# Description of collective magnetization processes with machine learning
models

Alexander Kornell1,2,∗, Lukas Exl3,4, Leoni Breth1,2, Johann Fischbacher1,2,
Alexander Kovacs1,2, Harald Oezelt1,2, Markus Gusenbauer1,2, Masao Yano5,
Noritsugu Sakuma5, Akihito Kinoshita5, Tetsuya Shoji5, Akira Kato5, Norbert J.
Mauser3, Thomas Schrefl1,2 1 Christian Doppler Laboratory for Magnet design
through physics informed machine learning, Viktor Kaplan-Straße 2E, 2700
Wiener Neustadt, Austria<EMAIL_ADDRESS>2 Department for Integrated
Sensor Systems, Danube University Krems, Viktor Kaplan-Straße 2E, 2700 Wiener
Neustadt, Austria  3 Research Platform MMM Mathematics-Magnetism-Materials c/o
Fak. Mathematik, University of Vienna, Oskar-Morgenstern-Platz 1, 1090 Vienna,
Austria  4 Wolfgang Pauli Institute c/o Fak. Mathematik, University of Vienna,
Oskar-Morgenstern-Platz 1, 1090 Vienna, Austria  5 Advanced Materials
Engineering Div., Toyota Motor Corporation, 1200, Mishuku Susono, Shizuoka
410-1193 Japan

###### Abstract

This work introduces a latent space method to calculate the demagnetization
reversal process of multigrain permanent magnets. The algorithm consists of
two deep learning models based on neural networks. The embedded Stoner-
Wohlfarth method is used as a reduced order model for computing samples to
train and test the machine learning approach. The work is a proof of concept
of the method since the used microstructures are simple and the only varying
parameters are the magnetic anisotropy axes of the hard magnetic grains. A
convolutional autoencoder is used for nonlinear dimensionality reduction, in
order to reduce the required amount of training samples for predicting the
hysteresis loops. We enriched the loss function with physical information
about the underlying problem which increases the accuracy of the machine
learning approach. A deep learning regressor is operating in the latent space
of the autoencoder. The predictor takes a series of previously encoded
magnetization states and outputs the next magnetization state along the
hysteresis curve. To predict the complete demagnetization loop, we apply a
recursive learning scheme.

††: Machine Learning: Science and Technology

##  1 Introduction

High performance permanent magnets play a key role in many technologies of
today’s life. They are necessary for cell phones, consumer electronic devices,
hybrid and electrical vehicles and sustainable energy production. Especially
the last two mentioned markets are emerging quickly and therefore the need for
permanent magnets will grow significantly in the future [1, 2, 3]. The state
of the art intermetallic phase of high performance permanent magnets is
Nd2Fe14B. Neodymium (Nd) is considered a critical element because of high
demand and possible supply risks [4]. Therefore, it is essential to avoid the
use of Nd while keeping the required magnetic properties for the target
applications [5].

High performance magnets are usually clustered by the magnetic field they are
able to provide and their resistance against opposite magnetic fields. These
macroscopic properties are reflected in the coercive field, the remanence and
the energy density product and emerge from microscopic magnetic properties and
the microstructure of the magnet [6]. Looking at this dependence it is clear
that the development of novel high performance magnets must be based on a
thorough understanding of the microscopic properties. The involved range of
length scales provides a particular challenge towards exact estimations at the
device level. Physical experiments are often costly and sometimes infeasible
and that is why accurate simulation strategies are necessary. This work
contributes to the multiscale description of permanent magnets, introducing a
methodology to predict magnetization reversal at the level of the granular
microstructure of a magnet.

The continuum theory of micromagnetism operates on a length scale, which is
small enough to model the microscopic properties of interest. Nevertheless, it
is worth mentioning that classical numerical algorithms based on this theory
are either costly in terms of computation time or require crude approximations
to bridge the length scales. These are the reasons why modern research on this
topic is now exploring algorithms based on machine learning as an efficient
alternative [7, 8, 9, 10]. Accurate machine learning models, however, can only
be developed if there is enough training data from experiments or simulations.
Within this approach the numerical algorithms, which are able to solve the
fundamental differential equations of micromagnetism, are the basis of the
models.

Successfully applying machine learning to calculate the figures of merit for
high performance magnets enables to pass microscopic properties through the
scales. This leads to a new approach of material design, which will provide an
efficient way to explore novel material combinations for rare-earth-free
permanent magnets.

##  2 Related Works

Our proposed approach is a latent space method. This means that one uses a
dimensionality reduction to create embedded images of the original input.
Since our method is data-driven, we use a convolutional autoencoder to span
the lower dimensional latent space. This is a deep learning model. Moreover,
the regressor is operating on the embedded images since they are of lower
dimension. The initial idea of this approach is coming from [11]. Kim et al.
present a generative model for synthesize fluid simulations based on a
convolutional autoencoder and an integrator operating in the latent space.
Reconstructing the velocity fields can be done approximately 700x faster than
re-simulating the data with the numerical algorithm.

In the field of micromagnetism, the work [10] proposes a machine learning
method to identify the nanocrytalline grain with the smallest switching field.
However, in this work the latent space is constructed by predefined attributes
of all grains composing the microstructure. In dynamic micromagnetism, Kovacs
et al. [9] recently proven that a latent space method can predict the magnetic
response of thin film magnets. They successfully use a machine learning
regressor within the latent space of convolutional autoencoder for the time
integration of the Landau-Lifshitz-Gilbert (LLG) equation [9]. Finally,
Schaffer et al. are comparing different dimensionality reduction methods for
constructing the latent space for a reduced order regressor [7].

##  3 Methods

###  3.1 Embedded Stoner-Wohlfarth Method

Our proposed method relies on deep learning. The main problem of these types
of machine learning models is that they heavily rely on a large dataset. In
that sense, it is crucial to use a very efficient simulation technique to be
able to calculate a rich enough dataset. This is the reason for choosing a
reduced order model for the necessary micromagnetic simulations to train and
test the machine learning method. The benefit of the embedded Stoner-Wohlfarth
method [12] is that it can capture low level information of interacting grains
while keeping the computation effort on the minimum.

The standard Stoner-Wohlfarth model describes a magnetic particle which is
small enough to only have a single domain and it has an elliptical shape.
Although the particle must be very small, it has to be large enough to have a
net magnetization. In this simple setup, important properties like the
switching field $H_{n}$ can be calculated analytically. In detail, the
external field required to switch the magnetization of the Stoner-Wohlfarth
particle is given by [13]:

$H^{0}_{n}=\frac{2(K_{u}+K_{d})}{\mu_{0}M_{s}}\textrm{,}$ (1)

assuming that the external field is exactly anti-parallel to the easy axis
$\mathbf{k}$ of the magnet, where $\mathbf{k}$ is a unit vector along the
magnetocrystalline anisotropy axis. Here $K_{u}$ is the anisotropy constant
and $K_{d}=\frac{1}{2}\mu_{0}M_{s}^{2}(N_{\bot}-N_{\parallel})$ is the
effective anisotropy constant coming from the demagnetization field [14] with
$N_{\parallel}$ and $N_{\bot}$ denoting the demagnetization factors of the
magnetization parallel and perpendicular to the easy axis. This field is also
referred to as the nucleation field of the magnet which is formally defined as
the minimal field which causes the magnetization to rotate out of its
preferred axis.

However, if the angle between the easy axis and the applied field is
$0<\Theta_{h}\leq\pi/2$, the field to switch the magnetization of the grain is
given by [10]:

$H_{n}=H_{n}^{0}\big{(}\cos^{2/3}(\Theta_{h})+\sin^{2/3}(\Theta_{h})\big{)}^{-3/2}\textrm{.}$
(2)

Figure 1:  Sketch of Stoner-Wohlfarth particle.

In addition, there is a reversible rotation of the magnetization of the grain,
whenever the applied field is not exactly anti-parallel to the easy axis of
the grain. Before the grain reverses its magnetization irreversible, the
magnetization rotates away from the easy axis until a critical angle is
reached. At this point the magnetization spontaneously changes its direction
towards a near direction of the external field. This process is schematically
shown in Figure 1. Finally, $\Theta$ converges to $\Theta_{h}$ if
$\mu_{0}M_{s}H_{ext}>>2(K_{u}+K_{d})$. The exact solution can be found in the
work [15] of J. Kalezhi and it will be used for the numerical calculations of
this work since it is very important in the practical use of the embedded
Stoner-Wohlfarth method.

The embedded Stoner-Wohlfarth method distributes virtual Stoner-Wohlfarth
particles in each grain whereby each particle has the same intrinsic magnetic
properties as the grain in which it is embedded. Within a grain, all Stoner-
Wohlfarth particles have the same easy axis. Therefore the intrinsic switching
field according to equation (2) is the same for all particles within the
grain. However, the field acting on the particle varies since it is a sum of
the external field and the local demagnetization fields of the other grains.
In addition, the switching field of a grain is defined by the reversal of the
embedded Stoner-Wohlfarth particles: It is expected that the complete grain
reverses its magnetization if the total field at one of the particles is
greater than the intrinsic switching field. This is justified because grains
of a Nd2Fe14B magnets contain no sites for domain wall pinning. In an external
field, an initially reversed domain will grow and the entire grain will become
reversed.

To avoid any naming complications, the total field $\mathbf{H}_{tot}$ is
introduced as the magnetic field at a Stoner-Wohlfarth particle coming from
the externally applied field and the demagnetization field of the grains. This
total magnetic field can be calculated by [10]:

$\mathbf{H}_{tot}(\mathbf{x})=\mathbf{H}_{ext}+\mathbf{H}_{demag}(\mathbf{x})\textrm{.}$
(3)

This equation explicitly uses the spatial coordinate $\mathbf{x}$ to underlie
that the total field $\mathbf{H}_{tot}$ is eventually different at each
discretization point of the system although the external field
$\mathbf{H}_{ext}$ is constant within the volume of interest. Here we neglect
ferromagnetic exchange interactions between the grains assuming that the
grains are separated by a non-magnetic intergranular phase.

Algorithm 1 Reduced Order Model

1:Inputs:

2: $\mathbf{M}_{0}$, $M_{s}$, $K_{a}$, $K_{d}$, $\mathbf{k}$,
$\mathbf{h}_{ext}$

3:Initialize:

4: MAXITER, $\Delta H_{ext}$, $H^{init}_{ext}$, $H_{ext}^{max}>0$
$H_{ext}\leftarrow H^{init}_{ext}$
$\mathbf{M}[H^{init}_{ext}][0]\leftarrow\mathbf{M}_{0}$

5:while $H_{ext}\leq H_{ext}^{max}$ do

6: for i = 1 to MAXITER do

7: for SW-particle j in microstructure do

8: $\mathbf{H}_{demag}\leftarrow$ calculate demagnetization field at SW-
particle j

9: $\mathbf{H}_{tot}\leftarrow\mathbf{H}_{demag}+H_{ext}\mathbf{h}_{ext}$

10: $\Theta_{h_{\textrm{tot}}}\leftarrow$ angle between total field and
$\mathbf{k}$

11: $H_{switch}\leftarrow$ equation (2)

12: if $||\mathbf{H}_{tot}||>H_{switch}$ then

13: switch all SW-particles of grain

14: $\mathbf{M}[H_{ext}][j]\leftarrow$ [15] adapted to magnetization

15: else

16: $\mathbf{M}[H_{ext}][j]\leftarrow$ [15] adapted to magnetization

17: end if

18: end for

19: if no particle switched then

20: break

21: end if

22: end for

23: $H_{ext}\leftarrow H_{ext}+\Delta H_{ext}$

24:end while

25:return $\mathbf{M}$

Algorithm 1 describes the embedded Stoner-Wohlfarth method in pseudo code. To
clarify uncertainties, it is worth to mention that the strength of the
external field is treated as a scalar value $H_{ext}$ while its direction is
given by the unit vector $\mathbf{h}_{ext}$ which is pointing in the opposite
direction of the initial saturation field. The index $j$ selects the
magnetization or the total field of the $j$-th particle.
The first step to simulate the magnetization reversal processes is to create a
multigrain structure where each grain has slightly different intrinsic
magnetic properties since this is the case in nature too. In this setup, the
difference is restricted to the easy axes $\mathbf{k}$ because the model is
strictly built of cubic grains. However, these properties of the model are the
inputs of Algorithm 1 in addition to an initial state of the magnetization.
Next, the size of the external field steps is defined. Furthermore, an
iteration for each field step is started where the total field is calculated
at the beginning. In addition, the magnetization is computed by the equations
of the Stoner-Wohlfarth model where a check if the total field exceeds the
switching field is needed. If this is the case, it is necessary to continue
the iteration since the total field for the other grains changes
significantly. Lastly the algorithm can move on to the next external field
step if the system reaches a stable state where no more switching fields are
exceeded.

###  3.2 Demagnetization Field

A very important computational step of Algorithm 1 is the calculation of the
demagnetization field in line 6. Within the framework of the embedded Stoner-
Wohlfarth method, it is assumed that each grain is uniformly magnetized and
that they are separated by an isolating boundary phase. Because
$\nabla^{\prime}\cdot\mathbf{M}(\mathbf{x}^{\prime})=0$ within each grain,
there are no field sources inside a grain. As a consequence, the demagnetizing
field can be expressed as sum over surface integrals [10]:

$\mathbf{H}^{eSW}_{demag}(\mathbf{x})=\frac{1}{4\pi}\sum_{i}^{N_{g}}\sum_{j}^{N_{s_{i}}}\int_{\partial
V^{\prime}_{ij}}\mathbf{M}\cdot\mathbf{n}_{ij}\frac{(\mathbf{x}-\mathbf{x}^{\prime})}{||\mathbf{x}-\mathbf{x}^{\prime}||^{3}}dS^{\prime}_{ij}\textrm{.}$
(4)

where $N_{g}$ is the total number of grains and $N_{s_{i}}$ is the number of
surfaces of the $i$-th grain. The vector $\mathbf{n}_{ij}$ is the outer normal
to surface $k$ of grain $j$.

Finally, it is possible to replace the surface integral by integrals over all
edges of the microstructure by again using the fact that the grains are
uniformly magnetized. Guptasarma and Singh [16] reported the necessary
formulas for calculating the demagnetization field for uniformly magnetized
polyhedrons. It is worth to mention that the necessary equations can be
implemented efficiently in a vectorized formulation. This is especially
important when the algorithm is implemented in a high level programming
language like Python. Without any further details, the final results are
mainly consisting of the following matrix vector product:

$\vec{H}^{ROM}_{demag}=\underline{\underline{B}}\cdot\vec{M}\textrm{,}$ (5)

where $\vec{M}$ and $\vec{H}^{ROM}_{demag}$ hold the magnetization and the
demagnetizing field of all grains, respectively. The matrix
$\underline{\underline{B}}\in R^{3N_{p}\times 3N_{g}}$ contains the spatial
components of the structure, where $N_{p}$ is the total number of Stoner-
Wohlfarth particles. One of the main benefits of this vectorized equation (5)
of the demagnetization field is that it can be implemented very efficiently on
graphic processors since it is possible to highly parallelize the computation
[12].

Nonetheless even an optimized algorithm has the drawback that the calculation
of the demagnetization field at each iteration has a theoretical runtime of
the order $\mathcal{O}(N_{g}N_{p})$ because it needs $N_{p}$ multiplications
for every grain. Going back to physics, one notices that the demagnetization
field decreases with increasing distance. This clearly indicates that
neighboring grains have an higher impact to the demagnetization field than
elements which are far away from each other. This means that grains which are
far away from the point of interest can be clustered without a big lack in
accuracy. A similar approximation is successfully applied by the Barns-Hut
tree method which is used to solve N-body problems in astrophysics or
electrodynamics [17, 18]. The tree code reduces the theoretical run time to
the order $\mathcal{O}(nlog(n))$ and there are libraries available which
implement this method. To use this idea, one needs to start by enumerating the
grains by their distances and add their values accordingly to $\vec{M}$ and
$\underline{\underline{B}}$. This enables to calculate the matrix vector
product by hierarchical matrices which are referred to as
$\mathcal{H}^{2}$-matrices. The main benefit of the $\mathcal{H}^{2}$-matrices
is that the calculation of the product can even be done with nearly a linear
order [19]. One example is the Python package h2tools [20].

###  3.3 Machine Learning Methods

The machine learning approach proposed in this work consists of two models: 1)
an convolutional autoencoder for dimensionality reduction, 2) a neural network
regressor for predicting magnetization states. In addition, the predictor is
applied to the latent space of the autoencoder. Figure 2 outlines the method
visually.

The algorithm starts by using the encoder of the autoencoder to obtain
embedded images of the magnetization at the $i-t,...,i-1$ steps of the
demagnetization curve. Furthermore, the predictor model takes these images in
the latent space and the external field of the $(i-1)$-th step of the
hysteresis as the input to predict the magnetization in the latent space at
the $i$-th step of the demagnetization curve. Finally the decoder must be used
to transform the new magnetization state back to the real space. This
predictor can be used to estimate the complete hysteresis of a magnetic
material by only using the first $k$ magnetization states since it can be
applied to its own predictions. In total this represents a method where the
reduced order model is replaced by a machine learning model and this should
lead to a substantial speedup. However, the regressor must calculate accurate
results that one can use this method.

Figure 2:  Schematic overview of the method to predict the next step of the
hysteresis by using an autoencoder and a predictor in the latent space.

####  3.3.1 Empirical Risk Minimization

The machine learning models used in this work are trained by supervised
learning. The goal of this technique is to establish models which can reliably
predict a variable $y\in\mathcal{Y}$, given an input $x\in\Omega$. A commonly
used technique to obtain such models is called emprical risk minimization.
Given the empirical risk $\mathcal{L}_{z}$ and the loss function
$\mathit{L}(\hat{y},y)$, the empirical risk minimization problem can be
defined as [21]:

$\hat{f}_{\mathit{H},z}={\arg\min}_{f\in\mathit{H}}\\{\mathcal{L}_{z}(f)\\}={\arg\min}_{f\in\mathit{H}}\left\\{\frac{1}{m}\sum^{m}_{i=1}\mathit{L}(\hat{y}_{i},y_{i})\right\\}\textrm{,}$
(6)

In detail, a loss function takes the predicted value $\hat{y}$ and the ground
truth $y$ and calculates a difference which is important for the underlying
problem. It is necessary to understand that there is no ’one fits all’ loss
function because the importance of the underlying error can vary with the
actual problem. However, some commonly used examples are the mean squared
loss, the cross entropy loss and the hinge loss. Furthermore, it is possible
to enrich the loss function by information of the underlying problem [9]. One
of the main topics of this work is to engineer appropriate loss functions to
predict the magnetization reversal process. This is done by including parts of
micromagnetism, like the preservation of the norm of the magnetization vector,
to this function. Furthermore, recursive learning of the predictor is realized
by the loss function.

However, training a machine learning model with empirical risk minimization is
basically a large algebraic minimization problem. Since it is expected that
the dataset is large, one usually uses gradient based minimization techniques.
Although the standard stochastic gradient descent method (SGD) is already a
viable algorithm for training a machine learning model, there are ways to
improve the algorithm by using additional information of the function to
minimize. One example is the Adam algorithm which uses the momentum of the
gradient and an adaptive learning rate to update the property of interest.
This method converges faster than the classical stochastic gradient descent
method and it has also a higher probability to escape local minima if the loss
surface is noisy [22]. However, very commonly used machine learning models for
dimensionality reduction are convolutional autoencoders based on neural
networks. For that type of models there is a further improvement to the Adam
algorithm which is named the Nadam method. The Nadam method essentially speeds
up the training of a convolutional autoencoder by reducing the number of
epochs needed to converge to a minimum [23].

####  3.3.2 Convolutional Autoencoder

Formally an autoencoder is an unsupervised neural network which tries to copy
the input to its output [21]. However, the interesting part is that there is
one central intermediate layer which has a different dimension than the input
layer. This technique is commonly used for dimensionality reduction or feature
extraction. The focus of this work lays on the first case where the
intermediate layer compresses the data to the most important features which
are necessary to obtain the original state, assuming that the method is
successfully reducing the dimension of the underlying problem.

An autoencoder consists of two parts, namely the encoder and the decoder. The
encoder maps the input data to a latent space where the dimension of this
space is defined by the central intermediate layer. In addition, the decoder
is able to decode the latent space representation of the input back to the
real space. Since the representation in the latent space is unknown upfront,
both parts of the autoencoder are trained together where the loss is usually
calculated in the real space and the weights and biases of both models are
updated at once. The following equation describes the empirical risk of an
autoencoder:

$\mathcal{L}_{AE}=\frac{1}{m}\sum_{k=1}^{m}\mathit{L}(D(E(x_{k})),x_{k})\textit{,}$
(7)

where $E:\Omega\rightarrow\mathcal{F}_{\Omega}$ is the encoder and
$D:\mathcal{F}_{\Omega}\rightarrow\Omega$ is the decoder. In addition, the
space $\mathcal{F}_{\Omega}\subset R^{d}$ is the latent space. When using this
technique for dimensionality reduction, the dimension of the latent space is
lower than the dimension of the real space. In that case one speaks of an
undercomplete autoencoder. On the other hand an overcomplete autoencoder would
have a latent space of higher dimension than the original space.

Convolutional autoencoders are neural networks which utilize convolutional
layers instead of dense layers. Convolutional layers restrict the input for
each neuron to just a few neurons of the last layer which are geometrically
nearby each other. This is done by using a filter or kernel with a specific
size for calculating the output of a layer. In detail, the filters given by
matrices with a previously defined dimension are applied to all input channels
of the layer and the depth of the output image is equal to the number of
filters applied within the layer [24]. This essentially means that a
convolutional layer transforms a volume of values into another volume of
values which eventually has a different dimension. Finally, the activation
function is applied component-wisely. As a consequence the total number of
trainable parameters is drastically reduced. This technique allows to identify
local characteristics of the image at the first layers while global
characteristics are recognized within the deeper layers of the network.
Although this method was initially developed for image recognition, it also
shows its strength when applied to physical problems because local
interactions eventually have an other impact to the system than global ones.

In addition, convolutional layers can include downsampling directly within the
layer by using a stride. This value defines how far the filter is moved from
one calculation to the next. For example a stride of two means that the filter
is shifted by two units. This procedure reduces the dimensionality already
within the convolutional layer because the number of elements in the output
matrix is roughly the number of total input values divided by the stride in
each dimension and for each filter. The main benefit of this procedure is that
it introduces trainable parameters for the dimensionality reduction step. This
is not the case for pooling layers since they are consisting of fixed
operations. Nevertheless, there is a drawback because the increased number of
weights and biases require more computational power to be trained. However,
recent researches show that there are applications where convolutional layers
with strides outperform the classical setup with pooling layers [25].

Using this concept, one usually builds the encoder by start stacking
convolutional layers. Normally, the dimension is already reduced in the first
layers by adding a stride or pooling layers. Furthermore, there are dense
layers in use when getting towards the latent space. The decoder reverses this
process by first enlarging the dimension with dense layers and later on using
transposed convolutional layers to get back the dimension of the real space. A
transposed convolutional layer is the inverse of a convolutional layer which
essentially means that it can decode the features of the input [26].

Figure 3:  Schematic overview of the reconstruction of one magnetization state
by using the autoencoder. Reconstructing a complete hysteresis requires to
apply this method to every state along the demagnetization curve.

##  4 Results

###  4.1 Data sets

The dataset used for the following experiments consists of simulated data
without practical experiments. The microstructures are generated by Neper
which is a tool for generating polycrystalline structures and meshing. In
detail, each structure is three dimensional and it consists of 64 cubic grains
aligned uniform in all directions. Furthermore, all grains have the same
intrinsic magnetic properties besides their easy axes. The easy axes are
randomly defined within an angle of 35 degree to the z-coordinate. In
addition, the embedded Stoner-Wohlfarth method is used to simulate the
magnetization reversal process of the microstructures. In detail, the external
field is varied from 8 to -8 Tesla in 1000 steps. This means that the external
field is reduced by -0.016 Tesla at each field step. The complete dataset
consists of 10 full simulation which leads to 10 000 different magnetization
states. For each full simulation, a different random distribution of the easy
axes is used. From this set nine simulations are used for training while one
simulation is the testset.

Figure 4 shows the simulated magnetization reversal process of one
microstructure. Furthermore, there are four magnetization states of the
structure visualized explicitly. To recap, this simulation corresponds to 1000
samples of the data set.

Figure 4:  Simulated magnetization reversal process of multigrain magnet,
color shows the magnetization along the z-axis.

Figure 4 indicates that the grains are switching in the range of -8 to -3
Tesla because the magnetization of the complete microstructure does not change
significantly before. This essentially means that only approximately 40 % of
the training data contains grains with a reversed magnetization. To mitigate
that problem, the training samples are weighted due to the change of the
$z$-component of the magnetization state in relation to the previous external
field step. The following formula provides the calculation scheme of the
weights:

$\eqalign{w_{1}=0\textrm{,}\cr\eqalign{w}_{i}=\sum_{j=1}^{64}|m_{j,z}(H_{ext}^{(i)})-m_{j,z}(H_{ext}^{(i-1)})|\textrm{
for }i=2,...1000\textrm{.}}$ (8)

Here $m_{j,z}(H_{ext}^{(i)})$ denotes the magnetization along the z-axis of
the $j$-th grain at the $i$-th external field step. Equation (8) assigns
larger weights to training samples which have a significant change in
magnetization in relation to the previous field step. This further has the
effect that the weights and biases of the models are more sensible to the
training samples which are partially reversed. In total this should improve
the performance of the models in the part of the demagnetization curve where
the magnetization is reversed.

The implementations of the machine learning models are done with the machine
learning framework Tensorflow 2 [27] used in Python 3.8. This is an open
source software package which was initially developed by Google and it
provides a set of software tools to efficiently use machine learning.
Furthermore, the activation function for all layers is the exponential linear
unit [28]. In addition, the kernels are initialized by the Tensorflow 2
function ’he_normal’ which draws the initial weights from a truncated normal
distribution around 0 where the standard deviation is related to the number
input weights. He et. al. have shown that this kernel initialization increases
the models accuracy in their work [29] in 2015.

The training of the network is done by backpropagation [21] where 20% of the
training samples are used for validation with a batch size of 64. Moreover,
the Nadam algorithm is used as the optimizer. The initial learning rate of the
method is set to 0.0001. In addition, this parameter of the optimizer is
reduced during training by the Keras callback ’ReduceLROnPlateau’ if the
monitored validation loss stays constant though a certain amount of epochs.
Additionally early stopping is used to avoid overfitting. This means that if
the validation loss does not decrease significantly after some epochs, the
training is stopped before reaching the maximal number of iterations through
the training samples.

###  4.2 Encoding magnetic states

The first question to answer is how to define an appropriate empirical risk
minimization problem for training the autoencoder. Since this is a regression
problem, the standard mean squared error is a reasonable loss function to
choose. In this case, this error is calculated as the sum of squared
differences of each entry of the magnetization vector. However, micromagnetism
also states that the length of the magnetization vector must be constant. This
implies that the loss function can be enriched by a term which takes care of
the preservation of the norm. Using these ideas, one can formulate the
candidate loss function by the following equation [9]:

$\mathit{L}_{AE}(\hat{x},x)=a_{1}\underbrace{\sum_{i,j,k=1}^{4}\sum_{l=1}^{3}(x_{ijkl}-\hat{x}_{ijkl})^{2}}_{\mathit{L}_{MSE}}+a_{2}\underbrace{\sum_{i,j,k=1}^{4}\left(1-\sqrt{\sum_{l=1}^{3}\hat{x}_{ijkl}^{2}}\right)^{2}}_{\mathit{L}_{Norm}}\textrm{,}$
(9)

where $\hat{x}$ is the reproduced image in the real space of the autoencoder.
The indices $i,j$ and $k$ number the grains on a regular geometric grid.
Furthermore, $l$ describes the magnetization component. Therefore $x_{ijkl}$
defines the $l$-th component of the unit vector of the magnetization in grain
$ijk$.

The first term of the right hand side of equation (9) defines the mean squared
error for this example. The second term of this formulation penalizes the
model if the reproduced image does not conserve the norm. Furthermore, the
parameters $a_{1}$ and $a_{2}$ describe the relative weights of the terms
since the model eventually produces better results if they are not uniformly
weighted. Last, it is worth to note that the loss function accounts only the
mean squared error if the parameter $a_{2}$ is set to zero. Figure 5 compares
loss functions for the convolutional autoencoder depending on $a_{1}$ and
$a_{2}$.

The architecture of the convolutional autoencoder used for the following
experiments is described in Table 1. The encoder consists of one convolutional
layer with four filters where each filter has the dimension of $2\times
2\times 2$ and it uses a stride of 2. In addition, there are three dense
layers. The latent space consists of 16 neurons which means that the
dimensionality is reduced by a factor of 12 since the image is described by
192 floats in the real space. The decoder applies the same setup of layers in
the reversed order. In total this leads to 4983 trainable parameters.

Layer (type) | Output Shape | Param #
---|---|---
input (InputLayer) | [(None, 4, 4, 4, 3)] | 0
conv3d (Conv3D) | (None, 2, 2, 2, 4) | 100
flatten (Flatten) | (None, 32) | 0
dense (Dense) | (None, 32) | 1056
dense_1 (Dense) | (None, 32) | 1056
dense_2 (Dense) | (None, 16) | 528
dense_3 (Dense) | (None, 16) | 272
dense_4 (Dense) | (None, 16) | 272
dense_5 (Dense) | (None, 32) | 544
dense_6 (Dense) | (None, 32) | 1056
reshape (Reshape) | (None, 2, 2, 2, 4) | 0
output (Conv3DTransp) | (None, 4, 4, 4, 3) | 99
Total params: 4,983 |  |
Trainable params: 4,983 |  |
Non-trainable params: 0 |  |
Table 1:  Autoencoder architecture for evaluating different loss functions.

Figure 5 shows the mean squared error and norm error dependent on the number
of epochs for the different loss functions. Note that the terms are separately
tracked without weighting during the training which ensures that they are
quantitatively comparable for the different setups. The graphs indicate that
the model does not loose the accuracy in terms of the mean squared error by
adding the penalization term. This means that including the norm in loss
function reduces the mean squared error simultaneously. Nevertheless, it is
necessary to mention that this behavior is only shown if the norm related term
is weighted smaller than the term which calculates the mean squared error. In
numbers this means that setting the relative weights $a_{1}$ to 0.85 and
$a_{2}$ to 0.15 enables the model to reach approximately the same minimum of
the mean squared error by less epochs. However, it is worth to note that
weighting the norm error equally to the mean squared error introduces a
significantly higher mean squared error.

As expected, adding the physical information of the preservation of the norm
to the loss function significantly reduces the error in the length of the
predicted magnetization vectors. In addition, the relative difference between
the norm error on the training set and the norm error on the validation set
decreases by enlarging the relative weight $a_{2}$. However, the preservation
of the norm is just additive information since this term does not relate the
predicted magnetization to the ground truth. This means that weighting the
norm error to high would lead bad predictions since the model would not learn
the underlying data distribution in any sense.

Figure 5:  Autoencoder loss contributions for different weights $a_{1}$ and
$a_{2}$. Blue: $a_{1}=1.00$ and $a_{2}=0.00$; Green: $a_{1}=0.85$ and
$a_{2}=0.15$; Red: $a_{1}=0.50$ and $a_{2}=0.50$.

Figure 6 shows the magnetization reversal process of the test microstructure
simulated by the embedded Stoner-Wohlfarth method. In addition, there are the
plots of the reproduced magnetization of the structure where the autoencoder
is trained with the different loss functions which are analyzed before. The
graphs are created by predicting the magnetization state of all steps of the
external field and plotting the $z$-component correspondingly.

In general the graphs indicate that the used model is able to reproduce the
demagnetization curve. Furthermore, one can see that the mean squared error is
a good error measurement for the underlying problem. Nevertheless, there are
differences in the performance related to the configuration of the loss
function. In detail, the plots show that using a model that minimizes a loss
which equally weights the norm error and the mean squared error performs worse
than a model which weights the mean squared error higher than the norm error.
However, weighting the norm conservation term by 15% provides a model with a
reasonably good reconstruction. Nonetheless it is worth to note that the model
in this test case gets inputs with a normalized magnetization. So for this
case, the autoencoder predicts accurately if it does not change the norm but
this is eventually not true if there is an additional machine learning model
which predicts the magnetization state of the microstructure in the latent
space. If the predictor produces embedded images which are not normalized in
the real space, the autoencoder must be able to do this normalization. In
total this is an additional argument to add the norm error to the loss
function since the mean squared error on its own does not include
normalization.

Figure 6:  Ground truth and reproduced magnetization states of the
convolutional autoencoder for different weights $a_{1}$ and $a_{2}$. Orange:
ground truth; Blue: $a_{1}=1.00$ and $a_{2}=0.00$; Green: $a_{1}=0.85$ and
$a_{2}=0.15$; Red: $a_{1}=0.50$ and $a_{2}=0.50$.

###  4.3 Predictor

The autoencoder can be used for dimensionality reduction but the main goal is
to predict the complete magnetization reversal process of unseen
microstructures. This means that the regressor must be able to predict
accurate results recursively from its own predictions. In general, the dataset
to train and test the latent space predictor is the same as for the
autoencoders but it needs to be preprocessed. In detail, the first thing to do
is to choose a single model for the dimensionality reduction. In the following
experiments this is the autoencoder given by Table 1 with the loss weights
$a_{1}=0.85$ and $a_{2}=0.15$. The encoder of this model is used to obtain the
embedded images of all states of magnetization of the 10 simulations. Again
one full simulation is taken as the unseen test set to evaluate the models.
The next step is to concatenate the training samples in batches of $t$
consecutive magnetization states. To avoid misunderstandings, it is worth to
note that one magnetization state can be part of at most $t$ different input
batches. In addition, the strength of the external field at the $(i-1)$-th
step is added to these batches. In total this leads to $(1000-t)$ inputs for
each simulation.

Figure 7:  Training process of a predictor model with t=4 and l=7.

However, the next question to answer is how to define a loss function for an
empirical risk minimization problem for the model. In general the predictor
should be able to estimate the complete demagnetization curve and not only the
next magnetization state. Therefore it must be able to provide accurate
results even if the inputs are magnetization states which are calculated by
the model itself. This is the reason why a standard mean squared error loss
function which only measures the difference to the next state in magnetization
eventually fails due to interfering errors. To avoid this problem, one can use
a loss function which also accounts for a mean squared error of further
predictions. In summary one training sample of the predictor has the following
structure:

$b^{(i)}:=(H_{ext}^{(i-1)},e^{(i-t)},...e^{(i)},e^{(i+1)},...,e^{(i+l-1)})=$
$=(H_{ext}^{(i-1)},E(\vec{M}(H_{ext}^{(i-t)})),...,E(\vec{M}(H_{ext}^{(i)})),E(\vec{M}(H_{ext}^{(i+1)})),...,E(\vec{M}(H_{ext}^{(i+l-1)})))\textrm{,}$

where $e^{(i)}:=E(\vec{M}(H_{ext}^{(i)}))$ denotes the encoded magnetization
of the $i$-th external field step. The $i$-th batch contains the encoded
magnetization patterns of the $t$ previous field steps and the $l$ future
field steps. Furthermore, the following notation is introduced:

$\epsilon^{(j)}:=\cases{e^{(j)}\quad\textrm{ if }j<i\\\
\hat{e}^{(j)}\quad\textrm{ else }}{\textrm{.}}$ (10)

Here $\hat{e}^{(j)}:=P(H_{ext}^{(i)},\epsilon^{(j-1)},...,\epsilon^{(j-t)})$
is the output of the predictor at the j-th step of the demagnetization curve.
Using these notations, the mean squared error of one prediction is given by
the following formula:

$\mathit{L}_{j}(\hat{e}^{(j)},e^{(j)})=\sum_{t=1}^{d}(e^{(j)}_{t}-\hat{e}^{(j)}_{t})^{2}\textrm{,}$
(11)

where $d$ is the number of latent space variables. Finally, it is necessary to
sum up the mean squared errors for each prediction to get the loss function:

$\mathit{L}_{PRED}(\hat{e},e)=\sum_{j=0}^{l-1}\mathit{L}_{j}(\hat{e}^{(j)},e^{(j)})\textrm{.}$
(12)

Figure 7 outlines the training process of the predictor graphically. The
sketch uses a model where $t=4$ and $l=7$. In this case, the neural network
takes the four previous magnetization states as an input. Furthermore, it is
applied recursively to estimate the next seven magnetization states and the
sum of the mean squared error of these predictions is minimized during the
training process.

Figure 8:  Demagnetization curve of microstructure calculated by the embedded
Stoner-Wohlfarth method and predicted by latent space regressor with different
values of $t$ and $l$.

Figure 8 shows the complete demagnetization curve of the test microstructure
predicted by the model with different configurations of $t$ and $l$. In
addition, there is the ground truth which is the calculated hysteresis by the
embedded Stoner-Wohlfarth method. The predictor model takes as an input the
first $t$ encoded magnetization states from the test set and calculates the
complete magnetization reversal recursively.

Although the minimization of the loss function ensures that each of the
configurations is able to do at least 4 predictions with sufficiently small
error, it is clearly displayed that $l$ must be greater than 4 to enable the
model to calculate the complete demagnetization curve reasonably well. In
detail, the model where $t=4$ and $l=4$ does not reproduce the hysteresis and
the oscillating predictions are eventually related interfering errors.
However, if $l=8$ or greater the model is able to estimate the curve of
interest. Nevertheless, it is interesting to see that the configuration $t=8$
and $l=8$ scores worse than the model with $t=4$ and $l=8$. In total the
simulations show that the configurations where $t$ is is at least one third
less than $l$ outperform the other combinations.

##  5 Summary and conclusion

This work outlines a machine learning approach to estimate the demagnetization
reversal process of multigrain microstructures. The method uses a
convolutional autoencoder and a latent space regressor based on neural
networks. Furthermore, the embedded Stoner-Wohlfarth method is used to
generate the dataset. This numerical method is efficient enough to calculate a
dataset to empower deep learning for predicting the hysteresis loop of an
unseen microstructure.

The analysis shows that it is possible to reproduce the demagnetization curve
with an autoencoder trained by the loss function given by equation (9).
However, the dominating term of the loss function must be the mean squared
error since weighting the norm related term too high leads to worse
reconstructions. Nevertheless, the results indicate that including the norm
error reduces the computational expense for training the model. Furthermore,
it eventually includes normalization capabilities to the autoencoder.

In summary the experiments with the latent space predictor prove that it is
possible to predict the hysteresis curve of an unseen microstructure by a
machine learning model operating in the latent space of an autoencoder.
However, there are some important facts which need to be fulfilled to enable
the method to succeed. The first is that the model based on neural networks
must have enough hidden layers to estimate the underlying distribution.
Furthermore, it is necessary to recursively apply the predictor during
training by using a loss function which takes the mean squared error of future
predictions into account. Moreover, simply increasing the number of inputs of
previous magnetization states does not increase the model performance as long
as the number of future predictions within the loss function is not changed
accordingly. The discussed results of the predictor take as an input the first
$t$ encoded magnetization states. However, different starting points lead to
the same results which most likely is related to the fact that the training
samples are differently weighted. To recap, datapoints where irreversible
magnetization reversal occurs have a higher impact to the machine learning
models than datapoints where only reversible magnetization rotation occurs.

A key finding of the work is that loss function engineering is necessary to
apply common machine learning techniques to micromagnetism. One insight which
was gained during this work is that adding the norm error to the loss function
of the autoencoder did provide better results for this setup than just taking
the standard mean squared error. In detail, adding this term with an
appropriate weight led to the same results for the mean squared error while
reducing the error in the norm. Moreover, the inclusion of this term reduced
the training time of the model since it was able to find a minimum of the loss
functions after less epochs. Furthermore, we were able to find differences
between the ground truth and the reproduced images which gives room for
improvement for this method.

In addition the experiments with the predictors clearly stated the fact that
it is necessary to account the error of future prediction during the training
of the model to enable it to predict the complete hysteresis curve. Therefore
a recursive loss function was developed which respects the mean squared error
of the future predictions. Furthermore, we were able to show that the neural
network for calculating the following steps of the demagnetization curve must
be deep enough to approximate the magnetization states accurately. Finally, we
evaluated architectures for the predictor by trying different combinations of
the hyperparameters which describe the number of input magnetization states
and the number of recursive iterations during the training. These experiments
showed that ratio between these parameters effects the accuracy of the model.

The goal of this work was a proof of concept of the machine learning method
for calculating the demagnetization curve of magnets with simple granular
geometry. Nevertheless, it is possible to extend this method for synthetic
polycrystalline material. However, more complex structures eventually need
more complex methods. In detail, the experiments with the norm error showed
that adding information to the autoencoder improves the model. Therefore it
would be interesting to apply additional information to the loss function.
This can be a term which penalizes the model if the predicted state is not in
its equilibrium. This idea comes from the work of [30] which introduces
physical informed neural networks where the underlying equations of
micromagnetism are part of the loss function. Moreover, another promising way
to improve the predictor is to enrich its input by information about the
microstructure and the intrinsic magnetic properties of the material. However,
since high dimensionality is expected to be a problem one has to find a way to
represent the microstructure itself within the latent space.

## Acknowledgments

Financial support by the Austrian Federal Ministry for Digital and Economic
Affairs, the National Foundation for Research, Technology and Development and
the Christian Doppler Research Association is gratefully acknowledged, as well
as the support by the Austrian Science Fund (FWF) under grant No. P31140-N32
and grant No. F65.

## References

  * [1] Meszaros F, Shatanawi M and Ogunkunbi G A 2020 Period. Polytech. Transp. Eng. 49 93–101 ISSN 1587-3811, 0303-7800 URL https://doi.org/10.3311/pptr.14037
  * [2] Ntanos S, Skordoulis M, Kyriakopoulos G, Arabatzis G, Chalikias M, Galatsidas S, Batzios A and Katsarou A 2018 Sustainability 10 2626 ISSN 2071-1050 URL https://doi.org/10.3390/su10082626
  * [3] Yang Y, Walton A, Sheridan R, Güth K, Gauß R, Gutfleisch O, Buchert M, Steenari B M, Van Gerven T, Jones P T and Binnemans K 2016 J. Sustain. Metall. 3 122–149 ISSN 2199-3823, 2199-3831 URL https://doi.org/10.1007/s40831-016-0090-4
  * [4] Bobba S, Carrara S, Huisman J, Mathieux F and Pavel C 2020 Critical raw materials for strategic technologies and sectors in the eu - a foresight study, 2020 Tech. rep. European Commission, Joint Research Centre
  * [5] Binnemans K, Jones P T, Müller T and Yurramendi L 2018 J. Sustain. Metall. 4 126–146 ISSN 2199-3823, 2199-3831 URL https://doi.org/10.1007/s40831-018-0162-8
  * [6] Fischbacher J, Kovacs A, Gusenbauer M, Oezelt H, Exl L, Bance S and Schrefl T 2018 J. Phys. D: Appl. Phys. 51 193002 ISSN 0022-3727, 1361-6463 URL https://doi.org/10.1088/1361-6463/aab7d1
  * [7] Schaffer S, Mauser N J, Schrefl T, Suess D and Exl L 2021 IEEE Transactions on Magnetics
  * [8] Exl L, Mauser N J, Schaffer S, Schrefl T and Suess D 2021 Journal of Computational Physics 444 110586
  * [9] Kovacs A, Fischbacher J, Oezelt H, Gusenbauer M, Exl L, Bruckner F, Suess D and Schrefl T 2019 J. Magn. Magn. Mater. 491 165548 ISSN 0304-8853 URL https://doi.org/10.1016/j.jmmm.2019.165548
  * [10] Exl L, Fischbacher J, Kovacs A, Özelt H, Gusenbauer M, Yokota K, Shoji T, Hrkac G and Schrefl T 2018 J. Phys. Mater. ISSN 2515-7639 URL https://doi.org/10.1088/2515-7639/aaf26d
  * [11] Kim B, Azevedo V C, Thuerey N, Kim T, Gross M H and Solenthaler B 2018 CoRR abs/1806.02071 (Preprint 1806.02071) URL http://arxiv.org/abs/1806.02071
  * [12] Fischbacher J, Kovacs A, Exl L, Kühnel J, Mehofer E, Sepehri-Amin H, Ohkubo T, Hono K and Schrefl T 2018 Scripta Mater. 154 253–258 ISSN 1359-6462 URL https://doi.org/10.1016/j.scriptamat.2017.11.020
  * [13] Özelt H 2018 Exchange coupled ferri-/ferromagnetic composite nanomagnets Ph.D. thesis Technische Universität Wien
  * [14] Kronmüller H, Durst K D and Martinek G 1987 J. Magn. Magn. Mater. 69 149–157 ISSN 0304-8853 URL https://doi.org/10.1016/0304-8853(87)90111-9
  * [15] Kalezhi J 2011 Modelling data storage in nano-island magnetic materials Ph.D. thesis University of Manchester, Faculty of Engineering and Physical Sciences
  * [16] Guptasarma D and Singh B 1999 Geophysics 64 70–74 ISSN 0016-8033, 1942-2156 URL https://doi.org/10.1190/1.1444531
  * [17] Iwasawa M, Namekata D, Sakamoto R, Nakamura T, Kimura Y, Nitadori K, Wang L, Tsubouchi M, Makino J, Liu Z, Fu H and Yang G 2020 The International Journal of High Performance Computing Applications 34 615–628 ISSN 1094-3420, 1741-2846 URL https://doi.org/10.1177/1094342020943652
  * [18] Gan Z and Xu Z 2014 Sci. China Math. 57 1331–1340 ISSN 1674-7283, 1869-1862 URL https://doi.org/10.1007/s11425-014-4783-5
  * [19] Börm S 2005 Computing 77 1–28 ISSN 0010-485X, 1436-5057 URL https://doi.org/10.1007/s00607-005-0146-y
  * [20] Mikhalev A Y and Oseledets I V 2015 Numer. Linear Algebra Appl. 23 230–248 ISSN 1070-5325 URL https://doi.org/10.1002/nla.2021
  * [21] Schaffer S, Mauser N, Schrefl T, Suess D and Exl L 2021 Machine learning methods for the prediction of micromagnetic magnetization dynamics Master’s thesis Universität Wien
  * [22] Kingma D P and Ba J 2017 arXiv:1412.6980 (Preprint 1412.6980)
  * [23] Dozat T 2016 (n.p.)
  * [24] Buduma N 2017 Fundamentals of Deep LearningDesigning Next-Generation MachineIntelligence Algorithms 1st ed (O’Reilly Media)
  * [25] Brown C J, Mullaney J R, Morrison J, Martin J W and Trombley T J 2015 USGS Publications Warehouse ISSN 2328-0328 (Preprint 1412.6806) URL https://doi.org/10.3133/sir20155057
  * [26] Zhang Y 2018 (n.p.)
  * [27] Abadi M, Agarwal A, Barham P, Brevdo E, Chen Z, Citro C, Corrado G S, Davis A, Dean J, Devin M, Ghemawat S, Goodfellow I, Harp A, Irving G, Isard M, Jia Y, Jozefowicz R, Kaiser L, Kudlur M, Levenberg J, Mané D, Monga R, Moore S, Murray D, Olah C, Schuster M, Shlens J, Steiner B, Sutskever I, Talwar K, Tucker P, Vanhoucke V, Vasudevan V, Viégas F, Vinyals O, Warden P, Wattenberg M, Wicke M, Yu Y and Zheng X 2015 TensorFlow: Large-scale machine learning on heterogeneous systems software available from tensorflow.org URL https://www.tensorflow.org/
  * [28] Clevert D A, Unterthiner T and Hochreiter S 2016 arXiv:1511.07289 (Preprint 1511.07289)
  * [29] He K, Zhang X, Ren S and Sun J 2015 IEEE International Conference on Computer Vision (ICCV 2015) 1502
  * [30] Kovacs A, Exl L, Kornell A, Fischbacher J, Hovorka M, Gusenbauer M, Breth L, Oezelt H, Yano M, Sakuma N, Kinoshita A, Shoji T, Kato A and Schrefl T 2022 Commun. Nonlinear Sci. 104 106041 ISSN 1007-5704 URL https://doi.org/10.1016/j.cnsns.2021.106041


# Can Calibration Improve Sample Prioritization?

Ganesh Tata
University of Alberta
<EMAIL_ADDRESS>
&Gautham Krishna Gudur11footnotemark: 1
Global AI Accelerator, Ericsson
<EMAIL_ADDRESS>
&Gopinath Chennupati
Amazon Alexa
<EMAIL_ADDRESS>
&Mohammad Emtiyaz Khan
RIKEN Center for AI Project
<EMAIL_ADDRESS>Both authors contributed equally to this work.

###### Abstract

Calibration can reduce overconfident predictions of deep neural networks, but
can calibration also accelerate training? In this paper, we show that it can
when used to prioritize some examples for performing subset selection. We
study the effect of popular calibration techniques in selecting better subsets
of samples during training (also called sample prioritization) and observe
that calibration can improve the quality of subsets, reduce the number of
examples per epoch (by at least 70%), and can thereby speed up the overall
training process. We further study the effect of using calibrated pre-trained
models coupled with calibration during training to guide sample
prioritization, which again seems to improve the quality of samples selected.

##  1 Introduction

Calibration is a widely used technique in machine learning to reduce
overconfidence in predictions. Modern deep neural networks are known to be
overconfident classifiers or predictors, and calibrated networks provide
trustworthy and reliable confidence estimates [1]. Hence, finding new
calibration techniques and improving them has been an active area of research
[1, 6, 10, 11].

In this paper, we ask if calibration aids in accelerating training by using
sample prioritization, i.e., we select training samples based on calibrated
predictions to better steer the training performance. We explore different
calibration techniques and focus on selecting a subset with the most
informative samples during each epoch. We observe that calibration performed
during training plays a crucial role in choosing the most informative subsets,
which in turn accelerates neural network training. We then investigate the
effect of an external pre-trained model which is well-calibrated (with larger
capacity) on the sample selection process during training.

Our contributions are as follows,

We provide an in-depth study analyzing the effect of various calibration
techniques on sample prioritization during training. We also consider pre-
trained calibrated target models and observe their effect on sample
prioritization along with calibration during training. We benchmark our
findings on widely used CIFAR-10 and CIFAR-100 datasets and observe the
improved quality of the chosen subsets across different subset sizes, which
ensures faster deep neural network training.

##  2 Background

###  2.1 Problem Statement

We formulate the problem in the paper as follows. A calibration technique $C$
is performed during training at each epoch, and a sample prioritization
function $a$ is then used to select the most informative samples for training
each subsequent epoch. We use Expected Calibration Error (ECE) for model
calibration [10], which measures the absolute difference between the model’s
accuracy and its confidence.

The paper discusses how a calibration technique $C$, when coupled with a
sample prioritization function $a$, affects the performance (accuracy and
calibration error (ECE)) of the model. In addition, we also observe if this
phenomenon can aid in faster and more efficient training. We hypothesize a
closer relationship between calibration and sample prioritization during
training, wherein the calibrated model probabilities at each epoch are used by
a sample prioritization criterion to select the most informative samples for
training each subsequent epoch.

###  2.2 Calibration

Calibration is a technique that curbs overconfident predictions in deep neural
networks, wherein the predicted (softmax) probabilities reflect true
probabilities of correctness (better confidence estimates) [1]. In this paper,
we consider various prominently used calibration techniques which are
performed during training.

Label Smoothing implicitly calibrates a model by discouraging overconfident
prediction probabilities during training [9]. The one-hot encoded ground truth
labels ($y_{k}$) are smoothened using a parameter $\alpha$, that is
$y^{LS}_{k}=y_{k}(1-\alpha)+\alpha/K$, where $K$ is the number of classes.
These smoothened targets $y^{LS}_{k}$ and predicted outputs $p_{k}$ are then
used to minimize the cross-entropy loss.

Mixup is a data augmentation method [14] which is shown to output well-
calibrated predictive scores [13], and is again performed during training.

$\displaystyle\bar{x}=\lambda x_{i}+(1-\lambda)x_{j}$
$\displaystyle\bar{y}=\lambda y_{i}+(1-\lambda)y_{j}$

where $x_{i}$ and $x_{j}$ are two input data points that are randomly sampled,
and $y_{i}$ and $y_{j}$ are their respective one-hot encoded labels. Here,
$\lambda\sim\text{Beta}(\alpha,\alpha)$ with $\lambda\in[0,1]$.

Focal Loss is an alternative loss function to cross-entropy which yields
calibrated probabilities by minimizing a regularized KL divergence between the
predicted and target distributions [8].

$L_{Focal}=-(1-p)^{\gamma}logp$

where $p$ is the probability assigned by the model to the ground-truth correct
class, and $\gamma$ is a hyperparameter. When compared with cross-entropy,
Focal Loss has an added factor that encourages the samples predicted with
correct classes to have lower probabilities. This enables the predicted
distribution to have higher entropy, thereby helping avoid overconfident
predictions.

###  2.3 Sample Prioritization

Sample prioritization is the process of selecting important samples during
different stages of training to accelerate the training process of a deep
neural network without compromising on performance. In this paper, we perform
sample prioritization during training using Max Entropy, which is a de facto
uncertainty sampling technique to select the most efficient samples at each
epoch.

Max Entropy selects the most informative samples (top-$k$) that maximize the
predictive entropy [12].

$\displaystyle\mathbb{H}$
$\displaystyle[y|x,D_{train}]:=-\sum_{c}p(y=c|x,D_{train})\log
p(y=c|x,D_{train})$

###  2.4 Pre-trained Calibrated Target models

Pre-trained models have been widely used in literature to obtain comprehensive
sample representations before training a downstream task [5]. We use a pre-
trained calibrated model with larger capacity which we call the target model
[3] and use the Max Entropy estimates obtained from this target model at each
epoch to select the samples, thereby guiding the corresponding epochs of the
model during training. Further, we call the model which is being trained as
the current model. In this paper, we perform sample prioritization with the
target model in addition to calibrating the current model.

Table 1: Test Accuracies (%) and ECEs (%) across various calibration
techniques and subset sizes with Resnet-34 as current model for both datasets.

Dataset | Calibration | 100% | 30% | 20% | 10%
---|---|---|---|---|---
Accuracy | ECE | Accuracy | ECE | Accuracy | ECE | Accuracy | ECE
CIFAR-10 |  | No Calibration
---
Cross-Entropy (Baseline)
94.1 | 4.1 | 93.6 | 5.33 | 93.86 | 4.01 | 93.23 | 5.2
| Label Smoothing
---
0.03/0.05/0.05/0.03
94 | 1.84 | 91.74 | 3.17 | 91.48 | 3.56 | 91.72 | 2.71
| Mixup
---
0.1/0.3/0.2/0.15
95.1 | 2.1 | 94.39 | 2.67 | 93.35 | 2.59 | 93.17 | 1.78
| Focal Loss
---
1/3/3/3
94.69 | 1.71 | 93.19 | 1.2 | 92.6 | 1.25 | 92.25 | 1.42
CIFAR-100 |  | No Calibration
---
Cross-Entropy (Baseline)
77.48 | 5.42 | 73.13 | 10.77 | 71.54 | 13.16 | 69.65 | 14.47
| Label Smoothing
---
0.03/0.03/0.03/0.09
77.05 | 4.88 | 72.21 | 3.45 | 70.93 | 5.75 | 68.63 | 5.67
| Mixup
---
0.15/0.15/0.15/0.35
78.68 | 3.59 | 73.57 | 1.49 | 72.02 | 2.4 | 69.1 | 1.16
| Focal Loss
---
1/3/3/5
78.59 | 3.57 | 71.86 | 1.67 | 70.61 | 3.25 | 65.81 | 1.82

##  3 Experiments and Results

We perform our experiments on CIFAR-10 and CIFAR-100 [4] datasets with the
setting mentioned in Section 2.1, and we use Resnet-34 [2] as the current
model. For both datasets, the initial training set consisting of 50,000
samples is split into 90% training data and 10% validation data, while the
test set contains 10,000 samples. In the sample prioritization setting, we
start with 10 warm-up epochs in which all samples are selected during training
(no subset selection), following which we select $n\%$ of total training
samples in each epoch using the Max Entropy criterion. We choose different
settings of subset sizes for each epoch with $n$ – {10, 20, 30}.

We use the Stochastic Gradient Descent optimizer with initial learning rates
of 0.01 and 0.1 for CIFAR-10 and CIFAR-100 respectively, trained for $200$
epochs with a cosine annealing [7] scheduler, weight decay of $5e^{-4}$ and
momentum of $0.9$ for both datasets. The models are trained using a V100 GPU.
We consider classification accuracies and ECEs across different calibration
techniques with their respective parameter sweeps as follows: Label Smoothing
($\alpha$) – {0.01, 0.03, 0.05, 0.07, 0.09}, Mixup ($\alpha$) – {0.1, 0.15,
0.2, 0.25, 0.3, 0.35} and Focal Loss ($\gamma$) – {1, 2, 3, 4, 5}. As
baselines, we consider uncalibrated models with standard cross-entropy loss.
For the target experiments, we choose Resnet-50 models [2] trained with Mixup
(with $\alpha=0.3$ for CIFAR-10, and $\alpha=0.25$ for CIFAR-100) as our
target models after performing parameter sweeps across all calibration
techniques.

###  3.1 Discussion on Results

Table 1 shows the test ECEs and accuracies for the current model across
different calibration techniques and subset sizes with Max Entropy criterion.
We can observe that all calibration techniques have lower test ECEs than their
respective uncalibrated models across all subset sizes for both datasets. This
demonstrates that performing calibration during training improves sample
prioritization. Moreover, these results indicate that there are no significant
trade-offs between model accuracy and model confidence (ECE) when calibration
is performed with sample prioritization. Figures 1(a) and 1(b) also illustrate
lower validation ECEs across training epochs for calibrated current models
when compared to their uncalibrated counterparts, with 30% subset size as an
instance. In particular, we observe that test accuracies for Mixup across all
subset sizes are consistently comparable and often higher than their
respective uncalibrated models.

(a) CIFAR-10

(b) CIFAR-100

Figure 1: Validation ECEs (%) for both datasets with 30% subset size for
current model. Table 2: Test Accuracies (%) and ECEs (%) across various
calibration techniques and subset sizes with Resnet-34 as current model for
both datasets, and Resnet-50 (Mixup) as target model.

Dataset | Calibration | 100% | 30% | 20% | 10%
---|---|---|---|---|---
Accuracy | ECE | Accuracy | ECE | Accuracy | ECE | Accuracy | ECE
CIFAR-10 |  | No Calibration
---
Cross-Entropy (Baseline)
94.1 | 4.1 | 93.95 | 4.04 | 93.43 | 4.9 | 93.16 | 4.11
| Label Smoothing
---
0.03/0.05/0.05/0.03
94 | 1.84 | 93.62 | 2.93 | 93.3 | 3.32 | 93.27 | 1.9
| Mixup
---
0.1/0.3/0.15/0.15
95.1 | 2.1 | 94.7 | 2.88 | 93.79 | 2.73 | 93.22 | 2.16
| Focal Loss
---
1/2/2/1
94.69 | 1.71 | 93.15 | 1.06 | 92.65 | 1.58 | 92.84 | 1.89
CIFAR-100 |  | No Calibration
---
Cross-Entropy (Baseline)
77.48 | 5.42 | 75.38 | 9.36 | 75.04 | 9.39 | 71.07 | 9.27
| Label Smoothing
---
0.03/0.03/0.03/0.09
77.05 | 4.88 | 76.06 | 2.28 | 75.27 | 2.67 | 72.59 | 1.63
| Mixup
---
0.15/0.2/0.15/0.15
78.68 | 3.59 | 75.62 | 0.86 | 74.78 | 1.43 | 70.32 | 0.86
| Focal Loss
---
1/2/3/2
78.59 | 3.57 | 74.89 | 2.37 | 73.73 | 1.43 | 70.89 | 1.51

As expected, we observe from Table 1 that the test accuracies reduce when the
subset size becomes smaller. Moreover, Mixup consistently has higher test
accuracies and low test ECEs across different subset sizes compared to other
calibration techniques for both datasets. This could be attributed to Mixup
being a data transformation/augmentation technique, thereby performing well
even in the low-data regime. Further, Figure 2 shows that training with Mixup
leads to a relatively balanced representation of classes in the chosen
subsets. In contrast, Label Smoothing and Focal Loss are loss-based
calibration techniques with no explicit transformation performed on the
underlying training data. Here, we note that Focal Loss can be more effective
when coupled with a post hoc calibration technique like temperature scaling
[8]. However, this setting is not applicable in our paper since we explicitly
focus on calibration techniques during training. In addition, Mixup and Focal
Loss are known to outperform Label Smoothing [13, 8].

Effectiveness of Target: Table 2 exhibits the results when a pre-trained
calibrated target model (Resnet-50 with Mixup) is used for guiding the current
model while performing sample prioritization with calibration during training.
Interestingly, a well-calibrated target model can boost the performance on
both datasets for under-performing calibration techniques (like Label
Smoothing) performed only on the current model without the target’s influence.
However, for calibration techniques that are already performing well (like
Mixup and Focal Loss), there is no significant loss in performance on
CIFAR-10, while there is a considerable improvement in performance in general
on CIFAR-100. This can be clearly observed when comparing Table 2 with Table
1. We assume that the performance of a current model trained with 100% data is
similar for all experiments performed with and without a target model for each
respective dataset.

(a) Mixup

(b) Focal Loss

(c) Label Smoothing

(d) No Calibration

(e) Random

Figure 2: Class Distribution across epochs for all calibration techniques with
Max Entropy (a)-(d) and Random Sampling (e) with 30% subset size for current
model on CIFAR-10.

Class Distribution: Figures 2(a) \- 2(d) show the class distribution across
training epochs for different calibration techniques with sample
prioritization performed using Max Entropy. For comparison, we also consider
Random sampling as a baseline (Figure 2(e)). As discussed above, performing
Mixup (Figure 2(a)) during training leads to a relatively balanced
representation of classes in each epoch. In contrast, Label Smoothing and
Focal Loss (Figures 2(b) and 2(c)) do not exhibit a balanced representation of
classes in the chosen subsets. The class representations are imbalanced for
the uncalibrated setting as well (Figure 2(d)).

##  4 Conclusion

In this paper, we investigate whether existing calibration techniques improve
sample prioritization during training. We empirically show that a deep neural
network calibrated during training selects better subsets of samples than an
uncalibrated model. We also demonstrate the effectiveness of pre-trained
calibrated target models in guiding sample prioritization during training,
thereby boosting calibration performance. Finally, since calibration aids in
sample prioritization by improving the quality of subsets, it ensures faster
neural network training.

## References

  * [1] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger.  On calibration of modern neural networks.  In International Conference on Machine Learning, pages 1321–1330. PMLR, 2017.
  * [2] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778, 2016.
  * [3] Dan Hendrycks, Kimin Lee, and Mantas Mazeika.  Using pre-training can improve model robustness and uncertainty.  In International Conference on Machine Learning, pages 2712–2721, 2019.
  * [4] Alex Krizhevsky and Geoffrey Hinton.  Learning multiple layers of features from tiny images.  Master’s thesis, University of Toronto, 2009.
  * [5] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In Advances in Neural Information Processing Systems, volume 25, 2012.
  * [6] Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon.  Accurate uncertainties for deep learning using calibrated regression.  In International Conference on Machine Learning, pages 2796–2804, 2018.
  * [7] Ilya Loshchilov and Frank Hutter.  Sgdr: Stochastic gradient descent with warm restarts.  In International Conference on Learning Representations, 2017.
  * [8] Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip Torr, and Puneet Dokania.  Calibrating deep neural networks using focal loss.  Advances in Neural Information Processing Systems, 33:15288–15299, 2020.
  * [9] Rafael Müller, Simon Kornblith, and Geoffrey E Hinton.  When does label smoothing help?  Advances in neural information processing systems, 32, 2019.
  * [10] Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht.  Obtaining well calibrated probabilities using bayesian binning.  In Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015\.
  * [11] Jeremy Nixon, Michael W Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran.  Measuring calibration in deep learning.  In CVPR Workshops, 2019.
  * [12] Claude Elwood Shannon.  A mathematical theory of communication.  Bell system technical journal, 27(3):379–423, 1948.
  * [13] Sunil Thulasidasan, Gopinath Chennupati, Jeff A Bilmes, Tanmoy Bhattacharya, and Sarah Michalak.  On mixup training: Improved calibration and predictive uncertainty for deep neural networks.  In Advances in Neural Information Processing Systems, volume 32, 2019.
  * [14] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz.  mixup: Beyond empirical risk minimization.  International Conference on Learning Representations, 2018.

##  Appendix A Common Samples between Epochs

Figures 3(a) and 3(b) show the percentage of common samples between
consecutive epochs for CIFAR-10 and CIFAR-100 respectively across various
calibration techniques, with Max Entropy and Random Sampling as the sample
prioritization criteria with 30% subset size. In all settings, sample
selection with Max Entropy leads to a significantly higher percentage of
common samples between consecutive epochs throughout training, as opposed to
random sampling which results in very few common samples. Although only a
small percentage of samples change across epochs, sample prioritization with
Max Entropy coupled with any calibration technique yields good classification
performance.

(a) CIFAR-10

(b) CIFAR-100

Figure 3: Common Samples (%) between epochs with 30% subset size for current
model.

##  Appendix B Validation Accuracies

We report the validation accuracies of CIFAR-10 and CIFAR-100 with 30% subset
size for the current model across all calibration techniques with Max Entropy
as the sample prioritization criterion in Figures 4(a) and 4(b).

(a) CIFAR-10

(b) CIFAR-100

Figure 4: Validation Accuracies (%) for both datasets with 30% subset size for
current model.

##  Appendix C Reliability Diagrams

We report the reliability diagrams of CIFAR-10 and CIFAR-100 with 30% subset
size for the current model across all calibration techniques with Max Entropy
as the sample prioritization criterion in Figure 5 and 6.

(a) Mixup

(b) Focal Loss

(c) Label Smoothing

(d) No Calibration

Figure 5: Reliability diagrams for CIFAR-10 with 30% subset size for current
model.

(a) Mixup

(b) Focal Loss

(c) Label Smoothing

(d) No Calibration

Figure 6: Reliability diagrams for CIFAR-100 with 30% subset size for current
model.