# Learning Order Parameters from Videos of Dynamical
Phases for Skyrmions with Neural Networks

Weidi Wang  Department of Physics, Beijing Normal University, Beijing 100875,
China  The Center for Advanced Quantum Studies, Beijing Normal University,
Beijing 100875, China  Zeyuan Wang  Department of Physics, Drexel University,
Philadelphia, PA 19104, USA  Yinghui Zhang  College of Education for the
Future, Beijing Normal University, Beijing 100875, China  Bo Sun
<EMAIL_ADDRESS>College of Education for the Future, Beijing Normal
University, Beijing 100875, China  School of Artificial Intelligence, Beijing
Normal University, Beijing 100875, China  Ke Xia<EMAIL_ADDRESS>The Center
for Advanced Quantum Studies, Beijing Normal University, Beijing 100875, China
Beijing Computational Science Research Center, Beijing 100193, China

###### Abstract

The ability to recognize dynamical phenomena (e.g., dynamical phases) and
dynamical processes in physical events from videos, then to abstract physical
concepts and reveal physical laws, lies at the core of human intelligence. The
main purposes of this paper are to use neural networks for classifying the
dynamical phases of some videos and to demonstrate that neural networks can
learn physical concepts from them. To this end, we employ multiple neural
networks to recognize the static phases (image format) and dynamical phases
(video format) of a particle-based skyrmion model. Our results show that
neural networks, without any prior knowledge, can not only correctly classify
these phases, but also predict the phase boundaries which agree with those
obtained by simulation. We further propose a parameter visualization scheme to
interpret what neural networks have learned. We show that neural networks can
learn two order parameters from videos of dynamical phases and predict the
critical values of two order parameters. Finally, we demonstrate that only two
order parameters are needed to identify videos of skyrmion dynamical phases.
It shows that this parameter visualization scheme can be used to determine how
many order parameters are needed to fully recognize the input phases. Our work
sheds light on the future use of neural networks in discovering new physical
concepts and revealing unknown yet physical laws from videos.

##  I Introduction

Humans, even young infants, can recognize and classify objects and their
motions, and interpret what has happened in them, deduce (to some extent) what
should happen, and imagine what would happen in other situations Yi  _et al._
(2019). This reflects the ability of human beings to reason logically based on
physical events and then to extract experience and laws (or patterns) from
them. Therefore, we can summarize the process of human discovering of physical
laws as (i) identifying and classifying physical events, (ii) abstracting
physical concepts, (iii) revealing physical laws. With the rapid development
of artificial intelligence, it has become possible to use machine learning to
perform the first two processes.

Machine learning has extremely broad applications in many fields and has
achieved breakthrough progress with bright prospects Jordan and Mitchell
(2015). Previous applications of machine learning techniques in condensed
matter physics centered on representing quantum states Carleo and Troyer
(2017), analyzing experimental data Zhang  _et al._ (2019); Rem  _et al._
(2019), classifying different phases and their transitions Wang (2016);
Carrasquilla and Melko (2017); Van Nieuwenburg  _et al._ (2017) and
discovering physical concepts Iten  _et al._ (2020).

One of the earliest machine learning algorithms was the artificial neural
network, a computational model designed to simulate biological learning
Goodfellow  _et al._ (2016). From the perspective of network architecture,
there are three fundamental neural networks: (i) feedforward neural network
(FNN), that can fit the nonlinear relationship between any two columnar data
Haykin (1994), (ii) convolutional neural network (CNN), which is suitable for
image data recognition and classification, and (iii) recurrent neural network
(RNN), that is applicable to processing sequential data. In Ref. Patterson and
Gibson (2017), the authors hold the belief that the input data type should
match the used network architecture. In other words, when the input is
columnar data, image data, or processing sequential data, the corresponding
network architecture is FNN, CNN, or RNN [specifically Long Short-Term Memory
(LSTM)], respectively. While when the input is video data, one should apply
the hybrid CNN and LSTM architecture.

The type of input data reflects the complexity of the problem under study, and
the application of machine learning techniques in the field of phase
transitions recognition ranges from simple to complex. Initially, machine
learning was applied to classify phases and phase transitions that are
primarily static, so the corresponding methods were generally FNN Iakovlev
_et al._ (2018, 2019); Venderley  _et al._ (2018) and CNN Araki  _et al._
(2019); Beach  _et al._ (2018); Park  _et al._ (2017). In the last two years,
RNN started being used for studying processing sequential data, mainly
including the identification of dynamical phases van Nieuwenburg  _et al._
(2018) and dynamical processes Deviatov  _et al._ (2019), as well as the
prediction of qubit dynamics Gupta and Biercuk (2018); Flurin  _et al._
(2020). However, the videos of dynamical phases or processes have not been
considered so far. There are two main reasons for this situation. The first is
that, compared with sequential data, the video data has one additional
dimension and a larger volume, thus contains more information and is more
difficult to identify and work with. Secondly, the identification of dynamical
phases and processes usually uses supervised learning, and this procedure
needs to label the training data manually, which requires a lot of time and
cost. Moreover, previous work van Nieuwenburg  _et al._ (2018); Deviatov  _et
al._ (2019); Gupta and Biercuk (2018); Flurin  _et al._ (2020) only regarded
neural networks as a black box, profound analysis of it being outside of the
interest of its authors. But it is essential for the future discovery of
physical laws to use neural networks for classifying videos of dynamical
phases and to analyze what they have learned.

In this paper we consider the dynamics of skyrmions based on a particle model,
since many kinds of systems can be adequately modeled as collectively
interacting particles moving in quenched disorder, with the transition from a
pinned phase to a sliding phase under an applied drive Reichhardt  _et al._
(2015). Examples include colloids on various types of substrates Bohlein  _et
al._ (2012), Wigner crystals and vortices in type-II superconductors Fily  _et
al._ (2010). In the following, we first use a variety of neural networks to
classify static and dynamical phases generated by this model and use the
trained neural networks to predict the phase boundaries. Then we analyze the
neural networks by feature map visualization and a parameter visualization
scheme. The latter is based on the idea of representation learning Bengio  _et
al._ (2013), combined with variational autoencoders Kingma and Welling (2013)
and supervised learning. This represents the main achievement of our work.
Last but not least we explore whether the network can determine the number of
order parameters by changing the number of neurons, and the influence of using
an unevenly distributed training set on the parameter visualization scheme.

##  II Model

We consider a particle model for skyrmions in metallic chiral magnets which
can describe the interaction between skyrmions and random disorder Lin  _et
al._ (2013). The equation of motion for a single skyrmion $i$ is Reichhardt
_et al._ (2015):

$\alpha_{d}\bm{v}_{i}=\bm{F}_{M}+\bm{F}^{ss}_{i}+\bm{F}^{sp}_{i}+\bm{F}_{D}.$
(1)

Here $\bm{v}_{i}$ means the skyrmion velocity. The damping term $\alpha_{d}$
adjusts the velocity to the net force acting on the skyrmion, while the first
term on the right-hand side is the Magnus term
$\bm{F}_{M}=-\alpha_{m}\hat{z}\times\bm{v}_{i}$ that turns the skyrmion
velocity into the perpendicular direction. According to Ref. Reichhardt  _et
al._ (2015), we impose the constraint $\alpha_{d}^{2}+\alpha_{m}^{2}=1$ and
take $\alpha_{m}/\alpha_{d}=9.962$. The skyrmion-skyrmion interaction force is
$\bm{F}^{ss}_{i}=F^{s0}\sum_{j=1}^{N_{s}}\hat{\bm{r}}_{ij}K_{1}(R_{ij}/\xi_{s})$,
where $F^{s0}=1$, $R_{ij}=\left|\bm{r}_{i}-\bm{r}_{j}\right|$,
$\hat{\bm{r}}_{ij}=(\boldsymbol{r}_{i}-\boldsymbol{r}_{j})/R_{ij}$, $K_{1}$ is
the modified Bessel function and $\xi_{s}=3$ represents the size of skyrmions.
The pinning force is
$\bm{F}^{sp}_{i}=\sum_{j=1}^{N_{p}}J_{d}\exp(-\bm{r}_{d}/\xi_{d})$, where
$J_{d}$ characterizes the strength of the pinning, $\bm{r}_{d}$ represents the
radial vector between the $i$th skyrmion and the $j$th pinning center, and
$\xi_{d}=0.3$ means the size of defects. The pinning force $\bm{F}^{sp}$ is
generated by a randomly placed, non-overlapping harmonic trap, and its maximum
pinning force is $\bm{F}_{p}$ Reichhardt  _et al._ (2015). The driving term is
$\bm{F}_{D}=2\pi\hbar e^{-1}\hat{z}\times\bm{J}_{B}$, where $\bm{J}_{B}$ is
the externally applied current. The system size is $L\times L$ with $L=36$,
while the density of skyrmions ($\rho_{s}$) and pinning sites ($\rho_{p}$) are
$0.1$ and $0.3$, respectively. In simulations, we use periodic boundary
conditions in the $x$ and $y$ directions. In order to express the various
interaction forces more concisely, we set the unit force to $F_{0}\sim
10^{-5}$ N/m and the unit length to $l_{0}\sim 10$ nm Reichhardt  _et al._
(2015).

Figure 1: The dynamical phase diagram for skyrmions as $\bm{F}_{p}$ and
$\bm{F}_{D}$ vary. MC: moving crystal; ML: moving liquid; PG: pinned amorphous
glass; PC: pinned crystal. The solid blue line is the phase boundary between
crystal and amorphous states, and the solid orange line is the phase boundary
between moving and pinned states. We used a grid of $840$ points on the square
lattice to construct the dynamical phase diagram. Since the initial positions
of skyrmions and defects are random, possibly affecting thus the formed phase
diagram, the value for each point is averaged over $10$ simulations.

The numerical details of Eq. (1) are shown in Ref. Lin  _et al._ (2013). In
each simulation, we calculate $4000$ iterations and draw a figure of skyrmions
distribution for each iteration. In order to make the motion information more
obvious, we take an image every 15 iterations, and thus we obtain almost $270$
images. Because a standard video has 30 frames per second, we finally produce
thus a nine-second video during each calculation. We can get different
$\bm{F}_{D}$ and $\bm{F}_{p}$ by changing the magnitude of the pinning
strength $J_{d}$ and the applied current $\bm{J}_{B}$, and then obtain four
different dynamical phases in video format (see Supplemental Material 111See
Supplemental Material at url for videos of the four dynamical phases and
further detailed information.). The diagram of four dynamical phases is shown
in Fig. 1. When $\bm{F}_{p}<0.055$, the skyrmions form a pinned crystal state
(PC), which transforms into a moving crystal state (MC) as the driving force
$\bm{F}_{D}$ increases. When $\bm{F}_{p}>0.055$, we first find a pinned
amorphous glass state (PG) at low drives. As the driving force $\bm{F}_{D}$
increases, it converts to moving liquid (ML) later and then to moving crystal.
From the diagram and videos of the four dynamical phases, it is easy to deduce
that the driving term $\bm{F}_{D}$ determines the velocity of skyrmions
movement, while the strength of the interaction between skyrmion and skyrmion
$\xi_{s}$ and the one between skyrmion and defect $\xi_{d}$ determine whether
it forms a crystal or an amorphous state.

##  III Classification of static and dynamical skyrmion phases

We study the classification of videos of four dynamical phases for the
particle-based skyrmion model. Our goal is to understand whether supervised
learning with neural networks can classify these phases exactly. In
particular, we also hope that the neural network can accurately predict the
phase boundary between them.

Image recognition is the basis of video recognition, so we first use the CNN
to recognize the images of static skyrmion crystal and amorphous phases.
Besides, we find that the problem of recognizing videos of dynamical phases
recognition is similar to the problem of video action recognition in machine
learning. In Ref. Carreira and Zisserman (2017), the authors summarize five
machine learning methods of video action recognition: (i) CNN-LSTM, (ii) three
dimensional convolutional neural network (3D-CNN), (iii) two-stream, (iv)
3D-fused two-stream, (v) two-stream 3D-CNN. Here we use the first two basic
methods (CNN-LSTM and 3D-CNN) to identify the dynamical phases of skyrmions.

Since our machine learning method belongs to supervised learning, we also need
to label the training image data and video data. From videos of the four
dynamical skyrmion phases one can observe that crystal and amorphous states
are divided by spatial information, moving and pinned states being moreover
separated by temporal information. In order to classify crystal and amorphous
states, we calculate the radial distribution function (RDF) and the standard
deviation of radial distribution function (SDRDF), whose value is denoted by
$S$. We also define mean-velocity ($\bar{v}$) to categorize moving and pinned
states (the corresponding expressions are shown in Appendix A). Apparently, we
can consider SDRDF as the spatial order parameter, and we can consider mean-
velocity as the temporal order parameter.

###  III.1 Classification of static skyrmion crystal and amorphous phases by
the CNN

We first train the CNN to distinguish the crystal phase with sixfold ordering
[Fig. 2(a)] from the unordered amorphous phase [Fig. 2(b)]. The $16800$
labeled images of crystal and amorphous phases are generated through
simulations at various interaction strengths ($\xi_{d}$ and $\xi_{s}$), in
which $80\%$ of the data are for training and $20\%$ for validation.

Figure 2: Examples of (a) crystal and (b) amorphous states. (c) The phase
diagram of crystal and amorphous states predicted by the CNN. The blue line is
the phase boundary between crystal and amorphous phase, the same as that in
Fig. 1, so the points on the left of the blue line represent the data for
crystals and the points on the right of the blue line represent the data for
amorphous states. After these test data are sent to the CNN, this will give
the predicted labels. The color of the dots represents the predicted label
(i.e., purple represents crystal, green represents amorphous state). The red
triangles represent the phase boundary predicted by CNN. The testing accuracy
is $98.33\%$ and $14$ test data were misclassified. These misclassified data
are green points in the crystal area, purple points in the amorphous area, and
the points between the predicted phase boundary and the simulated phase
boundary.

The CNN starts with a convolutional layer consisting of $32$ filters of size
$5\times 5$ with rectified linear (ReLu) activation functions. The output of
this layer is sent to the batch normlization (BN) layer before applying
$2\times 2$ max pooling. To extract features in a sufficient manner the
network is then fed into the same convolutional layer, BN layer, and into max
pooling layers with ReLu activation. After that, a fully-connected layer with
$1024$ ReLu units is followed before the ultimately softmax output layer with
two neurons. The actual training of the network is completed by minimizing the
cross entropy by using the Adam optimizer. Finally, we save the model
parameters (weights and biases) when the training is completed and the
validation accuracy reaches the best result. Considering the randomness of
neural networks for each training, we train five identical networks at the
same time and then select the optimal model parameter to test according to the
highest validation accuracy.

We wish to know whether the CNN can predict the phase boundary of skyrmion
crystal and amorphous states, so we evenly select 840 images in the phase
diagram of Fig. 1 as a test set to see the performance of the CNN on it. From
Fig. 2(c), we can see that the trained CNN can successfully recognize the
skyrmion crystal and amorphous states in the test set and can predict the
phase boundary (red triangles) which is in good agreement with that obtained
by calculating SDRDF (blue solid line). The good performance of the CNN on the
test set shows that the CNN can capture spatial information to distinguish
between crystal and amorphous states.

###  III.2 Classification of four dynamical skyrmion phases by the CNN-LSTM

Figure 3: (a) The structure of the hybrid CNN and LSTM. (b) The phase diagram
predicted by the CNN-LSTM. The blue and orange lines are the phase boundaries
between crystal and amorphous phases, moving and pinned phases, respectively,
which are the same as those in Fig. 1. After these test data are sent to the
CNN-LSTM, the network will give the prediction of labels. The color of the
dots represents the predicted label (i.e., purple represents ML, green
represents PG, magenta represents PC, and blue represents MC). The testing
accuracy is $94.40\%$ and $47$ test data were misclassified. These
misclassified data are the points with different colors in each area, such as
the green points in the PC area.

In this section, we show a universal architecture integrating CNN and LSTM
that can recognize the four dynamical phases of skyrmions in video format. In
the hybrid CNN and LSTM architecture, the CNN is specially designed to extract
spatial features, while the LSTM is used to extract temporal features. Then
temporal and spatial features are fused by the softmax function to achieve the
purpose of video recognition. Each input data is a video composed of $10$
frames of $36\times 36$ images which are the skyrmions distribution figures
drawn by the results of each iteration during the simulation. We train the
CNN-LSTM using a dataset of about $15,960$ videos, in which $80\%$ for
training and $20\%$ for validation.

The CNN-LSTM architecture is shown in Fig. 3(a). The structure of CNN here is
the same as that of CNN in Sec. III.1, except that the size of the
convolutional layer and max pooling layer is $3\times 3$. The output of CNN is
flattened to the LSTM consisting of $50$ neurons. The final layer of our
neural network is a softmax layer with four neurons, the output of which being
the probabilities of the four dynamical phases.

Additionally we evenly selected $840$ video data in the phase diagram of Fig.
1 as a test set to analyze the performance of the CNN-LSTM on it. As can be
seen from the predicted diagram in Fig. 3(b), the CNN-LSTM is able to classify
the four dynamical phases of skyrmions.

###  III.3 Classification of four dynamical skyrmion phases by the 3D-CNN

For the sake of getting the spatial and temporal information, a direct
approach to video recognition using deep networks is to combine the
convolutional operation with temporal features Herath  _et al._ (2017). Now we
use the 3D-CNN, which is superior to two-dimensional CNN Ji  _et al._ (2012),
for recognizing videos of four dynamical skyrmion phases.

Figure 4: The architecture of the 3D-CNN. The bule circle represents ReLu. We
use the same batch normalization and 3D max pooling layers. The first and
second 3D convolutional layers are also the same except the numbers of
convolutional kernels ($32$ and $16$ respectively).

The network architecture is shown in Fig. 4. The training data and test data
are the same as those used by the CNN-LSTM. After the training is finished, we
wish to know whether the 3D-CNN can predict the phase boundary of dynamical
phases for skyrmions. From Fig. 5 we can see that the trained 3D-CNN can
predict the crystal and amorphous phase boundary (red pentagram) which is in
good agreement with the phase boundary obtained by simulation (blue solid
line). Moreover, the trained 3D-CNN can predict the moving and pinned phase
boundary (green triangle) which is in good agreement with the phase boundary
obtained by simulation (orange solid line). In conclusion, the 3D-CNN can not
only classify four dynamical phases of skyrmions but is able to predict the
boundaries between them as well.

Figure 5: The phase diagram predicted by the 3D-CNN. The blue and orange lines
are the phase boundaries between crystal and amorphous states, moving and
pinned states, respectively, which are the same as those in Fig. 1. After
these test data are sent to the 3D-CNN, the network will give the prediction
of labels. The color of the dots represents the predicted label (i.e., purple
represents ML, green represents PG, magenta represents PC, and blue represents
MC). The red pentagram represents the predicted phase boundary between crystal
and amorphous phase, while the green triangle represents the predicted phase
boundary between moving and pinned phase. The testing accuracy is $97.74\%$
and $19$ test data were misclassified. These misclassified data are the points
with different colors in each area (e.g., the green points in the PC area) and
the points between the predicted phase boundary and the simulated phase
boundary.

##  IV Analysis of the classification processes

The results of our previous studies based on machine learning have shown that
neural networks can extract spatial and temporal information to recognize
static and dynamical phases of skyrmions. A question which raises naturally
is: what did the neural networks learn? In Ref. Wang (2016), the author finds
the physical concepts (e.g., the order parameter and structure factor) for
distinguishing phases by visualizing the samples in the four-dimensional
feature space composed of principal components. By visualizing the weights of
the hidden layer to analyze what the neural network has learned, other studies
find that the weight of the hidden layer is indeed related to some physical
quantities (e.g., the total magnetization and topological charge) Iakovlev
_et al._ (2018); Suchsland and Wessel (2018). This motivates us to perform a
similar analysis of our neural networks to explore what they learned.

###  IV.1 Feature map visualization in the CNN

Despite the success of CNN in many fields such as computer vision, it has long
been viewed as a “black box” and it is still unclear why it performs so well
Krizhevsky  _et al._ (2017). Although it is impossible to open this black box,
one can do some visualization investigations. The studies of visualization can
be divided into two main categories. One is feature map visualization, which
does not analyze the specific parameters of the convolution kernel, but first
selects some images, propagates them once forward in the existing model, and
then visualizes the feature map in the convolution kernel. The other is
parameter visualization that focuses on the analysis of parameters in the
convolution kernel or neurons.

We first visualize the feature maps of the CNN which has recognized the images
of skyrmion crystal and amorphous phases. We select respectively a crystal and
amorphous image as shown in Fig. 6(a) and 6(b). It can be seen that skyrmions
in the crystal are arranged in the direction of black and red arrows, while
skyrmions in the crystal are disordered. To find out whether the neural
network can learn the information of crystal orientation, after preprocessing,
the images are transferred to the CNN which has been trained before. Then we
visualize the feature maps of the first convolutional layer. Two feature maps
of the crystal and amorphous image are respectively shown in Fig. 6. Comparing
Fig. 6(c) and 6(d), we can see that the CNN has extracted the information of
oblique downward arrangement [i.e., the direction of the red arrow in Fig.
6(a)] from the images. From the comparison of Fig. 6(e) and 6(f), we can see
that the CNN has extracted the information of oblique upward arrangement
[i.e., the direction of the black arrow in Fig. 6(a)] from the images.
Therefore, we can conclude that the CNN extracts some information about the
spatial structure, and recognizes the images of crystal and amorphous phases
based on this information.

Figure 6: The original image of (a) crystal and (b) amorphous phase. (c) and
(e) are feature maps extracted from (a) by CNN. (d) and (f) are feature maps
extracted from (b) by CNN.

###  IV.2 Parameter visualization in the CNN

The feature map visualization visually shows the information of what CNN
learned from the original image. Now we want to perform a parameter
visualization to explore the relationship between the values of neuron outputs
in CNN and physical parameters (e.g., the order parameter). In Ref. Iten  _et
al._ (2020), the authors propose a scheme for visualizing the intermediate
parameters of a neural network based on the idea of representation learning.
They consider the former part of the neural network as an encoding process and
the latter part as a decoding process. When the training of a neural network
is completed, the information stored by neurons in the middle layer consists
of physical quantities that are closely related to the physical problem under
study. For example, in the study of damped pendulum motion, the information
preserved by neurons in the middle layer is related to the spring constant and
damping coefficient.

With this idea, we propose a scheme that can reveal what the neural networks
have learned and we refer to it as a parameter visualization scheme.
Specifically, we add a newly fully-connected layer in front of the original
CNN output layer and set the number of neurons to 1 because the classification
of skyrmion crystal and amorphous phases from physics requires only one order
parameter (i.e., SDRDF). And the information of input crystal and amorphous
images is compressed into one neuron in this fully-connected layer after a
series of convolutional and pooling operations. We are interested to know
whether this neuron output is related to SDRDF (the details of parameter
visualization scheme are shown in Appendix B).

Figure 7: The scatter diagram of the neuron output and SDRDF. The gray dashed
line is the horizontal line with a neuron output of $0.5$.

We first train the newly constructed CNN five times by the original training
set and save the model parameters after each completed training. Then we
randomly choose $2520$ images of crystal and amorphous phases, importing them
into the CNN, and output values of the neuron in the newly added fully-
connected layer. Note that we calculate SDRDF of each image sample before it
is sent to the neural network, and we can obtain the output of the
corresponding neuron after it is sent to the neural network. We can make a
scatter diagram of the neuron output and SDRDF (Fig. 7), where each blue dot
represents a image sample. In Fig. 7 we can see that in the region where
$S<0.4$ (the images are amorphous phases) the neuron output is mostly zero,
while in the region where $S>0.4$ (the images are crystal) the neuron output
is mostly one. Moreover, we find a mutation from 0 to 1 in the neuron output
near $S=0.4$. In summary, one can safely conclude that there is a strong
correlation between the neuron output and SDRDF.

We emphasize that only the images of skyrmion crystal and amorphous phases are
input to the CNN, while SDRDF is an artificially defined order parameter to
distinguish between skyrmion crystal and amorphous phases. Note also that the
information about SDRDF is not directly input into the CNN. However, the CNN
learner has found out meaningful information related to SDRDF by itself. Then
we can deduce that information similar to SDRDF is also stored in this neuron,
in other words, the CNN finds the order parameter of classifying crystal and
amorphous phases.

Furthermore, we can predict the critical value of the spatial order parameter
based on the relationship of the neuron output and SDRDF. We use the sigmoid
function to fit the relationship between the neuronal output and SDRDF.
Specifically, the sigmoid function is

$y(x)=\frac{1}{1+e^{-k(x-b)}}$ (2)

where the parameter $k$ determines the steepness of the sigmoid function. When
$k$ tends to 0, the line of fit is horizontal, indicating that the neuron
output does not correlate with the order parameter. When $k$ tends to
infinity, the line of fit is similar to a step function, indicating that the
neuron output has a strong correlation with the order parameter. The fitting
curve is shown in Fig. 7. It can be seen that the neuron output is $1$ in the
crystal part, while the neuron output is $0$ in the amorphous part. According
to the intuition about phase transitions, we can take the point where the
probability is exactly $0.5$ to be the inferred critical point $S_{0}$ Ch’ng
_et al._ (2017); Carrasquilla and Melko (2017); Liu and Van Nieuwenburg
(2018); Beach  _et al._ (2018); Dong  _et al._ (2019). Therefore, according to
the fitting curve, we can predict that the critical value of SDRDF is
$S_{CNN}=0.40009$, which is very little different from the real value
$S_{0}=0.40000$. In conclusion, CNN can not only extract information of the
order parameter from the original skyrmion crystal and amorphous images but
also can predict the critical value of the order parameter.

Figure 8: (a) the scatter diagram of the neuron one output with respect to
SDRDF and mean velocity, (b) the scatter diagram of the neuron two output with
respect to SDRDF and mean velocity. The gray dashed lines are the dividing
lines for order parameters. Namely, the gray dashed line in (a) is
$S_{0}=0.4$, and the gray dashed line in (b) is $\bar{v}_{0}=0.0014$.

###  IV.3 Parameter visualization in the 3D-CNN

We visualize now the neural network that identifies videos of four dynamical
skyrmion phases. Since we have demonstrated that the 3D-CNN is more successful
than the CNN-LSTM at classification, we consider only the parameter
visualization for the 3D-CNN. Similar to the previous parameter visualization
for the CNN, we add a fully-connected layer in front of the output layer of
the 3D-CNN. Because recognizing videos of four dynamical skyrmion phases
requires both temporal and spatial information, and physically requires two
order parameters (SDRDF and mean velocity), we set the number of neurons in
the fully-connected layer to $2$.

After training the new 3D-CNN, the best model parameters are saved and $2520$
samples (videos of four dynamical skyrmion phases) are input into the new
3D-CNN. Then we separately plot a scatter diagram of the output of two neurons
with respect to two order parameters [see Fig. 8(a) and 8(b)]. Each dot in the
figure represents a sample, and the color of dots represents the values of
neuron output. It can be seen in Fig. 8(a) that, when the mean velocity is
constant, the output of neuron one changes from $0$ to $1$ (the color changes
from green to blue) with the increase of SDRDF. However, when SDRDF is fixed,
the output of neuron one is constant no matter how the mean velocity varies,
which shows that the output of neuron one is only related to SDRDF. In other
words, neuron one stores only spatial information. Similarly, from Fig. 8(b),
when SDRDF is fixed, the output of neuron two changes from 0 to 1 (the color
changes from green to blue) as the mean velocity increases. However, when the
mean velocity is fixed, the output of neuron two is constant no matter how
SDRDF varies. It is shown that the output of neuron two is only related to the
mean velocity. Namely, neuron two stores only temporal information.

Figure 9: (a) The scatter diagram of the neuron one output and SDRDF. (b) The
scatter diagram of the neuron two output and mean velocity. The gray dashed
line is the horizontal line with a neuron output of $0.5$.

We emphasize that the input of the 3D-CNN consists only in videos of four
dynamical skyrmion phases, while the SDRDF and mean velocity are artificially
defined order parameters to distinguish videos of four dynamical skyrmion
phases, and the information of SDRDF and mean velocity is not directly entered
to the 3D-CNN. However, the 3D-CNN learner has found out meaningful
information related to SDRDF and mean velocity by itself. By visualizing the
output of two neurons in the 3D-CNN we find that the output of neuron one is
only related to SDRDF and the output of neuron two is only related to the mean
velocity, hence one can safely conclude that neuron one stores information
similar to the spatial order parameter and neuron two stores information
similar to the temporal order parameter. In other words, the 3D-CNN finds the
spatial order parameter and temporal order parameter.

Moreover, we can infer the critical values of the order parameters based on
the relationship between the neurons output and the order parameters. We plot
a scatter diagram of the output of neuron one versus SDRDF [Fig. 9(a)] and a
scatter of the output of neuron two versus mean velocity [Fig. 9(b)],
respectively. As can be seen from the scatter (red dots) in Fig. 9(a), neuron
one output has some correlation with SDRDF, and we use the sigmoid function to
fit this correlation (green line). Based on the fitting curve, when the output
of neuron one is $0.5$, we can predict the corresponding critical value
$S_{3D-CNN}=0.391\pm 0.002$ (real value $S_{0}=0.400$). Similarly, according
to Fig. 9(b), we can also use the sigmoid function to fit the relationship
between the output of neuron two and mean velocity, and predict the critical
value of mean velocity is $\bar{v}_{3D-CNN}=0.00121$ (real value
$\bar{v}_{0}=0.00140$).

It also needs to be added that when we perform the parameter visualization
scheme, we must select the best model which has the highest accuracy on the
validation set over the 5 training sessions. The reason is that when the
neural network can distinguish the input phases with maximum accuracy, the
input data is compressed to its simplest and best form at the same time. This
form is closest to the physical essence, and therefore the result visualized,
in this case, is the result shown in Fig. 8. Using other models for parameter
visualization, we would find at this point that the neural network does not
necessarily separate the temporal and spatial information, nor does it learn
both the temporal and spatial order parameters (the results of these parameter
visualizations are shown in Supplemental Materials).

##  V Test of parameter visualization scheme in the 3D-CNN

###  V.1 Variation of the number of neurons

After finding that the parameter visualization scheme works surprisingly well
(especially in the 3D-CNN) a related question arises: what happens when we
change the number of neurons ($N$) in the newly added fully-connected layer of
the 3D-CNN? To produce an answer to it we set the number of neurons to 1, 3,
and 4 respectively, to investigate the relations between the neurons output
and the order parameters.

Figure 10: The scatter diagram of the neuron output with respect to two order
parameters when the number of neurons in the newly added fully-connected layer
is $1$. The gray dashed line is the dividing line for the temporal order
parameter.

####  V.1.1 Case $N=1$

When the number of neurons (in the newly added fully-connected layer) is $1$,
we use the parameter visualization scheme and draw the scatter diagram of the
neuron output with respect to two order parameters (Fig. 10). We find that the
neuron output is only related to mean velocity, which means that the network
learns the information of the temporal order parameter. Besides, we also find
that the recognition accuracy of the 3D-CNN with one neuron is $10\%$ lower
than that of the 3D-CNN with two neurons. This suggests that using one neuron
is not sufficient to distinguish videos of four dynamical skyrmion phases, at
least two being needed for this. Correspondingly, to physically identify four
dynamical skyrmion phases, it is not enough to use one order parameter, at
least two being required. When we set the number of neurons in the 3D-CNN to 1
(compressing thus the information into one order parameter), the neural
network would have to discard some of the information. A related relevant
question is what information would be discarded by the neural network. The
above result shows that the neural network discards spatial information and
retains temporal information. The possible reason is that learning the
temporal order parameter is easier than learning the spatial order parameter.

####  V.1.2 Case $N=3$ or $N=4$

When the number of neurons is 3, we respectively plot the scatter diagrams of
the neurons output with respect to two order parameters [Fig. 11(a), 11(b),
11(c)], it can be seen that neurons one and two learn the temporal order
parameter, and neuron three learns the spatial order parameter, and the output
of neuron one is almost the same as that of neuron two. This means that the
information stored by one of neuron one and neuron two is redundant. In other
words, it is sufficient to use two order parameters to classify four dynamical
phases of skyrmions.

When the number of neurons is 4, we respectively plot the scatter diagrams of
the neurons output with respect to two order parameters (see Supplemental
Materials). We find that three neurons learn the spatial order parameter, and
the other neuron learns the temporal order parameter. In this case, it is
still possible to conclude that only two order parameters are needed to
classify four dynamical phases of skyrmions.

We compare the above two cases with the situation when the number of neurons
is 1. We find that when the number of neurons is 1, the 3D-CNN lacks
sufficient information and thus cannot fully classify the four dynamical
phases of skyrmions. When the number of neurons is 3 or 4, our results show
that the information of two neurons is enough. Therefore, it can be determined
that only two order parameters are needed to identify the four dynamical
phases of skyrmions. In conclusion, we can find out how many order parameters
are needed to fully classify the input phases by our methods, which is very
meaningful for exploring the unknown phases and phase transitions.

Figure 11: When the number of neurons in the newly added fully-connected layer
is $3$ (a) the scatter diagram of the neuron one output with respect to two
order parameters, (b) the scatter diagram of the neuron two output with
respect to two order parameters, (c) the scatter diagram of the neuron three
output with respect to two order parameters.

###  V.2 Variation of the distribution of samples in the training set

In the previous problems, the sample distribution in the training set we used
was uniform. In other words, the number of phases of each type in the training
set is equal (for example, if the total number of the training set is 16000
with four types of phases, the number of phases of each type would be 4000).
However it is difficult to ensure that the training samples we collected in
the real world are uniformly distributed, and we often deal with datasets that
are unevenly distributed. Consequently the inevitable question arises: is the
neural network still able to learn the order parameters from videos when the
sample is unevenly distributed?

To provide an answer to it we construct a new training set, and the
distribution of samples in the training set is no longer uniform, but similar
to the distribution in the phase diagram of Fig. 1, which was chosen because
it is closer to physical reality. Specifically, the proportions of PG, MC, PC,
and ML in the training set were $40\%$, $35\%$, $20\%$, and $5\%$,
respectively, and the total number of samples in the training set remains
$15960$.

We first use this new training set to train the 3D-CNN which has two neurons
in the newly added fully-connected layer and visualize the optimal model. We
find that the results are basically the same as those obtained in Sec. IV.3.
Two neurons learn the temporal and spatial order parameters, respectively. We
then proceed to vary the number of neurons to 1, 3, or 4 in order to explore
the effect of using an unevenly distributed training set on these cases. When
the number of neurons is $1$, the scatter diagram of the neuron output with
respect to two order parameters is shown in Fig. 12. We have two findings: (i)
compared with the usage of two neurons, the recognition accuracy reduces by
$5\%$; (ii) the ML region is empty, i.e. all ML phases are misclassified. So
when the number of neurons is 1, the 3D-CNN can only classify four types of
phases into three types. Strictly speaking, the 3D-CNN with only one neuron
cannot fully recognize the four dynamical phases of skyrmions, and at least
two neurons are needed. Since at least two order parameters are needed in
physics to distinguish the four dynamical phases of skyrmions, when we set the
number of neurons in the 3D-CNN to 1, the neural network has to discard some
information. As can be seen from our results above, the neural network learns
information about the PG, PC, and MC type phases and discards information
about the ML type phase, which explains why the recognition accuracy is
reduced by $5\%$ (since ML accounts for $5\%$ in the training set). When the
number of neurons is 3 or 4, we find that the results are essentially the same
as those presented in Sec. V.1.2. The neural network learns the temporal and
spatial order parameters, and the information learned by some neurons is
redundant.

In summary, when the number of neurons is 1, the neural network discards the
information of the phase with the least number of samples to ensure a higher
recognition accuracy, which also reflects the intelligence of the neural
network. When the number of neurons is 2, 3, or 4, the results are essentially
the same as before. This shows that the 3D-CNN trained with an unevenly
distributed training set can still learn two order parameters. Moreover, by
using the parameter visualization scheme and varying the number of neurons, we
can also determine how many order parameters are required to fully classify
the input phases.

Figure 12: The scatter diagram of the neuron output with respect to SDRDF and
mean velocity when the number of neurons is $1$ and the training set is
unevenly distributed.

##  VI Summary and outlook

We develop a general video recognition method to identify the physical
dynamical phases, and propose a parameter visualization scheme based on
representation learning to analyze the neural network. Then we demonstrate
that the neural network learns order parameters from videos. Our method can
give the number of order parameters (similar to determining the degree of
freedom of a system), without requiring numerical values or the mathematical
expressions of order parameters, which is a step forward in discovering
physical laws from videos.

From the perspective of future applications, the techniques of video
recognition can be widely used in physical experimental observations and
computational simulations. For example, employed for analyzing videos of the
magnetic dynamical process Seki  _et al._ (2020); Deviatov  _et al._ (2019),
it can discover important physical parameters throughout the process, and even
reveal physical laws. Or it can be used to discover some anomalies or novel
physical phenomena during dealing with experimental observations of scanning
tunneling microscopy.

Our method of parameter visualization comes from variational autoencoders in
representation learning. The general algorithm of variational autoencoders is
to construct a network with the same input and output, and then adjust the
number of neurons in the middle layer to get the minimal set of latent
parameters (similar to dimensionality reduction Hinton and Salakhutdinov
(2006) and data compression). However, our method is fundamentally different
from the general variational autoencoders, as the input and output are video
datasets and the corresponding labels, respectively. Therefore, our method can
be seen as a combination of supervised learning and variational autoencoders,
which makes it even further generalizable and capable of being applied to
other supervised learning scenarios.

We note that some recent work uses variational autoencoders to extract latent
parameters of partial differential equations from physical systems Lu  _et
al._ (2020); He and Pathak (2020); Wang  _et al._ (2019), but their datasets
are typically processing sequential data, which is not as complex as video
data. Moreover, they do not explicitly obtain partial differential equations.
For getting physical insights more in line with those of humans, perhaps
symbolic regression Udrescu and Tegmark (2020); Kim  _et al._ (2020) could be
used for delivering parsed expressions mapped by encoders and decoders Iten
_et al._ (2020). If symbolic regression can be combined with video recognition
techniques of supervised learning, it should be possible to obtain physical
insights from videos based on mathematical expressions.

##  VII Acknowledgments

We acknowledge Wenan Guo, Xintian Wu and Huaiyang Yuan for helpful discussions
about model simulations, and thank Lei Wang and Ka Shen for reading the
manuscript and helpful suggestions. Also, we would like to thank Yuanyuan Mi,
Zhuo Hao and Penghao Rao for technical assistance about machine learning. This
work was financially supported by the National Key Research and Development
Program of China (Grants No. 2017YFA0303300 and No. 2018YFB0407601), the
National Natural Science Foundation of China (Grants No. 61774017, No.
11734004, and No. 21421003), and NSAF (Grant No. U1930402). We also
acknowledge the National Supercomputer Center in Guangzhou (Tianhe II) for
providing the computing resources. The authors used Keras Chollet  _et al._
(2015) for machine learning.

##  Appendix A classification problem demonstration

The aim of this appendix is to clarify the classification problem of dynamical
skyrmion phases and how to make labels. From Ref. Reichhardt  _et al._ (2015),
we know that there are four dynamical phases in the system described by Eq.
(1), which have two types of characteristics: spatial characteristics (crystal
or amorphous states) and temporal characteristics (moving or pinned states).

In statistical mechanics, crystal and amorphous states can be uniquely
identified by RDF. The RDF $g(r)$ was introduced in order to describe how
density changes as a function of distance from a reference particle in a
system of particles Chandler (1987) and is given by

$g(r)=\frac{1}{\rho_{0}}\frac{n(r)}{V}$ (3)

where $\rho_{0}$ is density, $n(r)$ is the particle numbers in the ring from
$r$ to $r+dr$ and $V$ is the volume.

Generally speaking, crystal is long-range order, and the arrangement of atoms
has periodicity; noncrystal is short-range order, and the arrangement of atoms
has no periodicity. The RDF of crystal and amorphous states are shown in Fig.
13. The first and second peaks in RDF are the coordination numbers of the
first and second near neighbour. There are almost no resolvable peaks after
the third neighbor in the RDF of amorphous state, but there are still
resolvable peaks after the third neighbor in the RDF of crystal.

Figure 13: The RDF $g(r)$ as a function of the distance $r$ for crystal (a)
and amorphous states (b).

We calculate the standard deviation of RDF after the third peak (SDRDF) and
set the critical value $S_{0}=0.4$ to classify crystal and amorphous states.
We assume that the weak disorder regime ($S>S_{0}$) is crystal while the
strong disorder regime ($S\leq S_{0}$) is amorphous state. Therefore, when we
use the CNN to classify crystal and amorphous phases in Sec. III.1, MC and PC
are labeled by a label $p=(0,1)$; ML and PG are labeled by a label $p=(1,0)$.

Furthermore, we define mean-velocity ($\bar{v}$) to distinguish between moving
and pinned states and set the critical value to $\bar{v}_{0}=0.0014$ (unit
length per second) $\approx 10^{-11}$ m/s. We assume that the low velocity
regime ($\bar{v}\leq\bar{v}_{0}$) is pinned state while the high velocity
regime ($\bar{v}>\bar{v}_{0}$) is moving state.

In Sec. III.2 and III.3, the labels of four dynamical skyrmion phases used by
the CNN-LSTM and 3D-CNN are the same. At weak disorder and low mean-velocity
PC is labeled by a label $p=(1,0,0,0)$, and at weak disorder and high mean-
velocity MC is labeled by a label $p=(0,1,0,0)$, and at strong disorder and
low mean-velocity PG is labeled by a label $p=(0,0,1,0)$, and at strong
disorder and high mean-velocity ML is labeled by a label $p=(0,0,0,1)$.

##  Appendix B details of parameter visualization scheme

We propose a scheme to visualize the parameters in the neural network and in
this appendix we share some details about it. In general, we would take the
following steps (using the case when the number of neurons in the CNN is 1 as
an example):

  * •

step 1. Add a newly fully-connected layer in front of the output layer of the
original CNN, and set the number of neurons in the fully-connected layer to 1,
getting thus a new CNN;

  * •

step 2. Use the original training set to train the new CNN for five times and
save the model parameters after the training is completed each time.

  * •

step 3. Select the best model parameters with the highest accuracy of
validation set over the five training sessions and import them into the model.

  * •

step 4. 2520 samples (images of skyrmions crystal and amorphous phases) are
selected and input into the new CNN, which outputs the value of the neuron in
the newly added fully-connected layer and the values of the neurons in the
output layer (i.e., predicted labels).

  * •

step 5. Compare the predicted labels with the real labels and get accuracy. We
aim to investigate what the neural network learns. Since it is impossible for
the neural network to fully discriminate the input phases, we need to remove
those misidentified data for which the neural network has not learned the
features.

  * •

step 6. Draw a scatter diagram of the neuron output with respect to the
spatial order parameter.

When dealing with the problem of the 3D-CNN parameter visualization, the steps
that are different from those for the CNN parameter visualization are step 1,
step 4, and step 6. In the first step, the number of neurons in the fully-
connected layer is set to 2. In the fourth step, the samples we selected are
in video format. In the sixth step, we draw the scatter diagram of the outputs
of the neurons with respect to two order parameters.

## References

  * Yi  _et al._ (2019) K. Yi, C. Gan, Y. Li, P. Kohli, J. Wu, A. Torralba, and J. B. Tenenbaum, “Clevrer: Collision events for video representation and reasoning,” arXiv:1910.01442 (2019).
  * Jordan and Mitchell (2015) M. I. Jordan and T. M. Mitchell, “Machine learning: Trends, perspectives, and prospects,” Science 349, 255 (2015).
  * Carleo and Troyer (2017) G. Carleo and M. Troyer, “Solving the quantum many-body problem with artificial neural networks,” Science 355, 602 (2017).
  * Zhang  _et al._ (2019) Y. Zhang, A. Mesaros, K. Fujita, S. Edkins, M. Hamidian, K. Ch ng, H. Eisaki, S. Uchida, J. S. Davis, E. Khatami, _et al._ , “Machine learning in electronic-quantum-matter imaging experiments,” Nature 570, 484 (2019).
  * Rem  _et al._ (2019) B. S. Rem, N. Käming, M. Tarnowski, L. Asteria, N. Fläschner, C. Becker, K. Sengstock, and C. Weitenberg, “Identifying quantum phase transitions using artificial neural networks on experimental data,” Nat. Phys. , 1 (2019).
  * Wang (2016) L. Wang, “Discovering phase transitions with unsupervised learning,” Phys. Rev. B 94, 195105 (2016).
  * Carrasquilla and Melko (2017) J. Carrasquilla and R. G. Melko, “Machine learning phases of matter,” Nat. Phys. 13, 431 (2017).
  * Van Nieuwenburg  _et al._ (2017) E. P. Van Nieuwenburg, Y.-H. Liu, and S. D. Huber, “Learning phase transitions by confusion,” Nat. Phys. 13, 435 (2017).
  * Iten  _et al._ (2020) R. Iten, T. Metger, H. Wilming, L. Del Rio, and R. Renner, “Discovering physical concepts with neural networks,” Phys. Rev. Lett. 124, 010508 (2020).
  * Goodfellow  _et al._ (2016) I. Goodfellow, Y. Bengio, and A. Courville,  _Deep learning_ (MIT press, 2016).
  * Haykin (1994) S. Haykin,  _Neural networks: a comprehensive foundation_ (Prentice Hall PTR, 1994).
  * Patterson and Gibson (2017) J. Patterson and A. Gibson,  _Deep learning: A practitioner’s approach_ (” O’Reilly Media, Inc.”, 2017).
  * Iakovlev  _et al._ (2018) I. Iakovlev, O. Sotnikov, and V. Mazurenko, “Supervised learning approach for recognizing magnetic skyrmion phases,” Phys. Rev. B 98, 174411 (2018).
  * Iakovlev  _et al._ (2019) I. Iakovlev, O. Sotnikov, and V. Mazurenko, “Profile approach for recognition of three-dimensional magnetic structures,” Phys. Rev. B 99, 024430 (2019).
  * Venderley  _et al._ (2018) J. Venderley, V. Khemani, and E. A. Kim, “Machine learning out-of-equilibrium phases of matter,” Phys. Rev. Lett. 120, 257204 (2018).
  * Araki  _et al._ (2019) H. Araki, T. Mizoguchi, and Y. Hatsugai, “Phase diagram of a disordered higher-order topological insulator: A machine learning study,” Phys. Rev. B 99, 085406 (2019).
  * Beach  _et al._ (2018) M. J. Beach, A. Golubeva, and R. G. Melko, “Machine learning vortices at the kosterlitz-thouless transition,” Phys. Rev. B 97, 045207 (2018).
  * Park  _et al._ (2017) W. B. Park, J. Chung, J. Jung, K. Sohn, S. P. Singh, M. Pyo, N. Shin, and K.-S. Sohn, “Classification of crystal structure using a convolutional neural network,” IUCrJ 4, 486 (2017).
  * van Nieuwenburg  _et al._ (2018) E. van Nieuwenburg, E. Bairey, and G. Refael, “Learning phase transitions from dynamics,” Phys. Rev. B 98, 060301 (2018).
  * Deviatov  _et al._ (2019) A. Deviatov, I. Iakovlev, and V. Mazurenko, “Recurrent network classifier for ultrafast skyrmion dynamics,” Phys. Rev. Applied 12, 054026 (2019).
  * Gupta and Biercuk (2018) R. S. Gupta and M. J. Biercuk, “Machine learning for predictive estimation of qubit dynamics subject to dephasing,” Phys. Rev. Applied 9, 064042 (2018).
  * Flurin  _et al._ (2020) E. Flurin, L. S. Martin, S. Hacohen-Gourgy, and I. Siddiqi, “Using a recurrent neural network to reconstruct quantum dynamics of a superconducting qubit from physical observations,” Phys. Rev. X 10, 011006 (2020).
  * Reichhardt  _et al._ (2015) C. Reichhardt, D. Ray, and C. O. Reichhardt, “Collective transport properties of driven skyrmions with random disorder,” Phys. Rev. Lett. 114, 217202 (2015).
  * Bohlein  _et al._ (2012) T. Bohlein, J. Mikhael, and C. Bechinger, “Observation of kinks and antikinks in colloidal monolayers driven across ordered surfaces,” Nat. Mater. 11, 126 (2012).
  * Fily  _et al._ (2010) Y. Fily, E. Olive, N. Di Scala, and J. C. Soret, “Critical behavior of plastic depinning of vortex lattices in two dimensions: Molecular dynamics simulations,” Phys. Rev. B 82, 134519 (2010).
  * Bengio  _et al._ (2013) Y. Bengio, A. Courville, and P. Vincent, “Representation learning: A review and new perspectives,” IEEE Transactions on Pattern Analysis and Machine Intelligence 35, 1798 (2013).
  * Kingma and Welling (2013) D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” arXiv:1312.6114 (2013).
  * Lin  _et al._ (2013) S.-Z. Lin, C. Reichhardt, C. D. Batista, and A. Saxena, “Particle model for skyrmions in metallic chiral magnets: Dynamics, pinning, and creep,” Phys. Rev. B 87, 214419 (2013).
  * Note (1) See Supplemental Material at url for videos of the four dynamical phases and further detailed information.
  * Carreira and Zisserman (2017) J. Carreira and A. Zisserman, “Quo vadis, action recognition? a new model and the kinetics dataset,” in  _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ (2017) pp. 6299–6308.
  * Herath  _et al._ (2017) S. Herath, M. Harandi, and F. Porikli, “Going deeper into action recognition: A survey,” Image and Vision Computing 60, 4 (2017).
  * Ji  _et al._ (2012) S. Ji, W. Xu, M. Yang, and K. Yu, “3d convolutional neural networks for human action recognition,” IEEE Transactions on Pattern Analysis and Machine Intelligence 35, 221 (2012).
  * Suchsland and Wessel (2018) P. Suchsland and S. Wessel, “Parameter diagnostics of phases and phase transition learning by neural networks,” Phys. Rev. B 97, 174435 (2018).
  * Krizhevsky  _et al._ (2017) A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” Commun. ACM 60, 84 C90 (2017).
  * Ch’ng  _et al._ (2017) K. Ch’ng, J. Carrasquilla, R. G. Melko, and E. Khatami, “Machine learning phases of strongly correlated fermions,” Phys. Rev. X 7, 031038 (2017).
  * Liu and Van Nieuwenburg (2018) Y. H. Liu and E. P. Van Nieuwenburg, “Discriminative cooperative networks for detecting phase transitions,” Phys. Rev. Lett. 120, 176401 (2018).
  * Dong  _et al._ (2019) X.-Y. Dong, F. Pollmann, X.-F. Zhang, _et al._ , “Machine learning of quantum phase transitions,” Phys. Rev. B 99, 121104 (2019).
  * Seki  _et al._ (2020) S. Seki, M. Garst, J. Waizner, R. Takagi, N. Khanh, Y. Okamura, K. Kondou, F. Kagawa, Y. Otani, and Y. Tokura, “Propagation dynamics of spin excitations along skyrmion strings,” Nat. Commun. 11, 1 (2020).
  * Hinton and Salakhutdinov (2006) G. E. Hinton and R. R. Salakhutdinov, “Reducing the dimensionality of data with neural networks,” Science 313, 504 (2006).
  * Lu  _et al._ (2020) P. Y. Lu, S. Kim, and M. Soljačić, “Extracting interpretable physical parameters from spatiotemporal systems using unsupervised learning,” Phys. Rev. X 10, 031056 (2020).
  * He and Pathak (2020) H. He and J. Pathak, “An unsupervised learning approach to solving heat equations on chip based on auto encoder and image gradient,” arXiv:2007.09684 (2020).
  * Wang  _et al._ (2019) C. Wang, H. Zhai, and Y.-Z. You, “Emergent schrödinger equation in an introspective machine learning architecture,” Sci. Bull. 64, 1228 (2019).
  * Udrescu and Tegmark (2020) S.-M. Udrescu and M. Tegmark, “Symbolic pregression: Discovering physical laws from raw distorted video,” arXiv:2005.11212 (2020).
  * Kim  _et al._ (2020) S. Kim, P. Y. Lu, S. Mukherjee, M. Gilbert, L. Jing, V. Čeperić, and M. Soljačić, “Integration of neural network-based symbolic regression in deep learning for scientific discovery,” IEEE Transactions on Neural Networks and Learning Systems (2020), 10.1109/TNNLS.2020.3017010.
  * Chollet  _et al._ (2015) F. Chollet _et al._ , “Keras,” https://github.com/keras-team/keras (2015).
  * Chandler (1987) D. Chandler,  _Introduction to modern statistical mechanics_ (Oxford University Press, 1987) p. 288.

Supplemental Materials: Learning Order Parameters from Videos of Dynamical
Phases for Skyrmions with Neural Networks

##  Appendix S-1 Visualization of the model parameters for each training
session

In the parameter visualization scheme, to avoid the randomness of the neural
network, we trained the same neural network by the same dataset five times,
and then selected the model parameters with the highest accuracy of validation
set over five training sessions for visualization. Since the order of the
dataset is disrupted before each training session, this operation is
equivalent to performing five times simple cross-validations.

Here we deal with the problem of parameter visualization in the same way as
the previous problem of classification, for the reason that the two problems
are intrinsically consistent. The case with the highest recognition accuracy
corresponds to the case where the information is compressed to its simplest
and best form, i.e., closest to the physical essence.

Figure S-1: Two examples of visualizing the model parameters. (a) and (b) are
the results of the visualization of neurons one and two in the first example,
respectively. (c) and (d) are the results of the visualization of neurons one
and two in the second example, respectively.

Figure. S-1 shows examples of visualizing the model parameters derived from
two of the training sessions. The results of the first example [Fig. S-1(a)
and (b)] show that neuron one learns information about the temporal order
parameter, while neuron two learns the spatiotemporal mixed information. The
results of the second example [Fig. S-1(c) and (d)] show that neuron one
learns the spatiotemporal mixed information, while neuron two learns
information about the spatial order parameter.

##  Appendix S-2 Parameter visualization of the 3D-CNN with four neurons

When the number of neurons in the newly added fully-connected layer is $4$, we
use the parameter visualization scheme and respectively plot the scatter
diagrams of the neuron output with respect to two order parameters (Fig. S-2).
It can be seen that neuron 3 learns information about the spatial order
parameter, while neurons 1, 2, and 4 learn information about the temporal
order parameter. We can conclude that only two order parameters are needed to
identify the dynamical phases of skyrmions. Besides, we also test the case
where the number of neurons is 10. We find that the network can learn two
order parameters, but some neurons may learn the spatiotemporal mixed
information [e.g., Fig. S-1(b) and (c)]. The reason may be that as the number
of neurons increases, the information becomes more divergent and it becomes
more difficult to compress the information to the best and simplest form. This
makes it more difficult to learn order parameters.

Figure S-2: When the number of neurons is 4, (a) the scatter diagram of the
neuron 1 output with respect to two order parameters. (b) The scatter diagram
of the neuron 2 output with respect to two order parameters. (c) The scatter
diagram of the neuron 3 output with respect to two order parameters. (d) The
scatter diagram of the neuron 4 output with respect to two order parameters.