# Self-supervised Representations and Node Embedding Graph Neural Networks for
Accurate and Multi-scale Analysis of Materials

Jian-Gang Kong  School of Science, Chongqing University of Posts and
Telecommunications, Chongqing 400065, China  Ke-Lin Zhao  School of Science,
Chongqing University of Posts and Telecommunications, Chongqing 400065, China
Jian Li  School of Science, Chongqing University of Posts and
Telecommunications, Chongqing 400065, China  Institute for Advanced Sciences,
Chongqing University of Posts and Telecommunications, Chongqing 400065, China
Southwest Center for Theoretical Physics, Chongqing University, Chongqing
401331, China  Qing-Xu Li  School of Science, Chongqing University of Posts
and Telecommunications, Chongqing 400065, China  Institute for Advanced
Sciences, Chongqing University of Posts and Telecommunications, Chongqing
400065, China  Yu Liu  Inspur Electronic Information Industry Co., Ltd,
Beijing 100085, China  Rui Zhang<EMAIL_ADDRESS>Huaneng Guicheng Trust
Corp Ltd, ShenZhen, 518000,China  Jia-Ji Zhu<EMAIL_ADDRESS>School of
Science, Chongqing University of Posts and Telecommunications, Chongqing
400065, China  Institute for Advanced Sciences, Chongqing University of Posts
and Telecommunications, Chongqing 400065, China  Southwest Center for
Theoretical Physics, Chongqing University, Chongqing 401331, China  Kai Chang
<EMAIL_ADDRESS>SKLSM, Institute of Semiconductors, Chinese Academy of
Sciences, Beijing 100083, China  Center for Excellence in Topological Quantum
Computation, University of Chinese Academy of Sciences, Beijing 100190, China
Division of Quantum Materials and Devices, Beijing Academy of Quantum
Information Sciences, Beijing 100193, China

###### Abstract

Supervised machine learning algorithms, such as graph neural networks (GNN),
have successfully predicted material properties. However, the superior
performance of GNN usually relies on end-to-end learning on large material
datasets, which may lose the physical insight of multi-scale information about
materials. And the process of labeling data consumes many resources and
inevitably introduces errors, which constrains the accuracy of prediction. We
propose to train the GNN model by self-supervised learning on the node and
edge information of the crystal graph. Compared with the popular manually
constructed material descriptors, the self-supervised atomic representation
can reach better prediction performance on material properties. Furthermore,
it may provide physical insights by tuning the range information. Applying the
self-supervised atomic representation on the magnetic moment datasets, we show
how they can extract rules and information from the magnetic materials. To
incorporate rich physical information into the GNN model, we develop the node
embedding graph neural networks (NEGNN) framework and show significant
improvements in the prediction performance. The self-supervised material
representation and the NEGNN framework may investigate in-depth information
from materials and can be applied to small datasets with increased prediction
accuracy.

The traditional way of material design strongly relies on the experience and
intuition of experts. Trial-and-error experiments are performed to synthesize
new materials, usually requiring a long period and high cost. The traditional
computational methods are based on domain knowledge, including density
functional theory, molecular dynamics, and density matrix renormalization
group. A new methodology, the machine learning method, has emerged in material
science and has been successfully applied to predict various properties of
materials[1] and the inverse design of materials aiming at specific
properties[2].

The data representation of materials, the so-called material descriptor, is
one of the core elements for the application of machine learning in materials
science[3]. Manually constructed material descriptors can be classified into
two categories—local atomic descriptors and global descriptors[4, 5, 6, 7].
The local atomic descriptor[8, 9], e.g., orbital field matrix (OFM)[10],
describes an atom and its surrounding environment, which can be directly used
as the input of machine learning algorithm to study physical properties at the
atomic level[11, 10, 12]. The global descriptor describes bondings or
interactions between atoms, such as Coulomb matrix (CM)[13] and sine matrix
(SM)[14], which are suited for the global properties of the system.

The deep learning method[15], a rising star of the machine learning family,
has also achieved remarkable performance in materials science[16] with its
huge number of trainable parameters and high non-linearity. Moreover, compared
to manually constructed material descriptors, deep neural networks can be
trained to extract high-level material representations relevant to the target
material properties in an automatic way[17, 18]. Recently, Xie et al. [19]
developed a crystal graph convolutional neural networks (CGCNN) framework to
encode crystalline materials by using graph-structured data. The CGCNN has
achieved excellent performance in predicting several material properties, for
instance, formation energy and band gap.

However, there is still room for improvements in applying GNN to materials
science. First, the power of GNN relies on large datasets typically containing
more than 100,000 materials[20], whereas experimental or high-fidelity
computational datasets with more physical information are usually small. The
GNN on small datasets often overfits and significantly complicates the
training process. As a result, GNN loses its advantage over standard machine
learning algorithms combined with manually constructed descriptors if the
dataset is not large enough[21]. Second, the popular graph machine learning
algorithms in material science are primarily supervised learning that relies
on data with labels, so they cannot fully utilize the large amount of material
data that has not been labeled through calculations or experiments. On the
other hand, labeling data requires high experimental or computational costs in
material science, and the labeling process inevitably causes errors
simultaneously. Therefore, it is crucial to reduce the reliance on labeled
data.

The unsupervised pre-training strategy serves as an effective solution to
reduce the reliance on labeled data, which may substantially enhance the
prediction ability of the trained model. For example, the recently developed
ChatGPT is a powerful and incredibly insightful conversational AI[22]. As a
variation of the GPT series[23], the generalization ability of ChatGPT is
undoubtedly rooted in the generative pre-training (GPT) procedure on the
unprecedentedly massive dataset. While unsupervised or generative pre-training
has led to tremendous success in natural language processing[24] and computer
vision[25], this methodology still needs to be widely taken in material
science. Self-supervised learning on graphs is a type of unsupervised learning
which constructs learning tasks by utilizing the node information, edge
information of the graph itself[26], or augmentations of the graph[27]. The
self-supervised pre-training strategy may make it a cutting-edge research area
in the graph machine learning community[28] and possess the potential to train
GNN models for materials.

Here we propose a self-supervised learning strategy on crystal graphs for a
general multi-scale atomic representation using a computational material
dataset. This strategy enables the GNN to effectively capture the rules of
elements and the local structure in materials by recovering the randomly
masked atomic properties and the distance between atoms. The trained GNN
generates fixed-length, low-dimensional vector representations for atoms in
materials, storing physical knowledge transferred from the GNN model. By
concatenating the atomic representations of different GNN layers, the strategy
can effectively alleviate the over-smoothing problem caused by the deep GNN.
It can also fuse information within different spatial ranges, which helps
better capture the complex interactions in materials and gain physical
insights. Compared with the manually constructed descriptors, the self-
supervised learning strategy has several benefits: (1) The length of the
vector representation from the manually constructed descriptors increases
rapidly with the total number of elements in the dataset or the maximum number
of atoms in the unit cell, and may hinder its applications to more diverse
material datasets. Nevertheless, the length of the self-supervised atomic
representation is fixed. (2) Local atomic descriptors only encode the nearest
neighbor information, while our proposed multi-scale self-supervised atomic
representation may involve larger-scale information relevant to specific
properties. (3) Manually constructed descriptors usually incorporate domain-
specific knowledge, while self-supervised learning can extract more general
information from materials. Then we generate a dataset of experimental
magnetic moments as an illustrative example. The self-supervised atomic
representations can be averaged to study various material properties with
higher accuracy and lower computational cost than usual manually constructed
descriptors. They can also further boost the performance of the GNN model by
combining it with the graph structure of materials, which leads us to a node
embedding graph neural network (NEGNN) framework.

The paper is organized as follows: In Sec.II, we propose the self-supervised
learning strategy for the crystal graph neural network. In Sec.III-V, we
generate self-supervised atomic embeddings(NEs) from the self-supervised pre-
trained graph neural networks. We show its advantages by calculating the
cosine similarity, analyzing its prediction performance on magnetic moments,
and visualizing the low dimensional plots. Sec.VI demonstrates the benefits of
multiscale representations. In Sec.VII, we test the self-supervised graph
embeddings (GEs) for material properties and develop the NEGNN framework.
Finally, we summarize our framework and make comparisons of the NEGNN with
some other popular GNN models in Sec.VIII. Figure 1 illustrates the high-level
pipeline of our work.

Figure 1: The high-level pipeline of the proposed framework.

Results

Self-supervised learning on crystal graphs. We propose a new strategy of
performing self-supervised learning on crystal graphs using a computational
material dataset from the Materials Project[31]. The dataset size is chosen to
be 60,000, which is not large compared with a typical self-supervised learning
task but large enough to illustrate the effectiveness of our strategy. The
types of self-supervised learning tasks on graphs can be classified into three
categories: generative, contrastive, and predictive[28]. Our strategy are
mainly based on the predictive tasks for simplicity.

Figure 2: The self-supervised training of crystal graph neural networks and
the generation of node embeddings from the self-supervised pre-trained GNN.
(a) Predictive self-supervised training on crystal graphs (AB primitive cell
as an illustrative example), where the dark green (dark red) balls represent B
(A) atoms, the dark green (dark red) bar is the node vector corresponding to B
(A) atom, the red crosses denote randomly masked node or edge information
during training. A GNN is trained to recover the masked information based on
surrounding crystal environment. (b) 64-dimensional atomic vectors generated
from the self-supervised pre-trained GNN given a crystal graph as input (AB
primitive cell as an illustrative example), the dark red (dark green) bars at
the bottom are single-scale atomic vectors of different layers, while those on
the right are multi-scale atomic vectors.

For predictive tasks shown in Fig. 2, we randomly mask a certain proportion of
nodes in a crystal graph during training to ensure that the elemental
information of atoms corresponding to masked nodes is not available. The
masked information is the period and the group of an element, following the
conventions defined in the previous study[19], corresponding to classification
problems of 9 categories for period and 18 categories for group. Besides, we
can also randomly mask a certain proportion of edges connecting to masked
atoms which encode the distance between atoms, to perform a classification
problem of 41 categories. Then we train the GNN to reproduce the proper labels
based on the edges and neighbouring nodes of the masked atoms. As a result,
the model gradually learns the rules of chemical constituents of materials and
can capture high-level information about local structures. The loss function
of each classification problem in the predictive self-supervised learning is
the cross-entropy loss function $L_{pred}$ implemented in Pytorch[29], defined
as

$\displaystyle L_{pred}$ $\displaystyle=$
$\displaystyle\log\left(\mathbf{z},c\right)$ (1) $\displaystyle=$
$\displaystyle-z_{c}+\log\left(\sum_{j=1}^{M}\exp(z_{j})\right),$

where the output vector $\mathbf{z}$ is an $M$-dimensional vector, $M$ the
number of categories in the classification problem, and $c$ the proper label.
The definition of the total loss function in predictive self-supervised
learning is the sum of the loss functions of two node-level and one edge-level
classification tasks. The learning curves of the self-supervised learning task
are shown in Figs. 3. We can see that the learning curves of the predictive
task in Fig. 3 and Fig. 3 converge to different values depending on the number
of categories in the classification task. The number of categories
corresponding to the node is relatively small, 9 and 18 for period and group
numbers, respectively, therefore the accuracy can achieve almost 1. On the
other hand, the number of categories corresponding to the edge is relatively
large, 41 categories, and the accuracy eventually reaches nearly 80%. All the
training processes converge after 100 epochs.

Figure 3: The training curves of the self-supervised learning process. (a) The
training accuracy vs epoch plot for predictive self-supervised learning. (b)
The training loss vs epoch plot for predictive self-supervised learning.

For the GNN architecture, we first transform the node vector of length 92 and
the edge vector of length 41 to the initial embeddings of length 64 by a
linear layer as the preprocessing step. Then, we put the preprocessed crystal
graph into a 5-layer CGCNN and update the node vectors by the graph structure
through the message passing mechanism. We utilize the CGConv operator
implemented by Pytorch-geometric[30] as convolutional operations,

$\mathbf{x}_{i}^{{}^{\prime}}=\mathbf{x}_{i}+\sum_{j\in
N(i)}\sigma\left(\mathbf{z}_{i,j}\mathbf{W}_{f}+\mathbf{b}_{f}\right)\odot
g\left(\mathbf{z}_{i,j}\mathbf{W}_{s}+\mathbf{b}_{s}\right)$ (2)

where $\mathbf{z}_{i,j}=[\mathbf{x}_{i},\mathbf{x}_{j},\mathbf{e}_{ij}]$ is
the concatenation of the central atom $\mathbf{x}_{i}$, the neighboring atoms
$\mathbf{x}_{j}$ and the edges $\mathbf{e}_{i,j}$ between them. $\mathbf{W}$
and $\mathbf{b}$ are learnable weights. $\sigma$ and $g$ are sigmoid and soft
plus activation functions respectively. $\odot$ is element-wise product. Next,
we perform a BatchNorm layer to stabilize the training process, speed up the
convergence after each convolutional layer, and finally perform a ReLU
nonlinear layer and a Dropout layer to enhance the expressive and
generalization abilities.

The generation of atomic representations and over-smoothing problem in deep
GNN. We train a five-layer GNN model by self-supervised learning to capture
the high-level information about elements and local structures of materials in
the dataset. Once the GNN model is trained, we can generate 64-dimensional
atomic representations for each material through a single forward pass shown
in Fig. 2. Through the message passing mechanism of GNN, the atomic
representations of each layer are supposed to have some information about
previous layers. However, it may not be ideal to take the atomic
representation of the last layer as the final atomic representation in the
trained GNN model. On the one hand, deeper GNNs can capture long-range
interactions and extract more abstract and complex representations; on the
other hand, deep GNNs often suffer the over-smoothing problem[32, 33]. More
specifically, a $K$-layer GNN model means that nodes are updated by
aggregating the information from their K-hop neighborhoods. As the number of
layers increases, the receptive fields of different nodes will overlap
significantly so that the final representations of nodes tend to converge to
the same vector. For a CGCNN model, the over-smoothing problem may be more
severe for materials with less atoms in the primitive cell.

Figure 4: The cosine similarity plot. Cosine similarity plot between atomic
vectors of Na (K) atoms and Cl atoms in NaCl (KCl) primitive cell under
different GNN layers. The blue(red) lines correspond to single(multi)-scale
atomic vectors.

An excellent atomic descriptor should be able to distinguish between atoms
located in different crystal environments[3, 34]. We define a simple
indicator, cosine similarity, as the similarity measure of atomic
representations. The cosine similarity of $\mathbf{x}_{1}$ and
$\mathbf{x}_{2}$ is defined as
$\dfrac{\mathbf{x}_{1}\cdot\mathbf{x}_{2}}{x_{1}\times x_{2}}$. We take the
two-atom primitive cells of NaCl and KCl as examples, for which are expected
with over-smoothing problems. After putting the corresponding crystal graphs
into the five-layer self-supervised pre-trained GNN, we can get the node
embeddings (NE) of Na (K) atoms and Cl atoms of five different GNN layers. NE0
refer to the initial embeddings, and NE$i$, ($i=1,2,3,4,5$) refer to the
atomic representations of each GNN layers. The cosine similarity between Na
(K) and Cl embeddings under the same GNN layer is calculated to indicate the
degree of over-smoothing, the results are shown in Fig. 4. We can see that the
similarity increases rapidly, starting from the second GNN layer. The atomic
representations of the last layer reach the highest similarity, i.e. the
representation of Na (K) atom is almost indistinguishable from Cl atom, and
the atomic representations of the initial and first layers bear the lowest
similarity, which indicates that the atomic representations of the last GNN
layer significantly lose their local, atomic-level information.

Since the information about elements and the local environments is crucial to
the physical properties, we choose NE01 as our initial multi-scale atomic
representation, i.e., the concatenation of NE0 and NE1. The multi-scale atomic
representations have better distinguishability, as shown in Fig. 4. We further
fine-tune the target properties with corrections from a larger spatial scale
beyond NE01, and next generate an experimental magnetic moment dataset to
obtain some physical insights and illustrate our strategy.

The performance of self-supervised atomic representation on magnetic moments.
The magnetic moments in solids are different from the isolated magnetic
moments due to the crystal field environment and the interactions between
atoms. Therefore the study of the magnetic moments in solids provides an
excellent platform to verify the effects of environmental information within
the crystals at different ranges. Furthermore, the magnetic moment is the
critical quantity that determines the magnetic properties of materials, and it
is a crucial step of high-throughput screening and designing of conventional
magnetic materials[10, 35, 36] and magnetic topological materials[37, 38].

The dataset of experimental magnetic moments is generated from the
experimental antiferromagnetic material database MAGNDATA[39, 40]. This
database contains more than 1,500 experimental magnetic structures; after
removing the magnetic structures that cannot be processed by Pymatgen[41],
such as disordered magnetic structures and incommensurate magnetic structures,
there remain 1,137 magnetic materials, covering 79 elements in total. We
generate 13,276 atoms with nonzero magnetic moments, but we only retain atoms
with different magnetic moments in the same material to avoid possible
duplicate data. The remaining dataset of size 1816 covers 29 elements,
including 13 transition metal elements (Co, Rh, Ni, V, Mn, Mo, Fe, Re, Os, Cu,
Ir, Cr, Ru), 12 lanthanides (Ho, Ce, Tm, Dy, Pr, Tb, Nd, Yb, Eu, Gd, Er, Sm),
and four other elements (Ga, Np, U, Pu), where Fe, Mn, Co, and Ni has the
largest number of magnetic moments among all the elements.

We utilize the kernel ridge regression (KRR) algorithm in this work, which is
widely used in material science, and split the dataset into a training set and
a test set at a ratio of 8:2. Then we perform 10-fold cross-validation and
grid search on the training set to determine the best hyperparameters of the
regression model, and apply them to the test set. The hyperparameters of KRR
are kernels such as RBF (radial basis function) kernel or laplacian kernel,
where the kernel width is $\gamma$, and the regularization strength is
$\alpha$. The range of each hyperparameter is listed in Table 1.

Table 1: The hyperparameters of KRR and CGCNN.

Hyperparameters of KRR Range Hyperparameters of CGCNN Range kernels
[’laplacian’, ’rbf’] Learning rate [1e-5, 1e-4, $\cdots$, 1e-1] kernel width
$\gamma$ [1e-10, 1e-9, $\cdots$, 1e-0] The number of GNN layers [2,3,4,5]
regularization strength $\alpha$ [1e-10, 1e-9, $\cdots$, 1e-0] The number of
hidden layers [1,2,3,4] Dimension of node vectors in GNN [32, 64, 96, 128]

This cross-validation procedure usually gives a more reasonable evaluation for
small datasets[20]. We retrain the model based on the optimal set of
hyperparameters and obtain the scores on the test set. We repeat the previous
procedure five times using different train-test splits to get a reasonable
estimate of model performance on a small dataset. The final score is averaged
over five test scores, and the standard deviation over five test scores is
taken as uncertainty. However, for a dataset more prominent than 10,000, we
get a final score on the test set by a single train-validation-test split. The
metrics for evaluating the prediction performance of the model are mean
absolute error (MAE), mean squared error(MSE), R2 score, and Pearson
correlation coefficient (PCC). The inputs are the orbital field matrix (OFM)
and the atomic representations of different GNN layers and their
concatenations, i.e., NE0, NE1, …, NE01, … All the inputs are fixed-length
vectors.

Table 2: Prediction performance of the KRR model on the experimental magnetic
moment dataset when OFM and different single-scale atomic descriptors are
taken as inputs, where the uncertainties are the standard deviations of test
scores on five random test sets. The unit is $\mu_{B}$.

OFM NE0 NE1 NE2 NE3 NE4 NE5 MAE 0.860$\pm$0.107 0.997$\pm$0.086
0.805$\pm$0.066 0.898$\pm$0.070 0.956$\pm$0.055 1.039$\pm$0.073
0.997$\pm$0.081 MSE 1.477$\pm$0.230 1.798$\pm$0.281 1.536$\pm$0.224
1.822$\pm$0.305 2.067$\pm$0.300 2.271$\pm$0.354 2.085$\pm$0.335 R2
0.684$\pm$0.059 0.620$\pm$0.053 0.674$\pm$0.049 0.614$\pm$0.062
0.561$\pm$0.063 0.518$\pm$0.077 0.558$\pm$0.068 PCC 0.836$\pm$0.025
0.797$\pm$0.025 0.823$\pm$0.031 0.785$\pm$0.040 0.756$\pm$0.040
0.727$\pm$0.050 0.747$\pm$0.047

Let us compare the performance of different single-scale atomic descriptors in
predicting the local magnetic moments in solid materials. From Table 2, we can
see that the prediction ability of self-supervised atomic representations of
different GNN layers varies considerably, and the overall trend of MAE from
NE1 to NE5 is consistent with the cosine similarity shown in Fig. 4, which
indicates the significance of the distinguishability of atomic representations
in property prediction. However, the initial elemental embedding NE0 is an
exception. It shows the lowest cosine similarity among different
representations but the highest MAE in predicting magnetic moments. The
exception lies in the fact that NE0 contains no information of specific
crystal environments through convolutions. It contains only some low-level
information of materials through the weights of the linear layer, which is not
relevant enough to magnetic moments. On the other hand, it is worth noting
that the performance of NE1 or NE2 can be comparable to the manually
constructed atomic descriptor OFM. Furthermore, the length of the vector
representations of each GNN layer is only 64, much smaller than the
1056-dimensional vector representation of OFM, which means that we can achieve
similar or slightly higher accuracy in predicting magnetic moments at a lower
computational cost. We also test the performance of self-supervised atomic
vectors on magnetic moments with several other popular ML models, such as the
Support Vector Regression (SVR) and Random Forest (RF) regression in addition
to the KRR model, as shown in Fig. 5. We found that the SVR and KRR models
achieve similar prediction performances, while the RF model performs the worst
among them. It should also be noted that the prediction performances of the
three ML models share a similar trend from NE0 to NE5, which indicates the
robustness of self-supervised vectors against their scales.

Figure 5: The comparison of prediction performance among various machine
learning models on the magnetic moment dataset. The prediction MAE of the KRR
model, the SVR model and the RF model on the magnetic moments when NE0, NE1,
…, NE5 are taken as inputs vectors. Table 3: Prediction performance of the KRR
model on the experimental magnetic moment dataset when different multi-scale
self-supervised atomic descriptors as well as NE01+OFM are taken as inputs,
where the uncertainties are the standard deviations of test scores on five
random test sets. The unit is $\mu_{B}$.

NE01 NE012 NE0123 NE01234 NE012345 NE01+OFM MAE 0.719$\pm$0.054
0.726$\pm$0.043 0.737$\pm$0.043 0.749$\pm$0.039 0.739$\pm$0.044
0.800$\pm$0.070 MSE 1.289$\pm$0.226 1.318$\pm$0.235 1.334$\pm$0.215
1.341$\pm$0.227 1.330$\pm$0.227 1.371$\pm$0.241 R2 0.727$\pm$0.045
0.721$\pm$0.046 0.718$\pm$0.043 0.716$\pm$0.045 0.719$\pm$0.044
0.710$\pm$0.049 PCC 0.854$\pm$0.027 0.851$\pm$0.282 0.849$\pm$0.026
0.848$\pm$0.028 0.849$\pm$0.027 0.855$\pm$0.028

Then we construct the multi-scale atomic descriptors by concatenating the
atomic representations of different GNN layers, to combine the information
about the environment or interactions at different spatial ranges between
atoms. We can see from comparing Tables 2 and 3 that the overall performance
of multi-scale atomic descriptors is more satisfactory than that of a single
GNN layer. NE01 achieves the best performance, related to its lowest cosine
similarity shown in Fig. 4. From NE01 to NE012345, with the information of a
more extensive range, the performance is, however, declining. The reason might
be that the larger-scale information is more like the environmental noises and
irrelevant to the magnetic moments, which is related to the fact that the
magnetic moments of transition metals and lanthanides is mainly affected by
local crystal field and spin-orbit coupling, regardless of higher-order
environmental corrections. We also find that NE01 concatenating with OFM can
improve the performance of OFM alone. On the one hand, NE01 may contain more
physical inductive bias, i.e., high-level information about elements and
structures, increasing the expressive power of a single OFM; on the other
hand, NE01, as a highly compact local atomic descriptor, only has a length of
128 in vector representation, and significantly improve the prediction ability
of OFM. The MAE and MSE decrease 7.0% and 7.2% respectively, and the R2 and
PCC increase 3.8% and 2.3% respectively, with a slight decrease of
computational costs, which may contribute to the toolbox of feature
engineering[42].

Figure 6: The comparison between random representations and self-supervised
atomic representations on the magnetic moment dataset. The MAE of the KRR
model on the magnetic moment dataset when the self-supervised atomic
representations, SSL(red bars), and random atomic representations, Random(blue
bars), are taken as inputs. The green (cyan) vertical lines denote
uncertainties for SSL(Random).

The analysis of self-supervised atomic representations and dimensional
reduction. The model we trained on the computational material dataset can
generate atomic representations for unknown experimental magnetic materials,
whose overall sound performance shows that our self-supervised training
strategy can indeed guide the model to learn the essential features of
material data, i.e., rules of elements and structures.

In order to highlight the benefits of self-supervised learning, we compare the
atomic representations NE-SSL generated by the self-supervised pre-trained
model (single-scale descriptor in Table 2) and the atomic representations NE-
Random generated by the GNN model with randomly initialized weights, whose MAE
are shown in Fig. 6. First of all, we should note that the performance of NE-
Random of different GNN layers is not as bad as expected, which is also
indicated by the large positive R2 score in Supplemental Material. It makes
sense due to two facts: (1) the input crystal graphs already contain a rich
relational inductive bias; (2) the atomic representations obtained by the
message passing mechanism can receive the information about elements and
structures from the crystal graphs. Importantly, we can observe that the MAE
of atomic representations NE-SSL of different GNN layers is systematically
lower than the MAE of NE-Random, with a much smaller standard deviation, which
undoubtedly proves that the self-supervised learning can indeed optimize the
generated atomic representations.

To further characterize the rich information in the atomic representations
generated by the self-supervised pre-trained GNN model, we utilize the non-
linear, dimensional reduction method t-SNE[43] to map the 128-dimensional NE01
atomic representations to two-dimensional (2D) space, visualizing and labeling
these 2D points with element types and local environments.

Figure 7: t-SNE visualization of self-supervised atomic vectors NE-SSL and
random atomic vectors NE-Random with element labels and local environment
labels, where the different colors represent different elements or local
environments. (a) 2D distribution of NE-Random under element labels. (b) 2D
distribution of NE-SSL under element labels. (c) 2D distribution of NE-Random
under local environment labels. (d) 2D distribution of NE-SSL under local
environment labels. The meaning of abbreviations is listed as follows: oct
means octahedral, cp means close-packed, bcc means body-centered cubic, tet
means tetrahedron, and trig means trigonal bipyramidal.

We use Pymatgen’s tools to identify the local environment around an atom
located in the crystal. For the generated magnetic moment dataset, there are
685 magnetic atoms in the octahedral local environment, 186 atoms in the
close-packed local environment, 69 atoms in the body-centered cubic local
environment, 40 atoms in the tetrahedron local environment, and 35 atoms in
the trigonal bipyramidal local environment. The tool cannot accurately
identify the remaining 800 magnetic atoms, so they are not analyzed in this
study. The t-SNE visualizations of NE01 atomic representations NE-SSL and NE-
Random are shown in Figs. 7 \- 7. Figs. 7 and 7 show that the atomic vectors
NE-SSL generated by the self-supervised pre-trained GNN are well clustered
under element labels, especially for transition metal elements shown in Fig.
7. Lanthanides, such as Nd, Tb, and other grey-colored elements, are also
clustered in the upper left region of Fig. 7. In contrast, the random atomic
vectors NE-Random show no clear organizational pattern in Fig. 7, proving that
self-supervised learning can capture rich chemical rules.

The distribution of the local environments is shown in Figs. 7 and 7, where we
can see that the octahedral local environment is the most typical type since
magnetic atoms favor local octahedral environments to generate significant
magnetic moments. The distribution of the local octahedron environment is
dispersed into several small clusters due to the competition between different
types of elements. We can see that the distribution of NE-SSL has more
apparent patterns than of NE-Random. On the one hand, the distribution of the
local octahedral environment is more concentrated with relatively large
clusters; on the other hand, the patterns of body-centered cubic and local
tetrahedral environments are also apparent.

From previous analysis, we can see the GNN can indeed learn the general rules
of the elemental and structural distribution in materials. They keep the
learned rules in the form of weights that are further transferred to the
generated atomic representations. Therefore, NE-SSL has richer physical
information than NE-Random, leading to performance improvements.

The superiority of multi-scale atomic representation. To further analyze the
results shown in Table 2 and Table 3, we pick out transition metal and
lanthanide elements from a test set. The test set size is 255 for transition
metal elements and 93 for lanthanide elements, respectively. We record the MAE
of different atomic descriptors on the test set, shown in Fig. 8.

Figure 8: The comparison between single-scale and multi-scale atomic
representations on the magnetic moment dataset. The MAE of the KRR model on
the magnetic moments of transition metal(red) and lanthanide(blue) in a test
set given (a) single-scale and (b) multi-scale self-supervised atomic vectors
as input.

We can find that each atomic descriptor’s prediction error of lanthanide
elements is much larger than that of transition metal elements on the magnetic
moment. The reason is twofold. (1) It may be caused by the different training
data sizes of 1257 and 470 for transition metals and lanthanides,
respectively. The small amount of lanthanides may lead to poor fitting in the
training process. As a result, the machine learning model fails to learn the
general rules about the magnetic moments of lanthanides compared to transition
metals. (2) It may also result from the fact that the magnetic properties of
lanthanides are more complex than transition metal elements. From NE0 to NE1,
the prediction errors of transition metals and lanthanides drop significantly,
which means the near-site environment already contains the relevant
information about the local crystal field and spin-orbit coupling. For single-
scale atomic representations from NE1 to NE4, the prediction errors of
transition metals and lanthanides rapidly increase, indicating that atomic
representations of the deeper GNN layer have a sustained loss of information
at the local level, which corresponds to the over-smoothing problem; For
multi-scale atomic descriptors from NE01 to NE012345, with the information of
the larger spatial scales, the prediction errors of transition metals only
slightly fluctuate and the prediction errors of lanthanides increase at a much
smaller amount than those from NE1 to NE4. These results show that multi-scale
atomic representations can preserve the essential information related to the
magnetic moment and resist information loss and noises.

We can also see from Tables 2 and 3 that the MSE is much greater than the MAE
of each atomic descriptor, which indicates that there are outliers. So it is
interesting to analyze the parity plots of predicted values versus the
experimental values from the test set. We take the parity plot of NE01 as an
instance to disclose more physical insights.

Figure 9: The parity plots of NE and NE+OFM on the test dataset. (a) Parity
plot of the KRR model on a test set when the NE01 is taken as input, the green
and blue points are transition metals and lanthanides respectively, the texts
colored in red show prediction errors of the corresponding outliers. (b)
Parity plot of the KRR model on a test set when the NE01+OFM atomic
representation is taken as input, the red arrows indicate changes in
prediction performance after incorporating OFM with NE01 compared to (a).

First, it is worth mention that in the test set of size 364, 52.2% of elements
satisfying $|\mathrm{Exp}-\mathrm{Pred}|\leqslant 0.5\mu_{B}$, and 76.9% of
elements satisfying $|\mathrm{Exp}-\mathrm{Pred}|\leqslant 1.0\mu_{B}$. In
contrast, by the spin-density functional theory (SDFT), the errors of DFT
values to experimental values within $0.5\mu_{B}$ and $1\mu_{B}$ are 51.9% and
77.22%, respectively[36]. The SDFT calculation is also based on the MAGNDATA
database, however, with much more computational resources.

Next, we discuss the outliers in the parity plot shown in Fig. 9. (1) For
transition metal elements, the magnetic moment with the most significant
prediction error comes from the Mn atoms in Mn6Ni16Si7[44], which are
2.88$\mu_{B}$ and 2.93$\mu_{B}$ respectively, shown as two blue squares
located within $4\mu_{B}<\mathrm{Exp}<5\mu_{B}$ in Fig. 9. Mn6Ni16Si7 is a
material with rich physics in magnetism. For example, there is geometric
frustration of antiferromagnetism in the octahedral structure formed by six Mn
atoms below 197K. (2) For lanthanides, the most significant prediction error,
6.93 $\mu_{B}$, comes from the Dy atoms in Dy3Ru4Al12[45], shown as the green
pentagon located within $0<\mathrm{Exp}<1\mu_{B}$ in the upper left corner of
Fig. 9. The non-collinear magnetic structure of the Dy3Ru4Al12 is caused by
the competition between the RKKY interaction and the anisotropy induced by the
crystal field. (3) The prediction error of Tb atom in Tb5Ge4[46] is 4.13
$\mu_{B}$, shown as the green cross located within
$8\mu_{B}<\mathrm{Exp}<9\mu_{B}$ in Fig. 9, which exhibits strange behaviors
caused by spin reorientation transition of the canted antiferromagnetic
structure. (4) The prediction error of Gd atoms in GdVO4[47] is 4.89
$\mu_{B}$, shown as a green hexagon located at nearly $\mathrm{Exp}=7\mu_{B}$,
which has high specific heat and strong magnetocaloric effect above 2.5 K. The
four outliers mentioned above all have non-trivial magnetic structures or
magnetic properties, indicating that the revealed materials with significant
prediction errors have some curious physics and are worth further analysis.

Finally, we compare the parity plots of NE01 and NE01 concatenated with OFM
(NE01+OFM), shown in Fig. 9. We can see that the R2 score of NE01+OFM
increases 4.3% over NE01 alone with more explicit physical information
involved in OFM, which means that the overall prediction performance improves.
In particular, three lanthanides with the most significant prediction errors
marked in Fig. 9 are moving towards the experimental values. The significant
improvement in predicting the magnetic moment of the Gd atom is similar to the
SDFT calculation by adding Hubbard U in some lanthanides[36].

The self-supervised material representation and the NEGNN framework. After
analyzing the results of self-supervised atomic representations on the
magnetic moment dataset, we turn to study various kinds of material
properties. First, we treat the representations of different layers as
environmental corrections at different spatial ranges and fine-tune the object
properties by adjusting the environmental information included in the
descriptors. On the one hand, like the manually constructed local atomic
descriptors, we construct the graph embeddings (GE) of materials by simply
averaging the atomic representations of the same material. They can be
directly used as inputs of the KRR model to fit structure-property relations.
On the other hand, the expressive ability of GNN can be enhanced by
incorporating rich physical information or structural information[48, 49] into
the initial node vectors of original crystal graphs. Therefore, we introduce
the NEGNN framework, which combines self-supervised atomic representations
with graph structure of materials to perform end-to-end learning of specific
properties.

Figure 10: The prediction performance of multi-scale GEs on the formation
energy and band gap. The MAE of the KRR model on the (a) formation energy
dataset and (b) HSE bandgap dataset, when different multi-scale self-
supervised material representations are taken as inputs. GE012 achieves the
lowest prediction error indicated by the red circle. Table 4: Prediction
performance of the KRR model with different material descriptors on the
formation energy dataset and HSE bandgap dataset. We choose the self-
supervised multi-scale descriptors, GE012, with the best performance among the
descriptors shown in Fig. 10. The units of formation energy and HSE band gap
are eV/atom and eV respectively.

Formation Energy HSE band gap ESM ECM SM OFM GE012 SM OFM GE012 MAE 0.49 0.64
0.37 0.282$\pm 0.018$ 0.192$\pm$0.013 1.387$\pm$0.015 0.935$\pm$0.028
0.816$\pm$0.034 MSE — — — 0.158$\pm$0.017 0.081$\pm$0.010 3.464$\pm 0.178$
1.587$\pm$0.101 1.302$\pm$0.061 R2 — — — 0.857$\pm$0.014 0.927$\pm$0.005
0.305$\pm$0.040 0.680$\pm$0.036 0.738$\pm$0.020 PCC — — — 0.932$\pm$0.007
0.966$\pm$0.003 0.575$\pm$0.024 0.838$\pm$0.018 0.863$\pm$0.015

To highlight the general information about elements and structures within the
embeddings, we compare the performance of GE with several manually constructed
descriptors and the CGCNN model on various material properties such as
formation energy, HSE bandgap, and elastic properties. (1) We choose the
formation energy dataset from Ref. [14] and the corresponding 3,725 material
structures from the Materials Project. The MAE reported in the Ref. [14] for
Ewald sum matrix (ESM), extended Coulomb matrix (ECM) and SM are 0.49eV/atom,
0.64eV/atom and 0.37eV/atom respectively. (2) We choose the HSE bandgap
dataset from the MaterialGo website[50] and the corresponding 6,030 material
structures from the Materials Project. To perform the regression task, we only
hold the 2,711 materials labeled by non-zero bandgap and with no more than 50
atoms in a unit cell for a reasonable length of the matrix-based descriptors
such as SM, CM, and ESM. (3) We choose the elastic property dataset from the
Ref. [19] and the corresponding 3,204 material structures from the Materials
Project, which are labeled by three kinds of elastic properties: bulk modulus,
shear modulus, and Poisson ratio.

Figure 11: The prediction performance of multi-scale GEs on the elastic
properties. The MAE of the KRR model on the (a) bulk modulus dataset, (b)
shear modulus dataset, (c) Poisson ratio dataset when different self-
supervised multi-scale material representations are taken as inputs. The units
of bulk and shear modulus are taken as $\log(GPa)$. GE012345 achieves the
lowest prediction error on all of the three elastic properties indicated by
the red circle. Table 5: Prediction performance of the KRR model with
different material descriptors on the three elastic property datasets. We
choose the self-supervised multi-scale descriptors, GE012345, with the best
performance among the descriptors shown in Fig. 11. The units of bulk and
shear modulus are taken as $\log(GPa)$.

Bulk Modulus Shear Modulus Poisson Ratio OFM GE012345 OFM GE012345 OFM
GE012345 MAE 0.121$\pm$0.017 0.088$\pm$0.004 0.142$\pm$0.011 0.112$\pm$0.003
0.034$\pm$0.001 0.032$\pm$0.001 MSE 0.027$\pm$0.003 0.017$\pm$0.002
0.035$\pm$0.004 0.024$\pm$0.001 0.002$\pm$0.000 0.002$\pm$0.000 R2
0.774$\pm$0.021 0.860$\pm$0.014 0.727$\pm$0.034 0.817$\pm$0.017
0.408$\pm$0.026 0.471$\pm$0.028 PCC 0.894$\pm$0.006 0.931$\pm$0.008
0.868$\pm$0.007 0.910$\pm$0.012 0.648$\pm$0.016 0.752$\pm$0.015

Then we analyze the self-supervised multi-scale descriptors. First, we analyze
the interplay between the range information within GE and the material
properties. Fig. 10 shows that the GE012 achieves the best performance among
all multi-scale descriptors on formation energy, indicating that energy-
related material properties can be well decomposed into the contributions from
the local environment near the atom, consistent with the worse performance of
the descriptors from GE0123 to GE012345 which has more extensive environmental
information. It also holds for the bandgap dataset shown in Fig. 10 that the
local environment plays the crucial role, which may result from the principle
of electronic nearsightedness[51]. While we can see from Figs. 11 \- 11 that
the descriptors incorporating the most extensive range of environmental
information, namely GE012345, have achieved the best performance of all three
elastic properties, indicating that the elastic properties are less local than
formation energy and bandgap. Second, we compare the prediction performance of
the self-supervised descriptors and the manually constructed descriptors shown
in Tables 4 and 5. We can see that the self-supervised multi-scale descriptor,
GE, is significantly better than the popular descriptors ESM, ECM, SM, and
OFM, on all considered material properties. On the one hand, GE contain more
general and high-level material information extracted through self-supervised
learning instead of domain-specific information, such as Coulomb interaction
encoded in SM and orbital interaction encoded in OFM. On the other hand, the
compactness of GE reduces the possibility of overfitting and also results in
lower computational costs as shown in Supplementary Material.

Table 6: Prediction performance of CGCNN and NEGNN on all the material
properties.

Formation Energy HSE Band Gap Bulk Modulus Shear Modulus Poisson Ratio CGCNN
NEGNN CGCNN NEGNN CGCNN NEGNN CGCNN NEGNN CGCNN NEGNN MAE 0.190$\pm$0.027
0.180$\pm$0.020 0.792$\pm$0.056 0.705$\pm$0.024 0.066$\pm$0.008
0.066$\pm$0.010 0.100$\pm$0.006 0.096$\pm$0.004 0.033$\pm$0.002
0.029$\pm$0.001 MSE 0.081$\pm$0.010 0.060$\pm$0.011 1.452$\pm$0.658
1.039$\pm$0.104 0.012$\pm$0.003 0.010$\pm$0.002 0.017$\pm$0.001
0.017$\pm$0.001 0.002$\pm$0.000 0.002$\pm$0.000 R2 0.933$\pm$0.019
0.946$\pm$0.008 0.729$\pm$0.085 0.791$\pm$0.028 0.898$\pm$0.021
0.917$\pm$0.015 0.863$\pm$0.008 0.867$\pm$0.012 0.467$\pm$0.032
0.557$\pm$0.027 PCC 0.974$\pm$0.004 0.980$\pm$0.001 0.865$\pm$0.055
0.897$\pm$0.012 0.954$\pm$0.008 0.965$\pm$0.006 0.932$\pm$0.004
0.940$\pm$0.010 0.713$\pm$0.016 0.752$\pm$0.015

The NEGNN is distinct from the CGCNN, for that the input crystal graph is the
combination of self-supervised multi-scale atomic representation and graph
structure of materials. We choose NE01 to combine with graph structure without
loss of generality since it contains minimal sufficient information about
elements and local environments. The prediction performance of CGCNN and NEGNN
is shown in Table 6. For CGCNN and NEGNN, the hyperparameters shown in Table
are the learning rate, the number of GNN layers, the dimension of node vectors
within a GNN layer, and the number of hidden layers. The dimension of the
hidden layer is taken as twice that of node vectors in GNN, which is usually a
good choice. We also choose Adam as the optimizer for simplicity.

Compared with the best performance of the KRR model in Tables 4 and 5, CGCNN
shows improvement in accuracy yet loses computational efficiency on small
datasets[20]. Nevertheless, the self-supervised enhanced GNN, NEGNN, improves
the prediction performance on all material properties over CGCNN. The strength
of self-supervised learning is from the ability to learn more high-level rules
about materials. To check how the performance of self-supervised learning
depends on the size of datasets, we evaluate both the NEGNN and CGCNN on the
formation energy dataset (size 46,744) and bandgap dataset (size 27,430),
denoted by ”Formation Energy+” and ”Bandgap+” in Table 7. As we can see, NEGNN
shows 13.78% improvements over CGCNN in a larger bandgap dataset. In contrast,
it achieves the same MAE as the CGCNN on an even larger formation energy
dataset, indicating that the self-supervised learning strategy preserves its
strength for the larger datasets.

Table 7: Prediction performance of CGCNN and NEGNN on formation energy dataset
and bandgap dataset of larger size

Formation Energy+ Bandgap+ CGCNN NEGNN CGCNN NEGNN MAE 0.039 0.039 0.388 0.341
MSE — 0.005 — 0.325 R2 — 0.995 — 0.854 PCC — 0.998 — 0.924

The comparison of the NEGNN and other popular GNN frameworks. NEGNN’s main
focus is to incorporate as much pertinent material data as possible at the
beginning stage via two fresh methods. On the one hand, it can be a practical
technique for recovering the randomly masked material properties to input
essential information in a self-supervised manner; on the other hand,
multiscale representations can also integrate information at different
physical ranges and enhance the distinguishability between atomic vectors.

At this point, it is necessary to highlight the differences between various
GNN frameworks. The comparison between the original CGCNN[19], the primary
reference of our work, and the NEGNN has already been discussed in detail. The
SchNet[17] is one of the early GNN frameworks in material science, which
utilizes a specific-designed continuous filter as the message-passing layer,
primarily suitable for modeling small molecular systems. The MEGNet[18]
proposed by Chen et al. is another early GNN framework. The MEGNet unifies the
property prediction of crystals and molecules and incorporates a global state
vector for specific purposes. However, the differences between MEGNet and
NEGNN are distinctive: (a) The learned embeddings in MEGNet come from the
supervised training process, which benefits from labeled data. In contrast,
the NEGNN generates embeddings from self-supervised training, requiring only
the primitive structure information of crystals. The NEGNN may benefit when
the labels are computationally expensive or hard to acquire. (b) Regarding
interpretability, both works provide the t-SNE plot of elemental embeddings
and the rediscovery of chemical rules. Nevertheless, the NEGNN offers
additional degrees of freedom to extract physical information, i.e., the scale
of embeddings. The AtomSets[52] is a newly developed framework that aims to
utilize the pre-training of MEGNet to generate universal feature vectors and
improve prediction performance on small datasets. However, there are also
several significant differences between AtomSets and NEGNN: (a) For the pre-
training strategy, AtomSets performs transfer learning using large, labeled
datasets, while our self-supervised approach does not need any labeled
datasets; (b) The NEGNN utilizes multiscale embeddings with lower similarity
and can gain in-depth physical information and better interpretability by
adjusting the scale degree of freedom and analyzing the changes in the test
dataset. For instance, we can verify the importance of local environments to
both transition metal elements and lanthanide elements in Fig. 8; we can
identify some interesting magnetic materials with non-trivial magnetic
behavior through the parity plot in Fig. 9; we can also get a measure of the
locality of target material properties in Fig. 10 and Fig. 11. All of the
physical information is available by multiscale embeddings. Besides the two
early works, we note that a Crystal Twins[53] model was published after our
submission. The Crystal Twins leverage a contrastive training strategy based
on CGCNN with complicated loss functions. However, the cross-entropy loss
functions in NEGNN are more transparent for physical information.

The superiority of NEGNN over CGCNN in accuracy can be seen from TABLE 6 and
TABLE 7. The MEGNet shows MAEs of 0.032 eV/atom and 0.35 eV for formation
energy and band gap[18], using dataset sizes similar to those in TABLE 7. The
performance of MEGNet can be competing with NEGNN. However, only the NEGNN can
survive the computationally expensive or unavailable labels. We have also
tested the state-of-the-art framework, the ALIGNN[54], on the same datasets.
It achieved 0.028 eV/atom and 0.275 eV, which is much better than CGCNN and
NEGNN, and this may be attributed to incorporating more high-level material
information. Notwithstanding, our findings revealed that ALIGNN’s
computational cost is substantially higher than NEGNN’s since it requires
additional GNN layers to update the atomistic line graph expressing the three-
body interactions. This additional depth increases the number of training
parameters, thus significantly increasing computation time. NEGNN still has
the potential to improve by enlarging the unlabeled, pretraining dataset
compared to all the frameworks mentioned above.

Conclusion

We introduce low-dimensional, fixed-length atomic representations by self-
supervised learning on crystal graphs.Combining self-supervised descriptors
with standard machine learning models such as KRR can predict atomic
properties and several material properties more accurately than popular
manually constructed descriptors like OFM and SM while maintaining good
computational efficiency. A standard machine-learning model with self-
supervised atomic representations is more trouble-free to train than the GNN
model. It can avoid the overfitting problem usually suffered by the GNN models
on small datasets. By altering the range of environmental information in the
self-supervised atomic representation, we can gain a machine-learning model
with good physical insights. The predictive self-supervised pre-training
strategy can extract high-level information from unlabeled datasets and
incorporate prior information of materials into the GNN model. The multi-scale
embeddings can extract in-depth physical information with better
interpretability. Based on the strategy of self-supervised learning and the
generated multi-scale embeddings, we develop the knowledge-enhanced NEGNN
framework by combining the self-supervised atomic vectors with the GNN,
significantly improving the performance.

The NEGNN framework is promising for various applications and is open for
further developments. First, more self-supervised learning tasks can be
performed on crystal graphs by encoding explicit structural information like
bond angles, which can capture the more high-level information of materials
and transfer it to atomic representations; Second, self-supervised learning
can capture the local structures by recovering the distance between atoms, and
the GNN can be further regularized by reproducing global structural
information like crystal groups, which may be more potent in predicting
material properties; Third, we have demonstrated that the effectiveness of
self-supervised learning in predicting the atomic-level properties, especially
for magnetic moments. It is not difficult to generalize the self-supervised
atomic representation for other site-related properties in solid materials,
for example, implementing more powerful machine learning models for impurity
levels[55] or bandgap engineering[56, 57].

Code availability

The code and all the datasets of this paper can be available from the GitHub
[58].

## References

  * [1] Pilania, G., Wang, C., Jiang, X., Rajasekaran, S. & Ramprasad, R. Accelerating materials property predictions using machine learning. Sci. Rep. 3, 1-6 (2013).
  * [2] Noh, J., Gu, G., Kim, S. & Jung, Y. Machine-Enabled Inverse Design of Inorganic Solid Materials: Promises and Challenges. Chem. Sci. 11, 4871–4881 (2020).
  * [3] Ghiringhelli, L., Vybiral, J., Levchenko, S., Draxl, C. & Scheffler, M. Big Data of Materials Science - Critical Role of the Descriptor. Phys. Rev. Lett. 114, 105503 (2015).
  * [4] Ceriotti, M., Willatt, M. J. & Csányi, G. Machine Learning of Atomic-Scale Properties Based on Physical Principles. Handb. Mater. Model, 1911–1937 (2020).
  * [5] Musil, F. & Ceriotti, M. Machine Learning at the Atomic Scale. Chimia 73, 972–982 (2019).
  * [6] Musil, F. et al. Physics-inspired structural representations for molecules and materials. Chem. Rev. 121, 9759-9815 (2021).
  * [7] Langer, M. F., Goeßmann, A. & Rupp, M. Representations of molecules and materials for interpolation of quantum-mechanical simulations via machine learning. npj Comput. Mater. 8, 1-14 (2022).
  * [8] Behler, J. Atom-centered symmetry functions for constructing high-dimensional neural network potentials. J. Chem. Phys. 134, 074106 (2011).
  * [9] De, S., Bartók, A. P., Csányi, G. & Ceriotti, M. Comparing molecules and solids across structural and alchemical space. Phys. Chem. Chem. Phys. 18, 13754-13769 (2016).
  * [10] Pham, T. L. et al. Machine learning reveals orbital interaction in crystalline materials. Sci. Technol. Adv. Mater. 18 (2017).
  * [11] Unke, O. T. et al. Machine learning force fields. Chem. Rev. 121, 10142-10186 (2021).
  * [12] Bürkle, M., Perera, U., Gimbert, F., Nakamura, H., Kawata, M. & Asai, Yoshihiro. Deep-Learning Approach to First-Principles Transport Simulations. Phys. Rev. Lett. 126, 177701 (2021).
  * [13] Rupp, M., Tkatchenko, A., Müller, K.-R. & von Lilienfeld, O. A. Fast and Accurate Modeling of Molecular Atomization Energies with Machine Learning. Phys. Rev. Lett. 108, 058301(2012).
  * [14] Faber, F., Lindmaa, A., von Lilienfeld, O A. & Armiento, R. Crystal structure representations for machine learning models of formation energies. Int. J. Quantum Chem. 115, 1094-1101 (2015).
  * [15] LeCun, Y., Bengio, Y. & Hinton, G. Deep Learning. Nature 521, 436-444 (2015).
  * [16] Choudhary, K. et al. Recent advances and applications of deep learning methods in materials science. npj Comput. Mater. 8, 1-26 (2022).
  * [17] Schütt, K. T., Sauceda, H. E., Kindermans, P-J., Tkatchenko, A. & Müller, K-R. Schnet–a deep learning architecture for molecules and materials. J. Chem. Phys. 148, 241722 (2018).
  * [18] Chen, C., Ye, W., Zuo, Y., Zheng, C. & Ong, S. P. Graph networks as a universal machine learning framework for molecules and crystals. Chem. Mater. 31, 3564-3572 (2019).
  * [19] Xie, T. & Grossman, J. C. Crystal Graph Convolutional Neural Networks for an Accurate and Interpretable Prediction of Material Properties. Phys. Rev. Lett. 120, 145301 (2018).
  * [20] Dunn, A., Wang, Q,, Ganose, A., Dopp, D. & Jain, A. Benchmarking materials property prediction methods: the Matbench test set and Automatminer reference algorithm. npj Comput. Mater. 6, 138 (2020).
  * [21] Fung, V., Zhang, J., Juarez, E. & Sumpter, B. G. Benchmarking graph neural networks for materials chemistry. npj Comput. Mater. 7, 1-8 (2021).
  * [22] https://chat.openai.com/chat.
  * [23] Brown, T. et al. Language models are few-shot learners. Advances in Neural Information Processing Systems 33, 1877-1901 (2020).
  * [24] Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018).
  * [25] He, K., Fan, H., Wu, Y., Xie, S. & Girshick, R. Momentum contrast for unsupervised visual representation learning. Proc. IEEE Conf. Comput. Vis. Pattern Recog, 9729-9738 (2020).
  * [26] Hu, W. et al. Strategies for pre-training graph neural networks. arXiv preprint arXiv:1905.12265 (2019).
  * [27] You, Y. et al. Graph contrastive learning with augmentations. Advances in Neural Information Processing Systems 33, 5812-5823 (2020).
  * [28] Wu, L., Lin, H., Tan, C., Gao, Z, & Li, S. Z. Self-supervised learning on graphs: Contrastive, generative, or predictive. IEEE Transactions on Knowledge and Data Engineering (2021).
  * [29] https://pytorch.org.
  * [30] Fey, M. and Lenssen, J. E. Fast Graph Representation Learning with PyTorch Geometric. ICLR Workshop on Representation Learning on Graphs and Manifolds (2019).
  * [31] Jain, A. et al. Commentary: The Materials Project: A materials genome approach to accelerating materials innovation. APL Mater. 1, 011002 (2013).
  * [32] Oono, K. & Suzuki, T. Graph neural networks exponentially lose expressive power for node classification. arXiv preprint arXiv:1905.10947 (2019).
  * [33] Li, Q., Han, Z. & Wu, X. Deeper insights into graph convolutional networks for semi-supervised learning. Thirty-Second AAAI conference on artificial intelligence (2020).
  * [34] Pozdnyakov, S. N., Willatt, M. J., Bartók, A. P., Ortner, C., Csányi, G. & Ceriotti, M. Incompleteness of atomic structure representations. Phys. Rev. Lett. 125, 166001 (2020).
  * [35] Balluff, J., Diekmann, K., Reiss, G. & Meinert, M. High-throughput screening for antiferromagnetic Heusler compounds using density functional theory. Phys. Rev. Mater. 1, 034404 (2017).
  * [36] Huebsch, M.-T., Nomoto, T., Suzuki, M.-T. & Arita, R. Benchmark for Ab Initio Prediction of Magnetic Structures Based on Cluster-Multipole Theory. Phys. Rev. X 11, 011031 (2021).
  * [37] Xu, Y. et al. High-throughput calculations of magnetic topological materials. Nature 586, 702-707 (2020).
  * [38] Choudhary, K., Garrity, K. F., Ghimire, N. J., Anand, N. & Tavazza, F. High-throughput search for magnetic topological materials using spin-orbit spillage, machine learning, and experiments. Phys. Rev. B 103, 155131 (2021).
  * [39] Gallego, S. V. et al. MAGNDATA: towards a database of magnetic structures. I. The commensurate case. J. Appl. Crystallogr. 49, 1750-1776 (2016).
  * [40] Gallego, S. V. et al. MAGNDATA: towards a database of magnetic structures. II. The incommensurate case. J. Appl. Crystallogr. 49, 1941-1956 (2016).
  * [41] Shyue P. O. et al. Python Materials Genomics (pymatgen): A robust, open-source python library for materials analysis. Comput. Mater. Sci. 68, 314-319 (2013).
  * [42] Ward, L. et al. Matminer: An open source toolkit for materials data mining. Comput. Mater. Sci. 152, 60-69 (2018).
  * [43] Van der Maaten, L. & Hinton, G. Visualizing data using t-SNE. J. Mach. Learn. Res. 9 (2008).
  * [44] Ahmed, S. J., Greedan, J. E, Boyer, C. & Niewczas, M. Magnetic and structural studies of G-phase compound Mn6Ni16Si7. Inorg. Chem. 57, 14144-14151 (2018).
  * [45] Gorbunov, DI. et al. Electronic properties of a distorted kagome lattice antiferromagnet Dy3Ru4Al12. Phys. Rev. B 90, 094405 (2014).
  * [46] Ritter, C., Morellon, L., Algarabel, PA., Magen, C. & Ibarra, MR. Magnetic and structural phase diagram of Tb5 (Six Ge1-x)4. Phys. Rev. B 65, 094405 (2002).
  * [47] Palacios, E. et al. Magnetic structures and magnetocaloric effect in R VO4 (R= Gd, Nd). Phys. Rev. B 97, 214401 (2018).
  * [48] Karamad, M., Magar, R., Shi, Y., Siahrostami, S., Gates, I. D. & Farimani, A. B. Orbital graph convolutional neural network for material property prediction. Phys. Rev. Mater. 4, 093801 (2020).
  * [49] Banjade, H. et al. Structure motif–centric learning framework for inorganic crystalline systems. Sci. Adv. 7, eabf1754 (2021).
  * [50] Jie, J. et al. A new MaterialGo database and its comparison with other high-throughput electronic structure databases for their predicted energy band gaps. Sci. China: Technol. Sci. 62, 1423-1430 (2019).
  * [51] Prodan, E. & Kohn, W. Nearsightedness of electronic matter. Proc. Natl. Acad. Sci. 102, 11635-11638 (2005).
  * [52] Chen, C. & Ong, S. P. AtomSets as a hierarchical transfer learning framework for small and large materials datasets. npj Comput. Mater. 7, 173 (2021).
  * [53] Magar, R., Wang, Y. & Barati Farimani, A. Crystal twins: self-supervised learning for crystalline material property prediction. npj Comput. Mater. 8, 231 (2022).
  * [54] Choudhary, K. & DeCost, B. Atomistic line graph neural network for improved materials property predictions. npj Comput. Mater. 7, 185 (2021).
  * [55] Mannodi-Kanakkithodi, A. et al. Machine-learned impurity level prediction for semiconductors: the example of Cd-based chalcogenides. npj Comput. Mater. 6, 39 (2020).
  * [56] Sharma, V., Kumar, P., Dev, P. & Pilania, G. Machine learning substitutional defect formation energies in ABO3 perovskites. J. Appl. Phys. 128, 034902 (2020).
  * [57] Frey, N. C., Akinwande, D., Jariwala, D. & Shenoy, V. B. Machine learning-enabled design of point defects in 2d materials for quantum and neuromorphic information processing. ACS Nano 14, 13406-13417 (2020).
  * [58] https://github.com/Harrykjg-physics/self-supervised-atomic-representations.

Acknowledgements

This work was supported by the Scientific Research Program from Science and
Technology Bureau of Chongqing City (Grant No. cstc2020jcyj-msxmX0684), MOST
of China (Grants No. 2017YFA0303400 and No. 2018YFA0306101), the Science and
Technology Research Program of Chongqing Municipal Education Commission (Grant
No. KJQN202000639), the National Natural Science Foundation of China (NSFC)
(Grants No. 11974340, and No. 61674145), the Chinese Academy of Sciences
(Grants No. XDB28000000, No. QYZDJ-SSW-SYS001, and No. XXH13506-202) and in
part by the National Natural Science Foundation of China under Grant No.
12147102.

Author contributions

Conceptualization, J.-J.Z. and K.C.; investigation, J.-G.K., R.Z., J.-J.Z. and
K.C.; writing—original draft preparation, J.-G.K.; writing—review and editing,
all authors; visualization, J.-G.K. and J.L.; supervision, J.-J.Z. and K.C.;
funding acquisition, J.-J.Z., J.L., and K.C. All authors have read and agreed
to the published version of the manuscript.

Competing interests

The authors declare no competing interests.