# Characterizing out-of-distribution generalization of neural networks:
application to the disordered Su-Schrieffer-Heeger model

Kacper Cybiński1 Marcin Płodzień2 Michał Tomza1 Maciej Lewenstein2,3 Alexandre
Dauphin2,4 Anna Dawid5 1Faculty of Physics, University of Warsaw, Pasteura 5,
02-093 Warsaw, Poland  2ICFO - Institut de Ciències Fotòniques, The Barcelona
Institute of Science and Technology, Av. Carl Friedrich Gauss 3, 08860
Castelldefels (Barcelona), Spain  3ICREA, Pg. Lluís Campanys 23, 08010
Barcelona, Spain  4PASQAL SAS, 2 av. Augustin Fresnel Palaiseau, 91120, France
5Center for Computational Quantum Physics, Flatiron Institute, 162 Fifth
Avenue, New York, NY 10010, USA

###### Abstract

Machine learning (ML) is a promising tool for the detection of phases of
matter. However, ML models are also known for their black-box construction,
which hinders understanding of what they learn from the data and makes their
application to novel data risky. Moreover, the central challenge of ML is to
ensure its good generalization abilities, i.e., good performance on data
outside the training set. Here, we show how the informed use of an
interpretability method called class activation mapping (CAM), and the
analysis of the latent representation of the data with the principal component
analysis (PCA) can increase trust in predictions of a neural network (NN)
trained to classify quantum phases. In particular, we show that we can ensure
better out-of-distribution generalization in the complex classification
problem by choosing such an NN that, in the simplified version of the problem,
learns a known characteristic of the phase. We show this on an example of the
topological Su–Schrieffer–Heeger (SSH) model with and without disorder, which
turned out to be surprisingly challenging for NNs trained in a supervised way.
This work is an example of how the systematic use of interpretability methods
can improve the performance of NNs in scientific problems.

##  I Introduction and motivation

Machine learning (ML) promises a revolution in science, similar to the current
revolution in industry [1]. In quantum physics, neural networks (NNs) serve as
a flexible and promising representation of quantum states [2, 3, 4] and a
booster for quantum technologies [5, 6, 7], e.g., as entanglement classifiers
[8, 9]. Neural networks are especially promising in the detection of phases of
matter and have been used for classical [10, 11, 12, 13, 14, 15], quantum [16,
17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28], and topological [29, 30, 31,
32, 33, 34, 35, 36, 37] phase transitions with supervised [10, 25, 26, 27, 28,
11, 29, 30, 31] and unsupervised [32, 13, 12, 16, 17, 18, 19, 34, 20, 33, 21,
22, 23, 35, 14, 24, 36, 37, 15] approaches as well as for experimental data
[38, 39, 40, 41, 42]. Other examples include ML approaches relying on
dimensionality reduction [43, 44, 45, 46], kernel methods [47, 48, 49],
topological data analysis [50, 51, 52, 53], and quantum NNs [54, 55, 56].

However, before NNs join standard toolboxes for the analysis of phases of
matter, they need to become more interpretable (so we understand what they
learn) and reliable (so we can trust their predictions). Note that
interpretability is a stronger condition than reliability (if we understand
exactly how an NN makes its predictions, we usually can trust it). There are
extensive efforts to make automated approaches more interpretable [17, 57, 58,
23, 52, 59, 35, 60, 61], which should ultimately lead to learning phases of
matter and assisting physicists in understanding the learned order parameters
[62, 63, 64, 65, 41]. However, the community focuses predominantly on a
special representation of data, that is, spin configurations. Therefore, the
question of the interpretability and reliability of NNs applied to different
quantum data remains wide open.

Reliable NNs are expected to generalize robustly to new scenarios [66]. It is
especially challenging in the case of the out-of-distribution (OOD)
generalization when the test data come from a different distribution than the
training data. Naturally, no OOD generalization should be expected for
unrelated training and test distributions. However, a robust model also
performs well under a limited distribution shift. Such an OOD generalization
of a network can be checked when we have enough information on the test
distribution; for example, in a supervised scenario, we have access to labels
of some OOD test data. When we do not have such labels, our trust in the OOD
generalization of an NN has to be limited, especially in the presence of
spurious correlations in the training data.

In this work, we show how we can increase trust in the OOD generalization of
an NN in the absence of labeled OOD data. To this end, we study explanations
of NN predictions and the representation of data learned by an NN and discuss
patterns that correlate well with the network robustness. We perform this
analysis on an example of data coming from a prototypical topological
Hamiltonian, that is, the Su-Schrieffer-Heeger (SSH) model. The training data
come from the standard SSH model, while the OOD test data are from the SSH
model with the disorder, as presented schematically in figure 1(a).

Previous work [32] showed that standard convolutional neural networks (CNNs)
trained on the data from the SSH model struggle to generalize under the
disorder. The authors solved this problem by using a domain adversarial neural
network. Here, we do not aim to improve this solution. Instead, we want to
understand the reason for the failures of standard CNNs and propose tools that
can inform the user whether to expect an OOD generalization from a trained CNN
or not without labeled data.

This paper is structured as follows. We start by describing the data set and
the learning task of a CNN in sections II.1-II.2. In section II.3, we discuss
an interpretability technique called class activation mapping (CAM) [67],
which provides explanations of CNN predictions as sketched in figure 1(b). To
study the data representation learned by a CNN, we need a dimensionality
reduction technique such as principal component analysis (PCA), presented
schematically in figure 1(c), which we describe in section II.4. We present
and discuss our results in section III and conclude in section IV.

Figure 1: Unsupervised study of out-of-distribution generalization with class
activation mapping and principal component analysis. (a) We train multiple
randomly initialized CNNs to predict phases in the SSH model without the
disorder (simple regime) and study their OOD generalization to phases in the
SSH model with disorder (complex regime). Only a handful of trained CNNs learn
meaningful features (instead of spurious correlations) that allow for
successful OOD generalization. (b) We analyze what data features are the most
important for predictions of well- and poorly-generalizing CNNs with class
activation mapping (CAM) – an interpretability technique producing heatmaps
that after overlaying on input samples indicate the most important data
elements for the prediction. (c) With principal component analysis (PCA), we
study the similarity of the representation learned by poorly- and well-
generalizing CNNs for the data from the simple and complex regime. Panels (b)
and (c) are adapted from Refs. [68, 69], respectively.

##  II Methods

###  II.1 The model Hamiltonian

The model proposed by Su, Schrieffer, and Heeger (SSH) [70] describes spinless
fermions on a one-dimensional (1D) chain in a tight-binding approximation,
with staggered nearest-neighbor tunneling amplitudes. It is described by the
Hamiltonian

$\hat{H}_{0}=\textit{v}\sum_{n=1}^{N}\hat{c}^{\dagger}_{n}\hat{\sigma}_{x}\hat{c}_{n}+\textit{w}\sum_{n=1}^{N-1}\quantity(\frac{1}{2}\hat{c}^{\dagger}_{n}\quantity(\hat{\sigma}_{x}+i\hat{\sigma}_{y})\hat{c}_{n+1}+\text{h.c.}).$
(1)

The SSH Hamiltonian can have alternative definitions depending on the
conventions [71, 72, 73] (see appendix A for more details). We consider a
chain of $N$ unit cells. Within each unit cell, there are two sites that
belong to sublattice $A$ and $B$, respectively. The operator
$\hat{c}_{i}^{(\dagger)}=\quantity(\hat{c}_{A,i}^{(\dagger)},\hat{c}_{B,i}^{(\dagger)})$
denotes the annihilation(creation) operators on the respective sublattices in
the system. $\sigma_{i}$ denotes the Pauli matrices. The tunneling amplitudes
are different for intercell tunneling (between neighboring cells) – w and for
intracell tunneling (within a single unit cell) – v. For the remainder of this
work, we fix the intercell tunneling to $\textit{w}=1$.

Such a system exhibits two distinct phases: topological and trivial. They are
characterized by a different value of a topological invariant, called the
winding number, which arises from the chiral symmetry present in the system.
For periodic boundary conditions (PBC), the invariant is winding number
$\vartheta$. For open boundary conditions (OBC), the topological phase is
characterized by the real space winding number and by the presence of zero-
energy eigenstates, corresponding to the half-filling of the system [72].
These states are exponentially localized at the edges of the system. For this
reason, they are referred to as edge states or edge modes. The number of such
zero-energy edge states on one edge is equal the winding number of the bulk.
It can be shown by bulk-boundary correspondence [70], that both formulations
are equivalent; therefore, in this work we use both approaches.

The interplay between the topological invariants of the system and the
presence of the disorder can be studied with the following Hamiltonian [71],

$\hat{H}_{1}=\sum_{n=1}^{N-1}t_{n}\quantity(\frac{1}{2}\hat{c}_{n}^{\dagger}\left(\hat{\sigma}_{x}+i\hat{\sigma}_{y}\right)\hat{c}_{n+1}+\text{
h.c. })+\sum_{n=1}^{N}m_{n}\hat{c}_{n}^{\dagger}\hat{\sigma}_{x}\hat{c}_{n}.$
(2)

Here, the tunneling amplitudes $t_{n}$ and $m_{n}$ vary from site to site.
They include the added disorder in the following way

$t_{n}=\textit{w}+2W\omega_{n},\quad m_{n}=\textit{v}+W\omega^{\prime}_{n}.$
(3)

The parameter $W$ is the strength of the disorder, and $\omega_{n}$ is a
random variable drawn from the uniform distribution on the interval
$[-0.5,0.5]$. The topological invariants, to some extent, are robust to
disorder. However, for a strong enough value of the disorder, they are likely
to change. Therefore, to account for disorder in winding number calculation,
we use a handy approximation based on infinite system winding number
calculation [74] (see appendix B for details). The relationship between
winding number $\vartheta$ and disorder strength $W/\textit{w}$ for systems of
the symmetry class such as equation (2) has been thoroughly studied in Ref.
[71], where they proposed a phase diagram for this scenario. The phase diagram
we calculate using the infinite system winding number approximation is in good
agreement with their results.

###  II.2 Learning task and data set

In this work, we task a convolutional neural network (CNN) with learning a
mapping between a full set of eigenstates of the SSH Hamiltonian from equation
(1) and the corresponding winding number. We describe the CNN architecture and
training hyperparameters in detail in appendix C. To obtain the input data, we
compute the eigenstates of equation (1) using exact diagonalization [75] with
OBC. The corresponding winding numbers are computed by taking the same
Hamiltonian with the imposed PBC. This is an approximate calculation, which
translates into calculating the winding number of an infinite system [74].

The input data are matrices whose columns are squared moduli of the
eigenstates of the system, sorted by energy. We present exemplary input data
samples in figure 1(a). The labels are corresponding winding numbers. Because
of the presence of chiral symmetry, the energy spectrum is symmetric, and the
zero-energy edge states correspond to the two middle columns of an input data
sample. The amplitude of the edge states peaks at the edges and decays
exponentially into the bulk; therefore, we expect the edge pixels of the
middle columns to be the most indicative of these states in the topological
phase.

We want to understand the general behavior of NNs, so we study multiple
instances of CNNs obtained by training them using the same hyperparameters but
from different random initializations, following the default uniform
initialization in PyTorch [76]. We train and test our CNNs on data from a
system without added disorder, described by equation (1). We train them on
data sampled in equal numbers from both topological and trivial phases, far
from the phase transition. The loss function we optimize in this task is
defined by the binary cross-entropy (BCE) function [77]. We test CNNs on data
sampled uniformly over a whole range of $\textit{v}/\textit{w}$, including the
close vicinity of the phase transition. The $\textit{v}/\textit{w}$ values
used to generate data points in the three data sets (training, validation, and
test) were different to prevent possible leakage of information. Such choice
of testing points allows us to assess the networks’ generalization, that is,
its performance on previously unseen examples that are sampled from the same
underlying distribution of the data. Our understanding is that data that are
drawn from the same distribution share the same structure and set of features.

A much more challenging task for NNs than the in-distribution generalization
is to achieve out-of-distribution (OOD) generalization, that is, to perform
well on data that come from a different distribution than the training data.
We expect ODD generalization when OOD data have a similar structure and set of
meaningful features as training data. In our case, these OOD data are data
samples constructed from eigenstates of the disordered SSH model described by
equation (2). Many new features are added to the OOD data that arise from the
disorder-induced Anderson localization [78]. This results in more highly
localized eigenstates, which render the edge states less distinguishable with
increasing disorder strength. Especially dangerous to the OOD generalization
are spurious correlations, which can be present in the training data. They
refer to a situation where some features of the input data and the label
appear to be related to each other, but the relationship is coincidental or
confounded by an external variable [79].

To evaluate the network OOD generalization, we check its ability to recreate
the phase diagram faithfully. A viable error metric is the root mean square
error (RMSE). In our case, it is computed as the square root mean of an
element-wise square difference between the target ($y$) and the predicted
($\hat{y}$) winding numbers, for $N_{\mathrm{s}}^{\textit{v}}$ values of
intracell tunneling $\textit{v}/\textit{w}$, $N_{\mathrm{s}}^{W}$ values of
the disorder amplitude $W/\textit{w}$, and $N_{\mathrm{r}}$ realizations of
each disorder amplitude $W/\textit{w}$. The number of entries in the $y$ and
$\hat{y}$ arrays is therefore,
$N_{\mathrm{el}}\,=\,N_{\mathrm{s}}^{W}\,\cdot\,N_{\mathrm{s}}^{\textit{v}}\,\cdot\,N_{\mathrm{r}}$,
and the RMSE formula reads

$\mathrm{RMSE}(y,\hat{y})=\sqrt{\frac{1}{N_{\mathrm{el}}}\sum_{i=1}^{N_{\mathrm{el}}}\quantity(y_{i}-\hat{y}_{i})^{2}}\,.$
(4)

Another metric we use is OOD accuracy, defined as the percentage of correct
predictions out of $N_{\mathrm{el}}$ in the winding number prediction task.
Note that the network can output only binary predictions, however, when we
plot phase diagrams predicted by networks there are non-binary entries coming
from averaging over $N_{\mathrm{r}}$ realizations of each disorder amplitude
$W/\textit{w}$.

###  II.3 Class activation mapping (CAM)

Machine learning methods are usually black-box tools. They accomplish the
tasks that we give them at the cost of not being able to justify the outcome
they provide. This shortcoming is known as the lack of interpretability.
Interpretability can be understood as “the degree to which a human can
consistently predict the model’s result” [80]. There are numerous
interpretability techniques that aim at a better understanding of the network
reasoning. We recommend their overview in Ref. [81]. In this work, we apply
class activation mapping (CAM), a simple pixel attribution technique. CAM,
given an input sample, produces a map highlighting the areas of the input that
contribute the most to the prediction of a considered class.

CAM is an attribution technique tailored for use with CNNs and leverages their
design. A CNN produces different representations of the input data during a
forward pass through the network and encodes them in different channels. CAM
relies on the latent representation of the data present in the output channels
of the last convolutional layer in a network, visualized in figure 1(b) as
colored activation maps, and performs their weighted sum with weights
$\alpha_{k}$ [82]. The resulting map, after rescaling back to the original
data size, highlights the areas that were the most influential in predicting a
considered class. The crucial part is the choice of the weights $\alpha_{k}$.
The original formulation of the method relies on reducing each channel output
of the last convolutional layer to a single number with global average pooling
(GAP) layer [67]. The weights of fully-connected layer connecting those
numbers with the output corresponding to the considered class $C$ are then
used as $\alpha_{k}$ for the weighted average of convolved images (see
appendix D for more details). An obvious limitation of this approach is the
necessity for the NN architecture to include the GAP layer, which can limit
the applicability of CAM to existing trained models.

This technique can be made architecture-agnostic [83, 84, 85, 86, 87, 88]. The
simplest extension is Grad-CAM [84], which replaces the need for the GAP layer
with gradients computed by backpropagation of the class output to the last
convolutional layer. We applied both techniques in our analysis, but they gave
quantitatively the same results, which is why in the discussion we only
present the results obtained with CAM.

###  II.4 Dimensionality reduction

The information learned by an NN is stored throughout all its parameters.
Analyzing how data is represented in the parameter space could help
researchers extract this knowledge. However, even for small NNs, the
dimensionality of parameter space is of the order of thousands, and humans
have difficulty comprehending data in high dimensions. Thus, reducing data to
a small number of dimensions is helpful for visualization purposes and allows
us to gain precious insight into NNs’ inner workings [89, 90, 91, 92]. An
example of such insight might be to extract which areas in latent space
correspond to given learned concepts and, conversely, the relations and
distance between them the network has learned [93]. To take advantage of this
paradigm in our work, we use principal component analysis (PCA) - an
established linear dimensionality reduction technique.

PCA [94] relies on computing the principal components (PCs) of the data that
are first stacked to form a multidimensional tensor. The PCs are eigenvectors
of the covariance matrix of this tensor, represented schematically as orange
arrows in figure 1(c). The magnitude of their corresponding eigenvalues orders
them, so the first PC represents the direction of the highest data variance,
the second PC is orthogonal to it and describes the direction with the second
highest variance, etc. The original tensor is then projected into a subspace
spanned by the selected number of its PCs. The result is a tangible low-
dimensional representation of the data that presents the maximal variance of
the original high-dimensional space. The low-dimensional representation is
needed as a preprocessing step for some ML algorithms or visualization
purposes. Because of the linearity of this reduction technique, the distance
between the points and the density is primarily preserved. It can be treated
as representative of the distance in the original high-dimensional space.

##  III Results and discussion

###  III.1 Convolutional neural networks fail to generalize to data with
disorder

Convolutional neural networks succeeded in the training task posed by the
disorderless SSH system. The training of all CNN instances converged, and the
networks achieved perfect (100%) accuracy on both the training and validation
data sets. The performance was also very good for all the trained instances
when tasked with making predictions for in-distribution data. The accuracy
score for the in-distribution testing was greater than 95%. The only regions
where some test samples were misclassified were for data in the vicinity of
the phase transition, which were excluded from the training set.

Despite promising results within the training distribution, OOD generalization
proved to be a challenging task, as we show in table 1. We quantify the OOD
generalization by testing the trained networks on the phase diagram of the SSH
model, generated for 25 disorder realizations, and presented in the bottom-
left corner of figure 2. Upon many initializations, we have found that only as
little as $5\%$ of them manage to reproduce the shape of the phase diagram
correctly. We used RMSE as a metric of distinction between the well- and
poorly-generalizing models, with the threshold set at 0.2. By dividing these
models, we can also observe their OOD accuracy. The well-generalizing networks
achieve an OOD accuracy of 84% and more on the data from the whole phase
diagram. In contrast, a large majority (95%) of CNNs have OOD accuracy only
marginally higher than a random guess, as seen in the third row of table 1.
Interestingly, the well-generalizing networks tend to predict the phase
transition at values below $\textit{v}/\textit{w}=1$, closer to
$\textit{v}/\textit{w}\approx 0.9-0.95$, as visible in figures 2(h) and (j).
They may take into account the finite-size effects that the infinite-system
approximation of the winding number fails to account for.

We can improve this poor statistic by introducing slightly disordered
($W/\textit{w}\in\quantity{0.01,0.05}$) data to the training data set. We
treat this range of small disorders as a ’perturbative’ regime in which we can
label the slightly disordered data with winding numbers calculated for
respective disorderless points of the phase diagram. As a result, the ratio of
successful CNNs increases from $5\%$ to $22\%$. Other statistics remain
qualitatively the same in both well- and poorly-generating models. Performance
comparison of 100 randomly initialized CNNs trained only on disorderless data
and CNNs trained also with slightly disordered data is in table 1.

The networks exhibit this behavior despite showing stellar performance both in
the training regime and during the in-domain testing. Most CNNs fail even
though we provide them with all the necessary information to carry this
performance to a disordered regime, such as the disorder-robust connection
between zero-energy edge states and winding number. Moreover, the physically
motivated remedy only slightly improved the number of well-generalizing
networks. To understand these phenomena, we must lift the lid of the ’black
box’ and ask why CNNs do not generalize well.

Table 1: Statistics (means $\pm$ standard deviations) of 100 randomly initialized networks trained only on data without disorder, $W=0$, and on data including also slightly disordered samples, $W/\textit{w}\in\quantity{0,0.01,0.05}$. | CNNs trained on $W=0$ |  |  CNNs trained on $W/\textit{w}\in\quantity{0,0.01,0.05}$ |
---|---|---|---|---
OOD generalization | Well-generalizing | Poorly-generalizing |  Well-generalizing | Poorly-generalizing
Number of CNNs | 5 / 100 | 95 / 100 |  22 / 100 | 78 / 100
Training accuracy | $\sim 100\,\%$ | $\sim 100\,\%$ |  $\sim 100\,\%$ | $\sim 100\,\%$
Test accuracy | $95.5\pm 1.5\,\%$ | $97\pm 2\,\%$ |  $94.7\pm 1.9\,\%$ | $97\pm 2\,\%$
OOD accuracy | $84\pm 3\,\%$ | $55\pm 8\,\%$ |  $86\pm 3\,\%$ | $61\pm 9\,\%$
RMSE | $0.168\pm 0.025$ | $0.46\pm 0.07$ |  $0.153\pm 0.031$ | $0.41\pm 0.10$
Figure 2: Comparison between a well- and poorly-generalizing CNN through the
lens of CAM. First column: the expected CAM result for a CNN that focuses on
the edge states, obtained via a toy model in appendix E (first row), and the
target phase diagram (second row). (a) CAM-based explanation of the prediction
of one of many poorly-generalizing CNNs (here, the OOD accuracy of $50.3\%$)
for an input sample from the disorderless SSH model at
$\textit{v}/\textit{w}=0.27$. CNN places more importance on states around the
one- and three-fourths of the spectrum. (b) One of the well-generalizing CNNs
(OOD accuracy of $90.1\%$) in the same input sample pays more attention to the
localization of the edge states. However, (c) poorly-generalizing CNN (OOD
accuracy of $52.3\%$) can also pay attention to edge states, and (d) well-
generalizing CNN (OOD accuracy of $91.7\%$) can ignore edge states, rendering
the CAM analysis inconclusive. Panels (e)-(h) present the predicted phase
diagram of the corresponding CNNs. The phase diagrams are predicted winding
numbers $\vartheta$ averaged over $N_{\mathrm{r}}=25$ realizations of
disorder, with $N_{\mathrm{s}}^{\textit{v}}=200$ slices in
$\textit{v}/\textit{w}$, and $N_{\mathrm{s}}^{W}=500$ slices in $W/\textit{w}$
directions.

###  III.2 Analysis of the CNNs generalization’s failure

Most successfully trained CNNs failed to generalize from the simple training
regime to the disordered regime. This is especially surprising because there
is a robust indicator of the topological phase, namely the presence of edge
states. To understand what other features of the input data are instead
leveraged by networks to make their predictions, we use class activation
mapping (CAM) [67], described in section II.3.

A large majority of trained networks tend to ignore edge states in the middle
of the spectrum as well as the extremes of the spectrum, that is, the ground
state and the highest excited state (maximally filled state). Instead, they
look at the remaining states of the system. We see an example of such a
typical network in figure 2(a). Networks that ignore edge states in the
disorderless SSH model usually fail to generalize to the system with the
disorder, as seen in the corresponding predicted phase diagram in figure 2(e).
In contrast, a small number of networks focus more on the edge states in the
middle of the spectrum, corresponding to the half-filled system and
constituting the topological invariant of the system. An example of such a
network is shown in figure 2(b), which pays closer attention to the
localization at one of the system edges. This group of networks is more likely
to generalize well to the disordered data, see the predicted phase diagram in
figure 2(f), as it detects something related to the known topological
invariant. This analysis shows that we can increase our trust in the OOD
generalization of the network by making sure it looks at relevant features in
the known regime of the problem.

The analysis so far showed that networks tend to ignore edge states, but why
is this the case? We hypothesize that features must exist in the bulk that
allow a network to solve the learning task in the simple regime, i.e., that
correlate well with the label in the simple regime. Prompted by this analysis,
we compare each eigenstate of the SSH model for intracell tunneling
$\textit{v}=0$ to its respective eigenstate for different intracell tunneling
values. We plot the described fidelity of the pair of eigenstates in figure
3(a). As expected, we see that the fidelity of the edge states at
$\textit{v}=0$ and their counterparts for different $\textit{v}/\textit{w}$
changes rapidly throughout the phase diagram. However, we observe that the
fidelity of the bulk states also changes across $\textit{v}/\textit{w}$ and,
as such, can be used by a CNN to predict a label in the system without
disorder. It is only an example of a possible feature, and there can be any
number of more complex ones that the networks pick up from the data.
Importantly, they no longer correlate with the label in the disordered system,
as seen in figure 3(b). For this reason, we consider them to be spurious
correlations in the task of distinguishing between the topological and
topologically trivial phases.

Apparently, in this task, CNNs tend to learn some combination of features
related to numerous non-edge states, even if they are weakly correlated with
the label when taken separately. This observation echoes the results of the ML
community on the inherent trade-off of classifiers between the accuracy and
robustness. Tsipras et al. [95] showed rigorously on a simple example that
classifiers learn a combination of weakly correlated features to achieve
perfect accuracy, instead of relying on a single strongly correlated feature
that is also present in the data but does not allow for perfect accuracy. When
disorder enters the data, it can easily distort the combination of weakly
correlated features, harming robustness of a classifier. A solution to
achieving robust neural networks is adversarial training [96], which is
reminiscent of the original solution from Ref. [32], which used a domain
adversarial neural network. In our data distribution, though, the single
feature (edge states) should be perfectly correlated with the label (up to
finite-size effects), so the task of maximizing accuracy should not lead to
ignoring this feature. We do not know what causes CNNs to favor the
combination of weak correlators over the single predictive feature, but we
have two hypotheses. First, the preference towards multiple features may be
due to regularization that prevents NNs from relying on single features.
Second, it may come from the iterative nature of the training, which causes
multiple features with weak correlation to have an increasingly strong
training signal as they boost each other.

Figure 3: Fidelity between the eigenstate $e_{i}$ for $\textit{v}=0$ and the
respective eigenstate $e_{i}$ for different values $\textit{v}/\textit{w}$,
showcasing an example of correlations in data coming from the SSH model that
can be used by an NN to predict a label in a simple regime. (a) Apart from the
rapidly changing fidelity expected of the half-filled edge state, we can also
observe the changing fidelity of eigenstates closer to the extremes of the
spectrum. (b) Already in the weakly disordered case ($W/\textit{w}=0.15$)
fidelity of the states closer to the extremes of the spectrum becomes a poor
predictor of the phases. Only the disorder-robust fidelity of edge states
remains a valid feature. Note: The drop in fidelity of edge states around
$\textit{v}/\textit{w}=0.08$ in panel (a) and rapid oscillations for small
$\textit{v}/\textit{w}$ in panel (b) come from the hybrydization of the edge
states due to finite-size effects [70].

###  III.3 Unsupervised analysis of OOD generalizations of NNs

In the previous section, we have described how CAM guided us into
understanding that the majority of CNNs learn some spurious combination of
weakly correlated features (related to non-edge states) instead of a single
strong feature (related to edge states). An appropriate combination of weakly
correlated features allows for perfect accuracy in the simple regime but
ceases to correlate with the label in the disordered regime. The same CAM
analysis can be used to assess the OOD generalization of CNNs without access
to the labels. For example, if a CNN focuses on the edge state, this increases
the chance of a better OOD generalization of the model.

Although, overall, CAM is useful and can be used as the first step of the
generalization study, we have found that the CAM analysis is unfortunately
inconclusive and can only indicate CNN OOD generalization tendencies. In
particular, not every CNN that pays attention to the edge states exhibits good
OOD generalization – evidence for this is in panels (c) and (g) of figure 2.
We even find an opposite example where the CNN that does not put special focus
on edge states achieves great OOD accuracy, as seen in panels (d) and (h) of
figure 2. We share the CAM results for all 100 trained CNNs for all test data
online [97] for the reader interested in various possible behaviors of the
discussed CNNs. The reliability of CAM as the method to assess the
generalization could be improved if used on the test data from the disordered
SSH model. However, this would bring this explanation method into a noisy
regime where CAM is known to be fragile and unreliable. We advise the reader
against using this explanation method in the presence of noise and elaborate
on this topic in appendix D.

As a second technique to validate the OOD generalization of a network, we
propose to study the structure of data representation learned by trained CNNs.
When an NN processes an input sample, it generates different activations at
every network layer. We want to understand the relation between the
activations generated by data without the disorder and those generated by the
disordered data. We follow here the logic that data the network views as
similar should generate similar activations or, in other words, should have
similar learned representations. If an NN sees any meaningful relation between
the training data without the disorder and the OOD test data with the
disorder, their representations should be somewhat similar. Such a result
increases our trust in the OOD prediction made by a network.

The space of CNN activations is high-dimensional. In the case of a fully
connected layer, the dimension depends on the number of neurons. In the case
of a convolutional layer, the dimension is equal to the size of the convolved
data point. To study the relation between points in such a high-dimensional
space, we apply dimensionality reduction techniques that can bring the space
dimension down to, e.g., two while preserving some structure of the original
data. We use here principal component analysis (PCA), explained in section
II.4. To verify the results, we also used uniform manifold approximation and
projection (UMAP), which is a non-linear dimensionality reduction technique.
We discuss obtained results in appendix F.

Figure 4: Two-dimensional representation of the data learned by the
penultimate layer of a poorly- and well-generalizing CNN with inconclusive CAM
results from figure 2, obtained with PCA. (a) The well-generalizing CNN, which
according to CAM ignores the edge states, learned the representation in which
the disorderless data (first column) are clustered in two well-separated
clusters corresponding to two phases. Data with increasing disorder falls
firstly in the same two clusters formed by disorderless data, and then slowly
separates from them but keeps a weak connection. (b) The representation of
test data learned by the poorly-generalizing CNN, which according to CAM pays
close attention to the edge states, already for disorderless data (first
column) lacks well-defined two clusters. The data with disorder, forms more
and more separated third cluster, only weakly connected to one of the clusters
of disorderless data. Insets represent labels obtained with the infinite-
system approximation of the winding number.

We study the representation of the data learned by the penultimate layer of
CNN, that is, the neuron activations after applying the GAP and before
entering the last fully connected layer, with the softmax function, which
plays the role of a classifier. To this end, in figure 4 we present the
clustering obtained with PCA for data with different disorder strengths in the
two-dimensional versions of the high-dimensional representation space of the
penultimate CNN layer. The dimensionality reduction is applied to the
activations generated by all test data simultaneously, but we plot data
subsets for different $W/\textit{w}$’s in separate subplots to make the
analysis clearer. We exclude test data without disorder coming from the
vicinity of the transition ($\textit{v}/\textit{w}\in[0.9;1.1]$) to further
simplify the analysis.

Let us first focus on the data representation learned by the penultimate layer
of a well-generalizing CNN in figure 4(a). The first thing to notice is the
representation of the disorderless data (first column) that relies on two
well-separated clusters corresponding to two phases (light orange is trivial,
light blue is topological). In the next columns, we plot both the disorderless
data and the data with increasing disorder strength (indicated with darker
orange and blue). For small and intermediate disorder ($W/\textit{w}=0.15$ and
$0.5$), the disordered data are represented very similarly to the data without
disorder, in other words, their representation overlaps. With increasing
disorder ($W/\textit{w}=2$ and more), the data start to disconnect from the
original clusters. Interestingly, the topological data with disorder separate
faster from the disorderless data than the data from the topologically trivial
phase. Finally, only for large disorder the data form a separate third
cluster. This means that the network has a disjointed representation only of
strongly disordered data.

In contrast, the data representation learned by poorly-generalizing CNNs
already at the level of disorderless data do not form two well-defined
clusters corresponding to two phases. Especially the topological data without
disorder form quite a disconnected cluster, as seen with PCA in figure 4(b).
The data with disorder of $W/\textit{w}=0.5$ (third column) is already
separate from the original clusters. For strong disorders, data form three or
four separate clusters in the representation space of the penultimate layer.
This suggests that the network does not see continuity in the data as a
function of the disorder strength.

Notably, the two CNNs whose learned data representations we studied above are
the same CNNs whose CAM analysis rendered inconclusive results. That is, the
well-generalizing CNN is the one from figure 2(d),(h) that focuses less on
edge states. The poorly-generalizing CNN is the one from figure 2(c),(g) that
focuses prominently on edge states. This shows that data representation
analysis together with CAM can make stronger statements about the OOD
generalization of trained networks.

Finally, we make an analogous analysis for data representation across layers
of the well- and poorly-generalizing CNNs in appendix F, this time with UMAP.
It renders the same conclusions as PCA. However, we make an additional
interesting observation that UMAP performed on the input data with and without
disorder does better job in forming clusters corresponding to topological and
trivial phases than a majority of CNNs trained in a supervised way on data
without disorder.

##  IV Conclusion and outlook

In this work, we have presented how to increase the trust in predictions of a
neural network (NN) when the predictions are made on data from a different
distribution than the NN’s training data distribution and without access to
ground-truth labels. Such an out-of-distribution (OOD) generalization is a
desired property of a robust and reliable network and is difficult to achieve
or validate. We have shown how one can study the network to understand the way
it processes the data with two different tools. The first is an
interpretability technique that highlights parts of the data that were
important to the network classification decision, called class activation
mapping (CAM). The second is an analysis of the data representation learned by
the network, facilitated by the dimensionality reduction techniques, such as
principal component analysis (PCA), applied to the representation space of the
network. The data we have tackled are eigenstates of the Su-Schrieffer-Heeger
(SSH) Hamiltonian, coming from two regimes: with and without disorder. We
trained hundreds of convolutional networks (CNNs) on the data without disorder
and checked their in-distribution and out-of-distribution generalization to
data without and with disorder, respectively. We have made the following
observations.

  * •

An overwhelming majority ($95\%$) of successfully trained CNNs failed to
generalize to data with disorder. Expanding the training set with data with
slight disorder ($W/\textit{w}\leq 0.05$) improved these statistics only to an
extent ($78\%$).

  * •

The reason CNNs tend to fail is that they focus on non-generalizable features
of data that are nevertheless useful for the training task, as indicated by
the CAM analysis of CNNs’ predictions. For example, eigenvectors other than
the edge states carry correlations that can be leveraged to classify phases in
the disorderless data regime, but they disappear when disorder is added.

  * •

CAM analysis can be used to assert our trust in network prediction. If in the
disorderless regime a network pays attention to edge states, which are known
to be useful features, such a network tends to exhibit a better OOD
generalization. Surprisingly, such networks can still fail to generalize to
disordered data, making the CAM analysis inconclusive.

  * •

Moreover, CAM itself is also a fragile interpretation technique that performs
poorly in the presence of noise, which prevented its use to understand the
network predictions in the disordered regime.

  * •

Dimensionality reduction techniques such as PCA can be used to visualize the
data representation learned by networks. If a CNN has a disconnected (very
different) representation of disordered data with no connection to the
disorderless training data, the trust in its predictions on the disordered
data should be limited. A good predictor of the OOD generalization quality is
instead when a CNN represents the data with slight disorder similarly to its
disorderless training data and then the representation gets smoothly
disconnected with increasing disorder strength.

We conclude that, together, CAM and data representation analysis serve as
useful tools to gain additional insight and assess trust in NN predictions.
Given their low cost, we believe that their routine use can only benefit
members of the scientific community who have already added deep learning to
their computational toolbox. Such an analysis of NNs is especially needed in
light of our surprising observations that CNNs fail in a relatively simple
task that is generalizing to slight disorder when trained on the disorderless
SSH model, whereas physicists know that there exists a robust feature to solve
this task, such as the presence of edge states. This observation also
highlights that scientific data sets create new challenges for deep learning,
which can unravel unexpected behaviors and failure modes of NNs. At the same
time, the community needs to develop more robust tools to assess NN
performance in regimes without known ground truths, bearing in mind that some
of them fail in the presence of random or adversarial noise.

### Data availability

The source code and data that support the findings of this study is openly
available at
https://github.com/kcybinski/Interpreting_NNs_for_topological_phases_of_matter
[97].

###### Acknowledgements.

K.C. acknowledges the financial support from the Polish Ministry of Science
and Higher Education within the “Excellence initiative – research university”
program. M.T. acknowledge the National Science Centre Poland (grant no.
2020/38/E/ST2/00564) for the financial support and the Poland’s high-
performance computing infrastructure PLGrid (HPC Centers: ACK Cyfronet AGH)
for providing computer facilities and support (computational grant no.
PLG/2023/016878). ICFO group acknowledges support from European Research
Council AdG NOQIA; MCIN/AEI (PGC2018-0910.13039 /501100011033,
CEX2019-000910-S/10.13039/501100011033, Plan National FIDEUA
PID2019-106901GB-I00, Plan National STAMEENA PID2022-139099NB, I00, project
funded by MCIN/AEI/10.13039/501100011033 and by the “European Union
NextGenerationEU/PRTR" (PRTR-C17.I1), FPI); QUANTERA MAQS PCI2019-111828-2);
QUANTERA DYNAMITE PCI2022-132919, QuantERA II Programme co-funded by European
Union’s Horizon 2020 program under Grant Agreement No 101017733); Ministry for
Digital Transformation and of Civil Service of the Spanish Government through
the QUANTUM ENIA project call - Quantum Spain project, and by the European
Union through the Recovery, Transformation and Resilience Plan -
NextGenerationEU within the framework of the Digital Spain 2026 Agenda;
Fundació Cellex; Fundació Mir-Puig; Generalitat de Catalunya (European Social
Fund FEDER and CERCA program, AGAUR Grant No. 2021 SGR 01452, QuantumCAT
U16-011424, co-funded by ERDF Operational Program of Catalonia 2014-2020);
Barcelona Supercomputing Center MareNostrum (FI-2023-3-0024); Funded by the
European Union. Views and opinions expressed are however those of the
author(s) only and do not necessarily reflect those of the European Union,
European Commission, European Climate, Infrastructure and Environment
Executive Agency (CINEA), or any other granting authority. Neither the
European Union nor any granting authority can be held responsible for them
(HORIZON-CL4-2022-QUANTUM-02-SGA PASQuanS2.1, 101113690, EU Horizon 2020 FET-
OPEN OPTOlogic, Grant No 899794), EU Horizon Europe Program (This project has
received funding from the European Union’s Horizon Europe research and
innovation program under grant agreement No 101080086 NeQSTGrant Agreement
101080086 — NeQST); ICFO Internal “QuantumGaudi” project; European Union’s
Horizon 2020 program under the Marie Sklodowska-Curie grant agreement No
847648; “La Caixa” Junior Leaders fellowships, La Caixa” Foundation (ID
100010434): CF/BQ/PR23/11980043. Al.D. acknowledges the financial support from
a fellowship granted by la Caixa Foundation (ID 100010434, fellowship code
LCF/BQ/PR20/11770012). An.D. acknowledges the financial support from the
Foundation for Polish Science. The Flatiron Institute is a division of the
Simons Foundation.

##  Appendix A Alternate Su-Schrieffer-Heeger model formulations

The basic formulations of the SSH model [98] we consider is discussed by
Asboth et al. [70]:

$\hat{H}=\textit{v}\sum_{m=1}^{N}\outerproduct{m}{m}\otimes\hat{\sigma}_{x}+\textit{w}\sum_{m=1}^{N-1}\quantity(\outerproduct{m+1}{m}\otimes\frac{\hat{\sigma}_{x}+i\hat{\sigma}_{y}}{2}+\text{h.c.})\,.$
(5)

In
$\hat{c}_{i}^{(\dagger)}=\quantity(\hat{c}_{A,i}^{(\dagger)},\hat{c}_{B,i}^{(\dagger)})$
operators formulation it is,

$\hat{H}=\textit{v}\sum_{n=1}^{N}\hat{c}^{\dagger}_{n}\hat{\sigma}_{x}\hat{c}_{n}+\textit{w}\sum_{n=1}^{N-1}\quantity(\hat{c}^{\dagger}_{n}\frac{\hat{\sigma}_{x}+i\hat{\sigma}_{y}}{2}\hat{c}_{n+1}+\text{h.c.})\,.$
(6)

Other works we used for comparison of our results formulate it differently:

  1. 1.

Mondragon et al. [71]:

$\hat{H}_{SSH}=\sum_{n}\quantity{\textit{m}_{n}\hat{c}^{\dagger}_{n}\hat{\sigma}_{y}\hat{c}_{n}+\textit{t}_{n}\quantity(\hat{c}^{\dagger}_{n}\frac{\hat{\sigma}_{x}+i\hat{\sigma}_{y}}{2}\hat{c}_{n+1}+\text{h.c.})},$
(7)

  2. 2.

Meier et al. [72]:

$\hat{H}_{SSH}=\sum_{n}\quantity{\textit{m}_{n}\hat{c}^{\dagger}_{n}\hat{\sigma}_{x}\hat{c}_{n}+\textit{t}_{n}\quantity(\hat{c}^{\dagger}_{n+1}\frac{\hat{\sigma}_{x}-i\hat{\sigma}_{y}}{2}\hat{c}_{n}+\text{h.c.})},$
(8)

  3. 3.

Le et al. [73]:

$\hat{H}_{SSH}=\sum_{j,\sigma={A,B}}\quantity{-J\quantity[1+(-1)^{j}\Delta
t]\,\hat{c}^{\dagger}_{j+1,\sigma}\hat{c}_{j,\sigma}+\text{h.c.}}.$ (9)

Equations drawn from Refs. [71] and [72], allow for the presence of disorder
in the system through $t_{n}$ and $m_{n}$ tunneling amplitudes, corresponding
to the same inter- or intra- cell tunneling as w and v. The correspondence is:

  * •

intercell tunneling $\textit{w}\to\,\textit{t}_{n}$,

  * •

intracell tunneling $\textit{v}\to\,\textit{m}_{n}$.

The sign discrepancy between models from Ref. [71] and Ref. [72] in the
intercell part is due to the different site numeration. The former uses
$BABABA\dots$ convention, and the latter uses $ABABAB\dots$ convention, which
is the one we use. The difference is seen in figure 5. Another difference we
note is that in Ref. [71], the intracell tunneling amplitudes are purely
imaginary $\textit{m}_{n}$, while in Ref. [72], they are purely real
$\textit{m}_{n}$. Both yield the same results as long as one convention is
kept. In this work, we opt for real tunneling amplitudes.

Figure 5: Difference between SSH formulations from Refs. [71] and [72] is due
to different sublattice order, as seen in panels (a) and (b), respectively.
Both define distance within unit cell $d_{1}=0$, and distance between unit
cells as $d_{2}=1$. Clarification of this choice is needed for the further
transition to momentum space, as the ordering of A and B sites in each unit
cell is swapped.

##  Appendix B Calculation of winding number

Calculation demonstration. The routine we employ for the calculation of the
winding number [74] involves several steps. This section is to serve as a
step-by-step demonstration. It is accompanied by a mirror Jupyter Notebook in
our GitHub repository [97].

Demonstration parameters. In order to keep the calculations tractable, we
choose system size to be 6 sites (3 unit cells). We set the boundary
conditions to periodic. We generate the occupational basis, and sort it
lexicographically.

Winding number formula We want to use here the equation for the winding number
in the form [74]:

$\vartheta=\frac{1}{2\pi i}\oint_{BZ}\mathrm{Tr}(h^{-1}\partial_{k}h)$ (10)

It is defined as a $1^{\text{st}}$ Brillouin zone integral of a transformed
momentum space Hamiltonian. In the following sections we will derive it,
starting from Hamiltonian definition presented in equation (1).

Occupational basis Hamiltonian. First we define the Hamiltonian in the
occupational basis according to equation (1). In matrix form it is,

$H_{\mathrm{occ}}=\matrixquantity[0&\textit{v}&0&0&0&\textit{w}\\\
\textit{v}&0&\textit{w}&0&0&0\\\ 0&\textit{w}&0&\textit{v}&0&0\\\
0&0&\textit{v}&0&\textit{w}&0\\\ 0&0&0&\textit{w}&0&\textit{v}\\\
\textit{w}&0&0&0&\textit{v}&0].$ (11)

Momentum basis Hamiltonian. The winding number is defined as a $1^{\text{st}}$
Brillouin zone integral, so we must transfer the Hamiltonian to momentum
space. The condition to be met here is that a fermion only accumulates phase
when crossing between the neighboring Brillouin zones. Given our choice of
boundary conditions, this is achieved by addition of $e^{ik}$ multiplication
to the corner terms. These are the terms corresponding to transitioning the
periodic boundary. The resulting Hamiltonian is

$H_{\mathrm{occ}}^{\mathrm{k}}=\matrixquantity[0&\textit{v}&0&0&0&\textit{w}e^{ik}\\\
\textit{v}&0&\textit{w}&0&0&0\\\ 0&\textit{w}&0&\textit{v}&0&0\\\
0&0&\textit{v}&0&\textit{w}&0\\\ 0&0&0&\textit{w}&0&\textit{v}\\\
\textit{w}e^{ik}&0&0&0&\textit{v}&0].$ (12)

Chiral symmetry operator. The next step is to define the chiral symmetry
operator $\Gamma$. In this system it has the form

$\Gamma=\left[\begin{matrix}1&0&0&0&0&0\\\ 0&-1&0&0&0&0\\\ 0&0&1&0&0&0\\\
0&0&0&-1&0&0\\\ 0&0&0&0&1&0\\\ 0&0&0&0&0&-1\end{matrix}\right].$ (13)

Hamiltonian in chiral symmetry basis. We need to rewrite our Hamiltonian in
the eigenbasis of $\Gamma$. To this end, we must ensure the eigenvectors of
$\Gamma$ are also sorted lexicographically. The resulting change-of-basis
matrices $\chi,\chi^{\dagger}$ have the form

$\chi=\left[\begin{matrix}0&0&0&0&0&1\\\ 0&0&0&1&0&0\\\ 0&1&0&0&0&0\\\
0&0&0&0&1&0\\\ 0&0&1&0&0&0\\\
1&0&0&0&0&0\end{matrix}\right],\quad\chi^{\dagger}=\left[\begin{matrix}0&0&0&0&0&1\\\
0&0&1&0&0&0\\\ 0&0&0&0&1&0\\\ 0&1&0&0&0&0\\\ 0&0&0&1&0&0\\\
1&0&0&0&0&0\end{matrix}\right].$ (14)

Once we apply them to the Hamiltonian $H$ it becomes block off-diagonal,

$H=\chi
H_{\mathrm{occ}}^{\text{k}}\chi^{\dagger}=\left[\begin{matrix}0&0&0&v&0&we^{-ik}\\\
0&0&0&w&v&0\\\ 0&0&0&0&w&v\\\ v&w&0&0&0&0\\\ 0&v&w&0&0&0\\\
we^{ik}&0&v&0&0&0\end{matrix}\right]=\matrixquantity[0&h^{\dagger}\\\ h&0].$
(15)

Figure 6: Phase transition vs disorder strength. Comparison of winding number
computed for different disorder amplitudes $W/\textit{w}$, averaged over
$N_{\mathrm{r}}=25$ realizations of disorder. With increased disorder, the
phase transition point shifts to $\textit{v}/\textit{w}\geq 1$, as discussed
in section II.1. Use of infinite system calculation is visible by the position
of phase transition point, computed to happen at $\textit{v}/\textit{w}=1$ in
disorderless system ($W=0$). In a finite system, it would happen for a smaller
$\textit{v}/\textit{w}$ ratio [70].

Final calculation. Now the elements we need for the calculations according to
equation (10) have the form:

$h=\left[\begin{matrix}v&w&0\\\ 0&v&w\\\ we^{ik}&0&v\end{matrix}\right],\quad
h^{-1}=\frac{1}{v^{3}+w^{3}e^{ik}}\left[\begin{matrix}v^{2}&-vw&w^{2}\\\
w^{2}e^{ik}&v^{2}&-vw\\\
-vwe^{ik}&w^{2}e^{ik}&v^{2}\end{matrix}\right],\quad\partial_{k}h=\left[\begin{matrix}0&0&0\\\
0&0&0\\\ iwe^{ik}&0&0\end{matrix}\right],$
$h^{-1}\partial_{k}h=\frac{ie^{ik}}{v^{3}+w^{3}e^{ik}}\left[\begin{matrix}w^{2}&0&0\\\
-vw&0&0\\\
v^{2}&0&0\end{matrix}\right],\quad\tr\quantity[h^{-1}\partial_{k}h]=\frac{iw^{2}e^{ik}}{v^{3}+w^{3}e^{ik}}.$

We have arrived at the final formula for integration to obtain the winding
number. Now, it needs to be integrated over the $1^{\text{st}}$ Brillouin
Zone. Due to software stability reasons, we integrated it numerically for
$k\in\quantity[0,2\pi-\epsilon]$.

Universality. This calculation of the winding number also applies to the
disordered SSH Hamiltonian, defined by equation (2). The phase transition
point then drifts to values of $\textit{v}/\textit{w}\geq 1$ with an increase
of disorder amplitude $W/\textit{w}$. This is presented in figure 6.

##  Appendix C Architectures and hyperparameters

Data point. Single data point supplied to the network is a matrix of squared
coefficients of eigenstates of Hamiltonians defined by equations (1)
(represented in a lexicographically sorted occupational basis), (2) and its
corresponding label. Each column is a single normalized eigenstate. The label
is the winding number. A step-by-step demonstration of the calculation of the
winding number is presented in appendix B.

Basic training parameters. All throughout this work we keep the intercell
tunneling amplitude $\textit{w}=1$. The two phases of the SSH system are
probed by varying $\textit{v}/\textit{w}\in\quantity(0,2)$. In all data sets
(training, validation, test) both phases are equally numerous – 50% of all
data points represent each phase. To ensure linear independence of data in all
three sets, the intervals of $\textit{v}/\textit{w}$ were chosen with a
slightly offset initial point. That is $0.001,0.002,0.003$ for training,
validation, and test set, respectively. This ensures they all represent
distinct points in the phase diagram, and that in disorderless setting they
belong to the same distribution.

Data set composition. A part of what we would like to achieve is for the
network to learn the correct phase transition point by itself. Therefore, the
training and validation data sets do not contain data from the direct vicinity
of the phase transition. In these two sets, we vary the parameter v in the
range $\textit{v}/\textit{w}\in\quantity(0,0.8)\cup\quantity(1.2,2)$. The
composition of disorderless data sets is presented in the left column of Table
2.

Training extension – Disordered data. The extension that proved fruitful was
an addition of disordered data to the training. In this setting, we included
data from low-disorder regimes ($W/\textit{w}\in{0.01,0.05})$ in training and
validation sets. For each disorder amplitude, the data added to the training
set remained in $5:1$ ratio to the disorderless data. For the validation set,
the ratio was set at $4:1$. The composition of data sets used in both regimes
of training are presented in Table 2.

Generalization assessment. To assess the networks’ generalization, we tasked
it with recreating the target v vs $W$ phase diagram. To this end, we
generated $N_{\mathrm{r}}=25$ disorder realizations of the input data and
corresponding labels. We covered the range
$\textit{v}/\textit{w}\in\quantity[0,2]$, and $W/\textit{w}\in\quantity[0,5]$
for each realization. The step size in disorder amplitude strength is $\Delta
W/\textit{w}=0.1$, while the step size in intracell tunneling amplitude is
$\Delta\textit{v}/\textit{w}=0.01$. After collecting predictions for all
$N_{\mathrm{r}}$ realizations, the values in the generated phase diagram are
then averaged over $N_{\mathrm{r}}$ predicted labels for each pair of
$\quantity(\textit{v}/\textit{w},W/\textit{w})$.

Table 2: Composition of data sets used in networks’ training. The base setting involved only using data coming from the SSH model in equation (1). In an extension to this approach, data from disordered SSH model was added, see equation (2). Approach | Disorderless | Disordered data
---|---|---
Disorder | $W=0$ | $W=0$ | $W/\textit{w}=0.01$ | $W/\textit{w}=0.05$
Training points | 5000 | 5000 | 1000 | 1000
Validation points | 1000 | 1000 | 250 | 250
Figure 7: Architecture of the CNN used in this work.

Architecture. The architecture we use is a shallow convolutional neural
network (CNN), as presented in figure 7. It consists of three convolutional
layers with symmetric kernels. The initial convolution filter is of shape (4,
4), followed by convolutions with (3, 3) kernels. Before the final fully-
connected classification layer, we position the global average pooling (GAP)
layer in order to accommodate the requirements of the CAM attribution
technique.

Hyperparameters. We trained the networks using Stochastic Gradient Descent
(SGD). We explored various combinations of learning rate (LR), momentum,
weight decay, and batch size. The range of LRs we evaluated was
$10^{-6}-10^{0}$. The tested batch sizes were $64$ and $500$. The
hyperparameters we picked for training were: LR =$10^{-4}$, momentum $=0.1$,
weight decay $=0.1$, batch size $=64$. The networks were trained for up to 500
epochs, with early stopping allowed after 10 consecutive epochs of validation
loss rise, with warm-up of 50 epochs.

##  Appendix D Gradient-based interpretability techniques are fragile

Introduction to class activation mapping. CAM is an attribution technique
tailored for use with CNNs and leverages their design. A CNN produces
different representations of the input data during a forward pass through the
network and encodes them in different channels. CAM relies on the latent
representation of the data present in the $N_{\text{ch}}$ output channels of
the last convolutional layer in a network (activations $A_{k}$), by performing
their weighted sum with weights $\alpha_{k}$ [82],

$\text{CAM}^{C}(A)=\text{ReLU}~{}\quantity(\sum_{k=1}^{N_{\text{ch}}}\alpha_{k}A_{k})\,.$
(16)

Figure 8: Class activation mapping (CAM) generates a map that highlights
relevant parts of the input data for predicting a selected class $C$ (out of
$N_{C}$ classes), e.g., a pug out of all 1000 ImageNet categories. In the
final convolutional layer of the network, each of the $N_{ch}$ channels
generates a convolved image, $A_{k}$ (three out of $N_{ch}$ are marked with
blue, red, and green squares in the figure). Each activation can encode some
data features relevant to some of the classes. An intuitive example of such
features could be the shortened snout of a pug or cat’s whiskers. The GAP
layer then reduces each of these activations to its mean value (e.g., each
colored square got reduced to the scalar of the same color). Finally, the
fully-connected layer maps GAP values to the network outputs and its weights
are used as weights for the CAM. Adapted from Ref. [67].

The resulting map, after rescaling back to the original data size, highlights
the areas that were the most influential in predicting a considered class $C$.
The crucial part is the choice of the weights $\alpha_{k}$. The original
formulation of the method relies on reducing each channel output of the last
convolutional layer to a single number with global average pooling (GAP) layer
[67]. The weights of fully-connected layer connecting those numbers with the
output corresponding to the considered class $C$ are then used as $\alpha_{k}$
for the weighted average of convolved images. We present this explanation
graphically in figure 8.

CAM fails for the disordered regime. As discussed before, CAM provides
invaluable insight into possible correlations in disorderless data. However,
the results proved unreliable when applied to data from a disordered regime
($W/\textit{w}\leq 0.05$). Contrary to the previously observed and expected
tendencies, CAM importance maps were asymmetric and varied significantly
between neighboring v values and multiple realizations of the same point in
phase space. This example is presented in figures 9(c)-(d).

NN’s fragility. The CNNs have been shown to be prone to ’adversarial noise’
attacks [99, 100]. They can take various intricate forms, as discussed by Ref.
[101]. The simplest scenario of an adversarial noise attack is one where the
input image is perturbed in a deliberately engineered way to fool the CNN and
make it predict a different class while keeping the input visually
indistinguishable for a human. This class of attacks targets the predictive
power of CNN. However, the problem in our case is the more subtle –
predictions remain factual, and the gradient-based interpretation fails.

Figure 9: Interpretation fragility. (a) Gradient-based attribution techniques
(CAM included) rely on the direction of the gradient (pink arrow), so small
perturbation $\delta$ of test image $x_{t}$ can vary the basis of explanation,
leaving class prediction unaffected. Panel adapted from Ref. [102]. (b)
Thermometer encoding, a solution to this problem proposed by Buckman et al.
[103], is a discretization technique for encoding a floating point input
variable as a vector representing its magnitude. Adapted from Ref. [103]. (d)
In a disordered regime, CAM yields flawed interpretations. (c) The input is
axially symmetric – a property that the interpretation should preserve. (e)
This is remedied by using thermometer encoding at the cost of limited network
performance and a rise in computational cost.

Interpretation fragility. The topic of attacks on interpretation alone has
already been addressed by the computer science community [102, 104, 105]. The
intuition provided by Ref. [102] is as follows. Gradient-based techniques,
like CAM, rely on the direction one must go from any given test example to the
decision boundary. This is the geometric interpretation of the gradient in
parameter space. The decision boundary, being highly non-convex, makes it very
easy to perturb a given test image while still within the boundaries of its
class. This slight shift is enough for the gradient vector to point in another
direction, producing a different interpretation. See figure 9(a).

Thermometer encoding. Buckman et al. [103] have proposed using thermometer
encoding of input to remedy this problem. It belongs to a broader class of
approaches to reducing problem complexity via casting input with floating
point entries to discrete variables. A standard input discretization
technique, one-hot encoding, encodes a quantized floating point number as a
vector with the only non-zero entry marking the range in which it falls. Ref.
[103] argues that by mimicking the behavior of a mercury thermometer, encoding
the magnitude of the encoded input, the network gradients are better behaved.
This change, they argue, reduces the fragility of gradient-based network
attribution techniques. The thermometer encoding process is demonstrated in
figure 9(b).

Thermometer recovers CAM. We have implemented thermometer encoding with the
number of discretization bins ranging from 10 to 100 to test its applicability
to our problem. We have successfully recovered the insight provided by CAM, as
displayed in figure 9(e). We have found the $100$ discretization bins to be
the optimal value for the problem. This technique retrieves CAM interpretation
for all disorders in the range addressed by this study.

Cost of CAM interpretability. While gradient-based interpretation is
recovered, there is collateral damage done to network performance and training
costs. None out of 100 networks trained using this thermometer encoding
generalized well to data with disorder, even when trained on data with added
small disorder $W/\textit{w}\in\quantity{0,0.01,0.05}$. This is a significant
performance loss when compared to 22% of well-generalizing networks obtained
when the architecture is not adjusted to the thermometer encoding. There is
also a substantial rise in the computational cost of network training. For
$100$ discretization bins, the number of trainable parameters rises from
$\mathcal{O}(10^{3})$ to $\mathcal{O}(10^{9})$. This, in turn, extended the
training time of a single network from $\mathcal{O}(1\mathrm{min})$ to
$\mathcal{O}(2\mathrm{h})$. The training was performed in PyTorch [76] on a
CUDA-enabled NVIDIA GeForce GTX 1050 graphics card with 4 GB of GDDR5 VRAM.

##  Appendix E Toy model for CAM performance assessment

Figure 10: CAM analysis of a network trained and tested in a simplified
setting. All entries in data points are drawn from the uniform distribution
$U[0,0.1]$. In half of the data, we set the edge entries of two middle columns
to 1. This constitutes a ’Topological’ phase, with the second part labeled
’Trivial’ phase. CAM analysis of this setting confirms that the features the
network uses for class discrimination are the bright pixels.

The motivation behind our use of CAM can be demonstrated using a toy model of
the problem at hand, imitating the SSH model with a slight disorder. In this
setting, we create an artificial data set consisting of data points, where all
entries are drawn from the uniform distribution $U[0,0.1]$. We then separate
it into two parts and set the edge entries of two middle columns to 1. In this
setting, the phases are not distinguished by any correlations other than the
two bright pixels we set. CAM analysis of a network trained and tested in this
setting shows that the features the network uses for class discrimination are
the bright pixels. This result is shown in figure 10.

##  Appendix F Evolution of the data representation across layers of a neural
network

Representation varies between network layers. We can also study the evolution
of the data representation across layers, which gives additional insights into
the behavior of different networks. To this end, we now investigate outputs of
the three consecutive convolutional layers (Conv1, Conv2, Conv3), next to the
output of the GAP layer, which we have studied in the main text in figure 4.
Contrary to the discussion in the main text, here we do not remove the data
from the vicinity of the transition point. We also inspect the respective
representation of the test data without disorder and OOD test data with a
single realization of multiple disorder amplitudes
($W/\textit{w}\in\quantity{0.15,0.5,1,2}$). However, not all data sets can be
visualized accurately in a few dimensions using only linear transformations.
Therefore, we chose a nonlinear technique to visualize the underlying
structure of the latent space representations in the CNNs more accurately than
PCA, described in section II.4. To this end, we use the uniform manifold
approximation and projection (UMAP) [106].

Figure 11: Uniform manifold approximation and projection (UMAP) routine
sketch. UMAP is a two-step routine: first, it computes a graph that represents
the data. The second step is to learn a low-dimensional representation of the
graph while trying to preserve the local, global, and topological structure of
the data. This allows for efficient visualization of high-dimensional
datasets. Adapted from Ref. [107].

UMAP – nonlinear dimensionality reduction. As seen in figure 11, UMAP is a
two-step routine: first, it computes a graph that represents the data and then
learns a low-dimensional representation of the graph while trying to preserve
the local, global, and topological structure of the data. UMAP is good at
visualizing high-dimensional nonlinear data sets, but it has some drawbacks.
First, the embedding learning process is stochastic, which means that the
reproducibility of the results is limited. Second, it can distort the true
nature of data connectivity and density [108, 109, 110], so it is essential to
be cautious when using it to conclude the clustering or concentration of data.
It is a good practice to verify that the resulting number of clusters agrees
with what we expect to find in the data based on results from other
techniques. Despite its limitations, UMAP has proven to be computationally
inexpensive and is often used because of its ability to visualize high-
dimensional data efficiently. These are the reasons behind our decision to use
it to analyze the latent space representations in the CNNs we studied. The
UMAP embeddings are visualized in figures 12 and 13 for well- and poorly-
generalizing networks, respectively.

Figure 12: Evolution of the internal data representation with UMAP within
different layers of the well-generalizing CNN that does not focus on edge
states, as seen figure 2(d),(h). Figure 13: Evolution of the internal data
representation with UMAP within different layers of the poorly-generalizing
CNN that focuses on edge states, as seen figure 2(c),(g).

UMAP results for raw data. Analysis of raw data with UMAP (first row of
figures 12 and 13) allows for two interesting observations. Firstly,
surprisingly, the UMAP embedding is itself a better classifier of data from
disordered regimes than the majority of CNNs. The embedding of both classes in
such data overlaps with the embeddings of classes from a disorderless system.
This could allow for an informed decision about proper labels in the OOD
regime. Secondly, it is a testament to the previously mentioned stochasticity
of low-dimensional representations. The raw data subject to analysis are the
same in both figures, but their embedding is slightly different. This is a
result of UMAP’s limited reproducibility. Upon careful inspection, these UMAP
embeddings led to one more observation: that the data coming from the
topological class formed two disjoint clusters. This phenomenon is due to the
hybridization of the edge states [70].

Well-generalizing CNNs represent data with and without disorder similarly. Let
us first focus on the two-dimensional data representation generated by UMAP
for the well-generalizing network, as seen in figure 12. In the consecutive
columns, we plot the disorderless data and the data with increasing disorder
strength (indicated with darker orange and blue). In the first convolutional
layer (Conv1), the disordered data are represented very similarly to those
without the disorder. In other words, their representation overlaps. In the
following two layers (Conv2, Conv3), the class representations overlap for
intermediate disorder amplitudes ($W/\textit{w}\in\quantity{0.5,1}$. With
increasing disorder amplitude ($W/\textit{w}\geq 2$), the embedded data start
to disconnect from the original clusters. In particular, the embedded Conv3
layer activations, which initially clustered with the topological class, begin
forming a class of its own. This may signify a change in the data structure
the network has learned to observe. It also means that at this stage of the
forward pass, the network has a disjointed representation only of strongly
disordered data.

The lowest row presents embedded activations from the GAP layer, a
qualitatively different picture. Knowing the nature of GAP output, it seems
safe to assume that what we see here is an embedding of a unidimensional
manifold. As disorder increases, the initially disjoint clusters of
disorderless data gradually connect with data from increasingly disordered
regimes. This captures well the perceived similarity of both classes an
observer, artificial or human, has when observing the input data.

Poorly-generalizing CNNs have very different representation of data with and
without disorder. Let us now move on to the two-dimensional data
representation generated by UMAP for the poorly-generalizing network, as seen
in figure 13. In the first convolutional layer (Conv1), the disordered data
are represented very similarly to those without the disorder. This behaviour
is consistent with the one observed for the well-generalizing network. In the
following two layers (Conv2, Conv3), the class representations overlap for
intermediate disorder amplitudes ($W/\textit{w}\in\quantity{0.5,1}$. With
increasing disorder amplitude ($W/\textit{w}\geq 2$), the embedded data start
to disconnect from the original clusters. Given the poor generalizability we
expect from this network, it is not surprising to see them merge together, and
then finally be positioned near just one of the initial distributions. The
fact that it gets positioned near the topological phase representation might
suggest that this would be the phase predicted in high disorder regime, and
this notion turns out to be true here.

The lowest row of figure 12 is once again a qualitatively different picture
from the previous layers, but of the same nature as embedding of GAP layer
activations from well-generalizing network. This further strengthens the
assumption that is an embedding of a unidimensional manifold. Here, however,
it is hard to say surely that the initial clusters of disorderless data are
well separated. Their, already low, separation vanishes as disorder increases.
The disordered data then proceed to merge together and position in the range
of topological class. This, once again, correctly suggests that this would be
the phase predicted in high disorder regime.

## References

  * Dawid and LeCun [2023] A. Dawid and Y. LeCun, Introduction to latent variable energy-based models: A path towards autonomous machine intelligence (2023), arXiv:2306.02572 [cs.LG] .
  * Hermann et al. [2023] J. Hermann, J. Spencer, K. Choo, A. Mezzacapo, W. M. C. Foulkes, D. Pfau, G. Carleo, and F. Noé, Ab initio quantum chemistry with neural-network wavefunctions, Nat. Rev. Chem. 7, 692–709 (2023).
  * Medvidović and Moreno [2024] M. Medvidović and J. R. Moreno, Neural-network quantum states for many-body physics (2024), arXiv:2402.11014 [cond-mat.dis-nn] .
  * Lange et al. [2024] H. Lange, A. V. de Walle, A. Abedinnia, and A. Bohrdt, From architectures to applications: A review of neural quantum states (2024), arXiv:2402.09402 [cond-mat.dis-nn] .
  * Dawid et al. [2023] A. Dawid, J. Arnold, B. Requena, A. Gresch, M. Płodzień, K. Donatella, K. A. Nicoli, P. Stornati, R. Koch, M. Büttner, R. Okuła, G. Muñoz-Gil, R. A. Vargas-Hernández, A. Cervera-Lierta, J. Carrasquilla, V. Dunjko, M. Gabrié, P. Huembeli, E. van Nieuwenburg, F. Vicentini, L. Wang, S. J. Wetzel, G. Carleo, E. Greplová, R. Krems, F. Marquardt, M. Tomza, M. Lewenstein, and A. Dauphin, Modern applications of machine learning in quantum sciences (2023), arXiv:2204.04198 [quant-ph] .
  * Krenn et al. [2023] M. Krenn, J. Landgraf, T. Foesel, and F. Marquardt, Artificial intelligence and machine learning for quantum technologies, Phys. Rev. A 107, 010101 (2023).
  * Mafu [2024] M. Mafu, Advances in artificial intelligence and machine learning for quantum communication applications, IET Quantum Commun. 10.1049/qtc2.12094 (2024).
  * Lu et al. [2018] S. Lu, S. Huang, K. Li, J. Li, J. Chen, D. Lu, Z. Ji, Y. Shen, D. Zhou, and B. Zeng, Separability-entanglement classifier via machine learning, Phys. Rev. A 98, 012315 (2018).
  * Sanavio et al. [2023] C. Sanavio, E. Tignone, and E. Ercolessi, Entanglement classification via witness operators generated by support vector machine, Eur. Phys. J. Plus 138, 936 (2023).
  * Carrasquilla and Melko [2017] J. Carrasquilla and R. G. Melko, Machine learning phases of matter, Nat. Phys. 13, 431 (2017).
  * Li et al. [2018] C.-D. Li, D.-R. Tan, and F.-J. Jiang, Applications of neural networks to the studies of phase transitions of two-dimensional Potts models, Annals of Physics 391, 312 (2018).
  * Schäfer and Lörch [2019] F. Schäfer and N. Lörch, Vector field divergence of predictive model output as indication of phase transitions, Phys. Rev. E 99, 062107 (2019).
  * Bachtis et al. [2020] D. Bachtis, G. Aarts, and B. Lucini, Mapping distinct phase transitions to a neural network, Phys. Rev. E 102, 053306 (2020).
  * Liu et al. [2021] K. Liu, N. Sadoune, N. Rao, J. Greitemann, and L. Pollet, Revealing the phase diagram of Kitaev materials by machine learning: Cooperation and competition between spin liquids, Phys. Rev. Research 3, 023016 (2021).
  * Richter-Laskowska et al. [2023] M. Richter-Laskowska, M. Kurpas, and M. M. Maśka, Learning by confusion approach to identification of discontinuous phase transitions, Phys. Rev. E 108, 024113 (2023).
  * Van Nieuwenburg et al. [2017] E. P. L. Van Nieuwenburg, Y.-H. Liu, and S. D. Huber, Learning phase transitions by confusion, Nat. Phys. 13, 435 (2017).
  * Wetzel [2017] S. J. Wetzel, Unsupervised learning of phase transitions: From principal component analysis to variational autoencoders, Phys. Rev. E 96, 022140 (2017).
  * Liu and van Nieuwenburg [2018] Y.-H. Liu and E. P. L. van Nieuwenburg, Discriminative cooperative networks for detecting phase transitions, Phys. Rev. Lett. 120, 176401 (2018).
  * Ch’ng et al. [2018] K. Ch’ng, N. Vazquez, and E. Khatami, Unsupervised machine learning account of magnetic transitions in the Hubbard model, Phys. Rev. E 97, 013306 (2018).
  * Huembeli et al. [2019] P. Huembeli, A. Dauphin, P. Wittek, and C. Gogolin, Automated discovery of characteristic features of phase transitions in many-body localization, Phys. Rev. B 99, 104106 (2019).
  * Kottmann et al. [2020] K. Kottmann, P. Huembeli, M. Lewenstein, and A. Ac$́i$n, Unsupervised phase discovery with deep anomaly detection, Phys. Rev. Lett. 125, 170603 (2020).
  * Kottmann et al. [2021] K. Kottmann, P. Corboz, M. Lewenstein, and A. Ac$́i$n, Unsupervised mapping of phase diagrams of 2D systems from infinite projected entangled-pair states via deep anomaly detection, SciPost Phys. 11, 025 (2021).
  * Arnold et al. [2021] J. Arnold, F. Schäfer, M. Žonda, and A. U. J. Lode, Interpretable and unsupervised phase classification, Phys. Rev. Research 3, 033052 (2021).
  * Patel et al. [2022] Z. Patel, E. Merali, and S. J. Wetzel, Unsupervised learning of Rydberg atom array phase diagram with Siamese neural networks, New J. Phys. 24, 113021 (2022).
  * Broecker et al. [2017] P. Broecker, J. Carrasquilla, R. G. Melko, and S. Trebst, Machine learning quantum phases of matter beyond the fermion sign problem, Sci. Rep. 7, 8823 (2017).
  * Théveniaut and Alet [2019] H. Théveniaut and F. Alet, Neural network setups for a precise detection of the many-body localization transition: Finite-size scaling and limitations, Phys. Rev. B 100, 224202 (2019).
  * Dong et al. [2019] X.-Y. Dong, F. Pollmann, and X.-F. Zhang, Machine learning of quantum phase transitions, Phys. Rev. B 99, 121104 (2019).
  * Blücher et al. [2020] S. Blücher, L. Kades, J. M. Pawlowski, N. Strodthoff, and J. M. Urban, Towards novel insights in lattice field theory with explainable machine learning, Phys. Rev. D 101, 094507 (2020).
  * Zhang et al. [2018] P. Zhang, H. Shen, and H. Zhai, Machine learning topological invariants with neural networks, Phys. Rev. Lett. 120, 066401 (2018).
  * Tsai et al. [2020] Y.-H. Tsai, M.-Z. Yu, Y.-H. Hsu, and M.-C. Chung, Deep learning of topological phase transitions from entanglement aspects, Phys. Rev. B 102, 054512 (2020).
  * Baireuther et al. [2023] P. Baireuther, M. Płodzień, T. Ojanen, J. Tworzydło, and T. Hyart, Identifying Chern numbers of superconductors from local measurements, SciPost Phys. Core 6, 087 (2023).
  * Huembeli et al. [2018] P. Huembeli, A. Dauphin, and P. Wittek, Identifying quantum phase transitions with adversarial neural networks, Phys. Rev. B 97, 134109 (2018).
  * Rodriguez-Nieva and Scheurer [2019] J. F. Rodriguez-Nieva and M. S. Scheurer, Identifying topological order through unsupervised machine learning, Nat. Phys. 15, 790 (2019).
  * Greplova et al. [2020] E. Greplova, A. Valenti, G. Boschung, F. Schäfer, N. Lörch, and S. D. Huber, Unsupervised identification of topological phase transitions using predictive models, New J. Phys. 22, 045003 (2020).
  * Balabanov and Granath [2021] O. Balabanov and M. Granath, Unsupervised interpretable learning of topological indices invariant under permutations of atomic bands, Mach. Learn.: Sci. Technol. 2, 025008 (2021).
  * Yu et al. [2023] L.-W. Yu, S.-Y. Zhang, P.-X. Shen, and D.-L. Deng, Unsupervised learning of interacting topological phases from experimental observables, Fundamental Research 10.1016/j.fmre.2022.12.016 (2023).
  * Teng et al. [2023] Y. Teng, S. Sachdev, and M. S. Scheurer, Clustering neural quantum states via diffusion maps, Phys. Rev. B 108, 205152 (2023).
  * Rem et al. [2019] B. S. Rem, N. Käming, M. Tarnowski, L. Asteria, N. Fläschner, C. Becker, K. Sengstock, and C. Weitenberg, Identifying quantum phase transitions using artificial neural networks on experimental data, Nat. Phys. 15, 917 (2019).
  * Khatami et al. [2020] E. Khatami, E. Guardado-Sanchez, B. M. Spar, J. F. Carrasquilla, W. S. Bakr, and R. T. Scalettar, Visualizing strange metallic correlations in the two-dimensional fermi-hubbard model with artificial intelligence, Phys. Rev. A 102, 033326 (2020).
  * Käming et al. [2021] N. Käming, A. Dawid, K. Kottmann, M. Lewenstein, K. Sengstock, A. Dauphin, and C. Weitenberg, Unsupervised machine learning of topological phase transitions from experimental data, Mach. Learn.: Sci. Technol. 2, 035037 (2021).
  * Miles et al. [2023] C. Miles, R. Samajdar, S. Ebadi, T. T. Wang, H. Pichler, S. Sachdev, M. D. Lukin, M. Greiner, K. Q. Weinberger, and E.-A. Kim, Machine learning discovery of new phases in programmable quantum simulator snapshots, Phys. Rev. Res. 5, 013026 (2023).
  * Link et al. [2023] M. Link, K. Gao, A. Kell, M. Breyer, D. Eberz, B. Rauf, and M. Köhl, Machine learning the phase diagram of a strongly interacting fermi gas, Phys. Rev. Lett. 130, 203401 (2023).
  * Wang [2016] L. Wang, Discovering phase transitions with unsupervised learning, Phys. Rev. B 94, 195105 (2016).
  * Che et al. [2020] Y. Che, C. Gneiting, T. Liu, and F. Nori, Topological quantum phase transitions retrieved through unsupervised machine learning, Phys. Rev. B 102, 134213 (2020).
  * Long et al. [2020] Y. Long, J. Ren, and H. Chen, Unsupervised manifold clustering of topological phononics, Phys. Rev. Lett. 124, 185501 (2020).
  * Yang et al. [2021] Y. Yang, Z.-Z. Sun, S.-J. Ran, and G. Su, Visualizing quantum phases and identifying quantum phase transitions by nonlinear dimensional reduction, Phys. Rev. B 103, 075106 (2021).
  * Vargas-Hernández et al. [2018] R. A. Vargas-Hernández, J. Sous, M. Berciu, and R. V. Krems, Extrapolating quantum observables with machine learning: inferring multiple phase transitions from properties of a single phase, Phys. Rev. Lett. 121, 255702 (2018).
  * Greitemann et al. [2019a] J. Greitemann, K. Liu, L. D. C. Jaubert, H. Yan, N. Shannon, and L. Pollet, Identification of emergent constraints and hidden order in frustrated magnets using tensorial kernel methods of machine learning, Phys. Rev. B 100, 174408 (2019a).
  * Greitemann et al. [2021] J. Greitemann, K. Liu, and L. Pollet, The view of TK-SVM on the phase hierarchy in the classical kagome Heisenberg antiferromagnet, J. Phys.: Condens. Matter 33, 054002 (2021).
  * Leykam and Angelakis [2023] D. Leykam and D. G. Angelakis, Topological data analysis and machine learning, Adv. Phys.: X 8, 2202331 (2023).
  * Scheurer and Slager [2020] M. S. Scheurer and R.-J. Slager, Unsupervised machine learning and band topology, Phys. Rev. Lett. 124, 226401 (2020).
  * Cole et al. [2021] A. Cole, G. J. Loges, and G. Shiu, Quantitative and interpretable order parameters for phase transitions from persistent homology, Phys. Rev. B 104, 104426 (2021).
  * Park et al. [2022] S. Park, Y. Hwang, and B.-J. Yang, Unsupervised learning of topological phase diagram using topological data analysis, Phys. Rev. B 105, 195115 (2022).
  * Schuld and Killoran [2019] M. Schuld and N. Killoran, Quantum machine learning in feature Hilbert spaces, Phys. Rev. Lett. 122, 040504 (2019).
  * Caro et al. [2022] M. C. Caro, H.-Y. Huang, M. Cerezo, K. Sharma, A. Sornborger, L. Cincio, and P. J. Coles, Generalization in quantum machine learning from few training data, Nat. Comm. 13, 4919 (2022).
  * Liu et al. [2023] Y.-J. Liu, A. Smith, M. Knap, and F. Pollmann, Model-independent learning of quantum phases of matter with quantum convolutional neural networks, Phys. Rev. Lett. 130, 220603 (2023).
  * Dawid et al. [2020] A. Dawid, P. Huembeli, M. Tomza, M. Lewenstein, and A. Dauphin, Phase detection with neural networks: interpreting the black box, New J. Phys. 22, 115001 (2020).
  * Wetzel et al. [2020] S. J. Wetzel, R. G. Melko, J. Scott, M. Panju, and V. Ganesh, Discovering symmetry invariants and conserved quantities by interpreting siamese neural networks, Phys. Rev. Res. 2, 033499 (2020).
  * Dawid et al. [2021] A. Dawid, P. Huembeli, M. Tomza, M. Lewenstein, and A. Dauphin, Hessian-based toolbox for reliable and interpretable machine learning in physics, Mach. Learn.: Sci. Techn. 3, 015002 (2021).
  * Arnold and Schäfer [2022] J. Arnold and F. Schäfer, Replacing neural networks by optimal analytical predictors for the detection of phase transitions, Phys. Rev. X 12, 031044 (2022).
  * Wetzel [2024] S. J. Wetzel, Closed-form interpretation of neural network classifiers with symbolic regression gradients (2024), arXiv:2401.04978 [cs.LG] .
  * Wetzel and Scherzer [2017] S. J. Wetzel and M. Scherzer, Machine learning of explicit order parameters: From the Ising model to SU(2) lattice gauge theory, Phys. Rev. B 96, 184410 (2017).
  * Greitemann et al. [2019b] J. Greitemann, K. Liu, and L. Pollet, Probing hidden spin order with interpretable machine learning, Phys. Rev. B 99, 060404 (2019b).
  * Liu et al. [2019] K. Liu, J. Greitemann, and L. Pollet, Learning multiple order parameters with interpretable machines, Phys. Rev. B 99, 104410 (2019).
  * Miles et al. [2021] C. Miles, A. Bohrdt, R. Wu, C. Chiu, M. Xu, G. Ji, M. Greiner, K. Q. Weinberger, E. Demler, and E.-A. Kim, Correlator convolutional neural networks as an interpretable architecture for image-like quantum matter data, Nat. Comm. 12, 10.1038/s41467-021-23952-w (2021).
  * Tran et al. [2022] D. Tran, J. Z. Liu, M. W. Dusenberry, D. Phan, M. Collier, J. Ren, K. Han, Z. Wang, Z. E. Mariet, H. Hu, N. Band, T. G. J. Rudner, Z. Nado, J. van Amersfoort, A. Kirsch, R. Jenatton, N. Thain, E. K. Buchanan, K. P. Murphy, D. Sculley, Y. Gal, Z. Ghahramani, J. Snoek, and B. Lakshminarayanan, Plex: Towards reliability using pretrained large model extensions, in First Workshop on Pre-training: Perspectives, Pitfalls, and Paths Forward at ICML 2022 (2022) arXiv:2207.07411.
  * Zhou et al. [2016] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba, Learning deep features for discriminative localization, in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (2016) pp. 2921–2929, http://cnnlocalization.csail.mit.edu/.
  * Fischer [2022] J. S. Fischer, Class activation maps (2022).
  * Anna Dawid-Łękowska [2022] Anna Dawid-Łękowska, Quantum many-body physics with ultracold atoms and molecules: exact dynamics and machine learning, Ph.D. thesis, University of Warsaw, Universitat Politècnica de Catalunya. Institut de Ciències Fotòniques (2022).
  * Asbóth et al. [2016] J. K. Asbóth, L. Oroszlány, and A. Pályi, A Short Course on Topological Insulators (Springer International Publishing, 2016).
  * Mondragon-Shem et al. [2014] I. Mondragon-Shem, T. L. Hughes, J. Song, and E. Prodan, Topological Criticality in the Chiral-Symmetric AIII Class at Strong Disorder, Phys. Rev. Lett. 113, 046802 (2014).
  * Meier et al. [2018] E. J. Meier, F. A. An, A. Dauphin, M. Maffei, P. Massignan, T. L. Hughes, and B. Gadway, Observation of the topological Anderson insulator in disordered atomic wires, Science 362, 929 (2018).
  * Le et al. [2020] N. H. Le, A. J. Fisher, N. J. Curson, and E. Ginossar, Topological phases of a dimerized Fermi–Hubbard model for semiconductor nano-lattices, Npj Quantum Inf. 6, 24 (2020).
  * Fraxanet et al. [2021] J. Fraxanet, U. Bhattacharya, T. Grass, D. Rakshit, M. Lewenstein, and A. Dauphin, Topological properties of the long-range Kitaev chain with Aubry-André-Harper modulation, Phys. Rev. Res. 3, 013148 (2021).
  * Zhang and Dong [2010] J. M. Zhang and R. X. Dong, Exact diagonalization: the Bose-Hubbard model as an example, Eur. J. Phys. 31, 591 (2010).
  * Paszke et al. [2019] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, PyTorch: An imperative style, high-performance deep learning library (2019), arXiv:1912.01703 [cs.LG] .
  * Good [1952] I. J. Good, Rational decisions, J. R. Stat. Soc., Ser. B, Methodol. 14, 107 (1952).
  * Anderson [1958] P. W. Anderson, Absence of diffusion in certain random lattices, Phys. Rev. 109, 1492 (1958).
  * Ye et al. [2024] W. Ye, G. Zheng, X. Cao, Y. Ma, X. Hu, and A. Zhang, Spurious correlations in machine learning: A survey (2024), arXiv:2402.12715 [cs.LG] .
  * Kim et al. [2016] B. Kim, R. Khanna, and O. O. Koyejo, Examples are not enough, learn to criticize! Criticism for interpretability, in Advances in Neural Information Processing Systems, Vol. 29 (Curran Associates, Inc., 2016).
  * Molnar [2023] C. Molnar, Interpretable Machine Learning (published independently, 2023) https://christophm.github.io/interpretable-ml-book/.
  * Jung and Oh [2021] H. Jung and Y. Oh, Towards Better Explanations of Class Activation Mapping, in 2021 IEEE/CVF International Conference on Computer Vision (ICCV) (IEEE Computer Society, Los Alamitos, CA, USA, 2021) pp. 1316–1324.
  * Smilkov et al. [2017] D. Smilkov, N. Thorat, B. Kim, F. Viégas, and M. Wattenberg, Smoothgrad: removing noise by adding noise (2017).
  * Selvaraju et al. [2020] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, Grad-CAM: Visual explanations from deep networks via gradient-based localization, Int. J. Comput. Vis. 128, 336 (2020).
  * Chattopadhay et al. [2018] A. Chattopadhay, A. Sarkar, P. Howlader, and V. N. Balasubramanian, Grad-CAM++: Generalized Gradient-Based Visual Explanations for Deep Convolutional Networks, in 2018 IEEE Winter Conference on Applications of Computer Vision (WACV) (2018) pp. 839–847.
  * Desai and Ramaswamy [2020] S. Desai and H. G. Ramaswamy, Ablation-CAM: Visual Explanations for Deep Convolutional Network via Gradient-free Localization, in 2020 IEEE Winter Conference on Applications of Computer Vision (WACV) (2020) pp. 972–980.
  * Fu et al. [2020] R. Fu, Q. Hu, X. Dong, Y. Guo, Y. Gao, and B. Li, Axiom-based Grad-CAM: Towards Accurate Visualization and Explanation of CNNs (2020), arXiv:2008.02312 [cs.CV] .
  * Wang et al. [2020] H. Wang, Z. Wang, M. Du, F. Yang, Z. Zhang, S. Ding, P. Mardziel, and X. Hu, Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networks, in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (2020) pp. 111–119.
  * Erhan et al. [2009] D. Erhan, Y. Bengio, A. C. Courville, and P. Vincent, Visualizing higher-layer features of a deep network, in 2009 International Conference on Machine Learning (ICML) (2009).
  * Simonyan et al. [2014] K. Simonyan, A. Vedaldi, and A. Zisserman, Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps (2014), arXiv:1312.6034 [cs.CV] .
  * Nguyen et al. [2015] A. Nguyen, J. Yosinski, and J. Clune, Deep neural networks are easily fooled: High confidence predictions for unrecognizable images, in 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (IEEE, 2015).
  * Olah et al. [2017] C. Olah, A. Mordvintsev, and L. Schubert, Feature visualization, Distill 2, 10.23915/distill.00007 (2017).
  * Carter et al. [2019] S. Carter, Z. Armstrong, L. Schubert, I. Johnson, and C. Olah, Activation atlas, Distill 4, 10.23915/distill.00015 (2019).
  * Pearson [1901] K. Pearson, LIII. On lines and planes of closest fit to systems of points in space, Lond. Edinb. Dublin Philos. Mag. J. Sci. 2, 559 (1901).
  * Tsipras et al. [2020] D. Tsipras, S. Santurkar, L. Engstrom, A. Turner, and A. Madry, Robustness may be at odds with accuracy, in International Conference on Learning Representations (2020).
  * Madry et al. [2018] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, Towards Deep Learning Models Resistant to Adversarial Attacks, in International Conference on Learning Representations (ICLR) (2018).
  * Cybiński et al. [2023] K. Cybiński, M. Płodzień, M. Tomza, M. Lewenstein, A. Dauphin, and A. Dawid, Github Repository with accompanying code (2023), accessed: 2024-06-07.
  * Su et al. [1979] W. P. Su, J. R. Schrieffer, and A. J. Heeger, Solitons in Polyacetylene, Phys. Rev. Lett. 42, 1698 (1979).
  * Yao et al. [2020] Z. Yao, A. Gholami, K. Keutzer, and M. Mahoney, PyHessian: neural networks through the lens of the Hessian, in 2020 IEEE International Conference on Big Data (2020) pp. 581–590.
  * Duan et al. [2021] R. Duan, X. Mao, A. K. Qin, Y. Chen, S. Ye, Y. He, and Y. Yang, Adversarial Laser Beam: Effective Physical-World Attack to DNNs in a Blink, in 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021) pp. 16057–16066.
  * Wang et al. [2023] Y. Wang, T. Sun, S. Li, X. Yuan, W. Ni, E. Hossain, and H. V. Poor, Adversarial attacks and defenses in machine learning-powered networks: A contemporary survey (2023), arXiv:2303.06302 [cs.LG] .
  * Ghorbani et al. [2019] A. Ghorbani, A. Abid, and J. Zou, Interpretation of neural networks is fragile, Proceedings of the AAAI Conference on Artificial Intelligence 33, 3681 (2019).
  * Buckman et al. [2018] J. Buckman, A. Roy, C. Raffel, and I. Goodfellow, Thermometer encoding: One hot way to resist adversarial examples, in 2018 International Conference on Learning Representations (ICLR) (2018).
  * Dombrowski et al. [2019] A.-K. Dombrowski, M. Alber, C. Anders, M. Ackermann, K.-R. Müller, and P. Kessel, Explanations can be manipulated and geometry is to blame, in Advances in Neural Information Processing Systems, Vol. 32, edited by H. Wallach, H. Larochelle, A. Beygelzimer, F. d. Alché-Buc, E. Fox, and R. Garnett (2019).
  * Dombrowski et al. [2022] A.-K. Dombrowski, C. J. Anders, K.-R. Müller, and P. Kessel, Towards robust explanations for deep neural networks, Pattern Recognition 121, 108194 (2022), publisher: Elsevier BV.
  * McInnes et al. [2020] L. McInnes, J. Healy, and J. Melville, UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction (2020), arXiv:1802.03426 [cs, stat].
  * Sainburg et al. [2021] T. Sainburg, L. McInnes, and T. Q. Gentner, Parametric (neural network) embedding (2021).
  * Cooley et al. [2019] S. M. Cooley, T. Hamilton, J. C. J. Ray, and E. J. Deeds, A novel metric reveals previously unrecognized distortion in dimensionality reduction of scRNA-seq data (2019).
  * Wattenberg et al. [2016] M. Wattenberg, F. Viégas, and I. Johnson, How to use t-SNE effectively, Distill 10.23915/distill.00002 (2016).
  * Chari and Pachter [2023] T. Chari and L. Pachter, The specious art of single-cell genomics, PLOS Comput. Biol. 19 (2023).