# Identifying Dominant Industrial Sectors in Market States of the S&P 500
Financial Data

Tobias Wand1,2,∗ Martin Heßler1,2 Oliver Kamps2

(1Westfälische Wilhelms-Universität Münster, Institut für Theoretische Physik,
Germany
2Westfälische Wilhelms-Universität Münster, Center for Nonlinear Science,
Germany
∗Corresponding author<EMAIL_ADDRESS>
August 2022, Revised March 2023 )

###### Abstract

Understanding and forecasting changing market conditions in complex economic
systems like the financial market is of great importance to various
stakeholders such as financial institutions and regulatory agencies. Based on
the finding that the dynamics of sector correlation matrices of the S&P 500
stock market can be described by a sequence of distinct states via a
clustering algorithm, we try to identify the industrial sectors dominating the
correlation structure of each state. For this purpose, we use a method from
Explainable Artificial Intelligence (XAI) on daily S&P 500 stock market data
from 1992 to 2012 to assign relevance scores to every feature of each data
point. To compare the significance of the features for the entire data set we
develop an aggregation procedure and apply a Bayesian change point analysis to
identify the most significant sector correlations. We show that the
correlation matrix of each state is dominated only by a few sector
correlations. Especially the energy and IT sector are identified as key
factors in determining the state of the economy. Additionally we show that a
reduced surrogate model, using only the eight sector correlations with the
highest XAI-relevance, can replicate $90\%$ of the cluster assignments. In
general our findings imply an additional dimension reduction of the dynamics
of the financial market.

##### Keywords:

Econophysics, Finance, Explainable AI, Clustering, Market States

##  1 Introduction

The economy is an example of a complex system with a large number of
interacting parts [29]. Its non-stationarity makes the analysis much more
difficult than for a stationary system. Nevertheless the time evolution of
many non-stationary complex systems can be described as a sequence of
transient states, i.e. the system passes through a sequence of states where it
remains for a short time, before transitioning to the next state. Since this
represents an enormous reduction of the dimensionality of the system, the
identification of these states might open the possibility for a better
prediction and control of the system. Examples can be found in such diverse
fields as e.g. neurodynamics [2] and fluid dynamics [8]. Since in many cases
an underlying mathematical description is not available, one has to rely on
data driven methods to find such states, e.g. via clustering or similar
methods [19, 13]. Due to the increasing availability of data for the economy
nowadays, such methods can also be applied in this field of research. In [39]
the authors tried to identify such states in the economy by analysing
correlation matrices of daily S&P 500 data. Generally, the correlation matrix
between the returns of the different stocks is an object of extensive research
[5, 24, 30, 31, 33, 41].
In [39, 43, 50] the correlation matrices of the daily S&P 500 data could be
sorted into eight clusters, which can be interpreted as states of the economy
corresponding to e.g. economic growth phases or crises [39] and show locally
stable behaviour [43, 50]. Further research showed that subtracting the
dominant market mode identified in [24, 41] from the correlation matrix leads
to a reduced-rank correlation matrix that highlights the differences between
different market states and allows to identify exogenous events, precursors
for crises and collectivity of industrial sectors [16, 15, 17]. An alternative
approach to the identification of market states is described in [32], where
the author uses the clustering algorithm given in [11]. While this algorithm
uses the correlation matrices between the observations that would be sorted
into a proposed clustering structure $s$ to infer the optimal parameters of
the clustering algorithm, the observables of interest are the returns of the
individual companies. Note that an elegant interpretation of the individual
market states is given in [32]. The states are interpreted regarding the mean
returns of the companies conditional on the trading day being in a specific
state $s$. Thus, among other things, the author finds that the Oil & Gas and
Petroleum sectors have a dominating influence for one of the temporal
clusters.
The correlation matrices are also the focus of observation in [39] where eight
market states are identified. In [39] the differences between the market
states are qualitatively discussed by comparing their centroids to each other
and found different regimes of correlation structure: some states correspond
to strong overall correlations throughout the market, whereas other states
exhibit weak correlations outside of companies from the same industry sector
or anticorrelations between some sectors. However, these comparisons were
purely qualitative via plotting and comparing heat maps of the clusters’
centroids and thus, there has not yet been a quantitative analysis of what
differentiates these market states from each other. In this article our goal
is to identify the dominant factors within the different economic states in
the framework of [39, 43, 50] by using a method [22] from the emerging field
of Explainable Artificial Intelligence (XAI) which can be understood as a
collection of methods that shed light on the black box that most AI methods
seem to be from the user’s perspective [36, 45]. XAI allows a quantitative
ranking of how much a given observable has influenced the decision algorithm
(here: clustering algorithm) and therefore gives us deeper insights into the
differences between the market states than the analysis in [39]. Most
importantly, this methodology can allow us to perform a dimensionality
reduction by identifying the most important sector correlations without any
qualitative ambiguity. XAI is nowadays considered for applications in physics
[23, 40, 14] and finance alike [3] and in particular, financial regulators
have already embraced the methods of XAI to increase transparency for
customers with regards to AI-supported business decisions of financial
institutions [9].

In [32] the market states are characterized and interpreted in terms of the
clustered returns, whereas we consider the correlations of different market
sectors. Such an analysis is similar to the qualitative analysis of the states
in [39, 43, 50], but XAI can give us a more detailed analysis: The observables
in [39, 43, 50] and in our study are the correlation matrices and therefore,
the clustered variables are the correlations between different sectors. These
correlations, however, are strongly interdependent; not only because of the
economic real-world situation that they model, but also because of
mathematical constraints of the correlation matrix (its eigenvalues are non-
negative, meaning that its matrix entries cannot be chosen independently from
each other; cf. e.g. [4] for the problems that can arise from this fact in
practical applications). The XAI algorithm from [22] is well-suited to deal
with such dependencies because of its holistic approach: it starts with the
prediction of the k-means algorithm and goes backwards through the algorithmic
structure until it arrives at the input layer, where it distributes the
prediction relevance on all input variables (here: correlations)
simultaneously. I.e. it does not look at an isolated input variable in a
vacuum, but always takes the values of other (potentially highly
interdependent) variables into account.

Here, based on the hypothesis that the dynamics of the financial market can be
described by passing through a sequence of a limited number of transient
states, we use the k-means clustering method to identify the clusters. While
most of the above clustering analyses use a hierarchical k-means clustering,
we use regular k-means clustering. This has a big advantage because it can be
reformulated to allow the use of XAI, which can analyse the decisions of the
clustering algorithm [22]. This allows us to reveal the dominant factors of
the different market states and to compare the market states quantitatively,
thereby improving the qualitative comparison depicted in figure 2 of [39].
Moreover, as shown later in the article, the XAI selection allows for a
dimensionality reduction of the system by selecting the most relevant
correlations.

The remainder of this article is structured as follows: Section 2 explains how
the S&P 500 data was collected and preprocessed so that it could be used for
our analysis. Section 3 introduces the clustering method and the XAI method
before interpreting the XAI results. Also, it discusses a reduced model which
only uses the features (i.e. sector-sector correlations) with the best XAI
values to learn the clustering classification to test the usefulness of XAI
analyses for reducing the dimensionality of the data. Finally, section 4
summarises the results, compares them to previous works and suggests follow-up
research. The appendix includes some additional information on the XAI
algorithm in appendix A, some additional depictions of results in further
figures in appendix B and technical details on neural networks in appendix C.

##  2 Data Preparation

The S&P 500 is a stock index containing 500 large US companies traded at
American stock exchanges. Daily stock data from these companies were
downloaded via the Python package yfinance [1] for the time period between
1992 and 2012, which is the same time period as in [43]. Only stock data of
companies that were part of the S&P 500 index during at least 99.5% of the
time were used for this analysis. Following the approach in [50], the data is
aggregated according to the 10 GICS (Global Industry Classification Standard)
sectors representing individual parts of the economy [20]. Stock prices from
companies of the same GICS sector were added to calculate representative time
series $(S_{t})_{t=1,\dots,T}$ for each of the 10 sectors by adding the price
time series of the companies in the same economic sector, which corresponds to
the price of buying one stock of each. This drastically reduces the number of
free parameters $N(N-1)/2$ in the $N\times N$ correlation matrix from an order
of $10^{4}$ to exactly $45$ to ensure that there is enough data to estimate
each correlation, i.e. that the estimation is not underdetermined. If a
company’s price time series is not available for the full time period, the
remaining 0.5% of the data are interpolated linearly with the .interpolate()
method in pandas [42, 35]. Overall, there are 249 companies’ time series for
5291 trading days. Each sector’s returns $R_{t}=(S_{t+1}-S_{t})/S_{t}$ are
locally normalised to remedy the impact of sudden changes in the drift of the
time series with the method introduced in [48] as

$r_{t}=\frac{R_{t}-\langle R_{t}\rangle_{n}}{\sqrt{\langle
R^{2}_{t}\rangle_{n}-\langle R_{t}\rangle^{2}_{n}}}.$ (1)

Here, $\langle\dots\rangle_{n}$ denotes a local mean across the $n$ most
recent data points, i.e. $r_{t}$ is subjected to a standard normalisation
transformation with respect to the local mean and standard deviation (i.e.
volatility $\sigma$). Following [39], $n=13$ was chosen for the daily data.
For each time step $t$ and each pair of sectors $i,j$ the local correlation
coefficients

$C_{i,j}=\frac{\langle r^{(i)}_{t}r^{(j)}_{t}\rangle_{\tau}-\langle
r^{(i)}_{t}\rangle_{\tau}\langle
r^{(j)}_{t}\rangle_{\tau}}{\sigma^{(i)}_{\tau}\sigma^{(j)}_{\tau}}$ (2)

are calculated over a time period of $\tau=40$ trading days like in [39] (the
40 working days correspond to two trading months) with the local standard
deviations $\sigma^{(i)}_{\tau}$. The window is shifted by one trading day
each, resulting in overlapping windows, and hence, the window to calculate
$\langle\dots\rangle_{n}$ in equation 1 is also shifted by one day. This
approach is similar to the sector analysis in [43], where the correlations
between the individual companies’ stock time series are averaged within their
GICS sectors. However, our procedure of first computing the GICS sector time
series and second, calculating the correlation ensures that larger companies
are given more weight in the resulting correlation matrix than smaller ones.
It has to be stressed that the GICS sector classification has changed over the
considered time period. E.g. when [43] was published, there were only 10 GICS
sectors, whereas nowadays, there are 11. During the data period from 1992 to
2012, the exact composition of sectors and subsectors has also changed which
poses additional problems in sorting a company into a sector. Here, the
historical data is sorted into the sectors according to the companies’ GICS
classification at the end of the time period in 2012, reflecting the same
sectors as the ones used in [39].

##  3 Identifying the most relevant sector correlations within a market state

In this section we first give an overview of the clustering method with
further technical details explained in the appendix A, before using XAI to
shed light on how the clustering algorithm arrives at its results. Then, the
XAI results are discussed and interpreted. Finally, a reduced surrogate model
that only uses the highest-XAI features is used to see whether the reduction
to features with high XAI preserves most of the clustering algorithm’s
information.

###  3.1 Identifying Market States via K-Means Clustering

Because the analyses [39, 43, 50] unanimously revealed eight market states for
the time period analysed in this article, we set the number of clusters for
the k-means algorithm to eight. In contrast to the procedure in the
literature, however, regular k-means clustering is used instead of a
hierarchical binary k-means clustering in order to make the cluster centres
amenable to the XAI method suggested in [22]. The regular k-means clustering
method starts with a predefined number of $k$ clusters and randomly selects
$k$ initial cluster centres before iteratively assigning each daily
correlation matrix to its nearest cluster centre and updating the centres by
setting them as the cluster’s centroid until their centroid positions converge
[28]. The market correlation matrix C belongs to cluster $j$, if for the
cluster centroids $(\textbf{C}_{l})_{l=1,\dots,k}$ the inequality

$||\textbf{C}-\textbf{C}_{j}||<\underset{l\neq
j}{\min}||\textbf{C}-\textbf{C}_{l}||$ (3)

holds for a suitable distance metric $||\dots||$. Here, the Euclidean distance
is used.

Figure 1: This figure shows which cluster state is occupied by the market at
which point in time. Individual data points are depicted with low opacity and
the slight overlap between adjacent points increases the opacity and may
appear like a solid line.

The time series of the resulting clusters is depicted in figure 1 and shows a
qualitatively similar behaviour to those in [39, 43, 50]. In the observed time
period, some clusters cease to appear more than sporadically in the data after
some years (clusters 0, 1 and 5) and some only start to appear more than
sporadically years after the start of the observation time period (cluster 6,
7 and again 5). However, figure 1 shows some quantitative differences compared
to the aforementioned literature, in particular because the clusters in figure
1 often appear isolated and for short time periods apart from their main
emergence interval. This may be attributed to the slightly different
clustering technique used for this article and the three clustering analyses
[39, 43, 50] also show differences in their plots of clusters against time.

###  3.2 Explainable AI for Clustering

Given the results for the clusters we now want to explore whether the clusters
are dominated by certain industrial sectors and, if this is the case, which
sectors are significant for the identification of the market states
represented by the clusters. To this end we use the method developed in [22]
to analyse k-means through the lens of XAI. Therefore two steps are necessary
[22]: First, the clustering classifier has to be neuralised, i.e. rewritten as
a neural network. Then, an XAI method for neural networks is used for
analysing the neuralised clustering algorithm, namely the Layer-wise Relevance
Propagation (LRP) [38]. For each instance (data point), the XAI algorithm
estimates how much the values of its features contributed to the neural
network’s decision for this instance. It answers the question which particular
feature values led to $\textbf{C}(t)$ being sorted into cluster $j$ instead of
any other cluster. The contribution of the $i^{\textmd{th}}$ feature is
quantified in a relevance score $\rho_{i}$ which is calculated according to
equations (2) and (4) from [22] for each instance with respect to its cluster
assignment $j$ for all features $i$. Appendix A gives further details on the
implementation of this algorithm.

####  3.2.1 XAI: From Local to Global Relevances

This method produces relevance scores $\rho_{i}$ for each feature $i$ (for the
correlation between sectors) for every instance (i.e. for each trading day),
as depicted in the inset of figure 2 for the first day of cluster 2. However,
we wish to focus on the aggregated relevance of one specific feature to
understand which sector correlations are more important for the market state
than others. As an example, figure 2 shows the histogram of XAI relevances for
the correlation Energy/Materials for all trading days in cluster 2. Are such
values particularly high? Does the correlation between Energy and Materials
matter more for cluster 2 than other correlations? To answer such a question,
one has to use the XAI values of each instance (local values) for a global
aggregation to compare the aggregated influence of the Energy/Materials
correlation with other correlations. Using the mean relevances of each sector
seems reasonable, but the mean is distorted by some instances with very high
absolute values for some of their relevances. Hence, one can instead focus on
the median of each feature’s relevance for each cluster’s instances.
Alternatively, one can identify each instance’s feature with the highest
$\rho_{i}$ and then concentrate on the features that are most frequently the
most important feature of an instance within each cluster. Because it regards
the statistical mode (i.e. the most frequently observed value) and calculates
the mode of the mode of each instance’s $\rho_{i}$, we chose to name this
method "mode-mode". The next sections addresses the question, which of these
aggregation methods is more suitable for the data.

Figure 2: Inset: Barplot of the XAI importance scores for the the first day in
cluster 2, showing how much each feature contributed to the classification.
The mode (feature with highest XAI score) is depicted in red. Main figure:
Histogram of the XAI importance of the correlation Energy/Materials for all
instances (trading days) in cluster 2. Outliers like the one visible on the
far left of this histogram distort the mean. The median value is depicted in
red.

####  3.2.2 Determining Relevant Correlations

For each individual cluster, the XAI relevances of each instance are
aggregated to analyse the importance of any feature for the clustering
decision via two methods: Either the median $\hat{\rho}_{i}$ of the
$i^{\textmd{th}}$ feature or the mode-mode method is used. The latter method
calculates how often $n_{i}$ the $i^{\textmd{th}}$ feature is the most
relevant feature of an instance and ranks the features according to their
value $n_{i}$. Both the $(\hat{\rho}_{i})_{i}$ and $(n_{i})_{i}$ values are
sorted ascendingly as shown in a sample plot in figure 3. These plots can be
approximated by piecewise linear segments and sometimes, they show a
characteristic elbow shape: They start at a long plateau of low values and
later experience a drastic increase, resulting in an almost rectangular shape
(cf. figure 3, right). This shape allows to divide the features into two
groups. Therefore, a change point analysis can identify the most drastic
change in the slope of those plots and help us to achieve a clear threshold
between important features with high $\hat{\rho}_{i}$ or $n_{i}$ and features
with little importance [26]. Here we use the Bayesian change point analysis
implemented in the antiCPy package [18] based on the ideas in [7].
Like depicted in figure 3, the mode-mode values usually show a much more
pronounced elbow that makes change point selection easier and their estimated
change point is often at a much higher feature number than for the median
method. Therefore, if features on the right-hand side of the change point are
to be interpreted as highly relevant features, then the mode-mode method
allows for a more trustworthy and more exhaustive feature reduction than the
median method.

Figure 3: Comparison of ascendingly plotting the features according to the
median values XAI (left) and mode-mode XAI (right) in blue and their
respective change point estimation with antiCPy in red. Plots for the other
clusters are included as supplementary material in the appendix B.

We use the mode-mode method to differentiate between relevant and irrelevant
correlations, because the approach usually leads to a more parsimonious
feature reduction (i.e. higher interpretability) and often much clearer change
points (cf. elbow-like relevance plots in 3). Note that a relevant correlation
is not necessarily unusually strong or weak, but rather particularly useful to
differentiate between this cluster and the others. As an example, figure 4
shows the centroid (left), the aggregated XAI values (centre) and the change
point selection of relevant features (right) for cluster 2. The relevant
features cannot be detected trivially by e.g. just identifying particularly
high or low values in the centroid, but follow from a comparison to the other
centroids. Hence, the change point analysis of relevant values does not follow
trivially from the cluster’s centroids, but uncovers new information about the
data and yields a parsimonious selection of relevant economic sector
correlations.
A plot of all relevant correlations for each cluster is shown in figure 5. The
correlations of the Energy sector appear in states associated with clusters 1
and 6, the Utilities sector in 1 and 4 and the Telecommunication sector in 3
and 4. The IT sector dominates cluster 5 , influences cluster 0 and, alongside
Telecommunication Services, also influences cluster 2. States in cluster 7
only have a high relevance for the correlations between Energy/Financial
Services and IT/Telecommunication. Other clusters show a high influence of
Utilities and Telecommunication Services, which are both tied to the Energy or
IT sector, respectively: the Utilities sector includes companies which
distribute gas and electricity, while Telecommunication Services are related
to the physical infrastructure of information transportation.

Figure 4: Visualisation of different matrices that represent cluster 2:
Comparison between the cluster’s centroid, the mode-based aggregated XAI
values and the binary antiCPy-selected relevant XAI values. Brighter colours
denote higher values. Because the diagonal’s correlation values are always 1,
their values have been set to the matrices’ averages. Figure 5: Relevant
correlations (yellow) of the sectors for the 8 clusters in figure 1. Relevance
selection was done via the change point analysis explained in section 3.2.1 on
the mode-mode XAI values. The sectors’ abbreviations are E: Energy, M:
Materials, F: Financial Services, CS: Consumer Staples, CD: Consumer
Discretionary, U: Utilities, T: Telecommunication Services, IT: Information
Technology, I: Industrials and HC: Health Care. Note that the diagonal has
been set to zero relevance by default because its correlations are always 1.

###  3.3 Dimensionality Reduction for all Clusters

It makes sense to assume that features with a high relevance should be
particularly useful for predicting the original classifier. In particular,
this might reveal that only few features are necessary to predict the majority
of the cluster assignments. If we only wish to classify one cluster vs. all
the others, then the relevant features selected by the cutoff in figure 3 are
a reasonable choice for reducing the dimensionality of the data. But for an
all vs. all classifier, choosing all relevant features of all clusters would
barely reduce the number of features at all, because clusters 1 and 4 (cf.
figures 9 and 11) would already each contribute more than ten features to this
list of relevant features. Instead, one can restrict the analysis to the
single best feature for each cluster, resulting in eight features instead of
the original 45. This also constitutes a second chance to compare the mode-
mode selection to the median selection. If a model trained on such a
dimensionality reduction predicts most of the cluster assignments correctly,
it will serve as a self-consistency check by indicating that features with
high XAI relevance are also highly useful for predicting.
To test, whether a dimensionality reduction is possible, we train a neural
network as a surrogate model that should predict the original clustering
decisions, but restrict its data to only some of the original features (cf.
[36, chapter 8.6]). For each of the eight clusters, only the feature that has
the highest XAI relevance score is used to train a neural network as a
surrogate model. To check, if this surrogate model has a high accuracy with
respect to the original classifier, we also train neural networks with eight
features randomly selected from the leftover features. Accuracy is defined as
the percentage of correctly predicted cluster labels. A short introduction to
neural networks and details on the architecture of the used neural network are
given in the appendix C. Training, test and validation data each contained
$\frac{1}{3}$ of the full data. The networks were trained on (a) the eight
optimal mode-mode features, (b) the eight optimal median features and (c)
eight randomly selected features. For each group 100 networks were trained
each of which was initialized with a different random seed to marginally
change the performances in the groups (a) and (b). Also, each network in group
(c) has a different selection of random features. This results in a
distribution of network accuracies for each of the three groups. The eight
correlations used for the mode-mode method are: E/M, E/F, E/HC, M/U, F/IT,
CS/IT, U/T and U/IT (cf. the caption of figure 5 for the abbreviations).
The accuracy of predicting the original clustering label out of the eight
clusters was chosen as a goodness criterion for the different feature
selections. The results are depicted in the kernel density plots in figure 6,
which are smoothened histograms of the 100 accuracy values for each neural
network. The random feature selection achieves noticeably worse results than
the two XAI based selection methods. While the mode-mode
($\mu_{mode}\pm\sigma_{mode}=0.895\pm 0.004$) is slightly better than the
median method ($\mu_{median}\pm\sigma_{median}=0.882\pm 0.005$), their
confidence intervals (CIs) overlap at the $2\sigma$ level. Therefore, one
cannot safely conclude that one of them is better than the other. However,
their CIs for one $\sigma$ do not overlap and the mode-mode criterion
possesses superior explainability for the individual clusters as demonstrated
in figure 3. Thus, it is recommended to use the mode-mode instead of the
median.

Figure 6: With the eight mode-mode features, the eight best median features
and eight random features, 100 neural networks were trained each with a
different seed. Kernel density estimations show that the mode-mode and median
networks achieve a much higher accuracy than the randomly selected suboptimal
features.

##  4 Discussion

The goal of the research presented in this article was to answer the question
whether it is possible to identify sector correlations that dominate the
assignment of a trading day to a market state. To this end we used a
clustering procedure that allows for an interpretation by a method from the
field of XAI. We designed an aggregation procedure to combine these local
explanations to global feature relevances for the entire data. The resulting
relevance curves show pronounced elbow plots for the mode-mode aggregation
like in figure 3 which can be analysed with a change point identification
method to define cutoffs between irrelevant and relevant features.

Using the described methodology it is possible to show that the assignment of
a certain trading day of the S&P 500 market to a cluster (i.e. to a market
state) is dominated by only a few sector correlations. The correlations
involving the Energy or IT sector each make up three of the eight best
features selected by the mode-mode XAI aggregation, respectively, but E/IT is
not one of the eight best mode-mode features. Note that in figure 5, clusters
1, 2, 5 and 6 also show a high influence of IT and/or Energy. The frequent
occurrence of the IT sector among the highly relevant correlations may be
traced back to the fact that the observation period includes the build-up, the
emergence and burst of the dot-com bubble which saw the rise and rapid decline
of many IT companies. This is also reflected in the analyses in [39] and [50].
The high importance of the energy sector implies that this sector has major
influence in determining whether the economy is in a state of crisis or
growth, probably because all other economic activity requires energy for e.g.
transportation or production. While the high importance of the IT sector might
have to be attributed to the dot-com bubble, the continued influence of the
energy sector on the economic state might be a more reasonable conclusion from
this analysis. These results support previous studies that have identified the
energy sector as a key influence in economic activity [12, 21, 27].
We also showed that a neural network which is only trained on the features
with the highest XAI relevance scores can predict the trading day’s cluster
with a high degree of accuracy. In general our results imply that a further
dimensionality reduction for the description of the dynamics of the financial
market is possible. This can be particularly useful in high frequency trading
applications, where the time span of interest lies within the life expectancy
of a cluster. In this case, when data needs to be processed very quickly, it
may be too slow to work with the complete correlation matrix. Nevertheless,
our findings suggest that a reduced matrix only containing the most
informative correlations for this cluster can lower the computation times
without sacrificing too much accuracy. Note that the quantification of each
feature’s relevance was only possible due to the nature of the used XAI
algorithm. The other discussed approaches could perform a dimensionality
reduction only under the significantly higher effort of a less objective,
since qualitative attempt which can lead to ambiguous results.

###  4.1 Comparison to other Clustering Algorithms

Our analysis follows the work in [39, 43, 50] and treats the correlation
matrix of the returns as the observable that is to be sorted into clusters
(market states) via k-means. A different approach to the identification of
market states is given in [32], where the clustering algorithm in the maximum-
likelihood-framework of [11] is used. The author of [32] not only uses the
clustering algorithm to identify market states, but also to identify economic
sectors. These show strong overlap with the sectors in the SIC classification
which have been used as industry standards and recently been replaced by the
GICS sectors. Because of this strong overlap, we use the GICS classification
as a starting point for our analysis. In [32], daily prices from 1990 to 1999
have been analysed, whereas our work uses data from 1992 to 2012 in line with
[43, 50] and therefore has more recent data. While [32] gauges the importance
of business sectors for the market states by showing selected scatter plots of
the companies’ average returns for two different market states, we use an all-
vs.-all comparison of all market states with the methodology of XAI. Despite
the increased time frame and the different approach to clustering, our
analysis confirms some of the key findings in [32], namely the pronounced role
of the High-Tech companies (roughly corresponding to the IT sector in our
analysis) and the sectors of Oil & Gas and Petroleum companies (corresponding
to the Energy sector). While [32] also finds a peculiar behaviour of the Gold
& Silver mining companies, our analysis does not have such a high resolution
of detailed business sectors and instead those companies are part of the
Materials sector. While [32] does not find any peculiar behaviour for the
Finance sector (or Commercial Banks in [32]), the Finance sector appears twice
among our top 8 correlations, indicating an above average importance. This
probably corresponds to the fact that our data, unlike the time interval used
in [32], includes the 2008 financial crisis.
An interesting difference between our study and the analysis of market states
in [39] can be found by regarding the last state (8 in [39] and 7 in our
article): the qualitative state comparison in [39] simply finds an overall
high degree of correlation across the different sectors during this state,
whereas our analysis still identifies that the two sector correlations
Energy/Finance and Utilities/IT (cf. figure 4) have a dominant effect on this
cluster. The identification of those two dominant correlations was only
possible due to the quantitative approach of the XAI methodology.

###  4.2 Outlook

While this article used a standard k-means clustering algorithm and therefore
deviated from the hierarchical clustering in [39, 43, 50], this opens up
another possibility for follow-up research by combining these methods: using a
hierarchical clustering and using the XAI on each split in the hierarchy may
increase our understanding even further via XAI as an addition to the
intrinsic interpretability of hierarchical algorithms. This would potentially
tie together the explainability of XAI with the paradigm of interpretable
models presented in [44].
Despite the insightful analyses of financial correlation matrices in [39, 43,
50] and in this article, one has to keep in mind that the analysis of the
states is based on the matrix of Pearson’s correlation coefficients between
financial time series. Although this method is frequently used in financial
research [31, 24, 41, 33], it has been subject to criticism: As noted in [46],
Pearson’s correlation only provides inference on linear correlation, but not
on nonlinear correlations. This can also be seen in the stylised fact that
returns are uncorrelated, but squared returns are strongly correlated [29].
Instead, Spearman’s correlation may provide another view on correlations and
include nonlinear effects [47] and therefore can be used as another starting
point for follow-up research. Additionally, as already mentioned in section 2,
the GICS classification has been updated over time to reflect changes in the
economy’s structure. Because these updates also happened during the regarded
20 years time period, they may have affected the results of this analysis.
Moreover, there is a more subtle problem about the data that can be referred
to as a survivorship bias: Like [39, 43, 50], we only regard companies whose
stock prices are available for the entire time period. Companies that went
bankrupt or were merged into another company during the time period are
therefore completely disregarded by these analyses, although their
bankruptcies or takeovers certainly contain valuable information about the
state of the economy.

## Acknowledgements

Tobias Wand thanks Shanshan Wang, Henrik Bette (both University of Duisburg-
Essen) and Katrin Schmietendorf (WWU Münster) and the anonymous referees for
their valuable suggestions. Tobias Wand and Martin Heßler are funded by the
Studienstiftung des deutschen Volkes.

## References

  * [1] Ran Aroussi  “yfinance 0.1.70”, 2022
  * [2] Peter Ashwin and Marc Timme  “When instability makes sense”  In _Nature_ 436.7047, 2005, pp. 36–37
  * [3] Martin Berg and Ouren Kuiper  “XAI in the Financial Sector”, 2020
  * [4] Walter Böhm and Kurt Hornik  “Generating random correlation matrices by the simple rejection method: Why it does not work”  In _Statistics & Probability Letters_ 87, 2014, pp. 27–30
  * [5] Stephen J. Brown  “The Number of Factors in Security Returns”  In _The Journal of Finance_ 44.5 Wiley, 1989, pp. 1247–1262
  * [6] Francois Chollet  “Keras”  GitHub, 2015
  * [7] Volker Dose and Annette Menzel  “Bayesian analysis of climate change impacts in phenology”  In _Global Change Biology_ 10.2 Wiley, 2004, pp. 259–272
  * [8] Daniel Fernex, Bernd R. Noack and Richard Semaan  “Cluster-based network modeling—From snapshots to complex dynamical systems”  In _Science Advances_ 7.25 American Association for the Advancement of Science (AAAS), 2021
  * [9] BaFin - Federal Agency Financial Services Supervision  “Big data meets artificial intelligence”, 2018
  * [10] Aurelien Geron  “Hands-on machine learning with scikit-learn, keras, and TensorFlow”  Sebastopol, CA: O’Reilly Media, 2019
  * [11] Lorenzo Giada and Matteo Marsili  “Algorithms of maximum likelihood data clustering with applications”  In _Physica A: Statistical Mechanics and its Applications_ 315.3, 2002, pp. 650–664
  * [12] Giray Gozgor, Chi Keung Marco Lau and Zhou Lu  “Energy consumption and economic growth: New evidence from the OECD countries”  In _Energy_ 153, 2018, pp. 27–34
  * [13] Peter beim Graben and Axel Hutt  “Detecting Recurrence Domains of Dynamical Systems by Symbolic Dynamics”  In _Phys. Rev. Lett._ 110 American Physical Society, 2013, pp. 154101
  * [14] Christophe Grojean, Ayan Paul, Zhuoni Qian and Inga Strümke  “Lessons on interpretable machine learning from particle physics”  In _Nature Reviews Physics_ 4.5 Springer ScienceBusiness Media LLC, 2022, pp. 284–286
  * [15] Anton J Heckens and Thomas Guhr  “A new attempt to identify long-term precursors for endogenous financial crises in the market correlation structures”  In _Journal of Statistical Mechanics: Theory and Experiment_ 2022.4 IOP Publishing, 2022, pp. 043401
  * [16] Anton J Heckens, Sebastian M Krause and Thomas Guhr  “Uncovering the dynamics of correlation structures relative to the collective market motion”  In _Journal of Statistical Mechanics: Theory and Experiment_ 2020.10 IOP Publishing, 2020, pp. 103402
  * [17] Anton J. Heckens and Thomas Guhr  “New collectivity measures for financial covariances and correlations”  In _Physica A: Statistical Mechanics and its Applications_ 604, 2022, pp. 127704
  * [18] Martin Heßler and Oliver Kamps  “Bayesian on-line anticipation of critical transitions”  In _New Journal of Physics_ 24.6 IOP Publishing, 2022, pp. 063021
  * [19] Axel Hutt, Markus Svensen, F Kruggel and R Friedrich  “Detection of fixed points in spatiotemporal signals by clustering method”  In _Physical Review. E_ 61, 2000, pp. R4691–3
  * [20] S&P Dow Jones Indices  “Sector Classification” [Online; accessed 3 March 2022], https://www.msci.com/documents/10199/4547797/Effective+Until+February+28,+2014.xls, 2018
  * [21] Rafał Kasperowicz  “Electricity Consumption and Economic Growth: Evidence from Poland”  In _Journal of International Studies_ 1, 2014, pp. 46–57
  * [22] Jacob Kauffmann et al.  “From Clustering to Cluster Explanations via Neural Networks”  In _IEEE Transactions on Neural Networks and Learning Systems_ , 2022, pp. 1–15
  * [23] Arnd Koeppe et al.  “Explainable Artificial Intelligence for Mechanics: Physics-Explaining Neural Networks for Constitutive Models”  In _Frontiers in Materials_ 8, 2022
  * [24] Laurent Laloux, Pierre Cizeau, Jean-Philippe Bouchaud and Marc Potters  “Noise Dressing of Financial Correlation Matrices”  In _Phys. Rev. Lett._ 83 American Physical Society, 1999, pp. 1467–1470
  * [25] Will Landecker et al.  “Interpreting individual classifications of hierarchical networks”  In _2013 IEEE Symposium on Computational Intelligence and Data Mining (CIDM)_ , 2013, pp. 32–38
  * [26] Wolfgang Linden, Volker Dose and Udo Toussaint  “Bayesian Probability Theory”  Cambridge University Press, 2014
  * [27] Wen-Cheng Lu  “Electricity Consumption and Economic Growth: Evidence from 17 Taiwanese Industries”  In _Sustainability_ 9.1, 2017
  * [28] J. MacQueen  “Some methods for classification and analysis of multivariate observations”  In _Proc. of the Fifth Berkeley Symposium on Mathematical Statistics and Probability_ 5.1, 1967, pp. 281–297
  * [29] Rosario Mantegna and H. Stanley  “An Introduction to Econophysics”  Cambridge University Press, 2000
  * [30] Rosario Nunzio Mantegna  “Degree of correlation inside a financial market”  In _AIP Conference Proceedings_ 411.1, 1997, pp. 197–202
  * [31] Harry Markowitz  “Portfolio Selection”  In _The Journal of Finance_ 7.1 JSTOR, 1952, pp. 77
  * [32] Matteo Marsili  “Dissecting financial markets: sectors and states”  In _Quantitative Finance_ 2.4 Routledge, 2002, pp. 297–302
  * [33] Gautier Marti, Frank Nielsen, Mikołaj Bińkowski and Philippe Donnat  “A Review of Two Decades of Correlations, Hierarchies, Networks and Clustering in Financial Markets”  In _Progress in Information Geometry: Theory and Applications_ Springer International Publishing, 2021, pp. 245–274
  * [34] Martín Abadi et al.  “TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems” Software available from tensorflow.org, 2015
  * [35] Wes McKinney  “Data Structures for Statistical Computing in Python”  In _Proceedings of the 9th Python in Science Conference_ , 2010, pp. 56–61
  * [36] Christoph Molnar  “Interpretable Machine Learning”, 2022
  * [37] Grégoire Montavon, Wojciech Samek and Klaus-Robert Müller  “Methods for interpreting and understanding deep neural networks”  In _Digital Signal Processing_ 73, 2018, pp. 1–15
  * [38] Grégoire Montavon et al.  “Layer-Wise Relevance Propagation: An Overview”  In _Explainable AI. Interpreting, Explaining and Visualizing Deep Learning_ Springer International Publishing, 2019, pp. 193–209
  * [39] Michael C. Münnix et al.  “Identifying States of a Financial Market”  In _Scientific Reports_ 2.1 Springer ScienceBusiness Media LLC, 2012
  * [40] Mark S. Neubauer and Avik Roy  “Explainable AI for High Energy Physics”  In _arXiv_ , 2022  DOI: 10.48550/arxiv.2206.06632
  * [41] Vasiliki Plerou et al.  “Universal and Nonuniversal Properties of Cross Correlations in Financial Time Series”  In _Phys. Rev. Lett._ 83 American Physical Society, 1999, pp. 1471–1474
  * [42] Jeff Reback et al.  “pandas-dev/pandas: Pandas 1.4.3”  Zenodo, 2022
  * [43] Philip Rinn et al.  “Dynamics of quasi-stationary systems: Finance as an example”  In _EPL (Europhysics Letters)_ 110.6 IOP Publishing, 2015, pp. 68003
  * [44] Cynthia Rudin  “Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead”  In _Nature Machine Intelligence_ 1.5 Springer ScienceBusiness Media LLC, 2019, pp. 206–215
  * [45] “Explainable AI”  Springer International Publishing, 2019
  * [46] Leonidas Sandoval  “A map of the Brazilian stock market”  In _Advances in Complex Systems_ 15.05, 2012, pp. 1250042
  * [47] Leonidas Sandoval and Italo De Paula Franca  “Correlation of financial markets in times of crisis”  In _Physica A_ 391.1, 2012, pp. 187–208
  * [48] Rudi Schäfer and Thomas Guhr  “Local normalization. Uncovering correlations in non-stationary financial time series”  In _Physica A_ 389.18, 2010, pp. 3856–3865
  * [49] Avanti Shrikumar, Peyton Greenside and Anshul Kundaje  “Learning Important Features Through Propagating Activation Differences”  In _Proceedings of the 34th International Conference on Machine Learning_ 70, Proceedings of Machine Learning Research  PMLR, 2017, pp. 3145–3153
  * [50] Yuriy Stepanov et al.  “Stability and hierarchy of quasi-stationary states: financial markets as an example”  In _Journal of Statistical Mechanics: Theory and Experiment_ 2015.8 IOP Publishing, 2015, pp. P08011
  * [51] Jianming Zhang et al.  “Top-Down Neural Attention by Excitation Backprop”  In _Int. J. Comput. Vision_ 126.10 USA: Kluwer Academic Publishers, 2018, pp. 1084–1102

##  Appendix A Appendix: Details on Layer-Wise Relevance Propagation for
K-Means

These sections give a more detailed explanation of how to reformulate the
k-means clustering algorithm and how to use the Layer-wise Relevance
Propagation as described in [22].

###  A.1 Neuralisation of K-Means

It has been proven in the supplementary materials of [22] that the k-means
decision criterion (3) of cluster $j$ vs. the other clusters $l\neq j$ can be
rewritten in two layers and a third decision layer. This procedure is
described by algorithm 1. Note that while the k-means classifier tests all
clusters against each other, the neuralised classifier only tests if an
instance belongs to a specific cluster $j$ versus all other clusters $l\neq
j$. If the instance does not belong to the $j^{\textmd{th}}$ cluster,
algorithm 1 does not say which of the other clusters is correct. Therefore,
one needs as many neuralised classifiers as there are clusters to fully
neuralise equation (3).

Algorithm 1 Neuralisation of a K-Means Classifier

1:Goal: Test assignment of C to cluster $j$ against all alternative clusters
$l\neq j$.

2:Input: Instance C and cluster centroids $(\textbf{C}_{l})_{l}$

3:Procedure in three layers:

4:Calculate $\forall l\neq j$ $h_{l}=\textbf{w}_{l}\cdot\textbf{C}+b_{l}$ with
$\textbf{w}_{l}=2(\textbf{C}_{j}-\textbf{C}_{l})$ and
$b_{l}=||\textbf{C}_{l}||^{2}-||\textbf{C}_{j}||^{2}$

5:Calculate $f_{j}=\min_{l\neq j}h_{l}$

6:assign C to cluster $j$, if $f_{j}>0$

###  A.2 Layer-wise Relevance Propagation

Layer-wise Relevance Propagation is an XAI method that runs backwards through
a neural network. At each step, it calculates the relevance that the current
neuron has for the deeper layer and distributes it accordingly to the next
higher layer’s neurons. A conservation property ensures that the combined
relevance is preserved while being propagated to the higher layers and is
illustrated in figure 7. According to [38], this reasoning is analogous to
Kirchhoff’s laws in electrical circuits and used similarly in other XAI
techniques such as [25, 49, 51].
For the neuralised k-means algorithm 1, [22] suggests the following
propagation rule of relevance $\rho_{l}$ from $f_{j}$ backwards to $h_{l}$:

$\rho_{l}=\frac{\exp(-\beta h_{l})}{\sum_{l\neq j}\exp(-\beta h_{l})}f_{j}$
(4)

with the inverse mean $\beta=\mathbb{E}[f_{j}]^{-1}$ of $f_{j}$ over the
entire data. The second rule for the propagation of relevance $\rho_{l}$ of
each node $h_{l}$ to the original input features $c_{i}$ of C is given by

$\rho_{i}=\sum_{l\neq
j}\frac{(c_{i}-m_{i,l})w_{i,l}}{\sum_{i}(x_{i}-m_{i,l})w_{i,l}}\rho_{l}$ (5)

with $\textbf{m}_{l}=(\textbf{C}_{j}+\textbf{C}_{l})/2$ as the point directly
between the centroids and $w_{i,l}$ as the $i^{\text{th}}$ component of the
difference vector $\textbf{w}_{l}$ between the cluster centroids $j$ and $l$
(cf. the pseudocode in 1). Note that in our study, each $c_{i}$ is a
correlation like in equation (2).

Figure 7: Illustration of the Layer-wise Relevance Propagation, taken from
[37] and adapted under a CC BY 4.0 license
(https://creativecommons.org/licenses/by/4.0/) to stay consistent with our
article’s notation.

##  Appendix B Appendix: Elbow Plots

The same plots as in figure 3 are depicted in this section for the other seven
clusters. While a pronounced elbow is visible for all clusters in the mode-
mode aggregation, the median method often shows a continuum.

Figure 8: The same figures as in figure 3, but now for cluster 0. Figure 9:
The same figures as in figure 3, but now for cluster 1. Figure 10: The same
figures as in figure 3, but now for cluster 3. Figure 11: The same figures as
in figure 3, but now for cluster 4. Figure 12: The same figures as in figure
3, but now for cluster 5. Figure 13: The same figures as in figure 3, but now
for cluster 6. Figure 14: The same figures as in figure 3, but now for cluster
7.

##  Appendix C Appendix: Neural Network Architecture

The neural network used as a surrogate model in this article is a deep multi-
layer perceptron network for a classification task to predict which of the
eight clusters the instance belongs to. The architecture of such a network is
as follows: The basic unit of the network is a neuron or perceptron, which
gets an input $z\in\mathbb{R}$ and produces an output $u\in\mathbb{R}$ via an
activation function $f(z)=u$. Motivated by biological neurons, the activation
functions often only produce a significantly nonzero output (the neuron
"fires"), if the input is larger than some threshold. Common choices for
activation functions are given by

$\displaystyle\text{relu}(z)=\max(0,z)\hskip
28.45274pt\text{LeakyReLU}_{\alpha}(z)=\max(\alpha z,z)\textmd{ with free
parameter }\alpha$ (6)
$\displaystyle\text{selu}_{(s,a)}(z)=sz\mathbbm{1}_{z\leq
0}+sa(\exp{(z)}-1)\mathbbm{1}_{z<0}\textmd{ with }a=1.67326324\textmd{ and
}s=1.05070098\textmd{.}$

The neurons are organised in layers where the output of the $N_{n}$ neurons in
layer $n$ is used as inputs for the $N_{n+1}$ neurons in the $n+1^{\text{th}}$
layer. If the connections are dense, then all $N_{n}$ previous outputs are
combined in linear combinations to contribute to the next layer’s inputs. The
weights of the linear combinations are then optimised to achieve the best
accuracy. If the connections are in a dropout layer of rate $\beta$, then at
every optimisation step, $\beta$ of the connections are forcibly set to $0$ to
prevent an overfitting of the network (i.e. that the network is fit so closely
to the known data that it does not generalise well on unknown data).
The very first layer (input layer) receives the features (observables) of the
data and, in this case, the output layer is giving a prediction about which of
the 8 clusters the data belongs to.
The softmax function is often used to get from the continuous values of the
neurons’ outputs to an actual prediction. If the network has to choose between
$k$ possible predictions, then it will end with $k$ linear combinations
$l_{1},\dots,l_{k}$ of the neurons’ outputs in its penultimate layer. For the
$k$ possible classes it transforms these linear combinations into
probabilities $p_{1},\dots,p_{k}$ via the softmax function following

$\textmd{softmax}(l_{i})=\frac{\exp{(l_{i})}}{\sum_{i}\exp{(l_{i})}}.$

Note that $\sum_{i}p_{i}=1$. Finally, it uses the highest probability as its
prediction. Further details on the construction of neural networks can be
found e.g. in [10] and technical details on the implementation of the neural
network are given in [6, 34]. The network architecture used in this article is
detailed in table 1.

Layer | Dense | Dense | Dense | Dense | Dense | Dropout | Dense
---|---|---|---|---|---|---|---
Units | 256 | 128 | 128 | 1024 | 128 | - | 8
Activation | selu | relu | relu | LeakyReLu | LeakyReLu | - | softmax
Parameter | - | - | - | 0.05 | 0.01 | 0.3 | -
Table 1: Architecture of the neural network with 100 training epochs for the
surrogate model in section 3.2 from left to right. There are eight input units
for the eight features of the surrogate model and eight output units for the
eight clusters.