# Coarse graining correlation matrices according to macrostructures:
Financial markets as a paradigm

M. Mijaíl Martínez-Ramos  Instituto de Ciencias Físicas, Universidad Nacional
Autónoma de México, 62210 Cuernavaca, México  Parisa Majari  Instituto de
Ciencias Físicas, Universidad Nacional Autónoma de México, 62210 Cuernavaca,
México  Andres R. Cruz-Hernández  Instituto de Ciencias Físicas, Universidad
Nacional Autónoma de México, 62210 Cuernavaca, México  School of Management,
Universidad de los Andes, 111711, Bogotá, Colombia  Hirdesh K. Pharasi  School
of Engineering and Technology, BML Munjal University, Gurugram, Haryana
122413, India  Manan Vyas  Corresponding author<EMAIL_ADDRESS>Instituto
de Ciencias Físicas, Universidad Nacional Autónoma de México, 62210
Cuernavaca, México

###### Abstract

We analyze correlation structures in financial markets by coarse graining the
Pearson correlation matrices according to market sectors to obtain Guhr
matrices using Guhr’s correlation method according to Ref. [P. Rinn et. al.,
Europhysics Letters 110, 68003 (2015)]. We compare the results for the
evolution of market states and the corresponding transition matrices with
those obtained using Pearson correlation matrices. The behavior of market
states is found to be similar for both the coarse grained and Pearson
matrices. However, the number of relevant variables is reduced by orders of
magnitude.

Complex systems, Correlations, $k$-means clustering, Coarse graining,
Transition matrices, Multi dimensional scaling

##  I Introduction

Easy availability of financial markets data makes these ideal to explore new
aspects of complex systems. About 25 years ago, it was recognized that the
state of a financial market is largely determined by largest eigenvalue of the
correlation matrix of returns of the closing prices [1]. In 2012, the concept
of discrete ‘market states’ was introduced based on a clustering algorithm
applied in the space of correlation matrices [2] and has received considerable
attention in and outside financial markets [3, 4, 5, 6, 7, 8, 9, 10, 11, 12,
13, 14]. Note that the concept of ‘market states’ is different from ‘state of
the market’, which is defined by the largest eigenvalue [2]. This concept is
one option of what is often referred to as ‘market regimes’ by economists [11,
15], but the latter may use a wider range of parameters. The correlation
matrix has proven valuable in characterizing various aspects of stock returns
[16, 17], stock index returns and global stock returns [18], and stock return
volatility [19]. Precisely estimating the properties of correlation matrices
is crucial for a range of financial decision-making processes such as asset
allocation, portfolio optimization [20], and derivative pricing [21].

In Refs. [6, 8, 7], it was found that a $k$-means (in what follows “KM”)
clustering calculation produces results that have a marked dependence on the
average correlations or the largest eigenvalue of the Pearson correlation
matrix as can be expected from the standard literature [1, 22] 111 Indeed the
correlation of these two quantities over the time horizon from 2006 to 2023 is
above 0.9 [23]. Yet we see clear differences in the S&P 500 data for similar
average correlations in two and three-dimensional representations obtained by
dimensional scaling [24, 7, 6, 8, 25]. These results produce transition
matrices between states that have given a very reasonable account of states
where a crash may be imminent [8]. It is not far fetched to suspect that
correlations of sectors could give similar results with smaller effort and
possibly even clearer signals.

The simple-minded approach of averaging over the returns of stocks in each
sector is not satisfactory for a variety of reasons. Fortunately in Ref. [26]
a coarse-graining (in what follows “CG”) of the correlation was proposed,
where we average correlation matrix elements over sectorial blocks providing
sort of a covariance matrix over the market sectors. This matrix, which we
shall call the ‘sectorial Guhr matrix’, provides the appropriate tool for our
endeavor.

In the present paper, we shall analyze S$\&$P 500 and Nikkei 225 using the
stocks that exist over our time horizon, which we choose identical to Refs.
[8, 25]. We will thus repeat the analysis of Ref [8] in some detail and
compare the results of noise reduction by a power map (hereinafter “PM”) [27,
28, 29], and KM [30, 31] with the corresponding results for the sectorial Guhr
matrices. We also plot the dimensionally scaled matrices for visualization
purposes. Indeed, if we think of parametrizing the correlation matrix, for
example if we have $N=350$ stocks for S$\&$P 500 market (after purification of
the time series), we obtain 350 time series and thus a $350\times 350$
correlation matrix: due to the symmetry of the matrix and the diagonal
consisting of ones we have 61075 parameters [$N(N-1)/2$ parameters], which is
not very pleasant. The $10\times 10$ sectorial Guhr matrix for market sectors
will have $55$ parameters, which is not great but provides a dramatic
reduction.

In the next section, we shall detail the techniques and describe the data sets
we use, following ideas outlined in [6, 7, 8]. In section II, we will explain
the technique of Guhr et. al. [26] to obtain the sectorial Guhr matrices from
the Pearson correlation matrices. We shall construct the correlation matrices
as a function of the trading days with epoch lengths of 20 and one day shift
of the epoch for the daily closing price returns. Note that the closing prices
data are used as some random walk properties are expected for intra-day
trading and opening data seem to be more volatile than closing data; also they
don’t reflect the same degree of the decisions of investors/investment groups
that are based on fundamental data. We will then apply the clustering
technique. In section III, we show the corresponding results for S&P 500 and
Nikkei 225 with time horizon of January 2006 to December 2019. Note that we
exclude COVID period as there is a significant change in the structure of
market states observed recently [23], with an entirely new state appearing.
Yet only time can tell, if this is a change in the structure of the market or
a transitory single occurrence that is tapering off. We finally will present
conclusions and an outlook of possible extensions in the methods and in the
applications.

##  II Data and techniques used

We use the daily adjusted closing price $P_{i}=[p_{1},p_{2},...,p_{T}]$ of
$i$-th stock of S$\&$P 500 (USA) and Nikkei 225 indices over the period from
January 2006 to December 2019. Here $i=1,2,...,N$; N denotes the number of
stocks and $T$ is the time horizon. The S$\&$P 500 market consists of $N=350$
number of stocks over $T=3523$ trading days and the Nikkei 225 consists of
$N=150$ stocks over $T=3458$ trading days. Note that here we choose the same
time horizon as Ref. [8]. Note that $N$ is usually smaller than the number of
listed stocks as we use the criterion to exclude stocks that have not been
traded for more than two consecutive days during the chosen time horizon.

Using the time series for $P_{i}$, we compute time series for logarithmic
returns $r_{i}(t)=\ln p_{i}(t+\Delta t)-\ln p_{i}(t)$; $t=1,2,\ldots,T-1$.
Then, the Pearson correlation $C_{ij}(\tau)$ between stocks $i$ and $j$ is
defined as

$C_{ij}(\tau)={[r_{i}-\langle r_{i}\rangle][r_{j}-\langle
r_{j}\rangle]}/{\sqrt{\mbox{var}(r_{i})\;\mbox{var}(r_{j})}}\;;$ (1)

here, the symbol $\langle\ldots\rangle$ denotes average and var$(\ldots)$ is
the variance of the return time series for each epoch:
$\mbox{var}(r_{i})=\\{\sum_{j=\tau_{t}}^{\tau_{t^{\prime}}}r_{i,j}\\}/(\tau_{t^{\prime}}-\tau_{t}+1)$.
In Eq. (1), $\tau$ is the end date of the epoch which we shall choose as $20$
trading days, i.e. approximately one month. To obtain the time evolution we
shift the epochs by one day over the time horizon. Thus, we have $M=3503$ and
$3438$ correlations matrices for S$\&$P 500 and Nikkei 225, respectively.

Table 1: Sectors in S&P 500 market Label | Sectors
---|---
CD | Consumer Discretionary
CS | Consumer Staples
EG | Energy
FN | Financials
HC | Health Care
ID | Industrials
IT | Information Technology
MT | Materials
TC | Technology
UT | Utilities
Table 2: Sectors in Nikkei 225 market Label | Sectors
---|---
CG | Consumer Goods
CP | Capital Goods/Others
FN | Financials
MT | Materials
TC | Technology
UT | Transportation $\&$ Utilities

We apply the PM method as a noise suppression technique [27, 28, 29] on
Pearson correlation matrices using the following definition:

$C^{\prime}_{ij}(\tau)=sign(C_{ij})|C_{ij}|^{1+\epsilon}$ (2)

where $0<\epsilon<1$ is the noise-suppression parameter. Note that the
applying PM method on the CG matrices gives similar results as if we apply PM
method on the correlation matrices and then obtain the CG matrices.

Figure 1: Correlation matrix plot for the total time horizon considered
(January 2006 to December 2019) with (a) Pearson correlation and (b) sectorial
Guhr matrix for S&P 500. The market sectors (see Table 1) for the Guhr
matrices are not scaled according to the number of stocks in the particular
sector. Note that no negative correlations survive over the total time
horizon.

Figure 2: Same as Fig. 1 but for Nikkei 225. The market sectors (see Table 2)
for the Guhr matrices are not scaled according to the number of stocks in the
particular sector.

The stocks in the S&P 500 and Nikkei 225 markets are classified into
$N_{s}=10$ and $6$ sectors, respectively, as shown in tables 1 and 2. We then
obtain the ${N_{s}\times N_{s}}$ Guhr matrices by coarse graining Pearson
correlation matrices taking the average over different intra- and inter-
sectorial correlation matrix elements. Considering each intra- and inter-
sectorial blocks as shown in Figs. 1 and 2, we sum all the correlation matrix
elements inside the block (excluding the self-correlations) and divide by the
total number of correlation matrix elements $m$ in the block,

$G_{ij}=\displaystyle\frac{1}{m}\displaystyle\sum_{\alpha,\beta}C_{\alpha,\beta}\;\;\;\;\;\;\forall\;\;(\alpha\in
i,\beta\in j)\;.$ (3)

Note that the Guhr matrix $G$ is not a correlation matrix as its diagonal
matrix elements are not identity. An important feature of the Guhr matrices is
that we deal with low dimensional space $N_{s}(N_{s}+1)/2$, i.e., $55\;D$ for
S$\&$P 500, and $21\;D$ for Nikkei 225 markets which is much smaller than the
original $N(N-1)/2\;D$ one, which is not great but provides a dramatic
reduction. Figure 1 shows the correlation matrix plot for (a) Pearson and (b)
Guhr matrices for S&P 500 market over the total time horizon. Similarly, Fig.
2 shows the correlation matrix plot for (a) Pearson and (b) Guhr matrices for
Nikkei 225 market over the total time horizon. Note that the sectors for the
Guhr matrices are not scaled according to the number of stocks in the
particular sector. There are strong sectorial correlations in Energy and
Utilities sectors in S&P 500 markets and Financials sector in Nikkei 225
market.

We know that the average correlation (or equivalently the largest eigenvalue)
dominates the behavior of the correlation matrix as a function of time [2].
Thus, although we move in a space of $N\times(N-1)/2$ dimensions, a single
dimension dominates the evolution except for details. These very high
dimensions seem meaningless in view of the dominance of the average
correlation and are very unwieldy if we follow, say $N$, stocks. For
illustration purposes, DS was used in [24, 7], but the pictures show the
significance of the average correlation. We use dimensional scaling that
enables us to visualize objects (set of correlation matrices) from a higher
dimensional space to a low dimensional space [24]. The objects in the higher
dimensional space satisfy the distance/similarity between correlation matrices
at different times $\tau_{t}$ and $\tau_{t}^{\prime}$ and are defined as

$\xi(\tau_{t},\tau_{t}^{\prime})=\sum_{i<j}\left|C_{ij}(\tau_{t})-C_{ij}(\tau_{t}^{\prime})\right|\;.$
(4)

For completeness, we optimize the number of market states as a function of
power map exponent $\epsilon$ based on the minimum standard deviation of
intra-cluster distances $\sigma_{intra}$. The results are shown in Appendix A
as we do not find any significant differences with PM.

##  III Results and discussion

In this section, we first present the time evolution of market states for five
clusters by optimizing standard deviation of intra-cluster distance
$\sigma_{intra}$ without applying the PM and compare it to sectorial Guhr
matrices resulting from the same correlation matrices. For the analysis, we
first compute the Pearson correlation matrix using return time series and then
apply PM followed by CG to obtain the Guhr matrices for each epoch. Then,
using the Pearson and Guhr matrices, we do KM to obtain the market states. We
also dimensionally scale the correlation matrices to three dimensions and show
the 2$D$ projections. We shall do this first for S&P 500 and then for Nikkei
225.

###  III.1 Standard and Poors 500

(a) Pearson correlation matrices (b) sectorial Guhr matrices

Figure 3: Time evolution of five market states of the S&P 500 data using (a)
Pearson correlation matrices $C$ and (b) sectorial Guhr matrices $G$. The
state evolution is obtained after performing KM clustering on 3503 short-time
correlation matrices. Pearson correlation matrix elements are computed using
logarithmic return time series of adjusted closing prices with epochs of
length 20 trading days shifted by one trading day. The market states are
arranged in order of increasing average correlations: (a) (0.157, 0.281,
0.286, 0.433, 0.611) and (b) (0.160, 0.269, 0.373, 0.487, 0.654),
respectively.

(a) $\epsilon=0.0$, Pearson correlation matrices (b) $\epsilon=0.0$, sectorial
Guhr matrices

Figure 4: 3D dimensional scaling of 3503 correlation matrices of the five S&P
500 market states from January 2006 to December 2019 shown in Fig. 3: (a)
$\epsilon=0.0$, Pearson correlation matrices and (b) $\epsilon=0.0$, sectorial
Guhr matrices.

(a) Pearson correlation matrices (b) sectorial Guhr matrices

Figure 5: Transition matrices showing transitions between different market
states for S&P 500 in Fig. 3 obtained with (a) Pearson correlation matrices
and (b) sectorial Guhr matrices. The transition matrix corresponding to Guhr
matrices is near-tridiagonal. Also, the necessary criterion for Markovianity
given in Eq. (2) of [6] is fulfilled for both. The equilibrium distributions
corresponding to (a) and (b) are (0.2095, 0.2388, 0.1263, 0.2964, 0.129) and
(0.2204, 0.2658, 0.2279, 0.2045, 0.0814) respectively.

We begin our study by analyzing evolution of market states for Pearson
correlation matrices and sectorial Guhr matrices as shown in Fig. 3 for S&P
500. The state evolution is obtained after performing KM clustering on the
3503 short-time correlation matrices. Pearson correlation matrix elements are
computed using logarithmic return time series of adjusted closing prices with
epochs of length 20 trading days shifted by one trading day. The market states
are arranged in order of increasing average correlations. State 3 obtained
using Pearson correlation matrices is markedly different from the state 3
obtained using sectorial Guhr matrices. This difference is also reflected in
the dimensional scaling figures shown in Fig. 4. Also, the transition matrices
are different as shown in Fig. 5. Only the transition matrix corresponding to
sectorial Guhr matrices is near-tridiagonal. Also, the necessary criterion for
Markovianity given in Eq. (2) of [6] is fulfilled for both the transition
matrices. Using CG, the number of transitions from state 4 (near-critical
state) to state 5 (critical state) reduce considerably as can be seen from
Fig. 5. For the sectorial Guhr matrices, there are no transitions from states
1-3 to state 5. However, when we use PM (see Fig. A.2 in Appendix A), state 3
is almost identical for both Pearson correlation matrices and sectorial Guhr
matrices. As a consequence, the dimensionally scaled figure and transition
matrices for the two as shown in Figs. A.3 and A.4 are also very similar. Note
that dimensional scaling shows no state overlaps and transition matrices are
near-tridiagonal for both. Thus, CG does a similar job as PM and hence, PM
along with CG is same as CG. Note that typical state concentrations are seen
in Fig. 3 on higher side of correlations at the end of years 2008 and 2011
while the opposite happens with the quiet market region of the year 2017. In
the quiet market region of 2017, the absence of states 3-5 is prominent when
using sectorial Guhr matrices. These features are also seen when we apply PM
(see Fig. A.2 in Appendix A). Coarse graining and PM both reduce the number of
transitions from near-critical state to the critical state.

Note that the S&P 500 market with the chosen parameters shows a sequential
behavior of market states as a function of average correlations or
equivalently, largest eigenvalues. With optimal PM parameters, this behavior
is to be expected as a dominance of the largest eigenvalues is well-known [1].
We will call this behavior lineality of market states. This lineality has to
break down if we increase the states and indeed this will happen for the S&P
500 market if we do not apply PM. Nevertheless, in the CG picture, lineality
is conserved also for six clusters. Thus, in this CG case, transition matrix
will again become near-tridiagonal.

###  III.2 Nikkei 225

Figure 6: Time evolution of five market states of the Nikkei 225 data using
(a) Pearson correlation matrices $C$ and (b) sectorial Guhr matrices $G$. The
state evolution is obtained after performing KM clustering on 3438 short-time
correlation matrices. Pearson correlation matrix elements are computed using
logarithmic return time series of adjusted closing prices with epochs of
length 20 trading days shifted by one trading day. The market states are
arranged in order of increasing average correlations. The average correlations
for the states are (a) (0.215, 0.354, 0.421, 0.484, 0.642) and (b) (0.228,
0.348, 0.438, 0.543, 0.679) respectively.

(a) $\epsilon=0$, Pearson correlation matrices (b) $\epsilon=0$, sectorial
Guhr matrices

Figure 7: 3D dimensional scaling of 3438 correlation matrices of the five
Nikkei 225 market states from January 2006 to December 2019 shown in Fig. A.1:
(a) $\epsilon=0.0$, Pearson correlation matrices and (b) $\epsilon=0.0$,
sectorial Guhr matrices.

Figure 8: Transition matrices showing transitions between different market
states for Nikkei 225 in Fig. A.1 obtained with (a) Pearson correlation
matrices and (b) sectorial Guhr matrices. The transition matrix corresponding
to Guhr matrices is nearly tri-diagonal. Also, the necessary criterion for
Markovianity given in Eq. (2) of [6] is fulfilled for both. The equilibrium
distributions corresponding to (a) and (b) are
$(0.1635,0.3254,0.1118,0.2549,0.1444)$ and
$(0.1529,0.297,0.2699,0.1834,0.0968)$ respectively.

We perform similar analysis with Nikkei 225 market data and obtain similar
results as that of S&P 500 market data. Figures 6-8 show the corresponding
results of Nikkei 225 market for state evolution, dimensionally scaled
matrices projected to two dimensions and transition matrices, respectively.
Nevertheless, as seen from Fig. A.7 in Appendix A, PM (with optimal
$\epsilon$) does not remove market state overlaps even for Pearson correlation
matrices. This is in confirmation with previous studies [6, 7, 8].

##  IV Conclusions and future outlook

We have analyzed aspects of complex system dynamics using financial markets as
a paradigm. Using the logarithmic-return-time series of adjusted closing
prices with epochs of length 20 trading days shifted by one trading day, we
calculate the Pearson correlation matrices for both S&P 500 and Nikkei 225
markets. We obtain the sectorial Guhr matrices by dividing the correlation
matrices into sectorial blocks and averaging over intra- and inter-sectorial
blocks. We thus obtain $10\times 10$ and $6\times 6$ dimensional sectorial
Guhr matrices respectively for S&P 500 and Nikkei 225 markets. Using these
Pearson and Guhr matrices, we use KM clustering to obtain the market states.
Using these, we analyze the evolution of the market states and the
corresponding transition matrices. We also dimensionally scale Pearson and
sectorial Guhr matrices to three dimensions for visualization. As CG implies
averaging, it will suppress noise and indeed the lineal behavior in the market
states and quasi tri-diagonality of transition matrices are retained up to a
larger number of market states which will likely be useful for certain
applications. Note that there is no guarantee that this will be optimal
approach for risk assessment [8] in view of the fact that we do not use the
transition matrix as an additional criterion to optimize $\epsilon$ in
potential use of PM. We also checked the so-called Guhr covariance method
which yield similar but somewhat less convincing results.

We find that the behavior of the market states for the CG matrices is rather
similar to the Pearson correlation matrices. Note that the average correlation
which is the dominant feature is almost same for sectorial Guhr matrices and
Pearson correlation matrices. In the Appendix A, we discuss the influence of
applying PM to the Pearson correlation matrices and find that the resulting
Guhr matrices may also have improved transition matrices. Note that though the
features discussed in Ref. [23] will disappear if we use PM. The question
whether CG affects them or not will have to be analyzed in a future paper as
this state exists outside the time horizon of the present paper. Note that the
number of relevant variables is reduced to 55 and 21 respectively for S&P 500
and Nikkei 225 markets thus the making the problem more manageable. It will be
interesting to analyze more extreme CG and we plan to reduce sectorial Guhr
matrices to $2\times 2$ and possibly $3\times 3$ matrices. Preliminary work
has shown this to be promising.

##  V Acknowledgements

The authors are grateful to Anirban Chakraborti, Thomas Gorin, A. Raúl
Hernández Montoya, Christof Jung, Francois Leyvraz and Thomas H. Seligman for
their inputs and suggestions. P.M. acknowledges a fellowship from CONAHCYT
Project Fronteras 428214. We gratefully acknowledge financial support from
CONAHCYT project Fronteras 10872, CONAHCYT Project Fronteras 425854, and UNAM-
DGAPA PAPIIT AG101122.

## References

  * Mantegna and Stanley [2000] R. N. Mantegna and H. E. Stanley,  _An Introduction to Econophysics: Correlations and Complexity in Finance_ (Cambridge University Press, 2000).
  * Münnix  _et al._ [2012] M. C. Münnix, T. Shimada, R. Schäfer, F. Leyvraz, T. H. Seligman, T. Guhr, and H. E. Stanley, Identifying states of a financial market, Scientific Reports 2, 644 (2012).
  * Zhou  _et al._ [2018] L. Zhou, L. Qiu, C. Gu, and H. Yang, Immediate causality network of stock markets, Europhysics Letters 121, 48002 (2018).
  * Anand  _et al._ [2018] K. Anand, J. Khedair, and R. Kühn, Structural model for fluctuations in financial markets, Phys. Rev. E 97, 052312 (2018).
  * Tang  _et al._ [2018] J. Tang, L. Khoja, and H. R. Heinimann, Characterisation of survivability resilience with dynamic stock interdependence in financial networks, Applied Network Science 3, 23 (2018).
  * Pharasi  _et al._ [2018] H. K. Pharasi, K. Sharma, R. Chatterjee, A. Chakraborti, F. Leyvraz, and T. H. Seligman, Identifying long-term precursors of financial market crashes using correlation patterns, New J. Phys. 20, 103041 (2018).
  * Pharasi  _et al._ [2019] H. K. Pharasi, K. Sharma, A. Chakraborti, and T. H. Seligman, Complex market dynamics in the light of random matrix theory, in  _New Perspectives and Challenges in Econophysics and Sociophysics_, edited by F. Abergel, B. K. Chakrabarti, A. Chakraborti, N. Deo, and K. Sharma (Springer International Publishing, Cham, 2019) pp. 13–34.
  * Pharasi  _et al._ [2024] H. K. Pharasi, E. Seligman, S. Sadhukhan, P. Majari, and T. H. Seligman, Dynamics of market states and risk assessment, Physica A: Statistical Mechanics and its Applications 633, 129396 (2024).
  * Wang  _et al._ [2020] S. Wang, S. Gartzke, M. Schreckenberg, and T. Guhr, Quasi-stationary states in temporal correlations for traffic systems: Cologne orbital motorway as an example, Journal of Statistical Mechanics: Theory and Experiment 2020, 103404 (2020).
  * Nie [2020] C.-X. Nie, A network-based method for detecting critical events of correlation dynamics in financial markets, Europhysics Letters 131, 50001 (2020).
  * Heckens  _et al._ [2020] A. J. Heckens, S. M. Krause, and T. Guhr, Uncovering the dynamics of correlation structures relative to the collective market motion, Journal of Statistical Mechanics: Theory and Experiment 2020, 103402 (2020).
  * James  _et al._ [2022] N. James, M. Menzies, and K. Chin, Economic state classification and portfolio optimisation with application to stagflationary environments, Chaos, Solitons & Fractals 164, 112664 (2022).
  * Heckens and Guhr [2022a] A. J. Heckens and T. Guhr, A new attempt to identify long-term precursors for endogenous financial crises in the market correlation structures, Journal of Statistical Mechanics: Theory and Experiment 2022, 043401 (2022a).
  * Heckens and Guhr [2022b] A. J. Heckens and T. Guhr, New collectivity measures for financial covariances and correlations, Physica A: Statistical Mechanics and its Applications 604, 127704 (2022b).
  * Campbell  _et al._ [1998] J. Y. Campbell, A. W. Lo, and A. C. MacKinlay, The econometrics of financial markets, Econometric Theory 14, 671 (1998).
  * Mantegna [1999] R. Mantegna, Hierarchical structure in ﬁnancial markets, European Physical Journal B 11, 193 (1999).
  * Bonanno  _et al._ [2001] G. Bonanno, F. Lillo, and R. Mantegna, High-frequency cross-correlation in a set of stocks, Quantitative Finance 1, 96 (2001).
  * Bonanno  _et al._ [2000] G. Bonanno, N. Vandewalle, and R. Mantegna, Taxonomy of stock market indices, Physical Review E 62, R7615 (2000).
  * Micciché  _et al._ [2003] S. Micciché, G. Bonanno, F. Lillo, and R. Mantegna, Degree stability of a minimum spanning tree of price return and volatility, Physica A-Statistical Mechanics and its Applications 324, 66 (2003).
  * Tola  _et al._ [2008] V. Tola, F. Lillo, M. Gallegati, and R. N. Mantegna, Cluster analysis for portfolio optimization. journal of economic dynamics & control, Journal of Economic Dynamics and Control 32, 235 (2008).
  * Tumminello  _et al._ [2010] M. Tumminello, F. Lillo, and R. N. Mantegna, Correlation, hierarchies, and networks in financial markets, Journal of Economic Behavior and Organization 75, 40 (2010).
  * Bouchaud and Potters [2003] J.-P. Bouchaud and M. Potters,  _Theory of financial risk and derivative pricing: from statistical physics to risk management_ (Cambridge university press, 2003).
  * Martínez-Ramos  _et al._ [2023] M. M. Martínez-Ramos, M. Vyas, P. Majai, and T. H. Seligman, Covid anomaly in the correlation analysis of s&p 500 market states, (2023), arXiv:2308.14830 .
  * Seber [2009] G. A. Seber,  _Multivariate observations_ (John Wiley and Sons, 2009).
  * Pharasi  _et al._ [2023] H. K. Pharasi, S. Sadhukhan, P. Majari, A. Chakraborti, and T. H. Seligman, Market state dynamics in correlation matrix space, in  _Quantum decision theory and complexity modelling in economics and public policy_ (Springer, in press 2023).
  * Rinn  _et al._ [2015] P. Rinn, Y. Stepanov, J. Peinke, T. Guhr, and R. Schäfer, Dynamics of quasi-stationary systems: Finance as an example, Europhysics Letters 110, 68003 (2015).
  * Guhr and Kälber [2003] T. Guhr and B. Kälber, A new method to estimate the noise in financial correlation matrices, Journal of Physics A: Mathematical and General 36, 3009 (2003).
  * Schäfer and Guhr [2010] R. Schäfer and T. Guhr, Local normalization: Uncovering correlations in non-stationary financial time series, Physica A: Statistical Mechanics and its Applications 389, 3856 (2010).
  * Vinayak  _et al._ [2013] Vinayak, R. Schäfer, and T. H. Seligman, Emerging spectra of singular correlation matrices under small power-map deformations, Phys. Rev. E 88, 032115 (2013).
  * Teknomo [2006] K. Teknomo, K-means clustering tutorial, Medicine 100, 3 (2006).
  * Jain [2010] A. K. Jain, Data clustering: 50 years beyond k-means, Pattern recognition letters 31, 651 (2010).

##  Appendix A Optimization using Power Map on the Pearson correlation
matrices

In this section, We optimize the number of market states as a function of
power map exponent $\epsilon$ [27, 28, 29] applied to the Pearson correlation
matrices based on the minimum standard deviation of the intra-cluster
distances $\sigma_{intra}$ [8] of the resulting Guhr matrices. The results are
obtained using PM along with CG. Using the Pearson correlation matrices, we
apply the PM and coarse grain these to obtain the corresponding sectorial Guhr
matrices. Then, we use the sectorial Guhr matrices to obtain the intra-cluster
distances and we choose the parameters (number of clusters and $\epsilon$)
minimizing $\sigma_{intra}$. Same parameters are used for Pearson correlation
matrices as well.

###  A.1 S&P 500

Figure A.1 shows the classification of market states based on the minimum
standard deviation of intra-cluster distances $\sigma_{intra}$ for S$\&$P 500.
We apply PM to Pearson correlation matrices, do CG and then cluster the
sectorial Guhr matrices. Variance $\sigma_{intra}$ is calculated using the
standard deviation of intra-cluster measure $d_{intra}$ of 1000 different
initial conditions for different values of number of market states (or
clusters $k$) and $\epsilon$. The optimal values (corresponding to minimum
$\sigma_{intra}$) for S&P 500 market with time horizon 2006-2019, for $k\geq
4$, appears at $k=5$ and $\epsilon=0.5$.

Corresponding to this optimal choice of parameters, the results for state
evolution, dimensionally scaled matrices projected to two dimensions and
transition matrices are shown in Figs. A.2-A.4, respectively.

Figure A.1: Classification of market states based on the minimum standard
deviation of intra-cluster distances $\sigma_{intra}$ for S$\&$P 500 market.
We have used 1000 different initial conditions for the KM calculation of
$\sigma_{intra}$. These results are for cases for which we apply PM [Eq. (2)]
to correlation matrices, do the coarse graining and then cluster the Guhr
matrices. For $k\geq 4$, the minima of $\sigma_{intra}$ appears at $k=5$ and
$\epsilon=0.5$.

(a) Pearson correlation matrices (b) sectorial Guhr matrices

Figure A.2: Time evolution of five market states of the S&P 500 data using (a)
Pearson correlation matrices $C$ and (b) sectorial Guhr matrices $G$. The
state evolution is obtained after performing KM clustering on the 3503 short-
time correlation matrices with PM ($\epsilon=0.5$). Pearson correlation matrix
elements are computed using logarithmic return time series of adjusted closing
prices with epochs of length 20 trading days shifted by one trading day. The
market states are arranged in order of increasing average correlations. The
average correlations for the states are (a) (0.115, 0.208, 0.305, 0.414,
0.563) and (b) (0.116, 0.205, 0.302, 0.411, 0.581), respectively.

(a) $\epsilon=0.5$, Pearson correlation matrices (b) $\epsilon=0.5$, sectorial
Guhr matrices

Figure A.3: 3D dimensional scaling of 3503 correlation matrices of the five
S&P 500 market states from January 2006 to December 2019 shown in Fig. 6: (a)
$\epsilon=0.5$, Pearson correlation matrices and (b) $\epsilon=0.5$, sectorial
Guhr matrices. Note that the power mapped Pearson correlation matrices behave
essentially the same as sectorial Guhr matrices.

(a) $\epsilon=0.5$, Pearson correlation matrices (b) $\epsilon=0.5$, sectorial
Guhr matrices

Figure A.4: Transition matrices showing transitions between different market
states for S&P 500 in Fig. 6 obtained with (a) Pearson correlation matrices
and (b) sectorial Guhr matrices. The transition matrices are near-tridiagonal
and the necessary criterion for Markovianity given in Eq. (2) of [6] is
fulfilled, for both. The equilibrium distributions corresponding to (a) and
(b) are (0.307, 0.273, 0.236, 0.123, 0.061) and (b) (0.272, 0.297, 0.229,
0.147, 0.055), respectively.

###  A.2 Nikkei 225

Figure A.5: Classification of market states based on the minimum standard
deviation of intra-cluster distances $\sigma_{intra}$ for Nikkei 225 market.
We have used 1000 different initial conditions for the KM calculation of
$\sigma_{intra}$. These results are for cases for which we apply PM [Eq. (2)]
to correlation matrices, do the coarse graining and then cluster the Guhr
matrices. For $k\geq 4$, the minima of $\sigma_{intra}$ appears at $k=6$ and
$\epsilon=0.3$.

Figure A.5 shows the classification of market states based on the minimum
standard deviation of intra-cluster distances for $\sigma_{intra}$ for Nikkei
225. We apply PM to Pearson correlation matrices, do CG and then cluster the
sectorial Guhr matrices. Variance $\sigma_{intra}$ is calculated using the
standard deviation of intra-cluster measure $d_{intra}$ of 1000 different
initial conditions for different values of number of market states (or
clusters $k$) and $\epsilon$. The optimal values (corresponding to minimum
$\sigma_{intra}$) for Nikkei 225 market with time horizon 2006-2019, for
$k\geq 4$, appears at $k=6$ and $\epsilon=0.3$.

Corresponding to this optimal choice of parameters, the results for state
evolution, dimensionally scaled matrices projected to two dimensions and
transition matrices are shown in Figs. A.6-A.8, respectively.

(a) Pearson correlation matrices

(b) sectorial Guhr matrices

Figure A.6: Time evolution of five market states of the Nikkei 225 data using
(a) Pearson correlation matrices $C$ and (b) sectorial Guhr matrices $G$. The
state evolution is obtained after performing KM clustering on the 3438 short-
time correlation matrices with PM ($\epsilon=0.3$). Pearson correlation matrix
elements are computed using logarithmic return time series of adjusted closing
prices with epochs of length 20 trading days shifted by one trading day. The
market states are arranged in order of increasing average correlations. The
average correlations for the states are (a) (0.177, 0.287, 0.324, 0.369,
0.454, 0.600) and (b) (0.178, 0.279, 0.339, 0.375, 0.475, 0.616) respectively.

(a) $\epsilon=0.3$, Pearson correlation matrices (b) $\epsilon=0.3$, sectorial
Guhr matrices

Figure A.7: 3D dimensional scaling of 3438 correlation matrices of the five
Nikkei 225 market states from January 2006 to December 2019 shown in Fig. A.4:
(a) $\epsilon=0.3$, Pearson correlation matrices and (b) $\epsilon=0.3$, Guhr
matrices. Note that in this case, the PM will not establish near-linear
behavior for the market states of the Pearson correlation matrix while the
figure for sectorial Guhr matrix is not significantly affected.

(a) $\epsilon=0.3$, Pearson correlation matrices

(b) $\epsilon=0.3$, sectorial Guhr matrices

Figure A.8: Transition matrices showing transitions between different market
states for Nikkei 225 in Fig. A.4 obtained with (a) Pearson correlation
matrices and (b) sectorial Guhr matrices. The transition matrices are near-
tridiagonal and the necessary criterion for Markovianity given in Eq. (2) of
[6] is fulfilled, for both. The equilibrium distributions corresponding to (a)
and (b) are $(0.198,0.168,0.257,0.087,0.189,0.100)$ and
$(0.155,0.270,0.113,0.206,0.163,0.093)$ respectively.


spaces $M_{\nu}=\\{m\in M\mid\forall h\in\mathfrak{h},\,h.m=\nu(h)m\\}$, for
all $\nu\in\mathfrak{h}^{*}$, is finite dimensional. Then one may define ([25,
3.4.7]) the formal character ${\rm ch}\,M$ of $M$ as follows.

${\rm ch}\,M=\sum_{\nu\in\mathfrak{h}^{*}}\dim M_{\nu}\,e^{\nu}.$

Recall the set $E(\pi^{\prime})$ in subsection 6.1, and for all $\Gamma\in
E(\pi^{\prime})$, set
$\delta_{\Gamma}=w_{0}^{\prime}d_{\Gamma}-w_{0}d_{\Gamma}$, with
$d_{\Gamma}=\sum_{\gamma\in\Gamma}\varpi_{\gamma}$. By Prop. 6.2 and Theorem
6.1, the algebra of invariants
$\widetilde{C}_{\mathfrak{r}}^{U(\mathfrak{r}^{\prime})}$ is a polynomial
algebra in $\lvert E(\pi^{\prime})\rvert$ variables, each of them having
$\delta_{\Gamma}$, $\Gamma\in E(\pi^{\prime})$, as an $\mathfrak{h}$-weight.
Moreover for $\mathfrak{g}$ simple and $\pi^{\prime}\subsetneq\pi$ that is,
for $\mathfrak{p}\subsetneq\mathfrak{g}$, one has that $\delta_{\Gamma}\in
P^{+}(\pi)\setminus\\{0\\}$ for all $\Gamma\in E(\pi^{\prime})$ by [16,
5.4.3]. Then in this case ${\rm
ch}\,\widetilde{C}_{\mathfrak{r}}^{U(\mathfrak{r}^{\prime})}$ is well defined.

######  Proposition.

Assume, for all $\nu\in\mathfrak{h}^{*}$, that the weight space
$Sy(\tilde{\mathfrak{p}})_{\nu}$ is finite dimensional, so that the formal
character ${\rm{ch}}\,Sy(\tilde{\mathfrak{p}})$ is well defined. Then

${\rm ch}\,\widetilde{C}_{\mathfrak{r}}^{U(\mathfrak{r}^{\prime})}\leq{\rm
ch}\,Sy(\tilde{\mathfrak{p}})$

namely

$\prod_{\Gamma\in E(\pi^{\prime})}(1-e^{\delta_{\Gamma}})^{-1}\leq{\rm
ch}\,Sy(\tilde{\mathfrak{p}}).$

###### Proof.

Recall that the generalized Kostant filtration $\mathscr{F}_{K}$ is
decreasing, separated and that
$\mathscr{F}_{K}^{0}(U(\tilde{\mathfrak{p}}^{-})^{*})=U(\tilde{\mathfrak{p}}^{-})^{*}$.
Then, for every finite dimensional subspace $V$ of
$U(\tilde{\mathfrak{p}}^{-})^{*}$, there exists $N\in\mathbb{N}$ such that
$V\cap\mathscr{F}_{K}^{N}(U(\tilde{\mathfrak{p}}^{-})^{*})=\\{0\\}$. One
deduces easily that the graded vector space $gr_{K}(V)$ associated to the
induced generalized Kostant filtration on $V$ is isomorphic to $V$. Using
Theorem 7.6 and a same argument as in [16, 7.1] completes the proof. ∎

## References

  * [1]
  * [2] Borho W., Gabriel P., Rentschler R.: Primideale in Einhüllenden auflösbarer Lie-Algebren (Beschreibung durch Bahnenräume), Lecture Notes in Math. 357 Springer-Verlag, Berlin (1973).
  * [3] Bourbaki N.: Eléments de mathématique, Algèbre Chap. 1 à 3, Diffusion CCLS.
  * [4] Bourbaki N.: Eléments de mathématique, Groupes et Algèbres de Lie, Chapitres 4, 5 et 6, Masson, Paris et al. (1981).
  * [5] Bourbaki N.: Eléments de mathématique, Groupes et algèbres de Lie, Chapitres 7 et 8, Springer-Verlag, Berlin Heidelberg, 2006.
  * [6] Bourbaki N.:Eléments de mathématique, Algèbre commutative, Hermann, Paris 1968.
  * [7] Charbonnel J.Y., Moreau A.: The symmetric invariants of the centralizers and Slodowy grading, Math. Zeitschrift 282 (2016), no. 1-2, 273–339.
  * [8] Dixmier J.: Sur les représentations unitaires des groupes de Lie nilpotents II, Bull. Soc. Math. France 85 (1957), 325–388.
  * [9] Dixmier J.: Sur le centre de l’algèbre enveloppante d’une algèbre de Lie, C.R. Acad. Sc. Paris 265 (1967) 408–410.
  * [10] Dixmier J.: Algèbres Enveloppantes, Editions Jacques Gabay, les grands classiques Gauthier-Villars, Paris et al. (1974).
  * [11] Fauquant-Millet F.: Weierstrass sections for some truncated parabolic subalgebras, Journal of Algebra 580 (2021) 299–365.
  * [12] Fauquant-Millet F.: Sur la polynomialité de certaines algèbres d’invariants d’algèbres de Lie, Mémoire d’Habilitation à Diriger des Recherches,
http://tel.archives-ouvertes.fr/tel-00994655.

  * [13] Fauquant-Millet F.: About polynomiality of the Poisson Semicentre for Parabolic Subalgebras. In: Gorelik M., Hinich V., Melnikov A. (eds) Representations and Nilpotent Orbits of Lie Algebraic Systems, In Honour of the 75th Birthday of Tony Joseph. Progress in Mathematics, vol. 330 (2019) Birkhäuser, Cham.
https://doi.org/10.1007/978-3-030-23531-4

  * [14] Fauquant-Millet F.:Sur une algèbre parabolique $\bf P$ de $\check{\rm U}_{q}(\mathfrak{s}\mathfrak{l}_{n+1})$ et ses semi-invariants par l’action adjointe de $\bf P$, Bull. Sci. math. 122 (1998) 495–519.
  * [15] Fauquant-Millet F., Joseph A.: Sur les semi-invariants d’une sous-algèbre parabolique d’une algèbre enveloppante quantifiée, Transform. Groups 6 (2001) 125–142.
  * [16] Fauquant-Millet F., Joseph A.: Semi-centre de l’algèbre enveloppante d’une sous-algèbre parabolique d’une algèbre de Lie semi-simple, Ann. Sci. Éc. Norm. Sup. 38 (2005) 155–191.
  * [17] Fauquant-Millet F., Joseph A.: La somme des faux degrés - un mystère en théorie des invariants, Adv. Math. 217 (2008) 1476–1520.
  * [18] Fauquant-Millet F., Joseph A.: Adapted pairs and Weierstrass sections,
http://arxiv.org/abs/1503.02523.

  * [19] Fauquant-Millet F., Lamprou P.: Slices for maximal parabolic subalgebras of a semisimple Lie algebra, Transform. Groups 22 (2017) 911–932.
  * [20] Fauquant-Millet F., Lamprou P.: Polynomiality for the Poisson centre of truncated maximal parabolic subalgebras, J. Lie Theory 28 (2018) 1063–1094.
  * [21] Feigin E., Fourier G., Littelmann P.: PBW filtration and bases for irreducible modules in type ${\rm A}_{n}$, Transformation Groups 16, 71–89 (2011). https://doi.org/10.1007/s00031-010-9115-4
  * [22] Fittouhi Y., Joseph A.: Weierstrass sections for parabolic adjoint action in type A,
http://arxiv.org/abs/2001.00447.

  * [23] Humphreys J.E. : Introduction to Lie Algebras and Representation Theory, Graduate Texts in Mathematics, Springer-Verlag New-York, 1972.
  * [24] Joseph A.: A preparation theorem for the prime spectrum of a semisimple Lie algebra, J. Algebra 48 (1977) 241–289.
  * [25] Joseph A.: Quantum Groups and Their Primitive Ideals, Ergebnisse der Mathematik und ihrer Grenzgebiete, Springer Verlag, Berlin-Heidelberg-New-York, 1995.
  * [26] Joseph A.: On semi - invariants and index for biparabolic (seaweed) algebras I, J. Algebra 305 (2006) 487–515.
  * [27] Joseph A.: On semi - invariants and index for biparabolic (seaweed) algebras II, J. Algebra 312 (2007) 158–193.
  * [28] Joseph A.: Slices for biparabolic coadjoint actions in type $A$, J. Algebra 319 (2008) 5060–5100.
  * [29] Joseph A.: Compatible adapted pairs and a common slice theorem for some centralizers, Transform. Groups 13 (2008) 637–669.
  * [30] Joseph A.: An algebraic slice in the coadjoint space of the Borel and the Coxeter element, Adv. Math. 227 (2011) 522–585.
  * [31] Joseph A., Lamprou P.: Maximal Poisson commutative subalgebras for truncated parabolic subalgebras of maximal index in $\mathfrak{sl}_{n}$, Transform. Groups 12 (2007) 549–571.
  * [32] Joseph A., Shafrir D.: Polynomiality of invariants, unimodularity and adapted pairs, Transform. Groups 15 (2010) 851–882.
  * [33] Kostant B.: Lie group representations on polynomial rings, Amer. J. Math. 85 (1963) 327–404.
  * [34] Ooms A.: The polynomiality of the Poisson center and semi-center of a Lie algebra and Dixmier’s fourth problem, J. Algebra 477 (2017) 95-146.
  * [35] Ooms A., Van Den Bergh M.: A degree inequality for Lie algebras with a regular Poisson semi-center, J. Algebra 323 (2010) 305–322.
  * [36] Panyushev D.: On the coadjoint representation of $\mathbb{Z}_{2}$-contractions of reductive Lie algebras, Adv. Math. 213 (2007), no.1, 343–404.
  * [37] Panyushev D., Premet A., Yakimova O.: On symmetric invariants of centralisers in reductive Lie algebras, J. Algebra 313 (2007) 343–391.
  * [38] Panyushev D., Yakimova O.: A remarkable contraction of semisimple Lie algebras, Ann. Inst. Fourier (Grenoble) 62 (2012) 2053–2068.
  * [39] Panyushev D., Yakimova O.: Parabolic contractions of semisimple Lie algebras and their invariants, Sel. Math. New Ser. 19 (2013) 699–717.
  * [40] Panyushev D., Yakimova O.: Symmetric invariants related to representations of exceptional simple groups, Trans. Moscow Math. Soc. 78 (2017) 161–170.
  * [41] Popov V. L. Sections in invariant theory. The Sophus Lie Memorial Conference (Oslo 1992), Scand. Univ. Press, Oslo (1994) 315-361.
  * [42] Popov V. L., Vinberg E.B. Invariant Theory, in Encyclopaedia of Mathematical Sciences, Vol. 55, Algebraic Geometry IV, Springer-Verlag Berlin, Heidelberg 1994.
  * [43] Renault G.: Algèbre non commutative, Collection “Varia Mathematica”, Gauthier-Villars, 1975.
  * [44] Tauvel P., Yu R.W.T.: Lie Algebras and Algebraic Groups, Springer Monographs in Mathematics, Springer Verlag Berlin Heidelberg (2005).
  * [45] Yakimova O.: A counterexample to Premet’s and Joseph’s conjectures, Bull. Lond. Math. Soc. 39 (2007) 749–754.
  * [46] Yakimova O.: One-parameter contractions of Lie-Poisson brackets, J. Eur. Math. Soc. 16 (2014), 387–407.
  * [47] Yakimova O.: Symmetric invariants of $\mathbb{Z}_{2}$-contractions and other semi-direct products, Int. Math. Res. Notices 2017 (2017)1674–1716.
  * [48] Yakimova O.: Some semi-direct products with free algebras of symmetric invariants, In: Perspectives in Lie Theory, F. Callegaro, G. Carnovale, F. Caselli, C. De Concini, A. De Sole (eds.), Springer, Cham 19 (2017) 267–279.