# Dynamic portfolio optimization with
Inverse Covariance Clustering

Yuanrong Wang
Department of Computer Science
University College London
Gower Street, London WC1E 6BT
<EMAIL_ADDRESS>
&Tomaso Aste
Department of Computer Science
University College London
Gower Street, London WC1E 6BT
<EMAIL_ADDRESS>
Corresponding Author

###### Abstract

Market conditions change continuously. However, in portfolio’s investment
strategies, it is hard to account for this intrinsic non-stationarity. In this
paper, we propose to address this issue by using the Inverse Covariance
Clustering (ICC) method to identify inherent market states and then integrate
such states into a dynamic portfolio optimization process. Extensive
experiments across three different markets, NASDAQ, FTSE and HS300, over a
period of ten years, demonstrate the advantages of our proposed algorithm,
termed Inverse Covariance Clustering-Portfolio Optimization (ICC-PO). The core
of the ICC-PO methodology concerns the identification and clustering of market
states from the analytics of past data and the forecasting of the future
market state. It is therefore agnostic to the specific portfolio optimization
method of choice. By applying the same portfolio optimization technique on a
ICC temporal cluster, instead of the whole train period, we show that one can
generate portfolios with substantially higher Sharpe Ratios, which are
statistically more robust and resilient with great reductions in maximum loss
in extreme situations. This is shown to be consistent across markets, periods,
optimization methods and selection of portfolio assets.

_K_ eywords Dynamic Portfolio Optimization $\cdot$ Portfolio Management
$\cdot$ Financial Market States $\cdot$ Market Regimes $\cdot$ Temporal
Clustering $\cdot$ Information Filtering Networks $\cdot$ Covariance Structure

##  1 Introduction

In the field of asset management, the problem of portfolio allocation has
gained unprecedented popularity over the past few years. Constructing a good
portfolio combines the art and science of balancing between trade-offs and the
aim to meet long-term financial goals. The simple core of any portfolio
optimization is to assign optimal weights to each portfolio’s component in
order to minimize investment’s risk and maximize the return. In 1952,
Markowitz [1] demonstrated that, by assuming risk to be quantifiable by the
variance of the portfolio’s returns, the optimal weights which minimize
portfolio’s variance at a given average portfolio’s return can be computed
with a simple and exact formula. However, the Markowitz’s theoretical maximum
is attained only in-sample, on the train dataset, whereas off-sample, on the
test set where investment is made, performances of the Markowitz’s portfolio
can be largely sub-optimal.

Markowitz’s modern portfolio theory is the foundation to modern quantitative
asset management. There are however two main limitation in the Markowitz’s
assumptions. The first limitation concerns the use of the portfolio’s variance
as measure of risk. The variance (when defined) is indeed a measure for the
width of the distribution but there are other properties that are better
measure of risk (e.g. the value at risk) and might not be reducible to the
variance when the underlying probability distribution is not a location-scale.
The second limitation concerns the ability to estimate the (future) means and
covariance of the asset’s returns in the portfolio.

After Markowitz’s seminal work, many portfolio selection methodologies have
been introduced to cure the first limitation concerning the reliance on
variance for risk quantification and nowadays there are several well-
established approaches that go well beyond the use of variance as sole risk
measure [2]. Furthermore, with the enormous development of machine learning
optimization techniques there are presently virtually no limitations in
constructing optimal portfolios based on any kind of risk measure [3, 4, 5].

Addressing the second limitation is harder. Indeed, normally, one does not
have information from the future that would allow to set the future properties
of the asset’s multivariate distribution. Therefore, the reliance on past
observations and the assumption that they will significantly represent also
the future, is hard to avoid. Nonetheless, markets are not stationary, it is
common knowledge that they cyclically pass through bull and bear states and
occasionally deepen into crisis periods. For each of these periods the market
prices’ returns have different statistical properties and they are not
describable by means of a unique multivariate probability distribution. This
is especially relevant for factors that matter most to the management of
portfolio risk. Indeed, crisis periods have distribution with fatter tails and
they tend to be more asymmetrical with the left tail having larger probability
for large losses than the right tail for equivalent gains. Portfolio
constructions must take into account these differences and device different
investment strategies for each market condition. This is indeed the ground
basis for any dynamic asset allocation. However, such a wise allocation would
imply the knowledge of the future market state and forecasting it from past
observations is not an easy task.

In this paper, we provide an algorithm termed Inverse Covariance Clustering-
Portfolio Optimization (ICC-PO) to address the non-stationarity problem, by
identifying the inherent market states and forecast the most likely future
state. The Inverse Covariance Clustering (ICC) [6] is a novel temporal
clustering method for market states clustering. In this paper we propose to
make use of this temporal clustering classification, constructing different
optimal portfolios associated with two ICC market state clusters. The clusters
are constructed in the in-sample training set (the past) and then are used
separately to train the portfolio optimizer of choice which is then tested on
an off-sample period following the training set (the future). For the
optimization we used two approached based on the classical Markowitz’s
approach but devised to have only positive weights (no short-sellings). They
are the Sequential Least Square Quadratic Programming (SLS) and the Critical
Line Algorithm (CLA). Let us note, that the ICC-PO approach allows the use of
any optimization method of choice. We tested the approach with three extensive
experiments with daily data, from 2010 to 2020, from three different markets:
NASDAQ, FTSE and HS300. For each market, we selected 100 largest market
capitalization constituent stocks and quantified the off-sample performances
of portfolios constructed from in-sample training data using separately the
two ICC-market states. We demonstrate that the difference in returns and risks
(computed on the testing set) between the two optimal portfolios, constructed
from the two ICC-market states (on the training set), is very large with
Sharpe Ratios that more than double and with very large differences in the
likelihoods of large negative returns that can have up to three time smaller
quantiles (i.e. value at risk). We provide a simple criteria to forecast the
best performing out of sample market state which we named ‘State 0’. Our
results also show that sparsification of the inverse covariance matrix through
information filtering networks [7, 8] is improving the results, this is a
confirmation of a previous result [9] extended however in this paper to a
different dataset, different portfolio optimizers and different markets. The
robustness of the method is tested by gathering statistics over 100 re-
sampling of consecutive train-test sets randomly selected across the 10 years
period 2010 to 2020. Furthermore, reliance on portfolio basket choices is
tested by doing the same experiments with random selection of 100 stocks
instead of the 100 most capitalized.

The remaining of the paper is organized as follows: in Section 2, we review
literature on market states clustering, dynamic portfolio optimization, and
information filtering network applied in the experiments; in Section 3, we
outline experimental methodologies and implementations. The final results are
presented in Section 4 and discussed in Section 5.

##  2 Background literature

###  2.1 Mean-variance optimization

Despite the unquetionable merits and pioneering status of Markowitz’s mean-
variance optimization (MVO) approach, there are some major assumptions, and
several bad applications, that reduce its efficacy for practical
implementations. Firstly, MVO assumes that asset returns follow a finite-
variance distribution and higher moments are monotonic with variance. Many
financial theories simplify this assumption adopting a normal distribution,
and consequently the models that utilize such theories do not account for
extreme market situations. Moreover, the variance of a normal distribution as
a risk measure does not distinguish between upside and downside moves in the
market. Secondly, the MVO rely on the inversion of a covariance matrix and
this operation makes the method highly sensitive to estimation error
especially when the covariance is estimated on a relatively short time-period
and when such a past period is not representative of the future. Indeed,
historical financial market data is never a good representation of the true
underlying distribution as the observations are often partial. Furthermore,
most MVO implementations are assuming market stationarity, which is that the
mean and variance are assumed constant in each asset, while the correlation is
static between assets. MVO is designed to avoid unsystematic risks by
optimizing diversification. However, the systematic risks from market
movements are not addressed by the MVO methodology and this is usually the
most significant factor for investment decisions. Lastly, a single-period
investment will almost never work in reality. A constant re-allocation is
vital to respond to the rapidly changing environment.

###  2.2 Fat-tailed and asymmetric return distributions

Normal distributions do not represent well the observed probability
distribution of financial market’s assets prices returns. Indeed, they instead
have a larger number of small returns than what expected from normal
statistics, but also a larger number of very large positive and negative
returns of sizes that would be impossible with normal statistics [10, 11, 12].
They also have often asymmetric distributions with larger negative returns
(losses) more likely than large positive ones (gains). Several alternative
probability distributions have been used in the literature, namely, Student-t
[13, 14, 15], Laplace [16, 17, 18] and Pareto-Levy [19, 20] distributions. In
addition, alternative approaches to account for asymmetry have been taken into
account, with early works by Markowitz himself which in 1959 [21] employed
semi-covariance (the covariance from negative returns only) as a better risk
measure to better describe the downside market moves. Furthermore, limited
sample size is a critical contributing factor to estimation errors. Yet,
simply extending the sample size introduces data from events happened far in
the past which are likely to be less representative of present market
conditions. Hence, methods ranging from shrinkage [22], to LASSO
regularization [23, 24], and Monte Carlo based re-sampling [25, 26] have been
used to reduce this issue.

###  2.3 Non-stationarity and dynamic portfolio allocation

Assumptions regarding market stationarity and portfolio re-allocation are
often considered together, since multi-period investment is proven to be an
effective solution to mitigate the effect of Market turmoil. Several
contributions have shown that dynamic re-allocation brings improvements in the
resilience to market volatility with respect to the original single-period
portfolio diversification methods [27, 28, 29]. Nonetheless, such methods
still fail to address structured market movements. Indeed, accounting for such
changes requires to forecast the future market state. Further studies on
market states has been proposed to model and predict the intrinsic properties
of these dynamics, and two main streams are discussed below. The first one
uses Markov decision process to model the transition probability between
different market regimes. Currently, Hidden Markov Model (HMM) has
demonstrated great efficiency and validity [30]. However, it often encounters
problems mainly associated with the curse of dimensionality, as the
dimensionality of hidden states is linear to the number of assets considered
[31, 32]. On the other stream, researchers believe that market comprises mixed
multivariate distributions, and each state effectively corresponds to a
distribution. Hence, temporal clustering methods such as Gaussian Mixture [33,
34, 35], K-Nearest Neighbors (KNN) [36, 37, 38] have been applied for this
purpose. Then, portfolios can be re-adjusted according to the predicted state
with a selected re-allocation period. Yet, these methods often based on strong
assumptions and they are not originally designed for time-series, which
results issues e.g., Gaussian Mixture assumes Gaussian nature in all the base
distributions, and KNN overlooks temporal consistency between single data
point. This is also to some extent the approach of the decision-theoretic
Bayesian methods [39, 40, 41], such as the Black-Litterman model [42, 43],
which includes in the optimization a Bayesian prior on the future state. The
ICC-PO approach introduced in the present paper is in the same line of
temporal clustering methods just mentioned. However, in our case the temporal
clustering is the ICC method and we make no use of Gaussian Mixtures.

The traditional Markowitz model optimizes on a single-period only, and it
relies heavily on the assumption of constant asset mean vectors and covariance
matrix. Therefore, this static and long investment horizon is inadequate in a
dynamic market place. Yet, the mean-variance criteria inspires the development
in multi-period dynamic portfolio construction. The dynamic portfolio
optimization field currently follows two main streams. A discrete-time model
was proposed by Samuelson in 1969 [44] and developed since by Hakansson,
Grauer and others [45, 46, 47]. It separates an investment horizon into
discrete periods, and the portfolio can be reallocated at the end of each
period. In contrast, a continuous-time model was introduced by Merton [48] in
the same year, and together with further studies described the continuous
rebalancing of securities for a fixed planning horizon [49, 50, 51].

The two alternative assumptions that are often made in dynamic portfolio
optimization problems are market completeness and investment horizon. A
complete market is an approximation to the real market where friction,
transaction costs and asset liquidity exist, and dynamic portfolio has to
consider those real world factors [52, 53, 54, 55]. A more ideal scenario is
instead the incomplete market where some conditions are waived so that
research can only focus on dynamic asset selection process and ignore some
practical issues [56, 57, 58, 59]. Similarly, infinite horizon is a naive
assumption to finite horizon where investors will withdraw investment with an
exit time. The earlier pioneers [45, 49, 53] in this field, such as Samuelson
[44], Merton [48, 50], began with the infinite horizon assumption, while later
researchers in the 90s and the beginning of the millennium [60, 61, 55, 59]
led by He & Pearson [62] and Karatzas et.al [63] started to introduce the
finite horizon into the problem.

###  2.4 Market States Clustering

After the initial pitfall of Markov-Model-based methods [31, 32, 30], mainly
due to the curse of dimensionality, literature has started to look for
alternative methods to cluster similar temporal data points into a same group
based on certain comparison criteria. Such temporal clustering methods can
mostly be divided into two approaches: subsequent clustering and point
clustering. Subsequent clustering uses a sliding window to capture a period of
data points and analyze for recurrent patterns [64, 65]. The four main methods
of subsequent clustering are: (i) hierarchical [66, 67, 68]; (ii) partitioning
[69, 70]; (iii) density-based [71, 72, 73]and; (iv) pattern discovery [74, 75,
76]. These method have all shown applicability to financial data analysis and
portfolio construction. An alternative approach is point clustering that,
instead of measuring spatial similarity between two slices of time-series, it
looks at each temporal point individually, and assigns this multivariate
observation to an appropriate cluster based on distance metrics [77, 78, 79].
Hence, in point clustering, the choice of distance is core. In macroeconomics,
the market states are not the representation of solely upward or downward
trends of the market, but also the relative dynamics of equity prices, which
naturally makes correlations a convenient choice of collective dynamics. A
stationary correlation structure was assumed as the common approach in the
industry in the 90s [43, 80], which was, however, later shown to be overly
presumptive [81, 82, 83]. Consequently, research has been devoted to study
time-varying correlations. Models, such as Generalized Autoregressive
Conditional Heteroskedasticity (GARCH) [84] and the Dynamic Conditional
Correlation (DCC) [85] have been proposed for simulating and predicting this
dynamical correlation. However, most of these models suffer from the curse of
dimensionality and can only be applied to a limited number of assets, as
numbers of parameters increases super-linearly with the number of variables.

In 2017, Hallac et al. proposed the Toeplitz Inverse Covariance Clustering
(TICC) [86] algorithm, originally devised for electric vehicles action sensor.
It classifies states based on the likelihood measures of short subsequences of
observations and corresponding sparse precision matrix. After clustering, the
precision matrix of each state is estimated under a Toeplitz constraint.
Inspired by TICC, Procacci and Aste in 2020 [6] proposed a closed related
methodology names Inverse Covariance Clustering (ICC). This approach provides
a point clustering of observations also enforcing temporal consistency by
penalizing switching between states. The ICC method also uses sparse precision
matrices but sparsification is attained via information filtering networks
(see next Subsection). One main advantage of ICC, compared to TICC, is its
flexibility in the selection of similarity measures. It was also stated in
their original paper that different clustering distances separate market
states differently. For example, likelihood distance distinguishes better with
pre- and post-crisis period, Euclidean distance discriminates well between
bull and bear states, and Mahalanobis distance is a mixture of the above.

###  2.5 Information Filtering Networks

Many computational methods employ sparse approximation techniques to estimate
the inverse covariance matrix. The sparsification is effective because the
least significant components in a covariance matrix are often largely prone to
small changes and can lead to instability. Sparsified models filters out these
insignificant components, and thus improve the model resilience to noise. A
widely used approach for inverse covariance sparsification is GLASSO [87] that
uses $L_{1}$ norm regulatization. An alternative approach that uses
information filtering networks was shown to deliver better results with lower
computational burden and larger interpretability [8]. Information filtering
network analysis of complex system data has advanced significantly in the past
few years. Its aim is to model interactions in a complex system as a network
structure of elements (vertices) and interactions (edges). The first and best
know approach is the Minimum Spanning Tree (MST) that was firstly introduced
by Boruvka in 1926 [88] and it can be solved exactly (see [89] and [90] for
two common approaches). The MST reduces the structure to a connected tree
which retains the larger correlations. To better extract useful information,
Tumminello et al. [91] and Aste and Di Matteo [92] introduced the use of
planar graphs in the Planar Maximally Filtered Graph (PMFG) algorithm. Recent
studies have extended the approach to chordal graphs of flexible sparsity [7,
93]. Research fields ranging from finance [8] to neural systems [94] have
applied this approach as a powerful tool to understand high dimensional
dependency and construct a sparse representation. It was shown that, for
chordal information filtering networks, such as the Triangulated Maximally
Filtered Graph (TMFG) [7], one can obtain a sparse precision matrix that is
positively definite and has the structure of the network paving the way for a
proper $L_{0}$-norm topological regularization [95]. This approach has been
proved to be computationally more efficient and stable than GLASSO [87],
especially when few data points are available [8, 92].

###  2.6 Sparse inverse covariance for portfolio construction and market state
prediction

The application of sparse inverse covariance for portfolio construction has
been recently introduced in the literature [96, 97, 98]. The general approach
has been to make use of $L_{1}$-norm regularization via GLASSO [87]. In a
recent paper the sparsification methodology via information filtering
networks, was applied to the identification of inherent market states via ICC
[6]. In a following paper, the sparsification with TMFG information filtering
networks was applied to the portfolio construction problem revealing several
advantages with respect to traditional mean-variance methods [9].

##  3 Methodologies

In the present paper we combine ICC clustering with market state forecasting
to be used for portfolio optimization. Let us list in this Section the main
methods we use in our approach.

###  3.1 Inverse covariance temporal clustering for portfolio optimization
(ICC-PO)

Let’s consider a set of $n$ assets with $\mathbf{r}_{t}\in\mathbb{R}^{1\times
n}$ the vector of returns at time $t$. The corresponding vector of their
expected values is
$\boldsymbol{\mu}=\mathbb{E}(\mathbf{r}_{t})\in\mathbb{R}^{1\times n}$ and
their covariance matrix is
$\boldsymbol{\Sigma}=\mathbb{E}((\mathbf{r}_{t}-\boldsymbol{\mu})^{\top}(\mathbf{r}_{t}-\boldsymbol{\mu}))\in\mathbb{R}^{n\times
n}$. The ICC clustering method depends on the choice of a gain function,
$G_{t,k}$, which is a measure which qualifies the gain when the time $t$
returns, $\mathbf{r}_{t}$, are associated whith cluster $k$. Indeed, the ICC
approach gathers together in cluster $k$ observations that have the largest
gain in such a cluster with respect to any other cluster: $G_{t,k}>G_{t,h}$
for all $h\not=k$. For instance, in [6] it was used

$G^{Eu}_{t,k}=-(\mathbf{r}_{t}-\hat{\boldsymbol{\mu}}_{k})(\mathbf{r}_{t}-\hat{\boldsymbol{\mu}}_{k})^{\top}$
(1)

where $\hat{\boldsymbol{\mu}}_{k}$ is the sample mean return computed form the
observations in cluster $k$. This gain is minus the square of the euclidean
distance between the observation and the centroid of cluster $k$. A distance
associated with the likelihood for multivariate normal distributions is
instead

$G^{No}_{t,k}=\frac{1}{2}\ln|\hat{\boldsymbol{\Sigma}}_{k}^{-1}|-n\frac{d^{2}_{t,k}}{2},$
(2)

with

$d^{2}_{t,k}=(\mathbf{r}_{t}-\hat{\boldsymbol{\mu}}_{k})^{\top}\hat{\boldsymbol{\Sigma}}_{k}^{-1}(\mathbf{r}_{t}-\hat{\boldsymbol{\mu}}_{k})$
(3)

the Mahalanobis distance where $\hat{\boldsymbol{\Sigma}}_{k}$ is the sample
covariance computed form the observations in cluster $k$. While for the
multivariate Student-t one has

$G^{St}_{t,k}=\frac{1}{2}\ln|\hat{\boldsymbol{\Sigma}}_{k}^{-1}|-\frac{\nu+n}{2}\ln(1+\frac{d^{2}_{t,k}}{\nu})$
(4)

where, in this case, $\boldsymbol{\Sigma}_{k}^{-1}$ is the scale matrix, which
is $1-2/\nu$ times the sample covariance when $\nu>2$ and the covariance is
defined (note that also $\hat{\boldsymbol{\Sigma}}_{k}$ in the expression for
$d^{2}_{t,k}$ is the scale matrix, in this case).

We extensively tested all these gain functions observing that $G^{Eu}_{t,k}$
is particularly efficient is selecting clusters with prevalence of positive or
negative returns but it is performing poorly in the portfolio optimization
problem. The normal and Student-t likelihood related gains have similar
performances, but $G^{St}_{t,k}$ turns out to be in average superior and we
adopted it for the experiments we present in this paper. We also tested an
hybrid distance
$G_{t,k}=c_{1}\ln|\hat{\boldsymbol{\Sigma}}_{k}^{-1}|-c_{2}{d}^{2}_{t,k}$ with
the two arbitrary constants, $c_{1}$ and $c_{2}$, that allow to gauge between
the effects of the determinant of the covariance (an entropic term) and the
Mahalanobis distance term. The measure of the determinant of a covariance is
an equivalent estimation of the differential entropy of the multivariate
system, while Mahalanobis distance measures distance between points and
distributions.

ICC approach uses sparse inverse covariance that was shown to improve
considerably results over the full covariance. As sparsification technique we
used the sparse inverse constructed with TMFG information filtering graphs [7]
using the local-global (LoGo) inversion procedure described in [8], where the
elements of the inverse are computed by inverting local sample covariance
matrices from only four variables at the time and adding them up. The result
is a sparse inverse covariance with $3n-6$ non-zero entries in the upper
diagonal (instead of $n(n-1)/2$ in the full matrix). Such a matrix is
positively definited, if the number of observations is larger than four,
independently on the size of the whole matrix ($n\times n$). Sparse portfolios
are simply obtained by applying a portfolio optimization method (see next
subsection) with a sparse inverse covariance instead of a full covariance as
input.

A final key element of the ICC methodology is the temporal consistency of the
cluster that is imposed by penalizing frequent switches between clusters. In
this paper the penalizer parameter $\gamma$ is estimated in the train set
through a grid search so that the average cluster persistence is of a given
length (30 days in this paper).

The assignment of the temporal instance $t$ to a cluster number, $k_{t}$, is
performed iteratively starting from an initial random cluster assignment.
Specifically we evaluate the penalized gain

$\tilde{G}_{t,k_{t}}=G_{t,k_{t}}-\gamma\delta_{k_{t-1},k_{t}},$ (5)

and assign observation $t$ to the cluster with largest penalized gain. In the
previous expression, $\delta_{k_{t-1},k_{t}}$ is the Kronecker delta returning
one if $k_{t-1}=k_{t}$ and zero otherwise. After the assignment of the
time-$t$ observation to a given cluster $k_{t}$, all cluster parameters (means
and covariances) are recomputed with the new cluster assignments.

We then performed a mean variance portfolio optimization method independently
for each ICC state. Obtaining optimal weights associated with each temporal
cluster. To apply effectively such optimized weights to the portfolio problem
we have to forecast the state that is most likely to be predominant in the
future test set where the investment is performed. For this purpose we made
use of the short term persistence of such states and we assigned as most
likely future state the one that is predominant in the last part of the train
set. In this paper we consider two clusters only.

###  3.2 Portfolio optimization methods

Our proposed methodology is made of three main stages. First, we use ICC for
temporal clustering the train dataset into two market states. Second, we
forecast which of the two states will be predominant in the future, test
dataset, where the investment is made. Third, we perform portfolio
optimization using train data from the forecasted predominant ICC state. Our
approach is, to large extent, agnostic to the kind of optimization adopted. In
this paper, for the experiments, we used two, mean-variance optimizations
methods: 1. the Sequential Least Square Quadratic Programming approach and; 2.
the Critical Line Algorithm method. Let us briefly recall the basic elements
of these two portfolio optimization methods.

For the experiments in this paper Markowitz’s optimal weights can be computed
with the python package ’Numpy’ for direct matrix multiplication. The exact
solution is shown in Appendix E. In the literature, this solution is referred
to as ‘unconstrained’ because, beside the normalization and average
conditions, the weights have no other constraints. On the other hand, in some
practical cases, one might want to add further conditions to the weights. For
instance, many real world situations do not allow short selling, which hence
makes necessary to impose only positive weights in the range $w_{i}\in[0,1]$.
This constrained optimization problem cannot be any longer analytically solved
and numerical optimization methods must be adopted.

Two numerical optimization methods have been adopted in the experiment. The
sequential least square quadratic programming (SLS) [99, 100, 101] is
considered to be one of the most efficient computational method to solve
general nonlinear constrained optimization problems. Jackson et al. and
Cesarone et al. demonstrate its effectiveness in finance [102, 103]. There is
an easy-to-use package implemented in Python’s SciPy.optimize library [104]
which we applied in our experiments. The Critical Line Algorithm (CLA) is an
efficient alternative to the quadratic optimizer for mean-variance model, as
it is specifically designed for inequality portfolio optimization. It was
already originally introduced in the Markowitz Portfolio Selection paper [1],
and its computational implementation has become increasingly popular [105,
106]. CLA also solves constrained problems with conditions in inequalities,
but unlike SLS, it divides a constrained problem into series of unconstrained
sub-problems. In our experiment, to compute CLA optimization for portfolio
selection, we leveraged the implementation from the open-source portfoliolab
Python library from Hudson and Thames [107]. A key drawback of CLA is called
the Curse of Markowtitz, which is that a small change can lead to a very
unstable inverse covariance matrix calculation. Our employment of sparse
inverse covariance matrices via information filtering network produces more
robust results that are more resilient to noise produced by small changes, and
the overall model deliver better performances with respect to the model with
full inverse covarainces. Mathematical and algorithmic details of SLS and CLA
are included in included in Appendix E for reference.

In summary, these two portfolio optimization methodologies output optimal
portfolio weights $\mathbf{W}$ from an input constituted of: (i) a set of
observations $\mathbf{r}_{t}$; (ii) a vector of mean returns
$\boldsymbol{\mu}$; (iii) a covariance $\boldsymbol{\Sigma}$. As we shall see
shortly, in our implementation these inputs are provided in various
combinations including selecting from ICC states and sparsifying.

##  4 Implementation

###  4.1 Data

We carried out several experiments using historical financial time-series data
from three major capital markets: NASDAQ, FTSE and HS300. We selected 100
stocks from each of these three markets during the trading period between
01/01/2010 and 01/01/2020. For each stock, we calculated the daily log-return,
$r_{i}(t)=\log(P_{i}(t))-\log(P_{i}(t-1))$, using closing prices. For the 100
stocks, in the main paper, we selected the largest market capitalization
constituents but in the appendix we repeat the experiments with random
selection obtaining comparable results.

###  4.2 Experiments

The optimal portfolio weights are obtained from the data in the train set and
performances are measured over the test set where portfolio weights are left
unchanged. As performance indicators, we compute portfolio return, portfolio
standard deviation (i.e. volatility) and Sharpe ratio over the investment
horizon (test period). We report the annualized value of these quantities,
estimated as the daily values multiplied by $\sqrt{252}$. For statistical
robustness, for each market, we compute the above portfolio performance
indicators over 100 randomly chosen consecutive train-test periods within the
ten years dataset. Results are reported for the mean performances and the
5%-95% quantile ranges over such re-sampling.

The test set length (investment horizon) was established at 30 days which is a
reasonable value for practical applications, we however also report in
appendix results for horizons of 10, 20 and 100 days finding consistent
results. The train set length was established by performing experiments with
train sets of $L=0.5,1,2,3,4$ years. Figure 1 reports the annualized average
Sharpe Ratio computed on the test set as function of the train set length. One
can observe from the top figure that the lengths between one and two years
yields to consistent good performances. We adopted the period of 2 years as
optimal compromise between statistical robustness and best performances.

In the experiments, we first compute, on the training set, the ICC time
clusters assuming two states and Student-t log-likelihood, Eq.4 as gain
function. The choice of two states is for the seek of simplicity, we tested
also 3 states obtaining inferior but comparable results. We verified that
Student-t likelihood is best performing among the tested gain functions, in
Appendix we report results also for Normal log-likelihoods (Eq.2). The
switching penalty parameter $\gamma$ in ICC was set so that the average
cluster size is around 30 days, i.e. consistent with the 30-day investment
horizon. This selection of average cluster size and investment horizon is a
somehow arbitrary choice based on the effective threshold of the portfolio
performance measured by the Sharpe Ratio. We then labeled ‘Sparse 0’ the state
that is most abundant among the last 20 days of observations at the end of the
train period. Conversely, we labeled ‘Sparse 1’ the other. The term Sparse is
used to indicate that this portfolio uses sparse inverse covariance. To set
such a ‘prevalence period’ of 20 days we first performed a grid search over
the combination of training duration $L=0.5,1,2,3,4$ years, and using
prevalence periods of $10,20,30,100,L/2,L$ days. This search confirmed that
small values of prevalence periods, of $10,20,30$ days, provide better results
than larger prevalence periods. We therefore set a prevalence period of $20$
days as it provides the most consistent results across the grid search and it
is also consistent with the length of the test set.

The bottom plot in Figure 1 reveals that Sparse 0 has consistent better
performances over Full with, best results for training periods of one year.
Let us notice that, having a ICC average cluster size of 30 days, it makes
hard to cluster well a small training period of six months, and often
unbalanced clusters where one cluster dominates the period are obtained. On
the contrary, a large training duration (4 years) makes the model prone to
unnecessary patterns and noise, and in turn reduces performance. Thus we chose
1 year as best compromize for the length of the training set. In Appendix A
and C we see that similar results are obtained for the other two markets
(FTSE, HS300).

(a) Average Sharpe Ratio for 4 States

(b) The relative Sharpe Ratio between Sparse 0 and Full

Figure 1:  Sharpe Ratio for portfolios with 100 largest market capitalization
constituent stocks of NASDAQ Composite optimized using different training set
durations. The top subplot reports the average Sharpe Ratios ($SR$) with error
bars reporting 1 standard deviation, for Full, Sparse, Sparse 0 and 1,
statistics is on 100 training-testing periods chosen at random within the 10
years dataset. The bottom subplot report instead the relative Sharpe Ratios
between Sparse 0 and Full, $SR_{Sparse0}/SR_{Full}$.

We compute optimal portfolios using the two (SLS and CLA) optimization
methods. We trained each optimization method both on the whole train dataset
and, separately, on the two Sparse 0 and Sparse 1 states. We used the sample
means for each of the respective sets and either the ‘full’ sample covariances
(Pearson’s estimate) or the ‘sparse’ sample covariances (TMFG-LoGo estimate
[7, 8]). Therefore, for each optimization method we have four optimized
portfolios: two computed on the whole training set and with full or sparse
inverse covariance (named ‘Full’ and ’Sparse’ respectively); two computed on
the two ICC market states and with sparse inverse covariance (named ‘Sparse 0’
and ‘Sparse 1’). For benchmarking, these portfolios are also compared to a
portfolio with equal weights, $w_{i}=1/n$ named ‘Naive’. Overall, we have
therefore $4\times 3$ plus 1 differently optimized portfolios that are
recomputed 100 times over randomly sampled time-periods. Such optimized
portfolio weights are applied, for each of the three markets, to 100 most
capitalized stocks. In appendix we repeat the experiments for randomly
selected stocks.

##  5 Results

###  5.1 Log-likelihood

We computed the daily Student-t log-likelihood, using Eq.(4), for each of the
30-day investment horizon. Figure 2, reports for the averages of the
differences for each day between the log-likelihood of Sparse 0 and full and
also between Sparse 1 and full. The average is taken over the 100 random re-
sampling.

Figure 2, shows mostly positive gains for Sparse 0 indicating that, for most
days across the investment horizon, it has larger log-likelihoods than Full.
Sparse 1 gain instead reveals mostly negative results against full. This
therefore indicates that while Sparse 0 is, in average, a better model to
describe the multivariate nature of the log-returns in the test set with
respect to Full; instead, Sparse 1 is in average worst. Since both Sparse 0
and 1 were sparsified using TMFG, the difference between them must therefore
be a consequence of clustering. One might note that, even though some Sparse 1
log-likelihoods gains are in the positive domain, they anyway have smaller
magnitudes than their Sparse 0 counterparts. This result clearly shows the
effectiveness and importance of considering market states.

Figure 2:  Student-t log-likelihood for 100 largest market capitalization
constituent stocks of NASDAQ Composite v.s. number of days in the test period
after training. Each bar represents the average gain of the Sparse 0 (green)
or 1 (red) with respect to the Full in each day. Averages are over 100 re-
samplings.

Let us note that the two ICC clusters gather together observations that
maximize in-sample log-likelihood in the respective clustered periods. The
fact that these models (i.e. in-sample means and covariance) from these
clusters still correspond to different log-likelihood performances in the off-
sample test set indicates that the states are still relevant off-sample.
Further, the better off-sample performances of Sparse 0 state indicates
predictability; i.e. if one state outperforms another during the training
period, it will remain better performing throughout the test period.
Furthermore, to illustrate the universality of the log-likelihood results, we
have included similar Student-t log-likelihood (Appendix A) as well as Normal
log-likelihood (Appendix C) plots for random 100 stocks selections for all
three major indices (NASDAQ, FTSE, HS300), where similar patterns are
observed.

###  5.2 Portfolio Performance

We tested portfolio performances over 30-day investment horizon for 100
largest market capitalization constituent stocks of NASDAQ Composite computed
with the four portfolio optimization methods, SLS and CLA and using as inputs
Full, Sparse, Sparse 0 and Sparse 1. We also report the $1/n$ Naive
construction for benchmark. Tables 1 and 2 reports portfolio performances for
the combination of portfolio constructions (column ‘Solver’) and inputs
(column ‘State’). Performances are quantified in terms of annualized portfolio
return, annualized portfolio standard deviation (volatility) and the
annualized Sharpe ratio over 30-day investment horizon. We report the 5% and
95% quantiles and the means computed from the 100 random resampling of
consecutive training-investment periods chosen at random within the 10 years
dataset. The maximum returns and Sharpe Ratios, as well as the minimum
volatility, are highlighted in bold. Thus showing the best performer in each
market-solver combination. In addition, we highlight the minimum 5th
percentile returns to depict the state suffering the least loss, and the
maximum 95th percentile volatility to showcase the most stable state in
extreme market situations.

Solver | State | Return (%) |  (5,95)th percentile | Volatility (%) |  (5,95)th percentile | Sharpe |  (5,95)th percentile
---|---|---|---|---|---|---|---
|  $\frac{1}{n}$ Naive | 14.46 |  (-36,55) | 17.4 |  (14,28) | 1.536 |  (-1.4,4.3)
SLS | Full | 22.71 |  (-28,96) | 19.5 |  (14,28) | 1.627 |  (-1.4,5.4)
SLS | Sparse | 21.81 |  (-23,74) | 17.5 |  (14,26) | 1.764 |  (-1.0,6.2)
SLS | Sparse 0 | 29.04 |  (-6,66) | 16.0 |  (12,25) | 2.478 |  (-0.3,6.9)
SLS | Sparse 1 | 5.35 |  (-49,57) | 19.8 |  (14,34) | 0.978 |  (-2.3,4.6)
CLA | Full | 21.97 |  (-69,97) | 19.5 |  (14,31) | 1.541 |  (-2.1,6.5)
CLA | Sparse | 22.27 |  (-32,85) | 17.0 |  (12,27) | 1.758 |  (-1.9,6.5)
CLA | Sparse 0 | 28.73 |  (-27,76) | 15.8 |  (11,26) | 2.372 |  (-1.5,7.6)
CLA | Sparse 1 | 12.48 |  (-57,86) | 18.7 |  (12,32) | 0.964 |  (-2.9,6.6)
Table 1:  Portfolio performances obtained by using Student-t log-likelihood for ICC clustering. We report annualized return, annualized volatility and annualized Sharpe Ratio computed on 30 days investment period after the 1 year training set. The values are averages and 5th and 95th percentiles computed over 30-day investment horizon from obtained from 100 re-sampling of consecutive training-investment periods chosen at random within the 10 years dataset. The underlying assets are 100 largest market capitalization constituent stocks of NASDAQ Composite. Highlight in bold are return, volatility and Sharpe Ratio indicating the optimal state in each market solver combination, while highlights in 5th return and 95th volatility showcase the extreme behaviours (excluding the state Market). The state $1/n$ Naive is the equally weighted un-optimised portfolio and it is reported as benchmark.  Solver | State | Return (%) |  (5,95)th percentile | Volatility (%) |  (5,95)th percentile | Sharpe |  (5,95)th percentile
---|---|---|---|---|---|---|---
|  $\frac{1}{n}$ Naive | 14.46 |  (-36,55) | 17.4 |  (14,28) | 1.536 |  (-1.4,4.3)
SLS | Full | 22.98 |  (-28,96) | 19.3 |  (14,28) | 1.667 |  (-1.4,5.4)
SLS | Sparse | 21.96 |  (-23,74) | 17.3 |  (14,26) | 1.787 |  (-1.0,6.2)
SLS | Sparse 0 | 29.00 |  (-14,66) | 15.9 |  (12,23) | 2.260 |  (-0.8,4.6)
SLS | Sparse 1 | 6.84 |  (-43,63) | 19.5 |  (14,30) | 0.845 |  (-1.8,4.6)
CLA | Full | 20.63 |  (-76,97) | 19.5 |  (14,31) | 1.456 |  (-3.0,6.5)
CLA | Sparse | 21.15 |  (-53,85) | 17.0 |  (12,27) | 1.678 |  (-2.1,6.5)
CLA | Sparse 0 | 27.08 |  (-14,79) | 15.6 |  (10,30) | 2.175 |  (-0.7,6.6)
CLA | Sparse 1 | 11.54 |  (-69,77) | 18.6 |  (14,36) | 1.028 |  (-2.6,5.6)
Table 2: Portfolio performances obtained by using Normal log-likelihood for
ICC clustering. We report annualized return, annualized volatility and
annualized Sharpe Ratio computed on 30 days investment period after the 1 year
training set. The values are averages and 5th and 95th percentiles computed
over 30-day investment horizon from obtained from 100 re-sampling of
consecutive training-investment periods chosen at random within the 10 years
dataset. The underlying assets are 100 largest market capitalization
constituent stocks of NASDAQ Composite. Highlight in bold are return,
volatility and Sharpe Ratio indicating the optimal state in each market solver
combination, while highlights in 5th return and 95th volatility showcase the
extreme behaviours (excluding the state Market). The state $1/n$ Naive is the
equally weighted un-optimised portfolio and it is reported as benchmark.

From the mean values reported in Tables 1 (Student-t log-likelihood), we
observe that Sparse 0 outperforms Full, and this supremacy dominates for the
two solvers. More specifically, Sparse 0 is on average $29.3\%$, $19.5\%$ and
$53.1\%$ better in return, volatility and Sharpe Ratio than Full across all
two solvers. We observe instead that Sparse 1 is considerably worst than Full
indicating therefore that the significant gain of State 0 come from filtering
out the ‘disadvantageous’ Sparse 1 state rather than sparsification. This is
indeed confirmed by the small observed gains of Sparse over Full. these
results are confirmed by the analysis of the 5th and 95th quantiles where we
notice that Sparse 0 consistently achieves the least minimum extreme loss and
the least maximum extreme risk. Specifically, Sparse 0 on average loses
$66.0\%$ less and is $13.6\%$ less volatile than Full on 5th percentile return
and 95th percentile volatility respectively. In other words, the integrated
clustering portfolio optimization algorithm, ICC-PO, that we proposed can
boost returns with less risk than the traditional benchmark, as well as
provide extra resilience in extreme market situations.

To test the sensitivity of this method to the specific ICC clustering gain
function, we performed the same analysis using normal log-likelihood gain
function for ICC clustering. Results are reported in Table 2. Consistently
with the previous results, we observe $28.6\%$, $18.8\%$ and $42.0\%$
improvements in return, volatility and Sharpe Ratio, with $73.1\%$ and
$10.2\%$ gains in 5th percentile return and 95th percentile volatility. The
comparison illustrates a $11.1\%$ Sharpe Ratio improvement in Student-t log-
likelihood and a $7.1\%$ 5th percentile return advance in Normal log-
likelihood. In other words, Student-t is a better model to the market and
boosts portfolio performance. However, Normal log-likelihood generates a
higher resilience to extreme loss. Since the average gain in return and
volatility are similar in the two cases, the performance difference should
mainly come from general upward-shifted ranges in the Student-t Sharpe Ratio.

Similar tables of optimization results using 10, 20, 30 and 100-day investment
horizons, can be found in Appendix B for Student-t log-likelihood and D for
Normal log-likelihood. These experiments were carried over randomly selected
100 stocks baskets (instead of the 100 most capitalized ones); the set of 100
random stocks was re-chosen for each of the 100 re-sampling. Most of the
general patterns found earlier still hold regardless of the length of the
testing period and underlying assets. The relative difference, namely, the
gain between Sparse 0 and Full remains roughly the same. This consistency
further confirms the generality of our ICC-PO model. In this case we report
only the percentiles of the performance measures because being re-sampled on
different constituents, mean values might be misleading.

##  6 Discussion

The results presented in Section 5.1 quantitatively demonstrate an effective
gain in log-likelihood after applying temporal ICC clustering and computing
the optimized sparse portfolio associated to the most persistent ICC cluster
in the last 20 days of training (the Sparse 0 portfolio). We highlighted that
the additional gain in the ICC-PO construction is mainly a consequence of the
market states clustering and only partially consequence of sparsification.
These results are extremely robust showing comparable patterns across
experiments conducted for three major capital markets, using two different
solving methodologies, adopting four investment horizons and using both
Student-t and normal log-likelihoods gain functions (Appendices B and D). The
results in Appendices B and D obtained for 100 random stocks in the US, the UK
and the Chinese markets show a broader variability but overall well aligned
results. Our results also confirm the observation, by Procacci and Aste [9],
that models with larger likelihood better solve the portfolio optimization
problem.

As for the analysis on the 100 NASDAQ’s most capitalized stocks, also for the
random selection and the three markets we observe that the Normal log-
likelihood, Sparse 0 is $33.8\%$ less than Full in the 5th percentile Return,
whereas the Student-t log-likelihood is only $20.4\%$, which illustrates that
the Normal statistically loses less money in extreme situations. Namely, it
results that there are general advantages in using Student-t over Normal log-
likelihood, yet, the latter performs a better at limiting risks. The edge in
three main performance matrices depicts the Student-t’s better market
modelling property as suggested in the literature, especially for limited
sample daily log-return. In contrast, the mere pitfall in risk measures may
probably come from the fat-tail nature of the Student-t distribution.

It is difficult to assess the efficiency of ICC-PO by direct comparison to the
literature, since our focused result is the relative difference between Sparse
0 from Full. The most informative measurements to the general performance used
widely in the field of portfolio management are Sharpe Ratio (the risk
adjusted return), Jensen’s Alpha (the abnormal return over the theoretical
expectation), Treynor Ratio (the risk adjusted excess return from a risk-free
asset) and Roy Ratio (the risk adjusted excess return from the market index)
[108]. Literature identifies that Sharpe Ratio at values around 1 is commonly
considered as the boundary between a good and bad investment strategy, while
Sharpe Ratio at values around 2 represents an excellent standard, and 3 and
above are more likely to be achieved in a High Frequency Trading (HFT)
strategy [109, 110]. During the 10-year we investigated the annualized Sharpe
Ratio for NASDAQ-100, FTSE-250 and HS300 have been respectively equal to
$1.77$, $0.42$ and $1.07$ [111, 112, 113]. While our results for the various
portfolio construction combinations generally lie in a reasonable range around
these values, we note that the average Sharpe Ratio of the Sparse 0 based on
the 100 largest market capitalization stocks from NASDAQ is $2.425$, as well
as 100 random stocks from NASDAQ is $2.132$, from FTSE is $1.682$ and from
HS300 is $1.814$ greatly exceeding the index’s performances.

Apart from Sharpe Ratio for general performance assessment, risk is often a
critical consideration in portfolio investment due to the risk aversion nature
of investors and the quadratic utility function assumption. Two widely used
risk measures are value at risk (VaR) [114] and probable maximum loss (PML)
[115], which are interpreted as the minimum and the maximum loss expected in a
portfolio over a time period. As a proxy combination of VaR and PML, we
reported the 5th percentile Return in the random re-sampling. The observed
general $66.0\%$, $49.8\%$, $21.6\%$ and $32.4\%$ reductions in loss
respectively for largest-market-capitalization NASDAQ, NASDAQ, FTSE and HS300
are a highly significant result indicating likely large improvements of both
VaR and PML.

Lastly, as ICC-PO is computationally very efficient, it can be easily re-run
for every allocation window making dynamic portfolio allocation easy.

##  7 Conclusions

Portfolio optimization lays at the core of quantitative investment. Automation
in the dynamic allocation process is a challenging goal with a large community
of academics and practitioners dedicated to this task which requires a precise
and accurate modelling of the past market performance and a predictive
inference of the future market state. However, it is never an easy task to
predict the future, not to mention doing so constantly. Explanatory as they
are, only certain signals possess the forecasting ability and normally only
for a limited period of time. Hence, the results of our proposed algorithm
ICC-PO are worth to be mentioned. Indeed, we improve the equal weight
benchmark by over 50% in Sharpe Ratio, obtaining a statistically more robust
and resilient investment performance especially in the extreme market
situations with large reductions in losses.

In this paper we demonstrated that markets can be classified in different
states with distinct statistical properties. By using two states, classified
and clustered using log-likelihood as gain function and sparse inverse
covariance estimation, we have shown that the two clustered states continue to
be distinguishable in log-likelihood after the train (in-sample) period, with
one having systematically larger log-likelihood than the one computed from the
whole, unclustered, training sample. We have shown that the state with larger
log-likelihood tends to be the one also with largest likelihood in the last
period of training, indicating temporal persistence and providing a way for
predictability of the best performing state in the off-sample investment
period. Portfolios optimized with data from the best performing state’s
cluster give significantly better results than portfolios constructed from the
full dataset or the other state. This also confirm the intuitive argument (see
[9]) that a model with larger likelihood must perform better for portfolio
optimization purposes than a model with lower likelihood. These results were
tested extensively across a period of ten years, across three different
markets, with portfolios from two different optimizers, with clustering from
two different log-likelihoods, and both by using a selected group of most
capitalized stocks as well as by random picking a stock basket.

The choice of using two market states has been dictated by simplicity. Future
work will investigate the effect of the number of ICC clusters on the results.
Our results are based on a naive selection of stocks from major indices.
Hence, with a carefully designed portfolio basket, as commonly done in
industrial practices, we expect further improvement of the results. Also a
wider application in asset classes is a straightforward extension of the
method.

##  Appendix A Off sample log-likelihood and performances for Student-t log-
likelihood construction

In this appendix we investigate the effect of the length of the train set on
the Sharpe Ratio performance and off-sample (test set) log-likelihood using
100 randomly selected stocks drawn from NASDAQ, FTSE and HS300. They are in
the similar format as Figure 1 and 2 and demonstrate that identical patterns
exist regardless underlying assets and capital markets. In Figure 4, it is
noticeable that the green bars in general sit above 0 and the red are below 0,
which indicates the Sparse 0 has better off-sample log-likelihood than the
Full, as illustrated in Figure 2. In this appendix we compute Student-t
likelihoods.

Figure 3: Sharpe Ratio for portfolios with constituent stocks of three indices
optimized using different training set durations by using Student-t log-
likelihood for ICC clustering. The right subplot reports the average Sharpe
Ratios ($SR$) with 1 standard deviation for states, statistics is on 100
training-testing periods chosen at random within the 10 years dataset. The
left subplot report instead the relative Sharpe Ratios between Sparse 0 and
Full, $SR_{Sparse0}/SR_{Full}$.

Figure 4:  Student-t log-likelihood for constituent stocks of a) NASDAQ, b)
FTSE and c) HS300 Composite v.s. number of days in the test period after
training. Each bar represents the average gain of the Sparse 0 (green) or 1
(red) with respect to the Full in each day. Averages are over 100 re-
samplings.

##  Appendix B Portfolio Performances

In this appendix we extend the results in the main paper including 10, 20, 30
and 100-day investment horizons based on Student-t log-likelihood. Differently
from the main text, portfolio are constructed with 100 random stocks drawn
from NASDAQ, FTSE and HS300. They serve as complement and comparison for table
1. It is noticeable that although shorter testing period yields numerically
larger Sharpe Ratio due to a possible low Volatility and a overestimation of
annualized Return on small sample, the relative difference, namely, the gain
between the Sparse 0 and the Full remains roughly the same. This consistency
further confirms the generality of our model. Besides, the patterns in three
tables are generally consistent to the findings in section 5.2.

Market | Solver | State | Return (5, 95) percentile | Volatility (5, 95) percentile | Sharpe (5, 95) percentile
---|---|---|---|---|---
NASDAQ |  |  $\frac{1}{n}$ Naive | (-171,206) | (14.0,40.0) | (-8.7,12.5)
NASDAQ | SLS | Full | (-210,138) | (13.0,69.0) | (-7.2,9.1)
NASDAQ | SLS | Sparse | (-213,122) | (12.0,59.0) | (-7.6,9.1)
NASDAQ | SLS | Sparse 0 | (-157,125) | (9.0,52.0) | (-5.1,10.4)
NASDAQ | SLS | Sparse 1 | (-352,153) | (13.0,49.0) | (-7.1,7.6)
NASDAQ | CLA | Full | (-198,190) | (13.0,57.0) | (-6.6,10.1)
NASDAQ | CLA | Sparse | (-172,223) | (13.0,51.0) | (-8.4,10.3)
NASDAQ | CLA | Sparse 0 | (-181,185) | (10.0,49.0) | (-5.2,13.4)
NASDAQ | CLA | Sparse 1 | (-198,200) | (13.0,66.0) | (-8.3,7.4)
FTSE |  |  $\frac{1}{n}$ Naive | (-125,140) | (9.0,28.0) | (-9.0,17.0)
FTSE | SLS | Full | (-91,147) | (7.0,27.0) | (-8.1,20.5)
FTSE | SLS | Sparse | (-89,125) | (7.0,26.0) | (-8.8,15.5)
FTSE | SLS | Sparse 0 | (-64,150) | (8.0,22.0) | (-5.9,18.8)
FTSE | SLS | Sparse 1 | (-113,116) | (9.0,26.0) | (-8.9,11.6)
FTSE | CLA | Full | (-147,137) | (8.0,22.0) | (-6.9,17.6)
FTSE | CLA | Sparse | (-138,122) | (7.0,25.0) | (-9.0,18.0)
FTSE | CLA | Sparse 0 | (-119,129) | (7.0,23.0) | (-6.4,21.5)
FTSE | CLA | Sparse 1 | (-171,149) | (9.0,22.0) | (-10.5,16.3)
HS300 |  |  $\frac{1}{n}$ Naive | (-228,198) | (10.0,60.0) | (-7.7,10.8)
HS300 | SLS | Full | (-250,216) | (12.0,42.0) | (-8.3,15.8)
HS300 | SLS | Sparse | (-283,252) | (11.0,44.0) | (-8.1,15.3)
HS300 | SLS | Sparse 0 | (-181,284) | (11.0,46.0) | (-6.0,16.0)
HS300 | SLS | Sparse 1 | (-317,192) | (13.0,58.0) | (-7.9,9.9)
HS300 | CLA | Full | (-250,216) | (12.0,42.0) | (-8.3,15.8)
HS300 | CLA | Sparse | (-283,252) | (11.0,44.0) | (-8.1,15.3)
HS300 | CLA | Sparse 0 | (-194,284) | (11.0,41.0) | (-4.9,14.2)
HS300 | CLA | Sparse 1 | (-277,223) | (13.0,53.0) | (-9.2,8.6)
Table 3: Portfolio performances obtained by using Student-t log-likelihood for ICC clustering. We report annualized return, annualized volatility and annualized Sharpe Ratio computed on 10 days investment period after the 1 year training set. The values are averages and 5th and 95th percentiles computed over 10-day investment horizon from obtained from 100 re-sampling of consecutive training-investment periods chosen at random within the 10 years dataset. The underlying assets are constituent stocks of NASDAQ, FTSE and HS300. Highlight in bold are return, volatility and Sharpe Ratio indicating the optimal state in each market solver combination, while highlights in 5th return and 95th volatility showcase the extreme behaviours (excluding the state Market). The state $1/n$ Naive is the equally weighted un-optimised portfolio and it is reported as benchmark.  Market | Solver | State | Return (5, 95) percentile | Volatility (5, 95) percentile | Sharpe (5, 95) percentile
---|---|---|---|---|---
NASDAQ |  |  $\frac{1}{n}$ Naive | (-112,180) | (14.0,84.0) | (-4.7,5.9)
NASDAQ | SLS | Full | (-126,131) | (14.0,85.0) | (-4.4,6.5)
NASDAQ | SLS | Sparse | (-133,127) | (13.0,53.0) | (-5.1,6.5)
NASDAQ | SLS | Sparse 0 | (-124,120) | (12.0,83.0) | (-4.4,7.5)
NASDAQ | SLS | Sparse 1 | (-198,78) | (15.0,50.0) | (-6.1,4.5)
NASDAQ | CLA | Full | (-147,127) | (13.0,84.0) | (-4.7,5.4)
NASDAQ | CLA | Sparse | (-149,135) | (13.0,53.0) | (-4.6,6.4)
NASDAQ | CLA | Sparse 0 | (-101,144) | (12.0,46.0) | (-2.8,7.9)
NASDAQ | CLA | Sparse 1 | (-187,88) | (15.0,62.0) | (-5.4,4.1)
FTSE |  |  $\frac{1}{n}$ Naive | (-77,111) | (11.0,26.0) | (-4.7,8.1)
FTSE | SLS | Full | (-58,94) | (9.0,26.0) | (-4.5,11.4)
FTSE | SLS | Sparse | (-59,95) | (10.0,25.0) | (-4.9,10.6)
FTSE | SLS | Sparse 0 | (-39,96) | (9.0,17.0) | (-3.4,11.4)
FTSE | SLS | Sparse 1 | (-72,75) | (9.0,22.0) | (-5.4,7.5)
FTSE | CLA | Full | (-82,84) | (10.0,25.0) | (-5.6,10.3)
FTSE | CLA | Sparse | (-62,81) | (10.0,20.0) | (-6.2,10.1)
FTSE | CLA | Sparse 0 | (-59,79) | (9.0,22.0) | (-4.3,11.7)
FTSE | CLA | Sparse 1 | (-100,80) | (10.0,23.0) | (-6.0,9.0)
HS300 |  |  $\frac{1}{n}$ Naive | (-102,236) | (11.0,44.0) | (-4.4,9.3)
HS300 | SLS | Full | (-133,246) | (15.0,42.0) | (-5.1,10.7)
HS300 | SLS | Sparse | (-125,234) | (14.0,42.0) | (-5.0,10.3)
HS300 | SLS | Sparse 0 | (-101,218) | (13.0,40.0) | (-2.8,11.3)
HS300 | SLS | Sparse 1 | (-142,202) | (13.0,46.0) | (-5.2,7.9)
HS300 | CLA | Full | (-133,246) | (15.0,42.0) | (-5.1,10.7)
HS300 | CLA | Sparse | (-125,234) | (14.0,42.0) | (-5.0,10.3)
HS300 | CLA | Sparse 0 | (-62,247) | (12.0,41.0) | (-2.8,10.2)
HS300 | CLA | Sparse 1 | (-131,187) | (14.0,48.0) | (-5.0,8.2)
Table 4: Portfolio performances obtained by using Student-t log-likelihood for ICC clustering. We report annualized return, annualized volatility and annualized Sharpe Ratio computed on 20 days investment period after the 1 year training set. The values are averages and 5th and 95th percentiles computed over 20-day investment horizon from obtained from 100 re-sampling of consecutive training-investment periods chosen at random within the 10 years dataset. The underlying assets are constituent stocks of NASDAQ, FTSE and HS300. Highlight in bold are return, volatility and Sharpe Ratio indicating the optimal state in each market solver combination, while highlights in 5th return and 95th volatility showcase the extreme behaviours (excluding the state Market). The state $1/n$ Naive is the equally weighted un-optimised portfolio and it is reported as benchmark.  Market | Solver | State | Return (5, 95) percentile | Volatility (5, 95) percentile | Sharpe (5, 95) percentile
---|---|---|---|---|---
NASDAQ |  |  $\frac{1}{n}$ Naive | (-110,62) | (13.0,35.0) | (-4.5,5.7)
NASDAQ | SLS | Sparse | (-133,115) | (14.0,73.0) | (-4.7,4.7)
NASDAQ | SLS | Sparse 0 | (-99,112) | (14.0,71.0) | (-2.9,6.4)
NASDAQ | SLS | Sparse 1 | (-128,72) | (14.0,63.0) | (-4.4,4.4)
NASDAQ | CLA | Full | (-87,121) | (16.0,69.0) | (-3.0,5.8)
NASDAQ | CLA | Sparse | (-86,130) | (15.0,72.0) | (-2.6,6.2)
NASDAQ | CLA | Sparse 0 | (-40,134) | (14.0,74.0) | (-2.5,6.5)
NASDAQ | CLA | Sparse 1 | (-101,86) | (15.0,73.0) | (-3.3,4.0)
FTSE |  |  $\frac{1}{n}$ Naive | (-69,90) | (11.0,28.0) | (-3.0,6.6)
FTSE | SLS | Full | (-63,80) | (10.0,26.0) | (-4.7,8.0)
FTSE | SLS | Sparse | (-56,73) | (10.0,22.0) | (-4.8,8.3)
FTSE | SLS | Sparse 0 | (-52,87) | (9.0,20.0) | (-3.5,7.3)
FTSE | SLS | Sparse 1 | (-68,62) | (11.0,22.0) | (-4.5,6.3)
FTSE | CLA | Full | (-56,79) | (10.0,24.0) | (-4.8,8.0)
FTSE | CLA | Sparse | (-53,73) | (10.0,20.0) | (-4.6,9.1)
FTSE | CLA | Sparse 0 | (-47,72) | (9.0,20.0) | (-4.2,9.0)
FTSE | CLA | Sparse 1 | (-81,64) | (11.0,24.0) | (-5.8,7.1)
HS300 |  |  $\frac{1}{n}$ Naive | (-90,172) | (11.0,38.0) | (-3.3,7.1)
HS300 | SLS | Full | (-127,173) | (16.0,40.0) | (-4.0,6.3)
HS300 | SLS | Sparse | (-98,160) | (15.0,36.0) | (-4.1,7.2)
HS300 | SLS | Sparse 0 | (-65,172) | (13.0,43.0) | (-2.9,7.3)
HS300 | SLS | Sparse 1 | (-118,142) | (15.0,45.0) | (-3.8,5.7)
HS300 | CLA | Full | (-127,173) | (16.0,40.0) | (-4.0,6.2)
HS300 | CLA | Sparse | (-98,160) | (15.0,36.0) | (-4.1,7.2)
HS300 | CLA | Sparse 0 | (-64,173) | (13.0,38.0) | (-2.7,7.4)
HS300 | CLA | Sparse 1 | (-98,142) | (15.0,36.0) | (-4.2,5.4)
Table 5: Portfolio performances obtained by using Student-t log-likelihood for ICC clustering. We report annualized return, annualized volatility and annualized Sharpe Ratio computed on 30 days investment period after the 1 year training set. The values are averages and 5th and 95th percentiles computed over 30-day investment horizon from obtained from 100 re-sampling of consecutive training-investment periods chosen at random within the 10 years dataset. The underlying assets are constituent stocks of NASDAQ, FTSE and HS300. Highlight in bold are return, volatility and Sharpe Ratio indicating the optimal state in each market solver combination, while highlights in 5th return and 95th volatility showcase the extreme behaviours (excluding the state Market). The state $1/n$ Naive is the equally weighted un-optimised portfolio and it is reported as benchmark.  Market | Solver | State | Return (5, 95) percentile | Volatility (5, 95) percentile | Sharpe (5, 95) percentile
---|---|---|---|---|---
NASDAQ |  |  $\frac{1}{n}$ Naive | (-167,172) | (11.0,48.0) | (-7.6,8.4)
NASDAQ | SLS | Full | (-210,138) | (13.0,69.0) | (-7.2,9.1)
NASDAQ | SLS | Sparse | (-213,122) | (12.0,59.0) | (-7.6,9.1)
NASDAQ | SLS | Sparse 0 | (-189,160) | (11.0,41.0) | (-5.4,8.4)
NASDAQ | SLS | Sparse 1 | (-290,114) | (13.0,66.0) | (-8.5,5.6)
NASDAQ |  | Full | (-198,190) | (13.0,57.0) | (-6.6,10.1)
NASDAQ | CLA | Sparse | (-172,223) | (13.0,51.0) | (-8.4,10.3)
NASDAQ | CLA | Sparse 0 | (-165,187) | (11.0,46.0) | (-6.1,14.2)
NASDAQ | CLA | Sparse 1 | (-257,166) | (13.0,62.0) | (-6.8,6.6)
FTSE |  |  $\frac{1}{n}$ Naive | (-186,140) | (8.0,22.0) | (-10.3,23.0)
FTSE | SLS | Full | (-91,147) | (7.0,27.0) | (-8.1,20.5)
FTSE | SLS | Sparse | (-89,125) | (7.0,26.0) | (-8.8,15.5)
FTSE | SLS | Sparse 0 | (-75,161) | (7.0,21.0) | (-7.4,29.2)
FTSE | SLS | Sparse 1 | (-110,123) | (9.0,27.0) | (-9.5,13.6)
FTSE | CLA | Full | (-147,137) | (8.0,22.0) | (-6.9,17.6)
FTSE | CLA | Sparse | (-138,122) | (7.0,25.0) | (-9.0,18.0)
FTSE | CLA | Sparse 0 | (-80,145) | (8.0,23.0) | (-5.7,24.2)
FTSE | CLA | Sparse 1 | (-194,138) | (9.0,25.0) | (-12.0,16.3)
HS300 |  |  $\frac{1}{n}$ Naive | (-228,198) | (10.0,60.0) | (-7.7,10.7)
HS300 | SLS | Full | (-250,216) | (12.0,42.0) | (-8.3,15.8)
HS300 | SLS | Sparse | (-283,252) | (11.0,44.0) | (-8.1,15.3)
HS300 | SLS | Sparse 0 | (-237,249) | (11.0,50.0) | (-5.6,16.5)
HS300 | SLS | Sparse 1 | (-237,193) | (14.0,45.0) | (-8.4,9.3)
HS300 | CLA | Full | (-250,216) | (12.0,42.0) | (-8.3,15.8)
HS300 | CLA | Sparse | (-283,252) | (11.0,44.0) | (-8.1,15.3)
HS300 | CLA | Sparse 0 | (-186,298) | (10.0,44.0) | (-7.6,13.9)
HS300 | CLA | Sparse 1 | (-302,186) | (13.0,56.0) | (-7.5,9.5)
Table 6: Portfolio performances obtained by using Student-t log-likelihood for
ICC clustering. We report annualized return, annualized volatility and
annualized Sharpe Ratio computed on 100 days investment period after the 1
year training set. The values are averages and 5th and 95th percentiles
computed over 100-day investment horizon from obtained from 100 re-sampling of
consecutive training-investment periods chosen at random within the 10 years
dataset. The underlying assets are constituent stocks of NASDAQ, FTSE and
HS300. Highlight in bold are return, volatility and Sharpe Ratio indicating
the optimal state in each market solver combination, while highlights in 5th
return and 95th volatility showcase the extreme behaviours (excluding the
state Market). The state $1/n$ Naive is the equally weighted un-optimised
portfolio and it is reported as benchmark.

##  Appendix C Normal Log-likelihood: training duration and Off-sample Log-
likelihood

This Appendix C section includes Sharpe ratio against training duration plots
and off-sample Normal log-likelihood plots of 100 random stocks drawn from
NASDAQ, FTSE and HS300. They are in the similar format as Figure 1 and 2 and
demonstrate that identical patterns exist regardless underlying assets and
capital markets. In Figure 4, it is noticeable that the green bars in general
sit above 0 and the red are below 0, which indicates the Sparse 0 has better
off-sample log-likelihood than the Full, as illustrated in Figure 2.

Figure 5: Sharpe Ratio for portfolios with constituent stocks of three indices
optimized using different training set durations by using Normal log-
likelihood for ICC clustering. The right subplot reports the average Sharpe
Ratios ($SR$) with 1 standard deviation for states, statistics is on 100
training-testing periods chosen at random within the 10 years dataset. The
left subplot report instead the relative Sharpe Ratios between Sparse 0 and
Full, $SR_{Sparse0}/SR_{Full}$.

Figure 6:  Normal log-likelihood for constituent stocks of a) NASDAQ, b) FTSE
and c) HS300 Composite v.s. number of days in the test period after training.
Each bar represents the average gain of the Sparse 0 (green) or 1 (red) with
respect to the Full in each day. Averages are over 100 re-samplings.

##  Appendix D Off sample log-likelihood and performances for Normal log-
likelihood construction

In this appendix we perform the same kind of investigations as in the previous
appendix but ICC is computed using Normal log-likelihood. We notice similar
patterns but the Student-t log-likelihood result are more significant.
However, the Normal log-likelihood performs better in risk matrices.

Market | Solver | State | Return (5, 95) percentile | Volatility (5, 95) percentile | Sharpe (5, 95) percentile
---|---|---|---|---|---
NASDAQ |  |  $\frac{1}{n}$ Naive | (-171,206) | (14.0,40.0) | (-8.7,12.5)
NASDAQ | SLS | Full | (-192,190) | (13.0,57.0) | (-5.4,10.1)
NASDAQ | SLS | Sparse | (-160,223) | (13.0,51.0) | (-7.4,10.3)
NASDAQ | SLS | Sparse 0 | (-144,174) | (11.0,49.0) | (-5.0,12.6)
NASDAQ | SLS | Sparse 1 | (-181,218) | (14.0,59.0) | (-8.7,7.8)
NASDAQ | CLA | Full | (-192,198) | (13.0,57.0) | (-5.4,10.1)
NASDAQ | CLA | Sparse | (-160,223) | (13.0,51.0) | (-7.4,10.3)
NASDAQ | CLA | Sparse 0 | (-169,171) | (12.0,36.0) | (-5.0,14.2)
NASDAQ | CLA | Sparse 1 | (-256,144) | (14.0,67.0) | (-6.5,6.5)
FTSE |  |  $\frac{1}{n}$ Naive | (-161,117) | (7.0,34.0) | (-9.2,15.4)
FTSE | SLS | Full | (-163,116) | (8.0,28.0) | (-8.7,14.1)
FTSE | SLS | Sparse | (-148,108) | (8.0,33.0) | (-8.9,14.2)
FTSE | SLS | Sparse 0 | (-111,138) | (7.0,22.0) | (-7.1,18.8)
FTSE | SLS | Sparse 1 | (-199,118) | (9.0,40.0) | (-12.1,13.0)
FTSE | CLA | Full | (-163,116) | (8.0,28.0) | (-8.7,14.1)
FTSE | CLA | Sparse | (-148,108) | (8.0,33.0) | (-8.9,14.2)
FTSE | CLA | Sparse 0 | (-111,146) | (7.0,21.0) | (-8.3,18.0)
FTSE | CLA | Sparse 1 | (-176,123) | (8.0,36.0) | (-11.0,13.7)
HS300 |  |  $\frac{1}{n}$ Naive | (-228,198) | (10.0,60.0) | (-7.7,10.8)
HS300 | SLS | Full | (-250,216) | (12.0,42.0) | (-8.3,15.8)
HS300 | SLS | Sparse | (-283,252) | (11.0,44.0) | (-8.1,15.3)
HS300 | SLS | Sparse 0 | (-165,276) | (11.0,42.0) | (-5.6,15.2)
HS300 | SLS | Sparse 1 | (-289,176) | (13.0,44.0) | (-8.3,7.9)
HS300 | CLA | Full | (-250,216) | (12.0,42.0) | (-8.3,15.8)
HS300 | CLA | Sparse | (-283,252) | (11.0,44.0) | (-8.1,15.3)
HS300 | CLA | Sparse 0 | (-131,250) | (11.0,45.0) | (-5.3,18.8)
HS300 | CLA | Sparse 1 | (-331,218) | (12.0,58.0) | (-8.9,9.9)
Table 7: Portfolio performances obtained by using Normal log-likelihood for ICC clustering. We report annualized return, annualized volatility and annualized Sharpe Ratio computed on 10 days investment period after the 1 year training set. The values are averages and 5th and 95th percentiles computed over 10-day investment horizon from obtained from 100 re-sampling of consecutive training-investment periods chosen at random within the 10 years dataset. The underlying assets are constituent stocks of NASDAQ, FTSE and HS300. Highlight in bold are return, volatility and Sharpe Ratio indicating the optimal state in each market solver combination, while highlights in 5th return and 95th volatility showcase the extreme behaviours (excluding the state Market). The state $1/n$ Naive is the equally weighted un-optimised portfolio and it is reported as benchmark.  Market | Solver | State | Return (5, 95) percentile | Volatility (5, 95) percentile | Sharpe (5, 95) percentile
---|---|---|---|---|---
NASDAQ |  |  $\frac{1}{n}$ Naive | (-129,147) | (14.0,36.0) | (-4.9,7.8)
NASDAQ | SLS | Full | (-147,124) | (13.0,84.0) | (-4.7,5.4)
NASDAQ | SLS | Sparse | (-149,135) | (13.0,53.0) | (-4.6,6.4)
NASDAQ | SLS | Sparse 0 | (-98,133) | (12.0,64.0) | (-3.7,8.4)
NASDAQ | SLS | Sparse 1 | (-147,101) | (14.0,71.0) | (-5.6,5.3)
NASDAQ | CLA | Full | (-147,127) | (13.0,84.0) | (-4.7,5.4)
NASDAQ | CLA | Sparse | (-149,135) | (13.0,53.0) | (-4.6,6.4)
NASDAQ | CLA | Sparse 0 | (-95,127) | (11.0,51.0) | (-3.1,7.8)
NASDAQ | CLA | Sparse 1 | (-149,111) | (14.0,69.0) | (-4.8,5.5)
FTSE |  |  $\frac{1}{n}$ Naive | (-83,104) | (10.0,32.0) | (-6.9,9.1)
FTSE | SLS | Full | (-79,100) | (9.0,30.0) | (-5.5,9.5)
FTSE | SLS | Sparse | (-63,84) | (9.0,27.0) | (-5.6,9.6)
FTSE | SLS | Sparse 0 | (-49,82) | (9.0,27.0) | (-5.7,11.7)
FTSE | SLS | Sparse 1 | (-92,94) | (11.0,34.0) | (-5.6,8.0)
FTSE | CLA | Full | (-79,100) | (9.0,30.0) | (-5.5,9.5)
FTSE | CLA | Sparse | (-63,84) | (9.0,27.0) | (-5.6,9.6)
FTSE | CLA | Sparse 0 | (-68,82) | (9.0,23.0) | (-5.2,10.7)
FTSE | CLA | Sparse 1 | (-110,101) | (9.0,29.0) | (-7.3,8.3)
HS300 |  |  $\frac{1}{n}$ Naive | (-102,236) | (11.0,44.0) | (-4.4,9.3)
HS300 | SLS | Full | (-133,246) | (15.0,42.0) | (-5.1,10.7)
HS300 | SLS | Sparse | (-125,234) | (14.0,42.0) | (-5.0,10.3)
HS300 | SLS | Sparse 0 | (-92,231) | (13.0,45.0) | (-3.8,11.4)
HS300 | SLS | Sparse 1 | (-143,204) | (14.0,43.0) | (-5.6,7.4)
HS300 | CLA | Full | (-133,246) | (15.0,42.0) | (-5.1,10.7)
HS300 | CLA | Sparse | (-125,234) | (14.0,42.0) | (-5.0,10.3)
HS300 | CLA | Sparse 0 | (-94,223) | (12.0,46.0) | (-3.0,10.0)
HS300 | CLA | Sparse 1 | (-146,209) | (12.0,39.0) | (-5.5,8.0)
Table 8: Portfolio performances obtained by using Normal log-likelihood for ICC clustering. We report annualized return, annualized volatility and annualized Sharpe Ratio computed on 20 days investment period after the 1 year training set. The values are averages and 5th and 95th percentiles computed over 20-day investment horizon from obtained from 100 re-sampling of consecutive training-investment periods chosen at random within the 10 years dataset. The underlying assets are constituent stocks of NASDAQ, FTSE and HS300. Highlight in bold are return, volatility and Sharpe Ratio indicating the optimal state in each market solver combination, while highlights in 5th return and 95th volatility showcase the extreme behaviours (excluding the state Market). The state $1/n$ Naive is the equally weighted un-optimised portfolio and it is reported as benchmark.  Market | Solver | State | Return (5, 95) percentile | Volatility (5, 95) percentile | Sharpe (5, 95) percentile
---|---|---|---|---|---
NASDAQ |  |  $\frac{1}{n}$ Naive | (-112,137) | (15.0,41.0) | (-3.3,7.0)
NASDAQ | SLS | Full | (-105,113) | (16.0,71.0) | (-3.0,5.0)
NASDAQ | SLS | Sparse | (-135,120) | (15.0,73.0) | (-3.2,5.5)
NASDAQ | SLS | Sparse 0 | (-52,116) | (12.0,78.0) | (-2.6,5.6)
NASDAQ | SLS | Sparse 1 | (-169,86) | (16.0,74.0) | (-3.9,3.3)
NASDAQ | CLA | Full | (-105,113) | (16.0,71.0) | (-3.0,5.0)
NASDAQ | CLA | Sparse | (-135,120) | (15.0,73.0) | (-3.2,5.5)
NASDAQ | CLA | Sparse 0 | (-61,116) | (14.0,71.0) | (-2.5,6.2)
NASDAQ | CLA | Sparse 1 | (-146,78) | (15.0,79.0) | (-4.1,3.7)
FTSE |  |  $\frac{1}{n}$ Naive | (-46,68) | (11.0,31.0) | (-3.0,5.7)
FTSE | SLS | Full | (-45,75) | (11.0,26.0) | (-2.9,6.8)
FTSE | SLS | Sparse | (-48,69) | (11.0,24.0) | (-3.3,7.7)
FTSE | SLS | Sparse 0 | (-32,68) | (11.0,21.0) | (-2.6,8.2)
FTSE | SLS | Sparse 1 | (-69,67) | (11.0,29.0) | (-3.8,6.6)
FTSE | CLA | Full | (-45,75) | (11.0,26.0) | (-3.0,6.8)
FTSE | CLA | Sparse | (-48,69) | (11.0,24.0) | (-3.3,7.7)
FTSE | CLA | Sparse 0 | (-43,67) | (11.0,22.0) | (-3.0,7.6)
FTSE | CLA | Sparse 1 | (-56,58) | (12.0,29.0) | (-3.2,5.3)
HS300 |  |  $\frac{1}{n}$ Naive | (-91,165) | (11.0,51.0) | (-3.3,6.0)
HS300 | SLS | Full | (-127,168) | (16.0,42.0) | (-4.0,6.3)
HS300 | SLS | Sparse | (-94,163) | (15.0,38.0) | (-4.1,7.1)
HS300 | SLS | Sparse 0 | (-78,182) | (12.0,38.0) | (-3.0,6.5)
HS300 | SLS | Sparse 1 | (-121,135) | (14.0,55.0) | (-3.9,5.5)
HS300 | CLA | Full | (-127,168) | (16.0,42.0) | (-4.0,6.3)
HS300 | CLA | Sparse | (-94,163) | (15.0,38.0) | (-4.1,7.1)
HS300 | CLA | Sparse 0 | (-54,165) | (12.0,51.0) | (-2.3,7.8)
HS300 | CLA | Sparse 1 | (-110,138) | (14.0,43.0) | (-4.2,6.0)
Table 9: Portfolio performances obtained by using Normal log-likelihood for ICC clustering. We report annualized return, annualized volatility and annualized Sharpe Ratio computed on 30 days investment period after the 1 year training set. The values are averages and 5th and 95th percentiles computed over 30-day investment horizon from obtained from 100 re-sampling of consecutive training-investment periods chosen at random within the 10 years dataset. The underlying assets are constituent stocks of NASDAQ, FTSE and HS300. Highlight in bold are return, volatility and Sharpe Ratio indicating the optimal state in each market solver combination, while highlights in 5th return and 95th volatility showcase the extreme behaviours (excluding the state Market). The state $1/n$ Naive is the equally weighted un-optimised portfolio and it is reported as benchmark.  Market | Solver | State | Return (5, 95) percentile | Volatility (5, 95) percentile | Sharpe (5, 95) percentile
---|---|---|---|---|---
NASDAQ |  |  $\frac{1}{n}$ Naive | (-25,42) | (15.0,33.0) | (-1.3,2.6)
NASDAQ | SLS | Full | (-33,52) | (17.0,53.0) | (-1.4,2.8)
NASDAQ | SLS | Sparse | (-24,56) | (16.0,36.0) | (-1.2,3.1)
NASDAQ | SLS | Sparse 0 | (-17,38) | (14.0,37.0) | (-0.8,2.3)
NASDAQ | SLS | Sparse 1 | (-41,51) | (17.0,51.0) | (-1.9,2.1)
NASDAQ | CLA | Full | (-33,52) | (17.0,53.0) | (-1.4,2.8)
NASDAQ | CLA | Sparse | (-24,56) | (16.0,36.0) | (-1.2,3.1)
NASDAQ | CLA | Sparse 0 | (-24,52) | (15.0,36.0) | (-1.2,2.4)
NASDAQ | CLA | Sparse 1 | (-34,34) | (16.0,49.0) | (-1.7,1.7)
FTSE |  |  $\frac{1}{n}$ Naive | (-37,49) | (11.0,30.0) | (-2.3,5.3)
FTSE | SLS | Full | (-34,55) | (11.0,26.0) | (-2.2,5.0)
FTSE | SLS | Sparse | (-38,55) | (11.0,26.0) | (-2.1,5.0)
FTSE | SLS | Sparse 0 | (-27,44) | (10.0,18.0) | (-1.9,6.3)
FTSE | SLS | Sparse 1 | (-39,59) | (12.0,30.0) | (-2.3,5.1)
FTSE | CLA | Full | (-34,55) | (11.0,26.0) | (-2.2,5.0)
FTSE | CLA | Sparse | (-38,55) | (11.0,26.0) | (-2.1,5.0)
FTSE | CLA | Sparse 0 | (-28,57) | (11.0,18.0) | (-1.9,7.2)
FTSE | CLA | Sparse 1 | (-45,44) | (11.0,34.0) | (-2.6,4.5)
HS300 |  |  $\frac{1}{n}$ Naive | (-47,112) | (15.0,51.0) | (-1.9,5.0)
HS300 | SLS | Full | (-68,77) | (17.0,52.0) | (-2.0,3.8)
HS300 | SLS | Sparse | (-50,96) | (16.0,43.0) | (-1.8,4.9)
HS300 | SLS | Sparse 0 | (-60,109) | (15.0,46.0) | (-1.8,5.5)
HS300 | SLS | Sparse 1 | (-74,114) | (17.0,48.0) | (-2.0,4.1)
HS300 | CLA | Full | (-68,77) | (17.0,52.0) | (-2.0,3.8)
HS300 | CLA | Sparse | (-50,96) | (16.0,43.0) | (-1.8,4.9)
HS300 | CLA | Sparse 0 | (-54,86) | (15.0,51.0) | (-1.4,5.3)
HS300 | CLA | Sparse 1 | (-71,91) | (17.0,46.0) | (-2.0,4.3)
Table 10: Portfolio performances obtained by using Normal log-likelihood for
ICC clustering. We report annualized return, annualized volatility and
annualized Sharpe Ratio computed on 100 days investment period after the 1
year training set. The values are averages and 5th and 95th percentiles
computed over 100-day investment horizon from obtained from 100 re-sampling of
consecutive training-investment periods chosen at random within the 10 years
dataset. The underlying assets are constituent stocks of NASDAQ, FTSE and
HS300. Highlight in bold are return, volatility and Sharpe Ratio indicating
the optimal state in each market solver combination, while highlights in 5th
return and 95th volatility showcase the extreme behaviours (excluding the
state Market). The state $1/n$ Naive is the equally weighted un-optimised
portfolio and it is reported as benchmark.

##  Appendix E Portfolio Optimization

In the original Markowitz’s mean variance optimization approach, the portfolio
weights $\mathbf{W}=(w_{1},...,w_{n})\in\mathbb{R}^{1\times n}$ are chosen in
order to minimize portfolio’s variance $\sigma_{p}^{2}=\mathbf{W\Sigma
W}^{\top}$ for a given value, of the portfolio’s expected return
$\boldsymbol{\mu}{\mathbf{W}}^{\top}=\bar{r}_{p}$. Specifically,

$\displaystyle\mathbf{W}^{*}=\min_{{\mathbf{W}}}$
$\displaystyle\mathbf{W\Sigma W}^{\top}$ (6) s.t
$\displaystyle\mathbf{\mathds{1}W}^{\top}=1,$ and
$\displaystyle\boldsymbol{\mu}{\mathbf{W}}^{\top}=\bar{r}_{p},$

The exact solution can be obtained analytically by setting to zero the
derivatives with respect to $\mathbf{W}$, using the Lagrange multiplier
technique to account for the constraints. Namely the minimum of the following
Lagrangian is computed

$L({\mathbf{W}},\lambda)=\mathbf{W\Sigma
W}^{\top}+\lambda_{1}\boldsymbol{\mu}{\mathbf{W}}^{\top}+\lambda_{2}\mathbf{\mathds{1}W}^{\top},$
(7)

and the solution is

$\mathbf{W}^{*}=\mathbf{\Sigma^{-1}}(\lambda_{1}\boldsymbol{\mu}+\lambda_{2}\mathbf{\mathds{1}})^{\top},$
(8)

where $\lambda_{1}$ and $\lambda_{2}$ are the Lagrange multipliers.

The sequential least square quadratic programming (SLS) [99, 100, 101] is
considered to be one of the most efficient computational method to solve
general nonlinear constrained optimization problems. Jackson et al. and
Cesarone et al. demonstrate its effectiveness in finance [102, 103]. SLS
solves the optimization problem iteratively with a gradient descent strategy
starting with an initial setting ${\mathbf{W}}^{0}$, and updating
${\mathbf{W}}^{k+1}$ from ${\mathbf{W}}^{k}$ by:

${\mathbf{W}}^{k+1}={\mathbf{W}}^{k}+\alpha^{k}{\mathbf{d}}^{k}$ (9)

where ${\mathbf{d}}^{k}$ is the search direction at the $k$-th step and
$\alpha^{k}$ is the associated step size. In each iteration, the descent
search direction, ${\mathbf{d}}$, is determined by the solution of a sub-
problem. Given the loss function

$f({\mathbf{W}})=\mathbf{W\Sigma W}^{\top}$ (10)

that we want to minimize under a set of non-liner constraints
$g_{j}({\mathbf{W}})=0$ for $j\in[1,m_{e}]$ and $g_{j}({\mathbf{W}})\geq 0$
for $j\in[m_{e}+1,m]$, at each iteration, the problem of finding the optimal
descent direction can be addressed by solving the standard quadratic
programming sub-problem [116]:

$\displaystyle\mathbf{d}^{k+1}=\min_{{\mathbf{d}}}$
$\displaystyle{\frac{1}{2}{\mathbf{d}}\nabla^{2}L({\mathbf{W}}^{k},\boldsymbol{\lambda}){\mathbf{d}}^{\top}+\nabla
f({\mathbf{W}}^{k}){\mathbf{d}}^{\top}}$ (11) s.t $\displaystyle\nabla
g_{j}({\mathbf{W}}^{k}){\mathbf{d}}^{\top}+g_{j}({\mathbf{W}}^{k})=0,\;\;\;j=1,...,m_{e}$
$\displaystyle\nabla
g_{j}({\mathbf{W}}^{k}){\mathbf{d}}^{\top}+g_{j}({\mathbf{W}}^{k})\geq
0,\;\;\;j=m_{e}+1,...,m$

where $L({\mathbf{W}},\boldsymbol{\lambda})$ is the associated Lagrangian

$L({\mathbf{W}},\boldsymbol{\lambda})=f({\mathbf{W}})-\sum_{j=1}^{m}\lambda_{j}g_{j}({\mathbf{W}}).$
(12)

A step size $\alpha=1$ is optimal near a local optimum, but when far from the
optimum, the step size will need to be modified to guarantee a global
convergence. Han [117], Powell [118], Schittkowski [119] and Rockafellar [120]
have introduced the use of penalty functions in the nonlinear programming to
control the step size.

The Critical Line Algorithm (CLA) is an efficient alternative to the quadratic
optimizer for mean-variance model, as it is specifically designed for
inequality portfolio optimization. It was already originally introduced in the
Markowitz Portfolio Selection paper [1], and its computational implementation
has become increasingly popular [105, 106]. CLA also solves constrained
problems with conditions in inequalities, but unlike SLS, it divides a
constrained problem into series of unconstrained sub-problems by invoking the
concept of turning point. A turning point is a constrained minimum variance
portfolio whose vicinity contains other constrained minimum variance
portfolios of different free assets.

Similar to quadratic programming, an initial solution is required on the
constrained minimum variance frontier. To construct the initial solution,
assets are ranked with respect to their expected returns. Then, one increases
the weight of the first asset of the highest expected returns, $w_{1}$, from a
defined lower bound $l_{1}=0$ to an upper bound $u_{1}$ if $w_{1}\leq 1$.
Subsequently, the following assets have their weights increased until
$\sum_{i}{w_{i}}=1$. Typically, the weights of the first and the last few
assets are set to the upper and lower bound which are called bounded assets,
while only one in the middle has its weight between bounds and referred as the
free asset. The free weight is expressed as:

${w_{f}}=1-\sum_{i\in\mathds{U}}{w_{i}}-\sum_{i\in\mathds{L}}{w_{i}}$ (13)

where $\mathds{U}$ and $\mathds{L}$ represents two sets of upper and lower
bounded weights. Then in the following iterations, by decreasing the Lagrange
multiplier for the constraint on expected portfolio return, $\lambda$ to move
to the next lower turning point, two cases need to be considered to compute
$\mathbf{W}$. A formally free asset moves to its bound, or vice versa, a
bounded asset wants to become free. In both situations, the maximum threshold
$\lambda_{in}$ and $\lambda_{out}$ for the former and the later will be found.
Subsequently, the larger one characterises the new turning point, and the
asset is moved accordingly, and weights are re-assigned. As the free and
bounded assets do not interchange between turning points, the constrained
solution between two turning points is in fact the solution of unconstrained
optimization on only the free assets. Therefore, the constrained problem
reduces to solving the unconstrained problem on the free assets. When no new
threshold can be found, the lowest turning point is said to be reached and the
algorithm is terminated for the optimized $\mathbf{W}$.

## References

  * [1] Markowitz H.  Portfolio Selection.  The Journal of Finance. 1952;7(1).
  * [2] Hult H, Lindskog F, Hammarlid O, Rehn CJ.  Risk and Portfolio Analysis; 2012. .
  * [3] Samo YLK, Vervuurt A.  Stochastic Portfolio Theory: A Machine Learning Perspective.  Advanced Risk & Portfolio Management® Research Paper Series. 2016.
  * [4] Ban GY, Karoui NE, Lim AEB.  Machine Learning and Portfolio Optimization.  Manag Sci. 2018;64:1136–1154.
  * [5] Paiva FD, Cardoso RTN, Hanaoka GP, Duarte WM.  Decision-making for financial trading: A fusion approach of machine learning and portfolio selection.  Expert Syst Appl. 2019;115:635–655.
  * [6] Procacci PF, Aste T.  Forecasting market states.  Quantitative Finance. 2018;19:1491 – 1498.
  * [7] Massara GP, Matteo T, Aste T.  Network Filtering for Big Data: Triangulated Maximally Filtered Graph.  ArXiv. 2017;abs/1505.02445.
  * [8] Barfuss W, Massara GP, Di Matteo T, Aste T.  Parsimonious modeling with information filtering networks.  Physical Review E. 2016 Dec;94(6).  Available from: http://dx.doi.org/10.1103/PhysRevE.94.062306.
  * [9] Procacci PF, Aste T.  Portfolio Optimization with Sparse Multivariate Modelling.  arXiv. 2021;2103.15232.
  * [10] Officer R.  The Distribution of Stock Returns.  Journal of the American Statistical Association. 1972;67:807–812.
  * [11] Limpert E, Stahel WA.  Problems with Using the Normal Distribution â€“ and Ways to Improve Quality and Efficiency of Data Analysis.  PLoS ONE. 2011;6.
  * [12] Küchler U, Neumann K, Sørensen M, Streller A.  Stock returns and hyperbolic distributions.  Mathematical and Computer Modelling. 1999;29:1–15.
  * [13] Student.  On the probable error of the mean.  Biometrika. 1908;6:1–25.
  * [14] Peiró A.  The distribution of stock returns: international evidence.  Applied Financial Economics. 1994;4(6):431–439.
  * [15] Platen E, Rendek R.  Empirical Evidence on Student-t Log-Returns of Diversified World Stock Indices.  Journal of Statistical Theory and Practice. 2008;2(2):233–251.
  * [16] Wilson EB.  First and Second Laws of Error.  Journal of the American Statistical Association;18:841–851.
  * [17] Norton RM.  The Double Exponential Distribution: Using Calculus to Find a Maximum Likelihood Estimator.  The American Statistician. 1984;38:135–136.
  * [18] Eltoft T, Kim T, Lee TW.  On the multivariate Laplace distribution.  IEEE Signal Processing Letters. 2006;13:300–303.
  * [19] Mandelbrot B.  THE PARETO-LEVY LAW AND THE DISTRIBUTION OF INCOME*.  International Economic Review. 1960;1:79.
  * [20] Coronel-Brizio HF, Hernández-Montoya A.  On fitting the Pareto-Levy distribution to stock market index data: selecting a suitable cutoff value.  Physica A-statistical Mechanics and Its Applications. 2005;354:437–449.
  * [21] Stuart A, Markowitz H.  Portfolio Selection: Efficient Diversification of Investments.  A Quarterly Journal of Operations Research. 1959;10:253.
  * [22] Ledoit O, Wolf M.  Honey, I Shrunk the Sample Covariance Matrix.  Capital Markets: Asset Pricing & Valuation. 2003.
  * [23] Tibshirani R.  Regression Shrinkage and Selection via the Lasso.  Journal of the royal statistical society series b-methodological. 1996;58:267–288.
  * [24] Friedman JH, Hastie TJ, Tibshirani R.  Sparse inverse covariance estimation with the graphical lasso.  Biostatistics. 2008;9 3:432–41.
  * [25] Carsey T, Harden JJ.  Monte Carlo Simulation and Resampling Methods for Social Science; 2013\. .
  * [26] Manly B.  Randomization, Bootstrap and Monte Carlo Methods in Biology; 2020. .
  * [27] Berger PG, Ofek E.  Diversification’s effect on firm value.  Journal of Financial Economics. 1995;37:39–65.
  * [28] DeMiguel V, Garlappi L, Uppal R.  Optimal Versus Naive Diversification: How Inefficient is the 1/N Portfolio Strategy?  Review of Financial Studies. 2009;22:1915–1953.
  * [29] Schmitt J.  Portfolio Selection Efficient Diversification Of Investments; 2016. .
  * [30] Hamilton JD.  A New Approach to the Economic Analysis of Nonstationary Time Series and the Business Cycle.  Econometrica. 1989;57(2):357–384.  Available from: http://www.jstor.org/stable/1912559.
  * [31] Ramchand L, Susmel R.  Cross Correlations across Major International Markets.  Journal of Empirical Finance. 1998;5(4):397–416.
  * [32] Ang A, Bekaert G.  How do Regimes Affect Asset Allocation?  National Bureau of Economic Research; 2003. 10080.  Available from: http://www.nber.org/papers/w10080.
  * [33] Reynolds D.  Gaussian Mixture Models.  In: Encyclopedia of Biometrics; 2009. .
  * [34] Buckley I, Saunders D, Seco L.  Portfolio optimization when asset returns have the Gaussian mixture distribution.  Eur J Oper Res. 2008;185:1434–1461.
  * [35] Ketter W, Collins J, Gini ML, Gupta A, Schrater P.  Detecting and Forecasting Economic Regimes in Multi-Agent Automated Exchanges.  Econometrics eJournal. 2009.
  * [36] Delany SJ.  k-Nearest Neighbour Classifiers; 2007. .
  * [37] Nayak R, Mishra D, Rath A.  A Naïve SVM-KNN based stock market trend reversal analysis for Indian benchmark indices.  Appl Soft Comput. 2015;35:670–680.
  * [38] Kumar I, Dogra K, Utreja C, Yadav P.  A Comparative Study of Supervised Machine Learning Algorithms for Stock Market Trend Prediction.  2018 Second International Conference on Inventive Communication and Computational Technologies (ICICCT). 2018:1003–1007.
  * [39] Reyes EM, Ghosh S.  Bayesian Average Error-Based Approach to Sample Size Calculations for Hypothesis Testing.  Journal of Biopharmaceutical Statistics. 2013;23:569 – 588.
  * [40] Hee S, Hamborg T, Day S, Madan J, Miller F, Posch M, et al.  Decision-theoretic designs for small trials and pilot studies: A review.  Statistical Methods in Medical Research. 2016;25:1022 – 1038.
  * [41] Berry S, Carlin B, Lee JKJ, Müller P.  Bayesian Adaptive Methods for Clinical Trials; 2010. .
  * [42] Black F, Litterman RB.  Asset Allocation.  The Journal of Fixed Income. 1991;1(2):7–18.  Available from: https://jfi.pm-research.com/content/1/2/7.
  * [43] Black F, Litterman R.  Global Portfolio Optimization.  Financial Analysts Journal. 1992;48(5):28–43.
  * [44] Samuelson PA.  Lifetime Portfolio Selection By Dynamic Stochastic Programming.  The Review of Economics and Statistics. 1969;51(3):239–246.  Available from: http://www.jstor.org/stable/1926559.
  * [45] Hakansson NH.  Capital Growth and the Mean-Variance Approach to Portfolio Selection.  Journal of Financial and Quantitative Analysis. 1971;6(1):517–557.  Available from: https://EconPapers.repec.org/RePEc:cup:jfinqa:v:6:y:1971:i:01:p:517-557_02.
  * [46] Grauer RR, Hakansson NH.  On the Use of Mean-Variance and Quadratic Approximations in Implementing Dynamic Investment Strategies: A Comparison of Returns and Investment Policies.  Management Science. 1993;39(7):856–871.  Available from: https://EconPapers.repec.org/RePEc:inm:ormnsc:v:39:y:1993:i:7:p:856-871.
  * [47] Pliska SR.  Introduction to mathematical finance discrete time models.  Malden, Mass Blackwell; 1997.  Includes bibliographical references (p. [254]-256) and index.  Available from: http://openlibrary.org/books/OL661705M.
  * [48] Merton RC.  Lifetime Portfolio Selection under Uncertainty: The Continuous-Time Case.  The Review of Economics and Statistics. 1969;51(3):247–257.  Available from: http://www.jstor.org/stable/1926560.
  * [49] Karatzas I, Lehoczky J, Shreve S.  Optimal portfolio and consumption decisions for a “small investor” on a finite horizon.  Siam Journal on Control and Optimization. 1987;25:1557–1586.
  * [50] Merton RC.  Optimum consumption and portfolio rules in a continuous-time model.  Journal of Economic Theory. 1971;3(4):374–413.
  * [51] Bajeux-Besnainou I, Jordan JV, Portait R.  An Asset Allocation Puzzle: Comment.  American Economic Review. 2001 September;91(4):1170–1179.  Available from: https://www.aeaweb.org/articles?id=10.1257/aer.91.4.1170.
  * [52] Merton RC.  Lifetime Portfolio Selection under Uncertainty: The Continuous-Time Case.  The Review of Economics and Statistics. 1969;51:247–257.
  * [53] Cox J, Huang CF.  Optimal consumption and portfolio policies when asset prices follow a diffusion process.  Journal of Economic Theory. 1989;49:33–83.
  * [54] Björk T.  Arbitrage Theory in Continuous Time; 1999. .
  * [55] Jiao Y, Pham H.  Optimal investment with counterparty risk: a default-density model approach.  Finance and Stochastics. 2011;15:725–753.
  * [56] Karatzas I.  Optimization problems in the theory of continuous trading.  Siam Journal on Control and Optimization. 1989;27:1221–1259.
  * [57] Brennan M, Xia Y.  Dynamic Asset Allocation under Inflation.  Journal of Finance. 2000;57:1201–1238.
  * [58] Bielecki T, Jang I.  Portfolio optimization with a defaultable security.  Asia-Pacific Financial Markets. 2006;13:113–127.
  * [59] Jiao Y, Kharroubi I, Pham H.  Optimal investment under multiple defaults risk: a BSDE-decomposition approach.  Annals of Applied Probability. 2013;23:455–491.
  * [60] Zariphopoulou T.  A solution approach to valuation with unhedgeable risks.  Finance and Stochastics. 2001;5:61–82.
  * [61] Castañeda-Leyva N, Hernández-Hernández D.  Optimal Consumption-Investment Problems in Incomplete Markets with Stochastic Coefficients.  SIAM J Control Optim. 2005;44:1322–1344.
  * [62] He H, Pearson ND.  Consumption and portfolio policies with incomplete markets and short-sale constraints: The infinite dimensional case.  Journal of Economic Theory. 1991;54:259–304.
  * [63] Karatzas I, Lehoczky J, Shreve S, Xu G.  Martingale and duality methods for utility maximization in a incomplete market.  Siam Journal on Control and Optimization. 1991;29:702–730.
  * [64] Aghabozorgi S, Shirkhorshidi AS, Wah TY.  Time-series clustering - A decade review.  Inf Syst. 2015;53:16–38.
  * [65] Zolhavarieh S, Aghabozorgi S, Teh YW.  A Review of Subsequence Time Series Clustering.  The Scientific World Journal. 2014;2014.
  * [66] Navarro J, Frenk C, White S.  A Universal Density Profile from Hierarchical Clustering.  The Astrophysical Journal. 1997;490:493–508.
  * [67] Bhattacharjee B, Shafi M, Acharjee A.  Network mining based elucidation of the dynamics of cross-market clustering and connectedness in Asian region: An MST and hierarchical clustering approach.  J King Saud Univ Comput Inf Sci. 2019;31:218–228.
  * [68] Zeitsch PJ.  A jump model for credit default swaps with hierarchical clustering.  Physica A-statistical Mechanics and Its Applications. 2019;524:737–775.
  * [69] Madhulatha T.  An Overview on Clustering Methods.  ArXiv. 2012;abs/1205.1117.
  * [70] Dolnicar S.  A Review of Unquestioned Standards in Using Cluster Analysis for Data-Driven Market Segmentation; 2002. .
  * [71] Campello RJGB, Moulavi D, Sander J.  Density-Based Clustering Based on Hierarchical Density Estimates.  In: PAKDD; 2013. .
  * [72] Liu C, Cao Y.  Task re-pricing model based on density-based spatial clustering of applications.  Appl Soft Comput. 2020;96:106608.
  * [73] Tigani S, Chaibi H, Saadane R.  Gaussian Mixture and Kernel Density-Based Hybrid Model for Volatility Behavior Extraction From Public Financial Data.  Data. 2019;4:19.
  * [74] Aitken M, Brown P, Buckland C, Izan H, Walter TS.  Price clustering on the Australian Stock Exchange.  Pacific-basin Finance Journal. 1995;4:297–314.
  * [75] Ahn HJ, Cai J, Cheung YL.  Price Clustering on the Limit-Order Book: Evidence from the Stock Exchange of Hong Kong.  Journal of Financial Markets. 2005;8:421–451.
  * [76] Li F, Sheng H, Zhang D.  Event Pattern Discovery from the Stock Market Bulletin.  In: Discovery Science; 2002. .
  * [77] Kanungo T, Mount D, Netanyahu N, Piatko C, Silverman R, Wu A.  An Efficient k-Means Clustering Algorithm: Analysis and Implementation.  IEEE Trans Pattern Anal Mach Intell. 2002;24:881–892.
  * [78] Cheng Y.  Mean Shift, Mode Seeking, and Clustering.  IEEE Trans Pattern Anal Mach Intell. 1995;17:790–799.
  * [79] Alkhatib K, Najadat H, Hmeidi I, Shatnawi M.  Stock Price Prediction Using K-Nearest Neighbor (kNN) Algorithm; 2013\. .
  * [80] Duffie D, Pan J.  An Overview of Value at Risk; 1997. .
  * [81] Lin WL, Engle R, Ito T.  Do Bulls and Bears Move Across Borders? International Transmission of Stock Returns and Volatility.  Review of Financial Studies. 1994;7:507–538.
  * [82] Ang A, Bekaert G.  International Asset Allocation With Regime Shifts.  Review of Financial Studies. 2002;15:1137–1187.
  * [83] Musmeci N, Aste T, Matteo TD.  What does past correlation structure tell us about the future? An answer from network filtering.  arXiv: Portfolio Management. 2016.
  * [84] Bollerslev T.  Modelling the Coherence in Short-Run Nominal Exchange Rates: A Multivariate Generalized Arch Model.  The Review of Economics and Statistics. 1990;72(3):498–505.  Available from: http://www.jstor.org/stable/2109358.
  * [85] Engle R.  Dynamic Conditional Correlation: A Simple Class of Multivariate Generalized Autoregressive Conditional Heteroskedasticity Models.  Journal of Business & Economic Statistics. 2002;20(3):339–350.  Available from: http://www.jstor.org/stable/1392121.
  * [86] Hallac D, Vare S, Boyd S, Leskovec J.  Toeplitz Inverse Covariance-Based Clustering of Multivariate Time Series Data.  arXiv. 2018;1706.03161.
  * [87] Friedman J, Hastie T, Tibshirani R.  Sparse inverse covariance estimation with the graphical lasso.  Biostatistics. 2008;9 3:432–41.
  * [88] Nešetřil J, Milková E, Nešetřilová H.  Otakar Boruvka on minimum spanning tree problem Translation of both the 1926 papers, comments, history.  Discrete mathematics. 2001;233(1-3):3–36.
  * [89] Kruskal JB.  On the shortest spanning subtree of a graph and the traveling salesman problem; 1956. .
  * [90] Prim RC.  Shortest connection networks and some generalizations.  Bell System Technical Journal. 1957;36:1389–1401.
  * [91] Tumminello M, Aste T, Di Matteo T, Mantegna RN.  A tool for filtering information in complex systems.  Proceedings of the National Academy of Sciences. 2005;102(30):10421–10426.  Available from: https://www.pnas.org/content/102/30/10421.
  * [92] Aste T, Matteo T.  Sparse Causality Network Retrieval from Short Time Series.  Complex. 2017;2017:4518429:1–4518429:13.
  * [93] Massara GP, Aste T.  Learning Clique Forests.  ArXiv. 2019;1905.02266.
  * [94] Telesford QK, Simpson S, Burdette J, Hayasaka S, Laurienti P.  The Brain as a Complex System: Using Network Science as a Tool for Understanding the Brain.  Brain connectivity. 2011;1 (4):295–308.
  * [95] Aste T.  Topological regularization with information filtering networks.  arXiv preprint arXiv:200504692. 2020.
  * [96] Millington T, Niranjan M.  Robust Portfolio Risk Minimization Using the Graphical Lasso.  In: ICONIP; 2017. .
  * [97] Yuan X, Yu W, Yin Z, Wang G.  Improved Large Dynamic Covariance Matrix Estimation With Graphical Lasso and Its Application in Portfolio Selection.  IEEE Access. 2020;8:189179–189188.
  * [98] Lee TH, Seregina E.  Optimal Portfolio Using Factor Graphical Lasso.  arXiv: Econometrics. 2020.
  * [99] Kraft D.  A software package for sequential quadratic programming.  Tech Rep DFVLR-FB. 1988;88(28).
  * [100] Boggs PT, Tolle JW.  Sequential quadratic programming.  Acta Numerica. 1996;4(1).
  * [101] Nocedal J, Wright SJ.  Numerical Optimization.  Springer-Verlag. 2006.
  * [102] Jackson M, Staunton MD.  Quadratic Programming Applications in Finance Using Excel.  The Journal of the Operational Research Society. 1999;50(12).
  * [103] Cesarone F, Scozzari A, Tardella F.  Portfolio selection problems in practice: a comparison between linear and quadratic optimization models.  Computational Management Science. 2015;12(3).
  * [104] Scipy Minimization Library;.  Accessed: 2021-12-14.  https://docs.scipy.org/doc/scipy/reference/optimize.minimize-slsqp.html.
  * [105] Singh RH, Barford L, Harris FC.  Accelerating the critical line algorithm for portfolio optimization using gpus.  Advances in Intelligent Systems. 2016;448.
  * [106] Markowitz HM, Starer D, Fram H, Gerber S.  Avoiding the Downside: A Practical Review of the Critical Line Algorithm for Mean–Semivariance Portfolio Optimization.  In: Guerard JB, Ziemba WT, editors. HANDBOOK OF APPLIED INVESTMENT RESEARCH. World Scientific Book Chapters. World Scientific Publishing Co. Pte. Ltd.; 2020. p. 369–415.
  * [107] Bailey DH, de Prado ML.  An Open-Source Implementation of the Critical-Line Algorithm for Portfolio Optimization.  Algorithms. 2013;6(1).
  * [108] Narsoo J.  Performance Analysis of Portfolio Optimisation Strategies: Evidence from the Exchange Market.  International journal of economics and finance. 2017;9:124–132.
  * [109] Sharpe WF.  The Sharpe Ratio.  The Journal of Portfolio Management. 1994;21(1):49–58.  Available from: https://jpm.pm-research.com/content/21/1/49.
  * [110] Lo AW.  The Statistics of Sharpe Ratios.  Financial Analysts Journal. 2002;58(4):36–52.  Available from: https://doi.org/10.2469/faj.v58.n4.2453.
  * [111] Fiance Y. Shelton Capital Management Nasdaq-100 Index Fund Direct Shares;.  Available from: https://finance.yahoo.com/quote/NASDX/risk/.
  * [112] Fiance Y. iShares Core FTSE 100 UCITS ETF GBP (Dist) (ISF.L);.  Available from: https://finance.yahoo.com/quote/ISF.L?p=ISF.L&.tsrc=fin-srch.
  * [113] Zhang T.  Stock Picking Strategy Based on Exploration of Chip Distribution Indicators.  In: 2020 International Conference on Computing and Data Science (CDS); 2020. p. 276–282.
  * [114] Duffie D, Pan J.  An Overview of Value at Risk.  The Journal of Derivatives. 1997;4(3):7–49.  Available from: https://jod.pm-research.com/content/4/3/7.
  * [115] Ramsey AF, Goodwin BK.  Value-at-Risk and Models of Dependence in the U.S. Federal Crop Insurance Program.  Journal of Risk and Financial Management. 2019;12(2).  Available from: https://www.mdpi.com/1911-8074/12/2/65.
  * [116] Wilson RB.  A Simplicical Algorithm for Concave Programming.  In: PhD Dissertation. Harvard University; 1963. .
  * [117] Han SP.  A Globally Convergent Method for Nonlinear Programming.  JOptTheory Applic. 1997;22:248–256.
  * [118] Powell M.  A fast algorithm for nonlinearly constrained optimization calculations; 1978. .
  * [119] Hock W, Schittkowski K.  Test examples for nonlinear programming codes.  Journal of Optimization Theory and Applications. 1980;30:127–129.
  * [120] Rockafellar RT.  The Multiplier Method of Hestenes and Powell Applied to Convex Programming.  JOptTheory Applic. 1973;12:555–562.