# Variational Embedding Multiscale Sample Entropy:
complexity-based analysis for multichannel systems

Hongjian Xiao and Danilo P. Mandic

###### Abstract

To quantify the complexity of a system, entropy-based methods have received
considerable critical attention in real-world data analysis. Among numerous
entropy algorithms, amplitude-based formulas, represented by Sample Entropy,
suffer from a limitation of data length especially when it comes to practical
scenarios. And this shortcoming is further highlighted by involving coarse
graining procedure in multi-scale process. The unbalance between embedding
dimension and data size will undoubtedly result in inaccurate and undefined
estimation. To that cause, Variational Embedding Multiscale Sample Entropy is
proposed in this paper, which assign signals from various channels with
distinct embedding dimensions. And this algorithm is tested by both stimulated
and real signals. Furthermore, the performance of the new entropy is
investigated and compared with Multivariate Multiscale Sample Entropy and
Variational Embedding Multiscale Diversity Entropy. Two real-world database,
wind data sets with varying regimes and physiological database recorded from
young and elderly people, were utilized. As a result, the proposed algorithm
gives an improved separation for both situations.

###### Index Terms:

entropy, complexity, multi-channel system, physical signal analysis.

##  I Introduction

The investigation of complexity of real world signals we have witnessed a boom
in the last decades. Now recognized to have the same importance as the
properties in the time and frequency domain, the complexity of a data set is a
unique feature that can be utilized to understand a signal generating
mechanism via nonlinear analytical tools. The studies of complexity have
covered a wide spectrum from the fault diagnosis of rotating machine [1, 2, 3]
through to the early detection of disease and sickness in humans [4, 5, 6, 7].
Indeed, bio signals exhibit high degrees of irregularities and complex
dynamical behaviours [8]. Such nonlinear dynamics results from the
interactions between human body (organisms) and peripheral environment and
exhibits continuous fluctuations in time domain [9]. Complexity Loss Theory
(CLT) states the potential relationship between the complexity of physical
signals and health of an individual where the higher degree of complexity
indicates healthier condition of the individual [10]. However, new
developments have declared that pathology also exhibits an increase in
complexity based on structure, that is, a decrease of self-correlated
complexity will also be observed in healthy body [11].

Although the definition of structural complexity is inconsistent [12], there
are several commonly used methods to quantify the degree of dynamics where
entropy-based methodologies are popular ones. Compared to other methods to
estimate complexity of nonlinear systems, such as fractal dimension [8] and
recurrence plots [9], entropy analysis holds the advantages of simplicity and
noise robustness [13]. The features of the loss of complexity (LOC) manifest
themselves through, for example, an increase in randomness, less regularity,
breakdown of long-term correlations, multiscale variability, and time
irreversibility [12]. To this end, a large number of various entropy
algorithms have been proposed to quantify the different features of
complexity, or more precisely, the degree of complexity based on different
definitions.

The two early entropy algorithms that have been widely used are the
Approximate Entropy (ApEn) [14] and Sample Entropy (SampEn) [15], proposed in
1991 and 2000 respectively. Modified by ApEn, SampEn is able to reduce the
bias experienced by ApEn by removing self-matching and simultaneously has less
dependency on the data length giving relatively higher consistency [15]. Both
ApEn and SampEn were developed to quantify the randomness and irregularity of
the system. Generally speaking, the lower the value of SampEn, the less
complex the system. However, truly complex signals exhibit varying structures
across multiple time scales, while long-range correlations fail to be observed
by single-scale Sample Entropy analysis. To this end, Costa et al. introduced
a ‘coarse-graining’ procedure into the Sample Entropy methodology to verify
the structural complexity hidden in high scales, referred to as the Multiscale
Sample Entropy (MSE) [16]. This, in turn further spurred the development of
MSE algorithms including Composite Multiscale Sample Entropy [17] and Refined
Composite Multiscale Sample Entropy [18]. However, due to the ‘coarse-
graining’ procedure, the requirement of long data length remains even more
problemic and hard to be satisfied in most practical situations. In 2011,
Multivariate Multiscale Sample Entropy (MMSE) was introduced which
successfully combined dataset from multiple channels to estimate the dynamics
of the system more accurately and with shorter data length [19]. The key
improvement of MMSE is the form of composite delay vector which involves and
reconstructs data segments from multiple channels, whereby the inner
correlations among diverse signals are preserved [19]. The existing
multivariate multiscale entropies to date include:

  * •

Multivariate Multiscale Sample Entropy (MMSE) [19], a method which performs
joint multivariate analysis of physiological signals associated with multiple
channels.

  * •

Multivariate Multiscale Permutation Entropy (MMPE) [5], an extension of
standard Permutation Entropy [20] which inherits the desirable properties of
PE, such as fast computation and simple implementation.

  * •

Multivariate Multiscale Fuzzy Entropy (MMFE) [21] which combines Composite
Delay Vectors and Fuzzy Entropy [22], and exhibits more stable smoother
estimates than MMSE.

  * •

Variational embedding Diversity Entropy (veMDE) [23], a method developed on
the basis of Diversity Entropy [1] that combines angular distance and relative
probability, and exhibits a low computational load with similar performance as
MMPE.

Despite success, the inherent shortcomings of amplitude-based entropy
calculation still remain a major obstacle towards their more widespread use .
Other issues with current multivariate entropy methods are included as:

  1. 1.

The rule of thumb is that the requirement of data length is around $10^{m}$ to
$30^{m}$, where $m$ refers to the embedding dimension [24]. Hence, the choice
of the embedding dimension is limited by the available sample points.

  2. 2.

The ‘coarse-graining’ process further emphasizes the drawback of limited data
size, which causes inaccurate and undefined estimation for high scale
analysis.

  3. 3.

Amplitude-based distance is always sensitive to outliers such as noise and
artifacts.

  4. 4.

Poor quality of any single channel has a large impact on the multivariate
performance.

  5. 5.

Excessive computational load is required when implementing multi-channel
analyses.

Recently, Wang et al. [23] introduced a new way to combine datasets from
multiple channels into one entropy algorithm estimation, termed Variational
Embedding Multiscale Diversity Entropy. Here, inspired by [23], a new
multivariate entropy method is proposed based on Sample Entropy named
Variational Embedding Multiscale Sample Entropy (veMSE). This new method
offers the following advantages as:

  1. 1.

Complexity estimates at a higher embedding dimension are better defined with
limited data size.

  2. 2.

The requirement for the amount sample points is lower than current Sample
entropy-based methods.

  3. 3.

Strong noise-robustness is exhibited across the scales.

  4. 4.

The overall performance of multivariate estimate is independent on the quality
of any single-channel within a dataset.

  5. 5.

Less computational time is needed owing to straightforward and efficient
implementation.

The reminder of the paper is organized as follows: In Section II, the new
veMSE algorithm is outlined. Section III demonstrates the operation of veMSE
on simulated signals to give an initial insight with regards to the choice of
parameters. Then, based on the suggested parameter setting in Section III,
Section IV considers and discusses the properties of veMSE, including noise
robustness, directionality, and calculation efficiency. Next, veMSE is applied
to real-world signals as wind dynamics and heart rate variability in Section
V, and compared with the performance of univariate MSE and MMSE. Finally,
conclusions are given.

##  II Variational Embedding Multiscale Sample Entropy

Algorithm 1 Variational Embedding Multiscale Sample Entropy

Assume that there are $P$ channels measured from a system, where a signal
recorded from the $c^{th}$ channel is denoted by $x_{c}(i)$ and is of length
$N$, where $1\leq c\leq P,1\leq i\leq N$. Parameters involved in the veMSE
algorithm are the tolerance quotient $(r)$, embedding dimension $(m)$, scale
factor $(\tau)$ and time lag $(L)$. The detailed steps of veMSE are shown
below.

  1. 1)

Procedure of coarse graining is firstly applied to the original datasets for
all the channels. The scaled time series are calculated as

$y^{(\tau)}(j)=\frac{1}{\tau}\sum_{i=(j-1)\tau+1}^{j\tau}x(i),\quad 1\leq
j\leq N_{t},\;N_{t}=\frac{N}{\tau}.$ (1)

  2. 2)

For each channel, the embedding dimension is set as a variable. The dimension
for the $c^{th}$ channel is calculated as $m(c)=m+c-1$, as listed in Table. I.
Therefore, combined with the process of time delay, the embedding delay vector
of data $y^{(\tau)}(i),\;(1\leq i\leq N_{t})$ for channel $c$ is designated as
template, $Y_{c}^{(\tau)}(i)$ and calculated as

$Y_{c}^{(\tau)}(i)=[y^{(\tau)}(i),y^{(\tau)}(i+L)\;,\dots,\;y^{(\tau)}(i+n)],\;n_{c}=(m(c)-1)L.$
(2)

  3. 3)

Compute the Chebyshev distance between templates $Y_{c}^{(\tau)}$, where the
distance is defined according to the amplitude of the embedding vector as

$\begin{split}d=max\\{Y_{c}^{(\tau)}(i+k))-Y_{c}^{(\tau)}(j+k)\\},\\\
where\;0\leq k\leq m(c)-1,\;1\leq i,j\leq N_{t}-n_{c},\;i\neq j\end{split}$
(3)

  4. 4)

For each channel, the number of segments, $Y_{c}^{(\tau)}(j)$, within the
tolerance level $r$ of $Y_{c}^{(\tau)}(i)$, is recorded as $B_{c}(i)$. In
other words, $B_{c}(i)$ is the number of template matched or the number of
similar patterns in the dataset based on the boundary set, where the boundary
is defined by the tolerance, $r$. Therefore, the local probability of the
occurrence of template match for channel $c$ is

$R_{c}(i)=\frac{B_{c}(i)}{(N_{t}-n_{c}-1)}$ (4)

  5. 5)

Then, the global probability of the occurrence of template match for a channel
$c$ is calculated as

$\Phi(c)=\frac{1}{(N_{t}-n_{c})}\sum_{i=1}^{N_{t}-n_{c}}{R_{c}(i)}$ (5)

  6. 6)

Compute the sum of the global probability for all the channels as

$\Phi=\sum_{c=1}^{P}{\Phi(c)}$ (6)

Recall that unlike the MMSE algorithm (see Appendix.X), the embedding
dimension, $m(c)$, is diverse and varied along with the index of channel $c$.

  7. 7)

Modify the embedding dimension to $(m(c)+1)$. Hence, $n_{c}$ is adjusted to
$m(c)\times L$ and the Steps 2-6 are repeated to obtain global probability
with increased dimension $\Phi(m+1)$.

  8. 8)

Variational Embedding Multiscale Sample Entropy is finally obtained as

$veMSE=-ln\frac{\Phi(m+1)}{\Phi(m)}.$ (7)

The steps of proposed veMSE is given in Algorithm. 1. The key improvement of
veMSE is that it allows for the varying setting of the embedding dimension for
multi-channel signals as illustrated in Table. I, while simultaneously
maintaining the information within each channel. Compared to MMSE (see
Algorithm. 2 in Appendix), despite the fact that different embedding values
can be also achieved for each channel by MMSE, veMSE focuses more on the
information between data channels. Indeed, the composite delay vector in MMSE
successfully combines the embedding vectors of multichannel signals. However,
there is an discrepancy and bias between the similarity among the recombined
composite delay vectors and those among embedding vectors of the original
datasets. The proposed veMSE estimates the complexity information of signals
in multiple channels without influencing the correlation in each individual
signal. Secondly, due to the varying embedding dimensions, veMSE yields a
weighted contribution to multiple channels. That is, the probability
distribution of each channel is diverse, which serve as an amplification of
chaotic features in measured signals of multi-channel systems. Thirdly, since
the probability of occurrences of similar patterns will be processed multiple
times based on the number of channels, and the summation process will be
performed before the logarithm operation in the last step, the veMSE is
theoretically able to unveil the complexity properties under higher embedding
dimension but with the same data length, in comparison with other algorithms
based on Sample Entropy.

TABLE I: Relation between the embedding dimension ($m$) and index of channel ($c$) Index of channel $(c)$ |  1 |  2 | $\dots$ |  $c$
---|---|---|---|---
Embedding dimension $(m(c))$ |  $m$ |  $m+1$ | $\dots$ |  $m+c+1$

To show the performance of the new proposed veMSE, both synthetic signals and
real physical datasets are considered in the following sections. As one of the
popular multichannel entropy algorithms, MMSE will be employed as a benchmark
for veMSE under the same conditions.

##  III Results of veMSE on stimulated signals

In this section, synthetic signals generated based on five models will be
utilized to illustrate the performance of veMSE including Gaussian white
noise, flicker noise (coloured noise), and autoregressive (AR) models AR$(1)$,
AR$(2)$ and AR$(3)$. Standard deviations of all the generated signals were set
to $\sigma=1$. Coefficients of AR models are given in Table.II. Parameters
that will be discussed in the following subsections include embedding
dimension, data length, tolerance, and scale factor. To avoid unknown
influence and control variables, time lag were set to $L=1$ for all
operations.

TABLE II: coefficients of AR models Coefficients |  $a_{1}$ |  $a_{2}$ |  $a_{3}$
---|---|---|---
AR $(1)$ |  $0.5$ |  $-$ |  $-$
AR $(2)$ |  $0.5$ |  $0.25$ |  $-$
AR $(3)$ |  $0.5$ |  $0.25$ |  $0.125$

Figures in each subsection are presented in pairs to show the results for five
models. Upper panels give the curves of white noise and flick noise, while the
panels in the bottom depict the results of AR models in contrast to white
noise. Complexity curves of entropy values are plotted as error-bar figures,
averaged over outcomes of 20 realizations for each model.

###  III-A Varied Embedding Dimension (m)

Usually, for the implementation of SampEn-based algorithms, the embedding
dimension and data length are two parameters that are interdependent and
mutually coupled. Data size is restricted to between $(10^{m}-30^{m})$ as a
rule of thumb [25]. In real world, signals recorded do not have infinite
length and are generally limited by operation time and memory space.
Therefore, the embedding dimension is commonly set to $m=2$ or $m=3$ for a
signal with 1000 samples [26]. Higher values of embedding dimension with a
shortage of data size will cause unstable estimation as standard MMSE given in
Figure.1.

Figure 1: Operation of single-scale a) MMSE, and b) veMSE as a function of the
embedding dimension, m.

Results of veMSE as a function of embedding dimension are shown in Figure.1.
Each entropy is calculated based on signals from two channels. Except the
independent variable $m$, other parameters, such as scale factor and data
length, are set as constant values (1 and 1000 respectively). The tolerance,
$r$ is varying according to the total variance of the covariance matrix of
processed data sets as $r\times tr(S)$, and here the tolerance quotient was
fixed to $r=0.15$.

Figure.1, where embedding dimension is modified from 1 to 9, shows that unlike
MMSE, signals with a complex correlated structure are able to give a defined
entropy value even at high embedding dimensions, as e.g. AR(3) in scale of 7.
Also, signals with higher randomness are more likely to become unstable as the
embedding dimension increase. However, even in the case of Gaussian white
noise with highest randomness, here veMSE with the embedding dimension equal
to 5 was able to successfully and stably process the data. On the other hand,
traditional Multiscale Sample Entropy methods fail to give a defined value
with the embedding dimension higher than 3 under the same condition [11].
Therefore, from the viewpoint of estimation stability, veMSE exhibits a marked
improvement when it comes to complex information in high dimensions.

###  III-B Varied Data Length (N)

The data length of the signal is another limitation in addition to embedding
dimension when implementing entropy-based calculation, particularly in real
world processes. Amplitude-based entropy algorithms require at least 1000 data
points to guarantee a consistent estimation such as with Multiscale Sample
Entropy (MSE) and Multiscale Fuzzy Entropy (MFE) [22]. However, in real world
data, as in the analysis of heart rate variability for example, to obtain the
required data size for R-R intervals, a minimum of 5 minutes of the raw
Electrocardiograph (ECG) signals are needed. In practice, the implementation
of such a long-time recording in a controlled state is hard to be satisfied.
Compared to amplitude-based entropy methods, space distance-based entropy
algorithms such as Cosine Similarity Entropy, show less restriction to data
length, with a minimum of 700 samples required [11].

Figure 2: Operation of single-scale a) MMSE, and b) veMSE as a function of the
data length, N.

Figure.2 illustrates the performance of single scale of veMSE as a function of
data length in a logarithmic scale. The embedding dimension was set to $m=2$
and the choice of tolerance was the same as before. The values of veMSE for
white noise and $1/f$ noise were not defined before $N$ = 40, while AR(2) was
not defined with veMSE when $N$ was smaller than 30. The smallest sample sizes
for AR(1) and AR(3) were also $N$ = 40.

The standard deviation of the entropy results is gradually narrowing down with
an increase in data length, while the range for each error-bar is ascending
from top to bottom. As a result, a system with higher structure (AR(3))
reveals larger standard deviation. In addition, the consistency of the
estimation can be guaranteed as evidenced by the relative position of curves
in each graph maintains being unchanged as data length, $N$, increases. More
importantly, when analysing the white noise and flicker noise in the top
graph, the estimation at $N$ = 100-sample length could successfully separate
the complexity degrees of the two signals, while in the bottom graph, when the
data length reaches $N$ = 400 samples, there is no intersection region among
entropy values from different models. This illustrates that the requirement of
data length when applying veMSE is much lower than other entropy methods, e.g.
MMSE in Figure.2 gives the $N$ = 1300 is required for the separation of 4
models. Subsequently, this property gives the way to reveal the complexity
information under high scales, with limited data length, but with a stable
estimation, which better serves the balance between the dimension and data
size.

###  III-C Varied tolerance (r)

Figure 3: Operation of single-scale a) MMSE, and b) veMSE as a function of the
tolerance, r.

The tolerance, $r$, can be explained as the boundary of the similarity degree
among comparing templates. The SampEn-based algorithms limit the tolerance to
a hard threshold, as the Heaviside function in related to the standard
deviation of the original data. However, for a multivariate case with
multichannel data sets, only a single value is needed in the algorithm. As in
[19], the choice of tolerance of veMSE is dependent on the total variance of
the covariance matrix, $S$, of the analysed data sets. Therefore, the
tolerance was set as $r\times tr(S)$.

Figure.3 illustrates single scale entropy estimation as a function of
tolerance quotient, $r$, varying from 0.1 to 1.5 at 0.1 increments. The data
length and embedding dimension were separately fixed at $N$ = 1000 and $m$ = 2
to show the influence of varied tolerance setting. Observe from the figures
that for all curves, the increase of tolerance quotient results in a monotone
decrease in complexity estimation, which is the same as MMSE in Figure.3. All
can be obviously distinguished from each other before $r$ = 1 in veMSE. The
values after $r$ = 1 are too small to be differentiated. The gap among
different complexity estimations in the two figures obviously narrows down
after r = 0.5, therefore, supporting that the value of tolerance quotient to
be chosen below $r$ = 0.5.

###  III-D Varied Scale Factor

As noted by Costa et al. in [16], the multiscale analysis by integrating
consecutive coarse-graining procedure is of importance in signal processing
associated with hidden correlation structure in data. Based on the
aforementioned analysis of parameters involved in veMSE, the embedding
dimension was set to $m$ = 2 and the tolerance was chosen to $r$ = 0.15
multiplied by total variance of the covariance matrix. With regard to
performance of multiscale analysis, graphs of multichannel entropy results are
presented in response to the scale factor varying from $(\tau)$ = 1 to
$(\tau)$ = 40. Dual channel data with $N$ = 3000 sample data points for each
model were considered.

Figure 4: Operation of Multivariate Multiscale Sample Entropy [19] with the
embedding dimension set as [2 2].

Figure 5: Operation of Multivariate Multiscale Sample Entropy [19] with the
embedding dimension set as [2 3].

Figure 6: Operation of variational embedding Multiscale Diversity Entropy [23]
with the embedding dimension set as 2.

Figure 7: Operation of the proposed variational embedding Multiscale Sample
Entropy with the embedding dimension set as 2.

To further elucidate the extent of improvements of veMSE over the Multivariate
Multiscale Sample Entropy (MMSE) and Variational Embedding Multiscale
Diversity Entropy (veMDE), their performances were compared with the proposed
veMSE method. With the same data size, due to the variational dimension
feature of veMSE, two different settings of the parameter related to embedding
dimension for MMSE were applied and are shown in Figure.5 and Figure.5. The
performance of veMDE is given in Figure.7, with the embedding dimension set to
$m$ = 2, while the results of veMSE are presented in Figure.7.

These figures demonstrate that complex structure hidden in higher dimension is
hard to be unveiled by MMSE. The standard deviation of those 20 realizations
grows steadily as the scale factor increases. Overall, the dimension setting
pair $[2,2]$ gives a better performance in MMSE for the considered restricted
data length. However, even under the optimal dimension settings, as in
Figure.5, the complexity of AR(3) in purple and AR(2) in yellow fails to be
distinguished in multi-scale cases. As for veMDE in Figure.7, Diversity
Entropy is developed based on angular distance and distribution probability
[1]. As can be seen in the graph, veMDE reveals a consistent estimation for
each system, which exhibites a short-term correlation. Nevertheless, long-term
structural complexity of each model fails to be shown by veMDE. Besides, due
to the lack of ability to estimate long-term correlation, veMDE for AR models
which contain highly correlated structure can not be distinguished from white
noise in large scales.

On the other hand, consistent with the advantages as aforementioned
theoretical discussion, the merits of veMSE can be clearly seen from in
Figure.7. To better specify the improvement, the optimal dimension setting [2
2] of MMSE in Figure.5 was utilized to compare with veMSE. Observe from the
upper graphs of the two types of noise, although both of the two algorithms
were able to distinguish between the two models, the complexity of white noise
went down while the flicker noise maintained a certain level in spite of the
increasing scale. The range of entropy in error-bars for flicker noise based
on veMSE was much narrower than that in basis of MMSE especially in large
scales. Secondly, in the bottom graph, values of AR(2) and AR(3) (in yellow
and purple) fail to be fully separated by MMSE in the cases of high scale, as
stated above, while with the same data length, the separability of AR models
with different orders can be successfully accomplished in veMSE. It is
critical to apply the entropy calculation under multiscale situation since the
long-range correlation of the system is largely ignored in the analysis under
low scale. Next, it can be observed that minor difference exists between the
complexity estimation of the two models, referred to white noise and AR(1)
(blue and red line in bottom graph). Instead, the enhancement produced by
veMSE is particularly revealed on the analysis of highly correlated and
structural signals, systems with higher structural complexity.

Overall, according to the comparison of veMSE and MMSE as well as veMDE based
on the above five models, the veMSE provides a more stable estimation that can
better demonstrate complex temporal fluctuations. In addition, veMSE is
especially suitable for multiscale analysis of highly correlated signals which
exhibit variation of spatial-temporal patterns over a range of scales.

##  IV Properties of veMSE

We now elaborate on the three properties of the proposed entropy veMSE
algorithm: noise robustness, directionality, and calculation efficiency. The
parameters setting: data size $N$ = 3000; embedding dimension $m$ = 2;
tolerance $r$ = 0.15; and scale factor $\tau$ = 1. A bivariate system was
considered in the analysis. The results which depict the noise analysis and
directionality analysis based on proposed veMSE are presented in Figure.8 and
Figure.9 respectively. And corresponding performance of MMSE are exhibited in
Figure.8 and 9. The curves of time requirements of veMSE and MMSE are given in
Figure.10.

###  IV-A Noise Robustness

Robustness towards noise and artifacts is of critical importance in any
estimation. Given that it is infeasible to avoid the noise associated to
recording equipment and the ubiquity of artifacts in bio signals, for
instance, muscle and electro-magnetic artifacts pervasively exist in EEG-based
monitor [27], the noise-robustness property was tested by comparing complexity
estimation for AR models with and without noise. In Figure.8, from top to the
bottom, the first panel presents the curves for uncorrelated white Gaussian
noise (WGN), correlated flicker noise ($1/f$ noise) and coloured noise
containing both WGN and $1/f$ noise. It is clear that three systems with
different degrees of correlation can be separated by the veMSE. Adding white
noise will enhence the short-term correlation as shown at the beginning in the
top graph, where the yellow line (1/f + WGN) is as high as blue line (WGN),
while the long-term correlation is lower as scale factor increases (1/f $>$
1/f+WGN $>$ WGN). The first graph reveals that veMSE could correctly give a
complexity estimation, in line with the theoretical analysis, on the basis of
uncorrelated and correlated noise.

In the second and third panel of Figure.8, the results of veMSE for AR models
with uncorrelated white noise and correlated flicker noise are presented, to
contrast to the outcomes of pure AR signals in Figure.7. The amplitude of the
added noise signal was set to $20\%$ that for the AR signals. Compared to
Figure.7, the gaps between the complexity curves for the AR models of varying
order decrease with noise. However, even that the gap among distinct models is
narrowed down, separation can still be achieved to high scales in $20\%$ noisy
scenarios. While in case of MMSE, noisy AR signals with different complexity
cannot be well divided and the impact of noise was clearly shown in Figure.8.
Given these points, the performance of complexity estimation based on veMSE is
consistent with cases without noise. A unique feature of veMSE not present in
the other MSE algorithms, showing its potential to be applied in practical
recording data sets.

Figure 8: Illustration of noise robustness of a) standard MMSE and b) the
proposed veMSE.

###  IV-B Directionality

Next feature of veMSE that is going to be discussed is directionality. For
multivariate analysis, a problem is that the optimal ordering of the input
channel is unknown. Yet, without prior knowledge related to the optimal
channel order, the performance of the estimation will be impacted. To this
end, the directionality of the veMSE is analysed for bivariate systems.
Figure.9 reveals two graphs, each contains three pairs of curves. In top
graph, there are white noise with AR(1), AR(1) with AR(2) and AR(2) with
AR(3), with the order of input shown by legend (first present, first
processed). As can be seen from this figure, the lower scale estimate is
mainly influenced by the first input signal. For example, the blue line [WGN
AR(1)] can be clearly recognized lower than the red one [AR(1) WGN] at the
beginning especially, in the single scale case. As scale increases, the two
lines approach the same level. Similar trend can be seen for the other two
pairs.

Figure 9: Illustration of directionality of a) standard MMSE and b) the
proposed veMSE.

In the second graph of Figure.9, the analysed signals are AR(1), AR(2) and
AR(3) with one of the signals in each system associated with white noise. The
legend [AR AR+WGN] refers to cases where noise-free signals are the first
variate followed by noisy signals and vice versa for [AR+WGN AR]. This setting
of the inputs was used to stimulate real scenarios when dealing with multi-
channel signals, whereby one of them represents a poor recording with noise.
Considering the noise-robustness property of veMSE, the amplitude of noise
signals in this subsection is enlarged to the same level as for AR signals, to
demonstrate a clear difference when the input order is modified. As shown in
the figure, the inverted input orders can be reflected by different start
levels, while the curves then approach each other as well as end with similar
estimates. Therefore, regardless of the input order, the separation of
complexity levels of AR models can be successfully achieved. And observed in
Figure.9 in case of MMSE, the inverted input exhibited no influence on the
resulted curves in small scales, while similar performance as veMSE in the
larger scale.

As demonstrated in Figure.9, the reversed order has limited influence on the
estimation in high scales as all the paired curves approach to the same three
regions, respectively, while in spite of the modified order, the three models
can be separated. However, as similar phenomenon is shown in the top graph,
the varying order of the input signal will generate entropy values with
different degrees in small scale analysis when the input signals contain
distinct structure. Therefore, the direction of the input order needs to be
carefully considered when applying small scale analysis, and such
consideration can be ignored in high scale analysis with identical system
measurement.

###  IV-C Computational Complexity

Admittedly, entropy analysis based on multichannel signals is more time-
consuming than other single variate estimation as it should be. Nevertheless,
for potential applications as real time processing in the future, calculation
efficiency is one of the critical factors that need to be carefully
considered. Therefore, in this subsection, time consumption of veMSE is
discussed and compared with commonly used MMSE and the results are given
below.

Figure 10: Computation time for MMSE and the proposed veMSE with modified
parameters for white Gaussian input.

Figure.10 shows the processing time as a function of various modified
parameters when implementing the veMSE (blue line) and MMSE (red line). All
the curves are given as an average over 10 implement realizations. Each graph
is designed to reflect the behaviours for only one modified parameter, when
the independent variables in the following figures, from the left- to the
right-hand side, as the scale factor, length of data, number of channels and
embedding dimension. All the entropy calculations were set as single-variate
and bivariate processing by default. The data length and embedding dimension
were irrelevant variables were fixed to $N$ = 5000 and $m$ = 2.

Overall, the red line, representing computational time of MMSE, is above the
blue, that of veMSE, for all the scenarios. Actually, the increase of scale
factor and the decrease of data length reflect that when the data size after
‘coarse-graining’ procedure is lower than 1000, the times needed for the two
calculations are similar, as shown in the first graph where the scale factor
is higher than 5 or when the data length is shorter than 1000 in the second
graph. As for the influence of channel number, demonstrated in the third
graph, it is reasonable that MMSE need more time as the channel number
increases, because the key step for Sample Entropy is the ratio of conditional
probability for similar patterns between the embedding dimension, $m$, and its
increase, $m+1$, the number of possible ways to apply the ($m+1$) dimension is
equal to the number of channels involved when forming the composite delay
vector in MMSE. Therefore, the calculation with an increased embedding
dimension will be repeated $c$ times in MMSE, where $c$ denotes the number of
channels. Finally, in the relationship between an increased dimension and
computation time, the time difference roughly maintains a fixed value in spite
of the dimension changing in the last graph.

In general, with reference to the widely applied MMSE, the time needed for the
same amount of data with veMSE is shorter. This reflects that the calculations
efficiency of veMSE is higher than MMSE, which is of high potential interest
to implement real-time monitoring of human states in the future.

##  V Performance of veMSE on real signals

###  V-A Wind Dynamics

Having illustrated how veMSE performs on synthetic signals, we now examine the
performance of veMSE in real-world systems. In the first place, wind dynamics
were examined. The long-term correlations in wind dynamics has been revealed
in previous studies by using detrended fluctuation analysis (DFA) [28] and
standard Multivariate Multiscale Sample Entropy (MMSE) [19]. Here, the
proposed veMSE was utilized to exhibit an improved performance of
characterizing different dynamical complexity of wind regimes.

The applied database of wind was recorded by 3D ultrasonic anemometers at a
sampling frequency of 50 Hz. The recording process was implemented in the
Institute of Industrial Science of the University of Tokyo. Three channels
were obtained and they are in east-west, north-south, and vertical direction
respectively. The wind regimes containing different dynamics were defined as
low, medium, and high by the magnitude of wind speed as examples shown in
Figure. 11. The parameters involved in veMSE analysis were set to $m$ = 2, $L$
= 1, $r=0.2\times tr(S)$, and $N$ = 3000. Illustrations of entropy results
were averaged over 10 trials for each channel and exhibited as error-bar. In
addition, shuffled wind data sets were also tested and compared with obtained
wind dynamics, which were generated from the recorded wind signals but with a
random order. In this case, the possible correlations within signals were
broken down but the statistical properties were preserved.

Figure 11: Magnitude of the wind signal. The wind segments are defined as low,
medium, and high regimes.

Before the comparison between veMSE and MMSE, univariate Multiscale Sample
Entropy (MSE) was firstly applied to give an insight into the complexity of
the dynamical system to be processed. As shown in Figure.12, the randomized
data sets exhibit a white noise-like behaviour in terms of structural
complexity, which is expected as the correlation was destroyed while
shuffling. The MSE results of wind dynamics assigned the highest complexity to
the medium regime which is expected since the medium wind speed hold the
fewest constrains. Yet, the complexity of low wind regime was quantified
higher than the high regime which violates the intuition, as the high wind
regime also contains components with medium speed. Besides, each wind dynamics
exhibited lower complexity than the their randomized series, wrongly resulting
to the absence of long-term correlations within wind data sets.

The same trails were analyzed by Multivariate Multiscale Sample Entropy as
shown in Figure.12 which revealed the relation of the three wind regimes in a
expected way that mild wind exhibited the highest complexity, followed by the
high and low dynamics regimes. Further, all the three wind regimes were able
to present long-range correlation higher than their surrogate data sets.
However, overlapped area can be found across all the scales. While in
Figure.12, observe, as desired, the long-range correlation of different wind
speed was able to be detected resulting from the proposed veMSE with more
apparent separation on basis of the correct analysis compared to MSE and MMSE.

Figure 12: Results of a) standard single-variate MSE, b) standard multi-
channel MMSE, c) proposed veMSE based on wind dynamics and randomized
dynamics.

###  V-B Physiological Database

Now, physiological data sets were involved to exhibit the performance of
complexity quantification based on entropy analysis. The physiological signals
utilized is Fantasia database [29] that includes R-R intervals (RRI) and
interbreath interval (IBI) extracted from electrocardiograph (ECG) and
respiration signals respectively. The structure of long-term correlation in
heart rate variability and respiratory dynamics have been examined by
traditional method as Detrended Fluctuation Analysis (DFA) [30] and standard
MMSE [19].

Fantasia database were recorded from 20 young people (at the age of 21-34) and
20 elderly participants (ageing from 68-85) who were rigorously screened for
120 minutes. 10 subsets out of them were chosen in each group. The ECG and
respiration signals were recorded with sample frequency at 250 Hz. Then, RRI
and IBI signals were extracted and aligned. The surrogate series with
randomised order were analyzed along with RRI and IBI signals in each
methodology to reveal the effectiveness of the complexity. Parameters in
entropy analysis were set to $m$ = 2, $L$ = 1, $r=0.15\times tr(S)$, and $N$ =
4000.

In the first place, to give an overall view of the signals in each channel.
Univariate Multiscale Sample Entropy was applied to single channel separately
as exhibited in Figure.14 and Figure.14. As Complexity Loss Theory [31]
stated, the adaptive capacity of bio-systems is damaged by disease and aging.
The complexity of RRI and IBI signals recorded from young people is supposed
to be higher than that from elderly subjects when considering long-term
correlation. In respect of RRI in short-term correlation, MSE exhibited a
correct estimation under certain low scales. However, both MSE on basis of RRI
and IBI are unable to reveal the correct relation between the dynamics of
elder individuals and young people in a long-range term.

Figure 13: Operation of Univariate Multiscale Sample Entropy based on R-R
interval.

Figure 14: Operation of Univariate Multiscale Sample Entropy based on
interbreath interval (IBI).

Next, observed in Figure.16, MMSE was applied to the bivariate channels. The
resulted error bar indicated the higher dynamical complexity in young
participants than that in elderly individuals which conformed the complexity
loss theory with aging [31]. Moreover, both physiological data sets showed
higher long-range correlation than the randomized surrogate series in an
expected way, yet as scale increases, overlapping area can be observed due to
the lack of sample length after coarse graining process. While in contrast to
MMSE, performance of veMSE in Figure.16 remained the correct and clear
estimation but with higher stability particularly in large scales. In
practical scenarios, the size of recorded signals is mostly limited. Hence,
veMSE can better facilitate the analysis of physical and physiological signals
when identifying dynamical difference based on nonlinear features.

Figure 15: Operation of Multivariate Multiscale Sample Entropy.

Figure 16: Operation of Variational Embedding Multiscale Sample Entropy.

##  VI Conclusion

To conclude, the Variational Embedding Multiscale Sample Entropy (veMSE)
method has been introduced for enhanced complexity analysis of real-world
data. The results of this paper indicate that veMSE is capable of exhibiting
the complex features of the system in large scale with higher embedding
dimension compared to MMSE and veMDE. Besides, the utilization of multivariate
analysis via veMSE could guarantee an improvement in contrast to single-
variate analysis regardless of the quality of the recorded signals in sub-
channels. Moreover, veMSE shows a strong noise robustness feature and
meanwhile, the time consumption of veMSE is lower than MMSE under the same
conditions as well, and this improvement is particularly apparent as the
increase of available channels. The higher calculation efficiency is of high
potential interest to apply entropy analysis into scenarios where need real-
time processing or synchronized monitor in the future application.
Nevertheless, it should be noted that this method is restricted to amplitude-
based distance. Future research can be intended to put focus on angular
distance-based associated with variational embedding dimension methodology.

##  Appendix A Algorithm of Multivariate Multiscale Sample Entropy

Algorithm 2 Multivariate Multiscale Sample Entropy

The steps of standard Multivariate Multiscale Sample Entropy are given below.
For a multi-variate data set $\\{x_{c,i}\\}_{i=1}^{N},\,1\leq c\leq P$ with
length of $N$ and number of channel $P$. Manually selected parameters are the
embedding dimension ($M=[m_{1},m_{2},\dots,m_{P}]$), tolerance ($r$), time
delay ($L=[l_{1},l_{2},\dots,l_{P}]$) and scale factor ($\tau$):

  1. 1.

Normalize the original multi-variate data sets by subtracting the mean and
dividing by the standard deviation.

  2. 2.

Perform Coarse Graining Process to obtain the scaled multi-channel time series
$\\{y_{c,i}^{(\tau)}\\}_{j=1}^{N/\tau}$ following the equation

$y_{c,i}^{(\tau)}(j)=\frac{1}{\tau}\sum^{j+\tau/2-1}_{i=j-\tau/2-1}{x_{c}(i)},\quad
1\leq j\leq\frac{N}{\tau},\,c=1,2,\dots,P$

  3. 3.

Form the Composite Delay Vectors (CDV) $\textbf{Y}_{M}(i)$ according to $M$
and $L$ in a form as

$\displaystyle\textbf{Y}_{M}(i)=[$ $\displaystyle
y_{1,i},y_{1,i+l_{1}},\dots,y_{1,i+(m_{1}-1)l_{1})},$ $\displaystyle
y_{2,i},y_{2,i+l_{2}},\dots,y_{2,i+(m_{2}-1)l_{2})},$
$\displaystyle\qquad\qquad\qquad\vdots$ $\displaystyle
y_{c,i},y_{c,i+l_{c}},\dots,y_{c,i+(m_{c}-1)l_{c})},]$

  4. 4.

Compute the similarity for all pairwise CDVs ($Y_{M}(i)\,\&\,Y_{M}(j)$) based
on Chebyshev distance as

$D(i,j)=max\\{Y_{M}(i+k)-Y_{M}(j+k)||\,0\leq
k\leq((\sum_{c=1}^{P}{m_{c}})-1),i\neq j\\}$

  5. 5.

Calculate the number of matching patterns defined as similar pair $B(i)$ that
satisfies the criterion $D(i,j)\leq r$.

  6. 6.

Compute the local probability $C(i)$ and global probability $\Phi(i)$ of
$B(i)$ by

$C(i)=\frac{B(i)}{N-n-1},\quad\Phi=\frac{\sum_{i=1}^{N-n}{{C(i)}}}{N-n},\,n=max(M)*max(L)$

  7. 7.

Repeat Steps 3-6 with modified embedding dimension as ($m_{c}$+1) and obtain
the updated global probability as

$\Phi^{*}=\frac{\sum_{i=1}^{N-n}{{C^{*}(i)}}}{N-n},\,n=max(M^{*})*max(L)$

Recall that there are $P$ ways to increase the embedding dimension and the
modified global probability, $\Phi^{*}$, is the averaged result.

  8. 8.

Multivariate Multiscale Sample Entropy is defined as

$MMSE=-\ln{[\frac{\Phi^{*}}{\Phi}]}$

## Acknowledgment

We wish to thank Institute of Industrial Science of the University of Tokyo
for providing the wind data sets used in this article.

## References

  * [1] X. Wang, S. Si, and Y. Li, “Multiscale diversity entropy: A novel dynamical measure for fault diagnosis of rotating machinery,” IEEE Transactions on Industrial Informatics, vol. 17, no. 8, pp. 5419–5429, 2020.
  * [2] Y. Lv, R. Yuan, T. Wang, H. Li, and G. Song, “Health degradation monitoring and early fault diagnosis of a rolling bearing based on ceemdan and improved mmse,” Materials, vol. 11, no. 6, p. 1009, 2018.
  * [3] Y. Li, X. Wang, S. Si, and S. Huang, “Entropy based fault classification using the case western reserve university data: A benchmark study,” IEEE Transactions on Reliability, vol. 69, no. 2, pp. 754–767, 2019.
  * [4] D. O’Hora, S. Schinkel, M. J. Hogan, L. Kilmartin, M. Keane, R. Lai, and N. Upton, “Age-related task sensitivity of frontal eeg entropy during encoding predicts retrieval,” Brain topography, vol. 26, no. 4, pp. 547–557, 2013.
  * [5] F. C. Morabito, D. Labate, F. La Foresta, A. Bramanti, G. Morabito, and I. Palamara, “Multivariate multi-scale permutation entropy for complexity analysis of alzheimer’s disease eeg,” Entropy, vol. 14, no. 7, pp. 1186–1202, 2012.
  * [6] M.-X. Xiao, H.-C. Wei, Y.-J. Xu, H.-T. Wu, and C.-K. Sun, “Combination of rr interval and crest time in assessing complexity using multiscale cross-approximate entropy in normal and diabetic subjects,” Entropy, vol. 20, no. 7, p. 497, 2018.
  * [7] A. L. Goldberger, L. A. Amaral, J. M. Hausdorff, P. C. Ivanov, C.-K. Peng, and H. E. Stanley, “Fractal dynamics in physiology: alterations with disease and aging,” Proceedings of the national academy of sciences, vol. 99, no. suppl 1, pp. 2466–2472, 2002.
  * [8] T. Higuchi, “Approach to an irregular time series on the basis of the fractal theory,” Physica D: Nonlinear Phenomena, vol. 31, no. 2, pp. 277–283, 1988\.
  * [9] C. L. Webber Jr and J. P. Zbilut, “Dynamical assessment of physiological systems and states using recurrence plot strategies,” Journal of applied physiology, vol. 76, no. 2, pp. 965–973, 1994.
  * [10] L. Ni, J. Cao, and R. Wang, “Analyzing eeg of quasi-brain-death based on dynamic sample entropy measures,” Computational and mathematical methods in medicine, vol. 2013, 2013.
  * [11] T. Chanwimalueang and D. P. Mandic, “Cosine similarity entropy: Self-correlation-based complexity analysis of dynamical systems,” Entropy, vol. 19, no. 12, p. 652, 2017.
  * [12] M. D. Costa, C.-K. Peng, and A. L. Goldberger, “Multiscale analysis of heart rate dynamics: entropy and time irreversibility measures,” Cardiovascular Engineering, vol. 8, no. 2, pp. 88–93, 2008.
  * [13] Y. Zhang, J. Li, and J. Wang, “Exploring stability of entropy analysis for signal with different trends,” Physica A: Statistical Mechanics and its Applications, vol. 470, pp. 60–67, 2017.
  * [14] S. M. Pincus, “Approximate entropy as a measure of system complexity.,” Proceedings of the National Academy of Sciences, vol. 88, no. 6, pp. 2297–2301, 1991.
  * [15] J. S. Richman and J. R. Moorman, “Physiological time-series analysis using approximate entropy and sample entropy,” American Journal of Physiology-Heart and Circulatory Physiology, vol. 278, no. 6, pp. H2039–H2049, 2000.
  * [16] M. Costa, A. L. Goldberger, and C.-K. Peng, “Multiscale entropy analysis of complex physiologic time series,” Physical review letters, vol. 89, no. 6, p. 068102, 2002.
  * [17] S.-D. Wu, C.-W. Wu, S.-G. Lin, C.-C. Wang, and K.-Y. Lee, “Time series analysis using composite multiscale entropy,” Entropy, vol. 15, no. 3, pp. 1069–1084, 2013.
  * [18] S.-D. Wu, C.-W. Wu, S.-G. Lin, K.-Y. Lee, and C.-K. Peng, “Analysis of complex time series using refined composite multiscale entropy,” Physics Letters A, vol. 378, no. 20, pp. 1369–1374, 2014.
  * [19] M. U. Ahmed and D. P. Mandic, “Multivariate multiscale entropy: A tool for complexity analysis of multichannel data,” Physical Review E, vol. 84, no. 6, p. 061918, 2011.
  * [20] C. Bandt and B. Pompe, “Permutation entropy: a natural complexity measure for time series,” Physical review letters, vol. 88, no. 17, p. 174102, 2002\.
  * [21] M. U. Ahmed, T. Chanwimalueang, S. Thayyil, and D. P. Mandic, “A multivariate multiscale fuzzy entropy algorithm with application to uterine emg complexity analysis,” Entropy, vol. 19, no. 1, p. 2, 2017.
  * [22] W. Chen, Z. Wang, H. Xie, and W. Yu, “Characterization of surface emg signal based on fuzzy entropy,” IEEE Transactions on neural systems and rehabilitation engineering, vol. 15, no. 2, pp. 266–272, 2007.
  * [23] X. Wang, S. Si, and Y. Li, “Variational embedding multiscale diversity entropy for fault diagnosis of large-scale machinery,” IEEE Transactions on Industrial Electronics, 2021.
  * [24] R. Alcaraz, D. Abásolo, R. Hornero, and J. Rieta, “Study of sample entropy ideal computational parameters in the estimation of atrial fibrillation organization from the ecg,” in 2010 Computing in Cardiology, pp. 1027–1030, IEEE, 2010.
  * [25] C.-K. Peng, S. Havlin, H. E. Stanley, and A. L. Goldberger, “Quantification of scaling exponents and crossover phenomena in nonstationary heartbeat time series,” Chaos: an interdisciplinary journal of nonlinear science, vol. 5, no. 1, pp. 82–87, 1995.
  * [26] A. Humeau-Heurtier, “The multiscale entropy algorithm and its variants: A review,” Entropy, vol. 17, no. 5, pp. 3110–3123, 2015.
  * [27] S. Makeig, C. Kothe, T. Mullen, N. Bigdely-Shamlo, Z. Zhang, and K. Kreutz-Delgado, “Evolving signal processing for brain–computer interfaces,” Proceedings of the IEEE, vol. 100, no. Special Centennial Issue, pp. 1567–1584, 2012.
  * [28] R. Govindan and H. Kantz, “Long-term correlations and multifractality in surface wind speed,” EPL (Europhysics Letters), vol. 68, no. 2, p. 184, 2004.
  * [29] A. L. Goldberger, L. A. Amaral, L. Glass, J. M. Hausdorff, P. C. Ivanov, R. G. Mark, J. E. Mietus, G. B. Moody, C.-K. Peng, and H. E. Stanley, “Physiobank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals,” circulation, vol. 101, no. 23, pp. e215–e220, 2000.
  * [30] N. Iyengar, C. Peng, R. Morin, A. L. Goldberger, and L. A. Lipsitz, “Age-related alterations in the fractal scaling of cardiac interbeat interval dynamics,” American Journal of Physiology-Regulatory, Integrative and Comparative Physiology, vol. 271, no. 4, pp. R1078–R1084, 1996\.
  * [31] L. A. Lipsitz and A. L. Goldberger, “Loss of’complexity’and aging: potential applications of fractals and chaos theory to senescence,” Jama, vol. 267, no. 13, pp. 1806–1809, 1992.