# Topological data analysis of human vowels:
Persistent homologies across representation spaces

Guillem Bonafos1234, Pierre Pudlo13, Jean-Marc Freyermuth135, Samuel Tronçon4,
Arnaud Rey23

5This publication is based upon work supported by King Abdullah University of
Science and Technology Research Funding (KRF) under Award No.
ORFS-2022-CRG11-5025.2 1Aix Marseille Univ, CNRS, I2M, Marseille, France  2Aix
Marseille Univ, CNRS, LPC, Marseille, France  3Aix Marseille Univ, ILCB, Aix-
en-Provence, France  4Résurgences R&D, Arles, France

###### Abstract

Topological Data Analysis (TDA) has been successfully used for various tasks
in signal/image processing, from visualization to supervised/unsupervised
classification. Often, topological characteristics are obtained from
persistent homology theory. The standard TDA pipeline starts from the raw
signal data or a representation of it. Then, it consists in building a
multiscale topological structure on the top of the data using a pre-specified
filtration, and finally to compute the topological signature to be further
exploited. The commonly used topological signature is a persistent diagram (or
transformations of it). Current research discusses the consequences of the
many ways to exploit topological signatures, much less often the choice of the
filtration, but to the best of our knowledge, the choice of the representation
of a signal has not been the subject of any study yet. This paper attempts to
provide some answers on the latter problem. To this end, we collected real
audio data and built a comparative study to assess the quality of the
discriminant information of the topological signatures extracted from three
different representation spaces. Each audio signal is represented as i) an
embedding of observed data in a higher dimensional space using Taken’s
representation, ii) a spectrogram viewed as a surface in a 3D ambient space,
iii) the set of spectrogram's zeroes. From vowel audio recordings, we use
topological signature for three prediction problems: speaker gender, vowel
type, and individual. We show that topologically-augmented random forest
improves the Out-of-Bag Error (OOB) over solely based Mel-Frequency Cepstral
Coefficients (MFCC) for the last two problems. Our results also suggest that
the topological information extracted from different signal representations is
complementary, and that spectrogram's zeros offers the best improvement for
gender prediction.

###### Index Terms:

TDA, topologically-augmented machine learning, persistent homology,
representation space, signal classification, human vowel.

##  I Introduction

Topological Data Analysis (TDA) is a fast-growing research area that relies on
deep mathematical foundations [1, 2, 3]. It offers novel and potentially
fruitful angles of analysis of digital audio signals. This innovative approach
to data science is based on extracting information from the _shape of data_.

TDA has already been applied to various signal processing problems [4, 5]. It
starts from the assumption that the data have a shape [6] and it computes its
persistent homologies, which provide a compact representation of its
topological features. These are stable to perturbations of input data and
independent of dimensions and coordinate systems. However, this shape strongly
depends on the way a signal is represented (i.e., on the representation
space). The purpose of this work is to study how the computation of persistent
homologies depends on the chosen representation spaces, considering the
specific problem of vowels categorization in human language. We study the
impact of the representation space on the extracted topological information
and determine if accessing higher-dimensional persistent homologies allows
getting more discriminant information. We also discuss what is the best way to
summarize the information contained in a persistence diagram for our specific
classification tasks.

This article is organized as follows. First, we introduce the problem and its
rationale. Second, we describe in a nutshell the theory and the processing
pipeline of TDA. Third, we present the strategy for investigating the problem,
the nature of the acquired dataset and our classification aims. Fourth, we
report the main results that are extensively discussed in the last section.

##  II The problem

###  II-A Motivations for TDA

Topology is the branch of mathematics that deals with the qualitative
geometric information of a space [1]. The tools provided by algebraic topology
allow us to capture the shape of the data [7]. The topological approach frees
itself from the question of metrics and coordinates by studying the properties
of a space through its connectivity. It has an interesting explanatory power
thanks to its great potential for visualization, and the topological features
have a discriminant power which makes TDA a particularly interesting candidate
for the classification of natural signals. In this section, we give an
overview of application of TDA in signal processing. Additional details on the
theoretical foundations of TDA are given in Section III.

A central feature in TDA is the computation of persistent homologies [8].
Several pipelines have been proposed for the computation and use of such
topological descriptors in data analysis. It typically consists in calculating
persistent homologies on the input data (e.g. an audio signal) or a
representation of it (e.g., its spectrogram), vectorizing persistence
diagrams, and using these characteristics in a model [3, 9]. Persistence
diagrams and some of their representations have been shown to be stable
against noise [10, 3, 11], meaning that small perturbations of the input data
results in small changes in the persistent diagram. This stability makes the
topological approach an excellent candidate for the description of natural
signals. Altogether, the interesting perspectives offered by TDA to face Big
Data challenges combined with the fast development of tools for the efficient
computation of topological descriptors has led to a proliferation of studies
demonstrating the added value of TDA in a variety of contexts.

For example, [4, 5] demonstrate the usefulness of TDA for signal processing
and for the study of signals on graphs. Topological tools are also useful in
analyzing the shape of time series [12], for object detection in images [13]
or in sound detection [14]. Besides detection problems, topological
descriptors are also useful for the classification of sound signals [15] and
musical signals [16]. It impacts multiple scientific domains alike biology,
medicine, ecology [17], neurosciences, e.g., for fMRI [18] or EEG data [19,
20] for which TDA allows the construction of invariant signal descriptors.
Topological features are complementary to more classical descriptors, and they
allow capturing global and high-dimensional information useful for signal
analysis. However, the impact of the representation space of the signal on the
extracted topological information is, to our knowledge, still an unexplored
question.

###  II-B Representation space

Our study focuses on physical signals, in particular, sound signals. A raw
data signal can be represented as different _data objects_ , in different,
here-called, representation spaces. Figure 1 shows the same raw signal in
three different ways: as a spectrogram, viewed as a surface in 3D Euclidean
ambient space, as the spectrogram's zeros, or as a point cloud using Taken's
embedding.

(a) Wave

(b) Spectrogram's surface

(c) Spectrogram's zeros

(d) Taken's embedding

Figure 1: Different representation of the same signal. 1a is the initial wave
of the sound; 1b) is the surface of its spectrogram; 1c is the zeros of its
spectrogram in the time-frequency plane; 1d is its Taken's embedding.

Flandrin [21] refers to time-frequency analysis as the language of signal
processing. Among the plethora of existing representation methods, the
spectrogram is certainly the most widely used. Let us consider a sound signal
$x\in L_{1}(\mathbb{R})\cap L_{2}(\mathbb{R})$, its spectrogram
$S_{x}^{(h)}(t,\omega)$ is defined as:

$S_{x}^{(h)}(t,\omega)=|F_{x}^{(h)}(t,\omega)|^{2},$ (1)

where $(t,\omega)\in\mathbb{R}^{2}$ are time and frequency variables,
$h(t)\in\mathbb{R}$ is a window function and $F_{x}^{(h)}(t,\omega)$ is the
Short-Time Fourier Transform (STFT):

$F^{(h)}_{x}(t,\omega)=\int_{-\infty}^{+\infty}x(s)h(s-t)\exp\\{-i\omega(s-t/2)\\}ds.$
(2)

We can therefore represent the spectrogram of a one dimensional sound signal
$x$, with time window function $h(\cdot)$, as a surface
$\mathcal{S}_{x}^{(h)}$ in a 3D ambient Euclidean space where the dimensions
are time, frequency and amplitude,
$\mathcal{S}_{x}^{(h)}:=\left\\{\left(t,\omega,S^{(h)}_{x}(t,\omega)\right)|(t,\omega)\in\mathbb{R}^{2}\right\\}\subset\mathbb{R}^{3}$
(see Figure 1b). This representation has been proven effective in revealing
geometrical structures, enhancing classification performances [22]. Another
possibility is to rather consider the spectrogram's zeros as introduced in
[23]. Choosing the window function to be Gaussian, i.e,
$h(t)=\pi^{-1/4}\exp\left\\{-t^{2}/4\right\\}$, we readily show that the STFT
can be rewritten as follows:

$F_{x}^{(h)}(t,\omega)=\exp\left\\{-|z|^{2}/4\right\\}\mathcal{F}_{x}(z),$ (3)

where, $z=\omega+it$ and $\mathcal{F}_{x}(z)$ is the Bargmann transform of
$x$. It is easy to see that it is an entire function of order $2$, which
admits a Weierstrass-Hadamard form:

$\mathcal{F}_{x}(z)\propto\prod_{n=1}^{\infty}\left(1-\frac{z}{z_{n}}\right)\exp\left\\{\frac{z}{z_{n}}+\frac{1}{2}\left(\frac{z}{z_{n}}\right)^{2}\right\\},$
(4)

where $\mathcal{Z}_{x}:=\left\\{z_{n}=\omega_{n}+it_{n}\right\\}_{n}$ is the
set of zeroes of $\mathcal{F}_{x}(z)$. The spectrogram is therefore completely
characterized by its zeroes, in other words, it can be represented by a point
cloud in the time-frequency plane. [21] pioneered the idea of using
topological characteristics for describing a spectrogram by visualizing the
distribution of the edges of its zero-based Delaunay-triangulation. In the
sequel, we build on this idea for extracting topological characteristics using
the tools described hereafter in Section (III).

We introduce another representation of sound signals that is not related to
time-frequency analysis. Considering a sound signal as a discrete-time digital
audio recording (or time series) $\left\\{x_{1},\ldots,x_{T}\right\\}$ and
assuming it comes from a dynamic system, we can borrow the tools of dynamical
system analysis to find out an informative representation. The Taken's theorem
states that it is possible to obtain a representation of this time series that
is topologically equivalent to the attractor of the system via a delay
embedding, which contains useful information about the system [24]. We thus
transform the digital audio recording into a point cloud in a higher
dimensional space,
$\mathcal{P}_{D,x}=\left\\{p_{1},\ldots,p_{m}\right\\}\subset\mathbb{R}^{D}$,
each element $p_{i}\in\mathcal{P}_{D,x}$ is a vector of dimension $D$,
constructed by taking a time delay $\tau$:

$p_{i}=(x_{i},x_{i+\tau},x_{i+2\tau},...,x_{i+(D-1)\tau})^{\prime}.$ (5)

We therefore embed a one dimensional digital audio recording into a higher
dimensional space to obtain a point cloud. It is necessary to estimate the two
hyperparameters: $D$, the dimension of the space, and $\tau$, the time delay.
$D$ is estimated using the Cao's algorithm [25]. $\tau$ is selected using the
Average Mutual Information (AMI). The coordinates of the phase-space embedding
must be independent enough (to avoid aggregation around the diagonal in the
embedding). $\tau$ is then chosen to be the smallest value such that
$AMI(\tau)<\frac{1}{e}$. An example of this point cloud representation is
shown in Figure 1d.

###  II-C Same signal, different representations, different topological
characteristics

We described three ways of representing the same audio signal. It naturally
raises some questions, such as, is there a representation that carries more
topological (discriminant) information than the other? To get an intuition
about this question, Figure 2b shows three persistence diagrams computed on
the representations of the same signal illustrated on Figure 1 (see section
IV-C for more information about how this topological information is computed).
Although the signal is the same, the extracted topological characteristics are
very different because of the choice of the representation. Since the
representation spaces can be of different dimensions, the homological features
describing the topology of the object of these spaces will also be of
different dimensions. This raises another, more specific questions, is the
access to higher dimensions providing relevant information? We address these
issues in a quantitative way, through a case study, a classification problem
of human vowels.

(a) Persistence diagram of the spectrogram's surface

(b) Persistence diagram of the spectrogram's zeros

(c) Persistence diagram of the Taken's embedding

Figure 2: Three persistent diagrams computed for the three different
representation of the same signal of Figure 1. 2c is the persistence diagram
of the spectrogram's surface using sublevel sets; 2b is the persistence
diagram of the zeros of the spectrogram using an Alpha complex; 2c is the
persistence diagram of the Taken's embedding using an Alpha complex. The
points in the diagram represent the computed persistent homologies. The
different colors represent the different dimensions: black for $p=0$, red for
$p=1$ and blue for $p=2$.

##  III Topological Data Analysis: An overview

This section presents the theory and data analysis pipeline of TDA in a
nutshell. For more details, we refer the reader to two important textbooks
that offer a fairly broad presentation of TDA and its theoretical foundations
(i.e., [26, 27]) as well as the excellent introduction for data scientists
[3].

###  III-A Persistent homology

In a prosaic summary, TDA is the enumeration of holes of different dimensions
in the _shape of data_. This intuitive notion of holes is formalized by the
mathematical concept of a homology group. Homology groups allow us to treat
the holes of a topological space mathematically, by studying the connectivity
of the space. Let us denote as $H_{p}(\mathbb{M})$ the $p^{th}$ homology group
of the topological space $\mathbb{M}$. Every homology group has the
topological properties of its dimension. For $p=0$, $H_{0}(\mathbb{M})$ takes
the connected elements of $\mathbb{M}$, for $p=1$, $H_{1}(\mathbb{M})$ takes
the tunnels, for $p=2$, the voids. The rank of $H_{p}(\mathbb{M})$ defines the
$p^{th}$ Betti numbers, denotes as $\beta_{p}=rank(H_{p})$ hereafter. Betti
numbers basically count the number of different topological features. Homology
groups and Betti numbers are topological invariants which characterizes the
shape of $\mathbb{M}$.

Instead of the homology groups of $\mathbb{M}$, we are interested in its
persistent homology groups. We compute them through a filtration $F$ of
$\mathbb{M}$. A filtration $F$ is a parametrized nested family of subspaces
$F=(M_{r})_{r\in T},$ where $T\subseteq\mathbb{R},$ such that for any
$r,r^{\prime}\in T,$ if $r\leq r^{\prime},$ $M_{r}\subseteq M_{r^{\prime}},$
and $\mathbb{M}=\cup_{r\in T}M_{r}$. The parameter $r\in T$ is the scale
parameter. Let $f:\mathbb{M}\mapsto\mathbb{R}$ and $M_{r}=f^{-1}(-\infty,r]$
be the sublevel set for value $r$, the family $\left\\{M_{r}\right\\}_{r\in
T}$ is the sublevel set filtration. A filtration allows multiscale topological
description; for each value of $r$ we have the associated homology groups. The
persistence of homologies makes it possible to track the lifetime of
homologies, to determine their birth and their death according to the scale
parameter $r$.

We denote as $\mathbb{X}$ our data objects that can be either
$\mathcal{Z}_{x}$ or $\mathcal{P}_{x,D}$ (point clouds), or with the two-
dimensional surface $\mathcal{S}_{x}^{(h)}$. A point cloud typically does not
carry interesting topological information, nevertheless, it is possible to
retrieve some via the construction of a simplicial filtration on the top of
it. We can define a simplicial filtration by constructing a simplicial complex
on the data. The filtration is then a nested sequence of simplicial complexes,
for each of which we compute the (simplicial) homologies. In this paper, we
use the Alpha complexes, a family of subcomplexes of the Delaunay complex,
since it is equivalent but smaller than other complexes such as the Cech
complexes. For any $x\in\mathbb{X}$, with $\mathbb{X}\subset\mathbb{R}^{d}$,
and $r\in\mathbb{R}^{+}$, we define $B_{x}(r)=x+r\mathbb{B}^{d}$ the closed
ball centered at $x$ and of radius $r$. The union of these balls is the set of
points at a distance at most $r$ from at least one of the points of
$\mathbb{X}$. We define the Alpha complex

$Alpha(r)=\\{\sigma\subseteq\mathbb{X}|\cap_{x\in\sigma}R_{x}(r)\neq\emptyset\\},$
(6)

where $R_{x}(r)$ is the intersection of each Euclidean ball with its
corresponding Voronoi cell. In the context of filtration, we construct a
nested sequence of Alpha complexes on $\mathbb{X}$, taking an increasing value
of $r$.

We recover the homological information of $\mathbb{X}$ computed for different
scale values $r$. The birth and death of the topological features (the
elements that make up each homology group) are recorded and summarized in the
persistence diagram, for each value of $r$. Thus, each persistent topological
feature resulting from the filtration is expressed by a pair $(b_{i},d_{j})$,
the moment of its birth and its death, i.e., two values of $r$. The most
persistent homologies are those for which the difference $d_{j}-b_{j}$ is
maximal [28]. We define a persistence diagram as a multiset of points in
$\mathcal{D}:=D\times\\{1,...,P\\}$, where

$D:=\\{(b,d)\in\mathbb{R}^{2}|d\geq b\geq 0\\}.$ (7)

Each triplet $(b,d,p)\in\mathcal{D}$ represents a $p$-dimensional homological
feature that appears when $r=b$ and disappears when $r=d$ [29].

In our experiment, we use filtration adapted to the representation spaces.
More specifically, the alpha-complex filtration for Taken's embeddings and
spectrogram zeroes and the sublevel set filtration for the spectrogram
surface.

###  III-B Exploitation of the information from the diagram space

Persistence diagrams provide multiscale homological descriptions of the data.
However, the space of persistence diagrams has very complicated geometry and
topology which makes it difficult to exploit directly [30]. Many strategies
were developed in order to extract the relevant information and enable machine
learning applications. We refer the readers to [31, 32] that offer
comprehensive review of existing methods.

Common approaches consist in computing functional representations alike
persistent surfaces [33], persistence landscapes [34] or persistent
silhouettes [11], and then, to discretized them to be used as vector-based
input for machine learning algorithms. These representations have been proven
to enjoy stability properties [35] meaning that they are slightly modified by
small variations in the input data. Their nice properties and ease to compute
make then widely employed in applications. For example, [15] uses persistence
landscapes as input to a CNN for musical signal classification, and PLLay [36]
is a topological layer based on persistence landscapes.

Instead of functional representations, persistence diagrams can be summarized
by scalar descriptors. There is a myriad of such, among which, the $p$-norm of
the persistence diagram [37] or the persistent entropy [38]. Both are proven
to be stable to perturbations of the input data [39]. Aggregating those
descriptors together appears to be an interesting strategy. [40] construct a
stable vector of the persistence diagram from the distances between points and
between each point and the diagonal, while [14] use a whole set of descriptors
on the persistence diagrams. In the sequel, we will consider sets of
descriptors as well as some standard functional summaries.

##  IV Experiments

In this section, we investigate how the choice of representation space of a
digital audio signal affects topological information. This information is
quantified in terms of Out Of Bag (OOB) error in a supervised classification
problem inspired by the topical question in acoustic of vowel classification
[41, 42]. It finds applications in emotion classification [43], in evaluating
developmental trouble [44] or in order to distinguish between healthy patients
and patients with neurological disorder [45].

We collected our own dataset consisting of French-speaking adults pronouncing
vowels indoors. For each of these recordings, we extract homological
information from three possible representation spaces, as well as classical
frequency descriptors from the associated spectrogram. We identify the
recordings with the pronounced vowel, the individual pronouncing it, and their
gender. Thus, we can quantify whether topological descriptors provide
additional information to the more conventional frequency descriptors; whether
there is a difference in the topological information depending on the
representation space; whether there is a better way of vectorizing the
persistence diagrams. In order to visualize the extracted information in a
more qualitative way, we also learn the underlying manifold from the set of
all extracted topological features of each representation and visualize how
the signals are distributed over each manifold.

In what follows, we describe the recorded data and three associated
classification problems on which the test is carried out. Then, we present the
three representation spaces and associated filtration we use for each signal.
Finally, we present different procedures for vectorization of persistent
diagrams and the methodology we use to compare them.

###  IV-A Presentation of the data

The data are digital audio recordings of French-speaking adults. These
recordings were made in a controlled environment to keep low and stable
signal-to-noise ratio and to avoid junk sounds. We used a Zoom H6 recording
device with a stereophonic microphone XYH-6. The sampling rate of the
recordings is 44100 Hz, 16 bits. For the present analyses, we subsampled the
signals to 16kHz and convert them to mononophonic. We recorded 20 individuals,
15 women and 5 men. Each individual pronounced 8 vowels (ä, ~, , i, o, Õ, u,
y) in 7 conditions: natural, low voice, high voice, short, long, on an
ascending and descending scale. There were 10 utterances for each condition.
Therefore, each vowel has been recorded $1400$ times and each individual $560$
times. In total, $8400$ recordings were made by females and $2800$ by males.
The audio data set is freely accessible [46] and can be downloaded on
https://zenodo.org/record/7961904.

###  IV-B Comparison on three supervised classification problems

We consider the three following problems : prediction of the vowel, prediction
of the individual who pronounced the vowel, prediction of the gender of the
individual who pronounced the vowel. For each problem (and each representation
space), we use a random forest as a classification model with the same number
of $500$ trees. We report the OOB, which allows us to estimate the error by
cross-validation at a lower cost, for models using either topological
variables only, or frequency variables only, or both topological and frequency
variables (topologically-augmented).

###  IV-C Computation of the homological information

For each record, we compute the spectrogram with a Gaussian window of $11.6$
ms and an overlap of $90\%$. On the spectrogram's surface representation, we
apply sublevel set filtration to compute the persistent homologies while on
the spectrogram's zeroes, we use an alpha-complex filtration (see section
III).

For the representation using Taken's embeddings, we first estimate for each
record the two parameters required to build the embedding, $\tau$ according
the AMI and $D$ according to the Cao's algorithm. Then, we calculate the
embeddings of each record according to these two values. Finally, we harmonize
the dimension of embedding spaces over recordings by reducing the dimension to
3 via UMAP [47]. The records are represented by point clouds in
$\mathbb{R}^{3}$. On this space, we compute the persistent homologies using an
alpha-complex filtration.

At the end, each audio signal is associated to three persistent diagrams
carrying potentially different topological information.

We also compute the Mel-Frequency Cepstral Coefficients (MFCC) [48, 49]. Those
are more classical frequency descriptors of the signal, specifically designed
for human speech analysis and generally used in machine learning for
classification tasks involving human speech. They will serve as a baseline to
compare our topological descriptors. Finally, in a topologically-augmented
machine learning fashion, we will merge the MFCC with the topological
descriptors.

###  IV-D Extraction of the information from the persistence diagram and
comparison of the persistent variables

As discussed in part III-B, there are many ways to summarize the information
carried by persistence diagrams into variables that facilitate further
statistical/machine learning usage. We do not claim to be exhaustive in our
comparison. We follow various proposals from the literature to form a set of
variables computed on persistence diagrams, here so-called _persistent
variables_. In addition to persistent variables, we also compute functional
summaries. To be fair in comparison, we use the same classification model in
each case.

####  IV-D1 Persistent variables

Let $\mathcal{D}$ a persistence diagram, the multiset defined in equation (7).
We compute each variable for $p=\left\\{0,1\right\\}$, for the Taken's
embeddings we also compute for $p=2$.

From [38], we compute the persistent entropy $E_{p}$. Let
$L_{p}=\\{\ell_{i}=d_{i}-b_{i}|1\leq i\leq n\\}$ the set containing the
lifetime of each homological features of dimension $p$ of the diagram $D$. We
define the persistent entropy as

$E_{p}(F)=-\sum\limits_{i=1}^{n}p_{i}\log(p_{i}),$ (8)

where $p_{i}=\frac{\ell_{i}}{S_{L}}$ and
$S_{L}=\sum\limits_{i=1}^{n}\ell_{i}.$ We compute it for each dimension.

We follow [14] to compute several descriptors of the persistence diagram.
Using $L_{p}$, we compute some statistics from the vector:

  1. i)

the mean $\mu_{p}$.

  2. ii)

the variance $\sigma_{p}^{2}$.

  3. iii)

the 5 top longest lifetimes $L_{p,i}$ for $i=1,...,5$.

  4. iv)

two normalized longest lifetime,

$\frac{L_{p,1}}{|L_{p}|}$

.

  5. v)

and

$\frac{L_{p,1}}{\mu_{p}}.$

  6. vi)

the number of $\alpha$-long lifetime $N_{p,\alpha}$. We choose $\alpha$ to
distinguish between topological noise and information from the diagram. We
take $\alpha=0.05$.

$N_{p,\alpha}=\\#(L_{p,i}>\alpha).$

  7. vii)

the ratio of means

$\frac{\mu_{p_{1}}}{\mu_{p_{2}}}.$

$p_{1}$ and $p_{2}$ being two dimensions of homological features, with
$p_{1}<p_{2}$.

  8. viii)

the ratio of $\alpha$-long cycles

$\frac{N_{p_{1},\alpha}}{N_{p_{2},\alpha}}.$

  9. ix)

the products of top longest lifetimes

$L_{p_{1},i}\cdot L_{p_{2},i},$

for $i=1,...,6$.

  10. x)

the products

$L_{p_{1},i}\cdot(L_{p_{2},i}-L_{p_{2},i+1})$

for $i=1,...,6$.

  11. xi)

the Periodicity Score

$PS=1-\frac{L_{1,2}}{L_{1,1}}.$

  12. xii)

the Quasi-Periodicity Score

$QPS=L_{1,2}\cdot L_{2,1}.$

  13. xiii)

the Frequency Shift Score

$FSS=\frac{L_{2,1}\cdot L_{2,2}}{L_{1,1}}.$

We compute four more variables from [17]:

  * •

the persistent Betti number. For any pair of indices $0\leq k\leq l\leq n$ and
any dimension $p$, the $p^{th}$ persistent Betti number is

$\beta_{p}^{k,l}=\sum_{i\leq k}\sum_{j>l}\kappa_{p}^{i,j},$ (9)

where $\kappa_{p}^{i,j}$ is the number of $p$-dimensional homology that are
born at $X_{i}$ and die at $X_{j}$, $X_{i}$ and $X_{j}$ being two subsets of
the filtration. The persistent Betti number is the number of holes for each
dimension, and is called so in [17]. We follow [26] and call it the persistent
Betti number ;

  * •

the maximum hole lifetime in each dimension

$\text{max}_{p}=\max_{\ell_{i}\in L_{p}}(\ell_{i});$ (10)

  * •

the number of relevant holes

$\text{n\\_rel}_{p}=\sum_{\ell_{i}\in
L_{p}}f(\ell_{i},\max_{p},\text{ratio}),$ (11)

where $f(\ell_{i},\max_{p},\text{ratio})$ equals 1 if
$\ell_{i}\geq\max_{d}\times\text{ratio},$ 0 otherwise. It is the number of
points in the persistent diagram that are relatively distant from the
diagonal. We chose to count the holes with a lifetime at least greater than a
quarter of the longest (i.e., $\text{ratio}=0.25$) ;

  * •

the sum of all lifetime

$\text{sum}_{p}=\sum_{\ell_{i}\in L_{p}}(\ell_{i}).$ (12)

Following [37], we compute the $p$-norm of the persistence diagram,

$\lVert D\rVert_{p}=[\sum\limits_{u\in D}\text{pers}(u)^{p}]^{\frac{1}{p}},$
(13)

where $u$ is a point of the diagram and $\text{pers}(u)$ the absolute value of
the difference between the coordinates. In practice, we compute it for $p=2$.

The set of persistent variables is used alone in the different classification
problems, as well as in combination with the MFCCs. This data fusion method,
which we expect to be the most efficient according to the results from the
literature, can be found under the name "Topology augmented" in Table I, which
summarizes the results for the different problems.

####  IV-D2 Functional summary of the persistence diagrams

In addition to the set of persistent variables, we consider functional summary
of the persistent diagrams. After being discretized, they can use it as input
of a classification model. We present and test two of these methods.

First, we compute the silhouette of a persistence diagram [11]. It follows the
persistence landscapes [34, 35]. The silhouette summarizes the persistence
diagram in a single function. We choose this representation to illustrate the
mapping of the persistence diagram into a Hilbert space. We define it as

$\phi^{(\gamma)}(t)=\frac{\sum_{j=1}^{m}|d_{j}-b_{j}|^{\gamma}\lambda_{j}(t)}{\sum_{j=1}^{m}|d_{j}-b_{j}|^{\gamma}}.$
(14)

The silhouette takes a parameter $\gamma$. This determines whether all points
are treated equally ($\gamma$ small) or whether the most persistent pairs of
points are given more weight ($\gamma$ large). We set $\gamma=1$, because some
results show the importance of what is sometimes considered as "topological
noise" (e.g., [13]). We also have to fix the number of sample on which we
build the silhouette. We set $\text{nsample}=2^{9}=512$.

Next, we compute the persistence images of the diagrams [33]. The idea is to
map the persistence diagram to an integrable surface, so-called the
persistence surface

$\rho_{D}(u,v)=\sum_{(x,y)=(b,d-b)\in D}w(x,y)g_{(x,y)}(u,v),$

where $g_{(x,y)}(u,v):\mathbb{R}^{2}\rightarrow\mathbb{R}$ is bivariate
Gaussian distribution centered at each point $(x,y)=(b,d-b)\in D$ and
$w:\mathbb{R}^{2}\mapsto\mathbb{R}$ is a continuous and piecewise
differentiable weighting function. $(u,v)\in\mathcal{D}$ is a compact domain,
e.g., the domain of definition of the points $(x,y)=(b,d-b)\in D$. Then, we
divide the domain in a collection of non-overlapping subdomains, the pixels
$P_{i}$, with $\mathcal{D}=\bigcup P_{i}$. We integrate the persistence
surface over the fixed grid to define the persistence image, taking the
average of $\rho$ in each pixel,

$I_{P_{i}}(\rho_{D})=\int\int_{P_{i}}\rho_{D}(u,v)dudv.$ (15)

This outputs an image representing the persistence diagram, with a density
distribution that is more or less important depending on the distribution of
homologies on the persistence diagram. Again, we can use these vectors as
input to a classification model. A sampling parameter must be set, which is
fixed at 10.

###  IV-E Step-wise selection of the variables

As aforementioned in Section III-B, the question of how to extract information
from the persistence diagram for statistical purposes remains widely open and
is not to being addressed in this paper. We instead focus on the study of the
differences in persistent homologies between representation spaces of a given
signal, and we bring complementary information to the field by comparing the
most frequently occurring topological variables.

In order to carry out this comparison, we follow a step-wise strategy. For
each classification problem, we start with the complete set of persistent
variables. We estimate the model with this set of variables, and then we
remove the least important variable. We retrain the model with this new set of
variables. This process continues until we remove all the variables in the
training set. In the end, we retrain the model with the smallest OOB error. We
report in Table I the results of the best model, and we list the variables
present in the set of variables used to train this model. We count each time a
variable is in the set of the best model, for each problem and for each
representation.

This information can be found in the Table II. We can then compare whether
there is a difference according to the input representation, whether certain
topological variables carry more interesting information than others, or at
least whether they are more often found in the best model for the
classification tasks in question. Since our procedure includes MFCC, we
potentially remove some MFCC descriptors while keeping some topological
variables in models producing the best results.

There are three classification problems (prediction of the vowel, of the
gender, and of the individual), three initial signal representations
(spectrogram's surface, spectrogram's zeros, taken's embeddings). For each, we
follow our step-wise strategy to find the best model, eliminating iteratively
the less useful variable, with two conditions: with and without the MFCC. We
save the $18$ best models, and we count the remaining persistent variables in
each of them. Lastly, we follow the same procedure by including the
topological variables of all representations in the same model, for the three
problems. Thus, a variable can be counted in a maximum of 24 models.

We put in bold in Table II the variables that appear at least in 50% of the
best models. In the signal representation columns, each variable can appear at
best 12 times. Here again, we put in bold those that appear at least in 50% of
the best models. This way, it will be possible to compare whether certain
variables appear more than others globally, whatever the representation and
whatever the problem. We can also check if there are differences depending on
the representation space. For the sake of clarity, we resume the top 5 longest
lifetimes $L_{p,i}$ and their product $L_{p_{1},i}\cdot L_{p_{2},i+1}$ by a
row in Table II for each dimension. Also, the maximum count for these
variables is not 24, but $24\times 5=120$.

##  V Results

###  V-A Supervised problem

TABLE I: Comparison of Out Of Bag error (OOB) for different signal
representations

Signal Representation | Vectorizationa | OOB (%)
---|---|---
|  | Vowel | Gender | Individual
MFCC |  | 8.71 | 4.54 | 11.54
Spectrogram's Surface | Silhouettes $p=0$ | 72.97 | 20.37 | 73.72
| Silhouettes $p=1$ | 46.91 | 19.34 | 63.34
| Silhouettes $p=0,1$ | 45.04 | 16.44 | 53.86
| Persistent Image $p=0$ | 79.31 | 28.46 | 88.27
| Persistent Image $p=1$ | 79.26 | 28.73 | 87.97
| Persistent Image $p=0,1$ | 76.99 | 26.1 | 83.43
| Persistent Variables | 52.03 | 15.72 | 50.07
| Topology augmentedb | 8.43 | 4.71 | 10.42
Spectrogram's Zeros | Silhouettes $p=0$ | 79.89 | 23.69 | 78.21
| Silhouettes $p=1$ | 73.33 | 17.61 | 75.69
| Silhouettes $p=0,1$ | 70.11 | 16.14 | 67.5
| Persistent Image $p=0$ | 82.48 | 25.08 | 87.77
| Persistent Image $p=1$ | 70.5 | 16.79 | 71.15
| Persistent Image $p=0,1$ | 70.7 | 17.19 | 70.68
| Persistent Variables | 69.85 | 15.19 | 62.15
| Topology augmentedb | 8.03 | 5.55 | 9.24
Taken's embeddings | Silhouettes $p=0$ | 84.39 | 27.71 | 89.15
| Silhouettes $p=1$ | 81.51 | 25.32 | 84.44
| Silhouettes $p=2$ | 79.71 | 25.02 | 81.19
| Silhouettes $p=0,1$ | 78.03 | 24.59 | 80.63
| Silhouettes $p=0,2$ | 78.79 | 24.56 | 77.86
| Silhouettes $p=1,2$ | 78.56 | 24.68 | 77.22
| Silhouettes $p=0,1,2$ | 76.41 | 23.98 | 75.53
| Persistent Image $p=0$ | 85.11 | 29.63 | 90.24
| Persistent Image $p=1$ | 81.89 | 27.06 | 84.87
| Persistent Image $p=2$ | 83.5 | 27.15 | 87.42
| Persistent Image $p=0,1$ | 80.76 | 26.29 | 82.42
| Persistent Image $p=0,2$ | 81.54 | 25.94 | 82.93
| Persistent Image $p=1,2$ | 80.26 | 26.21 | 82.24
| Persistent Image $p=0,1,2$ | 79.53 | 25.76 | 79.86
| Persistent Variables | 69.85 | 20.27 | 60.89
| Topology augmentedb | 8.49 | 4.79 | 10.77
All together | Persistent Variables | 41.56 | 10.89 | 32.13
| Topology augmentedb | 7.98 | 6.01 | 10.3

  * a

Strategy to extract information from the persistence diagram

  * b

Signal representation = MFCC + persistent variables. We follow the step-wise
strategy presented in section IV-E on this set and present the result of the
best model.

The results for all supervised classification problems can be found in Table
I. All models are random forests with the same number of trees. For models
using as covariates either persistent variables or both persistent variables
and MFCCs, the reported results are those of the best model (i.e., the model
following the stepwise procedure, enjoying the lowest OOB).

####  V-A1 TDA is useful for signal classification

Topological information improves the results of two over three classification
problems. For vowel classification, the MFCCs alone obtain an OOB of 8.71%.
The addition of topological information improves the results, whatever the
chosen representation space. The OOB reduces to 8.43%, 8.03%, 8.49% using the
persistent variables obtained from spectrogram's surfaces, spectrogram's zeros
and Taken's embeddings, respectively. The best improvement is obtained by
taking all persistent variables from all representation spaces, resulting in
an OOB of 7.98%.

The individuals' classification, also benefits from topological information.
Indeed, MFCCs alone are outperformed when adding topological information,
whatever the representation space. The best results are obtained when the
persistent variables are extracted from the spectrogram's zeros, lowering the
OOB from 11.54% to 9.24%. On the contrary to vowel classification, taking all
persistent variables from all representation spaces deteriorates the results.

Finally, on the gender classification problem, topology augmented approach
fails to improve the results over the MFCCs alone, which exhibits the lowest
OOB at 4.54%.

####  V-A2 Topological information alone is not enough to discriminate the
signal

Topological information alone, whatever the chosen representation space, is
outperformed by classical frequency descriptors on each problem. For the vowel
classification problem, while the MFCCs alone achieve an OOB of 8.71%, the
best result using only topological variables is of about 41%. It is obtained
with persistent variables from all representations.

For the gender classification problem, using MFCCs alone give the best result,
with an OOB of 4.54%. Using topological information, the best models always
consider the persistent variables, with OOB ranging from 15.19% to 20.27%.
Aggregating the persistent variables of each representation improves the
results, with an OOB of 10.89%.

For the individual classification problem, it is once again the set of
persistence variables of all three representations, that obtains the best
results among models considering only topological information. It performs
poorly with an OOB of 32.13% while MFCCs alone reach OOB of 11.54%.

####  V-A3 Complementarity of representations

Topology augmented approaches improve the results over the MFCCs for both
vowel and individual classification. Comparing the topology augmented
approaches for the three representations on the vowel classification problem,
it is the variables extracted from the persistence diagrams of the
spectrogram's zeros that achieve the minimum OOB. For the individual
classification problem, the variables extracted from the persistence diagrams
of the zeros of the spectrograms also provide the best score for all the
approaches compared for this problem.

While the topology augmented approach with the addition of the persistent
variables of the spectrogram's zeros provides the best improvement for both
problems, there does not appear to be one better representation of the signal
or one with a more informative topology than the others. The differences are
held and, for the gender classification problem, the topology augmented models
with the persistence diagrams of the spectrogram's surfaces and Taken's
embeddings outperform the spectrogram's zeros. For the vowel classification
problem, the best model is the one taking the full set of persistent variables
in addition to the MFCCs, those computed on all representations.

The resulting 'data objects' have different topology, which would be
potentially complementary. Indeed, we note that, for each problem, the model
learned with the set of all persistent variables, computed on the three
representations, and without the MFCCs, performs better than each model
learned with the set of persistent variables computed on each representation.
Thus, for vowel classification, while the persistent variables of the
spectrogram's surfaces have an OOB of 52.03%, those of the spectrogram's zeros
69.85% and those of the Taken's embeddings 69.85%, the model trained on the
set of these persistent variables reaches an OOB of 41.56%. Moreover, by
adding the MFCCs, it is this set that gives the best results for this problem.
For gender classification, the persistent variables computed on the
spectrogram's surfaces have an OOB of 15.72%, those on the spectrogram zeros
15.19% and those on the Taken's embeddings 20.27%. All together, the model
still obtains a clear improvement, with an OOB of 10.89%. For classification
of the individuals, the persistent variables computed on the surfaces of the
spectrogram have an OOB of 50.07%, those of the zeros of the spectrogram
62.15%, those of the Taken's embdedings 60.89%. The model learned on the whole
of these variables obtains an OOB of 32.13%. Thus, we improve each time the
results by merging with all the persistent variables.

###  V-B Best topological variables

TABLE II: Most frequently kept persistent variables in each best model

Variable |  Frequency in the best model (%)a | Signal Representation
---|---|---
|  |  Spectrogram's Surface |  Spectrogram's Zeros |  Taken's embeddings
$\beta_{0}$ (9) |  62.5 |  33.33 |  41.67 |  50
$\beta_{1}$ |  33.33 |  50 |  0 |  16.67
$\beta_{2}$ |  25 |  $\emptyset$ |  $\emptyset$ |  50
$\max_{0}$ (10) |  29.17 |  58.33 |  0 |  0
$\max_{1}$ |  0 |  0 |  0 |  0
$\max_{2}$ |  0 |  $\emptyset$ |  $\emptyset$ |  0
$\text{n\\_rel}_{0}$ (11) |  0 |  0 |  0 |  0
$\text{n\\_rel}_{1}$ |  4.17 |  0 |  1 |  0
$\text{n\\_rel}_{2}$ |  0 |  $\emptyset$ |  $\emptyset$ |  0
$\text{sum}_{0}$ (12) |  41.67 |  66.67 |  0 |  2
$\text{sum}_{1}$ |  50 |  5 |  2 |  5
$\text{sum}_{2}$ |  8.33 |  $\emptyset$ |  $\emptyset$ |  16.67
$E_{0}$ (8) |  29.17 |  50 |  0 |  8.33
$E_{1}$ |  33.33 |  50 |  0 |  16.67
$E_{2}$ |  4.17 |  $\emptyset$ |  $\emptyset$ |  8.33
$\lVert D_{0}\rVert_{p}$ (13) |  37.5 |  50 |  16.67 |  8.33
$\lVert D_{1}\rVert_{p}$ |  50 |  25 |  50 |  25
$\lVert D_{2}\rVert_{p}$ |  0 |  $\emptyset$ |  $\emptyset$ |  0
$\mu_{0}$ (i) |  87.5 |  83.33 |  75 |  16.67
$\mu_{1}$ |  79.17 |  66.67 |  50 |  41.67
$\mu_{2}$ |  4.17 |  $\emptyset$ |  $\emptyset$ |  8.33
$\sigma^{2}_{0}$ (ii) |  45.83 |  33.33 |  41.67 |  16.67
$\sigma^{2}_{1}$ |  66.67 |  25 |  100 |  8.33
$\sigma^{2}_{2}$ |  0 |  $\emptyset$ |  $\emptyset$ |  0
$\frac{L_{0,1}}{|L_{0}|}$ (iv) |  25 |  33.33 |  0 |  16.67
$\frac{L_{1,1}}{|L_{1}|}$ |  33.33 |  58.33 |  0 |  8.33
$\frac{L_{2,1}}{|L_{2}|}$ |  0 |  $\emptyset$ |  $\emptyset$ |  0
$\frac{L_{0,1}}{\mu_{0}}$ v) |  54.17 |  41.67 |  41.67 |  25
$\frac{L_{1,1}}{\mu_{1}}$ |  37.5 |  50 |  8.33 |  16.67
$\frac{L_{2,1}}{\mu_{2}}$ |  0 |  $\emptyset$ |  $\emptyset$ |  0
$N_{0,\alpha}$ (vi) |  29.17 |  0 |  58.33 |  0
$N_{1,\alpha}$ |  25 |  0 |  8.33 |  41.67
$N_{2,\alpha}$ |  0 |  $\emptyset$ |  $\emptyset$ |  0
$\frac{\mu_{0}}{\mu_{1}}$ (vii) |  87.5 |  100 |  33.33 |  41.67
$\frac{\mu_{0}}{\mu_{2}}$ |  12.5 |  $\emptyset$ |  $\emptyset$ |  25
$\frac{\mu_{1}}{\mu_{2}}$ |  4.17 |  $\emptyset$ |  $\emptyset$ |  8.33
$\frac{N_{0,\alpha}}{N_{1,\alpha}}$ viii) |  20.84 |  0 |  33.33 |  8.33
$\frac{N_{0,\alpha}}{N_{2,\alpha}}$ |  0 |  $\emptyset$ |  $\emptyset$ |  0
$\frac{N_{1,\alpha}}{N_{2,\alpha}}$ |  4.17 |  $\emptyset$ |  $\emptyset$ |  8.33
$PS$ (xi) |  12.5 |  25 |  0 |  0
$QPS$ (xii) |  0 |  $\emptyset$ |  $\emptyset$ |  0
$FSS$ (xiii) |  0 |  $\emptyset$ |  $\emptyset$ |  0
$L_{0,i}{\textsuperscript{b}}$ (iii) |  21.67 |  36.67 |  5 |  1.67
$L_{1,i}{\textsuperscript{b}}$ |  50.83 |  43.33 |  51.67 |  6.67
$L_{2,i}{\textsuperscript{b}}$ |  5 |  $\emptyset$ |  $\emptyset$ |  10
$L_{0,i}\cdot L_{1,i}{\textsuperscript{b}}$ (ix) |  27.5 |  31.67 |  15 |  8.33
$L_{0,i}\cdot L_{2,i}{\textsuperscript{b}}$ |  0 |  $\emptyset$ |  $\emptyset$ |  0
$L_{1,i}\cdot L_{2,i}{\textsuperscript{b}}$ |  2.5 |  $\emptyset$ |  $\emptyset$ |  5
$L_{0,i}\cdot(L_{1,i}-L_{0,i+1}){\textsuperscript{b}}$ (x) |  5 |  5 |  3.33 |  1.67
$L_{0,i}\cdot(L_{2,i}-L_{0,i+1}){\textsuperscript{b}}$ |  0.3 |  $\emptyset$ |  $\emptyset$ |  1.67
$L_{1,i}\cdot(L_{2,i}-L_{1,i+1}){\textsuperscript{b}}$ |  0 |  $\emptyset$ |  $\emptyset$ |  0

  * a

We follow the step-wise strategy to choose the best model for each
classification problem. we count in how many models each variable appears.

  * b

We count if one of them is kept, $\forall i\in\\{1,...,6\\}$

For this classification problems, persistent silhouettes or persistent images
perform worse than the persistent variables in all scenarios. For this reason,
we build the topologically augmented approach only with the persistent
variables. They are therefore not taken into account in the stepwise procedure
to identify the best topological variable (see Section IV-E).. All the results
are presented in Table II.

In general, there are 8 persistent variables that we encounter at least in 50%
of the best models. These are the persistent Betti number in dimension $p=0,$
the sum of $L_{p}$ in dimension $p=1,$ the 2-norm of the persistence diagram
for $p=1,$ the average of $L_{p}$ for $p=0$ and $p=1$ and its variance for
$p=1$, the normalized longest lifetime for $p=0,$ with the normalization
computed with the mean of $L_{p}$ of this dimension, the ratio of the mean of
$L_{0}$ and $L_{1}$.

The number of variables retained in 50% of the best models varies
significantly from one representation space to another. We identified $11,\
5,\ 2$ persistent variables retained for the spectrogram's surface, for the
spectrogram's zeros and for the Taken's embedding, respectively.

There are two persistent variables that seem to stand out the most: the
persistent Betti number and the norm of the persistence diagram. The other
persistent variables which appear regularly are linked and summarized in the
vector $L_{p}$. We find statistics on this vector that summarize the
information of the persistence diagrams (mean, variance, normalized maximum),
whatever the initial representation of the signal.

###  V-C Unsupervised analysis, learning the manifold

(a) MFCC Vowel

(b) MFCC Sex

(c) MFCC subject

(d) Surface Vowel

(e) Surface Sex

(f) Surface Subject

(g) Zeros Vowel

(h) Zeros Sex

(i) Zeros Subject

(j) Takens Vowel

(k) Takens Sex

(l) Takens Subject

Figure 3: Projection of the records on the 2d space learned with UMAP,
depending on the input of the algorithm: MFCC, topological variables from the
spectrogram's surface, the spectrogram's zeros or the Taken's embedding.

In order to describe better the behavior of the homological features
associated to the different representation spaces, we provide a visual and
qualitative help using the Uniform Manifold Approximation and Projection for
Dimension Reduction (UMAP) algorithm. Figure 3 represents the projection of
the recordings onto the 2d space learned with the UMAP. It distinguishes MFCC,
persistent variables accordingly to the representation space, and the three
problems considered therein.

Figure 3 reveals that the manifold learned on the MFCCs creates clear clusters
for vowels. This is less the case for gender, where observations from both
classes are quite close in this space. Same comment applies for individuals.
Although some observations are centered, the distribution of individuals in
this space can be quite spread out. It is even harder to see a pattern for the
manifold learned on persistent variables, whatever the representation space.
In fact, the manifolds learned from the persistent variables are much more
connected, and we do not distinguish clear clusters in these spaces.

Interestingly, the overall shapes of manifolds learned on Taken's embeddings
and on spectrogram's surface look quite similar, and rotated by 180°. While
connectivity is still present, manifolds learned on topological variables
extracted from zero's spectrogram looks very different. This might be
explained by the number of coordinates in the representation space within the
persistent homology. Taken's embedding and the spectrogram's surface lie in a
3-dimensional space, whereas the spectrogram's zeros are in the plane.

##  VI Discussion

The main results that emerge from the present study are fourfold. Firstly,
topological information improved some classification results. Secondly, the
choice of the signal representation space has an impact on the topological
information that is extracted. Thirdly, the best way to use a persistence
diagram for a classification task depends on the problem. Finally, topological
information extracted from the spectrogram's zeroes appeared to be
particularly complementary to the MFCC and provided the best results out of
all tested models for individual prediction.

###  VI-A On the improvements on the prediction of labels

On the one hand, no matter how the information is extracted from the
persistence diagrams, topological information never outperformed the MFCCs. On
the other hand, adding topological variables to frequency variables improves
the results in two of the three considered classification problems. We thus
confirmed results already described in the literature [40, 20, 3]. Topological
descriptors of signals bring complementary information to MFCCs, which are
specifically designed for human speech analysis.

These results contribute to fill an existing gap in the analysis of time-
varying data, as noted in some recent reviews [32]. They are encouraging
enough to continue investigating the performance of topologically-augmented
machine learning approaches even for natural signal suffering from much lower
signal-to-noise ratio as one might consider robust filtration alike Distance-
To-Measure introduced in [50].

Our results show no improvement for the gender classification problem. Remark
that it is the simplest problem as it is binary classification, while the
other have 8 and 20 classes, for vowels and individuals prediction,
respectively. Another possible reason is the pitch difference between men and
women. Topological characteristics are certainly invariant to pitch
modulations, and MFCC's particularly well suited.

When the problem becomes more complicated, topological information proved to
be useful. The most notable improvement is for the prediction of the speaker.
Persistent homologies seem to carry this additional information needed to
improve the classification performance.

###  VI-B Different objects, different topologies

####  VI-B1 The difference between representation spaces

As already noticed, it is difficult to identify one representation of the
signal as being better than another. Nevertheless, there are two points to
bear in mind from the classification results according to the initial
representation space:

  * •

For the two problems where topological information improves the results, i.e.,
individual and vowel classification, the persistent homologies extracted from
the time-frequency plane of the spectrogram zeros improve the OOB error the
most. Taken's embeddings are the representation with the least improvement for
both problems. Thus, access to higher dimensional homologies does not seem to
provide discriminative information about the signal.

  * •

The aggregation of topological information is also advantageous. It allows the
best improvement in the prediction of vowels and, when it does not allow the
best improvement in the prediction of individuals, the improvement is better
than for the other two representations. Moreover, for the models trained only
on the persistent variables, without the MFCCs, the best results are always
obtained when all the representations are combined. The homological
information of the representation spaces therefore seems complementary.

This shows that a particular representation space, reveal different salient
topological features. Very often, Taken's embeddings are considered in
topological signal analysis. In this paper, we introduced two ways to derive
topological information from spectrogram that have the advantage of being more
interpretable.

####  VI-B2 A different perspective

The topological signatures we computed on the different signal representation
spaces are complementary to more classical descriptors such as MFCCs. This has
been highlighted in supervised classification problems [40, 20, 3].

The manifolds we learn from the persistent variables, presented in Figure 3,
are strongly connected. The topological approach is metric-free and based on
the connectivity [1]. A topological latent space does not clearly cluster the
data.

We find a similar observation in the topological autoencoder [51]. This
property makes entangled structures appear that are impossible in a clear
spatial separation. This would be useful to illustrate a hierarchy in the
data, with a parent-child structure.

The value of the topological approach depends on the problem [2] and is
particularly suited to problems requiring analysis of nested categories. If
one expects such a structure in the data, and we wish to highlight it, it is
interesting to examine its topology, in order to look at the problem from a
blind spot to more classical analysis tools.

###  VI-C On the more present persistent variables

Finally, regarding the vectorization of persistence diagrams, it seems
difficult to characterize topological signatures, at least among those tested.
We confirm a result already present in other reviews studying this question
[31], which find that the most interesting topological signatures seem to
depend on the studied data set. Here we found that mapping the persistence
diagram onto a functional space, such as [11, 33], was less efficient than
taking a set of scalars that summarize the information contained in the
diagram. Nevertheless, these approaches have the advantage of having well
established stability properties and have been used elsewhere [35, 15, 36].
This does not disqualify them for other problems, and they might even be
effective on this problem, if handled more carefully (for example, by tuning
finely the different parameters they depend on, or by computing well-chosen
summary data).

This question also raises the issue of what is considered to be topological
noise. Topological noise generally designates homological features with very
short lifespans. It can be seen on persistent diagrams as the set of points
along the main diagonal. We might ask ourselves, is topological noise really
noise? Indeed, it has already been noticed that topological noise can be
useful to characterize a signal [13]. The persistence diagrams obtained after
alpha-complex filtration on the zeros of the spectrograms are visually the
diagrams with the most persistent homologies, many of them close to the
diagonal, as can be seen in Figure 2b. Yet, it is this representation that
gives the best improvements. Perhaps these homologies are not just noise and
contain information. It is important to bear this in mind when choosing a way
to use the information contained in persistence diagrams, as not all methods
treat elements close to the diagonal in the same way.

Finally, we did not test all the existing approaches to this question. In
fact, we adopted a strategy consisting of extracting information from the
persistence diagrams, either by mapping persistence diagrams onto function
space (persistent landscape lives in Banach space) or by calculating vectors
to describe them.

##  VII Conclusion

We discussed the potential added value of the topological approach to sound
signal processing by studying the differences according to the representation
space of the signal. We tested it on three classification problems, predicting
the gender of the speaker, the pronounced vowel and the identity of the
speaker. For two of these problems, vowel and identity prediction, the
topological features improve the results compared to the baseline. Although it
is difficult to distinguish one representation space as being more informative
than another, it seems that the topological descriptors computed on each of
them are complementary. Our results suggest the use of less common
representations than Taken's embeddings, such as spectrogram's surfaces or
spectrogram's zeros. For individual classification, the zeros give the best
results with the topologically-augmented approach. Moreover, parameter
selection and interpretation of a spectrogram is simpler than Taken's
embeddings. We analyzed different ways of vectorizing information from
persistence diagrams. The best results were obtained using a set of persistent
variables regardless of the representation space. This shows that the
topological approach offers a complementary and interesting angle of analysis.
It is not sufficient on its own to discriminate the signal, but it does
provide additional information about its hierarchical structure. It would be
interesting to analyze more theoretically the complementarity of the
representation spaces of the signal, and to see the possibility of working
directly in the space of persistence diagrams.

In the end, we believe that TDA has a lot to offer to sound signal processing,
in particular for individual recognition problems.

## References

  * [1] G. Carlsson, ``Topology and data,'' Bulletin of the American Mathematical Society, vol. 46, no. 2, pp. 255–308, 2009.
  * [2] L. Wasserman, ``Topological Data Analysis,'' Annual Review of Statistics and Its Application, vol. 5, no. 1, pp. 501–532, 2018.
  * [3] F. Chazal and B. Michel, ``An Introduction to Topological Data Analysis: Fundamental and Practical Aspects for Data Scientists,'' Frontiers in Artificial Intelligence, vol. 4, 2021.
  * [4] S. Barbarossa and S. Sardellitti, ``Topological Signal Processing Over Simplicial Complexes,'' IEEE Transactions on Signal Processing, vol. 68, pp. 2992–3007, 2020.
  * [5] S. Barbarossa and S. Sardellitti, ``Topological Signal Processing: Making Sense of Data Building on Multiway Relations,'' IEEE Signal Processing Magazine, vol. 37, pp. 174–183, Nov. 2020.
  * [6] M. Ferri, ``Why Topology for Machine Learning and Knowledge Extraction?,'' Machine Learning and Knowledge Extraction, vol. 1, pp. 115–120, May 2018.
  * [7] A. J. Zomorodian, Topology for Computing.  Cambridge University Press, 1 ed., Jan. 2005.
  * [8] N. Otter, M. A. Porter, U. Tillmann, P. Grindrod, and H. A. Harrington, ``A roadmap for the computation of persistent homology,'' EPJ Data Science, vol. 6, p. 17, Dec. 2017.
  * [9] J.-D. Boissonnat, F. Chazal, and B. Michel, ``Topological Data Analysis,'' in Novel Mathematics Inspired by Industrial Challenges (M. Günther and W. Schilders, eds.), vol. 38, pp. 247–269, Cham: Springer International Publishing, 2022.
  * [10] D. Cohen-Steiner, H. Edelsbrunner, and J. Harer, ``Stability of Persistence Diagrams,'' Discrete & Computational Geometry, vol. 37, pp. 103–120, Jan. 2007.
  * [11] F. Chazal, B. T. Fasy, F. Lecci, A. Rinaldo, and L. Wasserman, ``Stochastic Convergence of Persistence Landscapes and Silhouettes,'' in Proceedings of the Thirtieth Annual Symposium on Computational Geometry, (Kyoto Japan), pp. 474–483, ACM, June 2014.
  * [12] L. M. Seversky, S. Davis, and M. Berger, ``On Time-Series Topological Data Analysis: New Data and Opportunities,'' in 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), (Las Vegas, NV, USA), pp. 1014–1022, IEEE, June 2016.
  * [13] V. Patrangenaru, P. Bubenik, R. L. Paige, and D. Osborne, ``Challenges in Topological Object Data Analysis,'' Sankhya A, vol. 81, pp. 244–271, Feb. 2019.
  * [14] T. Fireaizen, S. Ron, and O. Bobrowski, ``Alarm Sound Detection Using Topological Signal Processing,'' in ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), (Singapore, Singapore), pp. 211–215, IEEE, May 2022.
  * [15] J.-Y. Liu, S.-K. Jeng, and Y.-H. Yang, ``Applying Topological Persistence in Convolutional Neural Network for Music Audio Signals,'' arXiv:1608.07373 [cs], Aug. 2016.
  * [16] M. G. Bergomi and A. Baratè, ``Homological persistence in time series: An application to music classification,'' Journal of Mathematics and Music, vol. 14, pp. 204–221, May 2020.
  * [17] C. M. Pereira and R. F. de Mello, ``Persistent homology for time series and spatial data clustering,'' Expert Systems with Applications, vol. 42, pp. 6026–6038, Sept. 2015.
  * [18] A. Salch, A. Regalski, H. Abdallah, R. Suryadevara, M. J. Catanzaro, and V. A. Diwadkar, ``From mathematics to medicine: A practical primer on topological data analysis (TDA) and the development of related analytic tools for the functional discovery of latent structure in fMRI data,'' PLOS ONE, vol. 16, p. e0255859, Aug. 2021.
  * [19] F. Nasrin, C. Oballe, D. L. Boothe, and V. Maroulas, ``Bayesian Topological Learning for Brain State Classification,'' arXiv:1912.08348 [cs, stat], Dec. 2019.
  * [20] X. Xu, N. Drougard, and R. N. Roy, ``Topological Data Analysis as a New Tool for EEG Processing,'' Frontiers in Neuroscience, vol. 15, p. 761703, Nov. 2021.
  * [21] P. Flandrin, Explorations in Time-Frequency Analysis.  Cambridge University Press, first ed., Sept. 2018.
  * [22] J. Levy, A. Naitsat, and Y. Y. Zeevi, ``Classification of audio signals using spectrogram surfaces and extrinsic distortion measures,'' EURASIP Journal on Advances in Signal Processing, vol. 10, pp. 83–103, 2022.
  * [23] P. Flandrin, ``Time–Frequency Filtering Based on Spectrogram Zeros,'' IEEE Signal Processing Letters, vol. 22, pp. 2137–2141, Nov. 2015.
  * [24] F. Takens, ``Detecting strange attractors in turbulence,'' in Dynamical Systems and Turbulence, Warwick 1980 (D. Rand and L.-S. Young, eds.), vol. 898, pp. 366–381, Berlin, Heidelberg: Springer Berlin Heidelberg, 1981.
  * [25] L. Cao, ``Practical method for determining the minimum embedding dimension of a scalar time series,'' Physica D: Nonlinear Phenomena, vol. 110, pp. 43–50, Dec. 1997.
  * [26] H. Edelsbrunner and J. Harer, Computational Topology: An Introduction.  AMS Press, 2009.
  * [27] T. K. Dey and Y. Wang, Computational Topology for Data Analysis.  New York: Cambridge University Press, first edition ed., 2022.
  * [28] B. T. Fasy, F. Lecci, A. Rinaldo, L. Wasserman, S. Balakrishnan, and A. Singh, ``Confidence sets for persistence diagrams,'' The Annals of Statistics, vol. 42, Dec. 2014.
  * [29] V. Maroulas, F. Nasrin, and C. Oballe, ``A Bayesian Framework for Persistent Homology,'' SIAM Journal on Mathematics of Data Science, vol. 2, pp. 48–74, Jan. 2020.
  * [30] V. Divol and T. Lacombe, ``Understanding the topology and the geometry of the space of persistence diagrams via optimal partial transport,'' Journal of Applied and Computational Topology, no. 25, pp. 1–53.
  * [31] D. Barnes, L. Polanco, and J. A. Perea, ``A Comparative Study of Machine Learning Methods for Persistence Diagrams,'' Frontiers in Artificial Intelligence, vol. 4, p. 681174, July 2021.
  * [32] F. Hensel, M. Moor, and B. Rieck, ``A Survey of Topological Machine Learning Methods,'' Frontiers in Artificial Intelligence, vol. 4, p. 681108, May 2021.
  * [33] H. Adams, T. Emerson, M. Kirby, R. Neville, C. Peterson, P. Shipman, S. Chepushtanova, E. Hanson, F. Motta, and L. Ziegelmeier, ``Persistence Images: A Stable Vector Representation of Persistent Homology,'' The Journal of Machine Learning Research, vol. 18, p. 8830, Jan. 2017.
  * [34] P. Bubenik, ``Statistical Topological Data Analysis using Persistence Landscapes,'' The Journal of Machine Learning Research, vol. 16, no. 1, p. 26, 2015.
  * [35] P. Bubenik, ``The Persistence Landscape and Some of Its Properties,'' in Topological Data Analysis (N. A. Baas, G. E. Carlsson, G. Quick, M. Szymik, and M. Thaule, eds.), vol. 15, pp. 97–117, Cham: Springer International Publishing, 2020.
  * [36] K. Kim, M. Zaheer, F. Chazal, J. Kim, J. S. Kim, and L. Wasserman, ``PLLay: Efficient Topological Layer based on Persistence Landscapes,'' in NeurIPS 2020 - 34th Conference on Neural Information Processing Systems, (Vancouver / Virtuel, Canada), p. 13, 2020.
  * [37] D. Cohen-Steiner, H. Edelsbrunner, J. Harer, and Y. Mileyko, ``Lipschitz Functions Have L p -Stable Persistence,'' Foundations of Computational Mathematics, vol. 10, pp. 127–139, Apr. 2010.
  * [38] N. Atienza, R. Gonzalez-Diaz, and M. Rucco, ``Persistent entropy for separating topological features from noise in vietoris-rips complexes,'' Journal of Intelligent Information Systems, vol. 52, pp. 637–655, June 2019.
  * [39] N. Atienza, R. Gonzalez-Díaz, and M. Soriano-Trigueros, ``On the stability of persistent entropy and new summary functions for topological data analysis,'' Pattern Recognition, vol. 107, p. 107509, Nov. 2020.
  * [40] M. Carrière, S. Y. Oudot, and M. Ovsjanikov, ``Stable Topological Signatures for Points on 3D Shapes,'' Computer Graphics Forum, vol. 34, pp. 1–12, Aug. 2015.
  * [41] Y. Korkmaz, A. Boyacı, and T. Tuncer, ``Turkish vowel classification based on acoustical and decompositional features optimized by Genetic Algorithm,'' Applied Acoustics, vol. 154, pp. 28–35, Nov. 2019.
  * [42] G. P. Georgiou, ``Comparison of the prediction accuracy of machine learning algorithms in crosslinguistic vowel classification,'' Scientific Reports, vol. 13, p. 15594, Sept. 2023.
  * [43] S. Deb and S. Dandapat, ``Emotion Classification Using Segmentation of Vowel-Like and Non-Vowel-Like Regions,'' IEEE Transactions on Affective Computing, vol. 10, pp. 360–373, July 2019.
  * [44] J. Vavrina, P. Zetocha, and J. Tuckova, ``Detection of degree of developmental dysphasia based on methods of vowel analysis,'' in 2012 35th International Conference on Telecommunications and Signal Processing (TSP), pp. 503–507, July 2012.
  * [45] M. Vashkevich and Yu. Rushkevich, ``Classification of ALS patients based on acoustic analysis of sustained vowel phonations,'' Biomedical Signal Processing and Control, vol. 65, p. 102350, Mar. 2021.
  * [46] G. Bonafos, J.-M. Freyermuth, P. Pudlo, S. Tronçon, and A. Rey, ``French vowels,'' May 2023.
  * [47] L. McInnes, J. Healy, and J. Melville, ``UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction,'' Sept. 2020.
  * [48] S. Chachada and C.-C. J. Kuo, ``Environmental sound recognition: a survey,'' APSIPA Transactions on Signal and Information Processing, vol. 3, p. e14, 2014.
  * [49] J. Sueur, Sound Analysis and Synthesis with R.  New York, NY: Springer Berlin Heidelberg, 1st edition ed., 2018.
  * [50] F. Chazal, B. Fasy, F. Lecci, B. Michel, A. Rinaldo, and L. Wasserman, ``Robust topological inference: Distance to a measure and kernel distance,'' Journal of Machine Learning Research, vol. 18, 12 2014.
  * [51] M. Moor, M. Horn, B. Rieck, and K. Borgwardt, ``Topological Autoencoders,'' in arXiv:1906.00722 [Cs, Math, Stat], vol. PMLR 119, pp. 7045–7054, 2020.