# Detachment Problem – Application in Prevention of Information Leakage in
Stock Markets

Henri Hansen  Tampere University  Juho Kanniainen

###### Abstract

In this paper, we introduce Detachment Problem within the context of the well-
established independent cascade model. It can be seen as a generalized
Vaccination Problem, where we acknowledge that an individual may belong to
multiple circles. The aim is to optimally cut the individuals’ ties to these
circles to minimize the overall information transfer in a social network under
a scenario where a limited number of investor-circle pairs can be operated.
When an individual is isolated from a particular circle, it leads to the
eliminating of the connections to all the members of that circle, yet the
connections to other circles remain. This approach contrasts with the
conventional vaccination problem, which generally involves selecting a subset
of vertices for total elimination. In our case, the connections of individuals
to the circles are selectively, rather than entirely, eliminated.

Contextually, this article focuses on private information flows, specifically
within networks formed by memberships in circles of insiders. Our quasi-
empirical study uses simulated information flows on a truly observable
network, and the statistical properties of the simulated information flows are
matched with real-world data. In a broader context, this paper presents the
Detachment Problem as a versatile approach for optimal social distancing,
applicable across various scenarios.

We propose and define a concept of expected proportional outside influence, or
EPOI, as measure of how widespread information leak is. We also implement a
greedy algorithm for finding a set of detachments to minimize EPOI. For
comparison, we devise a simple heuristic based on minimal cut, to separate the
most influential circles from each other. We provide evidence that the greedy
algorithm is not optimal, and it is sometimes outperformed by the simple
heuristic minimum cut algorithm, However, the greedy algorithm outperforms the
cut algorithm in most cases. Further avenues of research are discussed.

###### keywords:

Detachment problem , Vaccination problem , Immunization problem , Influential
spreaders , Influence maximization , Insider information , Independent cascade
, Financial markets

††journal: the Journal of Pseudosciences

##  1 Introduction

The propagation of ideas, fake news, influence, and diseases through social
networks has been examined from numerous angles, encompassing fields such as
technology adoption, the spread of rumors, viral marketing initiatives, and
epidemiology, among others (Kempe et al., 2003; Nowzari et al., 2016; Li et
al., 2018; Sharma et al., 2019). This paper defines and introduces a new
problem, which we term the Detachment Problem.

In the Detachment Problem, an individual is connected to one or more circles.
A circle is simply a set which connects all its members in some way. In our
data this is the set a board of directors and high level managers of a
company, i.e., the set of insiders, but it can be participants in a given
meeting, a school class, etc in other applications. The objective is to
optimally select a given number of individual-circle pairs and remove the
selected individuals from the selected circles, to minimize the information
transfer in a social network. When a vertex is isolated from a given circle,
it leads to the eliminating its connections to all the members of that circle.
However, the connections to other circles remain. We address the detachment
problem assuming the dynamics of information spread follow the well-
established cascade model, known as the _independent cascade model_
(Goldenberg et al., 2001; Kempe et al., 2003).

The Detachment Problem can be seen as a generalized case of the _vaccination_
or _immunization problem_ , i.e., selecting an individual and either immunize
or, in the case of the detachment problem, partially isolate, from a given
social network (for vaccination and immunuzation problems, see, for example,
Pastor-Satorras and Vespignani, 2002; Gallos et al., 2007; Prieto Curiel and
González Ramírez, 2021). While conventional vaccination approaches focus on
immunizing a subset of vertices (individuals), the Detachment Problem differs
as it does not entail the complete removal of a vertex. Instead, it involves
cutting off its connections to specific circles. Additionally, this problem
bears a strong resemblance to the task of identifying influential spreaders or
nodes within a network, or influence maximization, as discussed in various
studies (Kitsak et al., 2010; Chen et al., 2012; Morone and Makse, 2015; Lü et
al., 2016). However, the distinctive aspect of the Detachment Problem lies not
in identifying and removing a vertex (individual) but in eliminating its ties
with certain circles to which it belongs.

The framework for the Detachment Problem outlined in the paper is highly
generic, with a primary emphasis on methodology. This problem is applicable in
a variety of contexts, ranging from the regulation of financial markets to the
containment of infectious diseases. To concretize the usefulness of the
framework, we focus on private information flows, in particular, in networks
that emerge from memberships in circles of insiders. The go-to model we use is
the well-known _independent cascade_ model (IC), which captures a simplified
susceptible-infected-recovered (SIR) model in a network of individuals. The
problem of _influence maximization_ is that of finding a subset of vertices to
”infect” in order to maximize the diffusion of information, and it has
received significant attention in the literature. From the perspective of the
application, this article addresses the question of preventing the spread of
illicit information, which might lead to, for example, insider trading.

We study the problem by modeling a particular network of insiders, i.e., board
members and key executives of companies listed in the Helsinki Stock exchange
in the year 2009. These persons form a social network, and their known
connections come from being members of the same inner circle in a company. As
many key personnel hold key positions in multiple companies, this network
represents a plausible medium through which inside information might spread
(Ahern, 2017; Berkman et al., 2020; Baltakienė et al., 2022; Baltakys et al.,
2023). Publicly traded companies are obligated to announce important
information publicly, but the insiders of these companies gain access to such
information before the information is made public. The use of such information
to make profitable trades in the stock market before the information is
publicized is known as _insider trading_ , and it is illegal in most
jurisdictions. Insider trading is estimated by various sources to result in
massive illicit gains and as such create a pernicious and hard-to-detect cost
in the economy, which makes our research question a pertinent one. What is
more, our model and results point to regulatory recommendations and allow us
to quantify the effect of the implementation of said recommendations.

##  2 Model and Problem formulation

###  2.1 Induced network

######  Definition 1.

Let $V$ be a set of vertices, and $\mathcal{I}=\\{I_{1},I_{2},\ldots I_{k}\\}$
where $I_{i}\subseteq V$ be a collection of sets of vertices called _circles_.
The _network induced by_ $\mathcal{I}$ is the graph $(V,E)$ where

$V=\bigcup_{I_{i}\in\mathcal{I}}I_{i},$ $E=\\{(u,v)\mid\exists i:u\in
I_{i}\wedge v\in I_{i}\\}.$

In other words, there is an edge between two vertices if there is a circle in
which both belong.

We associate a weight $w_{u,v}\in[0,1]$ for each $(u,v)\in E$, to denote a
probability that $u$ passes information forward to $v$. Then we call $(V,E,w)$
an _information flow network_.

###  2.2 The Independent Cascade Model

The so-called Independent cascade model (Goldenberg et al., 2001; Kempe et
al., 2003) describes a simple process by which information starting from an
initial set of vertices spreads through such a network. It is a variant of the
Susceptible - Infected - Recovered, or SIR-model, in which each vertex is
infected at most once in some time step, will attempt to infect all of its
neighbours in the next, and will be recovered and totally inert in all the
subsequent time steps.

Let $A_{0}\subseteq V$. We define a _cascade from $A_{0}$_ as random sequence
$\gamma(A_{0})=\langle A_{0},A_{1},\ldots,A_{n}\rangle$ where each
$A_{i}\subseteq V$ and for each $i=1,\ldots n$ we have

$A_{i}\cap\displaystyle\bigcup_{j<i}A_{j}=\emptyset$

and for every $v\in A_{i+1}$ there is some $u\in A_{i}$ such that $(u,v)\in E$
and $w_{u,v}\geq 0$. The cascade is generated by the following process, which
we call the _independent cascade process_ or IC-process:

  1. 1.

The initially infected set is the set $A_{0}$, and set $i=0$.

  2. 2.

If $A_{i}$ is empty, then stop; the cascade is finished and $n=i-1$.

  3. 3.

Let $A_{i+1}$ be a set.

  4. 4.

For each $u\in A_{i}$ and $v$ such that $(u,v)\in E$ and $v$ is not yet
infected: Add $v$ to the set $A_{i+1}$ with probability $w_{u,v}$; if $v$ is
added to $A_{i+1}$ mark it as influenced.

  5. 5.

Set $i=i+1$ and go to 2.

Given an information flow network and an initial set $A_{0}$, we denote by

$\sigma(A_{0})=E\left(\left|\bigcup_{A\in\gamma(A_{0})}A\right|\right).$

I.e, $\sigma(A_{0})$ is the expected number of vertices in the cascade and the
function $\sigma$ is known as the _influence function_. There is no closed-
form solution to the computation of the influence function (that we know of),
and it has been proposed that the computation of the influence function is
P#-complete(Shakarian et al. (2015))

###  2.3 The Detachment problem

Let $\mathcal{I}=\\{I_{1},I_{2},\ldots I_{k}\\}$ be a collection of circles
and let $G=(V,E,w)$ be the information flow network induced by $\mathcal{I}$.
Let $p$ be a distribution over $\mathcal{I}$, or, equivalently, a distribution
over the set $\\{1,\ldots,k\\}$. We define the _expected proportional outside
influence_ or EPOI of the information flow network as

$\rho_{\mathcal{I}}=\sum_{I\in\mathcal{I}}p(I)\cdot\left(\frac{\sigma(I)-|I|}{|V\setminus
I|}\right)$

In other words, the EPOI of the network is the expected proportion of non-
circle members influenced when a random circle is the source of a cascade.

Given a collection of circles $\mathcal{I}=\\{I_{1},\ldots,I_{k}\\}$, we
define the _detachment_ of a vertex $v$ from a given circle $I_{i}$ such that
$v\in I_{i}$, as the collection

$\det(\mathcal{I},(v,I_{i}))=\\{I_{1},\ldots,I_{i-1},I_{i}\setminus\\{v\\},I_{i+1},\ldots,I_{k}\\}$

and we generalize this to detaching a set of pairs
$S=\\{(v_{1},I_{i_{1}}),\ldots,(v_{m},I_{i_{m}})\\}$ in the obvious way, i.e,

$\det(\mathcal{I},S)=\det(\det(\cdots\det(\mathcal{I},(v_{1},I_{i_{1}}))\cdots,(v_{m-1},I_{i_{m-1}})),(v_{m},I_{i_{m}}))$

We now define the _$m$ -detachment problem_:

######  Problem 1.

Find the detachment $S$ such that $|S|=m$ and $\rho_{\det(\mathcal{I},S)}$ is
minimized.

##  3 Algorithms and Transformations

###  3.1 Bridge-block graph

######  Definition 2.

Let $V$ be a set of vertices and let $\mathcal{I}$ be a collection of circles.
The _Bridge-Block network_ (BBN) induced by $\mathcal{I}$ is a bipartite graph
$(\mathcal{I},U,F)$, where

  1. 1.

$\mathcal{I}=\\{I_{1},\ldots,I_{k}\\}$, is the set of _blocks_ , i.e., each
circle is called a block of the BBN.

  2. 2.

$U=\\{v\in V\mid\exists i,j:i\neq j\wedge v\in I_{i}\cap I_{j}\\}$, is the set
_bridges_ , i.e., each vertex that belongs to more than one circle, is called
a bridge of the BBN

  3. 3.

$F=\\{(I,v)\mid I\in\mathcal{I}\wedge v\in U\wedge v\in I\\}$, i.e., a block
and a bridge have an edge between them, if and only if the bridge belongs to
the block.

We will use the bridge-block graph in our algorithms, and make use of the fact
that a detachment of $(v,i)$ from $\mathcal{I}$, is equivalent to removing
exactly one edge, namely $(I_{i},v)$ from the bridge-block graph induced by
$\mathcal{I}$.

###  3.2 Greedy algorithm

Data: Circles $\mathcal{I}$, edge weights $w$, initial distribution $p$,
number of detachments $m$

$\rho_{min}\leftarrow\infty$;

for  _$i=1$ to $m$_ do

$(\mathcal{I},U,F)\leftarrow BBN(\mathcal{I})$;

for  _each $(I,v)\in U$_ do

$\mathcal{I}^{\prime}=\det(\mathcal{I},(v,I))$;

if  _$\rho_{\mathcal{I}^{\prime}} <\rho_{min}$_ then

$\rho_{min}\leftarrow\rho_{\mathcal{I}^{\prime}}$;

$\mathcal{I}^{*}\leftarrow\mathcal{I}^{\prime}$;

end if

end for

$\mathcal{I}\leftarrow\mathcal{I}^{*}$;

end for

Result: The network $\mathcal{I}^{*}$ with EPOI $\rho_{min}$

Algorithm 1 Greedy algorithm

A simple greedy algorithm for solving Problem 1 is given in Algorithm 1. The
strategy is very simple, but we present it here for illustrative purpose. The
algorithm itself is amenable to many heuristics and optimizations, such as
omitting vertices that have a particularly low centrality measure. We will
discuss these heurisics in a further chapter.

###  3.3 Min-cut

Because the computation of the EPOI is time consuming with Monte Carlo, there
is definitely a rationale for exploring methods that do not need the
evaluation of the EPOI, and rely purely on heuristics of the graphs instead.

Data: Circles $\mathcal{I}$, weights $w$ and initial distribution $p$

Choose $\mathcal{I}_{1}\subseteq\mathcal{I}$ and
$\mathcal{I}_{2}\subseteq\mathcal{I}$ such that
$\mathcal{I}_{1}\cap\mathcal{I}_{2}=\emptyset$ and $\sigma(\mathcal{I}_{1})$
and $\sigma(\mathcal{I}_{2})$ are large;

Heuristically determine weights for the BBN;

Add vertex $s$ and edge $(s,\mathcal{I}_{1})$, with a large weight;

Add vertex $t$ and edge $(\mathcal{I}_{2},t)$ with a large weight ;

Find Min-Cut for the BBN that separates $s$ and $t$ Result: The network
$\mathcal{I}^{\prime}$ that has 2 components

Algorithm 2 Minimum Cut

######  Theorem 1.

A (minimum) cut that separates two circles in a BBN is equivalent to a
detachment

The theorem is obvious, because in a BBN, all paths alternate between a block
and a bridge. The upside of Algorithm 2 compared with Algorithm 1 is that it
does not require the calculation of the EPOI at each step.

##  4 Empirical and random data

###  4.1 Empirical data

We will make use of data that comes from insider registries of companies
publicly listed on the Helsinki Stock exchange. We use the year 2007 as our
benchmark year, i.e., $I_{i}$ is chosen as the set of insiders for the company
$i$ on the last day of 2006. $I_{i}$ is the set of people who are considered
insiders of a given company $i$. The BBN of the insiders is not fully
connected, so we only choose the companies that are in the largest connected
component of the generated BBN. Our data has $|\mathcal{I}|=106$ companies,
with a total of $|V|=967$ insiders, of whom $|U|=140$ are bridges, i.e.,
individuals who are insiders in more than one company.

###  4.2 Random data generation

The process of generating randomized input for the algorithm involves a
systematic procedure designed to mimic the statistical properties of the
empirical data and with some modifications as to the parameters. We fixed the
ratios

$\alpha=\frac{|V|}{|\mathcal{I}|},~{}\beta=\frac{|V|}{|U|},~{}\gamma=\frac{|V|}{|F|}$

The generative process generates the circles via a distribution that is a mix
of uniform distibution and preferential selection, using a coefficient
$0\leq\rho\leq 1$. The circle assignment was a four-stage process.

  1. 1.

The first process assigns each element a unique circle as follows:

    1. (a)

First we assign one element to each of the circles.

    2. (b)

Let $A^{(}k)$ be the set of assigned elements on iteration $k$. One unassigned
element is picked, and the circle for it is chosen so that the probability of
being assigned into circle $I^{(k+1)}_{i}$ is

$(1-\rho)\frac{1}{|\mathcal{I}|}+\rho\frac{I^{(k)}_{i}}{A^{(k)}}$

    3. (c)

Repeat until each element has been assigned a circle.

  2. 2.

The BBN is then forced into being connected by generating $|\mathcal{I}|-1$
bridges

    1. (a)

An element is picked using uniform distribution, and its circle is chosen as
the root circle. Note that this means larger circles have higher probability
of being chosen.

    2. (b)

At every iteration, a random element $u$ that is not yet connected, is chosen
and added to a randomly chosen connected circle, thereby creating a bridge and
connecting all the elements from its the circle to the graph.

    3. (c)

this is repeated until all circles are connected to the root.

  3. 3.

The rest of the bridges are then created by choosing non-bridge elements at
random and adding them to random circles.

  4. 4.

The bridges are then added to further circles by choosing circles using the
same mixed distribution as in the first stage.

Figure 1: Comparison of empirical (blue) and generated (brown) graph
statistics

Figure 1 shows graphically a comparison of some graph characteristics of
empirical and generated graphs, with $\alpha,\beta$, and $\gamma$ chosen to be
equal to those measured empirically, and $\rho=0.3$. The empirical data had
three abnormally large circles that this generative model fails to march, but
otherwise the properties are very similar.

The transmission probabilities between nodes were generated from a Beta-
distribution with parameters $\alpha=20,\beta=80$, which gives the expected
value of $w$ at $\mu=0.2$, and a standard deviation $\sigma\approx 0.04$. We
empirically determined that the expected EPOI is very robustly about 0.79 over
all the random graphs in our class, including the empirical data.

##  5 Results

We generated 96 random graphs with number of vertices ranging from 100 to 1980
with the process described in the previous section. The objective was to gauge
the robustness and scalability of our two methods. We observed a relatively
stable EPOI over our data, and a modest but stable deterioration of the
algorithms as the size of the graph increases, which is to be expected. Below
we report the result for a sample of 15 of these random graphs.

Table 1: Numerical results n | Min-cut | EPOI cut | EPOI greedy | std of EPOI | base EPOI
---|---|---|---|---|---
280 | 2 | 0.766 | 0.705 | 0.000 | 0.809
320 | 3 | 0.720 | 0.641 | 0.000 | 0.789
680 | 1 | 0.768 | 0.720 | 0.000 | 0.783
740 | 2 | 0.756 | 0.713 | 0.000 | 0.770
860 | 2 | 0.779 | 0.736 | 0.000 | 0.790
880 | 1 | 0.766 | 0.754 | 0.000 | 0.779
900 | 2 | 0.785 | 0.755 | 0.000 | 0.798
1000 | 1 | 0.789 | 0.783 | 0.000 | 0.800
1020 | 3 | 0.734 | 0.742 | 0.000 | 0.786
1040 | 1 | 0.766 | 0.743 | 0.000 | 0.775
1500 | 1 | 0.782 | 0.775 | 0.000 | 0.790
1860 | 2 | 0.777 | 0.757 | 0.000 | 0.783
1920 | 3 | 0.766 | 0.744 | 0.000 | 0.778
1960 | 3 | 0.754 | 0.736 | 0.000 | 0.774
1980 | 1 | 0.789 | 0.783 | 0.000 | 0.793
Empirical (967) | 3 | 0.712 | 0.650 | 0.000 | 0.729

The greedy algorithm performed better than the cutting algorithm in all but
one case. In order to better understand what created the difference, we
conducted a number of comparisons of graph characteristics but found no
explaining factor. Further research is warranted in this respect.

##  6 Discussion

This short paper demonstrates the idea of the detachment problem, and is still
under work. We try to present here the main ideas of detachment, the sort of
networks that it might work on, and one heuristic approach.

Full combinatorial optimization is still intractable in this case, and the
cutting algorithm does not seem to provide a satisfactory heuristic even
compared with the greedy algorithm. Our tentative results already demonstrate
that in realistic problems, the greedy algorithm is not optimal, which is not
surprising. What may be surprising, is that a very simple heuristic algorithm
outperforms it on a problem with realistic characteristics.

## References

  * Ahern (2017) Ahern, K. R., 2017. Information networks: Evidence from illegal insider trading tips. Journal of Financial Economics 125 (1), 26–47.
  * Baltakienė et al. (2022) Baltakienė, M., Baltakys, K., Kanniainen, J., 2022. Trade synchronization and social ties in stock markets. EPJ Data Science 11 (1), 54.
  * Baltakys et al. (2023) Baltakys, K., Baltakienė, M., Heidari, N., Iosifidis, A., Kanniainen, J., 2023\. Predicting the trading behavior of socially connected investors: Graph neural network approach with implications to market surveillance. Expert Systems with Applications 228, 120285.
  * Berkman et al. (2020) Berkman, H., Koch, P., Westerholm, P. J., 2020. Inside the director network: When directors trade or hold inside, interlock, and unconnected stocks. Journal of Banking & Finance 118, 105892.
  * Chen et al. (2012) Chen, D., Lü, L., Shang, M.-S., Zhang, Y.-C., Zhou, T., 2012. Identifying influential nodes in complex networks. Physica a: Statistical mechanics and its applications 391 (4), 1777–1787.
  * Gallos et al. (2007) Gallos, L. K., Liljeros, F., Argyrakis, P., Bunde, A., Havlin, S., 2007. Improving immunization strategies. Physical Review E 75 (4), 045104.
  * Goldenberg et al. (2001) Goldenberg, J., Libai, B., Muller, E., 2001. Talk of the network: A complex systems look at the underlying process of word-of-mouth. Marketing letters 12 (3), 211–223.
  * Kempe et al. (2003) Kempe, D., Kleinberg, J., Tardos, É., 2003. Maximizing the spread of influence through a social network. In: Proceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining. pp. 137–146.
  * Kitsak et al. (2010) Kitsak, M., Gallos, L. K., Havlin, S., Liljeros, F., Muchnik, L., Stanley, H. E., Makse, H. A., 2010. Identification of influential spreaders in complex networks. Nature physics 6 (11), 888–893.
  * Li et al. (2018) Li, Y., Fan, J., Wang, Y., Tan, K.-L., 2018. Influence maximization on social graphs: A survey. IEEE Transactions on Knowledge and Data Engineering 30 (10), 1852–1872.
  * Lü et al. (2016) Lü, L., Chen, D., Ren, X.-L., Zhang, Q.-M., Zhang, Y.-C., Zhou, T., 2016. Vital nodes identification in complex networks. Physics reports 650, 1–63.
  * Morone and Makse (2015) Morone, F., Makse, H. A., 2015. Influence maximization in complex networks through optimal percolation. Nature 524 (7563), 65–68.
  * Nowzari et al. (2016) Nowzari, C., Preciado, V. M., Pappas, G. J., 2016. Analysis and control of epidemics: A survey of spreading processes on complex networks. IEEE Control Systems Magazine 36 (1), 26–46.
  * Pastor-Satorras and Vespignani (2002) Pastor-Satorras, R., Vespignani, A., 2002. Immunization of complex networks. Physical review E 65 (3), 036104.
  * Prieto Curiel and González Ramírez (2021) Prieto Curiel, R., González Ramírez, H., 2021. Vaccination strategies against covid-19 and the diffusion of anti-vaccination views. Scientific Reports 11 (1), 6626.
  * Shakarian et al. (2015) Shakarian, P., Bhatnagar, A., Aleali, A., Shaabani, E., Guo, R., Shakarian, P., Bhatnagar, A., Aleali, A., Shaabani, E., Guo, R., 2015. The independent cascade and linear threshold models. Diffusion in Social Networks, 35–48.
  * Sharma et al. (2019) Sharma, K., Qian, F., Jiang, H., Ruchansky, N., Zhang, M., Liu, Y., 2019. Combating fake news: A survey on identification and mitigation techniques. ACM Transactions on Intelligent Systems and Technology (TIST) 10 (3), 1–42.


# Non-Intrusive Speech Intelligibility Prediction for Hearing Impaired
Individuals using Self-Supervised Speech Representations††thanks: This work
was supported by the Centre for Doctoral Training in Speech and Language
Technologies (SLT) and their Applications funded by UK Research and Innovation
[grant number EP/S023062/1]. This work was also funded in part by TOSHIBA
Cambridge Research Laboratory

###### Abstract

Self-supervised speech representations (SSSRs) have been successfully applied
to a number of speech-processing tasks, e.g. as feature extractor for speech
quality (SQ) prediction, which is, in turn, relevant for assessment and
training speech enhancement systems for users with normal or impaired hearing.
However, exact knowledge of why and how quality-related information is encoded
well in such representations remains poorly understood. In this work,
techniques for non-intrusive prediction of SQ ratings are extended to the
prediction of intelligibility for hearing-impaired users. It is found that
self-supervised representations are useful as input features to non-intrusive
prediction models, achieving competitive performance to more complex systems.
A detailed analysis of the performance depending on Clarity Prediction
Challenge 1 listeners and enhancement systems indicates that more data might
be needed to allow generalisation to unknown systems and (hearing-impaired)
individuals.

Index Terms: hearing loss, metric prediction, neural networks, self-supervised
speech representations

##  1 Introduction

Age-related hearing loss (HL) is an increasingly prevalent problem in
countries with ageing populations worldwide. In the United Kingdom, for
example, approximately $12$ million people suffer from HL of greater than 25
decibels hearing level (dBHL); by $2035$ this is predicted to increase to
$14.2$ million [1]. HL can often impede an individual’s ability to participate
in a spoken conversation, especially in noisy environments, as parts of the
speech can become unintelligible [2]. As such, the development of methods to
increase speech intelligibility (SI) in assistive listening devices to
alleviate this is of paramount importance [3]. While there have been large
improvements in speech enhancement technology thanks to neural network-based
approaches [4, 5, 6] these can often be challenging to implement in small form
factor hearing aid (HA) hardware. Furthermore, given that the exact severity
and nature of hearing loss differs greatly between individuals, a ’one size
fits all’ approach is not viable. The Clarity Project [7] aims to improve the
design of hearing aids via two alternating challenges and related datasets
[8], the Clarity Enhancement Challenge (CEC) and the Clarity Prediction
Challenge (CPC). The CEC is concerned with the design of the actual
enhancement algorithm, while the CPC involves the prediction of the
intelligibility of the enhanced speech for hearing-impaired listeners. The
overall aim of the CPC is to produce systems that mimic the behaviour of
hearing-impaired listeners, reducing the need for expensive and time-consuming
human assessment by listening tests, while also providing a training target or
metric for enhancement systems.
SSSR have been found to be useful either as pretrained layers or feature
transformations in many speech-related applications [9, 10, 11]. It is
understood that self-supervised speech representation (SSSR)s are able to
encode and predict the context of the speech content in the input audio, and
thus model the patterns of spoken language. Recent work [12, 13, 14, 15, 16]
has found that in addition to speech content, SSSRs are also able to capture
information on potentially corrupting noise and distortion in the input audio.
In this work, SSSRs are used as a feature transformation for non-intrusive
neural speech intelligibility prediction networks, trained on the CPC1
challenge dataset. Non-intrusive metric prediction networks using SSSRs are
proposed to serve as feature extractors and analysed for different latent or
output SSSR layers to predict SI for hearing-impaired users.
The remainder of the paper is structured as follows. In Section 2, the SSSRs
used in this work are briefly introduced and Section 3 reviews the use of
SSSRs in related tasks. Section 4 introduces the dataset as used in this work
and in Section 5 the relationships between this dataset and SSSR distance
measures are examined. Finally, in Sections 6 and 7, experiments involving the
use of SSSR as feature representations in non-intrusive intelligibility
prediction networks are described, and the results analysed.

##  2 Self Supervised Speech Representations

Generally, SSSRs are neural networks that, given a waveform representation of
speech audio $\mathbf{s}[n]$, produce a final output which expresses the
_context_ of that input speech audio. As the name suggests, they are trained
in a self-supervised way, typically by _masking_ some segment of the input
audio representation and tasking the network in training to recreate the
masked segment [9]. Structurally, the networks consist of two distinct stages
as shown in Fig. 1.

SSSR
---
$\mathbf{S}_{\mathrm{OL}}$
---
$\mathbf{s}[n]$
---
CNN
---
$\mathcal{G}_{\mathrm{FE}}$
---
Transformers
---
$\mathcal{G}_{\mathrm{OL}}$
---
$\mathbf{S}_{\mathrm{FE}}$
---

Fig. 1: Representations extracted from SSSR model with time-domain input
signal $\mathbf{s}[n]$. Feature channels are sorted [17] and values normalised
for clarity.

The input waveform $\mathbf{s}[n]$ with discrete time index $n$ is first
processed by a number of 1-D convolutional layers, resulting in a two-
dimensional feature encoding representation. In the second stage, this
representation is processed by a number of Transformer [18] layers, to give
the final two-dimensional output. In both stages, one of the dimensions
represents time, while the other represents features. For a time domain signal
$\mathbf{s}[n]$, the output of the initial convolutional neural network (CNN)
encoder stage $\mathcal{G}_{\mathrm{FE}}$ of a SSSR is

$\mathbf{S}_{\mathrm{FE}}=\mathcal{G}_{\mathrm{FE}}(\mathbf{s}[n]),$ (1)

where operator $\mathcal{G}_{\mathrm{FE}}$ denotes the 1-D convolutional
layers encompassing the encoder. The subsequent processing by the Transformer
based stage can be defined by an additional operator
$\mathcal{G}_{\mathrm{OL}}$ denoting the Transformer layers encompassing the
final output stage, i.e.

$\mathbf{S}_{\mathrm{OL}}=\mathcal{G}_{\mathrm{OL}}(\mathcal{G}_{\mathrm{FE}}(\mathbf{s}[n])).$
(2)

Both signal representations $\mathbf{S}_{\mathrm{FE}}$ and
$\mathbf{S}_{\mathrm{OL}}$ have two dimensions: time $T$, depending on the
length of the input audio in block time, and feature dimension $F$.

The _ Cross-Lingual Speech Representation (XLSR)_ [19] is one of the SSSR
representations used in this work. It is a variant of the Wav2Vec2 [9]
structure. It is trained on $436,000$ hours of audio data from a number of
languages, including the BABEL dataset which contains potentially noisy
telephone conversation recordings. Its network is structured in the way
described above, with the outputs of $\mathcal{G}_{\mathrm{FE}}$ having a
feature dimension of $512$ and the final (transformer) outputs of
$\mathcal{G}_{\mathrm{OL}}$ having a feature dimension of $1024$. In this
work, the smallest version of XLSR sourced from the
HuggingFace††https://huggingface.co/facebook/wav2vec2-xls-r-300m is used.
_Hidden Unit Bidirectional Encoder Representations from Transformers (HuBERT)_
[10] differ w.r.t. the general training of a SSSR described above in that the
training target is a cluster of masked frames similar to BERT [20] rather than
the masked frame itself. However, its network structure follows the same
pattern, with the outputs of $\mathcal{G}_{\mathrm{FE}}$ having a feature
dimension of $512$ and the final output representation a feature dimension of
$768$. In this work, we use the HuBERT Large model trained on $960$ hours of
clean English speech sourced from the LibriSpeech [21] dataset, from the
Fairseq GitHub repository††https://github.com/facebookresearch/fairseq.

##  3 SSSRs for Metric Prediction

SSSRs have been applied to metric prediction tasks, typically to quality
prediction [22, 23]. In [13], XLSR representations are used as feature
extraction in a non-intrusive human MOS prediction network. Similarly, in [24]
SSSRs are used for the same quality prediction task, but they are fine-tuned
with a mean pooling layer rather than being used simply as feature extraction.
SSSRs were also applied to the CPC1 challenge in [25], where they were used as
feature extractors alongside spectrograms and learnable filter banks.
In all these cases, only the final SSSR output $\mathcal{G}_{\mathrm{OL}}$ was
considered. However, findings in [12] suggest that the output of the initial
encoding stage $\mathcal{G}_{\mathrm{FE}}$ better captures quality-related
information. As such, in this work, both representations stages are considered
and compared as feature transformations for speech intelligibility prediction.

##  4 Clarity Challenge 1 Data

The dataset for the first Clarity Prediction Challenge 1 (CPC1)[26] as used in
this work can be expressed as a series of sequences: ($\mathbf{\hat{s}}[n]$,
$\\{\mathbf{a}_{l},\mathbf{a}_{r}\\}$, $i$), which is generated as visualised
in Fig. 2.

Speech in noise (SPIN)
---
generator
Listener information
---
(audiogram)
Hearing
---
Aid (HA)
Hearing Loss
---
Simulator
$\mathbf{v}[n]$
---
noise
---
speech
---
$\mathbf{x}_{\\{l,r\\}}[n]$
---
$\mathbf{\hat{s}}[n]$
---
$\mathbf{\hat{s}^{\prime}}[n]$
---
$\mathcal{S}$
---
$\mathbf{a}_{\\{l,r\\}}$
---
$\mathbf{s}[n]$
---
Fig. 2: Signal generation for Clarity Prediction Challenge.

$\mathbf{\hat{s}}[n]$ represents the output of a hearing aid system for some
noisy speech input $\mathbf{x}[n]$, containing some clean speech
$\mathbf{s}[n]$. $\\{\mathbf{a}_{l},\mathbf{a}_{r}\\}$ are the left and right
audiogram representations of a particular listener’s hearing loss. Blue and
red box plots in Fig. 2 illustrate the HL distribution in the CPC1 dataset
from which the individual audiograms are sampled. Finally, $i$ represents the
intelligibility of the audio $\mathbf{\hat{s}}[n]$ for that listener, defined
as the percentage of words they were able to reproduce by speaking aloud
immediately after hearing the audio, compared to a ground truth transcription
of the speech also denoted as the _correctness_ of the listener’s response.
Additionally, audio $\mathbf{\hat{s}}^{\prime}[n]$ is defined as the output of
the baseline Cambridge MSBG hearing loss simulator (HLS), denoted here by
operator $\mathcal{S}$, cf. [27] for additional details on the clarity system.

$\mathbf{\hat{s}}^{\prime}[n]=\mathcal{S}(\mathbf{\hat{s}}[n],\\{\mathbf{a}_{l},\mathbf{a}_{r}\\})$
(3)

The signal $\mathbf{\hat{s}}^{\prime}[n]$ is an approximation of the audio
that is perceived by the hearing-impaired listener. This can be thought of as
encoding the hearing characteristics of the specific listener (audiogram)
within the signal. Note that all signals in the dataset are binaural i.e.
consist of left and right channels, denoted by $l$ and $r$, respectively.
The upper plot in Fig. 3 shows the distribution of correctness $i$ in the CPC1
training set. From this, it can be observed that in the majority of cases, the
listener was able to fully reproduce the speech in the audio they heard, i.e.
$i=100$ for $\approx 50\%$ of the assessed files. The next largest class is
where $i=0$, meaning that the listener was not able to reproduce any words in
the audio. This distribution is due to the more realistic in-the-wild SI
measurement strategy for the Clarity dataset [26] which is in contrast to lab-
based SI matrix tests [28]. The lower panel of Fig. 3 shows the average
correctness $i$ for each listener in the CPC1 training set. With the exception
of listener L0227, all of the listeners achieve similar performance.

Fig. 3: Histogram showing the distribution of ground truth correctness $i$ in
CPC1 training set (top) and a bar chart showing average correctness $i$ per
listener in the CPC1 training set (bottom). Dotted lines are respective
overall average values.

##  5 Analysing Relationships between SSSRs and Human Speech Intelligibility

In order to express the relationship between SSSRs and correctness $i$ in the
dataset, two distance measures are defined in a mean squared error (MSE)
sense:

$d_{\mathrm{FE}}=\frac{1}{TF}\sum_{t}^{T}\sum_{f}^{F}(\mathbf{S}_{\mathrm{FE}}[t,f]-\mathbf{{P}}_{\mathrm{FE}}[t,f])^{2}$
(4)
$d_{\mathrm{OL}}=\frac{1}{TF}\sum_{t}^{T}\sum_{f}^{F}(\mathbf{S}_{\mathrm{OL}}[t,f]-\mathbf{{P}}_{\mathrm{OL}}[t,f])^{2}$
(5)

The distance $d_{\mathrm{FE}}$ in (4) expresses the MSE distance between the
SSSR feature encoding layer representations $\mathbf{S}_{\mathrm{FE}}[t,f]$ of
the clean reference audio $\mathbf{s}[n]$ and the representations
$\mathbf{{P}}_{\mathrm{FE}}[t,f]$ of the test signal $\mathbf{p}[n]$, while
(5) expresses the MSE distance between the SSSR output layer representations
$\mathbf{S}_{\mathrm{OL}}[t,f]$ and $\mathbf{{P}}_{\mathrm{OL}}[t,f]$, with
$t$ and $f$ denoting block time and feature index, respectively. Note that
$\mathbf{p}[n]$ and is a placeholder for either the speech signal after HA
enhancement $\mathbf{\hat{s}}[n]$ or this signal after HLS processing
$\mathbf{\hat{s}}^{\prime}[n]$ as shown in Fig. 2. Distances in (4), (5) are
designed to express the distortion captured by the SSSR due to the
transformations which have been applied to $\mathbf{s}[n]$ to produce e.g.
$\mathbf{\hat{s}}^{\prime}[n]$, i.e. the artificial distortion/reverb added to
create $\mathbf{x}[n]$, enhancement by the hearing aid system (in
$\mathbf{\hat{s}}[n]$) and finally the HLS. In addition to distances (4) and
(5) the MSE distance between spectrogram representations of $\mathbf{s}[n]$
and $\mathbf{p}[n]$,

$d_{\mathrm{SG}}=\frac{1}{TF_{\mathrm{Hz}}}\sum_{t}^{T}\sum_{f_{\mathrm{Hz}}}^{F_{\mathrm{Hz}}}\left(\mathbf{S}_{\mathrm{SG}}[t,f_{\mathrm{Hz}}]-\mathbf{P}_{\mathrm{SG}}[t,f_{\mathrm{Hz}}]\right)^{2},$
(6)

will be analysed, with $f_{\mathrm{Hz}}$ and $F_{\mathrm{Hz}}$ denoting the
technical frequency and the highest frequency analysed, respectively. In the
following, the left (first) channel of the audio is used to compute the
distance measures (4), (5) and (6).

Table 1: Spearman and Pearson correlations between distance measures and
correctness values $i$ in the CPC1 training set, strongest correlations in
bold.

Representation, Distance | $\mathbf{p}[n]$ | Spearman | Pearson
---|---|---|---
SPEC, $d_{\mathrm{SG}}$, (6) | $\mathbf{\hat{s}}[n]$ | $-0.10$ | $-0.18$
SPEC, $d_{\mathrm{SG}}$, (6) | $\mathbf{\hat{s}}^{\prime}[n]$ | $-0.09$ | $-0.07$
$\mathrm{XLSR}$, $d_{\mathrm{FE}}$, (4) | $\mathbf{\hat{s}}[n]$ | $-0.13$ | $-0.16$
$\mathrm{XLSR}$, $d_{\mathrm{FE}}$, (4) | $\mathbf{\hat{s}}^{\prime}[n]$ | $-0.24$ | $-0.28$
$\mathrm{XLSR}$, $d_{\mathrm{OL}}$, (5) | $\mathbf{\hat{s}}[n]$ | $-0.26$ | $-0.27$
$\mathrm{XLSR}$, $d_{\mathrm{OL}}$, (5) | $\mathbf{\hat{s}}^{\prime}[n]$ | $-0.24$ | $-0.24$
$\mathrm{HuBERT}$, $d_{\mathrm{FE}}$, (4) | $\mathbf{\hat{s}}[n]$ | $\mathbf{-0.38}$ | $\mathbf{-0.47}$
$\mathrm{HuBERT}$, $d_{\mathrm{FE}}$, (4) | $\mathbf{\hat{s}}^{\prime}[n]$ | $-0.23$ | $-0.29$
$\mathrm{HuBERT}$, $d_{\mathrm{OL}}$, (5) | $\mathbf{\hat{s}}[n]$ | ${-0.10}$ | ${-0.17}$
$\mathrm{HuBERT}$, $d_{\mathrm{OL}}$, (5) | $\mathbf{\hat{s}}^{\prime}[n]$ | ${-0.28}$ | ${-0.32}$

Table 1 shows the Spearman and Pearson correlations of the MSE distances
(4)-(6) with the correctness values $i$ for the CPC1 training set. Absolute
correlations are low, but this is expected for the Clarity dataset (cf. [26]
and Section 4). Comparing the distances between the feature representations in
(4)-(6) and the intelligibility scores $i$ allows for an expression of how
distortion in the signal, which might affect intelligibility, is captured by
that feature representation. Interestingly, applying the hearing loss
simulation $\mathcal{S}$ in (3) does not uniformly improve the correlation
with $i$ across all distances in Table 1; only for the XLSR encoder output
representation distance $d_{\mathrm{FE}}$ and the HuBERT final output
representation distance $d_{\mathrm{OL}}$ does using
$\mathbf{\hat{s}}^{\prime}[n]$ lead to higher correlation than using
$\mathbf{\hat{s}}[n]$.

##  6 SSSR-based Intelligibility Prediction

This section proposes the use of SSSRs as features in non-intrusive neural
intelligibility prediction networks. Following the findings from Table 1, both
the hearing aid output signal $\mathbf{\hat{s}}[n]$ and that signal processed
by the hearing loss simulation $\mathbf{\hat{s}}^{\prime}[n]$ are used as the
input audio to the models, as no conclusive best representation is indicated
by these results.

###  6.1 Dataset

Models are trained on both the open and closed training and test sets detailed
in the CPC1 description [8]. The closed set has the same listeners and systems
for both train and testsets, while the open set has $5$ unseen listeners and
$1$ unseen system in its testset. For more detail as to how these sets are
differentiated, see [8]. A validation set is created using $10\%$ of the
available training data. As we are interested in non-intrusive predictors,
either the hearing aid output signal $\mathbf{\hat{s}}[n]$ or the hearing loss
simulated audio signal $\mathbf{\hat{s}}^{\prime}[n]$ are used to predict the
Correctness label $i$. For the closed set, the training set contains $4376$
utterances, the validation set $487$ and the test set $2421$. The training set
contains $3222$ utterances for the open set, $358$ for the validation set, and
$632$ for the test set.

###  6.2 Model Structure and Experiment Setup

A model structure inspired by [13] is chosen for the SI prediction network.
Five feature extraction methods are used; outputs of
$\mathcal{G}_{\mathrm{FE}}$ and $\mathcal{G}_{\mathrm{OL}}$ for both, HuBERT
and XLSR representations, as well as a spectrogram representation denoted as
$\mathrm{SPEC}$. After the feature extraction, the resultant representation is
processed by $2$ bidirectional long short-term memory (BLSTM) layers with an
input size equal to the feature dimension $F$ of the input and a hidden layer
size of $F/2$. The final layer is an attention pooling feed-forward layer,
similar to that in NISQA [22] with a single output neuron and a sigmoid
activation to output the predicted correctness $\hat{i}$ (normalised between
$0$ and $1$). Note that due to different dimensions $F$ of different feature
representations, the number of parameters in each network varies from
$923,906$ for the models using spectrogram representations to as many as
$14,701,570$ for the models using the XLSR output layer, i.e.
$\mathcal{G}_{\mathrm{OL}}$.
The two input audio representations $\mathbf{\hat{s}}[n]$ or
$\mathbf{\hat{s}}^{\prime}[n]$ are used, i.e. the output of the hearing aid
systems and the enhanced audio processed by the hearing loss simulation, as in
(3). As these audio representations have two channels, each channel is
processed by the model separately; during training, the loss for each channel
is computed and then summed before being back-propagated to the model. During
validation and testing, the maximum value between each channel is taken as an
approximation of the _better ear effect_ [29].
The spectrogram representation is created by a short time Fourier transform
(STFT) with a window length of $20$ ms, a hop length of $10$ ms and an FFT
size of $1024$. All audio is re-sampled to $16$ kHz such that it can be used
as inputs to the SSSR models.

##  7 Results

In addition to the intrusive (reference-signal-based) challenge baseline, the
best-performing non-intrusive entries to the challenge are reported in this
section as additional baselines, as the proposed system is also non intrusive.
Challenge entry E23 [30] makes use of contrastive predictive coding and vector
quantisation features. E06 [31] is similar to the proposed system, denoted by
$\mathrm{SPEC}$ in the following, but uses a CNN based network structure. E33
[25] also utilises SSSRs as feature extraction, but spectrogram and learnable
filterbank features are also used as model inputs. E29 [32] makes use of an
information-theory-inspired approach, wherein the difference between internal
representations in neural automatic speech recognition (ASR) systems is used
to approximate human intelligibility, and was the overall best non-intrusive
challenge entry.

###  7.1 Results on CPC1 Closed set

Table 2: Non-Intrusive Prediction Performance on the CPC1 closed set. Best performances for baselines and proposed methods in boldface font. Model Name | RMSE | Var | Spearman | Pearson
---|---|---|---|---
CPC1 Baseline | 28.50 | – | 0.62 | –
E23 [30] | 41.50 | – | 0.07 | –
E06 [31] | 32.00 | – | 0.43 | –
E33 [25] | 24.10 | – | 0.75 | –
E29 [32] | 23.30 | – | 0.77 | –
$\mathrm{SPEC}$ $\mathbf{\hat{S}}_{\mathrm{SPEC}}$ | 25.45 | 0.52 | 0.59 | 0.72
$\mathrm{SPEC}$ $\mathbf{\hat{S}^{\prime}}_{\mathrm{SPEC}}$ | 25.45 | 0.52 | 0.58 | 0.72
$\mathrm{HuBERT}\leavevmode\nobreak\ \mathbf{\hat{S}}_{\mathrm{FE}}$ | 30.82 | 0.61 | 0.44 | 0.56
$\mathrm{HuBERT}\leavevmode\nobreak\ \mathbf{\hat{S}^{\prime}}_{\mathrm{FE}}$ | 26.64 | 0.53 | 0.56 | 0.70
$\mathrm{HuBERT}\leavevmode\nobreak\ \mathbf{\hat{S}}_{\mathrm{OL}}$ | 24.76 | 0.50 | 0.59 | 0.74
$\mathrm{HuBERT}\leavevmode\nobreak\ \mathbf{\hat{S}^{\prime}}_{\mathrm{OL}}$ | 24.82 | 0.50 | 0.61 | 0.74
$\mathrm{XLSR}\leavevmode\nobreak\ \mathbf{\hat{S}}_{\mathrm{FE}}$ | 25.01 | 0.50 | 0.60 | 0.74
$\mathrm{XLSR}\leavevmode\nobreak\ \mathbf{\hat{S}^{\prime}}_{\mathrm{FE}}$ | 25.33 | 0.51 | 0.60 | 0.72
$\mathrm{XLSR}\leavevmode\nobreak\ \mathbf{\hat{S}}_{\mathrm{OL}}$ | 28.42 | 0.58 | 0.47 | 0.66
$\mathrm{XLSR}\leavevmode\nobreak\ \mathbf{\hat{S}^{\prime}}_{\mathrm{OL}}$ | 30.20 | 0.61 | 0.52 | 0.64

Table 2 shows the performance of the proposed systems for the CPC1 closed set.
All proposed systems show comparable performance with the best-performing
challenge entries, although, none of the proposed systems outperforms system
E29. It should be noted, however, that the computation overhead to implement
system E29 is significantly greater than any of the proposed systems here, as
several state-of-the-art ASR systems must be trained and fine-tuned for E29.
Of the proposed systems trained on the outputs of the hearing loss simulation
$\mathbf{\hat{s}}^{\prime}[n]$, the best performing is the model which uses
HuBERT output representations $\mathbf{\hat{S}^{\prime}}_{\mathrm{OL}}$ as
features. This is consistent with the findings in Table 1 which shows that the
distance measure using this representation had the highest correlation with
$i$ of those distances computed using $\mathbf{\hat{s}}^{\prime}[n]$. Of those
trained using the hearing aid outputs $\mathbf{\hat{s}}[n]$, HuBERT’s output
$\mathbf{\hat{S}}_{\mathrm{OL}}$ is also the best performing achieving near
identical performance to the $\mathbf{\hat{s}}^{\prime}[n]$ model. In terms of
the difference in performance between using earlier SSSR representations
$\mathcal{G}_{\mathrm{FE}}$ or output representations
$\mathcal{G}_{\mathrm{OL}}$ as features, this seems to depend on the SSSR
used; for HuBERT the output layers perform best, while for XLSR the feature
encoder layers show better performance.

Table 3: Non Intrusive Prediction Performance on the CPC1 open set. Model Name | RMSE | Var | Spearman | Pearson
---|---|---|---|---
CPC 1 Baseline | 36.50 | – | 0.53 | –
E23 [30] | 43.70 | – | 0.05 | –
E33 [25] | 28.9 | – | 0.65 | –
E29 [32] | 24.60 | – | 0.73 | –
$\mathrm{SPEC}\leavevmode\nobreak\ \mathbf{\hat{S}}_{\mathrm{SPEC}}$ | 32.84 | 1.29 | 0.35 | 0.50
$\mathrm{SPEC}\leavevmode\nobreak\ \mathbf{\hat{S}^{\prime}}_{\mathrm{SPEC}}$ | 29.16 | 1.15 | 0.57 | 0.60
$\mathrm{HuBERT}\leavevmode\nobreak\ \mathbf{\hat{S}}_{\mathrm{FE}}$ | 33.69 | 1.30 | 0.27 | 0.45
$\mathrm{HuBERT}\leavevmode\nobreak\ \mathbf{\hat{S}^{\prime}}_{\mathrm{FE}}$ | 35.31 | 1.40 | 0.19 | 0.24
$\mathrm{HuBERT}\leavevmode\nobreak\ \mathbf{\hat{S}}_{\mathrm{OL}}$ | 32.43 | 1.22 | 0.47 | 0.54
$\mathrm{HuBERT}\leavevmode\nobreak\ \mathbf{\hat{S}^{\prime}}_{\mathrm{OL}}$ | 29.66 | 1.14 | 0.60 | 0.61
$\mathrm{XLSR}\leavevmode\nobreak\ \mathbf{\hat{S}}_{\mathrm{FE}}$ | 31.83 | 1.26 | 0.49 | 0.52
$\mathrm{XLSR}\leavevmode\nobreak\ \mathbf{\hat{S}^{\prime}}_{\mathrm{FE}}$ | 30.86 | 1.19 | 0.56 | 0.56
$\mathrm{XLSR}\leavevmode\nobreak\ \mathbf{\hat{S}^{\prime}}_{\mathrm{OL}}$ | 31.85 | 1.25 | 0.42 | 0.49
$\mathrm{XLSR}\leavevmode\nobreak\ \mathbf{\hat{S}^{\prime}}_{\mathrm{OL}}$ | 34.54 | 1.36 | 0.26 | 0.37

###  7.2 Results of CPC1 Open set

Table 3 shows the performance of the proposed systems on the more challenging
CPC1 open set (cf. Section 4). Performance of the proposed systems is
significantly worse than that of the closed set for all systems, with a much
larger variance in MSE in all cases, but all proposed systems still outperform
the baseline. The poorer performance might be due to overfitting of the models
to the training data, (in particular to the enhancement systems in the
training set) as the test data contains unseen enhancement systems and
listeners. All of the models here perform similarly poorly.

###  7.3 System and Listener-wise Analysis

Fig. 4: System (top) and listener-wise (bottom) correctness prediction
$\hat{i}$ (l./green) vs. true $i$ (r./brown) using HuBERT output for
$\mathbf{\hat{S}}_{\mathrm{OL}}$ model on CPC1 closed set.

Fig. 5: System (top) and listener-wise (bottom) correctness prediction
$\hat{i}$ (l./green) vs. true $i$ (r./brown) using HuBERT outpur for
$\mathbf{\hat{S}^{\prime}}_{\mathrm{OL}}$ model on CPC1 closed set.

Fig. 6: System (top) and listener-wise (bottom) correctness prediction
$\hat{i}$ (l./green) vs. true $i$ (r./brown) using HuBERT output for
$\mathbf{\hat{S}}_{\mathrm{OL}}$ model on CPC1 open set. Listeners and Systems
unseen during training are bold.

Fig. 7: System (top) and listener-wise (bottom) correctness prediction
$\hat{i}$ (l./green) vs. true $i$ (r./brown) using HuBERT output for
$\mathbf{\hat{S}^{\prime}}_{\mathrm{OL}}$ model on CPC1 open set. Listeners
and Systems unseen during training are bold.

For further analysis, Figs. 4 and 5 show ground truth and predicted
correctness across the hearing aid systems and across the listeners in the
CPC1 open testset for the HuBERT $\mathbf{\hat{S}}_{\mathrm{OL}}$ and HuBERT
$\mathbf{\hat{S}^{\prime}}_{\mathrm{OL}}$ models, respectively. Both models
show similar performance across the different hearing aid systems, both
successfully assigning low scores to the audio enhanced by the E005 hearing
aid system. This indicates that the models are able (at some level) to detect
the distortions introduced by this enhancement. Similarly, there is little
difference in performance across the subset of listeners for the two models;
this suggests that the listener-specific hearing loss information which the
$\mathbf{\hat{S}^{\prime}}_{\mathrm{OL}}$ model has access to (encoded in the
audio) does not aid in the intelligibility prediction performance. It should
be noted that already the enhancement system (hearing aid) has (implicitly)
access to the hearing loss information and is expected to process its input
signal accordingly (cf. Fig. 2). Interestingly, both models overestimate the
intelligibility ratings of speaker L0227 who performs worse than average at
the intelligibility task (cf. Fig. 3). This suggests that L0227’s lower
performance is not due to their hearing loss but rather other unknown
factor(s); audiogram information for this listener does not show that they
have particularly severe hearing loss.
Figs. 7 and 6 show ground truth and predicted correctness across the hearing
aid systems and across the listeners for the more challenging CPC1 closed
testset for the HuBERT output for $\mathbf{\hat{S}}_{\mathrm{OL}}$ and HuBERT
output for $\mathbf{\hat{S}^{\prime}}_{\mathrm{OL}}$ models, respectively.
Systems and listeners which are unseen during the training of the models are
highlighted by bold-font. Here, the overfitting of the proposed system to the
hearing aid systems during training can be observed by the poor performance on
the unseen hearing aid system in the testset, E018. The overall lower
performance of the proposed systems on the closed set is shown by the
listener-wise plots, with both systems significantly overestimating the
correctness versus the true value; however the encoding of the hearing loss
information in $\mathbf{\hat{S}^{\prime}}_{\mathrm{OL}}$ does appear to have
some positive effect here.

##  8 Conclusion

This work explores the use of SSSR models as feature extraction for non-
intrusive speech intelligibility prediction networks in comparison to
traditional, spectrogram-based input. Both, the final SSSR representation and
the intermediate output of the SSSR feature encoder are compared for the first
time for an SI prediction task for hearing-impaired users. Results indicate
that encoding the hearing loss of a particular listener via (an additional)
hearing loss simulation does not typically improve performance. Additionally,
models tend to overfit to specific hearing aid systems, as demonstrated by the
results on the open set which might be alleviated by larger datasets released
in the future. The upcoming CPC2 challenge data, which includes twice the
number of enhancement systems, could mitigate this issue.

## References

  * [1] Neil Park,  “Population estimates for the UK, England and Wales, Scotland and Northern Ireland, provisional: mid-2019,”  Hampshire: Office for National Statistics, 2020.
  * [2] Christoph Völker, Anna Warzybok, and Stephan M.A. Ernst,  “Comparing Binaural Pre-processing Strategies III: Speech Intelligibility of Normal-Hearing and Hearing-Impaired Listeners,”  Trends in Hearing, vol. 19, 2015.
  * [3] Stefan Goetze, Feifei Xiong, Jan Rennies, Thomas Rohdenburg, and Jens-E. Appell,  “Hands-free telecommunication for elderly persons suffering from hearing deficiencies,”  in IEEE Int. Conf. on E-Health Networking, Application and Services (Healthcom’10), 2010.
  * [4] Harishchandra Dubey, Ashkan Aazami, Vishak Gopal, Babak Naderi, Sebastian Braun, Ross Cutler, Alex Ju, Mehdi Zohourian, Min Tang, Hannes Gamper, Mehrsa Golestaneh, and Robert Aichner,  “Icassp 2023 deep noise suppression challenge,” 2023.
  * [5] Szu-Wei Fu, Cheng Yu, Kuo-Hsuan Hung, Mirco Ravanelli, and Yu Tsao,  “Metricgan-u: Unsupervised speech enhancement/ dereverberation based only on noisy/ reverberated speech,”  2021\.
  * [6] George Close, Thomas Hain, and Stefan Goetze,  “MetricGAN+/-: Increasing Robustness of Noise Reduction on Unseen Data,”  in EUSIPCO 2022, Belgrade, Serbia, Aug. 2022.
  * [7] Simone Graetzer, Michael Akeroyd, Jon P. Barker, Trevor J. Cox, John F. Culling, Graham Naylor, Eszter Porter, and Rhoddy Viveros Muñoz,  “Clarity: Machine learning challenges to revolutionise hearing device processing,” 2020.
  * [8] Simone Graetzer, Jon Barker, Trevor J. Cox, Michael Akeroyd, John F. Culling, Graham Naylor, Eszter Porter, and Rhoddy Viveros Muñoz,  “Clarity-2021 Challenges: Machine Learning Challenges for Advancing Hearing Aid Processing,”  in Proc. Interspeech 2021, 2021, pp. 686–690.
  * [9] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli,  “wav2vec 2.0: A framework for self-supervised learning of speech representations,”  in Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, Eds. 2020, vol. 33, pp. 12449–12460, Curran Associates, Inc.
  * [10] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed,  “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,” 2021.
  * [11] Shu wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y. Lin, Andy T. Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, Tzu-Hsien Huang, Wei-Cheng Tseng, Ko tik Lee, Da-Rong Liu, Zili Huang, Shuyan Dong, Shang-Wen Li, Shinji Watanabe, Abdelrahman Mohamed, and Hung yi Lee,  “SUPERB: Speech Processing Universal PERformance Benchmark,”  in Proc. Interspeech 2021, 2021, pp. 1194–1198.
  * [12] George Close, William Ravenscroft, Thomas Hain, and Stefan Goetze,  “Perceive and predict: self-supervised speech representation based loss functions for speech enhancement,”  in Proc. ICASSP 2023, 2023.
  * [13] Bastiaan Tamm, Helena Balabin, Rik Vandenberghe, and Hugo Van hamme,  “Pre-trained speech representations as feature extractors for speech quality assessment in online conferencing applications,”  in Interspeech 2022. Sep 2022, ISCA.
  * [14] Tsun-An Hsieh, Cheng Yu, Szu-Wei Fu, Xugang Lu, and Yu Tsao,  “Improving perceptual quality by phone-fortified perceptual loss using wasserstein distance for speech enhancement,” 2020.
  * [15] Amitay Sicherman and Yossi Adi,  “Analysing discrete self supervised speech representation for spoken language modeling,”  in ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023, pp. 1–5.
  * [16] Ankita Pasad, Bowen Shi, and Karen Livescu,  “Comparative layer-wise analysis of self-supervised speech models,” 2023\.
  * [17] William Ravenscroft, Stefan Goetze, and Thomas Hain,  “Att-TasNet: Attending to Encodings in Time-Domain Audio Speech Separation of Noisy, Reverberant Speech Mixtures,”  Frontiers in Signal Processing, vol. 2, 2022.
  * [18] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin,  “Attention is all you need,”  in Advances in Neural Information Processing Systems, I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds., 2017, vol. 30.
  * [19] Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, Alexei Baevski, Alexis Conneau, and Michael Auli,  “XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale,”  in Proc. Interspeech 2022, 2022, pp. 2278–2282.
  * [20] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova,  “BERT: Pre-training of deep bidirectional transformers for language understanding,”  in Proc. of ACL 2019, Minneapolis, Minnesota, June 2019, pp. 4171–4186, Association for Computational Linguistics.
  * [21] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur,  “Librispeech: An asr corpus based on public domain audio books,”  in 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2015, pp. 5206–5210.
  * [22] Gabriel Mittag, Babak Naderi, Assmaa Chehadi, and Sebastian Möller,  “NISQA: A deep CNN-self-attention model for multidimensional speech quality prediction with crowdsourced datasets,”  in Interspeech 2021. aug 2021, ISCA.
  * [23] Erica Cooper, Wen-Chin Huang, Tomoki Toda, and Junichi Yamagishi,  “Generalization ability of mos prediction networks,” 2021.
  * [24] Helard Becerra, Alessandro Ragano, and Andrew Hines,  “Exploring the influence of fine-tuning data on wav2vec 2.0 model for blind speech quality prediction,”  in Proc. Interspeech 2022, 2022, pp. 4088–4092.
  * [25] Ryandhimas Edo Zezario, Fei Chen, Chiou-Shann Fuh, Hsin-Min Wang, and Yu Tsao,  “MBI-Net: A Non-Intrusive Multi-Branched Speech Intelligibility Prediction Model for Hearing Aids,”  in Proc. Interspeech 2022, 2022, pp. 3944–3948.
  * [26] Jon Barker, Michael Akeroyd, Trevor J. Cox, John F. Culling, Jennifer Firth, Simone Graetzer, Holly Griffiths, Lara Harris, Graham Naylor, Zuzanna Podwinska, Eszter Porter, and Rhoddy Viveros Munoz,  “The 1st Clarity Prediction Challenge: A machine learning challenge for hearing aid intelligibility prediction,”  in Proc. Interspeech 2022, 2022, pp. 3508–3512.
  * [27] Michael Anthony Stone and Brian C. J. Moore,  “Tolerable hearing aid delays. i. estimation of limits imposed by the auditory path alone using simulated hearing losses.,”  Ear and hearing, vol. 20 3, pp. 182–92, 1999.
  * [28] Birger Kollmeier, Anna Warzybok, Sabine Hochmuth, Melanie A. Zokoll, Verena Uslar, Thomas Brand, and Kirsten C. Wagener,  “The multilingual matrix test: Principles, applications, and comparison across languages: A review,”  International Journal of Audiology, vol. 54, no. sup2, pp. 3–16, 2015.
  * [29] Patrick M Zurek and GA Studebaker,  “Binaural advantages and directional effects in speech intelligibility,”  Acoustical factors affecting hearing aid performance, vol. 2, pp. 255–275, 1993.
  * [30] Alex F. McKinney and Benjamin Cauchi,  “Non-intrusive binaural speech intelligibility prediction from discrete latent representations,”  IEEE Signal Processing Letters, vol. 29, pp. 987–991, 2022.
  * [31] George Close, Samuel Hollands, Stefan Goetze, and Thomas Hain,  “Non-intrusive Speech Intelligibility Metric Prediction for Hearing Impaired Individuals,”  in Proc. Interspeech 2022, 2022, pp. 3483–3487.
  * [32] Zehai Tu, Ning Ma, and Jon Barker,  “Unsupervised Uncertainty Measures of Automatic Speech Recognition for Non-intrusive Speech Intelligibility Prediction,”  in Proc. Interspeech 2022, 2022, pp. 3493–3497.

  *[HL]: hearing loss
  *[dBHL]: decibels hearing level
  *[SI]: speech intelligibility
  *[HA]: hearing aid
  *[CEC]: Clarity Enhancement Challenge
  *[CPC]: Clarity Prediction Challenge
  *[SSSR]: self-supervised speech representation
  *[SSSRs]: self-supervised speech representation
  *[CNN]: convolutional neural network
  *[XLSR]: Cross-Lingual Speech Representation
  *[CPC1]: Clarity Prediction Challenge 1
  *[HLS]: hearing loss simulator
  *[MSE]: mean squared error
  *[BLSTM]: bidirectional long short-term memory
  *[STFT]: short time Fourier transform
  *[ASR]: automatic speech recognition