###### Abstract

We study the transport of an overdamped Brownian particle in a tilted rough
potential driven by a gaussian white noise.We conisder various forms of rough
potentials to analyse the effect of roughness on the diffusion and current
generated.The variation of current, diffusion coefficient, and peclet number
with the static tilting force, amplitude of roughness and temperature were
studied. We concluded that the roughness reduces the particle current at any
range of temperature. But at low temperatures, it enhances the diffusion
coeffeicient of the particle

Effect of roughness on the transport of an overdamped Brownian particle
Prathyusha S Nair1, Ronald Benjamin1

1Department of Physics, Cochin University of Science and Technology, Kochi

.

##  1 Introduction

Particle transport by random fluctuations from its environment has
implications in various fields of Science. The particles can achieve
unidirectional motion by rectifying these random fluctuations. Brownian
ratchets are devices that make use of these random fluctuations to move in a
particular directions by breaking the detailed balance symmetry. Tilting
ratchets, pulsating ratchets, temperature ratchets are some type of ratchets,
that has been studied so far. There lies different Physical, chemical and
Biological applications of Brownian ratchet mechanism. For example, motion of
kinesin motor protein on microtubules, transport of metastatic cancer cells,
particle separation and particle pre-concentration based on their size etc.

So far, in the study of particle transport smooth type of potentials were
considered. But, most of the Brownian particles encounter with a rough
environment. When Frauenfelder et al studied about the protein folding process
[1],[2], it was understood that this process resulted in some rough/rugged
behaviour in the energy landscape of proteins, which affected the movement of
proteins. Being inspired from these findings Zwanzig studied about the
diffusion in a rough potential[3]. He generated a rough potential by
superimposing a fast randomly oscillating perturbation on a smooth background
potential. He proved that the effective diffusion cofficient decreases in the
rough potential compared to the smooth potential. Later on, it was also found
that the particle current generated decreases in a rough potential, in the
overdamped regime[4].It was found that when the noise strength is very low the
roughness is found to enhance the particle transport, but when the noise
strength is large it hinders the particle transport [5]. The positive effect
of roughness on the particle transport was again proved in [6]. On choosing
suitbale parameter regime, the roughness can increase the particle current and
efficiency.

The diffusion of the particle in presence of a rough potential haven’t been
fully explored.So our aim is to study the behavior of diffusion and directed
current in a rough potential.In this paper we are investigating the effect of
roughness on the transport of an overdamped Brownian particle in the presence
of a static tilt. We have analysed transport parameters for different cases of
potentials. The model of the system and the potentials we have considered are
described in Sec.2. And we discuss about the numerical results in Sec.3. The
findings are concluded in Sec.4

##  2 Methods

Here we consider the motion of an overdamped Brownian particle in a periodic
rough potential V(x) which is subjected to a static tilting force F and random
thermal fluctuations $\xi$(t). The dynamics of the system is described by the
Langevin equation,

$\gamma\dot{x}=-V^{\prime}(x)+F+\xi(t)$ (1)

The thermal fluctuations are modeled by a gaussian white noise which
satisfies,

$\langle\xi(t)\rangle=0$ (2) $\langle\xi(t)\xi(t^{\prime})\rangle=2\gamma
k\textsubscript{b}T\delta(t-t^{\prime}).$ (3)

The periodic potential V(x) contains a smooth part $V_{0}$(x) and a rough part
$V_{1}$(x) in the form,

$V(x)=V_{0}(x)+V_{1}(x)$ (4)

We start by considering a sinusoidal function as the smooth potential and
roughness also of sinusoidal form(Case I).

$V_{0}(x)=sin(2\pi x/L)$ (5) $V_{1}(x)=\epsilon(c_{1}\sin(\lambda_{1}\Omega
x)+c_{2}\sin(\lambda_{2}\Omega x)).$ (6)

Here L is the spatial period of the potential $V_{0}(x)$ and $\epsilon$ is the
amplitude of roughness. $\lambda_{1}\Omega$, $\lambda_{2}\Omega$ and
$c_{1}$,$c_{2}$ are the periodicities and amplitudes of the two sine functions
in $V_{1}$. Higher the values of $\lambda$s, denser the perturbations. The
perturbation becomes deeper on increasing the $\epsilon$. The values of
parameters ,$c_{1}$,$c_{2}$,$\lambda_{1}$, $\lambda_{2}$ are taken as 1, 0.5,
1, and 2 respectively. The spatial period L is taken as 2$\pi$.

Next we consider the case, where the unperturbed potential is a spatially
asymmetric function.(Case II). The potential $V_{0}(x)$ takes the form,

$V_{0}(x)=-[\sin(x)+\frac{1}{4}\sin(2x+\psi-\frac{\pi}{2})]$ (7)

The rough part of the potential is given by, equation (6). Here $\psi$ is the
parameter that breaks the symmetry of the potential.The potential is periodic
with period L =2$\pi$.

As a next case (Case III ) a piece wise linear potential is taken for
analysis.

$V_{0}=\begin{cases}\frac{x}{\alpha},&0\leq x<\alpha\\\
\frac{\lambda-x}{\lambda-\alpha},&\alpha<x<\lambda\end{cases}$ (8)

where $\alpha$ is the asymmetry parameter and $\lambda$ is the spatial period
of the system. When $\alpha$=0.5 then, the potemtial is symmetric in space.
Here a regular sinusoidal potential is considered as the roughness. Therefore
the total potential will be,

$V(x)=[V_{0}(x)-\epsilon\cos(2\pi kx)/2]/N$ (9)

where k is the wavenumber which is an odd integer and N is the normalization
factor. Here we have taken N as (1+$\epsilon$).

The effective transport of a Brownian particle is defined by the coherency of
the system. The more coherent the particles are the more effecyive is the
transport. So here we study some parameters that measures the effectiveness of
the transport. The quantity of primary interest is the particle current
generated. It is defined as,

$\langle\dot{x}\rangle=\lim_{t\to\infty}\frac{\langle x(t)\rangle}{t}.$ (10)

The brackets $\langle$..$\rangle$ denotes the ensemble average of the
quantity. We then study how much the particle trajectory has spread. This is
quantified by the effective diffusion coefficient, which is given by,

$D\textsubscript{eff}=\lim_{t\to\infty}\frac{\langle x^{2}(t)\rangle-\langle
x(t)\rangle^{2}}{2t}.$ (11)

The Peclet number is a transport parameter that incoperates velocity of the
ratchet and coherency of the ratchet. It is described as,

$Pe=\frac{L\langle v\rangle}{D_{eff}}$ (12)

where v is the velocity of the particle, $D_{eff}$ is the effective diffusion
coefficient. L is the characteristic length of the sysytem. Generally we take
the spatial period of the potential as the characteristic length.

The effective diffusion coefficient for a particle in a periodic potential .
[7],[8],[9] is defined according to the Refs as,

$D_{eff}=\frac{D_{O}\int_{x_{0}}^{x_{0}+L}I_{+}(x)[I_{-}(x)]^{2}\frac{dx}{L}}{[\int_{x_{0}}^{x_{0}+L}I_{-}(x)\frac{dx}{L}]^{3}}$
(13)

where,

$I_{\pm}(x)=\mp D_{0}^{-1}e^{\pm V(x)/k_{b}T}\int_{x}^{x\mp L}e^{\mp
V(x)/k_{b}T}dy$ (14)

in which $x_{0}$ is an arbitrary position and $D_{0}=k_{b}T/\gamma$. And the
particle current of the system is equal to [7, 8, 9],

$\langle\dot{x}\rangle=N^{-1}(1-e^{-LF/k_{b}T})$ (15)

Here we have carried out numerical simulations to calculate the transport
properties. Equation (1) has been numerically integrated using Euler’s method
with time step dt $\leq$ $10^{-4}$ for Case I and Case III. The Case II was
simulated using stochastic Runge kutta method of order two with time step dt
$\leq$ $10^{-4}$ . The total time of observation was taken as 1000 units. The
initial position of the particle was randomly chosen from a uniform
distribution in the interval [0,1]. The transport properties were calculated
by averaging over 1000 trajectories.

##  3 Results and Discussions

In this study we have considered different types of potentials to analyse the
effect of roughness on the system. Lets take the Case I. To investigate how
the roughness changes the transport proprties, we first analyse the particle
current as a function of tilting force for different values of amplitude of
roughness $\epsilon$. In Fig 1a and Fig.2 we can see that the current
increases monotonically as a function of F. Increasing the tilting force F,
helps the particle to over come the potential barrier. To analyse the effect
of roughness we focus on to the values of F upto 10(Fig. 1.b, Fig.3). We can
see that roughness doesnot contribute to the transport. The particle current
is high when there is no roughness. The current decreases with the roughness
$\epsilon$. Fig 4.also affirms this observation. For different
temperatures,increasing the roughness seems to hinder the particle current.
When the amplitude of roughness increases it becomes difficult for the
particle to overcome each potential minima due to the rough part.

Figure 1: a) The particle current J as a function of Tilting force for various
values of $\epsilon$ at temperature T=0.1 and $\lambda_{1}$=10
$\lambda_{2}$=20. b)J as a function of F where F$\leq$ 10(Case -I)

And when we vary the temperature, the current is found to increases with
it(see Fig. 5). In the absence of roughness the J is approximately same for
all values of Temperature. When the temperature is very much lower than the
value of $\epsilon$ (less than 0.05), the current J is negligible in presence
of roughness. With such a low value of temperature it seems difficult for the
particle cross the potential minima. The temperature determines the noise
strength. So only in presence of higher noise strength the particle get enough
pull to easily move out of the potential well.

Figure 2: The particle current J as a function of Tilting force F for various
values of $\epsilon$ at temperature T=0.01 and $\lambda_{1}$=10
$\lambda_{2}$=20(Case -I) Figure 3: The particle current J as a function of
Tilting force F for various values of $\epsilon$ at temperature T=0.01 and
$\lambda_{1}$=10 $\lambda_{2}$=20 for smaller values of F.(Case -I)

The effective diffusion coefficient is numerically calculated using
equation(11). The effective diffusion coefficient is high when the particle
diffuses over the potential barrier. In Fig.6 and Fig.7 we have studied the
variation of diffusion coefficient as a function of the tilting force F at two
different temperatures.

Figure 4:  The particle current as a function of the amplitude of roughness
$\epsilon$ for various Temperatures when the tilting force F =4 and
$\lambda_{1}$=10 $\lambda_{2}$=20.(Case-I) Figure 5: The particle current J as
a function of temperature T for various values of $\epsilon$ when tilting
force F=2 and $\lambda_{1}$=10 $\lambda_{2}$=20. (Case-I) Figure 6: The
Effective diffusion coefficient as a function of tilting force F for different
values of $\epsilon$ at a temperature T =0.1 and $\lambda_{1}$=10,
$\lambda_{2}$=20. (Case-I) Figure 7: The Effective diffusion coefficient as a
function of tilting force F for different values of $\epsilon$ at a
temperature T =0.01 and $\lambda_{1}$=10, $\lambda_{2}$=20. (Case-I) Figure 8:
The Effective diffusion coefficient as a function of $\epsilon$ for different
values of temperature when the tilting force F =4 and $\lambda_{1}$=10,
$\lambda_{2}$=20.(Case-I) Figure 9: The Effective diffusion coefficient as a
function of temperature T for various values of $\epsilon$ when tilting force
F=2 and $\lambda_{1}$=10 $\lambda_{2}$=20.(Case-I)

When the tilting force is greater than zero (F>0), slope of the effective
potential is negative and the particle moves in the positive direction. At a
critical value of force F it over comes the force due to the potential. And
the particle diffuses out from the potential minima. The peaks in the curve
suggests that the tilting force F has over come the force due to the potential
barrier. We can see that till a particular value of F the roughness seems to
hinder the particle diffusion. And after that point the roughness is enhancing
the effective diffusion coefficient upto a value of F. The roughness does not
seem to have much effect on $D_{eff}$. When the temperature decreases the
position of critical values of F shifts to the left. The observation we had
about the effect of rouhness on $D_{eff}$is affirmed in Fig.8.

Figure 10: The Peclet number as a function of tilting force F for different
values of $\epsilon$ at a temperature T =0.1 and $\lambda_{1}$=10,
$\lambda_{2}$=20. (Case-I) Figure 11: The Peclet number as a function of
tilting force F for different values of $\epsilon$ at a temperature T =0.01
and $\lambda_{1}$=10, $\lambda_{2}$=20. (Case-I) Figure 12: The Peclet number
Pe as a function of $\epsilon$ for different values of temperature when the
tilting force F =4 and $\lambda_{1}$=10, $\lambda_{2}$=20.(Case-I) Figure 13:
The Peclet number Pe as a function of temperature T for various values of
$\epsilon$ when tilting force F=2 and $\lambda_{1}$=10
$\lambda_{2}$=20.(Case-I)

At very low temperatures the effective diffusion coefficient increases upto a
particular value of roughness and then it decreases. And the roughness seems
to have negligible effect on Diffusion coefficient at higher temperatures.
This is again clarified by varying Diffusion coefficient as a function of
temperature Fig.9. As we increase the temperature we find that the diffusion
coefficient is almost the same for different values of roughness.

Now we move to Case II. Here we have taken the asymmetric case by choosing the
aymmetry parameter as $\psi=0.5\pi$. The parameters of rough part (Equation 7)
$c_{1}$,$c_{2}$, $\lambda_{1}$, $\lambda_{2}$ are chosen as 1, 0.5, 1,and 2.
We have analysed particle current as a function of tilting force for different
values of $\epsilon$ at different temperatures and different $\Omega$s
(Fig.14, Fig.15, Fig.16). As we increase the tilting force the effective
potential becomes more and more tilted, which helps the particle to overcome
the potential barrier easily.Here the slope of the potential is negative and
hence the particle moves in the positive direction. It is evident fro thee
figures that, the roughness reduces the particle current for smaller vaues fo
tilting force. But at higher values of F, roughness doesnot seem to effect the
particle cuurent. The particle current is almost same for various values of
roughness.

Figure 14:  The particle current J as a function of Tilting force for various
values of $\epsilon$ at temperature T=0.1 and $\Omega=20$ .(Case -II) Figure
15:  The particle current J as a function of Tilting force for various values
of $\epsilon$ at temperature T=0.2 and $\Omega=20$ .(Case -II)

The effect of roughness on the particle current is visible from Fig.18 and
Fig.19. As we increase the amplitude of roughness, it becomes difficult for
the particle to jump out of these minima of the rough potential. At higher
temperatures the particle current will be higher. And when we increase the
tilting force, the maximum value of the current increases.

To study the behaviour of diffusion in the second case we varied the diifusion
coefficient as a function of the static tilting force for different amplitude
of roughness at different temperatures(Fig.21,22,23). The diffuion coefficient
$D_{eff}$ increases with the tilting force upto a critical value and then it
decreases. In the rough potential the tilting force has to overcome the force
due to the smooth part and force due to the rough part of the potential. Hence
there occurs multiple peaks in the graph each time the particle moves out of
the potential minima.

Figure 16:  The particle current J as a function of Tilting force for various
values of $\epsilon$ at temperature T=0.05 and $\Omega=20$ .(Case -II) Figure
17: The particle current J as a function of $\epsilon$ for different values of
temperature when the tilting force F =2 and $\Omega$ =20 (Case-II). Figure 18:
The particle current J as a function of $\epsilon$ for different values of
temperature when the tilting force F =10 and $\Omega$ =20 (Case-II). Figure
19: The particl current J as a function oftemperature T for different values
of $\epsilon$ at F=2 with $\Omega=20$ Figure 20: The effective diffusion
coefficient as a function of tilting force F for different values of
$\epsilon$ at temperature T=0.05 with $\Omega=20$ Figure 21: The effective
diffusion coefficient as a function of amplitude of roughness $\epsilon$ for
different values of temperature T at F=10 with $\Omega=20$ Figure 22: The
effective diffusion coefficient as a function of amplitude of roughness
$\epsilon$ for different values of temperature T at F=5 with $\Omega=20$
Figure 23: The effective diffusion coefficient as a function of amplitude of
roughness $\epsilon$ for different values of temperature T at F=2 with
$\Omega=20$ Figure 24: The effective diffusion coefficient as a function of
temperature T for different values of $\epsilon$ at F=2 with $\Omega=20$
Figure 25: The Peclet number as a function of amplitude of roughness
$\epsilon$ for different values of temperature T at F=2 with $\Omega=20$
Figure 26: The Peclet number as a function of amplitude of roughness
$\epsilon$ for different values of temperature T at F=5 with $\Omega=20$
Figure 27: The Peclet number as a function of temperature T for different
values of $\epsilon$ at F=2 with $\Omega=20$

##  4 Conclusions

On analysing the behaviour of current in various potential, we found that, the
roughness does have a negative impact on the particle current. The particle
current was found to decrease with roughness at all range of temperatures we
have considered. But at low temperatures the diffusion of the particle is
being enhanced by the roughness. The diffusion coefficient of the particle is
found to increase with roughness after a critical value of F, at lower
temperatures. At higher temperatures roughness doesn’t effect the effective
diffusion coefficient. So we concluded that roughness does not promote
directed transport at lower temperatures.

## References

  * [1] Frauenfelder, H., Sligar, S. G., & Wolynes, P. G.(1991). The energy landscapes and motions of proteins. Science, 254(5038), 1598-1603.
  * [2] Ansari, A., Berendzen, J., Bowne, S. F., Frauenfelder, H., Iben, I. E., Sauke, T. B., … & Young, R. D.(1985) Protein states and proteinquakes. Proceedings of the National Academy of Sciences, 82(15), 5000-5004.
  * [3] Zwanzig, R. Diffusion in a rough potential. Proceedings of the National Academy of Sciences 1988, 85(7), 2029-2030.
  * [4] Mondal, D., Ghosh, P. K., & Ray, D. S.(2009). Noise-induced transport in a rough ratchet potential. The Journal of chemical physics, 130(7), 074703.
  * [5] Barik, D. (2021). Roughness in the periodic potential enhances transport in a driven inertial ratchet. Physical Review E, 104(2), 024103.
  * [6] Camargo, S., & Anteneodo, C. (2018). Impact of rough potentials in rocked ratchet performance. Physica A: Statistical Mechanics and its Applications, 495, 114-125.
  * [7] Reimann, P., Van den Broeck, C., Linke, H., Hänggi, P., Rubi, J. M., & Pérez-Madrid, A. (2001). Giant acceleration of free diffusion by use of tilted periodic potentials. Physical review letters, 87(1), 010602.
  * [8] Reimann, P., Van den Broeck, C., Linke, H., Hänggi, P., Rubi, J. M., & Pérez-Madrid, A.(2002). Diffusion in tilted periodic potentials: Enhancement, universality, and scaling. Physical Review E, 65(3), 031104.
  * [9] Heinsalu, E., Tammelo, R., & Örd, T. (2004). Diffusion and current of Brownian particles in tilted piecewise linear potentials: Amplification and coherence. Physical Review E, 69(2), 021111.


# Improving Acoustic Word Embeddings through Correspondence Training of Self-
supervised Speech Representations

Amit Meghanani, Thomas Hain
Speech and Hearing Research Group
Department of Computer Science, The University of Sheffield, United Kingdom
<EMAIL_ADDRESS>
###### Abstract

Acoustic word embeddings (AWEs) are vector representations of spoken words. An
effective method for obtaining AWEs is the Correspondence Auto-Encoder (CAE).
In the past, the CAE method has been associated with traditional MFCC
features. Representations obtained from self-supervised learning (SSL)-based
speech models such as HuBERT, Wav2vec2, etc., are outperforming MFCC in many
downstream tasks. However, they have not been well studied in the context of
learning AWEs. This work explores the effectiveness of CAE with SSL-based
speech representations to obtain improved AWEs. Additionally, the capabilities
of SSL-based speech models are explored in cross-lingual scenarios for
obtaining AWEs. Experiments are conducted on five languages: Polish,
Portuguese, Spanish, French, and English. HuBERT-based CAE model achieves the
best results for word discrimination in all languages, despite HuBERT being
pre-trained on English only. Also, the HuBERT-based CAE model works well in
cross-lingual settings. It outperforms MFCC-based CAE models trained on the
target languages when trained on one source language and tested on target
languages.

Improving Acoustic Word Embeddings through Correspondence Training of Self-
supervised Speech Representations

Amit Meghanani, Thomas Hain Speech and Hearing Research Group Department of
Computer Science, The University of Sheffield, United Kingdom
<EMAIL_ADDRESS>

##  1 Introduction

Self-supervised learning (SSL)-based speech representations are becoming
popular in speech processing and producing state-of-the-art results in many
downstream tasks such as automatic speech recognition, speaker verification,
keyword spotting, voice conversion, etc Yang et al. (2021). These
representations are obtained using self-supervised learning on large amounts
of unlabelled speech data. Wav2vec2 Baevski et al. (2020), HuBERT Hsu et al.
(2021), and WavLM Chen et al. (2022) are a few examples of such SSL-based
speech models. However, representations obtained from these models have not
been extensively explored in the context of learning acoustic word embeddings
(AWEs). AWEs are fixed-dimensional vector representations of spoken words that
find applications in various downstream tasks, such as query-by-example search
Settle et al. (2017); Yuan et al. (2018); Hu et al. (2021), keyword spotting
Barakat et al. (2011), providing clues for human lexical processing Matusevych
et al. (2020), hate speech detection in low resource settings Jacobs et al.
(2023), etc.

Recently, the work Sanabria et al. (2023) proposed extracting AWEs from SSL-
based speech representations using a mean pooling mechanism. The authors
suggest that SSL-based speech representations, which are contextualized, can
be effectively converted into AWEs using a straightforward pooling mechanism.
On the other hand, Correspondence Auto-Encoder (CAE) based training strategies
for AWEs Kamper (2019) using MFCC Davis and Mermelstein (1980) features are
shown to be promising in the literature. Correspondence training involves an
auto-encoder where a spoken word serves as the input to the encoder, and the
target output of the decoder is a different instance of the same spoken word.
This approach helps to preserve acoustic-phonetic information while filtering
out unnecessary details such as speaker, acoustic environment, and duration,
etc. Both encoder and decoder are typically recurrent neural networks (RNNs).
More details about the model will be presented in Sec. 2 and Sec. 4.2.
Correspondence training has also been explored in the work Meghanani and Hain
(2024) to improve content representations of SSL-based speech models.

The work Lin et al. (2023) uses a Correspondence Transformer Encoder (CTE) for
obtaining robust AWEs, trained from scratch and a large-scale unlabelled
speech corpus. In contrast, in this work, pre-trained SSL speech models are
coupled with a simple RNN based auto-encoder for correspondence training to
obtain robust AWEs. This work attempts to use the correspondence training of
auto-encoder to obtain the AWEs by leveraging SSL-based speech representations
instead of MFCC features as input features to the CAE model. Further, cross-
lingual capabilities are also examined for SSL-based AWEs trained with CAE
method. The SSL models (HuBERT, Wav2vec2, and WavLM) used in this work are
pre-trained on English data. However, it has been demonstrated that these
models work well as feature extractors for the all the languages considered in
this study. The performances on the word-discrimination task for all the
languages (Polish, Portuguese, Spanish, and French) are as good as on the
English language (Sec. 5). A detailed analysis is also conducted to assess the
importance of contextual information in spoken words by comparing feature
extraction with and without context. For this work, we obtain spoken words for
all five languages from the subset of Multilingual LibriSpeech (MLS) Pratap et
al. (2020) dataset. The derived dataset consists of five languages (Polish,
Portuguese, Spanish, French, English) with start and end timestamps of spoken
words 111https://github.com/Trikaldarshi/SSL_AWE. We chose MLS dataset for our
experiments as many previous works Matusevych et al. (2020); Abdullah et al.
(2021a, 2022, b); Kamper et al. (2021) on AWEs rely on GobalPhone Schultz et
al. (2013) dataset, which is not freely available.

The main contributions of this work are as follows:

  1. 1.

Utilizing corresponding training with SSL-based speech representations to
obtain highly discriminative AWEs.

  2. 2.

Showing effectiveness of SSL models, pre-trained only on English, as feature
extractors in cross-lingual scenarios for obtaining high-quality AWEs.

  3. 3.

Quantitatively demonstrating that incorporating the context of the spoken word
in SSL-based speech representations leads to the production of more robust
AWEs.

The rest of the paper is as follows: Sec. 2 describes the correspondence auto-
encoder methodology to obtain AWEs; Sec. 3 describes the data preparation and
data statistics; Sec. 4 describes the details of the experiments; Sec. 5
describes the results and analysis; Sec. 6 concludes the work with possible
future directions. Sec. 7 describes the limitations of the work.

##  2 Methodology

Correspondence auto-encoder is trained with input as a spoken word and target
output as a different instance of the same spoken word. Typically, Recurrent
Neural Network (RNN) based encoder and decoder are used, hence the model is
referred to as CAE-RNN. The rationale behind this training method is that CAE-
RNN will preserve only the acoustic-phonetic information and filter out the
other unnecessary information factors such as speaker, duration, acoustic
environment, etc Kamper (2019). Fig. 1 shows the CAE-RNN model setup. The
input to the ENC is a sequence of acoustic feature vectors
($X={X_{1},X_{2},...,X_{m}}$) of a spoken word. The target output is the
sequence of acoustic feature vectors of the different instance of the same
spoken word ($X^{\prime}={X^{\prime}_{1},X^{\prime}_{2},...,X^{\prime}_{n}}$).
The encoder produces the AWE (e) of the spoken word $X$, which is then fed to
the decoder to reconstruct $X^{\prime}$. The output of the decoder is
represented as $Y={Y_{1},Y_{2},...,Y_{n}}$. The mean squared loss function for
a single training pair ($X,X^{\prime}$) can be described as following:

Figure 1: CAE-RNN training setup for extracting AWEs Kamper (2019).

$L=\sum_{k=1}^{n}||X^{\prime}_{k}-Y_{k}||^{2}$ (1)

, where $X^{\prime}={X^{\prime}_{1},X^{\prime}_{2},...,X^{\prime}_{n}}$ is the
target output and $Y={Y_{1},Y_{2},...,Y_{n}}$ is the output of the decoder as
shown in Fig. 1.

##  3 Data Preparation

Data Statistics | Polish | Portuguese | Spanish | French | English
---|---|---|---|---|---
Train | Dev | Test | Train | Dev | Test | Train | Dev | Test | Train | Dev | Test | Train | Dev | Test
# Spoken Words | 104448 | 4595 | 4563 | 117820 | 4964 | 4659 | 82258 | 3721 | 3601 | 84267 | 3004 | 3114 | 92352 | 3147 | 3192
# Unique Spoken Words | 9346 | 3818 | 3887 | 9785 | 3696 | 3539 | 7085 | 2678 | 2769 | 7221 | 2394 | 2433 | 7157 | 2448 | 2527
# Speaker | 11 | 4 | 4 | 42 | 10 | 10 | 79 | 20 | 20 | 120 | 18 | 18 | 182 | 42 | 41
Total Duration (hours) | 18.5 | 0.8 | 0.8 | 21.9 | 0.93 | 0.87 | 14.7 | 0.67 | 0.65 | 14.9 | 0.55 | 0.56 | 16.6 | 0.56 | 0.57

Table 1: A summary of the data statistics for all five languages across the
train, dev, and test splits

The Multilingual LibriSpeech (MLS) dataset Pratap et al. (2020) is utilized to
obtain spoken words. Five languages, namely Polish, Portuguese, Spanish,
French, and English, are selected from MLS. For each language, approximately
25,000 utterances are selected for the training set, 500 for the development
set, and 500 for the test set. These selected utterances are force-aligned to
obtain the spoken word boundaries using the Montreal Forced Aligner toolkit
McAuliffe et al. (2017). Only spoken words with a duration of 0.5 seconds or
longer are included in the derived dataset, following the standard practice in
the literature He et al. (2017). Spoken words with a frequency greater than 50
or less than 5 are excluded from the derived dataset. Table 1 presents a
summary of the statistics for the final extracted dataset , encompassing all
five languages. The speakers across different sets are non-overlapping, which
is a desirable characteristic for evaluating AWEs as they should be robust to
speaker variations. Polish language had limited available data and
consequently has the fewest number of speakers, while English has the highest
number. The duration in Table 1 represents the total time duration of spoken
words across the different sets.

##  4 Experimental Setup

Experiments are conducted on all the five languages with SSL-based speech
representations as input features extracted from Wav2vec2, HuBERT, and WavLM.
Experiments are also conducted with MFCC as input features. First, the feature
extraction methods for various SSL-based speech representations and MFCCs are
described. Then, the configuration of the CAE-RNN model is explained, along
with the mean pooling baseline Sanabria et al. (2023) and the AE-RNN method
(without correspondence training), for comparison. Next, the word
discrimination task is described, which is used for evaluating the quality of
the extracted AWEs. Finally, the training details of the CAE-RNN and other
models are provided.

###  4.1 Feature Extraction

####  4.1.1 SSL-based Speech Representations

SSL models are pre-trained on large amount of unlabelled speech data. The task
defined for the pre-training is known as the pretext task. Each model differs
based on how the pretext task was defined, the data used for pre-training, and
the model architecture. In this work, the “BASE” architectures of Wav2vec2,
HuBERT, and WavLM model 222https://github.com/pytorch/fairseq (all with
$\approx$ 95M parameters) are used for feature extraction. All these models
are pre-trained on 960 hours of LibriSpeech data Panayotov et al. (2015). A
“BASE" architecture typically has a multi-layer CNN-based feature encoder
followed by 12 Transformer layers. In this work, representations from each
model are extracted from the final (i.e. 12th) Transformer layer. For all the
above mentioned SSL models, 768-dimensional feature vectors are obtained for
each spoken word at a framerate of 20 ms.

SSL-based speech representations are extracted in two different ways: the
first one is extracted using the context around the spoken word, and the other
one is extracted without the context, as described here:

  1. 1.

With context: In this case, first the SSL-based speech representations of the
entire spoken utterance are computed and then the time boundaries of the
spoken word is used to get the representations of the segment belonging to the
spoken word. This ensures that the extracted representations capture the
context around the spoken word as the entire utterance is processed by the SSL
model. Let us assume $U$ represents an utterance and $X$ represents a spoken
word instance present in the utterance $U$ with start and end timestamps
denoted as $t_{1}$ and $t_{2}$. If $f$ represents the SSL model, then the SSL-
based speech representation for the entire utterance is computed $Z=f(U)$.
Then the speech representations for the spoken word $X$ will be
$Z_{t_{1}:t_{2}}$.

  2. 2.

Without context: In this case, no context is considered and SSL-based speech
representations are extracted by inputting only speech segments belonging to
the spoken words to the SSL models. Hence, in this case, the SSL-based speech
representation for the spoken word $X$ will be $Z=f(U_{t_{1}:t_{2}})$.

####  4.1.2 MFCC Features

For each spoken word, 20-dimensional MFCC features are extracted with 30 ms
window size and 20 ms shift along with delta and delta-delta features, which
leads to 60-dimensional MFCC feature vectors.

###  4.2 Model Details

A 4-layer Bidirectional GRU with a hidden dimension of 256 is used for both
the encoder and decoder in the CAE-RNN model. Dropout rate is set to 0.2. The
final hidden state of the encoder-GRU is fed to a fully connected layer to
obtain 128-dimensional AWE (e) as shown in Fig. 1. This embedding is then fed
to the decoder at each time step as input to the decoder Kamper (2019). The
output of the decoder is then fed to a fully connected layer to produce the
target output.

A regular auto-encoder RNN (AE-RNN) model is also used as one of the baselines
with similar configurations. AE-RNN model is an auto-encoder model where input
and target output is exactly the same spoken word, i.e. input-output training
pair is ($X,X$). A mean pooling model is also used as baseline Sanabria et al.
(2023), which does not require any training. This method computes the mean of
the SSL-based speech representations to get the 768-dimensional AWE of a
spoken word.

###  4.3 Word Discrimination Task

To evaluate the AWEs, the same-different word-discrimination task is used
Kamper et al. (2015); Carlin et al. (2011). First, all possible spoken word
pairs are generated. For example, if there are total N spoken words, then the
total generated spoken word pairs for comparison will be
$\binom{N}{2}=\frac{N(N-1)}{2}$. After that, the cosine distance between the
AWEs of these pairs are computed and compared with a threshold to decide
whether the spoken words are same or different. The average precision (AP) is
calculated by varying all the possible threshold values, which is the area
under the precision-recall curve. AP is reported for the same-different word
discrimination task. Word-discrimination task is applied on the test set,
which has unseen speakers during training. Also, a subset of the test set is
created for all five languages in such a way that none of the words in the
subset are encountered during training. This particular subset is referred to
as $\text{test}^{\prime}$. The word-discrimination task is also conducted on
$\text{test}^{\prime}$. The total number of generated spoken word pairs for
both the test and $\text{test}^{\prime}$ sets is described in Table 2 for all
languages.

| Spoken Word
---
Pairs (in million)
Polish | Portuguese | Spanish | French | English
test | 10.4 M | 10.8 M | 6.4 M | 4.8 M | 5.1 M
$\text{test}^{\prime}$ | 4.7 | 2.6 M | 1.8 M | 1.3 M | 1.4 M

Table 2: The total number of spoken word pairs generated for the test and
$\text{test}^{\prime}$ sets.

###  4.4 Training Details

The total number of generated correspondence training pairs ($X,X^{\prime}$)
for each language is as follows: 9,55,106 for Polish, 11,63,468 for
Portuguese, 7,80,197 for Spanish, 7,95,613 for French, and 9,72,532 for
English. The remaining training details are as follows for various inputs:

  * •

SSL-based Speech Representations as Input: CAE-RNN models are trained for 30
epochs, using a learning rate of 0.0001 with Adam optimizer and a batch size
of 512. In each run, the model with the best performance on the development
set in terms of word-discrimination is selected as the final model for
evaluation on the test set. AE-RNN models are trained for 50 epochs, keeping
all other parameters same as mentioned above for the CAE-RNN models.

  * •

MFCC as Input: Both AE-RNN and CAE-RNN models with MFCC as inputs are trained
for 100 epochs, using a learning rate of 0.0001 with Adam optimizer. The batch
size for the AE-RNN model was chosen as 64, while for the CAE-RNN model it was
set to 256 based on preliminary experiments for better convergence. Similarly
to the previous case, the model with the best performance on the development
set in terms of word-discrimination is selected as the final model for
evaluation on the test set.

Model | Polish | Portuguese | Spanish | French | English
---|---|---|---|---|---
AE-RNN | 0.20 | 0.10 | 0.17 | 0.01 | 0.01
CAE-RNN | 0.56 | 0.41 | 0.57 | 0.43 | 0.24

Table 3: AP on the test set for word-discrimination task using MFCCs as input
features for AE-RNN and CAE-RNN models in different languages.

Model | Polish | Portuguese | Spanish | French | English
---|---|---|---|---|---
AE-RNN | 0.21 | 0.10 | 0.24 | 0.03 | 0.01
CAE-RNN | 0.54 | 0.47 | 0.63 | 0.57 | 0.33

Table 4: AP on the $\text{test}^{\prime}$ set for word-discrimination task
using MFCCs as input features for AE-RNN and CAE-RNN models in different
languages.

##  5 Results and Analysis

AWE Extraction Method |  Input Features | Polish | Portuguese | Spanish | French | English
---|---|---|---|---|---|---
| with
---
context
| without
---
context
| with
---
context
| without
---
context
| with
---
context
| without
---
context
| with
---
context
| without
---
context
| with
---
context
| without
---
context
Mean Pooling | Wav2vec2 | 0.01 | 0.00 | 0.00 | 0.00 | 0.02 | 0.00 | 0.03 | 0.01 | 0.02 | 0.00
WavLM | 0.07 | 0.00 | 0.01 | 0.00 | 0.03 | 0.00 | 0.05 | 0.02 | 0.07 | 0.03
HuBERT | 0.17 | 0.33 | 0.10 | 0.15 | 0.14 | 0.32 | 0.22 | 0.31 | 0.15 | 0.24
AE-RNN | Wav2vec2 | 0.10 | 0.00 | 0.08 | 0.00 | 0.15 | 0.01 | 0.13 | 0.01 | 0.07 | 0.00
WavLM | 0.34 | 0.14 | 0.21 | 0.11 | 0.43 | 0.20 | 0.34 | 0.23 | 0.26 | 0.17
HuBERT | 0.44 | 0.40 | 0.36 | 0.27 | 0.58 | 0.52 | 0.45 | 0.40 | 0.36 | 0.34
CAE-RNN | Wav2vec2 | 0.86 | 0.71 | 0.86 | 0.63 | 0.93 | 0.79 | 0.71 | 0.61 | 0.82 | 0.52
WavLM | 0.86 | 0.72 | 0.76 | 0.63 | 0.92 | 0.85 | 0.70 | 0.61 | 0.66 | 0.51
HuBERT | 0.90 | 0.82 | 0.88 | 0.71 | 0.95 | 0.89 | 0.74 | 0.65 | 0.86 | 0.65

Table 5: AP scores for the word-discrimination task on the test set using SSL-
based speech representations as input features for all five languages. AWE
extraction methods include mean pooling Sanabria et al. (2023), AE-RNN, and
CAE-RNN.

Table 3 shows the baseline results with MFCC features as input for the AE-RNN
and CAE-RNN models. This demonstrates the effectiveness of the CAE-RNN model
over the AE-RNN model for the word-discrimination task, as the CAE-RNN
consistently outperforms the AE-RNN for all languages. Table 4 presents the
results for the derived subset of the test set ($\text{test}^{\prime}$) with
similar trends. It is worth noting that the AP on the $\text{test}^{\prime}$
set is relatively better than that of the original test set in most cases.
This is likely due to the fact that the number of spoken word pairs generated
for the evaluation on the $\text{test}^{\prime}$ is significantly fewer
compared to the original test set, as mentioned in Table 2.

Table 5 displays the results obtained from using various SSL-based speech
representations (Wav2vec2, WavLM, and HuBERT) as input features, combined with
different AWE extraction methods (mean pooling, CAE-RNN, and AE-RNN). The
results presented in Table 5 represent the AP for the word-discrimination task
on the test set, employing different SSL-based speech representation feature
extraction setups (‘with context’ and ‘without context’). From Table 5, it is
evident that the AWEs derived ‘with context’ exhibit greater robustness. The
AP on the test set for all languages is significantly better when utilizing
SSL-based speech representations ‘with context’ compared to the feature
extraction ‘without context’. As shown in Table 5, the CAE-RNN model
demonstrates superior performance when using SSL-based speech representations
as input features compared to the MFCC-based baseline model (Table 3) across
all languages. Furthermore, Table 5 provides a comparison of the CAE-RNN model
with other baseline models (mean pooling and AE-RNN) when utilizing SSL-based
speech representations as input features. CAE-RNN consistently outperforms
both the AE-RNN and mean pooling methods for all languages and SSL models.
Another advantage of the CAE-RNN model over mean pooling is that the AWEs
obtained from CAE-RNN have a dimension of 128, while mean pooling-based AWEs
are 768-dimensional. Based on the results presented in Table 5, it is evident
that the HuBERT features consistently achieve the best performance across all
configurations and languages. Specifically, when using the CAE-RNN method for
AWE extraction and SSL-based speech representations extracted ‘with context’,
the HuBERT achieves the highest AP on the test set: 0.90 for Polish, 0.88 for
Portuguese, 0.95 for Spanish, 0.74 for French, and 0.86 for English. The
performance order can be sorted as HuBERT $>$ Wav2vec2 $>$ WavLM $>$ MFCCs
when using the CAE-RNN-based AWE model and SSL-based speech representations
extracted ‘with context’.

Table 6 presents the results for the $\textbf{test}^{\prime}$ set, which
includes unseen words and speakers. The models exhibit similar trends in
performance in this case as well. This provides evidence that the proposed
methodology performs equally well on unseen words. One interesting finding is
that the SSL-based speech representations considered in this work were pre-
trained solely on English language. Despite this, they are capable of
generating meaningful features for other languages, resulting in good
performance as demonstrated in Table 5 and 6 for the word-discrimination task.

AWE Extraction Method |  Input Features | Polish | Portuguese | Spanish | French | English
---|---|---|---|---|---|---
| with
---
context
| without
---
context
| with
---
context
| without
---
context
| with
---
context
| without
---
context
| with
---
context
| without
---
context
| with
---
context
| without
---
context
Mean Pooling | Wav2vec2 | 0.01 | 0.01 | 0.01 | 0.00 | 0.04 | 0.01 | 0.06 | 0.01 | 0.03 | 0.00
WavLM | 0.05 | 0.00 | 0.02 | 0.01 | 0.05 | 0.01 | 0.10 | 0.05 | 0.09 | 0.04
HuBERT | 0.18 | 0.31 | 0.11 | 0.17 | 0.19 | 0.40 | 0.30 | 0.40 | 0.18 | 0.29
AE-RNN | Wav2vec2 | 0.09 | 0.01 | 0.09 | 0.01 | 0.24 | 0.02 | 0.21 | 0.03 | 0.08 | 0.01
WavLM | 0.33 | 0.12 | 0.25 | 0.12 | 0.55 | 0.28 | 0.44 | 0.33 | 0.29 | 0.23
HuBERT | 0.44 | 0.41 | 0.41 | 0.31 | 0.69 | 0.64 | 0.55 | 0.50 | 0.43 | 0.41
CAE-RNN | Wav2vec2 | 0.87 | 0.72 | 0.90 | 0.65 | 0.96 | 0.85 | 0.84 | 0.73 | 0.89 | 0.67
WavLM | 0.85 | 0.72 | 0.81 | 0.68 | 0.95 | 0.90 | 0.83 | 0.75 | 0.75 | 0.62
HuBERT | 0.90 | 0.83 | 0.91 | 0.75 | 0.97 | 0.93 | 0.86 | 0.81 | 0.93 | 0.75

Table 6: AP scores for the word-discrimination task on the
$\text{test}^{\prime}$ set using SSL-based speech representations as input
features for all five languages. AWE extraction methods include mean pooling
Sanabria et al. (2023), AE-RNN, and CAE-RNN..

###  5.1 Cross-lingual Analysis

To assess the effectiveness of SSL speech representation-based CAE-RNN models
in cross-lingual settings, a CAE-RNN model trained on one source language
(English in this case) is evaluated on four different target languages. This
evaluation can be considered a ‘zero-shot’ evaluation, as no training data
from the target languages is required. Table 7 displays the results in terms
of AP for the word-discrimination task on the test set and
$\text{test}^{\prime}$ set for the four target languages (Polish, Portuguese,
Spanish, and French). In this scenario as well, the HuBERT-based CAE-RNN model
achieves the best performance across all languages, except for French where
Wav2vec2 performs the best. It is worth noting that the CAE-RNN model in the
‘zero-shot’ setting outperforms the mean pooling method (Table 5 and 6)
Sanabria et al. (2023) and the CAE-RNN model trained on the target language
with MFCC features (Table 3 and 4). The mean pooling method Sanabria et al.
(2023) can be considered a ‘zero-shot’ AWE extraction method, as it does not
involve additional training on top of the pre-trained SSL models. In a ‘zero-
shot’ setup for target languages, using a CAE-RNN trained on a well-resourced
source language can offer an advantage over the mean pooling method. In
conclusion, SSL-based CAE-RNN models have fairly good performance when used
crosslingually. There have been earlier studies Kamper et al. (2021) on
acoustic word embeddings for zero-resource languages using multilingual
transfer with MFCC features, which worked well. Also, intuitively, some
generalisation was expected as the aim of modelling is to compress a small
segment of speech into a fixed dimensional vector. There might be a language
effect on pre-trained SSL speech models but the basic speech properties are
still invariant to changes in the language. The cross-lingual ability of SSL-
based CAE-RNN models to obtain AWEs can support many applications such as
speech search, indexing and discovery systems for languages with low-resources
Kamper et al. (2021).

Input Features | Polish | Portuguese | Spanish | French
---|---|---|---|---
test | $\text{test}^{\prime}$ | test | $\text{test}^{\prime}$ | test | $\text{test}^{\prime}$ | test | $\text{test}^{\prime}$
Wav2vec2 | 0.57 | 0.57 | 0.48 | 0.54 | 0.60 | 0.68 | 0.52 | 0.68
WavLM | 0.48 | 0.47 | 0.36 | 0.40 | 0.54 | 0.63 | 0.50 | 0.64
HuBERT | 0.59 | 0.60 | 0.50 | 0.56 | 0.62 | 0.69 | 0.48 | 0.65
MFCC | 0.18 | 0.20 | 0.11 | 0.15 | 0.22 | 0.29 | 0.22 | 0.35

Table 7: AP for the word-discrimination task with CAE-RNN model trained on
English language with various input features and tested on other four
languages.

Word1 | Word2 |  | Cosine Distance
---
(Mean Pooling)
| Cosine Distance
---
(CAE-RNN)
Description
aside | aside | 0.23 | 0.01 | Same word
aside | ideas | 0.56 | 0.99 | Anagram pair
this | this | 0.33 | 0.11 | Same word
this | hits | 0.38 | 0.50 | Anagram pair
no | no | 0.30 | 0.02 | Same word
no | on | 0.63 | 0.69 | Anagram pair

Table 8: Comparison of cosine distances between AWEs of same spoken word pairs
and anagram pairs. HuBERT features are used for both CAE-RNN and mean pooling
method.

Figure 2: t-SNE visualisation of the AWEs derived from HuBERT-based CAE-RNN
model for all five languages. From each language, all spoken instances of the
top 7 words with the highest frequency count from the test set are chosen.

###  5.2 Analysis of Anagram Pairs

Anagrams are words that can be formed by rearranging the letters of another
word. Analysing anagram pairs provides insights into the impact of letter
order on AWE representation. Robust AWEs should capture the letter order in
spoken words. For this analysis, same spoken word pairs and anagram pairs are
chosen from different speakers. Ideally, the cosine distance between the same
word pairs should be close to 0, while anagram word pairs should be close to
1. In Table 8, HuBERT-based CAE-RNN AWEs demonstrate cosine distances of
approximately 0.01, 0.11, and 0.02 for the same spoken word pairs ‘aside’,
‘this’, and ‘no’, respectively. The anagram pairs of the words ‘aside’,
‘this’, and ‘no’ (i.e., ‘ideas’, ‘hits’, and ‘on’) have distances of 0.99,
0.50, and 0.69, respectively, for the HuBERT-based CAE-RNN model. These values
are significantly better for both the same word pairs and anagram word pairs
when compared to the HuBERT-based mean pooling method Sanabria et al. (2023).
This indicates that the HuBERT-based CAE-RNN model accurately captures the
letter order in a word compared to the mean pooling baseline Sanabria et al.
(2023).

###  5.3 AWE Visualisation

t-SNE visualization is used to plot the 2-dimensional representations of the
derived AWEs for all five languages. From each language, all spoken instances
of the top 7 words with the highest frequency count from the test set are
chosen. The plots demonstrate distinct and well-separated clusters for each
spoken word across all languages. One interesting pattern can be observed for
the Polish language, where the clusters of the spoken words ‘Owadów’ and
‘Owady’ share the boundary and are closely related in the AWE space. This is
likely due to the fact that the first four letters of both the words (o, w, a,
d) are shared and these words only differ in their endings.

##  6 Conclusions and Future Work

It has been demonstrated that SSL-based speech representations with CAE-RNN
models outperform mean pooling and AE-RNN models across all languages. They
also outperform MFCC-based models. Among all the SSL models, HuBERT performs
the best when used as input for the CAE-RNN model, outperforming models such
as Wav2vec2 and WavLM. Notably, despite being pre-trained on English data, the
SSL models exhibit excellent performance on other languages, showcasing their
cross-lingual generalization capability for AWE extraction.

Furthermore, quantitative analysis reveals that incorporating context
information of the spoken word leads to more robust AWEs. The HuBERT-based
CAE-RNN model trained on English language and tested on other target languages
outperforms the mean pooling method and the CAE-RNN model trained on the
target language using MFCC features. This ‘zero-shot’ method to obtain robust
AWEs for the target language can be useful in applications for low-resource
languages Jacobs et al. (2023). An analysis was also conducted to show that
the CAE-RNN model effectively captures the order of letters in a word.

In future work, experiments will be conducted with the “LARGE" variation of
SSL models, as well as multilingual pre-trained SSL models such as
Wav2vec2-XLSR Conneau et al. (2020). Additionally, an interesting experiment
would involve training a single universal AWE model on all languages and
comparing its performance with language-specific AWE models. Further research
will focus on measuring the performance gains of SSL-based CAE-RNN models on
downstream tasks such as query-by-example search Settle et al. (2017); Yuan et
al. (2018); Hu et al. (2021) and keyword spotting Shin et al. (2022).

##  7 Limitations

This work is focused on the extraction of AWEs and measuring their quality
solely based on the word discrimination task. No downstream applications such
as query-by-example search and keyword spotting, have been discussed using the
improved AWEs. In this work, only the “BASE” versions of the SSL-based speech
models are explored for experiments and analysis. There are other variations,
such as “LARGE” version, for which this study can be extended. All the
languages considered in this work belong to the Indo-European language family.
This work does not contain the analysis of languages that belong to another
language family, such as Dravidian or Afroasiatic language families. This work
does not deal with layer-wise analysis, which can provide better insights for
further improving the AWEs.

##  8 Acknowledgments

This work was supported by the Centre for Doctoral Training in Speech and
Language Technologies (SLT) and their Applications funded by UK Research and
Innovation [grant number EP/S023062/1]. This work was also funded in part by
LivePerson, Inc.

## References

  * Abdullah et al. (2021a) Badr Abdullah, Iuliia Zaitova, Tania Avgustinova, Bernd Möbius, and Dietrich Klakow. 2021a.  How familiar does that sound? cross-lingual representational similarity analysis of acoustic word embeddings.  In _Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP_ , pages 407–419, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Abdullah et al. (2021b) Badr M. Abdullah, Marius Mosbach, Iuliia Zaitova, Bernd Möbius, and Dietrich Klakow. 2021b.  Do Acoustic Word Embeddings Capture Phonological Similarity? An Empirical Study.  In _Proc. Interspeech 2021_ , pages 4194–4198.
  * Abdullah et al. (2022) Badr M. Abdullah, Bernd Möbius, and Dietrich Klakow. 2022.  Integrating Form and Meaning: A Multi-Task Learning Model for Acoustic Word Embeddings.  In _Proc. Interspeech 2022_ , pages 1876–1880.
  * Baevski et al. (2020) Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020.  wav2vec 2.0: A framework for self-supervised learning of speech representations.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 12449–12460. Curran Associates, Inc.
  * Barakat et al. (2011) M. S. Barakat, C. H. Ritz, and D. A. Stirling. 2011.  Keyword spotting based on the analysis of template matching distances.  In _2011 5th International Conference on Signal Processing and Communication Systems (ICSPCS)_ , pages 1–6.
  * Carlin et al. (2011) Michael A. Carlin, Samuel Thomas, Aren Jansen, and Hynek Hermansky. 2011.  Rapid evaluation of speech representations for spoken term discovery.  In _Proc. Interspeech 2011_ , pages 821–824.
  * Chen et al. (2022) Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng, Xiangzhan Yu, and Furu Wei. 2022.  Wavlm: Large-scale self-supervised pre-training for full stack speech processing.  _IEEE Journal of Selected Topics in Signal Processing_ , 16(6):1505–1518.
  * Conneau et al. (2020) Alexis Conneau, Alexei Baevski, Ronan Collobert, Abdel rahman Mohamed, and Michael Auli. 2020.  Unsupervised cross-lingual representation learning for speech recognition.  In _Interspeech_.
  * Davis and Mermelstein (1980) S Davis and P Mermelstein. 1980.  Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences.  _IEEE Trans. Acoust., Speech, Signal Process._ , 28(4):357–366.
  * He et al. (2017) Wanjia He, Weiran Wang, and Karen Livescu. 2017.  Multi-view recurrent neural acoustic word embeddings.  In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_.
  * Hsu et al. (2021) Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021.  Hubert: Self-supervised speech representation learning by masked prediction of hidden units.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 29:3451–3460.
  * Hu et al. (2021) Yushi Hu, Shane Settle, and Karen Livescu. 2021.  Acoustic span embeddings for multilingual query-by-example search.  In _2021 IEEE Spoken Language Technology Workshop (SLT)_ , pages 935–942.
  * Jacobs et al. (2023) Christiaan Jacobs, Nathanaël Carraz Rakotonirina, Everlyn Asiko Chimoto, Bruce A. Bassett, and Herman Kamper. 2023.  Towards hate speech detection in low-resource languages: Comparing asr to acoustic word embeddings on wolof and swahili.  _ArXiv_ , abs/2306.00410.
  * Kamper (2019) Herman Kamper. 2019.  Truly unsupervised acoustic word embeddings using weak top-down constraints in encoder-decoder models.  _ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 6535–3539.
  * Kamper et al. (2015) Herman Kamper, Micha Elsner, Aren Jansen, and Sharon Goldwater. 2015.  Unsupervised neural network based feature extraction using weak top-down constraints.  In _2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 5818–5822.
  * Kamper et al. (2021) Herman Kamper, Yevgen Matusevych, and Sharon Goldwater. 2021.  Improved acoustic word embeddings for zero-resource languages using multilingual transfer.  _IEEE/ACM Trans. Audio, Speech and Lang. Proc._ , 29:1107–1118.
  * Lin et al. (2023) Jingru Lin, Xianghu Yue, Junyi Ao, and Haizhou Li. 2023.  Self-Supervised Acoustic Word Embedding Learning via Correspondence Transformer Encoder.  In _Proc. INTERSPEECH 2023_ , pages 2988–2992.
  * Matusevych et al. (2020) Yevgen Matusevych, Herman Kamper, and Sharon Goldwater. 2020.  Analyzing autoencoder-based acoustic word embeddings.  _CoRR_ , abs/2004.01647.
  * McAuliffe et al. (2017) Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. 2017.  Montreal forced aligner: Trainable text-speech alignment using kaldi.  In _Interspeech_.
  * Meghanani and Hain (2024) Amit Meghanani and Thomas Hain. 2024.  Score: Self-supervised correspondence fine-tuning for improved content representations.  In _ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_.
  * Panayotov et al. (2015) Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015.  Librispeech: An asr corpus based on public domain audio books.  In _2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 5206–5210.
  * Pratap et al. (2020) Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert. 2020.  Mls: A large-scale multilingual dataset for speech research.  _Interspeech 2020_.
  * Sanabria et al. (2023) Ramon Sanabria, Hao Tang, and Sharon Goldwater. 2023.  Analyzing acoustic word embeddings from pre-trained self-supervised speech models.  In _ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 1–5.
  * Schultz et al. (2013) Tanja Schultz, Ngoc Thang Vu, and Tim Schlippe. 2013.  Globalphone: A multilingual text & speech database in 20 languages.  In _2013 IEEE International Conference on Acoustics, Speech and Signal Processing_ , pages 8126–8130.
  * Settle et al. (2017) Shane Settle, Keith Levin, Herman Kamper, and Karen Livescu. 2017.  Query-by-example search with discriminative neural acoustic word embeddings.  In _INTERSPEECH_.
  * Shin et al. (2022) Hyeon-Kyeong Shin, Hyewon Han, DoYeon Kim, Soo-Whan Chung, and Hong-Goo Kang. 2022.  Learning audio-text agreement for open-vocabulary keyword spotting.  In _Interspeech_.
  * Yang et al. (2021) Shu-Wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Lai, Kushal Lakhotia, Yist Y. Lin, Andy T. Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, Tzu hsien Huang, Wei-Cheng Tseng, Ko tik Lee, Da-Rong Liu, Zili Huang, Shuyan Dong, Shang-Wen Li, Shinji Watanabe, Abdel rahman Mohamed, and Hung yi Lee. 2021.  Superb: Speech processing universal performance benchmark.  In _Interspeech_.
  * Yuan et al. (2018) Yougen Yuan, Cheung-Chi Leung, Lei Xie, Hongjie Chen, Bin Ma, and Haizhou Li. 2018.  Learning Acoustic Word Embeddings with Temporal Context for Query-by-Example Speech Search.  In _Proc. Interspeech 2018_ , pages 97–101.