# Open-vocabulary keyword spotting in any language through multilingual
contrastive speech-phoneme pretraining

Jian Zhu Farhan Samir Changbing Yang Jahurul Islam
University of British Columbia
<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

In this paper, we introduce a massively multilingual speech corpora with fine-
grained phonemic transcriptions, encompassing more than 115 languages from
diverse language families. Based on this multilingual dataset, we propose
CLAP-IPA, a multilingual phoneme-speech contrastive embedding model capable of
open-vocabulary matching between speech signals and phonemically transcribed
keywords or arbitrary phrases. The proposed model has been tested on two
fieldwork speech corpora in 97 unseen languages, exhibiting strong
generalizability across languages. Comparison with a text-based model shows
that using phonemes as modeling units enables much better crosslinguistic
generalization than orthographic texts.

##  1 Introduction

Key Word Spotting (KWS) is a classic task of identifying specific key words in
streaming speech, with primary applications in voice assistants and
interfaces. Traditional KWS systems used to limit to a fixed set of key words
in a certain language. It still remains a challenging task to develop open-
vocabulary KWS systems that can deal with any input, and even more challenging
to develop such a system for any language in the world López-Espejo et al.
(2021). Currently most open-vocabulary KWS models still require a few samples
to adapt to new words and struggles to perform zero-shot adaption. Effort has
been made to multilingual KWS models but they are mostly limited to a dozen of
languages Mazumder et al. (2021a); Lei et al. (2023); Reuter et al. (2023) and
has not been shown to generalize to arbitrary unseen languages effectively.

With the availability of multilingual data and the rapid development of neural
speech models, accumulating evidence indicates that scaling up the
multilingual data can tremendously improve the performance of multilingual
speech processing Conneau et al. (2020); Babu et al. (2021); Radford et al.
(2023). However, multilingual KWS has yet to benefit from the scale of
multilingual data.

To tackle these challenges, we proposed Contrastive Language-Audio Pretraining
with International phonetic Alphabet (CLAP-IPA), a phoneme-to-speech retrieval
model based on the contrastive dual encoder architecture of CLAP Wu et al.
(2023) but pretrained with phoneme-speech pairs. Trained on our carefully
curated multilingual speech corpora with phonemic transcriptions, the proposed
model is capable of performing zeroshot open-vocabulary KWS in any language,
including languages not seen by the model during training.

Our major contributions can be summarized as follows:  1) We have curated one
of the largest phonemically transcribed speech corpora in 115 languages from
public data, examined by trained phoneticians, totaling over 1000 hours;  2)
We proposed CLAP-IPA, a multilingual phoneme-to-speech retrieval model that
works in any language without adaptation;  3) Our analysis indicates that
phonemes serve as more effective modeling units than texts in current
multilingual tasks. Phonemes, being shared across all languages, enhance
knowledge transfer within training data.

In addition to KWS, we envision that CLAP-IPA and the curated multilingual
dataset will benefit more downstream tasks and applications in multilingual
speech processing. To facilitate future research, we will release our dataset,
scripts and pre-trained models at: https://github.com/lingjzhu/clap-ipa.

##  2 Backgrounds

###  2.1 Phoneme-based speech datasets

Most speech corpora are distributed as audio-text pairs, as texts are far more
easier to collect than phonemic transcriptions, the creation of which requires
expert knowledge in phonetics and/or the target language. However, texts are
usually not good representations of human speech, as they vary considerably
across languages and often do not correspond to the actual pronunciation.
Compared to the diversity of orthographic systems, phonemes, or symbols from
the International Phonetic Alphabet (IPA), are compact and universal
representations of all human speech. In fieldwork, many spoken languages,
especially dialects and indigenous languages, do not have an orthographic
writing system, where phonemic transcriptions are the only viable form of
transcription. For multilingual representations, IPA symbols can often be a
better choice, as all languages can be decomposed into a limited and unified
set of symbols.

Recent studies proposed that hidden units learned by self-supervised speech
models or pretrained neural codec can be used as modeling units Nguyen et al.
(2023); Wang et al. (2023). While these hidden units are highly correlated to
phonemes Sicherman and Adi (2023), they are not invariant across systems and
far less interpretable than phonemes Gat et al. (2022).

There are scattering effort to create open-source phoneme-based speech corpora
in linguistics and speech processing community. Most of such corpora are
collected from fieldwork recordings, notably DoReCo Paschen et al. (2020), the
Pangloss Collection Michailovsky et al. (2014) and the UCLA phonetic Corpus Li
et al. (2021). These collections of fieldwork recordings usually consist of
dozens of low-resource languages, most of which are transcribed phonemically.
While these corpora encompass dozens of low-resource languages transcribed
phonemically, the diversity is accompanied by limitations. Most languages are
represented by only a few hours of recordings from a highly restricted pool of
speakers, and phonemic transcriptions are often sparse and inconsistent across
languages. For large-scale phoneme-based corpora, VoxCommunis Ahn and Chodroff
(2022) is another effort to create large-scale a phoneme-based speech corpus
based on 57 languages in the Common Voice dataset Ardila et al. (2020) and the
Epitran G2P system Mortensen et al. (2018). Despite these efforts,
phonemically transcribed speech corpora are still rare and limited in
languages.

###  2.2 Spoken keyword detection and retrieval

KWS is a classic task in spoken language processing, receiving wide attention
from the speech community López-Espejo et al. (2021). Deep neural networks
have become the dominant paradigm in KWS systems since the past decade (e.g.,
Chen et al., 2014; Tang and Lin, 2018; Rybakov et al., 2020; Berg et al.,
2021). However, most of these systems are based on multiclass-classifications
of a set of pre-defined keywords, making them hard to adapt to new keywords
outside training set.

While most research in keyword spotting focus predominantly in English, in
recent years, there have been increased interests in building multilingual
keyword detection systems that can adapt to new words or even new languages
through few-shot learning Mazumder et al. (2021a); Lei et al. (2023); Reuter
et al. (2023). Nevertheless, existing multilingual KWS systems face
limitations in terms of the number of supported languages, and adapting to new
words in different languages often necessitates few-shot adaptations rather
than zero-shot predictions.

Another approach for keyword detection is based on dense retrieval systems
popularized by CLIP Radford et al. (2021) and the subsequent CLAP Wu et al.
(2023) and CLARA Noriy et al. (2023). Speech, text or data in other modalities
sequences are projected into dense vectors in shared latent spaces. Some text-
to-speech dense retrieval systems are based on semantics rather than the
surface acoustic form of speech signals Duquenne et al. (2021); Khurana et al.
(2022); Zhu et al. (2022a), yet this contrastive learning paradigm has also
been applied successfully to build open-vocabulary KWS systems Nishu et al.
(2023).

While texts are the primary modeling units in most systems, there are studies
showing the effectiveness of using IPA symbols to achieve open-vocabulary
generalization Tanaka et al. (2001); Shin et al. (2022); Lee and Cho (2023);
Reuter et al. (2023). Built on these prior efforts, we scaled up the phoneme-
based open-vocabulary KWS models to more languages to achieve crosslinguistic
generalization.

| Train (hrs) | Dev (hrs) | Test (hrs) | Total (hrs) | Languages | Avg. Dur (hrs)
---|---|---|---|---|---|---
FLEURS-Phonemes | 544.02 | 73.46 | 162.06 | 779.54 | 77 | 10.12
MSWC-Phonemes | 485.35 | 64.08 | 64.11 | 613.44 | 36 | 17.04
Doreco-Phonemes | 13.70 | - | 5.29 | 18.99 | 44 | 0.44
Table 1: Detailed statistics of curated speech corpora.

##  3 Dataset curation

High-quality data is essential to model performance, yet phonemically
transcribed speech corpora are rare. So we made endeavors to create a large-
scale phonemic transcriptions for public speech corpora, encompassing 115
languages across language families. Transcribing speech signals into phonemic
transcriptions often require years of expertise in phonetics, making it hard
to create high-quality phonemic transcriptions at scale. This process can be
automated through grapheme-to-phoneme conversion (G2P), a process of
converting orthographic transcriptions into phonemic transcriptions through
pronunciation dictionaries and/or statistical models.

###  3.1 Phonemic transcriptions

We primarily made use of three multilingual speech datasets, FLEURS Conneau et
al. (2023), Multilingual Spoken Words Corpus (MSWC) Mazumder et al. (2021b)
and DoReCo Paschen et al. (2020).

#### FLEURS

We used two multilingual G2P systems, Epitran Mortensen et al. (2018) and
CharsiuG2P Zhu et al. (2022b), to create phonemic transcriptions. As these two
systems cover a overlapping but slightly different set of languages, combining
them allowed to maximize the diversity of languages. Before preprocessing, we
removed any texts with Arabic numbers or texts in non-alphabetic writing
systems but contains alphabetic code-switching, as G2P systems cannot process
them correctly.

Texts in most languages were already normalized and segmented in FLEURS. Yet
some Asian languages require special treatment as their orthographic systems
do not explicitly mark word boundaries using spaces. For Mandarin Chinese,
G2PW Chen et al. (2022) was used to create the Pinyin romanizations from
unsegmented texts, which were then mapped to IPA symbols. For Thai, we used
PyThaiNLP Phatthiyaphaibun et al. (2016) to perform word segmentation and G2P.
For Japanese, the word segmentation were first performed with Fugashi McCann
(2020) before G2P was applied. We had to exclude Khmer data due to the failure
to find a good tool for Khmer word segmentation.

#### MSWC

As MSWC is a word-level speech corpus, creating phonemic transcriptions were
straightforward. CharsiuG2P and Epitran were deployed to transcribe the
orthographic words in selected languages to phonemic sequences. To strike a
balance between diversity and quantity, we limited the minimum frequency of
each word to 10 to filter out low frequency words, as low frequency samples
are prone to error. For words with more than 10 samples, 10 of them will be
randomly selected from the pool of samples for that word. After filtering, we
ended up with 1,611,790 spoken word tokens, amounting to around 448 hours of
speech.

#### DoReCo

The original DoReCo data were distributed as hour-long recordings, so we
segmented them into individual utterances based on the sentence boundaries in
the provided annotations. For DoReCo, all languages were transcribed as
phonemes using X-SAMPA notations. We simply converted the X-SAMPA
transcription to IPA symbols, as there is a one-to-one mapping between these
two systems. Utterances with incomplete transcriptions or loud background
noises were discarded.

###  3.2 Dataset validation

Automatically generated multilingual datasets are often subject to various
quality issues Kreutzer et al. (2022). To ensure the quality of phonemic
transcriptions, trained phoneticians had listened to at least ten random
samples in each language to determine whether phonemic transcriptions and
speech signals were approximately matched. However, we were aware that, as IPA
is not an exact reflection of human speech, a true transcription does not
always exist for every utterance Ladefoged and Halle (1988); Ladefoged (1990);
Pandeli et al. (1997). Even trained phoneticians often disagree on the
phonemic transcriptions of the same utterance, due to factors including
psycho-acoustic constraints, phonetic training, the influence of mother tongue
and language-specific conventions Kerswill and Wright (1990); Raymond et al.
(2002); Pitt et al. (2005); Heselwood (2013).

As G2P systems are based on rules or pronunciation dictionaries, they reflect
how a word should be pronounced rather than how a word is pronounced. Given
the high variability (e.g., phonetic reduction, coarticulation) in speech
signals, it is not always possible for the G2P phonemic transcriptions to
exactly match the audio. We applied a relatively relaxed standard for the
generated transcriptions: as long as the speech signal approximately matches
more than 80% of the transcription, it is considered valid. While we made best
efforts to validate the transcription quality, we acknowledge that there are
still transcriptions errors in the dataset. A summary of the final dataset is
presented at Table 1 and more detailed information on individual languages can
be found at Appendix A.

##  4 Method

###  4.1 Contrastive learning

Given a speech signal $\mathbf{S}=[s_{1},s_{2},\cdots,s_{m}]$ and a phonemic
sequence $\mathbf{P}=[p_{1},p_{2},\cdots,p_{n}]$, $n\leq m$, our task of
phoneme-to-speech retrieval is learn two encoders $f_{S}$ and $f_{T}$ to map
both sequences into fixed-dimensional vectors in the same latent space, such
that it can rank samples based on the similarity function
$sim(f_{S}(\mathbf{P}),f_{T}(\mathbf{S}))$, where $sim(\cdot)$ is usually the
cosine similarity
$sim(\mathbf{x},\mathbf{y})=\frac{\mathbf{xy}}{|\mathbf{x}||\mathbf{y}|}$.

Here we adopt the same contrastive learning framework as CLAP Wu et al.
(2023), as it has been proven to be one of the most effective strategy to
learn high quality cross-modal representations. There are two separate
encoders to process phoneme data $\mathbf{P}$ and speech data $\mathbf{S}$,
transforming them into phoneme embedding and speech embedding. The CLIP loss
function Radford et al. (2021) relies on the softmax-based InfoNCE loss Oord
et al. (2018) to contrast a data sample with its positive and negative
samples, which has been widely adopted for multi-modal or uni-modal
contrastive training. In this study, we use SigLIP, a simpler sigmoid-based
loss that has been shown to as effective as the softmax-based CLIP loss Zhai
et al. (2023). Given $\boldsymbol{x}_{i}=f_{S}(\mathbf{P}_{i})$ and
$\boldsymbol{y}_{i}=f_{T}(\mathbf{S}_{i})$, it is defined as follows.

$\mathcal{L}=-\frac{1}{|\mathcal{B}|}\sum_{i=1}^{|\mathcal{B}|}\sum_{j=1}^{|\mathcal{B}|}\underbrace{\log\frac{1}{1+e^{z_{ij}(-t\mathbf{x}_{i}\cdot\mathbf{y}_{j}+b)}}}_{\mathcal{L}_{ij}}$
(1)

where $t$ and $b$ are learnable parameters that were updated during training.
$z_{ij}$ is the ground truth label, $z_{ij}=1$ for positive pairs and
$z_{ij}=0$ for negative pairs. Following the recommendation by Zhai et al.
(2023), we initialized $t=\log 10$ and $b=-10$.

###  4.2 Speech encoder

The speech encoder has the same transformer encoder architecture as the
Whisper's encoder. The weights were initialized with Whisper's pretrained
encoder weights, whereas the decoder was discarded. The original Whisper
encoder does not accept attention masks, but these padding tokens can biases
the model during pooling. So attention masks were also passed to the speech
encoder and the final fixed-dimensional embedding through self-attention
pooling on non-padded hidden states. Self-attention pooling has been shown to
outperform mean and max pooling for speech embeddings. For speech data
augmentation, SpecAugment Park et al. (2019) was applied during training using
the default hyperparameters in Whisper training.

###  4.3 Phoneme encoder

As the set of IPA symbols are small, we trained a specialized tokenizer for
IPA symbols, including all base IPA symbols as well as diacritics including
tonal notations, stress marks and tie bars for affricates. Upon inspection, we
noticed that the tie bars were inconsistently labeled in our source datasets
(e.g., [tS] vs. [t͡S]) and stress marks tend to be a language-specific
phenomenon Gordon and Roettger (2017). Yet we preserved these idiosyncratic
labels to enhance the diversity of our data, so that the final phoneme encoder
can accept the highly variable inputs in practice. All other non-IPA
characters were removed from our data.

The phoneme tokenizer was trained using the unigram algorithm Kudo (2018) with
sentencepiece package 111https://github.com/google/sentencepiece. The
tokenizer was trained on all phonemic transcriptions in our datasets, with a
vocabulary of 500 and byte-fallback for unknown characters. As our data are
exclusively phonemes, a specialized tokenizer would be more conducive to the
current task and more parameter efficient.

For the phoneme encoder, we used the BERT architecture Devlin et al. (2019)
with mean pooling of the final hidden states as the fixed-dimensional
representation. The phoneme encoder was pretrained on a corpus of phonemic
transcriptions using standard masked language modeling (MLM) as detailed in
Devlin et al. (2019). Given that phoneme sequences are of less complexity than
texts, the masking probablity was set to 30%. The training data were pooled
from diverse sources, including FLEURS-P, MSWC-P, DoReCo-P, pronunciation
dictionaries in CharsiuG2P Zhu et al. (2022b), and Vox Communis Ahn and
Chodroff (2022). The final pretraining corpus consists of 11 million samples
in more than 110 languages. We pretrained three phoneme encoders of different
sizes, matching hyperparameters including number of layers, hidden dimensions,
and number of attention heads to the corresponding Whisper encoder (tiny, base
and small).

###  4.4 Phonetically informed hard negatives

Negative samples are critical to contrastive learning. Part of CLIP's success
can be attributed to the large training batch sizes and the resulting large
in-batch negative samples Radford et al. (2021); Cherti et al. (2023). Given
our hardware constraints, we instead constructed high-quality hard negatives
to compensate for the small in-batch negatives.

IPA transcriptions are supposed to be invariant symbolic representations of
speech signals, so we can create hard negatives that are highly similar to the
original pronunciation by manipulating the IPA symbols. In phonetics and
phonology, a minimal pair refers to a pair of word that only differ in one
phoneme in pronunciation Ladefoged and Johnson (2014), like [bæt] vs. [bæd].
To generate hard negatives, for a phonemic sequence of length $L$, we randomly
select $k$ phonemes ($0<k<L$), and implemented three operations, inserting a
new phoneme, deleting the existing phoneme and replacing the existing phoneme
with a new phoneme, which was determined on-the-fly with the same probability
during training. An independent contemporary work also adopts the same
strategy to generate hard negatives for KWS with effective improvement Nishu
et al. (2023).

For word-level training, we considered minimal pairs as natural hard negatives
with $k=1$, which can be easily constructed by manipulating the ground truth
transcription. For sentence-level training, however, constructing such minimal
pairs is almost possible, given that the longer sentence transcriptions are
more likely to contain inaccurate phonemes. We relaxed the standard for
sentence-level hard negatives by using a corruption rate of 10% such that
$k=\lfloor{0.1L}\rfloor$.

Method Data Phonemes $\rightarrow$Speech Speech$\rightarrow$Phonemes
Speech$\rightarrow$Speech EER(%) $\downarrow$ AUC(%) $\uparrow$ EER(%)
$\downarrow$ AUC(%) $\uparrow$ EER(%) $\downarrow$ AUC(%) $\uparrow$ CMCD Shin
et al. (2022) LPE 8.42 96.7 - - - - LPH 32.90 73.58 - - - - PhonMatchNet Lee
and Cho (2023) LPE 2.80 99.29 - - - - LPH 18.82 88.52 - - - - CED Nishu et al.
(2023) LPE 1.7 99.84 - - - - LPH 14.4 92.7 - - - - CLAP-IPA-base (text) LPE
6.0 98.31 6.29 98.19 3.43 99.45 LPH 31.14 74.8 28.61 78.48 25.67 81.9 CLAP-
IPA-base LPE 1.3 99.88 1.37 99.88 1.95 99.79 LPH 23.03 84.58 22.87 85.06 24.23
83.52 CLAP-IPA-tiny LPE 1.32 99.89 1.37 99.89 2.37 99.7 LPH 24.09 83.75 23.16
84.72 24.87 83.87 CLAP-IPA-base LPE 1.16 99.91 1.18 99.9 1.9 99.79 LPH 22.98
84.61 22.4 85.49 24.51 83.3 CLAP-IPA-small LPE 1.48 99.88 1.41 99.89 2.17
99.74 LPH 23.63 84.25 23.09 84.96 24.13 83.77

Table 2: Results on the Libriphrase Evaluation set

Model Data Phonemes$\rightarrow$Speech Speech$\rightarrow$Phonemes
Speech$\rightarrow$Speech Hit@1 $\uparrow$ mAP $\uparrow$ Hit@1 $\uparrow$ mAP
$\uparrow$ Hit@1 $\uparrow$ mAP $\uparrow$ CLAP-IPA-base (no-hard-neg-text)
MSWC-P 13.51 12.7 12.01 17.46 75.62 63.15 CLAP-IPA-base (no-hard-neg-phonemes)
MSWC-P 79.28 68.74 67.51 75.81 77.24 66.25 CLAP-IPA-tiny MSWC-P 81.98 72.76
71.59 79.5 79.52 70.25 CLAP-IPA-base MSWC-P 81.68 72.76 72.43 79.87 79.64
70.19 CLAP-IPA-small MSWC-P 81.38 72.32 71.59 79.54 79.94 70.95 CLAP-IPA-base
(no-hard-neg-text) FLEURS-P 8.48 10.22 9.18 11.27 85.78 86.54 CLAP-IPA-base
(no-hard-neg-phonemes) FLEURS-P 86.4 87.4 85.34 89.24 91.25 91.7 CLAP-IPA-tiny
FLEURS-P 95.76 95.9 94.43 96.17 96.02 96.33 CLAP-IPA-base FLEURS-P 95.76 96.32
94.79 96.44 97.61 97.66 CLAP-IPA-small FLEURS-P 95.23 95.71 95.14 96.79 97.79
97.99

Table 3: Evaluation results on five unseen languages.

##  5 Experiments

###  5.1 Training details

We trained three variants of models, CLAP-IPA-tiny, CLAP-IPA-base and CLAP-
IPA-small, all of them were matched to the default encoder parameters of
Whisper models under the same tiny, base and small suffixes. The speech
encoder and phoneme encoder were symmetric, or exactly matched in
hyperparameters. For the speech encoder, SpecAugment Park et al. (2019) was
enabled with default parameters as in HuggingFace Whisper checkpoints. Our
training dataset included the training set of FLEURS-P, MSWC-P and DoReCo-P
corpora. By default, all models were trained with paired speech recordings and
their phonemic transcriptions. All models were trained for 100k iterations
with the Adam optimizer Kingma and Ba (2015); Loshchilov and Hutter (2018)
with the same hypterparameters.

For controlled comparison between phoneme-based and text-based models, we also
trained two base models on the same FLEURS-P and MSWC-P subset but with only
phonemic or text transcriptions. These two models were matched in total
parameters, training data and all other hyperparameters during training.

###  5.2 Evaluation datasets

To evaluate the model's performance in crosslinguistic retrieval tasks, we
made use of several evaluation datasets covering a wide range of topologically
diverse languages.

#### Libriphrase

To compared with existing models, we first tested on a popular KWS dataset,
Libriphrase Shin et al. (2022), as an out-of-domain evaluation dataset, since
our models were not trained on their training sets. We used Equal Error Rate
(EER) and the Area under Curve (AUC) scores to compare model performance,
consistent with common practices.

#### Unseen languages

We also evaluated all models on five unseen languages with typological
diversity from FLEURS-P and MSWC-P. We isolated five language from MSWC-P and
FLEURS-P, namely, Vienamese (vie), Tamil (tam), Hausa (hau), Georgian (geo)
and Odia (ori). For FLEURS-P, the test sets of these five languages were
directly used. However, for MSWC-P, due to data scarcity, we pooled all
training, validation and tests of these five languages together to form a
larger and more challenging benchmark. Hit@1 and mean average precision (mAP)
were used to measure the cross-linguistic retrieval performance of all models.

#### Fieldwork languages

To further validate the crosslinguistic generalizability of the proposed
models, we tested them on 97 unseen languages from linguistic fieldwork
recordings. We used the UCLA phonetic Corpus Li et al. (2021) as a word-level
test set and a subset of the DoReCo corpus Paschen et al. (2020) as an
utterance-level test set. All together there were 97 unseen languages in these
two test sets, which makes them suitable for evaluating the proposed model's
capacity of crosslinguistic generalizability. Given that we did not have
multiple samples for the same word or utterance, we only evaluated on phoneme-
to-speech and speech-to-phoneme retrieval.

##  6 Findings

In this section, we summarize our major findings of our dataset construction
and CLAP-IPA experiments.

Model | UCLA | DoReCo
---|---|---
Phonemes$\rightarrow$Speech | Speech$\rightarrow$Phonemes | Phonemes$\rightarrow$Speech | Speech$\rightarrow$Phonemes
Hit@1 | mAP | Hit@1 | mAP | Hit@1 | mAP | Hit@1 | mAP
CLAP-IPA-tiny | 39.19 | 50.19 | 40.26 | 51.86 | 78.91 | 83.84 | 78.96 | 83.68
CLAP-IPA-base | 45.36 | 56.52 | 45.59 | 56.99 | 82.59 | 86.79 | 82.5 | 86.52
CLAP-IPA-small | 49.67 | 60.35 | 49.82 | 60.82 | 82.04 | 86.38 | 81.98 | 86.22
\- No pretraining |  |  |  |  |  |  |  |
CLAP-IPA-tiny | 44.75 | 55.87 | 45.13 | 56.04 | 74.36 | 80.26 | 74.14 | 80.04
CLAP-IPA-base | 46.51 | 57.3 | 46.8 | 57.78 | 77.24 | 82.75 | 77.55 | 82.79
CLAP-IPA-small | 47.85 | 58.84 | 48.42 | 59.63 | 82.49 | 86.96 | 82.27 | 86.71
\- No hard negatives |  |  |  |  |  |  |  |
CLAP-IPA-base | 39.91 | 51.11 | 41.18 | 52.39 | 68.0 | 74.9 | 67.71 | 74.57
Table 4: Evaluation results for fieldwork speech corpora

#### Multilingual models might not hold an apparent advantage over well-
engineered monolingual models in high-resource languages.

As shown in Table 2, compared to other state-of-the-art KWS models, Cross-
Modal Correspondence Detector (CMCD) Shin et al. (2022), PhonMatchNet Lee and
Cho (2023), and Common Embedding based Detector (CED) Nishu et al. (2023),
CLAP-IPA models generally achieve better performance in Libriphrase-Easy test
set, though they had not been trained on the Libriphrase training set. While
CLAP-IPA has not surpassed PhonMatchNet and CED in Libriphrase-Hard, its
performance remains competitive.

However, these prior KWS models are late-interaction based models, which are
extremely inefficient for large-scale retrieval among a large number of
candidates, as each unique combination of query-candidate pairs must be passed
to the model to compute similarity scores. For CLAP-IPA, however, most of the
representations in a speech corpora can be precomputed and stored in vector
databases to facilitate ranking and retrieval. CLAP-IPA not only can perform
phoneme-to-speech retrieval but also speech-to-speech retrieval, thereby being
more versatile.

Query | Output Type | Retrieved candidates (ranked from high to low)
---|---|---
étá | Most similar | Ètá, éta, lāta, étá, ìtà, ait:a, mÈtá, me∥[ta, p͡tá, aita, ÈtÉ, atsa, àtá, eit@
| Most dissimilar | bÕ*̇ti, tju:riS, sorNgi, aBu”Ru, mbúruù, sumbuN, buluz, SìZìZ, tSungu
Table 5: Sample output of Speech$\rightarrow$phonemes retrieval by CLAP-IPA-
small, given the query speech [étá].

#### Phoneme-based modeling is critical to the crosslinguistic
generalizability of CLAP-IPA models.

Table 3 and 4 indicates that the proposed model exhibits strong generalization
capabilities to unseen languages. Generally speaking, all phoneme-based models
perform better on utterance-level datasets (FLEURS-P and DoReCo-P) than on
word-level datasets (MSWC-P and UCLA Phonetic Corpus), because the longer the
phoneme sequence, the more likely that it is distinct in a pool of candidates.
Text-based models are struggling to generalize to unseen languages, as these
unseen languages have their own distinct writing systems (e.g, Vienamese and
Tamil) that are not seen in training languages. Comparison between the text-
based and phoneme-based models in Table 2 and Table 3 clearly shows that it is
the use of phonemes as modeling units that enables the crosslinguistic
generalizability, since they can represent all languages using the same set of
symbols.

We also analyzed the model performance qualitatively. The example presented in
Table 5 shows the ranking of retrieved phonemic transcriptions (among 5000
potential candidates in 95 languages) with respect to the speech query [étá]
in the UCLA Phonetic Corpus. The similarity assigned by CLAP-IPA-small was
highly consistent with human perception. The top-ranked candidates shared the
same number of syllables and extremely similar syllable structure as the
query, and captured the articulatory features of the query extremely well even
across languages.

Figure 1: Correlation of model performance on individual languages with
training hours by language. Languages are represented by their ISO 639-3
codes.

#### The number of training hours of individual languages does not predict the
performance by language in phoneme-based models. All languages, regardless of
being high-resource or low-resource, benefits from the multilingual knowledge
transfer in phoneme-based modeling.

It has been reported that there is a strong correlation between text-based
multilingual ASR performance on individual languages and training hours of
individual languages Radford et al. (2023); Rouditchenko et al. (2023). We
conducted similar analysis on the relationship between the amount of training
data and the retrieval performance for all languages in the test set of
FLEURS-P. We also confirm this observations for the text-based model, as there
is a moderate correlation between Hit@1 and the amount of training hours
(Spearman's $\rho:0.42$; $p\leq 0.0002$). However, for the phoneme-based
model, there is no significant correlation between training hours and final
performance by language (Spearman's $\rho:0.14$; $p=0.22$). As shown in Figure
1, the phoneme-based model outperforms the text-based model in every language
by a large margin, especially for languages with less training data.

Since the orthography varies across languages and is usually not an accurate
reflection of pronunciation, many low-resource languages are not reaping the
full benefits of large-scale multilingual data in this cross-modal task in
text-based models. Close inspection shows that, for the five zero-shot
languages, the text-based model generalizes well to Hausa (Latin alphabet) but
cannot process Vienamese and Tamil at all. For non-zero-shot languages, the
text-based models also significantly underperforms in languages with non-Latin
alpahbet, such as Japanese, Arabic and Cantonese.

In contrast, the phoneme-based model is capable of making full use of the
commonalities shared by human speech. Regardless of the amount of training
data, the phoneme-based model achieves near perfect performance in retrieval
in almost all seen languages (see Figure 1). Even for unseen languages in a
zero-shot setting, phoneme-based models can generalize successfully
crosslinguistically, making them extremely useful not only in low-resource but
also in zero-resource scenarios. Similar observation about the efficiency of
IPA representations in multilingual settings has been made in another study
Feng et al. (2023).

#### Hard negatives are critical to improving crosslinguistic retrieval, while
pretraining provides limited, inconsistent benefits.

Table 3 and 4 suggests that, despite the simplicity of the method, models
trained with phonetically informed hard negatives have significant improvement
over those without, as hard negatives force models to learn to distinguish the
nuances in pronunciation. However, pretrained the phoneme encoders on phonemic
transcripts did not always help but might be adverse to performance in some
cases (see Table 4). Our hypothesis is that, compared to orthographic texts,
phonemic sequences are limited in the number of symbols, simple in structure
and unambiguous in their meanings, making them easy to learn even without
large-scale pretraining.

#### Creating high-quality phonemic transcriptions remains extremely
challenging for most world languages.

During our dataset construction, we were unable to process many languages due
to the lack of pronunciation dictionaries, text transcriptions, or relevant
NLP tools (e.g., the lack of good word segmentation tools for some East/South
East Asian languages). While there are large-scale speech corpora encompassing
more than 1000 languages Salesky et al. (2020); Pratap et al. (2023), textual
or phonemic labels cannot be easily obtained for most of them, limiting their
usage in many research applications. Our effort marks a small step in creating
the high-quality phonemic transcriptions to more languages, yet much is still
needed to be done for more languages in the world.

##  7 Conclusions

In this study, we introduce a phonemically transcribed multilingual speech
corpus with human validation as well as CLAP-IPA, a contrastive phoneme-to-
speech model that can perform phoneme-to-speech and speech-to-speech
retrieval. Evaluation results show that using IPA symbols as modeling units
can effectively enable CLAP-IPA to generalize to unseen languages,
highlighting the benefits of incorporating linguistic knowledge with deep
learning methods.

While we only focus on KWS in the current study, we believe that the
multilingual dataset has great potentials to benefit other related tasks in
multilingual speech processing, such as multilingual phoneme recognition and
multilingual speech synthesis. In the future, we will continue to create
phonemic transcriptions for more data in more diverse languages to scale up
the data size and diversity and we will also explore more efficient
architectures to perform the given task.

##  8 Limitations

Our study is still limited in several aspects. First, the proposed models are
still not optimized in terms of computational efficiency. Since most KWS
applications are running on mobile devices with limited computational power,
the proposed models still have too many model parameters to run efficiently on
mobile devices. Moreover, speech sequences are usually much longer than text
sequence. Self-attention with quadratic complexity might not be the most
suitable architecture for processing speech. More efforts are needed to make
such multilingual models are efficient. Secondly, the number of languages
studied in our paper is still limited and might be biased towards languages
that are relatively high-resource. They are not representative of the global
language landscape. There are much more low-resource or endangered languages
we are not able to include due to the lack of various resources. To promote
linguistic inclusion and fairness, we will continue to improve the language
diversity of our research in the future.

## Acknowledgements

This research was enabled in part through the computational resources and
services provided by Advanced Research Computing at the University of British
Columbia and in part by support provided by the Digital Research Alliance of
Canada.

## References

  * Ahn and Chodroff (2022) Emily Ahn and Eleanor Chodroff. 2022.  VoxCommunis: A corpus for cross-linguistic phonetic analysis.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 5286–5294, Marseille, France. European Language Resources Association.
  * Ardila et al. (2020) Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. 2020\.  Common voice: A massively-multilingual speech corpus.  In _Proceedings of the Twelfth Language Resources and Evaluation Conference_ , pages 4218–4222, Marseille, France. European Language Resources Association.
  * Avanzi et al. (2022) Mathieu Avanzi, Marie-José Béguelin, Gilles Corminboeuf, Federica Diémoz, and Laure Anne Johnsen. 2022.  French (Swiss) DoReCo dataset.  In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Université Lyon 2), Berlin & Lyon.
  * Aznar (2022) Jocelyn Aznar. 2022.  Nisvai DoReCo dataset.  In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Université Lyon 2), Berlin & Lyon.
  * Babu et al. (2021) Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, et al. 2021\.  Xls-r: Self-supervised cross-lingual speech representation learning at scale.  _arXiv preprint arXiv:2111.09296_.
  * Berg et al. (2021) Axel Berg, Mark O’Connor, and Miguel Tairum Cruz. 2021.  Keyword Transformer: A Self-Attention Model for Keyword Spotting.  In _Proc. Interspeech 2021_ , pages 4249–4253.
  * Bogomolova et al. (2022) Natalia Bogomolova, Dmitry Ganenkov, and Nils Norman Schiborr. 2022.  Tabasaran DoReCo dataset.  In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Université Lyon 2), Berlin & Lyon.
  * Burenhult (2022) Niclas Burenhult. 2022.  Jahai DoReCo dataset.  In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Université Lyon 2), Berlin & Lyon.
  * Chen et al. (2014) Guoguo Chen, Carolina Parada, and Georg Heigold. 2014.  Small-footprint keyword spotting using deep neural networks.  In _2014 IEEE international conference on acoustics, speech and signal processing (ICASSP)_ , pages 4087–4091. IEEE.
  * Chen et al. (2022) Yi-Chang Chen, Yu-Chuan Steven, Yen-Cheng Chang, and Yi-Ren Yeh. 2022.  g2pW: A Conditional Weighted Softmax BERT for Polyphone Disambiguation in Mandarin.  In _Proc. Interspeech 2022_ , pages 1926–1930.
  * Cherti et al. (2023) Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. 2023\.  Reproducible scaling laws for contrastive language-image learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2818–2829.
  * Cobbinah (2022) Alexander Yao Cobbinah. 2022.  Baïnounk Gubëeher DoReCo dataset.  In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Université Lyon 2), Berlin & Lyon.
  * Conneau et al. (2020) Alexis Conneau, Alexei Baevski, Ronan Collobert, Abdelrahman Mohamed, and Michael Auli. 2020.  Unsupervised cross-lingual representation learning for speech recognition.  _arXiv preprint arXiv:2006.13979_.
  * Conneau et al. (2023) Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna. 2023.  Fleurs: Few-shot learning evaluation of universal representations of speech.  In _2022 IEEE Spoken Language Technology Workshop (SLT)_ , pages 798–805. IEEE.
  * Cowell (2022) Andrew Cowell. 2022.  Arapaho DoReCo dataset.  In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Université Lyon 2), Berlin & Lyon.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Duquenne et al. (2021) Paul-Ambroise Duquenne, Hongyu Gong, and Holger Schwenk. 2021.  Multimodal and multilingual embeddings for large-scale speech mining.  _Advances in Neural Information Processing Systems_ , 34:15748–15761.
  * Däbritz et al. (2022) Chris Lasse Däbritz, Nina Kudryakova, Eugénie Stapert, and Alexandre Arkhipov. 2022.  Dolgan DoReCo dataset.  In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Université Lyon 2), Berlin & Lyon.
  * Döhler (2022) Christian Döhler. 2022.  Komnzo DoReCo dataset.  In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Université Lyon 2), Berlin & Lyon.
  * Feng et al. (2023) Siyuan Feng, Ming Tu, Rui Xia, Chuanzeng Huang, and Yuxuan Wang. 2023.  Language-Universal Phonetic Representation in Multilingual Speech Pretraining for Low-Resource Speech Recognition.  In _Proc. INTERSPEECH 2023_ , pages 1384–1388.
  * Forker and Schiborr (2022) Diana Forker and Nils Norman Schiborr. 2022.  Sanzhi Dargwa DoReCo dataset.  In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Université Lyon 2), Berlin & Lyon.
  * Franjieh (2022) Michael Franjieh. 2022.  Fanbyak DoReCo dataset.  In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Université Lyon 2), Berlin & Lyon.
  * Garcia-Laguia (2022) Alexandro Garcia-Laguia. 2022.  Northern Alta DoReCo dataset.  In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Université Lyon 2), Berlin & Lyon.
  * Gat et al. (2022) Itai Gat, Felix Kreuk, Ann Lee, Jade Copet, Gabriel Synnaeve, Emmanuel Dupoux, and Yossi Adi. 2022.  On the robustness of self-supervised representations for spoken language modeling.  _arXiv preprint arXiv:2209.15483_.
  * Gippert (2022) Jost Gippert. 2022.  Svan DoReCo dataset.  In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Université Lyon 2), Berlin & Lyon.
  * Gordon and Roettger (2017) Matthew Gordon and Timo Roettger. 2017.  Acoustic correlates of word stress: A cross-linguistic survey.  _Linguistics Vanguard_ , 3(1):20170007.
  * Griscom (2022) Richard Griscom. 2022.  Asimjeeg Datooga DoReCo dataset.  In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Université Lyon 2), Berlin & Lyon.
  * Gusev et al. (2022) Valentin Gusev, Tiina Klooster, Beáta Wagner-Nagy, and Alexandre Arkhipov. 2022\.  Kamas DoReCo dataset.  In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Université Lyon 2), Berlin & Lyon.
  * Güldemann et al. (2022) Tom Güldemann, Martina Ernszt, Sven Siegmund, and Alena Witzlack-Makarevich. 2022\.  N$||$ng DoReCo dataset.  In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Université Lyon 2), Berlin & Lyon.
  * Haig et al. (2022) Geoff Haig, Maria Vollmer, and Hanna Thiele. 2022.  Northern Kurdish (Kurmanji) DoReCo dataset.  In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Université Lyon 2), Berlin & Lyon.
  * Harvey (2022) Andrew Harvey. 2022.  Gorwaa DoReCo dataset.  In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Université Lyon 2), Berlin & Lyon.
  * Heselwood (2013) Berry Heselwood. 2013.  _Phonetic Transcription in Theory and Practice_.  Edinburgh University Press.
  * Kazakevich and Klyachko (2022) Olga Kazakevich and Elena Klyachko. 2022.  Evenki DoReCo dataset.  In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Université Lyon 2), Berlin & Lyon.
  * Kerswill and Wright (1990) Paul Kerswill and Susan Wright. 1990.  The validity of phonetic transcription: Limitations of a sociolinguistic research tool.  _Language variation and change_ , 2(3):255–275.
  * Khurana et al. (2022) Sameer Khurana, Antoine Laurent, and James Glass. 2022.  Samu-xlsr: Semantically-aligned multimodal utterance-level cross-lingual speech representation.  _IEEE Journal of Selected Topics in Signal Processing_ , 16(6):1493–1504.
  * Kim (2022) Soung-U Kim. 2022.  Jejuan DoReCo dataset.  In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Université Lyon 2), Berlin & Lyon.
  * Kingma and Ba (2015) Diederik Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _International Conference on Learning Representations (ICLR)_ , San Diega, CA, USA.
  * Kreutzer et al. (2022) Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoît Sagot, Clara Rivera, Annette Rios, Isabel Papadimitriou, Salomey Osei, Pedro Ortiz Suarez, Iroro Orife, Kelechi Ogueji, Andre Niyongabo Rubungo, Toan Q. Nguyen, Mathias Müller, André Müller, Shamsuddeen Hassan Muhammad, Nanda Muhammad, Ayanda Mnyakeni, Jamshidbek Mirzakhalov, Tapiwanashe Matangira, Colin Leong, Nze Lawson, Sneha Kudugunta, Yacine Jernite, Mathias Jenny, Orhan Firat, Bonaventure F. P. Dossou, Sakhile Dlamini, Nisansa de Silva, Sakine Çabuk Ballı, Stella Biderman, Alessia Battisti, Ahmed Baruwa, Ankur Bapna, Pallavi Baljekar, Israel Abebe Azime, Ayodele Awokoya, Duygu Ataman, Orevaoghene Ahia, Oghenefego Ahia, Sweta Agrawal, and Mofetoluwa Adeyemi. 2022.  Quality at a glance: An audit of web-crawled multilingual datasets.  _Transactions of the Association for Computational Linguistics_ , 10:50–72.
  * Krifka (2022) Manfred Krifka. 2022.  Daakie DoReCo dataset.  In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Université Lyon 2), Berlin & Lyon.
  * Kudo (2018) Taku Kudo. 2018.  Subword regularization: Improving neural network translation models with multiple subword candidates.  _arXiv preprint arXiv:1804.10959_.
  * Ladefoged (1990) Peter Ladefoged. 1990.  The revised international phonetic alphabet.  _Language_ , 66(3):550–552.
  * Ladefoged and Halle (1988) Peter Ladefoged and Morris Halle. 1988.  Some major features of the international phonetic alphabet.  _Language_ , 64(3):577–582.
  * Ladefoged and Johnson (2014) Peter Ladefoged and Keith Johnson. 2014.  _A course in phonetics_.  Cengage learning.
  * Lee and Cho (2023) Yong-Hyeok Lee and Namhyun Cho. 2023.  PhonMatchNet: Phoneme-Guided Zero-Shot Keyword Spotting for User-Defined Keywords.  In _Proc. INTERSPEECH 2023_ , pages 3964–3968.
  * Lei et al. (2023) Lei Lei, Guoshun Yuan, Hongjiang Yu, Dewei Kong, and Yuefeng He. 2023.  Multilingual customized keyword spotting using similar-pair contrastive learning.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_.
  * Li et al. (2021) Xinjian Li, David R Mortensen, Florian Metze, and Alan W Black. 2021.  Multilingual phonetic dataset for low resource speech recognition.  In _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 6958–6962. IEEE.
  * López-Espejo et al. (2021) Iván López-Espejo, Zheng-Hua Tan, John HL Hansen, and Jesper Jensen. 2021\.  Deep spoken keyword spotting: An overview.  _IEEE Access_ , 10:4169–4199.
  * Loshchilov and Hutter (2018) Ilya Loshchilov and Frank Hutter. 2018.  Decoupled weight decay regularization.  In _International Conference on Learning Representations_.
  * Mazumder et al. (2021a) Mark Mazumder, Colby Banbury, Josh Meyer, Pete Warden, and Vijay Janapa Reddi. 2021a.  Few-Shot Keyword Spotting in Any Language.  In _Proc. Interspeech 2021_ , pages 4214–4218.
  * Mazumder et al. (2021b) Mark Mazumder, Sharad Chitlangia, Colby Banbury, Yiping Kang, Juan Manuel Ciro, Keith Achorn, Daniel Galvez, Mark Sabini, Peter Mattson, David Kanter, et al. 2021b.  Multilingual spoken words corpus.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_.
  * McCann (2020) Paul McCann. 2020.  fugashi, a tool for tokenizing Japanese in python.  In _Proceedings of Second Workshop for NLP Open Source Software (NLP-OSS)_ , pages 44–51, Online. Association for Computational Linguistics.
  * Michailovsky et al. (2014) Boyd Michailovsky, Martine Mazaudon, Alexis Michaud, Séverine Guillaume, Alexandre François, and Evangelia Adamou. 2014.  Documenting and researching endangered languages: the pangloss collection.  _Language Documentation & Conservation_, 8:119–135.
  * Michaud (2022) Alexis Michaud. 2022.  Yongning Na DoReCo dataset.  In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Université Lyon 2), Berlin & Lyon.
  * Mortensen et al. (2018) David R Mortensen, Siddharth Dalmia, and Patrick Littell. 2018.  Epitran: Precision g2p for many languages.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_.
  * Mosel (2022) Ulrike Mosel. 2022.  Teop DoReCo dataset.  In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Université Lyon 2), Berlin & Lyon.
  * Nguyen et al. (2023) Tu Anh Nguyen, Eugene Kharitonov, Jade Copet, Yossi Adi, Wei-Ning Hsu, Ali Elkahky, Paden Tomasello, Robin Algayres, Benoît Sagot, Abdelrahman Mohamed, and Emmanuel Dupoux. 2023.  Generative spoken dialogue language modeling.  _Transactions of the Association for Computational Linguistics_ , 11:250–266.
  * Nishu et al. (2023) Kumari Nishu, Minsik Cho, Paul Dixon, and Devang Naik. 2023.  Flexible keyword spotting based on homogeneous audio-text embedding.  _arXiv preprint arXiv:2308.06472_.
  * Noriy et al. (2023) Kari A Noriy, Xiaosong Yang, Marcin Budka, and Jian Jun Zhang. 2023.  Clara: Multilingual contrastive learning for audio representation acquisition.  _arXiv preprint arXiv:2310.11830_.
  * Oord et al. (2018) Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018.  Representation learning with contrastive predictive coding.  _arXiv preprint arXiv:1807.03748_.
  * O'Shannessy (2022a) Carmel O'Shannessy. 2022a.  Light Warlpiri DoReCo dataset.  In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Université Lyon 2), Berlin & Lyon.
  * O'Shannessy (2022b) Carmel O'Shannessy. 2022b.  Warlpiri DoReCo dataset.  In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Université Lyon 2), Berlin & Lyon.
  * Ozerov (2022) Pavel Ozerov. 2022.  Anal DoReCo dataset.  In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Université Lyon 2), Berlin & Lyon.
  * Pandeli et al. (1997) Helen Pandeli, Joseph F Eska, Martin J Ball, and Joan Rahilly. 1997.  Problems of phonetic transcription: the case of the hiberno-english slit-t.  _Journal of the International Phonetic Association_ , 27(1-2):65–75.
  * Park et al. (2019) Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le. 2019.  Specaugment: A simple data augmentation method for automatic speech recognition.  _Interspeech 2019_.
  * Paschen et al. (2020) Ludger Paschen, François Delafontaine, Christoph Draxler, Susanne Fuchs, Matthew Stave, and Frank Seifart. 2020.  Building a time-aligned cross-linguistic reference corpus from language documentation data (doreco).  In _Proceedings of the Twelfth Language Resources and Evaluation Conference_ , pages 2657–2666.
  * Phatthiyaphaibun et al. (2016) Wannaphong Phatthiyaphaibun, Korakot Chaovavanich, Charin Polpanumas, Arthit Suriyawongkul, Lalita Lowphansirikul, and Pattarawat Chormai. 2016.  PyThaiNLP: Thai Natural Language Processing in Python.
  * Pitt et al. (2005) Mark A Pitt, Keith Johnson, Elizabeth Hume, Scott Kiesling, and William Raymond. 2005.  The buckeye corpus of conversational speech: Labeling conventions and a test of transcriber reliability.  _Speech Communication_ , 45(1):89–95.
  * Ponsonnet (2022) Maïa Ponsonnet. 2022.  Dalabon DoReCo dataset.  In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Université Lyon 2), Berlin & Lyon.
  * Pratap et al. (2023) Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoorv Vyas, Maryam Fazel-Zarandi, et al. 2023\.  Scaling speech technology to 1,000+ languages.  _arXiv preprint arXiv:2305.13516_.
  * Quesada et al. (2022) Juan Diego Quesada, Stavros Skopeteas, Carolina Pasamonik, Carolin Brokmann, and Florian Fischer. 2022.  Cabécar DoReCo dataset.  In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Université Lyon 2), Berlin & Lyon.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR.
  * Radford et al. (2023) Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023.  Robust speech recognition via large-scale weak supervision.  In _International Conference on Machine Learning_ , pages 28492–28518. PMLR.
  * Raymond et al. (2002) William D Raymond, Mark Pitt, Keith Johnson, Elizabeth Hume, Matthew Makashay, Robin Dautricourt, and Craig Hilts. 2002.  An analysis of transcription consistency in spontaneous speech from the buckeye corpus.  In _Seventh International Conference on Spoken Language Processing_.
  * Reiter (2022) Sabine Reiter. 2022.  Cashinahua DoReCo dataset.  In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Université Lyon 2), Berlin & Lyon.
  * Reuter et al. (2023) Paul M Reuter, Christian Rollwage, and Bernd T Meyer. 2023.  Multilingual query-by-example keyword spotting with metric learning and phoneme-to-embedding mapping.  In _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 1–5. IEEE.
  * Riesberg (2022) Sonja Riesberg. 2022.  Yali (Apahapsili) DoReCo dataset.  In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Université Lyon 2), Berlin & Lyon.
  * Ring (2022) Hiram Ring. 2022.  Pnar DoReCo dataset.  In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Université Lyon 2), Berlin & Lyon.
  * Rose (2022) Françoise Rose. 2022.  Mojeño Trinitario DoReCo dataset.  In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Université Lyon 2), Berlin & Lyon.
  * Rouditchenko et al. (2023) Andrew Rouditchenko, Sameer Khurana, Samuel Thomas, Rogerio Feris, Leonid Karlinsky, Hilde Kuehne, David Harwath, Brian Kingsbury, and James Glass. 2023\.  Comparison of Multilingual Self-Supervised and Weakly-Supervised Speech Pre-Training for Adaptation to Unseen Languages.  In _Proc. INTERSPEECH 2023_ , pages 2268–2272.
  * Rybakov et al. (2020) Oleg Rybakov, Natasha Kononenko, Niranjan Subrahmanya, Mirkó Visontai, and Stella Laurenzo. 2020.  Streaming Keyword Spotting on Mobile Devices.  In _Proc. Interspeech 2020_ , pages 2277–2281.
  * Salesky et al. (2020) Elizabeth Salesky, Eleanor Chodroff, Tiago Pimentel, Matthew Wiesner, Ryan Cotterell, Alan W Black, and Jason Eisner. 2020.  A corpus for large-scale phonetic typology.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4526–4546, Online. Association for Computational Linguistics.
  * Schnell (2022) Stefan Schnell. 2022.  Vera'a DoReCo dataset.  In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Université Lyon 2), Berlin & Lyon.
  * Seifart (2022a) Frank Seifart. 2022a.  Bora DoReCo dataset.  In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Université Lyon 2), Berlin & Lyon.
  * Seifart (2022b) Frank Seifart. 2022b.  Resígaro DoReCo dataset.  In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Université Lyon 2), Berlin & Lyon.
  * Shin et al. (2022) Hyeon-Kyeong Shin, Hyewon Han, Doyeon Kim, Soo-Whan Chung, and Hong-Goo Kang. 2022\.  Learning Audio-Text Agreement for Open-vocabulary Keyword Spotting.  In _Proc. Interspeech 2022_ , pages 1871–1875.
  * Sicherman and Adi (2023) Amitay Sicherman and Yossi Adi. 2023.  Analysing discrete self supervised speech representation for spoken language modeling.  In _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 1–5. IEEE.
  * Skopeteas et al. (2022) Stavros Skopeteas, Violeta Moisidi, Nutsa Tsetereli, Johanna Lorenz, and Stefanie Schröter. 2022.  Urum DoReCo dataset.  In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Université Lyon 2), Berlin & Lyon.
  * Tanaka et al. (2001) Kazuyo Tanaka, Yoshiaki Itoh, Hiroaki Kojima, and Nahoko Fujimura. 2001.  Speech data retrieval system constructed on a universal phonetic code domain.  In _IEEE Workshop on Automatic Speech Recognition and Understanding, 2001. ASRU'01._ , pages 323–326. IEEE.
  * Tang and Lin (2018) Raphael Tang and Jimmy Lin. 2018.  Deep residual learning for small-footprint keyword spotting.  In _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 5484–5488. IEEE.
  * Teo (2022) Amos Teo. 2022.  Sümi DoReCo dataset.  In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Université Lyon 2), Berlin & Lyon.
  * Thieberger (2022) Nick Thieberger. 2022.  Nafsan (South Efate) DoReCo dataset.  In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Université Lyon 2), Berlin & Lyon.
  * Vanhove (2022) Martine Vanhove. 2022.  Beja DoReCo dataset.  In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Université Lyon 2), Berlin & Lyon.
  * Vydrina (2022) Alexandra Vydrina. 2022.  Kakabe DoReCo dataset.  In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Université Lyon 2), Berlin & Lyon.
  * Wang et al. (2023) Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. 2023.  Neural codec language models are zero-shot text to speech synthesizers.  _arXiv preprint arXiv:2301.02111_.
  * Wegener (2022) Claudia Wegener. 2022.  Savosavo DoReCo dataset.  In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Université Lyon 2), Berlin & Lyon.
  * Wichmann (2022) Søren Wichmann. 2022.  Texistepec Popoluca DoReCo dataset.  In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Université Lyon 2), Berlin & Lyon.
  * Witzlack-Makarevich et al. (2022) Alena Witzlack-Makarevich, Saudah Namyalo, Anatol Kiriggwajjo, and Zarina Molochieva. 2022.  Ruuli DoReCo dataset.  In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Université Lyon 2), Berlin & Lyon.
  * Wu et al. (2023) Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. 2023.  Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation.  In _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 1–5. IEEE.
  * Xu and Bai (2022) Xianming Xu and Bibo Bai. 2022.  Sadu DoReCo dataset.  In Frank Seifart, Ludger Paschen, and Matthew Stave, editors, _Language Documentation Reference Corpus (DoReCo) 1.2_. Leibniz-Zentrum Allgemeine Sprachwissenschaft & laboratoire Dynamique Du Langage (UMR5596, CNRS & Université Lyon 2), Berlin & Lyon.
  * Zhai et al. (2023) Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023.  Sigmoid loss for language image pre-training.  _arXiv preprint arXiv:2303.15343_.
  * Zhu et al. (2022a) Jian Zhu, Zuoyu Tian, Yadong Liu, Cong Zhang, and Chia-Wen Lo. 2022a.  Bootstrapping meaning through listening: Unsupervised learning of spoken sentence embeddings.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 1134–1154, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Zhu et al. (2022b) Jian Zhu, Cong Zhang, and David Jurgens. 2022b.  ByT5 model for massively multilingual grapheme-to-phoneme conversion.  In _Proc. Interspeech 2022_ , pages 446–450.

##  Appendix A Dataset statistics

Table 7, LABEL:app:fleurs_stats and LABEL:app:doreco_stats provide tabulated
summaries of the detailed statistics of our curated datasets.

##  Appendix B Training hyperparameters

For pre-training, we trained three variants of BERT from scratch using only
phonemic transcriptions. We adopted the AdamW optimizer with an initialize
learning rate of $1e-4$ and cosine scheduling with a warm-up step of 1000. All
models were trained for 60k iterations before stopping. All training processes
were completed on a single V100 GPU of 32 GB.

All hyperparameters for CLAP-IPA models were listed in Table 6. By default,
all models were trained on a single V100 with 32GB of memory. The training
time for 100k steps ranged from 17 hours for CLAP-IPA-tiny to 41 hours for
CLAP-IPA-small.

Hyperparameters | CLAP-IPA-tiny | CLAP-IPA-base | CLAP-IPA-small
---|---|---|---
Hidden dimensions | 384 | 512 | 768
Num. Layers | 4 | 6 | 12
Num. Att. Heads | 6 | 8 | 12
Intermediate size | 1536 | 2048 | 3072
Initial learning rate | 1e-4
Scheduler | Cosine Scheduler
Warm-up steps | 500
Total training steps | 100k
FLEURS-P batch size | 32
MSWC-P batch size | 256
DoReCo-P batch size | 32
Gradient checkpointing | True
Mixed Precision | float16
Max. Gradient Norm for Gradient Clipping | 10
Table 6: Hyperparameters for training CLAP-IPA models.

Language ISO 639-3 Family Train (hrs) Dev (hrs) Test (hrs) Avg.Phones Avg.Dur.
(s) Arabic ara Indo-European 0.79 0.11 0.11 6.57 (1.58) 1 (0) Catalan cat
Indo-European 61.38 8.12 8.07 7.51 (2.34) 1 (0) Czech cze Indo-European 3.01
0.41 0.39 6.2 (1.92) 1 (0) Dutch dut Indo-European 6.42 0.84 0.84 7.2 (2.79) 1
(0) English eng Indo-European 125.61 16.29 16.52 6.84 (2.34) 1 (0) Esperanto
epo Constructed 8.48 1.13 1.12 6.95 (1.97) 1 (0) Estonian est Uralic 2.51 0.34
0.33 6.35 (2.13) 1 (0) French fra Indo-European 62.69 8.31 8.31 5.8 (2.03) 1
(0) German ger Indo-European 83.23 10.99 10.96 8.81 (3.41) 1 (0) Irish gle
Indo-European 0.48 0.07 0.07 4.17 (1.45) 1 (0) Greek gre Indo-European 0.71
0.1 0.1 5.71 (2.09) 1 (0) Interlingua ina Constructed 0.53 0.06 0.05 5.8
(1.84) 1 (0) Indonesian ind Austronesian 1.74 0.25 0.25 6.04 (1.88) 1 (0)
Italian ita Indo-European 18.42 2.46 2.43 7.35 (2.45) 1 (0) Kyrgyz kir Turkic
1.52 0.23 0.21 6.89 (2.43) 1 (0) Lithuanian lit Indo-European 0.7 0.04 0.06
6.88 (2.22) 1 (0) Maltese mlt Afro-Asiatic 1.12 0.15 0.16 6.04 (2.75) 1 (0)
Mongolian mon Mongolic 1.48 0.2 0.21 5.5 (1.73) 1 (0) Polish pol Indo-European
14.39 1.93 1.94 6.9 (2.09) 1 (0) Portuguese por Indo-European 7.16 0.95 0.95
6.52 (2.07) 1 (0) Romanian ron Indo-European 0.5 0.1 0.08 6.43 (2.31) 1 (0)
Russian rus Indo-European 18.48 2.46 2.44 8.48 (2.88) 1 (0) Slovak slo Indo-
European 0.08 0.01 0.01 6.28 (2.14) 1 (0) Slovenian slv Indo-European 0.27
0.05 0.05 5.16 (1.56) 1 (0) Spanish spa Indo-European 40.04 5.35 5.32 7.64
(2.49) 1 (0) Swedish swe Indo-European 1.18 0.16 0.16 5.36 (1.93) 1 (0) Tatar
tat Turkic 3.76 0.5 0.48 6.18 (1.84) 1 (0) Turkish tur Turkic 2.82 0.38 0.39
6.95 (2.27) 1 (0) Ukrainian ukr Indo-European 1.87 0.25 0.26 6.76 (2.3) 1 (0)
Welsh wel Indo-European 13.6 1.8 1.81 5.76 (1.96) 1 (0) Mandarin cmn Sino-
Tibetan 0.4 0.05 0.05 8.68 (2.26) 1 (0)

Table 7: Statistics of languages in MSWC-Phonemes. All samples are padded to be clips of 1 second. (Avg.Phones: average number of phonemes in each word; Avg.Dur.: average duration of each clip).  Table 8: Detailed statistics of DoReCo-Phonemes. (Avg. Phones: average number of phonemes in each word; Avg.Dur.: average duration of each clip). Language |  ISO 693-3 |  Avg. Dur (s) |  Total duration (hrs) |  Avg. Phones |  Family | Split |  Citation
---|---|---|---|---|---|---|---
Komnzo |  tci |  2.59 |  0.27 |  29.99 |  Yam | train |  (Döhler, 2022)
Vera'a |  vra |  3.55 |  0.57 |  43.03 |  Austronesian | train |  (Schnell, 2022)
Sanzhi Dargwa |  na |  4.85 |  0.17 |  44.82 |  Nakh-Daghestanian | train |  (Forker and Schiborr, 2022)
Urum |  uum |  4.63 |  0.37 |  45.75 |  Turkic | test |  (Skopeteas et al., 2022)
Beja |  bej |  2.32 |  0.36 |  24.97 |  Afro-Asiatic | test |  (Vanhove, 2022)
Light Warlpiri |  na |  3.47 |  0.47 |  32.75 |  Mixed Language | train |  (O'Shannessy, 2022a)
Kamas |  xas |  3.60 |  0.84 |  24.71 |  Uralic | train |  (Gusev et al., 2022)
Nafsan (South Efate) |  erk |  6.10 |  0.36 |  50.83 |  Austronesian | test |  (Thieberger, 2022)
Tabasaran |  tab |  4.16 |  0.21 |  42.31 |  Nakh-Daghestanian | train |  (Bogomolova et al., 2022)
Savosavo |  svs |  5.17 |  0.82 |  49.35 |  Isolate | train |  (Wegener, 2022)
Sümi |  nsm |  2.74 |  0.14 |  32.59 |  Sino-Tibetan | train |  (Teo, 2022)
French (Swiss) |  fra |  2.75 |  0.31 |  32.61 |  Indo-European | test |  (Avanzi et al., 2022)
Northern Alta |  aqn |  2.78 |  1.04 |  25.94 |  Austronesian | train |  (Garcia-Laguia, 2022)
Jejuan |  jje |  2.59 |  0.03 |  24.43 |  Koreanic | train |  (Kim, 2022)
Jahai |  jhi |  3.61 |  0.45 |  32.74 |  Austroasiatic | test |  (Burenhult, 2022)
Nisvai |  none |  3.11 |  0.56 |  42.22 |  Austronesian | test |  (Aznar, 2022)
Warlpiri |  wbp |  3.64 |  0.94 |  30.84 |  Pama-Nyungan | test |  (O'Shannessy, 2022b)
Fanbyak |  fnb |  2.81 |  0.22 |  27.29 |  Austronesian | train |  (Franjieh, 2022)
Bora |  boa |  4.40 |  0.34 |  41.49 |  Boran | train |  (Seifart, 2022a)
Yongning Na |  nru |  4.23 |  0.30 |  33.15 |  Sino-Tibetan | train |  (Michaud, 2022)
Dalabon |  ngk |  2.46 |  0.08 |  23.46 |  Gunwinyguan | train |  (Ponsonnet, 2022)
Sadu |  na |  2.75 |  0.15 |  22.78 |  Sino-Tibetan | train |  (Xu and Bai, 2022)
Teop |  tio |  2.96 |  0.65 |  30.62 |  Austronesian | train |  (Mosel, 2022)
Cashinahua |  cbs |  3.58 |  0.73 |  33.55 |  Pano-Tacanan | train |  (Reiter, 2022)
Dolgan |  dlg |  4.24 |  0.69 |  43.55 |  Turkic | test |  (Däbritz et al., 2022)
Anal |  anm |  3.02 |  0.37 |  26.43 |  Sino-Tibetan | train |  (Ozerov, 2022)
Baïnounk Gubëeher |  bab |  3.13 |  0.40 |  30.95 |  Atlantic-Congo | train |  (Cobbinah, 2022)
Texistepec Popoluca |  poq |  2.65 |  0.08 |  28.50 |  Mixe-Zoque | train |  (Wichmann, 2022)
Daakie |  ptv |  3.22 |  0.22 |  34.87 |  Austronesian | train |  (Krifka, 2022)
Ning |  ngh |  2.67 |  0.12 |  22.96 |  Tuu | train |  (Güldemann et al., 2022)
Ruuli |  ruc |  3.13 |  0.32 |  34.99 |  Atlantic-Congo | train |  (Witzlack-Makarevich et al., 2022)
Cabécar |  cjp |  3.61 |  0.38 |  39.62 |  Chibchan | test |  (Quesada et al., 2022)
Evenki |  evn |  3.89 |  0.66 |  31.71 |  Tungusic | train |  (Kazakevich and Klyachko, 2022)
Arapaho |  arp |  3.99 |  0.87 |  32.95 |  Algic | train |  (Cowell, 2022)
Svan |  sva |  4.77 |  0.56 |  47.85 |  Kartvelian | train |  (Gippert, 2022)
Resígaro |  rgr |  5.45 |  1.27 |  33.31 |  Arawakan | train |  (Seifart, 2022b)
Yali (Apahapsili) |  na |  2.38 |  0.04 |  32.34 |  Nuclear Trans New Guinea | test |  (Riesberg, 2022)
Asimjeeg Datooga |  na |  2.81 |  0.28 |  28.30 |  Nilotic | train |  (Griscom, 2022)
Northern Kurdish (Kurmanji) |  kmr |  4.39 |  0.54 |  50.76 |  Indo-European | test |  (Haig et al., 2022)
Gorwaa |  gow |  2.95 |  0.28 |  31.29 |  Afro-Asiatic | train |  (Harvey, 2022)
Pnar |  pbv |  8.28 |  0.29 |  72.74 |  Austroasiatic | test |  (Ring, 2022)
Kakabe |  kke |  4.14 |  0.59 |  33.07 |  Mande | train |  (Vydrina, 2022)
Mojeño Trinitario |  trn |  5.66 |  0.65 |  48.42 |  Arawakan | train |  (Rose, 2022)
Table 9: Detailed statistics of FLEURS-Phonemes. (Avg.Phones: average number of phonemes in each word; Avg.Dur.: average duration of each clip). Language | ISO 639-3 | Family | Train (hrs) | Dev (hrs) | Test (hrs) | Avg. Dur (s) | Avg. Phones
---|---|---|---|---|---|---|---
Afrikaans | afr | Indo-European | 2.71 | 0.48 | 0.66 | 11.95 | 4.69
Amharic | amh | Afro-Asiatic | 8.26 | 0.57 | 1.28 | 11.91 | 7.25
Arabic | ara | Afro-Asiatic | 4.93 | 0.75 | 1.12 | 10.2 | 6.63
Azerbaijani | aze | Turkic | 6.89 | 1.1 | 2.42 | 12.27 | 6.33
Belarusian | bel | Indo-European | 7.3 | 1.37 | 3.11 | 13.87 | 6.08
Bulgarian | bul | Indo-European | 7.05 | 0.85 | 1.44 | 10.65 | 5.47
Bengali | ben | Indo-European | 8.18 | 1.21 | 2.75 | 12.67 | 5.84
Bosnian | bos | Indo-European | 7.57 | 1.1 | 2.47 | 11.4 | 5.42
Catalan | cat | Indo-European | 5.77 | 1.09 | 2.43 | 11.34 | 4.64
Cebuano | ceb | Austronesian | 9.33 | 0.72 | 1.77 | 13.26 | 4.54
Mandarin Chinese | cmn | Sino-Tibetan | 6.04 | 0.87 | 2 | 10.37 | 3.81
Czech | cze | Indo-European | 6.38 | 0.82 | 1.91 | 10.76 | 5.58
Welsh | wel | Indo-European | 9.12 | 1.49 | 3.32 | 12.98 | 4.27
Danish | dan | Indo-European | 5.75 | 0.99 | 2.26 | 10.69 | 4.4
German | ger | Indo-European | 6.88 | 1.06 | 2.46 | 11.16 | 5.61
Greek | gre | Indo-European | 7.51 | 0.64 | 1.47 | 10.69 | 5.14
English | eng | Indo-European | 5.64 | 0.88 | 1.39 | 9.79 | 4.41
Spanish | spa | Indo-European | 6.73 | 1.17 | 2.45 | 11.24 | 4.87
Estonian | est | Uralic | 5.38 | 1.02 | 2.37 | 10.57 | 6.35
Fula | ful | Niger-Congo | 10.27 | 0.84 | 2.12 | 14.35 | 4.16
Finnish | fin | Uralic | 6.75 | 1.18 | 2.58 | 11.61 | 7.04
Irish | gle | Indo-European | 9.31 | 1.24 | 2.76 | 14.54 | 3.68
Galician | glg | Indo-European | 5.12 | 0.89 | 2.06 | 10.31 | 5.01
Hausa | hau | Afro-Asiatic | 10.09 | 1.25 | 2.47 | 15.21 | 4.36
Hindi | hin | Indo-European | 5.14 | 0.63 | 1.11 | 11.01 | 4.08
Croatian | hrv | Indo-European | 8.78 | 0.85 | 1.98 | 11.14 | 5.43
Hungarian | hun | Uralic | 7.01 | 1.14 | 2.45 | 10.85 | 5.76
Indonesian | ind | Austronesian | 6.94 | 0.97 | 1.89 | 12.18 | 5.79
Icelandic | ice | Indo-European | 2.11 | 0.1 | 0.14 | 10.8 | 5.53
Italian | ita | Indo-European | 6.86 | 1.31 | 2.8 | 11.52 | 4.97
Japanese | jpn | Japonic | 5.06 | 0.67 | 1.52 | 11.63 | 3.63
Javanese | jav | Austronesian | 8.6 | 0.94 | 2.22 | 12.98 | 5.47
Georgian | geo | Kartvelian | 3.87 | 0.99 | 2.37 | 11.31 | 7.14
Kazakh | kaz | Turkic | 8.91 | 1.29 | 3.02 | 13.55 | 6.78
Korean | kor | Koreanic | 5.68 | 0.57 | 1.03 | 12.14 | 7.17
Kyrgyz | kir | Turkic | 6.99 | 1.1 | 2.52 | 11.45 | 6.83
Lao | lao | Kra-Dai | 5.58 | 0.47 | 1.09 | 13.41 | 21.33
Lithuanian | lit | Indo-European | 7.28 | 0.97 | 2.32 | 10.96 | 6.33
Maori | mri | Austronesian | 13.34 | 1.86 | 4.53 | 19.34 | 3.48
Macedonian | mac | Indo-European | 5.14 | 1.05 | 2.47 | 10.5 | 5.35
Malayalam | mal | Indo-European | 7.37 | 1.36 | 2.86 | 12.28 | 10.23
Mongolian | mon | Mongolic | 8.63 | 0.97 | 2.21 | 12.19 | 5.52
Marathi | mar | Indo-European | 9.48 | 1.23 | 3.04 | 12.96 | 6
Malay | msa | Austronesian | 7.28 | 0.79 | 1.82 | 11.8 | 5.99
Maltese | mlt | Afro-Asiatic | 7.5 | 1.24 | 2.81 | 12.31 | 4.68
Burmese | bur | Sino-Tibetan | 10.07 | 1.49 | 3.25 | 14.56 | 12.17
Norwegian | nob | Indo-European | 7.96 | 0.43 | 0.93 | 12.06 | 4.54
Dutch | dut | Indo-European | 5.81 | 0.38 | 0.77 | 9.18 | 4.89
Nyanja | nya | Niger-Congo | 8.23 | 1.2 | 2.77 | 14.53 | 5.99
Oromo | orm | Afro-Asiatic | 5.11 | 0.05 | 0.13 | 13.46 | 5.41
Oriya | ori | Indo-European | 2.42 | 1 | 2.25 | 11.33 | 6.5
Punjabi | pan | Indo-European | 4.96 | 0.63 | 1.48 | 11.49 | 4.07
Polish | pol | Indo-European | 7.23 | 0.73 | 1.63 | 10.71 | 5.66
Portuguese | por | Indo-European | 7.77 | 1.06 | 2.5 | 12.49 | 4.7
Romanian | ron | Indo-European | 7.65 | 0.88 | 1.95 | 11.46 | 5.31
Russian | rus | Indo-European | 6.28 | 0.92 | 1.94 | 10.97 | 6.24
Sindhi | snd | Indo-European | 9.15 | 1.1 | 2.55 | 12.15 | 4.41
Slovak | slo | Indo-European | 4.55 | 0.92 | 2.1 | 10.8 | 5.58
Slovenian | slv | Indo-European | 5.78 | 0.74 | 1.78 | 10.2 | 5.43
Shona | sna | Niger-Congo | 7.56 | 1.27 | 3.03 | 14.12 | 6.88
Somali | som | Afro-Asiatic | 9.84 | 1.26 | 3.03 | 14.04 | 4.77
Serbian | srp | Indo-European | 8.14 | 0.7 | 1.66 | 12.05 | 5.25
Swedish | swe | Indo-European | 6.34 | 0.79 | 1.82 | 11.64 | 5.08
Swahili | swa | Niger-Congo | 10.1 | 0.69 | 1.54 | 14.72 | 5.15
Tamil | tam | Indo-European | 6.34 | 1.04 | 1.61 | 12.5 | 8.12
Telugu | tel | Indo-European | 5.87 | 0.75 | 1.11 | 11.64 | 7.03
Tajik | tgk | Indo-European | 6.52 | 0.77 | 1.96 | 13.43 | 5.39
Thai | tha | Kra-Dai | 6.21 | 1.14 | 2.56 | 11.34 | 4.83
Turkish | tur | Turkic | 6.43 | 0.94 | 2.09 | 11.77 | 6.48
Ukrainian | ukr | Indo-European | 6.7 | 0.78 | 1.78 | 10.82 | 5.87
Urdu | urd | Indo-European | 5.34 | 0.64 | 0.66 | 11.17 | 3.9
Uzbek | uzb | Turkic | 7.6 | 0.99 | 2.25 | 11.8 | 6.58
Vietnamese | vie | Austroasiatic | 6.71 | 1.01 | 2.33 | 10.97 | 4.07
Xhosa | xho | Niger-Congo | 9.78 | 1.27 | 2.91 | 12.96 | 7.19
Yoruba | yor | Niger-Congo | 8.46 | 1.56 | 3.26 | 15.48 | 3.48
Cantonese Chinese | yue | Sino-Tibetan | 5.56 | 0.93 | 2.07 | 12.31 | 3.96
Zulu | zul | Niger-Congo | 11.05 | 1.31 | 3.03 | 17.3 | 7.23