# DASB - Discrete Audio and Speech Benchmark

Pooneh Mousavi1,2, Luca Della Libera1,2, Jarod Duret3, Artem Ploujnikov4,2,
Cem Subakan5,2,1, Mirco Ravanelli1,2,4
1Concordia University 2Mila - Quebec AI Institute 3Avignon Université
4Université de Montréal 5Université Laval

###### Abstract

Discrete audio tokens have recently gained considerable attention for their
potential to connect audio and language processing, enabling the creation of
modern multimodal large language models. Ideal audio tokens must effectively
preserve phonetic and semantic content along with paralinguistic information,
speaker identity, and other details. While several types of audio tokens have
been recently proposed, identifying the optimal tokenizer for various tasks is
challenging due to the inconsistent evaluation settings in existing studies.
To address this gap, we release the Discrete Audio and Speech Benchmark
(DASB), a comprehensive leaderboard for benchmarking discrete audio tokens
across a wide range of discriminative tasks, including speech recognition,
speaker identification and verification, emotion recognition, keyword
spotting, and intent classification, as well as generative tasks such as
speech enhancement, separation, and text-to-speech. Our results show that, on
average, semantic tokens outperform compression tokens across most
discriminative and generative tasks. However, the performance gap between
semantic tokens and standard continuous representations remains substantial,
highlighting the need for further research in this field.

##  1 Introduction

Traditional speech and audio processing systems have long relied on
handcrafted low-level features such as Mel-Frequency Cepstral Coefficients and
Filterbanks [1]. Recently, self-supervised learning (SSL) led to outstanding
performance improvements by learning more complex, robust, and general speech
features through deep neural networks. Notable models include Wav2Vec2 [2],
WavLM [3], and HuBERT [4]. In all these cases, the rich information in speech
and audio signals is encoded into a sequence of continuous vectors. Even
though continuous vectors have proven effective in capturing the complex
details embedded in speech and audio, there is a growing interest in discrete
representations. Discrete audio representations, known as audio tokens,
transform the original waveform into a finite set of vectors. These tokens are
derived using methods such as quantization of self-supervised learning (SSL)
models [5, 6, 7], neural compression techniques (codecs)[8, 9], or hybrid
approaches [8, 9] that combine both methods.

What is driving the interest in audio tokens? Arguably, this trend is linked
to the remarkable success of autoregressive Large Language Models (LLMs) such
as LLama [10], PALM [11], BERT [12], and GPT [13]. Unlike audio, these models
operate on text, which is inherently discrete. Inspired by their
effectiveness, researchers are exploring audio language models [14, 15, 16,
17, 18, 19, 20, 21] by representing the audio as a sequence of discrete
tokens. Moreover, audio and text tokens can be naturally combined, paving the
way for the development of modern multi-modal LLMs [22] capable of processing
audio, text, and visual data. Discrete tokens also simplify audio generation
tasks like speech enhancement and synthesis by turning them into
classification problems instead of regression models [23]. Finally, they also
enable efficient data compression for better transmission and storage. The
main drawback of audio tokens is the inevitable loss of information introduced
by the discretization process. We ideally aim for audio tokens that preserves
crucial information of the original waveform, including phonetic and
linguistic content, speaker identities, emotions, and other paralinguistic
cues. However, despite the growing trend toward audio tokens, there is still a
lack of standardized evaluation benchmarks, with different studies employing
varied experimental settings [24, 25, 26, 27, 28]. Without a consistent
framework for measuring and comparing performance, it becomes challenging to
determine which audio tokens perform optimally across various tasks.

Figure 1: The workflow of DASB consists of three steps. First, a discrete
audio encoder converts the audio signal into discrete tokens (left). Then, the
tokens are combined using attention and fed to a neural model for the final
prediction (middle). For generative tasks, the predicted tokens are passed to
a discrete decoder, which converts them back into an audio waveform (right).
Both the encoder and decoder are pretrained and frozen during downstream model
training.

To address this gap, we introduce the Discrete Audio and Speech Benchmark
(DASB). DASB systematically assesses various audio tokens across several
common speech processing tasks. In particular, our contribution is the
following:

  * •

We benchmark a diverse set of discrete audio encoders from all three
categories: semantic (Discrete HuBERT, Discrete WavLM, Discrete Wav2Vec2),
compression (EnCodec [29], DAC [30]), and hybrid (SpeechTokenizer [8]).

  * •

We consider a wide range of discriminative tasks, including speech, speaker,
emotion recognition, keyword spotting, and intent classification. We also
tackle generative tasks, such as speech enhancement, separation, and text-to-
speech. For a more reliable assessment, we consider different downstream
architectures for each task, following the insights in [31]. To the best of
our knowledge, this is the first comprehensive benchmark of audio tokens that
covers both discriminative and generative tasks.

  * •

We publicly release DASB111https://github.com/speechbrain/benchmarks/tree/DASB
as a modular code repository built on the popular SpeechBrain [32] toolkit and
licensed under Apache 2.0.

##  2 Related Work

Several research efforts have recently explored using discrete audio tokens as
an alternative to continuous features. Some studies focused on using discrete
features for speech recognition and speech translation [28, 27, 33],
specifically evaluating the tokens obtained from the quantized versions of the
HuBERT model. Similarly, Yang et al. [34] examined discrete features for
speech recognition and text-to-speech. Audio tokens have been proposed for
speech enhancement as well. For example, Wang et al. [26] investigated the
application of semantic tokens to speech enhancement, whereas Erdogan et al.
[35] proposed a hybrid tokenizer called TokenSplit for both speech enhancement
and separation.

While previous studies investigated the use of compression or hybrid tokens
[36, 37, 38], these efforts were often limited to specific applications and a
few audio tokenizers. In particular, previous benchmarking attempts focused on
one category of tokenizers, either semantic or compression-based, and mostly
on discriminative or generative tasks. For instance, Puvvada et al. [25]
compared the performance of EnCodec and DAC [30] for speaker verification,
speaker diarization, and speech recognition. Mousavi et al. [39] benchmarked
various discriminative and generative tasks with semantic tokens. Wu et al.
[24] provided a comprehensive study of the quality of resynthesized sound with
compression and hybrid tokenizers. The latter attempt used pretrained models
for speech, speaker, and emotion recognition, and assessed how much
information is preserved by feeding them resynthesized audio. However, it did
not address the direct use of tokenized input for training downstream tasks,
nor did it deeply analyze the role of semantic tokens. Our analyses, instead,
suggest that semantic tokens outperform other tokenizers.

To the best of our knowledge, the proposed DASB benchmark is the first to
compare several audio tokenizers from three categories (semantic, compression,
and hybrid) across many discriminative and generative speech tasks of broad
practical interests. Moreover, unlike previous works on discrete audio tokens,
we draw inspiration from the findings presented in [31] for reliably
benchmarking continuous SSL representations and we consider different
downstream architectures for each task. Similar to the approach taken by
SUPERB [40] for continuous representation, we offer a standardized evaluation
benchmark where researchers can easily evaluate novel audio tokens.

##  3 Benchmark Design

The pipeline of DASB, illustrated in Fig. 1, consists of three components:
Audio Encoder, Downstream Model, and Audio Decoder. The main features of the
considered tokenizers are summarized in Table 1, while Figure 2 reports the
time and memory resources required by both encoders and decoders. The
following subsections describe each module in detail.

###  3.1 Discrete Audio Encoder

The audio encoder converts the audio signal into a sequence of discrete
tokens. It is pretrained on large amounts of unlabeled data and remains frozen
during the training of downstream tasks. Different encoders may compress the
information in the original waveform at different rates. The compression level
is measured by the bitrate, defined as:

$\text{bitrate}=\log_{2}{V}\cdot C\cdot R,$ (1)

where $C$ is the number of codebooks, $V$ is the number of vectors in each
codebook (vocabulary), and $R$ is the rate of codes per second. It is worth
mentioning that a single sequence of tokens might be insufficient to capture
the rich and complex information embedded in speech signals. The encoders thus
often output multiple discrete sequences, with each sequence corresponding to
a different codebook $C$. The encoders can operate at different bitrates
simply by adjusting the number of codebooks $C$. For a fairer comparison, we
define three different distinct bitrate ranges that we have identified from
the literature [41, 42, 43]: low (0-1.5 kbps), medium (2.9-6 kbps), and high
(24 kbps). We consider this approach to prevent the trivial conclusion that
some audio tokens perform better than others simply due to a higher bitrate.

The design of DASB is flexible, allowing for easy integration and benchmarking
of various tokenizers. Using the terminology from [44, 8], we categorize audio
tokens into three classes: semantic, compression, and hybrid tokenizers.

Table 1: Key Features of the Discrete Audio Encoders. #Params is computed for
medium bitrate.

Model | #Params | Sampling Rate | Bitrate (kbps) | #Codebooks
---|---|---|---|---
low | medium | high | low | medium | high
Discrete HuBERT | 309.0M | 16KHz | 0.98 | 2.9 | - | 2 | 6 | -
Discrete WavLM | 309.0M | 16KHz | 0.98 | 2.9 | - | 2 | 6 | -
Discrete Wav2Vec2 | 309.0M | 16KHz | 0.98 | 2.9 | - | 2 | 6 | -
EnCodec [29] | 17.9M | 24KHz | 1.5 | 6.0 | 24.0 | 2 | 8 | 32
DAC [30] | 22.4M | 24KHz | 1.5 | 6.0 | 24.0 | 2 | 8 | 32
SpeechTokenizer [8] | 85.3M | 16KHz | 1.0 | 4.0 | - | 2 | 8 | -

Semantic tokens [5, 6, 7] are generated by clustering or quantizing layers
from SSL models [2, 3, 4]. The tokenization process typically involves
selecting specific layers from a pretrained SSL model and applying the k-means
algorithm to group their representations. Semantic tokens primarily capture
high-level information, such as phonetic, semantic, and syntactic information.
They are not optimized for waveform reconstruction, making them potentially
better suited for discriminative tasks like speech recognition. Recent
studies, however, have shown that semantic tokens can also be effective in
generative tasks [26, 34, 39]. We adopt the tokenization algorithm proposed in
[39]. In particular, we consider three widely-used open-source SSL models:
Wav2Vec2-large, WavLM-large, and HuBERT-large, each composed of 24 layers.
Then, we cluster six of these layers using the k-means algorithm and select
two layers from the lower part (1, 3) to capture low-level information, two
from the middle layers (7, 12), and two from the higher layers (18, 23) to
encode content and meaning as well.

Compression tokens [43, 29, 30] are mainly used for audio compression. They
are trained to accurately reconstruct the original audio, making them
potentially suitable for audio generation tasks. We integrated two publicly
available compression-based tokenizers in our baseline. EnCodec [29] has three
main components: (i) an encoder network $E$ consisting of a 1D convolution
followed by a two-layer LSTM that processes the audio input and produces a
latent representation $z$; (ii) a quantization layer $Q$ that compresses $z$
into $z_{q}$ using Residual Vector Quantization (RVQ) [43], where distinct
codebooks quantizes residuals in multiple steps; and (iii) a decoder network
$G$ that mirrors the encoder and reconstructs the time-domain signal $\hat{x}$
from $z_{q}$. The system is trained end-to-end to minimize reconstruction loss
over time and frequency domains. It also adopts a perceptual loss using
discriminators at different resolutions. EnCodec offers multiple models at low
to medium bitrates (1.5 to 24 kbps). DAC [30] is an improved version of
EnCodec. It combines advances in high-fidelity audio generation with better
vector quantization techniques from the image domain, along with improved
adversarial and reconstruction losses. DAC also supports quantizer dropout,
allowing a single model to support variable bitrates.

Hybrid tokenizers [8, 9] unify semantic and acoustic tokens by disentangling
different aspects of speech information hierarchically. SpeechTokenizer [8] is
a unified speech tokenizer for large language models. It combines semantic and
acoustic tokens, separating different speech information across RVQ layers.
The model is based on RVQ-GANs, similar to EnCodec, and uses a convolutional
encoder-decoder network from EnCodec. A two-layer BiLSTM replaces the original
two-layer LSTM to improve semantic modeling. A semantic teacher guides the
first RVQ quantizer, allowing the first layer tokens to capture content
information effectively. With a residual structure, the subsequent quantizers
capture the remaining paralinguistic information. SpeechTokenizer employs
HuBERT as the semantic teacher to capture content information. The training
procedure maximizes the cosine similarity between the RVQ layer outputs and
the semantic teacher representations. HuBERT Layer 9 units represent semantic
tokens, while EnCodec codes represent acoustic tokens.

Figure 2: Time and memory required to process an utterance of 16 seconds for
encoders and decoders of the considered audio tokenizers on an NVIDIA GeForce
RTX 3070 GPU @ 8 GB.

###  3.2 Downstream Model

In this step, we employ neural networks to solve supervised tasks of common
interest. To achieve this, we first assign each discrete token to a
corresponding embedding vector through a lookup table. Subsequently, we
dynamically combine the embeddings from different codebooks using attention,
enabling the model to adjust the importance of codebooks for the specific task
of interest. The attention mechanism consists of a simple multi-layer
perceptron (MLP) that takes the embeddings of the audio tokens as input. The
MLP generates a score for each selected codebook, which is normalized by a
softmax function as shown in the following equations:

$\displaystyle
z_{c,t}=f\big{(}\text{emb}(d_{c,t})\big{)},\;a_{c,t}=\frac{\exp(z_{c,t})}{\sum_{k=1}^{C}\exp(z_{k,t})},\;h_{t}=\sum_{c}a_{c,t}z_{c,t},$
(2)

where $d_{c,t}$ is the discrete token obtained from codebook $c$, at time $t$
and, $z_{c,t}$ represents the score assigned to codebook $c$ at time $t$ by
the MLP function $f$. The function $\text{emb}(\cdot)$ refers to the lookup
table that assigns embeddings to each discrete token. The variable $a_{c,t}$
denotes the attention assigned to the codebook $c$ at time $t$, and lastly
$h_{t}$ is the representation that is fed to the downstream MLP model. We
would like to note that the MLP learns different codebook combinations $h_{t}$
at each time step $t$ depending on the attention weights $a_{c,t}$, enabling
the model to extract the necessary information when required. We also
considered summing all embeddings, as has been done in previous literature
[25, 24], but found that attention weights performed slightly better.

The combined representations $h_{t}$ are fed into neural models designed for
different tasks. The downstream models are jointly trained with their
attention and embedding layers in an end-to-end fashion. For discriminative
tasks, the model outputs either a single prediction (e.g., for emotion
recognition) or a sequence of predictions (e.g., for speech recognition). For
generative tasks (e.g., speech enhancement), the neural network outputs the
targeted tokens per codebook, with the output shape being $C\times L$, where
$L$ is the sequence length. The predicted audio tokens are converted into an
audio waveform via an audio decoder, as explained in the next section.

###  3.3 Discrete Audio Decoder

The decoder, used for generative tasks only, converts the predicted tokens
into audio signals. The decoder is frozen during training. The choice of
decoder depends on the encoder used in the first step. For compression and
hybrid tokenizers, we use their built-in decoder. For semantic tokens, we use
the scalable vocoder proposed in [39], which is a modified HiFi-GAN [45]
pretrained with LibriSpeech-960h [46]. The scalable vocoder accepts a variable
number of multi-layer semantic tokens as input and can handle different
bitrates using a layer dropout mechanism.

##  4 Experiments

In the following sections, we describe the discriminative and generative tasks
considered in our experiments. For a more reliable evaluation, we tested two
downstream architectures for each task. For detailed information about the
hyperparameters used in each experiment, refer to Appendix F.

###  4.1 Discriminative Tasks

For the downstream architectures and training procedures, we follow the best-
performing approaches for classic continuous self-supervised representations
proposed in [31].

  * •

Automatic Speech Recognition (ASR): The goal of ASR is converting speech
signals into written text. We address two ASR tasks. The first task involves
English ASR using the popular LibriSpeech dataset [46]. Training and
validation are performed on the train-clean-100 and dev-clean subsets,
respectively, while testing is conducted on the test-clean and test-other
subsets. The downstream architecture for this task consists of two layers of
Bidirectional Long Short-Term Memory (BiLSTM) followed by a linear layer for
mapping audio to characters. The second architecture utilizes ContextNet [47]
with unitary strides to maintain the frame rate of the encoder models.
Additionally, we explore low-resource languages, specifically Welsh (Cymraeg)
and Basque (Euskera) datasets extracted from CommonVoice 17.0 [48]. Here, we
evaluate the performance using both the BiLSTM architecture and a two-layer
dense neural network mapping frame representations to character probabilities.
We use the Word Error Rate (WER) as the error metric for all ASR tasks.

  * •

Speaker Identification/Verification (SID, SV): Speaker Identification involves
classifying each utterance by its speaker identity as a multi-class
classification, with the same predefined set of speakers for both training and
testing. The evaluation metric is the accuracy. Automatic Speaker Verification
(ASV), instead, involves training a binary classifier to determine whether the
speakers in a pair of utterances are the same. The evaluation metric adopted
in this case is the equal error rate (EER). We use the widely-used VoxCeleb1
[49] train and test splits for both tasks. First, we test the X-vector [50]
architecture with AM-Softmax [51] loss for training the speaker embeddings.
For verification, we use the cosine similarity between speaker
representations. As a second architecture, we replace the X-vectors with an
ECAPA-TDNN neural network [52].

  * •

Emotion Recognition (ER): The task involves predicting one of the four
classes: happy, sad, angry, and neutral. We use the popular IEMOCAP [53]
dataset, which contains about 10k samples from 10 speakers. As a first
architecture, we directly input the representations into a linear
classification layer after averaging them along the time axis. For the second
downstream architecture, we use ECAPA-TDNN. The evaluation metric is the
accuracy.

  * •

Intent Classification (IC): This task aims to determine the intention or
purpose given utterance a speech recording. In particular, we here aim to
classify each utterance into one of 18 scenarios, including calendar, email,
and alarm. For this task, we utilize the SLURP dataset [54], which comprises
around 72k audio recordings of single-turn user interactions with a home
assistant. We employ ECAPA-TDNN and a two-layer BiLSTM (followed by a linear
classifier) as downstream architectures. We evaluate the performance using the
accuracy.

  * •

Keyword Spotting (KS): Keyword Spotting involves detecting predefined keywords
by classifying utterances into a set of specified words. We use the Speech
Commands dataset v1.0 [55] for this task, as done in SUPERB. The dataset
includes ten classes of keywords, a class for silence, and an unknown class to
account for false positives. We employ both the X-vector and ECAPA-TDNN
architectures. The evaluation metric is the accuracy.

###  4.2 Generative Tasks

  * •

Speech Enhancement (SE): Speech enhancement aims to improve audio quality by
cleaning up noisy input recordings. For this task, we utilize the popular
VoiceBank dataset [56]. We employ two downstream architectures: a non-
autoregressive Conformer encoder [57], and a convolutional recurrent deep
neural network (CRDNN). The input tokens are extracted from the noisy signal,
while target tokens from the clean one. Training is performed using the cross-
entropy loss. The speech quality is assessed using the deep noise suppression
mean opinion score (DNSMOS) [58]. The intelligibility is evaluated through the
differential word error rate (dWER) [59], which measures the WER between the
transcribed enhanced signal and the transcribed target signal. The
transcriptions are obtained using the small version of Whisper [60].
Additionally, to measure speaker fidelity, we use the cosine similarity
(SpkSim) between X-vectors extracted from the enhanced signal and the target
signal using the base variant of WavLM [3] fine-tuned for speaker
verification.

  * •

Speech Separation (SS): Speech separation aims to isolate individual voices
from an audio recording containing multiple speakers. For this task, we use
the Libri2Mix dataset [61], which contains mixtures of two overlapping
speakers. We employ two downstream architectures: a non-autoregressive
Conformer encoder [57], and a convolutional recurrent deep neural network
(CRDNN). The input tokens are extracted from the mixture, while target tokens
from each of the two sources. Training is performed using the cross-entropy
loss with permutation invariant training [62]. To measure performance, we
employ the same metrics as speech enhancement.

  * •

Text-to-Speech (TTS): The task involves generating raw speech audio from a
given text input. For downstream architectures, we consider both a small and a
large autoregressive Transformer [63]. We train all models on the LJSpeech
dataset [64]. To assess the speech quality, we use a pretrained UTokyo-SaruLab
System (UTMOS) [65], which is specifically designed for TTS and trained to
predict human Mean Opinion scores. To measure pronunciation accuracy, we use
the dWER metric. This involves comparing the transcriptions provided by a
speech recognizer for the synthesized speech sample with the transcriptions
from the ground truth. As for enhancement and separation, we considered the
small version of Whisper [60].

Table 2: Benchmarking results for discriminative tasks.

Models/Tasks | ASR-En | ASR-multiling | ER | IC | KS | SI | SV
---|---|---|---|---|---|---|---
WER $\downarrow$ |  WER $\downarrow$ | ACC $\uparrow$ | ACC $\uparrow$ | ACC $\uparrow$ | ACC $\uparrow$ | EER $\downarrow$
Clean | Other | Welsh | Basque
Low Bitrate
Discrete Hubert | 8.99 | 21.14 | 58.50 | 26.83 | 57.20 | 68.70 | 90.54 | 0.90 | 24.99
Discrete WavLM | 11.72 | 27.56 | 60.37 | 28.63 | 59.80 | 73.40 | 97.94 | 0.70 | 26.02
Discrete Wav2Vec2 | 12.14 | 28.65 | 66.30 | 32.25 | 57.80 | 74.10 | 96.16 | 0.40 | 33.53
EnCodec | 52.37 | 77.04 | 92.01 | 58.20 | 44.70 | 31.50 | 86.00 | 58.30 | 17.40
DAC | 63.96 | 83.61 | 94.86 | 66.29 | 49.20 | 22.10 | 81.00 | 45.10 | 20.62
SpeechTokenizer | 19.77 | 43.12 | 76.67 | 47.92 | 49.10 | 57.90 | 95.09 | 47.40 | 20.41
Medium Bitrate
Discrete Hubert | 7.91 | 18.95 | 54.77 | 23.63 | 62.10 | 70.50 | 94.69 | 67.40 | 15.71
Discrete WavLM | 8.52 | 20.35 | 54.22 | 22.06 | 57.60 | 78.00 | 98.09 | 80.80 | 8.00
Discrete Wav2Vec2 | 8.76 | 21.32 | 60.39 | 26.64 | 59.10 | 75.10 | 96.64 | 65.47 | 17.64
EnCodec | 46.80 | 74.24 | 91.23 | 47.95 | 51.30 | 31.40 | 88.70 | 91.90 | 7.81
DAC | 59.54 | 81.48 | 97.43 | 56.16 | 45.80 | 18.90 | 76.60 | 83.80 | 11.78
SpeechTokenizer | 18.32 | 41.21 | 75.17 | 38.94 | 52.10 | 57.80 | 94.86 | 91.40 | 7.88
High Bitrate
EnCodec | 45.18 | 72.56 | 93.40 | 87.65 | 46.40 | 19.60 | 83.60 | 92.81 | 7.18
DAC | 99.53 | 99.38 | 99.40 | 99.68 | 46.00 | 15.70 | 75.20 | 85.61 | 10.89
Continuous Baseline
SSL | 3.370 | 7.04 | 41.77 | 14.32 | 63.10 | 86.10 | 99.00 | 99.70 | 2.10

Table 3: Benchmarking results for generative tasks. N.C. indicates “Not
Converged".

Models/Tasks | SE | SS | TTS
---|---|---|---
DNSMOS $\uparrow$ |  dWER $\downarrow$ |  SpkSim $\uparrow$ |  DNSMOS $\uparrow$ |  dWER $\downarrow$ |  SpkSim $\uparrow$ |  UTMOS $\uparrow$ |  dWER $\downarrow$
Low Bitrate
Discrete HuBERT | 3.33 | 15.47 | 0.824 | 3.52 | 80.86 | 0.840 | 3.24 | 2.55
Discrete WavLM | 3.26 | 16.52 | 0.830 | 3.43 | 62.34 | 0.847 | 3.84 | 3.01
Discrete Wav2Vec2 | 3.55 | 18.86 | 0.779 | 3.75 | 96.70 | 0.787 | 3.32 | 3.45
EnCodec | 3.15 | 34.35 | 0.852 | 3.11 | 83.55 | 0.877 | 1.46 | 8.85
DAC | 3.30 | 57.41 | 0.853 | 3.01 | 102.00 | 0.854 | 1.97 | 10.68
SpeechTokenizer | 3.18 | 30.13 | 0.858 | 3.13 | 85.25 | 0.874 | 2.51 | 3.69
Medium Bitrate
Discrete HuBERT | 3.48 | 12.62 | 0.875 | 3.70 | 66.29 | 0.891 | 3.80 | 3.40
Discrete WavLM | 3.48 | 10.18 | 0.889 | 3.68 | 34.03 | 0.912 | 3.82 | 2.45
Discrete Wav2Vec2 | 3.54 | 17.60 | 0.858 | 3.75 | 78.42 | 0.866 | 3.68 | 2.89
EnCodec | 3.10 | 19.07 | 0.885 | 3.09 | 48.57 | 0.906 | 1.50 | 94.6
DAC | 3.49 | 31.14 | 0.906 | 3.26 | 55.43 | 0.924 | 1.71 | 71.26
SpeechTokenizer | 3.49 | 23.44 | 0.876 | 3.42 | 60.75 | 0.906 | 1.96 | 53.26
High Bitrate
EnCodec | 2.87 | 68.22 | 0.814 | 2.95 | 97.73 | 0.839 | N.C | N.C
DAC | 2.95 | 46.07 | 0.860 | 2.53 | 208 | 0.784 | N.C | N.C
Continuous Baseline
SSL | 3.49 | 4.92 | 0.928 | 3.68 | 9.97 | 0.939 | 3.71 | 2.94

##  5 Results

###  5.1 Comparison of Discrete Audio Models

Tables 2 and 3 show the performance of discriminative and generative tasks
among the two downstream architectures explored. For each value in the table,
we report the best performance obtained with the two downstream architectures.
Detailed results for each architecture, along with the settings for each
experiment using continuous SSL models, are provided in Appendix E.

We observe a significant variation in the tokenizer performance across
different tasks. This result suggests that the optimal choice of tokenizer
depends on the specific task at hand. However, some interesting patterns
emerge. For instance, semantic tokens significantly outperform compression
tokens for most discriminative tasks. This trend is due to the ability of
semantic tokens to capture high-level information from the audio signal as
also observed in existing findings in the literature [44]. The only exceptions
are speaker recognition tasks, where EnCodec achieves the best results. This
suggests that compression tokens better encode speaker information. It is
consistent with a previous study [66] that shows discrete tokens obtained from
quantization of SSL layers remove speaker information.

Semantic tokens show the best performance for generative tasks as well,
achieving the highest MOS and dWER scores, indicating better overall quality
and intelligibility in the generated outputs. However, for preserving speaker
identity, compression tokens are more effective, as shown by superior speaker
similarity (SpkSim) metrics. We think our findings for generative tasks are
particularly interesting. While prior research efforts [26, 39] explored the
use of semantic tokens for generation, they did not include a comparison with
the performance of compression tokens. It is important to remark that the
success observed with both semantic tokens relies heavily on the effectiveness
of the decoder architecture used in our benchmark. Our scalable decoder
minimizes distortions and artifacts in the generated speech, leading to better
performance on various generative tasks.

In Table 4 (right), we present the ranking aggregation for the considered
tokenizers (medium bitrate). Each model is individually ranked for every task,
and we compute the average position across all ranks. This analysis shows that
discrete WavLM generates the top-performing audio tokens. While the continuous
version of WavLM ranks highest in the SUPERB benchmark, our findings
demonstrate for the first time that this model maintains strong performance
even after tokenization.

Our comparison between discrete tokens and the best continuous baseline
reveals a significant performance gap favoring continuous representations.
This suggests that tokenization loses valuable information, such as phonetics,
speaker identity, and emotion. Addressing this information loss is a key
challenge for future generations of audio tokens.

###  5.2 Impact of Bitrate

We also study the impact of different bitrates on the performance of the
tokenizers. Tables 2 and 3 show that a medium bitrate achieves the best
results for both discriminative and generative tasks. Interestingly, higher
bitrates, when available (e.g., for EnCodec and DAC), tend to degrade
performance. While higher bitrates can potentially preserve more information,
they also increase the output dimensionality of the model, making the task
more challenging to solve. In some cases, we found the task so challenging
with high bitrates that the model did not converge, as observed in the case of
TTS. It is worth noting that semantic tokens have a lower bitrate than
compression tokens, as shown in Table 1. For example, in the medium bitrate
range, discrete WavLM has a bitrate of 2.9 kbps, while EnCodec has 6.0 kbps.
This difference is due to the varying number of codebooks (6 vs. 8) and
sampling rates (16 kHz vs. 24 kHz). Despite their lower bitrate, semantic
tokens provide better performance.

Another aspect we investigate is the efficiency of the encoders and decoders,
as this could impact some applications. Figure 2 shows the time and memory
usage for each encoder-decoder pair across all bitrate ranges. For semantic
tokens, the encoder is a large neural network and is computationally
demanding. In contrast, the decoders are based on a compact HiFi-GAN model and
are very efficient. For streaming tasks [67] where time and memory are
critical, EnCodec turned out to be a better candidate due to the efficiency of
both its encoder and decoder models.

Table 4: (left) Evaluating various discrete decoders on the speech re-
synthesis task (medium bitrate). (right) Ranking aggregation for models
(medium bitrate).

Models/Metrics | DNSMOS $\uparrow$ | dWER $\downarrow$ | SpkSim $\uparrow$
---|---|---|---
Discrete HuBERT | 3.68 | 6.60 | 0.92
Discrete WavLM | 3.64 | 5.19 | 0.94
Discrete Wav2Vec2 | 3.71 | 8.72 | 0.91
EnCodec | 3.54 | 2.16 | 0.98
DAC | 3.74 | 2.36 | 0.99
SpeechTokenizer | 3.58 | 5.12 | 0.94
Continuous SSL | 3.73 | 2.33 | 0.98

Model | Disc. | Gen. | Comb.
---|---|---|---
Discrete HuBERT | 2.66 | 3.62 | 3.11
Discrete WavLM | 2.00 | 2.75 | 1.94
Discrete Wav2Vec2 | 3.33 | 2.68 | 3.41
EnCodec | 4.11 | 3.93 | 4.23
DAC | 5.55 | 4 .06 | 4.64
SpeechTokenizer | 3.44 | 3.81 | 3.64

###  5.3 Analysis of Discrete Audio Decoder

Finally, we present a comparative evaluation of the decoders in Table 4
(left). The decoder evaluation is conducted on the LibriSpeech test-clean
subset using a speech re-synthesis task, where we extract the tokens from each
discrete audio encoder and reconstruct the speech using the associated
decoders. Then, we evaluate the reconstructed speech based on speaker
similarity, Mean Opinion Score (MOS), and differential Word Error Rate (dWER).
The goal of this experiment is to establish if a given system is able to
provide a high-fidelity reconstruction of the input audio after encoding it in
the discrete space. This is especially important to establish for generative
tasks.

The results show that the built-in decoders of compression tokens outperform
other models in preserving speaker similarity, further confirming that current
semantic tokens do not adequately preserve speaker information. Compression-
based tokens also achieve better dWER scores. However, in terms of speech
quality (assessed with DNSMOS), there are no significant differences between
semantic and compression-based tokens. This trend indicates that while
semantic tokens produce good-quality audio, they may be slightly more prone to
semantic degradation (e.g., mispronunciations of words or phonemes). As
expected, continuous baselines perform better than their discrete
counterparts. Additional analysis on low and high settings can be found in
Appendix D.

##  6 Conclusion

This paper introduces DASB, a comprehensive benchmark designed to evaluate the
performance of discrete audio tokens across diverse tasks of broad interest.
We employ various evaluation metrics, downstream architectures, and bitrates
for more robust comparisons. Interestingly, our findings reveal that semantic
tokens outperform, on average, compression tokens in both generative and
discriminative tasks. In particular, discrete WavLM emerged as the top-
performing model, making it a natural candidate for adoption in multi-modal
text+audio LLMs. A significant performance gap, however, persists when
compared to traditional self-supervised continuous representations. This
highlights the need for further research, which we believe is essential for
better incorporating audio tokens into large multimodal language models.

One limitation we encountered is the proprietary nature of some audio
tokenizers, such as Soundstream zeghidour2021soundstream [43], which are not
publicly accessible. Additionally, the benchmark is currently limited to
speech tasks, but we plan to broaden it including music and sound processing.
Our goal is to help the research community establish a shared benchmark and
evaluation protocol for discrete audio representations. We will thus keep
expanding DASB by continuously incorporating novel tokenizers and tasks.

##  7 Acknowledgements

We acknowledge the support of the Natural Sciences and Engineering Research
Council of Canada (NSERC) and the Digital Research Alliance of Canada
(alliancecan.ca). We thank OVHCloud for donating part of the GPU computing
resources needed for this work.

## References

  * [1] Lawrence Rabiner and Biing-Hwang Juang.  Fundamentals of Speech Recognition.  Prentice-Hall Signal Processing Series, 1993.
  * [2] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli.  wav2vec 2.0: A framework for self-supervised learning of speech representations.  In International Conference on Neural Information Processing Systems (NeurIPS), volume 33, pages 12449–12460, 2020.
  * [3] Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al.  WavLM: Large-scale self-supervised pre-training for full stack speech processing.  IEEE Journal of Selected Topics in Signal Processing, 16(6):1505–1518, 2022.
  * [4] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed.  HuBERT: Self-supervised speech representation learning by masked prediction of hidden units.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451–3460, 2021.
  * [5] Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Abdelrahman Mohamed, and Emmanuel Dupoux.  Speech resynthesis from discrete disentangled self-supervised representations.  In Interspeech, pages 3615–3619, 2021.
  * [6] Dan Wells, Hao Tang, and Korin Richmond.  Phonetic analysis of self-supervised representations of English speech.  In Interspeech, pages 3583–3587, 2022.
  * [7] Yu-An Chung, Yu Zhang, Wei Han, Chung-Cheng Chiu, James Qin, Ruoming Pang, and Yonghui Wu.  w2v-BERT: Combining contrastive learning and masked language modeling for self-supervised speech pre-training.  In IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 244–250, 2021.
  * [8] Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, and Xipeng Qiu.  SpeechTokenizer: Unified speech tokenizer for speech large language models.  In International Conference on Learning Representations (ICLR), 2024.
  * [9] Zhihao Du, Shiliang Zhang, Kai Hu, and Siqi Zheng.  FunCodec: A fundamental, reproducible and integrable open-source toolkit for neural speech codec.  arXiv preprint arXiv:2309.07405, 2023.
  * [10] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  arXiv preprint arXiv:2307.09288, 2023.
  * [11] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al.  PaLM: scaling language modeling with pathways.  Journal of Machine Learning Research, 24, 2024.
  * [12] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Conference of the North American Chapter of the Association for Computational Linguistics (NAACL): Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, 2019.
  * [13] Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang.  GPT understands, too.  AI Open, 2023.
  * [14] Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu.  Neural discrete representation learning.  In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, 2017.
  * [15] Paul K. Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Zalán Borsos, Félix de Chaumont Quitry, Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, Hannah Muckenhirn, et al.  AudioPaLM: A large language model that can speak and listen.  arXiv preprint arXiv:2306.12925, 2023.
  * [16] Jiaming Wang, Zhihao Du, Qian Chen, Yunfei Chu, Zhifu Gao, Zerui Li, Kai Hu, Xiaohuan Zhou, Jin Xu, Ziyang Ma, Wen Wang, Siqi Zheng, Chang Zhou, Zhijie Yan, and Shiliang Zhang.  LauraGPT: Listen, attend, understand, and regenerate audio with GPT.  arXiv preprint arXiv:2310.04673, 2023.
  * [17] Tianrui Wang, Long Zhou, Ziqiang Zhang, Yu Wu, Shujie Liu, Yashesh Gaur, Zhuo Chen, Jinyu Li, and Furu Wei.  VioLA: Unified codec language models for speech recognition, synthesis, and translation.  arXiv preprint arXiv:2305.16107, 2023.
  * [18] Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, and Furu Wei.  Neural codec language models are zero-shot text to speech synthesizers.  arXiv preprint arXiv:2301.02111, 2023.
  * [19] Xiaofei Wang, Manthan Thakker, Zhuo Chen, Naoyuki Kanda, Sefik Emre Eskimez, Sanyuan Chen, Min Tang, Shujie Liu, Jinyu Li, and Takuya Yoshioka.  SpeechX: Neural codec language model as a versatile speech transformer.  arXiv preprint arXiv:2308.06873, 2023.
  * [20] Andrea Agostinelli et al.  MusicLM: Generating music from text.  arXiv preprint arXiv:2301.11325, 2023.
  * [21] Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi.  AudioGen: Textually guided audio generation.  In International Conference on Learning Representations (ICLR), 2023.
  * [22] Gemini Team, Google.  Gemini: A family of highly capable multimodal models.  arXiv preprint arXiv:2312.11805, 2023.
  * [23] Ian J. Goodfellow, Yoshua Bengio, and Aaron Courville.  Deep Learning.  MIT Press, 2016.
  * [24] Haibin Wu, Ho-Lam Chung, Yi-Cheng Lin, Yuan-Kuei Wu, Xuanjun Chen, Yu-Chi Pai, Hsiu-Hsuan Wang, Kai-Wei Chang, Alexander H Liu, and Hung-yi Lee.  Codec-SUPERB: An in-depth analysis of sound codec models.  arXiv preprint arXiv:2402.13071, 2024.
  * [25] Krishna C. Puvvada, Nithin Rao Koluguri, Kunal Dhawan, Jagadeesh Balam, and Boris Ginsburg.  Discrete audio representation as an alternative to Mel-spectrograms for speaker and speech recognition.  In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 12111–12115, 2024.
  * [26] Ziqian Wang, Xinfa Zhu, Zihan Zhang, YuanJun Lv, Ning Jiang, Guoqing Zhao, and Lei Xie.  SELM: Speech enhancement using discrete tokens and language models.  In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 11561–11565, 2024.
  * [27] Dong Zhang, Rong Ye, Tom Ko, Mingxuan Wang, and Yaqian Zhou.  DUB: Discrete unit back-translation for speech translation.  In Findings of the Association for Computational Linguistics: ACL, pages 7147–7164, 2023.
  * [28] Xuankai Chang, Brian Yan, Kwanghee Choi, Jee-Weon Jung, Yichen Lu, Soumi Maiti, Roshan Sharma, Jiatong Shi, Jinchuan Tian, Shinji Watanabe, Yuya Fujita, et al.  Exploring speech recognition, translation, and understanding with discrete speech units: A comparative study.  In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 11481–11485, 2024.
  * [29] Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi.  High fidelity neural audio compression.  Transactions on Machine Learning Research (TMLR), 2023.
  * [30] Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar.  High-fidelity audio compression with improved RVQGAN.  In International Conference on Neural Information Processing Systems (NeurIPS), 2023.
  * [31] Salah Zaiem, Youcef Kemiche, Titouan Parcollet, Slim Essid, and Mirco Ravanelli.  Speech self-supervised representation benchmarking: Are we doing it right?  In Interspeech, pages 2873–2877, 2023.
  * [32] Mirco Ravanelli, Titouan Parcollet, Peter Plantinga, Aku Rouhe, Samuele Cornell, Loren Lugosch, Cem Subakan, Nauman Dawalatabad, Abdelwahab Heba, Jianyuan Zhong, et al.  SpeechBrain: A general-purpose speech toolkit.  arXiv preprint arXiv:2106.04624, 2021.
  * [33] Xuankai Chang, Brian Yan, Yuya Fujita, Takashi Maekaku, and Shinji Watanabe.  Exploration of efficient end-to-end ASR using discretized input from self-supervised learning.  In Interspeech, pages 1399–1403, 2023.
  * [34] Yifan Yang, Feiyu Shen, Chenpeng Du, Ziyang Ma, Kai Yu, Daniel Povey, and Xie Chen.  Towards universal speech discrete tokens: A case study for ASR and TTS.  In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 10401–10405, 2024.
  * [35] Hakan Erdogan, Scott Wisdom, Xuankai Chang, Zalán Borsos, Marco Tagliasacchi, Neil Zeghidour, and John R Hershey.  TokenSplit: Using discrete speech representations for direct, refined, and transcript-conditioned speech separation and recognition.  In Interspeech, pages 3462–3466, 2023.
  * [36] Siyang Wang and Éva Székely.  Evaluating text-to-speech synthesis from a large discrete token-based speech language model.  In Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING), pages 6464–6474, 2024.
  * [37] Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al.  Neural codec language models are zero-shot text to speech synthesizers.  arXiv preprint arXiv:2301.02111, 2023.
  * [38] Eugene Kharitonov, Damien Vincent, Zalán Borsos, Raphaël Marinier, Sertan Girgin, Olivier Pietquin, Matt Sharifi, Marco Tagliasacchi, and Neil Zeghidour.  Speak, read and prompt: High-fidelity text-to-speech with minimal supervision.  Transactions of the Association for Computational Linguistics, 11:1703–1718, 2023.
  * [39] Pooneh Mousavi, Jarod Duret, Salah Zaiem, Luca Della Libera, Artem Ploujnikov, Cem Subakan, and Mirco Ravanelli.  How should we extract discrete audio tokens from self-supervised models?, 2024.
  * [40] Shu wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y. Lin, Andy T. Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, et al.  SUPERB: Speech Processing Universal PERformance Benchmark.  In Interspeech, pages 1194–1198, 2021.
  * [41] Jean-Marc Valin, Koen Vos, and Timothy Terriberry.  Definition of the Opus audio codec.  Technical report, 2012.
  * [42] Martin Dietz, Markus Multrus, Vaclav Eksler, Vladimir Malenovsky, Erik Norvell, Harald Pobloth, Lei Miao, Zhe Wang, Lasse Laaksonen, Adriana Vasilache, et al.  Overview of the EVS codec architecture.  In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5698–5702, 2015.
  * [43] Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi.  SoundStream: An end-to-end neural audio codec.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, pages 495–507, 2021.
  * [44] Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, and Neil Zeghidour.  AudioLM: A language modeling approach to audio generation.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:2523–2533, 2023.
  * [45] Dongchao Yang, Songxiang Liu, Rongjie Huang, Jinchuan Tian, Chao Weng, and Yuexian Zou.  HiFi-Codec: Group-residual vector quantization for high fidelity audio codec.  arXiv preprint arXiv:2305.02765, 2023.
  * [46] Matěj Korvas, Ondřej Plátek, Ondřej Dušek, Lukáš Žilka, and Filip Jurčíček.  Free English and Czech telephone speech corpus shared under the CC-BY-SA 3.0 license.  In International Conference on Language Resources and Evaluation (LREC), pages 4423–4428, 2014.
  * [47] Wei Han, Zhengdong Zhang, Yu Zhang, Jiahui Yu, Chung-Cheng Chiu, James Qin, Anmol Gulati, Ruoming Pang, and Yonghui Wu.  ContextNet: Improving convolutional neural networks for automatic speech recognition with global context.  In Interspeech, pages 3610–3614, 2020.
  * [48] Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber.  Common Voice: A massively-multilingual speech corpus.  In Language Resources and Evaluation Conference (LREC), pages 4218–4222, 2020.
  * [49] Arsha Nagrani, Joon Son Chung, and Andrew Zisserman.  VoxCeleb: A large-scale speaker identification dataset.  In Interspeech, pages 2616–2620, 2017.
  * [50] David Snyder, Daniel Garcia-Romero, Gregory Sell, Daniel Povey, and Sanjeev Khudanpur.  X-vectors: Robust DNN embeddings for speaker recognition.  In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5329–5333, 2018.
  * [51] Feng Wang, Jian Cheng, Weiyang Liu, and Haijun Liu.  Additive margin softmax for face verification.  IEEE Signal Processing Letters, 25(7):926–930, 2018.
  * [52] Brecht Desplanques, Jenthe Thienpondt, and Kris Demuynck.  ECAPA-TDNN: Emphasized channel attention, propagation and aggregation in TDNN based speaker verification.  In Interspeech, pages 3830–3834, 2020.
  * [53] Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan.  IEMOCAP: Interactive emotional dyadic motion capture database.  Language resources and evaluation, 42:335–359, 2008.
  * [54] Emanuele Bastianelli, Andrea Vanzo, Pawel Swietojanski, and Verena Rieser.  SLURP: A spoken language understanding resource package.  In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7252–7262, 2020.
  * [55] Pete Warden.  Speech Commands: A dataset for limited-vocabulary speech recognition.  arXiv preprint arXiv:1804.03209, 2018.
  * [56] Cassia Valentini-Botinhao, Xin Wang, Shinji Takaki, and Junichi Yamagishi.  Investigating RNN-based speech enhancement methods for noise-robust text-to-speech.  In Speech Synthesis Workshop, pages 146–152, 2016.
  * [57] Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang.  Conformer: Convolution-augmented transformer for speech recognition.  In Interspeech, pages 5036–5040, 2020.
  * [58] Chandan KA Reddy, Vishak Gopal, and Ross Cutler.  DNSMOS P.835: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors.  In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022.
  * [59] Zhong-Qiu Wang et al.  Sequential multi-frame neural beamforming for speech separation and enhancement.  In IEEE Spoken Language Technology (SLT) Workshop, pages 905–911, 2021.
  * [60] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever.  Robust speech recognition via large-scale weak supervision.  arXiv preprint arXiv:2212.04356, 2022.
  * [61] Joris Cosentino, Manuel Pariente, Samuele Cornell, Antoine Deleforge, and Emmanuel Vincent.  LibriMix: An open-source dataset for generalizable speech separation.  arXiv preprint arXiv:2005.11262, 2020.
  * [62] M. Kolbæk, D. Yu, Z.-H. Tan, and J. Jensen.  Multitalker speech separation with utterance-level permutation invariant training of deep recurrent neural networks.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 25:1901–1913, 2017.
  * [63] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In International Conference on Neural Information Processing Systems (NeurIPS), pages 6000–6010, 2017.
  * [64] Keith Ito.  The LJ speech dataset.  https://keithito.com/LJ-Speech-Dataset/, 2017.
  * [65] Saeki Takaaki et al.  UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022.  In Interspeech, pages 4521–4525, 2022.
  * [66] Benjamin van Niekerk, Marc-André Carbonneau, Julian Zaïdi, Matthew Baas, Hugo Seuté, and Herman Kamper.  A comparison of discrete and soft speech units for improved voice conversion.  In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6562–6566, 2022.
  * [67] Yi-Chiao Wu, Israel D. Gebru, Dejan Marković, and Alexander Richard.  Audiodec: An open-source streaming high-fidelity neural audio codec.  In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5, 2023.
  * [68] Harishchandra Dubey, Ashkan Aazami, Vishak Gopal, Babak Naderi, Sebastian Braun, Ross Cutler, Alex Ju, Mehdi Zohourian, Min Tang, Mehrsa Golestaneh, et al.  ICASSP 2023 deep noise suppression challenge.  IEEE Open Journal of Signal Processing, 2024.
  * [69] Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter.  Audio Set: An ontology and human-labeled dataset for audio events.  In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 776–780, 2017.
  * [70] Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra.  FSD50K: an open dataset of human-labeled sound events.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:829–852, 2021.
  * [71] Dmitry Bogdanov, Minz Won, Philip Tovstogan, Alastair Porter, and Xavier Serra.  The MTG-Jamendo dataset for automatic music tagging.  In International Conference on Machine Learning (ICML), 2019.
  * [72] Gautham J Mysore.  Can we automatically transform speech recorded on common consumer devices in real-world environments into professional production quality speech?—A dataset, insights, and challenges.  IEEE Signal Processing Letters, 22(8):1006–1010, 2014.
  * [73] Junichi Yamagishi, Christophe Veaux, and Kirsten MacDonald.  CSTR VCTK Corpus: English multi-speaker corpus for CSTR voice cloning toolkit (version 0.92), 2019.
  * [74] Zafar Rafii, Antoine Liutkus, Fabian-Robert Stöter, Stylianos Ioannis Mimilakis, and Rachel Bittner.  The MUSDB18 corpus for music separation.  https://doi.org/10.5281/zenodo.1117372, 2017.
  * [75] Jonathan Shen, Ruoming Pang, Ron J. Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, Rif A. Saurous, Yannis Agiomvrgiannakis, and Yonghui Wu.  Natural TTS synthesis by conditioning WaveNet on Mel spectrogram predictions.  In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4779–4783, 2018.
  * [76] Hideyuki Tachibana, Katsuya Uenoyama, and Shunsuke Aihara.  Efficiently trainable text-to-speech system based on deep convolutional networks with guided attention.  In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4784–4788, 2018.

##  Appendix A General Information

###  A.1 Computational Resources

We designed our benchmark to be computationally accessible. Every task runs on
GPUs with 32 GB or more of VRAM. Tasks like keyword spotting takes only 8
hours, while speech recognition (e.g, ASR-Basque) might require about 30 hours
on a single NVIDIA V100 GPU.

###  A.2 Impact

We believe DASB can have a positive impact on the research community. We do
not foresee a direct negative societal impact or misuse of our benchmark.
However, we acknowledge that DASB can potentially accelerate progress in
multi-modal large language models, which, in turn, have a wide range of
potential positive and negative uses that society is still working to assess.

###  A.3 Hosting and Maintenance Plan

DASB platform is hosted and version-tracked via GitHub. It is available at
https://github.com/speechbrain/benchmarks. DASB is a community-driven and
open-source initiative. We plan to extend it by running additional experiments
and including new audio tokenizers and tasks. We welcome external
contributors.

###  A.4 Licensing

Our work is licensed under Apache 2.0
(https://www.apache.org/licenses/LICENSE-2.0).

Table 5: Licenses for the models used in our benchmark.

Model | License
---|---
HuBERT-large | Apache 2.0
WavLM-large | ATTRIBUTION-SHAREALIKE 2.0
Wav2vec2-large | Apache 2.0
EnCodec | MIT license
DAC | MIT license
SpeechTokenizer | Apache 2.0

###  A.5 Author Statement

We, the authors, will bear all responsibility in case of violation of rights.

##  Appendix B Discrete Audio Models Details

Table 6: Features of the Considered Discrete Audio Encoders.

Model | Dataset | Repo
---|---|---
Discrete Hubert [39] | LibriSpeech960[46] | huggingface.co/poonehmousavi/SSL_Quantization
Discrete WavLM [39] | LibriSpeech960[46] | huggingface.co/poonehmousavi/SSL_Quantization
Discrete Wav2Vec2 [39] | LibriSpeech960[46] | huggingface.co/poonehmousavi/SSL_Quantization
EnCodec [29] |  DNS [68], CommonVoice [48], AudioSet [69], |
| github.com/facebookresearch/encodec |
DAC [30] |  DAPS[72], DNS [68], CommonVoice [48], |
| github.com/descriptinc/descript-audio-codec |
SpeechTokenizer [8] | LibriSpeech960[46] | github.com/ZhangXInFD/SpeechTokenizer

##  Appendix C Dataset and Downstream Models

Table 7 provides a summary of the datasets and the two downstream
architectures used for each task.

Table 7: Dataset and Downstream Models

Dataset | Task | 1st Architecture | 2nd Architecture | Dataset Link
---|---|---|---|---
LibriSpeech [46] | Speech Recognition | BiLSTM | ContextNet | openslr.org/12
CommonVoice 17.0 [48] | Speech Recognition | BiLSTM | Linear | commonvoice.mozilla.org/en/datasets
VoxCeleb1 [49] | speaker verification/identification | ECAPA-TDNN | X-Vectors | robots.ox.ac.uk/ vgg/data/voxceleb/vox1.html
IEMOCAP [53] | Emotion Recognition | ECAPA-TDNN | Time-Pooling + Linear | sail.usc.edu/iemocap/
Speech Commands [55] | Keyword Spotting | X -Vectors | ECAPA-TDNN | tensorflow.org/datasets/catalog/speech_commands
SLURP [54] | Intent Classification | BiLSTM + Linear | Time-Pooling + Linear | zenodo.org/record/4274930
VoiceBank [56] | Speech Enhancement | Conformer | CRDNN | datashare.ed.ac.uk/handle/10283/2791
Libri2Mix [61] | Speech Separation | Conformer | CRDNN | github.com/JorisCos/LibriMix
LJSpeech [64] | Text-to-Speech | Shallow Transformer | Deep Transformer | keithito.com/LJ-Speech-Dataset/

##  Appendix D Additional Analysis of Discrete Audio Decoders

In this section, we expand on the results from Section 5.3 by including low
and high bitrates in addition to the medium bitrate. Additionally, for speaker
similarity, we measure the cosine similarity between X-vectors extracted from
the reconstructed and target signals using two different models: WavLM (SpkSim
WavLM) and ECAPA-TDNN (SpkSim ECAPA), both fine-tuned for speaker
verification. When analyzing low and high bitrate settings, distinct trends
emerge compared to the medium bitrate. At low bitrates, the models perform
worse in terms of speaker similarity, MOS, and dWER, as expected. This is
particularly pronounced for compression-based decoders, where the degradation
is more significant in terms of dWER. In contrast, at high bitrate, there is
an overall small improvement in all metrics. In particular, the DAC model
consistently outperforms EnCodec across all evaluated metrics, even for high
bitrate settings.

The analysis shows that bitrate significantly impacts the performance of
discrete decoders. Higher bitrates better preserve speech characteristics and
result in lower error rates, while lower bitrates degrade these aspects. This
highlights the trade-off between bitrate and speech synthesis quality.

Table 8: Evaluation of various discrete decoders for the speech re-synthesis
task at low, medium, and high bitrates.

Models/Metrics |  SpkSim ECAPA $\uparrow$ |  SpkSim WavLM $\uparrow$ |  DNSMOS $\uparrow$ |  dWER $\downarrow$
---|---|---|---|---
Low Bitrate
Discrete Hubert | 0.34 | 0.87 | 3.51 | 8.25
Discrete WavLM | 0.34 | 0.88 | 3.45 | 7.01
Discrete Wav2Vec2 | 0.28 | 0.82 | 3.73 | 10.58
EnCodec | 0.52 | 0.92 | 3.20 | 11.71
DAC | 0.54 | 0.91 | 3.42 | 14.02
SpeechTokenizer | 0.22 | 0.72 | 3.21 | 45.7
Medium Bitrate
Discrete Hubert | 0.47 | 0.92 | 3.68 | 6.60
Discrete WavLM | 0.53 | 0.94 | 3.64 | 5.19
Discrete Wav2Vec2 | 0.43 | 0.91 | 3.71 | 8.72
EnCodec | 0.87 | 0.98 | 3.54 | 2.16
DAC | 0.87 | 0.99 | 3.74 | 2.36
SpeechTokenizer | 0.65 | 0.94 | 3.58 | 5.12
High Bitrate
EnCodec | 0.94 | 0.99 | 3.69 | 1.47
DAC | 0.98 | 100 | 3.79 | 0.73

##  Appendix E Additional Results

Tables 9 \- 12 show the results obtained with 2 different downstream
architectures. Note that table 7 indicates the first and second architectures
explored for each task. For the continuous baseline, we follow the same
architecture as the discrete experiments except for TTS. For the TTS
continuous baseline, we use a modified Tacotron2 [75] architecture enhanced
with guided attention [76] that predicts SSL representations instead of Mel
spectrograms.

Varying the architecture of the downstream decoder leads to significant
variations in task performance. For ASR tasks, BiLSTM performs better. For
classification tasks, ECAPA-TDNN shows the best performance, except for
keyword spotting where X-vector is slightly better. For speech enhancement and
separation, Conformer shows the best performance. For TTS, a notable pattern
is observed: semantic tokens yield the best results with shallow models, while
acoustic and hybrid tokens perform better with deeper models but still
underperform compared to semantic tokens. One reason might be that discrete
SSL models retain higher-level features closer to phonetic transcriptions,
requiring lower-capacity models to capture the relationship between raw text
and such representations. Higher-capacity models can lead to slower training
and potential overfitting. In contrast, shallow models appear to underfit
acoustic tokens, resulting in high dWERs and speech-like sounds with only
surface resemblance to the original sentence, rather than intelligible speech.

Table 9: Results for discriminative tasks with the first downstream
architecture.

Models/Tasks | ASR-En | ASR-multiling | ER | IC | KS | SI | SV
---|---|---|---|---|---|---|---
WER $\downarrow$ |  WER $\downarrow$ | ACC $\uparrow$ | ACC $\uparrow$ | ACC $\uparrow$ | ACC $\uparrow$ | EER $\downarrow$
Clean | Other | Welsh | Basque
Low Bitrate
Discrete Hubert | 8.99 | 21.14 | 58.50 | 26.83 | 57.20 | 68.70 | 90.54 | 0.90 | 24.99
Discrete WavLM | 11.72 | 27.56 | 60.37 | 28.63 | 59.80 | 73.40 | 97.94 | 0.70 | 26.02
Discrete Wav2Vec2 | 12.14 | 28.65 | 66.30 | 32.25 | 57.80 | 74.10 | 96.16 | 0.40 | 33.53
EnCodec | 52.37 | 77.04 | 92.01 | 58.20 | 44.70 | 31.50 | 86.00 | 58.30 | 17.40
DAC | 63.96 | 83.61 | 94.86 | 66.29 | 49.20 | 22.10 | 81.00 | 1.10 | 29.99
SpeechTokenizer | 19.77 | 43.12 | 76.67 | 47.92 | 49.10 | 57.90 | 95.09 | 47.40 | 20.41
Medium Bitrate
Discrete Hubert | 7.91 | 18.95 | 54.77 | 23.63 | 62.10 | 70.50 | 94.69 | 67.40 | 15.71
Discrete WavLM | 8.52 | 20.35 | 54.22 | 22.06 | 57.60 | 78.00 | 98.09 | 80.80 | 8.00
Discrete Wav2Vec2 | 8.76 | 21.32 | 60.39 | 26.64 | 59.10 | 75.10 | 96.64 | 65.47 | 17.64
EnCodec | 46.80 | 74.24 | 91.23 | 47.95 | 51.30 | 31.40 | 88.70 | 91.90 | 7.81
DAC | 59.54 | 81.48 | 97.43 | 56.16 | 45.80 | 18.90 | 76.60 | 83.80 | 11.78
SpeechTokenizer | 18.32 | 41.21 | 75.17 | 38.94 | 52.10 | 57.80 | 94.86 | 91.40 | 7.88
High Bitrate
EnCodec | 45.18 | 72.56 | 93.40 | 87.65 | 46.40 | 19.60 | 83.60 | 92.81 | 7.18
DAC | 99.53 | 99.38 | 99.40 | 99.74 | 46.00 | 15.70 | 75.20 | 85.61 | 10.89
Continuous Baseline
SSL | 3.370 | 7.04 | 41.77 | 14.32 | 63.10 | 86.10 | 99.00 | 99.70 | 2.10

Table 10: Results for discriminative tasks with the second downstream
architecture.

Models/Tasks | ASR-En | ASR-multiling | ER | IC | KS | SI | SV
---|---|---|---|---|---|---|---
WER $\downarrow$ |  WER $\downarrow$ | ACC $\uparrow$ | ACC $\uparrow$ | ACC $\uparrow$ | ACC $\uparrow$ | EER $\downarrow$
Clean | Other | Welsh | Basque
Low Bitrate
Discrete Hubert | 11.99 | 23.45 | 97.30 | 99.20 | 61.40 | 57.80 | 90.54 | 15.60 | 18.34
Discrete WavLM | 14.98 | 30.32 | 100.00 | 99.29 | 61.20 | 57.70 | 97.80 | 14.00 | 18.45
Discrete Wav2Vec2 | 15.45 | 35.30 | 99.89 | 98.39 | 59.50 | 60.20 | 96.52 | 5.80 | 23.5830
EnCodec | 90.84 | 94.97 | 99.88 | 100.00 | 40.20 | 18.40 | 88.50 | 34.70 | 23.08
DAC | 125.00 | 119.00 | 100.00 | 100.00 | 47.20 | 17.70 | 83.90 | 21.50 | 27.92
SpeechTokenizer | 26.50 | 47.91 | 100.00 | 99.11 | 50.40 | 49.10 | 94.55 | 24.50 | 25.92
Medium Bitrate
Discrete Hubert | 10.91 | 21.65 | 95.81 | 97.20 | 64.80 | 59.10 | 94.38 | 14.80 | 17.67
Discrete WavLM | 11.12 | 22.63 | 96.51 | 97.38 | 60.60 | 63.80 | 97.85 | 25.60 | 15.74
Discrete Wav2Vec2 | 12.48 | 25.40 | 94.34 | 95.76 | 61.90 | 63.00 | 96.92 | 11.70 | 19.07
EnCodec | 124.00 | 119.00 | 99.82 | 99.98 | 41.50 | 18.20 | 89.10 | 30.10 | 21.73
DAC | 124.00 | 122.00 | 99.96 | 100.00 | 46.80 | 16.40 | 80.30 | 15.90 | 29.75
SpeechTokenizer | 118.00 | 117.00 | 97.08 | 96.49 | 56.60 | 47.60 | 89.10 | 32.80 | 20.15
High Bitrate
EnCodec | 124.00 | 122.00 | 100.00 | 99.93 | 43.40 | 17.10 | 80.40 | 23.40 | 25.52
DAC | 122.00 | 122.00 | 99.74 | 100.00 | 47.70 | 15.40 | 76.00 | 16.40 | 29.94
Continuous Baseline
SSL | 10.05 | 13.80 | 68.72 | 48.60 | 68.60 | 75.20 | 98.70 | 88.40 | 4.31

Table 11: Results for generative tasks with the first downstream architecture.
N.C. indicates “Not Converged".

Models/Tasks | SE | SS | TTS
---|---|---|---
DNSMOS $\uparrow$ |  dWER $\downarrow$ |  SpkSim $\uparrow$ |  DNSMOS $\uparrow$ |  dWER $\downarrow$ |  SpkSim $\uparrow$ |  UTMOS $\uparrow$ |  dWER $\downarrow$
Low Bitrate
Discrete Hubert | 3.33 | 15.47 | 0.824 | 3.52 | 80.86 | 0.840 | 3.25 | 2.93
Discrete WavLM | 3.26 | 16.52 | 0.830 | 3.43 | 62.34 | 0.847 | 2.74 | 12.69
Discrete Wav2Vec2 | 3.55 | 18.86 | 0.779 | 3.75 | 96.70 | 0.787 | 3.32 | 3.45
EnCodec | 3.15 | 34.35 | 0.852 | 3.11 | 83.55 | 0.877 | 1.46 | 8.84
DAC | 3.30 | 57.41 | 0.853 | 3.01 | 102.00 | 0.854 | 1.97 | 10.66
SpeechTokenizer | 3.18 | 30.13 | 0.858 | 3.13 | 85.25 | 0.874 | 2.55 | 16.42
Medium Bitrate
Discrete HuBERT | 3.48 | 12.62 | 0.875 | 3.70 | 66.29 | 0.891 | 3.78 | 16.09
Discrete WavLM | 3.48 | 10.18 | 0.889 | 3.68 | 34.03 | 0.912 | 3.77 | 14.76
Discrete Wav2Vec2 | 3.54 | 17.60 | 0.858 | 3.75 | 78.42 | 0.866 | 3.68 | 5.98
EnCodec | 3.10 | 19.07 | 0.885 | 3.09 | 48.57 | 0.906 | 1.43 | 92.4
DAC | 3.49 | 31.14 | 0.906 | 3.26 | 55.43 | 0.924 | 1.71 | 71.25
SpeechTokenizer | 3.49 | 23.44 | 0.876 | 3.42 | 60.75 | 0.906 | 1.97 | 53.26
High Bitrate
EnCodec | 2.87 | 68.22 | 0.814 | 2.95 | 97.73 | 0.839 | N.C | N.C
DAC | 2.95 | 46.07 | 0.860 | 2.53 | 208 | 0.784 | N.C | N.C
Continuous Baseline
SSL | 3.49 | 4.92 | 0.928 | 3.68 | 9.97 | 0.939 | 2.86 | 4.687

Table 12: Results for generative tasks with the second downstream
architecture. N.C. indicates “Not Converged".

Models/Tasks | SE | SS | TTS
---|---|---|---
DNSMOS $\uparrow$ |  dWER $\downarrow$ |  SpkSim $\uparrow$ |  DNSMOS $\uparrow$ |  dWER $\downarrow$ |  SpkSim $\uparrow$ |  UTMOS $\uparrow$ |  dWER $\downarrow$
Low Bitrate
Discrete HuBERT | 3.31 | 13.98 | 0.821 | 3.52 | 97.58 | 0.817 | 3.24 | 2.55
Discrete WavLM | 3.27 | 16.50 | 0.825 | 3.44 | 60.12 | 0.834 | 3.84 | 3.01
Discrete Wav2Vec2 | 3.55 | 17.17 | 0.777 | 3.74 | 95.20 | 0.785 | 3.32 | 3.45
EnCodec | 3.16 | 45.07 | 0.851 | 2.32 | 99.17 | 0.495 | 1.40 | 53.50
DAC | 3.31 | 59.28 | 0.863 | 2.88 | 138 | 0.804 | 1.81 | 19.77
SpeechTokenizer | 3.32 | 27.54 | 0.860 | 3.14 | 90.68 | 0.846 | 2.51 | 3.69
Medium Bitrate
Discrete HuBERT | 3.48 | 13.71 | 0.857 | 3.72 | 91.63 | 0.843 | 3.80 | 3.40
Discrete WavLM | 3.47 | 9.63 | 0.878 | 3.68 | 37.53 | 0.902 | 3.82 | 2.45
Discrete Wav2Vec2 | 3.53 | 16.58 | 0.853 | 3.74 | 83.86 | 0.831 | 3.68 | 2.89
EnCodec | 3.04 | 84.78 | 0.796 | 3.06 | 70.25 | 0.882 | 1.50 | 94.62
DAC | 3.35 | 71.07 | 0.831 | 3.12 | 95.11 | 0.872 | 1.47 | 77.00
SpeechTokenizer | 2.12 | 99.62 | 0.549 | 3.10 | 89.78 | 0.862 | 1.85 | 64.26
High Bitrate
EnCodec | 2.93 | 95.06 | 0.780 | 2.96 | 157 | 0.768 | N.C | N.C
DAC | 2.22 | 99.62 | 0.571 | 2.31 | 234 | 0.787 | N.C | N.C
Continuous Baseline
SSL | 3.42 | 6.05 | 0.861 | 3.43 | 24.92 | 0.873 | 2.86 | 4.687

##  Appendix F Hyperparameters

We maintain the embedding dimension at 1024 for consistency across all
experiments. Training continues until convergence or until the validation loss
stops improving. For detailed settings for each experiment, please visit the
GitHub repository.


# NDNSD: Service Publishing and Discovery in NDN

Saurab Dulal  University of Memphis
<EMAIL_ADDRESS>Lan Wang  University of Memphis
<EMAIL_ADDRESS>
###### Abstract

Service discovery is a crucial component in today’s massively distributed
applications. In this paper, we propose NDNSD – a fully distributed and
general-purpose service discovery protocol for Named Data Networking (NDN). By
leveraging NDN’s data synchronization capability, NDNSD offers a high-level
API for service publishing and discovery. We present NDNSD’s main design
features including hierarchical naming, service information specification, and
service accessibility. We also implemented two other discovery schemes, one
reactive and one proactive, and compared them with NDNSD. Our evaluation shows
that NDNSD achieves (a) lower latency, lower overhead, and same reliability
compared to the reactive scheme, and (b) comparable latency, lower overhead at
larger scale, and higher reliability compared to the proactive scheme.

###### Index Terms:

Information Centric Networking, Named Data Networking, Service Discovery, NDN
Synchronization

##  I Introduction

Modern applications rely on many services, e.g., cloud computing, file
storage, and databases, to function properly. These services are provided by
not only stationary servers, but also mobile devices offering computing and
data service. _Service discovery_ is the process of finding desired service(s)
in a distributed environment.

TCP/IP has a plethora of protocols ([1, 2, 3, 4, 5]) at different layers to
facilitate service discovery. These protocols have a major limitation stemming
from the inability of the network layer to recognize names used by the
application layer ([6, 7]). For example, a printer application cannot identify
itself as /edu/abc/library/printer1 to the network, but instead has to rely on
its IP address and port number. This creates a semantic mismatch between the
two layers and thus requires either a dedicated server (e.g., the resource
directory in CoAP) or a name resolution service such as DNS. This dependency
is cumbersome for decentralized applications or edge applications. Popular
edge applications such as AWS IoT [8], Google Chromecast [9], and Azure Sphere
[10] mostly depend on the cloud, not a local device, for service registration
and discovery. This design incurs extra delay for edge devices to discover
services residing in close vicinity. More seriously, a disruption in the
connectivity to the cloud will disrupt the whole application.

Named Data Networking (NDN) is a new data-centric Internet architecture [11].
It uses application-level names directly in the network layer, so there is no
need to resolve the application names into addresses and ports (and there are
no addresses in the network layer). Moreover, NDN’s distributed dataset
synchronization protocols [12], i.e., Sync, can make the discovery process
fully distributed by directly synchronizing the service information among the
participating nodes, thereby eliminating the requirement of a centralized
entity to facilitate the service discovery process. Several service discovery
solutions ([13, 14, 15, 16, 17]) have been proposed for NDN (and its
predecessor CCN). However, we found various limitations in these solutions
(Section II-C) such as high overhead, being limited to a specific environment,
or requiring separate servers to store service information.

In this paper, we propose NDNSD, a fully distributed, general-purpose, and
scalable service discovery protocol for NDN. NDNSD offers a sync-based high-
level API for publishing and discovering services, obtaining measurement
information, and controlling access to service information. We have
implemented NDNSD and evaluated it by comparing it to two other discovery
schemes, one reactive and one proactive. Our evaluation shows that NDNSD
achieves (a) lower latency, lower overhead, and same reliability compared to
the reactive scheme, and (b) comparable latency, lower overhead at a larger
scale, and higher reliability compared to the proactive scheme.

##  II Background and Related Work

###  II-A NDN

Named Data Networking (NDN) is an evolving data-centric Internet architecture.
Every piece of content in NDN is named (e.g. /edu/abc/servers/cygnux/info),
carried in one or more data packets. The content is fetched using Interests
whose name matches the data name. NDN changes IP’s host-centric communication
by decoupling data packets from their producers. The decoupling is feasible in
NDN because data is signed by its producer at the time of creation and thus
its authenticity can be verified by other nodes. Once decoupled, data can be
served by any node that stores a copy of it.

An NDN forwarder, e.g., the NDN Forwarding Daemon (NFD) [18], implements the
network-layer protocols needed for name-based communication. The forwarder
consists of three core components: Pending Interest Table (PIT), Content Store
(CS), and Forwarding Information Base (FIB). The PIT records incoming
Interests, not yet satisfied, and also aggregates them if the same Interest
has already been received. The CS caches previously received data packets to
satisfy incoming Interests. If an Interest is not satisfied by the CS or
matches the PIT, it is forwarded via one or more interfaces with the help of
the FIB and a forwarding strategy. Once a data packet is received, it is
forwarded to all the incoming interfaces of the matching Interests recorded in
the PIT, thus multicast is supported natively in NDN. In addition, since each
data packet follows the reverse path of the matching Interests, routers can
measure path performance (e.g., RTT), which enables adaptive forwarding
decisions [19].

Figure 1: Example of NDN Synchronization Process (“I” and “D” represent Sync
Interest and Sync Data, respectively.)

###  II-B Data Synchronization in NDN (Sync)

NDN synchronization, or Sync in short, provides a powerful abstraction above
the Interest-Data exchange to facilitate multi-party communication [12]. Sync
ensures that every participating node has up-to-date information of their
shared distributed dataset by encoding the set of data names in a compact form
(i.e., “state”) and exchanging the state among the nodes using Sync Interests
(Figure 1). Once a new data packet is published, Sync updates its local state
and sends the new state and new data name in a Sync Data packet to all other
nodes (Figure 1), which can now fetch the data using the new data name.

###  II-C Related Work

Mark Mosko suggested using Sync for service discovery [13]. Devices use a
well-defined namespace such as “/parc/printers/” to advertise the manifest of
service records. However, Mosko’s design lacks protocol details, API
specifications, and access control. Moreover, it was designed for a different
ICN architecture, i.e. CCNx 1.0, as opposed to NDN. Ravindran et.al. [14]
proposed two different service discovery protocols: neighbor discovery
protocol (NDP) for locally reachable CCNx nodes neighbors, and Service Publish
and Discovery (SPDP) for discovering remote services. SPDP uses a recursive
query that propagates hop-by-hop among the reachable adjacencies running SPDP
instances. Data containing the service list is aggregated by the respective
instances and is sent back to the original requestor. This approach searches
for services one hop at a time, so it may take a long time if services are
multiple hops away.

NDNe [16] uses an expanded ring search technique along with the broadcast for
service discovery in edge environments. The request is first broadcasted to a
1-hop neighbor (TTL is used for hop count). If no reply is received within the
pre-defined timeout, the request is sent to 2-hop neighbors. This process
repeats until either the service is found, or the consumer gives up. The
expanded ring search can be expensive and may not scale if there is a large
number of consumers.

Similar to our work, Mastorakis et. al. use Sync among multiple edge computing
servers (ECS) to facilitate service discovery [17]. ECS are special nodes in
the network that maintain service information available from service
providers. A discovery Interest (e.g. /discovery/ecs/) from an application is
matched with a suitable service by the closest ECS. However, maintaining ECS
is an extra infrastructure requirement for service discovery – this may be
feasible for a particular edge computing application, but not in the general
case.

##  III Design

Figure 2: Application workflow showing all layers involved in service
publishing and discovery process

We designed NDNSD to be a fully distributed and general-purpose service
discovery protocol for NDN that works in a wide range of environments, e.g.,
LAN, WAN, and IoT. It provides an API for applications to advertise and
discover services. Internally, it uses NDN Sync [12] for service announcement
and discovery.

We view service discovery as a pub-sub problem, i.e. publishing service
information and subscribing to such information. Participants in this pub-sub
system can be one of three types: i) those advertising services via publishing
service information, e.g., an NDN repository providing persistent storage
service to other devices, ii) those discovering services via subscribing to
service information, e.g., a data collection application on a sensor that
needs to use a remote persistent storage service, and iii) those doing both,
e.g., a group of computers that dynamically share their spare computation
resources with each other. Since Sync provides transport service to NDN
applications, it can be used to realize a pub-sub system, as shown by Nichols
[20]. Unlike other pub-sub systems designed for TCP/IP or ICN, Sync-based pub-
sub systems do not require central servers, changes in the network layer, or a
name resolution system. Instead, publishers and subscribers simply agree on a
common name for the Sync group, and the subscribers will be notified by the
Sync protocol of the new data names whenever a publisher publishes new data.

Given Sync’s ability to support pub-sub models, we use it for distributed
service discovery. Service publishers and finders can use a semantically
meaningful name for their sync group. e.g.,
“/edu/abc/library/printers/NDNSD/discovery” for publishing and discovering the
services of all the printers in a university library. Similarly,
“/edu/abc/cs/ndnrepo/NDNSD/discovery” can be used to discover NDN repositories
in a computer science department. These semantic names from the application
layer are directly used in the NDN network layer for rendezvous, thus there is
no need for name resolution or central servers. Additionally, we use name-
based access control (NAC-ABE) to control the accessibility of sensitive
services by an unauthorized user. The focus of our work is therefore
threefold: (a) design the naming scheme and structure of service information
(Section III-B), (b) publish and discover the service information, and (c)
Service-info accessibility. Figure 2 shows the workflow of NDNSD among several
components of the system. In the following sections, we detail the NDNSD
protocol design.

Figure 3: NDNSD Namespace Design

###  III-A Hierarchical Namespace

NDNSD uses two separate namespaces for discovery and application service
information, and their design is illustrated in Figure 3

####  III-A1 Discovery Namespace

Service publishers and finders use a Sync group to rendezvous with each other
and we call the name prefix of their Sync group _Discovery Name Prefix_. As
shown in Figure 3, the discovery name prefix starts with a routable root
component, which can be an organization, e.g., /edu/abc, or an application,
e.g., /app/mhealth. Depending on the specific application scenario, there can
be more name components to further constrain the service scope, such as
physical location (e.g., university library), organization entity (e.g.,
computer science department), and target users (e.g., computer science
students). The next component is service type. For example, image-proc
represents all the image processing services, while image-proc/rcnn includes
only those running the RCNN algorithm. The above name components are provided
by the application to NDNSD through its API (see Section III-C). Finally, the
last two name components are added by NDNSD to yield the discovery name
prefix, e.g., /edu/abc/image-proc/rcnn/NDNSD/discovery. “NDNSD” differentiates
NDNSD’s data from other protocols’ data, and “discovery” differentiates
NDNSD’s service discovery Sync data from its service-info data.

The above design does not impose a strict limit on the depth of the namespace
hierarchy. Moreover, applications have the flexibility to use semantically
meaningful names. Note, however, that service publishers and finders in one
application need to agree on a service type and avoid collision with other
applications. There may be well-known service types emerging as NDNSD is
gradually adopted.

####  III-A2 Service Information Namespace

Each service provider publishes detailed information about its service in the
relevant Sync group (see the previous section) so that service finders can
choose the most suitable provider. Figure 3 illustrates the design of our
service information namespace. It has the same first three name components,
i.e., root, service scope (optional), and service type, as the discovery
namespace. However, it requires a service identifier after service type, e.g.,
“rcnn1” after /edu/abc/image-proc/rcnn, to identify a specific service
provider, which is supplied by the application. The next two name components,
“NDNSD” and “service-info”, are added by NDNSD. For example,
/edu/abc/cs/servers/cygnux/NDN/service-info is the name prefix of the service
information about the server _cygnux_ in the computer science department. The
last component is a sequence number for the service information – whenever the
server’s information changes, the sequence number is increased. The service
provider will publish the new name through Sync, which will notify the service
finders of the new name so they can fetch the new service information.

Note that service publishers should have a valid certificate for the name they
want to use to advertise their service, as NDN requires every data packet to
be signed by the public key of the data publisher.

###  III-B Service Information

Figure 4: Sample Service Publisher Configuration File

(The above sample file is written in Boost Info Format [21])

Service information is a collection of information used by a service provider
to advertise its service. Since NDNSD is a generic service discovery protocol,
the service information design should support a wide variety of applications.
As shown in Figure 4, it is composed of three blocks: service info namespace,
required service detail, and optional service detail. The items in the
namespace block are used to construct discovery and service information name
prefixes (Section III-A) to advertise the service. Using the two service
detail blocks, providers can list details of their service using as many key-
value pairs as needed. The required block contains the _service name_ , i.e.,
how to reach the service, and _lifetime_ , i.e., how long this service
information is valid. The optional block contains more application-specific
information. For example, an image-processing service can list details about
the model used to process the image and release version. IoT applications can
provide details about sensor type, location, available memory, sleep time,
processing capabilities, etc.

###  III-C API and Protocol Interactions

NDNSD provides the following API for applications to publish and discover
service information:

Service Publisher receives and stores the service information from the
application. It uses the root, service scope, and service type information in
the namespace block to form a discovery prefix (e.g.,
/edu/abc/library/printers/NDNSD/discovery) and joins the corresponding sync
group. The publisher also uses the root, service scope, service type, and
service identifier to construct the service-info data name, e.g.,
/edu/abc/printers/printer1/NDNSD/service-info/1, creates a data packet with
this name, and stores information from the service-detail blocks in the
content of the packet.

Service Finder accepts service discovery requests from an application. Each
request contains root, service scope, and service type which are combined to
form a discovery prefix. Next, the finder joins the sync group identified by
the discovery prefix, fetches the service information from all the service
providers, and sends it back to the application. In addition, the finder can
provide measurement information to the application on demand (Section III-D).

###  III-D Measurement Information

Measurement information is crucial for applications to determine the Quality
of Service and perform load balancing. In order to help service finders make
better decisions in selecting service providers, NDNSD computes statistics
such as round-trip time (RTT), retransmission count, and timeouts, while
fetching the service information or probing a service provider using its
service name. The measurements are exposed to the application via the Service
Finder’s API.

###  III-E Service-info Accessibility

Figure 5: Use of NAC-ABE in Our System. Note: content in the context of NDNSD
is service information

Sensitive services may require some form of access control such that only
authorized users can access their service information. For example, a printer
in the CS department chair’s office may want to restrict public use. We use
Name-Based Access Control with Attribute-based Encryption (NAC-ABE) [22] to
control the visibility of service information. As shown in Figure 5, NAC
encrypts the service information with policies composed of attributes, and
generates decryption keys for only those with appropriate access rights. For
example, the printer in the Chair’s office can advertise a printer service and
encrypt the service information with the policy (“CS Chair” or “CS Admin
Assistant”). Now to decrypt this service information, the finder needs to have
a decryption key with either the “CS Chair” or the “CS Admin Assistant”
attribute.

Note that NAC is not used for all the advertised services by default. It is up
to each service provider to decide whether to use NAC, based on the
sensitivity of the services it offers. For example, public printers do not
need access control on their service information.

(a)

(b)

(c)

Figure 6: Figure (a), (b), and (c) shows the median, 75th, and 90th-percentile
service discovery latency stretch, respectively, for each successive
publication

##  IV EVALUATION

We evaluate NDNSD by comparing it with two simple service discovery schemes,
one is proactive and the other is reactive. In the _Proactive_ scheme, the
service provider periodically multicasts its service to the entire network
using a notification Interest. The Interest consists of the discovery prefix
(e.g. /edu/abc/cs/servers), the application service name (e.g.
/edu/abc/cs/servers/cygnux/service-info), and a sequence number. The sequence
number is increased whenever the service status is updated. An application
interested in the service listens to the multicast, uses the application
service name to construct a service-info Interest, and fetches the
corresponding service-info using unicast. The sequence number tells the finder
if the multicast was already received and, if so, the finder avoids fetching
the same service-info. In this scheme, multicast Interests from multiple
service providers (of the same service) are not aggregated due to the presence
of the application service name. This can significantly increase packet
overhead with an increasing number of providers. In the _Reactive_ scheme, the
finder multicasts service discovery Interests to the network. Since there can
be more than one service provider, the finder iteratively sends multicast
Interests, with known providers carried in the application parameter field.
This process continues until the Interest times out, which means the service-
info from all the providers have been fetched. Each provider, after receiving
the multicast Interest, checks if its name is present in the application
parameter. If so, it ignores the Interest. Otherwise, it sends back its
service-info. This iterative process is also periodic, which helps the finder
fetch all the updates from the providers. For a fair comparison, we set the
frequency of periodic Interest multicasts for all the schemes to once every
second in our experiments.
Setup For our evaluation, we emulate multiple service providers offering
Computation Service. Each provider updates its service-info, such as available
GPUs, TPUs, Disk Storage, and Server-load, every 10 seconds for a total of 30
updates. We use Mini-NDN [23], a lightweight network emulator tool, and the
NDN testbed topology111NDN testbed topology: http://ndndemo.arl.wustl.edu/
consisting of 37 nodes and 97 links for all experiments.

###  IV-A Performance Metrics

We use the following metrics in our evaluation:

a) Service Discovery Latency Stretch is the ratio between the actual service-
info latency (i.e., time to discover some service-info data) and the expected
minimum of the same.

b) Normalized Packet Overhead (per node) is the ratio between the overhead per
node and the expected minimum number of packets per node. For example, suppose
there are 5 service finders and 2 service providers each publishing 30 times
at an interval of 10 seconds, the expected minimum number of packets = number
of providers $\times$ number of links in both directions $\times$ number of
publications = 2 $\times$194 $\times$ 30 = 11640. For NDNSD, the overhead
consists of (i) Sync Interest and Data packets used by the service providers
and finders for synchronization, and (ii) the duplicate NACKs sent by the
nodes to notify the downstream of receiving the same Interest twice. For the
Proactive scheme, the overhead consists of periodic service multicast
Interests from the service providers and the duplicate NACKs. For the Reactive
scheme, the overhead consists of periodic multicasts to find service from the
finders and the duplicate NACKs. Additionally, for all the schemes, extra
service-info Interests and data packets are also counted as an overhead.
Ideally, there should only be one Interest and one Data packet for each
publication.

c) Satisfaction Ratio for a specific service is defined as the number of
service providers learned by a finder divided by the total number of service
providers offering the same service.

###  IV-B Emulation Results

In Figure 6, we present the median, 75th, and 90th-percentile of the Service
Discovery Latency Stretch. The results show that both NDNSD and the Proactive
scheme have similar low stretches. This is because in both schemes, the
service provider advertises its service to the entire network after it is
published or updated. The advertisement helps the finders to fetch the
service-info immediately after receiving the multicast. Figure 6 also shows
that the Reactive scheme performed worst among the three, because in this
scheme the publisher needs to wait for a multicast Interest from the finder to
send back the service-info. Thus, delayed arrival of the Interests will
increase the service-info latency, which explains its high stretch.
Occasionally, in the Reactive scheme, Interest arrival and service
advertisement can happen at the same time, resulting in a short service-info
latency equal to the delay between the service provider and the finder. This
can be seen in Figure 6(a) & (b) where the Reactive stretch is lower than the
other two schemes and close to 1.

Figure 7: Normalized Packet Overhead

In Figure 7, we present the Normalized Packet Overhead of the different
schemes. The overhead in the Proactive scheme remains constant as the number
of providers increases because the expected and actual number of packets
increased by the same factor due to the lack of Interest aggregation in the
Proactive scheme. It can be seen that for the other two schemes, NDNSD and
Reactive, the overhead starts decreasing as the number of providers increases.
This is because of the Interest aggregation in both schemes. In addition,
NDNSD performed much better than the Reactive scheme. This is because: (a)
NDNSD fetches service-info using unicast whereas the Reactive scheme uses
multicast, (b) The Reactive scheme always uses one extra final multicast
Interest, which times out, to make sure service-info from all the providers
are fetched, and (c) a greater number of multicast packets in the Reactive
scheme leads to more duplicate NACKs. Thus, we can expect that NDNSD will
perform better than the Reactive scheme, even if the number of publishers
grows much higher.

Finally, we ran a set of experiments introducing 1, 2, and 4 percent link loss
per link and compared the Satisfaction Ratios. We observed that both NDNSD and
the Reactive scheme were able to receive all the service-info data regardless
of the link losses, thereby maintaining a 100% satisfaction ratio. For NDNSD,
Sync actively synchronizes the publications using the states among all
participating nodes. Thus, advertisements from all the service providers reach
the finder. Once an advertisement is received, NDNSD fetches the service-info
and does multiple retransmissions if needed. In the Reactive case, the
finder’s periodic discovery Interest is able to retrieve/recover all service-
info data. In contrast, in the Proactive case, there is no finder
retransmission, so lost service-info packets cannot be recovered. Hence, we
observed 99%, 97.5%, and 97% satisfaction ratios for 1, 2, and 4 percent link
loss, respectively.

##  V Conclusion and Future Works

We have presented NDNSD, a fully distributed, general-purpose protocol for
service publishing and discovery in NDN. We use a hierarchical namespace for
NDNSD, which provides applications with fine-grained control over the
advertised names of their service(s). Moreover, we leverage NDN Sync to make
service discovery independent of any external infrastructure, and designed the
service information to support a wide variety of applications. NDNSD provides
measurement information to applications to allow better decision-making when
selecting service providers, and is able to control the accessibility of
service information using NAC. Finally, we have shown that NDNSD outperforms
two other baseline service discovery solutions.

We plan to do the following in our next steps: (a) evaluate NDNSD’s
performance in applications such as building management systems and improve
our design and implementation accordingly, (b) refine the collection and
representation of measurement information, and (c) conduct evaluation over NDN
testbed and release NDNSD to the public.

## ACKNOWLEDGMENT

This work was supported by the National Science Foundation awards 1629769 and
2019085. We thank the anonymous reviewers for their insightful feedback.

## References

  * [1] M. Jeronimo and J. Weast, _UPnP design by example: a software developer’s guide to universal plug and play_. Intel Press, 2003.
  * [2] M. Boucadair, R. Penno, and D. Wing, “Universal plug and play (upnp) internet gateway device-port control protocol interworking function (igd-pcp iwf),” _RFC 6970_ , 2013.
  * [3] “IEEE standard for local and metropolitan area networks – station and media access control connectivity discovery,” _IEEE Std 802.1AB-2005_ , pp. 1–176, 2005.
  * [4] L. Smith, C. Roe, and K. S. Knudsen, “A jini/sup tm/ lookup service for resource-constrained devices,” in _Proceedings 2002 IEEE 4th International Workshop on Networked Appliances_ , 2002, pp. 135–144.
  * [5] E. Guttman, “Service location protocol: Automatic discovery of IP network services,” _IEEE Internet Computing_ , vol. 3, no. 4, pp. 71–80, 1999.
  * [6] W. Shang, Y. Yu, R. Droms, and L. Zhang, “Challenges in IoT networking via TCP/IP architecture,” _NDN Technical Report 0038_ , 2016.
  * [7] W. Shang, A. Bannis, T. Liang, Z. Wang, Y. Yu, A. Afanasyev, J. Thompson, J. Burke, B. Zhang, and L. Zhang, “Named Data Networking of Things,” in _2016 IEEE First International Conference on Internet-of-Things Design and Implementation (IoTDI)_. IEEE, 2016, pp. 117–128.
  * [8] Amazon Inc., “AWS IoT,” https://aws.amazon.com/iot/solutions/connected-home/., (Accessed on 07/05/2022).
  * [9] S. Weber, _Chromecast Users Manual: Stream Video, Music, and Everything Else You Love to Your TV_. USA: Weber Systems Inc., 2014.
  * [10] Mircosoft Inc., “Azure iot edge, 2020,” https://azure.microsoft.com/en-us/services/iot-edge/., (Accessed on 07/05/2022).
  * [11] L. Zhang, A. Afanasyev, J. Burke, V. Jacobson, K. Claffy, P. Crowley, C. Papadopoulos, L. Wang, and B. Zhang, “Named data networking,” _ACM SIGCOMM Computer Communication Review_ , vol. 44, no. 3, pp. 66–73, 2014.
  * [12] P. Moll, W. Shang, Y. Yu, A. Afanasyev, and L. Zhang, “A survey of distributed dataset synchronization in named data networking,” NDN, Technical Report NDN-0053, Revision 2, 2021.
  * [13] M. Mosko, “CCNx 1.0 collection synchronization,” in _PARC Technical Report_. Palo Alto Research Center, Inc., 2014.
  * [14] R. Ravindran, T. Biswas, X. Zhang, A. Chakraborti, and G. Wang, “Information-centric networking based homenet,” in _2013 IFIP/IEEE International Symposium on Integrated Network Management (IM 2013)_. IEEE, 2013, pp. 1102–1108.
  * [15] ndn lite, “Ndn-lite service discovery,” 2018, accessed: 2020-04-22. [Online]. Available: https://github.com/named-data-iot/ndn-lite/wiki/Service-Discovery
  * [16] M. Amadeo, C. Campolo, and A. Molinaro, “NDNe: Enhancing Named Data Networking to support cloudification at the edge,” _IEEE Communications Letters_ , vol. 20, no. 11, pp. 2264–2267, 2016.
  * [17] S. Mastorakis, A. Mtibaa, J. Lee, and S. Misra, “ICedge: When edge computing meets information-centric networking,” _IEEE Internet of Things Journal_ , 2020.
  * [18] NDN Project Team, “Named Data Networking Forwarding Daemon,” https://github.com/named-data/NFD, (Accessed on 07/05/2022).
  * [19] C. Yi, A. Afanasyev, I. Moiseenko, L. Wang, B. Zhang, and L. Zhang, “A case for stateful forwarding plane,” _Computer Communications_ , vol. 36, no. 7, pp. 779–791, 2013.
  * [20] K. Nichols, “Lessons learned building a secure network measurement framework using basic ndn,” in _Proceedings of the 6th ACM Conference on Information-Centric Networking_ , 2019, pp. 112–122.
  * [21] M. Kalicinski and S. Redl, “Boost. propertytree,” _Online: http://www. boost. org/doc/libs/1_ , vol. 42, no. 0, 2008.
  * [22] Z. Zhang, Y. Yu, S. K. Ramani, A. Afanasyev, and L. Zhang, “NAC: Automating access control via Named Data,” in _IEEE Military Communications Conference (MILCOM)_ , 2018, pp. 626–633.
  * [23] NDN Project Team, “Mini-NDN GitHub,” https://github.com/named-data/mini-ndn, (Accessed on 07/05/2022).