# An Efficient End-to-End Approach to Noise Invariant Speech Features via
Multi-Task Learning

Heitor R. Guimarães, Arthur Pimentel, Anderson R. Avila, Mehdi Rezagholizadeh,
Boxing Chen, and Tiago H. Falk

###### Abstract

Self-supervised speech representation learning enables the extraction of
meaningful features from raw waveforms. These features can then be efficiently
used across multiple downstream tasks. However, two significant issues arise
when considering the deployment of such methods “in-the-wild”: (i) Their large
size, which can be prohibitive for edge applications; and (ii) their
robustness to detrimental factors, such as noise and/or reverberation, that
can heavily degrade the performance of such systems. In this work, we propose
RobustDistiller, a novel knowledge distillation mechanism that tackles both
problems jointly. Simultaneously to the distillation recipe, we apply a multi-
task learning objective to encourage the network to learn noise-invariant
representations by denoising the input. The proposed mechanism is evaluated on
twelve different downstream tasks. It outperforms several benchmarks
regardless of noise type, or noise and reverberation levels. Experimental
results show that the new Student model with 23M parameters can achieve
results comparable to the Teacher model with 95M parameters. Lastly, we show
that the proposed recipe can be applied to other distillation methodologies,
such as the recent DPWavLM. For reproducibility, code and model checkpoints
will be made available at https://github.com/Hguimaraes/robustdistiller.

###### Index Terms:

Speech Representation Learning, Self-supervised Learning, Speech pre-training,
Environmental Robustness, Domain Adaptation, Knowledge Distillation.

##  I Introduction

Self-supervised speech representation learning (S3RL) has become commonplace
in speech processing applications. These methods allow us to learn meaningful
and disentangled universal features from high-dimensional data. These features
are attained in a self-supervised manner by exploiting the inner structure of
the data modality through a pretext task [1]. Later, these universal
representations serve as input to different downstream systems, such as speech
and/or speaker recognizers. Today, Wav2Vec 2.0 [2], HuBERT [3], and WavLM [4]
are the most widely used models to extract cross-task speech representations.

Universal representations can be important for edge applications, where
storage capacity may be limited and storing the weights of different models is
unfeasible. Universal speech representations, however, face certain
limitations in edge applications. First, their performance drastically drops
in unseen environmental (test) conditions not seen during training [5].
Second, in some configurations, the representations rely on a very large
number of parameters, thus making them impractical for resource-constrained
devices. For instance, while the HuBERT-_base_ [3] model has 95 million
parameters, the XLS-R [6], a multilingual model, can range up to 2 billion.

To overcome these limitations, different techniques have been applied. For
example, the work in [5] showed that the HuBERT representation can be more
robust to unseen conditions via domain adversarial training and data
augmentation. Therefore, the so-called “Robust HuBERT” model was developed.
Alternately, different compression techniques have been explored to reduce the
footprint of those models, with promising results seen using knowledge
distillation (KD). For example, the work in [7] showed that HuBERT could be
compressed by almost a factor of four via KD, resulting in the so-called
DistilHuBERT model. Notwithstanding, recent results have shown that
DistilHuBERT has sensitivity to unseen test conditions [8], especially those
related to environmental noise and reverberation. New compression techniques
using distillation and pruning have been shown to improve performance in clean
settings. For instance, the DPWavLM model [9] benefits from these innovations.
Furthermore, experiments show these methods can reduce gaps between compressed
and larger models in noise. However, there is still room for improvement,
especially on speaker-related downstream tasks. As such, to date, only a
handful of systems have successfully combined model compression and
environmental robustness for edge speech applications. This paper aims to fill
this gap.

More specifically, the present study expands on the authors’ previous work
[10] and proposes RobustDistiller. The novel training recipe was designed to
compress universal models while making them more resilient to unseen noise
perturbations commonly encountered “in the wild”. Our new contributions can be
summarized as follows:

  1. 1.

We propose a feature denoising KD recipe that helps the Student model learn
noise-invariant features, making speech applications more robust in noisy
environments.

  2. 2.

Through extensive experimentation, we show that the denoising step introduced
during distillation gives the Student model better disentanglement properties.
Hence, this modification leads to better generalization.

  3. 3.

The proposed recipe is tested across representations (i.e., HuBERT, Wav2Vec
2.0, WavLM, and Robust HuBERT [5]), showing its generalizability.

  4. 4.

Empirically, our recipe shows that a Student model can outperform its robust
Teacher in noisy scenarios for content-related tasks with SNR ranging from
$[-5,5]$ dB.

  5. 5.

Lastly, although our method is built on top of the DistilHuBERT for most
experiments, the proposed recipe is shown to be flexible and to work alongside
other compression strategies, such as DPWavLM [9].

The remainder of this paper is organized as follows. Section II presents
background material and related works. Section III describes the proposed
method, while Section IV presents the experimental setup. Section V presents
the obtained results and discusses them in light of existing literature.
Finally, Section VII presents the conclusions and future research directions.

##  II Background and Related Work

###  II-A Self-Supervised Speech Representation Learning

Learning to extract relevant features in an unsupervised manner is a
challenging machine-learning task. Recently, the Wav2Vec 2.0 [2] was proposed
as a specific model for S3RL. The key principle behind the model is to learn
acoustic features directly from speech utterances. The architecture consists
of a multi-layer convolutional encoder that outputs intermediate latent speech
representations for every 20 ms of audio. Next, some latent representations
are randomly masked, as in a masked language modeling (MLM) problem. The
context network, i.e., a large Transformer encoder, then attempts to predict
the discrete speech units from a codebook obtained through a Gumbel-Softmax
distribution [11]. The Wav2Vec 2.0 model is optimized via a contrastive loss.

Predictive pretext tasks are another popular paradigm for learning
representations from speech signals. In this category, the model directly
relies on a multi-class classification approach. Today, HuBERT and WavLM are
representative state-of-the-art models in this category, showing top
performances across several tasks. The HuBERT and WavLM architectures are very
similar to Wav2Vec 2.0 and consist of a convolutional feature encoder and a
large Transformer encoder.

HuBERT’s training objective is to solve a masked-language modeling (MLM) task.
But instead of directly predicting the continuous representation from the
masked acoustic frame, the model supervisory signal is a discrete phonetic
unit obtained via an offline K-means clustering. Finally, a BERT-like
prediction loss is applied over the masked regions, forcing the model to learn
a combined acoustic and language model.

Similarly to HuBERT, WavLM [4] is also a predictive model, but shown to be
more robust against environmental factors. This was achieved by training the
model on a large and diverse dataset, not constrained to only audiobooks, as
other predecessors. Moreover, two augmentation strategies are performed during
the model training phase: (i) add overlapping speech to the utterances; and
(ii) contaminate the signals with additive noise. Thus, in addition to the
predictive task, the model needs to implicitly perform a speech-denoising step
to be able to predict the discrete units correctly.

###  II-B Speech processing Universal PERformance Benchmark

As more universal representations emerge, the need for a standard method to
compare them is needed. The Speech processing Universal PERformance Benchmark
(SUPERB) [12] is a leaderboard to benchmark the performance of new universal
representations on a set of 10 tasks in a standardized manner. The ten tasks
fall under four main categories: content, speaker, semantic, and
paralinguistic. Under the content category, tasks include keyword spotting
(KS), phoneme recognition (PR), automatic speech recognition (ASR), and query-
by-example (QbE). The speaker category includes speaker identification (SID),
automatic speaker verification (ASV), and speaker diarization (SD). Semantic
tasks, in turn, include spoken intent classification (IC) and spoken slot
filling (SF). Finally, the paralinguistic task include speech emotion
recognition (ER). The interested reader is referred to [12] for more details
about the SUPERB tasks and the datasets used.

More recently, the SUPERB-Speech Generation (SG) benchmark has been proposed
in [13]. It extends the tasks from the original SUPERB to include generative
tasks. This new benchmark includes speech translation, speech enhancement,
source separation, voice conversion, and out-of-distribution ASR. We are
particularly interested in two downstream tasks relevant to “in-the-wild”
scenarios, i.e., speech enhancement (SE) and source separation (SS).

Here, we also adopt the “score” metric used in [14] to compute a general score
across all the different tasks of the SUPERB challenge, thus allowing the
assessment of a representation’s success across tasks. This is important, as
different tasks can have different figures-of-merit (e.g., accuracy or equal
error rate), thus simple averaging or summing would not suffice. The score
metric is given by:

$\mathrm{Score}(u)=\frac{1000}{|T|}\Sigma_{t}^{T}\frac{1}{|I_{t}|}\Sigma_{i}^{I_{t}}\frac{s_{t,i}(u)-s_{t,i}(\mathrm{base})}{s_{t,i}(\mathrm{SOTA})-s_{t,i}(\mathrm{base})},$
(1)

where $u$ is the upstream model being evaluated (e.g., WavLM), $s_{t,i}$ is
the $i$-th metric from the task $t$. Following this nomenclature,
$s_{t,i}(\mathrm{SOTA})$ and $s_{t,i}(\mathrm{base})$ represent the best and
worst metrics achieved in the given task $t$ for the metric $i$, respectively.
$|T|$ is the total of tasks being measured and $|I_{t}|$ the total of metrics
reported for that given task. The final score ranges from 0 to 1000, with
higher values indicating better overall model performance across all tasks.

###  II-C Model Compression and Noise Robustness: Prior Art

While S3RL has become ubiquitous across speech tasks, models are still
sensitive to conditions typically observed in edge applications. For example,
recent research has shown that when test data is corrupted by factors not seen
during training (e.g., reverberation and noise), the distribution shift
hampers model performance [10]. This has been shown to be common across speech
representations, as they are usually trained on extensive unlabelled (clean)
data from audiobook readings [15, 16]. Techniques that have been explored to
mitigate this out-of-domain issue include domain adaptation and domain
generalization [17]. Robust HuBERT [5], for example, proposes to use domain
adversarial training (DAT) [18] to learn noise-invariant features, using the
HuBERT model as a baseline, and improved robustness to noisy test conditions
have been reported [5].

Moreover, edge applications are commonly limited in storage capacity and
processing power. For example, the HuBERT model can range from 95 million to 1
billion 32-bit float parameters [3] due to the size of its Transformer
network. Such requirements may place strain on edge devices. To counter this
issue, different compression schemes have been explored. DistilHuBERT [7], for
example, relies on knowledge distillation to compress the HuBERT model by
almost a factor of four. More specifically, the layerwise distillation
mechanism occurs with a Student model of only two Transformer layers. The
representation is fed to three prediction heads that approximate the latent
representations from the {4th, 8th, and 12th} layers of the Teacher model.
After training, the prediction heads are discarded. DistilHuBERT optimizes a
loss function that combines reconstruction elements with a cosine similarity
metric. LightHuBERT [19], in turn, is an alternative compression methodology
for HuBERT which consists of pruning structured parameters and a two-step
distillation mechanism. Although LightHuBERT is currently one of the best
compression methodologies for S3RL models, their recipe is computationally
expensive and can be prohibitive to train on academic resources.

Due to its ease of use, DistilHuBERT has been extended in recent works to
improve its performance under clean settings. Two similar approaches,
FitHuBERT [20] and the Deep-versus-Wide (DW) [21] method, were developed
concomitantly as an extension of the DistilHuBERT recipe. In their work, the
authors propose deeper Transformer encoder networks for the Student model
while making it thinner. Furthermore, DW further explores the use of layer-to-
layer (L2L) prediction to compute the similarity between Teacher and Student
embeddings, unlike DistilHuBERT which uses prediction heads only on top of the
last Transformer layer. Empirically, the authors show that L2L distillation
for deeper models can work better than the original DistilHuBERT.

Lastly, a more recent work termed DPHuBERT [9] was proposed using distillation
and structured pruning techniques (DP) to achieve better model performance on
several tasks from SUPERB. The training consists of a two-step recipe. First,
a Student model is created as a copy of the Teacher model. In this step,
during the distillation, the Student weights are regularized such that 75% of
them are eliminated with the structured pruning method. In the second step, a
final distillation procedure is performed to keep the representations from the
small Student model as similar to the Teacher’s as possible.

As can be seen, the methods discussed above either focus on improving the
robustness of universal representations to unseen conditions or on enabling
the compression of model sizes. To the best of our knowledge, only one work
[22], developed concurrently with ours, has investigated the compression and
robustness of S3RL models. Inspired by the Robust HuBERT model, the authors
combine domain adversarial training (DAT) and the bootstraping-your-own-
latents (BYOL) [23] style of training. Specifically, the Teacher and Student
networks receive different distorted views of the same utterance as input.
Unlike our work, the HuBERT model is distilled and evaluated only on three
downstream tasks. We also encountered three key differences from our work: (i)
We learn noise-invariance via a multi-task learning approach instead of DAT,
and only the Student model receives distorted inputs; (ii) we show that the
RobustDistiller can be applied to other Teacher models beyond just HuBERT; and
(iii) we extend our analysis to all ten tasks from the SUPERB benchmark and
two tasks from the SUPERB-SG benchmark. Our approach, termed RobustDistiller,
extends both the DistilHuBERT and DPHuBERT recipes to generate robust
universal representations. The proposed method is described in detail next.

Figure 1: Diagram of the proposed RobustDistiller recipe. The input of the
Student model can be a noisy waveform. The Teacher, on the other hand,
receives the clean waveform. The Student must learn clean features and
simultaneously retrieve as much information as possible to reconstruct a clean
utterance.

##  III RobustDistiller: Joint distillation and noise-invariant representation
extraction

The RobustDistiller recipe was first introduced in [10] and shown to compress
the HuBERT-_base_ representation to a quarter of its original size, thus
reducing the number of parameters from 94 million to 23 million approximately.
This was achieved while creating a more robust model against environmental
artifacts. Initially, RobustDistiller was proposed as an extension of the
DistilHuBERT recipe with a robustness-first principle. Herein, we show that
our proposed recipe can be extended to other distillation recipes, as
discussed and empirically shown in the following sections. Two main
modifications are proposed to improve the robustness of the Student models:
(i) a feature denoising knowledge distillation step that induces the Student
model to learn noise-invariant representations; and (ii) a multi-task learning
approach via a signal enhancement step where, given the last hidden state from
the Student model obtained from a noisy signal, we reconstruct the clean
waveform or estimate a binary mask to be applied on the STFT of the noisy
input. Figure 1 shows the schematics of the proposed model. More details about
these modifications are given next.

###  III-A Modification #1: Feature Denoising Knowledge Distillation

As stated earlier, one of the main challenges that S3RL models face is domain
shift, where the training and real-world test data differ significantly in
e.g., channel conditions and environmental noises. This can result in system
performance degradation, rendering the models less effective for practical
use. Data augmentation has been proposed to address this issue by increasing
data diversity and reducing model bias [24].

Inspired by the WavLM training scheme, here we introduce an online
contamination method during the distillation process. Our method involves
training a Student model to reconstruct the clean representations of a Teacher
model while being fed with noisy data as input. During training, a batch of
clean speech utterances is uniformly sampled and subjected to one of the
following actions: (i) no changes are made to the training utterance; (ii) the
utterance is contaminated with additive noise with a signal-to-noise ratio
(SNR) chosen randomly from $[0,20]$ dB; (iii) the speech waveform is convolved
with a random room impulse response; and (iv) both noise and reverberation are
applied simultaneously. Mathematically, the goal of feature denoising
knowledge distillation is to minimize the distance between the Student and
Teacher’s representation, as follows:

$\min_{\theta}~{}\operatorname{\mathbb{E}}_{x\sim\mathcal{D}}\left[\frac{1}{m}\sum_{i=1}^{m}\sum_{\ell\in
L}\mathcal{L}_{D}\left(f_{T}^{(\ell)}(\boldsymbol{x}_{i};\phi),f_{S}^{(\ell)}(\hat{\boldsymbol{x}}_{i};\theta)\right)\right],$
(2)

where $\boldsymbol{x}_{i}$ is sample from an unlabeled dataset
$\mathcal{D}=\\{x_{i}\mid i=1,...,m\\}$ and $\hat{\boldsymbol{x}}_{i}$ is the
distorted view of such sample. $f_{T}^{(\ell)}$ is the hidden representation
of the $\ell$-th layer from the Teacher model parametrized by $\phi$, and
$f_{S}^{(\ell)}$ is the Student prediction that approximates the Teacher
representation. $L$ is the set of layers chosen to be distilled (e.g., 4th,
8th, and 12th if following the DistilHuBERT recipe). Lastly, $\mathcal{L}_{D}$
follows the same definition of [7] as a combination of L1 loss and a cosine
similarity between representations.

With this feature denoising step, our central hypothesis is that the model
must learn to disentangle the speech signal information from the background
noise, leveraging more robust (noise-invariant) features. Note that this
methodology can be applied to both layer-wise distillation and L2L methods.

###  III-B Modification #2: Multi-task learning

Speech enhancement is a fundamental task in speech processing aimed at
improving the quality of a speech signal by removing detrimental factors. This
paper proposes a multi-task learning approach inspired by speech enhancement
and motivated by recent advancements in speech representation learning [25].
In this setting, our proposed approach involves reconstructing the clean
waveform from the learned representation of the Student model via an
additional enhancement head in complement to the layerwise distillation
process. Unlike traditional enhancement techniques, we aim to ensure that the
upstream model carries sufficient information about the speech signal rather
than focusing on optimizing a quality metric.

To achieve this goal, we explored two architectures for the enhancement head.
First, we experiment with an enhancement head similar to the encoder of the
Student model. Specifically, we employ a BiLSTM layer followed by seven
transposed convolutions and GELU activation functions. This method aims to
reconstruct the waveform directly from the last hidden state of our Student
model. The main limitation of this proposed head is that the enhancement head
is large, with roughly 17 million parameters. Although we will discard the
enhancement head after training, it can be a memory bottleneck during
training.

Next, we also experiment with a lightweight enhancement head that operates in
the time-frequency domain. This enhancement head has only 5 million
parameters, not impacting the original distillation time too much. Herein, we
estimate a binary mask that will be applied to the magnitude of the noisy
signal’s short-time Fourier transform (STFT). Later, we reconstruct the clean
waveform to measure the quality of the enhanced signal (e.g., PESQ [26]) by
using the noisy phase of the signal. This enhancement head consists of a
single BiLSTM network with three layers and a sigmoid activation function to
create the binary mask.

Additionally, we experiment with three different loss functions: L1-, L2-, and
a multi-resolution STFT-loss to guide the enhancement head. In our
experiments, we will discuss the impact of each choice in more detail. We
refer to the combination of these two procedures, i.e., feature denoising
knowledge distillation and multi-task learning, as the RobustDistiller recipe,
which can be applied across unsupervised representations. In our experiments,
we build on the work of [10] and apply the RobustDistiller recipe to distill
four popular speech representations: _Wav2Vec 2.0_ [2], HuBERT [3], WavLM [4],
as well as the more recent Robust HuBERT[5].

###  III-C Notes on extending RobustDistiller to DP-methods

It is interesting to notice that, although RobustDistiller’s recipe was
originally developed on top of the DistilHuBERT method, we can expand this
idea to other methods as well. To test this hypothesis, in this work we
explore expanding the DPWavLM model with our modifications. More specifically,
our proposed modifications are inserted during the second distillation step of
the recipe, after the distillation-plus-pruning step. We opted for this
implementation to reduce the computational cost; however, it is an interesting
future work to explore the effects of robust distillation on both steps.

##  IV Experimental Setup

###  IV-A Datasets

Similar to DistilHuBERT, the distillation recipe of the proposed model is
based on the clean speech utterances of the LibriSpeech corpus [15], a dataset
with 960 hours of audiobook recordings with a 16 kHz sampling rate derived
from the LibriVox project. Additionally, to augment the training data, two
noise datasets are used, namely MUSAN [27] and UrbanSound8K [28], as well as
the OpenSLR28 dataset [29] of room impulse responses (RIR). The noise datasets
contain approximately 15 hours of recordings in a wide variety of categories,
from office-like noises, natural sounds, babble noise, and urban sounds, such
as engine idling and siren. We removed the children playing and street music
categories to focus on non-speech like noise sources in this first analysis.
All the utterances are resampled to 16 kHz. Lastly, the OpenSLR28 dataset
contains 325 RIRs recorded in real environments, from small meeting rooms to
large auditoriums.

At test time, two additional noise datasets are used to test the model
performance under unseen conditions. The first is the Acoustic Scene
Classification from the Detection and Classification of Acoustic Scenes and
Events Challenge (DCASE2020) [30], comprised of 64 hours of audio recordings
in 10 acoustic scenes, with four different recording devices in 12 cities. For
the reverberation condition, we used the OpenSLR26 dataset [29], which
contains 60,000 simulated room impulse responses corresponding to various
small-, medium-, and large-sized rooms.

###  IV-B Pre-training

Here, we evaluate the effectiveness of the proposed methodology through a
series of experiments. For the distillation process, we selected four Teacher
models: _Wav2Vec 2.0_ base, HuBERT base, WavLM base+, and Robust HuBERT. The
distillation process, as outlined in [7], was applied to these models,
resulting in models with the _Distiller_ prefix. We should note that the
distillation process was applied to the 4th, 8th, and 12th layers of all
Teacher models, which may not be optimal for models other than HuBERT [7]. For
those interested in achieving maximum task-based performance with distilled
S3RL, refer to [31]. Therefore, the reported results may underestimate the
achievable accuracy. In the next section, we also show the experimental
results of applying our recipe in the DPWavLM method. We follow closely the
original procedure described in its original paper.

For all experiments derived from the DistilHuBERT recipe, upstream models were
trained using a single NVidia A100 GPU. When utilizing feature denoising KD
only, the proposed method requires approximately 30 hours to train, whereas
adding the waveform denoising head increases training time to approximately 43
hours. For the experiments based on the DPWavLM recipe, we follow the
configurations on the original paper, training with 4 NVidia A100 GPU for
approximately 8h. Unless stated otherwise, the reported results were achieved
using the feature denoising KD and the STFT-denoising head with the L1 loss.
We employed the AdamW optimizer with a batch size of 24 utterances for 200k
iterations, and the learning rate linearly decays from $2\times 10^{-4}$ to
zero after 14k updates.

###  IV-C Downstream Tasks, Testing conditions, and Figures-of-Merit

The ten original tasks from the SUPERB benchmark [12] and two from the SUPERB-
SG benchmark [13] are used to gauge the benefits of the proposed method. It is
important to emphasize that since our focus is on the upstream model, we do
not perform hyperparameter-tuning on the SUPERB downstream tasks; hence we use
the default parameters reported in the SUPERB benchmark. Consequently, for the
clean test scenario, some of the results reported herein for the baselines may
differ from those reported on the SUPERB leaderboard. Nonetheless, the
performance trend and gaps reported here are similar to those seen on the
leaderboard.

Four evaluation scenarios are considered for analysis for all downstream tasks
in our work: clean $(c)$, noise-only $(n)$, reverberation-only $(r)$, and
noise-plus-reverberation $(n+r)$. The clean scenario meets the same criteria
as the test set as SUPERB downstream tasks. For the noise-only setting,
additive noise with signal-to-noise ratios ranging from $[-5,20]$ dB are
sampled from the DCASE2020 dataset and added to the utterances. For the
reverberation-only condition, in turn, room impulse responses are uniformly
sampled from the OpenSLR26 dataset and convolved with the test set utterance.
Lastly, the noise-plus-reverberation condition applies both noise and
reverberation jointly as described in the $(n)$ and $(r)$ settings, thus
representing the most challenging configuration. A fixed custom seed is
applied for the random processes to ensure all models are evaluated with the
same deterioration.

As for figures-of-merit, each downstream task has its own evaluation metric.
For the classification tasks, i.e., KS, SID, IC, and ER, the standard accuracy
metric (ACC) is reported. For the SF task, both F-score and Character Error
Rate (CER) are used. Word-Error rate (WER), Phoneme-Error rate (PER), and
Diarization-Error rate (DER) are the metrics used for the ASR, PR, and SD
tasks, respectively. For the ASV task, the equal-error rate (EER) is used. For
the QbE task, we report the maximum term weighted value (MTWV) metric. Next,
the SE task is evaluated using quality and intelligibility metrics: the
Perceptual Evaluation of Speech Quality (PESQ) [26], the Short-Time Objective
Intelligibility (STOI) [32], and the Scale-Invariant Signal-to-Distortion
Ratio (SI-SDR) [33]. Lastly, the SS task is evaluated using the SI-SDR metric
as well. More details about the figures-of-merit can be found in [12, 13].

##  V Experimental Results and Discussion

TABLE I: Experimental results for content-related tasks on the SUPERB. Bold
represents the best result and underline the second best for the compressed
models.

| Upstream |  KS \- ACC $(\uparrow)$ |  PR \- PER $(\downarrow)$ |  ASR \- WER $(\downarrow)$ |  QbE \- MTWV $(\uparrow)$
---|---|---|---|---|---
| (c) | (n) | (r) | (n + r) | (c) | (n) | (r) | (n + r) | (c) | (n) | (r) | (n + r) | (c) | (n) | (r) | (n + r)
| Teachers
| Wav2Vec 2.0 Base [2] | 96.20 | 86.50 | 64.43 | 56.25 | 5.48 | 17.61 | 18.86 | 36.59 | 6.44 | 24.55 | 27.26 | 56.06 | 8.44 | 7.25 | 7.33 | 6.20
| HuBERT Base [3] | 96.30 | 84.55 | 61.70 | 51.61 | 5.05 | 15.07 | 14.41 | 31.12 | 6.43 | 20.12 | 19.26 | 42.47 | 6.52 | 6.54 | 5.53 | 5.15
| WavLM Base+ [4] | 96.88 | 90.07 | 78.64 | 72.38 | 3.93 | 11.33 | 10.83 | 30.63 | 5.87 | 14.36 | 13.94 | 39.17 | 6.25 | 6.02 | 6.16 | 5.29
| Robust HuBERT [5] | 96.33 | 92.15 | 74.81 | 66.50 | 5.38 | 9.84 | 13.92 | 28.38 | 6.75 | 14.74 | 21.55 | 41.73 | 5.50 | 5.92 | 5.83 | 5.04
| Distillation-only recipes
| Distiller (Wav2Vec 2.0) [7] | 95.33 | 82.21 | 55.18 | 45.37 | 15.70 | 37.65 | 67.54 | 77.62 | 13.00 | 46.89 | 113.63 | 181.22 | 6.56 | 4.80 | 5.00 | 5.17
| Distiller (HuBERT) [7] | 96.14 | 86.34 | 59.20 | 54.53 | 14.15 | 32.95 | 51.40 | 66.87 | 13.26 | 40.55 | 72.07 | 112.73 | 6.72 | 6.14 | 4.99 | 4.44
| Distiller (WavLM) [7] | 96.36 | 88.48 | 61.34 | 57.58 | 11.57 | 26.70 | 49.87 | 64.19 | 12.30 | 37.40 | 91.46 | 153.59 | 6.17 | 6.34 | 5.80 | 4.49
| Distiller (Robust HuBERT) [7] | 96.62 | 87.76 | 58.65 | 50.70 | 12.42 | 28.57 | 56.10 | 71.54 | 12.69 | 36.27 | 91.28 | 145.56 | 6.53 | 6.02 | 4.59 | 4.18
| FitHuBERT [20] | 96.59 | 89.78 | 61.31 | 56.77 | 13.07 | 27.32 | 38.45 | 55.30 | 11.56 | 34.20 | 54.54 | 101.33 | 6.21 | 5.79 | 5.06 | 4.38
| [Ours] RD (Wav2Vec 2.0) | 96.14 | 91.98 | 85.49 | 78.94 | 16.49 | 24.86 | 26.77 | 38.82 | 13.53 | 25.19 | 27.24 | 43.36 | 6.27 | 6.12 | 5.89 | 4.78
| [Ours] RD (HuBERT) | 96.07 | 92.31 | 85.43 | 79.32 | 15.07 | 22.71 | 23.74 | 35.16 | 13.37 | 24.44 | 25.60 | 40.81 | 6.82 | 7.35 | 6.94 | 6.02
| [Ours] RD (WavLM) | 96.79 | 93.48 | 87.76 | 82.31 | 13.38 | 20.68 | 21.43 | 32.49 | 13.02 | 23.75 | 24.46 | 39.73 | 5.89 | 6.03 | 5.99 | 5.44
| [Ours] RD (Robust HuBERT) | 96.56 | 92.99 | 85.69 | 79.81 | 13.53 | 20.58 | 21.50 | 32.38 | 13.17 | 23.97 | 25.31 | 40.85 | 6.61 | 6.57 | 6.66 | 5.19
| Distillation-_plus_ -Pruning recipes
| LightHuBERT [19] | 96.11 | 86.69 | 72.35 | 62.25 | 6.14 | 16.92 | 12.15 | 27.42 | 8.22 | 25.33 | 20.30 | 43.80 | 5.90 | 6.17 | 6.22 | 4.90
| DPHuBERT [9] | 96.30 | 84.49 | 63.84 | 56.05 | 9.41 | 24.28 | 29.40 | 49.03 | 10.60 | 32.76 | 40.12 | 76.46 | 6.28 | 5.91 | 4.96 | 5.13
| DPWavLM [9] | 96.20 | 88.12 | 60.40 | 55.08 | 8.23 | 18.81 | 25.95 | 43.78 | 10.14 | 26.17 | 37.20 | 67.93 | 6.04 | 5.46 | 5.20 | 5.05
| [Ours] RD (DPWavLM) | 96.69 | 92.66 | 88.96 | 83.93 | 8.26 | 13.19 | 12.87 | 21.44 | 9.98 | 18.37 | 18.34 | 31.59 | 6.00 | 6.04 | 5.85 | 5.00

###  V-A Performance across SUPERB tasks

In this section, we report the accuracy achieved across the 10 SUPERB tasks
with the four baseline Teacher models: Wav2Vec 2.0 base, HuBERT base, WavLM
base, and Robust HuBERT, each with 95 million parameters. Moreover, we also
report results achieved with different compression recipes, namely
_LightHuBERT_ (27M parameters), four distilled versions of the Teacher models
using the DistilHuBERT recipe (23M parameters, each), the FitHuBERT model (23M
parameters), and the DPHuBERT and DPWavLM (24M parameters, each). Finally, we
report the performance achieved with the proposed RobustDistiller recipe for
the four Teacher models, represented by the acronym _RD_ in the tables. The
results are shown for the clean test conditions (column ‘(c) in the tables),
noise-only (n), reverberation-only (r), and noise-plus-reverberation (n+r)
conditions. The performance metric used by each task is displayed, as well as
an up- or down-arrow indicating if higher or lower metric values are better,
respectively. Subsection _5)_ will provide a more detailed discussion of the
”distillation-plus-pruning” compression recipe.

####  V-A1 Robustness of Content-related Tasks

First, we start looking at the robustness experiments, the central part of our
contribution. Table V shows the obtained results for the four content-related
tasks. As can be seen, for the KS task, the _WavLM_ model showed the best
performance among the Teacher models for three of the four conditions, with
Robust HuBERT showing the highest accuracy for the noise-only condition. In
the clean and noise-only conditions, the compressed benchmarks performed
similarly to the Teacher models; however, the performance degraded
substantially under reverberation-only and reverberation-plus-noise settings.
The proposed methodology outperformed the original Distiller recipe across all
scenarios, even in clean conditions. In the challenging noisy conditions, the
RobustDistiller framework allowed for compressed models to even outperform
their larger Teacher versions, thus showing the advantage of the feature
denoising KD and multi-task learning for speech representation learning and
compression.

For the PR and ASR tasks, WavLM and Robust HuBERT continued to show the best
accuracy across the Teachers, but with substantial performance drops in the
three noisy conditions. For these two tasks, unlike KS, the Distiller
compression methodology highly degraded performance with FitHuBERT achieving
the top performance on the clean tested conditions. The proposed
RobustDistiller framework was able to outperform the original distillation
methodology and FitHuBERT on noisy conditions, but not the recipes based on
distillation-plus-pruning (e.g., DPWavLM and LightHuBERT). Lastly, for the QbE
experiments, the proposed methodology outperformed all the baseline
distillation-only schemes and Teacher models, thus further corroborating the
importance of the proposed method.

####  V-A2 Robustness of Speaker-related Tasks

Next, Table V-A2 presents the results for the three speaker-related tasks. For
these tasks, reverberation was shown to be a critical nuance factor for the
downstream tasks, including the Teacher models, corroborating previous
findings [34]. For instance, the SID task is based on the VoxCeleb1 [35]
dataset, a classification problem with more than one thousand classes. Small
distribution shifts can substantially affect system performance. Reverberation
is known to affect, for example, pitch information, which could be conveyed by
the different representations. Of the benchmark compression methods,
LightHuBERT and DP-HuBERT-based methods again came out on top for non-robust
Students, especially for the conditions involving reverberation.
Notwithstanding, the proposed compression recipe achieved the best accuracy in
these conditions, often outperforming the respective Teachers.

TABLE II: Experimental results for speaker-related tasks on the SUPERB. Bold
represents the best result and underline the second best for the compressed
models.

| Upstream |  SID \- ACC $(\uparrow)$ |  ASV \- EER $(\downarrow)$ |  SD \- DER $(\downarrow)$
---|---|---|---|---
| (c) | (n) | (r) | (n + r) | (c) | (n) | (r) | (n + r) | (c) | (n) | (r) | (n + r)
| Teachers
| Wav2Vec 2.0 Base [2] | 75.17 | 41.79 | 5.36 | 2.80 | 6.01 | 15.33 | 28.45 | 34.05 | 6.08 | 8.33 | 40.44 | 46.91
| HuBERT Base [3] | 81.43 | 52.44 | 12.37 | 6.46 | 5.50 | 12.64 | 24.41 | 28.80 | 5.88 | 8.02 | 28.51 | 32.55
| WavLM Base+ [4] | 72.14 | 49.52 | 14.08 | 6.74 | 4.34 | 10.16 | 20.88 | 28.90 | 3.58 | 5.50 | 16.25 | 21.11
| Robust HuBERT [5] | 73.08 | 48.43 | 6.99 | 4.24 | 5.26 | 10.05 | 26.01 | 31.28 | 4.79 | 6.21 | 19.51 | 22.49
| Distillation-only recipes
| Distiller (Wav2Vec 2.0) [7] | 73.63 | 41.17 | 3.07 | 1.49 | 7.27 | 17.35 | 39.18 | 41.90 | 6.59 | 8.31 | 59.72 | 65.24
| Distiller (HuBERT) [7] | 72.28 | 45.01 | 5.11 | 2.95 | 6.59 | 13.94 | 29.32 | 33.84 | 5.98 | 7.95 | 39.51 | 41.98
| Distiller (WavLM) [7] | 79.78 | 52.19 | 6.07 | 3.51 | 5.69 | 11.51 | 33.02 | 35.82 | 5.75 | 7.74 | 53.52 | 60.11
| Distiller (Robust HuBERT) [7] | 77.37 | 50.67 | 5.15 | 3.07 | 6.08 | 12.08 | 36.90 | 40.01 | 6.12 | 7.71 | 42.67 | 49.27
| FitHuBERT [20] | 54.09 | 33.04 | 7.22 | 4.04 | 6.72 | 13.86 | 33.49 | 37.75 | 6.84 | 8.99 | 33.79 | 40.63
| [Ours] RD (Wav2Vec 2.0) | 76.27 | 61.06 | 40.18 | 28.86 | 7.31 | 10.69 | 13.51 | 16.89 | 5.22 | 6.99 | 7.69 | 9.39
| [Ours] RD (HuBERT) | 77.91 | 61.83 | 43.56 | 31.66 | 6.23 | 9.23 | 11.66 | 15.01 | 5.19 | 6.40 | 8.01 | 10.46
| [Ours] RD (WavLM) | 78.39 | 60.73 | 42.04 | 30.02 | 5.83 | 8.93 | 11.24 | 14.59 | 4.92 | 6.28 | 9.01 | 12.09
| [Ours] RD (Robust HuBERT) | 77.52 | 61.96 | 41.79 | 29.66 | 6.59 | 9.64 | 12.22 | 16.02 | 5.31 | 6.53 | 8.15 | 10.63
| Distillation-_plus_ -Pruning recipes
| LightHuBERT [19] | 72.03 | 41.57 | 17.50 | 8.68 | 5.57 | 12.80 | 14.81 | 23.56 | 6.08 | 7.96 | 19.38 | 21.73
| DPHuBERT [9] | 76.33 | 45.79 | 8.13 | 4.27 | 5.54 | 12.93 | 23.93 | 30.80 | 6.16 | 7.97 | 26.18 | 32.08
| DPWavLM [9] | 82.18 | 53.65 | 12.48 | 6.61 | 5.58 | 10.66 | 29.69 | 34.84 | 5.56 | 7.18 | 36.53 | 42.88
| [Ours] RD (DPWavLM) | 84.01 | 67.69 | 49.63 | 35.06 | 5.05 | 8.14 | 9.89 | 13.99 | 5.08 | 6.37 | 6.64 | 8.45

####  V-A3 Robustness of Semantic-related Tasks

For the semantic-related evaluations, Table V-A3 shows the results for two
tasks: IC and SF. Baseline compression highly affected system performance in
noisy conditions, with distillation-plus-pruning recipes again showing
improved accuracy across all tested conditions. The proposed RobustDistiller
consistently outperformed the original Distiller and FitHuBERT recipes in
several cases, achieving results in line with LightHuBERT, DPHuBERT, or even
surpassing the Teacher model (e.g., in the IC task under the n+r condition).

TABLE III: Experimental results for semantics-related tasks on the SUPERB.
Bold represents the best result and underline the second best for the
compressed models.

| Upstream |  IC \- ACC $(\uparrow)$ |  SF \- F1 $(\uparrow)$ |  SF \- CER $(\downarrow)$
---|---|---|---|---
| (c) | (n) | (r) | (n + r) | (c) | (n) | (r) | (n + r) | (c) | (n) | (r) | (n + r)
| Teachers
| Wav2Vec 2.0 Base [2] | 92.83 | 69.26 | 68.78 | 50.88 | 88.30 | 74.06 | 70.20 | 47.61 | 24.77 | 41.73 | 48.47 | 68.11
| HuBERT Base [3] | 98.34 | 78.70 | 75.77 | 59.08 | 88.52 | 76.90 | 77.16 | 57.37 | 25.20 | 38.98 | 40.92 | 59.58
| WavLM Base+ [4] | 98.84 | 83.13 | 89.35 | 68.20 | 90.14 | 83.74 | 85.62 | 67.73 | 21.92 | 30.08 | 29.71 | 49.06
| Robust HuBERT [5] | 98.66 | 91.43 | 85.32 | 71.10 | 88.83 | 84.14 | 78.75 | 61.75 | 24.24 | 31.25 | 38.61 | 55.69
| Distillation-only recipes
| Distiller (Wav2Vec 2.0) [7] | 86.05 | 39.81 | 29.24 | 17.37 | 82.28 | 60.58 | 25.22 | 16.13 | 35.88 | 59.88 | 88.42 | 93.66
| Distiller (HuBERT) [7] | 94.12 | 48.72 | 47.72 | 24.31 | 82.12 | 64.14 | 34.43 | 20.83 | 35.21 | 55.44 | 82.26 | 90.56
| Distiller (WavLM) [7] | 94.89 | 63.25 | 48.43 | 31.69 | 84.76 | 68.74 | 36.72 | 21.43 | 32.89 | 50.60 | 80.47 | 89.66
| Distiller (Robust HuBERT) [7] | 95.86 | 67.07 | 46.30 | 28.34 | 85.19 | 68.71 | 29.08 | 17.06 | 30.98 | 49.69 | 85.84 | 92.92
| FitHuBERT [20] | 94.89 | 63.56 | 67.20 | 45.48 | 84.75 | 70.68 | 59.90 | 39.04 | 31.03 | 46.97 | 59.73 | 76.47
| [Ours] RD (Wav2Vec 2.0) | 89.30 | 71.47 | 71.00 | 54.47 | 80.61 | 73.68 | 72.12 | 60.32 | 37.58 | 46.92 | 49.71 | 62.51
| [Ours] RD (HuBERT) | 93.78 | 78.96 | 78.01 | 66.28 | 82.26 | 75.04 | 74.45 | 61.97 | 35.70 | 45.24 | 46.65 | 59.96
| [Ours] RD (WavLM) | 95.23 | 82.05 | 86.45 | 72.21 | 82.46 | 75.93 | 75.38 | 62.88 | 35.88 | 43.94 | 45.85 | 58.55
| [Ours] RD (Robust HuBERT) | 95.81 | 81.12 | 84.16 | 64.59 | 83.27 | 75.81 | 74.89 | 62.44 | 34.42 | 43.73 | 46.72 | 58.82
| Distillation-_plus_ -Pruning recipes
| LightHuBERT [19] | 98.60 | 83.73 | 88.19 | 70.45 | 87.70 | 74.82 | 79.78 | 60.67 | 26.04 | 42.24 | 38.57 | 58.33
| DPHuBERT [9] | 97.89 | 73.45 | 72.29 | 50.25 | 87.41 | 72.91 | 68.74 | 44.37 | 28.36 | 44.85 | 52.17 | 72.89
| DPWavLM [9] | 98.58 | 83.94 | 77.72 | 61.24 | 88.03 | 78.25 | 75.05 | 55.23 | 25.93 | 38.72 | 44.78 | 63.15
| [Ours] RD (DPWavLM) | 98.39 | 92.33 | 92.67 | 83.18 | 88.21 | 82.69 | 84.03 | 72.69 | 26.80 | 34.84 | 34.67 | 47.72

####  V-A4 Robustness of Paralinguistic-related Tasks

Finally, for the ER task, Table V-A4 summarizes our findings. Compression was
shown to be fairly resilient to the noisy conditions and resulted in
recognition accuracy in line with those obtained with the Teacher models. The
RobustDistiller method achieved the highest accuracy of all tested
distillation-only compression models and was the closest to the accuracy of
the Teacher models, despite requiring four times fewer parameters. FitHuBERT
showed the best accuracy on the noise-only scenario among the compressed
models.

TABLE IV: Experimental results for the paralinguistic task (emotion
recognition) on the SUPERB. Bold represents the best result and underline the
second best for the compressed models.

| Upstream |  ER \- ACC $\uparrow$
---|---|---
| (c) | (n) | (r) | (n + r)
| Teachers
| Wav2Vec 2.0 Base [2] | 63.01 | 51.46 | 32.70 | 27.02
| HuBERT Base [3] | 64.75 | 53.45 | 40.72 | 29.12
| WavLM Base+ [4] | 67.96 | 55.08 | 36.57 | 27.14
| Robust HuBERT [5] | 64.59 | 57.17 | 41.44 | 33.36
| Distillation-only recipes
| Distiller (Wav2Vec 2.0) [7] | 61.01 | 49.94 | 24.14 | 23.69
| Distiller (HuBERT) [7] | 62.55 | 47.89 | 31.35 | 25.77
| Distiller (WavLM) [7] | 62.09 | 51.14 | 29.24 | 25.27
| Distiller (Robust HuBERT) [7] | 64.02 | 51.60 | 29.29 | 25.08
| FitHuBERT [20] | 61.35 | 52.17 | 36.25 | 28.06
| [Ours] RD (Wav2Vec 2.0) | 63.57 | 50.50 | 40.08 | 29.98
| [Ours] RD (HuBERT) | 61.46 | 48.06 | 35.40 | 27.28
| [Ours] RD (WavLM) | 62.76 | 51.50 | 39.73 | 32.65
| [Ours] RD (Robust HuBERT) | 62.81 | 52.10 | 34.72 | 27.36
| Distillation-_plus_ -Pruning recipes
| LightHuBERT [19] | 63.88 | 48.72 | 35.51 | 29.84
| DPHuBERT [9] | 63.06 | 52.90 | 35.45 | 27.44
| DPWavLM [9] | 65.61 | 57.58 | 36.87 | 29.96
| [Ours] RD (DPWavLM) | 64.87 | 61.10 | 59.78 | 49.04

####  V-A5 Generalizing to other Teacher-Student recipes

The last subgroup in Tables V to V-A4 describes the results obtained from
models that use distillation-plus-pruning recipes. Empirically, these models
achieve the best overall performance among the compressed models, especially
on content-related tasks (e.g., ASR). Beyond a similar generalization to its
Teacher on clean scenarios, these models also exhibit some degree of
robustness to adverse conditions. For instance, the LightHuBERT model is one
of the most resilient to the reverberation condition, sometimes even
outperforming its Teacher, the HuBERT model. In this work, to show how
flexible our recipe is, we selected the DPWavLM model due to its excellent
generalization capabilities and feasibility for training under academic
resources. DPWavLM consists of two stages. First, distillation and pruning are
jointly performed. In the second stage, only distillation is used to adjust
the Student. We specifically insert our two modifications of the
RobustDistiller in the second stage.

Our results show that the RobustDistiller recipe can increase the robustness
of the Student model and help with the generalization for clean scenarios, as
observed in several tasks such as ASR, KS, SID, and SD. For the noisy
scenarios, the gap between clean and noisy settings is significantly smaller.
For instance, the RobustDistiller on top of the DPWavLM in the noise-plus-
reverberation scenario for the IC task has a drop of 15.21% percentage points
in accuracy. In contrast, Robust HuBERT, a robust model four times the size of
our Student, has a drop of 27.56%.

####  V-A6 Overall Performance across tasks and scenarios

Lastly, Table V-A6 shows the overall score (Eq. 1) grouped by task type and
evaluation scenario. We also present the number of parameters and the number
of multiply-accumulate operations (MACs) for each model. As expected for
models tested under clean settings, the Teacher models have better overall
performance when compared to their smaller counterparts. For the compressed
models, we observe that distillation-plus-pruning takes the lead across clean
tasks, with a commendable performance for DPWavLM and LightHuBERT.
Nonetheless, the proposed RobustDistiller shows that it can not only improve
“in-the-wild” scenarios, but also help the model generalize better for clean
settings. For instance, for content- and speaker-related tasks, DPWavLM
trained with the proposed recipe improves the overall performance under clean
scenarios, with 17.15% and 20.22% increase in the score, respectively.
Speaker-related tasks, in general, greatly benefit from the proposed robust
distillation recipe for both distillation-plus-pruning and distillation-only
methods.

For noisy conditions, we observe that RobustDistiller can drastically improve
across all four groups of tasks, sometimes even surpassing robust Teacher
models, such as WavLM and Robust HuBERT. Comparing non-robust distillation-
only methods with the proposed RobustDistiller on semantics-related tasks with
reverberation, we can observe a Score gain of 2.6 times. Similar results can
be observed on speaker-based tasks. Overall, the gains achieved with
RobustDistiller motivate the development of distillation methods with a
robustness-first principle during its development.

TABLE V: Overall Score results for each scenario on the original 10 SUPERB
tasks grouped by category. Bold represents the best result and underline the
second best for the compressed models.

| Upstream | #params | MACs | Content-related Tasks | Speaker-related Tasks | Semantic-related Tasks | Paralinguistics-related Tasks
---|---|---|---|---|---|---|---
| (M) | (G) | (c) | (n) | (r) | (n + r) | (c) | (n) | (r) | (n + r) | (c) | (n) | (r) | (n + r) | (c) | (n) | (r) | (n + r)
| Teachers
| Wav2Vec 2.0 Base [2] | 95 | 1669 | 841 | 686 | 750 | 712 | 459 | 221 | 260 | 214 | 671 | 576 | 668 | 533 | 289 | 270 | 240 | 131
| HuBERT Base [3] | 95 | 1669 | 702 | 630 | 605 | 600 | 605 | 450 | 431 | 398 | 885 | 719 | 784 | 685 | 539 | 421 | 465 | 214
| WavLM Base+ [4] | 95 | 1670 | 814 | 780 | 817 | 759 | 868 | 752 | 560 | 466 | 1000 | 908 | 974 | 857 | 1000 | 545 | 349 | 136
| Robust HuBERT [5] | 95 | 1669 | 604 | 827 | 726 | 697 | 651 | 678 | 431 | 405 | 921 | 982 | 876 | 817 | 516 | 703 | 485 | 382
| Distillation-only recipes
| Distiller (Wav2Vec 2.0) [7] | 23 | 786 | 123 | 0 | 37 | 123 | 248 | 143 | 0 | 0 | 71 | 0 | 0 | 0 | 0 | 155 | 0 | 0
| Distiller (HuBERT) [7] | 23 | 786 | 289 | 314 | 242 | 254 | 371 | 338 | 254 | 247 | 393 | 160 | 210 | 90 | 222 | 0 | 202 | 82
| Distiller (WavLM) [7] | 23 | 786 | 362 | 461 | 290 | 224 | 579 | 515 | 131 | 123 | 529 | 388 | 233 | 154 | 156 | 246 | 143 | 62
| Distiller (Robust HuBERT) [7] | 23 | 786 | 405 | 405 | 132 | 121 | 471 | 483 | 148 | 132 | 609 | 431 | 161 | 91 | 433 | 281 | 145 | 55
| FitHuBERT [20] | 22 | 3122 | 395 | 455 | 365 | 331 | 67 | 126 | 257 | 219 | 559 | 442 | 565 | 408 | 50 | 324 | 340 | 172
| [Ours] RD (Wav2Vec 2.0) | 23 | 786 | 195 | 628 | 740 | 695 | 413 | 702 | 885 | 898 | 127 | 549 | 688 | 647 | 368 | 198 | 447 | 248
| [Ours] RD (HuBERT) | 23 | 786 | 265 | 781 | 851 | 871 | 555 | 819 | 928 | 942 | 375 | 649 | 766 | 758 | 65 | 13 | 316 | 141
| [Ours] RD (WavLM) | 23 | 786 | 346 | 700 | 795 | 833 | 633 | 830 | 915 | 921 | 434 | 699 | 840 | 814 | 253 | 273 | 437 | 354
| [Ours] RD (Robust HuBERT) | 23 | 786 | 363 | 742 | 839 | 785 | 499 | 792 | 908 | 909 | 502 | 690 | 816 | 753 | 259 | 319 | 297 | 145
| Distillation-_plus_ -Pruning recipes
| LightHuBERT [19] | 27 | 861 | 538 | 585 | 754 | 652 | 473 | 345 | 634 | 546 | 861 | 717 | 903 | 792 | 413 | 63 | 319 | 243
| DPHuBERT [9] | 24 | 654 | 459 | 387 | 451 | 489 | 516 | 380 | 420 | 355 | 788 | 577 | 674 | 488 | 295 | 379 | 317 | 148
| DPWavLM [9] | 24 | 589 | 462 | 524 | 470 | 512 | 638 | 614 | 321 | 266 | 870 | 785 | 774 | 672 | 662 | 734 | 357 | 247
| [Ours] RD (DPWavLM) | 24 | 589 | 541 | 792 | 845 | 852 | 766 | 917 | 1000 | 1000 | 854 | 945 | 972 | 1000 | 556 | 1000 | 1000 | 1000

###  V-B Performance on SUPERB-SG Tasks: Speech Enhancement and Source
Separation

For downstream generative tasks, our assumption is that feature denoising KD
and multi-task learning will lead to improved disentanglement in the latent
space, resulting in improved results. Table V-B reports the results obtained
for the SE and SS tasks. One of the first observations for these tasks is that
compressed models do not have a significant performance gap compared to their
Teacher model, as has happened on other tasks. For distillation-only recipes,
the proposed modifications led to improvements in both SE and SS compared to
their original recipe. For distillation-plus-pruning recipes, however, there
is no clear pattern on whether RobustDistiller helps or not. Furthermore,
examining the overall results for smaller models, robust or not, these
experiments suggest that there is still room for improvements on generative
tasks.

TABLE VI: Experimental results on the SUPERB-SG benchmark for Speech
Enhancement (SE) and Source Separation (SS) tasks. Bold represents the best
result and underline the second best for the compressed models.

| Upstream | SE | SS | Score $(\uparrow)$
---|---|---|---|---
| STOI $(\uparrow)$ | PESQ $(\uparrow)$ | SI-SDR $(\uparrow)$ | SI-SDR $(\uparrow)$
| Teachers
| Wav2Vec 2.0 Base [2] | 94.83 | 2.93 | 9.34 | 10.02 | 715.07
| HuBERT Base [3] | 94.86 | 2.98 | 9.05 | 9.97 | 726.45
| WavLM Base+ [4] | 95.17 | 3.04 | 9.41 | 11.37 | 988.09
| Robust HuBERT [5] | 94.93 | 3.00 | 9.16 | 9.86 | 751.27
| Distillation-only recipes
| Distiller (Wav2Vec 2.0) [7] | 94.71 | 2.91 | 9.41 | 9.48 | 633.51
| Distiller (HuBERT) [7] | 94.76 | 2.96 | 9.24 | 9.39 | 656.65
| Distiller (WavLM) [7] | 94.88 | 2.98 | 8.71 | 10.03 | 702.04
| Distiller (Robust HuBERT) [7] | 94.85 | 2.94 | 9.44 | 9.32 | 671.14
| FitHuBERT [20] | 94.31 | 2.79 | 7.75 | 6.02 | 0.00
| [Ours] RD (Wav2Vec 2.0) | 94.89 | 2.99 | 9.19 | 10.01 | 752.93
| [Ours] RD (HuBERT) | 94.90 | 3.00 | 9.47 | 9.80 | 770.31
| [Ours] RD (WavLM) | 94.95 | 2.99 | 9.46 | 10.17 | 802.38
| [Ours] RD (Robust HuBERT) | 94.82 | 2.99 | 9.14 | 9.73 | 711.64
| Distillation-_plus_ -Pruning recipes
| LightHuBERT [19] | 94.72 | 2.96 | 9.11 | 9.91 | 683.06
| DPHuBERT [9] | 94.81 | 2.92 | 9.09 | 9.80 | 662.85
| DPWavLM [9] | 94.92 | 3.02 | 9.52 | 10.56 | 861.84
| [Ours] RD (DPWavLM) | 94.95 | 3.00 | 9.53 | 10.47 | 846.80

###  V-C Speech-noise disentanglement

The effects of feature denoising knowledge distillation and the multi-task
learning during distillation proved especially useful in clean and noisy
settings. Initially, we hypothesized that both paradigms helped the model to
better generalize by reinforcing the disentanglement of speech features from
noise factors. In Figure 2, we further explore this hypothesis and test the
effects that noise has on the latent space of three different models: HuBERT,
Distiller (HuBERT), and RobustDistiller (HuBERT), from top to bottom,
respectively. Left-side plots are for the clean utterances and right-side for
noise under a 0 dB SNR setting. Each individual point is an utterance, and its
shape/color represents one out of eight randomly-selected speakers from the
LibriSpeech dataset. First, we compute a global average pooling of each frame
of the tested representation. Then we reduce the dimensionality of the signal
to two dimensions via a tSNE transformation. As can be seen, the
representations are able to accurately discriminate the speakers in the clean
condition. In turn, under the noisy setting, only the RobustDistiller method
preserves the disentanglement across speakers.

Figure 2: tSNE visualization on the effect of noise on the latent
representation of (top to bottom) HuBERT, Distiller (HuBERT), and
RobustDistiller (HuBERT), respectively. Clean signals are shown on the left,
while noisy on the right. Data are from eight randomly-selected speakers of
LibriSpeech.

###  V-D Impact of the Enhancement Head

Figure 3: Measuring the impact of different enhancement heads and loss
functions for our proposed multi-task learning paradigm.

To evaluate the effectiveness of the proposed modification #2 (i.e., adding
the enhancement head), we conducted experiments with models capable of
reconstructing either the original waveform or the clean STFT magnitude, using
different enhancement losses. During the distillation phase, we measured
reconstruction metrics (e.g., PESQ and SI-SDR) every 50 steps for each model
configuration. Once the distilled model was obtained, we fine-tuned it for
three different tasks: KS, IC, and ER.

Figure 3 (i) depicts the results for the IC task under the four tested
scenarios. The dashed lines indicate the outcome for a model without the
enhancement head but which has the feature denoising knowledge distillation
step. Figure 3 (ii) and (iii) display the pretraining PESQ and SI-SDR,
respectively.

As can be seen, the combination that achieves the best overall performance
relies on an enhancement head that estimates the binary mask for the STFT
magnitude and uses an L1-loss function. This result highly correlates with the
best reconstruction metrics observed during the pretraining phase. We
hypothesize that, for small-capacity enhancement heads that estimate a binary
mask, the greater the reconstruction during the training, the better the final
disentanglement because the Student model is learning to preserve more speech
features in their latent space. In particular, for the IC task in Figure 3(i),
we observe an improvement in three of the four tested scenarios using the
enhancement head. For the reverberation scenario, the usage of our proposed
enhancement head slightly improved upon the setting where we use only feature
denoising KD; thus, a limitation of this work is to experiment with better
architectures that can handle better dereverberation.

Despite achieving poor reconstruction metrics during pretraining, the second-
best overall result across the three tasks goes for the model reconstructing
the raw waveform directly and using an L1-loss function. Going from a very
compressed latent representation to a high-dimensional signal (i.e., raw
waveform) is difficult. Furthermore, inserting a more robust enhancement head
can soften the job from the Student model, thus undermining the full potential
of creating an encoder that disentangles noise from speech features as we
planned.

###  V-E Discussion on distilling from already robust Teachers

One important question that may arise from the analysis herein is the
following: What if we distill a representation that was built to be robust; do
we get better results than developing a robust distillation process? To answer
this question, we zoom into the results in Table V-A6. In this analysis, we
consider the WavLM and Robust HuBERT models to be robust Teachers since both
were augmented with noise during the training phase. First, considering
distillation-only recipes, we observe that the Distiller recipe with HuBERT
and Robust HuBERT leads to the former Student being more robust in the noise-
only setting. However, both have a huge gap compared to the Teacher model.
When applying our methodology, RobustDistiller, we observe a large improvement
in all three noisy scenarios.

Interestingly, RobustDistiller with HuBERT or the Robust HuBERT models
generate Students with similar performance. Thus, a two-step process of first
making the Teacher model and then distilling it is not necessary. It is
computationally better to distill a non-robust model directly with
RobustDistiller, and we can achieve similar results.

Lastly, similar results can be observed when using the distillation-plus-
pruning methods. From the beginning, DPWavLM can preserve much of the
robustness of WavLM. However, if trained with RobustDistiller, we can further
improve its performance in noisy settings and also its generalizability in
clean scenarios, as observed for content- and speaker-related tasks.

###  V-F Effect of noise type in low SNR regimes

Results above have shown the sensitivity of different upstream models to the
presence of noise during testing. An in-depth analysis showed that this effect
was particularly true at SNR regimes below an SNR of 10 dB. Here, we explore
if noise type under these SNR regimes play a role on final system accuracy.
For simplicity, here we focus only on the KS task and noise levels at the SNR
range of $[-5,5]$ dB. The DCASE2020 noise dataset used for testing was
comprised of noise signals under three categories: indoor (e.g., shopping
mall, airport), outdoor (e.g., park, public square), and transportation (e.g.,
bus, metro, tram). Figure 4 shows the accuracy achieved under the clean
condition, as well as the three different noise types for the WavLM Base+
Teacher model, as well as this model distilled with the benchmark distiller
and the proposed RobustDistiller frameworks. As can be seen, for the benchmark
systems, indoor noise causes the most significant decline in accuracy, which
could be attributed to the combined impact of reverberation and additive
noise. Transportation noise had the least impact on the benchmark methods. The
proposed approach, in turn, resulted not only in the highest overall accuracy
across the three noise types, but showed to be independent of noise type, thus
further validating the noise robustness properties of the proposed solution.

Figure 4: KS accuracy per noise type with clean speech levels used for
referencing.

###  V-G Effects of room size

Lastly, we explore the impact of room size on system performance. Room sizes
are directly proportional to reverberation times, with larger rooms typically
resulting in lower performance. For this investigation, we use the IC task and
HuBERT-based variations and the recorded RIRs from small, medium, and large-
sized rooms, as described in Section IV-A. Figure 5 depicts the accuracy
achieved for each of the three room sizes. As expected, accuracy decreases as
room sizes increase. The two proposed RobustDistiller setups achieved the
highest accuracy across the three room sizes, with accuracy values at times
outperforming the Teacher model, whilst requiring just one-quarter of the
parameters. These findings are promising and suggesting that robust
distillation for edge speech applications may indeed be possible via a robust
distillation process.

Figure 5: IC performance per room size for different HuBERT distillation
schemes.

##  VI Conclusion

In this work, we describe RobustDistiller, a methodology to improve the
environmental robustness of the distillation process of speech representation
learning algorithms. In particular, two modifications are proposed to original
distillation recipes: (i) feature denoising knowledge distillation; and (ii)
multi-task learning, where a speech enhancement step is jointly performed to
reinforce the disentanglement of speech features from noise factors. We show
that our recipe can be adapted to both layerwise distillation (e.g.,
DistilHuBERT) and distillation-plus-pruning (e.g., DPHuBERT) methods. We test
the robustness of our model across clean and noisy conditions on top of 12
speech-processing tasks from SUPERB and SUPERB-SG. Experimental results show
the proposed method frequently outperforming the original Teacher models with
roughly four times fewer parameters. Overall, the proposed RobustDistiller on
top of the DPWavLM recipe resulted in the best performance across tasks, thus
suggesting that their use could be suitable for “in-the-wild” edge speech
applications.

## References

  * [1] A. Mohamed, H.-y. Lee, L. Borgholt, J. D. Havtorn, J. Edin, C. Igel, K. Kirchhoff, S.-W. Li, K. Livescu, L. Maaløe, T. N. Sainath, and S. Watanabe, “Self-supervised speech representation learning: A review,” _IEEE Journal of Selected Topics in Signal Processing_ , vol. 16, no. 6, pp. 1179–1210, 2022.
  * [2] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 12 449–12 460, 2020.
  * [3] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 29, pp. 3451–3460, 2021.
  * [4] S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao _et al._ , “Wavlm: Large-scale self-supervised pre-training for full stack speech processing,” _IEEE Journal of Selected Topics in Signal Processing_ , 2022.
  * [5] K. P. Huang, Y.-K. Fu, Y. Zhang, and H. yi Lee, “Improving Distortion Robustness of Self-supervised Speech Processing Tasks with Domain Adaptation,” in _Proc. Interspeech 2022_ , 2022, pp. 2193–2197.
  * [6] A. Babu, C. Wang, A. Tjandra, K. Lakhotia, Q. Xu, N. Goyal, K. Singh, P. von Platen, Y. Saraf, J. Pino _et al._ , “Xls-r: Self-supervised cross-lingual speech representation learning at scale,” _arXiv preprint arXiv:2111.09296_ , 2021.
  * [7] H.-J. Chang, S.-w. Yang, and H.-y. Lee, “Distilhubert: Speech representation learning by layer-wise distillation of hidden-unit bert,” in _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2022, pp. 7087–7091.
  * [8] H. Guimarães, A. Pimentel, A. Avila, M. Rezagholizadeh, and T. H. Falk, “An exploration into the performance of unsupervised cross-task speech representations for ”in the wild” edge applications,” 2023.
  * [9] Y. Peng, Y. Sudo, S. Muhammad, and S. Watanabe, “DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech Models,” in _Proc. INTERSPEECH 2023_ , 2023, pp. 62–66.
  * [10] H. R. Guimarães, A. Pimentel, A. R. Avila, M. Rezagholizadeh, B. Chen, and T. H. Falk, “Robustdistiller: Compressing universal speech representations for enhanced environment robustness,” in _ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2023, pp. 1–5.
  * [11] E. Jang, S. Gu, and B. Poole, “Categorical reparameterization with gumbel-softmax,” in _International Conference on Learning Representations_ , 2017. [Online]. Available: https://openreview.net/forum?id=rkE3y85ee
  * [12] S.-w. Yang, P.-H. Chi _et al._ , “Superb: Speech processing universal performance benchmark,” _arXiv preprint arXiv:2105.01051_ , 2021.
  * [13] H.-S. Tsai, H.-J. Chang, W.-C. Huang, Z. Huang, K. Lakhotia, S.-w. Yang, S. Dong, A. Liu, C.-I. Lai, J. Shi, X. Chang, P. Hall, H.-J. Chen, S.-W. Li, S. Watanabe, A. Mohamed, and H.-y. Lee, “SUPERB-SG: Enhanced speech processing universal PERformance benchmark for semantic and generative capabilities,” in _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , S. Muresan, P. Nakov, and A. Villavicencio, Eds. Dublin, Ireland: Association for Computational Linguistics, May 2022, pp. 8479–8492. [Online]. Available: https://aclanthology.org/2022.acl-long.580
  * [14] T.-h. Feng, A. Dong, C.-F. Yeh, S.-w. Yang, T.-Q. Lin, J. Shi, K.-W. Chang, Z. Huang, H. Wu, X. Chang _et al._ , “Superb@ slt 2022: Challenge on generalization and efficiency of self-supervised speech representation learning,” in _2022 IEEE Spoken Language Technology Workshop (SLT)_. IEEE, 2023, pp. 1096–1103.
  * [15] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: an asr corpus based on public domain audio books,” in _2015 IEEE international conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 2015, pp. 5206–5210.
  * [16] J. Kahn, M. Rivière _et al._ , “Libri-light: A benchmark for ASR with limited or no supervision,” in _Proc. IEEE ICASSP_ , 2020, pp. 7669–7673.
  * [17] W.-N. Hsu, A. Sriram, A. Baevski, T. Likhomanenko, Q. Xu, V. Pratap, J. Kahn, A. Lee, R. Collobert, G. Synnaeve _et al._ , “Robust wav2vec 2.0: Analyzing domain shift in self-supervised pre-training,” _arXiv preprint arXiv:2104.01027_ , 2021.
  * [18] Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. March, and V. Lempitsky, “Domain-adversarial training of neural networks,” _Journal of Machine Learning Research_ , vol. 17, no. 59, pp. 1–35, 2016. [Online]. Available: http://jmlr.org/papers/v17/15-239.html
  * [19] R. Wang, Q. Bai, J. Ao, L. Zhou, Z. Xiong, Z. Wei, Y. Zhang, T. Ko, and H. Li, “Lighthubert: Lightweight and configurable speech representation learning with once-for-all hidden-unit bert,” _arXiv preprint arXiv:2203.15610_ , 2022.
  * [20] Y. Lee, K. Jang, J. Goo, Y. Jung, and H. R. Kim, “FitHuBERT: Going Thinner and Deeper for Knowledge Distillation of Speech Self-Supervised Models,” in _Proc. Interspeech 2022_ , 2022, pp. 3588–3592.
  * [21] T. Ashihara, T. Moriya, K. Matsuura, and T. Tanaka, “Deep versus Wide: An Analysis of Student Architectures for Task-Agnostic Knowledge Distillation of Self-Supervised Speech Models,” in _Proc. Interspeech 2022_ , 2022, pp. 411–415.
  * [22] K.-P. Huang, Y.-K. Fu, T.-Y. Hsu, F. R. Gutierrez, F.-L. Wang, L.-H. Tseng, Y. Zhang, and H.-y. Lee, “Improving generalizability of distilled self-supervised speech processing models under distorted settings,” in _2022 IEEE Spoken Language Technology Workshop (SLT)_ , 2023, pp. 1112–1119.
  * [23] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar _et al._ , “Bootstrap your own latent-a new approach to self-supervised learning,” _Advances in neural information processing systems_ , vol. 33, pp. 21 271–21 284, 2020.
  * [24] M. L. Seltzer, D. Yu, and Y. Wang, “An investigation of deep neural networks for noise robust speech recognition,” in _2013 IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2013, pp. 7398–7402.
  * [25] H. Wang, Y. Qian, X. Wang, Y. Wang, C. Wang, S. Liu, T. Yoshioka, J. Li, and D. Wang, “Improving noise robustness of contrastive speech representation learning with speech reconstruction,” in _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2022, pp. 6062–6066.
  * [26] A. Rix, J. Beerends, M. Hollier, and A. Hekstra, “Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs,” in _2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.01CH37221)_ , vol. 2, 2001, pp. 749–752 vol.2.
  * [27] D. Snyder, G. Chen, and D. Povey, “MUSAN: A Music, Speech, and Noise Corpus,” 2015.
  * [28] J. Salamon, C. Jacoby, and J. P. Bello, “A dataset and taxonomy for urban sound research,” in _22nd ACM International Conference on Multimedia (ACM-MM’14)_ , Orlando, FL, USA, Nov. 2014, pp. 1041–1044.
  * [29] T. Ko, V. Peddinti, D. Povey, M. L. Seltzer, and S. Khudanpur, “A study on data augmentation of reverberant speech for robust speech recognition,” in _Proc. IEEE ICASSP_ , 2017, pp. 5220–5224.
  * [30] A. Mesaros, T. Heittola, and T. Virtanen, “A multi-device dataset for urban acoustic scene classification,” in _Proceedings of the Detection and Classification of Acoustic Scenes and Events 2018 Workshop (DCASE2018)_ , November 2018, pp. 9–13. [Online]. Available: https://dcase.community/documents/workshop2018/proceedings/DCASE2018Workshop\\_Mesaros\\_8.pdf
  * [31] A. Pimentel, H. R. Guimarães, A. Avila, and T. H. Falk, “Environment-aware knowledge distillation for improved resource-constrained edge speech recognition,” _Applied Sciences_ , vol. 13, no. 23, 2023. [Online]. Available: https://www.mdpi.com/2076-3417/13/23/12571
  * [32] C. H. Taal, R. C. Hendriks, R. Heusdens, and J. Jensen, “An algorithm for intelligibility prediction of time–frequency weighted noisy speech,” _IEEE Transactions on Audio, Speech, and Language Processing_ , vol. 19, no. 7, pp. 2125–2136, 2011.
  * [33] J. Le Roux, S. Wisdom, H. Erdogan, and J. R. Hershey, “Sdr–half-baked or well done?” in _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2019, pp. 626–630.
  * [34] T. H. Falk and W.-Y. Chan, “Modulation spectral features for robust far-field speaker identification,” _IEEE Transactions on Audio, Speech, and Language Processing_ , vol. 18, no. 1, pp. 90–100, 2010.
  * [35] A. Nagrani, J. S. Chung, and A. Zisserman, “Voxceleb: a large-scale speaker identification dataset,” in _INTERSPEECH_ , 2017.

|  Heitor R. Guimarães (Student Member, IEEE) received his B.Eng. degree in
Computer and Information Engineering from the Federal University of Rio de
Janeiro (UFRJ); and the M.Sc. degree in Electrical Engineering from the
University of São Paulo (USP). He is pursuing a Ph.D. in Telecommunications at
the Institut National de la Recherche Scientifique (INRS - EMT). His research
interests include self-supervised learning, domain adaptation, adversarial
attacks, and efficient deep learning (e.g., model compression) for speech
applications.
---|---
|  Arthur Pimentel received his B.Sc. degree in electronics engineering from
the Federal University of Pernambuco (UFPE), Recife, Brazil, in 2021; and the
M.Sc. degree in Telecommunications at the Institut National de la Recherche
Scientifique (INRS - EMT). He is pursuing a Ph.D. in Telecommunications at
INRS - EMT. His research interests include signal processing, self-supervised
speech representation learning and deepfake detection.
---|---
|  Anderson Avila is an Assistant Professor at INRS-EMT, working in the INRS-
UQO Joint Research Unit in Cybersecurity. His research background is on
machine learning and signal processing applied to natural language processing.
During his PhD, Dr. Avila worked on the development of new models for speech
quality assessment and on the robustness of voice biometrics. Prior to joining
INRS-UQO, Dr. Avila was a researcher scientist in natural language and speech
processing, working on projects related to model compression, low-latency and
robustness of spoken language understanding. Dr. Avila received his BSc in
Computer Science from the Federal University of São Carlos, his MSc from
Federal University of ABC and his PhD from INRS.
---|---
|  Mehdi Rezagholizadeh obtained a PhD from McGill University in Electrical
and Computer Engineering (Centre of Intelligent Machines) in 2016. He joined
Huawei in January 2017 and his research focus has been on different deep
learning and NLP projects such as generative adversarial networks, neural
machine translation, adversarial neural text generation, and efficient NLP for
pre-trained language models. He is now a principal research scientist and he
has been leading the NLP team of Huawei Noah’s Ark Lab in Canada since 2018.
Mehdi has contributed to more than 15 patents and 50 published papers in top
journals and conferences. Moreover, Mehdi has organized several academic and
industrial workshops such as NeurIPS 2021, 2022, and 2023 ENLSP workshops. He
has served as technical committee member of ACL Rolling Review 21, ACL
(22,23), EMNLP (20,21,22), NAACL 21, and EACL 21.
---|---
|  Boxing Chen is the Chief Researcher of Montreal Research Centre for Huawai
Technologies Canada, where he focuses on natural language processing and
machine learning. Prior to joining Huawai, Dr. Chen held various positions,
including Senior Staff Algorithm Expert at Alibaba Group’s Machine
Intelligence Lab and Research Officer at the National Research Council Canada
(NRC), etc. He has co-authored over 100 papers in AI conferences and journals,
received the ”Best Paper Award” at MT Summit 2013 and a nomination for the
same award at ACL 2013. He has also served as an area chair for conferences
such as ACL, EMNLP, COLING, and AACL, and his teams have secured first place
in over 20 machine translation competitions.
---|---
|  Tiago H. Falk received the B.Sc. degree in electronics engineering from
UFPE, Recife, Brazil, in 2002, and the M.Sc. and Ph.D. degrees in electrical
and computer engineering from Queen’s University, Kingston, ON, Canada, in
2005 and 2008, respectively. From 2008 to 2010, he was a Postdoctoral Fellow
with the University of Toronto, Toronto, ON, Canada. Since 2010, he has been
with INRS-EMT, University of Quebec, Quebec city, QC, Canada, where he is
currently a tenured Full Professor, the Director of the
Multimodal/Multisensory Signal Analysis and Enhancement Lab, and co-Director
of the INRS-UQO Joint Research Unit on Cybersecurity. His research interests
include signal processing and machine learning for next-generation secure
human-machine interfaces.
---|---