# Whisper-SV: Adapting Whisper for Low-data-resource Speaker Verification

Li Zhang1,∗, Ning Jiang2,∗ , Qing Wang1, Yue Li1, Quan Lu2, and Lei Xie1,#
1Audio, Speech and Language Processing Group (ASLP@NPU), School of Computer
Science,
Northwestern Polytechnical University (NPU), Xi’an, China
2Mashang Consumer Finance Co.,Ltd, Beijing, China * Co-first author.#
Corresponding author.

###### Abstract

Trained on 680,000 hours of massive speech data, Whisper is a multitasking,
multilingual speech foundation model demonstrating superior performance in
automatic speech recognition, translation, and language identification.
However, its applicability in speaker verification (SV) tasks remains
unexplored, particularly in low-data-resource scenarios where labeled speaker
data in specific domains are limited. To fill this gap, we propose a
lightweight adaptor framework to boost SV with Whisper, namely Whisper-SV.
Given that Whisper is not specifically optimized for SV tasks, we introduce a
representation selection module to quantify the speaker-specific
characteristics contained in each layer of Whisper and select the top-k layers
with prominent discriminative speaker features. To aggregate pivotal speaker-
related features while diminishing non-speaker redundancies across the
selected top-k distinct layers of Whisper, we design a multi-layer aggregation
module in Whisper-SV to integrate multi-layer representations into a singular,
compacted representation for SV. In the multi-layer aggregation module, we
employ convolutional layers with shortcut connections among different layers
to refine speaker characteristics derived from multi-layer representations
from Whisper. In addition, an attention aggregation layer is used to reduce
non-speaker interference and amplify speaker-specific cues for SV tasks.
Finally, a simple classification module is used for speaker classification.
Experiments on VoxCeleb1, FFSVC, and IMSV datasets demonstrate that Whisper-SV
achieves EER/minDCF of 2.22%/0.307, 6.14%/0.488, and 7.50%/0.582,
respectively, showing superior performance in low-data-resource SV scenarios.

###### Index Terms:

Whisper, speaker verification, adaptor, low-data-resource

##  I Introduction

Speaker verification (SV) is the process of confirming if an individual is who
they claim to be [1]. In recent years, deep learning has shown remarkable
success in SV tasks. However, current methods often rely on large amounts of
labeled training speech [2, 3, 4, 5, 6], and the performance significantly
declines in challenging scenarios with limited speaker labeled data, such as
far-field and multi-language SV tasks. This decline is primarily due to the
scarcity of large-scale speech datasets with speaker labels in these low-data-
resource scenarios [7, 8] and the lack of robustness in speaker models trained
on conventional Mel-Frequency Cepstral Coefficients (MFCC) and Filter Bank
(Fbank) features.

In general, data augmentation is commonly used in low-data-resource scenarios.
Data augmentation includes various transformations of speech (e.g. add noise
[9], reverberation [10], speed perturbation [11] and SpecAug [12]) and
introducing out-of-domain datasets which may lead to domain mismatch problems
[13, 14]. In addition to the above data augmentation methods, leveraging pre-
trained speech foundation models for downstream low-data-resource tasks has
recently become increasingly popular due to their extensive exposure to
massive amounts of data, enhancing their generalization capabilities for
various tasks. Chen and Rudnicky [15] explored wav2vec2.0 fine-tuning for
improved low-data-resource speech emotion recognition. Gupta et al. [16]
enhanced language identification in low-data-resource languages (Indian) by
exploiting learned features with wav2vec2.0. Zhao and Zhang [17] exploited and
analyzed a series of pre-trained models for speech recognition in 15 low-
resource languages. Notably, using pre-trained large foundation speech models
in downstream low-data-resource tasks facilitates the effective utilization of
vast amounts of out-of-domain data to enhance model robustness and mitigate
the domain mismatch issue typically associated with introducing out-of-domain
data. Moreover, since these large foundation models are trained on extensive
out-of-domain data, they often achieve good performance in downstream tasks
without requiring a large number of trainable parameters.

Recently, a multi-task multi-lingual model called Whisper [18] was proposed,
which was an encoder-decoder transformer architecture trained on a large-scale
dataset of 680k hours of speech data, showing superior performances in various
tasks, including multilingual automatic speech recognition (ASR), speech
translation (ST), and language identification (LID). Although trained on vast
data showing great potential in low-data-resource scenarios, Whisper is
optimized for ASR, ST, and LID tasks, ignoring SV tasks that specialize in
extracting and analyzing individual vocal characteristics for identity
verification. Therefore, we aim to explore its efficacy in SV, particularly
focusing on low-data-resource SV scenarios where speaker-labeled data in the
target domain is limited.

In this paper, we propose Whisper-SV, a lightweight adaptor framework to
transfer Whisper for SV tasks, particularly benefiting low-data-resource SV
scenarios. Given that Whisper is not explicitly designed for SV tasks, it is
crucial to determine which layers of Whisper contain more discriminative
speaker characteristics for SV, and how to effectively aggregate
representations of multi-layer of Whisper for SV tasks. Consequently, Whisper-
SV is elaborately designed to comprise four modules: a pre-trained Whisper
module, a representation selection module, a multi-layer aggregation module,
and a classification module, respectively. Specifically, the pre-trained
Whisper module is employed to provide robustness and generalized
representations derived from Whisper pre-trained on diverse and massive speech
datasets. The representation selection module quantifies the speaker-specific
information within representations of each layer in Whisper and selects top-k
layers containing crucial speaker-related characteristics. Subsequently, the
multi-layer aggregation module is designed to aggregate pivotal speaker-
related information while reducing redundant features across the top-k
distinct layers of Whisper. Finally, the classification module is tasked with
speaker classification.

The contributions of this work are as follows:

  * •

We propose Whisper-SV, an adaptor framework for low-data-resource SV tasks,
leveraging a small number of trainable parameters and a limited amount of
speech data to transfer the robustness and generalization of representations
from Whisper to low-data-resource SV.

  * •

We introduce a representation selection module within Whisper-SV to quantify
the speaker-specific information in representations of each layer in Whisper
and select the top-k layers containing the most speaker-related information.
Specifically, we train individual SV models for each layer of Whisper and
evaluate their performance.

  * •

We design a multi-layer aggregation module in Whisper-SV to integrate multi-
layer representations into a singular, compacted representation for SV. This
module employs convolutional layers with shortcut connections among different
layers to refine speaker characteristics derived from multi-layer
representations. Additionally, an attention aggregation layer reduces non-
speaker information and amplifies speaker-specific cues for SV tasks.

We extensively validate Whisper-SV through experiments conducted on low-data
resource datasets (VoxCeleb1 [19], FFSVC [7], and IMSV [8]). The experimental
results demonstrate that Whisper-SV achieves superior performance in low-data-
resource SV scenarios despite having a small number of trainable parameters.

The remainder of the paper is structured as follows: Section II discusses
related works, Section III outlines Whisper-SV, Section IV presents the
experimental setup, Section V showcases results and analysis, Section VI
concludes the paper, and Section VII explores future work.

##  II Related Work

In this session, we discuss the related works of this paper from three
perspectives: the application of Whisper in downstream tasks, the usage of
pre-trained models in SV, and research on low-data-resource SV.

###  II-A Whisper for Downstream Task

Whisper is trained on massive and diverse multilingual weakly supervised
speech, specially tailored for multilingual ASR, ST, and LID tasks [18]. The
performance of Whisper achieves high-quality recognition results on various
benchmarks without fine-tuning on specific datasets. Recently, some
researchers have focused on adapting Whisper’s generalization and robustness,
which learned from massive amounts of data, to various downstream tasks. Some
studies delve into reducing the size of Whisper on streaming ST [20] and
efficient ASR [21]. Some strive to adapt Whisper for child speech recognition
[22, 23]. In addition to ASR, ST, and LID applications, some researchers are
dedicated to transferring Whisper for other applications. Kodali et al. [24]
investigate the embeddings of Whisper in the vocal intensity category. Wang et
al. [25] introduce a speech-based in-context learning (SICL) approach with
Whisper for test-time adaptation. Zezario et al. [26] leverage the acoustic
features from Whisper for robust speech assessment and Rathod et al. [27] use
the embeddings of Whisper for dysarthric severity-level classification. Ameer
et al. [28] enhance stuttered speech classification by optimizing Whisper’s
encoder layer. WhisperSeg [29] utilizes Whisper for human and animal voice
activity detection, and Berns et al. [30] investigate Whisper on speaker and
language change detection. Most of the above studies focus on utilizing
representations of Whisper for downstream classification or detection tasks,
which indicates the representations of Whisper encapsulate comprehensive
speech features, encompassing acoustic characteristics, speaker attributes,
vocal details, etc. Meanwhile, the generalization and robustness of Whisper in
various speech tasks motivate us to explore its application in SV tasks.

###  II-B Pretrained Model for SV

In the field of SV, pre-trained models usually fall into two categories,
supervised pre-trained models and self-supervised learning (SSL) models
respectively. Specifically, the supervised pre-trained models are trained with
large-scale speaker-labeled datasets [31, 3, 2] and optimized for speaker
classification tasks [32]. The supervised pre-trained models tailored for SV
are usually trained with large-scale speaker-labeled datasets (e.g. VoxCeleb
[3], AISHELL-2 [33], CNCELEB [34]). The supervised pre-trained models designed
for SV are commonly employed for initializing SV models [14, 35, 36] or
applying domain adaptation techniques [37, 38, 39] to transfer their
robustness and generalization to low-data-resource SV scenarios. The SSL
models are trained for general speech tasks (e.g. wavLM [40], Hubert [41],
wav2vec [42]). Many researchers leverage the weighted embeddings from all
transformer layers and the conv-extractor module of SSL as features to train
SV models [43, 44]. In addition, some studies focus on fine-tuning [45, 46]
partial or entire SSL models for SV tasks, which introduce a large number of
training parameters and extend the training duration. Whisper is a weak-
supervised learning model for multi-lingual ASR, ST, and LID tasks. While some
researchers have applied Whisper to various downstream tasks (e.g., speaker
detection [30], vocal intensity categorization [24], dysarthric severity-level
classification [27], etc.), its potential in SV tasks remains unexplored and
our paper aims to fill this gap.

Figure 1:  The architecture of Whisper-SV includes four modules: (a) a pre-
trained Whisper module for providing robust and generalized representations,
(b) a representation selection module for selecting the top-k layers
containing significant speaker-specific characteristics, (c) a multi-layer
aggregation module to aggregate representations from multiple layers of
Whisper, and (d) a speaker classifier module for speaker classification.

###  II-C Low-data-resource SV

Low-data-resource SV refers to developing SV systems with limited data
resources, such as those involving minor languages and far-field SV
applications. The limitation in data resources may arise from having a small
number of speakers, insufficient diversity in speech content, or a lack of
varied acoustic environments within the dataset. The primary challenge lies in
the scarcity of data that effectively captures the full spectrum of complexity
and variability inherent in real-world speech, which is essential for training
robust SV systems. To address the challenge of limited data availability, a
straightforward approach is to augment the low-resource speech data.
Conventional methods include adding noise [9] and reverberation [10], speed
perturbation [11], and SpecAug [12], among others. Some generative methods
have also been employed to perform augmentation on low-data-resource SV [47,
35]. Besides synthesizing and augmenting low-resource speech, a large amount
of out-of-domain data can also be utilized to improve speaker recognition
performance in low-resource scenarios. Whether through synthetic data
augmentation or the direct incorporation of out-of-domain speech data, the
low-resource problem is transformed into a domain mismatch problem [39, 38,
37]. Instead of introducing out-of-domain speech data, we explore the
potential of the pre-trained large model Whisper to improve low-data-resource
SV tasks.

##  III Methods

The schematic diagram of Whisper-SV is depicted in Fig. 1, encompassing four
modules: (a) a pre-trained Whisper module, (b) a representation selection
module, (c) a multi-layer aggregation module, and (d) a speaker classifier
module. The pre-trained Whisper module is to provide robust and generalized
bottleneck representations from Whisper. Given that the principal training
objective of Whisper is aligned with ASR, ST, and LID rather than SV tasks,
the representation generated by the decoder of Whisper encompasses features
that are more closely associated with ASR, ST, and LID. Consequently, in this
paper, we focus on leveraging the representations produced by the encoder of
Whisper to boost SV tasks. In addition, we incorporate a representation
selection module for quantifying the speaker-specific information in
representations of each layer in Whisper and select the top-k layers with
prominent speaker-related discriminative features for SV. The multi-layer
aggregation module is designed to integrate features across different layers
of Whisper for SV. Finally, a simple classifier module is used for speaker
classification.

###  III-A Revisiting Whisper

Whisper explores the potential of weakly supervised speech processing systems
trained on diverse and extensive speech datasets. The evaluation results of
Whisper have showcased the capacity of these systems to approach human-level
performance when provided with sufficient training data [18]. Whisper is based
on a classical encoder-decoder Transformer [48] architecture and the encoder
is formed by two convolutional layers with a kernel size of 3, followed by a
sinusoidal positional encoding, and a stacked set of Transformer blocks. The
decoder uses the learned positional embeddings and the same number of
Transformer blocks as the encoder. Whisper is particularly optimized for ASR,
ST, and LID tasks. Still, many studies delve into transferring the
generalization and robustness of Whisper to other downstream tasks (e.g.,
vocal intensity classification [24], dysarthric severity-level classification
[27], speaker activity detection [30, 29], etc.). This paper focuses on
adapting Whisper for SV tasks with a lightweight adaptor framework (Whisper-
SV). In Whisper-SV, we use the encoder of the pre-trained Whisper as the
bottleneck feature extractor for SV. As illustrated in Fig. 1 (a), the encoder
of Whisper transforms the input audio signals into a series of high-level
feature representations.

Suppose we have a set of input audios, denoted as $X$. The corresponding
speaker labels for $X$ are given by
$Y=\\{y_{1},y_{2},y_{3},\ldots,y_{i},\ldots,y_{M}\\}$, and $M$ signifies the
total number of speakers in the training set. The encoder blocks of Whisper
are $F_{W}=\\{f_{w}^{1},f_{w}^{2},f_{w}^{3}...f_{w}^{l}...f_{w}^{L}\\}$, where
$L$ is the layer number of Whisper encoder. After all input audio re-sampled
to 16,000 Hz, we extract the multi-layer bottleneck representations from pre-
trained Whisper as follows:

$F_{W}(X)=\\{f_{w}^{1}(X),f_{w}^{2}(X),f_{w}^{3}(X),...f_{w}^{l}(X)...,f_{w}^{L}(X)\\},$
(1)

where $l\in\\{1,2,3,..,L\\}$ and $F_{W}(X)$ are the collections of
representations of all layers in the Whisper encoder.

###  III-B Representation Selection Module

Considering that Whisper is not explicitly optimized for SV tasks, the
representation selection module is designed to evaluate speaker-specific
characteristics within the representations of each layer in Whisper to
identify the top-k layers with the most significant discriminative speaker
information. The representation selection module comprises a temporal pooling
layer $\rho$, evaluation models
$F_{S}=\\{f_{s}^{1},f_{s}^{2},f_{s}^{3},...,f_{s}^{i},...,f_{s}^{L}\\}$, and a
strategy to select the top-k contribution layers for SV, as depicted in Fig. 1
(b). The temporal pooling layer $\rho$ is used to pool the bottleneck
representations extracted from Whisper on the temporal dimension to the
duration of the input audio. The evaluation models $F_{S}$, trained
individually on the representations extracted from each layer of the Whisper
encoder and subsequently evaluated on the validation dataset, are employed to
quantitatively assess the speaker-related features encapsulated within the
representation of each layer in Whisper. Suppose the assessment error rates of
evaluation models are
$P_{S}=\\{P_{s}^{1},P_{s}^{2},...,P_{s}^{i},...,~{}P_{s}^{L}\\}$ and $L$ is
the layers of Whisper encoder. The error rate of the speaker model trained by
the i-th layer representations is
$P_{s}^{i}=\gamma(f_{s}^{i}(\rho(f_{w}^{i}(X))))$, where $\gamma$ is the error
metric of evaluation models $F_{S}$. $f_{s}^{i}$ is the i-th layer evaluation
model. $\rho$ is the temporal pooling layer. The collection of error rates for
speaker models trained individually on representations from each layer of
Whisper is formulated as:

$\begin{split}P_{S}=\\{\gamma(f_{s}^{i}(\rho(f_{w}^{i}(X))))\mid
i=1,2,\ldots,L\\}\end{split}$ (2)

In Eq. (2), we obtain the error rates of speaker models trained on the
representations from each layer. We then identify the top-k layers by
selecting those with the lowest error rates. The selected $k$ layers of
Whisper with the lowest error rate (the top-k layers containing the highest
speaker-specific cues) are $P_{S}^{top-k}$:

$\displaystyle
P_{S}^{top-k}=\\{P_{s}^{{}^{\prime}1},P_{s}^{{}^{\prime}2},P_{s}^{{}^{\prime}3},\ldots,P_{s}^{{}^{\prime}k}\\}$
(3) $\displaystyle\quad=\text{{argsort}}_{\text{{ascend}}}(P_{S})[1:k].$

The representations associated with $P_{S}^{top-k}$ (contain crucial speaker-
specific characteristics) in the top-k layers of Whisper are denoted as:

$\tilde{F}_{W}^{top-k}(X)=\\{\tilde{F}_{w}^{1}(X),\tilde{F}_{w}^{2}(X),\tilde{F}_{w}^{3}(X),\ldots,\tilde{F}_{w}^{k}(X)\\}.$
(4)

###  III-C Multi-layer Aggregation Module

To aggregate multi-layer representations from Whisper into a single, condensed
representation for SV, we introduce a multi-layer aggregation module to
amalgamate the representations from the top-k layers in Whisper for SV. As
illustrated in Fig. 1 (c), the multi-layer aggregation module consists of zoom
layers, multi-level aggregation layers based on 1D convolution, and an
attention aggregation layer for fusing speaker-specific cues and suppressing
speaker-irrelevant information from representations of top-k layers in
Whisper.

The zoom layers
$F_{z}=\\{F_{z}^{1},F_{z}^{2},F_{z}^{3},...,F_{z}^{l},F_{z}^{k}\\}$ are used
to scale the dimensions of representations extracted from the Whisper encoder.
The zoom layers are essentially one-dimensional convolutional layers
engineered to modulate the channel dimension to a reduced scale. The primary
function of these zoom layers is to attenuate extraneous information about
non-speakers while concurrently extracting salient discriminative features for
SV tasks. This process not only enhances the model’s specificity but also
contributes to a reduction in the overall parameter footprint of SV models.
The outputs of the zoom layers are:

$F_{z}^{top-k}(X)=\\{F_{z}^{1}(\tilde{F}_{w}^{1}(X)),F_{z}^{2}(\tilde{F}_{w}^{2}(X)),...,F_{z}^{k}(\tilde{F}_{w}^{k}(X))\\}.$
(5)

Subsequently, the multi-level aggregation layers based on 1D convolution are
introduced to aggregate speaker-specific cues across representations of top-k
layers in the Whisper encoder. Besides the top-1 representation of
$F_{z}^{1}(\tilde{F}_{w}^{1}(X))$, the other scaled representations are
individually fed into separate convolution blocks, and each convolution block
consisting of one convolution layer, one ReLU layer, and one batch
normalization layer. There are $k-1$ convolution blocks representing as
$F_{M}=\\{F_{m}^{2},...,F_{m}^{k}\\}$. To better aggregate information across
multiple layers, shortcut links are used among convolution blocks, where the
input to the convolution block of the $i$-th layer is the sum of the output
from the convolution block of the $(i-1)$-th layer and the output from the
zoom layer of the $i$-th layer. The input of the $i$-th convolutional block
is:

$F_{m}^{i}(X)=F_{m}^{(i-1)}(F_{z}^{i-1}(\tilde{F}_{w}^{i-1}(X)))\oplus
F_{z}^{i}(\tilde{F}_{w}^{i}(X)),$ (6)

where $2<i\leq k$ and $\oplus$ represents the short-cut connection. The
outputs of the $k-1$ convolution modules and the scaling layer of the top-1
layer together form the input to the attention aggregation layer.

$F_{a}(X)=\frac{1}{T}(cat[F_{z}^{1}(X),F_{m}^{2}(X),...F_{m}^{k}(X)]),$ (7)

where $F_{a}(X)\in\mathbb{R}^{B\times C\times T}$ and $B$, $C$, $T$ are the
batch size, channels, and frame numbers of the aggregated representation.

In the attention aggregation layer, we employ both squeeze-and-excitation (SE)
attention [49] and self-attention [48] to aggregate representations from
different layers of the whisper encoder for SV tasks. However, the results
show that SE attention performs better, possibly because SE attention
recalibrates feature channels, reinforcing the learning of important features
while suppressing unimportant ones. Different channels may contain different
semantic information, and SE attention can effectively distinguish the
importance of speaker-related characteristics. The formulation of the SE
attention is expressed as:

$M^{C}=\delta(W_{2}^{C\times C^{\prime}}*(relu((W_{1}^{C^{\prime}\times
C}*F_{a}(X))))),$ (8)

where $*$ means matrix multiplication. $W_{1}^{C^{\prime}\times C}$ and
$W_{2}^{C\times C^{\prime}}$ are two fully connected layers that capture the
inter-dependence of channels in the aggregated representation $F_{a}(X)$. The
dimensional reduction factor $\frac{C}{C^{\prime}}$ indicates the aggregated
representation reduction ratio to avoid the parameters overhead. Finally, a
sigmoid function $\delta$ scales the channel-wise weights. The attention mask
is a set of attention factors predicted by supervised speaker classification
loss aiming to emphasize essential speaker-specific features and compress the
non-speaker information from Whisper. The output of the multi-layer
aggregation layer is:

$F^{\prime}_{a}(X)=M^{C}\cdot F_{a}(X),$ (9)

where $\cdot$ is element-point multiplication.

###  III-D Speaker Classifier Module

As illustrated in Fig. 1 (d), the speaker classifier module consists of a
convolutional layer, an attention statistical pooling (ASP) layer [4], a
linear classification layer, and AAM-softmax [50] as the loss function.
Suppose the parameters of the speaker classifier module are $F_{c}$, and the
final prediction speaker labels are $F_{c}(F^{\prime}_{a}(X))$. The loss is
formulated as:

$\displaystyle\begin{split}&\mathcal{L}(X,Y)=\mathop{\mathbb{E}}_{(X,Y)\sim\mathcal{P}}[F_{c}(F^{\prime}_{a}(X)),Y)],\end{split}$
(10)

where $F_{c}$ denotes all layers of the speaker classifier module.

###  III-E Parameters Analysis

In Whisper-SV, the parameter size of each module during the inference and
training stages is shown in TABLE I. The Representation Selection Module is
used before training to evaluate which layers of Whisper’s embeddings have
more distinctive speaker-related information, and it does not participate in
training and inference. The multi-layer aggregation module and the speaker
classifier module undergo joint training. The pre-trained Whisper module is a
feature extractor and is not involved in the training process. To conserve
time and computational resources, we extract data representations using the
pre-trained Whisper model once and refrain from reutilizing it throughout the
iteration process.

TABLE I: Parameters of Whisper-SV in Inference and Training Stage

Module |  | Inference
---
Parameters (M)
| Trainable
---
Parameters (M)
(a) Pre-trained Whisper | 1550 | 0
(b) Representation Selection Module | 0 | 0
(c) Multi-layer Aggregation Module | 4.610 | 4.610
(d) Speaker Classifier Module | 0.725 | 0.725

##  IV Experimental Setup

###  IV-A Dataset

In this paper, we evaluate the performance of Whisper-SV on VoxCeleb1 [19],
FFSVC [7], and IMSV [8] datasets. VoxCeleb1 is collected from open-source
media. The VoxCeleb1 training set comprises 1211 speakers, with a total
duration of approximately 340 hours. FFSVC is a multi-channel far-field
speaker recognition dataset. In this paper, we utilize the speech recordings
from the second channel and iPhone in both the training and test sets of FFSVC
(referred to as ’FFSVC’ hereafter) for conducting experiments. Specifically,
the FFSVC dataset comprises 120 speakers and approximately 288 hours of audio.
IMSV is an indic-multilingual speaker recognition dataset. The IMSV training
set comprises 50 speakers, with a total duration of approximately 100 hours.
The test set we use includes VoxCeleb-O [19], FFSVC2020 Task1 development
trials (the second channel speech for testing) [7], and IMSV development
trials [8]. The data volume of VoxCeleb1, FFSVC, and IMSV is comparatively
limited, rendering them suitable as low-resource datasets for SV tasks.
Furthermore, to assess Whisper-SV’s performance in scenarios with severe data
limitations, we conduct experiments by randomly partitioning training datasets
from VoxCeleb1, FFSVC, and IMSV into fractions of the original data
volume—specifically, whole, half (1/2), quarter (1/4), and one_eighth (1/8)
while ensuring a consistent number of speakers in the training process.

In addition, we compare the results of Whisper-SV with data augmentation
methods. Data augmentation is performed on Musan [9] and room impulse response
[10] datasets. It’s worth noting that we do not apply any data augmentation
during the training process of Whisper-SV.

###  IV-B Configurations

Model Configurations: In this paper, we employ the Whisper-large-v2 [18] as
the pre-trained Whisper module. The evaluation model in the representation
selection module is ECAPA-TDNN with configurations of [1024, 1024, 1024, 3072]
[4]. Ablation experiments indicate that Whisper-SV performs optimally with a
top-k value of 4 while having a relatively modest number of parameters. The
configurations of components of the multi-layer aggregation module and speaker
classification module are illustrated in TABLE II.

TABLE II: The configurations of Whisper-SV.

Layer | Channels | Kernel | Dilation
---|---|---|---
Zoom Layer | 128 | 5 | 1
Conv1d in Multi-layer Aggregation Module | 512 | 3 | 3
Attention Aggregation Layer | 128 | - | -
Conv1d in Speaker Classifier Module | 256 | 1 | 1
Attention Statistic Pooling Layer | 128 | - | -

Training Configurations: In this paper, we employ two types of feature
extraction methods. The first method is applied to the input features of
Whisper, which computes 80-dimensional log magnitude Mel spectrogram features
using 25-millisecond windows and a 10-millisecond stride. The second method
utilizes 80-dimensional Filterbank (Fbank) features with a 25ms window size
and a 10ms frameshift, serving as the baseline for training the comparison
models. Before feature extraction, the input audio is resampled at 16,000 Hz.
The loss function employed is the additive angular margin softmax (AAM-
softmax) [50] with a margin parameter set to 0.2 and a scale parameter set to
30. Additionally, a weight decay of 2e-5 is applied in the training process.
The optimization employs the Adam optimizer with a cyclic learning rate, which
varies between 1e-8 and 1e-3, following the triangular policy [51].

###  IV-C Comparison Methods

We conduct a comparative analysis of Whisper-SV and other competitive methods.
These methods include ECAPA-TDNN trained with Fbank (ECAPA-TDNN (Fbank)) and
representations (the selected top-1 representations by the representation
selection module) of Whisper (ECAPA-TDNN (Whisper)), data augmentation,
supervised pre-trained SV models, SSL pre-trained models, and some domain
adaptation methods in low-data-resource SV. The comparison methods are listed
in TABLE III.

TABLE III: Comparison Methods

Category | Methods
---|---
Baseline | ECAPA-TDNN (Fbank) [4]
ECAPA-TDNN (Whisper)
Data Augmentation |  | Data augmentation (add noise [52] and reverberation [52],
---
SpecAug [12], speedPerturb [11])
SSL Method | Entirely finetuning Hubert for SV (EF-Hubert-large-finetune) [45]
Entirely finetuning wav2vec for SV (EF-wav2vec-large-finetune) [45]
Domain Adaptation | Wasserterin domain adversarial [39] training
MMD tranfer learning [38]
WTR finetuning [14]
Model and Loss Improvements | Dual-model self-regularization and fusion (DualA-tFC) [53]
Angular margin prototypical loss (AMP Loss) [54]
ConvNext-Tiny [55]
Lighten CNN-LSMT [56]

###  IV-D Metric Score

In the test phase, we use cosine similarity as the scoring criterion. The
performance metrics are equal error rate (EER) and minimum detection cost
function (minDCF) [57] which is evaluated with $P_{target}=0.01$,
$C_{miss}=C_{fa}=1$. Furthermore, we incorporate two critical metrics: the
number of parameters (measured in millions, M) and the floating-point
operations per second (FLOPs, quantified in giga, G). In this paper, all FLOPs
calculations are performed with a batch size of 1.

##  V Experimental Results

###  V-A Analysis of Representations of Whisper

Considering that Whisper is purpose-built for multi-lingual ASR, ST, and LID,
to tailor Whisper for SV tasks effectively, it is crucial to quantify speaker-
specific cues inherent within representations of each layer in Whisper. To
thoroughly analyze the speaker-related discriminative information encapsulated
within representations of each distinct layer, we independently train the SV
model utilizing representations extracted from each layer of Whisper and
assess the performance of each layer’s representations in SV. The experimental
results are depicted in Fig. 2. We employ a composite evaluation metric
($(EER+10*minDCF)/2$) to identify the top-k layers with the lowest error rate
to determine the optimal representation extraction layers for training SV
models in Whisper. This approach provides a comprehensive assessment by
jointly considering the EER and minDCF metrics, as illustrated by the green
lines in Fig. 2.

Figure 2: Experimental results of ECAPA-TDNN trained with representations
extracted from each layer of Whisper. The red dots indicate the top four
lowest combined EER and minDCF ((EER +10*minDCF)/2).

Fig. 2 displays experimental results from three datasets (VoxCeleb1, FFSVC,
and IMSV), indicating that Whisper’s representations around the 20th layer
exhibit optimal speaker discriminative characteristics for SV tasks.
Furthermore, the trend observed in the three sub-graphs suggests diminishing
speaker-specific information of Whisper representations before layer 15.
Consequently, our experiments thoroughly analyze the performance of
representations from layer 32 to layer 11 of Whisper in SV tasks. In Fig. 2
(a), (b) and (c), the four red dots represent the top-4 layers with
significant speaker discriminative characteristics for the corresponding
datasets. The results of the selected top-4 layers are illustrated in TABLE
IV. As evident from TABLE IV, the layers containing the most crucial speaker-
discriminative information within the Whisper framework are predominantly
centered around the 20th layer, located in the middle rear of the Whisper
encoder. Based on these experimental findings, we are going to identify and
harness the top-k layers that demonstrate optimal performance impacts on the
SV model. These selected layers are integrated within a multi-layer
aggregation module specifically designed for boosting SV tasks.

TABLE IV: EER (%) and minDCF (p=0.01) of ECAPA-TDNN trained with
representations of top-4 layers of Whisper.

Training Dataset | Layer Index | EER | minDCF | (EER+minDCF*10)/2
---|---|---|---|---
VoxCeleb1 | 22 | 2.92 | 0.391 | 3.415
21 | 2.94 | 0.413 | 3.535
20 | 3.02 | 0.416 | 3.590
23 | 3.05 | 0.418 | 3.615
FFSVC | 21 | 8.59 | 0.835 | 8.470
18 | 8.79 | 0.847 | 8.630
20 | 8.89 | 0.845 | 8.670
17 | 8.96 | 0.853 | 8.745
IMSV (1/4) | 20 | 8.50 | 0.649 | 7.495
19 | 8.67 | 0.649 | 7.580
21 | 8.68 | 0.650 | 7.590
18 | 8.77 | 0.662 | 7.695

###  V-B Experimental Results on VoxCeleb1

The experimental results on VoxCeleb1 are presented in TABLE V. TABLE V
highlights that using raw representations extracted directly from Whisper (Raw
Whisper Representation) for SV is unviable, yielding an EER of 44.37% and a
minDCF of 0.999. This challenge primarily stems from Whisper not being
optimized explicitly for SV tasks. Therefore, it is essential to adapt Whisper
for SV through the development of Whisper-SV.

When compared to training ECAPA-TDNN directly using the optimal
representations of the top-1 layer in Whisper (ECAPA-TDNN (Whisper)), Whisper-
SV demonstrates a relative improvement of 23.9%/21.4% on EER/minDCF.
Meanwhile, the outcomes of ECAPA-TDNN (Whisper) surpass those achieved by
ECAPA-TDNN trained with Frank (ECAPA-TDNN (Fbank)). This observation
underscores that representations extracted from Whisper encompass richer
discriminative speaker-specific features SV tasks. In addition, when compared
with fine-tuning self-supervised models (wav2vec2.0 [42] and Hubert [41]) for
SV, Whisper-SV achieves lower EER and minDCF with a smaller number of
trainable parameters and computational efficiency. Furthermore, Whisper-SV
outperforms the Siamese capsule back-end processing method and the binary
networks in SV tasks.

TABLE V: Experimental results on VoxCeleb (EER(%), minDCF (p=0.01)).

Method |  Training Dataset |  Trainable Parameters (M) | Flops (G) | VoxCeleb-O
---|---|---|---|---
EER | minDCF
ECAPA-TDNN (Fbank) [4] | VoxCeleb1 | 20.768 | 9.6 | 4.37 | 0.379
w/ Data augmentation | VoxCeleb1 | 20.768 | 9.6 | 3.02 | 0.326
Raw Whisper Representation | VoxCeleb1 | 0 | 9.6 | 44.37 | 0.999
ECAPA-TDNN (Whisper) | VoxCeleb1 | 20.768 | 9.6 | 2.92 | 0.391
EF-Hubert-large-finetune [45] | VoxCeleb1 | 300 | 9.6 | 2.36 | -
EF-wav2vec-large-finetune [45] | VoxCeleb1 | 317 | - | 3.42 | -
Siamese Capsule [58] | VoxCeleb1 | 8.3 | - | 3.14 | -
Linear Weighted Sum - 32 Layers | VoxCeleb1 | 1.215 | 0.296 | 3.26 | 0.369
Random Four Layer - Layer [22,21,1,8] | VoxCeleb1 | 5.336 | 0.643 | 2.97 | 0.417
Average Pooling Top-4 Layers | VoxCeleb1 | 2.597 | 0.581 | 2.91 | 0.330
Whisper-SV | VoxCeleb1 | 5.336 | 0.643 | 2.22 | 0.307
w/o Zoom Layer (No Zoom Layer) | VoxCeleb1 | 21.647 | 5.458 | 2.77 | 0.428

Additionally, we explore alternative methods for using Whisper
representations, including 1) the commonly used linear weighted sum of
representations from all 32 layers (Linear Weighted Sum - 32 Layers), 2)
training Whisper-SV with representations from randomly selected four layers
(Random Four Layers - Layer [22, 21, 1, 8]), 3) excluding the use of the zoom
layer (i.e., not performing dimensionality reduction on Whisper
representations) (w/o Zoom Layer), and 4) using average pooling of
representations from the top-4 layers instead of the multi-layer aggregation
module (Average Pooling - Top-4 Layers). As the experimental results shown in
TABLE V indicate, the performance of these alternative approaches has not
surpassed that of Whisper-SV. The analysis of these alternative methods is as
follows. The Linear Weighted Sum of 32 layers may introduce non-speaker-
related information, as Whisper was initially trained for ASR tasks, not SV
tasks. Not all layer representations are equally suitable for SV tasks.
Randomly selecting four layers may degrade SV performance if shallow
representations are chosen, as illustrated in Fig. 2, where shallow
representations perform poorly for SV tasks. Removing the zoom layer
significantly increases the number of parameters, as Whisper representations
have a dimensionality of 1280. It may also lead to overfitting, as the zoom
layer is primarily used for dimensionality reduction and extracting speaker-
related information from Whisper representations. The use of average pooling
of representations from the top-k layers, instead of the original multi-layer
aggregation, does not purify and fuse representations from different layers,
introducing non-speaker-related information. Average pooling weakens the
discriminative power of speaker-related solid representations.

###  V-C Experimental Results on FFSVC

The experimental results for FFSVC are presented in TABLE VI. Notably, the
experimental results of Whisper-SV outperform those of ECAPA-TDNN (Fbank) and
ECAPA-TDNN (Whisper). Furthermore, we discover that utilizing only 1/4 of the
FFSVC training dataset yields superior results compared to using the entire
dataset. This can be attributed to Whisper’s robust denoising and
discriminative feature extraction capabilities, and make Whisper-SV
necessitate fewer speech samples from a single speaker. Excessive data of per
speaker may result in overfitting. Therefore, Whisper-SV is well-suited for
data low-resource SV scenarios.

Additionally, we conduct comparison experiments between Whisper-SV and domain
adaptation methods (vanilla fine-tuning, WTR finetuning, and Wasserterin DA),
which are trained with VoxCeleb2 [3] and FFSVC dataset. The results indicate
that Whisper-SV trained with only 1/4 of the FFSVC data outperforms vanilla
fine-tuning and Wasserterin DA methods. Utilizing a vanilla fine-tuned
Whisper-SV model pre-trained on VoxCeleb1, we observe a notable 5.9% decrease
in minDCF relative to the minDCF achieved by the WTR method, despite the WTR
method exhibiting a lower EER. Meanwhile, Whisper-SV maintains a relatively
low number of trainable parameters and FLOPs. These findings demonstrate that
Whisper-SV can perform better with lower data requirements and fewer trainable
parameters.

TABLE VI: Experimental results on FFSVC (EER(%), minDCF (p=0.01)).

Method | Training Dataset |  Trainable Parameters (M) | FLOPs (G) | FFSVC
---|---|---|---|---
EER | minDCF
ECAPA-TDNN (Fbank) [4] | FFSVC | 20.768 | 4.814 | 9.49 | 0.810
\+ Data augmentation | FFSVC | 20.768 | 4.814 | 8.55 | 0.774
ECAPA-TDNN (Whisper) | FFSVC | 20.768 | 4.814 | 8.59 | 0.835
ECAPA-TDNN (Whisper) | FFSVC (1/4) | 20.768 | 4.814 | 8.56 | 0.828
Vanilla Finetuning | VoxCeleb2 + FFSVC | 20.768 | 4.814 | 8.44 | 0.548
WTR [14] | VoxCeleb2 + FFSVC | 20.768 | 4.814 | 5.53 | 0.519
Wasserterin DA [59] | VoxCeleb2 + FFSVC | 20.769 | 4.815 | 5.78 | 0.5697
Whisper-SV | FFSVC (1/4) | 5.336 | 0.643 | 7.90 | 0.631
Whisper-SV Finetuning | VoxCeleb1 + FFSVC (1/4) | 5.336 | 0.643 | 6.14 | 0.488

###  V-D Experimental Results on IMSV

The experimental results for IMSV are displayed in TABLE VII. As demonstrated
in TABLE VI, the IMSV results also reveal that using only 1/4 of the training
data leads to lower EER results than utilizing the entire dataset. Compared
with ECAPA-TDNN (Whisper), Whisper-SV shows a relative reduction of 11.7%.

TABLE VII: Experimental results on IMSV (EER(%), minDCF (p=0.01)).

Method | Training Dataset |  Trainable Parameters (M) | FLOPs (G) | IMSV
---|---|---|---|---
EER | minDCF
ECAPA-TDNN (Fbank) [4] | IMSV (1/4) | 20.768 | 4.814 | 11.51 | 0.845
\+ Data augmentation | IMSV (1/4) | 20.768 | 4.814 | 9.23 | 0.845
ECAPA-TDNN (Whisper) | IMSV | 20.768 | 4.814 | 8.90 | 0.604
ECAPA-TDNN (Whisper) | IMSV (1/4) | 20.768 | 4.814 | 8.50 | 0.649
ConvNext-Tiny [55] | IMSV | 27.796 | 2.903 | 19.01 | 0.987
AMP Loss [54] | IMSV | 20.768 | 4.814 | 7.97 | 0.827
CNN-LSTM [56] | IMSV | 0.587 | 0.025 | 43.86 | 0.987
Whisper-SV | IMSV (1/4) | 5.336 | 0.643 | 7.50 | 0.582

###  V-E Extension Experiments on VoxCeleb2

To further validate the effectiveness of Whisper-SV on general SV tasks, we
conduct experiments on the VoxCeleb2 dataset as shown in TABLE VIII. When
VoxCeleb2 is used as the training set, Whisper-SV achieves EER/minDCF of
1.71%/0.211 on the VoxCeleb-O. This performance, compared to the first-place
result in VoxSRC2019 using ResNet34 with 256 channels [60, 61], exhibits a
slight increase in EER by 0.29%. However, it surpasses the results obtained
with the X-vector model [62, 52] of equivalent parameter sizes.

TABLE VIII: Experimental results on VoxCeleb2 (EER(%), minDCF (p=0.01)).

Method |  Training Dataset |  Trainable Parameters (M) | Flops (G) | VoxCeleb-O
---|---|---|---|---
EER | minDCF
ResNet34 (256) [60] | VoxCeleb2 (Data Aug) | 7.031 | 2.929 | 1.42 | -
X-vector [62] | VoxCeleb2 (Data Aug) | 4.357 | 0.724 | 2.41 | 0.260
ECAPA-TDNN [4] | VoxCeleb2 (Data Aug) | 20.768 | 0.814 | 0.87 | 0.107
RawNet3 [63] | VoxCeleb2 & VoxCeleb1 (Data Aug) | - | - | 0.89 | 0.0659
MFA-Conformer [64] | VoxCeleb2 & VoxCeleb1 & SITW(Data Aug) | 20.5 | - | 0.64 | 0.081
Whisper-SV | VoxCeleb2 | 5.336 | 0.643 | 1.71 | 0.211

###  V-F Ablation and Visualization

Data Reduction Ablation To further investigate whether Whisper-SV performs
better than Fbank in low-resource scenarios, we explore the correlation
between the reduction in data and the performance of SV. The experimental
results are illustrated in Fig. 3. This analysis is carried out after
partitioning the training data into various fractions, including the whole
dataset, half, quarter, and one-eighth. The results presented in Fig. 3 show
that the VoxCeleb1 dataset exhibits superior performance when utilizing the
whole dataset. However, as the amount of data decreases, the performance
degradation of ECAPA-TDNN trained with Fbank features is significantly more
pronounced than Whisper-SV’s results.

Meanwhile, experimental results from the FFSVC and IMSV datasets indicate that
Whisper-SV achieves the lowest EER and minDCF when only a quarter of the data
volume is used. In contrast, ECAPA-TDNN trained with Fbank achieves optimal
results when the whole dataset is employed. These results suggest that Whisper
representations contain a higher density of speaker-specific features while
effectively filtering out extraneous noise elements. On the other hand, an
excessive volume of speech data per speaker may lead to overfitting, making
Whisper representations more advantageous for SV tasks in low-resource data
scenarios.

The training and validation losses are depicted in Fig. 4. Fig. 4 indicates
that during the FFSVC training process, training with whole and half of the
data shows signs of overfitting starting at epochs 7 and 8, as evidenced by
increasing validation loss. In contrast, training with 1/8 and 1/16 of the
data exhibits underfitting, whereas training with 1/4 of the data mitigates
overfitting without leading to underfitting.

Figure 3: Experimental results of ECAPA-TDNN and Whisper-SV trained with
different proportions of training data (The training duration (hours)
corresponding to the value in parentheses on the x-axis).

Figure 4: The training and validation loss value in the training process on
FFSVC.

Zoom Layer Channel Ablation Subsequently, we examine the influence of varying
the scale of whisper representations within the zoom layer on the efficacy of
Whisper-SV. The experimental results presented in TABLE IX reveal that
Whisper-SV attains optimal performance across three distinct datasets when the
whisper representation is scaled to a dimensionality 128. Meanwhile, Whisper-
SV with 128 channels in the zoom layer demonstrates fewer learnable parameters
and better performance. This indicates that Whisper-SV is distinguished by its
low computational complexity and minimal trainable parameters.

TABLE IX: Experimental results on different channels in the zoom layer
(EER(%), minDCF (p=0.01)).

Zoom layer Channel |  Trainable Parameters (M) | Voxceleb1 | IMSV | FFSVC
---|---|---|---|---
EER | minDCF | EER | minDCF | EER | minDCF
512 | 19.22 | 2.67 | 0.347 | 7.77 | 0.572 | 8.37 | 0.768
256 | 9.845 | 2.54 | 0.365 | 7.33 | 0.585 | 8.10 | 0.789
128 | 5.335 | 2.22 | 0.307 | 7.50 | 0.582 | 7.90 | 0.631
64 | 3.586 | 3.63 | 0.398 | 8.55 | 0.612 | 9.43 | 0.803

Channel Ablation in Multi-layer Aggregation Module  The multi-layer
aggregation module, consisting of 1D convolutional blocks, aggregates multi-
layer representations extracted from Whisper. The experimental results of
different channels in the multi-layer aggregation module are illustrated in
TABLE X. The experimental results in TABLE X indicate that Whisper-SV exhibits
optimal performance when the channel count of the multi-layer aggregation
module is configured to 512.

TABLE X: Comparison of experimental results on different channels in the
multi-layer aggregation module (EER(%), minDCF (p=0.01)).

Channels in Multi-layer Aggregation Module |  Trainable Parameters (M) | Voxceleb1 | IMSV | FFSVC
---|---|---|---|---
EER | minDCF | EER | minDCF | EER | minDCF
512 | 5.335 | 2.22 | 0.307 | 7.50 | 0.582 | 7.90 | 0.631
256 | 4.251 | 2.22 | 0.317 | 7.18 | 0.601 | 8.10 | 0.789
128 | 3.774 | 2.51 | 0.351 | 7.974 | 0.67 | 8.79 | 0.814

Figure 5: Embedding visualization of different SV models.

Different Top-k Ablation  Fig. 6 displays the results of the ablation study
conducted with different top-k values. Specifically, k is from the set of {1,
2, 3, 4, 5, 6, 7, 8}. The experimental results indicate that Whisper-SV
achieves its best performance when k=4, signifying that selecting the most
discriminative representations from the top four layers yields the best
results. Interestingly, aggregating an excessive number of layers in Whisper
representations does not reduce EER and minDCF. This lack of improvement may
be attributed to the inclusion of non-speaker-related features introduced by
the extensive layers of Whisper representations. Furthermore, the excessive
aggregation of representations may lead to model overfitting.

Figure 6: Experimental results on different top-k on VoxCeleb1 / FFSVC / IMSV.

Embedding Visualization To further elucidate the discrimination of speaker
embeddings extracted from different SV models, we employ T-SNE [65] to
illustrate the raw whisper representations, embeddings generated by the ECAPA-
TDNN (Fbank), embeddings generated by ECAPA-TDNN (Whisper), and embeddings
generated by Whisper-SV models. The distribution of the above embeddings is
depicted in Fig. 5. In Fig. 5, we can see that irrespective of the testing
dataset, utilizing raw whisper representations for speaker identification is
infeasible. The entanglement of speaker embeddings demonstrates we need an
adaptor to transfer Whisper to SV tasks. Observing the embedding distribution
on VoxCeleb1, it becomes apparent that ECAPA-TDNN (Whisper) exhibits enhanced
intra-class cohesion compared to ECAPA-TDNN (Fbank). Furthermore, Whisper-SV
demonstrates improved cohesion in its embedding distribution and greater
inter-class separation. These observations hold when analyzing the results on
FFSVC, further substantiating that speaker embeddings of Whisper-SV are more
discriminative and robust. The visualization results for Whisper-SV on IMSV
show an improvement in classification performance on hard samples when
contrasted with ECAPA-TDNN (Fbank) and ECAPA-TDNN (Whisper).

##  VI Conclusion

This paper proposes an adaptor framework named Whisper-SV, designed to adapt
Whisper for SV tasks, particularly benefiting low-data-resource SV scenarios.
Whisper-SV consists of four distinct modules: a pre-trained Whisper module, a
representation selection module, a multi-layer aggregation module, and a
speaker classifier module. Specifically, the pre-trained Whisper module is the
robust and generalized representation extractor. As Whisper is not explicitly
tailored for SV tasks, the representation selection module is introduced to
assess speaker identity characteristics within each layer of Whisper. It
selects the top-k layers with prominent speaker-related features.
Subsequently, the multi-layer aggregation module is designed to fuse
representations from the selected top-k layers to a compacted representation
abundant in speaker-specific discriminative characteristics. Finally, the
classifier module is used for speaker classification. Attributable to
Whisper’s pre-training on massive and diverse datasets, Whisper-SV achieves
remarkable performance with a modest number of trainable parameters and a
limited amount of training speech. This demonstrates the suitability of
Whisper-SV for low-data-resource SV tasks. In addition, experimental results
and analysis illustrate the superior performance of Whisper-SV compared to
other competitive methods in low-data-resource SV tasks.

##  VII Future Work

Whisper-SV achieves superior performance in low-data-resource SV tasks.
However, since Whisper is a parameter-heavy, large speech foundation model, it
is not well-suited for SV applications requiring low computational resources
and fast inference speeds. Therefore, in future work, we will focus on
developing SV models with Whisper using transfer learning methods to reduce
reliance on Whisper during inference.

## References

  * [1] Ravika Naika,  “An overview of automatic speaker verification system,”  Proc. ICICC, pp. 603–610, 2018.
  * [2] Lantian Li, Ruiqi Liu, Jiawen Kang, Yue Fan, Hao Cui, Yunqi Cai, Ravichander Vipperla, Thomas Fang Zheng, and Dong Wang,  “Cn-celeb: multi-genre speaker recognition,”  Speech Communication, vol. 137, pp. 77–91, 2022.
  * [3] Joon Son Chung, Arsha Nagrani, and Andrew Zisserman,  “Voxceleb2: Deep speaker recognition,”  arXiv preprint arXiv:1806.05622, 2018.
  * [4] Brecht Desplanques, Jenthe Thienpondt, and Kris Demuynck,  “ECAPA-TDNN: Emphasized channel attention, propagation and aggregation in tdnn based speaker verification,”  in Conference of the International Speech Communication Association (INTERSPEECH), 2020.
  * [5] Tianyan Zhou, Yong Zhao, and Jian Wu,  “Resnext and Res2net structures for speaker verification,”  in 2021 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2021, pp. 301–307.
  * [6] Fei Xie, Dalong Zhang, and Chengming Liu,  “Global–local self-attention based transformer for speaker verification,”  Applied Sciences, vol. 12, no. 19, pp. 10154, 2022.
  * [7] Xiaoyi Qin, Ming Li, Hui Bu, Rohan Kumar Das, Wei Rao, Shrikanth Narayanan, and Haizhou Li,  “The FFSVC 2020 evaluation plan,”  in Conference of the International Speech Communication Association (INTERSPEECH), 2020.
  * [8] Jagabandhu Mishra, Mrinmoy Bhattacharjee, and SR Mahadeva Prasanna,  “I-MSV 2022: Indic-multilingual and multi-sensor speaker verification challenge,”  in International Conference on Speech and Computer. Springer, 2023, pp. 437–445.
  * [9] David Snyder, Guoguo Chen, and Daniel Povey,  “Musan: A music, speech, and noise corpus,”  arXiv preprint arXiv:1510.08484, 2015.
  * [10] Emanuel AP Habets,  “Room impulse response generator,”  Technische Universiteit Eindhoven, Tech. Rep, vol. 2, no. 2.4, pp. 1, 2006.
  * [11] Shiqing Yang and Min Liu,  “Data augmentation for speaker verification,”  in Proceedings of the 2022 6th International Conference on Electronic Information Technology and Computer Engineering, 2022, pp. 1247–1251.
  * [12] Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le,  “Specaugment: A simple data augmentation method for automatic speech recognition,”  in Conference of the International Speech Communication Association (INTERSPEECH), 2019.
  * [13] Aleksei Gusev, Vladimir Volokhov, Alisa Vinogradova, Tseren Andzhukaev, Andrey Shulipa, Sergey Novoselov, Timur Pekhovsky, and Alexander Kozlov,  “STC-Innovation speaker recognition systems for far-field speaker verification challenge 2020.,”  in Conference of the International Speech Communication Association (INTERSPEECH), 2020, pp. 3466–3470.
  * [14] Li Zhang, Qing Wang, Hongji Wang, Yue Li, Wei Rao, Yannan Wang, and Lei Xie,  “Distance-based weight transfer for fine-tuning from near-field to far-field speaker verification,”  in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5.
  * [15] Li-Wei Chen and Alexander Rudnicky,  “Exploring wav2vec 2.0 fine tuning for improved speech emotion recognition,”  in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5.
  * [16] Shivang Gupta, Kowshik Siva Sai Motepalli, Ravi Kumar, Vamsi Narasinga, Sai Ganesh Mirishkar, and Anil Kumar Vuppala,  “Enhancing language identification in indian context through exploiting learned features with wav2vec2. 0,”  in International Conference on Speech and Computer. Springer, 2023, pp. 503–512.
  * [17] Jing Zhao and Wei-Qiang Zhang,  “Improving automatic speech recognition performance for low-resource languages with self-supervised models,”  IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1227–1241, 2022.
  * [18] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever,  “Robust speech recognition via large-scale weak supervision,”  in International Conference on Machine Learning. PMLR, 2023, pp. 28492–28518.
  * [19] Arsha Nagrani, Joon Son Chung, Weidi Xie, and Andrew Zisserman,  “Voxceleb: Large-scale speaker verification in the wild,”  Computer Speech & Language, vol. 60, pp. 101027, 2020.
  * [20] Dominik Macháček, Raj Dabre, and Ondřej Bojar,  “Turning Whisper into real-time transcription system,”  in Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics: System Demonstrations, 2023.
  * [21] Hang Shao, Wei Wang, Bei Liu, Xun Gong, Haoyu Wang, and Yanmin Qian,  “Whisper-KDQ: A lightweight Whisper via guided knowledge distillation and quantization for efficient asr,”  arXiv preprint arXiv:2305.10788, 2023.
  * [22] Rishabh Jain, Andrei Barcovschi, Mariam Yiwere, Peter Corcoran, and Horia Cucu,  “Adaptation of Whisper models to child speech recognition,”  arXiv preprint arXiv:2307.13008, 2023.
  * [23] Juan Camilo Vásquez-Correa and Aitor Álvarez Muniain,  “Novel speech recognition systems applied to forensics within child exploitation: Wav2vec2. 0 vs. Whisper,”  Sensors, vol. 23, no. 4, pp. 1843, 2023.
  * [24] Manila Kodali, Sudarsana Reddy Kadiri, and Paavo Alku,  “Classification of vocal intensity category from speech using the wav2vec2 and Whisper embeddings,”  in Conference of the International Speech Communication Association (INTERSPEECH), 2023.
  * [25] Siyin Wang, Chao-Han Huck Yang, Ji Wu, and Chao Zhang,  “Can Whisper perform speech-based in-context learning,”  arXiv e-prints, p. arXiv:2309.07081, Sept. 2023.
  * [26] Ryandhimas E Zezario, Yu-Wen Chen, Szu-Wei Fu, Yu Tsao, Hsin-Min Wang, and Chiou-Shann Fuh,  “A study on incorporating Whisper for robust speech assessment,”  arXiv preprint arXiv:2309.12766, 2023.
  * [27] Siddharth Rathod, Monil Charola, and Hemant A Patil,  “Noise robust Whisper features for dysarthric severity-level classification,”  in International Conference on Pattern Recognition and Machine Intelligence. Springer, 2023, pp. 708–715.
  * [28] Huma Ameer, Seemab Latif, Rabia Latif, and Sana Mukhtar,  “Whisper in focus: Enhancing stuttered speech classification with encoder layer optimization,”  arXiv preprint arXiv:2311.05203, 2023.
  * [29] Nianlong Gu, Kanghwi Lee, Maris Basha, Sumit Kumar Ram, Guanghao You, and Richard Hahnloser,  “Positive transfer of the Whisper speech transformer to human and animal voice activity detection,”  bioRxiv, pp. 2023–09, 2023.
  * [30] Tijn Berns, Nik Vaessen, and David A van Leeuwen,  “Speaker and language change detection using wav2vec2 and Whisper,”  arXiv preprint arXiv:2302.09381, 2023.
  * [31] Jee-weon Jung, Hee-Soo Heo, Bong-Jin Lee, Jaesong Lee, Hye-jin Shim, Youngki Kwon, Joon Son Chung, and Shinji Watanabe,  “Large-scale learning of generalised representations for speaker recognition,”  arXiv preprint arXiv:2210.10985, 2022.
  * [32] Yu Zheng, Jinghan Peng, Yihao Chen, Yajun Zhang, Min Liu, and Minqiang Xu,  “The SpeakIn system for far-field speaker verification challenge 2022,”  in Conference of the International Speech Communication Association (INTERSPEECH), 2022.
  * [33] Jiayu Du, Xingyu Na, Xuechen Liu, and Hui Bu,  “Aishell-2: Transforming mandarin asr research into industrial scale,”  arXiv preprint arXiv:1808.10583, 2018.
  * [34] Yue Fan, JW Kang, LT Li, KC Li, HL Chen, ST Cheng, PY Zhang, ZY Zhou, YQ Cai, and Dong Wang,  “CN-Celeb: a challenging chinese speaker recognition dataset,”  in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 7604–7608.
  * [35] Li Zhang, Yue Li, Namin Wang, Jie Liu, and Lei Xie,  “Npu-hc speaker verification system for far-field speaker verification challenge 2022,”  in Conference of the International Speech Communication Association (INTERSPEECH), 2022.
  * [36] Li Zhang, Jian Wu, and Lei Xie,  “NPU speaker verification system for interspeech 2020 far-field speaker verification challenge,”  in Conference of the International Speech Communication Association (INTERSPEECH), 2020.
  * [37] Li Zhang, Qing Wang, Kong Aik Lee, Lei Xie, and Haizhou Li,  “Multi-level transfer learning from near-field to far-field speaker verification,”  in Conference of the International Speech Communication Association (INTERSPEECH), 2021.
  * [38] Weiwei Lin, Man-Mai Mak, Na Li, Dan Su, and Dong Yu,  “Multi-level deep neural network adaptation for speaker verification using mmd and consistency regularization,”  in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 6839–6843.
  * [39] Johan Rohdin, Themos Stafylakis, Anna Silnova, Hossein Zeinali, Lukáš Burget, and Oldřich Plchot,  “Speaker verification using end-to-end adversarial language adaptation,”  in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, pp. 6006–6010.
  * [40] Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al.,  “Wavlm: Large-scale self-supervised pre-training for full stack speech processing,”  IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1505–1518, 2022.
  * [41] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed,  “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 3451–3460, 2021.
  * [42] Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli,  “wav2vec: Unsupervised pre-training for speech recognition,”  arXiv preprint arXiv:1904.05862, 2019.
  * [43] Rostislav Makarov, Nikita Torgashov, Alexander Alenin, Ivan Yakovlev, and Anton Okhotnikov,  “ID R&D system description to VoxCeleb speaker recognition challenge 2022,”  ID R&D Inc.: New York, NY, USA, 2022.
  * [44] Jaesung Huh, Andrew Brown, Jee-weon Jung, Joon Son Chung, Arsha Nagrani, Daniel Garcia-Romero, and Andrew Zisserman,  “VoxSRC 2022: The fourth VoxCeleb speaker recognition challenge,”  in Conference of the International Speech Communication Association (INTERSPEECH), 2023.
  * [45] Yingzhi Wang, Abdelmoumene Boumadane, and Abdelwahab Heba,  “A fine-tuned wav2vec 2.0/hubert benchmark for speech emotion recognition, speaker verification and spoken language understanding,”  arXiv preprint arXiv:2111.02735, 2021.
  * [46] Sergey Novoselov, Galina Lavrentyeva, Anastasia Avdeeva, Vladimir Volokhov, and Aleksei Gusev,  “Robust speaker recognition with transformers using wav2vec 2.0,”  arXiv preprint arXiv:2203.15095, 2022.
  * [47] Zhiming Wang, Furong Xu, Kaisheng Yao, Yuan Cheng, Tao Xiong, and Huijia Zhu,  “AntVoice neural speaker embedding system for ffsvc 2020.,”  in Conference of the International Speech Communication Association (INTERSPEECH), 2021, pp. 1069–1073.
  * [48] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin,  “Attention is all you need,”  Advances in neural information processing systems, vol. 30, 2017.
  * [49] Jie Hu, Li Shen, and Gang Sun,  “Squeeze-and-excitation networks,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 7132–7141.
  * [50] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou,  “Arcface: Additive angular margin loss for deep face recognition,”  in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 4690–4699.
  * [51] Leslie N Smith,  “Cyclical learning rates for training neural networks,”  in WACV. IEEE, 2017, pp. 464–472.
  * [52] David Snyder, Daniel Garcia-Romero, Gregory Sell, Daniel Povey, and Sanjeev Khudanpur,  “X-vectors: Robust dnn embeddings for speaker recognition,”  in 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2018, pp. 5329–5333.
  * [53] Yibo Duan, Yanhua Long, and Jiaen Liang,  “Dual-model self-regularization and fusion for domain adaptation of robust speaker verification,”  Speech Communication, vol. 155, pp. 103001, 2023.
  * [54] Dat Vi Thanh, Thanh Pham Viet, and Trang Nguyen Thi Thu,  “Deep speaker verification model for low-resource languages and vietnamese dataset,”  in Proceedings of the 35th Pacific Asia Conference on Language, Information and Computation, 2021, pp. 442–451.
  * [55] Maksym Kizitskyi, Olena Turuta, and Oleksii Turuta,  “Improving speaker verification model for low-resources languages,”  in 7th International Conference on Computational Linguistics and Intelligent Systems, 2023.
  * [56] Zitian Zhao, Hancong Duan, Geyong Min, Yue Wu, Zilei Huang, Xian Zhuang, Hao Xi, and Meirong Fu,  “A lighten CNN-LSTM model for speaker verification on embedded devices,”  Future Generation Computer Systems, vol. 100, pp. 751–758, 2019.
  * [57] Alvin Martin and Mark Przybocki,  “The nist 1999 speaker recognition evaluation—an overview,”  Digital signal processing, vol. 10, no. 1-3, pp. 1–18, 2000.
  * [58] Amirhossein Hajavi and Ali Etemad,  “Siamese capsule network for end-to-end speaker recognition in the wild,”  in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 7203–7207.
  * [59] Kai Liu and Huan Zhou,  “Text-independent speaker verification with adversarial learning on short utterances,”  in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 6569–6573.
  * [60] Joon Son Chung, Arsha Nagrani, Ernesto Coto, Weidi Xie, Mitchell McLaren, Douglas A Reynolds, and Andrew Zisserman,  “Voxsrc 2019: The first voxceleb speaker recognition challenge,”  arXiv preprint arXiv:1912.02522, 2019.
  * [61] Hossein Zeinali, Shuai Wang, Anna Silnova, Pavel Matějka, and Oldřich Plchot,  “But system description to voxceleb speaker recognition challenge 2019,”  arXiv preprint arXiv:1910.12592, 2019.
  * [62] Shuai Wang, Yexin Yang, Yanmin Qian, and Kai Yu,  “Revisiting the statistics pooling layer in deep speaker embedding learning,”  in 2021 12th International Symposium on Chinese Spoken Language Processing (ISCSLP). IEEE, 2021, pp. 1–5.
  * [63] Jee-weon Jung, You Jin Kim, Hee-Soo Heo, Bong-Jin Lee, Youngki Kwon, and Joon Son Chung,  “Pushing the limits of raw waveform speaker recognition,”  arXiv preprint arXiv:2203.08488, 2022.
  * [64] Yang Zhang, Zhiqiang Lv, Haibin Wu, Shanshan Zhang, Pengfei Hu, Zhiyong Wu, Hung-yi Lee, and Helen Meng,  “Mfa-conformer: Multi-scale feature aggregation conformer for automatic speaker verification,”  arXiv preprint arXiv:2203.15249, 2022.
  * [65] Laurens Van der Maaten and Geoffrey Hinton,  “Visualizing data using t-SNE.,”  Journal of machine learning research, vol. 9, no. 11, 2008.