# Joint Speaker Encoder and Neural Back-end Model for Fully End-to-End
Automatic Speaker Verification with Multiple Enrollment Utterances

Chang Zeng , , Xiaoxiao Miao , , Xin Wang , ,
Erica Cooper , and Junichi Yamagishi Manuscript received XXX. This study was
partially supported by JST CREST Grants (JPMJCR18A6 and JPMJCR20D3), MEXT
KAKENHI Grants (21K17775, 21H04906, 21K11951, 18H04112), and Google AI for
Japan program. (Corresponding author: Chang Zeng.)Chang Zeng, Xiaoxiao Miao,
Xin Wang, Erica Cooper and Junichi Yamagishi are with the National Institute
of Informatics, 2-1-2 Hitotsubashi Chiyoda-ku, Tokyo 101-8340, Japan (e-mail:
{zengchang, xiaoxiaomiao, wangxin, ecooper, jyamagis}@nii.ac.jp).
0000-0002-4882-1823 0000-0002-6645-6524 0000-0001-8246-0606
0000-0002-2978-2793 0000-0003-2752-3955

###### Abstract

Conventional automatic speaker verification systems can usually be decomposed
into a front-end model such as time delay neural network (TDNN) for extracting
speaker embeddings and a back-end model such as statistics-based probabilistic
linear discriminant analysis (PLDA) or neural network-based neural PLDA
(NPLDA) for similarity scoring. However, the sequential optimization of the
front-end and back-end models may lead to a local minimum, which theoretically
prevents the whole system from achieving the best optimization. Although some
methods have been proposed for jointly optimizing the two models, such as the
generalized end-to-end (GE2E) model and NPLDA E2E model, all of these methods
are designed for use with a single enrollment utterance. In this paper, we
propose a new E2E joint method for speaker verification especially designed
for the practical case of multiple enrollment utterances. In order to leverage
the intra-relationship among multiple enrollment utterances, our model comes
equipped with frame-level and utterance-level attention mechanisms. We also
utilize several data augmentation techniques, including conventional noise
augmentation using MUSAN and RIRs datasets and a unique speaker embedding-
level mixup strategy for better optimization.

###### Index Terms:

neural network, automatic speaker verification, deep learning, attention, data
augmentation

##  I Introduction

Automatic speaker verification (ASV), which verifies and compares the speaker
identity of an input utterance with enrollment data, is an extensively used
biometric method. Depending on whether there are restrictions on the content
of the speech, ASV systems can be divided into text-dependent ones, where the
content of the spoken utterance is specified, or text-independent ones in
which there are no restrictions on the linguistic content. With the
development of deep learning, deep neural networks (DNN) have been
successfully applied to both text-independent and text-dependent ASV tasks.
The mainstream ASV models have evolved from statistical generative models such
as Gaussian mixture models (GMM), universal background models (UBM)[1] and
i-vectors [2] to DNN-based discriminative models such as d-vector [3, 4],
x-vector [5, 6, 7], and r-vector [8]. One of the earliest DNN-based approaches
is the d-vector [3], which takes frame-level acoustic features and their
context as the input to predict the frame-level speaker-related
representation, requiring post-processing to obtain the prediction of one
utterance. More recently, DNN-based ASV systems utilize more efficient methods
that directly extract utterance or segment-level speaker representative
vectors (also called speaker embeddings) from the hidden layer of a DNN.

State-of-the-art DNN-based ASV systems can be broadly categorized into two
groups: stage-wise approaches and fully end-to-end (E2E) approaches. The
stage-wise approaches consist of a neural speaker encoder for extracting
speaker embeddings as front-end processing and a back-end model for measuring
the similarity of speaker embeddings, in which the processes in the two stages
are trained separately. The front-end speaker encoder mainly consists of a
frame-level feature extractor to capture discriminative speaker information, a
pooling layer to accumulate the frame-level features into an utterance-level
representation, and a loss function for the optimization.

Two typical stage-wise methods are time delay neural network (TDNN)-based
x-vector and residual neural network (ResNet)-based r-vector. The x-vector
utilizes TDNN as a frame-level feature extractor, followed by a statistics
pooling layer to aggregate frame-level features into an utterance-level
vector, which is then passed on to the following fully connected classifier.
Instead of using TDNN, a 1-dimensional convolutional neural network, ResNet-
based r-vector leverages 2-dimensional convolutional neural networks with
residual connections, which is beneficial for building deeper neural networks
as the frame-level feature extractor. Analogous to x-vector, r-vector also
uses a temporal pooling layer for aggregating frame-level information into a
single utterance-level speaker vector, and the following classifier uses the
aggregated vector as input to categorize speakers. In addition to the
frameworks of speaker encoders mentioned earlier, various approaches primarily
related to the pooling methods and loss functions have also been proposed to
improve the performance further. For example, a self-attention mechanism [7,
9] was introduced into the pooling layer to calculate attentive statistics for
more discriminative speaker representations. As for the loss functions of the
speaker encoder, both metric learning-based loss functions such as triplet
loss [10, 11] and margin-based softmax loss functions such as angular softmax
loss [12, 13] and additive margin softmax loss [14, 15] have shown superior
performance in ASV tasks. It should be noted that the loss functions used in
the speaker encoder are not verification-based ones but classification-based
ones.

A back-end model, which measures the similarity of enrollment utterances and a
testing utterance, is another crucial part of a modern ASV system. Cosine
similarity is a simple but widely used back-end model for the front-end neural
speaker encoder with a margin-based softmax loss function. Compared with
cosine similarity, PLDA [16] is a more robust back-end model that can further
disentangle speaker and channel factors in a latent space. Neural networks
have also been used as the back-end model [17], and it was reported through a
challenge competition [18] that this model outperforms the conventional PLDA
model [18]. Moreover, there have been attempts to 1) combine a neural speaker
encoder with a neural back-end, 2) design a fully E2E architecture that takes
a pair of utterances as the input and generates the similarity score of trials
directly from the inputs, and 3) optimize both of them by means of
verification-based loss functions [17, 18].

Even though many novel ASV approaches are proposed every year, there is one
scenario that has not been fully explored: a module that handles multiple
enrollment utterances. For practical cases where it is more reliable to use
multiple enrollment utterances, the operation of simply concatenating
waveforms or averaging speaker embeddings is generally applied to these
multiple enrollment utterances to obtain a single speaker representative
vector [19, 20, 21], which is then treated as a single enrollment utterance.

Figure 1: General architecture of ASV.

In our previous study [22], we demonstrated that widely used simple operations
such as concatenation and averaging for multiple enrollment utterances cannot
efficiently leverage the intra-relationships among them. In order to better
handle multiple enrollment utterances, a new neural back-end called attention
back-end, which utilizes a scaled-dot self-attention mechanism [23] to learn
the acoustic variations among a varying number of enrollment utterances, has
been proposed, along with a feed-forward self-attention mechanism [24] to
aggregate multiple speaker embeddings into a single speaker representative
vector by using an adaptive weighted sum.

In this paper, to handle the case of multiple enrollment utterances more
appropriately, we extend our earlier proposed attention back-end to an E2E
approach by jointly fine-tuning the front-end speaker encoder and the
attention back-end simultaneously. We argue that the intra-speaker acoustic
variations contained in multiple enrollment utterances can also overcome some
of the intrinsic biases of the pretrained speaker encoder introduced by
classification-based learning [12, 25, 14] or metric learning [26, 10, 11].
Additionally, considering the problem of heavily unbalanced positive and
negative trials existing in the trials-sampling method [22], focal loss [27]
is selected to replace the binary cross-entropy loss to dynamically resample
positive and negative trials and emphasize the contribution of difficult
trials at the same time. We also introduce a unique embedding-level mixup [28,
29]-based data augmentation strategy to increase the diversity of the training
data.

The rest of this paper is presented as follows: In Section II, we provide an
overview of representative works, including the details of front-end speaker
encoders, frequently used loss functions, and both conventional and neural
back-end models. Section III introduces the new fully E2E approach extending
our previously proposed attention back-end to handle the case of multiple
enrollment utterances. The method of sampling training trials for multiple
enrollment utterances is also described in detail in this section. Section IV
lists the loss functions utilized in both the attention back-end and the
proposed fully E2E model. Section V describes the signal-level and embedding-
level data augmentation techniques we applied in ASV. In Section VI, we report
the experimental results of the proposed fully E2E method on the CNCeleb
dataset for the case of multiple enrollments by comparing it with models that
use separately-trained speaker encoder and back-end models as well as other
SOTA E2E methods. We present an experiment in which we stack our proposed
points gradually to determine the individual improvements. A deeper analysis
of the effect of the number of enrollment utterances is also provided in this
section. We conclude Section VII with a brief summary.

##  II Related work

###  II-A Stage-Wise Architecture of ASV

An automatic speaker verification system usually consists of a front-end
speaker encoder for extracting a speaker embedding and a back-end scoring
model for giving a pair-wise score, which measures the similarity between
enrollment utterances and the testing utterance. The whole system is
illustrated at the top of Fig. 1.111In addition to the modules mentioned
above, there is a preprocessing module that conducts centering, length
normalization, and other operations on the speaker embedding.

####  II-A1 Front-End Speaker Encoder

##### Architecture of Speaker Encoder

The speaker encoder, which projects a duration-varied utterance or acoustic
features to a fixed-length discriminative speaker embedding, is an essential
component for speaker verification systems. Generally speaking, the
architecture of the speaker encoder (illustrated at the bottom of Fig. 1) can
be divided into three parts, including a frame-level feature extractor, a
pooling layer for aggregation, and a fully connected layers-based classifier.

Many components have been proposed to extract discriminative speaker
embeddings based on the above architecture. TDNN [30] and ResNet [31] were the
first to be applied as the backbone of speaker encoders in [5, 6, 10], and
many variants [7, 32, 33, 34] have appeared since then. This paper utilizes
some representative speaker encoders in both the baseline systems and the
proposed fully E2E approach (described in detail in Section VI-B).

##### Attention Mechanism for Speaker Encoder

The attention mechanism is an essential component of the speaker encoder and
can be incorporated into the frame-level feature extractor and the pooling
layer for aggregation. Although there are many attention-based methods for
speaker encoders, such as vector-based attentive pooling [9], multi-head
attentive pooling [35], multi-resolution multi-head attentive pooling [36],
and others, in this section we review only the Squeeze-and-Excitation (SE)
channel attention [37] for frame-level feature extraction and the attentive
statistics pooling layer [7] for aggregation, as they are used in most of the
models in this paper.

SE channel attention for frame-level feature extraction is introduced to
integrate global information by re-scaling the outputs of convolutional layers
that only focus on local information. In this paper we only examine the
incorporation of the SE module with a 1-dimensional convolutional layer, but a
2-dimensional convolutional layer can also be used.

The squeeze operation first averages the frame-level feature map obtained from
a 1-dimensional convolutional layer with $C$ channels denoted as
$\boldsymbol{M}=[\boldsymbol{m}_{1},\cdots,\boldsymbol{m}_{T}]=[\boldsymbol{f}_{1},\cdots,\boldsymbol{f}_{C}]\top\in\mathbb{R}^{C\times{T}}$
along the time domain, and obtains a vector
$\boldsymbol{z}=\frac{1}{T}\sum_{i}^{T}\boldsymbol{m}_{i}$. Then, a compact
excitation representation
$\boldsymbol{s}=[s_{1},\cdots,s_{C}]^{\top}=\sigma(\boldsymbol{W}_{2}^{\top}f(\boldsymbol{W}_{1}^{\top}\boldsymbol{z}+\boldsymbol{b}_{1})+\boldsymbol{b}_{2})$
is obtained by inputting $\boldsymbol{z}$ into two fully connected (FC)
layers, where $\sigma(\cdot)$ is a sigmoid function and $\boldsymbol{W}_{1}$
and $\boldsymbol{W}_{2}$ are $C\times{\frac{C}{r}}$ and
${\frac{C}{r}\times{C}}$ matrices that control the dimensions of hidden
features with a ratio $r$.

The final output of the SE module is obtained by rescaling the output at each
channel $c$ with its channel attention weight $s_{c}$, that is,
$\boldsymbol{\tilde{M}}=[\boldsymbol{\tilde{f}}_{1},\cdots,\boldsymbol{\tilde{f}}_{C}]\top=[s_{1}\boldsymbol{f}_{1},\cdots,s_{C}\boldsymbol{f}_{C}]\top$.

Attentive statistics pooling (ASP) applies attention mechanisms in the speaker
verification task for aggregation. It concatenates the attention-based
weighted mean and standard deviation of frame-level feature maps into a vector
$\boldsymbol{c}=\text{Concat}(\boldsymbol{\tilde{\mu}},\boldsymbol{\tilde{\sigma}})$,
where $\boldsymbol{\tilde{\mu}}$ and $\boldsymbol{\tilde{\sigma}}$ are the
weighted mean and standard deviation, respectively, and they are calculated
with an attention weight $\alpha_{t}$ at each time step $t$ as follows:

$\displaystyle\boldsymbol{\tilde{\mu}}$
$\displaystyle=\sum_{t}^{T}\alpha_{t}\cdot\boldsymbol{m}_{t},$ (1)
$\displaystyle\boldsymbol{\tilde{\sigma}}$
$\displaystyle=\sqrt{\sum_{t}^{T}\alpha_{t}\boldsymbol{m}_{t}\odot\boldsymbol{m}_{t}-\boldsymbol{\tilde{\mu}}\odot\boldsymbol{\tilde{\mu}}},$
(2) $\displaystyle\alpha_{t}$
$\displaystyle=\frac{\text{exp}(\boldsymbol{v}^{\top}f(\boldsymbol{W}^{\top}\boldsymbol{m}_{t}+\boldsymbol{b})+k)}{\sum_{t}^{T}\text{exp}(\boldsymbol{v}^{\top}f(\boldsymbol{W}^{\top}\boldsymbol{m}_{t}+\boldsymbol{b})+k)},$
(3)

where $\boldsymbol{v}$, $\boldsymbol{W}$, $\boldsymbol{b}$, and $k$ are
learnable parameters and $f(\cdot)$ is a non-linear function such as ReLU or
Tanh.

##### Loss Function for Speaker Encoder

The loss functions for the speaker encoder can be divided into two categories:
metric learning (such as triplet loss [10]) and softmax-based variants (such
as plain softmax [5] and additive margin softmax (AM-softmax) [14, 15]).
Compared to the former, softmax-based variants possess comparable performance
and a simpler training pipeline without using any sampling method for training
trials, which is usually used in metric learning [38]. Below, we review both
plain softmax and AM-softmax because both of them are used in our experiments.

Plain softmax loss is used for classification, as

$\displaystyle\mathcal{L}_{\text{softmax}}$ $\displaystyle=$
$\displaystyle-\frac{1}{N}$
$\displaystyle\sum_{n=1}^{N}\log\frac{e^{||\boldsymbol{w}_{i}||\cdot||\boldsymbol{e}_{n}||\cos\theta_{ni}}}{e^{||\boldsymbol{w}_{i}||\cdot||\boldsymbol{e}_{n}||\cos\theta_{ni}}+\sum_{j\neq
i}^{M}e^{||\boldsymbol{w}_{j}||\cdot||\boldsymbol{e}_{n}||\cos\theta_{nj}}},$
(4)

where $\theta_{ni}$ is the angle between the speaker embedding
$\boldsymbol{e}_{n}$ of the $n$-th training sample and the weight vector
$\boldsymbol{w}_{i}$ of its target class $i$. Here, the decision boundary
between two different classes is decided by the length of the weight vector
and the corresponding angle between the weight vector and the speaker
embedding. This property leads to poor discrimination when using superficial
cosine similarity as a back-end model. Therefore, the speaker encoder trained
with the plain softmax loss function usually extracts the speaker embedding
from the penultimate hidden layer, as in [6, 7], and a PLDA back-end processes
these embeddings for further improvement.

Considering the undesirable properties of plain softmax loss, many softmax-
based variants have been proposed [12, 15, 25]. AM-softmax is a variant that
can enlarge the distance between decision boundaries among different classes,
as

$\mathcal{L}_{\text{am}}=-\frac{1}{N}\sum_{n=1}^{N}\log\frac{e^{s(\cos\theta_{ni}-m)}}{e^{s(\cos\theta_{ni}-m)}+\sum_{j\neq
i}^{M}e^{s\cos\theta_{nj}}},$ (5)

where almost all annotations are the same as in Eq. (II-A1) except for the two
hyper-parameters $s$ and $m$, which rescale the value of $\cos\theta_{ni}$ and
control the decision boundary margin among different classes, respectively.

####  II-A2 Back-end

##### Cosine Similarity

Cosine score can measure the similarity of two speaker embeddings distributed
in high-dimensional space. It can be formulated as

$\text{Cos}(\boldsymbol{e}_{i},\boldsymbol{e}_{j})=\frac{\boldsymbol{e}_{i}\top\boldsymbol{e}_{j}}{||\boldsymbol{e}_{i}||\cdot||\boldsymbol{e}_{j}||},$
(6)

where $\boldsymbol{e}_{i}$ and $\boldsymbol{e}_{j}$ are speaker embeddings
from the same or different speakers. It has been frequently used as a scoring
method for ASV systems trained using discriminative loss functions [10, 13,
32, 33]. In the case of multiple enrollment utterances, a frequently used
operation is to average the speaker embeddings
$\boldsymbol{c}_{n}=\frac{1}{L}\sum_{l=1}^{L}\boldsymbol{e}_{l}$, where $L$ is
the number of enrollment utterances per speaker. Alternatively, a speaker
embedding may be extracted from concatenated waveforms, or from acoustic
features [22].

##### Probabilistic Linear Discriminant Analysis

Probabilistic Linear Discriminant Analysis (PLDA) is another popular method
for scoring in speaker verification—not only in statistics-based methods such
as i-vector but also in neural network-based methods such as x-vector. There
are many variants of PLDA models, but here we only review the simple Gaussian
PLDA (GPLDA) model:

$\boldsymbol{e}_{n}=\boldsymbol{e}_{\mu}+\boldsymbol{F}\boldsymbol{\omega}_{n}+\boldsymbol{\epsilon}_{n},$
(7)

where $\boldsymbol{e}_{\mu}$ represents the center vector of the training
speaker embeddings. In GPLDA, both the latent variable
$\boldsymbol{\omega}_{n}$ and residual term $\boldsymbol{\epsilon}_{n}$ for
the $n$-th utterance are assumed to follow Gaussian distributions, that is,
$\boldsymbol{\omega}_{n}\sim\mathcal{N}(0,\mathbf{I})$ and
$\boldsymbol{\epsilon}_{n}\sim\mathcal{N}(0,\mathbf{\Sigma})$, where
$\mathbf{I}$ and $\boldsymbol{\Sigma}$ represent identity and covariance
matrices, respectively. The score of a pair of speaker embeddings is given as
follows for a single-session enrollment case:

$\displaystyle P(\boldsymbol{e}_{i},\boldsymbol{e}_{j})$
$\displaystyle=\boldsymbol{e}_{i}^{\top}\boldsymbol{Q}\boldsymbol{e}_{i}+\boldsymbol{e}_{j}^{\top}\boldsymbol{Q}\boldsymbol{e}_{j}+2\boldsymbol{e}_{i}^{\top}\boldsymbol{P}\boldsymbol{e}_{j},$
(8) $\displaystyle\boldsymbol{P}$
$\displaystyle=\boldsymbol{\Sigma}^{-1}_{tot}\boldsymbol{\Sigma}_{ac}(\boldsymbol{\Sigma}_{tot}-\boldsymbol{\Sigma}_{ac}\boldsymbol{\Sigma}^{-1}_{tot}\boldsymbol{\Sigma}_{ac})^{-1},$
(9) $\displaystyle\boldsymbol{Q}$
$\displaystyle=\boldsymbol{\Sigma}^{-1}_{tot}-(\boldsymbol{\Sigma}_{tot}-\boldsymbol{\Sigma}_{ac}\boldsymbol{\Sigma}^{-1}_{tot}\boldsymbol{\Sigma}_{ac})^{-1},$
(10) $\displaystyle\boldsymbol{\Sigma}_{tot}$
$\displaystyle=\boldsymbol{F}\boldsymbol{F}^{\top}+\boldsymbol{\Sigma},$ (11)
$\displaystyle\boldsymbol{\Sigma}_{ac}$
$\displaystyle=\boldsymbol{F}\boldsymbol{F}^{\top},$ (12)

where $\\{\boldsymbol{\mu},\boldsymbol{F},\boldsymbol{\Sigma}\\}$ are
trainable parameters that are estimated using of the Expectation-Maximization
(EM) algorithm.

There are several ways of handling the multi-session enrollment case with PLDA
[39, 40]. In light of the results reported in [39, 40], we adopt the average
of multiple speaker embeddings in our baseline models due to its better
performance.

##### Neural Probabilistic Linear Discriminant Analysis

Inspired by the PLDA pair-wise scoring method, [17] proposed a neural network
model to simulate the behavior of PLDA, called neural PLDA (NPLDA). Referring
to the PLDA implementation in Kaldi [41], NPLDA uses two fully connected
layers as substitutes for the LDA and centering processing. Then, two semi-
orthogonal matrices are leveraged to represent the corresponding
$\boldsymbol{Q}$ and $\boldsymbol{P}$ in Eq. (8), respectively, which measure
the similarity between the enrollment and testing speaker embeddings.
According to [17], initializing the parameters of NPLDA from a well-trained
PLDA model is an important trick to obtain a large improvement over the
conventional PLDA model. For the scenario of multiple enrollment utterances,
there is no special description in [17]. Therefore, we average all enrollment
speaker embeddings as one speaker-representative vector and use NPLDA to
compute the score of this vector and the testing speaker embedding as a
baseline system.

##### Loss Function for Neural Back-end

In general, a back-end model in speaker verification measures the similarity
of a trial pair, which contains one or more enrollment utterances and one
testing utterance. Therefore, the following binary cross-entropy (BCE) loss
function is a good choice to evaluate how well the back-end model is trained:

$\displaystyle\mathcal{L}_{\text{BCE}}=$
$\displaystyle-\sum_{k=1}^{K}[\mathds{1}_{y_{k}=1}\mathrm{log}\sigma(s_{k})+\mathds{1}_{y_{k}\neq
1}\mathrm{log}(1-\sigma(s_{k}))],$ (13)

where $s_{k}$ is the similarity score of the $k$-th trial pair, and
$\mathds{1}$ is an indicator function. The ground-truth label $y_{k}=1$
denotes that the trial pair is positive, i.e., that both the enrollment and
test trial are from the same speaker, while $y_{k}\neq 1$ denotes that it is
negative.

The normalized minimum detection cost function (minDCF) formulated by (14), is
frequently used to evaluate the performance of ASV systems by considering
practical cases, as

$\displaystyle C_{norm}(\beta,\eta)=P_{miss}(\eta)+\beta P_{fa}(\eta),$ (14)

where $\eta$ is a threshold value to make a hard decision. $P_{miss}$ and
$P_{fa}$ are the probability of missing and false alarms decided by $\eta$,
respectively. $\beta$ is a predefined application-dependent number represented
by:

$\displaystyle\beta=\frac{C_{fa}(1-P_{target})}{C_{miss}P_{target}},$ (15)

where $C_{miss}$ and $C_{fa}$ are the costs assigned to miss and false alarms,
respectively, and $P_{target}$ is the prior probability of a target trial
[42].

However, it cannot be used as a loss function due to its non-differentiable
property. There has been an attempt [17] to optimize the minDCF metric
directly for training so that it can be introduced in gradient-based
optimization by approximating it in a differentiable way, as:

$\displaystyle\mathcal{L}_{\text{aDCF}}=\min\limits_{\eta}C_{norm}^{(soft)}(\beta,\eta),$
(16)

where the parameter $\theta$ is differentiable by introducing it into an
approximate version of $P_{miss}$ and $P_{fa}$, which is calculated as:

$\displaystyle P_{miss}^{(soft)}(\eta)$
$\displaystyle=\frac{\sum_{k=1}^{K}y_{k}[1-\sigma(\delta(s_{k}-\eta))]}{\sum_{k=1}^{K}y_{k}},$
(17) $\displaystyle P_{fa}^{(soft)}(\eta)$
$\displaystyle=\frac{\sum_{k=1}^{K}(1-y_{k})\sigma(\delta(s_{k}-\eta))}{\sum_{k=1}^{K}(1-y_{k})}.$
(18)

Here, $k$ is the trial index, $s_{k}$ is the system score, $y_{k}$ denotes the
ground-truth label for trial $k$, and $\sigma$ denotes a sigmoid function. $K$
is the total number of trials in the mini-batch over which the cost is
computed. By choosing a large enough value for the warping factor $\delta$,
the approximation can be made arbitrarily close to the actual detection cost
function for a wide range of thresholds [17].

Figure 2: NPLDA-E2E architecture. The speaker encoder and the NPLDA are
trained independently at first, and the two models are then concatenated for
fine-tuning in E2E style.

###  II-B Fully End-to-End Method

There are several definitions of E2E in the context of speaker verification.
For example, some papers [10, 43, 44] have described their systems, which
train a speaker encoder with a discriminative loss function and use cosine
score to measure the similarity between enrollment and testing speaker
embeddings extracted from the well-trained speaker encoder, as E2E speaker
verification systems. However, in the current work, we refer to a system that
is fully trained directly from waveforms or acoustic features and optimized
with a speaker verification metric, directly outputting the similarity score
for a trial pair in the testing stage, as a fully E2E system (e.g.,
generalized E2E (GE2E) [4], E2E NPLDA [42], and so on [45, 46, 47]). In this
section, we review only the recently proposed representative NPLDA-E2E
architecture, which demonstrated a successful application in the VOiCES
challenge [18].

This approach concatenates a speaker encoder, which is TDNN in [42], and an
NPLDA back-end model in order to optimize an ASV system with the approximated
DCF loss function in Eq. (16) simultaneously. Figure 2 illustrates the
enrollment and evaluation process of this fully E2E model. Additionally, a
novel sampling method of trials is proposed in [42] to simulate the enrollment
and evaluation of the testing stage during the training stage, which improves
the performance due to the resultant consistency. In the case of multiple
enrollment utterances, the enrollment speaker embeddings are simply averaged
before being processed by the NPLDA layers.

Figure 3: Architecture of proposed fully E2E method. The speaker encoder is
trained as a classification task with noise augmentation [48, 49], SpecAugment
[50], and a mixup strategy [28, 29] at first stage. The well-trained speaker
encoder is then concatenated with a randomly initialized attention back-end
for the second E2E fine-tuning stage.

##  III End-to-End Joint Model of Speaker Encoder and Attention Back-end

As discussed in the previous section, a practical speaker verification system
is divided into two parts: a front-end speaker encoder for extracting a
speaker embedding and a back-end for scoring a pair-wise trial. However, the
independent optimization of these separate components is prone to falling into
an undesirable local minimum [46, 42]. Several joint-optimization methods have
been proposed to solve this problem by combining the front-end speaker encoder
and the back-end scoring model [47, 4, 42], and have shown substantial
improvements compared to separate speaker encoders and back-ends.
Nevertheless, none of the above-mentioned models have the capability to make
full use of the intra-relationships among multiple enrollment utterances. In
this section, we propose a new architecture for a fully E2E model that not
only follows the E2E training pipeline but also leverages the intra-
relationships among multiple enrollment utterances. Considering the difficulty
of training a fully E2E neural network directly, we adopt a two-stage training
strategy (shown in Fig. 3) in which a speaker encoder is firstly optimized
with a softmax or AM-softmax loss function and then the parameters of the
speaker encoder are transferred to the fully E2E model for fine-tuning.

###  III-A Discriminative Speaker Encoder

In the first stage of optimization, many common speaker encoders (e.g. TDNN,
ResNet) can be trained as a good seed model for the second fine-tuning stage.
To improve the robustness against complicated scenarios such as noisy
environments, we make use of a few data augmentation tricks, including noise
augmentation, mixup, and SpecAug [50], to increase the diversity of training
data.

###  III-B Fully End-to-End Fine-tuning

In the fine-tuning stage, the well-trained speaker encoder is concatenated
with a randomly initialized attention back-end component, and the concatenated
model is optimized using an E2E training method with a verification-based loss
function. The input of the model is the acoustic features of utterances, and
the output is the scores of pair-wise trials. We implemented multi-level
aggregation methods, including frame-level attention aggregation in some
speaker encoder components and utterance-level attention aggregation in the
attention back-end component, to leverage the intra-relationships of multiple
enrollment utterances.

####  III-B1 Trial Sampling Method

When training the fully E2E model, we adopt a sampling method that allows each
speaker in the same mini-batch to possess an equal number of utterances
instead of randomly sampling utterances in the same manner as the first
training stage. Speaker embeddings can be extracted from the acoustic features
of the utterances in a mini-batch by means of a speaker encoder module (e.g.
TDNN, ResNet) in our fully E2E method, as Fig. 4 shows. Considering that
speakers in a mini-batch can be viewed as either target or non-target
speakers, we rearrange the speaker embeddings in one mini-batch to define
various positive and negative pairs. Table I gives a simple example to
illustrate our method of rearrangement. Assuming there are three speakers,
$A$, $B$, and $C$, and four utterances whose indices range from 1 to 4 per
speaker, we select one utterance as a test trial and use the others of the
same speaker can be used as enrollments for positive pairs. Negative pairs can
be formed by selecting a test trial denoted as ($\checkmark$) from one speaker
and enrollments denoted as ($\times$, $\times$, $\times$) from other speakers.

TABLE I: Composition of pairs of (test-trial, enrollment-data) for training back-end model and ground-truth labels from mini-batch. A, B, and C are speaker IDs, and 1, 2, 3, and 4 are his/her audio IDs. $\checkmark$and $\times$denote test and enrollment audio files, respectively.  | $A$ | $B$ | $C$ |  |  |  |
---|---|---|---|---|---|---|---
| 1 | 2 | 3 | 4 | 1 | 2 | 3 | 4 | 1 | 2 | 3 | 4 |  | Test | Enroll | Label
Trials to be used for training | $\checkmark$ | $\times$ | $\times$ | $\times$ |  |  |  |  |  |  |  |  |  | $\boldsymbol{q}_{A1}$ | $\boldsymbol{h}_{A1}$ | P
$\checkmark$ |  |  |  |  | $\times$ | $\times$ | $\times$ |  |  |  |  |  | $\boldsymbol{q}_{A1}$ | $\boldsymbol{h}_{B1}$ | N
$\checkmark$ |  |  |  |  |  |  |  |  | $\times$ | $\times$ | $\times$ |  | $\boldsymbol{q}_{A1}$ | $\boldsymbol{h}_{C1}$ | N
$\times$ | $\checkmark$ | $\times$ | $\times$ |  |  |  |  |  |  |  |  |  | $\boldsymbol{q}_{A2}$ | $\boldsymbol{h}_{A2}$ | P
| $\checkmark$ |  |  | $\times$ |  | $\times$ | $\times$ |  |  |  |  |  | $\boldsymbol{q}_{A2}$ | $\boldsymbol{h}_{B2}$ | N
| $\checkmark$ |  |  |  |  |  |  | $\times$ |  | $\times$ | $\times$ |  | $\boldsymbol{q}_{A2}$ | $\boldsymbol{h}_{C2}$ | N
|  |  |  |  |  | ⋮ |  |  |  |  |  |  |  |  |
$\times$ | $\times$ | $\times$ |  |  |  |  |  |  |  |  | $\checkmark$ |  | $\boldsymbol{q}_{C4}$ | $\boldsymbol{h}_{A4}$ | N
|  |  |  | $\times$ | $\times$ | $\times$ |  |  |  |  | $\checkmark$ |  | $\boldsymbol{q}_{C4}$ | $\boldsymbol{h}_{B4}$ | N
|  |  |  |  |  |  |  | $\times$ | $\times$ | $\times$ | $\checkmark$ |  | $\boldsymbol{q}_{C4}$ | $\boldsymbol{h}_{C4}$ | P

####  III-B2 Utterance-level Attention

After sampling and rearranging numerous trials from the data in a batch, the
training process is the same as the inference process. Therefore, in the rest
of this section, we only describe the details of how inference is implemented
in the fully E2E model. Assuming a speaker has $K$ enrollment utterances,
which may vary for different speakers, $K$ speaker embeddings
$\\{\boldsymbol{e}_{1},\cdots,\boldsymbol{e}_{K}\\}$ transformed by speaker
encoder $\text{Enc}(\cdot)$ are stacked to form a speaker embedding matrix
$\boldsymbol{E}\in\mathbb{R}^{K\times{D}}$ as input to the following module.
Here, $D$ is the dimension of the speaker embedding generated by the speaker
encoder. Since we argue that the relationship among multiple enrollment
utterances is important in order to make full use of them, we utilize multi-
head scaled dot self-attention (SDSA) [23] here to project them to a latent
matrix denoted by $\boldsymbol{H}\in\mathbb{R}^{K\times{D}}$, as

$\boldsymbol{H}={\rm{Concat}}(\boldsymbol{H}_{1},\boldsymbol{H}_{2},\cdots,\boldsymbol{H}_{d_{1}})\boldsymbol{W}^{O}+\boldsymbol{E},$
(19)

where $d_{1}$ represents the number of heads in SDSA. The $i$-th sub-matrix
$\boldsymbol{H}_{i}\in\mathbb{R}^{K\times\frac{D}{d_{1}}}$ is computed by

$\boldsymbol{H}_{i}=\text{Attn}_{1}(\boldsymbol{Q}_{i},\boldsymbol{K}_{i},\boldsymbol{V}_{i})=\text{Softmax}(\frac{\boldsymbol{Q}_{i}\boldsymbol{K}_{i}^{\top}}{\sqrt{{D}/{d_{1}}}})\boldsymbol{V}_{i}.$
(20)

As described in [23], $\boldsymbol{Q}_{i}$, $\boldsymbol{K}_{i}$, and
$\boldsymbol{V}_{i}$ are query, key, and value matrices, respectively, which
can be obtained by applying linear transformations with trainable parameter
matrices of $\boldsymbol{W}_{i}^{Q}$, $\boldsymbol{W}_{i}^{K}$, and
$\boldsymbol{W}_{i}^{V}$ on input matrix $\boldsymbol{E}$. Through the multi-
head mechanism in Eq. (19), the information of different positions of
$\boldsymbol{E}$ can be attended from different representation sub-spaces
[23].

Figure 4: Proposed method showing expanded detail of the attention back-end
structure. The speaker encoder is pretrained at stage 1 (S1). The whole
system, which consists of the pretrained speaker encoder and a randomly
initialized attention back-end, is fine-tuned at stage 2 (S2).

As $K$ is a varying number, it is necessary to aggregate the latent matrix
$\boldsymbol{H}$ to a speaker representative vector $\boldsymbol{h}$. In order
to do this, another attention mechanism called multi-head feed-forward self-
attention (FFSA) [24] is utilized, as

$\boldsymbol{h}={\text{Concat}}(\boldsymbol{h}_{1},\boldsymbol{h}_{1},\cdots,\boldsymbol{h}_{d_{2}}),$
(21)

where $d_{2}$ represents the number of heads in FFSA. For each head vector
$\boldsymbol{h}_{j}\in\mathbb{R}^{\frac{D}{d_{2}}}$, it is derived as

$\boldsymbol{h}_{j}=\text{Attn}_{2}{(\boldsymbol{\tilde{H}}_{j})}=\text{Softmax}(\boldsymbol{v}_{j}^{\top}\text{Tanh}(\boldsymbol{W}_{j}\boldsymbol{\tilde{H}}_{j}^{\top}))\boldsymbol{\tilde{H}}_{j},$
(22)

where the adaptive weights of $\boldsymbol{h}_{j}$ are learned by a small feed
forward neural network. The trainable parameters
$\boldsymbol{W}_{j}\in\mathbb{R}^{D_{2}\times{\frac{D}{d_{2}}}}$ and
$\boldsymbol{v}_{j}\in\mathbb{R}^{D_{2}}$ are applied on input
$\boldsymbol{\tilde{H}}_{j}$ in order, which is a sub-matrix of
$\boldsymbol{H}$, with successive non-linear transformation Tanh and Softmax,
respectively.

####  III-B3 Score Calibration

After aggregating the enrollment embeddings via attention mechanisms, the
similarity between an evaluating speaker embedding $\boldsymbol{q}$, which is
extracted by the speaker encoder, and the claimed speaker representative
vector $\boldsymbol{h}$ is measured by cosine score, as

$\displaystyle\mathrm{Cos}(\boldsymbol{q},\boldsymbol{h})$
$\displaystyle=\frac{\boldsymbol{q}^{\top}\boldsymbol{h}}{||\boldsymbol{q}||\cdot||\boldsymbol{h}||}.$
(23)

Next, a logistic regression (LR) is used for better calibration, as

$\displaystyle P(\boldsymbol{q},\boldsymbol{h})$
$\displaystyle=\frac{1}{1+\exp^{-s}}=\frac{1}{1+\exp^{-a\,\text{Cos}(\boldsymbol{q},\boldsymbol{h})-b}},$
(24)

where $a$ and $b$ are learnable parameters of LR and
$P(\boldsymbol{q},\boldsymbol{h})$ denotes the probability that the speaker of
the evaluating utterance is the same as that of the enrollment utterances.

##  IV Loss Functions

Using a classification-based loss function in the speaker encoder training
stage would be inconsistent with the verification process. Therefore, in the
fine-tuning stage, we utilize the weighted sum of the attention mechanism-
based generalized E2E (AGE2E) loss and the binary focal loss as the criterion
to supervise the fine-tuning of the proposed method, as

$\mathcal{L}=\lambda\mathcal{L}_{\mathrm{AGE2E}}+(1-\lambda)\mathcal{L}_{\mathrm{FOCAL}},$
(25)

where AGE2E loss is a variant of the generalized E2E (GE2E) loss and is
designed to better leverage the training trial pairs selected by our trial
sampling method in each mini-batch [22]. At the same time, the focal loss is
expected to alleviate the issue of unbalanced positive and negative training
pairs stemming from the trial sampling method. It also encourages the model to
focus on difficult trial pairs. The hyper-parameter $\lambda$ is set to 0.6 in
this paper.

###  IV-A Attention-based Generalized End-to-End Loss

The AGE2E loss is used in our model to reduce within-class variance and
enlarge between-class distances. Unlike BCE, AGE2E loss exploits the
information of the trial pairs in the mini-batch. Let us consider the example
in Table I and define $\boldsymbol{q}_{lm}$ as a speaker embedding vector
extracted from the $m$-th test trial of speaker $l$. Let us also define
$\boldsymbol{h}_{nm}$ as the vector $\boldsymbol{h}$ of Eq. (21) extracted
from the enrollment set that contains multiple audio files uttered by speaker
$n$ (except the $m$-th test trial). Given all the $\boldsymbol{q}_{lm}$ and
$\boldsymbol{h}_{nm}$ from the mini-batch, the AGE2E loss is computed by

$\displaystyle\mathcal{L}_{\mathrm{AGE2E}}$
$\displaystyle=-\sum_{{}^{\forall}l,m}\log\frac{\exp^{P(\boldsymbol{q}_{lm},\boldsymbol{h}_{lm})}}{\sum_{{}^{\forall}n}\exp^{P(\boldsymbol{q}_{lm},\boldsymbol{h}_{nm})}}.$
(26)

Although the definition above is similar to the GE2E loss [4], a crucial
difference is on $\boldsymbol{h}$. While the original GE2E simply averages
multiple embeddings per speaker and uses the resulting centroid as
$\boldsymbol{h}$, we utilize the attention mechanism in Eqs. (21–22) to
compute the $\boldsymbol{h}$. This is why we refer to it as attention-based
GE2E. The attention-based $\boldsymbol{h}$ is arguably more powerful than the
simple centroid to represent the speaker in the embedding space.

###  IV-B Focal Loss

As discussed in [22], the trial sampling strategy results in a heavily
unbalanced data ratio of positive and negative trials. Furthermore, BCE loss
in [22] treats difficult and easy trials equally, which is not ideal in our
use case. We use focal loss [27, 51] as a substitute for BCE loss to alleviate
this problem.

The focal loss used for a binary classification task is defined on the basis
of the BCE loss and can be written as

$\displaystyle\mathcal{L}_{\text{FOCAL}}$ $\displaystyle=$
$\displaystyle-\sum_{{}^{\forall}l,m,n}[\mathds{1}_{l=n}\alpha(1-P(\boldsymbol{q}_{lm},\boldsymbol{h}_{nm}))^{\gamma}\log
P(\boldsymbol{q}_{lm},\boldsymbol{h}_{nm})$ $\displaystyle+\mathds{1}_{l\neq
n}(1-\alpha)P(\boldsymbol{q}_{lm},\boldsymbol{h}_{nm})^{\gamma}\log(1-P(\boldsymbol{q}_{lm},\boldsymbol{h}_{nm}))],$
(27)

where the hyper-parameter $\alpha$ controls the balance between positive and
negative classes, and
$(1-P(\boldsymbol{q}_{lm},\boldsymbol{h}_{nm}))^{\gamma}$ and
$P(\boldsymbol{q}_{lm},\boldsymbol{h}_{nm})^{\gamma}$ are modulating factors
that dynamically adjust the weight of the loss for each trial pair. The
_focusing_ hyper-parameter $\gamma\geq 0$ further controls the scale of the
modulating factors. When $\gamma=0$ and $\alpha=0.5$, the focal loss function
is equivalent to the BCE loss.

The focal loss encourages the model to focus on difficult pairs. Let us
consider the case where $l\neq n$, i.e., the test trial and enrollment
utterances, are from different speakers. If the input pair is easy to
classify, the E2E model should produce a small
$P(\boldsymbol{q}_{lm},\boldsymbol{h}_{nm})$, and the resulting modulating
factor $P(\boldsymbol{q}_{lm},\boldsymbol{h}_{nm})^{\gamma}$ also becomes
small. Consequently, the loss from this easy pair is down-weighted. In
contrast, if the model produces a large
$P(\boldsymbol{q}_{lm},\boldsymbol{h}_{nm})$ close to 1, the modulation factor
$P(\boldsymbol{q}_{lm},\boldsymbol{h}_{nm})^{\gamma}$ will also be close to 1,
and the loss from this difficult pair is not down-weighted. Thus, the
difficult trial pair contributes more to the training loss as well as to the
gradient. By replacing the BCE loss with focal loss and adjusting the two
hyper-parameters $\\{\alpha,\gamma\\}$, we observed that this mechanism could
indeed help the model converge to a better state in the experiments.

##  V Multi-level Data Augmentation

Data augmentation plays a key role in building a state-of-the-art ASV system
[5, 33, 29, 32]. In addition to simple noise augmentation, SpecAug [50] and
speaker augmentation [52] are frequently used to increase the robustness of an
ASV system. Moreover, a mixup strategy, which minimizes the vicinal risk [28]
in training, has also been investigated at both the signal and acoustic
feature levels [29]. Instead of the implementation in [29], we applied the
mixup strategy at the speaker embedding level to augment the training data in
the fine-tuning stage. As far as we know, this is the first time an embedding-
level mixup strategy has been applied to the ASV task. We explain it in detail
after first introducing the standard signal-level augmentation method.

###  V-A Signal-level Augmentation

In our proposed method, we use the MUSAN [48] and RIRs [49] corpora for
signal-level data augmentation. Here, we follow the KALDI [41] CNCeleb recipe
to implement data augmentation in both the speaker encoder pretraining stage
and the fully E2E fine-tuning stage.

We also use mixup [28, 29] from acoustic features is also used in the
pretraining stage to improve the generalization performance of the model.
Mixup can be viewed as a data augmentation method, and we use it here to
create new vicinal samples from the acoustic feature level. The mixup
operation can be written as

$\displaystyle\tilde{\boldsymbol{x}}=\beta_{1}\boldsymbol{x}_{1}+(1-\beta_{1})\boldsymbol{x}_{2},$
(28)
$\displaystyle\tilde{\boldsymbol{y}}=\beta_{1}\boldsymbol{y}_{1}+(1-\beta_{1})\boldsymbol{y}_{2},$
(29)

where $\\{\boldsymbol{x}_{1},\boldsymbol{y}_{1}\\}$ and
$\\{\boldsymbol{x}_{2},\boldsymbol{y}_{2}\\}$ are the pairs of acoustic
features and one-hot encodings of target labels from two randomly selected
data samples. The weight $\beta_{1}$ can be generated from a Beta distribution
with a hyper-parameter $\psi_{1}$, i.e.,
$\beta_{1}\sim\text{Beta}(\psi_{1},\psi_{1})$. In all experiments in this
paper, the parameter $\psi_{1}$ is set to 1.0.

The new data sample $\\{\tilde{\boldsymbol{x}},\tilde{\boldsymbol{y}}\\}$ is
used as augmented data to train the neural speaker encoder for extracting
speaker embeddings. In the speaker encoder pretraining stage, AM-softmax or
other margin-based variants on the augmented data can be written as

$\displaystyle\mathcal{L}_{am}(\tilde{\boldsymbol{x}},\tilde{\boldsymbol{y}})=\beta_{1}\mathcal{L}_{am}({\tilde{\boldsymbol{x}}},\boldsymbol{y}_{1})+(1-\beta_{1})\mathcal{L}_{am}({\tilde{\boldsymbol{x}}},\boldsymbol{y}_{2}).$
(30)

###  V-B Embedding-level Augmentation

Signal-level augmentation has exhibited success in the automatic speaker
verification task [5, 32, 29]. However, in our system, we hypothesize that the
mixup augmentation at the embedding level is also helpful, since the
distribution of training pairs in our fully E2E method is heavily unbalanced
due to adopting the sampling method in III-B. The embedding-level mixup
strategy is applied on the speaker embedding $\boldsymbol{q}$ of the test
trial, as shown in Fig. 4. It is inserted before computing the cosine
similarity between $\boldsymbol{h}$ and $\boldsymbol{q}$. This operation can
be formulated as

$\displaystyle\tilde{\boldsymbol{q}}=\beta_{2}\boldsymbol{q}_{1}+(1-\beta_{2})\boldsymbol{q}_{2},$
(31) $\displaystyle\tilde{y}=\beta_{2}y_{1}+(1-\beta_{2})y_{2},$ (32)

where $\boldsymbol{q}_{1}$ and $\boldsymbol{q}_{2}$ are testing speaker
embeddings of the test branch in Fig. 4. $y_{1}$ and $y_{2}$ are the
corresponding labels of the pairs of these testing speaker embeddings and a
speaker representative vector $\boldsymbol{h}$. The weight $\beta_{2}$ is also
generated from a Beta distribution with a hyper-parameter $\psi_{2}$, which is
set to 1.0 in this paper. The similarity between the mixup augmented testing
speaker embedding $\tilde{\boldsymbol{q}}$ and the speaker representative
vector $\boldsymbol{h}$ is computed by using cosine score followed by logistic
regression for score calibration. Therefore, the loss function for embedding-
level mixup can be written as

$\displaystyle\mathcal{L}(\tilde{\boldsymbol{q}},\boldsymbol{h},\tilde{y})=\beta_{2}\mathcal{L}({\boldsymbol{\tilde{q}}},\boldsymbol{h},y_{1})+(1-\beta_{2})\mathcal{L}({\boldsymbol{\tilde{q}}},\boldsymbol{h},y_{2}).$
(33)

TABLE II: TDNN encoder architecture used as a front-end model in our experiment. The normal cross-entropy based on speaker labels was used to train this model.  Layer | Layer context | Total contexts | Input $\times$ output
---|---|---|---
TDNN-ReLU1 | [-2,+2] | 5 |  $C$ $\times$ 512
TDNN-ReLU2 | {-2, 0, +2} | 9 |  1536 $\times$ 512
TDNN-ReLU3 | {-3, 0, +3} | 15 |  1536 $\times$ 512
TDNN-ReLU4 | {0} | 15 |  512 $\times$ 512
TDNN-ReLU5 | {0} | 15 |  512 $\times$ 1500
SP | [0, T) | $T$ |  1500$T$ $\times$ 3000
FC1 | {0} | $T$ |  3000 $\times$ 512
FC2 | {0} | $T$ |  512 $\times$ 512
Softmax | {0} | $T$ |  512 $\times$ $M$
TABLE III: TDNN-ASP encoder architecture used as a front-end model in our experiment. The normal cross-entropy based on speaker labels was used to train this model.  Layer | Layer context | Total contexts | Input $\times$ output
---|---|---|---
TDNN-ReLU1 | [-2,+2] | 5 |  $C$ $\times$ 512
TDNN-ReLU2 | {-2, 0, +2} | 9 |  1536 $\times$ 512
TDNN-ReLU3 | {-3, 0, +3} | 15 |  1536 $\times$ 512
TDNN-ReLU4 | {0} | 15 |  512 $\times$ 512
TDNN-ReLU5 | {0} | 15 |  512 $\times$ 1500
ASP | [0, T) | $T$ |  1500$T$ $\times$ 3000
FC1 | {0} | $T$ |  3000 $\times$ 512
FC2 | {0} | $T$ |  512 $\times$ 512
Softmax | {0} | $T$ |  512 $\times$ $M$
TABLE IV: ECAPA-TDNN encoder architecture used as a front-end model in our experiment. The normal cross-entropy based on speaker labels was used to train this model.  Layer | Layer context | Total contexts | Input $\times$ output
---|---|---|---
TDNN-ReLU1 | [-2,+2] | 5 |  $C$ $\times$ 512
SE-Res2TDNN1 | {-2, 0, +2} | 9 |  1536 $\times$ 512
SE-Res2TDNN2 | {-3, 0, +3} | 15 |  1536 $\times$ 512
SE-Res2TDNN3 | {-4, 0, +4} | 24 |  1536 $\times$ 512
Concat | - | - |  512 $\times$ 1536
TDNN-ReLU2 | {0} | 24 |  1536 $\times$ 1536
ASP | [0, T) | $T$ |  1536$T$ $\times$ 3072
FC1 | {0} | $T$ |  3072 $\times$ 512
FC2 | {0} | $T$ |  512 $\times$ 512
Softmax | {0} | $T$ |  512 $\times$ $M$
TABLE V: SEResNet34 architecture used as a front-end model in our experiment. AM-softmax loss function was used to train this model.  Layer | Kernel size | Stride | Output
---|---|---|---
Conv1 |  3 $\times$ 3 $\times$ 32 |  1 $\times$ 1 |  $L$ $\times$ 64 $\times$ 32
SE-Res1 |  3 $\times$ 3 $\times$ 32 |  1 $\times$ 1 |  $L$ $\times$ 64 $\times$ 32
SE-Res2 |  3 $\times$ 3 $\times$ 64 |  2 $\times$ 2 |  $L/2$ $\times$ 32 $\times$ 64
SE-Res3 |  3 $\times$ 3 $\times$ 128 |  2 $\times$ 2 |  $L/4$ $\times$ 64 $\times$ 128
SE-Res4 |  3 $\times$ 3 $\times$ 256 |  2 $\times$ 2 |  $L/8$ $\times$ 128 $\times$ 256
Flatten | - | - |  $L/8$ $\times$ 2048
ASP | - | - | 4096
FC | 512 | - | 512
AM-softmax | $N$ | - | $N$

##  VI Experiments

###  VI-A Datasets

We conducted all experiments on the CNCeleb [19, 20] dataset, collected from
Chinese celebrities in 11 different genres. It contains two subsets called
CNCeleb1 [19] and CNCeleb2 [20]. CNCeleb1 has 800 speakers for training and
200 for evaluation, and CNCeleb2 has 2,000 speakers for training. The dataset
includes more than 600,000 utterances in total, and the duration is more than
1,000 hours. In contrast to the single-genre data of the widely used VoxCeleb
dataset [53, 54], CNCeleb contains 11 genres that span interviews, singing,
conversation, speech, and so on. The mismatch of genre between enrollment
utterance and testing utterance leads to performance degradation of successful
speaker verification models such as x-vector and ResNet with margin-based
softmax on the VoxCeleb dataset [20]. In our experimental configuration,
training data from CNCeleb1 and CNCeleb2 were used for training, and the
performance evaluation was conducted on the CNCeleb1 test dataset.

In addition to the training and evaluation datasets mentioned above, we use
the MUSAN [48] and RIRs [49] datasets for noise augmentation. The former
contains three types of noise, and the latter contains reverberation data in
several different conditions.

###  VI-B Speaker Encoders

In our experiments, we trained four different but representative speaker
encoders with 23-dimensional Mel-Frequency Cepstrum Coefficients (MFCCs) as
speaker embedding extractors. In the training stage of the speaker encoder,
noise augmentation with MUSAN, RIRs, mixup, and SpecAug [50] are utilized to
augment the training data. For each speaker encoder, we utilized cosine
similarity, PLDA, NPLDA, and attention back-end with the AGE2E loss and the
focal loss on the corresponding speaker embeddings as baseline systems.

####  VI-B1 TDNN

The first speaker encoder is TDNN, the detailed configuration of which is
provided in Table II. This speaker encoder uses TDNN as a frame-level feature
extractor, simple statistics pooling for aggregation, and a plain softmax loss
function, without using any attention mechanism.

####  VI-B2 TDNN-ASP

The TDNN-ASP speaker encoder, whose architecture is shown in Table III, is
similar to the above TDNN model except for the aggregation method. It uses
attentive statistics pooling [7], which summarizes frames using different
weights generated by the attention mechanism.

####  VI-B3 ECAPA-TDNN

ECAPA-TDNN incorporates SE channel attention into the frame-level feature
extractor and concatenates shallow and deep feature maps before the attention-
based aggregation layer. Note that [32] used only one FC layer after
aggregation and utilized additive angular margin softmax (AAM-softmax) as the
loss function, but in our baseline system, we follow the architecture of
x-vector in [5], which used two FC layers after aggregation and plain cross-
entropy as the loss function. The details are provided in Table IV.

####  VI-B4 SE-ResNet34

The final speaker encoder is SE-ResNet34, whose architecture is shown in Table
V. It embeds SE channel attention in a ResNet34 model to make use of global
information. It also uses AM-softmax as the loss function to maximize the
between-class distance and minimize within-class variance.

TABLE VI: Comparison of fully E2E method with separated speaker encoder and back-ends. “Concat” represents concatenating operation. “Mean” represents averaging operation. Speaker encoder | Back-end | E2E | Enroll process. | EER(%) | minDCF(0.01)
---|---|---|---|---|---
TDNN | Cosine | $\times$ | Concat | 15.25 | 0.6784
TDNN | Cosine | $\times$ | Mean | 14.03 | 0.6700
TDNN [20] | PLDA | $\times$ | Concat | 12.52 | –
TDNN | PLDA | $\times$ | Concat | 12.09 | 0.6105
TDNN | PLDA | $\times$ | Mean | 13.29 | 0.6522
TDNN | NPLDA | $\times$ | Concat | 11.11 | 0.5274
TDNN | NPLDA | $\times$ | Mean | 10.63 | 0.5335
TDNN | Attention | $\times$ | Attention | 9.49 | 0.5405
TDNN | Attention | $\checkmark$ | Attention | 8.89 | 0.5207
TDNN-ASP | Cosine | $\times$ | Concat | 13.56 | 0.6460
TDNN-ASP | Cosine | $\times$ | Mean | 11.63 | 0.6157
TDNN-ASP | PLDA | $\times$ | Concat | 11.63 | 0.6157
TDNN-ASP | PLDA | $\times$ | Mean | 10.67 | 0.5990
TDNN-ASP | NPLDA | $\times$ | Concat | 10.90 | 0.5322
TDNN-ASP | NPLDA | $\times$ | Mean | 10.46 | 0.5414
TDNN-ASP | Attention | $\times$ | Attention | 9.54 | 0.5447
TDNN-ASP | Attention | $\checkmark$ | Attention | 9.02 | 0.5238
ECAPA-TDNN | Cosine | $\times$ | Concat | 13.19 | 0.6270
ECAPA-TDNN | Cosine | $\times$ | Mean | 11.79 | 0.6117
ECAPA-TDNN | PLDA | $\times$ | Concat | 10.54 | 0.5806
ECAPA-TDNN | PLDA | $\times$ | Mean | 9.51 | 0.5665
ECAPA-TDNN | NPLDA | $\times$ | Concat | 10.09 | 0.5186
ECAPA-TDNN | NPLDA | $\times$ | Mean | 9.70 | 0.5186
ECAPA-TDNN | Attention | $\times$ | Attention | 8.63 | 0.4958
ECAPA-TDNN | Attention | $\checkmark$ | Attention | 8.04 | 0.4879
SE-ResNet34 | Cosine | $\times$ | Concat | 11.23 | 0.5903
SE-ResNet34 | Cosine | $\times$ | Mean | 10.26 | 0.5687
SE-ResNet34 | PLDA | $\times$ | Concat | 10.81 | 0.6078
SE-ResNet34 | PLDA | $\times$ | Mean | 10.48 | 0.6006
SE-ResNet34 | NPLDA | $\times$ | Concat | 49.17 | 1.0000
SE-ResNet34 | NPLDA | $\times$ | Mean | 53.91 | 1.0000
SE-ResNet34 | Attention | $\times$ | Attention | 9.14 | 0.5414
SE-ResNet34 | Attention | $\checkmark$ | Attention | 8.62 | 0.5229

###  VI-C Training Methodology

Since we conducted several experiments on the speaker encoder, back-end model,
and fully E2E model separately, in this section, we describe the training
methodology for each of these models individually.

For the speaker encoder, we used the above models to extract speaker
embeddings for all back-end models. TDNN series were trained using the AdamW
optimizer [55] for 21 epochs with a learning rate of 0.001, a momentum of
0.99, and a second-order momentum of 0.999. We used cosine annealing with a
restart strategy [56] to schedule the learning rate with three epochs and two
factors. The SE-ResNet34 was trained using a simple SGD optimizer for 12
epochs with a learning rate of 0.01, a momentum of 0.9, and a learning rate
scheduler that multiplies the learning rate by 0.1 if no improvement on the
validation loss is seen for 10,000 iterations.222It is implemented using the
ReduceOnPlateau Pytorch API [57]. All speaker encoders were trained using the
ASV-Subtools [58] toolkit.

Four back-end models were compared in our experiments. Three of these were
trainable ones, namely, PLDA, NPLDA, and attention back-end; cosine similarity
was also used. For PLDA, we followed the implementation of KALDI [41] in which
two-covariance PLDA training follows mean subtraction for centralization and
linear discriminant analysis (LDA) for dimension reduction from 512 to 256.
The PLDA model was trained using all speaker embeddings extracted from the
training data (including the augmented data) for ten iterations using the EM
algorithm. As for the NPLDA model, we used the configuration of [17] and all
speaker embeddings from the training data to train it. Parameters of the
previous PLDA model were used to initialize the NPLDA model. We trained the
attention back-end by using the same configuration as [22] for 40 epochs.

In order to compare other fully E2E methods with our proposed joint approach,
we also trained three baseline systems: the tuple E2E (TE2E) [47], generalized
E2E (GE2E) [4] and NPLDA E2E [42] models. Both the TE2E and GE2E models were
trained using an SGD optimizer for 50 epochs. Its learning rate was
initialized at 0.01 and decayed by 0.1 every 15 epochs. As for the NPLDA E2E
model, we followed the training configuration described in [42]. The
previously trained speaker encoder and NPLDA back-end were used to initialize
the NPLDA E2E model for better optimization. For training our joint model, the
parameters of the pretrained speaker encoder were used to initialize the
speaker encoder component, and the back-end component was initialized
randomly. The model was trained using an SGD optimizer with a learning rate
initialized at 0.0001 and decayed by 0.95 every epoch for 50 epochs.

###  VI-D Comparison of Fully E2E Method with Separated Speaker Encoders and
Back-ends

In the first experiment, in order to determine the effect of E2E joint
optimization, all the mentioned speaker encoders and back-end models discussed
above were trained, and then compared with the proposed fully E2E models by
jointly optimizing the combined speaker encoders with attention back-end
models. The results of EER and minDCF(0.01) are divided into four groups in
accordance with the type of speaker encoders, as shown in Table VI. As we can
see, for all the speaker encoders we used, jointly optimizing the pretrained
speaker encoder with the attention back-end model outperforms the
corresponding separated speaker encoders with all back-end models. These
results also demonstrate that our previously proposed attention back-end in
[22] surpasses other conventional back-end models and the neural network-based
NPLDA in the case of multiple enrollment utterances.

Additionally, no matter which speaker encoder or which back-end model we used,
the mean operation of averaging speaker embeddings on multiple enrollments is
better than the operation of concatenating waveforms of various enrollments.
Therefore, in the following experiments, apart from the attention mechanism
for the enrollment process and the proposed fully E2E method, all other
reported results are based on the mean operation for multiple enrollments.

###  VI-E Comparison with Other E2E methods

Various previously developed E2E methods [47, 4, 42] have also reported
excellent results for the speaker verification task. In this part, we compared
our proposed fully E2E method with these SOTA E2E methods based on the ECAPA-
TDNN speaker encoder, as shown as Table VII. The ECAPA-TDNN speaker encoder
was first trained based on a speaker classification task and then fine-tuned
with different E2E methods. The GE2E method is an improved version of the TE2E
method, and as we can see in the table, GE2E is much better than TE2E in terms
of the EER and minDCF(0.01) metrics. Our proposed fully E2E method
outperformed both the GE2E and NPLDA-E2E methods in terms of the EER metric,
by relatively 21.4% and 13.0%, respectively, which indicates that the
attention mechanism in our method is much more suitable for handling the case
of multiple enrollments. As for the minDCF(0.01) metric, our fully E2E method
is slightly better than NPLDA-E2E, which is far better than the GE2E method.

TABLE VII: Comparison with other E2E methods based on ECAPA-TDNN speaker encoder. Method |  EER(%) |  minDCF(0.01)
---|---|---
TE2E [47] | 10.64 | 0.5769
GE2E [4] | 10.23 | 0.5535
NPLDA-E2E-aDCF [42] | 9.24 | 0.4922
Fully E2E | 8.04 | 0.4879

###  VI-F Effect of Proposed Elements

There are three novel points in our proposed fully E2E method. To figure out
the contribution of each component, we conducted an experiment that integrated
each part one by one to make the contribution of each element clear. In
addition, the separated ECAPA-TDNN speaker encoder with the conventional PLDA
back-end model and the neural network-based attention back-end model are
compared, as shown in Table VIII. From the first and second lines of this
table, it is obvious that the attention back-end proposed in [22] can make
better use of multiple enrollments compared with the PLDA back-end no matter
which evaluation metrics we used. Comparing the second and third lines, we
found that fully E2E fine-tuning can overcome some of the intrinsic biases in
the speaker encoder introduced by the classification-based loss function.
After applying E2E fine-tuning, the EER metric was improved by 4.8%
relatively, although the minDCF(0.01) metric was slightly worse than the
attention back-end. From the fourth line, we found that focal loss is able to
dynamically alleviate the problem of unbalanced data, as mentioned in the
previous section. Finally, the fifth line proves that embedding-level mixup
can improve the performance further by increasing the diversity of training
data.

TABLE VIII: Effect of proposed elements based on ECAPA-TDNN speaker encoder. System | EER(%) | minDCF(0.01)
---|---|---
ECAPA-TDNN/PLDA | 9.51 | 0.5665
ECAPA-TDNN/Attention backend [22] | 8.93 | 0.5043
+E2E fine-tune | 8.50 | 0.5075
++Focal loss | 8.26 | 0.4909
+++Embedding mixup | 8.04 | 0.4879

###  VI-G Effect of Number of Enrollment Utterances

To further investigate the performance of our proposed E2E method, we
experimented with varying the number of enrollment utterances made by
individual speakers. Instead of using the official CNCeleb evaluation
protocol, we selected speakers from this protocol whose numbers of enrollments
are equal to or more than five as enrollment speakers for a fair comparison.
In this experiment, the number of enrollment utterances is gradually increased
from $K=1$ to $K\geq 5$ to examine the effect of the number of enrollments.
The results shown in Table IX indicate that enrollment with multiple
utterances is much better than using a single one in all situations, as
expected. Moreover, as shown in Fig. 5, it is obvious that our fully E2E model
not only performs better for multiple enrollment utterances but is also
competitive in the case of a single enrollment utterance.

TABLE IX: Breakdown EER(%) results according to number of enrollment utterances of individual speakers included in the eval set. For enrollment processing, concatenation and mean operation were used for ECAPA-TDNN cases. System | $K=1$ | $K=2$ | $K=3$ | $K=4$ | $K\geq 5$
---|---|---|---|---|---
ECAPA-TDNN/PLDA | 18.44 | 15.49 | 12.85 | 12.49 | 10.11
ECAPA-TDNN/NPLDA | 18.15 | 15.50 | 12.72 | 12.14 | 9.71
ECAPA-TDNN/NPLDA-E2E | 17.76 | 14.44 | 12.09 | 11.57 | 9.49
ECAPA-TDNN/Attention backend | 17.90 | 14.56 | 10.83 | 10.14 | 8.40
Fully E2E | 17.57 | 14.08 | 10.22 | 9.64 | 7.96
Figure 5: Effect of the number of enrollments. The horizontal axis represents
the number of enrollments, where 5 means equal to or higher than 5. The left
vertical axis shows EER in percentage. The right vertical axis shows the
relative improvements in percentage compared with the single enrollment case
for each system.

##  VII Conclusion

In this paper, we proposed a fully E2E method based on our previously proposed
attention back-end model for ASV in order to handle the case of multiple
enrollments. To solve the problem of unbalanced training data and to increase
the diversity of this data, we incorporate the focal loss as well as a unique
embedding-level mixup strategy to optimize the models in a fully E2E style.
Experiments on the CNCeleb dataset showed that the proposed approach achieved
relative improvements of 21.4% and 13.0% over the two compared SOTA E2E
methods, respectively. Moreover, compared with separated ECAPA-TDNN speaker
encoder and back-end models, our method outperformed the best baseline by an
average of 6.8% relatively, thus demonstrating the advantages of the proposed
fully E2E method. We also conducted an experiment to determine the effect of
the number of enrollment utterances and found that our fully E2E method not
only performed better for multiple enrollment utterances but was also
competitive in the case of a single enrollment utterance.

In future work, we will investigate a new direction considering how to
simultaneously solve mismatch problems, including general training and testing
mismatch, enrollment and evaluation mismatch, and spoofing attacks. We believe
it will make the automatic speaker verification system more practical,
especially considering the threat of recent deepfake techniques.

## References

  * [1] D. A. Reynolds, T. F. Quatieri, and R. B. Dunn, “Speaker verification using adapted gaussian mixture models,” _Digital signal processing_ , vol. 10, no. 1-3, pp. 19–41, 2000.
  * [2] N. Dehak, P. J. Kenny, R. Dehak, P. Dumouchel, and P. Ouellet, “Front-end factor analysis for speaker verification,” _IEEE Transactions on Audio, Speech, and Language Processing_ , vol. 19, no. 4, pp. 788–798, 2010.
  * [3] E. Variani, X. Lei, E. McDermott, I. L. Moreno, and J. Gonzalez-Dominguez, “Deep neural networks for small footprint text-dependent speaker verification,” in _2014 IEEE international conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 2014, pp. 4052–4056.
  * [4] L. Wan, Q. Wang, A. Papir, and I. L. Moreno, “Generalized end-to-end loss for speaker verification,” in _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2018, pp. 4879–4883.
  * [5] D. Snyder, D. Garcia-Romero, D. Povey, and S. Khudanpur, “Deep neural network embeddings for text-independent speaker verification.” in _Interspeech_ , 2017, pp. 999–1003.
  * [6] D. Snyder, D. Garcia-Romero, G. Sell, D. Povey, and S. Khudanpur, “X-vectors: Robust dnn embeddings for speaker recognition,” in _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2018, pp. 5329–5333.
  * [7] K. Okabe, T. Koshinaka, and K. Shinoda, “Attentive statistics pooling for deep speaker embedding,” _arXiv preprint arXiv:1803.10963_ , 2018.
  * [8] H. Zeinali, S. Wang, A. Silnova, P. Matějka, and O. Plchot, “But system description to voxceleb speaker recognition challenge 2019,” _arXiv preprint arXiv:1910.12592_ , 2019.
  * [9] Y. Wu, C. Guo, H. Gao, X. Hou, and J. Xu, “Vector-based attentive pooling for text-independent speaker verification.” in _INTERSPEECH_ , 2020, pp. 936–940.
  * [10] C. Li, X. Ma, B. Jiang, X. Li, X. Zhang, X. Liu, Y. Cao, A. Kannan, and Z. Zhu, “Deep speaker: an end-to-end neural speaker embedding system,” _arXiv preprint arXiv:1705.02304_ , 2017.
  * [11] H. Bredin, “Tristounet: triplet loss for speaker turn embedding,” in _2017 IEEE international conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 2017, pp. 5430–5434.
  * [12] W. Liu, Y. Wen, Z. Yu, M. Li, B. Raj, and L. Song, “Sphereface: Deep hypersphere embedding for face recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 212–220.
  * [13] Z. Huang, S. Wang, and K. Yu, “Angular softmax for short-duration text-independent speaker verification.” in _Interspeech_ , 2018, pp. 3623–3627.
  * [14] F. Wang, J. Cheng, W. Liu, and H. Liu, “Additive margin softmax for face verification,” _IEEE Signal Processing Letters_ , vol. 25, no. 7, pp. 926–930, 2018.
  * [15] H. Wang, Y. Wang, Z. Zhou, X. Ji, D. Gong, J. Zhou, Z. Li, and W. Liu, “Cosface: Large margin cosine loss for deep face recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 5265–5274.
  * [16] S. Ioffe, “Probabilistic linear discriminant analysis,” in _European Conference on Computer Vision_. Springer, 2006, pp. 531–542.
  * [17] S. Ramoji, P. Krishnan, and S. Ganapathy, “Nplda: A deep neural plda model for speaker verification,” _arXiv preprint arXiv:2002.03562_ , 2020.
  * [18] M. K. Nandwana, J. Van Hout, C. Richey, M. McLaren, M. A. Barrios, and A. Lawson, “The voices from a distance challenge 2019.” in _Interspeech_ , 2019, pp. 2438–2442.
  * [19] Y. Fan, J. Kang, L. Li, K. Li, H. Chen, S. Cheng, P. Zhang, Z. Zhou, Y. Cai, and D. Wang, “Cn-celeb: a challenging chinese speaker recognition dataset,” in _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 7604–7608.
  * [20] L. Li, R. Liu, J. Kang, Y. Fan, H. Cui, Y. Cai, R. Vipperla, T. F. Zheng, and D. Wang, “Cn-celeb: multi-genre speaker recognition,” _Speech Communication_ , 2022.
  * [21] K. A. Lee, A. Larcher, G. Wang, P. Kenny, N. Brümmer, D. Van Leeuwen, H. Aronowitz, M. Kockmann, C. Vaquero, B. Ma _et al._ , “The reddots data collection for speaker recognition,” in _Interspeech 2015_ , 2015.
  * [22] C. Zeng, X. Wang, E. Cooper, X. Miao, and J. Yamagishi, “Attention back-end for automatic speaker verification with multiple enrollment utterances,” in _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2022, pp. 6717–6721.
  * [23] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in neural information processing systems_ , 2017, pp. 5998–6008.
  * [24] C. Raffel and D. P. Ellis, “Feed-forward networks with attention can solve some long-term memory problems,” _arXiv preprint arXiv:1512.08756_ , 2015\.
  * [25] J. Deng, J. Guo, N. Xue, and S. Zafeiriou, “Arcface: Additive angular margin loss for deep face recognition,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 4690–4699.
  * [26] R. Hadsell, S. Chopra, and Y. LeCun, “Dimensionality reduction by learning an invariant mapping,” in _2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06)_ , vol. 2. IEEE, 2006, pp. 1735–1742.
  * [27] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, “Focal loss for dense object detection,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2980–2988.
  * [28] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “mixup: Beyond empirical risk minimization,” _arXiv preprint arXiv:1710.09412_ , 2017.
  * [29] Y. Zhu, T. Ko, and B. Mak, “Mixup learning strategies for text-independent speaker verification.” in _Interspeech_ , 2019, pp. 4345–4349.
  * [30] K. J. Lang, A. H. Waibel, and G. E. Hinton, “A time-delay neural network architecture for isolated word recognition,” _Neural networks_ , vol. 3, no. 1, pp. 23–43, 1990.
  * [31] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [32] B. Desplanques, J. Thienpondt, and K. Demuynck, “Ecapa-tdnn: Emphasized channel attention, propagation and aggregation in tdnn based speaker verification,” _arXiv preprint arXiv:2005.07143_ , 2020.
  * [33] J. S. Chung, J. Huh, S. Mun, M. Lee, H. S. Heo, S. Choe, C. Ham, S. Jung, B.-J. Lee, and I. Han, “In defence of metric learning for speaker recognition,” _arXiv preprint arXiv:2003.11982_ , 2020.
  * [34] H. S. Heo, B.-J. Lee, J. Huh, and J. S. Chung, “Clova baseline system for the voxceleb speaker recognition challenge 2020,” _arXiv preprint arXiv:2009.14153_ , 2020.
  * [35] Y. Zhu, T. Ko, D. Snyder, B. Mak, and D. Povey, “Self-attentive speaker embeddings for text-independent speaker verification.” in _Interspeech_ , vol. 2018, 2018, pp. 3573–3577.
  * [36] Z. Wang, K. Yao, X. Li, and S. Fang, “Multi-resolution multi-head attention in deep speaker embedding,” in _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 6464–6468.
  * [37] J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 7132–7141.
  * [38] B. Harwood, V. Kumar BG, G. Carneiro, I. Reid, and T. Drummond, “Smart mining for deep metric learning,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 2821–2829.
  * [39] K. A. Lee, A. Larcher, C. H. You, B. Ma, and H. Li, “Multi-session plda scoring of i-vector for partially open-set speaker detection,” in _Annual Conference of the International Speech Communication Association (Interspeech)_ , 2013.
  * [40] P. Rajan, A. Afanasyev, V. Hautamäki, and T. Kinnunen, “From single to multiple enrollment i-vectors: Practical plda scoring variants for speaker verification,” _Digital Signal Processing_ , vol. 31, pp. 93–101, 2014.
  * [41] D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlicek, Y. Qian, P. Schwarz _et al._ , “The kaldi speech recognition toolkit,” in _IEEE 2011 workshop on automatic speech recognition and understanding_ , no. CONF. IEEE Signal Processing Society, 2011.
  * [42] S. Ramoji, P. Krishnan, and S. Ganapathy, “Neural plda modeling for end-to-end speaker verification,” _arXiv preprint arXiv:2008.04527_ , 2020.
  * [43] C. Zhang and K. Koishida, “End-to-end text-independent speaker verification with flexibility in utterance duration,” in _2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_. IEEE, 2017, pp. 584–590.
  * [44] Y. Li, F. Gao, Z. Ou, and J. Sun, “Angular softmax loss for end-to-end speaker verification,” in _2018 11th International Symposium on Chinese Spoken Language Processing (ISCSLP)_. IEEE, 2018, pp. 190–194.
  * [45] J. Rohdin, A. Silnova, M. Diez, O. Plchot, P. Matějka, and L. Burget, “End-to-end dnn based speaker recognition inspired by i-vector and plda,” in _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2018, pp. 4874–4878.
  * [46] J.-W. Jung, H.-S. Heo, I.-H. Yang, H.-J. Shim, and H.-J. Yu, “A complete end-to-end speaker verification system using deep neural networks: From raw signals to verification result,” in _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2018, pp. 5349–5353.
  * [47] G. Heigold, I. Moreno, S. Bengio, and N. Shazeer, “End-to-end text-dependent speaker verification,” in _2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2016, pp. 5115–5119.
  * [48] D. Snyder, G. Chen, and D. Povey, “Musan: A music, speech, and noise corpus,” _arXiv preprint arXiv:1510.08484_ , 2015.
  * [49] T. Ko, V. Peddinti, D. Povey, M. L. Seltzer, and S. Khudanpur, “A study on data augmentation of reverberant speech for robust speech recognition,” in _2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2017, pp. 5220–5224.
  * [50] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, “Specaugment: A simple data augmentation method for automatic speech recognition,” _arXiv preprint arXiv:1904.08779_ , 2019.
  * [51] S. Wang, Z. Huang, Y. Qian, and K. Yu, “Discriminative neural embedding learning for short-duration text-independent speaker verification,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 27, no. 11, pp. 1686–1696, 2019.
  * [52] H. Yamamoto, K. A. Lee, K. Okabe, and T. Koshinaka, “Speaker augmentation and bandwidth extension for deep speaker embedding.” in _Interspeech_ , 2019, pp. 406–410.
  * [53] A. Nagrani, J. S. Chung, and A. Zisserman, “Voxceleb: a large-scale speaker identification dataset,” _arXiv preprint arXiv:1706.08612_ , 2017.
  * [54] J. S. Chung, A. Nagrani, and A. Zisserman, “Voxceleb2: Deep speaker recognition,” _arXiv preprint arXiv:1806.05622_ , 2018.
  * [55] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” _arXiv preprint arXiv:1711.05101_ , 2017.
  * [56] Ilya Loshchilov and Frank Hutter, “SGDR: stochastic gradient descent with warm restarts,” in _5th International Conference on Learning Representations, ICLR_ , 2017.
  * [57] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga _et al._ , “Pytorch: An imperative style, high-performance deep learning library,” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [58] F. Tong, M. Zhao, J. Zhou, H. Lu, Z. Li, L. Li, and Q. Hong, “Asv-subtools: Open source toolkit for automatic speaker verification,” in _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2021, pp. 6184–6188.