# DT-SV: A Transformer-based Time-domain Approach for Speaker Verification
††thanks: † Corresponding Author: Jianzong Wang<EMAIL_ADDRESS>
Nan Zhang, Jianzong Wang†, Zhenhou Hong, Chendong Zhao, Xiaoyang Qu, Jing Xiao
Ping An Technology (Shenzhen) Co., Ltd., Shenzhen, China
Emails<EMAIL_ADDRESS><EMAIL_ADDRESS>
{zhenhouhong, cd896614<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Speaker verification (SV) aims to determine whether the speaker’s identity of
a test utterance is the same as the reference speech. In the past few years,
extracting speaker embeddings using deep neural networks for SV systems has
gone mainstream. Recently, different attention mechanisms and Transformer
networks have been explored widely in SV fields. However, utilizing the
original Transformer in SV directly may have frame-level information waste on
output features, which could lead to restrictions on capacity and
discrimination of speaker embeddings. Therefore, we propose an approach to
derive utterance-level speaker embeddings via a Transformer architecture that
uses a novel loss function named diffluence loss to integrate the feature
information of different Transformer layers. Therein, the diffluence loss aims
to aggregate frame-level features into an utterance-level representation, and
it could be integrated into the Transformer expediently. Besides, we also
introduce a learnable mel-fbank energy feature extractor named time-domain
feature extractor that computes the mel-fbank features more precisely and
efficiently than the standard mel-fbank extractor. Combining Diffluence loss
and Time-domain feature extractor, we propose a novel Transformer-based time-
domain SV model (DT-SV) with faster training speed and higher accuracy.
Experiments indicate that our proposed model can achieve better performance in
comparison with other models.

###### Index Terms:

speaker verification, Transformer, raw waveform, learnable fbank extractor,
diffluence loss

##  I Introduction

Speaker verification (SV) is a binary classification task that answers the
question whether an unknown utterance belongs to its claimed identity.
Usually, it can be divided into two categories: text-dependent speaker
verification (TD-SV) and text-independent speaker verification (TI-SV)[1].
Therein, TI-SV has no constraint on text content and speakers can say anything
to the verification system, which brings great convenience to the users, hence
we focus on TI-SV in this work.

Generally speaking, there are two kinds of models in the research of SV: the
statistical model and the neural network model[2]. As an efficient statistical
model, i-vector[3] achieves great success in TI-SV task. It compresses both
speaker and channel information into a fixed-dimensional space called total
variability subspace. Recently, with the increasing of the scale of labeled
data, more and more researchers start to pay attention to the neural network
model. Under the supervised learning framework, the model could automatically
learn the speaker representation through the data-driven training method.

Deep neural networks have been shown to be useful for extracting speaker-
discriminative feature vectors independently from the i-vector framework [3].
With the help of an amount of training data, such approaches have obtained
much better results, particularly under the condition of short-duration
utterances. In 2014, Ehsan et al.[4] introduced the neural network into
speaker verification. At the training step, four fully connected layers are
utilized for speaker classification. Meanwhile, the speaker embedding
(‘d-vector’) is calculated from averaging the last hidden layer’s output over
frames in verification step. Using this pipeline, more well-designed neural
networks such as convolutional neural network(CNNs)[5, 6] and recurrent neural
networks(RNNs)[7] have been proposed for SV task. However, it still needs more
powerful deep neural networks to better extract the speaker feature.

The attention mechanism[8, 9] is a powerful method which offers a way to
obtain an even more discriminative utterance-level feature by explicitly
selecting frame-level representations that better represent speaker
characteristics. Nowadays, Transformer with self-attention mechanism have
become an effective model in a variety of application fields[10, 11, 12],
which mainly focus on the processing of deep neural networks on certain areas
of feature maps or certain temporal slots, including the scenario of SV[2].
The success achieved by this approach in SV with self-attention mechanism
allows models to learn the frame-level features, which are more precise to
represent the speaker characteristics. Yet, there are still two issues. First,
since it can not extract speaker-discriminative features from the raw speech
input, it relies on the mel-fbank or MFCC features. Secondly, the original
Transformer structure may have frame-level information waste on output
features, which could lead to restrictions on capacity and discrimination of
speaker embeddings.

Based on above issues, we propose a Transformer-based Time-domain model for SV
named DT-SV with two major improvements in comparison to the original
Transformer in SV, a novel loss function named diffluence loss and a learnable
mel-fbank feature extractor named time-domain feature extractor. Features
extracted by this extractor will be more appropriate to the neural network
since the time-domain feature extractor is learned from the raw data
distribution. Also, diffluence loss contributes to better obtain the
utterance-level embedding, which summarizes the information from other
embeddings via a self-attention mechanism. Specifically, this loss indicates
the distance among features of the first frame and other frames on each layer,
thus enhancing the speaker-related information in the utterance-level
embedding while weakening the speaker-related information among other frame
embeddings. In particular, we propose two architectures with different numbers
of layers, which are DT-SV-light and DT-SV. Both of them are well-performed in
our experiments while the DT-SV-light could achieve competitive performance
compared to other light models with 10 times smaller GFLOPs.

Our main contributions are as follows:

  * •

An utterance-level speaker representation based on Transformer is presented,
via proposing a novel diffluence loss to aggregate frame-level features on
each layer to an utterance-level speaker representation while weaken the
speaker-related information in the frame-level embeddings.

  * •

A learnable mel-fbank features extractor named time-domain feature extractor
is introduced, which could extract features from speech signal like mel-fbank
more precisely and efficiently than the standard mel-fbank extractor.

  * •

A Transformer-based time-domain speaker verification model is designed to take
advantage of the above two modules, achieving faster training speed and higher
accuracy.

##  II Related Work

###  II-A Speaker Embedding Extractor

During the last few years, how to obtain a single fixed dimension vector to
represent an utterance has drawn most researchers’ attention in this field.
The success of i-vector[3] extracted from GMM-UBM models[13], one of the
earliest speaker embeddings, stimulated the interest among researchers to
search for better speaker embeddings. In the meantime, with the increase of
available data for training, speaker embeddings based on deep learning methods
have gained popularity. The d-vector[4] is one of the earliest DNN-based
embeddings, the core idea of which is to assign the ground-truth speaker
identity of an utterance as the labels of frames belonging to this utterance
in the training phase, which transfers the model training to a classification
problem. Soon after, the x-vectors[14, 15] extracted from the Time Delay
Neural Networks(TDNNs) [16] based system have consistently outperformed
i-vector based systems and therefore been widely used in state-of-the-art
systems. Considering spectrogram representations of speech as images, speaker
embeddings extracted from pretrained residual networks (ResNets) [17, 18] with
a considerable number of images, have performed even better than x-vectors
recently. In order to represent variable utterance lengths, different pooling
methods [19, 15, 20] have been proposed to aggregate the information across
the utterance into a single vector. The s-vectors [21] extracted from speaker
classification architectures based on Transformer’s encoder, attend to all
features that capture speaker characteristics over the entire utterance,
achieving greater performance than x-vectors.

###  II-B Transformer in Speech

The attention-based model has nowadays shown its powerful ability on
representation learning in speech processing fields. There have been a few
studies to explore incorporating attention at the model level and prove such
attention mechanism is effective in aggregating frame-level features. Zhang et
al.[22] proposed the time-frequency attention module for speaker verification
to help the system to focus on important regions of short utterances along the
time and the frequency domain, extracting more effective speaker
discriminative information. In [18], an end-to-end speaker verification system
employing time-frequency and channel attention hierarchically in ResNet
framework is presented for speaker verification as well.

Lately, the architecture named Transformer with stacked self-attention at the
encoder and a multi-head topology have been explored extensively for SV.
Inspired by the next sentence prediction task of BERT[23], an architecture
called TESA [21] based on Transformer’s encoder is proposed as a replacement
for conventional PLDA-based speaker verification to capture the speaker
characteristics better, outperforming the PLDA[24] on the same dataset.
Besides, Zhu et al.[25] proposed a method to create fixed-dimensional
representations for SV with a serialized multi-layer multi-head attention
mechanism containing a stack of self-attention modules, which is designed to
aggregate and propagate attentive statistics from one layer to the next in a
serialized manner. Different from other studies which re-designs the inner
structure of the attention module, we strictly follows original Transformer,
providing the simple but effective modifications.

Figure 1:  A diagram of DT-SV, which is based on Transfomer with two major
improvements in green part. Therein, the Transformer block is the same as the
usual Transformer layer and TDFE represents Time-domain Feature Extractor.

##  III Proposed Method

###  III-A Model Architecture

A common practice in all deep learning methods addressing the SV task is to
perform speaker classification first. Then, utterance-specific fixed-dimension
embeddings are obtained from the speaker classification network by different
pooling methods. These embeddings are immediately fed to a SV system to
validate the identity claimed by the speaker.

In the framework based on the original transformer structure in SV, the input
is mel-fbank features while the output is the speaker classification loss.
Similar to the Transformer applied to natural language processing like
BERT[23] and computer vision like ViT[26], an extra learnable embedding is
added to the sequence as the first frame embedding in our proposed model. The
final hidden state corresponding to this frame serves as the aggregate
utterance-specific embedding for speaker classification, ignoring the
information in other frames. However, we noticed that the speaker-related
information not only exists in the first frame embedding but also is contained
in embeddings of other ignored frames via our experiments. For the better
capacity and discrimination of speaker embeddings, the ignored information
should be further aggregated to form an utterance-level representation. To
address this issue, we proposed a novel SV model based on the original
Transformer with some minor modifications on loss function.

As illustrated in Fig.1, the major differences between our proposed DT-SV and
the original Transformer-based model are time-domain feature extractor(TDFE)
and diffluence loss. Therein, TDFE is a front module to extract features from
the raw waveform input directly, and diffluence loss is the sum of distances
between the utterence-level embedding and each frame-level embedding in every
layer. Particularly, such loss utilizes all features over the entire
utterance, which is more suitable in capturing speaker characteristics on an
utterance. Meanwhile, the join of DTFE makes DT-SV completely an end-to-end SV
model directly from a raw input to classification results, which accelerates
forward propagation speed and model training speed since all operations could
be implemented on GPU because of the raw time domain input.

###  III-B Time-domain Feature Extractor

In general, the standard mel-fbank feature extractor consists of following
modules: pre-emphasis, framing, window, short-time Fourier transform(STFT),
energy spectrum(ES) and mel filter. As is conveyed by Fig.2, which is the
framework of TDFE, all these parts are combined into a black box operation of
a convolutional layer and a fully-connected layer.

Figure 2: The framework of time-domain feature extractor. The processes in the
standard mel-fbank feature extractor, including pre-emphasis, framing, window,
STFT and ES are replaced by a convolutional layer, and the mel filter is
substituted with a fully-connected layer.

As features are extracted via a convolutional layer and a fully-connected
layer in TDFE, we denote the input tensor as $\mathbf{X}\in\mathbb{R}^{A\times
T\times D}$, where $A$ is the amplititude number, $T$ is the time dimension
and $D$ is the dimension of input features. The convolution operation
convolves with a weight matrix $\mathbf{W}_{c}\in\mathbb{R}^{1\times K\times
D\times O}$, where $K$ is the kernel size and $O$ is the dimension of output
features. Likewise, a weight matrix $\mathbf{W}\in\mathbb{R}^{A\times T\times
O\times D}$ is applied in the fully-connected operation. Mathematically, the
process of TDFE could be formulated as:

$\mathbf{E}=\mathcal{F}(\mathbf{X}\ast\mathbf{W}_{c})$ (1)
$\mathbf{Y}=\mathcal{F}(\mathbf{W}\times\mathbf{E})$ (2)

where $\ast$ denotes the convolution operation and $\times$ denotes the matrix
multiply, $\mathcal{F(\cdot)}$ represents the activation function,
$\mathbf{Y}$ is the output feature likes the mel-fbank feature.

It is worth noting that the general steps to obtain the mel-fbank feature from
a speech signal are inspired by the biology of human perception. However,
features extracted by our proposed TDFE are learned from the distribution of
real-world data, which are more suitable for subsequent processing of neural
networks.

###  III-C Diffluence Loss

To address restrictions for capacity and discrimination of speaker embeddings
since the original transformer structure may have frame-level information
waste on output features, the diffluence loss is designed. As we mentioned
before, the diffluence loss is indicated the distance between the utterence-
level embedding and other frame-level embeddings in each layer. Thus, it could
enhance the speaker-related information in the utterence-level embedding.

The loss function for training consists of following two parts, classification
loss $L_{C}$ and diffluence loss $\mathfrak{L}_{D}$, as formulated in Eq.(3).

$L=L_{C}-\mathfrak{L}_{D}$ (3)

Therein, the classification loss is a cross-entropy function in the standard
classification problem. Here, we use the AAM-Softmax loss function [27], as
follows:

$\displaystyle L_{C}$
$\displaystyle=-\frac{1}{N}\sum^{N}_{n=1}\log{\frac{\phi_{n}}{\phi_{n}+\psi_{n}}}$
(4) $\displaystyle\phi_{n}$ $\displaystyle=e^{\tau\cos(\theta_{l_{n},n}+m)}$
$\displaystyle\psi_{n}$ $\displaystyle=\sum^{J}_{j=1,j\neq
l_{n}}e^{\tau\cos(\theta_{j,n})}$

where $\tau$ is a scaling factor for preventing gradients too small during the
training process, $m$ is the hyper-parameters.

For the diffluence loss $\mathfrak{L}_{D}$, the Kullback-Leibler divergence,
which is widely applied in the field of machine learning and deep learning, is
utilized as a representation of distance:

$\mathfrak{L}_{D}=\frac{1}{LT}\sum^{L}_{l=1}\sum^{T}_{i=1}KL(\mathbf{v}_{l0}||\mathbf{v}_{li})$
(5)

where the $\mathbf{v}_{li}$ represents the $i$-th output embedding in the
$l$-th layer, KL$(\cdot)$ denotes the Kullback-Leibler divergence. Specially,
when $l=1$, the $KL(\mathbf{v}_{l0}||\mathbf{v}_{li})$ is called attention
distance.

Besides, the cosine function could also serve as $\mathfrak{L}_{D}$, which is
an another common measurement of the distance between distributions. The
relevant results of comparative experiment can be found in Section IV.

##  IV EXPERIMENT AND ANALYSIS

In this section, we firstly describe some datasets utilized in following
experiments, and then detailedly analyze the experimental results accordingly.

###  IV-A Dataset

CN-CELEB: CN-Celeb [28] is a large-scale speaker recognition dataset collected
“in the wild”. Specifically, the dataset contains more than 130000 utterances
from 1000 Chinese celebrities, and covers 11 different genres in real world,
including entertainment, interview, singing, play, movie, vlog, live
broadcast, speech, drama, recitation and advertisement. The speech of a
particular speaker may be in more than 5 genres. The diversity in genres makes
CN-Celeb more representative for the true scenarios in unconstrained
conditions, but also more challenging.

VoxCeleb2: VoxCeleb is a large-scale speaker identification dataset consisting
of short clips of human speech. The entire dataset involves two parts:
VoxCeleb1 [29] and VoxCeleb2 [19]. VoxCeleb2 dataset consists over 1 million
utterances for over 6,000 celebrities, extracted from videos uploaded to
YouTube. VoxCeleb1 dataset consists of about 150,000 utterances from 1251
different celebrities. The utterances are collected from YouTube videos in
which the celebrities belong to different races and have a wide range of
accents. In our work, the proposed DT-SV is trained on the VoxCeleb2 while
evaluated on the VoxCeleb1, including VoxCeleb1-E and VoxCeleb1-H test sets.

###  IV-B Implementation details

In order to compare experimental results equitably, all settings in our
experiments are consistent with those in baselines[30], except for the loss
function. Thus, we utilize similar network structure, data processing,
training, and testing strategies in our experiments.

Network structure: As we mentioned in Section I, two different architectures
are proposed in our work. To be specific, DT-SV-light is “lighter” on three
aspects including the number of layers, dimensions and attention heads than
DT-SV, as presented in Table I. Consequently, the number of parameters of DT-
SV-light is much less than DT-SV, which makes DT-SV-light much easier to apply
in more scenarios like mobile devices.

TABLE I: Model hyper-parameters of DT-SV-light and DT-SV. Model | DT-SV-light | DT-SV
---|---|---
Params(M) | 1.0 | 21.5
Layers | 4 | 6
Dims | 128 | 128/256/512
Attention Heads | 4 | 8

Metric: Following the general practice, equal error rate (EER) and minimum
detection cost function (minDCF) are chosen to evaluate the performance. Some
parameters maintain the same as those in [29], where the target probability
$P_{tar}$ is 0.01, $C_{fa}$ and $C_{fr}$ has the same weight of 1.0.

Training: In our design for ablation experiments, four different models with
different layers or dimensions are trained to assess the effects of
hyperparameters. All training settings are presented in Table II, one of these
models has 4 layers and 128 dimensions, while the other three models all have
6 layers with exponentially increasing dimensions(128, 256 and 512
respectively). For the two models with 128 dimensions, the learning rate decay
and the weight decay are both set as 0.97 and $10^{-5}$ respectively while the
learning rate are 0.001 and 0.0005. For the other two models with 256
dimensions and 512 dimensions, the learning rate, the learning rate decay and
the weight decay are basically the same with 0.001, 0.99 and $5\times 10^{-5}$
respectively. To make a relatively fair comparison, the number of epoches is
progressive increase because of different model sizes. Besides, for all
models, the optimizer is Adam [31] and the batch size is set to 150.

TABLE II: Training setting of different DT-SV models. “L” is layer, “D” is dimension, “LR” is learning rate, “LRD” is learning rate decay, “WD” is weight decay. #L | D | LR | LRD | WD | epoch
---|---|---|---|---|---
4 | 128 | 0.001 | 0.97 | $10^{-5}$ | 20
6 | 128 | 0.0005 | 0.97 | $10^{-5}$ | 24
6 | 256 | 0.001 | 0.99 | $5\times 10^{-5}$ | 26
6 | 512 | 0.001 | 0.99 | $5\times 10^{-5}$ | 30

Different training techniques: Table III shows the progressive improvements of
DT-SV while employing different training techniques. To verify the
effectiveness of common techniques, diffluence loss is not added for the time
being. Starting from DT-SV-light, which is regard as baseline, with 3.19% EER,
the multiple training techniques are applied to improve the performence. The
first one is SpecAugment [32] and Data Augmentation that could make the
distribution of the input raw wave data more diverse. Thus, it lowers the EER
to 2.84%, or reduces 0.35% EER compared to the baseline. Then, with the help
of Relative Position Embedding [33] that could enhance the information of
relative position, the EER of the model decreases by another 0.5%. In general,
making a model bigger and deeper is always an effective method for acquiring
better representations. As shown in Table III, the model with higher multi-
layer perceptron(MLP) dimensions and more Transformer blocks could reduce the
EER greatly. Consequently, all these four training techniques are employed in
the following experiments.

TABLE III: Ablation path of our proposed DT-SV-light baseline. All the models are carried out on the same GPU devices for fair comparison. The performance can be boosted from 3.19 to 2.07 (-1.12) using the proposed techniques.  Training Techniques | EER(%)
---|---
Basline(DT-SV-light with AAM-Softmax) | 3.19
\+ SpecAugment & Data Augmentation | 2.84 (-0.35)
\+ Relative Position Embedding | 2.69 (-0.50)
\+ Higher MLP dimension ($256\rightarrow 512$) | 2.13 (-1.07)
\+ More Transformer Blocks ($4\rightarrow 6$) | 2.07 (-1.12)
TABLE IV: Comparison of different models on CN-CELEB. Front-end | Input | Params(M) | GFLOPs | EER(%) | minDCF
---|---|---|---|---|---
x-vector [28] | MFCC | 4.2 | 2.164 | 14.78 | 16.51
r-vector [34] | Fbank | - | - | 13.43 | N/R
Fast-ResNet34 [35] | Raw waveform | 1.9 | 5.874 | 13.12 | 0.611
DT-SV-light | Raw waveform | 1.0 | 0.390 | 12.54 | 0.562
TABLE V: Results for speaker verification on the VoxCeleb1, VoxCeleb1-E and VoxCeleb-H test sets. N/R : Not report results. AP: Angular Prototypical. DLoss represents the diffluence loss. Front-end Model | Input | Loss | VoxCeleb1 | VoxCeleb1-E | VoxCeleb1-H
---|---|---|---|---|---
EER(%) | MinDCF | EER(%) | MinDCF | EER(%) | MinDCF
Thin-ResNet34 [17] | Magnitude | Softmax | 2.87 | N/ R | 2.95 | N/R | 4.93 | N/R
RawNet2 [36] | Raw waveform | Softmax | 2.48 | N/R | 2.57 | N/R | 4.89 | N/R
Wav2spk [37] | Raw waveform | AM-softmax | 1.95 | 0.203 | N/R | N/R | N/R | N/R
raw-x-vector [38] | Raw waveform | AM-softmax | N/R | N/R | 2.64 | N/R | 4.34 | N/R
ICspk [39] | Raw waveform | AP | 1.92 | 0.137 | 1.94 | 0.141 | 3.78 | 0.237
TESA [21] | MFCC | Softmax | 2.37 | 0.2 | N/R | N/R | N/R | N/R
serialized attention [25] | MFCC | cross entropy | N/R | N/R | 2.36 | 0.242 | 3.99 | 0.375
DT-SV | Raw waveform | AAM-softmax | 2.07 | 0.154 | 2.11 | 0.161 | 4.31 | 0.245
DT-SV | Raw waveform | AAM-softmax+DLoss(Cosine) | 1.95 | 0.132 | 1.94 | 0.138 | 3.96 | 0.247
DT-SV | Raw waveform | AAM-softmax+DLoss(KL) | 1.92 | 0.130 | 1.91 | 0.136 | 3.72 | 0.225

###  IV-C Results and Analysis

Attention distance: Here the impact of layers on performance of DT-SV is
explored. The Fig. 3 displays the distance between attention heads at
different layers. Therein, dots with the same color represent the distance
between the first attention head and the other attention heads in the specific
layer. The distance of attention head is calculated by averaging distances of
the first frame embedding and the other frame embeddings, as Eqn. (5) in Sec.
III-C when $l=1$.

As illustrated in Fig. 3(a), when the layer of Transformer is 6, the attention
distance of the first five layers is relatively scattered. Following the
second layer that is the most scattered, the degree of scattering from the
third layer to the fifth layer is similar to the first layer. The attention
distances are the most clustered in the sixth layer. Moreover, as shown in
Fig. 3(b) in which the number of layers of Transformer is 10, the overall
trend is basically similar to Fig. 3(a). Specifically, the attention distances
in the first five layers are relatively scattered, and the distance is
gradually gathering from the sixth layer.

From the whole trend of attention distance in Fig. 3, we notice that each
layer is broadly more clustered than its former layer from the second layer,
which indicates that diffluence loss could allow DT-SV to integrate
information across the entire utterance to the lower layers. And the distance
is stable from the sixth layer, that is to say, the differences among the
frame-level embeddings in these layers is slight. The results inspire us that
the number of layers of the Transformer in the DT-SV could be set as 4 to hold
fewer parameters and achieve decent performance. Alternatively, such number
could also be set as 6 to obtain a more powerful model.

(a) Attention distance when the layer of Transformer is 6.

(b) Attention distance when the layer of Transformer is 10.

Figure 3: The attention distance in each layer.

Comprehensive comparison: The metrics of different models on the CN-CELEB are
listed in Table IV. Notice that we select the DT-SV-light to compared with the
other three models, wherein the inputs of x-vector and r-vector are the MFCC
or Fbank features extracting from the waveform by the standard preprocessing
in Fig.2 while the input of the Fast-ResNet34 is the raw waveform. As
presented in Table IV, the DT-SV-light achieves the best performance with
12.54% EER, which proves that the TDFE could acquire much better features than
the standard preprocessing. Moreover, the fact that the DT-SV-light could
achieve decent performance with fewer parameters suggests its enormous
potential in industrial applications.

To futher demonstrate the effectiveness of our proposed model, we conduct more
comprehensive comparison experiments, as the Table V shows, in which the other
results are cited from the literatures. Therein, the input of these models are
raw waveforms except Thin-ResNet34, TESA and serialized attention model,
though TESA and serialized attention are also based on Transformer.

Obviously, the proposed DT-SV could achieve lower EER than most comparative
models on all datasets even without the diffluence loss. Furthermore, the
better performance of the DT-SV after combining with the diffluence loss,
especially that calculated by KL divergence, indicates that the diffluence
loss could enhance the related-speaker information effectively. Besides, in
comparison to TESA and serialized attention that based on Transformer with
well-designed inner structures, the proposed DT-SV based on original
Transformer with some minor modifications, such as diffluence loss, is quite
simple yet extremely effective. More significantly, such loss could be easily
incorporated into any existing Transformer-based achitectures.

##  V Conclusion

In this work, we propose an improved Transformer-based time-domain approach
for speaker verification named DT-SV, which leverages time-domain feature
extractor and diffluence loss to capture speaker characteristics in an
utterance having greater capacity and discrimination. Specifically, the time-
domain feature extractor could extract features from the raw speech input more
precisely and efficiently than the standard mel-fbank extractor. Moreover, a
novel loss function named diffluence loss could enhance speaker-related
information in the utterance-level embedding while weaken such information in
other frame-level embeddings. As our experiments demonstrated, the proposed
DT-SV could achieve competitive results on the datasets employed. Besides, our
proposed architecture explores the Transformer mechanism for the speaker
verification task and hopefully gives an inspiration for other related tasks.
Last but not least, the diffluence loss could be flexibly embedded into other
analogous structures.

## Acknowledgment

This paper is supported by the Key Research and Development Program of
Guangdong Province under grant No.2021B0101400003. Corresponding author is
Jianzong Wang from Ping An Technology (Shenzhen) Co., Ltd (jzwang@188.com).

## References

  * [1] X. Qu, J. Wang, and J. Xiao, “Evolutionary algorithm enhanced neural architecture search for text-independent speaker verification,” in _Proc. Interspeech_ , 2020, pp. 961–965.
  * [2] V. Mingote, A. Miguel, A. Ortega, and E. Lleida, “Memory layers with multi-head attention mechanisms for text-dependent speaker verification,” in _2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2021, pp. 6154–6158.
  * [3] N. Dehak, P. J. Kenny, R. Dehak, P. Dumouchel, and P. Ouellet, “Front-end factor analysis for speaker verification,” _IEEE Transactions on Audio, Speech, and Language Processing_ , vol. 19, no. 4, pp. 788–798, 2010.
  * [4] E. Variani, X. Lei, E. McDermott, I. L. Moreno, and J. Gonzalez-Dominguez, “Deep neural networks for small footprint text-dependent speaker verification,” in _2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2014, pp. 4052–4056.
  * [5] J. Balian, R. Tavarone, M. Poumeyrol, and A. Coucke, “Small footprint text-independent speaker verification for embedded systems,” in _2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2021, pp. 6179–6183.
  * [6] S. Han, J. Byun, and J. W. Shin, “Time-domain speaker verification using temporal convolutional networks,” in _2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2021, pp. 6688–6692.
  * [7] H. Tak, J. Patino, M. Todisco, A. Nautsch, N. Evans, and A. Larcher, “End-to-end anti-spoofing with RawNet2,” in _2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2021, pp. 6369–6373.
  * [8] D. Bahdanau, K. H. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” in _3rd International Conference on Learning Representations, ICLR 2015_ , 2015.
  * [9] N. Zhang, J. Wang, W. Wei, X. Qu, N. Cheng, and J. Xiao, “Cacnet: Cube attentional cnn for automatic speech recognition,” in _2021 International Joint Conference on Neural Networks (IJCNN)_ , 2021, pp. 1–7.
  * [10] M. India, P. Safari, and J. Hernando, “Double multi-head attention for speaker verification,” in _2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2021, pp. 6144–6148.
  * [11] H. Wu, X. Li, A. T. Liu, Z. Wu, H. Meng, and H.-y. Lee, “Adversarial defense for automatic speaker verification by cascaded self-supervised learning models,” in _2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2021, pp. 6718–6722.
  * [12] Z. Hong, J. Wang, X. Qu, J. Liu, C. Zhao, and J. Xiao, “Federated Learning with Dynamic Transformer for Text to Speech,” in _Proc. Interspeech_ , 2021, pp. 3590–3594.
  * [13] D. A. Reynolds, T. F. Quatieri, and R. B. Dunn, “Speaker verification using adapted gaussian mixture models,” _Digital Signal Processing_ , vol. 10, no. 1-3, pp. 19–41, 2000.
  * [14] D. Snyder, D. Garcia-Romero, D. Povey, and S. Khudanpur, “Deep neural network embeddings for text-independent speaker verification,” in _2017 Conference of the International Speech Communication Association (Interspeech)_ , 2017, pp. 999–1003.
  * [15] D. Snyder, D. Garcia-Romero, G. Sell, D. Povey, and S. Khudanpur, “X-vectors: Robust dnn embeddings for speaker recognition,” in _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2018, pp. 5329–5333.
  * [16] A. Waibel, T. Hanazawa, G. Hinton, K. Shikano, and K. J. Lang, “Phoneme recognition using time-delay neural networks,” _IEEE Transactions on Acoustics, Apeech, and Signal Processing_ , vol. 37, no. 3, pp. 328–339, 1989\.
  * [17] A. Nagrani, J. S. Chung, W. Xie, and A. Zisserman, “Voxceleb: Large-scale speaker verification in the wild,” _Computer Speech & Language_, vol. 60, p. 101027, 2020.
  * [18] C. Wang, J. Yi, J. Tao, Y. Bai, and Z. Tian, “Hierarchically attending time-frequency and channel features for improving speaker verification,” in _2021 12th International Symposium on Chinese Spoken Language Processing (ISCSLP)_. IEEE, 2021, pp. 1–5.
  * [19] J. S. Chung, A. Nagrani, and A. Zisserman, “Voxceleb2: Deep speaker recognition,” in _2018 Conference of the International Speech Communication Association (Interspeech)_ , 2018, pp. 1086–1090.
  * [20] K. Okabe, T. Koshinaka, and K. Shinoda, “Attentive statistics pooling for deep speaker embedding,” in _2018 Conference of the International Speech Communication Association (Interspeech)_ , 2018, pp. 2252–2256.
  * [21] N. J. M. S. Mary, S. Umesh, and S. V. Katta, “S-vectors and TESA: Speaker embeddings and a speaker authenticator based on transformer encoder,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 30, pp. 404–413, 2022.
  * [22] Y. Zhang, H. Yu, and Z. Ma, “Speaker verification system based on deformable cnn and time-frequency attention,” in _2020 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)_. IEEE, 2020, pp. 1689–1692.
  * [23] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT_ , 2019.
  * [24] S. Ioffe, “Probabilistic linear discriminant analysis,” in _European Conference on Computer Vision_. Springer, 2006, pp. 531–542.
  * [25] H. Zhu, K. A. Lee, and H. Li, “Serialized multi-layer multi-head attention for neural speaker embedding,” in _2021 Conference of the International Speech Communication Association (Interspeech)_ , 2021, pp. 106–110.
  * [26] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” in _9th International Conference on Learning Representations, ICLR 2020_ , 2020.
  * [27] J. Deng, J. Guo, N. Xue, and S. Zafeiriou, “Arcface: Additive angular margin loss for deep face recognition,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 4690–4699.
  * [28] Y. Fan, J. Kang, L. Li, K. Li, H. Chen, S. Cheng, P. Zhang, Z. Zhou, Y. Cai, and D. Wang, “CN-CELEB: a challenging chinese speaker recognition dataset,” in _2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 7604–7608.
  * [29] A. Nagrani, J. S. Chung, and A. Zisserman, “VoxCeleb: A large-scale speaker identification dataset,” in _2017 Conference of the International Speech Communication Association (Interspeech)_ , 2017, pp. 2616–2620.
  * [30] H. Zeinali, L. Burget, J. Rohdin, T. Stafylakis, and J. H. Cernocky, “How to improve your speaker embeddings extractor in generic toolkits,” in _2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2019, pp. 6141–6145.
  * [31] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [32] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, “Specaugment: A simple data augmentation method for automatic speech recognition,” in _2019 Conference of the International Speech Communication Association (Interspeech)_ , 2019, pp. 2613–2617.
  * [33] Z. Dai, Z. Yang, Y. Yang, J. G. Carbonell, Q. Le, and R. Salakhutdinov, “Transformer-XL: Attentive language models beyond a fixed-length context,” in _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , 2019, pp. 2978–2988.
  * [34] Z. Chen, S. Wang, and Y. Qian, “Self-supervised learning based domain adaptation for robust speaker verification,” in _I2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2021, pp. 5834–5838.
  * [35] J. S. Chung, J. Huh, S. Mun, M. Lee, H. S. Heo, S. Choe, C. Ham, S. Jung, B.-J. Lee, and I. Han, “In defence of metric learning for speaker recognition,” in _2020 Conference of the International Speech Communication Association (Interspeech)_ , 2020, pp. 2977–2981.
  * [36] J.-w. Jung, S.-b. Kim, H.-j. Shim, J.-h. Kim, and H.-J. Yu, “Improved RawNet with feature map scaling for text-independent speaker verification using raw waveforms,” in _2020 Conference of the International Speech Communication Association (Interspeech)_ , 2020, pp. 1496–1500.
  * [37] W. Lin and M.-W. Mak, “Wav2Spk: A simple DNN architecture for learning speaker embeddings from waveforms,” in _2020 Conference of the International Speech Communication Association (Interspeech)_ , 2020, pp. 3211–3215.
  * [38] G. Zhu, F. Jiang, and Z. Duan, “Raw-x-vector: Multi-scale time domain speaker embedding network,” _arXiv preprint arXiv:2010.12951_ , 2020.
  * [39] J. Peng, X. Qu, J. Wang, R. Gu, and J. Xiao, “ICSpk: Interpretable complex speaker embedding extractor from raw waveform,” in _2021 Conference of the International Speech Communication Association (Interspeech)_ , 2021, pp. 511–515.