# Improving the Adversarial Robustness for Speaker Verification by Self-
Supervised Learning

Haibin Wu*, Xu Li*, Andy T. Liu, Zhiyong Wu, Helen Meng Hung-yi Lee * Equal
contribution.Haibin Wu, Andy T. Liu and Hung-yi Lee are with the Graduate
Institute of Communication Engineering, National Taiwan University, Taiwan
(e-mail: {f07921092, f07942089, hungyilee}@ntu.edu.tw).Xu Li and Helen Meng
are with the Department of Systems Engineering and Engineering Management, The
Chinese University of Hong Kong, Hong Kong (email:
{xuli,hmmeng}@se.cuhk.edu.hk).Zhiyong Wu is with the Shenzhen International
Graduate School, Tsinghua University (email: zywu@se.cuhk.edu.hk).

###### Abstract

Previous works have shown that automatic speaker verification (ASV) is
seriously vulnerable to malicious spoofing attacks, such as replay, synthetic
speech, and recently emerged adversarial attacks. Great efforts have been
dedicated to defending ASV against replay and synthetic speech; however, only
a few approaches have been explored to deal with adversarial attacks. All the
existing approaches to tackle adversarial attacks for ASV require the
knowledge for adversarial samples generation, but it is impractical for
defenders to know the exact attack algorithms that are applied by the in-the-
wild attackers. This work is among the first to perform adversarial defense
for ASV without knowing the specific attack algorithms. Inspired by self-
supervised learning models (SSLMs) that possess the merits of alleviating the
superficial noise in the inputs and reconstructing clean samples from the
interrupted ones, this work regards adversarial perturbations as one kind of
noise and conducts adversarial defense for ASV by SSLMs. Specifically, we
propose to perform adversarial defense from two perspectives: 1) adversarial
perturbation purification and 2) adversarial perturbation detection. The
purification module aims to alleviate the adversarial perturbations in the
samples and pull the contaminated adversarial inputs back towards the decision
boundary. Experimental results show that our proposed purification module
effectively counters adversarial attacks and outperforms traditional filters
from both alleviating the adversarial noise and maintaining the performance of
genuine samples. The detection module aims to detect adversarial samples from
genuine ones based on the statistical properties of ASV scores derived by ASV
integrating with different number of SSLMs. Experimental results show that our
detection module effectively shields the ASV by detecting adversarial samples
with an accuracy of around 80%. Moreover, since there is no common metric for
evaluating the adversarial defense performance for ASV, this work also
formalizes evaluation metrics for adversarial defense considering both
purification and detection based approaches into account. We sincerely
encourage future works to benchmark their approaches based on the proposed
evaluation framework.

###### Index Terms:

automatic speaker verification, adversarial attacks, adversarial defense,
self-supervised learning.

##  I Introduction

Automatic speaker verification (ASV) deals with a task of certifying whether a
certain piece of utterance belongs to a given speaker. It has been intensively
studied for decades, and a large variety of cutting-edge ASV models have been
proposed [1]. These ASV models can be classified into three most
representative frameworks: 1) i-vector based speaker embedding systems [2, 3,
4, 5, 6], 2) deep neural network (DNN) based speaker embedding systems [7, 8,
9, 10], and 3) end-to-end speaker verification systems [11, 12, 13]. ASV is
one of the most essential technology for biometric identification and is
widely used in real-world applications, including smartphones, banking
systems, IoT devices, etc.

Given that such applications are security-critical, the robustness of ASV
systems is of high relevance. However, recent works have shown that ASV
systems are vulnerable to malicious attacks, such as impersonation [14, 15],
replay [15, 16], synthetic speech [17, 18, 19] and very lately emerged
adversarial attacks [20, 21]. These attacks can pose severe threats to the
state-of-the-art (SOTA) ASV systems. To tackle this issue, ASVspoof challenge
series [22, 23, 24] have been held to develop strong countermeasures [25, 26,
27] mainly against replay and synthetic speech. Due to the very late emergence
of adversarial attacks, the defense approaches against them are rare and
limited, which is the focus of this paper.

The concept of adversarial attack was first proposed by [28]. They slightly
modified the genuine clean samples by deliberately crafted tiny perturbations,
and image classification models with super-human performance would
embarrassingly predict wrong results. This shortcoming of image classification
models is particularly surprising because the modiﬁcations are often
imperceptible, or barely perceptible, to humans.

Not only computer vision models but also audio processing models are subject
to adversarial attacks. Carlini and Wagner [29] successfully conduct
adversarial attack against the DeepSpeech [30], a SOTA automatic speech
recognition (ASR) neural network model. Given any piece of audio, whether
speech, silence, or music, the authors can perform adversarial attack to craft
another piece of audio, which is over 99% similar to the original one, but can
be predicted by the ASR model as any transcription predefined by the authors
with 100% success rate. [31, 32, 33, 34, 35, 36, 37, 38] also conduct
adversarial attack towards ASR from different aspects. Existing works also
demonstrated that other speech processing tasks, including anti-spoofing
models for ASV [39, 40, 41, 42], music classification [43], sound event
classification [44] and source separation [45], were vulnerable to adversarial
attacks.

Also, previous works [21, 46, 47, 48, 49, 50, 51, 52, 53, 20] showed that
adversarial attacks can cause catastrophic performance drop of the SOTA ASV
systems, which undermines the usage of ASV systems in the real-world. [21] is
among the first ones to investigate the vulnerability of ASV models to
adversarial attacks. They demonstrate the end-to-end text-dependent ASV models
can be fooled by malicious attackers both in cross-dataset and cross-feature
scenarios. Some early works attack ASV systems with few speakers [51, 50].
What’s more, SOTA ASV models, including i-vector and x-vector systems, are
also under the radar of adversarial attacks. [53] benchmarks the adversarial
robustness of three x-vector systems, which are the SOTA models in recent ASV
evaluations, and shows they are subject to adversarial attacks. Li et al. [46]
illustrates the GMM i-vector system can be hallucinated by adversarial attacks
and the adversarial samples crafted from i-vector systems attain the
transferability to x-vector systems. In order to attack real-world
applications possibly, attackers also pour great efforts to make adversarial
attacks over-the-air, inaudible and universal. Li et al. [49] first
demonstrate that the x-vector system, one of the SOTA multi-class speaker
recognition systems, are vulnerable to over-the-air adversarial attacks in the
physical world, rather than just feeding the adversarial examples directly
into the ASV systems. [48] explores the existence of master voices, i.e.,
adversarial utterances which can mimic a large number of users by an
adversarial optimization method. Wang et al. [52] introduce the psychoacoustic
principle [54] of frequency masking to generate more inaudible adversarial
perturbations. The experimental results on Aishell-1 dataset [55] show that
their proposed adversarial attack method can manipulate the x-vector based
speaker recognition system identify a piece of audio, whether speech or music,
as any target (i.e., adversary-desired) speaker. Xie et al. [47] proposes the
first audio-agnostic, real-time, over-the-air adversarial perturbation against
the x-vector based ASV systems. With the pervasiveness of ASV systems in
safety-critical environment, mitigating the vulnerability of ASV models to
adversarial attacks is of high priority.

However, how to conduct adversarial defense effectively for ASV systems still
remains an open question. Previous attempts [56, 57, 58] require knowledge
about attack methods’ details during adversarial sample generation. It is
impractical that the defenders know exactly the adversarial attack algorithms
which the in-the-wild attackers will choose in advance.

We make the first attempt to conduct adversarial defense for ASV by self-
supervised learning without knowing the adversarial sample generation process.
Self-supervised learning has aroused keen attention recently. The self-
supervised learning based models possess the merits of alleviating the
superficial noise in the inputs and extracting the pivotal information from
the contaminated inputs after pretraining. To some extent, adversarial noise
is also a kind of noise, so we propose a defense framework based on self-
supervised learning based model. The self-supervised learning based model
adopted to purify the adversarial perturbation and reform the adversarial
samples in this work is called self-supervised learning based reformer (SSLR).
The defense framework consists of an adversarial perturbation purification
module which attempts to pull the contaminated adversarial inputs back towards
the decision boundary based on the purification ability of the SSLR, and an
adversarial perturbation detection module that aims at directly detecting the
adversarial samples.

Early works about adversarial attack [21, 51, 50, 47, 49] and defense [56, 58]
focus on speaker classiﬁcation tasks with few speakers. In this work, we focus
on the speaker verification task, which is an open-set problem and more
challenging than close-set classification tasks. We adopt Voxceleb1 [59] as
our benchmark dataset, which includes 148,642 short clips of human speech for
1,251 speakers. Our experimental results for voxceleb1 demonstrate the
proposed framework performs adversarial defense effectively for SOTA ASV
models from both the purification and detection perspectives.

Our contributions are as follows:

  * •

This paper is among the first ones to perform adversarial defense for ASV
models without knowing the adversarial sample generation process. As the
beginning work of this direction, there is no baseline work for reference. So
we firstly utilize traditional filters as countermeasures for adversarial
attacks and employ them as our baseline.

  * •

The proposed adversarial defense framework based on self-supervised learning
takes adversarial perturbation purification and detection into account. It
effectively improves the robustness of SOTA ASV models against adversarial
attacks. Specifically, the adversarial perturbation purification module
effectively counters adversarial attacks and outperforms traditional filters
from both alleviating the adversarial noise and maintaining the performance of
genuine samples. Also, the adversarial perturbation detection module
effectively shields the ASV by detecting the adversarial samples with an
accuracy of around 80%.

  * •

We also firstly formalize the evaluation metrics for adversarial defense on
ASV, as shown in Section VI-C, and sincerely encourage future works to
benchmark their approaches base on the evaluation metrics.

##  II Terminology

In this section, we summarize the terminologies adopted in this work.

  * •

Non-target trial: The enrollment and testing utterances belong to two
different speakers.

  * •

Target trial: The enrollment and testing utterances belong to the same
speaker.

  * •

Target system: The system that attackers attempt to attack.

  * •

Substitute system: The system that is used for approximating gradients for
adversarial sample generation when the target system is unavailable to
attackers.

  * •

False acceptance rate (FAR): The percentage of non-target trials in which
unauthorised persons are incorrectly accepted.

  * •

False rejection rate (FRR): The percentage of target trials where authorised
persons are incorrectly rejected.

  * •

Equal error rate (EER): The error rate under the operation point where FAR
equals to FRR.

##  III Related work

Existing approaches to protect ASV systems against adversarial attacks can be
divided into two categories: purification methods and detection methods.

Purification methods regard the adversarial perturbations within modified
samples as a particular kind of noise. They tackle this problem by either
developing some filters [60] in front of ASV systems to purify the samples or
leveraging adversarial training [61, 28] to make ASV systems themselves robust
against this kind of noise. After applying purification methods, adversarial
samples become less offensive, and thus given the adversarial samples, ASV can
give the same predictions as the corresponding genuine ones. Wang et al. [56]
and Wu et al. [42] harness the idea of adversarial training to alleviate the
vulnerability of ASV models and anti-spooﬁng models for ASV, respectively.
[56] mixes the adversarial objective function with the classification
objective as regularization while [42] injects adversarial samples into
training set for data augmentation. Adversarial training requires knowledge
about the attack methods for adversarial sample generation and is sensitive to
specific attack algorithms. However, it does not make sense that the system
designers know the exact attack algorithms to be adopted by attackers. Zhang
et al. [58] propose to train an independent DNN filter and apply it before the
ASV system to mitigate the adversarial perturbations and purify the
adversarial samples. However, this method [58] requires adversarial samples to
train the filtering module, so the defense method is specific and even over-
fitting to the attack algorithms for generating adversarial samples. [42]
equips the anti-spoofing models for ASV with hand-crafted filters, including
Gaussian filter, median filter, and mean filter, to counter the adversarial
attacks. Hand-crafted filters haven’t been used for adversarial defense on
ASV. We adopt the above filters to protect ASV systems and compare their
performance with our proposed method.

While detection methods aim at distinguishing adversarial samples from genuine
ones, and the samples being classified as adversarial ones will be detected
and discarded directly and ASV systems refuse to make prediction for them. A
detection network is introduced by Li et al. [57] proposes a detection network
for ASV to distinguish the adversarial samples from genuine samples. However,
training such a detection network also needs to know the specific attack
algorithms to generate adversarial samples.

To our best knowledge, only a few approaches [56, 57, 58] were proposed to
protect ASV systems against adversarial attacks, and how to effectively purify
and mitigate the adversarial perturbations for ASV systems is still an open
question. In this work, we propose to defend ASV systems against adversarial
attacks by the SSLR models. Specifically, the proposed defense framework is
composed of an adversarial purification module and an adversarial perturbation
detection module, which mitigates the adversarial noise or directly detects
the adversarial samples, respectively. What’s more, compared with previous
works [56, 57, 58], our method doesn’t require the knowledge of adversarial
samples generation process in advance. Experimental results show that our
proposed method can effectively protect ASV systems against adversarial
attacks from both purification and detection perspectives.

Our previous work [62] adopts the self-supervised learning based model as a
deep filter to alleviate adversarial attack for the r-vector system. In
contrast to [62], we further investigate the potential of self-supervised
learning based model against adversarial attack for ASV from different
perspectives:

  * •

[62] merely considers self-supervised learning models for purification use,
yet this work harnesses the self-supervised learning model for both
purification and detection use.

  * •

In contrast to [62], which only conducts adversarial defense for the r-vector
system, this work performs experiments on both r-vector and x-vector systems
to illustrate our proposed defense method is general.

  * •

Previous work [62] only implements one masking strategy to pre-train the self-
supervised learning model. In this work, we adopt several masking strategies
to train various self-supervised learning models and compare the performance
of different masking strategies for adversarial defense.

  * •

In comparison to [62], which simply and crudely uses EER to evaluate the
defense performance, this work systematically formalizes the evaluation
metrics for adversarial defense on ASV (see Section VI-C).

##  IV Background

Figure 1: (a) Illustration of the SSLR models training. (b) Adversarial
defense by cascaded SSLR models. (c) Automatic speaker verification.

###  IV-A Automatic speaker verification

Automatic speaker verification (ASV) aims at confirming a speaker identity
claim given a segment of speech. The speaker identity claim is usually
presented by an enrollment utterance to the ASV system, whereas the segment of
speech serves as a testing utterance in verification scenarios. An ASV system
usually consists of three parts: feature engineering, speaker embedding
extraction, and similarity scoring. Feature engineering converts an input
waveform into frame-level acoustic features, such as Mel-frequency cepstral
coefficients (MFCCs), filter-banks, and spectrograms. These acoustic features
contain variation trends in frequency domain that can reflect speaker cues.
Speaker embedding extraction aims at projecting acoustic features of an
utterance into a speaker-related embedding space. It is conducted by a variety
of speaker extraction front-ends, such as i-vector system [2, 3, 4, 5], time
delay neural network (TDNN) x-vector system [9] and resnet r-vector system
[63]. Similarity scoring defines a distance metric in the speaker embedding
space to measure the speaker similarity between the enrollment and testing
embeddings. The higher the score is, the more likely the two utterances belong
to the same speaker. The scoring function usually adopts cosine [64] and
probabilistic linear discriminant analysis (PLDA) [65] functions.

To formulate an ASV system mathematically, we denote the speaker embedding
extraction as $g$ with parameters $\boldsymbol{\theta_{1}}$ and the similarity
scoring function as $S$ with parameters $\boldsymbol{\theta_{2}}$. Given an
enrollment acoustic features $\boldsymbol{X_{i}}$ and a testing acoustic
features $\boldsymbol{X_{j}}$, the extracted speaker embedding $e$ and output
similarity score $s$ are formulated as:

$\displaystyle
e_{i}=g_{\boldsymbol{\theta_{1}}}(\boldsymbol{X_{i}}),e_{j}=g_{\boldsymbol{\theta_{1}}}(\boldsymbol{X_{j}})$
(1) $\displaystyle s=S_{\boldsymbol{\theta_{2}}}(e_{i},e_{j})$ (2)

###  IV-B Adversarial sample generation

To conduct adversarial attack, attackers will deliberately elaborate a tiny
perturbation and add it to genuine samples to generate adversarial samples. In
a realistic scenario of attacking ASV systems, attackers only have access to
the testing utterance (e.g. $X$ in Fig. 1 (c)) and add adversarial
perturbations (e.g. $\delta$ in Fig. 1 (c)) on it to fool the ASV. Suppose we
have an enrollment utterance $i$ and a testing utterance $j$ in each trial. We
add perturbations into the acoustic features of the testing utterance
$\boldsymbol{X_{j}}$ to degrade the ASV. For target trials, attackers aim at
enlarging the distance of the enrollment and testing representations and
minimizing the similarity score so that the ASV will false reject this trial.
While for non-target trials, attackers aim at making the enrollment and
testing representations closer to each other, and maximizing the similarity
score so that the ASV will false accept the imposters.

Different attacking algorithms utilize different adversarial perturbation
strategies. This work adopts the BIM [66] algorithm to generate adversarial
samples, which is one of the most effective attacking methods. We utilize the
target system directly, i.e. the r-vector (or x-vector) system in this work,
to calculate gradients for adversarial perturbation generation, unless
otherwise specified.

As illustrated in Eq. 3 and 6, BIM perturbs $\boldsymbol{X_{j}}$ in different
directions for target and non-target trials. For target trials, BIM sets the
parameter $k=-1$ and minimize the similarity score, while for non-target
trials, BIM sets the parameter $k=1$ to maximize the similarity score.
Mathematically, to begin with the genuine input
$\boldsymbol{\tilde{X}_{j}^{0}}=\boldsymbol{X_{j}}$, BIM perturbs
$\boldsymbol{X_{j}}$ towards the gradient of the objective $S$ w.r.t.
$\boldsymbol{X_{j}}$ in an iterative manner, as formulated in Eq. 3 and 6.
Such an iterative optimization strategy can efficiently figure out adversarial
perturbations to make ASV behave incorrectly. In the equations, $\alpha$,
$\epsilon$ and $N$ are the parameters of the BIM algorithm. Specifically,
$\alpha$ is the step size, $\epsilon$ is the perturbation degree and $N$
denotes the number of iterations. The function
$clip_{\boldsymbol{X_{j}},\epsilon}(\boldsymbol{X})$ holds the norm
constraints by applying element-wise clipping such that
$\|{\boldsymbol{X}-\boldsymbol{X_{j}}}\|_{\infty}\leq\epsilon$.

$\displaystyle\boldsymbol{\delta}$ $\displaystyle=\alpha
sign(\nabla_{\boldsymbol{\tilde{X}_{j}^{n}}}k\times
S_{\boldsymbol{\theta_{2}}}(g_{\boldsymbol{\theta_{1}}}(\boldsymbol{X_{i}}),g_{\boldsymbol{\theta_{1}}}(\boldsymbol{\tilde{X}_{j}^{n}}))),$
$\displaystyle\boldsymbol{\tilde{X}_{j}^{n+1}}$
$\displaystyle=clip_{\boldsymbol{X_{j}},\epsilon}(\boldsymbol{\tilde{X}_{j}^{n}}+\boldsymbol{\delta}),~{}~{}~{}\text{for
$n=0,...,N-1$}$ (3) $\displaystyle k$
$\displaystyle=\\{\begin{array}[]{cc}-1,&\text{target trial}\\\ 1,&\text{non-
target trial}\end{array}$ (6)

##  V Self-supervised learning

Self-supervised learning has been widely applied in different domains; here,
we mainly focus on self-supervised learning for speech processing. There are
two major branches of self-supervised learning methods in speech domain:
Contrastive Predictive Coding (CPC) and Reconstruction.

The CPC method [67, 68, 69, 70, 71, 72] uses autoregressive modeling of
representations in the feature space, where the model learns to determine the
near future frames in an acoustic sequence while contrasting with frames from
a more distant temporal location. The contrastive loss pulls temporally nearby
representations closer and pushes temporally distant ones further. The CPC
contrastive objective does not directly predict acoustic features, so it is
not suitable for this work.

On the other hand, another branch of work uses reconstruction objectives for
self-supervised learning. Unlike the CPC objective, acoustic features are
predicted as the models’ output, and a reconstruction loss is computed across
the predicted features and ground-truth features. The Autoregressive
Predictive Coding (APC) methods [73, 74] predict future frames like a
recurrent-based LM [75] while conditioning on past frames. In other approaches
[76, 77, 78], parts of the input acoustic features are randomly masked by a
variety of masking strategies, and the L1 reconstruction loss is set as the
target. The random masking strategies of these approaches [76, 77, 78] are
inspired by the Masked Language Model (MLM) task from BERT [79, 80, 81]; hence
these models can be seen as a speech version of BERT. After pretraining, these
models attain the ability to alleviate the superficial noise and keep the
pivotal information in the inputs. Their inference time is fast since they are
Transformer [82] models which are parallelizable in time (unlike
autoregressive models).

Figure 2: The illustration of different inputs with various masking
strategies. The masked part is highlighted by an orange block. (a) is the
original MFCC, and (b), (c), (d) are the MFCC modified by channel masking,
time masking and magnitude masking, respectively.

In this work, we employ the speech BERT trained by denoising tasks as self-
supervised learning based reformer to alleviate or detect the adversarial
perturbation for ASV. Inspired by [76, 77, 78], we adopt a variety of masking
strategies to perturb the inputs and train the self-supervised learning
models. During training, the perturbed acoustic features are fed into the SSLR
models, and the L1 reconstruction loss between the outputs and the clean
acoustic features is derived as the loss function to optimize model
parameters. Fig. 2 illustrates a sample of 24-dimensional MFCC feature
sequences from the Voxceleb1 dataset. Fig. 2.(a) is the original plot of MFCC,
and three masking strategies are shown below:

  * •

In Fig. 2.(b), we adopt the channel masking by randomly selecting a block of
consecutive channels with length $W_{C}$ and masking the values within the
block to zero for all time steps across the input sequence. The channel
masking strategy is originated from SpecAugment [83]. During training, the
masked features are fed into the SSLR. However, clean samples may be taken as
inputs into the SSLR models during the inference stage, which will cause
inconsistency and mismatch problems. So, for $1/(W_{C}+1)$ of the time during
training, none of the channels will be masked such that the model can receive
the entire channel information, which will alleviate the mismatch problem of
model’s inputs between training time and inference time.

  * •

Fig. 2.(c) illustrates the time masking process. We randomly select $P_{T}$
percentage frames out of the total frames, and the selected $P_{T}$ percentage
frames are made of several blocks of contiguous frames with width $W_{T}$
along time. Then, we do masking within the frames according to the following
strategy: 1) 80% of the time, all the selected frames are set to zero. (the
first orange block in Fig. 2.(c)) 2) 10% of the time, we randomly select other
frames to replace them. (the second orange block in Fig. 2.(c)) 3) 10% of the
time, the selected frames remain unchanged. (the third orange block in Fig.
2.(c)) Case 3) also addresses the inconsistency problem between training and
inference. Through time masking, model needs to do reconstruction by
bidirectional representations which contain both past and future context.

  * •

We do magnitude masking by adding Gaussian noise to the whole input acoustic
features with probability $P_{N}$, as shown in Fig. 2.(d). Magnitude masking
can be seen as a kind of data augmentation.

The self-supervised learning based reformer trained by channel masking, time
masking, and magnitude masking are denoted as SSLR-C, SSLR-T, and SSLR-M,
respectively. The permutation of the above three masking methods is adopted to
enlarge and diversify the pool of SSLR models, i.e. SSLR-TC111SSLR-TC means
the self-supervised learning reformer trained by the combination of time
masking and channel masking., SSLR-TM, SSLR-CM, SSLR-TCM are trained for
conducting adversarial defense.

Figure 3: Flow of detection.

##  VI Proposed adversarial defense method

We adopt the self-supervised learning based reformer (SSLR) for adversarial
defense as it attains the ability of purifying the superficial perturbations
and maintaining the pivotal information in the inputs after pretraining. Based
on SSLR, we propose a defense framework which includes one purification
module, named adversarial perturbation purification, and one detection module,
named adversarial perturbation detection, for defending ASV systems against
adversarial samples. In contrast to existing works [56, 57, 58], our proposed
framework doesn’t require knowledge of the process for adversarial sample
synthesis. The two modules in our defense framework are both orthogonal to
adversarial training [56, 42]. The purification module pulls adversarial
samples back towards the decision boundary, which is effective for purifying
tiny adversarial perturbations. The detection module aims at differentiating
the adversarial samples from genuine samples by approximating the statistical
properties of genuine samples and setting a threshold.

###  VI-A Adversarial perturbation purification

The SSLR models are trained to alleviate the superficial noise in the inputs,
extract pivotal information to the hidden representations, and use the hidden
representations to reconstruct the clean inputs. During pretraining, we train
the SSLR models with adequate data for it to learn the ability of purification
and denoising. Based on it, SSLR models attain the ability to alleviate
adversarial noise during the reconstruction process, as adversarial noise is
also a kind of noise to some extent. From the perspective of attackers, they
attempt to manipulate the ASV systems to predict scores higher than the
threshold for non-target trials, and scores lower than the threshold for
target trials by crafting adversarial perturbations and adding them to the
genuine inputs. We use our proposed defense module, cascaded SSLR models, to
purify the adversarial noise, reform the adversarial samples and pull it back
to the decision boundary as shown in Fig. 1.b. Each SSLR slot in Fig. 1.b will
do reconstruction of the inputs and help alleviate the superficial noise in
the inputs. Different masking configurations will equip SSLR models with
different abilities of denoising. In order to enlarge the diversity of self-
supervised learning models and compare the denoising ability of different
masking configurations, we trained $M$ SSLR models with different masking
settings in advance. As shown in Fig. 1.b, we select one of the $M$ SSLR
models, and put it into the $K$ SSLR blocks. During inference, before the
audio is fed into the ASV system, the adversarial perturbation purification
module will alleviate the adversarial noise block by block and pull it back to
the decision boundary.

[58] trained a DNN based filter to protect the ASV systems. But training such
a filter [58] requires the defender to model the attackers and know exactly
the adversarial sample synthesis process. Contrary to [58], the training
process of our SSLR models doesn’t rely on any knowledge about adversarial
samples generation.

###  VI-B Adversarial perturbation detection

The SSLR models attain the capacity to purify the superficial noise in the
adversarial samples and pull them back towards the decision boundary, while
keep the key information of genuine samples and do nearly lossless
reconstruction. Take a non-target trial as an example, the score of the
genuine sample is below the threshold, and the ASV scores change little after
being transformed by the cascaded SSLR models. As a result, the variation of
ASV scores of genuine samples after passing through different numbers of SSLR
models is small. By contrast, the ASV score for the adversarial sample is over
the threshold. The SSLR models will try to purify the adversarial noise and
decrease the ASV score of the adversarial sample to make the score below the
threshold. So the variations for adversarial samples will change more
significantly than genuine samples after being transformed by the cascaded
SSLR models. Based on it, we want to harness the different variations of ASV
scores to distinguish adversarial samples from genuine samples. Specifically,
as illustrated in Fig. 3, we concatenate $K$ SSLR blocks and put one SSLR
model into each block. Then we provide the input $X$ into SSLR-1, passing it
through different number of SSLR models to generate reconstructed outputs
$\\{X^{\prime}_{1},X^{\prime}_{2}...,X^{\prime}_{K}\\}$. Based on
$\\{X,X^{\prime}_{1},X^{\prime}_{2}...,X^{\prime}_{K}\\}$, $K+1$ ASV scores
$\\{s^{\prime}_{0},s^{\prime}_{1},s^{\prime}_{2}...,s^{\prime}_{K}\\}$ can be
derived, as illustrated in Fig. 1. We notice that the score variations of
$\\{s^{\prime}_{0},s^{\prime}_{1},s^{\prime}_{2}...,s^{\prime}_{K}\\}$ are
different for genuine and adversarial samples, i.e. adversarial samples have
larger score variations than genuine ones. Thereby, we decide to apply
statistical moments to illustrate the score variations and use them to
distinguish adversarial and genuine samples. Specifically, we suppose that we
derive $K$ scores for a trial, denoted by $s_{j}$ where $j\in\\{1,2,...,K\\}$.
Then, we compute the $k$-th moment of these scores according to Eq. 7 and 8:

$\displaystyle\overline{s}=\frac{1}{K}\sum_{j=1}^{K}s_{j}$ (7) $\displaystyle
m_{k}=\frac{1}{K}\sum_{j=1}^{K}(s_{j}-\overline{s})^{k}$ (8) $\displaystyle
t_{k}=|m_{k}|$ (9)

where $\overline{s}$ is the average of the computed ASV scores, and $m_{k}$ is
the $k$-th moment of these scores. Since the sign of each moment has no
information on the scale of the variation, we just use the absolute value of
the moment ($t_{k}$) to indicate whether a sample is adversarial or not, as
shown in Eq. 9. Finally, $t_{k}$ is utilized as the detection score $d_{i}$
for each trial $i$.

Previous work [57] directly trained a detector based on both genuine and
adversarial samples and it requires the knowledge of the specific attack
algorithm for generating adversarial samples. However, training our detector
is independent of any knowledge for adversarial sample generation.

###  VI-C Evaluation Metrics

Since previous works on ASV adversarial defense are rare and limited, and also
there is not a common metric to evaluate the defense performance, we propose
several evaluation metrics in this work.

For adversarial perturbation purification module, we propose to utilize GenEER
to evaluate the performance of ASV systems on genuine samples, while utilize
AdvFAR and AdvFRR to evaluate the performance of ASV under adversarial
attacks. In realistic scenarios, system designers are usually not aware of
adversarial samples and determine the system operation point $\tau_{ASV}$ that
minimizes certain metrics evaluated on genuine samples only. To this end, we
determine the $\tau_{ASV}$ based on the EER metrics on genuine samples, as
shown in Eq. 10, 11 and 12. Then, GenEER is derived by Eq. 13 that reflects
the system EER on genuine samples.

$\displaystyle
GenFAR(\tau)=\frac{|\\{s_{i}\geq\tau:i\in\mathbb{T}_{gen\\_ntgt}\\}|}{|\mathbb{T}_{gen\\_ntgt}|}$
(10) $\displaystyle
GenFRR(\tau)=\frac{|\\{s_{i}<\tau:i\in\mathbb{T}_{gen\\_tgt}\\}|}{|\mathbb{T}_{gen\\_tgt}|}$
(11)
$\displaystyle\tau_{ASV}=\\{\tau\in\mathbb{R}:GenFAR(\tau)=GenFRR(\tau)\\}$
(12) $\displaystyle GenEER=GenFAR(\tau_{ASV})=GenFRR(\tau_{ASV})$ (13)

where $\mathbb{T}_{gen\\_tgt}$ and $\mathbb{T}_{gen\\_ntgt}$ represent the
trial sets consisting of genuine target and genuine non-target trials,
respectively, $s_{i}$ denotes the ASV score for the trial $i$ and
$|\mathbb{A}|$ denotes the number of elements in set $\mathbb{A}$. To evaluate
ASV performance on adversarial samples, AdvFAR and AdvFRR are derived by Eq.
14 and 15 given the system operation point $\tau_{ASV}$.

$\displaystyle
AdvFAR=\frac{|\\{s_{i}\geq\tau_{ASV}:i\in\mathbb{T}_{adv\\_ntgt}\\}|}{|\mathbb{T}_{adv\\_ntgt}|}$
(14) $\displaystyle
AdvFRR=\frac{|\\{s_{i}<\tau_{ASV}:i\in\mathbb{T}_{adv\\_tgt}\\}|}{|\mathbb{T}_{adv\\_tgt}|}$
(15)

where $\mathbb{T}_{adv\\_tgt}$ and $\mathbb{T}_{adv\\_ntgt}$ represent the
trial sets consisting of adversarial target and adversarial non-target trials,
respectively. Finally, we evaluate our adversarial purification module in two
aspects: 1) the reduction of system’s AdvFAR and AdvFRR under adversarial
attacks; 2) the influence on system’s GenEER on genuine samples.

For adversarial perturbation detection module, we propose to utilize
$EER_{det}$ for evaluating the detector’s discrimination ability between
adversarial and genuine samples. $EER_{det}$ is achieved by setting a
threshold $\tau_{det}$ of detection scores $d_{i}$ where the FAR of the
adversarial samples and the FRR of the genuine samples equal to each other.
This process can be formulated from Eq. 16 to Eq. 19.

$\displaystyle
FAR_{det}(\tau)=\frac{|\\{d_{i}<\tau:i\in\mathbb{T}_{adv}\\}|}{|\mathbb{T}_{adv}|}$
(16) $\displaystyle
FRR_{det}(\tau)=\frac{|\\{d_{i}\geq\tau:i\in\mathbb{T}_{gen}\\}|}{|\mathbb{T}_{gen}|}$
(17)
$\displaystyle\tau_{det}=\\{\tau\in\mathbb{R}:FAR_{det}(\tau)=FRR_{det}(\tau)\\}$
(18) $\displaystyle EER_{det}=FAR_{det}(\tau_{det})=FRR_{det}(\tau_{det})$
(19)

where $\mathbb{T}_{adv}$ and $\mathbb{T}_{gen}$ denote the trial sets
consisting of adversarial and genuine trials, respectively.

Moreover, with the consideration that adversarial and genuine samples are
pooled together in realistic scenarios, we further pool adversarial and
genuine samples together and compute a joint-FAR (j-FAR) and a joint-FRR
(j-FRR) for system evaluation. These two metrics can reflect a trade-off
performance of our defense approaches between alleviating adversarial attacks
and preserving ASV performance on genuine samples. Our experiments are
conducted where adversarial and genuine samples are combined in equal
proportion, unless otherwise specified. Moreover, in order to investigate the
variation of j-FAR and j-FRR with respective to different proportions of
adversarial samples, we also conduct experiments where adversarial and genuine
samples are combined with a range of proportion values, as discussed in
Section VIII-A.

For the purification module, we pool adversarial and genuine samples together
with a certain portion, then derive j-FAR and j-FRR as Eq. 22 and 23.

$\displaystyle\mathbb{T}_{joint\\_ntgt}=\mathbb{T}_{adv\\_ntgt}\cup\mathbb{T}_{gen\\_ntgt}$
(20)
$\displaystyle\mathbb{T}_{joint\\_tgt}=\mathbb{T}_{adv\\_tgt}\cup\mathbb{T}_{gen\\_tgt}$
(21)
$\displaystyle\text{j-FAR}=\frac{|\\{s_{i}\geq\tau_{ASV}:i\in\mathbb{T}_{joint\\_ntgt}\\}|}{|\mathbb{T}_{joint\\_ntgt}|}$
(22)
$\displaystyle\text{j-FRR}=\frac{|\\{s_{i}<\tau_{ASV}:i\in\mathbb{T}_{joint\\_tgt}\\}|}{|\mathbb{T}_{joint\\_tgt}|}$
(23)

While for the detection module, notice that the detector aims at detecting all
adversarial samples and rejecting them directly, the ground truth operation
for all adversarial trials is rejection. Hence, different from the trial set
partition for the purification module (Eq. 20 and 21), we categorize the
adversarial target trials into the partition of non-target trials for the
detection module, as shown in Eq. 24 and 25. Then, j-FAR and j-FRR are derived
by the operation points of both the detector and ASV, as shown in Eq. 26 and
27, respectively.

$\displaystyle\mathbb{T}_{joint\\_ntgt}=\mathbb{T}_{adv\\_ntgt}\cup\mathbb{T}_{adv\\_tgt}\cup\mathbb{T}_{gen\\_ntgt}$
(24) $\displaystyle\mathbb{T}_{joint\\_tgt}=\mathbb{T}_{gen\\_tgt}$ (25)
$\displaystyle\text{j-FAR}=\frac{|\\{d_{i}<\tau_{det}\cap
s_{i}>=\tau_{ASV}:i\in\mathbb{T}_{joint\\_ntgt}\\}|}{|\mathbb{T}_{joint\\_ntgt}|}$
(26) $\displaystyle\text{j-FRR}=\frac{|\\{d_{i}\geq\tau_{det}\cup
s_{i}<\tau_{ASV}:i\in\mathbb{T}_{joint\\_tgt}\\}|}{|\mathbb{T}_{joint\\_tgt}|}$
(27)

##  VII Experimental setup

###  VII-A Dataset

In this work, we conduct experiments on Voxceleb1 [59] dataset, which consists
of short clips of human speech. There are in total 148,642 utterances for 1251
speakers. We develop our ASV system on the train and development partitions
while reserve 4,874 utterances of the testing partition for evaluating our ASV
system and generating adversarial samples. Notice that generating adversarial
samples is time-consuming and resource-consuming. Without loss of generality,
we randomly select 1000 trials out of 37,720 trials provided in [59], to
generate adversarial samples.

###  VII-B ASV setup

In this work, we develop two ASV systems to demonstrate the robustness of our
proposed methods. They are the x-vector system [9] and the r-vector system
[63].

The x-vector system is configured as [9], except that additive angular margin
(AAM)-softmax loss [84] with hyper-parameters {$m=0.3$, $s=32$} is used for
training. Extracted x-vectors are centered and projected to a 200-dimensional
vector by LDA, then length-normalized before PLDA modeling.

The r-vector system has the same architecture as [63], and AAM-softmax loss
[84] with hyper-parameters {$m=0.2$, $s=30$} is used for training. Extracted
r-vectors are centered and length-normalized before cosine scoring.

TABLE I: ASV performance with genuine and adversarial inputs. x-vector system | complete trials | 1K trials
---|---|---
EER (%) | minDCF | EER (%) | minDCF
genuine input | 5.97 | 0.515 | 6.59 | 0.641
adversarial input | 39.87 | 0.995 | 39.18 | 0.994
r-vector system | complete trials | 1K trials
EER (%) | minDCF | EER (%) | minDCF
genuine input | 8.40 | 0.639 | 8.87 | 0.792
adversarial input | 65.92 | 1.000 | 66.23 | 1.000

###  VII-C ASV performance with genuine and adversarial inputs

Table I shows the x-vector and r-vector system performance on the complete
trials provided in [59] and also the selected 1K trials. The system
performance is evaluated by equal error rate (EER) and minimum detection cost
function (minDCF) with a prior probability of target trials as 0.01.

We observe that system performance seriously degrades after performing
adversarial attacks for both x-vector and r-vector systems, which verifies the
effectiveness of the adversarial attack algorithm and the vulnerability of
SOTA ASV models to adversarial attacks. Besides, we observe consistent
performance trends between the ASV performance on the complete trials and the
selected 1K trials, which indicates it is reasonable to evaluate the defense
approaches on the 1K trials. And we do further experiments on the 1K trials.

###  VII-D SSLR setup

The dataset for pretraining the SSLR models is voxceleb2 [85] which includes
2442 hours raw speech. There are in total 1,128,242 utterances for 6112
speakers in voxceleb2. We use both the training and testing set of voxceleb2
for pretraining. Seven SSLR models with different masking strategies are
pretrained: SSLR-T, SSLR-M, SSLR-C, SSLR-TM, SSLR-TC, SSLR-CM, SSLR-TCM. The
structures of the seven models are the same, they consist of three layers of
transformer encoders with multi-head self attention [86] and two feed-forward
network. The time masking percentage $P_{T}$ is 15%, the width of time masking
$W_{T}$ is 7, the length of channel masking $W_{C}$ is 5, and magnitude
masking probability $P_{N}$ is set to 0.15, and the Gaussian noise for
magnitude masking is with zero mean and 0.2 variance. The acoustic feature for
training the models is 24 dimensional MFCC extracted by standard Kaldi [87].
The Adam optimizer [88] is adopted to pretrain the SSLR models guided by L1
reconstruction loss.

(a)

(b)

(c)

(d)

Figure 4: ASV system performance with different number of cascaded SSLR
models. (a) and (b) show the AdvFAR and AdvFRR of the r-vector system,
respectively. (c) and (d) show the AdvFAR and AdvFRR of the x-vector system,
respectively.

##  VIII Experimental Results: Adversarial Samples Purification

This section evaluates the adversarial purification performance of the SSLR
models. Two situations are taken into account according to the knowledge that
attackers know about the SSLR models. Firstly, we assume that attackers are
unaware of the SSLR models, and adversarial samples are generated with the ASV
model only. This situation is the most practical one since the realistic
attacking scenarios are mainly black-box ones, and the attackers cannot access
the model internals. Secondly, we leave some knowledge of SSLR models to
attackers to demonstrate our methods’ robustness.

###  VIII-A Attackers are unaware of the SSLR models

In this section, we assume that attackers are unaware of the SSLR models in
front of ASV. The AdvFAR and AdvFRR of the r-vector system integrated with
different number of SSLR models are shown in Fig. 4 (a) and Fig. 4 (b),
respectively. We observe that AdvFAR and AdvFRR decrease dramatically as ASV
integrates more SSLR models, which indicates the effectiveness of our proposed
SSLR models on purifying adversarial samples. Specifically, we observe that
SSLR models pretrained by more than one masking strategy have better
performance than those with only one masking strategy. Accordingly, SSLR-TCM
achieves the best defense performance where AdvFAR decreases from over 85% to
around 20% and AdvFRR decreases from over 50% to around 20%, as shown in Fig.
4. One possible explanation is that integrating different masking strategies
strengthens the denoising ability of SSLR, then equips the SSLR models with
better capacity of purifying adversarial samples. Similar results on the
x-vector system are observed in Fig. 4 (c) and Fig. 4 (d), which indicates
that the defense capacity of SSLR models can generalize well to different ASV
systems.

Figure 5: The j-FRR performance for the r-vector system integrated with
different filters under a range of combination portion values.

Because this paper makes the first attempt to conduct adversarial defense by
adversarial perturbation purification without knowing the adversarial sample
generation procedure and there is no baseline for comparison, we employ
traditional filters for defense and set them as baselines. Table II
illustrates the AdvFAR, AdvFRR, and GenEER of the r-vector system integrated
with the SSLR and various traditional filters. We observe that all the SSLR
and filters can effectively purify adversarial samples, resulting in lower
AdvFAR and AdvFRR than the ASV system without any defense methods. However,
all of them negatively influence genuine samples, resulting in higher GenEER
than the original ASV. A possible reason is that the SSLR models and filters
cannot perfectly reconstruct the genuine samples, so some noise is inevitably
added into the genuine samples, which results in higher GenEER. However, as
shown in Table II, our proposed SSLR models have better performance on both
purifying the adversarial samples and maintaining the genuine samples than
traditional filters.

Moreover, our experiments show that integrating ASV with either one SSLR-C or
one SSLR-CM also significantly decreases AdvFAR and AdvFRR, as shown in Table
II. However, their negative influences on genuine samples are large, resulting
in a large GenEER. Based on our experiments, integrating ASV with more than
one SSLR-C or SSLR-CM will seriously degrade ASV system performance on genuine
samples. Also, it has no benefit in improving the performance of adversarial
sample purification. For example, cascading 2 SSLR-C (or SSLR-CM) results in a
larger GenEER of 28.13% (or 30.52%), AdvFAR of 26.39% (or 29.18%) and AdvFRR
of 30.09% (or 33.12%). Thereby, for the configurations of SSLR-C and SSLR-CM,
we only consider cascading one block for performance comparison. While for
SSLR-TCM, we consider cascading 9 blocks for performance comparison as it
achieves the best performance on purifying adversarial samples and preserving
the ASV performance on genuine samples. Cascading 9 SSLR-TCM achieves the best
results on all the metrics, suggesting our SSLR models’ effectiveness in
purifying adversarial perturbations. Similar observations on the x-vector
system are illustrated in Table III.

TABLE II: The adversarial samples purification effectiveness comparison between the SSLR models and traditional filters. The ASV system is the r-vector system. | AdvFAR (%) | AdvFRR (%) | GenEER (%)
---|---|---|---
NA | 87.36 | 51.95 | 8.87
mean | 29.55 | 28.79 | 27.71
median | 30.86 | 29.65 | 27.27
gaussian | 32.16 | 32.68 | 30.95
1*SSLR-C | 22.49 | 26.41 | 22.59
2*SSLR-C | 29.18 | 33.12 | 29.5
1*SSLR-CM | 22.86 | 24.46 | 22.39
2*SSLR-CM | 26.39 | 30.09 | 27.1
1*SSLR-TCM | 52.04 | 51.95 | 9.19
9*SSLR-TCM | 21.00 | 22.29 | 14.59
TABLE III: The adversarial samples purification effectiveness comparison between the SSLR models and traditional filters. The ASV system is the x-vector system. | AdvFAR (%) | AdvFRR (%) | GenEER (%)
---|---|---|---
NA | 35.32 | 44.16 | 6.59
mean | 26.95 | 26.41 | 25.79
median | 25.28 | 25.97 | 23.60
gaussian | 29.00 | 28.57 | 28.20
1*SSLR-C | 16.91 | 23.59 | 19.50
2*SSLR-C | 24.72 | 31.60 | 26.69
1*SSLR-CM | 16.55 | 22.29 | 18.00
2*SSLR-CM | 23.98 | 28.79 | 25.19
1*SSLR-TCM | 29.93 | 37.23 | 6.30
9*SSLR-TCM | 16.54 | 16.88 | 11.50

Here we evaluate the j-FAR and j-FRR performance for both r-vector and
x-vector system (totally 4 settings) under a range of combination portion
values. These 4 settings have similar performance trends, and Fig. 5
illustrates the j-FRR for the r-vector system. In the condition of ASV-only
(NA+ASV in Fig. 5), we observe that j-FRR decreases as the portion of genuine
samples increases. And all filters benefit ASV mostly in the area with high
portion of adversarial samples. Specifically, our proposed SSLR models perform
much better than traditional filters. However, as the portion of genuine
samples increases, the benefits decrease. ASV-only begins to outperform ASV
integrated with traditional filters at the ratio of 1.0, and outperform ASV
integrated with SSLR models at the ratio of 4.0. The reason is that as the
portion of genuine samples increases, the negative influence of filters on
genuine samples outweighs the positive influence on adversarial samples. Due
to limited space, for the rest of this paper, we compute j-FAR and j-FRR by
combining adversarial and genuine samples in equal proportion, unless
otherwise specified.

###  VIII-B Attackers are aware of the SSLR models

This section gives a case study to show the robustness of our adversarial
perturbation purification module to a more severe attacking scenario where
attackers have some knowledge about the SSLR. Specifically, we consider two
scenarios, where 1) attackers are aware of the SSLR but do not know the
specific masking strategies during training, 2) attackers know the specific
masking strategies and can access the internal parameters of the SSLR. The
SSLR model used for defense in this subsection is SSLR-TCM. To mimic such two
scenarios, the SSLR models used for generating adversarial samples for
Scenario 1) and 2) are SSLR-TC and SSLR-TCM, respectively. In this work, we
only give a case study to cascade one SSLR model in front of ASV to generate
adversarial samples for both Scenario 1 and 2.

Figure 6: The AdvFRR performance trends for the r-vector system according to
different knowledge of SSLR that attackers can access.

This section compares the attack destructiveness in the above two scenarios
and the normal scenario where attackers are unaware of the SSLR models. The
AdvFAR and AdvFRR for the r-vector and x-vector systems (totally 4 settings)
have similar trends, and Fig. 6 only demonstrates the AdvFRR of the r-vector
system due to limited space222For new manuscripts, the page limit is 13 pages.
We can put the full results in the revised version if the reviewers consider
it necessary. By comparing the three lines in figures, we observe that as
attackers can access more knowledge of the SSLR models, the generated
adversarial samples become more effective. However, even if attackers can
access the complete parameters of the defense SSLR (SSLR-TCM in our case), the
attack destructiveness is still alleviated by integrating ASV with more SSLR
models.

Considering a realistic scenario where adversarial and genuine samples are
pooled together, we evaluate j-FAR and j-FRR of the ASV systems under three
different scenarios. Table IV illustrates the performance of the r-vector
system under three different scenarios. We also observe similar results on the
x-vector system. From the table, we observe that our proposed SSLR outperforms
traditional filters for both j-FAR and j-FRR in all three scenarios, which
illustrates the effectiveness of SSLR on both purifying adversarial samples
and also preserving ASV performance on genuine samples. Notice that in some
scenarios, cascading a Gaussian filter may lead to worse performance on j-FRR
than ASV only. The reason is that the Gaussian filter has larger negative
influences on genuine samples, which outweighs the positive influences on
adversarial samples.

###  VIII-C ASV fine-tuning

To alleviate the negative effects of the SSLR models on genuine samples, we
make a further attempt to fine-tune the ASV models to better fit with the data
generated by passing genuine samples through the SSLR models. Specifically, we
pass the training data $D$, the development set of Voxceleb1, into the SSLR
models (9 SSLR-TCM models) to generate $\widetilde{D}$, and use the
combination of $D$ and $\widetilde{D}$ to fine-tune the ASV models several
epochs. The fine-tune epochs are 3 epochs and 6 epochs for r-vector and
x-vector, respectively. After fine-tuning, the negative effects are obviously
reduced and the GenEERs are 11.99% and 8.29% for 9 SSLR-TCM incorporating with
the r-vector and x-vector systems, respectively. In the meantime, it is also
observed that the purification performance on adversarial samples is as
effective as the previous one without fine-tuning333For new manuscripts, the
page limit is 13 pages. We can put the full result table in the revised
version if the reviewers consider it necessary.

TABLE IV: The r-vector system performance under different conditions. (Each column refers to the system utilized to generate adversarial samples, while each row refers to the system for defense.) | NA+ASV | SSLR-TC+ASV | SSLR-TCM+ASV
---|---|---|---
j-FAR (%) | j-FRR (%) | j-FAR (%) | j-FRR (%) | j-FAR (%) | j-FRR (%)
NA+ASV | 48.04 | 30.41 | 32.34 | 31.39 | 32.71 | 31.39
mean+ASV | 28.62 | 28.25 | 28.81 | 28.46 | 28.72 | 28.68
median+ASV | 29.00 | 28.46 | 28.90 | 28.25 | 29.28 | 28.46
gaussian+ASV | 31.51 | 31.82 | 31.78 | 31.71 | 31.60 | 31.60
9*SSLR-TCM+ASV | 17.84 | 18.51 | 20.26 | 20.89 | 23.23 | 23.48

##  IX Experimental Results: Adversarial Perturbation Detection

Figure 7: The t-SNE visualization of 10-dimensional scores for adversarial and
genuine samples.

In this section, we perform adversarial defense via a detection module. We
found that ASV scores of adversarial samples change larger after integrating
the ASV with the SSLR models than genuine samples. If we derive ASV scores by
passing the data through ASV integrated with different numbers of SSLR models
and concatenating these scores as a vector, the distributions of the vectors
for adversarial and genuine samples will be different. To demonstrate this
property, given an utterance, we pass it through ASV integrated with 0 to 9
SSLR models to derive 10 scores as a 10-dimensional vector. Then, we use t-SNE
to project this 10-dimensional vector into a 2-dimensional space and visualize
both adversarial and genuine samples, as shown in Fig. 7. This figure shows
that adversarial and genuine samples can be well separated even in this
2-dimensional space. Besides, target and non-target trials are separated well.
This phenomenon indicates that the vectors’ statistics between adversarial and
genuine samples are different, and motivates us to use the properties for
detection.

As the scores of genuine and adversarial samples attain different statistical
properties, we do experiments based on different orders of moments. The
results are shown in Table V. The detection EER ($EER_{det}$) for the r-vector
and x-vector systems are measured by optimizing a threshold. We observe that
such a detection approach can effectively detect adversarial samples for both
systems. Specifically, a higher order statistical moment is more effective on
detection. The 5-th order moment can achieve an $EER_{det}$ of 20.4% and 22.5%
for the r-vector system and x-vector system, respectively. It indicates that
the detection accuracy of the r-vector and x-vector systems are 79.6% and
77.5%, respectively.

TABLE V: The detection equal error rate ($EER_{det}$) based on different statistical moments. | 2nd | 3rd | 4th | 5th
---|---|---|---|---
r-vec | 23.3 | 20.9 | 21.5 | 20.4
x-vec | 26.6 | 23.0 | 24.1 | 22.5
TABLE VI: The j-FAR and j-FRR of ASV and ASV integrated with the detection approach. | r-vec | x-vec
---|---|---
j-FAR (%) | j-FRR (%) | j-FAR (%) | j-FRR (%)
NA+ASV | 48.05 | 8.87 | 31.47 | 6.71
det+ASV | 9.95 | 23.16 | 9.75 | 13.2

We also conduct experiments based on the pool of adversarial and genuine
samples and compare the j-FAR and j-FRR between ASV only and ASV with the
detection module, as Eq. 26 and 27. As shown in Eq. 24 and 25, we treat all
adversarial samples and genuine non-target samples as negative ones, while we
treat genuine target samples as positive ones.

The j-FAR and j-FRR performance is demonstrated in Table VI. We observe that
the j-FARs for both r-vector and x-vector systems dramatically decrease after
integrating ASV with our detection approach, verifying the effectiveness of
our detection module in such a tandem system. The j-FRRs for both r-vector and
x-vector systems increase after applying the detection method. The reason is
that our detection module will false reject some genuine target samples.
However, we observe that the positive effect on j-FARs vastly outweighs the
negative impact on j-FRRs. The increase of FAR is more dangerous than the rise
of FRR in practical scenarios. For example, in financial applications, the
cost of a false alarm is usually much higher than a false reject.

##  X Conclusion

In this work, we propose to harness self-supervised learning based reformer
for adversarial defense from both adversarial perturbation purification and
adversarial perturbation detection perspectives. On the one hand, the
adversarial perturbation purification effectively alleviates the adversarial
noise in the inputs and pulls the contaminated adversarial samples back
towards the decision boundary. The SSLR degrades genuine GenEER very little
after fine-tuning ASV (Sec. VIII-C) – 3.12% absolute and 1.70% absolute for
r-vector and x-vector systems, respectively. This is a mild negative effect
outweighed by the performance improvement of ASV systems under adversarial
attack as shown in Figure 4 \- AdvFAR decreases from 87.36% to 21.00% and from
35.32% to 16.54% for r-vector and x-vector respectively, and AdvFRR decreases
from 51.95% to 22.29% and from 44.16% to 16.88% for r-vector and x-vector
respectively. Also, our proposed method outperforms the traditional filters,
which are firstly adopted for adversarial defense on ASV and set up as a
baseline, not only on purifying the adversarial perturbation, but also on
maintaining the performance of genuine samples. On the other hand, the
adversarial perturbation detection module successfully detects the adversarial
samples with an accuracy of 79.6% and 77.5% for r-vector and x-vector systems,
respectively (Table V).

We also firstly benchmark the evaluation metrics for adversarial defense on
ASV to make it easier for future works to conduct a fair comparison. How to
perfectly perform adversarial defense for ASV remains an open question, and we
provide a potential direction for defense. We wish our method will provide
inspiration and open a door for future research about adversarial defense on
ASV.

## References

  * [1] M.-W. Mak and J.-T. Chien, Machine learning for speaker recognition.  Cambridge University Press, 2020.
  * [2] N. Dehak, P. Kenny, R. Dehak, P. Dumouchel, and P. Ouellet, “Front-end factor analysis for speaker verification,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 19, no. 4, pp. 788–798, 2010.
  * [3] P. Kenny, “A small footprint i-vector extractor,” in Odyssey 2012-The Speaker and Language Recognition Workshop, 2012.
  * [4] S. Prince and J. Elder, “Probabilistic linear discriminant analysis for inferences about identity,” in 11th International Conference on Computer Vision, pp. 1–8, IEEE, 2007.
  * [5] D. Garcia-Romero and C. Y. Espy-Wilson, “Analysis of i-vector length normalization in speaker recognition systems,” in Twelfth annual conference of the international speech communication association, 2011.
  * [6] Y. Lei, N. Scheffer, L. Ferrer, and M. McLaren, “A novel scheme for speaker recognition using a phonetically-aware deep neural network,” in 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1695–1699, IEEE, 2014.
  * [7] E. Variani, X. Lei, E. McDermott, I. Moreno, and J. Gonzalez-Dominguez, “Deep neural networks for small footprint text-dependent speaker verification,” in ICASSP, pp. 4052–4056, IEEE, 2014.
  * [8] D. Snyder, D. Garcia-Romero, D. Povey, and S. Khudanpur, “Deep neural network embeddings for text-independent speaker verification.,” in Interspeech, pp. 999–1003, 2017.
  * [9] D. Snyder, D. Garcia-Romero, G. Sell, D. Povey, and S. Khudanpur, “X-vectors: Robust dnn embeddings for speaker recognition,” in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5329–5333, IEEE, 2018.
  * [10] X. Li, J. Zhong, J. Yu, S. Hu, X. Wu, X. Liu, and H. Meng, “Bayesian x-vector: Bayesian neural network based x-vector system for speaker verification,” arXiv preprint arXiv:2004.04014, 2020.
  * [11] S.-X. Zhang, Z. Chen, Y. Zhao, J. Li, and Y. Gong, “End-to-end attention based text-dependent speaker verification,” in 2016 IEEE Spoken Language Technology Workshop (SLT), pp. 171–178, IEEE, 2016.
  * [12] G. Heigold, I. Moreno, S. Bengio, and N. Shazeer, “End-to-end text-dependent speaker verification,” in 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5115–5119, IEEE, 2016\.
  * [13] D. Snyder, P. Ghahremani, D. Povey, D. Garcia-Romero, Y. Carmiel, and S. Khudanpur, “Deep neural network-based speaker embeddings for end-to-end speaker verification,” in Spoken Language Technology Workshop (SLT), pp. 165–170, IEEE, 2016.
  * [14] E. Zetterholm, M. Blomberg, and D. Elenius, “A comparison between human perception and a speaker verification system score of a voice imitation,” evaluation, vol. 119, pp. 116–4, 2004.
  * [15] Z. Wu, N. Evans, T. Kinnunen, J. Yamagishi, F. Alegre, and H. Li, “Spoofing and countermeasures for speaker verification: A survey,” speech communication, vol. 66, pp. 130–153, 2015.
  * [16] Z. Wu, S. Gao, E. Cling, and H. Li, “A study on replay attack and anti-spoofing for text-dependent speaker verification,” in Signal and Information Processing Association Annual Summit and Conference (APSIPA), pp. 1–5, IEEE, 2014.
  * [17] V. Shchemelinin and K. Simonchik, “Examining vulnerability of voice verification systems to spoofing attacks by means of a TTS system,” in ICSC, pp. 132–137, Springer, 2013.
  * [18] V. Shchemelinin, M. Topchina, and K. Simonchik, “Vulnerability of voice verification systems to spoofing attacks by tts voices based on automatically labeled telephone speech,” in International Conference on Speech and Computer, pp. 475–481, Springer, 2014.
  * [19] T. Kinnunen, Z. Wu, K. Lee, F. Sedlak, E. Chng, and H. Li, “Vulnerability of speaker verification systems against voice conversion spoofing attacks: The case of telephone speech,” in ICASSP, pp. 4401–4404, IEEE, 2012.
  * [20] R. K. Das, X. Tian, T. Kinnunen, and H. Li, “The attacker’s perspective on automatic speaker verification: An overview,” arXiv preprint arXiv:2004.08849, 2020.
  * [21] F. Kreuk, Y. Adi, M. Cisse, and J. Keshet, “Fooling end-to-end speaker verification with adversarial examples,” in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1962–1966, IEEE, 2018.
  * [22] J. Yamagishi, M. Todisco, M. Sahidullah, H. Delgado, X. Wang, N. Evans, T. Kinnunen, K. A. Lee, V. Vestman, and A. Nautsch, “Asvspoof 2019: The 3rd automatic speaker verification spoofing and countermeasures challenge database,” 2019.
  * [23] T. Kinnunen, M. Sahidullah, H. Delgado, M. Todisco, N. Evans, J. Yamagishi, and K. A. Lee, “The asvspoof 2017 challenge: Assessing the limits of replay spoofing attack detection,” 2017.
  * [24] Z. Wu, T. Kinnunen, N. Evans, J. Yamagishi, C. Hanilçi, M. Sahidullah, and A. Sizov, “Asvspoof 2015: the first automatic speaker verification spoofing and countermeasures challenge,” in Sixteenth Annual Conference of the International Speech Communication Association, 2015.
  * [25] A. Gomez-Alanis, J. A. Gonzalez-Lopez, and A. M. Peinado, “A kernel density estimation based loss function and its application to asv-spoofing detection,” IEEE Access, vol. 8, pp. 108530–108543, 2020.
  * [26] G. Lavrentyeva, S. Novoselov, A. Tseren, M. Volkova, A. Gorlanov, and A. Kozlov, “Stc antispoofing systems for the asvspoof2019 challenge,” arXiv preprint arXiv:1904.05576, 2019.
  * [27] X. Li, N. Li, C. Weng, X. Liu, D. Su, D. Yu, and H. Meng, “Replay and synthetic speech detection with res2net architecture,” arXiv preprint arXiv:2010.15006, 2020.
  * [28] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, “Intriguing properties of neural networks,” arXiv preprint arXiv:1312.6199, 2013.
  * [29] N. Carlini and D. Wagner, “Audio adversarial examples: Targeted attacks on speech-to-text,” in 2018 IEEE Security and Privacy Workshops (SPW), pp. 1–7, IEEE, 2018.
  * [30] A. Hannun, C. Case, J. Casper, B. Catanzaro, G. Diamos, E. Elsen, R. Prenger, S. Satheesh, S. Sengupta, A. Coates, et al., “Deep speech: Scaling up end-to-end speech recognition,” arXiv preprint arXiv:1412.5567, 2014.
  * [31] X. Yuan, Y. Chen, Y. Zhao, Y. Long, X. Liu, K. Chen, S. Zhang, H. Huang, X. Wang, and C. A. Gunter, “Commandersong: A systematic approach for practical adversarial voice recognition,” in 27th $\\{$USENIX$\\}$ Security Symposium ($\\{$USENIX$\\}$ Security 18), pp. 49–64, 2018.
  * [32] L. Schönherr, K. Kohls, S. Zeiler, T. Holz, and D. Kolossa, “Adversarial attacks against automatic speech recognition systems via psychoacoustic hiding,” arXiv preprint arXiv:1808.05665, 2018.
  * [33] H. Yakura and J. Sakuma, “Robust audio adversarial example for a physical attack,” arXiv preprint arXiv:1810.11793, 2018.
  * [34] R. Taori, A. Kamsetty, B. Chu, and N. Vemuri, “Targeted adversarial examples for black box audio systems,” in 2019 IEEE Security and Privacy Workshops (SPW), pp. 15–20, IEEE, 2019.
  * [35] Y. Qin, N. Carlini, G. Cottrell, I. Goodfellow, and C. Raffel, “Imperceptible, robust, and targeted adversarial examples for automatic speech recognition,” in International Conference on Machine Learning, pp. 5231–5240, PMLR, 2019\.
  * [36] M. M. Cisse, Y. Adi, N. Neverova, and J. Keshet, “Houdini: Fooling deep structured visual and speech recognition models with adversarial examples,” Advances in neural information processing systems, vol. 30, pp. 6977–6987, 2017.
  * [37] D. Iter, J. Huang, and M. Jermann, “Generating adversarial examples for speech recognition,” Stanford Technical Report, 2017.
  * [38] M. Alzantot, B. Balaji, and M. Srivastava, “Did you hear that? adversarial examples against automatic speech recognition,” arXiv preprint arXiv:1801.00554, 2018.
  * [39] S. Liu, H. Wu, H.-y. Lee, and H. Meng, “Adversarial attacks on spoofing countermeasures of automatic speaker verification,” arXiv preprint arXiv:1910.08716, 2019.
  * [40] H. Wu, A. T. Liu, and H.-y. Lee, “Defense for black-box attacks on anti-spoofing models by self-supervised learning,” arXiv preprint arXiv:2006.03214, 2020.
  * [41] Y. Zhang, Z. Jiang, J. Villalba, and N. Dehak, “Black-box attacks on spoofing countermeasures using transferability of adversarial examples,” Proc. Interspeech 2020, pp. 4238–4242, 2020.
  * [42] H. Wu, S. Liu, H. Meng, and H.-y. Lee, “Defense against adversarial attacks on spoofing countermeasures of asv,” arXiv preprint arXiv:2003.03065, 2020\.
  * [43] C. Kereliuk, B. L. Sturm, and J. Larsen, “Deep learning and music adversaries,” IEEE Transactions on Multimedia, vol. 17, no. 11, pp. 2059–2071, 2015.
  * [44] V. Subramanian, E. Benetos, and M. B. Sandler, “Robustness of adversarial attacks in sound event classification,” 2019.
  * [45] N. Takahashi, S. Inoue, and Y. Mitsufuji, “Adversarial attacks on audio source separation,” arXiv preprint arXiv:2010.03164, 2020.
  * [46] X. Li, J. Zhong, X. Wu, J. Yu, X. Liu, and H. Meng, “Adversarial attacks on gmm i-vector based speaker verification systems,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6579–6583, IEEE, 2020.
  * [47] Y. Xie, C. Shi, Z. Li, J. Liu, Y. Chen, and B. Yuan, “Real-time, universal, and robust adversarial attacks against speaker recognition systems,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1738–1742, IEEE, 2020.
  * [48] M. Marras, P. Korus, N. D. Memon, and G. Fenu, “Adversarial optimization for dictionary attacks on speaker verification.,” in Interspeech, pp. 2913–2917, 2019.
  * [49] Z. Li, C. Shi, Y. Xie, J. Liu, B. Yuan, and Y. Chen, “Practical adversarial attacks against speaker recognition systems,” in Proceedings of the 21st International Workshop on Mobile Computing Systems and Applications, pp. 9–14, 2020.
  * [50] G. Chen, S. Chen, L. Fan, X. Du, Z. Zhao, F. Song, and Y. Liu, “Who is real bob? adversarial attacks on speaker recognition systems,” arXiv preprint arXiv:1911.01840, 2019.
  * [51] Y. Gong and C. Poellabauer, “Crafting adversarial examples for speech paralinguistics applications,” arXiv preprint arXiv:1711.03280, 2017.
  * [52] Q. Wang, P. Guo, and L. Xie, “Inaudible adversarial perturbations for targeted attack in speaker recognition,” arXiv preprint arXiv:2005.10637, 2020.
  * [53] J. Villalba, Y. Zhang, and N. Dehak, “x-vectors meet adversarial attacks: Benchmarking adversarial robustness in speaker verification,” Proc. Interspeech 2020, pp. 4233–4237, 2020.
  * [54] Y. Lin and W. H. Abdulla, “Principles of psychoacoustics,” in Audio Watermark, pp. 15–49, Springer, 2015.
  * [55] H. Bu, J. Du, X. Na, B. Wu, and H. Zheng, “Aishell-1: An open-source mandarin speech corpus and a speech recognition baseline,” in 2017 20th Conference of the Oriental Chapter of the International Coordinating Committee on Speech Databases and Speech I/O Systems and Assessment (O-COCOSDA), pp. 1–5, IEEE, 2017.
  * [56] Q. Wang, P. Guo, S. Sun, L. Xie, and J. H. Hansen, “Adversarial regularization for end-to-end robust speaker verification.,” in Interspeech, pp. 4010–4014, 2019.
  * [57] X. Li, N. Li, J. Zhong, X. Wu, X. Liu, D. Su, D. Yu, and H. Meng, “Investigating robustness of adversarial samples detection for automatic speaker verification,” arXiv preprint arXiv:2006.06186, 2020.
  * [58] H. Zhang, L. Wang, Y. Zhang, M. Liu, K. A. Lee, and J. Wei, “Adversarial separation network for speaker recognition,” Proc. Interspeech 2020, pp. 951–955, 2020.
  * [59] A. Nagrani, J. S. Chung, and A. Zisserman, “Voxceleb: a large-scale speaker identification dataset,” arXiv preprint arXiv:1706.08612, 2017.
  * [60] J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132–7141, 2018.
  * [61] I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” arXiv preprint arXiv:1412.6572, 2014.
  * [62] H. Wu, X. Li, A. T. Liu, Z. Wu, H. Meng, and H. yi Lee, “Adversarial defense for automatic speaker verification by cascaded self-supervised learning models,” 2021.
  * [63] H. Zeinali, S. Wang, A. Silnova, P. Matějka, and O. Plchot, “BUT system description to Voxceleb speaker recognition challenge 2019,” arXiv preprint arXiv:1910.12592, 2019.
  * [64] N. Dehak, R. Dehak, J. R. Glass, D. A. Reynolds, P. Kenny, et al., “Cosine similarity scoring without score normalization techniques.,” in Odyssey, p. 15, 2010.
  * [65] P. Kenny, T. Stafylakis, P. Ouellet, M. J. Alam, and P. Dumouchel, “Plda for speaker verification with utterances of arbitrary duration,” in 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 7649–7653, IEEE, 2013.
  * [66] A. Kurakin, I. Goodfellow, and S. Bengio, “Adversarial machine learning at scale,” arXiv preprint arXiv:1611.01236, 2016.
  * [67] A. van den Oord, Y. Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” 2018.
  * [68] M. Rivière, A. Joulin, P.-E. Mazaré, and E. Dupoux, “Unsupervised pretraining transfers well across languages,” 2020.
  * [69] K. Kawakami, L. Wang, C. Dyer, P. Blunsom, and A. van den Oord, “Learning robust and multilingual speech representations,” 2020.
  * [70] S. Schneider, A. Baevski, R. Collobert, and M. Auli, “wav2vec: Unsupervised pre-training for speech recognition,” Interspeech, 2019.
  * [71] A. Baevski, S. Schneider, and M. Auli, “vq-wav2vec: Self-supervised learning of discrete speech representations,” arXiv preprint arXiv:1910.05453, 2019\.
  * [72] A. Baevski, M. Auli, and A. Mohamed, “Effectiveness of self-supervised pre-training for speech recognition,” 2019.
  * [73] Y.-A. Chung, W.-N. Hsu, H. Tang, and J. Glass, “An unsupervised autoregressive model for speech representation learning,” in Interspeech, 2019.
  * [74] Y.-A. Chung and J. Glass, “Generative pre-training for speech with autoregressive predictive coding,” in ICASSP, 2020.
  * [75] T. Mikolov, M. Karafiát, L. Burget, J. Černockỳ, and S. Khudanpur, “Recurrent neural network based language model,” in Eleventh annual conference of the international speech communication association, 2010.
  * [76] A. T. Liu, S.-W. Li, and H. yi Lee, “Tera: Self-supervised learning of transformer encoder representation for speech,” 2020.
  * [77] A. T. Liu, S.-w. Yang, P.-H. Chi, P.-c. Hsu, and H.-y. Lee, “Mockingjay: Unsupervised speech representation learning with deep bidirectional transformer encoders,” ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), May 2020.
  * [78] P.-H. Chi, P.-H. Chung, T.-H. Wu, C.-C. Hsieh, S.-W. Li, and H. yi Lee, “Audio albert: A lite bert for self-supervised learning of audio representation,” 2020\.
  * [79] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” 2018.
  * [80] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” 2019.
  * [81] Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, “Albert: A lite bert for self-supervised learning of language representations,” 2019.
  * [82] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” 2017.
  * [83] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, “Specaugment: A simple data augmentation method for automatic speech recognition,” arXiv preprint arXiv:1904.08779, 2019.
  * [84] X. Xiang, S. Wang, H. Huang, Y. Qian, and K. Yu, “Margin matters: Towards more discriminative deep neural network embeddings for speaker recognition,” in 2019 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pp. 1652–1656, IEEE, 2019.
  * [85] J. S. Chung, A. Nagrani, and A. Zisserman, “Voxceleb2: Deep speaker recognition,” arXiv preprint arXiv:1806.05622, 2018.
  * [86] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” arXiv preprint arXiv:1706.03762, 2017.
  * [87] D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlicek, Y. Qian, P. Schwarz, J. Silovsky, G. Stemmer, and K. Vesely, “The kaldi speech recognition toolkit,” in ASRU, 2011.
  * [88] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.