# ASVspoof 2021: Towards Spoofed and Deepfake Speech Detection in the Wild

Xuechen Liu, Xin Wang, _Member, IEEE_ , Md Sahidullah, _Member, IEEE_ , Jose
Patino, Héctor Delgado, Tomi Kinnunen, _Member, IEEE_ , Massimiliano Todisco,
_Member, IEEE_ , Junichi Yamagishi, _Senior Member, IEEE_ , Nicholas Evans,
_Member, IEEE_ , Andreas Nautsch, Kong Aik Lee, _Senior Member, IEEE_ Xuechen
Liu and Md Sahidullah are with Université de Lorraine, CNRS, Inria, LORIA,
F-54000, Nancy, France (e-mail<EMAIL_ADDRESS>md.sahidullah@inria.fr).
Md Sahidullah is now an independent research consultant.Xin Wang and Junichi
Yamagishi are with the Digital Content and Media Sciences Research Division,
National Institute of Informatics, Tokyo 101-8430, Japan (e-mail:
<EMAIL_ADDRESS>jyamagis@nii.ac.jp).Jose Patino, Massimiliano Todisco,
Andreas Nautsch, and Nicholas Evans are with Digital Security Department,
EURECOM (Campus SophiaTech), 06410 Biot, France (e-mail:
<EMAIL_ADDRESS><EMAIL_ADDRESS>evans@eurecom.fr). Jose Patino is
now with Cerence Inc. Andreas Nautsch is now with Avignon Université.Héctor
Delgado is with Nuance Communications, C/ Gran Vía 39, 28013 Madrid, Spain
(e-mail: hector.delgado@nuance.com).Tomi Kinnunen and Xuechen Liu are with
School of Computing, University of Eastern Finland, FI-80101, Joensuu, Finland
(e-mail: tomi.kinnunen@uef.fi).Kong Aik Lee is with the Institute for Infocomm
Research, A⋆STAR, 138632 Singapore (e-mail: lee_kong_aik@i2r.a-star.edu.sg).

###### Abstract

Benchmarking initiatives support the meaningful comparison of competing
solutions to prominent problems in speech and language processing. Successive
benchmarking evaluations typically reflect a progressive evolution from ideal
lab conditions towards to those encountered in the wild. ASVspoof, the
spoofing and deepfake detection initiative and challenge series, has followed
the same trend. This article provides a summary of the ASVspoof 2021 challenge
and the results of 37 participating teams. For the logical access task,
results indicate that countermeasures solutions are robust to newly introduced
encoding and transmission effects. Results for the physical access task
indicate the potential to detect replay attacks in real, as opposed to
simulated physical spaces, but a lack of robustness to variations between
simulated and real acoustic environments.The DF task, new to the 2021 edition,
targets solutions to the detection of manipulated, compressed speech data
posted online. While detection solutions offer some resilience to compression
effects, they lack generalization across different source datasets. In
addition to a summary of the top-performing systems for each task, new
analyses of influential data factors and results for hidden data subsets, the
article includes a review of post-challenge results, an outline of the
principal challenge limitations and a road-map for the future of ASVspoof.

Link to the ASVspoof challenge and related resources:
https://www.asvspoof.org/index2021.html

###### Index Terms:

ASVspoof, spoofing, countermeasures, deepfakes, presentation attack detection

##  I Introduction

Biometric systems implemented with automatic speaker verification (ASV)
technology [1] are vulnerable to _spoofing attacks_ [2] whereby an adversary
attempts to masquerade as another individual through the presentation of
manipulated speech data [3]. There are four principal means to generate
attacks: _impersonation_ , _voice conversion_ (VC), _text-to-speech synthesis_
(TTS), and _replay_. Impersonation attacks have received the least attention;
successful attacks typical require special expertise, e.g. of professional
impersonators. In contrast, VC, TTS and replay spoofing attacks can all be
mounted using readily available software toolkits and consumer devices and
have hence received greater attention [3]. The threat posed by such techniques
is now well recognised, particularly in academia and increasingly in industry
[4, 5, 6, 7].

The ASVspoof initiative and challenge series [8] was conceived to foster the
development of countermeasures (CMs) to protect against the manipulation of
ASV systems from spoofing attacks. ASVspoof has designed, collected and
distributed substantial databases of both bona fide and spoofed data, with the
latter being generated with a broad range of state-of-the-art VC and TTS
technologies, recording and replay devices. They have been used in the series
of four biennial benchmarking challenges, the results of which show tremendous
progress in spoofing detection reliability.

The first edition of ASVspoof in 2015 [8] focused on the development of CMs
for the detection of VC and TTS attacks. For the subsequent edition in 2017,
the focus switched to replay attacks. The third edition, ASVspoof 2019, was
the first to address all three attack types through two separate tasks
involving: (i) a logical access (LA) scenario involving VC and TTS attacks
generated using a set of different statistical and neural methods; (ii) a
physical access (PA) scenario involving replay attacks implemented in a large
number of simulated acoustic environments.

While the overall scope of ASVspoof challenges has been steadily expanding
with the introduction of new attack algorithms, evaluation scenarios, and
performance metrics [9], the technical quality of the audio data to date has
remained high; the data is relatively free of additive noise and encoding,
compression and transmission artifacts. While experiments with clean data
detached from the complexities of the real world have a legitimate role in
basic research, the prolonged use of such data may promote the development of
CM solutions which fail to generalize to in-the-wild application scenarios
where external distortions are commonplace. Furthermore, the past ASVspoof
training, development and evaluation sets all originate from the same VCTK
source corpus [10]. This may hinder the design of CMs which generalize to
other domains with different speech characteristics.

Figure 1: The ASVspoof 2021 challenge consists of three different tasks:
logical access (LA); physical access (PA); deepfake (DF). While the general
theme of exposing spoofing countermeasures (CMs) to more realistic conditions
is common to all three tasks, each was effectively run as a separate sub-
challenge. The database and results for each task are described in the
sections indicated.

As illustrated in Fig. 1, the move towards an evaluation using speech data
which is more representative of practical applications was the essence of the
ASVspoof 2021 challenge. It sought to benchmark the latest CM solutions in
more realistic conditions in which speech data undergoes coding, compression
and transmission across telephony channels (LA task, left in Fig. 1), or
undergoes acoustic propagation in a real physical space (PA task, middle in
Fig. 1). ASVspoof 2021 also introduced the new deepfake (DF) task (right in
Fig. 1) in which adversaries aim not to deceive an ASV system, but merely
fabricate an utterance in the voice of a target speaker. The goal may be to
harm the reputation of a well-known personality or to manipulate society by
disseminating disinformation or fake news [11, 12]. The aim behind the new DF
task is to assess the robustness of spoofing detection solutions when used to
detect compressed manipulated speech data of varying characteristics posted
online. The scenario is simulated by processing audio files from different
sources with various codecs used in social media.

Described in this paper are the key advances made since previous editions of
ASVspoof, together with details of the dataset design, collection and protocol
policies. Also described are the challenge results and principal techniques
that are common to the top-performing systems for all three sub-tasks. The
study substantially extends the preliminary workshop version [13] with an in-
depth analysis of influential data factors that impact upon CM performance.
Additionally, we present a summary of system descriptions submitted by
participating teams and identify the most promising detection algorithms and
techniques. The article also contains a new, detailed analysis of results for
specially-designed _hidden subsets_ and includes a survey of _post-challenge_
studies and related work outside of the scope of ASVspoof. None of these were
presented in [13].

The new insights presented in this paper, together with newly released meta-
data for the three ASVspoof 2021 task databases111The meta-data was used in
the post-evaluation analysis presented herein and is publicly available via
the challenge website: https://www.asvspoof.org/index2021.html. should be
helpful to those looking to participate in future ASVspoof challenges as well
as all those working in the field. Toward the end of the paper we reflect upon
the limitations and key lessons learned from ASVspoof 2021, with a discussion
of current ideas and directions for future research.

##  II ASVspoof 2021 Challenge Outline

In this section we outline the three challenge tasks, each an independent sub-
challenge with their own corresponding evaluation database. All three are
partitioned into progress and evaluation subsets in addition to a number of
hidden subsets. Progress subsets of modest size were used for intermediate
assessment prior to the evaluation submission deadline for which participants
submitted scores for the full evaluation database comprising the three
subsets. Only progress and evaluation subsets are discussed in this section,
with corresponding results being presented in Section III. Hidden subsets and
results are presented in Section IV.

For all three tasks, all data was distributed in free lossless audio codec
(FLAC) format with a sampling rate of 16 kHz and with 16-bit quantisation.
There were no new training data for ASVspoof 2021, though protocols for
training and development using data released through previous ASVspoof
editions were made available. Challenge rules and participant guidelines are
available in the challenge evaluation plan [14].

TABLE I: Number of trials in each task. Task | Subset | Bona fide | Spoofed
---|---|---|---
LA | Progress | 1,676 | 14,788
Evaluation | 14,816 | 133,360
PA | Progress | 14,472 | 72,576
Evaluation | 94,068 | 627,264
DF | Progress | 5,768 | 53,557
Evaluation | 14,869 | 5,19,059

###  II-A Logical Access (LA)

The ASVspoof 2015 and 2019 editions both incorporated LA tasks in which all
speech signals were clean, i.e. without either additive noise, reverberation
or other sources of channel variation. As a consequence, the research effort
focused upon the identification of key techniques for the capturing and
classification of processing artifacts which result from the generation of
spoofed speech signals using either TTS or VC algorithms. Such ideal
conditions are unrealistic and other work has shown the likely degradation in
CM performance when they are deployed in practical scenarios such as person
authentication over telephony channels [15, 16].

The ASVspoof 2021 LA task was hence designed to reduce the gap between ideal
laboratory conditions and those to be expected in the wild. This was achieved
with the transmission of both bona fide and spoofed speech across real
telephony systems, including a voice-over-internet-protocol (VoIP) system and
a public switched telephone network (PSTN). The 2021 LA challenge hence
focuses on the study of robustness to nuisance variation from compression,
packet loss and other artifacts stemming from different bandwidths,
transmission infrastructures and bitrates, etc.; the consideration of additive
noise remains for a future challenge edition.

TABLE II: Summary of LA data conditions. Underlined factors appear in evaluation subset but not in progress subset Cond. | Codec |  | Sampling
---
rate
Transmission | Bitrate
LA-C1 | - | 16 kHz | - | 250 kbps
LA-C2 | a-law | 8 kHz | VoIP | 64 kbps
LA-C3 | unk. + $\mu$-law | 8 kHz | PSTN + VoIP | \- / 64 kbps
LA-C4 | G.722 | 16 kHz | VoIP | 64 kbps
LA-C5 |  $\mu$-law | 8 kHz | VoIP | 64 kbps
LA-C6 | GSM | 8 kHz | VoIP | 13 kbps
LA-C7 | OPUS | 16 kHz | VoIP | VBR 16 kbps

Speech data are sourced from the ASVspoof 2019 LA evaluation database, itself
derived from the VCTK database [10]. Spoofed trials are generated using one of
13 different VC, TTS or hybrid spoofing attack algorithms (A07-A19 in [17]).
For all but one condition, _both_ bona fide _and_ spoofed data undergo
transmission across either a VoIP or PSTN+VoIP network using one of six codecs
giving the seven evaluation conditions listed in Table II. The choice of
codecs is broadly representative of traditional or legacy codecs (a-law,
G.722, etc.) and modern IP streaming codecs (OPUS) in use today.

The reference condition C1 is identical to the ASVspoof 2019 LA scenario, i.e.
with neither encoding nor transmission. Conditions C2 and C4-C7 correspond to
transmission across an Asterisk222https://www.asterisk.org private branch
exchange (PBX) system using codecs operating at sampling rates of either 8 kHz
(C2: a-law, C4: $\mu$-law, C5: GSM) or 16 kHz (C4: G.722, C7: OPUS). The
bitrate of each data condition (last column of Table II) was set commensurate
with the codec, with the lowest being 13 kbps for condition C6. Each utterance
is transmitted in its own individual VoIP channel, by automatically generated
Asterisk call files, and between a pair of session initiation protocol (SIP)
endpoints either within a local area network connection, or from an endpoint
hosted in France to endpoints in Italy or Singapore. Condition C3 involves
transmission in Spain over a PSTN system for which codec conditions are
uncontrollable and unknown. Calls are initiated from a mobile smartphone
whereas data recordings are made at a SIP endpoint hosted on a professional
VoIP system which uses a $\mu$-law codec operating with an 8 kHz sampling
rate. Condition C3 may hence reflect the application of multiple, unknown
intermediate transcodings and network transmissions. No codec information nor
any external metadata was provided in the audio file headers. The number of
trials in the LA progress and evaluation subsets is shown in Table I. The
progress subset contains a modest number of C1-C4 utterances, while the
evaluation subset contains the remaining C1-C4 utterances in addition to the
full set of C5-C7 utterances.

The distribution of speakers and spoofing attacks is balanced in each
condition, implying that differences in detection performance can be
attributed reliably to the variations in encoding and transmission. While the
total number of speakers is the same as that of the ASVspoof 2019 LA
evaluation set, the 2021 database contains a substantial number of previously
unexposed, new bona fide utterances collected from the same set of speakers in
addition to new spoofed utterances generated with the same attack algorithms.

Last, with no new matched training nor development data being provided for the
2021 edition, challenge rules dictate the use for training purposes of _only_
ASVspoof 2019 LA training and development subsets which contain only clean
data (no similar encoding and transmission). Use of the 2019 evaluation subset
was strictly forbidden. The challenge for the ASVspoof 2021 LA task is hence
to design spoofing CMs which generalise well to unknown channel variation.
While the use of external _speech_ data was not permitted, the challenge rules
allow the use of external _non-speech_ resources such as noise samples,
impulse responses, and audio compression software for training or data
augmentation [14].

###  II-B Physical Access (PA)

Post-challenge analysis of the ASVspoof 2019 PA results showed evidence of
over-fitting to simulated attacks; CM performance for attacks recorded in real
physical spaces was generally found to be worse than for simulated attacks
[18]. Even so, the use of simulation allows for the generation of plentiful
data in an unlimited number of different, simulated physical spaces at little
cost and hence remains an attractive alternative to recordings in real
physical spaces.

The ASVspoof 2021 PA task was designed to encourage progress in
generalisation, specifically CMs trained and developed using simulated attack
data which reliably detect presentation attacks made in real physical spaces.
The evaluation data hence comprises bona fide speech and replayed recordings
both collected in a variety of real physical spaces. Bona fide data comprises
670 utterances from the VCTK corpus[10]333This is the same set of VCTK
utterances used to create the ASVspoof 2019 PA evaluation set [17, Fig. 2,
#15]. which are presented to an ASV system using a high-quality loudspeaker
with a reasonably flat frequency response [19]. Recordings are made in 162
($=9\times 3\times 6$) acoustic environments comprising nine different rooms
${S}_{\text{asv}}\in\\{\text{R1},\text{R2},\cdots,\text{R9}\\}$, three
different ASV microphones
${Q}_{\text{asv,m}}\in\\{\text{M1},\text{M2},\text{M3}\\}$, and six different
talker-to-ASV distances (and angles)
${D}_{\text{s}}\in\\{\text{D1},\text{D2},\cdots,\text{D6}\\}$ listed in Table
III. The talker-to-ASV distances are illustrated in Fig. 2.

TABLE III: Summary of PA data conditions. Underlined factors appear only in
the evaluation subset and not in the progress subset. Technical details of
microphone and replay devices are available in the supplementary material.

$S_{\text{asv}}$ / $S_{\text{a}}$ | Room size |
---|---|---
$w\times d\times h(m)$ |  |  $D_{\text{s}}$ / $D_{\text{s}}^{\prime}$ / $D_{\text{a}}$ | Angle, Dis.(m)
R1 / r1 | $8.0\times 8.0\times 2.4$ |  | D1 / d1 / c1 |  $15^{\circ}$, 2.0
R2 / r2 | $6.0\times 5.0\times 2.3$ |  | D2 / d2 / c2 |  $45^{\circ}$, 1.5
R3 / r3 | $6.6\times 5.0\times 2.4$ |  | D3 / d3 / c3 |  $75^{\circ}$, 1.0
R4 / r4 | $7.5\times 7.7\times 2.6$ |  | D4 / d4 / c4 |  $90^{\circ}$, 0.5
R5 / r5 | $7.2\times 4.0\times 2.3$ |  | D5 / d5 / c5 |  $120^{\circ}$, 1.25
R6 / r6 | $4.5\times 6.5\times 2.5$ |  | D6 / d6 / c6 |  $150^{\circ}$, 1.75
R7 / r7 | $4.5\times 2.4\times 2.4$ |  ${D}_{\text{s}}$: Talker-to-ASV distance
R8 / r8 | $7.1\times 4.8\times 2.5$ |  ${D}^{\prime}_{\text{s}}$: Attacker-to-ASV distance
R9 / r9 | $5.9\times 4.0\times 2.8$ |  ${D}_{\text{a}}$: Attacker-to-talker distance
${S}_{\text{asv}}$: room for voice presentation |  |  |
${S}_{\text{a}}$: room for replay acquisition |  |  |
$Q_{\text{asv,m}}$ / $Q_{\text{a,m}}$ | Quality |  | $Q_{\text{a, s}}$ | Quality
M1 / m1 | Medium |  | s2 | Low
M2 / m2 | High |  | s3 | Medium
M3 / m3 | Low |  | s4 | High
${Q}_{\text{asv,m}}$: ASV microphone |  ${Q}_{\text{a,s}}$: Attacker replay device
${Q}_{\text{a,m}}$: Attacker microphone |  |  |

Replays are made in the same set of rooms according to different attacker
factors including the room size
${S}_{\text{a}}\in\\{\text{r1},\text{r2},\cdots,\text{r9}\\}$444We use
${S}_{\text{asv}}$ and ${S}_{\text{a}}$ to denote the rooms in which
utterances are presented to an ASV microphone and the rooms in which recording
of the target speaker’s speech are made by the attacker, respectively., the
attacker microphone device
${Q}_{\text{a,m}}\in\\{\text{m1},\text{m2},\text{m3}\\}$, the attacker replay
device ${Q}_{\text{a,s}}\in\\{\text{s2},\text{s3},\text{s4}\\}$555s1 denotes
the simulated bona fide talker., and the attacker-to-talker distance
${D}_{\text{a}}\in\\{\text{c2},\text{c3},\text{c4}\\}$. Attacker replay
devices are situated at the same positions as bona fide talkers
${D}^{\prime}_{\text{s}}\in\\{\text{d1},\cdots,\text{d6}\\}$666Different to
the notations used in [13], the attacker-to-ASV and the attacker-to-talker
distances in this paper are denoted by d* and c*, respectively. c2, c3, and c4
are the same positions as d2, d3, and d4, respectively.. Since the number of
exhaustive attack factor combinations is large (1,458 $=9\times 3\times
3\times 3\times 6$), a non-exhaustive policy was adopted. First, the nine
rooms are divided into three groups777$\\{$R1/r1, R2/r2, R3/r3$\\}$,
$\\{$R4/r4, R5/r5, R6/r6$\\}$, $\\{$R7/r7, R8/r8, R9/r9$\\}$. so that each
group contains one large, one medium, and one small sized room. Replay attacks
in each room $S_{\text{asv}}$ are presented to the ASV microphone using a
single replay device and speech data recorded in three $S_{\text{a}}$ rooms in
the same room group. Recordings are made using the full set of three attacker
microphones ${Q}_{\text{a,m}}$, each set at one of the three attacker-to-
talker distances ${D}_{\text{a}}$. Accordingly, replayed data presented in
each room $S_{\text{asv}}$ cover nine different combinations of replay device,
attacker room, and attacker microphone. The nine combinations are different
for each room in which replays are made.

Figure 2: Illustration of talker-to-ASV distance
${D}_{\text{s}}\in\\{\text{D1},\text{D2},\cdots,\text{D6}\\}$. The full set of
ASV microphones $\\{\text{M1},\text{M2},\text{M3}\\}$ are used in all
positions. Attacker-to-ASV distance $\mathcal{D}_{\text{s}}^{\prime}$ can be
plotted similarly by replacing the talker with the attacker replay device.
Distances and angles depicted for D1 and D2 only; those for D3-6 are listed in
Table III.

Recordings and replays are made using VCTK source utterances at a sampling
rate of 48 kHz. After collection, all bona fide and spoofed data are
downsampled to 16 kHz using a non-causal Hamming-windowed sinc filter
truncated to a length of $2^{14}+1$ samples. This downsampling configuration
is the same as that used in collecting the ASVspoof 2019 PA database. Leading
and trailing non-speech segments of greater than 0.3 seconds are truncated.
The set of trials are then divided into utterance-disjoint progress and
evaluation subsets. Both sets include data from the full set of speakers. As a
further test of generalisation, a subset of replay trials corresponding to the
factors underlined in Table III were withheld from the progress subset and
reserved exclusively for the evaluation subset. For other factors, 80% of the
bona fide and 70% of the replay trials are assigned to the evaluation set,
with the remaining data constituting the progress subset. The number of trials
in each set are listed in Table I.

###  II-C DeepFake (DF)

Evaluation data for the ASVspoof 2021 DF task is a collection of bona fide and
spoofed speech utterances processed with different lossy codecs used typically
for media storage. The data is encoded and then decoded to recover
uncompressed audio. This process introduces distortions which depend on the
codec and its configuration. Generic end-user applications are envisaged,
namely we aim to promote solutions for the detection of deepfakes in
compressed audio used in television and media hosted on news websites and
social media platforms etc. In contrast to the LA and PA scenarios, the DF
task does not involve the use of an ASV system. Accordingly, whereas the t-DCF
[9] is used as the primary metric for LA and PA tasks, that for the DF task is
the CM equal error rate (EER).

While the evaluation database originates partly from the ASVspoof 2019 LA
evaluation set, there are two additional sets of source data. They are the
2018 [20] and 2020 [21] _voice conversion challenge_ (VCC) databases. Both are
publicly available but neither has been used previously for ASVspoof
challenges. In contrast to LA and PA evaluation data, both of which are
derived solely from the VCTK database [10], the VCC 2018 and 2020 challenge
data are derived from DAPS [22] and EMIME [23] corpora, respectively. We
included all the audio data from the two VCC challenges, including source and
target speaker training data and all data submitted by VCC participants.
Combined, these two additional source datasets contain bona fide speech
collected from a total of 26 additional speakers (12 in VCC 2018, 14 in VCC
2020) and a large number of spoofed utterances generated using VC attack
algorithms not used in generating any of the past ASVspoof challenge
databases. The aim was to address CM generalisation across different codecs as
well as different source databases (domains) and spoofing attacks.

The full DF evaluation database is hence generated using in excess of 100
different spoofing attack algorithms generated by the large number teams or
individuals who contributed to the ASVspoof 2019 database [17], or by VCC
participants. It should still be noted that, since many of the VC approaches
share similar voice coders (vocoders), we are most interested in analysing the
dependence of CM performance upon broad vocoder categories (rather than
individual VC systems). To this end, the set of vocoders are divided into five
broad categories: _traditional_ , _neural autoregressive_ , _neural non-
autoregressive_ , _waveform concatenation_ and _unknown_. The unknown category
covers missing or incomplete VC system descriptions.

Audio data from the three sources (ASVspoof 2019, VCC 2018, VCC 2020) was
processed with the set of codecs giving the nine evaluation conditions shown
in Table IV. The ‘no codec’ condition C1 corresponds to original audio
samples. Conditions C2 and C3 both use an mp3 codec. C4 and C5 use an m4a
codec with advanced audio coding,888https://trac.ffmpeg.org/wiki/Encode/AAC
(referred ). whereas both C6 and C7 use an ogg Vorbis
codec.999https://xiph.org/vorbis/ (referred ). The differences between
conditions with the same codec lie in the use of different, variable bit rate
(VBR) configurations, one lower and one higher, as illustrated in the right-
most column of Table IV. Two additional conditions C8 and C9 involve the
successive application of two different codecs, one with a lower VBR, the
other with a higher VBR. The goal for these two dual-codec conditions is to
study whether spoofing artefacts persist in the case of distortion introduced
through transcoding. Such transcoding could take place, for instance, when an
adversary acquires a sample of compressed bona fide speech for a given target
speaker, e.g. a podcast posted online, uses it somehow to generate a
manipulated or deepfake sample, and then uploads the result to a social media
website which uses a different audio compression technique. We used
ffmpeg101010http://ffmpeg.org/ and sox111111http://sox.sourceforge.net/
toolkits in creating the DF evaluation data. The number of trials for the DF
subset is shown in Table I.

TABLE IV: Summary of DF evaluation conditions. Each of these conditions also includes different vocoder types as sub-conditions. Underlined factors only appear in evaluation subset but not in progress subset. Cond. | Compression | Bitrate
---|---|---
DF-C1 | – | 256 kbps
DF-C2 | Low mp3 |  $\sim$80-120 kbps
DF-C3 | High mp3 |  $\sim$220-260 kbps
DF-C4 | Low m4a |  $\sim$20-32 kbps
DF-C5 | High m4a |  $\sim$96-112 kbps
DF-C6 | Low ogg |  $\sim$80-96 kbps
DF-C7 | High ogg |  $\sim$256-320 kbps
DF-C8 | mp3$\rightarrow$ m4a |  $\sim$80-120 kbps, $\sim$96-112 kbps
DF-C9 | ogg $\rightarrow$ m4a |  $\sim$80-96 kbps, $\sim$96-112 kbps
TABLE V: Empirical description of top-5 submissions for each task. Teams are presented in order according to results of pooled normalized min t-DCF. LFCC: Linear frequency cepstral coefficients [24]. L-VQT: long-term variable Q transform [25]. SEnet: Squeeze-and-excitation network [26]. SFR-CNN: sparse feature reactivation CNN block, from CondenseNetV2 [27].  Task | Team ID | Data Augmentation | Acoustic Feature | Classifier | System Fusion
---|---|---|---|---|---
_LA_ | T23 | Trans codec | Raw waveform; Raw spec | LightCNN;ResNet;LSTM | Weighted score average
T35 | Additive noise | LFCC | ResNet | Score average
T19 | RIR;MUSAN | Mel spec | ResNet;SEnet | Weighted score average
T36 | Speed perturbation | Raw Waveform;Mel spec | TDNN;ResNet;MLP | Score average
T20 | RIR;MUSAN;Media codec | Linear Filter Bank | ResNet;MLP | Score average
_PA_ | T07 | Speed perturbation | Log spec | VAE;GMM | Score average
T16 | In-house replay | Linear spec;Raw spec;Phase spectrum | SEnet | Score average
T23 | RIR;MUSAN | Raw spec | ResNet | Weighted score average
T01 | - | linear spec;Mel spec | LightCNN | Score average
T04 | RIR;Speed perturbation | Mel spec; CQT spec | TDNN | Score average
_DF_ | T23 | Media codec | Raw Waveform; Raw spec | LightCNN;ResNet;LSTM | Weighted score average
T20 | RIR;MUSAN;Media codec | Linear filter bank | ResNet;MLP | Score average
T08 | Trans codec;Media codec | CQT spec | LightCNN | Score average
T06 | Trans codec;Media codec | Linear spec; Raw spec | SEnet;TDNN;GMM | Weighted score average
T22 | Media codec | L-VQT spec | SFR-CNN | -

##  III ASVspoof 2021 Challenge Results

A summary of results for all three tasks is shown in Fig. 4. Baseline systems
(B0$n$) and submissions ( T$nn$)121212Submissions with no identifier
correspond to teams that failed to submit a valid system description. are
ranked according their performance on the evaluation subset and primary
metric. For all three tasks, CM performance is illustrated in terms of the EER
in red bars (bottom axes), which is the primary metric only for the new DF
task. For LA and PA tasks, the primary metric is the min t-DCF [9]. These
results are shown in blue bars (top axes). The _ASV floor_ [9, Eq. (11) and
Section V], which corresponds to the t-DCF in the case of a perfect CM but
imperfect ASV system, is illustrated with a vertical blue line. The gap
between blue bars and blue lines give an indication of CM performance. Table V
shows a description of the best performing submissions for each task.

###  III-A Logical Access (LA)

####  III-A1 Full challenge results

Results for the LA task are shown in Fig. 4a. Many CMs outperform the best
baseline B03, some by a large margin. The top systems show t-DCFs close to the
ASV floor, indicating CMs with low error rates. For the majority of systems
the gap in performance for progress and evaluation subsets is modest. The
over-fitting to the progress subset seen for other systems is likely caused by
differences between the known (C1-C4) and unknown (C5-C7) conditions, the
latter of which appear only in the evaluation subset.

####  III-A2 Influential data factors

The left-most pane of Fig. 6a shows the distribution in t-DCF for the top-10
systems for each of the seven conditions shown in Table II. For reference, the
distribution for pooled conditions is shown in the right-most pane. The
distribution in t-DCF for wideband conditions (no codec, G.722, OPUS) is lower
than for narrowband conditions (a-law, PSTN, u-law and GSM), indicating the
general importance of information at higher frequencies to CM performance.
Among the narrowband conditions, lower bit rates and uncontrolled transmission
(GSM and unk. + $\mu$-law, respectively – see Table II) lead to worse
performance.

The middle pane of Fig. 6a shows the distribution in t-DCF for the three
transmission routes detailed in Section II-A, namely routes across a LAN and
from France to Italy (FR-IT) and from France to Singapore (FR-SG). The
similarity in t-DCF for all three shows that transmission routes have little
impact upon CM performance; they are not dissimilar to the distribution for
pooled results. This would suggest that future challenge editions could use
simpler LAN routes only.

Fig. 3 depicts a condition/attack analysis, with attack algorithms on the
vertical axis and evaluation conditions on the horizontal axis. The color of
each circle encodes the median min t-DCF for the top-10 systems, while the
radius represents the inter-quartile range. For some attacks and some
conditions the min t-DCF is consistently high. For attack A18, and to a lesser
extent also A17, there is greater variation across conditions, with the median
t-DCF ranging from approximately 0.4 for condition C1 (no encoding) to over
0.8 for C3 (unk. + $\mu$-law) and approximately 0.7 for C6 (GSM). These
observations show that A18, an attack which is already difficult to detect
even in the absence of encoding, becomes disproportionately more difficult to
detect after low bandwidth encoding.

Figure 3: Statistics on tDCF values of top-10 LA submissions in sub-
conditions. Dot size is decided by inter-quartile range of tDCF distribution,
and color is decided by its median.

####  III-A3 Top-performing systems

A summary of the top-5 systems is presented to the top of Table V. All use
some form of data augmentation, though no common form can be identified. Most
are ensemble systems, and most operate upon short-term spectral features or
raw waveforms. Most use a ResNet classier or variant, with other types of
convolutional networks also being popular. Fusion strategies include weighted
averaging, using either uniformly or empirically set weights.

###  III-B Physical Access (PA)

####  III-B1 Full challenge results

A summary of results for the PA task is shown in Fig. 4b. While almost half of
the systems outperform the best B01 baseline, the performance of all systems
is substantially higher than the ASV floor of 0.12. This would suggest that
the PA task is more challenging than the LA task. The difficulty is likely
caused by the difference between simulated replay attacks (data used for
training and development) [17, Section 2.2.3] and replay attacks recorded in
real physical spaces (evaluation data), e.g., differences in room acoustics
and noise conditions. We also observe only modest gaps between performance for
progress and evaluation subsets indicating that, while error rates remain
high, system performance is stable across different evaluation conditions
(rooms and devices, see Section II-B).

(a) LA (b) PA (c) DF

Figure 4: ASVspoof 2021 results for LA, PA, DF tracks. Results shown in terms
of pooled normalised minimum t-DCF and pooled EER (%). Blue line marks ASV
floor. Only teams that submitted system description are marked with ID.
Figure 5: Break-down of the EER in the VCC 2018/2020 subset of the DF track,
according to the codecs and the type of vocoder used by the VC system.

(a) LA (b) PA

(c) DF

Figure 6: Boxplots of evaluation set min t-DCF or EERs of top-10 submissions
decomposed over different factors. Markers are top-1 submission ($\diamond$),
B01 (o), B02 ($\triangleleft$), B03 ($\medwhitestar$), and B04 ($\times$).

####  III-B2 Influential data factors

Fig. 6c shows min t-DCF results pooled over the top-10 and baseline systems
for a selection of attacker and environment factors. The rooms used for replay
acquisition and voice presentation are not included in the analysis because we
observe no substantial correlation between the room size and the min t-DCF
values. However, we observe expected impacts related to the quality of the
attacker microphone. For example, higher-quality attacker microphones m1 and
m2 lead to higher min t-DCFs than the lower-quality microphone m3. This is to
be expected because spoofed data acquired using a higher-quality microphone
$Q_{\text{a, m}}$ introduces less distortion and are hence more challenging to
detect.

The min t-DCF is also higher for a better quality attacker replay device
$Q_{\text{a, s}}=\text{s3}$ or s4. Similarly, shorter attacker-to-talker
distances $D_{\text{a}}$ also lead to worse performance. At the closest
position $D_{\text{a}}=\text{c4}$, min t-DCF values are the highest.

The attacker-to-ASV distance $D^{\prime}_{\text{s}}$ is observed to be
inversely correlated with the min t-DCT. However, as expected, if the talker
is closer to the ASV microphone (i.e., moving from $D_{\text{s}}=\text{D1}$ to
D4), then the min t-DCF decreases.

####  III-B3 Top-performing systems

A summary of the top-5 PA systems is presented in the middle of Table V.
Similar to the LA task, most use data augmentation, once again with various
different approaches, and all are ensemble systems. Linear, raw or mel scaled
spectrogram features are most common, whereas classifiers are diverse. The
top-1 system uses both frame-level and temporal-level features and the
parallel combination of variational autoencoder (VAE) and Gaussian mixture
model (GMM) classifiers. Like for the LA task, all the top-5 systems use score
averaging.

###  III-C DeepFake (DF)

####  III-C1 Full challenge results

A summary of results for the DF task is shown in Fig. 4c. The main observation
is the striking difference in results for progress and evaluation subsets.
While 23 (out of 33) systems have EERs of less than 10% for the progress
subset, and while the best performing system even has an EER of less than 1%,
_all_ have EERs exceeding 15% for the evaluation set. This is a strong
indication of over-fitting. Despite high EERs, however, 18 systems
outperformed the best B04 baseline system.

####  III-C2 Influential data factors

The left-most pane of Fig. 6c shows the distribution in EER performance for
each compression method, with the distribution for pooled conditions being
shown in the right-most pane. For mp3 compression, a higher VBR (C3) gives a
lower median EER than for a lower VBR (C2). For m4a and ogg codecs, there is
no similar effect between the low and high bit rates for either codec. EERs
for the ogg codec are lower than those for both mp3 and m4a codecs, no matter
what the VBR. Interestingly, EERs for the two double compression methods are
only modestly higher than those for the single ogg compression, but below
those for either mp3 or m4a compression, indicating some resilience to
transcoding.

The middle pane of Fig. 6c shows the distribution in EER for each source
database. EERs are substantially higher for VCC datasets than for the ASVspoof
2019 LA database. This is likely due to the use of the latter in the progress
set only and is hence a consequence over-fitting, or lack of generalisation to
mis-matched VCC source data. Further analysis of this observation is provided
in Section IV-C.

Finally, Fig. 5 shows a breakdown in the EER for pooled VCC 2018 and 2020
source data according to the type of vocoder and codec. The results indicate
that, for any given codec, both types of neural vocoders yield higher EERs
than more traditional vocoders or waveform concatenation.

####  III-C3 Top-performing systems

A summary of the top-5 DF submissions is illustrated to the bottom of Table V.
With one exception, all submissions are ensemble systems. All use data
augmentation. While there are different approaches, all use some form of media
codec augmentation.131313We use _media codec_ to refer to generic codecs
implemented in various software packages. The challenge participants used mp3,
m4a, ogg, opus, mp4, aac, alac, vorbis, wma and sbc, along with other open-
source toolkits for media codec augmentation. Acoustic features are diverse. A
range of different classifiers are also used, including mostly convolutional
neural networks, but with simpler MLP and GMM classifiers also featuring among
the top-5 submissions. Once again, all top-5 systems use score averaging for
fusion.

##  IV Hidden Subsets and Additional Analyses

We now turn attention to results for the hidden subsets first mentioned in
Section II-B before presenting other analyses. The purpose of hidden subsets
is to assess the dependence of CM performance upon certain data
characteristics that were withheld from challenge participants. They were
unaware of their inclusion and treated hidden subsets in the same manner as
evaluation data. Hidden subsets were not used in deriving challenge results or
rankings.

###  IV-A The role of non-speech

The databases for all three tasks contain a hidden subset of utterances from
which non-speech intervals are automatically removed. The motivation is to
assess performance when spoofed _speech_ detectors are constrained to operate
only upon _speech_ segments; recent reports (e.g. [28, 29, 30]) suggest that
some CMs might rely upon information within non-speech segments. While non-
speech information might serve as legitimate cues for spoofing detection,
adversaries could easily remove non-speech segments to gain an advantage.
Alternatively, cues within non-speech intervals might correspond to unintended
database design artifacts. It is hence of interest to determine any
differences in CM performance when they are restricted to using information
from speech segments only.

(a) LA (b) PA (c) DF

Figure 7: Comparison of top-10 CMs performance on evaluation subset (with
silence) and hidden subset (w/o silence). For LA and DF, min t-DCFs over VC-
and TTS-based spoofing attacks are plotted separately. Markers are top-1
system ($\diamond$), B01 (o), B02 ($\triangleleft$), B03 ($\medwhitestar$),
and B04 ($\times$).

Bona fide and spoofed utterances are selected at random from evaluation
subsets giving the number of utterances shown in Table I for LA, PA and DF
tasks. An energy-based voice activity detector with the recommended threshold
settings [31, Section 5.1] is then used to remove segments labeled as non-
speech in addition to any segments of less than 50ms duration. For the LA
task, the hidden subset contains C1-C7 utterances that are all treated by a
VAD to remove non-speech segments. So as to focus upon speech/non-speech
effects only, rather than variations in room characteristics and replay
configurations etc., the PA hidden subset contains data from the D4 talker-to-
ASV distance only (90∘, 0.5 m from the ASV microphone as per Table IV). The
hidden subset for the DF task comprises utterances for all conditions C1-9,
but ASVspoof 2019 LA source data only.

Contrasting min t-DCF/EER results for evaluation subset (w/non-speech) and
hidden subset (w/o non-speech segments) are plotted in Fig. 7 for the LA task
(a), the PA task (b) and DF task (c). Boxplots for the top-10 LA systems are
shown separately for VC-based (left), TTS-based (middle) and pooled (right)
attacks.141414Following the descriptions in [17], A17, A18, and A19 are
grouped as VC-based attacks. All others are grouped as TTS-based attacks.
While the contrast is greatest for TTS attacks, min t-DCFs calculated from
utterances containing non-speech segments (w/non-speech) are substantially
lower than those calculated from utterances without non-speech segments (w/o).
LA source data contain long non-speech segments at the start and/or end of
each utterance. While VC spoofing attack algorithms reproduce these
characteristics, TTS algorithms produce utterances without such long non-
speech segments. Their length may hence serve as a cue to distinguish between
bona fide and spoofed utterances, accounting for why their absence leads to
degraded performance. The reliance upon such cues may not lead to reliable
detection in the wild. Observations for the DF task shown in Fig. 7c are much
the same as for the LA task. Without the use of non-speech segments,
performance is substantially degraded, more so for TTS attacks than VC
attacks.

Boxplots for the top-10 PA systems are shown in Fig. 7b for replay attacks
recorded in real physical spaces, with (evaluation subset) and without (hidden
subset) non-speech segments (two left-most boxes). The length of non-speech
intervals is not expected to have an impact since recording and replaying do
not alter utterance duration. Nonetheless, without non-speech segments,
performance is notably worse.

Any asymmetry in the _length_ of non-speech segments in bona fide and spoofed
speech should not serve as a detection cue. The _contents_ of non-speech
segments, however, is a different issue. Non-speech segments, while being
information that is easily removed by the adversary, are an innate
characteristics of recorded speech which might still provide legitimate
spoofing detection cues.

###  IV-B Real v/s simulated replay attack

The second hidden subset in the PA database contains _simulated_ replayed
data. The motivation is to determine whether reliable estimates of performance
can also be derived using evaluation data that is also simulated instead of
being recorded in real physical spaces. Simulated bonafide and spoofed data
are created using the same tool as that used in generating the ASVspoof 2019
PA database [17, Section 4.3]. Room impulse responses and background noises
are extracted from swept-sine signals recorded in the same rooms and with the
same set of microphones. For the same reasons outlined above, data is
simulated for the D4 talker-to-ASV distance only.

Results shown in the right-most pane of Fig. 7b depict the distribution in min
t-DCFs for the top-10 PA systems for the hidden, simulated subset which
contains non-speech intervals. The median min t-DCF for the simulated subset
is much higher than that for the evaluation subset. Further analysis shows
that four out of the top-10 PA systems’ min t-DCFs for the simulated subset
are higher than 0.99. These results discourage the use of simulated evaluation
data to estimate CM performance in real physical environments.

###  IV-C Performance gap for DF progress and evaluation subsets

Fig. 4 shows a substantial gap in performance for progress and evaluation
subsets for the DF task. As evident from results shown in Fig. 6c, the gap
relates in part to the inclusion of two previously unexposed source corpora.
To explore the cause for this gap, we examined the class- and data source-
conditional CM score distributions for the top-5 submissions. Results are
shown in Fig. 8 along with distributions for the best B04 baseline. While
similar findings were observed for compressed data conditions, the plots shown
correspond to non-compressed data (condition C1 in Table IV).

While, for any given system, score distributions for the spoofed class (bottom
row) are reasonably well aligned, there are significant differences in the
distributions for the bona fide class (top row). Scores for the two VCC source
corpora are consistently lower than those for the ASVspoof 2019 LA corpora.
This shift leads to greater overlap in the distributions for bona fide and
spoofed classes. The use of only ASVspoof 2019 LA data for training leads to
over-fitting and models which generalise poorly to VCC data in the evaluation
subset, greater confusion between bona fide and spoofed trials, and hence
degraded detection performance.

As a further numerical quantification, Table VI displays the EERs with either
bonafide or spoof trials constrained to specific subsets. We find that
whenever either one of the two VCC subsets is included on bonafide side (the
first three rows), the EERs are very high ($\sim 15\%\dots 34\%$). In stark
contrast, however, when both VCC’18 and VCC’20 trials are _excluded_ from
bonafide side (the last three rows), the EERs of the six CMs are substantially
lower ($<1\%$ for T23) – regardless of whether the _spoof_ trials include
VCC’18 or VCC’20 data.

These findings, and given the performance mismatch between evaluation and
hidden track which excludes silences, motivated a deeper exploration of the
possible role of silence in the VCC bonafide score distribution shift. To this
end, for each audio segment from C1 (pooled from the progress and evaluation
subsets), the _proportion of detected nonspeech_ was calculated and used as
recognition score. EER computed from these scores reflects the degree of
similarity of the corresponding positive and negative trial sets in terms of
nonspeech proportion. These EERs, for different combination of positive and
negative trials, are shown in Table VII. The trends are similar to those shown
in Table VI; EERs are substantially lower when ASVspoof 2019 bonafide data is
used as positive set, particularly when the negative set contains VCC 2018 and
VCC 2020 spoof data. The last two rows show an alternative experiment were
both positive and negative sets consists of bona fide data only. EER values
between 11% and 14% 10% indicate that the nonspeech proportion distribution
for ASVspoof and VCC bonafide data are reasonably well separated. It would not
be a surprise that submitted CMs learned the bonafide class based (at least,
partially) on the implicit different silence distribution of the ASVspoof
bonafide data (the only data source included in the training set). This in
turn makes them fail when an unseen bonafide dataset does not follow the same
silence distribution.

Figure 8: Score distributions of bonafide (top) and spoof (bottom) for top-5 DF subsystem and baseline B04, broken down according to three data sources used. TABLE VI: EERs (%) for the DF track with all compressors pooled, but bonafide/spoof trials selected from different combinations of source datasets. Bonafide | Spoof |  |  |  |  |  |
---|---|---|---|---|---|---|---
trials | trials | T23 | T20 | T08 | T06 | T22 | B04
All | All | 15.63 | 16.04 | 18.29 | 19.01 | 19.22 | 22.38
VCC18 | All | 23.39 | 27.64 | 31.03 | 30.87 | 29.81 | 34.44
VCC20 | All | 16.98 | 19.77 | 21.55 | 22.88 | 25.00 | 31.48
ASV19 | ASV19 | 0.33 | 2.41 | 3.75 | 8.82 | 2.76 | 6.21
ASV19 | VCC18+20 | 0.60 | 1.98 | 2.78 | 5.83 | 1.93 | 3.09
ASV19 | All | 0.55 | 2.05 | 2.92 | 6.32 | 2.11 | 3.56
TABLE VII: Similar to Table VI, but using the proportion of detected nonspeech frames by VAD as spoofing detection score. Only trials of C1 were considered. Positive set | Negative set | EER
---|---|---
All bona | All spoof | 37.59
VCC18 bona | All spoof | 44.17
VCC20 bona | All spoof | 51.14
ASV19 bona | ASV19 spoof | 24.98
ASV19 bona | VCC18+20 spoof | 9.19
ASV19 bona | All spoof | 10.36
ASV19 bona | VCC18 bona | 11.90
ASV19 bona | VCC20 bona | 14.00

##  V Post-Challenge Studies and Related Work

Each edition of ASVspoof addresses new challenges but also leads to the
identification of new challenges and concerns which have stimulated a number
of post-challenge studies. The 2021 edition is no exception. A selection of
such studies is discussed below, with a summary of related results shown in
Table VIII.151515The selection is based on searches of “ASVspoof 2021” and
“Deepfake” in the ISCA archive and IEEExplore. They pertain to LA and DF tasks
and (1) the shift from the use of acoustic features toward end-to-end (E2E)
architectures with raw waveform inputs and (2) the popularity of data
augmentation (DA).

Originating from [32] and based on RawNet [33] and graph attention networks,
AASIST [34] is representative of progress in the use of E2E architectures. The
RawNet in AASIST has further been replaced by various _self-supervised
learning_ (SSL) front-ends, including wav2vec [35, 36, 37]. While the use of
SSL brings substantial improvements in performance, the use of models pre-
trained on external data is not compliant with the ASVspoof training protocol.
A study of different SSL frontends and related techniques can be found in
[36]. While conventional acoustic features remain popular [38], E2E
architectures and SSL-based frontends have led to substantial improvements in
performance for the LA task, even for single systems (not ensembles), with
best t-DCF being 0.2066 (best EER 0.28%) for the ASVspoof 2021 LA database
[35]. Such result comes from the use of SSL with data augmentation. Post-
challenge studies that employ ASVspoof 2021 PA data and work on replay attacks
are rarely seen.

An investigation of DA for the E2E solution which operates directly upon raw
waveform inputs, named _RawBoost_ , was reported in [39]. The approach is
based upon the combination of linear and non-linear filtering in addition to
signal-dependent and signal-independent additive noise. Investigation on DA
techniques applied on anti-spoofing can be found in [38, 40, 41]. In these
studies a number of approaches to DA have been used. Compression, codec and
channel effect, bandwidth difference augmentation, as well as time- frequency-
masking [42]. All these studies confirmed that the robustness of spoofed audio
detection is substantially improved when DA techniques are employed.

Finally, other anti-spoofing / deepfake detection challenges and databases
have emerged since 2021. The first _audio deep synthesis detection_ (ADD)
challenge [43] was launched to evaluate the threat of low-quality and
partially-spoofed audio [44] in order to address real-time and more
challenging scenarios. The summary of the results indicates the fragility of
synthesis detection model against lower quality and unseen fake audio,
especially when partially-spoofed audio is presented. The spoofing-aware
speaker verification (SASV) challenge [45], which exploits the same ASVspoof
2019 LA source database as the ASVspoof 2021 LA evaluation, provides protocols
for the design of not just CM solutions, as in the case of ASVspoof, but also
ASV systems and alternative architectures for their combination. Three
different metrics support the assessment of spoofing robustness, automatic
speaker verification and combined, spoofing aware speaker verification. A
summary of SASV results [45] anticipates the significance of improving the
reliability of both ASV and CM sub-systems, including the joint optimization
schemes.

TABLE VIII: Selected post-challenge results from published papers. Top-1 submission in main challenge and two best baselines are added for reference. |  | min t-DCF | EER (%) |  |  |  | EER (%)
---|---|---|---|---|---|---|---
LA | [35] | 0.2066 | 0.82 |  | DF | [35] | 2.85
T23 | 0.2176 | 1.32 |  | [37] | 4.98
[37] | 0.2780 | 3.54 |  | [36] | 6.18
[38] | 0.2882 | 4.66 |  | [38] | 14.27
B03 | 0.3445 | 9.26 |  | T23 | 15.64
[36] | 0.3590 | 7.18 |  | B04 | 22.38
B04 | 0.4257 | 9.50 |  | B03 | 23.48

##  VI Limitations and future directions

While we are hopeful that ASVspoof has made substantial contributions to open
data and reproducible research in anti-spoofing, like any evaluation, it is
not without limitations. Some of the most relevant issues that are fundamental
to the future ASVspoof road-map are discussed in the following.

Realistic audio data – ASVspoof 2021 made important steps towards evaluation
with data that is more representative of realistic conditions. Other steps
remain to be taken, especially the consideration of background noise for the
LA task, the collection of recordings from human talkers as opposed to
loudspeakers for the PA task, and the use of real social media for the DF
task.

Role of non-speech – The results for hidden data sets confirm that non-speech
intervals can play a role in the detection of spoofed speech, particularly for
LA and DF tasks. While non-speech intervals are an innate characteristic of
natural speech, and while their contents can help to distinguish between bona
fide and spoofed speech, their length is a database characteristic that should
not serve as a cue for detection. It is of interest to investigate the
generation or conversion of non-speech regions in the case of TTS and VC
attacks as well as the difference between cues used for detection in non-
speech and speech intervals.

Training data policy – The use of a strict training protocol ensures the
meaningful comparability of results generated by different teams and is
fundamental to most benchmarking evaluations. Even so, post-evaluation results
show the benefits of a relaxed training policy which allows the use of larger,
more complex models trained using external data, e.g. via semi-supervised
learning. To allow for their further exploration in a benchmarking evaluation
setting, future editions of ASVspoof could hence allow a dual training policy,
one fixed, the other relaxed.

Diversity in attacks – The spoofing attacks in the 2021 LA and DF databases
were generated with TTS and VC algorithms that were at the state of the art
prior to 2020. TTS and VC technology has advanced rapidly in recent years,
especially in terms of requiring far less target speaker training data [46,
47, 48]. The next ASVspoof edition calls for a renewed spoofed data collection
effort and the exploration of vulnerabilities to the latest attack techniques.

Diversity in audio data – ASVspoof data are exclusively English-language,
read-speech sourced from the VCTK corpus [10]. The inclusion of two unexposed
datasets in the DF database exposed a lack of CM generalization. The cause is
due mostly to differences in the characteristics of _bona fide_ speech and
score distributions. This may suggest over-fitting to speakers in the training
datasets. In the future, ASVspoof should increase diversity not only in
spoofing attacks, but also in bona fide source data, including collection in
different recording environments, languages and speaking styles, among other
factors.

Extensible evaluation – The two ASV-related tasks, LA and PA, used a fixed ASV
system, with participants developing only the spoofing CM. The literature
shows an increasing interest in the joint optimisation of both CMs and ASV
systems and alternative combination architectures. The spoofing-aware speaker
verification (SASV) challenge [45] was launched to support research in this
direction. SASV challenge results [45] have shown the need for larger labeled
datasets for the learning of speaker embedding extractors and end-to-end
solutions. ASVspoof and SASV will likely merge for a common evaluation in the
future. This will require not only a huge data collection effort but further
thought to evaluation metrics; ASVspoof uses an ASV-constrained t-DCF, whereas
the SASV challenge uses a set of three different EER-based metrics.

Integration of multiple scenarios – There is no reason why replay attacks
cannot be made in LA scenarios (variable microphone, telephony applications),
nor why TTS and VC attacks cannot be replayed in PA scenarios (fixed
microphone, access control applications). Their exploration will increase the
complexity of both protocol design and data collection, but remains an
important future direction for ASVspoof.

Partially spoofed audio – ASVspoof has considered only the scenario in which
utterances are either bona fide or spoofed in _full_. _Partial_ spoofs, where
only a potentially short interval or perhaps a single word is substituted or
manipulated, are known to pose a greater detection challenge [44]. The
partially spoofed scenario is perhaps most relevant to the DF task in which
there is no ASV system. This is because partially-spoofed utterances of the
shortest duration may be unlikely to provoke ASV false alarms, but can still
fundamentally change the meaning of a given phrase161616An example might be
the manipulation of a phrase within a video posted to social media from _‘I
won the election’_ to _‘I lost the election’_.. Nonetheless, utterances in
which a greater proportion of the utterance is spoofed still stand to fool the
ASV system used in LA and PA tasks while also being more difficult to detect
with a CM. Depending of the application, given a partially spoofed utterance,
it might then be of interest to estimate bona fide and spoofed labels at the
segment level, e.g. to _explain_ the classifier decision.

Adversarial attacks – Many examples in the literature show the threat to
reliable ASV stemming from adversarial attacks [49, 50]. These target the ASV
system, but rarely the CM. With estimates of security being only as strong as
the adversary model, future editions of ASVspoof should consider adversarial
attacks that target _both_ the ASV _and_ the CM.

##  VII Conclusions

The ASVspoof 2021 challenge was designed to foster progress in reliable
automatic speaker verification, spoofing and deepfake detection in more
realistic and practical scenarios. The challenge attracted submissions from 37
teams across the three tasks, each of whom submitted valid challenge scores
and system descriptions. This paper provides an overview of the challenge
datasets and an analysis of results. We also report an analysis of various
data-related issues, in addition to new results for hidden data subsets. They
reveal some challenge limitations, prompting ideas for future challenge
editions.

Results show that the transmission of speech data across real telephony
systems in a logical access task causes only modest degradation to spoofing
detection reliability and that estimates of performance from transmission
across a local area network are as reliable as estimates from transmission
across more geographically distant endpoints. Impacts on performance caused by
compression effects in a deepfake task are also modest, but results show a
lack of genearlisation to different source data. Data augmentation is common
to all top-performing systems for both logical access and deepfake tasks.

The physical access task appears to be the most challenging of the three,
likely due to the mismatch between training and evaluation data. Results show
the added difficulty when attacks are recorded with higher quality microphones
and replayed with high-quality loudspeakers, both at a short distance from the
talker and microphone respectively, or when the automatic speaker verification
microphone is of lower quality.

Future editions of ASVspoof will continue the path towards ever-more realistic
conditions and will consider stronger adversaries, larger databases containing
greater variation in acoustic conditions and spoofing attacks, data collected
from a larger speaker population and an additional, relaxed training data
policy. Finally, ASVspoof and SASV challenges will merge to allow greater
flexibility in the form of jointly-optimised CM and ASV sub-systems and
alternative combination architectures.

## Acknowledgments

The ASVspoof 2021 organising committee extends its sincere gratitude and
appreciation to the 72 participants and teams. Since participants are assured
of anonymity, we regret that we cannot acknowledge them here by name. This
work was supported jointly by the French National Research Agency (grant No.
ANR-18-JSTS-0001) and the Japan Science and Technology Agency (grant No.
JPMJCR18A6) under project VoicePersonae. The work also received partial
support from Academy of Finland (proj. 349605, SPEECHFAKES) and MEXT KAKENHI
Grants (21K17775, 21H04906). The work also received partial support from
Region Grand Est, France, and from the Science and Engineering Research
Council, Agency for Science, Technology and Research (A⋆STAR), Singapore,
through its Council Research Fund (Project No. CR-2021-005).

## References

## References

  * [1] Zhongxin Bai and Xiao-Lei Zhang  “Speaker recognition based on deep learning: An overview”  In _Neural Networks_ 140, 2021, pp. 65–99  DOI: https://doi.org/10.1016/j.neunet.2021.03.004
  * [2] “ISO/IEC 30107. Information technology – biometric presentation attack detection”  ISO, Geneva, Switzerland, 2016
  * [3] Zhizheng Wu et al.  “Spoofing and countermeasures for speaker verification: A survey”  In _speech communication_ 66 Elsevier, 2015, pp. 130–153
  * [4] “Use real-time caller authentication with Voice ID” Accessed: 2022-09-10, https://docs.aws.amazon.com/connect/latest/adminguide/voice-id.html
  * [5] “What is a deepfake? Everything you need to know about the AI-powered fake media” Accessed: 2022-09-10, https://www.businessinsider.com/guides/tech/what-is-deepfake?IR=T
  * [6] “Deepfake Audio Detection in the Call Center” Accessed: 2022-09-10, https://www.pindrop.com/blog/deepfake-audio-detection-in-the-call-center
  * [7] “Facebook, Microsoft Back Contest to Better Detect Deepfakes” Accessed: 2022-09-10, https://www.wired.com/story/facebook-microsoft-contest-better-detect-deepfakes/
  * [8] Zhizheng Wu et al.  “ASVspoof 2015: the first automatic speaker verification spoofing and countermeasures challenge”  In _Proc. Interspeech_ , 2015, pp. 2037–2041
  * [9] Tomi Kinnunen et al.  “Tandem Assessment of Spoofing Countermeasures and Automatic Speaker Verification: Fundamentals”  In _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ 28, 2020, pp. 2195–2210  DOI: 10.1109/TASLP.2020.3009494
  * [10] Christophe Veaux, Junichi Yamagishi and Kirsten MacDonald  “CSTR VCTK Corpus: English Multi-speaker Corpus for CSTR Voice Cloning Toolkit” http://dx.doi.org/10.7488/ds/1994  University of Edinburgh. The Centre for Speech Technology Research (CSTR), 2017
  * [11] “Deepfakes – The Good, The Bad, And The Ugly” Accessed: 2022-09-10, https://www.forbes.com/sites/bernardmarr/2022/01/11/deepfakes--the-good-the-bad-and-the-ugly/?sh=4a5516f24f76
  * [12] “Fact check: The deepfakes in the disinformation war between Russia and Ukraine” Accessed: 2022-09-10, https://www.dw.com/en/fact-check-the-deepfakes-in-the-disinformation-war-between-russia-and-ukraine/a-61166433
  * [13] Junichi Yamagishi et al.  “ASVspoof 2021: accelerating progress in spoofed and deepfake speech detection”  In _Proc. ASVspoof Challenge workshop_ , 2021, pp. 47–54  DOI: 10.21437/ASVSPOOF.2021-8
  * [14] Héctor Delgado et al.  “ASVspoof 2021: Automatic speaker verification spoofing and countermeasures challenge evaluation plan”  In _arXiv:2109.00535_ , 2021
  * [15] Hector Delgado et al.  “Impact of Bandwidth and Channel Variation on Presentation Attack Detection for Speaker Verification”  In _Proc. BIOSIG_ , 2017, pp. 1–6  DOI: 10.23919/BIOSIG.2017.8053510
  * [16] Galina Lavrentyeva et al.  “Phonespoof: A New Dataset for Spoofing Attack Detection in Telephone Channel”  In _Proc. ICASSP_ , 2019, pp. 2572–2576  DOI: 10.1109/ICASSP.2019.8682942
  * [17] Xin Wang et al.  “ASVspoof 2019: A large-scale public database of synthesized, converted and replayed speech”  In _Computer Speech & Language_ 64, 2020, pp. 101114  DOI: 10.1016/j.csl.2020.101114
  * [18] Andreas Nautsch et al.  “ASVspoof 2019: Spoofing Countermeasures for the Detection of Synthesized, Converted and Replayed Speech”  In _IEEE Transactions on Biometrics, Behavior, and Identity Science_ 3.2, 2021, pp. 252–265  DOI: 10.1109/TBIOM.2021.3059479
  * [19] PreSonus  “Sceptre S8 Studio Monitor Tech Sheet”  URL: https://pae-web.presonusmusic.com/downloads/products/pdf/Sceptre_S8_Studio_Monitor_techsheet.pdf
  * [20] Jaime Lorenzo-Trueba et al.  “The Voice Conversion Challenge 2018: Promoting Development of Parallel and Nonparallel Methods”  In _Proc. Odyssey_ , 2018, pp. 195–202  DOI: 10.21437/Odyssey.2018-28
  * [21] Zhao Yi et al.  “Voice Conversion Challenge 2020 –- Intra-lingual semi-parallel and cross-lingual voice conversion”  In _Proc. Joint Workshop for the Blizzard Challenge and Voice Conversion Challenge 2020_ , 2020, pp. 80–98  DOI: 10.21437/VCC˙BC.2020-14
  * [22] Gautham J. Mysore  “Can we Automatically Transform Speech Recorded on Common Consumer Devices in Real-World Environments into Professional Production Quality Speech? – A Dataset, Insights, and Challenges”  In _IEEE Signal Processing Letters_ 22.8, 2015, pp. 1006–1010  DOI: 10.1109/LSP.2014.2379648
  * [23] Mirjam Weste  “The EMIME Bilingual Database”, 2010
  * [24] Xinhui Zhou et al.  “Linear versus mel frequency cepstral coefficients for speaker recognition”  In _Proc. ASRU_ , 2011, pp. 559–564  DOI: 10.1109/ASRU.2011.6163888
  * [25] Jialong Li et al.  “Long-term variable Q transform: A novel time-frequency transform algorithm for synthetic speech detection”  In _Digital Signal Processing_ 120, 2022, pp. 103256  DOI: https://doi.org/10.1016/j.dsp.2021.103256
  * [26] Jie Hu et al.  “Squeeze-and-Excitation Networks”  In _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 42.8, 2020, pp. 2011–2023  DOI: 10.1109/TPAMI.2019.2913372
  * [27] Le Yang et al.  “CondenseNet V2: Sparse Feature Reactivation for Deep Networks”  In _Proc. CVPR_ , 2021, pp. 3568–3577  DOI: 10.1109/CVPR46437.2021.00357
  * [28] Bhusan Chettri, Emmanouil Benetos and Bob L.. Sturm  “Dataset Artefacts in Anti-Spoofing Systems: A Case Study on the ASVspoof 2017 Benchmark”  In _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ 28, 2020, pp. 3018–3028  DOI: 10.1109/TASLP.2020.3036777
  * [29] Yuxiang Zhang, Wenchao Wang and Pengyuan Zhang  “The Effect of Silence and Dual-Band Fusion in Anti-Spoofing System”  In _Proc. Interspeech_ , 2021, pp. 4279–4283  DOI: 10.21437/Interspeech.2021-1281
  * [30] Nicolas Müller et al.  “Speech is Silver, Silence is Golden: What do ASVspoof-trained Models Really Learn?”  In _Proc. ASVspoof Challenge workshop_ , 2021, pp. 55–60  DOI: 10.21437/ASVSPOOF.2021-9
  * [31] Tomi Kinnunen and Haizhou Li  “An overview of text-independent speaker recognition: From features to supervectors”  In _Speech communication_ 52.1 Elsevier, 2010, pp. 12–40  DOI: 10.1016/j.specom.2009.08.009
  * [32] Hemlata Tak et al.  “End-to-End anti-spoofing with RawNet2”  In _Proc. ICASSP_ , 2021, pp. 6369–6373  DOI: 10.1109/ICASSP39728.2021.9414234
  * [33] Jee-weon Jung et al.  “Improved RawNet with Feature Map Scaling for Text-Independent Speaker Verification Using Raw Waveforms”  In _Proc. Interspeech_ , 2020, pp. 1496–1500  DOI: 10.21437/Interspeech.2020-1011
  * [34] Jee-weon Jung et al.  “AASIST: Audio Anti-Spoofing Using Integrated Spectro-Temporal Graph Attention Networks”  In _Proc. ICASSP_ , 2022, pp. 6367–6371  DOI: 10.1109/ICASSP43922.2022.9747766
  * [35] Hemlata Tak et al.  “Automatic Speaker Verification Spoofing and Deepfake Detection Using Wav2vec 2.0 and Data Augmentation”  In _Proc. Odyssey_ , 2022, pp. 112–119  DOI: 10.21437/Odyssey.2022-16
  * [36] Xin Wang and Junichi Yamagishi  “Investigating Self-Supervised Front Ends for Speech Spoofing Countermeasures”  In _Proc. Odyssey_ , 2022, pp. 100–106  DOI: 10.21437/Odyssey.2022-14
  * [37] Juan M. Martín-Doñas and Aitor Álvarez  “The Vicomtech Audio Deepfake Detection System Based on Wav2vec2 for the 2022 ADD Challenge”  In _Proc. ICASSP_ , 2022, pp. 9241–9245  DOI: 10.1109/ICASSP43922.2022.9747768
  * [38] Ariel Cohen et al.  “A study on data augmentation in voice anti-spoofing”  In _Speech Communication_ 141, 2022, pp. 56–67  DOI: https://doi.org/10.1016/j.specom.2022.04.005
  * [39] Hemlata Tak et al.  “Rawboost: A Raw Data Boosting and Augmentation Method Applied to Automatic Speaker Verification Anti-Spoofing”  In _Proc. ICASSP_ , 2022, pp. 6382–6386  DOI: 10.1109/ICASSP43922.2022.9746213
  * [40] Juan M. Martín-Doñas and Aitor Álvarez  “The Vicomtech Audio Deepfake Detection System Based on Wav2vec2 for the 2022 ADD Challenge”  In _Proc. ICASSP_ , 2022, pp. 9241–9245  DOI: 10.1109/ICASSP43922.2022.9747768
  * [41] Abderrahim Fathan, Jahangir Alam and Woo Hyun Kang  “Mel-Spectrogram Image-Based End-to-End Audio Deepfake Detection Under Channel-Mismatched Conditions”  In _Proc. ICME_ , 2022, pp. 1–6  DOI: 10.1109/ICME52920.2022.9859621
  * [42] Daniel S. Park et al.  “SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition”  In _Proc. Interspeech_ , 2019, pp. 2613–2617  DOI: 10.21437/Interspeech.2019-2680
  * [43] Jiangyan Yi et al.  “ADD 2022: the first Audio Deep Synthesis Detection Challenge”  In _Proc. ICASSP_ , 2022, pp. 9216–9220  DOI: 10.1109/ICASSP43922.2022.9746939
  * [44] Lin Zhang et al.  “The PartialSpoof Database and Countermeasures for the Detection of Short Generated Audio Segments Embedded in a Speech Utterance”  In _ArXiv_ abs/2204.05177, 2022
  * [45] Jee-weon Jung et al.  “SASV 2022: The First Spoofing-Aware Speaker Verification Challenge”  In _Proc. Interspeech_ , 2022
  * [46] Erica Cooper et al.  “Zero-shot multi-speaker text-to-speech with state-of-the-art neural speaker embeddings”  In _Proc. ICASSP_ , 2020, pp. 6184–6188
  * [47] Sung-Feng Huang et al.  “Meta-TTS: Meta-learning for few-shot speaker adaptive text-to-speech”  In _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ 30, 2022, pp. 1558–1571
  * [48] “YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for Everyone”  In _Proc. ICML_ , 2022, pp. 2709–2720
  * [49] Jesús Villalba, Yuekai Zhang and Najim Dehak  “x-Vectors Meet Adversarial Attacks: Benchmarking Adversarial Robustness in Speaker Verification”  In _Proc. Interspeech_ , 2020, pp. 4233–4237  DOI: 10.21437/Interspeech.2020-2458
  * [50] Sonal Joshi et al.  “Study of Pre-Processing Defenses Against Adversarial Attacks on State-of-the-Art Speaker Recognition Systems”  In _IEEE Transactions on Information Forensics and Security_ 16, 2021, pp. 4811–4826  DOI: 10.1109/TIFS.2021.3116438

## Supplementary material

TABLE IX: Details on microphone and replay devices used in PA data recording. Numbers are from official specifications. Replay device | s2 | s3 | s4
---|---|---|---
Brand | SONY | NEUMANN | GENELC
SRS-XB43 | KH 80 DSP | 8030 CP
Lower cut-off freq. (–6 dB) in Hz | 20 | 53 | 47
Upper cut-off freq. (–6 dB) in Hz | 20k | 22k | 25k
Passband free-field FR ($\pm$ 2dB) in Hz | UNAVAL. | 59 - 20k | 54 - 20k
|  |  |
Microphone | M1/m1 | M2/m2 | M3/m3
Brand | Marantz | M-Audio | iPad Air
MPM-1000 | Uber-mic | MEMS mic.
FR in Hz | 20 - 20k | 30 - 20k | UNAVAIL.
SNR in dB | 77 | 110 | UNAVAIL.
Sensitivity in dB | 45 | 38 ($\pm$ 2) | UNAVAIL.

###  VII-A Devices used in PA data recording

The replay and recording devices used to create PA data is shown in Table. IX.
All setups are cited from the specifications available at the official
websites.

###  VII-B Description of Top-Performing Systems

The top-5 systems for each scenario are ranked and selected respectively based
on minimum t-DCF. The architectures are presented in terms of block diagrams
which were drawn based on the system descriptions and feedback from the
challenge participants.

For a unified presentation, we break down each system into four cascaded
processing blocks:

  * •

Audio pre-processor includes the steps to partition data and process input
wave files. Most of the methods in this step are applied only at training
phase.

  * •

Input processor contains the acoustic frontends used for further processing.
Learnable acoustic frontends are also categorized to this part.

  * •

Classifier covers the neural network as well as statistical models for the
countermeasure system. Online augmentation methods and training strategies (if
any) are also covered here.

  * •

Post-processor contains score-domain operations, including score fusion and
normalization.

In the following (starting from next page) we give short descriptions to the
architecture of the illustrated systems.

Figure 9: Illustration of top-5 systems for the LA scenario. The descending
order is based on performance via minimum t-DCF values on evaluation set.
Numbers in brackets are the values. Dash lines indicate steps/modules which
are only applied for training, while solid lines indicate ones applied for
both training and testing.

####  VII-B1 Logical Access (LA)

T23: Multiple sub-systems are trained based on the codec-augmented and trimmed
audio. Most of them are based on various types spectral features and two
parallel classifiers, LightCNN (LCNN) and ResNet. They are forwarded to LSTMs
and the scores are summed up with weights. Further, another sub-system uses a
SincNet frontend and an LSTM backend. The output score of this subsystem is
further fused with the weighted averaged score above.

T35: The input audio is processed using pre-emphasis and a-law companding
algorithm. LFCC features are extracted from the processed audio and fed into
two ResNet-based classifiers. The output scores from the two classifiers are
averaged.

T19: The training data is augmented via RIR and MUSAN datasets, followed by
trimming and additive noise. The mel spectrograms are fed into a ResNet with
squeeze-and-excitation in its building blocks. The output scores are averaged
via pre-defined empirical weights.

T36 is a combination of multiple classifiers based on RawNet and ECAPA-TDNN.
The training data is augmented via additive noise and 3-fold speed
perturbation with parameters of 0.9, 1.0 and 1.1. Three types of feature
extractors are used — raw waveform, mel spectrogram and a learnable LEAF
frontend, whose outputs are fed into one RawNet and two ECAPA-TDNNs. The
output scores are combined using equal weights.

T20 uses a ResNet classifier trained with log filterbanks. The training data
is augmented via RIR, MUSAN and media codecs such as mp3 and m4a. The log
filterbanks are augmented via zero-value masking at frequency domain
(FreqAugment). A fully-connected layer is added after the ResNet and trained
with _large margin cosine loss_ (LMCL).

Figure 10: Illustration of top-5 systems for the PA scenario. The descending
order is based on performance via minimum t-DCF values on evaluation set.
Numbers in brackets are the values. Dash lines indicate steps/modules which
are only applied for training, while solid lines indicate ones applied for
both training and testing.

####  VII-B2 Physical Access (PA)

T07 is an one-class learning framework based on _Gaussian mixture models_
(GMM) and _variation autoencoder_ (VAE). The training data is augmented via
room simulation 171717via _pyroomacoustics_ :
https://github.com/LCAV/pyroomacoustics and 2-fold speed perturbation with
parameters of 0.9 and 1.1. The acoustic feature used is log spectrogram, which
is fed into a neural vocoder to create filtered spectrograms. Applied
candidates for the vocoder are WORLD, HifiGAN, and MelGAN. The original and
processed spectrograms, which are time-frequency representations, are averaged
via the temporal axis. One GMM and one VAE are trained via the temporal-
averaged spectrograms and another GMM is trained via the original log
spectrogram and the filtered spectrogram. The output scores are combined via
equally weighting.

T16 uses a combination of SEnet and GMM trained with various short-term
spectral features. The data is augmented via in-house replay data. The scores
are fused using empirically weighted averaging.

T23: A parallel combination of multiple sub-systems trained on trimmed speech
segments, augmented via RIR, MUSAN and other additive noises. Mel spectrogram
with various number of frequency bins and LEAF frontend are the feature
extractors. Classifiers are implemented via ResNet, whose output are forwarded
to scors via _attentive statistical pooling_ (ASP). Final score is obtained
via empirically weight averaging.

T01: The pre-emphasized audio is input to produce three different types of
acoustic features: LFCCs, magnitude spectrogram (MagSpec), and MFCCs. Each of
them is fed into a LightCNN classifier and the output score is obtained by
fusing the output of the three classifiers using equal weights.

T04: A TDNN network with statistical pooling. The pre-emphasized audio is used
to produce another three types of features: MFCCs, CQCCs and CQT spectrogram.

Figure 11: Illustration of top-5 systems for the DF scenario. The descending
order is based on performance via equal error rates on evaluation set. Numbers
in brackets are the error rates. Dash lines indicate steps/modules which are
only applied for training, while solid lines indicate ones applied for both
training and testing.

####  VII-B3 DeepFake (DF)

T23 is a parallel hybrid of three classifiers. Two acoustic frontends process
the trimmed and codec-augmented audio: Mel spectrogram extractor and a
convolutive frontend built via sinc function. The former is fed into a ResNet
and a LightCNN and the latter is fed into a RawNet. The outputs of the three
nets is feed into separate bi-directional LSTMs to produce scores, which are
finally fused via empirically weight averaging.

T20: The feature extractor, classifier and scoring scheme is the same as one
used by the same team in the LA scenario. The training data is augmented via
media codecs such as mp3 and m4a, and other additive noises.

T08 fuses multiple LightCNN classifiers using LFCCs and CQT spectrograms. The
input data is augmented using multiple media and transmission
codecs181818G722, alaw, G723, G726, gsm, opus, SPEEX, mp3, m4a, ogg, mp2.
LFCCs are used to train one LightCNN and the remaining classifiers are trained
on the other feature. The scores are processed via min-max normalization,
before being averaged.

T06 is a combination of a TDNN classifier trained on LFCCs, four SEnet modules
trained on LFCCs, two GMMs trained on _product spectral cepstral coefficients_
(PSCCs), and three TDNNs trained separately on the two mentioned features and
_log linear filterbank energies_ (LLFBs).

T22 is a convolutional network based on CondenseV2Net, trained on log power
spectrum derived from _long-term variable Q transform_ (L-VQT). The training
data is augmented via mp3, m4a, wma and trimmed to 4-second chunks.

###  VII-C Hidden Track Results

The comparison of EERs for evaluation set and hidden track are presented in
Fig. 12. We can see that for LA and DF scenarios, while for the main track the
EERs for the top systems are all lower than 10%, for the hidden track the
number raised to more than 20%. Such pattern can also be found for the PA
scenario, with narrower performance gap between the main and hidden tracks.

The results indicate the effect of silence on the spoofing detection.
Meanwhile, the different gap between PA and the other two scenarios may due to
the artifact on the length of silence injected in the spoofed and bonafide
training data — for LA and DF, where synthetic speeches are produced via TTS
systems, such difference is large. For PA where the attack comes from
replaying the audio, such difference becomes smaller.

###  VII-D Challenge Overall Results

We present full statistical results on progress and evaluation sets in Table
X, XI and XII, for LA, PA and DF scenarios respectively. Same as last section,
the team IDs are kept consistent with ones used in the main paper. The IDs
starting with ‘B’ are IDs of the baseline systems provided by us, for
comparability concern. For LA and PA we present both t-DCF and EER results,
while for DF we report EERs.

(a) LA (b) PA (hidden track 1) (c) PA (hidden track 2) (d) DF

Figure 12: Comparison of CMs EERs on evaluation set and hidden tracks. Blue
dots correspond to top 1-10 submissions. Red dots are other submissions. Best
viewed in color.

TABLE X: ASVspoof 2021 progress and evaluation results for LA. Results shown in terms of pooled normalised minimum t-DCF and pooled EER [%]. |  | Progress set | Evaluation set
---|---|---|---
# | ID | t-DCF | EER | t-DCF | EER
1 | T23 | 0.1816 | 0.89 | 0.2176 | 1.32
2 | T35 | 0.2115 | 2.61 | 0.2480 | 2.77
3 | T19 | 0.2175 | 2.69 | 0.2495 | 3.13
4 |  | 0.2119 | 2.51 | 0.2500 | 2.81
5 | T36 | 0.2373 | 3.69 | 0.2531 | 3.10
6 | T20 | 0.2137 | 2.39 | 0.2608 | 3.21
7 | T08 | 0.2376 | 3.23 | 0.2672 | 3.62
8 | T16 | 0.2394 | 3.39 | 0.2689 | 3.63
9 |  | 0.2436 | 3.11 | 0.2725 | 3.61
10 | T04 | 0.2372 | 4.54 | 0.2747 | 5.58
11 | T06 | 0.2476 | 5.61 | 0.2853 | 5.66
12 |  | 0.2556 | 4.22 | 0.2880 | 5.01
13 | T03 | 0.2461 | 3.65 | 0.2882 | 4.66
14 |  | 0.2556 | 5.90 | 0.2893 | 5.70
15 | T31 | 0.2622 | 5.00 | 0.3094 | 5.46
16 | T17 | 0.2990 | 7.10 | 0.3279 | 7.19
17 | T07 | 0.3029 | 7.88 | 0.3310 | 8.23
18 | T30 | 0.2999 | 7.58 | 0.3362 | 8.89
19 | B03 | 0.3152 | 8.90 | 0.3445 | 9.26
20 | T02 | 0.3220 | 7.71 | 0.3445 | 7.79
21 | T14 | 0.3167 | 8.82 | 0.3451 | 8.98
22 | T11 | 0.3015 | 6.80 | 0.3666 | 7.19
23 | T34 | 0.3688 | 13.25 | 0.4059 | 13.45
24 | B04 | 0.4152 | 9.49 | 0.4257 | 9.50
25 | T15 | 0.2986 | 9.67 | 0.4890 | 14.68
26 | B01 | 0.4948 | 15.80 | 0.4974 | 15.62
27 | T25 | 0.2086 | 2.03 | 0.5148 | 13.75
28 | T32 | 0.4256 | 10.93 | 0.5270 | 12.90
29 |  | 0.5887 | 19.04 | 0.5748 | 18.50
30 | B02 | 0.5836 | 21.13 | 0.5758 | 19.30
31 |  | 0.2528 | 5.26 | 0.5775 | 14.28
32 | T01 | 0.2773 | 6.32 | 0.6204 | 15.95
33 |  | 0.2792 | 7.87 | 0.6288 | 15.87
34 | T24 | 0.3108 | 10.85 | 0.6320 | 15.98
35 |  | 0.3233 | 9.72 | 0.6370 | 16.27
36 | T29 | 0.3316 | 11.87 | 0.6741 | 17.41
37 |  | 0.3679 | 12.54 | 0.6812 | 17.66
38 | T12 | 0.6900 | 26.26 | 0.7227 | 26.41
39 |  | 0.4382 | 13.77 | 0.7233 | 19.19
40 |  | 0.6123 | 23.16 | 0.8521 | 26.14
41 |  | 0.9988 | 53.16 | 1.0000 | 53.81
TABLE XI: ASVspoof 2021 progress and evaluation results for PA. Results shown in terms of pooled normalised minimum t-DCF and pooled EER [%]. |  | Progress set | Evaluation set
---|---|---|---
# | ID | t-DCF | EER | t-DCF | EER
1 | T07 | 0.6736 | 23.60 | 0.6824 | 24.25
2 | T16 | 0.6790 | 25.37 | 0.7122 | 27.59
3 | T23 | 0.7019 | 24.22 | 0.7296 | 26.42
4 | T01 | 0.6925 | 25.73 | 0.7446 | 28.36
5 | T04 | 0.6676 | 24.80 | 0.7462 | 29.00
6 |  | 0.7207 | 27.23 | 0.7469 | 29.22
7 | T33 | 0.7263 | 27.00 | 0.7648 | 29.55
8 | T08 | 0.7177 | 25.95 | 0.7670 | 29.02
9 | T37 | 0.7912 | 32.39 | 0.8216 | 35.07
10 | T27 | 0.8102 | 31.62 | 0.8307 | 32.00
11 | T26 | 0.7837 | 26.91 | 0.8362 | 29.61
12 | T28 | 0.8498 | 30.94 | 0.8879 | 32.96
13 |  | 0.8934 | 35.16 | 0.9265 | 37.10
14 | B01 | 0.9062 | 36.33 | 0.9434 | 38.07
15 | T03 | 0.9084 | 36.37 | 0.9444 | 38.07
16 |  | 0.9193 | 35.78 | 0.9530 | 38.50
17 | T09 | 0.9531 | 34.39 | 0.9666 | 34.77
18 | B02 | 0.9747 | 39.79 | 0.9724 | 39.54
19 | T11 | 0.9836 | 40.91 | 0.9939 | 42.55
20 |  | 0.9855 | 40.38 | 0.9945 | 42.98
21 | B03 | 0.9827 | 42.16 | 0.9958 | 44.77
22 | B04 | 0.9993 | 46.03 | 0.9997 | 48.60
23 | T10 | 0.9996 | 45.10 | 1.0000 | 45.50
TABLE XII: ASVspoof 2021 progress and evaluation results for DF. Results shown in terms of pooled normalised minimum t-DCF and pooled EER [%]. |  | Progress set | Evaluation set
---|---|---|---
# | ID | EER | EER
1 | T23 | 0.24 | 15.64
2 | T20 | 1.79 | 16.05
3 | T08 | 2.93 | 18.30
4 |  | 1.65 | 18.80
5 | T06 | 8.29 | 19.01
6 | T22 | 2.17 | 19.22
7 | T03 | 2.32 | 19.24
8 | T01 | 4.72 | 19.70
9 | T36 | 0.78 | 20.23
10 | T19 | 3.00 | 20.33
11 | T31 | 6.88 | 20.33
12 |  | 2.25 | 20.63
13 |  | 2.00 | 20.82
14 | T16 | 1.98 | 20.84
15 |  | 4.84 | 21.61
16 |  | 5.06 | 21.67
17 |  | 2.37 | 22.03
18 |  | 6.10 | 22.38
19 | B04 | 6.10 | 22.38
20 | T25 | 4.18 | 22.62
21 | B03 | 11.61 | 23.48
22 |  | 0.10 | 23.57
23 |  | 5.36 | 23.88
24 | T35 | 0.42 | 24.12
25 |  | 11.82 | 24.89
26 | T30 | 4.66 | 25.21
27 | B02 | 21.01 | 25.25
28 | T26 | 16.40 | 25.41
29 | B01 | 17.63 | 25.56
30 |  | 3.40 | 26.67
31 | T21 | 17.20 | 28.96
32 |  | 9.07 | 29.25
33 |  | 29.63 | 29.75

###  VII-E Useful Resources

We provide useful resources for the challenge participants as well as the
research community to reproduce the baselines and conduct their own
experiments.

  * •

Link to the datasets, in a per-scenario manner:
LA - https://zenodo.org/record/4837263,
PA - https://zenodo.org/record/4834716,
DF - https://zenodo.org/record/4835108.

  * •

Link to the keys and meta-labels, in a per-scenario manner:
LA - https://www.asvspoof.org/asvspoof2021/LA-keys-full.tar.gz,
PA - https://www.asvspoof.org/asvspoof2021/PA-keys-full.tar.gz,
DF - https://www.asvspoof.org/asvspoof2021/DF-keys-full.tar.gz.

  * •

The challenge baseline repository, which is open-sourced with ASV scores,
baseline CM systems and the evaluation package. One can visit
https://github.com/asvspoof-challenge/2021 for more details.