# Evince the artifacts of Spoof Speech by blending Vocal Tract and Voice
Source Features

Tadipatri Uday Kiran Reddy , Sahukari Chaitanya Varun, Kota Pranav Kumar
Sankala Sreekanth, Kodukula Sri Rama Murty
Indian Institute of Technology Hyderabad, Telangana, India
{ee19btech11038, ee19bech11040, ee19btech11051, ee20resch11011, ksrm}
@iith.ac.in

###### Abstract

With the rapid advancement in synthetic speech generation technologies, great
interest in differentiating spoof speech from the natural speech is emerging
in the research community. The identification of these synthetic signals is a
difficult task not only for the cutting-edge classification models but also
for humans themselves. To prevent potential adverse effects, it becomes
crucial to detect spoof signals. From a forensics perspective, it is also
important to predict the algorithm which generated them to identify the
forger. This needs an understanding of the underlying attributes of spoof
signals which serve as a signature for the synthesizer. This study emphasizes
the segments of speech signals critical in identifying their authenticity by
utilizing the Vocal Tract System(VTS) and Voice Source(VS) features.

In this paper, we propose a system that detects spoof signals as well as
identifies the corresponding speech-generating algorithm. We achieve 99.58% in
algorithm classification accuracy. From experiments, we found that a VS
feature-based system gives more attention to the transition of phonemes,
while, a VTS feature-based system gives more attention to stationary segments
of speech signals. We perform model fusion techniques on the VS-based and VTS-
based systems to exploit the complementary information to develop a robust
classifier. Upon analyzing the confusion plots we found that WaveRNN is poorly
classified depicting more naturalness. On the other hand, we identified that
synthesizer like Waveform Concatenation, and Neural Source Filter is
classified with the highest accuracy. Practical implications of this work can
aid researchers from both forensics (leverage artifacts) and the speech
communities (mitigate artifacts).

###### Index Terms:

Synthetic Speech Attribution, X-vector, Bicoherence, LP Residual,
Coarticulation

##  I Introduction

Speech synthesis is the process of generating natural speech using machines.
With the rapid improvements in deep learning technologies, speech synthesis
algorithms have remarkably improved the genuineness of synthesized speech. The
advances in speech synthesis algorithms made them deploy in various
applications - aiding vocally handicapped, assisting vision-impaired humans
through screen readers, voice dubbing in multi-media applications, and human-
machine interactions, etc. Although speech synthesis algorithms have abundant
usage in improving the quality of life, they pose a severe threat in forensic
science if used negatively. For example, the advancements in speech synthesis
algorithms severely threaten the usage of biometric voice systems—especially
the voice biometrics deployed in machine-critical areas such as financial
transactions and forensic applications. The forger may exploit the speech
synthesis systems to create illegal audio copies to defame public figures. The
recent occurrence of mishaps [1, 2] illustrates the adverse consequences
caused by voice impersonators. Following the threats of synthesis systems, it
is vital to study the defence strategies that develop an algorithm to
discriminate synthetic speech from natural speech. Correspondingly, there have
also been many studies that investigate the usage of different features that
could be used for differentiating spoof signals from natural signals. Some
studies suggest that features consisting of high-frequency information have
pronounced effects on spoof speech [3, 4].
Although the recent models perform reasonably well in discriminating synthetic
speech from natural speech, they lack an interpretation of the artifacts
captured by the model. Finding the artifacts captured by the model to detect
the fake speech will aid forensic experts in narrowing down their search area
for finding the synthesizer. Detecting the underlying synthesizer helps
identify the falsifier. It also helps reduce models’ complexity by explicitly
capturing the possible artifacts in the feature extraction. Moreover, the
synthetic speech groups could focus more on them to improve the naturalness of
the generated speech.

Naturally, speech is produced by modulating the expelled air from the lungs
through the movements of articulators (nose, mouth, etc.). Hence the air from
the lungs acts as a source of energy, and articulators act as a modulator to
produce intelligible sounds called phonemes. Natural variations are inevitable
in delivering speech sounds. Majorly the natural variations are introduced at
the system and source levels while generating the speech. Speech synthesis
algorithms aim to mimic this natural system and may fail to synthesize the
subtle variations introduced by the human speech production system.

In that direction, the authors of [5] claimed that there exist certain
phonemes that synthesized models are not able to generate precisely. They
empirically showed that synthetic systems fail to generate fricatives
(/S/,/SH/), nasals (/M/,/N/), and vowels (/Y/) compared to the other phonemes.
Similarly, the work in [6] shows that the silence regions contain
discriminative patterns to classify synthetic speech from natural speech. They
found that synthetic models face difficulty in synthesizing the statistically-
realistic silence intervals.

In the literature, various features are used to capture the artifacts
introduced at the Vocal Track System (VTS) and Voice Source (VS). Spectral
magnitude features (Log-Mel filter bank energies) are mainly used to capture
VTS characteristics. To analyze VS, Linear Prediction (LP) residuals are used
extensively. Many works [7, 8, 9, 10, 11] have shown the complementary
information present on these features. Hence, to capture and improve the
performance of synthetic systems, it is essential to explore both features.

This work explores spectral magnitude features ( Log-Mel filter bank energies
) and LP residuals to perform synthetic speech algorithm classification. As
the spectrum of LP residuals is relatively flat, we have proposed an end-to-
end feature extraction module to extract more discriminative patterns from the
LP residuals. The spectral magnitude features from the speech signal or the
end-to-end features learned from the LP residual are used with the X-Vector
[12] framework to perform synthetic speech classification. Further, the
attention layer in the X-Vector architecture is used to analyze the artifacts
learned by the model in discriminating the synthetic speech. We found that the
synthetic speech detection systems trained with different features tend to
attend to various artifacts of synthesis algorithms. Specifically, the VTS
features tend to attend more to stationary speech segments. On the other hand,
the VS features tend to attend more to the co-articulation variations, i.e.,
phoneme transitions. Later, the subsystems built on different features are
fused at different architecture levels to improve classification accuracy.
Finally, we identify two sets of algorithms that perform worst using our
system, indicating more naturalness and, on the contrary, a few which are
perfectly classified, i.e., more synthetic. Implications of this work can aid
researchers from forensics (leveraging artifacts) and speech synthesis
(mitigating the artifacts).

The rest of the paper is organized as follows. Section $\mathrm{II}$ describes
the source-system model of the speech production mechanism and motivates the
need to extract VTS features and VS features. Section $\mathrm{III}$ describes
the details of the VTS feature extraction. Section $\mathrm{IV}$ describes the
DNN architecture of a synthetic speech classifier. Section $\mathrm{V}$
describes the VS feature extraction and DNN models to extract high-level
representations from it. Subsequently, Section $\mathrm{VI}$ contains
information about the experimental setup, including the implementation details
and datasets used. Section $\mathrm{VII}$ discusses the results of the models
used and includes details of the experiments conducted by fusion. In addition,
it also contains a comparison of the proposed models and their variants.
Section $\mathrm{VII}$ analyzes the artifacts learned by the synthetic speech
classification models by exploring the attention modules and proposes a method
to fuse the subsystems to improve the performance. Finally, $\mathrm{VIII}$
concludes the paper with a summary of contributions and the future scope of
the work.

##  II Motivation for feature extraction

In this section, we briefly describe one of the speech production models known
as the Source-System model.

In the literature, many researchers have proposed models [13, 14, 15, 16, 17]
to describe the speech production system and model the non-linearities.
However, in this study, we restrict ourselves to Source-System Model as
described in the paper [18]. This model assumes linearity and that the source
and system work independently to produce speech.

$\displaystyle s[n]=v[n]*e[n]$ (1)

Where $s[n]$, $v[n]$, and $e[n]$ are the speech signal, impulse response of
the vocal tract filter, and excitation signal. The speech signal is the output
of the time-varying digital filter ($v[n]$) with time-varying excitation
($e[n]$) [19], which can be mathematically modeled as Eq 1.

Figure 1: Histogram of Local Jitter and Local Shimmer.

Figure 2: Feature extraction from linear prediction residuals

Figure 3: Synthetic speech classification architecture

Synthetic speech algorithms seek to mimic this natural system, where they may
fail to incorporate subtle features of speech production, such as

  * •

Prosodic variations such as jitter, shimmer:

Jitter is a parameter to capture frequency variations in a time period as
described in Eq 2. Here $T_{i}$ is the duration of a period and $N$ is the
number of periods.

$\text{Jitter (in
\%)}=100\frac{N}{N-1}\frac{\sum_{i=1}^{N-1}|T_{i}-T_{i+1}|}{\sum_{i=1}^{N}T_{i}}$
(2)

Shimmer is a parameter to capture amplitude variations in a time period as
described in Eq 3. Here $A_{i}$ is the amplitude in period and $N$ is the
number of periods.

$\text{Shimmer (in
\%)}=100\frac{N}{N-1}\frac{\sum_{i=1}^{N-1}|A_{i}-A_{i+1}|}{\sum_{i=1}^{N}A_{i}}$
(3)

The excitation source of synthetic speech is more periodic and less noisy;
therefore less jitter and shimmer in the impulse train, as pointed out in the
paper [20]. Among synthetic speech algorithms, there could be distinct
patterns in these impulse trains, causing variations in jitter and shimmer.
Figure 1 shows the histogram of Local Jitter and Local Shimmer of different
synthetic speech signals. It is evident that jitter and shimmer variations
among these algorithms are different, which gives us the motivation to look at
the features that capture such information.

  * •

Co-articulation of phoneme sequences:

Coarticulation is the context-dependent variation of phonemes’ articulatory
and acoustic realization, especially of consonants. The generation of highly
natural and intelligible speech requires the simulation of realistic
articulatory movements. Phoneme transitions in synthetic speeches tend to be
sharper than that observed in natural peech[21, 22, 23]. Hence, we could
emphasize the coarticulation capturing features for the classification problem
to attribute as discriminatory evidence for the given synthesizer.

In the following sections, we discuss the features which could potentially
capture the artifacts in spoof speech; Log Mel Spectrogram, capturing VTS
features, and Linear Predictive Residuals, capturing VS features.

##  III Vocal Tract System Features: Log-Mel filter bank energies

The Vocal Tract System can be considered a concatenation of acoustic tubes.
The resonances of the vocal tract manifest as formant frequencies in the
spectrogram of the speech signal. Hence capturing the formant frequencies of
the speech signal can convey information about the vocal tract system used in
the speech production process. Many works [24, 25, 26, 27, 28] utilize the
MFCC (mel-frequency cepstral coefficients), and LPCC (linear prediction
cepstral coefficients) features to capture the magnitude information.

Log Mel filter bank energies are also one of the popular ways to extract
information about the vocal tract system[29]. The extracted energies consist
of up to 2nd-order statistics conveying vocal tract features in the speech
production model.

##  IV Synthetic speech classification - X-vector

This section provides the details of the synthetic speech classification
architecture. X-vector framework [12] originally proposed for speaker
classification is explored to classify synthetic speech from natural speech.
As shown in Figure 3, the X-vector framework consists of three submodules: (i)
Frame level encoder, (ii) Multi-head self-attentive pooling layer, and (iii)
segment level encoder. Frame level encoder comprises 1D convolutional layers
and is framed to extract high-level features representing the artifacts
introduced by the synthetic algorithms. Multi-head self-attentive pooling
layer attentively pools the information from variable-length high-level
representations into a fixed dimensional embedding. Finally, a set of Dense
layers in the segment-level encoder outputs the logits of the classes.
Finally, the cross-entropy loss function is used to train the network.

###  IV-A Multi-Head Self-Attention

In this work, we assume that all the frames may not be equally important to
discriminate the synthetic algorithms. The results in [5] validate the
assumption by showing certain phonemes (/S/,/SH/,/M/,/N/) are relatively more
important than other phonemes. Hence we have used multi-head self-attention
modules to highlight the speech regions that are more discriminated among the
synthetic algorithms. In further sections, we identify the artifacts of the
synthetic algorithms by plotting these attention values over the speech
signal, as shown in Figure 5.

Let $(\textbf{x}_{i})_{i=1}^{T}\in\mathbb{R}^{n}$ be the sequence of feature
vectors extracted from the input speech signal and $f:\mathbb{R}^{n\times
T}\mapsto\mathbb{R}^{d\times T^{\prime}}$ be the frame level encoder function
in the X-vector framework. Frame-level encoder extracts high-level
representations from the sequence of features vectors

$\textbf{h}_{1},\textbf{h}_{2},...,\textbf{h}_{T^{\prime}}=f(\textbf{x}_{1},\textbf{x}_{2},...,\textbf{x}_{T_{i}})$
(4)

Where $(\textbf{h}_{i})_{i=1}^{T^{\prime}}\in\mathbb{R}^{d}$ are the high-
level representations extracted from the frame level encoder. Time axis
dimension $T^{\prime}$ depends on the effective receptive field, and stride
[30] of the frame level encoder $f$.

Self-attention module relatively weights the frame level representations and
computes weighted stats (mean, variance) to obtain the fixed dimensional
embedding. The relative importance of the frame-level representations is
calculated using the encoder-decoder framework. The encoder in this work
consists of the 1D convolutional layer and projects the frame-level
representations to lower dimensional space. The decoder calculates the
relative importance of frames using the encoded information. Let
$s_{e}:\mathbb{R}^{d\times T^{\prime}}\mapsto\mathbb{R}^{b\times T^{\prime}}$
and $s_{d}:\mathbb{R}^{b\times T^{\prime}}\mapsto\mathbb{R}^{d\times
T^{\prime}}$ be the encoder and decoder in the self-attention module. Then the
relative importance is computed as follows

$\textbf{e}_{1},\textbf{e}_{2},...,\textbf{e}_{T^{\prime}}=s_{e}(\textbf{h}_{1},\textbf{h}_{2},...,\textbf{h}_{T}^{\prime})$
(5)
$\textbf{a}_{1},\textbf{a}_{2},...,\textbf{a}_{T^{\prime}}=Softmax(s_{d}(\textbf{e}_{1},\textbf{e}_{2},...,\textbf{e}_{T^{\prime}}))$

Where $(\textbf{e}_{i})_{i=1}^{T^{\prime}}\in\mathbb{R}^{b}$ are low
dimensional encoded frames,
$(\textbf{a}_{i})_{i=1}^{T^{\prime}}\in\mathbb{R}^{d}$ are relative weights of
$(\textbf{h}_{i})_{i=1}^{T}\in\mathbb{R}^{d}$ computed from the self-attention
module and $Softmax(.)$ activation function is operated along the time axis.
Notice that the self-attention module assigns the relative weights to both
time and channel dimensions. Further the statistics of the relatively weighted
frame-level representation ( $(\textbf{a}_{i}\odot\textbf{h}_{i})_{i=1}^{T}$ )
are computed to obtain the fixed dimensional embedding. A single self-
attention layer may not be able to efficiently assign relative weights to
multiple sets of complimentary feature vectors. Hence, the multi-head self-
attention module incorporates multiple self-attention layers to obtain the
fixed dimension embeddings to capture various aspects of frame-level
representations efficiently. The fixed-dimensional embeddings extracted from
each head of the multi-head self-attentive module are concatenated to
incorporate the information from multiple views.

Table I: Effective receptive field and stride of proposed models Architecture | Receptive Field | Effective Stride
---|---|---
LPR-DNN | 2160 (135 ms) | 64 (4 ms)
LMS-DNN | 4960 (310 ms) | 12 (0.75 ms)

###  IV-B LMS-DNN: VTS feature-based DNN

To exploit the vocal tract features, we built an X-Vector [12] network using
Log-Mel Spectrogram features as shown in Figure 3. The Mel Spectrogram
features are passed into frame level encoder. After obtaining a compact
representation $(128\times T^{\prime})$, $T^{\prime}$ is governed by the
receptive field and stride as shown in Table I. Next, we perform a self-
attention module ($s_{e}:\mathbb{R}^{128\times
T^{\prime}}\mapsto\mathbb{R}^{64\times T^{\prime}}$, and
$s_{d}:\mathbb{R}^{64\times T^{\prime}}\mapsto\mathbb{R}^{128\times
T^{\prime}}$) followed by a pooling layer to normalize the features across
time steps to obtain a fixed 256 (concatenation of mean and variance of a
128-dimensional vector)-dimensional vector. Finally, we use the logistic
regression layer to perform algorithm classification.

Figure 4: Log-applied bicoherence magnitude plots on LP Residuals for samples
of 1 natural and 3 different synthetic speech algorithms with common speaker
(LA_0089) and common transcript (IS THERE ON HIS HANDS.). These plots are
displayed on a common intensity scale of [10,25] to enhance visibility.

##  V Voice Source Features: Linear Prediction Residuals

Capturing the magnitude spectral features (2nd-order statistics) for speaker
verification[31] tasks has been one of the standard approaches. The DL
community has developed TTS systems that primarily rely on loss functions that
minimize the distance between 2nd Order statistics [32, 33, 34]. Therefore,
using only 2nd-order features might not provide complete discriminatory
evidence among algorithms. This emerges the need for looking at features that
capture higher-order statistics.

There have been some established works [7, 8, 9, 10, 11] which stated that the
source and the system features, complementary within themselves, could be
utilized together for analysis of speech signals. The voice source features to
capture the vocal cords’ characteristics in speech production. The phase
spectral information is complementary to information from VTS features[35, 36]
as we are limiting to 2nd-order statistics. And one such higher-order
statistical feature exploited in this work is LP Residuals.

Linear Prediction (LP) analysis [37, 38, 39] is a methodology where the
magnitude spectral envelope is repressed by first finding the magnitude
spectral envelope and then operated with an inverse formulation which results
in residual rich in higher order correlations, with phase content. This
feature, called the LP residual, captures the voice source characteristics.
Analytically, the VTS response is modeled as a linear prediction filter. LP
Residual is obtained as a result of inverse filtering the speech signal with
an estimated VTS response.

$\displaystyle s[n]=\sum_{k=1}^{K}a_{k}s[n-k]+Ge[n]$ (6) $\displaystyle
S(z)=V(z)E(z);\;\;\;V(z)=\frac{G}{A(z)}=\frac{G}{1-\sum\limits_{k=1}^{K}a_{k}z^{-k}}$
(7) $\displaystyle E(z)=V(z)^{-1}S(z)$ (8)

Where $V(z)$ is the VTS response, $a_{k}$ are the linear coefficients, and $K$
is the order of the Linear Prediction.

For our study, we perform all-pole modeling of LP Residual[38]. The order is
chosen such that we have one resonance for each kHz ($K=\left\lceil
4+f_{s}/1000\right\rceil$) [40]. We ensure that the order is odd to constrain
the LP analysis with at least one real pole [41].

According to the Source-System model, the excitation source of voiced sounds
is ideally an impulse train. However, natural signals are different from the
ideal scenario, and the excitation source neither has a consistent
period(jitter) nor a constant amplitude(shimmer), making it less predictable,
as shown in Figure 1. On the other hand, LP Residuals of spoof signals are
periodic and have relatively constant periods making them more predictable,
and this is consistent with all the spoof algorithms. This is a potential
discriminative feature that can be exploited to classify spoof signals.

To validate the use of this feature, we perform Bispectrum [42] analysis on
the LP residuals, a third-order statistic (or skewness) study. We use
bicoherence (Equation V), which is the normalized bispectrum making phase
content more pronounced [43].

$\displaystyle B_{coher}(\omega_{1},\omega_{2})=$ (9)
$\displaystyle\frac{1}{W}\frac{\sum_{w=0}^{W-1}X(\omega_{1})X(\omega_{2})X^{*}(\omega_{1}+\omega_{2})}{\sqrt{\sum_{w=0}^{W-1}|X(\omega_{1})X(\omega_{2})|^{2}\sum_{w=0}^{W-1}|X^{*}(\omega_{1}+\omega_{2})|^{2}}}$

As shown in Figure 4, the patterns observed in bicoherence
plots111https://in.mathworks.com/matlabcentral/fileexchange/3013-hosa-higher-
order-spectral-analysis-toolbox suggest that the LP residuals have captured
the higher-order correlations highlighting phase information (LP residuals are
obtained from inversion of the magnitude spectrum of the signal) and support
its utilization as discriminatory evidence.

Hence, we extracted more exhaustive information from a speech sample by fusing
both features which are highlighted in Figure 5.

Figure 5: Attention maps on a synthetic speech signal with an utterance of
”THERE WAS A SUBSTANTIAL EXPLOSION.”, NOTE:- GREEN bands indicate attention by
LPR-DNN whereas RED represents LMS-DNN

###  V-A LPR-DNN: VS feature-based DNN

We extract low-dimensional features from the LP Residuals using
1D-convolutional filters and non-linear activation functions. LP Residuals are
obtained by inverse filtering the speech signal with an envelope - the so-
called VTS Response. Therefore the resultant spectrum of LP Residuals is
relatively flat, and traditional feature extraction methods such as spectral
magnitude will most likely be inefficient. This motivates us to introduce a
feature extractor. Since the usage of LP Residuals for the Spoof
detection/classification task is studied much less in the literature, we
propose a DNN-based feature extractor consisting of 1D-Convolutional filter
banks with non-linear activation functions. Figure 3 describes the parameters
used in the feature extractor; we utilize $Dropout$ layers to equally train
all the coefficients of 1D-filters [44]. $Layer$ $Normalisation$ [45] is
applied at end of every layer to avoid vanishing gradients while training the
network. We extract $(64\times T)$ dimensional features from the feature
extractor.

Later, frame-level encoding is performed on these features using the X-Vectors
system. After obtaining a compact representation (128 x $T^{\prime}$), where
$T^{\prime}$ is governed by receptive field and stride from Table I. The rest
of the network follows the description provided in Figure 3. It is worth
noting that attention values will be different than that of LMS-DNN because
here, inputs consist of additional information. Figure 5 shows bands of
attention on speech signal, and there is no overlap between LPR-DNN and LMS-
DNN bands. This indicates the presence of complementariness in VTS features
and VS features.

##  VI Experimental Setup

###  VI-A Implementation details

The hardware specifications for conducting the studies are Intel Xeon Silver
$4114$ CPU @ $2.20$GHz, and four NVIDIA GeForce RTX $2080$ Ti GPUs. The
software packages utilized are PyTorch library for training the neural
networks and scikit for computing LP Residuals. This link
222https://github.com/TUdayKiranReddy/SPCUP2022consists of source codes for
replicating all the results reported in this paper.

Class | Input/Ouput | Generator
---|---|---
A01 | Text/(MCC, F0) | WaveNet [46]
A02 | Text/(MCC, F0, BAP) | WORLD [47]
A03 | Text/(MCC, F0, BAP) | WORLD [47]
A04, A16 | Text/(MCC, F0) | Waveform Concatenation
A05 |  $Speech^{\rm a}$/(MFCC, F0, AP) | WORLD [47]
A06, A19 |  $Speech^{\rm a}$/LPC | Spectral filtering + OLA
A07 | Text/(MCC, F0, BAP) | WORLD [47]
A08 | Text/(MCC, F0) | Neural source-filter
A09 | Text/(MCC, F0) | Vocaine [48]
A10 | Text/Mel-Spectrograms | WaveRNN [49]
A11 | Text/Mel-Spectrograms | Griffin-Lim [50]
A12 | Text/(F0, Linguistic Features) | WaveNet [46]
A13 |  $Speech^{\rm b}$/MCC | Waveform filtering
A14 |  $Speech^{\rm b}$/(MCC, F0, BAP) | STRAIGHT [51]
A15 |  $Speech^{\rm b}$/(MCC, F0) | Waveform Concatenation
A17 |  $Speech^{\rm a}$/(MCC, F0) | Waveform filtering
A18 |  $Speech^{\rm a}$/MFCC | MFCC Vocoder
Natural | - | -
Table II: ASVSpoof19 dataset description

###  VI-B Dataset

For conducting the studies, we utilize the Logical Access portion of the data
from the ASVSpoof 2019 data corpus[52]. The description of each class is
described in Table II. We have a total of 18 classes ($17$ Synthetic + $1$
Natural). However, the ASVSpoof19 dataset consists of distinct speakers in
training and evaluation which best suits spoof classification (binary). Here
the work is primarily concerned with synthetic speech algorithm
classification; therefore we have pooled that data corpus and created custom
data splits. We have created two versions of custom data splits as shown in
Table III,

  * •

Custom Split-1 or CS1 In this data split, the whole ASVSpoof19 data is
accumulated and we have made 40%, 10%, and 50% split for training, validation,
and evaluation. We ensured that all the algorithms are uniformly spread over
all three data sets.

  * •

Custom Split-2 or CS2 To avoid overfitting the models by learning speaker
information, we have created CS2 data splits such that there are distinct
speakers in training and evaluation. We have 10 speakers in com mon for both
training and validation, to ensure that model is performing relatively good on
seen speakers. The proportions are mentioned in III.

Table III: Dataset composition (∗ represents common speakers) | CS1 | CS2
---|---|---
#Speakers | Train | 20 | 30 + 10*
Validation | 20 | 17 + 10*
Evaluation | 67 | 50
Proportion | Train | 40% | 33.25%
Validation | 10% | 16.11%
Evaluation | 50% | 50.64%

In this work, we use speech signals synthesized at a sampling frequency of
$16KHz$. To compute LP Residuals, we perform a $23rd$ order (for the reasons
mentioned in section V) LP analysis. To compute log mel spectrogram, we take
$80$ mels, $512$ bin FFT, window length of $400$ samples ($25$ms), and hop
length of $160$ ($10$ms).

##  VII Results and Experiments

In this section, evaluation performance and ablation analysis of the proposed
architectures are studied. We train our models LPR+DNN and LMS+DNN and discuss
the obtained performances. We perform various experiments on the proposed
architectures. First, we fuse the networks with various fusing paradigms and
validate the hypothesis of complementary information. Finally, we remove
silence regions in the training data to prevent overfitting the data [6]. This
operation emphasizes on capturing the effects of phonemic variations in
synthetic speech, rather than silent regions.

Loss function: We use cross entropy function as a loss function to optimize
our DNN models,
$\mathcal{L}_{(\boldsymbol{X},\boldsymbol{y})}(\boldsymbol{\theta})=H(\boldsymbol{y},{LR_{1}}\left(f\left(\boldsymbol{\theta},\boldsymbol{X}\right)\right))$.
We use Adam optimizer to train our network.
Evaluation Metrics:- We report algorithm classification accuracy (in %, higher
the better) to evaluate the performance of our models.

LPR-DNN vs LMS-DNN: In Table IV, we clearly observe that LPR-DNN is
outperforming LMS-DNN. This is evident from the hypothesis we gave earlier. LP
Residuals have more discriminative features because we are removing the VTS
response by inverse filtering it because the speech generation algorithms
typically optimize 2nd order features. The best performance obtained among
these two algorithms is by LPR-DNN with an algorithm classification accuracy
of 98.95%.

Table IV: Results of various models (Algorithm Accuracy %)

Classifier type | Architecture | CS1 | CS2
---|---|---|---
Evaluation | Evaluation
Without silence removal | LPR-DNN | 98.95% | 96.02%
LMS-DNN | 96.42% | 96.04%
LPR+LMS-DNN* | 99.58% | 98.66%
LMS-LPR-DNN | 99.06% | 97.65%
With silence removal | LPR-DNN | 96.37% | 98.42%
LMS-DNN | 83.08% | 90.44%
LPR+LMS-DNN* | 98.66% | 99.27%
LPR+LMS-DNN | 98.57% | 96.29%

Figure 6: t-SNE embeddings for processed vector from X-vectors

###  VII-A Model Fusion

Employing various fusion paradigms in speech community [53, 54] have shown
significant performance leaps. As discussed earlier, VS and VTS features have
complementary information, which we would like to exploit in this experiment
by performing two different kinds of fusion.

####  VII-A1 LPR+LMS-DNN: Intermediate fusion [55]

This fusion paradigm is achieved by fusing features at the architecture level;
typically, the concatenation of features occurs in the common latent space of
different systems. In this architecture variant, we concatenate features
obtained after frame-level encoding and perform multi-headed self-attention,
statistical pooling, and logistic regression.

####  VII-A2 LPR+LMS-DNN*: Late fusion [55]

One elementary fusion technique is known as score level fusion, where we
perform weighted averages on logits obtained from different systems. We
perform the same using LPR-DNN and LMS-DNN with equal weights.

###  VII-B Comparisons among variants of proposed models

The proposed models are hyper-tuned to achieve the best performance, and Table
IV consists of the performances achieved. To avoid over-fitting DNNs due to
distinct silent regions present in spoof algorithms[6], we also experimented
with silence removal using the Voice Activity Detector (VAD) system [56].
Below are the four major points that we would like to accentuate.

####  VII-B1 VS features are a cut above VTS features

As we observed discriminative features of VS in the previous sections, it is
evident from the performance on LPR-DNN, which is at least $2\%$ superior to
LMS-DNN.

####  VII-B2 Performance leap with fusion techniques

As anticipated, fusion techniques showed better performance than LPR-DNN and
LMS-DNN, which is a popular paradigm in the DNN community to achieve better
results. This performance leap is observed in all the studies performed, such
as with/without silence removal and inference with distinct speakers. We
plotted t-SNE [57] embeddings onto $2D$ space to evaluate the separability,
which is shown in Figure 6. It is observed that clusters in LMS-DNN embeddings
are merged into others and are close together. However, after performing
feature fusion, the clusters are clearly separated. This is yet another piece
of evidence for the hypothesis that VS and VTS features consist of
complementary information.

####  VII-B3 Over-fitting of VTS features

On the CS1 data set, the performance dip when silence regions are removed in
LMS-DNN is significant, implying that LMS-DNN is over-fitting the data by
exploiting the amount of silence region present in synthetic speech signals.
In contrast, LPR-DNN shows robustness to the higher amount of silence regions.

####  VII-B4 Robustness with unseen speakers

Speaker over-fitting is a generic issue in speech recognition/verification. To
test this, we have created CS2 dataset and trained on it appropriately. The
results obtained have very slight variations compared to CS1, implying that
proposed DNNs are robust to new speakers and can identify the underlying
algorithm characteristics.

##  VIII Analysis and Inference

In this section, we analyze the self-attentive modules and visualize the
portion of the speech signal highlighted. We pass a speech signal with an
utterance of ”THERE WAS A SUBSTANTIAL EXPLOSION” to both, LP residual and Mel
Spectrogram-based Models. We look at the attention values, which is a matrix
of size $(\\#Channels,T^{\prime})$, and $T^{\prime}$ is determined by the
effective receptive field and stride as mentioned in I. For visualization, we
take the mean of the matrix along the channel axis and later binarize it with
the mean over time as the threshold.

From Figure 5, we find that LPR-DNN gives more attention to the transition of
phonemes. In literature, it is known as coarticulation[21]. This is evident
from formant plots where the green bands(LPR-DNN) are highlighted at the
portions where formants change significantly, indicating the change in
phoneme. In contrast, LMS-DNN focuses on stationary parts of phonemes where
the formants are relatively constant.

Typically, when there is a change from voiced-to-unvoiced sound or vice-versa,
the source signal changes from random noise to impulse trains. This change is
captured by LPR-DNN, and it is different for different algorithms, therefore
providing stronger discrimination compared to LMS-DNN.

Figure 7: ROC plots for finding EER of LMS-DNN, LPR-DNN and fused LPR-LMS-DNN.
The EER for the case of LPR-LMS-DNN (green) is observed to be so low that the
plot almost overlaps with the axes

To observe the benefit of utilizing the complementariness of the features
captured by LPR-DNN and LMS-DNN, we take the case of the binary spoof
classifier. By plotting the ROC curve, we find that there is a significant
reduction of EER value for the fused model when compared to the individual
models. For LPR-DNN and LMS-DNN, the EER is 0.013114 and 0.017282
respectively, whereas the EER for the fused model is 0.000989. By maximizing
the information extracted from the speech signal, we can observe an increase
in classification accuracy.

(a) LMS-DNN

(b) LPR-DNN

Figure 8: Confusion matrix plots for proposed networks based on the waveform
generator of the synthesizer

To understand the attributions of the spoof samples, we have analyzed the
confusion plots based on the waveform generator of the underlying synthesizer
as shown in Figure 8. The confusion plots suggest that the synthesizers are
identified correctly for most of the waveform generators. It can be observed
from Figure 8(a) that the LMS-DNN performs better on waveform generators which
are based on Waveform Concatenation, Neural Source Filter, and Vocaine.

On the contrary, Figure 8(b) shows that the rest of the synthesizers are much
more accurately identified by the LPR-DNN. It can be observed that the LPR-DNN
performance is much more pronounced in the case of Griffin-Lim, Waveform
Filtering, STRAIGHT and Spectral Filtering + OLA. However, we observe that the
spoof samples generated from the WaveRNN seem to be the most confusing class
for both the designed architectures. The confusion happens to be between
WaveRNN and the speech samples generated by Wavenet and the natural class.

##  IX Conclusion

We proposed a spoof speech algorithm classifier using DNN by using voice
source (LP Residuals) and vocal tract (Mel Spectrograms) features. By
combining evidence from these two features, we achieved a performance leap
indicating complementary information between them. In a further investigation
into the attention maps of DNN over the spoof speech, we observed that LPR-DNN
focused on phoneme transitions, and in contrast, LMS-DNN is focused more on
the stationary portion of phonemes. We conclude that artifacts in spoof speech
are deductive using higher-order correlations and the research in spoof speech
generation algorithms has to consider improving natural phonetic transitions.
This study could be utilized for enhancing the synthetic generation techniques
by improving on the defects identified. Also, an exhaustive study on phonemes
could be performed to identify which phoneme utterance could be more critical
in spoof detection.

##  X Acknowledgement

We sincerely thank SIP Lab, IIT Hyderabad, for providing the computation power
needed for the studies and the IEEE SP Cup 2022 organizers for the problem
statement and the initial dataset.

## References

  * [1] C. Stupp, “Fraudsters Used AI to Mimic CEO’s Voice in Unusual Cybercrime Case.” https://www.wsj.com/articles/fraudsters-use-ai-to-mimic-ceos-voice-in-unusual-cybercrime-case-11567157402, Aug 30 2019.
  * [2] D.-S.-S. Engineering-Team, “When acting turns criminal: Deepfakes and voice impersonators in the cybercriminal underground.” https://www.digitalshadows.com/blog-and-research/deepfakes-and-voice-impersonators-in-the-cybercriminal-underground/, Dec 1 2021.
  * [3] J. Yang and R. K. Das, “Long-term high frequency features for synthetic speech detection,” Digital Signal Processing, vol. 97, p. 102622, 2020.
  * [4] M. Sahidullah, T. Kinnunen, and C. Hanilçi, “A comparison of features for synthetic speech detection,” 09 2015.
  * [5] H. Dhamyal, A. Ali, I. A. Qazi, and A. A. Raza, “Using self attention dnns to discover phonemic features for audio deep fake detection,” in 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 1178–1184, 2021.
  * [6] D. Mari, F. Latora, and S. Milani, “The sound of silence: Efficiency of first digit features in synthetic audio detection,” 2022.
  * [7] H. Wakita, “Residual energy of linear prediction applied to vowel and speaker recognition,” IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. 24, no. 3, pp. 270–271, 1976.
  * [8] M. Faundez-Zanuy and D. Rodriguez-Porcheron, “Speaker recognition using residual signal of linear and nonlinear prediction models,” 01 1998.
  * [9] W.-C. Hsu and J.-N. Sun, “A study of the usefulness of linear prediction residual for speaker recognition,” Advanced Science Letters, vol. 9, pp. 754–761, 04 2012.
  * [10] W.-C. Hsu, W.-H. Lai, and W.-P. Hong, “Usefulness of residual-based features in speaker verification and their combination way with linear prediction coefficients,” in Ninth IEEE International Symposium on Multimedia Workshops (ISMW 2007), pp. 246–251, 2007.
  * [11] S. Mahadeva Prasanna, C. S. Gupta, and B. Yegnanarayana, “Extraction of speaker-specific excitation information from linear prediction residual of speech,” Speech Communication, vol. 48, no. 10, pp. 1243–1261, 2006.
  * [12] D. Snyder, D. Garcia-Romero, G. Sell, D. Povey, and S. Khudanpur, “X-vectors: Robust dnn embeddings for speaker recognition,” in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5329–5333, 2018.
  * [13] A. Lacroix, “Speech production-physics, models and prospective applications,” in ISPA 2001. Proceedings of the 2nd International Symposium on Image and Signal Processing and Analysis. In conjunction with 23rd International Conference on Information Technology Interfaces (IEEE Cat., pp. 3–, 2001.
  * [14] W. Zhu and H. Kasuya, “A new speech synthesis system based on the arx speech production model,” in Proceeding of Fourth International Conference on Spoken Language Processing. ICSLP ’96, vol. 3, pp. 1413–1416 vol.3, 1996.
  * [15] K. Iso, “Speech recognition using dynamical model of speech production,” in 1993 IEEE International Conference on Acoustics, Speech, and Signal Processing, vol. 2, pp. 283–286 vol.2, 1993.
  * [16] L. Mitiche and A. B. H. Adamou-Mitiche, “Second order speech model based on ga’s,” in 2016 4th International Conference on Control Engineering & Information Technology (CEIT), pp. 1–4, 2016.
  * [17] C.-W. Kim, “Models of speech production,” in Formal Aspects of Cognitive Processes (T. Storer and D. Winter, eds.), (Berlin, Heidelberg), pp. 142–158, Springer Berlin Heidelberg, 1975.
  * [18] G. Fant, Acoustic theory of speech production.  No. 2, Walter de Gruyter, 1970.
  * [19] R. Singh, B. Raj, and D. Gencaga, “Forensic anthropometry from voice: An articulatory-phonetic approach,” in 2016 39th International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO), pp. 1375–1380, 2016.
  * [20] A. J. Rozsypal and B. F. Millar, “Perception of jitter and shimmer in synthetic vowels,” Journal of Phonetics, vol. 7, no. 4, pp. 343–355, 1979\.
  * [21] P. Menzerath and A. de Carvalho Lacerda, “Koartikulation, steuerung und lautabgrenzung : eine experimentelle untersuchung,” 1933.
  * [22] B. H. Story and K. Bunton, “An acoustically-driven vocal tract model for stop consonant production,” Speech Communication, vol. 87, pp. 1–17, 2017.
  * [23] P. Birkholz, “Modeling consonant-vowel coarticulation for articulatory speech synthesis,” PLOS ONE, vol. 8, pp. 1–17, 04 2013.
  * [24] A. Meghanani, A. C. S., and A. G. Ramakrishnan, “An exploration of log-mel spectrogram and mfcc features for alzheimer’s dementia recognition from spontaneous speech,” in 2021 IEEE Spoken Language Technology Workshop (SLT), pp. 670–677, 2021.
  * [25] F. Beritelli and R. Grasso, “A pattern recognition system for environmental sound classification based on mfccs and neural networks,” in 2008 2nd International Conference on Signal Processing and Communication Systems, pp. 1–4, 2008.
  * [26] W. J. Poser, “Douglas o’shaughnessy, speech communication: Human and machine. reading, massachusetts: Addison-wesley publishing company, 1987. pp. xviii 568\. isbn 0-201-16520-1.,” vol. 20, p. 52–54, Cambridge University Press, 1990\.
  * [27] C. V. Cotton and D. P. W. Ellis, “Spectral vs. spectro-temporal features for acoustic event detection,” in 2011 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pp. 69–72, 2011.
  * [28] V. Peltonen, J. Tuomi, A. Klapuri, J. Huopaniemi, and T. Sorsa, “Computational auditory scene recognition,” in 2002 IEEE International Conference on Acoustics, Speech, and Signal Processing, vol. 2, pp. II–1941–II–1944, 2002\.
  * [29] Y. Gao, T. Vuong, M. Elyasi, G. Bharaj, and R. Singh, “Generalized spoofing detection inspired from audio generation artifacts,” 2021.
  * [30] W. Luo, Y. Li, R. Urtasun, and R. Zemel, “Understanding the effective receptive field in deep convolutional neural networks,” 2017.
  * [31] F. Bimbot, J.-F. Bonastre, C. Fredouille, G. Gravier, I. Magrin-Chagnolleau, S. Meignier, T. Merlin, J. Ortega-García, D. Petrovska-Delacrétaz, and D. A. Reynolds, “A tutorial on text-independent speaker verification,” EURASIP Journal on Advances in Signal Processing, vol. 2004, no. 4, pp. 1–22, 2004.
  * [32] Y. Wang, R. Skerry-Ryan, D. Stanton, Y. Wu, R. J. Weiss, N. Jaitly, Z. Yang, Y. Xiao, Z. Chen, S. Bengio, Q. Le, Y. Agiomyrgiannakis, R. Clark, and R. A. Saurous, “Tacotron: Towards end-to-end speech synthesis,” 2017.
  * [33] A. v. d. Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu, “Wavenet: A generative model for raw audio,” 2016.
  * [34] S. O. Arik, M. Chrzanowski, A. Coates, G. Diamos, A. Gibiansky, Y. Kang, X. Li, J. Miller, A. Ng, J. Raiman, S. Sengupta, and M. Shoeybi, “Deep voice: Real-time neural text-to-speech,” 2017.
  * [35] K. Murty and B. Yegnanarayana, “Combining evidence from residual phase and mfcc features for speaker recognition,” vol. 13, pp. 52–55, 2006.
  * [36] R. Gonzalez, Better Than MFCC Audio Classification Features, pp. 291–301.  10 2013.
  * [37] G. Pop and D. Burileanu, “Towards detection of synthetic utterances in romanian language speech forensics,” in 2021 International Conference on Speech Technology and Human-Computer Dialogue (SpeD), pp. 80–84, 2021.
  * [38] K. S. R. Murty, V. Boominathan, and K. Vijayan, “Allpass modeling of lp residual for speaker recognition,” in 2012 International Conference on Signal Processing and Communications (SPCOM), pp. 1–5, 2012.
  * [39] J. Makhoul, “Linear prediction: A tutorial review,” Proceedings of the IEEE, vol. 63, no. 4, pp. 561–580, 1975.
  * [40] L. R. Rabiner, B. Gold, and C. K. Yuen, “Theory and application of digital signal processing,” IEEE Transactions on Systems, Man, and Cybernetics, vol. 8, no. 2, pp. 146–146, 1978.
  * [41] T. Quatieri, Discrete-time Speech Signal Processing: Principles and Practice.  Pearson Education, 2002.
  * [42] A. K. Singh and P. Singh, “Detection of ai-synthesized speech using cepstral & bispectral statistics,” in 2021 IEEE 4th International Conference on Multimedia Information Processing and Retrieval (MIPR), pp. 412–417, 2021.
  * [43] C. K. Kovach, H. Oya, and H. Kawasaki, “The bispectrum and its relationship to phase-amplitude coupling,” NeuroImage, vol. 173, pp. 518–539, 2018.
  * [44] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: A simple way to prevent neural networks from overfitting,” Journal of Machine Learning Research, vol. 15, no. 56, pp. 1929–1958, 2014.
  * [45] J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,” 2016.
  * [46] A. van den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu, “Wavenet: A generative model for raw audio,” in Arxiv, 2016.
  * [47] M. MORISE, F. YOKOMORI, and K. OZAWA, “World: A vocoder-based high-quality speech synthesis system for real-time applications,” IEICE Transactions on Information and Systems, vol. E99.D, no. 7, pp. 1877–1884, 2016.
  * [48] Y. Agiomyrgiannakis, “Vocaine: The vocoder and applications in speech synthesis,” pp. 4230–4234, 04 2015.
  * [49] N. Kalchbrenner, E. Elsen, K. Simonyan, S. Noury, N. Casagrande, E. Lockhart, F. Stimberg, A. v. d. Oord, S. Dieleman, and K. Kavukcuoglu, “Efficient neural audio synthesis,” 2018.
  * [50] D. Griffin and J. Lim, “Signal estimation from modified short-time fourier transform,” IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. 32, no. 2, pp. 236–243, 1984.
  * [51] H. Kawahara, “Straight, exploitation of the other aspect of vocoder: Perceptually isomorphic decomposition of speech sounds,” Acoustical Science and Technology, vol. 27, no. 6, pp. 349–353, 2006.
  * [52] X. Wang, J. Yamagishi, M. Todisco, H. Delgado, A. Nautsch, N. Evans, M. Sahidullah, V. Vestman, T. Kinnunen, K. A. Lee, L. Juvela, P. Alku, Y.-H. Peng, H.-T. Hwang, Y. Tsao, H.-M. Wang, S. L. Maguer, M. Becker, F. Henderson, R. Clark, Y. Zhang, Q. Wang, Y. Jia, K. Onuma, K. Mushika, T. Kaneda, Y. Jiang, L.-J. Liu, Y.-C. Wu, W.-C. Huang, T. Toda, K. Tanaka, H. Kameoka, I. Steiner, D. Matrouf, J.-F. Bonastre, A. Govender, S. Ronanki, J.-X. Zhang, and Z.-H. Ling, “Asvspoof 2019: A large-scale public database of synthesized, converted and replayed speech,” 2019.
  * [53] N. Chauhan, T. Isshiki, and D. Li, “Speaker recognition using fusion of features with feedforward artificial neural network and support vector machine,” in 2020 International Conference on Intelligent Engineering and Management (ICIEM), pp. 170–176, 2020.
  * [54] S. Toshniwal, A. Kannan, C.-C. Chiu, Y. Wu, T. N. Sainath, and K. Livescu, “A comparison of techniques for language model integration in encoder-decoder speech recognition,” 2018.
  * [55] D. Ramachandram and G. W. Taylor, “Deep multimodal learning: A survey on recent advances and trends,” IEEE Signal Processing Magazine, vol. 34, no. 6, pp. 96–108, 2017.
  * [56] Y.-Y. Yang, M. Hira, Z. Ni, A. Chourdia, A. Astafurov, C. Chen, C.-F. Yeh, C. Puhrsch, D. Pollack, D. Genzel, D. Greenberg, E. Z. Yang, J. Lian, J. Mahadeokar, J. Hwang, J. Chen, P. Goldsborough, P. Roy, S. Narenthiran, S. Watanabe, S. Chintala, V. Quenneville-Bélair, and Y. Shi, “Torchaudio: Building blocks for audio and speech processing,” arXiv preprint arXiv:2110.15018, 2021.
  * [57] L. van der Maaten and G. Hinton, “Visualizing data using t-sne,” Journal of Machine Learning Research, vol. 9, no. 86, pp. 2579–2605, 2008.