XX Month, XXXX XX Month, XXXX XX Month, XXXX XX Month, XXXX XX Month, XXXX
XXXX.2022.1234567

Corresponding author: Sarina Meyer (email: sarina.meyer@ims.uni-stuttgart.de).
* Equal contribution. This study is supported by the Carl Zeiss foundation,
JST CREST Grants (JPMJCR18A6), and MEXT KAKENHI Grant (22K21319).

# VoicePAT: An Efficient Open-source Evaluation Toolkit for Voice Privacy
Research

Sarina Meyer${}^{\textbf{1}}$ Student Member  IEEE  and Xiaoxiao
Miao${}^{\textbf{2}}$ Member  IEEE
and Ngoc Thang Vu${}^{\textbf{1}}$ Member  IEEE  Institute for Natural
Language Processing, University of Stuttgart, Germany  National Institute of
Informatics, Japan

###### Abstract

Speaker anonymization is the task of modifying a speech recording such that
the original speaker cannot be identified anymore. Since the first Voice
Privacy Challenge in 2020, along with the release of a framework, the
popularity of this research topic is continually increasing. However, the
comparison and combination of different anonymization approaches remains
challenging due to the complexity of evaluation and the absence of user-
friendly research frameworks. We therefore propose an efficient speaker
anonymization and evaluation framework based on a modular and easily
extendable structure, almost fully in Python. The framework facilitates the
orchestration of several anonymization approaches in parallel and allows for
interfacing between different techniques. Furthermore, we propose
modifications to common evaluation methods which make the evaluation more
powerful and reduces their computation time by 65 to 95%, depending on the
metric. Our code is fully open source.

speaker anonymization, voice privacy, privacy evaluation

IEEE Copyright Notice

©2024 IEEE. Personal use of this material is permitted. Permission from IEEE
must be obtained for all other uses, in any current or future media, including
reprinting/republishing this material for advertising or promotional purposes,
creating new collective works, for resale or redistribution to servers or
lists, or reuse of any copyrighted component of this work in other works.

This work has been submitted to the IEEE Open Journal of Signal Processing
(OJSP) - ICASSP 2024 .

Figure 1: Proposed framework, consisting of separate anonymization and
evaluation branches.

##  1 INTRODUCTION

Speaker anonymization [1] is a task in which speech recordings are
automatically modified in a way that the original speaker becomes
unidentifiable from the audio, usually by changing the voice towards the voice
of an artificial target speaker. The goal of this process is to preserve the
speaker’s voice privacy while keeping enough information from the input to use
the anonymized audio in downstream tasks (e.g., speech recognition [2]). In
order to foster research in this topic, the Voice Privacy Challenge (VPC) has
been held in 2020 and 2022 [1, 3]. The open-source framework accompanying the
challenges, consisting of code bases, baseline and evaluation models,
datasets, and techniques, have had a great influence on the field, with most
approaches using at least part of the VPC framework.

However, the purpose of the VPC framework is to support the challenges and
therefore needs more flexibility to be easily usable, adaptable, and
extendable outside of this context. It is heavily based on the Kaldi toolkit
[4], which, due to the complex nature of mismatching objectives of the
toolkit, leads to its processes being intransparent and complicated. Finally,
due to its static structure, the framework introduces a lot of redundant
computations, making the use of the VPC framework, in many cases, inefficient.

As voice privacy research is an important and timely topic for society and
politics, we need a robust and modular framework that allows us to quickly
test ideas without struggling too much with the infrastructure and that
facilitates researchers’ collaboration. In this paper, we propose a framework
for speaker anonymization. The framework is implemented almost fully in Python
and consists of two branches for anonymization and evaluation, as shown in
Figure 1. Both branches exhibit a modular structure in which single components
can be skipped or added. The methods and models in each component can be
easily exchanged by alternatives. The control over the composition of a
pipeline is done exclusively via configuration files such that different
speaker anonymization systems (SASs) or evaluation metrics can be compared
without much effort. We extend and improve commonly used evaluation methods by
employing the ESPnet [5] and SpeechBrain [6] toolkits for evaluation instead
of Kaldi [4]. Furthermore, we significantly speed up the evaluation by
combining data reduction and finetuning techniques for privacy and utility
metrics. Overall, we believe this framework will be an essential tool for
advancing research in voice privacy and facilitating participation in future
Voice Privacy Challenges in the field.

Our contributions are as follows:

  * •

We propose improvements to standard evaluation methods for speaker
anonymization which make these evaluations more efficient, easier to use, and
more feasible for intermediate evaluations.

  * •

We show through experiments on two primary VPC baselines and two state-of-the-
art SASs that these improvements provide stronger privacy tests, while
requiring 65 to 95% less time to execute.

  * •

We release all code in a new toolkit,
VoicePAT111https://github.com/DigitalPhonetics/VoicePAT (Voice Privacy
Anonymization Toolkit), which further includes pipelines for running different
state-of-the-art SASs.

  * •

Due to its modularity, extending VoicePAT with more anonymization systems and
evaluation metrics is easily possible, enabling the comparison of different
approaches within one framework and probing the effectiveness of the
anonymization using a more diverse attack landscape.

##  2 Background and Related Work

Before describing our proposed framework and evaluation, we will first give
some background about the current state of the topic, existing evaluation
metrics, and frameworks.

###  2.1 SPEAKER ANONYMIZATION

The goal of an SAS as defined by VPC [1, 3] is to automatically modify the
voice in an audio recording such that the original speaker cannot be
identified anymore. Following VPC baseline models, approaches can be
distinguished into methods based on digital signal processing and
disentanglement. Whereas the former perceptively modify speech to conceal the
original speaker’s identity [1, 3], they are less effective than
disentanglement-based methods [7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17], which
involves three steps as shown in the left of Figure 1:

(1) Disentanglement of Original Speech: (i) Frame-level content features
extraction via automatic speech recognition (ASR) models [2, 18] or self-
supervised learning (SSL) based content encoders [10, 11]; (ii) Frame-level
prosody features extraction, e.g., F0 using the YAAPT algorithm [19]; (iii)
Utterance-level speaker embeddings extraction, e.g., from pre-trained x-vector
[20] or ECAPA-TDNN [21] models for automatic speaker verification (ASV).

(2) Modification of original features: This is a crucial step to hide the
original speaker’s identity. Most works focus on modifying the original
speaker embeddings, assuming that identity is mainly encoded in them.
Typically, a selection-based speaker anonymizer [22], such as the baselines BL
1.a and BL 1.b of the VPC, replaces an original speaker with an anonymized
speaker vector. This anonymized vector is a mean speaker vector derived from a
randomly chosen set of speaker vectors in an external pool. However, the
averaging process on speaker vectors leads to limited speaker diversity in the
generated anonymized voices and serious speaker privacy leakage problems when
facing stronger attacker [12]. To mitigate these problems, recent works adopt
DNN-based anonymizers. For example, [8, 9] introduce a Wasserstein generative
adversarial network (GAN) that is trained to turn random noise into artificial
speaker embeddings that follow a similar distribution as the original speaker
vectors. Another approach employs an orthogonal householder neural network
(OHNN) to rotate original speaker vectors, ensuring anonymized vectors follow
the original space, maintaining speech naturalness. The parameters of OHNN-
based anonymizer are trained using classification and similarity losses,
encouraging distinct speaker identities [12].

(3) Generation of anonymized speech: The anonymized speaker, prosody, and
content features are then fed into a synthesis model [23, 24] to generate
anonymized speech.

###  2.2 EVALUATION

Speaker anonymization commonly has two objectives: (1) privacy: protecting the
identity of a speaker, and (2) utility: keeping other attributes of the
original audio needed for use in downstream applications (e.g., linguistic
content, naturalness, prosody, speaker emotion) unchanged. The challenge is to
optimize an SAS to achieve a trade-off between both objectives, whereby the
weighting between them and the utility assessment metrics depend on the
application.

####  2.2.1 SPEAKER PRIVACY PROTECTION

To evaluate the effectiveness of preserving speaker identity against different
attackers, it is most common to compute the equal error rate (EER) using ASV
evaluation models. For this, an attacker is comparing the anonymized trial
utterance processed by users against enrollment utterances in different attack
conditions with either a model trained on original ($ASV_{\text{eval}}$) or on
anonymized ($ASV_{\text{eval}}^{\text{anon}}$) data [1, 3]:

  * •

Unprotected (OO): a baseline metric to assess the strength of the
$ASV_{\text{eval}}$ attacker when both enrollment and trial utterances are not
anonymized (i.e., original).

  * •

Ignorant (OA): attackers are unaware of the anonymization, they use original
enrollment data and $ASV_{\text{eval}}$ to infer the identity of anonymized
trial utterances.

  * •

Lazy-informed (AA-lazy): attackers use anonymized enrollment speech, generated
with the same SAS but inaccurate parameters, and $ASV_{\text{eval}}$ to detect
identities.

  * •

Semi-informed (AA-semi): the attacker is similar to the lazy-informed one, but
employs a more powerful $ASV_{\text{eval}}^{\text{anon}}$ model trained on
anonymized speech, which helps to reduce the mismatch between the original and
anonymized speech to infer the speaker identity.

For a successful anonymization, an EER close to 50% is targeted. An
alternative to the EER metric is using the log-likelihood-ratio cost function
$C_{llr}$ as discrimination loss ($C_{llr}^{min}$) and calibration loss
($C_{llr}-C_{llr}^{min}$) [1]. Other privacy metrics include the linkability
($D_{\leftrightarrow}^{sys}$) between two utterances [25, 26], the de-
identification ($De_{ID}$) based on voice similarity matrices [27], and the
expected ($D_{ECE}$) and worst-case ($\log(l)$) privacy disclosure metrics of
the ZEBRA framework [28].

####  2.2.2 SPEECH UTILITY PRESERVATION

As most applications require the anonymization to retain the linguistic
content of the speech, the primary utility evaluation is performed with ASR
and measured as word error rate (WER). The VPC introduces two models for this:
$ASR_{\text{eval}}$ (A-lazy) is trained on the original data, and
$ASR_{\text{eval}}^{\text{anon}}$ (A-semi) is trained on the anonymized data,
similar to the attackers in privacy evaluation. The lower the WER is, the
better.

Another common utility metric is the gain of voice distinctiveness
$\text{G}_{\text{VD}}$ [26] that assesses how well the ability of
distinguishing different speakers is kept during anonymization. If the voice
distinctiveness in the anonymized space is the same as in the original space,
the $\text{G}_{\text{VD}}$ is close to zero. If it is improved, the score is
above zero, otherwise below. The metric is closely related to the privacy
metrics and is computed using $ASV_{\text{eval}}$ and voice similarity
matrices.

Depending on the application, other speech utility metrics could be included,
e.g., prosody preservation via pitch correlation as done in the VPC 2022 [3].

For simplicity, we will focus on EER as privacy indicator, and WER and
$\text{G}_{\text{VD}}$ for measuring utility in our experiments.

###  2.3 EXISTING FRAMEWORKS AND THEIR LIMITATIONS

To support participation in the challenges, the VPC published an open-source
framework with code for all baselines and evaluation metrics. However, this
framework lacks the flexibility to skip single steps in its run pipeline,
rerunning only parts of it, or adding new metrics. Most of the algorithms are
written in the C++-based Kaldi toolkit which is challenging to maintain and
lacks compatibility with standard Python-based speech processing models.
Furthermore, the evaluation models included in the framework may take several
days for computations. Combined with the difficulty of skipping previously
computed calculations, performing a full anonymization with subsequent
evaluation in the VPC framework is complicated and expensive, potentially
discouraging new researchers from working on SAS development. Motivated by
similar concerns about the framework, [29] recently presented an alternative
evaluation framework written in Python and exhibiting modular and extendable
structures. However, they do not test their framework with standard SAS
approaches nor compare their evaluation metrics with the ones in the VPC
framework. This makes it difficult to assess the quality of their
improvements. We thus find a lack of suitable anonymization and evaluation
frameworks for this topic. Hence, we propose a new alternative in this paper.

##  3 PROPOSED EFFICIENT FRAMEWORK

(a) Privacy

(b) Utility

Figure 2: Comparisons of VPC and proposed privacy and utility evaluation
models. Note that although $\text{G}_{\text{VD}}$ is a utility metric, we
plotted $\text{G}_{\text{VD}}$ evaluation time in the privacy subplot as it is
computed by $ASV_{eval}$.

The proposed framework for speaker anonymization research consists of two
pipeline branches, shown in Figure 1: One for the anonymization process and
one for evaluation. Both branches consist of different modules which in turn
hold various models and algorithms that can be used to fulfill the role of the
module. All parameters for selecting the order and type of modules and their
constituents, as well as all other settings for running a pipeline, are given
in configuration files. Modules can in principle be exchanged, extended, or
skipped if the general objective allows it. In this way, the framework
provides a flexible option to modify existing approaches, combine ideas from
different systems, and test the effect of single components in a controlled
fashion.

###  3.1 ANONYMIZATION BRANCH

The idea of the anonymization branch is to provide a platform for researchers
developing an SAS to evaluate their ideas quickly. Ideally, if they only want
to test a minor change like a different speaker embedding modification
mechanism, they would only have to add a new model (a Python class) to the
speaker embedding modification module and adjust the configuration file. If
the modification is radical, they might need to add a new module and pipeline.

Generally, an SAS consists of the following components: (1) a configuration
file, (2) a pipeline, (3) a collection of modules, and (4) one specific model
in each module. The configuration file specifies the pipeline (e.g., the GAN-
based pipeline of [9]), which then defines the obligatory and optional modules
of the pipeline and their processing order. Each module might have different
models or approaches, e.g., different speech synthesis models. The selection
of one model per module and the inclusion of optional modules are specified in
the configuration file. Per default, the output of each module is saved to
disk. This makes it possible to skip the computation of one module if it has
been computed before and its input has not changed, and thus, to test minor
modifications more efficiently.

###  3.2 EVALUATION BRANCH

Following the standards of voice privacy research, the evaluation branch is
divided into two modules: privacy and utility. Each module corresponds to one
evaluation aspect and can consist of one or several metrics. For example, ASV
is one module of privacy evaluation and is mainly measured by one metric, EER.
Multiple models or algorithms are feasible for the ASV module. Similar to the
anonymization branch, all settings are again set in configuration files. To
further improve the efficiency of the proposed framework, we employ more
powerful ASV and ASR models, explore the training strategies, and modify the
computation of $\text{G}_{\text{VD}}$ as described below.

####  3.2.1 Evaluation models

In the VPC framework, EER and $\text{G}_{\text{VD}}$ are computed using a
Kaldi-based x-vector speaker encoder with a PLDA distance model [20], and WERs
are computed using a Kaldi-based TDNN-F model [2]. As ASV and ASR technology
develops, it is important to examine the impact of advanced models on the
respective evaluation results. Aiming to find the most reliable choice, we
propose using evaluation models based on the state-of-the-art toolkits
Speechbrain [6] for ASV and ESPnet [5] for end-to-end ASR, as shown in Figure
2.

Both toolkits are developed using PyTorch [30] in the context of research, and
they meet two requirements for our purposes: (1) They provide a user-friendly
approach to modify training recipes which cover a wide range of
hyperparameters and architecture choices for the models. (2) Both toolkits are
continuously developed, ensuring they incorporate the latest advancements in
both ASV and end-to-end ASR techniques regularly. For ASV evaluation models,
we present choices including the Speechbrain-based x-vector and the cutting-
edge ECAPA-TDNN, featuring both cosine and PLDA back-ends. For ASR evaluation
models, we provide an ESPnet-based Transformer encoder-based Connectionist
Temporal Classification (CTC) ASR model with Attention Encoder Decoder (AED)
[31]. A transformer-based language model is trained using LibriSpeech-train-
clean-360 [32] once and used for decoding.

####  3.2.2 Modifications for $\text{G}_{\text{VD}}$

The gain of voice distinctiveness metric $G_{\text{VD}}$ [27] is defined as
the diagonal dominance ratio of two voice similarity matrices, one for the
original speaker space and one for the anonymized one. In the VPC framework,
those similarity scores are computed by the $ASV_{\text{eval}}$ model trained
on the original data. $ASV_{\text{eval}}$ yields more reliable scores for the
original data but introduces a mismatch when applied to anonymized speech. We
are interested in exploring different evaluation models for computing
similarity scores: (1) using the $ASV_{\text{eval}}$ model; (2) using the
$ASV_{\text{eval}}^{anon}$ model; (3) using $ASV_{\text{eval}}$ for original
data and $ASV_{\text{eval}}^{anon}$ for anonymized data to see whether this
could improve the accuracy of similarity scores for different types of data.
Furthermore, different from the VPC framework, where all utterances of each
speaker are considered for similarity computation, we enhance efficiency by
randomly selecting 5 utterances per speaker to compute the log-likelihood
ratios between two speakers.

####  3.2.3 Training strategy of $ASV_{\text{eval}}^{\text{anon}}$ and
$ASR_{\text{eval}}^{anon}$ models

Four models are required for privacy and utility evaluation, as described in
Section 2.2.2: $ASV_{\text{eval}}$ and $ASR_{\text{eval}}$, trained on the
original LibriSpeech-train-clean-360 dataset, are directly provided by the VPC
platform. Thus, we will assess only their evaluation times, not the training
time. $ASV_{\text{eval}}^{\text{anon}}$ and $ASR_{\text{eval}}^{anon}$ are
trained from scratch like the original models, but on the anonymized
LibriSpeech-train-clean-360 processed by the same evaluated SAS. This requires
extra time for anonymizing training data and conducting the training process,
in addition to the evaluation time.

Since the anonymization of the entire training dataset is quite time-
consuming, we aim to explore the impact of obtaining
$ASV_{\text{eval}}^{\text{anon}}$ and $ASR_{\text{eval}}^{anon}$ by finetuning
the pretrained $ASV_{\text{eval}}$ and $ASR_{\text{eval}}$, respectively,
using only a subset of the anonymized data to eliminate the necessity of
anonymizing the entire dataset. We consider two techniques for data reduction:
(1) choosing a limited number of utterances from all the speakers, and (2)
selecting all utterances from a specific subset of speakers. By using this
approach, we can balance the trade-off between anonymization, training time,
and the effectiveness of the $ASV_{\text{eval}}^{\text{anon}}$ and
$ASR_{\text{eval}}^{anon}$.

##  4 EXPERIMENTS

In this section, we compare the effectiveness of different evaluation models
in measuring privacy and utility performance across various SASs. The
evaluation models used here are trained on either the entire original or
anonymized LibriSpeech-train-clean-360 dataset. Once the evaluation models are
chosen, the focus shifts to exploring the training strategy for the semi-
informed evaluation models, particularly concerning the amount of training
data.

To draw more general conclusions, we choose diverse SASs, including both
traditional and state-of-the-art methods, to generate anonymized speech and
evaluate it using our proposed VoicePAT. The notations for these SASs are
listed in Table 1. All experiments are performed on the VCTK [33] and
LibriSpeech [32] test sets as given by the VPC. The results consistently
report the average score on them. Further settings, e.g., training
hyperparameters, can be found in our source code. All time measurements apply
to experiments conducted on single NVIDIA A100 GPU, except ASR evaluation
using 4 GPUs.

Table 1:  Notations for the evaluated SASs. GST means global style token [9],
ROH means random orthogonal Householder [12].

Notation |  Content encoder |  Speaker encoder |  Syn. model |  Speaker anon.
---|---|---|---|---
BL 1.a [3] | TDNN-F | x-vector | SS-AM+NSF | Select.
BL 1.b [3] | TDNN-F | x-vector | HiFi-GAN+NSF | Select.
GAN [9] | Branchformer+CTC/AED | GST-based | FastSpeech2 + HiFi-GAN | GAN
OHNN [12] | SSL | ECAPA | HiFi-GAN |  ROH

###  4.1 CHOICE OF EVALUATION MODELS

####  4.1.1 ASV evaluation models

Table 2: Comparison of four privacy attack models using the x-vector and ECAPA
speaker encoders with PLDA and cosine as distance measures. Privacy scores for
each SAS and attack condition are given as EER in %. $\uparrow$ means higher
values are better, while $\downarrow$ means lower values are better.

SAS | Eval | OO $\downarrow$ | OA $\uparrow$ | AA-lazy $\uparrow$ | AA-semi $\uparrow$
---|---|---|---|---|---
BL 1.a | x-vector + cosine | 9.24 | 54.38 | 26.39 | 11.73
x-vector + PLDA | 7.18 | 52.56 | 26.41 | 8.66
ECAPA + cosine | 3.11 | 52.85 | 25.65 | 7.62
ECAPA + PLDA | 3.49 | 51.62 | 25.06 | 7.69
BL 1.b | x-vector + cosine | 9.24 | 52.30 | 25.31 | 11.33
x-vector + PLDA | 7.18 | 51.49 | 25.98 | 8.27
ECAPA + cosine | 3.11 | 49.82 | 24.67 | 7.21
ECAPA + PLDA | 3.49 | 50.84 | 23.10 | 8.51
OHNN | x-vector + cosine | 9.24 | 49.65 | 45.55 | 47.09
x-vector + PLDA | 7.14 | 49.34 | 45.55 | 42.47
ECAPA + cosine | 3.11 | 48.22 | 44.26 | 44.62
ECAPA + PLDA | 3.63 | 50.18 | 44.44 | 41.47
GAN | x-vector + cosine | 9.24 | 53.05 | 48.57 | 43.11
x-vector + PLDA | 7.03 | 52.80 | 48.71 | 44.12
ECAPA + cosine | 3.11 | 51.45 | 46.11 | 44.61
ECAPA + PLDA | 3.68 | 50.85 | 47.57 | 47.20

Table 2 lists the mean EERs for various SASs under all conditions computed by
different evaluation models. First, no matter which ASV attacker is used, the
EERs of BL 1.a and BL 1.b decrease by around 25% under the AA-lazy and 7%-11%
under the AA-semi condition, indicating severe privacy leakage when facing the
stronger, semi-informed attack model $ASV_{\text{eval}}^{\text{anon}}$.
However, the EERs of OHNN\- and GAN-based SASs yield over 40% across all the
attack conditions, showing remarkable privacy protection capabilities. In
order to decide which evaluation model provides the best results, we look at
the OO condition, where no SAS is applied and it is expected to achieve very
low EERs on this original data. The model using ECAPA-TDNN and cosine distance
achieves with 3.11% the lowest EER and can therefore be considered as the best
ASV evaluation model. This is consistent with the findings in the ASV field
[34, 35].

Table 3: EERs and WERs obtained by the proposed and the VPC evaluation models.
The proposed models are ECAPA-TDNN + cosine for ASV and transformer-based
CTC/AED ASR. -l and -s stand for -lazy and -semi.

|  | EER (%) $\uparrow$ | WER (%) $\downarrow$
---|---|---|---
SAS | Eval | OO $\downarrow$ | OA | AA-l | AA-s | O | A-l | A-s
BL 1.a | VPC | 3.58 | 50.07 | 30.31 | 10.31 | 8.48 | 8.96 | 7.79
Proposed | 3.11 | 52.85 | 25.65 | 7.62 | 7.24 | 8.00 | 7.66
BL 1.b | VPC | 3.58 | 51.11 | 28.98 | 9.93 | 8.48 | 10.16 | 7.61
Proposed | 3.11 | 49.82 | 24.67 | 7.21 | 7.24 | 8.42 | 7.50
OHNN | VPC | 3.58 | 49.93 | 45.94 | 42.15 | 8.48 | 10.24 | 8.01
Proposed | 3.11 | 48.22 | 44.26 | 44.62 | 7.24 | 7.92 | 7.70
GAN | VPC | 3.58 | 51.09 | 46.49 | 44.33 | 8.48 | 8.47 | 6.86
Proposed | 3.11 | 51.45 | 46.11 | 44.61 | 7.24 | 8.00 | 7.03

Accordingly, we choose ECAPA-TDNN + cosine as the primary proposed ASV
evaluation model in all following experiments. In the central columns in Table
3, we compare the EERs with this proposed model to the x-vector + PLDA model
of the VPC toolkit. Compared to the VPC model, the proposed one consistently
achieves lower or similar EERs across all the conditions and SASs. This means
that the proposed ASV model is a stronger attacker, which is reasonable as the
proposed model exhibits a more powerful ability to infer the speaker’s
identity.

Moreover, the proposed ASV model reduces the time needed for training it and
performing the evaluation (Figure 2(a)). Instead of 6 hours for training the
VPC model (which includes the x-vector encoder and the PLDA), our proposed
ECAPA-TDNN + cosine model only requires 2 hours. The effect on evaluation time
is smaller though still noticeable, from 30 minutes to 20 when using the
proposed model.

####  4.1.2 ASR evaluation models

The right columns of Table 3 summarize the WERs for both original (O) and
anonymized data (A-lazy, A-semi), using the VPC (TDNN-F) or proposed
(Transformer-based CTC/AED) evaluation models. One common trend for all SASs
is that the proposed ASR model achieves notably lower WERs in comparison to
the VPC model. For the A-semi condition, the utilization of
$ASR_{\text{eval}}^{\text{anon}}$ can reduce the mismatch between original and
anonymized data, further decreasing the WERs.

Another interesting observation is that for the VPC model, the WERs of A-semi
decoded by $ASR_{\text{eval}}^{\text{anon}}$ are consistently lower than the O
condition decoded by $ASR_{\text{eval}}$. In contrast, for the proposed model,
the original data yield the lowest WERs for most SASs, regardless of the
training data of ASR evaluation models, except for the GAN-based SAS. Possible
reasons could be either that the $ASR_{\text{eval}}$ provided by VPC was not
adequately trained or the structure of this model is not powerful enough. In
contrast, our proposed ASR model is more powerful in achieving accurate
results.

Regarding training and evaluation time for the ASR models (Figure 2(b)), we
observe that our model increases the time for ASR evaluation from 5 hours of
the VPC model to 6 hours. However, the training of the proposed model takes 20
hours, significantly less than the 72 hours required by the VPC model.
Moreover, comparing the A-lazy and A-semi results show that training the ASR
model on anonymized data, resulting in the $ASR_{\text{eval}}^{\text{anon}}$
model, has less effect for the proposed model than the one of the VPC. It can
therefore be argued that training the $ASR_{\text{eval}}^{\text{anon}}$ model
for each evaluation and using the A-semi condition is not necessary with the
proposed model. This is further supported by arguing that using an ASR model
specifically trained on anonymized data might not be realistic in actual
applications. Thus, by using a more robust ASR model and reverting to only the
A-lazy condition, we could effectively reduce the evaluation time by 92% from
77 hours of the VPC to 6 hours222In the A-semi condition, we have a reduction
by 66% (from 77h to 26)..

Table 4: Comparisons of $\text{G}_{\text{VD}}$ obtained by the VPC and
proposed ECAPA-TDNN + cosine evaluation models. The evaluation models can be
either only $ASV_{\text{eval}}$, $ASV_{\text{eval}}^{anon}$, or a combination
of both ($ASV_{\text{eval}}$ for original and $ASV_{\text{eval}}^{anon}$ for
anonymized data). #utts per spk=5 means randomly selecting 5 utterances per
speaker for similarity computation.

SAS | Eval | # utts per spk | $ASV_{\text{eval}}$ | $ASV_{\text{eval}}^{\text{anon}}$ | Both
---|---|---|---|---|---
BL 1.a | VPC | all | -7.71 | 0.18 | -1.11
Proposed | all | -6.57 | 1.37 | 0.03
Proposed | 5 | -6.79 | 1.46 | 0.11
BL 1.b | VPC | all | -7.29 | 0.13 | -1.09
Proposed | all | -7.40 | 1.72 | 0.62
Proposed | 5 | -7.39 | 1.80 | 0.72
OHNN | VPC | all | -1.34 | -0.11 | -1.10
Proposed | all | -1.53 | 1.69 | 0.23
Proposed | 5 | -1.57 | 1.65 | 0.15
GAN | VPC | all | -0.84 | 0.68 | 0.11
Proposed | all | -1.74 | 2.64 | 2.03
Proposed | 5 | -1.62 | 2.63 | 1.96

###  4.2 GAIN OF VOICE DISTINCTIVENESS

Table 4 lists $\text{G}_{\text{VD}}$ results for various SASs computed by the
VPC and proposed evaluation models. Looking at the $\text{G}_{\text{VD}}$
achieved by $ASV_{\text{eval}}$, we can see: (1) Anonymized speech generated
through the OHNN\- and GAN-based models exhibit higher voice distinctiveness
than BL 1.a and BL 1.b, with $\text{G}_{\text{VD}}$ closer to zero. (2) The
VPC and the proposed models achieve both either similar or the proposed one
lower $\text{G}_{\text{VD}}$ values. (3) Comparing the proposed model
utilizing all utterances and 5 utterances per speaker reveals similar results,
indicating the use of 5 utterances may suffice for small test sets with
limited voice variation333LibriSpeech and VCTK contain read speech segments
extracted from longer recordings.. At the same time, using only 5 utterances
per speaker reduces the time for computing the $\text{G}_{\text{VD}}$
drastically from 6 hours to only 20 minutes (Figure 2(a)). Thus, we can speed
up the $\text{G}_{\text{VD}}$ evaluation by 95% without reducing the result
quality.

However, when employing $ASV_{\text{eval}}^{anon}$ or a combination of both
$ASV_{\text{eval}}$ and $ASV_{\text{eval}}^{anon}$, the $\text{G}_{\text{VD}}$
is significantly higher and often above zero, indicating a positive gain in
voice distinctiveness. The difference between using the VPC models and the
proposed ones is more notable, whereby the proposed models suggest almost no
difference between the SASs anymore. This shows that $\text{G}_{\text{VD}}$ is
an unstable metric that highly depends on the model used for evaluation. In
order to measure voice distinctiveness precisely, it may be necessary to
consider downstream tasks like speaker diarization [36], instead of relying
solely on the $\text{G}_{\text{VD}}$ metric.

all$0$$4$$8$$12$$16$Error Rate %ASV (EER)ASR (WER)905010500200# utts per spk#
spk (a) Error rates for $ASV_{\text{eval}}^{anon}$ and
$ASR_{\text{eval}}^{anon}$

all905010020k40k60k80k100kTraining data [#
utts]all500200AnonymizationTrainingTotalData02004006008001kTime [min]# utts
per spk# spk (b) Time and data efficiency for $ASV_{\text{eval}}^{anon}$

Figure 3: Effect of different training strategies for
$ASV_{\text{eval}}^{anon}$ and $ASR_{\text{eval}}^{anon}$ on (a) evaluation
metrics for BL 1.a, and (b) data and time efficiency. The strategies involve
finetuning with data reduction, either by restricting the number of utterances
per speaker or the number of speakers. They are compared against using all
data for training the models from scratch.

###  4.3 TRAINING STRATEGY FOR $ASV_{\text{eval}}^{\text{anon}}$ AND
$ASR_{\text{eval}}^{anon}$ MODELS

Figure 3 shows the influence of the different data reduction strategies on the
privacy scores and evaluation efficiency for BL 1.a. The experiments are
conducted across various $ASV_{\text{eval}}^{\text{anon}}$ trained using
different amounts of anonymized speech data.

It can be observed from the results in Figure 3(a) that the more we decrease
the amount of training data, the more the EER increases. This is especially
problematic for #utts per spk=10 because its scores might suggest that the
SAS’s privacy protection would be better than it actually is. For WER, the
effect of data reduction is negligible as it only changes from 7.66% WER (all)
to 7.91% (#utts per spk=10) in the worst case.

The biggest impact of the different training strategies for
$ASV_{\text{eval}}^{\text{anon}}$ can be seen in Figure 3(b). It shows that
the largest factor of time needed for privacy evaluation comes from the
requirement of having to anonymize the training data for each evaluation run.
Decreasing the amount of training data therefore means less time being spent
on anonymizing it, thus, the time cost of an evaluation decreases linearly
with the reduction of data.

Based on these results, we found that #utts per spk=50 provides the best
balance between EER increase and cost.444We validated this finding using other
SASs as well. We omitted them due to limited space.. This data reduction
decreases the total evaluation time for ASV evaluation from 16 hours (14.25
hours for anonymization, 2 hours for training) to 7 hours (6.25 hours for
anonymization, 49 minutes for training). Compared to 20.25 hours needed in the
VPC evaluation framework (14.25 hours for anonymization, 6 hours for
training), this reduces the time needed for creating the
$ASV_{\text{eval}}^{\text{anon}}$ model by 65%. Therefore, we recommend to use
finetuning with this setting at least during SAS development, and only revert
to training on all data from scratch for final evaluation.

##  5 ANALYSIS

This section delves deeper into privacy results by examining resynthesis
performance and summarizing rankings when employing various evaluation models
for different SASs.

Table 5: Comparison of EERs (%) and WERs (%) on the resynthesized conditions
without anonymization. The proposed models are ECAPA-TDNN + cosine for ASV and
transformer-based CTC/AED ASR.

|  | EER (%) | WER (%)
---|---|---|---
SAS | Eval | OO | OR | RR-lazy | O | R-lazy
BL 1.a | VPC | 3.58 | 17.53 | 13.24 | 8.48 | 10.06
Proposed | 3.11 | 22.47 | 14.10 | 7.24 | 11.00
BL 1.b | VPC | 3.58 | 21.25 | 14.62 | 8.48 | 12.73
Proposed | 3.11 | 28.45 | 15.28 | 7.24 | 9.75
OHNN | VPC | 3.30 | 7.60 | 4.82 | 8.48 | 9.45
Proposed | 3.11 | 10.05 | 4.28 | 7.24 | 7.58
GAN | VPC | 3.29 | 23.09 | 16.08 | 8.48 | 8.16
Proposed | 3.11 | 27.09 | 17.68 | 7.24 | 7.85

###  5.1 RESYNTHESIS

In the privacy evaluation, we obtain one privacy score in form of EER for each
model and attack condition. However, as all tested SASs employ a speech
synthesis step after the actual anonymization method, it is not clear whether
the anonymization power of an SAS comes actually from this anonymization
method or from the synthesis. We therefore explore this by testing the
resynthesis performance of each SAS. For this, we generate a new version of
the evaluation data per SAS in which the anonymization method is skipped and
instead the original speaker vector is used for synthesis. We evaluate two new
conditions for $ASV_{\text{eval}}$: OR and RR-lazy, with original or
resynthesized enrollment data, respectively, and resynthesized trial data. For
$ASR_{\text{eval}}$, we test the decoding performance on the resynthesized
data (R-lazy). In both cases, we compare to the performance on the original
data.

Table 5 shows the results. Except for the OHNN-based SAS, all SASs clearly
exploit the synthesis to increase the privacy protection and do not rely only
on their anonymization method. The OHNN-based SAS, on the other hand, has
almost no identification loss during resynthesis. However, the synthesis in
all SASs lead to an increase in WER and thus reduced intelligibility.

Using the synthesis to increase the privacy protection is not necessarily a
drawback of the BL 1.a, BL 1.b and GAN-based SAS. However, researchers might
not be aware of this effect and might put too much focus on optimizing their
anonymization method instead of the synthesis. It also decreases the control
one has about the actual outcome of the SAS. A related issue was already
observed by [37] about the vocoder drift of speaker vectors during
anonymization.

###  5.2 EFFECT ON RANKING

In this paper, we presented new evaluation models and strategies for training
the lazy-informed attackers as alternatives to the evaluation framework of the
VPC. We have shown that the perceived strength of an SAS depends on the models
it has been evaluated with, however, it is also important to analyze how the
relative performance of multiple SAS in comparison (i.e., their ranking) is
influenced by the choice of evaluation models.

Comparing the scores for different SAS in the lazy- and semi-informed attack
conditions as shown in Table 3 reveals consistently the same ranking for the
level of privacy protection (with GAN\- and OHNN-based being partly on equal
places): (1) GAN-based, (2) OHNN-based, (3) BL 1.a, and (4) BL 1.b. This is
regardless of whether the VPC evaluation models or the proposed ones are
applied, and also regardless of the training strategy. For ASR evaluation, on
the other hand, the ranking of SASs’ performances does not stay consistent but
changes depending on the ASR model used for evaluation. However, the WER
scores of all SASs are relatively similar to each other when using the
proposed evaluation models. Thus, we argue that this change in ranking for
utility evaluation is a rather small effect.

##  6 DISCUSSION

Which evaluation models should we consider? For privacy evaluation, we
proposed and evaluated various attacker models and found that different attack
models do not influence the ranking of privacy protection ability for various
SASs, even though they produce different privacy scores. However, we only
tested a limited number of attackers from the same ASV family. It is possible
that an attacker using a different technique, e.g., conformer-based [38] or
SSL-based ASV models [39], might show different trends.

Moreover, the choice of attack conditions is still heavily based on
assumptions. It is unclear whether the semi-informed attackers are realistic
or what we could assume about the knowledge and dedication of real attackers.
Hopefully, challenges like a Voice Privacy Attacker Challenge555A Voice
Privacy Attacker Challenge was initially planned for a workshop at INTERSPEECH
2023, see https://www.voiceprivacychallenge.org. will provide new insights and
perspectives.

Overall, we saw a particular trade-off between the quality of objective
results and their usability in terms of time consumption for computation, at
least for privacy metrics. Retraining the $ASV_{\text{eval}}^{anon}$ from
scratch on the full anonymized training data seems to lead to the strongest
attacker. However, it is costly, which can be significantly reduced by a
finetuning strategy that leads to an attacker of only slightly worse strength.
We propose using this alternative training strategy at least during SAS
development to speed up voice privacy research. However, a full retraining on
all data might still be a better option to assess the full privacy
capabilities.

What are the drawbacks of the current evaluation metrics? A more problematic
metric, according to our experiments, is $\text{G}_{\text{VD}}$. We tested
three model approaches ($ASV_{\text{eval}}$, $ASV_{\text{eval}}^{anon}$, and
the combination of both), but their results differ considerably. It is unclear
which approach is better suited for measuring the preservation of voice
distinctiveness among anonymized speech. We conclude that we need a more
robust alternative, e.g., to use the anonymized dataset to perform downstream
speaker verification or speaker diarization tasks.

What is missing? An ideal state an SAS should reach to fulfill the requirement
of successful and guaranteed full privacy protection has yet to be defined. It
is unknown when an SAS could be considered good enough for using it on data
where privacy protection matters, or how otherwise the remaining privacy risk
could be measured666A first approach for privacy risk assessment was proposed
in the ZEBRA framework [28] but is so far not used as a standard metric in
research..

In summary, being open source, the proposed framework serves as a platform for
unifying researchers and research on this topic. Researchers can add their own
SASs and evaluation metrics to the framework such that large-scale and
extensive evaluations and comparisons would be possible without much
additional effort. In this way, we hope that this framework will help towards
finding answers to the questions above, and towards the development of
powerful anonymization tools.

##  7 CONCLUSION

We proposed a new Python-based and modular open-source framework for speaker
anonymization research. It allows combining, managing, and evaluating several
anonymization approaches within one platform that is simple to apply and
extend. We further present various improvements to standard evaluation
techniques for speaker anonymization. Specifically, we exchange previous
Kaldi-based evaluation models with more powerful techniques using the ESPnet
and SpeechBrain toolkits. Moreover, we showed that we could decrease the time
required for evaluation by up to 95% by reducing training and test data while
keeping the quality of the evaluations at comparative levels. We anticipate
that these changes to common development and evaluation procedures will
significantly facilitate and support speaker anonymization research in the
near future.

## References

  * [1] N. Tomashenko, X. Wang, E. Vincent, J. Patino, B. M. L. Srivastava, P.-G. Noé, A. Nautsch, N. Evans, J. Yamagishi, B. O’Brien, et al.,  “The VoicePrivacy 2020 challenge: Results and findings,”  Computer Speech & Language, vol. 74, no. C, Jul 2022.
  * [2] D. Povey, G. Cheng, Y. Wang, K. Li, H. Xu, M. Yarmohammadi, and S. Khudanpur,  “Semi-orthogonal low-rank matrix factorization for deep neural networks.,”  in Proc. Interspeech, 2018, pp. 3743–3747.
  * [3] N. Tomashenko, X. Wang, X. Miao, H. Nourtel, P. Champion, M. Todisco, E. Vincent, N. Evans, J. Yamagishi, and J. F. Bonastre,  “The VoicePrivacy 2022 challenge evaluation plan,”  arXiv preprint arXiv:2203.12468, 2022.
  * [4] D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlicek, Y. Qian, P. Schwarz, J. Silovsky, G. Stemmer, and K. Vesely,  “The kaldi speech recognition toolkit,”  in Proc. IEEE ASRU, Dec. 2011.
  * [5] S. Watanabe, T. Hori, S. Karita, T. Hayashi, J. Nishitoba, Y. Unno, N. Enrique Yalta Soplin, J. Heymann, M. Wiesner, N. Chen, A. Renduchintala, and T. Ochiai,  “ESPnet: End-to-end speech processing toolkit,”  in Proc. Interspeech, 2018, pp. 2207–2211.
  * [6] M. Ravanelli, T. Parcollet, P. Plantinga, A. Rouhe, S. Cornell, L. Lugosch, C. Subakan, N. Dawalatabad, A. Heba, J. Zhong, J.-C. Chou, S.-L. Yeh, S.-W. Fu, C.-F. Liao, E. Rastorgueva, F. Grondin, W. Aris, H. Na, Y. Gao, R. D. Mori, and Y. Bengio,  “SpeechBrain: A general-purpose speech toolkit,” 2021,  arXiv:2106.04624.
  * [7] S. Meyer, F. Lux, P. Denisov, J. Koch, P. Tilli, and N. T. Vu,  “Speaker anonymization with phonetic intermediate representations,”  in Proc. Interspeech, 2022, pp. 4925–4929.
  * [8] S. Meyer, P. Tilli, P. Denisov, F. Lux, J. Koch, and N. T. Vu,  “Anonymizing speech with generative adversarial networks to preserve speaker privacy,”  in Proc. IEEE SLT. IEEE, 2023, pp. 912–919.
  * [9] S. Meyer, F. Lux, J. Koch, P. Denisov, P. Tilli, and N. T. Vu,  “Prosody is not identity: A speaker anonymization approach using prosody cloning,”  in Proc. IEEE ICASSP. IEEE, 2023, pp. 1–5.
  * [10] X. Miao, X. Wang, E. Cooper, J. Yamagishi, and N. Tomashenko,  “Language-independent speaker anonymization approach using self-supervised pre-trained models,”  in Proc. Odyssey, 2022, pp. 279–286.
  * [11] X. Miao, X. Wang, E. Cooper, J. Yamagishi, and N. Tomashenko,  “Analyzing language-independent speaker anonymization framework under unseen conditions,”  in Proc. Interspeech, 2022, pp. 4426–4430.
  * [12] X. Miao, X. Wang, E. Cooper, J. Yamagishi, and N. Tomashenko,  “Language-independent speaker anonymization using orthogonal householder neural network,”  IEEE/ACM Trans. Audio, Speech, and Language Processing (in press), 2023,  arXiv:2305.18823.
  * [13] C. O. Mawalim, K. Galajit, J. Karnjana, S. Kidani, and M. Unoki,  “Speaker anonymization by modifying fundamental frequency and x-vector singular value,”  Computer Speech & Language, vol. 73, pp. 101326, 2022.
  * [14] P. Champion, A. Larcher, and D. Jouvet,  “Are disentangled representations all you need to build speaker anonymization systems?,”  in Proc. Interspeech, 2022, pp. 2793–2797.
  * [15] A. S. Shamsabadi, B. M. L. Srivastava, A. Bellet, N. Vauquier, E. Vincent, M. Maouche, M. Tommasi, and N. Papernot,  “Differentially private speaker anonymization,”  Proc. Privacy Enhancing Technologies, vol. 2023, no. 1, pp. 98–114, Jan. 2023.
  * [16] H. Turner, G. Lovisotto, and I. Martinovic,  “Generating identities with mixture models for speaker anonymization,”  Computer Speech & Language, vol. 72, pp. 101318, 2022.
  * [17] J. Yao, Q. Wang, L. Zhang, P. Guo, Y. Liang, and L. Xie,  “NWPU-ASLP system for the voiceprivacy 2022 challenge,”  in Proc. 2nd Symp. on Security and Privacy in Speech Communication, 2022.
  * [18] V. Peddinti, D. Povey, and S. Khudanpur,  “A time delay neural network architecture for efficient modeling of long temporal contexts,”  in Proc. Interspeech, 2015, pp. 3214–3218.
  * [19] K. Kasi and S. A. Zahorian,  “Yet another algorithm for pitch tracking,”  in Proc. IEEE ICASSP, 2002, vol. 1, pp. 361–364.
  * [20] D. Snyder, D. Garcia-Romero, G. Sell, D. Povey, and S. Khudanpur,  “X-vectors: Robust dnn embeddings for speaker recognition,”  in Proc. IEEE ICASSP. IEEE, 2018, pp. 5329–5333.
  * [21] B. Desplanques, J. Thienpondt, and K. Demuynck,  “ECAPA-TDNN: Emphasized channel attention, propagation and aggregation in TDNN based speaker verification,”  in Proc. Interspeech, 2020, pp. 3830–3834.
  * [22] B. M. L. Srivastava, N. Tomashenko, X. Wang, E. Vincent, J. Yamagishi, M. Maouche, A. Bellet, and M. Tommasi,  “Design choices for x-vector based speaker anonymization,”  in Proc. Interspeech, 2020, pp. 1713–1717.
  * [23] X. Wang, S. Takaki, and J. Yamagishi,  “Neural source-filter-based waveform model for statistical parametric speech synthesis,”  in Proc. IEEE ICASSP, 2019, pp. 5916–5920.
  * [24] J. Kong, J. Kim, and J. Bae,  “HiFi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis,”  in Proc. NeurIPS, 2020, pp. 17022–17033.
  * [25] M. Gomez-Barrero, J. Galbally, C. Rathgeb, and C. Busch,  “General framework to evaluate unlinkability in biometric template protection systems,”  IEEE Trans. Information Forensics and Security, vol. 13, no. 6, pp. 1406–1420, 2018.
  * [26] M. Maouche, B. M. L. Srivastava, N. Vauquier, A. Bellet, M. Tommasi, and E. Vincent,  “A comparative study of speech anonymization metrics,”  in Proc. Interspeech, 2020, pp. 1708–1712.
  * [27] P.-G. Noé, J.-F. Bonastre, D. Matrouf, N. Tomashenko, A. Nautsch, and N. Evans,  “Speech pseudonymisation assessment using voice similarity matrices,”  in Proc. Interspeech, 2020, pp. 1718–1722.
  * [28] A. Nautsch, J. Patino, N. Tomashenko, J. Yamagishi, P.-G. Noé, J.-F. Bonastre, M. Todisco, and N. Evans,  “The privacy ZEBRA: Zero evidence biometric recognition assessment,”  in Proc. Interspeech, 2020, pp. 1698–1702.
  * [29] C. Franzreb, T. Polzehl, and S. Moeller,  “A comprehensive evaluation framework for speaker anonymization systems,”  in Proc. 3rd Symp. Security and Privacy in Speech Communication, 2023.
  * [30] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala,  “PyTorch: An imperative style, high-performance deep learning library,”  in Advances in Neural Information Processing Systems 32, pp. 8024–8035. Curran Associates, Inc., 2019.
  * [31] S. Watanabe, T. Hori, S. Kim, J. R. Hershey, and T. Hayashi,  “Hybrid CTC/attention architecture for end-to-end speech recognition,”  IEEE Journal of Selected Topics in Signal Processing, vol. 11, no. 8, pp. 1240–1253, 2017.
  * [32] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur,  “Librispeech: An ASR corpus based on public domain audio books,”  in Proc. IEEE ICASSP, 2015, pp. 5206–5210.
  * [33] J. Yamagishi, C. Veaux, and K. MacDonald,  “CSTR VCTK corpus: English multi-speaker corpus for CSTR voice cloning toolkit (version 0.92),” 2019.
  * [34] Q. Wang, K. A. Lee, and T. Liu,  “Scoring of large-margin embeddings for speaker verification: Cosine or PLDA?,”  in Proc. Interspeech, 2022, pp. 600–604.
  * [35] Z. Li, R. Xiao, H. Chen, Z. Zhao, W. Wang, and P. Zhang,  “How to make embeddings suitable for plda,”  Computer Speech & Language, vol. 81, pp. 101523, 2023.
  * [36] X. Anguera, S. Bozonnet, N. Evans, C. Fredouille, G. Friedland, and O. Vinyals,  “Speaker diarization: A review of recent research,”  IEEE/ACM Trans. Audio, Speech, and Language Processing, vol. 20, no. 2, pp. 356–370, 2012.
  * [37] M. Panariello, M. Todisco, and N. Evans,  “Vocoder drift in x-vector–based speaker anonymization,”  in Proc. Interspeech, 2023, pp. 2863–2867.
  * [38] Y. Zhang, Z. Lv, H. Wu, S. Zhang, P. Hu, Z. Wu, H.-y. Lee, and H. Meng,  “MFA-Conformer: Multi-scale feature aggregation conformer for automatic speaker verification,”  in Proc. Interspeech, 2022, pp. 306–310.
  * [39] S. Chen, Y. Wu, C. Wang, Z. Chen, Z. Chen, S. Liu, J. Wu, Y. Qiao, Furu W., J. Li, and X. Yu,  “Unispeech-sat: Universal speech representation learning with speaker aware pre-training,”  in Proc. IEEE ICASSP. IEEE, 2022, pp. 6152–6156.


# Proper orthogonal decomposition vs. Fourier analysis for extraction of
large-scale structures of thermal convection

Supriyo Paul  Computational Fluid Dynamics Team, Centre for Development of
Advanced Computing Pune,
Pune 411007, India111Present Address: Department of Physics, A.C. College,
Jalpaiguri 735101, India. E-mail<EMAIL_ADDRESS>
E-mail<EMAIL_ADDRESS>Mahendra K. Verma  Department of Physics, IIT
Kanpur, Kanpur 208016, India

###### Abstract

We performed a comparative study of extraction of large-scale flow structures
in Rayleigh Bénard convection using proper orthogonal decomposition (POD) and
Fourier analysis. We show that the free-slip basis functions capture the flow
profiles successfully for the no-slip boundary conditions. We observe that the
large-scale POD modes capture a larger fraction of total energy than the
Fourier modes. However, the Fourier modes capture the rarer flow structures
like flow reversals better. The flow profiles of the dominant POD and Fourier
modes are quite similar. Our results show that the Fourier analysis provides
an attractive alternative to POD analysis for capturing large-scale flow
structures.

###### keywords:

Proper Orthogonal Decomposition; Fourier Analysis; Convection

##  1 Introduction

In fluid flows, the large-scale structures play a major role in its dynamics.
Hence, an identification of such structures is critical for understanding
fluid flows. Proper Orthogonal Decomposition (POD) [1, 2, 3] is one of the
popular methods for this task. In this paper we employ a POD scheme called
“snapshot method” [4, 5], in which a number of uncorrelated and discrete time
snapshots of the flow field are used.

An alternate tool for identifying large-scale structures is Fourier analysis
[6], which is relatively easier to compute. Recently, Chandra and Verma [7]
showed that the Fourier modes play a critical role in the reversal dynamics of
turbulent convection. Note that a large number of low-dimensional models have
been constructed using the large-scale Fourier modes. There are several low-
dimensional models based on POD modes as well [8, 9], but Fourier modes are
more amenable for this purpose.

In the present work, we perform a comparative study between POD and Fourier
analysis by employing them to Rayleigh-Bénard convection in a 2D box. We will
emphasise similarities and dissimilarities between these two diagnostics
tools.

##  2 Large-scale Structures of RBC in a Two-dimensional Box

We simulate Rayleigh-Bénard convection (RBC) in a square box of unit
dimension. For the velocity field we assume no-slip boundary conditions
($u=v=0$) on all the walls, and for the temperature field we consider the top
and bottom walls to be perfectly conducting, while the side-walls to be
insulating. The relevant nondimensionalized equations under Boussinesq
approximation for the flow are

$\displaystyle\partial_{t}{\bf v}+({\bf v}\cdot\nabla){\bf v}$
$\displaystyle=$ $\displaystyle-\nabla P+RaPrT\hat{y}+Pr\nabla^{2}{\bf v},$
(1) $\displaystyle\partial_{t}T+({\bf v}\cdot\nabla)T$ $\displaystyle=$
$\displaystyle\nabla^{2}T,$ (2) $\displaystyle\nabla\cdot{\bf v}$
$\displaystyle=$ $\displaystyle 0.$ (3)

where ${\mathbf{v}}=u\hat{x}+v\hat{y}$ is the velocity field, $T$ is the
temperature field, the Rayleigh number $Ra$ is the ratio of the buoyancy term
and the nonlinear term, the Prandtl number $Pr$ is the ratio of the kinematic
viscosity and the thermal diffusivity, and $\hat{y}$ is the buoyancy
direction.

We solve the equations in a unit square box for Prandtl number $Pr=1$, and
Rayleigh number $Ra=2\times 10^{7}$. For this parameter, the flow is
turbulent. The flow also exhibits flow reversals, that is, probes near the
vertical walls exhibit random reversals of the velocity field. For the
simulation we use spectral element code NEK5000 [10], and employ $28\times 28$
spectral elements with seventh-order polynomials inside each element. Thus the
overall resolution of the simulation is $196\times 196$ grids. The
aforementioned resolution is sufficient to resolve the boundary layers. We
remark that a two-dimensional RBC in an experiment can be realised when the
depth of the setup is much smaller compared to the height and the width; in
such systems, the modes along the depth are not generated in a significant
manner.

We ran our simulation till the system attains a steady state. During the
steady state, we focus our attention on the flow during an interval from
thermal diffusive time $t_{A}=12.765$ to $t_{B}=12.792$, between which a flow
reversal is observed. We analyse the dominant flow structures during this
interval.

For the POD analysis[4, 5] we take 1000 snapshots of the flow in the
aforementioned interval in an equidistant manner. We interpolate the
simulation data on a uniform mesh of resolution ($192\times 192$). For the POD
analysis, we construct 1000 vectors using the two components of the velocity
and the temperature field. After this we construct a correlation matrix of
these vectors[4, 5]. We construct the first ten most energetic POD modes using
the correlation matrix[4, 5]. We extract the first 10 POD modes, which contain
97.5% of the total energy. In Fig. 1 we illustrate the energy ratio
$E_{p}/E_{1}$, where $E_{p}$ represents the energy of the $p$-th POD mode.
Clearly, $E_{p}/E_{1}$ decreases sharply with $p$ ($E_{1}/E_{2}\sim 14$).
Also, the first POD mode contains 88% of the total energy. The first three POD
modes are exhibited in Fig. 2 as $P1$-$P3$.

Figure 1: Ratio of the energies of different POD modes with the 1st POD mode.
Also the ratio $E_{m,n}/E_{1,1}$, where $E_{m,n}$ is the energy of the $(m,n)$
Fourier mode.

The flow velocity at all the four walls is zero, i.e., ${\bf v}=0$. A spectral
decomposition of the no-slip boundary condition involves Chebyshev polynomials
that resolve the boundary layers quite efficiently. Note however that the flow
structures in the boundary layers are of “small scales”, and they can be
ignored while computing the large-scale flow structures. Chandra and Verma [7]
observed that the free-slip basis functions defined below

$\displaystyle u$ $\displaystyle=$
$\displaystyle\sum\limits_{m,n}\hat{u}_{m,n}({\bf k})\sin(m\pi x)\cos(n\pi y)$
$\displaystyle v$ $\displaystyle=$
$\displaystyle\sum\limits_{m,n}\hat{v}_{m,n}({\bf k})\cos(m\pi x)\sin(n\pi y)$
(4)

capture the large-scale flow structures quite well. Here $k_{x}=m\pi$,
$k_{y}=n\pi$ with $m,n$ as positive integers, $\hat{u}_{m,n},\hat{v}_{m,n}$
are the Fourier components of $u$ and $v$ respectively [7]. Note that the
free-slip basis functions do not satisfy the no-slip boundary condition. The
success of the free-slip basis functions in capturing the large-scale
structures is due to the fact that ignored modes at the boundary layers belong
to the small-scale structures.

The temperature field satisfies the conducting boundary conditions at the
horizontal plates ($T$=constant), and insulating boundary conditions
($\partial T/\partial x=0$) at the vertical walls. It is customary to separate
$T$ into conducting and convective part, i.e.,

$T(x,y)=\bar{T}(y)+\theta(x,y)$ (5)

where $\bar{T}(y)=1-y$ is the conduction profile for the nondimensionalized
system, and $\theta$ is the temperature fluctuations over $\bar{T}(y)$. The
aforementioned thermal boundary conditions yields:

$\displaystyle\theta$ $\displaystyle=$
$\displaystyle\sum\limits_{m,n}\hat{\theta}_{m,n}({\bf k})\cos(m\pi
x)\sin(n\pi y)$ (6)

The Fourier transforms can be performed for each snapshot independently, which
is one of the main advantages of this analysis, contrary to POD that requires
many snapshots. However to study the dynamics and evolution of the flow
structures during a reversal, we study 1000 snapshots of the flow. We compute
the Fourier modes for each frame [Eqs. (4, 6)]. The energy of the higher
wavenumber modes decreases sharply, consistent with the Kolmogorov theory of
turbulence. Since our focus is on the large-scale structures, we take the
first 10 Fourier modes that contain approximately 93% of the total kinetic
energy. In Fig. 1 we plot the ratio $E_{m,n}/E_{1,1}$, where $E_{m,n}$
represents the kinetic energy of the $(m,n)$ Fourier mode. The Fourier
analysis reveals that the modes $(1,1)$, $(2,2)$, and $(1,3)$ are the most
dominant modes in the flow. The first Fourier mode has 58% of the total
energy. The Fourier modes of the flow are shown in right column of Fig. 2 as
$F1$-$F3$.

Figure 2: Plots of the three most energetic POD modes (left column) and the
corresponding Fourier modes (right column). The top three POD modes are (P1)
POD-1, (P2) POD-2, and (P3) POD-3. The top three energetic Fourier modes are
(F1) (1,1) mode, (F2) (2,2) mode and (F3) (1,3) mode.

When we compare the two methods for the extraction of large-scale flow
structures, we observe that POD modes are more optimal than the Fourier modes.
For example, the first 10 POD and first 10 Fourier modes contain respectively
97.5% and 93% of the total energies. Also, the first POD mode contains 88% of
the total energy, but the corresponding Fourier mode contains only 58% of the
total energy. However, the flow structures of the first three POD modes are
distinctly similar in their shape to the first three Fourier modes, but the
higher order modes differ. Note however that the directions of the velocity
fields of the POD and Fourier modes are anti-correlated, hence the amplitudes
of these modes are also opposite to each other (to be discussed below).

The amplitude of a POD mode is computed by projecting the snapshot to the POD
mode. The time series of the first five Fourier and POD modes are exhibited in
Fig. 3 that shows that the amplitudes of the POD and Fourier modes are anti-
correlated to each other. This is consistent with the anti-correlation of the
velocity fields for the corresponding POD and Fourier modes (see Fig. 2).
During the flow reversal between $t=12.78$ and $t=12.785$, the first Fourier
mode, as well as the first POD mode, change sign. The second and third modes
also show sharp variations during the flow reversal [7]. The fourth and fifth
Fourier modes also show noticeable variations during the reversal process, but
the corresponding POD modes do not show noticeable variations during this
period. This is due to the fact that top three POD modes are most dominant in
the flow, while the fourth and the fifth POD modes are quite weak. Thus both
Fourier and POD modes provide valuable information on the dynamics of flow
reversal, details of which can be found in Sergent and Podvin [8, 9], and
Chandra and Verma [7].

The reconstruction of the snapshots using the POD and Fourier modes is shown
in Fig. 4 where we show the 5-th and 670-th snapshots of the flow. The 5-th
snapshot is reproduced accurately by both the POD and Fourier analysis.
However for the 670-th snapshot, which depicts the corner-roll flow structure,
POD reconstruction is poorer than the Fourier one. The reason for the
discrepancy is due to the averaging process of the POD analysis. On the
average, the first POD mode is more dominant than the second POD-mode
($E_{2}/E_{1}=1/14$). For the 670-th snapshot, the second POD mode is the most
important mode, but its contribution towards the reconstruction of the
snapshot get suppressed by the most dominant first POD mode. On the other
hand, Fourier modes correctly reconstructs the 670-th snapshot since each
snapshot has its own set of Fourier modes, and the (2,2) mode is more dominant
than the (1,1) mode for the 670-th snapshot. Hence Fourier reconstruction is
better than the POD reconstruction for dynamic flows.

Figure 3:  Plot of the time series of the most energetic five Fourier and POD
modes of the vertical velocity $v$ during the reversal. POD coefficients are
scaled appropriately for visual clarity. Figure 4: Reconstruction of the 5th
and 670th snapshots using the first ten POD and ten Fourier modes. Both the
POD and Fourier reconstruction are reasonably good, but the the 670th snapshot
is better reconstructed using the Fourier modes.

##  3 Conclusions and Discussions

In this paper we proposed that the low-wavenumber Fourier modes are good
candidates for identifying large-scale flow structures, and they provide an
alternative to POD analysis. In support of our argument, we perform a
comparative study of the POD and Fourier analysis of 1000 snapshots of the
flow profiles in a two-dimensional Rayleigh Bénard convection.

The low wavenumber Fourier modes capture the large-scale structures of the
flow quite well. However, the first $N$ POD modes contain more energy than the
corresponding Fourier modes. Due to this, a reproduction of a flow pattern
requires fewer POD modes than the Fourier modes, but the difference is not
very significant. Note however that Fourier modes can be computed for each
snapshot separately, but the POD analysis requires a large set of snapshots. A
simple estimation shows that the Fourier analysis is computationally less
expensive than the POD analysis. This is useful while dealing with large
three-dimensional datasets. Moreover, the Fourier modes have simpler visual
interpretations than the POD modes.

A major objection to the usage of Fourier analysis is its inapplicability to
no-slip boundary conditions. However, the small-scale structures of a boundary
layer do not contribute significantly to the large-scale structures. The RBC
example discussed in this paper and in Chandra and Verma [7] show that free-
slip basis can capture the large scale flow structures even under no-slip
geometries quite well.

Thus, POD and Fourier analysis have their own advantages and disadvantages.
For idealised geometries like a box, Fourier analysis provides an attractive
alternative to POD.

## Acknowledgment

The authors thank M. Chandra for sharing with us the RBC data. SP thanks
C-DAC’s CFD group for support and encouragement. The analysis was performed on
PARAM-YUVA.

## References

  * [1] D. D. Kosambi, J. Ind. Math. Soc., 7, 76 (1943).
  * [2] J. L. Lumley, “The structure of inhomogeneous turbulent flows,” in Atmospheric Turbulence and Radio Wave Propagation, (Nauka, Moscow), p. 166, (1967).
  * [3] G. Berkooz, P. Holmes, and J. L. Lumley, Annu. Rev. Fluid Mech. 25, 539 (1993).
  * [4] L. Sirovich, Q. Appl. Math. 45, 561 (1987); L. Sirovich, Q. Appl. Math. 45, 573 (1987); L. Sirovich, Q. Appl. Math. 45, 583 (1987).
  * [5] L. Sirovich, L., Maxey, and H. Tarman, “Analysis of turbulent thermal convection”, In Sixth Symposium on Turbulent Shear Flow, Toulouse, France, p. 68. Springer (1989); A. E. Deane and L. Sirovich, J. Fluid Mech. 222, 231 (1991).
  * [6] M. Lesieur,“Turbulence in Fluids”, (Kluwer Academic, Dordrecht), (1990).
  * [7] M. Chandra and M. K. Verma, Phys. Rev. E 83, 067303 (2011); M. Chandra and M. K. Verma, Phys. Rev. Lett. 110, 114503 (2013)
  * [8] B. Podvin and A. Sergent, Phys. Fluids 24, 105106 (2012); J. Fluid Mech. 766, 172 (2015).
  * [9] J. Bailon-Cuba, O. Shishkina, C. Wagner, and J. Schumacher, Phys. Fluids 24, 107101 (2012).
  * [10] Paul F. Fischer, James W. Lottes, and Stefan G. Kerkemeier, “nek5000 Web page”, http://nek5000.mcs.anl.gov, (2008).