# Minimal diagrams of composite flat knots

Jie Chen

###### Abstract.

The connected sum of two flat knots depends on the choice of diagrams and
basepoints. We show that any minimal crossing diagram of a composite flat knot
is a connected sum diagram. We also show the crossing number of flat knots is
super-additive under connected sum.

###### Key words and phrases:

flat knot, connected sum, composite knot

###### 2020 Mathematics Subject Classification:

57K10; 57K12

##  1\. Introduction

A famous conjecture in classical knot theory asserts that the crossing number
is additive under connected sum. It is known that the crossing number
satisfies $cr(K_{1}\\#K_{2})\leq cr(K_{1})+cr(K_{2})$, so it is sub-additive,
and additivity has been established for certain families of knots (e.g.
alternating knots). In general, though, it remains a difficult open problem to
prove additivity for all knots.

In this paper, we consider the analogous question about the crossing number of
the connected sum of _flat knots_. Flat knots were first introduced by
Kauffman [kauffman99], and they were also studied by Turaev [turaev04], who
called them _virtual strings_. A flat knot (or virtual string) can be
represented as a regular immersed curve $\alpha:S^{1}\to\Sigma$ in a compact
oriented surface $\Sigma$, up to homotopy. The operation of connected sum is
not well-defined; it depends on the choice of diagrams and basepoints on the
flat knots.

The main result in this paper is Theorem 3.5; it shows that any minimal
crossing diagram of a composite flat knot is a connected sum diagram. By way
of application, we use it to show that the crossing number is super-additive
under connected sum of flat knots. Namely, we show that
$cr(\alpha_{1}\\#\alpha_{2})\geq cr(\alpha_{1})+cr(\alpha_{2})$ (Corollary 1).
This inequality holds for any composite flat knot obtained as the connected
sum of $\alpha_{1}$ and $\alpha_{2}$, regardless of the choice of diagrams and
basepoints. There are examples to show the inequality is sometimes strict, but
if the connected sum is formed using two minimal crossing diagrams, then
Proposition 3.3 applies to give an equality of crossing numbers.

##  2\. Preliminaries

In this section, we introduce flat knots and links. They can be described
alternatively in terms of flat link diagrams or flat Gauss diagrams. The flat
Reidemeister moves generate an equivalence relation called homotopy.

######  Definition 2.1.

A _flat link diagram_ is a $4$-valent planar graph, where each vertex is now
allowed to be either a _flat crossing_ or virtual crossing as in Figure 1.

Figure 1. A flat crossing (left) and virtual crossing (right)

######  Definition 2.2.

Two flat link diagrams are said to be _homotopic_ if they are related by a
finite sequence of _flat Reidemeister moves_ shown in Figure 2 along with
planar isotopies.

Figure 2. Flat Reidemeister moves

For a flat link diagram $D$, its _crossing number_ is denoted $cr(D)$ and
defined as the number of flat crossings in $D$. Note that the virtual
crossings of $D$ do not contribute to $cr(D).$ The crossing number of a flat
link $\alpha$ is denoted $cr(\alpha)$ and defined as the minimum, over all
diagrams $D$ for $\alpha$, of the crossing number of $D$. Thus, $cr(\alpha)$
is an invariant of the flat link.

Flat links arise naturally from virtual links by flattening all of the
classical crossings [kauffman99]. When we refer to a component of a flat link,
we mean a connected component of a virtual link overlying it. In this paper,
we will be mainly interested in flat knots, which are flat links with one
component.

######  Definition 2.3.

For a flat knot diagram with $n$ classical crossings, its _flat Gauss diagram_
is a counterclockwise oriented skeleton with $2n$ points and $n$ arrows. Every
arrow encodes a crossing as shown in Figure 3. The order of the points on the
skeleton determines the adjacency of the crossings in the flat knot diagram.

Figure 3. Flat crossings to Gauss diagram arrows

Figure 4. Flat virtual trefoil and its Gauss diagram

Every flat knot diagrams determines an associated Gauss diagram, and
conversely any Gauss diagram determines a flat knot diagram. The flat
Reidemeister moves translate into moves on Gauss diagrams, and these are shown
in Figure 5. Note that VR1, VR2, VR3 and FR4 do not change the flat Gauss
diagram.

For any classical knot, there is a sequence of Reidemeister moves to convert
any given diagram into a a minimal crossing diagram. However, the sequence may
increase the number of crossings before achieving a minimal crossing diagram.

For flat knots, the following result shows that one can convert any flat knot
diagram into a minimal crossing diagram without introducing any new crossings.
The following theorem is a statement of the monotonicity result for flat
knots, [hassscott, manturovbook, cahn, david].

######  Theorem 2.4.

For any flat knot diagram, there exists a sequence of flat Reidemeister moves
such that the number of crossings monotonically decreases until one achieves a
minimal crossing diagram. Further, any two minimal diagrams of the same flat
knot are related by a sequence of FR3 moves.

The monotonicity result for flat knots and links has a somewhat storied
history. In [hassscott], Hass and Scott showed that any curve on a surface can
be monotonically reduced to one with minimal intersection number. In
[MR2008880], Kadokami claimed that any flat link diagram can be monotonically
reduced to a minimal crossing diagram, but Gibson found counterexamples in
[gibson]. In [cahn], Cahn proved Kadokami’s claim for flat knots, and in
[david], Freund gave a complete solution by proving Kadokami’s claim for
nonparallel flat links.

Figure 5. Flat Reidemeister moves on Gauss diagrams

##  3\. Composite flat knots and crossing number

In this section, we define the connected sum for flat knots and consider the
crossing number of composite flat knots. We will show that every minimal
crossing diagram of a composite flat knot is a connected sum diagram.

We assume all diagrams in this section are oriented. We use $D^{\bullet}$ to
denote an oriented flat knot diagram $D$ together with a choice of basepoint
$p$ on it. Given two flat knot diagrams $D_{1},D_{2}$ with basepoints
$p_{1},p_{2}$, the connected sum is denoted
$D_{1}^{\bullet}\\#D_{2}^{\bullet}$. The operation of connected sum is not
well-defined for flat knots; it depends on the diagrams and basepoints, cf.
Figure 6.

This example also shows that the decomposition theorem, a theorem stating
every flat knot can be uniquely decomposed as a connected sum of nontrivial
prime flat knots, does not exist for flat knots; also see [Matveev].

Figure 6. Connected sum of flat knots depends on basepoints

######  Definition 3.1.

A flat knot $\alpha$ is said to be _composite_ if it can be represented by a
connected sum diagram $D_{1}^{\bullet}\\#D_{2}^{\bullet}$ such that $\alpha$
is not equivalent to $D_{1}$ or $D_{2}$. A flat knot $\alpha$ is said to be
_prime_ if it is not a composite flat knot.

######  Definition 3.2.

Let $D_{1},D_{2}$ be flat knot Gauss diagrams. The _set of the permutant
diagrams_ is defined to be the set of the connected sum Gauss diagrams

$[D_{1}\\#D_{2}]=\\{D_{1}^{\bullet}\\#D_{2}^{\bullet}\mid
D_{i}^{\bullet}\text{ equals $D_{i}$ with a choice of basepoint}\\}.$

Note that the connected sum of two trivial flat knot diagrams can produce a
nontrivial flat knot. For example, the flat knots in Figure 7 are nontrivial
connected sums of two and three trivial diagrams, respectively. Similarly, one
can construct a nontrivial flat knot as the connected sum of $n$ trivial
diagrams. These flat knots can be shown to be different by computing their
based matrices and associated characteristic polynomials, see [turaev04, chen-
thesis].

Figure 7. Flat knots 4.10 and 6.2064

######  Proposition 3.3.

Let $D_{1}$ and $D_{2}$ be two minimal crossing flat Gauss diagrams. Then any
permutant diagram $D\in[D_{1}\\#D_{2}]$ is a minimal crossing number diagram
of its flat knot type.

###### Proof.

Obviously, $cr(D)=cr(D_{1})+cr(D_{2})$. For the sake of contradiction, suppose
the crossing number of $D$ can be reduced. By Theorem 2.4, the process must
involve finitely many FR3 moves in Figure 9 and decreasing FR1, FR2 moves in
Figure 8.

Suppose FR3 can be applied on three arrows, with one from $D_{1}$ and two from
$D_{2}$. This can only happen in the versions a, b, c and d of FR3, in the
left-to-right direction, as shown in Figure 9. However, such Gauss diagrams of
$D_{1}$ or $D_{2}$ are not minimal crossing. For example, as shown in Figure
11, if the connected sum is formed at the red dashed line, then applying an
FR1 on $D_{1}$ reduces the crossing number. The same argument applies to
FR3(b,c,d) as well. Therefore, no FR3 moves can be applied so that the arrows
involved are from both $D_{1},D_{2}$. Since $D_{1},D_{2}$ are minimal Gauss
diagrams, the diagram obtained after FR3 moves should still be a connected sum
diagram of minimal crossing Gauss diagrams.

Now consider the decreasing FR1,2 applied on a connected sum diagram. Any such
move applied to $D$ can be also applied to either $D_{1}$ or $D_{2}$. Then no
FR1,2 can be applied to $D$ to reduce its crossing number. ∎

Figure 8. The decreasing FR1 and FR2 moves on Gauss diagrams

Figure 9. FR3 moves on Gauss diagrams

A _long flat knot diagram_ is an oriented $4$-valent planar graph with only
flat crossings and virtual crossings that coincides with the $x$-axis outside
of some compact set; see Figure 10. Two long flat knot diagrams are said to be
_equivalent_ if they are related by a finite sequence of flat Reidemeister
moves and planar isotopies. A _long flat knot_ is an equivalence class of long
flat knot diagrams.

The _closure_ of a long virtual knot is the flat knot obtained by replacing
the $x$-axis outside the compact set with a half circle. If $D$ is a flat knot
diagram, then removing a point from $D$ gives a long flat knot diagram
$D^{\bullet}$, and the closure of $D^{\bullet}$ is $D$.

Figure 10. A long flat knot diagram

######  Lemma 3.4.

Let $D$ be a connected-sum diagram $D=D_{1}^{\bullet}\\#D_{2}^{\bullet}$,
where $D_{1}^{\bullet}$ and $D_{2}^{\bullet}$ are both nontrivial as long flat
knots. Among all flat Reidemeister moves, only increasing FR2-moves can change
$D$ to a non-connected-sum diagram.

###### Proof.

Observe that if a diagram on the left hand side of a decreasing FR1,2-move in
Figure 8 is a connected sum, then so is the right hand side. Moreover, no FR3
move can transform a connected-sum diagram to a non-connected sum: as shown in
Figure 11 if the connected sum is addressed at the red dashed line, then the
blue dash-dotted line addresses the connected sum on the right hand side.
Other FR3 moves follow similarly.

Figure 11. FR3(a) move on a Gauss diagram

∎

######  Theorem 3.5.

Any minimal crossing diagram of a composite flat knot is a connected sum
diagram.

###### Proof.

A composite round flat knot has a Gauss diagram
$D=D_{1}^{\bullet}\\#D_{2}^{\bullet}$, where $D_{1}^{\bullet}$ and
$D_{2}^{\bullet}$ are two Gauss diagrams with a choice of basepoint on them,
and removing the basepoint from either of them produces a nontrivial long flat
knot. Assume $D^{\prime}$ is a minimal crossing diagram that is flat
equivalent to $D$. By Theorem 2.4, $D^{\prime}$ is obtained from $D$ by
finitely many FR3-moves and decreasing FR1,2-moves. Then by Lemma 3.4,
$D^{\prime}$ is a connected sum diagram. ∎

Primality is a flat knot invariant. We can apply Theorem 3.5 to the tabulation
process: if a minimal diagram of a flat knot $\alpha$ is not a connected sum,
then $\alpha$ is a prime flat knot. One can also find a minimal crossing
diagram for a flat knot in [flatknotinfo]. Furthermore, it also implies that
the flat knot crossing number is super-additive for connected sums.

######  Corollary 3.6.

For any $D\in[D_{1}\\#D_{2}]$, let $\alpha,\alpha_{1},\alpha_{2}$ be the flat
knot types of $D,D_{1},D_{2}$, respectively. Then the crossing number
satisfies

(1) $cr(\alpha)\geq cr(\alpha_{1})+cr(\alpha_{2})$

###### Proof.

The inequality holds if $cr(\alpha_{i})=0$ for $i=1$ or $i=2$.

Assume $\alpha$ is a composite knot and $cr(\alpha_{i})>0$ for $i=1,2$. Then
by Theorem 3.5, a minimal diagram $D^{\prime}$ of $\alpha$ is a connected sum
Gauss diagram. Then $D^{\prime}\in[D^{\prime}_{1}\\#D^{\prime}_{2}]$ for some
Gauss diagrams $D^{\prime}_{1},D^{\prime}_{2}$ of $\alpha_{1},\alpha_{2}$,
respectively. Then we have

$cr(\alpha)=cr(D^{\prime})=cr(D^{\prime}_{1})+cr(D^{\prime}_{2})\geq
cr(\alpha_{1})+cr(\alpha_{2}).$

∎

For example, the flat knot diagrams on the left of Figure 6 are equivalent to
the flat unknot. However, their connected sum, shown on the right top, is a
nontrivial flat knot with $4$ crossings. It is also worth mentioning that
Proposition 3.3 implies that equality holds in Equation (1) when the summands
$D_{1}$ and $D_{2}$ are minimal crossing diagrams.

## Acknowledgements

The results in this paper are derived from the Ph.D. thesis [chen-thesis].

## References

Department of Mathematics & Statistics, McMaster University
Hamilton, Ontario, Canada L8S 4K1

E-mail address<EMAIL_ADDRESS>


rowsep=0pt

# Speaker anonymization using neural audio codec language models

###### Abstract

The vast majority of approaches to speaker anonymization involve the
extraction of fundamental frequency estimates, linguistic features and a
speaker embedding which is perturbed to obfuscate the speaker identity before
an anonymized speech waveform is resynthesized using a vocoder. Recent work
has shown that x-vector transformations are difficult to control consistently:
other sources of speaker information contained within fundamental frequency
and linguistic features are re-entangled upon vocoding, meaning that
anonymized speech signals still contain speaker information. We propose an
approach based upon neural audio codecs (NACs), which are known to generate
high-quality synthetic speech when combined with language models. NACs use
quantized codes, which are known to effectively bottleneck speaker-related
information: we demonstrate the potential of speaker anonymization systems
based on NAC language modeling by applying the evaluation framework of the
Voice Privacy Challenge 2022.

Index Terms—  Speaker anonymization, neural audio codec, language modeling

##  1 Introduction

_Speaker anonymization_ involves the task of processing a speech signal to
conceal the identity of the speaker while retaining the spoken content and
other para-linguistic attributes such as intonation and prosody. As defined by
the Voice Privacy Challenge [1], a speaker anonymization system should provide
a certain trade-off between _privacy protection_ and _utility preservation_.
The former is measured by the difficulty of an attacker to recover the
identity of the original speaker from an anonymized signal via automatic
speaker verification (ASV). The latter is assessed primarily by the
reliability of an automatic speech recognition (ASR) system to transcribe the
anonymized waveform, among other, secondary metrics such as pitch correlation
and gain of voice distinctiveness [2].

Most speaker anonymization systems are based on often-incremental changes to
original work in [3], which operates upon three distinct components extracted
from an input waveform: an F0 curve encoding prosodic information; some form
of linguistic features encoding the spoken content; an x-vector [4] encoding
the speaker identity. The x-vector is perturbed to conceal the identity of the
speaker and then fed to a vocoder with the other two components in order to
synthesize an anonymized waveform. This approach assumes that speaker
information is contained entirely within the x-vector, even if this is known
not to be the case [5, 6]. Residual speaker information captured in linguistic
features and the F0 curve is re-entangled with the anonymized x-vector upon
vocoding. An x-vector extracted afresh from the anonymized utterance then
still contains speaker information which can be used by an adversary to
reverse the anonymization and re-identify the speaker [5, 7]. Other
researchers [5] have found that the level of speaker information contained in
linguistic features can be reduced through their quantization.

Motivated by their successful application to numerous audio synthesis tasks
[8, 9], we have sought to exploit the potential of neural audio codec (NAC)
language modeling to design a speaker anonymization system that better
suppresses speaker information and hence provides an improved trade-off
between anonymization and utility. Such an approach is appealing since
linguistic features are not used directly for waveform synthesis, as with
previous approaches: instead, they are used to infer a set of NAC acoustic
tokens with a language model. These features are quantized and therefore have
the potential to bottleneck speaker information and improve anonymization. The
final waveform is synthesized by decoding the acoustic tokens with a decoder
neural network. We hope that the representational power of NACs should help to
preserve speech quality and utility.

##  2 Related work

In the following we describe some related research which provided motivation
for the work presented in this paper.

X-vector–based speaker anonimization - The original x-vector–based pipeline
introduced in [3] is the basis of much of the related work reported recently.
An approach to dispense with the intermediate acoustic model was proposed in
[2]. More refined x-vector anonymization functions were proposed in later work
[10, 11], with some [12] achieving notable improvements to privacy protection
levels, albeit under the assumption that the attacker does not have full
knowledge of the anonymization system. Whatever the approach, x-vector
perturbation does not prevent speaker-related information contained in the F0
curve and linguistic features from _leaking_ into the anonymized waveform upon
vocoding. As a result, the x-vector which can be re-extracted from the
anonymized waveform by a privacy adversary who wishes to reidentify the
speaker tends to _drift_ away from that at the vocoder input [13]. While the
drift can be beneficial to anonymization, it hinders the design of more
effective anonymization functions and can also be inverted by an adversary to
undo the privacy protection [14]. Attempts to sanitize speaker information
from linguistic features have also been explored, e.g. [7] based on the
concept of differential privacy, which reports improvements to privacy at the
cost of reduced utility and pitch correlation. The same issue was tackled in
[5] by means of feature quantization, which is shown to be effective as a
speaker information bottleneck, though with a degradation to utility. In this
paper, we follow a similar approach, but propose a completely new synthesis
pipeline in which we avoid the leakage of information between different speech
components by design.

NAC language modeling - NACs were proposed recently for audio compression [15,
16]. They consist in convolutional autoencoders that compress audio to low-
bitrate, tokenized representations which support high-fidelity decoding. Due
to their discretized nature, encoded representations can be modeled using
techniques normally used for language-related tasks, such as transformers.
This idea was first introduced in [8] for a variety of audio generation tasks,
and tailored to text-to-speech (TTS) in [9]. A transformer is used to convert
the input (be it audio or text) to a set of high-level _semantic tokens_.
These are then fed to another transformer which converts them into _NAC
acoustic tokens_ which can be decoded to resynthesize an audio signal.

The same technique can also be applied to voice conversion [8, 17], and is
hence ideally suited to speaker anonymization. NAC language models operate on
quantized codes, which are known to be beneficial to privacy protection [5].
Moreover, such models appear to naturally disentangle linguistic information
into semantic tokens, while encoding speaker information and recording
conditions mostly in acoustic tokens [8]. Hence, we propose a speaker
anonymization system whereby an input utterance is re-synthesized by means of
a NAC language model. Semantic tokens are kept unchanged, while acoustic
tokens are substituted with those of a different speaker, the goal being to
preserve the linguistic content of the signal while suppressing information
related to the original speaker.

Fig. 1: Diagram of the proposed anonymization system.

##  3 Proposed approach

###  3.1 Neural audio codec language modeling

A diagram of the proposed system is shown in Figure 1. Following [9], it is
comprised of a semantic encoder, a NAC (encoder and decoder), a pair of
transformers and a pool of speaker prompts. They are described in the
following.

The semantic encoder produces high-level semantic representations of the input
signal using a codebook of $N_{S}$ quantized embeddings. The output is a
sequence of integers $\mathbf{s}\in\\{1,\dots,N_{S}\\}^{T_{S}}$, where $T_{S}$
is the number of frames and where each integer is a codeword index.

The NAC is an encoder-decoder architecture. The encoder maps input waveforms
to a quantized, compressed representation from which the decoder reconstructs
a high-fidelity waveform. Efficient compression is achieved with a set of $Q$
hierarchical codebooks. Lower level codebooks capture coarser waveform
characteristics, while finer details are captured by higher level codebooks.
Following [8, 9], we refer to the first $Q_{C}$ codebooks as ‘coarse
codebooks’, and to the last $Q-Q_{C}$ codebooks as ‘fine codebooks’. All have
$N_{Q}$ codewords so that the output of the encoder is
$\mathbf{\tilde{\mathbf{a}}}\in\\{1,\dots,N_{Q}\\}^{Q\times T_{A}}$, where
$T_{A}$ is the number of frames into which the input is divided.

The coarse and fine transformers estimate a set of acoustic tokens
$\mathbf{a}$ from a prompt of input semantic tokens $\mathbf{s}$ and acoustic
tokens $\mathbf{\tilde{\mathbf{a}}}$. Essentially, the transformers attempt to
predict what semantic information should ‘sound like’ in the domain of
quantized acoustic tokens. The coarse transformer autoregressively predicts
coarse acoustic tokens, i.e. the codewords belonging to the coarse codebooks.
More specifically, for frame $t$, the transformer predicts the probability
distribution of token $\mathbf{a}_{q,t}$ conditioned on the following
elements: the semantic prompt $\mathbf{s}$, the coarse tokens from the
acoustic prompt $\mathbf{\tilde{\mathbf{a}}}_{<Q_{C},:}$, and all previous
predictions.111In practice, the sequence upon which to perform regression is
flattened to
$(\mathbf{s},\mathbf{\tilde{\mathbf{a}}},\mathbf{a}_{1,1},\mathbf{a}_{2,1},\dots,\mathbf{a}_{Q_{C},1},\mathbf{a}_{1,2},\mathbf{a}_{2,2},\dots,\mathbf{a}_{Q_{C},2},\dots).$
See [8, 9] for more details. The modeled distribution is therefore

$p\left(\mathbf{a}_{q,t}\middle|\mathbf{s},\mathbf{\tilde{\mathbf{a}}}_{<Q_{C},:},\mathbf{a}_{<Q_{C},<t},\mathbf{a}_{<q,t}\right)$
(1)

for $q\in[1,Q_{C}]$. The fine transformer is instead non-autoregressive. It
estimates the tokens of codebook $q$ using all tokens belonging to codebooks
$<q$ and all tokens of the acoustic prompt $\mathbf{\tilde{\mathbf{a}}}$, thus
modeling the distribution

$p\left(\mathbf{a}_{q,:}\middle|\mathbf{\tilde{\mathbf{a}}},\mathbf{a}_{<q,:}\right)$
(2)

for every $q\in[Q_{C}+1,Q].$ Once the acoustic tokens $\mathbf{a}$ have been
predicted for all codebooks $q\in[1,Q]$, they can be input into the decoder to
synthesize an anonymized waveform.

The pool of speaker prompts is a set of acoustic tokens extracted by the NAC
encoder from utterances belonging to a set of external speakers. Those
speakers are referred to as _pseudo-speakers_ , since they replace the
original speaker in the anonymized utterance. As suggested in [8], acoustic
tokens, especially the coarse tokens, can capture information related to the
speaker identity. We use them to perform voice conversion, as detailed in the
following.

###  3.2 Anonymization technique

A set of semantic tokens $\mathbf{s}$ is first extracted from the input
utterance. They encode the high-level spoken content. Their quantization helps
to suppress speaker-related information.

A pseudo-speaker is chosen by randomly selecting an acoustic prompt
$\mathbf{\tilde{\mathbf{a}}}$ from the speaker prompt pool. Anonymization can
be performed at either speaker or utterance levels. At the _speaker level_ ,
anonymization is performed using the same speaker prompt for each utterance
corresponding to any one speaker. In contrast, for _utterance level_
anonymization, a speaker prompt is selected at random for each utterance.
While several anonymization systems include techniques to synthesize
fictitious voices [5, 10, 11, 12], here we use real voices as pseudo-speakers
to focus our analysis on the intrinsic anonymization capability of the NAC
language model.

Prompted with $\mathbf{s}$ and $\mathbf{\tilde{\mathbf{a}}}$, the coarse and
fine transformers generate a set of acoustic tokens $\mathbf{a}$ which reflect
the semantic information of the original utterance, but the acoustic
characteristics of the pseudo-speaker. Acoustic tokens $\mathbf{a}$ are fed to
the decoder which synthesizes the anonymized output waveform.

System | LibriSpeech |  |  |  | VCTK |  |  |
---|---|---|---|---|---|---|---|---
| EER (%) | WER (%) | $G_{VD}$ | $\boldsymbol{\rho}^{F_{0}}$ | EER (%) | WER (%) | $G_{VD}$ | $\boldsymbol{\rho}^{F_{0}}$
Original | 4.4 | 4.2 | 0 | 1 | 3.2 | 12.8 | 0 | 1
Original (eval. pipeline of [18]) | 1.5 | 2.5 | n.a. |  | 1.1 | 7.6 | n.a. |
B1b [2] | 8.6 | 4.4 | -5.8 | 0.78 | 9.7 | 10.7 | -7.1 | 0.81
T11 [10] | 20.6 | 3.9 | -19.0 | 0.68 | 39.7 | 7.9 | -18.4 | 0.73
Champion et al. [5] | 17.5 | 4.5 | n.a. | 0.67 | 28.0 | 10.0 | n.a. | 0.73
Champion et al. (noise on F0) [5] | 23.4 | 4.6 | n.a. | 0.52 | 40.8 | 10.3 | n.a. | 0.60
Ours | 28.5 | 7.5 | -1.5 | 0.68 | 45.5 | 18.9 | -2.1 | 0.74
Ours (eval. pipeline of [18]) | 34.1 | 4.6 | n.a. |  | 36.6 | 15.5 | n.a. |
Table 1: Results of the analyzed systems on the Voice Privacy Challenge 2022
test subsets.

##  4 Experimental setup

Our codebase is branched from Bark,222The original source code is available at
www.github.com/suno-ai/bark, though we built our system from the port included
in the CoquiTTS library available at www.github.com/coqui-ai/TTS. Our source
code, as well as audio samples, will be made available upon publication. an
open-source, NAC-based TTS system which is very similar to VALL-E [9]. The
modules described in Section 3.1 are all taken from Bark. The semantic encoder
has a HuBERT backbone [19] and a LSTM [20] back-end which predicts the
semantic token associated to the HuBERT feature vector output at each frame.
The semantic dictionary is of size $N_{S}=10000$. The coarse and fine
transformers are 12-layer GPT-like models [21], each having $Q=8$ different
codebooks with $N_{Q}=1024$ codewords. The first $Q_{C}=2$ codebooks are
considered coarse. The NAC is EnCodec [16]. The difference between Bark and
our system is that, being a TTS model, Bark estimates semantic tokens
$\mathbf{s}$ corresponding to an input text using a further _semantic
transformer_. In our implementation, we bypass the semantic transformer and
use ground truth semantic tokens from the input waveform thereby providing for
voice conversion instead of TTS. With this setup, we are able to use pre-
trained Bark modules off-the-shelf, without the need for any training.

We adopt the Voice Privacy Challenge 2022 protocol [2] for evaluation. The
test set comprises subsets of the _LibriSpeech_ [22] and _VCTK_ [23]
databases. The pool of speaker prompts is taken from the Bark voice library.
It consists of 130 utterances collected from speakers of different gender and
nationality.333www.github.com/suno-ai/bark/tree/main/bark/assets/prompts/v2
The threat model is the _semi-informed_ attack described in [2]. Trial
utterances are anonymized at the _speaker level_. The attacker is assumed to
have access to the anonymization system. They anonymize a set of external data
(librispeech-clean-360) at the _utterance level_ and use it to train an ASV
system (a TDNN with a PLDA back-end [24]). They also have access to original
(non-protected) enrollment utterances which they anonymize at the _speaker
level_. The attacker thus has enrollment and trial utterances both of which
are anoymized and uses an ASV system to verify whether they correspond to the
same speaker. The attacker has no knowledge of which pseudo-speaker was used
for anonymization on the test utterance and will hence likely select a
different pseudo-speaker to anonymize the enrollment utterance. The privacy
metric is the resulting EER estimated from a large number of ASV trials.
Utility is assessed by training an ASR system on the same anonymized version
of librispeech-clean-360 and by estimating the word error rate (WER) from
anonymized test data. Two additional metrics are defined in the VoicePrivacy
Challenge evaluation plan [2]. The first is the F0 curve correlation
$\boldsymbol{\rho}^{F0}$ between original and anonymized utterance which is
used as a measure of prosody preservation. The second is the gain of voice
distinctiveness $G_{VD}$ which is used to estimate how well the anonymized
voices of different speakers can be distinguished [2].

We adopt the B1b and T11 participant system from the Voice Privacy Challenge
2022, in addition to the system proposed by Champion et al. [5] as baselines.
System T11 is the non-TTS system that achieved the highest privacy level in
the 2022 challenge.444Results available at
www.voiceprivacychallenge.org/results-2022. The overall highest privacy level
was in fact achieved by a TTS-based system [11] that barely passed the prosody
preservation requirement of scoring $\boldsymbol{\rho}^{F_{0}}>0.3$. In
general, TTS-based systems are known to almost completely erase speaker
information, at the cost of a severe loss of intonation and prosody.
Therefore, we do not include [11] in our comparative analysis. The work in [5]
was the first to propose the use of codebook-based feature quantization.

##  5 Results

Results are shown separately for LibriSpeech and VCTK test sets in Table 1.
Our system achieves the highest privacy levels: 28.5% EER for LibriSpeech;
45.5% EER for VCTK. The substantially lower EERs of 17.5% and 28.0% for the
two test sets and the system of Champion et al. suggest that our quantization
approach is more effective in removing speaker information than that proposed
in [5]. In an effort to further suppress speaker information, Champion et al.
also experimented with the addition of Gaussian noise to the input F0 curve.
Improvements to privacy nonetheless result in a lower pitch correlation
$\boldsymbol{\rho}^{F0}\approx 0.55$.555This result is provided by courtesy of
the main author of [5]. For our method, the pitch correlation is in the order
of $\boldsymbol{\rho}^{F0}\approx 0.7$ on average, and compares favorably with
that of other systems in the literature [2]. In terms of privacy protection,
our model also comfortably outperforms the T11 system by 8% and 5% EER for
LibriSpeech and VCTK test sets respectively. The gain in voice distinctiveness
for the T11 system are also low. This is not surprising since the system maps
all speakers to similar pseudo-speakers. In contrast, our system gives values
of $G_{VD}\approx-2$, denoting substantially better speaker distinctiveness.

However, utility estimates for our model are lower than that of other systems.
The WER increase from 4.2% (original data) to 7.5% for the LibriSpeech subset
and from 12.8% to 18.9% for the VCTK subset. Similar issues have also been
reported in the literature. The authors of [8] show that the NAC copy-
synthesis of LibriSpeech test-clean dataset causes an increases to the WER of
its own ASR system from 2.5% to 6%, with similar results being reported in
[9]. Nevertheless, informal listening tests on our data do not reveal any
notable artifacts or degradation to intelligibility.

In an attempt to shed light on the cause for this phenomenon, we repeated
similar experiments using a different ASR architecture, namely that reported
in [18], which is retrained according to the same setup described in Section
4. The issue persists. The WER increases from 2.5% to 4.6% for the LibriSpeech
subset and from 7.6% to 15.5% for the VCTK subset. These findings suggest that
the degradation to utility is more dependend on the NAC language model than on
the ASR system. As suggested in [8], this could be due to the quality of some
pseudo-speaker prompts, since the extracted fine acoustic tokens tend also to
capture aspects of the (potentially poor) _recording conditions_ , the
characteristics of which are then transferred to anonymized outputs. More
thorough experimentation to help us better understand this phenomena is
already underway.

##  6 Conclusions

We present a novel approach to speaker anonymization based on a neural audio
codec (NAC) language model. Our system performs voice conversion by extracting
a set of semantic tokens from an input signal and using them to estimate a set
of acoustic tokens belonging to a different speaker, which in turn are used to
synthesize an anonymized speech signal with a NAC decoder. The quantized
nature of the semantic and acoustic tokens successfully bottlenecks speaker-
related information delivering substantially improved anonymization
performance without compromising prosody or speaker distinctiveness. While
informal listening tests show that anonymized signals are of high quality and
intelligibility, automatic transcription with a speech recognition system
shows a modest degradation to utility. Future work should investigate
strategies to better protect utility while retaining the benefits to privacy
safeguard, such as using high-quality speaker prompts or fine-tuning parts of
the system with utility preservation constraints.

## References

  * [1] N. Tomashenko, Brij Mohan Lal Srivastava, Xin Wang, Emmanuel Vincent, Andreas Nautsch, Junichi Yamagishi, Nicholas Evans, Jose Patino, Jean-François Bonastre, Paul-Gauthier Noé, and Massimiliano Todisco,  “Introducing the VoicePrivacy Initiative,”  in Proc. Interspeech 2020, 2020, pp. 1693–1697.
  * [2] Natalia Tomashenko, Xin Wang, Xiaoxiao Miao, Hubert Nourtel, Pierre Champion, Massimiliano Todisco, Emmanuel Vincent, Nicholas Evans, Junichi Yamagishi, and Jean-François Bonastre,  “The voiceprivacy 2022 challenge evaluation plan,” 2022.
  * [3] Fuming Fang, Xin Wang, Junichi Yamagishi, Isao Echizen, Massimiliano Todisco, Nicholas Evans, and Jean-Francois Bonastre,  “Speaker Anonymization Using X-vector and Neural Waveform Models,”  in Proc. 10th ISCA Workshop on Speech Synthesis (SSW 10), 2019, pp. 155–160.
  * [4] Brecht Desplanques, Jenthe Thienpondt, and Kris Demuynck,  “ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification,”  in Proc. Interspeech 2020, 2020, pp. 3830–3834.
  * [5] Pierre Champion, Anthony Larcher, and Denis Jouvet,  “Are disentangled representations all you need to build speaker anonymization systems?,”  in Interspeech 2022. Sept. 2022, pp. 2793–2797, ISCA.
  * [6] Pierre Champion, Denis Jouvet, and Anthony Larcher,  “A Study of F0 Modification for X-Vector Based Speech Pseudonymization Across Gender,” 2021.
  * [7] Ali Shahin Shamsabadi, Brij Mohan Lal Srivastava, Aurélien Bellet, Nathalie Vauquier, Emmanuel Vincent, Mohamed Maouche, Marc Tommasi, and Nicolas Papernot,  “Differentially private speaker anonymization,”  Proceedings on Privacy Enhancing Technologies, vol. 1, pp. 98–114, 2023.
  * [8] Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, and Neil Zeghidour,  “AudioLM: a Language Modeling Approach to Audio Generation,” July 2023,  arXiv:2209.03143 [cs, eess].
  * [9] Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, and Furu Wei,  “Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers,” Jan. 2023,  arXiv:2301.02111 [cs, eess].
  * [10] Jixun Yao, Qing Wang, Li Zhang, Pengcheng Guo, Yuhao Liang, and Lei Xie,  “NWPU-ASLP System for the VoicePrivacy 2022 Challenge,”  in Proc. 2nd Symposium on Security and Privacy in Speech Communication, 2022.
  * [11] Sarina Meyer, Pascal Tilli, Florian Lux, Pavel Denisov, Julia Koch, and Ngoc Thang Vu,  “Cascade of phonetic speech recognition, speaker embeddings gan and multispeaker speech synthesis for the VoicePrivacy 2022 Challenge ,”  in Proc. 2nd Symposium on Security and Privacy in Speech Communication, 2022.
  * [12] Xiaoxiao Miao, Xin Wang, Erica Cooper, Junichi Yamagishi, and Natalia Tomashenko,  “Language-independent speaker anonymization using orthogonal householder neural network,”  arXiv preprint arXiv:2305.18823, 2023.
  * [13] Michele Panariello, Massimiliano Todisco, and Nicholas Evans,  “Vocoder drift compensation by x-vector alignment in speaker anonymisation,”  in Proc. 3nd Symposium on Security and Privacy in Speech Communication, 2023.
  * [14] Michele Panariello, Massimiliano Todisco, and Nicholas Evans,  “Vocoder drift in x-vector–based speaker anonymization,”  in Proc. INTERSPEECH 2023, 2023, pp. 2863–2867.
  * [15] Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi,  “SoundStream: An End-to-End Neural Audio Codec,”  IEEE/ACM Transactions on Audio, Speech and Language Processing, vol. 30, pp. 495–507, Nov. 2021.
  * [16] Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi,  “High Fidelity Neural Audio Compression,” Oct. 2022,  arXiv:2210.13438 [cs, eess, stat].
  * [17] Zhichao Wang, Yuanzhe Chen, Lei Xie, Qiao Tian, and Yuping Wang,  “Lm-vc: Zero-shot voice conversion via speech generation based on language models,” 2023.
  * [18] Francesco Nespoli, Daniel Barreda, Jöerg Bitzer, and Patrick A. Naylor,  “Two-Stage Voice Anonymization for Enhanced Privacy,”  in Proc. INTERSPEECH 2023, 2023, pp. 3854–3858.
  * [19] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed,  “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,”  IEEE/ACM Trans. Audio, Speech and Lang. Proc., vol. 29, pp. 3451–3460, oct 2021.
  * [20] Sepp Hochreiter and Jürgen Schmidhuber,  “Long short-term memory,”  Neural Comput., vol. 9, no. 8, pp. 1735–1780, nov 1997.
  * [21] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.,  “Language models are unsupervised multitask learners,”  OpenAI blog, vol. 1, no. 8, pp. 9, 2019.
  * [22] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur,  “Librispeech: An ASR corpus based on public domain audio books,”  in 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2015, pp. 5206–5210.
  * [23] Junichi Yamagishi, Christophe Veaux, and Kirsten MacDonald,  “CSTR VCTK Corpus: English Multi-speaker Corpus for CSTR Voice Cloning Toolkit,”  University of Edinburgh. The Centre for Speech Technology Research, 2019.
  * [24] Vijayaditya Peddinti, Daniel Povey, and Sanjeev Khudanpur,  “A time delay neural network architecture for efficient modeling of long temporal contexts,”  in Proc. Interspeech 2015, 2015, pp. 3214–3218.