[affiliation=1]XinWang [affiliation=2]TomiKinnunen [affiliation=3]Kong AikLee
[affiliation=4]Paul-GauthierNoé [affiliation=1]JunichiYamagishi

ISCA Copyright Notice

©2024 ISCA. By each submission of a paper to INTERSPEECH-2024, the author(s)
hereby warrant(s) that the manuscript has been submitted to ISCA (the
International Speech Communication Association) for publication in the
INTERSPEECH 2024 Proceedings, IS ORIGINAL AND HAS NOT BEEN SUBMITTED FOR
PUBLICATION OR PUBLISHED ELSEWHERE and that all trademark use within the
manuscript has been credited to its owner or written permission to use the
name has been granted. In addition, the author(s) acknowledge(s) that all
images such as tables, screenshots, graphics, etc., do not have a copyright
that is held by a third party. The author(s) agree(s) to, and hereby assign(s)
all rights, title and interest, including copyrights, in and to the manuscript
to ISCA. The author(s) retain(s) the rights to any intellectual property
developed by the author(s) and included in the manuscript. Note that the ISCA
reviewing process is confidential. The copyright is provisionally transferred
to ISCA at the time of submission of the article. This copyright is
definitively assigned to ISCA if the manuscript is accepted after the
reviewing process. If the manuscript is not accepted, the copyright transfer
to ISCA is thus abrogated.

This work is accepted by the ISCA INTERSPEECH 2024.

    @inproceedings{wangRevisiting2024,
      title = {Revisiting and Improving Scoring Fusion for
           Spoofing-aware Speaker Verification Using Compositional
           Data Analysis},
      booktitle = {Proc. {{Interspeech}}},
      author = {Wang, Xin and Kinnunen, Tomi and Kong Aik, Lee and
          Noé, Paul-Gauthier and Yamagishi, Junichi},
      year = {2024},
      pages = {(accepted)}
    }

# Revisiting and Improving Scoring Fusion for Spoofing-aware Speaker
Verification Using Compositional Data Analysis

###### Abstract

Fusing outputs from automatic speaker verification (ASV) and spoofing
countermeasure (CM) is expected to make an integrated system robust to zero-
effort imposters and synthesized spoofing attacks. Many score-level fusion
methods have been proposed, but many remain heuristic. This paper revisits
score-level fusion using tools from decision theory and presents three main
findings. First, fusion by summing the ASV and CM scores can be interpreted on
the basis of compositional data analysis, and score calibration before fusion
is essential. Second, the interpretation leads to an improved fusion method
that linearly combines the log-likelihood ratios of ASV and CM. However, as
the third finding reveals, this linear combination is inferior to a non-linear
one in making optimal decisions. The outcomes of these findings, namely, the
score calibration before fusion, improved linear fusion, and better non-linear
fusion, were found to be effective on the SASV challenge database.

###### keywords:

speaker verification, anti-spoofing, fusion, log-likelihood ratio, ternary
classification

##  1 Introduction

Automatic speaker verification (ASV) systems are vulnerable to spoofed
synthetic speech data that clone the target speakers' voices [1]. By combining
ASV and spoofing countermeasure (CM), which ideally should reject any spoofed
data, we expect to be able to construct a spoofing-aware ASV (SASV) system
that only accepts speech data from real (bona fide) humans who match the
target speaker's identity.

An SASV system can be a cascade of ASV and CM sub-systems [2]. The ASV and CM
make separate binary decisions, and the input is accepted if it is accepted by
both sub-systems. Another design used by many SASV systems is to produce one
score and make one binary decision. Let the input be
$\bm{x}=(\bm{x}^{(p)},\bm{x}^{(r)})\in\mathcal{X}$, where $\bm{x}^{(p)}$ and
$\bm{x}^{(r)}$ are the probe (test) and reference (enrollment) samples,
respectively, and can be waveforms, features, etc. An SASV system defines a
scoring function $g:\mathcal{X}\rightarrow\mathbb{R}$ that maps $\bm{x}$ into
a score $s_{\text{sasv}}$. If $s_{\text{sasv}}$ surpasses a threshold, the
system takes the action of $\mathtt{accept}$, claiming that $\bm{x}_{p}$ is
bona fide _and_ matches the speaker identity in $\bm{x}_{r}$. Otherwise, it
takes the action of $\mathtt{reject}$.

This paper focuses on the SASV systems that produce $s_{\text{sasv}}$ using
score-level fusion. The idea is to use ASV and CM sub-systems to obtain an ASV
score $s_{\text{asv}}$ and CM score $s_{\text{cm}}$ then use a score-level
fusion function $h:\mathbb{R}\times\mathbb{R}\rightarrow\mathbb{R}$ to merge
the scores into $s_{\text{sasv}}$. Note that there are feature-level fusion
and end-to-end methods [3, 4, 5], which do not need to produce
$\\{s_{\text{asv}},s_{\text{cm}}\\}$. We argue that score-level fusion is
useful because $\\{s_{\text{asv}},s_{\text{cm}}\\}$ provides invaluable extra
explanations on the model decision. Some evaluation metrics, such as tandem
equal error rate (t-EER) [6], also require separated $s_{\text{asv}}$ and
$s_{\text{cm}}$.

An example SASV system using score-level fusion (baseline B1 of the SASV
challenge [5]) is illustrated in Fig. 1. Its ASV sub-system extracts
embeddings from $\bm{x}^{(p)}$ and $\bm{x}^{(r)}$ and computes their cosine
similarity as $s_{\text{asv}}$, which indicates the degree that $\bm{x}^{(p)}$
and $\bm{x}^{(r)}$ match in terms of speaker identity. The CM sub-system
extracts an embedding from $\bm{x}^{(p)}$ and produces a score
$s_{\text{cm}}\in\mathbb{R}$ using a neural network. A higher $s_{\text{cm}}$
indicates that $\bm{x}^{(p)}$ is more likely to be bona fide. Hence, it is
intuitively reasonable to fuse the scores by
$s_{\text{sasv}}=s_{\text{asv}}+s_{\text{cm}}$.

More advanced fusion functions have been proposed in the SASV and other
biometrics fields [7, 2]. Although most are intuitively reasonable, for
example, normalizing the numeric range of $s_{\text{asv}}$ and $s_{\text{cm}}$
before summation [5], the link between practice and theory is not always
clear. Therefore, the implicit assumptions, link to optimal decisions, and
room for improvement remain elusive.

Starting from the summation of the ASV and CM scores, we scrutinize the score
fusion using tools from decision theory. After reformulating SASV as a
classification task that involves three data classes but requires a binary
decision, we interpret SASV scoring from the perspective of compositional data
analysis and show that the summation of $s_{\text{asv}}$ and $s_{\text{cm}}$
is a reasonable choice if the ASV and CM scores are proper log-likelihood
ratios (LLRs). Hence, the interpretation indicates that score calibration
before fusion is beneficial. The interpretation also leads to an improved
fusion method that uses probabilistic models to compute LLRs for summation.
Finally, the summation-based fusion is analyzed from the perspective of
optimal decision policy and found to be theoretically inferior to a non-linear
fusion method.

Input pair $\bm{x}=(\bm{x}^{(p)},\bm{x}^{(r)})$ ASV embedding CM embedding
Cosine similarity Neural network +$s_{\text{sasv}}$$\bm{x}^{(p)}$
$\bm{x}^{(r)}$ $s_{\text{asv}}$$s_{\text{cm}}$ Figure 1: Example SASV with
score-level fusion (B1 in [5])

The above interpretations of SASV have not been presented as far as the
authors are aware. They differ from one existing study that treats
$\\{s_{\text{asv}},s_{\text{cm}}\\}$ as posterior probabilities [8].
Experiments were conducted on the SASV challenge database [5]. The calibration
before fusion was found to be effective, especially for linear fusion by
summation. The linear and non-linear fusion of LLRs outperformed baselines,
and the latter outperformed the systems fusing posterior probabilities [8].
The results are reproducible. 111Code site: https://github.com/nii-
yamagishilab/SpeechSPC-mini.

##  2 Interpreting and improving SASV score fusion

###  2.1 SASV is not fusion of congruent binary classifiers

By definition, an SASV system should only $\mathtt{accept}$ if the input is
uttered by the bona fide target speaker; otherwise, $\mathtt{reject}$. Fusing
ASV and CM scores for a binary decision in the SASV system seems to be similar
to the score fusion in multi-modal biometric verification [9, 10].

Let $\bm{\mathrm{X}}{}=\\{\bm{x}{}_{1},\cdots,\bm{x}{}_{K}\\}$ be the inputs
to a biometric verification system with $K$ modalities (e.g., speech and
face). The system determines whether inputs match the claimed identity
(${H}{}_{\mathsf{tar}{}}$) or not (${H}{}_{\mathsf{non}{}}$). Assuming that
$\bm{x}_{k}$ are statistically independent, we can use the Bayes' formula [11]
to obtain

$\log\frac{{P({H}{}_{\mathsf{tar}{}}|\bm{\mathrm{X}}{})}}{1-{P({H}{}_{\mathsf{tar}{}}|\bm{\mathrm{X}}{})}}=\log\frac{\pi_{\mathsf{tar}{}}}{1-\pi_{\mathsf{tar}{}}}+\sum_{k=1}^{K}\mathsf{llr}^{\mathsf{tar}{}}_{\mathsf{non}{}}(\bm{x}_{k}),$
(1)

where
$\mathsf{llr}^{\mathsf{tar}{}}_{\mathsf{non}{}}(\bm{x}_{k})\triangleq\log\frac{p(\bm{x}_{k}|{H}{}_{\mathsf{tar}{}})}{p(\bm{x}_{k}|{H}{}_{\mathsf{non}{}})}$
is an LLR produced by the $k$-th sub-system, and $\pi_{\mathsf{tar}{}}$ is the
prior probability of class $\mathsf{tar}{}$. Assuming that the LLRs and priors
are equal to the ground truth, our decision strategy for minimizing the
decision error rate is to accept ${\mathsf{tar}{}}$ if and only if
$\sum_{k=1}^{K}\mathsf{llr}^{\mathsf{tar}{}}_{\mathsf{non}{}}(\bm{x}_{k})+\log\frac{\pi_{\mathsf{tar}{}}}{1-\pi_{\mathsf{tar}{}}}>0$
[11]. The sum of LLRs is the weight of evidence to favor $\mathsf{tar}{}$,
which is known as _independent additivity_ of LLRs [12].

We may hastily treat SASV as biometric verification with two modalities,
namely, ASV and CM. Thus, fusion by $s_{\text{cm}}+s_{\text{asv}}$ can be
interpreted as the sum of LLRs, given that the scores approximate the LLRs.
However, unlike Eq. (1) where all sub-systems deal with
$\\{{H}{}_{\mathsf{tar}{}},{H}{}_{\mathsf{non}{}}\\}$, ASV and CM have
different hypotheses: for CM, the input is either bona fide
(${H}{}_{\mathsf{bona}{}}$) or spoofed (${H}{}_{\mathsf{spoof}{}}$). With four
hypotheses involved, independent additivity of LLRs and Eq. (1) no longer hold
[12, sec.4.3].

###  2.2 Interpreting SASV using compositional data analysis

A better starting point is to have three exhaustive and mutually exclusive
hypotheses [2, 13]: the input probe is bona fide and matches the reference
(${H}{}_{\mathsf{tar.bf}{}}$), bona fide and unmatched
(${H}{}_{\mathsf{non.bf}{}}$), or spoofed (${H}{}_{\mathsf{spf}{}}$). 222It is
unnecessary to separate ${H}{}_{\mathsf{spf}{}}$ into matched and non-matched
cases since both should be rejected [6]. To interpret this classification task
with three classes but two actions $\\{\mathtt{accept},\mathtt{reject}\\}$, we
use _compositional data analysis_ [14, 15].

Let
$\bm{P}=[P({H}{}_{\mathsf{spf}{}}|\bm{x}),P({H}{}_{\mathsf{non.bf}{}}|\bm{x}),P({H}{}_{\mathsf{tar.bf}{}}|\bm{x})]^{\top}$
and
$\bm{\pi}=[\pi_{\mathsf{spf}{}},\pi_{\mathsf{non.bf}{}},\pi_{\mathsf{tar.bf}{}}]^{\top}$
be the vectors of posterior and prior probabilities, respectively. Let the
vector of likelihoods be
$\bm{w}=[p(\bm{x}|{H}{}_{\mathsf{spf}{}}),p(\bm{x}|{H}{}_{\mathsf{non.bf}{}}),p(\bm{x}|{H}{}_{\mathsf{tar.bf}{}})]^{\top}$.
Because $\sum_{i}\pi_{i}=\sum_{i}P_{i}=1$, where $i$ is the index of the
vector's element, $\bm{\pi}$ and $\bm{P}$ are compositional vectors on the
probability simplex $\mathbb{S}^{3}$. Hence, $\bm{P}$ (and $\bm{\pi}$) can be
equivalently represented by a vector in 2D Euclidean space
$\tilde{\bm{P}}=[\tilde{{P}_{1}},\tilde{{P}_{2}}]^{\top}\in\mathbb{R}^{2}$.

Among many possible choices for $\mathbb{S}^{3}\rightarrow\mathbb{R}^{2}$, an
isometric-log-ratio (ILR) transformation [16]
$\tilde{\bm{P}}=\operatorname{ilr}(\bm{P})$ defines

$\displaystyle\tilde{{P}}_{1}=\frac{1}{\sqrt{2}}\log\frac{P_{2}}{P_{1}},\quad\tilde{{P}}_{2}=\frac{1}{\sqrt{6}}\log\frac{P_{3}P_{3}}{P_{1}P_{2}},$
(2)

where the scalars $\frac{1}{\sqrt{2}}$ and $\frac{1}{\sqrt{6}}$ are due to the
Aitchison orthonormal basis [15, Sec.10.2][16]. The same transformation
$\operatorname{ilr}(\cdot)$ can be applied to $\bm{\pi}$ and $\bm{w}$.
Particularly, for $\tilde{\bm{w}}$, we have

$\begin{split}\tilde{{w}}_{1}&=\frac{1}{\sqrt{2}}\log\frac{w_{3}w_{2}}{w_{1}w_{3}}=\frac{1}{\sqrt{2}}\Big{(}\mathsf{llr}^{\mathsf{tar.bf}{}}_{\mathsf{spf}{}}(\bm{x})-\mathsf{llr}^{\mathsf{tar.bf}{}}_{\mathsf{non.bf}}(\bm{x})\Big{)},\end{split}$
(3)
$\begin{split}\tilde{{w}}_{2}&=\frac{1}{\sqrt{6}}\log\frac{w_{3}w_{3}}{w_{1}w_{2}}=\frac{1}{\sqrt{6}}\Big{(}\mathsf{llr}^{\mathsf{tar.bf}{}}_{\mathsf{non.bf}}(\bm{x})+\mathsf{llr}^{\mathsf{tar.bf}{}}_{\mathsf{spf}{}}(\bm{x})\Big{)}.\end{split}$
(4)

With the $\tilde{\bm{P}},\tilde{\bm{\pi}}$, and $\tilde{\bm{w}}$, it is known
that $\tilde{\bm{P}}=\tilde{\bm{\pi}}+\tilde{\bm{w}}$ [15, Sec.3.3.3]. This
equation has a similar form of ``posterior log-odds = prior log-odds + LLR''
to Eq. (1). Similar to the sum of LLRs in Eq. (1),
$\tilde{\bm{w}}=[\tilde{{w}_{1}},\tilde{{w}_{2}}]^{\top}$ can be interpreted
as weights of evidence, albeit in a hierarchical manner [15] as illustrated in
Fig. 2. A larger $\tilde{w}_{2}$ favors ${H}{}_{\mathsf{tar.bf}{}}$ over the
other two, and it is the $\tilde{w}_{1}$ that discriminates
${H}{}_{\mathsf{spf}{}}$ from ${H}{}_{\mathsf{non.bf}{}}$.

${H}_{\mathsf{spf}}$${H}_{\mathsf{non.bf}}$${H}_{\mathsf{tar.bf}}$$\tilde{w}_{1}$$\tilde{w}_{2}$
Figure 2: Bifurcating tree for ternary hypothesis test based on compositional
data analysis [15]

The above description indicates that $\tilde{w}_{2}$ is what an SASV system
needs to produce. This observation paves the way for interpreting and
improving the SASV score fusion methods.

###  2.3 Fusion method 1: summation of ASV and CM scores

Note that ASV is designed to discriminate $\mathsf{tar.bf}{}$ and
$\mathsf{non.bf}{}$, while CM differentiates $\mathsf{tar.bf}{}$ from
$\mathsf{spf}{}$. If their scores approximate the LLRs, i.e.,
$s_{\text{asv}}\approx\mathsf{llr}^{\mathsf{tar.bf}{}}_{\mathsf{non.bf}}(\bm{x})$
and
$s_{\text{cm}}\approx\mathsf{llr}^{\mathsf{tar.bf}{}}_{\mathsf{spf}{}}(\bm{x})$,333For
a speaker-independent CM, we can assume $p(\bm{x}|\mathsf{tar.bf}{})\approx
p(\bm{x}|\mathsf{non.bf}{})\approx p(\bm{x}|\mathsf{bona}{})$ [17] and
$\mathsf{llr}^{\mathsf{tar.bf}{}}_{\mathsf{spf}{}}(\bm{x})\approx\mathsf{llr}^{\mathsf{bona}{}}_{\mathsf{spf}{}}(\bm{x})$.
we can follow Eq. (4) and compute $s_{\text{sasv}}$ as

$\tilde{{w}}_{2}\approx
s_{\text{sasv}}=\frac{1}{\sqrt{6}}(s_{\text{asv}}+s_{\text{cm}}).$ (5)

Hence, the summation-based score fusion is interpretable from the perspective
of compositional data analysis. Note that the factor $\frac{1}{\sqrt{6}}$ does
not affect the discrimination power of $s_{\text{sasv}}$.

The interpretation may be disappointing since Eq. (5) is the same as what we
have done, but it suggests a practice that is either ignored or conducted
arbitrarily: _the ASV and CM scores need to well approximate the LLRs before
summation._ One way to do so is score calibration [10]. For example, let
$\\{f_{\text{asv}},f_{\text{cm}}\\}$ be calibration functions, both of which
have a form of affine transformation $f(x)=ax+b$. We can learn parameters of
$\\{f_{\text{asv}},f_{\text{cm}}\\}$ on a development set through logistic
regression [18], after which we compute
$s_{\text{sasv}}=\frac{1}{\sqrt{6}}\big{[}f_{\text{asv}}(s_{\text{asv}})+f_{\text{cm}}(s_{\text{cm}})\big{]}$.

###  2.4 Fusion method 2: summation of LLRs

Rather than calibrating the ASV and CM scores to produce the LLRs (i.e.,
discriminative calibration [19]), we can treat the scores as features and
carry out generative calibration [19]. Let us create a feature vector
$\bm{s}=[s_{\text{asv}},s_{\text{cm}}]^{\top}$ and introduce likelihood
functions $p(\bm{s};\bm{\theta}_{\mathsf{tar.bf}}{})$,
$p(\bm{s};\bm{\theta}_{\mathsf{spf}{}})$, and
$p(\bm{s};\bm{\theta}_{\mathsf{non.bf}{}})$ for the three classes. Assuming
$\bm{s}$ encodes all the information of $\bm{x}$ and
$\frac{p(\bm{x}|{H}{}_{1})}{p(\bm{x}|{H}{}_{2})}=\frac{p(\bm{s}|\bm{\theta}_{1})}{p(\bm{s}|\bm{\theta}_{2})}$,
based on Eq. (4), we can compute

$s_{\text{sasv}}=\frac{1}{\sqrt{6}}\Big{(}\mathsf{llr}^{\mathsf{tar.bf}{}}_{\mathsf{non.bf}}(\bm{s})+\mathsf{llr}^{\mathsf{tar.bf}{}}_{\mathsf{spf}{}}(\bm{s})\Big{)},$
(6)

where
$\mathsf{llr}^{\mathsf{tar.bf}{}}_{\mathsf{non.bf}}(\bm{s})\triangleq\log\frac{p(\bm{s};\bm{\theta}_{\mathsf{tar.bf}{}})}{p(\bm{s};\bm{\theta}_{\mathsf{non.bf}{}})}$
and
$\mathsf{llr}^{\mathsf{tar.bf}{}}_{\mathsf{spf}}(\bm{s})\triangleq\log\frac{p(\bm{s};\bm{\theta}_{\mathsf{tar.bf}{}})}{p(\bm{s};\bm{\theta}_{\mathsf{spf}{}})}$.
Each $p(\bm{s};\bm{\theta})$ can be a Gaussian or another parametric
distribution, and its parameter set $\bm{\theta}$ can be estimated by
maximizing the likelihood on development data.

During testing, we compute $p(\bm{s};\bm{\theta}_{\mathsf{tar.bf}}{})$,
$p(\bm{s};\bm{\theta}_{\mathsf{spf}{}})$, and
$p(\bm{s};\bm{\theta}_{\mathsf{non.bf}{}})$ of a test sample and plug them
into Eq. (6). Since the selected parametric distributions may be different
from the actual distribution of $\bm{s}$, the estimated LLRs may be
inaccurate. Hence, we can optionally apply discriminative calibration on the
LLRs before fusion.

###  2.5 Fusion method 3: nonlinear fusion of LLRs

The interpretation so far leads to the simple form of score fusion in Eqs. (5)
and (6). Importantly, however, it also reveals a major limitation for making
decisions. Recall from Sec. 2.1 that, for a single ASV system ($K=1$), we
accept ${\mathsf{tar}{}}$ if and only if
$\mathsf{llr}^{\mathsf{tar}{}}_{\mathsf{non}{}}(\bm{s})+\log\frac{\pi_{\mathsf{tar}{}}}{1-\pi_{\mathsf{tar}{}}}>0$.
For SASV, a similar decision policy is to accept ${\mathsf{tar.bf}{}}$ if and
only if $\tilde{{P}}_{2}=\tilde{{\pi}}_{2}+\tilde{{w}}_{2}>0$. Given
$\tilde{{P}}_{2}$ defined in Eq. (2), this condition can be written as

$P({H}{}_{\mathsf{tar.bf}{}}|\bm{x})^{2}>P({H}{}_{\mathsf{non.bf}{}}|\bm{x})P({H}{}_{\mathsf{spf}{}}|\bm{x}),$
(7)

or equivalently,

$\begin{split}\mathsf{llr}^{\mathsf{tar.bf}{}}_{\mathsf{non.bf}}(\bm{s})+&\mathsf{llr}^{\mathsf{tar.bf}{}}_{\mathsf{spf}{}}(\bm{s})>\log\frac{\pi_{\mathsf{spf}{}}\pi_{\mathsf{non.bf}{}}}{\pi_{\mathsf{tar.bf}{}}^{2}}.\end{split}$
(8)

However, this decision policy cannot minimize the decision cost even if when
the true LLRs and priors are given.

It can be shown that, for SASV with equal decision costs on false rejection of
$\mathsf{tar.bf}{}$ and false acceptance of $\mathsf{non.bf}$ or
$\mathsf{spf}$, the optimal decision policy is to accept $\mathsf{tar.bf}{}$
when 444It is different from the condition
$P({H}{}_{\mathsf{tar.bf}{}}|\bm{x})>P({H}{}_{\mathsf{non.bf}{}}|\bm{x})$ AND
$P({H}{}_{\mathsf{tar.bf}{}}|\bm{x})>P({H}{}_{\mathsf{spf}{}}|\bm{x})$ [15,
sec.10.3], which maximizes the decision utility [20] with a scalar diagonal
utility matrix.

$P({H}{}_{\mathsf{tar.bf}{}}|\bm{x})>P({H}{}_{\mathsf{non.bf}{}}|\bm{x})+P({H}{}_{\mathsf{spf}{}}|\bm{x}),$
(9)

or with Bayes' formula and logarithm applied,

$-\log[(1-\rho)e^{-\mathsf{llr}^{\mathsf{tar.bf}{}}_{\mathsf{non.bf}}(\bm{s})}+\rho
e^{-\mathsf{llr}^{\mathsf{tar.bf}{}}_{\mathsf{spf}{}}(\bm{s})}]>-\log\beta,$
(10)

where
$\beta\triangleq\frac{\pi_{\mathsf{tar.bf}{}}}{\pi_{\mathsf{non.bf}{}}+\pi_{\mathsf{spf}{}}}$
and
$\rho\triangleq\frac{\pi_{\mathsf{spf}{}}}{\pi_{\mathsf{non.bf}{}}+\pi_{\mathsf{spf}{}}}$.
555$\rho$ is the same as the spoof prevalence prior in t-EER [6].

Inequality. (7) is a necessary but not sufficient condition to achieve Ineq.
(9). For example, a sample with
$[P({H}{}_{\mathsf{spf}{}}|\bm{x}),P({H}{}_{\mathsf{non.bf}{}}|\bm{x}),P({H}{}_{\mathsf{tar.bf}{}}|\bm{x})]=[0.05,0.65,0.3]$
will be rejected by Ineq. (9) but falsely accepted by (7). The overall
decision cost cannot be lower than that based on the optimal decision policy
in Ineq. (9) (or (10)), given that we know the true LLRs and priors.

To support the above argument, we generated ASV and CM scores using Gaussian
distributions and a uniform prior, which enables us to compute the true LLRs
and compared the linear and non-linear decision boundaries. As Fig. 3
indicates, the non-linear one based on Ineq. (10) leads to a lower decision
cost, particularly, a lower false acceptance rate. Note that, Ineqs. (8) and
(10) are agnostic to the data distributions, and we use Gaussians in
simulation only for convenience.

Figure 3: Scatter plot of ASV
$\mathsf{llr}^{\mathsf{tar.bf}{}}_{\mathsf{non.bf}}(\bm{s})$ and CM
$\mathsf{llr}^{\mathsf{tar.bf}{}}_{\mathsf{spf}{}}(\bm{s})$ from simulated
data. Dashed and solid lines are decision boundaries based on Ineqs. (8) and
(10), respectively, given true flat prior. Green solid line is based on Ineq.
(10) but mis-matched priors
${\pi}_{\mathsf{spf}}{}=0.05,{\pi}_{\mathsf{non.bf}}{}=0.05,{\pi}_{\mathsf{tar.bf}}{}=0.9$.
Note that LLRs rather than raw scores are plotted.

(a) B1

(b) B1c

(c) L2c

(d) L3c

Figure 4: Distributions of CM, ASV, and fused SASV scores. Bona fide data of
target speakers ($\mathsf{tar.bf}{}$), those of non-target speakers
($\mathsf{non.bf}{}$), and spoofed data ($\mathsf{spf}{}$) are in different
colors. Each vertical line in bottom plane marks SASV-EER threshold.

On the basis of Ineq. (10), we define a new fusion method

$s_{\text{sasv}}=-\log\big{[}(1-\rho)e^{-\mathsf{llr}^{\mathsf{tar.bf}{}}_{\mathsf{non.bf}}(\bm{s})}+\rho
e^{-\mathsf{llr}^{\mathsf{tar.bf}{}}_{\mathsf{spf}{}}(\bm{s})}\big{]}.$ (11)

The usage is similar to that in Sec. 2.4, but we plug the LLRs into Eq. (11).
Unfortunately, the prior-dependent $\rho$ cannot be decoupled from the LLRs,
and it needs to be decided, for example, by grid search on a development set.
The green solid line in Fig. 3 shows the decision boundary when $\rho$ is
computed from different priors. It performs reasonably well.

As a special case, if each $p(\bm{s};\bm{\theta})$ is implemented as a
Gaussian distribution, Eq. (11) turns out to be equivalent to the so-called
Gaussian back-end fusion [21, Eq.(1)]. The link between Gaussian back-end
fusion and the optimal decision has not been shown before as far as the
authors are aware. Detailed proof is available on the code site (see link on
page 1).

##  3 Experiments

We verified the effectiveness of the three fusion methods on the SASV
challenge dataset [5] and the official training, development, and evaluation
splits.

###  3.1 Model recipe and evaluation metrics

All eight experimental systems were based on the open-sourced SASV challenge
baseline B1 (Fig. 1) and used the pre-trained ECAPA-TDNN-based ASV [22] and
AASIST-based CM [23] embedding extractors, which were provided by the
challenge organizers. The differences in the systems are

  * •

B1 is a replicate of SASV B1. The ASV branch requires no training, and the
neural network in the CM branch is trained using binary cross-entropy given
$s_{\text{cm}}$ and the labels for CM. The SASV score is computed by
$s_{\text{sasv}}=s_{\text{cm}}+s_{\text{asv}}$.

  * •

L2 is similar to B1 but uses Eq. (6) for score fusion. Each
$p(\bm{s};\bm{\theta})$ is a Gaussian with a full covariance matrix.

  * •

L3 follows L2 but uses Eq. (11) for score fusion.

  * •

B1c, L2c, and L3c are variants of B1, L2, and L3, respectively, but calibrated
scores or LLRs before fusion. The calibration function is $f(x)=ax+b$.

  * •

B1v2 (B2-v2 in [5]) and Post (PR-S-F in [8]) were included for reference. They
interpret scores as posterior probabilities and fuse by
$s_{\text{sasv}}=\sigma(s_{\text{cm}})+s_{\text{asv}}$ and
$s_{\text{sasv}}=\sigma(s_{\text{cm}})\times\sigma(s_{\text{asv}})$,
respectively [24]. The $\sigma(\cdot)$ is a sigmoid function.

The parameters $\bm{\theta}$, $\rho$, and $\\{a,b\\}$ for calibration were
tuned on the development set.

All systems have about 82.5 k trainable parameters in the the neural network
for CM, which has three hidden layers with 258, 128, and 64 neurons,
respectively. All layers used a leaky ReLU activation with a negative slope
value of 0.3. All systems were trained on a Tesla V100 card using the Adam
optimizer with a learning rate of $5\times 10^{-5}$, batch size of 24, and
weight decay with a coefficient of $1\times 10^{-3}$. The maximum number of
training epochs was 10, and the checkpoint with the best CM EER on the
development set was used for evaluation. The implementation was done using
SpeechBrain [25], and the training took less than one day per system.

The SASV-EER [5] was used to gauge the discrimination power of
$s_{\text{sasv}}$. The log-likelihood-ratio cost $\mathrm{Cllr}$ [26] was also
computed and decomposed into $\mathrm{Cllr}_{\text{min}}$ and
$\mathrm{Cllr}_{\text{calib}}$ to measure the discrimination and calibration
performance of $s_{\text{sasv}}$. The t-EER [6] was computed given
$\\{s_{\text{asv}},s_{\text{cm}}\\}$ or the LLRs. Each system was trained and
evaluated for six rounds, where each round used a different seed to randomly
initialize the neural network in the CM sub-system. The averaged results are
listed in Table 1. The confidence interval of SASV-EER was using the
Interspeech official toolkit with the default setting.

Table 1: Results on SASV evaluation set. Each number is averaged over six
training and evaluation rounds. Calibration (discriminative) is not on
$s_{\text{sasv}}$ but on scores or LLRs to be fused. Darker cell color
indicates worse result in each row.

ID | B1 | B1c | L2 | L2c | L3 | L3c | B1v2 | Post
---|---|---|---|---|---|---|---|---
Fusion | Eq. (5) | Eq. (6) | Eq. (11) | [5] | [8]
Calibration | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\times$
SASV-EER (%) | 20.46 | 2.73 | 3.31 | 1.56 | 1.44 | 1.43 | 1.60 | 1.55
conf. ($\alpha=5\%$) | $\pm 0.40$ | $\pm 0.27$ | $\pm 0.31$ | $\pm 0.23$ | $\pm 0.23$ | $\pm 0.23$ | $\pm 0.22$ | $\pm 0.24$
$\mathrm{Cllr}$ | 2.17 | 1.09 | 1.04 | 0.14 | 0.18 | 0.16 | 0.96 | 0.84
$\mathrm{Cllr}_{\text{min}}$ | 0.52 | 0.11 | 0.13 | 0.07 | 0.06 | 0.07 | 0.08 | 0.07
$\mathrm{Cllr}_{\text{calib}}$ | 1.64 | 0.98 | 0.91 | 0.07 | 0.11 | 0.10 | 0.88 | 0.78
t-EER (%) | 2.10 | 2.10 | 1.68 | 1.68 | 1.68 | 1.68 | 2.19 | 2.21

###  3.2 Results

Calibration before fusion was useful: As reported in the original paper [5],
B1 achieved an SASV-EER of around 20%. As Fig. 4(a) shows, due to the large
difference in the dynamic ranges of ASV and CM scores, the latter dominated
the fused SASV score. The values of $\mathrm{Cllr}$ and
$\mathrm{Cllr}_{\text{calib}}$ were higher than 1 bit, indicating that the
SASV scores were not useful for making decisions. By calibrating
$s_{\text{asv}}$ and $s_{\text{cm}}$ before fusion, B1c reduced the SASV EER
to 2.73%, and $\mathrm{Cllr}$-based metrics also improved. From Fig. 4(b), we
can observe that the distribution of $s_{\text{asv}}$ was `stretched out', and
the fused $s_{\text{sasv}}$ better separated the three classes of data.

When fusing LLRs, the system pairs {L2, L2c} and {L3, L3c} showed that the
gain of the calibration was diminishing. This is understandable because the
dynamic range of ASV and CM scores are handled by the covariance matrix of
$p(\bm{s};\bm{\theta})$. However, calibrating before fusion was still helpful,
especially in the case of linear fusion in Eq. (6). Compared with L2, L2c's
EER was reduced to 1.56%, and its $\mathrm{Cllr}$ was only 0.14 bits.

Non-linear fusion of LLRs was better: Rather than fusing the ASV and CM
scores, L2, L3, and their variants fused the estimated LLRs. The nonlinear
fusion of LLRs in L3c led to the best SASV-EER in our experiments. The
difference between L3c and L2c was small, possibly because the test data set
of the SASV challenge can be well separated using a linear decision boundary
in the space of LLRs, as Fig. 4(c) and (d) show.

Compared with B1v2 and Post, which treat the ASV and CM scores as posterior
probabilities, L2c, L3, and L3c performed similarly or better in terms of
SASV-EER, and their $\mathrm{Cllr}_{\text{calib}}$s were much lower. The
results suggest that the fusion methods based on compositional data analysis
are also practically useful.

Finally, the t-EER measures the discrimination power of
$\\{s_{\text{asv}},s_{\text{cm}}\\}$ or LLRs and is agnostic to their
calibration and fusion. Hence, the t-EERs of B1 and B1c are the same.
Similarly, the four systems fusing LLRs have the same t-EER. It was observed
that using the LLRs led to a lower t-EER than raw or sigmoid-transformed
$s_{\text{asv}}$ and $s_{\text{cm}}$. This is evidence to encourage the sub-
systems to produce LLRs.

##  4 Conclusions

This paper presented an alternative interpretation of SASV score fusion based
on compositional data analysis. We suggested practices and improved fusion
methods that may have been ignored or used ad hoc. First, the calibration of
scores or LLRs before fusion was found to be helpful, especially when linearly
fusing raw scores or LLRs. Second, rather than fusing raw scores, estimating
and fusing LLRs turned out to be better in experiments. Third, the linear
fusion of LLRs was found to be inferior to the non-linear one in both analysis
and experiments. What is presented in this paper is by no means the only way
to interpret SASV fusion, thus is expected to foster other studies.

##  5 Acknowledgements

This work was supported by JST, PRESTO Grant Number JPMJPR23P9, Japan. It was
supported in part by the Academy of Finland under Grant 349605, project
"SPEECHFAKES".

## References

  * [1] Z. Wu, N. Evans, T. Kinnunen, J. Yamagishi, F. Alegre, and H. Li, ``Spoofing and countermeasures for speaker verification: A survey,'' _Speech Communication_ , vol. 66, pp. 130–153, Feb. 2015.
  * [2] A. Hadid, N. Evans, S. Marcel, and J. Fierrez, ``Biometrics systems under spoofing attack: An evaluation methodology and lessons learned,'' _IEEE Signal Processing Magazine_ , vol. 32, no. 5, pp. 20–30, 2015.
  * [3] A. Sizov, E. Khoury, T. Kinnunen, Z. Wu, and S. Marcel, ``Joint speaker verification and antispoofing in the i-vector space,'' _IEEE Transactions on Information Forensics and Security_ , vol. 10, no. 4, pp. 821–832, 2015.
  * [4] A. Gomez-Alanis, J. A. Gonzalez-Lopez, S. P. Dubagunta, A. M. Peinado, and M. Magimai.-Doss, ``On joint optimization of automatic speaker verification and anti-spoofing in the embedding space,'' _IEEE Transactions on Information Forensics and Security_ , vol. 16, pp. 1579–1593, 2021.
  * [5] J.-w. Jung, H. Tak, H.-j. Shim, H.-S. Heo, B.-J. Lee, S.-W. Chung, H.-J. Yu, N. Evans, and T. Kinnunen, ``SASV 2022: The first spoofing-aware speaker verification challenge,'' in _Proc. Interspeech_ , 2022, pp. 2893–2897.
  * [6] T. H. Kinnunen, K. A. Lee, H. Tak, N. Evans, and A. Nautsch, ``T-eer: Parameter-free tandem evaluation of countermeasures and biometric comparators,'' _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , pp. 1–16, 2023.
  * [7] I. Chingovska, A. Anjos, and S. Marcel, ``Anti-spoofing in action: Joint operation with a verification system,'' in _2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , 2013, pp. 98–104.
  * [8] Y. Zhang, G. Zhu, and Z. Duan, ``A probabilistic fusion framework for spoofing aware speaker verification,'' in _Proc. Odyssey_ , Jun. 2022, pp. 77–84.
  * [9] K. Nandakumar, Y. Chen, S. C. Dass, and A. Jain, ``Likelihood ratio-based biometric score fusion,'' _IEEE transactions on pattern analysis and machine intelligence_ , vol. 30, no. 2, pp. 342–347, 2007.
  * [10] G. S. Morrison, ``Tutorial on logistic-regression calibration and fusion: Converting a score to a likelihood ratio,'' _Australian Journal of Forensic Sciences_ , vol. 45, no. 2, pp. 173–197, 2013.
  * [11] R. O. Duda, P. E. Hart, and D. G. Stork, _Pattern Classification_ , 2nd ed. New York: Wiley, 2001.
  * [12] E. T. Jaynes, _Probability Theory: The Logic of Science_. Cambridge university press, 2003.
  * [13] I. Chingovska, A. R. Dos Anjos, and S. Marcel, ``Biometrics evaluation under spoofing attacks,'' _IEEE transactions on Information Forensics and Security_ , vol. 9, no. 12, pp. 2264–2276, 2014.
  * [14] J. Aitchison, ``The statistical analysis of compositional data,'' _Journal of the Royal Statistical Society: Series B (Methodological)_ , vol. 44, no. 2, pp. 139–160, 1982.
  * [15] P.-G. Noé, ``Representing evidence for attribute privacy : Bayesian updating, compositional evidence and calibration,'' Theses, Université d'Avignon, Apr. 2023.
  * [16] J. J. Egozcue, V. Pawlowsky-Glahn, G. Mateu-Figueras, and C. Barcelo-Vidal, ``Isometric logratio transformations for compositional data analysis,'' _Mathematical geology_ , vol. 35, no. 3, pp. 279–300, 2003\.
  * [17] T. Kinnunen, H. Delgado, N. Evans, K. A. Lee, V. Vestman, A. Nautsch, M. Todisco, X. Wang, M. Sahidullah, J. Yamagishi, and D. A. Reynolds, ``Tandem assessment of spoofing countermeasures and automatic speaker verification: Fundamentals,'' _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 28, pp. 2195–2210, 2020.
  * [18] N. Brümmer and G. R. Doddington, ``Likelihood-ratio calibration using prior-weighted proper scoring rules,'' in _Proc. Interspeech_ , 2013, pp. 1976–1980.
  * [19] N. Brummer, A. Swart, and D. V. Leeuwen, ``A comparison of linear and non-linear calibrations for speaker recognition,'' in _Proc. Odyssey_ , 2014, pp. 14–18.
  * [20] X. He, C. E. Metz, B. M. Tsui, J. M. Links, and E. C. Frey, ``Three-class ROC analysis-A decision theoretic approach under the ideal observer framework,'' _IEEE Transactions on Medical Imaging_ , vol. 25, no. 5, pp. 571–581, 2006.
  * [21] M. Todisco, H. Delgado, K. A. Lee, M. Sahidullah, N. Evans, T. Kinnunen, and J. Yamagishi, ``Integrated presentation attack detection and automatic speaker verification: Common features and Gaussian back-end fusion,'' in _Proc. Interspeech_ , 2018, pp. 77–81.
  * [22] B. Desplanques, J. Thienpondt, and K. Demuynck, ``ECAPA-TDNN: Emphasized channel attention, propagation and aggregation in TDNN based speaker verification,'' in _Proc. Interspeech_ , 2020, pp. 3830–3834.
  * [23] J.-w. Jung, H.-S. Heo, H. Tak, H.-j. Shim, J. S. Chung, B.-J. Lee, H.-J. Yu, and N. Evans, ``AASIST: Audio anti-spoofing using integrated spectro-temporal graph attention networks,'' in _Proc. ICASSP_ , 2022, pp. 6367–6371.
  * [24] J. Kittler, M. Hatef, R. Duin, and J. Matas, ``On combining classifiers,'' _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 20, no. 3, pp. 226–239, Mar. 1998.
  * [25] M. Ravanelli, T. Parcollet, P. Plantinga, A. Rouhe, S. Cornell, L. Lugosch, C. Subakan, N. Dawalatabad, A. Heba, J. Zhong, J.-C. Chou, S.-L. Yeh, S.-W. Fu, C.-F. Liao, E. Rastorgueva, F. Grondin, W. Aris, H. Na, Y. Gao, R. D. Mori, and Y. Bengio, ``SpeechBrain: A general-purpose speech toolkit,'' 2021.
  * [26] N. Brummer, ``Measuring, refining and calibrating speaker and language information extracted from speech,'' Ph.D. dissertation, Stellenbosch: University of Stellenbosch, 2010.
  * [27] A. Nautsch, ``Speaker recognition in unconstrained environments.'' Ph.D. dissertation, Darmstadt University of Technology, Germany, 2019.
  * [28] D. R. Castro, ``Forensic evaluation of the evidence using automatic speaker recognition systems,'' Ph.D. dissertation, Universidad autónoma de Madrid, 2007.
  * [29] N. Tomashenko, B. M. L. Srivastava, X. Wang, E. Vincent, A. Nautsch, J. Yamagishi, N. Evans, J. Patino, J.-F. Bonastre, P.-G. Noé, and M. Todisco, ``Introducing the VoicePrivacy Initiative,'' in _Proc. Interspeech_ , ISCA, Oct. 2020, pp. 1693–1697.

©2024 Xin Wang

##  Appendix A Appendix

###  A.1 Optimal decision policy

We elaborate on the decision policy and linear and non-linear decision
boundaries presented in Sec. 2.5.

####  A.1.1 SASV hypotheses and actions

In SASV, or the joint operation of ASV and CM, the input pair of the probe and
reference $\bm{x}=(\bm{x}^{(p)},\bm{x}^{(r)})\in\mathcal{X}$, either as
waveforms or feature vectors, can be from one of the three possible classes
$\mathcal{Y}=\\{{H}{}_{\mathsf{tar.bf}}{},{H}{}_{\mathsf{non.bf}}{},{H}{}_{\mathsf{spf}}{}\\}$[2]:

$\displaystyle\mathsf{tar.bf}{}:$ $\displaystyle\,\bm{x}_{p}\text{ is bona
fide \emph{and} matches the speaker identity in }\bm{x}_{r},$
$\displaystyle\mathsf{non.bf}{}:$ $\displaystyle\,\bm{x}_{p}\text{ is bona
fide \emph{but} the speaker identity does not match},$
$\displaystyle\mathsf{spf}{}:$ $\displaystyle\,\bm{x}_{p}\text{ is spoofed}.$

In the last class, the SASV system does not need to differentiate whether a
spoofed probe $\bm{x}_{p}$ sounds like the target speaker. A more detailed
discussion is given in Sec. A.4.

SASV is used to accept only bona fide speech uttered by the target speaker.
Therefore, an SASV system can take one of two possible actions
$\mathcal{A}=\\{\mathtt{accept},\mathtt{reject}\\}$:

$\displaystyle\mathtt{accept}:$ $\displaystyle\,\bm{x}_{p}\text{ is claimed to
be from }\mathsf{tar.bf}{}\text{ and accepted},$
$\displaystyle\mathtt{reject}:$ $\displaystyle\,\bm{x}_{p}\text{ is claimed to
be NOT from }\mathsf{tar.bf}{}\text{ and rejected}.$

In the case of $\mathtt{reject}$, SASV does not need to differentiate whether
$\bm{x}_{p}$ is $\mathsf{non.bf}{}$ or $\mathsf{spf}{}$.

####  A.1.2 Decision cost and optimal decision policy

An SASV system may take wrong actions. A sample from $\mathsf{tar.bf}{}$ may
be falsely rejected (missed), while samples from $\mathsf{non.bf}{}$ or
$\mathsf{spf}{}$ may be falsely accepted. We can assign a decision cost to
each incorrect decision. Given an action $a\in\mathcal{A}$ and the ground-
truth label $y\in\mathcal{Y}$, we use $C(a,y)\geq 0$ to denote the decision-
cost function $c:\mathcal{A}\times{\mathcal{Y}}\rightarrow\mathbb{R}^{+}$. A
general definition of $C(a,y)$ is given in Table 2.

Following the Bayesian decision theory [11], we can compute the conditional
risk $R(a|\bm{x})$ when taking an action $a$ given the input sample $\bm{x}$
as

$R(a|\bm{x})=\sum_{y\in\mathcal{Y}}C(a,y)P(y|\bm{x}),$ (12)

where $P(y|\bm{x})$ is the posterior probability. By plugging in the possible
values of $C(a,y)$ listed in Table 2, we can compute the conditional risks
when taking the two possible actions

$\displaystyle R(\mathtt{accept}|\bm{x})$
$\displaystyle=C_{\text{fa}}^{\mathsf{non.bf}{}}{}P({H}{}_{\mathsf{non.bf}}{}|\bm{x})+C_{\text{fa}}^{\mathsf{spf}{}}{}P({H}{}_{\mathsf{spf}}{}|\bm{x}),$
(13) $\displaystyle R(\mathtt{reject}|\bm{x})$
$\displaystyle=C_{\text{miss}}^{\mathsf{tar.bf}{}}{}P({H}{}_{\mathsf{tar.bf}}{}|\bm{x}).$
(14)

To minimize the overall risk, the Bayes decision rule tells us to select the
action with a smaller conditional risk [11]. Accordingly, the decision policy
is

${a}=\begin{cases}\mathtt{accept},\quad\text{if
}R(\mathtt{reject}|\bm{x})>R(\mathtt{accept}|\bm{x})\\\
\mathtt{reject},\quad\text{if }R(\mathtt{reject}|\bm{x})\leq
R(\mathtt{accept}|\bm{x})\\\ \end{cases}.$ (15)

In other words, the optimal decision policy for an SASV system is to
$\mathtt{accept}$ the input probe if the conditional risk
$R(\mathtt{accept}|\bm{x})$ is smaller than $R(\mathtt{reject}|\bm{x})$.
Otherwise, the system should $\mathtt{reject}$ the input.

Table 2: Values of decision cost function $C(a,y)$ for SASV. Subscript fa means false acceptance.  |  | Action $a$
---|---|---
True class label $y$ |  | $\mathtt{accept}$ | $\mathtt{reject}$
$\mathsf{tar.bf}{}$ |  | 0 | $C_{\text{miss}}^{\mathsf{tar.bf}{}}$
$\mathsf{non.bf}{}$ |  | $C_{\text{fa}}^{\mathsf{non.bf}{}}$ | 0
$\mathsf{spf}{}$ |  | $C_{\text{fa}}^{\mathsf{spf}{}}$ | 0

With Eqs. (13) and (14), we re-write the condition to accept the input, i.e.,
$R(\mathtt{reject}|\bm{x})>R(\mathtt{accept}|\bm{x})$, as

$\begin{split}C_{\text{miss}}^{\mathsf{tar.bf}{}}{}P({H}{}_{\mathsf{tar.bf}}{}|\bm{x})&>\\\
C_{\text{fa}}^{\mathsf{non.bf}{}}{}&P({H}{}_{\mathsf{non.bf}}{}|\bm{x})+C_{\text{fa}}^{\mathsf{spf}{}}{}P({H}{}_{\mathsf{spf}}{}|\bm{x}).\end{split}$
(16)

Using the Bayes' formula, we write the left and right hand sides as

$C_{\text{miss}}^{\mathsf{tar.bf}{}}{}P({H}{}_{\mathsf{tar.bf}}{}|\bm{x})=C_{\text{miss}}^{\mathsf{tar.bf}{}}{}\frac{p(\bm{x}|{H}{}_{\mathsf{tar.bf}}{})\pi_{\mathsf{tar.bf}{}}}{p(\bm{x})},$
(17)

and

$\begin{split}&C_{\text{fa}}^{\mathsf{non.bf}{}}{}P({H}{}_{\mathsf{non.bf}}{}|\bm{x})+C_{\text{fa}}^{\mathsf{spf}{}}{}P({H}{}_{\mathsf{spf}}{}|\bm{x})\\\
=&C_{\text{fa}}^{\mathsf{non.bf}{}}{}\frac{p(\bm{x}|{H}{}_{\mathsf{non.bf}}{})\pi_{\mathsf{non.bf}{}}}{p(\bm{x})}+C_{\text{fa}}^{\mathsf{spf}{}}{}\frac{p(\bm{x}|{H}{}_{\mathsf{spf}}{})\pi_{\mathsf{spf}{}}}{p(\bm{x})}.\end{split}$
(18)

Plugging the above two equations into Ineq. (16) and cancel $p(\bm{x})$ on
both sides of the inequality, Ineq. (16) can be written in terms of
likelihoods and priors as

$\begin{split}C_{\text{miss}}^{\mathsf{tar.bf}{}}{}p(\bm{x}|{H}{}_{\mathsf{tar.bf}}{})\pi_{\mathsf{tar.bf}{}}&>\\\
C_{\text{fa}}^{\mathsf{non.bf}{}}{}p(\bm{x}|{H}{}_{\mathsf{non.bf}}{})&\pi_{\mathsf{non.bf}{}}+C_{\text{fa}}^{\mathsf{spf}{}}{}p(\bm{x}|{H}{}_{\mathsf{spf}}{})\pi_{\mathsf{spf}{}}.\end{split}$
(19)

By dividing both sides by
$C_{\text{miss}}^{\mathsf{tar.bf}{}}{}p(\bm{x}|{H}{}_{\mathsf{tar.bf}}{})$, we
obtain

$\begin{split}\pi_{\mathsf{tar.bf}{}}>\frac{C_{\text{fa}}^{\mathsf{non.bf}{}}{}p(\bm{x}|{H}{}_{\mathsf{non.bf}}{})}{C_{\text{miss}}^{\mathsf{tar.bf}{}}{}p(\bm{x}|{H}{}_{\mathsf{tar.bf}}{})}\pi_{\mathsf{non.bf}{}}+\frac{C_{\text{fa}}^{\mathsf{spf}{}}{}p(\bm{x}|{H}{}_{\mathsf{spf}}{})}{C_{\text{miss}}^{\mathsf{tar.bf}{}}{}p(\bm{x}|{H}{}_{\mathsf{tar.bf}}{})}\pi_{\mathsf{spf}{}},\end{split}$
(20)

which can be further written as

$\begin{split}\pi_{\mathsf{tar.bf}{}}>\frac{C_{\text{fa}}^{\mathsf{non.bf}{}}{}}{C_{\text{miss}}^{\mathsf{tar.bf}{}}{}}e^{-\mathsf{llr}^{\mathsf{tar.bf}{}}_{\mathsf{non.bf}{}}(\bm{x})}\pi_{\mathsf{non.bf}{}}+\frac{C_{\text{fa}}^{\mathsf{spf}{}}{}}{C_{\text{miss}}^{\mathsf{tar.bf}{}}{}}e^{-\mathsf{llr}^{\mathsf{tar.bf}{}}_{\mathsf{spf}{}}(\bm{x})}\pi_{\mathsf{spf}{}},\end{split}$
(21)

where the LLRs are defined as
$\mathsf{llr}^{\mathsf{tar.bf}{}}_{\mathsf{non.bf}{}}(\bm{x})\triangleq\log\frac{p(\bm{x}|{H}{}_{\mathsf{tar.bf}{}})}{p(\bm{x}|{H}{}_{\mathsf{non.bf}{}})}$
and
$\mathsf{llr}^{\mathsf{tar.bf}{}}_{\mathsf{spf}{}}(\bm{x})\triangleq\log\frac{p(\bm{x}|{H}{}_{\mathsf{tar.bf}{}})}{p(\bm{x}|{H}{}_{\mathsf{spf}{}})}$.

Because the samples from $\mathsf{non.bf}{}$ and $\mathsf{spf}{}$ should be
rejected, they can be conceptually grouped together as a ``negative'' class.
The prior probability of having a sample from this ``negative'' class is
therefore $\pi_{\mathsf{non.bf}{}}+\pi_{\mathsf{spf}{}}$, and, among the
``negative'' samples, the proportion of $\pi_{\mathsf{spf}}$ is defined as

$\rho\triangleq\frac{\pi_{\mathsf{spf}{}}}{\pi_{\mathsf{non.bf}{}}+\pi_{\mathsf{spf}{}}}.$
(22)

The ratio $\rho$ is referred to as _spoof prevalence prior_ in the t-EER [6].
Furthermore, The ratio of the priors of the $\mathsf{tar.bf}{}$ against the
``negative class'' can be defined as

$\beta\triangleq\frac{\pi_{\mathsf{tar.bf}{}}}{\pi_{\mathsf{non.bf}{}}+\pi_{\mathsf{spf}{}}}=\frac{\pi_{\mathsf{tar.bf}{}}}{1-\pi_{\mathsf{tar.bf}{}}}.$
(23)

By dividing both sides of Ineq. (21) with
$\pi_{\mathsf{non.bf}{}}+\pi_{\mathsf{spf}{}}$, we finally obtain

$\begin{split}\beta>\frac{C_{\text{fa}}^{\mathsf{non.bf}{}}{}}{C_{\text{miss}}^{\mathsf{tar.bf}{}}{}}e^{-\mathsf{llr}^{\mathsf{tar.bf}{}}_{\mathsf{non.bf}{}}(\bm{x})}(1-\rho)+\frac{C_{\text{fa}}^{\mathsf{spf}{}}{}}{C_{\text{miss}}^{\mathsf{tar.bf}{}}{}}e^{-\mathsf{llr}^{\mathsf{tar.bf}{}}_{\mathsf{spf}{}}(\bm{x})}\rho,\end{split}$
(24)

In short, to make the optimal decision, an SASV system computes the right hand
side of Ineq. (24) given the LLRs and the priors. The value can be interpreted
as a `score'. The system then compares the score with the left hand side to
make a decision. If the score is smaller than $\beta$, the system
$\mathtt{accept}$ the input.

Note that the procedure assumes that a lower score indicates favors
$\mathtt{accept}$ more. In most implementations, it is more intuitive if a
_higher_ score indicates favors $\mathtt{accept}$ more. This can be easily
done re-written Ineq.(24), for example, as

$\begin{split}-\Big{[}\frac{C_{\text{fa}}^{\mathsf{non.bf}{}}{}}{C_{\text{miss}}^{\mathsf{tar.bf}{}}{}}e^{-\mathsf{llr}^{\mathsf{tar.bf}{}}_{\mathsf{non.bf}{}}(\bm{x})}(1-\rho)+\frac{C_{\text{fa}}^{\mathsf{spf}{}}{}}{C_{\text{miss}}^{\mathsf{tar.bf}{}}{}}e^{-\mathsf{llr}^{\mathsf{tar.bf}{}}_{\mathsf{spf}{}}(\bm{x})}\rho\Big{]}>-\beta.\end{split}$
(25)

We can further take $\log(\cdot)$ on both sides. Notice that the left hand
side increases if the two LLRs increase. In other words, if the LLRs give more
weights to $\mathsf{tar.bf}{}$, the decision favors $\mathtt{accept}$ more.

####  A.1.3 On decoupling LLR and priors

Let us consider an extreme case without spoofed data, i.e., $\rho=0$ and
$\pi_{\mathsf{spf}}{}=0$. Ineq. (24) can be written as

$\begin{split}\beta>\frac{C_{\text{fa}}^{\mathsf{non.bf}{}}{}}{C_{\text{miss}}^{\mathsf{tar.bf}{}}{}}e^{-\mathsf{llr}^{\mathsf{tar.bf}{}}_{\mathsf{non.bf}{}}(\bm{x})},\end{split}$
(26)

or, with $-\log(\cdot)$ applied to both sides of the inequality,

$\begin{split}\mathsf{llr}^{\mathsf{tar.bf}{}}_{\mathsf{non.bf}{}}(\bm{x})>\log\frac{1-\pi_{\mathsf{tar.bf}}{}}{\pi_{\mathsf{tar.bf}}{}}\frac{C_{\text{fa}}^{\mathsf{non.bf}{}}{}}{C_{\text{miss}}^{\mathsf{tar.bf}{}}{}}.\end{split}$
(27)

This is the optimal decision policy for ASV as a typical binary classification
task [27, (2.33)].

Note that the LLRs are on the left hand side, while the priors (and decision
costs) are on the right hand side. Because the LLRs are decoupled from the
priors, the system can be designed to produce the LLRs without considering the
priors. It is the user, who deploys the system into an application, assign the
decision costs, asserted priors, and accordingly the decision threshold. The
decoupled LLRs and priors are preferred in a forensic context [28, 27]. A
similar decision policy can be derived when $\rho=1$, i.e., CM without ASV.

Unfortunately, when $\rho\in(0,1)$, the LLRs and priors cannot be decoupled.
The $\rho$ is needed to compute the score, and the SASV system designer has to
estimate the value of the $\rho$, for example, on the basis of the statistics
from the development data. In our experiment, $\rho$ was found through grid
search by minimizing the decision risks on the development data.

###  A.2 Optimal decision policy when decision costs are equal

For an actual application, we need to assign values to the decision costs on
the basis of the application's requirements. For example, the cost for missing
a sample of $\mathsf{tar.bf}{}$ may be high for a smart-speech device since
the user may be annoyed if their voice is falsely rejected. However, the costs
for falsely accepting $\mathsf{spf}{}$ and $\mathsf{non.bf}{}$ may be high for
online banking. False acceptance of imposters may lead to disastrous financial
loss.

Following the common assumptions in other studies (e.g., [21]), we now
consider a particular case in which
$C_{\text{fa}}^{\mathsf{non.bf}{}}{}=C_{\text{fa}}^{\mathsf{spf}{}}{}=C_{\text{miss}}^{\mathsf{tar.bf}{}}{}$.
Accordingly, the optimal policy of action for the SASV system is to
$\mathtt{accept}$ when

$\begin{split}P({H}{}_{\mathsf{tar.bf}}{}|\bm{x})>P({H}{}_{\mathsf{non.bf}}{}|\bm{x})+P({H}{}_{\mathsf{spf}}{}|\bm{x}),\end{split}$
(28)

or, equivalently,

$\begin{split}\beta>e^{-\mathsf{llr}^{\mathsf{tar.bf}{}}_{\mathsf{non.bf}{}}(\bm{x})}(1-\rho)+e^{-\mathsf{llr}^{\mathsf{tar.bf}{}}_{\mathsf{spf}{}}(\bm{x})}\rho.\end{split}$
(29)

As mentioned above, the SASV system needs to compute the right hand side of
Ineq. (29) as a score. The score is then compared with $\beta$ to make a
decision.

In practice, we assume that a higher SASV score indicates that the input is
more likely from $\mathsf{tar.bf}{}$. Hence, we take the negative logarithm
over the two sides of Ineq. (29) and obtain

$\begin{split}-\log\big{[}e^{-\mathsf{llr}^{\mathsf{tar.bf}{}}_{\mathsf{non.bf}{}}(\bm{x})}(1-\rho)+e^{-\mathsf{llr}^{\mathsf{tar.bf}{}}_{\mathsf{spf}{}}(\bm{x})}\rho\big{]}>-\log\beta.\end{split}$
(30)

Taking the logarithm makes the left hand side similar to conducting a log-sum-
exp operation on the LLRs, 666https://en.wikipedia.org/wiki/LogSumExp albeit a
weighted summation with weights $\rho$ and $1-\rho$.

In implementation, the SASV system computes the score
$s_{\text{sasv}}=-\log\big{[}e^{-\mathsf{llr}^{\mathsf{tar.bf}{}}_{\mathsf{non.bf}{}}(\bm{x})}(1-\rho)+e^{-\mathsf{llr}^{\mathsf{tar.bf}{}}_{\mathsf{spf}{}}(\bm{x})}\rho\big{]}$
for an input $\bm{x}$. It compares $s_{\text{sasv}}$ with the threshold
$-\log\beta$ for making the decision.

Recall that if we assume the ASV and CM score vector
$\bm{s}=[s_{\text{asv}},s_{\text{cm}}]^{\top}$ encodes all the relevant
information about the input sample $\bm{x}$, we can compute the LLRs given
$\bm{s}$ as the observed data. We then obtain the scoring function in Eq. (11)
of Sec. 2.5.

####  A.2.1 Link to Gaussian fusion back-end

Note that

$\begin{split}s_{\text{sasv}}&=-\log\big{[}e^{-\mathsf{llr}^{\mathsf{tar.bf}{}}_{\mathsf{non.bf}{}}(\bm{x})}(1-\rho)+e^{-\mathsf{llr}^{\mathsf{tar.bf}{}}_{\mathsf{spf}{}}(\bm{x})}\rho\big{]}\\\
&=-\log\Big{[}\frac{p(\bm{x}|{H}{}_{\mathsf{non.bf}{}})}{p(\bm{x}|{H}{}_{\mathsf{tar.bf}{}})}(1-\rho)+\frac{p(\bm{x}|{H}{}_{\mathsf{spf}{}})}{p(\bm{x}|{H}{}_{\mathsf{tar.bf}{}})}\rho\Big{]}\\\
&=-\log\Big{[}\frac{(1-\rho)p(\bm{x}|{H}{}_{\mathsf{non.bf}{}})+\rho
p(\bm{x}|{H}{}_{\mathsf{spf}{}})}{p(\bm{x}|{H}{}_{\mathsf{tar.bf}{}})}\Big{]}\\\
&=\log\frac{p(\bm{x}|{H}{}_{\mathsf{tar.bf}{}})}{(1-\rho)p(\bm{x}|{H}{}_{\mathsf{non.bf}{}})+\rho
p(\bm{x}|{H}{}_{\mathsf{spf}{}})}.\end{split}$ (31)

This equation has the same form as the so-called Gaussian fusion back-end [21,
Eq.(1)], but we have shown how it is derived from the optimal decision policy,
given the decision-cost matrix defined in Table 2 and configuration
$C_{\text{fa}}^{\mathsf{non.bf}{}}{}=C_{\text{fa}}^{\mathsf{spf}{}}{}=C_{\text{miss}}^{\mathsf{tar.bf}{}}{}$.

Note that the Gaussian fusion back-end parameterizes each likelihood function
using a Gaussian distribution, but any other parametric distribution can be
used.

####  A.2.2 Link to the sub-optimal decision policy

In Sec. 2.5, we showed that the linear fusion of LLRs is linked to the
decision policy in Ineqs. (7) and (8). For convenience, we write Ineqs. (7)
again below

$P({H}{}_{\mathsf{tar.bf}{}}|\bm{x})^{2}>P({H}{}_{\mathsf{non.bf}{}}|\bm{x})P({H}{}_{\mathsf{spf}{}}|\bm{x}),$
(32)

It has been described that any vector of posterior probabilities
$\bm{P}=[P({H}{}_{\mathsf{spf}{}}|\bm{x}),P({H}{}_{\mathsf{non.bf}{}}|\bm{x}),P({H}{}_{\mathsf{tar.bf}{}}|\bm{x})]^{\top}$
that satisfies Ineq. (28) also satisfies Ineq. (32), but the reverse is not
always true. In other words, Ineq. (32) is a necessary but not sufficient
condition to achieve Ineq. (28).

Ineq. (32) is necessary to achieve Ineq. (28): Note that
$P({H}{}_{\mathsf{tar.bf}{}}|\bm{x})+P({H}{}_{\mathsf{non.bf}{}}|\bm{x})+P({H}{}_{\mathsf{spf}{}}|\bm{x})=1$.
Therefore, Ineq. (28) can be written as

$\begin{split}P({H}{}_{\mathsf{tar.bf}}{}|\bm{x})&>P({H}{}_{\mathsf{non.bf}}{}|\bm{x})+P({H}{}_{\mathsf{spf}}{}|\bm{x})\\\
&=1-P({H}{}_{\mathsf{tar.bf}}{}|\bm{x}),\end{split}$ (33)

or, equivalently, $P({H}{}_{\mathsf{tar.bf}}{}|\bm{x})>0.5$. Therefore, for a
$\bm{P}$ that satisfies Ineq. (28), we know that
$P({H}{}_{\mathsf{tar.bf}}{}|\bm{x})>0.5$,
$P({H}{}_{\mathsf{non.bf}}{}|\bm{x})<0.5$,
$P({H}{}_{\mathsf{spf}}{}|\bm{x})<0.5$, and

$P({H}{}_{\mathsf{tar.bf}}{}|\bm{x})^{2}>0.25>P({H}{}_{\mathsf{non.bf}}{}|\bm{x})P({H}{}_{\mathsf{spf}}{}|\bm{x}).$
(34)

Hence, Ineq. (32) is a necessary condition of Ineq. (28).

Ineq. (32) is not a sufficient condition of Ineq. (28): In Seq. 2.5, we
provided an example $\bm{P}=[0.05,0.65,0.3]$ that satisfies Ineq. (32) but not
Ineq. (28): $0.3^{2}>0.05\times 0.65$ but $0.3<0.05+0.65$. Hence, Ineq. (32)
is not a sufficient condition of Ineq. (28).

###  A.3 Comparing with ternary classification

In Sec.A.1, we defined SASV as a classification task with two actions but
three data classes. We may compare it with a ternary classification task. For
convenience, let the three classes be
$\mathcal{Y}=\\{{H}{}_{1},{H}{}_{2},{H}{}_{3}\\}$. The three possible actions
are to assign the input to one of the three classes.

####  A.3.1 Minimizing the decision cost

We can define $C(a,y)$ in Table 3. According to the Bayes decision rule, the
system should classify the input sample $\bm{x}$ to ${H}{}_{1}$ if

$\sum_{i}C_{i1}P({H}{}_{i}|\bm{x})<\sum_{i}C_{i2}P({H}{}_{i}|\bm{x})$ (35)

_and_

$\sum_{i}C_{i1}P({H}{}_{i}|\bm{x})<\sum_{i}C_{i3}P({H}{}_{i}|\bm{x}).$ (36)

Similar conditions can be written for accepting ${H}{}_{2}$ and ${H}_{3}$.

If $C_{ii}=0$, i.e., no cost for correct action, the condition is

$\begin{split}C_{21}P({H}{}_{2}|\bm{x})+&C_{31}P({H}{}_{3}|\bm{x})\\\
&<C_{12}P({H}{}_{1}|\bm{x})+C_{32}P({H}{}_{3}|\bm{x})\end{split}$ (37)

_and_

$\begin{split}C_{21}P({H}{}_{2}|\bm{x})+&C_{31}P({H}{}_{3}|\bm{x})\\\
&<C_{23}p({H}{}_{2}|\bm{x})+C_{13}p({H}{}_{1}|\bm{x}).\end{split}$ (38)

If we further set $C_{32}=C_{23}=0$ as Table 4 shows, the condition becomes

$\begin{split}C_{21}P({H}{}_{2}|\bm{x})+C_{31}P({H}{}_{3}|\bm{x})<C_{12}P({H}{}_{1}|\bm{x})\end{split}$
(39)

_and_

$\begin{split}C_{21}P({H}{}_{2}|\bm{x})+C_{31}P({H}{}_{3}|\bm{x})<C_{13}p({H}{}_{1}|\bm{x}).\end{split}$
(40)

Merging the two sub-conditions, we further get

$\begin{split}C_{21}P({H}{}_{2}|\bm{x})+C_{31}P({H}{}_{3}|\bm{x})<\min(C_{13},C_{12})p({H}{}_{1}|\bm{x}).\end{split}$
(41)

Table 3: Values of decision cost function $C(a,y)$ for ternary classification.  |  | Action $a$
---|---|---
True class label $y$ |  | 1 | 2 | 3
1 |  | $C_{11}$ | $C_{12}$ | $C_{13}$
2 |  | $C_{21}$ | $C_{22}$ | $C_{23}$
3 |  | $C_{31}$ | $C_{32}$ | $C_{33}$
Table 4: Example values of decision cost function $C(a,y)$. |  | Action $a$
---|---|---
True class label $y$ |  | 1 | 2 | 3
1 |  | 0 | $C_{12}$ | $C_{13}$
2 |  | $C_{21}$ | 0 | $0$
3 |  | $C_{31}$ | 0 | 0
Table 5: Values of decision utility function $U(a,y)$ for ternary classification task.  |  | Action $a$
---|---|---
True class label $y$ |  | 1 | 2 | 3
1 |  | $U_{11}$ | $U_{12}$ | $U_{13}$
2 |  | $U_{21}$ | $U_{22}$ | $U_{23}$
3 |  | $U_{31}$ | $U_{32}$ | $U_{33}$
Table 6: Example values of decision utility function $U(a,y)$. |  | Action $a$
---|---|---
True class label $y$ |  | 1 | 2 | 3
1 |  | $U$ | 0 | 0
2 |  | 0 | $U$ | 0
3 |  | 0 | 0 | $U$

If we further assume $C_{12}=C_{13}$, we get

$\begin{split}C_{21}P({H}{}_{2}|\bm{x})+C_{31}P({H}{}_{3}|\bm{x})<C_{12}p({H}{}_{1}|\bm{x}),\end{split}$
(42)

which has a similar form to the condition for SASV in Ineq. (16).

####  A.3.2 Maximizing the decision utility

In contrast to minimizing the decision cost, we may conduct a Bayesian
decision by maximizing the utility [20]. Let us define a utility function
$U(a,y)$ that returns the utility (or reward) when the system the action $a$
while the ground-truth label is $y$. A generic value table of $U(a,y)$ for a
ternary classification task is shown in Table 5. Note that, different from the
decision cost, we should set utilities so that $U_{ii}>U_{ij}\geq 0,i\neq{j}$.

The system accepts ${H}{}_{1}$ if

$\sum_{i}U_{i1}P({H}{}_{i}|\bm{x})>\sum_{i}U_{i2}P({H}{}_{i}|\bm{x})$ (43)

_and_

$\sum_{i}U_{i1}P({H}{}_{i}|\bm{x})>\sum_{i}U_{i3}P({H}{}_{i}|\bm{x}).$ (44)

Similar conditions can be written for accepting ${H}{}_{2}$ and ${H}_{3}$.

Although the condition looks similar to that for minimizing the decision cost,
a nuance is that we can define a diagonal utility matrix, i.e., $U_{ii}>0$ and
$U_{ij}=0,i\neq{j}$. This means that only the correct decision receives a
reward. In this case, the condition to accept ${H}{}_{1}$ can be written as

$U_{11}P({H}{}_{1}|\bm{x})>U_{22}P({H}{}_{2}|\bm{x})$ (45)

_and_

$U_{11}P({H}{}_{1}|\bm{x})>U_{33}P({H}{}_{3}|\bm{x}).$ (46)

If we further set $U_{ii}=U_{jj}=U,\forall{i,j}$ as Table 6 shows, the
condition becomes

$P({H}{}_{1}|\bm{x})>P({H}{}_{2}|\bm{x})\land
P({H}{}_{1}|\bm{x})>P({H}{}_{3}|\bm{x}),$ (47)

where $\land$ is the logical operator AND.

In [15, sec.10.3], Ineq. (47) and its relationship with compositional data
analysis is discussed. Inequality. (47) is different from Ineq. (28): the
former is derived when maximizing the decision utility given a scalar utility
matrix.

###  A.4 On spoofed sample

A spoofed sample should be rejected by SASV regardless of whether it sounds
like the target speaker. If the spoofed probe is forged by an attacker (i.e.,
biometric imposters in ISO/IEC 30107), it is supposed to sound like the target
speaker's voice; however, the algorithm to create the spoofed probe may not be
good enough to mimic the voice. Hence, the spoofed sample should be rejected
regardless of its similarity to the target speaker's voice.

Suppose the `spoofed' sample is created by an authentic user who wants to make
it less similar to their own voice (i.e., biometric concealers). In that case,
this application is considered more relevant to voice anonymization [29]
rather than SASV. Of course, an SASV system should also reject the `spoofed'
sample since it intended use is to conceal the speaker's identity.