# Tackling Spoofing-Aware Speaker Verification with Multi-Model Fusion

###### Abstract

Recent years have witnessed the extraordinary development of automatic speaker
verification (ASV). However, previous works show that state-of-the-art ASV
models are seriously vulnerable to voice spoofing attacks, and the recently
proposed high-performance spoofing countermeasure (CM) models only focus
solely on the standalone anti-spoofing tasks, and ignore the subsequent
speaker verification process. How to integrate the CM and ASV together remains
an open question. A spoofing aware speaker verification (SASV) challenge has
recently taken place with the argument that better performance can be
delivered when both CM and ASV subsystems are optimized jointly. Under the
challenge’s scenario, the integrated systems proposed by the participants are
required to reject both impostor speakers and spoofing attacks from target
speakers, which intuitively and effectively matches the expectation of a
reliable, spoofing-robust ASV system. This work focuses on fusion-based SASV
solutions and propose a multi-model fusion framework to leverage the power of
multiple state-of-the-art ASV and CM models. The proposed framework vastly
improves the SASV-EER from 8.75% to be 1.17%, which is 86% relative
improvement compared to the best baseline system in the SASV challenge.

##  1 Introduction

Automatic speaker verification (ASV) is a technology to verify claimed
speakers by the given speech segments. Resorting to the power of deep neural
networks (DNNs) in representation learning, speaker embedding-based ASV
approaches have achieved state-of-the-art performances on several benchmark
datasets [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]. Despite the impressive
performance, current ASV models are vulnerable to spoofing attacks, which
include audio replay, text-to-speech (TTS) and voice conversion (VC) [13, 14,
15, 16, 17, 18], back-door attacks [19] and related emerged adversarial
attacks [20, 21]. Various methods have been proposed to tackle adversarial
attacks [22, 23, 24, 25, 26, 27, 28, 29].

At the same time, led by a series of challenges [13, 14, 15, 16], many
research efforts have investigated countermeasure (CM), also known as anti-
spoofing, to defend and detect spoofing attacks. The current methods utilize
end-to-end neural network structures [30, 31] to distinguish between spoofing
speech from bona fide speech by either the raw speech signal or hand-crafted
features. Recently, CM systems have been significantly improved by a series of
works [30, 31, 32, 33, 34, 35, 36, 37]. CM models are vulnerable to
adversarial attacks [38], and some efforts are dedicated to address such
attacks [39, 40].

While ASV and CM systems have been successfully developed as individual tasks,
limited efforts have been devoted to their integration. Jung et al. [41]
recently held a spoofing aware speaker verification (SASV) challenge, which is
a special session in ISCA INTERSPEECH 2022, with the argument that better
performance can be delivered when CM and ASV subsystems are both optimized.
They released official protocols and baselines based on the ASVspoof 2019
dataset [42]. Compared with previous anti-spoofing challenges that measure CM
systems based on fixed ASV models, this challenge considers CM and ASV
together as an integrated system and uses a variant of classic equal error
rate (SASV-EER) as a primary assessment metric. Under this metric, the
integrated systems are required to reject both impostors speakers and spoofing
attacks from target speakers, which intuitively and effectively matches the
expectation of a reliable, spoofing-robust ASV system.

In this work, we focus on fusion-based SASV solutions and propose a multi-
model fusion framework to leverage the power of multiple state-of-the-art ASV
and CM models. The ASV and CM models we covered include MFA-Conformer [8],
Resnet34 [10], ECAPA-TDNN [9] and AASIST [35], AASIST-L, RawGAT-ST [36]. Based
on the proposed framework, we study combinations of different ASV and CM
models. They vastly improve the SASV-EER from 8.75% to 1.17%, which achieves
86% relative improvement compared to the best baseline system in the SASV
challenge.

Figure 1: The proposed multi-model fusion framework.

##  2 Related work

###  2.1 Automatic speaker verification

Automatic speaker verification aims to determine whether a given utterance
belongs to the claimed speaker. Most modern ASV approaches are based on
speaker embeddings, i.e., representing speech segments into fix-length
identity vectors. Speaker verification can be achieved by measuring the
distance between different speaker embeddings. These methods train the models
to distinguish speakers from the variable-length speech signal. Then an
embedding of the bottleneck layer is extracted as a speaker-dependent vector
to identify speaker information. Following the speaker embedding scheme,
current ASV systems mainly involve two steps: the speaker embeddings are
extracted from the enrollment and test speech to respectively represent the
identification of the target speaker and verified speech segment. Then, a
metric is used to score the test speech based on enrollment speech, where the
metric can be cosine distance or more advanced statistical models like PLDA
[43]. After many years’ development, many technologies and typologies have
been introduced to further improve speaker embedding models, including better
structures [5, 9, 10], training schemes [44, 45] and pooling approaches [46,
47].

###  2.2 Countermeasures

Countermeasure approaches for ASV systems are developed to detect the spoofing
samples generated by attack techniques, including text-to-speech, voice
conversion, and audio replay. Conventional CM models use a Gaussian Mixture
Model (GMM) classifier as back-end with different input features [48, 49, 50].
In [51], a combination of CNN and RNN models is proposed as a DNN-based
solution. [31] introduced Resnet in this task, which is further improved by
[34] using large margin cosine loss and frequency masking data augmentation.
[36] was first to introduce RawNet that directly uses raw speech waveform as
input. [52] introduced graph attention networks (GAT) with the ability of
modeling the relationships between neighbouring sub-bands or segments of
spectrum features. Based on GAT, a series of extensions are proposed in [36,
35] using both raw audio and spectrum features. Besides, a one-class learning
method was proposed in [32] for better generalization ability. Such previous
efforts presented a variety of anti-spoofing models with high performance.

##  3 Methodology

###  3.1 SASV systems

There are two main categories of spoofing-aware speaker verification (SASV)
systems: multi-task learning based systems and fusion based systems. The
multi-task learning systems are trained by both the anti-spoofing loss and
speaker verification loss. The speaker verification task and anti-spoofing
task share the same input features and the embedding backbone layers. However,
such a system requires both speaker labels and anti-spoofing labels for
training. Only the ASVspoof dataset [53] can fulfill such requirements. The
ASVspoof dataset contains only a small number of speakers, which will induce
the trained multi-task learning system to overfit to only a small number of
speakers, resulting limited generalization capacity to real-world applications
[41]. Another disadvantage of such systems is that the learning objectives of
anti-spoofing and speaker verification systems are different – anti-spoofing
models rely on the device information to detect the spoofing utterances, yet
the speaker verification systems tend to remove such information which is
incongruent with speaker discrimination [54].

The other kind of SASV system is the fusion-based systems, which may involve
embedding-level or score-level fusion. Such systems can rely on state-of-the-
art speaker verification models trained on large-scale datasets [4, 5] and
state-of-the-art spoofing countermeasure models.

Thus, the SASV challenge adopts the second type of systems as baselines. We
will introduce the details of such systems in the next subsection.

###  3.2 Baseline systems

The SASV Challenge 2022 provides two baselines. The organizers provided one
pre-trained speaker verification model, ECAPA-TDNN, and one anti-spoofing
model, AASIST. Both models attain state-of-the-art performance for their own
tasks.

Baseline1 is a score-level fusion method. For a given trial, which consists of
an enrollment utterance and a testing utterance, ECAPA-TDNN will be used to
obtain the enrollment embedding and the testing embedding from the enrollment
and testing utterances. Then the cosine score of the enroll embedding and
testing embedding is calculated. The anti-spoofing score is also obtained from
the spoofing countermeasure (CM), AASIST, given the testing utterance as the
input. Subsequent addition of the SV and CM scores produces the final score,
which indicates whether such a trail is target trial or not.

Baseline2 is an embedding-level fusion method. ECAPA-TDNN is used to obtain
the enrollment embedding and testing embedding, and AASIST is used to obtain
the countermeasure embedding. These three embeddings are concatenated and fed
to a deep neural network to predict whether the trial belongs to the target or
not.

###  3.3 Proposed systems

The baselines of the SASV Challenge do achieve better performance, compared
with using speaker verification models or anti-spoofing models alone, as shown
in Table 4. However, Baseline1 does not consider that the score scales of ASV
score and CM score are in different ranges, and Baseline2 does not consider
fine-grained fusion. Thus they cannot take good advantage of the state-of-the-
art ASV and anti-spoofing models in boosting their performances.

This motivates us to propose an ensemble model for the SASV task. We will
first report the details of the framework followed by rational analysis. The
proposed framework is shown in Figure 1. ASV-1, ASV-2, …, ASV-$m$ denote $m$
ASV pre-trained models. CM-1, CM-2, …, CM-$n$ denote $n$ countermeasure
models. $x_{e}$ and $x_{t}$ are the enrollment and testing utterances,
respectively. Given a trial composed of an enrollment and testing utterance,
$m$ cosine scores, $s_{sv}^{1},s_{sv}^{2},...,s_{sv}^{m}$ are calculated by
$m$ pre-trained speaker verification models. $\oplus$ denotes the
concatenation procedure. $f$ and $g$ are the trainable countermeasure (CM)
layer and prediction layer, respectively. Given the testing utterance, $n$
testing embeddings, $h_{1},h_{2},...,h_{n}$ are extracted by $n$ pre-trained
anti-spoofing models, respectively. Then the $n$ testing embeddings are
firstly concatenated, and then fed into the countermeasure layer to get the
countermeasure score $s_{cm}$. We use the cross-entropy loss on $s_{cm}$ to
guide the CM layer learn to distinguish between the target trials and other
trials:

$L_{cm}=-log\frac{exp(s^{l}_{cm})}{\sum_{j=0}^{1}exp(s^{j}_{cm})},$ (1)

where $l\in\\{0,1\\}$ is the label, indicating whether the trail is target or
not. Then we concatenate $s_{cm}$ and $s_{sv}^{1},s_{sv}^{2},...,s_{sv}^{m}$,
and then input them to the prediction layer to get the final prediction
$\hat{y}$ Cross-entropy is derived to train the entire model:

$L_{pr}=-log\frac{exp(\hat{y}_{l})}{\sum_{j=0}^{1}exp(\hat{y}_{j})},$ (2)

where $l\in\\{0,1\\}$ is the label, denoting whether the input trail is target
or not. To be specific, 1 denotes the non-spoofed target trial. The final loss
is:

$L=L_{cm}+L_{pr}.$ (3)

Previous years have witnessed the development of state-of-the-art speaker
verification models that are trained on very large datasets. Well-trained ASV
models attain very good performance for in-domain data as shown in Table
LABEL:tab:asv_eer, and have good generalization and reliability for out-of-
domain data as shown in Table 4. As a result, we directly use the scores
provided by the ASV for fusion. Anti-spoofing models detect artifacts located
in specific spectrum segments or temporal segments [35], and different anti-
spoofing models focus on different kinds of artifact. Thus we propose to
ensemble different CM embeddings from a group of countermeasure models. The
results shown in Table 4 and Figure 2 show the effectiveness of the proposed
method.

##  4 Experimental setup

###  4.1 Dataset

The SASV Challenge restricts the training data for ASV to the VoxCeleb2
development set, and the training data for the anti-spoofing model to ASVspoof
2019 training set. We strictly follow the constraints released by the
Challenge and more details can be found in [41].

Released with Automatic Speaker Verification Spoofing And Countermeasures
Challenge 2019, the ASVspoof 2019 dataset aims to determine whether the
advances in text-to-speech and voice conversion technology pose increased
threat to automatic speaker verification and the reliability of spoofing
countermeasures [53]. It aligns the anti-spoofing task closely with the ASV
task, and sparks the community’s interest in the SASV Challenge. The ASVspoof
2019 dataset consists of bona fide speech trails collected from the VCTK
corpus [55], and spoofing attack trials synthesised by text-to-speech, voice
conversion, or replay. The speakers, with their corresponding bona fide and
spoofing trails, are partitioned into three subsets, namely the training (20
speakers with 2,580 bona fide trails and 22,800 spoofing trails), development
(20 speakers with 2,458 bona fide trails and 22,296 spoofing trails) and
evaluation sets (67 speakers with 7,355 bona fide trails and 63,882 spoofing
trails). Moreover, the spoofing attack algorithms used in the evaluation set
are different from those in training and development set, which is intended to
promote the generalization performance of the developed systems. Based on the
ASVspoof 2019 dataset, the SASV Challenge provided the development and
evaluation protocols which list target, non-target, and spoof trials, and for
each them, multiple enrolment utterances exist.

For ASV training, VoxCeleb2 [5] is a benchmark dataset for speaker
verification, containing audio clips collected from YouTube. There are totally
1,092,009 utterances from 5,994 speakers in the VoxCeleb2 development set.

###  4.2 Evaluation metrics

Three kinds of EERs, SV-EER SPF-EER and SASV-EER, are used by the SASV
Challenge, and SASV-EER on the ASVspoof 2019 evaluation set is adopted as the
main evaluation metric. Equal error rate (EER) is the overall error rate when
a threshold is set such that the false accept rate equals the false reject
rate. The lower the EER, the better the discrimination capacity the model.
Widely used in binary classification problems, it is a more objective and
comprehensive metric than accuracy, especially on data with unbalanced
classes.

In this challenge, EERs are used to evaluate the performance of the models on
three sub-tasks: ASV, anti-spoofing, and SASV [41]. The definition of the EERs
are showed in Table LABEL:tab:metric. For all three sub-tasks, trails from the
target speaker are treated as positive samples, while the definition of
negative samples is different. For the SASV sub-task, both the non-target
speakers’ bona fide speech trails and spoofing trails are considered negative
samples. For ASV sub-task, only the non-target speakers’ bona fide speech
trails are considered as negative samples. For anti-spoofing sub-task, only
spoofing trails aimed at target speaker are considered as negative samples,
which is different from other general anti-spoofing tasks.

Table 1:  Description of EERs. ”+” denotes positive samples, and ”-” denotes negative samples. Blank denotes samples not considerated in the calculation of the EER. Metrics | Target | Non-target | Spoof
---|---|---|---
SASV-EER | + | - | -
SV-EER | + | - |
SPF-EER | + |  | -

###  4.3 Implementation setup

####  4.3.1 ASV

Following the requirements of the SASV challenge, we only use the development
set of VoxCeleb2 as the training data. During training, we randomly extract
3-second segment of each segments and use the on-the-fly feature extraction
procedure. The input features for training are 80-dimensional Mel spectrogram
with the window length of 25 ms and the hop size of 10 ms. No data
augmentation and voice activity detection are applied.

For training the speaker verification backbone models, we adopt three well-
known structures, namely, MFA-Conformer [8], ECAPA-TDNN [9] and Resnet34 [56].
For the ECAPA-TDNN, we use the pre-trained model provided by the SASV
Challenge. MFA-Conformer and Resnet34 are trained by the additive margin
Softmax (AM-Softmax) loss [57] with the margin as 0.2 and the scaling factor
as 30. The speaker embedding dimensions of all three models are 192. Adam
optimizer with an initial learning rate of 0.001 is adopted and the learning
rate decreases by 50% every 4 epoch. Also the weight decay is set as 1e-7 to
avoid overfitting.

Then we evaluate the three models using equal error rate (EER) on the
VoxCeleb1-O testing set [4]. Cosine scoring is implemented and the results are
shown in Table LABEL:tab:asv_eer. As we can see, the MFA-Conformer backbone
achieves the best EER of 0.91%, which outperforms the ECAPA-TDNN provided by
the SASV challenge.

Table 2: ASV performance on VoxCeleb1-O testing set. Model | Frontend | EER(%)
---|---|---
MFA-Conformer | Mel spectrogram | 0.91
Resnet34 | Mel spectrogram | 1.50
ECAPA-TDNN | Mel spectrogram | 0.96

####  4.3.2 Anti-spoofing

For the anti-spoofing model, we adopt AASIST [35], AASIST-L, and RawGAT-ST
[36]. AASIST-L is the light version of AASIST. The AASIST pre-trained model is
provided by the SASV challenge. The training data for the three models is from
the logical access of ASVspoof 2019 [53], which is composed of spoofing audios
generated by text-to-speech and voice conversion. The evaluation metrics for
the anti-spoofing model on the evaluation set are min-tDCF and EER. As we can
see in Table LABEL:tab:cm_eer, AASIST is with the best performance, namely the
lowest min-tDCF and EER.

Table 3: Anti-spoofing model performance on ASVspoof 2019 evaluation set. Model | Frontend | min-tDCF | EER(%)
---|---|---|---
RawGAT-ST | Raw waveform | 0.0335 | 1.06
AASIST-L | Raw waveform | 0.0309 | 0.99
AASIST | Raw waveform | 0.0275 | 0.83

####  4.3.3 Fusion method

We adopt the models in Section 4.3.1 and Section 4.3.2 as the ASV models and
CM models, respectively, as shown in Figure 1. For the CM layer $f$ in Figure
1,we use three fully connected layers with hidden dimensions 256, 128 and 64
respectively. For the prediction layer $g$, we use one linear fully connected
layer. We train our model using Adam optimizer with an initial learning rate
as 0.0001. The batch size is 32 and the epoch number is set as 200. We
selected the best model based on the development set of the ASVspoof 2019
dataset.

##  5 Experimental results and analysis

Table 4: Performance of all systems on the ASVspoof 2019 development and evaluation dataset. Three evaluation metrics, namely SV-EER, SPF-EER and SASV-EER are illustrated. | Model |  SV-EER |  SPF-EER |  SASV-EER
---|---|---|---|---
| dev | eval | dev | eval | dev | eval
(A1) | ECAPA-TDNN | 1.64 | 1.86 | 20.28 | 30.75 | 17.37 | 23.84
(A2) | MFA-Conformer | 1.61 | 1.38 | 19.94 | 30.22 | 16.91 | 23.28
(A3) | Resnet34 | 1.68 | 1.08 | 17.40 | 29.76 | 14.62 | 22.69
(B1) | AASIST | 46.01 | 49.24 | 0.07 | 0.67 | 15.86 | 24.38
(B2) | AASIST-L | 48.30 | 49.04 | 0.13 | 0.84 | 15.72 | 24.81
(B3) | RawGAT-ST | 51.25 | 49.24 | 0.34 | 0.96 | 15.96 | 24.85
(C1) |  Baseline1 [41] | 14.89 | 35.10 | 6.94 | 0.50 | 2.09 | 19.15
(C2) |  Baseline2 [41] | 14.38 | 16.01 | 0.01 | 1.23 | 5.41 | 8.75
(D1) | MFA-Conformer_AASIST | 1.48 | 1.47 | 0.2 | 1.08 | 0.88 | 1.35
(D2) | ECAPA-TDNN_AASIST | 1.48 | 1.58 | 0.2 | 1.06 | 0.99 | 1.42
(D3) | Resnet34_AASIST | 1.49 | 1.02 | 0.20 | 1.53 | 0.88 | 1.32
(D4) | MFA-Conformer_AASIST-L | 1.68 | 1.68 | 0.15 | 2.03 | 1.21 | 1.83
(D5) | ECAPA-TDNN_AASIST-L | 1.61 | 1.62 | 0.19 | 2.18 | 1.10 | 1.92
(D6) | Resnet34_AASIST-L | 1.68 | 1.69 | 0.18 | 2.01 | 1.15 | 1.84
(D7) | MFA-Conformer_RawGAT-ST | 2.16 | 2.09 | 0.35 | 0.78 | 1.55 | 1.82
(D8) | ECAPA-TDNN_RawGAT-ST | 2.56 | 2.17 | 0.04 | 0.78 | 1.81 | 1.94
(D9) | Resnet34_RawGAT-ST | 2.09 | 1.97 | 0.40 | 0.79 | 1.55 | 1.69
(E1) | MFA-Conformer_CM-ALL | 1.91 | 1.66 | 0.20 | 0.64 | 1.01 | 1.30
(E2) | ECAPA-TDNN_CM-ALL | 1.39 | 1.73 | 0.20 | 0.74 | 0.81 | 1.40
(E3) | Resnet34_CM-ALL | 1.28 | 1.12 | 0.26 | 1.43 | 0.74 | 1.32
(F1) | SV-ALL_AASIST | 1.42 | 1.30 | 0.27 | 1.61 | 0.81 | 1.41
(F2) | SV-ALL_AASIST-L | 1.42 | 1.33 | 0.47 | 3.99 | 0.88 | 2.95
(F3) | SV-ALL_RawGAT-ST | 1.82 | 1.64 | 0.4 | 0.82 | 1.28 | 1.39
(G1) | SV-ALL_CM-ALL | 1.27 | 1.20 | 0.20 | 1.15 | 0.81 | 1.17

In this section, we will first show the experimental results, followed by the
experimental observations and analysis.

(a) MFA-Conformer

(b) ECAPA-TDNN

(c) Resnet34

(d) AASIST

(e) AASIST-L

(f) RawGAT-ST

(g) Baseline1

(h) Baseline2

(i) SV-ALL_CM-ALL

Figure 2: The histogram plots of different systems.

###  5.1 Experimental results

Table 4 shows the three evaluation metrics provided by the SASV Challenge on
the ASVspoof 2019 development and evaluation sets. A1-A3 denote using only
speaker verification model. B1-B3 denote using only anti-spoofing model. C1
and C2 are the two baselines provided by the SASV challenge. D1-D9 denote
using the fusion of one speaker verification model and one anti-spoofing
model, e.g., MFA-Conformer_AASIST means the combination of MFA-Conformer as
speaker verification and AASIST as the anti-spoofing model. E1-E3 denote using
the fusion of one speaker verification model with all the anti-spoofing
models. F1-F3 denote the fusion of all three speaker verification models and
one anti-spoofing model. G1 denotes the fusion of all the speaker verification
models and anti-spoofing models.

Figure 2 illustrates the histogram plots of all the systems. All three kinds
of trials, including target, non-target and spoof trials, are taken into
account. The x-axis shows the prediction scores that indicate whether the
trial is from the target or not. (a)-(c) are the plots for systems only using
speaker verification models, including MFA-Conformer, ECAPA-TDNN and Resnet34.
(d)-(f) are the plots for systems only using the anti-spoofing models,
including AASIST, AASIST-L and RawGAT-ST. (g) and (h) are the plots for
Baseline1 and Baseline2. (i) is the plot for the system using all three
speaker verification models and anti-spoofing models for fusion.

###  5.2 Observations and analysis

We have the following observations and analysis:

Only using speaker verification models. A1-A3 models, which are state-of-the-
art ASV frameworks, performed well on the sub-task speaker verification, as
shown in Table LABEL:tab:asv_eer. Regarding the speaker verification
performance, they generalize well to the ASVspoof 2019 evaluation set, e.g.,
the evaluation SV-EERs for the ECAPA-TDNN, MFA-Conformer and Resnet34 are
1.86%, 1.38% and 1.08%, respectively. However, they give poor results on
detecting spoofing attacks, with evaluation SPF-EERs being 30.75%, 30.22% and
29.76% for the ECAPA-TDNN, MFA-Conformer and Resnet34, respectively. Also, as
shown in Figure 2 (a)-(c), according to the distribution of the predicted
scores, ASV models attain poor ability in distinguishing between spoofing
trials and target trials, although they easily discriminate between target and
non-target trials. Finally, the SASV-EERs are spoiled by the spoofing trials.
This phenomenon is reasonable, as the ASV is trained to distinguish different
speakers. And in order to acquire good speaker verification ability, the ASV
model sometimes has to remove the incongruent information for speaker
discrimination, such as microphone information, channel information, yet such
information is helpful for anti-spoofing tasks.

Only using anti-spoofing models. CM models (B1-B3) work well in detecting
spoofing trials, showing satisfactory performance on SPF-EER, e.g. AASIST,
AASIST-L and RawGAT-ST are with the evaluation SPF-EER of 0.67%, 0.84% and
0.96% respectively as shown in Table 4. However, because of their speaker-
unrelated training objective, they are incapable of verifying speakers,
yielding SV-EERs close to 50%, which indicates that their prediction for
speaker verification task is nearly a random guess. Unsurprisingly, we also
note that the histograms of the target and non-target speakers almost
completely overlap in Figure 2 (d)-(f), which is another evidence for their
poor ability to distinguish among speakers. As a result, the performance of
spoofing-aware speaker verification of the countermeasure models are limited,
e.g., AASIST, AASIST-L and RawGAT-ST are with the evaluation SASV-EERs of
24.38%, 24.81% and 24.85%, respectively. The anti-spoofing models are trained
to distinguish whether an utterance is spoof or not, and the training
criterion has no relation to speaker verification.

Two baselines. Compared to the individual models, the two baseline fusion
models are superior to using only ASV models or the spoofing countermeasure
models alone in the SASV Challenge. Among them, Baseline2 achieves an
evaluation SASV-EER of 8.75%, which is better than Baseline1 with an
evaluation SASV-EER of 19.15%, which is expectable. This is because simple
score-level fusion used in Baseline1 does not guarantee that the output scores
of ASV and countermeasure belong to a unified space. The ASV scores range from
-1 to 1, yet the countermeasure scores range from -20 to 15. Computing simple
additions will make the countermeasure score dominate the ASV score. Figure 2
(g)(h) show that the distributions of the scores predicted by Baseline1 for
target and non-target speak-ers are highly overlapped, while those of
Baseline2 are much more separated. Baseline2 uses trainable deep neural
networks to do better fusion among the ASV features and countermeasure
features, and they can make a proportion of non-target trials distinguishable
from the target trials as shown in Figure 2 (h). However, the performance of
Baseline2 on SV sub-task is still not satisfactory, which motivates us to
explore further possibilities for fusion-based models.

Proposed approach. The proposed models fuse 3 SOTA ASV models (A1-A3) and 3
SOTA anti-spoofing CM models (B1-B3). We have developed a total of 16 models
with comprehensive experiments based on the proposed multi-model fusion
framework. As shown in Table 4, all the proposed models outperform the
baseline models with a large margin on the SASV-EER metrics, while retaining
universally good performances on SV-EER and SPF-EER. Empowered by the proposed
framework, fusion models own significant discrimination capacity regarding the
three tasks. Among them, 9 models (3 by 3), D1-D9, are fused from the
permutations of ASV-CM model pairs. D1-D3, which incorporate ASSIST as the CM
model, have better results than fusion models incorporating other single CM
model. D4-D6, which incorporate ASSIST-L as the CM model, show slight over-
fitting on anti-spoofing task and lower EER on the ASV task compared with the
respective individual models. D7-D9, which incorporate RawGAT-ST as the CM
model, retain superior performance on anti-spoofing task but compromise the
SV-EER. Further, we constructed 6 more models by involving all 3 ASV models
with single CM model (E1-E3), and by involving single ASV model with all 3 CM
models (F1-F3), respectively. The models show better performance than D1-D9 in
general, benefiting from the more comprehensive model ensemble. F2 shows over-
fitting in the anti-spoofing and SASV tasks. Finally, to push the limits of
the multi-modal fusion framework, we incorporate all 3 ASV models and all 3 CM
models and construct SV-ALL_CM-A model (G1). As expected, the model yields a
SASV-EER of 1.17% given the evaluation trials, which gives the best
performance among all models and achieves an 86.63% relative boost compared to
Baseline2. Besides, G1 gave performances comparable with relevant SOTA models
in the other two sub-tasks. Figure 2 (i) also shows that the distributions of
the 3 classes are well-separated, which verifies the effectiveness of the
proposed method.

##  6 Conclusion

In order to take advantage of multiple state-of-the-art ASV and CM models, we
propose a fusion-based framework to solve the SASV task. We comprehensively do
the experiments to show the effectiveness of the proposed method. The best
fusion model effectively decreases the SASV-EER of the best baseline from
8.75% to 1.17%, achieving an improvement of 86% relative (6.58% absolute).

##  7 Acknowledgements

This research is funded by the Centre for Perceptual and Interactive
Intelligence, an InnoCentre of The Chinese University of Hong Kong. This work
was done while Haibin Wu was a visiting student at The Chinese University of
Hong Kong. Haibin Wu is supported by Google PhD Fellowship Scholarship.

## References

  * [1] David Snyder, Daniel Garcia-Romero, Gregory Sell, Daniel Povey, and Sanjeev Khudanpur,  “X-vectors: Robust dnn embeddings for speaker recognition,”  in ICASSP 2018. IEEE, 2018, pp. 5329–5333.
  * [2] Ehsan Variani, Xin Lei, Erik McDermott, Ignacio Lopez Moreno, and Javier Gonzalez-Dominguez,  “Deep neural networks for small footprint text-dependent speaker verification,”  in 2014 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2014, pp. 4052–4056.
  * [3] Koji Okabe, Takafumi Koshinaka, and Koichi Shinoda,  “Attentive statistics pooling for deep speaker embedding,”  in Interspeech, 2018, pp. 2252–2256.
  * [4] Arsha Nagrani, Joon Son Chung, and Andrew Zisserman,  “Voxceleb: a large-scale speaker identification dataset,”  arXiv preprint arXiv:1706.08612, 2017.
  * [5] Joon Son Chung, Arsha Nagrani, and Andrew Zisserman,  “Voxceleb2: Deep speaker recognition,”  arXiv preprint arXiv:1806.05622, 2018.
  * [6] Yue Fan, JW Kang, LT Li, KC Li, HL Chen, ST Cheng, PY Zhang, ZY Zhou, YQ Cai, and Dong Wang,  “Cn-celeb: a challenging chinese speaker recognition dataset,”  in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 7604–7608.
  * [7] Lantian Li, Ruiqi Liu, Jiawen Kang, Yue Fan, Hao Cui, Yunqi Cai, Ravichander Vipperla, Thomas Fang Zheng, and Dong Wang,  “Cn-celeb: multi-genre speaker recognition,”  Speech Communication, 2022.
  * [8] Yang Zhang, Zhiqiang Lv, Haibing Wu, Shanshan Zhang, Pengfei Hu, and Zhiyong Wu,  “Mfa-conformer: Multi-scale feature aggregation conformer for automatic speaker verification,”  arXiv preprint, 2022.
  * [9] Brecht Desplanques, Jenthe Thienpondt, and Kris Demuynck,  “Ecapa-tdnn: Emphasized channel attention, propagation and aggregation in tdnn based speaker verification,”  arXiv preprint arXiv:2005.07143, 2020.
  * [10] Hossein Zeinali, Shuai Wang, Anna Silnova, Pavel Matějka, and Oldřich Plchot,  “But system description to voxceleb speaker recognition challenge 2019,”  arXiv preprint arXiv:1910.12592, 2019.
  * [11] Lantian Li, Yixiang Chen, Ying Shi, Zhiyuan Tang, and Dong Wang,  “Deep speaker feature learning for text-independent speaker verification,”  arXiv preprint arXiv:1705.03670, 2017.
  * [12] Lantian Li, Yiye Lin, Zhiyong Zhang, and Dong Wang,  “Improved deep speaker feature learning for text-dependent speaker recognition,”  in 2015 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA). IEEE, 2015, pp. 426–429.
  * [13] Zhizheng Wu, Tomi Kinnunen, et al.,  “Asvspoof 2015: the first automatic speaker verification spoofing and countermeasures challenge,”  in Sixteenth Annual Conference of the International Speech Communication Association, 2015.
  * [14] Tomi Kinnunen, Md Sahidullah, et al.,  “The asvspoof 2017 challenge: Assessing the limits of replay spoofing attack detection,”  ISCA (the International Speech Communication Association), 2017\.
  * [15] Massimiliano Todisco, Xin Wang, et al.,  “Asvspoof 2019: Future horizons in spoofed and fake audio detection,”  arXiv preprint arXiv:1904.05441, 2019.
  * [16] Junichi Yamagishi, Xin Wang, et al.,  “Asvspoof 2021: accelerating progress in spoofed and deepfake speech detection,”  arXiv preprint arXiv:2109.00537, 2021.
  * [17] Jiangyan Yi, Ruibo Fu, Jianhua Tao, Shuai Nie, Haoxin Ma, Chenglong Wang, Tao Wang, Zhengkun Tian, Ye Bai, Shan Liang, Shiming Wang, Shuai Zhang, Xinrui Yan, Le Xu, and Haizhou Li,  “Add 2022: the first audio deep synthesis detection challenge,”  in ICASSP. IEEE, 2022.
  * [18] Xin Wang and Junich Yamagishi,  “A comparative study on recent neural spoofing countermeasures for synthetic speech detection,”  arXiv preprint arXiv:2103.11326, 2021.
  * [19] Tongqing Zhai, Yiming Li, Ziqi Zhang, Baoyuan Wu, Yong Jiang, and Shu-Tao Xia,  “Backdoor attack against speaker verification,”  in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 2560–2564.
  * [20] Felix Kreuk, Yossi Adi, Moustapha Cisse, and Joseph Keshet,  “Fooling end-to-end speaker verification with adversarial examples,”  in ICASSP. IEEE, 2018, pp. 1962–1966.
  * [21] Rohan Kumar Das, Xiaohai Tian, Tomi Kinnunen, and Haizhou Li,  “The attacker’s perspective on automatic speaker verification: An overview,”  arXiv preprint arXiv:2004.08849, 2020.
  * [22] Qing Wang, Pengcheng Guo, Sining Sun, Lei Xie, and John HL Hansen,  “Adversarial regularization for end-to-end robust speaker verification.,”  in Interspeech, 2019, pp. 4010–4014.
  * [23] Xu Li, Na Li, Jinghua Zhong, Xixin Wu, Xunying Liu, Dan Su, Dong Yu, and Helen Meng,  “Investigating robustness of adversarial samples detection for automatic speaker verification,”  arXiv preprint arXiv:2006.06186, 2020.
  * [24] Hanyi Zhang, Longbiao Wang, Yunchun Zhang, Meng Liu, Kong Aik Lee, and Jianguo Wei,  “Adversarial separation network for speaker recognition,”  Proc. Interspeech 2020, pp. 951–955, 2020.
  * [25] Haibin Wu, Xu Li, Andy T Liu, Zhiyong Wu, Helen Meng, and Hung-yi Lee,  “Adversarial defense for automatic speaker verification by cascaded self-supervised learning models,”  in ICASSP. IEEE, 2021, pp. 6718–6722.
  * [26] Haibin Wu et al.,  “Improving the adversarial robustness for speaker verification by self-supervised learning,”  arXiv preprint arXiv:2106.00273, 2021.
  * [27] Sonal Joshi, Jesús Villalba, Piotr Żelasko, Laureano Moro-Velázquez, and Najim Dehak,  “Adversarial attacks and defenses for speaker identification systems,”  arXiv preprint arXiv:2101.08909, 2021.
  * [28] Haibin Wu, Yang Zhang, Zhiyong Wu, Dong Wang, and Hung-yi Lee,  “Voting for the right answer: Adversarial defense for speaker verification,”  arXiv preprint arXiv:2106.07868, 2021.
  * [29] Haibin Wu et al.,  “Spotting adversarial samples for speaker verification by neural vocoders,”  arXiv preprint arXiv:2107.00309, 2021.
  * [30] Joao Monteiro and Jahangir Alam,  “Development of voice spoofing detection systems for 2019 edition of automatic speaker verification and countermeasures challenge,”  in 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2019, pp. 1003–1010.
  * [31] Joao Monteiro, Jahangir Alam, and Tiago H Falk,  “Generalized end-to-end detection of spoofing attacks to automatic speaker recognizers,”  Computer Speech & Language, vol. 63, pp. 101096, 2020.
  * [32] You Zhang, Fei Jiang, and Zhiyao Duan,  “One-class learning towards synthetic voice spoofing detection,”  IEEE Signal Processing Letters, vol. 28, pp. 937–941, 2021.
  * [33] Haibin Wu, Heng-Cheng Kuo, Naijun Zheng, Kuo-Hsuan Hung, Hung-yi Lee, Yu Tsao, Hsin-min Wang, and Helen Meng,  “Partially fake audio detection by self-attention-based fake span discovery,”  arXiv preprint arXiv:2202.06684, 2022.
  * [34] Tianxiang Chen, Avrosh Kumar, Parav Nagarsheth, Ganesh Sivaraman, and Elie Khoury,  “Generalization of audio deepfake detection,”  in Proc. Odyssey 2020 The Speaker and Language Recognition Workshop, 2020, pp. 132–137.
  * [35] Jee-weon Jung, Hee-Soo Heo, Hemlata Tak, Hye-jin Shim, Joon Son Chung, Bong-Jin Lee, Ha-Jin Yu, and Nicholas Evans,  “Aasist: Audio anti-spoofing using integrated spectro-temporal graph attention networks,”  arXiv preprint arXiv:2110.01200, 2021.
  * [36] Hemlata Tak, Jee-weon Jung, Jose Patino, Madhu Kamble, Massimiliano Todisco, and Nicholas Evans,  “End-to-end spectro-temporal graph attention networks for speaker verification anti-spoofing and speech deepfake detection,”  arXiv preprint arXiv:2107.12710, 2021.
  * [37] Xu Li et al.,  “Replay and synthetic speech detection with res2net architecture,”  in ICASSP. IEEE, 2021, pp. 6354–6358.
  * [38] Songxiang Liu, Haibin Wu, Hung-yi Lee, and Helen Meng,  “Adversarial attacks on spoofing countermeasures of automatic speaker verification,”  in ASRU. IEEE, 2019, pp. 312–319.
  * [39] Haibin Wu, Andy T Liu, and Hung-yi Lee,  “Defense for black-box attacks on anti-spoofing models by self-supervised learning,”  arXiv preprint arXiv:2006.03214, 2020.
  * [40] Haibin Wu, Songxiang Liu, Helen Meng, and Hung-yi Lee,  “Defense against adversarial attacks on spoofing countermeasures of asv,”  in ICASSP. IEEE, 2020, pp. 6564–6568.
  * [41] Jee-weon Jung, Hemlata Tak, Hye-jin Shim, Hee-Soo Heo, Bong-Jin Lee, Soo-Whan Chung, Hong-Goo Kang, Ha-Jin Yu, Nicholas Evans, and Tomi Kinnunen,  “Sasv challenge 2022: A spoofing aware speaker verification challenge evaluation plan,”  arXiv preprint arXiv:2201.10283, 2022.
  * [42] Junichi Yamagishi, Massimiliano Todisco, et al.,  “Asvspoof 2019: The 3rd automatic speaker verification spoofing and countermeasures challenge database,”  2019\.
  * [43] Sergey Ioffe,  “Probabilistic linear discriminant analysis,”  in European Conference on Computer Vision. Springer, 2006, pp. 531–542.
  * [44] Zhifu Gao, Yan Song, Ian Vince McLoughlin, Pengcheng Li, Yiheng Jiang, and Li-Rong Dai,  “Improving aggregation and loss function for better embedding learning in end-to-end speaker verification system.,”  in INTERSPEECH, 2019, pp. 361–365.
  * [45] Jianfeng Zhou, Tao Jiang, Zheng Li, Lin Li, and Qingyang Hong,  “Deep speaker embedding extraction with channel-wise feature responses and additive supervision softmax loss function.,”  in Interspeech, 2019, pp. 2883–2887.
  * [46] Weidi Xie, Arsha Nagrani, Joon Son Chung, and Andrew Zisserman,  “Utterance-level aggregation for speaker recognition in the wild,”  in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, pp. 5791–5795.
  * [47] Nanxin Chen, Jesús Villalba, and Najim Dehak,  “Tied mixture of factor analyzers layer to combine frame level representations in neural speaker embeddings.,”  in INTERSPEECH, 2019, pp. 2948–2952.
  * [48] Tanvina B Patel and Hemant A Patil,  “Combining evidences from mel cepstral, cochlear filter cepstral and instantaneous frequency features for detection of natural vs. spoofed speech,”  in Sixteenth annual conference of the international speech communication association, 2015.
  * [49] Md Sahidullah, Tomi Kinnunen, and Cemal Hanilçi,  “A comparison of features for synthetic speech detection,”  ISCA (the International Speech Communication Association), 2015\.
  * [50] Massimiliano Todisco, Héctor Delgado, and Nicholas WD Evans,  “A new feature for automatic speaker verification anti-spoofing: Constant q cepstral coefficients.,”  in Odyssey, 2016, vol. 2016, pp. 283–290.
  * [51] Chunlei Zhang, Chengzhu Yu, and John HL Hansen,  “An investigation of deep-learning frameworks for speaker verification antispoofing,”  IEEE Journal of Selected Topics in Signal Processing, vol. 11, no. 4, pp. 684–694, 2017.
  * [52] Hemlata Tak, Jee-weon Jung, Jose Patino, Massimiliano Todisco, and Nicholas Evans,  “Graph attention networks for anti-spoofing,”  arXiv preprint arXiv:2104.03654, 2021.
  * [53] Xin Wang, Junichi Yamagishi, Massimiliano Todisco, Héctor Delgado, Andreas Nautsch, Nicholas Evans, Md Sahidullah, Ville Vestman, Tomi Kinnunen, Kong Aik Lee, et al.,  “Asvspoof 2019: A large-scale public database of synthesized, converted and replayed speech,”  Computer Speech & Language, vol. 64, pp. 101114, 2020.
  * [54] Hye-jin Shim, Jee-weon Jung, Ju-ho Kim, and Ha-jin Yu,  “Integrated replay spoofing-aware text-independent speaker verification,”  Applied Sciences, vol. 10, no. 18, pp. 6292, 2020.
  * [55] Junichi Yamagishi, Christophe Veaux, and Kirsten MacDonald,  “Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit (version 0.92),” 2019.
  * [56] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,  “Deep residual learning for image recognition,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  * [57] Hao Wang et al.,  “Cosface: Large margin cosine loss for deep face recognition,”  in IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 5265–5274.


11institutetext: The Hong Kong University of Science and Technology, Hong
Kong, China
11email<EMAIL_ADDRESS>Equal contribution;
🖂 corresponding author.

# Few Shot Medical Image Segmentation with Cross Attention Transformer

Yi Lin† Yufan Chen† Kwang-Ting Cheng  and Hao Chen🖂

###### Abstract

Medical image segmentation has made significant progress in recent years. Deep
learning-based methods are recognized as data-hungry techniques, requiring
large amounts of data with manual annotations. However, manual annotation is
expensive in the field of medical image analysis, which requires domain-
specific expertise. To address this challenge, few-shot learning has the
potential to learn new classes from only a few examples. In this work, we
propose a novel framework for few-shot medical image segmentation, termed CAT-
Net, based on cross masked attention Transformer. Our proposed network mines
the correlations between the support image and query image, limiting them to
focus only on useful foreground information and boosting the representation
capacity of both the support prototype and query features. We further design
an iterative refinement framework that refines the query image segmentation
iteratively and promotes the support feature in turn. We validated the
proposed method on three public datasets: Abd-CT, Abd-MRI, and Card-MRI.
Experimental results demonstrate the superior performance of our method
compared to state-of-the-art methods and the effectiveness of each component.
Code: https://github.com/hust-linyi/CAT-Net.

###### Keywords:

Few Shot Cross Attention Iterative Refinement.

##  1 Introduction

Automatic segmentation of medical images is a fundamental step for a variety
of medical image analysis tasks, such as diagnosis, treatment planning, and
disease monitoring [2, 1]. The emergence of deep learning (DL) has enabled the
development of many medical image segmentation methods, which have achieved
remarkable success [4, 12, 32, 3, 10]. Most of the existing methods follow a
fully-supervised learning paradigm, which requires a considerable amount of
labeled data for training. However, the manual annotation of medical images is
time-consuming and labor-intensive, limiting the application of DL in medical
image segmentation. Specifically for the 3D volumetric medical images (e.g.,
CT, MRI), the manual annotation is even more challenging which requires the
annotators to go through hundreds of 2D slices for each 3D scan.

To address the challenge of manual annotation, various label-efficient
techniques have been explored, such as self-supervised learning [15], semi-
supervised learning [30, 31], and weakly-supervised learning [11]. Despite
leveraging information from unlabeled or weakly-labeled data, these techniques
still require a substantial amount of training data [21, 16], which may not be
practical for novel classes with limited examples in the medical domain. This
limitation encourages the few-shot learning paradigm [22, 24, 6, 28] to be
applied to medical image segmentation. Specifically, the few-shot learning
paradigm aims to learn a model from a small number of labeled data (denoted as
support) and then apply it to a new task (denoted as query) with only a few
labeled data without any retraining. Considering the hundreds of organs and
countless diseases in the human body, FSL brings great potential to the
various medical image segmentation tasks where a new task can be easily
investigated in a data-efficient manner.

Most few-shot segmentation methods follow the learning-to-learn paradigm,
which aims to learn a meta-learner to predict the segmentation of query images
based on the knowledge of support images and their respective segmentation
labels. The success of this paradigm depends on how effectively the knowledge
can be transferred from the support prototype to the query images. Existing
few-shot segmentation methods mainly focus on the following two aspects: (1)
how to learn the meta-learner [17, 26, 14]; and (2) how to better transfer the
knowledge from the support images to the query images [23, 27, 18, 13, 5, 25].
Despite prototype-based methods having shown success, they typically ignore
the interaction between support and query features during training. In this
paper, as shown in Fig. 1(a), we propose CAT-Net, a Cross Attention
Transformer network for few-shot medical image segmentation, which aims to
fully capture intrinsic classes details while eliminating useless pixel
information and learn an interdependence between the support and query
features. Different from the existing FSS methods that only focus on the
single direction of knowledge transfer (i.e., from the support features to the
query features), the proposed CAT-Net can boost the mutual interactions
between the support and query features, benefiting the segmentation
performance of both the support and query images. Additionally, we propose an
iterative training framework that feed the prior query segmentation into the
attention transformer to effectively enhance and refine the features as well
as the segmentation. Three publicly available datasets are adopted to evaluate
our CAT-Net, i.e., Abd-CT [9], Abd-MRI [8], and Card-CT [33]. Extensive
experiments validate the effectiveness of each component in our CAT-Net, and
demonstrate its state-of-the-art performance.

##  2 Method

###  2.1 Problem Definition

Few-shot segmentation (FSS) aims to segment novel classes by just a few
samples with densely-annotated samples. In FSS, the dataset is divided into
the training set $\mathbb{D}_{\text{train}}$, containing the base classes
$\mathbb{C}_{\text{train}}$, and the test set $\mathbb{D}_{\text{test}}$,
containing the novel classes $\mathbb{C}_{\text{test}}$, where
$\mathbb{C}_{\text{train}}\cap\mathbb{C}_{\text{test}}=\emptyset$. To obtain
the segmentation model for FSS, the commonly used episode training approach is
employed [29]. Each trainig/testing episode ($S_{i},Q_{i}$) instantiates a
$N$-way $K$-shot segmentation learning task. Specifically, the support set
$S_{i}$ contains $K$ samples of $N$ classes, while the query set $Q_{i}$
contains one sample from the same class. The FSS model is trained with
episodes to predict the novel class for the query image, guided by the support
set. During inference, the model is evaluated directly on
$\mathbb{D}_{\text{test}}$ without any re-training. In this paper, we follow
the established practice in medical FSS [7, 15, 20] that consider the 1-way
1-shot task.

Figure 1: (a) Overview of the CAT-NET; (b) The architecture of CMAT module.

###  2.2 Network Overview

The Overview of our CAT-Net is illustrated in Fig. 1(a). It consists of three
main components: 1) a mask incorporated feature extraction (MIFE) sub-net that
extracts initial query and support features as well as query mask; 2) a cross
masked attention Transformer (CMAT) module in which the query and support
features boost each other and thus refined the query prediction; and 3) an
iterative refinement framework that sequentially applies the CMAT modules to
continually promote the segmentation performance. The whole framework can be
trained in an end-to-end fashion.

###  2.3 Mask Incorporated Feature Extraction

The Mask Incorporate Feature Extraction (MIFE) sub-net takes query and support
images as input and generates their respective features, integrated with the
support mask. A simple classifier is then used to predict the segmentation for
the query image. Specifically, we first employ a feature extractor network
(i.e., ResNet-50) to map the query and support image pair $I^{q}$ and $I^{s}$
into the feature space, producing multi-level feature maps $F^{q}$ and $F^{s}$
for query and support image, respectively. Next, the support mask is pooled
with $F^{s}$ and then expanded and concatenated with both $F^{q}$ and $F^{s}$.
Additionally, the segmentation mask of query image in MIFE is further
concatenated with the query feature to strengthen the correlation between
query and support features via a pixel-wise similarly map. Finally, the query
feature is processed by a simple classifier to get the query mask. Further
details of the MIFE architecture can be found in the supplementary material.

###  2.4 Cross Masked Attention Transformer

As shown in Fig. 1(b), the cross masked attention Transformer (CMAT) module
comprises three main components: 1) a self-attention module for extracting
global information from query and support features; 2) a cross masked
attention module for transferring foreground information between query and
support features while eliminating redundant background information, and 3) a
prototypical segmentation module for generating the final prediction of the
query image.

Self-Attention Module. To capture the global context information of every
pixel in the query feature $F_{0}^{q}$ and support features $F_{0}^{s}$, the
initial features are first flattened into 1D sequences and fed into two
identical self-attention modules. Each self-attention module consists of a
multi-head attention (MHA) layer and a multi-perceptron (MLP) layer. Given an
input sequence $S$, the MHA layer first projects the sequence into three
sequences $K$, $Q$, and $V$ with different weights. The attention matrix $A$
is then calculated as:

$A(Q,K)=\dfrac{QK^{T}}{\sqrt{d}}$ (1)

where $d$ is the dimension of the input sequence. The attention matrix is then
normalized by a softmax function and multiplied by the value sequence $V$ to
get the output sequence $O$:

$O=\text{softmax}(A)V$ (2)

The MLP layer is a simple $1\times 1$ convolution layer that maps the output
sequence $O$ to the same dimension as the input sequence $S$. Finally, the
output sequence $O$ is added to the input sequence $S$ and normalized using
layer normalization (LN) to obtain the final output sequence $X$. The output
feature sequence of the self-attention alignment encoder is represented by
$X^{q}\in\mathbb{R}^{HW\times D}$ and $X^{s}\in\mathbb{R}^{HW\times D}$ for
query and support features, respectively.

Cross Masked Attention Module. We utilize cross masked attention to
incorporate query features and support features with respect to their
foreground information by constraining the attention region in attention
matrix with support and query masks. Specifically, given the query feature
$X^{q}$ and support features $X^{s}$ from the aforementioned self-attention
module, we first project the input sequence into three sequences $K$, $Q$, and
$V$ using different weights, resulting in $K^{q}$, $Q^{q}$, $V^{q}$, and
$K^{s}$, $Q^{s}$, $V^{s}$, respectively. Taking the support features as an
example, the cross attention matrix is calculated by:

$\text{A}(K^{q},Q^{s})=\dfrac{(K^{q})^{T}Q^{s}}{\sqrt{d}}$ (3)

We expand and flatten the binary query mask $M^{q}$ to limit the foreground
region in attention map. The masked cross attention (MCA) map is computed as:

$\text{MCA}(K^{q},Q^{s},V^{q},{M}^{s})=M^{s}\cdot
V^{q}(\text{softmax}(A(K^{q},Q^{s})))$ (4)

Similar to self-attention, the support feature is processed by MLP and LN
layer to get the final enhanced query features $F^{s}_{1}$. Similarly, the
enhanced query feature $F^{q}_{1}$ is obtained with foreground information
from the query feature.

Prototypical Segmentation Module. Once the enhanced query and support features
are obtained, the prototypical segmentation is used to obtain the final
prediction. First, a prototype of class $c$ is built by masked average pooling
of the support feature $F^{s}_{1}$ as follows:

$p_{c}=\dfrac{1}{K}\sum^{K}_{k=1}\dfrac{\sum_{k,x,y}F^{s}_{i,(k,x,y)}m^{s}_{(k,x,y,c)}}{\sum_{x,y}m^{s}_{(k,x,y,c)}}$
(5)

where $K$ is the number of support images, and $m^{s}_{(k,x,y,c)}$ is a binary
mask that indicates whether pixel at the location $(x,y)$ in support feature
$k$ belongs to class $c$. Next, we use the non-parametirc metric learning
method to perform segmentation. The prototype network calculates the distance
between the query feature vector and the prototype $P={\\{P_{c}|c\in C\\}}$.
Softmax function is applied to produce probabilistic outputs for all classes,
generating the query segmentation:

$\hat{M}^{q}_{i,(x,y)}=\text{softmax}\big{(}\alpha\text{cos}(F^{q}_{i,(x,y)},p_{c})\cdot{\text{softmax}(\alpha\text{cos}(F^{q}_{i,(x,y)},p_{c}))}\big{)}$
(6)

where $\text{cos}(\cdot)$ denotes cosine distance, $\alpha$ is a scaling
factor that helps gradients to back-propagate in training. In our work,
$\alpha$ is set to 20, same as in [29].

Additionally, we design a double threshold strategy to obtain query
segmentation. Specifically, we set the first threshold $\tau$ to 0.5 to obtain
the binary query mask ${M}^{q}$, which is used to calculate the Dice loss and
update the model. Then, the second threshold $\hat{\tau}$ is set to 0.4 to
obtain the dilated query mask $\hat{M}^{q}$, which is used to generate the
enhanced query feature $F^{q}_{2}$ in the next iteration. The second threshold
$\hat{\tau}$ is set lower than the first threshold $\tau$ to prevent some
foreground pixels from being mistakenly discarded. The query segmentation mask
${M}^{q}$ and dilated mask $\hat{M}^{q}$ are represented by:

${M}^{q}_{i}=\left\\{\begin{aligned} 1,\quad&M^{q}_{i,(x,y)}>\tau\\\
0,\quad&M^{q}_{i,(x,y)}<\tau\\\
\end{aligned}\right.\qquad\hat{M}^{q}_{i}=\left\\{\begin{aligned}
1,\quad&M^{q}_{i,(x,y)}>\hat{\tau}\\\ 0,\quad&M^{q}_{i,(x,y)}<\hat{\tau}\\\
\end{aligned}\right.$ (7)

###  2.5 Iterative Refinement framework

As explained above, the CMAT module is designed to refine the query and
support features, as well as the query segmentation mask. Thus, it’s natural
to iteratively apply this sub-net to get the enhanced features and refine the
mask, resulting in a boosted segmentation result. The result after the $i$-th
iteration is represented by:

$(F^{s}_{i},F^{q}_{i},M^{q}_{i},\hat{M}^{q}_{i})=\text{CMAT}(F_{i-1}^{s},F_{i-1}^{q},\hat{M}_{i-1}^{q},M^{s})$
(8)

The subdivision of each step can be specifically expressed as:

$(F^{s}_{i},F^{q}_{i})=\text{CMA}(F_{i-1}^{s},F_{i-1}^{q},\hat{M}_{i-1}^{q},M^{s})$
(9)
$(M^{q}_{i},\hat{M}_{i}^{q})=\text{Proto}(F_{i}^{s},F_{i}^{q},M^{s},\tau,\hat{\tau})$
(10)

where $\text{CMA}(\cdot)$ indicates the self-attention and cross masked
attention module, and $\text{Proto}(\cdot)$ represents the prototypical
segmentation module.

##  3 Experiment

###  3.1 Dataset and Evaluation Metrics

We evaluate the proposed method on three public datasets, i.e., Abd-CT [9],
Abd-MRI [8], and Card-MRI [33]. Abd-CT contains 30 abdominal CT scans with
annotations of left and right kidney (LK and RK), spleen (Spl), liver (Liv).
Abd-MRI contains 20 abdominal MRI scans with annotations of the same organs as
Abd-CT. Card-MRI includes 35 cardiac MRI scans with annotations of left
ventricular blood pool (LV-B), left ventricular myocardium (LV-M), and right
ventricle (RV). We use the Dice score as the evaluation metric following [15,
20].

To ensure a fair comparison, all the experiments are conducted under the 1-way
1-shot scenario using 5-fold cross-validation. We follow [15] to remove all
slices containing test classes during training to ensure that the test classes
are all unseen during validation. In each fold, we follow [15, 7, 20] that
takes the last patient as the support image and the remaining patients as the
query (setting i@). We further propose a new validation setting (setting ii@)
that takes every image in each fold as a support image alternately and the
other images as the query. The averaged result of each fold is reported. It
could evaluate the generalization ability of the model by reducing the affect
of support image selection.

###  3.2 Implementation Details

The proposed method is implemented using PyTorch. Each 3D scan is sliced into
2D slices and reshaped into 256$\times$256 pixels. Common 3D image pre-
processing techniques, such as intensity normalization and resampling, are
applied to the training data. We apply episode training with 20$k$ iterations.
SGD optimizer is adopted with a learning rate of 0.001 and a batch size of 1.
Each episode training takes approximately 4 hours using a single NVIDIA RTX
3090 GPU.

Table 1: Comparison with state-of-the-art methods in Dice coefficient (%) on
Abd-CT and Abd-MRI, and Card-MRI datasets under setting i@ &<EMAIL_ADDRESS>
|  |  Abd-CT [8] |  Abd-MRI [9] |  Card-MRI [33]
---|---|---|---|---
| Methods | LK | RK | Spl. | Liv. | Avg. | LK | RK | Spl. | Liv. | Avg. | LV-B | LV-M | RV | Avg.
Setting i@ |  SE-Net [19] | 32.83 | 14.84 | 0.23 | 0.27 | 11.91 | 62.11 | 61.32 | 51.80 | 27.43 | 50.66 | 58.04 | 25.18 | 12.86 | 32.03
PA-Net [29] | 37.58 | 34.69 | 43.73 | 61.71 | 44.42 | 47.71 | 47.95 | 58.73 | 64.99 | 54.85 | 70.43 | 46.79 | 69.52 | 62.25
ALP-Net [15] | 63.34 | 54.82 | 60.25 | 73.65 | 63.02 | 73.63 | 78.39 | 67.02 | 73.05 | 73.02 | 61.89 | 87.54 | 76.71 | 75.38
AD-Net [7] | 63.84 | 56.98 | 61.84 | 73.95 | 64.15 | 71.89 | 76.02 | 65.84 | 76.03 | 72.70 | 65.47 | 88.36 | 78.35 | 77.39
Q-Net [20] | 63.26 | 58.37 | 63.36 | 74.36 | 64.83 | 74.05 | 77.52 | 67.43 | 78.71 | 74.43 | 66.87 | 89.63 | 79.25 | 78.58
Ours | 63.36 | 60.05 | 67.65 | 75.31 | 66.59 | 74.01 | 78.90 | 68.83 | 78.98 | 75.18 | 66.85 | 90.54 | 79.71 | 79.03
Setting ii@ |  ALP-Net [15] | 65.99 | 59.49 | 65.02 | 73.50 | 66.05 | 70.17 | 77.05 | 67.71 | 72.45 | 71.85 | 61.61 | 87.13 | 77.35 | 75.36
AD-Net [7] | 67.35 | 59.88 | 64.35 | 76.78 | 67.09 | 72.26 | 76.57 | 67.89 | 73.96 | 72.67 | 65.08 | 86.26 | 76.50 | 75.95
Q-Net [20] | 66.25 | 62.36 | 67.35 | 77.33 | 68.32 | 73.96 | 81.07 | 65.39 | 72.36 | 73.20 | 66.35 | 88.40 | 79.37 | 78.04
Ours | 68.82 | 64.56 | 66.02 | 80.51 | 70.88 | 75.31 | 83.23 | 67.31 | 75.02 | 75.22 | 67.21 | 90.54 | 80.34 | 79.36

###  3.3 Comparison with State-of-the-Art Methods

We compare the proposed CAT-Net with state-of-the-art (SOTA) methods,
including SE-Net [19], PANet [29], ALP-Net [15], and AD-Net [7], and Q-Net
[20]. PANet [29] are the typical prototypical FSS method in the natural image
domain, SE-Net [19], ALP-Net [15], AD-Net [7], and Q-Net [20] are the most
representative work in medical FSS task. Experiment results presented in Table
1 demonstrate that the proposed method outperforms SOTAs on all three datasets
under both setting i@ and setting<EMAIL_ADDRESS>Under setting i@, the proposed CAT-Net
achieves 66.59% Dice on Abd-CT, 75.18% Dice on Abd-MRI, and 79.03% Dice on
Card-MRI in Dice, outperforming SOTAs by 1.76%, 0.75%, and 0.45%,
respectively. Under setting ii@, CAT-Net achieves 70.88% Dice on Abd-CT,
75.22% Dice on Abd-MRI, and 79.36% Dice on Card-MRI, outperforming SOTAs by
2.56%, 2.02% and 1.32%, respectively. The consistent superiority of our method
to SOTAs on three datasets and under two evaluation settings indicates the
effectiveness and generalization ability of the proposed CAT-Net. In addition,
the qualitative results in Fig. 2 demonstrate that the proposed method is able
to generate more accurate and detailed segmentation results compared to SOTAs.

Figure 2: Qualitative results of our method on Abd-CT and Abd-MRI.

###  3.4 Ablation Study

We conduct an ablation study to investigate the effectiveness of each
component in CAT-Net. All ablation studies are conducted on Abd-MRI under
setting<EMAIL_ADDRESS>
Effectiveness of CMAT Block: To demonstrate the importance of our proposed
CAT-Net in narrowing the information gap between the query and supporting
images and obtaining enhanced features, we conducted an ablation study.
Specifically, we compared the results of learning foreground information only
from the support (S$\to$Q) or query image (Q$\to$S) and obtaining a single
enhanced feature instead of two (S$\leftrightarrow$Q). It can be observed that
using the enhanced query feature (S$\to$Q) achieves 66.72% in Dice,
outperforming only using the enhanced support feature (Q$\to$S) by 0.74%. With
our CMAT block, the mutual boosted support and query feature
(S$\leftrightarrow$Q) could improve the Dice by 1.90%. Moreover, the iteration
refinement framework consistently promotes the above three variations by
0.96%, 0.56%, and 2.26% in Dice, respectively.

S$\to$Q | Q$\to$S | S$\leftrightarrow$Q | Iter | Dice | Improve
---|---|---|---|---|---
✓ |  |  |  | 66.72 | -
| ✓ |  |  | 65.98 | -0.74
|  | ✓ |  | 68.62 | +1.90
✓ |  |  | ✓ | 67.68 | +0.96
| ✓ |  | ✓ | 66.54 | +0.56
|  | ✓ | ✓ | 70.88 | +2.26
Table 2: Effectiveness of each component. S$\to$Q and Q$\to$S denote one
branch CAT-Net to enhance support or query feature, respectively.
S$\leftrightarrow$Q indicates applying cross attention to both S and Q.

Figure 3: The influence of different numbers of iteration CMAT modules.

Influence of Iterative Mask Refinement Block: To determine the optimal number
of iterative refinement CMAT block, we experiment with different numbers of
blocks. In Fig. 3, we observe that increasing the number of blocks results in
improved performance, with a maximum improvement of 2.26% in Dice when using 5
blocks. Considering the performance gain between using 4 and 5 CMAT blocks was
insignificant, we hence opt to use four CMAT blocks in our final model to
strike a balance between efficiency and performance.

##  4 Conclusion

In this paper, we propose CAT-Net, Cross Attention Transformer network for
few-shot medical image segmentation. Our CAT-Net enables mutual interaction
between the query and support features by the cross masked attention module,
enhancing the representation abilities for both of them. Additionally, the
proposed CMAT module can be iteratively applied to continually boost the
segmentation performance. Experimental results demonstrated the effectiveness
of each module and the superior performance of our model to the SOTA methods.
In the future, we plan to extend our CAT-Net from 2D to 3D networks, explore
the application of our model to other medical image segmentation tasks, as
well as the extension of our model to other clinical applications, such as
rare diseases and malformed organs, where data and annotations are scarce and
costly.

## References

  * [1] Che, H., Chen, S., Chen, H.: Image quality-aware diagnosis via meta-knowledge co-embedding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 19819–19829 (2023)
  * [2] Che, H., Cheng, Y., Jin, H., Chen, H.: Towards generalizable diabetic retinopathy grading in unseen domains. arXiv preprint arXiv:2307.04378 (2023)
  * [3] Che, H., Jin, H., Chen, H.: Learning robust representation for joint grading of ophthalmic diseases via adaptive curriculum and feature disentanglement. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 523–533. Springer (2022)
  * [4] Chen, J., Lu, Y., Yu, Q., Luo, X., Adeli, E., Wang, Y., Lu, L., Yuille, A.L., Zhou, Y.: TransUnet: Transformers make strong encoders for medical image segmentation. arXiv preprint arXiv:2102.04306 (2021)
  * [5] Fan, Q., Pei, W., Tai, Y.W., Tang, C.K.: Self-support few-shot semantic segmentation. In: European Conference on Computer Vision (ECCV). pp. 701–719. Springer (2022)
  * [6] Garcia, V., Bruna, J.: Few-shot learning with graph neural networks. In: International Conference on Learning Representations (ICLR) (2018)
  * [7] Hansen, S., Gautam, S., Jenssen, R., Kampffmeyer, M.: Anomaly detection-inspired few-shot medical image segmentation through self-supervision with supervoxels. Medical Image Analysis 78, 102385 (2022)
  * [8] Kavur, A.E., Gezer, N.S., Barış, M., Aslan, S., Conze, P.H., Groza, V., Pham, D.D., Chatterjee, S., Ernst, P., Özkan, S., et al.: Chaos challenge-combined (CT-MR) healthy abdominal organ segmentation. Medical Image Analysis 69, 101950 (2021)
  * [9] Landman, B., Xu, Z., Igelsias, J., Styner, M., Langerak, T., Klein, A.: Miccai multi-atlas labeling beyond the cranial vault–workshop and challenge. In: Proc. MICCAI Multi-Atlas Labeling Beyond Cranial Vault—Workshop Challenge. vol. 5, p. 12 (2015)
  * [10] Lin, Y., Liu, L., Ma, K., Zheng, Y.: Seg4reg+: Consistency learning between spine segmentation and cobb angle regression. In: Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part V 24. pp. 490–499. Springer International Publishing (2021)
  * [11] Lin, Y., Qu, Z., Chen, H., Gao, Z., Li, Y., Xia, L., Ma, K., Zheng, Y., Cheng, K.T.: Label propagation for annotation-efficient nuclei segmentation from pathology images. arXiv preprint arXiv:2202.08195 (2022)
  * [12] Lin, Y., Zhang, D., Fang, X., Chen, Y., Cheng, K.T., Chen, H.: Rethinking boundary detection in deep learning models for medical image segmentation. In: International Conference on Information Processing in Medical Imaging. pp. 730–742. Springer (2023)
  * [13] Liu, Y., Liu, N., Yao, X., Han, J.: Intermediate prototype mining transformer for few-shot semantic segmentation. In: Advances in Neural Information Processing Systems (NeurIPS) (2022)
  * [14] Luo, X., Tian, Z., Zhang, T., Yu, B., Tang, Y.Y., Jia, J.: Pfenet++: Boosting few-shot semantic segmentation with the noise-filtered context-aware prior mask. arXiv preprint arXiv:2109.13788 (2021)
  * [15] Ouyang, C., Biffi, C., Chen, C., Kart, T., Qiu, H., Rueckert, D.: Self-supervised learning for few-shot medical image segmentation. IEEE Transactions on Medical Imaging 41(7), 1837–1848 (2022)
  * [16] Pan, W., Yan, J., Chen, H., Yang, J., Xu, Z., Li, X., Yao, J.: Human-machine interactive tissue prototype learning for label-efficient histopathology image segmentation. In: International Conference on Information Processing in Medical Imaging. pp. 679–691. Springer (2023)
  * [17] Pandey, P., Vardhan, A., Chasmai, M., Sur, T., Lall, B.: Adversarially robust prototypical few-shot segmentation with neural-ODEs. In: International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI). pp. 77–87. Springer (2022)
  * [18] Peng, B., Tian, Z., Wu, X., Wang, C., Liu, S., Su, J., Jia, J.: Hierarchical dense correlation distillation for few-shot segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 23641–23651 (2023)
  * [19] Roy, A.G., Siddiqui, S., Pölsterl, S., Navab, N., Wachinger, C.: ‘squeeze & excite’ guided few-shot segmentation of volumetric images. Medical Image Analysis 59, 101587 (2020)
  * [20] Shen, Q., Li, Y., Jin, J., Liu, B.: Q-Net: Query-informed few-shot medical image segmentation. arXiv preprint arXiv:2208.11451 (2022)
  * [21] Siam, M., Oreshkin, B.N., Jagersand, M.: AMP: Adaptive masked proxies for few-shot segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 5249–5258 (2019)
  * [22] Snell, J., Swersky, K., Zemel, R.: Prototypical networks for few-shot learning. Advances in Neural Information Processing Systems (NeurIPS) 30 (2017)
  * [23] Sun, L., Li, C., Ding, X., Huang, Y., Chen, Z., Wang, G., Yu, Y., Paisley, J.: Few-shot medical image segmentation using a global correlation network with discriminative embedding. Computers in Biology and Medicine 140, 105067 (2022)
  * [24] Sung, F., Yang, Y., Zhang, L., Xiang, T., Torr, P.H., Hospedales, T.M.: Learning to compare: Relation network for few-shot learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 1199–1208 (2018)
  * [25] Tang, H., Liu, X., Sun, S., Yan, X., Xie, X.: Recurrent mask refinement for few-shot medical image segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 3918–3928 (2021)
  * [26] Tian, Z., Lai, X., Jiang, L., Liu, S., Shu, M., Zhao, H., Jia, J.: Generalized few-shot semantic segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022)
  * [27] Tian, Z., Zhao, H., Shu, M., Yang, Z., Li, R., Jia, J.: Prior guided feature enrichment network for few-shot segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence (2020)
  * [28] Vinyals, O., Blundell, C., Lillicrap, T., Wierstra, D., et al.: Matching networks for one shot learning. Advances in Neural Information Processing Systems (NeurIPS) 29 (2016)
  * [29] Wang, K., Liew, J.H., Zou, Y., Zhou, D., Feng, J.: PANet: Few-shot image semantic segmentation with prototype alignment. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 9197–9206 (2019)
  * [30] Xu, Z., Wang, Y., Lu, D., Yu, L., Yan, J., Luo, J., Ma, K., Zheng, Y., Tong, R.K.y.: All-around real label supervision: Cyclic prototype consistency learning for semi-supervised medical image segmentation. IEEE Journal of Biomedical and Health Informatics 26(7), 3174–3184 (2022)
  * [31] Yang, X., Lin, Y., Wang, Z., Li, X., Cheng, K.T.: Bi-modality medical image synthesis using semi-supervised sequential generative adversarial networks. IEEE journal of biomedical and health informatics 24(3), 855–865 (2019)
  * [32] Zhang, D., Lin, Y., Chen, H., Tian, Z., Yang, X., Tang, J., Cheng, K.T.: Deep learning for medical image segmentation: tricks, challenges and future directions. arXiv preprint arXiv:2209.10307 (2022)
  * [33] Zhuang, X.: Multivariate mixture model for myocardial segmentation combining multi-source images. IEEE Transactions on Pattern Analysis and Machine Intelligence 41(12), 2933–2946 (2018)