# Cooperative Multi-Agent Transfer Learning with Level-Adaptive Credit
Assignment

Tianze Zhou1111Work done during an internship at Noah’s Ark Lab, Huawei
Technologoies. Fubiao Zhang1 Kun Shao2 Kai Li3 Wenhan Huang3 Jun Luo2 Weixun
Wang4 Yaodong Yang2 Hangyu Mao2 Bin Wang2 Dong Li2 Wulong Liu2 Jianye Hao 2

1 Beijing Institute of Technology
{tianzezhou<EMAIL_ADDRESS>
2 Noah’s Ark Lab, Huawei Technologoies
{shaokun2<EMAIL_ADDRESS>
3 Shanghai Jiao Tong University
4 Tianjin University

###### Abstract

Extending transfer learning to cooperative multi-agent reinforcement learning
(MARL) has recently received much attention. In contrast to the single-agent
setting, the coordination indispensable in cooperative MARL constrains each
agent’s policy. However, existing transfer methods focus exclusively on agent
policy and ignores coordination knowledge. We propose a new architecture that
realizes robust coordination knowledge transfer through appropriate
decomposition of the overall coordination into several coordination patterns.
We use a novel mixing network named level-adaptive QTransformer (LA-
QTransformer) to realize agent coordination that considers credit assignment,
with appropriate coordination patterns for different agents realized by a
novel level-adaptive Transformer (LA-Transformer) dedicated to the transfer of
coordination knowledge. In addition, we use a novel agent network named
Population Invariant agent with Transformer (PIT) to realize the coordination
transfer in more varieties of scenarios. Extensive experiments in StarCraft II
micro-management show that LA-QTransformer together with PIT achieves superior
performance compared with state-of-the-art baselines.

##  1 Introduction

Coordination in multi-agent reinforcement learning (MARL) is a popular topic
in fields ranging from robotics [9, 8], computer games [2, 33] to
recommendation systems [35]. Centralized training with decentralized execution
(CTDE) is a popular regime in cooperative MARL to realize efficient agent
coordination. Existing CTDE research covers important topics such as division
of agents [27], diversification [32] and exploration [19]. Recent works [29,
11, 1, 17, 16] have also started to make progress in transfer learning in
cooperative MARL. For example, Liu et al. [16] use policy distillation [22] to
achieve fixed agent transfer learning. However, the agent population varies in
different tasks in most cases. To solve this problem, DyAN [29] uses a graph
neural network to adapt to dynamic agent population. UPDeT [11] uses
Transformer[24] to realize a universal and transferable agent policy network
to achieve agent-level knowledge transfer. However, these methods all focus on
the transfer of individual agent policy and ignore the coordination knowledge.
Unlike single-agent tasks, cooperative multi-agent tasks require the
coordination of multiple agents. Ignoring coordination knowledge may lead to
biased transfer because the difference in coordination implies a difference in
agent policy.

In cooperative MARL, while the joint policy differs from task to task, the
underlying coordination may be decomposed into several patterns that remain
valid across different tasks. Figure 1 illustrates this point in StarCraft II,
where agents tend to form three different coalition patterns [10] with
different coalition patterns accomplishing different sub-tasks. By leveraging
decomposition according to coordination patterns, we may achieve robust
coordination knowledge transfer. The example in Figure 1 also suggests that
the coordination patterns tend to involve a regular number of agents, such as
pairwise coordination patterns, triplet coordination, etc. This means
knowledge transfer on coordination policy is manageable in terms of scale.

Figure 1: Coordination patterns in StarCraft II.

Transformer [24] is a popular module to capture the relationship among
elements and is widely used in nature language process [14] and computer
vision [5]. In this paper, we use Transformer to capture the correlation
between agents and construct the coordination patterns. However, the
traditional Transformer module can only construct the pairwise coordination
pattern. While stacking multiple Transformer modules could allow us to go
beyond the pairwise pattern, this approach is not suitable for large-scale
multi-agent scenarios due to the enormous computational cost. Instead, we
propose the level-adaptive Transformer (LA-Transformer), which can adaptively
capture agent-specific coordination levels and realize coordination patterns
involving a variable number of agents. We realize the LA-Transformer using
both hard attention and a hybrid-based method. The hard LA-Transformer focuses
on the most appropriate coordination level, while the hybrid LA-Transformer
merges the features from multiple levels.

Proper credit assignment is essential for coordination among multiple agents
in both policy-based [6, 30] and value-based [23] cooperative MARL. The credit
each agent receives must reflect their contribution towards the coordinated
performance. In CTDE for MARL, a trainable mixing network is often used to
implement the required credit assignment. We follow this practice and
introduce LA-Transformer into the design of a novel mixing network named
level-adaptive QTransformer (LA-QTransformer). Compared with other mixing
networks such as QMIX [21] and Qatten [31], LA-QTransformer is a more
expressive mechanism for coordination policy learning and coordination
knowledge transfer.

However, redesigning the mixing network alone cannot achieve adequate
coordination transfer because the dynamic agent population size limits the
agent policy to be reused. Previous methods implicitly assume that the joint
action space has a fixed dimension or only small and medium-sized scenarios
need to be handled. Such assumptions make it hard to apply them in real
scenarios. To handle this problem, we design a novel agent structure called
Population Invariant agent with Transformer (PIT) to realize generalized
coordination knowledge transfer in scenarios of different scales with variable
agent numbers.

Evaluation of our new method with the SMAC benchmark [25] shows that it
outperforms current SOTA methods in transfer scenarios as well as non-transfer
scenarios. In addition, a curricular training experiment with an increasing
number of agents validated the robustness of our method. Finally, we
demonstrate the interpretability of the proposed modules and confirm the
contribution of LA-Transformer with ablation studies.

##  2 Backgrounds

###  2.1 Cooperative multi-agent Q-learning

The fully cooperative MARL task can be formulated as a Dec-POMDP [20]. A tuple
can represent Dec-POMDP $\left<I,S,U,Z,P,R,O,n,\gamma\right>$, where $s\in S$
represents the global state of the environment. At any time, each agent $i\in
I\equiv\\{1,...,n\\}$ interacts with the environment by generating
corresponding action $u_{i}\in{U}$ through it’s local observation vector
$z_{i}\in Z$ according to the observation function $O(s,i)$. The overall
objective is to maximize the cumulative reward $R$ from environmental
feedback. The environment receives the joint action $\mathbf{a}$, and
transfers to the next state $\mathbf{s^{\prime}}$ according to the state
transition function $P\left(s^{\prime}\mid s,\mathbf{a}\right)$. $n$ defines
the number of agents, and $\gamma$ represents the discount factor.

Centralized training with decentralized execution [15] (CTDE) is a popular
regime to address the Dec-POMDP problem. In the CTDE framework, the mixing
network is introduced to merge all individual Q values into $Q_{tot}$:

$\displaystyle
Q_{tot}(\mathbf{\tau},\mathbf{u},s;\theta)=f([Q_{i}(\tau^{i},u^{i})]_{i}^{n},s;\theta).$
(1)

And then TD-learning is used to train the whole network

$\displaystyle\mathcal{L}(\theta)=\sum_{i=1}^{b}\left[\left(y_{i}^{tot}-Q_{tot}(\mathbf{\tau},\mathbf{u},s;\theta)\right)^{2}\right],$
(2)

where $b$ is the batch size of replay buffer, $y^{\text{tot
}}=r+\gamma\max_{\mathbf{u}^{\prime}}Q_{\text{tot
}}\left(\tau^{\prime},\mathbf{u}^{\prime},s^{\prime};\theta^{-}\right)$, and
$\theta^{-}$ is the parameter of target network.

###  2.2 Multi-agent transfer learning

The basic idea behind transfer learning is that the knowledge acquired from
previous tasks can be reused to accelerate learning drastically, and it makes
the learning of complex tasks feasible [4, 3]. Due to the complexity of MARL,
multi-agent transfer learning is not a straightforward extension of single-
agent transfer learning. In the multi-agent setting, the policy mapping
expands from a single agent to multiple agents, and the dimension of the
mapping is varying with specific tasks:

$\displaystyle J_{p}\rightarrow J_{c}:A_{1}\times\ldots\times A_{n}\rightarrow
A_{1}^{\prime}\times\ldots\times A_{m}^{\prime}$ (3)

where $J_{p}$ and $J_{c}$ represent the joint-policy in the previous task and
the current task, respectively, and $n,m$ shows the number of agents in these
tasks.

[4] divides multi-agent transfer learning (MATL) into two main types: the
intra-agent transfer and the inter-agent transfer. The intra-agent transfer
focuses on the relationship between the source tasks and the target tasks,
while the inter-agent transfer pays more attention to reusing knowledge
received from communication with other agents.

###  2.3 Transformer

The Transformer is an attention-based neural network structure widely used in
nature language process and computer vision. The traditional Transformer
module consists of two sub-structures, the attention module and the feed-
forward network. Soft attention and hard attention are two approaches to
realize the attention mechanism. Soft attention takes the softmax function to
calculate the input elements relationship.

$\displaystyle\text{ Attention
}(\mathbf{Q},\mathbf{K},\mathbf{V})=\operatorname{softmax}\left(\frac{\mathbf{Q}\mathbf{K}^{T}}{\sqrt{d_{k}}}\right)\mathbf{V},$
(4)

where $\mathbf{Q},\mathbf{K},\mathbf{V}$ represents the query, keys, values of
input elements respectively and $\sqrt{d_{k}}$ is the normalization
coefficient.

Due to element weights is calculated directly, soft attention is fully
differentiable. However, the softmax function weakens the ground truth
element’s weight, limiting the actual performance. Hard attention overcomes
the limitation of soft attention by selecting the sole element. However, this
selecting operation is non-differentiable. Gumbel softmax [12] is a popular
trick to approximate hard attention performance while keeping the neural
network back-propagation differentiable. To enhance the representation of the
embedding features, Transformer utilizes a feed-forward network. The feed-
forward network contains a 1-D convolutional layer and a layer-normalization
module.

Transformer is suitable to capture elements relationship in cooperative MARL.
Due to Transformer’s flexible I/O characteristic, it can handle dynamic
element inputs. Compared with RNN-based methods, Transformer does not care
about the order of elements and can process elements in parallel.

##  3 Methods

In this section, we design a novel value-based framework to realize
coordination knowledge transfer in cooperative MARL. Figure 2 describes the
whole structure of our methods. It contains a mixing network level-adaptive
QTransformer (LA-QTransformer) that utilizes the level-adaptive Transformer
(LA-Transformer) module to realize the coordination knowledge transfer and the
agent network, Population Invariant agent with Transformer (PIT), to achieve
coordination transfer in universal scenarios.

Figure 2: Left: The structure of PIT. Firstly, The observation separated
explicitly is flowed into the property group module to generate the different
class embedding features. The GRU core is utilized to merge all embedding
features. Finally, the adaptive action module utilizes the GRU embedding
feature and the property group module embedding features to generate dynamic
agent actions. Middle: The whole structure of our methods. Right: The
structure of LA-QTransformer. LA-QTransformer first separates the state
features into different class entities’ features and utilizes the encoder
layers to encode the features into the same dimension. Then the LA-Transformer
module is used to generate multi-level coordination patterns and
merges(selects) the appropriate coordination patterns. The multi-head
attention module then integrates the coordination patterns and generates agent
credit values.

###  3.1 Level-Adaptive QTransformer

####  3.1.1 Level-Adaptive Transformer

Figure 3: Two implementations of LA-Transformer.

The Transformer module can be applied to generate the pairwise coordination
patterns via capturing the relationship of input elements. However, only
considering the pairwise coordination patterns can not achieve general
coordination transfer. To generate coordination patterns on multiple levels, a
native method is stacking Transformer modules. However, this is unrealistic in
large-scale multi-agent scenarios due to its massive memory consumption.
Besides, stacking modules blurs different level relationships and makes it
impossible to determine which level works. In this section, we propose an
efficient module named level-adaptive Transformer (LA-Transformer) to generate
multi-level coordination patterns and capture the most suitable coordination
pattern. We implement LA-Transformer with two methods, including hard
attention and the hybrid method, as shown in Figure 3.

In the traditional Transformer, the pairwise coordination patterns are
generated via:

$\displaystyle\mathbf{Q,K,V}=\operatorname{MLP}_{Q,K,V}(\mathbb{s}),$ (5)
$\displaystyle
c=\operatorname{Softmax}\left(\frac{\mathbf{Q}\mathbf{K}^{T}}{\sqrt{d_{k}}}\right)\mathbf{V},$
(6)

where $c$ is the coordination pattern.
$\frac{\mathbf{Q}\mathbf{K}^{T}}{\sqrt{d_{k}}}$ can be approximated as the
coordination relationship. Compared with traditional stacking operations, we
fix $k$ and $v$ to realize linearly increasing coordination level and prevent
feature blurring, respectively. The specific coordination level is calculated
via:

$\displaystyle c_{i}$
$\displaystyle=\operatorname{Softmax}\left(\frac{\mathbf{c_{i-1}}\mathbf{K}^{T}}{\sqrt{d_{k}}}\right)\mathbf{V},$
(7)

where $c_{i-1}$ is the coordination pattern of previous level.

Furthermore, we analyze that the upper bound of the coordination pattern level
does not need to be huge. The relationship mapping of the adjacent levels
tends to be stationary as the level increases according to the Brouwer
Fixpoint Theorem [13]. With the coordination level increasing, there exist two
coordination patterns with adjacent level $c_{i},c_{j}$ that their difference
tends to zero:

$\displaystyle\left\|c_{i}-c_{j}\right\|_{2}\leq\epsilon,\ \exists i,j,\;i\geq
j\geq 0,$ (8)

Due to the difference among agents, we propose two methods to generate
different levels of coordination patterns for different agents.

LA-Transformer (hard). LA-Transformer (hard) utilizes the hard attention
mechanism to select the most appropriate coordination patterns. To achieve the
hard attention-based LA-Transformer, we first utilize the MLP function to
encode the initial embedding features to get the key of the hard attention,
i.e., $\mathbf{k_{e}}$, and then take the Gumbel softmax function to generate
the mask on different levels.

$\displaystyle\mathbf{k_{e}}$
$\displaystyle=\operatorname{MLP}_{K}(\mathbb{s}),$ (9)
$\displaystyle\mathbf{mask}$
$\displaystyle=\operatorname{gumbel\\_softmax}(k_{e}[c_{1},...c_{i}])$ (10)

Then, we take the mask to select the agent-specific coordination pattern.

$\displaystyle\mathbf{c_{hard}}=\mathbf{mask}\times[c_{1},...,c_{i}]$ (11)

The advantage of utilizing the mask to select is that it can explicitly
provide the value of the level.

LA-Transformer (hybrid). Although the LA-Transformer (hard) can explicitly
select a coordination pattern, it inevitably ignores some essential details.
So we propose the hybrid level-adaptive Transformer, which generates the
level-adaptive coordination pattern via adaptively fusing coordination
patterns from all levels. Specifically, we take the MLP function to do the
fuse operation.

$\displaystyle\mathbf{c_{hybrid}}$
$\displaystyle=\operatorname{MLP}([c_{1},...,c_{i}])$ (12)

Finally, the coordination patterns $\mathbf{c_{hard}}$ or
$\mathbf{c_{hybrid}}$ flows into a FeedForward Module to enhance the
representation ability and generate $\mathbf{c_{final}}$.

####  3.1.2 Level-Adaptive Transformer-based mixing network (LA-QTransformer)

Limited by the dueling structure mixing network (such as QPLEX) large search
space, we take the popular QMIX-like monotonic mixing network as the baseline
model. Our framework can be expressed as
$Q_{tot}=\sum_{i}^{m}w_{i}(s)q_{i}(o_{i},\tau_{i})$, where $w_{i}(s)$ is the
non-negative parameter realized by the proposed mixing network (LA-
QTransformer), and $q_{i}(o_{i},\tau_{i})$ is the individual Q value of agent
$i$.

The right part of figure 2 shows the structure of LA-QTransformer. LA-
QTransformer has two essential modules, the coordination decomposition module
and the coordination integration module. In the coordination decomposition
module, LA-QTransformer utilizes the LA-Transformer module to generate
different coordination patterns for different agents. After that, LA-
QTransformer utilizes a multi-head attention module to combine all agents’
coordination patterns and generate agents’ credits.

Due to the input entity’s different classes and dimensions, we first divide
the state features into several entities via the entity class and utilize the
MLP-based encoder to embed all entities’ features into the same dimension.
Then we take the preprocessed state features into the LA-Transformer module to
generate suitable coordination patterns. After that, we utilize the multi-head
attention module to combine all coordination patterns and generate the credit
assignment weights. Finally, LA-QTransformer takes the dot product operation
to merge agents’ Q values and generate the total Q value $Q_{total}$. In
addition, a bias term is used to make up for the residual.

###  3.2 Population invariant agent via Transformer (PIT)

To realize the coordination transfer in more general scenarios, we design the
Population Invariant agent via Transformer (PIT), as shown in the left of
Figure 2. PIT has three main parts, the Property Group Module, the GRU Core,
and the Adaptive Action Module. In the Property Group Module, PIT explicitly
groups the input entities via entities’ property and generates group
embeddings. The GRU Core merges all group embeddings. The Adaptive Action
Module makes it adaptive to the dynamic action space from different scenarios.

Property Group Module. We first divide the observation into the agent
attribute features $o_{self}$, and several group feature sets
$\mathbf{o_{special}}$ via the entity’s property. For example, in SMAC, the
role property, such as ally and enemy, can be used to divide groups. Due to
solid relevance in the same groups, we introduce the Transformer module to
generate adaptable and general relevant embeddings. To solve the dynamic
entity population problem, we unify all entities in the same groups and
represent these features on the group level. Inspired by the mean-field method
and attention mechanism, we represent the group features with the mean
features of all inner group entities $e_{m}$ and the most relevant features
$e_{i}$.

GRU Core. The GRU Core utilizes the agent-self features $e_{self}$ and
different group representation $e_{m,i},e_{a,i}$ to capture the temporal
change of the group feature and merge all of these features.

Adaptive Action Module. Inspired by the Action Semantics Network [28], we
classify actions to adapt to dynamic action space. We divide the total action
space into self-related actions, such as move and no-op, and interacting
actions, such as attack. Considering that the interacting action is highly
related to interacting entities, we utilize the embedding features generated
from Transformer to construct interacting action. Note that some properties do
not include the action attribute, so we add an action mask module to block
unrelated actions.

The details of PIT can be seen in Appendix.

##  4 Experimental results

Figure 4: The performance of PIT LA-QTransformer and PIT in SMAC. We compare
our methods with several baselines in homogeneous and heterogeneous scenarios.

We evaluate the performance of PIT and LA-QTransformer in SMAC, which is a
popular MARL benchmark based on the real-time strategy game StarCraft II. In
SMAC, each unit is controlled by an independent agent with local observation.
In contrast, the opponent’s units are controlled by the built-in rule-based
AI. To test the robustness of our methods, all experiments are run with five
random seeds and evaluated under seven threads in parallel.

###  4.1 Baseline performance

Figure 4 shows the performance in small-sized homogeneous, large-sized
homogeneous, and median-size heterogeneous scenarios, respectively. In the
simple 3m scenario, both LA-QTransformer(hybrid) and LA-QTransformer(hard)
achieve excellent performance. In more challenging scenarios, LA-
QTransformer(hybrid) outperforms other baseline methods. However, due to
missing details of selecting one specific coordination pattern, LA-
QTransformer(hard) method shows suboptimal performance.

###  4.2 Performance in transfer learning setting

Figure 5: The experimental results of PIT and UPDeT with mixing modules, LA-
QTransformer and QMIX. Agents are well-trained to converge in the original
scenarios and then fine-tuned in the latter scenarios.

Figure 5 shows the evaluated performance of PIT with LA-Transformer and QMIX
and current state-of-the-arts multi-agent transfer learning method UPDeT. The
experimental results present the performance of different modules in the
transfer learning setting.

Transfer in different scales. Figure 5(a) shows that both two LA-QTransformer
methods can achieve excellent coordination knowledge transfer while QMIX is
unstable. In Figure 5(b), we conduct a more extensive transfer test in
different scales (more than two times). Due to the enormous scale changes, LA-
QTransformer needs to regenerate proper coordination patterns to adapt to the
difference of scenarios, which shows little instability at the beginning of
the training process. However, LA-QTransformer has strong adaptability and can
quickly converge to the optimal policy. Besides, the experimental result
demonstrates that PIT is superior to UPDeT.

Transfer in different difficulty levels. We evaluate the performance of
coordination knowledge transfer in two scenarios with different difficulty
levels (8m_vs_9m is simple and 5m_vs_6m is complex). Figure 5(c) and 5(d) show
the transfer from simple to complex and from complex to simple respectively.
LA-QTransformer shows an advantage in the jumpstart (the initial performance)
and the asymptotic performance. Due to no coordination knowledge transfer,
QMIX’s performance is unstable in transferring from simple to complex.

Transfer in heterogeneous scenarios. As shown in Figure 5(e) and 5(f), we test
the performance in two heterogeneous scenarios with different scales. The LA-
Transformer(hybrid) outperforms other baselines because the coordination
patterns in heterogeneous scenarios are also helpful. In heterogeneous
scenarios, the coordination patterns in different types of agents have obvious
differences, making LA-Transformer(hard) more likely to miss valuable
information. This results in suboptimal performance. Besides, the empirical
results show that UPDeT does not perform well in heterogeneous scenarios.

###  4.3 Performance with curricular transfer learning

Figure 6: The curricular transfer experiment in SMAC scenarios.

We extend our methods to curricular learning and make the scaling up
experiment. Figure 6 shows that agents are firstly trained in the 5m_vs_6m
scenario and then transferred to the 8m_vs_9m scenario with 2M training steps,
and finally tested in the 10m_vs_11m scenario. According to the results,
curricular learning can correct coordination patterns and generate more
general coordination patterns. The performance in the 10m_vs_11m scenario
shows that LA-QTransformer with PIT achieves excellent performance, even
without any further training.

###  4.4 Interpretation of LA-QTransformer

(a) Credit assignment values in two different scenarios

(b) Weight of pairwise-coordination.

Figure 7: The interpretation of LA-QTransformer.

Transfer ability. Figure 7 (a) shows the temporal credit assignment values of
LA-QTransformer, and it verifies the feasibility of coordination transfer. The
Figure shows that the second half credit assignment values of the 8m_vs_9m
scenario have a remarkable similarity with that in the 5m_vs_6m scenario,
demonstrating the coordination of the coordination policies.

Coordination level of LA-QTransformer. In Figure 7 (b), we present the
pairwise coordination weight in LA-QTransformer and find that the complex
scenario needs higher-level coordination. Because the average pairwise
coordination weight of 5m_vs_6m scenario is smaller than that in the 8m_vs_9m
scenario. This also indicates that learning the whole coordination policy has
difficulty in realizing the coordination knowledge transfer due to policy
overfitting.

Initial win rate analysis. Table 1 emphasizes the jumpstart performance of
different mixing networks under the PIT. The jumpstart metric can measure the
agent performance without training and shows method generalization. Compared
with QMIX, LA-QTransformer can learn a more robust coordination policy via
coordination decomposition.

Table 1: Test win rate of PIT without training. new (origin) | LA-QTransformer(hybrid) | LA-QTransformer(hard) | QMIX
---|---|---|---
5m (3m) | 97.2% | 97.1% | 95.3%
7m (3m) | 97.2% | 96.9% | 95.9%
8m9m (5m6m) | 95.1% | 97.1% | 96.1%
5m6m (8m9m) | 83.2% | 77.0% | 21.7%
1s2z (2s3z) | 98.1% | 97.1% | 91.5%
2s3z (1s2z) | 90.8% | 47.7% | 84.8%

###  4.5 Ablation study on ML-Transformer

We evaluate the difference between the LA-Transformer and the traditional
Transformer stacking method. Figure 8 shows that stacking two Transformer
layers does not lead to any performance improvement. In cooperative MARL,
stacking Transformer modules blurs different level relationships and has
difficulty in capturing the proper coordination level. However, LA-Transformer
can explicitly distinguish the differences between different coordination
levels and perceive the suitable coordination level. LA-Transformer(hard) can
be explained as the combination of the two stacking Transformer modules and
select the proper coordination level, while LA-Transformer(hybrid) implicitly
generates the coordination level. The initial win rate verifies that.

Figure 8: Ablation study on the Transformer stacking method.

##  5 Related work

Credit assignment methods in cooperative multi-agent Q-learning. VDN [23] uses
a simple sum operation to generate the global Q value. QMIX [21], as an
extension of VDN, introduces the monotonicity hypothesis to satisfy the IGM
condition and uses the hyper-network to achieve it [7]. Qatten [31] introduces
the multi-head attention mechanism to construct the mixing network to obtain
better performance. QPLEX [26] uses the dueling network to avoid direct
optimization from monotonicity assumptions. Previous methods always focus on
the whole coordination policy and can achieve excellent performance in several
cooperative tasks. However, learning on the whole coordination level may lead
to the over-fitting coordination policy and is unsuitable for transfer
learning tasks.

Multi-agent transfer learning methods. Current MATL has two branches: the
auxiliary training technique and the adaptive network structure. Reusing
replay buffer and policy distillation are the prevalent auxiliary training
methods. [29] improves the efficiency of value-based MATL by reusing the
transition data generated in previous scenarios. Inspired by policy
distillation [22], Liu et al. [16] proposes to transfer the knowledge learned
in a single agent to multiple agents and uses the n-step return to approximate
the difference of the local environment dynamics. This can achieve selective
migration and avoid the negative transfer.

In adaptive network structure methods, agents can directly reload the previous
knowledge via adapting to the dynamic observation and action shape. DyAN [29]
uses the graph neural network to decompose the observation into each entity
node to deal with the uncertain population of entities. Unlike DyAN, EPC-
MADDPG [17] merges varying entity features to fixed-dimensional features with
the attention mechanism. UPDeT [11] firstly proposes to use Transformer to
handle dynamic features. It separates the observation features into several
entity-based features and uses the Transformer module to generate different
actions. However, previous methods are all limited in realizing the
coordination knowledge transfer.

##  6 Conclusion

In this paper, we propose a novel mixing network for cooperative MARL, called
LA-QTransformer, to achieve coordination knowledge transfer. Compared with the
agent-level knowledge transfer, coordination transfer has better
generalization and scalability. Our network first decomposes the correlations
among agents into a series of agent-specific coordination patterns via the
level-adaptive Transformer (LA-Transformer) and then integrates the
coordination patterns for the purpose of credit assignment. To ensure the
coordination knowledge transfer in more varieties of scenarios, we design a
novel agent structure named population invariant agent with Transformer.
Experiments on the SMAC benchmarks show that LA-QTransformer can achieve
excellent coordination policy transfer and outperforms current SOTA baselines.

Through experiments, we notice that curriculum learning can correct the
coordination patterns and realize efficient coordination transfer. An
interesting question then is how to design a systematic curriculum to achieve
more efficient coordination transfer. Moreover, it should be noted that while
we only realized coordination knowledge transfer in multi-agent Q-learning, we
may also consider extending our approach to policy-based methods, such as
MADDPG [18] and MAPPO [34].

## References

  * Agarwal et al. [2019] Akshat Agarwal, Sumit Kumar, and Katia Sycara.  Learning transferable cooperative behavior in multi-agent teams.  _arXiv preprint arXiv:1906.01202_ , 2019.
  * Berner et al. [2019] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al.  Dota 2 with large scale deep reinforcement learning.  _arXiv preprint arXiv:1912.06680_ , 2019.
  * Boutsioukis et al. [2011] Georgios Boutsioukis, Ioannis Partalas, and Ioannis Vlahavas.  Transfer learning in multi-agent reinforcement learning domains.  In _European Workshop on Reinforcement Learning_ , pages 249–260. Springer, 2011.
  * Da Silva and Costa [2019] Felipe Leno Da Silva and Anna Helena Reali Costa.  A survey on transfer learning for multiagent reinforcement learning systems.  _Journal of Artificial Intelligence Research_ , 64:645–703, 2019.
  * Dosovitskiy et al. [2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Foerster et al. [2018] Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson.  Counterfactual multi-agent policy gradients.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 32, 2018.
  * Ha et al. [2016] David Ha, Andrew Dai, and Quoc V Le.  Hypernetworks.  _arXiv preprint arXiv:1609.09106_ , 2016.
  * Haarnoja et al. [2018a] Tuomas Haarnoja, Sehoon Ha, Aurick Zhou, Jie Tan, George Tucker, and Sergey Levine.  Learning to walk via deep reinforcement learning.  _arXiv preprint arXiv:1812.11103_ , 2018a.
  * Haarnoja et al. [2018b] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  _arXiv preprint arXiv:1801.01290_ , 2018b.
  * Horling and Lesser [2004] Bryan Horling and Victor Lesser.  A survey of multi-agent organizational paradigms.  _Knowledge Engineering Review_ , 19(4):281–316, 2004.
  * Hu et al. [2021] Siyi Hu, Fengda Zhu, Xiaojun Chang, and Xiaodan Liang.  Updet: Universal multi-agent reinforcement learning via policy decoupling with transformers.  _arXiv preprint arXiv:2101.08001_ , 2021.
  * Jang et al. [2016] Eric Jang, Shixiang Gu, and Ben Poole.  Categorical reparameterization with gumbel-softmax.  _arXiv preprint arXiv:1611.01144_ , 2016.
  * Kellogg et al. [1976] R Bruce Kellogg, Tien-Yien Li, and James Yorke.  A constructive proof of the brouwer fixed-point theorem and computational results.  _SIAM Journal on Numerical Analysis_ , 13(4):473–483, 1976.
  * Kitaev et al. [2020] Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya.  Reformer: The efficient transformer.  _arXiv preprint arXiv:2001.04451_ , 2020.
  * Kraemer and Banerjee [2016] Landon Kraemer and Bikramjit Banerjee.  Multi-agent reinforcement learning as a rehearsal for decentralized planning.  _Neurocomputing_ , 190:82–94, 2016.
  * Liu et al. [2019] Yong Liu, Yujing Hu, Yang Gao, Yingfeng Chen, and Changjie Fan.  Value function transfer for deep multi-agent reinforcement learning based on n-step returns.  In _IJCAI_ , pages 457–463, 2019.
  * Long et al. [2020] Qian Long, Zihan Zhou, Abhibav Gupta, Fei Fang, Yi Wu, and Xiaolong Wang.  Evolutionary population curriculum for scaling multi-agent reinforcement learning.  _arXiv preprint arXiv:2003.10423_ , 2020.
  * Lowe et al. [2017] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch.  Multi-agent actor-critic for mixed cooperative-competitive environments.  _arXiv preprint arXiv:1706.02275_ , 2017.
  * Mahajan et al. [2019] Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson.  Maven: Multi-agent variational exploration.  _arXiv preprint arXiv:1910.07483_ , 2019.
  * Oliehoek et al. [2016] Frans A Oliehoek, Christopher Amato, et al.  _A concise introduction to decentralized POMDPs_ , volume 1.  Springer, 2016.
  * Rashid et al. [2018] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson.  Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning.  _arXiv preprint arXiv:1803.11485_ , 2018.
  * Rusu et al. [2015] Andrei A Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirkpatrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell.  Policy distillation.  _arXiv preprint arXiv:1511.06295_ , 2015.
  * Sunehag et al. [2017] Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al.  Value-decomposition networks for cooperative multi-agent learning.  _arXiv preprint arXiv:1706.05296_ , 2017.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _arXiv preprint arXiv:1706.03762_ , 2017.
  * Vinyals et al. [2017] Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, Alireza Makhzani, Heinrich Küttler, John Agapiou, Julian Schrittwieser, et al.  Starcraft ii: A new challenge for reinforcement learning.  _arXiv preprint arXiv:1708.04782_ , 2017.
  * Wang et al. [2020a] Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang.  Qplex: Duplex dueling multi-agent q-learning.  _arXiv preprint arXiv:2008.01062_ , 2020a.
  * Wang et al. [2020b] Tonghan Wang, Heng Dong, Victor Lesser, and Chongjie Zhang.  Roma: Multi-agent reinforcement learning with emergent roles.  In _Proceedings of the 37th International Conference on Machine Learning_ , 2020b.
  * Wang et al. [2019] Weixun Wang, Tianpei Yang Yong Liu, Jianye Hao, Xiaotian Hao, Yujing Hu, Yingfeng Chen, Changjie Fan, and Yang Gao.  Action semantics network: Considering the effects of actions in multiagent systems.  _arXiv preprint arXiv:1907.11461_ , 2019.
  * Wang et al. [2020c] Weixun Wang, Tianpei Yang, Yong Liu, Jianye Hao, Xiaotian Hao, Yujing Hu, Yingfeng Chen, Changjie Fan, and Yang Gao.  From few to more: Large-scale dynamic multiagent curriculum learning.  In _AAAI_ , pages 7293–7300, 2020c.
  * Wang et al. [2020d] Yihan Wang, Beining Han, Tonghan Wang, Heng Dong, and Chongjie Zhang.  Off-policy multi-agent decomposed policy gradients.  _arXiv preprint arXiv:2007.12322_ , 2020d.
  * Yang et al. [2020a] Yaodong Yang, Jianye Hao, Ben Liao, Kun Shao, Guangyong Chen, Wulong Liu, and Hongyao Tang.  Qatten: A general framework for cooperative multiagent reinforcement learning.  _arXiv preprint arXiv:2002.03939_ , 2020a.
  * Yang et al. [2020b] Yaodong Yang, Ying Wen, Jun Wang, Liheng Chen, Kun Shao, David Mguni, and Weinan Zhang.  Multi-agent determinantal q-learning.  In _International Conference on Machine Learning_ , pages 10757–10766. PMLR, 2020b.
  * Ye et al. [2020] Deheng Ye, Zhao Liu, Mingfei Sun, Bei Shi, Peilin Zhao, Hao Wu, Hongsheng Yu, Shaojie Yang, Xipeng Wu, Qingwei Guo, et al.  Mastering complex control in moba games with deep reinforcement learning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 6672–6679, 2020.
  * Yu et al. [2021] Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre Bayen, and Yi Wu.  The surprising effectiveness of mappo in cooperative, multi-agent games.  _arXiv preprint arXiv:2103.01955_ , 2021.
  * Zhao et al. [2017] Xiangyu Zhao, Liang Zhang, Long Xia, Zhuoye Ding, Dawei Yin, and Jiliang Tang.  Deep reinforcement learning for list-wise recommendations.  _arXiv preprint arXiv:1801.00209_ , 2017.


# ECAPA2: a hybrid neural network architecture and training strategy for
robust speaker embeddings

###### Abstract

In this paper, we present ECAPA2, a novel hybrid neural network architecture
and training strategy to produce robust speaker embeddings. Most speaker
verification models are based on either the 1D- or 2D-convolutional operation,
often manifested as Time Delay Neural Networks or ResNets, respectively.
Hybrid models are relatively unexplored without an intuitive explanation what
constitutes best practices in regard to its architectural choices. We motivate
the proposed ECAPA2 model in this paper with an analysis of current speaker
verification architectures. In addition, we propose a training strategy which
makes the speaker embeddings more robust against overlapping speech and short
utterance lengths. The presented ECAPA2 architecture and training strategy
attains state-of-the-art performance on the VoxCeleb1 test sets with
significantly less parameters than current models. Finally, we make a pre-
trained model publicly available to promote research on downstream tasks.

Index Terms—  speaker verification, speaker embeddings, ECAPA2

##  1 Introduction

Speaker verification tries to determine if two speech utterances originate
from the same speaker. In recent years, the field has gained significant
performance improvements due to the availability of large, labeled datasets
[1, 2] and the development of specialized neural network architectures [3, 4].

Most speaker verification architectures are based on the 1D- or
2D-convolutional operation. Examples of the former include Time Delay Neural
Networks (TDNNs) such as the popular x-vector model [3] and ECAPA-TDNN [4].
2D-convolutional architectures are mostly based on the ResNet architecture,
such as the fwSE-ResNet model presented in [5]. Recently, hybrid architectures
[5, 6] that try to combine the benefits of both convolutional operations have
been proposed.

However, relatively little research is done on assessing the impact of the
usage of either or a combination of these architectural choices besides raw
speaker verification performance. For example, the authors of [7] propose an
architecture consisting of separate 1D- and 2D-convolutional subnetworks
processing the input independently. The resulting model outperforms the
singular TDNN- and ResNet-based speaker verification models. A similar
observation is made in system fusions often employed in speaker verification
competitions, where the fusion of TDNN- and ResNet-based models proves to be
the most complementary, indicating both models learn distinct speaker
characteristics [8]. A 2D-convolutional stem on top of the ECAPA-TDNN model is
proposed in [5]. As the kernels of a TDNN-based model span the complete
frequency range, the 2D-convolutional stem should alleviate the limited
capability of the TDNN architecture to model frequency-independent features. A
similar approach is used in [6], resulting in a performance improvement
compared to a regular TDNN-based architecture.

In this paper, we perform a series of model interpretability analyses to
better understand the impact of architectural choices on the resulting speaker
embeddings. This includes a feature ablation analysis to assess input
robustness and neuron conductance experiments to determine the impact of
different kernel types in the network. Subsequently, we base our proposed
ECAPA2 architecture on the findings of the aforementioned model
interpretability analysis. In addition, we enhance the model training strategy
to produce speaker embeddings which are robust against overlapping speakers
and short utterance durations. Finally, we provide a publicly
available111huggingface.co/Jenthe/ECAPA2, pre-trained model with
straightforward APIs to extract embeddings to foster further research on the
usage of speaker embeddings in downstream applications.

##  2 Architectural analysis

In this section, we provide a series of model interpretability experiments to
establish the structural differences of 1D- and 2D-convolutional speaker
verification architectures and the characteristics of the resulting speaker
embeddings. The 1D- and 2D-convolutional architectures are represented by the
E-TDNN and ResNet34 models described in [4], respectively. Both models perform
similar on the VoxCeleb1 test sets [2] and have a comparable number of
parameters. The input features consists of 80-dimensional Mel-filterbanks.
Architectural details and the training strategy can be found in the
accompanying paper [4]. All interpretability experiments in this section are
performed using these models unless described otherwise.

Fig. 1: Effect of temporal frame masking on a TDNN- and ResNet-based speaker
verification model measured on the VoxCeleb1-O test set.

Fig. 2: Effect of input filterbank masking on a TDNN- and ResNet-based speaker
verification model measured on the VoxCeleb1-O test set.

###  2.1 Input feature robustness

First, we want to establish how the different architectures behave towards
alterations of the input features. Figure 1 depicts the impact on the
VoxCeleb1-O test set when masking a varying amount of consecutive temporal
frames of the input Mel-filterbanks. While the baseline performance is nearly
identical, we see a greater degradation of the TDNN-based architecture in
comparison to the ResNet model when the amount of masked frames grows. A
similar behaviour can be observed in Figure 2, which depicts the speaker
verification performance when masking varying numbers of filterbanks. Again,
the ResNet-based model seems much more robust against alterations to its input
features, with a relatively modest decrease in performance when the number of
masked filterbanks is low.

These results corroborate the notion that ResNet-based architectures learn
spatially invariant features [5]. While a TDNN-based model could theoretically
learn kernels spanning a limited range of frequencies at different positions,
the characteristics of a 1D-convolutional kernel seems to push it towards
learning features depending on the full frequency spectrum. This results in a
severe degradation when not all frequency information is available.

###  2.2 Effective receptive fields

The receptive field of a model indicates the region of the input space that
influences the response of an individual neuron in a network. We can
distinguish between the theoretical receptive field, which simply defines the
region of the input which can affect a neuron, and the effective receptive
field (ERF), which provides a measurement of the proclivity of different input
regions to affect a neuron [9]. It has been shown that a limited ERF can have
a negative impact on performance [9, 10].

We calculate the ERFs of the speaker verification models similarly to [9] by
placing a gradient signal of 1 at the spatially centered neuron in the layer
before the pooling operation. Subsequently, the input gradients are gathered
by backpropagating the gradient signal trough the randomly initialized model.
We disabled striding in the ResNet models as to focus the analysis on the
effect of the convolutional layers.

Figure 3 depicts the ERF of the spatially centered neuron in the last layer
before the pooling operation of both models. We observe the same Gaussian-
shaped receptive field as reported in [9] for both architectures. In contrast
to the TDNN architecture, the ERF of the ResNet model is centered around the
frequency dimension of the output neuron. This indicates that the neurons
before the pooling operation are more inclined to focus on input features
around the center of the receptive field, potentially not exploiting frequency
information at the edges of its ERF.

TDNN (8 x 1D-conv)

ResNet (34 x 2D-conv)

ResNet (34 x 2D-conv) + TDNN (1 x 1D-conv)

Fig. 3: Effective receptive field (top) and conductance (bottom) of the
spatially centered neuron before the pooling layer in speaker verification
architectures.

Figure 4 illustrates the relationship between the depth of a ResNet-based
model and its corresponding ERF. We also plot the gradient response of the
TDNN model. We observe a tendency towards a uniform receptive field in the
frequency dimension of the ResNet models when the number of convolutional
layers increases. With a growing number of convolutional layers, the receptive
field is expanding, which results in a more uniform area around the mean of
the Gaussian reflecting the ERF. This indicates that larger ResNet models are
eventually more inclined to exploit the complete frequency range, although at
a significant computational cost due to the increased convolutional
operations.

Fig. 4: Intersection of the effective receptive field at the corresponding
temporally centered input frame of a spatially centered output neuron before
the pooling layer in increasingly larger ResNet speaker verification
architectures. Notice the tendency of the receptive field to become uniform
across the frequency dimension when the number of 2D-convolutional layers
increases.

###  2.3 Neuron conductance

To establish the impact of the ERF on a trained speaker verification model, we
apply a neuron conductance analysis following the method described in [11].
Neuron conductance is based on the Integrated Gradients [12] attribution
method and assigns importance scores to input features by integrating the
gradients of the output of the model with respect to the inputs along a path
from a baseline to the desired input. Examples of neuron conductance responses
are depicted in Figure 3 based on a speech sample of 4 seconds. We note that
these results where consistent across different input utterances and pre-
pooling neurons.

We observe that the attributions of both the TDNN- and ResNet-based model are
closely related to their corresponding ERFs. The TDNN model has a uniform-like
attribution in the frequency dimension while the ResNet network has a much
more localized attribution with the magnitude of the attributions at the edges
of its ERF diminishing quickly. This corroborates the observation that the
Gaussian-like ERF of a singular ResNet-based architecture results in pre-
pooling features based on a limited input frequency range. While large ResNet
models somewhat alleviate this issue, as depicted in Figure 4, we attempt to
solve this more efficiently with the proposed ECAPA2 architecture.

Fig. 5: Topology of the ECAPA2 architecture. T denotes the number of temporal
frame-level features and k indicates kernel size.

##  3 Proposed ECAPA2 architecture

In this section, we describe and motivate our proposed ECAPA2 architecture
based on observations from the previous section. Mainly, we want our model (1)
to be robust against the negative performance impact of input alterations
observed in Section 2.1, and (2) to have pre-pooling hidden features with an
ERF covering the complete frequency range uniformly to exploit all frequency
information. ECAPA2 achieves this by defining two main modules, each focusing
on either local or global frequency regions. An overview of the final
architecture is depicted in Figure 5.

###  3.1 Local feature extractor

The first module of the proposed ECAPA2 model consists of a cascade of Local
Feature Extractor (LFE) blocks. Each LFE block consists of three
2D-convolutional operations followed by the previously proposed frequency-wise
Squeeze-Excitation (fwSE) module [5] as depicted in Figure 6. This enables the
network to learn robust, spatially-invariant features and counteracts the
sensitivity to input alterations as observed in Section 2.1. The fwSE module
allows the model to inject global context information in the intermediate
hidden features, resulting in more capable frame-level representations. A
learnable positional encoding vector across the frequency dimension is added
to enable the module to integrate frequency positional information into the
features as shown in [5].

Strided convolutions in the frequency dimension are applied at specific
locations in the LFE module to widen the receptive field. This has the
additional benefit of increasing the computational efficiency by downsampling
the hidden feature map dimensions. The strided convolutions are only
introduced later in the network to avoid the loss of potential useful
information in the frequency dimension.

The output hidden features of the LFE block will focus on a local frequency
region due to the Gaussian-like shape of the corresponding ERF, similar to
ResNet-based speaker verification architectures. This makes it harder for the
network to model more expressive features based on the complete frequency
range. While increasing the number of 2D-convolutional layers will make the
ERF more uniform, this requires a substantial increase in model parameters. We
attempt to solve this limitation efficiently with a subsequent Global Feature
Extractor (GFE) module.

Fig. 6: Local Feature Extractor block of the ECAPA2 architecture. k denotes
kernel size with s indicating an optional striding value. h determines the
hidden feature dimension of the fwSE block.

###  3.2 Global feature extractor

The GFE module consists of a small TDNN network to integrate the frequency
information learned in the local feature extractor and is depicted in Figure
5. The kernel size of each 1D-convolutional layer is 1, except for the Res2Net
[13] 1D-convolutional layer, which is set to 3. The kernel characteristics of
1D-convolutions create a uniform ERF across the frequency dimension. This is
illustrated in Figure 3, where the addition of a single TDNN layer at the end
of a ResNet-based structure results in a uniform ERF in the frequency
dimension of the pre-pooling hidden features. By placing the 1D-convolutional
layers at the end of the network, we also circumvent the robustness and
spatial dependency issues covered in Section 2.1. Subsequently, we use
channel-dependent attentive statistics (CAS) pooling [4] to integrate global
context in the attention module and project the pooled statistics to a
192-dimensional speaker embedding using a linear layer.

##  4 Proposed training strategy

The architecture described in Section 3 is optimized using the subcenter
Additive Angular Margin (AAM) softmax loss function [14, 15]. AAM promotes
intra-class compactness and inter-class dispersion by applying a margin
penalty on the target class during training. The subcenters mitigates the
performance impact of potential noisy samples during training due to the short
utterance cropping and aggressive augmentation methods often applied in
speaker verification training stages. To obtain speaker embeddings which are
robust against overlapping speakers and short-duration utterances, two
additional training enhancements are employed.

###  4.1 Overlapping speaker robustness

We incorporate the recently proposed margin-mixup training strategy [16] to
support an embedding space with overlapping speakers. During margin-mixup, the
network has to predict the target classes of an input speech mixture
consisting of two speakers with a random energy mixing ratio $\lambda$. The
margin penalty of the AAM-softmax loss function is applied proportionally to
both target classes according to $\lambda$. More details can be found in the
accompanying paper [16].

###  4.2 Short utterance robustness

Most speaker verification systems are trained and tested with the assumption
of long input utterances. However, in real-world scenarios this assumption
does not always hold. Previous work has already shown that models trained with
long utterance conditions perform poorly on short speech segments and vice
versa [17]. We alleviate this issue by altering the large-margin fine-tuning
(LM-FT) stage [8] with a variable length training (VLT) strategy. During the
LM-FT stage, we randomly crop the input utterance between 1 and 5 seconds with
a probability of $\alpha$, otherwise we take the standard crop length of 5
seconds. We argue this should enable the model to generate features fit for
both short and long input utterances.

##  5 Experimental setup

We train our ECAPA2 model using the development partition of the VoxCeleb2 [2]
dataset. We use a two-fold speed augmentation with factors 0.9 and 1.1 to
create additional speakers from the training set, similar to [18]. We take
random crops of 2 seconds and apply a random augmentation using the MUSAN [19]
(noise, music, babble) and RIR dataset [20] (reverb). We use 256-dimensional
FFTs as input with a window and hop length of 25 ms and 10 ms, respectively.
SpecAugment [21] is applied to increase robustness of the speaker embeddings
by randomly masking between 0 and 5 time frames and 0 and 32 FFT bins.

The AAM-softmax margin penalty is set to 0.2 while the $\alpha$ and $\beta$
parameter of the margin-mixup training protocol [16] is set to 0.05. The
number of subcenters is set to 2 for each class. Our model is trained using
the Adam [22] optimizer with a cyclical learning rate (CLR) [23] using the
triangular2 policy with a minimum and maximum learning rate of 1e-8 and 1e-3,
respectively. The cycle length is set to 120k steps. Weight decay is applied
on all layers with a value of 2e-4.

After the initial training stage, we apply the LM-FT strategy as proposed [8].
Additionally, we employ our proposed short utterance sampling strategy
described in Section 4.2 with $\alpha=0.4$. During this stage, the AAM-softmax
penalty and maximum crop size are increased to 0.4 and 5 seconds,
respectively. The MUSAN-, RIR- and SpecAugment-based augmentations are
disabled. The sampling probability of the speed-augmented speakers is reduced
to 0.2 to prevent domain mismatch. The CLR cycle length is set to 60k
iterations while the maximum learning rate is decreased to 1e-5. For both the
initial and fine-tuning stage, the system is trained for one cycle using a
batch size of 256 and 512, respectively.

We apply top-500 adaptive s-normalization [24] on the cosine similarity scores
of the verification trials with an imposter cohort existing of the average of
the length-normalized embeddings for each speaker in the training partition of
VoxCeleb2. Finally, a logistic regression based calibration stage is applied
as described in [8] with the utterance duration as our only quality
measurement.

|  | Standard Benchmarks | Custom
---|---|---|---
System | Params | Vox1-O | Vox1-E | Vox1-H | Vox1-M | Vox1-S
| EER | MinDCF | EER | MinDCF | EER | MinDCF | EER | EER
SE-ResNet-100 [25] | 40M | 0.43 | 0.032 | 0.53 | 0.058 | 1.04 | 0.105 | - | -
SE-ResNet-100 + CAS [25] | 42M | 0.36 | 0.037 | 0.55 | 0.060 | 1.05 | 0.104 | - | -
ResNet-101-64 [26] | 206M | 0.50 | 0.035 | 0.64 | 0.051 | 0.97 | 0.078 | - | -
ECAPA-TDNN [4] | 14M | 0.87 | 0.106 | 1.12 | 0.131 | 2.12 | 0.210 | 24.78 | 11.05
ECAPA-CNN-TDNN [5] | 60M | 0.61 | 0.037 | 0.76 | 0.079 | 1.32 | 0.135 | 22.07 | 9.23
fwSE-ResNet-87 [27] | 30M | 0.50 | 0.037 | 0.71 | 0.077 | 1.26 | 0.120 | 21.32 | 9.02
ECAPA2 | 27M | 0.34 | 0.029 | 0.52 | 0.058 | 0.99 | 0.098 | 17.42 | 7.92
Table 1: Speaker verification performance of the proposed ECAPA2 model
compared to other state-of-the-art architectures.

Speaker verification performance is verified on the standard VoxCeleb1 test
sets with the equal error rate (EER) and MinDCF metric using a $P_{target}$
value of $10^{-2}$ with $C_{FA}$ and $C_{Miss}$ set to 1. To verify
performance on overlapping speakers, we use the Vox1-M test set as introduced
in [16], which consists of the same trials as Vox1-O with the trial utterances
heavily mixed with audio from an interfering speaker. To asses performance on
short-duration utterances, we create an additional Vox1-S test set based on
Vox1-O with the trial utterances randomly cropped between 0.5 and 2 seconds.
No score normalization and calibration are applied when validating on Vox1-M
and Vox1-S.

##  6 Results

The results of the proposed ECAPA2 model on the standard VoxCeleb1 benchmarks
and our additional Vox1-M and Vox1-S test sets is given in Table 1. We compare
the ECAPA2 model with the previous ECAPA-TDNN architecture and the more recent
speaker verification models fwSE-ResNet-87 [27] and ECAPA-CNN-TDNN [5]. We
also include the best published single-system EER results on the VoxCeleb1
test sets of models trained only on the development part of VoxCeleb2. The
number of parameters is based on the embedding extraction partition of the
models during inference.

ECAPA2 attains state-of-the-art performance on the Vox1-O and Vox1-E test sets
and is only surpassed with a minor margin on Vox1-H by the ResNet-101-64
system described in [26], a model using significantly more parameters due to a
16-head attention module in the pooling layer. This trend continues, with the
three best competing systems in Table 2 consisting of large ResNet-based
models with a significantly higher number of parameters. This corroborates our
notion that incorporating a small TDNN-based subnetwork acting as a global
feature extractor can attain similar or better results compared to singular
deep ResNet-based architectures in a more efficient manner.

Compared to the fwSE-ResNet-87 model, ECAPA2 gains a 18.3% EER improvement on
our Vox1-M test set, relatively. This supports the findings in [16] which
states that speaker verification models trained with a single speaker
assumption perform poor on overlapping speech. Likewise, ECAPA2 attains a
relative performance improvement of 12.2% on the short utterance Vox1-S test
set compared to the fwSe-ResNet-87 system, reinforcing the notion that current
state-of-the-art speaker verification models are not optimally trained to
handle short utterances.

| Method | EER | MinDCF
---|---|---|---
| ECAPA2 | 0.34 | 0.029
A | no global module | 0.40 | 0.034
B | small global module | 0.38 | 0.032
C | big global module | 0.35 | 0.029
Table 2: Ablation study of ECAPA2 on the Vox1-O test set.

To determine the impact of the GFE module in the ECAPA2 architecture, we
perform an ablation analysis with the results given in Table 2. In experiment
A, we trained the model without the global module, making it structurally
similar to a ResNet-based model with 60 layers. This results in a degradation
on the EER of 15% relative on the Vox1-O test set and signifies that the
global module can counteract the weaknesses of ResNet-based models presented
in Section 2. In experiment B, the global module is replaced with one
1D-convolutional layer with a kernel size equal to 1. This minimal global
module still improves upon a singular ResNet-based architecture but is
outperformed by the proposed architecture, showing that the global module can
benefit from additional complexity. However, incorporating an additional
Res2Conv1D block in the proposed ECAPA2 model did not improve results as
tested in experiment C.

Configuration | Vox1-O | Vox1-M | Vox1-S
---|---|---|---
baseline | 0.34 | 17.42 | 7.92
no margin-mixup | 0.35 | 24.04 | 7.96
no VLT | 0.38 | 17.64 | 8.89
VLT ($\alpha=0.1$) | 0.37 | 17.56 | 8.66
VLT ($\alpha=0.9$) | 0.49 | 17.41 | 7.15
Table 3: Analysis of proposed training strategy.

The impact of the proposed training strategy is given in Table 3. Training
with margin-mixup improves results on Vox1-M with 27.5% EER relative, while
having no significant impact on the non-overlapping test sets Vox1-O and
Vox1-S. The proposed short utterance sampling strategy improves upon fine-
tuning with only long utterances with 10.9% relative in Vox1-S. Additional
experiments with a low and high $\alpha$ cropping probability hyperparameter
shows that the baseline configuration of $\alpha=0.4$ gains the best results
on Vox1-S without impacting performance on the regular test sets.

##  7 Conclusion

In this paper, we introduced ECAPA2, a novel hybrid neural network
architecture for robust speaker embeddings. By addressing the limitations of
existing speaker verification models, ECAPA2 attains state-of-the-art
performance on the VoxCeleb1 test sets with significantly fewer parameters.
Additionally, the proposed training strategy successfully improves the
resilience of the embeddings against overlapping speech and short utterance
lengths.

## References

  * [1] Arsha Nagrani, Joon Son Chung, and Andrew Zisserman,  “VoxCeleb: A large-scale speaker identification dataset,”  in INTERSPEECH 2017, 2017, pp. 2616–2620.
  * [2] Joon Son Chung, Arsha Nagrani, and Andrew Zisserman,  “VoxCeleb2: Deep speaker recognition,”  in INTERSPEECH 2018, 2018, pp. 1086–1090.
  * [3] David Snyder, Daniel Garcia-Romero, Gregory Sell, Daniel Povey, and Sanjeev Khudanpur,  “X-vectors: Robust dnn embeddings for speaker recognition,”  in ICASSP 2018, 2018, pp. 5329–5333.
  * [4] Brecht Desplanques, Jenthe Thienpondt, and Kris Demuynck,  “ECAPA-TDNN: Emphasized channel attention, propagation and aggregation in TDNN based speaker verification,”  in INTERSPEECH 2020, 2020, pp. 3830–3834.
  * [5] Jenthe Thienpondt, Brecht Desplanques, and Kris Demuynck,  “Integrating frequency translational invariance in TDNNs and frequency positional information in 2D ResNets to enhance speaker verification,”  in INTERSPEECH 2021, 2021, pp. 2302–2306.
  * [6] Zhuxin Chen, Duisheng Chen, Hanyu Ding, and Yue Lin,  “The NetEase Games system description for text-dependent sub-challenge of SDSVC 2020,” .
  * [7] Woo Hyun Kang and Jahangir Alam,  “Investigation on Deep Speaker Embedding Extraction Methods for Multi-Genre Speaker Verification,”  in Proc. The Speaker and Language Recognition Workshop (Odyssey 2022), 2022, pp. 376–383.
  * [8] Jenthe Thienpondt, Brecht Desplanques, and Kris Demuynck,  “The idlab voxsrc-20 submission: Large margin fine-tuning and quality-aware score calibration in dnn based speaker verification,”  in ICASSP 2021, 2021, pp. 5814–5818.
  * [9] Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel,  “Understanding the effective receptive field in deep convolutional neural networks,”  in Advances in Neural Information Processing Systems, D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, Eds., 2016, vol. 29.
  * [10] Andre Araujo, Wade Norris, and Jack Sim,  “Computing receptive fields of convolutional neural networks,”  Distill, 2019,  https://distill.pub/2019/computing-receptive-fields.
  * [11] Kedar Dhamdhere, Mukund Sundararajan, and Qiqi Yan,  “How important is a neuron,”  in International Conference on Learning Representations, 2019.
  * [12] Mukund Sundararajan, Ankur Taly, and Qiqi Yan,  “Axiomatic attribution for deep networks,”  in Proceedings of the 34th International Conference on Machine Learning - Volume 70, 2017, ICML’17, p. 3319–3328.
  * [13] Shanghua Gao, Ming-Ming Cheng, Kai Zhao, Xinyu Zhang, Ming-Hsuan Yang, and Philip H. S. Torr,  “Res2Net: A new multi-scale backbone architecture,”  IEEE TPAMI, 2019.
  * [14] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou,  “Arcface: Additive angular margin loss for deep face recognition,”  in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 4685–4694.
  * [15] Jiankang Deng, Jia Guo, Tongliang Liu, Mingming Gong, and Stefanos Zafeiriou,  “Sub-center ArcFace: Boosting face recognition by large-scale noisy web faces,”  in Proc. ECCV, 2020, pp. 741–757.
  * [16] Jenthe Thienpondt, Nilesh Madhu, and Kris Demuynck,  “Margin-mixup: A method for robust speaker verification in multi-speaker audio,”  in ICASSP 2023, 2023, pp. 1–5.
  * [17] Jee-Weon Jung, Hee-Soo Heo, Hye-Jin Shim, and Ha-Jin Yu,  “Short utterance compensation in speaker verification via cosine-based teacher-student learning of speaker embeddings,”  in 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2019, pp. 335–341.
  * [18] Hitoshi Yamamoto, Kong Aik Lee, Koji Okabe, and Takafumi Koshinaka,  “Speaker Augmentation and Bandwidth Extension for Deep Speaker Embedding,”  in INTERSPEECH 2019, 2019, pp. 406–410.
  * [19] David Snyder, Guoguo Chen, and Daniel Povey,  “Musan: A music, speech, and noise corpus,”  arXiv preprint arXiv:1510.08484, 2015.
  * [20] Tom Ko, Vijayaditya Peddinti, Daniel Povey, Michael L. Seltzer, and Sanjeev Khudanpur,  “A study on data augmentation of reverberant speech for robust speech recognition,”  in ICASSP 2017, 2017, pp. 5220–5224.
  * [21] Daniel S. Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D. Cubuk, and Quoc V. Le,  “SpecAugment: A simple data augmentation method for automatic speech recognition,”  in INTERSPEECH 2019, 2019, pp. 2613–2617.
  * [22] Diederik Kingma and Jimmy Ba,  “Adam: A method for stochastic optimization,”  Proc. ICLR, 2015.
  * [23] Leslie N. Smith,  “Cyclical learning rates for training neural networks,”  in 2017 IEEE Winter Conference on Applications of Computer Vision, 2017, pp. 464–472.
  * [24] Pavel Matejka, Ondřej Novotný, Oldřich Plchot, Lukas Burget, Mireia Diez, and Jan Černocký,  “Analysis of score normalization in multilingual speaker recognition,”  in INTERSPEECH 2017, 2017, pp. 1567–1571.
  * [25] Rostislav Makarov, Nikita Torgashov, Alexander Alenin, Ivan Yakovlev, and Anton Okhotnikov,  “ID R&D system description to voxceleb speaker recognition challenge 2022,”  VoxCeleb Speaker Recognition Challenge Workshop 2022, 2022.
  * [26] Miao Zhao, Yufeng Ma, Min Liu, and Minqiang Xu,  “The speakin system for voxceleb speaker recognition challange 2021,” 2021.
  * [27] Jenthe Thienpondt, Brecht Desplanques, and Kris Demuynck,  “Tackling the score shift in cross-lingual speaker verification by exploiting language information,”  in ICASSP 2022, 2022, pp. 7187–7191.