# Heterogeneous Federated Learning

Fuxun Yu1, Weishan Zhang1, Zhuwei Qin1, Zirui Xu1
Di Wang2, Chenchen Liu3, Zhi Tian1, Xiang Chen1
1George Mason University, 2Microsoft
3University of Maryland, Baltimore County
{fyu2, wzhang23, zqin, zxu21, ztian1<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Federated learning learns from scattered data by fusing collaborative models
from local nodes. However, due to chaotic information distribution, the model
fusion may suffer from structural misalignment with regard to unmatched
parameters. In this work, we propose a novel federated learning framework to
resolve this issue by establishing a firm structure-information alignment
across collaborative models. Specifically, we design a feature-oriented
regulation method ($\Psi$-Net) to ensure explicit feature information
allocation in different neural network structures. Applying this regulating
method to collaborative models, matchable structures with similar feature
information can be initialized at the very early training stage. During the
federated learning process under either IID or non-IID scenarios, dedicated
collaboration schemes further guarantee ordered information distribution with
definite structure matching, so as the comprehensive model alignment.
Eventually, this framework effectively enhances the federated learning
applicability to extensive heterogeneous settings, while providing excellent
convergence speed, accuracy, and computation/communication efficiency.

##  1 Introduction

Federated Learning (FL) has drawn great attention with outstanding
collaborative training performance and data privacy supportability [6]. It is
commonly achieved by fusing collaborative nodes’ homogeneous neural networks
through Federated Averaging (FedAvg), which generates a global model by
averaging local parameters with the same coordinates [6, 13]. However, FedAvg
suffers from non-negligible accuracy drop due to inevitable heterogeneity:
highly Non-IID data across nodes may cause huge divergence within the
parameters to be averaged and thus significantly skewed results [18, 4].

Recent works further elaborate on such heterogeneity from a structural
alignment perspective [7, 2, 12]. Although local neural network models have
the homogeneous architecture, the parameters with the same model coordinates
might learn different features at different levels, resulting in chaotic
parameter distribution across collaborative nodes. Therefore, the conventional
coordinates-based averaging scheme might fuse unmatched parameters and
continuously hurt the global convergence [13, 15].

Several Federated Learning optimization works have been proposed to leverage
this structural alignment property (i.e., “permutation invariance” [13]), such
as Representation Matching [7], Bayesian Matching [16], FedMA [13], etc. They
conform to a common methodology: after a certain amount of local training,
they evaluate parameter similarity across local models and reorganize the
permutation, so that approximate parameters could be averaged together.
Although outperformed conventional schemes like FedAvg, these methods still
have certain limitations, such as inaccurate parameter matching, extra
computation/communication overhead, limited heterogeneous applicability, and
compromised data privacy.

To tackle these limitations, we propose a novel FL framework to resolve this
issue by establishing not only structure but also information alignment across
collaborative models. Specifically, we design a feature-oriented regulation
method ($\Psi$-Net), which can effectively identify local models’ structural
hierarchies and grouped components that are associated with particular classes
and corresponding features. Based on such explicit feature information
allocation, matchable structures with similar feature information can be
initialized at the very early training stage, and further guarantees ordered
information distribution with definite structure matching along the whole
training process. Fig. 1 provides a set of intuitive comparisons for parameter
matching across collaborative models. With the proposed feature allocated
regulation, our model’s encoded feature distribution conform to structural
alignments with corresponding classes, while regular FedAvg’s model fusion
suffers from significant parameter mismatching and therefore expected
performance degradation.

We conduct extensive experiments on general datasets (CIFAR10, 100) with two
model architectures to regulate (VGG16 and MobileNet). Our work demonstrates
significant improvement in both convergence speed and accuracy, outperforming
previous state-of-the-art works by large margins (+2.5%$\sim$4.6%) with
significantly less computation and communication cost. Especially under
highly-skewed non-IID scenarios, our work still performs effective and robust
model alignment to facilitate the convergence while most previous methods
cannot. To our best knowledge, this is the very first work that solves the FL
alignment problem directly from a model regulation perspective, which
significantly improves the FL applicability in various heterogeneous settings.

Figure 1: The feature encoding comparison between FedAvg and our proposed
framework. The color of each neuron is determined by its top response class to
indicate its learned feature, as defined in Eq. 2. (a) The original FedAvg
with chaotic feature encoding can suffer from feature averaging conflicts
among different nodes. (b) & (c) In contrast, our framework enforces
structurally-aligned feature encoding and alleviates such averaging conflicts
in both IID and non-IID cases. Experiments are conducted with 10 collaborative
nodes (VGG9 on CIFAR10). IID: Each node has data of 10 classes. Non-IID: Each
node has data of 5 varied classes.

##  2 Background and Motivation

#### Neural Network Permutation Invariance.

From a permutation perspective, the model parameters (e.g., neuron weight
matrix $\Omega_{i}$ in the $i$th node’s model) can be decoupled as
$\omega_{i}\Pi_{i}$, where $\omega_{i}$ defines parameter values and $\Pi_{i}$
defines the alignment sequence. For $N$ collaborative training nodes with the
homogeneous neural network architecture, the model permutation invariance can
be formulated as:

$F(X)=(\omega_{0}\Pi_{0})\Pi_{0}^{T}X=\cdots=(\omega_{i}\Pi_{i})\Pi_{i}^{T}X=\cdots=(\omega_{N}\Pi_{N})\Pi_{N}^{T}X.\hskip
14.22636pt$ (1)

Although the local models across collaborative nodes can achieve the same
global function $F$ on the input set $X$, the local models’ $\omega$ and $\Pi$
can vary from model to model [13, 15]. In such circumstances, their encoded
feature distribution can vary due to the permutation matrix $\Pi$, causing
considerable parameter mismatching ($\omega_{i}\Pi_{i}\neq\omega_{j}\Pi_{j}$)
and thus collaborative training performance degradation.

#### Matched Averaging with Permutation Adjustment.

To achieve FL model alignment, previous methods mainly improve FedAvg by
iteratively matching approximate parameters with permutation adjustment before
each global averaging cycle. Their common approach is to first identify
approximate parameters by evaluating certain parameter characteristics (e.g.,
weight value or activation degree) across collaborative nodes based on a
similarity metric (mostly MSE loss). Then, they will search an adjustment
matrix $\Pi_{i}^{trans}$ for reordering to achieve the minimum divergence with
the averaging target: $\omega_{i}\Pi_{i}\Pi^{trans}\approx\omega_{j}\Pi_{j}$.
Specifically, such a problem can be resolved by optimization algorithms
similar to solving the Wasserstein Barycenter problem [11].

#### Limitations of the State of the Art.

Although the current parameter matching works alleviated the FL model
alignment issue, they still suffer from certain limitations:

(1) Inaccurate Parameter Matching: The state of the art’s (SOTA) structural
alignment accurateness is limited by the quantitative parameter similarity
evaluation. The parameters with the least MSE difference in weight/activation
may carry distinct feature information, and therefore the forced matching
without qualitative verification can cause catastrophic information loss [5,
9];

(2) Heterogeneous Applicability: SOTA assumes different local models have
mostly matchable parameters ($\omega_{i}\approx\omega_{j}$). While the
practical FL heterogeneity may involve highly Non-IID local data and even
asymmetric learning tasks, leading huge model alignment complexity [7, 4];

(3) Performance Overhead: The post-training parameter matching incurs extra
computation load for parameter similarity evaluation and permutation
adjustment optimization. Meanwhile, parameter similarity evaluation across
nodes also introduces considerable communication overhead;

(4) Data Privacy: Moreover, the activation-based similarity evaluation
requires data sharing between different nodes, thus the data privacy can be
compromised [11].

#### Design Motivation: Feature-Level Alignment.

We expect to relieve the above problems from two aspects: On one hand, beyond
feature-agnostic parameter similarity evaluation, we would further interpret
the encoded feature information in neural network components and promote the
parameter similarity assessment to the feature level. On the other hand, we
would design a feature-oriented model regulation method to ensure explicit
feature allocation in specific model structures. Rather than post-training
alignment, such a regulation method adjusts local model architecture according
to their data and task distribution at the very early training stage, and
continuously maintain not only structure but also information alignment for
better FL performance.

Motivated by these two aspects, our work begins with the interpretation of
encoded feature information in neural networks. Specifically, we adopt a
neuron’s class response preference to indicate its learned feature [9, 8, 10,
19]: Starting with single fully-connected layer for classification tasks, such
a preference can be measured by observing the neuron’s activation response
$A(x_{c})$ on inputs $x$ from $C$ classes, as well as its gradients $\partial
Z_{c}/\partial A(x_{c})$ (or weights $w_{c}$ equally) towards a class $c$’s
prediction confidence $Z_{c}$. Combining these two factors and further
generalizing to a multi-layer convolutional neural network, a neuron’s learned
feature can be indicated by its class preference vector:

$P=[p_{1},\ldots,p_{c},\ldots,p_{C}],\hskip
14.22636pt\text{where}\leavevmode\nobreak\
p_{c}=\sum_{b}^{B}A(x_{c,b})*\frac{\partial Z_{c}}{\partial A(x_{c,b})},$ (2)

where $A\leavevmode\nobreak\ (x_{c,b})$ denotes activations and $\partial
Z_{c}/\partial A(x_{c,b})$ denotes gradients from class $c$’s confidence, both
of which are averaged on $B$ batch trials. The effectiveness of such an
interpretation approach is illustrated in Fig. 1, which demonstrates the
parameter/feature divergence from a functionality perspective and highlights
the advantages of our feature-level structural information alignment.

#### Our Work: Feature-Allocated Model Regulation and Collaboration.

Based on the feature interpretation, our major work focuses on the second
motivation aspect. Specifically, we propose a novel federated learning
framework including feature-regulated model design and a set of robust
collaboration policies under various heterogeneous settings. Fig. 2 presents
the framework overview. Leveraging feature distribution exploration with Eq.
3, we designed a $\Psi$-Net model structure that consists of shared layers and
grouped layers, which are expected to share fundamental features with all
learning classes and isolate class-specific features within exclusive
structures, respectively. By regulating existing model architectures into such
a $\Psi$-Net structure, we can maintain features’ consistent structural
allocation in different training stages, and thereby establishing the basis
for the feature level model alignment. The collaboration policies can further
specify the optimal $\Psi$-Net configuration and particular model fusion
schemes for various FL scenarios. Not only capable for regular IID and non-IID
data distribution, our framework can even support extreme heterogeneous cases,
where certain classes might be unlearnable on particular collaborative nodes.
Eventually, this framework is expected to effectively enhance the FL
applicability to extensive heterogeneous settings, while providing excellent
convergence speed, accuracy, and computation/communication efficiency.

##  3 $\Psi$-Net Structure for Feature Allocation

###  3.1 Feature Allocation Definition

For a given neural network, the goal of feature allocation can be defined by a
structure-feature mapping relationship ($M$) between the model meta-structure
sets ($S$) and task sets ($T$):

$M=\\{s^{t}_{i}:t\leavevmode\nobreak\ \leavevmode\nobreak\
\text{or}\leavevmode\nobreak\ <t_{0},...,t_{n}>\\},\\\ \leavevmode\nobreak\
\leavevmode\nobreak\ \text{s.t.}\leavevmode\nobreak\ \leavevmode\nobreak\
\forall i\neq j,\leavevmode\nobreak\ \leavevmode\nobreak\ M(s_{i})\cap
M(s_{j})=\emptyset,$ (3)

where $s^{t}_{i}$ indicates one meta-structure (e.g., one set of neurons in
the neural network) mapped to one (or one set of) task feature $t$, and
$M(s^{t1})\cap M(s^{t2})=\emptyset$ enforces that any individual task feature
can be assigned to only one meta-architecture.

In general, due to the insufficient interpretability, it’s hard to either find
the meta-structure $s_{i}$ or interpret the feature $t$ in a convolutional
neural network, not to mention manipulating its feature encoding mapping. But
interestingly, the AlexNet authors found that AlexNet learns distinct shape-
oriented and color-oriented features in its two separated convolution groups
[1], so did ShuffleNet [17] with their findings that features become biased
among different convolutional groups. The two networks both utilize the group
convolution structure. Although for different purposes, they demonstrate the
similar feature regulation effects. Following such observation, our hypothesis
is in the densely-connected convolutional structures, the training gradients
carrying task features can usually backpropagate through the entire network,
leading to random feature encoding orders. In contrast, the group convolution
separates different groups and build implicit boundaries between them, thus
preventing gradients flowing interleaved and achieving certain feature
regulation effects.

From this perspective, we propose our $\Psi$-net regulation utilizing grouped
convolutional and linear layers to build a meta-structure sets $S$. Meanwhile,
by allocating the class logit $t$ to the structure group $s_{i}$, we assume
that the feature of task $t$ is also allocated to the structure,
$s_{i}\rightarrow s^{t}_{i}$. In this way, the meta-structure $s^{t}_{i}$
always acts as the feature anchor for task feature $t$ regardless of on which
nodes $i$, thus ensuring the feature alignment within structure $s^{t}$ among
all nodes:

$\Omega_{t}\leftarrow
s^{t}_{i}\approx...s^{t}_{j}\approx...s^{t}_{N},\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \forall\leavevmode\nobreak\ i,j\in
N.$ (4)

Meanwhile, even though different nodes can have varied structure orders, their
local structure $s^{t}_{i}$ can still be explictly mapped with the
$\Omega_{t}$, which is the $t$-th meta-structure in the global model structure
$\Omega$. With such a feature alignment property, we are then able to enforce
the structural feature alignment and achieve even heterogeneous model
collaboration, as we will show later.

Figure 2: Our proposed heterogeneous learning framework includes two major
steps: (I) We utilize group-convolution based $\Psi$-net structure as a
regulation for feature allocation. (II) Then we propose the heterogeneous
$\Psi$-net FL policy by grouped-shared parameter averaging to handle both IID
and non-IID data distributions.

###  3.2 $\Psi$-Net Structure Implementation

As shown in Fig. 2 (a), $\Psi$-Net structure regulation is composed of two
major parts to achieve grouped-based feature allocation: Non-grouped shared
layers (lower convolutional layers) and grouped layers (higher convolutional
and fully-connected layers).

#### Shared Layers.

The design of shared layers is due to the shallow layers learn mostly basic
features that are useful for most tasks and show less feature divergence. In
such cases, grouping these layers can prevent neurons in one group from
receiving gradients from other groups, leading to sub-optimal learning
performance. Therefore, the shallow layers are reserved shared without
grouping.

To determine an appropriate number of shared layers to reserve, we evaluate
the feature divergence of layer $l$ based on the total variance ($TV$) of
neurons’ feature encoding:

$TV_{l}=\sum\nolimits_{i}^{L}\frac{1}{L}||P_{l,i}-E(P_{l,i})||_{2},\leavevmode\nobreak\
\leavevmode\nobreak\ \text{where}\leavevmode\nobreak\ \leavevmode\nobreak\
P_{l,i}=[\leavevmode\nobreak\
p_{l,1},...,p_{l,c},...,p_{l,N}]/\sum\nolimits_{j}^{N}{p_{l,j}},$ (5)

where we use the neuron’s class preference vector $P_{l,i}$ in Eq. 2 as an
indication of its learned feature.

From shallow to deep layers, the layer-wise feature divergence can surge to
high points in some medium levels. We thus evaluate the feature divergence and
empirically determine a sharing depth to start the grouped layer construction.
In later experiments, we will evaluate our depth selection and show that our
$\Psi$-net structure is fairly robust to the depth hyper-parameter selection.

#### Group Convolutional Layers.

For the deeper convolutional layers, the encoded class feature are more
diverged and thus easier to conflict with each other during weight averaging.
Therefore, we utilize group convolution and construct separable group
convolutions. In each group, the convolutional neuron can only receive/output
the feature channels within the current group. Thus, the backpropagated
gradients carrying feature information can only flow through the individual
group, avoiding the feature interleaving between different groups.

Considering the practical training dataset complexity, the number of groups
cannot be always “one-to-one” matched to the training tasks. Therefore, on big
datasets (e.g., CIFAR100 with 100 classes), we also supports “one-to-many”
mapping, i.e., multiple class logits mapped to one convolutional group, so as
to achieve high scalability but without sacrificing the feature regulation
performance too much. As we will show later, both one-to-one mapping on small
datasets (CIFAR10) and one-to-many mapping on mediate datasets (CIFAR100) can
achieve as good structural feature regulation benefits.

#### Grouped Fully-Connected Layers.

For the fully-connected (FC) layers, the previous FC layers often connect all
input convolutional neurons to the class logits, and thus cannot regulating
which convolutional group learns the corresponding task features. In
$\Psi$-net, we also decouple the original FC layer into groups, i.e., multiple
linear layers with same number as the convolutional groups. One linear layer
connects the class logit(s) with the mapped convolutional groups only, thus
enforcing the gradients flowing backwards to the mapped group without any
leakage. Multi-layer FC layers can be decoupled similarly across layers, which
we omit for simplicity.

#### Group Normalization Layers.

Previous works have shown that batch normalization layers can influence the FL
performance as different local models tend to collect non-consistent batch
mean and variance statistics (especially in non-IID cases), thus degrading the
performance [4]. Our $\Psi$-net structure design, by enforcing the similar
feature encoding, alleviates the batch statistics divergence within each
group. Therefore, we propose to incorporate the Group Normalization (GN) layer
[14] into our $\Psi$-net structure to improve the model convergence
performance. Specifically, we add the GN layer after each grouped
convolutional layer, both with the same number of groups to ensure the batch
statistic normalization is conducted within each group. For the shared layers,
we can regard it as a layer with group number equal to 1, which deduces to the
normal batch normalization. In the later evaluation, we will demonstrate the
effectiveness of GN in our $\Psi$-net structure.

Overall, by our $\Psi$-net structure, we can isolate different convolutional
groups, so as the task features between them. Each conv/fc group only learns
the specified task features from the assigned class logits by gradient back-
propagation, enabling a pre-aligned feature allocation during local training.
Also, $\Psi$-net regulation is highly configurable and supports dynamic
structure-feature mapping, which provides high flexibility in handling
training datasets with different complexity.

##  4 $\Psi$-Net Collaboration for Heterogeneous Federated Learning

In this section, we present the particular FL collaboration framework based on
the $\Psi$-Net structure regulation through different training stages. To
demonstrate the framework’s applicability for various heterogeneous settings,
we target both IID and non-IID data distribution scenarios. Especially, we
also take an extreme non-IID case into consideration: certain classes’ data
volume might be too small or missing on particular nodes, resulting in
asymmetric learning tasks across local models.

#### Global & Local Model Initialization.

Regarding the heterogeneity across collaborative nodes, the $\Psi$-Net based
global and local models might be initialized differently for training scenario
adaptation.

Global Model: Given a global task with multiple learning classes and a target
neural network architecture to transform (e.g., VGG16), the proposed
regulation method will first identify the depth boundary between shared and
grouped layers based on Eq. 5. Regardless of IID or non-IID scenarios, the
number of separated layer groups will be determined with the mapping schemes
described in Sec. 3 to guarantee the expected granularity and scalability with
different learning class volumes.

Local Model: In IID and general non-IID data distribution cases, local models
share the homogeneous structure with the global model. While in the extreme
non-IID cases, where particular nodes are unable to learn certain classes,
heterogeneous models will be initialized for the asymmetric learning tasks.
From the structure-information alignment perspective, the structure groups
corresponding to unlearnable classes will be trimmed, as they cannot learn
useful information to contribute.

Such a trimming process is implemented differently following the mapping
scheme adopted by the global model: With the “one-to-one” mapping scheme, any
structure group without local task data being present would be trimmed. While
with “one-to-many”, the trimming of one structure group is conducted only if
all the mapped tasks $M(s_{c})$ are not present in node $i$’s local data
$LD(i)$:

$s_{c}=\textbf{1},\leavevmode\nobreak\ \leavevmode\nobreak\
\text{iff}\leavevmode\nobreak\ \leavevmode\nobreak\ M(s_{c})\cap
LD(i)=\emptyset\leavevmode\nobreak\ ,\leavevmode\nobreak\ \leavevmode\nobreak\
\text{otherwise}\leavevmode\nobreak\ \leavevmode\nobreak\ \textbf{0},$ (6)

where $M(s_{c})$ is the task set mapped to the current conv/fc group. Better
rules may be determined considering the amount of data or abandoning isolated
classes, which we leave as future work.

#### Local Model Training.

Local training is conducted epoch-wise as regular FL schemes. However,
different from SOTA, the structure-information alignment is continuously
guaranteed without considerable overheads, such as parameter sharing,
evaluation, and post-training alignment. Moreover, the nodes with asymmetric
tasks further trim local models, leading to less computation cost.

#### Aligned Model Averaging.

The averaging will be conducted periodically every $e$ epochs.

Shared layers will be synchronized by local models from all $N$ collaborative
nodes. As they extract fundamental features with less class-specific
divergence, an global averaging can be directly applied:

$\Omega_{shared}=Avg(\omega_{shared}^{i}),\leavevmode\nobreak\ \hskip
14.22636pti\in\\{1,\ldots,N\\},$ (7)

where $\omega_{shared}^{i}$ denotes the weights of the shared layers from the
$i$th node’s local model. Moreover, such uniformed shared layers also benefit
for global consensus under heterogeneous scenarios.

Grouped layers conduct model averaging only within matched groups, which are
supposed to have the same primary learning class to alleviate potential
feature conflicts:

$\Omega_{c}=Avg(\omega_{c}^{\mathbf{I}_{c}}),\leavevmode\nobreak\ \hskip
14.22636pt\text{where}\leavevmode\nobreak\ \hskip
5.69054pt\mathbf{I}_{c}=\\{i,\leavevmode\nobreak\
\text{s.t.}\leavevmode\nobreak\ c\in M(i)\\},$ (8)

where $\Omega_{c}$ denotes the averaged weights of the grouped layer
structures for the $c$th learning class, $\mathbf{I}_{c}$ indicates the
indices of collaborative nodes where class $c$ presents in their local data
sets. Ideally, all collaborative nodes should be included in $\mathbf{I}_{c}$.
However, when asymmetric learning tasks occur, $\mathbf{I}_{c}$ will exclude
certain unsupportive local models for class $c$.

Normalization layers, including batch and group normalization layer, are
averaged in a similar way: For batch normalization in shared layers, we
average the “shift” and “scale” parameters, as well as the the “running_mean”
and “running_variance” statistics for each layer. For group normalization in
the grouped layers, only the “shift” and “scale” parameters are averaged
within matched groups, as the other two are subject to the batch change.

#### Heterogeneous Model Collection.

Taking the aforementioned extreme non-IID cases into consideration, the final
global model can be considered as a collection of heterogeneous local
structures:

$\Omega^{Global}=\\{\Omega_{shared},\leavevmode\nobreak\
\Omega_{1},\ldots,\leavevmode\nobreak\ \Omega_{C}\\}.$ (9)

Overall, along with the feature allocated structure regulation, our framework
can avoid the collaborative conflicts and achieve continuously feature
alignment regulation during different FL stages. In next section, we conduct
systematical evaluation of our framework.

##  5 Experiment

#### Experiment Setup.

We evaluate our proposed framework with image classification tasks on CIFAR10
and CIFAR100. Three neural network architecture, namely VGG9, VGG16, and
MobileNetv1, are adopted to demonstrate the generality of our structure
regulation method. For local data distributions, we consider both IID and non-
IID scenarios, where the non-IID can have only partial classes of the dataset.
State-of-the-art works including FedMA [13] and FedProx [3] are compared to
demonstrate the efficiency and effectiveness of our proposed framework. Code
will be released.

###  5.1 Convergence Performance Improvement Evaluation

In this section, we first demonstrate our framework’s advanced performance in
terms of convergence accuracy and speed. As aforementioned, we adopt the “one-
to-one” mapping scheme to generate 10 structure groups with $\Phi$-Net on
CIFAR10 for fine-grained feature allocation. On CIFAR100, we also utilize the
10-group structure with the “one-to-many” mapping in order to achieve the
required scalability with increased dataset complexity.

#### Accuracy Improvement (Non-IID to IID).

We demonstrate our framework’s full-spectrum accuracy improvement in Table. 2.
The experimental setting $N*C$ indicates there are $N$ collaborative nodes,
and each node has $C$ classes present in the local data distribution. The
global dataset is split randomly and equally onto different nodes without any
duplication. The results reveal that our framework consistently outperforms
the baseline – FedAvg by +1%$\sim$4% accuracy on VGG9. FedAvg on MobileNetV1
seems to suffer more from the highly-skewed non-IID data distribution. As a
result, our framework could achieve +6%$\sim$19% accuracy on MobileNetV1.

#### Accuracy Improvement (Node Scalability).

Similar experiments are conducted to evaluate our framework’s generality with
node scaling. Specifically, we scale the number of collaborative nodes from 10
to 100 with the non-IID data distribution (each node only have 5 classes
present in the local data distribution). Without loss of generality, our
framework provides consistently better performance ranging from +2%$\sim$4% on
VGG9 and +5%$\sim$11% on MobileNetV1.

The underlying reasons of our accuracy improvement is due to the structurally
aligned feature distribution across local models, as demonstrated in Fig. 1.
Such feature alignment alleviates the parameter averaging conflicts and
providing higher convergence accuracy. This also benefits the convergence
speed as discussed in the next section.

#### Convergence Speed Improvement.

Four settings are selected similar with CIFAR10 non-IID spectrum settings,
e.g., 10 nodes with each node having partial classes presented. Fig. 3 shows
the test accuracy curves of FedAvg and ours through the entire training
process. In all the non-IID cases, our method consistently shows higher
convergence speed with only 50-80 rounds for the optimal performance, while
FedAvg usually needs at least 100 rounds. One exception is the 10x100 IID
setting, the initial convergence of FedAvg seems to be faster, but our method
soon exceeds it after 50 epochs and finally achieves +4% accuracy than FedAvg.

Table 1: Non-IID Spectrum (CIFAR10). | N * C | 10x3 | 10x4 | 10x5 | 10x10
---|---|---|---|---|---
VGG9 | FedAvg | 82% | 84% | 85% | 88%
Ours | 83% | 88% | 88% | 90%
MbNet | FedAvg | 67% | 71% | 79% | 85%
Ours | 86% | 88% | 90% | 91%

Table 2: Node Scalability (CIFAR10). | N * C | 10x5 | 20x5 | 50x5 | 100x5
---|---|---|---|---|---
VGG9 | FedAvg | 85% | 86% | 83% | 83%
Ours | 88% | 88% | 86% | 87%
MbNet | FedAvg | 79% | 85% | 81% | 78%
Ours | 90% | 90% | 89% | 88%

Figure 3: Convergence Speed Comparison between FedAvg and Proposed Method
(VGG16 on CIFAR100). Figure 4: Local Epoch Influence and Local Training
Budgets Comparison with SOTA.

###  5.2 Performance Comparison with SOTA (FedProx & FedMA)

In this part, we conduct performance comparison between our work with SOTA
methods including FedProx and FedMA. We compare our performance in two
aspects: (1) convergence accuracy under different local training epoch numbers
$e$; (2) the optimal performance under given local training budgets. The
overall results are shown in Fig. 4, where the SOTA results are referred from
[13]. All experiments use the same setting (VGG9, 16 nodes) with the dirichlet
data distribution of CIFAR10.

#### Communication Efficiency with Different Local Epochs.

As previous work studied, with higher local training epochs, the frequency of
the communication in FL could be effectively reduced. However, longer local
training epochs can also lead to sub-optimal performance, since the model
collaboration are less frequent with potential model divergence. Therefore, we
aim to achieve high convergence accuracy with as longer local epochs as
possible. Fig. 4 (a) shows the convergence accuracy performance comparison
under local epoch settings {10, 20, 50, 70, 100, 150}. All models are trained
with 54 averaging rounds as per settings in [13] for fair comparison. Clearly,
our framework shows the best performance under all settings, improving FedMA
by +3%$\sim$5% accuracy.

#### Computational Efficiency with Local Training Budgets.

Specifically, the results of baseline FedAvg and SOTA works are their
corresponding optimal performance reported in [13]. As we can see that, our
method could achieve +0.6%$\sim$2.4% accuracy than FedMA with similar or less
local training budgets. For FedAvg and FedProx, our method could achieve +3.0%
and +4.0% accuracy improvement with the same training budget, respectively.

###  5.3 Ablation Study of Design Components

We study the performance influence of three design strategies in our
framework, including the normalization layer influence, grouping number
selection, as well as the sharing layer depth selection.

#### Normalization Strategy Analysis.

For normalization strategy analysis, we compare our work and FedAvg under the
data distribution of (VGG9, CIFAR10, 10x4 non-IID). The results are shown in
Fig. 6. The baseline FedAvg without normalization achieves 84.13% accuracy.
FedAvg+BN helps improve the accuracy to 85.46%, while FedAvg+GN hurts the
model performance, degrading the accuracy to 83.34%. Utilizing the same GN
settings, our work instead achieves the best accuracy 88.26%. This further
demonstrates that our grouped structure helps alleviate the statistics
divergence within groups, thus the group normalization could benefit the model
performance.

#### Grouping Number Analysis.

In this part, the performance robustness of our model is demonstrated with
complementary benefits under different group number selection. The overall
results are shown in Fig. 6 (VGG16, CIFAR100, 10x50 non-IID). Our 10-group and
20-group structures both achieve the optimal performance around $\sim$68%,
providing +2.7% accuracy improvment than FedAvg – 65.3% with the original
structure. The 100-group structure, though achieving sub-optimal accuracy
improvement (+1.9% than FedAvg), shows the highest convergence speed (the
green curve). We hypothesize the convergence speedup is brought by its most
fine-grained feature alignment benefits (1 class mapped to 1 group). But
oppositely, with 100 groups, the per-path capacity can be limited. Though sub-
optimal accuracy, it verifies the group size trade-off as discussed in Sec. 3.

#### Grouping Depth Analysis.

We also demonstrate that our framework’s performance is robust to the grouping
depth hyper-parameter selection in Figure 7. Specifically, our model shows
nearly optimal performance 89%$\sim$90% in large ranges of depth selection,
e.g., from C3-2 to C5-2 on CIFAR10. Also, the total variance of the model (a
pre-trained model with minimum 50 epoch pre-training) offers good indication
of the layer-wise feature divergence.

Figure 5: Normalization Strategy Analysis.

Figure 6: Number of Groups Analysis.

[table]  VGG16 C1-2 C2-2 C3-2 C4-2 C5-2 FC None CIFAR10 82% 87% 89% 90% 89%
87% 86% CIFAR100 53% 62% 67% 68% 69% 68% 65%

Figure 7: Grouping Depth Sensitivity. Left: Total Variance of each layer.
Right: Performance under different grouping depth, showing our models’
performance is robust to depth selection (10x5, 10x50 non-IID).

##  6 Conclusion

In this paper, we proposed a novel federated learning framework to resolve FL
models’ chaotic fusion problem by establishing a firm structure-information
alignment. In the process of proposing this method, we adopted feature-
oriented parameter interpretation and proposed a novel feature-oriented
regulation method ($\Psi$-Net) to ensure explicit feature information
allocation in different neural network structures. Such a regulation method
adjusts the local model architecture according to their data and task
distribution at the very early training stage, and continuously maintain not
only structure but also information alignment for better FL performance. To
our best knowledge, this is the very first work that solves the FL alignment
problem directly from a model regulation perspective, which significantly
improves the FL performance and applicability in various heterogeneous
settings.

## References

  * [1] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In Advances in neural information processing systems, pages 1097–1105, 2012.
  * [2] M. I. Leontev, V. Islenteva, and S. V. Sukhov.  Non-iterative knowledge fusion in deep convolutional neural networks.  Neural Processing Letters, 51(1):1–22, 2020.
  * [3] T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith.  Federated optimization in heterogeneous networks.  In Proceedings of the 3rd MLSys Conference, 2018.
  * [4] X. Li, K. Huang, W. Yang, S. Wang, and Z. Zhang.  On the convergence of fedavg on non-iid data.  In Proceedings of the 8th International Conference on Learning Representations (ICLR), 2019.
  * [5] Z. Liu, M. Sun, T. Zhou, G. Huang, and T. Darrell.  Rethinking the value of network pruning.  In Proceedings of the 7th International Conference on Learning Representations (ICLR), 2019.
  * [6] H Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, et al.  Communication-efficient learning of deep networks from decentralized data.  arXiv preprint arXiv:1602.05629, 2016.
  * [7] H. Mostafa.  Robust federated learning through representation matching and adaptive hyper-parameters.  arXiv preprint:1912.13075, 2019.
  * [8] Z. Qin, F. Yu, C. Liu, and X. Chen.  How convolutional neural network see the world-a survey of convolutional neural network visualization methods.  arXiv preprint:1804.11191, 2018.
  * [9] Z. Qin, F. Yu, C. Liu, and X. Chen.  Functionality-oriented convolutional filter pruning.  In Proceedings of 30th British Machine Vision Conference (BMVC), 2019.
  * [10] W. Samek, A. Binder, G. Montavon, S. Lapuschkin, and K. Müller.  Evaluating the visualization of what a deep neural network has learned.  IEEE transactions on neural networks and learning systems, 28(11):2660–2673, 2016.
  * [11] S. P. Singh and M. Jaggi.  Model fusion via optimal transport.  arXiv preprint:1910.05653, 2019.
  * [12] J. Smith and M. Gashler.  An investigation of how neural networks learn from the experiences of peers through periodic weight averaging.  In Proceedings of the 16th IEEE International Conference on Machine Learning and Applications (ICMLA), pages 731–736. IEEE, 2017.
  * [13] H. Wang, M. Yurochkin, Y. Sun, D. Papailiopoulos, and Y Khazaeni.  Federated learning with matched averaging.  In Proceedings of the 8th International Conference on Learning Representations (ICLR), 2020.
  * [14] Yuxin Wu and Kaiming He.  Group normalization.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 3–19, 2018.
  * [15] M. Yurochkin, M. Agarwal, S. Ghosh, K. Greenewald, N. Hoang, and Y. Khazaeni.  Probabilistic federated neural matching.  2018\.
  * [16] M. Yurochkin, M. Agarwal, S. Ghosh, K. Greenewald, T. N. Hoang, and Y. Khazaeni.  Bayesian nonparametric federated learning of neural networks.  In Proceedings of the 36th International Conference on Machine Learning (ICML), pages 7252–7261, 2019.
  * [17] Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun.  Shufflenet: An extremely efficient convolutional neural network for mobile devices.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6848–6856, 2018.
  * [18] Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra.  Federated learning with non-iid data.  arXiv preprint arXiv:1806.00582, 2018.
  * [19] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba.  Learning deep features for discriminative localization.  In Proceedings of Computer Vision and Pattern Recognition (CVPR), pages 2921–2929, 2016.


# Standardisation of Data Formats in Gamma-ray astronomy

C. Nigro1 and T. Hassan2
on behalf of the Data Formats for Gamma-ray Astronomy forum contributors

###### Abstract

The operation of the future Cherenkov Telescope Array (CTA), the next
generation of Imaging Atmospheric Cherenkov Telescopes (IACTs), as an Open
Observatory requires data products and analysis tools to be accessible and
usable by the scientific community. This requirement has impelled gamma-ray
astronomers to develop open-source science tools and standardised data
formats. The objective of this presentation is to provide a perspective on the
current effort to define a prototypical format for high-level gamma-ray
astronomical data. The initiative, started by the IACT community in 2016, has
gained in the years 2018 and 2019 full recognition thanks to the first public
data releases in such format by current-generation IACTs, followed by a series
of papers employing it either to test the capabilities of open-source gamma-
ray analysis tools or to showcase examples of reproducible, multi-instrument
analysis.

1Institut de Física d’Altes Energies (IFAE), Barcelona, Spain;
<EMAIL_ADDRESS>
2Centro de Investigaciones Energéticas, Medioambientales y Tecnológicas
(CIEMAT), Madrid, Spain<EMAIL_ADDRESS>
##  1 Introduction

With the noticeable exception of Fermi-LAT, current instruments for gamma-ray
astronomy rely on independent proprietary software and data for their
scientific analyses. These technical differences not only prevent multi-
instrument analyses to be performed within a common framework, but also impede
any legacy data products to be disseminated without releasing the
corresponding analysis software. Furthermore, the next instrument for gamma-
ray astronomy at Very High Energies (VHE, $E>100\,{\rm GeV}$), CTA (CTA
Consortium 2019), will be the first IACT to be operated as an Open
Observatory. The requirement on its analysis tools and data to be accessible
and usable to the astronomical community has led to the development of open-
source science tools, ctools (Knödlseder et al. 2016) and gammapy (Deil et al.
2017a), and to the definition of a preliminary standardised data format for
gamma-ray data (Deil et al. 2017b). This initiative, focus of our review, will
be described in Section 2. In Section 3 we will instead describe the projects
that implemented such data standard and used it either for the purpose of
validating the above mentioned science tools or to illustrate the ease of
multi-instrument analysis once data standardisation has been achieved.

##  2 The Data Formats for Gamma-ray Astronomy Forum

The discussion around a common format for high-level gamma-ray astronomical
data was started in early 2016 and has been channeled in the so-called Data
Formats for Gamma-ray Astronomy forum (Deil et al. 2017b). The data level
chosen to be standardised is the so-called Data Level 3 (DL3, Contreras et al.
2015) which encapsulates lists of events classified as $\gamma$ rays and the
Instrument Response Function (IRF) in the data units of a FITS file (Wells et
al. 1981). The specifications of the format, establishing the content and the
naming scheme of the FITS files are written in the form of a sphinx
documentation hosted on github111https://github.com/open-gamma-ray-
astro/gamma-astro-data-formats and built via readthedocs222https://gamma-
astro-data-formats.readthedocs.io/en/latest/, as shown in Figure 1. The open
access to the format allows both gamma-ray astronomers producing the actual
data and science tools developers to modify or expand it following the github
workflow.

Figure 1.: github repository (left) and online documentation (right) hosting
the Data Formats for Gamma-ray Astronomy specifications.

##  3 Projects using the Data Level 3 Format

Though initiated with the objective of CTA, the DL3 format can accommodate the
data of current-generation IACTs. In what follows we present some projects
already implementing and testing DL3 data products and open-source science
tools.

###  3.1 The H.E.S.S. First Public Test Data Release

The H.E.S.S. collaboration was the first to make publicly available a small
sample of its earliest observations (2004-2006) in DL3 format (H. E. S. S.
Collaboration 2018)333https://www.mpi-hd.mpg.de/hfm/HESS/pages/dl3-dr1/. The
release contains $30\,{\rm h}$ of observations of sources representing
different science cases (Crab Nebula, PKS 2155-304, MSH 15-52 and RX
J1713.7-3946) and $20\,{\rm h}$ of observations of sky patches devoid of
gamma-ray sources, usable to build a background model (off-source
observations). The aim of this release is to advertise the endeavour on the
data standardisation and to allow for a double feedback: both on the side of
the data format and on the science tools that will employ it.

###  3.2 Towards open and reproducible multi-instrument analysis in gamma-ray
astronomy

Figure 2.: Fit of the Crab Nebula spectrum obtained using the individual
datasets in Nigro et al. (2019) and combining them in a single likelihood
(red).

The project, described in Nigro et al. (2019), provides the first example of
fully-reproducible, multi-instrument analysis of gamma-ray data. Observations
of the Crab Nebula by Fermi-LAT and by four of the operating IACTs (MAGIC,
VERITAS, FACT and H.E.S.S.444The H.E.S.S. dataset is a taken from the H.E.S.S.
public data release.) converted into DL3 format, are analysed with gammapy. A
one-dimensional binned likelihood analysis (obtaining photon counts versus
energy via aperture photometry) is performed to estimate the spectrum of the
Crab Nebula using all the datasets jointly (Figure 2). A possible modification
of the likelihood to account for the systematic uncertainties in the energy
scales of the different instruments is also discussed. The project constitute
the first joint data release by IACTs. Full reproducibility is ensured by
providing all the data, analysis scripts and tutorial notebooks publicly on
github555https://github.com/open-gamma-ray-astro/joint-crab along with a conda
environment installing all the software dependencies. An ancillary docker
container666https://hub.docker.com/r/gammapy/joint-crab guarantees a
reproducibility longer than the life span of the software used.

###  3.3 Analysis of the H.E.S.S. public data release with ctools

The H.E.S.S. DL3 data release was analysed with ctools in Knödlseder et al.
(2019). The science tool was thus far tested only with CTA simulations and the
publication represents the first test of its capabilities with actual IACT
data. A recipe is presented to build a background model starting from the Off
observations, the latter is used for spectral (and morphological - where
possible) analysis of the 4 sources included in the H.E.S.S. DL3 data release.
Results of binned and unbinned three-dimensional likelihoods (accounting at
once for sources position, morphology and spectra) are compared against the
ones of one-dimensional binned likelihoods. The internal consistency of the
ctools results obtained with the different likelihood methods and their
agreement with the results from literature extracted from larger datasets of
the same sources show the maturity of the analysis tool (Figure 3). The paper
also illustrates the capability of ctools to simultaneously analyse Fermi-LAT
and IACT data within the same likelihood framework.

Figure 3.: Validation of science tools on the extended source RX J1713.7-3946.
Left: comparison of the result of the ctools unbinned analysis and background
model in Knödlseder et al. (2019) against the literature. Right: comparison of
the ctools unbinned and gammapy binned analysis using the background model in
Mohrmann et al. (2019) against the H.E.S.S. propiretary software and the
literature.

###  3.4 Validation of Open-source Science Tools and Background Model
Construction in $\gamma$-ray Astronomy

Mohrmann et al. (2019) aim at validating both ctools and gammapy by comparing
their results when analysing the H.E.S.S. DL3 data release. Here another
procedure for the construction of a background model is illustrated, this time
though relying on a large private dataset of H.E.S.S. off-source observations
($\sim 4000\,{\rm h}$). The results of three-dimensional and one-dimensional
analyses by the two science tools are compared against the results obtained
with one of the H.E.S.S. proprietary analysis chain on the same datasets and
against the literature. For both likelihood methods an excellent agreement is
found with the results of the H.E.S.S. analysis software.

##  4 Conclusion

The specifications of the Data Formats for Gamma-ray astronomy forum have been
recently employed to produce the first public IACT data release and to
validate the analysis methods of the science tools currently in development.
DL3 data constitutes the first example of multi-instrument gamma-ray data
analysed with a fully-reproducible pipeline. The VHE community has been
building crucial expertise in data standardisation that will be essential in
the future operation of CTA.

## References

  * Contreras et al. (2015) Contreras, J. L., et al. 2015, in 34th International Cosmic Ray Conference (ICRC2015), vol. 34 of International Cosmic Ray Conference, 960. 1508.07584
  * CTA Consortium (2019) CTA Consortium 2019, Science with the Cherenkov Telescope Array (World Scientific Publishing Co. Pte. Ltd.)
  * Deil et al. (2017a) Deil, C., et al. 2017a, in 35th International Cosmic Ray Conference (ICRC2017), vol. 301 of International Cosmic Ray Conference, 766. 1709.01751
  * Deil et al. (2017b) — 2017b, in 6th International Symposium on High Energy Gamma-Ray Astronomy, vol. 1792 of American Institute of Physics Conference Series, 070006\. 1610.01884
  * H. E. S. S. Collaboration (2018) H. E. S. S. Collaboration 2018, arXiv e-prints, arXiv:1810.04516. 1810.04516
  * Knödlseder et al. (2016) Knödlseder, J., et al. 2016, A&A, 593, A1. 1606.00393
  * Knödlseder et al. (2019) — 2019, A&A, 632, A102. 1910.09456
  * Mohrmann et al. (2019) Mohrmann, L., et al. 2019, A&A, 632, A72. 1910.08088
  * Nigro et al. (2019) Nigro, C., et al. 2019, A&A, 625, A10. 1903.06621
  * Wells et al. (1981) Wells, D. C., et al. 1981, A&AS, 44, 363


# In-the-wild Speech Emotion Conversion Using Disentangled Self-Supervised
Representations and Neural Vocoder-based Resynthesis

Navin Raj Prabhu⋆†, Nale Lehmann-Willenbrock†, Timo Gerkmann⋆

###### Abstract

Speech emotion conversion aims to convert the expressed emotion of a spoken
utterance to a target emotion while preserving the lexical information and the
speaker’s identity. In this work, we specifically focus on in-the-wild emotion
conversion where parallel data does not exist, and the problem of
disentangling lexical, speaker, and emotion information arises. In this paper,
we introduce a methodology that uses self-supervised networks to disentangle
the lexical, speaker, and emotional content of the utterance, and subsequently
uses a HiFiGAN vocoder to resynthesise the disentangled representations to a
speech signal of the targeted emotion. For better representation and to
achieve emotion intensity control, we specifically focus on the arousal
dimension of continuous representations, as opposed to performing emotion
conversion on categorical representations. We test our methodology on the
large in-the-wild MSP-Podcast dataset. Results reveal that the proposed
approach is aptly conditioned on the emotional content of input speech and is
capable of synthesising natural-sounding speech for a target emotion. Results
further reveal that the methodology better synthesises speech for mid-scale
arousal (2 to 6) than for extreme arousal (1 and 7).

##  1 Introduction

One way in which emotions are expressed by individuals in social interactions
is via speech signals [1]. In the context of human-machine interaction
systems, the generation of spoken dialogue is a fundamental facet of natural
interaction between humans and machines [2, 3]. More importantly, to improve
the naturalness of machine communication, the generation of emotionally
expressive speech is required. While speech generation technologies have been
making significant progress [4, 5], emotionally expressive speech generation
is still a challenge [6, 7]. Speech emotion conversion (SEC) is a technique
that aims to convert the expressed emotion of a spoken utterance to a target
emotion while preserving the lexical and the speaker information [8, 6].
Therefore, SEC has a crucial application in building next-generation human-
machine interaction systems, aiming at equipping them with the ability to
interact with social and emotional intelligence.

Emotion can be represented either categorically or as dimensional
representations. As categorical representations, Ekman’s six basic emotions
[9] (e.g., anger, happy) are commonly used [10]. However, emotion is a fuzzy
construct with fuzzy class boundaries [11], and such discrete representations
do not aptly capture the subtle difference between human emotions [11]. To
overcome this, the circumplex model [11] captures emotional expressions using
two continuous and independent dimensions, i.e., _arousal_ (relaxed or passive
vs. aroused or activated) and _valence_ (positive vs. negative) [12, 13]. In
SEC research, efforts have also been made to control the intensity of
categorical emotion representations, e.g., using mixed emotion representations
[3] or modeling emotion intensity as an auxiliary task [14]. Note that SEC
using the dimensional representations directly archives intensity control, as
opposed to an additional effort in the categorical representation case.

Figure 1: Illustration of the training process. During inference, $e$ is
replaced by the target emotion $\bar{e}$. Orange blocks denote trainable
blocks, and blue blocks are pretrained.

SEC datasets are primarily acted-out [10, 15, 7, 16], as opposed to in-the-
wild improvised datasets [12]. Acted-out databases make a strong assumption on
the availability of parallel utterances [6], i.e., one where each source
utterance also has a ground-truth utterance of a target emotion, which is
resource inefficient to collect [17, 18] and techniques relying on them lack
scalability [6].

In this work, we specifically focus on non-parallel data. Models trained on
non-parallel data are scalable to different emotion types [6], as they are not
restricted by the emotion pairs trained on. However, they are also more
challenging than modeling parallel data, as the problem of _disentanglement_
arises [6, 8]. For SEC on non-parallel data, a disentanglement method is
required to decompose the input speech signal into several constituents:
emotion, lexical, and speaker information, encoded in respective latent
representations. During inference, the latent representation encoding emotion
is modified to achieve emotion conversion. Existing works have primarily
employed variational auto-encoders (VAE) [19, 15] and sequence-to-sequence
encoder-decoders [14, 3] to achieve disentanglement.

Self-supervised learning (SSL) techniques that leverage large unlabeled
datasets have shown to be promising in several tasks, most importantly its
state-of-the-art performance in speech emotion recognition (SER) [20, 21] and
speaker voice conversion [22]. Motivated by this, in this paper, we propose a
novel methodology that employs SSL models to disentangle emotion, lexical, and
speaker identity representations, and uses a HiFiGAN [4] to resynthesise the
disentangled representations into emotional speech. For a better
representation of emotion and better intensity control in emotion conversion,
in this paper, we will represent emotion using the continuous arousal
dimension. Moreover, we train and validate the proposed methodology on the in-
the-wild MSP-Podcast v1.10 dataset [12]. To the best of our knowledge, this
work is the first in the literature to perform SEC on an in-the-wild setting
and on arousal, while existing literature has primarily focused on categorical
emotion (e.g., anger, sad, neutral, happy) [15, 17, 18] and acted-out datasets
(e.g., IEMOCAP, ESD) [10, 7].

##  2 Background

###  2.1 Disentangled SSL representations

SSL techniques have shown great success in several downstream tasks such as
automatic speech recognition [23], phoneme segmentation [24], speaker
verification [25], and SER [20, 21]. Existing literature well documents that
representations obtained from SSL models finetuned on a specific task have
exclusive information on that particular task. Analysis presented in [26]
reveals that discrete representations obtained from HuBERT contain exclusively
phoneme and lexical information, without any speaker or F0 information. In
[25], the HuBERT framework was used to build the WavLM which was then fine-
tuned exclusively to preserve speaker identity [25]. Motivated by this, in
this paper, we use SSL networks to disentangle lexical and speaker information
from input speech.

###  2.2 Speech resynthesis

While disentanglement is only the first step of SEC, the subsequent step is to
synthesise the disentangled SSL representations into natural-sounding speech
that is also conditioned on the emotion content. Traditionally, neural network
based vocoders conditioned on the log Mel-spectogram enabled generating
natural-sounding speech [27]. More recently, techniques have been proposed to
directly synthesise natural-sounding speech from SSL-based speech
representations [22, 5, 24, 26], the technique termed as resynthesis. In [22],
HiFiGAN was used to resynthesise speech from disentangled representations
obtained from three SSL models: HuBERT for lexical content, VQ-VAE for F0
information, and a speaker verification model. In this paper, we follow a
similar approach, that is to perform resynthesis by training a HiFiGAN on
disentangled SSL representations. The crucial difference between our work and
[22] is that we employ this technique in SEC, thereby also requiring us to
condition the HiFiGAN on emotion information.

##  3 Proposed Methodology

The task of SEC is formulated as follows: given a single-channel audio input
of a spoken utterance $\mathbf{X}_{l,s,e}\in\mathbb{R}^{1\times T}$, with
lexical content $l$, speaker identity $s$, and annotated arousal score $e$,
with its raw signal represented as a sequence of samples
$x=(x_{1},...,x_{T})$, we design a system to synthesise
$\widehat{\mathbf{Y}}_{l,s,\bar{e}}\in\mathbb{R}^{1\times T}$, a single-
channel audio that preserves the lexical content $l$ and speaker identity $s$
of $\mathbf{X}_{l,s,e}$, and converts the arousal of $\mathbf{X}_{l,s,e}$ to
target arousal $\bar{e}$. To this, we introduce an SSL-based HiFiGAN network
(see Fig. 1) that consists of (i) _SSL encoders_ that disentangles lexical
$l$, speaker $s$, and emotion $e$ information, and (ii) a _HiFiGAN decoder_
that is trained to resynthesise emotion converted speech
$\widehat{\mathbf{Y}}_{l,s,\bar{e}}$ from the disentangled representations.
The network does not rely on parallel data, where the ground-truth
${\mathbf{Y}}_{l,s,\bar{e}}$ exists, and is generative in nature.

###  3.1 Disentanglement encoders

Lexical Encoder: The input to the lexical encoder $E_{l}$ is the time-domain
signal $x$, and its output is a sequence of low frequency representations
$E_{l}({x})=(l_{1},...,l_{T^{\prime}})$. As $E_{l}$, we choose the pretrained
SSL-based HuBERT model. Following [22], using the $k$-means algorithm, we
convert the continuous HuBERT representations into discrete representations
denoted as $z_{l}=(z_{1},..\\-.,z_{T^{\prime}})$, where each unit $z_{i}$ is a
positive integer and $z_{i}\in{0,1,..,K}$. Existing works [22, 5], have shown
such discrete representations $z_{i}$ to be more related to the phonemes of
the respective utterance.
Speaker Encoder: Similar to $E_{l}$, the input to the speaker encoder $E_{s}$
is the time-domain signal $x$. As $E_{l}$, the pretrained WavLM speaker
verification model [25] is used. The output of $E_{s}$ is a continuous
$d-vector$ speaker representation $z_{s}\in\mathbb{R}^{512}$. Unlike the
lexical representation $z_{l}$, $z_{s}$ is a global representation that
encodes speaker information of the whole utterance. To account for the
mismatch in frequency, $z_{s}$ is concatenated on all $T^{\prime}$ frames of
$z_{l}$ for the resynthesis and is denoted as $z_{T^{\prime}}=(z_{l},z_{s})$.
Emotion encoder: To encode emotional information in the disentanglement phase,
as the emotion encoder $E_{e}$ we use simple trainable linear layers. The
input to the encoder $E_{e}$ is the ground-truth annotated arousal scalar
label $e\in\mathbb{R}$ and $1\leq e\leq 7$, and the output is the emotion
representation $z_{e}\in\mathbb{R}^{128}$. Similar to $z_{s}$, $z_{e}$ is also
a global representation that encodes speaker information of the whole
utterance $x$, and similarly $z_{e}$ is also concatenated on all frames of
$z_{l}$ for the resynthesis and is denoted as
$z_{T^{\prime}}=(z_{l},z_{s},z_{e})$.

###  3.2 Resynthesis decoder

Inspired by existing works on voice conversion [22, 5] and SEC [24], we adopt
a modified version of the HiFiGAN [4] to resynthesise the disentangled encoder
representations. Note that [24] is the closest to our work. However, in this
work, we extend the HiFiGAN to non-parallel data and also tackle the thereby
arising problem of disentanglement.

The HiFiGAN is trained using a generator $G$ and two discriminator networks,
the multi-period and the multi-scale discriminators, $D_{p}$ and $D_{s}$,
respectively [4, 22]. The generator $G$ has a series of blocks composed of a
transposed convolution and a dilated residual layer (see Fig. 1). The
transposed convolutions upsample the representations to match the input number
of samples $T$. The dilated residual layers increase the receptive field. The
input to $G$ is the concatenated disentangled representations
$z_{T^{\prime}}=(z_{l},z_{s},z_{e})$ and the output is the resynthesised
speech $\widehat{\mathbf{Y}}_{l,s,\bar{e}}$ conditioned on lexical, speaker
and emotion information.

The multi-period discriminator $D_{p}$ consists of six sub-discriminators each
operating on different period hops of the input and generated speech: 2, 3, 4,
5, 7, and 11. Similarly, the multi-scale discriminator $D_{s}$ uses three sub-
discriminators operating at different scales: the original scale, x2
downsampled scale, and x4 downsampled scale.

###  3.3 Loss functions

HiFiGAN is a generative network trained using adversarial learning strategy
[4]. Following [22], for resynthesised output speech signal
$\widehat{\mathbf{Y}}$ represented as $\hat{y}=(\hat{y}_{1},...,\hat{y}_{T})$,
each of the sub-discriminators $D_{j}$ in $D$ is trained to minimize the
following adversarial losses (${L}_{adv}$ and ${L}_{D}$):

$\displaystyle{L}_{adv}(D_{j},G)$
$\displaystyle=\sum_{x}||1-D_{j}(\hat{y})||^{2}_{2},$ (1)
$\displaystyle{L}_{D}(D_{j},G)$
$\displaystyle=\sum_{x}[||1-D_{j}({x})||^{2}_{2}+||D_{j}(\hat{x})||^{2}_{2}],$
(2)

where $x$ is the input speech and
$\hat{y}=G(z_{T^{\prime}})=G(z_{l},z_{s},z_{e})$.

Along with ${L}_{adv}$ and ${L}_{D}$, a reconstruction loss term ${L}_{recon}$
and a feature-matching loss ${L}_{fm}$ [28] is also added to the loss
function. ${L}_{recon}$ measures the Mel-spectrogram reconstruction between
input $x$ and resynthesised output $\hat{y}$:

$\displaystyle{L}_{recon}(G)=\sum_{x}||\phi(x)-\phi(\hat{y})||_{1},$ (3)

where $\phi$ is a function computing Mel-spectrogram. The ${L}_{fm}$ term is
the distance between the discriminator activations of input $x$ and
resynthesised output $\hat{y}$:

$\displaystyle{L}_{fm}(D_{j},G)=\sum_{x}\sum_{i=1}^{R}\dfrac{1}{M_{i}}||\psi_{i}(x)-\psi_{i}(\hat{y})||_{1},$
(4)

where $\psi_{i}$ is the function that extracts the activations of the $i$-th
discriminator layer, and, $M_{i}$ and $R$ are the number of features in $i$
and the number of layers in $D_{j}$, respectively.

While the concatenation of emotion label embeddings $z_{e}$ to the input of
generator $G$ already conditions the resynthesised output $\hat{y}$, we also
included an SER loss to the loss function. To achieve this, we use a pre-
trained SSL-based SER system $E_{SER}$ introduced in [21]. The $E_{SER}$
network was built by fine-tuning the Wav2Vec2-Large-Robust network [23] on the
MSP-Podcast (v1.7) dataset [12]. The SER loss term ${L}_{SER}$ is formulated
as,

$\displaystyle{L}_{SER}=\sum_{x}[1-{L}_{ccc}(e,E_{SER}(\hat{y}))],$ (5)

where ${L}_{ccc}$ is the concordance correlation coefficient (CCC) [29] that
measures similarity between two variables, $e$ is the ground-truth emotion of
input, and $E_{SER}(\hat{y})$ is the predicted emotion for resynthesised
speech $\hat{y}$. The ${L}_{ccc}$ measure varies between $-$1 and $+$1, where
$+$1 denotes perfect similarity, therefore $1-{L}_{ccc}$ is minimized during
training.

The final loss for generator $G$ and discriminator $D$ is:

$\displaystyle{L}_{G}(D,G)=$
$\displaystyle\sum_{j=1}^{J}[{L}_{adv}(D_{j},G)+\lambda_{fm}{L}_{fm}(D_{j},G)]$
$\displaystyle+\lambda_{r}{L}_{recon}(G)+\lambda_{SER}{L}_{SER},$ (6)
$\displaystyle{L}_{D}(D,G)=$ $\displaystyle\sum_{j=1}^{J}{L}_{D}(D_{j},G),$
(7)

where $J$ is the number of sub-discriminators in $D$. Following [22], we set
$\lambda_{fm}=2$ and $\lambda_{r}=45$. $\lambda_{SER}$ was set to 1 after
preliminary results supported this setup.

Model versions | WVMOS | SER Error
---|---|---
$L_{mse}$ | $L_{abs}$
$z_{l}$ | 1.80 | $-$ | $-$
$z_{l}+z_{s}$ | 2.29 | $-$ | $-$
$z_{l}+z_{s}+z_{e}$ | 2.64 | 0.0971 | 0.2642
$z_{l}+z_{s}+z_{e}+L_{SER}$ | 3.26∗ | 0.0843∗ | 0.2442∗
Table 1: Overall performance of model versions. * indicates statistically
significant improvements in results.

##  4 Experimental Setup

Dataset: The dataset used in this work is the _in-the-wild_ MSP-Podcast
dataset (v1.10) [12]. To the best of our knowledge, this is the first work in
literature that performs SEC on an in-the-wild dataset. The dataset consists
of approximately 166hrs of audio collected from podcasts. Emotion is labeled
at the utterance-level in terms of arousal, valence, and dominance. In this
work, we only use the arousal annotations for SEC. The arousal annotations,
collected on a scale of 1 to 7, are distributed with $\mu=$4 and
$\sigma=$0.95, denoting that there are comparatively fewer samples on the
extremes of the scales (1 and 7) than on middle regions (3 to 5). Note this
also reflects the nature of emotions in-the-wild podcasts and also in any
real-world scenario, where extreme emotions are expected to be sparse.
Validation measures: We validate the proposed methodology in terms of both the
SEC capabilities and the naturalness of synthesised speech. As the measure of
SEC capability, we use the mean-squared ${L}_{mse}$ and mean-absolute
${L}_{abs}$ errors, calculated between the target arousal $\bar{e}$ and the
SER prediction on the resynthesised output $E_{SER}(\hat{y})$. As the measure
of the naturalness of $\hat{y}$, we use the wav2vec mean-opinion score (WVMOS)
[30]. WVMOS is an objective speech quality measure based on wav2vec2.0 [23]
and is fine-tuned on the mean-opinion scores (on a scale of 1 to 5) obtained
from the listening tests of the 2018 Voice Conversion Challenge [31]. As the
listening tests primarily focused on the naturalness of the voice conversion
task, it makes the WVMOS well suited for non-intrusively measuring the
naturalness of $\hat{y}$. Note here that, as we do not use parallel data we
can only rely on non-intrusive measures that do not require the ground-truth
audio of emotion conversion $\mathbf{Y}_{l,s,\bar{e}}$. Statistical
significance for improved performance is estimated using one-tailed $t$-test
on error distributions, asserting significance for _p_ -values $\leq 0.05$.

##  5 Results and Discussion

Overall performance:  Firstly, we compare the overall performance of four
different versions of the proposed methodology: (i) $z_{l}$, which only uses
the lexical representations for resynthesis, (ii) $z_{l}+z_{s}$, which also
uses the speaker information $z_{s}$, (iii) $z_{l}+z_{s}+z_{e}$, which also
uses the emotion label embeddings $z_{e}$ thereby also conditioning on emotion
information, and (iv) $z_{l}+z_{s}+z_{e}+L_{SER}$ which further conditions on
emotion information by including $L_{SER}$ in the loss function. As this is
the first work in literature to perform SEC in terms of arousal, we do not
have any baselines, rather we present different versions of the proposed
methodology for comparison.

From the results presented in Table 1, we note the following: firstly, with
the increasing addition of disentangled representations the HiFiGAN is capable
of producing more natural-sounding speech. The usage of all three
representations along with the inclusion of SER loss
$z_{l}+z_{s}+z_{e}+L_{SER}$ achieves the best WVMOS score of $3.26$ with
statistical significance. This also validates the SEC capability of the
proposed approach, as conditioning on emotion should also improve the
naturalness of synthesised speech and it is reflected in the WVMOS score. In
[22], it is suggested to also include F0 representations to obtain better
natural-sounding speech. However, in our case, we do not use F0
representations for resynthesis as conditioning on F0 will also affect the
conditioning on emotion representations $z_{e}$, given that F0 and emotion are
highly correlated [32]. Secondly, including the SER loss term $L_{SER}$ for
further conditioning on emotional content results in an improved SEC
performance. The model achieves an $L_{mse}$ of $0.0843$ and $L_{abs}$ of
$0.2442$, improving over $z_{l}+z_{s}+z_{e}$ with statistical significance.
The SEC capabilities can be further noted in the audio examples presented
online111https://uhh.de/inf-sp-emovc-itg .

(a) $L_{mse}$ trends

(b) WVMOS trends

Figure 2: Class-wise performance for target arousal $\bar{e}$, with respect to
the input segment sizes.

Influence of segment size:  While training HiFiGAN, to account for varying
utterance lengths, segments of consistent lengths are randomly sampled from
different regions of the full utterance and then are used for training. In
[22], it is noted that the HiFiGAN is robust enough to resynthesise natural-
sounding speech even when trained on a small segment size (0.75s). However,
for modeling emotion the context is important [33], thereby requiring larger
segment sizes to aptly condition on the emotion content.

To investigate the influence of segment sizes on SEC, we trained the proposed
methodology on varying segment sizes, and the results are presented in Fig. 2.
From the results, it is noted that larger segment sizes improve SEC
performances, both in terms of $L_{mse}$ and WVMOS. While the improvements on
naturalness (WVMOS) are clearly noted, improvements on $L_{mse}$ are larger
for extreme emotion values (1 and 7) than mid-ranges of emotion (3 to 5).

Performance on different arousal classes:  Existing research has shown that
SEC techniques generally tend to perform well on certain emotion pairs than
others. For example, in [17], it is noted that the emotion pair of angry-sad
is easier to convert than the happy-angry pair. To investigate this, Fig. 2
also present performances with respect to each of the arousal classes ranging
from 1 to 7. In terms of the SEC capability, the $L_{mse}$ error is noted to
be higher for extreme arousal classes (1 and 7). However, for mid-scale
arousal values between 2 and 6 the $L_{mse}$ error is comparatively smaller.
In terms of the naturalness of synthesised speech, the WVMOS score is lower
for high arousal classes (6 and 7) and is comparatively better for other
arousal classes (1 to 5). This reveals that the proposed methodology is
capable of synthesising more natural-sounding speech when reducing the arousal
of the input speech, than when increasing the arousal. Overall, the results
reveal that the methodology better synthesises speech for mid-scale arousal
than for extreme arousal.

Figure 3: Sample log-energy spectrogram of emotion converted speech, along
with comparisons on pitch contours.

Qualitative analysis of spectrograms:  Sample spectrograms of emotion
converted speech can be seen in Fig. 3, for the ground-truth speech of arousal
$e=3.20$, for synthesised speech of reduced arousal $\bar{e}=1$, and for
increased arousal $\bar{e}=7$. Comparing the spectrograms of arousal $1$ and
arousal $7$, from the marked eclipses, we can observe that for increased
arousal of $7$ the spectrograms have larger magnitudes in the mid-frequencies.
This reveals that the model associates larger frequency magnitudes for high
arousal speech, than for low arousal speech. Along with the spectrograms, we
also plot the pitch contours of the respective speech signals. From the pitch
contours, it can be noted that the synthesised speech for high arousal
($\bar{e}=7$) has a higher mean and variability of pitch, than that of both
the ground-truth speech ($e=3.20$) and the synthesised speech for low arousal
($\bar{e}=1$). This aligns with existing literature that associates high
intensity of emotion with an increased mean pitch [14]. This difference in
pitch is also clearly notable in the audio examples available online1. These
results validate that the proposed model successfully performs SEC by aptly
conditioning on the emotion content.

##  6 Conclusion

In this paper, as the first in literature, we tackle the problem of in-the-
wild speech emotion conversion on continuous arousal representations, as
opposed to acted speech and categorical representations. In-the-wild datasets
lack parallel utterances and thereby the problem of disentangling the lexical,
speaker, and emotion information arises. To tackle this, we introduced a novel
methodology that uses SSL encoders for disentanglement, and a HiFiGAN vocoder
to resynthesise emotion conditioned speech from the disentangled SSL
representations. We validated the network on an in-the-wild dataset, in terms
of its emotion conversion capability, using a pretrained SER system, and the
naturalness of synthesised speech, using WVMOS. Results reveal that the
network is capable of synthesising natural-sounding speech with emotion
conversion. Further analysis revealed that the network better synthesises
emotional speech for mid-scale arousal than for extreme arousal. Finally, the
pitch contour analysis showed that the synthesised speech for high arousal has
a higher mean and variability of pitch than that of both the ground-truth
speech and the synthesised speech for low arousal.

## References

  * [1] B. W. Schuller, “Speech emotion recognition: Two decades in a nutshell, benchmarks, and ongoing trends,” Communications of the ACM, vol. 61, no. 5, pp. 90–99, 2018.
  * [2] J. Crumpton and C. L. Bethel, “A survey of using vocal prosody to convey emotion in robot speech,” International Journal of Social Robotics, vol. 8, pp. 271–285, 2016.
  * [3] K. Zhou, B. Sisman, R. Rana, B. W. Schuller, and H. Li, “Speech synthesis with mixed emotions,” IEEE Tran. on Affective Computing, 2022.
  * [4] J. Kong, J. Kim, and J. Bae, “Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis,” Advances in Neural Inf. Proc. Sys., NeurIPS, vol. 33, 2020.
  * [5] W.-N. Hsu, T. Remez, B. Shi, J. Donley, and Y. Adi, “Revise: Self-supervised speech resynthesis with visual input for universal and generalized speech regeneration,” in Proc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 2023.
  * [6] A. Triantafyllopoulos, B. W. Schuller, G. İymen, M. Sezgin, X. He, Z. Yang, P. Tzirakis, S. Liu, S. Mertes, E. André, et al., “An overview of affective speech synthesis and conversion in the deep learning era,” Proc. of the IEEE, 2023.
  * [7] K. Zhou, B. Sisman, R. Liu, and H. Li, “Emotional voice conversion: Theory, databases and esd,” Speech Communication, vol. 137, pp. 1–18, 2022.
  * [8] Z. Du, B. Sisman, K. Zhou, and H. Li, “Disentanglement of Emotional Style and Speaker Identity for Expressive Voice Conversion,” in Interspeech, 2022\.
  * [9] P. Ekman and W. V. Friesen, “Constants across cultures in the face and emotion.,” Journal of personality and social psychology, vol. 17, no. 2, p. 124, 1971.
  * [10] C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan, “Iemocap: Interactive emotional dyadic motion capture database,” Language resources and evaluation, vol. 42, no. 4, pp. 335–359, 2008.
  * [11] J. A. Russell, “A circumplex model of affect.,” Journal of personality and social psychology, vol. 39, no. 6, p. 1161, 1980.
  * [12] L. Martinez-Lucas, M. Abdelwahab, and C. Busso, “The msp-conversation corpus,” Interspeech, 2020.
  * [13] N. Raj Prabhu, G. Carbajal, N. Lehmann-Willenbrock, and T. Gerkmann, “End-to-end label uncertainty modeling for speech-based arousal recognition using Bayesian neural networks,” in Interspeech, Sep 2022.
  * [14] K. Zhou, B. Sisman, R. Rana, B. W. Schuller, and H. Li, “Emotion intensity and its control for emotional voice conversion,” IEEE Tran. on Affective Computing, vol. 14, no. 1, pp. 31–48, 2023.
  * [15] K. Zhou, B. Sisman, R. Liu, and H. Li, “Seen and unseen emotional style transfer for voice conversion with a new emotional speech dataset,” in IEEE Int. Conf. on Acoustics, Speech, Sig. Proc., ICASSP, pp. 920–924, IEEE, 2021.
  * [16] F. Burkhardt, A. Paeschke, M. Rolfes, W. F. Sendlmeier, B. Weiss, et al., “A database of german emotional speech.,” in Interspeech, vol. 5, pp. 1517–1520, 2005.
  * [17] G. Rizos, A. Baird, M. Elliott, and B. Schuller, “Stargan for emotional speech conversion: Validated by data augmentation of end-to-end emotion recognition,” in IEEE Int. Conf. on Acoustics, Speech, Sig. Proc., ICASSP, 2020.
  * [18] K. Zhou, B. Sisman, and H. Li, “Transforming spectrum and prosody for emotional voice conversion with non-parallel training data,” in The Speaker and Language Recognition Workshop (Speaker Odyssey), 05 2020.
  * [19] K. Zhou, B. Sisman, and H. Li, “Vaw-gan for disentanglement and recomposition of emotional elements in speech,” in IEEE Spoken Language Technology Workshop, 2021.
  * [20] D. de Oliveira, N. Raj Prabhu, and T. Gerkmann, “Leveraging semantic information for efficient self-supervised emotion recognition with audio-textual distilled models,” in Interspeech, 2023.
  * [21] J. Wagner, A. Triantafyllopoulos, H. Wierstorf, M. Schmitt, F. Burkhardt, F. Eyben, and B. W. Schuller, “Dawn of the transformer era in speech emotion recognition: closing the valence gap,” IEEE Tran. on Pattern Analysis and Machine Intelligence, 2023.
  * [22] A. Polyak, Y. Adi, J. Copet, E. Kharitonov, K. Lakhotia, W.-N. Hsu, A. Mohamed, and E. Dupoux, “Speech Resynthesis from Discrete Disentangled Self-Supervised Representations,” in Interspeech, 2021.
  * [23] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” Advances in Neural Inf. Proc. Sys., NeurIPS, vol. 33, pp. 12449–12460, 2020.
  * [24] F. Kreuk, J. Keshet, and Y. Adi, “Self-Supervised Contrastive Learning for Unsupervised Phoneme Segmentation,” in Interspeech, pp. 3700–3704, 2020\.
  * [25] S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao, J. Wu, L. Zhou, S. Ren, Y. Qian, Y. Qian, M. Zeng, and F. Wei, “Wavlm: Large-scale self-supervised pre-training for full stack speech processing,” IEEE Journal of Selected Topics in Signal Processing, vol. 16, pp. 1505–1518, 2021.
  * [26] A. Sicherman and Y. Adi, “Analysing discrete self supervised speech representation for spoken language modeling,” in IEEE Int. Conf. on Acoustics, Speech, Sig. Proc., ICASSP, pp. 1–5, 2023\.
  * [27] A. van den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu, “Wavenet: A generative model for raw audio,” in 9th ISCA Speech Synthesis Workshop, pp. 125–125, 2016\.
  * [28] A. B. L. Larsen, S. K. Sønderby, H. Larochelle, and O. Winther, “Autoencoding beyond pixels using a learned similarity metric,” in Int. Conf. Machine Learning (ICML), 2016.
  * [29] L. I.-K. Lin, “A Concordance Correlation Coefficient to Evaluate Reproducibility,” Biometrics, vol. 45, p. 255, Mar. 1989.
  * [30] P. Andreev, A. Alanov, O. Ivanov, and D. Vetrov, “Hifi++: A unified framework for bandwidth extension and speech enhancement,” in IEEE Int. Conf. on Acoustics, Speech, Sig. Proc., ICASSP, pp. 1–5, 2023\.
  * [31] J. Lorenzo-Trueba, J. Yamagishi, T. Toda, D. Saito, F. Villavicencio, T. Kinnunen, and Z.-H. Ling, “The voice conversion challenge 2018: Promoting development of parallel and nonparallel methods,” in Interspeech, Apr 2018\.
  * [32] F. Eyben, K. R. Scherer, B. W. Schuller, J. Sundberg, E. Andre, C. Busso, L. Y. Devillers, J. Epps, P. Laukka, S. S. Narayanan, and K. P. Truong, “The geneva minimalistic acoustic parameter set (gemaps) for voice research and affective computing,” IEEE Tran. on Affective Computing, vol. 7, pp. 190–202, Apr 2016.
  * [33] G. Ramet, P. N. Garner, M. Baeriswyl, and A. Lazaridis, “Context-aware attention mechanism for speech emotion recognition,” in IEEE Spoken Language Technology Workshop, pp. 126–131, 2018.