# Wait or Not to Wait: Evaluating Trade-Offs between Speed and Precision in
Blockchain-based Federated Aggregation ††thanks: * These authors have equal
contribution

Huong Nguyen*, Tri Nguyen*, Lauri Lovén, Susanna Pirttikangas  Center for
Ubiquitous Computing, University of Oulu, Finland
<EMAIL_ADDRESS>
###### Abstract

This paper presents a fully coupled blockchain-assisted federated learning
architecture that effectively eliminates single points of failure by
decentralizing both the training and aggregation tasks across all
participants. Our proposed system offers a high degree of flexibility,
allowing participants to select shared models and customize the aggregation
for local needs, thereby optimizing system performance, including accurate
inference results. Notably, the integration of blockchain technology in our
work is to promote a trustless environment, ensuring transparency and non-
repudiation among participants when abnormalities are detected. To validate
the effectiveness, we conducted real-world federated learning deployments on a
private Ethereum platform, using two different models, ranging from simple to
complex neural networks. The experimental results indicate comparable
inference accuracy between centralized and decentralized federated learning
settings. Furthermore, our findings indicate that asynchronous aggregation is
a feasible option for simple learning models. However, complex learning models
require greater training model involvement in the aggregation to achieve high
model quality, instead of asynchronous aggregation. With the implementation of
asynchronous aggregation and the flexibility to select models, participants
anticipate decreased aggregation time in each communication round, while
experiencing minimal accuracy trade-off.

###### Index Terms:

Federated learning, Blockchain, Personalized model, Customization.

##  I Introduction

Machine learning approaches are in high demand across various domains, driven
by the data’s rapid growth. However, traditional machine learning often
requires data to be transferred to a centralized computation for a robust
model formation, leading to communication costs, privacy leakage, and security
concerns. In response, Federated Learning (FL) [1] aims to reduce the
dependence of machine learning approaches on centralized computation.

In more detail, FL allows an array of participants to partake in training
while preserving privacy through distributed training tasks. Essentially,
instead of transferring data to a centralized computation, FL distributes the
training tasks to various participants for local model training. Once trained,
local models are sent to a server for aggregation into a global model, which
is consequently shared with the participants to update their local models
accordingly. By using FL, challenges associated with communication costs, data
privacy, and security concerns can be mitigated. This characteristic of FL,
along with its potential to maintain data privacy, significantly contributes
to making it an attractive approach for modern computing paradigms. However,
the conventional FL still encounters several issues, including single-point
failure, malicious participants, unfair incentives, local model verification,
and authorization.

Previous works have proposed solutions that address these issues by leveraging
blockchain technology. Blockchain offers a trustless environment by means of a
decentralized, immutable database among participants. As an example, to avoid
single-point failure, the central aggregator can be shifted to a network of
participants in a decentralized area [2, 3]. Another solution involving
blockchain is to verify local model updates before aggregating them to the
global model, which can be auditable via evidence in blockchain [3, 4]. To
encourage participants in training model tasks, some researchers have proposed
using incentive strategies based on blockchain technology [5, 6, 4]. While
these works have shown promise in addressing some FL issues, the intersection
of blockchain and FL has raised other problems. These include trade-offs
between system performance and inference accuracy, particularly when using
asynchronous communication for aggregation in different scenarios, such as
centralized and decentralized settings. Also, most of previous works focus on
permissioned blockchain like Hyperledger
Fabric111https://www.hyperledger.org/projects/fabric and blockchain-based
simulation where the system is closed and lacks flexibility or dynamicity.
Hence, a real deployment for permissionless blockchain is an open question.

This work presents a feasible architecture and deployment for blockchain-based
FL that addresses centralized FL issues. The offered architecture eliminates
the aggregator as a central entity in traditional FL and instead allows the
aggregation task to be deployed and maintained by any participants in the
system. This architecture transforms FL from a traditional to a decentralized
environment, with blockchain technology leveraged as the primary solution.
Furthermore, by enabling the selection of models for personalized aggregation,
our method effectively prevents the integration of poisoning (intended) or
noisy models (unintended), which can lead to lower accuracy in future
aggregations. Therefore, this architecture facilitates asynchronous
communication, allowing aggregation, called asynchronous aggregation, to
proceed without waiting for all local training models to be completed.
Compared to prior related works, which primarily consisted of surveys and
limited experimentation, our research uniquely focuses on investigating the
impact of model complexities on the performance of asynchronous aggregation in
blockchain-based FL. Also, our work conducted on a private Ethereum aims to
uncover potential challenges that may arise in real-world deployments, issues
that simulations cannot fully capture. The decision to conduct a real
deployment allows us to bridge the gap between theoretical simulations and
practical implementation, providing valuable insights into the question: “How
do different model complexities influence the performance of blockchain-based
FL aggregation?”

The experimental results demonstrate the feasibility of our proposal in simple
learning models using asynchronous aggregation. However, complex learning
models require a larger number of training models to be involved in
aggregation to achieve high accuracy. Additionally, the experiments reveal a
notable similarity in inference accuracy between centralized and decentralized
FL settings.

The remaining of the paper is structured as follows: Section II provides an
overview of the background and related work. Section III introduces the
proposed scheme, while Section IV presents experimental details and results.
Finally, Section V summarizes this work.

##  II Background and Related Work

###  II-A Backgrounds

####  II-A1 System models of blockchain-leveraged Federated Learning

One of the interesting considerations is the harmonization of architecture
between FL and blockchain architecture. In detail, initial FL is introduced
with a “master-slave” architecture, while decentralization is the foundation
of blockchain technology. Regarding survey works [7, 8], FL leverages the
blockchain in three directions from the architecture perspective. The first
view of blockchain-based FL is about fully coupled, which points to the FL
clients handling blockchain and aggregator tasks. Semi-coupled blockchain-
based FL is about the deployment of FL aggregation to blockchain participants,
and FL clients train and submit local models to blockchain participants for
aggregation. Loosely coupled between blockchain and FL is a concept in which
the blockchain provides services, for example, access management or reputation
for enhancing FL features, especially from a security viewpoint.

####  II-A2 Blockchain performance

Given the variety in blockchain platforms, an initial comparative analysis by
[9] highlights that Fabric exhibits lower latency and higher throughput than
Ethereum. Furthering this research, [10] developed BLOCKBENCH, a tool for
evaluating different blockchain platforms. Results from BLOCKBENCH demonstrate
that Fabric achieves superior throughput performance when compared to both
Proof-of-Work (PoW) and Proof-of-Authority variants of Ethereum. Additionally,
in terms of computational and memory demands, Hyperledger Fabric consumes the
fewest resources. Nonetheless, the study does not provide detailed information
on block and transaction capacities for each blockchain setup, particularly
noting the absence of difficulty configuration details in PoW-based Ethereum.
Also, the limited interactions among smart contracts pose another area of
concern. Therefore, previous studies [11, 12] detailed the inverse effect of
block size and throughput. Particularly, those works evaluate the system’s
performance in various scenarios increasing the block size, which findings are
about the more number of network participants and the high capacity of
messages influences the system performance, especially in throughput.
Regarding the influence of system performance via the number of participants
in a blockchain-based FL, Peng et al. [4] find a half reduction in throughput
and an increase in execution time from the doubled number of participants. To
boost the blockchain performance, Sethi et al. [13] point the usage of
reinforcement learning to predict the PoW’s difficulty in consensus rounds to
enhance blockchain performance, especially in the usage of blockchain-based FL
where the number of participants is flexible.

Due to the previous comprehensive works in blockchain performance in different
scenarios, those findings related to network measurement are accepted in this
work, which means the asynchronous aggregation with less communication gains
achievements in comparison with the synchronous federated aggregation waiting
and aggregating the entire local models.

###  II-B Related work

One of the initial proposals for integrating blockchain to enhance security in
FL is presented in [3]. This study introduces a semi-coupled blockchain-based
FL framework that leverages blockchain technology to prevent privacy leaks.
Specifically, it utilizes blockchain to manage data flows and implement a
verification mechanism that supports a fully decentralized environment.
However, the experimental setup described is relatively simple, involving a
single blockchain node, and it does not address the impact of complex models
on asynchronous aggregation.

Flchain [14] also explores a semi-coupled blockchain-based FL system, designed
to bolster the security of FL through the creation of channels for learning
multiple global models. In this system, local devices first train their models
independently before exchanging them via blockchain to form a cohesive global
model. This approach allows devices the flexibility to select channels or
global models that best suit their training needs. Despite its innovative
approach, this early work on blockchain integration in FL lacks detailed
experimental validation and comprehensive settings description.

Lu et al. [15] discuss the application of blockchain combined with FL to
ensure privacy-preserving data sharing within the industrial IoT. A
permissioned blockchain is employed to manage the computation and storage of
FL models, facilitating the auditing of data usage permissions. In other
words, [15] is a semi-coupled blockchain-based FL for a marketplace which
deals with the privacy-preserving aspect in sharing data from data holders.
While this study builds upon a promising concept and conducts experiments with
two datasets, it falls short in extensively exploring the effects of varying
model complexities or the practical deployment challenges encountered in
blockchain-based FL systems.

Kang et al. [16] proposed a blockchain-empowered federated edge learning. The
study employs a dual blockchain structure: a main public blockchain for global
models and consortium sub-chains for local models from edge servers. This
setup supports decentralization while catering to real-time applications. They
introduced a “Proof-of-Verifying” consensus to filter malicious updates,
focusing on gradient compression to improve communication efficiency. This
consensus evaluates local updates against a testing dataset using a threshold,
ensuring only valid updates contribute to the global model.

From FL’s aggregation aspect, Ramanan et al. [17]’s BAFFLE introduces a novel
approach to FL by employing blockchain technology to facilitate aggregator-
free operations. It leverages smart contracts for the aggregation of the
global model, eliminating the need for a central aggregator. Despite its
innovative approach to decentralizing model aggregation and enhancing
security, BAFFLE faces significant challenges with increased computation and
communication costs. The system incorporates smart contracts for interactions
and updates in the blockchain, ensuring a seamless blend of local training
model updates and global model aggregation. Similar to BAFFLE, PIRATE [18]
utilizes blockchain technology for decentralized learning, storing each
community’s global model on separate blockchains. This facilitates inter-
community model updates. PIRATE addresses Byzantine attacks, which can occur
during model parameter updates and gradient aggregations, by enhancing the
system’s resilience against such threats in distributed learning environments.

Li et al. [19] introduce a blockchain-based FL framework named BFLC, which
utilizes a committee consensus mechanism to enhance both efficiency and
security. This framework employs blockchain technology for securely storing
global models and exchanging local model updates. BFLC improves performance
through a committee selection consensus mechanism and uses smart contracts for
autonomously aggregating global models, ensuring that only authorized devices
can contribute updates. This approach significantly enhances the security and
efficiency of FL systems.

Shayan et al. [2] leverage blockchain and cryptographic techniques for
privacy-preserving machine learning process through a semi-coupled blockchain-
based FL. Their system, Biscotti, is noted for its resilience to fault-
tolerant and malicious adversaries. In Biscotti, each blockchain’s block
represents a training iteration, in which a selected committee aggregates and
verifies local updates before issuing rewards to trainers. These rewards, such
as reputed scores, help in the selection of committee members for aggregating
models in FL and gaining consensus among blockchain participants.

Peng et al. [4] present VFChain, a semi-coupled blockchain-based FL for
auditability and verifiability. A selected committee aggregates models
alongside verifiable proofs recorded on blockchain for auditability. The
committee, selected from a reputation system, receives local gradients from
participants for aggregation. The experiments are based on a centralized
Fabric platform and a convolution neural network, and include mechanisms for
verifying global models through voting procedures; however, it does not
support personalized aggregation in the aggregation process, limiting its
flexibility.

Despite enabling decentralized features, those previous works face other
issues regarding performance and decentralized issues. BLADE-FL [6] builds on
the integration of blockchain to enhance FL, aiming to overcome efficiency
limitations seen in prior works. It emphasizes privacy, resource allocation,
and addressing free-rider issues, distinguishing itself by requiring specific
contributions for maintaining the blockchain and local model training. BLADE-
FL introduces a mechanism for assigning training tasks and rewards, addresses
participant anonymity by adding random noise to data (though it affects
learning performance), and proposes a method using pseudo-noise sequences to
identify and penalize non-contributing participants.

Regarding the communication aspect, Wilhelmi et al. [20] explore blockchain’s
role in decentralizing FL to form semi-coupled blockchain-based FL focusing on
the freshness of model updates via the “age-of-block” metric to measure time
differences between local model updates. This concept helps analyze how
blockchain’s real-world application affects learning accuracy, addressing
potential inconsistencies and network delays, especially blockchain fork
issues. Their findings highlight the FL framework’s resilience in maintaining
accuracy across various environmental configurations.

Xu et al. [5] introduce BESIFL, a loosely coupled blockchain-based FL, to
prevent malicious nodes through an incentive mechanism. BESIFL aims to enhance
inference accuracy by selecting credible nodes using a reputation-based
incentive system. The BESIFL employs a credit-based approach for node
selection, distinguishing training and verification participants through two
specific scores derived from a contribution-based incentive mechanism. Via
token-based reward and punishment, smart contract-based model demander playing
a role as the FL server for aggregation offers reward or punishment training
or malicious participants. This incentive strategy enhances the accuracy
compared to previous models. The experiments are deployed in Hyperledger
Fabric with a centralized server with a Kafka setting, and IPFS is utilized
for decentralized storage, with hash data and reputation scores stored on the
blockchain for verification. Despite these advancements, the study does not
explore how varying complexities of models might affect inference accuracy,
which remains an open area.

Finally, while prior works have shown promise in addressing certain FL
challenges, the majority of the existing literature focuses on permissioned
blockchains such as Hyperledger Fabric, or blockchain-based simulations with
closed systems lacking flexibility and dynamism. A prevailing weakness among
prior studies is also the inadequate evaluation of their systems’
compatibility across a spectrum of model complexities. This leaves a
significant gap in exploring real deployments for permissionless blockchains,
which is exactly where our work delves into the intricate details of this
intersection under multiple settings, particularly the accuracy comparison
between centralized and decentralized settings along with the variety of model
complexities.

##  III fully coupled blockchain-based Federated Learning

###  III-A Decentralization via Blockchain-based FL

Figure 1: Vanilla vs proposed FL architecture illustrations

Due to single-point failure from traditional FL, leveraging blockchain
technology regarding fully coupled integration is a promising solution for
shifting to a decentralized setting. Following traditional FL, the aggregation
duty as centralization is to form a global model based on an array of local
models; however, from the vulnerable perspective, single-point failure is a
downside, and the transformation to decentralization is a promising solution.
In particular, Figure 1 illustrates the comparison between Vanilla and
decentralized learning. Based on the prior works, we found that the
traditional FL systems work with two layers: the client layer, which handles
data generation, self-processing, and making local models, and the aggregator
layer forming the global models. With the decentralized aspect, this research
attempts to rearrange the architecture and roles in the system, where the data
generator is separated/organized into one layer meanwhile the worker node, as
well as the aggregator, are merged into one layer. Consequently, we will have
multiple aggregators working as peers in an asynchronous blockchain-assisted
FL system, supporting in detecting nodes with abnormal models. With fully
coupled blockchain interference, the proposed system gains a lot of perks,
which we consider three prominent cases:

  * •

Case 1 - flexibility in choosing shared models and customizing an aggregation
for the local usage

  * •

Case 2 - flexibility in aggregation time or asynchronous aggregation

  * •

Case 3 - ensuring non-repudiation of the participant about their models

Leveraging the fact that peers will automatically share their new data with
others, therefore, when we put the blockchain into an FL system, peers - those
working as local workers and aggregators here will broadcast their local model
(weights) to the rest of the participants in the network. As a consequence,
each peer will have all the shared models from others and is expected to have
the ability to choose ones to join its customized aggregation process freely
or asynchronous aggregation. To do this, a test dataset is prepared to
evaluate the fitness of the shared model. If the evaluation is over a pre-set
threshold, the worker will then include that model in their aggregation
process; otherwise, it will be ignored. This will help when the updated model
is too general and contains weights from the outliers due to the heterogeneous
data from other regions or scopes, leading to an accuracy decrease.

###  III-B Asynchronous Aggregation in Decentralized Learning

Due to the large communication from the bottleneck, the proposed learning
system will be an asynchronous system, where there is no need to wait for all
participants in the network to complete their local training to start the
aggregation progress. Working in the conventional method, where each peer
awaits all other peers’ updates for aggregation will create a heavier
computational load for workers. This becomes more observable when we have
millions of participants in the network and each has to evaluate the relevance
and suitability of numerous shared models before starting their customized
aggregation. From that, bottlenecks can be formed at peer-level and the
performance is then reduced. Therefore, each aggregator can desire how many
local updates she/he would use to aggregate instead of waiting for total
update models. However, the issue from this consideration leads to local
generalization and reduces the knowledge from the global. Based on this
analysis, we come up with a contemplation that varying complexities in models
may impact the inference accuracy in the asynchronous aggregation processes of
blockchain-based federated learning systems.

Regarding the flexible configuration protocol and dynamic participants, the
system can operate in either decentralized learning with various customized
models or FL regarding the identical global model. In the first case, the
system can go as decentralized learning which points to the freedom of
customizing the current peer’s model with an arbitrary number of local models
to satisfy the peer’s specific test dataset. Meanwhile, the second method
supports gaining a global viewpoint from the entire network’s peers to vote
and support a bunk of local models regarding a consensus like a global model;
however, instead of a fixed single aggregator, this mechanism allows any peer
to become the aggregator to avoid the single-point failure.

###  III-C Hierarchy of blockchain-based FL

For a sophisticated blockchain-based FL design, we deliberate the adoption of
a middle layer by which the data is transferred to the closest local server
instead of traditional FL, which requires training in data generation before
sending it to a centralized aggregation server. Regarding the lack of data in
training, the local server gathers data nearby before training a local model.
In detail, local servers communicate and exchange their local models after
training. At this stage, the peers will have two options for aggregation: (1)
customizing an arbitrary number of local models or (2) agreeing on a common
block of local updates as representing the potential global update for the
next state. Besides, the usage of this layer enables the intelligence at the
edge that has been discussed as the most interesting topic from various
communities [21], regarding the cloud-edge continuum, due to its availability
and performance.

##  IV Evaluation

###  IV-A Setup and Environment

####  IV-A1 Blockchain Configuration

We decided to utilize the Ethereum platform222https://ethereum.org/en/ to
deploy our proposal. The main idea with Ethereum is a well-known
permissionless blockchain, which should fit our proposal as aforementioned.
Due to the permissionless blockchain platform, the computation cost from PoW
consensus cannot be avoided; however, Ethereum enables openness in
collaboration learning. The workflow of Ethereum can be viewed in Figure 2. In
the beginning, from the yellow part of the figure, the data is collected from
the data generator for training before sharing it with other participants in
the blockchain network. Parallelly, blockchain participants execute a PoW
procedure to determine the leader of the current consensus round in the blue
region of Figure 2. Once the leader is determined, a potential block is
broadcast from the leader for verification from other participants in regions
red and green, respectively.

Figure 2: Ethereum setting for the experiment via three peers deploying
Ethereum application denoted Geth with different tasks: (a) Interaction
between data generator and blockchain participants, (b) PoW consensus to
select a leader, (c) Block formation, (d) Block validation

The blockchain experiments are based on the Ethereum framework which setup
encompassed three virtual entities, called Geth:1.10.12 along with
Truffle:5.4.28333https://archive.trufflesuite.com/, on a physical machine: an
Intel(R) Core(TM) i7-8700 CPU @ 3.20 GHz, 3.19 GHz, 32 GB RAM, and 64-bit OS
x64-based processor. Concretely, based on the VirtualBox tool, each virtual
machine resource has 6 GB RAM and 4 processors. The asynchronous aggregation
is based on a solidity-language smart contract. Regarding the interaction
between FL local model training and blockchain-based smart contract, we
utilize web3444https://web3js.readthedocs.io/en/v1.2.1/getting-started.html
library and asynchronous communication via NodeJS for propagating local models
and collecting aggregated one for the next iterator. Furthermore, building
upon the system performance findings from previous works discussed in Section
II-A2, we have configured Ethereum without block size and transaction size
constraints. Specifically, we ensure that the transaction size exceeds the
model’s size, as detailed in the gas conversion aspect outlined in [12]. This
adjustment accounts for Ethereum’s complexity, which is measured in units of
gas.

####  IV-A2 Federated Learning Configuration

Regarding the training, we come up with three considerations that need to be
decided, including dataset, models, and training.

Dataset: In determining the selection criteria for these datasets, we
conducted a thorough review of prior works pertaining to FL and found that
MNIST, CIFAR-10, and CIFAR-100 are commonly employed as benchmark datasets. We
then opted to use CIFAR-10 due to its colored images and the limited number of
classes (ten). This decision was made in contrast to MNIST, which contains
only binary images, and CIFAR-100, with a significantly larger number of
labels (one hundred) while maintaining the same dataset size. The CIFAR-10
dataset consists of 60000 images in the size of 32x32, number of classes in
total is 10 with a variety of labels on animals (bird, cat, deer, dog, frog,
horse) and vehicles (airplane, automobile, ship, truck).

Model: In the model selection phase, we considered running our experiment on
multiple models, ranging from simple Neural Network (Simple NN) architectures
to more complex ones, such as EfficientNet, developed by Google. Specifically,
we have opted to work with both Simple NN model and the most resource-
efficient variant of EfficientNet, denoted as EfficientNet-B0.

Training: We use PyTorch to implement the training with chosen models: Simple
NN and EfficientNet-B0. The Simple NN model is constructed from scratch with
only 62K parameters and approximately 248KB in size, whereas, for
EfficientNet-B0 (parameters count 5.3M, size 21.2MB), we employ transfer
learning by modifying its final layer. FedAvg[1] algorithm is utilized for
model aggregation during our FL training process.

###  IV-B Experiments and Results

####  IV-B1 Experiments

Regarding the experiments, we illustrate two scenarios of deploying FL in
centralized and decentralized environments. In both cases, we consider to
assess the fitness of the updated model. Specifically, Section III points out
the usage of test data from each participant to evaluate the receiving model
updates before deciding to aggregate at each communication round.

Centralized setting: In the centralized scenario, we deploy a Vanilla FL with
ten communication rounds in which each training device trains five epochs for
local updates before sending for aggregation. In detail, three virtual
machines representing three local devices A, B, and C, are used for training
local models before sharing them with the aggregator to aggregate the global
model. This global is then shared back to local devices for the next round.
Besides, we present two types of aggregation from the aggregator, called
“consider” and “not consider”. “not consider” is Vanilla’s traditional
approach, where the aggregator collects and aggregates all three local
updates. Meanwhile, in the “consider” case, the aggregator chooses the best
combination of local models via a default test set.

Decentralized setting: In the decentralized scenario, we deploy a blockchain-
based FL with Ethereum. In particular, three virtual machines implement both
local training and aggregation tasks, instead of using a specific aggregator
for global aggregation. Therefore, each device’s results vary based on the
diversity of aggregation. For the next training round, the combined model with
the highest accuracy from the last round is utilized. As a result, the chosen
model combination may vary between rounds. In cases where multiple
combinations achieve the same performance or accuracy, the device selects one
of them randomly. We also keep the same FL training as Vanilla’s setting,
which is ten communication rounds and five epochs for training at local
devices.

####  IV-B2 Results

Regarding the experimental results, we investigate the similarity as well as
the main difference findings in Vanilla and blockchain-based FL settings.

Vanilla setting: In Table I, we report the accuracy values of participants in
each communication round, considering two aggregation types at the aggregator:
“consider” and “not consider”. These values are measured by evaluating the
aggregated model’s performance on the test data from individual local clients.
We also include Figure 3 to visualize the reported values. Regarding
comparative results, we observe that having the option to select models at the
aggregator (denoted as “consider”) generally results in better accuracy for
participating clients, especially when training simple models. The difference
here is minor, at only 0.65%, between 59.53% (“not consider”) and 60.18%
(“consider”). On the other hand, when dealing with more complex models, we
achieve higher performance (around 85-86% across clients), accompanied by some
fluctuations between these two aggregation types, as depicted in Figure 3(b).
Those results can point out the similarity in the accuracy despite the keeping
or discarding of imperfect models in the two aggregation approaches. Due to
this finding, the blockchain-based FL is expected to be a potential solution
for unfit or abnormal model detection in the aggregation process.

(a) SimpleNN

(b) Efficient B0

Figure 3: Vanilla FL: Test accuracy (%) on two aggregation types TABLE I: Vanilla FL: Clients’ test accuracy (%) on two aggregation types Model | Client | Params | round
---|---|---|---
|  |  | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10
Simple NN | A | Consider | 0.2263 | 0.3733 | 0.4325 | 0.4747 | 0.5096 | 0.5316 | 0.5591 | 0.5803 | 0.592 | 0.6018
Not consider | 0.2855 | 0.3831 | 0.4493 | 0.4937 | 0.5258 | 0.54 | 0.5577 | 0.5711 | 0.5799 | 0.5953
B | Consider | 0.2319 | 0.3733 | 0.4325 | 0.4747 | 0.5096 | 0.5316 | 0.5591 | 0.5803 | 0.592 | 0.6018
Not consider | 0.2855 | 0.3831 | 0.4493 | 0.4937 | 0.5258 | 0.54 | 0.5577 | 0.5711 | 0.5799 | 0.5953
C | Consider | 0.2263 | 0.3733 | 0.4325 | 0.4747 | 0.5096 | 0.5316 | 0.5591 | 0.5803 | 0.592 | 0.6018
Not consider | 0.2855 | 0.3831 | 0.4493 | 0.4937 | 0.5258 | 0.54 | 0.5577 | 0.5711 | 0.5799 | 0.5953
Efficient-B0 | A | Consider | 0.8018 | 0.8394 | 0.8525 | 0.8532 | 0.8567 | 0.8565 | 0.858 | 0.8597 | 0.8499 | 0.8578
Not consider | 0.7986 | 0.8421 | 0.8511 | 0.8585 | 0.8556 | 0.8578 | 0.8561 | 0.858 | 0.859 | 0.8583
B | Consider | 0.7991 | 0.8421 | 0.853 | 0.8533 | 0.8563 | 0.8546 | 0.8587 | 0.8585 | 0.8513 | 0.8597
Not consider | 0.7982 | 0.8422 | 0.8534 | 0.8553 | 0.8568 | 0.8596 | 0.8541 | 0.8596 | 0.862 | 0.8576
C | Consider | 0.8021 | 0.8423 | 0.8503 | 0.853 | 0.8586 | 0.8538 | 0.8586 | 0.8613 | 0.8502 | 0.858
Not consider | 0.7974 | 0.8411 | 0.8513 | 0.8548 | 0.8587 | 0.859 | 0.8568 | 0.8585 | 0.8589 | 0.8576

Decentralized setting: Concerning experiments of blockchain-based FL, the
accuracy for each round is reported in Table II, III, and IV, corresponding to
the respective device. Overall, the experiments with Simple NN models
demonstrate consistent accuracy values across participants, while variations
in accuracy values are observed among different devices in the case of
Efficient-B0 models. In terms of model combination, the one that has the
highest aggregation accuracy varies across communication rounds, particularly
for simple models. However, for a more complex model like Efficient-B0,
aggregating all models consistently yields the highest accuracy in most
rounds, except for specific instances: round 1 at client A with model
combination BC (78.97% vs. 78.9% for the ABC model), round 3 at client A with
combination BC (85.23% vs. 85.21% for ABC), round 1 at client B with model
combination BA (78.97% vs. 78.94% for ABC), round 3 at client B with model BA
(84.08% vs. 83.69% for ABC), round 1 at client C with model combination AB
(79.01% vs. 78.81% for ABC), and round 9 at client C with model CB (85.41% vs.
85.13% for ABC). As shown in Figure 4, the similarity of various aggregations
is evident among devices in the case of Simple NN, but this does not hold for
Efficient-B0. However, it is worth noting that when training complex models
like Efficient-B0, it is more beneficial for participating clients to
collaborate by combining their local models with others, as using solely their
local models consistently results in lower or sub-optimal performance. The
blockchain-based FL experiment highlights that aggregating the entire set of
models in complex models yields superior results, whereas, in the case of
simple models, accuracy remains relatively consistent.

Consequently, a trade-off question, regarding the aggregation time and model
quality, is raised: “Should we prioritize waiting for all models for
aggregation, or accept a slight reduction in accuracy to expedite the process
asynchronously?” The response to this question is formulated through a
detailed analysis of specific applications, which may vary in requirements.

(a) Client A

(b) Client B

(c) Client C

Figure 4: Blockchain-based FL: Test accuracy (%) on different model combinations TABLE II: Blockchain-based FL: Test accuracy (%) on different model combinations - Client A Model | Params from | Round
---|---|---
|  | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10
Simple NN | A | 0.1435 | 0.3143 | 0.3901 | 0.4343 | 0.4686 | 0.4923 | 0.5295 | 0.5405 | 0.559 | 0.5719
A,B | 0.1732 | 0.3193 | 0.3909 | 0.4365 | 0.4666 | 0.5027 | 0.5245 | 0.5465 | 0.567 | 0.5777
A,C | 0.1732 | 0.3193 | 0.3909 | 0.4365 | 0.4666 | 0.5027 | 0.5245 | 0.5465 | 0.567 | 0.5777
B,C | 0.1732 | 0.3193 | 0.3909 | 0.4365 | 0.4666 | 0.5027 | 0.5245 | 0.5465 | 0.567 | 0.5777
A, B, C | 0.1 | 0.3161 | 0.3921 | 0.4348 | 0.4678 | 0.5008 | 0.5287 | 0.5503 | 0.5676 | 0.5815
Efficient-B0 | A | 0.7753 | 0.7976 | 0.8121 | 0.8154 | 0.8148 | 0.8279 | 0.8261 | 0.8247 | 0.8315 | 0.8219
A,B | 0.7876 | 0.8237 | 0.8391 | 0.843 | 0.843 | 0.848 | 0.8507 | 0.8512 | 0.8483 | 0.8396
A,C | 0.7905 | 0.8222 | 0.8406 | 0.8443 | 0.8441 | 0.8489 | 0.8476 | 0.8495 | 0.8505 | 0.8449
B,C | 0.7924 | 0.8266 | 0.8415 | 0.8399 | 0.8409 | 0.8523 | 0.8493 | 0.8479 | 0.8486 | 0.8438
A, B, C | 0.789 | 0.8357 | 0.8473 | 0.8492 | 0.8492 | 0.8521 | 0.8534 | 0.8533 | 0.8526 | 0.8498
TABLE III: Blockchain-based FL: Test accuracy (%) on different model combinations - Client B Model | Params from | Round
---|---|---
|  | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10
Simple NN | B | 0.1435 | 0.3143 | 0.3901 | 0.4343 | 0.4686 | 0.4923 | 0.5295 | 0.5405 | 0.559 | 0.5719
B, A | 0.1732 | 0.3193 | 0.3909 | 0.4365 | 0.4666 | 0.5027 | 0.5245 | 0.5465 | 0.567 | 0.5777
B, C | 0.1732 | 0.3193 | 0.3909 | 0.4365 | 0.4666 | 0.5027 | 0.5245 | 0.5465 | 0.567 | 0.5777
A, C | 0.1732 | 0.3193 | 0.3909 | 0.4365 | 0.4666 | 0.5027 | 0.5245 | 0.5465 | 0.567 | 0.5777
A, B, C | 0.1 | 0.3161 | 0.3921 | 0.4348 | 0.4678 | 0.5008 | 0.5287 | 0.5503 | 0.5676 | 0.5815
Eficient-B0 | B | 0.7761 | 0.7971 | 0.8121 | 0.8138 | 0.8143 | 0.8258 | 0.8222 | 0.8247 | 0.8292 | 0.8192
B, A | 0.7897 | 0.8281 | 0.8408 | 0.842 | 0.8443 | 0.8503 | 0.849 | 0.8498 | 0.8498 | 0.8433
B, C | 0.7888 | 0.8253 | 0.8394 | 0.8398 | 0.8411 | 0.8501 | 0.8477 | 0.8475 | 0.8494 | 0.8399
A, C | 0.7886 | 0.8252 | 0.8363 | 0.8398 | 0.8432 | 0.8469 | 0.8482 | 0.8466 | 0.8477 | 0.8439
A, B, C | 0.7894 | 0.8369 | 0.8369 | 0.8495 | 0.8471 | 0.8547 | 0.8549 | 0.8554 | 0.8518 | 0.8523
TABLE IV: Blockchain-based FL: Test accuracy (%) on different model combinations - Client C Model | Params from | Round
---|---|---
|  | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10
Simple NN | C | 0.1435 | 0.3143 | 0.3901 | 0.4343 | 0.4686 | 0.4923 | 0.5295 | 0.5405 | 0.559 | 0.5719
C, A | 0.1732 | 0.3193 | 0.3909 | 0.4365 | 0.4666 | 0.5027 | 0.5245 | 0.5465 | 0.567 | 0.5777
C, B | 0.1732 | 0.3193 | 0.3909 | 0.4365 | 0.4666 | 0.5027 | 0.5245 | 0.5465 | 0.567 | 0.5777
A, B | 0.1732 | 0.3193 | 0.3909 | 0.4365 | 0.4666 | 0.5027 | 0.5245 | 0.5465 | 0.567 | 0.5777
A, B, C | 0.1 | 0.3161 | 0.3921 | 0.4348 | 0.4678 | 0.5008 | 0.5287 | 0.5503 | 0.5676 | 0.5815
Efficient-B0 | C | 0.7705 | 0.7966 | 0.8142 | 0.8101 | 0.8122 | 0.8306 | 0.8225 | 0.8247 | 0.8275 | 0.8173
C, A | 0.7887 | 0.8236 | 0.8419 | 0.8452 | 0.844 | 0.851 | 0.8448 | 0.8495 | 0.8506 | 0.8456
C, B | 0.7899 | 0.8253 | 0.8377 | 0.8372 | 0.8426 | 0.8502 | 0.8441 | 0.8503 | 0.8541 | 0.8428
A, B | 0.7901 | 0.8233 | 0.8409 | 0.8428 | 0.8456 | 0.848 | 0.8444 | 0.8501 | 0.8509 | 0.847
A, B, C | 0.7881 | 0.8373 | 0.8429 | 0.8493 | 0.8516 | 0.8564 | 0.8514 | 0.8547 | 0.8513 | 0.8546

##  V Conclusion

This study presents a real-world Ethereum-based FL deployment, encompassing
varied testing across various model complexities. Our findings show that the
conventional averaging approach on all models, such as Vanilla FL, fails to
consistently yield optimal inference results and leads to subpar system
performance. Instead, empowering clients to select only the appropriate models
for their aggregation proved to be a more effective strategy in balancing
speed and inference accuracy trade-offs. This, however, is contingent on the
complexity of the training model. For simple and small models, asynchronous
aggregation offers a viable and advantageous alternative, eliminating the need
to wait for all clients to participate while maintaining the quality of
aggregated models. On the other hand, the selective combinations in the
asynchronous aggregation process may not always result in the highest
precision in training a large and complex model. Yet, they still can maintain
a good aggregated model with only negligible accuracy loss (less than 0.5%).
Besides, we also observe from our real-world deployment the occurrence of
resource exhaustion due to dual tasks on one peer (mining and training model),
a scenario that similar research with simulation experiments do not encounter.

The integration of blockchain technology in our system ensures participants
cannot deny their authorship, providing strong evidence against detected
abnormal clients. This contributes to the security and integrity of
collaborative learning systems, paving the way for more reliable and
trustworthy implementations in real-world settings. It is noted that, in the
scope of this work, abnormalities do not necessarily imply malicious intent or
attempts to disrupt the system; they may arise from the natural data
heterogeneity across clients.

Consequently, our future work will focus more on deploying and evaluating the
robustness of this method on the non-repudiation in various poisonous data
attacks and different datasets. Also, the impact of an arbitrary number of
local updates on each peer in asynchronous communication is another intriguing
question we aim to explore for optimal values.

## Acknowledgements

This research is supported by the Research Council of Finland through the 6G
Flagship program (Grant 318927), and by Business Finland through the Neural
pub/sub research project (8754/31/2022) and Enabling Metaverse (EMETA)
(8719/31/2022).

## References

  * [1] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y. Arcas, “Communication-Efficient Learning of Deep Networks from Decentralized Data,” in _Proceedings of the 20th International Conference on Artificial Intelligence and Statistics_ , ser. Proceedings of Machine Learning Research, A. Singh and J. Zhu, Eds., vol. 54. PMLR, 20–22 Apr 2017, pp. 1273–1282.
  * [2] M. Shayan, C. Fung, C. J. M. Yoon, and I. Beschastnikh, “Biscotti: A blockchain system for private and secure federated learning,” _IEEE Transactions on Parallel and Distributed Systems_ , vol. 32, no. 7, pp. 1513–1525, 2021.
  * [3] S. Awan, F. Li, B. Luo, and M. Liu, “Poster: A reliable and accountable privacy-preserving federated learning framework using the blockchain,” in _Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security_ , ser. CCS ’19. New York, NY, USA: Association for Computing Machinery, 2019, p. 2561–2563.
  * [4] Z. Peng, J. Xu, X. Chu, S. Gao, Y. Yao, R. Gu, and Y. Tang, “Vfchain: Enabling verifiable and auditable federated learning via blockchain systems,” _IEEE Transactions on Network Science and Engineering_ , vol. 9, no. 1, pp. 173–186, 2022.
  * [5] Y. Xu, Z. Lu, K. Gai, Q. Duan, J. Lin, J. Wu, and K.-K. R. Choo, “Besifl: Blockchain-empowered secure and incentive federated learning paradigm in iot,” _IEEE Internet of Things Journal_ , vol. 10, no. 8, pp. 6561–6573, 2023.
  * [6] C. Ma, J. Li, L. Shi, M. Ding, T. Wang, Z. Han, and H. V. Poor, “When federated learning meets blockchain: A new distributed learning paradigm,” _IEEE Computational Intelligence Magazine_ , vol. 17, no. 3, pp. 26–33, 2022\.
  * [7] Z. Wang and Q. Hu, “Blockchain-based federated learning: A comprehensive survey,” 2021.
  * [8] J. Zhu, J. Cao, D. Saxena, S. Jiang, and H. Ferradi, “Blockchain-empowered federated learning: Challenges, solutions, and future directions,” _ACM Comput. Surv._ , vol. 55, no. 11, feb 2023.
  * [9] S. Pongnumkul, C. Siripanpornchana, and S. Thajchayapong, “Performance analysis of private blockchain platforms in varying workloads,” in _2017 26th International Conference on Computer Communication and Networks (ICCCN)_ , 2017, pp. 1–6.
  * [10] T. T. A. Dinh, R. Liu, M. Zhang, G. Chen, B. C. Ooi, and J. Wang, “Untangling blockchain: A data processing view of blockchain systems,” _IEEE Transactions on Knowledge and Data Engineering_ , vol. 30, no. 7, pp. 1366–1385, 2018.
  * [11] H. Nguyen, T. Nguyen, T. Leppänen, J. Partala, and S. Pirttikangas, “Situation awareness for autonomous vehicles using blockchain-based service cooperation,” in _Advanced Information Systems Engineering_. Cham: Springer International Publishing, 2022, pp. 501–516.
  * [12] T. Nguyen, H. Nguyen, J. Partala, and S. Pirttikangas, “Trustedmaas: Transforming trust and transparency mobility-as-a-service with blockchain,” _Future Generation Computer Systems_ , vol. 149, pp. 606–621, 2023.
  * [13] P. Sethi, T. Nguyen, M. R. Chowdhury, S. Pirttikangas, and A. P. da Silva, “Fair consensus in blockchain with heterogeneous miners using reinforcement learning aided adaptive proof-of-work,” in _2024 IEEE 21st Consumer Communications & Networking Conference (CCNC)_, 2024, pp. 937–942.
  * [14] U. Majeed and C. S. Hong, “Flchain: Federated learning via mec-enabled blockchain network,” in _2019 20th Asia-Pacific Network Operations and Management Symposium (APNOMS)_ , 2019, pp. 1–4.
  * [15] Y. Lu, X. Huang, Y. Dai, S. Maharjan, and Y. Zhang, “Blockchain and federated learning for privacy-preserved data sharing in industrial iot,” _IEEE Transactions on Industrial Informatics_ , vol. 16, no. 6, pp. 4177–4186, 2020\.
  * [16] J. Kang, Z. Xiong, C. Jiang, Y. Liu, S. Guo, Y. Zhang, D. Niyato, C. Leung, and C. Miao, “Scalable and communication-efficient decentralized federated edge learning with multi-blockchain framework,” in _Blockchain and Trustworthy Systems_ , Z. Zheng, H.-N. Dai, X. Fu, and B. Chen, Eds. Singapore: Springer Singapore, 2020, pp. 152–165.
  * [17] P. Ramanan and K. Nakayama, “Baffle : Blockchain based aggregator free federated learning,” in _2020 IEEE International Conference on Blockchain (Blockchain)_ , 2020, pp. 72–81.
  * [18] S. Zhou, H. Huang, W. Chen, P. Zhou, Z. Zheng, and S. Guo, “Pirate: A blockchain-based secure framework of distributed machine learning in 5g networks,” _IEEE Network_ , vol. 34, no. 6, pp. 84–91, 2020.
  * [19] Y. Li, C. Chen, N. Liu, H. Huang, Z. Zheng, and Q. Yan, “A blockchain-based decentralized federated learning framework with committee consensus,” _IEEE Network_ , vol. 35, no. 1, pp. 234–241, 2021.
  * [20] F. Wilhelmi, E. Guerra, and P. Dini, “On the decentralization of blockchain-enabled asynchronous federated learning,” in _2023 IEEE 9th International Conference on Network Softwarization (NetSoft)_ , 2023, pp. 408–413.
  * [21] E. Peltonen, I. Ahmad, A. Aral, M. Capobianco, A. Y. Ding, F. Gil-Castiñeira, E. Gilman, E. Harjula, M. Jurmu, T. Karvonen, M. Kelanti, T. Leppänen, L. Lovén, T. Mikkonen, N. Mohan, P. Nurmi, S. Pirttikangas, P. Sroka, S. Tarkoma, and T. Yang, “The many faces of edge intelligence,” _IEEE Access_ , vol. 10, pp. 104 769–104 782, 2022.


# On the Utility of Self-supervised Models for Prosody-related Tasks

###### Abstract

Self-Supervised Learning (SSL) from speech data has produced models that have
achieved remarkable performance in many tasks, and that are known to
implicitly represent many aspects of information latently present in speech
signals. However, relatively little is known about the suitability of such
models for prosody-related tasks or the extent to which they encode prosodic
information. We present a new evaluation framework, “SUPERB-prosody,”
consisting of three prosody-related downstream tasks and two pseudo tasks. We
find that 13 of the 15 SSL models outperformed the baseline on all the
prosody-related tasks. We also show good performance on two pseudo tasks:
prosody reconstruction and future prosody prediction. We further analyze the
layerwise contributions of the SSL models. Overall we conclude that SSL speech
models are highly effective for prosody-related tasks. We release our
code111https://github.com/JSALT-2022-SSL/superb-prosody for the community to
support further investigation of SSL models’ utility for prosody.

Index Terms—  Speech Self-Supervised Learning, Representation Learning,
Pretrained Models, Prosody, Pragmatics

(0cm,10cm) Copyright 2022 IEEE. Published in the 2022 IEEE Spoken Language
Technology Workshop (SLT) (SLT 2022), scheduled for 19-22 January 2023 in
Doha, Qatar. Personal use of this material is permitted. However, permission
to reprint/republish this material for advertising or promotional purposes or
for creating new collective works for resale or redistribution to servers or
lists, or to reuse any copyrighted component of this work in other works, must
be obtained from the IEEE. Contact: Manager, Copyrights and Permissions / IEEE
Service Center / 445 Hoes Lane / P.O. Box 1331 / Piscataway, NJ 08855-1331,
USA. Telephone: + Intl. 908-562-3966.

##  1 Introduction

Self-supervised Learning (SSL) has revolutionized research in many areas of
artificial intelligence, including speech processing. SSL pre-trained speech
models have shown remarkable performance and generalizability across a wide
range of tasks [1, 2, 3]. However, we do not currently have a good
understanding of what knowledge these models capture nor of the limits of
their power. This is true in particular for the prosodic aspects of speech.

The speech signal contains not only lexical but prosodic information. Broadly
speaking, the latter has three realms of function: paralinguistic,
phonological, and pragmatic. Paralinguistic functions, such as marking speaker
identity and expressing emotion, are largely conveyed by prosodic settings
that are stable over the span of many utterances, and are often evident from
any sample of just a few syllables. Phonological functions, notably marking
the identity of syllables and words with tones and stress patterns, are
largely conveyed by prosodic features whose temporal occurrence is tightly
linked to the units they mark. The utilities of SSL models for these two
realms have been demonstrated, by many recent studies using tasks from the
SUPERB [1] benchmark, among others.

However, for the third realm, the realm of pragmatic function, the question of
the utility of SSL models has remained open. Functions in this realm, include
managing turn-taking, marking topic structure and information structure, and
expressing engagement, stance, attitude, and intent. These pragmatic functions
are especially important in dialog, and we expect that future dialog systems
will need more prosodic competence, in order to enable more satisfying user
experiences and to support interaction in novel genres and situations [4]. In
many cases, these functions are expressed using multistream temporal
configurations of low-level prosodic features, where these configurations can
last from a few hundred milliseconds to several seconds [5, 6], and may be
only loosely aligned with the lexical content. As these configurations are
fundamentally different from the forms of prosody in the other two realms, it
is an open question whether SSL models are also useful for this realm.

Accordingly our research question is whether pre-trained models have utility
for prosody-conveyed pragmatic functions. We investigate this in four ways.
First, we assemble a set of prosody-intensive tasks and measure how well pre-
trained models support them. Second, we use pitch and energy reconstruction
pseudo-tasks to measure how well these models represent prosodic information.
Third, we evaluate the utility of these models for the prediction of future
pitch and energy. Fourth, we probe the pre-trained models to see in which
layers prosodic information is likely represented.

The main contributions of this paper are: 1) The finding that pre-trained SSL
models indeed can provide value for prosody-intensive tasks, often reaching
state-of-the-art performance. 2) Results for 15 recent SSL models span
different model architectures and pre-training objectives. 3) Analysis of the
representation of prosody in SSL models, including layerwise analysis. 4) An
open-source evaluation framework, SUPERB-prosody, examines the prosodic
prowess of SSL models.

##  2 RELATED WORK

The most directly relevant study [7] aimed primarily to evaluate a model for
producing de-identified representations of speech, but includes three aspects
that are very relevant to our research question. First, for evaluation
purposes, several “spoken language understanding” tasks were selected, of
which three of these were both pragmatics-relevant and prosody-intensive.
Second, six pre-trained models were tested against this task set, showing
various levels of performance. Third, probing their own model, VQP, provided
evidence that it was encoding, to some extent, several prosodic features.
Taken together these results suggest that the answer to our research question
is yes, but the case is not settled for two reasons. First, neither their
performance results nor their probing results were compared against non-pre-
trained baselines, leaving open the question of whether the pre-trained models
were in fact providing any benefit. Second, many aspects of their methods are
unclear, and no code is available to enable replication. Thus, we need further
investigation, and an open and transparent evaluation framework for SSL
models. Very recent work has found that SSL models are helpful for predicting
some perceptions of speaking style, but require additional downstream
sequence-modeling layers for best performance [8].

Evaluation benchmarks have been critical in supporting and evaluating the rise
of SSL in the speech field. NOSS [9] is a benchmark for non-semantic
downstream tasks. SUPERB [1] broadly examines SSL models for content, speaker,
semantic and paralinguistic aspects, demonstrating that SSL models generalize
across diverse downstream tasks. SUPERB-sg [2] enhances the SUPERB benchmark
with more challenging semantic and generative tasks. SLUE [10], another recent
benchmark, targets spoken language understanding tasks. However, up to now,
the speech community lacks an evaluation benchmark/framework to measure the
prosodic utility of SSL models.

Analysis of speech SSL models has recently received significant attention,
with previous works examining various aspects [11, 12, 13, 7]. [11, 13] mainly
focus on analyzing lexical information in SSL models. [12] investigate
acoustic, syntactic, and semantic characteristics, but they only experiment on
two SSL models and probe only by utterance-wise regression tasks. Accordingly,
more focused analysis is needed, especially regarding prosodic information.

Fig. 1: Diagram of SUPERB-Prosody framework. We extract the hidden
representations from a frozen SSL model, and lightweight linear models are
used for each downstream task. $P_{i}$ means the value of the rule-based
prosodic feature at time frame $i$.

Model | Network | #Params | Stride | Input | Corpus | Pretraining
---|---|---|---|---|---|---
FBANK | - | 0 | 10ms | waveform | - | -
APC [14] | 3-GRU | 4.11M | 10ms | FBANK | LS 360 hr | F-G
VQ-APC [15] | 3-GRU | 4.63M | 10ms | FBANK | LS 360 hr | F-G + VQ
NPC [16] | 4-Conv, 4-Masked Conv | 19.38M | 10ms | FBANK | LS 360 hr | M-G + VQ
Mockingjay [17] | 12-Trans | 85.12M | 10ms | FBANK | LS 360 hr | time M-G
TERA [18] | 3-Trans | 21.33M | 10ms | FBANK | LS 960 hr | time/freq M-G
modified CPC [19] | 5-Conv, 1-LSTM | 1.84M | 10ms | waveform | LL 60k hr | F-C
wav2vec [20] | 19-Conv | 32.54M | 10ms | waveform | LS 960 hr | F-C
vq-wav2vec [21] | 20-Conv | 34.15M | 10ms | waveform | LS 960 hr | F-C + VQ
DistilHuBERT [22] | 7-Conv 2-Trans | 23.49M | 20ms | waveform | LS 960 hr | KD
wav2vec 2.0 Base [23] | 7-Conv 12-Trans | 95.04M | 20ms | waveform | LS 960 hr | M-C + VQ
wav2vec 2.0 Large [23] | 7-Conv 24-Trans | 317.38M | 20ms | waveform | LL 60k hr | M-C + VQ
HuBERT Base [24] | 7-Conv 12-Trans | 94.68M | 20ms | waveform | LS 960 hr | M-P + VQ
HuBERT Large [24] | 7-Conv 24-Trans | 316.61M | 20ms | waveform | LL 60k hr | M-P + VQ
WavLM Base [25] | 7-Conv 12-Trans | 94.68M | 20ms | waveform | LL 60k hr | M-P + VQ
WavLM Large [25] | 7-Conv 24-Trans | 316.62M | 20ms | waveform | Mix 94k hr | M-P + VQ

Table 1: SSL models examined. #Params includes parameters for both pre-
training and inference. LS = LibriSpeech and LS = LibriLight. For the pre-
training methods, VQ = vector quantization, F = future, M = masked, G =
generation, C = contrastive discrimination, P = token
prediction/classification, and KD = knowledge distillation.

##  3 The SUPERB-Prosody Framework

This section introduces our tasks — classification, prosody reconstruction,
and future value prediction task — and our evaluation framework, including the
upstream/downstream setup.

###  3.1 Classification tasks

To evaluate the pragmatic and prosody-related abilities of pretrained models,
we need a set of “prosody-intensive” tasks, that is, tasks where it is known
that prosodic abilities are useful. We chose three well-curated, open-source,
utterance-classification datasets, involving sentiment, sarcasm, and
persuasiveness. To briefly describe each task:

Sentiment Analysis (SA) involves detecting the degree of positive or negative
feeling in an utterance. While sentiment and emotion are often conflated, and
similar methods may work for both tasks, sentiment is less visceral and of
greater practical importance. Specifically we chose the CMU-MOSEI [26] corpus,
in which each utterance is labeled from $-3$ to $3$, representing the degree
of sentiment. Following previous works, we experiment with two settings:
binary classification, with the dataset split by the labels in $[-3,0)$ and
$(0,3]$, and seven-category classification. The evaluation metric is accuracy.

Sarcasm Detection (SarD) is perhaps the most obviously prosody-intensive task,
as a mismatch between the lexical content and the prosodic message is
frequently the major marker of sarcasm. We chose the MUStARD [27] corpus, in
which each utterance has a label of $0$ or $1$, for sarcasm or non-sarcasm. We
follow the speaker-dependent setup in the original paper [27], using five-fold
cross-validation for evaluation. The evaluation metric is the F1 score.

Persuasiveness Prediction (PP) is the task of detecting whether a presentation
is likely to be convincing to others. Correlates include how pleasant the
speaker’s voice is and their perceived confidence and dominance, all of which
involve prosody. We chose the POM [28] corpus. The labels are 0 or 1, for
persuasive or non-persuasive. The evaluation metric is accuracy.

###  3.2 Prosody reconstruction

Prosody Reconstruction (ProP) is a pseudo-task designed to test whether SSL
models embed specific prosody features in their hidden representation. As our
target, we chose the two most commonly used prosody features, pitch and
energy. Given the SSL features, we use a lightweight linear downstream model
to predict each. The pitch is represented in log scale, using as targets the
values computed by
pYAAPT222http://bjbschmitt.github.io/AMFM_decompy/pYAAPT.html. For energy, we
use the librosa toolkit333https://github.com/librosa/librosa, again using a
log scale. As data, we use LibriTTS [29], a multi-speaker text-to-speech
dataset, and for both features, the evaluation metric is the Mean Square Error
(MSE) of the differences. No loss is computed for unvoiced frames, that is,
frames where pYAAPT detects no pitch.

###  3.3 Future value prediction

Future Value Prediction (FVP) is designed to test whether the output from SSL
models can predict future prosody. The task setting is similar to the prosody
reconstruction task, using pitch and energy features as the prediction targets
and using LibriTTS dataset with MSE objective. The task is, given the
information from $t=1$ to the current frame $i$, to predict the value at
$i+h$, where $h$ is the prediction horizon. Four prediction horizons $h$,
namely $0.12$, $0.24$, $0.50$, and $1.00$ seconds, are used. Because most SSL
speech models are not causal (due to the inclusion of either self-attention or
bi-directional connections), we only test causal SSL models or attention-based
SSL models for which we can apply an attention mask to avoid cheating with
future information444We exclude WavLM for this task because it uses a gated
relative position bias, so simply modifying the attention mask would cause a
model mismatch between training and inference.. The evaluation metrics are
MSE.

Fig. 2: Results for Sentiment Analysis (SA), Sarcasm Detection (SarD),
Persuasiveness Prediction (PP), and Prosody Reconstruction (PR). State-of-the-
art (SOTA) values are from [26] for SA, [30] for SarD, [31] for PP, and
REAPER[32] for pitch reconstruction. Text-only baselines are only for the
prosody-intensive downstream tasks, and thus not available for PR.

###  3.4 Evaluation framework

As suggested by Figure 1, our framework consists of (1) an upstream SSL speech
model and (2) a linear downstream model for probing. Following the procedure
of SUPERB [1], the parameters of the upstream models are fixed for all the
downstream tasks. For each frame of the input, we extract the representations
$\mathbf{x_{i}}$ from each hidden layer $i$ of the upstream model, and we
aggregate those hidden representations per utterance into
$\mathbf{y}=\sum_{i}^{L}w_{i}\mathbf{x_{i}}$ where the $w_{i}$ are trained per
task. The resulting $\mathbf{y}$, a two-dimensional matrix (time $\times$
aggregated features), is the input for the downstream model.

####  3.4.1 Upstream models: Speech SSL models

The SSL models tested are summarized in Table 1. These are a diverse
collection, including modified CPC [19], APC [14], VQ-APC [15], NPC [16], TERA
[18], vq-wav2vec [21], DistilHuBERT [22], HuBERT [24], wav2vec [20], wav2vec
2.0 [23], and WavLM [25]. The selected SSL models span different network
architectures and pre-training objectives.

####  3.4.2 Downstream models: linear probing model

For the classification tasks, SA, SarD, and PP, the representation
$\mathbf{y}$ is mean-pooled along the time axis, forming a dense vector of
dimension (time × aggregated features) to (aggregated features). This vector
is then fed to a simple linear model to project from model dimension to 1. The
training objective is Cross-Entropy Minimization.

In ProR and FVP, the goal is to predict the fine-grained prosodic information.
We use the frame-level representation from each time step of $\mathbf{y}$ as
the input, and the downstream model is a linear model, projecting from model
dimension to 1. MSE minimization is the training objective. We try multiple
learning rates for each task ($[1e^{-2}$, $1e^{-3}$, $1e^{-4}$, $1e^{-5}$,
$1e^{-6}]$), and report the best performance. The training step is 3000 for
SarD and 50000 for the other tasks.

####  3.4.3 Baselines

As a baseline feature set, we use“FBANK,” the 80-dimensional Log Mel
Filterbank features with delta and delta-delta features (240 dimensions in
total), chosen because this is known to work well for many speech tasks. For
the classification tasks, FBANK features are average-pooled across each
utterance. For pitch reconstruction, we also compare the performance of
another high-quality off-the-shelf pitch extractor, Talkin’s REAPER. For
future value prediction (FVP) we design a baseline, FBANK + RNN, which feeds
filterbank features from $t=1$ to $i$ into a one-layer uni-directional
Recurrent Neural Network (RNN) with 128 hidden size to predict the value at
$t=i+h$.

As an additional point of comparison, we also explore the text-only
performance for each classification task. Since all datasets contain ground
truth speech transcription, we take these transcriptions as the input data. As
the NLP model we use the pre-trained RoBERTa [33] for SarD and SA, and
Longformer [34] for PP555Longformer is based on RoBERTa, but can accept up to
4096 tokens (versus 512 tokens for RoBERTa), and is used here because the
transcriptions of PP are too long for RoBERTa.. We follow the typical method
to fine-tune the pre-trained NLP model: extract the sentence embedding from
the [CLS] token’s embedding, and feed it to a linear classification model. No
parameters are frozen during fine-tuning. We vary the learning rate
($[1e^{-3}$, $1e^{-4}$, $1e^{-5}$, and $1e^{-6}]$) and report the best
performance.

Fig. 3: The contribution analysis for each task. The darker the color, the
higher the contribution. We only include models which have 12 layers of
representations.

Model | Pitch w/ Prediction Horizon (s)$\downarrow$ | Energy w/ Prediction Horizon (s)$\downarrow$
---|---|---
0.12 |  0.24 |  0.50 |  1.00 |  0.12 |  0.24 |  0.50 |  1.00
FBANK + RNN | 0.049 | 0.104 | 0.142 | 0.157 | 0.52 | 1.42 | 2.06 | 2.46
APC | 0.033 | 0.043 | 0.052 | 0.053 | 0.91 | 1.46 | 1.98 | 2.37
modified CPC | 0.038 | 0.051 | 0.062 | 0.065 | 0.79 | 1.37 | 1.94 | 2.47
wav2vec | 0.053 | 0.064 | 0.075 | 0.075 | 0.68 | 1.15 | 1.70 | 2.31
Mockingjay | 0.069 | 0.077 | 0.081 | 0.099 | 0.48 | 1.92 | 2.17 | 2.43
wav2vec 2.0 Base | 0.038 | 0.047 | 0.047 | 0.049 | 0.44 | 0.85 | 1.24 | 1.49
wav2vec 2.0 Large | 0.035 | 0.039 | 0.045 | 0.046 | 0.43 | 0.80 | 1.23 | 1.31
HuBERT Base | 0.029 | 0.036 | 0.041 | 0.042 | 0.39 | 0.70 | 1.12 | 1.42
HuBERT Large | 0.025 | 0.027 | 0.028 | 0.037 | 0.41 | 0.77 | 1.21 | 1.36

Table 2: The MSE loss for FVP. Lower values indicate better prediction, of
future pitch or energy.

| SarD F1$\uparrow$ | PP Acc$\uparrow$ | SA Acc$\uparrow$
---|---|---|---
Layer selection | (0,1,12) | (10,11,12) | (0,1,12) | (10,11,12) | (0,1,8) | (7,8,9)
wav2vec 2.0 Base | 70.6 | 66.3 | 81.3 | 81.3 | 73.6 | 74.0
HuBERT Base | 72.0 | 70.0 | 85.8 | 83.8 | 75.2 | 76.2

Table 3: Experimental results for layer-limited feature integration for SarD,
PP, and SA. The best layer, underlined, is chosen based on the contribution
analysis.

Method | ZH-p$\downarrow$ | PO-p$\downarrow$ | ZH-e$\downarrow$ | PO-e$\downarrow$
---|---|---|---|---
FBANK | 0.050 | 0.096 | 0.849 | 0.477
vq-wav2vec | 0.022 | 0.097 | 0.490 | 0.339
wav2vec 2.0 Base | 0.012 | 0.039 | 0.338 | 0.160
HuBERT Base | 0.009 | 0.042 | 0.232 | 0.149
WavLM Base | 0.008 | 0.019 | 0.233 | 0.192

Table 4: MSE for prosody reconstruction, showing cross-lingual
transferability. p = pitch, e = energy.

##  4 Main Results

###  4.1 SSL models perform well on prosody-intensive tasks

The experimental results for SA, SarD, and PP are shown in Figure 2. In Figure
2 (a) and (b), we can see that all SSL models yield better SA performance than
the baseline FBANK features in both 2-label and 7-label evaluations. The large
SSL models, wav2vec 2.0, HuBERT, and WavLM, even improve on the state-of-the-
art (SOTA) performance [26] in the 2-label setup. In the 7-label setup, WavLM
Large outperforms audio-only SOTA performance. Around one-third of the SSL
models show better performance than the text-only baseline, confirming the
value of acoustic information for SA.

Figure 2 (c) shows the SarD result. Although Mockingjay, wav2vec, and vq-
wav2vec show inferior performance to the baseline FBANK, the other SSL models
do well, with DistilHuBERT, HuBERT, and WavLM improving on the previous audio-
only SOTA [30]. In SarD, all acoustic models, both SSL models and FBANK,
outperform the text-only baseline, confirming that SarD requires acoustic
information beyond content for prediction.

Lastly, for the PP results, Figure 2 (d), shows that all SSL models yield
better performance than the FBANK features and the previous audio-only SOTA
[31]. APC, VQ-APC, vq-wav2vec, and HuBERT obtains superior or equal accuracy
to the text-only baseline.

###  4.2 SSL models encode prosodic information

While the results above suggest that the SSL models are encoding prosodic
information, there is also direct evidence from the ProR and FVP tasks, as
seen in Figure 2 and Table 2.

For PR, the two features, pitch and energy, have slightly different results.
In pitch reconstruction, Figure 2 (e), all the SSL models perform better than
baseline FBANK except for vq-wav2vec. Several SSL models surpass REAPER
performance, with the WavLM the best. As for energy reconstruction, Figure 2
(f) shows that all SSL models greatly improved over baseline FBANK. Although
generation-based SSL models perform well on pitch reconstruction, they did
relatively worse on energy reconstruction. On the other hand, the SSL models
pre-trained by masked contrastive discrimination/token prediction show strong
performance on both pitch and energy reconstruction. Overall, we observe that
SSL models indeed encode prosodic information.

FVP is more challenging than ProR since it requires the model to capture both
global and local prosodic information for successful future prediction. From
Table 2, we see, unsurprisingly, that the larger prediction horizons make
prediction harder. For pitch, HuBERT Large gets the best performance,
outperforming other SSL models and baseline FBANK+RNN at all four horizons.
Although the pre-training objectives of APC, modified CPC, and wav2vec involve
future generation/discrimination, they still result in inferior performance to
wav2vec 2.0 and HuBERT. As for future energy prediction, only wav2vec 2.0 and
HuBERT consistently outperform baseline FBANK. In general, we observe that
some SSL models are not good at FVP, but wav2vec 2.0 and HuBERT outperform
FBANK by a large margin. This result suggests that wav2vec 2.0 and HuBERT do
have the capability to encode and summarize relevant prosodic information.

##  5 Further Analysis

###  5.1 Layerwise contribution analysis

In order to estimate the contribution of each layer, we consider two factors.
First, we use the weight $w_{i}$ from each layer (through the weighted-sum
mechanism) to the downstream model. Second, because typical feature magnitudes
may vary, we also consider the values of the hidden representation
$\mathbf{x}$ in each layer, as measured by the L2-norm of feature values for
the testing data. After getting the whole L2-norm features for each layer for
each sample, we take the mean across samples to get the feature magnitude
estimation. The contribution for each layer is then defined as:
$c_{i}=||\mathbf{x_{i}}||_{2}\times w_{i}$, where $i$ means the layer number.

The results are shown in Figure 3. From Figure 3(a), we can see that for most
SSL models, the contribution is strongest in the first few layers for both
pitch and energy reconstruction. This shows that SSL tends to best represent
prosodic information in the front. One exception is Mockingjay, where the
largest contribution is located in the last layer. Because Mockingjay’s pre-
training objective is to reconstruct frame-wise features, it is unsurprising
that the later layers contain a good representation of low-level prosody.

However, for the classification tasks SarD and PP, Figure 3 (c) and (d) shows
that the distribution of layer contributions is smooth, suggesting that both
tasks need information across multiple layers. As previous work has shown that
later layers represent more content information [11, 25], this suggests that
both prosodic and content information are needed for SarD and PP. As for SA,
in Figure 3(e), we observe a high contribution value in the latter layers for
the high-performing models HuBERT and WavLM. This suggests that SA might only
require content information to perform well.

###  5.2 Feature integration

To further examine whether the encodings in the first few layers bring the
most benefit, we designed a new experiment for three prosody-intensive tasks
using wav2vec 2.0 and HuBERT. We compare two settings, concatenation of 1) the
features from the first two layers and the best layer we discovered666The best
layer is determined by the contribution analysis above., and 2) the best layer
with its two neighbor layers. These concatenated features are passed to the
downstream model. The downstream model size of the two settings is the same,
so we can compare the performance fairly. If the first setting is better than
the second setting, this would indicate that early-layer (low-level)
information indeed most benefits the final results.

The results are shown in Table 3. In SarD and PP, we see the integration of
the first two layers yields better performance, which means the low-level
information improves the modeling of SarD and PP. Yet the use of low-level
information does not improve SA performance, suggesting that SA may rely on
content rather than just low-level information.

###  5.3 Cross-lingual transferability

Further, we did a preliminary investigation of whether SSL models may have
cross-lingual transferability for prosodic information, using the ProR task as
in previous experiments: specifically, attempting pitch and energy
reconstruction. While the pitch of one frame is primarily a physical
phenomenon, in context there may be language dependencies. All the SSL models
being pre-trained in English, we try this for Mandarin and Polish, using data
from AISHELL-3 [35] multilingual LibriSpeech (MLS) [36], respectively. Table 4
summarizes the results. We note that WavLM is good at pitch reconstruction,
and HuBERT is superior at energy reconstruction.

##  6 Conclusion

In this work, we explore the utility of SSL models for prosody-conveyed
pragmatic functions. Using the newly-proposed SUPERB-prosody evaluation
framework, we find that SSL models provide significant value for prosody-
intensive tasks, and that they are good at extracting prosodic information in
pseudo tasks. Furthermore, we analyze the layer contribution and discover that
most SSL models tend to store prosodic information in the first few layers.
However, the field still lacks a good understanding of why different SSL
models are better for different tasks [37], and this is an important topic for
future work.

##  7 Acknowledgement

We thank the Taiwan Web Service and the National Center for High-performance
Computing (NCHC) of the National Applied Research Laboratories (NARLabs) in
Taiwan for providing computing and storage resources. Part of the work
presented here was carried out during the 2022 Jelinek Memorial Summer
Workshop on Speech and Language Technologies at Johns Hopkins University,
which was supported with unrestricted gifts from Amazon, Microsoft, and
Google.

## References

  * [1] Shu wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y. Lin, Andy T. Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, Tzu-Hsien Huang, Wei-Cheng Tseng, Ko tik Lee, Da-Rong Liu, Zili Huang, Shuyan Dong, Shang-Wen Li, Shinji Watanabe, Abdelrahman Mohamed, and Hung yi Lee,  “SUPERB: Speech Processing Universal PERformance Benchmark,”  in Proc. Interspeech, 2021, pp. 1194–1198.
  * [2] Hsiang-Sheng Tsai, Heng-Jui Chang, Wen-Chin Huang, Zili Huang, Kushal Lakhotia, Shu-wen Yang, Shuyan Dong, Andy Liu, Cheng-I Lai, Jiatong Shi, et al.,  “SUPERB-SG: Enhanced speech processing universal performance benchmark for semantic and generative capabilities,”  in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022, pp. 8479–8492.
  * [3] Abdelrahman Mohamed, Hung-yi Lee, Lasse Borgholt, Jakob D Havtorn, Joakim Edin, Christian Igel, Katrin Kirchhoff, Shang-Wen Li, Karen Livescu, Lars Maaløe, et al.,  “Self-supervised speech representation learning: A review,”  arXiv preprint arXiv:2205.10643, 2022.
  * [4] Matthew Marge, Carol Espy-Wilson, Nigel G. Ward, et al.,  “Spoken language interaction with robots: Recommendations for future research,”  Computer Speech & Language, vol. 71, pp. 101255, 2022.
  * [5] Nigel G. Ward,  Prosodic Patterns in English Conversation,  Cambridge University Press, 2019.
  * [6] Oliver Niebuhr,  “Stepped intonation contours: A new field of complexity,”  in Tackling the Complexity of Speech, R. Skarnitzl and O. Niebuhr, Eds., pp. 39–74. Charles University Press, 2015.
  * [7] Jack Weston, Raphael Lenain, Udeepa Meepegama, and Emil Fristed,  “Learning de-identified representations of prosody from raw audio,”  in Proceedings of the 38th International Conference on Machine Learning, Marina Meila and Tong Zhang, Eds. 2021, vol. 139 of Proceedings of Machine Learning Research, pp. 11134–11145, PMLR.
  * [8] Diego Aguirre, Nigel Ward, Jonathan E Avila, and Heike Lehnert-LeHouillier,  “Comparison of models for detecting off-putting speaking styles,”  Proc. Interspeech, pp. 2303–2307, 2022.
  * [9] Joel Shor, Aren Jansen, Ronnie Zvi Maor, Oran Lang, Omry Tuval, Félix de Chaumont Quitry, Marco Tagliasacchi, Ira Shavitt, and Dotan Emanuel,  “Towards learning a universal non-semantic representation of speech,”  in Proc. Interspeech, 2020.
  * [10] Suwon Shon, Ankita Pasad, Felix Wu, Pablo Brusco, Yoav Artzi, Karen Livescu, and Kyu J Han,  “SLUE: New benchmark tasks for spoken language understanding evaluation on natural speech,”  in International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 7927–7931.
  * [11] Ankita Pasad, Ju-Chieh Chou, and Karen Livescu,  “Layer-wise analysis of a self-supervised speech representation model,”  in 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2021, pp. 914–921.
  * [12] Jui Shah, Yaman Kumar Singla, Changyou Chen, and Rajiv Ratn Shah,  “What all do audio transformer models hear? probing acoustic representations for language delivery and its structure,”  arXiv preprint arXiv:2101.00387, 2021.
  * [13] Yu-An Chung, Yonatan Belinkov, and James Glass,  “Similarity analysis of self-supervised speech representations,”  in International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 3040–3044.
  * [14] Yu-An Chung, Wei-Ning Hsu, Hao Tang, and James Glass,  “An Unsupervised Autoregressive Model for Speech Representation Learning,”  in Interspeech, 2019, pp. 146–150.
  * [15] Yu-An Chung, Hao Tang, and James Glass,  “Vector-quantized autoregressive predictive coding,”  in Interspeech, 2020, pp. 3760–3764.
  * [16] Alexander H Liu, Yu-An Chung, and James Glass,  “Non-autoregressive predictive coding for learning speech representations from local dependencies,”  arXiv preprint arXiv:2011.00406, 2020.
  * [17] Andy T Liu, Shu-wen Yang, Po-Han Chi, Po-chun Hsu, and Hung-yi Lee,  “Mockingjay: Unsupervised speech representation learning with deep bidirectional transformer encoders,”  in International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 6419–6423.
  * [18] Andy T Liu, Shang-Wen Li, and Hung-yi Lee,  “Tera: Self-supervised learning of transformer encoder representation for speech,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 2351–2366, 2021.
  * [19] Morgane Rivière, Armand Joulin, Pierre-Emmanuel Mazaré, and Emmanuel Dupoux,  “Unsupervised pretraining transfers well across languages,”  in ICASSP, 2020, pp. 7414–7418.
  * [20] Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli,  “wav2vec: Unsupervised pre-training for speech recognition.,”  in Interspeech, 2019.
  * [21] Alexei Baevski, Steffen Schneider, and Michael Auli,  “vq-wav2vec: Self-supervised learning of discrete speech representations,”  in ICLR, 2020.
  * [22] Heng-Jui Chang, Shu-wen Yang, and Hung-yi Lee,  “DistilHuBERT: Speech representation learning by layer-wise distillation of hidden-unit BERT,”  in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 7087–7091.
  * [23] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli,  “wav2vec 2.0: A framework for self-supervised learning of speech representations,”  in NeurIPS, 2020.
  * [24] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed,  “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 3451–3460, 2021.
  * [25] Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al.,  “Wavlm: Large-scale self-supervised pre-training for full stack speech processing,”  arXiv preprint arXiv:2110.13900, 2021.
  * [26] AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency,  “Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph,”  in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Melbourne, Australia, July 2018, pp. 2236–2246, Association for Computational Linguistics.
  * [27] Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihalcea, and Soujanya Poria,  “Towards multimodal sarcasm detection (an _Obviously_ perfect paper),”  in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, July 2019, pp. 4619–4629, Association for Computational Linguistics.
  * [28] Sunghyun Park, Han Suk Shim, Moitreya Chatterjee, Kenji Sagae, and Louis-Philippe Morency,  “Computational analysis of persuasiveness in social multimedia: A novel dataset and multimodal prediction approach,”  in Proceedings of the 16th International Conference on Multimodal Interaction, 2014, pp. 50–57.
  * [29] Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J. Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu,  “LibriTTS: A corpus derived from librispeech for text-to-speech,”  CoRR, vol. abs/1904.02882, 2019.
  * [30] Ning Ding, Sheng-wei Tian, and Long Yu,  “A multimodal fusion method for sarcasm detection based on late fusion,”  Multimedia Tools and Applications, vol. 81, no. 6, pp. 8597–8616, 2022.
  * [31] Behnaz Nojavanasghari, Deepak Gopinath, Jayanth Koushik, Tadas Baltrušaitis, and Louis-Philippe Morency,  “Deep multimodal fusion for persuasiveness prediction,”  in Proceedings of the 18th ACM International Conference on Multimodal Interaction, 2016, pp. 284–288.
  * [32] David Talkin,  “Reaper: Robust epoch and pitch estimator,”  https://github.com/google/REAPER, 2015.
  * [33] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov,  “Roberta: A robustly optimized BERT pretraining approach,”  arXiv preprint arXiv:1907.11692, 2019.
  * [34] Iz Beltagy, Matthew E Peters, and Arman Cohan,  “Longformer: The long-document transformer,”  arXiv preprint arXiv:2004.05150, 2020.
  * [35] Yao Shi, Hui Bu, Xin Xu, Shaoji Zhang, and Ming Li,  “Aishell-3: A multi-speaker Mandarin TTS corpus and the baselines,”  2015\.
  * [36] Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert,  “MLS: A large-scale multilingual dataset for speech research,”  ArXiv, vol. abs/2012.03411, 2020.
  * [37] Sri Harsha Dumpala, Chandramouli Shama Sastry, Rudolf Uher, and Sageev Oore,  “On Combining Global and Localized Self-Supervised Models of Speech,”  in Proc. Interspeech 2022, 2022, pp. 3593–3597.