# Knowledge Distillation and Training Balance for Heterogeneous Decentralized
Multi-Modal Learning over Wireless Networks ††thanks: B. Yin and Z. Chen are
with Cooperative Medianet Innovation Center and Shanghai Key Laboratory of
Digital Media Processing and Transmission, Shanghai Jiao Tong University,
Shanghai, China (e-mail: {yinbsh, zhiyongchen}@sjtu.edu.cn). M. Tao is with
Department of Electronic Engineering, Shanghai Jiao Tong University, China
(e-mail: mxtao@sjtu.edu.cn). _(Corresponding author: Zhiyong Chen, Meixia
Tao)_

Benshun Yin, Zhiyong Chen, _Senior Member, IEEE_ , and Meixia Tao, _Fellow,
IEEE_

###### Abstract

Decentralized learning is widely employed for collaboratively training models
using distributed data over wireless networks. Existing decentralized learning
methods primarily focus on training single-modal networks. For the
decentralized multi-modal learning (DMML), the modality heterogeneity and the
non-independent and non-identically distributed (non-IID) data across devices
make it difficult for the training model to capture the correlated features
across different modalities. Moreover, modality competition can result in
training imbalance among different modalities, which can significantly impact
the performance of DMML. To improve the training performance in the presence
of non-IID data and modality heterogeneity, we propose a novel DMML with
knowledge distillation (DMML-KD) framework, which decomposes the extracted
feature into the modality-common and the modality-specific components. In the
proposed DMML-KD, a generator is applied to learn the global conditional
distribution of the modality-common features, thereby guiding the modality-
common features of different devices towards the same distribution. Meanwhile,
we propose to decrease the number of local iterations for the modalities with
fast training speed in DMML-KD to address the imbalanced training. We design a
balance metric based on the parameter variation to evaluate the training speed
of different modalities in DMML-KD. Using this metric, we optimize the number
of local iterations for different modalities on each device under the
constraint of remaining energy on devices. Experimental results demonstrate
that the proposed DMML-KD with training balance can effectively improve the
training performance of DMML.

###### Index Terms:

Decentralized learning, multi-modal learning, knowledge distillation, training
balance

##  I Introduction

Machine learning, exemplified by deep neural networks, has achieved
significant breakthroughs in various domains [1], such as object detection and
image classification. With the rapid expansion of the Internet of Things
(IoT), a substantial volume of data is being generated by distributed IoT
devices, such as wireless security cameras. To harness the vast amount of data
for intelligent applications, edge learning [2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
is emerging as a promising research field.

Federated learning (FL) [5, 7, 6] is a edge learning algorithm that can
collaboratively train a neural network with the local dataset of wireless
devices. Compared with uploading large amounts of raw data to the edge server
for centralized learning, FL can protect the data privacy of users. However,
certain problems persist in FL systems. Firstly, relying on an edge server for
model aggregation may introduce a single point of failure, where an attack on
the server can disrupt the entire FL system [12]. Secondly, considering the
vast scalability of the edge computing networks, a single edge server cannot
effectively aggregate all the updates from a large number of devices. Thirdly,
transmitting the model parameters to the edge server is susceptible to
security vulnerabilities in realistic wireless networks, such as malicious
threats that can tamper with or steal local update information[13].
Consequently, FL may be not applicable in some scenarios, making it necessary
to apply the decentralized learning [8, 9, 10] approach that eliminates the
reliance on a server.

On the other hand, with the rapid development of sensor technology, the
physical environment can be detected from various modalities. Similar to how
humans perceive the world through multiple senses, multi-modal sensory data
can provide comprehensive information from multiple aspects. Multi-modal
learning [14, 15] has become a recent trend in machine learning, which can
obtain performance improvement compared to single-modal learning by
integrating multi-modal data. For example, human action recognition [14] and
multi-modal sentiment analysis [15] can extract multi-modal feature
representation from different modalities including audio and vision, thereby
improving prediction performance. Multiple sensor data including LiDAR, GPS
and camera images can be used to speed up the sector selection in the mmWave
band for vehicular mobility scenarios[16].

Existing decentralized learning approaches [8, 9, 10] have not considered the
characteristics of multi-modal learning. Compared to decentralized single-
modal learning, the efficient training of decentralized multi-modal learning
(DMML) can not be achieved by simply changing the training model from single-
modal to multi-modal. Besides the challenge of non-independent and identically
distributed (non-IID) data among different devices, the DMML systems also face
the issue of modality heterogeneity. Due to variations in detection
environments and device types, the modalities of data collected by different
wireless devices may differ. For example, some automobiles may be equipped
with visual sensors but lack radar sensors. In dynamic environments, certain
sensors may not always be available, leading to the absence of certain
modalities in the collected data. Therefore, the data on different devices may
exhibit different combinations of modalities. While many decentralized
learning approaches have been proposed to address the challenge of non-IID
data across devices, they may have a potential limitation, as they typically
require all participating devices to have the same modality. In multi-modal
learning, different neural networks are typically used to extract feature
representations for different modalities. For instance, the transformer model
[17] can be used to process textual data, while convolutional neural networks
are employed for image data processing. Modality heterogeneity implies that
the models trained on different devices are different. Besides, jointly
training a multi-modal network with multiple input modalities enables the
network to learn correlated features across different modalities. However, the
absence of certain modalities for some devices increases the difficulty of
learning the cross-modal features in DMML.

Figure 1: Training imbalance of DMML with CREMA-D dataset [18].

In addition to non-IID and modality heterogeneity, another challenge of DMML
is the imbalanced training of different modalities [19, 20, 21]. The
performance of the individually trained single-modal model may surpass that of
the jointly trained multi-modal network [20]. In [19], it is theoretically
proved that when training a single-modal network, features can be sufficiently
learned, resulting in good performance. However, modality competition exists
when training a multi-modal network. Only a subset of modalities’ feature
extraction networks can adequately learn the feature representations, leading
to imbalanced training across different modalities. As shown in Fig. 1, we
estimate the accuracy of the two modalities under the joint training. The
training of the visual modality is insufficient, which can not achieve the
accuracy obtained from training the visual-only network. In DMML systems, the
local training of a device with multiple modalities is inherently imbalanced.
Due to the modality heterogeneity and non-IID data, the training imbalance of
different modalities varies across devices, which consequently exacerbates the
imbalance after the parameter aggregation among devices. These issues motivate
us to propose the knowledge distillation and training balance method to
improve the learning performance of DMML over wireless networks.

###  I-A Related Work

Several recent works [16, 22, 23, 24, 25] have considered multi-modal learning
in federated learning systems. In [16], it assumes that all vehicles have the
same sensors, ensuring identical modalities across devices. To compensate for
missing sensor data during model inference, [16] utilizes previous data from
the same sensor. The training scheme of federated multi-modal learning is
improved in [23, 24, 25] to address the issue of modality heterogeneity. In
[23], a dynamic and multi-view graph structure corresponding to the devices is
applied on the edge server to automatically capture the relationships among
devices, which is utilized to achieve correlated local model updates among
devices. Using the global model and the local model’s outputs with respect to
a public dataset, an inter-modal contrastive objective is designed in [24] to
complement for the absent modality in some users, and an intra-modal
contrastive objective is proposed to regularize models to converge towards the
global consensus. To learn the cross-modal features, the modality-agnostic
features and the modality-specific features are extracted from the data of
each modality separately in [25], and a modality discriminator is employed to
guide the training of the feature extraction networks. However, the algorithm
designed specifically for federated learning cannot be directly applied to
decentralized learning due to its reliance on the server.

Existing decentralized learning methods [8, 9, 10] focus on the single-modal
learning, such as training single-modal image classifiers. Many works have
addressed the non-IID data problem in decentralized learning [26, 27, 28]. The
authors in [26] conduct detailed experiments to demonstrate the impact of non-
IID data on the performance of decentralized learning. To deal with the non-
IID data, a cross-gradient aggregation algorithmic framework is proposed in
[27]. In [28], the mixing-weights of devices are dynamically updated to
improve the personalized model for each device’s task. Besides, some works
have studied the impact of topology on decentralized learning [29, 30, 31,
32]. The authors in [29] theoretically quantify how the topology affects the
convergence of decentralized learning. In [30], an integer programming problem
is formulated to find the best suitable topology at each training round. In
[31], the topology of devices is optimized to save the transmission energy
consumption. It is proved in [32] that the condition for achieving consensus
in decentralized learning can be relaxed to the requirement that the merged
topology of multiple training rounds is a connected graph.

Several methods [20, 22, 21, 33] have been proposed to mitigate the challenge
of imbalanced multi-modal training. In [20], the overfitting and
generalization behaviors of the training multi-modal model are quantified by
the overfitting-to-generalization ratio, which is minimized to determine
optimal gradient blending weights of different modalities. Based on the
assumption of convex loss functions, the gradient blending is extended from
centralized learning to federated learning in [22]. The authors in [21]
propose to adaptively control the optimization of each modality based on the
discrepancy of their contribution towards the learning objective, thereby
addressing the training imbalance of different modalities. In [33], the
learning processes of single-modal and multi-modal networks are decoupled, and
the learning rates of different modalities are dynamically balanced using an
adaptive tracking factor. Different from the aforementioned approaches, we
propose a training balance metric specific designed for decentralized learning
in this paper.

Figure 2: A decentralized multi-modal learning system.

###  I-B Contributions and Outline

Motivated by the above, we propose the decentralized multi-modal learning with
knowledge distillation (DMML-KD) framework to deal with the problem of non-IID
data and modality heterogeneity. Furthermore, we design a training balance
method to improve the learning performance of decentralized multi-modal
learning over wireless networks. The main contributions are summarized as
follows:

  * •

We propose the DMML-KD framework to improve the training performance under the
non-IID data and modality heterogeneity. We firstly decompose the extracted
feature into the modality-common and the modality-specific components. Then we
use the generator to learn the global conditional distribution of the
modality-common features. The devices can utilize the generator to guide its
modality-common features towards the global distribution, thereby potentially
capturing correlated features across different modalities.

  * •

We develop a method to control the number of local iterations for different
modalities to deal with the imbalanced training of different modalities.
Compared to adjusting the learning rates [21, 33] of different modalities, the
proposed method can reduce the computational burden of devices, thereby
reducing the energy consumption.

  * •

We design a performance metric based on the parameter variation to evaluate
the training speed of different modalities. Based on the metric, we optimize
the number of iterations for different modalities. Experimental results show
that the proposed DMML-KD framework with training balance can effectively
improve the training performance of DMML.

The rest of this paper is organized as follows. The system model and the
proposed DMML-KD framework are introduced in Section II. The training balance
method and the optimization problem formulation are presented in Section III.
The detail of the proposed training balance algorithm is introduced in
Sections IV. Finally, extensive experimental results are presented in Section
V, and conclusions are drawn in Section VI.

##  II System Model and DMML-KD framework

In this section, we introduce a decentralized multi-model learning system,
focusing on the design of the system model and loss functions. We then develop
the DMML-KD framework.

Figure 3: A multi-modal neural network with the concatenation fusion method.

###  II-A System Model

We consider a decentralized multi-modal learning system as shown in Fig. 2,
which consists of a set of wireless devices $\mathcal{K}=\\{1,2,...,K\\}$. The
set of modalities of device $k$ is $\mathcal{M}_{k}\subseteq\mathcal{M}$,
where $\mathcal{M}=\\{1,2,...,M\\}$ contains all modalities in the considered
system. Note that $\mathcal{M}_{k}$ varies across different devices. As shown
in Fig. 2, smartphones possess data with two modalities, namely images and
audio, whereas smart glasses exclusively comprise the image modality. Each
device $k\in\mathcal{K}$ has its local dataset
$\\{(\\{\bm{x}^{m}_{k,1}\\}_{m\in\mathcal{M}_{k}},y_{k,1}),...,(\\{\bm{x}^{m}_{k,D_{k}}\\}_{m\in\mathcal{M}_{k}},y_{k,D_{k}})\\}$
with the size $D_{k}$. $\bm{x}^{m}_{k,1},...,\bm{x}^{m}_{k,D_{k}}$ are the raw
data of the modality $m\in\mathcal{M}_{k}$. $y_{k,1},...,y_{k,D_{k}}$ refer to
the corresponding label. It is assumed that the data of different modalities
on each device is temporally aligned.

The topology of the considered system can be regarded as an undirected graph.
The set of neighboring devices of device $k$ is denoted as $\mathcal{R}_{k}$.
Let $\mathcal{R}^{m}_{k}\subseteq\mathcal{R}_{k}$ be the set of neighboring
devices that possess data with modality $m$. There exists a device-to-device
(D2D) link between device $k$ and its neighboring device
$k^{{}^{\prime}}\in\mathcal{R}_{k}$, which can be used to transmit model
parameters. After completing the local update in each training round, each
device transmits the local model to its neighboring devices using frequency-
division multiplexing.

Through the collaborative learning of devices, the objective of this system is
to find the optimal parameters $\bm{w}^{\ast}$ that minimizes the global loss
function

$F(\bm{w})=\frac{1}{K}\sum_{k\in\mathcal{K}}F_{k}(\bm{w}).$ (1)

The component
$F_{k}(\bm{w})=\frac{1}{D_{k}}\sum_{d=1}^{D_{k}}f(\\{\bm{x}^{m}_{k,d}\\}_{m\in\mathcal{M}_{k}},y_{k,d};\bm{w})$
is the task-specific loss function of device $k$, where $f(\cdot)$ is
determined by the specific task, e.g., cross-entropy.

Table I: Summary of Notations Notation |  Definition
---|---
$\mathcal{K}$ |  Set of devices
$\mathcal{M}_{k}$ |  Set of modalities on device $k$
$\bm{x}_{k,d}^{m}$, $y_{k,d}$ |  Raw data for modality $m$ and label of $d$-th data on device $k$
$\mathcal{R}_{k}$ |  Set of neighboring devices of device $k$
$\mathcal{R}^{m}_{k}$ |  Set of neighboring devices that possess data with modality $m$ of device $k$
$\hat{\bm{h}}^{m}_{k,d}$, $\check{\bm{h}}^{m}_{k,d}$ |  Modality-common and modality-specific feature extracted from the data of modality $m$
$\hat{\bm{w}}^{n}_{k,t}$ |  Modality-common classifier obtained by device $k$ after $n$-th iteration of $t$-th global round
$\check{\bm{w}}^{n,m}_{k,t}$ |  Modality-specific classifier of modality $m$ obtained by device $k$ after $n$-th iteration of $t$-th global round
$\bm{w}^{n,m}_{k,t}$ |  Feature extractor of modality $m$ obtained by device $k$ after $n$-th iteration of $t$-th global round
$N_{k}$ |  Number of local iterations in each round on device $k$
$\bm{W}^{m}_{k,t}$,$\check{\bm{W}}^{m}_{k,t}$,$\hat{\bm{W}}_{k,t}$ |  Aggregated parameter of $\bm{w}^{N_{k},m}_{k,t}$, $\check{\bm{w}}^{N_{k},m}_{k,t}$ and $\hat{\bm{w}}^{N_{k}}_{k,t}$ in $t$-th round on device $k$

###  II-B Multi-Modal Neural Networks with Feature Decomposition

In this paper, we take multi-modal neural networks with the widely used fusion
method, i.e., concatenation [21]. Fig. 3 depicts an original multi-modal
neural network that uses the concatenation fusion method. The data
$\bm{x}^{m}$ of modality $m$ is processed using the network with the
parameters $\bm{w}^{m}$ to extract the feature representation $\bm{h}^{m}$.
The features from different modalities are concatenated, and then processed
using the linear classifier with the weight $\tilde{\bm{w}}$ and the bias
$\bm{b}$ to obtain the output $\bm{c}$, i.e.,

$\displaystyle\bm{c}$
$\displaystyle=\tilde{\bm{w}}[(\bm{h}^{1})^{T},(\bm{h}^{2})^{T},...,(\bm{h}^{M})^{T}]^{T}+\bm{b}$
$\displaystyle=\sum_{m=1}^{M}\tilde{\bm{w}}^{m}\bm{h}^{m}+\bm{b}.$ (2)

The classifier $\tilde{\bm{w}}\in\mathbb{R}^{C\times ML}$ can be decomposed
into $M$ components corresponding to different modalities, i.e.,
$\tilde{\bm{w}}=[\tilde{\bm{w}}^{1},\tilde{\bm{w}}^{2},...,\tilde{\bm{w}}^{M}]$,
where $\tilde{\bm{w}}^{m}\in\mathbb{R}^{C\times L}$. $C$ represents the number
of categories in the dataset. $L$ is the length of the features $\bm{h}^{m}$,
i.e., $\bm{h}^{m}\in\mathbb{R}^{L\times 1}$. The network in Fig. 3 extracts a
specific feature from each modality’s data, which is not suitable for
scenarios with modality absence. This is because a device lacking modality $m$
can not learn the correlated features between its own modalities and modality
$m$.

Therefore, as shown in Fig. 4, we decompose the feature extracted from each
modality into the modality-common component $\hat{\bm{h}}^{m}$ and the
modality-specific component $\check{\bm{h}}^{m}$ based on the network in Fig.
3, i.e.,

$\displaystyle[(\hat{\bm{h}}^{m})^{T},(\check{\bm{h}}^{m})^{T}]^{T}=\phi(\bm{x}^{m};\bm{w}^{m}),$
(3)

where $\phi(\cdot;\bm{w}^{m})$ refers to the feature extraction network with
the parameter $\bm{w}^{m}$. The modality-common features extracted from the
data of different modalities within the same category are expected to be in
the same distribution. This facilitates the application of knowledge
distillation to learn the correlated features between modalities in scenarios
with modality absence. Subsequently,
$\hat{\bm{h}}^{m}\in\mathbb{R}^{\hat{L}\times 1}$ and
$\check{\bm{h}}^{m}\in\mathbb{R}^{\hat{L}\times 1}$ are processed by the
common classifier $\hat{\bm{w}}$ and the modality-specific classifier
$\check{\bm{w}}^{m}$ respectively, i.e.,

$\displaystyle\bm{c}=\sum_{m=1}^{M}(\hat{\bm{w}}\hat{\bm{h}}^{m}+\check{\bm{w}}^{m}\check{\bm{h}}^{m})+\bm{b}.$
(4)

The whole model is
$\bm{w}=\\{\bm{w}^{1},\bm{w}^{2},...,\bm{w}^{M},\hat{\bm{w}},\check{\bm{w}}^{1},\check{\bm{w}}^{2},...,\\\
\check{\bm{w}}^{M},\bm{b}\\}$.

Due to the modality absence, only a subset of the model $\bm{w}$ is trained on
each device. Specifically, device $k$ utilizes the data $\bm{x}_{k,d}^{m}$
from its available modalities $m\in\mathcal{M}_{k}$ to obtain the output
$\bm{c}_{k,d}$, i.e.,

$\displaystyle\bm{c}_{k,d}=\sum_{m\in\mathcal{M}_{k}}(\hat{\bm{w}}_{k,t}^{n}\hat{\bm{h}}_{k,d}^{m}+\check{\bm{w}}_{k,t}^{n,m}\check{\bm{h}}_{k,d}^{m})+\bm{b}_{k,t}^{n},$
(5)

where $\bm{w}^{n,m}_{k,t}$ and $\check{\bm{w}}_{k,t}^{n,m}$ are the parameter
of the feature extraction network and the modality-specific classifier
corresponding to modality $m$, respectively. $\hat{\bm{w}}_{k,t}^{n}$ is the
parameter of the common classifier. $\bm{b}_{k,t}^{n}$ is the bias. They are
obtained by device $k$ after the $n$-th iteration of the $t$-th global round.
$[(\hat{\bm{h}}_{k,d}^{m})^{T},(\check{\bm{h}}_{k,d}^{m})^{T}]^{T}$ is the
feature extracted from $\bm{x}_{k,d}^{m}$ using the network with the parameter
$\bm{w}^{n,m}_{k,t}$. To effectively decompose the modality-common feature
$\hat{\bm{h}}_{k,d}^{m}$ and the modality-specific feature
$\check{\bm{h}}_{k,d}^{m}$, the following loss functions are employed.

Figure 4: A multi-modal neural network with feature decomposition.

####  II-B1 Similarity Loss

To make the modality-common features $\hat{\bm{h}}_{k,d}^{m}$ of different
modalities be in the same distribution, we minimize the following loss
function to approximate the distributions obtained by processing
$\hat{\bm{h}}^{m}_{k,d}$ from different modalities using the common
classifier.

$\displaystyle
F^{sim}_{k}{=}\frac{1}{D_{k}}\sum_{d=1}^{D_{k}}\sum_{(m,m^{{}^{\prime}})\in\hat{\mathcal{M}}_{k}}\frac{KL(\sigma(\hat{\bm{w}}^{n}_{k,t}\hat{\bm{h}}^{m}_{k,d})||\sigma(\hat{\bm{w}}^{n}_{k,t}\hat{\bm{h}}^{m^{{}^{\prime}}}_{k,d}))}{|\mathcal{M}_{k}|(|\mathcal{M}_{k}|-1)},$
(6)

where $\hat{\mathcal{M}}_{k}$ is the set of permutations formed by selecting
any two different elements from $\mathcal{M}_{k}$, which has
$|\mathcal{M}_{k}|(|\mathcal{M}_{k}|-1)$ elements. The number of elements in
$\mathcal{M}_{k}$ is denoted by $|\mathcal{M}_{k}|$. $\sigma(\cdot)$ refers to
the softmax function. KL-divergence [34] can be used to measure the divergence
between two different distributions $\bm{z}$ and $\hat{\bm{z}}$. It is defined
by

$\displaystyle
KL(\bm{z}||\hat{\bm{z}})=\sum_{i}z^{i}\ln\frac{z^{i}}{\hat{z}^{i}},$ (7)

where $z^{i}$ and $\hat{z}^{i}$ represent the $i$-th component of $\bm{z}$ and
$\hat{\bm{z}}$ respectively. The loss $F^{sim}_{k}$ can only be computed by
devices that have multiple modalities. The devices with only one modality need
the generator to guide their modality-common features towards the common
distribution, which will be introduced in the following section.

####  II-B2 Auxiliary Classification Loss

The distributions of the modality-common features for different data
categories are distinct. To capture the category information within these
features, we minimize the auxiliary classification loss as following

$\displaystyle
F^{cls}_{k}=\frac{1}{D_{k}|\mathcal{M}_{k}|}\sum_{d=1}^{D_{k}}\sum_{m\in\mathcal{M}_{k}}CE(\sigma(\hat{\bm{w}}^{n}_{k,t}\hat{\bm{h}}^{m}_{k,d}),y_{k,d}),$
(8)

where $CE(\sigma(\hat{\bm{w}}^{n}_{k,t}\hat{\bm{h}}^{m}_{k,d}),y_{k,d})$ is
the cross-entropy loss calculated with
$\sigma(\hat{\bm{w}}^{n}_{k,t}\hat{\bm{h}}^{m}_{k,d})$ and the label
$y_{k,d}$.

####  II-B3 Difference Loss

To ensure that the modality-common and modality-specific features capture
distinct aspects of the input data, we utilize the orthogonality loss as
follows [35]

$\displaystyle
F^{dif}_{k}=\sum_{m\in\mathcal{M}_{k}}\|(\hat{\bm{H}}^{m}_{k})^{T}\check{\bm{H}}^{m}_{k}\|_{2}^{2},$
(9)

where
$\hat{\bm{H}}^{m}_{k}{=}[\hat{\bm{h}}^{m}_{k,1},\hat{\bm{h}}^{m}_{k,2},...,\hat{\bm{h}}^{m}_{k,D_{k}}]\in\mathbb{R}^{\hat{L}\times
D_{k}}$ and
$\check{\bm{H}}^{m}_{k}=[\check{\bm{h}}^{m}_{k,1},\check{\bm{h}}^{m}_{k,2},...,\check{\bm{h}}^{m}_{k,D_{k}}]\in\mathbb{R}^{\hat{L}\times
D_{k}}$.

Therefore, the loss function of the feature decomposition is summarized as
follows

$\displaystyle
F^{dec}_{k}=\alpha_{1}F^{sim}_{k}+\alpha_{2}F^{cls}_{k}+\alpha_{3}F^{dif}_{k},$
(10)

where $\alpha_{1}$, $\alpha_{2}$ and $\alpha_{3}$ are hyper-parameters. This
loss is minimized to optimize the parameters of the feature extractors and the
common classifier, i.e.,
$\\{\bm{w}^{n,m}_{k,t}\\}_{m\in\mathcal{M}_{k}}\cup\\{\hat{\bm{w}}^{n}_{k,t}\\}$.

Figure 5: Proposed decentralized multi-modal learning with knowledge
distillation.

###  II-C Decentralized Learning

The execution process of decentralized learning can be divided into two
stages. In the local update stage, each device updates the model parameter
using its data. In the parameter aggregation stage, each device transmits the
locally updated parameter to its neighboring devices and aggregates its
parameter with the received parameters. The decentralized learning process for
the multi-modal network presented in the above section is introduced as
follows.

####  II-C1 Local Update Stage

Due to the different modalities across devices, the model trained on different
devices differ during the local update stage. Specifically, the model
parameters $\bm{w}^{n,m}_{k,t},\check{\bm{w}}^{n,m}_{k,t}$ associated with the
modality $m\in\mathcal{M}_{k}$ and the common parameters
$\hat{\bm{w}}^{n}_{k,t},\bm{b}_{k,t}^{n}$ are used to calculate the output
$\bm{c}_{k,d}$. These parameters are updated by device $k$. The parameters
corresponding to the other modalities, i.e.,
$\\{\bm{w}^{n,m}_{k,t},\check{\bm{w}}^{n,m}_{k,t}\\}_{m\notin\mathcal{M}_{k}}$,
are not used in the forward propagation (FP) and the backward propagation
(BP), thus they are not updated. The parameter that can be updated by device
$k$ is denoted by
$\bm{w}^{n}_{k,t}=\\{\bm{w}^{n,m}_{k,t},\check{\bm{w}}^{n,m}_{k,t}\\}_{m\in\mathcal{M}_{k}}\cup\\{\hat{\bm{w}}^{n}_{k,t},\bm{b}_{k,t}^{n}\\}$.
The update equation of the parameters is

$\displaystyle\bm{w}^{n}_{k,t}=\bm{w}^{n-1}_{k,t}-\eta_{t}\nabla
F^{tot}_{k}(\bm{w}^{n-1}_{k,t}),$ (11)

for $n=1,2,...,N_{k}$. $N_{k}$ is the number of iterations of the local
update, which is determined by the number of local epochs, the amount of local
data and the batch size of device $k$. The total loss $F^{tot}_{k}$ about the
training of the multi-modal neural network is given by

$\displaystyle F^{tot}_{k}=F_{k}+F^{dec}_{k}+\alpha_{4}F^{kd}_{k},$ (12)

where $\alpha_{4}$ is a hyper-parameter. The loss $F^{kd}_{k}$ about knowledge
distillation is introduced in the subsequent section. The initial parameter of
device $k$ in the $t$-th global round is $\bm{w}^{0}_{k,t}=\bm{W}_{k,t-1}$,
where $\bm{W}_{k,t-1}$ is the aggregated model parameter of device $k$
obtained in the $(t-1)$-th round.

####  II-C2 Parameter Aggregation Stage

In the parameter aggregation stage, device $k$ sends the parameters
$\bm{w}^{N_{k},m}_{k,t}$ and $\check{\bm{w}}^{N_{k},m}_{k,t}$ to its
neighboring devices that possess data with modality $m$, i.e.,
$k^{{}^{\prime}}\in\mathcal{R}_{k}^{m}$. In contrast, the parameters
$\hat{\bm{w}}^{N_{k}}_{k,t},\bm{b}_{k,t}^{N_{k}}$ are transmitted to all the
neighboring devices, i.e., $k^{{}^{\prime}}\in\mathcal{R}_{k}$. The parameters
specific to modality $m$ can only be exchanged between neighboring devices
that possess data with modality $m$ because the feature extraction networks
for different modalities are distinct. Then device $k$ aggregates its
parameters with the received parameters by

$\displaystyle\bm{W}^{m}_{k,t}{=}\sum_{k^{{}^{\prime}}\in\mathcal{R}_{k}^{m}\cup\\{k\\}}\xi^{m}_{k,k^{{}^{\prime}}}\bm{w}^{N_{k^{{}^{\prime}}},m}_{k^{{}^{\prime}},t},$
$\displaystyle\check{\bm{W}}^{m}_{k,t}{=}\sum_{k^{{}^{\prime}}\in\mathcal{R}_{k}^{m}\cup\\{k\\}}\xi^{m}_{k,k^{{}^{\prime}}}\check{\bm{w}}^{N_{k^{{}^{\prime}}},m}_{k^{{}^{\prime}},t},$
$\displaystyle\hat{\bm{W}}_{k,t}=\sum_{k^{{}^{\prime}}\in\mathcal{R}_{k}\cup\\{k\\}}\xi_{k,k^{{}^{\prime}}}\hat{\bm{w}}^{N_{k^{{}^{\prime}}}}_{k^{{}^{\prime}},t},$
$\displaystyle\bm{B}_{k,t}=\sum_{k^{{}^{\prime}}\in\mathcal{R}_{k}\cup\\{k\\}}\xi_{k,k^{{}^{\prime}}}\bm{b}^{N_{k^{{}^{\prime}}}}_{k^{{}^{\prime}},t},$
(13)

where $\xi^{m}_{k,k^{{}^{\prime}}}\geq 0$ and $\xi_{k,k^{{}^{\prime}}}\geq 0$
are the aggregation coefficients for the parameters specific to modality $m$
and the common parameters, respectively. They satisfy
$\sum_{k=1}^{K}\xi^{m}_{k,k^{{}^{\prime}}}=1$,
$\sum_{k^{{}^{\prime}}=1}^{K}\xi^{m}_{k,k^{{}^{\prime}}}=1$,
$\sum_{k=1}^{K}\xi_{k,k^{{}^{\prime}}}=1$ and
$\sum_{k^{{}^{\prime}}=1}^{K}\xi_{k,k^{{}^{\prime}}}=1$. In particular,
$\xi^{m}_{k,k^{{}^{\prime}}}=0$ for
$k^{{}^{\prime}}\notin\mathcal{R}_{k}^{m}\cup\\{k\\}$, and
$\xi_{k,k^{{}^{\prime}}}=0$ for
$k^{{}^{\prime}}\notin\mathcal{R}_{k}\cup\\{k\\}$. The aggregated parameter
$\bm{W}_{k,t}=\\{\bm{W}^{m}_{k,t},\check{\bm{W}}^{m}_{k,t}\\}_{m\in\mathcal{M}_{k}}\cup\\{\hat{\bm{W}}_{k,t},\bm{B}_{k,t}\\}$
is used as the initial parameter for the next local update stage of device
$k$. The local update and parameter aggregation are performed for many global
rounds to obtain the desired learning performance.

If decentralized training is conducted solely using the task-specific loss
function $F_{k}$, device $k$ can only learn the correlated features among its
own modalities $m\in\mathcal{M}_{k}$ and cannot acquire the features related
to the other modalities $m\notin\mathcal{M}_{k}$, which degrades the
performance of DMML. To address this issue, we apply the knowledge
distillation to DMML.

###  II-D DMML with Knowledge Distillation

While some works have integrated knowledge distillation with federated
learning [36, 37], to the best of our knowledge, knowledge distillation has
not yet been applied to DMML.

As shown in Fig. 5, in the proposed training scheme, each device can utilize a
generator to learn the conditional distribution of the modality-common
features under different categories. Due to the non-IID data and modality
heterogeneity, the locally-trained generator of each device can only learn the
local distribution. By means of parameter exchange and aggregation, the
generator can learn the global conditional distribution of the modality-common
features. Consequently, devices can utilize the generator to guide their
modality-common features towards the global distribution. In other words, a
part of features from each modality can be guided towards a common
distribution, thereby potentially capturing correlated features across
different modalities.

####  II-D1 Knowledge Extraction

To make the generator $\bm{G}^{n}_{k,t}$ learn the knowledge about the
distribution of the modality-common features, each device utilizes the trained
common classifier $\hat{\bm{w}}^{N_{k}}_{k,t}$ to guide the training of the
generator with the following loss

$\displaystyle
F^{gen}_{k}=\mathbb{E}_{\begin{subarray}{c}\bm{h}^{G}_{k}=\bm{G}^{n}_{k,t}(\bm{z},y),\\\
\bm{z}\sim\mathcal{N}(0,\bm{I}),y\in\mathcal{C}_{k}\end{subarray}}[CE(\sigma(\hat{\bm{w}}^{N_{k}}_{k,t}\bm{h}^{G}_{k}),y){+}\beta\|\bm{h}^{G}_{k}-\bar{\bm{h}}^{y}_{k,t}\|_{2}],$
(14)

where $\mathcal{C}_{k}$ is the set of labels possessed by device $k$. Given a
label $y\in\mathcal{C}_{k}$, the generator $\bm{G}^{n}_{k,t}$ can map a
Gaussian random variable $\bm{z}\sim\mathcal{N}(0,\bm{I})$ to a feature
$\bm{h}^{G}_{k}$. $\beta$ is a hyper-parameter. By minimizing the loss
function $F^{gen}_{k}$, the generated feature $\bm{h}^{G}_{k}$ is expected to
be correctly classified by the common classifier, while approaching the mean
of the true features, i.e.,
$\bar{\bm{h}}^{y}_{k,t}=\frac{1}{|Y_{k}^{y}||\mathcal{M}_{k}|}\sum_{d\in
Y_{k}^{y}}\sum_{m\in\mathcal{M}_{k}}\hat{\bm{h}}^{m}_{k,d}$. The set
$Y_{k}^{y}=\\{d|y_{k,d}=y\\}$ contains the indices of data with the label $y$.
This enables the generator to learn the conditional distribution of the
modality-common features.

The generator locally trained on each device can only learn the local
distribution because of the non-IID data and modality heterogeneity. To learn
the global conditional distribution of the modality-common features, each
device can send its generator $\bm{G}^{\hat{N}_{k}}_{k,t}$ to its neighboring
devices and aggregate its generator with the received generators.

$\displaystyle\bar{\bm{G}}_{k,t}=\sum_{k^{{}^{\prime}}\in\mathcal{R}_{k}\cup\\{k\\}}\xi_{k,k^{{}^{\prime}}}\bm{G}^{\hat{N}_{k}}_{k^{{}^{\prime}},t},$
(15)

where $\hat{N}_{k}$ is the number of local training iterations for the
generator of device $k$.

####  II-D2 Knowledge Distillation

In the next training round, the devices can utilize the aggregated generator
$\bar{\bm{G}}_{k,t}$ to guide its modality-common features towards the global
distribution by minimizing the following loss

$\displaystyle
F^{kd}_{k}{=}\frac{1}{D_{k}|\mathcal{M}_{k}|}\sum_{d=1}^{D_{k}}\sum_{m\in\mathcal{M}_{k}}$
$\displaystyle\mathbb{E}_{\begin{subarray}{c}\bm{h}^{G}_{k}=\bar{\bm{G}}_{k,t}(\bm{z},y_{k,d}),\\\
\bm{z}\sim\mathcal{N}(0,\bm{I})\end{subarray}}[KL(\sigma(\hat{\bm{w}}^{n}_{k,t+1}\hat{\bm{h}}^{m}_{k,d})$
$\displaystyle\qquad\qquad||\sigma(\hat{\bm{w}}^{n}_{k,t+1}\bm{h}^{G}_{k}))].$
(16)

Minimizing the divergence between
$\sigma(\hat{\bm{w}}^{n}_{k,t+1}\bm{h}_{k,d})$ and
$\sigma(\hat{\bm{w}}^{n}_{k,t+1}\bm{h}^{G}_{k})$ enables the modality-common
features of device $k$ to approach the common distribution learned by the
generator. The overall training process of the proposed DMML-KD is outlined in
Algorithm 1.

Algorithm 1 Proposed decentralized multi-modal learning with knowledge
distillation

Input: $\bullet$ The initial parameters of the multi-modal network
$\bm{W}_{k,0}$ and the generator $\bar{\bm{G}}_{k,0}$ for all devices
$\bullet$ The learning rate $\eta_{t}$ and $\eta_{t}^{G}$ of the multi-modal
network and the generator
$\bullet$ The aggregation coefficients $\xi^{m}_{k,k^{{}^{\prime}}}$ and
$\xi_{k,k^{{}^{\prime}}}$ for all devices

1: for training round $t=1$ to $T$ do

2: for all devices $k=1$ to $K$ in parallel do

3: Set $\bm{w}^{0}_{k,t}=\bm{W}_{k,t-1}$,
$\bm{G}^{0}_{k,t}=\bar{\bm{G}}_{k,t-1}$

4: for iteartion $n=1$ to $N_{k}$ do

5: Train the multi-modal network with $F^{tot}_{k}$

6: end for

7: Calculate the average feature $\bar{\bm{h}}_{k,t}^{y}$

8: for iteartion $n=1$ to $\hat{N}_{k}$ do

9: Use $\hat{\bm{w}}^{N_{k}}_{k,t}$ and $\bar{\bm{h}}_{k,t}^{y}$ to train the
generator with $F^{gen}_{k}$

10: end for

11: Transmit the parameters $\bm{w}^{N_{k},m}_{k,t}$ and
$\check{\bm{w}}^{N_{k},m}_{k,t}$ to its neighboring devices
$k^{{}^{\prime}}\in\mathcal{R}_{k}^{m}$

12: Transmit the parameters
$\hat{\bm{w}}^{N_{k}}_{k,t},\bm{b}_{k,t}^{N_{k}},\bm{G}_{k,t}^{\hat{N}_{k}}$
to all the neighboring devices $k^{{}^{\prime}}\in\mathcal{R}_{k}$

13: Aggregate the parameters with (II-C2) and (15)

14: end for

15: end for

##  III Training Balance and Problem Formulation

To address the issue of imbalanced training across different modalities, we
propose a method that controls the number of local iterations for different
modalities. Compared to adjusting the learning rates [21, 33] of different
modalities, controlling the number of iterations can reduce the computational
load of devices, thereby reducing the energy consumption and the training
latency. To achieve the balanced training by controlling the number of
iterations, we introduce a novel performance metric based on the parameter
variation, which can evaluate the training speed of different modalities.
Furthermore, we formulate an optimization problem to optimize the number of
iterations for different modalities in each training round.

###  III-A Control over the Number of Iterations

In the local update process introduced in Section II, device $k$ performs
joint training in each iteration using the data from all the modalities it
possesses. In other words, the parameters
$\bm{w}^{n,m}_{k,t},\check{\bm{w}}^{n,m}_{k,t}$ corresponding to the
modalities in $\mathcal{M}_{k}$ are updated in each iteration. Due to the
different training speeds of different modalities, it is unreasonable to
impose the same number of iterations on different modalities. In this paper,
we reduce the number of iterations for the modalities that exhibit faster
training speed, thereby achieving the training balance across different
modalities.

Specifically, we use $P_{k,t}^{n,m}=1$ to denote that the parameters
corresponding to the modality $m$ are trained by the device $k$ in the $n$-th
iteration of the $t$-th round, otherwise $P_{k,t}^{n,m}=0$. Therefore, the
output of device $k$ in the $n$-th iteration of the $t$-th round is given by

$\displaystyle\bm{c}_{k,d}=\sum_{m\in\mathcal{M}_{k}}P_{k,t}^{n,m}(\hat{\bm{w}}_{k,t}^{n-1}\hat{\bm{h}}_{k,d}^{m}+\check{\bm{w}}_{k,t}^{n-1,m}\check{\bm{h}}_{k,d}^{m})+\bm{b}_{k,t}^{n-1}.$
(17)

If $P_{k,t}^{n,m}=0$, the feature extraction network $\bm{w}_{k,t}^{n-1,m}$
corresponding to modality $m$ for device $k$ is not utilized to extract
features in the $n$-th iteration. Because the features
$[(\hat{\bm{h}}_{k,d}^{m})^{T},(\check{\bm{h}}_{k,d}^{m})^{T}]^{T}$ is not
extracted, the classifiers
$\hat{\bm{w}}_{k,t}^{n-1},\check{\bm{w}}_{k,t}^{n-1,m}$ do not process them.
Therefore, the parameters $\bm{w}_{k,t}^{n-1,m},\check{\bm{w}}_{k,t}^{n-1,m}$
corresponding to modality $m$ is not updated, i.e.,
$\bm{w}_{k,t}^{n,m}=\bm{w}_{k,t}^{n-1,m}$ and
$\check{\bm{w}}_{k,t}^{n,m}=\check{\bm{w}}_{k,t}^{n-1,m}$. The common
classifier $\hat{\bm{w}}_{k,t}^{n-1}$ can be updated because it may process
the modality-common features from other modalities. If $P_{k,t}^{n,m}=0$ for
all the modalities in $\mathcal{M}_{k}$, the whole model
$\\{\bm{w}^{n-1b,m}_{k,t},\check{\bm{w}}^{n-1,m}_{k,t}\\}_{m\in\mathcal{M}_{k}}\cup\\{\hat{\bm{w}}^{n-1}_{k,t},\bm{b}_{k,t}^{n-1}\\}$
is not updated in the $n$-th iteration.

We use the integer $0\leq N^{m}_{k,t}\leq N_{k}$ to represent the number of
iterations performed by the device $k$ on the parameters corresponding to the
modality $m$ in the $t$-th round. $N^{m}_{k,t}$ is obtained with the metric
and optimization problem described later. Once $N^{m}_{k,t}$ is obtained, the
value of $P_{k,t}^{n,m}$ for each iteration is randomly selected from
$\\{0,1\\}$ under the condition that
$\sum_{n=1}^{N_{k}}P_{k,t}^{n,m}=N^{m}_{k,t}$.

Different from the parameters
$\\{\bm{w}^{n,m}_{k,t},\check{\bm{w}}^{n,m}_{k,t}\\}_{m\notin\mathcal{M}_{k}}$,
which remain unchanged throughout the local update of device $k$, the
parameters
$\\{\bm{w}^{n,m}_{k,t},\check{\bm{w}}^{n,m}_{k,t}\\}_{m\in\mathcal{M}_{k}}$
are not updated in a part of local iterations. The updated parameters
$\\{\bm{w}^{N_{k},m}_{k,t},\check{\bm{w}}^{N_{k},m}_{k,t}\\}_{m\in\mathcal{M}_{k}}$
need to be transmitted to the neighboring devices in $\mathcal{R}_{k}^{m}$.

###  III-B Performance Metric for Training Balance

In this paper, we evaluate the training speed of different modalities by
measuring the relative parameter variation of the feature extractor for each
modality, i.e.,

$\displaystyle\frac{\|\bm{W}_{k,t}^{m}-\bm{W}_{k,t-1}^{m}\|_{2}}{\|\bm{W}_{k,t}^{m}\|_{2}}$
$\displaystyle=\frac{\|\sum_{k^{{}^{\prime}}=1}^{K}\xi^{m}_{k,k^{{}^{\prime}}}\bm{w}^{N_{k^{{}^{\prime}}},m}_{k^{{}^{\prime}},t}-\bm{W}_{k,t-1}^{m}\|_{2}}{\|\bm{W}_{k,t}^{m}\|_{2}}$
$\displaystyle\stackrel{{\scriptstyle(a)}}{{\leq}}\sum_{k^{{}^{\prime}}=1}^{K}\xi^{m}_{k,k^{{}^{\prime}}}\frac{\|\bm{w}^{N_{k^{{}^{\prime}}},m}_{k^{{}^{\prime}},t}-\bm{W}_{k,t-1}^{m}\|_{2}}{\|\bm{W}_{k,t}^{m}\|_{2}}$
(18)

where step (a) is due to the convexity of $\|\cdot\|_{2}$ and
$\sum_{k^{{}^{\prime}}=1}^{K}\xi^{m}_{k,k^{{}^{\prime}}}=1$. We use
$\varphi_{k,k^{{}^{\prime}},t}^{m}=\frac{\|\bm{w}^{N_{k^{{}^{\prime}}},m}_{k^{{}^{\prime}},t}-\bm{W}_{k,t-1}^{m}\|_{2}}{\|\bm{W}_{k,t}^{m}\|_{2}}$
to denote the relative parameter variation between the parameter
$\bm{w}^{N_{k^{{}^{\prime}}},m}_{k^{{}^{\prime}},t}$ obtained by the locally
update of device $k^{{}^{\prime}}$ and the aggregated parameter
$\bm{W}_{k,t-1}^{m}$ of device $k$.

Generally, a larger loss function leads to larger gradients, resulting in
bigger parameter variations. As training approaches convergence, the
parameters tend to stabilize, indicating smaller variations in parameters. If
the parameter variation of the parameters corresponding to modality $m$ is
smaller than that of the other modalities, the loss function for modality $m$
is smaller and closer to convergence. Hence, the number of iterations
corresponding to modality $m$ needs to be reduced. This implies that the
number of iterations should be inversely proportional to the magnitude of
parameter variations.

However, we cannot directly determine the number of iterations for modality
$m$ on device $k$ based on the parameter variation
$\frac{\|\bm{W}_{k,t}^{m}-\bm{W}_{k,t-1}^{m}\|_{2}}{\|\bm{W}_{k,t}^{m}\|_{2}}$.
This is because the parameter variation
$\frac{\|\bm{W}_{k,t}^{m}-\bm{W}_{k,t-1}^{m}\|_{2}}{\|\bm{W}_{k,t}^{m}\|_{2}}$
of device $k$ is influenced not only by its local update, i.e.,
$\varphi_{k,k,t}^{m}$, but also by the parameter update of its neighboring
devices, i.e.,
$\varphi_{k,k^{{}^{\prime}},t}^{m},k^{{}^{\prime}}\in\mathcal{R}_{k}^{m}$, as
shown in (III-B). These influences are combined with the aggregation
coefficient $\xi^{m}_{k,k^{{}^{\prime}}}$. For modality $m$ with the small
parameter variation on device $k$, it is necessary to not only reduce the
number of iterations for modality $m$ on device $k$, but also consider
reducing the number of iterations for modality $m$ on the neighboring device
$k^{{}^{\prime}}\in\mathcal{R}_{k}^{m}$. Meanwhile, when reducing the number
of iterations for modality $m$ on device $k$, it is important to consider the
impact on other devices after the parameter aggregation, which can be
quantified by the parameter variation $\varphi_{k,k^{{}^{\prime}},t}^{m}$.

Based on the above analysis, we develop the following performance metric of
training balance

$\displaystyle\gamma_{t+1}^{m}$
$\displaystyle=\sum_{k=1}^{K}\sum_{k^{{}^{\prime}}=1}^{K}\xi^{m}_{k,k^{{}^{\prime}}}\frac{N^{m}_{k^{{}^{\prime}},t+1}}{\varphi_{k,k^{{}^{\prime}},t}^{m}+\varepsilon}$
$\displaystyle=\sum_{k^{{}^{\prime}}=1}^{K}N^{m}_{k^{{}^{\prime}},t+1}\sum_{k=1}^{K}\frac{\xi^{m}_{k,k^{{}^{\prime}}}}{\varphi_{k,k^{{}^{\prime}},t}^{m}+\varepsilon},$
(19)

where $\varepsilon>0$ is a very small number to avoid division by zero. After
the parameter aggregation in the $t$-th round, the number of iterations
$N^{m}_{k^{{}^{\prime}},t+1}$ of the $(t+1)$-th round is optimized to make
$\gamma_{t+1}^{m}$ be close for different modalities. It can be observed from
(III-B) that if the parameter variation $\varphi_{k,k^{{}^{\prime}},t}^{m}$ is
big, the number of iterations is expected to be high. The term
$N^{m}_{k^{{}^{\prime}},t+1}\sum_{k=1}^{K}\frac{\xi^{m}_{k,k^{{}^{\prime}}}}{\varphi_{k,k^{{}^{\prime}},t}^{m}+\varepsilon}$
can be regarded as the influence of device $k^{{}^{\prime}}$, after
$N^{m}_{k^{{}^{\prime}},t+1}$ iterations, on other devices that require
aggregation with $\bm{w}^{N_{k^{{}^{\prime}}},m}_{k^{{}^{\prime}},t+1}$.

###  III-C Problem Formulation

Optimizing the number of iterations $N^{m}_{k,t}$ can have an impact on the
energy consumption of devices. The energy consumption of devices is primarily
attributed to two factors: the local update and the parameter transmission.
Specifically, the energy consumption of device $k$ in the $t$-th round is
given by [38]

$\displaystyle E_{k,t}=E^{cmp}_{k,t}+E^{com}_{k,t}$
$\displaystyle=\frac{\epsilon_{k}f_{k}^{2}}{e_{k}}(\sum_{m\in\mathcal{M}_{k}}N^{m}_{k,t}(O^{m}{+}\hat{O}){+}N^{\max}_{k,t}(O^{b}{+}O^{G}_{F}){+}\hat{N}_{k}O^{G})$
$\displaystyle\qquad+\sum_{m\in\mathcal{M}_{k}}\sum_{k^{{}^{\prime}}\in\mathcal{R}^{m}_{k}}\frac{p_{k}\Upsilon^{m}}{\mathcal{B}\log(1{+}\frac{p_{k}g_{k,k^{{}^{\prime}},t}^{2}}{\mathcal{B}\mathcal{N}_{0}})}$
$\displaystyle\qquad+\sum_{k^{{}^{\prime}}\in\mathcal{R}_{k}}\frac{p_{k}\Upsilon}{\mathcal{B}\log(1{+}\frac{p_{k}g_{k,k^{{}^{\prime}},t}^{2}}{\mathcal{B}\mathcal{N}_{0}})},$
(20)

where $O^{m}$ is the total floating point operations (FLOPs) of performing the
FP and BP for the feature extractor and classifier specific to modality $m$ in
each iteration. $\hat{O}$ and $O^{b}$ represent the total FLOPs of the FP and
BP for the common classifier and the bias in each iteration, respectively. The
FLOPs of the FP for the generator are denoted as $O^{G}_{F}$ in each
iteration, while the total FLOPs of FP and BP for the generator is denoted as
$O^{G}$ in each iteration. $f_{k}$ and $e_{k}$ are the computing frequency and
the number of FLOPs per cycle for device $k$, respectively. $\epsilon_{k}$ is
the effective capacitance coefficient of computing chip for device $k$.
$N^{\max}_{k,t}$ is the maximum number of iterations that device $k$ can
perform under the constraints of $N_{k}$ and the remaining energy. $\Upsilon$
denotes the total size (_in bit_) of the common classifier, the bias and the
generator. $\Upsilon^{m}$ denotes the total size of the parameters specific to
modality $m$. Device $k$ transmits the parameters to its neighbors using a
channel with a bandwidth of $\mathcal{B}$ through the transmission power
$p_{k}$. The channel gain between device $k$ and $k^{{}^{\prime}}$ in the
$t$-th round is $g_{k,k^{{}^{\prime}},t}$. The spectral density of the
additive white Gaussian noise (AWGN) is $\mathcal{N}_{0}$.

In this paper, we optimize the number of iterations for the $(t+1)$-th round
to achieve balanced training across devices for different modalities. The
optimization problem can be formulated as

$\displaystyle\mathcal{P}:\min_{\\{N^{m}_{k,t+1}\\}}\leavevmode\nobreak\ $
$\displaystyle\frac{\sum_{m=1}^{M}(\gamma_{t+1}^{m}-\frac{1}{M}\sum_{m^{{}^{\prime}}=1}^{M}\gamma_{t+1}^{m^{{}^{\prime}}})^{2}}{\sum_{k=1}^{K}\sum_{m\in\mathcal{M}_{k}}N^{m}_{k,t+1}}$
$\displaystyle s.t.\leavevmode\nobreak\ $ $\displaystyle E_{k,t+1}\leq
E_{k,t+1}^{\max},\forall k,$ (21) $\displaystyle 0\leq N^{m}_{k,t+1}\leq
N_{k},N^{m}_{k,t+1}\in\mathbb{Z},\forall m\in\mathcal{M}_{k},\forall k,$ (22)
$\displaystyle\sum_{k^{{}^{\prime}}\in\mathcal{R}_{k}^{m}}(\Delta
N^{m}_{k,t+1}{-}\Delta N^{m}_{k^{{}^{\prime}},t+1})^{2}{\leq}\delta,\forall
m\in\mathcal{M}_{k},\forall k.$ (23)

The constraint (21) represents that the energy consumption $E_{k,t+1}$ cannot
exceed the remaining energy $E_{k,t+1}^{\max}=E_{k,t}^{\max}-E_{k,t}$. The
constraint (22) denotes the feasible region of $N^{m}_{k,t+1}$. The term
$\Delta N^{m}_{k,t+1}=N^{\max}_{k,t+1}-N^{m}_{k,t+1}$ denotes the number of
iterations reduced for modality $m$ of device $k$ in the $(t+1)$-th round. A
larger value of $\Delta N^{m}_{k,t+1}$ indicates greater energy saving of
device $k$. For fair consideration, it is expected that the saved number of
iterations of different devices are similar, as shown by the constraint (23).
The optimization objective is to minimize the variances of
$\gamma_{k,t+1}^{m}$ across all modalities. Meanwhile, the total number of
iterations across all devices is maximized to avoid slow convergence due to
insufficient iteration per round. After each round of parameter aggregation,
it is required to solve this optimization problem to obtain the number of
iterations $N^{m}_{k^{{}^{\prime}},t+1}$ in the next round.

##  IV Training Balance Algorithm Design

The optimization problem $\mathcal{P}$ is non-convex. Therefore, we develop a
low-complexity algorithm to obtain the approximate solution $N^{m}_{k,t+1}$ in
this section, and introduce how to obtain the solution in practical systems.

###  IV-A Approximate Solution of Problem $\mathcal{P}$

We first estimate the maximum number of iterations $N^{\max}_{k,t+1}$ that
device $k$ can perform under the constraints (21) and (22). If device $k$ has
sufficient energy, i.e., the energy consumption $E_{k,t+1}$ calculated using
(III-C) with $N_{k,t+1}^{m}=N_{k}$ is less than $E_{k,t+1}^{\max}$, then we
have $N^{\max}_{k,t+1}=N_{k}$. Otherwise, $N^{\max}_{k,t+1}$ is the maximum
number of iterations that can be performed with the device’s remaining energy,
i.e., $N^{\max}_{k,t+1}$ is the maximum integer that satisfies the condition
where the energy consumption $E_{k,t+1}$ calculated with
$N_{k,t+1}^{m}=N^{\max}_{k,t+1}$ is less than $E_{k,t+1}^{\max}$.

Then the variances of $\gamma_{k,t+1}^{m}$ across all modalities are taken
into consideration, which can be minimized with the following condition

$\displaystyle\gamma_{t+1}^{m}=\frac{1}{M}\sum_{m^{{}^{\prime}}=1}^{M}\gamma_{t+1}^{m^{{}^{\prime}}},\forall
m.$ (24)

The condition (24) indicates that
$\gamma_{t+1}^{m}=\gamma_{t+1}^{m^{{}^{\prime}}}$, $\forall
m,m^{{}^{\prime}}$. From (III-B), we can observe that $\gamma_{t+1}^{m}$ is
determined by the number of iterations corresponding to modality $m$, i.e.,
$N_{k,t+1}^{m},\forall k$. By maximizing
$\sum_{k=1}^{K}\sum_{m\in\mathcal{M}_{k}}N^{m}_{k,t+1}$ while satisfying the
condition (24), we can obtain the modality that uses the maximum feasible
number of iterations, i.e.,

$\displaystyle\hat{m}=\arg\min_{m}\sum_{k^{{}^{\prime}}=1}^{K}N^{\max}_{k^{{}^{\prime}},t+1}\sum_{k=1}^{K}\frac{\xi^{m}_{k,k^{{}^{\prime}}}}{\varphi_{k,k^{{}^{\prime}},t}^{m}+\varepsilon}.$
(25)

The number of iterations corresponding to the other modalities $m\neq\hat{m}$
should be reduced in order to make $\gamma_{t+1}^{m}$ close to
$\gamma_{t+1}^{\hat{m}}$. To satisfy the constraint (23), which is to make
$\Delta N^{m}_{k,t+1}$ similar across different devices, we employ a method of
uniformly decreasing the number of iterations. As shown in Algorithm 2, the
initial number of iterations for each device
$k\in\mathcal{K}^{m}=\\{k|m\in\mathcal{M}_{k},|\mathcal{M}_{k}|>1\\}$ is set
as $N^{\max}_{k,t+1}$, then the iterations for the devices
$k\in\mathcal{K}^{m}$ are successively decreased until the following condition
holds

$\displaystyle\sum_{k^{{}^{\prime}}=1}^{K}N^{m}_{k^{{}^{\prime}},t+1}\sum_{k=1}^{K}\frac{\xi^{m}_{k,k^{{}^{\prime}}}}{\varphi_{k,k^{{}^{\prime}},t}^{m}+\varepsilon}\leq\gamma_{t+1}^{\hat{m}}.$
(26)

Since the devices with only one modality do not suffer from the training
imbalance across modalities, their iteration count is set as
$N^{\max}_{k,t+1}$.

Algorithm 2 Obtain the number of iterations

Input: The maximum number of iterations $N^{\max}_{k,t+1},\forall k$; The
coefficients
$\frac{\xi^{m}_{k,k^{{}^{\prime}}}}{\varphi_{k,k^{{}^{\prime}},t}^{m}+\varepsilon},\forall
m\in\mathcal{M}_{k},\forall k,\forall k^{{}^{\prime}}$.

1: Obtain $\hat{m}$ with (25)

2: for $m=1$ to $M$ do

3: Set $N^{m}_{k,t+1}=N^{\max}_{k,t+1},\forall k\in\mathcal{K}^{m}$

4: repeat

5: Select the next device $k\in\mathcal{K}^{m}$

6: Let $N^{m}_{k,t+1}-1\rightarrow N^{m}_{k,t+1}$

7: until the condition (26) holds

8: end for

Output: The number of iterations $N^{m}_{k,t+1},\forall
m\in\mathcal{M}_{k},\forall k$

Algorithm 3 Proposed DMML-KD with training balance

Input: The initial parameters $\bm{W}_{k,0}$, $\bar{\bm{G}}_{k,0}$; The
learning rate $\eta_{t}$ and $\eta_{t}^{G}$; The aggregation coefficients
$\xi^{m}_{k,k^{{}^{\prime}}}$ and $\xi_{k,k^{{}^{\prime}}}$.

1: for training round $t=1$ to $T$ do

2: for all devices $k=1$ to $K$ in parallel do

3: Set $\bm{w}^{0}_{k,t}=\bm{W}_{k,t-1}$,
$\bm{G}^{0}_{k,t}=\bar{\bm{G}}_{k,t-1}$

4: Randomly choose $P_{k,t}^{n,m}$ with
$\sum_{n=1}^{N_{k}}P_{k,t}^{n,m}=N^{m}_{k,t}$

5: for iteartion $n=1$ to $N_{k}$ do

6: Train the multi-modal networks with $P_{k,t}^{n,m}=1$ using $F^{tot}_{k}$

7: end for

8: Calculate the average feature $\bar{\bm{h}}_{k,t}^{y}$

9: for iteartion $n=1$ to $\hat{N}_{k}$ do

10: Use $\hat{\bm{w}}^{N_{k}}_{k,t}$ and $\bar{\bm{h}}_{k,t}^{y}$ to train the
generator with $F^{gen}_{k}$

11: end for

12: Transmit $\bm{w}^{N_{k},m}_{k,t}$ and $\check{\bm{w}}^{N_{k},m}_{k,t}$ to
devices $k^{{}^{\prime}}\in\mathcal{R}_{k}^{m}$. Transmit
$\hat{\bm{w}}^{N_{k}}_{k,t},\bm{b}_{k,t}^{N_{k}},\bm{G}_{k,t}^{\hat{N}_{k}}$
to devices $k^{{}^{\prime}}\in\mathcal{R}_{k}$

13: Aggregate the parameters with (II-C2) and (15)

14: Obtain the coefficients $N^{\max}_{k,t+1}$ and
$\varphi_{k,k^{{}^{\prime}},t}^{m}$.

15: Send $N^{\max}_{k,t+1}$ and
$\frac{\xi^{m}_{k,k^{{}^{\prime}}}}{\varphi_{k,k^{{}^{\prime}},t}^{m}+\varepsilon},\forall
m\in\mathcal{M}_{k},\forall k^{{}^{\prime}}$ to the cluster head

16: end for

17: The cluster head utilizes Algorithm 2 to obtain $N^{m}_{k,t+1}$ and sends
them back to the respective devices

18: end for

###  IV-B Training Balance in Practice

Based on the above description, we can observe that obtaining the iterations
$N^{m}_{k,t+1},\forall m\in\mathcal{M}_{k},\forall k$ requires the
coefficients $N^{\max}_{k,t+1}$ and
$\frac{\xi^{m}_{k,k^{{}^{\prime}}}}{\varphi_{k,k^{{}^{\prime}},t}^{m}},\forall
k,\forall k^{{}^{\prime}},\forall m$. The maximum iteration $N^{\max}_{k,t+1}$
can be obtained by device $k$. The aggregation coefficients
$\xi^{m}_{k,k^{{}^{\prime}}},\forall k^{{}^{\prime}},\forall m$ are stored on
device $k$. The parameter variation $\varphi_{k,k^{{}^{\prime}},t}^{m},\forall
k^{{}^{\prime}},\forall m$ can be calculated by device $k$ after receiving the
parameters from other devices in the parameter aggregation stage of the $t$-th
round. To obtain the number of iterations, each device needs to send its
coefficients to the cluster head, which can be the device connected to the
maximum number of devices. After receiving the coefficients sent by the other
devices, the cluster head utilizes Algorithm 2 to obtain the number of
iterations for different modalities on each device and sends them back to the
respective devices. The overall process of the proposed DMML-KD with training
balance is outlined in Algorithm 3.

Due to the significantly smaller size of the coefficients $N^{\max}_{k,t+1}$
and $\frac{\xi^{m}_{k,k^{{}^{\prime}}}}{\varphi_{k,k^{{}^{\prime}},t}^{m}}$
compared to the parameter size of the training model, the overhead of
transmitting these coefficients can be ignored. Additionally, the cluster head
only needs to execute a simple loop to obtain the number of iterations
$N^{m}_{k,t+1}$. The proposed training balance method exhibits low complexity.

Figure 6: Three different distributions of data categories. (a) $\Phi=$IID.
(b) $\Phi=0.3$. (c) $\Phi=0.5$.

##  V Simulation Results

In this section, we present the detail of the simulation and evaluate the
performance of the proposed DMML-KD with the training balance algorithm.

###  V-A Simulation Setup

Table II: Simulation parameters Parameter | Value
---|---
Bandwidth $\mathcal{B}$ | 500 KHz
Noise power spectral density $\mathcal{N}_{0}$ | -140 dBm/Hz
Transmission power $p_{k}$ | $0.1W$
CPU frequency $f_{k}$ | 1 GHz
Number of FLOPs per cycle $e_{k}$ | 4
Effective capacitance coefficient $\epsilon_{k}$ | $2\times 10^{-28}$
Number of global rounds $T$ | 80
FLOPs specific to audio and visual $O^{1},O^{2}$ | $\\{5.73,5.82\\}{\times}10^{10}$
Parameter size specific to audio and visual $\Upsilon^{1},\Upsilon^{2}$ | $\\{8.94,8.95\\}{\times}10^{7}$
FLOPs of common classifier $\hat{O}$ | $4.92{\times}10^{4}$
Parameter size $\Upsilon$ | $1.35{\times}10^{6}$
FLOPs of generator $O^{G}_{F}$, $O^{G}$ | $\\{2.7,5.4\\}{\times}10^{6}$

####  V-A1 Simulation Environment

A decentralized learning system with $K=12$ devices is considered in the
simulation. The devices are uniformly distributed within a circular area with
a diameter of 100 meters. The distance between device $k$ and
$k^{{}^{\prime}}$ is denoted as $d_{k,k^{{}^{\prime}}}$. Device $k$ is
connected to the devices within a range of 50 meters around it. The channel
gain $g_{k,k^{{}^{\prime}},t}$ follows the Rayleigh distribution with the mean
$10^{-PL(d_{k,k^{{}^{\prime}}})/20}$. The path loss is
$PL(d_{k,k^{{}^{\prime}}})(dB)=32.4+20\log_{10}(\hat{f}_{k}^{carrier})+20\log_{10}(d_{k,k^{{}^{\prime}}})$
and the carrier frequency is $\hat{f}_{k}^{carrier}=2.6$ GHz. The simulation
parameters of the considered system are summarized in Table II.

####  V-A2 Dataset Partition

The training dataset is CREMA-D [18], which is a sentiment analysis dataset.
CREMA-D consists of two modalities: audio and visual. It consists of 7,442
video clips, each lasting 2-3 seconds. For the visual data, we extract one
frame from each video and crop it to the size of $224\times 224\times 3$,
which serves as the input for the feature extractor. The audio data is
processed into a spectrogram with the size $257\times 188$ using the librosa
[39]. To evaluate the performance of the proposed algorithm in different
scenarios, we use three different distributions of modalities, i.e.,
$\Gamma=1$, $\Gamma=0.5$ and $\Gamma=0$. Additionally, we apply three
different distributions of data categories, i.e., $\Phi=$IID, $\Phi=0.3$ and
$\Phi=0.5$. They are combined in pairs to form nine different scenarios. The
specific settings for the distributions are as follows:

  * •

$\Gamma=1$: All devices possess data with both audio and visual modalities.

  * •

$\Gamma=0.5$: Half of the devices have data with both audio and visual
modalities, while the other devices have data with only one of the modalities,
either audio or visual.

  * •

$\Gamma=0$: All devices have data with only one of the modalities, either
audio or visual.

  * •

$\Phi=$IID: The data on each device is randomly selected from the entire
dataset, as shown in Fig.6 (a).

  * •

$\Phi=0.3$: 30% data on each device belongs to one category, while the
remaining data is randomly composed of the other category data, as shown in
Fig.6 (b).

  * •

$\Phi=0.5$: 50% data on each device belongs to one category, while the
remaining data is randomly composed of other category data, as shown in Fig.6
(c).

We employ ResNet-18 [40] to extract features from both the audio and visual
data. The length of the extracted features is $L=512$, and the length
$\hat{L}=256$. The common classifier and the modality-specific classifiers are
the fully connected layer with 256 input nodes and $C=6$ output nodes. The
FLOPs and the parameter size of the training network are summarized in Table
II.

Figure 7: Audio and visual accuracy with $\Gamma=0$.

####  V-A3 Hyper-parameters of DMML-KD

The generator is a two-layer fully connected network with the nodes [70, 512,
256] sequentially. The learning rate of the multi-modal network and the
generator are $5\times 10^{-4},1\times 10^{-4}$, respectively. The number of
iterations for the generator is 50 in each round. The hyper-parameters
$\alpha_{1},\alpha_{2},\alpha_{3},\alpha_{4},\beta$ in the loss functions are
all set as 1. The losses are optimized with the Adam optimizer [41]. The
aggregation coefficients are obtained with the Metropolis-Hastings weights
[42].

###  V-B Performance Evaluation

Figure 8: Test accuracy with $\Phi=$IID and (a) $\Gamma=1$, (b) $\Gamma=0.5$.

Figure 9: Test accuracy with $\Phi=$0.3 and (a) $\Gamma=1$, (b) $\Gamma=0.5$.

Figure 10: Test accuracy with $\Phi=$0.5 and (a) $\Gamma=1$, (b) $\Gamma=0.5$.
Figure 11: Audio and visual accuracy with $\Gamma=1$. Figure 12: Audio and
visual accuracy with $\Gamma=0.5$.

The test accuracies of DMML and DMML-KD under different data partitions are
shown in Fig. 7, Fig. 8, Fig. 9 and Fig. 10. Specifically, in the case of
$\Gamma=0$, the model trained on devices is single-modal, and there is no
issue of imbalanced training across modalities. Therefore, training balancing
is not performed in the case of $\Gamma=0$. For the cases of $\Gamma=1$ and
$\Gamma=0.5$, the test accuracies of different devices are averaged to obtain
the curves shown in Fig. 8, Fig. 9 and Fig. 10. In the case of $\Gamma=0$, the
test accuracies of devices with only the visual modality and only the audio
modality are separately averaged to obtain the visual accuracy and audio
accuracy, as shown in Fig. 7. To estimate the visual accuracy and audio
accuracy of the multi-modal network in the cases of $\Gamma=1$ and
$\Gamma=0.5$, we use the following term [21] as the approximated prediction of
modality $m\in\\{1,2\\}$,

$\displaystyle\bm{c}^{m}_{k,d}=\hat{\bm{w}}_{k,t}^{n}\hat{\bm{h}}_{k,d}^{m}+\check{\bm{w}}_{k,t}^{n,m}\check{\bm{h}}_{k,d}^{m}+\frac{\bm{b}_{k,t}^{n}}{2}.$
(27)

The estimated accuracies are shown in Fig. 11 and Fig. 12.

####  V-B1 Knowledge Distillation

As shown in Fig. 8, Fig. 9 and Fig. 10, knowledge distillation can effectively
improve the accuracy. In the case of $\Gamma=0$, the single-modal model
trained with DMML cannot learn the correlated features of the other modality.
As shown in Fig. 7, after applying knowledge distillation, the visual and
audio accuracy are both improved, which indicates that knowledge distillation
can transfer feature knowledge of the other modality.

Moreover, as shown in Fig. 8 (a), Fig. 9 (a) and Fig. 10 (a), the performance
gain between DMML-KD without balance and DMML is 2.69% in the case of
$\Phi=$IID. In contrast, the performance gains for the cases of $\Phi=0.3$ and
$\Phi=0.5$ are 4.15% and 4.04%, respectively. This is because the performance
of DMML-KD is almost unchanged in the presence of non-IID data, indicating
that DMML-KD can deal with the non-IID data.

####  V-B2 Training Balance

As shown in Fig. 7, when training a single-modality network with DMML, both
the visual and audio feature extractor are trained effectively. The visual
accuracy can approach or even exceed the audio accuracy. However, the visual
accuracy is significanty lower than that of the audio accuracy when joint
training the multi-modal network, as shown in Fig. 11. The visual accuracy in
Fig. 12 is higher than that in Fig. 11, which is because a portion of devices
train the visual-only network under $\Gamma=0.5$. These results indicate that
training a multi-modal network jointly may lead to the insufficient training
of the visual feature extractor.

As shown in Fig. 8, Fig. 9 and Fig. 10, compared to DMML-KD without balance,
the proposed training balance method can further improve the accuracy. As
shown in Fig. 11 and Fig. 12, although DMML-KD without balance can improve the
visual and audio accuracy compared to DMML, the training of the visual
modality is still not sufficiently compared to the audio modality. The visual
accuracy is improved by the proposed training balance method. Besides, it can
be observed from Fig. 11 that different category distributions can impact the
visual accuracy obtained with DMML. In contrast, our proposed DMML-KD with
training balance can achieve comparable or even higher visual accuracy than
the audio accuracy across different non-IID scenarios.

Apart from the performance gain, the proposed training balance method can also
reduce the energy consumption of devices, as shown in Table III. Compared with
DMML, training and transmitting the generator in DMML-KD add some
computational load on devices. However, the required FLOPs of training the
generator is much smaller than that of the feature extraction networks. The
energy saved in devices by reducing the number of iterations exceeds the
energy consumption added by training the generator.

Table III: Averaged energy consumption per device with $\Gamma=1$ Distribution | DMML ($\times 10^{4}$ J) | DMML-KD with balance($\times 10^{4}$ J)
---|---|---
$\Phi=$IID | 1.737 | 1.652
$\Phi=0.3$ | 1.749 | 1.645
$\Phi=0.5$ | 1.749 | 1.671

##  VI Conclusion

In this paper, we propose the DMML with knowledge distillation to enhance the
training performance in the presence of non-IID data and modality
heterogeneity over wireless networks. The proposed method utilizes a generator
to learn the global conditional distribution of modality-common features and
guide the modality-common features of different devices towards the same
distribution. Furthermore, we design a strategy to reduce the number of local
iterations for rapidly converging modalities, addressing the issue of training
imbalance. The optimization of the number of local iterations for different
modalities on each device is achieved based on the proposed performance
metric. Experimental results show that the proposed DMML-KD framework with
training balance significantly improves the training performance of DMML.

## References

  * [1] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” _Nature_ , vol. 521, no. 7553, pp. 436–444, May 2015.
  * [2] S. Deng, H. Zhao, W. Fang, J. Yin, S. Dustdar, and A. Y. Zomaya, “Edge intelligence: The confluence of edge computing and artificial intelligence,” _IEEE Internet of Things Journal_ , vol. 7, no. 8, pp. 7457–7469, Aug. 2020\.
  * [3] G. Zhu, D. Liu, Y. Du, C. You, J. Zhang, and K. Huang, “Toward an intelligent edge: Wireless communication meets machine learning,” _IEEE Communications Magazine_ , vol. 58, no. 1, pp. 19–25, Jan. 2020.
  * [4] X. Wang, Y. Han, V. C. Leung, D. Niyato, X. Yan, and X. Chen, “Convergence of edge computing and deep learning: A comprehensive survey,” _IEEE Communications Surveys & Tutorials_, vol. 22, no. 2, pp. 869–904, Secondquarter 2020.
  * [5] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in _Proc. Artificial intelligence and statistics_ , 2017, pp. 1273–1282.
  * [6] B. Yin, Z. Chen, and M. Tao, “Predictive gan-powered multi-objective optimization for hybrid federated split learning,” _to appear in IEEE Transactions on Communications_ , May 2023.
  * [7] C. Feng, Z. Zhao, Y. Wang, T. Q. S. Quek, and M. Peng, “On the design of federated learning in the mobile edge computing systems,” _IEEE Transactions on Communications_ , vol. 69, no. 9, pp. 5902–5916, Sept. 2021.
  * [8] M. Chen, D. Gündüz, K. Huang, W. Saad, M. Bennis, A. V. Feljan, and H. V. Poor, “Distributed learning in wireless networks: Recent progress and future challenges,” _IEEE Journal on Selected Areas in Communications_ , vol. 39, no. 12, pp. 3579–3605, Dec. 2021.
  * [9] Z. Jiang, G. Yu, Y. Cai, and Y. Jiang, “Decentralized edge learning via unreliable device-to-device communications,” _IEEE Transactions on Wireless Communications_ , vol. 21, no. 11, pp. 9041–9055, 2022.
  * [10] H. Xing, O. Simeone, and S. Bi, “Federated learning over wireless device-to-device networks: Algorithms and convergence analysis,” _IEEE Journal on Selected Areas in Communications_ , vol. 39, no. 12, pp. 3723–3741, 2021.
  * [11] B. Yin, Z. Chen, and M. Tao, “Dynamic data collection and neural architecture search for wireless edge intelligence systems,” _IEEE Transactions on Wireless Communications_ , vol. 22, no. 1, pp. 688–703, 2023.
  * [12] D. C. Nguyen, M. Ding, Q.-V. Pham, P. N. Pathirana, L. B. Le, A. Seneviratne, J. Li, D. Niyato, and H. V. Poor, “Federated learning meets blockchain in edge computing: Opportunities and challenges,” _IEEE Internet of Things Journal_ , vol. 8, no. 16, pp. 12 806–12 825, 2021.
  * [13] C. Ma, J. Li, M. Ding, H. H. Yang, F. Shu, T. Q. S. Quek, and H. V. Poor, “On safeguarding privacy and security in the framework of federated learning,” _IEEE Network_ , vol. 34, no. 4, pp. 242–248, 2020.
  * [14] R. Gao, T.-H. Oh, K. Grauman, and L. Torresani, “Listen to look: Action recognition by previewing audio,” in _Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , June 2020, pp. 10 457–10 467.
  * [15] D. Hazarika, R. Zimmermann, and S. Poria, “Misa: Modality-invariant and -specific representations for multimodal sentiment analysis,” in _Proc. ACM International Conference on Multimedia_ , 2020, pp. 1122–1131.
  * [16] B. Salehi, J. Gu, D. Roy, and K. Chowdhury, “Flash: Federated learning for automated selection of high-band mmwave sectors,” in _Proc. IEEE Conference on Computer Communications_ , 2022, pp. 1719–1728.
  * [17] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Proc. Advances in Neural Information Processing Systems_ , vol. 30, 2017.
  * [18] H. Cao, D. G. Cooper, M. K. Keutmann, R. C. Gur, A. Nenkova, and R. Verma, “Crema-d: Crowd-sourced emotional multimodal actors dataset,” _IEEE Transactions on Affective Computing_ , vol. 5, no. 4, pp. 377–390, 2014.
  * [19] Y. Huang, J. Lin, C. Zhou, H. Yang, and L. Huang, “Modality competition: What makes joint training of multi-modal network fail in deep learning? (Provably),” in _Proc. International Conference on Machine Learning_ , vol. 162, Jul. 2022, pp. 9226–9259.
  * [20] W. Wang, D. Tran, and M. Feiszli, “What makes training multi-modal classification networks hard?” in _Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , June 2020, pp. 12 695–12 705.
  * [21] X. Peng, Y. Wei, A. Deng, D. Wang, and D. Hu, “Balanced multimodal learning via on-the-fly gradient modulation,” in _Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , June 2022, pp. 8238–8247.
  * [22] S. Chen and B. Li, “Towards optimal multi-modal federated learning on non-iid data with hierarchical gradient blending,” in _Proc. IEEE Conference on Computer Communications_ , 2022, pp. 1469–1478.
  * [23] J. Chen and A. Zhang, “Fedmsplit: Correlation-adaptive federated multi-task learning across multimodal split networks,” in _Proc. ACM SIGKDD Conference on Knowledge Discovery and Data Mining_ , 2022, pp. 87–96.
  * [24] Q. Yu, Y. Liu, Y. Wang, K. Xu, and J. Liu, “Multimodal federated learning via contrastive representation ensemble,” in _Proc. International Conference on Learning Representations_ , 2023.
  * [25] X. Yang, B. Xiong, Y. Huang, and C. Xu, “Cross-modal federated human activity recognition via modality-agnostic and modality-specific representation learning,” in _Proc. AAAI Conference on Artificial Intelligence_ , vol. 36, no. 3, Jun. 2022, pp. 3063–3071.
  * [26] K. Hsieh, A. Phanishayee, O. Mutlu, and P. Gibbons, “The non-IID data quagmire of decentralized machine learning,” in _Proc. International Conference on Machine Learning_ , vol. 119, Jul. 2020, pp. 4387–4398.
  * [27] Y. Esfandiari, S. Y. Tan, Z. Jiang, A. Balu, E. Herron, C. Hegde, and S. Sarkar, “Cross-gradient aggregation for decentralized learning from non-iid data,” in _Proc. International Conference on Machine Learning_ , vol. 139, Jul. 2021, pp. 3036–3046.
  * [28] S. Li, T. Zhou, X. Tian, and D. Tao, “Learning to collaborate in decentralized learning of personalized models,” in _Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , June 2022, pp. 9766–9775.
  * [29] T. Vogels, H. Hendrikx, and M. Jaggi, “Beyond spectral gap: the role of the topology in decentralized learning,” in _Proc. Advances in Neural Information Processing Systems_ , vol. 35, 2022, pp. 15 039–15 050.
  * [30] H. Xu, M. Chen, Z. Meng, Y. Xu, L. Wang, and C. Qiao, “Decentralized machine learning through experience-driven method in edge networks,” _IEEE Journal on Selected Areas in Communications_ , vol. 40, no. 2, pp. 515–531, 2022\.
  * [31] J.-J. Kuo, C.-W. Ching, H.-S. Huang, and Y.-C. Liu, “Energy-efficient topology construction via power allocation for decentralized learning via smart devices with edge computing,” _IEEE Transactions on Green Communications and Networking_ , vol. 5, no. 4, pp. 1806–1819, 2021.
  * [32] A. Koloskova, N. Loizou, S. Boreiri, M. Jaggi, and S. Stich, “A unified theory of decentralized SGD with changing topology and local updates,” in _Proc. International Conference on Machine Learning_ , vol. 119, Jul. 2020, pp. 5381–5393.
  * [33] Y. Sun, S. Mai, and H. Hu, “Learning to balance the learning rates between various modalities via adaptive tracking factor,” _IEEE Signal Processing Letters_ , vol. 28, pp. 1650–1654, 2021.
  * [34] S. Kullback and R. A. Leibler, “On information and sufficiency,” _The annals of mathematical statistics_ , vol. 22, no. 1, pp. 79–86, 1951.
  * [35] K. Bousmalis, G. Trigeorgis, N. Silberman, D. Krishnan, and D. Erhan, “Domain separation networks,” in _Proc. Advances in Neural Information Processing Systems_ , vol. 29, 2016.
  * [36] E. Jeong, S. Oh, H. Kim, J. Park, M. Bennis, and S.-L. Kim, “Communication-efficient on-device machine learning: Federated distillation and augmentation under non-iid private data,” _arXiv preprint arXiv:1811.11479_ , 2018.
  * [37] Z. Zhu, J. Hong, and J. Zhou, “Data-free knowledge distillation for heterogeneous federated learning,” in _Proc. International Conference on Machine Learning_ , vol. 139, Jul. 2021, pp. 12 878–12 889.
  * [38] T. D. Burd and R. W. Brodersen, “Processor design for portable systems,” _Journal of VLSI signal processing systems for signal, image and video technology_ , vol. 13, no. 2-3, pp. 203–221, 1996.
  * [39] B. McFee, C. Raffel, D. Liang, D. P. Ellis, M. McVicar, E. Battenberg, and O. Nieto, “librosa: Audio and music signal analysis in python,” in _Proc. python in science conference_ , vol. 8, 2015, pp. 18–25.
  * [40] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proc. IEEE Conference on Computer Vision and Pattern Recognition_ , June 2016.
  * [41] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in _Proc. International Conference on Learning Representations_ , 2015.
  * [42] L. Xiao and S. Boyd, “Fast linear iterations for distributed averaging,” _Systems $\&$ Control Letters_, vol. 53, no. 1, pp. 65–78, 2004.