# Task-Oriented Mulsemedia Communication using Unified Perceiver and Conformal
Prediction in 6G Metaverse Systems

Hongzhi Guo, and Ian F. Akyildiz, H. Guo is with the University of Nebraska-
Lincoln, Lincoln, NE, 68588, USA. E-mail<EMAIL_ADDRESS>I. F. Akyildiz is
with Truva Inc., Alpharetta, GA 30022, USA. E-mail<EMAIL_ADDRESS>
###### Abstract

The growing prominence of extended reality (XR), holographic-type
communications, and metaverse demands truly immersive user experiences by
using many sensory modalities, including sight, hearing, touch, smell, taste,
etc. Additionally, the widespread deployment of sensors in areas such as
agriculture, manufacturing, and smart homes is generating a diverse array of
sensory data. A new media format known as multisensory media (mulsemedia) has
emerged, which incorporates a wide range of sensory modalities beyond the
traditional visual and auditory media. 6G wireless systems are envisioned to
support the internet of senses, making it crucial to explore effective data
fusion and communication strategies for mulsemedia. In this paper, we
introduce a task-oriented multi-task mulsemedia communication system named
MuSeCo, which is developed using unified Perceiver models and Conformal
Prediction. This unified model can accept any sensory input and efficiently
extract latent semantic features, making it adaptable for deployment across
various Artificial Intelligence of Things (AIoT) devices. Conformal Prediction
is employed for modality selection and combination, enhancing task accuracy
while minimizing data communication overhead. The model has been trained using
six sensory modalities across four classification tasks. Simulations and
experiments demonstrate that MuSeCo can effectively select and combine sensory
modalities, significantly reduce end-to-end communication latency and energy
consumption, and maintain high accuracy in communication-constrained systems.

###### Index Terms:

Artificial Intelligence of Things (AIoT), conformal prediction, mulsemedia
communication, multimodal machine learning, semantic communication, unified
Perceiver.

##  I Introduction

Human perception relies on a complex interplay of senses, including the five
fundamental ones: sight, hearing, touch, smell, and taste. Recent research has
discovered that the human brain contains a multisensory correlation detector
that optimally selects and combines sensory inputs [1, 2]. This capability is
not unique to humans; even simple organisms exhibit similar sensory
integration. Traditionally, the utilization of a large number of senses for a
single task has been constrained. Prior to the era of 5G wireless systems, the
transmission of novel sensory modalities—such as touch, smell, taste, and
wind—faced significant challenges due to the constraints of communication and
sensing technologies. However, recent advancements in 5G have dramatically
reduced end-to-end latency and increased data rates, facilitating the
transmission of new sensory modalities, including haptic signals via the
tactile internet [3]. With the ongoing development of 6G wireless systems, it
is anticipated that even more sensory modalities will be transmitted to
fulfill the vision of the Internet of Senses [4]. This progression will
enhance Extended Reality (XR)—encompassing Augmented Reality (AR), Mixed
Reality (MR), and Virtual Reality (VR)—as well as Holographic-Type
Communications (HTC), which are spatial computing technologies capable of
generating hyperrealistic objects and environments [5, 6]. Additionally,
digital twins of humans, animals, and virtual settings rely extensively on
multiple sensory modalities. Head-Mounted Displays (HMDs) for XR and HTC are
equipped with integrated sensors and actuators which are pivotal in supporting
the collection and rendering of diverse sensory data. As XR and HTC become
essential tools for metaverse applications, there is a growing imperative to
develop multisensory media (mulsemedia) that integrates a broad array of
sensory modalities [7, 8, 9].

Mulsemedia represents an advancement over traditional multimedia,
incorporating multiple sensory modalities beyond the visual and auditory.
Unlike a multimodal dataset, which is inherently static and pre-collected,
mulsemedia is a dynamic concept focusing on real-time streaming and
interactive applications [9]. Although existing multimodal data processing
techniques can facilitate the development of mulsemedia systems, mulsemedia
communication consists of two distinct categories: First, the precise
reconstruction of mulsemedia at remote destinations, which prioritizes
reconstruction fidelity [7]; and second, the application of mulsemedia to
complete specific tasks such as prediction, classification, and
identification, which emphasizes semantic fidelity. This paper addresses the
latter, developing task-oriented mulsemedia communication systems. We explore
an edge computing architecture, as depicted in Fig. 1, where Internet of
Things (IoT) sensors and devices transmit unimodal data to edge servers, such
as gateways and base stations. These edge servers aggregate the unimodal data
to process mulsemedia, subsequently transmitting the relevant results to user
devices. While existing IoT systems are capable of supporting such an
architecture, three major research challenges remain, fundamentally limiting
system performance in terms of task accuracy, end-to-end latency, and energy
consumption.

Figure 1: Illustration of mulsemedia communication and multi-task machine
learning in edge networks. The IoT sensors can capture senses including but
not limited to sight, hearing, touch, smell, taste, temperature, moisture, and
motion.

First, the heterogeneity in data dimensions and quality among various sensory
modalities presents significant challenges in data encoding, communication,
and fusion. Visual data such as videos and images typically dominate in
volume, whereas data from pressure sensors and motion tracking systems are
comparatively limited. In communication systems that utilize data packets, it
is imperative to prioritize smaller sensory modalities. Failure to do so may
result in the loss of an entire sensory modality with a single data packet
loss. Furthermore, sensory data captured at different times necessitates
sophisticated synchronization techniques. The presence of low-quality or
misaligned sensory data can adversely affect the efficacy of data fusion and,
consequently, the outcomes of related tasks.

Second, IoT devices exhibit considerable heterogeneity in their computing,
storage, and communication capabilities. Traditional multimodal data
processing employs modality-specific machine learning (ML) models. For
instance, images and audio are processed using two independent models to
extract latent features, which are subsequently fused to generate task-
specific outputs. For mulsemedia, which involves multiple sensory modalities,
this approach could require a simple IoT device to support numerous modality-
specific models, potentially exceeding the device’s capacity, as depicted in
Fig. 2. Furthermore, the challenge is compounded as feature extraction models
are often tailored to specific tasks. Employing a single sensory modality for
multiple tasks necessitates the use of a variety of models. Developing a
unified solution capable of processing arbitrary sensory modality for multiple
tasks remains a challenge.

Last, a sensory modality may not contain information related to a specific
task, or such information can be distorted by noise or interference. In
traditional multimodal ML systems, data is preprocessed and selected to ensure
relevance to the task; for instance, a text file synchronized with an audio
file. However, in real-time streaming scenarios, an audio file might contain
background noises that not only obscure the content but, in severe cases, can
make the associated file irrelevant. Thus, it is crucial to assess the task-
related information present in each sensory modality and to intelligently
combine these modalities to optimize task performance. This necessitates a
dynamic system capable of evaluating and integrating sensory data in real time
to effectively address these challenges.

To address the aforementioned challenges, our objective is to develop a
unified ML model capable of encoding any sensory modality into homogeneous
latent data suitable for transmission, while also adaptively selecting
essential sensory modalities to minimize communication end-to-end latency,
reduce energy consumption, and enhance task accuracy.

First, we propose a distributed and unified mulsemedia communication and data
fusion architecture, named MuSeCo, employing the Perceiver model [10] and the
HighMMT multimodal transformer [11]. This architecture allows for the
processing of sensory modalities of arbitrary data dimensions through the
unified Perceiver model, which transforms these inputs into homogeneous latent
data of significantly reduced size compared to the raw data. Notably, the
Perceiver model, in contrast to the traditional Transformer [12], can
accommodate inputs of any dimension and often requires fewer parameters due to
its reuse of attention layers. MuSeCo can be seamlessly integrated into IoT
devices equipped with various sensors; for instance, both a camera and a
microphone can operate using the same trained MuSeCo model without
necessitating any modality-specific modifications. As illustrated in Fig. 3,
each IoT device is equipped with a unified Perceiver module, eliminating the
need for multiple modality-specific ML models in devices with multiple
sensors.

Figure 2: Modality-specific machine learning model for IoT devices. A single
device can have multiple models for different sensory modalities (such as the
five human basic senses: sight, hearing, touch, smell and taste).

Second, we enhance the MuSeCo framework with a sensory modality selection and
combination mechanism employing Conformal Prediction [13]. The human brain
uses a multisensory correlation detector for causal inference, as highlighted
in [1]. The HighMMT model, however, operates without a clear evaluation of
each sensory modality’s role. Differently, MuSeCo leverages Conformal
Prediction to assess the correlation among modalities and adjust the
individual weights of modalities in the final output accordingly. Sensory
modalities that exhibit higher certainty and greater relevance to the task are
assigned larger weights. To optimize the integration of sensory data, we
propose two combination approaches: the equal-weight combination (EWC) and the
set-size-scaled combination (SSSC). The architecture of the proposed MuSeCo
system is depicted in Fig. 4.

Finally, we introduce an adaptive extension named MuSeCo-Adaptive, designed to
dynamically transmit Conformal Prediction scores and latent semantic data.
This innovation aims to strike a balance among minimal end-to-end latency,
reduced energy consumption, and maintained high task accuracy in systems
constrained by data communication limitations. We assess the performance of
both MuSeCo and MuSeCo-Adaptive using four multimodal datasets encompassing
six sensory modalities. Notably, the unified model successfully executes four
distinct tasks without the need for retraining. The sensory modalities
considered include image, audio, text, force, proprioception, and control
signals.

The remainder of this paper is organized as follows: Section II introduces
related work in the field. Section III details the proposed MuSeCo
architecture, including the development of the Perceiver model and the
distributed communication system model. Section IV discusses the methodologies
for adaptive modality selection and combination and introduces the data
communication protocols. In Section V, the system’s performance is evaluated
through both simulation and experimental methodologies. Finally, Section VI
concludes the paper.

Figure 3: Unified Perceiver model for IoT devices. A single model can be used
to process multiple sensory modalities (such as the five human basic senses:
sight, hearing, touch, smell and taste) for multiple tasks.

##  II Related Work

Two research domains relevant to this paper are semantic communication and
multimodal machine learning. In semantic communication, rather than
transmitting conventional bit streams, transmitters encode and send semantic
data to enhance reliability and minimize communication overhead [14]. This
approach serves two primary purposes. First, it facilitates data
reconstruction where the transmitter encodes semantic information, enabling
the receiver to reconstruct the same modality with high-precision semantic
details. This approach has been successfully implemented in various media
including text [15, 16], images and videos [17], and audio [18]. Second,
semantic communication is employed in task-oriented contexts such as image
retrieval, machine translation, and visual question answering [19, 20]. Task-
oriented semantic communication significantly reduces communication overhead
by requiring only the transmission of task-relevant information, thereby
reducing the volume of encoded data.

Previous research has also delved into multimodal semantic communication. For
instance, in [20], image and text modalities are integrated for the visual
question answering task. The work in [21] addresses the audio-visual parsing
task, while [22] considers modalities including text, image, and speech. These
studies typically employ modality-specific encoders, meaning each sensory
modality is processed using an encoder specifically designed for that type.
This paper distinguishes itself from the existing literature by utilizing a
unified encoder based on the Perceiver model, which eliminates the need for
modality-specific encoders. This unified encoder is capable of handling
arbitrary types of modalities, regardless of their input dimensions. Thus, it
is not confined to conventional modalities such as text, image, or speech,
offering a more versatile and comprehensive approach to multimodal semantic
communication.

The design of the proposed MuSeCo architecture is inspired by the advancements
in general intelligence for multimodal agents [23]. There is a trend in the
development of Artificial Intelligence (AI) agents that possess general
intelligence capabilities, enabling them to process multimodal data across
multiple tasks. For instance, the study in [24] reports on a unified
Transformer model trained to concurrently learn 7 tasks over 8 datasets,
primarily utilizing visual and language modalities. Similarly, the works in
[11, 25] feature unified Transformer models that handle 10 modalities for 15
tasks. While these models achieve state-of-the-art (SOTA) performance with
fewer parameters, their architectures are centralized, which limits their
applicability in distributed data communication environments. In contrast, the
MuSeCo architecture reorganizes and distributes the attention modules in
Perceiver [10] and HighMMT [11] to IoT devices and edge servers. This
decentralized approach facilitates more flexible and scalable applications in
real-world environments. Additionally, unlike the Perceiver and HighMMT models
that rely on static, pre-collected data, MuSeCo incorporates a modality
selection and combination module that adaptively processes available sensory
modalities. This feature enables dynamic handling of data inputs, enhancing
the system’s efficacy in diverse operational contexts.

The modality selection and combination process within the MuSeCo architecture
utilizes Conformal Prediction, which offers prediction certainty scores and
sets, rather than merely providing a straightforward prediction result [26,
27]. While ML models are capable of generating outputs for classification
tasks, these outputs typically lack information regarding the certainty of the
predictions. Conformal Prediction addresses this limitation by generating a
classification set, denoted as ${\bm{u}}$, along with a confidence level
$1-\alpha$. This ensures that the coverage of Conformal Prediction will
include the true class within ${\bm{u}}$ with a probability of $1-\alpha$.
Both theoretical analyses and practical implementations have demonstrated the
use of Conformal Prediction in enhancing ML systems [28, 27, 13]. In this
paper, we leverage the Softmax outputs from the Perceiver model as the basis
for generating conformal prediction results. These results are then utilized
to select sensory modalities and determine optimal modality combination
coefficients. This method enriches the predictive reliability and efficacy of
the system by integrating a quantitative measure of confidence into the
decision-making process, thereby enhancing the overall performance of the
multimodal integration.

##  III System Architecture and Modeling

Figure 4: MuSeCo system architecture using unified Perceiver encoder and
Conformal Prediction for Artificial Intelligence of Things.

In this section, we introduce the architecture of the proposed MuSeCo system,
which is designed for distributed deployment across AIoT systems. We also
develop an analytical model to describe the operations within this system. As
depicted in Fig. 4, the system incorporates $n_{m}$ IoT devices, such as
Raspberry Pis, each equipped with one or more sensors. Additionally, there are
$n_{d}$ user devices, each tasked with potentially different applications. At
the network’s edge, a gateway or base station hosts an edge server. This
server is responsible for collecting sensory data, conducting mulsemedia
modality analysis, and generating classification outputs based on this data.

TABLE I: Mathematical notation. Symbol |  Description
---|---
${\bm{M}}$ |  Sensory modality data
$n_{m}$ |  Number of IoT devices
$T_{b}$ |  Computation load of Perceiver module
$n_{t,i}$ |  Number of sensory modalities for Task $i$
${\bm{p}}$ |  Position embedding
${\bm{e}}$ |  Modality embedding
${\bm{L}}$ |  Latent semantic data
${\bm{c}}$ |  Softmax score
${\bm{u}}$ |  Conformal prediction set
$c_{s}$ |  Server computing capacity
$c_{iot}$ |  IoT device computing capacity
$r_{t}$ |  Communication data rate
$P_{t}$ |  Communication transmission power
$\Gamma_{s}$ |  Computing energy efficiency of the server
$\Gamma_{iot}$ |  Computing energy efficiency of the IoT device
$p_{h}$ |  Percentage of data samples only use Conformal Prediction in MuSeCo-Adaptive
$|{\bm{M}}|$ |  The overall number of bits used to represent all data in ${\bm{M}}$ in binary.

We consider a scenario with $n_{t}$ tasks, where the $i$-th task utilizes a
distinct set of sensory modalities denoted by
${\mathcal{T}}_{i}=\\{{\bm{M}}_{i,1},{\bm{M}}_{i,2},\ldots\\}$, with
${\bm{M}}_{i,j}$ representing the $j$-th sensory modality data for the $i$-th
task. In this paper, we assume that each task uses data from unique sensors,
implying that the sets of modalities for different tasks are disjoint:

$\displaystyle{\mathcal{T}}_{i}\cap{\mathcal{T}}_{j}=\emptyset\quad\text{for
}i\neq j.$ (1)

For instance, the image data of digits used in Task 1 (AV-MNIST) is not
utilized in Task 2 (UR-FUNNY) as detailed in Table III. Henceforth, unless
otherwise noted, we omit the task subscript $i$, and the discussions are
applicable to any task.

###  III-A Unimodal Encoder

The architecture of MuSeCo is shown in Fig. 4 where MuSeCo distributes the
unimodal feature encoding to IoT devices and leaves the multimodal fusion in
the edge server. This can reduce the communication overhead. In the following,
we introduce the unimodal encoder in MuSeCo.

A key challenge in feature encoding is that different modalities have
drastically different data dimension and format. For example, an image and a
force signal both can be represented in matrix format, but their dimensions
are different. For example, as shown in Table III, an image in Task 2 is
20$\times$371 and a force signal in Task 4 is 16$\times$7\. Moreover, images
are organized as spatial pixels and force signals are time-series data. Due to
the lack of unified tools, existing works use modality-specific ML models to
extract features and fuse them in later stages [22].

Transformer is widely used in image, text, and audio machine learning [12]. It
is also adopted in multi-task multimodal semantic communication [22]. However,
Transformer models cannot effectively reduce input data dimension and, thus,
data communication overhead. Thus, modality-specific Transformer and dimension
reduction models must be developed to process individual modality. This is
mainly due to the query-key-value (QKV) attention mechanism that is used in
Transformer.

Let ${\bm{M}}\in\mathbb{R}^{l_{m}\times d_{m}}$ denote the input sensory
modality and $f_{sa}\left({\bm{M}}\right)\in\mathbb{R}^{l_{m}\times d_{a}}$
represent the self-attention operation on ${\bm{M}}$, where
$d_{a}=d_{m}/h_{m}$ and $h_{m}$ is the number of self-attention module. In
Transformer, $f_{sa}(\cdot)$ is written as,

$\displaystyle
f_{sa}\left({\bm{M}}\right)=\operatorname{softmax}\left(\frac{{\bm{Q}}{\bm{K}}^{T}}{\sqrt{d_{a}}}\right){\bm{V}},$
(2)

where ${\bm{Q}}\in\mathbb{R}^{l_{m}\times d_{a}}$,
${\bm{K}}\in\mathbb{R}^{l_{m}\times d_{a}}$, and
${\bm{V}}\in\mathbb{R}^{l_{m}\times d_{a}}$ , and
$\operatorname{softmax}(\cdot)$ is the Softmax function. The QKV matrices are
generated using linear projections:

${\bm{Q}}={\bm{M}}{\bm{W}}_{Q},\quad{\bm{K}}={\bm{M}}{\bm{W}}_{K},\quad{\bm{V}}={\bm{M}}{\bm{W}}_{V},$

where $\bm{W}_{Q}\in\mathbb{R}^{d_{m}\times
d_{a}},\bm{W}_{K}\in\mathbb{R}^{d_{m}\times
d_{a}},\bm{W}_{V}\in\mathbb{R}^{d_{m}\times d_{a}}$. The multi-head self-
attention module can be modeled as

$\displaystyle
f_{mas}({\bm{M}})={\bm{M}}+[f_{sa}({\bm{M}}),\cdots,f_{sa}({\bm{M}})]{\bm{W}},$
(3)

where there are $h_{m}$ $f_{sa}({\bm{M}})$ in the bracket and
${\bm{W}}\in\mathbb{R}^{d_{m}\times d_{m}}$. The Transformer encoder generates
output:

$\displaystyle{\bm{L}}_{out}=f_{msa}({\bm{M}})+f_{ffn}(f_{msa}({\bm{M}})),$
(4)

where $f_{ffn}(\cdot)$ is the feedforward network. Note that, the output
${\bm{L}}_{out}$ and input ${\bm{M}}$ have the same dimension. As a result, if
the input has a large size, the Transformer encoder cannot generate latent
data with reduced dimension. For multimodal sensory data with different input
dimensions, modality-specific Transformer model must be designed, such as the
work in [22]. Additional steps are needed to combine different sensory
modalities after Transformer encoding.

Perceiver is developed by modifying Transformer [10]. In Transformer, the
complexity of the self attention is ${\mathcal{O}}(l_{m}^{2})$ which depends
on the input size. To address this issue, the Perceiver uses asymmetric QKV
attention module. In particular, ${\bm{Q}}$ becomes a projection of learned
latent data with dimension $l_{p}$ and $K$ and $V$ are projections of the
input data. Here, $l_{p}\ll l_{m}$ and complexity is reduced to
${\mathcal{O}}(l_{p}\cdot l_{m})$. As shown in Fig. 5, the output of the cross
attention layer has the same dimension as the latent matrix and the complexity
is reduced to ${\mathcal{O}}(l_{p}^{2})$ in the latent space. The Perceiver
uses a Fourier feature for position encoding and the small latent space allows
deep learning with a large number of layers. Also, Perceiver reuses parameters
in different layers which significantly reduces overall parameter number in
the system.

Figure 5: Illustration of the Perceiver architecture.

In order to accept all input modalities, the input ${\bm{M}}$ is concatenated
with one-hot modality embedding ${\bm{e}}$, position embedding ${\bm{p}}$ and
zero padding to make all input modalities the same size, i.e.,

$\displaystyle{\tilde{\bm{M}}}=[{\bm{M}};{\bm{e}};{\bm{p}};{\bm{0}}].$ (5)

Therefore, we need to select the sensory modality with the largest size and
use zero padding for other smaller modalities.

The Perceiver-based unimodal encoder can be written as:

$\displaystyle{\bm{L}}_{u}=f_{uni}({\tilde{\bm{M}}},{\bm{L}}_{0};\theta_{A}),$
(6)

where ${\bm{L}}_{u}\in{\mathbb{R}}^{l_{p}\times l_{a}}$ which is the output
latent data, ${\bm{L}}_{0}\in{\mathbb{R}}^{l_{p}\times l_{a}}$ is the trained
input latent data, and $\theta_{A}$ is the unimodal encoder parameters. Note
that, ${\bm{L}}_{u}$ and ${\bm{L}}_{0}$ have the same dimension. The IoT
device can directly send ${\bm{L}}_{u}$ to the edge server for further
multimodal fusion. Note that, compare to the raw sensory data, such as image
and audio, ${\bm{L}}_{u}$ can be much smaller due to the reduced dimension of
the latent space. For example, the latent dimension in Section V is
20$\times$64 which is much smaller than the image dimension in Task 2, i.e.,
$112\times 112$ as shown in Table III. This could reduce the data
communication overhead. However, for some time-series sensory modalities, such
as force and position, ${\bm{L}}_{u}$ can be larger than the raw data. Since
${\bm{L}}_{u}$ has the same dimension for all sensory modalities, it would be
over compressed if we make its dimension smaller than that of the smallest raw
sensory modality.

Note that, $f_{uni}(\cdot,\theta_{A})$ is the module named Unimodal Encoder A
in Fig. 4 and it will be trained together with the Multimodal Cross Attention
A in the edge server which will be introduced next.

###  III-B Multimodal Cross Attention

The edge server integrates latent data generated by Unimodal Encoder A in Fig.
4. Following the multimodal integration approach proposed in [11], the
Perceiver is used to integrate latent data as shown in Fig. 6. If the $i$-th
task requires $n_{t,i}$ modalities, then the model will run the multimodal
cross attention module for $n_{t,i}(n_{t,i}-1)$ times to cover all
combinations among the sensory modalities. The multimodal cross attention
architecture is the same as the Perceiver used in the Unimodal Encoder A.
Thus, the multimodal cross attention module can be modeled as

$\displaystyle{\bm{c}}_{mul}=$ $\displaystyle
f_{ffn}^{mul}(f_{mul}({\bm{L}}_{1},{\bm{L}}_{2};\theta_{A}^{m}),f_{mul}({\bm{L}}_{1},{\bm{L}}_{3};\theta_{A}^{m}),\cdots,$
$\displaystyle
f_{mul}({\bm{L}}_{n_{t,i}-1},{\bm{L}}_{n_{t,i}};\theta_{A}^{m}))$ (7)

where $f_{ffn}^{mul}(\cdot)$ is a two-layer feedforward network with output
size being the class number, $f_{mul}(\cdot;\theta_{A}^{m})$ is the Perceiver
architecture with parameter $\theta_{A}^{m}$. Then, using
$\operatorname{argmax}({\bm{c}}_{mul})$, we can obtain the classification
class label.

Unimodal Encoder A will be trained together with Multimodal Cross Attention A
using Cross Entropy Loss. It is the HighMMT model proposed in [11] by
connecting Unimodal Encoder A and Multimodal Cross Attention A. In this paper,
it is named as Distributed-HighMMT since the two modules are in different
devices.

Figure 6: The HighMMT architecture in [11] using unimodal encoder and
multimodal cross attention.

###  III-C Conformal Prediction in MuSeCo

To further reduce the communication overhead, the IoT device can perform more
processing towards completing the task. For example, the IoT device can send
unimodal classification results directly to the edge server, and the
multimodal fusion at the edge server can adopt majority vote by considering
each modality equally important. However, the sensory modalities are
heterogeneous in data quality and dimension and they cannot be considered
equally.

The key idea in MuSeCo is that the IoT device uses unimodal Conformal
Prediction [13] to evaluate its certainty about the classification results. If
the certainty is high, the sensory modality has strong task-related
information and can be fused in the edge server with high weights. However, if
the certainty is low, the sensory modality does not have sufficient task-
related information and it can confuse the multimodal cross attention module
in the edge server. Thus, the certainty is an important parameter that will be
generated by the conformal prediction.

In order to get the Conformal Prediction, the latent output of the Unimodal
Encoder A is fed to another unimodal encoder together with the input sensory
modality data, which can be modeled as

$\displaystyle{\bm{l}}_{cp}=f_{uni}({\tilde{\bm{M}}},{\bm{L}}_{u};\theta_{B}),$
(8)

where $\theta_{B}$ is the parameter of the Unimodal Encoder B in Fig. 4. Note
that, the output ${\bm{l}}_{cp}$ is a vector which is the last column of the
latent matrix. Then, a two-layer feedforward network
$f_{ffn}^{uni}({\bm{l}}_{cp};\theta_{p},{\mathcal{T}}_{i})$ is used, where
$\theta_{p}$ is the parameter of the feedforward network and
${\mathcal{T}}_{i}$ is the task. Note that, this output head
$f_{ffn}^{uni}({\bm{l}}_{cp};\theta_{p},{\mathcal{T}}_{i})$ is task-dependent
and we need one feedforward network for each task, as shown in Fig. 7. The
first layer’s input size is the length of ${\bm{l}}_{cp}$ and the second
layer’s output size is the class number of the dataset. In the end, Softmax
function is used to generate scores, which are

$\displaystyle{\bm{c}}=\operatorname{softmax}({\bm{l}}_{cp}),$ (9)

where ${\bm{c}}\in{\mathbb{R}}^{n_{c}}$ and $n_{c}$ is the class number of the
task.

Figure 7: Illustration of the Conformal Prediction outputs for 4 tasks. Each
task has its own output head to generate certainty scores and a prediction
set.

By using $\operatorname{argmax}({\bm{c}})$ we can obtain the predicted class.
The model can be trained using Cross Entropy Loss. Note that, Unimodal Encoder
A is trained before Unimodal Encoder B and they are not trained together.

However, using $\operatorname{argmax}({\bm{c}})$ we can only obtain a
prediction without showing any certainty. Conformal Prediction can transform
${\bm{c}}$ to scores which can represent the certainty. The intuitive
understanding is that if the certainty of the prediction is high, the largest
number in ${\bm{c}}$ must be close 1. On the other hand, if the certainty of
the prediction is low, the largest number is comparable to the second largest
number. Nevertheless, the Softmax outputs cannot represent real probabilities
to accurately represent certainty. Transformation is needed to convert Softmax
outputs to prediction certainty.

To obtain the Conformal Prediction score, a calibration dataset with $n_{cal}$
data examples ($x$,$y$) is needed which cannot be part of the training
dataset, where we use $x$ as the input and $y\in\\{1,2,\cdots,n_{c}\\}$ as the
label. According to [13], the score can be defined as
${c}_{cp}=1-{\bm{c}}[y]$, where ${\bm{c}}[y]$ denotes the $y$-th element (true
label) in vector ${\bm{c}}$. The $n_{cal}$ scores can be ranked in an array
${\bm{s}}$. Given a percentage $\alpha$, we can obtain a quantile ${\hat{q}}$
using

$\displaystyle{\mathbb{P}}({c}_{cp}<{\hat{q}})\geq 1-\alpha,\text{ for
}\forall{c}_{cp}\in{\bm{s}}.$ (10)

For a new data example ($x$,$\cdot$) with an unknown label, the Softmax output
values in ${\bm{c}}$ will be compared with $1-{\hat{q}}$. If any value in
${\bm{c}}$ is larger than $1-{\hat{q}}$, the associated class will be added to
a prediction set ${\bm{u}}$. For example, as shown in Fig. 7, if the Softmax
outputs are higher than a threshold $1-{\hat{q}}$, the associated class is
added to the prediction set. In this way, it is nearly guaranteed that [27]

$\displaystyle{\mathbb{P}}(y\in{\bm{u}})\approx 1-\alpha,$ (11)

where $y$ is the true label. Note that, the size $|{\bm{u}}|$ is related to
the prediction certainty, i.e., a small size means the model provides high
certainty, vice versa.

Using the conformal prediction, the IoT device can send the Softmax outputs
${\bm{c}}$ and the predicted set ${\bm{u}}$ to the edge server. Then, the edge
server can integrate multiple sensory modalities and make the final
prediction. The objective is to combine sensory modalities using their
certainty. For a task ${\mathcal{T}}$ with sensory modalities
$\\{{\bm{M}}_{1},{\bm{M}}_{2},\cdots\\}$, the edge server receives the
associated Softmax scores $\\{{\bm{c}}_{1},{\bm{c}}_{2},\cdots\\}$ and
associated prediction sets $\\{{\bm{u}}_{1},{\bm{u}}_{2},\cdots\\}$ from IoT
devices.

We consider the following two approaches to combine the Softmax outputs, named
equal-weight combination (EWC) and set-size-scaled combination (SSSC). In EWC,
all sensory modalities are considered equally. The Softmax outputs are added
together, upon which the predict label ${\hat{y}}$ can be obtained, i.e.,

$\displaystyle{\bm{c}}_{t}=\sum_{i=1}^{|{\mathcal{T}}|}{\bm{c}}_{i},\text{ and
}{\hat{y}}={\operatorname{argmax}}({\bm{c}}_{t}),$ (12)

where $|{\mathcal{T}}|$ is the number of sensory modalities in the task. In
this way, even the sensory modality quality is low, it will be considered the
same as the high-quality sensory modality. The certainty of each sensory
modality is not used in EWC.

To address this issue, we use the prediction set size as an indicator of
certainty and scale the Softmax outputs. In SSSC, the modality combination is
written as

$\displaystyle{\bm{c}}_{t}=\left(\sum_{m=1}^{|{\mathcal{T}}|}\frac{{\bm{c}}_{m}}{|{\bm{u}}_{m}|^{\beta}}\right)/\left(\sum_{m=1}^{|{\mathcal{T}}|}\frac{1}{|{\bm{u}}_{m}|^{\beta}}\right),\text{
and }{\hat{y}}={\operatorname{argmax}}({\bm{c}}_{t}),$ (13)

where $\beta\geq 1$ is a hyperparameter and $|{\bm{u}}|$ is the Conformal
Prediction set size. When the task class number is small, e.g., binary
classification with 2 classes, the prediction set size $|{\bm{u}}|$ is also
small (no larger than 2) but this does not represent strong certainty. By
using a large $\beta$, we can penalize large sets and reward small sets. For
example, consider that $\beta=2$. For a set size 1, its Softmax outputs would
not be changed, while for a set size 2, its Softmax outputs would be scaled by
0.25, which means higher uncertainty reduces its weight. Using SSSC, we can
mitigate the impact of modality heterogeneity by reducing the low-quality
sensory modality’s weight in the combination stage.

Up to this point, we have introduced two approaches for task-oriented
mulsemedia communication. First, we introduce the Distributed-HighMMT which
uses the Unimodal Encoder A and Multimodal Cross Attention A, as shown in Fig.
4. The IoT devices send unimodal latent data to the edge server for multimodal
fusion. Second, we introduce the MuSeCo model where the Unimodal Encoder A and
B generate Conformal Prediction scores and sets which are sent to the edge
server for weighted multimodal combination. In the next Section, we will
introduce the MuSeCo-Adaptive which can combine the Distributed-HighMMT and
MuSeCo. Also, we introduce the communication protocol and the evaluation
parameters.

##  IV Adaptive Mulsemedia Communication System

MuSeCo effectively reduces data communication overhead by transmitting only
Conformal Prediction scores and sets, though it typically achieves lower
classification accuracy compared to Distributed-HighMMT, as demonstrated in
Fig. 13. Unlike Distributed-HighMMT, which requires all sensory modalities to
function and fails in their absence, MuSeCo is capable of making predictions
with just a single sensory modality. Consequently, this presents a trade-off
among classification accuracy, reliability, and data communication overhead.
To address this and achieve a balanced solution, we introduce the MuSeCo-
adaptive.

###  IV-A Adaptive Model Selection

The basic idea of MuSeCo-Adaptive is to use the combined Softmax outputs
${\bm{c}}_{t}$ in Equ. (12) and Equ. (13). If we rank the elements in
${\bm{c}}_{t}$ and the largest value is much larger than the second largest
one, this means the modality combination has reached to a strong agreement.
However, if the largest value is similar to the second largest one, the result
of the modality combination is vague.

Figure 8: An example of the combined score ${\bm{c}}_{t}$ for a binary
classification problem.

In Fig. 8, we use a toy example to show the idea. We consider a binary
classification problem with two sensory modalities. The unimodal Softmax
outputs are ${\bm{c}}_{1}=[x_{1}^{1},x_{2}^{1}]$ and
${\bm{c}}_{2}=[x_{1}^{2},x_{2}^{2}]$. If we use the EWC in Equ. (12), we have
${\bm{c}}_{t}=({\bm{c}}_{1}+{\bm{c}}_{2})/2$. In Fig. 8, the line
$x_{1}=x_{2}$ is a boundary. If $c_{t}$ is on the left-hand side, the result
prefers class 2, vice versa. The ${\bm{c}}_{1}$, ${\bm{c}}_{2}$, and
${\bm{c}}_{t}$ must be on the line $x_{1}+x_{2}=1$. Next, we consider the
following three cases:

  * •

${\bm{c}}_{1}$ is close to [1,0] and ${\bm{c}}_{2}$ is on the right-hand side
of the boundary. In this case, ${\bm{c}}_{t}$ will be close to [1,0] and the
two sensory modalities reach to an agreement. Using EWC or SSSC, we can obtain
the same prediction result.

  * •

${\bm{c}}_{1}$ is close to [1,0] and ${\bm{c}}_{2}$ is close to [0,1]. In this
case, ${\bm{c}}_{t}$ will be close to $x_{1}=x_{2}$. It is challenging to
predict the label since the two sensory modalities provide contradict
evidence.

  * •

Both ${\bm{c}}_{1}$ and ${\bm{c}}_{2}$ are close to the boundary
$x_{1}=x_{2}$. Similar to the previous case, the model cannot make a strong
decision.

Generally, if $\max({\bm{c}}_{t})$ is smaller than a threshold, the approach
based on Softmax outputs and Conformal Prediction is not efficient and we
should use the Multimodal Cross Attention module. To determine the optimal
threshold, we reuse the calibration dataset that was used for Conformal
Prediction.

(a) Task 1.

(b) Task 2.

(c) Task 3.

(d) Task 4.

Figure 9: Softmax score distribution of equal-weight combination (EWC) for
correctly and wrongly classified samples.

(a) Task 1.

(b) Task 2.

(c) Task 3.

(d) Task 4.

Figure 10: Softmax score distribution of set-size-scaled combination (SSSC)
for correctly and wrongly classified samples.

In Fig. 9 and Fig. 10 we show the density of Softmax score for test samples
with correct and wrong labels using EWC and SSSC, respectively. We use MuSeCo
and evaluate 4 tasks given in Table III. As discussed before, IoT devices send
Softmax outputs and conformal prediction sets to the edge server. We consider
the data can be precisely received without any error. Then, the edge server
uses EWC and SSSC with $\beta=1$ to combine sensory modalities. As we can see
in Fig. 9, the majority of the correctly classified samples have higher
Softmax scores than the misclassified samples in Task 1, Task 3, and Task 4
using SSSC. In Table II, we provide a comprehensive comparison of the results
in Fig. 9 and Fig. 10. We show the 30% and 70% percentile of the classified
samples with correct and wrong labels, respectively. For example, the 30%
percentile of the classified samples with correct labels for Task 1 using EWC
is 0.37 which means 30% of correctly classified samples’ combined Softmax
scores are smaller than 0.37, while 70% of the misclassified samples’ softmax
scores are smaller than 0.36, which shows that the combined Softmax scores of
the correct and wrong groups have different distributions. A large difference
between the two percentiles indicates that the combined Softmax scores of
correct and wrong samples are more different. Moreover, if we use the 30%
percentile of the correctly classified samples as a threshold to obtain a
subset of the original dataset, in which all samples have the maximum Softmax
score larger than the threshold. In this subset, we can obtain the accuracy
and the covered sample percentage in Table II. We can consider these samples
are simple ones and those below the threshold are complex ones. As we can see
from the table, for all the four tasks, more than 57% samples’s maximum
Softmax scores are above the threshold and the obtained accuracy is higher
than using the complex Multimodal Cross Attention module (the SOTA in Table
III).

TABLE II: Comparison between EWC and SSSC based on the Softmax score distribution. Tasks |  Correct (30%) |  Wrong (70%) |  Accuracy (%) |  Sample Percentage (%)
---|---|---|---|---
Task 1 | EWC |  0.37 |  0.36 |  86.02 |  56.92
SSSC |  0.47 |  0.47 |  82.19 |  56.84
Task 2 | EWC |  0.56 |  0.59 |  71.90 |  59.57
SSSC |  0.55 |  0.60 |  69.10 |  68.59
Task 3 | EWC |  0.75 |  0.73 |  88.47 |  58.17
SSSC |  0.75 |  0.74 |  88.49 |  59.74
Task 4 | EWC |  0.40 |  0.44 |  97.85 |  68.38
SSSC |  0.66 |  0.49 |  98.92 |  68.38

It should be noted that, the numbers of correctly classified and misclassified
samples are different. Although the distributions for Task 2 are similar, the
accuracy remains high. Also, we notice that EWC and SSSC have similar
performance in Tasks 1 to 4. Since here we do not consider noises in sensory
modality, there is no significant differences in the predictions sets.

Based on the intuitive insights and the empirical analysis, we have shown that
a percentile can be used to obtain a threshold. As we can see in Table II, the
30% percentile varies across different tasks and combination approaches. To
address this issue, we use calibration datasets for each task to obtain the
threshold percentile. First, by using the calibration datasets
$\\{x_{cal},y_{cal}\\}$ and a percentage $\alpha_{2}$, we obtain a quantile
for each task ${\hat{q}}_{i}^{e}$ by only considering the Softmax score of the
correctly predicted samples. Note that, this quantile is used in the edge
server, which is different from the quantile used in Conformal Prediction.
Above the threshold, we consider the sensory modalities have consistent
semantic meaning and the combined Softmax score will be used to generate the
prediction outputs, while below the threshold, the Multimodal Cross Attention
A module is used.

###  IV-B Mulsemedia Communication Protocol

Based on previous analyses, we present the MuSeCo-Adaptive communication
protocol, which is also shown in Fig. 11. The user device can request for task
results. The edge server identifies the task-related sensory modalities,
sensors, and IoT devices and then send requests to identified IoT devices.
Each selected IoT device collects sensory data and use Unimodal Encoder A and
B to generate Softmax scores and Conformal Prediction set which are sent to
the edge server. If the combined maximum score is higher than the threshold
$\alpha_{2}$, the sample is considered simple and the edge server will send
the results to the user device and complete the process.

If the combined maximum score is lower than the threshold $\alpha_{2}$, the
sample is considered complex. It is challenging to classify this sample using
EWC or SSSC. The edge server will request for more latent data from the
Unimodal Encoder A. Since the latent data was used to generate Conformal
Prediction sets, IoT devices do not need to recompute and they can send saved
latent data. After receiving the latent data, the edge server uses the
Multimodal Cross Attention A module to generate the final classification
result which will be sent to the user device.

Figure 11: Illustration of MuSeCo-Adaptive communication protocol.

In order to comprehensively compare the performance of MuSeCo and MuSeCo-
Adaptive, we introduce five communication approaches with various setups. The
five approaches include,

  * •

A1 (Raw Data): The IoT devices send raw sensory data to the edge server. The
edge server has HighMMT model [11] to process the sensory data and generate
classification results.

  * •

A2 (Task Results): Each IoT device uses Unimodal Encoder A and B to generate
classification results and sends them to the edge server. The edge server
performs a majority vote (randomly select when the sensory modality number is
even) and send results to user device.

  * •

A3 (Latent Data): IoT devices send latent data output from Unimodal Encoder A
to the edge server, where Multimodal Cross Attention A module is used to
generate the task outputs which are sent to user devices.

  * •

A4 (MuSeCo: Softmax Scores and Conformal Prediction Sets): IoT devices send
Softmax scores and Conformal Prediction sets to the edge server. The edge
server uses SSSC to select and combine sensory modalities to generate task
outputs and send to user devices.

  * •

A5 (MuSeCo-Adaptive: Adaptive Transmission): IoT devices send Softmax scores
and Conformal Prediction sets to the edge server. If the edge server decides
that the sample is a complex one, it requests for latent data from each IoT
device. Then, it uses Multimodal Cross Attention A module to generate the task
outputs. The details are shown in Fig. 11.

Figure 12:  An example of the approaches A1 to A5 for Task 1 (AV-MNIST).

An example of the 5 approaches for Task 1 is given in Fig. 12. There are two
IoT devices and they are equipped with image and audio sensors, respectively.
In A1, the IoT devices send image and audio spectrogram to the edge server
which uses the HighMMT model to identify the digit. In A2, each IoT device
sends its classification result to the edge server which will select the most
frequent result. In A3, each IoT device sends its latent data to the edge
server, and the edge server uses Multimodal Cross Attention A module to
generate the classification result. In A4, each IoT device sends the unimodal
Softmax score and Conformal Prediction sets with 90% certainty to the edge
server. For example, Sensor 1 has a high uncertainty and the largest three
Softmax scores are close which are associated with the numbers 3, 9, and 2.
With 90% confidence, the true digit is within the set [3,9,2,0,8,4,7,1,6]. The
large set size indicates that Sensor 1’s result is not reliable. On the
contrary, Sensor 2’s set only has number 3 which indicates high confidence. In
SSSC combination, Sensor 1’s Softmax score will be discounted by $1/9^{\beta}$
and then added with Sensor 2’s Softmax score. In A5, each IoT device sends the
same Conformal Prediction information as that in A4 to the edge server. The
calibration threshold is 0.4983. If the combined maximum score (e.g., using
SSSC) is higher than 0.4983, the edge server will proceed to make a
prediction. In this example, the maximum combined score is 0.8745 which is
higher than the threshold and the result is digit 3. However, if the maximum
combined score is lower than 0.4983, the edge server will request for latent
data and the Multimodal Cross Attention A module will be used.

Next, we introduce the communication system and evaluation parameters, which
are followed by comprehensive comparisons in the next section.

###  IV-C Communication and Evaluation Parameters

In this paper, we evaluate the trade-offs among the task accuracy, end-to-end
latency, and energy consumption. In order to simulate a wireless communication
system with various signal-to-noise ratios (SNR), we convert float decimal
data into binary data and then send the binary data using modulated symbols.
We consider uncoded modulation schemes and each float decimal number is
encoded into 18 bits with 9 bits for the integer part and 9 bits for the
fraction part. As we can see in Fig. 12, the data transmitted from the sensor
device is drastically different using different approaches. For example, in A1
the transmitted raw sensory data is 240 kbits and in A3 the transmitted latent
data is 23 kbits, while it is only 36 bits in A2. We consider the raw sensory
data and latent data are transmitted using 64QAM modulation due to their large
data size, while classification results, Softmax scores, and Conformal
Prediction sets are sent using QPSK. IoT devices use the same transmission
power $P_{t}$ for data communication.

We assume all the IoT devices have the same computing capacity which is
represented by $c_{iot}$. The computing capacity is measured using Floating-
point Operations Per Second (FLOPS). The edge server’s computing capacity is
$c_{s}$ and one Perceiver unit (i.e., Unimodal Encoder A and B, the Multimodal
Cross Attention A) requires $T_{b}$ floating-point operations.

The overall required computation for task $i$ is related to the number of
modalities and the communication approaches. For A1 the total computation is

$\displaystyle T_{1}\approx
T_{b}(|{\mathcal{T}}|)+T_{b}(|{\mathcal{T}}|-1)|{\mathcal{T}}|.$ (14)

The end-to-end latency is defined as the time from the user device requests
for task output to the time it receives the result. Since the majority of the
latency consists of the transmission of raw data from IoT devices and the
computation in the edge server, the end-to-end latency for a task $i$ is
approximated by

$\displaystyle\tau_{1}=\sum_{m=1}^{|{\mathcal{T}}|}|{\bm{M}}_{m}|/r_{t}+T_{1}/c_{s}.$
(15)

where $r_{t}$ is the data rate. The IoT devices use Time Division Multiple
Access (TDMA) and the data transmission is sequential.

Similarly, the energy consumption for task $i$ is approximated using the
communication and computation energy consumption, which is

$\displaystyle E_{1}\approx
P_{t}\times(\sum_{m=1}^{|{\mathcal{T}}|}|{\bm{M}}_{m}|/r_{t})+T_{1}/\Gamma_{s},$
(16)

where $\Gamma_{s}$ is the energy efficiency of the server, which is measured
in FLOPS/W (FLOPs/Joule) [29]. The task accuracy is measured based on the ML
model outputs and the ground truth labels.

In A2, the majority of the computation task is performed in the IoT device and
the latency is

$\displaystyle\tau_{2}\approx 2T_{b}/c_{iot}.$ (17)

Here, we implicitly assume that all the IoT devices’ computations are
parallel. The total energy consumption can be written as

$\displaystyle E_{2}\approx 2T_{b}(|{\mathcal{T}}|)/\Gamma_{iot},$ (18)

where $\Gamma_{iot}$ is the energy efficiency of the IoT device. The
communication latency and communication energy consumption are neglected since
each IoT device only sends its classification task results.

In A3, the latency consists of three major parts, i.e., the computation in the
IoT device, the latent data transmission from the IoT device to the edge
server, and the computation in the edge server. Thus, we have

$\displaystyle\tau_{3}\approx
T_{b}/c_{iot}+\sum_{m=1}^{|{\mathcal{T}}|}|{\bm{L}}_{u,m}|/r_{t}+(|{\mathcal{T}}|-1)(|{\mathcal{T}}|)T_{b}/c_{s},$
(19)

where ${\bm{L}}_{u,m}$ the latent data of the $m$-th sensory modality for a
task. The total energy consumption can be written as

$\displaystyle E_{3}\approx
T_{b}(|{\mathcal{T}}|)/\Gamma_{iot}+\sum_{m=1}^{|{\mathcal{T}}|}|{\bm{L}}_{u,m}|P_{t}/r_{t}+(|{\mathcal{T}}|-1)(|{\mathcal{T}}|)T_{b}/\Gamma_{s}.$
(20)

In A4, the latency $\tau_{4}$ and energy consumption $E_{4}$ can be
approximated using $\tau_{2}$ and $E_{2}$, respectively. The Softmax scores
and Conformal Prediction sets have limited data and the computation is mainly
performed in IoT devices.

In A5, the latency and energy consumption depend on the percentage of simple
samples, which is denoted by $p_{h}$. For simple samples, the latency is
$\tau_{4}$ and the energy consumption is $E_{4}$. For complex samples, the
latency consists of three major parts: the unimodal encoding using Unimodal
Encoder A and B, latent data transmission, and Multimodal Cross Attention
computation. This is because the MuSeCo-Adaptive uses MuSeCo first and then
uses latent data, as shown in Fig. 11. Here, we implicitly assume the steps
(5) to (7) in Fig. 11 take negligible time. As a result, the latency of A5 is

$\displaystyle\tau_{5}$ $\displaystyle\approx
p_{h}(2T_{b}/c_{iot})+(1-p_{h})(2T_{b}/c_{iot}+\sum_{m=1}^{|{\mathcal{T}}|}|{\bm{L}}_{u,m}|/r_{t}$
$\displaystyle+(|{\mathcal{T}}|-1)(|{\mathcal{T}}|)T_{b}/c_{s}),$ (21)

and the total energy consumption is

$\displaystyle E_{5}$ $\displaystyle\approx
p_{h}(2T_{b}(|{\mathcal{T}}|)/\Gamma_{iot})+(1-p_{h})(2T_{b}(|{\mathcal{T}}|)/\Gamma_{iot}$
$\displaystyle+\sum_{m=1}^{|{\mathcal{T}}|}|{\bm{L}}_{u,m}|P_{t}/r_{t}+(|{\mathcal{T}}|-1)(|{\mathcal{T}}|)T_{b}/\Gamma_{s}).$
(22)

The above latency and energy consumption models can be applied to any task.
The only difference is the task sensory modality number and input sensory
modality data volume.

##  V Mulsemedia Communication Performance Evaluation

The performance evaluation uses 4 tasks, including Task 1 AV-MNIST [30], Task
2 UR-FUNNY [31], Task 3 MOSEI [32], and Task 4 modified PUSH [33]. The sensory
modality dimension and the SOTA classification accuracy are provided in Table
III. Task 1 uses both image and audio spectrogram to identify digits 0 to 9.
It is a modified version of the original MNIST dataset. Since both image and
audio MNIST have been extensively studied, in AV-MNIST the image quality is
reduced and additional noises are added to the audio for multimodal
classification. Task 2 and Task 3 are humor detection and emotion
classification tasks, respectively. Task 4 is generated from the MUJOCO PUSH
task [33] where a robot arm pushes a circular puck on a surface. The original
task is to predict the 2D location of the unknown object when the robot is
manipulating the object. Since Task 1 to 3 are classification problems, we
transform Task 4 into a classification problem by dividing the surface into 9
equal-sized regions. The prediction problem becomes a classification problem
of identifying the region that the object will be located in. We import the
dataset from the MultiBench dataset [25]. Our simulation and experiments are
performed using NVIDIA V100 GPU with 32 GB memory. We train Distributed-
HighMMT and MuSeCo each for 3 times and test their performance. The learning
rate is 0.001 and the weight decay is 0.001. The depth of each Perceiver
module is 1, the number of latents is 20, the latent dimension is 64, the
number of heads for cross attention is 1, the number of heads for latent self
attention is 6, and the cross attention head dimension and latent self
attention head dimension are both 64. The models are offline trained and
deployed for online execution.

TABLE III: Task sensory modality, input dimension and state of the art accuracy. | Task 1 | Task 2 | Task 3 | Task 4
---|---|---|---|---
Sensory Modality | Image | Audio | Image | Audio | Text | Image | Audio | Text |  Gripper Pos |  Gripper Sensors | Image | Control
Dimension | 28$\times$28 | 112$\times$112 | 20$\times$371 | 20$\times$81 | 50$\times$300 | 50$\times$35 | 50$\times$74 | 50$\times$300 |  16$\times$3 |  16$\times$7 | 16$\times$32$\times$32 | 16$\times$7
SOTA | 71.1 | 66.2 | 80.2 | —
*The SOTA accuracy is obtained from [11]. Task 1 is the AV-MNIST [30], Task 2 is the UR-FUNNY [31], Task 3 is the MOSEI [32], and Task 4 is
modified PUSH [33].

###  V-A Reliable Communication

The development of 5G and Beyond wireless systems supports ultra-reliable
communication for IoT systems, particularly the Ultra-Reliable Low Latency
Communication (URLLC) service. In this subsection, we consider the wireless
communication is reliable and all data can be transmitted with high fidelity,
i.e., the data quality is limited by the sensor precision rather than data
communication.

Figure 13:  Classification accuracy for 4 tasks using approaches A1 to A5. The
single sensory modality performance is labeled as M1 to M4. UNI represents the
HighMMT model for uni-task using all available sensory modalities in the
associated task.  Figure 14:  End-to-end latency using approaches A1 to A5
with different communication data rates for Task 1. Figure 15:  Energy
consumption using approaches A1 to A5 with different communication data rates
Task 2.

In Fig. 13, the classification accuracy of each task using approaches A1 to A5
are shown. For Conformal Prediction $\alpha$ is set as 0.1 and the percentile
threshold for adaptive model selection is obtained using a percentage of 30%
for correctly classified calibration samples. In order to better understand
the performance, we also provide M1 to M4 and the UNI model. The single
sensory modality performance for each task is labeled as M1 to M4. For
example, Task 1 only has two sensory modalities and the accuracy is obtained
using a unified single modality model (trained using 6 modalities with all 12
sensory data) for 4 tasks. UNI represents the performance of the HighMMT model
trained for uni-task using all available sensory modalities in the associated
task.

First, each sensory modality has different task-related information. For
example, M2 in Task 1 has much higher accuracy than M1. The uni-task (UNI)
model for Task 1 performs slightly better than only using sensory modality 2.
This shows that the sensory modalities are imbalanced in providing task-
related information. Although deep learning models can extract underlying
information and outperform single modality performance, e.g., Task 1 and Task
2, they can also perform similar as the single modality model (e.g., Task 3).

In A2, all sensory modalities are considered equally important. Due to the
imbalanced task-related information, the low-accuracy sensory modalities will
create negative impact on the final decision. As a result, A2 always performs
the worst. A1 and A3 have similar performance except for Task 4. They are not
exactly the same due to the limitation of GPU memory and we used a reduced
test data size for A3. Note that, A1 is the same as the HighMMT in [11]. We
notice that for all 4 tasks either A3 or A4 can achieve the best performance
and A5’s performance is in between A3’s and A4’s since it is a combination of
them. Overall, A4 (MuSeCo) and A5 (MuSeCo-Adaptive) can achieve the highest
average accuracy of 74.5% and 74.3%, which are similar to A1 (73.9%) and A3
(73.3%) and significantly better than A2 (64.1%).

To evaluate the latency and energy consumption, we consider the IoT device is
a typical microcomputer, such as Raspberry pi3, with energy efficiency
$\Gamma_{iot}=0.813$ GFLOPS/W and computing capacity $c_{iot}=3.62$ GFLOPS.
The edge server is a powerful machine with $\Gamma_{s}=2.13$ GFLOPS/W and
computing capacity $c_{s}=428$ GFLOPS, e.g., Intel E5-2640 v3. We use fvcore
library to count the number of FLOPS of the unimodal encoder and the
multimodal cross attention module. They have the same structure and we
consider the number of FLOPS are the same which is around 500 MFLOPS. The
transmission power $P_{t}$ is 100 mW. The data rate using 64QAM modulation is
shown as the x-axis in Fig. 14 and Fig. 15. Since all the 4 tasks have similar
latency and energy consumption performance, we only use Task 1 as an example.
In the simulation, the average percentage ($p_{h}$) of samples using Conformal
Prediction in A5 is 53.71%.

As shown in Fig. 14 and Fig. 15, A1 is the best solution if there is no
limitation on data communication, i.e., the data rates can be high enough
(larger than 1.5 Mbps). The latency and energy consumption of performing
computations in the IoT device is higher than that in the edge server.
Therefore, it is faster and more energy-efficient to offload all computations
to the edge server when data communication is fast. However, since the
datasets have low-quality images and videos (e.g., 28 $\times$ 28), the
required data rate is low. If the task requires high-resolution images and
videos, the required data rates can be much higher than 1.5 Mbps.

When the data rate is lower than 400 kbps, A2 and A4 have the lowest latency.
In this scenario, the overall latency is constrained by communication latency.
Since A2 only sends classification results and A4 sends Softmax scores and
Conformal Prediction sets, the communication latency is low compared with
sending raw data and latent data. Between 400 kbps and 1500 kbps, A3 has the
lowest latency. In this data rate interval, the communication latency is
smaller than sending raw data. Moreover, the offloading of multimodal cross
attention computation to the edge server also reduces computation latency. The
latency of A5 is between A3 and A4 when the performance is constrained by
communication. As the available data rates increase, the latency caused by
sending latent data and multimodal cross attention computation in the edge
server are negligible and the latency of A5 converges to that of A2 and A4.

Similarly, when the data rate is low, the energy consumption of communication
is high and A2 and A4 have the lowest energy consumption, as shown in Fig. 15.
Similar to the latency, as data rates increase, A3 and then A1 have the lowest
energy consumption. This is mainly due to the high energy efficiency of the
edge server and more computation workload is offloaded to the edge server.

From the above discussions, we can see that A4 (MuSeCo) demonstrates high
accuracy, low latency, and low energy consumption when the system is
constrained by data communication. There is a transition interval in data rate
where A3 performs better than any other approaches. When the system is not
constrained by data communication, A1 can achieve the lowest latency and
energy consumption. Note that, the exact data rates of the transition interval
depend on the specific task and raw data size. Task-dependent communication
and computation approaches can further improve the performance.

###  V-B Unreliable Communication

Although 5G and Beyond wireless systems have significantly improved
communication reliability and data rates, the IoT networks still experience
unreliable transmission occasionally due to interference, limited
infrastructure support, and attacks. In the following, we evaluate two
scenarios. First, a single sensory modality is affected by unreliable
communication and all other sensory modalities are reliably received by the
edge server. Second, all sensory modalities are affected by unreliable
communication. Here, we consider Additive White Gaussian Noise (AWGN) channel.

Figure 16:  Impact of imbalanced sensory modalities due to unreliable data
communication.

In all 4 tasks, the first sensory modality M1’s SNR is set to 10 and 20 dB and
all other sensory modalities’ SNR are set to 30 dB. As we can see in Fig. 16,
a single modality with high noise power can affect the overall accuracy
compared with the results in Fig. 13. First, A1’s accuracy is significantly
reduced since the raw sensory data is highly noisy which makes it challenging
to classify the sample. Since the model is trained using clean data, the
impact of noise can dramatically change the accuracy. For example, the
accuracy of Task 1 is reduced from around 70% to around 20%. The accuracy of
A1 for Task 3 is stable since M1 in Task 3 does not have strong task-related
information compared with M3. Since A2 sends unimodal classification results
and any error in data communication can cause significant errors, its accuracy
is also affected by the noisy channel. But the impact is mitigated when there
are many sensory modalities, e.g., Task 4 has 4 sensory modalities and A2’s
accuracy remains stable. In A3, latent data is sent to the edge server.
Similar to A1, the noisy latent data of M1 creates negative impact on the
multimodal cross attention and significantly reduces the classification
accuracy. Although the accuracy of A4 and A5 are also affected by the noisy
channel, the performance is among the best or close to the best in Task 1, 2
and 4. Also, due to the noise, the classification accuracy has a larger
variance compared with the results in Fig. 13.

Figure 17: Impact of noisy sensory modalities due to unreliable data
communication.

In Fig. 17, we consider two SNRs, i.e., 15 and 25 dB, for all sensory
modalities. When SNR is 25 dB, we can consider the performance is similar to
the reliable communication shown in Fig. 13. As we can see, A4 and A5 remain
to be two reliable solutions without significant reduction in accuracy among
the four tasks. The results for Task 3 is slightly different from other tasks.
In particular, A1’s accuracy decreases but not as significant as other tasks.
A1 and A3 experience significant drop since the data is noisy. Although A2
performs better than A1 and A3 at low SNR, it cannot intelligently combine
multimodal sensory data and its performance is worse than A4 and A5.

In general, when the data communication is reliable A1, A3, A4, and A5 have
similar performance in task accuracy, but A1, A3, and A4 demonstrates low
latency and low energy consumption at different communication data rates.
However, considering that A4’s accuracy in reliable communication systems is
slightly lower than that of A3 in 3 out of the 4 considered tasks (see Fig.
13), A5 can be a substitution of A4 to obtain higher accuracy with longer
latency and higher energy consumption. With noisy communication channels, A4
and A5 demonstrates better performance compared with other solutions.

##  VI Conclusion

The Artificial Intelligence of Things (AIoT) enhances data fusion and
communication efficiency, but the need for varied machine learning (ML) models
to process different sensory modalities complicates the system and challenges
large-scale deployment. In our study, we introduce MuSeCo and MuSeCo-Adaptive
for AIoT, employing a unified Perceiver model and Conformal Prediction. This
approach allows IoT devices equipped with various sensors to use a single
Perceiver model for data encoding. Conformal Prediction offers a lightweight
solution for multimodal data fusion, which can select and integrate imbalanced
multimodal data and reduce communication overhead. The model is trained across
six sensory modalities for four distinct tasks. Evaluation results reveal that
MuSeCo achieves high accuracy, minimal end-to-end latency, and low energy
consumption in systems constrained by data communication.

## References

  * [1] J. Pesnot Lerousseau, C. V. Parise, M. O. Ernst, and V. van Wassenhove, “Multisensory correlation computations in the human brain identified by a time-resolved encoding model,” _Nature communications_ , vol. 13, no. 1, p. 2489, 2022.
  * [2] C. V. Parise and M. O. Ernst, “Correlation detection as a general mechanism for multisensory integration,” _Nature communications_ , vol. 7, no. 1, p. 11543, 2016.
  * [3] K. Antonakoglou, X. Xu, E. Steinbach, T. Mahmoodi, and M. Dohler, “Toward haptic communications over the 5g tactile internet,” _IEEE Communications Surveys & Tutorials_, vol. 20, no. 4, pp. 3034–3059, 2018.
  * [4] R. Joda, M. Elsayed, H. Abou-Zeid, R. Atawia, A. B. Sediq, G. Boudreau, M. Erol-Kantarci, and L. Hanzo, “The internet of senses: Building on semantic communications and edge intelligence,” _IEEE Network_ , 2022.
  * [5] I. F. Akyildiz and H. Guo, “Wireless Communication Research Challenges for Extended Reality (XR),” _ITU Journal on Future and Evolving Technologies_ , vol. 3, no. 2, pp. 273–287, 2022.
  * [6] I. F. Akyildiz and H. Guo, “Holographic-type communication: A new challenge for the next decade,” _ITU Journal on Future and Evolving Technologies_ , vol. 3, no. 2, pp. 421–442, 2022.
  * [7] A. Covaci, L. Zou, I. Tal, G.-M. Muntean, and G. Ghinea, “Is multimedia multisensorial?-a review of mulsemedia systems,” _ACM Computing Surveys (CSUR)_ , vol. 51, no. 5, pp. 1–35, 2018.
  * [8] A. C. da Silveira and C. A. S. Santos, “Ongoing challenges of evaluating mulsemedia qoe,” in _Proceedings of the 2nd Workshop on Multisensory Experiences-SensoryX’22_. SBC, 2022.
  * [9] I. F. Akyildiz, H. Guo, R. Dai, and W. Gerstacker, “Mulsemedia communication research challenges for metaverse in 6G wireless systems,” _ITU Journal on Future and Evolving Technologies, Volume 4 (2023), Issue 4, Pages 562-579_ , vol. 4, no. 4, pp. 562–579, 2023.
  * [10] A. Jaegle, F. Gimeno, A. Brock, O. Vinyals, A. Zisserman, and J. Carreira, “Perceiver: General perception with iterative attention,” in _International conference on machine learning_. PMLR, 2021, pp. 4651–4664.
  * [11] P. P. Liang, Y. Lyu, X. Fan, J. Tsaw, Y. Liu, S. Mo, D. Yogatama, L.-P. Morency, and R. Salakhutdinov, “High-modality multimodal transformer: Quantifying modality & interaction heterogeneity for high-modality representation learning,” _Transactions on Machine Learning Research_ , 2022\.
  * [12] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [13] A. N. Angelopoulos, S. Bates _et al._ , “Conformal prediction: A gentle introduction,” _Foundations and Trends in Machine Learning_ , vol. 16, no. 4, pp. 494–591, 2023.
  * [14] G. Shi, Y. Xiao, Y. Li, and X. Xie, “From semantic communication to semantic-aware networking: Model, architecture, and open problems,” _IEEE Communications Magazine_ , vol. 59, no. 8, pp. 44–50, 2021.
  * [15] L. Yan, Z. Qin, R. Zhang, Y. Li, and G. Y. Li, “Resource allocation for text semantic communications,” _IEEE Wireless Communications Letters_ , vol. 11, no. 7, pp. 1394–1398, 2022.
  * [16] H. Xie, Z. Qin, G. Y. Li, and B.-H. Juang, “Deep learning enabled semantic communication systems,” _IEEE Transactions on Signal Processing_ , vol. 69, pp. 2663–2675, 2021.
  * [17] P. Jiang, C.-K. Wen, S. Jin, and G. Y. Li, “Wireless semantic communications for video conferencing,” _IEEE Journal on Selected Areas in Communications_ , vol. 41, no. 1, pp. 230–244, 2022.
  * [18] Z. Weng and Z. Qin, “Semantic communication systems for speech transmission,” _IEEE Journal on Selected Areas in Communications_ , vol. 39, no. 8, pp. 2434–2444, 2021.
  * [19] H. Xie, Z. Qin, X. Tao, and K. B. Letaief, “Task-oriented multi-user semantic communications,” _IEEE Journal on Selected Areas in Communications_ , vol. 40, no. 9, pp. 2584–2597, 2022.
  * [20] H. Xie, Z. Qin, and G. Y. Li, “Task-oriented multi-user semantic communications for vqa,” _IEEE Wireless Communications Letters_ , vol. 11, no. 3, pp. 553–557, 2021.
  * [21] P. Wang, J. Li, C. Liu, X. Fan, M. Ma, and Y. Wang, “Distributed semantic communications for multimodal audio-visual parsing tasks,” _IEEE Transactions on Green Communications and Networking_ , 2024.
  * [22] G. Zhang, Q. Hu, Z. Qin, Y. Cai, G. Yu, and X. Tao, “A unified multi-task semantic communication system for multimodal data,” _IEEE Transactions on Communications_ , 2024.
  * [23] Z. Durante, Q. Huang, N. Wake, R. Gong, J. S. Park, B. Sarkar, R. Taori, Y. Noda, D. Terzopoulos, Y. Choi _et al._ , “Agent ai: Surveying the horizons of multimodal interaction,” _arXiv preprint arXiv:2401.03568_ , 2024\.
  * [24] R. Hu and A. Singh, “Unit: Multimodal multitask learning with a unified transformer,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 1439–1449.
  * [25] P. P. Liang, Y. Lyu, X. Fan, A. Agarwal, Y. Cheng, L.-P. Morency, and R. Salakhutdinov, “Multizoo & multibench: A standardized toolkit for multimodal deep learning,” _The Journal of Machine Learning Research_ , vol. 24, no. 1, pp. 11 056–11 062, 2023.
  * [26] G. Shafer and V. Vovk, “A tutorial on conformal prediction.” _Journal of Machine Learning Research_ , vol. 9, no. 3, 2008.
  * [27] A. N. Angelopoulos, S. Bates, M. Jordan, and J. Malik, “Uncertainty sets for image classifiers using conformal prediction,” 2021.
  * [28] R. F. Barber, E. J. Candes, A. Ramdas, and R. J. Tibshirani, “Conformal prediction beyond exchangeability,” _The Annals of Statistics_ , vol. 51, no. 2, pp. 816–845, 2023.
  * [29] R. Desislavov, F. Martínez-Plumed, and J. Hernández-Orallo, “Trends in ai inference energy consumption: Beyond the performance-vs-parameter laws of deep learning,” _Sustainable Computing: Informatics and Systems_ , vol. 38, p. 100857, 2023.
  * [30] V. Vielzeuf, A. Lechervy, S. Pateux, and F. Jurie, “Centralnet: a multilayer approach for multimodal fusion,” in _Proceedings of the European Conference on Computer Vision (ECCV) Workshops_ , 2018, pp. 0–0.
  * [31] M. K. Hasan, W. Rahman, A. Bagher Zadeh, J. Zhong, M. I. Tanveer, L.-P. Morency, and M. E. Hoque, “UR-FUNNY: A multimodal language dataset for understanding humor,” in _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. Hong Kong, China: Association for Computational Linguistics, Nov. 2019, pp. 2046–2056. [Online]. Available: https://www.aclweb.org/anthology/D19-1211
  * [32] A. B. Zadeh, P. P. Liang, S. Poria, E. Cambria, and L.-P. Morency, “Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph,” in _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 2018, pp. 2236–2246.
  * [33] M. A. Lee, B. Yi, R. Martín-Martín, S. Savarese, and J. Bohg, “Multimodal sensor fusion with differentiable filters,” in _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2020, pp. 10 444–10 451.