# Non-contact Multimodal Indoor Human Monitoring Systems: A Survey

Le Ngu Nguyen⋆ Praneeth Susarla⋆ Anirban Mukherjee⋆† Manuel Lage Cañellas⋆
Constantino Álvarez Casado⋆ Xiaoting Wu⋆
Olli Silvén† Dinesh Babu Jayagopi† Miguel Bordallo López⋆ Praneeth Susarla, Le
Ngu Nguyen, Anirban Mukherjee, and Manuel Lage Cañellas have contributed
equally to this work.This research has been supported by the Academy of
Finland 6G Flagship program under Grant 346208 and PROFI5 HiDyn under Grant
32629, and the InSecTT project, which is funded under the European ECSEL Joint
Undertaking (JU) program under grant agreement No 876038. ⋆ Center for Machine
Vision and Signal Analysis, University of Oulu, Finland
† International Institute of Information Technology, Bangalore, India

###### Abstract

Indoor human monitoring systems leverage a wide range of sensors, including
cameras, radio devices, and inertial measurement units, to collect extensive
data from users and the environment. These sensors contribute diverse data
modalities, such as video feeds from cameras, received signal strength
indicators and channel state information from WiFi devices, and three-axis
acceleration data from inertial measurement units. In this context, we present
a comprehensive survey of multimodal approaches for indoor human monitoring
systems, with a specific focus on their relevance in elderly care. Our survey
primarily highlights non-contact technologies, particularly cameras and radio
devices, as key components in the development of indoor human monitoring
systems. Throughout this article, we explore well-established techniques for
extracting features from multimodal data sources. Our exploration extends to
methodologies for fusing these features and harnessing multiple modalities to
improve the accuracy and robustness of machine learning models. Furthermore,
we conduct comparative analysis across different data modalities in diverse
human monitoring tasks and undertake a comprehensive examination of existing
multimodal datasets. This extensive survey not only highlights the
significance of indoor human monitoring systems but also emphasizes their
versatile applications. In particular, we emphasize their critical role in
enhancing the quality of elderly care, offering valuable insights into the
development of non-contact monitoring solutions applicable to the needs of
aging populations.

##  I Introduction

Figure 1: Non-contact elderly monitoring system

One important objective of the WHO Global strategy and action plan on ageing
and health is to customize healthcare systems to the needs of elderly people
[1]. Indoor human monitoring is an essential component of an elderly care
system, providing critical decision-making, personalized care, emotional
support or interdisciplinary collaboration, ultimately enhancing the overall
effectiveness and reliability of remote care delivery [2]. Indoor human
monitoring systems observe the indoor environments using sensors such as
cameras [3], WiFi devices and radars [4], or wearable devices [5]. Every
sensor possesses its own set of advantages and disadvantages, making them
unique in their capabilities and limitations. For example, a camera can
capture more reliable information about the users but can also intrude on
their privacy leading to discomfort and objections from the individuals being
monitored. Ambient WiFi signals can be analyzed to infer human activities in a
non-intrusive manner with limited reliability. Radar devices can measure
subtle chest displacements to estimate vital signs but may struggle to
identify objects and are expensive to monitor objects with optimal
performance. While the unimodal usage of these sensors posses limitations, a
combination of visual data and wireless signals is more robust and practical,
providing holistic information to implement non-contact human monitoring
systems [6].

Multimodal systems provides a multi-aspect view on current situations. For
instance, in robotics and smart factories, multiple sensors attached to
different locations and devices facilitate contextual understanding and
intelligent decision-making functions [7]. A multimodal system can adapt
itself to different scenarios based on the user’s settings, providing a
flexible approach that respects individual requirements and privacy concerns.
For example, while cameras and radio devices could be acceptable in a living
room, only radio equipment could be installed in a bedroom in order to keep
the user’s privacy. This adaptability extends to how different modalities
collaborate. To serve specific tasks optimally, such as activity recognition
in indoor monitoring scenarios, modalities can be paired in innovative ways. A
camera, for example, can guide a radar device to the best location for
measuring respiration and heart rate [8]. Similarly, radio-based positioning
data can be utilized as input for a vision-based human detection algorithm
[9]. Through these combined approaches, a multimodal system enhances accuracy,
reliability, and context awareness [7]. It also offers the benefits of being
robust, redundant, and adaptable to various monitoring needs while preserving
user privacy [8].

In this work, we survey the existing multimodal systems for different human
monitoring tasks in indoor scenarios. Indoor human monitoring systems
typically involve various tasks such as activity recognition, gait analysis,
vital signs measurement, user localization, and user identification. Elderly
care, particularly in the context of telecare systems [2], has gained immense
consideration due to the increasing number of senior individuals living
independently [10] [11]. Our research primarily focuses on this domain, while
also shedding light on other applications such as assisting patients at home.
These human monitoring systems not only bolster a sense of safety but also
detect crucial events (e.g., falling and onset of diseases) in order to
facilitate timely intervention. Over time, the continuous monitoring offered
by these systems aids healthcare professionals in observing the progression of
chronic conditions, laying the foundation for tailored treatment plans. In
order to gain insights on how to realize a state-of-the-art elderly monitoring
system, we explore a wide range of sensors and machine learning techniques,
with a special focus on non-contact methods. We followed the Preferred
Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guideline
[12] to search and select relevant studies for this survey. First, we defined
the keywords for searching on Google Scholar, IEEE Xplore, ACM Digital
Library, and Springer Link. The relevent studies should include these
keywords:  1. human monitoring,  2. indoor environments, indoor settings,
indoor scenarios,  3. multimodal,  4. multiple sensors, multi-sensor, and  5.
non contact.  Then, we refined the found articles using the inclusion and
exclusion criteria:

  * •

Inclusion criteria:  1. Articles are peer-reviewed.  2. Articles are related
to human monitoring.  3. Articles utilize multiple sensors to collect and
process multimodal data to implement human monitoring applications, as well as
showing the advantages of multi-sensor systems.  4. Techniques in the articles
can be applied to elderly monitoring systems.

  * •

Exclusion criteria:  1. Articles focus on outdoor applications of multi-sensor
systems and multimodal data (such as autonomous driving).  2. Articles utilize
multiple sensors that are carried by the users or attached to the human
bodies.  3. Articles analyze multimodal data for applications not directly
related to human subjects, such as vehicle or animal tracking.

The earliest related work that we can find is published by Krahnstoever et al.
[13] in 2005, which embedded Radio Frequency Identification (RFID) tags into
objects in order to augment visual data for object tracking and activity
recognition. After collecting the relevant articles, we analyzed them to
identify tasks required in a multi-sensor system for monitoring elderly
people. These comprehensive monitoring systems undertake a series of crucial
tasks to ensure the well-being and safety of elderly residents [11].
Initially, they focus on the precise recognition of elderly users and their
daily activities, forming a baseline understanding of their routines.
Subsequently, these systems take on essential responsibilities such as
conducting gait analysis and monitoring vital signs to track evolving health
conditions, offering timely assistance as needed. In addition to health-
related functions, these systems extend their capabilities to encompass
surveillance tasks, including the creation of 3D models of indoor environments
for precise localization of elderly individuals during suspicious movements,
thus ensuring not only their physical security but also their overall well-
being in the context of independent living. An example of an advanced non-
contact human monitoring system like elderly care can be summarized in terms
of sensors, methods, challenges, and applications as shown in Figure 1. We
organize the survey around the techniques to implement these tasks, which are
essential in the context of elderly care and applicable to related human
monitoring systems. The tasks include activity recognition (see Section II-A),
vital sign measurement (see Section II-B), user identification (see Section
II-C), localization (see Section II-D), and 3D modeling of the environment
(see Section II-E). In addition, we complement these sections with a small set
of articles related to alternative sensing methods, which give context to the
survey.

For developing non-contact human monitoring systems, we focus on the usage of
non-contact sensors such as cameras and radio devices. We also include their
interactions with other complementary sensors such as on-body inertial
measurement units, pressure plates, microphones, and wearable vital sign
monitoring devices. These main sensors, each with its unique strengths and
limitations, form the core of elderly monitoring solutions. They offer
complementary insights and capabilities that, when combined thoughtfully,
create a comprehensive monitoring solution tailored to specific needs of the
users and constraints of the environment. An example of multimodal data
captured with a diverse set of sensors is depicted in Figure 2.

Figure 2: From left to right: RGB camera, Depth camera, 3D Depth camera,
Thermal camera and mmWave radar data.

RGB Cameras: Red-Green-Blue (RGB) cameras are the visual workhorses of elderly
monitoring systems. They excel at capturing rich visual information, including
color, texture, and shape, providing a comprehensive view of the monitored
environment [14]. Their versatility is evident in numerous applications, from
tracking user activities to detecting anomalies. However, they are susceptible
to environmental factors such as varying light conditions and occlusions,
which can impact their performance. Additionally, privacy concerns often arise
due to the intrusive nature of constant visual monitoring. Nevertheless, their
ubiquity and ability to offer real-time insights make them indispensable in
many monitoring scenarios.

Depth Cameras: Depth cameras, utilizing either time-of-flight or stereoscopic
methods, are specialized sensors designed to capture accurate 3D spatial and
shape information. These characteristics make them particularly well-suited
for tasks that demand precise depth perception, such as fall detection and
gesture recognition. One drawback is that they do not capture color or texture
data, features that are sometimes essential for specific applications. To
overcome this limitation, depth cameras are often integrated with RGB cameras
to form RGB-D systems [15]. These hybrid sensors offer a balanced data set
that includes color, texture, and depth, thereby contributing to a more
comprehensive monitoring solution.

Thermal Cameras: Infrared cameras are effective in dark environments, making
them valuable for monitoring when visibility is low. However, they also lack
color and texture information and are sensitive to heat sources. Their use is
ideal for scenarios where visual information is not critical, and the focus is
on temperature-related monitoring. Their cost limits their usage to certain
scenarios. Consequently, there are few public datasets of thermal data and
models trained on these datasets, comparing to those of RGB and RGB-D
modalities.

Radio Devices: Radio-frequency sensors, including radar and WiFi devices, have
gained prominence for their ability to operate in non-line-of-sight scenarios
and their robustness in variable lighting conditions [16]. These sensors have
applications in tracking user movements, detecting vital signs, and even
through-wall sensing. Nevertheless, they share a common limitation with other
non-visual sensors, as they lack color, texture, and shape data. They often
require sophisticated postprocessing techniques to extract interpretable
information, but their unique capabilities make them valuable components in
multimodal elderly monitoring systems.

Complementary Sensors: These sensors collectively form a group that
complements the strengths and weaknesses of visual and radio sensors in
elderly monitoring systems. Inertial measurement units, commonly found in
devices like smartphones and smartwatches, offer cost-effective means of
tracking user movements and activities but lack visual information [5].
Ambient sensors [17], which measure environmental factors like temperature,
humidity, and object usage, provide insights into the living conditions but
offer limited information about individual users. Pressure-sensitive sensors
integrated in surfaces, such as the floor, are versatile, aiding in gait
analysis and fall detection; however, their installation and maintenance can
be complex. Audio sensors, represented by microphones, have numerous
applications and are commonly integrated into devices, making them a practical
choice for audio-related monitoring [18]. Nevertheless, they do not provide
visual information, and privacy concerns regarding voice data are significant.
These complementary sensors can enhance the overall monitoring system by
providing context and supplementary data, making them valuable additions to
not only an elderly monitoring system but also to general human monitoring
solutions.

In our survey, we study the utilization of multimodal approaches for different
tasks and scenarios in indoor non-contact human monitoring systems, focusing
on monitoring elderly individuals [11]. This survey is useful for researchers
aiming to leverage multimodal approaches in developing an elderly monitoring
solution and related systems. We aims to link the sensors and methods to
provide a logical view of the multimodal approaches. Besides offering a deep
dive into current practices, it identifies gaps in the existing literature and
suggests prospective avenues for exploration, especially in the context of
multi-modality.

The contribution of this survey is listed as follows: • We investigate systems
that utilize vision and radio-frequency signals, with the supplement of other
sensors. • We summarize methods of combining vision, radio, and other sensing
modalities in different levels: early, intermediate, and late fusion (or
decision fusion). • We highlight the advantages of integrated vision and radio
devices, in comparison to other systems that leverage only cameras or radio
sensing. • We present challenges and limits of the existing multimodal human
monitoring systems, as well as proposing potential solutions to handle these
issues.

The existing surveys on indoor human monitoring systems concentrate on either
individual modalities or one task, such as vision-based activity recognition
[3] [19], RGB-based and skeloton-based activity recognition [20], action
recognition based on wearable sensors [5], RF-based human behaviour
recognition [16] [4] [21], and vital signs measurement based on WiFi CSI [22].
Among surveys focusing on multimodal approaches, we can found those
investigating activity recognition [14], object detection [23], fusion methods
for multimodal data [24], and classification models [25]. In these surveys,
multimodal approaches have shown their advantages. For example, Fan et al.
[26] utilized a video-based captioning method to improve the radio-based
approach in diary generation. Their combination leveraged an accurate modality
to enhance a privacy-aware sensing device. In vital sign monitoring, a camera
can guide the radar to the sensing region on the human body [8]. These
examples have proved that an optimal integration of multiple modalities can
improve and extend the utility of human sensing systems while mitigating the
risk involving the collection of sensitive data.

##  II Human monitoring tasks on multimodal data

In this section, we identify important tasks that are required in a elderly
monitoring system. They are: activity recognition, vital sign measurement,
identification, localization, and 3D scene modeling. We describe popular
techniques that are utilized to implement these tasks.

###  II-A Activity recognition

In this section, we investigate human activity recognition systems using
cameras, microphones, wearable sensors, and radio devices. The input of an
activity recognition model is a feature vector $v\in\mathbb{R}^{k}$, where $k$
is the number of features, and its output is an activity such as standing,
walking, falling, and lying. The feature vectors are extracted from one sensor
or combined from multiple modalities. Sun et al. [14] provided a comprehensive
survey on human activity recognition from single-modality to multi-modality
perspectives.

TABLE I: Human activity recognition using multi-modal non-contact human assisting systems Articles |  Sensors |  Multimodal approaches |  Objectives
---|---|---|---
Zhao et al. [27] |  sub-GHz WiFi, RGB camera |  radio features with camera supervision |  Human pose estimation in indoor activities such as walking, sitting, taking stairs, waiting for elevators, opening doors, and talking to friends
Zhao et al. [28] |  Force sensor, Kinect camera, and wearable accelerometer |  Correlative memory neural network (CorrMNN) to measure the correlation in bimodal gait data |  Learning gait differences between three neurodegenerative diseases, between patients with different severity levels of Parkinson’s disease, and between healthy individuals and patients
Shao et al. [29] |  4 Kinect cameras (skeleton data and RGB image data) |  Long Short-Term Memory (LSTM) model for skeleton data, two Convolutional Neural Network (CNN) models for RGB videos (front and side perspectives), fusing silhouette features, and decision level fusion |  Detection depression in a dataset of 200 postgraduate students (86 depressive ones)
Zou et al. [30] |  sub-GHz WiFi, RGB camera |  decision-level |  sitting, standing, walking when the light is on or off
Ardianto et al. [31] |  Infrared (IR), RGB-D |  feature fusion |  standing, walking, and running
De et al. [32] |  Infrared, RGB-D |  feature fusion |  NTU RGB+D dataset: 82 daily individual actions, 26 two-person actions, and 12 medical condition actions (neeze/cough, staggering, falling down, headache, chest pain, back pain, neck pain, nausea/vomiting, fan self, yawn, stretch oneself, blow nose)
Krahnstoever et al. [13] |  RFID in objects, RGB camera |  feature fusion |  recognizing human-object interaction: person approaching (an object), taking/examining/reading/replacing an object, and person leaving
Memmesheimer et al. [33] |  WiFi, Camera, inertial sensor, motion capture |  transforming all data to one common representation (2D images) as input for CNN |  Daily activities from NTU RGB+D, UTD-MHAD (camera and IMU), and Simulate (RGBD)
Li et al. [34] |  UWB radar, Wearable inertial sensors |  |  Activity recognition and fall detection
Fan et al. [26] |  Camera, WiFi |  Multimodal feature alignment |  Daily activity recognition and diary, using video-based captioning to improve the performance of the radio-based method.

Under the unimodal approaches, recent human activity recognition (HAR) systems
have proposed training the learning models from a single data source
information such as video, audio, acceleration, and wireless signal. Unimodal
activity recognition using vision or radio waves have been extensively studied
in the literature [35][36][37]. RGB videos were leveraged by Robertson et al.
[35] to perform activity recognition. Nie et al. [36] analyzed the three-
dimensional (3D) scenes by observing human trajectory in the environment. Han
et al. [37] used thermal infrared imaging data and performed human motion
detection independent of the lighting conditions, colors of human surfaces, or
backgrounds. Using radio waves, Li et al. [38] performed action recognition in
smart home systems where cameras are difficult to utilize due to inadequate
lighting, and privacy concerns.

Multi-modal HAR applications utilize the data from multiple modalities such as
RGB, depth, infrared camera, wearable sensors, and skeleton data. Most of the
recently proposed multi-modal HAR applications are applied on the
MSRDailyActivity3D, UTH-MHAD, NTU RGB+D benchmark multimodal datasets and are
broadly classified into co-learning-based and fusion-based methods. Co-
learning methods explore the learned knowledge from auxiliary sensor data and
use them to assist the learning processor of a different multi-modal sensor
data source. Fusion-based methods generally involve multiple combinations of
visual modalities such as RGB - depth - skeleton data, visual and non-visual
modalities such as audio - acceleration - RGB - depth.

Recent works [39, 40, 41, 42] proposed multimodal datasets for HAR using the
modalities from different radio, radar and camera information. Bocus et al.
[40] proposed a HAR multimodal dataset using Wi-Fi network interface card
(NIC) interface, passive WiFi radar (PWR), Ultra-wide band (UWB), vision
camera and infrared sensor information. Gou et al. [41] proposed a human
gesture-based activity recognition by combining Kinect-camera information with
CSI information of the WiFi devices. The proposed work also published a WiAR
dataset. Using the OPERAnet dataset [40], Koupai et al. [43] implemented a
fusion-based vision transformer approach for HAR on PWR radar, CSI data, and
vision data. Similarly, Li et al. [42] combined camera with FMCW radar
information for activity recognition in dark and occlusion conditions. The
authors also released the multimodal dataset. Although the aforementioned
multi-modality fusion-based HAR methods have achieved promising results on
some benchmark datasets, the task of effective modality fusion is still
largely open. Specifically, most of the existing multi-modality methods have
complicated architectures that require high computational costs. In Table I,
most work focused on daily activities of healthy subjects. Some of them aim to
detect falling and health-related movements (e.g., sneezing, coughing, and
staggering) [32] [34]. In an elderly monitoring system, it is useful to
integrate an automatic diary generator [26], which may help medical experts to
assess long-term conditions and treatment effectiveness.

Analysing the walking parameters of a patient provides useful information
about movement disorders and neurodegenerative diseases. It is challenging to
distinguish between normal and abnormal gait features as well as between
different severity levels of diseases. To overcome this problem, Zhao et al.
[44] aggregated data from multiple sensors to learn the differences between
three neurodegenerative diseases, between patients with different severity
levels of Parkinson’s disease, and between healthy individuals and patients.
Although the subjects in [29] were not elderly people, Shao et al. [29]
illustrated the feasibility of detecting depression with Kinect cameras. For
RGB data, they combined silhouette features from the front and the side view
to produce predictions, which were fused at the decision level with the
skeleton-based LSTM outputs. These methods are useful for an elderly
monitoring system, especially for long-term assessment of neurodegenerative
diseases and mental disorders.

Human Activity Recognition (HAR) remains an important area of research in the
domain of elderly care, with an application to multiple other use cases. While
single-modal systems provide a foundational understanding and an array of
capabilities, the evolving landscape of multi-modal systems harnesses the
synergies between different modalities, offering a more comprehensive
perspective of human activities. Recent research [26] [33] have showed that
fusing data from multiple modalities can significantly improve the accuracy
and robustness of activity detection. From the recent literature, it can be
seen that gait analysis has proven crucial in distinguishing health states
using multiple sensors and algorithms. The highlighted studies [28] [29],
although focusing on specific conditions, reveal the potential for broader
applications in elderly monitoring. However, challenges persist, particularly
in seamlessly integrating and processing diverse data streams without
incurring exorbitant computational costs. Future research should aim to
improve sensor integration through the harmonization of data and the use of
unified platforms, and the generalization across diverse populations, by
designing more inclusive data collection plans [45] and delving into
stratified models. Moreover, while a majority of research has been directed at
detecting standard daily activities, there’s an emerging emphasis on
recognizing health-related movements and incidents specific to the elderly,
such as falls [34]. This not only enhances the safety for seniors living
independently but also provides valuable insights to healthcare professionals
for optimizing care regimens. In this context it would be interesting to see
the evolution of HAR systems specifically tailored for specific demographics,
such as the elderly. Future works may also consider the development of
adaptive algorithms, capable of personalizing activity recognition based on
individual behavior patterns (e.g., extracting from the activity diary [26]),
thus paving the way for more personalized and effective elderly care
solutions.

###  II-B Vital sign measurement

TABLE II: Vital signs monitoring Articles |  Modalities |  Scenarios |  Methods
---|---|---|---
He et al. [46] |  Xethru IR-UWB and Kinect, Vernier belt |  respiration rate of 1-2 subjects sitting stationary; classifying eupnea, Cheyne-Stokes respiration, Kussmaul respiration, and apnea |  using the depth camera to detect and localize the subject; then, selecting the radar at the optimal location; random forest classifier
Ren et al. [47] |  Radar and optical camera based techniques |  detecting vital signs such as respiratory rate (RR), heart rate (HR), and blood oxygen saturation in the sitting position, varying illumination |  using UWB stepped-frequency continuous-wave radar and imaging photoplethysmography (iPPG) techniques to measure vital signs
Yang et al. [48] |  FMCW radar and a digital camera |  Heart rate (HR) and breathing rate (BR), pre-exercise  and post-exercise conditions, three subjects |  the vital signs were extracted from the STFT spectrograms with the graph-based segmentation algorithm
Xie et al. [49] |  a pair of co-located UWB and depth sensors, medical devices (Nonin LifeSense II and Masimo Pulse Oximeter) for ground truth |  Heart rate and respiratory rate, sitting, 1 - 3 subjects |  identifying four typical temporal and spectral patterns and a suitable RR/HR estimator for each, possible to monitor 3 subjects separating at only 20 cm
Shokouhmand et al. [8] |  red-green-blue-depth (RGB-D) camera and radar |  respiratory rate (RR) and heart rate (HR), sitting |  The camera estimates the human torso landmarks and a processing unit constantly adapts the radar beams to the direction of the subjects
Yang et al. [50] |  impulse-radio UWB (IR-UWB) radar, optical and depth-sensing camera |  respiration rate, sitting, 1 - 2 subjects |  face detection algorithm based on the RGB video images, using the camera and utilizes the distance information acquired to find the subejcts and search the respiration waveform
Chian et al. [51] |  Two radars and one thermal camera, |  respiration rates and heart rate, 1 - 5 subjects sitting, Vernier respiration belt and Polar H10 as reference senros |  Using a thermal camera to detect the number of people and their locations, identifying the respiration rates and heartbeat rates of multiple people.
Vilesov et al. [52] |  TI AWR1443 FMCW radar and Zed2 RGB camera; Philips MX800 clinical patient monitor for ground-truthing |  heart rate prediction of 91 sitting volunteers (28 light, 49 medium, and 14 dark skin), fairness in terms of similarity of outcome between the light and dark skin tone groups |  Two-agent minimax problem: a fusion network mapping from the unimodal waveforms to the fused multimodal waveform and a discriminator minimizing the mutual information between the estimated waveform and skin-tone attribute

Obtaining physiological data such as respiration rate and heart rate plays a
crucial role in monitoring health and well-being of individuals. In this
section, we explore the non-invasive vital sign measurement techniques that
utilize videos and radio frequency signals.

Unimodal vital-sign monitoring using vision or radio waves has been
extensively studied in the literature [53, 54, 55]. Soto el al. [53] provided
a comprehensive survey of wireless fidelity (WiFi)-based vital-sign monitoring
using channel state information (CSI) information. Selvaraju et al. [54]
conducted a complete review of the recent advances in camera-based techniques
for monitoring of five vital signs including heart rate, respiratory rate,
blood pressure, body skin temperature, and oxygen saturation. Zhang et al.
[55] discussed the overview of three types of electromagnetic radar systems
such as continuous-wave, frequency modulated (FM), and UWB to measure human
physiological signs. These studies demonstrate the potential of one modality
such as camera, WiFi signal, or radar sensor for non-contact vital sign
monitoring in a variety of applications.

However, there are still several challenges to overcome under unimodal vital
sign monitoring approaches. For example, systems based on radio-frequency
signals suffer from signal interference and body motion artifacts. On the
other hand, camera-based systems are less accurate in poor-lighting
conditions. Multimodal approaches can address such limitations and increase
the reliability of non-contact vital sign monitoring [55]. He et al. [46]
combined depth cameras and radars to localize and estimate vital signs for one
or two subjects. The use of a 3D depth camera (Kinect) helps to ensure that
the subjects being monitored are within the coverage area of three
ultrawideband radars. These radars, which use impulse radio technology, are
specifically designed for measuring respiration rate and detecting respiration
patterns. Yang et al. [50] also reported a system for monitoring
simultaneously the respiration of two subjects with a hybrid camera-radar
prototype. The system was built with an impulse-radio ultra wide-band (IR-UWB)
radar module and an optical and depth-sensing camera module. Subject detection
is done by the camera and uses the distance information to guide the signal
processing of the radar. Providing that context to the radar facilitates the
extraction of respiration information. Instead of depth camera, Chian et al.
[51] proposed a system to track and identify the respiration rates and
heartbeat rates of multiple people using a thermal camera and a Doppler radar.
The thermal camera detects the number of people and their movements, and the
doppler radars estimates the respiration and heartbeat. Other studies focused
on the performance of the radar and optical techniques. For example, Ren et
al. measured respiratory rate, heart rate and blood oxygen saturation with a
combination of ultrawided stepped stepped-frequency continuous-wave radar and
a camera for photoplethysmography (iPPG). In the study, they pointed out the
differences of the modalities: The radar can be used for seeing-through walls,
and optical technique is uniquely capable of measuring $SpO_{2}$. Yang et al.
[48] used a FMCW radar and a digital camera to measure the vital signs in
real-life situations. The graph-based method provides HR measurements that are
highly correlated with the golden standard device. Similarly, heart and
respiratory rates can be obtained using a pair of co-located UWB and depth
sensors identifying the spectral patterns [49].The work focuses in the
robustness against harmonics and intermodulation. Likelihood of different
respiration rate and heart rate are studied to detect corrupted signals.
Shokouhmand et al. [8] developed a system where the respiration rate and heart
rate are monitored in real time by radar and a depth camera. While the depth
camera tracks the locations of subjects, the radar estimates their respiratory
and heart rates. A novel method is designed to optimize the regions of
interest for monitoring the respiration rate and heart rate. A combination of
micro Doppler motion effect and photoplethysmography was used by Rong et al.
[56] for cardiac pulse detection. The breath rate and heart rate are detected
micro Doppler motion effect using microwave frequency while the volumetric
change of the heart is measured by photoplethysmography. In the study, they
introduced the concept of terahertz-wave-plethysmography which detected blood
volume changes in the upper dermis tissue. Also implementing the fusion method
for rPPG with a radar and a camera, Vilesov et al. [52] ensured the fairness
in terms of similarity of the estimated heart rates between the light and dark
skin tone groups. They formulated the problem as a two-agent minimax
optimization procedure: a fusion network mapping from the unimodal waveforms
to the fused multimodal waveform and a discriminator minimizing the mutual
information between the estimated waveform and skin-tone attribute.

Two common challenging situations in non-contact vital sign measurement are
body movement and multi-target monitoring. The former can be tackled with
movement reduction or cancellation by radars [57] [58] [59]. The latter can be
mitigated either by radars [60] [61] or by integrating cameras to track the
targets and direct the radars towards the optimal region for measurement [8].

Recent research on non-contact vital sign measurements underscores the
evolution from unimodal to multimodal monitoring approaches. While camera-
based techniques offer depth perception and subject localization, their
performance often diminishes under poor lighting conditions. On the other
hand, radio-frequency methods, though robust in various environmental
conditions, grapple with interference from body motions and other electronic
devices. We can particularly highlight the emerging trend of integrating
visual and RF modalities [8] [51] [46] [52], which serves to counteract the
individual limitations of each method. Such fusion notably enhances the
system’s ability to navigate challenges, especially in situations involving
body movement and multi-target monitoring. In this context, future efforts
might benefit from developing advanced algorithms for seamless integration of
multiple data sources, investigating the impact of environmental factors on
multimodal systems and exploring the potential of newer sensing technologies.

###  II-C User identification

TABLE III: User identification Articles |  Modalities |  Features |  Methods & Results
---|---|---|---
Shi et al. [62] |  mmWave radar, RGB camera |  Fusing features from gait energy images and time-Doppler spectrograms |  121 subjects (72 men and 49 women) with eight views and three walking conditions, experimenting with different fusion methods (voting-based decision fusion, feature concatenation, and feature fusion based on learned weights) to show that concatenating features along the along the space and temporal dimensions achieved the best accurary (up to 95.4%)
Li et al. [63] |  RFID, Kinect |  RFID phase features (velocity, average phase change,…), RSSI features, Kinect skeleton motion features |  ranking correlation between bodies Kinect and tags of the RFID reader, SVM classifier, assigning IDs to individuals within 4 seconds with 96.6% accuracy
Chen et al. [64] |  WiFi, RGB camera |  WiFi CSI and videos, device AoA using phase differences between antennas, LSTM |  vision-assisted calibration and Bayesian inference to improve the device AoA estimation
Cao et al. [65] |  WiFi, RGB camera |  PointNet-based learned features from CSI and RGB |  PointNet with attention mechanism, LSTM encoder-decoder network
Liu et al. [66] |  WiFi, RGB camera |  WiFi Fine Timing Measurements, inertial data, and RGB-D videos |  LSTM as feature extractor to generate motion trajectories, LSTM as feature extractor, Maximum Matching Weighted Bipartite Graphs for matching track and phone IDs
Deng et al. [67] |  WiFi, RGB camera |  CSI, Lightweight Residual Convolution Network for feature learning |  LRCN, LSTM, feature fusion, 94.2% recognition accuracy
Fang et al. [68] |  WiFi, panaromic camera |  WiFi CSI, panaromic camera |  estimating the Angle of Arrival (AoA) of the smartphone, cross-modal trajectory matching to determine the identity of the individuals
Luchetti et al. [69] |  UWB, RGB camera |  Locations |  UWB device and camera to track tags carried by users

This section focuses on techniques and systems that leverage multimodal data
to infer the user identities across all modalities. Users can be identified
using gait analysis, which employ various sensors such as cameras, pressure
sensors, radars, and accelerometers [70] [71]. An identification model maps a
feature vector $v\in\mathbb{R}^{k}$ to an identity. Machine learning methods
can be leveraged to aligning and transforming features of one modality to
another [72].

Due to their ubiquity in indoor environment, WiFi signals are combined with
visual information for user identification. Deng et al. [67] utilized WiFi
signals and videos for human identification. They proposed a two-stream
network architecture based on the Lightweight Residual Convolution Network
(LRCN) to learn features for WiFi and video data. The learned features were
concatenated as inputs to a LSTM model to recognize users with an accuracy of
94.2%. The EyeFi system of Fang et al. [68] integrated a WiFi chipset into a
panaromic overhead camera in order to combine motion trajectories from both
vision and RF modalities for human identification. They trained a neural
network based on a student-teacher model to estimate the Angle of Arrival
(AoA) of WiFi packets from the CSI values.

In some scenarios, one user moves from a camera-allowed region to a camera-
restricted area. This problem requires solutions to find the alignment between
two or more modalities. To address the cross-modal human re-identification
problem, Cao et al. [65] extracted point cloud data from both modalities and
match them. They utilized a PointNet-based model to extract features from CSI
and RGB. Their PointNet architecture integrated the attention mechanism and a
LSTM encoder-decoder network to implement user re-identification across two
modalities. Li et al. [63] extracted RFID phase features (such as velocity and
average phase change), RSSI features, and Kinect-based skeleton features for
data association. Their method ranked the correlation between the body motion
generated by Kinect and RFID tags The system could assign IDs to individuals
with the accuracy of 96.6% accuracy. Liu et al. [66] leveraged WiFi Fine
Timing Measurements (FTM) and inertial measurement unit (IMU) sensor data
captured by the user’s smartphone to associate the user detected on a camera
footage with their corresponding smartphone identifier (e.g. WiFi MAC
address), using a recurrent multi-modal deep neural network. Chen et al. [64]
used a multi-antenna WiFi radio and a camera to monitor users in the area. The
approach processed WiFi CSI data of WiFi packets to extract features on
locations, motion, and activities of users. It fused these features with
visual information to find the best matches of device carriers in videos with
network IDs of the phones. Liu et al. [66] analysed WiFi Fine Timing
Measurements, inertial data, and RGB-D videos to generate motion trajectories,
using a LSTM model. Then, they implemented Maximum Matching Weighted Bipartite
Graphs in order to match visual tracks and phone IDs. Luchetti et al. [69]
leveraged UWB devices and cameras to track tags carried by users.

The integration of multimodal data for user identification showcases the
potential of synergizing diverse modalities, with WiFi signals and visual cues
being at the forefront. Current methodologies predominantly employ machine
learning techniques, from LSTMs to PointNet architectures, to align [63] [66],
transform [64], and extrapolate features [68] from one modality to another,
achieving impressive identification accuracies. However, challenges such as
transitions from camera-enabled to camera-restricted zones, interference, and
real-time processing are still largely unaddressed. Future research should
focus int the seamless fusion of data, addressing privacy concerns, and
adapting these techniques to heterogeneous environments and ensuring
consistent performance across varying conditions.

###  II-D User localization

TABLE IV: User localization Articles |  Modalities |  Features |  Methods & Results
---|---|---|---
Suwannaphong et al. [73] |  Bluetooth, Camera |  BLE RSSI (camera for ground-truth) |  RSSI augmentation is done from multiple BLE devices using camera ground-truths and one-shot learning
Jiang et al. [74] |  Bluetooth, Camera |  BLE 5.1, visual point cloud, and IMU |  locating the BLE transmitter in the camera coordinate, homography-based matching mechanism to obtain the depth information of the target BLE device, 3.4$\deg$ angular error and 8.4cm position error in median, outperforming the state-of-the-art by 48% and 8%, respectively; overall accuracy over 90%
Ishihara et al. [75] |  Bluetooth, Camera |  CNN ouput |  image-based localization by integrating Bluetooth radio-wave signal readings, using a dual-stream CNN
Deng et al. [76] |  Radar, Camera |  mmWave Radar and Camera Fusion |  Object detection on drones
Li et al. [77] |  LiDAR, Radar, Camera |  LiDAR, Radar, and camera for object detection and tracking |  multiple modality combinations, LiDAR: point-wise concantenation, Radar: pillar feature encoder, camera: image feature encoder
Li et al. [78] |  Radar, Camera |  mmWave radar IWR6843ISK-ODS and RGB camera |  distingushing living pedestrian and portrait billboard, a feature fusion network of mmWave radar and computer vision based on attention mechanism, mAP of 97.7%
Papaioannou et al. [79] |  WiFi, Camera |  WiFi-based and Camera-based trajectories |  Using WiFi measurements to merge camera-based tracklets based on tracklet trees
Zhao et al. [80] |  WiFi, Camera |  WiFi-based and Camera-based trajectories |  Using WiFi CSI to aid distance calculation based on inertial data, then combining it with vision-based localization, YOLO V2 for target detection
Cai et al. [81] |  WiFi, Camera |  Trajectories |  Vision-based tracking to extract the object trajectory, radio signals identify anonymous visual tracks and correct errors
Xu et al. [82] |  WiFi, Camera |  Visual data, radio (RSS), and inertial data to extract trajectories |  Augmented particle filters, Trajectory fusion, localization accuracy of 0.7m, outperforming the state-of-the-art systems by ¿70%
Zhang et al. [44] |  UWB, Infrared Camera |  Infrared on-board camera locating by identifying artificial landmarks attached to the ceiling, TOA-based UWB positioning method |  Asymmetric Double-sided Two-way ranging, Extended Kalman filter (EKF) is used to fuse the real-time location information
Nguyen et al. [83] |  UWB, Infrared Camera, IMU |  Trajectories from camera, IMU, and UWB |  Monocular camera, a 6-DoF IMU, and a single UWB anchor for robot localization
Liu et al. [84] |  UWB, Camera |  Trajectories from UWB and monocular camera |  EKF (extended Kalman filter), resolving scale ambiguity and repositioning of the monocular ORB-SLAM, indoor positioning accuracy of 0.2m
Varotto et al. [85] |  Radio receiver, Camera |  Target locations |  Drone with a radio receiver and a camera for detecting a radio-emitting target, recursive Bayesian Estimation scheme that uses camera observations to refine radio measurements
Streubel et al. [86] |  Indoor Pedestrian Tracking |  Camera and Radar (MIMO-FMCW) |  Fusion of stereo camera and radar targets to track pedestrians
Pearce et al. [87] |  Tracking human movement patterns |  mmWave and Camera (as supervision signals) |  Framework for training a mmWave radar model with a camera for labeling the data and supervising the radar model
Lim et al. [88] |  Human detection and tracking |  FMCW Radar, Depth, IMU, and RGB Camera Data |  3D convolution based supervised regression model, Out-of-range pixel classification model
Cai et al. [89] |  Human detection and tracking |  mmWave radar and thermal camera |  Uncertainty-guided fusion framework using Bayesian Neural Network to jointly extract features for for human detection in visual degradation conditions

Within the context of indoor human centered monitoring systems, one of the
essential tasks is indoor localization. An localization system aims to
estimate the user location $x\in\mathbb{R}^{3}$ using a feature vector
$v\in\mathbb{R}^{k}$. User positions and movement trajectories
$t=[x_{1},\dots,x_{i},\dots,x_{n}]$ provide useful measurements on the well-
being of the individuals living at their home.

Multimodal data improves localization in challenging situations such as
cluttered or dark environments in indoor scenarios. For example, in a factory,
surveillance cameras can be obstructed by equipment. Tarkowski et al. [90]
proposed a hybrid localization system combining vision-based, radio-based and
inertial techniques to solve localization issues in difficult and complicated
industrial situations. Similarly, Woznica et al. [91] and Wang et al.[92]
tackled the problem of localization using multimodal analysis of radio and
vision. In a dim lighting museum setting with multiple people moving around
exhibit stands, Papaioannou et al. [79] integrated WiFi measurements to
concatenate visual tracklets into accurate and continuous paths to reduce the
localization error to below 1 meter. Zhao et al. [80] enhances indoor
localization of users with multimodal sensing: camera, wearable IMU sensors,
and ambient CSI data of WiFi signal. They estimated distance and orientation
in indoor environments to achieve sub-meter accuracy. Zhang et al. [44]
introduced a positioning method that combined infrared vision and ultra-
wideband radio device. The former identified landmarks attached to the ceiling
while the latter used an adaptive weight positioning algorithm to improve the
accuracy at the edge of the UWB coverage area. Extended Kalman filter (EKF)
was used to fuse their location information. Xu et al. [82] introduced an
multi-sensor localization system with an accuracy of 0.7m in indoor
environments. The sensors included: cameras, RSS of wireless signals, and
wearable inertial sensors. iVR used an augmented particle filter to combine
multiple unimodal locations. Jiang et al. [74] deployed BLE 5.1, visual point
cloud, and IMU to reduce the positioning error to 3.4$\deg$ angular error and
8.4cm position error in median, which outperformed the state-of-the-art by 48%
and 8%, respectively. Liu et al. [84] utilized UWB and monocular camera to
resolve scale ambiguity and repositioning of the monocular ORB-SLAM. In an
retail environment, Sturari et al. [93] proposed a sensor fusion system
consisting active radio beacons and RGB-D cameras. This system was used for
customer indoor localization and tracking. A mobile application received (with
a frequency of 5Hz) raw data (beacon MAC address, broadcasting power, and
RSSI) to get the device position. On the other hand, the RGB-D cameras
monitored an area to track people and propagate position data. Following the
research direction of combining visual and radio data, other researchers
leverage specially-designed equipment for sensing: radar. Streubel et al. [86]
combined camera and radar features to detect and track indoor pedestrians.
They have taken stereo camera features and multiple input multiple output
(MIMO) frequency modulated continuous wave (FMCW) radar features, and then
used a standard linear Kalman filter based method for tracking the humans, and
have shown how using a heterogenous sensor fusion gives better performance
when compared with the sensors individually. The visual richness of videos can
be exploited to enhance radar systems. Pearce et al. [87] leveraged camera
data to train an mmWave radar system to track and classify human movement
patterns. Although RGB cameras have achieved great performance in human
detection and tracking, they are susceptible to visual degradation. To address
this issue, Cai et al. [89] introduced a uncertainty-guided fusion framework
based on a Bayesian Neural Network to combine thermal videos and mmWave radar
signals for detecting multiple users.

Different modalities can be combined in a parallel manner to improve the
performance of object detection and localization systems. Ishihara et al. [75]
enhanced image-based localization accuracy by integrating Bluetooth radio-wave
signal readings, using a dual-stream CNN. Qiu et al. [9] used the radio
localization and identifier information from the radio signals to assist the
human detection. On the other hand, one modality can be leveraged to increase
the measurement accuracy of the other. For instance, Varotto et al. [85]
detected a radio-emitting target with an aerial robot equipped with a radio
receiver and a camera. They implemented a Recursive Bayesian Estimation scheme
that utilized camera observations to refine radio measurements. Deng et al.
[76] introduced Geryon using camera and mmWave radar on drones for object
detection. The visual information assisted radar to reduce the aggravated
sparsity and noise of point-cloud data. On the other hand, a radar-based
saliency extraction algorithm decreased bandwidth consumption and offloading
latency of camera. Beyond camera and radar combination, Lim et al. [88]
provided an FMCW radar dataset that is synchronized with RGB-D and also
inertial measurement unit (IMU) measurements. It includes indoor scenes of
different rooms and including different numbers of people and stationary
obstructions, and the authors have built models to perform human tracking with
radar and depth information. Compared to traditional RGB-D cameras, they have
shown that using radar results in far lesser noise and better understanding of
varying velocity.

Vision-based tracking offers high localization accuracy but suffers from
identity switching and fragmentation errors when multiple users are in close
to each other or occluded. On the other hand, radio-based tracking is reliable
in object detection and identification but less accurate in localization. Cai
et al. [81] leveraged radio-based identification and localization techniques
to augment anonymous visual tracks with identity information and enhance the
accuracy of vision-based object detection. Nguyen et al. [83] deployed a
monocular camera, a 6-DoF IMU, and a single unknown Ultra-wideband (UWB)
anchor to implement accurate and drift-reduced localization. They processed
the UWB measurements by leveraging the propagated data computed by the visual-
inertial-odometry (VIO) pipeline.

The recent literature underscores the significance of multimodal data in
enhancing indoor localization, particularly in complex scenarios like
cluttered spaces or areas with low light. The synergy of different modalities,
be it Bluetooth and cameras [73] [74], radars and LiDARs [78], thermal camera
and mmWave radar [89], or UWB, camera, and inertial sensors [83], offers a
robust solution to limitations inherent to each individual system. Despite the
recent effort, some challenges still persist. Vision-based systems present
obvious problems in occluded spaces while radio-based system still show a
relatively low accuracy and the combination approaches are tailored according
to modalities and use cases. An emerging and promising research direction,
self-supervised learning, could be exploited by having certain modalities aid
in the annotation of others, such as computer vision models providing labels
for radio-based systems. This could result in larger-scale databases with more
variability on scenarios and modalities, potentially improving the reliability
in dynamically-changing environments.

###  II-E Depth estimation

TABLE V: Depth estimation Articles |  Task |  Modalities |  Methodology
---|---|---|---
Xu et al. [94] |  Depth Estimation for indoor Mapping |  mmWave and Lidar (as supervision signals) |  3D convolution based supervised regression model, Out-of-range pixel classification mode
Lu et al. [95] |  Scene understanding in smoke-filled environments |  Cross-modal supervision of LiDAR to radar data |  Conditional-GAN-based approach, classifying obstacles on the reconstructed grid map (such as door, lift, wall, and glass)
Long et al. [96] |  Obstacle detection in diverse illumination conditions |  RGB-Depth data with mmWave radar data |  Particle filter data fusion to reduce uncertainties and obtain more accurate state estimation
Ding et al. [97] |  3D human mesh estimation |  RGB/IR images and mmWave point clouds |  Combining outputs of PointNet, ShapeNet, and CamNet as inputs to a vertex-based Skinned Multi-Person Linear Model to estimate body shapes

In this section, we discuss the methods and applications which use multimodal
data for depth estimation and related tasks such as 3D modelling and scene
understanding. Depth estimation is useful to enhance the environmental
perception capability of the intelligent machines in monitoring and
surveillance tasks. Several works have utilized multimodal data for
understanding scenes and objects in the 3D setting [94][95][96]. Combining the
discussed multimodal approaches are useful estimating in depth related tasks.
Modalities such as radio along with RGB data help in aiding the perception of
depth which is a crucial step for the efficient understanding of human-
centered 3D indoor environments.

The abilities of radio signals to provide information invariant to color and
lighting condition make it a useful modality, when being combined with RGB
features such as color, shape, and textual information. For instance, Long et
al. [96] proposed a data fusion method based on Particle Filter, which
effectively utilized RGB-Depth data with mmWave radar data to accurately
detect multiple objects with varying distances and orientations, even in
diverse illumination conditions.

Cross-modal supervision is another approach to leverage the strengths of
different modalities. Xu et al. [94] present a depth map regression model for
imaging radar perception by using mmWave radar with LiDAR based supervision.
Their approach produces depth maps that visually match LiDAR ground truth, and
it is shown how their model successfully detects elements like floors and
ceilings that are mostly undetectable and are crucial for indoor mapping
tasks. Some challenge discussed in this work includes a lack of understanding
for glass objects, which the authors believe could be mitigated by performing
better understanding of materials, and also some noisy near-range observations
and due to the complicated reflection path. Apart from complications due to
obstructions and material hindrances, there can also be situations when smoke
and dust obscure vision, rendering camera and laser based devices inoperable.
In a similar combination, to understanding the objects and obstacles in indoor
mapping, Lu et al. [95] proposed a solution, called milliMap, to tackle the
scene understanding challenges in smoke-filled environments. Their
conditional-GAN-based approach handled sparsity and multi-path noise of mmWave
radar signals by combining cross-modal supervision from a LiDAR and the strong
geometric priors of indoor spaces. Along with indoor mapping, their approach
also classified the semantics of the obstacles on their reconstructed grid map
(such as door, lift, wall, and glass) which improved the scene reconstruction
accuracy.

Reconstructing the 3D model of a human body is essential in human monitoring
systems. In this application, multimodal data has showed their advantages
since multiple modalities such as RGB/IR cameras and mmWave radars can
complement each other. For example, Ding et al. [97] fused features from
RGB/IR images and mmWave radar point cloud to estimate a 3D human body mesh.

The potential of multimodal data sources, focusing on RGB, radar, and other
radio devices, is still largely unexplored in the field of depth estimation
[94], 3D modeling [97], and scene understanding [95] [96]. The combination of
RGB color and the depth awareness of radar and radio signals could potentially
offer a solution to conventional challenges such as obstructions, occlusions,
or variable illumination [96]. However, while significant progress has been
made in indoor mapping, some challenges persist. These include occlusions by
walls, difficulties in understanding glass objects, noise in near-range
observations, and complications from partially transparent substances such as
smoke and dust [94][95]. There is an immense potential in analysis of multi-
path propagation effects of radio-waves, which could be used by enhanced
fusion techniques that complement cameras and dealing with ambiguous monocular
information.

##  III Multimodal methods

(a) Combination

(b) Transformation

(c) Collaboration

Figure 3: Categorization of multimodal approaches

Within the ambit of multimodal monitoring systems, we present a distinct
categorization of approaches, as illustrated in Figure 3. The categories
include:  1. the combination of various sensing modalities to yield a unified
output,  2. the transformation of one modality into another, and  3. the
mutual collaboration between different modalities, where each one aids in
refining the other.  Historical reviews and surveys [24][98] predominantly
shine light on the first two approaches—combining and transforming multimodal
data. The essence behind these techniques is to distill an optimal
representation from the data conglomerate. Diverging slightly, we delve deeper
into the notion of collaboration. In this paradigm, one modality’s data can
influence and optimize another’s sensor operations, thereby enhancing the
system’s overall performance. This intriguing interplay among modalities,
especially collaboration, will be further elaborated upon in the subsequent
sections.

###  III-A Multimodal fusion

Multimodal fusion methods combine visual data and radio signals to improve the
performance of machine learning models (see Figure 4) in terms of accuracy and
reliability. For example, Qiu et al. [9] performed a radio position
information-aided human detection to enhance the reliability and accuracy of
human camera-based detection. Krahnstoever et al. [13] augmented vision-based
human motion tracking with radio frequency identification (RFID) technology.
In a similar line of work, Sturari et al. [99] developed a sensor fusion
system of active radio beacons and RGB-D cameras for investigating shopper
movements and behavior in retail environments. Stotko et al.[100] used RGB-D
and IR for reconstructing geometry and the spatially varying surface albedo of
a scene. Similarly, Muaaz et al. [101] perform human activity recognition by
combining Wi-Fi (CSI) with wearable inertial measurement unit (IMU) in Android
phones. Qin et al. [102] fused time series data from different wearable
sensors to recognize human activity with a residual network. In a attempt to
generalize action recognition across various modalities, Memmesheimer et al.
[33] experimented with four datasets containing CSI informatoin from WiFi
devices, skeleton information from camera and inertial, and body movements
from a motion capture system.

Figure 4: Combination methods: (a) Early Fusion (b) Intermediate Fusion (c)
Late Fusion

The literature has started to explore the diverse fusion techniques that
integrate vision and wireless communication technologies for a more accurate
prediction of activities and scenes [9]. By combining various sensing
modalities, such as infrared images, radio position data, and RGB videos, we
can achieve more robust results, especially in challenging settings such as
low-light conditions. The intersection of tools, from RGB-D cameras to Wi-Fi
CSI and wearable sensors, can enrich our understanding of multimodal fusion.
However, there are still challenges in seamlessly merging data from different
sources while maintaining consistent performance in different situations.
Future research should focus on refining fusion methods, enhancing the
richness of datasets, and creating systems that can adjust and learn in real-
time to better capture and interpret dynamic indoor environments and human
actions.

###  III-B Transformation

When working with information from various sources, one task is to transform
or map data collected with one sensing modality to that of others. This is
essential in a multi-sensor elderly monitoring system since we need to match
feature vectors between different modalities. For example, it is possible to
install a camera in a living room but that would violate the user in a
bedroom. Instead, a non-vision modality such as a radar should be deployed. A
transformation technique is useful to map visual data to radar data in order
to identify one user moving between these two rooms.

####  III-B1 Non-learning transformation

Non-learning approaches rely on domain knowledge to map data collected by
different sensor types (i.e. different modalities) to the same representation.
Different types of sensors can capture data in the same format. For example,
point cloud can be extracted from both radio frequency signals and videos. Cao
et al. [65] utilized PointNet-based models to extract point cloud data from
WiFi CSI and RGB features. They matched the data points in both modalities for
user re-identification. Multivariate signals can be plotted, saved as images,
and processed by convolutional neural networks (CNNs). Memmesheimer et al.
[33] transformed inertial data, and WiFi CSI into 2D images as input to a CNN-
based classifier. Another solution is to select one modality as the anchor
space which other modalities project their data into. Dimitrievski et al.
[103] first used the Fast-CNN vision detector to generate a set of bounding
boxes around detected objects. Then, they projected the boxes into the range-
azimuth space of the radar data. Similarly, Cui et al. [104] generated
bounding boxes with YOLOv3 on the image plane. After that, they projected the
detections from the pixel coordinate to the radar coordinate. The non-learning
techniques rely on domain knowledge and they need to be customized for
different sets of sensors.

####  III-B2 Learning-based transformation

Learning techniques rely on neural networks such as encoder-decoder to learn
the common representation of different modalities. Autoencoder is well-known
for unsupervised representations learning. It contains two component: one is
an encoder and the other is a decoder. The encoder transform the input feature
vector into a compressed representation (i.e., latent representation). The
decoder reconstructs the input from this latent representation in such a way
that the reconstruction error is minimized. Multimodal user identification
involves the transformation of data from multiple sources to a common form.
Cao et al. [105] introduced ViTag in order to link a sequence of visual
bounding boxes with inertial data and Wi-Fi Fine Time Measurements (FTM) from
smartphones. The system used a multimodal LSTM encoder-decoder network to
perform cross-modal translation that reconstructed IMU and FTM readings from
camera-based bounding boxes. Then, the transformed output was matched with the
observed data. Piechocki et al. [106] learned a common latent representation
of videos and WiFi signals for activity recognition. Especially, their
generative model can reconstruct data of a missing modality. These
transformation models can be considered as a universal solution for any
sensors but they require a significant amount of data for training.

In the exploration of the literature on data transformation within multi-
sensor systems, two distinct strategies emerged: non-learning and learning-
based techniques. The non-learning techniques rely mostly on domain knowledge,
converting sensor data into shared representations by leveraging inherent
similarities between modalities or designating one modality as the anchor
space. While effective, they demand customization for different sensor
combinations, posing a challenge to scalability. Conversely, learning-based
transformations propose the utilization of neural network architectures, such
as autoencoders, to learn shared-modality representations. While they offer
more universal applicability, their dependence on vast training datasets
remains a limitation. Moving forward, the research community could focus on
creating hybrid transformation models that merge domain knowledge with
adaptive learning to ensure robust, scalable, and efficient data
transformation in dynamic multi-sensor environments.

###  III-C Collaboration

In recent years, the concept of modality collaboration has emerged as a
possible approach to enhance the performance and reliability of multi-sensor
systems. By leveraging the strengths of individual sensing modalities,
researchers aim to address their inherent limitations, using collaborative
schemes with other sensors. This collaboration manifests in two primary forms:
co-learning and cross-modal guidance, both of which exploit synergies between
modalities to deliver enhanced outcomes in a variety of applications. Co-
learning is about the transfer of learned knowledge between modalities. In
cross-modal guidance, one modality can control another in order to enhance the
performance of the whole multimodal system.

####  III-C1 Co-learning

Co-learning through multiple modalities can be useful to perform robust and
reliable action recognition in non-contact human assisting systems. For
example, using vision alongside radio waves as the prior knowledge can be
helpful to overcome the shortcomings of radio signals. Similarly, radio signal
data can be useful to assist vision-based sensors and perform reliable
activity recognition under low illumination, dark, and occlusion conditions.
Zhao et al. [27] estimated human pose through walls and occlusions using
cross-modal teacher-student network that transferred the visual knowledge of
human pose from the video-based teacher network and supervised the RF-based
student network to predict keypoint confidence maps. Bocus et al. [39]
proposed a ultra wideband dataset (UWB) for human activity recognition (HAR)
built with COTS RF device input data and camera ground truth. They used
commercial off-the-shelf UWB-equipment to generate the dataset against camera
ground truths. Human pose-estimation using radio frequency signals with the
help of camera ground-truth have been studied by several research groups
[107][108][109]. Zou et al. [30] introduces WiVi which identifies common human
activities in an accurate and device-free manner using commercial WiFi-enabled
IoT devices and camera, and by performing multimodal fusion at the decision
level to combine the strength of these modalities. Fei et al. [110], proposed
a WiFi-based human perception or detection with camera-based annotations.

Co-learning methodologies have paved the way for enhanced action recognition
in non-contact human assisting systems. The most typical approaches have
focused on integrating visual cues with radio wave insights. However,
challenges such as efficient multi-modal data integration and the optimization
of cross-modal translation techniques still remain, while adaptive fusion
strategies could be a reasonable future research direction.

####  III-C2 Cross-modal supervision

The concept of cross-modal supervision stands out as a method where
researchers leverage the unique strengths of one modality to enhance the
performance of other modalities in a task.

Considering the challenge of labelling mmWave radar data, Pearce et al. [87]
have proposed a framework to train a mmWave radar with a camera that solves
the problem and helps in annotating the radar data. The work included
recordings in indoor scenario, and uses camera and radar in a teacher-student
fashion, which they show can be useful in tracking and classifying human
movement patterns. Shokouhmand et al. [8] introduced a camera-guided radar to
monitor vital signs. An RGB-D camera detected the human torso landmarks which
was used to steer the radar beams to the direction of the subjects. In
multimodal localization, Qiu et al. [9] utilized a radio-based positioning
technique to improve the reliability and accuracy of human camera-based
detection. They leveraged the region proposals generated from radio
localization to suppress the false detections and reduce miss detections. On
the other hand, He et al. [46][111] implemented an hybrid approach that used
both modalities for localization and leveraged visual data to select the
optimal radar for vital sign measurement. Another approach is to use a well-
established model to train another model (on another modality). Zhao et al.
[27] designed a cross-modal teacher-student network which transferred the
visual knowledge of human pose to radio-frequency signals. Two or more
modalities can supplement each other in various scenarios. Human assisting
systems that integrate cameras and radio devices have showed their
effectiveness in various application domains. Radio waves aid visual data in
scenarios that hinder the operations of cameras, e.g. occluded or private
areas [27]. One of the important challenges in tracking humans in indoor
settings is occlusion of the subjects by walls. Using only camera device
identifying the movement patterns and pose becomes difficult in such
scenarios. The ability of radar to pass through walls help in tackling the
issue. Song et al. [112] developed a method to reconstruct 3D pose of human
subjects who are hidden. Combining a UWB MIMO radar and a 3D pose
reconstruction model based on camera images to generate labels, they
successfully predicted the 3D pose of hidden human targets. Similarly, Zhao et
al. [27] estimated wall-obstructed humans using WiFi signals. Using RF and RGB
data, they have built a teacher-student network and performed cross-modal
supervision for accurate estimation of 2D human pose. On the other hand,
cameras can support radio devices in cases that reduce the performance of the
latter, such as multi-users scenarios [49] and body movement [113], predicting
the optimal beam such that the received signal power is maximized [114]. The
combination is beneficial since these two modalities can complement each
other.

From camera-augmented radar to radar-guided camera approaches, researchers are
demonstrating the viability of cross modal collaborative systems in addressing
inherent modality limitations. However, challenges persist in optimizing these
architectures, especially in ensuring real-time modality synchronization and
seamless data fusion. Future research should explore dynamic modality
collaboration mechanisms [115], that can adapt to unsynchronized data streams,
and the investigation of adaptive modality weighting based on environmental
contexts.

##  IV Discussion

In this section, we discuss shortcomings of existing multimodal datasets.
Then, we address recent trends of machine learning methods and issues of
multimodal data.

###  IV-A Public datasets of visual and non-visual modalities

We select several existing datasets that include visual (e.g. RGB videos and
depth videos) and non-visual modalities (e.g., RF signals and inertial data).
These datasets are described succintly in Table VI. Other surveys have been
considered different combinations of modalities, such as RGB and depth data
[20].

TABLE VI: Multimodal datasets and state-of-the-art results Datasets |  Modalities |  Description
---|---|---
Berkeley MHAD [116] |  12 RGB cameras, two Microsoft kinect cameras, six wearable acceleration sensors, and four microphones |  11 actions performed by 12 subjects (seven males and five females)
UTD-MHAD [117] |  Kinect camera and one wearable inertial sensor |  Eight subjects (four males and four females) performed 27 actions
UR Fall Detection [118] |  Kinect cameras on the sideview and one inertial measurement unit was worn near the pelvis |  Five persons, ADLs (walking, sitting, crouching down, leaning down/picking up objects from the floor, and lying on a settee), three types of falls (forward, backward, and lateral)
CZU-MHAD [119] |  RGBD camera (Kinect), skeleton data, 10 wearable sensors |  Five subjects, No description on annotation
Opportunity++ [17] |  Sideview and topview cameras, inertial motion capture system, an UWB localization system, motion and switch sensors in objects |  Early-morning routine of four subjects: preparing a coffee and sandwich, then having the breakfast, and finally cleaning the kitchen by putting food and utensils in place.
OPERAnet [40] |  radio frequency and vision-based sensors |  8 hours, 2 rooms, 6 participants, 6 activities, namely, sitting down on a chair, standing from sit, lying down on the ground, standing from the floor, walking and body rotating, UWB-anchor-based positioning, activities: labels captured when activities were performed (following the script or manual annotation)
Alkhateeb [120] |  radar, mmWave radio, camera, 3D LiDAR and gps receiver |  The dataset is a collection of scenarios with each scenario containing multi-modal sensing and communication data. The collected scenarios are for applications such as beam prediction, user identification, object localization, object detection, and blockage identification
Chen et al. [121] |  4D imaging radar, RGBD camera |  20 volunteers in 6 different environments. The mmWave radar can achieve better body reconstruction accuracy than the RGB camera but worse than the depth camera
Topham et al. [122] |  Two digital cameras and a wearable digital goniometer |  Casual walks from 64 participants, in both indoor and outdoor real-world environments. Annotated with Human Pose Estimation using OpenPose 2D HPE23 system (75 anatomical keypoints); then, all authors manually performed a visual check.

The datasets presented exhibit a consistent trend towards integrating both
visual modalities, such as RGB and depth videos, and non-visual modalities
like RF signals, inertial data, and wearables. A common pattern is the use of
Kinect cameras, which offer depth information, coupled with wearables or other
non-visual sensors. Subjects across datasets typically perform a range of
everyday activities, ensuring diverse motion patterns for robust model
training. However, the number of subjects, actions, and the specificity of
annotations vary. Notably absent is the integration of newer sensing
technologies and modalities not yet common in current datasets. For future
datasets, there is a potential emphasis on more complex, real-world scenarios,
a broader array of sensing technologies, and more detailed, possibly
automated, annotation methodologies.

####  IV-A1 Synchronization

Synchronizing multimodal data streams is a cornerstone task in the successful
combination and subsequent analysis of heterogeneous sensor data. Misalignment
or asynchrony can introduce significant errors and adversely affect downstream
processing and insights. We investigate methods to synchronize and align
signals captured by different sensors at various sampling rates. Capturing
multiple modalities on the same machine ensures shared timestamps, providing a
direct way to reduce asynchrony [121]. Another prevalent strategy is clock
configuration to a unified time server [40][121]. For instance, Bocuset
al.[40] employed a local Network Time Protocol (NTP) server to achieve
synchronization across modalities. In practical scenarios like activity
recognition, actions that create distinguishable events in the data can serve
as markers for the onset and conclusion of data capture sessions, such as a
jump. Manual checks can identify misalignments between modalities, which are
then adjusted accordingly [121].

The effective synchronization of multimodal data is imperative for meaningful
analysis and application. Future efforts may focus on more automated and
robust methods that can handle a wide variety of sensors, environmental
conditions, and real-world scenarios.

####  IV-A2 Annotation

The process of annotation, a fundamental step in curating and preparing data
for various tasks, especially in machine learning and computer vision, often
hinges on human intervention. This human-centric nature can inadvertently
introduce errors. Addressing such inaccuracies becomes paramount to ensuring
reliable and effective datasets. Engaging multiple annotators to review the
same dataset can diminish the impact of individual biases and reduce the
potential for mistakes. In the context of activity labeling, Bocus et al. [40]
implemented an application to issue verbal instruction to the subjects and
recorded the timestamps. As a backup method, an observer monitored the
subjects and input the start and stop moments of the activities. One challenge
when dealing with multiple data sources is to separate and associate data. For
example, a system with camera and radio devices can monitor several users
simultaneously. The views of both modalities contain multiple targets. Signal
processing and machine learning techniques are required to isolate the targets
and correspond data from two modalities to the right target. Hence, the
accuracy of the annotation depends on these techniques. Li et al. [38] relied
on AlphaPose to annotate the videos, then used them to train the WiFi-based
pose estimation model. Topham et al. [122] utilized the OpenPose system for
pose annotatation and later manually performed a visual verification. In vital
sign measurement, the ground-truth data are obtained from a single wearable
device [51] [46] [49].

While technological advancements are steadily improving annotation techniques,
the fusion of automation with human oversight continues to be a vital
combination. Future research could further explore advanced algorithms and
frameworks to streamline this process, by providing better pre-annotations
[87] [46], or limiting the needed human intervention with active learning
schemes [123].

####  IV-A3 Participant recruitment

A common issue of the existing datasets is that the ratio of female and males
subjects is not balanced. For example, Berkeley MHAD [116] contains data of
seven males and five females while CZU-MHAD [119] only has data of male
participants. Future datasets should focus on inclusiveness, ensuring
representativeness and fairness of all possible user groups [45] [52].

###  IV-B Multimodal architectures based on transformers

Transformer models [124] have the capability of representing diverse relations
between inputs from one modality or multiple modalities (e.g., text, images,
audio, and radio signals) [125]. We can define a cross-modal transformer of
input sequences (tokens) from two modalities: images $\mathbf{X}_{1}$ and
radio signals $\mathbf{X}_{2}$. We add positional encodings (PE) to the
embeddings to capture the order of tokens in each sequence:
$\mathbf{Z}_{1}=PE(\mathbf{X}_{1})$ and $\mathbf{Z}_{2}=PE(\mathbf{X}_{2})$.
Within each modality, we apply a self-attention (SA) mechanism to capture
relationships between tokens:
$SA(\mathbf{Z})=SA(\mathbf{Q},\mathbf{K},\mathbf{V})$, where
$\mathbf{Q}=\mathbf{Z}\mathbf{W}^{Q}$, $\mathbf{K}=\mathbf{Z}\mathbf{W}^{K}$,
and $\mathbf{V}=\mathbf{Z}\mathbf{W}^{V}$; $\mathbf{W}^{Q}$, $\mathbf{W}^{K}$,
and $\mathbf{W}^{V}$ are the three projection matrices. Depending on the
applications, we can implement masked self-attention mechanisms to leverage
additional knowledge in the transformer models [125]. Multiple self-attention
layers can be combined to form a multi-head self-attention (MHSA) structure.
Then, the output of the multi-head self-attention layer goes through a feed-
forward network (FFN). To allow information flow between modalities, we use a
cross-modal attention mechanism:
$\mathbf{Z}_{1}=MHSA(\mathbf{Q}_{2},\mathbf{K}_{1},\mathbf{V}_{1})$ and
$\mathbf{Z}_{2}=MHSA(\mathbf{Q}_{1},\mathbf{K}_{2},\mathbf{V}_{2})$. For other
cross-modal attention mechanisms, Xu et al. [125] provided technical details
and comparison between them.

Cross-modal transformers on multimodal data create the possibility to
implement self-supervised learning based on cross-modal supervision methods
[87] [46] (see Section III-C). For example, in activity recognition, state-of-
the-art vision-based models, which are publicly available, can provide labels
(i.e. self-supervised signals) to train models based on RF signals. In
deployment, the system uses mostly radio devices for classifying users’
activities to ensure efficiency and privacy [126]. Videos captured by the
camera are only investigated in emergency cases, where visual details are
required to clarify the current situation.

Comparing to activity recognition, vital signs measurement, user
identification, and user localization, there have been fewer articles [96]
[94] [95] on combining cameras and radio devices for 3D scene understanding in
indoor settings (see Section II-E). Through adapting 3D scene understanding
techniques for autonomous driving in outdoor environments [123] [127] [126],
we can integrate characteristics of RF signal propagation and visual data to
infer the 3D representation of an environment more accurately. For example,
such task can be implemented using a spatio-contextual fusion transformer
[126], which leveraged cross-attention layers to complement spatial and
contextual information of videos and radio signals. In addition, other
modalities such as audio can be integrated to enhance the 3D scene
understanding task [128].

###  IV-C Explainability and security

An important aspect for healthcare applications is the explainability of the
models. Nourani et al. [129] discussed various challanges in Explainable AI,
and their system outputs human understandable explanations for activity
recognition tasks. Uddin et al. [130] performed body sensor-based activity
recognition system using Neural Structured Learning (NSL), and and then uses
Local Interpretable Model-Agnostic Explanations (LIME) to explain the model’s
decisions. Schmidt et al. [131] perform scene context change detection and
classification based on RF signals with expert driven neural explainability.
Multimodal explainable approaches ensure that models are not only accurate but
also understandable and transparent They become crucial for increasing trust
and keeping ethical standards, which are especially important in healthcare
systems.

Striking the right balance between efficient monitoring and safeguarding
personal data is a critical task in every system. Figure 5 provides a visual
comparison of various systems, illustrating the trade-offs involved. In the
activity recognition task, this aspect of multimodal data was analyzed by Sun
et al. [14]. We should develop a guideline to handle the security and privacy
issues of multiple modalities, as well as balancing their utility and risk
trade-off.

Figure 5: Comparison of wearable, audio-based, vision-based, and radio-based
systems

Another risk of using multiple modalities is that one compromised sensor may
reveal sensitive data collected by other sensors. This can be implemented with
the same techniques to deal with partial failure of sensors in a multimodal
system. For instance, Li et al. [77] integrated LiDAR, millimeter-wave radar,
and Camera for 3D object detection and tracking, in some scenarios of missing
modalities. Lee et al. [132] compensated corrupted data of RGB cameras, depth
cameras, and force sensors in robotics.

Despite recent advances, ensuring security and maintaining privacy remain
pertinent research topics [14], especially focusing on cases when the
malfunction or compromise of one modality [77] [132] can cascade
vulnerabilities throughout the whole sensing and monitoring system.

###  IV-D Integrated sensing and communication

The recent integrated sensing and communication (ISAC) paradigm [133] aims to
leverage the sensing ability of communication devices to collect information
of environments and users. An ISAC system results from a coordinated or joint
design of sensing and communication functionalities to achieve low-cost
operation, efficient use of limited spectrum, compact-size devices, additional
sensing services for wireless networks, and overall improvement of system
performance. ISAC systems offer a wide range of applications, including
activity recognition [134], localization and tracking [135], vital signs
monitoring [136], and environment reconstruction [137]. The paradigm
facilitates the design of a multimodal elderly monitoring system that unifies
efficient communication and accurate sensing among diverse sensors.

##  V Conclusion

Throughout this survey, we delved into the multifaceted world of multimodal
approaches that play a pivotal role in enhancing human monitoring systems. Our
primary focus was on the combination of cameras and radio devices, showcasing
how they work hand-in-hand to enhance each other’s strengths. Their
collaboration stands out especially in elderly monitoring systems where non-
contact methods are paramount for both efficiency and user comfort [11]. We
analyzed the literature in each one of the sub-tasks and components of a truly
multimodal monitoring system, summarizing the recent research, enumerating the
remaining challenges and devising possible lines for future research. When
analyzing the available resources, we also pointed out the shortcomings in
existing datasets, underscoring the need for more comprehensive and versatile
data collections. In addition, we discussed potential topics of interest that
can be seen throughout all technical implementations. From our analysis, it
can be seen that working with multimodal data presents its own set of
challenges, but the potential benefits for improving human monitoring systems
are well worth the effort.

## ACKNOWLEDGEMENTS

This research has been supported by the Academy of Finland 6G Flagship program
under Grant 346208 and PROFI5 HiDyn program under Grant 32629, and by the
InSecTT project under the European ECSEL Joint Undertaking (JU) program grant
No 876038.

## References

  * [1] World Health Organization, “Progress report on the United Nations decade of healthy ageing, 2021-2023,” in _UN Decade of Healthy Ageing_ , 2023.
  * [2] H. Gokalp and M. Clarke, “Monitoring activities of daily living of the elderly and the potential for its use in telecare and telehealth: a review,” _TELEMEDICINE and e-HEALTH_ , 2013.
  * [3] J. K. Aggarwal and L. Xia, “Human activity recognition from 3d data: A review,” _Pattern Recognition Letters_ , 2014.
  * [4] J. M. Fernandes, J. S. Silva, A. Rodrigues, and F. Boavida, “A survey of approaches to unobtrusive sensing of humans,” _ACM Computing Surveys_ , 2022\.
  * [5] J. Wang, Y. Chen, S. Hao, X. Peng, and L. Hu, “Deep learning for sensor-based activity recognition: A survey,” _Pattern recognition letters_ , vol. 119, pp. 3–11, 2019.
  * [6] A. Haque, A. Milstein, and L. Fei-Fei, “Illuminating the dark spaces of healthcare with ambient intelligence,” _Nature_ , 2020.
  * [7] X. Li, Q. Yu, B. Alzahrani, A. Barnawi, A. Alhindi, D. Alghazzawi, and Y. Miao, “Data fusion for intelligent crowd monitoring and management systems: A survey,” _IEEE Access_ , 2021.
  * [8] A. Shokouhmand, S. Eckstrom, B. Gholami, and N. Tavassolian, “Camera-augmented non-contact vital sign monitoring in real time,” _IEEE Sensors Journal_ , 2022.
  * [9] C. Qiu, D. Zhang, Y. Hu, H. Li, Q. Sun, and Y. Chen, “Radio-assisted human detection,” _IEEE Transactions on Multimedia_ , 2022.
  * [10] M. Plöthner, K. Schmidt, L. de Jong, J. Zeidler, and K. Damm, “Needs and preferences of informal caregivers regarding outpatient care for the elderly: a systematic literature review.” _BMC Geriatrics_ , 2019.
  * [11] V. Stanford, “Using pervasive computing to deliver elder care,” _IEEE Pervasive computing_ , 2002.
  * [12] M. J. Page, J. E. McKenzie, P. M. Bossuyt, I. Boutron, T. C. Hoffmann, C. D. Mulrow, L. Shamseer, J. M. Tetzlaff, E. A. Akl, S. E. Brennan _et al._ , “The prisma 2020 statement: an updated guideline for reporting systematic reviews,” _International Journal of Surgery_ , 2021.
  * [13] N. Krahnstoever, J. Rittscher, P. Tu, K. Chean, and T. Tomlinson, “Activity recognition using visual tracking and rfid,” in _2005 Seventh IEEE Workshops on Applications of Computer Vision (WACV/MOTION’05)-Volume 1_ , vol. 1. IEEE, 2005, pp. 494–500.
  * [14] Z. Sun, Q. Ke, H. Rahmani, M. Bennamoun, G. Wang, and J. Liu, “Human action recognition from various data modalities: A review,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2022.
  * [15] J. Li, W. Gao, Y. Wu, Y. Liu, and Y. Shen, “High-quality indoor scene 3d reconstruction with rgb-d cameras: A brief review,” _Computational Visual Media_ , vol. 8, pp. 1–25, 03 2022.
  * [16] S. Yousefi, H. Narui, S. Dayal, S. Ermon, and S. Valaee, “A survey on behavior recognition using wifi channel state information,” _IEEE Communications Magazine_ , vol. 55, no. 10, pp. 98–104, 2017.
  * [17] M. Ciliberto, V. Fortes Rey, A. Calatroni, P. Lukowicz, and D. Roggen, “Opportunity++: A multimodal dataset for video-and wearable, object and ambient sensors-based human activity recognition,” _Frontiers in Computer Science_ , 2021.
  * [18] P. P. Morita, K. S. Sahu, and A. Oetomo, “Health monitoring using smart home technologies: Scoping review,” _JMIR mHealth and uHealth_ , 2023.
  * [19] M. Yuan, S. Wei, J. Zhao, and M. Sun, “A systematic survey on human behavior recognition methods,” _SN Computer Science_ , vol. 3, no. 1, pp. 1–25, 2022\.
  * [20] C. Wang and J. Yan, “A comprehensive survey of rgb-based and skeleton-based human action recognition,” _IEEE Access_ , 2023.
  * [21] S. Wang and G. Zhou, “A review on radio based activity recognition,” _Digital Communications and Networks_ , 2015.
  * [22] J. C. Soto, I. Galdino, E. Caballero, V. Ferreira, D. Muchaluat-Saade, and C. Albuquerque, “A survey on vital signs monitoring based on wi-fi csi data,” _Computer Communications_ , 2022.
  * [23] X. Tang, Z. Zhang, and Y. Qin, “On-road object detection and tracking based on radar and vision fusion: A review,” _IEEE Intelligent Transportation Systems Magazine_ , 2022.
  * [24] T. Baltrušaitis, C. Ahuja, and L.-P. Morency, “Multimodal machine learning: A survey and taxonomy,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2018.
  * [25] W. C. Sleeman, R. Kapoor, and P. Ghosh, “Multimodal classification: Current landscape, taxonomy and future directions,” _ACM Computing Surveys_ , 2022\.
  * [26] L. Fan, T. Li, Y. Yuan, and D. Katabi, “In-home daily-life captioning using radio signals,” in _European Conference on Computer Vision_ , 2020.
  * [27] M. Zhao, T. Li, M. Abu Alsheikh, Y. Tian, H. Zhao, A. Torralba, and D. Katabi, “Through-wall human pose estimation using radio signals,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 7356–7365.
  * [28] A. Zhao, J. Li, J. Dong, L. Qi, Q. Zhang, N. Li, X. Wang, and H. Zhou, “Multimodal gait recognition for neurodegenerative diseases,” _IEEE transactions on cybernetics_ , 2021.
  * [29] W. Shao, Z. You, L. Liang, X. Hu, C. Li, W. Wang, and B. Hu, “A multi-modal gait analysis-based detection system of the risk of depression,” _IEEE Journal of Biomedical and Health Informatics_ , 2022.
  * [30] H. Zou, J. Yang, H. Prasanna Das, H. Liu, Y. Zhou, and C. J. Spanos, “Wifi and vision multimodal learning for accurate and robust device-free human activity recognition,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops_ , 2019, pp. 0–0.
  * [31] S. Ardianto and H.-M. Hang, “Multi-view and multi-modal action recognition with learned fusion,” in _2018 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)_. IEEE, 2018, pp. 1601–1604.
  * [32] A. M. De Boissiere and R. Noumeir, “Infrared and 3d skeleton feature fusion for rgb-d action recognition,” _IEEE Access_ , vol. 8, pp. 168 297–168 308, 2020.
  * [33] R. Memmesheimer, N. Theisen, and D. Paulus, “Gimme signals: Discriminative signal encoding for multimodal activity recognition,” in _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2020, pp. 10 394–10 401.
  * [34] H. Li, A. Shrestha, F. Fioranelli, J. Le Kernec, and H. Heidari, “Hierarchical classification on multimodal sensing for human activity recogintion and fall detection,” in _2018 IEEE SENSORS_ , 2018.
  * [35] N. Robertson and I. Reid, “A general method for human activity recognition in video,” _Computer Vision and Image Understanding_ , vol. 104, no. 2-3, pp. 232–248, 2006.
  * [36] Y. Nie, A. Dai, X. Han, and M. Nießner, “Pose2room: Understanding 3d scenes from human activities,” _arXiv preprint arXiv:2112.03030_ , 2021.
  * [37] J. Han and B. Bhanu, “Human activity recognition in thermal infrared imagery,” in _2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05)-Workshops_. IEEE, 2005, pp. 17–17.
  * [38] T. Li, L. Fan, M. Zhao, Y. Liu, and D. Katabi, “Making the invisible visible: Action recognition through walls and occlusions,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 872–881.
  * [39] M. J. Bocus and R. Piechocki, “A comprehensive ultra-wideband dataset for non-cooperative contextual sensing,” _Scientific Data_ , vol. 9, no. 1, pp. 1–13, 2022.
  * [40] M. J. Bocus, W. Li, S. Vishwakarma, R. Kou, C. Tang, K. Woodbridge, I. Craddock, R. McConville, R. Santos-Rodriguez, K. Chetty _et al._ , “Operanet, a multimodal activity recognition dataset acquired from radio frequency and vision-based sensors,” _Scientific data_ , 2022.
  * [41] L. Guo, L. Wang, J. Liu, W. Zhou, and B. Lu, “Huac: Human activity recognition using crowdsourced wifi signals and skeleton data,” _Wireless Communications and Mobile Computing_ , vol. 2018, 2018.
  * [42] T. Li, L. Fan, M. Zhao, Y. Liu, and D. Katabi, “Making the invisible visible: Action recognition through walls and occlusions,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 872–881.
  * [43] A. K. Koupai, M. J. Bocus, R. Santos-Rodriguez, R. J. Piechocki, and R. McConville, “Self-supervised multimodal fusion transformer for passive activity recognition,” _arXiv preprint arXiv:2209.03765_ , 2022.
  * [44] Q. Zhang and Y. Li, “Indoor positioning method based on infrared vision and uwb fusion,” in _Journal of Physics: Conference Series_ , 2021.
  * [45] N. L. Bragazzi, R. Khamisy-Farah, and M. Converti, “Ensuring equitable, inclusive and meaningful gender identity-and sexual orientation-related data collection in the healthcare sector: insights from a critical, pragmatic systematic review of the literature,” _International Review of Psychiatry_ , 2022.
  * [46] S. He, Z. Han, C. Iglesias, V. Mehta, and M. Bolic, “A real-time respiration monitoring and classification system using a depth camera and radars,” _Frontiers in Physiology_ , 2022.
  * [47] L. Ren, L. Kong, F. Foroughian, H. Wang, P. Theilmann, and A. E. Fathy, “Comparison study of noncontact vital signs detection using a doppler stepped-frequency continuous-wave radar and camera-based imaging photoplethysmography,” _IEEE Transactions on Microwave Theory and Techniques_ , vol. 65, no. 9, pp. 3519–3529, 2017.
  * [48] X. Yang, Z. Zhang, X. Li, Y. Zheng, and Y. Shen, “Remote radar-camera vital sign monitoring system using a graph-based extraction algorithm,” in _2021 46th International Conference on Infrared, Millimeter and Terahertz Waves (IRMMW-THz)_. IEEE, 2021, pp. 1–2.
  * [49] Z. Xie, B. Zhou, X. Cheng, E. Schoenfeld, and F. Ye, “Vitalhub: Robust, non-touch multi-user vital signs monitoring using depth camera-aided uwb,” in _IEEE International Conference on Healthcare Informatics_ , 2021.
  * [50] C. Yang, B. Bruce, X. Liu, B. Gholami, and N. Tavassolian, “A hybrid radar-camera respiratory monitoring system based on an impulse-radio ultrawideband radar,” in _Annual International Conference of the IEEE Engineering in Medicine & Biology Society_, 2020.
  * [51] D.-M. Chian, C.-K. Wen, C.-J. Wang, M.-H. Hsu, and F.-K. Wang, “Vital signs identification system with doppler radars and thermal camera,” _IEEE Transactions on Biomedical Circuits and Systems_ , vol. 16, no. 1, pp. 153–167, 2022.
  * [52] A. Vilesov, P. Chari, A. Armouti, A. B. Harish, K. Kulkarni, A. Deoghare, L. Jalilian, and A. Kadambi, “Blending camera and 77 ghz radar sensing for equitable, robust plethysmography,” _ACM Transactions on Graphics_ , 2022\.
  * [53] J. C. Soto, I. Galdino, E. Caballero, V. Ferreira, D. Muchaluat-Saade, and C. Albuquerque, “A survey on vital signs monitoring based on wi-fi csi data,” _Computer Communications_ , vol. 195, pp. 99–110, 2022.
  * [54] V. Selvaraju, N. Spicher, J. Wang, N. Ganapathy, J. M. Warnecke, S. Leonhardt, R. Swaminathan, and T. M. Deserno, “Continuous monitoring of vital signs using cameras: a systematic review,” _Sensors_ , vol. 22, no. 11, p. 4097, 2022.
  * [55] L. Zhang, C. Fu, C. Li, and H. Hong, “Rf and camera-based vital signs monitoring applications,” in _Contactless Vital Signs Monitoring_. Elsevier, 2022, pp. 303–326.
  * [56] Y. Rong, P. C. Theofanopoulos, G. C. Trichopoulos, and D. W. Bliss, “A new principle of pulse detection based on terahertz wave plethysmography,” _Scientific reports_ , vol. 12, no. 1, pp. 1–15, 2022.
  * [57] E. Cardillo, C. Li, and A. Caddemi, “Vital sign detection and radar self-motion cancellation through clutter identification,” _IEEE Transactions on Microwave Theory and Techniques_ , vol. 69, no. 3, pp. 1932–1942, 2021.
  * [58] T. K. V. Dai, K. Oleksak, T. Kvelashvili, F. Foroughian, C. Bauder, P. Theilmann, A. E. Fathy, and O. Kilic, “Enhancement of remote vital sign monitoring detection accuracy using multiple-input multiple-output 77 ghz fmcw radar,” _IEEE Journal of Electromagnetics, RF and Microwaves in Medicine and Biology_ , vol. 6, no. 1, pp. 111–122, 2022.
  * [59] K.-C. Peng, M.-C. Sung, F.-K. Wang, and T.-S. Horng, “Noncontact vital sign sensing under nonperiodic body movement using a novel frequency-locked-loop radar,” _IEEE Transactions on Microwave Theory and Techniques_ , vol. 69, no. 11, pp. 4762–4773, 2021.
  * [60] Y. Wang, Y. Shui, X. Yang, Z. Li, and W. Wang, “Multi-target vital signs detection using frequency-modulated continuous wave radar,” _EURASIP Journal on Advances in Signal Processing_ , vol. 2021, no. 1, pp. 1–19, 2021.
  * [61] C. Feng, X. Jiang, M.-G. Jeong, H. Hong, C.-H. Fu, X. Yang, E. Wang, X. Zhu, and X. Liu, “Multitarget vital signs measurement with chest motion imaging based on mimo radar,” _IEEE Transactions on Microwave Theory and Techniques_ , vol. 69, no. 11, pp. 4735–4747, 2021.
  * [62] Y. Shi, L. Du, X. Chen, X. Liao, Z. Yu, Z. Li, C. Wang, and S. Xue, “Robust gait recognition based on deep cnns with camera and radar sensor fusion,” _IEEE Internet of Things Journal_ , 2023.
  * [63] H. Li, P. Zhang, S. Al Moubayed, S. N. Patel, and A. P. Sample, “Id-match: A hybrid computer vision and rfid system for recognizing individuals in groups,” in _CHI Conference on Human Factors in Computing Systems_ , 2016\.
  * [64] H. Chen, S. Munir, and S. Lin, “Rfcam: Uncertainty-aware fusion of camera and wi-fi for real-time human identification with mobile devices,” _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ , 2022.
  * [65] D. Cao, R. Liu, H. Li, S. Wang, W. Jiang, and C. X. Lu, “Cross vision-rf gait re-identification with low-cost rgb-d cameras and mmwave radars,” _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ , 2022.
  * [66] H. Liu, A. Alali, M. Ibrahim, B. B. Cao, N. Meegan, H. Li, M. Gruteser, S. Jain, K. Dana, A. Ashok, B. Cheng, and H. Lu, “Vi-fi: Associating moving subjects across vision and wireless sensors,” in _ACM/IEEE International Conference on Information Processing in Sensor Networks_ , 2022.
  * [67] L. Deng, J. Yang, S. Yuan, H. Zou, C. X. Lu, and L. Xie, “Gaitfi: Robust device-free human identification via wifi and vision multimodal learning,” _IEEE Internet of Things Journal_ , 2022.
  * [68] S. Fang, T. Islam, S. Munir, and S. Nirjon, “Eyefi: Fast human identification through vision and wifi-based trajectory matching,” in _International Conference on Distributed Computing in Sensor Systems_ , 2020.
  * [69] A. Luchetti, A. Carollo, L. Santoro, M. Nardello, D. Brunelli, and P. Bosetti, “Human identification and tracking using ultra-wideband-vision data fusion in unstructured environments,” _Acta IMEKO e-Journal of the International Measurement Confederation_ , 2021.
  * [70] C. Wan, L. Wang, and V. V. Phoha, “A survey on gait recognition,” _ACM Computing Surveys_ , 2018.
  * [71] A. Nambiar, A. Bernardino, and J. C. Nascimento, “Gait-based person re-identification: A survey,” _ACM Computing Surveys_ , 2019.
  * [72] A. Sepas-Moghaddam and A. Etemad, “Deep gait recognition: A survey,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2023.
  * [73] T. Suwannaphong, R. McConville, and I. Craddock, “Radio signal strength indication augmentation for one-shot learning in indoor localisation,” in _Proceedings of the 1st ACM Workshop on Smart Wearable Systems and Applications_ , 2022, pp. 7–12.
  * [74] W. Jiang, F. Li, L. Mei, R. Liu, and S. Wang, “Visble: Vision-enhanced ble device tracking,” in _IEEE International Conference on Sensing, Communication, and Networking_ , 2022.
  * [75] T. Ishihara, K. M. Kitani, C. Asakawa, and M. Hirose, “Deep radio-visual localization,” in _IEEE Winter Conference on Applications of Computer Vision (WACV)_ , 2018.
  * [76] K. Deng, D. Zhao, Q. Han, S. Wang, Z. Zhang, A. Zhou, and H. Ma, “Geryon: Edge assisted real-time and robust object detection on drones via mmwave radar and camera fusion,” _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ , 2022.
  * [77] Y. Li, J. Deng, Y. Zhang, J. Ji, H. Li, and Y. Zhang, “EZFusion: A close look at the integration of lidar, millimeter-wave radar, and camera for accurate 3d object detection and tracking,” _IEEE Robotics and Automation Letters_ , 2022.
  * [78] H. Li, R. Liu, S. Wang, W. Jiang, and C. X. Lu, “Pedestrian liveness detection based on mmwave radar and camera fusion,” in _IEEE International Conference on Sensing, Communication, and Networking_ , 2022.
  * [79] S. Papaioannou, H. Wen, A. Markham, and N. Trigoni, “Fusion of radio and camera sensor data for accurate indoor positioning,” in _IEEE International Conference on Mobile Ad Hoc and Sensor Systems_ , 2014.
  * [80] Y. Zhao, J. Xu, J. Wu, J. Hao, and H. Qian, “Enhancing camera-based multimodal indoor localization with device-free movement measurement using wifi,” _IEEE Internet of Things Journal_ , 2020.
  * [81] J. Cai and H. Cai, “Robust hybrid approach of vision-based tracking and radio-based identification and localization for 3d tracking of multiple construction workers,” _Journal of Computing in Civil Engineering_ , 2020\.
  * [82] J. Xu, H. Chen, K. Qian, E. Dong, M. Sun, C. Wu, L. Zhang, and Z. Yang, “Ivr: Integrated vision and radio localization with zero human effort,” _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ , 2019.
  * [83] T. H. Nguyen, T.-M. Nguyen, and L. Xie, “Range-focused fusion of camera-imu-uwb for accurate and drift-reduced localization,” _IEEE Robotics and Automation Letters_ , 2021.
  * [84] F. Liu, J. Zhang, J. Wang, H. Han, and D. Yang, “An uwb/vision fusion scheme for determining pedestrians’ indoor location,” _Sensors_ , 2020.
  * [85] L. Varotto, A. Cenedese, and A. Cavallaro, “Probabilistic radio-visual active sensing for search and tracking,” in _European Control Conference_ , 2021\.
  * [86] R. Streubel and B. Yang, “Fusion of stereo camera and mimo-fmcw radar for pedestrian tracking in indoor environments,” in _2016 19th International Conference on Information Fusion (Fusion)_. IEEE, 2016, pp. 565–572.
  * [87] A. Pearce, J. A. Zhang, and R. Xu, “A combined mmwave tracking and classification framework using a camera for labeling and supervised learning,” _Sensors_ , vol. 22, no. 22, p. 8859, 2022.
  * [88] T.-Y. Lim, S. A. Markowitz, and M. N. Do, “Radical: A synchronized fmcw radar, depth, imu and rgb camera data dataset with low-level fmcw radar signals,” _IEEE Journal of Selected Topics in Signal Processing_ , vol. 15, no. 4, pp. 941–953, 2021.
  * [89] K. Cai, Q. Xia, P. Li, J. Stankovic, and C. X. Lu, “Robust human detection under visual degradation via thermal and mmwave radar fusion,” in _International Conference on Embedded Wireless Systems and Networks_ , 2023\.
  * [90] M. Tarkowski, K. Bizewski, M. Rzymowski, K. Nyka, and L. Kulas, “Wireless multimodal localization sensor for industrial applications,” in _2016 21st International Conference on Microwave, Radar and Wireless Communications (MIKON)_. IEEE, 2016, pp. 1–4.
  * [91] P. Woznica, M. Tarkowski, M. Plotka, and L. Kulas, “Rf indoor positioning system supported by wireless computer vision sensors,” in _2014 20th International Conference on Microwaves, Radar and Wireless Communications (MIKON)_. IEEE, 2014, pp. 1–3.
  * [92] C.-S. Wang and L.-C. Cheng, “Rfid & vision based indoor positioning and identification system,” in _2011 IEEE 3rd international conference on communication software and networks_. IEEE, 2011, pp. 506–510.
  * [93] M. Sturari, D. Liciotti, R. Pierdicca, E. Frontoni, A. Mancini, M. Contigiani, and P. Zingaretti, “Robust and affordable retail customer profiling by vision and radio beacon sensor fusion,” _Pattern Recognition Letters_ , 2016\.
  * [94] R. Xu, W. Dong, A. Sharma, and M. Kaess, “Learned depth estimation of 3d imaging radar for indoor mapping,” in _2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2022, pp. 13 260–13 267.
  * [95] C. X. Lu, S. Rosa, P. Zhao, B. Wang, C. Chen, J. A. Stankovic, N. Trigoni, and A. Markham, “See through smoke: robust indoor mapping with low-cost mmwave radar,” in _Proceedings of the 18th International Conference on Mobile Systems, Applications, and Services_ , 2020, pp. 14–27.
  * [96] N. Long, K. Wang, R. Cheng, K. Yang, and J. Bai, “Fusion of millimeter wave radar and rgb-depth sensors for assisted navigation of the visually impaired,” in _Millimetre Wave and Terahertz Sensors and Technology XI_ , vol. 10800. SPIE, 2018, pp. 21–28.
  * [97] H. Ding, Z. Chen, C. Zhao, F. Wang, G. Wang, W. Xi, and J. Zhao, “MI-Mesh: 3D human mesh construction by fusing image and millimeter wave,” _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ , 2023.
  * [98] W. Guo, J. Wang, and S. Wang, “Deep multimodal representation learning: A survey,” _IEEE Access_ , vol. 7, pp. 63 373–63 394, 2019.
  * [99] M. Sturari, D. Liciotti, R. Pierdicca, E. Frontoni, A. Mancini, M. Contigiani, and P. Zingaretti, “Robust and affordable retail customer profiling by vision and radio beacon sensor fusion,” _Pattern Recognition Letters_ , vol. 81, pp. 30–40, 2016.
  * [100] P. Stotko, M. Weinmann, and R. Klein, “Albedo estimation for real-time 3d reconstruction using rgb-d and ir data,” _ISPRS Journal of Photogrammetry and Remote Sensing_ , vol. 150, pp. 213–225, 2019.
  * [101] M. Muaaz, A. Chelli, A. A. Abdelgawwad, A. C. Mallofré, and M. Pätzold, “Wiwehar: Multimodal human activity recognition using wi-fi and wearable sensing modalities,” _IEEE Access_ , 2020.
  * [102] Z. Qin, Y. Zhang, S. Meng, Z. Qin, and K.-K. R. Choo, “Imaging and fusing time series for wearable sensor-based human activity recognition,” _Information Fusion_ , vol. 53, pp. 80–87, 2020.
  * [103] M. Dimitrievski, L. Jacobs, P. Veelaert, and W. Philips, “People tracking by cooperative fusion of radar and camera sensors,” in _IEEE Intelligent Transportation Systems Conference_ , 2019.
  * [104] F. Cui, Y. Song, J. Wu, Z. Xie, C. Song, Z. Xu, and K. Ding, “Online multi-target tracking for pedestrian by fusion of millimeter wave radar and vision,” in _IEEE Radar Conference_ , 2021.
  * [105] B. B. Cao, A. Alali, H. Liu, N. Meegan, M. Gruteser, K. Dana, A. Ashok, and S. Jain, “Vitag: Online wifi fine time measurements aided vision-motion identity association in multi-person environments,” in _IEEE International Conference on Sensing, Communication, and Networking_ , 2022.
  * [106] R. J. Piechocki, X. Wang, and M. J. Bocus, “Multimodal sensor fusion in the latent representation space,” _Scientific Reports_ , 2023.
  * [107] C. Xie, D. Zhang, Z. Wu, C. Yu, Y. Hu, Q. Sun, and Y. Chen, “Accurate human pose estimation using rf signals,” in _2022 IEEE 24th International Workshop on Multimedia Signal Processing (MMSP)_. IEEE, 2022, pp. 1–6.
  * [108] A. Sengupta, F. Jin, R. Zhang, and S. Cao, “mm-pose: Real-time human skeletal posture estimation using mmwave radars and cnns,” _IEEE Sensors Journal_ , vol. 20, no. 17, pp. 10 032–10 044, 2020.
  * [109] W. Jiang, H. Xue, C. Miao, S. Wang, S. Lin, C. Tian, S. Murali, H. Hu, Z. Sun, and L. Su, “Towards 3d human pose construction using wifi,” in _Proceedings of the 26th Annual International Conference on Mobile Computing and Networking_ , 2020, pp. 1–14.
  * [110] F. Wang, S. Zhou, S. Panev, J. Han, and D. Huang, “Person-in-wifi: Fine-grained person perception using wifi,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 5452–5461.
  * [111] S. He, V. Mehta, and M. Bolic, “A joint localization assisted respiratory rate estimation using ir-uwb radars,” in _Annual International Conference of the IEEE Engineering in Medicine & Biology Society_, 2020.
  * [112] Y. Song, T. Jin, Y. Dai, Y. Song, and X. Zhou, “Through-wall human pose reconstruction via uwb mimo radar and 3d cnn,” _Remote Sensing_ , vol. 13, no. 2, p. 241, 2021.
  * [113] C. Gu, G. Wang, Y. Li, T. Inoue, and C. Li, “A hybrid radar-camera sensing system with phase compensation for random body movement cancellation in doppler vital sign detection,” _IEEE transactions on microwave theory and techniques_ , vol. 61, no. 12, pp. 4678–4688, 2013.
  * [114] G. Charan, T. Osman, A. Hredzak, N. Thawdar, and A. Alkhateeb, “Vision-position multi-modal beam prediction using real millimeter wave datasets,” in _2022 IEEE Wireless Communications and Networking Conference_ , 2022.
  * [115] D. Li, J. Xu, Z. Yang, Q. Zhang, Q. Ma, L. Zhang, and P. Chen, “Motion inspires notion: Self-supervised visual-lidar fusion for environment depth estimation,” in _Annual International Conference on Mobile Systems, Applications and Services_ , 2022.
  * [116] F. Ofli, R. Chaudhry, G. Kurillo, R. Vidal, and R. Bajcsy, “Berkeley mhad: A comprehensive multimodal human action database,” in _IEEE Workshop on Applications of Computer Vision_ , 2013.
  * [117] C. Chen, R. Jafari, and N. Kehtarnavaz, “Utd-mhad: A multimodal dataset for human action recognition utilizing a depth camera and a wearable inertial sensor,” in _IEEE International Conference on Image Processing_ , 2015.
  * [118] B. Kwolek and M. Kepski, “Human fall detection on embedded platform using depth maps and wireless accelerometer,” _Computer Methods and Programs in Biomedicine_ , 2014.
  * [119] X. Chao, Z. Hou, and Y. Mo, “Czu-mhad: A multimodal dataset for human action recognition utilizing a depth camera and 10 wearable inertial sensors,” _IEEE Sensors Journal_ , vol. 22, no. 7, pp. 7034–7042, 2022.
  * [120] A. Alkhateeb, G. Charan, T. Osman, A. Hredzak, J. Morais, U. Demirhan, and N. Srinivas, “Deepsense 6g: A large-scale real-world multi-modal sensing and communication dataset,” _IEEE Communications Magazine_ , 2023.
  * [121] A. Chen, X. Wang, S. Zhu, Y. Li, J. Chen, and Q. Ye, “Mmbody benchmark: 3d body reconstruction dataset and analysis for millimeter wave radar,” in _ACM International Conference on Multimedia_ , 2022.
  * [122] L. K. Topham, W. Khan, D. Al-Jumeily, A. Waraich, and A. J. Hussain, “A diverse and multi-modal gait dataset of indoor and outdoor walks acquired using multiple cameras and sensors,” _Scientific Data_ , 2023.
  * [123] A. Sengupta, A. Yoshizawa, and S. Cao, “Automatic radar-camera dataset generation for sensor-fusion applications,” _IEEE Robotics and Automation Letters_ , 2022.
  * [124] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , 2017.
  * [125] P. Xu, X. Zhu, and D. A. Clifton, “Multimodal learning with transformers: A survey,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2023.
  * [126] Y. Kim, S. Kim, J. W. Choi, and D. Kum, “CRAFT: Camera-radar 3D object detection with spatio-contextual fusion transformer,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2023.
  * [127] Y. Lei, Z. Wang, F. Chen, G. Wang, P. Wang, and Y. Yang, “Recent advances in multi-modal 3d scene understanding: A comprehensive survey and evaluation,” _arXiv preprint arXiv:2310.15676_ , 2023.
  * [128] H. Yun, J. Na, and G. Kim, “Dense 2d-3d indoor prediction with sound via aligned cross-modal distillation,” in _IEEE/CVF International Conference on Computer Vision_ , 2023.
  * [129] M. Nourani, C. Roy, T. Rahman, E. D. Ragan, N. Ruozzi, and V. Gogate, “Don’t explain without verifying veracity: An evaluation of explainable ai with video activity recognition,” _arXiv preprint arXiv:2005.02335_ , 2020.
  * [130] M. Z. Uddin and A. Soylu, “Human activity recognition using wearable sensors, discriminant analysis, and long short-term memory-based neural structured learning,” _Scientific Reports_ , 2021.
  * [131] S. Schmidt, J. Stankowicz, J. Carmack, and S. Kuzdeba, “Riftnext: Explainable deep neural rf scene classification,” in _Proceedings of the 3rd ACM Workshop on Wireless Security and Machine Learning_ , 2021, pp. 79–84.
  * [132] M. A. Lee, M. Tan, Y. Zhu, and J. Bohg, “Detect, reject, correct: Crossmodal compensation of corrupted sensors,” in _IEEE International Conference on Robotics and Automation_ , 2021.
  * [133] H. Viswanathan and P. E. Mogensen, “Communications in the 6g era,” _IEEE Access_ , 2020.
  * [134] X. Li, Y. Cui, J. A. Zhang, F. Liu, D. Zhang, and L. Hanzo, “Integrated human activity sensing and communications,” _IEEE Communications Magazine_ , 2022\.
  * [135] A. Adhikary, M. S. Munir, A. D. Raha, Y. Qiao, Z. Han, and C. S. Hong, “Integrated sensing, localization, and communication in holographic mimo-enabled wireless network: A deep learning approach,” _IEEE Transactions on Network and Service Management_ , 2023.
  * [136] Z. Zhao, R. Liu, and J. Li, “Integrated sensing and communication based breath monitoring using 5g network,” in _International Wireless Communications and Mobile Computing (IWCMC)_ , 2023.
  * [137] Z. Zhou, X. Li, J. He, X. Bi, Y. Chen, G. Wang, and P. Zhu, “6g integrated sensing and communication - sensing assisted environmental reconstruction and communication,” in _IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2023.

  *[HAR]: human activity recognition
  *[3D]: three-dimensional
  *[NIC]: network interface card
  *[PWR]: passive WiFi radar
  *[UWB]: Ultra-wide band
  *[WiFi]: wireless fidelity
  *[CSI]: channel state information
  *[FM]: frequency modulated