# Empowering cyberphysical systems of systems with intelligence

Stavros Nousias1, Nikos Piperigkos1, Gerasimos Arvanitis1,2, Apostolos
Fournaris, Aris S. Lalos1 and Konstantinos Moustakas2
1Industrial Systems Institute, Athena Research Center, Patras Science Park,
Greece
2Dept. of ELectrical and Computer Engineering, University of Patras, Greece

###### Abstract

Cyber Physical Systems have been going into a transition phase from individual
systems to a collecttives of systems that collaborate in order to achieve a
highly complex cause, realizing a system of systems approach. The automotive
domain has been making a transition to the system of system approach aiming to
provide a series of emergent functionality like traffic management,
collaborative car fleet management or large-scale automotive adaptation to
physical environment thus providing significant environmental benefits (e.g
air pollution reduction) and achieving significant societal impact. Similarly,
large infrastructure domains, are evolving into global, highly integrated
cyber-physical systems of systems covering all parts of the value chain. In
practice, there are significant challenges in CPSoS applicability and
usability to be addressed, i.e. even a small CPSoS such as a car consists
several subsystems Decentralization of CPSoS appoints tasks to individual CPSs
within the System of Systems. CPSoSs are heterogenous systems. They comprise
of various, autonomous, CPSs, each one of them having unique performance
capabilities, criticality level, priorities and pursued goals. all CPSs must
also harmonically pursue system-based achievements and collaborate in order to
make system-of-system based decisions and implement the CPSoS functionality.
This survey will provide a comprehensive review on current best practices in
connected cyberphysical systems. The basis of our investigation is a dual
layer architecture encompassing a perception layer and a behavioral layer.
Perception algorithms with respect to scene understanding (object detection
and tracking, pose estimation), localization mapping and path planning are
thoroughly investigated. Behavioural part focuses on decision making and human
in the loop control.

###### Index Terms:

IEEE, IEEEtran, journal, LaTeX, paper, template.

##  I Introduction

The Cyber-Physical System domain the past few years have been going into a
transition phase from individual systems operating isolated to a collection of
systems that collaborate to achieve a highly complex cause, realizing a system
of systems approach. There is a significant investment in Cyber-Physical
Systems of Systems both within or outside Europe for domains, like automotive,
industrial manufacturing, railways, aerospace, smart buildings, logistics,
energy, industrial processes, that have a significant impact on the European
economy and society.

The automotive domain, for example, that provides jobs for 12 million people
and accounts for 4% of the EU’s GDP including sales and maintenance for 4.3
million, and transport for 4.8 million, has been thoroughly investing in CPSs
inside cars either to provide elaborate control for traditional automotive
processes (like brake system, steering etc) or to introduce new concepts like
autonomous/semi-autonomous driving. This domain has been making a transition
to the system of system approach aiming to provide a series of emergent
functionality like traffic management, collaborative car fleet management or
large-scale automotive adaptation to the physical environment thus providing
significant environmental benefits (e.g air pollution reduction) and achieving
significant societal impact.

Similarly, large infrastructure domains, like industrial manufacturing with
more than 30 million employees, a turnover of € 6,410 billion, and a value-
added of € 1,590 billion in 2010 are evolving into global, highly integrated
cyber-physical systems of systems that go beyond pure production and that
covers all parts of the value chain, including research, design, and service
provision. This novel approach can enable a high level of flexibility that can
be interpreted to fast adaptation to customer requirements, a high degree of
product customization and better industrial sustainability.

Achieving collective behaviour for CPSoS based solutions in large scale
control processes will help citizens improve their quality of life through
smart, safe, and secure cities, energy-efficient buildings and cars, and green
infrastructures (traffic management, lighting, water and waste management);
and smart devices and services for smart home functionality, home monitoring,
health services, and assisted living.

However, in practice, there are significant challenges in CPSoS applicability
and usability to be addressed to take full advantage of the CPSoS benefits and
sustain/extend their growth. The fact that even a small CPSoS, (eg. a
connected car) consists of several subsystems and executes thousands of lines
of code highlights the complexity of the system of system solution and the
extremely elaborate CPSoS orchestration which highlights the need for an
approach beyond traditional control and management center[1].

Given this, having a centralized authority that handles all CPSoS processes,
subsystems and control loops seems to be very hard to capture and implement
thus pointing to a different design, control and management approach.
Decentralization of CPSoS processes and overall functionality by appointing
tasks to individual CPSs within the System of Systems can be a reasonable
solution, yet still, the collaborative mechanism between CPSs (that constitute
the CPSoS behaviour) remains a point of research since appropriate tools and
methodologies are needed to assess that the expected system of system
functional requirements is retained (the CPSoS operates as it should be) and
the non-functional requirements are matched (the CPSoS remains resilient, safe
and efficient).

CPSoSs are heterogeneous systems. They consist of various, autonomous, CPSs,
each one of them having unique performance capabilities, criticality level,
priorities and pursued goals. CPSs in general are self-organized and, on
several occasions, they may have conflicting goals thus competing to get
access to common resources. However, from a CPSoS perspective, all CPSs must
also harmonically pursue system-based achievements and collaborate to make
system-of-system based decisions and implement the CPSoS behaviour.
Considering that CPSoS consists of many CPSs, finding the methodology to
achieve such an equilibrium in a decentralized way is not an easy task. The
above issue becomes more complex when we also consider the amount of data to
be exchanged between CPSs and the processing of those data. The collection of
data and the data analytics need to be refined in such a way that only the
important information is extracted and forwarded to other CPSs and the overall
system. Also, Mechanisms to handle, in a distributed way, large amounts of
data are needed to extract cognitive patterns and detect abnormalities. Thus,
some data classification, labelling and refinement mechanisms should be put in
place locally (in each CPS) to offload the complexity and communication
overhead at the system of system level[2].

In the above-described setup, we cannot overlook the fact that CPSoS depend on
humans since humans are part of the CPSoS functionality and services, interact
with the CPSs and contribute to the CPSoS behaviour. Operators and managers
play a key role in the operation of CPSoS and take many important decisions
while in several cases human CPS users are the key player in the CPSoS main
role (thus forming Cyber-Physical Human Systems). Thus, we need to structure a
close symbiosis between computer-based systems and human operators/users and
constantly enhance human situational awareness as well as devise a
collaborative mechanism on handling CPSoS decisions, forcing the CPSoS to
comply with human guidelines and reactions. Novel approaches on Human Machine
Interfaces that employ eXtended Reality (XR) principles need to be devised to
help humans gain fast and easy to grasp insight into the CPSoS processes but
also to enrol them seamlessly to the CPSoS operation.

Finally, it cannot be overlooked that security and trust in CPSoS operation
must be retained at all costs since CPSoSs are physically entangled systems
and are in close integration with humans. Security breaches can lead to
serious incidents that may affect human lives (in automotive, energy,
aerospace, railways, industrial domain etc.). The autonomous nature of the
CPSoS, the high heterogeneity and the use of legacy components, however, makes
traditional security measures hard to apply, thus highlighting the need for a
new, CPS applicable security and trust mechanism that must be applied to the
system from design time and follow the CPSoS design operate continuum, thus
constantly be updated, reconfigured and redesigned according to cybersecurity
abnormalities. To achieve that, security components must be modelled based on
the security-by-design principle considering that they may be placed in CPSs
with various, different, security needs and performance capabilities. Such
components must be realized during the design/redesign of the CPSoS while in
parallel specialized security monitoring mechanisms and tools must be
introduced in the autonomous CPSs and the System as a whole, so that they can
detect, identify, respond and mitigate a security attack in the presence of
unforeseen conditions that may emerge during CPSoS operation including
resilience failures.

This survey will provide a comprehensive review of current best practices in
connected cyber-physical systems. The basis of our investigation is a dual-
layer architecture encompassing a perception layer and a behavioural layer.
Perception algorithms concerning scene understanding (object detection and
tracking, pose estimation), localization mapping and path planning are
thoroughly investigated. The behavioural part focuses on decision making and
human in the loop control. The rest of this survey is organized as follows.
Section II analyses aspects of the perception layer focusing on 2D and 3D
object detection and scene analysis, localization, SLAM and path planning and
human-centric perception. Section III focuses on the behavioural layer, while
Section IV concludes this survey paper.

##  II Perception layer

###  II-A Object detection and scene analysis

####  II-A1 Object detection from 2D images

Object detection has been evolved considerably since the appearance of deep
convolutional neural networks [3]. Nowadays, there are two main branches of
proposed techniques. In the first one, the object detectors, using two stages,
generate region proposals which are subsequently classified in the categories
that are determined by the application at hand (e.g., vehicles, cyclists and
pedestrians, in the case of autonomous driving). Some important,
representative, high-performance examples of this first branch are Faster
R-CNN [4], Region-based Fully Convolutional Network (R-FCN) [5], Feature
Pyramid Network (FPN) [6] and Mask R-CNN [7]. In the second branch, object
detection is cast to a single-stage, regression-like task with the aim to
provide directly both the locations and the categories of the detected
objects. Notable examples, here, are Single Shot MultiBox Detector (SSD) [8],
SqueezeDet [9], YOLOv3 [10] and EfficientDet [11].

####  II-A2 Object detection from 3D images

Object detection in LIDAR point clouds is a three-dimensional problem where
the sampled points are not uniformly distributed over the objects in the scene
and do not directly correspond to a cartesian grid. 3D object detection is
dominantly performed with 3D convolutional networks due to the irregularity
and lack of apparent structure in the point cloud. Several transformations
take place to match the point cloud to feature maps that are forwarded into
deep networks. Commendable detection outcomes appear in the literature as
early as 2016. Li et al.[12] projected the 3D points in a 2D map and employed
2D fully convolutional networks to successfully detect cars in a LIDAR point
cloud reaching an accuracy of 71.0% for car detection of moderate difficulty.
A follow-up paper [13] proposes 3D fully convolutional networks reporting
accuracy of 75.3% for car detection of moderate difficulty. However, since
dense 3D fully convolutional networks demonstrate high execution times, Yan et
al.[14] investigated an improved sparse convolution method for such networks,
which significantly increases the speed of both training and inference.
According to KITTI benchmarks, the reported accuracy reaches 78.6% for car
detection of moderate difficulty. To revisit 2D convolutions in 3D object
detection Pointpillars [15] proposed a novel encoder that utilizes PointNets
to learn a representation of point clouds organized in vertical columns
(pillars) and subsequently employed a series of 2D convolutions. Pointpillars
reported accuracy of 77.28% in the same category. Shi et al. proposed
PointRCNN [16] for 3D object detection from raw point cloud. They devised a
two stages stage approach where the first stage yields a bottom-up 3D proposal
generation the second stage refines the proposals in the canonical coordinates
to obtain the final detection results, reporting accuracy of 78.70%. An
extended variation of PointRCNN is the part-aware and aggregation neural
network(Part-$A^{2}$ Net). The part-aware stage for the first time fully
utilizes free-of-charge part supervisions derived from 3D ground-truth boxes
to simultaneously predict high-quality 3D proposals and accurate intra-object
part locations. Then the part-aggregation stage learns to re-score the box and
refines the box location by exploring the spatial relationship of the pooled
intra-object part locations. The reported accuracy reaches 79.40%.

###  II-B Localization, SLAM and Path planning

Unmanned vehicles, either ground (UGV), aerial (UAV) or underwater (UUV), are
prominent CPSoS. Typical examples include autonomous vehicles and robots,
operating for a variety of different civilian and military challenging tasks.
At the same time, the prototyping of 5G and V2X (e.g. V2V and V2I) related
communication protocols enable the close collaboration of vehicles, to address
their main individual or collective goals. Autonomous vehicles with inter-
communication and network abilities are known as Connected and Automated
Vehicles (CAV), being part of the more general concept of Connected CPSoS. The
main focus of CAV’s related technologies is to increase and improve safety,
security and energy consumption of (cooperative or not) autonomous driving, by
the strict control of vehicle’s position and motion [17]. At a higher level,
CAV have the potential for a further enhancement of the transportation
sector’s overall performance.

Perception and scene analysis ability are fundamental for a vehicle’s reliable
operation. Computer vision-based object detection and tracking should be seen
as a first (though necessary) pre-processing step, feeding more sophisticated
operational modules of vehicle [18]. The latter is imperative to have accurate
knowledge of both its own and its neighbours’ (vehicles, pedestrians or static
landmarks) position, in order to design efficiently the future motion actions,
i.e. to determine the best possible velocity, acceleration, yaw rate, etc.
These motion actions primarily focus on e.g. keeping safe inter-vehicular
distances, eco-friendly driving by reducing gas emissions, etc. The above
challenges can be addressed in the context of Localization, SLAM and Path
planning, which are discussed below:

####  II-B1 Localization

Localization module is responsible for providing absolute position information
to the vehicles. Global Navigation Satellite Systems (GNSS), like GPS, Beidou,
Glonass, etc., are usually exploited for that purpose. GPS sensor is currently
employed as the most common commercial device. It is straightforward to couple
or fuse GPS information with Inertial Measurement Unit (IMU) readings [19], to
design a complete Inertial Navigation System (INS) providing positioning,
velocity and timing solutions (PVT). IMU sensor consists of gyroscopes and
accelerometers for measuring yaw rate and acceleration (in $x$,$y$,$z$
directions) of vehicle. Additionally, odometers and wheel sensors [20] can
also be utilized. However, even highly reliable IMU sensors suffer from
accumulative or drift error, significantly reducing their consistency as the
vehicle is moving. Another limitation of stand-alone GPS localization is
directly related to GPS itself. Its accuracy is highly degraded in dense urban
canyons or tunnels [21], even exceeding 10$m$ error. Main sources of GPS
signal degradation are due to [19] satellite clock error, receiver clock
error, ionosphere delay, tropospheric delay, multi-path, etc. Moreover, it is
vulnerable to cyber-attacks [22], like spoofing or jamming. The former causes
an intentionally ”wrong” position, even kilometres away from expected GPS
measurement. The latter poses a rather more severe threat since it totally
blocks the GPS signal. Several alternative approaches relying on ground base
stations have been developed for enhancing localization accuracies, such as
Assisted GPS (AGPS) or Differential GPS (DGPS). However, they are also
susceptible to multi-path effect and signal blockage [23]. The desired
localization error, as it has been reported in the literature, should be lower
than 1$m$ (where-in-lane accuracy) [24] to meet the standards of autonomous
driving. For example, if a vehicle is localized on the curb instead of the
road, it may lead to a serious accident with pedestrians or other vehicles.
Therefore, it is quite clear that for obtaining the desired positioning
solutions, other types of advanced sensors, like LIDAR, Camera, RADAR, etc.,
must be additionally taken into account. Moreover, the emergence of V2V
communications in the context of the Internet of Things (IoT), facilitates the
exploitation of both onboard and off-board information, in order to design a
more robust localization system. This collaborating multi-modal fusion of
heterogeneous measurements is known as Cooperative Localization (CL), a rather
recent and very promising technique that is able to tackle the limitations and
drawbacks of GPS/IMU localization. Each vehicle can now receive external
information (like absolute position, relative distance, velocity,
acceleration, etc.) from nearby vehicles, infrastructure or pedestrians,
effectively assisting its own localization system.

There are many existing works [21], [25], [26], [27], [28] that survey related
aspects, challenges and algorithms of CL. For example, [21] provides an
overview of current trends and future applications of Localization (not only
CL) in autonomous vehicles environments. The discussed techniques are mainly
distinguished on the basis of the utilized sensor. Ranging measurements like
relative distance and angle can also be extracted through the V2V abilities of
CL. Common ranging techniques include Time of Arrival (TOA), Angle of Arrival
(AoA), Time Difference of Arrival (TDOA), Received Signal Strength (RSS), etc.
The works of [25], [26] delve into detailed mathematical modelling of CL
tasks. More specifically, [25] exploits various criteria to categorize related
algorithms:

  1. 1.

Measurement type: The sensor or ranging technique being used for localization.
V2V communications enable different ranging methods to be used (as mentioned
above).

  2. 2.

Centralized vs Distributed: Centralized algorithms require nodes/vehicles of
the network to broadcast their measurements to a fusion centre (e.g. cloud or
some leader-vehicle), responsible for all the computations. Although higher
accuracy can be achieved, limitations like communication overhead,
computational power, network size, fusion centre malfunctioning etc, must be
taken into account. On the contrary, with distributed processing architecture,
the computations are assigned to each vehicle which interacts only with close
neighbours.

  3. 3.

One-shot vs Tracking: One-shot refer to methods that do not exploit any past
information. On the other hand, tracking has to do with algorithms that, apart
from measurements, employ kinematic models in order to approximate the actual
movement of vehicles. Tracking methods exploit Bayesian estimators as
mentioned below.

  4. 4.

Fusion estimator: Multi-modal fusion is vital for increased location
estimation accuracy. Fusion can be effectively performed using well-known
estimators like Least Squares (LS), Maximum Likelihood (ML), Minimum Mean
Square Error (MMSE), Maximum A Posteriori (MAP), etc. One-shot ML estimator
coincides with (weighted by measurement noise variance) LS when the
measurements are corrupted by Gaussian noise. MMSE and MAP are common Bayesian
estimators which treat the unknown vehicle’s position as a random variable,
instead of a deterministic value as one-shot do. Kalman, Extended Kalman and
Unscented Kalman Filters (KF, EKF, UKF) are prominent examples of MMSE
estimators. Belief propagation and factor graphs optimization is also an
important MAP tool.

The [26] formulates a distributed Gradient Descent (GD) algorithm as LS
solution and the Bayesian factor graph approach of Sum Product Algorithm over
Wireless Networks (SPAWN). In general, distributed and tracking/Bayesian
algorithms are more attractive to perform CL. An overview of distributed
localization algorithms in IoT is also given in [27]. The authors discuss in
addition the proposed distributed geometric framework of DILOC, as well as the
extended versions of DLRE and DILAND, which facilitates the design of a linear
localization algorithm. These methods require the vehicle to be inside the
convex full formed by 3 neighbouring anchors (nodes with known and fully
accurate positions) and to compute its barycentric coordinates with respect to
neighbours. However, major challenges are related to mobile scenarios due to
varying topologies, as well as how feasible the presence of anchors will be in
automotive applications. An interesting approach is discussed in [29], where
mobile agents in general, try to cooperatively estimate their position as well
as to track non-cooperative objects. The authors developed a distributed
particle filter-based belief propagation approach with message passing, though
they consider the presence of anchor nodes. Furthermore, the computational and
communication overhead may be a serious limitation towards real-time
implementation. In [30], a novel distributed technique is proposed to improve
the stand-alone GNSS accuracy of vehicles. Once again, noncooperative objects
or features (e.g. trees, pedestrians, etc.) are exploited in order to improve
location accuracy. Features are cooperatively detected by vehicles using their
onboard sensors (e.g. LIDAR), where a perfect association is assumed. These
Vehicle-to-Features measurements are fused with GNSS in the context of a
Bayesian message-passing approach and KF. Experimental evaluation was assessed
using SUMO simulator, however, the number of detected features, as well as
communication overhead, should be taken into serious account. The work of [31]
extends [30], by proposing a distributed data association framework for
features and vehicles. Data association was based on belief propagation.
Validation was performed in realistic urban traffic conditions. The main
aspect of [31], as well as [30], is that vehicles must reach a consensus about
features state, in order to improve their location. Graph Laplacian CL has
been introduced in [32], [33]. Centralized or distributed Laplacian
Localization formulates a LS optimization problem, which fuses the
heterogeneous inter-vehicular measurements along with the V2V connectivity
topology through the linear Laplacian operator. EKF and KF based solutions
have been proposed for addressing CL in tunnels [34], [35] when the GPS signal
may be blocked. A distributed robust cubature KF enhanced by Huber
M-estimation is presented in [36]. The method is used to tackle the challenges
of data fusion under the presence of outliers. Pseudo-range measurements from
satellites are also considered during the fusion process. Authors in [37]
developed a distributed Bayesian CL method for localizing vehicles in the
presence of Non-Line-of-Sight range measurements and spoofed vehicles. They
focused primarily on ego vehicle location estimation and abnormal vehicles
detection rates.

####  II-B2 SLAM

Simultaneous Localization and Mapping (SLAM) is also a relevant task of
Localization. It refers to the problem of mapping an environment using
measurements from sensors (e.g. Camera or LIDAR) on-board the vehicle or
robot, whilst at the same time estimating the position of that sensor relative
to the map. Although when stated in this way SLAM can appear to be quite an
abstract problem robust and efficient solutions to the SLAM problem are
critical to enabling the next wave of intelligent mobile devices. SLAM in its
general form tries to estimate over a time period, the poses of the
vehicle/sensor and the landmarks’ position of the map, given control input
measurements, provided by odometry sensors onboard the vehicle and
measurements with respect to landmarks. Therefore, we have mainly two
subsystems: Front-end, which detects the landmarks of the map and correlates
them with the poses, and the Back-end, which casts an optimization problem in
order to estimate the pose and the location of landmarks.

SLAM techniques can be distinguished to either Visual or LIDAR based Odometry
(VO and LO) solutions, reflecting camera or LIDAR as the main sensor to be
exploited:

  1. 1.

To compute the local position and motion of a camera, VO algorithms must
estimate the transformation that the camera undergoes between the current
frame and a reference frame. The reference frame can be defined by the
previous frame in the input sequence, some keyframe in the recent past, or a
collection of frames from the recent past. In each case, the task is to
estimate the transformation that takes information in the camera’s current
frame into the frame of reference of the past frame(s). This task can be seen
as an optimisation problem where the cost is given by the residual between the
information measured in the current frame and corresponding information
derived and reprojected from the reference frames. The vast majority of VO
algorithms use feature-based approaches (e.g. [38]). The image is decomposed
to a sparse set of interesting points, where for each interest point location
the local appearance of the image is described by a feature vector that is
invariant to camera transformations. The feature vectors are associated
between the input frame and reference to form a set of geometric constraints
from which we can derive the camera motion and scene structure. In this case,
the cost function is formulated by the difference between the measured
reprojection location of these interest points between frames, referred to as
the reprojection error. However, some known limitations of feature-based
methods include i) extraction of interesting points and feature vectors may be
expensive (using well-known algorithms like SHIFT or SURF), ii) they are prone
to errors in areas where there is a low number of interesting points, etc. On
the contrary, dense or direct VO approaches [39, 40] focus on minimizing the
(geometric) reprojection error, aiming to directly minimise the photometric
error between pixels in the optimization problem. State-of-the-art VO
algorithms include Direct Sparse Odometry (DSO) [41], ORB-SLAM [42] and ORB-
SLAM2 [43].

  2. 2.

LIDAR sensor provides dense 3D point clouds of vehicle’s surroundings. The
goal of LO is to estimate the pose of the vehicle by accumulating the
transformation between consecutive frames of 3D point clouds. The existing LO
solutions can be divided into two groups: point-wise and feature-wise methods.
Point-wise methods estimate the relative transformation directly using the raw
3D points while feature-wise methods try to utilize more sophisticated
characteristics of the point cloud such as the edge and planar feature points.
The most well-known pointwise LO method is the iterative close point (ICP)
[44]. ICP operates at a point-wise level and directly matches two frames of
the point cloud by finding the correspondences. One of the major drawbacks of
the ICP is that when the frames include large quantities of points, ICP may
suffer from a high computational load arising from the point cloud
registration. Many variants of ICP have been proposed to improve its
efficiency and accuracy, such as the Trimmed ICP [45] and Normal ICP [46]. To
avoid the high computational load resulting from using the entire set of raw
points, the feature-based LO methods extract a set of representative features
from the raw points. The fast point feature histogram (FPFH) was proposed in
[47] to extract and describe important features. The FPFH enables the
exploration of the local geometry and the transformation is optimized by
matching the one-by-one FPFH-based correspondence. Another well-known feature-
based LO method is LOAM [48]. Theoretically, LOAM integrates the properties of
both the point-wise and the feature-wise methods. On the one hand, to decrease
the computational load of typical ICP, LOAM proposed to extract two kinds of
feature points, the edge and planar, respectively. The extraction of the
feature is simply based on the smoothness of a small region near a given
feature point. Different from the FPFH which provides multiple categories of
features based on its descriptors, LOAM involves only two feature groups.
Another popular variant of LOAM, is Lego-LOAM [49].

####  II-B3 Path planning

Connected Advanced Driver Assistance Systems (ADAS) help to reduce road
fatalities and have received considerable attention in the research and
industrial societies [50]. Recently, there is a shift of focus from individual
drive-assist technologies like power steering, anti-lock braking systems
(ABS), electronic stability control (ESC), adaptive cruise control (ACC) to
features with a higher level of autonomy like collision avoidance, crash
mitigation, autonomous drive and platooning. More importantly, grouping
vehicles into platoons [51, 52] has received considerable interest, since it
seems to be a promising strategy for efficient traffic management and road
transportation, offering several benefits in highway and urban driving
scenarios related to road safety, highway utility and fuel economy.

To maintain the cooperative motion of vehicles in a platoon, the vehicles
exchange their information with the neighbours using V2V and V2I [53]. The
advances in V2X communication technology [53, 52] enable multiple automated
vehicles to communicate with one another, exchanging sensor data, vehicle
control parameters and visually detected objects facilitating the so-called 4D
cooperative awareness (e.g., identification/detection of occluded pedestrian,
cyclists or vehicles).

Several works have been proposed for tackling the problems of cooperative path
planning. Many of them focus on providing spacing policies schemes using both
centralized and decentralized model predictive controllers. Though very few
take into account the effect of network delays, which are inevitable and can
deteriorate significantly the performance of distributed controllers.

The authors in [54], presented a unified approach to cooperative path-planning
using Nonlinear Model Predictive Control with soft constraints at the planning
layer. The framework additionally accounts for the planned trajectories of
other cooperating vehicles ensuring collision avoidance requirements.
Similarly, a multi-vehicle cooperative control system is proposed in [55, 56]
with a decentralized control structure, allowing each automated vehicle to
conduct path planning and motion control separately. The authors in [51]
present a robust decentralised state-feedback controller in the discrete-time
domain for vehicle platoons, considering identical vehicle dynamics with
undirected topologies. An extensive study of their performance under random
packet drop scenarios is also provided, highlighting their robustness in such
conditions. The authors in [57] have extended decentralized MPC schemes to
incorporate also the predicted trajectories of human driving vehicles. Such
solutions are expected to enable the co-existence of vehicles supporting
various levels of autonomy, ranging from L0 (manual operation) to L5 (fully
autonomous operation) [58].

Additionally to the cooperative path planning mechanisms, spacing policies and
controllers have also received increased interest towards ensuring collision
avoidance by regulating the speeds of the vehicles forming a platoon. Two
different types of spacing policies can be found in the literature, i.e., the
constant-spacing policy [59] and the constant-time-headway spacing policy
(e.g., focusing on maintaining a time gap between vehicles in a platoon
resulting in spaces that increase with velocity) [60]. In both categories,
most works, use a one direction control strategy. At this point, it should be
mentioned that in a one-directional strategy the vehicle controller processes
the measurements which are received from leading vehicles. Similarly, a
bidirectional platoon control scheme takes into consideration the state of
vehicles in front and behind (see [61]). In most of the cooperative platooning
approaches, the vehicle platoons are formulated as double-integrator systems
that deploy decentralised bidirectional control strategies similar to
mass–spring–damper systems. This model is widely deployed since it is capable
of characterising the interaction of the vehicles with uncertain environments
and thus is more efficient in stabilising the vehicle platoon system in the
presence of modelling errors and measurement noise. Though, it should be noted
that the effect of network delays on the performance of such systems, has not
been extensively studied, despite the fact that time delays, including sensor
detective delay, braking delay and fuel delay not only seems to be inevitable
but also is expected to deteriorate significantly the performance of the
distributed controllers.

TABLE I: Cooperative Localization methods

Fusion algorithm(s) | Survey | Centralized solution | Distributed solution | Benefits | Limitations | Reference
---|---|---|---|---|---|---
LS, GD and SPAWN | - | - |  | Two state-of-the art algorithms | Large number of iterations and information exchange are required to reach good solution |  Wymeersch et al. [26]
Particle filter based belief propagation | - | - |  | Distributed tracking of mobile nodes and non cooperative objects | Nodes have to reach consensus on objects’ position |  Meyer et al. [29]
EKF | - | - |  | Overall location estimation in harsh conditions and realistic network simulation | Lacks evaluation for the individual vehicle |  Elazab et al. [34]
Cubature KF and Huber M-estimation | - | - |  | Robust location estimation in the presence of measurement outliers | Not considering the impact of dynamic VANET’s topology |  Liu et al. [36]
- |  | - | - | Complete survey about the different fusion algorithms and technologies for CL | - |  Buehrer et al. [25]
- |  | - | - | Complete survey about the different fusion algorithms and technologies for CL, including SLAM methods | - |  Kuutti et al. [21]
Geometric algorithms |  | - |  | Linear and distributed approach based on sophisticated selection of neighbors | Developed mainly for static scenarios |  Safavi et al. [27]
Gaussian message passing and KF | - |  |  | Distributed CL method relying on the cooperatively detection of features | Vehicles have to reach consensus on features’ position |  Soatti et al. [30]
- |  | - |  | Detailed book about the current and potential status of CL methods | - |  Gao et al. [28]
Particle filter based belief propagation | - | - |  | Distributed data association approach | Vehicles have to reach consensus on features’ position |  Brambilla et al. [31]
Graph Laplacian processing | - |  | - | Fusion of three measurement modalities via linear LS | No motion model is concerned |  Piperigkos et al. [32]
Graph Laplacian processing | - |  |  | Fusion of three measurement modalities via linear LS | No motion model is concerned |  Piperigkos et al. [33]
KF and ML | - | - |  | Effective and simple implementation of cooperative awareness | Measurement model is rather abstract, not discussing in detail how it can be formulated |  Yang et al. [35]
Bayesian approach | - | - |  | Accurate location estimation in harsh conditions | Only ego vehicle location is assessed |  Zhao et al. [37]

TABLE II: SLAM methods based on VO and LO solutions Camera |  LIDAR |  Benefits |  Limitations |  Reference
---|---|---|---|---
- |  |  Fundamental work |  High computational load |  ICP [44]
- |  |  Variant of ICP |  Improves the computational complexity of ICP |  TICP [45]
|  - |  Fundamental feature based approach |  Challenging the extraction of feature points |  Klein et al. [38]
- |  |  Exploits a set of representative features from raw point cloud |  Lacks evaluation in different weather and lighting conditions |  FPFH [47]
|  - |  Directly minimize the photometric error between pixels |  Sensitive to image noise |  Steinbrucker et al. [39]
|  - |  Directly minimize the photometric error between pixels |  Sensitive to image noise |  Whelan et al. [40]
- |  |  State-of-the-art LO solution |  Lacks evaluation in different weather and lighting conditions |  LOAM [48]
|  - |  State-of-the-art VO solution |  Lacks evaluation in different weather and lighting conditions |  ORB-SLAM [42]
- |  |  Variant of ICP |  Improves the computational complexity of ICP |  NICP [46]
|  - |  State-of-the-art VO solution |  Lacks evaluation in different weather and lighting conditions |  ORB-SLAM2 [43]
|  - |  State-of-the-art VO solution |  Lacks evaluation in different weather and lighting conditions |  DSO [41]
- |  |  State-of-the-art LO solution |  Lacks evaluation in different weather and lighting conditions |  LeGO-LOAM[49]
TABLE III: Path planning methods Cooperative path planning |  Spacing controller mechanism |  Year |  Reference
---|---|---|---
- |  |  2013 |  Ghasemi et al. [61]
|  - |  2015 |  Kuriki et al. [56]
- |  |  2017 |  Liu et al. [59]
- |  |  2017 |  Liu et al. [60]
|  - |  2018 |  Viana et al. [57]
|  - |  2019 |  Viana et al. [54]
|  - |  2019 |  Huang et al. [55]
|  - |  2019 |  Taeihagh et al. [58]

###  II-C Human centric perception

Human, as a part of a CPSoS, plays an important role to the functionality of
the system. The humans’ role in such complicated systems (e.g., CPSoS) is
vital since they react and collaborate with the machines, providing them with
useful feedback and affecting the way that these systems work. Humans can
provide valuable input both in an active (on purpose) or in a passive (without
consideration) way. For example, an input such as a gesture, or voice can be
used as an order or command to control the operation of a system via a Human-
Machine Interface (HMI). On the other hand, pose estimation or biometrics,
like heart rate, could be taken into account by a decision component,
resulting in a corresponding change of the system’s functionality for security
reasons (e.g., when a user’s fatigue has been detected).

The following sections present some human-related inputs (e.g., behaviour,
characteristics, etc.,) that can be beneficially used in CPSoSs, according to
the literature.

  * •

Biometrics and Biometric Recognition. The most well-known and most frequently
used biometrics, related to humans, are face, fingerprint, iris, EEG, EGG,
respiratory, and heart rate. Some of them are unique for each person so they
can be used for human identification while others can be used for monitoring
the humans’ state or their special cognitive situation of a specific time
period. The use of biometrics covers a large variety of tasks and applications
in CPSoSs.

Regarding the face of a human as a biometric, the related tasks can be face
detection [62, 63, 64, 65], face alignment [66, 67], face recognition [68],
face tracking [69, 70], face classification/verification [71], and face
landmarks extraction [72, 73, 74]. Fingerprint [75, 76, 77], palmprint [78]
and iris/gaze [79, 80] are mainly used for user’s identification tasks due to
their uniqueness for each person. EEG [81, 82, 83], EGG, respiratory [84, 85,
86], heart rate [87, 88, 89] are used for the user’s state monitoring. Besides
the fact that they can provide valuable information, their usage in real
applications is difficult to be applied due to the special wearable devices
that it is required for the capturing.

The choice of which specific biometric will be utilized depends on the use
case scenario, the availability and feasibility of using a sensor (e.g., it
will be placed in a stationary location or it has to be constantly wearable
during the operation), the special power consumption needs of each sensor, the
accuracy and the latency.

One other important issue, which needs to be taken under serious consideration
before the use of a biometric in real systems, is the privacy and security of
these sensitive data since they must be protected via encoding in order to be
anonymously stored or used.

  * •

Person Identification. Person identification is a common image retrieval
problem, where the objective of this task is the recognition of a specific
person’s identity by usually using only a single image, captured by a camera.

Generally, the person identification task is a more complicated and
challenging problem in comparison with the identification using only the face,
since face identification is applied in a more controlled environment (e.g.,
use of a smaller captured frame, the user has to remove glasses, hat and other
accessories to be identified). On the other hand, person identification has to
deal with more complex issues like the different points of view, light and
weather conditions, different resolution of the camera, types of clothes and a
large variety of background contexts.

Person identification has shown great usability in applications related to
CPSoSs, mostly for security purposes. Its utility has been marked specifically
when it is applied “in the wild” and in uncontrolled environments where other
biometrics are not feasible to be used due to technical constraints. Nowadays
approaches usually use deep networks to perform reliable and accurate results.

Authors in [90] proposed an additive distance constraint approach with similar
labels loss to learn highly discriminative features for person re-
identification. In [91], the authors proposed a deep model (PurifyNet) to
address the issue of the person re-identification task with label noise which
has limited annotated samples for each identity. In [92], an unsupervised re-
identification deep learning approach was used that is capable of
incrementally discovering discriminative information from automatically
generated person tracklet data.

  * •

Human Pose Estimation and Action Recognition.

Human pose estimation and recognition of the human’s action have been proved
as particularly valuable tasks in nowadays video-captured applications related
to CPSoSs. They can be utilized in a variety of fields such as ergonomics
assessment, safe training of new operators, fatigue and drowsiness detection
of the user, human-machine interactions, prediction of operator’s next action
for avoiding accidents through changing the operation of a machine, dangerous
moving monitoring in insecure workspace areas.

A restriction that can negatively affect and obstruct the quality of the
results of these tasks is the limited coverage area of the camera.
Nevertheless, this limitation can be overcome using new types of sensors and
tools like Inertial Measurement Units (IMUs), whole-body tracking system (e.g.
SmartsuitPro, Xsens) etc., [93].

In [94], an approach is presented that exploits visual cues from human pose to
solve industrial scenarios for safety applications in CPSs. In [95], three
modalities (i.e., 3D skeletons, body part images and motion history image) are
integrated into a hybrid deep learning architecture for human action
recognition. The authors in [96] proposed a skeleton-based approach utilizing
Spatio-temporal information and CNNs for the classification of human
activities. The authors in [97] presented an indoor monitoring reconfigurable
CPS that uses embedded local nodes (Nvidia Jetson TX2), proposing learning
architectures to address Human Action Recognition.

  * •

Hand Gesture Recognition.

Hand gesture recognition tasks can be a very useful tool for interactions with
machines or subsystems in CPSoSs [98], and particularly in applications where
the user is not allowed to have physical hand contact with a machine due to
security reasons. This task mainly consists of three sequential steps which
are hand detection, hand tracking, and finally gesture recognition. This means
that hand gesture recognition can occur either by a single image (i.e., static
gesture recognition) or by a sequence of images (i.e., dynamic gesture
recognition). The first strategy looks more like a retrieval problem where the
gesture of the image has to match with a known predefined gesture from a
dataset of gestures. The second is a more complicated problem but it is more
useful since can cover the requirements of a bigger variety of real problems
[99].

Gesture recognition is a very common task in Human-Computer Interaction.
Nonetheless, the recognition of complex patterns demands accurate sensors and
sufficient computational power [100]. Additionally, we have to refer that
visual computing plays an important role in CPSoSs, especially in these
applications where the visual gesture recognition system relies on multi-
sensor measurements [101, 102].

Authors in [98] presented a control interface for cyber-physical systems that
interprets and executes commands in a human-robot shared workspace using a
gesture recognition approach. The authors in [103] tried to address the
problem of personalized gesture recognition for cyber-physical environments,
proposing an event-driven service-oriented framework. While in other gesture
recognition applications, a body-worn setup was proposed, which supplements
the omnipresent 3 DoF motion sensors with a set of ultrasound transceivers
[104].

  * •

Speech and speaker recognition.

Speech recognition is a sub-category of a more generic research area related
to the domain of Natural Language Processing (NLP). The main objective of
speech recognition is to automatically translate the content of the entire
speech (or the most significant part of it) into text or other recognizable
forms from the computers. Assuming that the recording and processing of speech
do not require a special sensor, but just a simple audio recorder, we can
understand how easy to use this information is. Additionally, speech can be
applied without any physical contact interaction, making it an ideal signal
for HMI applications.

Speech recognition tasks can be utilized in the smart input system [105, 106],
automatic transcription system [107, 108], smart voice assistant [109],
computer-assisted speech [110], rehabilitation [111, 112] and language
teaching.

Similar to the face recognition task that focuses on the recognition of an
individual human using the facial information that is enclosed in a single
image, the speaker recognition task tries to achieve the same goal using the
vocal tone information of the subject. Speaker recognition is one of the most
basic components for human identification, which has various applications in
many CPSoSs. Additionally, fusion schemes can be used combining both speaker
recognition and face recognition for more secure integrations [113].

A speaker recognition system consists of three separate parts, namely the
speech acquisition module, the feature extraction and selection module, and
finally the pattern matching and classification module. In CPSoSs, the
implementation of an automatic speech recognition system relies on a voice
user interface so that humans to interact with robots or other CPS components.
Nevertheless, this type of interface can not replace the classical GUIs but it
can intensify them by providing, in some cases, a more efficient way of
interaction.

The authors in [114] developed a technique to train a Neural Network (NN) on
the extracted Mel-frequency Cepstral Coefficient (MFCC) features from audio
samples to increase the recognition accuracy of the short utterance speaker
recognition system. In [115], the authors tried to improve the robustness of
speaker identification, using a Stacked Sparse Denoising Auto-encoder.

TABLE IV: Datasets for face recognition, detection and facial landmarks extraction tasks. DATASET |  Short Description |  Link of the Dataset |  Paper Name
---|---|---|---
Helen |  Helen dataset consists of 2330 images (400x400 pixels) with labeled facial components which are manually annotated, containing contours near to eyes, eyebrows, nose, lips and jawline. |  http://www.ifp.illinois.edu/~vuongle2/helen/ |  Interactive Facial Feature Localization [116]
AFW |  AFW (Annotated Faces in the Wild) is a face detection dataset consisting of 205 images with 468 faces. Each face image is labelled with at most 6 landmarks with visibility labels, as well as a bounding box. |  https://www.ics.uci.edu/~xzhu/face/ |  Face detection, pose estimation, and landmark localization in the wild [117]
300W |  300-W dataset consists of 300 Indoor and 300 Outdoor “in the wild” images, covering a large variety of identity, expression, illumination conditions, pose, occlusion and face size. |  https://ibug.doc.ic.ac.uk/resources/300-W/ |  300 Faces in-the-Wild Challenge: The First Facial Landmark Localization Challenge [118]
LFPW |  The Labeled Face Parts in the Wild (LFPW) consists of 1,432 faces from images which are downloaded from the web (e.g., google.com, flickr.com, and yahoo.com). |  https://neerajkumar.org/databases/lfpw/ |  Localizing parts of faces using a consensus of exemplars [119]
AFLW |  The Annotated Facial Landmarks in the Wild (AFLW) consists of 25,000 faces that are annotated with up to 21 landmarks per image. The images have been gathered from Flickr, covering a large variety of poses, expressions, ethnicities, ages, genders and environmental conditions. |  https://www.tugraz.at/institute/icg/research/team-bischof/lrs/downloads/aflw/ |  Annotated Facial Landmarks in the Wild: A large-scale, real-world database for facial landmark localization [120]
AFLW2000-3D |  AFLW2000-3D dataset consists of 2,000 images that have been annotated using 68 points representing 3D facial landmarks. This dataset is usually used for evaluation of 3D facial landmark detection models. |  http://www.cbsr.ia.ac.cn/users/xiangyuzhu/projects/3DDFA/main.htm |  Face Alignment Across Large Poses: A 3D Solution [121]
300-VW |  300 Videos in the Wild (300-VW) is a dataset for evaluating facial landmark tracking algorithms in the wild. Each video of this dataset is almost 1 minute in duration (at 25-30 fps). Each frame of all videos has been annotated in the same way as the 300 W dataset. |  https://ibug.doc.ic.ac.uk/resources/300-VW/ |  Offline Deformable Face Tracking in Arbitrary Videos [122]
COCO-WholeBody |  This dataset is an extension of COCO dataset cavering a whole-body annotation (i.e., face, hand, feet) |  https://github.com/jin-s13/COCO-WholeBody |  Whole-Body Human Pose Estimation in the Wild [123]
MALF |  MALF consists of 5,250 images with 11,931 faces in total. This dataset is the first face detection dataset that supports fine-gained evaluation. |  http://www.cbsr.ia.ac.cn/faceevaluation/ |  Fine-grained Evaluation on Face Detection in the Wild [124]
FDDB |  FDDB dataset consists of 2,845 images with 5,171 annotated faces. |  http://vis-www.cs.umass.edu/fddb/index.html |  FDDB: A Benchmark for Face Detection in Unconstrained Settings [125]
TABLE V: Datasets of images with iris. DATASET |  Short Description |  Link of the Dataset |  Paper Name
---|---|---|---
UBIRIS.v2 |  The UBIRIS.v2 dataset consists of 11,102 images of iris that were captured from 261 subjects, with 10 images for each subject. The images were acquired using a variety of different conditions like distance, motion and different visible wavelengths. They have been also affected by real noise. |  http://iris.di.ubi.pt/ubiris2.html |  The UBIRIS.v2: A Database of Visible Wavelength Iris Images Captured On-the-Move and At-a-Distance [126]
OpenEDS |  Open Eye Dataset (OpenEDS) consists of images with eyes captured using a virtual-reality head display. This dataset was collected from 152 individual participants and is divided into four subsets. |  https://research.fb.com/programs/openeds-challenge |  OpenEDS: Open Eye Dataset [127]
TABLE VI: Datasets for pose estimation. DATASET |  Short Description |  Link of the Dataset |  Paper Name
---|---|---|---
COCO |  The Microsoft Common Objects in Context (MS COCO) consists of 328,000 images. This dataset is a general-proposed large-scale object detection, segmentation, key-point detection, and captioning dataset containing also labeled human’s poses. |  https://cocodataset.org/ |  Microsoft COCO: Common Objects in Context [128]
MPII |  The MPII Human Pose Dataset consist of 25,000 images of which 15,000 images are training samples, 3,000 images are validation samples and the rest 7,000 images are testing samples. The single-person poses are manually annotated with up to 16 body joints. The images are taken from YouTube videos covering 410 different human activities. |  http://human-pose.mpi-inf.mpg.de/ |  2D Human Pose Estimation: New Benchmark and State of the Art Analysis [129]
DensePose |  DensePose-COCO is a large-scale ground-truth dataset with image-to-surface correspondences which are manually annotated from 50,000 images of the COCO dataset and train DensePose-RCNN, to densely regress part-specific UV coordinates within every human region at multiple frames per second. |  http://densepose.org/ |  DensePose: Dense Human Pose Estimation in the Wild [130]
LSP |  The Leeds Sports Pose (LSP) dataset consists of 2,000 images of sportspersons in total gathered from Flickr, 1,000 for training and 1,000 for testing. This dataset is used for human pose estimation and each image is annotated with 14 joint locations. |  https://dbcollection.readthedocs.io/en/latest/datasets/leeds_sports_pose_extended.html |  Clustered Pose and Nonlinear Appearance Models for Human Pose Estimation [131]
JHMDB |  JHMDB is a recognition dataset that consists of 960 video sequences belonging to 21 actions. This dataset is a subset of the larger HMDB51 dataset which has been collected from digitized movies and YouTube videos. |  http://jhmdb.is.tue.mpg.de/ |  Towards Understanding Action Recognition [132]
Unite the People |  Unite The People dataset is mainly used for 3D body estimation. The images come from an extended version of LSP dataset, as well as the single person-tagged people from the MPII Human Pose Dataset. The images are labeled with different types of annotations such as segmentation labels, pose or 3D representation. |  https://files.is.tuebingen.mpg.de/classner/up/ |  Unite the People: Closing the Loop Between 3D and 2D Human Representations [133]
TABLE VII: Datasets for hand and gesture recognition. DATASET |  Short Description |  Link of the Dataset |  Paper Name
---|---|---|---
HandNet |  The HandNet dataset contains the depth images of 10 participants’ hands non-rigidly deforming in front of a RealSense RGB-D camera. The annotations were generated by a magnetic annotation technique. 6D pose is available for the center of the hand as well as the five fingertips (i.e. position and orientation of each). |  http://www.cs.technion.ac.il/~twerd/HandNet/ |  Rule of thumb: Deep derotation for improved fingertip detection [134]
EgoGesture |  The EgoGesture dataset consists of 2,081 RGB-D videos, 24,161 gesture samples and 2,953,224 frames from 50 distinct subjects. |  http://www.nlpr.ia.ac.cn/iva/yfzhang/datasets/egogesture.html |  EgoGesture: A New Dataset and Benchmark for Egocentric Hand Gesture Recognition [135]
NVGesture |  The NVGesture dataset consists of 1,532 dynamic gestures categorized into 25 classes. The dataset is separated into 1,050 samples for training and 482 for testing. The application in which it can be used is for touchless driver controlling. |  https://research.nvidia.com/publication/online-detection-and-classification-dynamic-hand-gestures-recurrent-3d-convolutional |  Online Detection and Classification of Dynamic Hand Gestures With Recurrent 3D Convolutional Neural Network [136]
IPN Hand |  The IPN Hand is a dataset consisting of videos with sufficient size, variation, and real-world elements capable to be used by deep neural networks for training and evaluation. The application on which this dataset focuses is dynamic hand gesture recognition. |  https://github.com/GibranBenitez/IPN-hand |  Real-time Hand Gesture Detection and Classification Using Convolutional Neural Networks [137]
MLGESTURE |  MlGesture consists of more than 1300 hand gesture videos from 24 participants and features 9 different hand gesture symbols. The dataset has been recorded in a car with 5 different sensor types at two different viewpoints and it can be used for hand gesture recognition tasks. |  https://iiw.kuleuven.be/onderzoek/eavise/mlgesture/home |  Low-latency hand gesture recognition with a low resolution thermal imager [138]
TABLE VIII: Datasets of action recognition. DATASET |  Short Description |  Link of the Dataset |  Paper Name
---|---|---|---
UCF101 |  This dataset consists of 13,320 video clips ($\sim$ 27 hours) from Youtube, classified into 101 categories and into 5 types (i.e., Body motion, Human-human interactions, Human-object interactions, Playing musical instruments and Sports). |  https://www.crcv.ucf.edu/data/UCF101.php |  UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild [139]
Kinetics |  It is a high-quality dataset of videos used for human action recognition. The dataset consists of around 500,000 labeled video clips of 10 seconds covering 600 human action classes with at least 600 video clips for each action class. |  https://deepmind.com/research/open-source/kinetics |  The Kinetics Human Action Video Dataset [140]
HMDB51 |  The HMDB51 is a dataset consisting of 6,849 video clips from 51 action categories (such as “jump”, “kiss” and “laugh”). Each category containing at least 101 clips. |  https://serre-lab.clps.brown.edu/resource/hmdb-a-large-human-motion-database/ |  HMDB: A large video database for human motion recognition [141]
ActivityNet |  The ActivityNet contains 200 different types of activities and a total of 849 hours of videos collected from YouTube. It is one of the largest datasets regarding the number of activity categories and a number of videos. |  http://activity-net.org/ |  ActivityNet: A Large-Scale Video Benchmark for Human Activity Understanding [142]
NTU RGB+D |  NTU RGB+D consists of 56,880 video clips of 60 action classes collected from 40 subjects. The actions can be generally divided into three categories: 40 daily actions (e.g., drinking, eating, reading), nine health-related actions (e.g., sneezing, staggering, falling down), and 11 mutual actions (e.g., punching, kicking, hugging). |  http://rose1.ntu.edu.sg/datasets/actionrecognition.asp |  NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis [143]
KTH |  The KTH dataset contains six actions: walk, jog, run, box, hand-wave, and hand clap by 25 different individuals, in different environments outdoor (s1), outdoor with scale variation (s2), outdoor with different clothes (s3), and indoor (s4). |  https://www.csc.kth.se/cvap/actions/ |  Recognizing Human Actions: A Local SVM Approach [144]
Composable activities dataset |  This dataset consists of 693 annotated videos of activities in 16 classes performed by 14 individuals. |  https://ialillo.sitios.ing.uc.cl/ActionsCVPR2014/ |  Discriminative Hierarchical Modeling of Spatio-Temporally Composable Human Activities [145]
HACS |  HACS dataset contains 504K videos (shorted than 4 minutes) collected from YouTube, categorized in 200 action classes. It is used human action recognition. |  http://hacs.csail.mit.edu/ |  HACS: Human Action Clips and Segments Dataset for Recognition and Temporal Localization [146]
TABLE IX: Datasets for speech recognition. DATASET |  Short Description |  Link of the Dataset |  Paper Name
---|---|---|---
LibriSpeech |  This dataset consist of approximately 1,000 hours of audiobooks. |  http://www.openslr.org/12 |  Librispeech: An ASR corpus based on public domain audio books [147]
Speech Commands |  Speech Commands consists of 65,000 of 30 short words $\sim$ one second long. It is a collection of spoken words by thousands of different people, designed for the training and evaluation of keyword spotting systems. |  https://ai.googleblog.com/2017/08/launching-speech-commands-dataset.html |  Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition [148]
MuST-C |  MuST-C currently represents the largest publicly available multilingual corpus for speech translation from English into several languages. It covers eight languages. It consists of hundred hours of audio recordings from English TED Talks. |  https://ict.fbk.eu/must-c/ |  MuST-C: A multilingual corpus for end-to-end speech translation [149]
Common Voice |  Common Voice is a dataset of 9,283 recorded hours that consists of audio files and corresponding text files including demographic metadata like age, sex, and accent. |  https://commonvoice.mozilla.org/en/datasets |  Common Voice: A Massively-Multilingual Speech Corpus [150]
Libri-Light |  Libri-Light is a collection of over 60K hours of spoken English suitable for training speech recognition systems under limited or no supervision. |  https://github.com/facebookresearch/libri-light |  Libri-Light: A Benchmark for ASR with Limited or No Supervision [151]
THCHS-30 |  THCHS-30 is a free Chinese speech database that can be used for speech recognition systems. |  http://166.111.134.19:7777/data/thchs30/README.html |  THCHS-30 : A Free Chinese Speech Corpus [152]
VOICES |  This dataset consists of speech recorded by far-field microphones in noisy room conditions for using in speech and signal processing approaches. |  https://registry.opendata.aws/lab41-sri-voices/ |  Voices Obscured in Complex Environmental Settings (VOICES) corpus [153]
LibriCSS |  LibriCSS is a real recorded dataset that simulates conversations where are captured by far-field microphones. |  https://github.com/chenzhuo1011/libri_css |  Continuous speech separation: dataset and analysis [154]
SPEECH-COCO |  SPEECH-COCO contains 616,767 audios generated using text-to-speech (TTS) synthesis. The audio files are paired with images. |  https://zenodo.org/record/4282267 |  PEECH-COCO: 600k Visually Grounded Spoken Captions Aligned to MSCOCO Data Set [155]

##  III Behavioral layer

In each CPSoS, the knowledge, senses and expertise of humans constitute
important informative values that can be taken into account for the insurance
of its operational excellence. However, a major concern, which was needed to
be addressed in an early age of CPSoS’s evolution, is in which way these human
abstract features can be accessible and understandable by the CPSoSs.

A way to enter the human, as a separate component into a CPSoS, is by
introducing an anthropocentric mechanism which is known in the literature as
the human-in-the-loop approach [156][157]. This mechanism allows a direct way
that humans can continuously interact with the CPSoSs’ control loops in both
directions of the system (i.e., taking and giving inputs).

Even though common CPSoSs are human-centred systems, in which the human
constitutes an essential part of the system, unfortunately, in many real
cases, these systems still consider the human as an external and unpredictable
element, without taking its importance into deeper consideration. The main
vision of the researchers and engineers is to create a human-machine
symbiosis, integrating humans as holistic beings within CPSoSs. In this way,
CPSoSs have to support a more tight bond with the human element, through
human-in-the-loop controls taking into account human’s features like intents,
psychological and cognitive states, emotions and actions that all of them can
be deduced through sensors data and signal processing approaches.

Additionally, engineers, which design and develop new generations CPSoSs, have
to understand and realize which are these specific features of the CPSoSs that
make them be different from the traditional CPSs. One of these things is the
human-in-the-loop mechanism that allows to CPSoS taking advantage of some
special humans’ characteristics, which make them superiors in comparison with
the machines. The technological assessments are not mature yet to directly
integrate these human-oriented characteristics into machines and robots, so
the use of the human-in-the-loop component is essential to serve the initial
goals of a CPSoS. These characteristics, as have been proposed by the
literature [158], are presented below:

  * •

Cognition. Humans have a totally different way to observe a situation,
understand a problem and make final decisions based on lacking data than
computers do. Human’s cognition is the combined result of knowledge,
experience, inspiration and intuition where no nowadays machine can overcome
or even approach in some way.

  * •

Predicatibility. People are not programmed to perform the same task, in the
same way, every time that they try. In some cases, this would be a problem,
especially when they have to follow very specific instructions. This
characteristic might make them less reliable than a simple computer that just
follows precise orders. However, this unpredictable behaviour could be
beneficial in a critical situation, which may suddenly appear and that has not
been distinctly defined in the script of the instructions. The ability of
humans to be easily adapted to unknown situations make them a perfect
component to provide out-of-the-box solutions in hazardous circumstances.

  * •

Motivation. Humans, from their nature, usually require incentives and become
more productive when they assure them. Motivation can guide a human to perform
more effort to a task than what is required. On the other hand, computers and
machines follow a very specific pipeline of work and they are not able to
change the way that they perform a task in order to enhance their
productivity.

The human-in-the-loop applications can be separated into three main categories
in respect to the type of input that human provides:

  1. 1.

these applications that the human has a head role and directly control the
functionality of the CPSoS as a user or operator

  2. 2.

these applications where the system passively monitors humans (e.g.,
biometrics, pose e.g.,) and based on them it takes decisions for appropriate
actions

  3. 3.

these which are a hybrid combination of the aforementioned two types

Direct human control of a CPSoS. The applications of this category can be
separated into two different sub-categories, related to the degree of freedom
that the CPSoS has in order to make its own decision and takes unsupervised
actions.

In the first sub-category, operators manage a process that is close to an
autonomous task. This means that the system has full control of its action
however the user is responsible to adjust some parameters that may affect the
functionality of the system when it is required for external reasons. An
example that can describe a scenario like this, is when an operator sets new
values, to specific parameters on a machine in the industry, for changing the
operation of the assembly line (e.g., for a new product).

In the second sub-category, the operator has even more power and plays a more
active role in the process, by directly controlling some tasks, setting
explicit commands for the operation of the machines or robots. An example of
this scenario is when an expert operator has to take complete control of a
robotic arm remotely, for repairing purposes.

Human monitoring applications. The applications of this category represented
by systems that passively monitor humans’ actions, behaviour and biometrics
using the acquired data of sensors to make appropriate decisions or to display
the information. Based on the type of reaction of the system, the applications
can be separated into two types, namely the open-loop and closed-loop systems.

Open-loop systems continuously monitor humans and just visualize (e.g., smart
glasses) or send a report with appropriate results, which may be useful or
interesting for the operator. The system does not take any further action in
this case. The presented results can cover (i) the first level of information,
(ii) the second level of information and (iii) KPIs. First level information
includes measurements that usually are directly received by the sensors (e.g.,
heart rate, respiratory, blinking of eyes). After the appropriate process of
the first level information, it can be produced the second level information
(e.g. drowsiness, awareness, anxiety level) that corresponds to a higher
contextual meaning.

Closed-loop systems use the received information of the sensors and the
processing results in order to take an action. For example in an automotive
use case, if critical drowsiness of a driver is detected by the system then
the car would take full control of the vehicle and additionally would
appropriately inform the driver of his condition, as a supplementary task.

Hybrid Systems Hybrid systems take human-centric sensing information as
feedback to perform an open or closed-loop action but additionally, at the
same time, they also take into account the direct human inputs and
preferences. An example in the manufacturing use case, the system monitors the
operator’s actions, while he collaborates with a robot, and provides
appropriate guided instructions. However, the level of detail of the guided
assistant can be modified by the personalized preferences of the user that can
be related by the level of his experience.

Humans play an important role in CPSoSs. Their contribution can be summarized
into three categories (i) for data acquisition, (ii) for inference related to
their state and (iii) for the actuation of an action to complete a task of
their own or to collaborate with other components of the system [159]. More
specifically:

  1. 1.

Data acquisition:

     * •

Human as an informer. Humans provide the system with information through the
wearable sensors that may carry or through other devices that monitor them.

     * •

Human as a communicator to transfer condensed knowledge. Humans are excellent
communicators. They have the special ability to easily understand complicated
information, make conclusions and pass to the system with filtered, useful and
deductive information.

  2. 2.

State inference:

     * •

Human as an insider component. Training algorithms and machine learning
approaches can be used for the recognition of the human’s state (e.g.,
cognitive, physical, emotional, phycological, etc.,) which may affect the good
functionality of a CPSoS or put the safety of the user at risk. When a
problematic state of a user is identified, the system can change the typical
operation that runs, with the purpose to protect him, or just to inform him
with an appropriate message or warning.

     * •

Human as a feedback component. Based on the state of the user, the system may
provide suggestions or recommendations to them. The acceptance of these
suggestions by the users can be further utilized by the system as useful
feedback to relate user’s preferences and corresponding user’s states,
providing more personalized solutions in future similar situations.

  3. 3.

Actuation:

     * •

Human as actuators. The actions that a human, as a part of a CPSoS, can do,
are: (i) to set the values of some parameters, (ii) to execute specific tasks
or (iii) to take the entire control of the system, if required.

Human-machine-interface (HMI) is referred to as the medium that is utilized
for the direct communication between humans and machines. This interface is
used to facilitate a physical interaction between humans and machines [160].
Typically, the classical HMI system makes up of some common hardware
components, like a screen and keyboard, and software with specialized
functionalities, performing in this way a graphical user interface (GUI). All
sensors and wearable devices, which are connected with humans or other
components of the CPSoS, are also part of an HMI. The HMI has very extensive
usage in CPSoSs, allowing each part of the CPSoS to directly interact with a
human and vice versa, creating a synergy loop between cyber-physical systems
and humans. In the future, HMIs will be also able to have social cohesion
between humans and machines.

The authors in [161] suggested that the main representative of HMI tools in
CPSoSs, which are mainly used for the communication between humans and
machines, are automatic speech recognition, gesture recognition, and extended
reality, which can be represented by augmented or virtual reality. In such
implementations, a touch screen can be the place where humans and machines
meet each other in order to visualize, manipulate and exchange their goals,
models, and ideas. The authors in [162], presented a framework that is capable
of visually acquiring information from HMIs in order to detect and prevent
human-in-the-loop errors in the control rooms of Nuclear Power plants. The
intelligent and adaptable CPSoSs expect the automation systems to be
decentralized and support ”Plug-and-Produce” features. In this way, the HMIs
have to dynamically update and adapt display screens and support elements to
facilitate the work of the operators like IO fields and buttons [163]. The
authors in [164], proposed a graphical HMI mechanism for intelligent and
connected in-vehicle systems in order to offer a better experience to
automotive users. While in [165], the authors presented a way of connecting an
HMI with a software model of an embedded control system and thermodynamic
models in a hybrid co-simulation.

Augmented reality (AR) is the technological tool that allows the enrichment of
the physical world utilizing digital content and information superimposed on
top of a perceived representation of the real environment Theoretically, this
means that any of the senses that a human has would potentially benefit from
this technology. Nevertheless, in practice, real applications, using augmented
reality technology, usually render visual, audio or haptic virtual information
to the physical space of the user.

Recently, modern AR devices (like goggles, smart glasses, head-mounted
displays (HMD) and tablets) have been tremendously evolved to provide great
functionalities to their users. However, there are still open issues related
to the problem of how comfortable and productive can be their use. To overcome
these limitations, authors in [166] proposed a mechanism, namely “accented
visualization” that allows adapting additional data, presented by AR devices,
according to the user’s current interest, attention and focus.

In [167], a paradigm of CPSs is presented showing how it can be implemented in
the pre-construction industry by integrating it with the AR technology to make
real-time decisions. Linking AR with CPS offers promising perspectives for
process-centric guidance of CPS users, where manual tasks can be guided and
monitored, enabling a much better traceability [168]. However, AR applications
are still not widely used in industry, which may lay at the complex industrial
requirements (e.g., technical, environmental and regulative) [169].

Prediction of operator’s intentions is a task that can improve the
effectiveness of collaboration between CPSoSs and humans. An accurate
prediction can be very essential, especially in industrial scenarios where the
resilience and safety of all CPSoSs components mostly depend on the mutual
understanding between humans and CPSoSs. So, it seems necessary to design and
develop reliable, robust and accurate human behaviour modelling techniques
that are capable to predict the human’s actions or behaviour.

On the one hand, operators are mainly responsible for their own safety when
they are in the same working environment with a cobot, performing
collaborative tasks. However, on the other hand, CPSoSs must have smart
components that are able to identify, understand and even predict operators’
intentions with a primary goal to protect them from a serious injury, for
instance. For this implementation, a continuous video capturing component can
be used by the prediction system to detect, track, recognize human’s gestures
or postures and an artificial intelligence component to predict human’s
intention.

The system can anticipate when unexpected human operations have been detected
or specific human activity patterns have been predicted [170]. In the
meanwhile, the cobot can perform other tasks [171]. In literature, a lot of
different approaches have been presented to solve the problem of prediction
operator’s intentions, such as a framework for the prediction of human
intentions from RGBD data [172]. A sparse Bayesian learning-based human
intention predictor to predict the future human desired position [173]. A
temporal CNN with a convolution operator for human trajectory prediction
[174]. A system that detects human intentions through a recursive Bayesian
classifier, exploiting head, and hand tracking data [172]. A human intention
inference system that uses Expectation-Maximization algorithm with online
model learning [175].

Awareness

Situation Awareness is used to describe the level of awareness that
operators/drivers/users have of the situation in order to perform tasks
successfully [176]. Based on the definition in [177], situational awareness
needs to include four specific requirements:

  1. 1.

to easily receive information from the environment.

  2. 2.

to integrate this information with relevant internal knowledge, creating a
mental model of the current situation.

  3. 3.

to use this model to direct further perceptual exploration in a continual
perceptual cycle.

  4. 4.

to anticipate future events.

Taking these four requirements into account, situational awareness is defined
as the continuous extraction of environmental information, the integration of
this information with previous knowledge to form a coherent mental picture,
and the use of that picture in directing further perception and anticipating
future events. The system will be able to monitor and understand the user’s
state (e.g. fatigue, cognitive level, etc) in order to produce personalized
alarms, warnings, information and suggestions to the users. A situational
awareness application could also provide:

  * •

Information streams regarding the task underway improving focus

  * •

Personalized reminders regarding other parallel or scheduled tasks
significantly improving response time

  * •

Notifications and visual aids regarding imminent dangers or accident-related
factors

  * •

Environmental values and real-time measurements of sensors

  * •

KPIs visualizing the effectiveness of the CPSoS functionality

Situation awareness is important in cases where a user must intervene in
operations and cooperations of highly automated systems in order to correct
failed autonomous decisions in CPSoSs [178]. It is also an effective method to
keep the mechanical parts of a system as well as the operators secure and
safe, so that it can be classified into two groups, human or computer
awareness [179]. Moreover, situational awareness for security reasons is very
important, since it can be used to inform the user about an attack that takes
place in real-time [180].

Nevertheless, CPSoSs have also to overcome some challenges in respect of
human-in-the-loop component, which are:

  * •

Processing in real-time. The complexity of CPSoSs, consisting of a variety of
different components, leads to the production of a huge amount of data
instantly. The processing of all these data and the real-time decisions making
are challenging tasks, especially when the human-in-the-loop component is
apparent since human’s safety is the most important issue and the processes
related to it would be handle with distinct sensitivity.

The processing of data in batches could be a solution to this challenge.
However, this approach is not reliable in critical situations of CPSoSs that
may appear, where vital and accurate decisions have to be made quickly to
protect human life and security. So in other words, the real-time data
processing framework requires the ability of the system to handle large
amounts of data with very low latency and in relative high performance.

  * •

Online streaming of data. CPSoSs is a system of systems that are connected to
each other, collaborating and transferring also in real-time useful
information and data. The requirement of real-time processing results in the
need for the processing of these data in an online streaming mode.

The challenge, in the case of online streaming, is due to the fact that data
is transferred in an ordered sequence of instances that usually can be
accessed once or a small number of times due to limited computing and storage
capabilities.

The tremendous growth of data demands switching from traditional data
processing solutions to systems, which can process a continuous stream of
real-time data.

  * •

High-dimensional data. High-dimensional data are referred to those data whose
dimensions are more complex than the ordinary data. The presence of high-
dimensional data is becoming a very common issue in many real-world
applications of CPSoSs. The processing of high-dimensional data that has been
acquired by different sensors and devices presents a fundamental challenge
leading to the need for more sophisticated methods to be developed.

High-frequency data are referred to as those data that usually appear as time
series and the update of their values happens very fast (i.e., new
observations take place every milliseconds-second). The appropriate handle of
high-frequency data is essential for contemporary CPSoSs.

Processing of these data introduces new challenges to decision-making tasks,
especially when a human takes part in the CPSoS as a human-in-the-loop
component.

  * •

Unsupervised learning in data of CPSoSs. Unsupervised learning is a type of
learning that tries to autonomously discover hidden patterns in untagged data.
This is a very useful method to be applied in real-time applications where the
observed data has a large variety in comparison with those of a restricted
dataset. However, at the same time is a very challenging task when it is
applied in CPSoSs that require accurate and precise results and usually, there
are no “ground truth” data for the evaluation of the method’s accuracy [181].

##  IV Conclusion

This survey aimed to provide a comprehensive review on current best practices
in connected cyberphysical systems adopting a dual architecture approach with
a perception and a behavioral layer. Cooperative path planning is also
discussed in the context of autonomous vehicles. Several works have been
proposed for tackling the problems of cooperative path planning. Many of them
focus on providing spacing policies schemes using both centralized and
decentralized model predictive controllers. Though very few take into account
the effect of network delays, which are inevitable and can deteriorate
significantly the performance of distributed controllers. The individual nodes
solve a simultaneous localization and mapping (SLAM) problem refering to the
problem of mapping an environment using measurements from sensors (e.g. Camera
or LIDAR) on-board the vehicle or robot, whilst at the same time estimating
the position of that sensor relative to the map. Object detection from either
2D imagery and 3D LIDAR point clouds provide range measurements through the
application of deep neural networks.

In the presented setup, CPSoS depend on humans since humans are part of the
CPSoS functionality and services, the interact with the CPSs and contribute to
the CPSoS behavior. Human, as a part of a CPSoS, plays an important role to
the functionality of the system. The humans’ role in such complicated systems
(e.g., CPSoS) is vital since they react and collaborate with the machines,
providing them with useful feedback and affecting the way that these systems
work. Humans can provide valuable input both in an active (on purpose) or in a
passive (without consideration) way. Operators and managers play a key role in
the operation of CPSoS and take many important decisions while in several
cases human CPS users are key player in the CPSoS main role thus forming Cyber
Physical Human Systems. Human-in-the-loop approach refers to an
anthropocentric mechanism that allows a direct way that humans can
continuously interact with the CPSoSs’ control loops in both directions of the
system (i.e., taking and giving inputs).

## References

  * [1] S. Engell, R. Paulen, M. A. Reniers, C. Sonntag, and H. Thompson, “Core research and innovation areas in cyber-physical systems of systems,” in _International Workshop on Design, Modeling, and Evaluation of Cyber Physical Systems_. Springer, 2015, pp. 40–55.
  * [2] R. Atat, L. Liu, J. Wu, G. Li, C. Ye, and Y. Yang, “Big data meet cyber-physical systems: A panoramic survey,” _IEEE Access_ , vol. 6, pp. 73 603–73 636, 2018.
  * [3] Z.-Q. Zhao, P. Zheng, S.-t. Xu, and X. Wu, “Object detection with deep learning: A review,” _IEEE transactions on neural networks and learning systems_ , vol. 30, no. 11, pp. 3212–3232, 2019.
  * [4] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: towards real-time object detection with region proposal networks,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 39, no. 6, pp. 1137–1149, 2016\.
  * [5] J. Dai, Y. Li, K. He, and J. Sun, “R-fcn: Object detection via region-based fully convolutional networks,” _arXiv preprint arXiv:1605.06409_ , 2016.
  * [6] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie, “Feature pyramid networks for object detection,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 2117–2125.
  * [7] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask r-cnn,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2961–2969.
  * [8] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg, “Ssd: Single shot multibox detector,” in _European conference on computer vision_. Springer, 2016, pp. 21–37.
  * [9] B. Wu, F. Iandola, P. H. Jin, and K. Keutzer, “Squeezedet: Unified, small, low power fully convolutional neural networks for real-time object detection for autonomous driving,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , 2017, pp. 129–137.
  * [10] J. Redmon and A. Farhadi, “Yolov3: An incremental improvement,” _arXiv preprint arXiv:1804.02767_ , 2018.
  * [11] M. Tan, R. Pang, and Q. V. Le, “Efficientdet: Scalable and efficient object detection,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 10 781–10 790.
  * [12] B. Li, T. Zhang, and T. Xia, “Vehicle detection from 3d lidar using fully convolutional network,” _arXiv preprint arXiv:1608.07916_ , 2016.
  * [13] B. Li, “3d fully convolutional network for vehicle detection in point cloud,” in _2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2017, pp. 1513–1518.
  * [14] Y. Yan, Y. Mao, and B. Li, “Second: Sparsely embedded convolutional detection,” _Sensors_ , vol. 18, no. 10, p. 3337, 2018.
  * [15] A. H. Lang, S. Vora, H. Caesar, L. Zhou, J. Yang, and O. Beijbom, “Pointpillars: Fast encoders for object detection from point clouds,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 12 697–12 705.
  * [16] S. Shi, X. Wang, and H. Li, “Pointrcnn: 3d object proposal generation and detection from point cloud,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 770–779.
  * [17] U. Montanaro, S. Dixit, S. Fallah, M. Dianati, A. Stevens, D. Oxtoby, and A. Mouzakitis, “Towards connected autonomous driving: review of use-cases,” _Vehicle System Dynamics_ , vol. 57, no. 6, pp. 779–814, 2018.
  * [18] A. Eskandarian, C. Wu, and C. Sun, “Research advances and challenges of autonomous and connected ground vehicles,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 22, no. 2, pp. 683–711, 2021.
  * [19] A. Noureldin, T. B. Karamat, and J. Georgy, _Fundamentals of Inertial Navigation, Satellite-based Positioning and their Integration_. Springer Berlin Heidelberg, 2013.
  * [20] I. Skog and P. Handel, “In-car positioning and navigation technologies—a survey,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 10, no. 1, pp. 4–21, 2009.
  * [21] S. Kuutti, S. Fallah, K. Katsaros, M. Dianati, F. Mccullough, and A. Mouzakitis, “A survey of the state-of-the-art localization techniques and their potentials for autonomous vehicle applications,” _IEEE Internet of Things Journal_ , vol. 5, no. 2, pp. 829–846, 2018.
  * [22] K. Ren, Q. Wang, C. Wang, Z. Qin, and X. Lin, “The security of autonomous driving: Threats, defenses, and future directions,” _Proceedings of the IEEE_ , vol. 108, no. 2, pp. 357–372, 2020.
  * [23] N. Alam and A. G. Dempster, “Cooperative positioning for vehicular networks: Facts and future,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 14, no. 4, pp. 1708–1717, 2013.
  * [24] J. B. P. Neto, L. C. Gomes, F. M. Ortiz, T. T. Almeida, M. E. M. Campista, L. H. M. Costa, and N. Mitton, “An accurate cooperative positioning system for vehicular safety applications,” _Computers & Electrical Engineering_, vol. 83, p. 106591, 2020.
  * [25] R. M. Buehrer, H. Wymeersch, and R. M. Vaghefi, “Collaborative sensor network localization: Algorithms and practical issues,” _Proceedings of the IEEE_ , vol. 106, no. 6, pp. 1089–1114, 2018.
  * [26] H. Wymeersch, J. Lien, and M. Z. Win, “Cooperative localization in wireless networks,” _Proceedings of the IEEE_ , vol. 97, no. 2, pp. 427–450, 2009\.
  * [27] S. Safavi, U. A. Khan, S. Kar, and J. M. F. Moura, “Distributed localization: A linear theory,” _Proceedings of the IEEE_ , vol. 106, no. 7, pp. 1204–1223, 2018.
  * [28] C. Gao, _Cooperative localization and navigation : theory, research, and practice_. Boca Raton: Taylor & Francis, CRC Press, 2019.
  * [29] F. Meyer, O. Hlinka, H. Wymeersch, E. Riegler, and F. Hlawatsch, “Distributed localization and tracking of mobile networks including noncooperative objects,” _IEEE Transactions on Signal and Information Processing over Networks_ , vol. 2, no. 1, pp. 57–71, 2016.
  * [30] G. Soatti, M. Nicoli, N. Garcia, B. Denis, R. Raulefs, and H. Wymeersch, “Implicit cooperative positioning in vehicular networks,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 19, no. 12, pp. 3964–3980, 2018.
  * [31] M. Brambilla, M. Nicoli, G. Soatti, and F. Deflorio, “Augmenting vehicle localization by cooperative sensing of the driving environment: Insight on data association in urban traffic scenarios,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 21, no. 4, pp. 1646–1663, 2020.
  * [32] N. Piperigkos, A. S. Lalos, K. Berberidis, and C. Anagnostopoulos, “Cooperative multi-modal localization in connected and autonomous vehicles,” in _2020 IEEE 3rd Connected and Automated Vehicles Symposium (CAVS)_ , 2020.
  * [33] N. Piperigkos, A. S. Lalos, and K. Berberidis, “Graph based cooperative localization for connected and semi-autonomous vehicles,” in _2020 IEEE 25th International Workshop on Computer Aided Modeling and Design of Communication Links and Networks (CAMAD)_ , 2020.
  * [34] M. Elazab, A. Noureldin, and H. S. Hassanein, “Integrated cooperative localization for vehicular networks with partial GPS access in urban canyons,” _Vehicular Communications_ , vol. 9, pp. 242–253, 2017.
  * [35] P. Yang, D. Duan, C. Chen, X. Cheng, and L. Yang, “Multi-sensor multi-vehicle (MSMV) localization and mobility tracking for autonomous driving,” _IEEE Transactions on Vehicular Technology_ , vol. 69, no. 12, pp. 14 355–14 364, 2020.
  * [36] J. Liu, B. gen Cai, and J. Wang, “Cooperative localization of connected vehicles: Integrating GNSS with DSRC using a robust cubature kalman filter,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 18, no. 8, pp. 2111–2125, 2017.
  * [37] J. Zhao, Y. Zhang, S. Ni, and Q. Li, “Bayesian cooperative localization with NLOS and malicious vehicle detection in GNSS-challenged environments,” _IEEE Access_ , vol. 8, pp. 85 686–85 697, 2020.
  * [38] G. Klein and D. Murray, “Parallel tracking and mapping for small AR workspaces,” in _2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality_. IEEE, 2007\.
  * [39] F. Steinbrucker, J. Sturm, and D. Cremers, “Real-time visual odometry from dense RGB-d images,” in _2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops)_. IEEE, 2011.
  * [40] T. Whelan, H. Johannsson, M. Kaess, J. J. Leonard, and J. McDonald, “Robust real-time visual odometry for dense RGB-d mapping,” in _2013 IEEE International Conference on Robotics and Automation_. IEEE, 2013.
  * [41] J. Engel, V. Koltun, and D. Cremers, “Direct sparse odometry,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 40, no. 3, pp. 611–625, 2018.
  * [42] R. Mur-Artal, J. M. M. Montiel, and J. D. Tardos, “ORB-SLAM: A versatile and accurate monocular SLAM system,” _IEEE Transactions on Robotics_ , vol. 31, no. 5, pp. 1147–1163, 2015.
  * [43] R. Mur-Artal and J. D. Tardos, “ORB-SLAM2: An open-source SLAM system for monocular, stereo, and RGB-d cameras,” _IEEE Transactions on Robotics_ , vol. 33, no. 5, pp. 1255–1262, 2017.
  * [44] P. Besl and N. D. McKay, “A method for registration of 3-d shapes,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 14, no. 2, pp. 239–256, 1992.
  * [45] Y. Makihara, M. Takizawa, Y. Shirai, J. Miura, and N. Shimada, “Object recognition supported by user interaction for service robots,” in _Object recognition supported by user interaction for service robots_ , vol. 3. IEEE, 2002, pp. 561–564.
  * [46] J. Serafin and G. Grisetti, “NICP: Dense normal based point cloud registration,” in _2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2015.
  * [47] R. B. Rusu, N. Blodow, and M. Beetz, “Fast point feature histograms (FPFH) for 3d registration,” in _2009 IEEE International Conference on Robotics and Automation_. IEEE, 2009\.
  * [48] J. Zhang and S. Singh, “LOAM: Lidar odometry and mapping in real-time,” in _Robotics: Science and Systems X_. Robotics: Science and Systems Foundation, 2014.
  * [49] T. Shan and B. Englot, “LeGO-LOAM: Lightweight and ground-optimized lidar odometry and mapping on variable terrain,” in _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2018.
  * [50] E. Uhlemann, “Connected-vehicles applications are emerging [connected vehicles],” _IEEE Vehicular Technology Magazine_ , vol. 11, no. 1, pp. 25–96, 2016.
  * [51] K. Halder, U. Montanaro, S. Dixit, M. Dianati, A. Mouzakitis, and S. Fallah, “Distributed h controller design and robustness analysis for vehicle platooning under random packet drop,” _IEEE Transactions on Intelligent Transportation Systems_ , 2020.
  * [52] P. Wang, B. Di, H. Zhang, K. Bian, and L. Song, “Platoon Cooperation in Cellular V2X Networks for 5G and Beyond,” _IEEE Transactions on Wireless Communications_ , vol. 18, no. 8, pp. 3919–3932, 2019\.
  * [53] L. Hobert, A. Festag, I. Llatser, L. Altomare, F. Visintainer, and A. Kovacs, “Enhancements of v2x communication in support of cooperative autonomous driving,” _IEEE communications magazine_ , vol. 53, no. 12, pp. 64–70, 2015\.
  * [54] I. B. Viana, H. Kanchwala, and N. Aouf, “Cooperative trajectory planning for autonomous driving using nonlinear model predictive control,” in _2019 IEEE International Conference on Connected Vehicles and Expo (ICCVE)_ , 2019, pp. 1–6.
  * [55] Z. Huang, D. Chu, C. Wu, and Y. He, “Path planning and cooperative control for automated vehicle platoon using hybrid automata,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 20, no. 3, pp. 959–974, 2019.
  * [56] Y. Kuriki and T. Namerikawa, “Formation control with collision avoidance for a multi-uav system using decentralized mpc and consensus-based control,” in _2015 European Control Conference (ECC)_ , 2015, pp. 3079–3084.
  * [57] I. B. Viana and N. Aouf, “Distributed cooperative path-planning for autonomous vehicles integrating human driver trajectories,” in _2018 International Conference on Intelligent Systems (IS)_ , 2018, pp. 655–661.
  * [58] A. Taeihagh and H. S. M. Lim, “Governing autonomous vehicles: emerging responses for safety, liability, privacy, cybersecurity, and industry risks,” _Transport Reviews_ , vol. 39, no. 1, pp. 103–128, 2019.
  * [59] Y. Liu, B. Xu, and Y. Ding, “Convergence analysis of cooperative braking control for interconnected vehicle systems,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 18, no. 7, pp. 1894–1906, 2017.
  * [60] Y. Liu, C. Pan, H. Gao, and G. Guo, “Cooperative spacing control for interconnected vehicle systems with input delays,” _IEEE Transactions on Vehicular Technology_ , vol. 66, no. 12, pp. 10 692–10 704, 2017.
  * [61] A. Ghasemi, R. Kazemi, and S. Azadi, “Stable decentralized control of a platoon of vehicles with heterogeneous information feedback,” _IEEE Transactions on Vehicular Technology_ , vol. 62, no. 9, pp. 4299–4308, 2013.
  * [62] A. Claudi, P. Sernani, G. Dolcini, L. Palazzo, and A. F. Dragoni, “A hierarchical hybrid model for intelligent cyber-physical systems,” in _2013 Proceedings of the 11th Workshop on Intelligent Solutions in Embedded Systems (WISES)_ , 2013, pp. 1–6.
  * [63] J. Isern, F. Barranco, D. Deniz, J. Lesonen, J. Hannuksela, and R. R. Carrillo, “Reconfigurable cyber-physical system for critical infrastructure protection in smart cities via smart video-surveillance,” _Pattern Recognition Letters_ , vol. 140, pp. 303–309, 2020. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0167865520304098
  * [64] L. W. Chen, Y. F. Ho, M. F. Tsai, H. M. Chen, and C. F. Huang, “Cyber-physical signage interacting with gesture-based human–machine interfaces through mobile cloud computing,” _IEEE Access_ , vol. 4, pp. 3951–3960, 2016.
  * [65] J. Galbally, P. Ferrara, R. Haraksim, A. Psyllos, and L. Beslay, “Study on face identification technology for its implementation in the schengen information system,” _Joint Res. Centre, Ispra, Italy, Rep. JRC-34751_ , 2019\.
  * [66] V. G. Kaburlasos, C. Lytridis, C. Bazinas, S. Chatzistamatis, K. Sotiropoulou, A. Najoua, M. Youssfi, and O. Bouattane, “Head pose estimation using lattice computing techniques,” in _2020 International Conference on Software, Telecommunications and Computer Networks (SoftCOM)_ , 2020, pp. 1–5.
  * [67] V. G. Kaburlasos, C. Lytridis, C. Bazinas, G. A. Papakostas, A. Naji, M. H. Zaggaf, K. Mansouri, M. Qbadou, and M. Mestari, “Structured human-head pose representation for estimation using fuzzy lattice reasoning (flr),” in _2020 Fourth International Conference On Intelligent Computing in Data Sciences (ICDS)_ , 2020, pp. 1–5.
  * [68] A. Makovetskii, V. Kober, A. Voronin, and D. Zhernov, “Facial recognition and 3d non-rigid registration,” in _2020 International Conference on Information Technology and Nanotechnology (ITNT)_ , 2020, pp. 1–4.
  * [69] S. Li, K. N. Ngan, R. Paramesran, and L. Sheng, “Real-time head pose tracking with online face template reconstruction,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 38, no. 9, pp. 1922–1928, 2016\.
  * [70] C. J. Pereira Passarinho, E. Ottoni Teatini Salles, and M. Sarcinelli Filho, “Face tracking in unconstrained color videos with the recovery of the location of lost faces,” _IEEE Latin America Transactions_ , vol. 13, no. 1, pp. 307–314, 2015.
  * [71] G. Arvanitis, K. Moustakas, and N. Fakotakis, “Online biometric identification with face analysis in web applications,” in _Speech and Computer_ , A. Ronzhin, R. Potapova, and G. Németh, Eds. Cham: Springer International Publishing, 2016, pp. 515–522.
  * [72] M. Jeong, B. C. Ko, S. Kwak, and J.-Y. Nam, “Driver facial landmark detection in real driving situations,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 28, no. 10, pp. 2753–2767, 2018.
  * [73] S. E. Eskimez, R. K. Maddox, C. Xu, and Z. Duan, “Noise-resilient training method for face landmark generation from speech,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 28, pp. 27–38, 2020\.
  * [74] H. J. Lee, S. T. Kim, H. Lee, and Y. M. Ro, “Lightweight and effective facial landmark detection using adversarial learning with face geometric map generative network,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 30, no. 3, pp. 771–780, 2020.
  * [75] O. S. Okpara and G. Bekaroo, “Cam-wallet: Fingerprint-based authentication in m-wallets using embedded cameras,” in _2017 IEEE International Conference on Environment and Electrical Engineering and 2017 IEEE Industrial and Commercial Power Systems Europe (EEEIC / I CPS Europe)_ , 2017, pp. 1–5.
  * [76] D. Valdes-Ramirez, M. A. Medina-Pérez, R. Monroy, O. Loyola-González, J. Rodríguez, A. Morales, and F. Herrera, “A review of fingerprint feature representations and their applications for latent fingerprint identification: Trends and evaluation,” _IEEE Access_ , vol. 7, pp. 48 484–48 499, 2019\.
  * [77] J. Preciozzi, G. Garella, V. Camacho, F. Franzoni, L. Di Martino, G. Carbajal, and A. Fernandez, “Fingerprint biometrics from newborn to adult: A study from a national identity database system,” _IEEE Transactions on Biometrics, Behavior, and Identity Science_ , vol. 2, no. 1, pp. 68–79, 2020.
  * [78] X. Wang, H. Gong, H. Zhang, B. Li, and Z. Zhuang, “Palmprint identification using boosting local binary pattern,” in _18th International Conference on Pattern Recognition (ICPR’06)_ , vol. 3, 2006, pp. 503–506.
  * [79] F. Vicente, Z. Huang, X. Xiong, F. De la Torre, W. Zhang, and D. Levi, “Driver gaze tracking and eyes off the road detection system,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 16, no. 4, pp. 2014–2027, 2015.
  * [80] C.-C. Lai, S.-W. Shih, and Y.-P. Hung, “Hybrid method for 3-d gaze tracking using glint and contour features,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 25, no. 1, pp. 24–37, 2015.
  * [81] A. Laghari, Z. A. Memon, S. Ullah, and I. Hussain, “Cyber physical system for stroke detection,” _IEEE Access_ , vol. 6, pp. 37 444–37 453, 2018.
  * [82] A. Pandey, R. Sequeria, P. Kumar, and S. Kumar, “A multistage deep residual network for biomedical cyber-physical systems,” _IEEE Systems Journal_ , vol. 14, no. 2, pp. 1953–1962, 2020.
  * [83] S. Lou, Y. Feng, G. Tian, Z. Lv, Z. Li, and J. Tan, “A cyber-physical system for product conceptual design based on an intelligent psycho-physiological approach,” _IEEE Access_ , vol. 5, pp. 5378–5387, 2017.
  * [84] Z. Jiang, M. Hu, Z. Gao, L. Fan, R. Dai, Y. Pan, W. Tang, G. Zhai, and Y. Lu, “Detection of respiratory infections using rgb-infrared sensors on portable device,” _IEEE Sensors Journal_ , vol. 20, no. 22, pp. 13 674–13 681, 2020\.
  * [85] E. Saatci and E. Saatci, “Determination of respiratory parameters by means of hurst exponents of the respiratory sounds and stochastic processing methods,” _IEEE Transactions on Biomedical Engineering_ , pp. 1–1, 2021\.
  * [86] F. Meng, Y. Shi, N. Wang, M. Cai, and Z. Luo, “Detection of respiratory sounds based on wavelet coefficients and machine learning,” _IEEE Access_ , vol. 8, pp. 155 710–155 720, 2020.
  * [87] I. H. Rao, N. A. Amir, H. Dagale, and J. Kuri, “e-surakshak: A cyber-physical healthcare system with service oriented architecture,” in _2012 International Symposium on Electronic System Design (ISED)_ , 2012, pp. 177–182.
  * [88] B. Prado, D. Dantas, K. Bispo, T. Fontes, G. Santana, and R. Silva, “A virtual prototype semihosting approach for early simulation of cyber-physical systems,” in _2018 IEEE Symposium on Computers and Communications (ISCC)_ , 2018, pp. 00 208–00 213.
  * [89] A. J. Majumder, M. Elsaadany, J. A. Izaguirre, and D. R. Ucci, “A real-time cardiac monitoring using a multisensory smart iot system,” in _2019 IEEE 43rd Annual Computer Software and Applications Conference (COMPSAC)_ , vol. 2, 2019, pp. 281–287.
  * [90] G. Li, L. Huang, L. Tang, C. Han, Y. Chen, H. Xie, S. Li, and G. Xu, “Person re-identification using additive distance constraint with similar labels loss,” _IEEE Access_ , vol. 8, pp. 168 111–168 120, 2020.
  * [91] M. Ye and P. C. Yuen, “Purifynet: A robust person re-identification model with noisy labels,” _IEEE Transactions on Information Forensics and Security_ , vol. 15, pp. 2655–2666, 2020.
  * [92] M. Li, X. Zhu, and S. Gong, “Unsupervised tracklet person re-identification,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 42, no. 7, pp. 1770–1782, 2020.
  * [93] F. Hu, Q. Hao, Q. Sun, X. Cao, R. Ma, T. Zhang, Y. Patil, and J. Lu, “Cyberphysical system with virtual reality for intelligent motion recognition and training,” _IEEE Transactions on Systems, Man, and Cybernetics: Systems_ , vol. 47, no. 2, pp. 347–363, 2017.
  * [94] S. O. B. Islam, W. A. Lughmani, W. S. Qureshi, A. Khalid, M. A. Mariscal, and S. Garcia-Herrero, “Exploiting visual cues for safe and flexible cyber-physical production systems,” _Advances in Mechanical Engineering_ , vol. 11, no. 12, p. 1687814019897228, 2019. [Online]. Available: https://doi.org/10.1177/1687814019897228
  * [95] H. El-Ghaish, M. E. Hussien, A. Shoukry, and R. Onai, “Human action recognition based on integrating body pose, part shape, and motion,” _IEEE Access_ , vol. 6, pp. 49 040–49 055, 2018.
  * [96] P. Nikolov, O. Boumbarov, A. Manolova, K. Tonchev, and V. Poulkov, “Skeleton-based human activity recognition by spatio-temporal representation and convolutional neural networks with application to cyber physical systems with human in the loop,” in _2018 41st International Conference on Telecommunications and Signal Processing (TSP)_ , 2018, pp. 1–5.
  * [97] D. Deniz, F. Barranco, J. Isern, and E. Ros, “Reconfigurable cyber-physical system for lifestyle video-monitoring via deep learning,” in _2020 25th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA)_ , vol. 1, 2020, pp. 1705–1712.
  * [98] G. Horváth and G. Erdős, “Gesture control of cyber physical systems,” _Procedia CIRP_ , vol. 63, pp. 184–188, 2017, manufacturing Systems 4.0 – Proceedings of the 50th CIRP Conference on Manufacturing Systems. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S2212827117305000
  * [99] H.-R. Choi and T. Kim, “Combined dynamic time warping with multiple sensors for 3d gesture recognition,” _Sensors_ , vol. 17, no. 8, 2017. [Online]. Available: https://www.mdpi.com/1424-8220/17/8/1893
  * [100] F. Grützmacher, B. Beichler, C. Haubelt, and B. Theelen, “Dataflow-based modeling and performance analysis for online gesture recognition,” in _2016 2nd International Workshop on Modelling, Analysis, and Control of Complex CPS (CPS Data)_ , 2016, pp. 1–8.
  * [101] J. Posada, C. Toro, I. Barandiaran, D. Oyarzun, D. Stricker, R. de Amicis, E. B. Pinto, P. Eisert, J. Döllner, and I. Vallarino, “Visual computing as a key enabling technology for industrie 4.0 and industrial internet,” _IEEE Computer Graphics and Applications_ , vol. 35, no. 2, pp. 26–40, 2015.
  * [102] H. Aviles-Arriaga, L. Sucar, and C. Mendoza, “Visual recognition of similar gestures,” in _18th International Conference on Pattern Recognition (ICPR’06)_ , vol. 1, 2006, pp. 1100–1103.
  * [103] Y. Lou, W. Wu, R.-D. Vatavu, and W.-T. Tsai, “Personalized gesture interactions for cyber-physical smart-home environments,” _Science China Information Sciences_ , vol. 60, no. 7, p. 072104, Oct 2016. [Online]. Available: https://doi.org/10.1007/s11432-015-1014-7
  * [104] V. Putz, J. Mayer, H. Fenzl, R. Schmidt, M. Pichler-Scheder, and C. Kastl, “Cyber–physical mobile arm gesture recognition using ultrasound and motion data,” in _2020 IEEE Conference on Industrial Cyberphysical Systems (ICPS)_ , vol. 1, 2020, pp. 203–208.
  * [105] P. Wang, “Research and design of smart home speech recognition system based on deep learning,” in _2020 International Conference on Computer Vision, Image and Deep Learning (CVIDL)_ , 2020, pp. 218–221.
  * [106] Y. Han, J. Hyun, T. Jeong, J.-H. Yoo, and J. W.-K. Hong, “A smart home control system based on context and human speech,” in _2016 18th International Conference on Advanced Communication Technology (ICACT)_ , 2016, pp. 165–169.
  * [107] J. Chaloupka, P. Červa, J. Silovský, J. Žd’ánský, and J. Nouza, “Modification of the speech feature extraction module for the improvement of the system for automatic lectures transcription,” in _Proceedings ELMAR-2012_ , 2012, pp. 223–226.
  * [108] J. Chaloupka, J. Nouza, J. Malek, and J. Silovsky, “Phone speech detection and recognition in the task of historical radio broadcast transcription,” in _2015 38th International Conference on Telecommunications and Signal Processing (TSP)_ , 2015, pp. 1–4.
  * [109] S. Subhash, P. N. Srivatsa, S. Siddesh, A. Ullas, and B. Santhosh, “Artificial intelligence-based voice assistant,” in _2020 Fourth World Conference on Smart Trends in Systems, Security and Sustainability (WorldS4)_ , 2020, pp. 593–596.
  * [110] C. Tejedor-García, D. Escudero-Mancebo, E. Cámara-Arenas, C. González-Ferreras, and V. Cardeñoso-Payo, “Assessing pronunciation improvement in students of english using a controlled computer-assisted pronunciation tool,” _IEEE Transactions on Learning Technologies_ , vol. 13, no. 2, pp. 269–282, 2020.
  * [111] J. Aishwarya, P. P. Kundapur, S. Kumar, and K. S. Hareesha, “Kannada speech recognition system for aphasic people,” in _2018 International Conference on Advances in Computing, Communications and Informatics (ICACCI)_ , 2018, pp. 1753–1756.
  * [112] T. A. Mariya Celin, G. Anushiya Rachel, T. Nagarajan, and P. Vijayalakshmi, “A weighted speaker-specific confusion transducer-based augmentative and alternative speech communication aid for dysarthric speakers,” _IEEE Transactions on Neural Systems and Rehabilitation Engineering_ , vol. 27, no. 2, pp. 187–197, 2019.
  * [113] J. Zhang and D. Tao, “Empowering things with intelligence: A survey of the progress, challenges, and opportunities in artificial intelligence of things,” 2020.
  * [114] Z. Kozhirbayev, B. A. Erol, A. Sharipbay, and M. Jamshidi, “Speaker recognition for robotic control via an iot device,” in _2018 World Automation Congress (WAC)_ , 2018, pp. 1–5.
  * [115] Z. Wang, S. Duan, C. Zeng, X. Yu, Y. Yang, and H. Wu, “Robust speaker identification of iot based on stacked sparse denoising auto-encoders,” in _2020 International Conferences on Internet of Things (iThings) and IEEE Green Computing and Communications (GreenCom) and IEEE Cyber, Physical and Social Computing (CPSCom) and IEEE Smart Data (SmartData) and IEEE Congress on Cybermatics (Cybermatics)_ , 2020, pp. 252–257.
  * [116] V. Le, J. Brandt, Z. Lin, L. Bourdev, and T. S. Huang, “Interactive facial feature localization,” in _Computer Vision – ECCV 2012_ , A. Fitzgibbon, S. Lazebnik, P. Perona, Y. Sato, and C. Schmid, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2012, pp. 679–692.
  * [117] X. Zhu and D. Ramanan, “Face detection, pose estimation, and landmark localization in the wild,” in _2012 IEEE Conference on Computer Vision and Pattern Recognition_ , 2012, pp. 2879–2886.
  * [118] C. Sagonas, G. Tzimiropoulos, S. Zafeiriou, and M. Pantic, “300 faces in-the-wild challenge: The first facial landmark localization challenge,” in _2013 IEEE International Conference on Computer Vision Workshops_ , 2013, pp. 397–403.
  * [119] P. N. Belhumeur, D. W. Jacobs, D. J. Kriegman, and N. Kumar, “Localizing parts of faces using a consensus of exemplars,” in _CVPR 2011_ , 2011, pp. 545–552.
  * [120] M. Köstinger, P. Wohlhart, P. M. Roth, and H. Bischof, “Annotated facial landmarks in the wild: A large-scale, real-world database for facial landmark localization,” in _2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops)_ , 2011, pp. 2144–2151.
  * [121] X. Zhu, Z. Lei, X. Liu, H. Shi, and S. Z. Li, “Face alignment across large poses: A 3d solution,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 146–155.
  * [122] G. G. Chrysos, E. Antonakos, S. Zafeiriou, and P. Snape, “Offline deformable face tracking in arbitrary videos,” in _2015 IEEE International Conference on Computer Vision Workshop (ICCVW)_ , 2015, pp. 954–962.
  * [123] S. Jin, L. Xu, J. Xu, C. Wang, W. Liu, C. Qian, W. Ouyang, and P. Luo, “Whole-body human pose estimation in the wild,” in _Computer Vision – ECCV 2020_ , A. Vedaldi, H. Bischof, T. Brox, and J.-M. Frahm, Eds. Cham: Springer International Publishing, 2020, pp. 196–214.
  * [124] B. Yang, J. Yan, Z. Lei, and S. Z. Li, “Fine-grained evaluation on face detection in the wild,” in _Automatic Face and Gesture Recognition (FG), 11th IEEE International Conference on_. IEEE, 2015.
  * [125] V. Jain and E. Learned-Miller, “Fddb: A benchmark for face detection in unconstrained settings,” University of Massachusetts, Amherst, Tech. Rep. UM-CS-2010-009, 2010.
  * [126] H. Proenca, S. Filipe, R. Santos, J. Oliveira, and L. A. Alexandre, “The ubiris.v2: A database of visible wavelength iris images captured on-the-move and at-a-distance,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 32, no. 8, pp. 1529–1535, 2010.
  * [127] S. J. Garbin, Y. Shen, I. Schuetz, R. Cavin, G. Hughes, and S. S. Talathi, “Openeds: Open eye dataset,” 2019.
  * [128] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _Computer Vision – ECCV 2014_ , D. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars, Eds. Cham: Springer International Publishing, 2014, pp. 740–755.
  * [129] M. Andriluka, L. Pishchulin, P. Gehler, and B. Schiele, “2d human pose estimation: New benchmark and state of the art analysis,” in _Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition_ , ser. CVPR ’14. USA: IEEE Computer Society, 2014, p. 3686–3693. [Online]. Available: https://doi.org/10.1109/CVPR.2014.471
  * [130] R. A. Güler, N. Neverova, and I. Kokkinos, “Densepose: Dense human pose estimation in the wild,” in _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 7297–7306.
  * [131] S. Johnson and M. Everingham, “Clustered pose and nonlinear appearance models for human pose estimation,” in _Proceedings of the British Machine Vision Conference_. BMVA Press, 2010, pp. 12.1–12.11, doi:10.5244/C.24.12.
  * [132] H. Jhuang, J. Gall, S. Zuffi, C. Schmid, and M. J. Black, “Towards understanding action recognition,” in _2013 IEEE International Conference on Computer Vision_ , 2013, pp. 3192–3199.
  * [133] C. Lassner, J. Romero, M. Kiefel, F. Bogo, M. J. Black, and P. V. Gehler, “Unite the people: Closing the loop between 3d and 2d human representations,” in _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017, pp. 4704–4713.
  * [134] A. Wetzler, R. Slossberg, and R. Kimmel, “Rule of thumb: Deep derotation for improved fingertip detection,” in _Proceedings of the British Machine Vision Conference (BMVC)_. BMVA Press, September 2015, pp. 33.1–33.12. [Online]. Available: https://dx.doi.org/10.5244/C.29.33
  * [135] Y. Zhang, C. Cao, J. Cheng, and H. Lu, “Egogesture: A new dataset and benchmark for egocentric hand gesture recognition,” _IEEE Transactions on Multimedia_ , vol. 20, no. 5, pp. 1038–1050, 2018.
  * [136] P. Molchanov, X. Yang, S. Gupta, K. Kim, S. Tyree, and J. Kautz, “Online detection and classification of dynamic hand gestures with recurrent 3d convolutional neural networks,” in _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016, pp. 4207–4215.
  * [137] O. Köpüklü, A. Gunduz, N. Kose, and G. Rigoll, “Real-time hand gesture detection and classification using convolutional neural networks,” in _2019 14th IEEE International Conference on Automatic Face Gesture Recognition (FG 2019)_ , 2019, pp. 1–8.
  * [138] M. Vandersteegen, W. Reusen, K. V. Beeck, and T. Goedemé, “Low-latency hand gesture recognition with a low resolution thermal imager,” in _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , 2020, pp. 440–449.
  * [139] K. Soomro, A. R. Zamir, and M. Shah, “Ucf101: A dataset of 101 human actions classes from videos in the wild,” 2012.
  * [140] W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan, F. Viola, T. Green, T. Back, P. Natsev, M. Suleyman, and A. Zisserman, “The kinetics human action video dataset,” 2017.
  * [141] H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre, “Hmdb: A large video database for human motion recognition,” in _2011 International Conference on Computer Vision_ , 2011, pp. 2556–2563.
  * [142] F. C. Heilbron, V. Escorcia, B. Ghanem, and J. C. Niebles, “Activitynet: A large-scale video benchmark for human activity understanding,” in _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2015, pp. 961–970.
  * [143] A. Shahroudy, J. Liu, T.-T. Ng, and G. Wang, “Ntu rgb+d: A large scale dataset for 3d human activity analysis,” 2016.
  * [144] C. Schuldt, I. Laptev, and B. Caputo, “Recognizing human actions: a local svm approach,” in _Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004._ , vol. 3, 2004, pp. 32–36 Vol.3.
  * [145] I. Lillo, A. Soto, and J. C. Niebles, “Discriminative hierarchical modeling of spatio-temporally composable human activities,” in _2014 IEEE Conference on Computer Vision and Pattern Recognition_ , 2014, pp. 812–819.
  * [146] H. Zhao, A. Torralba, L. Torresani, and Z. Yan, “Hacs: Human action clips and segments dataset for recognition and temporal localization,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 8668–8678.
  * [147] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: An asr corpus based on public domain audio books,” in _2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2015, pp. 5206–5210.
  * [148] P. Warden, “Speech commands: A dataset for limited-vocabulary speech recognition,” 2018.
  * [149] R. Cattoni, M. A. Di Gangi, L. Bentivogli, M. Negri, and M. Turchi, “Must-c: A multilingual corpus for end-to-end speech translation,” _Computer Speech and Language_ , vol. 66, p. 101155, 2021. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0885230820300887
  * [150] R. Ardila, M. Branson, K. Davis, M. Henretty, M. Kohler, J. Meyer, R. Morais, L. Saunders, F. M. Tyers, and G. Weber, “Common voice: A massively-multilingual speech corpus,” 2019.
  * [151] J. Kahn, M. Rivière, W. Zheng, E. Kharitonov, Q. Xu, P.-E. Mazaré, J. Karadayi, V. Liptchinsky, R. Collobert, C. Fuegen _et al._ , “Libri-light: A benchmark for asr with limited or no supervision,” in _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 7669–7673.
  * [152] D. Wang and X. Zhang, “Thchs-30 : A free chinese speech corpus,” 2015.
  * [153] C. Richey, M. A. Barrios, Z. Armstrong, C. Bartels, H. Franco, M. Graciarena, A. Lawson, M. K. Nandwana, A. Stauffer, J. van Hout, P. Gamble, J. Hetherly, C. Stephenson, and K. Ni, “Voices obscured in complex environmental settings (voices) corpus,” 2018.
  * [154] Z. Chen, T. Yoshioka, L. Lu, T. Zhou, Z. Meng, Y. Luo, J. Wu, X. Xiao, and J. Li, “Continuous speech separation: dataset and analysis,” 2020.
  * [155] W. Havard, L. Besacier, and O. Rosec, “Speech-coco: 600k visually grounded spoken captions aligned to mscoco data set,” _arXiv preprint arXiv:1707.08435_ , 2017.
  * [156] M. Gaham, B. Bouzouia, and N. Achour, _Human-in-the-Loop Cyber-Physical Production Systems Control (HiLCP2sC): A Multi-objective Interactive Framework Proposal_. Cham: Springer International Publishing, 2015, pp. 315–325.
  * [157] B. Hadorn, M. Courant, and B. Hirsbrunner, “Towards human-centered cyber-physical systems: a modeling approach,” Université de Fribourg, Tech. Rep., 2016.
  * [158] S. K. Sowe, E. Simmon, K. Zettsu, F. de Vaulx, and I. Bojanova, “Cyber-physical-human systems: Putting people in the loop,” _IT Professional_ , vol. 18, no. 1, pp. 10–13, 2016.
  * [159] D. S. Nunes, P. Zhang, and J. Sá Silva, “A survey on human-in-the-loop applications towards an internet of all,” _IEEE Communications Surveys Tutorials_ , vol. 17, no. 2, pp. 944–965, 2015.
  * [160] A. Ajoudani, A. M. Zanchettin, S. Ivaldi, A. Albu-Schäffer, K. Kosuge, and O. Khatib, “Progress and prospects of the human–robot collaboration,” _Autonomous Robots_ , vol. 42, no. 5, pp. 957–975, Jun 2018. [Online]. Available: https://doi.org/10.1007/s10514-017-9677-2
  * [161] D. Gorecky, M. Schmitt, M. Loskyll, and D. Zühlke, “Human-machine-interaction in the industry 4.0 era,” in _2014 12th IEEE International Conference on Industrial Informatics (INDIN)_ , 2014, pp. 289–294.
  * [162] H. V. Singh and Q. H. Mahmoud, “Eye-on-hmi: A framework for monitoring human machine interfaces in control rooms,” in _2017 IEEE 30th Canadian Conference on Electrical and Computer Engineering (CCECE)_ , 2017, pp. 1–5.
  * [163] M. Shakil and A. Zoitl, “Towards a modular architecture for industrial hmis,” in _2020 25th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA)_ , vol. 1, 2020, pp. 1267–1270.
  * [164] Z. Wang, J.-J. Han, and T. Miao, “A framebuffer oriented graphical human-machine interaction mechanism for intelligent in-vehicle systems,” in _2019 International Conference on Internet of Things (iThings) and IEEE Green Computing and Communications (GreenCom) and IEEE Cyber, Physical and Social Computing (CPSCom) and IEEE Smart Data (SmartData)_ , 2019, pp. 202–207.
  * [165] N. Pedersen, T. Bojsen, and J. Madsen, “Co-simulation of cyber physical systems with hmi for human in the loop investigations,” in _Proceedings of the Symposium on Theory of Modeling & Simulation_, ser. TMS/DEVS ’17. San Diego, CA, USA: Society for Computer Simulation International, 2017.
  * [166] A. Ivaschenko, A. Khorina, and P. Sitnikov, “Accented visualization by augmented reality for smart manufacturing aplications,” in _2018 IEEE Industrial Cyber-Physical Systems (ICPS)_ , 2018, pp. 519–522.
  * [167] C. M. Lukman Khalid, M. S. Fathi, and Z. Mohamed, “Integration of cyber-physical systems technology with augmented reality in the pre-construction stage,” in _2014 2nd International Conference on Technology, Informatics, Management, Engineering Environment_ , 2014, pp. 151–156.
  * [168] K. Kammerer, R. Pryss, K. Sommer, and M. Reichert, “Towards context-aware process guidance in cyber-physical systems with augmented reality,” in _2018 4th International Workshop on Requirements Engineering for Self-Adaptive, Collaborative, and Cyber Physical Systems (RESACS)_ , 2018, pp. 44–51.
  * [169] M. Lorenz, S. Knopp, and P. Klimant, “Industrial augmented reality: Requirements for an augmented reality maintenance worker support system,” in _2018 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)_ , 2018, pp. 151–153.
  * [170] A. M. Zanchettin, A. Casalino, L. Piroddi, and P. Rocco, “Prediction of human activity patterns for human–robot collaborative assembly tasks,” _IEEE Transactions on Industrial Informatics_ , vol. 15, no. 7, pp. 3934–3942, 2019.
  * [171] M. A. R. Garcia, R. Rojas, L. Gualtieri, E. Rauch, and D. Matt, “A human-in-the-loop cyber-physical system for collaborative assembly in smart manufacturing,” _Procedia CIRP_ , vol. 81, pp. 600–605, 2019, 52nd CIRP Conference on Manufacturing Systems (CMS), Ljubljana, Slovenia, June 12-14, 2019\. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S2212827119304676
  * [172] A. Casalino, C. Messeri, M. Pozzi, A. M. Zanchettin, P. Rocco, and D. Prattichizzo, “Operator awareness in human–robot collaboration through wearable vibrotactile feedback,” _IEEE Robotics and Automation Letters_ , vol. 3, no. 4, pp. 4289–4296, 2018.
  * [173] K. Li, R. Chen, T. Nuchkrua, and S. Boonto, “Dual loop compliant control based on human prediction for physical human-robot interaction,” in _2019 58th Annual Conference of the Society of Instrument and Control Engineers of Japan (SICE)_ , 2019, pp. 459–464.
  * [174] D. Zhao and J. Oh, “Noticing motion patterns: A temporal cnn with a novel convolution operator for human trajectory prediction,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 2, pp. 628–634, 2021.
  * [175] H. C. Ravichandar and A. P. Dani, “Human intention inference using expectation-maximization algorithm with online model learning,” _IEEE Transactions on Automation Science and Engineering_ , vol. 14, no. 2, pp. 855–868, 2017.
  * [176] M. R. Endsley, “Toward a theory of situation awareness in dynamic systems,” _Human Factors_ , vol. 37, no. 1, pp. 32–64, 1995. [Online]. Available: https://doi.org/10.1518/001872095779049543
  * [177] M. Vidulich, C. Dominguez, E. Vogel, and G. McMillan, “Situation awareness: Papers and annotated bibliography,” ARMSTRONG LAB WRIGHT-PATTERSON AFB OH CREW SYSTEMS DIRECTORATE, Tech. Rep., 1994.
  * [178] L. Horváth, “Situation-awareness in model of cyber physical system,” in _2020 IEEE 18th World Symposium on Applied Machine Intelligence and Informatics (SAMI)_ , 2020, pp. 17–22.
  * [179] Z. Yang, T.-H. Li, and W.-Q. Jiang, “Situation awareness for cyber-physical system: A case study of advanced metering infrastructure,” in _2018 IEEE International Conference on Prognostics and Health Management (ICPHM)_ , 2018, pp. 1–6.
  * [180] M. Joo, J. Seo, J. Oh, M. Park, and K. Lee, “Situational awareness framework for cyber crime prevention model in cyber physical system,” in _2018 Tenth International Conference on Ubiquitous and Future Networks (ICUFN)_ , 2018, pp. 837–842.