# Integrating Sensing and Communication in Cellular Networks via NR Sidelink

Dariush Salami, Ramin Hasibi, Stefano Savazzi Tom Michoel, and Stephan Sigg D.
Salami and S. Sigg are with the Department of Communications and Networking,
Aalto University, Finland.
E-mails: {dariush.salami<EMAIL_ADDRESS>Hasibi and T. Michoel are
with the Department of Informatics at the University of Bergen.
E-mails: {ramin.hasibi<EMAIL_ADDRESS>Savazzi is a with the National
Research Council of Italy (CNR), IEIIT institute, Milano, Italy.
E-mail<EMAIL_ADDRESS>
###### Abstract

RF-sensing, the analysis and interpretation of movement or environment-induced
patterns in received electromagnetic signals, has been actively investigated
for more than a decade. Since electromagnetic signals, through cellular
communication systems, are omnipresent, RF sensing has the potential to become
a universal sensing mechanism with applications in smart home, retail,
localization, gesture recognition, intrusion detection, etc. Specifically,
existing cellular network installations might be dual-used for both
communication and sensing. Such communications and sensing convergence is
envisioned for future communication networks. We propose the use of NR-
sidelink direct device-to-device communication to achieve device-initiated,
flexible sensing capabilities in beyond 5G cellular communication systems. In
this article, we specifically investigate a common issue related to sidelink-
based RF-sensing, which is its angle and rotation dependence. In particular,
we discuss transformations of mmWave point-cloud data which achieve rotational
invariance, as well as distributed processing based on such rotational
invariant inputs, at angle and distance diverse devices. To process the
distributed data, we propose a graph based encoder to capture spatio-temporal
features of the data and propose four approaches for multi-angle learning. The
approaches are compared on a newly recorded and openly available dataset
comprising 15 subjects, performing 21 gestures which are recorded from 8
angles.

###### Index Terms:

RF communication and sensing convergence, NR Sidelink, Point cloud, Activity
recognition, Gesture recognition

##  I Introduction

RF convergence describes the use of a common transceiver for both
communication and sensing. In particular, the recent advances in RF-sensing,
such as localization and recognition has demonstrated that ubiquitous cellular
installations may not only enable seamless connectivity, but, utilizing the
same system, it may further seamless localization and environmental perception
capabilities. However, as the available bandwidth is naturally limited [1],
RF-sensing has to share resources with communication systems. In particular,
as shown in [2], resource sharing is more efficient than reserving fixed-size
resources for both communication and sensing. In this regard, modulating
communication symbols with radar waveforms, as well as the utilization of
communication signals for localization and environmental perception have been
investigated [3].

In view of the current standardization efforts, neither of these approaches
were deeply investigated for future communication systems. However, 3GPP has
introduced a mechanism for devices to reserve channel resources in the system:
the LTE/NR sidelink feature (PC5 interface), which has been introduced first
in LTE releases 12 and 13 for device-to-device communication. Specifically, a
user equipment (UE) device may request resources for sidelink communication
from the corresponding gNB. According to the standard, the sidelink
communication then occurs via the Physical Sidelink Shared Channel (PSSCH). We
propose to utilize the sidelink functionality for device-based RF-sensing (cf.
figure 1). In particular, a UE would request sidelink resources from the
respective gNB, and then operates, for instance, an FMCW radar in the assigned
frequency range111Both licensed and unlicensed bands may be allocated for NR
sidelink use.

Figure 1: Integration of sensing capabilities into next generation cellular
communication systems exploiting sidelink device-to-device communication

In addition, we investigate and address a practical issue in RF-sensing
systems, specifically in distributed sidelink-sensing settings, which is
frequently ignored in the literature. The relative orientation of the RF-
interface and the subject (e.g. in gesture sensing) or (moving) object (e.g.
environmental perception) significantly impacts the observed electromagnetic
pattern and hence the recognition accuracy. We demonstrate this impact in an
experimental study with 15 subjects, in which gestures are simultaneously
recorded from 8 different angles. A learning system is proposed which is
capable of recognizing gestures from multiple angles. For this, we employ in a
sidelink-sensing scenario multi-angle learning, four different pooling
mechanisms to weigh the relevant angles, as well as rotationally independent
representations of the received patterns. The system is compared to
traditional RF-sensing approaches and shows superior recognition accuracy.

Our contributions are

  * •

a scheme to achieve RF-convergence in cellular communication systems utilizing
NR sidelink for flexible UE-initiated and CRAN controlled resource allocation.

  * •

four approaches for multi-angle learning to fuse data from distributed
sidelink-based RF-sensors

  * •

a novel graph based encoder to capture spatio-temporal features of point
clouds

  * •

an openly available dataset for a NR sidelink scenarios collected from 15
subjects from 8 angles with more than 25,000 samples

  * •

an open source data collection
platform222https://version.aalto.fi/gitlab/salamid1/mmwave-data-collector for
mmWave radars as well as the source code of the proposed models
333https://version.aalto.fi/gitlab/salamid1/AngleRadar

The proposed system for RF-convergence via NR sidelink will enable
simultaneous communication and sensing and seamlessly integrates into future
cellular communication networks by widening the sidelink communication
capabilities.

Possible applications, for instance, may be the support of ultra-reliable and
low-latency communication (URLLC) through pro-active corrective actions (e.g.
switching the serving gNB ahead of a predicted signal drop). Particularly, the
system may exploit RF sensing for channel prediction [4, 5, 6]. RF-data
provides rich features to predict sudden events (e.g., blockages), which
cannot be done with current, purely channel analytic solutions. Further
examples are ambient assisted living scenarios, gaming, intrusion detection,
as well as elderly/remote care, where pre-installed cellular communication
hardware may be utilized for the recognition of situations, gestures, motion
and mobility.

##  II Related Work

Device-free gesture and activity recognition, by analysing time or frequency
domain patterns of electromagnetic signals, achieves high recognition accuracy
[7]. For instance, the Channel State Information (CSI), phase and Received
Signal Strength (RSS) on channels of multi-antenna wireless interfaces provide
rich information and enable accurate recognition of motion [8, 9]. In
particular, via micro Doppler variations, whole-body motion can be
distinguished [10]. Furthermore, detection of minute movement, such as
respiration, pulse [11] or even emotion [12] may be obtained using Frequency
Modulated Continuous Wave (FMCW) radars. In addition, via RF-holography, the
3D perception of objects [13] is feasible. The technology is capable of joint
motion detection for multiple moving targets [14] by signal processing to
separate motion of these targets from a single stream and by installing
multiple Tx/Rx points.

These impressive results, together with the realization that electromagnetic
signals are omnipresent nowadays through cellular system deployment, explain
the interest in integrating sensing capabilities into communication systems. A
good overview on RF communications and sensing convergence research is given
in [15]. In particular, the authors argue that the wireless medium is limited
and thus has to be shared for both communication and sensing systems. For
instance, DARPA has proposed shared spectrum access for radar and
communications444http://www.darpa.mil/program/shared-spectrum-access-for-
radar-and-communications. They propose cooperative spectrum sharing, in which
information is shared between the communication and the radar system in near
real time, which implies the co-existence of two separate systems. In
contrast, to achieve communication and sensing via a single, integrated
system, several authors have proposed communication that is embedded in
signals transmitted by radar devices (radar-embedded communications) [16, 17,
18, 19]. For instance, [3] propose a dual-use radar-communication multicarrier
waveform, in which different subcarriers are assigned to different subsystems
to achieve RF-convergence. In contrast, the authors of [1] investigate the use
of guard bands of a linear frequency modulated radar waveform for
communications and study how the waveform affects the symbol error rate.
Additionally, they also derive the radar’s probability of detection when the
QPSK RF carriers are injected into the radar signal. A good overview over
recent results on RF convergence is given in [2], together with a proposal for
a joint radar, communications, positioning, navigation and timing system.

In contrast to this previous work, we suggest to utilize the NR sidelink
device-to-device communication functionality that has been first introduced by
3GPP in releases 12 and 13 for LTE. In particular, we suggest that UEs request
resources from the system for RF sensing to instrument mmWave radar sensing on
the granted resources. Hence, spectrum sharing would be dynamically adaptive
to the RF-sensing need, indicated autonomously by UE devices and controlled by
the system to balance communication and sensing needs in any given situation.

Recently, a related concept has been discussed in [20]. The authors propose to
continuously track neighbouring devices by estimating angle of arrival (AoA),
time of arrival (ToA) and received signal strength (RSS) from NR sidelink
reference signals exchanged between the devices. In contrast, we propose to
use the bandwidth granted for NR sidelink communication for mmWave radar
sensing (e.g. FMCW), instead of utilizing the packet-based communication
through the shared and controlled sidelink channels PSCCH and PSSCH.

In our work we conduct an RF-sensing instrumentation utilizing the Texas
Instruments IWR1443555https://www.ti.com/product/IWR1443 sensor, which is a
Frequency-Modulated Continuous Wave (FMCW) –Multiple-Input and Multiple-Output
(MIMO) radar operating in the 77-81 GHz RF band. In particular, the sensor
generates point clouds reflected from objects in the environment to yield a
time-varying signal in an $x$-$y$-$z$ coordinate system [9].

A temporal point-cloud is a sequence of frames through time, each of which
consists of an unordered set of points [21, 22]. Fuelled by commercial
availability of point cloud generating mmWave radars, mmWave radar point cloud
based sensing has been actively investigated for e.g. hand tracking [23],
gesture recognition [24] activity recognition [25], gait recognition [26], or
positioning [27].

Direct point cloud processing has first been achieved by PointNet [28], which
extracts point-wise spatial features and aggregates them through permutation-
invariant pooling operations. To tackle the problem of learning local
structures in PointNet, PointNet++ [29] was proposed. In particular, the
authors have introduced a Set Abstraction (SA) layer to recursively apply a
simplified version of PointNet on the input point cloud capturing both local
and global features. Following a graph based approach, [30] proposed Dynamic
Edge Convolution (DEC) to build graphs by applying a nearest neighbour search
on point clouds and by processing them using Message Passing Neural Networks
(MPNN). Although these models have shown significant improvements on static
point cloud processing including shape classification and semantic
segmentation, they lack the ability to capture temporal dependencies such as
in gestures and movement.

To process dynamic point clouds, a combination of Recurrent Neural Networks
(RNNs) with either 3D-Convolutional Neural Networks (CNNs) or PointNet++
layers has been proposed in [9, 22, 31]. Furthermore, a modified LSTM layer,
[21], was introduced to propagate temporal information while preserving the
spatial structure. All these models, however, expect a point cloud from a
single sensor and are trained with respect to a single unique operating angle
with respect to the observed gesture or movement. Naturally, single angle and
single sensor operations are constrained and constitute only a special case in
realistic instrumentation, where multiple sensors in an environment might
contribute point clouds of a scene from various different angles. In such
case, synchronized operation across sensors is necessary and angles might
deviate from the optimal angle used for training. We propose models which are
resilient to changes in angle and which are also capable to operate on varying
number of input angles for environmental perception.

##  III RF-sensing for cellular communications

RF convergence describes the shared use of the available bandwidth in a
communication system for both communication and sensing. Traditional
approaches have designed signals to either embed communication in
transmissions by radar devices, or have utilized communication signals for
sensing [2]. In both cases, a degradation in performance compared to a
communication-only (sensing-only) system is expected [1].

However, the demand in sensing or communication services differs in its
distribution both temporally and spatially. Especially in cellular systems,
the RF-sensing demand which is short-range, is primarily associated to UEs
while a gNB has a higher communication demand since it connects to multiple
UEs. For efficiency reasons, resources reserved for UE-centered spatially
constrained sensing should not constrain communication or RF-sensing of remote
UEs in the same cell. A mechanism that supports this paradigm is the NR-
sidelink device-to-device communication. In particular, in NR-sidelink, a UE
may request resources for spatially constrained sidelink operation. We propose
to implement RF-sensing utilizing FMCW on the reserved sidelink resources.

In the following, we briefly introduce the specifics of NR sidelink
communication, further propose RF-sensing via sidelink, and discuss angle
dependency in NR sidelink based RF-sensing.

Figure 2: Data processing and aggregation chain for multiple devices in
proximity conducing joint NR-sidelink-based RF sensing

###  III-A NR sidelink (R17)

3GPP introduced the LTE sidelink feature (PC5 interface) in release 12 and 13
for device to device communication with minor involvement of the eNB [32]
(device-to-device (D2D) proximity service (ProSe)). In particular, two or more
UEs in proximity can be known by each other using direct signalling and
reserved resources. To enable the functionality, a new node, ProSe (Proximity
Services) function is introduced of which each PLMN possesses one [33].
Authorization to use direct discovery or direct communication of a UE is done
by this node. The ProSe function provides Radio resource parameters to the UEs
for out of coverage communication. UEs are further capable of direct discovery
(identifying another UE in proximity) via E-UTRA direct radio signals.
Sidelink communication is conducted via the Physical Sidelink Shared Channel
PSSCH (for ProSe application data; same as LTE PDSCH, QPSK & 16QAM), the
Physical Sidelink Control Channel PSCCH (for control information to decode
PSSCH, similar to PDCCH, QPSK), Physical Sidelink Discovery Channel PSDCH and
Physical Sidelink Broadcast Chanel PSBCH (for MIB-SL information). Resource
pools are set of resources that are used for sidelink transmission. Both
control and data information is transmitted using a resource pool. The Radio
resources for direct communication can be requested by the UE autonomously or
given by the gNB. To initialize device-to-device communication, the UE sends
ProSe UE information indication informing the network that it wants to use
direct communication. gNB will in turn give radio resource information in an
RRC message along with SL-RNTI for that UE to decode DCI6663GPP TR 23.725
V16.2.0 (2019-06), 3rd Generation Partnership Project; Technical Specification
Group Services and System Aspects; Study on enhancement of Ultra-Reliable Low-
Latency Communication (URLLC) support in the 5G Core network (5GC)(Release
16). The gNB then assigns the dedicated resources to the UE and sends a grant
using DCI††footnotemark:  scrambled with SL-RNTI and the UE uses this
information to send data in sidelink to UE2 using SCI 0 for control
information before data is sent via PSSCH between a pair of UEs.

3GPP introduced in their Release 15 of the 5G new radio (NR) specification the
Sidelink technology for cellular vehicle-to-everything communication (C-V2X).
In the following release 16, physical layer, protocol and core network
functionalities supporting NR sidelink for Broadcast, Groupcast, and Unicast
were added together with positional reference signal (PRS) and a sounding
reference signal (SRS) 7773GPP TR 22.186 V16.2.0 (2019-06), 3rd Generation
Partnership Project; Technical Specification Group Services and System
Aspects; Enhancement of 3GPP support for V2X scenarios; stage 1 (Release 16).
The 3GPP Release 17 standardization is investigating enhancements to NR
sidelink to support higher reliability for V2V services, power saving features
for pedestrian/vulnerable Road Users, commercial D2D use-cases etc. The NR
Release 16 sidelink design assumes two frequency ranges, up-to 6 GHz and a
Subcarrier Spacing (SCS) of 15 KHz, 30 KHz and 60 KHz, as well as a frequency
range from 6 GHz to 52.6 GHz and with a SCS of 60 KHz and 120 KHz.

The Physical Sidelink Control Channel (PSCCH) carries Sidelink Control
Information (SCI) and is time multiplexed with the associated Physical
Sidelink Shared Channel (PSSCH). Furthermore, a two-stage SCI was introduced,
separating the content of the SCI payload into two parts. The first SCI part
contains sensing information and is broadcast to the surrounding UEs
(containing information related to QoS priority of data, occupied resource
blocks, resource reservation interval etc.), whereas the second SCI
transmission carries information related data decoding of Physical SL Shared
Channel (PSSCH) [34]. Furthermore, higher reliability for unicast and
groupcast transmission is achieved with the introduction of the new Physical
Sidelink Feedback Channel (PSFCH) to carry HARQ feedback in the last OFDM
symbol of a slot. In order to perform efficient link adaptation for unicast
transmissions, feedback of the channel quality information (CQI) is supported
from the receiver UE.

###  III-B NR sidelink-based RF-sensing

NR-sidelink is designed to support D2D communication utilizing a battery of
the usual broadcast, control, shared and feedback channels (PSBCH, PSCCH,
PSSCH) together with a number of reference signals (primary (S-PSS), secondary
(S-SSS), phase tracking (PF-RS), and CSI-RS). NR sidelink is expected to work
in both licensed and unlicensed bands and operates out-of-coverage, in partial
coverage or in-coverage. We propose a new mode of operation for NR-sidelink:
environmental sensing. In particular, a UE would request resources for RF-
sensing from the gNB or CRAN, which in turn informs the UE about the assigned
resource block (steps 1 and 2 in figure 3). The UE may then utilize these
resources, e.g. operating as a FMCW mmWave radar to sense an environment and
gestures (steps a-d in the figure). If multiple UEs in proximity participate
in the sensing of an environmental situation, the initiating UE first
establishes a sidelink communication channel among the UEs, and shares
information on the allocated channel resources (steps 3 and 4). In such case,
the features extracted are to be aggregated across these devices (step c).
After the sensing operation is completed, the UE informs the gNB that the
resources may be released again (cf. steps 5-7 in figure 3).

Figure 3: RF-convergence achieved via the allocation of sidelink resources and
FMCW mmWave sensing in a cellular communication system. In the case of
multiple participating UEs for the sensing (indicated in blue in the figure),
feature aggregation and communication across UEs is achieved via a sidelink
communication channel.

The environmental sensing processing chain for each UE participating in the
sidelink sensing is illustrated in figure 2. In the case of a single UE, the
device extracts the data, processes and segments it and utilize the extracted
features for the classification. When multiple UEs collaborate (in blue in the
figure) by jointly sensing an environmental situation, each device will
extract the data, preprocess, segment it and extract features from the local
data stream. These features are then shared via the sidelink with the
initiating UE, which in turn aggregates the features and infers the respective
class (cf. figure 2).

###  III-C Angle dependency in RF-sensing

RF-sensing collaboration between multiple devices in proximity touches a
fundamental problem in RF-sensing. The observed patterns which originate from
reflections off the same moving object differ among sensing UEs conditioned on
their distance, angle and translation relative to the object. A gesture
recorded from a $0^{\circ}$ angle might have a very different shape when
recorded e.g. from a $90^{\circ}$ angle as illustrated in figure 4.

Figure 4: Illustration of the fusion of multi-angle point-cloud data. Clapping
gestures recorded with the mmWave radar simultaneously from different angles
are merged into a join representation of the gesture.

In order to mitigate this issue, the data captured at different angles is to
be fused in order to achieve a global joint representation of the data that
may be fed into the classification algorithm. For this, we have experimented
with rotation-invariant representations of the data, e.g. by applying
principal component analysis on the data [35] and also with mapping the data
into alternative higher dimensional representations [36]. In particular,
L1-norm maximizing algorithms, such as [37, 38] generally achieve good results
(L2-norm based algorithms are more susceptible to noise and outliers in the
point cloud data). However, we observed that, for the data set we utilized,
the large number of points reflected by the torso of the person, and which is
similar for all gestures conducted, causes significant confusion among classes
when applying PCA to achieve rotational invariance. While we are currently
working on a solution for this problem, we have, for this article, instead
implemented the feature fusion within a neural network architecture as
described in section IV-A and in section IV-B. When multiple UEs collaborate
by jointly sensing the same situation, the neural network can be also trained,
or updated periodically, using a distributed approach that leverages the NR-
sidelink, namely the Federated Learning, described in section V.

##  IV Instrumentation

(a) Crossing chrip interference.

(b) Parallel chirp interference.

Figure 5: Crossing and parallel chirp interference between transmitted chirp
(victim) and aggressor chirp.

Although using multiple NR sidelink-based radars for sensing the environment
can potentially address the shadowing effect and the low-resolution of the
radar in z-axis [9], it also gives rise to issues due to the interference
between radars including noise floor degradation, blind-spots at certain
ranges or directions, as well as ghost objects. In this section, we discuss
two different types of interference that can occur with FMCW radars: crossing
chirp interference and parallel chirp interference. As shown in Fig. 5.a,
crossing interference occurs when one chirp (referred to as aggressor in the
following) crosses the chirp of another chirp (referred to as victim, since it
is falling victim to the interference of the aggressor). This type of
interference typically increases the noise floor resulting in a reduction in
Signal to Noise Ratio (SNR) of the real targets thereby affecting detection
and creating momentary blind-spots. The glitch duration in crossing
interference is given by:

$glitch\\_duration=\frac{bandwidth}{|slope_{aggressor}-slope_{victim}|}$ (1)

According to Equation 1, the glitch duration for two crossing interferers is
typically low and affects few samples.

The parallel interference is shown in Fig. 5.b. This type of interference
occurs when the aggressor chirp and the victim chirp have the same slope. If
the delay in the start of a chirp between different radars is within
microsecond, the aggressor chirp will be within the bandwidth of the entire
chirp of the victim. This type of interference results in ghost objects at
random distance with random velocity that do not exist in the environment but
are detected by the radar. Since such interference will occur only when the NR
sidelink operating radars start nearly simultaneously, the probability of it
is small. During the experiments we employed the built-in capability of the
radar in interference detection to avoid any interference issues in the
recorded dataset.

###  IV-A Addressing angle dependency in NR sidelink based RF-sensing

Figure 6: Schematic of the proposed encoder. (a) Point cloud generated by the
radar for a gesture; (b) The graph representation of the gesture after
applying the proposed K-NN to reflect the temporal dependency; (c) Edge
features for each incident edge at a central point $i$ are calculated in this
step; (d) The representation for all the nodes are calculated by applying an
aggregation function over the edge features

(a) Each angle representation vector $R$ is fed into the same $\phi$ function;
outputs are aggregated through different pooling mechanisms

(b) Different $\phi$ functions are applied on each angle representation $R$;
output probability is the normalized sum of the per class score

Figure 7: Schematic view of the two proposed approaches: (a) orientation
independent and (b) orientation tracking

For each angle $a$ in the set of all angles of the NR-sidelink based radars, a
motion point cloud $X_{a}=\\{x_{1},\dots,x_{n}\\}$ is available in which each
point has a feature set of $x_{i}=\\{f_{i}^{1},...,f_{i}^{F}\\}$. For each of
the points in $X_{a}$, a frame number $f_{i}^{s}\in x_{i}$ is part of this
feature set which defines the temporal aspect of the motion point cloud. To
process the motion point cloud corresponding to a radar, a graph is generated
and processed to obtain the representation of a gesture for each angle. To
capture the temporal evolution of the gesture, the graph is generated over the
nearest neighbours in temporal (frame-wise) dimension so that each point is
connected to the closest points in the Euclidean space from the previous
frames. To achieve this, a masking scheme is utilized in which the masked set
of points $\mathcal{F}_{x_{i}}$ for $x_{i}$ is obtained through:

$\mathcal{F}_{x_{i}}=\\{x_{j}:\forall x_{j}\in X,f_{j}^{s}>f_{i}^{s}\\}.$ (2)

Moreover, the distance between two points is defined as the Euclidean distance
of all the corresponding features of points including $f_{i}^{s}$ and is
obtained through:

$D_{x_{i},x_{j}}=\begin{cases}||x_{i}-x_{j}||:x_{i},x_{j}\in X_{a},&\text{if
}x_{j}\notin\mathcal{F}_{x_{i}},\\\ \infty,&\text{otherwise},\end{cases}$ (3)

where $D_{x_{i},x_{j}}$ and $||.||$ denote the distance between $x_{i}$ and
$x_{j}$ and the L2 norm, respectively. Finally, for each point K nearest
points are chosen as neighbours in the graph according to $D_{x_{i},x_{j}}$,
resulting in a Temporal graph
$\mathcal{G}_{a}=(\mathcal{V}_{a},\mathcal{E}_{a})$ where
$\mathcal{E}\subseteq X_{a}\times X_{a}$ is the set of directed edges between
each point and its neighbours.

To process the generated graph, an Edge Convolution Network (ECN) [39] is
applied and a vector of gesture representations is gathered. In the ECN, each
convolution layer follows a message passing algorithm, in which features of
each point in the graph gradually propagate to its neighbours as messages and
the incoming messages for each point are aggregated with the features of the
point itself using learnable, differentiable functions. In this network, in
each convolution layer, the hidden representation of each point is updated
through:

$\displaystyle h_{i}^{0}$ $\displaystyle=x_{i}\setminus\\{f_{i}^{s}\\},$ (4)
$\displaystyle h_{i}^{l}$
$\displaystyle=\underset{j:(i,j)\in\mathcal{E}}{\Gamma}M_{\theta}(h_{i},h_{j}-h_{i}),$

in which, $h_{i}^{l}$ is the hidden representation of point $i$ in convolution
layer $l$, $\setminus$ is the set subtraction operator, message function
$M_{\theta}:\mathbb{R}^{F}\times\mathbb{R}^{F}\rightarrow\mathbb{R}^{F^{\prime}}$
is an Multi Layer Perceptron function with learnable weight set of $\theta$,
$\Gamma$ is a channel-wise symmetric max pooling function [40] applied on the
messages of the edge emanating from each neighbor. To obtain the
representation vector of each angle $a$ motion point cloud $\mathcal{R}_{a}$,
a symmetric max pooling function of $\Gamma$ is applied on the hidden
representation of all points in $X_{a}$ as:

$\mathcal{R}_{a}=\underset{i\in\mathcal{V}}{\Gamma}h_{i}.$ (5)

###  IV-B Rotation-resilient Classification of gestures

The set of all angle motion point cloud representation vectors is denoted by
$\mathbb{G}=\bigcup_{a\in\mathcal{A}}R_{a}$. In order to predict the gesture
from $\mathbb{G}$, two different approaches of angle invariant and user
orientation tracking are considered. The pipeline of each is demonstrated in
Fig. 7. In the first approach (Fig. 7(a)), a set of permutation invariant
functions with respect to elements of $R_{a}$ are used to process $\mathbb{G}$
and to predict the gesture label. In the second method (Fig. 7(b)), for each
vector, a set of learnable functions is considered. The details of each method
are presented in the following.

####  IV-B1 Angle invariant prediction

For this approach, each $\mathcal{R}_{a}$ is fed into a function that
calculates the embedding of the vector independent of others and the output of
embeddings are aggregated through a pooling mechanism $\gamma$. This method is
commonly used to process sets of vectors [41] and is achieved via

$P=\rho[\gamma(\phi(R_{1}),\phi(R_{2}),\dots,\phi(R_{m}))].$ (6)

In equation 6, $P$ is the probability of each class, $\rho$ and $\phi$ are
learnable functions. For our method, we employ 1-dimensional convolution
layers for $\phi$ and MLPs followed by a Softmax function for $\rho$. For the
pooling operator, we provide a choice between different pooling methods of Max
pool [40], Attention pool [41], and element wise summing referred to as Vote
pool. Each of the pooling operators result in a different prediction model
that has different performances when it comes to processing the gesture set
$\mathbb{G}$. As opposed to the traditional maxpooling method which does not
contain any learnable weights, the Attention pooling operator, aggregates the
angle representation vectors into a single vector using learnable
differentiable weights that optimizes the final gesture representation through
the multi-head self-attention mechanism [42]. In the case of Vote pool, $\phi$
is replaced with a multi layered perceptron function for which the size of the
last layer is equal to the number of gesture classes and $\rho$ is the Softmax
function. The advantage of this method is that we do not need to keep track of
the orientation of the user with respect to NR sidelink based radars in case
the user changes its position with respect to the surrounding sidelink
operating radars.

####  IV-B2 User orientation tracking prediction

In this method, each $\mathcal{R}_{a}$ is fed into a separate function to
embed calculation which are specific to each angle $a$. The output of each
function is then a separate prediction for the label of a gesture. The final
prediction of the gesture is obtained through a voting mechanism between all
the function outputs. This method follows the formula

$P=\rho[\Sigma(\phi_{1}(R_{1}),\phi_{2}(R_{2}),\dots,\phi_{m}(R_{m})].$ (7)

In equation 7, $\Sigma$ is an element-wise summation function and $\rho$ is
the normalization function for probability calculation. In this approach, we
need to know which radar is located at which angle with respect to the user to
feed the input motion point cloud to the corresponding $\phi_{a}$. For the
$\phi$ functions we employ MLPs followed by a Softmax function to predict each
class probability.

##  V Learning, processing and data fusion in distributed scenarios

In this section, distributed processing and machine learning (ML) tools are
proposed to leverage the sidelink resources for rotation-resilient gesture
classification. In particular, we assume that the radars are equipped with a
low-power processing unit that supports gradient-based neural network model
optimization [43] as well as real-time data fusion and classification. Radars
can be randomly distributed in the space however, to simplify the reasoning,
we assume that these are deployed according to a ring topology as depicted in
Fig 8(a).

Conventional machine learning systems fuse and process the motion point clouds
of each radar on a data center, typically colocated with the gNB. However,
such centralized processing compromises data privacy, lacks scalability and
often requires an intensive use of the uplink radio channel, for moving raw
datasets. A distributed policy based on a federated learning (Federated
Learning) tool is explored that leverages device-to-device sidelink
communications. Federated Learning [43] is a recently proposed distributed
machine learning paradigm that allows the parameters of a Machine Learning
model to be collectively optimized across several resource-constrained
wireless devices [44] equipped with low-power tensor processing units. The
proposed Federated Learning system lets the radar devices act as local
learners, beside point cloud data producers. The learnable model parameters
described in Section IV (the MLP function $\theta$ and the classifier
$\phi(.)$) are first optimized locally from training data. Next, the radars
mutually exchange the local models using the sidelink resources. The NR
sidelink operating radars in-turn improve the local parameters by fusing the
received contributions. This procedure continues for a new optimization round
until the model satisfies a target accuracy using validation data. Federated
Learning is privacy-preserving by design, as it keeps the raw motion point
clouds $X_{a}$ on the radar devices [43]. It also unloads the uplink cellular
link and obviates the need for a data center.

Targeting rotation-resilient classification of gestures, some considerations
on Federated Learning implementation are required. In user orientation
tracking, the learned functions are location specific, namely specific to the
radar relative position (angle $a$): each radar thus autonomously learns the
local angle representation and the classifier parameters $\phi_{i}(.)$ using
training data. For angle invariant prediction, the model parameters $\phi(.)$
and $\gamma$ in eq. (6) are learned collectively via Federated Learning: on
each round, the fusion of the received models can be orchestrated by the
initiating UE and implemented via distributed weighted averaging [45].
Federated Learning can be used both for initial training of model parameters
and for periodic retraining or update. In what follows, considering the
complexity of the model, we adopt the latter approach.

##  VI Evaluation

In this section, we evaluate the performance of the proposed models in terms
of recognition accuracy under various conditions. The evaluation metrics we
use are balanced average accuracy and Area Under the ROC Curve (AUC). Balanced
average accuracy is defined as the average of the recall obtained on each
class. AUC is equal to the probability that a classifier will rank a randomly
chosen positive instance higher than a randomly chosen negative one which in
turn quantifies the discriminatory power of the classifier.

###  VI-A Data collection

(a) positioning of the radars with respect to the participant

(b) actual environment with a participant performing a gesture and radars
recording

Figure 8: Experimental setup for data collection.

We utilize mmWave FMCW radars i.e. IWR1443 for the experiments. These radars
transmit a sinusoidal signal with linearly increasing frequency, called chirp,
and process the reflected signal to extract information like range, velocity,
and the angle of the targets in the environment. The starting frequency of the
chirps is 77 GHz and the final, highest frequency is 81 GHz, resulting in a 4
GHz bandwidth. The evaluation kit produces point clouds from dense tensors
through a signal processing pipeline [9].

Following the same gesture set in [9], we collected 21 classes of gestures
shown in Fig. 12 from 15 subjects repeating each gesture 10 times. We
installed 8 radars around the participant in a 1.5m distance. The positioning
of the radars with respect to the participant is shown in Fig. 8(a) and Fig.
8(b). We recruited 15 participants (7 female) aged 19–50, weight 50–105 kg,
and height 1.55–2 m for data collection. Participants were unpaid volunteers
recruited via mailing lists, external advertisements, and flyers distributed
among several of our university’s departments. Before starting the data
collection process, the experimenter demonstrated the correct way of
performing gestures through pre-recorded videos. Moreover, while recording,
the experimenter verified gestures by observing a visualized point cloud on
the screen. In case the gesture was performed incorrectly, the participant was
asked to repeat the gesture. To collect the data from different angles, as
shown in Fig. 8(b) we connected 8 radars to a laptop through USB ports. The
final dataset consists of 25,200 gesture samples.

To receive the point cloud from each radar and save them to files, we
developed an open source software called mmWave Data Collector which supports
multiple IWR1443 and AWR1642 radars. mmWave Data Collector uses Robot
Operating System (ROS)888https://www.ros.org/ extending its functionalities to
support more than one sensor simultaneously.

For the train/test data split we follow a population dependent cross
validation strategy to answer the question that how well the models perform
recognizing gestures from unseen users. We use 10 users for train/validation
and 3 users for inference purpose.

###  VI-B Data Preprocessing

Since we record each gesture for 3 seconds, they have different different
number of frames. However, the baseline models require a fixed number of
frames and points in each frame. After receiving moving point clouds from the
radar, we divide the points in such a way that we have the same number of
frames. After this, we perform point re-sampling on each frame to fix the
number of points in frames for all samples. To fix the number of frames, we
empirically distribute the points into 32 frames (for a point cloud with $n$
points, the first $n/32$ points are considered as the first frame, the second
$n/32$ are considered as the second frame, and so on). Moreover, to fix the
number of points in each frame while preserving the spatial structure, we
utilize the density-based re-sampling method introduced in [46]. Assume $n/32$
as the desired number of points in each frame. To down-sample the points,
$\mathcal{K}$-means algorithm is used by setting $\mathcal{K}$ equal to $n/32$
and selecting the centroids of the clusters as the points in the frame. To up-
sample the points, Agglomerative Hierarchical Clustering (AHC) is applied
iteratively and the centroids of the clusters are added as new points to the
frame until we have the desired number of points. We empirically set the
number of points in each frame to 32.

###  VI-C Data Augmentation

To increase the generalizability of the models (see section IV-A), we apply
five data augmentation techniques on each batch during the training phase.
Random translation of the gesture up-to 10cm, random scaling between 0.8 and
1.25, random jittering (point-wise translation) based on a Gaussian
distribution with $\mu=0$ and $\sigma=0.01$, random clipping of 0.03m, and
random shuffling of points in each frame to preserve the spatial and temporal
dependencies while changing the input representation are the five data
augmentation techniques applied on-the-fly.

###  VI-D Model Implementation

The proposed models are implemented using PyTorch [47] and PyTorch Geometric
[48]. For initial training of the models, including baselines and the proposed
models except for Pantomime and PointGest, we used a server with 64GB of RAM
and a Tesla V100 16GB GPU. Federated Learning, discussed previously, can be
also used to distribute the training tasks on the radar devices sharing the
NR-sidelink. Since Pantomime and PointGest are computationally expensive, we
used a Tesla V100 32GB GPU to train them.

During the training phase, we also utilize an early stopping strategy with a
patience of 100 epochs to avoid over-fitting of the models. During training,
if no improvement is observed on the validation set in terms of cost function
within the patience period, the training is stopped and the best model is
saved for the inference phase. We use negative log likelihood between softmax
followed by logarithm of class scores and the ground-truth labels as the loss
function [49]. Adam Optimizer [50] with a step-decay strategy to decrease
learning rate is used to optimize the loss function:

$L_{r}=L_{init}\cdot d_{r}^{\lfloor\frac{e}{e_{r}}\rfloor}$ (8)

In equation 8, $L_{r}$ is the learning rate used at each epoch, $L_{init}$ is
the initial value of the learning rate, $d_{r}$ is the drop rate after every
$e_{r}$ epochs, $e$ is the current epoch and $\lfloor\cdot\rfloor$ is the
floor operator. In our setup $L_{init}$ is $0.001$, $d_{r}$ is $0.5$, and
$e_{r}$ is $80$.

###  VI-E Classification Results

In Table I, the performance of our proposed models (Max Pool, Attention Pool,
Vote Pool, Orientation Tracking) is compared to the baseline models PointNet++
[29], DEC [39], PointLSTM [21], PointGest [22], and Pantomime [9]. In
PointNet++ and DEC, the data are combined through time and angle resulting in
a single point cloud representing the whole gesture from different angles
since they are designed to process static point clouds. For PointLSTM,
PointGest, and Pantomime, the four dimensional (4D; x-y-z and time) point
clouds from different angles are combined frame-wise resulting in a temporal
4D point cloud representing the gesture from different angles since the models
are able to process 4D point clouds. In this section, we assume that all
angles are available in the inference phase and the orientation of the target
w.r.t. the radars is known. As illustrated in Table I, two of our models,
attention pool and orientation tracking, outperform state-of-the-art in terms
of both average accuracy and AUC. The orientation tracking approach achieves
an accuracy and AUC of 100.

Model | Acc. | AUC
---|---|---
Pointnet++ | 82.51 | 98.21
Dynamic Edge Convolution (DEC) | 96.20 | 99.99
PointLSTM | 96.67 | 99.95
PointGest | 95.45 | 99.93
Pantomime | 98.63 | 99.97
Max pool(ours) | 95.40 | 99.83
Attention pool(ours) | 98.73 | 100
Vote pool(ours) | 98.41 | 99.73
Orientation tracking(ours) | 100 | 100
TABLE I: Comparison with the state of the art on the collected dataset when
all the angles are available in the inference phase. The best results per
column are denoted in bold typeface.

###  VI-F Angle Drop-out Results

Figure 9: The performance of the proposed models compared to the baselines
when not all the angles are available in the inference phase. Different number
of angles are randomly removed in the inference phase.

In realistic scenarios, we might not have all angles available in the
inference phase. In this section, we evaluate the performance of our proposed
models as well as the baselines when different number of angles are available
in the inference phase (from 1 to 7). For each batch (a set of 32 gestures),
we randomly remove the specified number of angles and evaluate the performance
of the models. We repeat each setting 10 times and report the average accuracy
and standard deviation of each model in Fig. 9. First, as we decrease the
number of available angles, the standard deviation for all the models
increases since not all angles have the same amount of information (see
section VI-I). Second, the orientation tracking approach achieves a notable
accuracy even for recognizing gestures using a single angle compared to the
rest of the models. The drop in the accuracy for the orientation tracking
model when we go from 8 angles to 1 angle is only 22% while for rest of the
models it is from 55% to 90%. Moreover, for extreme settings in which we have
only one or two angles available, all the proposed models outperform the
baselines since the baseline models are not capable of taking into account the
angles. This observation suggests that dedicating an encoder with an
independent set of weights for each angle increases the generalizability of
the model for settings with varying number of available radars (angles).
However, this improvement comes with a cost of requiring knowledge of the
target’s orientation with respect to the NR sidelink operating radars.

###  VI-G Paired Angle Recognition Results

In realistic scenarios, a person is not surrounded by 8 RF sensors as in our
experimental setting and indeed, the actual angle of a sidelink operating
radar with respect to the person performing the gesture might be random and
arbitrary. This means that data is produced only by a few selected angles. For
the recognition of gestures, we show that the orientation tracking approach is
significantly better than the base line models (see section VI-F). In
particular, we pair angles, ($0^{\circ}$, $45^{\circ}$), ($90^{\circ}$,
$135^{\circ}$), ($180^{\circ}$, $225^{\circ}$), and ($270^{\circ}$,
$315^{\circ}$), to evaluate the models when data from only two deterministic
angles are available. All models were trained on data from all angles while in
the inference phase, the data was provided only from one of these angle pairs.
As shown in Fig. 10, the baseline models fail to generalize on a limited
number of angles suggesting that they overfit the train data where all the
angles are available. However, the proposed methods attention pool, vote pool,
and orientation tracking outperform the baselines in all the four combination
of angles. As we mentioned in section VI-F, the orientation tracking approach
is resilient to the number of available angles in the inference phase
suggesting that it is able to generalize on various number of angles.

|  |  |  |
---|---|---|---|---
(a) PointNet++ | (b) DEC | (c) PointLSTM | (d) PointGest | (e) Pantomime
|  |  |  |
(f) Max Pool | (g) Attention Pool | (h) Vote Pool | (i) Orientation Tracking |

Figure 10: The performance of proposed models compared to the baseline models
when we have different pair of angles in the inference phase. The metric shown
in these plots is the average accuracy. For each model, we evaluate the
performance for four different pairs of ($0^{\circ}$, $45^{\circ}$),
($90^{\circ}$, $135^{\circ}$), ($180^{\circ}$, $225^{\circ}$), and
($270^{\circ}$, $315^{\circ}$).

.

###  VI-H Angle Permutation Results

Figure 11: The performance of the orientation tracking approach when the
orientation of the user is unknown. We feed data from specified number of
angles to other encoders instead of their dedicated ones.

To evaluate the performance of the orientation tracking approach when the
orientation of the subject is unknown in the inference phase, we permute
different number of angles. As shown in Fig. 11, as we increase the number of
angles for which the data is not fed to the dedicated encoder, the standard
deviation of the average accuracy increases. Moreover, as expected, the
accuracy decreases by increasing the number of swapped angles. As a result, it
is crucial for orientation tracking approach to know the orientation of the
user with respect to the NR sidelink operating radars.

###  VI-I Angle Importance

Figure 12: Gesture set used in the experiments: (a) ‘lateral raise’, (b) ‘two-
hand lateral-raise’, (c) ‘two-hand lateral-to-front’, (d) ‘lateral-to-front’,
(e) ‘two-hand inward circles’ (f) ‘left-arm circle’, (g) ‘two-hand outward
circles’, (h) ‘right-arm circle’, (i) ‘lift’, (j) ‘pull’, (k) ‘two-hand pull’,
(l) ‘push’, (m) ‘two-hand push’, (n) ‘push-down’, (o) ‘swipe right’, (p)
‘swipe left’, (q) ‘throw’, (r) ‘two-hand throw’, (s) ‘circle counter-
clockwise’. (t) ‘circle clockwise’, (u) ‘arms swing’. In right side of each
gesture, the importance of radars for the orientation tracking (the best)
model is shown in a polar chart.

Data coming from sidelink operating radars in different angles, differ even
for the same gesture conducted due to the orientation change and the shadowing
effect from a participant’s body. In Fig. 12, the importance of each angle for
each gesture in the orientation tracking approach is shown in a polar chart.
For most gestures, the radars in front ($0^{\circ}$, $45^{\circ}$, and
$315^{\circ}$) have the highest impact while the radar at the back
($180^{\circ}$) has the least importance. However, for few gestures like (q),
(r), and (u) where the hands are visible from a back view, the impact of the
radar at $180^{\circ}$ increases. Moreover, for the gestures that happen on
one side of the body e.g. (d) and (n), the radars on the same side are of
higher importance compared to the radars on the other side.

##  VII Conclusion

We have proposed a mechanism for RF-convergence in cellular communication
systems. In particular, we suggest to integrate RF-sensing with NR sidelink
device-to-device communication, which is since release 12 part of the 3GPP
cellular communication standards. We specifically investigated a common issue
related to NR sidelink based RF-sensing, which is its angle and rotation
dependence. In particular, we discussed transformations of mmWave point-cloud
data which achieve rotational invariance, as well as distributed processing
based on such rotational invariant inputs at distributed, angle and distance
diverse devices. Further, and to process the distributed data, we proposed a
graph based encoder to capture spatio-temporal features of the data as well as
four approaches for multi-angle learning. The approaches are compared on a
newly recorded and openly available dataset comprising 15 subjects, performing
21 gestures which are recorded from 8 angles. We were able to show that our
data aggregation and processing toolchain outperforms the state-of-the-art
point-cloud based gesture recognition approaches for angle-diverse gesture
recordings.

## Acknowledgment

Part of the calculations presented above were performed using computer
resources within the Aalto University School of Science “Science-IT” project.

This project has received funding from the European Union’s Horizon 2020
research and innovation programme under the Marie Skłodowska-Curie Grant
agreement No. 813999. We further appreciate partial funding through the
ERANET-COFUND (H2020) CHIST-ERA III project RadioSense.

## References

  * [1] J. C. Rohde and R. A. Romero, “Detection performance of embedded qpsk onto lfm waveform guard bands for rf convergence,” in _2021 IEEE Radar Conference (RadarConf21)_. IEEE, 2021, pp. 1–6.
  * [2] A. Herschfelt, A. Chiriyath, D. W. Bliss, C. D. Richmond, U. Mitra, and S. D. Blunt, “Vehicular rf convergence: Simultaneous radar, communications, and pnt for urban air mobility and automotive applications,” in _2020 IEEE Radar Conference (RadarConf20)_. IEEE, 2020, pp. 1–6.
  * [3] M. Bică and V. Koivunen, “Multicarrier radar-communications waveform design for rf convergence and coexistence,” in _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2019, pp. 7780–7784.
  * [4] W. Chen, K. Niu, D. Wu, D. Zhao, L. Wang, and D. Zhang, “A contactless gesture interaction system using lte (4g) signals,” in _Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers_ , 2019, pp. 260–263.
  * [5] W. Chen, K. Niu, D. Zhao, R. Zheng, D. Wu, W. Wang, L. Wang, and D. Zhang, “Robust dynamic hand gesture interaction using lte terminals,” in _2020 19th ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN)_. IEEE, 2020, pp. 109–120.
  * [6] S. Savazzi, S. Sigg, F. Vicentini, S. Kianoush, and R. Findling, “On the use of stray wireless signals for sensing: A look beyond 5g for the next generation of industry,” _Computer_ , vol. 52, no. 7, pp. 25–36, 2019.
  * [7] S. Savazzi, S. Sigg, M. Nicoli, V. Rampa, S. Kianoush, and U. Spagnolini, “Device-free radio vision for assisted living: Leveraging wireless channel quality information for human sensing,” _IEEE Signal Processing Magazine_ , vol. 33, no. 2, pp. 45–58, 2016.
  * [8] X. Ma, Y. Zhao, L. Zhang, Q. Gao, M. Pan, and J. Wang, “Practical device-free gesture recognition using wifi signals based on metalearning,” _IEEE Transactions on Industrial Informatics_ , vol. 16, no. 1, pp. 228–237, 2019.
  * [9] S. Palipana, D. Salami, L. A. Leiva, and S. Sigg, “Pantomime: Mid-air gesture recognition with sparse millimeter-wave radar point clouds,” _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ , vol. 5, no. 1, pp. 1–27, 2021.
  * [10] S. Skaria, A. Al-Hourani, M. Lech, and R. J. Evans, “Hand-gesture recognition using two-antenna doppler radar with deep convolutional neural networks,” _IEEE Sensors Journal_ , vol. 19, no. 8, pp. 3041–3048, 2019.
  * [11] X. Wang, C. Yang, and S. Mao, “Resilient respiration rate monitoring with realtime bimodal csi data,” _IEEE Sensors Journal_ , vol. 20, no. 17, pp. 10 187–10 198, 2020.
  * [12] M. Raja, A. Exler, S. Hemminki, S. Konomi, S. Sigg, and S. Inoue, “Towards pervasive geospatial affect perception,” _GeoInformatica_ , vol. 22, no. 1, pp. 143–169, 2018.
  * [13] P. M. Holl and F. Reinhard, “Holography of wi-fi radiation,” _Physical review letters_ , vol. 118, no. 18, p. 183901, 2017.
  * [14] M. Zhao, T. Li, M. Abu Alsheikh, Y. Tian, H. Zhao, A. Torralba, and D. Katabi, “Through-wall human pose estimation using radio signals,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 7356–7365.
  * [15] B. Paul, A. R. Chiriyath, and D. W. Bliss, “Survey of rf communications and sensing convergence research,” _IEEE Access_ , vol. 5, pp. 252–270, 2016\.
  * [16] S. D. Blunt, P. Yatham, and J. Stiles, “Intrapulse radar-embedded communications,” _IEEE Transactions on Aerospace and Electronic Systems_ , vol. 46, no. 3, pp. 1185–1200, 2010.
  * [17] M. L. Parmentar and R. A. Romero, “Opportunistic techniques to inject ofdm subcarriers into navigational radar spectral guard bands and mainlobe for spectrum sharing,” in _2019 IEEE Radar Conference (RadarConf)_. IEEE, 2019, pp. 1–6.
  * [18] J. G. Metcalf, C. Sahin, and S. D. Blunt, “Impact of adjacent/overlapping communication waveform design within a radar spectrum sharing context,” in _2020 IEEE International Radar Conference (RADAR)_. IEEE, 2020, pp. 472–477.
  * [19] C. Y. Liu and R. A. Romero, “Deep neural network detection for pulsed radar-embedded m-psk communications,” in _2020 17th European Radar Conference (EuRAD)_. IEEE, 2021, pp. 238–241.
  * [20] Y. Lu, J. Talvitie, M. Koivisto, E. Rastorgueva-Foi, T. Levanen, E. S. Lohan, and M. Valkama, “Joint positioning and tracking via nr sidelink in 5g-empowered industrial iot,” _arXiv preprint arXiv:2101.06003_ , 2021.
  * [21] Y. Min, Y. Zhang, X. Chai, and X. Chen, “An efficient pointlstm for point clouds based gesture recognition,” in _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 5760–5769.
  * [22] D. Salami, S. Palipana, M. Kodali, and S. Sigg, “Motion pattern recognition in 4d point clouds,” in _2020 IEEE 30th International Workshop on Machine Learning for Signal Processing (MLSP)_. IEEE, 2020, pp. 1–6.
  * [23] Z. Dong, F. Li, J. Ying, and K. Pahlavan, “A model-based rf hand motion detection system for shadowing scenarios,” _IEEE Access_ , vol. 8, pp. 115 662–115 672, 2020.
  * [24] H. Liu, Y. Wang, A. Zhou, H. He, W. Wang, K. Wang, P. Pan, Y. Lu, L. Liu, and H. Ma, “Real-time arm gesture recognition in smart home scenarios via millimeter wave sensing,” _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ , vol. 4, no. 4, pp. 1–28, 2020\.
  * [25] A. D. Singh, S. S. Sandha, L. Garcia, and M. Srivastava, “Radhar: Human activity recognition from point clouds generated through a millimeter-wave radar,” in _Proceedings of the 3rd ACM Workshop on Millimeter-wave Networks and Sensing Systems_ , 2019, pp. 51–56.
  * [26] Z. Meng, S. Fu, J. Yan, H. Liang, A. Zhou, S. Zhu, H. Ma, J. Liu, and N. Yang, “Gait recognition for co-existing multiple people using millimeter wave sensing,” in _In Proc. of AAAI_ , vol. 34, no. 01, 2020, pp. 849–856.
  * [27] P. Zhao, C. X. Lu, J. Wang, C. Chen, W. Wang, N. Trigoni, and A. Markham, “mID: Tracking and identifying people with millimeter wave radar,” in _DCOSS_ , 2019, pp. 33–40.
  * [28] C. R. Qi, H. Su, K. Mo, and L. J. Guibas, “Pointnet: Deep learning on point sets for 3d classification and segmentation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 652–660.
  * [29] C. R. Qi, L. Yi, H. Su, and L. J. Guibas, “Pointnet++: Deep hierarchical feature learning on point sets in a metric space,” in _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , ser. NIPS’17, 2017.
  * [30] Y. Wang, Y. Sun, Z. Liu, S. E. Sarma, M. M. Bronstein, and J. M. Solomon, “Dynamic graph cnn for learning on point clouds,” _Acm Transactions On Graphics (tog)_ , vol. 38, no. 5, pp. 1–12, 2019.
  * [31] J. Owoyemi and K. Hashimoto, “Spatiotemporal learning of dynamic gestures from 3d point cloud data,” in _2018 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2018, pp. 1–5.
  * [32] R. Molina-Masegosa, J. Gozalvez, and M. Sepulcre, “Configuration of the c-v2x mode 4 sidelink pc5 interface for vehicular communication,” in _2018 14th International conference on mobile ad-hoc and sensor networks (MSN)_. IEEE, 2018, pp. 43–48.
  * [33] X. Lin, J. G. Andrews, A. Ghosh, and R. Ratasuk, “An overview of 3gpp device-to-device proximity services,” _IEEE Communications Magazine_ , vol. 52, no. 4, pp. 40–48, 2014.
  * [34] X. Wang, S. Mao, and M. X. Gong, “An overview of 3gpp cellular vehicle-to-everything standards,” _GetMobile: Mobile Computing and Communications_ , vol. 21, no. 3, pp. 19–25, 2017.
  * [35] I. Jolliffe, “Principal component analysis,” _Encyclopedia of statistics in behavioral science_ , 2005.
  * [36] X. Li, R. Li, G. Chen, C.-W. Fu, D. Cohen-Or, and P.-A. Heng, “A rotation-invariant framework for deep point cloud analysis,” _IEEE Transactions on Visualization and Computer Graphics_ , 2021.
  * [37] N. Kwak, “Principal component analysis based on l1-norm maximization,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 30, no. 9, pp. 1672–1680, 2008.
  * [38] F. Nie, H. Huang, C. Ding, D. Luo, and H. Wang, “Robust principal component analysis with non-greedy l1-norm maximization,” in _Twenty-Second International Joint Conference on Artificial Intelligence_ , 2011.
  * [39] Y. Wang, Y. Sun, Z. Liu, S. E. Sarma, M. M. Bronstein, and J. M. Solomon, “Dynamic graph CNN for learning on point clouds,” _CoRR_ , vol. abs/1801.07829, 2018. [Online]. Available: http://arxiv.org/abs/1801.07829
  * [40] I. J. Goodfellow, Y. Bengio, and A. Courville, _Deep Learning_. Cambridge, MA, USA: MIT Press, 2016, http://www.deeplearningbook.org.
  * [41] J. Lee, Y. Lee, J. Kim, A. R. Kosiorek, S. Choi, and Y. W. Teh, “Set transformer,” _CoRR_ , vol. abs/1810.00825, 2018. [Online]. Available: http://arxiv.org/abs/1810.00825
  * [42] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” 2017.
  * [43] P. Kairouz and H. McMahan, _Advances and Open Problems in Federated Learning_ , ser. Foundations and Trends in Machine Learning Series. Now Publishers, 2021. [Online]. Available: https://books.google.it/books?id=_A2EzgEACAAJ
  * [44] M. M. Amiri and D. Gündüz, “Federated learning over wireless fading channels,” _IEEE Transactions on Wireless Communications_ , vol. 19, no. 5, pp. 3546–3557, 2020.
  * [45] S. Savazzi, M. Nicoli, M. Bennis, S. Kianoush, and L. Barbieri, “Opportunities of federated learning in connected, cooperative, and automated industrial systems,” _IEEE Communications Magazine_ , vol. 59, no. 2, pp. 16–21, 2021\.
  * [46] G. Cohen, M. Hilario, H. Sax, S. Hugonnet, and A. Geissbuhler, “Learning from imbalanced data in surveillance of nosocomial infection,” _Artificial intelligence in medicine_ , vol. 37, no. 1, pp. 7–18, 2006.
  * [47] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga _et al._ , “Pytorch: An imperative style, high-performance deep learning library,” _arXiv preprint arXiv:1912.01703_ , 2019.
  * [48] M. Fey and J. E. Lenssen, “Fast graph representation learning with pytorch geometric,” _arXiv preprint arXiv:1903.02428_ , 2019.
  * [49] I. Goodfellow, Y. Bengio, and A. Courville, _Deep Learning_. MIT Press, 2016, http://www.deeplearningbook.org.
  * [50] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.

|  Dariush Salami received his BSc and MSc degrees from Shahid Beheshti
University and Amirkabir University of Technology in Software Engineering in
2016 and 2019, respectively. He is currently a Marie Skłodowska Curie fellow
in ITN-WindMill project and a PhD researcher at the department of
communications and networking at Aalto University. He is mainly focused on
Machine Learning for Wireless Communications and Sensing especially in mmWave
range.
---|---
|  Ramin Hasibi received his the BSc and MSc from Isfahan University of
Technology and Amirkabir University of Technology in Information Technology
Engineering in 2016 and 2019, respectively. He is currently a Ph.D. researcher
at the department of informatics, University of Bergen where his main research
focus is on Graph Representation Learning and Graph Neural Networks as well as
their application in different domains.
---|---
|  Stefano Savazzi received the M.Sc. degree and the Ph.D. degree (Hons.) in
ICT from the Politecnico di Milano, Italy, in 2004 and 2008, respectively. In
2012, he joined the Institute of Electronics, Computer and Telecommunication
Engineering (IEIIT), Consiglio Nazionale delle Ricerche (CNR), as a
Researcher. He has coauthored over 100 scientific publications. His current
research interests include distributed signal processing, learning and
networking aspects for the Internet of Things, radio vision and localization.
Dr. Savazzi won the Dimitris N. Chorafas Foundation Award in 2008. He is
serving as Associate Editor for Frontiers in Communications and Networks and
Topic Editor for Sensors (MDPI).
---|---
|  Tom Michoel is Professor in bioinformatics at the Computational Biology
Unit at the Department of Informatics at the University of Bergen since 2018,
and was an independent group leader in computational biology at the University
of Edinburgh (2012-2018) and the University of Freiburg (2010-2012). He
obtained the MSc degree in Physics (1997) and PhD degree in Mathematical
Physics (2001) from the KU Leuven, and was a postdoctoral researcher in
mathematics (UC Davis, 2001-2002), theoretical physics (KU Leuven, 2002-2004),
and bioinformatics and systems biology (Ghent University, 2004-2010). His
research focus in the last five years has been on developing methods,
algorithms, and software for causal inference and Bayesian network learning
from high-dimensional omics data, supported by grants from the BBSRC
(2015-2016), the NIH (2016-2019), the MRC (2017-2021), and the Norwegian
Research Council (2021-2024).
---|---
|  Stephan Sigg received his M.Sc. degree in computer science from TU
Dortmund, in 2004 and his Ph.D. degree from Kassel University, in 2008. Since
2015 he is an assistant professor at Aalto University, Finland. He is a member
of the editorial board of the Proceedings of the ACM on Interactive, Mobile,
Wearable and Ubiquitous Technologies as well as of the Elsevier journal of
Computer Communications. He has served as a TPC member of renowned conferences
including IEEE PerCom, IEEE ICDCS, etc. His research interests include Ambient
Intelligence, in particular, Pervasive sensing, activity recognition, usable
security algorithms for mobile distributed systems.
---|---


[a,b]Antonino Troja

#  Euclid in a nutshell

Isaac Tutusaus  Jenny G. Sorce  on behalf of the Euclid Consortium

###### Abstract

Euclid is a European Space Agency (ESA) mission designed to constrain the
properties of dark energy and gravity via weak gravitational lensing and
galaxy clustering. It will carry out a wide area imaging and spectroscopy
survey in visible and near-infrared bands, covering approximately $15\,000$
$\deg^{2}$ of the extragalactic sky in six years. Euclid will be equipped with
a 1.2 m diameter Silicon Carbide (SiC) mirror telescope feeding two
instruments built by the Euclid Consortium: a high-quality panoramic visible
imager and a near-infrared photometer and spectrograph. These proceedings
briefly describe the satellite and its instruments, which are optimised for
pristine point spread function and reduced stray light, producing very crisp
images. Furthermore, we summarise the survey strategy, the global scheduling,
and the preparations for the satellite commissioning and the Science Data
Centers to produce scientific data.

##  1 The Euclid mission

The Euclid mission [1] is a survey aiming at describing the nature and the
physics of the dark components of the Universe that drive its dynamics: dark
matter and dark energy. To do so, the satellite will measure billions of
galaxy shapes and tens of millions of galaxy spectra from a privileged
position: the Lagrangian point L2 in the two-body system Sun-Earth. The huge
amount of data collected during the 6 years lifetime of the mission will
permit us not only to go deeper in the understanding of the Universe’s dark
sector, fulfilling the requirement of the primary science but also to expand
the knowledge of many astrophysical fields, that will be gathered under the
so-called legacy science. Both of them are described in the two proceedings
papers accompanying this one [2, 3].

In this document, the Euclid mission is concisely described, as well as the
instruments on-board. In the last section, we give an overview of global
scheduling of the mission.

###  1.1 The space survey

The Euclid spacecraft will observe the visible and near-infrared sky through a
wide area imaging and spectroscopic survey, the Euclid Wide Survey (EWS, [4]).
The EWS will cover approximately $15\,000\deg^{2}$ during the nominal six-
years mission lifetime (Fig. 1). Together with the EWS, Euclid will also
perform an additional deep survey (DS).

Figure 1: Euclid possible wide survey area is bounded by blue lines. The
observed regions are coloured blue, the unobserved in green. The darker the
blue, the earlier that patch of the sky is scanned. DS regions are shown below
the Mollweide projection of the full sky, where the EWS area is depicted.

The EWS is the core of the mission. It will allow us to detect the signatures
of dark matter and dark energy, with the goal of better understanding the
accelerated expansion of the Universe and the nature of dark energy. The two
main probes of the EWS are galaxy clustering (GC) and weak lensing (WL). GC
will be measured from tens of millions of spectroscopic redshifts, in the
range $0.7<z<1.8$ and an accuracy of $0.001(1+z)$ at $68\%$ confidence level.
WL will be measured using $\sim 1.5$ billion galaxy shapes and will be coupled
to photometric redshifts (photo-z). Together with complementary ground-based
photometry, the near-infrared passbands enable the calculation of the mean
photo-z of the tomographic redshift bins with an accuracy of $0.002\,(1+z)$
[5]. WL measurements will be complemented by further cosmological probes
derived from the same data and combined with external ones, e.g., clusters of
galaxies, and cosmic microwave background (CMB) cross-correlation.

The EWS will cover more than one-third of the sky, excluding the Galactic and
the ecliptic planes. The EWS is estimated to have an average limited magnitude
of $26.5$ mag in the visible band $I_{\scriptscriptstyle\rm E}$ and $24.5$ mag
in the near-infrared bands $Y_{\scriptscriptstyle\rm E}$,
$J_{\scriptscriptstyle\rm E}$, and $H_{\scriptscriptstyle\rm E}$ ($5\sigma$
point-like source, magnitudes are in the AB magnitude system, [4]). On the
spectroscopic side, the $H_{\alpha}$ line flux limit is $2\times
10^{-16}\,\mbox{erg}^{-1}\,\mbox{cm}^{-2}\,\mbox{s}^{-1}$ at $1600$ nm.

The DS is designed to observe dimmer objects in smaller areas, covering at
least $40\deg^{2}$ over three fields in the northern and southern Galactic
hemispheres. DS will provide science calibration data sets to the EWS and data
for legacy science, like faint high-redshift galaxies, quasars, and active
galactic nuclei (AGN).

####  1.1.1 The survey strategy

Every day, Euclid can observe a circle in ecliptic coordinates, whose size has
a latitude varying width. The Sun-spacecraft axis will move $\sim
1$\mathrm{\SIUnitSymbolDegree}$$ per day, forcing the satellite to be as much
as possible perpendicular to the Sun. In this way, the Sun-shield of the
spacecraft will face the Sun at any time, maintaining the thermal stability of
the mirrors and the instruments.

Each tile of the survey will be observed in ‘step-and-stare’ mode, i.e.,
taking imaging and spectroscopic measurements after pointing the telescope to
a fixed position of the sky before moving to the next one. The Euclid field of
view (FoV), common to the instruments, is about $0.56\deg^{2}$. For each tile,
Euclid will follow a dithering technique, consisting in taking a series of
four observations, slightly changing the telescope pointing direction between
them. This strategy aims at correcting in part instrumental effects on the
images, for example, due to the gap between detectors in the focal planes.

###  1.2 The ground segment

Euclid will collect more than half a million visual and near-infrared images
during the 6 years of the mission. These images will be transferred to Earth
daily. The data stream is expected to be approximately $100\,\rm{GB}$ per day,
compressed by a factor of about $2.2$. The data taken by the satellite will be
complemented by data from ground-based telescopes, covering the same sky as
the EWS. The Science Ground Segment (SGS) will process and analyse the
heterogeneous data sets composed of space and ground-based images.
Furthermore, the SGS will be responsible for the Euclid archives and the
production of the official Euclid data release.

##  2 Instruments

Two modules set up the Euclid satellite: the Service Module (SVM) and the
Payload Module (PLM). The SVM is under the responsibility of Thales Alenia
Space (TAS), which is the prime contractor for the entire satellite. It
comprises the instruments like the star tracker, gyroscopes, micro-motions
systems, Attitude and Orbit Control Systems (AOCS), the thermal regulations
system, the downlink communication system, tanks for hydrazine and cold gases,
and the Fine Guidance Sensor (FGS).

The PLM is under the responsibility of Airbus (Defence and Space) and contains
the telescope and the two main scientific instruments, the visual imager (VIS,
[6]) and the Near-Infrared Spectrometer and Photometer (NISP, [7]), in
addition to the PLM thermal control system. VIS and NISP were both delivered
by the Euclid Consortium.

###  2.1 The telescope

The telescope is an off-axis 3-mirror Korsch cold telescope. The primary
mirror has a diameter of 1.2 meter, with a $\sim 0.91\deg^{2}$ FoV. During the
whole operative life of the survey, the main mirror will be cooled and
maintained at a temperature of $T<130\,\rm K$. The thermal stability is
assessed at around $50\,\rm mK$. The three mirrors are built in Silicon
Carbide (SiC), a material that ensures excellent thermo-elasticity and
stiffness. Furthermore, it is immune to ionizing radiations, guaranteeing
high-performance when in space. The secondary mirror is provided with a 5
degrees-of-freedom mechanism (M2M) that allows movements of the mirror of the
order of $1$\mathrm{\SIUnitSymbolMicro m}$$, allowing in-flight operations
aimed to correct focus and tilt.

Figure 2: VIS (left) and NISP (right) focal plane. Courtesy of TAS.

###  2.2 VIS

VIS is the key to obtaining WL measurements. VIS will measure the shapes of
galaxies in the visible spectrum, covering the 550 to 900 nm wavelength region
using a broad-band filter. An array of $6\times 6$ e2V CCDs is mounted on the
VIS focal plane (Fig. 2), equivalent to about $0.56\deg^{2}$ FoV. Each CCD is
made of $4096\times 4132$ $12\mu$m $None$ pixels. The image quality is
expected to be around $None$, allowing measurements of galaxies shapes with
high-accuracy.

###  2.3 NISP

Figure 3: Transmission curves of the $Y_{\text{E}}$, $J_{\text{E}}$, and
$H_{\text{E}}$ filters and the blue (BGS) and red grims (RGS). Figure based on
[5]

NISP will observe the sky in the near-infrared spectrum (wavelength coverage
of 900–2020 nm, Fig. 3), acquiring photometric and spectroscopic data of the
extra-galactic sources in the FoV. The NISP focal plane (Fig. 2) is equipped
with an array of $4\times 4$ HAWAII-H2RG detectors. Each detector is composed
of $2048\times 2048$ $18\mu$m $None$ pixels and the full FoV is as large as
the VIS one.

Near-infrared photometry will be performed on the same sources whose image is
collected by VIS. Three filters will be used for photometry, $Y_{\text{E}}$,
$J_{\text{E}}$, and $H_{\text{E}}$, covering wavelengths in the range
$950-2020$ nm (Fig. 3). The three filters are hosted on the Filter Wheel
(FWA), together with two other slots: OPEN, used to let the light pass without
filtering, to get its spectrum through the Grism Wheel (GWA), and CLOSED, to
measure dark frames, useful to estimate the thermal response of the detectors.

NISP will get the spectra of all the sources in the FoV through slitless
spectroscopy. The light coming from the telescope will be dispersed through
one of the grisms mounted on the GWA (namely RGS000, RGS180, and RGS270), in
the wavelength range between 1200 and 1850 nm. Each grism will give the
spectra along a different direction. These spectra will be combined offline in
order to purify the spectrum of each source from contaminants. The grism
RGS270 will not be used because of non-conformity discovered in 2020 [8]. The
observations with RGS000 and RGS180 will be complemented by two observations
made again with RGS000 and RGS180, tilted by
$-4$\mathrm{\SIUnitSymbolDegree}$$ and $+4$\mathrm{\SIUnitSymbolDegree}$$,
respectively. A fourth grism (BGS000) will cover the range between 920 and
1400 nm, and it will be used only in DS.

##  3 Global scheduling

The Euclid spacecraft launch is scheduled for Q3 2023. Euclid will reach L2 in
about 1 month. During the transfer phase to the target orbit, a 1-month-long
commissioning phase will take place. After commissioning, a performance
verification (PV) phase will be executed for 2 months, before starting the
actual data collection. Three main data releases are expected. The first one
(DR1) 24 months from the beginning of the survey, with data from about
$2500\deg^{2}$ of the sky. The second data release (DR2) is expected 2 years
after DR1, covering about $7500\deg^{2}$. The third data release (DR3) will be
delivered after 3 further years from DR2, with the full EWS area observed [4].

## Acknowledgments

IT acknowledges funding from the Swiss National Science Foundation and from
the European Research Council (ERC) under the European Union’s Horizon 2020
research and innovation program (Grant agreement No. 863929). JS acknowledges
support from the French Agence Nationale de la Recherche for the LOCALIZATION
project under grant agreements ANR-21-CE31-0019. The Euclid Consortium
acknowledges the ESA and a number of agencies and institutes that have
supported the development of Euclid, in particular the Academy of Finland, the
Agenzia Spaziale Italiana, the Belgian Science Policy, the Canadian Euclid
Consortium, the French Centre National d’Etudes Spatiales, the Deutsches
Zentrum für Luft- und Raumfahrt, the Danish Space Research Institute, the
Fundação para a Ciência e a Tecnologia, the Ministerio de Ciencia e
Innovación, the National Aeronautics and Space Administration, the National
Astronomical Observatory of Japan, the Netherlandse Onderzoekschool Voor
Astronomie, the Norwegian Space Agency, the Romanian Space Agency, the State
Secretariat for Education, Research and Innovation (SERI) at the Swiss Space
Office (SSO), and the United Kingdom Space Agency. A complete and detailed
list is available on the Euclid web site (http://www.euclid-ec.org).

## References

  * [1] R. Laureijs, et al., _Euclid Definition Study Report_ , arXiv e-prints arXiv:1110.3193 (2011), doi:10.48550/arXiv.1110.3193
  * [2] I. Tutusaus et al., _Euclid: performance on main cosmological parameter science_ , in proceedings of ICHEP2022, Proceedings of Science 2022.
  * [3] J.G. Sorce et al., _Euclid Legacy Science prospects_ , in proceedings of ICHEP2022, Proceedings of Science 2022.
  * [4] Euclid Collaboration: R. Scaramella et al., _Euclid preparation I. The Euclid Wide Survey_ , A&A, Volume 662 (2022), doi:10.1051/0004-6361/202141938
  * [5] Euclid Collaboration: M. Schirmer et al., _Euclid preparation XVIII. The NISP photometric system_ , A&A, Volume 662 (2022), doi:10.1051/0004-6361/202142897
  * [6] M. Cropper et. al., _VIS: the visible imager for Euclid_ , in proceedings of Space Telescopes and Instrumentation 2018: Optical, Infrared, and Millimeter Wave; SPIE 2018, Vol. 10698, doi: 10.1117/12.2315372
  * [7] T. Maciaszek et al., _Euclid near-infrared spectrometer and photometer instrument flight model presentation, performance, and ground calibration results summary_ , in proceedings of Space Telescopes and Instrumentation 2022: Optical, Infrared, and Millimeter Wave; SPIE 2022, Vol. 12180, doi: 10.1117/12.2630338
  * [8] Laureijs, R., et. al. _Euclid mission status after mission critical design_ , in proceedings of Space Telescopes and Instrumentation 2020: Optical, Infrared, and Millimeter Wave; SPIE 2020, Vol. 11443, doi:10.1117/12.2563145