E-mails: {paschooley<EMAIL_ADDRESS>
# Radar Human Motion Classification Using Multi-Antenna System

Patrick A Schooley  School of Engineering, Widener University, Chester, PA
19013, USA  Syed A. Hamza  School of Engineering, Widener University, Chester,
PA 19013, USA

###### Abstract

This paper considers human activity classification for an indoor radar system.
Human motions generate nonstationary radar returns which represent Doppler and
micro-Doppler signals. The time-frequency (TF) analysis of micro-Doppler
signals can discern subtle variations on the motion by precisely revealing
velocity components of various moving body parts. We consider radar for
activity monitoring using TF-based machine learning approach exploiting both
temporal and spatial degrees of freedom. The proposed approach captures
different human motion representations more vividly in joint-variable data
domains achieved through beamforming at the receiver. The radar data is
collected using real time measurements at 77 GHz using four receive antennas,
and subsequently micro-Doppler signatures are analyzed through machine
learning algorithm for classifications of human walking motions. We present
the performance of the proposed multi antenna approach in separating and
classifying two closely walking persons moving in opposite directions.

##  1 Introduction

Human motion recognition (HMR) finds important applications in a large variety
of scenarios ranging from gesture recognition for smart homes, detecting
events of interest for automatic surveillance, behavioral analysis, Gait
abnormality recognitions, health monitoring in care facilities and
rehabilitation services to enable independent living for elderly [1, 2, 3, 4,
5, 6].

Contactless sensing of human motions has gained traction because of the
obvious benefits of being non obtrusive. It does not require any user
intervention and as such the users are not required to wear specific devices
to be tracked via smart phone applications [7, 8, 9]. Radar systems are at the
forefront of remote sensing technologies as they provide robust non contact
monitoring that is not affected by lighting conditions. Additionally, active
RF (radio frequency) sensing provides 4D imaging capabilities by explicitly
measuring the scatterer velocity in addition to range and 2D angular
localization. This is unlike other remote sensing sensors of human motions
such as visual-based systems that require additional pre-processing and
filtering operations to accurately discern small movements [10, 11]. Also
radar images are privacy preserving as high resolution imaging radar renders
silhouette type portrait revealing little identifiable information as opposed
to camera-based systems.

We investigate a human activity monitoring system to concurrently monitor
movements of multiple persons. This could facilitate to separately record the
activities of multiple persons in detail. For example, in health care
facility, the task of care givers can be eased by attending to the needs of
several care receivers at the same time. Radar is a proven technology for
target detection, localization and tracking. Imaging radars are getting
attention recently because of their added capability of classifying different
targets. Radar returns classification could be performed after localizing the
target in range, azimuth and/or Doppler.

In this paper, we consider the radar human motion classification by attempting
to localize the motions to a given azimuth directions. The proposed
beamforming approach can reduce the system cost and alleviate the need of
using multiple radars as proposed in [10, 12, 13, 14]. Azimuth filtering is
achieved by applying beamforming to the receiver array. Specifically, we
consider the task of classifying two persons walking closely in opposite
directions at different azimuth angles. We aim to correctly pair the direction
of motion to the corresponding azimuth angle. This is achieved by jointly
processing the two spectrograms obtained in the directions of both motions. In
this case, the received data is filtered using beamforming with two separate
sets of beamformer weights. Subsequently, classification is performed by
jointly processing the time frequency signature of both azimuth directions.
The proposed scheme works adequately when the two motions are not completely
separable in azimuth. This could either be due to the close proximity of the
two motions in the azimuth or high sidelobes of the beamformer.

Figure 1: Processing sequence

The rest of the paper is organized as follows: In the next section, we
describe the data set and time frequency representation domain. The 2D PCA and
beamformer image fusion for multi-person monitoring is discussed in Section 3.
Experimental results are shown in Section 4, while the conclusion is given in
Section 5.

##  2 Radar return signal analysis

The complex valued raw data matrix $\mathbf{s}(n,m)\in C^{N*M}$ of the
frequency-modulated continuous wave (FMCW) radar is obtained through spatially
processing the radar returns by an $M$ element uniformly spaced antenna array.
The data is collected over $N$ temporal sampling instances. The receiver array
vector $\mathbf{s}(m)\in C^{M}$ at time instant $n$ corresponds to the
$n_{th}$ row of $\mathbf{s}(n,m)$ and is given by,

$\mathbf{s}(m)=\sum_{l=1}^{L}\alpha_{l}\mathbf{a}(\theta_{l})+\mathbf{v}(m),$
(1)

where, $\mathbf{a}({\theta_{l}})$ $\in\mathbb{C}^{M}$ is the steering vector
corresponding to the azimuth direction $\theta_{l}$ of the scatterer, and is
defined as follows,

$\mathbf{a}({\theta_{l}})=[1\,\,\,e^{j(2\pi/\lambda)dcos(\theta_{l})}\,.\,.\,.\,e^{j(2\pi/\lambda)d(M-1)cos(\theta_{l})}]^{T}.$
(2)

Here, $d$ is the inter-element spacing and $\alpha_{l}$ $\in\mathbb{C}$ is the
complex amplitude of the radar return. The additive Gaussian noise
$\mathbf{v}(m)$ $\in\mathbb{C}^{M}$ has variance $\sigma_{v}^{2}$. The
elements of the received data vector $\mathbf{s}(m)$ are combined linearly by
the $M$-sensor beamformer that strives to spatially filter the reflections
from all other directions except the signal in the direction of beamformer
look angle $\theta_{k}$. The spatially filtered signal vector
$\mathbf{x}({\theta_{k}})$ $\in\mathbb{C}^{N}$ after beamforming is given by,

$\mathbf{x}({\theta_{k}})=\mathbf{s}(n,m)\mathbf{w}^{H}({\theta_{k}}),$ (3)

where $\mathbf{w}({\theta_{k}})=\mathbf{a}^{H}({\theta_{k}})$ are the complex
beamformer weights pointing towards $\theta_{k}$.

The spatially filtered signal vector $\mathbf{x}({\theta_{k}})$ is reshaped
into a two-dimensional matrix, $\mathbf{x}_{\theta_{k}}(p,q)$. This is
achieved by segmenting the $N$ dimensional vector $\mathbf{x}({\theta_{k}})$,
such that, the $P$ samples collected within a pulse repetition interval (PRI)
are stacked into a $P$ dimensional column. There are $Q$ such columns within
$\mathbf{x}_{\theta}(p,q)$ where $Q=N/P$ is the number of PRIs processed
within the observation time $N$. The range-map, $\mathbf{r}_{\theta_{k}}(p,q)$
is obtained by applying the column-wise Discrete Fourier Transform (DFT)
operation which is given by,

$\mathbf{r}_{\theta_{k}}(l,q)=\sum_{p=0}^{P-1}\mathbf{x}_{\theta_{k}}(p,q)e^{-j(2\pi
lp/N)}$ (4)

We observe the data in the TF domain after localizing the motion in azimuth
and range bins of interest. The spectrogram is used as the TF signal
representation, showing the variation of the signal power as a function of
time $n$ and frequency $k$. The spectrogram of a periodic version of a
discrete signal $\mathbf{v}_{\theta_{k}}(n)$, is given by [15, 16, 17],

$\mathbf{d}_{\theta_{k}}(n,k)=|\sum_{m=0}^{H-1}\mathbf{h}(m)\mathbf{v}_{\theta_{k}}(n-m)e^{-j(2\pi
km/H)}|^{2},$ (5)

where
$\mathbf{v}_{\theta_{k}}=\sum_{l=r_{l}}^{r_{u}}\mathbf{r}_{\theta_{k}}(l,q)$
is obtained by collapsing the range dimension beginning from lower range bin
$r_{l}$ to highest range bin $r_{h}$. Tapering window $\mathbf{h}$ of length
$H$ is applied to reduce the sidelobes. The spectrograms reveal the the
different velocities, accelerations and higher order moments which cannot be
easily modeled or assumed to follow specific nonstationary structures [18]. We
observe the motion of two humans walking closely in opposite directions at
different azimuth angles. We aim to correctly pair the direction of motion to
the corresponding azimuth angle. This is achieved by jointly processing two
spectrograms, $\mathbf{v}_{\theta_{1}}(n,k)$ and
$\mathbf{v}_{\theta_{2}}(n,k)$ which are respectively localized at azimuth
angles $\theta_{1}$ and $\theta_{2}$. It is clear that the concurrent motions
of multiple objects are hard to be distinguished in azimuth by only using a
single antenna.

##  3 Feature extraction and classification

We adopt Two-Dimensional Principal Component Analysis (2-D PCA) for
dimensionality reduction to draw the most pertinent features from spectrograms
[19, 20]. The features obtained from individual spectrograms are jointly
classified with the Nearest Neighbor (NN) classifier [21].

The 2-D PCA is performed on the covariance matrix $\mathbf{R}_{\theta_{k}}$
which is computed as follows,

$\mathbf{R}_{\theta_{k}}=\sum_{i=0}^{T-1}{\mathbf{X}^{i}}^{H}_{\theta_{k}}\mathbf{X}^{i}_{\theta_{k}},$
(6)

where, $\mathbf{X}^{i}_{\theta_{k}}$ is the normalized spectrogram for the
$i_{th}$ example and $T$ are the total training examples. The
eigendecomposition of $R_{\theta_{k}}$ is performed and the individual train
images are projected onto the subspace spanned by the $K$ dominant
eigenvectors of $R_{\theta_{k}}$ corresponding to the $K$ largest eigenvalues.
It is noted that two spectrogram images are generated per example, by using
two different sets of beamformer weights. After separately performing the 2-D
PCA, the projected spectrograms are vectorized and concatenated before
training the NN classifier, as shown in Fig. 1.

The overall data collection, preprocessing and classification can be described
by the following steps.

###  3.1 Data Collection and Preprocessing

  * •

PRI is set to 1 ms, and each data example is observed over the time period of
12 s, resulting in $Q=12000$ slow time samples.

  * •

ADC sampling rate is 512 ksps, rendering 512 fast time samples per PRI,
resultantly the length of data vector is $N=6144000$.

  * •

The received data $\mathbf{s}(n,m)\in C^{N*M}$, is collected through $M=4$
element receive array, with an inter-element spacing of $\lambda/2$ ($\lambda$
is the wavelength corresponding to the operating frequency), therefore the
dimensionality of received raw data matrix is $6144000\times 4$.

  * •

Beamforming is performed on the raw data matrix, resulting in a spatially
filtered $\mathbf{x}({\theta_{k}})$ vector of dimensions $6144000\times 1$.
Two such vectors are generated in the directions of each motion ${\theta_{1}}$
and ${\theta_{2}}$.

  * •

Each vector $\mathbf{x}({\theta_{k}})$ is reshaped into a $512\times 12000$
matrix. After applying columnwise DFT, and identifying the range bins of
interest, the corresponding rows are summed together, resulting in
$\mathbf{v}_{\theta_{k}}=\sum_{l=r_{l}}^{r_{u}}\mathbf{r}_{\theta_{k}}(l,q)$,
which is of dimensions $12000\times 1$.

  * •

Two spectrogram $\mathbf{d}_{\theta_{1}}$ and $\mathbf{d}_{\theta_{2}}$, each
of dimensions $384\times 128$ is obtained, where the window length is 128.

###  3.2 Training

  * •

Normalize both spectrograms of all training examples $\mathbf{v}_{\theta_{k}}$
by subtracting the average.

  * •

Perform the eigendecomposition of the covariance matrices estimated according
to (6) and select $K$ dominant eigenvectors. Note that covariance matrices are
separately evaluated for both angles.

  * •

Project the training spectrograms onto the subspace spanned by selected
eigenvectors, resulting in two projected spectrograms per example. Each has
the dimensionality $128\times K$.

  * •

After vectorizing each spectrogram, both are concatenated into a $256K\times
1$ vector and trained through NN classifier.

###  3.3 Testing

  * •

Normalize the testing image and project onto the subspace obtained in the
training process.

  * •

The testing image is then passed onto trained NN classifier for prediction.

Table 1: Confusion matrix, Number of principal eigenvectors, $K$=1 Classified/Actual Class |  Class-1 |  Class-2
---|---|---
Class-1 |  97.6% |  0%
Class-2 |  2.4% |  100%
Table 2: Confusion matrix, Number of principal eigenvectors, $K$=2 Classified/Actual Class |  Class-1 |  Class-2
---|---|---
Class-1 |  100% |  0%
Class-2 |  0% |  100%

##  4 Experimental Results

In this section, we demonstrate the effectiveness of beamformer image fusion
for monitoring multiple persons in the field of view. We consider two classes,
Class-1 and Class-2, of motions. For Class-1 motions, one person moves
radially towards the radar at an azimuth angle of $\theta_{1}$ while at the
same time another person moves radially away from the radar at an azimuth
angle of $\theta_{2}$. For Class-2 motion, the two persons perform the same
walking motion but in an opposite direction, i.e., this time the person at
azimuth angle $\theta_{1}$ moves radially away from the radar, whereas the
other person concurrently moves radially towards the radar at an azimuth angle
of $\theta_{2}$.

Figure 2 shows the time frequency signature of Class-1 motion. Note the TF
domain depicts both strong positive and negative Doppler, where both
frequencies are present at the same time. The time frequency signature of the
Class-1 motion is processed after applying beamforming weights to the output
of all receivers. The beamformer points to $\theta_{1}$ direction. Figure 3
shows the time frequency signature of the beamformer pointing towards
$\theta_{1}$. It is evident from the time frequency signature that the
beamformer attempted to filter the positive frequencies. However, the positive
Doppler is not completely eliminated due to the sidelobes of the beamformer.
The beamformer pointing towards $\theta_{2}$ emphasizes the positive Doppler
while attempting to mitigate the strong negative Doppler as depicted in Fig.
4. On the other hand, Fig. 5 shows the TF spectrum of the Class-2 motion. It
is clear that Figs. 2 and 5 are very similar and are of little use to classify
the two motion classes. However, the spatially filtered TF spectrums for
Class-2 are flipped for respective look directions when compared to Class-1
motion as shown in Figs. 6 and 7. Therefore, the two beamformed TF spectrums
when processed jointly can potentially classify the two motion classes with
reasonable accuracy, therefore mapping the direction of motion to the correct
azimuth angle.

Figure 2: Time Frequency signature of Class-1 motions processed through single
antenna (without beamforming) Figure 3: Time Frequency signature of Class-1
motions processed through beamformer pointing towards $\theta_{1}$. Figure 4:
Time Frequency signature of Class-1 motions processed through beamformer
pointing towards $\theta_{2}$. Figure 5: Time Frequency signature of Class-2
motions processed through single antenna (without beamforming) Figure 6: Time
Frequency signature of Class-2 motions processed through beamformer pointing
towards $\theta_{1}$. Figure 7: Time Frequency signature of Class-2 motions
processed through beamformer pointing towards $\theta_{2}$.

The radar system used in the experiments is AWR2243 from Texas Instruments
having four receiver and three transmitters. The center frequency is 77 GHz,
whereas the bandwidth is 5 GHz. The obtained data set is analyzed in MATLAB
R2020B. The data set contains 120 samples, 60 samples for each motion class
collected through six subjects. For the 2D PCA-based classifications, the
training set consists of 46 samples for each class (92 samples in total), and
each sample are two 128 × 384 images. The rest of the data set is used for
testing. The first image is generated by steering the beamformer to
$\theta_{1}$. Likewise, the other image is generated by changing the steering
angle to $\theta_{2}$. Thus, for each motion class two 128× 384 × 46 matrices
are generated. After performing the dimensionality reduction through 2D PCA,
both beamformed TF images are concatenated together and passed on to NN
classifier. The confusion matrix for fused spectrogram is shown in Tables 1,
depicting the correspondence between the actual and the classified class. It
shows that using only the dominant eigenvector renders a classification
accuracy of around $97.6\%$ and $100\%$ for Class-1 and Class-2 respectively.
More specifically, Table 1 depicts that the around $97.6\%$ of actual Class-1
motions are classified as Class-1, while the rest $2.4\%$ are declared as
Class-2. The success rate can be improved by including an additional principal
component. Table 2 shows that the proposed scheme has a $100\%$ success rate
when two principal eigenvectors are employed.

##  5 Conclusion

In this paper, we introduced an approach that observes the time frequency
representation of radar returns from different azimuth angles. We provided an
effective means to discern combinations of multiple motions occurring at
different angular directions from the radar. The proposed approach
successfully maps the actual motions to the corresponding angular locations
and is found to be effective when the spectrograms are not completely
separable in angle.

##  6 ACKNOWLEDGMENT

The authors would like to thank Andrew Lichtenwalner, Daniel Galvao, Ryan
Cummings and Connor Ryan for their assistance with data collection.

## References

  * [1] H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre, “HMDB: A large video database for human motion recognition,” in _2011 International Conference on Computer Vision_ , 2011, pp. 2556–2563.
  * [2] M. Amin, _Radar for Indoor Monitoring: Detection, Classification, and Assessment_. CRC Press, 2017.
  * [3] S. Skaria, A. Al-Hourani, M. Lech, and R. J. Evans, “Hand-gesture recognition using two-antenna doppler radar with deep convolutional neural networks,” _IEEE Sensors Journal_ , vol. 19, no. 8, pp. 3041–3048, 2019\.
  * [4] P. Wang, W. Li, P. Ogunbona, J. Wan, and S. Escalera, “RGB-D-based human motion recognition with deep learning: A survey,” _Computer Vision and Image Understanding_ , vol. 171, pp. 118–139, 2018.
  * [5] A. K. Seifert, M. G. Amin, and A. M. Zoubir, “Toward unobtrusive in-home gait analysis based on radar micro-doppler signatures,” _IEEE Transactions on Biomedical Engineering_ , vol. 66, no. 9, pp. 2629–2640, 2019\.
  * [6] J. Liu, C. Gu, Y. Zhang, and J. Mao, “Analysis on a 77 ghz mimo radar for touchless gesture sensing,” _IEEE Sensors Letters_ , vol. 4, no. 5, pp. 1–4, 2020.
  * [7] M. G. Amin, Y. D. Zhang, F. Ahmad, and K. C. D. Ho, “Radar signal processing for elderly fall detection: The future for in-home monitoring,” _IEEE Signal Processing Magazine_ , vol. 33, no. 2, pp. 71–80, 2016.
  * [8] G. Gennarelli, F. Soldovieri, and M. Amin, “Radar for indoor surveillance: state of art and perspectives,” in _Multimodal Sensing: Technologies and Applications_ , E. Stella, Ed., vol. 11059, International Society for Optics and Photonics. SPIE, 2019, pp. 1 – 10. [Online]. Available: https://doi.org/10.1117/12.2527660
  * [9] S. A. Shah and F. Fioranelli, “RF sensing technologies for assisted daily living in healthcare: A comprehensive review,” _IEEE Aerospace and Electronic Systems Magazine_ , vol. 34, no. 11, pp. 26–44, 2019.
  * [10] B. Jokanović and M. Amin, “Fall detection using deep learning in range-doppler radars,” _IEEE Transactions on Aerospace and Electronic Systems_ , vol. 54, no. 1, pp. 180–189, 2018.
  * [11] S. Z. Gurbuz and M. G. Amin, “Radar-based human-motion recognition with deep learning: Promising applications for indoor monitoring,” _IEEE Signal Processing Magazine_ , vol. 36, no. 4, pp. 16–28, 2019.
  * [12] Z. Chen, G. Li, F. Fioranelli, and H. Griffiths, “Dynamic hand gesture classification based on multistatic radar micro-doppler signatures using convolutional neural network,” in _2019 IEEE Radar Conference (RadarConf)_ , 2019, pp. 1–5.
  * [13] F. Fioranelli, J. Patel, S. Z. Gürbüz, M. Ritchie, and H. Griffiths, “Multistatic human micro-doppler classification with degraded/jammed radar data,” in _2019 IEEE Radar Conference (RadarConf)_ , 2019, pp. 1–6.
  * [14] S. Gurbuz, J. Soraghan, A. Balleri, and C. Clemente, “Micro-doppler based in-home aided and unaided walking recognition with multiple radar and sonar systems,” _IET Radar, Sonar & Navigation_, vol. 11, 06 2016.
  * [15] L. Cohen, “Time-frequency distributions-a review,” _Proceedings of the IEEE_ , vol. 77, no. 7, pp. 941–981, 1989.
  * [16] P. Flandrin, _Time-Frequency/Time-Scale Analysis_. Academic Press, 1999.
  * [17] V. Chen and H. Ling, _Time-Frequency Transforms for Radar Imaging and Signal Analysis_. Artech House, 2002.
  * [18] P. Setlur, M. Amin, and F. Ahmad, “Analysis of micro-Doppler signals using linear FM basis decomposition,” in _Radar Sensor Technology X_ , R. N. Trebits and J. L. Kurtz, Eds., vol. 6210, International Society for Optics and Photonics. SPIE, 2006, pp. 201 – 211. [Online]. Available: https://doi.org/10.1117/12.669003
  * [19] B. Erol and M. Amin, “Generalized PCA fusion for improved radar human motion recognition,” in _2019 IEEE Radar Conference (RadarConf)_ , 2019, pp. 1–5.
  * [20] A. Malhi and R. X. Gao, “PCA-based feature selection scheme for machine defect classification,” _IEEE Transactions on Instrumentation and Measurement_ , vol. 53, no. 6, pp. 1517–1525, 2004.
  * [21] M. G. Amin and R. G. Guendel, “Radar human motion recognition using motion states and two-way classifications,” in _2020 IEEE International Radar Conference (RADAR)_ , 2020, pp. 1046–1051.