# Enhancing Dimension-Reduced Scatter Plots with Class and Feature Centroids

Daniel B. Hier, Tayo Obafemi-Ajayi, Gayla R. Olbricht, Devin M. Burns, Sasha
Petrenko, Donald C. Wunsch II Daniel Hier, Gayla Olbricht, Sasha Petrenko,
Devin Burns, and Donald Wunsch II are with Missouri University of Science &
Technology (Missouri S & T, Rolla, MO, USA. Email: {hierd, olbrichtg, burnsde,
petrenkos<EMAIL_ADDRESS>Obafemi-Ajayi is with Missouri State
University, Springfield, MO, USA. Email<EMAIL_ADDRESS>Partial support from the Missouri S & T Kummer Institute, the Mary Finley
Endowment, and the Missouri S & T Intelligent Systems Center are gratefully
acknowledged.

###### Abstract

Dimension reduction is increasingly applied to high-dimensional biomedical
data to improve its interpretability. When datasets are reduced to two
dimensions, each observation is assigned an x and y coordinates and is
represented as a point on a scatter plot. A significant challenge lies in
interpreting the meaning of the x- and y-axes due to the complexities inherent
in dimension reduction. This study addresses this challenge by using the x and
y coordinates derived from dimension reduction to calculate class and feature
centroids, which can be overlaid onto the scatter plots. This method connects
the low-dimension space to the original high-dimensional space. We illustrate
the utility of this approach with data derived from the phenotypes of three
neurogenetic diseases and demonstrate how the addition of class and feature
centroids increases the interpretability of scatter plots.

Clinical relevance: Overlaying class and feature centroids onto two-
dimensional scatter plots derived from high-dimensional biomedical datasets
enhances their interpretability.

##  I Introduction

Big data and precision medicine initiatives are creating large datasets with
numerous observations and many features [1, 2]. In particular, genomic and
molecular biological datasets are characterized by high dimensionality [3, 4].
The goal of dimension reduction is to simplify the dataset by reducing the
number of features while retaining as much of the original information as
possible. Dimension reduction is especially important when dealing with high-
dimensional data, where a large number of features can lead to complexity,
increased computational costs, and the risk of overfitting by machine learning
models [5, 6, 7].

Figure 1:  Stacked bar chart of phenotypes (features) by disease type. Each
column shows the frequency of one of the 31 available phenotype superclasses.
The three most frequent phenotypes were incoordination, hyperreflexia, and eye
movement abnormalities.

High-dimensional datasets can be visualized as scatter plots when reduced to
two dimensions. A variety of dimensionality reduction methods are available,
including locally linear embedding (LLE), isometric feature mapping (ISOMAP),
multidimensional scaling (MDS), uniform manifold approximation and projection
(UMAP), principal component analysis (PCA), and t-distributed stochastic
neighbor embedding (t-SNE) [8, 9, 10, 11, 12, 13]. These methods vary in
stochasticity, linearity, sensitivity to parameter changes, and their ability
to preserve the local and global structure of the data. The interpretation of
scatter plots after dimension reduction remains complex. After dimension
reduction, the x and y axes cannot be easily labeled and are typically
expressed in arbitrary units that reflect the operations of complex algorithms
[14].

We present a method to enhance the interpretability of two-dimensional scatter
plots obtained after dimension reduction. Feature and class centroids are
overlaid onto the plots to enhance interpretability. This approach addresses
the challenge of interpreting the axes of dimension-reduced plots.

To demonstrate the feasibility of this approach, we used it to visualize the
phenotypes of three rare neurogenetic diseases: hereditary spastic paraparesis
(HSP), hereditary cerebellar ataxia (CA), and Charcot-Marie-Tooth disease
(CMT). Gene mutations cause all [15, 16, 17]. We sought to capture the
phenotypic variability between these diseases with a scatter plot.
Specifically, we reduced a high-dimensional space with 31 available phenotype
categories (Fig. 1) to two dimensions and overlaid class and feature
centroids. Before dimension reduction, the data had 31 columns (each column
representing a phenotype feature) and 235 rows (each row representing a
variant of either Charcot-Marie-Tooth disease, cerebellar ataxia, or
hereditary spastic paraparesis).

Figure 2:  SHAP values used to identify the most important features predicting
class membership. Bar length is proportional to influence on a class i.e.
incoordination favored cerebellar ataxia; hyporeflexia: Charcot-Marie-Tooth
disease; and hypertonia: hereditary spastic paraparesis. Top $10$ features
were retained for further analysis. Figure 3: Summary of dimension reduction
strategy. The Human Phenotype Ontology (HPO) has 8743 terms; the 235
neurogenetic disease cases used 790 of these terms; subsumption reduced the
phenotypes to 31 categories; UMAP and t-SNE reduced the dimensions to 2.

Relevant Prior Work. Reviews of dimension reduction methods for the
visualization of high-dimensional datasets are available [18, 12, 19, 8, 13].
Based on quantitative benchmarks and end-user preferences, [20] found that
t-SNE and UMAP outperformed other dimension reduction methods. UnProjection
[21] is a project that uses neural networks to reconnect the low-dimension
space to the high-dimension space. Interaxis is a project that allows end
users to interactively manipulate the axes of low-dimensional scatter plots to
explore the influence of features on the plotted positions of markers [22]. In
similar work, Eckelt et al. [23] have suggested adding group centroids to low-
dimensional embeddings of high-dimensional datasets. [24] has used annotations
to make disease-phenotype plots based on t-SNE more interpretable. [25] have
created t-viSNE as a visual interactive inspector of t-SNE scatter plots. It
can generate interpretive feature bar charts based on the t-SNE plots that are
the circumscribed areas of interest. [26] introduced NoLiES, an interactive
system to improve the interpretability of low-dimensional embeddings by
circumscribing rangesets on the plots.

##  II Methods

Data acquisition. The variants belonging to the three neurogenetic diseases
(Charcot-Marie-Tooth disease, hereditary cerebellar ataxia, and hereditary
spastic paraplegia) were extracted from Orphadata [27], resulting in a total
of $235$ variants. The phenotypes for each variants were coded as terms from
the Human Phenotype Ontology (HPO) [28]. Nine hundred and seventy unique HPO
terms were needed to describe the phenotypes of the 235 variants of
neurogenetic diseases (Fig. 3. Only HPO terms that were signs or symptoms of
neurogenetic disease were retained. Radiological and laboratory findings were
excluded. Each HPO term was binary coded (present or absent), creating an
array with $235$ rows (each row was a variant of one of the three neurogenetic
diseases), two columns of labels (disease name and variant name), and $970$
columns of features (each a phenotype description that was binary coded as 0
or 1 for absent or present). For example, Row 1 had a disease name of Charcot-
Marie-Tooth disease, a variant name of Charcot-Marie-Tooth disease type 1A,
and phenotype features of hyporeflexia, gait disturbance, pes cavus, distal
muscle weakness, distal sensory impairment, skeletal muscle atrophy, sensory
ataxia, gait imbalance, kyphoscoliosis, paresthesia, calf muscle hypertrophy,
diaphragmatic weakness, spontaneous pain sensation, shoulder pain, and
hyperactive deep tendon reflexes (all coded 1 for present). Among the $235$
neurogenetic diseases were $125$ variants of cerebellar ataxia (CA), $33$
variants of Charcot-Marie-Tooth disease (CMT), and $77$ variants of hereditary
spastic paraparesis (HSP).

Dimension reduction. The number of phenotype features (HPO terms) was reduced
from 970 to 31 by the reduction by subsumption method [29, 30]. This method is
applicable here because of the hierarchical structure of the HPO. Most
terminology ontologies (HPO included) are based on a subsumptive containment
hierarchy with classes hierarchically organized from the general to the
specific. Each child class inherits properties from its parent class. This
inheritance of properties from a parent is called subsumption. The reduction
by subsumption procedure replaces terms that are highly specific in meaning
and lower in the ontology hierarchy with more general terms higher in the
hierarchy. For example, a more general term such as hyporeflexia can serve as
a container for more specific HPO terms such as areflexia, lower limb
hyporeflexia, areflexia of the lower limbs, reduced tendon reflexes, absent
Achilles reflex, hyporeflexia of the upper limbs, and hyporeflexia of the
lower limbs that are subsumed by the category of hyporeflexia. After replacing
more specific phenotype terms with more general phenotype categories by
subsumption (Fig. 3), the dimension-reduced data array had $235$ rows, two
label columns, and $31$ feature columns. The data array was reduced to two
dimensions using t-SNE [31] (perplexity = 50, metric = euclidean).

Calculation of class and feature centroids. Dimension reduction by t-SNE
created x, y coordinates for each observation. The x and y coordinates were
concatenated to the original $235\times 33$ data matrix, yielding a $235\times
35$ matrix. The concatenated data matrix was sorted and filtered by either the
variant name (class) column or any of the $31$ phenotype (feature) columns. We
calculated class centroids (Equation 1) for the three ground truth classes
(cerebellar ataxia, Charcot-Marie-Tooth disease, and hereditary spinal
paraparesis). We calculated feature centroids for the 31 feature categories
(Equation 2). For plotting purposes, we looked for the most important features
that predicted class membership. We fitted an xgboost  classifier [32] to the
data and used the Shapley Additive exPlanations (SHAP) values [33] to find the
most important phenotype features for predicting class membership (Fig. 2).
SHAP applies game theory to identify features most predictive of class
membership .

Scatter plots. Based on their x, y coordinates from t-SNE, observations were
plotted as markers on a scatter plot (plt.scatter from matplotlib). Markers
were color-coded based on ground truth labels. Class and feature centroids
were plotted, color-coded, and annotated by the same methods.

Given a data array D, with a set of features F and class labels L. Class
Centroid Calculation:
For class $c$ in L, where $N_{c}$ is the number of observations in class $c$:
$\text{Class Centroid (C}_{x},C_{y})=\left(\frac{\sum x_{i}}{N_{c}},\frac{\sum
y_{i}}{N_{c}}\right)$ (1) where $(x_{i},y_{i})$ are the coordinates of
observations in class $c$. Feature Centroid Calculation:
For feature $f$ in F, where $N_{f}$ is the number of observations with feature
$f$: $\text{Feature Centroid}(F_{x},F_{y})=\left(\frac{\sum
x_{i}}{N_{f}},\frac{\sum y_{i}}{N_{f}}\right)$ (2) where $(x_{i},y_{i})$ are
coordinates of observations with feature $f$.

Figure 4:  Scatter plot of 235 neurogenetic disease variants in two-
dimensional space. The x and y coordinates were calculated by t-SNE based on
the 31 phenotype features. Figure 5: Same scatter plot as Fig. 4 with markers
colored by their ground truth labels. Note that markers form three groupings
based on their class membership. There are 33 variants of Charcot-Marie-Tooth
disease, 77 variants of hereditary spastic paraparesis, and 125 variants of
cerebellar ataxia shown. Each variant has a distinct phenotype. Figure 6:
Feature centroids for ten of the most important features. The x- and
y-coordinates of these features have been calculated based on the x- and
y-coordinates of the disease variants that exhibit these features
(phenotypes). See Methods for details. Figure 7:  Same data as Fig. 5. Feature
centroids have been added to the scatter plot. Although the scatter plot is
cluttered, note the proximity of Charcot-Marie-Tooth disease to feature
centroids for weakness, atrophy, and hyporeflexia; the proximity of cerebellar
ataxia markers to the feature centroids for eye movements, incoordination,
tremor, and cognitive; and the proximity of hereditary spastic paraparesis
markers to hypertonia, hyperreflexia, and sensory feature centroids. Figure 8:
Class centroids for the three neurogenetic disease classes, compared to Fig.
5.  Figure 9: Composite scatter plot that combines the feature centroids from
Fig. 6 with the class centroids from Fig. 8.

##  III Results

We downloaded the phenotypes of 235 neurogenetic disease variants from
Orphadata. Orphadata encodes phenotypes with terms from the Human Phenotype
Ontology [28]. Orphadata uses $970$ unique terms from the HPO to describe the
phenotypes of these disease variants. We used reduction by subsumption to
collapse these 970 terms into 31 phenotype categories (Figs. 3 and 1). We used
t-SNE to reduce the data to two dimensions. This approach provided x and y
coordinates for each observation, enabling us to calculate the x,y coordinates
of each observation (disease variant) as well as the mean x,y coordinates of
the class and feature centroids.

The 235 observations can be represented as markers on a scatter plot (Fig. 4).
Since the ground truth label is available for each observation, each marker
can be color-coded by class name (Fig. 5). We calculated the feature centroids
for each of the 31 phenotype categories (Equation 2) and the class centroids
for the three disease classes (Equation 1). Rather than plot all 31 feature
centroids on a scatter plot, we used Shapley Additive exPlanations (SHAP) to
find the most informative features (sometimes influential or important) (Fig.
2). The centroids of the 10 most informative phenotype features are shown in
Fig. 6. These feature centroids were overlaid onto a scatter plot with all 235
observations (Fig. 7). We plotted class centroids (Fig. 8) and then combined
the feature centroids and the class centroids into a single plot (Fig. 9).

##  IV Discussion

Dimension reduction is a powerful tool for gaining insight into high-dimension
data. As Rudin et al. [14] commented, even in data science, a picture is worth
a thousand words. Dimension reduction (DR) techniques take, as input, high-
dimensional data and project it down to a lower-dimensional space (usually 2D
or 3D) so that a human can better comprehend it. However, interpreting low-
dimension representations of high-dimension spaces is difficult. The x- and
y-axis of two-dimensional spaces derived from dimension reduction are in
arbitrary units and have no simple interpretation. Rudin et al.[14] consider
the interpretation of spaces created by dimension reduction one of the grand
challenges of explainable AI and ask Can the DR transformation from high- to
low-dimensions be made more interpretable or explainable? We suggest that the
overlay of class and feature centroids onto dimension-reduced scatter plots
more interpretable. When datasets are reduced to two dimensions, x and y
coordinates allow each observation to be represented as a marker on a scatter
plot (Figs. 4 and 5). These x and y coordinates also allow a calculation of
feature and class centroids which can be overlaid onto scatter plots (Figs. 8
and 6).

The overlay of feature and class centroids onto dimension-reduced scatter
plots offers new avenues for interpretation (Fig. 9). One approach is to
examine scatter plots on a quadrant-by-quadrant basis and look for classes and
features that share the same quadrant. For example, in Fig. 9, the upper left
quadrant is inhabited by the class CA (cerebellar ataxia) and the features
incoordination, cognitive, tremor, hyperreflexia, and incoordination.
Importantly, tremor, eye movement abnormalities, and incoordination are key
presenting signs of cerebellar ataxia. Another approach is to use the
proximity between class and feature centroids to interpret the dimension-
reduced scatter plot. For example, the centroid for hereditary spastic
paraparesis (HSP) is closest to the features of hypertonia, sensory loss,
atrophy, weakness, and hyperreflexia (Fig. 9), and signs and symptoms of this
disorder.

Although our novel approach, which relies on calculating class and feature
centroids, appears promising, it does have limitations. Since most dimension
reduction methods are nonlinear, there is no straightforward way to interpret
distances between centroids on dimension-reduced plots (Figs. 6 and 8). The
relative distances between centroids must be interpreted with caution. Second,
more work needs to be done on scalability. Our studies were carried out on a
small $235\times 31$ data array; additional work on larger datasets is needed.
Third, more work needs to be done on generalizability. Our dataset was based
on neurological phenotypes that were binary-coded. Further work can assess
whether the same methods can be applied to continuous or ordinal data from
other domains of interest.

Nevertheless, this approach to improving the interpretability dimension-
reduced scatter plots has advantages. First, adding feature centroids provides
a link to the underlying feature space, aiding in understanding the
significance of the x and y axes. Second, replacing observation markers with
class centroids improves readability (Fig. 8). Third, the calculation of
feature and class centroids is straightforward and the resulting centroids are
easily plotted using matplotlib. This method could be useful in biology,
medicine, and bioinformatics, where dimension-reduced scatter plots are widely
used.

## Data and code availability

Data for the neurogenetic diseases and Python code are available on the
project GitHub site at https://github.com/ACIL-Group/CENTROIDS and archived at
Zenodo [34].

## References

  * [1] T. Hulsen, S. S. Jamuar, A. R. Moody, J. H. Karnes, O. Varga, S. Hedensted, R. Spreafico, D. A. Hafler, and E. F. McKinney, “From big data to precision medicine,” _Frontiers in medicine_ , vol. 6, p. 34, 2019.
  * [2] B. B. Misra, C. Langefeld, M. Olivier, and L. A. Cox, “Integrated omics: tools, advances and future approaches,” _Journal of molecular endocrinology_ , vol. 62, no. 1, pp. R21–R45, 2019.
  * [3] J. Hua, W. D. Tembe, and E. R. Dougherty, “Performance of feature-selection methods in the classification of high-dimension data,” _Pattern Recognition_ , vol. 42, no. 3, pp. 409–424, 2009.
  * [4] R. Clarke, H. W. Ressom, A. Wang, J. Xuan, M. C. Liu, E. A. Gehan, and Y. Wang, “The properties of high-dimensional data spaces: implications for exploring gene and protein expression data,” _Nature reviews cancer_ , vol. 8, no. 1, pp. 37–49, 2008.
  * [5] R. E. Colgan, D. E. Gutierrez, J. Sundram, and G. B. Tenali, “Analysis of medical data using dimensionality reduction techniques,” _Przeglad Elektrotechniczny_ , vol. 89, pp. 279–281, 2013.
  * [6] K. Al-Jabery, T. Obafemi-Ajayi, G. Olbricht, and D. Wunsch, _Computational Learning Approaches to Data Analytics in Biomedical Applications_. Academic Press, 2019.
  * [7] J. T. De Souza, A. C. De Francisco, and D. C. De Macedo, “Dimensionality reduction in gene expression data sets,” _IEEE Access_ , vol. 7, pp. 61 136–61 144, 2019.
  * [8] B. Ghojogh, A. Ghodsi, F. Karray, and M. Crowley, “Multidimensional scaling, sammon mapping, and isomap: Tutorial and survey,” _arXiv preprint arXiv:2009.08136_ , 2020.
  * [9] D. Kobak and P. Berens, “The art of using t-sne for single-cell transcriptomics,” _Nature communications_ , vol. 10, no. 1, p. 5416, 2019.
  * [10] G. C. Linderman, M. Rachh, J. G. Hoskins, S. Steinerberger, and Y. Kluger, “Fast interpolation-based t-sne for improved visualization of single-cell rna-seq data,” _Nature methods_ , vol. 16, no. 3, pp. 243–245, 2019.
  * [11] M. Wattenberg, F. Viégas, and I. Johnson, “How to use t-sne effectively,” _Distill_ , vol. 1, no. 10, p. e2, 2016.
  * [12] F. Anowar, S. Sadaoui, and B. Selim, “Conceptual and empirical comparison of dimensionality reduction algorithms (pca, kpca, lda, mds, svd, lle, isomap, le, ica, t-sne),” _Computer Science Review_ , vol. 40, p. 100378, 2021.
  * [13] S. Ayesha, M. K. Hanif, and R. Talib, “Overview and comparative study of dimensionality reduction techniques for high dimensional data,” _Information Fusion_ , vol. 59, pp. 44–58, 2020.
  * [14] C. Rudin, C. Chen, Z. Chen, H. Huang, L. Semenova, and C. Zhong, “Interpretable machine learning: Fundamental principles and 10 grand challenges,” _Statistic Surveys_ , vol. 16, pp. 1–85, 2022.
  * [15] J.-M. Vallat, C. Goizet, M. Tazir, P. Couratier, L. Magy, and S. Mathis, “Classifications of neurogenetic diseases: An increasingly complex problem,” _Revue Neurologique_ , vol. 172, no. 6-7, pp. 339–349, 2016.
  * [16] P. Lallemant-Dudek and A. Durr, “Clinical and genetic update of hereditary spastic paraparesis,” _Revue neurologique_ , vol. 177, no. 5, pp. 550–556, 2021.
  * [17] D. Pareyson and C. Marchesi, “Diagnosis, natural history, and management of charcot–marie–tooth disease,” _The Lancet Neurology_ , vol. 8, no. 7, pp. 654–667, 2009.
  * [18] Y. Wang, H. Huang, C. Rudin, and Y. Shaposhnik, “Understanding how dimension reduction tools work: an empirical approach to deciphering t-sne, umap, trimap, and pacmap for data visualization,” _The Journal of Machine Learning Research_ , vol. 22, no. 1, pp. 9129–9201, 2021.
  * [19] E. S. Ortigossa, F. F. Dias, and D. C. d. Nascimento, “Getting over high-dimensionality: How multidimensional projection methods can assist data science,” _Applied Sciences_ , vol. 12, no. 13, p. 6799, 2022.
  * [20] J. Xia, Y. Zhang, J. Song, Y. Chen, Y. Wang, and S. Liu, “Revisiting dimensionality reduction techniques for visual cluster analysis: an empirical study,” _IEEE Transactions on Visualization and Computer Graphics_ , vol. 28, no. 1, pp. 529–539, 2021.
  * [21] M. Espadoto, G. Appleby, A. Suh, D. Cashman, M. Li, C. E. Scheidegger, E. W. Anderson, R. Chang, and A. C. Telea, “Unprojection: Leveraging inverse-projections for visual analytics of high-dimensional data,” _IEEE Transactions on Visualization and Computer Graphics_ , 2021.
  * [22] H. Kim, J. Choo, H. Park, and A. Endert, “Interaxis: Steering scatterplot axes via observation-level interaction,” _IEEE transactions on visualization and computer graphics_ , vol. 22, no. 1, pp. 131–140, 2015.
  * [23] K. Eckelt, A. Hinterreiter, P. Adelberger, C. Walchshofer, V. Dhanoa, C. Humer, M. Heckmann, C. Steinparz, and M. Streit, “Visual exploration of relationships and structure in low-dimensional embeddings,” _IEEE Transactions on Visualization and Computer Graphics_ , 2022.
  * [24] W. Xu, X. Jiang, X. Hu, and G. Li, “Visualization of genetic disease-phenotype similarities by multiple maps t-sne with laplacian regularization,” _BMC medical genomics_ , vol. 7, no. 2, pp. 1–9, 2014.
  * [25] A. Chatzimparmpas, R. M. Martins, and A. Kerren, “t-visne: Interactive assessment and interpretation of t-sne projections,” _IEEE transactions on visualization and computer graphics_ , vol. 26, no. 8, pp. 2696–2714, 2020.
  * [26] J.-T. Sohns, M. Schmitt, F. Jirasek, H. Hasse, and H. Leitte, “Attribute-based explanation of non-linear embeddings of high-dimensional data,” _IEEE Transactions on Visualization and Computer Graphics_ , vol. 28, no. 1, pp. 540–550, 2021.
  * [27] S. Maiella, A. Rath, C. Angin, F. Mousson, and O. Kremp, “Orphanet and its consortium: where to find expert-validated information on rare diseases,” _Revue neurologique_ , vol. 169, pp. S3–8, 2013.
  * [28] P. N. Robinson, “Deep phenotyping for precision medicine,” _Human mutation_ , vol. 33, no. 5, pp. 777–780, 2012.
  * [29] D. C. Wunsch III and D. B. Hier, “Subsumption reduces dataset dimensionality without decreasing performance of a machine learning classifier,” in _2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC)_. IEEE, 2021, pp. 1618–1621.
  * [30] D. B. Hier, R. Yelugam, M. D. Carrithers, and D. C. Wunsch III, “The visualization of orphadata neurology phenotypes,” _Frontiers in Digital Health_ , vol. 5, p. 1064936, 2023.
  * [31] L. van der Maarten and G. Hinton, “Visualizing data using t-sne.” _Journal of machine learning research_ , vol. 9, no. 11, 2008.
  * [32] T. Chen and C. Guestrin, “XGBoost: A scalable tree boosting system,” in _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , ser. KDD ’16. New York, NY, USA: ACM, 2016, pp. 785–794. [Online]. Available: http://doi.acm.org/10.1145/2939672.2939785
  * [33] S. M. Lundberg and S.-I. Lee, “A unified approach to interpreting model predictions,” in _Advances in Neural Information Processing Systems 30_ , I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds. Curran Associates, Inc., 2017, pp. 4765–4774. [Online]. Available: http://papers.nips.cc/paper/7062-a-unified-approach-to-interpreting-model-predictions.pdf
  * [34] D. Hier, T. Obafemi-Ajayi, G. R. Olbricht, D. M. Burns, S. Petrenko, and D. C. Wunsch, II, “Acil-group/centroids,” Jan. 2024. [Online]. Available: https://doi.org/10.5281/zenodo.10465863


# An Integrated Visual System for Unmanned Aerial Vehicles Tracking and
Landing on the Ground Vehicles

Kangcheng Liu∗, Xunkuai Zhou, Benyun Zhao, Huosen Ou, and Ben M. Chen
∗Kangcheng Liu is the corresponding author.

###### Abstract

The vision of unmanned aerial vehicles is very significant for UAV-related
applications such as search and rescue, landing on a moving platform, etc. In
this work, we have developed an integrated system for the UAV landing on the
moving platform, and the UAV object detection with tracking in the complicated
environment. Firstly, we have proposed a robust LoG-based deep neural network
for object detection and tracking, which has great advantages in robustness to
object scale and illuminations compared with typical deep network-based
approaches. Then, we have also improved based on the original Kalman filter
and designed an iterative multi-model-based filter to tackle the problem of
unknown dynamics in real circumstances of motion estimations. Next, we
implemented the whole system and do ROS Gazebo-based testing in two
complicated circumstances to verify the effectiveness of our design. Finally,
we have deployed the proposed detection, tracking, and motion estimation
strategies into real applications to do UAV tracking of a pillar and obstacle
avoidance. It is demonstrated that our system shows great accuracy and
robustness in real applications.

##  I Introduction and Related Work

Visual tracking and detection play a key role in all kinds of UAV navigation
applications [1]. It has a wide range of applications such as UAV search and
rescue, UAV detection and tracking, UAV surveillance and environmental
monitoring [2], security surveillance, geographical mapping, power-line, and
pipeline inspection, an autonomous inspection of large-scale bridges,
warehouse management, and logistic delivery. However, the visual tracking and
detection of the target objects are of great significance to improve the
autonomy of unmanned aerial systems. However, some great challenges remain.
First, the detection of the target object needs to be realized in real-time
for the UAV. Currently, most current deep neural network-based approaches
merely focus on the development of sophisticated network architecture, and
pre-training algorithms to solve the detection in diverse modalities. However,
the efficient algorithms which can be deployed on the UAV platform have not
been sufficiently explored. Also, the robustness is poor when faced with low
illuminations and rapid rotations. In order to track the UAV effectively, we
need to do the motion estimation and tracking of the target object to perform
the landing task.

Figure 1: The Detailed System Framework of the Computer Vision System for ROS-
based UAV Target Tracking and Landing. The color blue indicates our proposed
modules and the color green indicates other modules to fulfill the tracking
and landing task.

The Kalman Filter [3, 4, 5] has been proven extensively to be an effective
approach to achieving estimation of some dynamic variables given the sensor
measurements observed over a period of time. However, the traditional Kalman
Filter can not tackle the problem of sensor noise as well as the nonlinear
motion patterns of the target.

Yang et al. use a fuzzy logic complementary Kalman Filter (KF) based on visual
and IMU data for the landing of the UAV [6]. Yuan Wei et al. [7] use the
radars installed on vehicles or the UAVs for tracking applications. Ashraf
Qadir et al. use the onboard visual tracking system to implement a Kalman
Filter-based visual tracking system, and the system is capable of continuously
detecting the object if the tracking failure occurs [8]. But the real flight
test of them remains the future work. Zhao [9] et al. proposes a visual ground
target tracking strategy for the rotorcraft UAV. Oh, et al. propose an
autonomous visual tracking algorithm with Extended Kalman Filter (EKF) for
micro aerial vehicles [10]. They have proposed an efficient object-tracking
algorithm for UAVs, and effective ground object tracking can be achieved.
Recently, various learning-based methods have been proposed for object
segmentation, detection and tracking [11, 12, 13, 14, 15, 16, 17, 18], but the
deep learning-based methods suffer greatly from the poor generalization
capacity and the large computational and memory cost [19, 20, 21].

In order to tackle the problems above, in the vision-based object detection
and tracking, we have proposed to use a Laplacian of Gaussian filter and used
it to construct a convolutional network, which makes it more appropriate for
real-time object detection. It is demonstrated that our method can achieve
real-time performance in an unknown environment. The recognition rate can
achieve 45 frames per second, which fulfills the real-time requirements. The
UAV vision-based applications are very fundamental to all related applications
[15, 22, 23]. However, great challenges remain. The first is that the typical
visual detection system can not handle the rotation of the targeted object and
low illuminations, which makes the subsequent tracking and landing difficult.
The second is that the previous Kalman filter-based motion estimation suffers
from low accuracy and will greatly decrease the success rate in fulfilling the
task of tracking and landing. As shown in Fig. 2, to tackle the challenges
mentioned above, in this paper, we have proposed an integrated system for UAV
tracking and landing applications. Taking the RGB-D images as input, we
utilize our proposed Laplacian of Gaussian (LoG) filter to construct the deep
neural networks to perform the object detection, which achieves robustness and
accuracy under low illumination. We have proposed to use iterative multi-model
methods based on the original Kalman Filter to improve the accuracy in
tracking and motion estimation. Also, we have integrated our proposed approach
with other robotics modules such as SLAM and motion/task planning as a whole
system to perform UAV-based tracking and landing of the target objects in real
applications. The deep learning-based methods have been demonstrated to be
very effective in object recognition and tracking [13, 14, 11, 24, 25, 26, 22,
27]. In summary, we have the following prominent contributions:

  1. 1.

We have proposed a general network for object detection and integrated it with
ROS for real robotics search and rescue applications. Moreover, we have
integrated the Laplacian of Gaussian (LoG) filter into the deep neural
networks and it is demonstrated that the LoG-based method has a great
advantage in robustness to object scale and illuminations.

  2. 2.

We have done real experiments to demonstrate the effectiveness of our proposed
approach. It turns out that the IMM-based filter in motion estimation shows
satisfactory accuracy under various circumstances. We have also done real UAV
experiments to demonstrate the effectiveness of our design.

  3. 3.

We have also integrated our method with the point clouds segmentation methods
for dynamic objects removal, and also we have integrated the proposed approach
with motion planning approaches, which realize the real demos of the UAV
landing on the UGV moving platform, as well as UAV based motion estimation of
the moving/boxes pillars.

##  II Proposed Methodology

In this work, we have two prominent contributions. The first is that we
proposed Laplacian of Gaussian (LoG) filter to construct the deep neural
networks to perform the object detection. The second is that we have proposed
to use iterative multi-model methods based on the original Kalman Filter to
improve the accuracy in tracking and motion estimation. The details of these
two contributions will be illustrated in detail in the remaining of this
Section.

Figure 2: Detailed illustration of the proposed object detection system. The
green parts are the user configuration and input configuration of the detailed
model. The blue part is the deep learning-based model. The orange part is for
the visualization and publishing of our detection results. And the red part is
for the image or video stream input.

###  II-A The Iterative Multi-model Filter for dynamic object tracking

####  II-A1 Kalman Filter

Kalman filter is an algorithm that uses linear system state equations to
optimally estimate system state through system input and output observation
data. Since the observation data includes the influence of noise and
interference in the system, the optimal estimation can also be regarded as a
filtering process. The core meaning is that the Kalman filter can estimate the
state from the noisy data process very well. Moreover, the Kalman filter is
also one of the breakthrough technologies used in Apollo’s moon landing.
Kalman filtering is also a recursive filtering algorithm based on state space
in the time domain, which is easy to be realized in real time on computer, and
has a small amount of computation and storage. This method can deal with the
filtering problem of multi-variable non-stationary random processes and the
filtering problem of time-varying systems. For example, in the course of the
flight, the disturbance encountered by an aircraft is usually time-varying
non-stationary noise. At this time, the Kalman filter can be used to
effectively remove the interference and obtain more real state estimation
data. To improve the detection and tracking performance, and improve the
precision of positional prediction, in our application, we also develop the
real-time object tracking algorithm based on the basic idea of the Kalman
filter. The Kalman filter is utilized in this experiment to remove the noise
in the observer and controller of the control system and minimize the number
of squared errors. The advantage of the Kalman filter is that it can make the
optimal state estimation of the system by taking advantage of measurement
data. Kalman filter is essential because the noise and disturbance in the
system influence the true data in the measurement. For autonomous Unmanned
aerial vehicles in our applications, denote the initial state matrix of the
UAV as $X$, the initial process covariance matrix as $P$, which denotes the
error in the state estimation. Next, the initial state becomes previous.
Utilize subscript $K$ to represent each state in the iteration cycle. In the
next time step, the current state becomes the previous one. Then the new state
can be predicted based on the physical model and previous state, which can be
formulated as:

$X_{K}=AX_{K-1}+BU_{K-1}+W_{K-1}$ (1)
$Z_{k}^{{}^{\prime}}=AP_{K-1}A^{T}+Q_{k}$ (2)

The matrix $A$ is the $n\times n$ system matrix. The matrix $B$ represents the
function of the input to state, which is called the input matrix or control
matrix. The matrix $W_{K-1}$ denotes the predicted state noise matrix. Where
the matrix $U$ denotes the control variable matrix, $Q$ denotes the process
noise covariance matrix, which keeps the state covariance matrix from becoming
too small or going to zero. Denote $P_{k}^{{}^{\prime}}$ as the prior
estimation of the state, it can be represented as:

$P_{k}^{{}^{\prime}}=AP_{K-1}A^{T}+Q_{k}$ (3)

Let $A,B$ and $C$ denote the adaptation matrices, which convert the input
state to the process state. And $Y$ denotes the measurement of state, $Z$
denotes the measurement noise. Then the measurement from sensors can be
denoted as:

$Y_{k}=CX_{K}^{\star}+Z_{k}$ (4)

According to the , we can calculate the Kalman Gain $K$ as:

$K=\frac{P_{k}^{{}^{\prime}}H}{HP^{{}^{\prime}}_{k}{H}^{T}+R}$ (5)

Where $K$ is the Kalman Gain, $R$ is the sensor noise or the measurement
covariance matrix. And $H$ is the conversion matrix to make the size
consistent. Then the state update can be formulated as:

$X_{k}=X_{k}^{{}^{\prime}}+K(Y_{k}-HX_{k}^{{}^{\prime}})$ (6)

And the covariance update can be formulated as:

$P_{k}=(I-KH)P_{k}^{{}^{\prime}}$ (7)

Where $I$ is the identity matrix. Then for the $k_{th}$ time step, the state
matrix $X_{k}$ and the process covariance matrix which represents an error in
the estimate can be obtained. Note that the adjustment of Kalman Gain is
essentially the adjustment of the noise value of $Q$ and $R$. Note that:

  1. 1.

The smaller the $K$ is, we can trust more on the estimation of the model.

  2. 2.

The bigger the $K$ is, we can trust more on the estimation of the sensors.

  3. 3.

The value of $K$ is related to the accuracy of the sensors and the error
within the environment.

It can be seen from the Eq. 6 that, we directly use the difference between the
prediction value and the estimation value. We use the parameter $K$ to
determine we trust more on the observation value $Z$ or the prediction value
$X$.

We apply the Kalman filter to the horizontal position and velocity of the UAV
on the world coordinate. Because the x-direction and the y-direction of the
horizontal coordinate are independent of each other, we merely need to set a
directional state in the Kalman Filter. The state $x$ can be the position in
the UAV-based visual tracking of moving object application using RGB-D camera.
The position and velocity of the ground vehicle to be tracked can be set as:

$x=\left[\begin{array}[]{ccc}x\\\ vx\\\ \end{array}\right]$ (8)

The next state estimation of the vehicle can be represented as:

$x^{-}_{k+1}=\left[\begin{array}[]{ccc}x^{-}_{k+1}\\\ vx^{-}_{k+1}\\\
\end{array}\right]=\left[\begin{array}[]{ccc}x_{k}+vx_{k}\Delta t+w_{k}\\\
vx_{k}+w_{v,k}\\\ \end{array}\right]$ (9)

Then we can also obtain the system matrix $A$ from the partial derivative of
$f$ with respect to $x_{k}$ as follows:

$A=\left[\begin{array}[]{ccc}1&\delta t\\\ 0&1\\\ \end{array}\right]$ (10)

Then we can also obtain the $P_{k}^{{}^{\prime}}$, which is the prior
estimation of the state, it can be represented as:

$P_{k}^{{}^{\prime}}=AP_{K-1}A^{T}+Q_{k}$ (11)

Also, the $K,X_{k},P_{k}$ can be obtained which are the Kalman Gain, the
state, and the covariance update. Then the motion estimation of the target UGV
can be achieved.

Figure 3: The Coordinate Transformation Illustrations.

####  II-A2 Proposed Iteractive Multi-model (IMM) Methods

However, in real systems, the object may have great mobility, and it may take
sudden turning and acceleration. Merely utilizing the original Kalman Filter
may not realize the most ideal results, and adaptive methods must be taken.
The iterative multi-model (IMM) methods [28] overcome the limitations
mentioned above. The output of the filter will be the weighted average of
estimation from multiple filters. The weight is the probability of correctly
describing the model at the current moment.

####  II-A3 Proposed Laplacian of Gaussian-based Object Tracking Methods

We have proposed the Laplacian of Gaussian-based object tracking results. And
we have fused the Laplacian of Gaussian (LoG) operator into modern deep neural
networks such as ResNext. The ResNext-based network has acceptable efficiency
and we have designed methods to integrate the LoG operator seamlessly into the
modern ResNext architecture. We have utilized some of our previous network
designs mentioned in our FG-Net [11, 12, 29, 13, 14, 30, 31]. According to our
experiments, utilizing our proposed Laplacian of Gaussian operator, the
performance of object detection and tracking can be significantly boosted. And
we can still maintain the real-time performance and inference speed in
fulfilling the tasks of aerial UAV-based object detection and tracking.

Figure 4: Detailed illustration of our proposed system architecture Figure 5:
The Real-world UAV-Based Tracking Results. The tracked objects are denoted by
the bounding boxes. Figure 6: Real-world experiments of the object tracking of
the moving boxes (pillars). The figure below shows the real experiments of the
UAV tracking and the figure above shows the simulation of the UAV tracking of
the moving boxes (pillars). The IMM-based filter is utilized to do the motion
estimation, and the LoG filter-based deep neural network is utilized to detect
and track the moving pillars. TABLE I: The Comparisons Results of the
detection performance in with fast rotation in the object box of 30 ∘ per
second.

Methods | Success | Failure in Detect
---|---|---
Ours (LoG Network) | 7 | 0
Typical DCNN of ResNeXt [32] | 5 | 2
Tradictional HOG based Method [33] | 2 | 5

TABLE II: The Comparisons Results of the detection performance with very low
illumination.

Methods | Success | Failure in Detect
---|---|---
Ours (LoG Network) | 7 | 0
Typical DCNN of ResNext [32] | 3 | 4
Tradictional HOG based Method [33] | 1 | 6

TABLE III: The Comparisons of The Tracking Accuracy and Computational Cost
Tested in the Real scenario tracking a Moving Pillar.

Methods | Computational Time (ms/Image) | Max Error (cm)
---|---|---
Ours (w/o LoG based Neural Net, w/ IMM) | 51.58 | 2.11
Ours (Utilized the original Kalman Filter) | 75.8 | 3.85
Ours (Utilized the IMM Filter) | 76.3 | 1.57

##  III Experimental Results

###  III-A Detection and Tracking Results

We have summarized the related results for detection and tracking in Table I
for fast rotations and Table II for low illumination. It can be demonstrated
our proposed approach has satisfactory performance under fast rotation and low
illumination compared with ResNeXt-based methods and traditional methods. For
our LoG based network, we adopt the same experimental settings with the state-
of-the-art approach ResNext [32]. It can be seen that the our proposed
approach will not fail with fast rotations and very low illuminations.
Therefore, the robustness of our proposed approach is demonstrated.

###  III-B Motion Estimation Performance

We have summarized the related results of motion estimation projected onto 2D
planes and the real experiments of the motion estimation of the moving pillar
by the onboard Nvidia TX2 GPU on the UAV, as shown in Fig. 6. It is
demonstrated by experiments that the object’s position can be tracked very
precisely. Also, we have compared our proposed approach and traditional Kalman
Filter based approaches for motion estimation. As shown in Table III, our
proposed IMM filter based object tracking can realize more accurate object
motion estimation. The maximum error in the motion estimation is 1.57 cm,
which reduces the maximum tracking error by 2.28 cm compared with the original
Kalman Filter. Also. it can be demonstrated that the LoG based neural network
for detection also enhances the motion estimation accuracy. In conclusion, our
proposed approach can achieve accurate motion estimation with satisfactory
efficiency. As shown in Fig. 6, the real experiments of the object tracking of
the moving boxes (pillars) and the related simulations are demonstrated. The
figure below shows the real experiments of the UAV tracking and the figure
above shows the simulation of the UAV tracking of the moving boxes (pillars).
The IMM-based filter is utilized to do the motion estimation, and the LoG
filter based deep neural network is utilized to detect the moving pillars.

###  III-C Final Demos of UAV landing on the Moving Platform

Finally, we have deployed our method for a real application case of the UAV
landing on the moving platform. We use the proposed network to do object
detection and tracking. Based on the detection and tracking results, we use
our proposed IMM filter to do the motion estimation as well as to track the
target object. The simulation experiments of UAV tracking and landing on the
moving UGV platform through obstacles are shown in Fig. 5 and Fig. 6. The UAV
can take off and detect the objects precisely, and then track the object
continuously. The UAV can autonomously approach the target, and then landing
on the moving platform. Finally, the co-moving of the UAV and UGV can be
achieved with great robustness. Also, as shown in Fig. 5 and Fig. 6. the UAV
can land very precisely, thanks to the accurate motion estimations of position
and velocity. The demos of the UAV tracking of the UGVs through various of
obstacles in the road circumstances are illustrated in Fig. 5 and Fig. 6. It
can be demonstrated that the UAV can take off and approach the target
autonomously, and then track the target UGV continuously in the obstacle. The
UAV and the UGV can navigate intelligently through the obstacles. Finally, the
UAV can find the way out with the UGV. The real experiments of UAV landing on
the moving platform is shown in Fig. 5 and 6, it can be seen that the UAV can
achieve accurate and robust landing with our proposed approaches. We have
compared the tracking trajectory of the UAV compared with the target ground
truth trajectory. The UAV can realize very precise tracking of the target. It
further demonstrates that our proposed approach can successfully fulfill the
UAV tracking and landing task with satisfactory accuracy and robustness in
motion estimation.

##  IV Conclusions

In this work, we have proposed an integrated visual system for unmanned aerial
vehicles landing on the targeted movingc UGV platform. We have proposed a
systematical design of the vision systems for the UAV tracking and landing
applications. Firstly, we have integrated the Laplacian of Gaussian filter
into deep neural networks. It has shown great merits in robustness to rotation
and low illumination. Secondly, we have designed an iterative multi-model
Kalman filter adapted based on the original Kalman Filter for object tracking,
which achieves great accuracy. Finally, we have integrated our proposed
approach with other robotics modules such as SLAM and motion/task planning as
a whole system to perform UAV tracking and landing in real applications. UAV
Vision is important [34, 35]. Our integrated visual system is very important
for future UAV-based detection and tracking applications such as autonomous
landing on moving vehicles.

## References

  * [1] H. Duan, P. Li, Y. Shi, X. Zhang, and C. Sun, “Interactive learning environment for bio-inspired optimization algorithms for uav path planning,” _IEEE Transactions on Education_ , vol. 58, no. 4, pp. 276–281, 2015.
  * [2] K. Liu, Y. Qu, H.-M. Kim, and H. Song, “Avoiding frequency second dip in power unreserved control during wind power rotational speed recovery,” _IEEE transactions on power systems_ , vol. 33, no. 3, pp. 3097–3106, 2017.
  * [3] G. Welch, G. Bishop _et al._ , “An introduction to the kalman filter,” 1995\.
  * [4] K. Liu, “A robust and efficient lidar-inertial-visual fused simultaneous localization and mapping system with loop closure,” in _2022 12th International Conference on CYBER Technology in Automation, Control, and Intelligent Systems (CYBER)_. IEEE, 2022, pp. 1182–1187.
  * [5] K. Liu, X. Zhou, B. Zhao, H. Ou, and B. M. Chen, “An integrated visual system for unmanned aerial vehicles following ground vehicles: Simulations and experiments,” in _2022 IEEE 17th International Conference on Control & Automation (ICCA)_. IEEE, 2022, pp. 593–598.
  * [6] Q. Yang and L. Sun, “A fuzzy complementary kalman filter based on visual and imu data for uav landing,” _Optik_ , vol. 173, pp. 279–291, 2018.
  * [7] Y. Wei, T. Hong, and M. Kadoch, “Improved kalman filter variants for uav tracking with radar motion models,” _Electronics_ , vol. 9, no. 5, p. 768, 2020.
  * [8] A. Qadir, J. Neubert, W. Semke, and R. Schultz, “On-board visual tracking with unmanned aircraft system (uas),” in _Infotech@ Aerospace 2011_ , 2011, p. 1503.
  * [9] X. Zhao, Q. Fei, and Q. Geng, “Vision based ground target tracking for rotor uav,” in _2013 10th IEEE International Conference on Control and Automation (ICCA)_. IEEE, 2013, pp. 1907–1911.
  * [10] S.-M. Oh and E. Johnson, “Relative motion estimation for vision-based formation flight using unscented kalman filter,” in _AIAA guidance, navigation and control conference and exhibit_ , 2007, p. 6866.
  * [11] K. Liu, Z. Gao, F. Lin, and B. M. Chen, “Fg-net: Fast large-scale lidar point cloudsunderstanding network leveraging correlatedfeature mining and geometric-aware modelling,” _arXiv preprint arXiv:2012.09439_ , 2020.
  * [12] ——, “Fg-conv: Large-scale lidar point clouds understanding leveraging feature correlation mining and geometric-aware modeling,” in _2021 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2021, pp. 12 896–12 902.
  * [13] K. Liu, X. Han, and B. M. Chen, “Deep learning based automatic crack detection and segmentation for unmanned aerial vehicle inspections,” in _2019 IEEE International Conference on Robotics and Biomimetics (ROBIO)_. IEEE, 2019, pp. 381–387.
  * [14] Z. Yuzhi, P. Lai-Man, W. Xuehui, L. Kangcheng, Z. Yujia, Y. Wing-Yin, X. Pengfei, and X. Jingjing, “Legacy photo editing with learned noise prior,” _arXiv preprint arXiv:2011.11309_ , 2020.
  * [15] K. Liu and H. Ou, “A light-weight lidar-inertial slam system with high efficiency and loop closure detection capacity,” in _2022 International Conference on Advanced Robotics and Mechatronics (ICARM)_. IEEE, 2022, pp. 284–289.
  * [16] K. Liu, “Semi-supervised confidence-level-based contrastive discrimination for class-imbalanced semantic segmentation,” in _2022 12th International Conference on CYBER Technology in Automation, Control, and Intelligent Systems (CYBER)_. IEEE, 2022, pp. 1230–1235.
  * [17] K. Liu, Y. Qu, H.-M. Kim, and H. Song, “Avoiding frequency second dip in power unreserved control during wind power rotational speed recovery,” _IEEE Transactions on Power Systems_ , vol. 33, no. 3, pp. 3097–3106, 2018.
  * [18] Y. Zhao, L.-M. Po, T. Lin, X. Wang, K. Liu, Y. Zhang, W.-Y. Yu, P. Xian, and J. Xiong, “Legacy photo editing with learned noise prior,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2021, pp. 2103–2112.
  * [19] K. Liu, “Robust industrial uav/ugv-based unsupervised domain adaptive crack recognitions with depth and edge awareness: From system and database constructions to real-site inspections,” in _Proceedings of the 30th ACM International Conference on Multimedia_ , 2022, pp. 5361–5370.
  * [20] K. Liu and B. M. Chen, “Industrial uav-based unsupervised domain adaptive crack recognitions: From system setups to real-site infrastructural inspections,” _IEEE Transactions on Industrial Electronics_ , 2022.
  * [21] K. Liu, Y. Zhao, Q. Nie, Z. Gao, and B. M. Chen, “Weakly supervised 3d scene segmentation with region-level boundary awareness and instance discrimination,” in _European Conference on Computer Vision 2022 (ECCV 2022)_. Springer, Cham, 2022, pp. 37–55.
  * [22] K. Liu, “An integrated lidar-slam system for complex environment with noisy point clouds,” _arXiv preprint arXiv:2212.05705_ , 2022.
  * [23] K. Liu, A. Xiao, J. Huang, K. Cui, Y. Xing, and S. Lu, “D-lc-nets: Robust denoising and loop closing networks for lidar slam in complicated circumstances with noisy point clouds,” in _2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2022, pp. 12 212–12 218.
  * [24] K. Liu, X. Zhou, and B. M. Chen, “An enhanced lidar inertial localization and mapping system for unmanned ground vehicles,” in _2022 IEEE 17th International Conference on Control & Automation (ICCA)_. IEEE, 2022, pp. 587–592.
  * [25] K. Liu, Y. Zhao, Z. Gao, and B. M. Chen, “Weaklabel3d-net: A complete framework for real-scene lidar point clouds weakly supervised multi-tasks understanding,” in _2022 International Conference on Robotics and Automation (ICRA)_. IEEE, 2022, pp. 5108–5115.
  * [26] K. Liu and H. Ou, “A light-weight lidar-inertial slam system with loop closing,” _arXiv preprint arXiv:2212.05743_ , 2022.
  * [27] K. Liu, “An enhanced lidar-inertial slam system for robotics localization and mapping,” _arXiv preprint arXiv:2212.05743_ , 2022.
  * [28] L.-Q. Li, Y.-c. Sun, and Z.-X. Liu, “Maximum fuzzy correntropy kalman filter and its application to bearings-only maneuvering target tracking,” _International Journal of Fuzzy Systems_ , vol. 23, no. 2, pp. 405–418, 2021\.
  * [29] K. Liu, Z. Gao, F. Lin, and B. M. Chen, “Fg-net: A fast and accurate framework for large-scale lidar point cloud understanding,” _IEEE Transactions on Cybernetics_ , 2022.
  * [30] G. Yang*, K. Liu*, J. Zhang, B. Zhao, Z. Zhao, X. Chen, and B. M. Chen, “Datasets and processing methods for boosting visual inspection of civil infrastructure: A comprehensive review and algorithm comparison for crack classification, segmentation, and detection,” _Construction and Building Materials_ , vol. 356, p. 129226, 2022.
  * [31] K. Liu, Y. Zhao, Q. Nie, Z. Gao, and B. M. Chen, “Ws3d supplementary material,”,” in _European Conference on Computer Vision (ECCV). Springer, Cham_ , 2022, pp. 37–55.
  * [32] S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He, “Aggregated residual transformations for deep neural networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017, pp. 1492–1500.
  * [33] N. Dalal and B. Triggs, “Histograms of oriented gradients for human detection,” in _2005 IEEE computer society conference on computer vision and pattern recognition (CVPR’05)_ , vol. 1. Ieee, 2005, pp. 886–893.
  * [34] K. Liu, G. Yang, J. Zhang, Z. Zhao, X. Chen, and B. M. Chen, “Datasets and methods for boosting infrastructure inspection: A survey on defect segmentation and detection,” in _2022 IEEE 17th International Conference on Control & Automation (ICCA)_. IEEE, 2022, pp. 23–30.
  * [35] K. Liu, “Rm3d: Robust data-efficient 3d scene parsing via traditional and learnt 3d descriptors-based semantic region merging,” _International Journal of Computer Vision_ , vol. 1, no. 3, pp. 1–30, 2022.


#  $ER=EPR$ and Non-perturbative action integrals for quantum gravity

Salwa Alsaleh<EMAIL_ADDRESS>Department of Physics and Astronomy, King
Saud University, Riyadh 11451, Saudi Arabia  Lina Alasfar
<EMAIL_ADDRESS>Université de Clermont Auvergne, 24 Avenue des Landais
F-63177 Aubière Cedex, France

(27th August 2024)

###### Abstract

In this paper, we construct and calculate a non-perturbative path integrals in
a multiply-connected space-time. This is done by by summing over homotopy
classes of paths. The topology of the space-time is defined by Einstein-Rosen
bridges (ERB) forming from the entanglement of quantum foam described by
virtual black holes. As these ‘bubbles’ are entangled, they are connected by
Plankian ERB’s because of the $ER=EPR$ conjecture. Hence the space-time will
possess a large first Betti number $B1$. For any compact 2-surface in the
space-time, the topology (in particular the homotopy) of that surface is not-
trivial, due to the large number of Plankian ERB’s that define homotopy though
this surface. The quantisation of space-time with this topology - along with
the proper choice of the 2-surfaces - is conjectured to allow anon-
perturbative path integrals of quantum gravity theory over the space-time
manifold.

###### pacs:

04.70.Dy , 04.60.Kz, 04.60.Pp, 04.60.-m

##  I Introduction

In the past few years, a new approach in understanding and constructing
quantum geometry had emerged from linking geometric quantities to information-
theoretic one. This idea first emerged from the calculation of entropy of
spherically symmetric isolated horizons (SSIH) in Loop quantum gravity
immirzi1997real ; PhysRevLett.80.904 . Later, the emergence of holographic
principle to include the ‘entanglement entropy’ of the vacuum to be linked to
the are of the surface enclosing a geodesic-generated ball
reznik2003entanglement . A deeper link link between quantum geometry and
information theory was established ER1 , as a solution to AMPS / Firewall
paradox almheiri2013black . This solution suggested that for two black holes
collapsed from an entangled matter, these black holes would be connected by an
Einstein-Rosen Bridge (ERB). This conjecture became widely known as the
$ER=EPR$ conjecture. One can go even further with conjecture and claim that
for any pair of entangled particles, there is a Plankian ERB, that matter will
not fall into it. However, this can considered in an abstract geometric sense
ER2 .

The development of this conjecture has raised the possibility that space-time
could be interwoven by entanglement, and wormholes could be holding the space-
time together, This idea has been applied in category theory and topological
quantum field theory to show that quantum fluctuations correspond to formation
of wormholes and vice versa baez2014wormholes . Technically speaking, the
$ER=EPR$ raises the possibility that space-time could multiply connected.
However, how one could start doing physics in multiply-connected space-time ?
More importantly, how to formulate the physic of multiply connected space-
time; and would it help to do so ?

The most basic way to start is by replacing the points of the space-time
manifold $\mathcal{M}$ by a non-Abelian algebra generators,establishing non-
commutative geometry Nicolini:2005vd ; Gingrich:2010ed ,

$[q^{\mu},q^{\nu}]=i\ell_{p}^{2}g^{\mu\nu}.$ (1)

This is a direct result of combining gravitation theory with uncertainty
relation of quantum mechanics. It is immediately recognised that $\ell_{p}$ is
the Plank length, and the relation (1) implies that space-time at the Plank
scale consists of what-so-called quantum foam wheeler1 . First described by J.
Wheeler and later studied in detail by S. Hawking hawking1996virtual , this
quantum foam consists of a sea of virtual black holes, which Hawking described
them as bubbles. These bubbles make the space-time posses a large second Betti
number. Moreover, in his paper, Hawking argued that the space-time is simply
connected and admits a topology $S^{2}\times S^{2}$ or $K_{3}\times K_{3}$
…etc.

Nevertheless, these bubbles can form the multiply-connected space-time we are
looking for if we considered the $ER=EPR$ conjecture. In this paper, we shall
discuss how to define a multiply-connected space-time combing both ideas
discussed above. Next, show how this picture could help in the theory of
quantum gravity, after that an application for this conjecture is discussed at
the hand-waving level by recovering the Bekenstein-Hawking formula for Black
hole entropy. Finally, we will point out the possible steps for this programme
that ought to be taken in order to make the argument in this paper more
formal.

##  II The space-time at the Plank scale

Let $\mathcal{M}$ be a 4-dimensional $C^{k}$-manifold that admits a topology
$\mathbb{R}\times\Sigma$, we deal with globally hyperbolic space-time in order
to prepare for canonical formalism of gravity. If any field was put on
$\mathcal{M}$, by assigning smooth sections
$\varsigma(t,x);t,x\in\mathbb{R}\times\Sigma$ over $\mathcal{M}$ to an
associated bundle $\mathcal{F}$. This field is quantised by standard canonical
quantisation rules, taking the field to be a scalar field
$\phi(t,x)=\varsigma(t,x)$, the quantisation for the field is given by the
commutation relation for $\phi$ and its conjugate momentum $\pi(t,x)$ which is
itself a section $\varpi(t,x)=\frac{d\varsigma(t,x)}{dt}$,

$\displaystyle[\phi(t_{0},x),\phi(t_{0},x^{\prime})]=0,$
$\displaystyle[\pi(t_{0},x),\pi(t_{0},x^{\prime})]=0,$
$\displaystyle[\phi(t_{0},x),\pi(t_{0},x^{\prime})]=i\delta(x-x^{\prime}).$
(2)

These rules imply a Fock space 111 Since we are dealing with quantum field
theory on curved manifold, the Fock space and vacuum spates are by no means
unique and thereby we can define a Hamiltonian $H$, whose expectation value
over any vacuum state diverges as it is known from standard quantum field
theory. The only non- perturbative method to remove such divergence is to
impose some UV-cutoff on the modes of the filed. This cutoff condition can be
made because of the gravity theory present on $\mathcal{M}$. For general
relativity (GR) one can find this cutoff term to be for modes with wavelengths
less than or equal to the plank length $\ell_{p}=1$ in Plankian units In the
semi-classical GR, the geometry itself fluctuates like the quantum fields that
lie on it to show this take a small 3-space like surface $\sigma$, let it be a
geodesic-generated ball with radius $l$. We now integrate Einstein field
equations over this ball (in some chart defined over that small ball),

$\frac{1}{4\pi}\int_{\sigma}d\sigma^{\nu}G_{\mu\nu}=2\int_{\sigma}d\sigma^{\nu}\langle\
\hat{T}_{\mu\nu}\rangle.$ (3)

The LHS is equal to the radius of curvature domain of small region of space
time $R_{\mu}$. For the RHS, we consider the stress energy tensor of the
scalar field $\phi$ we get,

$R_{\mu}=\int d^{3}k\left(\omega_{k}+k_{i}\right)\langle
0|\hat{a}^{\dagger}_{k}\hat{a}_{k}+\hat{a}_{k}\hat{a}^{\dagger}_{k}|0\rangle.$
(4)

The operators $\hat{a}^{\dagger}_{k},\hat{a}_{k}$ are the creation and
inhalation operators of the defined Fock space. Here, the Latin indices run
from $1$ to $3$ and the Greek ones run from $0$ to $3$, and $\omega_{k}$ is
the normal mode given by $\sqrt{k_{j}k^{j}}$. The expression (4) can be
further treated yielding,

$R_{\mu}=\int d^{3}k\sum_{k=0}^{\infty}\omega_{k}$ (5)

Thus the radius of curvature domain is divergent, implying that the geometry
is not supposed to exist with field fluctuating like that. Hence, locality and
flat geometry cannot coexist. To get rid of that divergence we use the
uncertainty relation $\Delta R_{\mu}x_{\mu}\sim 1$ . This is derived from
Hawking’s assumption about the quantum structure of space-time at the micro
scale hawking1996virtual , as a sea of virtual blackholes. By this assumption,
we cannot define the field nor the geometry at one point, but rather a region
with minimal length. Thus geometry fluctuation is a key element in the
stability of classical geometry. Concluding geometry must me quantised, i.e.
the space-time is made from discrete bubbles of geometry. This result is
obtained without assumptions outside GR and quantum theory.Virtual black holes
were also studied in third quantised formalism of canonical quantum gravity
faizal2012some , in dilatorn gravity theories
LABEL:Grumiller:2002dm,Grumiller:2001rg,Grumiller:2000ah, and in other
alternative theories as in generalised uncertainty principle and their
phenomenological implications in proton decay alsaleh2017virtual  and higher
spin theories from Teukolsky equations Prestidge:1998bk . Virtual black holes
are also important in understanding the black hole information paradox
Calmet:2014uaa , for example, it has been shown that the S-matrix of
scattering processes by virtual black hole is incoherent Hawking:1997ia ,
leading to a possible indication that information paradox could be resolved by
studying the phenomenology of scattering by virtual black holes
Grumiller:2004yq .

The predictions of quantum mechanics, these nearby virtual black holes are
entangled. In order to show this, consider black holes in a box , each black
hole will emit Hawking radiation to be absorbed by the others, even when the
system reaches thermal equilibrium, this process will continue to happen. If
we assign a density matrix for this system, it will evolve to a mixed states
density matrix over time, even if we started with a pure state (each state
describes a Black hole.). Viz., black holes will get entangled by absorbing
each other’s Hawking radiation ER1 .

Similar argument can be made to the virtual black holes, that make up the
quantum foam. Another way to show that the quantum foam is indeed entangled,
and hence is filled with Plankian Wormholes, is the work done by
baez2014wormholes , using category theory, they concluded that the pair
production of quantum fluctuation correspond to formation of a wormhole.
Although their argument was for topological field theories, it can be proven
that this argument could be extended to general relativity if we considered it
as a form of BF theory. Identifying the form $B=\ast[e\wedge e]$, where $e$
are the triebein fields corresponding to the 3 metric $\gamma$ of $\Sigma$,
then the Palatini action reads,

$S_{P}=\int_{\Sigma}\text{Tr}(\ast[e\wedge e]\wedge F).$ (6)

However, it remains to be proven explicitly that these results will hold for
full relativistic gravity theory.

Assigning to each unit volume on the 3 spacelike surface $\Sigma$ a quantum
state $|\psi\rangle$, this unit area can carry only 1 bit of information,
either it contains a virtual black hole $|1\rangle$ or not $|0\rangle$ Hence,
we can write the quantum state for such unit area,

$|\psi\rangle=\frac{1}{\sqrt{2}}\left(|1\rangle+|0\rangle\right).$ (7)

We observe that given any 2-surface $\Delta$, enclosing a region of $\Sigma$ ,
we can establish near the boundary states of the volumes in the ‘in’ and ‘out’
regions. These states describe entangled states,

$|\Psi\rangle=\frac{1}{\sqrt{2}}\left(|1_{in}\rangle\otimes|1_{out}\rangle+|0_{in}\rangle\otimes|0_{out}\rangle\right).$
(8)

The state $|\Psi\rangle$ belongs to the Hilbert space
$\mathcal{H}=\mathcal{H}_{in}\otimes\mathcal{H}_{out}$ of the space-time
volumes. This construction of space-time from entanglement does not require
holography as in van2010building , but basically shares the same aim. In order
to do so, it is needed to define the topology of the space-time having this
construction.

##  III  The topology of the entangled quantum foam

Since these volume elements (making up) the quantum foam are entangled, using
the $ER=EPR$ conjecture, each bubble of geometry is connected to another by a
wormhole/ ERB. Making the space-time multiply connected. However, in 3
manifolds, we cannot clearly define the first Betti number corresponding the
the first fundamental group. The first fundamental group, appears (well
defined) for 2 surfaces. Taking the surface $\Delta$ in the previous section,
and analysing its homotopy we find that, and due to the $ER=EPR$. We can
define loops between the ‘in’ and ‘out’ regions that cannot be shrunk to a
point. These loops are made by passing through the wormholes made by the
entangled bubbles. The surface $\Delta$ is them homeomorphic to an
$N$-punctured sphere, $N+1$ being proportional to the area of the surface in
Plank-area units.

The first fundamental group for the surface $\Delta$ is generated by $N$
generators, each go around each puncture made by a wormhole. Note that we
should only consider bubbles entangled in the region $Ar(\Delta)\pm 1$, the
entanglement for bubbles further from Plank length is negligible. Hence the
rank of this free group is proportional to $Ar(\Delta)$. The calculation of
the proportionality constant is rather difficult, because we need to take into
an account that the classical area is obtained only by filling the surface
with infinite circles, while in quantum geometry, the ’quantum’ area is a bit
less, due to packing factor (see Tammes Problem, Ref tammes1930origin ). The
exact number of generator is the number of wormholes forming between bubbles
filling the area $Ar(\Delta-1)$ and $Ar(\Delta+1)$, assuming all the black
holes in these two layers are entangled, the number of the wormholes forming
is the area of the surface $\Delta$ that comes between the layers multiplied
by a filling factor, i.e. how many bubbles one can fit in the volume covered
by $Ar(\Delta+1)$ minus the volume covered by $Ar(\Delta-1)$.

##  IV Calculation of path integrals

The topology of space-time could allow a non perturbative calculation of path
integrals, automatically removing divergences that appear in the standard path
integrals. Since we have found$\mathcal{M}$ to be multiply connected, we can
look at its universal covering space, and preform path integration on the
covering space. Instead of integrating over all paths of the same homotopy
class, one can instead integrate over homotopy classes in the universal cover.
Nevertheless, this is not helpful in unbounded spaces, as one still gets
infinite classes to integrate over. Rather, one may take two points in
$\mathcal{M}$, $p,q$. Surround $p$ with a 2-spacelike surface $\sigma_{p}$ and
$q$ with another $\sigma_{q}$. Homotopy classes of paths between $p$ and $q$
will be finite due to the topology of the surfaces $\sigma_{p}$ and
$\sigma_{q}$. One can preform a non perturbative integration over the homotopy
classes.

Nevertheless, there is a rather peculiar feature of this technique, that is
the dependence of the value of the path integral of the choice of surfaces. It
remains an unsolved problem that needs special attention. This ‘freedom’ in
choosing the surfaces, could correspond to gauge freedom, that one can mod
out. Alternatively, there is no freedom and it is fixed by the theory of
gravity, like the case with the horizon of a black hole, this surface is fixed
by gravity, and one needs to use - only- for the calculation of action on that
surface. Hence the general technique for calculating path integrals in a
multiply-connected space-time could be summarised in the following points,

  * •

Pick two points in the space-time manifold $\mathcal{M}$.

  * •

Surround these points by 2 spacelike surfaces.

  * •

Identify the homotopy classes and loops on these surfaces, then left them up
to the associated bundle in which the field of concern ’lives’. Such than one
can preform parallel transport along the loops.

  * •

Define the fundamental groups for the surfaces, and find a proper
representation for these groups (loop representation).

  * •

Using the above representation, preform the standard technique for path
integration in multiply connected space-time,

$\int\mathcal{D}[f]e^{-iS[f,f,_{\mu}]}\longrightarrow\sum_{\alpha\in\pi_{1}(\sigma_{p}\cup\sigma_{q})}\chi(\alpha)K^{\alpha}.$
(9)

Where $\alpha$ is the homotopy class, $\chi$ is a representation of the
fundamental group and $K^{\alpha}$ is the transition amplitude via the
homotopy class $\alpha$.

##  V Conclusion

The action path integrals over a multiply connected spaces could be shown to
be non-peturbative and yield exact results as one might sum over equivalent
classes of paths that have the same winding number. In fact, in this paper, we
have seen that merging the ideas of minimal length and creation of pair black
holes hawking1996virtual , along with the ER=EPR conjecture ER1 ; ER2  could
yield a multiply-connected space-time. Hence path integrals over space-time
with gravity could be non-peturbative.

It could be shown in greater detail that the creation and inhalation of black-
holes as a result of third-quantised theory of geometry faizal2012some . Hence
one could also apply the non-peturbative techniques to geometry itself. There
is a lot to add to this proposal in order to use it for exact calculation of
path integrals , and/or quantisation of geometry. Based on the microscopic
structure of space-time, and to generalise the calculations in topological
field theory baez2014wormholes  to the 4D relativistic space-time, that is
non-topological.

As the topology is scale-independent,the non trivial topology of scape-time
caused by the virtual wormholes forming the quantum foam, could have a low-
energy effects. It would be interesting to investigate such effects in future
work, such as the Aharonov-Bohm effect, corrections to scattering amplitudes
in QED and QCD and shifts in energy levels of nuclear and atomic systems.

## Acknowledgements

This research project was supported by a grant from the “ Research Center of
the Female Scientiffic and Medical Colleges” , Deanship of Scientiffic
Research, King Saud University.

## References

  * (1) Ahmed Almheiri, Donald Marolf, Joseph Polchinski, and James Sully.  Black holes: complementarity or firewalls?  Journal of High Energy Physics, 2013(2):1–20, 2013.
  * (2) Salwa Alsaleh, Abeer Al-Modlej, and Ahmed Farag Ali.  Virtual black holes from generalized uncertainty principle and proton decay.  arXiv preprint arXiv:1703.10038, 2017.
  * (3) A. Ashtekar, J. Baez, A. Corichi, and K. Krasnov.  Quantum geometry and black hole entropy.  Phys. Rev. Lett., 80:904–907, Feb 1998.
  * (4) John C Baez and Jamie Vicary.  Wormholes and entanglement.  Classical and Quantum Gravity, 31(21):214007, 2014.
  * (5) Xavier Calmet.  Virtual Black Holes, Remnants and the Information Paradox.  Class. Quant. Grav., 32(4):045007, 2015.
  * (6) Mir Faizal.  Some aspects of virtual black holes.  Journal of Experimental and Theoretical Physics, 114(3):400–405, 2012.
  * (7) Hrant Gharibyan and Robert F Penna.  Are entangled particles connected by wormholes? evidence for the er= epr conjecture from entropy inequalities.  Physical Review D, 89(6):066001, 2014.
  * (8) Douglas M. Gingrich.  Noncommutative geometry inspired black holes in higher dimensions at the LHC.  JHEP, 05:022, 2010.
  * (9) D. Grumiller.  Virtual black holes and the S-matrix.  Int. J. Mod. Phys., D13:1973–2002, 2004.
  * (10) S. W. Hawking and Simon F. Ross.  Loss of quantum coherence through scattering off virtual black holes.  Phys. Rev., D56:6403–6415, 1997.
  * (11) Stephen W Hawking.  Virtual black holes.  Physical Review D, 53(6):3099, 1996.
  * (12) Giorgio Immirzi.  Real and complex connections for canonical gravity.  Classical and Quantum Gravity, 14(10):L177, 1997.
  * (13) Juan Maldacena and Leonard Susskind.  Cool horizons for entangled black holes.  Fortschritte der Physik, 61(9):781–811, 2013.
  * (14) Piero Nicolini, Anais Smailagic, and Euro Spallucci.  Noncommutative geometry inspired Schwarzschild black hole.  Phys. Lett., B632:547–551, 2006.
  * (15) Tim Prestidge.  Higher spin field equations in a virtual black hole metric.  Phys. Rev., D58:124022, 1998.
  * (16) Benni Reznik.  Entanglement from the vacuum.  Foundations of Physics, 33(1):167–176, 2003.
  * (17) Pieter Merkus Lambertus Tammes.  On the origin of number and arrangement of the places of exit on the surface of pollen-grains.  PhD thesis, University of Groningen, 1930.
  * (18) Mark Van Raamsdonk.  Building up space–time with quantum entanglement.  International Journal of Modern Physics D, 19(14):2429–2435, 2010\.
  * (19) John A Wheeler.  On the nature of quantum geometrodynamics.  Annals of Physics, 2(6):604–614, 1957.


* [134] S. D. Okegbile, J. Cai, C. Yi, and D. Niyato, “Human digital twin for personalized healthcare: Vision, architecture and future directions,” _IEEE Network_ , Jul. 2022.
  * [135] M. Y. Lim, J. Dias, R. Aylett, and A. Paiva, “Creating adaptive affective autonomous npcs,” _Autonomous Agents and Multi-Agent Systems_ , vol. 24, pp. 287–311, Mar. 2012.
  * [136] H. Duan, J. Li, S. Fan, Z. Lin, X. Wu, and W. Cai, “Metaverse for social good: A university campus prototype,” in _Proceedings of the 29th ACM International Conference on Multimedia_ , Oct. 2021, pp. 153–161.
  * [137] V. Taecharungroj, ““what can chatgpt do?” analyzing early reactions to the innovative ai chatbot on twitter,” _Big Data and Cognitive Computing_ , vol. 7, no. 1, p. 35, Feb. 2023.
  * [138] L. Rydell, “Predictive algorithms, data visualization tools, and artificial neural networks in the retail metaverse,” _Linguistic and Philosophical Investigations_ , no. 21, pp. 25–40, 2022.
  * [139] A. Hodorog, I. Petri, and Y. Rezgui, “Machine learning and natural language processing of social media data for event detection in smart cities,” _Sustainable Cities and Society_ , vol. 85, p. 104026, Oct. 2022.
  * [140] B. Osatuyi, “Information sharing on social media sites,” _Computers in Human Behavior_ , vol. 29, no. 6, pp. 2622–2631, Nov. 2013.
  * [141] L. Juhász and H. H. Hochmair, “Analyzing the spatial and temporal dynamics of snapchat,” in _AnaLysis, Integration, Vision, Engagement (VGI-ALIVE) Workshop_ , Jun. 2018.
  * [142] R. Wilken and L. Humphreys, “Placemaking through mobile social media platform snapchat,” _Convergence_ , vol. 27, no. 3, pp. 579–593, Jun. 2021.
  * [143] D. A. Parry, J. T. Fisher, H. Mieczkowski, C. J. Sewall, and B. I. Davidson, “Social media and well-being: A methodological perspective,” _Current Opinion in Psychology_ , vol. 45, p. 101285, Jun. 2022.
  * [144] A. Mottelson, A. Muresan, K. Hornbæk, and G. Makransky, “A systematic review and meta-analysis of the effectiveness of body ownership illusions in virtual reality,” _ACM Transactions on Computer-Human Interaction_ , 2023\.
  * [145] G. Schalk, D. J. McFarland, T. Hinterberger, N. Birbaumer, and J. R. Wolpaw, “Bci2000: a general-purpose brain-computer interface (bci) system,” _IEEE Transactions on Biomedical Engineering_ , vol. 51, no. 6, pp. 1034–1043, May 2004.
  * [146] C.-T. Lin, Y.-C. Chen, T.-Y. Huang, T.-T. Chiu, L.-W. Ko, S.-F. Liang, H.-Y. Hsieh, S.-H. Hsu, and J.-R. Duann, “Development of wireless brain computer interface with embedded multitask scheduling and its application on real-time driver’s drowsiness detection and warning,” _IEEE Transactions on Biomedical Engineering_ , vol. 55, no. 5, pp. 1582–1591, Apr. 2008.
  * [147] W. He, Y. Zhao, H. Tang, C. Sun, and W. Fu, “A wireless bci and bmi system for wearable robots,” _IEEE Transactions on Systems, Man, and Cybernetics: Systems_ , vol. 46, no. 7, pp. 936–946, Dec. 2015.
  * [148] S. R. A. Jafri, T. Hamid, R. Mahmood, M. A. Alam, T. Rafi, M. Z. Ul Haque, and M. W. Munir, “Wireless brain computer interface for smart home and medical system,” _Wireless Personal Communications_ , vol. 106, no. 4, pp. 2163–2177, Jun. 2019.
  * [149] C.-T. Lin, C.-J. Chang, B.-S. Lin, S.-H. Hung, C.-F. Chao, and I.-J. Wang, “A real-time wireless brain-computer interface system for drowsiness detection,” _IEEE Transactions on Biomedical Circuits and Systems_ , vol. 4, no. 4, pp. 214–222, Apr. 2010.
  * [150] A. I. N. Alshbatat, P. J. Vial, P. Premaratne, and L. C. Tran, “Eeg-based brain-computer interface for automating home appliances,” _Journal of Computers_ , 2014.
  * [151] J. Rosenthal, A. Sharma, E. Kampianakis, and M. S. Reynolds, “A 25 mbps, 12.4 pj/b dqpsk backscatter data uplink for the neurodisc brain–computer interface,” _IEEE Transactions on Biomedical Circuits and Systems_ , vol. 13, no. 5, pp. 858–867, Aug. 2019.
  * [152] K. F. Navarro, “Wearable, wireless brain computer interfaces in augmented reality environments,” in _International Conference on Information Technology: Coding and Computing, 2004. Proceedings. ITCC 2004._ , vol. 2. IEEE, Apr. 2004, pp. 643–647.
  * [153] N. Q. Hieu, D. T. Hoang, D. N. Nguyen, and E. Dutkiewicz, “Enabling immersion and presence in the metaverse with over-the-air brain-computer interface,” _arXiv preprint arXiv:2303.10577_ , 2023.
  * [154] R. C. Moioli, P. H. Nardelli, M. T. Barros, W. Saad, A. Hekmatmanesh, P. E. G. Silva, A. S. de Sena, M. Dzaferagic, H. Siljak, and W. Van Leekwijck, “Neurosciences and wireless networks: The potential of brain-type communications and their applications,” _IEEE Communications Surveys & Tutorials_, vol. 23, no. 3, pp. 1599–1621, 2021.
  * [155] C. Perfecto, M. S. Elbamby, J. Del Ser, and M. Bennis, “Taming the latency in multi-user vr 360°: A qoe-aware deep learning-aided multicast framework,” _IEEE Transactions on Communications_ , vol. 68, no. 4, pp. 2491–2508, Jan. 2020.
  * [156] X. Wei, C. Yang, and S. Han, “Prediction, communication, and computing duration optimization for vr video streaming,” _IEEE Transactions on Communications_ , vol. 69, no. 3, pp. 1947–1959, Nov. 2020.
  * [157] P. Xiang, H. Shan, Z. Zhang, L. Yu, and T. Q. Quek, “Noma based vr video transmissions exploiting user behavioral coherence,” in _2020 IEEE Wireless Communications and Networking Conference (WCNC)_. IEEE, May 2020, pp. 1–6.
  * [158] N. Q. Hieu, D. N. Nguyen, D. T. Hoang, and E. Dutkiewicz, “When virtual reality meets rate splitting multiple access: A joint communication and computation approach,” _IEEE Journal on Selected Areas in Communications_ , vol. 41, no. 5, pp. 1536–1548, Jan. 2023.
  * [159] Y. Mao, O. Dizdar, B. Clerckx, R. Schober, P. Popovski, and H. V. Poor, “Rate-splitting multiple access: Fundamentals, survey, and future research trends,” _IEEE Communications Surveys & Tutorials_, Jul. 2022.
  * [160] Z. Qin, X. Tao, J. Lu, and G. Y. Li, “Semantic communications: Principles and challenges,” _arXiv preprint arXiv:2201.01389_ , 2021.
  * [161] W. Klimesch, H. Schimke, and J. Schwaiger, “Episodic and semantic memory: an analysis in the eeg theta and alpha band,” _Electroencephalography and clinical Neurophysiology_ , vol. 91, no. 6, pp. 428–441, Dec. 1994.
  * [162] S.-H. Lee, M. Lee, and S.-W. Lee, “Neural decoding of imagined speech and visual imagery as intuitive paradigms for bci communication,” _IEEE Transactions on Neural Systems and Rehabilitation Engineering_ , vol. 28, no. 12, pp. 2647–2659, Nov. 2020.
  * [163] M. Kritzler, M. Funk, F. Michahelles, and W. Rohde, “The virtual twin: Controlling smart factories using a spatially-correct augmented reality representation,” in _Proceedings of the Seventh International Conference on the Internet of Things_ , Oct. 2017, pp. 1–2.
  * [164] B. Moya, A. Badías, I. Alfaro, F. Chinesta, and E. Cueto, “Digital twins that learn and correct themselves,” _International Journal for Numerical Methods in Engineering_ , vol. 123, no. 13, pp. 3034–3044, Jul. 2022\.
  * [165] L. U. Khan, W. Saad, D. Niyato, Z. Han, and C. S. Hong, “Digital-twin-enabled 6g: Vision, architectural trends, and future directions,” _IEEE Communications Magazine_ , vol. 60, no. 1, pp. 74–80, Jan. 2022.
  * [166] M. Dunleavy and C. Dede, “Augmented reality teaching and learning,” _Handbook of Research on Educational Communications and Technology_ , pp. 735–745, 2014.
  * [167] X. Zhang, L. Yao, D. Zhang, X. Wang, Q. Z. Sheng, and T. Gu, “Multi-person brain activity recognition via comprehensive eeg signal analysis,” in _Proceedings of the 14th EAI International Conference on Mobile and Ubiquitous systems: Computing, Networking and Services_ , Nov. 2017, pp. 28–37.
  * [168] M. J. Eugster, T. Ruotsalo, M. M. Spapé, I. Kosunen, O. Barral, N. Ravaja, G. Jacucci, and S. Kaski, “Predicting term-relevance from brain signals,” in _Proceedings of the 37th International ACM SIGIR Conference on Research & Development in Information Retrieval_, Jul. 2014, pp. 425–434.
  * [169] H. Ji, J. Li, R. Lu, R. Gu, L. Cao, and X. Gong, “Eeg classification for hybrid brain-computer interface using a tensor based multiclass multimodal analysis scheme,” _Computational Intelligence and Neuroscience_ , vol. 2016, Jan. 2016.
  * [170] R. Ju, C. Hu, and Q. Li, “Early diagnosis of alzheimer’s disease based on resting-state brain networks and deep learning,” _IEEE/ACM Transactions on Computational Biology and Bioinformatics_ , vol. 16, no. 1, pp. 244–257, 2017\.
  * [171] A. Mohammed, M. Zamani, R. Bayford, and A. Demosthenous, “Toward on-demand deep brain stimulation using online parkinson’s disease prediction driven by dynamic detection,” _IEEE Transactions on Neural Systems and Rehabilitation Engineering_ , vol. 25, no. 12, pp. 2441–2452, Jul. 2017.
  * [172] R. Yuste, S. Goering, G. Bi, J. M. Carmena, A. Carter, J. J. Fins, P. Friesen, J. Gallant, J. E. Huggins, and J. Illes, “Four ethical priorities for neurotechnologies and ai,” _Nature_ , vol. 551, no. 7679, pp. 159–163, Nov. 2017.
  * [173] Z. Zheng, S. Xie, H. Dai, X. Chen, and H. Wang, “An overview of blockchain technology: Architecture, consensus, and future trends,” in _2017 IEEE International Congress on Big Data (BigData congress)_. IEEE, Jun. 2017, pp. 557–564.
  * [174] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, “Intriguing properties of neural networks,” in _2nd International Conference on Learning Representations, ICLR 2014_ , Jan. 2014.
  * [175] A. B. Arrieta, N. Díaz-Rodríguez, J. Del Ser, A. Bennetot, S. Tabik, A. Barbado, S. García, S. Gil-López, D. Molina, and R. Benjamins, “Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai,” _Information Fusion_ , vol. 58, pp. 82–115, 2020.
  * [176] L. Drew, “The ethics of brain-computer interfaces,” _Nature_ , vol. 571, no. 7766, pp. S19–S19, Jul. 2019.
  * [177] Braincom, “High-density cortical implants for cognitive neuroscience and rehabilitation of speech using brain-computer interfaces,” Accessed: Aug. 15, 2023. [Online]. Available: http://www.braincom-project.eu/about/
  * [178] X. Meng, R. Jiang, D. Lin, J. Bustillo, T. Jones, J. Chen, Q. Yu, Y. Du, Y. Zhang, and T. Jiang, “Predicting individualized clinical measures by a generalized prediction framework and multimodal fusion of mri data,” _Neuroimage_ , vol. 145, pp. 218–229, 2017.
  * [179] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in Neural Information Processing Systems_ , vol. 30, 2017.

|  Howe Yuan Zhu received his Ph.D. degree in computer science at the
University of Technology Sydney in 2022. He received his BSc in Biomedical
Engineering at the University of Sydney in 2017. He is currently a research
associate at the University of Technology Sydney and a core GrapheneX-UTS
Human-centric Artificial Intelligence Centre (HAI) member. His current
research interests are XR/VR/AR, brain-computer interfaces, robotics,
artificial intelligence in robotics, and human-computer interaction. He is
actively exploring translational applications of wearable XR technologies for
medical applications and understanding human factors for assistive
technologies.
---|---
|  Nguyen Quang Hieu received the B.E. degree in Hanoi University of Science
Technology, Vietnam in 2018. He is currently a Ph.D. student at School of
Electrical and Data Engineering, University of Technology (UTS), Sydney,
Australia. Before joining UTS, he was a research assistant at School of
Computer Science and Engineering, Nanyang Technological University, Singapore.
His research interests include wireless communications, network optimization,
reinforcement learning, and deep learning..
---|---
|  Dinh Thai Hoang (M’16, SM’22) is currently a faculty member at the School
of Electrical and Data Engineering, University of Technology Sydney,
Australia. He received his Ph.D. in Computer Science and Engineering from the
Nanyang Technological University, Singapore 2016. His research interests
include emerging wireless communications and networking topics, especially
machine learning applications in networking, edge computing, and
cybersecurity. He has received several precious awards, including the
Australian Research Council Discovery Early Career Researcher Award, IEEE TCSC
Award for Excellence in Scalable Computing for Contributions on “Intelligent
Mobile Edge Computing Systems” (Early Career Researcher), IEEE Asia-Pacific
Board (APB) Outstanding Paper Award 2022, and IEEE Communications Society Best
Survey Paper Award 2023. He is currently an Editor of IEEE TMC, IEEE TWC, IEEE
TCCN, IEEE TVT, and IEEE COMST.
---|---
|  Diep N. Nguyen (Senior Member, IEEE) received the M.E. degree in electrical
and computer engineering from the University of California at San Diego
(UCSD), La Jolla, CA, USA, in 2008, and the Ph.D. degree in electrical and
computer engineering from The University of Arizona (UA), Tucson, AZ, USA, in
2013. He is currently a faculty member with the Faculty of Engineering and
Information Technology, University of Technology Sydney (UTS), Sydney, NSW,
Australia. Before joining UTS, he was a DECRA Research Fellow with Macquarie
University, Macquarie Park, NSW, Australia, and a Member of the Technical
Staff with Broadcom Corporation, San Jose, CA, USA, and ARCON Corporation,
Boston, MA, USA, and consulting the Federal Administration of Aviation,
Washington, DC, USA, on turning detection of UAVs and aircraft, and the U.S.
Air Force Research Laboratory, USA, on anti-jamming. His research interests
include computer networking, wireless communications, and machine learning
application, with emphasis on systems’ performance and security/privacy. Dr.
Nguyen received several awards from LG Electronics, UCSD, UA, the U.S.
National Science Foundation, and the Australian Research Council. He has
served on the Editorial Boards of the IEEE Transactions on Mobile Computing,
IEEE Communications Surveys & Tutorials (COMST), IEEE Open Journal of the
Communications Society, and Scientific Reports (Nature’s).
---|---
|  Distinguished Professor Chin-Teng Lin received a Bachelor’s of Science from
National Chiao-Tung University (NCTU), Taiwan in 1986, and holds Master’s and
PhD degrees in Electrical Engineering from Purdue University, USA , received
in 1989 and 1992, respectively. He is currently a distinguished professor and
Co-Director of the Australian Artificial Intelligence Institute within the
Faculty of Engineering and Information Technology at the University of
Technology Sydney, Australia. He is also an Honorary Chair Professor of
Electrical and Computer Engineering at NCTU. For his contributions to
biologically inspired information systems, Prof Lin was awarded Fellowship
with the IEEE in 2005, and with the International Fuzzy Systems Association (
IFSA) in 2012. He received the IEEE Fuzzy Systems Pioneer Award in 2017. He
has held notable positions as editor-in-chief of IEEE Transactions on Fuzzy
Systems from 2011 to 2016; seats on Board of Governors for the IEEE Circuits
and Systems (CAS) Society (2005-2008), IEEE Systems, Man, Cybernetics (SMC)
Society (2003-2005), IEEE Computational Intelligence Society (2008-2010);
Chair of the IEEE Taipei Section (2009-2010); Chair of IEEE CIS Awards
Committee (2022, 2023); Distinguished Lecturer with the IEEE CAS Society
(2003- 2005) and the CIS Society (2015-2017); Chair of the IEEE CIS
Distinguished Lecturer Program Committee (2018-2019); Deputy Editor-in-Chief
of IEEE Transactions on Circuits and Systems-II (2006-2008); Program Chair of
the IEEE International Conference on Systems, Man, and Cybernetics (2005); and
General Chair of the 2011 IEEE International Conference on Fuzzy Systems. Prof
Lin is the co-author of Neural Fuzzy Systems (Prentice-Hall) and the author of
Neural Fuzzy Control Systems with Structure and Parameter Learning (World
Scientific). He has published more than 425 journal papers including about 200
IEEE journal papers in the areas of neural networks, fuzzy systems, brain-
computer interface, multimedia information processing, cognitive neuro-
engineering, and human-machine teaming, that have been cited more than 34,000
times. Currently, his h-index is 90, and his i10-index is 401.
---|---