# Revisiting Domain-Adaptive 3D Object Detection by Reliable, Diverse and
Class-balanced Pseudo-Labeling

Zhuoxiao Chen1 Yadan Luo1 Zheng Wang2 Mahsa Baktashmotlagh1 Zi Huang1
1The University of Queensland 2University of Electronic Science and Technology
of China
{zhuoxiao.chen, y.luo, m.baktashmotlagh<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Unsupervised domain adaptation (DA) with the aid of pseudo labeling techniques
has emerged as a crucial approach for domain-adaptive 3D object detection.
While effective, existing DA methods suffer from a substantial drop in
performance when applied to a multi-class training setting, due to the co-
existence of low-quality pseudo labels and class imbalance issues. In this
paper, we address this challenge by proposing a novel ReDB framework tailored
for learning to detect all classes at once. Our approach produces Reliable,
Diverse, and class-Balanced pseudo 3D boxes to iteratively guide the self-
training on a distributionally different target domain. To alleviate
disruptions caused by the environmental discrepancy (e.g., beam numbers), the
proposed cross-domain examination (CDE) assesses the correctness of pseudo
labels by copy-pasting target instances into a source environment and
measuring the prediction consistency. To reduce computational overhead and
mitigate the object shift (e.g., scales and point densities), we design an
overlapped boxes counting (OBC) metric that allows to uniformly downsample
pseudo-labeled objects across different geometric characteristics. To confront
the issue of inter-class imbalance, we progressively augment the target point
clouds with a class-balanced set of pseudo-labeled target instances and source
objects, which boosts recognition accuracies on both frequently appearing and
rare classes. Experimental results on three benchmark datasets using both
voxel-based (i.e., SECOND) and point-based 3D detectors (i.e., PointRCNN)
demonstrate that our proposed ReDB approach outperforms existing 3D domain
adaptation methods by a large margin, improving 23.15% mAP on the nuScenes
$\rightarrow$ KITTI task. The code is available at
https://github.com/zhuoxiao-chen/ReDB-DA-3Ddet.

##  1 Introduction

Figure 1: Top: An illustration of the domain gap in 3D point clouds. Bottom:
Average Precision (AP) drop of ST3D [70] when applied to the multi-class
adaptation from nuScenes to KITTI (left), and from Waymo to KITTI (right).

As LiDAR-based 3D object detection continues to gain traction in various
applications such as robotic systems [1, 40, 60, 56, 83, 74] and self-driving
automobiles [8, 59, 42, 61, 2, 78, 37, 39, 32, 25, 38], it becomes
increasingly vital to address the challenges of deploying detectors in real-
world scenes. The primary obstacles stem from the discrepancy between the
training and test point cloud data, which are commonly curated from different
scenes, locations, times, and sensor types, creating a domain gap. This domain
gap mainly comes from the object shift and environmental shift, and can
significantly degrade the prediction accuracy of 3D detectors. Object shift
[62, 36] refers to the changes in the spatial distribution, point density, and
scale of objects between the training and test domains. For instance, the
average length of cars in the Waymo dataset [52] is significantly different
from the one in the KITTI dataset [15] by around 0.91 meters [62].
Environmental shift, on the other hand, arises from composite differences in
the surrounding environment such as inconsistent beam numbers, angles, point
cloud ranges, and data acquisition locations. For example, in Fig 1, Waymo
generates 3D scenes by 64-beam LiDAR sensors, while nuScenes [3] comprises
more sparse 32-beam environments with double large beam angles.

The co-existence of object shift and environmental shift poses a great
challenge to the 3D detection models to be deployed in the wild. To combat
such a problem, domain-adaptive 3D detection approaches [36, 71, 70, 79, 80]
have been exploited to adapt the model trained on a labeled dataset (i.e.,
source domain) to an unlabeled dataset from a different distribution (i.e.,
target domain). In this area, pioneering research of ST3D [70] presents a
self-training paradigm that generates pseudo-labeled bounding boxes to
supervise the subsequent learning on the target point clouds. In the same
vein, later studies [36, 79, 71, 76, 77] seek different solutions based on
self-supervised techniques, such as mean-teacher framework [36] and
contrastive learning [79] for stable optimization and obtaining better
embeddings.

Revisiting Domain-adaptive 3D Detection Setup. The aforementioned domain-
adaptive 3D detection approaches have typically followed a single-class
training setting, where the models are trained to adapt to each class
separately. While it is more practical and fairer to train the models with all
classes, our empirical study has shown that the detection performance of prior
works decreases considerably when switching to a multi-class setting (Fig 1).
Such an average precision (AP) drop can be attributed to the poor quality of
produced pseudo labels (i.e., erroneous and redundant) and the lower
recognition accuracy of rare classes (e.g., 91 times fewer bicycles than cars
in Waymo [52]).

In this work, we propose a novel ReDB framework (Fig 2) that aims to generate
Reliable, Diverse, class-Balanced pseudo labels for the domain-adaptive 3D
detection task. Our approach addresses the challenge of multi-class learning
by incorporating the following three mechanisms:

Reliability: Cross-domain Examination. To remove erroneous pseudo labels of
high confidence and avoid error accumulation in self-training, we introduce a
cross-domain examination (CDE) strategy to assess pseudo label reliability.
After copying the pseudo-labeled target objects into a familiar source
environment, the reliability is measured by the consistency i.e.,
Intersection-over-Union (IoU) between two box predictions in the target domain
and the source domain, respectively. Any objects with low IoUs will be
considered unreliable and then discarded. To prevent point conflicts between
the source and target point clouds, we remove the source points that fall
within the region where pseudo-labeled objects will be copied to. The proposed
CDE ensures that the accepted pseudo-labeled instances are domain-agnostic and
less affected by environmental shifts.

Diversity: OBC-based Downsampling. To avoid redundant pseudo labels that are
frequently and similarly scaled, it is important to prevent the trained
detector from collapsing into a fixed pattern that may only detect certain
types of objects (e.g., small-sized cars), and miss other instances of unique
styles (e.g., buses and trucks). In order to enhance geometric diversity, we
derive a metric called overlapped boxes counting (OBC) metric to uniformly
downsample the pseudo labels. The metric design is motivated by the
observation that 3D detector tends to predict more boxes for objects with
uncommon geometries, as they are harder to localize with only a few tight
boxes. We count the number of regressed boxes surrounding each detected object
as OBC and use kernel density estimation (KDE) to estimate its empirical
distribution. We then apply downsampling according to the inverse probability
of KDE, effectively reducing the number of pseudo labels in the high-density
OBC region, where objects have similar and frequent geometries. By learning
from a diverse subset of pseudo labels, the 3D detector can better identify
objects of various scales and point densities, potentially mitigating object
shift.

Balance: Class-balanced Self-Training. Despite the fact that reliable and
diverse pseudo labels can be selected by the previous two modules, there still
exists a significant inter-class imbalance. To achieve class-balanced self-
training, we randomly inject pseudo-labeled objects into each target point
cloud, with an equal number of samples from each category. By learning from
such class-balanced target data, the model can better grasp the holistic
semantics of the target labels. To enable a smooth transition of learning from
the source data to the target data, we first augment the target data with
labeled source objects in a class-balanced manner for the initial few steps.
We then gradually reduce the ratio of source objects and increase the number
of ReD labels as the self-training proceeds. This progressive class-balanced
self-training allows the model to adapt stably to the target domain, enhancing
recognition for both frequently appearing and rare classes.

Our work rectifies the setup of domain-adaptive 3D detection to a multi-class
scenario and proposes a novel ReDB framework for pseudo labeling in domain-
adaptive 3D detection. Extensive experiments on three large-scale testbeds
evidence that the proposed ReDB is of exceptional adaptability for both voxel-
based and point-based contemporary 3D detectors in varying environments,
improving 3D mAP of 20.66% and 23.15% over state-of-the-art methods on the
nuScenes $\rightarrow$ KITTI tasks, respectively.

Figure 2: The overall framework of the proposed method: we firstly obtain high
confident pseudo labels from Stage 1. In Stage 2, we check the reliability of
pseudo labels by the cross-domain examination (CDE) and reduce geometrical
redundancy by OBC-based downsampling. Finally, we progressively augment target
point clouds by injecting ReD labels and source labels for class-balanced
self-training in Stage 3.

##  2 Related Work

3D Object Detection from Point Clouds. With the growing advancement of neural
networks over the last decade, the 3D point clouds can be effectively encoded
by deep models to either point-based representation or grid-based
representation. The series of point-based detectors [48, 73, 72, 50, 41, 58,
81] directly encode 3D objects from raw points using PointNet encoder, then
generate 3D proposals from point features. This type of encoding strategy can
aid the model better preserve and learn the geometric properties of 3D
objects. While, the grid-based detectors [11, 24, 69, 82, 68, 22, 63, 49, 9,
75, 47, 53, 12] rely on convolution neural networks for efficiently capturing
regularly voxelized point clouds. Hybrid methods [18, 20, 71, 70] take
advantages of both paradigms, yielding better results at the expense of
processing speed. However, the preceding work overlooks the domain gap in
different 3D scenes such that could scarcely be applied to unseen scenarios.

Domain Adaptation for 3D Detection. To surmount such difficulty, unsupervised
domain adaptation (UDA) strives to transfer knowledge from a labelled source
domain to an unlabeled target domain. There are primarily two types of
strategies to eliminate the discrepancy between two domains. One is the
adversarial-based approach [13, 14, 57, 44, 19, 10, 6, 34, 35, 33, 27, 21]
that learns the domain-invariant features. The other is the statistical-based
approach [31, 30, 29, 51, 5, 64, 65, 26, 28] that seeks to minimize the
distribution distance between two domains. Despite the fact that UDA
techniques have been extensively investigated in the last decade, few UDA
approaches aim to address the domain shift existing in 3D object detection.
The distribution shifts of 3D detection are identified as dissimilar object
statistics [62], weather effects [67, 17], sensor difference [43, 16, 66] or
synthesis-to-real gap [45, 7, 23]. To close the domain shift, an adversarial-
based approach [80] was proposed to match features of different scales and
ranges. Another streams of work focus on generating and enhancing 3D pseudo
labels through memory bank [70, 71], mean-teacher paradigm [36] or contrastive
learning [79]. However, existing methods necessitate either extra
computational time or significant memory usage. Besides that, these methods
generate low-quality pseudo-labels induced by environmental gaps and numerous
analogous geometrical features, and overlook the class-imbalanced issue.

##  3 Methodology

###  3.1 Problem Definition

In this section, we mathematically formulate the problem of unsupervised
domain adaptation for 3D object detection and set up the notations. The main
objective is to adapt a 3D object detector trained on the labeled point clouds
$\\{(X^{s}_{i},Y^{s}_{i})\\}^{N_{s}}_{i=1}$ from the source domain, to the
unlabeled data $\\{X^{t}_{i}\\}^{N_{t}}_{i=1}$ in the target domain. $N_{s}$
and $N_{t}$ indicate the number of point clouds in the source and target
domains, respectively. The 3D box annotation of the $i$-th source point cloud
is denoted as $Y^{s}_{i}=\\{b_{j}\\}^{B_{i}}_{j=1}$ and
$b_{j}=(x,y,z,w,l,h,\theta,c)\in\mathbb{R}^{8}$, where $B_{i}$ is the total
number of labeled boxes in $X^{s}_{i}$, $(x,y,z)$ represents the box center
location, $(w,l,h)$ is the box dimension, $\theta$ is the box orientation, and
$c\in\\{1,...,C\\}$ is the object category.

###  3.2 Overall Framework

Our approach consists of three stages to facilitate knowledge transfer from
the label-rich source domain to the label-scarce target domain, as illustrated
in Fig 2. In Stage 1, the 3D detector (e.g., SECOND [68] or PointRCNN [48]) is
pre-trained on the source domain using the standard random object scaling
(ROS) augmentation [70]. Upon reaching convergence of pre-training, the
unlabeled point clouds are passed to the pre-trained detector to generate
pseudo-labels $\\{\widehat{Y}^{t}_{i}\\}^{N_{t}}_{i=1}$ of high confidence for
the target data $\\{X^{t}_{i}\\}^{N_{t}}_{i=1}$. Specifically, the produced
pseudo labels will undergo scrutiny by the cross-domain examination (CDE)
strategy (Sec 3.3) and down-sampled by the OBC-based diversity module (Sec
3.4), forming a subset of reliable and diverse (ReD) objects associated with
pseudo labels. The details of stage 2 are illustrated in Fig 4. During model
self-training on the target set in Stage 3, we randomly inject ReD target
objects and source objects into each target point clouds in a class-balanced
manner (Sec 3.5), with the ratio of source samples gradually decreasing. The
3D detector is iteratively trained to adapt by alternating between Stage 2 and
Stage 3.

Figure 3: The proposed cross-domain examination (CDE) strategy involves
copying pseudo target points $X^{ps}$ in blue, with red and yellow boxes
representing initial predictions in the target and source environments,
respectively. Green boxes denote the ground truth for reference. If the
intersection over union (IoU) between the target and source predictions is
large, like in the first example, the reliability is accepted. However, in
cases where $\hat{b}$ is not detected or the IoU is small, like in the second
and third examples, these labels are considered unreliable due to
inconsistencies caused by environmental shift.

###  3.3 Reliability: Cross-domain Examination

The proposed cross-domain examination (CDE) strategy aims to identify high-
quality pseudo labels that are robust to the environmental shift. First, given
a 3D detector $F(\,\cdot\,;\theta^{s})$ pre-trained on the source domain, we
can obtain the initial pseudo labels by inferring a target point cloud
$X_{i}^{t}$ as:

$\displaystyle\widehat{Y}_{i}^{t}=\\{\hat{b}_{j}\\}^{\widehat{B}_{i}}_{j=1}=F(X_{i}^{t};\theta^{s}),$
(1)

where $\theta^{s}$ is the pre-trained weights, and
$\widehat{Y}^{t}\in\mathbb{R}^{\widehat{B}\times 8}$ denotes the pseudo labels
with the model confidence greater than $\delta_{\operatorname{pos}}$.
$\widehat{B}_{i}$ is the number of obtained pseudo labels. Below, we omit the
subscript $i$ for simplicity. To attain a real source environment, we randomly
sample a source point cloud $X^{s}\sim rand(\\{(X^{s}_{i})\\}^{N_{s}}_{i=1})$
and copy-paste the target point clouds $X^{ps}\subset X^{t}$ that fall within
$\widehat{Y}^{t}$, into the sampled source point cloud as shown in Fig 3. To
prevent conflicts between existing points and the pseudo points to be pasted,
we apply the object points removal operator $\texttt{R}(\,\cdot\,)$ to remove
source points residing in the area overlapped with the copied target samples
$X^{ps}$. Now, we can safely move the target points to the source point clouds
as $\texttt{R}(X^{s})\oplus X^{ps}$, where $\oplus$ represents point
concatenation. We generate new predictions $\widetilde{Y}^{t}$ of high
confidence for the concatenated source point clouds as:

$\displaystyle\widetilde{Y}_{i}^{t}=\\{\tilde{b}_{j}\\}^{\widetilde{B}_{i}}_{j=1}=F(\texttt{R}(X^{s})\oplus
X^{ps});\theta^{s}),$ (2)

where $\widetilde{B}_{i}$ represents the number of boxes in the updated source
clouds. To examine the resiliency of the pseudo-labeled object
$\hat{b}\in\hat{Y}^{t}$ to the environmental shift, we compute the
intersection over union (IoU) between the initial predicted box $\hat{b}$ and
its corresponding prediction $\tilde{b}\in\tilde{Y}^{t}$ in the source
environment. Through comparing the IoU to a predefined IoU threshold
$\delta_{\operatorname{cde}}$, we can effectively discriminate reliable pseudo
labels from unreliable ones by using the following criterion:

$\displaystyle
f_{\operatorname{CDE}}(\hat{b};\tilde{b})=\begin{cases}\hat{b}&\text{IoU}(\hat{b},\tilde{b})\geq\delta_{\operatorname{cde}},\\\
\emptyset&\text{otherwise}.\end{cases}$ (3)

We apply the CDE only at the first round of pseudo labelling because the pre-
trained detector has no knowledge about target environments. The process of
proposed CDE strategy is presented in Fig 3 with three examples for
illustration.

Figure 4: In Stage 2 of our approach, we input each target point cloud $X^{t}$
to the pre-trained detector and obtain 3D box predictions. By applying NMS, we
select confident and non-overlapped boxes $\hat{b}$ as pseudo labels.
Simultaneously, we compute OBC score for each pseudo-labeled $\hat{b}$, based
on prior-NMS boxes $\hat{b}^{prior}$. Next, we use CDE to add qualified
$\hat{b}$ to a pool. After inferring all target data, we globally downsample
the pseudo-labeled boxes in the pool using OBC, resulting in
$\mathcal{D}_{\operatorname{ReD}}$. In Stage 3, we augment each target point
cloud by sampling high-quality objects from $\mathcal{D}_{\operatorname{ReD}}$
(Eq 6) in a class-balanced manner during self-training. Stage 2 and 3 are
alternated until convergence.

###  3.4 Diversity: OBC-based Downsampling

Although we have already removed unreliable pseudo labels, the remaining label
set may contain objects with similar patterns. This hinders the model from
learning a more comprehensive distribution of the target domain. To ensure
that the pseudo label set is representative of the broad spectrum of geometric
characteristics present in target data, we investigate ways to properly
measure object diversity. Our empirical study shows that 3D detectors generate
more box predictions around unfamiliar objects, as visualized in Fig 5. It is
observed that, the box predictions (in yellow) before non-maximum suppression
(NMS) have obvious overlaps (i.e., $\text{IoU}>\delta_{\operatorname{obc}}$)
with the corresponding final predicted box (in red). Besides, predicted boxes
for low-density objects typically orientate differently, while the regressed
box scales for large objects have a high variance.

This observation motivates us to derive the concept of overlapped boxes
counting (OBC), which counts the number of predicted boxes that surround the
same object with the $\text{IoU}>\delta_{\operatorname{obc}}$. The OBC can
comprehensively reflect the uniqueness in geometric characteristics of
objects, including scales, locations, point densities. A qualitative analysis
of OBC is provided in Sec 4.3, which evidence that pseudo-labeled target
objects with high OBC values tend to be uncommon in different geometric
aspects. To estimate the probability density function (PDF) of OBC values, we
use bar plots to show the frequency of different OBC values, and fit it with
Kernel Density Estimation (KDE) (as shown in the blue curve of Fig 5). It can
be seen from the plot that the distribution of OBC is very skewed, implying
that the majority of geometric features are quite analogous since fell at high
frequency interval (i.e., from 3 to 12). Conversely, objects with rare
geometrics fell on the long distribution tail (i.e., from 12 to 30). To get a
diverse subset and exclude too many objects in high density regions, we
propose to leverage the inverse KDE function (shown in red curve) to assign a
low sampling weight for pseudo-labeled objects fell in dense regions, while
high sampling probability for objects in tail regions. In particular, we
firstly collect the set $\mathcal{O}$ of all OBC values with the cardinality
$\hat{B}=\sum_{i=1}^{N_{t}}\hat{B}_{i}$:

$\mathcal{O}=\\{f_{c}(\hat{b}_{j})|b_{j}\in\hat{Y}_{i}^{t}\\}_{i=1}^{N_{t}},\vspace{-1ex}$
(4)

where $f_{c}(\,\cdot\,)$ is the function to count OBC value for each pseudo-
labeled box $\hat{b}$. The KDE with the Gaussian kernel of the random variable
$O$ can be calculated with a finite set of $\mathcal{O}$:

$f_{\operatorname{KDE}}(O)=\frac{1}{\hat{B}\sigma\sqrt{2\pi}}\sum_{j=1}^{\hat{B}}e^{-\frac{1}{2}(\frac{f_{c}(\hat{b}_{j})-O}{\sigma})^{2}},$
(5)

where $\sigma$ is the standard deviation. The sampled subset
$\mathcal{D}_{\operatorname{ReD}}$ of size $\hat{B}/d$ are uniformly
downsampled from the inverse KDE, where $d>1$ is the down sampling rate.
Finally, the sampled subset $\mathcal{D}_{\operatorname{ReD}}$ contains
pseudo-labelled objects with more uniformly distributed OBC values, indicating
less common patterns and more diverse geometrics (e.g., object scales,
distances and densities).

Figure 5: An illustration of overlapped boxes counting (OBC). The upper part
shows box predictions before NMS generated around three positively predicted
objects associated with different OBC values. The bottom part is the plot
demonstrating the distributions of OBC values of all detected objects, with
the fitted KDE (blue) and inverse KDE (red) for diverse sampling.

###  3.5 Balance: Class-balanced Self-Training

Regardless OBC-based downsampling creates a pool of reliable and intra-class
diverse (ReD) pseudo-labels, but the inter-class imbalance issue still poses a
challenge for multi-class adaptation. We aim to leverage a class-balanced
paradigm to inject the generated pseudo labels to each target point cloud,
during the self training stage:

$\displaystyle\widehat{Y}^{r}$
$\displaystyle=\\{\\{\hat{b}_{j}^{1}\\}^{S^{r}}_{j=1}\oplus\cdots\oplus\\{\hat{b}_{j}^{C}\\}^{S^{r}}_{j=1}\\}\sim\mathcal{D}_{\operatorname{ReD}},$
(6)

where $\oplus$ represents the point cloud concatenation and the superscript
$1,\cdots,C$ of $\hat{b}$ indicates the predicted class of the pseudo label,
and $S^{r}$ is the sampling number of target ReD labels for each class. To
moderate and stable the adaptation process with the huge domain gap, we
augment the target data by injecting source ground truth (GT) labels with ROS
augmentation:

$\displaystyle P^{g}$ $\displaystyle=\\{Y_{1}^{s}\oplus\dots\oplus
Y_{N_{s}}^{s}\\},$ (7) $\displaystyle Y^{g}$
$\displaystyle=\\{\\{b_{j}^{1}\\}^{S^{g}}_{j=1}\oplus\cdots\oplus\\{b_{j}^{C}\\}^{S^{g}}_{j=1}\\}\sim
P^{g},$

where $S^{g}$ is the GT sampling number for each class, $P^{g}$ is the source
GT pool containing labelled boxes from all source point clouds. We gradually
increase $S^{r}$ and reduce $S^{g}$ during the self training, that enables 3D
detector to smoothly generalize to the target domain. Finally, we concatenate
the sampled target pseudo-labeled ReD boxes and source GT boxes to the initial
pseudo labels, including the points $X^{r}$ and $X^{g}$ inside target ReD
boxes and source GT boxes, respectively:

$\displaystyle\widehat{Y}^{t}$
$\displaystyle\leftarrow\widehat{Y}^{t}\oplus\widehat{Y}^{r}\oplus Y^{g},$ (8)
$\displaystyle X^{t}$ $\displaystyle\leftarrow X^{t}\oplus X^{r}\oplus X^{g}.$

Finally, we have each target point cloud augmented with class-balanced ReDB
pseudo labels as $(X_{i}^{t},\widehat{Y}_{i}^{t})$, and train the detector as:

$\displaystyle
F(\,\cdot\,;\theta^{t})\leftarrow\\{(X^{t}_{i},\widehat{Y}^{t}_{i})\\}^{N_{t}}_{i=1}.$
(9)

We alternatively generate target pseudo labels (Stage 2) and self-train the
model (Stage 3), until the 3D detector fully is optimized, as illustrated in
Fig 2. The detailed Algorithm is summarized in the supplementary materials.

Table 1:  Experiment results of three adaptation tasks are presented, with the reported average precision (AP) for bird’s-eye view ($\text{AP}_{\text{BEV}}$) / 3D ($\text{AP}_{\text{3D}}$) of car, pedestrian, and cyclist with IoU threshold set to 0.7, 0.5, and 0.5 respectively. When the KITTI is the target domain, we report the AP at Moderate difficulty. The last column shows the mean AP for all classes. We indicate the best adaptation result by bold and we highlight the row representing our method. Task | Method | Car | Pedestrian | Cyclist | Mean AP
---|---|---|---|---|---
Waymo $\rightarrow$ KITTI | Source Only | 51.48 / 19.78 | 40.80 / 31.26 | 47.63 / 35.45 | 46.64 / 28.83
SN |  76.61 / 54.14 |  52.48 / 48.20 | 34.56 / 32.74 | 54.55 / 45.03
ST3D | 77.62 / 49.24 | 44.45 / 42.04 | 47.74 / 42.95 | 56.60 / 44.70
ST3D++ | 77.68 / 50.03 | 49.09 / 46.19 | 51.50 / 47.70 | 59.42 / 47.97
ReDB |  80.37 / 54.12 |  51.01 / 48.20 |  52.05 / 47.97 |  61.14 / 50.10
Oracle | 83.29 / 73.45 | 46.64 / 41.33 | 62.92 / 60.32 | 64.28 / 58.37
Waymo $\rightarrow$ nuScenes | Source Only |  30.64 / 17.18 | 1.81 / 0.58 | 0.97 / 0.88 | 11.14 / 6.22
SN | 29.56 / 17.78 | 1.33 / 1.07 | 2.92 / 2.61 | 11.27 / 7.15
ST3D | 28.42 / 17.83 | 1.64 / 1.39 | 4.01 / 3.54 | 11.36 / 7.59
ST3D++ |  28.87 / 19.15 | 1.82 / 1.58 | 4.09 / 3.74 | 11.60 / 8.16
ReDB |  30.12 / 18.56 |  2.47 / 2.14 |  6.56 / 5.19 |  13.05 / 8.63
Oracle | 51.88 / 34.87 | 25.24 / 18.92 | 15.06 / 11.73 | 30.73 / 21.84
nuScenes $\rightarrow$ KITTI | Source Only | 39.15 / 7.65 | 21.54 / 16.87 | 6.31 / 2.44 | 22.33 / 8.98
SN | 56.08 / 28.67 | 23.05 / 16.84 | 5.11 / 2.32 | 28.08 / 15.94
ST3D | 71.50 / 48.09 | 22.64 / 17.61 | 7.86 / 5.20 | 34.00 / 23.64
ST3D++ | 69.90 / 44.62 | 24.11 / 18.20 | 10.14 / 6.39 | 33.75 / 21.64
ReDB |  74.23 / 51.31 |  25.95 / 18.38 |  13.82 / 8.64 |  38.00 / 26.11
Oracle | 83.29 / 73.45 | 46.64 / 41.33 | 62.92 / 60.32 | 64.28 / 58.37
Table 2:  Experiment results at Hard difficulty of adapting to the KITTI dataset. We report average precision (AP) for bird’s-eye view ($\text{AP}_{\text{BEV}}$) / 3D ($\text{AP}_{\text{3D}}$) of car, pedestrian, and cyclist with IoU threshold set to 0.7, 0.5, and 0.5 respectively. The last column shows the mean AP for all classes. We indicate the best adaptation result by bold, and we highlight the row representing our method. Task | Method | Car | Pedestrian | Cyclist | Mean AP
---|---|---|---|---|---
Waymo $\rightarrow$ KITTI | SN |  75.57 / 53.87 |  49.15 / 45.58 | 33.09 / 31.19 | 52.60 / 43.55
ST3D | 75.31 / 48.01 | 41.42 / 43.02 | 45.76 / 40.76 | 54.16 / 43.93
ST3D++ | 75.54 / 48.03 | 45.63 / 42.00 | 47.12 / 43.70 | 56.10 / 44.58
ReDB |  78.24 / 52.56 |  46.01 / 43.22 |  49.72 / 45.73 |  57.99 / 47.17
nuScenes $\rightarrow$ KITTI | SN | 54.29 / 25.81 | 21.43 / 15.09 | 5.13 / 2.31 | 26.95 / 14.40
ST3D | 68.21 / 43.44 | 21.30 / 15.96 | 7.33 / 5.08 | 32.28 / 21.49
ST3D++ | 68.29 / 40.85 | 22.16 / 16.31 | 7.32 / 3.91 | 32.59 / 20.36
ReDB |  69.76 / 46.17 |  25.01 / 18.88 |  9.54 / 6.15 |  34.77 / 23.73

##  4 Experiments

###  4.1 Experimental Setup

Datasets. Experiments are carried out on three widely used LiDAR 3D object
detection datasets: KITTI [15], Waymo [52], and nuScenes [3]. We follow [70,
71] to address different realistic scenarios of 3D domain adaptation: (i)
Adaptation across domains with object shifts (e.g., scale, point density),
(ii) Adaptation across domains with environmental shift (e.g., data generation
regions, LiDAR beams, angels and range).

Baseline Methods. We compare the proposed method with baselines using voxel-
based backbone (e.g., SECOND). (i) Source Only refers to the pre-trained model
from the source domain evaluated directly on the target domain without
adaptations; (ii) SN [62] takes size statistics of target objects and rescale
source objects during pre-training; (iii) ST3D [70] is the pioneering pseudo
labelling-based method that does not require any knowledge about backbone
models and dataset statistics; (iv) ST3D++ [71] is the extended version of
ST3D by domain-speciﬁc batch normalization [4], achieving the SOTA
performance. (v) Oracle means a fully supervised model trained on the target
domain. We also compare with baselines that only work for the point-based
backbone (e.g., PointRCNN). (vi) $\textsc{SF-UDA}^{\textsc{3D}}$ [46] seeks
the best scale for pseudo-labeling from consecutive frames by estimating
motion coherence. (vii) MLC-Net [36] is a mean-teacher [54] framework aiming
to stable optimize the self-training.

Implementation Details. Unlike the previous approach, we pre-trained and self-
trained a single 3D detection backbone for all categories, simultaneously,
rather than unfairly train a single-class model. Our code is built on the
point cloud detection codebase OpenPCDet [55] and runs on two NVIDIA Tesla
V100 GPUs. For both backbones, we set the batch size to 4 for each GPU. We set
hyperparameter $\delta_{\operatorname{cde}}=0.6$,
$\delta_{\operatorname{pos}}=0.6$, $\delta_{\operatorname{obc}}=0.3$, $d=5$
and epochs to 90 for all tasks. Regarding the class-balanced sampling, we set
$S^{r}=5$ and $S^{g}=10$, then increase $S^{r}$ and decrease $S^{g}$ by $2$,
at each round of pseudo label generation. The implementation code will be
available. To fairly evaluate baselines and the proposed method, we utilize
the KITTI evaluation metric for evaluation on the three classes: car
(equivalent to the vehicle for similar classes in the Waymo), pedestrians and
cyclists (equivalent to bicyclist and motorcyclist in nuScenes). Based on the
official KITTI evaluation metric, we report the average precision for each
class, in both 3D (i.e., $\text{AP}_{\text{3D}}$) and the bird’s eye view
(i.e., $\text{AP}_{\text{BEV}}$) over 40 recall positions, with IoU threshold
0.7 for cars and 0.5 for pedestrians and cyclists. We also calculate the mean
AP for all classes in 3D and BEV, denoted as $\text{mAP}_{\text{3D}}$ and
$\text{mAP}_{\text{BEV}}$, respectively.

Table 3: Performance comparisons ($\text{AP}_{\text{BEV}}$ and
$\text{AP}_{\text{3D}}$) with PointRCNN backbone on nuScenes $\rightarrow$
KITTI task. ${\ddagger}$ indicates the results reported in the original paper,
where only a single category was adapted.

|  | Car | Pedestrian | Cyclist | Average
---|---|---|---|---|---
| Method | Easy | Mod. | Hard | Easy | Mod. | Hard | Easy | Mod. | Hard | Easy | Mod. | Hard
3D | Source Only | 42.77 | 32.11 | 28.75 | 43.26 | 37.29 | 33.16 | 3.28 | 4.09 | 4.18 | 29.77 | 24.60 | 22.03
SN | 66.56 | 50.32 | 45.92 | 42.96 | 37.15 | 32.45 | 9.07 | 7.57 | 7.42 | 39.53 | 31.68 | 28.60
ST3D | 48.85 | 41.90 | 38.92 | 45.67 | 38.71 | 33.09 | 26.50 | 19.35 | 18.38 | 40.34 | 33.32 | 30.13
ST3D++ | 60.45 | 49.36 | 45.88 | 50.77 | 42.43 | 36.64 | 27.20 | 17.94 | 16.52 | 46.14 | 36.58 | 33.01
$\textsc{SF-UDA}^{\textsc{3D}}$ ${\ddagger}$ | 68.80 | 49.80 | 45.00 | - | - | - | - | - | - | - | - | -
MLC-Net ${\ddagger}$ | 71.30 | 55.40 | 49.00 | - | - | - | - | - | - | - | - | -
ReDB | 71.45 | 57.9 | 53.91 | 52.32 | 44.33 | 37.95 | 45.13 | 32.93 | 31.05 | 56.30 | 45.05 | 40.97
BEV | Source Only | 80.91 | 60.75 | 56.00 | 45.84 | 39.78 | 35.45 | 3.38 | 4.45 | 4.57 | 43.37 | 34.99 | 32.00
SN | 81.17 | 63.34 | 56.80 | 44.80 | 38.73 | 34.39 | 9.65 | 9.39 | 8.93 | 45.21 | 37.15 | 33.67
ST3D | 85.61 | 69.48 | 64.52 | 47.31 | 39.96 | 35.05 | 32.60 | 23.06 | 21.73 | 55.17 | 44.17 | 40.43
ST3D++ | 85.81 | 69.17 | 64.74 | 52.68 | 45.10 | 39.30 | 31.30 | 20.29 | 18.42 | 56.59 | 44.85 | 40.82
| ReDB | 91.50 | 76.01 | 71.42 | 54.83 | 45.87 | 39.34 | 48.44 | 35.49 | 33.23 | 64.93 | 52.46 | 48.00

Table 4: Ablative study of different modules on adapting Waymo to KITTI task. |  |  |  |  3D Detection mAP |  BEV Detection mAP
---|---|---|---|---|---
GT | PS | CDE | OBC | Easy | Moderate | Hard | Easy | Moderate | Hard
- | - | - | - | $48.41$ | $42.26$ | $40.18$ | $60.68$ | $53.52$ | $51.01$
$\surd$ | - | - | - | $54.10$ | $45.21$ | $42.42$ | $63.13$ | $56.73$ | $54.00$
- | $\surd$ | - | - | $51.92$ | $46.04$ | $43.68$ | $62.57$ | $54.57$ | $51.32$
- | $\surd$ | $\surd$ | - | $54.04$ | $47.07$ | $44.90$ | $64.70$ | $58.15$ | $55.69$
- | $\surd$ | - | $\surd$ | $53.99$ | $47.66$ | $44.65$ | $65.72$ | $58.51$ | $55.36$
$\surd$ | $\surd$ | $\surd$ | $\surd$ | 56.52 | 50.10 | 47.17 | 67.36 | 61.14 | 57.99

###  4.2 Main Results and Analysis

We conducted comprehensive experiments on three different 3D adaptation tasks
under a multi-class training setting, as summarized and reported in Tab 1. We
can clearly observe that ReDB consistently improves the performance on Waymo
$\rightarrow$ KITTI and nuScenes $\rightarrow$ KITTI by a large margin of
21.27% and 17.13% in terms of $\text{mAP}_{\text{3D}}$, which largely close
the performance gap between Source Only and Oracle. When comparing with the
latest SOTA method ST3D++, our ReDB exhibits superior performance in terms of
$\text{mAP}_{\text{BEV}}$ for all three adaptation tasks, with improvements of
2.9%, 12.5%, and 12.6%, respectively. It is worth noting that the proposed
ReDB gains significantly more performance for the last two tasks (i.e., Waymo
$\rightarrow$ nuScenes and nuScenes $\rightarrow$ KITTI) than the first task
(i.e., Waymo $\rightarrow$ KITTI), indicating the ReDB is more effective for
adapting to 3D scenes with larger environmental gaps. It is further evident
that the ReDB method stands out for its well-balanced performance across all
categories, while all baselines are biased towards the most frequently
appearing class (i.e. car) and underperformed in rare classes (i.e. pedestrian
and cyclist). Overall, the proposed ReDB excels all baselines on both
$\text{mAP}_{\text{3D}}$ and $\text{mAP}_{\text{BEV}}$ across all scenarios of
3D adaptation tasks.

Performance Analysis under Difficult Conditions. To corroborate the
effectiveness of the ReDB approach against the baseline methods, we present
additional experimental results in Tab 2, using a more challenging evaluation
metric: the average precision (AP) at Hard difficulty level defined by KITTI
dataset. It is observed that the proposed ReDB outperforms the SOTA baseline
ST3D++ by 5.81% of $\text{mAP}_{\text{3D}}$ for adapting from Waymo to KITTI.
Concerning a more challenging adaptation task (i.e., nuScenes → KITTI)
involving significant environmental shifts in beam numbers, angles and the
point cloud range, the proposed ReDB exhibits superior performance over all
other baseline models across all categories: 16.55% of
$\text{mAP}_{\text{3D}}$ than the SOTA baseline. Thus, our approach surpasses
the baselines by a large margin when assessed with KITTI metric at Hard
difficulty, indicating the efficacy of the ReDB in facilitating effective
generalization of 3D detectors to difficult target objects.

###  4.3 Component Analysis

Figure 6: Case studies of the overlapped boxes counting. We visualize the
predicted boxes before NMS (in yellow) and its count, with the corresponding
geometric description. By inverse frequency sampling of OBCs, geometrical
diverse objects are added to the ReD Pool.

Ablation Study. To investigate the impact of the derived CDE-based reliable
pseudo label generation, OBC-based downsampling and class-balanced self-
training, we compare five variants of the ReDB model in the task of adapting
Waymo to KITTI, shown in Tab 4. GT indicates the source ground truth sampling
and PS indicates the target pseudo label sampling. For fairness, we keep the
same hyperparameters for all ablation variants. To explore the effectiveness
of the proposed CDE and OBC modules, we remove each of them and observe a
dramatic $\text{mAP}_{\text{3D}}$ drop of 4.9% (row-5) and 5.34% (row-4)
compared to the full model (row-6), respectively. To validate the proposed
class-balanced self-training, we find when comparing to the non-sampling
baseline (row-1), the source ground truth sampling (row-2) and the target
pseudo label sampling (row-3) increase $\text{mAP}_{\text{3D}}$ of 7% and
8.9%, respectively. It is noticeable that, source ground truth sampling
(row-2) produces remarkable $\text{mAP}_{\text{3D}}$ gains at easy level
(5.69%) but very limited gains at moderate and hard levels (2.95% and 2.24%),
caused by a lack of knowledge about diverse geometrical features in the target
domain. Therefore, removing any module from the proposed ReDB induces a clear
drop in 3D / BEV mAP scores, confirming the importance of each component that
contributes to multi-class 3D domain adaptation.

Sensitivity to Detector Architecture. To validate the sensitivity of
performance to choices of voxel-based and point-based detectors, we plug the
proposed ReDB paradigm into PointRCNN. As reported in Tab 3, our approach
consistently outperforms the latest SOTA method ST3D++ by 23.32% and 24.11%
regarding $\text{mAP}_{\text{3D}}$ at moderate and hard levels, respectively.
Notably, our multi-class method has even achieved superior performance (10.02%
and 19.8%) compared to MLC-Net and $\textsc{SF-UDA}^{\textsc{3D}}$, both of
which were under the unfair single-class training setting and are exclusively
designed for the point-based 3D detector. The outstanding results of the
proposed method using a variety of 3D detectors, demonstrate the ReDB is a
model-agnostic and insensitive to detector architecture.

Figure 7: Hyperparameter sensitivity of $\delta_{\operatorname{cde}}$ and $d$.

Sensitivity to the hyperparameters $\delta_{\operatorname{cde}}$ and $d$. We
show the sensitivity of our approach to varying the CDE IoU threshold
$\delta_{\operatorname{cde}}$ and diversity downsampling rate $d$ in Fig 7. We
vary the value of $\delta_{\operatorname{cde}}$ and $d$, from $[0.3,0.6]$ and
$[2,10]$, respectively, while fixing the other configurations to the default
setting. We can observe that, when $\delta_{\operatorname{cde}}$ varies, there
is only 2.69% and 2.07% fluctuation on $\text{mAP}_{\text{3D}}$ and
$\text{mAP}_{\text{BEV}}$ scores, respectively. Regarding the variation of
$d$, we notice when $d$ is set as a large value (e.g., $d=10$), the
performance drops within an acceptable range: 1.37% in
$\text{mAP}_{\text{3D}}$ and 2.57% in $\text{mAP}_{\text{BEV}}$. Such a small
drop could be caused by the insufficient scale of the down-sampled ReD pool.
Hence, selecting a proper value of downsampling rate (e.g., $d=5$), reaches
the best results. Little fluctuation caused by varying
$\delta_{\operatorname{cde}}$ and $d$ evidence that, the proposed method is
robust to different choices of hyperparameters.

Visualization of OBC-based Diversity. To intuitively understand the merits of
our proposed diversity module, Fig 6 visualizes the counted predicted boxes
before NMS for different pseudo-labeled target objects. We can see that most
objects with small OBC values (e.g., between 5 and 8) are, (1) typically
closer to the LiDAR sensor, (2) with a complete object shape, (3) and commonly
small-sized objects. These objects usually have highly similar and complete
geometric features, making up the majority of the dataset. In contrast,
objects with high OBCs are usually diverse in one or more aspects of
geometrical representations. Regarding the object size, large-sized objects
tend to yield high OBC scores (e.g., 21 and 26). In addition to object volume,
we can also find objects that are obviously far away from the LiDAR center can
produce higher OBC values (from 16 to 23), and low-density and heavily
occluded objects also have high OBC values of 18 and 19, respectively. More
statistical analysis can be found in supplementary materials. Therefore, the
proposed OBC metric can effectively quantify the diversity of pseudo-labels in
multiple dimensions of geometric features, aiding the 3D detector to learn a
more diverse distribution of target objects, thereby alleviating
multidimensional object shift.

##  5 Conclusion

This paper revisits the setting of unsupervised domain-adaptive 3D detection
and investigates the generation of reliable, diverse, and class-balanced
pseudo labels for effective multi-class adaptation. The quantitative and
qualitative analysis demonstrates the effectiveness of the derived cross-
domain examination and OBC-based downsampling strategies, allowing pseudo
labels that are robust to environmental and object shifts to be preserved. The
class-balanced self-training helps detectors to learn frequently appearing and
rare classes simultaneously. The proposed approach is proven to be versatile
and can accommodate voxel-based and point-based 3D detectors.

##  6 Acknowledgments

This work was supported by Australian Research Council (DP230101196,
CE200100025).

## References

  * [1] Syeda Mariam Ahmed, Yan Zhi Tan, Chee-Meng Chew, Abdullah Al Mamun, and Fook Seng Wong.  Edge and corner detection for unorganized 3d point clouds with application to robotic welding.  In Proc. International Conference on Intelligent Robots and Systems (IROS), pages 7350–7355, 2018.
  * [2] Eduardo Arnold, Omar Y. Al-Jarrah, Mehrdad Dianati, Saber Fallah, David Oxtoby, and Alex Mouzakitis.  A survey on 3d object detection methods for autonomous driving applications.  IEEE Transactions on Intelligent Transportation Systems, 20(10):3782–3795, 2019.
  * [3] Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom.  nuscenes: A multimodal dataset for autonomous driving.  In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 11618–11628, 2020.
  * [4] Woong-Gi Chang, Tackgeun You, Seonguk Seo, Suha Kwak, and Bohyung Han.  Domain-specific batch normalization for unsupervised domain adaptation.  In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 7354–7362, 2019.
  * [5] Chao Chen, Zhihang Fu, Zhihong Chen, Sheng Jin, Zhaowei Cheng, Xinyu Jin, and Xian-Sheng Hua.  Homm: Higher-order moment matching for unsupervised domain adaptation.  In Proc. Conference on Artificial Intelligence (AAAI), pages 3422–3429, 2020.
  * [6] Zhuoxiao Chen, Yadan Luo, and Mahsa Baktashmotlagh.  Conditional extreme value theory for open set video domain adaptation.  In Proc. ACM International Conference on Multimedia in Asia (MM Asia), 2022.
  * [7] Robert DeBortoli, Fuxin Li, Ashish Kapoor, and Geoffrey A. Hollinger.  Adversarial training on point clouds for sim-to-real 3d object detection.  IEEE Robotics and Automation Letters, 6(4):6662–6669, 2021.
  * [8] Boyang Deng, Charles R. Qi, Mahyar Najibi, Thomas A. Funkhouser, Yin Zhou, and Dragomir Anguelov.  Revisiting 3d object detection from an egocentric perspective.  In Proc. Annual Conference on Neural Information Processing (NeurIPS), pages 26066–26079, 2021.
  * [9] Jiajun Deng, Shaoshuai Shi, Peiwei Li, Wengang Zhou, Yanyong Zhang, and Houqiang Li.  Voxel R-CNN: towards high performance voxel-based 3d object detection.  In Proc. Conference on Artificial Intelligence (AAAI), pages 1201–1209, 2021.
  * [10] Zhijie Deng, Yucen Luo, and Jun Zhu.  Cluster alignment with a teacher for unsupervised domain adaptation.  In Proc. International Conference on Computer Vision (ICCV), pages 9943–9952, 2019.
  * [11] Martin Engelcke, Dushyant Rao, Dominic Zeng Wang, Chi Hay Tong, and Ingmar Posner.  Vote3deep: Fast object detection in 3d point clouds using efficient convolutional neural networks.  In Proc. International Conference on Robotics and Automation (ICRA), pages 1355–1361, 2017.
  * [12] Lue Fan, Ziqi Pang, Tianyuan Zhang, Yu-Xiong Wang, Hang Zhao, Feng Wang, Naiyan Wang, and Zhaoxiang Zhang.  Embracing single stride 3d object detector with sparse transformer.  In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 8448–8458, 2022.
  * [13] Yaroslav Ganin and Victor S. Lempitsky.  Unsupervised domain adaptation by backpropagation.  In Proc. International Conference on Machine Learning (ICML), volume 37, pages 1180–1189, 2015.
  * [14] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor S. Lempitsky.  Domain-adversarial training of neural networks.  Journal of Machine Learning Research, 17:59:1–59:35, 2016.
  * [15] Andreas Geiger, Philip Lenz, and Raquel Urtasun.  Are we ready for autonomous driving? the KITTI vision benchmark suite.  In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3354–3361, 2012.
  * [16] Qiqi Gu, Qianyu Zhou, Minghao Xu, Zhengyang Feng, Guangliang Cheng, Xuequan Lu, Jianping Shi, and Lizhuang Ma.  PIT: position-invariant transform for cross-fov domain adaptation.  In Proc. International Conference on Computer Vision (ICCV), pages 8741–8750, 2021.
  * [17] Martin Hahner, Christos Sakaridis, Mario Bijelic, Felix Heide, Fisher Yu, Dengxin Dai, and Luc Van Gool.  Lidar snowfall simulation for robust 3d object detection.  In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 16343–16353, 2022.
  * [18] Chenhang He, Ruihuang Li, Shuai Li, and Lei Zhang.  Voxel set transformer: A set-to-set approach to 3d object detection from point clouds.  In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 8407–8417, 2022.
  * [19] Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei A. Efros, and Trevor Darrell.  Cycada: Cycle-consistent adversarial domain adaptation.  In Proc. International Conference on Machine Learning (ICML), volume 80, pages 1994–2003, 2018.
  * [20] Jordan S. K. Hu, Tianshu Kuai, and Steven L. Waslander.  Point density-aware voxels for lidar 3d object detection.  In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 8459–8468, 2022.
  * [21] Xin Jin, Cuiling Lan, Wenjun Zeng, and Zhibo Chen.  Re-energizing domain discriminator with sample relabeling for adversarial domain adaptation.  In Proc. International Conference on Computer Vision (ICCV), pages 9154–9163, 2021.
  * [22] Alex H. Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom.  Pointpillars: Fast encoders for object detection from point clouds.  In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 12697–12705, 2019.
  * [23] Alexander Lehner, Stefano Gasperini, Alvaro Marcos-Ramiro, Michael Schmidt, Mohammad-Ali Nikouei Mahani, Nassir Navab, Benjamin Busam, and Federico Tombari.  3d-vfield: Adversarial augmentation of point clouds for domain generalization in 3d object detection.  In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 17274–17283, 2022.
  * [24] Bo Li.  3d fully convolutional network for vehicle detection in point cloud.  In Proc. International Conference on Intelligent Robots and Systems (IROS), pages 1513–1518, 2017.
  * [25] Buyu Li, Wanli Ouyang, Lu Sheng, Xingyu Zeng, and Xiaogang Wang.  GS3D: an efficient 3d object detection framework for autonomous driving.  In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1019–1028, 2019.
  * [26] Shuang Li, Binhui Xie, Qiuxia Lin, Chi Harold Liu, Gao Huang, and Guoren Wang.  Generalized domain conditioned adaptation network.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(8):4093–4109, 2022.
  * [27] Xiaofeng Liu, Zhenhua Guo, Site Li, Fangxu Xing, Jane You, C.-C. Jay Kuo, Georges El Fakhri, and Jonghye Woo.  Adversarial unsupervised domain adaptation with conditional and label shift: Infer, align and iterate.  In Proc. International Conference on Computer Vision (ICCV), pages 10347–10356, 2021.
  * [28] Mingsheng Long, Yue Cao, Zhangjie Cao, Jianmin Wang, and Michael I. Jordan.  Transferable representation learning with deep adaptation networks.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(12):3071–3085, 2019.
  * [29] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I. Jordan.  Learning transferable features with deep adaptation networks.  In Proc. International Conference on Machine Learning (ICML), volume 37, pages 97–105, 2015.
  * [30] Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I. Jordan.  Unsupervised domain adaptation with residual transfer networks.  In Proc. Annual Conference on Neural Information Processing (NeurIPS), pages 136–144, 2016.
  * [31] Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I. Jordan.  Deep transfer learning with joint adaptation networks.  In Proc. International Conference on Machine Learning (ICML), volume 70, pages 2208–2217, 2017.
  * [32] Yadan Luo, Zhuoxiao Chen, Zijian Wang, Xin Yu, Zi Huang, and Mahsa Baktashmotlagh.  Exploring active 3d object detection from a generalization perspective.  In Proc. International Conference on Learning Representations (ICLR), 2023.
  * [33] Yadan Luo, Zi Huang, Zijian Wang, Zheng Zhang, and Mahsa Baktashmotlagh.  Adversarial bipartite graph learning for video domain adaptation.  In Proc. ACM International Conference on Multimedia (MM), pages 19–27, 2020.
  * [34] Yadan Luo, Zijian Wang, Zhuoxiao Chen, Zi Huang, and Mahsa Baktashmotlagh.  Source-free progressive graph learning for open-set domain adaptation.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9):11240–11255, 2023.
  * [35] Yadan Luo, Zijian Wang, Zi Huang, and Mahsa Baktashmotlagh.  Progressive graph learning for open-set domain adaptation.  In Proc. International Conference on Machine Learning (ICML), volume 119 of Proceedings of Machine Learning Research, pages 6468–6478, 2020.
  * [36] Zhipeng Luo, Zhongang Cai, Changqing Zhou, Gongjie Zhang, Haiyu Zhao, Shuai Yi, Shijian Lu, Hongsheng Li, Shanghang Zhang, and Ziwei Liu.  Unsupervised domain adaptive 3d detection with multi-level consistency.  In Proc. International Conference on Computer Vision (ICCV), pages 8846–8855, 2021.
  * [37] Jiageng Mao, Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li.  3d object detection for autonomous driving: A review and new outlooks.  CoRR, abs/2206.09474, 2022.
  * [38] Scott McCrae and Avideh Zakhor.  3d object detection for autonomous driving using temporal lidar data.  In Proc. International Conference on Image Processing (ICIP), pages 2661–2665, 2020.
  * [39] Gregory P. Meyer, Ankit Laddha, Eric Kee, Carlos Vallespi-Gonzalez, and Carl K. Wellington.  Lasernet: An efficient probabilistic 3d object detector for autonomous driving.  In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 12677–12686, 2019.
  * [40] Hector A. Montes, Justin Le Louedec, Grzegorz Cielniak, and Tom Duckett.  Real-time detection of broccoli crops in 3d point clouds for autonomous robotic harvesting.  In Proc. International Conference on Intelligent Robots and Systems (IROS), pages 10483–10488, 2020.
  * [41] Xuran Pan, Zhuofan Xia, Shiji Song, Li Erran Li, and Gao Huang.  3d object detection with pointformer.  In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 7463–7472, 2021.
  * [42] Rui Qian, Xin Lai, and Xirong Li.  3d object detection for autonomous driving: A survey.  Pattern Recognition, 130:108796, 2022.
  * [43] Christoph B Rist, Markus Enzweiler, and Dariu M Gavrila.  Cross-sensor deep domain adaptation for lidar detection and segmentation.  In Proc. Intelligent Vehicles Symposium, (IV), pages 1535–1542, 2019.
  * [44] Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada.  Maximum classifier discrepancy for unsupervised domain adaptation.  In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3723–3732, 2018.
  * [45] Khaled Saleh, Ahmed Abobakr, Mohammed Hassan Attia, Julie Iskander, Darius Nahavandi, Mohammed Hossny, and Saeid Nahavandi.  Domain adaptation for vehicle detection from bird’s eye view lidar point cloud data.  In Proc. International Conference on Computer Vision, pages 3235–3242, 2019.
  * [46] Cristiano Saltori, Stéphane Lathuilière, Nicu Sebe, Elisa Ricci, and Fabio Galasso.  Sf-uda${}^{\mbox{3d}}$: Source-free unsupervised domain adaptation for lidar-based 3d object detection.  In Proc. International Conference on 3D Vision (3DV), pages 771–780, 2020.
  * [47] Guangsheng Shi, Ruifeng Li, and Chao Ma.  Pillarnet: Real-time and high-performance pillar-based 3d object detection.  In Proc. European Conference on Computer Vision (ECCV), volume 13670, pages 35–52, 2022.
  * [48] Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li.  Pointrcnn: 3d object proposal generation and detection from point cloud.  In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–779, 2019.
  * [49] Shaoshuai Shi, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li.  From points to parts: 3d object detection from point cloud with part-aware and part-aggregation network.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(8):2647–2664, 2021.
  * [50] Weijing Shi and Raj Rajkumar.  Point-gnn: Graph neural network for 3d object detection in a point cloud.  In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1708–1716, 2020.
  * [51] Baochen Sun and Kate Saenko.  Deep CORAL: correlation alignment for deep domain adaptation.  In Proc. European Conference on Computer Vision Workshops (ECCV Workshops), volume 9915 of Lecture Notes in Computer Science, pages 443–450, 2016.
  * [52] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Yu Zhang, Jonathon Shlens, Zhifeng Chen, and Dragomir Anguelov.  Scalability in perception for autonomous driving: Waymo open dataset.  In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2443–2451, 2020.
  * [53] Pei Sun, Mingxing Tan, Weiyue Wang, Chenxi Liu, Fei Xia, Zhaoqi Leng, and Dragomir Anguelov.  Swformer: Sparse window transformer for 3d object detection in point clouds.  In Proc. European Conference on Computer Vision (ECCV), volume 13670, pages 426–442, 2022.
  * [54] Antti Tarvainen and Harri Valpola.  Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.  In Proc. Annual Conference on Neural Information Processing (NeurIPS), pages 1195–1204, 2017.
  * [55] OpenPCDet Development Team.  Openpcdet: An open-source toolbox for 3d object detection from point clouds.  https://github.com/open-mmlab/OpenPCDet, 2020.
  * [56] Jonathan Tremblay, Thang To, and Stan Birchfield.  Falling things: A synthetic dataset for 3d object detection and pose estimation.  In Proc. IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPR Workshops), pages 2038–2041, 2018.
  * [57] Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell.  Adversarial discriminative domain adaptation.  In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2962–2971, 2017.
  * [58] Haiyang Wang, Shaoshuai Shi, Ze Yang, Rongyao Fang, Qi Qian, Hongsheng Li, Bernt Schiele, and Liwei Wang.  Rbgnet: Ray-based grouping for 3d object detection.  In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1100–1109, 2022.
  * [59] Jun Wang, Shiyi Lan, Mingfei Gao, and Larry S. Davis.  Infofocus: 3d object detection for autonomous driving with dynamic information modeling.  In Proc. European Conference on Computer Vision (ECCV), volume 12355, pages 405–420. Springer, 2020.
  * [60] Li Wang, Ruifeng Li, Jingwen Sun, Xingxing Liu, Lijun Zhao, Hock Soon Seah, Chee Kwang Quah, and Budianto Tandianus.  Multi-view fusion-based 3d object detection for robot indoor scene perception.  Sensors, 19(19):4092, 2019.
  * [61] Yan Wang, Wei-Lun Chao, Divyansh Garg, Bharath Hariharan, Mark E. Campbell, and Kilian Q. Weinberger.  Pseudo-lidar from visual depth estimation: Bridging the gap in 3d object detection for autonomous driving.  In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 8445–8453, 2019.
  * [62] Yan Wang, Xiangyu Chen, Yurong You, Li Erran Li, Bharath Hariharan, Mark E. Campbell, Kilian Q. Weinberger, and Wei-Lun Chao.  Train in germany, test in the USA: making 3d object detectors generalize.  In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 11710–11720, 2020.
  * [63] Yue Wang, Alireza Fathi, Abhijit Kundu, David A. Ross, Caroline Pantofaru, Thomas A. Funkhouser, and Justin Solomon.  Pillar-based object detection for autonomous driving.  In Proc. European Conference on Computer Vision (ECCV), volume 12367 of Lecture Notes in Computer Science, pages 18–34, 2020.
  * [64] Zijian Wang, Yadan Luo, Zi Huang, and Mahsa Baktashmotlagh.  Prototype-matching graph network for heterogeneous domain adaptation.  In Proc. ACM International Conference on Multimedia (MM), pages 2104–2112, 2020.
  * [65] Zixin Wang, Yadan Luo, Peng-Fei Zhang, Sen Wang, and Zi Huang.  Discovering domain disentanglement for generalized multi-source domain adaptation.  In Proc. International Conference on Multimedia and Expo (ICME), pages 1–6, 2022.
  * [66] Yi Wei, Zibu Wei, Yongming Rao, Jiaxin Li, Jie Zhou, and Jiwen Lu.  Lidar distillation: Bridging the beam-induced domain gap for 3d object detection.  In Proc. European Conference on Computer Vision (ECCV), volume 13699 of Lecture Notes in Computer Science, pages 179–195, 2022.
  * [67] Qiangeng Xu, Yin Zhou, Weiyue Wang, Charles R. Qi, and Dragomir Anguelov.  SPG: unsupervised domain adaptation for 3d object detection via semantic point generation.  In Proc. International Conference on Computer Vision (ICCV), pages 15426–15436, 2021.
  * [68] Yan Yan, Yuxing Mao, and Bo Li.  SECOND: sparsely embedded convolutional detection.  Sensors, 18(10):3337, 2018.
  * [69] Bin Yang, Wenjie Luo, and Raquel Urtasun.  PIXOR: real-time 3d object detection from point clouds.  In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 7652–7660, 2018.
  * [70] Jihan Yang, Shaoshuai Shi, Zhe Wang, Hongsheng Li, and Xiaojuan Qi.  ST3D: self-training for unsupervised domain adaptation on 3d object detection.  In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 10368–10378, 2021.
  * [71] Jihan Yang, Shaoshuai Shi, Zhe Wang, Hongsheng Li, and Xiaojuan Qi.  St3d++: denoised self-training for unsupervised domain adaptation on 3d object detection.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022\.
  * [72] Zetong Yang, Yanan Sun, Shu Liu, and Jiaya Jia.  3dssd: Point-based 3d single stage object detector.  In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 11037–11045, 2020.
  * [73] Zetong Yang, Yanan Sun, Shu Liu, Xiaoyong Shen, and Jiaya Jia.  STD: sparse-to-dense 3d object detector for point cloud.  In Proc. International Conference on Computer Vision (ICCV), pages 1951–1960, 2019.
  * [74] Cang Ye and Xiangfei Qian.  3-d object recognition of a robotic navigation aid for the visually impaired.  IEEE Transactions on Neural Systems and Rehabilitation Engineering, 26(2):441–450, 2018.
  * [75] Tianwei Yin, Xingyi Zhou, and Philipp Krähenbühl.  Center-based 3d object detection and tracking.  In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 11784–11793, 2021.
  * [76] Yurong You, Katie Luo, Cheng Perng Phoo, Wei-Lun Chao, Wen Sun, Bharath Hariharan, Mark E. Campbell, and Kilian Q. Weinberger.  Learning to detect mobile objects from lidar scans without labels.  In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1120–1130, 2022.
  * [77] Yurong You, Cheng Perng Phoo, Katie Luo, Travis Zhang, Wei-Lun Chao, Bharath Hariharan, Mark Campbell, and Kilian Q. Weinberger.  Unsupervised adaptation from repeated traversals for autonomous driving.  In Proc. Annual Conference on Neural Information Processing (NeurIPS), 2022.
  * [78] Yurong You, Yan Wang, Wei-Lun Chao, Divyansh Garg, Geoff Pleiss, Bharath Hariharan, Mark E. Campbell, and Kilian Q. Weinberger.  Pseudo-lidar++: Accurate depth for 3d object detection in autonomous driving.  In Proc. International Conference on Learning Representations (ICLR), 2020.
  * [79] Yihan Zeng, Chunwei Wang, Yunbo Wang, Hang Xu, Chaoqiang Ye, Zhen Yang, and Chao Ma.  Learning transferable features for point cloud detection via 3d contrastive co-training.  In Proc. Annual Conference on Neural Information Processing (NeurIPS), pages 21493–21504, 2021.
  * [80] Weichen Zhang, Wen Li, and Dong Xu.  SRDAN: scale-aware and range-aware domain adaptation network for cross-dataset 3d object detection.  In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6769–6779, 2021.
  * [81] Yifan Zhang, Qingyong Hu, Guoquan Xu, Yanxin Ma, Jianwei Wan, and Yulan Guo.  Not all points are equal: Learning highly efficient point-based detectors for 3d lidar point clouds.  In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 18931–18940, 2022.
  * [82] Yin Zhou and Oncel Tuzel.  Voxelnet: End-to-end learning for point cloud based 3d object detection.  In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4490–4499, 2018.
  * [83] Zhengxue Zhou, Leihui Li, Alexander Fürsterling, Hjalte Joshua Durocher, Jesper Mouridsen, and Xuping Zhang.  Learning-based object detection and localization for a mobile robot manipulator in SME production.  Robotics and Computer-Integrated Manufacturing, 73:102229, 2022\.


# Evaluation of AI-Supported Input Methods in Augmented Reality Environment

Akos Nagy
Department of Networks & Digital Media
School of Computer Science & Maths,
ECE, Kingston University
Kingston upon Thames, UK
<EMAIL_ADDRESS>
&Thomas Lagkas
Department of Computer Science
International Hellenic University Kavala Campus
Greece & South-East European Research Centre
<EMAIL_ADDRESS>
&Panagiotis Sarigiannidis
Department of Electrical
and Computer Engineering
University of Western Macedonia
Kozani,Greece
<EMAIL_ADDRESS>
&Vasileios Argyriou
Department of Networks & Digital Media
School of Computer Science & Maths,
ECE, Kingston University
Kingston upon Thames, UK
<EMAIL_ADDRESS>

###### Abstract

Augmented Reality (AR) solutions are providing tools that could improve
applications in the medical and industrial fields. Augmentation can provide
additional information in training, visualization, and work scenarios, to
increase efficiency, reliability, and safety, while improving communication
with other devices and systems on the network. Unfortunately, tasks in these
fields often require both hands to execute, reducing the variety of input
methods suitable to control AR applications. People with certain physical
disabilities, where they are not able to use their hands, are also negatively
impacted when using these devices. The goal of this work is to provide novel
hand-free interfacing methods, using AR technology, in association with AI
support approaches to produce an improved Human-Computer interaction solution.

_K_ eywords Augmented Reality, HCI, hands-free interfaces, AI-guided
interaction

##  1 Introduction

Augmented Reality is a rapidly evolving technology that has the potential to
transform a wide range of industries, from healthcare [1, 2, 3] to
manufacturing [4, 5]. By overlaying digital information onto the physical
world, provided by systems on a network [6], AR offers a powerful tool for
training, visualization, and work scenarios, enabling users to increase
efficiency, reliability, and safety. However, one of the challenges facing AR
is the limited range of input methods that are available for controlling AR
applications, particularly in scenarios where both hands are required to
perform a task. This limitation not only impacts the usability of AR but also
has significant implications for people who, either require both hands for
work processes, observed in industrial or medical fields, or with certain
physical disabilities, who may not be able to use their hands to interact with
these devices [7, 8].

A variety of input methods are available using AR technologies[9, 10], most of
which face different limitations in working environments [11, 12, 13]. Hand
tracking allows the user to use their hand to interact with virtual objects or
perform certain gestures to communicate with the device, which limits the use
case when both hands are required while working. Additional peripherals can
provide inputs, but these devices add additional cost to purchase, while
operating them can require non-natural, or cause limited movement[14, 15, 16,
17]. To realise these AR interfaces and solutions, technologies based on
computer vision, 3D reconstruction, scene analysis, motion estimation, and
object detection are required [18, 19]. Freehand input control offers
potential solutions to these challenges, allowing users to interact with AR
applications without the need to use their hands. This technology enables
users to control AR applications through natural body movements, such as head
movements and voice commands [20], freeing up their hands for other tasks.
Voice commands can be limited in certain environments, where the general
background noise level is too high for reliable voice perception, [21, 22].

As a viable interaction solution, this work focuses on improving head
orientation-based input methods by evaluating head- and image-based input
solutions assisted by Interpolation and Gravity-Map AI support approaches and
compares them with classical input methods, such as mouse and gamepad. The
main contributions of this paper are the following:

  * a)

Introduce a novel AI-supported interfacing approach to improve user experience

  * b)

Defined evaluation framework for comparing input methods, including task
specification and evaluation metrics

  * c)

Evaluation of AI-support-assisted alternative input approaches and comparison
with classical input methods

The paper is structured as follows: Section 2 provides a review off related
research and methods. The proposed AI-enabled interaction methods for hands-
free applications are detailed in section 3. Section 4 showcases the results
using both the classic and the proposed interaction methods in a comparative
study. Finally, the conclusion is presented in section 5.

##  2 RELATED WORK

Human-Computer Interaction (HCI) is a core element of AR use cases.
Augmentation allows adaptive HCIs, that form to the human body, or allows
itself to be an input device. Numerous studies contribute to finding the most
optimal solutions in a wide range of scenarios.

Frusto-Pascual [23] examines different virtual keyboard positions and
interaction feedback methods for character input in augmented reality. Two
categories of keyboard position conditions are presented: viewpoint bound and
non-dominant hand bound conditions. Viewpoint bound conditions are defined
based on the position and orientation of the head-mounted display (HMD), while
non-dominant hand bound conditions are defined based on the position and
orientation of the non-dominant hand. Two forms of visual feedback and
guidance for interaction with the keyboard are evaluated: Raycast and Glow.
Raycast is a continuous guidance ray to the keyboard, while Glow is a visual
representation of the dominant hand index fingertip. Both feedback methods are
combined in the "Both combined" condition.

A system proposed by Chen [24] is developed within a VR environment that
allows freehand manipulation of 3D objects. The system includes a VR
disambiguation menu with preview cubes, which vary in timing and modality
based on the techniques. The three promising input modalities, explored for
hands-free operation in VR are head gaze, speech, and foot. Also, the timing
of disambiguation is explored. Although the methods are proposed related to VR
environment, are applicable using AR devices.

The foot-based method presents some limitations that may make it unsuitable,
as mentioned before, for some users and certain environments. One significant
limitation is that not everyone can use their feet effectively, such as
individuals with certain disabilities or injuries that affect their lower
extremities. Consequently, the foot-based method may not be an inclusive
solution, and alternative selection methods should be considered to
accommodate a broader range of users.

Moreover, the foot-based method requires additional equipment to track the
user’s foot movements accurately, which could increase the complexity of the
system and may not be practical in industrial environments. For example,
workers in manufacturing or construction sites may be wearing heavy-duty work
boots, steel-toed shoes, or other safety equipment that could interfere with
the accuracy of the foot-tracking.

Eye gaze-based interfaces for human-computer interaction are becoming
increasingly popular and are being used in various sectors such as tele-
operation [25]. In tele-operation, the operator’s eyes are occupied with
monitoring the environment and their hands are busy controlling tasks. Eye
gaze-based interfaces provide a natural user interface that is becoming
increasingly popular and could become the interface of choice for many
devices. Researchers have introduced various eye gaze tracking systems for
tele-operation, including those for people with disabilities. A cost-effective
and precise eye tracker has been developed, using an eye camera with two IR-
emitting LEDs next to the camera lens [26]. A gaze tracking system for tele-
operation for people with physical disabilities was introduced, using an
analog video-type camera to capture images and a gaze estimation algorithm to
control the remote [27]. An eye tracking-based mobile robot for tele-operation
was also proposed, using a Microsoft Kinect 3D camera sensor for eye tracking.
An eye gaze-based interface for a smart wheelchair was developed to assist
people with locomotor disabilities. The system includes image processing for
eye tracking and a module for controlling the movement of the wheelchair. It
also includes added functionality such as sending messages through a
smartphone and controlling electrical devices [28].

A study [13] conducted in 2021 investigates the effectiveness of a gaze-
adaptive user interface (UI) for Augmented Reality in comparison to an always-
on interface, presented. Previous research has suggested that eye gaze could
be a potential solution to the attention dilemma that arises when users must
shift their focus between the real world and AR content [11, 12]. The study
presents the findings of an empirical investigation that evaluated the
performance of gaze-adaptive UI in comparison to the always-on interface. The
participants were asked to perform tasks that involved shifting their focus
between the real world and virtual content. The results showed that most
participants preferred the gaze-adaptive UI and found it less distracting. The
gaze UI was faster, more intuitive, and perceived as easier to use when
focusing on reality. When focusing on virtual content, the always-on interface
was faster, but the user preferences were divided.

Figure 1: AR - AI Control Support Architecture [13]

The proposed system, as presented with Figure 1, provides additional
information when the user gazes at a particular point, triggered by the
temporal and spatial activation dimensions. Temporal activation determines how
long a user needs to look at an object to trigger the AR panel, while spatial
activation defines the size of the area a user needs to gaze at. The paper
aims to formalize the properties of a gaze-adaptive AR system and better
ground the design and study efforts.

Figure 2: AR - AI Control Support Architecture

##  3 METHODOLOGY

Augmented Reality devices have the potential to revolutionize the way
industrial workers receive or interact with information. AR devices can
provide hands-free access to information and instructions and communication
with network-connected devices, allowing workers to focus on their tasks while
still having the necessary information at their disposal. This can be achieved
through the use of head-mounted displays, smart glasses, or even smartphones.
The AR technology overlays digital information in the real-world environment,
providing workers with real-time access to important data such as
instructions, diagrams, and technical specifications.

The present study focuses on Hand-free controls supported HCI in cases where
using hands is not applicable. These cases include work environments where the
person requires both hands to perform the tasks required by the processes,
i.e.: industrial or medical fields. Other scenarios where due to an injury or
a medical condition, where the person is not capable of using their hands also
limit the usage of AR devices. Alternatively, a voice command could be used in
these cases, but in certain environments, due to the high volume of background
noise or the necessity of a quiet environment may not allow a such solution.

The following section describes a methodology of an AI-assisted architecture
that supports hand-free control using AR technologies. The comparative study
provides an evaluation of the performance and usability of the proposed hand-
free control solutions compared to traditional input methods such as keyboard
and mouse. The aim is to assess the potential of AR technologies for creating
more intuitive and natural user interfaces for various applications.

While the mouse on personal computers, gamepads on console devices, and touch
screens on mobile devices are part of the most common human-computer
interfaces, in certain scenarios, these approaches might not be the most
suitable. Physical limitations may be caused by the working environment, like
industrial settings, where personnel requires both hands to perform the
required actions, or due to disabilities or injuries, where the person’s
physical capabilities are impaired, these devices, due to their design, are
not suitable for human-computer interactions.

The goal of the following proposed framework for HCI is to design a general
pipeline architecture, demonstrated in Figure 2, involving AI-assisted
solutions, that allows the use of a variety of alternative input methods while
minimizing the discrepancy in usability compared to classical input devices.
In our study, we consider two input mechanisms as classical input devices, the
mouse, most commonly used with personal computers, and gamepads, which are
mainly used with console devices and dominantly for entertainment purposes. As
alternative input methods, assisted image and face tracking-based solutions
are proposed with applications in immersive environments. The face tracking
method uses computer vision algorithms to determine the position and
orientation of the user’s head, and based on these parameters provides
constantly input to the corresponding system, while the image tracking
solution again based on computer vision methods, detects the selected image
target. and constantly tracks its position and orientation in the 3D world
coordinates.

###  3.1 AI Support Approaches

The proposed AI-assisted solutions, integrated into the architecture, are
designed to reduce the difference between the classical input devices and the
alternative input solutions or devices. Using the input vector these solutions
will alter the data before providing it to the testing application. The
current study considers three different solutions as part of the experiment:

Interpolation is a powerful technique in computer graphics and computer vision
that can be used to estimate and derive values between known data points. In
the context of hand-free control, the interpolation technique allows for the
deviation of the input vector toward the intended target based on a specific
set of limitations and criteria. The input vector’s deviation is determined by
considering the distance of the target from the input vector and the original
orientation of the input vector.

This balance between forced, where the AI is influencing the input method, and
free control, where there is no influence, is crucial in ensuring that the
system is both responsive and intuitive to the user’s input, while also
maintaining a level of control over the target object. The technique allows
for smooth and gradual adjustments to the input vector so that the target can
be reached in an efficient and accurate manner.

The Pseudo code for the implementation is as seen in Algorithm 1.

Algorithm 1 InterpolationPrediction

1:

2:$start$: current coordinate,

3:$moveVec$: current input vector ,

4:$influence$, default(0.8f)

5:$number$: number of iterations, how many possible future positions need to
be predicted; default(1)

6: $p$: an array of predicted influenced points

7:function InterpolationPrediction($start$, $moveVec$, $target$,
$influence=0.8$, $number=1$)

8: $p\leftarrow[start,start+moveVec]$

9:

10: for $i\leftarrow 0$ to $number$ do

11: $c\leftarrow p[\operatorname{length}(p)-2]$

12: $tVec\leftarrow target-c$

13:

14: if $moveVec.\operatorname{mag}()==0\lor tVec.\operatorname{mag}()==0$ then

15: append $c$ to $p$

16: continue

17: end if

18:

19: $tVecNorm\leftarrow\operatorname{normalize}(tVec)$

20: $moveVecNorm\leftarrow\operatorname{normalize}(moveVec)$

21:

22:
$mod\leftarrow\operatorname{max}(tVecNorm.\operatorname{dot}(nVecNorm),influence)-influence$

23: $mod\leftarrow mod\times(1/(1-influence))$

24:

25: $fVec\leftarrow\operatorname{lerp}(moveVecNorm,tVecNorm,mod)$

26:
$modDist\leftarrow\operatorname{min}(tVec.\text{mag}(),moveVec.\text{mag}())$

27: $next\leftarrow fVec\times modDist+c$

28: append $next$ to $p$

29: end for

30:

31: return elements of $p$ starting from index $1$

32:end function

The Gravity-Map approach solution for controlling input vectors is a more
flexible approach compared to the Interpolation approach and can be used to
control the deviation of the input vector toward multiple target objects. In
this approach, the interface is divided into two areas, with one area being
defined as the area of effect, and the other being the unaffected area. The
area of effect is determined by the proximity of the target objects to the
current focus position. When the cursor position is within the area of effect,
the input vector is deviated toward the intended target object based on its
distance and original orientation.

The Gravity-Map approach is different from other interpolation techniques in
that it considers the relationship between the input vector and multiple
target objects, rather than just one target object at a time. This allows for
a more intuitive control experience, as the input vector is pulled towards the
target objects that are closest and within reach. Furthermore, dividing the
interface into two areas provides a balance between forced and free control.
In the area of effect, the input vector is automatically directed toward the
target objects, providing assistance for users. However, outside of this area,
the input vector remains unaltered, giving users the freedom to move and
control the cursor as they see fit.

The Pseudo code for the implementation is as seen in Algorithm 2.

Algorithm 2 Calculate influence vector at coordinate

1:

2:targets (list of rectangles with $x,y$, width, height attributes),

3:$PX$ (x coordinate of a point),

4:$PY$ (y coordinate of a point),

5:$influenceDistance$ (the max distance a given target is allowed to influence
the outcome)

6:influence vector at point described by $PX$ and $PY$

7:function GravityMapInfluence($targets$, $PX$, $PY$, $influenceDistance$)

8: $retInfluence\leftarrow(0,0)$;

9: for $i\leftarrow 0$ to $\text{length}(targets)$ do

10: $closestPoint\leftarrow$ calculate closest point within the target area to
$(PX,PY)$;

11: if $(PX,PY)$ is within the target area then

12: return $(0,0)$;

13: end if

14: if distance between $closestPoint$ and $(PX,PY)$ is greater than
$influenceDistance$ then

15: continue;

16: end if

17: $direction\leftarrow$ calculate direction vector towards $closestPoint$;

18: $distanceRatio\leftarrow$ distance between $closestPoint$ and $(PX,PY)$ /
$influenceDistance$;

19: $influence\leftarrow direction\times(1-distanceRatio^{2})$;

20: $retInfluence\leftarrow retInfluence+influence$;

21: end for

22: return $\mathbf{retInfluence}$;

23:end function

###  3.2 Data-collection Methods

Data collection has been performed via a data evaluation application (Figure
3), which encompasses three modes, each with its own task to perform and
metrics to evaluate. The study involved 20 participants.

Figure 3: Evaluation application (in follow mode) screenshot with crosshair
and target (white)

Locate Mode

In this mode, a series of static targets are displayed within the 3D
environment. The participant’s objective is to align the focus point, which is
represented with a crosshair in the application, with the target. The task
consists of multiple targets, each of which constitutes a separate subtask,
organized in a sequential manner with only one target being visible at any
given time. In this mode, we measure the average time it takes for a
participant to reach the target position with the various input methods and
solutions. Each target has a defined window of availability, beyond which it
will be removed, and the corresponding subtask will be considered as failed.

Select Mode

In this mode, the participant’s task is to precisely position the crosshair,
which serves as the center of the screen, onto the static targets in a 3D
environment. The evaluation consists of a series of subtasks, each of which
involves positioning the crosshair onto a target and maintaining it there
continuously for a set amount of time. The targets are presented in a
sequential manner, with only one target being visible at a time.

The accuracy of the various input methods and solutions was assessed by
conducting experiments and measuring the ability of the input method to stay
within a designated region of the screen. The success rate of each input
method was calculated based on the number of successfully completed subtasks,
and the extra time spent on the target was recorded for subtasks that took
longer than necessary to complete.

Follow Mode

In the Follow Mode, the participants are tasked with tracking a moving target,
as opposed to a static target in the other modes. The targets move along a
predefined path at predetermined speeds to control the experiment and
determine the relationship between success and speed. The objective is to
position the crosshair, which serves as the center of the screen, onto the
target and follow it along the designated path between the start and end
marks. This mode evaluates the precision of the input solutions in a dynamic
target scenario, similar to the Select Mode.

##  4 Results

###  4.1 Quantitative Metrics

In this section, we will discuss the various metrics used for the Quantitative
results. In the section, the focus point refers to the center of the screen,
represented by a crosshair in the testing application, as displayed on Figure
4.

Figure 4: Evaluation application screenshot with crosshair and target (cyan)

Locate Mode

Avg. Reach Time: Let $S_{locate}$ be the set of sub-tasks and let $n$ be the
total number of sub-tasks in $S_{locate}$. Let $S^{\prime}_{locate}$ be the
subset of $S_{locate}$ consisting of the sub-tasks that were completed
successfully. A sub-task constitutes success if the focus point overlaps with
the target object. For every sub-task, we record two timestamps,
$TargetAppearedTimeStamp$, when the target object appeared on the game, and
$TargetReachedTimeStamp$, when the focus point reached the target object. From
these values, we calculate a $ReachTime$ for every sub-task, which is the
difference between the previous two timestamps.

$\text{Avg. Reach Time}=\frac{1}{n}\sum_{s\in
S^{\prime}_{locate}}s_{ReachTime}$

$s_{ReachTime}=s_{TargetAppearedTimeStamp}-s_{TargetReachedTimeStamp}$

Target Reached %: Let $S_{locate}$ be the set of sub-tasks and let $n$ be the
total number of sub-tasks in $S_{locate}$. Let $S^{\prime}_{locate}$ be the
subset of $S_{locate}$ consisting of the sub-tasks that were completed
successfully. This formula calculates the average of successful sub-tasks by
dividing the sum of all successful sub-tasks by the total number of sub-tasks
in $S_{locate}$.

$\text{Target Reached \%}=\frac{1}{n}\sum_{s\in S^{\prime}_{locate}}s$

Select Mode:

Avg. Extra Time Required: Let $S_{Select}$ be the set of sub-tasks in Locate
Mode and let $n$ be the total number of sub-tasks in $S_{Select}$. Let
$S^{\prime}_{Select}$ be the subset of $S_{Select}$ consisting of the sub-
tasks that were completed successfully. A sub-task constitutes success if the
focus point overlaps with the target object for a set $SelectTarget$ time.

$\text{Avg. Extra Time Required}=\frac{1}{n}\sum_{s\in
S^{\prime}_{select}}(s_{T_{overlap}}-SelectTarget)$

Target Selected %: Let $S_{Select}$ be the set of sub-tasks in Select Mode and
let $n$ be the total number of sub-tasks in $S_{Select}$. Let
$S^{\prime}_{Select}$ be the subset of $S_{Select}$ consisting of the sub-
tasks that were completed successfully. This formula calculates the average of
successful sub-tasks by dividing the sum of all successful sub-tasks by the
total number of sub-tasks in $S_{Select}$.

$\text{Target Selected \%}=\frac{1}{n}\sum_{s\in S^{\prime}_{select}}s$

Follow Mode:

Avg. Follow %: Let $S_{Follow}$ be the set of sub-tasks in Follow Mode and let
$n$ be the total number of sub-tasks in $S_{Follow}$. Let
$S^{\prime}_{Follow}$ be the subset of $S_{Follow}$ consisting of the sub-
tasks that were completed successfully. This formula calculates the average
percentage of overlap time during the successful sub-tasks by dividing the sum
of all percentages of overlap time for sub-tasks by the total number of sub-
tasks in $S^{\prime}_{Follow}$. The percentage of overlap time for a sub-task
is the overlap time divided by the fly-time of the sub-tasks. Fly-time is the
total amount of time the target object is available on the screen, and overlap
time is the time the focus point is overlapping with the target object during
the sub-task.

$\text{Avg. Follow \%}=\frac{1}{n}\sum_{s\in
S^{\prime}_{Follow}}(s_{OverlapTime}/s_{FlyTime})$

Moving Target Touched % : Let $S_{Follow}$ be the set of sub-tasks in Follow
Mode and let $n$ be the total number of sub-tasks in $S_{Follow}$. Let
$S^{\prime}_{Follow}$ be the subset of $S_{Follow}$ consisting of the sub-
tasks that were completed successfully. This formula calculates the average of
successful sub-tasks by dividing the sum of all successful sub-tasks by the
total number of sub-tasks in $S_{Follow}$. A sub-task constitutes success if
the focus point overlaps with the target object.

$\text{Moving Target Touched \% }=\frac{1}{n}\sum_{s\in S^{\prime}_{Follow}}s$

###  4.2 Data evaluation

The evaluated data is separated into three tables, Table 1, Table 2 and Table
3, one for each of the data-collection modes, displaying the previously
discussed metrics associated with the modes. Each table is also separated into
two sections to showcase the differences between the input methods without AI
support, as baseline values, and with AI support, to evaluate the efficiency
of the support approaches. Each section has the best-performing values
highlighted.

The results show that the Mouse input method is the overall best-performing
among the input methods evaluated in this study. Among the AI-supported input
methods, the Head movement-based, Gravity-Map approach performed the best, in
certain aspects even better than the Mouse input.

Locate Mode

While the Mouse input method performed 48% better than the overall average in
terms of Reach Time, it’s only 18% better than the Gravity-Map assisted Head
input, which has gained a 42% improvement compared to not having AI support.
Image-based input was the worst-performing, not AI-supported method, and while
the Gravity-Map AI-support approach slightly improves the method, using the
Interpolation approach increased the Average Reach Time by 18%.

Table 1: OUTCOMES FOR ALL THE INPUT METHODS IN LOCATE MODE

| Input | Target Reached % | Avg. Reach Time
---|---|---|---
No AI Support | Mouse | 98.3% | 0.89s
Gamepad | 97.8% | 1.34s
Head | 96.0% | 1.86s
Image | 75.5% | 2.17s
AI Support | Head - Interpolation | 96.2% | 1.78s
Head - Gravity-Map | 99.78% | 1.08s
Image - Interpolation | 76.1% | 2.56s
Image - Gravity-Map | 83.6% | 1.82s
| Overall Average | 90.4% | 1.69s

Select Mode

The Mouse input method performed over 9X better than the overall average in
terms of Extra Time Required, due to the fact that a mouse does not have
continuous input like the alternative input methods. While using the
alternative methods, there’s a continuous movement, even in the target (head,
image) is seemingly motionless. The Head-based input using the Gravity-Map
approach performed over 4X better, and became the third fastest solution, than
the Head-based input without AI support, which required 2X more time to
perform the tasks than the overall average, being the overall slowest
solution. In terms of the Select Mode, a higher Extra Time Required value
indicates a less stable input solution when required to dwell on a fixed
point.

Table 2: OUTCOMES FOR ALL THE INPUT METHODS IN SELECT MODE

| Input | Target Selected % | Avg. Extra Time Required
---|---|---|---
No AI Support | Mouse | 100% | 0.042s
Gamepad | 95.8% | 0.099s
Head | 85.5% | 0.779s
Image | 70.5% | 0.470s
AI Support | Head - Interpolation | 81.9% | 0.659s
Head - Gravity-Map | 99.6% | 0.178s
Image - Interpolation | 72.6% | 0.494s
Image - Gravity-Map | 90.1% | 0.343s
| Overall Average | 87.0% | 0.383s

Follow Mode

The worst-performing solution was the Image-based input assisted by the
Interpolation AI-support, resulting in only 50% Avg. Follow % compared to the
average and 20%-25%, compared to the best-performing solutions. The two best-
performing solutions were the Mouse and Head-based input method, assisted by
the Gravity-Map AI-support approach. Only these two solutions were able to
perform better than the overall average, in terms of Avg. Follow %, with the
Mouse-based input method performing 16% better than the second-best solution.

Table 3: OUTCOMES FOR ALL THE INPUT METHODS IN FOLLOW MODE

| Input | Moving Target Touched % | Avg. Follow %
---|---|---|---
No AI Support | Mouse | 99.4% | 48.1%
Gamepad | 99.1% | 19.3%
Head | 83.3% | 14.2%
Image | 75.5% | 17.9%
AI Support | Head - Interpolation | 81.5% | 11.9%
Head - Gravity-Map | 99.5% | 41.3%
Image - Interpolation | 66.2% | 11.1%
Image - Gravity-Map | 83.9% | 14.9%
| Overall Average | 86.0% | 22.3%

Overview

We consider Target Reached %, Target Selected % and Moving Target Touched % as
measures of success within their own data-collection mode, for all of these
values relate to the ratio between successful and failed sub-tasks. Figure 5
displays the relative success rate for each input method. Input methods are
shown along the X-axis while the success rate is along the Y-axis. Vertical
bars represent the success rate for each input method with different colors
correlating to different data-collection modes. Horizontal lines represent the
average success values for each of the input methods. We calculate this value
by averaging Target Reached %, Target Selected %, and Moving Target Touched %
values. Each color represents each input method. Input methods without AI
support are represented by dashed lines, while AI-supported methods are
represented by dotted lines, to further increase readability. The diagram was
also limited to the [65-100] range on the Y-axis, to emphasize the
differences, that on a wider range would be not clear.

Figure 5: Graphical overview of "Success %" for each input-method and their
relative average values

##  5 CONCLUSIONS

Considering that mouse (and keyboard) are the main input peripherals used in
everyday life when controlling a personal computer, it was expected that the
input method will produce the best performance. Based on the results we can
conclude that using the Gravity-Map AI Support to aid alternative input
methods, such as face tracking, significantly improves the viability of the
method. The results show, that Head-tracking based input using the Gravity-Map
AI support approach performs the closest to the mouse-based input. Taking into
account that most of the participants only saw head tracking-based input
methods first time during the testing process, we can presume that increased
familiarity with the technology will lead to higher performance.

##  6 ACKNOWLEDGMENT

This project has received funding from the European Union’s Horizon Europe
Research and Innovation programme under grant agreement No. 101070181.

## References

  * [1] D. Parsons and K. MacCallum, “Current perspectives on augmented reality in medical education: Applications, affordances and limitations,” Advances in Medical Education and Practice, vol. 12, pp. 77–91, 2021.
  * [2] C. Dennler, D. E. Bauer, and A.-G. Scheibler, “Augmented reality in the operating room: a clinical feasibility study,” BMC Musculoskeletal Disorders, vol. 22, no. 1, p. 451, 2021.
  * [3] S. Hell and V. Argyriou, “Machine learning architectures to predict motion sickness using a virtual reality rollercoaster simulation tool,” in 2018 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR), pp. 153–156, 2018.
  * [4] V. Reljić, I. Milenković, S. Dudić, J. Šulc, and B. Bajči, “Augmented reality applications in industry 4.0 environment,” Applied Sciences, vol. 11, no. 12, 2021.
  * [5] X. Xu, Y. Lu, B. Vogel-Heuser, and L. Wang, “Industry 4.0 and industry 5.0—inception, conception and perception,” Journal of Manufacturing Systems, vol. 61, pp. 530–535, 2021.
  * [6] J. C. Kim, T. H. Laine, and C. Åhlund, “Multimodal interaction systems based on internet of things and augmented reality: A systematic literature review,” Applied Sciences, vol. 11, no. 4, 2021.
  * [7] M. Zilak, Z. Car, and I. Culjak, “A systematic literature review of handheld augmented reality solutions for people with disabilities,” Sensors, vol. 22, no. 20, 2022.
  * [8] Z. Rashid, J. Melià-Seguí, R. Pous, and E. Peig, “Using augmented reality and internet of things to improve accessibility of people with motor disabilities in the context of smart cities,” Future Generation Computer Systems, vol. 76, pp. 248–261, 2017.
  * [9] H. Kato, M. Billinghurst, I. Poupyrev, K. Imamoto, and K. Tachibana, “Virtual object manipulation on a table-top ar environment,” in Proceedings of the International Symposium on Augmented Reality (ISAR 2000), pp. 111–119, Oct. 2000.
  * [10] M. Billinghurst, H. Kato, and S. Myojin, “Advanced interaction techniques for augmented reality applications,” The Human Interface Technology New Zealand (HIT Lab NZ), vol. 1, 2001.
  * [11] K. Pfeuffer and et al., “ARtention: a design space for gaze-adaptive user interfaces in augmented reality,” Comput. Graph., vol. 95, pp. 1–12, 2021\.
  * [12] R. Vertegaal et al., “Attentive user interfaces,” Communications of the ACM, vol. 46, no. 3, pp. 30–33, 2003.
  * [13] R. Piening, K. Pfeuffer, A. Esteves, T. Mittermeier, S. Prange, P. Schröder, and F. Alt, “Looking for info: Evaluation of gaze based information retrieval in augmented reality,” in Human-Computer Interaction – INTERACT 2021, (Cham), pp. 544–565, Springer International Publishing, 2021.
  * [14] M. Whitlock, E. Harnner, J. R. Brubaker, S. Kane, and D. A. Szafir, “Interacting with distant objects in augmented reality,” in 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), pp. 42–48, IEEE, 2018\.
  * [15] F. Müller, J. McManus, S. Günther, M. Schmitz, M. Mühlhäuser, and M. Funk, “Mind the tap: Assessing foot-taps for interacting with head-mounted displays,” in Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, (New York, NY, USA), pp. 477:1–477:13, 2019.
  * [16] V. Bloom, D. Makris, and V. Argyriou, “Clustered spatio-temporal manifolds for online action recognition,” in 2014 22nd International Conference on Pattern Recognition, pp. 3963–3968, 2014.
  * [17] R. J. K. Jacob, Eye Tracking in Advanced Interface Design, pp. 258–290.  1995\.
  * [18] V. Argyriou, M. Petrou, and S. Barsky, “Photometric stereo with an arbitrary number of illuminants,” Computer Vision and Image Understanding, vol. 114, no. 8, pp. 887–900, 2010.
  * [19] V. Argyriou, “Performance study of gradient correlation for sub-pixel motion estimation in the frequency domain,” IEE Proceedings - Vision, Image and Signal Processing, vol. 152, pp. 107–114(7), February 2005.
  * [20] M. Bates, “Models of natural language understanding,” Proc. Natl. Acad. Sci. USA, vol. 92, no. 22, p. 9977, 1995.
  * [21] F. Manuri and G. Piumatti, “A preliminary study of a hybrid user interface for augmented reality applications,” IEEE, 8 2015.
  * [22] M. H. Ali, M. M. Jaber, S. K. Abd, A. Rehman, M. J. Awan, D. Vitkutė-Adžgauskienė, R. Damaševičius, and S. A. Bahaj, “Harris hawks sparse auto-encoder networks for automatic speech recognition system,” Applied Sciences, vol. 12, no. 3, 2022.
  * [23] M. Frutos-Pascual, C. Gale, C. Harrison, Jake M.and Creed, and I. Williams, “Character input in augmented reality: An evaluation of keyboard position and interaction visualisation for head-mounted displays,” in Human-Computer Interaction – INTERACT 2021, (Cham), pp. 480–501, Springer International Publishing, 2021.
  * [24] L. Chen, R. Balakrishnan, and T. Grossman, “Disambiguation techniques for freehand object manipulations in virtual reality,” in 2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), pp. 285–292, Mar. 2020.
  * [25] S. Mahmud, X. Lin, and J.-H. Kim, “Interface for human machine interaction for assistant devices: A review,” in 2020 10th Annual Computing and Communication Workshop and Conference (CCWC), pp. 0768–0773, 2020.
  * [26] H. Ho, “Low cost and better accuracy eye tracker,” in 2014 International Symposium on Next-Generation Electronics (ISNE), pp. 1–2, May 2014.
  * [27] M. Yu, X. Wang, Y. Lin, and X. Bai, “Gaze tracking system for teleoperation,” in The 26th Chinese Control and Decision Conference (2014 CCDC), pp. 4617–4622, May 2014.
  * [28] D. Gego, C. Carreto, and L. Figueiredo, “Teleoperation of a mobile robot based on eye-gaze tracking,” in 2017 12th Iberian Conference on Information Systems and Technologies (CISTI), (Lisbon), 2017.