* (16) A. Peres, Quantum Theory: Concepts and Methods (Kluwer, Dordrecht, 1993), p 161.
  * (17) A. Aspect, P. Grangier, and G. Roger, “Experimental realization of Einstein-Podolsky-Rosen-Bohm Gedankenexperiment: a new violation of Bell’s inequalities,” Phys. Rev. Lett., 49, 91, doi:10.1103/PhysRevLett.49.91 (1982).
  * (18) S. L. Altmann, “Hamilton, Rodrigues, and the quaternion scandal,” Mathematics Magazine, 62, 291 (1989).
  * (19) J. Christian, “Failure of Bell’s theorem and the local causality of entangled photons,” doi:10.48550/arXiv.1005.4932 (2010).
  * (20) G. Weihs et al., “Violation of Bell’s inequality under strict Einstein locality conditions,” Phys. Rev. Lett., 81, 5039, doi:10.1103/PhysRevLett.81.5039 (1998).
  * (21) A. Aspect, “Bell’s inequality test: more ideal than ever.” Nature 398, 189, doi:10.1038/18296 (1999).
  * (22) J. S. Bell, “Bertlmann’s socks and the nature of reality,” Journal de Physique, Colloque C2, Suppl. au numero 3, Tome 42 (1981). pp C2 41-61.
  * (23) D. Bohm, Quantum Theory (Prentice-Hall, Inc., Englewood Cliffs, New Jersey, 1951).
  * (24) Calculus with Analytic Geometry (Worth Publishers, Inc., New York, 1978).


# Recognition and Estimation of Human Finger Pointing with an RGB Camera for
Robot Directive

Eran Bamani, Eden Nissinman, Lisa Koenigsberg, Inbar Meir, Yoav Matalon and
Avishai Sintov All authors are with the School of Mechanical Engineering, Tel-
Aviv University, Israel. Corresponding author e-mail<EMAIL_ADDRESS>
###### Abstract

In communication between humans, gestures are often preferred or complementary
to verbal expression since the former offers better spatial referral. Finger
pointing gesture conveys vital information regarding some point of interest in
the environment. In human-robot interaction, a user can easily direct a robot
to a target location, for example, in search and rescue or factory assistance.
State-of-the-art approaches for visual pointing estimation often rely on depth
cameras, are limited to indoor environments and provide discrete predictions
between limited targets. In this paper, we explore the learning of models for
robots to understand pointing directives in various indoor and outdoor
environments solely based on a single RGB camera. A novel framework is
proposed which includes a designated model termed PointingNet. PointingNet
recognizes the occurrence of pointing followed by approximating the position
and direction of the index finger. The model relies on a novel segmentation
model for masking any lifted arm. While state-of-the-art human pose estimation
models provide poor pointing angle estimation accuracy of 28∘, PointingNet
exhibits mean accuracy of less than 2∘. With the pointing information, the
target is computed followed by planning and motion of the robot. The framework
is evaluated on two robotic systems yielding accurate target reaching.

##  I Introduction

Finger Finger pointing is a universal gesture where a human extends the arm
and index finger in direction of a desired target [1]. As such, it is a
prominent form of communication between humans in various cultures in order to
direct attention to some region of interest in the environment [2]. Tomasello
[3] argued that gestures, and pointing in particular, preceded vocal
communications as gestures provide better spatial referral then speech.
Pointing naturally and rapidly directs human attention to external targets
while speech does not. Infants, for instance, use pointing before they learn
to pronounce words [4]. Furthermore, in multi-cultural interaction, pointing
is the preferred form of communication to cope with the language barrier [5].
Consequently, pointing is an indispensable tool for conveying important
information. Pointing recognition in Human-Robot Interaction (HRI), therefore,
is a necessary form of communication for natural and easy directives to
robots.

Figure 1: A user directs a quadruped robot to a target position by pointing.
The robot observes the user through an RGB camera. PointingNet identifies a
pointing gesture and estimates its position and direction. Once the target has
been calculated, the robot plans motion and moves to the target.

HRI is a wide field of study that explores means of mutual interaction between
humans and robots. An interaction between a human and a robot might occur in
the form of information exchange, collaboration or guidance. In general,
gesture recognition is a popular non-verbal approach for HRI and has
historically been approached by conventional classification techniques
utilizing handcrafted features [6, 7, 8]. Nevertheless, these methods have
been found to be weak against the complexity of human motions. Consequently,
significant effort has been put in modeling and understanding intents of human
users through natural gestures [9]. By using natural human gestures, and
mainly pointing as done between humans, directive instructions to a robot can
easily be commanded while reducing the cognitive burden imposed on the user
[10]. The cognitive load on users can be reduced as they do not have to spend
much mental effort learning and remembering complex commands or navigating
through menus of digital interfaces. For instance, first responders can
instantly direct a robot to assist while performing life saving tasks.
Similarly, medical personnel can direct a robot to fetch tools or supplies.

Figure 2: Illustration of the proposed framework for a robot reach to a
pointed target. Pointing recognition acts as a trigger. That is, once pointing
of the user has been identified, the pointing position and direction are
estimated and used by the planner for planning the motion of the robot in the
instructed direction.

In order for a robot to understand the pointing gesture of a human, it must
first recognize the occurrence of pointing and then estimate direction.
Various approaches have been proposed for the estimation of pointing direction
while few addressed the recognition problem. Direction estimation methods can
be divided into two categories based on the sensing modality: wearable devices
and visual perception. Work with wearable devices for pointing estimation have
used Inertial Measurement Units (IMU) [11] and Electro-Myography (EMG) [12] to
sense motion and pose of the arm. However, these devices often require
constant carrying of the sensors along with micro-controllers on the user’s
body, which may be cumbersome. More importantly, they prevent spontaneous
pointing towards a robot and ability for occasional users to direct it.

As opposed to wearable sensing, visual perception allows spontaneous
observation of any frequent user with one or more cameras. Much work has been
done on pointing detection using multiple RGB cameras or a depth (stereo)
camera [13, 14, 15]. Nickel and Stiefelhagen [16], for instance, used a depth
camera along with a prior probabilistic body model in order to observe the
forearm and head, and identify direction of pointing gestures. While depth
data enables easier access to pointing gestures, depth cameras are limited to
short-range in indoor environments and work poorly outdoors. In addition, they
require an additional hardware setup and limit the generality of the method
[17]. However, only few addressed the pointing problem using a single RGB
camera [18]. The work in [19] used a simple monocular camera along with a
classifier to predict the target on an indoor floor from a set of discrete
points in the proximal surroundings of the user. Features from both the arm
and head were extracted and initiation of estimation was triggered using voice
commands. In [17], detection of whether the hand is pointing on objects was
performed. The detection was conducted with a camera set above the table in a
constructed environment. A Convex-Hull (CH) of the hand assisted in verifying
that exactly one finger is stretched out. However, the method was limited to
detection from a specific camera angle and other directions were not
demonstrated. Also, the work is restricted to a table top scenario with a
small viewing angle. These few methods that utilize only a single RGB camera
are demonstrated in a limited setting while considering rough estimation of
pointing direction toward large objects.

Many advances in HRI rely on human pose estimation models often termed
Skeleton models. These models use either depth or RGB cameras to find a set of
key-points that can represent the pose of the human body. Recent work used the
OpenPose library [20] to extract a 2D skeleton model of the user from a single
RGB camera [18]. Spatial pose of the user’s arm was extracted from the 2D
image by calibration of fixed distances of a specific human user. This makes
the method designated for only a single user while the pointing accuracy is
not clear. Then, the pointing direction is extracted analytically from the
detected arm joints and was used to direct a drone to one of several building
windows. Pointing direction estimation using prominent skeleton models (e.g.,
MediaPipe [21], VNect [22], OpenPose [23]) provide real-time 3D pose
estimation with a simple RGB camera. Results to be presented in this work show
that these models provide poor results that cannot be used for accurate
prediction of pointing direction. In addition, the key-points of the skeleton
are provided with respect to some root and not relative to the camera. Hence,
arm location relative to the robot equipped with a camera cannot be acquired.

In order for the robot to infer the target that the user is pointing to, the
robot must accurately estimate the position of the finger and its direction
with respect to the robot’s coordinate frame. The accuracy is even more
important in an environment with poor diversity [24]. For example, when
commanded through pointing to go to a specific window out of many in a
building, a search and rescue drone must accurately identify the correct one.
Slight variations in estimation may result in failure. Some approaches measure
pointing by the direction of the forearm [12], the index finger [25] or the
vector formed by the root of the nose and index finger [26]. However, no
thorough analysis was performed to determine the pointing measurement that
best reflects the intent of the user. Most approaches are based on forearm
direction estimation [27, 28]. Yet, the forearm and index finger are not
always co-linear while the index finger is considered more accurate in
pointing. Work that attempted to estimate the direction of the index finger
focused on a limited discrete setting and solved a classification problem to
distinguish between several objects on an indoor table top [25, 29]. In
conclusion and to the best of the authors knowledge, no work has addressed a
continuous pointing estimation problem based on observing the index finger by
an RGB camera while able to work indoor and outdoor, and to recognize the
occurrence of pointing.

In this paper, we investigate the ability to learn robust models for pointing
recognition and estimation in diverse environments. The aim is for a mobile
robot to recognize pointing, estimate target location and move to it (Figure
1). First, a thorough experimental analysis is provided in order to determine
the accuracy of the three pointing measurement approaches. Then, a full
framework for accurate pointing recognition and estimation is proposed solely
based on RGB images. The framework is illustrated in Figure 2. A novel model
termed PointingNet is proposed in which two main components are the pointing
recognition and estimation models. Both models are based on a unique
segmentation model aimed to mask any lifted arm in the scene. Then, the
recognition model classifies the segmented arm in order to determine whether
pointing is exhibited. If so, a novel estimation model is triggered to
approximate the position and direction of the index finger with respect to the
camera. The robot, in turn, computes the desired target and moves towards it.
To summarize, the contributions of this work are as follows:

  * •

A comparative analysis is presented for measurement approaches of pointing
direction considering the intended target of the user.

  * •

The framework includes a recognition model that identifies when the user is
pointing.

  * •

A novel pointing estimation model is proposed for estimating the position and
direction of the index finger during pointing. Evaluation over a diverse test
set shows highly accurate estimations.

  * •

A unique segmentation model is used in order to focus the recognition and
estimation on the pointing arm. The segmentation model outputs a mask for any
visibly lifted arm in the image.

  * •

Unlike prior work, the proposed models are based on a simple RGB camera and
can provide continuous estimations in various indoor and outdoor environments
based solely on observation of the user’s arm.

  * •

The proposed framework is demonstrated and evaluated on two mobile robots.

  * •

Trained model and labeled pointing dataset are available open-source for the
benefit of the community111To be available upon acceptance for publication.
Images will be modified in order to protect the privacy of the participants..

The method proposed in this work has potential applications in additional
areas such as virtual and augmented reality, interactive presentations and
directive reasoning of various other gestures. In addition, the approach can
be used in gaming and theme-parks to enable participants the interaction with
a game environment through natural gestures. In this work, we assume that the
camera is on the robot and, therefore, reference is with respect to the robot.
However, an alternative application can have a designated station with a
camera for the user to naturally direct or tele-operate a remote robot.

##  II Related Work

TABLE I: State-of-the-art comparison of pointing recognition methods | Recognition | Body | Skeleton | IF/FA/EF | Indoor | Outdoor | RGB/Depth/ |  Continuous/
---|---|---|---|---|---|---|---|---
Paper |  | posture |  |  |  |  | Wearable |  Discrete
Sikeridis et al. [11] | ✗ | ✗ | - | FA | - | - | Wearable | C
Haque et al. [12] | ✓ | ✗ | - | FA | - | - | Wearable | C
Nickel and Stiefelhagen [16] | ✓ | ✓ | ✗ | FA/EF | ✓ | ✗ | Depth | C
Jirak et al. [17] | ✓ | ✗ | ✗ | IF | ✓ | ✗ | RGB | D
Medeiros et al. [18] | ✗ | ✓ | ✓ | FA | ✗ | ✓ | RGB | D
Martin et al. [19] | ✗ | ✓ | ✗ | - | ✓ | ✗ | RGB | D
Hu et a. [25] | ✗ | ✗ | ✗ | IF | ✓ | ✗ | 2$\times$RGB | C
Azari et al. [26] | ✗ | ✓ | ✗ | EF | ✓ | ✗ | Depth | C
Lai et al. [27] | ✓ | ✓ | ✓ | FA | ✓ | ✗ | Depth | C
Kuramochi et al. [28] | ✗ | ✗ | ✓ | FA | ✓ | ✗ | 2$\times$RGB | C
Shukla et al. [29] | ✗ | ✗ | ✗ | IF | ✓ | ✗ | Depth | D
Tölgyessy et al. [30] | ✗ | ✗ | ✓ | FA | ✓ | ✗ | Depth | C
Pietroszek et al. [31] | ✗ | ✗ | - | FA | - | - | Wearable | C
Das [32] | ✗ | ✗ | ✗ | IF | ✓ | ✗ | Depth | C
Kehl and Van Gool [33] | ✓ | ✓ | ✗ | EF | ✓ | ✗ | 8$\times$RGB | C
Proposed method | ✓ | ✗ | ✗ | IF | ✓ | ✓ | RGB | C

Existing methods for point detection can be divided into three categories
based on the used sensing system: depth camera, RGB camera and wearable
device. The majority of work with visual perception are based on depth cameras
or multiple RGB ones from different pointing views. Early work used depth
camera images to estimate the pointing targets on a table [13]. Nickel and
Stiefelhagen [16] identified direction of pointing gestures through head and
hands by clustering human skin color based on [34]. The body model is also
used along with a Hidden Markov model to detect the occurrence of pointing.
Another method detected and approximated the pointing direction from two
orthogonal camera views [25]. Recently, two fish-eye cameras were installed on
both sides of a display and used to capture images of a wide area in front of
the display. With such setting, the system can recognize the user’s skeleton
and estimate pointing direction towards the display [28]. Similar work [35]
relied on contour detection along with CH of the hand while using close-up
Kinect-based images of hands. Hence, it is not clear whether the method is
applicable for real world scenarios.

Finger data is not easy to extract in an unconstructed environment due to its
small area it occupies in the image. Hence, some work observes alternative or
additional body features. The common approach is to determine pointing
direction based on the forearm posture [16, 28]. The work in [27, 30] used a
depth camera to identify the vector formed by the elbow and wrist extracted
from a skeleton model [36]. Intersection of the vector with the floor was then
computed in order to determine the desired goal location for a mobile robot.
Note that in [27], detection of a pointing gesture was performed by solely
observing whether the height of the arm is within some threshold relative to
the shoulder. A different approach detected the positions of the head and hand
with a depth camera, and considered the vector connecting them as the pointing
direction [26]. In the same work, the hand closest to the camera was
heuristically chosen as the pointing one. The work in [29], on the other hand,
proposed a method that solely observes the hand independent of body postures.
The work offered a model-free probabilistic method to identify the pointed
object from a known set. However, the work relied on a depth camera in a
limited setting of a single viewing angle above a tabletop.

While visual perception is the leading approach for pointing and gesture
recognition, a different approach worth mentioning is the use of wearable
devices. For instance, two Inertial Measurement Units (IMU) were positioned on
the forearm of a user during pointing on a presentation screen [11]. Often,
Electro-Myography (EMG) is used alongside an IMU. In EMG, electrical muscle
signals are sensed and mapped to limb movements [37]. While an IMU can
approximate motion and posture, EMG can also provide hand gesture recognition.
EMG along with an IMU was used for gesture recognition and detection of
forearm pointing direction [12]. Similarly, EMG and IMU were used on the
user’s forearm to control a drone through obstacles [38]. However, IMU is
prone to drift and requires frequent calibrations [17]. A different approach
used an off-the-shelf smartwatch to point and interact with a large display
[31]. As mentioned in the previous section, wearable devices require carrying
hardware on the body and prevent spontaneous or occasional pointing by
arbitrary users.

Table I summarizes the state-of-the-art on pointing detection and estimation.
The table sorts various characteristics of prior methods which include the
following. Recognition: whether the proposed method identifies the occurrence
of pointing; Body posture: whether the method requires some body posture
information in addition to the arm; Skeleton: whether the method utilizes some
skeleton library for human pose estimation; IF/FA/EF: whether the method
observes the direction of the Index Finger (IF), Forearm (FA) or Eyes-Finger
(EF) vector; Indoor/Outdoor: was the method demonstrated in indoor and outdoor
environments; RGB/Depth/Wearable: what is the sensing technology used for
pointing data acquisition; and, Continuous/Discrete: does the method provide
continuous pointing estimation or classifies from a discrete set of regions or
objects. The table shows high dependence of vision-based methods on depth
cameras and, therefore, the majority of the methods cannot be used outdoors.
In addition, forearm detection is the leading approach while observing the
index finger is less common, and with RGB camera in particular.

##  III Preliminary Pointing Analysis

As discussed above, prior pointing work considered the measurement of either
the forearm (FA) vector formed by the wrist and elbow points [16, 28], index
finger (IF) direction [17] or the direction of the vector connecting the user
eyes (root of nose) and finger (EF) [33, 16, 26]. Illustration of these
measurement types is seen in Figure 3. Yet and to the best of the authors
knowledge, a thorough experimental comparison between the three measurement
approaches has never been conducted.

Figure 3: Three approaches for measuring pointing direction: forearm vector
formed by the wrist and elbow points, index finger vector and the vector
connecting the user eyes and finger. Pointing error $e_{k}$ is calculated
according to the minimal distance between the target center point $\mathbf{g}$
and the calculated direction vector $\mathbf{v}_{k}$ passing through some key-
point $\mathbf{p}_{k}$ where $k=\\{\texttt{FA},\texttt{IF},\texttt{EF}\\}$.

Irrelevant of the hardware and any learning method to estimate the pointing
direction, we wish to examine how accurate is each of the measuring
approaches. Hence, an experiment was designed and conducted in which seven
participants were asked to point towards a pre-defined target. Several
reflective markers were positioned on designated eyeglasses, finger (Figure
7b) and forearm (wrist and elbow) of the user as well as on the target. Using
a motion capture system, the target position along with poses of user index
finger, forearm and root of the nose were captured during pointing to the
target. Users were instructed to point to the target in various positions with
respect to the target and various body poses such as standing, sitting down
and crouching. Also, the users were specifically asked to look at the target
during pointing. For each measuring approach, the pointing direction
$\mathbf{v}_{k}\in\mathbb{R}^{3}$ and a key-point
$\mathbf{p}_{k}\in\mathbb{R}^{3}$ were calculated where
$k=\\{\texttt{FA},\texttt{IF},\texttt{EF}\\}$. The key-point is the position
of the pointing finger for IF and EF approaches, or wrist for FA approach.
Then, the pointing error $e_{k}$ is the minimal distance between a line in
direction $\mathbf{v}_{k}$ passing through $\mathbf{p}_{k}$ and the target
center point $\mathbf{g}\in\mathbb{R}^{3}$, given by

$e_{k}=\frac{\|(\mathbf{g}-\mathbf{p}_{k})\times\mathbf{v}_{k}\|}{\|\mathbf{v}_{k}\|}.$
(1)

Note that the experiment does not consider gaze but solely the position of the
root of the nose as commonly done in pointing recognition work. Measuring gaze
usually requires a camera fixed on the head of the user tracking pupil
directions.

Figure 4: Pointing accuracy with regards to the distance of the user from the
pointed target for three measurement approaches.

The comparative results of the pointing experiment for the three measuring
approaches are reported in Figure 4. The figure presents the mean and standard
deviation pointing accuracies with respect to the distance from the target.
The overall mean errors are 491.9 mm, 333.3 mm and 157.8 mm for the FA, IF and
EF, respectively. It is clear that FA measurement is the least accurate since
some deviation angle at the wrist often exist. In addition, the results show
that EF measurement is the most accurate while IF measurement has a somewhat
larger mean error. While the EF approach is seen to be the most accurate, it
exhibits several disadvantages. First, an average of 51% of individuals are
more likely to exhibit congruent gaze with the target while pointing [39]. The
degree of variability between individuals is high and some may settle for an
implicit gaze with no observable head shift. In addition, turning the head and
gazing towards the target is performed for a short period of time of
approximately 670 milliseconds [40]. This is even more significant during
cognitive overload where the user is performing various tasks such as during
driving. Capturing the short time instant of user head-turn during pointing
can be complex if it indeed occurs. Finally, having a pointing recognition
model that depends both on the detection of finger and head limits the
efficiency of the method in occluded environments where both must be visible.
Therefore, the work in this paper is focused on the recognition of pointing
solely by observing and labeling the position and direction of the index
finger. Hence, the method is more robust to occlusions and user out-of-frame
scenarios.

##  IV Methods

###  IV-A Problem Formulation

We consider a scenario where a user is standing in front of a mobile robot. It
is assumed that the user is within the image frame of a standard RGB camera
mounted on the robot. If the user is not visible, spatial voice localization
can be used, for example, to attract the attention of the robot [41]. However,
such a problem is not in the scope of this work. Once the user exhibits a
pointing gesture, the robot is required to recognize its occurrence, estimate
direction, and move towards the pointed target. A scheme for this is
illustrated in Figure 2. It is important to note that pointing, in this work,
is the extension of the arm along with the index finger while flexing the
other fingers into the palm. While they can be included in future efforts, we
do not consider other approaches such as using more than one finger or non-
manual gestures such as head movements [1].

To achieve the above goal and provide a pointing framework, we address two
problems: 1) Derive a recognition model where the occurrence of a pointing
gesture with the index finger is identified; and, 2) derive an estimation
model for the pointing direction along with the position of the pointing
finger. Images are continuously perceived by the robot in real-time. In
parallel, a recognition model $h(I_{t})\in\\{0,1\\}$ observes image $I_{t}$ at
time $t$ for whether pointing is visible $h(I_{t})=1$ or not $h(I_{t})=0$. The
recognition model acts as a trigger for direction estimation and motion to the
target. The estimation model $\Gamma(I_{t})$, in turn, is a regression problem
aimed to map image $I_{t}$ to the following pointing parameters: index finger
position $\mathbf{p}=(x_{p},y_{p},z_{p})\in\mathbb{R}^{3}$ and pointing
direction described by the pitch $\beta\in\mathbb{R}$ and yaw
$\gamma\in\mathbb{R}$ angles. Pointing is, therefore, described by the feature
vector $\mathbf{v}=(x_{p},y_{p},z_{p},\beta,\gamma)$. These pointing
parameters are defined relative to the coordinate frame $\mathcal{O}_{c}$ of
the camera as seen in Figure 5. Yaw angle $\gamma$ is the rotation about the
$z_{c}$ axis of $\mathcal{O}_{c}$. Pitch $\beta$, on the other hand, is the
angle of the finger axis with the $x_{c}-y_{c}$ plane.

Without loss of generality, we consider the problem of reaching the target
goal. That is, the estimated pointing direction along with the position of the
finger enable the robot to pinpoint the target location as illustrated in
Figure 1. It is assumed that the user is pointing with only one arm, either
right or left, while the second can do any other task. Motion planning will
then be used to reach the target. In the general case and not in the scope of
this work, some additional context may be required for the robot to reason
about the target. For example, the user may instruct the robot to reach a door
or a window. Similarly, the user can command the robot to bring a specific
object. In both cases, the user will point in the corresponding direction and
the robot will attempt to identify the target in the estimated direction.

Figure 5: Illustration of the position $\mathbf{p}$ and direction
$\hat{\mathbf{x}}$ definitions of a pointing finger with respect to the
coordinate frame of the camera $\mathcal{O}_{c}$. Direction can also be
represented by angles $\beta$ and $\gamma$.  Figure 6: Flowchart of the
proposed PointingNet model for recognizing the event of pointing, and
estimating finger position and pointing direction. A segmentation model termed
PointingNet-S is a main component in which a mask of a lifted arm is
outputted. Then, the cropped arm image is passed through a CNN for
classification of whether pointing is visible. If pointing is indeed
identified, estimation model PointingNet-E is triggered where its input is the
concatenation of the original image, the masked image and an estimated depth
image generated by MiDaS. The output of the estimation model is the position
of the finger and direction of pointing, both with respect to the coordinate
frame of the camera.

###  IV-B Overview of Approach

In order to address the above objectives, we present the PointingNet model
seen in Figure 6. PointingNet includes three components: PointingNet-S model
for arm segmentation, recognition model for identifying the occurrence of
pointing and PointingNet-E model for estimating feature vector $\mathbf{v}$. A
user, in most cases, will be pointing in an unstructured environment with many
diverse objects in the background. Hence, these objects may obscure the
pointing arm and, thus, make the distinction between foreground and background
objects challenging. This may be even worse as the user is farther from the
camera and more objects are captured in the frame. Therefore, segmentation is
the first step in order to focus the recognition and estimation models on the
arm and minimize background noise. We propose to train a segmentation model
PointingNet-S for automated masking of any lifted human arms in the frame,
either pointing or not.

The segmentation is further used for pointing recognition and feature vector
estimation. Segmentation can improve recognition by providing a clear and
focused representation of the objects or regions of interest within an image.
This allows a classifier to concentrate on the most relevant features and
ignore background or irrelevant information, leading to better accuracy and
reduced over-fitting. Hence, segmentation is used to crop the region of the
arm and provide an input to a classification network. We note that, in
preliminary study, a segmentation model was tested for only pointing arms
which would obviate the need for the recognition model. However, the model
provided poor results with many false negatives. Hence, a segmentation for any
lifted arm followed by a recognition model provides best results.

The segmented mask is also used, along with a depth estimation of the scene,
as input to the PointingNet-E model for estimating feature vector
$\mathbf{v}$. We next describe the collection process of three datasets
followed by a description of the segmentation, recognition and estimation
models. Finally, we describe the mapping of the estimated information to robot
motion to the pointed target.

###  IV-C Data Collection

To achieve the above goal, three datasets are collected for training the
models. All datasets are collected by using a simple RGB camera. The first
dataset $\mathcal{S}$ is collected for training model PointingNet-E . A set of
$N_{s}$ images is collected where either the right or left arm is put in some
arbitrary pose not necessarily while pointing. Consequently, the dataset is in
the form $\mathcal{S}=\\{I_{1},\ldots,I_{N_{s}}\\}$ where $I_{k}$ is an image.
Similarly, dataset $\mathcal{H}$ is collected for training the recognition
model having $N_{h}$ images. Each image $I_{i}$ is labeled with
$l_{i}\in\\{0,1\\}$ for whether pointing is visible ($l_{i}=1$) or not
($l_{i}=0$). Examples of labeled images are seen in Figure 7a. Thus, the
resulting data is the set
$\mathcal{H}=\\{(I_{1},l_{1}),\ldots,(I_{H_{h}},l_{N_{h}})\\}$.

|
---|---
(a) | (b)
Figure 7: Data collection for (a) recognition model by labeling images with
pointing (left) or not pointing (right), and (b) estimation model by labeling
images with a motion capture system observing a finger marker relative to a
camera marker.

The third dataset $\mathcal{P}$ includes data for training the PointingNet-E
model. While the previous two datasets are collected in various indoor and
outdoor environments, the third dataset requires a Motion Capture (MoCap)
System and, therefore, collected only indoors. Nevertheless, segmentation of
the arm aims to allow the model to also perform well outdoors. Each image
$I_{j}$ in $\mathcal{P}$ is taken by the RGB camera and labeled using the
MoCap with feature vector $\mathbf{v}_{j}$. Vector $\mathbf{v}_{j}$ is
obtained using a finger marker (with reflective markers as seen in Figure 7b)
measured by the MoCap relative to a base marker on the RGB camera. The
acquisition and labeling process yield dataset
$\mathcal{P}=\\{(I_{1},\mathbf{v}_{1}),\ldots,(I_{N_{p}},\mathbf{v}_{N_{p}})\\}$
of $N_{p}$ samples.

In all training datasets, data is collected by several users in various
distances of the user from the camera in $0.5-5$ meters range, and in
different locations within the image frame. Furthermore, we consider pointing
with yaw angles in the range $\gamma\in[-125^{\circ},125^{\circ}]$. Angles not
in the range will most likely not be visible to the robot and some
intermediate pointing would have to be done to change its perspective. In
order to generate a robust model, some images exhibit pointing where the user
is occluded and most of the body is not visible except the pointing arm.
Occlusion can occur by either an obstacle (example at the bottom of Figure 7a)
or having a large portion of the user out of the image frame. In addition,
pointing is performed while arbitrarily switching between right and left arms.

All images in the training dataset go through image augmentation. In addition
to increasing the dataset, the augmentation encourages the model to recognize
pointing in diverse and noisy environments, and also given partial information
[42]. Hence, the model will be able to perform well in changing environments
which include lighting variations and occlusions. Applied augmentation
techniques include color jitter, random Cutout and image blur. Augmentation
examples of images in the training set are seen in Figure 8.

|
---|---
(a) | (b)
|
(c) | (d)
Figure 8: Augmentation examples of images in the training data: (a) original
image, (b) random cut-out from image (black rectangle on hand), (c) image blur
and (d) color jitter.

###  IV-D Pointing Segmentation

A segmentation model breaks down an image into several segments of interests.
Such process enables reducing the complexity of the problem and assists in
focusing on semantically meaningful regions for further image processing. In
this part, we build a segmentation model that can segment the user’s arm prior
to identifying if pointing occurs. For that purpose, all images in dataset
$\mathcal{S}$ are pre-processed and labeled by masking the corresponding user
arm. The entire visible arm is masked including the hand and fingers. Hence,
the pre-processed dataset is now
$\tilde{\mathcal{S}}=\\{(I_{k},s_{k})\\}_{k=1}^{N_{s}}$ where $s_{k}$ is the
set of masked pixels in image $I_{k}$.

Using the collected data, a segmentation model termed PointingNet-S is
proposed. First, a pre-trained Wide Residual Network (WideResNet) [43] is used
with 101 layers to encode the images. WideResNet increases the width of the
network, allowing it to capture complex features and patterns in the data.
After encoding the image, the latent space representation is reshaped by a set
of four convolutional operations, each having a different kernel. The first
two kernels are odd-sized kernels of $11\times 1$ and $1\times 11$. An odd-
sized kernel enables centering over the input data during convolution while
having rotational symmetry. The symmetry can simplify the computation and make
the network more robust to rotational variations in the input data. Two
additional convolutions are Atrous with a rate of two and three. The Atrous
convolution dilates the window size while incorporating gaps between data
values so to increase the field of view without increasing the number of
weights. Finally, the four convolutions are concatenated and fed into a
standard decoder comprising 48 convolutional layers. In addition, skip
connections are added between the encoder and decoder and between the
convolutions to avoid overfitting, improve gradient flow and encourage the
reuse of features. The model is trained with dataset $\tilde{\mathcal{S}}$ and
a Binary Cross-Entropy (BCE) with logits loss. The segmentation model is
illustrated in Figure 6. In conclusion, we acquire a segmentation model
$\hat{s}_{i}=S(I_{i})$ that outputs the predicted set of pixels $\hat{s}_{i}$
masking any lifted arm in image $I_{i}$.

###  IV-E Pointing Recognition

Object recognition refers to techniques or algorithms for recognizing objects
within images. Neural-networks, which are trained to recognize patterns in
data, are efficient for such tasks and include techniques of classification,
detection and segmentation [44, 45]. In our pointing recognition problem, the
task involves identifying whether the user seen in a query image exhibits a
pointing gesture. In order to eliminate background noise and interferences,
the PointingNet-S model described above is used to detect any lifted arm in
the image, crop out the rest of the image and proceed to further
classification. Therefore, dataset $\mathcal{H}$ is pre-processed to include
only cropped images of lifted arms. That is, each image $I_{i}\in\mathcal{H}$
is passed through the segmentation model and cropped to the minimal area
rectangle bounding predicted label $\hat{s}_{i}=S(I_{i})$. The pre-processed
dataset $\tilde{\mathcal{H}}=\\{(\tilde{I}_{i},l_{i})\\}_{i=1}^{N_{h}}$ is
acquired where $\tilde{I}_{i}$ is the cropped image of $I_{i}$.

Using $\tilde{\mathcal{H}}$, a recognition model $h(I_{i})$ is trained for
binary classification of images. Specifically, the model would determine
whether an arm seen in an inputted cropped image exhibits pointing or not.
Various classification models can be used to solve the pointing recognition
task. Comparative analysis to be presented in the evaluation section show that
marginal performance differences exist between a simple NN models (e.g.,
Fully-Connected NN or Convolutional-NN) to more complex pre-trained models
(e.g., GoogLeNet [45]). In addition, the former provides lower-computational
costs and more suitable for real-time predictions. Therefore, we propose the
use of a Convolutional NN (CNN) architecture with four convolutional layers.
The output of the final convolutional layer is flattened and fed into a series
of eight fully-connected (FC) layers used for binary classification. The
output of the final FC layer is passed through a Sigmoid activation function
in order to obtain a predicted probability for whether the input image
exhibits pointing.

###  IV-F Pointing Estimation

Once a pointing gesture has been recognized, the pointing feature vector
$\mathbf{v}$ is to be estimated. As discussed in Section I, common approaches
for human pose estimation are based on learning keypoints on the human body
forming a skeletal representation (e.g., MediaPipe [46] and VNect [22]). As
will be shown in the Evaluation Section, computing either forearm [18, 27] or
index finger direction from these keypoints results in poor estimation
accuracy. In addition, these keypoints are measured with respect to some
arbitrary coordinate frame and not relative to the camera. Hence, state-of-
the-art human pose estimation approaches cannot be used to extract the
position of the pointing finger.

The prominent disadvantage of using regular RGB images is the lack of spatial
information and, in particular, the user’s 3D pose relative to the camera.
Therefore, we incorporate the MiDaS Monocular Depth Estimation model [47, 48]
which adds depth information to the observed RBG image. MiDaS uses a single
monocular camera to estimate the depth of objects in the scene with respect to
the camera. Hence, each image $I_{j}\in\mathcal{P}$ is passed through the
MiDaS model to generate a corresponding RGB depth image $D_{j}$. Furthermore,
arm segmentation can focus the model on regions of interest. Hence, the
predicted mask $\hat{s}_{j}=S(I_{j})$ is used to generate a binary image
$B_{j}$ (i.e., white and black for arm and non-arm regions, respectively).
Finally, the images $I_{j}$, $D_{j}$ and $B_{j}$ are concatenated to a seven-
channel image $C_{j}$. This configuration enables to observe spatial
relations, to omit irrelevant features and to lower the required amount of
data for training the pointing estimation model. While the full arm is
segmented, the label for training PointingNet-E is the direction of the index
finger. The index finger is included in the segmented arm. Also, the model
receives additional information in the image, both in $I_{j}$ and $D_{j}$,
including the standing pose of the human body with approximated depth
information. Therefore, the model can learn dependencies of body and arm
postures to the direction of index finger.

The proposed PointingNet-E for pointing estimation encodes the multi-channel
image $C_{j}$ to acquire an estimated feature vector
$\tilde{\mathbf{v}}_{j}=\Gamma(C_{j})$. PointingNet-E is based on an eight-
layers ConvNeXt encoder [49]. The key innovation of ConvNeXt is its use of
channel groups within the network for efficient capturing of diverse features
in input images. By combining different types of convolutions within a block,
ConvNeXt can capture both low-level and high-level features while maintaining
a low computational cost. We modify the ConvNeXt to include skip-connections.
That is, the output of layer $i$ is passed to layer $i+1$ and also
concatenated with the output of layer $i+2$ after down-sampling. These
multiple pathways provide a form of redundancy that enables advanced
information sharing and makes the network more robust to noisy or incomplete
data.

The output of the ConvNeXt encoder is further passed through three different
Fully-Connected Neural-Network (FC-NN) blocks. Preliminary results have shown
that it is significantly more accurate to estimate each feature in pointing
vector $\mathbf{v}_{i}$ with a designated FC-NN rather than with a single FC-
NN for all features. Hence, each of the three FC-NN blocks outputs either
angle $\beta_{i}$, angle $\gamma_{i}$ or position $\mathbf{p}_{i}$ of pointing
feature vector $\mathbf{v}_{i}$. All FC-NN’s consist of four FC layers having
Rectifier Linear Unit (ReLU) activation functions in between. However, only
the FC-NN’s of the angles have a Sigmoid activation function at the output.
The model is trained to minimize a loss function summing the Root Mean Square
Error (RMSE) and Mean Squared Error (MSE) of the angles and position,
respectively, over dataset $\mathcal{P}$. In conclusion, given an image
$I_{j}$ with exhibited pointing, depth estimation $D_{j}$ and binary arm mask
$B_{j}$ are extracted so to estimate the feature vector
$\tilde{\mathbf{v}}_{j}=\Gamma(C_{j})$.

###  IV-G Robot motion

In the idling state, the robot will loop through a stream of real-time images
acquired from the camera. The images will pass through the pointing
recognition model $h$. Once a pointing gesture is identified, the estimation
model $\Gamma$ is triggered to output an estimated feature vector
$\tilde{\mathbf{v}}$. Hence, the estimated position of the finger
$\tilde{\mathbf{p}}^{(c)}$ and direction angles $\tilde{\beta},\tilde{\gamma}$
are now known. The superscript $(\cdot)^{(c)}$ indicates coordinates with
respect to the frame $\mathcal{O}_{c}$ of the camera. The unit vector
$\hat{\mathbf{x}}^{(c)}$ in the direction of pointing is acquired by

$\hat{\mathbf{x}}^{(c)}=\begin{pmatrix}\cos\tilde{\mathbf{\gamma}}\cos\tilde{\mathbf{\beta}}\\\
\sin\tilde{\mathbf{\gamma}}\cos\tilde{\mathbf{\beta}}\\\
-sin\tilde{\mathbf{\beta}}\end{pmatrix}.$ (2)

Given the homogeneous transformation matrix $A_{r,c}\in SE(3)$ mapping from
$\mathcal{O}_{c}$ to the robot coordinate frame $\mathcal{O}_{r}$, finger
position and pointing direction are expressed in $\mathcal{O}_{r}$ according
to

$\begin{pmatrix}\hat{\mathbf{x}}^{(r)}\\\
1\end{pmatrix}=A_{r,c}\begin{pmatrix}\hat{\mathbf{x}}^{(c)}\\\ 1\end{pmatrix}$
(3)

and

$\begin{pmatrix}\tilde{\mathbf{p}}^{(r)}\\\
1\end{pmatrix}=A_{r,c}\begin{pmatrix}\tilde{\mathbf{p}}^{(c)}\\\
1\end{pmatrix}.$ (4)

Furthermore, it is assumed that the environment of the robot is fully known
such that distance $d$ from the finger to the floor plane along vector
$\hat{\mathbf{x}}^{(r)}$ can be estimated. For instance, if both user and
robot are on a flat horizontal floor, distance $d$ can be calculated according
to simple proportionality

$d=-\frac{\tilde{p}_{z}^{(r)}+h}{\hat{x}_{z}^{(r)}}$ (5)

where $\tilde{p}_{z}^{(r)}$ and $\hat{x}_{z}^{(r)}$ are the $z$ components of
$\tilde{\mathbf{p}}^{(c)}$ and $\hat{\mathbf{x}}^{(r)}$, respectively, and $h$
is the height of the robot. Generally, the target location $\mathbf{g}$ in
$\mathcal{O}_{r}$ for the robot to reach is acquired according to the
resultant vector

$\mathbf{g}=\tilde{\mathbf{p}}^{(r)}+\hat{\mathbf{x}}^{(r)}d$ (6)

as seen in Figure 9. Once the target has been identified, the robot can walk
directly to it if the environment is known to be obstacle-free. Alternatively,
the robot can apply some motion planning algorithm [50] to plan a more complex
path considering obstacles in the environment.

Figure 9: Once the target $\mathbf{g}$ is computed, the robot can plan and
control the motion to it (dashed curve). All vectors are expressed in the
coordinate frame of the robot $\mathcal{O}_{r}$ after mapping from frame
$\mathcal{O}_{c}$.

##  V Model Evaluation

This section tests and analyzes the proposed PointingNet model including
analysis of arm segmentation, recognition model for classifying occurrences of
pointing and direction estimation. We describe the collection of training and
test data followed by an analysis of the model components. All model training
and evaluations were performed on a Linux Ubuntu (18.04 LTS) machine with
Intel Xeon Gold 6230R (20 Cores of 2.1GHz) CPU and four NVIDIA GeForce RTX
2080TI GPU (11GB of RAM). Neural-Network models were trained and tested using
PyTorch. Videos of the data collection, experiments and demonstrations can be
seen in the supplementary material.

###  V-A Database and Descriptors

As discussed in Section IV-C, three datasets $\mathcal{S}$, $\mathcal{H}$ and
$\mathcal{P}$ were collected for arm segmentation, pointing recognition and
pointing direction estimation, respectively. For the collection of these
datasets, 12 participants were recruited where nine and three participants
contributed training and test data, respectively. Participants and recorded
environments in the test set were not included in the collection of training
data. Participants include four females and eight males with maximum and
minimum heights of 1.56 m and 1.91 m, respectively. All image data was
collected with a simple web-camera yielding images of size $480\times 640$.
However, in order to enable robust usage with any camera, the images were
reduced to resolution $288\times 384$. To enhance the labeling process of the
information, the acquired data for segmentation was obtained at a resolution
of $1920\times 1080$ and later reduced. This allows for a more comprehensive
visualization of objects or regions of interest and facilitates the
identification and annotation of such elements.

Both datasets $\mathcal{S}$ and $\mathcal{H}$ were collected by the
participants in indoor and outdoor environments. Participants stood in various
locations in-front of the camera and up to five meters from it. During
recording, no other participant was visible in the image. Each participant was
asked to randomly point at any desired direction with a sole constraint of
pointing within the range of $\gamma\in[-125^{\circ},125^{\circ}]$.
Participants arbitrarily switched between their right and left arm. Also,
participants arbitrarily chose to have short or long sleeves during recording.
The collection yielded $N_{s}=17,698$ and $1,800$ train and test samples,
respectively, for dataset $\mathcal{S}$. In addition to many images of
pointing, the dataset includes images of lifted arms in various tasks such as
waving or holding an object. To allow for robust segmentation models, some
images included scenarios without any participant or with features resembling
a human arm (e.g., unused coat or shirt). These images were given a blank mask
(i.e., no segmentation). To generate the masked dataset $\tilde{\mathcal{S}}$,
images in $\mathcal{S}$ were pre-processed and labeled by masking the
corresponding user arm with the V7 auto-annotate tool.

Dataset $\mathcal{H}$ for classification includes $N_{h}=20,000$ training
samples and $2,000$ test samples. Half of $\mathcal{H}$ is of participants
exhibiting pointing in the images and labeled $l_{i}=1$. Images of the other
half, labeled $l_{i}=0$, include participants doing various tasks while
standing, walking, sitting or crouching but without pointing. The third data
set $\mathcal{P}$ includes $N_{p}=186,300$ and $17,458$ train and test
samples, respectively, of images labeled with pointing feature vectors.
Feature vector $\mathbf{v}_{i}$ was measured by a MoCap system with eight
OptiTrack Prime 41 cameras as discussed in Section IV-C and demonstrated in
Figure 7b. Since a MoCap was used, the collection was only performed indoors.

###  V-B PointingNet-S evaluation

As discussed in Section IV-D, segmentation focuses the recognition and
estimation models on the user’s arm. PointingNet-S is a designated model for
segmentation of lifted arms in images. We compare PointingNet-S to various
state-of-the-art segmentation models including Autoencoder, U-Net, U-Net++,
Feature Pyramid Network (FPN), ResNet-101, DenseNet and DeepLabV3+. The
baseline is a simple Autoencoder in which the original image is encoded and
reconstructed to the masked variant [51]. U-Net is a widely used image
segmentation model for biomedical applications where the image is contracted
and expanded in a U shaped path of convolutional layers [52]. U-Net++ is an
extension of the U-Net architecture with additional skip-connections to
improve segmentation accuracy [53]. Similarly, FPN uses a pyramid-shaped
feature hierarchy to extract features at different scales for object
segmentation [54]. ResNet-101 [55] and DenseNet-201 [56] are deep learning
models that use residual connections and densely connected blocks,
respectively, to improve accuracy and reduce overfitting. DeepLabV3+ model
uses dilated convolution and an encoder-decoder structure with a ResNet-101
backbone to preserve fine details [57]. These models are commonly used to
identify and segment various objects in images in different applications. They
provide different trade-offs between accuracy and computational efficiency.
Hence, the comparison is important since the performance of the models vary
based on the specific segmentation task and dataset.

All evaluated models are trained with dataset $\tilde{\mathcal{S}}$ and
evaluated on the independent test set. The performances of all models are
evaluated with the Intersection over Union (IoU). IoU calculates the
percentage ratio between overlapping and unified areas of the ground-truth and
predicted masks. Table II reports the comparative IoU results between all
methods. While some models achieved adequate accuracy, PointingNet-S is shown
to provide superior results by a large margin. Figure 10 shows examples of
predicted segmentation of the arms in various scenarios. PointingNet-S is able
to segment lifted arms in complex scenarios including multiple users in the
frame, occlusions and user pointing while body is out-of-frame.

TABLE II: Arm Segmentation Evaluation Results Models |  | IoU
---|---|---
Autoencoder |  | 66.9%
U-Net |  | 60.1%
U-Net++ |  | 86.6%
FPN |  | 69.2%
ResNet-101 |  | 78.5%
DenseNet-201 |  | 82.9%
DeepLabV3+ |  | 86.1%
PointingNet-S |  | 94.3%
Figure 10: Arm segmentation examples outputted by PointingNet-S in various
scenarios of multiple participants, occlusions, dual-arm lift, sitting down
and holding objects. The segmentation is performed for any lifted arm
regardless of pointing or not. Classification later determines if pointing
indeed occurs.

###  V-C Pointing Recognition Evaluation

Once segmentation of the arm has been acquired, the image is cropped to
include only the arm. As discussed in Section IV-E, the cropped image is
passed through the classification model in order to determine if pointing
occurs. In our proposed approach, a CNN network is trained to perform the
classification. We compare the model with various other models including FC-
NN, ResNet-18, Visual Geometry Group (VGG) and GoogLeNet. VGG is a popular
model for object recognition and segmentation that uses a deep convolutional
architecture [58]. Similarly, GoogLeNet is a deep CNN having a unique
inception module that incorporates multiple filters in parallel [45]. In
addition, we compare classification with and without segmentation of the arm.
When not including segmentation, the entire image is passed through the model.

Table III reports classification success rate for all models with and without
segmentation. First, not using segmentation exhibits poor results and
validates the need to focus the model on the arm. In addition, the results
with segmentation show marginal differences between simple models, such as FC-
NN and CNN, to the more complex models with advantage to the latter. However,
the marginal improvement may not justify the significant increase in
computational cost of runtime and memory usage. Hence, we use the proposed CNN
for further experiments while more complex models are optional.

Figure 11 reports the classification success rate and certainty with regards
to the users distance from the camera. The success rate and model certainty
for correct predictions are stable and high. However, certainty for incorrect
predictions is low. Hence, one can identify and ignore erroneous ones.

TABLE III: Pointing Recognition Evaluation Results Models |  | Segmentation
---|---|---
| w/o |  | w/
FC-NN |  | - |  | 96.6%
CNN |  | 66.7% |  | 97.2%
ResNet-18 |  | 74.6% |  | 98.1%
VGG-19 |  | 75.1% |  | 98.5%
GoogLeNet |  | 78.9% |  | 98.7%
---
Figure 11: Classification (top) success rate and (bottom) certainty with
regards to the distance of the user’s arm from the camera.

###  V-D PointingNet-E Evaluation

Once pointing is identified, PointingNet-E is triggered. In this section, we
evaluate the performance of the model on the test data and in real-time
predictions. PointingNet-E model was trained with dataset $\mathcal{P}$ as
discussed in Section IV-F. Accuracy of the model over the test data is
evaluated using Root Mean Square Error (RMSE) for the finger position and Mean
Absolute Error (MAE) for yaw and pitch angles, all with respect to the
measured ground-truth.

TABLE IV: Pointing direction estimation results |  | Position | Yaw | Pitch
---|---|---|---|---
|  | RMSE (mm) | MAE (∘) | MAE (∘)
MediaPipe | (index finger) | - | 26.1$\pm$12.2 | 17.3$\pm$16.0
(forearm) | - | 29.6$\pm$10.7 | 15.8$\pm$12.5
VNect | (forearm) | - | 46.8$\pm$26.3 | 30.5$\pm$17.4
OpenPose | (forearm) | - | 33.7$\pm$26.3 | 16.6$\pm$10.6
CNN |  | 241.1$\pm$8.1 | 41.2$\pm$8.3 | 44.3$\pm$21.2
VGG-19 |  | 95.4$\pm$4.7 | 29.1$\pm$7.8 | 40.4$\pm$19.7
DenseNet |  | 162.8$\pm$79.4 | 21.9$\pm$10.1 | 13.5$\pm$4.5
PointingNet-E |  $I_{i}$ only | 216.5$\pm$55.1 | 38.2$\pm$6.7 | 17.9$\pm$7.7
PointingNet-E |  $I_{i}$ & $D_{i}$ | 94.3$\pm$4.18 | 7.17$\pm$6.1 | 3.43$\pm$2.4
PointingNet-E |  $I_{i}$ & $B_{i}$ | 77.1$\pm$84.7 | 8.4$\pm$3.6 | 7.8$\pm$6.3
PointingNet-E | $C_{i}$ | 61.3$\pm$30 | 1.4$\pm$1.3 | 0.61$\pm$0.63

Table IV presents an overall comparison of the proposed approach with various
known models. First, we evaluate the pointing accuracy with state-of-the-art
skeleton-based human pose estimation models including MediaPipe [21], VNect
[22] and OpenPose [20]. These models are pre-trained and used as provided
through open-source distribution. While VNect and OpenPose offer spatial pose
estimation of only the forearm, MediaPipe can provide spatial estimation of
the finger as well. Hence, pointing is evaluated solely using forearm
keypoints for the former two while using also the index finger for the latter.
Human pose estimation models are not trained nor designated for the pointing
estimation problem. Therefore, all results with these models yield poor
direction accuracy. We note also that the position of the finger is given with
respect to some unknown coordinate frame and cannot be transferred to the
camera’s coordinate frame as required. Hence, skeleton-based human pose
estimation models are shown to be inadequate for pointing applications and
cannot be used in practice for directing robots.

The proposed model is next compared to various deep NN models including a
simple CNN, VGG-19 and DenseNet. The input for these models is only the
original image $I_{i}$. Furthermore, PointingNet-E is benchmarked across
different inputs: only the original image $I_{i}$; a concatenation of $I_{i}$
with the MiDaS depth approximation $D_{i}$; a concatenation of $I_{i}$ with
arm mask $B_{i}$; and, the concatenation $C_{i}$ of all three $I_{i}$, $B_{i}$
and $D_{i}$. The results in Table IV show poor accuracy for CNN, VGG-19 and
DenseNet as well as for PointingNet with only $I_{i}$, both in position and
direction. However, including $B_{i}$ or $D_{i}$ to $I_{i}$ significantly
reduces estimation errors. These results, therefore, highlight the importance
of each of the three components in predicting the position and direction of
pointing gestures. Consequently, having all three images $I_{i}$, $B_{i}$ and
$D_{i}$ provides significantly lower position and direction errors (less than
2∘ in average). Figure 12 shows PointingNet-E estimation accuracy with regards
to distance of the user from the camera. Results show slight error increase
with distance while still being fairly accurate. In addition, Figures 13 and
14 show the prediction errors of the yaw and pitch angles, respectively, with
respect to the angles themselves. The plots show relatively uniform error
distribution across the learned range. These results validate the
effectiveness and feasibility of PointingNet-E for accurate estimation of
pointing.

Heatmaps of yaw, pitch and position estimation errors with respect to data
sizes can be seen in Figures 15-17. The errors are calculated over the test
data while increasing the number of points $N_{p}$ in $\mathcal{P}$ for
PointingNet-E and the number of points $N_{s}$ in $\mathcal{S}$ for
PointingNet-S. For each cell in the heatmap, PointingNet-S was trained with
$N_{s}$ training data and then used to segment $N_{p}$ data for training
PointingNet-E. The figures also show curves for the error during the increase
of training data for PointingNet-E while having PointingNet-S trained with all
available data. The accuracy increases with the addition of pointing examples.
In addition, the results show the importance of a sufficiently trained
segmentation model which also emphasize the contribution of arm segmentation
to the pointing estimation accuracy.

Figure 18 shows an example for one pointing test image inputted to
PointingNet-E and the images extracted from its layers. The initial layers of
the modified ConvNeXt predominantly detect the edges of the image with a some
focus on the arm. In the following layers, there is a stronger emphasis on the
region of interest, i.e., the arm, compared to the background areas. The
output of the fifth layer highlights the pose of the arm. In the final layers,
the model prioritizes the finger and the entire arm region receiving the
highest intensity. While it is hard to reason about the exact features learned
by the model, visualization of the learned features in each layer of
PointingNet-E provides implications that the model learns to distinguish
between the different elements of the image and focuses on arm and finger
posture. Presumably, PointingNet-E learns to predict finger position and
direction by observing the index finger, but based also on learning patterns
of arm postures.

---
Figure 12: Prediction errors of (top) yaw and pitch angles and (bottom)
position with regards to the distance of the user’s arm from the camera.
Figure 13: Prediction error distribution of the yaw $\gamma$ angle. Figure 14:
Prediction error distribution of the pitch $\beta$ angle. Figure 15: Heatmap
of the yaw angle error (∘) with regards to the sizes of $\mathcal{P}$ and
$\mathcal{S}$ to train PointingNet-E and PointingNet-S, respectively. The
black curve (with the right $y$-axis) shows the error of the yaw angle
estimation with PointingNet-E when using the fully trained PointingNet-S.
Figure 16: Heatmap of the pitch angle error (∘) with regards to the sizes of
$\mathcal{P}$ and $\mathcal{S}$ to train PointingNet-E and PointingNet-S,
respectively. The black curve (with the right $y$-axis) shows the error of the
pitch angle estimation with PointingNet-E when using the fully trained
PointingNet-S.  Figure 17: Heatmap of the position error ($mm$) with regards
to the sizes of $\mathcal{P}$ and $\mathcal{S}$ to train PointingNet-E and
PointingNet-S, respectively. The black curve (with the right $y$-axis) shows
the error of the position estimation with PointingNet-E when using the fully
trained PointingNet-S.  Figure 18: Original image (left) of a pointing
participant and the outputted images from the first (1) to last (2) layers of
the ConvNeXt in PointingNet-E .

Next, the ability of the model to provide pointing position and direction
estimation in real-time is demonstrated. Here, the test participant moved the
arm fairly slow while changing pointing directions. The camera acquired images
in frequency of 38 Hz. Prediction accuracy matches the ones reported in Table
IV. Furthermore, Figure 19 shows an example of one real-time prediction
session. The mean yaw and pitch angles along the session are 1.78∘ and 0.8∘,
respectively. Similarly, the position errors along the $x$-, $y$\- and
$z$-directions are 102.7 mm, 20.66 mm and 21.83 mm, respectively.
PointingNet-E predictions closely match the ground-truth with slightly larger
errors along the $x$-axis. Overall, the results provide strong evidence that
our model is able to accurately predict the direction and position of pointing
gestures in real-time.

Figure 19: Real-time prediction of pointing parameters including (top) yaw and
pitch angles and (bottom) position of the user’s finger.

###  V-E Edge Cases

We next evaluate the performance of PointingNet in specific but interesting
edge cases where accurate predictions may be difficult. Note that previous
experiments included these edge cases in the test data. However, we provide a
designated analysis in this section to better emphasize the abilities of
PointingNet. Designated test sets were collected for six edge cases including:
gloved hands, out-of-frame participant, partly occluded participant, multi-
users in the image, pointing while seated and dual-arm lift.

In the gloved hands case, participants pointed while wearing blue or black
surgical gloves (Figure 20a). This also evaluates model robustness to skin
color. In the out-of-frame case, participants were positioned outside the
camera’s frame with only their pointing arm visible (Figure 20b). This
demonstrates the independence of the model from other body parts as required
in other methods. Similarly, during occlusion, part of the participants body
is not visible. Hence, the participants stood behind large objects (e.g. a
chair and a ladder) with their bodies partially occluded while their pointing
arms are clearly visible (Figure 20c). In multi-user cases, several
participants are seen in the image while only one is pointing (Figure 20d). We
do not consider cases where more than one participant is pointing. In the
fifth case, participants pointed while sitting down (Figure 20e). Finally, the
dual-arm case involves participants lifting both arms (Figure 20f) while one
is pointing and the other is performing other tasks (e.g. head scratching or
waving). All test sets were collected and labeled as described previously.

Table V summarizes the classification success rate and pointing estimation for
all six edge cases. The success rate remains high with a slight decrease for
occlusions, multi-users and dual-arm. Occlusions and out-of-frame are
relatively similar scenarios. However, the arm in out-of-frame scenarios is in
the foreground and clearly visible. In occlusions, on the other hand, the arm
is in the background while foreground objects have the focus. This is likely
why occlusion images receive lower success rate.

When comparing pointing estimation with the general error, positional errors
negligibly increase by 11 mm on average. However, the yaw and pitch errors
increase by an average of 8.08∘ and 7.84∘, respectively. Estimating pointing
direction in these cases is somewhat harder and the mean errors are larger
compared to the general error. Nevertheless, these errors remain low and
feasible for accurate pointing. One may improve accuracy in these cases by
adding corresponding image samples to the training data.

Figure 20: Examples of the six evaluated pointing edge cases: (a) user wearing gloves, (b) user is out-of-frame and only the pointing arm is visible, (c) user is partly occluded by objects in the scene, (d) multiple people in the scene where the pointing user is in the background, (e) user is sitting down and, (f) both arms of the user are lifted. TABLE V: Recognition success rate and estimation accuracy for several edge cases | Success | Position | Pitch | Yaw
---|---|---|---|---
| (%) | (mm) | (∘) | (∘)
Gloves | 94.9 | 81.3$\pm$44.0 | 10.89$\pm$6.9 | 11.02$\pm$3.2
Out-of-frame | 95.5 | 75.5$\pm$35.8 | 10.22$\pm$3.7 | 12.44$\pm$6.1
Occlusions | 89.5 | 70.9$\pm$29.6 | 11.75$\pm$5.9 | 12.86$\pm$3.7
Multi-users | 90.5 | 72.3$\pm$34.1 | 6.57$\pm$2.98 | 7.43$\pm$4.54
Sitting-down | 97.9 | 68.8$\pm$35.6 | 3.87$\pm$1.88 | 5.08$\pm$2.12
Dual-arm | 87.8 | 70.1$\pm$38.8 | 7.45$\pm$2.61 | 8.03$\pm$2.94

##  VI Robot Experiments

Following the evaluation of PointingNet, the effectiveness and practicality of
the model is experimented over robotic platforms in pointing-based human
guidance. The user’s ability to guide the robot towards a desired goal is an
essential aspect of HRI, as it involves direct communication and coordination
between the human and the robot. The following experiments provide an
assessment of the model’s feasibility for real-world applications over two
different robotic platforms. The experimental setups are first described
followed by results.

(a) |  (b)
---|---

Figure 21: Two robotic platforms used in the experiments: (a) Unitree Go1
quadruped robot equipped with a simple RGB web camera for pointing recognition
and a GPS-RTK for localization. (b) Clearpath Husky 4-wheel rover equipped
with a RealSense camera (only RGB data is used) for pointing recognition and a
2D LiDAR for localization.

###  VI-A Setup

####  VI-A1 Software

The PointingNet model was wrapped into a Robot Operating System (ROS) node.
Hence, the framework described in Figure 2 is implemented in ROS and predicts
pointing in real-time. The recognition model, with internal PointingNet-S
segmentation, loops until pointing is identified. Once identified, the
estimation model PointingNet-E is triggered to predict feature vector
$\mathbf{v}$ that corresponds to the captured image. In practice, ROS loops
for several more images after the trigger in order for the pointing arm to
complete its motion and become static. In our experiments, we assume a flat
horizontal floor. Hence, the pointed target is approximated according to
(5)-(6). The target is then transferred to a planning and control node which
is implemented individually for each of the two robotic platforms described
next. In both platforms, however, real-time calculations are performed over a
central computer where WiFi communication enables bi-directional transfer of
sensed information and motion commands.

####  VI-A2 Go1 quadruped robot

The first platform is the Unitree Go1 quadruped robot seen in Figure 21a. A
web camera was mounted on top of the robot in order to observe pointing users.
Furthermore, a GPS-RTK board and an Inertial Measurement Unit (IMU) are used
for global localization and heading calculations, respectively. The robot is
controlled using the manufacturer’s high-level ROS API. Hence, the planning
and control node commands the robot to move directly towards the target
location and stop once reached. The experimental trials were conducted in an
open environment without any obstacles.

####  VI-A3 Husky Rover

The second platform is the Husky 4-wheel rover by Clearpath. The rover is
equipped with a Sick 2D LiDAR for localization. Hence, prior to the
experiments, the robot mapped the environment in order to localize itself such
that it is able to pin-point any pointed target within the generated map. To
demonstrate the ability of PointingNet to acquire images from various cameras,
a RealSense RGB-Depth camera was mounted on the robot as seen in Figure 21b.
Nevertheless, only RGB data was obtained from the camera. When experimenting
with the rover, the pitch angle was ignored. Hence, the robot is instructed to
move along a line formed by the projection of the pointed vector on the floor
starting from the position of the finger. The ROS node for planning and
control will direct the robot to reach the start of the projected line and
move along it until encountering an obstacle. In the experimental trials, a
small bench was set as the desired target such that the robot can identify it
as an obstacle and stop in front of it.

TABLE VI: Experimental results for robot reaching to pointed targets  System | Success | Target reach | Distance
---|---|---|---
rate (%) | error (m) | to target (m)
Quadruped | 100 | 0.36$\pm$0.28 | 3.8$\pm$0.95
Rover | 86.7 | 0.32$\pm$0.2 | 7.2$\pm$2.5
Figure 22: Snapshots of a pointing experiment with the quadruped robot. The
top image shows the moment of pointing along with (a) the image observed by
the robot and (b) estimation of pointing direction. The robot reached the
target with an accuracy of 0.07 m. Figure 23: Snapshots of a pointing
experiment with the quadruped robot. The top image shows the moment of
pointing along with (a) the image observed by the robot and (b) estimation of
pointing direction. The robot reached the target with an accuracy of 0.13 m.

###  VI-B Results

A set of experimental trials was conducted for each of the robot platforms.
Each set included 15 pointing trials to arbitrary targets with respect to the
user and robot. In each trial, the user and robot were randomly re-located
such that the user stood in front of the robot in a random relative distance.
For the quadruped robot, a random target was marked on the floor. Similarly, a
small bench was moved to random positions for the rover to reach. For both
robots and in each trial, the user was instructed to point toward the target.

Table VI summarizes the success rate and mean errors over the 15 pointing
trials for each robot. Errors were measured after the robot has stopped at the
target and taken from the center of the marked target or bench to the center
of the robot. The table also shows the mean distance of the robot from the
desired target at the time of pointing. To test various scenarios, the tested
distances for the rover were larger than for the quadruped robot. Figures
22-24 and Figure 25 show several pointing trials of the quadruped robot and
rover, respectively. While the quadruped robot reached the vicinity of the
target on all trials, the rover missed the bench two times. It is important to
note that the reported deviations from the target are also a result of
localization errors. The GPS-RTK of the quadruped robot suffered from
localization drift and, therefore, errors are larger than for the rover over
shorter travel distances. Similarly, some directive routes for the rover had
lower mapping features for localization and, consequently, the rover missed
two targets. Nevertheless, the results show that the pointing estimation
provides sufficient accuracy for the robot to reach a user desired target
through natural pointing and RGB images.

Figure 24: Snapshots of a pointing experiment with the quadruped robot. The
top image shows the moment of pointing along with (a) the image observed by
the robot and (b) estimation of pointing direction. The robot reached the
target with an accuracy of 0.09 m. Figure 25: Pointing experiment with a
4-wheel rover: (top row, left to right) the rover identifies pointing, plans
motion and moves towards the target (red bench); (bottom row) (a) image
observed by the rover, (b) estimation of pointing direction and (c) planning
and controlling the motion in ROS-Gazebo. Purple arrow denotes the position
and pointing direction estimation of the index finger with regards to the
rovers map and at its initial pose. The rover reached the target with an
accuracy of 0.3 m.

##  VII Conclusions

In this work, a full framework was proposed for a robot to reason about a
pointing user through an RGB camera and act to reach the pointed target. The
framework includes the novel PointingNet model for recognition of a pointing
gesture and estimation of its direction relative to the camera on the robot.
The PointingNet-S model in the background segments any observed lifted arm in
the image for further parsing by the two models. Arm segmentation by
PointingNet-S along with MiDaS depth approximation have been shown to be
crucial for accurate estimation of pointing position and direction. A
comprehensive analysis over a large and diverse test set has validated the
high accuracy of PointingNet compared to state-of-the-art approaches with
skeleton-based models in particular. PointingNet was also successfully tested
on possible edge cases such as occlusions and multiple participants in the
observed image.

The experimental results over two robotic platforms have shown sufficient
accuracy for the robot to reach a human desired target through natural
pointing. In real-world scenarios, our natural pointing model can be
accompanied with verbal instructions for the robot to reach a certain object
or place. Verbal communication provides context regarding the desired task and
can better direct the robot. We leave this for future efforts. Future work may
also consider incorporating gaze estimation from the image during user glance
at the target. As discussed in Section III, agile capturing will be required
as users tend to take a very short glance at the desired target.

## References

  * [1] K. Cooperrider, J. Slotta, and R. Núñez, “The preference for pointing with the hand is not universal,” _Cognitive Science_ , vol. 42, no. 4, pp. 1375–1390, 2018.
  * [2] S. Kita, _Pointing: A foundational building block in human communication_. Mahwah, NJ: Lawrence Erlbaum, 2003.
  * [3] M. Tomasello, _Origins of Human Communication_. The MIT Press, 08 2008.
  * [4] E. Bates and F. Dick, “Language, gesture, and the developing brain,” _Developmental Psychobiology_ , vol. 40, no. 3, pp. 293–310, 2002.
  * [5] G. Hewes, “Gesture language in culture contact,” _Sign Language Studies_ , vol. 4, no. 3, pp. 1–34, 1974.
  * [6] H.-K. Lee and J.-H. Kim, “An hmm-based threshold model approach for gesture recognition,” _IEEE Transactions on pattern analysis and machine intelligence_ , vol. 21, no. 10, pp. 961–973, 1999.
  * [7] N. H. Dardas and N. D. Georganas, “Real-time hand gesture detection and recognition using bag-of-features and support vector machine techniques,” _IEEE Transactions on Instrumentation and measurement_ , vol. 60, no. 11, pp. 3592–3607, 2011.
  * [8] M. E. Hussein, M. Torki, M. A. Gowayyed, and M. El-Saban, “Human action recognition using a temporal hierarchy of covariance descriptors on 3d joint locations,” in _Twenty-third international joint conference on artificial intelligence_ , 2013.
  * [9] Y. Ji, Y. Yang, F. Shen, H. T. Shen, and X. Li, “A survey of human action analysis in hri applications,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 30, no. 7, pp. 2114–2128, 2020.
  * [10] V. Villani, L. Sabattini, C. Secchi, and C. Fantuzzi, “A framework for affect-based natural human-robot interaction,” in _IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN)_ , 2018, pp. 1038–1044.
  * [11] D. Sikeridis and T. Antonakopoulos, “An imu-based wearable system for automatic pointing during presentations,” _Image Processing & Communications_, vol. 21, 04 2017.
  * [12] F. Haque, M. Nancel, and D. Vogel, “Myopoint: Pointing and clicking using forearm mounted electromyography and inertial motion sensors,” in _Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems_ , 2015, p. 3653–3656.
  * [13] E. Littmann, A. Drees, and H. Ritter, “Neural recognition of human pointing gestures in real images,” _Neural Process. Lett._ , vol. 3, no. 2, p. 61–71, 1996.
  * [14] C.-B. Park and S.-W. Lee, “Real-time 3d pointing gesture recognition for mobile robots with cascade hmm and particle filter,” _Image and Vision Computing_ , vol. 29, no. 1, pp. 51–63, 2011.
  * [15] K. Nagai, H. Sakabe, and M. Ohka, “Finger direction recognition toward human-and-robot cooperative tasks,” in _International Symposium on Micro-NanoMechatronics and Human Science (MHS)_ , 2017, pp. 1–3.
  * [16] K. Nickel and R. Stiefelhagen, “Visual recognition of pointing gestures for human–robot interaction,” _Image and Vision Computing_ , vol. 25, no. 12, pp. 1875–1884, 2007.
  * [17] D. Jirak, D. Biertimpel, M. Kerzel, and S. Wermter, “Solving visual object ambiguities when pointing: an unsupervised learning approach,” _Neural Computing and Applications_ , pp. 1–23, 2020.
  * [18] A. Medeiros, P. Ratsamee, J. Orlosky, Y. Uranishi, M. Higashida, and H. Takemura, “3d pointing gestures as target selection tools: Guiding monocular uavs during window selection in an outdoor environment,” _Robomech J_ , vol. 8, no. 11, 2021.
  * [19] C. Martin, F.-F. Steege, and H.-M. Gross, “Estimation of pointing poses for visually instructing mobile robots under real world conditions,” _Robotics and Autonomous Systems_ , vol. 58, no. 2, pp. 174–185, 2010.
  * [20] D. Osokin, “Real-time 2d multi-person pose estimation on cpu: Lightweight openpose,” 2018. [Online]. Available: https://arxiv.org/abs/1811.12004
  * [21] C. Lugaresi, J. Tang, H. Nash, C. McClanahan, E. Uboweja, M. Hays, F. Zhang, C.-L. Chang, M. Yong, J. Lee, W.-T. Chang, W. Hua, M. Georg, and M. Grundmann, “Mediapipe: A framework for perceiving and processing reality,” in _Third Workshop on Computer Vision for AR/VR at IEEE Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * [22] D. Mehta, S. Sridhar, O. Sotnychenko, H. Rhodin, M. Shafiei, H.-P. Seidel, W. Xu, D. Casas, and C. Theobalt, “VNect: Real-time 3d human pose estimation with a single RGB camera,” _ACM Transactions on Graphics (TOG)_ , vol. 36, no. 4, pp. 1–14, 2017.
  * [23] Z. Cao, G. Hidalgo Martinez, T. Simon, S. Wei, and Y. A. Sheikh, “Openpose: Realtime multi-person 2d pose estimation using part affinity fields,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2019.
  * [24] C. Deuerlein, F. Müller, and P. Heß, “Case study of human pointing gestures and their data analysis,” _Procedia CIRP_ , vol. 88, pp. 223–228, 2020, 13th CIRP Conference on Intelligent Computation in Manufacturing Engineering, 17-19 July 2019, Gulf of Naples, Italy.
  * [25] K. Hu, S. Canavan, and L. Yin, “Hand pointing estimation for human computer interaction based on two orthogonal-views,” in _International Conference on Pattern Recognition_ , 2010, pp. 3760–3763.
  * [26] B. Azari, A. Lim, and R. T. Vaughan, “Commodifying pointing in hri: Simple and fast pointing gesture detection from rgb-d images,” _Conference on Computer and Robot Vision (CRV)_ , pp. 174–180, 2019.
  * [27] Y. Lai, C. Wang, Y. Li, S. S. Ge, and D. Huang, “3d pointing gesture recognition for human-robot interaction,” in _Chinese Control and Decision Conference (CCDC)_ , 2016, pp. 4959–4964.
  * [28] A. Kuramochi and T. Komuro, “3d hand pointing recognition over a wide area using two fisheye cameras,” in _HCI International - Late Breaking Papers: Multimodality, eXtended Reality, and Artificial Intelligence_ , 2021, pp. 58–67.
  * [29] D. Shukla, O. Erkent, and J. Piater, “Probabilistic detection of pointing directions for human-robot interaction,” in _International Conference on Digital Image Computing: Techniques and Applications (DICTA)_ , 2015, pp. 1–8.
  * [30] M. Tölgyessy, M. Dekan, F. Duchoň, J. Rodina, P. Hubinský, and L. Chovanec, “Foundations of visual linear human–robot interaction via pointing gesture navigation,” _International Journal of Social Robotics_ , vol. 9, pp. 509–523, 2017.
  * [31] K. Pietroszek, L. Tahai, J. R. Wallace, and E. Lank, “Watchcasting: Freehand 3d interaction with off-the-shelf smartwatch,” in _IEEE Symposium on 3D User Interfaces (3DUI)_ , 2017, pp. 172–175.
  * [32] S. S. Das, “Precise pointing direction estimation using depth data,” in _IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN)_ , 2018, pp. 202–207.
  * [33] R. Kehl and L. Van Gool, “Real-time pointing gesture recognition for an immersive environment,” in _IEEE International Conference on Automatic Face and Gesture Recognition_ , 2004, pp. 577–582.
  * [34] J. Yang, W. Lu, and A. Waibel, “Skin-color modeling and adaptation,” in _Computer Vision — ACCV’98_ , R. Chin and T.-C. Pong, Eds., 1997, pp. 687–694.
  * [35] L. Zhou, Z. Feng, Z. Cai, and X. Yang, “A Kinect-based finger pointing detection algorithm,” in _Second IYSF Academic Symposium on Artificial Intelligence and Computer Engineering_ , W. Qin, Ed., vol. 12079. International Society for Optics and Photonics, 2021, pp. 26 – 31.
  * [36] J. Shotton, R. Girshick, A. Fitzgibbon, T. Sharp, M. Cook, M. Finocchio, R. Moore, P. Kohli, A. Criminisi, A. Kipman, and A. Blake, “Efficient human pose estimation from single depth images,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 35, no. 12, pp. 2821–2840, 2013\.
  * [37] Z. Khokhar, Z. Xiao, and C. Menon, “Surface EMG pattern recognition for real-time control of a wrist exoskeleton,” _Biomedical engineering online_ , vol. 9, p. 41, Aug. 2010.
  * [38] J. DelPreto and D. Rus, “Plug-and-play gesture control using muscle and motion sensors,” in _ACM/IEEE International Conference on Human-Robot Interaction_ , 03 2020, pp. 439–448.
  * [39] N. Caruana, C. Inkley, P. Nalepka, D. Kaplan, and M. Richardson, “Gaze facilitates responsivity during hand coordinated joint attention,” _Scientific Reports_ , vol. 11, no. 1, pp. 1–11, 2021.
  * [40] A. G. Gomaa, G. Reyes, A. Alles, L. H. Rupp, and M. Feld, “Studying person-specific pointing and gaze behavior for multimodal referencing of outside objects from a moving vehicle,” _Proceedings of the 2020 International Conference on Multimodal Interaction_ , pp. 501–509, 2020.
  * [41] J. Gonzalez-Billandon, G. Belgiovine, M. Tata, A. Sciutti, G. Sandini, and F. Rea, “Self-supervised learning framework for speaker localisation with a humanoid robot,” in _IEEE International Conference on Development and Learning (ICDL)_ , 2021, pp. 1–7.
  * [42] J. Han, P. Fang, W. Li, J. Hong, M. A. Armin, , I. Reid, L. Petersson, and H. Li, “You only cut once: Boosting data augmentation with a single cut,” in _International Conference on Machine Learning (ICML)_ , 2022.
  * [43] S. Zagoruyko and N. Komodakis, “Wide residual networks,” _arXiv preprint arXiv:1605.07146_ , 2016.
  * [44] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” _Communications of the ACM_ , vol. 60, no. 6, pp. 84–90, 2017.
  * [45] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 1–9.
  * [46] V. Bazarevsky, I. Grishchenko, K. Raveendran, T. Zhu, F. Zhang, and M. Grundmann, “Blazepose: On-device real-time body pose tracking,” _arXiv preprint arXiv:2006.10204_ , 2020.
  * [47] R. Ranftl, K. Lasinger, D. Hafner, K. Schindler, and V. Koltun, “Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer,” _IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)_ , 2020.
  * [48] R. Ranftl, A. Bochkovskiy, and V. Koltun, “Vision transformers for dense prediction,” _ArXiv preprint_ , 2021.
  * [49] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie, “A convnet for the 2020s,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 11 976–11 986.
  * [50] S. M. LaValle, _Planning Algorithms_. Cambridge University Press, 2006.
  * [51] S. Karimpouli and P. Tahmasebi, “Segmentation of digital rock images using deep convolutional autoencoder networks,” _Computers & Geosciences_, vol. 126, pp. 142–150, 2019.
  * [52] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18_. Springer, 2015, pp. 234–241.
  * [53] Z. Zhou, M. M. R. Siddiquee, N. Tajbakhsh, and J. Liang, “Unet++: Redesigning skip connections to exploit multiscale features in image segmentation,” _IEEE transactions on medical imaging_ , vol. 39, no. 6, pp. 1856–1867, 2019\.
  * [54] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie, “Feature pyramid networks for object detection,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 2117–2125.
  * [55] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [56] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 4700–4708.
  * [57] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam, “Encoder-decoder with atrous separable convolution for semantic image segmentation,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 801–818.
  * [58] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _arXiv preprint arXiv:1409.1556_ , 2014.