# AirWare: Utilizing Embedded Audio and Infrared Signals for In-Air Hand-
Gesture Recognition

Nibhrat Lohia, Raunak Mundada, Eric C. Larson Nibhrat Lohia and Raunak Mundada
are Alumni of the Department of Statistical Science, Southern Methodist
University, Dallas, TX, 75206.
E-mail<EMAIL_ADDRESS>Eric C. Larson is a Professor in the Department of
Computer Science, Southern Methodist University, Dallas, TX, 75206Manuscript
received July, 2018.

###### Abstract

We introduce AirWare, an in-air hand-gesture recognition system that uses the
already embedded speaker and microphone in most electronic devices, together
with embedded infrared proximity sensors. Gestures identified by AirWare are
performed in the air above a touchscreen or a mobile phone. AirWare utilizes
convolutional neural networks to classify a large vocabulary of hand gestures
using multi-modal audio Doppler signatures and infrared (IR) sensor
information. As opposed to other systems which use high frequency Doppler
radars or depth cameras to uniquely identify in-air gestures, AirWare does not
require any external sensors. In our analysis, we use openly available APIs to
interface with the Samsung Galaxy S5 audio and proximity sensors for data
collection. We find that AirWare is not reliable enough for a deployable
interaction system when trying to classify a gesture set of 21 gestures, with
an average true positive rate of only 50.5% per gesture. To improve
performance, we train AirWare to identify subsets of the 21 gestures
vocabulary based on possible usage scenarios. We find that AirWare can
identify three gesture sets with average true positive rate greater than 80%
using 4–7 gestures per set, which comprises a vocabulary of 16 unique in-air
gestures.

###### Index Terms:

Convolutional Neural Networks, Deep Learning, Doppler, Gesture Recognition

##  1 Introduction

Communicating through hand gestures is ubiquitous across cultures.
Incorporating hand gestures into machine interaction has proven difficult
because one must reliably detect the gestures and infer their meaning
[lee2010search]. Even so, with the increasing variety of devices that can
interact with humans in more natural ways, in-air hand gesture recognition
systems have grown in popularity. Major technology companies like Google
[Soli], Microsoft [gupta2012soundwave], Amazon, and HP have released devices
that recognize some basic in-air hand gestures. To achieve reliability, these
devices employ specialized sensors or vision systems, like the Microsoft
Kinect, which increase cost and reduce the potential ubiquity of the device.
This is worrying because the use of in-air hand gestures to interact with a
machine is often desired only in niche scenarios when touch is inappropriate
or difficult. This is especially true for mobile devices (1) when the device
is small and touch is harder to use without occluding screen content (such as
a watch) or (2) in situational impairments, like wearing gloves or cooking,
when hands get dirty and touching a smart-phone or tablet is not desired
[dumas2009multimodal, wobbrock2006future]. There are also scenarios where in-
air gestures may add to the user experience, such as gaming or productivity
applications.

In this study, we investigate methods for detecting and classifying in-air
gestures using commodity sensors on a smartphone. More specifically, we
present AirWare, a system that fuses the information from the on-board
infrared (IR) proximity sensor of a Samsung Galaxy S5 with the Doppler shifts
detected by the microphone. Like previous work [aumi2013doplink,
chen2014airlink, gupta2012soundwave, sun2013spartacus] we play an inaudible
tone from the speakers and record from the microphone continuously. Using
signal processing and machine learning, we fuse the parameters of the IR
proximity sensor with the Doppler features to predict a large vocabulary of
different in-air gestures.

Our approach differs from previous work in that we (1) combine complementary
sensors that are already embedded in the mobile phone and (2) attempt to
classify a relatively large vocabulary of different gestures. Most previous
works only cover a few basic interactions (like panning), which does not
provide a rich interaction modality.

To inform the design and evaluate AirWare, we conducted a user study with 13
participants that performed each gesture several times (load balanced in terms
of presentation order). We show that, on average, AirWare can recognize the
full gesture vocabulary with only 50.47% average true positive rate per
gesture per user on 21 gestures. We conclude that the full 21 gesture
vocabulary is not accurate enough to support user interaction. However, using
various reduced vocabularies of 4–7 gestures, AirWare can achieve average true
positive rates greater than 75% for each reduced vocabulary. In total, the
reduced gesture sets comprise 16 unique gestures. We enumerate our
contributions as follows:

  1. 1.

We investigate the performance of fusing Doppler gesture sensing techniques
with the embedded IR proximity sensor using various machine learning
algorithms. We also compare the performance of a number of convolutional
neural network architectures.

  2. 2.

A human subjects evaluation: we validate the technology in a user study with
13 participants.

  3. 3.

We compare two different methods for collecting gesture data. The first
requires the IR sensor to be activated and the second is a free-form system.
We conclude that the free-form system creates variability in the way gestures
are performed such that the machine learning algorithms cannot readily
identify the gestures. Therefore, the AirWare system requires users to be
instructed on how to perform each gesture.

  4. 4.

We investigate personalized calibration to boost the recognition true positive
rate of the classifier, as well as providing an out-of-the-box gesture
recognition system, showing that user calibration improves the performance of
AirWare. We also investigate the amount of training data required to calibrate
the AirWare system, concluding that 2-3 examples per gesture are needed to
properly calibrate the system.

  5. 5.

We investigate a number of reduced gesture vocabularies that tailor to
different application use cases, showing average true positive rates greater
than 80% among subsets of gestures. When comprising the different subset, we
conclude that AirWare can support about 16 total gestures. We also conclude
that some gesture combinations cannot be supported, such as simultaneously
identifying pans and flicks.

##  2 Related Work

Dating back to 1980, in-air gesture sensing was achieved using commodity
cameras with a high degree of success [rubine1991automatic]. The RGB image of
a user-facing camera was used to detect and follow hand movements
[hilliges2009interactions, rautaray2015vision]. Even so, privacy concerns and
the requirements of processing video (battery life, lag time) limited the
impact of the technology [hinckley2003synchronous, locken2012user,
song2014air, suarez2012hand]. To mitigate these concerns, researchers have
been innovating in how they sense hand motions. In Samsung’s Galaxy S4 and S5
smart-phones, a dedicated infrared proximity sensor is used to detect hand
motions above the phone, sensing velocity, angle, and relative distance of
hand movements. The estimation is coarse, but allows for recognition of a
number of panning gestures. We note that the IR proximity sensor is not unique
the Samsung smartphones, but is used by a number of different manufacturers.
However, these manufacturers typically do not provide access the the sensor
via a developer API. With this in mind, the AirWare methodology could be
applied to these phones in the future, once the sensors become accessible.

gupta2012soundwave used an inaudible tone played on speakers and sensed the
Doppler reflections to determine when a user moved their hands toward or away
from an interface. aumi2013doplink, bannis2014adding, sun2013spartacus, and
chen2014airlink extended this work to detect pointing and flick gestures
toward an array of objects (including smartphones) using the Doppler effect.
These previous works typically recognize 2–4 gestures and many employ more
than one set of speaker and microphone. In contrast, AirWare attempts to
classify 21 gestures and various subsets ranging from 4–7 gestures per set.
AirWare is able to classify such a large vocabulary of different gestures
because the IR and audio Doppler combination provides complementary sensing
information without any external sensors.

There have also been a number of innovative solutions that use infrared
illuminating pendants [starner2000gesture], magnetic sensors [chen2013utrack],
side mounted light sensors [butler2008sidesight], and even the unmodified GSM
antenna radiation [zhao2014sideswipe]. However, AirWare is more ambitious in
the vocabulary size of gestures we attempt to classify, as well as unique in
terms of the fused sensor outputs investigated. Moreover, AirWare does not use
external sensors, but instead employs already embedded sensors from the mobile
phone.

raj2012ultrasonic review the HCI uses of Doppler sensing from ultrasonic range
finders. Although this requires an extra sensor, it uses many of the same
techniques as audio Doppler sensing. By sending a set of ”pings” into the
environment, the proximity of the hand (or any object) can be ascertained with
high accuracy. AirWare shares some similarity in sensing techniques as we also
employ a proximity sensor. Even so, AirWare uses the embedded Galaxy S5
proximity sensor, which is considerably less precise than the external
ultrasonic sensor employed by previous works.

butler2008sidesight produce IR sensor boards attached to a mobile device that
succeed at identifying single- and multi-finger gestures adjacent to a device;
however, this work does not address in-air gestures, use built-in hardware,
combine modalities, or address differentiating a large gesture vocabulary.

Figure 1: Progression of the AirWare interface used in our data collection.

kim2016hand use micro-Doppler signatures with a convolution neural network.
These Doppler signatures are measured by continuous wave Doppler radar at
5.8GHz (rather than the audio Doppler signal employed by AirWare). Their work
encourages us to utilize convolutional neural networks to recognize in-air
gestures. Their system classifies a set of 10 gestures with a five-fold cross
validation accuracy of 85.6%. For a reduced gesture set, the accuracy
increases to 93.1%. Although the system performs well, hand motions of the
gestures are controlled. For example, swiping left to right was a quick snap
that involved the wrist and all five fingers. However, for swiping right to
left, the wrist was no longer stationary and moved with only three fingers
involved. AirWare, however, does not impose such restrictions on the users
motions and, instead, relies on the IR proximity sensor measurement along with
Doppler signatures to classify the gestures. Moreover, AirWare employs
convolutional networks on multiple sensor sources using embedded sensors,
rather than an external, high frequency RF radar system.

Similar to kim2016hand, kim2016human use deep convolution neural networks with
continuous wave Doppler radars (i.e., using RF signals) for detection of
humans and, to some degree, hand gestures. It is important to note that Kim
uses specialized high frequency radar equipment. However, AirWare employs low-
frequency audio Doppler from commodity hardware. Moreover, we employ infrared
proximity sensors to create additional information that Doppler shifts do not
capture, such as the occurrence of movements transverse to the sensing
apparatus. raj2012ultrasonic used a similar high-frequency device setup to
classify most basic gestures. The results from these works were promising but
fall in the same category of adding external sensors for recognition. Most
have used ultra high frequency sonars for collecting Doppler signatures, with
some specific spatial arrangements in some cases, thus causing greater
frequency shifts which are relatively easier to classify.

##  3 Theory of Operation

In this section, we outline the different properties of each sensing modality:
Doppler and IR proximity. We also posit an argument for why combining these
modalities is inherently complementary. These sections also summarize the ways
in which we access and pre-process each signal.

Figure 2: Average true positive rate for different short time Fourier
transform parameters. A grid search reveals that a window size of 4096, 50%
overlap, and use of $\pm 16$ bins from $f_{0}$ performs the best.

###  3.1 Audio Doppler from Speaker and Microphone

Audio Doppler sensing for gesture recognition is discussed in detail in a
number of papers [gupta2012soundwave, aumi2013doplink]. Our method generally
follows that of other papers. We play an 18 kHz sine wave from the speakers of
the mobile phone, while continuously sampling from the microphone at 48 kHz.
This means that a constant 18 kHz sine wave will be sampled from the
microphone. When an object moves toward or away form a stationary phone, the
microphone can detect Doppler frequency reflections. These manifest as
additional reflections, added to the 18 kHz sine wave. The change in frequency
is given by:

$\Delta f=\frac{f_{0}v}{c}\cos(\theta)$ (1)

where $c$ is the speed of sound, $v$ is the velocity of the object, $\theta$
is the angle between the motion of the object and the microphone, and $f_{0}$
is the frequency of the sine wave played from the speakers. When analyzing the
signal, faster motions toward the microphone result in more pronounced
frequency increases. Movement away from the microphone results in frequency
decreases. The angle between the object movement and the microphone is a key
factor. Movement transverse to the microphone results in no frequency shift.
Movement directly toward or away the microphone maximizes the possible
frequency change. As such, the frequency reflections form different hand
gestures at different angles will manifest differently in the Doppler signal.
More than just frequency, the surface area of the object determines the
magnitude changes of the frequency reflections. This means it is possible to
detect the difference between waving a finger at the microphone versus waving
a hand at the same velocity. To capture these changes in the frequency over
time, like previous approaches, we use the short time Fourier transform
(STFT). Specifically, we use a sampling rate of 48 kHz. We use a Hamming
window to reduce spectral leakage. Other parameters of the STFT need to be
chosen to trade off resolution in time and resolution in frequency (i.e., a
classic signal processing problem). It was unclear what trade-off between time
and frequency to employ in order to capture the motion of the hand and
fingers. For instance, choosing a large window size would increase our
frequency resolution (i.e., our ability to discern Doppler shifts), but would
also reduce our time resolution (i.e., our ability to observe quick
movements). As such, we decided to grid search different STFT parameters. Each
different configuration resulted in slightly different frequency profiles that
could be used as features in our machine learning algorithms. We investigated
18 different configurations based upon the following combinations of
parameters:

  * •

window size (and FFT size) of 1024, 2048, and 4096 samples

  * •

overlap between windows of 25%, 50%, and 75%

  * •

number of frequency bins above and below $f_{0}$ to include as features, 8 or
16

The average true positive rate per gesture per user of the different
configurations is shown in Figure 2. Many configurations result in similar
performance, but the best configuration was found to be: window size of 4096
points, 50% overlap, and 16 bins above and below $f_{0}$. More details about
the machine learning and cross validation techniques are discussed later. We
save the STFT for three seconds of time data (discarding the initial startup
windows). An example of the STFT with the best found configuration can be seen
in Figure 3.

Figure 3: Zoomed spectrogram of three different gestures. Top of plots show
the output of the IR sensor angle and velocity.

To normalize and control dynamic range, we take the decibel magnitude of the
STFT. The implementation of the STFT grid search and feature extraction
techniques have been made open source and are available at [opensourceRaunak].

###  3.2 IR Sensing

The infrared proximity sensor for the Samsung Galaxy S5 is a set of four
infrared sensors surrounding an infrared LED. Infrared light reflects back
towards the four sensors when an object, like the hand, is above any of the
sensors. By detecting which sensors are activated first and in which order,
the sensor can infer what angle an object is moving laterally. The time
difference in which each of the sensors is excited determines the approximate
velocity of the object as it enters or exits the sensor area. When coupled
with Doppler sensing, these two sensing modalities have a number of
complementary features.

###  3.3 Complementary Sensors

While Doppler sensing can provide rich information about the direction of
motion towards or away from the microphone, about the relative velocity of
movement, and the relative surface area of the object, it is blind to absolute
trajectory. That is, movement right and left can look identical because they
lie in the same plane. Moreover, perpendicular movement to the microphone may
not cause any Doppler shifts and different combinations of velocity and
incidence angle can manifest similarly in the Doppler signal. For a number of
in-air gestures, this directionality and velocity are critical to
understanding the gesture. Fortunately, these lateral movements are exactly
what the IR sensor is designed to detect, which adds a complementary source of
information. However, the use of an IR proximity sensor is not a panacea. The
infrared sensor is often blind to gestures that occur at a distance from the
sensor and it is often unable to distinguish motions that have a “straight-on”
trajectory. In this way, the sensing modalities of infrared and Doppler are
quite complementary as outlined below:

Doppler Sensing:

  * •

Sensitive to motion towards and away from the microphone but agnostic to
lateral motions. Angle of motion results in different reflected frequencies.

  * •

Sensitive to the surface area of the object. Larger objects result in larger
magnitude reflections.

  * •

Sensitive to the overall velocity towards or away from the microphone. Higher
velocity movements result in different reflected frequencies

  * •

Sensitive to motions at both far and near distances from phone.

IR Proximity Sensor:

  * •

Sensitive to motion lateral to sensor and, to some degree, motions toward the
sensor

  * •

Can discern angle of lateral motions but not when motion is directed towards
sensor

  * •

sensitive to the overall velocity of lateral motions

  * •

Only sensitive to motions that occur relatively close to the sensor

Given these properties, it is easier to understand why the combination of
sensors reveals complementary details about in-air hand motions. While the IR
sensor can detect panning motions, it cannot distinguish if the motion comes
from fingers or the palm of the hand. While the Doppler signal shows velocity
of the hand as it passes by the microphone, it cannot discern if the motion is
from left to right or from top to bottom and vice-versa. These types of
differences are imperative to understand a large vocabulary of in-air hand
gestures. We investigate several methods to combine the IR sensor stream and
the STFT through traditional machine learning algorithms and convolutional
neural networks.

##  4 Spectrogram Processing

In this stage, our aim is to process and combine the features from the STFT
and the IR sensors so that they can be analyzed by a machine learning
algorithm. We start by finding the magnitude of the generated 18 kHz tone
across the entire STFT, $M_{0\mathrm{dB}}(t)$, where $t$ denotes the frame
number $(0,1,\ldots,99)$. We then isolate a band of magnitudes around the
frequency in a range of frequency bins above and below $f_{0}$. The number of
bins above and below the $f_{0}$ is a grid-searched parameter in our analysis.
We found that using 16 bins above and below the $f_{0}$ is sufficient for
classification. These ranges are shown in Figure 3. During this feature
extraction, we also eliminate the magnitude of $f_{0}$ from the spectrogram,
as the value is relatively constant in magnitude and therefore has limited
predictive capability as a feature.

After processing the STFT, we process the features extracted from the Samsung
Galaxy S5’s IR sensor. The sensor interface uses a “push” style API where the
application subscribes to notifications when the sensor is activated. The
notification includes the speed (a normalized value between 0 and 100) and
angle of any detected movements. The angle is an average of the entering and
exit angles. However, gestures that do not move laterally across the sensor
typically register as having zero velocity and zero degree angle because the
sensor cannot validly estimate the movement (but detects that an object is
close to the sensor). Each time we are notified of a movement, we log the
event and time-stamp when the event occurred.

###  4.1 Segmentation

When a user performs a gesture, it may or may not activate the IR sensor. We
performed two rounds of data collection. The first did not require that users
activate the IR sensor with the gesture and the second did require that the IR
sensor be activated. The segmentation procedure differs slightly between these
two scenarios. When we required the user to activate the IR sensor,
segmentation was straightforward: we buffer the audio signal 1.25 seconds
before and after the IR activation. When we did not require the IR sensor to
activate, we buffered 1.25 seconds before and after any “event of interest.”
We define this event to be when either the IR sensor is activated or when the
magnitude of frequency bins directly greater than and less than $f_{0}$
increase by 10 dB. Intuitively, this occurs when there is enough motion to
cause reflections of the Doppler audio signal. We also note that, when not
requiring the IR sensor to be activated, we expect an increased number of
false positives because any motion might trigger the segmentation algorithm.
Positively, requiring the IR sensor to be activated by the gesture can be
considered an effective means of reducing false positives. Negatively, it also
requires users to manipulate their gestures in a way that they always trigger
the sensor at the top of the phone. This limitation is discussed in more depth
in the next section.

Once all data is collected for all users, we employ normalization of each of
the IR features (angle and velocity) and of the entire spectrogram magnitudes
such that the all features are zero mean and unit standard deviation.

##  5 Experimental Methodology

In our pilot tests, we asked participants to perform each of the 21 gestures
in the way “that made the most sense to them.” In this way, we sought to
collect more realistic data where participants could be trained simply from a
textual prompt of what the gesture was, without explicit training or
demonstration. Therefore, we thought the gesture would be more intuitive to
the user (since they exhibit their internalization of the gesture, rather than
mimicking a gesture they were shown). However, this data was never
classifiable at a rate more than chance. We abandoned the idea that a large
gesture vocabulary could be collected without explicitly demonstrating the
gestures to participants. Based on our experience in the pilot study, we
decided to update our methodology to include showing videos of the gestures
being performed. Participants were then asked to perform the gesture to
demonstrate their understanding. Therefore, all participants were shown how to
perform the gestures and participants demonstrated their understanding to the
researchers before data collection started. Practically, this also means that
new users of AirWare would also need to go through the same instructional
videos to learn how to perform the gestures in the vocabulary. We see this as
a necessary limitation of the AirWare system: without an instructional phase,
there is too much variability among the gestures performed to detect them
reliably.

Because our pilot study had uncovered that gesture consistency might be
problematic, we conducted a user study in two phases. We chose two phases
because it was unclear how to define what a “proper” gesture consisted of.
Each phase differed in what the data collection application judged to be a
properly performed gesture. In the first phase, we collected gesture data from
the participants for every gesture in our vocabulary regardless of whether the
IR sensor was activated. That is, the user performed a gesture based upon
their memory of how the gesture was performed from the instructional videos.
In the second phase, we only informed the participant that a gesture was
performed successfully when the IR proximity sensor was activated. That is,
they were asked to repeat the gesture until they learned how to perform the
gesture while also activating the sensor at the top of the phone screen. In
this way, users needed to manipulate the way they performed the gesture such
that they understood where the proximity sensors was physically located on the
phone and how to activate it with each gesture.

Different users participated in each phase to protect against crossover
effects. That is, no user participated in both phases of the data collection.
We show later on that requiring the IR sensor to be activated greatly
increases the ability of the machine learning algorithm to correctly identify
the gesture. Practically, this means that the AirWare system will almost
certainly require an “instructional application” that trains users to perform
the gestures, and then verifies that the user understands how to perform the
gesture such that the IR sensor is activated. While this is an additional
limitation of the system because it imposes constraints on the gestures, such
an instructional application would likely be required no matter what, as
learning to perform 21 gestures for any user without some instruction can be
considered a daunting task.

In the first phase, 8 participants were recruited from university classes (age
range: 19-30, Male: 60%). During a session, participants were introduced to
the AirWare data collection mobile application and a demonstration of all the
gestures were given via the video recording. Participants were then instructed
to show the researcher each gesture. They weren’t told about the sensor
locations on the phone. The ambient environment was relatively quiet and
without many acoustic disturbances. Participants were instructed to hold the
smart-phone in one hand and perform gestures “above” the phone with the other
hand.

In the second phase, 13 participants were recruited (age range: 19-30, Male:
66%). Participants were similarly introduced to the data collection
application but were also instructed about the location of IR proximity sensor
on the phone (as described). The user interface showed the participant
whenever the IR sensor detected a movement through an animated label on the
application. The gesture data was registered only when the IR sensor detected
a movement; otherwise, the interface prompted the user to repeat the gesture.
On average, users had some initial trouble learning how to manipulate the
sensor for some gestures such as “tap” but were quickly able to alter their
strategy to tap towards the top of the phone (where the IR sensor was
located). All users were able to successfully activate the IR sensor after two
or three trials per gesture.

Figure 4: Examples of the different gestures predicted in the AirWare
vocabulary.

###  5.1 Gesture Vocabulary

Participants performed 21 different gestures as instructed on the screen of
the phone via a custom data collection app. A gesture name would appear on the
screen and the user would perform the in-air gesture (Figure 4). All sensor
data was saved locally on the phone for later processing. Users went through
each gesture one time as practice (practice data was not used in analysis) and
then were presented with a random permutation of the gestures. For
participants in the second phase, the practice session lasted as long as was
needed for the subject to learn how to activate the IR sensor. In all, each
participant performed between 5 and 10 iterations of each gesture. The
different number of gestures per participant is an artifact of the way the
gestures were randomly presented. We let participants perform gestures for 45
minutes and then ended the session. On average, each participant performed
about 250 gestures. Note that the smartphone was used for data collection
only. Subsequent analysis was performed offline.

The initial gestures chosen were based upon an informal review of other in-air
gesture sensing systems. After the initial pilot study, we refined the
gestures based upon informal discussions about what gestures were most
intuitive to perform. The final gesture set consisted of:

  * •

Flick left/right/up/down (quick hand movement from wrist)

  * •

Pan left/right/up/down (hand flat, movement from elbow)

  * •

Slice left/right (a fast “sword” motion diagonally across phone)

  * •

Zoom in/out (whole hand)

  * •

Whip (motion towards phone like a holding a whip)

  * •

Snap (similar to whip but snapping while moving toward the phone)

  * •

Magic wand (slow waving of the fingers towards the palm)

  * •

Click, double click (with finger, but not touching screen)

  * •

Tap, double tap (with full hand)

  * •

Circle (circular motion above the phone, hand flat)

  * •

Erase (moving hand back and forth)

In our pilot studies, more gesture were included in the vocabulary. However,
some gestures which users felt were awkward or unintuitive were removed. These
included gestures such as hand wobble, finger wave in/out, and push/pull
gestures.

##  6 Machine Learning Description

To create, train and validate machine learning algorithms we use a combination
of packages in Python. Specifically, we use the “scikit-learn” library
[pedregosa2011scikit] and Keras [chollet2015keras] with the TensorFlow
[tensorflow2015-whitepaper] back-end. We chose to investigate several
different machine learning baselines and also several different convolutional
neural network architectures. It was unclear what neural network architecture
and parameters of the architecture would be optimal, so we chose to train
several variants and perform hyper parameter tuning for each architecture.

###  6.1 Baseline Models - Traditional ML algorithms

For baseline comparison, we investigated several traditional machine learning
algorithms including multi-layer perceptrons, linear support vector machines,
and random forests. The doppler information is pre-processed using Principal
Component Analysis (PCA) to reduce the dimensions. The IR information is
averaged across time steps for each gesture. For each model, the hyper-
parameters as well as the number of principal components for doppler
information are selected via a randomized grid searching strategy. Based on
our grid searching results, hyper-parameters for each model are as follows:

Random Forest:

  * •

Number of trees/estimators: 1000

  * •

Bootstrap: True

  * •

Node split criterion: Gini-index

  * •

Maximum Features: $\sqrt{N}$

  * •

Number of Principal Components: 100

Support Vector Machines:

  * •

Kernel: Linear kernel

  * •

Penalty parameter: 10

  * •

Number of Principal Components: 100

Multi-layer Perceptron:

  * •

Hidden Layer 1 unit size: 500

  * •

Hidden Layer 2 unit size: 250

  * •

Early Stopping: True

  * •

Gradient Solver: Stochastic Gradient Descent

  * •

Activation Unit: Tanh

  * •

L2 Regularization: 0.01

  * •

Number of Principal Components: 100

We choose these algorithms because they span a wide variation of properties
including various decision boundary capabilities such as linear versus
arbitrary.

###  6.2 Convolutional Network Architectures

In addition to these baseline machine learning models, we chose to investigate
four different convolutional neural network architectures. The differences
between each model come from the number of convolutional and dense layers
employed, as well as the type of convolution employed (one dimensional versus
two dimensional). The first three models all employed one-dimensional
convolutions. This makes the processing more similar to approaches used in
natural language processing than in image processing. In text processing, one-
dimensional convolutional filters are typically convolved with the word
embedding matrix over a sequence [severyn2015learning]. The fourth model
employed two dimensional convolutional filters on the input spectrogram, which
is more common in image processing.

Each architecture follows from the basic diagram shown in Figure 5. In this
architecture, the spectrogram and IR signals are processed separately, through
similar convolutional branches of the network. They are then concatenated and
passed to dense hidden layers. The differences among the three models that
employed one-dimensional convolutions is the depth of the convolutional and
dense layers employed. The most simple architecture employs two convolutional
layers and two dense layers, followed by an output layer. Another model
employs three convolutional layers, and another model employs three dense
layers. In all models, every convolutional layer is followed by a max pool
layer. $L_{2}$ regularization is used in all convolutional layers to minimize
over-fitting. Rectified Linear Unit (ReLU) activations are used everywhere
except the final layer to speed up the training and avoid unstable gradients.
Finally a Softmax layer is used at the output which finally classifies the
gestures into 21 different classes. To more clearly reference each of the four
models, we refer to each model by number. Note that all models use two
convolutional layers to analyze the IR proximity sensor, but different number
of layers for the spectrogram branch of the network:

  * •

Model 1: 1D convolutional filters, two convolutional spectrogram layers, two
dense layers

  * •

Model 2: 1D convolutional filters, two convolutional spectrogram layers, four
dense layers

  * •

Model 3: 1D convolutional filters, three convolutional spectrogram layers,
four dense layers

  * •

Model 4: 2D convolutional filters, two convolutional spectrogram layers, two
dense layers

Figure 5: Diagram of the convolutional neural networks investigated. We vary
the number of layers in the convolutional layers and number of dense layers as
part of our analysis.

When training the network, we apply random perturbations to the input
spectrogram and IR sequences to help avoid over fitting and increase
generalization performance (i.e., data expansion). We randomly shift the data
temporally up to 10%. That is, we shift the entire spectrogram sequence
forward or backward in time randomly by up to 10%. The sequences are 2.5
seconds in duration, so this means that the the spectrogram and/or the IR
stream might shift by 250 ms. This is applied to the IR data and the
spectrogram separately (resulting in different random time shifts). This helps
with generalization performance because the Samsung Gesture API is somewhat
inconsistent in the timings for when it provides the push notification that
the IR sensor has been activated. Therefore this data expansion mirrors the
actual use case well.

###  6.3 Hyper-parameter Tuning

Based on the works of Bengio [bengio2012practical] and Bergstra et al.
[bergstra2011algorithms] we chose to use the Tree-structured Parzen Estimator
approach for hyper parameter tuning. These works established this estimation
approach to be superior for tuning hyper-parameters compared to randomized
search. During tuning, we only vary the number of convolutional filters and
kernel size for the spectrogram branch of the network because the signal size
is relatively more complex than the IR sensor stream. The filters applied to
the IR signal are held constant at 2 one dimensional filters with kernel
length of 2. The following parameters were tuned:

  * •

L2 Regularization: Normal Distribution with mean 0.001 and s.d 0.0001

  * •

Learning Rate ($10^{-X}$): [-6, -5, -4, -3, -2, -1, 0]

  * •

Number of convolution filters: [8, 16, 32, 64]

  * •

Kernel Size: [2, 3, 5]

  * •

Dropout: Uniform Distribution [0, 0.99]

  * •

Number of hidden layer units: [32, 64, 128, 256, 512]

  * •

Kernel weight initializers: [He normal and uniform distributions], [Glorot
normal and uniform distributions], [LeCun normal and uniform distributions]

Each of the four models underwent hyper parameter tuning. When comparing the
different architectures, we use the best set of hyper parameters found for
each architecture.

##  7 Results and Summary

We divide our results into three overarching sections: comparisons between
segmentations that require IR activation versus not requiring IR activation,
classification with the full gesture set, and classification with multiple
subsets of the gesture vocabulary.

###  7.1 IR Activation Segmentation Comparison

In this section, we compare the predictive ability of gestures collected
requiring that the IR sensor be activated versus not requiring the IR sensor
to activate to segment gestures. Recall that these data sets are collected
using separate experiments and different users. In each scenario, we train the
models using leave-one-subject-out cross validation. That is, no subject’s
data is simultaneously used for training and testing. For our evaluation
metric, we choose the average true positive rate per gesture. Because class
imbalance exists, accuracy is not a good indicator of performance as classes
that occur less often will receive less weight in the evaluation. Moreover,
binary scores like recall and precision are harder to interpret when micro or
macro averaged. Per-class true positive rate, alternatively, captures how well
we perform for each gesture. For this analysis, we choose to use the random
forest baseline model, as it is the best performing baseline model (discussed
later). Table I describes the per-class true positive rate for requiring
versus not requiring the IR sensor to be activated for the random forest
model. The main conclusion we draw from Table I is that requiring the IR
sensor to activate does increase the performance of the AirWare algorithm.
Moreover, there are other advantages for requiring that the sensor be
activated such as reducing false positives and reducing needless computation.
This is because the audio Doppler signal is likely to result in a number of
false positives from movement by the user and near the user; whereas the IR
sensor is relatively robust to these types of noise. However, requiring that
the IR sensor be activated also requires users to manipulate the way they
perform in-air gestures to activate the sensor. In this way, the AirWare
system will likely require some instruction to the users for how to reliably
perform different gestures. Thus, in the remainder of our analysis we only use
the gesture set that requires IR activation to segment gestures.

TABLE I: Average true positive rate per class for differing IR sensor activation. | Per Class True Positive Rate
---|---
IR Sensor Activation | Average | STD Error
Required, N=13 | 38.92% | 0.01
Not Required, N=8 | 13.71% | 0.02
Majority | 5.34% | N/A
Chance | 4.76% | N/A

###  7.2 IR Activation and Doppler Signatures

Figure 6: Average true positive rate per gesture per user comparison for using
only IR activation, only Doppler signatures and combined sensor information to
classify the 21 class gesture set.

In this section, we analyze the performance of using IR sensor data only,
Doppler data only, and the combined sensors. With this analysis we seek to
understand how advantageous it is to combine the sensors. To investigate this
research question, our network is modified to use only the IR activation
information or only the Doppler signature information. From Figure 5 this
corresponds to only using one of the input branches in the network. The
performance of the model trained using individual sensor information is
compared with the performance of the model trained using the combined sensor
information. The cross validation strategy used in all cases is leave-one-
subject-out, wherein we train the model on $N-1$ users’ data and test the
model on the $N$th user data. If we only look at the best performing models
from Figure 6, we see that we are able to achieve an average true positive
rate of 35% per class with a standard error of 0.02% when only the IR sensor
information is used. Note that model 2 and 3 are identical when only using the
IR signal branch. In comparison, using audio Doppler only sensor information
results in a best performing model with average true positive rate of 13% with
a standard error of 0.03%. From Figure 6, we can see that the performance with
only IR information is better than performance of using Doppler only
regardless of the machine learning model employed. However, when we combine
the information from the two modalities, the performance improves for all
convolutional neural network models and the random forest model. Thus, we
conclude that combining the two sensing modalities is advantageous for in-air
gesture recognition, resulting in a performance increase of about 10% average
true positive rate per gesture. The improvements are statistically significant
based upon a two-tailed T-test ($p<0.01$). In all analyses in the remainder of
the paper, we use the combined Doppler and IR sensor modalities as features
for the machine learning models.

Figure 7: Overview of three different cross-validation strategies invetigated.

###  7.3 Full 21-Gesture Vocabulary

We now investigate the performance of the baseline machine learning models as
well as different convolutional neural network architectures described in the
previous section using different cross-validation strategies. We analyze the
performance of the classifiers through the following cross validation
strategies, each with its own practical implications. Also, an overview of
each cross validation strategy is shown in Figure 7.

Figure 8: Average true positive rate per gesture per user comparison after
combining the IR activation and Doppler signatures for all baseline and
convolutional neural network models

####  7.3.1 Leave One User Out

We explore the performance of our classifiers using ‘leave one subject out’
cross validation strategy. The strategy used in this case is to train the
model on data from $N-1$ users and test it on the $N$th user, as described in
Figure 7. This approach analyzes whether we can classify the gestures
successfully without requiring the system to be calibrated. This implies that
for practical implementations, we can directly use a pre-trained, out-of-the-
box classifier to classify the gestures. Through this strategy, we try to
generalize the learning of our classifier across different types of users.
This is the ideal scenario for a gesture system, requiring no user input or
calibration before use. As can been seen from Figure 8, average true positive
rate per gesture per user ranges from approximately 19% to 46% for different
classifiers. Our best performing model in this case is ’Model 3’ which is a
deeper network in terms of the number of convolutional layers and dense
layers.

Breaking down the performance of Model 3 by users Figure 9 we see that our
network performs well, except for users 1, 3, and 4. We are able to achieve an
average true positive rate per gesture per user of 45.19% with a standard
error of 0.02%. We conclude that the performance of the leave-one-user-out
model is not sufficient for a gesture recognition system. Therefore, we
explored other cross validation strategies that assume a calibration phase is
employed.

####  7.3.2 Personalized Model for Each User

In this analysis, we analyze the performance of our classifier by calibrating
the model to each user. For each user, we perform a 5-iteration 60% training
and 40% testing stratified, shuffled split, as described in Figure 7. The test
size is chosen to make sure there are at least 2 instances of each class are
present in the test data [pedregosa2011scikit]. In effect, we train 13
independent models using only data collected from a specific individual for
training and testing. We would like to see if the variation in gesture
performance within the user is able to predict the classes successfully. This
cross validation mirrors a use case where users would need to provide example
gestures to the system as a calibration phase. This is less ideal in terms of
practical usage but may be necessary to increase performance. From Figure 8,
we see that, on average, the performance deteriorates for all the models as
compared to leave one subject out. In this case, Random Forest performs
equally well compared to ’Model 3’ at approximately 24% average true positive
rate per gesture per user. From Figure 9, we can see that none of the users
benefit from a fully personalized model when compared with ‘leave one subject
out’ performance. We are able to achieve an average true positive rate of
23.68% per class with a standard error of 0.03% across users for the 21-class
gesture set. It is unclear, however, if the personalized models do not perform
consistently because the training data is limited. Convolutional networks tend
to require large amounts of training data to perform well, so it is possible
that the decrease in performance is due to a significantly smaller training
set. This motivates us to combine our two cross validation strategies in order
to increase the amount of training data, but also employ a personalized
calibration procedure.

Figure 9: Average true positive rate per gesture per user comparison for the
best performing architecture, Model 3. Results are shown for all employed
cross validation strategies.

####  7.3.3 User Calibrated Model

In this cross validation strategy, we combine knowledge from the previous two
strategies to test the performance of the model. We first split the data based
on the user; $N-1$ users’ data in the training set. From the $N^{th}$ user
data, we perform a 5-fold 60%-40% stratified shuffle split, as done for the
personalized model. We then combine the training data from the $N-1$ users
with the 60% split of training data from the $N^{th}$ user and use the
remaining 40% of data from the $N^{th}$ user as a testing set, as shown in
Figure 7. The model performance for each user improves significantly for all
users with this training strategy. Thus, the model learns from other users as
well as the test user to classify the gestures of the test user. Note that
this training strategy, like the fully personalized model, assumes that a
calibration procedure will occur for each user of AirWare. From Figure 9, we
see that model 3 is the best performing amongst all the models. We are able to
achieve an average true positive rate of 50.45% per class per user with a
standard error of 0.03% across users for the 21-class gesture set.

If we investigate the most common confusions, we see that click gets
misclassified as double-click 26% of the times, pan down gets misclassified as
flick down (18%) and pan right gets misclassified as flick right (21%),
suggesting a close similarity between their Doppler signatures and IR
activations. The best performing gestures are erase, pans, and snap which have
average true positive rates of 89%, 72%, and 72%, respectively.

Finally, we also wish to understand about how much training data is required
before the performance of the different models begins to saturate. That is,
about how many calibration examples are required before the performance
plateaus? To investigate this question we look at the training curves for
‘Model 3’ and ‘Random Forest’ since these are the best performing models.

Figure 10: Training Curve for Model 3 and Random Forest using ‘User-
calibrated’ cross-validation strategy

Figure 10 shows the performance of ‘Model 3’ and ‘Random Forest’ as we
gradually increase the percentage of calibration data from 10% to 50%. We
increase the training size by 10% and evaluated the models using the remaining
data from the user not used in calibration. As we can see in Figure 10, both
‘Model 3’ and the ‘Random Forest’ model gradually increase performance as more
user-specific calibration data is added. Moreover, both models begin to
saturate between 30% and 50% of training data used from the user. If we assume
saturation is achieved at 50%, this corresponds to the system needing 2–3
examples of each gesture from the user during calibration.

From the above results, we can clearly see that fusing the two different
sensing modalities allows increased performance as compared to only using the
individual sensor information. However, the overall performance of 50.45% is
still dramatically less than what would be needed for a practical gesture
recognition system. We conclude that we cannot support the full 21 gesture
vocabulary at a given time. However, it may be possible to select subsets of
the gestures from the full vocabulary. Thus, we explore what simplifications
to the vocabulary can be made to increase the per gesture true positive rate
to a point of usability.

###  7.4 Using Feature Subsets

We now seek to understand if the performance of the classifier can be improved
by reducing the simultaneous number of gestures that a classifier must
distinguish for a given application. In this scenario, we wish to divide the
gestures into smaller subsets based upon what combinations of gestures are
most appropriate for different categories of applications. We assume that the
application in question somehow instructs the user of what gestures are
currently supported. If a user were to perform an unsupported gesture, the
system would misinterpret that gesture. Sub-setting the gesture set, we
enumerate 4 different categories: Generic, Mapping, and Gaming. Each gesture
set comprises 4 to 7 gestures. Together, these categories include 16 distinct
gestures. Thus, the vocabulary is large, but managed by never having more than
7 gestures available at a time. We test the performance of the model for these
reduced gesture sets using ‘user calibrated’ strategy discussed above. We also
employ the most accurate architecture as selected through previous analysis
and parameters remain the same as from our previous hyper-parameter tuning.
Confusion matrices are generated in the same manner as previously discussed. A
summary of the different reduced sets overall and per user appears in Figure
11. As shown, there are a number of users for which the system works well for
and a number of individuals that it does not always achieve high true positive
rate. In particular, users 1, 3, and 4 have reduced recognition rates compared
to other users. Upon review of the data, these also corresponded to users that
did not perform many practice trials while learning the gestures. These users
only practiced the gestures one time compared to other participants performing
gestures multiple times before they reported that they were ready to start the
experiment. As such, these participants may have rushed through the learning
of the gestures or not taken the experiment as seriously as others.

Figure 11: Performance comparison across users for different reduced gesture
sets Figure 12: Aggregate confusion matrix for user calibrated performance of
the reduced gesture sets.

Generic is comprised of a total of 7 gestures: double-tap, flicks
up/down/left/right, snap and erase. This set is most likely to be used by
interfaces that require generic up/down/left/right interactions as well as
some selection and undo interactions such as when interacting with a web
browser. With these 7 gestures, AirWare is able to achieve an average true
positive rate of 82.4% per gesture across users with a standard error of 0.02%
(Figure 11). Inherently, the double tap and snap gestures are performed in a
similar fashion, which accounts for most confusions. From Figure 12, we can
see the model confuses between these two gestures often. Similarly, we can see
that flick up and flick right are confused by the model. This is likely the
result of these gestures being performed quickly or sloppily by
participants—resulting in a number of ‘diagonal’ motions rather than
explicitly downward or leftward motions. Because of this, it might be
warranted to limit flick interactions to only left/right or only up/down.
Furthermore eliminating the snap gesture would help increase the true positive
rate of the double tap gesture.

Mapping set is focused on more immersive applications such as maps where
zooming and panning are core requirements. It consists of Zoom in/away, Pan
up/down/left/right and erase (a total of 7 gestures). We are able to achieve
an average true positive rate of 82.9% with a standard error of 0.02. (See
Figure 11.) From Figure 12, we can see the model confuses between pan down
with pan left or pan right gestures the most. This is a similar phenomenon to
users performing flicks at diagonal angles rather than directly lateral to the
phone. Even so, most pans are classified accurately.

Gaming set consists of snap, slice left/right and Whip (a total of 4 gestures)
focusing on specialized applications like gaming. We are able to achieve an
average true positive rate of 86.5% per gesture across users with a standard
error of 0.03%. (See Figure 11.) From Figure 12, we can see the model confuses
between snap and whip gestures the most, but mostly all gestures are
classified accurately.

The reduced gesture sets can be combined in different scenarios and for
different applications to achieve a gesture vocabulary of 16 unique in-air
gestures with about 80% or better average true positive rate per gesture. Many
gestures surpass 90% true positive rate. The reduced gesture sets presented
here are only an example of possible subsets. Depending on the needs of the
application, a number of reduced sets could be deployed by the AirWare system
for different usage scenarios to support an even larger vocabulary. With this
level of performance, we believe AirWare could be used in modern smartphones
for a number of application scenarios. However, the full 21 gesture vocabulary
is not accurate enough to be deployed by a gesture recognition system.

##  8 Discussions and Future Work

Despite the good performance of AirWare, there are limitations in our study
that we wish to specifically mention. First, our evaluation does not
incorporate any specific user interface task. That is, the gesture performed
by the participants were in random order and did not have an action task
associated to them. It might be possible that when users employ these gestures
for specific tasks, the way they are performed may alter in comparison to our
user study. Also taking into consideration that the gestures were recorded in
a relatively controlled setting, the performance can depreciate in real-life
environments having external acoustic disturbances or when the phone is held
in different positions. Even so, previous Doppler based research has shown the
method to be robust to many acoustic environments [gupta2012soundwave].

We have shown that user specific calibration significantly boosts
classification performance. However, this limits the scalability of our
approach because it requires users to provide calibration examples. Once
calibration examples are collected, the architecture must be retrained. This
retraining is limited by the current computing power of smart-phones.
Considering the complexity of algorithm, calibration would require cloud
support to be realistic. Even so, once trained, the neural network
architecture, short-time Fourier transform, and IR values are computationally
efficient and easy to compute from the smart-phone in real-time. The battery
considerations for such an implementation are not too much of a concern
because gestures only need to be processed when the IR sensor is activated.
This helps to further reduce the computational cost of the AirWare approach.
Even so, because we require the IR sensor to be activated, users may feel like
some gestures in the vocabulary are awkward or unintuitive to perform. This
limitation would also require that users are instructed on how to perform each
of the gestures and then the system would need to verify that the user could
complete the gestures reliably. While a limitation, we foresee this process as
something that could be incorporated into the calibration phase of the system.

We would also like to point out the problems with the Samsung S5 gesture
sensing API. Samsung has deprecated support for the device and access to the
sensor output is limited. Moreover, there is no way to access the raw sensor
values without rooting the phone. This limits the impact of our current
approach to pervade the current market, but doesn’t limit the research
contribution. This deprecation did affect our user study. Because the sensor
API was deprecated, many of the angle and velocity measures were flagged
“unknown.” We removed those incomplete records from our dataset but the
reliability of the sensor reading is called into question. As such, our
results might represent a lower bound of performance and may be further
increased with more reliable sensor readings or more expressive IR sensor
data.

Finally, we have not investigated user adoption of our vocabulary set nor have
we investigated impact of our large gesture vocabulary. We leave these
limitations to future work.

##  9 Conclusions

In conclusion, we presented AirWare, a technology that fuses the output of an
embedded smart-phone microphone and proximity sensor to recognize a gesture
set of 21 in-air hand-gestures with 50.47% average true positive rate per
gesture. While we show that combining two different sensor information streams
can significantly increase performance, we conclude that the full 21 gesture
vocabulary cannot be reliably classified for use in a deployed gesture
recognition system. However, AirWare can achieve a reliable true positive rate
per gesture for a number of reduced vocabulary gesture sets. In particular,
AirWare can achieve true positive rates of greater than 80% true positive rate
for Generic, Mapping, and Gaming gesture sets. Using these gesture sets,
AirWare can reliably classify a vocabulary of 16 unique gestures, with 4–7
gestures supported at any given time.

## Acknowledgments

The authors would like to thank students Rowdy Howell and Arya McCarthy for
their contributions in developing the AirWare mobile data collection
application.

Nibhrat Lohia completed his Masters in Statistics from the Dedman College of
Humanities and Sciences, Southern Methodist University and is currently
working as a Data Scientist in Copart, Inc. His research interests lie in the
application areas of deep neural nets and machine learning in general.
---
Raunak Mundada graduated with a Masters degree in Applied statistics and data
analysis from Southern Methodist University. His research interest lies in
machine learning and applied statistics. He is currently working as a
Statistician at GM Financial.
---
Eric C. Larson is an Assistant Professor in the department of Computer Science
and Engineering in the Bobby B. Lyle School of Engineering, Southern Methodist
University. His main research interests are in machine learning, sensing, and
signal/image processing for ubiquitous computing applications. He received his
Ph.D. in 2013 from the University of Washington. He received his B.S. and M.S.
in Electrical Engineering in 2006 and 2008, respectively, at Oklahoma State
University.
---


CRaWl [67] | 30.75 $\scriptstyle\pm$0.20 | 29.86 $\scriptstyle\pm$0.25 | 63.66 $\scriptstyle\pm$0.56 | 70.25 $\scriptstyle\pm$0.52
AgentNet | 26.22 $\scriptstyle\pm$0.26 | 25.49 $\scriptstyle\pm$0.27 | 67.24 $\scriptstyle\pm$0.65 | 72.33 $\scriptstyle\pm$0.62

Table 6: Test and validation average precision (%) on the OGB-MolPCBA dataset
and accuracy (%) on OGB-PPA datasets. We were unable to train ESAN model on
OGB-PPA dataset, as a machine with 256GB of main memory did not have enough
RAM to pre-process the graphs for training (OOM). Note that the pre-processed
graphs for the much smaller DD dataset (Table 4) already took 68GB of space on
disk and required 120GB of main memory to load.

We also test our model on a popular ZINC-12K molecular graph regression
dataset [24; 40], where the models need to predict the constrained solubility
of the molecule. While this property is more defined by the global structure
of the molecule, so might not align very well with AgentNet inductive bias
(Appendix F), some local features, like cycle counts are important for this
metric [24]. Originally Dwivedi et al. [24] recommended constraining the
models to have either 100K or 500k parameters for this benchmark (not all
baselines state that they follow this). Since our model is less parameter
efficient, compared to a standard GNN, due to the use of multiple MLPs for
every step and concatenation of various embeddings as input to those MLPs, we
chose to use the 500k parameter budget. This corresponds to an embedding size
of $88$. In Table 7 we can see that this model performs comparably to GIN. If
we instead would use an embedding size of $128$, which would correspond to
around 100k parameters for a traditional GNN architecture, such as GIN,
AgentNet outperforms quite a few of the expressive baselines [51; 4; 71; 28;
8; 11; 9; 67] and half of the different ESAN [8] configurations which use
different sub-graph construction techniques. Some of the baselines depend on
explicitly pre-computed structural features, such as cycles [4; 28; 11; 9]
which are known to be important in this task [24]. It is also worth noting
that for some models such as GIN the performance on this task becomes worse
when the parameter count is increased [24].

Model | ZINC (MAE)
---|---
PPGN [51] | 0.256 $\scriptstyle\pm$0.054
GIN [75] | 0.252 $\scriptstyle\pm$0.017
PNA [18] | 0.188 $\scriptstyle\pm$0.004
DGN [4] | 0.168 $\scriptstyle\pm$0.003
HIMP [28] | 0.151 $\scriptstyle\pm$0.006
SMP [71] | 0.138 $\scriptstyle\pm$?
ESAN (Edge Dropout) [8] | 0.172 $\scriptstyle\pm$0.005
ESAN (Node Dropout) [8] | 0.166 $\scriptstyle\pm$0.004
ESAN (Ego Graphs) [8] | 0.107 $\scriptstyle\pm$0.005
ESAN (Ego Graphs + Root ID) [8] | 0.102 $\scriptstyle\pm$0.003
GSN [11] | 0.108 $\scriptstyle\pm$0.018
CIN-small [9] | 0.094 $\scriptstyle\pm$0.004
CIN [9] | 0.079 $\scriptstyle\pm$0.006
CRaWl [67] | 0.085 $\scriptstyle\pm$0.004
AgentNet | 0.258 $\scriptstyle\pm$0.033
AgentNet (Embedding size = 128) | 0.144 $\scriptstyle\pm$0.016

Table 7: Mean absolute error on the ZINC 12k dataset.

For the OGB experiments, we used the best hyperparameters from OGB-MolHIV
(batch size of 64, 128 hidden units) and $k=26$ agents with $\ell=16$ steps
for OGB-MolPCBA, while $k=150$ agents with $\ell=8$ steps were used for OGB-
PPA, to account for larger graph size and memory constraints. As standard,
results are reported over 10 random seeds [38]. For ZINC we used a batch size
of $128$, $\ell=16$ steps, $k=37$ agents, and either $90$ or $128$ hidden
units and train it for $2000$ epochs. In this case, we use 4 random seeds to
report the results, following Dwivedi et al. [24].

##  Appendix I Importance of different model steps

In this section, we experimentally verify, that all of the model steps as
described in Section 3 and Figure 1 are necessary. Besides testing Random Walk
AgentNet, in Table 8 we also check how removing the node update step
(AgentNet, No Node Update) or neighborhood update step (AgentNet, No
Neighborhood Update) affects the model’s ability to solve hard synthetic
tasks. An important thing to note here is that normally, only nodes that have
an agent on them perform the neighborhood update step. Meaning, that even if
we remove the node update, there is still information accessible to the model
about which neighboring nodes and how many times were visited, but its unknown
by which agent. To account for this we also provide a model (AgentNet, No Node
Update, and Neighborhood Update For All), which skips the node update step and
performs neighborhood update for all nodes, not just ones that have an agent
on them. As you can see in Table 8 this model is no longer expressive, while
the other two versions do still have much better than random accuracy.
However, we see that their predictions are no longer perfect. When we do not
use the neighborhood update it becomes harder for the model to observe the
structural information, especially if we want to find cliques, for which we
need to check the neighborhood (Figure 1 (d)). Interestingly, AgentNet with no
node update does quite well on all tasks, as visit counts can act as a
surrogate for random features. Although performance is slightly worse than
that of GIN with random features on all but the 2-WL task. In Table 9 we can
see that these performance differences also transfer to the real-world OGB-
MolHIV graph classification task. It is worth noting that random walk AgentNet
achieves quite a close result to the full AgentNet. The reasons for this can
be twofold: 1) as we have seen in Table 1 this formulation is able to
recognize cycles, which are one of the most important structures in molecules;
2) as we have $k\approx n$ agents the whole graph can be explored even with
random walks.

Model |  $4$-cycles [59] |  Circular Skip Links [15] | 2-WL
---|---|---|---
GIN [75] | 50.0 $\scriptstyle\pm$0.0 | 10.0 $\scriptstyle\pm$0.0 | 50.0 $\scriptstyle\pm$0.0
GIN with random features [64; 1] | 99.7 $\scriptstyle\pm$0.4 | 95.8 $\scriptstyle\pm$2.1 | 92.4 $\scriptstyle\pm$1.6
Random Walk AgentNet | 100.0 $\scriptstyle\pm$0.0 | 100.0 $\scriptstyle\pm$0.0 | 50.5 $\scriptstyle\pm$4.5
Simplified AgentNet | 100.0 $\scriptstyle\pm$0.0 | 100.0 $\scriptstyle\pm$0.0 | 100.0 $\scriptstyle\pm$0.0
AgentNet, No Neighborhood Update | 75.8 $\scriptstyle\pm$17.9 | 40.6 $\scriptstyle\pm$5.8 | 51.2 $\scriptstyle\pm$2.0
AgentNet, No Node Update | 92.4 $\scriptstyle\pm$9.7 | 89.8 $\scriptstyle\pm$8.9 | 98.9 $\scriptstyle\pm$0.6
AgentNet, No Node Update and Neighborhood Update For All | 50.0 $\scriptstyle\pm$0.0 | 10.0 $\scriptstyle\pm$0.8 | 49.7 $\scriptstyle\pm$1.2
AgentNet | 100.0 $\scriptstyle\pm$0.0 | 100.0 $\scriptstyle\pm$0.0 | 100.0 $\scriptstyle\pm$0.0

Table 8: Ablation on how different AgentNet modules influence expressiveness
on datasets unsolvable by 1-WL (GIN). Removing any of the AgentNet steps as
described in Section 3 reduces models expressiveness.

| OGB-MolHIV
---|---
Model | Validation | Test
GIN [75] | 82.32 $\scriptstyle\pm$0.90 | 75.58 $\scriptstyle\pm$1.40
GIN \+ virtual node [75] | 84.79 $\scriptstyle\pm$0.68 | 77.07 $\scriptstyle\pm$1.49
ESAN [8]* | 84.28 $\scriptstyle\pm$0.90 | 78.00 $\scriptstyle\pm$1.42
Random Walk AgentNet | 85.61 $\scriptstyle\pm$1.29 | 77.89 $\scriptstyle\pm$1.29
AgentNet, No Neighborhood Update | 84.06 $\scriptstyle\pm$0.76 | 75.41 $\scriptstyle\pm$1.41
AgentNet, No Node Update | 84.70 $\scriptstyle\pm$1.40 | 76.71 $\scriptstyle\pm$1.37
AgentNet, No Node Update and Neighborhood Update For All | 84.40 $\scriptstyle\pm$0.89 | 75.47 $\scriptstyle\pm$0.85
AgentNet | 84.77 $\scriptstyle\pm$0.92 | 78.33 $\scriptstyle\pm$0.69

Table 9: Ablation on how different AgentNet modules influence test and
validation ROC-AUC (%) on the OGB-MolHIV dataset. *Best result achieved by any
version.

For synthetic tasks in Table 8 we used the same experimental setup as before
(Appendix D.1) with a grid search over the hyperparameters, while for OGB-
MolHIV in Table 9, we used the best hyperparameters of the full AgentNet
(Appendix D.2).

##  Appendix J Selecting the number of agents

Figure 5: AgentNet ROC-AUC (%) on the OGB-MolHIV when using different number
of agents (mean $n=26$). Gray line marks the mean number of nodes in the
dataset.

One of the hyperparameters that could be tuned for the model is the number of
agents. To avoid an extensive grid search we normally set the number of agents
to the mean number of nodes in the dataset. Besides matching the computational
complexity of traditional GNNs, one reason for this is that we want that most
of the nodes in the graph would be visited by at least some agent. Another
reason is that node embedding vectors, where agents record information are of
finite size. If we have many more agents than nodes (e.g. a $1000$ agents on a
$10$ node graph) there could be a negative interplay between the agents, where
they would corrupt the information stored on nodes by other agents. Choosing
to use around $n$ agents helps to avoid such negative interplay. As could be
seen from Figure 2 (d), AgentNet performance is stable over a wide range of
step and agent counts, but ideally, the model should visit the majority of the
nodes if one wants to maximize accuracy, while further step and agent count
increase beyond this does not bring much benefit. Here in Figure 5 we also
provide an ablation on the effects the number of agents has on AgentNet ROC-
AUC on OGB-MolHIV dataset, when using the best hyperparameters from the
original experiment (Appendix D.2). We again see that having around $n$ agents
is a good choice and that in fact, having $2n$ or $4n$ agents results in a
slightly lower performance than when using less than $\frac{n}{2}$ agents.
This again suggests that having only a fraction of the nodes active at a given
time step can be a simple way to reduce the computational burden.

##  Appendix K Important subgraphs

As we have discussed in the main paper, one of the potential benefits of
AgentNet is that the agents can learn to prioritize important substructures.
Here we investigate the MUTAG dataset [54], which is comprised of mutagenic
and non-mutagenic molecules. It is known that a good indicator that a molecule
is mutagenic is the presence of NO2 groups [21]. This dataset was constructed
such that all molecules have at least one NO2 group, however, mutagenic
molecules still tend to have more such groups. In Figure 6 we can indeed see
that agents in AgentNet indeed visit such informative subgroups more often.

Figure 6: AgentNet node visitation heat map on test graphs from the MUTAG
dataset. The brighter the color, the more often the given node has been
visited. Agents prioritize some important substructures (NO2) when they move
around the graph.