| 60 | $0(0)$ | $0(0)$ | $1$ | $1$ | $0(0)$ | $0(0)$ | $1$ | $1$ | $0.001$ | $0.001$ | $1$ | $1$ | $0.004$ | $0.004$ | $1$ | $1$
| 80 | $0(0)$ | $0(0)$ | $1$ | $1$ | $0(0)$ | $0(0)$ | $1$ | $1$ | $0.001$ | $0.001$ | $1$ | $1$ | $0.004$ | $0.004$ | $1$ | $1$
| 100 | $0(0)$ | $0(0)$ | $1$ | $1$ | $0(0)$ | $0(0)$ | $1$ | $1$ | $0.001$ | $0.001$ | $1$ | $1$ | $0.004$ | $0.004$ | $1$ | $1$
Mixture | 1 | $0(0)$ | $0(0)$ | $1$ | $1$ | $0(0)$ | $0(0)$ | $1$ | $1$ | $0.06$ | $0.01$ | $0.93$ | $0.99$ | $0.004$ | $0.004$ | $1$ | $1$
| 5 | $0(0)$ | $0(0)$ | $1$ | $1$ | $0(0)$ | $0(0)$ | $1$ | $1$ | $0.001$ | $0.001$ | $1$ | $1$ | $0.004$ | $0.004$ | $1$ | $1$
| 10 | $0(0)$ | $0(0)$ | $1$ | $1$ | $0(0)$ | $0(0)$ | $1$ | $1$ | $0.001$ | $0.001$ | $1$ | $1$ | $0.004$ | $0.004$ | $1$ | $1$
| 20 | $0(0)$ | $0(0)$ | $1$ | $1$ | $0(0)$ | $0(0)$ | $1$ | $1$ | $0.001$ | $0.001$ | $1$ | $1$ | $0.004$ | $0.004$ | $1$ | $1$
| 40 | $0(0)$ | $0(0)$ | $1$ | $1$ | $0(0)$ | $0(0)$ | $1$ | $1$ | $0.001$ | $0.001$ | $1$ | $1$ | $0.004$ | $0.004$ | $1$ | $1$
| 60 | $0(0)$ | $0(0)$ | $1$ | $1$ | $0(0)$ | $0(0)$ | $1$ | $1$ | $0.001$ | $0.001$ | $1$ | $1$ | $0.004$ | $0.004$ | $1$ | $1$
| 80 | $0(0)$ | $0(0)$ | $1$ | $1$ | $0(0)$ | $0(0)$ | $1$ | $1$ | $0.001$ | $0.001$ | $1$ | $1$ | $0.004$ | $0.004$ | $1$ | $1$
| 100 | $0(0)$ | $0(0)$ | $1$ | $1$ | $0(0)$ | $0(0)$ | $1$ | $1$ | $0.001$ | $0.001$ | $1$ | $1$ | $0.004$ | $0.004$ | $1$ | $1$
Variance shift | 1 | $0.01(0)$ | $0(0)$ | $1$ | $1$ | $1.73(0.59)$ | $2.10(0.59)$ | $0.93$ | $0.81$ | $0.07$ | $0.01$ | $0.93$ | $0.99$ | $0.006$ | $0.004$ | $0.99$ | $1$
| 5 | $0.42(0.07)$ | $0.40(0.08)$ | $0.99$ | $1$ | $4.42(0.72)$ | $7.30(0.70)$ | $0.73$ | $0.64$ | $0.001$ | $0.001$ | $1$ | $1$ | $0.004$ | $0.004$ | $1$ | $1$
| 10 | $0.39(0.06)$ | $0.22(0.06)$ | $1$ | $1$ | $8.69(0.66)$ | $13.12(0.73)$ | $0.55$ | $0.40$ | $0.001$ | $0.001$ | $1$ | $1$ | $0.004$ | $0.004$ | $1$ | $1$
| 20 | $0(0)$ | $0(0)$ | $1$ | $1$ | $13.43(0.78)$ | $18.12(0.69)$ | $0.35$ | $0.07$ | $0.001$ | $0.001$ | $1$ | $1$ | $0.004$ | $0.004$ | $1$ | $1$
| 40 | $0(0)$ | $0(0)$ | $1$ | $1$ | $18.01(0.68)$ | $19.82(0.68)$ | $0.11$ | $0$ | $0.001$ | $0.001$ | $1$ | $1$ | $0.004$ | $0.004$ | $1$ | $1$
| 60 | $0(0)$ | $0(0)$ | $1$ | $1$ | $19.19(0.55)$ | $19.98(0.94)$ | $0.02$ | $0$ | $0.001$ | $0.001$ | $1$ | $1$ | $0.004$ | $0.004$ | $1$ | $1$
| 80 | $0(0)$ | $0(0)$ | $1$ | $1$ | $19.64(0.47)$ | $20(1)$ | $0$ | $0$ | $0.001$ | $0.001$ | $1$ | $1$ | $0.004$ | $0.004$ | $1$ | $1$
| 100 | $0(0)$ | $0(0)$ | $1$ | $1$ | $19.82(0.64)$ | $20(1)$ | $0$ | $0$ | $0.001$ | $0.001$ | $1$ | $1$ | $0.004$ | $0.004$ | $1$ | $1$
Heavy tail | 1 | $0.05(0)$ | $0(0)$ | $1$ | $1$ | $1.65(0.54)$ | $1.70(0.54)$ | $0.96$ | $0.99$ | $0.03$ | $0.004$ | $0.96$ | $0.99$ | $0.01$ | $0.005$ | $0.98$ | $0.99$
| 5 | $0.04(0)$ | $0.02(0)$ | $1$ | $1$ | $2.89(0.71)$ | $4.53(0.74)$ | $0.91$ | $0.76$ | $0.001$ | $0.001$ | $1$ | $1$ | $0.004$ | $0.004$ | $1$ | $1$
| 10 | $0(0)$ | $0(0)$ | $1$ | $1$ | $4.49(0.78)$ | $7.87(0.73)$ | $0.78$ | $0.64$ | $0.001$ | $0.001$ | $1$ | $1$ | $0.004$ | $0.004$ | $1$ | $1$
| 20 | $0(0)$ | $0(0)$ | $1$ | $1$ | $5.66(0.76)$ | $11.73(0.75)$ | $0.77$ | $0.42$ | $0.001$ | $0.001$ | $1$ | $1$ | $0.004$ | $0.004$ | $1$ | $1$
| 40 | $0(0)$ | $0(0)$ | $1$ | $1$ | $9.40(0.79)$ | $16.41(0.78)$ | $0.54$ | $0.20$ | $0.001$ | $0.001$ | $1$ | $1$ | $0.004$ | $0.004$ | $1$ | $1$
| 60 | $0(0)$ | $0(0)$ | $1$ | $1$ | $11.02(0.74)$ | $18.06(0.82)$ | $0.52$ | $0.16$ | $0.001$ | $0.001$ | $1$ | $1$ | $0.004$ | $0.004$ | $1$ | $1$
| 80 | $0(0)$ | $0(0)$ | $1$ | $1$ | $12.53(0.77)$ | $18.51(0.90)$ | $0.41$ | $0.09$ | $0.001$ | $0.001$ | $1$ | $1$ | $0.004$ | $0.004$ | $1$ | $1$
| 100 | $0(0)$ | $0(0)$ | $1$ | $1$ | $13.17(0.75)$ | $19.07(0.97)$ | $0.30$ | $0.06$ | $0.001$ | $0.001$ | $1$ | $1$ | $0.004$ | $0.004$ | $1$ | $1$
Kurtosis | 1 | $0(0)$ | $0(0)$ | $1$ | $1$ | $1.23(0.42)$ | $1.55(0.52)$ | $0.96$ | $0.95$ | $0.002$ | $0.001$ | $0.99$ | $1$ | $0.004$ | $0.004$ | $1$ | $1$
| 5 | $0(0)$ | $0(0)$ | $1$ | $1$ | $1.75(0.59)$ | $3.54(0.70)$ | $0.96$ | $0.88$ | $0.001$ | $0.001$ | $1$ | $1$ | $0.004$ | $0.004$ | $1$ | $1$
| 10 | $0(0)$ | $0(0)$ | $1$ | $1$ | $2.81(0.66)$ | $6.41(0.76)$ | $0.94$ | $0.75$ | $0.001$ | $0.001$ | $1$ | $1$ | $0.004$ | $0.004$ | $1$ | $1$
| 20 | $0(0)$ | $0(0)$ | $1$ | $1$ | $4.63(0.71)$ | $9.90(0.78)$ | $0.84$ | $0.51$ | $0.001$ | $0.001$ | $1$ | $1$ | $0.004$ | $0.004$ | $1$ | $1$
| 40 | $0(0)$ | $0(0)$ | $1$ | $1$ | $5.70(0.73)$ | $13.43(0.77)$ | $0.74$ | $0.28$ | $0.001$ | $0.001$ | $1$ | $1$ | $0.004$ | $0.004$ | $1$ | $1$
| 60 | $0(0)$ | $0(0)$ | $1$ | $1$ | $7.06(0.75)$ | $16.38(0.81)$ | $0.72$ | $0.23$ | $0.001$ | $0.001$ | $1$ | $1$ | $0.004$ | $0.004$ | $1$ | $1$
| 80 | $0(0)$ | $0(0)$ | $1$ | $1$ | $8.11(0.79)$ | $17.50(0.83)$ | $0.71$ | $0.13$ | $0.001$ | $0.001$ | $1$ | $1$ | $0.004$ | $0.004$ | $1$ | $1$
| 100 | $0(0)$ | $0(0)$ | $1$ | $1$ | $8.83(0.78)$ | $18.52(0.89)$ | $0.55$ | $0.09$ | $0.001$ | $0.001$ | $1$ | $1$ | $0.004$ | $0.004$ | $1$ | $1$

###  F.2 The Semi-BNP GAN

Now, we examine the performance of the proposed GAN through additional
datasets, the details of which are given below. The generated samples are
shown in Figures 10. Generally, the generated images using semi-BNP GAN show
better resolution than the FNP GAN. The MMD scores presented in Table 3 are
also evidence to demonstrate this claim. To further assess the performance of
MMD-based GANs, we report the commonly used Fréchet inception distance (FID)
and the Kernel inception distance (KID) metrics (Bińkowski et al., 2018).
These metrics are well-suited for evaluating the performance of GANs. The
corresponding scores999The codes to compute the KID and FID are available at
https://github.com/mbinkowski/MMD-GAN/blob/master/gan/compute_scores.py. are
reported in Table 3. Similar to our MMD scores, the smaller values of FID and
KID show better performance of the GAN.

####  F.2.1 Bone Marrow Biopsy Dataset (Tomczak and Welling, 2016):

The bone marrow biopsy (BMB) dataset is a collection of histopathology of BMB
images corresponding to 16 patients with some types of blood cancer and
anemia: 10 patients for training, 3 for testing, and 3 for validation. This
dataset contains 10,800 images in the size of $28\times 28$ pixels, 6,800 of
which are considered for the training set. The rest of the images have been
divided into two sets of equal size for testing and validation. The whole
dataset can be found at
https://github.com/jmtomczak/vae_householder_flow/tree/master/datasets/histopathologyGray.
The results based on 6800 training images are presented in Figure 10-(a-c).

####  F.2.2 Labeled Faces in the Wild Dataset (Huang et al., 2008):

The labeled faces in the wild dataset (LFD) include 13,000 facial image
samples with 1,024 ($32\times 32$) dimensions. The dataset is available at
https://conradsanderson.id.au/lfwcrop/.

####  F.2.3 Brain Tumor MRI Dataset (Nickparvar, 2021):

In the last experiment, we consider a more challenging medical dataset
including brain MRI images available at https://www.kaggle.com/dsv/2645886.
This dataset has two groups including training and testing sets. Both are
classified into four classes: glioma, meningioma, no tumor, and pituitary. To
train the networks, we consider all 5,712 training images. The images vary in
size and have extra margins. We use a pre-processing
code101010https://github.com/masoudnick/Brain-Tumor-MRI-
Classification/blob/main/Preprocessing.py to remove margins and then resize
images to $50\times 50$ pixels. We also scale the pixel value of prepared
images to range 0-1 to make the range of distribution of feature values equal
and prevent any errors in the backpropagation computation.

(a) Training data (b) Semi-BNP-MMD GAN

(c) MMD-FNP GAN  (d) Training data

(e) Semi-BNP-MMD GAN  (f) MMD-FNP GAN

(g) Training data (h) Semi-BNP-MMD GAN

(i) MMD-FNP GAN

Figure 10: Generated samples of sizes ($6\times 6$) from semi-BNP-MMD and MMD-FNP GAN for the BMB and LFW datasets using a mixture of Gaussian kernels in 40,000 iterations. Table 3: The values of MMD, KID, and FID scores for four groups of datasets considering $n_{mb}=1000$ and $r_{mb}=1000$ in (24). Scores | Dataset
---|---
MNIST | BMB | LFW | MRI
Semi-BNP | FNP | Semi-BNP | FNP | Semi-BNP | FNP | Semi-BNP | FNP
MMD | $0.0384$ | $0.0404$ | $0.0285$ | $0.0315$ | $0.0281$ | $0.0302$ | $0.2059$ | $0.2231$
KID | $0.0034$ | $0.0046$ | $0.0030$ | $0.0036$ | $0.0019$ | $0.0026$ | $0.0260$ | $0.0264$
FID | $35.560$ | $37.934$ | $17.006$ | $17.264$ | $14.010$ | $14.473$ | $87.975$ | $87.831$

##  Appendix G More Discussion on the Potential Research

GANs are increasingly used in medical imaging applications which are effective
tools for tasks such as medical imaging reconstructions. The synthetic images
generated have often been proven to be valuable especially when the original
image is noisy or expensive to obtain. GANs have also been used for generating
images in cross-modality synthesis problems, where we observe magnetic
resonance imaging (MRI) for a given patient but want to generate computed
tomography (CT) images for that same patient (Wolterink et al., 2017). This
type of generative method for medical imaging can drastically reduce the time
and cost of obtaining data if the quality of the synthetic examples is
sufficiently high. GANs have also been used in a diagnostic capacity–for
example, in detecting brain lesions in images (Alex et al., 2017).

Here, the GAN is trained by distinguishing between labeled data of brain
images that contain and do not contain lesions. Then, the discriminator of the
GAN is used to detect brain lesions on new images. However, GANs are far less
commonly used for tasks like diagnosis. According to a survey on medical
imaging research in GANs, less than $10\%$ of the top papers surveyed were
dedicated towards making diagnoses, whereas the vast majority of papers were
dedicated towards generating realistic synthetic examples of medical images
for further analysis (Yi et al., 2019). We believe this is because where the
cost of making errors in diagnosis is immediately consequential to people,
unlike other AI applications where GANs are largely used.

We plan to extend the current work by mapping the data to a lower dimensional
space using an auto-encoder, a dimensionality reduction model helps to reduce
the noise in data and tries to optimize the cost function between the real
data and fake data in the code space. Then, we will propose a 3D semi-BNP GAN
in the code space to improve the ability of the GAN to generate medical
datasets. The auto-encoder method should further reduce the chance of mode
collapse and the 3D semi-BNP GAN will reduce the blurriness of the generated
samples that may be caused by using the auto-encoder. In future work, our
model will be able to generate 3D images and, hence, increase the resolution
of images, especially for MRI images. We hope that our future work will make
an impact in the field of medical imaging.

##  Appendix H Notations

Notation | Definition
---|---
$N(\cdot,\cdot)$ | Normal distribution
$LN(\cdot,\cdot)$ | Lognormal distribution
$t_{3}(\cdot,\cdot)$ |  $t$-distribution with 3 degrees of freedom
$LG(\cdot,\cdot)$ | Logistic distribution
$B_{d}$ |  $d\times d$ matrix with $0.25$ on the main diagonal and $0.2$ off the diagonal
$\mathbf{c}_{d}$ |  $d$-dimensional column vector of $c$’s
$I_{d}$ |  $d\times d$ identical matrix

  * •

In all distribution notations, the first component represents the mean vector
and the second component
represents the covariance matrix.


# WhisperWand: Simultaneous Voice and Gesture Tracking Interface

Yang Bai<EMAIL_ADDRESS>of Maryland College Park ,  Irtaza Shahid
<EMAIL_ADDRESS>of Maryland College Park ,  Harshvardhan Takawale
<EMAIL_ADDRESS>of Maryland College Park and  Nirupam Roy
<EMAIL_ADDRESS>of Maryland College Park

###### Abstract.

This paper presents the design and implementation of WhisperWand, a
comprehensive voice and motion tracking interface for voice assistants.
Distinct from prior works, WhisperWand is a precise tracking interface that
can co-exist with the voice interface on low sampling rate voice assistants.
Taking handwriting as a specific application, it can also capture natural
strokes and the individualized style of writing while occupying only a single
frequency. The core technique includes an accurate acoustic ranging method
called Cross Frequency Continuous Wave (CFCW) sonar, enabling voice assistants
to use ultrasound as a ranging signal while using the regular microphone
system of voice assistants as a receiver. We also design a new optimization
algorithm that only requires a single frequency for time difference of
arrival. WhisperWand prototype achieves 73 $\mu$m of median error for 1D
ranging and 1.4 mm of median error in 3D tracking of an acoustic beacon using
the microphone array used in voice assistants. Our implementation of an in-air
handwriting interface achieves 94.1% accuracy with automatic handwriting-to-
text software, similar to writing on paper (96.6%). At the same time, the
error rate of voice-based user authentication only increases from $6.26\%$ to
$8.28\%$.

##  1\. Introduction

Voice assistants (also commonly known as smart speakers) have become a common
household device. Nearly half (48%) of US internet users own one or more smart
speakers (vastatone, ) and global sales of such devices are likely to surpass
200 million annually (vastattwo, ). This proliferation gives rise to an
ecosystem of innovative acoustic sensing and perception capabilities around
these devices beyond traditional speech recognition. Researchers have
developed a spectrum of applications on voice assistants including health
monitoring (nandakumar2015contactless, ; wang2021using, ; zhang2020your, ),
gesture recognition (gupta2012soundwave, ; ruan2016audiogest, ), acoustic
analytics (ahmed2020preech, ; vaidya2019you, ), and environmental monitoring
(voloc, ; wang2021mavl, ).

Simultaneous motion tracking on these voice assistants can open up a wide
range of new applications. Precise motion tracking can be used as a
handwriting interface for nonvocal communication, two-factor authentication
using voice and signature, hand gesture can enable nonvocal interactions to
these ubiquitous devices for voice-impaired persons and elderly people.
Moreover, a simultaneous voice and gesture interface can be an authentication
channel using a library of pre-defined hand motions. However, existing
acoustic motion tracking studies cannot be applied on voice assistants that
can only support a limited audible bandwidth(e.g., Amazon Echo (0-8kHz)
(samplingrate, ; shen2020voice, ; chan2019contactless, ; taori2019targeted, ),
Google Nest Audio(0-10kHz) (googlenestaudio, ), Google Nest Mini (0-15.3kHz)
(googlenestmini, ), Sonos Move (0-18.5kHz) (sonosmove, )) due to two reasons:
(1) To achieve inaudibility, existing studies use the near-ultrasound band
(18-24kHz) for motion tracking, which is not supported by low-end voice
assistants. (2) Directly applying wide-band tracking on low-end devices
interferes with the voice interface. In this paper, we push the limit of
acoustic motion tracking accuracy to a few tens of micro-meter using a single
frequency, without any interference to the voice interface.

Figure 1. WhisperWand enables voice assistants to capture human voice and
track motions simultaneously on a voice assistant that can only support 0-8kHz
frequency band. The motion tracking only occupies a single frequency, without
interference to the human voice.

With the advent of smart devices, several techniques have been proposed to
enable a virtual motion tracking interface in the air (nandakumar2016fingerio,
; mao2016cat, ; zhang2017soundtrak, ; wang2019millisonic, ). While a lot of
applications are possible with these techniques, they are far from being
precise enough from supporting applications such as natural writing interface.
Existing works also require a wide-band signal for motion tracking, hinder the
co-existence with voice interface on low sampling rate devices. We ask the
question, How precisely can we track motion using a small bandwidth or just
one frequency?

A popular approach for contactless motion detection repurposes the principles
of active sonar. The Frequency Modulated Continuous Wave (FMCW) sonar
technique is particularly effective for detecting small motions due to its
high sensitivity and range resolution. Here an acoustic speaker generates a
chirp signal with continuously varying frequency within a fixed bandwidth and
the receiving microphone detects the distance of the source from the phase
delay of the received chirp. Unfortunately, the resolution of the FMCW sonar
is limited by its bandwidth. Even if a system does not mind using audible
sounds, it can use a maximum bandwidth of around $8$kHz on most of the
commodity devices. Moreover, using the entire audible bandwidth prevents the
system from simultaneously processing human voice signal.

This paper presents WhisperWand, an inaudible motion tracking interface for
voice assistants. The system shows fine-grained motion detection in a real-
world scenario using only a single frequency without affecting the device’s
voice recognition capabilities. WhisperWand achieves these targets by using a
very high-frequency ultrasound signal for ranging. Although regular voice
assistants are not capable of receiving ultrasound signals, we show that the
inherent non-linearity present in the off-the-shelf microphone systems provide
a natural multiplication operation in the acoustic signal path. If signals are
carefully designed, they can leverage this implicit multiplication operation
to shift down a high-frequency signal to the low-frequency band, which the
microphone can readily record without any modification to the hardware. We
build on this intuition to develop our Cross-Frequency Continuous Wave (CFCW)
sonar that generates ultrasound for ranging but receives and processes the
signal at the regular audible range in the off-the-shelf voice assistants. The
high-frequency signals offer a clear advantage in the resolution of distance
tracking while inaudible to human of all ages. Moreover, through careful
design, CFCW sonar only occupies one frequency at the audible band, minimizing
interference the to voice interface. Next generations of voice enabled devices
are coming with ultrasound speaker for signal generation. However, for devices
with the built-in speakers that cannot produce these high-frequency ultrasound
signals, we design an external add-on speaker module capable of producing the
required ultrasound signals. This speaker module operates as a stand-alone
external speaker connected through the audio port of the smart speaker.

In addition to the low-latency high-resolution 3D trajectory tracking,
WhisperWand takes one of the most precise motions – natural strokes of
spontaneous handwriting as an application. The personalized style of
handwriting includes subtle sub-mm movements of the pen tip continuously
moving in the air. Moreover, handwriting recovery will need to translate the
arbitrary 3D trajectory of the pen to a 2D writing recognizable by humans and
machines. We address the first challenge in our system, called WhisperWand, by
developing a novel motion-tracking technique and the second challenge using a
sequence of post-processing methods customized for the handwriting interface.
WhisperWand applies a geometric mesh parameterization technique to recover the
handwriting from an arbitrary virtual 3D surface to a flattened projection on
a 2D plane. The post-processing eliminates any stray trajectory due to pen-
lift and unwanted movements of the pen.

This paper focuses on developing a precise in-air motion tracking system for
voice assistants that can co-exist with the voice interface. We assess the
limits of ranging and localization with CFCW sonar and application-specific
challenges such as removing pen lifts and flattening writing surfaces. We made
the following contributions in this project:

  * $\bullet$

Designed, implemented, and evaluated a new ranging technique, cross-frequency
continuous radar, that can capture minute motions such as individualized
writing styles in signature. The accuracy of cross-frequency continuous radar
is $10\times$ higher than the most precise acoustic-based ranging technique.

  * $\bullet$

Developed an inaudible pure tone-based motion tracking system that can co-
exist with voice interface on low sampling rate voice assistants that can only
support a frequency band below 8kHz.

  * $\bullet$

Designed a series of processing methods to recover handwriting from a 3D
trajectory drawn in the air. The output produces handwriting that retains the
individualized writing styles and is legible to humans as well as machines.

  * $\bullet$

Developed a hardware/software prototype of the end-to-end system. We plan to
make the designs and codes open-source for the community to reproduce,
evaluate, and build on the WhisperWand system.

##  2\. Application scenarios

Non-verbal interaction interface: Co-exist with voice interface, WhisperWand
can be a means for nonverbal communication with voice interfaces for
interacting inconspicuously or even covertly. The user can write comments or
draw symbols while listening to a particular portion of a song and this note
will be saved on the timeline. The stylus for writing in the 3D space can be a
controller for complex interactions with a voice assistant useful for gaming
or gesture-based password. It can enable a writing interface for taking quick
handwritten notes on voice assistants. One example is annotating a live audio
(e.g., a song or a speech) played by the speaker.  Authentication with
physical signature: It is natural to wonder if a voice assistant can capture a
reliable physical signature, will it open electronic-signature services on
this new platform? Today voice assistants can read out emails to us, make
online payments, and even read bedtime stories. We can imagine in the future a
voice assistant reading out a legal document to a user and collecting a
signature for acknowledgment or on the receipt of a transaction. This sign-in-
the-air interface can also verify liveness and enable multi-factor
authentication for a secure session of interaction with the voice assistant.

Potential applications with CFCW sonar: In addition to interaction and
handwriting interface, the core ranging and localization with the cross-
frequency sonar can lead to applications that require high-resolution
tracking. Examples include tele-operation and remote surgery with precise hand
motion tracking, real-time orientation estimations for AR/VR systems, and
bodily vibration and tremor detection. We look forward to applying CFCW to
more applications requiring precise motion tracking.

##  3\. Primer: Location from phase

Distance estimation is a fundamental building block of localization, and its
accuracy depends on the Time of Flight (ToF) estimation. The phase of a
coherent signal is a measurable parameter that continuously changes over time.
Therefore, if the phase of the received signal is estimated correctly, it can
tell the time delay of signal propagation or the ToF from its origin at the
beacon. When transmitting a pure tone, the received signal can be represented
as $S_{R}=sin(2\pi f(t-t_{p}))$. Here $f$ is the frequency of the signal, and
$t_{p}$ is the time delay of propagation. Through analyzing the phase of
recorded signal $2\pi ft_{p}$, we can calculate the time of arrival $t_{p}$.
The phase shift of two adjacent samples can be formulated as $\Delta\phi=2\pi
f\Delta d/c$, where $\Delta d$ is the distance change within the time period
and c is the speed of sound.

###  3.1. Advantage of pure-tone based ranging

FMCW is a widely accepted technique for ranging and localization. However, it
has two limitations. FMCW transmits a chirp that occupies a wide frequency
band, and the resolution of distance tracking is proportional to
$\frac{c}{B}$, where $c$ is speed of sound and $B$ is the bandwidth of chirp.
To achieve higher resolution, a wide frequency band is occupied. If the band
overlaps with the 0-8kHz frequency band of voice, it disables the voice
interface. Existing works use near-ultrasound frequency band for the chirp,
but not all the devices support near-ultrasound frequency band. Voice
assistants like Amazon Alexa and Amazon Echo only support 0-8kHz frequency
band. Similar to FMCW, Time-of-Arrival (ToA) and Time-Difference-of-Arrival
(TDoA) also require a wide frequency band for precise correlation. The
advantage of pure-tone based ranging is that it only requires one frequency,
minimizing the interference with voice interface and other applications
requiring a wide frequency band.

Figure 2. The phase offset caused by interference leads to smaller distance
offset on high-frequency signal over low-frequency signal.

###  3.2. Advantages of high-frequency signal

Distance resolution is proportional to the frequency. The sensitivity of
distance measurement using phase depends on the wavelength of the signal as
the phase changes $2\pi$ radian per wavelength. With same distance change, a
larger phase change is shown in higher frequency signal. The microphones of
the voice assistants operate at the low-frequency audible signals of longer
wavelength leading to poor ranging and localization performance. High-
frequency signal leads to smaller distance offset under interference. Let us
assume that the interference caused by environmental noise or multipath
creates an indirect path with a lower amplitude than the direct path in phase
domain. With same distance change, Figure 2 shows the complex representation
of phase change for both high-frequency and low-frequency signal. The line-of-
sight path is shown in blue and the interference is shown in red. The sum of
the signal in green is the captured phase at the receiver side. Suppose the
object moves in distance $d$, the theoretical phase shift is $e^{2\pi fd/c}$.
The maximum phase error $\Delta\phi$ caused by the interference is
$sin^{-1}(\frac{A_{2}}{A_{1}})$ (wang2019millisonic, ). If we convert the
phase offset to distance offset, $\Delta d=\frac{\Delta\phi c}{2\pi f}$. As
shown in Figure 2 (b)(c), the key observation is the distance offset caused by
interference is proportional to frequency. In other words, high-frequency
signal is more robust to interference in distance tracking. Therefore, we use
high frequency signal for ranging.

###  3.3. Challenges of pure tone-based ranging

Vulnerable to multipath distortion. Unmodulated pure tone-based systems suffer
from an environmental multipath effect that degrades the phase accuracy of the
signal. Although pure tone ranging is accurate, multipath fading severely
impacts the accuracy. The non-line-of-sight signals fall in the same
frequency. This inseparable overlap interferes with the phase values, leading
to inaccurate distance estimation. Inability to obtain initial distance. While
pure-tone based ranging can capture precise distance change, it cannot capture
the absolute initial distance due to phase wrapping. The phase of absolute
distance is $2n\pi+\phi$, phase tracking can only capture $\phi$, without
information of the number of wrapping $n$. We address these limitations in our
novel ranging technique explained next.

##  4\. Cross-frequency sonar design

WhisperWand develops an acoustic ranging method that uses two completely
separate bands for the transmit signal and received signal – leading to the
ranging technique we call Cross-Frequency Continuous Wave (CFCW) sonar. If
works, CFCW sonar can introduce several crucial advantages to sensing
applications with these household devices. (a) (Accuracy) Given it can
leverage high-frequency ultrasound signals for ranging, it can have orders of
magnitude higher accuracy compared to audible frequency-based techniques. (b)
(Inaudibility) It allows accessing a wide band of inaudible frequencies with
regular microphones, while existing techniques can only use a small band of
‘near-ultrasound’ frequencies between 18-24 kHz. (c) (Power and hardware
simplicity) CFCW sonar operates at a low frequency which requires lower
sampling rates. It offers similar accuracy to ultrasound with low complexity
hardware as the power consumption and processing and storage requirements of
the platform increase with the frequency of operation.

The CFCW sonar leverages implicit frequency translation possible in regular
microphones. It carefully designs its ultra-sound transmit signals such that
they can leverage the fundamental nonlinearity in microphones for automatic
down conversion of the signals to the low-frequency recording range of the
microphones. While nonlinear frequency conversion enables the core CFCW
technique, it requires further signal design to eliminate environmental
effects for effective distance tracking and start point detection for
WhisperWand. We explain the step-by-step signal model next.

###  4.1. Implicit frequency and phase translation

Microphone non-linearity primer:  Commonly microphone is a linear system,
which means the signal recorded by the microphone is a linear combination of
the input signals. If the input signal is $S$, the recorded signal $S_{out}$
is $S_{out}=A_{1}S$, where $A_{1}$ is the complex gain. However, as introduced
in Backdoor (roy2017backdoor, ), the microphone shows non-linearity when the
frequency of the input signal is above $25kHz$. The recorded signal $S_{out}$
can be modeled as $S_{out}=\sum_{n=1}^{N}A_{n}S^{n}$. Since the third and
upper terms are too weak and thus can be ignored, the $S_{out}$ can be
represented as $S_{out}=A_{1}S+A_{2}S^{2}$. As shown in Figure 3, we transmit
two ultrasound pure tones together as $S=sin(2\pi f_{1}t)+sin(2\pi f_{2}t)$,
where $f_{1}$ and $f_{2}$ are 45kHz and 38kHz. With nonlinearity of
microphone, the recorded signal also includes the components produced by
$S^{2}$, which are $f_{1}+f_{2}$ and $f_{1}-f_{2}$. While $f_{1}+f_{2}$ is
higher than the threshold that can be recorded by the off-the-shelf
microphones, $f_{1}-f_{2}$ can be recorded with proper selection of
frequencies, as shown in Figure 3.  Interestingly, if phase change is happened
in frequency $f_{1}$, the phase change can be mapped to the down-converted
frequency $f_{1}-f_{2}$. As $S=sin(2\pi f_{1}t+\phi)+sin(2\pi f_{2}t)$, the
received signal at frequency $f_{1}-f_{2}$ is $sin(2\pi(f_{1}-f_{2})t+\phi)$.
Therefore, we map the phase of frequency $f_{1}$ on $f_{1}-f_{2}$ by down-
converting the frequency using non-linearity of microphone. Here we call
$f_{1}$ as primary channel and $f_{2}$ as secondary channel. The primary
signal source act as an acoustic beacon attached to the target object. The
secondary signal source is fixed using an add-on module to the voice
assistant. The secondary source emits a low-power signal that illuminates only
the microphone array of the voice assistant. These signals combine inside the
microphones to create a different signal that carries the phase offset between
the primary and secondary tones. The phase difference is due to the distance
of the target from the voice assistant. We verify if an off-the-shelf voice
assistant can show such implicit frequency shift. We use an Amazon Echo Dot
device to simultaneously record the human voice and the shifted tracking
signal successfully as shown in Figure 4.

Figure 3. The non-linearity of the microphone multiplies ultrasound signals
implicitly shifts the frequency to the audible band.

Figure 4. (a) Spectrogram and (b) FFT of the signal captured by Alexa. Motion
tracking only occupies 7kHz frequency.

###  4.2. Multipath avoidance

WhisperWand relies on the accurate phase estimation of the signal for ToF
measurement, however phase of the received signal can be severely affected by
environmental effects of which multipath signal propagation is the most
significant. Multipath is a natural phenomenon where a signal, after leaving
the transmitter, reflects off objects in the environment to create replicas
and the replicas propagate through paths of different delays before combining
at the receiver. The lengths of these individual paths decide the phase delays
of the replicas and therefore their superimposition leads to an unknown
amplitude and phase of the received signal. For location tracking, WhisperWand
requires the phase of the direct line-of-sight (LOS) path, but any non-line-
of-sight (NLOS) path can introduce an unpredictable error to the phase.
However, note that there is only one LOS path, and its length is shorter than
all NLOS paths. NLOS paths are indirect paths and therefore arrive at the
receiver later than the LOS signal. This provides a window of opportunity to
extract the undistorted phase from the LOS path before the multipath replicas
superimpose on this signal as explained next. Potentially the phase estimation
system can use a part of the signal clear from multipath superposition to get
an accurate phase of the transmit signal.

Theoretically, in a pulse-based probing signal, there exists a time window
when only the LOS signal is present at the receiving sensor. The length of
this time window WIN-LOS is equal to the delay of the first NLOS path which is
an unknown environmental parameter. The smallest delay of the NLOS path, which
leads to the smallest size of the time window, is related to the distance of
the closest large objects to the device. Another parameter that impacts the
worst-case estimate of the time-window WIN-LOS is the FFT resolution.
WhisperWand estimates the phase by first calculating the FFT coefficients of
the time series data of length WIN-LOS seconds. This signal is sampled by the
microphone at 16 kHz (standard sample rate of Alexa) after the nonlinear
conversion of the ultrasound signal. A too-short WIN-LOS will lead to wider
FFT bins combing a wider band of frequencies to the same bin. At the baseband,
the microphone also records ambient sounds, such as human voices and other
household noise. The energy of these sounds is mostly limited within 4 kHz
(wang2021mavl, ) and the CFCW signal is mapped to 7 kHz at the baseband.
Therefore, we limit the FFT bin width to a maximum of 2 kHz, which in turn
limits the minimum size of WIN-LOS to 0.5 ms around 17cm in distance. A pulsed
signal can hop to a different frequency within this time to avoid delayed NLOS
signals. As explained next, the CFCW receiver can continuously operate on a
constant frequency while the ultrasound-ranging signals hop.

Figure 5. An illustration of transparent frequency hopping. Although the
transmitted frequencies keep changing, the frequency of the received signal is
constant. The spurious signal does not interfere with the frequency of
tracking.

Transparent frequency hopping.  In CFCW sonar, the received frequency
$f_{rcv}$ is different from the dual frequencies ($f_{sty}$ and $f_{base}$)
transmitted in the air and it is equal to the difference between the two
transmitted frequencies ($f_{rcv}=f_{sty}-f_{base}$). Therefore, if the
transmitted frequencies change in a synchronized way to maintain a constant
difference between them, the received signal will remain constant. In other
words, we can design a frequency hopping for the transmitted signal pair while
its effect is transparent to the received signal. We used this observation to
avoid multipath in the CFCW sonar.

The dynamic multipath distortion is primarily caused by the $f_{sty}$ signal
source, which is facing towards the air, changes location over time, and is of
relatively higher power to have stronger NLOS reflections. The $f_{base}$
signal source, on the other hand, is placed in a static location only a few
centimeters away from the microphone array. Moreover, the power of the
$f_{base}$ signal is kept low – just enough to reach all the microphones in
the array – making NLOS multipath from this signal too weak to cause a
significant impact on the received difference signal. Therefore, if $f_{sty}$
and $f_{base}$ signals hop to new values $\hat{f}_{sty}$ and $\hat{f}_{base}$
respectively, delayed NLOS components of $f_{sty}$ will only mix with the
current $\hat{f}_{base}$ to create a spurious signal at frequency
($f_{sty}-\hat{f}_{base}$). This spurious frequency does not interfere with
the received CFCW frequency $f_{rcv}$ leaving it unaffected by the multipath
effect. Figure 5 shows this transparent frequency hopping-based multipath
avoidance where the transmitter’s frequencies keep hopping to new values
together every 3 ms.

Phase unwrapping.  Phase unwrapping is a classical approach to recover the
original phase value from the wrapped phase value. The phase value is wrapped
within $[-\pi,\pi]$, and the goal of phase unwrapping is to reconstruct the
continuous phase by removing the ”sudden jumps”. The true phase (i.e., true
phase without wrapping) can be reconstructed as long as the difference between
the subsequent phases is less than $\pi$. To satisfy this condition, we need
to guarantee the speed is less than $\frac{c}{2f_{1}\Delta T}$. With a
$f_{sty}$ 40kHz and $\Delta T$ 3ms, the maximum speed is 1.41m/s. It is
revealed that the peak velocity of hand movement is 2.7m/s, and the peak
velocity of hand gesture is 1.8m/s (degoede2001quickly, ). Meaning our system
can result in an error when the moving speed is high, limiting us to achieve
high accuracy with even higher frequency. To solve this problem, we induce
speed to estimate the direction of phase rotation. Ideally, when the speed was
positive in the last samples and was not decreasing dramatically, it is more
likely the phase is rotating anti-clockwise. Otherwise, when the speed was
negative (the direction of movement is opposite) and was not decreasing in the
last samples, it is rotating clockwise. In our system, we only require the
absolute difference between subsequent phases is within $2\pi$, thus the
highest speed we can support is 2.83m/s. To make sure there is an overlap
between the two transmitted signals with a supported distance of 70cm, we set
a frame rate of 333. Why synchronization is not required? We do not require
synchronization between transmitters and receivers. The reason is even with
synchronization, we still cannot find the initial distance by using a single
frequency. A precise correlation for ToF requires a wide band of frequency. To
get the absolute distance for localization, we design an algorithm to detect
the distance at the start point without the requirement of synchronization or
a wide band signal as explained next.

Figure 6. Illustration of start point detection technique. Although the
optimization function is non-convex, true start point has the global minimum
optimization error.

Start point detection. Without synchronization between transmitters and
receivers, our start point detection algorithm builds on a traditional
approach called Time Difference of Arrival (TDoA). TDoA approach locates a
target at intersections of hyperbolas or hyperboloids that are generated with
foci at each fixed receiver of a pair. Existing algorithm of estimating the
time delay between two receivers is to find the time samples that have highest
correlation between the transmitted and received signals. However, the time
delay is not precise enough due to two reasons: (1) The time resolution of the
correlation is limited by the sampling rate of received signal. (2) A precise
correlation requires a sharp pulse as transmit signal, which requires a wide
frequency band. Unfortunately, using wide frequency band hints the co-
existence between voice interface and tracking for low sampling rate devices
like voice assistants. Since the distance between the microphones is only
3.6cm on voice assistants, the distance drift causes even larger error on
location estimation.

To address the issue mentioned above, we take advantage of the precise phase
from CFCW sonar. Suppose the wrapped phase difference captured by a microphone
pair is $\theta_{ij}$, and the true phase difference is $2n\pi+\theta_{ij}$,
where $n$ is the number of wrapping. Based on the theory of triangle, the
maximum distance difference from speaker to two microphones is the distance
between two microphones. Thus, suppose the range between the pair of
microphones is $d$ and the wave length of signal is $\lambda$, we can define
the range of $n$ as $n\in[-\frac{d}{\lambda},\frac{d}{\lambda}]$, where n is
an integer. We find that with multiple pairs of microphones, only when the
vector $N$ map with the ground truth, the hyperbolas can cross on the same
point, i.e., the start point. In simulation, we loop through every possible
$N$ for microphone pairs to optimize the location $P$ and the corresponding
error $E(P)$. The optimized location $\hat{P}$ is

(1) $\hat{P}=\operatorname*{argmin}_{P}\sum_{i\neq
j}\left\|\lambda(N_{ij}+\frac{\theta_{ij}}{2\pi})-c\tau_{ij}(P)\right\|^{2}$

where $i$ and $j$ represents the ids of microphones, $\lambda$ is the wave
length, $N_{ij}$ is the number of phase wrapping for time difference of
arrival $\tau_{ij}$, $c$ is speed of sound, and $\theta_{ij}$ is the captured
phase difference between two microphones, i.e., $\theta_{i}-\theta_{j}$. As
shown in Figure 6, although there are many other replica peaks, only the true
start point has the minimum optimization error. Therefore, our optimization
problem aims to find $\hat{N}$ and $\hat{P}$ that gives the minimum error.
Since bruteforcely looping all possible $N$ is computationally heavy, the
optimization boils down to the minimization of

(2) $\hat{P},\hat{N}=\operatorname*{argmin}_{P,N}\sum_{i\neq
j}\left\|\lambda(N_{ij}+\frac{\theta_{ij}}{2\pi})-c\tau_{ij}(P)\right\|^{2}$

where $N\in[\frac{-D}{\lambda},{\frac{D}{\lambda}}]$ and $D$ is the vector of
distance between each pair of microphones. As shown in Figure 6, this
objective function is non-convex, with many local minima. Moreover, this
function is discrete, with $N$ as integers. To solve this challenge, we apply
a non-convex optimization algorithm called genetic algorithm
(mirjalili2019genetic, ) that also accept integer parameters. By minimizing
the loss, we optimize the $N$ and $P$ simultaneously to find the true start
point with global minima. This approach frees up the requirement of using a
wide band signal to capture the time delay for TDoA.

##  5\. Application-specific design

Precise motion tracking on voice assistants can benefit a wide range of
existing applications and open up possibilities for new ones. Several non-
verbal human-machine interfaces can be developed with gesture and motion
tracking. We have developed natural handwriting detection as the
representative application. Detection of words written in the air is a classic
application explored with a variety of sensing modalities. We pushed the
boundary of this application by not only enabling fine-grained motion tracking
required for subtle strokes created during sub-mm movements of the fingers,
but also developed techniques for post-processing the data so that even the
handwriting can be verified from the notes. Interestingly, during the process
of motion detection, WhisperWand’s signal does not interfere with the voice
signal and therefore the voice assistant can perform its regular operations
while simultaneously capturing the writing.

WhisperWand detects the pen strokes by localizing the acoustic source (the
primary signal). However, recovering meaningful writing from the continuous
trajectory of the pen tip will require addressing two challenges. (a) Pen-lift
elimination: While writing on paper or other solid surfaces, we often lift the
tip to jump to the next stroke which is disconnected from the current one, as
shown in Figure 7. This event is transparent while writing on a solid surface
like a paper or a tablet screen as it does not generate any strokes. However,
3D tracking-based writing detection cannot naturally distinguish a pen-lift
event from the actual strokes. (b) Flattening virtual surface: Given the user
can write at any angle on her assumed surface in the air and the surface is
not flat, the projection of the 3D trajectory on the horizontal plane leads to
distortion of the strokes. The distortion often makes the writing illegible to
both humans and automatic text recognizing software.

###  5.1. Localization in 3D space

Distances from at least three spatially separated microphones enable us to
apply the traditional trilateration method (doukhnitch2008efficient, ) to find
the 3D location of the target in space. We use distance measurements from all
the available microphones for location estimation using a multi-lateration
technique. Therefore, the location of the target should be somewhere on a
sphere of radius $d_{m}$, centered at the microphone’s location.
Theoretically, we should be able to find the location of the target by solving
the system of $N$ equations. However, in practical systems, the individual
distance measurements are not perfect due to noise, and therefore, the spheres
do not have a common solution. We use the following optimization to find the
estimated location of the target in 3D space. It essentially minimizes the
distance of the estimated location from all the spheres.

(3)
$\operatorname*{argmin}_{x,y,z}\sum_{i=1}^{N}\sqrt{(x-x_{i})^{2}+(y-y_{i})^{2}+(z-z_{i})^{2}-d^{2}_{i}}$

Figure 7. An analysis of the handwriting process. Each character has two to
four clusters that have slower moving speed than the strokes and pen lifts.

###  5.2. Removing pen-lift from virtual surface

The 3D trajectory contains both the actual pen strokes, which are a part of
the writing, as well as the stray movements, like pen-lifts. As shown in
Figure 8(a), the written word ‘fit’ is unrecognizable from the raw trajectory
of the pen because of unwanted marks due to pen-lifts in between two
disconnected strokes. To avoid this error, we will first need to identify the
accurate start and end of the pen-lift trajectories.

Figure 8. An example of pen lift removal of the word ”fit”.

Handwritten letters and drawings can be viewed as a collection of stroke
segments of different curvatures and lengths. Each of these segments starts
and ends with a change in direction of the stroke. While it is difficult to
quantify an angle or sharpness in the change of directions of segments due to
various writing styles and user-dependent features, the velocity of the pen
movement offers a reliable feature. The segment-by-segment annotated
handwriting reveals that the speed of the pen comes to a low value at the end
of a segment and just before the beginning of the next, as shown in Figure 7.
We adopted this intuition by first calculating the velocity of the pen between
every two samples of location on the trajectory.

A cluster of 3D location points that corresponds to the local minima of the
time-varying velocity indicates the start or end of a segment. Note that these
clusters i.e., the ends of the segments are necessarily belong to the
imaginary writing plane in the air. Now it is left to identify a pair of such
clusters indicating the start/end of a pen-lift. To this end, first, we fit a
triangular plane to each three of the clusters. A stroke segment joining two
clusters of this triangle is either entirely a valid stroke or a pen-lift. We
calculate the orthogonal distance of all points of a segment from this plane
to identify a pen-lift segment by its off-plane movement. In a few rare cases,
the user slows down during a pen-lift movement which erroneously produces a
cluster on a pen-lift trajectory. Our algorithm mistakenly includes these pen-
lift trajectories as valid strokes. We correct this by fitting a third-order
plane through all the clusters. Given the variations in the imaginary writing,
a plane is smooth corresponding to sudden movements for pen-lifts, the
spurious clusters on the pen-lift trajectory stand out from this fitted plane.
These clusters are then pruned before applying the triangle fitting mentioned
above.

###  5.3. Flattening writing surface

WhisperWand aims to develop a natural writing interface on the voice
assistants and therefore, does not restrict the orientation of the writing
plane or the writing style. The user can simply write in the air near the
voice assistant and WhisperWand attempts to produce the writing (or drawing)
in a human and machine-readable form. A simple projection of the trajectory on
to the horizontal plane can severely squeeze and distort the writing.

For the conversion of the 3D trajectory to 2D writing, we first identify the
writing surface from the 3D point cloud of the trajectory points by fitting a
third-order surface to them. This smooth surface is the assumed virtual plane
on which the user has written. Next, we take an orthogonal projection of the
points from the 3D pen trajectory on this surface. Note that the pen-lift and
other stray marks are already removed from this trajectory data in the
previous step. The next step is to project this arbitrary writing surface,
along with the writings, on the 2D horizontal plane. The curves on the writing
surface will require careful flattening to avoid any distortion of the written
strokes during projection. We refer to the techniques used in mesh
parameterization for this purpose. While there are several techniques for
curved surface flattening and transformation (floater2005surface, ;
sheffer2007mesh, ), we adopted an isometric mapping (Isomap)
(tenenbaum2000global, ) based approach. Isomap is a non-linear dimensionality
reduction technique and several recent techniques (caixeta2021robust, ;
yousaf2020extended, ; acosta2016geodesic, ; zhou2004iso, ) build on this core
idea to flatten triangularized mesh surfaces. This approach particularly suits
our purpose as the algorithm keeps the geodesic distance between the points of
the writing unchanged. Although Isomap is not the fastest algorithm for
surface flattening, it provides robustness and applies to surfaces with sharp
curves that occasionally appear on our unsupported writing surface.

Table 1. Comparison with prior works on acoustic motion tracking.

System | Setup | Tech. | Audible | 1D/3D Accu. | Refresh rate | Range | Mic sep. | Compt.a
---|---|---|---|---|---|---|---|---
CAT (mao2016cat, ) | Speaker-Phone | FMCW | N | 4mm/9mm | 25Hz | 7m | 90cm | No
SoundTrak (zhang2017soundtrak, ) | Speaker-Watch | Phase | Y | –/13mm | 86Hz | 20cm | 4cm | No
MilliSonic (wang2019millisonic, ) | Microphone-Phone | FMCW + phase | N | 0.7mm/2.6mm | 40Hz | 3m | 6-15cm | No
WhisperWand | Alexa-speaker | CFCW + phase | N | 0.07mm/1.4mm | 333Hz | 70cm | 3.6cm | Yes

  * a

The column of ’Compt.’ indicates if the solution is compatible with voice
interfaces on low sampling rate devices like Amazon Alexa.

##  6\. Implementation

To validate our implementation of WhisperWand, we build a hardware prototype
and data processing pipeline. As shown in Figure 9(a), we use a 7-microphone
array placed horizontally on the table and two low-cost($0.5) ultrasound
speakers (ultrasoundspeaker, ). The size of the microphone array is the same
as Amazon Echo, with distances between microphones as 3.6cm. We used
omnidirectional ADMP401 MEMS microphones (devices2013admp401, ) sampled at
16kHz, similar to the components used in the Amazon device (alexaMic, ). The
microphones are sampled simultaneously using a multi-channel data acquisition
system (Keysight, ). The secondary ultrasound speaker is placed at the same
plane as the microphone array, 15cm away from the center. The primary speaker
is equipped with a pen-shape stylus or act as a transmitter to project signal
in the air. The speakers are driven and synchronized by a Keysight 33500B
function generator. The collected data is processed offline using Matlab
scripts on a computer. For 3D accuracy evaluation, we use Vicon (vicon, )
system as our ground truth, as shown in Figure 9(b).

Figure 9. (a) Hardware setup for system evaluation and (b) Vicon system for
ground truth of 3D accuracy.

##  7\. Evaluation

Table 1 summarizes the performance of WhisperWand and compares with existing
techniques.

###  7.1. Location tracking performance

Ranging accuracy. To get an accurate ground truth, we mount the primary
speaker on a linear actuator (LinearActuator, ) that has a movement accuracy
of $11\mu m$ to get precise ground truth. The speaker moves from 10cm to 40cm.
To evaluate the impact of frequency, for each frequency for the primary
channel, we collect data 5 times. We maintain a 7kHz separation between the
primary and the secondary frequencies. Figure 10(a) shows the cumulative
distribution function (CDF) plot of the ranging error with frequencies 20kHz,
40kHz, 60kHz, and 80kHz. The signals are transmitted by transmitters resonate
at 20kHz (20kHzspeaker, ), 40kHz (ultrasoundspeaker, ), 60kHz (60kHzspeaker,
), and 80kHz (80kHzspeaker, ). Note that there is no non-linearity of the
microphone for 20kHz signal, so we directly estimate the distance from the
20kHz frequency. The frequency was hopped between the base frequency and 2kHz
higher. This plot shows that the accuracy increases with the higher
frequencies, with median errors of 0.48mm, 0.16mm, 0.076mm, and 0.073mm. The
accuracy difference between 60kHz and 80kHz is small. WhisperWand’s range can
be increased by increasing the power of the primary speaker. We evaluate the
performance with different distances of the stylus from the voice assistant in
Section 7.8.

Figure 10. The CDF of (a) ranging error and (b) 3D location tracking error of
WhisperWand.

3D location tracking accuracy. We measure the 3D localization and tracking
accuracy with the distance estimations from 7 microphones in the array and the
3D localization algorithm described in Section 5.1. To get the ground truth,
we use the optoelectronic motion capture system of Vicon with 0.2mm tracking
accuracy with a frame rate of 250Hz. We also use the same frame rate so that
the distance estimations can be matched. The 3D region of detection is
$0.5m\times 0.5m\times 0.4m$. The frequency of the primary speaker is
40/42kHz. Figure 10(b) shows the CDF of 3D localization for WhisperWand. The
median error is 1.4mm with a 3.6cm distance between the microphones. We
evaluated WhisperWand with the similar microphone array configuration
available in off-the-shelf voice assistants. Figure 11 shows some qualitative
examples of 3D tracking. Here the red lines indicate the ground truth
trajectory captured by the Vicon system, and the blue line is the trajectory
estimated with WhisperWand.

Figure 11. 3D tracking examples. The red and blue lines represent estimates
from Vicon and WhisperWand respectively.

###  7.2. Performance of voice recovery

WhisperWand is capable of simultaneously recovering voice and localization
signals since WhisperWand only occupies 7kHz frequency, minimizing
interference to the frequency band of voice. In this section, we evaluate the
quality of the recovered voice in the presence of a ranging signal. We apply
an existing i-vector-based speech-independent speaker authentication
(garcia2011analysis, ) and Google speech-to-text conversion (speechToText, )
techniques for the evaluation of voice recovery. We first train the speaker
authentication model with 9 males and 9 females in the Pitch Tracking Database
from Graz University of Technology (PTDB-TUG) (database, ), where each subject
has 236 8-second voice pieces. Then we enroll one male and one female using
three voice samples. We use the rest 233 samples for testing. To get the
ground truth of the system, we first use the original voice pieces for
testing. After that, we play the samples while collecting the 7kHz handwriting
tracking signal, and save them for testing. We show the accuracy with 7kHz in
Figure 12, the error rate only increases from $6.26\%$ to $8.28\%$, which
indicates the speaker authentication system is robust despite simultaneous
reception of handwriting tracking signal. For speech-to-text conversion, we
have one male and one female speaking 5 sentences each during the hand
movement with and without WhisperWand. We feed the recorded signals to the
speech-to-text converter without any change. The word recognition accuracies
of converted text messages are both $96.0\%$. One example of the sentence is
”Hi Alexa, I am using AirMarker to draw a star”. The recognized text is ”Hi
Alexa, I am using hair marker to draw a star”.

Figure 12. Speaker authentication without (left) and with (right) the 7kHz
handwriting tracking signal.

###  7.3. Performance under human voice and environmental noise

To evaluate the performance with human speaking while writing in the air, we
perform the same process of evaluation as the 3D localization. The only
difference is we play the noises at the same time. We test the robustness
under 50/60/70dB of human voices. Note that the highest strength of human
speaking in daily life is 50dB. Even when the sound level is 70dB, the median
error of WhisperWand is 2.4mm, as shown in Figure 13(b), meaning the human
voice does not affect its accuracy with a 16kHz sampling rate. Figure 13(a)
shows an example of the spectrogram of the recorded signal. There is a clear
separation between the frequencies of a human voice, environmental noise, and
the 7kHz down-converted signal for tracking. We also play three representative
types of environmental noises in the home, mall, and restaurant. The sound
levels are 60dB. The median error of WhisperWand with noises in malls and
restaurants are 1.4mm and 1.6mm, respectively. The error under home noise is
3.4mm which is relatively high. The reason is that the frequency of running
water has a component of 7kHz.

Figure 13. Spectrogram of the recorded signal under human voice (left) and 3D
results under different types of human voices and environmental noises
(right).

###  7.4. Performance of handwriting recovery

To test our performance of handwriting recovery, we write both paragraphs and
single words in the air. For the paragraph recovery, we write six
representative paragraphs in the air and use Google Keep to convert images
from text with an accuracy of $92.4\%$. In Figure 14, we show two
representative examples. To verify our accuracy of in-air writing-to-text
conversion, we recruit ten participants (2 female and 8 male) to do a small-
scale user study. The users were asked to draw 5 words each. Our objective is
to compare the accuracy of writing to text conversion with handwriting on
paper and in the air. To enable a fair comparison, we attach the primary
speaker to an apple pencil. After capturing both the on-paper and in-air
writings, we feed them into Google Keep for writing-to-text conversion and
compare the accuracy. We compare the accuracy of each word from the ten users,
which are ”alexa”, ”home”, ”okay”, ”sign”, and ”word”. The accuracy is the
number of correctly detected characters over the number of characters. We find
the overall accuracy of WhisperWand and on-paper writing are $94.1\%$ and
$96.6\%$, which means WhisperWand shows comparable accuracy with writing on
paper.

Figure 14. Examples of paragraphs detection.

###  7.5. Performance of signature recovery

To test the performance of signature recovery, we do a survey from both
professional and general public. Our objective is to compare the similarity of
the handwritten signatures and in-the-air recovered signatures. Not only the
written text can be recognized, but the biometric characteristics in
handwritten signature actions can be accurately captured, so that signature
verification is possible. For human study verification, we first recruit 20
participants (4 female and 16 male) to sign their signatures both on an iPad
using an apple pencil, and in the air. To avoid the inconsistency between the
two signatures, we attach the primary speaker to the apple pencil as the
ground truth.

(a) Professional signature verification: After capturing both on-tablet and
in-air writings, we first ask a signature verification professional to give
his opinion on if the 20 pairs of signatures are matched or not. The
professional treats 18 out of 20 pairs are matched, resulting in an accuracy
of $90\%$. The strategy of signature matching follows two rules: (1) the
structure of the signatures should be the same, including the shapes of the
whole signature and each character. (2) The angles of the pen touching and
lifting the surface should follow the same trend.

(b) Opinion of general public: After that, we select five pairs of signatures
and post a survey on Amazon Mechanical Turk. 30 people participated in the
survey to share their opinions on if the in-air signatures are real or forged.
Among the 150 ratings, $85.3\%$ of them are more than $50\%$ sure that the
pair of signatures are from the same person. This result means WhisperWand not
only can capture the recognizable written characters but also can capture
precise biometric characteristics of handwritten signature actions, which can
be used for signature verification. Several examples of signatures are shown
in Figure 15.

Figure 15. Examples of signature pairs on iPad tablet (top) and recovered in-
air signature (bottom).

Figure 16. Scenes of different multipath (a) wall, (b) clutters (3) both
clutter and wall.

###  7.6. Robustness to multipath

We consider the common multipath when putting Alexa at different locations in
a room, including a wall in behind, in clutter, and both in clutter and a wall
in behind. The scenes are shown in Figure 16 and results in Figure 17(a). When
there is multipath, the median errors are still within 1.4mm, which means
WhisperWand is robust to multipath. To test the effectiveness of the cross-
frequency approach for multipath elimination, we do the same experiment with a
wall behind, but only using one pair of frequencies. As shown in Figure 17(a),
the median error increases to 10mm, meaning our multipath elimination approach
is effective.

Figure 17. (a)Comparison of the performance under multipath with/without CFCW
for multipath elimination. (b)Performance of WhisperWand with different
frequency resolutions under Doppler shift.

###  7.7. Robustness to Doppler effect

To test the effect of moving speed, we draw three shapes at speeds of 0.2m/s,
0.5m/s, 0.8m/s, and 1m/s. The frequency of the primary speaker is 40/42kHz.
The maximum frequency shift caused by the Doppler effect is 126Hz. As shown in
Figure 17(b), the median errors are all within 1.9mm when the speed is within
0.8m/s. The error increases to 2.6mm when the speed is 1m/s. This result shows
our system is still reliable with a 1m/s moving speed, which is an extreme
case of hand movement in the air.

###  7.8. Impacts of external conditions

Distance to the microphone. Figure 18 shows the CDF plot of 10-40cm and
41-70cm for 40/42kHz and 60/62kHz primary channel, respectively. The plot
shows that the system performs better in the 10-40cm range, with median errors
of 0.14mm and 0.07mm for 40/42kHz and 60/62kHz frequencies. It also confirms
that higher frequency performs better following the theory. When the distance
increases to 41-70cm, the median errors are 0.23mm and 0.24mm. The increase of
error is because the SNR of the acoustic signal reduces with distance.
Moreover, when the distance increases, 60/62kHz frequency does not perform
better than 40/42kHz. The reason is the higher frequency signal attenuates
faster than lower frequency signals. Overall, the median error is still within
0.25mm in 41-70cm distance.

Figure 18. 1D results for 40/42kHz (left) and 60/62kHz (right) for different
distances.

Figure 19. Performance under different plane shifts (left) and size of written
word (right).

Plane shift. We evaluate the performance when writing at different planes,
including flat plane on top, flat plane besides, slant plane besides, and
vertical plane on top. As shown in Figure 19(a), aside from writing on the
flat plane beside the array, the median errors are all within 2mm. The median
error of writing on the flat plane beside the array is 3.3mm. The reason is
when writing on the same plane with the microphone array, the pattern of
distance changing is similar for all the microphones. Thus, same distance
estimation error will cause a bigger location error.

Size of the word. To test the effectiveness of writing recovery with different
sizes, we write the word ”home” in widths from 3 to 20cm. As shown in Figure
19(b), we find that even in a width of 3cm (0.75cm per character), we still
can recover the word clearly and can be recognized by Google Keep. This result
means WhisperWand is applicable for capturing regular sizes of writing for
humans.

##  8\. Related Work

The literature is rich in spatial analysis and localization. We sample below
three topics closely related to this work.

Acoustic motion tracking. Acoustic signals are actively being explored by the
research community for precise localization and tracking (wang2021mavl, ;
cao2020earphonetrack, ; huang2014shake, ; yun2015turning, ). Existing studies
detect minute body movements to monitor sleep apnea events
(nandakumar2015contactless, ) and breathing for adults (xu2019breathlistener,
) and infants (wang2019contactless, ). AAMouse (yun2015turning, ) applies
Doppler shifts of acoustic signals to track hand movements in real-time. CAT
(mao2016cat, ) further enhances the tracking accuracy by analyzing both FMCW
and Doppler shift of acoustic signals. FMCW maps time difference to frequency
shift, without the need for precise synchronization. SoundTrak
(zhang2017soundtrak, ) tracks accumulated phase shift for continuous tracking
of a speaker in the air. MilliSonic (wang2019millisonic, ) also applies the
FMCW signal on hand tracking. To achieve sub-millimeter level accuracy, it
tracks the phase shift of FMCW. On the other hand, LLAP (wang2016device, ) and
FingerIO (nandakumar2016fingerio, ) achieve device-free mm-level finger motion
tracking. In our work, we explore the possibility of finer localization that
does not interfere with voice interface on low sampling rate voice assistants.
The CFCW sonar technique used in WhisperWand enables household acoustic
devices to achieve tens of micrometer-level motion tracking accuracy.

RF and inertial sensor-based motion tracking. We have seen rich literature on
developing radio frequency techniques for localization and tracking, such as
commercial battery-free tags (jiang20193d, ; jin2018rf, ; jin2018wish, ;
wang2013dude, ; wang2021locating, ; xiao2017one, ; yang2014tagoram, ) and
custom RF backscatter (chang2018rf, ; chuo2017rf, ; kotaru2017localizing, ;
ma20163d, ; nandakumar20183d, ; vasisht2018body, ; wang2019rfid, ). However,
they still cannot outperform acoustic motion tracking due to the tradeoff
between latency, frame rate, and motion detection. Systems which can provide
centimeter-level results either require a highly constrained environment
(wang2013dude, ; yang2014tagoram, ) or have frame rates less than 1 Hz
(ma2017minding, ; li2009multifrequency, ) since these systems need to step
over hundreds of megahertz of bandwidth and take several seconds to compute a
single location estimate. More recently, novel systems have been proposed to
significantly reduce the latency of localization systems to tens of
milliseconds. However these systems either require using multiple antennas on
the backscatter tags (nandakumar20183d, ) or using multiple RF transmitters by
scanning over the different spectrum (luo20193d, ). Inertial sensor-based
motion capture is based on miniature intertial sensors, sensor fusion
algorithm, and biomechanical models (xsens, ; synertial, ; cao2021itracku, ).
Inertial sensors can only measure the motions, but not the absolute locations.

Cross frequency sonar. We are not the first to exploit non-linearity to down-
convert the frequency of the recorded signal. The notion of exploiting non-
linearity was originally studied in 1957 by Westervelt’s seminal theory
(westervelt1957scattering, ; westervelt1951theory, ). The closer use of the
non-linearity to our System name is Backdoor (roy2017backdoor, ) which shows
that ultrasound signals can be designed to become recordable by unmodified
microphones. More studies use non-linearity of microphone for side-channel
attack (farrukh2021s3, ; ramesh2021acoustics, ; liu2020maghacker, ;
zhang2017dolphinattack, ), indoor localization (lin2019rebooting, ), and
communication (bai2020batcomm, ).

##  9\. Conclusion

This paper develops an acoustic motion tracking interface for voice
assistants. WhisperWand proposes a high-resolution CFCW-based distance and
tracking estimation algorithm leveraging the nonlinearity of the microphone.
Moreover, WhisperWand is the first motion tracking work that can co-exist with
voice interface on low sampling rate voice assistants. Evaluations of the
prototype show a 1-D ranging error of 73 micro-meter and below 1.4 millimeters
of median error in 3D trajectory tracking. This paper presents multiple
benchmarks, pilot user studies and evaluation for some specific applications.
The design of WhisperWand enables an interface on voice assistants for
capturing handwritten notes, drawings, and signatures and can extend to a non-
verbal mode of human-machine interaction.

## References

  * (1) Google nest audio speaker test: Focusing on the sound.  https://www.dxomark.com/google-nest-audio-speaker/review-focusing-on-the-sound/, 2020.  Last accessed 17 August 2022.
  * (2) Google nest mini speaker review.  https://www.rtings.com/speaker/reviews/google/nest-mini, 2021.  Last accessed 17 August 2022.
  * (3) Half of us internet users report smart speaker ownership.  https://www.marketingcharts.com/industries/retail-and-e-commerce-119306, 2021.  Last accessed 17 August 2022.
  * (4) Sonos move speaker review.  https://www.rtings.com/speaker/reviews/sonos/move, 2021.  Last accessed 17 August 2022.
  * (5) 20khz speaker.  https://www.digikey.com/en/products/detail/pui-audio,-inc./ASX05408-HD-R/7227653utm_adgroup=Speakers&utm_source=google&utm_medium=cpc&utm_campaign=Shopping_Product_Audio, 2022.
  * (6) 60khz ultrasound speaker.  https://www.steminc.com/PZT/en/ultrasonic-air-transducer-60-khz, 2022.
  * (7) 80khz ultrasound speaker.  https://www.steminc.com/PZT/en/ultrasonic-air-transducer-80-khz, 2022.
  * (8) Amazon alexa speechrecognizer api.  https://developer.amazon.com/en-US/docs/alexa/alexa-voice-service/speechrecognizer.html, 2022.
  * (9) Google speech-to-text api.  https://cloud.google.com/speech-to-text, 2022.
  * (10) High speed linear actuator.  https://dovermotion.com/products/legacy-products/high-speed-linear-actuator-sas/, 2022.
  * (11) Pmm-3738-vm1010-r mems microphone.  https://www.puiaudio.com/products/pmm-3738-vm1010-r, 2022.
  * (12) Smart speakers - statistics and facts.  https://www.statista.com/topics/4748/smart-speakers/#topicHeader__wrapper, 2022.  Last accessed 17 August 2022.
  * (13) Acosta, D. A., Ruiz, O. E., Arroyave, S., Ebratt, R., Cadavid, C., and Londono, J. J. Geodesic-based manifold learning for parameterization of triangular meshes.  International Journal on Interactive Design and Manufacturing (IJIDeM) 10, 4 (2016), 417–430.
  * (14) Ahmed, S., Chowdhury, A. R., Fawaz, K., and Ramanathan, P. Preech: a system for privacy-preserving speech transcription.  In 29th $\\{$USENIX$\\}$ Security Symposium ($\\{$USENIX$\\}$ Security 20) (2020), pp. 2703–2720.
  * (15) Bai, Y., Liu, J., Lu, L., Yang, Y., Chen, Y., and Yu, J. Batcomm: enabling inaudible acoustic communication with high-throughput for mobile devices.  In Proceedings of the 18th Conference on Embedded Networked Sensor Systems (2020), pp. 205–217.
  * (16) Caixeta, R. M., and Costa, J. F. C. L. A robust unfolding approach for 3-d domains.  Computers & Geosciences 155 (2021), 104844.
  * (17) Cao, G., Yuan, K., Xiong, J., Yang, P., Yan, Y., Zhou, H., and Li, X.-Y. Earphonetrack: involving earphones into the ecosystem of acoustic motion tracking.  In Proceedings of the 18th Conference on Embedded Networked Sensor Systems (2020), pp. 95–108.
  * (18) Cao, Y., Dhekne, A., and Ammar, M. Itracku: tracking a pen-like instrument via uwb-imu fusion.  In Proceedings of the 19th Annual International Conference on Mobile Systems, Applications, and Services (2021), pp. 453–466.
  * (19) Chan, J., Rea, T., Gollakota, S., and Sunshine, J. E. Contactless cardiac arrest detection using smart devices.  NPJ digital medicine 2, 1 (2019), 1–8.
  * (20) Chang, L., Xiong, J., Wang, J., Chen, X., Wang, Y., Tang, Z., and Fang, D. Rf-copybook: A millimeter level calligraphy copybook based on commodity rfid.  Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 4 (2018), 1–19.
  * (21) Chuo, L.-X., Luo, Z., Sylvester, D., Blaauw, D., and Kim, H.-S. Rf-echo: A non-line-of-sight indoor localization system using a low-power active rf reflector asic tag.  In Proceedings of the 23rd Annual International Conference on Mobile Computing and Networking (2017), pp. 222–234.
  * (22) Co., M. M. Ultrasound speaker.  https://www.murata.com/-/media/webrenewal/products/sensor/ultrasonic/open/datasheet_maopn.ashx, 2020.
  * (23) DeGoede, K. M., Ashton-Miller, J. A., Liao, J. M., and Alexander, N. B. How quickly can healthy adults move their hands to intercept an approaching object? age and gender effects.  The Journals of Gerontology Series A: Biological Sciences and Medical Sciences 56, 9 (2001), M584–M588.
  * (24) Devices, A. Admp401: Omnidirectional microphone with bottom port and analog output, 2013.
  * (25) Doukhnitch, E., Salamah, M., and Ozen, E. An efficient approach for trilateration in 3d positioning.  Computer communications 31, 17 (2008), 4124–4129.
  * (26) Farrukh, H., Yang, T., Xu, H., Yin, Y., Wang, H., and Celik, Z. B. S3: Side-channel attack on stylus pencil through sensors.  Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5, 1 (2021), 1–25.
  * (27) Floater, M. S., and Hormann, K. Surface parameterization: a tutorial and survey.  Advances in multiresolution for geometric modelling (2005), 157–186.
  * (28) Garcia-Romero, D., and Espy-Wilson, C. Y. Analysis of i-vector length normalization in speaker recognition systems.  In Twelfth annual conference of the international speech communication association (2011).
  * (29) Gupta, S., Morris, D., Patel, S., and Tan, D. Soundwave: using the doppler effect to sense gestures.  In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (2012), pp. 1911–1914.
  * (30) Huang, W., Xiong, Y., Li, X.-Y., Lin, H., Mao, X., Yang, P., and Liu, Y. Shake and walk: Acoustic direction finding and fine-grained indoor localization using smartphones.  In IEEE INFOCOM 2014-IEEE Conference on Computer Communications (2014), IEEE, pp. 370–378.
  * (31) Jiang, C., He, Y., Yang, S., Guo, J., and Liu, Y. 3d-omnitrack: 3d tracking with cots rfid systems.  In 2019 18th ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN) (2019), IEEE, pp. 25–36.
  * (32) Jin, H., Wang, J., Yang, Z., Kumar, S., and Hong, J. Rf-wear: Towards wearable everyday skeleton tracking using passive rfids.  In Proceedings of the 2018 ACM International Joint Conference and 2018 International Symposium on Pervasive and Ubiquitous Computing and Wearable Computers (2018), pp. 369–372.
  * (33) Jin, H., Wang, J., Yang, Z., Kumar, S., and Hong, J. Wish: Towards a wireless shape-aware world using passive rfids.  In Proceedings of the 16th Annual International Conference on Mobile Systems, Applications, and Services (2018), pp. 428–441.
  * (34) Keysight.  https://www.keysight.com/us/en/products/waveform-and-function-generators.html, 2020\.
  * (35) Kotaru, M., Zhang, P., and Katti, S. Localizing low-power backscatter tags using commodity wifi.  In Proceedings of the 13th International Conference on emerging Networking EXperiments and Technologies (2017), pp. 251–262.
  * (36) Li, X., Zhang, Y., and Amin, M. G. Multifrequency-based range estimation of rfid tags.  In 2009 IEEE International Conference on RFID (2009), IEEE, pp. 147–154.
  * (37) Lin, Q., An, Z., and Yang, L. Rebooting ultrasonic positioning systems for ultrasound-incapable smart devices.  In The 25th Annual International Conference on Mobile Computing and Networking (2019), pp. 1–16.
  * (38) Liu, Y., Huang, K., Song, X., Yang, B., and Gao, W. Maghacker: eavesdropping on stylus pen writing via magnetic sensing from commodity mobile devices.  In Proceedings of the 18th International Conference on Mobile Systems, Applications, and Services (2020), pp. 148–160.
  * (39) Luo, Z., Zhang, Q., Ma, Y., Singh, M., and Adib, F. 3d backscatter localization for $\\{$Fine-Grained$\\}$ robotics.  In 16th USENIX Symposium on Networked Systems Design and Implementation (NSDI 19) (2019), pp. 765–782.
  * (40) Ma, Y., Hui, X., and Kan, E. C. 3d real-time indoor localization via broadband nonlinear backscatter in passive devices with centimeter precision.  In Proceedings of the 22nd Annual International Conference on Mobile Computing and Networking (2016), pp. 216–229.
  * (41) Ma, Y., Selby, N., and Adib, F. Minding the billions: Ultra-wideband localization for deployed rfid tags.  In Proceedings of the 23rd annual international conference on mobile computing and networking (2017), pp. 248–260.
  * (42) Mao, W., He, J., and Qiu, L. Cat: high-precision acoustic motion tracking.  In Proceedings of the 22nd Annual International Conference on Mobile Computing and Networking (2016), pp. 69–81.
  * (43) Mirjalili, S. Genetic algorithm.  In Evolutionary algorithms and neural networks. Springer, 2019, pp. 43–55.
  * (44) Nandakumar, R., Gollakota, S., and Watson, N. Contactless sleep apnea detection on smartphones.  In Proceedings of the 13th annual international conference on mobile systems, applications, and services (2015), pp. 45–57.
  * (45) Nandakumar, R., Iyer, V., and Gollakota, S. 3d localization for sub-centimeter sized devices.  In Proceedings of the 16th ACM Conference on Embedded Networked Sensor Systems (2018), pp. 108–119.
  * (46) Nandakumar, R., Iyer, V., Tan, D., and Gollakota, S. Fingerio: Using active sonar for fine-grained finger tracking.  In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (2016), pp. 1515–1525.
  * (47) Processing, S., and Laboratory, S. C. Pitch tracking database.  https://www.spsc.tugraz.at/databases-and-tools/ptdb-tug-pitch-tracking-database-from-graz-university-of-technology.html, 2019.
  * (48) Ramesh, S., Xiao, R., Maiti, A., Lee, J. T., Ramprasad, H., Kumar, A., Jadliwala, M., and Han, J. Acoustics to the rescue: Physical key inference attack revisited.  In 30th USENIX Security Symposium (USENIX Security 21) (2021), pp. 3255–3272.
  * (49) Roy, N., Hassanieh, H., and Roy Choudhury, R. Backdoor: Making microphones hear inaudible sounds.  In Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services (2017), pp. 2–14.
  * (50) Ruan, W., Sheng, Q. Z., Yang, L., Gu, T., Xu, P., and Shangguan, L. Audiogest: enabling fine-grained hand gesture detection by decoding echo signal.  In Proceedings of the 2016 ACM international joint conference on pervasive and ubiquitous computing (2016), pp. 474–485.
  * (51) Sheffer, A., Praun, E., Rose, K., et al. Mesh parameterization methods and their applications.  Foundations and Trends® in Computer Graphics and Vision 2, 2 (2007), 105–171.
  * (52) Shen, S., Chen, D., Wei, Y.-L., Yang, Z., and Choudhury, R. R. Voice Localization Using Nearby Wall Reflections.  Association for Computing Machinery, New York, NY, USA, 2020.
  * (53) Shen, S., Chen, D., Wei, Y.-L., Yang, Z., and Choudhury, R. R. Voice localization using nearby wall reflections.  In Proceedings of the 26th Annual International Conference on Mobile Computing and Networking (2020), pp. 1–14.
  * (54) Synertial.  Synertial motion capture.  https://www.synertial.com/, 2021.
  * (55) Taori, R., Kamsetty, A., Chu, B., and Vemuri, N. Targeted adversarial examples for black box audio systems.  In 2019 IEEE security and privacy workshops (SPW) (2019), IEEE, pp. 15–20.
  * (56) Technologies, X. Xsens motion capture.  https://www.xsens.com/, 2021.
  * (57) Tenenbaum, J. B., Silva, V. d., and Langford, J. C. A global geometric framework for nonlinear dimensionality reduction.  science 290, 5500 (2000), 2319–2323.
  * (58) UK, V. M. S. L. Vicon motion systems.  https://www.vicon.com/, 2022.
  * (59) Vaidya, T., and Sherr, M. You talk too much: Limiting privacy exposure via voice input.  In 2019 IEEE Security and Privacy Workshops (SPW) (2019), IEEE, pp. 84–91.
  * (60) Vasisht, D., Zhang, G., Abari, O., Lu, H.-M., Flanz, J., and Katabi, D. In-body backscatter communication and localization.  In Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication (2018), pp. 132–146.
  * (61) Wang, A., and Gollakota, S. Millisonic: Pushing the limits of acoustic motion tracking.  In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (2019), pp. 1–11.
  * (62) Wang, A., Nguyen, D., Sridhar, A. R., and Gollakota, S. Using smart speakers to contactlessly monitor heart rhythms.  Communications biology 4, 1 (2021), 1–12.
  * (63) Wang, A., Sunshine, J. E., and Gollakota, S. Contactless infant monitoring using white noise.  In The 25th Annual International Conference on Mobile Computing and Networking (2019), pp. 1–16.
  * (64) Wang, J., and Katabi, D. Dude, where’s my card? rfid positioning that works with multipath and non-line of sight.  In Proceedings of the ACM SIGCOMM 2013 conference on SIGCOMM (2013), pp. 51–62.
  * (65) Wang, J., Pan, C., Jin, H., Singh, V., Jain, Y., Hong, J. I., Majidi, C., and Kumar, S. Rfid tattoo: A wireless platform for speech recognition.  Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 3, 4 (2019), 1–24.
  * (66) Wang, J., Zhang, J., Li, K., Pan, C., Majidi, C., and Kumar, S. Locating everyday objects using nfc textiles.  In Proceedings of the 20th International Conference on Information Processing in Sensor Networks (co-located with CPS-IoT Week 2021) (2021), pp. 15–30.
  * (67) Wang, M., Sun, W., and Qiu, L. $\\{$MAVL$\\}$: Multiresolution analysis of voice localization.  In 18th USENIX Symposium on Networked Systems Design and Implementation (NSDI 21) (2021), pp. 845–858.
  * (68) Wang, W., Liu, A. X., and Sun, K. Device-free gesture tracking using acoustic signals.  In Proceedings of the 22nd Annual International Conference on Mobile Computing and Networking (2016), pp. 82–94.
  * (69) Westervelt, P. J. The theory of steady forces caused by sound waves.  The Journal of the Acoustical Society of America 23, 3 (1951), 312–315.
  * (70) Westervelt, P. J. Scattering of sound by sound.  The Journal of the Acoustical Society of America 29, 2 (1957), 199–203.
  * (71) Xiao, F., Wang, Z., Ye, N., Wang, R., and Li, X.-Y. One more tag enables fine-grained rfid localization and tracking.  IEEE/ACM Transactions on Networking 26, 1 (2017), 161–174.
  * (72) Xu, X., Yu, J., Chen, Y., Zhu, Y., Kong, L., and Li, M. Breathlistener: Fine-grained breathing monitoring in driving environments utilizing acoustic signals.  In Proceedings of the 17th Annual International Conference on Mobile Systems, Applications, and Services (2019), pp. 54–66.
  * (73) Yang, L., Chen, Y., Li, X.-Y., Xiao, C., Li, M., and Liu, Y. Tagoram: Real-time tracking of mobile rfid tags to high precision using cots devices.  In Proceedings of the 20th annual international conference on Mobile computing and networking (2014), pp. 237–248.
  * (74) Yousaf, M., Rehman, T. U., and Jing, L. An extended isomap approach for nonlinear dimension reduction.  SN Computer Science 1, 3 (2020), 1–10.
  * (75) Yun, S., Chen, Y.-C., and Qiu, L. Turning a mobile device into a mouse in the air.  In Proceedings of the 13th Annual International Conference on Mobile Systems, Applications, and Services (2015), pp. 15–29.
  * (76) Zhang, C., Xue, Q., Waghmare, A., Jain, S., Pu, Y., Hersek, S., Lyons, K., Cunefare, K. A., Inan, O. T., and Abowd, G. D. Soundtrak: Continuous 3d tracking of a finger using active acoustics.  Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 2 (2017), 1–25.
  * (77) Zhang, F., Wang, Z., Jin, B., Xiong, J., and Zhang, D. Your smart speaker can” hear” your heartbeat!  Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 4 (2020), 1–24.
  * (78) Zhang, G., Yan, C., Ji, X., Zhang, T., Zhang, T., and Xu, W. Dolphinattack: Inaudible voice commands.  In Proceedings of the 2017 ACM SIGSAC conference on computer and communications security (2017), pp. 103–117.
  * (79) Zhou, K., Synder, J., Guo, B., and Shum, H.-Y. Iso-charts: stretch-driven mesh parameterization using spectral analysis.  In Proceedings of the 2004 Eurographics/ACM SIGGRAPH symposium on Geometry processing (2004), pp. 45–54.