# Quantum coherence effects on inelastic thermoelectric devices: From diodes
to transistors

Bei Cao  Jiangsu Key Laboratory of Micro and Nano Heat Fluid Flow Technology
and Energy Application, School of Physical Science and Technology, Suzhou
University of Science and Technology, Suzhou, 215009, China  Chongze Han
Jiangsu Key Laboratory of Micro and Nano Heat Fluid Flow Technology and Energy
Application, School of Physical Science and Technology, Suzhou University of
Science and Technology, Suzhou, 215009, China  Xiang Hao  Jiangsu Key
Laboratory of Micro and Nano Heat Fluid Flow Technology and Energy
Application, School of Physical Science and Technology, Suzhou University of
Science and Technology, Suzhou, 215009, China  Chen Wang  Department of
Physics, Zhejiang Normal University, Jinhua, Zhejiang 321004, China  Jincheng
Lu<EMAIL_ADDRESS>Jiangsu Key Laboratory of Micro and Nano Heat Fluid
Flow Technology and Energy Application, School of Physical Science and
Technology, Suzhou University of Science and Technology, Suzhou, 215009, China

###### Abstract

We present a study on inelastic thermoelectric devices, wherein charge
currents and electronic and phononic heat currents are intricately
interconnected. The employment of double quantum dots in conjunction with a
phonon bath positions them as promising candidates for quantum thermoelectric
diodes and transistors. Within this study, we illustrate that quantum
coherence effects yield significant charge and Seebeck rectification effects.
It’s worth noting that, while the thermal transistor effect is observable in
the linear response regime, especially when phonon-assisted inelastic
processes dominate the transport, quantum coherence does not enhance thermal
amplification. Our work provides valuable insights for the optimization of
general thermoelectric devices.

##  I Introduction

Phonon-based thermoelectricity has garnered significant research attention,
owing to their relevance in both fundamental physics and cutting-edge energy
applications Li  _et al._ (2012); Thierschmann  _et al._ (2015); Benenti  _et
al._ (2017); Arrachea (2023). These studies sit at the intersection of
mesoscopic physics and open quantum systems, exploring topics such as
thermoelectric energy conversion in phonon-thermoelectric devices Jiang  _et
al._ (2012); Jiang and Imry (2017), and quantum dot circuit-quantum-
electrodynamics systems Ren  _et al._ (2012); Liu  _et al._ (2019); Lu  _et
al._ (2023a). Recent research has uncovered various quantum phenomena in
quantum dots subjected to voltage bias and temperature gradients, revealing
quantum coherence effects in electron/phonon transport Bu  _et al._ (2017);
Brandner  _et al._ (2017); Camati  _et al._ (2019); Liu and Segal (2021);
Aimet and Kwon (2023); Behera  _et al._ (2023). Coherence offers a promising
viewpoint for investigating non-equilibrium open quantum systems, providing
valuable insights for both theoretical foundations and practical applications
Scully  _et al._ (2003); Engel  _et al._ (2007); Vaziri and Plenio (2010);
Sánchez  _et al._ (2021); Ivander  _et al._ (2022). However, a significant gap
in the field lies in the exploration of phonon-thermoelectric devices with
quantum coherence effects as potential diodes and transistors.

Recent years have witnessed an emerging field of research focusing on
inelastic thermoelectricity, which has opened up new possibilities for high-
performance thermoelectric materials Wang  _et al._ (2022); Entin-Wohlman  _et
al._ (2010); Lu  _et al._ (2021); Tesser  _et al._ (2023). The control and
separation of heat and electrical currents on mesoscopic scales have gained
fundamental importance, as underscored in recent investigations into inelastic
thermoelectric transport Pekola and Karimi (2021); Lü  _et al._ (2016); Li
_et al._ (2021); Nian  _et al._ (2023). In this paper, we delve into the realm
of inelastic thermoelectric transport facilitated by phonons and demonstrate
that quantum coherence provides a promising avenue for realizing quantum
thermoelectric devices. Employing the quantum master equation with full
counting statistics approach, we showcase how the nonlinearity induced by
electron-phonon interactions allows for significant charge and Seebeck
rectification effects through careful tuning of the quantum dot’s energy
levels. Furthermore, we illustrate that a three-terminal setup can exhibit
thermal transistor effects even within the linear transport regime, although
heat amplification effects do not benefit from quantum coherence. Our work
establishes a robust platform offering unprecedented control over both charge
and heat. Throughout this paper, we employ natural units, setting
$\hbar=k_{B}=e\equiv 1$.

Figure 1: (a) Illustration of the three-terminal inelastic thermoelectric
devices. An electron left the source into the left quantum dot with energy
$E_{l}$ hops to the right quantum dot with a different energy $E_{r}$ assisted
by a phonon from the phonon bath (with temperature $T_{\rm ph}$). The
temperatures and chemical potentials of two electric reservoirs are $T_{L(R)}$
and $\mu_{L(R)}$, respectively. The temperature of phonon bath is $T_{\rm
ph}$. $t$ is the tunneling between the two quantum dots, $\gamma_{l}$
($\gamma_{r}$) is the coupling between the dots and the left (right)
reservoir, $\lambda_{q}$ is the strength of electron-phonon interaction. (b)
The charge current $I_{e}$ as a function of level detuning $E_{r}-E_{l}$. The
other parameters are $\mu=6\,{\rm meV}$, $\Delta\mu=2\,{\rm meV}$, $E_{l}=0$,
$\Gamma_{L}=\Gamma_{R}=\Gamma_{\rm ph}=6\,{\rm meV}$, $t=8\,{\rm meV}$,
$k_{B}T_{L}=k_{B}T_{R}=k_{B}T_{\rm ph}=10\,{\rm meV}$.

##  II Three-terminal double-quantum-dot system

In our construction (see Fig. 1(a)), a double quantum dot system is coupled to
a phononic heat bath ($\rm ph$) and exchanging energy with an electronic
reservoirs, $L$ and $R$. Our model is described by the Hamiltonian Jiang  _et
al._ (2012)

$\hat{H}=\hat{H}_{\rm DQD}+\hat{H}_{\rm e-ph}+\hat{H}_{\rm lead}+\hat{H}_{\rm
tun}+\hat{H}_{\rm ph},$ (1)

with $\hat{H}_{\rm
DQD}=\sum_{i=l,r}E_{i}\hat{d}_{i}^{\dagger}\hat{d}_{i}+t(\hat{d}_{l}^{\dagger}\hat{d}_{r}+{\rm
H.c.})$ describes the quantum dots with their energies $E_{i}$, tunneling $t$,
and electron operators, $\hat{H}_{\rm
e-ph}=\sum_{q}\lambda_{q}\hat{d}_{l}^{\dagger}\hat{d}_{r}(\hat{a}_{q}+\hat{a}^{\dagger}_{q})+{\rm
H.c.}$ accounts for electron-phonon interactions, $\hat{H}_{\rm
ph}=\sum_{q}\omega_{q}\hat{a}^{\dagger}_{q}\hat{a}_{q}$ represents the phonon
bath, $\hat{H}_{\rm
lead}=\sum_{j=L,R}\sum_{k}\varepsilon_{jk}\hat{d}_{jk}^{\dagger}\hat{d}_{jk}$
characterizes the electronic reservoirs’ energy levels, $\hat{H}_{\rm
tun}=\sum_{k}\gamma_{lk}\hat{d}_{l}^{\dagger}\hat{d}_{Lk}+\sum_{k}\gamma_{rk}\hat{d}_{r}^{\dagger}\hat{d}_{Rk}+{\rm
H.c.}$ involves tunneling between the quantum dots and the reservoirs.

To describe the system quantum mechanically, a characteristic function
$\mathcal{Z}(\lambda_{e},\lambda_{E},\lambda_{\rm ph})$ is defined with
counting parameters for charge ($\lambda_{e}$), energy ($\lambda_{E}$), and
phonons $\lambda_{\rm ph}$. Time evolution follows the Heisenberg
representation, and averages are taken with respect to an initial density
matrix factored between the system and baths Esposito  _et al._ (2009);
Campisi  _et al._ (2011). The quantum dots’ eigenstates ($|\chi\rangle$) and
their populations ($|{\mathbf{P}}\rangle$ = $\langle 0|\rho_{S}|0\rangle$,
$\langle D|\rho_{S}|D\rangle$, $\langle d|\rho_{S}|d\rangle$, $\langle
D|\rho_{S}|d\rangle$, $\langle d|\rho_{S}|D\rangle$) evolve according to a
matrix equation

$\displaystyle\dfrac{d|{\mathbf{P}}^{\lambda}(t)\rangle}{dt}={\mathbf{H}}(\lambda)|{\mathbf{P}}^{\lambda}(t)\rangle,$
(2)

where $|{\mathbf{P}}^{\lambda}(t)\rangle$ is a $5\times 5$ matrix (the
detailed expressions see Ref. Lu  _et al._ (2023b)). The charge current
flowing from the drain into the system emerges as

$I_{e}^{R}=\frac{\partial{\mathcal{G}}}{\partial(i\lambda_{e})}|_{\lambda=0},$
(3)

The symbol ${\mathcal{G}}$ is the cumulant generating function and signifies
the eigenvalues of the evolution matrix ${\mathbf{H}}$, specifically those
with the largest real part Wang  _et al._ (2017). The energy current is

$I_{E}^{R}=\frac{\partial{\mathcal{G}}}{\partial(i\lambda_{E})}|_{\lambda=0}.$
(4)

The electronic heat current extracted from the right reservoir is defined as
$I_{Q}^{R}=I_{E}^{R}-\mu_{R}I_{e}^{R}$ Yamamoto and Hatano (2015). The
phononic heat current is

$I_{Q}^{\rm ph}=\frac{\partial{\mathcal{G}}}{\partial(i\lambda_{\rm
ph})}|_{\lambda=0}.$ (5)

Similarly, the particle current $I_{e}^{L}$ and energy current $I_{E}^{L}$
flowing from the left ($L$) reservoir into the central system can also be
obtained by introducing the counting functions. Charge conservation implies
that $I_{e}^{L}+I_{e}^{R}=0$, while the energy conservation requires that
$I_{E}^{L}+I_{E}^{R}+I_{Q}^{\rm ph}=0$ Jiang and Imry (2016). The net charge
current flowing from the left reservoir to the right one is then

$I_{e}=\frac{1}{2}(I_{e}^{R}-I_{e}^{R}),$ (6)

and the net electronic heat current exchanged between the left and right
reservoir is

$I_{Q}^{e}=\frac{1}{2}(I_{Q}^{R}-I_{Q}^{R}).$ (7)

Before delving into detailed discussions, it is essential to note that the
methods employed in our study rely on weak system-reservoir couplings, denoted
as $\gamma_{i}$ ($i=l,r,\mathrm{ph}\ll k_{B}T$). In a prior work Lu  _et al._
(2023b), we showcased the feasibility of periodically driven thermoelectric
engines in a three-terminal double quantum dot system. Notably, we observed a
substantial enhancement in both output work and efficiency due to the presence
of quantum coherence. In Figure 1(b), with a fixed chemical potential
difference of $\Delta\mu=2,\mathrm{meV}$, we present the charge current as a
function of the quantum dot energy difference $E_{r}-E_{l}$ for both coherent
and incoherent transport. It is evident that quantum coherence, characterized
by nonzero off-diagonal elements in the system’s density matrix, leads to a
notable improvement in the charge current when compared to the scenarios
without quantum coherence.

##  III Rectification (diode) effects

In this section, we will illustrate that these three-terminal configurations
effectively serve as rectifiers Yang  _et al._ (2018); Zhang and Su (2021);
Wang  _et al._ (2019); Diaz and Sanchez (2021); Tesser  _et al._ (2022);
Khandelwal  _et al._ (2023). Our system, renowned for its combined thermal and
electrical transport capabilities, offers the potential for unconventional
rectification phenomena. Notably, it can induce charge rectification owing to
a temperature difference, in addition to showcasing the conventional charge
and heat rectification effects Zhang  _et al._ (2010); Ren and Zhu (2013). As
we have observed, in a nonequilibrium setup, both charge and heat currents are
defined, and their rectification behavior can exhibit significant variations.
It’s important to note that rectification effects become observable only
beyond the linear response regime Yang  _et al._ (2023a). The magnitude of the
rectification effect is defined by

$\displaystyle
R_{e}=\frac{I_{e}(\Delta\mu)+I_{e}(-\Delta\mu)}{|I_{e}(\Delta\mu)|+|I_{e}(-\Delta\mu)|},$
(8)

for charge rectification induced by chemical potential, and the rectification
induced by temperature gradient, termed as Seebeck rectification, isJiang  _et
al._ (2015)

$\displaystyle R_{et}=\frac{I_{e}(\Delta T)+I_{e}(-\Delta T)}{|I_{e}(\Delta
T)|+|I_{e}(-\Delta T)|},$ (9)

In Fig. 2 we demonstrate both charge and Seebeck-rectification effects in
three-terminal quantum dot systems. Obviously, from the Figs. 2(a) and 2(b),
we can see that the elastic heat and charge currents (i.e., $\Gamma_{\rm
ph}=0$) are almost symmetrical but the inelastic currents i.e., $\Gamma_{\rm
ph}\neq 0$) are asymmetric. The elastic currents are antisymmetric with
respect to forward and backward voltage bias and the inelastic transport
processes induced by the electron-phonon interaction can break the mirror
symmetry. The asymmetric behavior in the magnitude of inelastic current that
is the reason for both the charge and Seebeck-rectification effects Jiang  _et
al._ (2015). Figures 2(c) -2(f) give the dependencies of the rectification
effects on the quantum dots energies $E_{l}$ and $E_{r}$. The red and blue
areas in the figure indicate that rectification has achieved good results.
Both the charge $R_{e}$ and Seebeck $R_{et}$ rectification coefficients are
antisymmetric around the line of $E_{l}$ and $E_{r}$.

Furthermore, we can make a comparative analysis between the charge
rectification and Seebeck rectification effects in coherent and incoherent
scenarios. As depicted in Figs.2(c)-2(e) and Figs.2(d)-2(f), it becomes
evident that the rectification effect in thermoelectric devices, when
considering quantum coherence, is significantly more pronounced than in
incoherent cases. Hence, we can conclude that quantum coherence not only
amplifies the current magnitudes but also intensifies the asymmetry of
inelastic currents, thereby enhancing the rectification effects.

Figure 2: Charge, and Seebeck-rectification effects. (a) Charge current
$I_{e}$ as a function of $\Delta\mu$ for different $\Gamma_{\rm ph}$. The
parameters are $\mu=0$, $E_{l}=5\,{\rm meV}$, $E_{r}=7\,{\rm meV}$,
$k_{B}T_{\rm ph}=10\,{\rm meV}$. (b) Charge current $I_{e}$ as a function of
$\Delta T$ for different $\Gamma_{\rm ph}$, where $\Delta\mu=0$,
$E_{l}=5\,{\rm meV}$, $E_{r}=7\,{\rm meV}$. The charge rectification $R_{e}$
as functions of quantum dot energies $E_{l}$ and $E_{r}$ for (c) coherent
transport and (e) incoherent transport, where $\Delta\mu=10\,{\rm meV}$ and
$k_{B}T_{\rm ph}=10\,{\rm meV}$. The Seebeck-rectification $R_{et}$ as
functions of quantum dot energies $E_{l}$ and $E_{r}$ for (d) coherent
transport and (f) incoherent transport,, where $\Delta\mu=0$ and $k_{B}T_{\rm
ph}=15\,{\rm meV}$. The other parameters are $\Gamma_{L}=\Gamma_{R}=6\,{\rm
meV}$, $t=8\,{\rm meV}$, $k_{B}T_{L}=k_{B}T_{R}=10\,{\rm meV}$.  Figure 3: (a)
Schematic outlines the functioning of a conventional transistor. A small
charge current originating from the emitter to the base governs a
significantly larger charge current between the emitter and collector. The
ratio between these two charge currents is the defining characteristic of the
transistor effect (b) Scheme of a three-terminal quantum-dot system that can
serve as a phonon-thermoelectric transistor. Here, a modest heat current
travels from the source lead to the phonon bath ($I_{Q}^{\rm ph}$), and it
exerts control over a substantial heat current flowing from the source lead to
the drain lead ($I_{Q}^{R}$). This ratio between the two heat currents is
known as the heat current amplification factor ($\alpha$) and serves as the
hallmark of the thermal transistor effect. (c) The heat current amplification
factor $\alpha$ as a function of quantum dot energy difference $E_{l}-E_{r}$
for the cases of coherence and incoherence. The other parameters are
$\mu=6\,{\rm meV}$, $\Delta\mu=0$, $E_{l}=12\,{\rm meV}$,
$\Gamma_{L}=\Gamma_{R}=\Gamma_{\rm ph}=6\,{\rm meV}$, $t=8\,{\rm meV}$,
$k_{B}T_{L}=k_{B}T_{R}=k_{B}T_{\rm ph}=10\,{\rm meV}$. (d) The heat current
amplification factor $\alpha$ as functions of quantum dot energy $E_{l}$ and
$E_{r}$ The other parameters are same with Fig. 3(c).

##  IV Thermal transistor effect in the linear-response regime

Heat amplification, a phenomenon where a minor adjustment in the base current
yields a substantial impact on the current flowing between the collector and
emitter, has led to the development of quantum thermal transistors, enabling
precise control over energy transport Joulain  _et al._ (2016); Guo  _et al._
(2018); Yang  _et al._ (2023b). As depicted in Fig.3(a), this schematic
outlines the operation of a conventional transistor. In our research, we
investigate three-terminal quantum-dot systems as thermal transistors, as
depicted in Fig.3(b).

Traditionally, it was widely accepted that nonlinear transport was a
prerequisite for thermal transistor effects. Negative differential thermal
conductance was considered a key condition for the emergence of such effects,
as outlined in Li  _et al._ (2012, 2004, 2006). However, a groundbreaking
perspective was introduced in Ref.Jiang  _et al._ (2015), proposing that
thermal transistor effects could occur in the linear-transport regime,
particularly when dominated by phonon-assisted inelastic transport. In our
subsequent work Lu  _et al._ (2019, 2023a), we further demonstrated the
existence of linear thermal transistor effects in circuit-quantum-
electrodynamics systems across a wide range of quantum dot energies and light-
matter interactions. Furthermore, we showed that thermal transistor effects
can manifest even without these conditions in Brownian thermal transistors.
Analyzing the statistical distributions of the thermal amplification factor
within the Gaussian fluctuation framework provided further insights Lu  _et
al._ (2020). However, it’s important to note that all these studies were
limited to the quantum incoherent regime, and the influence of quantum
coherence on the thermal transistor effect remains unexplored.

In the scenario of exclusively thermal conduction (with the electrochemical
potential difference set to zero), the system’s linear thermal transport
properties can be aptly described by Sánchez  _et al._ (2017); Wang  _et al._
(2018)

$\displaystyle\left(\begin{array}[]{cccc}I_{Q}^{R}\\\ I_{Q}^{\rm
ph}\end{array}\right)=\left(\begin{array}[]{cccc}K_{RR}&K_{RP}\\\
K_{PR}&K_{PP}\end{array}\right)\left(\begin{array}[]{cccc}T_{R}-T_{L}\\\
T_{\rm ph}-T_{L}\end{array}\right),$ (10)

One of our key findings is the potential for a thermal transistor effect to
emerge within the linear-response regime. To be more specific, the
amplification factor for heat current is defined as follows:

$\displaystyle\alpha=\Big{|}\frac{\partial_{T_{\rm
ph}}I_{Q}^{R}}{\partial_{T_{\rm ph}}I_{Q}^{P}}\Big{|}=\frac{K_{RP}}{K_{PP}},$
(11)

As demonstrated in Figures 3(c) and 3(d), we have observed that the
coefficient $\alpha$ can be greater than 1 due to the inelastic transport
process and displays high sensitivity to the quantum dot energies, $E_{l}$ and
$E_{r}$, which can be easily manipulated through gate voltages in experimental
setups. Initially, we observed that the thermal amplification effect could be
achieved within a specific parameter range when $E_{r}-E_{l}>8\,\rm{meV}$ in
the context of incoherent transport. However, in the case of thermal
transistors accounting for quantum coherence, while thermal amplification
effects remain achievable, the range where $\alpha>1$ becomes notably
restricted, specifically within
$11\,\mathrm{meV}<E_{r}-E_{l}<13\,\mathrm{meV}$. In accordance with the
findings in Ref.Jiang  _et al._ (2015), the thermal amplification coefficient
$\alpha$ is determined as $\alpha\,=\,|E_{l}/(E_{r}-E_{l})|$ in the regime of
weak electron-phonon coupling using the Fermi Golden rule method. In this
context, the primary driver of the transistor effect is the ratio of
incoherent thermal conductivity. Therefore, when accounting for the quantum
coherence effect, although thermal amplification remains feasible within the
linear response range, as depicted in Fig.3(c), it narrows the parameter space
for achieving the thermal transistor effect, which is not advantageous. In
summary, we conclude that quantum coherence effects do not facilitate the
thermal transistor effect in the linear-transport regime, despite their
significant impact on currents, as illustrated in Fig. 1(b).

##  V Conclusion

In conclusion, this study delves into the intriguing domain of inelastic
thermoelectric devices, revealing their potential as quantum thermoelectric
diodes and transistors via inelastic transport processes induced by electron-
phonon interactions. We have demonstrated that precise adjustment of quantum
dot energy levels can yield significant charge and Seebeck rectification
effects, which are further enhanced when quantum coherence is considered.
Additionally, our research has unveiled the capability of a three-terminal
configuration to produce thermal transistor effects due to the inelastic
transport, even within the linear transport regime. However, it’s important to
note that quantum coherence does not contribute to heat amplification effects.
In summary, our study underscores the significance of quantum coherence in the
realm of inelastic thermoelectric devices and opens the door to innovative
applications in the field of quantum thermoelectricity.

##  VI Acknowledgments

This work was supported by the funding for the National Natural Science
Foundation of China under Grant Nos. 12305050, Jiangsu Key Disciplines of the
Fourteenth Five-Year Plan (Grant No. 2021135), the Natural Science Foundation
of Jiangsu Higher Education Institutions of China (Grant No. 23KJB140017), and
the Opening Project of Shanghai Key Laboratory of Special Artificial
Microstructure Materials and Technology.

## References

  * Li  _et al._ (2012) N. Li, J. Ren, L. Wang, G. Zhang, P. Hänggi, and B. Li, “Phononics: Manipulating heat flow with electronic analogs and beyond,” Rev. Mod. Phys. 84, 1045–1066 (2012).
  * Thierschmann  _et al._ (2015) H. Thierschmann, R. Sánchez, B. Sothmann, F. Arnold, C. Heyn, W. Hansen, H. Buhmann, and L. W. Molenkamp, “Three-terminal energy harvester with coupled quantum dots,” Nat. Nanotech. 10, 854 (2015).
  * Benenti  _et al._ (2017) G. Benenti, G. Casati, K. Saito, and R. S. Whitney, “Fundamental aspects of steady-state conversion of heat to work at the nanoscale,” Phys. Rep. 694, 1 (2017).
  * Arrachea (2023) L. Arrachea, “Energy dynamics, heat production and heat–work conversion with qubits: toward the development of quantum machines,” Reports on Progress in Physics 86, 036501 (2023).
  * Jiang  _et al._ (2012) J.-H. Jiang, O. Entin-Wohlman, and Y. Imry, “Thermoelectric three-terminal hopping transport through one-dimensional nanosystems,” Phys. Rev. B 85, 075412 (2012).
  * Jiang and Imry (2017) J.-H. Jiang and Y. Imry, “Enhancing thermoelectric performance using nonlinear transport effects,” Phys. Rev. Applied 7, 064001 (2017).
  * Ren  _et al._ (2012) J. Ren, J.-X. Zhu, J. E. Gubernatis, C. Wang, and B. Li, “Thermoelectric transport with electron-phonon coupling and electron-electron interaction in molecular junctions,” Phys. Rev. B 85, 155443 (2012).
  * Liu  _et al._ (2019) H. Liu, C. Wang, L.-Q. Wang, and J. Ren, “Strong system-bath coupling induces negative differential thermal conductance and heat amplification in nonequilibrium two-qubit systems,” Phys. Rev. E 99, 032114 (2019).
  * Lu  _et al._ (2023a) J. Lu, R. Wang, C. Wang, and J.-H. Jiang, “Thermoelectric rectification and amplification in interacting quantum-dot circuit-quantum-electrodynamics systems,” Entropy 25 (2023a), 10.3390/e25030498.
  * Bu  _et al._ (2017) K. Bu, U. Singh, S.-M. Fei, A. K. Pati, and J. Wu, “Maximum relative entropy of coherence: An operational coherence measure,” Phys. Rev. Lett. 119, 150405 (2017).
  * Brandner  _et al._ (2017) K. Brandner, M. Bauer, and U. Seifert, “Universal coherence-induced power losses of quantum heat engines in linear response,” Phys. Rev. Lett. 119, 170602 (2017).
  * Camati  _et al._ (2019) P. A. Camati, J. F. G. Santos, and R. M. Serra, “Coherence effects in the performance of the quantum otto heat engine,” Phys. Rev. A 99, 062103 (2019).
  * Liu and Segal (2021) J. Liu and D. Segal, “Coherences and the thermodynamic uncertainty relation: Insights from quantum absorption refrigerators,” Phys. Rev. E 103, 032138 (2021).
  * Aimet and Kwon (2023) S. Aimet and H. Kwon, “Engineering a heat engine purely driven by quantum coherence,” Phys. Rev. A 107, 012221 (2023).
  * Behera  _et al._ (2023) J. Behera, S. Bedkihal, B. K. Agarwalla, and M. Bandyopadhyay, “Quantum coherent control of nonlinear thermoelectric transport in a triple-dot aharonov-bohm heat engine,” Phys. Rev. B 108, 165419 (2023).
  * Scully  _et al._ (2003) M. O. Scully, M. S. Zubairy, G. S. Agarwal, and H. Walther, “Extracting work from a single heat bath via vanishing quantum coherence,” Science 299, 862–864 (2003).
  * Engel  _et al._ (2007) G. S. Engel, T. R. Calhoun, E. L. Read, T.-K. Ahn, T. Mančal, Y.-C. Cheng, R. E. Blankenship, and G. R. Fleming, “Evidence for wavelike energy transfer through quantum coherence in photosynthetic systems,” Nature 446, 782 (2007).
  * Vaziri and Plenio (2010) A. Vaziri and M. B. Plenio, “Quantum coherence in ion channels: resonances, transport and verification,” New J. Phys. 12, 085001 (2010).
  * Sánchez  _et al._ (2021) R. Sánchez, C. Gorini, and G. Fleury, “Extrinsic thermoelectric response of coherent conductors,” Phys. Rev. B 104, 115430 (2021).
  * Ivander  _et al._ (2022) F. Ivander, N. Anto-Sztrikacs, and D. Segal, “Quantum coherence-control of thermal energy transport: the v model as a case study,” New J. Phys. 24, 103010 (2022).
  * Wang  _et al._ (2022) R. Wang, C. Wang, J. Lu, and J.-H. Jiang, “Inelastic thermoelectric transport and fluctuations in mesoscopic systems,” Adv. Phys.: X 7, 2082317 (2022).
  * Entin-Wohlman  _et al._ (2010) O. Entin-Wohlman, Y. Imry, and A. Aharony, “Three-terminal thermoelectric transport through a molecular junction,” Phys. Rev. B 82, 115314 (2010).
  * Lu  _et al._ (2021) J. Lu, J.-H. Jiang, and Y. Imry, “Unconventional four-terminal thermoelectric transport due to inelastic transport: Cooling by transverse heat current, transverse thermoelectric effect, and maxwell demon,” Phys. Rev. B 103, 085429 (2021).
  * Tesser  _et al._ (2023) L. Tesser, R. S. Whitney, and J. Splettstoesser, “Thermodynamic performance of hot-carrier solar cells: A quantum transport model,” Phys. Rev. Appl. 19, 044038 (2023).
  * Pekola and Karimi (2021) J. P. Pekola and B. Karimi, “Colloquium: Quantum heat transport in condensed matter systems,” Rev. Mod. Phys. 93, 041001 (2021).
  * Lü  _et al._ (2016) J.-T. Lü, J.-S. Wang, P. Hedegård, and M. Brandbyge, “Electron and phonon drag in thermoelectric transport through coherent molecular conductors,” Phys. Rev. B 93, 205404 (2016).
  * Li  _et al._ (2021) Y. Li, W. Li, T. Han, X. Zheng, J. Li, B. Li, S. Fan, and C.-W. Qiu, “Transforming heat transfer with thermal metamaterials and devices,” Nature Reviews Materials 6, 488–507 (2021).
  * Nian  _et al._ (2023) L.-L. Nian, S. Hu, L. Xiong, J.-T. Lü, and B. Zheng, “Photon-assisted electron transport across a quantum phase transition,” Phys. Rev. B 108, 085430 (2023).
  * Esposito  _et al._ (2009) M. Esposito, U. Harbola, and S. Mukamel, “Nonequilibrium fluctuations, fluctuation theorems, and counting statistics in quantum systems,” Rev. Mod. Phys. 81, 1665–1702 (2009).
  * Campisi  _et al._ (2011) M. Campisi, P. Hänggi, and P. Talkner, “Colloquium: Quantum fluctuation relations: Foundations and applications,” Rev. Mod. Phys. 83, 771–791 (2011).
  * Lu  _et al._ (2023b) J. Lu, Z. Wang, J. Ren, C. Wang, and J.-H. Jiang, “Coherence-enhanced thermodynamic performance in periodically driven thermoelectric heat engines,” arXiv:2310.10465 (2023b).
  * Wang  _et al._ (2017) C. Wang, J. Ren, and J. Cao, “Unifying quantum heat transfer in a nonequilibrium spin-boson model with full counting statistics,” Phys. Rev. A 95, 023610 (2017).
  * Yamamoto and Hatano (2015) K. Yamamoto and N. Hatano, “Thermodynamics of the mesoscopic thermoelectric heat engine beyond the linear-response regime,” Phys. Rev. E 92, 042165 (2015).
  * Jiang and Imry (2016) J.-H. Jiang and Y. Imry, “Linear and nonlinear mesoscopic thermoelectric transport with coupling with heat baths,” C. R. Phys. 17, 1047 – 1059 (2016).
  * Yang  _et al._ (2018) Y. Yang, H. Chen, H. Wang, N. Li, and L. Zhang, “Optimal thermal rectification of heterojunctions under fourier law,” Phys. Rev. E 98, 042131 (2018).
  * Zhang and Su (2021) Y. Zhang and S. Su, “Thermal rectification and negative differential thermal conductance based on a parallel-coupled double quantum-dot,” Physica A: Statistical Mechanics and its Applications 584, 126347 (2021).
  * Wang  _et al._ (2019) C. Wang, D. Xu, H. Liu, and X. Gao, “Thermal rectification and heat amplification in a nonequilibrium v-type three-level system,” Phys. Rev. E 99, 042102 (2019).
  * Diaz and Sanchez (2021) I. Diaz and R. Sanchez, “The qutrit as a heat diode and circulator,” New J. Phys. 23, 125006 (2021).
  * Tesser  _et al._ (2022) L. Tesser, B. Bhandari, P. Andrea Erdman, E. Paladino, R. Fazio, and F. Taddei, “Heat rectification through single and coupled quantum dots,” New J. Phys. 24, 035001 (2022).
  * Khandelwal  _et al._ (2023) S. Khandelwal, M. Perarnau-Llobet, S. Seah, N. Brunner, and G. Haack, “Characterizing the performance of heat rectifiers,” Phys. Rev. Res. 5, 013129 (2023).
  * Zhang  _et al._ (2010) L. Zhang, J.-S. Wang, and B. Li, “Ballistic thermal rectification in nanoscale three-terminal junctions,” Phys. Rev. B 81, 100301 (2010).
  * Ren and Zhu (2013) J. Ren and J.-X. Zhu, “Theory of asymmetric and negative differential magnon tunneling under temperature bias: Towards a spin seebeck diode and transistor,” Phys. Rev. B 88, 094427 (2013).
  * Yang  _et al._ (2023a) Y.-j. Yang, Y.-q. Liu, and C.-s. Yu, “Quantum thermal diode dominated by pure classical correlation via three triangular-coupled qubits,” Phys. Rev. E 107, 064125 (2023a).
  * Jiang  _et al._ (2015) J.-H. Jiang, M. Kulkarni, D. Segal, and Y. Imry, “Phonon thermoelectric transistors and rectifiers,” Phys. Rev. B 92, 045309 (2015).
  * Joulain  _et al._ (2016) K. Joulain, J. Drevillon, Y. Ezzahri, and J. Ordonez-Miranda, “Quantum thermal transistor,” Phys. Rev. Lett. 116, 200601 (2016).
  * Guo  _et al._ (2018) B.-q. Guo, T. Liu, and C.-s. Yu, “Quantum thermal transistor based on qubit-qutrit coupling,” Phys. Rev. E 98, 022118 (2018).
  * Yang  _et al._ (2023b) Y. Yang, Y. Zhao, and L. Zhang, “A high-performance thermal transistor based on interfacial negative differential thermal resistance,” Applied Physics Letters 122, 232201 (2023b).
  * Li  _et al._ (2004) B. Li, L. Wang, and G. Casati, “Thermal diode: Rectification of heat flux,” Phys. Rev. Lett. 93, 184301 (2004).
  * Li  _et al._ (2006) B. Li, L. Wang, and G. Casati, “Negative differential thermal resistance and thermal transistor,” Applied Physics Letters 88, 143501 (2006).
  * Lu  _et al._ (2019) J. Lu, R. Wang, J. Ren, M. Kulkarni, and J.-H. Jiang, “Quantum-dot circuit-qed thermoelectric diodes and transistors,” Phys. Rev. B 99, 035129 (2019).
  * Lu  _et al._ (2020) J. Lu, R. Wang, C. Wang, and J.-H. Jiang, “Brownian thermal transistors and refrigerators in mesoscopic systems,” Phys. Rev. B 102, 125405 (2020).
  * Sánchez  _et al._ (2017) R. Sánchez, H. Thierschmann, and L. W. Molenkamp, “All-thermal transistor based on stochastic switching,” Phys. Rev. B 95, 241401 (2017).
  * Wang  _et al._ (2018) C. Wang, X.-M. Chen, K.-W. Sun, and J. Ren, “Heat amplification and negative differential thermal conductance in a strongly coupled nonequilibrium spin-boson system,” Phys. Rev. A 97, 052112 (2018).


Manuscript submitted to ACM Manuscript submitted to ACM

# MagSurface: Wireless 2D Finger Tracking Leveraging Magnetic Fields

Sarnab Bhattacharya<EMAIL_ADDRESS>University of Texas at Austin2501
SpeedwayAustinTexas78705 ,  Keum San Chun<EMAIL_ADDRESS>University of
Texas at Austin2501 SpeedwayAustinTexas78705 and  Edison Thomaz
<EMAIL_ADDRESS>University of Texas at Austin2501 SpeedwayAustinTexas78705

###### Abstract.

With the ubiquity of touchscreens, touch input modality has become a popular
way of interaction. However, current touchscreen technology is limiting in its
design as it restricts touch interactions to specially instrumented touch
surfaces. Surface contaminants like water can also hinder proper interactions.
In this paper, we propose the use of magnetic field sensing to enable finger
tracking on a surface with minimal instrumentation. Our system, MagSurface,
turns everyday surfaces into a touch medium, thus allowing more flexibility in
the types of touch surfaces. The evaluation of our system consists of
quantifying the accuracy of the system in locating an object on 2D flat
surfaces. We test our system on three different surface materials to validate
its usage scenarios. A qualitative user experience study was also conducted to
get feedback on the ease of use and comfort of the system. Localization error
as low as a few millimeters was achieved.

HCI, finger-tracking

††copyright: none††ccs: Human-centered computing Touch screens††ccs: Human-
centered computing Pointing devices

(a)

(b)

Figure 1. (a) MagSurface System consisting of electromagnets mounted on a
custom designed 3D printed structure and wearable band-aid, and, (b) The
wireless finger tracking device was built on a flexible PCB and consists of
three functional blocks: wireless micro-controller unit (MCU), sensing unit,
and antenna.

##  1\. Introduction

Over the last decade, touch input has become the most common human-computer
interface for interacting with mobile devices and large displays. The
popularity of touchscreens, and the intuitive manner in which they support
direct manipulation (hutchins1985direct, ) has brought touch technologies to a
variety of physical interfaces that require user input such as computer
screens, music players, ebook readers and automotive dashboards. However,
while largely commoditized for smartphones and personal devices, current
touchscreen technology is difficult to adapt for every applications that
require human machine interaction. This is because today’s touchscreens are
most commonly built using resistive or capacitive sensing technology applied
to a glass surface. As a result, precision touchscreens tend to be not only
relatively small and expensive, but also suffer from a few important
shortcomings, such as not performing well in the presence of water, e.g., in
the rain, and being available only as a strictly smooth surface. In light of
these drawbacks, the research community has explored a variety of techniques
to enhance touch based human machine interaction paradigms. These have
included supplementing objects with touch input modality, and allowing for
more flexibility in the types of surfaces where touch technologies can be
applied (ono2013touch, ; poupyrev2016project, ; Sato:2012:TET:2207676.2207743,
; Xiao:2013:WRE:2470654.2466113, ).

In this paper, we present a technique for wireless finger tracking based on
magnetic field sensing. Our approach extends touch input to any flat surface,
addressing some of the limitations of current touchscreen technologies. We
demonstrate our technique through a system implementation called MagSurface.
MagSurface is comprised of two components, (1) a pair of electromagnets and
(2) a custom wearable device, both shown in figure 1(a). The electromagnets
generate a time varying magnetic field which is captured by a magnetometer in
the wearable device, which is small and can be easily attached to the user’s
fingertips or place on a ring. Limitations of capacitive touchscreens such as
sub-optimal operation in the presence of water or dirt do not hinder our
approach. Likewise, our method is not affected by irregularities of the
tracking surface, i.e., the tracking surface does not have to be smooth. These
attributes make our approach particularly suitable for applications situated
in outdoor environments (e.g., industrial applications), in settings where
there might be a need to operate a computing device with dirty hands or
through a dirty surface, (e.g., kitchen, machine shop), and in environments
whose input surfaces must be sterilized with chemicals that could impact or
damage the electronics in traditional touchscreens.

The contributions of this work can be summarized as follows:

  * •

A system design and implementation comprised of an electromagnet bar for
generating magnetic fields and a lightweight, low power, and wireless wearable
to perform tracking of a user’s fingers on 2D surfaces.

  * •

An algorithm for finger tracking by iteratively reducing position error which
can be run on an embedded micro-controller.

  * •

An evaluation of the system on 3 surface materials, metallic, wood and
acrylic, in which we showed our approach produces localization errors not
greater than 0.25cm.

  * •

A qualitative user experience survey with 12 participants, whose results
demonstrated the viability of the approach and highlighted areas for
improvement.

##  2\. Related Work

Before touchscreens in mobile devices, finger tracking was made popular
through trackpads on laptop computers, which have been long studied
(epps1986comparison, ), and custom camera-based multi-touch systems such as
Microsoft Surface (de2010function, ). Over the years, various technologies
have been developed to support touch including Interpolated Force Sensitive
Resistance (IFSR), which support multitouch interactions and can acquire high-
quality anti-aliased pressure images at high frame rates
(Rosenberg:2009:UIM:1576246.1531371, ).

Current touchscreens typically use transparent resistive or capacitive panels
overlayed on glass screens. However, as touch interactions have become
increasingly more popular, a large body of research has examined new technical
approaches for enabling touch-enabled interfaces in everyday surfaces and
objects. These methods have explored a wide range of sensing modalities and
form factors. For instance, Takeoka et al. installed multi-layered infrared
lasers to detect not only hand and finger contact but also their orientation
(Takeoka:2010:ZIG:1936652.1936668, ). SmartSkin instrumented an entire
tabletop with capacitive leads to act as a touch interface, with support for
hover detection (Rekimoto:2002:SIF:503376.503397, ). The sections below
describe several techniques that have been proposed by researchers towards
improving and augmenting touch interfaces.

###  2.1. Optical Sensing

A large body of work focusing on vision-based and optical-based finger
tracking exist. Harrison et al. and others experimented with a portable
projector and Kinect system to enable touch and gesture interactions anywhere
(Harrison:2011:OWM:2047196.2047255, ). Similar methods include
(Wilson:2005:PCI:1095034.1095047, )(Xiao:2013:WRE:2470654.2466113,
)(Wilson:2004:TIT:1027933.1027946, ). SymmetriSense is a system utilizing
commodity cameras to detect touches on glossy surfaces
(Yoo:2016:SEN:2858036.2858286, ). Frustrated Total Internal Reflection (FTIR)
methods, which can be thought of as a combination of optical and vision based
system like (Hofer:2009:FFM:1517664.1517730,
)(Echtler:2009:IFE:1731903.1731909, ), have also yielded good results.
Commercial products like EvoMouse (evomouse, ) and LeapMotion (leap, ) have
used Infra Red (IR) reflection based sensing to track user fingers.

###  2.2. Acoustic Sensing

Acoustic approaches to detect touches, taps, knocks or flicks on surfaces have
been extensively explored. Kim et al. used commercial smartphone microphones
to enable tap interactions on a variety of surfaces
(Kim:2018:ULA:3274783.3274848, ). Other sound-based techniques to localize
touch events and gestures have also been proposed by
(Sun:2018:VST:3241539.3241568, )(7964907, )(5655110, ). Acoustic approaches
can be efficient in detecting and localizing taps. However, tracing a finger
as it is dragged on a surface can be more challenging with a sound-only
approach. SoundTrak, on the other hand, performs continuous finger tracking in
3D space by capturing and processing sound generated from a small speaker
attached to the user’s finger (Zhang:2017:SCT:3120957.3090095, ).

###  2.3. Magnetic Field Sensing

Magnetic Field (MF) sensing has long been considered a viable, accurate and
occlusion-free method of tracking objects. Raab et al. details a series of
transformations to convert non-linear magnetic field equations to directly
solvable linear equations (4102227, ). This sets the foundation for using MF
sensing to track objects in 3D space. Some simple approaches include using
permanent magnets attached to fingers (Harrison:2009:AWH:1622176.1622199, ),
or in rings (Ashbrook:2011:NSE:1978942.1979238, ) to provide 1D input to
smartwatches. Others proposed covering an entire interaction space with
magnetic sensors (Liang:2012:GAS:2380116.2380157, ). Han et al. demonstrated
that it is possible to track a magnet in 2D using a pair of magnetometers
(4421009, ). However, their assumptions made about the relative orientations
of the magnet might not always hold true in all scenarios. uTrack built on
this concept to propose a system for tracking the position of a permanent
magnet in 3D space (Chen:2013:UIU:2501988.2502035, ). However, they relied on
an exhaustive search of the orientation space which may be too computation
intensive for real-time low power embedded applications. Chen et al. proposed
a system to track fingers with high precision in 3D
(Chen:2016:FTP:2858036.2858125, ). Active electromagnets instead of permanent
magnets were used to filter out stray or ambient magnetic fields. Four
magnetometers were used to resolve the positioning ambiguity that needs to be
overcome when using MF sensing. The magnetometers were placed on a fixed base
while the active electromagnets were placed on fingers. Dai et al. used both
active electromagnets and geomagnetic fields to perform 6D tracking using one
electromagnet and one magnetometer (dai20176, ). Their algorithm relied on
geomagnetic fields for orientation and used that information to calculate the
3D position. This algorithm is quite complex and would be hard to perform on a
small low cost embedded processor in real time. Ge et al. proposed a unique
aparatus with rotating electromagnets (ge2009novel, ). In theory it is a novel
implementation but mechanical components like rotating electromagnets are
difficult to implement and are prone to failures.

###  2.4. Instrumented Objects

Finally, there has been strong interest in sensing touch in irregular shapes
and beyond surfaces. Sato et al. adopted an approach of embedding capacitive
sensors on everyday objects to sense touch and gesture
(Sato:2012:TET:2207676.2207743, ). Skintrack instrumented the human arm by
attaching electrodes on the wrist and wearing a signal emitting ring on the
finger to continuously track finger movement on the forearm and palm
(Zhang:2016:SUB:2858036.2858082, ). Zhang et al. proposes a unique way of
either fabricating objects, or coating objects with a conductive material
(Zhang:2017:ELT:3025453.3025842, ). Electrodes are then attached to the object
and using electric field tomography, touch and finger tracking can be
performed. Instrumenting surfaces or objects provide very accurate detection
and tracking of touch and other input modalities like pressure orientation,
etc. But these approaches require extra hardware and computational power. They
also tend to be specific to the particular setting, and thus are not easily
portable.

##  3\. Motivation

Most acoustic approaches suffer from being unable to continuously track
objects. They are also susceptible to noise. Thus it would not be ideal to use
them to simulate or replace a touchpad or touchscreen. Multi-finger
interactions would also be hard to integrate using acoustic sensing. Occlusion
was a major issue with vision/optical based systems. Vision based systems also
have to perform intensive computations which would not be ideal for a low
power embedded device. MF sensing is not hindered by occlusion and addresses
this issue. But in most of the previous work, some previous assumption about
the relative orientation was made or a compute intensive search was conducted
to find the orientations. Chen et al.(Chen:2016:FTP:2858036.2858125, ) came up
with a orientation invariant method of tracking objects in 3D using MF sensing
but their system utilized a lot of hardware to surmount the problem of not
knowing relative orientations apriori.

Our design was driven by the desire to avoid instrumenting the interaction
space with specialized and hard to set up sensors which would reduce
portability and increase costs. Instead, we aimed for a portable approach; to
use the system, a user would only need to place the electromagnets on a
surface of choice and attach a minimal wearable piece to a finger. For user
comfort, the wearable device on the finger was made very small (10x8 $mm^{2}$)
as shown in figure 1(b). Bluetooth Low Energy(BLE) was chosen as the wireless
link for communicating data from the system as most consumer devices come with
BLE chips built-in and thus would offer easier integration. 2D tracking offers
some benefits with regards to reduction in the amount of hardware needed to
accurately track objects using MF sensing. The magnetometer is attached to the
user’s finger and the electromagnets are used as a static anchor. This offers
3 advantages:

  * •

The wearable part is small compared to a bulky electromagnet and would offer
more ease of use.

  * •

It is low power and thus can be powered by a small coin cell battery. The
electromagnets on the surface can be powered by larger batteries and can even
be built into devices or furniture.

  * •

To expand the system for multi-finger use, more magnetometers would need to be
used. This is easier than adding another electromagnet with its characteristic
AC frequency and accordingly adding another filter on the processing side.

Exploiting the geometry of the setup and the constraint that the interaction
space is 2D, the system uses minimal hardware and algorithms that are not
computationally intensive to track a user’s finger.

##  4\. Algorithm

MagSurface uses inexpensive off the shelf hardware. We have designed some
simple, intuitive algorithms to utilize this hardware setup to accurately
track objects in 2D. Our method uses a modified trilateration technique to
calculate the distance between the magnetometer and the electromagnets based
on magnet field strength (figure 2(a)). MagSurface can be placed on any flat
surface and our current implementation is capable of 2D finger tracking within
a 10x10 $cm^{2}$ rectangular area.

(a)

(b)

Figure 2. (a) Trilateration with 2 anchors (C1 and C2) gives 2 probable
locations (A and B) of where the object might be. The distance D between the 2
anchors and the 2 radii R1 and R2 are known quantities. A is where our object
actually is, and, (b) Magnetic field lines generated by a small magnetic
source projected on a 2D plane. The magnetic field vector at the sensor can be
decomposed into the radial and angular components

###  4.1. Trilateration

Central to our tracking algorithm is trilateration, a technique that consists
of being able to determine the exact location of an object if the distance of
the object from anchors position points are known. Trilateration, as the name
suggests, implies the need to know the distance between an object and 3 anchor
points to calculate the location of an object. This is true for locating an
object on a 2D surface.

With 3 anchor points, it is possible to precisely calculate the location of an
object in 2D space. But in our case we only have two anchors – the two
electromagnets. Using two anchors, we can localize the object to two locations
(A and B) as illustrated in Figure 2(a). The center of the two circles (C1 and
C2) are the two electromagnets. In our specific case we have assumed that
interactions by a particular user will only be performed on one side of the
electromagnets. Thus we can safely discard one of the two possible locations
where our target object could be and lock in on the exact location of the
tracked object.

###  4.2. Magnetic Positioning

Our approach leverages magnetic waves generated by alternating voltage to
obtain the distance of our object from the electromagnets, i.e. anchor points.
Since the system tracks interactions on a 2D surface, we only consider
magnetic field lines in the plane on which the centers of the electromagnets
and the object lies, i.e. the surface on which we are interacting. From Figure
2(b) it can be observed that the strength of the magnetic field H needs 2
factors to be defined, the radial distance r from the magnets center to the
sensor, and the angle $\mathit{\theta}$ between the magnets north pole and the
sensor. Both of these values are unknown. H is generally decomposed into its
radial and angular terms $\mathbf{H_{r}}$ and $\mathbf{H_{\theta}}$
respectively. The values of these quantities are given by:

(1) $||\mathbf{H_{r}}||=M\ cos\theta/2\pi r^{3}$ (2)
$||\mathbf{H_{\theta}}||=M\ sin\theta/4\pi r^{3}$

Here, M is the magnetic moment of the electromagnet and relates to the
permeability of the core, current through the coil, and the cross-sectional
area of the coil.

Input: $\mathbf{H_{20}},\ \mathbf{H_{30}},\ K,\ D$

Output: X, Y

$\theta_{20}\leftarrow 0,\ \theta_{30}\leftarrow 0$;

for  _$k\leftarrow 1$ to $N$_ do

$r_{20}\ calculated\ from\ \mathbf{H_{20}}\ and\ \theta_{20}\ using\ equ\
\ref{eq:mag_strength}$;

$r_{30}\ calculated\ from\ \mathbf{H_{30}}\ and\ \theta_{30}\ using\ equ\
\ref{eq:mag_strength}$;

$(X,Y)\ calculated\ from\ r_{20},\ r_{30}\ and\ D$;

$\theta_{20}\leftarrow arctan(X/Y)$;

$\theta_{30}\leftarrow arctan((D-X)/Y)$;

end for

Algorithm 1 Converging to correct $\mathit{\theta}$ and r

As (Chen:2016:FTP:2858036.2858125, ) has shown, unless the relative
orientation of the electromagnet and our sensor is known, it is very difficult
to compute $\mathbf{H_{r}}$ and $\mathbf{H_{\theta}}$ from the data obtained
from the magnetometer. The magnetometer we used is a 3-axis magnetometer,
which outputs the magnetic field strength detected on each of its axes as
$\mathbf{H_{x}}$, $\mathbf{H_{y}}$ and $\mathbf{H_{z}}$ for x-axis, y-axis and
z-axis respectively. To obtain $\mathbf{H_{r}}$ and $\mathbf{H_{\theta}}$ from
these values would require either knowing the orientation beforehand and
applying a transform or exhaustively searching the rotation space for the
global optimal rotation that transforms ($\mathbf{H_{x}}$, $\mathbf{H_{y}}$,
$\mathbf{H_{z}}$) to ($\mathbf{H_{r}}$, $\mathbf{H_{\theta}}$, 0)
(Chen:2013:UIU:2501988.2502035, ). Both of these methods are not feasible as
neither the relative orientation is provided apriori nor do we want to use the
computational capability to do an exhaustive search of the rotation space in
real-time. So instead, we calculate the total magnetic field strength given
by:

(3) $||\textbf{H}||^{2}=H_{x}^{2}+H_{y}^{2}+H_{z}^{2}=K\times
r^{-6}\times(3*cos^{2}\theta+1)$

The K in equation 3 is a constant. We still have an under-constrained equation
as we are trying to figure out both r and $\mathit{\theta}$ from only one
equation. This is similar to the approach adopted by
(Chen:2016:FTP:2858036.2858125, ), but unlike their method of using multiple
magnetometers to resolve the ambiguity, we use an iterative algorithm to
accurately estimate both r and $\mathit{\theta}$ from one equation itself by
leveraging the geometric constraints of the system. It should also be noted
that these equations only hold when we treat the magnet as a small dipole. In
other words, the distance from the magnet should be greater than the dimension
of the magnets.

Our algorithm solves the problem of estimating both r and $\mathit{\theta}$
from one equation by taking advantage of the geometric constraints and
symmetries in our system. While calculating the distance from the wearable to
one electromagnet is key, it is critical to consider the entire system as a
whole. The 2 electromagnets are driven by an alternating current(AC) source.
One magnet is driven by a 20Hz sine wave while the other is driven by a 30Hz
sine wave. As they are on different frequencies the magnetic fields do not
interfere with each other. Stray magnetic fields and the earth’s magnetic
field, a DC field, also do not affect the generated magnetic fields unless
they fall in the operating frequency band of the electromagnets. The
magnetometer on the wearable device samples at 100Hz, which has a Nyquist cut
off frequency of 50Hz, well above the frequency of our electromagnets. A Fast
Fourier Transform(FFT) is performed on the magnetometer data and the frequency
components at 20Hz and 30Hz are obtained. This data is fed into our algorithm
as $\mathbf{H_{20}}$ and $\mathbf{H_{30}}$ respectively, as shown in algorithm
1.

The K term in the algorithm is the constant determined by the physical
properties of the electromagnet. As both our electromagnets are the same, we
can use the same K for both of them. The D term is also a constant as it is
the distance between the centers of the 2 electromagnets as shown in Figure
3(a). The distance between our sensor and the 20Hz electromagnet and the 30Hz
electromagnet is represented by $r_{20}$ and $r_{30}$ respectively and the
angle is represented by $\theta_{20}$ and $\theta_{30}$. Initially, both
$\theta_{20}$ and $\theta_{30}$ are set to 0. Values of $r_{20}$ and $r_{30}$
are calculated using these values of $\theta$ by using equation 3. After
obtaining the distance from the 2 electromagnets, relative position of the
sensor represented by X and Y can be calculated by standard geometric
equations. Once we obtain an initial value of X and Y, we can update our
values of $\theta_{20}$ and $\theta_{30}$.

We conducted tests to see how fast the two $\theta s$ converge and observed
that they converged in under 5 iterations in almost all cases. Thus N can be
set to 5 without any loss in accuracy. Furthermore, after the initial
convergence to the correct $\theta$ values, the next $\theta$ values can use
the previous calculated $\theta$ values as an initial guess for faster
convergence. We consider our update rate of 100Hz to be sufficient as it is
very unlikely that a user’s finger will move a large distance within tens of
miliseconds. Consequently, the previous $\theta$ values will be very close to
the present $\theta$ values.

(a)

(b)

Figure 3. (a) Geometry of our setup. $EM_{20}$ and $EM_{30}$ are the centers
of the electromagnets generating 20Hz and 30Hz sine waves respectively. (X,Y)
is the position of our sensor considering $EM_{20}$ as origin, and, (b) 20Hz
sine wave produced by the tone generating circuit. The Blue waveform is the
raw output of the DAC. The Yellow waveform is after low pass filtering.

###  4.3. Calibration

Before the user can begin using the system, a calibration procedure is needed.
This step takes no more than a few seconds and requires the user to place his
finger on a marked predefined position on the electromagnet bar while wearing
the sensor. The calibration position on the bar was set to the midway point
between the two electromagnets. During the calibration step, the system
calculates the K term in equation 3 for the electromagnets. We have included
this step to make our system compatible with any kind of magnet being driven
by any amount of current. As K captures most of the magnets’ parameters, as
will be explained in Section 7, calibrating our system makes it magnet
invariant. Additionally, constant field disturbances present in the
environment are accounted for to a certain degree in the calibration step.

###  4.4. Localization Algorithm

Our algorithm consists of the following steps:

  1. (1)

Read magnetic sensor value and add it to input buffer

  2. (2)

For all 3 axes, apply band-pass filtering and do an FFT on the filtered value.

  3. (3)

From the FFT calculate $\mathbf{H_{20}}$ and $\mathbf{H_{30}}$

  4. (4)

Apply algorithm 1 to obtain the coordinates of the sensor

The input buffer can store up to 50 sensor readings. Thus each new sample is
combined with 49 previous samples to compute the FFT. We have incorporated a
combination of multithreaded and single threaded segments in our algorithm.
The data collection and finding positions are done in serial while the signal
conditioning and FFT is done in parallel for each axis. All our code is
implemented in python using Numpy and Scipy libraries. We have also used the
win32api library to control the mouse pointer from our python program.

Implementing the algorithm on a PC or laptop is straightforward. Taking it a
step further we implemented the tracking algorithm on the embedded micro-
controller itself. It had a hardware Floating Point Unit(FPU) to speed up
floating point calculations. The signal acquisition, filtering, buffering, FFT
calculation and convergence algorithm were all implemented on the embedded
processor. Due to lack of memory and compute power a smaller buffer of 20
samples was used, this did not lead to significant tracking accuracy loss. The
system was timed to run the entire algorithm in under 5ms. This indicated even
at a sampling rate of 200Hz the system will be able to process and track the
user’s finger. Processing on the edge like so reduces the burden of the
central computer having to dedicate compute cycles for running the algorithm.
This computation might be significant if multi-finger tracking is implemented.
At a 100Hz update rate performing the computation on chip consumed around 2mA
current on average. The battery we used was rated at 20mAh which means the
system can be used for around 10 hours continuously before needing to be
recharged.

Material | Mean Error(cm)
---|---
Metallic Tabletop | 0.21 $\pm$ 0.12
Wood | 0.20 $\pm$ 0.13
Acrylic | 0.22 $\pm$ 0.11
Table 1. Table showing average localization errors on different surfaces over
the 25 points

##  5\. Hardware

The MagSurface hardware consists of two components:

  * •

An electromagnet bar with two electromagnets, driven by an AC signal source

  * •

A wireless wearable device containing the magnetometer sensor for location
tracking and a micro-controller unit (MCU) with a BLE radio for data
transmission

###  5.1. Electromagnet Bar

The electromagnet bar contains 2 electromagnets with their magnetic axis
exactly parallel to each other. This reduces the computations involved when
calculating the position of the finger. The 2 electromagnet centers have to be
kept level and separated by a fixed distance. This is achieved by mounting
them on a 3D printed enclosure.

The electromagnets generate sinusoidal magnetic waves. AC fields as opposed to
DC magnetic fields are used as it is much easier to reduce interference due to
stray field with a tuned filter. Another advantage is that the earth’s
substantial magnetic field will not interfere with our measurements as that is
a DC value which can be filtered out. A class D audio amplifier was used to
drive the electromagnets; it provides stereo channel audio output, making it
ideal for driving the 2 electromagnets. Furthermore, a gain control for tuning
the current was also incorporated. To generate the tone, we employed an
Atmega328P and a dual channel Digital to Analog Converter (DAC) to generate
sinusoidal wave-forms at frequencies of 20Hz and 30Hz as shown in figure 3(b).
The DAC produces discrete level sine waves, which are smoothed with a low pass
filter.

###  5.2. Finger Tracking Wearable

To sense the magnetic fields, we built a small and lightweight wireless
wearable device (See Figure 1(b)). The device was built on a flexible PCB and
consists of three functional blocks: wireless MCU, sensing unit, and antenna.
For the wireless MCU, CC2640 from Texas Instruments was used due to its low
power consumption and small package size. As for the sensing subsystem, a
MPU-9250 module(mpu9250, ) from InvenSense was used. The MPU-9250 contains a
6-axis intertial motion unit (IMU) and a 3-axis magnetometer. For this
specific application, only 3-axis magnetometer sensor was activated. Lastly,
in the antenna block, we used a 2.45 GHz chip antenna and an impedance-matched
balun filter from Johanson Technology. Touch on surfaces is detected by a
pressure sensor connected to the MCU’s digital input pin.

(a)

(b)

Figure 4. (a) The test bench used for testing the accuracy of the system on
various surfaces. 25 points on a grid of 10x10 $cm^{2}$ dimension was used to
evaluate the system at different sensor positions. Measurements shown on the
image are in millimeter, and, (b) Screenshot from the game the participants
were asked to play. The Red dot was controlled by finger movements to dodge
the black blocks.

##  6\. Evaluation and Results

Localization accuracy of the system was evaluated and a user study to
understand the acceptance of the system among users was conducted.

(a)

(b)

(c)

Figure 5. Scatter plots showing the actual position (red) and the system
determined position (blue) on different surfaces namely (a)Metal, (b)Wood, and
(c)Acrylic. The axes measurements are in cm. The scatter plots represent the
evaluation grid shown in Figure 4(a).

###  6.1. Localization Accuracy

The accuracy of tracking was measured on a 5x5 grid of points (Figure 4(a)).
The relative locations of the grid points with respect to the electromagnetic
bar were predefined. When a user places a finger on these predefined points
while wearing the device, the tracking algorithm predicts the location of the
finger. The accuracy was computed by obtaining the mean absolute error (MAE)
between the predicted location and the true location across the 25 center
points of the 5x5 grid (Table 1). This approach is similar to the one used by
(4421009, ). We also examined the accuracy of the system on three different
surfaces namely acrylic, wood and metal as these three surfaces are widely
used in our daily lives. The scatter plots in figure 5 depict the location of
the 25 grid points (ground truth, in red) and the inferred location of the
finger for each one of the points (in blue) for each one of the 3 surfaces
tested.

###  6.2. Qualitative User Experience Study

A user study with a convenience sample of 12 participants (9 male, 3 female)
aged 19 to 26 years-old was conducted to gain insights into the overall user
experience of the system. In the study, participants wore the tracking device
on their index finger and used it like a pointing device to play a game. The
game consisted of moving a red dot and dodging moving blocks on the computer
screen. Figure 4(b) shows a screenshot of the game. Over time, the number of
blocks and the speed at which they moved increased. In other words the
difficulty increased as time passed. Participants were given 3 tries. After
that the participants were asked to fill a survey which asked about the user
experience of using the system.

Overall, participants reacted positively to the experience (”This seems to not
be tiring or hard to use for extended amount of time”, ”its nice to use my own
finger to make movement, mouse can sometimes cause pain”). A couple of
participants commented on latency (”sometimes there is delay between of the
movement between finger and controller”, ”I would love to suggest fixing the
lagging so its more responsive and accurate”). Finally, some participants also
noticed a small amount of cursor jitter (”it would be good if the movement
also stops when i am idle”, ”at moving the jitter is negligible but at slower
speed or standstill, the jitter becomes increasingly noticeable”). It was
observed that this jitter was maximum at distances far away from the magnets
as well as distances closer to the magnets. This corresponds to the maximum
error regions that we observed during our evaluation of the accuracy of the
system.

##  7\. Discussion

###  7.1. Results

An interesting observation from the error plots is that the localization error
is more pronounced at points further away and also very close to the
electromagnets. Error at distant points is to be expected as the magnetic
fields decay very rapidly with distance. For distances very close to the
magnet, the error can be explained by the characteristics of our model. As
stated earlier, the magnetic equations consider the electromagnets as small
magnetic dipoles. But at distances very close to the electromagnets, the
dimension of the electromagnets themselves are comparable to the distance from
the magnet to the sensor, which results in inaccuracies. Another observation
is that the error on different surfaces remained more or less same which
indicated that the calibration step might be compensating for differences in
environmental conditions.

###  7.2. User Feedback

The most prominent feedback was about the shakiness of the cursor when the
participant kept their finger idle. Due to small variations in the output of
the localization algorithm, in each iteration the new position changed
slightly. This in turn caused the cursor on the screen to shake. A fix to this
would be to introduce a dead-zone in the sensor movement, i.e., if the
movement is small, ignore it completely. Another common feedback was to ”fix
the lagging so the system is more responsive”. We use a sampling rate of 100Hz
for the magnetometer. But the samples go to a FiFo which is 50 samples long
for calculating the FFT. So there is a small latency in the movement of the
finger being translated to a cursor movement. One way to address this issue
would be to increase the sampling rate. There are magnetometers that can
sample at 800Hz which would make the system much more responsive to position
changes. Reducing the buffer size to 20 for our on device processing did
improve the response rate by a good margin.

###  7.3. Custom Electromagnets

For our electromagnets, we used 2 off-the-shelf electromagnets whose exact
specifications we could not confirm. This was possible because, as previously
described, our calibration step makes it possible for our system to be
electromagnet invariant. We compute the value for the magnetic field strength
K used in equation 4. K is proportional to 4 factors:

(4)
$\textbf{K}\propto\mathbf{\mu}\times\textbf{N}\times\textbf{I}\times\textbf{A}$

$\mathbf{\mu}$ is the magnetic permeability of the core material, N is the
number of turns of the coil, I is the current flowing through the coil and A
is the cross sectional area of the coil.

In designing custom electromagnets, we would choose to use a core material
with larger relative magnetic permeability. This would allow us to reduce the
cross sectional area and number of turns, thereby reducing the size of the
electromagnet without reducing the magnetic field strength of the generated
fields. It would also enable us to reduce the current flowing through the coil
thus making the system more power efficient.

###  7.4. Enlarging Tracking Area

The operation of MagSurface is by no means restricted to the 10x10 $cm^{2}$
rectangular area used to evaluate it. There are 2 ways to increase the size of
the finger tracking space: (1) make the generated magnetic field stronger, or
(2) make the sensors more sensitive.

The magnetic field can be strengthened by raising K, which can be achieved by
increasing any of the 4 factors present in equation 4. We would not want to
increase A or N as that would make the system bulky and reduce portability.
Alternatively, we could use better core materials or drive more current. These
approaches come with their own limitations. Using a better core material will
make the system expensive, and increasing current would make the system energy
inefficient and also cause heating issues. We have to consider all these
trade-offs when designing the system to balance both the pros and the cons.

Using better sensors would be another option. The sensor that we are currently
using has a resolution of 0.6$\mu$T. There are sensors which have a resolution
of 0.01$\mu$T. But that high level of sensitivity also comes with trade-offs,
either they have a low sampling rate, or they are very expensive, or they have
a large physical size. For our purpose, the system requires high sampling rate
for low latency tracking and small physical size for portability and ease of
use.

There is still a limit to how big of an interaction space can be used. As
magnetic fields are inversely proportional to the cube of the distance,
increasing the magnetic strength by 8 would only lead to a sensing distance
increase of 2. So there is a very sharp diminishing return.

###  7.5. Richer Interactions

In this work, we demonstrate the viability of MagSurface with just one
magnetometer, placed on the finger. Since the magnetometer does not affect the
magnetic field being generated, an interactive experience leveraging multiple
fingers, and thus multiple magnetometers, could be designed with our proposed
approach. In future work, we would like to investigate the space of gestural
interactions using more than one finger, e.g., zoom, scroll and multi-finger
swipe. Additionally, multi-user interaction in the same space is another area
that can be explored.

The number of sensors that can be added would be limited by the BLE data rate.
Each sensor would have a rate at which it would send its position. The maximum
number of sensors n theoretically supported would be

(5) $\textbf{n}=\frac{max\ BLE\ throughput}{Device\ update\ rate\times Data\
per\ update}$

##  8\. Usage Scenarios

We envision the use of MagSurface both as a way to extend the input space of
existing devices and as replacement to existing touchscreens and touchpads in
specific scenarios where these technologies might present shortcomings.

#### Extended Interactive Surface

By incorporating MagSurface into computing devices such as laptops and
tablets, the touch-interactive surface of these devices could be expanded
beyond their built-in touchpad or touchscreen. Furthermore as the wearable
sensing device can sit on top of a finger (i.e., on the fingernail), or placed
on a ring, it would not hinder other forms of input like typing on the
keyboard or using a mouse. This usage scenario is similar to the one laid out
by Mistry and Maes (Mistry:2011:MCM:1979742.1979715, ).

#### Smart Furniture

As smart rooms become more common, MagSurface provides an opportunity to
embedded touch input into furniture without modifying its design and
aesthetic. As an example, MagSurface’s electromagnets could be placed within a
sofa arm, turning the arm into a tracking surface. By wearing the wireless
finger tracking device on a ring, users could control a smart TV, sound system
or IoT devices without the need for an additional device (e.g., remote control
with a trackpad). This scenario is possible because MagSurface permits finger
tracking even on surfaces that are corrugated or covered by fabric, i.e., not
perfectly flat.

#### Wet Lab and Surgery Room

In a wet lab or surgery room, where scientists and doctors might handle
hazardous materials or be in contact with contagious substances (e.g., blood),
maintaining a sterile environment is paramount. For this reason, hands must be
regularly de-germed and the utilization of gloves is a strict requirement.
Gloves should be taken off during computer use however, which is highly
disruptive. Computer hardware, especially keyboards and touch surfaces, can be
contaminated with microorganisms when touched by contaminated hands. By
wearing MagSurface’s wireless finger tracking device, it would be possible to
let doctors and scientists interact with a computer via touch while wearing
gloves on a surface which can be heavily disinfected with chemicals that might
not be suitable for existing touchscreen electronics.

#### COVID-19

The COVID-19 virus is very infectious and is known to spread indirectly via
contact with surfaces. In many places such as airport check self check in
counters, grocery self checkout, etc. people have to use a touchscreen. In
this situation strong disinfectants need to be used on public touchscreens to
prevent the spread of the virus. Using such strong chemicals can cause damage
to traditional touchscreens, replacement of which can be a costly affair. Our
approach lets any surface become a touchscreen, thus enabling regular
tabletops of even replaceable and disposable plastic or paper sheets to become
touch surfaces. Additionally the wearable part can be embedded inside a sealed
plastic mould which can be clipped onto the finger. Thus the wearable device
can also be disinfected after use.

##  9\. Conclusion

This paper presents an approach that leverages magnetic fields to enable
finger tracking on any flat surface. We demonstrated an implementation of our
approach with MagSurface, a system comprised of an electromagnet bar and a
wearable band-aid. We performed an evaluation of the system on 3 surface
materials, metallic, wood and acrylic, in which we showed that our approach
produces localization errors not greater than 0.25cm. Despite opportunities
for improvements, our method proved viable as a way to bring finger tracking
to settings where a perfectly smooth, clean and dry sensing surface cannot be
expected.

## References

  * (1)
  * (2) 2010\.  EvoMouse.  (2010).  https://www.celluon.com/evomouse/
  * (3) 2010\.  Leap Motion.  (2010).  https://www.leapmotion.com/
  * (4) 2015\.  InvenSense, MPU-9250 Nine-Axis MEMS MotionTracking Device.  (2015).  http://www.invensense.com/-products/motion-tracking/9-axis/mpu-9250/
  * (5) Daniel Ashbrook, Patrick Baudisch, and Sean White. 2011.  Nenya: Subtle and Eyes-free Mobile Input with a Magnetically-tracked Finger Ring. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’11). ACM, New York, NY, USA, 2043–2046.  DOI:http://dx.doi.org/10.1145/1978942.1979238
  * (6) Ke-Yu Chen, Kent Lyons, Sean White, and Shwetak Patel. 2013.  uTrack: 3D Input Using Two Magnetic Sensors. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST ’13). ACM, New York, NY, USA, 237–244.  DOI:http://dx.doi.org/10.1145/2501988.2502035
  * (7) Ke-Yu Chen, Shwetak N. Patel, and Sean Keller. 2016.  Finexus: Tracking Precise Motions of Multiple Fingertips Using Magnetic Sensing. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI ’16). ACM, New York, NY, USA, 1504–1514.  DOI:http://dx.doi.org/10.1145/2858036.2858125
  * (8) Houde Dai, Shuang Song, Xianping Zeng, Shjian Su, Mingqiang Lin, and Max Q-H Meng. 2017.  6-D electromagnetic tracking approach using uniaxial transmitting coil and tri-axial magneto-resistive sensor.  IEEE Sensors Journal 18, 3 (2017), 1178–1186.
  * (9) Isabelo de los Reyes, Nathanael Roberton, Brian Calvery, Timothy JE Turner, Adrian Chandley, Daniel Makoski, Paul Henderson, Egor Nikitin, Tarek Elabbady, Phillip Joe, and others. 2010.  Function-oriented user interface.  (June 1 2010).  US Patent 7,730,425.
  * (10) Florian Echtler, Andreas Dippon, Marcus Tönnis, and Gudrun Klinker. 2009.  Inverted FTIR: Easy Multitouch Sensing for Flatscreens. In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces (ITS ’09). ACM, New York, NY, USA, 29–32.  DOI:http://dx.doi.org/10.1145/1731903.1731909
  * (11) Brian W Epps. 1986.  Comparison of six cursor control devices based on Fitts’ law models. In Proceedings of the Human Factors Society Annual Meeting, Vol. 30. SAGE Publications Sage CA: Los Angeles, CA, 327–331.
  * (12) Xin Ge, Dakun Lai, Xiaomei Wu, and Zuxiang Fang. 2009.  A novel non-model-based 6-DOF electromagnetic tracking method using non-iterative algorithm. In 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE, 5144–5117.
  * (13) Xinying Han, Hiroaki Seki, Yoshitsugu Kamiya, and Masatoshi Hikizub. 2007\.  Wearable handwriting input device using magnetic field. In SICE Annual Conference 2007. 365–368.  DOI:http://dx.doi.org/10.1109/SICE.2007.4421009
  * (14) Chris Harrison, Hrvoje Benko, and Andrew D. Wilson. 2011.  OmniTouch: Wearable Multitouch Interaction Everywhere. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST ’11). ACM, New York, NY, USA, 441–450.  DOI:http://dx.doi.org/10.1145/2047196.2047255
  * (15) Chris Harrison and Scott E. Hudson. 2009.  Abracadabra: Wireless, High-precision, and Unpowered Finger Input for Very Small Mobile Devices. In Proceedings of the 22Nd Annual ACM Symposium on User Interface Software and Technology (UIST ’09). ACM, New York, NY, USA, 121–124.  DOI:http://dx.doi.org/10.1145/1622176.1622199
  * (16) Ramon Hofer, Daniel Naeff, and Andreas Kunz. 2009.  FLATIR: FTIR Multi-touch Detection on a Discrete Distributed Sensor Array. In Proceedings of the 3rd International Conference on Tangible and Embedded Interaction (TEI ’09). ACM, New York, NY, USA, 317–322.  DOI:http://dx.doi.org/10.1145/1517664.1517730
  * (17) Edwin L Hutchins, James D Hollan, and Donald A Norman. 1985.  Direct manipulation interfaces.  Human-computer interaction 1, 4 (1985), 311–338.
  * (18) Hyosu Kim, Anish Byanjankar, Yunxin Liu, Yuanchao Shu, and Insik Shin. 2018.  UbiTap: Leveraging Acoustic Dispersion for Ubiquitous Touch Interface on Solid Surfaces. In Proceedings of the 16th ACM Conference on Embedded Networked Sensor Systems (SenSys ’18). ACM, New York, NY, USA, 211–223.  DOI:http://dx.doi.org/10.1145/3274783.3274848
  * (19) Rong-Hao Liang, Kai-Yin Cheng, Chao-Huai Su, Chien-Ting Weng, Bing-Yu Chen, and De-Nian Yang. 2012.  GaussSense: Attachable Stylus Sensing Using Magnetic Sensor Grid. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology (UIST ’12). ACM, New York, NY, USA, 319–326.  DOI:http://dx.doi.org/10.1145/2380116.2380157
  * (20) J. Liu, Y. Chen, M. Gruteser, and Y. Wang. 2017.  VibSense: Sensing Touches on Ubiquitous Surfaces through Vibration. In 2017 14th Annual IEEE International Conference on Sensing, Communication, and Networking (SECON). 1–9.  DOI:http://dx.doi.org/10.1109/SAHCN.2017.7964907
  * (21) Pranav Mistry and Pattie Maes. 2011.  Mouseless: A Computer Mouse As Small As Invisible. In CHI ’11 Extended Abstracts on Human Factors in Computing Systems (CHI EA ’11). ACM, New York, NY, USA, 1099–1104.  DOI:http://dx.doi.org/10.1145/1979742.1979715
  * (22) Makoto Ono, Buntarou Shizuki, and Jiro Tanaka. 2013.  Touch & activate: adding interactivity to existing objects using active acoustic sensing. In Proceedings of the 26th annual ACM symposium on User interface software and technology. ACM, 31–40.
  * (23) Ivan Poupyrev, Nan-Wei Gong, Shiho Fukuhara, Mustafa Emre Karagozler, Carsten Schwesig, and Karen E Robinson. 2016.  Project Jacquard: interactive digital textiles at scale. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 4216–4227.
  * (24) F. H. Raab, E. B. Blood, T. O. Steiner, and H. R. Jones. 1979\.  Magnetic Position and Orientation Tracking System.  IEEE Trans. Aerospace Electron. Systems AES-15, 5 (Sep. 1979), 709–718.  DOI:http://dx.doi.org/10.1109/TAES.1979.308860
  * (25) Jun Rekimoto. 2002.  SmartSkin: An Infrastructure for Freehand Manipulation on Interactive Surfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’02). ACM, New York, NY, USA, 113–120.  DOI:http://dx.doi.org/10.1145/503376.503397
  * (26) Ilya Rosenberg and Ken Perlin. 2009.  The UnMousePad: An Interpolating Multi-touch Force-sensing Input Pad. In ACM SIGGRAPH 2009 Papers (SIGGRAPH ’09). ACM, New York, NY, USA, Article 65, 9 pages.  DOI:http://dx.doi.org/10.1145/1576246.1531371
  * (27) Munehiko Sato, Ivan Poupyrev, and Chris Harrison. 2012.  Touché: Enhancing Touch Interaction on Humans, Screens, Liquids, and Everyday Objects. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’12). ACM, New York, NY, USA, 483–492.  DOI:http://dx.doi.org/10.1145/2207676.2207743
  * (28) A. Sulaiman, K. Poletkin, and A. W. H. Khong. 2010.  Source Localization in the Presence of Dispersion for Next Generation Touch Interface. In 2010 International Conference on Cyberworlds. 82–86.  DOI:http://dx.doi.org/10.1109/CW.2010.72
  * (29) Ke Sun, Ting Zhao, Wei Wang, and Lei Xie. 2018.  VSkin: Sensing Touch Gestures on Surfaces of Mobile Devices Using Acoustic Signals. In Proceedings of the 24th Annual International Conference on Mobile Computing and Networking (MobiCom ’18). ACM, New York, NY, USA, 591–605.  DOI:http://dx.doi.org/10.1145/3241539.3241568
  * (30) Yoshiki Takeoka, Takashi Miyaki, and Jun Rekimoto. 2010.  Z-touch: An Infrastructure for 3D Gesture Interaction in the Proximity of Tabletop Surfaces. In ACM International Conference on Interactive Tabletops and Surfaces (ITS ’10). ACM, New York, NY, USA, 91–94.  DOI:http://dx.doi.org/10.1145/1936652.1936668
  * (31) Andrew D. Wilson. 2004.  TouchLight: An Imaging Touch Screen and Display for Gesture-based Interaction. In Proceedings of the 6th International Conference on Multimodal Interfaces (ICMI ’04). ACM, New York, NY, USA, 69–76.  DOI:http://dx.doi.org/10.1145/1027933.1027946
  * (32) Andrew D. Wilson. 2005.  PlayAnywhere: A Compact Interactive Tabletop Projection-vision System. In Proceedings of the 18th Annual ACM Symposium on User Interface Software and Technology (UIST ’05). ACM, New York, NY, USA, 83–92.  DOI:http://dx.doi.org/10.1145/1095034.1095047
  * (33) Robert Xiao, Chris Harrison, and Scott E. Hudson. 2013.  WorldKit: Rapid and Easy Creation of Ad-hoc Interactive Applications on Everyday Surfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’13). ACM, New York, NY, USA, 879–888.  DOI:http://dx.doi.org/10.1145/2470654.2466113
  * (34) Chungkuk Yoo, Inseok Hwang, Eric Rozner, Yu Gu, and Robert F. Dickerson. 2016.  SymmetriSense: Enabling Near-Surface Interactivity on Glossy Surfaces Using a Single Commodity Smartphone. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI ’16). ACM, New York, NY, USA, 5126–5137.  DOI:http://dx.doi.org/10.1145/2858036.2858286
  * (35) Cheng Zhang, Qiuyue Xue, Anandghan Waghmare, Sumeet Jain, Yiming Pu, Sinan Hersek, Kent Lyons, Kenneth A. Cunefare, Omer T. Inan, and Gregory D. Abowd. 2017b.  SoundTrak: Continuous 3D Tracking of a Finger Using Active Acoustics.  Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1, 2, Article 30 (June 2017), 25 pages.  DOI:http://dx.doi.org/10.1145/3090095
  * (36) Yang Zhang, Gierad Laput, and Chris Harrison. 2017a.  Electrick: Low-Cost Touch Sensing Using Electric Field Tomography. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI ’17). ACM, New York, NY, USA, 1–14.  DOI:http://dx.doi.org/10.1145/3025453.3025842
  * (37) Yang Zhang, Junhan Zhou, Gierad Laput, and Chris Harrison. 2016.  SkinTrack: Using the Body As an Electrical Waveguide for Continuous Finger Tracking on the Skin. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI ’16). ACM, New York, NY, USA, 1491–1503.  DOI:http://dx.doi.org/10.1145/2858036.2858082