# Analysis of Time-Distance Helioseismology for Detection of Emerging Active
Regions

John T. Stefan Department of Physics, New Jersey Institute of Technology,
Newark, NJ 07102  Alexander G. Kosovichev Department of Physics, New Jersey
Institute of Technology, Newark, NJ 07102  NASA Ames Research Center, Moffett
Field, Mountain View, CA 94040  Andrey M. Stejko Department of Physics, New
Jersey Institute of Technology, Newark, NJ 07102

(Received 12/02/20; Revised 02/01/21; Revised 02/23/21; Accepted -)

###### Abstract

A time-distance helioseismic technique, similar to the one used by Ilonidis et
al, is applied to two independent numerical models of subsurface sound-speed
perturbations to determine the spatial resolution and accuracy of phase travel
time shift measurements. The technique is also used to examine pre-emergence
signatures of several active regions observed by the Michelson Doppler Imager
(MDI) and the Helioseismic Magnetic Imager (HMI). In the context of similar
measurements of quiet sun regions, three of the five studied active regions
show strong phase travel time shifts several hours prior to emergence. These
results form the basis of a discussion of noise in the derived phase travel
time maps and possible criteria to distinguish between true and false positive
detection of emerging flux.

helioseismology – solar active regions – solar magnetic flux emergence

##  1 Introduction

As society relies more heavily on space-based infrastructure, there is an
increasing need for reliable prediction and forecasting of space weather and
geomagnetic activity. It is well-understood by now that the Sun is the main
influence on the geospace environment; therefore, solar variability must form
the basis of any space weather forecasting tools. Decades of observation have
shown that the Sun is subject to a 22-year magnetic cycle, with sunspot number
and active region size, strength, and location being the main manifestations
of this cycle. Active regions are characterized by their strong magnetic
fields which contribute to their high solar flare productivity, relative to
other areas of the Sun. While determining when an active region will flare
remains a complex and open problem, forecasting the development of active
regions is slightly more straightforward.

A well-regarded hypothesis for the development of active regions is the rising
flux tube model. MHD modeling has shown that strong shearing, such as found at
the boundary between the solar radiative and convective zones, can turn weak
poloidal magnetic fields into strong toroidal fields (Hughes et al., 1998;
Brummell et al., 2002). The resulting concentration of magnetic field, or flux
tube, experiences a buoyant force which causes the tube to bend and rise to
the surface (Cline et al., 2003). This upside-down ”u”-shaped tube explains
the bipolar nature of many active regions (Parker, 1955). Importantly, the
sub-surface origin of active regions should allow for detection before the
magnetic flux reaches the solar surface.

There are many helioseismic techniques which are useful in performing this
detection, some of which have already had success in identifying existing
active regions on the far-side of the Sun. Far-side imaging is accomplished
primarily using two different methods: acoustic holography and time-distance
analysis. In acoustic holography, the far-side wavefield is obtained from
near-side measurements and inhomegeneities in the wavefield, which are
measured by the relative phase-shift, can be attributed to the strength of
magnetic field (Malanushenko et al., 2004; Gizon et al., 2018). Time-distance
analysis, as the name suggests, measures shifts in phase travel time for waves
travelling between two selected points. Applying specific Fourier filters to
the data and methodically selecting the points isolates waves which bounce
through the solar interior several times to the far-side and back to the near-
side. This multi-skip analysis relies on local conditions perturbing the phase
travel time, and combinations of measurement configurations work to refine the
signal (Hartlep et al., 2008; Zhao, 2007).

The previously mentioned works, generally, have only focused on the detection
of existing active regions on the far-side surface, though the same methods
have also been applied to the pre-emergence detection of near-side active
regions. Investigation of helioseismic holographic techniques go as far back
as the 1990’s, with notable developments made by Lindsey & Braun (1997) in
detecting sub-surface flows prior to active region emergence. A detailed study
of emerging active regions using the holographic technique was performed by
Birch et al. (2013) and statistically significant differences in the mean
travel time deviation were observed. These mean travel time deviations are
used by the authors to estimate the strength of converging flows surrounding
the rising magnetic flux. Additionally, more recent tests on models of
sunspots using holography, as in Felipe et al. (2016), have attempted to
separate the effects of thermal and magnetic contributions to acoustic travel
times.

Some early examples of studies performed with the time-distance technique are
Kosovichev et al. (2000) and Kosovichev et al. (2001), where the theoretical
framework for time-distance helioseismology as well as detailed inversion
procedures are provided. The authors apply their inversion procedure to an
emerging active region observed by the Michelson Doppler Imager (MDI), where
they successfully identify subsurface structures and estimate the rate of
emergence of the magnetic flux. Kholikov (2013) also studies pre-emergence
active regions, some of which are the same regions as in Kosovichev et al.
(2000); it is found here that most travel time maps of active regions show a
10-15 second perturbation around the eventual location of the region on the
solar surface. Further discussion of helioseismic methods and results are
given in Gizon (2006), as well as a summary of efforts to detect sub-surface
sound speed perturbations.

Of particular note are the results of Ilonidis et al. (2012) and Ilonidis et
al. (2013); the first work establishes a measurement procedure which produces
a reported SNR of 4.2-4.9, while the second work further explains the
procedure and examines other helioseismic signatures, such as frequency
shifts, which may be indicative of emerging magnetic flux. The distinguishing
features of this method are the division of annuli into several pairs of
opposing arcs, as well as averaging of cross-correlation functions over these
arcs and over groups of annuli. Using their method, the authors report a
positive detection of emerging magnetic flux as well as estimates of the rate
of flux emergence.

In this work, we use a modified version of the time-distance method developed
by Ilonidis et al. (2012) to detect phase travel time shifts prior to the
emergence of several active regions. We first test the method on numerical
simulations to determine the sensitivity to sub-surface sound-speed
perturbations. We then examine two active regions already studied by Ilonidis
et al. (2011), NOAA AR 10488 and AR 07978, with data obtained by MDI. We also
examine three more recent active regions, AR 11158, AR 12257, and AR 12772
with data obtained by the Helioseismic and Magnetic Imager (HMI). The results
of the active region analysis are compared with a similar analysis of quiet
sun regions, where the ambient magnetic field is relatively uniform and no
significant features develop over the duration of data collection. We discuss
characteristics of noise in the phase travel time measurements of these quiet
sun regions, and suggest criteria to distinguish between true positive and
false positive emerging flux detection.

##  2 Methods and Data

Half of the observational data used are MDI Dopplergrams covering an area of
30 by 30 degrees, with a resolution of 0.12 degrees per pixel, and 480 frames
with 60 second cadence. The remaining half of observational data used are HMI
Dopplergrams, covering identical spatial extent and duration, with 45 second
cadence and a downgraded resolution of 0.12 degrees per pixel. The high
resolution of HMI - 0.06 degrees per pixel - does not noticeably improve phase
travel time measurements, and only increases computation time. Both the HMI
and MDI Dopplergrams are remapped to a Postel’s projection and tracked at the
Carrington rotation rate. We use simulated data from two separate 3D
hydrodynamic models for testing, the first is developed by Hartlep et al.
(2011) (referred to as Model 1) and the second developed by Stejko et al.
(2020) (referred to as Model 2). Data from model 1 has a spatial resolution of
0.7 degrees per pixel and 60 second cadence, and data from model 2 has the
same resolution and cadence as MDI.

###  2.1 Measuring Shifts in Phase Travel Time

To begin, the data is treated with a dual Fourier filter; the first is a flat-
top frequency filter which preserves oscillations only in the 2 to 5 mHz
range, the resonant portion of the acoustic power spectrum. The second Fourier
filter is a piece-wise Gaussian in phase speed, $v_{p}=\omega/k$, and the
parameters are selected to isolate oscillations with a turning point between
40 and 70 Mm beneath the photosphere. The Standard Solar Model (Christensen-
Dalsgaard et al., 1996) is used to estimate the sound speed at these depths,
which are $v_{p1}=92$ km/s at 40 Mm and $v_{p2}=127$ km/s at 70 Mm.
Oscillations with phase speeds slower than $v_{p1}$ are filtered with a
Gaussian centered at $v_{p1}$ with a width of $\delta v=8.7$ km/s.
Oscillations between between $v_{p1}$ and $v_{p2}$ are untreated, and the
oscillations above $v_{p2}$ are filtered with a Gaussian centered at $v_{p2}$
with the same width as the lower filter. In other words, the filters used are

$F_{1}(\nu)=\left\\{\begin{array}[]{cc}1&2mHz\leq\nu\leq 5mHz\\\
0&otherwise\end{array}\right.$

and

$F_{2}(v_{p})=\left\\{\begin{array}[]{cc}\exp\left[-\dfrac{\left(v_{p}-v_{p1}\right)^{2}}{2{\delta
v}^{2}}\right]&v_{p}<v_{p1}\\\ 1&v_{p1}\leq v_{p}\leq v_{p2}\\\
\exp\left[-\dfrac{\left(v_{p}-v_{p2}\right)^{2}}{2{\delta
v}^{2}}\right]&v_{p2}<v_{p}\end{array}\right.,$

and Figure 1 shows a typical power spectrum with the bounds of the Fourier
filters highlighted in white.

Now that the data has been filtered to isolate oscillations with appropriate
turning points, the cross-correlations are computed as in Duvall et al.
(1993). Each pixel in the image has its corresponding spherical coordinates
determined based on the heliographic location of the image’s center pixel, and
concentric rings centered on a given pixel are selected with radii equal to
half the horizontal travel distance of the desired oscillations. The radius of
the first ring is 4.56 degrees, corresponding to oscillations with the turning
point at 40 Mm, and the radii increase in increments of 0.12 degrees - MDI
spatial resolution - up to 8.16 degrees, corresponding to oscillations with
the turning point at 70 Mm. Once the rings have been identified, 5 different
configurations are used to compute the cross-correlation: the first
configuration subdivides the ring into 3 opposing pairs of arcs, the second
into 4 opposing pairs, and so on until the fifth configuration with 7 opposing
pairs. Signals within these arcs are spatially averaged and used to compute a
point-to-point cross-correlation for the selected turning point depth, arc
configuration, and spatial position.

The cross-correlations for positive and negative time lags are computed
separately and are fit to a Gabor wavelet (Nigam et al., 2007) of the form

$\Psi(\tau)=A\cos\left(\omega_{0}(\tau-\tau_{ph})\right)\exp\left[-\frac{1}{2}\gamma^{2}(\tau-\tau_{g})^{2}\right],$

where $\tau$ is the time lag, $\tau_{ph}$ is the phase travel time, $\tau_{g}$
is the group travel time, $\omega_{0}$ is the dominant frequency of
oscillation, and $\gamma$ is the decay rate of the Gaussian envelope. The
cross-correlations are averaged over spatial position and are then fit using a
non-linear least squares algorithm so that the phase travel time is a function
of annulus radius and number of segments used to compute the cross-
correlations. The initial values used for phase travel time, group travel
time, and frequency are their theoretical values for the given horizontal
travel distance. The cross-correlations are then shifted in time so that the
mean phase travel times for each annulus radius are the same, allowing the
cross-correlations to be averaged over distance. The cross-correlation for
each pixel is then fit again to obtain the phase travel time, as a function of
number of segments used. The phase travel time maps are then averaged to
produce a final travel time map.

While we use the same cross-correlation procedure as in Ilonidis et al.
(2012), we choose to use a different Gabor wavelet fitting procedure with the
differences being the following. First, the authors average their cross-
correlations over the different cases of number of arcs used, where we keep
these cross-correlations separate. The phase travel times differ slightly
between the widest and narrowest arcs, presumably because of waves which
scatter at the turning point leading to a ”bent” ray path and slightly longer
travel time. Narrower arcs more closely approximate the traditional, straight
ray path while the wider arcs are more likely to capture a perturbed ray path,
causing the disparity in phase travel time. Second, Ilonidis et al. (2012) use
Quiet Sun values for their pixel-by-pixel fitting, where we only use these
values to perform an initial fit on the spatially averaged cross-correlation
which serves as an initial guess for our pixel-by-pixel fit. Finally, the
authors use the Quiet Sun phase travel time to determine the shift for the
cross-correlations to be averaged over distance, while we use the phase travel
time from the spatially-averaged cross-correlation for each distance to
determine the appropriate shift. These procedures produce identical shifts if
the difference between the phase travel time for Active Regions and Quiet Sun
regions is less than the observational cadence, but this cannot be guaranteed
for ray paths which are especially long or encounter significant
perturbations.

It can be shown - as in Kosovichev & Duvall (1997) \- that the first order
perturbation to phase travel time due to a change in sound speed is

$\delta\tau_{mean}=\int_{\Gamma}\dfrac{\delta c}{c}Uds,$ (1)

where $\Gamma$ is the acoustic ray path, $\delta c$ is the perturbation to the
sound speed $c$, and $U=\omega/k$ is the phase slowness. The mean perturbation
to phase travel time, $\delta\tau_{mean}$, is computed as the simple average
of the travel time perturbations from the positive and negative time lags,
$\delta\tau_{\pm}=\tau_{\pm}-\langle\tau_{\pm}\rangle$. The reference phase
travel time for the positive or negative time lag, $\langle\tau_{\pm}\rangle$,
is obtained by taking a spatial average of the phase travel time maps. For
perturbations which occupy a small portion of the map, this average should be
approximately equal to the quiet Sun phase travel time.

##  3 Results

###  3.1 Testing on Numerical Simulations: Model 1

We begin with testing the time-distance method on numerical data from Model 1
(Hartlep et al., 2011), with two 5% reductions in sound speed of varying
spatial extent. Both perturbations are functions of depth (z) and angular
distance ($\alpha$) from the horizontal center,

$\delta c=\dfrac{A}{4}\left(1+\cos\left(\pi\dfrac{z-z_{0}}{\delta
z}\right)\right)\left(1+\cos\left(\pi\dfrac{\alpha}{\delta\alpha}\right)\right).$

The large perturbation has a horizontal width ($2\delta\alpha$) of 360 Mm and
the smaller perturbation has a horizontal width ($2\delta\alpha$) of 180 Mm;
both have a radius ($\delta z$) of 20 Mm in the vertical direction and are
centered 50 Mm ($z_{0}$) beneath the model’s surface. The large perturbation
(Figure 2a) is very well-resolved, with the strongest signal covering a
circular area roughly 200 Mm in diameter. The smaller perturbation (Figure 2b)
is distinguishable from the background noise, but the spatial features are
less clear than in the larger perturbation. Additionally, the magnitude of the
travel time shift is reduced in the smaller perturbation, though this is
expected as acoustic waves will travel less distance within the perturbed
region which reduces the shift in travel time.

To highlight the region where the phase travel time is significantly
perturbed, we set a 2$\sigma$ threshold of 8.20 seconds and overlay this
contour with a black line in Figures 2a and 2b. This threshold is obtained by
excluding data within 200 Mm of the map center which leaves only the
unperturbed regions, and the standard deviation is computed from the remaining
phase travel times. The shape of the contour in both cases differs slightly
from the circularly-symmetric perturbations, indicating that detected spatial
features may not necessarily coincide with in-situ conditions. This is further
supported by the discrepancy in Figure 2a between the perturbation width of
360 Mm and a width of 200 Mm for the detected feature. Analysis of a different
data set from Model 1 in Ilonidis et al. (2013) also yielded such a
discrepancy. The contouring also highlights some anomalous travel times in the
lower parts of Figures 2a and 2b for which we have no certain explanation.
While it is expected that some pixels not associated with the perturbation
will still exceed the threshold, both figures show a definite latitudinal
dependence of the phase travel time deviation. The grid of Model 1 is uniform
in latitude, so these anomalies may arise from the changing spatial resolution
closer to the poles.

###  3.2 Testing on Numerical Simulations: Model 2

We perform tests using Model 2 (Stejko et al., 2020) on two characteristics of
the time-distance method: dependence of sensitivity on depth and perturbation
width, and the accuracy of the travel time measurements. Figure 3 shows six
travel time maps of a perturbations with a radius in the vertical direction of
20 Mm. The left column contains perturbations with a horizontal radius of 50
Mm and the right column contains perturbations with a horizontal radius of 20
Mm. From top to bottom, the depths of the perturbations are 40, 50, and 60 Mm.
Each perturbation has an amplitude of 5% of the background sound speed and is
Gaussian in all directions. As with Model 1, we use the standard deviation of
unperturbed regions to set a 2$\sigma$ threshold to highlight strongly
perturbed regions. Four unperturbed cases are used for this computation, and
the resulting travel time maps are shown in Figure 4. The 2$\sigma$ contour is
also applied to these maps.

The most surprising feature of the perturbed travel time maps is the lack of
spatial information; the perturbations are Gaussian in the (x,y)-plane which
should imply a circular travel time shift. We see instead that the spatial
structure is convolved with the background noise, and the travel time shift
contribution added to underlying inhomogeneities. This is particularly true
for the perturbation placed at a depth of 40 Mm, which is the upper limit of
the vertical region that we sample. Furthermore, there is very little change
in the shape of the travel time measurements between the 20 and 50 Mm
perturbations at 50 and 60 Mm. Some similarities are expected as the
oscillation excitation functions are the same, but the only distinguishing
feature between the two are the magnitude of the travel time shifts. The
larger sound speed perturbation produces travel time shifts which are two to
three seconds greater than the smaller perturbation. The travel time shifts
also increase in magnitude with the depth of the perturbations; since the
sound speed perturbation is proportional to the local sound speed, we should
expect the phase travel time to vary proportionally as well.

To test the accuracy of the travel time measurements, we place a 5% reduction
in sound speed at 55 Mm below the model’s surface, with a radius of 20 Mm in
all directions. The theoretical travel time shift is computed using equation 1
and the standard solar model (Christensen-Dalsgaard et al., 1996) is used as
the background mesh. The travel time shifts are sampled similarly to the time-
distance measurement method, with signal averaged between several arc pairs,
though we select only one ray per arc pair. This is in contrast to the cross-
correlation procedure, where the Doppler signal is averaged over a given arc,
which is analogous to averaging over many rays in the ray-path computation.
The theoretical travel time map has the mean subtracted as is done with the
measurements of the perturbation, and the results for both are shown in Figure
5. The area in the measured travel time map with the greatest amplitude
roughly corresponds to the full-width half-maximum of the theoretical signal.
Interestingly, the measured travel time shifts are several seconds shorter
than what would be expected, though as previously discussed, the spatial
structure of the perturbation is convolved with the background noise. While
this difference can be explained by the noise level (4.79 seconds), it’s
expected that this noise would contribute equally to an increase in the travel
time in at least some pixels. However, we observe far more reductions in the
travel time magnitude than increases.

###  3.3 Detection of Emerging Flux and Comparison with Quiet Sun Measurements

We apply the time-distance method to five active regions using Dopplergrams
from HMI and MDI, and first examine NOAA ARs 10488 and 7978 which were
previously studied by Ilonidis et al. (2011), who measured strong travel time
shifts several hours prior to the emergence of the active regions. To
quantitatively evaluate the overall strength of the travel time shifts, we
define the perturbation index to be the sum of negative travel time shifts
with amplitude two times greater than the standard deviation of the data
series, multiplied by the area these travel time shifts occupy. For example, a
feature measuring 50 Mm2 with a magnitude of 10 seconds will have a
perturbation index of 500 s Mm2. Thus the perturbation index will increase if
either the area or the magnitude of a travel time shift increases.

AR 10488 shows a small increase in the perturbation index 25 hours before
emergence and stronger increase five hours before emergence, as shown in
Figure 6a. A large peak in the perturbation index appears roughly 25 hours
after AR 10488 begins emerging; extended observations by Ilonidis et al.
(2011) show that the magnetic flux rate decreases one day after emergence,
before abruptly increasing. We interpret this strong peak as additional flux
which strengthens the existing magnetic field. The perturbation index of AR
7978, shown in Figure 7a is slightly more noisy than that of AR 10488, and
shows a strong peak 20 hours before emergence. The magnetic flux of the active
region increases slowly for about 15 hours after emerging, and then
experiences a strong increase in the remaining 15 hours of observation. A
moderate peak centered around the time of emergence precedes the change in the
flux rate by 13 to 15 hours.

AR 12772 is a relatively weak active region, with an unsigned magnetic field
not exceeding 850 Gauss. We are still able to detect a significant peak in the
perturbation index (Figure 8a) 20 hours before the region emerges. In
contrast, AR 11158 is a fairly strong active region, with the magnetic field
exceeding 1000 Gauss at the end of our observations. The configuration of the
field also has a more significant bipolar structure than AR 12772. For this
region, we see a gradual rise in the perturbation index (Figure 9a) 25 hours
prior to emergence which peaks around 10 hours before the unsigned flux begins
increasing. Additionally, there are some smaller changes following initial
emergence which precede sharp changes in the unsigned magnetic flux. Finally,
we see a moderate increase in the perturbation index of AR 12257 (Figure 10a)
20 hours before the region emerges, and a slightly smaller increase centered
around the region’s emergence. This second increase is more representative of
expected emergence behavior: a gradual rise in the perturbation index,
followed by a sharp decline as the concentration of magnetic flux rises out of
focus.

In contrast to the perturbation index of the active regions, the Quiet Sun
regions show no gradual increases of the perturbation index, as observed in
some active regions, over 60 hours of observation (Figure 11). The
perturbation index of the Quiet Sun region at a latitude of +15∘ does show a
rapid increase at 38 hours followed by a gradual decline, though this isn’t
observed in the other Quiet Sun regions or any of the active regions. We
expect, and observe in AR’s 11158 and 12257, an opposite trend where there is
a gradual increase in the perturbation index followed by a sharp decline. This
can be interpreted as a concentration of magnetic flux just emerging into
focus at 70 Mm, slightly perturbing the phase travel time. As the flux rises,
it encounters more acoustic rays in our observed depths of 40-70 Mm, further
perturbing the measured phase travel time. The perturbation index will peak
once the lower extent of the magnetic flux passes through a depth of 70 Mm,
then decrease quicker than the initial rise. This asymmetrical development
comes from the uneven sensitivity of acoustic rays which we use to probe the
interior. As evidenced in Figure 3, deeper perturbations can cause a stronger
phase travel time shift as compared to a perturbation with equal magnitude at
a shallower depth. Since acoustic rays which penetrate to 70 Mm have less
curvature than the rays which penetrate to 40 Mm, a wave packet will travel
more distance in the region close to the turning point, making it more
sensitive to travel time shifts.

##  4 Discussion and Conclusion

Aside from the rate at which the perturbation index rises, we should also take
into account the magnitude of the peak when evaluating whether or not an
active region will emerge. The perturbation index of Quiet Sun regions can
occasionally reach values close to 13000 s Mm2, though it is more likely that
it will fluctuate around a mean of 4000 to 6000 s Mm2. If we choose the strict
criteria that a pre-emergence active region can only be identified by
exceeding any Quiet Sun perturbation index values (i.e exceed 13000 s Mm2),
then a successful identification can be made in three of the five active
regions studied - not including the secondary emergence signature detected in
AR 10488.

While the perturbation index is a useful tool in looking for pre-emergence
signatures, it is difficult to to determine the uncertainty associated with
it, and this is especially true for observational data. Our procedure involves
several non-linear operations - such as the Gabor wavelet fitting - which
confounds the uncertainties of the original measurements. It could be measured
from the previously discussed acoustic models without any sound-speed
perturbations, but these models lack true convection and other important
processes which may distort the phase travel time. We decide to compute the
uncertainty using the Quiet Sun measurements, under the assumption that there
are no significant inhomogeneities present. We compute the RMS of the phase
travel time deviation for each frame in the 60-hour series of the three
latitudes studied, 4.307 seconds, and use this as the noise level. The
uncertainty in the perturbation index at each frame is then computed as the
area of the detected perturbation multiplied by the noise level.

Unfortunately, detailed spatial features cannot be observed using the time-
distance method, as the average wavelength of oscillations which reach our
observation depths is around 3 Mm. For this reason, we use MDI resolution
(0.12 degrees per pixel) in our HMI observations, as the travel time will have
minimal variation between pixels at a resolution of 0.06 degrees per pixel.
This also speeds up the computation for HMI phase travel time maps compared to
using the native resolution. While we cannot measure significantly detailed
spatial features, the placement of positive and negative travel time shifts
relative to the eventual positions of active regions is interesting. At the
time of peak emergence, there are areas of positive travel time shift
coincident with the eventual location on the surface of magnetic flux. The
negative travel time shifts, which are used to compute the perturbation index,
are adjacent to the eventual concentrations of magnetic flux on the surface.

In earlier stages of emergence, we measure negative travel time shifts
coincident with the surface magnetic flux, as in the work of Ilonidis et al.
(2011), though the later, adjacent shifts are stronger in our observations. In
response to the results of Ilonidis et al. (2011), Braun (2012) comment that
their signatures could not be detected using a different method, and that the
results are inconsistent with models of emerging flux. In response, Ilonidis
et al. (2012) point to differences in the measurement techniques, including
their higher SNR, and offer some possible physical processes which may alter
the measured phase travel times. One of the primary arguments made by Braun
(2012) is that direct perturbations to the sound speed due to the magnetic
field are two orders of magnitude smaller than perturbations due to flows
induced by the magnetic field. Braun (2012) cite Birch et al. (2010), who find
that the radial flows are strongest above the magnetic flux and azimuthal
flows are strongest adjacent to the flux; the authors also state that the
azimuthal flows tend to dominate. The information provided by Braun (2012) and
Birch et al. (2010) may explain our observation of strong travel time shifts
adjacent to the active regions prior to emergence, and magnetic stresses and
flow perturbations may contribute to the observed travel time shifts. It
should also be noted that while sound-speed perturbations in simulations can
produce travel time shifts similar to the detected pre-emergence signatures,
this does not necessarily imply that those signatures are caused by a sound-
speed perturbation.

In this work, we used a time-distance method based on the ray approximation to
compute phase travel time differences. We first applied this method to two
numerical models in order to evaluate the sensitivity and spatial resolution.
We find that large perturbations to sound speed are easily detectable, with
moderate spatial detail. Smaller perturbations can be detected with the loss
of some spatial information, though detection is difficult if the perturbation
is too small or weak.

We then applied the time-distance method to five active regions - two observed
by MDI and three by HMI - as well as Quiet Sun regions where there was no
significant enhancement of the magnetic field. Measuring the perturbation
index, a proxy for strength of travel time shifts, we are able to definitively
identify three of five active regions before emergence, as well as secondary
development in an additional active region. We confirm the detection of AR’s
10488 and 7978 by Ilonidis et al. (2011), and find that the spatial position
of travel time shifts in all studied active regions support the results of
Braun (2012) and Birch et al. (2010).

There is still much work to be done in developing this time-distance method.
While our results show the method is viable, the relationship between the
magnitude and duration of travel time shifts to the strength and size of an
active region are still unclear. In further studies, we will perform a broad
statistical survey of historical active regions observed by MDI and HMI. It is
suspected that temporal development of the perturbation index can disentangle
the false positives in Quiet Sun measurements from true identification of
emerging flux, and this statistical survey should be able to validate or
invalidate this hypothesis.

This work was supported by NASA grants NNX14AB70G, 80NSSC19K0268,
80NSSC19K1436, 80NSSC20K0602, and 80NSSC20K1870. The authors would like to
thank Dr. Thomas Hartlep for his publicly available solar acoustic models, and
the NASA Advanced Supercomputing Division and the Stanford Solar Group for
providing access to observational data and computing facilities. Special
thanks to the Heliophysics Modeling and Simulation Group at NASA Ames Research
Center for their advice and support.

## References

  * Birch et al. (2010) Birch, A. C., Braun, D. C., & Fan, Y. 2010, ApJ, 723, L190, doi: 10.1088/2041-8205/723/2/L190
  * Birch et al. (2013) Birch, A. C., Braun, D. C., Leka, K. D., Barnes, G., & Javornik, B. 2013, ApJ, 762, 131, doi: 10.1088/0004-637X/762/2/131
  * Braun (2012) Braun, D. C. 2012, Science, 336, 296, doi: 10.1126/science.1215425
  * Brummell et al. (2002) Brummell, N., Cline, K., & Cattaneo, F. 2002, MNRAS, 329, L73, doi: 10.1046/j.1365-8711.2002.05183.x
  * Christensen-Dalsgaard et al. (1996) Christensen-Dalsgaard, J., Dappen, W., Ajukov, S. V., et al. 1996, Science, 272, 1286, doi: 10.1126/science.272.5266.1286
  * Cline et al. (2003) Cline, K. S., Brummell, N. H., & Cattaneo, F. 2003, ApJ, 588, 630, doi: 10.1086/373894
  * Duvall et al. (1993) Duvall, T. L., J., Jefferies, S. M., Harvey, J. W., & Pomerantz, M. A. 1993, Nature, 362, 430, doi: 10.1038/362430a0
  * Felipe et al. (2016) Felipe, T., Braun, D. C., Crouch, A. D., & Birch, A. C. 2016, ApJ, 829, 67, doi: 10.3847/0004-637X/829/2/67
  * Gizon (2006) Gizon, L. 2006, Modern Physics Letters A, 21, 1701, doi: 10.1142/S0217732306021207
  * Gizon et al. (2018) Gizon, L., Fournier, D., Yang, D., Birch, A. C., & Barucq, H. 2018, A&A, 620, A136, doi: 10.1051/0004-6361/201833825
  * Hartlep et al. (2011) Hartlep, T., Kosovichev, A. G., Zhao, J., & Mansour, N. N. 2011, Sol. Phys., 268, 321, doi: 10.1007/s11207-010-9544-1
  * Hartlep et al. (2008) Hartlep, T., Zhao, J., Mansour, N. N., & Kosovichev, A. G. 2008, ApJ, 689, 1373, doi: 10.1086/592721
  * Hughes et al. (1998) Hughes, D. W., Falle, S. A. E. G., & Joarder, P. 1998, MNRAS, 298, 433, doi: 10.1046/j.1365-8711.1998.01622.x
  * Ilonidis et al. (2013) Ilonidis, S., Zhao, J., & Hartlep, T. 2013, ApJ, 777, 138, doi: 10.1088/0004-637X/777/2/138
  * Ilonidis et al. (2011) Ilonidis, S., Zhao, J., & Kosovichev, A. 2011, Science, 333, 993, doi: 10.1126/science.1206253
  * Ilonidis et al. (2012) Ilonidis, S., Zhao, J., & Kosovichev, A. 2012, Science, 336, 296, doi: 10.1126/science.1215539
  * Ilonidis et al. (2012) Ilonidis, S., Zhao, J., & Kosovichev, A. G. 2012, in Astronomical Society of the Pacific Conference Series, Vol. 462, Progress in Solar/Stellar Physics with Helio- and Asteroseismology, ed. H. Shibahashi, M. Takata, & A. E. Lynas-Gray, 283.  https://arxiv.org/abs/1203.2546
  * Kholikov (2013) Kholikov, S. 2013, Sol. Phys., 287, 229, doi: 10.1007/s11207-013-0321-9
  * Kosovichev & Duvall (1997) Kosovichev, A. G., & Duvall, T. L., J. 1997, Astrophysics and Space Science Library, Vol. 225, Acoustic tomography of solar convective flows and structures, ed. F. P. Pijpers, J. Christensen-Dalsgaard, & C. S. Rosenthal, 241–260, doi: 10.1007/978-94-011-5167-2_26
  * Kosovichev et al. (2001) Kosovichev, A. G., Duvall, T. L., J., & Scherrer, P. H. 2001, in Astronomical Society of the Pacific Conference Series, Vol. 248, Magnetic Fields Across the Hertzsprung-Russell Diagram, ed. G. Mathys, S. K. Solanki, & D. T. Wickramasinghe, 169
  * Kosovichev et al. (2000) Kosovichev, A. G., Duvall, T. L. _Jr., J., & Scherrer, P. H. 2000, Sol. Phys., 192, 159, doi: 10.1023/A:1005251208431
  * Lindsey & Braun (1997) Lindsey, C., & Braun, D. C. 1997, ApJ, 485, 895, doi: 10.1086/304445
  * Malanushenko et al. (2004) Malanushenko, A., Braun, D., Kholikov, S., Leibacher, J., & Lindsey, C. 2004, in Multi-Wavelength Investigations of Solar Activity, ed. A. V. Stepanov, E. E. Benevolenskaya, & A. G. Kosovichev, Vol. 223, 283–284, doi: 10.1017/S1743921304005940
  * Nigam et al. (2007) Nigam, R., Kosovichev, A. G., & Scherrer, P. H. 2007, ApJ, 659, 1736, doi: 10.1086/512535
  * Parker (1955) Parker, E. N. 1955, ApJ, 121, 491, doi: 10.1086/146010
  * Stejko et al. (2020) Stejko, A. M., Kosovichev, A. G., & Mansour, N. N. 2020, 3D GLASS – Acoustic Solar Modeling of Background Flows.  https://arxiv.org/abs/2011.03131
  * Zhao (2007) Zhao, J. 2007, ApJ, 664, L139, doi: 10.1086/520837

Figure 1: Power spectrum of a typical datacube used to compute phase travel
time maps. The upper and lower horizontal white lines correspond to the edges
of the flat-top filter at 2 and 5 mHz. The diagonal white lines form the
center for the upper and lower phase speed filters. A Gaussian phase speed
filter of width 8.7 km/s is applied to data outside the white box. Figure 2:
Measurements of phase travel time deviation for the sound speed perturbations
in Model 1. a) Measurements for the perturbation with horizontal width of 360
Mm; b) Measurements for the perturbation with horizontal width of 180 Mm. The
solid black line denotes the 2$\sigma$ threshold of 8.20 seconds derived from
the unperturbed regions of the model. Figure 3: Travel time shifts measured in
Model 2 from a 5% increase in sound speed with a vertical radius of 20 Mm, at
a depth of 40 Mm (a,b), 50 Mm (c,d), and 60 Mm (e,f). The horizontal radius is
50 Mm for (a,c,e) and 20 Mm for (b,d,f). The solid black line denotes the
2$\sigma$ threshold of 9.58 seconds derived from empty model runs. Figure 4:
Travel time shifts measured in Model 2 for the unperturbed case. These maps
are used to determine the $2\sigma$ threshold (shown here by the black
contours) of Figures 3 and 5. Figure 5: Comparison between the measured (a)
and theoretical (b) travel time shifts for a 5% reduction in sound speed. The
scale is adjusted to align with the maximum and minimum of the theoretical
travel time shifts. The solid black line denotes the 2$\sigma$ threshold of
9.58 seconds derived from empty model runs. Figure 6: Analysis of AR 10488. a)
Perturbation index over time (black) and the unsigned magnetic flux (red); the
vertical red dashed line indicates the approximate time of emergence. b)
Travel time map at the time of peak perturbation index; the regions which meet
the 2$\sigma$ threshold are bounded by the solid dashed line. c) Magnetogram
of the active region at the end of the data series. Figure 7: Analysis of AR
7978. a) Perturbation index over time (black) and the unsigned magnetic flux
(red); the vertical red dashed line indicates the approximate time of
emergence. b) Travel time map at the time of peak perturbation index; the
regions which meet the 2$\sigma$ threshold are bounded by the solid black
line. c) Magnetogram of the active region at the end of the data series.
Figure 8: Analysis of AR 12772. a) Perturbation index over time (black) and
the unsigned magnetic flux (red); the vertical red dashed line indicates the
approximate time of emergence. b) Travel time map at the time of peak
perturbation index; the regions which meet the 2$\sigma$ threshold are bounded
by the solid black line. c) Magnetogram of the active region at the end of the
data series. Figure 9: Analysis of AR 11158. a) Perturbation index over time
(black) and the unsigned magnetic flux (red); the vertical red dashed line
indicates the approximate time of emergence. b) Travel time map at the time of
peak perturbation index; the regions which meet the 2$\sigma$ threshold are
bounded by the solid black line. c) Magnetogram of the active region at the
end of the data series. Figure 10: Analysis of AR 12257. a) Perturbation index
over time (black) and the unsigned magnetic flux (red); the vertical red
dashed line indicates the approximate time of emergence. b) Travel time map at
the time of peak perturbation index; the regions which meet the 2$\sigma$
threshold are bounded by the solid black line. c) Magnetogram of the active
region at the end of the data series. Figure 11: Perturbation index of 3 Quiet
Sun regions with heliographic latitude +15∘ (black), 0∘ (blue), and -15∘
(red), and equal longitude.


# Co-speech Gestures for Human-Robot Collaboration ††thanks: Project funding
was received from EU’s Horizon 2020 research and innovation programme, grant
no. 871449 (OpenDR).

Akif Ekrekli  Engineering and
Natural Sciences
Tampere University
Tampere, Finland
<EMAIL_ADDRESS>Alexandre Angleraud  Engineering and
Natural Sciences
Tampere University
Tampere, Finland
<EMAIL_ADDRESS>Gaurang Sharma  Engineering and
Natural Sciences
Tampere University
Tampere, Finland
<EMAIL_ADDRESS>Roel Pieters  Engineering and
Natural Sciences
Tampere University
Tampere, Finland
<EMAIL_ADDRESS>
###### Abstract

Collaboration between human and robot requires effective modes of
communication to assign robot tasks and coordinate activities. As
communication can utilize different modalities, a multi-modal approach can be
more expressive than single modal models alone. In this work we propose a co-
speech gesture model that can assign robot tasks for human-robot
collaboration. Human gestures and speech, detected by computer vision and
speech recognition, can thus refer to objects in the scene and apply robot
actions to them. We present an experimental evaluation of the multi-modal co-
speech model with a real-world industrial use case. Results demonstrate that
multi-modal communication is easy to achieve and can provide benefits for
collaboration with respect to single modal tools.

###### Index Terms:

Human-robot collaboration, multi-modal perception, speech recognition, gesture
detection, object detection

##  I Introduction

Fluent interaction between human and robot requires reliable perception to
capture the commands of a person. While recent approaches in deep learning [1]
have established impressive tools to detect e.g., human pose, gestures and
speech, single tools alone can not always convey easily the commands intended
[2]. Reasons for this are the limited expressions available for different
modes of communication and the limitations in perception performance. Human
hand gestures, for example, contain much less information content than speech.
On the other hand, gesture detection can be done much quicker than speech
recognition, leading to a faster response time. These conflicting properties
motivate to combine multiple perception tools into a single multi-modal
detection model that utilizes communication from human to robot for assigning
tasks and coordinating the collaboration. In this work we compare different
perception tools and analyse them with respect to their suitability for human-
robot collaboration. A co-speech gesture model is then developed that combines
speech, human hand gestures and object detection to achieve effective
communication of desired robot tasks, such as picking human-specified objects
and robot to human hand-overs (see Fig. 1). The developments are intended for
industrial human-robot collaboration where a collaborative robot shares its
tasks, and works in close collaboration with, a human operator. Our
contributions are:

  * •

Human speech and hand gesture perception methods to command robot actions

  * •

Co-speech gesture model that combines human natural speech and hand gestures
to command robot actions

  * •

Experimental evaluation of the co-speech gesture model in an industrial human-
robot collaborative use case

Figure 1: Co-speech gesture model that combines a speech phrase, human gesture
detection and object perception to command robot actions.

##  II Related work

###  II-A Human-Robot Collaboration

Collaboration between human and robot is often targeted for industrial
manufacturing [3], as both robot and human have unique skills that complement
each other. Different interfaces that enable the collaboration have been
analyzed, providing clear directions on how the collaboration benefits the
tasks [4]. Approaches include voice processing, gesture recognition, haptic
interaction, and even brainwave perception. Often machine [5] and deep [6]
learning are used as enabling perception tool [1] to classify and recognize
the person and objects in the environment [7].

Figure 2: Co-speech gesture model that takes input from speech commands,
gesture recognition and object detection to generate robot actions for human-
robot collaboration. Sensor fusion enables the human to refer to specific
objects (<rod, rocker, arm, this, that>) and apply actions to them (<pick,
place, give me>).

###  II-B Human Perception

Visual detection of a person in the scene has been an active area of research
[8]. Different visual modalities have been utilized [9], such as RGB and depth
information [10]. Multi-modal approaches that utilize RGB-D data are popular
as well [11]. Human pose estimation goes a step further than human detection
by estimating the 3D pose of a human and their individual skeleton joints
[12], which can be used as input for gesture detection. Utilizing speech for
commanding robots has been demonstrated with short verbal commands for task
coordination [13] and task programming [14]. As extension to short speech
commands, natural language as instructions to robots has been used for
planning [15] and allocation [16] of tasks to be performed by the robot.

Multi-modal human-robot collaboration using gestures and speech simultaneously
has been demonstrated for a human interacting with the humanoid robot NAO in
[17], where short phrases and gestures are utilized to indicate human actions.
Collaboration between a robot arm and a human worker is also demonstrated in
[18], where a set of gestures and speech commands are perceived individually
to produce the same input for robot actions. As comparison, our work considers
an industrial scenario with a collaborative robot where speech phrases and
gestures are combined to assign tasks to the robot.

##  III Methods and Tools

###  III-A Perception Tools

The perception tools utilized in this work are integrated in a common
framework for isolated and human-robot collaborative tasks. For human
perception, Lightweight OpenPose, a human skeleton detection tool [12] is
used, which takes images (RGB) as input and returns skeleton node points as
output. For interaction, the wrist node of the skeleton is taken and, when
presented in a certain image area, serves as trigger for robot actions (e.g.,
stop, continue) or refers to certain objects in the scene (i.e., detected
objects pointed to). In the latter case, the detected object that is closest
to the wrist node is selected for robot action execution. Speech recognition
is enabled by Vosk [19] for the detection of pre-defined input commands and
phrases. This set of words and sentences relate to available actions of the
robot and locations in the scene, as described in Table I. The model is
configured by filtering out unnecessary words that are unsuitable for robot
instructions. Objects in the scene are detected by a neural network
(Detectron2 [20]) trained on a custom dataset collected for the use case [21].

###  III-B Multi-modal Perception Methods

The perception tools can be used in different ways to allow for sensor
redundancy, sensor multi-modality and sensor information fusion, as follows.

  * •

Sensor redundancy \- multiple sensors are used to command the same robot
actions, e.g., speech or hand gesture to stop robot motion

  * •

Sensor multi-modality \- different sensor modalities are used to command
individual robot actions, e.g., speech provides the robot actions, vision
detects human gestures

  * •

Sensor-fusion \- different sensor modalities are combined to command a single
robot action, e.g., speech provides robot action, vision provides specific
object location as pointed to by the human

While sensor redundancy and multi-modality is supported and demonstrated in
Section IV, we emphasize our contributions to the fusion of multiple sensor
outputs into a single robot command, as explained in the following section.

TABLE I: Perception methods’ input and output Method |  Input |  Output
---|---|---
Wrist detection |  RGB image of the scene (human front-facing) Human gesture by moving wrist to certain image location |  Robot stop/continue actions
Speech recognition |  Robot action commands: <pick, place, give, go, stop, pause, continue>
Workspace commands: <rod, home, arm, me>
Human speech requests: <place rod>, <go home>, <give me another rocker arm>, <pick up the last rod> |  Robot motion Gripper actions Robot to human hand-over Robot stop/continue actions
Object detection |  RGB image of the scene (top-down) |  Detected objects in the scene Valid target location for robot
Co-speech gesture |  <pick rod> \+ pointing gesture + object detection
<give me this rod> \+ pointing gesture + object detection
<give me that rocker arm> \+ pointing gesture + object detection |  Robot motion Gripper actions Robot to human hand-over

###  III-C Co-speech Gesture Model

The single-modal visual and speech perception models are fused into a multi-
modal perception model by combining speech commands, pointing gestures and
object detection (see Fig. 2). Several examples of these co-speech gestures
are described in Table I. The human can refer to individual objects in the
scene by speech (e.g., <rod>, <rocker arm>) and pointing to them, and apply
specific robot actions by speech commands (e.g., picking with <pick>, placing
with <place>, robot to human hand-over with <give>).

Depending on the object, different robot actions are possible, as specified
beforehand. For example, objects can be picked up from the table, placed in
specified locations and handed over to the person. Object detection returns a
list of objects in the scene, which can be verbally referred to by their
class. Pointing gesture detection allows to refer to specific objects in the
scene by relating the pointing gesture location to detected object locations.
Robot actions are therefore commanded by specific action verbs and object
classes, complimented by gestures to provide fine-grained object references
(see Fig. 2).

##  IV Experimental Results

###  IV-A Industrial Use Case

The considered use case replicates an industrial assembly task that in current
situation is done manually by human operators. The solution we propose
introduces a collaborative robot as assistive tool to the assembly station,
under control of the person. This means that the assembly work is coordinated
by the human, with the robot assisting in tasks that the human decides.
Available robot actions are to move to certain locations in the work space,
pick objects that are detected on the table, place objects to specified
locations or hand them over to the human. In addition, coordinated actions
include the stopping and continuing of robot actions during execution, for
human visual inspection of the objects placed by the robot. Human commands can
be communicated by hand gestures and/or speech, with different levels of
functionality as described in Table I. The setup for experiments is depicted
in Fig. 3 and includes two cameras (Intel Realsense D435) for visual
perception (one front-facing for wrist detection; Fig. 3(b) and one top-down
for object detection; Fig. 3(c)) and a microphone for speech recognition.
Computation is performed on a standard Desktop PC running Ubuntu Linux with
Nvidia GTX 1080 Ti GPU, and all robot (Franka Emika) communication and control
utilizes ROS. All tools are open-source available to utilize or replicate:
https://github.com/opendr-eu/opendr.

###  IV-B Human Gesture Detection

Results for the visual wrist detection tool are depicted in Fig. 3(b), which
highlights both detected human wrists. When one of the wrists is detected
inside one of the squares, this is taken as trigger for referring to certain
robot actions or objects in the scene. For example, to stop robot motion, the
left wrist should be detected in the top left square and to continue robot
motion, the right wrist should be detected in the top right square. Pointing
gestures are interpreted in a similar manner. When the human points to a
certain object, first the left or right wrist needs to be detected in either
of the lower two squares in the image, after which the location to the closest
detected object is determined. Performance of the skeleton detection tool has
been reported in the original paper [12]. In our use case the detection
accuracy of the wrists inside a square is consistent around 90%, as assessed
from 20-second interval tests for different squares. This is satisfactory for
effective collaboration.

###  IV-C Object Detection

Results of visual object detection are depicted in Fig. 3(c), which has the
different detected objects annotated by colored bounding boxes (yellow for the
rocker arms and blue for the rods). As objects are detected in image space,
careful calibration of both cameras ensures the detected objects can be picked
from the table and that pointing gestures can refer to the same object in both
camera frames. In our use case the detection accuracy of all classes is over
90%.

(a) (b) (c)

Figure 3: Experimental setup with a human pointing at an object for robot
picking (a). One camera is human front-facing to capture human hand gestures
(b), one camera is mounted on the robot (eye-in-hand) for object detection on
the table (c).

###  IV-D Speech Recognition

Results of speech recognition were found satisfactory, as in most cases the
spoken commands are recognized correctly. Performance, as reported in [19],
depends on the language skills of the person giving commands, as in certain
cases non-native English speakers had to speak more clear to achieve correct
speech recognition. Besides the speech recognition itself, the speech tool was
improved by including a voice activity detector and a time-delay filter (0.5
seconds) to consider the natural pause in human speech. This resulted in a
delay of $\approx 1.9$ seconds between a verbal command and the recognized
speech (average of 50 trials with different commands).

###  IV-E Co-speech Gesture Model Performance

The co-speech gesture model has all three perception models running in
parallel, decreasing slightly the running performance of the skeleton
detection tool (i.e., 24 fps with image size of 1920$\times 1080$). Object
detection achieves a frame rate of 4.5 fps with image size of
1280$\times$720\. Extended experiments were performed to test the co-speech
tool in a collaborative assembly scenario. This included a human and robot
performing assembly steps to an engine, with parts that are either mounted by
the person or by the robot. Parts assembled by the person are picked by the
robot from the table and handed over to the human, and parts assembled by the
robot are picked by the robot from the table and directly mounted to the
engine. Coordination of the tasks and requesting robot actions is done by the
person via the co-speech gesture model. In addition, the human can halt and
continue robot tasks at any time, by both gesture and speech commands (<stop>,
<pause>, <continue>).

Single commands - Fig. 6(a-b) depict the human commanding a stop and continue
gesture, respectively. Fig. 6(c) shows the human commanding the robot to move
to its ’home’ configuration by the phrase <ok, go home>. For this, the home
location is preprogrammed in the software scripts.

Speech phrases - Fig. 6 depicts how human speech alone can be utilized to
command robot actions, by the phrase <give me another rocker arm>. From the
recognized speech, the tool extracts relevant words and connects these to
robot actions and objects in the scene. In this case <give me> refers to a
robot to human hand-over, <rocker arm> refers to the rocker arm class in the
object detection model, and <another> implies any of the detected rocker arms,
meaning the first in the returned detection list. As a result, the command
phrase initiates all required robot actions and starts executing them one-by-
one, as shown in Fig. 6(ac).

Co-speech commands - Fig. 6 depicts examples of the co-speech gesture model
that utilizes a human speech phrase and pointing gesture to achieve robot
actions applied to specified objects in the scene. In this case, as a pointing
gesture is detected by the wrist detection tool, the closest specified object
to the human wrist is selected for the robot actions. A video of the co-speech
gesture model demonstrates all commands from Fig. 6-6:
https://youtu.be/b_ISrhOlcC8. This shows the collaborative tasks, where the
human coordinates the actions of the robot with four pick and place actions
and four robot to human hand-overs. Human inspection is done after object
placement by stopping robot motion with a speech command. In total, the
experiment includes over 20 speech commands and seven co-speech gestures to
coordinate the shared task.

(a) (b) (c)

Figure 4: Single command gestures stop (a), continue (b) and speech (c).

(d) (e) (f)

Figure 5: Speech phrase to achieve robot to human hand-over.

(a) (b) (c)

Figure 6: Co-speech gestures to achieve specified robot actions to objects.

##  V Discussion and Limitations

Sensor redundancy enables different modalities to command the same robot
action. This was demonstrated for stopping and continuing robot motion and
actions by hand gestures (see Fig. 6) and by speech commands. While hand
gestures can be detected at relatively high rate (>24 FPS), it can takes
several image frames before a correct prediction occurs. On the other hand,
speech commands can have considerable delay even when a first verbal command
is correctly recognized.

While in most cases the co-speech gesture model achieves the intended robot
commands and collaboration, some limitations are identified. First, detection
of the human wrist in a specific image location requires careful human hand
motion. As alternative, human hand gestures could be recognized directly from
a dedicated model [22]. In our case, inference time and detection accuracy
were the main reasons for utilizing a skeleton detection model instead.
Second, the relation between human pointing and objects in the scene needs
precise camera calibration, such that the same object is referred to in both
images. This can be circumvented by using a single camera for both visual
perception tools, with RGB and depth perception functionalities.

##  VI Conclusions

This work investigated how multiple perception tools can be utilized and
combined for effective human-robot collaboration. Human hand gestures and
speech, as well as object detection, provide the input for robot actions, as
coordinated by a person. Single modal perception serves to command basic robot
actions (stop, continue) by gesture or speech. A co-speech gesture model is
developed that combines human speech phrases, pointing gestures and object
detection to command robot actions (pick and place, robot to human hand-overs)
to specified objects in the scene. Experimental results demonstrate that co-
speech gestures can be easily utilized for coordinating a shared task between
human and robot.

## References

  * [1] N. Robinson _et al._ , “Robotic vision for human-robot interaction and collaboration: A survey and systematic review,” _ACM Trans. Hum. -Robot Interact._ , vol. 12, no. 1, pp. 1–66, 2023.
  * [2] S. Gross and B. Krenn, “A communicative perspective on human–robot collaboration in industry: Mapping communicative modes on collaborative scenarios,” _Int. J. of Social Robotics_ , pp. 1–18, 2023.
  * [3] V. Villani, F. Pini, F. Leali, and C. Secchi, “Survey on human–robot collaboration in industrial settings: Safety, intuitive interfaces and applications,” _Mechatronics_ , vol. 55, pp. 248–266, 11 2018.
  * [4] L. Wang _et al._ , “Symbiotic human-robot collaborative assembly,” _CIRP Annals_ , vol. 68, pp. 701–726, 2019.
  * [5] F. Semeraro, A. Griffiths, and A. Cangelosi, “Human–robot collaboration and machine learning: A systematic review of recent research,” _Robot. Comput. Integr. Manuf._ , vol. 79, p. 102432, 2023.
  * [6] N. Sünderhauf _et al._ , “The limits and potentials of deep learning for robotics,” _Int. J. Rob. Res._ , vol. 37, pp. 405–420, 4 2018.
  * [7] J. Fan, P. Zheng, and S. Li, “Vision-based holistic scene understanding towards proactive human–robot collaboration,” _Robot. Comput. Integr. Manuf._ , vol. 75, p. 102304, 6 2022.
  * [8] A. Zacharaki, I. Kostavelis, A. Gasteratos, and I. Dokas, “Safety bounds in human robot interaction: A survey,” _Safety Science_ , vol. 127, p. 104667, 7 2020.
  * [9] T. Linder, N. Vaskevicius, R. Schirmer, and K. O. Arras, “Cross-modal analysis of human detection for robotics: An industrial case study,” in _IEEE Int. Conf. Intell. Robots Syst._ , 9 2021, pp. 971–978.
  * [10] E. Magrini _et al._ , “Human-robot coexistence and interaction in open industrial cells,” _Rob. Comp. Integr. Manuf._ , vol. 61, p. 101846, 2 2020.
  * [11] C. R. Qi _et al._ , “Frustum pointnets for 3D object detection from RGB-D data,” in _IEEE Conf. Comput. Vis. Pattern Recognit._ , 2018.
  * [12] D. Osokin, “Real-time 2D multi-person pose estimation on CPU: Lightweight OpenPose,” _arXiv preprint arXiv:1811.12004_ , 2018.
  * [13] A. Angleraud _et al._ , “Coordinating shared tasks in human-robot collaboration by commands,” _Front. Robot. AI._ , vol. 8, 10 2021.
  * [14] T. B. Ionescu and S. Schlund, “Programming cobots by voice: A human-centered, web-based approach,” _Proc. CIRP_ , vol. 97, pp. 123–129, 2021.
  * [15] A. Boteanu _et al._ , “A model for verifiable grounding and execution of complex natural language instructions,” in _IEEE Int. Conf. Intell. Robots Syst._ , 2016, pp. 2649–2654.
  * [16] J. K. Behrens _et al._ , “Specifying dual-arm robot planning problems through natural language and demonstration,” _IEEE Robotics and Automation Letters_ , vol. 4, no. 3, pp. 2622–2629, 2019.
  * [17] P. Bremner and U. Leonards, “Efficiency of speech and iconic gesture integration for robotic and human communicators-a direct comparison,” in _IEEE Int. Conf. Robot. Autom._ , 2015, pp. 1999–2006.
  * [18] H. Chen, M. C. Leu, and Z. Yin, “Real-time multi-modal human–robot collaboration using gestures and speech,” _J. Manuf. Sci. Eng._ , vol. 144, no. 10, p. 101007, 2022.
  * [19] Alpha Cephei, “Vosk Speech Recognition Toolkit,” https://github.com/alphacep/vosk-api, 2023.
  * [20] Y. Wu, A. Kirillov, F. Massa, W.-Y. Lo, and R. Girshick, “Detectron2,” https://github.com/facebookresearch/detectron2, 2019.
  * [21] G. Sharma, R. Pieters, and A. Angleraud, “Engine assembly dataset,” http://dx.doi.org/10.5281/zenodo.7669593, Feb. 2023.
  * [22] O. Mazhar _et al._ , “A real-time human-robot interaction framework with robust background invariant hand gesture detection,” _Robot. Comput. Integr. Manuf._ , vol. 60, pp. 34–48, 12 2019.