# Are the dynamics of wall turbulence in minimal channels and larger domain
channels equivalent? A graph-theoretic approach

Ahmed Elnahhas1 Emma Lenz2 Parviz Moin1 Adrián Lozano-Durán3 and H. Jane Bae2
1 Center for Turbulence Research, Stanford University, Stanford CA, 94305 USA
2 Graduate Aerospace Laboratories, California Institute of Technology,
Pasadena CA, 91125 USA  3 Department of Aeronautics & Astronautics,
Massachusetts Institute of Technology, Cambridge MA, 02139 USA
<EMAIL_ADDRESS>
###### Abstract

This work proposes two algorithmic approaches to extract critical dynamical
mechanisms in wall-bounded turbulence with minimum human bias. In both
approaches, multiple types of coherent structures are spatiotemporally
tracked, resulting in a complex multilayer network. Network motif analysis,
i.e., extracting dominant non-random elemental patterns within these networks,
is used to identify the most dominant dynamical mechanisms. Both approaches,
combined with network motif analysis, are used to answer whether the main
dynamical mechanisms of a minimal flow unit (MFU) and a larger unconstrained
channel flow, labeled a full channel (FC), at $Re_{\tau}\approx 180$, are
equivalent. The first approach tracks traditional coherent structures defined
as low- and high-speed streaks, ejections, and sweeps. It is found that the
roll-streak pairing, consistent with the current understanding of self-
sustaining processes, is the most significant and simplest dynamical mechanism
in both flows. However, the MFU has a timescale for this mechanism that is
approximately $2.83$ times slower than that of the FC. In the second approach,
we use semi-Lagrangian wavepackets and define coherent structures from their
energetic streak, roll, and small-scale phase space. This method also shows
similar motifs for both the MFU and FC. It indicates that, on average, the
most dominant phase-space motifs are similar between the two flows, with the
significant events taking place approximately $2.21$ times slower in the MFU
than in the FC. This value is more consistent with the implied timescale ratio
of only the slow speed streaks taking part in the roll-streak pairing
extracted using the first multi-type spatiotemporal approach, which is
approximately $2.17$ slower in the MFU than the FC.

##  1 Introduction

It has long been hypothesized that wall-bounded turbulence can be decomposed
into a set of coherent structures that coexist and interact [1]. This concept
can be interpreted kinematically by synthesizing the turbulent flow field as a
superposition of building block eddies; the attached eddy model (AEM) adopts
this approach for high-$Re_{\tau}$ flows using a hierarchy of eddies scaling
with distance from the wall [2, 3]. A more comprehensive model would account
for the dynamics of the building block eddies, both in isolation as well as
for their interactions. The most famous dynamic coherent structure model of
wall-bounded turbulence is the roll-streak self-sustaining process (SSP) [4,
5, 6, 7, 8, 9, 10, 11, 12]. This SSP is primarily extracted from simulations
of minimal flow units (MFU) at low $Re_{\tau}$, which isolate a single scale
of wall-bounded turbulence, as well as a single instance of the SSP at a time
by enforcing severe periodicity [4, 5, 6, 12]. These MFU simulations have also
been extended to larger scales by isolating structures in the logarithmic
region of the flow where similar dynamical cycles were observed [7, 8, 9, 10,
11]. Taking the AEM, one can theorize that a “dynamic”-AEM can be constructed
based on the findings from MFUs of the buffer and logarithmic regions of the
flow by superposing SSPs at each scale of motion and neglecting strong
interscale interactions. However, recent experimental observations of
inner/outer modulation and large-scale streak aggregation suggest more to the
story [13, 14, 15, 16]. Hence, a few questions immediately arise concerning
the current understanding of wall-bounded turbulence: Are the dynamics
extracted from MFUs relevant to wall-bounded turbulent flows in large
unbounded domains at the same $Re_{\tau}$? Or do interactions between
different instances of SSPs heavily influence each other, perhaps altering
their timescales? Are other, possibly dominant, dynamic cycles suppressed in
the artificially restricted domains of MFUs? Finally, what is the importance
of interscale interactions of different-sized SSPs at larger $Re_{\tau}$? To
understand the dynamical interactions amongst the same scales of motion and
inter-scale, we need to find a systematic and algorithmic way to extract and
study these dynamics. In this study, we propose such a framework and apply it
first to study whether the dynamics of the MFU exist unaltered in large domain
channel simulations, henceforth referred to as full channels (FC), at
$Re_{\tau}=180$, or not.

Recently, coherent structure identification and temporal tracking algorithms
have been developed to understand the basic dynamical properties of select
eddy definitions, such as momentum-carrying structures, velocity streaks, and
energy-containing eddies [17, 18, 19, 20]. These algorithms are based on
spatial and temporal clustering of grid points that satisfy certain
thresholding and proximity criteria. The result is a directed graph where the
nodes represent the coherent structures, and the edges represent temporal
connections between the structures. Therefore, advanced graph processing tools
can extract information about these coherent structures’ spatiotemporal
properties. Extending the clustering and tracking algorithms to track several
definitions of eddies simultaneously, along with the interconnections between
eddy types, results in a higher dimensional graph, where the eddy type acts as
the new dimension, called a multilayered network [21, 22]. We use network
motifs and graphlet search algorithms [23, 24] to analyze and compare
dynamically significant patterns extracted from the buffer layer MFU and large
domain channel flows.

A network motif of a complex network is a pattern occurring at a significantly
higher frequency than in a random network of the same size [23]. Network
motifs have been identified in various biological networks, such as those
responsible for gene expression and neuronal cell interactions [25]. They have
also been identified in larger-scale ecological and engineering networks [23].
Since our directed multilayer network has time embedded in its construction,
detecting motifs accounts for the most significant dynamical patterns between
eddies of various types directly. Figure 1(a) illustrates the proposed
approach. Each node in the network represents the full spatiotemporal history
of a coherent structure, and the edges represent the interactions between two
coherent structures at some point during their evolution. In the context of
the current question, finding the most dominant interaction pattern compared
to a set of random networks helps us isolate the key dynamical mechanism in
both the FC and the MFU and compare their similarity. Applications of network
analysis to fluid flow are only recent phenomena [18, 26, 27, 28, 29, 30, 31,
32, 33, 34], and applications to the study of the dynamics of coherent
structures have not been attempted in fully developed wall-bounded turbulent
flows. The results of this spatiotemporal approach are supplemented with
motifs extracted from an energetic phase-space constructed using semi-
Lagrangian wavepackets illustrated in figure 1(b).

(a)

(b)

Figure 1: (a) The proposed multi-type spatiotemporal tracking and motif
detection methodology. Different types of eddies are clustered simultaneously,
and the resulting multilayer network is searched for patterns called network
motifs. The lighter structures represent earlier times in the evolution of a
spatiotemporal eddy. Green lines represent the intertype connections
transferred from single-time connections to branches to trees. Orange cross-
lines indicate intra-type branch links that are not considered simultaneously
“primary” for tree clustering. (b) The proposed semi-Lagrangian wavepacket
approach where the greyed areas are MFU-sized boxes, which are allowed to
advect and collect quantities for the phase-space-temporal dynamical analysis.
Black lines are the spanwise centroids of low-speed streaks projected onto the
wall. The MFU is shown for comparison.

The rest of the paper is organized as follows. Section 2 describes the
datasets utilized to perform the analysis. In section 3, the multi-type
coherent structure tracking and motif detection algorithms are described, and
the results of their application to the MFU and the FC at $Re_{\tau}=180$ are
discussed. In section 4, the semi-Lagrangian wavepackets and phase-space
temporal motif analysis approach is explained, and the results of applying it
to the same datasets are discussed. Finally, conclusions and future work are
presented in section 5.

##  2 Simulation details

Let $u_{\tau}$ and $\nu$ denote the flow’s friction velocity and kinematic
viscosity, respectively. The two flows analyzed are direct numerical
simulations (DNS) of an MFU and an FC inside a channel of half-height
$\delta$. The friction Reynolds number for both flows is
$Re_{\tau}\equiv\delta u_{\tau}/\nu\approx 180$. In this flow, $x$, $y$, and
$z$ are the streamwise, wall-normal, and spanwise directions, respectively,
corresponding to their respective mean and fluctuating velocities $U$ and $u$,
$V$ and $v$, and $W$ and $w$. The domain of the MFU has a size of
$\\{0.564\pi\times 2\times 0.282\pi\\}\delta$, and that of the FC has a size
of $\\{8\pi\times 2\times 3\pi\\}\delta$. The resolution in inner units of
both simulations are $\\{\Delta x^{+},\Delta y^{+}_{min},\Delta
y^{+}_{max},\Delta z^{+}\\}=\\{6.05,0.17,7.52,6.05\\}$, which is consistent
with prior DNS simulations. Flow fields are stored every $\Delta t^{+}\approx
1$ for $T^{+}=2000$ and $T^{+}=25000$ inner time units for the FC and MFU,
respectively. These correspond to approximately $11\delta/u_{\tau}$ and
$135\delta/u_{\tau}$, respectively.

##  3 Multi-type spatiotemporal coherent-structure tracking and motif
detection

###  3.1 Multilayer network generation

The multi-type coherent-structure tracking algorithm is a three-step process,
schematically illustrated in figure 1(a):

Step 1: In this work, the eddies considered are the low- and high-speed
streaks and wall-normal streamwise momentum flux eddies, namely ejections and
sweeps, interchangeably referred to as Q2 and Q4, respectively [17, 20]. At
each time instant, each of these eddies is defined as the set of connected
points in space, $\Omega$, satisfying the following inequalities

$\big{\\{}(x,y,z):|uv|>\alpha_{uv}u^{\prime}(y)v^{\prime}(y)\quad\&\quad\begin{cases}\bigg{(}\frac{\int_{\Omega}u~{}dV_{\Omega}}{\int_{\Omega}dV_{\Omega}}<0\bigg{)}~{}\&~{}\bigg{(}\frac{\int_{\Omega}v~{}dV_{\Omega}}{\int_{\Omega}dV_{\Omega}}>0\bigg{)}\bigg{\\}}\implies\mathrm{Ejections~{}(Q2)}\\\\[10.0pt]
\bigg{(}\frac{\int_{\Omega}u~{}dV_{\Omega}}{\int_{\Omega}dV_{\Omega}}>0\bigg{)}~{}\&~{}\bigg{(}\frac{\int_{\Omega}v~{}dV_{\Omega}}{\int_{\Omega}dV_{\Omega}}<0\bigg{)}\bigg{\\}}\implies\mathrm{Sweeps~{}(Q4)}\end{cases}\mathrm{,}$
(1)
$\big{\\{}(x,y,z):\sqrt{u^{2}+w^{2}}>\alpha_{str}u^{\prime}(y)\quad\&\quad\begin{cases}u<0\big{\\}}\implies\mathrm{Slow~{}Streak}\\\
u>0\big{\\}}\implies\mathrm{Fast~{}Streak}\end{cases}{,}$ (2)

where $u^{\prime}(y)$ and $v^{\prime}(y)$ are the root-mean-squared streamwise
and wall-normal velocity fluctuations extracted from the corresponding
simulation, i.e., the MFU and the FC, $\alpha_{\\{uv,str\\}}$ are scaling
coefficients for the thresholds, and $V_{\Omega}$ is the physical volume
associated with the set of points, $\Omega$. The clustering process is known
to have a percolation crisis, and we use the values
$\\{\alpha_{\tau},\alpha_{str}\\}=\\{1.30,1.75\\}$, which maximize the number
of clusters identified per timestep on average. Each region, $\Omega$, is
labeled as a coherent structure at some instant in its evolution. Once
subdivided into these four sets, contiguous time snapshots construct their
temporal evolution.

(a)

(b)

Figure 2: (a) Histogram indicating the distribution of how many individual
attached eddy branches are grouped into trees using the simultaneous primary
branch interaction metric in Equation (3). (b) A multilayer network with each
node representing the full spatiotemporal evolution of an attached eddy tree
placed at its temporal centroid. The links between the nodes indicate an
interaction or overlap at some point during the lifetime of the two linked
spatiotemporal eddies. Only a subset of the complete network in the FC is
shown.

Step 2: The details of the structure tracking in time are described in [18].
In summary, object volumetric overlap between timesteps is used to construct a
graph with edges representing connections in time and objects at different
time steps representing nodes. While previously this was done for only a
single type of object, this procedure is repeated $N^{2}$ times where $N$ is
the number of eddy types. The graph is first clustered into branches, which
are the links between objects that mutually experience the least relative
change from one timestep to the next.

Step 3: The branches are then further partitioned to generate subgraphs or
trees, which are collections of branches that interact in a “primary” way. We
define a primary branch connection using the relative overlap metric from
branch $i$ to $j$ of the same eddy type and vice versa, which is defined as
follows

$O_{i\rightarrow j}=\frac{\int V_{i}\cap V_{j}dt_{i}}{\int
V_{i}dt_{i}}\quad\text{and}\quad O_{j\rightarrow i}=\frac{\int V_{j}\cap
V_{i}dt_{j}}{\int V_{j}dt_{j}}\text{,}$ (3)

where the integration is over the lifetime of the source branch and $V_{i}$ is
the volume of a branch at a given time across its lifetime. The intersecting
volume is computed at the same absolute time for both branches. The branch
link $j$ to branch $i$ with the highest overlap metric is considered the
primary branch in the positive or negative time directions. If two branches
are each other’s primary links, they are clustered into a subgraph or tree.
This process is applied recursively, and each resulting tree is subsequently
referred to as a spatiotemporal eddy. An illustration of the partitioning when
two branches do not point to each other as primary branch connections is
represented with the crossed orange lines in figure 1(a).

During step 2 and step 3 of the process, the physical connections between
eddies of different types are transferred from single-time connections at the
end of step 1 to connections between branches and, finally, trees at the end
of steps 2 and 3, respectively. This results in a large, complex, multilayered
network that encodes spatiotemporal interactions between eddies of the same
type and those between eddies of different types. These intertype connections
are represented with green lines in the schematic illustration of figure 1(a).

After the three-step process, many of the spatiotemporal eddies are small
structures existing in the center of the channel, which, if tracked, result in
an overly large/complex multilayer network. To ease the subsequent analysis,
we limit ourselves to interactions between attached eddies that live for a
minimal amount of time, i.e., those who, at some point during their lifetime,
had a minimal wall-normal height (on either wall) of $y^{+}_{min}<20$ [18,
20], and have lived for longer than $T^{+}_{life}=20$. Furthermore, they must
have interacted with another attached eddy during their lifetime. The volumes
occupied by the ejections, sweeps, slow, and fast streaks that satisfy these
criteria and subsets of them in both the MFU and the FC are presented in table
3.1. The applied criteria only restrict the minority of the attached eddies by
volume, approximately $11.5-17.5\%$, from participating in the subsequent
motif analysis. Figure 2(a) showcases the result of the step 3 partitioning of
branches into trees using the overlap metric for attached eddies, where the
size of the trees in terms of branches is exponentially decaying, and no
single tree contains more than $\mathcal{O}(10)$ branches. Given this
partitioning, figure 2(b) shows a subset of the multilayer network that
emerges due to all the interactions of attached spatiotemporal ejections,
sweeps, and slow and fast streaks in the FC where each dot contains within it
the entirety of the detailed spatiotemporal evolution of each of the
spatiotemporal eddies.

Table 1:  The average percentage of the domain’s volume occupied by certain
types of eddies that satisfy particular criteria in both the full channel and
minimal flow unit. $\mathcal{A}:{y_{min}^{+}<20}$.
$\mathcal{T}:{T_{life}^{+}>20}$. $\mathcal{I}:\text{Interaction with another
$\mathcal{A}-$\text{valid eddy}}$. $\mathcal{S}_{1}^{\\{F,M\\}}:\text{Involved
in most significant roll-streak motif}$. $\mathcal{F}_{2}^{M}:\text{Involved
in similar motif only presented in MFU}$. The two values for
$\mathcal{S}_{1}^{M}$ and $\mathcal{F}_{2}^{M}$ eddies are averages performed
over the entire temporal extent of the simulation and only over the times when
constituent eddies in the motifs are alive, respectively.

Minimal Flow Unit |  | Full Channel
---|---|---
Filter | Q2 & Q4 | Streaks |  | Filter | Q2 & Q4 | Streaks
$\mathcal{A}$ | 5.12% | 2.55% |  | $\mathcal{A}$ | 5.19% | 2.43%
$\mathcal{A}~{}\&~{}\mathcal{T}$ | 4.90% | 2.52% |  | $\mathcal{A}~{}\&~{}\mathcal{T}$ | 4.52% | 2.15%
$\mathcal{A}~{}\&~{}\mathcal{T}~{}\&~{}\mathcal{I}$ | 4.43% | 2.31% |  | $\mathcal{A}~{}\&~{}\mathcal{T}~{}\&~{}\mathcal{I}$ | 4.16% | 2.10%
$\mathcal{A}~{}\&~{}\mathcal{T}~{}\&~{}\mathcal{I}~{}\&~{}\mathcal{S}^{M}_{1}$ | 1.09-3.87% | 0.72-2.44% |  | $\mathcal{A}~{}\&~{}\mathcal{T}~{}\&~{}\mathcal{I}~{}\&~{}\mathcal{S}^{F}_{1}$ | 2.04% | 1.01%
$\mathcal{A}~{}\&~{}\mathcal{T}~{}\&~{}\mathcal{I}~{}\&~{}\mathcal{F}^{M}_{2}$ | 2.56-4.14% | 1.10-2.15% |  | — | — | —
|  |  |  |  |  |

###  3.2 Motif detection

The resulting multilayer network, at the tree level, is fed into an open-
source motif detection code, FAst Network MOtif Detection (FANMOD) [35], which
uses the exhaustive enumerate subgraph (ESU) algorithm to enumerate all
isomorphic subgraphs, $s$, of a certain size in our multilayer network [36].
The frequency of each isomorphic subgraph $s$, given by $m(s)$, is compared
with statistics from a set of randomly generated networks to determine over-
representation in the original network, i.e., significant motifs.

In this case, the randomly generated networks are constructed from our
original network by randomly switching edges between nodes in the network
while maintaining its size, $|G|$. The sizes of the original network for each
flow are $|G|_{FC}=31855$, and $|G|_{MFU}=1251$. Since their criteria are
mutually exclusive, ejections and sweeps cannot form connections, as is the
case between fast and slow streaks. This constraint is reflected in the random
networks generated by maintaining the local colored degree vector of each node
during the randomization [37]. This means that, for example, if a sweep were
originally connected to two fast streaks and one slow streak, it would remain
connected to two fast streaks and one slow streak after the randomization. We
can, therefore, interpret the randomization as a reordering in space and time
of the spatiotemporal eddies while maintaining their allowable interactions.

The frequency of each subgraph in the random graph is given by $m_{r}(s)$. The
mean and variance of $m_{r}(s)$ are computed from an ensemble of random graphs
and denoted $\overline{m_{r}(s)}$ and $Var(m_{r}(s))$, respectively. Enough
random graphs were generated so that the mean and the variance converged. This
was done with $100$ and $1000$ random graphs for the FC and MFU, respectively.
A significance score was determined for each isomorphic subgraph based on the
random mean and standard deviation. Furthermore, to account for shifts in the
scores associated with the differences in the sizes of the networks, a
normalization based on the law of large numbers utilizing $|G|$ is added. The
significance score is, therefore,

$\widetilde{W}(s)=\frac{m(s)-\overline{m_{r}(s)}}{\left(Var(m_{r}(s))|G|\right)^{1/2}}.$
(4)

###  3.3 Results

The results of the motif analysis indicate that there are no $3$-node motifs,
i.e., none where significantly different than random, and we focus our
discussions on $4$-node motifs. The schematics in table 2 illustrate the
results for the $4$-node motifs. The motifs are labeled as
$\\{\mathcal{S},\mathcal{F}\\}^{\\{F,M\\}}_{\\#}$ with $\mathcal{S}$ or
$\mathcal{F}$ denoting motifs ordered by significance score and frequency,
respectively, and the superscripts $F$ and $M$ denoting the flow, FC and MFU,
respectively. The left portion of the schematics in the table shows the top
six to seven motifs in descending order of significance score. The two flows
share the most significant motifs, where the top six in the MFU are found in
the top seven of the FC. However, besides the most dominant motif, the order
of the other five is shifted around in the MFU with respect to the FC, as
denoted by the two-headed black arrows. The most dominant $4$-node motif for
both flows is a pairing of a slow and a fast streak interacting with an
ejection and a sweep, consistent with the roll-streak SSP, labeled from now on
as $\mathcal{S}_{1}^{\\{F,M\\}}$. This likely indicates that the roll-streak
pairing is, in fact, the simplest form of spatiotemporal organization
exhibited in these single-scale wall-bounded flows. The right portion of the
schematics in the table shows the top six motifs in descending order of
frequency. The two flows share five of the top six between them. However,
their orders are shifted around in a similar manner to the top six most
dominant motifs. Furthermore, motif $\mathcal{F}_{2}^{M}$ strongly resembles
the structure of $\mathcal{S}_{1}^{\\{F,M\\}}$ except for a missing link
between slow streaks and sweeps. This is likely due to the confined domain
restricting the emergence of the structures simultaneously. Whether this
confinement leads to some ordering in the emergence and death of the
constituent eddies of this motif is examined further below.

Furthermore, table 2 reports the significance scores and frequency of
occurrence of each motif. Note that while the frequency of the most
significant motif in both flows seems to be low compared to all possible
$4$-node induced subgraphs in either $G_{FC}$ or $G_{MFU}$, that does not mean
that the constituent eddies are few or not substantial. In fact, table 3.1
shows that approximately $50\%$ of the attached eddies by volume in the FC
participate in this motif. For the MFU, the volume occupied by these eddies
averaged over the entire temporal extent of the simulation indicates that only
$25-30\%$ of them participate in the motif. However, there are times when no
eddies participating in this motif are present in the simulation domain. Thus,
if we condition the average on the times when the eddies participating in the
motif are present, i.e., alive, the average volume of ejections and sweeps is
around $87\%$ of their unconditioned average volume, and that of streaks is
even larger than the unconditioned average volume of attached streaks
throughout the simulation. A plausible explanation is as follows: In the MFU,
on average, only a single instance of an SSP can exist at a time, and once the
large-scale streaky structures break down in what is known as bursting [38],
the current tracking procedure sees no attached structures. While this also
happens in the case of the FC, since there is always at least one instance of
this motif active in the domain, conditionally averaging over the times when a
constituent eddy is alive is redundant. However, for the MFU, there are time
periods where we are averaging over zero volume due to the presence of only
small scales. Overall, this implies that in subsequent analyses, we should
include the simultaneous tracking of small scales in the motifs we search for
to find the full cycle of the SSP. Finally, it is interesting to note that all
the most frequent yet marginally significant motifs, except for
$\mathcal{F}_{2}^{M}$, in both the FC and MFU have an equal significance score
in each flow, respectively. This indicates that their frequencies are linked
during the construction of the random networks and that they are most likely
kinematically constrained to one another. We, therefore, focus our dynamical
analyses on the eddies participating in each of the motif instances of
$\mathcal{S}_{1}^{\\{F,M\\}}$, albeit including data from motif
$\mathcal{F}^{M}_{2}$ in the case of the MFU due to its similarity to
$\mathcal{S}_{1}^{M}$.

Table 2: The dynamical $4$-node motifs detected using the ESU algorithm implemented in FANMOD in both the FC and the MFU using the aforementioned random network model. The left portion shows motifs ordered in descending order of significance score, indicating the strongest non-random events in both domains. The right portion shows motifs ordered in descending order of frequency. The reported motifs exclude any with a significance score below $\widetilde{W}(s)<0.05$. The top and bottom rows of motif schematics represent those extracted from the FC and the MFU, respectively. Above and below the motif schematics, the significance score, $\widetilde{W}(s)$, and frequency percentage, $\mathcal{F}(s)$, i.e., the percentage of occurrences in the original networks with respect to the total number of all 4-node subgraphs present in the MFU and FC networks, which were 1811927 and 7057, respectively, are presented along with their labels. The bounding red box indicates that the most non-random dynamical pattern, $\mathcal{S}_{1}^{\\{F,M\\}}$, is the roll-streak pairing and is consistent amongst both flow domains. The bounding dashed purple box shows a similar motif to $\mathcal{S}_{1}^{\\{F,M\\}}$, $\mathcal{F}_{2}^{M}$, that only emerges in the MFU. $\newmoon$ $\rightarrow$ Ejections. $\newmoon$ $\rightarrow$ Sweeps. $\newmoon$ $\rightarrow$ Fast streaks. $\newmoon$ $\rightarrow$ Slow streaks. | Significance Ordering | Frequency Ordering
---|---|---
FC | $\mathcal{S}_{1}^{F}$ | $\mathcal{S}_{2}^{F}$ | $\mathcal{S}_{3}^{F}$ | $\mathcal{S}_{4}^{F}$ | $\mathcal{S}_{5}^{F}$ | $\mathcal{S}_{6}^{F}$ | $\mathcal{S}_{7}^{F}$ | $\mathcal{F}_{1}^{F}$ | $\mathcal{F}_{2}^{F}$ | $\mathcal{F}_{3}^{F}$ | $\mathcal{F}_{4}^{F}$ | $\mathcal{F}_{5}^{F}$ | $\mathcal{F}_{6}^{F}$
$\widetilde{W}(s)$ | 4.71 | 3.76 | 3.50 | 3.47 | 3.13 | 3.04 | 2.56 | 0.13 | 0.13 | 0.13 | 0.13 | 0.13 | 0.13
$\mathcal{F}(s)$ | 0.26 | 0.19 | 0.16 | 0.14 | 0.15 | 0.09 | 0.09 | 7.28 | 7.10 | 6.10 | 4.95 | 3.53 | 3.36
|
MFU | $\mathcal{S}_{1}^{M}$ | $\mathcal{S}_{2}^{M}$ | $\mathcal{S}_{3}^{M}$ | $\mathcal{S}_{4}^{M}$ | $\mathcal{S}_{5}^{M}$ | $\mathcal{S}_{6}^{M}$ | — | $\mathcal{F}_{1}^{M}$ | $\mathcal{F}_{2}^{M}$ | $\mathcal{F}_{3}^{M}$ | $\mathcal{F}_{4}^{M}$ | $\mathcal{F}_{5}^{M}$ | $\mathcal{F}_{6}^{M}$
$\widetilde{W}(s)$ | 1.54 | 1.30 | 0.83 | 0.79 | 0.73 | 0.38 | — | 0.37 | 0.05 | 0.37 | 0.37 | 0.37 | 0.37
$\mathcal{F}(s)$ | 0.72 | 0.47 | 0.34 | 0.35 | 0.14 | 0.10 | — | 15.20 | 8.86 | 8.67 | 3.83 | 3.49 | 3.25
|  |  |  |  |  |  |  |  |  |  |  |  |

Table 3.3 shows the average lifetimes of each motif,
$\overline{T^{+}}_{life}$, defined as the time from when the first
participating eddy emerges to when the final participating eddy dies, in both
the FC and MFU for motifs $\mathcal{S}_{1}^{\\{F,M\\}}$ and
$\mathcal{F}_{2}^{M}$. Simultaneously, it shows the average advection
velocity, $\overline{u^{+}}_{adv}$, of the participating eddies in the
streamwise direction. It is observed that $\overline{T^{+}}_{life}$ is
approximately $2.83$ times longer in the MFU compared to its FC counterpart.
The advection velocities of the constituent eddies of these motifs,
$\overline{u^{+}}_{adv}$, range from $1.3$ to $3.1$ times slower in the MFU
compared to its FC counterpart, with the primary reduction present in the
ejections and the slow streaks associated with them. This indicates that while
the main dynamical mechanism in both flows is the same, they differ in
timescales substantially. In fact, it will be shown that the reduction in the
speed of slow streaks in the MFU of approximately $2.17$ times compared to
those in the FC, is consistent with phase-space motifs extracted using the
semi-Lagrangian wavepacket basis analysis to come, which employs the presence
of slow streaks as a conditioning variable to determine the advection velocity
of the wavepackets. Figure 3 shows the probability density function (PDF) of
$T^{+}_{life}$ for the $\mathcal{S}_{1}^{F}$ motif, highlighting that the most
probable instances of this motif happen over a single eddy turnover timescale,
$\delta/u_{\tau}$, with a majority ($>95\%$) not lasting longer than two eddy
turnover times.

Table 3: The average lifetime in inner units, $\overline{T^{+}}_{life}$, of
the instances of the most significant motifs in each of the two flows,
$\mathcal{S}_{1}^{\\{F,M\\}}$, along with the average advection velocity in
inner units, $\overline{u^{+}}_{adv}$, of the constituent eddies taking part
in these significant motifs. In the case of the MFU, the eddies taking part in
$\mathcal{F}_{2}^{M}$ are also included in these averages due to the
similarity of the motif and the lack of samples.

$\overline{T^{+}}_{life}$
---
Motif Identifier | Minimal Flow Unit | Full Channel
$\mathcal{S}_{1}^{F,M}$ | 601 | 212
$\overline{u^{+}}_{adv}$
Eddy Type | Minimal Flow Unit | Full Channel
Ejections | 4.50 | 14.07
Sweeps | 11.01 | 14.30
Fast Streaks | 9.28 | 11.86
Slow Streaks | 6.00 | 13.00
|  |

Figure 3: PDF of the lifetime of the roll-streak motif instances,
$\mathcal{S}_{1}^{F}$, extracted from the FC. The two vertical dashed lines
are placed at $\delta/u_{\tau}$ and $2\delta/u_{\tau}$, respectively.

Furthermore, we note that the motifs presented in table 2, and in particular
$\mathcal{S}_{1}^{\\{F,M\\}}$, are not partitioned into their $2\times 4!$
possible emergence/death orderings. That is because no sufficiently prominent
ordering was found in the FC. The emergence ordering that occurred the most,
$9.91\%$ of the time, was fast streak $\rightarrow$ sweep $\rightarrow$ slow
streak $\rightarrow$ ejection. The death order that occurred the most,
$8.53\%$ of the time, was ejection $\rightarrow$ sweep $\rightarrow$ slow
streak $\rightarrow$ fast streak. Therefore, we comment on the ordering trends
for each individual eddy type, regardless of which permutation this ordering
belongs to. Table 4 reports these statistics. Overall, in the FC, ejections
are more likely to emerge last and die first, sweeps are more likely to emerge
at any order and die at any order, albeit with a slightly higher tendency to
die first similar to the ejections. On the other hand, fast streaks are more
likely to emerge first and die last, whereas slow streaks are slightly more
likely to emerge and die in the middle ordering of the motif. It is apparent
that the average emergence and death orderings for the fast streaks and the
ejections are inverted with respect to each other.

In the MFU, the emergence and death orderings of the ejections are inverted
with respect to those of the FC, where they now appear first and die last.
This indicates their longer timescales, consistent with their slower advection
velocities. Sweeps, however, tend to appear last and die first. Fast streaks
are approximately equally likely to appear first, second, or third and die
second or third. Finally, slow streaks appear last, with the sweeps, and die
either first or third. These differences between the two flows are probably
due to the confinement effects of periodic boundary conditions, which the MFU
is subject to.

Finally, streaks, being larger coherent structures than ejections and sweeps,
are associated with multiple ejections and sweeps events during their
lifetime. This was verified spatiotemporally by examining their type-dependent
degree distribution, which is not shown here for brevity. As such, it is
possible that a clearer picture of order would appear if, rather than
individual motif instances, clusters of individual motif instances are
examined [39].

Table 4: The percentage of time that ejections, sweeps, fast streaks, and slow streaks emerge and die in a particular order for motifs $\mathcal{S}_{1}^{F}$ for the FC, and $\mathcal{S}_{1}^{M}$ and $\mathcal{F}_{2}^{M}$ for the MFU. FC | Emergence(%) | Death(%)
---|---|---
Eddy Type | $1^{\mathrm{st}}$ | $2^{\mathrm{nd}}$ | $3^{\mathrm{rd}}$ | $4^{\mathrm{th}}$ | $1^{\mathrm{st}}$ | $2^{\mathrm{nd}}$ | $3^{\mathrm{rd}}$ | $4^{\mathrm{th}}$
Ejections | 14.2 | 20.7 | 24.5 | 40.6 | 35.1 | 27.2 | 21.8 | 15.9
Sweeps | 28.8 | 25.3 | 21.6 | 24.3 | 32.8 | 23.2 | 21.9 | 22.1
Fast Streaks | 37.6 | 26.1 | 22.1 | 14.2 | 11.4 | 20.1 | 28.8 | 39.7
Slow Streaks | 19.4 | 31.8 | 27.9 | 20.9 | 20.7 | 29.5 | 27.5 | 22.3
MFU | Emergence(%) | Death(%)
Eddy Type | $1^{\mathrm{st}}$ | $2^{\mathrm{nd}}$ | $3^{\mathrm{rd}}$ | $4^{\mathrm{th}}$ | $1^{\mathrm{st}}$ | $2^{\mathrm{nd}}$ | $3^{\mathrm{rd}}$ | $4^{\mathrm{th}}$
Ejections | 49.3 | 24.7 | 17.9 | 8.1 | 9.6 | 15.8 | 17.9 | 56.7
Sweeps | 13.0 | 18.9 | 29.2 | 38.9 | 42.9 | 27.5 | 19.4 | 10.2
Fast Streaks | 25.3 | 29.6 | 35.2 | 9.9 | 15.2 | 38.3 | 28.3 | 18.2
Slow Streaks | 12.4 | 26.8 | 17.8 | 43.0 | 32.3 | 18.3 | 34.5 | 14.9
|  |  |  |  |  |  |  |

##  4 Semi-Lagrangian wavepackets and phase-space temporal motifs

###  4.1 Defining the size of the semi-Lagrangian wavepackets

In addition to coherent structures, integral quantities of the flow also
provide insight into the flow dynamics. While we can collect statistics on
integral quantities directly in the MFU, we require an equally sized domain of
integration in the FC to make meaningful comparisons between the two flows. To
do so, we utilize semi-Lagrangian wavepackets in the FC to collect statistics
on these integral quantities. These wavepackets will be advected downstream
following an advection speed to track the relevant structures. To minimize the
effect of shearing, we choose the domain to have $y^{+}\leq 40$, as the
structures in this region are known to advect at a similar advection velocity
[18]. In the wall-parallel directions, the wavepackets, and hence the domain
of integration, are sized to match the extents of the MFU. Therefore, we
consider wavepackets of size
$(L_{x}^{+},L_{y}^{+},L_{z}^{+})=(320.5,40,160.2)$. As the FC domain is
considerably larger than the MFU, we can track many wavepackets simultaneously
in each flow field, as seen in figure 1(b).

###  4.2 Production–dissipation trajectories and advection speed

We first consider trajectories of production and dissipation,

$\llangle P\rrangle=-\frac{1}{L_{y}}\int_{0}^{L_{y}}\langle
uv\rangle\frac{\partial U}{\partial
y}\,\mathrm{d}y,\quad\llangle\varepsilon\rrangle=\frac{1}{L_{y}}\int_{0}^{L_{y}}2\nu\langle
S_{ij}S_{ij}\rangle\,\mathrm{d}y,$ (5)

in the minimal and full channels, where $S_{ij}$ is the rate-of-strain tensor,
and $\langle\cdot\rangle$ indicates averages in the streamwise and spanwise
directions in the domain of integration.

(a)

(b)

(c)

(d)

Figure 4: Production-dissipation of the (a) MFU and (b) FC. Contours are PDF
at 10%, 40%, and 70% of occurrences, and arrows show the mean trajectory
computed by taking averages over $50\times 50$ bins. Perpendicular lines show
quadrant divisions. (c) Relative transition probability between quadrants for
the (c) MFU and (d) FC. Horizontal entries are the initial quadrant, and
vertical entries are the next quadrant.

The production and dissipation are calculated at each timestep for the MFU and
the different instances of wavepackets in the FC. The joint PDF of those two
quantities is shown in figure 4(a,b) for the two cases. The PDF is compiled as
a histogram over $50\times 50$ bins, and the arrows represent the mean
evolution velocity of all the states within a particular bin. These results
are consistent with those observed in [38]. We also observe that the mean
evolution velocities of the MFU and FC are similar; however, the PDF of the FC
has a larger spread of production and dissipation values than the MFU,
indicating a much more intense production and dissipation events in the FC.

The production-dissipation space is then divided into quadrants following
[38], with the origin at the mean production and dissipation, and the axes
determined by the principal axes of the PDF. Although the FC has different
mean production-dissipation values and principal axes compared to the MFU, the
results were not sensitive to the choice of the mean values and principal
axes, so the quadrants based on the MFU were used for consistency.

The probability of moving from one quadrant to another, referred to as the
transition probability, was determined following the method of [27] and is
shown in figure 4(c,d). The transition probabilities only consider the
relative probability of moving to different quadrants and do not consider the
probability of staying in the same quadrant. Interestingly, figure 4(c,d) show
a high probability of moving from $I$ to $III$ and from $IV$ to $II$, whereas
(a,b) do not show arrows pointing in those directions. This is not necessarily
a contradiction because figure 4(c,d) consider only the probability of moving
from one quadrant to the other, while figure 4(a,b) also consider the speed of
evolution of the state. To quantify the error between transition probabilities
in the MFU and FC, the transition probabilities are assembled into a vector,
and the error is defined as the norm of the difference in the probability
vectors between the MFU and FC.

It was noted that the choice of advection speed has a significant effect on
the calculated quantities for the full channel. To determine the best
advection method, we compare the production–dissipation plot for the MFU and
FC using different advection methods and compare the transition probabilities.
The advection velocity that minimized error was the average velocity of only
the regions of the flow contained in low-speed streaks, defined as

$\\{(x,y,z):\sqrt{u^{2}+w^{2}}>\alpha u_{\tau}\quad\&\quad u<0\\}\mathrm{,}$
(6)

at $y^{+}=40$, with $\alpha=3.4$ [20]. Other definitions of advection
velocities were considered, including average velocity over the whole box,
average velocity at several $y$-locations, and average velocity conditioned to
negative $u$-fluctuations, but all had higher errors in the production-
dissipation cycle. Interestingly, [18] shows that these structures advect with
the mean velocity at $y^{+}=40$. However, in the case of production-
dissipation cycles, this velocity resulted in higher error than the velocity
conditioned to streaks at $y^{+}=40$. For the remainder of the section, we use
the definition given by equation (6). This leads to an average advection
velocity in the streamwise direction of $10.7u_{\tau}$ which is similar, but
not exactly equal to, the advection velocity of slow streaks extracted in the
FC using coherent structure tracking as presented in table 3.3. Note that the
advection velocity reported in table 3.3 is of the volumetric centroid of the
streak, i.e., its group velocity, rather than the average of the velocity
field occupied by the region of the slow streak only at $y^{+}=40$.

###  4.3 Temporal motif of energy contained in streak, roll, and small scales

Additionally, we track energy in eddies related to the self-sustaining
process. The magnitude of the streaks is defined by the magnitude of the
streamwise velocity captured in the
$(k_{x},k_{z})\equiv(k_{x}^{*}L_{x}/(2\pi),k_{z}^{*}L_{z}/(2\pi))=(0,1)$ mode
of the MFU domain,

$\llangle\hat{u}_{0,1}\rrangle=\frac{1}{L_{y}}\int_{0}^{L_{y}}(\hat{u}(k_{x}=0,k_{z}=1)^{2})^{1/2}\,\mathrm{d}y,$
(7)

where $k_{x}^{*}$ and $k_{z}^{*}$ are the dimensional wavenumbers. Similarly,
the magnitude of the rolls is defined by the wall-normal and spanwise velocity
magnitude of the $(k_{x},k_{z})=(0,1)$ mode of the MFU domain,

$\llangle\hat{r}\rrangle=\frac{1}{L_{y}}\int_{0}^{L_{y}}\left(\hat{v}(k_{x}=0,k_{z}=1)^{2}+\hat{w}(k_{x}=0,k_{z}=1)^{2}\right)^{1/2}\,\mathrm{d}y.$
(8)

Finally, the magnitude of the small scales is defined as the sum of all
velocity magnitude contained in smaller scales,

$\llangle\hat{s}\rrangle=\frac{1}{L_{y}}\int_{0}^{L_{y}}\left(\sum_{(k_{x},k_{z})\neq(0,0),(0,\pm
1)}\hat{u}(k_{x},k_{z})^{2}+\hat{v}(k_{x},k_{z})^{2}+\hat{w}(k_{x},k_{z})^{2}\right)^{1/2}\,\mathrm{d}y.$
(9)

The definitions are consistent with the definition of a streak and roll in an
MFU, where a single pair of infinitely long low- and high-speed streaks are
present.

The magnitude of streaks, rolls, and small scales are analyzed similarly to
production and dissipation. Distribution of the quantities and average
trajectories are shown in figure 5(a,b). Since three quantities were
considered, the phase space was divided into eight octants, with the origin at
the mean value of all three for the MFU. Principal axes of the data were
identified, but they were very closely aligned with the Cartesian axes, so
Cartesian axes were used to divide the space into octants, as shown in figure
5(a,b). The octants were named with a 3-digit binary key, with 0 for below-
average magnitude and 1 for above-average magnitude. The order of entries is
(streaks, rolls, small scales), so for example, octant $(0,1,1)$ represents
low magnitude contained in streaks, high magnitude contained in rolls, and
high magnitude contained in small scales. Transition probabilities were
calculated between the octants, shown in 5(c,d). Generally, transition
probabilities in the MFU and FC have many similarities. The MFU has a larger
spread of probability values, and it has more asymmetry in the direction of
movement between octants. For example, $(0,1,0)\to(0,0,0)$ has a significantly
higher probability than $(0,0,0)\to(0,1,0)$, but in the FC, both directions
have a similar probability.

(a)

(b)

(c)

(d)

Figure 5: Magnitude of the streak, roll, and small scales for the (a) MFU and
(b) FC. Contours are at 10% and 50% of occurrences, and arrows show the mean
trajectory. Black lines show octant divisions. Relative transition probability
between octants for the (c) MFU and (d) FC. Horizontal entries are the initial
octant, and vertical entries are the next octant.

We then form a graph using the trajectories. Each time the trajectory enters a
new octant, it is defined as an event. The PDF of the duration of single
events is shown in 6(a). Events in the MFU had longer durations than the FC,
with the average duration of an MFU event being 2.21 times as long as the
average FC event. This value is close to the ratio of lifetimes of the
spatiotemporal motifs reported in table 3.3 of $2.83$ and is even closer to
the ratio of timescales implied by the advection velocities of the slow
streaks between the two flows of approximately $2.17$. The trajectory can be
recast as a series of events, which then form the graph, where each event is a
node and the transition from one event to the next event is an edge between
the two nodes. Some events occurred for short durations, especially in the FC,
so events lasting $\Delta t^{+}\leq 5$ were removed, and the trajectory was
reassembled. This corresponds to the region of figure 6(a) where the duration
of events in the FC started to plateau. It was found that the motif results
were not sensitive to a cutoff value of $\Delta t^{+}=10$. The resulting graph
sizes were 17,952 events for the MFU and 90,579 events for the FC.

(a)

(b)

(c)

Figure 6: (a) PDF of duration of single events for the MFU (blue) and FC
(red). The vertical line at $\Delta t^{+}=5$ indicates the filter cutoff. (b)
PDF of duration of two-event subgraphs for the MFU (blue) and FC (red). Solid
lines represent all two-event subgraphs, dashed lines represent the
$(1,1,0)\to(1,1,1)$ motif, and dot-dashed lines represent the
$(1,0,0)\to(1,0,1)$ motif. (c) PDF of duration of three-node subgraphs for the
MFU (blue) and FC (red). Solid lines represent all three-event subgraphs,
dashed lines represent the $(0,0,1)\to(1,0,1)\to(1,0,0)$ motif, and dot-dashed
lines represent the $(0,0,0)\to(1,0,0)\to(1,1,0)$ motif.

The graphs were analyzed for motifs following the method of [23]. Because each
node in the graph had one incoming and one outgoing connection, the method
could be simplified and was much less computationally expensive than the
method described in section 3. Due to the restricted nature of the
trajectories, there were $8^{n}$ possible isomorphic subgraphs of $n$ nodes.
To analyze the trajectory for $n$-node motifs, the trajectory was broken down
into sequences of $n$ event-induced subgraphs, each classified as one of the
$8^{n}$ isomorphic subgraphs. The number of times each isomorphic subgraph
occurred was then tabulated. The significance of each of the induced subgraphs
were computed following equation 4.

The first five 2-node and 3-node subgraphs with the highest significance are
reported in tables 4.3 and 4.3. Generally, motifs in the MFU had a higher
significance $\widetilde{W}(s)$ compared to the FC. This was expected since
the MFU is more dynamically restricted, so the dynamics were frequently
repeated.

When considering 2-node motifs, all identified motifs in the MFU, and three of
the top five motifs in the FC, involve changing energy in the small scales.
subgraphs $(1,1,0)\to(1,1,1)$ and $(1,0,0)\to(1,0,1)$ were among the five most
significant motifs in both the MFU and FC. These motifs represent physical
phenomena, with the flow initially containing high energy in large-scale
structures and moving to a state with energy in both large and small-scale
structures, reflective of the energy cascade.

The 2-node motifs provide insight into the directionality of the trajectories.
For example, in the MFU, $(0,0,0)\to(0,0,1)$ and $(0,0,1)\to(0,0,0)$ have very
similar significance values, which suggests that both directions have similar
importance to the dynamics. However, $(0,1,1)\to(0,1,0)$ has much higher
significance than $(0,1,0)\to(0,1,1)$, with significance scores
$\widetilde{W}_{(0,1,1)\to(0,1,0)}=0.3276$ and
$\widetilde{W}_{(0,1,0)\to(0,1,1)}=0.0236$. This matches the physical
understanding of the SSP because $(0,1,1)\to(0,1,0)$ reflects the step in the
SSP where streamwise rolls are formed from small-scale structures by a
nonlinear process. $(0,1,0)\to(0,1,1)$ is not understood as a part of the SSP,
so it has a lower significance. Interestingly, there are variations in
directionality between the MFU and FC. For example, in the FC,
$(1,0,0)\to(1,0,1)$ and $(1,0,1)\to(1,0,0)$ are similar, with

$\frac{\widetilde{W}_{(1,0,0)\to(1,0,1)}}{\widetilde{W}_{(1,0,1)\to(1,0,0)}}=1.95.$

In the MFU, $(1,0,0)\to(1,0,1)$ is much more significant than
$(1,0,1)\to(1,0,0)$, with

$\frac{\widetilde{W}_{(1,1,0)\to(1,1,1)}}{\widetilde{W}_{(1,1,1)\to(1,1,0)}}=4.54.$

When considering the ratios of the number of occurrences, rather than the
significance, similar values are found. This suggests that the directionality
of the movement between these octants is more significant in the MFU than the
FC.

When considering 3-node motifs, several very rare subgraphs had high
significance because they were also very rare in the randomly generated
networks. Therefore, we did not report subgraphs that occurred fewer than 40
times. These rare subgraphs tended to contain diagonal movements or back-and-
forth movements between octants. The five subgraphs with the highest
significance meeting the occurrence threshold are reported in table 4.3.
Again, there is strong similarity between the MFU and FC, and the MFU tended
to have higher significance values.

Motifs $(0,0,1)\to(1,0,1)\to(1,0,0)$ and $(0,0,0)\to(1,0,0)\to(1,1,0)$ were
among the five most significant motifs in both the MFU and FC. While most
subgraphs had similar significance rankings between the MFU and FC, there were
a few that were very different. For example, $(0,0,0)\to(0,0,1)\to(1,0,1)$ had
the third highest significance in the MFU, with
$\widetilde{W}_{(0,0,0)\to(0,0,1)\to(1,0,1)}=0.0410$. However in the FC,
$(0,0,0)\to(0,0,1)\to(1,0,1)$ was the 349th most significant subgraph, with a
negative significance value of
$\widetilde{W}_{(0,0,0)\to(0,0,1)\to(1,0,1)}=-.0013$. This suggests that there
are dynamical patterns that are very significant to the MFU but not the FC.
This is similar to the re-ordering of the most significant, as well as most
common, yet dominant, spatiotemporal motifs in table 2. However, in general,
the similarities between the motifs indicate that there is significant
dynamical similarity between the MFU and FC.

The duration of significant motifs was compared to the duration of all
subgraphs taken together, and the results are shown in figure 6(b,c). Duration
was determined by summing the duration of the events making up the subgraph.
In particular, motifs that were among the top five most significant in both
the MFU and FC were considered. For two node motifs, this was
$(1,1,0)\to(1,1,1)$ and $(1,0,0)\to(1,0,1)$, and for three node motifs, this
was $(0,0,1)\to(1,0,1)\to(1,0,0)$ and $(0,0,0)\to(1,0,0)\to(1,1,0)$. The FC
tended to have faster times than the MFU, and this was seen in both the
duration of all events, and the duration of individual subgraphs. Even though
the same subgraphs were occurring with similar significance values in the MFU
and FC, they tended to have shorter durations in the FC. Further, the
durations of the significant subgraphs did not scale proportionally between
the MFU and FC. For example, $(0,0,1)\to(1,0,1)\to(1,0,0)$ was generally
slower than the background turbulence (represented by the solid lines in
figure 6) in the FC, but faster than the background turbulence in the MFU.

Table 5:  Five most significant 2-node motifs. Bold-face motifs are common
between the MFU and FC. $\widetilde{W}(s)$ is the significance score, and
Occ., or occurrence, is the number of times the subgraph occurs in the real
trajectory.

Minimal Flow Unit |  | Full Channel
---|---|---
$s$ | $\widetilde{W}(s)$ | Occ. |  | $s$ | $\widetilde{W}(s)$ | Occ.
$\boldsymbol{(1,1,0)\rightarrow(1,1,1)}$ | 0.370 | 1669 |  |  | $\boldsymbol{(1,1,0)\rightarrow(1,1,1)}$
---
0.326 | 2664
$(0,1,1)\rightarrow(0,1,0)$ | 0.328 | 816 |  | ${(1,1,0)\rightarrow(1,0,0)}$ | 0.244 | 1181
${(0,0,0)\rightarrow(0,0,1)}$ | 0.297 | 1109 |  | $(1,1,1)\rightarrow(1,1,0)$ | 0.229 | 2024
$(0,0,1)\rightarrow(0,1,1)$ | 0.290 | 1141 |  |  | $\boldsymbol{(1,0,0)\rightarrow(1,0,1)}$
---
0.191 | 526
| $\boldsymbol{(1,0,0)\rightarrow(1,0,1)}$
---
0.279 | 610 |  | ${(1,1,1)\rightarrow(1,0,1)}$ | 0.178 | 888
|  |  |  |  |  |

Table 6: Five most significant 3-node motifs, excluding those with fewer than
12 occurrences. Bold-face motifs are common between the MFU and FC.
$\widetilde{W}(s)$ is the significance score, and Occ., or occurrence, is the
number of times the subgraph occurs in the real trajectory.

Minimal Flow Unit |  | Full Channel
---|---|---
$s$ | $\widetilde{W}(s)$ | Occ. |  | $s$ | $\widetilde{W}(s)$ | Occ.
$\mathbf{(0,0,1)\to(1,0,1)\to(1,0,0)}$ | 0.0466 | 79 |  | ${(0,1,1)\to(1,1,1)\to(1,1,0)}$ | 0.0178 | 910
${(0,1,0)\to(1,1,0)\to(1,0,0)}$ | 0.0432 | 283 |  | $\mathbf{(0,0,0)\to(1,0,0)\to(1,1,0)}$ | 0.0167 | 282
$(0,0,0)\to(0,0,1)\to(1,0,1)$ | 0.0410 | 138 |  | ${(1,0,1)\to(1,1,1)\to(1,0,1)}$ | 0.0156 | 127
$(1,1,1)\to(1,1,0)\to(0,1,0)$ | 0.0305 | 89 |  | $\mathbf{(0,0,1)\to(1,0,1)\to(1,0,0)}$ | 0.0153 | 106
$\mathbf{(0,0,0)\to(1,0,0)\to(1,1,0)}$ | 0.0296 | 261 |  | $(0,0,1)\to(1,0,1)\to(1,1,1)$ | 0.0132 | 206
|  |  |  |  |  |

##  5 Conclusions

Interpreting wall-bounded turbulence as a collection of interacting coherent
structures is a long-standing paradigm. However, it needs a consistent
framework or an algorithmic approach to extract and analyze important
dynamical events. In this work, we attempt to do so by employing coherent-
structure tracking, semi-Lagrangian wavepackets, and analysis techniques from
network science and graph theory. As a proof of concept, we had two aims.
First, to extract the signature of the self-sustaining process
algorithmically. Second, to analyze the dynamics of the extracted self-
sustaining process in both a minimal flow unit and a large unrestricted domain
to understand how they differ.

To do so, we first extended the coherent structure tracking algorithms of [18]
to allow for simultaneously tracking multiple sets of eddies. In this study,
we limited ourselves to tracking attached ejections, sweeps, and slow and fast
streaks simultaneously. The spatiotemporal interactions between these eddies
are encoded into a complex multilayer network, which is then searched for
significantly non-random patterns given an appropriate model for a random
network, referred to as network motifs [23]. This analysis extracted the roll-
streak pairing as the most significant and perhaps simplest non-random
dynamical pattern in both flows, satisfying the first goal. However, it was
detected that other significant, as well as frequent, yet dominant patterns
were ordered differently between the two flows. This indicates that while the
most prominent dynamical mechanism is the same between the MFU and the FC,
there are others that are substantially different in both frequency and
significance between the two flows. Furthermore, even for the most significant
dynamical pattern, $\mathcal{S}_{1}^{\\{F,M\\}}$, the average timescale was
$2.83$ longer in the MFU than the FC.

We simultaneously employed semi-Lagrangian wavepackets to analyze the phase-
space dynamics of entire MFU-sized regions advecting within the FC, and
compared them to those of the MFU. Within the octant space defined by streak,
roll, and small-scale magnitude, graphs were constructed from the sequence of
transition events from one octant to the next. Overall, it was found that both
the 2-node and 3-node motifs were mostly similar between the MFU and the FC,
with 3-node motifs representing transitions from one scale to another.
However, a few key differences were found. First, the significance scores,
$\widetilde{W}(s)$, were higher for the MFU compared to FC, presumably due to
the restricted dynamics of the MFU. Second, the average timescales of each
motif were once again faster in the FC compared to MFU, with the average
duration of an MFU event being $2.21$ slower than that of an FC event,
consistent with the motif lifetimes extracted using spatiotemporal tracking,
and in particular with the implied ratio of timescales for the slow speed
streaks of $2.17$.

Overall, an algorithmic multi-eddy-type spatiotemporal tracking approach to
detecting key dynamical mechanisms in wall-bounded turbulence was proposed,
developed, and tested on a control experiment. It was found that the roll-
streak SSP is the lowest form of spatiotemporal organization, and that its
timescale in the MFU is substantially slower compared to its unrestricted
counterpart. This key finding was corroborated with a semi-Lagrangian
wavepacket approach combined with temporal motif extraction in phase space. In
the future, we aim to extend this approach to detect key interscale dynamics
at higher $Re_{\tau}$, and extract higher-order mechanisms by comparing to
more complex random network models that constrain more than just the colored-
degree distribution.

## Acknowledgments

A. Elnahhas and P. Moin acknowledge support from NASA’s Transformational Tools
and Technologies project under grant no. 80NSSC20M0201. E. Lenz acknowledges
support from the NSF Graduate Research Fellowship under grant no. DGE-1745301.
All authors acknowledge insightful comments from José Cardesa on an earlier
version of the manuscript. All authors acknowledge the European Research
Council (ERC) under the Caust grant ERC-AdG-101018287, and Prof. J. Jiménez
for the participation in the 2023 Madrid Summer Turbulence Workshop.

## References

## References

  * [1] Robinson S K 1991 Coherent motions in the turbulent boundary layer Annu. Rev. Fluid Mech. 23 601–639
  * [2] Townsend A A 1976 The Structure of Turbulent Shear Flow (Cambridge, UK: Cambridge University Press)
  * [3] Marusic I and Monty J P 2019 Attached eddy model of wall turbulence Annu. Rev. Fluid Mech. 51 49–74
  * [4] Jiménez J and Moin P 1991 The minimal flow unit in near-wall turbulence J. Fluid Mech. 225 213–240
  * [5] Hamilton J M, Kim J and Waleffe F 1995 Regeneration mechanisms of near-wall turbulence structures J. Fluid Mech. 287 317–348
  * [6] Jiménez J and Pinelli A 1999 The autonomous cycle of near-wall turbulence J. Fluid Mech. 389 335–359
  * [7] Flores O and Jiménez J 2010 Hierarchy of minimal flow units in the logarithmic layer Phys. Fluids 22 71704
  * [8] Hwang Y and Cossu C 2010 Self-sustained process at large scales in turbulent channel flow Phys. Rev. Letters 105 044505
  * [9] Hwang Y and Cossu C 2011 Self-sustained processes in the logarithmic layer of turbulent channel flows Phys. Fluids 23 061702
  * [10] Bae H J and Lozano-Durán A 2019 A minimal flow unit of the logarithmic layer in the absence of near-wall eddies and large scales Annu. Res. Briefs, Center for Turbulence Research, Stanford University, 247–256
  * [11] Bae H J, Lozano-Durán A and McKeon B 2021 Nonlinear mechanism of the self-sustaining process in the buffer and logarithmic layer of wall-bounded flows J. Fluid Mech. 914 A3
  * [12] Lozano-Durán A, Bae H J and Encinar M P 2020 Causality of energy-containing eddies in wall turbulence J. Fluid Mech. 882 A2
  * [13] Hutchins N and Marusic I 2007 Evidence of very long meandering features in the logarithmic region of turbulent boundary layers J. Fluid Mech. 579 1–28
  * [14] Hutchins N and Marusic I 2007 Large-scale influences in near-wall turbulence Phil. Trans. Royal Soc. A: Math. Phys. Eng. Sci. 365 647–664
  * [15] Marusic I, Mathis R and Hutchins N 2010 Predictive model for wall-bounded turbulent flow Science 329 193–196
  * [16] Zhou Z, Xu C X and Jiménez J 2022 Interaction between near-wall streaks and large-scale motions in turbulent channel flows J. Fluid Mech. 940 A23
  * [17] Lozano-Durán A, Flores O and Jiménez J 2012 The three-dimensional structure of momentum transfer in turbulent channels J. Fluid Mech. 694 100–130
  * [18] Lozano-Durán A and Jiménez J 2014 Time-resolved evolution of coherent structures in turbulent channels: characterization of eddies and cascades J. Fluid Mech. 759 432–471
  * [19] Dong S, Huang Y, Yuan X and Lozano-Durán A 2020 The coherent structure of the kinetic energy transfer in shear turbulence J. Fluid Mech. 892 A22
  * [20] Bae H J and Lee M 2021 Life cycle of streaks in the buffer layer of wall-bounded turbulence Phys. Rev. Fluids 6 064603
  * [21] Boccaletti S, Bianconi G, Criado R, Del Genio C I, Gómez-Gardeñes J, Romance M, Sendiña-Nadal I, Wang Z and Zanin M 2014 The structure and dynamics of multilayer networks Phys. Reports 544 1–122
  * [22] Kivelä M, Arenas A, Barthelemy M, Gleeson J P, Moreno Y, Porter A M and Estrada E 2014 Multilayer networks J. Comp. Networks 2 203–271
  * [23] Milo R, Shen-Orr S, Itzkovitz S, Kashtan N, Chklovskii D and Alon U 2002 Network motifs: Simple building blocks of complex networks Science 298 824–827
  * [24] Ribeiro P, Paredes P, Silve M E P, Aparicio D and Silva F 2021 A survey on subgraph counting ACM Comp. Surveys 54 A28
  * [25] Alon U 2007 Network motifs: theory and experimental approaches Nature Rev. Gen. 8 450–461
  * [26] Taira K, Nair A G and Brunton S L 2016 Network structure of two-dimensional decaying isotropic turbulence J. Fluid Mech. 795 R2
  * [27] Schmid P J, García-Gutierrez A and Jiménez J 2018 Description and detection of burst events in turbulent flows J. Phys.: Conf. Ser. 1001, Third Madrid Summer School for Turbulence, 012015
  * [28] Iacobello G, Scarsoglio S, Kuerten J G M and Ridolfi L 2019 Lagrangian network analysis of turbulent mixing J. Fluid Mech. 865 546–562
  * [29] Krueger P S, Hahsler M, Olinick E V, Williams S H and Zharfa M 2019 Quantitative classification of vortical flows based on topological features using graph matching Proc. Royal Soc. A: Math. Phys. Eng. Sci. 475 A8
  * [30] Fernex D, Noack B R and Semaan R 2021 Cluster-based network modeling: From snapshots to complex dynamical systems Science Adv. 7 5006–5022
  * [31] Iacobello G, Ridolfi L and Scarsoglio S 2021 Large-to-small scale frequency modulation analysis in wall-bounded turbulence via visibility networks J. Fluid Mech. 918 A13
  * [32] Li H, Fernex D, Semaan R, Tan J, Morzyński M and Noack B R 2021 Cluster-based network model J. Fluid Mech. 906 A21
  * [33] Perrone D, Kuerten J G M, Ridolfi L and Scarsoglio S 2021 Network analysis of Reynolds number scaling in wall-bounded lagrangian mixing Phys. Rev. Fluids 6 124501
  * [34] Taira K and Nair A G 2022 Network-based analysis of fluid flows: Progress and outlook Prog. Aero. Sci. 131 100823
  * [35] Wernicke S and Rasche F FANMOD: a tool for fast network motif detection Bioinformatics.  22 1152–1153
  * [36] Wernicke S Efficient detection of network motifs IEEE/ACM trans. comp. biology & bioinformatics  3 347–359
  * [37] Ribeiro P and Silva F 2014 Discovering colored network motifs Complex Networks V: Proceedings of the 5th Workshop on Complex Networks 107–118
  * [38] Jiménez J, Kawahara G, Simens M P, Nagata M and Shiba M 2005 Characterization of near-wall turbulence in terms of equilibrium and “bursting” solutions Phys. Fluids 17 015105
  * [39] Benson A R, Gleich D F and Leskovec J 2016 Higher-order organization of complex networks Science, 353 163–166


# Prosody for Intuitive Robotic Interface Design:
It’s Not What You Said, It’s How You Said It

Elaheh Sanoubari1, Atil Iscen2, Leila Takayama3, Stefano Saliceti2,
Corbin Cunningham1 and Ken Caluwaerts2
1Google Labs 2Google DeepMind 3Hoku Labs. {eqs, atil, ssaliceti, corbinc,
<EMAIL_ADDRESS><EMAIL_ADDRESS>
(2024)

###### Abstract.

In this paper, we investigate the use of ‘prosody’ (the musical elements of
speech) as a communicative signal for intuitive human-robot interaction
interfaces. Our approach, rooted in Research through Design (RtD), examines
the application of prosody in directing a quadruped robot navigation. We
involved ten team members in an experiment to command a robot through an
obstacle course using natural interaction. A human operator, serving as the
robot’s sensory and processing proxy, translated human communication into a
basic set of navigation commands, effectively simulating an intuitive
interface. During our analysis of interaction videos, when lexical and visual
cues proved insufficient for accurate command interpretation, we turned to
non-verbal auditory cues. Qualitative evidence suggests that participants
intuitively relied on prosody to control robot navigation. We highlight
specific distinct prosodic constructs that emerged from this preliminary
exploration and discuss their pragmatic functions. This work contributes a
discussion on the broader potential of prosody as a multifunctional
communicative signal for designing future intuitive robotic interfaces,
enabling lifelong learning and personalization in human-robot interaction.

Human-Robot Interaction, Prosody, Robotic Interfaces

††copyright: acmlicensed††journalyear: 2024††conference:  ACM/IEEE
International Conference on Human Robot Interaction (HRI); March 11-15, 2024;
Boulder, CO, USA

##  1\. Introduction

In 1968, Douglas Engelbart’s ’mother of all demos’ showcased the first
Graphical User Interface (GUI), revolutionizing human-machine interaction and
shaping the future of computers for decades to come. Engelbart posed this
question, “If in your office, you as an intellectual worker were supplied with
a computer display, backed up by a computer that was alive for you all day,
and was instantly responsive to every action you had, how much value would you
derive from that?” We extend the inquiry: What if the responsive computer was
a robot? An important distinction that sets Engelbart’s computer apart from a
robot is that the latter is a mobile agent capable of navigating through our
physical space. Understanding and following navigation commands is paramount
to a mobile robot that is instantly responsive to humans. Motivated by this,
we set out to explore how we might design an intuitive control interface that
allows a co-located human to guide a robot’s navigation. Adopting a Research
through Design (RtD) approach, we invited participants to guide a quadruped
robot through an obstacle course using any communication that felt natural to
them. To simulate an intuitive interface, a human ’wizard’ acted as the
robot’s sensory and cognitive proxy, interpreting and executing participants’
commands.

In our analysis of interaction videos, we observed that prosodic cues (musical
elements of speech) played a crucial role in accurately interpreting commands
when lexical and visual cues were inadequate. These cues, found unexpectedly,
supplied essential context to disambiguate numerous commands. Although the
wizard could discern and react to these cues while operating the robot, they
were not apparent in the transcribed commands. We discuss related work and
present an overview of the concept of prosody in section 2, describe our
methodology in section 3, and present an overview of specific prosodic
patterns that we found evidence for in section 5.

Prosody is a rich communicative signal that carries information relevant to
the task of controlling a co-located robot’s navigation, and as such, it holds
promise for influencing the design of intuitive robotic interfaces. While
Natural Language Understanding (NLU) for robot control is gaining interest,
prosody can complement this by providing supplementary context that is
difficult or inefficient to express through spoken language alone. Prosody’s
multifunctional nature and the nuanced control people have over it make it an
appealing signal for effective and user-friendly robotic control interfaces.
Computational systems that objectively capture prosodic cues via acoustic
analysis of speech signal can outperform humans (Ward, 2019b). We extend this
discussion in section 6.

Finally, prosody’s pragmatic and paralinguistic functions make it relevant to
lifelong learning and personalization in human-robot interaction (HRI).
Pragmatic functions of prosody, such as directing attention, expressing
uncertainty, prioritizing information, and coordinating actions are crucial
aspects for effective HRI and life-long in-context learning. Additionally,
prosody conveys paralinguistic information, such as user traits (e.g., age),
emotions (e.g., anger), and cognitive states (e.g., tiredness), which are
valuable in many robotic applications, and can facilitate speaker
identification and personalized HRI experiences. As a preliminary exploration
of prosody in HRI, this work aims to promote utilizing prosodic cues for
designing novel intuitive robotic control interfaces.

##  2\. Background & Related Work

To facilitate effective HRI, we need interfaces that enable seamless
bidirectional human-robot communication. Traditionally, research in human-
robot communication has concentrated on strictly-defined problem domains. This
has involved examining the effects of specific communication modalities (such
as ‘gaze’) using the robot either as a stimulus to assess the limits of human
reactions, or as a means to explore design variations by altering one modal
aspect at a time (e.g., the ‘duration’ of gaze) and gauging its impact on
human behavior (Admoni and Scassellati, 2017). Despite robots performing well
in demos involving a single communication medium, such performance is limited
in that it relies on stringent constraints and tight coupling of both the
environment and the user (Marge et al., 2022).

Natural human interaction involves a great deal of redundancy, as the same
message is typically communicated through multiple channels such as words,
prosody, gaze, posture, facial expressions, hand gestures, and actions (Admoni
and Scassellati, 2017; Marge et al., 2022; Gaschler et al., 2012a; Ward,
2019a; Gaschler et al., 2012b). We advocate for moving towards holistic and
intuitive robotic interfaces designed to align with natural human
interactions. We define an intuitive interface for human-robot interaction to
be an integrated multi-modal communication system that satisfies two general
requirements:

  1. (1)

Has competencies that allow it to: read embodied signals, generate robot
messages, and transmit them to humans effectively such that they are
immediately and easily understood.

  2. (2)

Has affordances that allow it to: receive system-directed messages via natural
human interaction, interpret them to extract action-relevant directives, and
facilitate action.

Along this line, building robot controllers based on spoken interaction is
gaining traction (Marge et al., 2022). Prior work that has outlined 25
recommendations as key advances needed for effective spoken interaction with
robots, emphasizes the requirement to ‘better exploit prosodic information’
(Marge et al., 2022). While speech, encompasses both words and prosody,
traditional speech recognition systems primarily focus on words, often
overlooking the rich information embedded in prosody. While this omission may
be inconsequential in certain applications, such as asking a voice assistant
to play music, it becomes more significant in the context of robots, given
that they are mobile agents embodied in our space. In certain HRI instances,
what is conveyed via prosody can be more important than the the actual words
spoken. As an example, Marge et al. (Marge et al., 2022) point to how the word
“oops” can vary in meaning from a ‘minor mistake’ to a ‘significant surprise’,
with prosody being the key indicator of the severity of the situation.

Prosody, the musical aspect of speech, encompasses elements such as
fundamental frequency (F0), loudness, timing, and spectrual information.
Prosody is pervasive in everyday life and plays a vital role in communication
by conveying information that extends beyond the literal meaning of words.
Prosodic constructs, defined as ‘temporal configurations of prosodic features
that carry specific meanings’ (Ward, 2019a) are complex, but systemic.
Prosodic constructs transcend direct alignment with words, can vary in degree,
and can be superimposed on other prosodic features to convey different
meanings or nuances (Ward, 2019b). Effective use of prosody can significantly
enhance communication and its lack can lead to confusion or delayed responses
from interaction partners.

Prosody serves many pragmatic functions; for example, we use specific
configurations of prosodic elements for directing attention (“over there!”),
conveying uncertainty (“the round one?”), establishing priorities (“help!”),
and coordinating action (“three, two, one… go!”) (Marge et al., 2022).
Moreover, English prosody includes specific constructs that distinguish
different dialogical activities, such as explaining, arguing, or making
decisions (Ward, 2019a). Such prosodic features can be objectively extracted
via accoustic processing of speech signals (e.g., (Eyben et al., 2010a;
Schuller and Batliner, 2013)). In fact computational models can already
outperform humans in detecting certain prosodic cues. For example, Skantze has
proposed a model using LSTM Recurrent Neural Networks can predict whether a
conversational turn-shift will occur or not in pauses, better than human
observers, using a fairly basic set of five prosodic features (Skantze, 2017).
The ability to extract prosodic information from human communication signals
is not merely beneficial for long-term HRI, but it is rather essential for
effectively organizing in-context continual robot learning.

##  3\. Methodology

To explore intuitive robotic interfaces, we took a Research through Design
(RtD) approach: a methodology where design practice is used as a means to
conduct research, which may involve iterative designing and prototyping to
explore complex problems, generate ‘tacit knowledge’, and propose innovative
solutions (Lupetti et al., 2021; Mascitelli, 2000).

The robot used in this study was a small quadruped robot developed in-house,
that weighs approximately $13.5\text{\,}\mathrm{kg}$, and stands at
approximately $0.4\text{\,}\mathrm{m}$ tall (see Fig. 1). To prototype an
intuitive interface, we invited human participants to guide a co-located
quadruped robot through an obstacle course, while a human operator wizarded
(Riek, 2012) the robot by interpreting human instructions in real-time and
translating them into basic robot navigation commands. The navigation
controller used by the wizard was a command-line interface that facilitated
interaction through keyboard inputs, enabling seven distinct actions: moving
forward, backward, left, right; turning left or right; and stopping (see Fig.
1).

Figure 1. Robot and the controller actions.

###  3.1. Procedure

We set up the experiment room as depicted in Fig. 2. Participants were told
the goal of this study is to explore how people might give a quadruped robot
navigation commands via natural human interaction. We asked them to guide the
robot through the obstacle course by giving it instructions in any way that
felt natural to them. Specifically, we asked participants to navigate the
robot (initially placed at the center of the room, point A) to the three balls
in RGB order: first the red ball (point R), then the green ball (point G), and
finally the blue ball (point B); then, navigate back to the center of the room
(point A), while avoiding the obstacle cones.

Figure 2. Experimental Setup, and a snapshot from the study

###  3.2. Participants and Data

We invited 10 individuals with moderate to high familiarity with the robot who
regularly operated quadrupeds (on a daily to weekly basis). Participants had
varying job titles, such as research scientist, roboticist, robotics engineer,
robot operator, and software engineer. The collected interaction data amounted
to 1.5 hours of recorded video, which was transcribed into a total of 194
verbal commands.

##  4\. Analysis

We conducted qualitative analysis by identifying and organizing patterns in
the data through thematic analysis (Braun and Clarke, 2006) and grouping
related ideas using affinity diagramming (Holtzblatt et al., 2004). Given our
primary goal of understanding the minimum technical requirements for designing
an interface that could replace the human ‘wizard’, our initial approach was
to analyze the ‘transcripts’ of user commands. However, we quickly realized
that the majority of the transcribed verbal commands were inherently
ambiguous; that is, it was challenging to map specific lexical commands such
as ’go around’, to precise controller actions (as depicted in Fig. 1).

Our initial attempt to address command ambiguity involved incorporating visual
cues. While this improved overall clarity, it did not resolve the issue; for
instance, the word ‘nice’ was used in varied contexts, sometimes indicating
‘keep going’, other times indicating ‘stop’, rendering both lexical and visual
cues ineffective in disambiguating the intended meaning. Further examination
of these ambiguous cases revealed a key insight: the distinguishing factor
often lay in the presence of different prosodic cues in the audio of the
speech. This led to the realization that prosodic cues were essential for
accurate command interpretation in many cases.

##  5\. Observations: Prosody for HRI

Below, we introduce a few of the specific prosodic constructs identified in
our qualitative analysis, elaborate their pragmatic functions, and provide
examples of observed qualitative evidence.

###  5.1. High-Priority Interpolation Construction

This prosodic pattern, characterized by a slow rise in pitch, low intensity,
and a fast speaking rate, focuses on marking utterances that deviate from the
normal topic structure, such as priority topics (i.e., things that must be
done immediately). Additional features often present include a subsequent
silence and breathy voice (see chapter 12 (Ward, 2019a)). In this study, we
observed that participants frequently employed this prosodic construct to
‘convey urgency’ in their commands. Exemplifying the use of this construct, in
one interaction instance, as the robot approached a cone, initially, the
participant calmly instructed the robot, ‘turn a bit to the left and stop…’;
however, as the robot took another step and dangerously neared the cone, the
participant’s repeated and hastened ‘left-left-left-left!’ This served as a
prosodic cue to convey a ‘shift in urgency’ and signal to the robot the
critical need to turn left immediately.

###  5.2. Minor Third Construction

This prosodic pattern is characterized with two (typically elongated) regions
of flat pitch with a short-intensity dip. According to Ward (Ward, 2019a),
this construct generally functions as a cue for the listener to take action
under specific conditions: (i) when there is a single clearly appropriate
action; (ii) when this action is required (e.g., by a social norm); (iii) when
the action is simple to execute; and (iv) when it should be carried out
immediately. In our qualitative analysis, we noted evidence for three distinct
variations of this prosodic pattern, each serving a different pragmatic
function.

####  5.2.1. Desist

This variation of Minor Third construction often features a downstepped
‘stop’, effectively cueing the cessation of an action (Ward, 2019a). This
pattern can be compelling even without words, as demonstrated when
discouraging a toddler from reaching for cookies before snack time with a
simple ‘uh oh’. An example of this construct in our study is a participant
commanding ‘keep going…’ as the robot was moving towards the ball, followed by
a harsh down-stepped ‘stop!’ as it critically approaches the wall.

####  5.2.2. Calling

A variation of minor third construct is frequently used for calling someone,
to the extent that prior work refers to it as as the “Calling Contour” (Ladd
Jr, 1978). An example of this from our study is an interaction wherein a
participant used this construct to call the robot using a two-syllable word in
place of a name, ‘dog-go’ (elongated, with the first syllable at a higher
pitch), to get the robot to turn and direct its attention to the speaker, from
across the room.

####  5.2.3. Reprimand

Another variant of Minor Third construct uses superimposed clipped ends to cue
a strong reprimand, with glottal stops after each syllable turning the generic
action-cueing effect into a more controlling one (Ward, 2019a), such as in
saying ‘bad dog’!

###  5.3. Backchannelling Construction

This construct is characterized by its lengthened, quiet utterances with a
typically flat pitch and slightly creaky voice. Its primary function is to
‘encourage the continuation of an action’. It is often used in communication
to establish shared knowledge or to serve as a subtle cue for continuation
(Ward, 2019a). An interaction instance in our study that exemplifies this
construct is a participant saying ’nice… nice…’, rhythmically and calmly, to
encourage the robot to keep walking in the same manner, speed and direction.

###  5.4. Positive Assessment Construction

This construct is characterized by a region of a relatively high pitch, then a
region of increased loudness with clear voicing, and finally a clipped ending
marked by a sharp drop in intensity. Its main function is to ‘express positive
assessment’ (Ward, 2019a). An example of this in our study was an interaction
instance in which after the robot successfully completed the task, the
participant patted its back and used this construct to utter ‘good # boy’!

##  6\. Discussion

Our study focused on exploring intuitive robotic interfaces through a Research
through Design approach. Participants instructed a quadruped robot in an
obstacle course, with their interactions recorded. Our observations revealed
that participants intuitively relied on various prosodic constructs to convey
key information. These included (a) the interpolation construct to signal
urgency, (b) the minor third construct for enforcing immediate cessation,
calling, and indicating reprimand, (c) the backchannelling construct to
encourage the continuation of an action, and (d) the positive assessment
construct for providing positive reinforcement.

Prosody not only facilitated the communication of time-sensitive commands in
our study but also played a crucial role in disambiguating communicative
signals. A notable example, discussed earlier in the paper, is the varied use
of the word ‘nice’– This word was sometimes used to indicate ‘keep going’, and
at other times, ‘stop’. Our analysis of prosodic cues revealed that in
instances where ’nice’ meant ‘keep going’, the backchannelling construct was
employed; conversely, when it was used to signal ‘stop’, it was in using the
minor third (desist) construct. This highlights how prosody can provide vital
context to seemingly straightforward lexical terms.

While in this study, interpretation of prosodic cues was done via a human
wizard, features such as fundamental frequency (F0), loudness, sound and
silence duration and other spectral information can be objectively extracted
using frame-by-frame acoustic analysis of real-time spoken interaction,
captured on a live mic. Given an annotated dataset, such prosodic features can
then be fed into models powerful enough to detect temporal dependencies
between them. As discussed earlier, computational systems using basic sets of
prosodic features paired with classfiers may outperform humans in detection
for many tasks. For future work exploring computational prosody, we recommend
considering openSMILE (Eyben et al., 2010b) (an open-source toolkit) and this
starter bibliography by Ward (Ward and Levow, 2021).

Furthermore, gaining a deeper understanding of prosody is pragmatic uses not
just for robots to make sense of human communication, but also for them to
generate communicative signals that people can intuitively understand. As
discussed earlier, specific prosodic constructs can indicate either positive
assessment, or reprimand. A robot detecting a prosodic ‘reprimand’ cue, can
interpret it as in-context punitive feedback and use it for adjustment or
cessation of an action, effectively leveraging prosody for reinforcement
learning. In addition, the robot can proceed to generate prosodic cues to
convey a “sincere” apology, for example by employing a non-verbal tone that
matches the ‘giving in’ construct (a prosodic configuration that conveys
sincerity in an apology, which children learn to use early in life. For
details, see (Ward, 2019a)). That is, communicative nuances embedded in
prosody can be used to enhance a robot’s ability to learn and respond
appropriately and empathetically.

A key advantage of prosody lies in its gradational nature, which allows us to
not only communicate cues such as urgency, but also discern the relative
intensity expressed in those cues. That is, we can listen to two lexically-
identical utterances and understand which one sounds more urgent. In other
words, prosody is ‘a matter of degree’, and humans have fine control over its
subtle variations in speech (Ward, 2019b), which allows for nuanced control.
It is worth noting that by applying a low-pass filter to speech, it is
possible to only preserve prosodic elements, discarding verbal content while
still enabling meaningful communication.

Computational prosody for HRI presents significant challenges. Disentangling
prosody from lexical content, while extracting cues relevant to robot actions
necessitates sophisticated approaches. Furthermore, integrating these prosodic
cues with lexical content in a way that maintains contextual understanding
introduces further complexities. Adding proficiency in non-verbal modalities
can improve the robustness of future intuitive robotic interfaces. To achieve
this, a deeper scientific understanding of prosody with an emphasis on ‘cross-
modality integration’ is needed.

Finally, we argue that prosody lends itself well to designing control
interfaces for for mobile agents, because it mimics how we communicate with
animals– not using our words, but using our prosody (Gergely et al., 2023;
Korcsok et al., 2020). In fact, work on evolutionary biology suggests that
“affective prosody in human acoustic communication has deep-reaching
phylogenetic roots, deriving from precursors already present and relevant in
the vocal communication systems of nonhuman mammals” (Zimmermann et al.,
2013). Throughout history, humans have partnered with animals (non-human
intelligent mobile agents) to extend their own embodiment and do things they
could not do on their own (Darling, 2021). We have trained animals to be
receptive to our communication and learn from us, by devising basic sets of
‘controller commands’ (e.g., American Kennel Club (AKC) recommends 5 basic
commands for training puppies (Lunchick, 2023)). Looking ahead, future
research could take inspiration and investigate how prosodic cues might be
leveraged for designing basic set of robot control commands.

##  7\. Conclusion

The study explores intuitive robotic interfaces using a Research through
Design approach. Participants interacted with a quadruped robot on an obstacle
course, using natural interaction to convey instructions. A key finding
includes evidence for intuitive use of prosody for signaling urgency,
stopping, encouragement, punitive and positive feedback. We highlight the role
of prosody in communicating time-sensitive commands and disambiguating
communicative signals, and emphasize its importance in adaptive robotic
learning and communication. This work underscores the potential of prosody for
designing future intuitive robotic control interfaces and enabling continual
learning and personalization in HRI.

## References

  * (1)
  * Admoni and Scassellati (2017) Henny Admoni and Brian Scassellati. 2017.  Social eye gaze in human-robot interaction: a review.  _Journal of Human-Robot Interaction_ 6, 1 (2017), 25–63.
  * Braun and Clarke (2006) Virginia Braun and Victoria Clarke. 2006.  Using thematic analysis in psychology.  _Qualitative research in psychology_ 3, 2 (2006), 77–101.
  * Darling (2021) Kate Darling. 2021\.  _The new breed: what our history with animals reveals about our future with robots_.  Henry Holt and Company.
  * Eyben et al. (2010a) Florian Eyben, Anton Batliner, and Bjoern Schuller. 2010a.  Towards a standard set of acoustic features for the processing of emotion in speech.. In _Proceedings of Meetings on Acoustics_ , Vol. 9. AIP Publishing.
  * Eyben et al. (2010b) Florian Eyben, Martin Wöllmer, and Björn Schuller. 2010b.  Opensmile: the munich versatile and fast open-source audio feature extractor. In _Proceedings of the 18th ACM international conference on Multimedia_. 1459–1462.
  * Gaschler et al. (2012a) Andre Gaschler, Kerstin Huth, Manuel Giuliani, Ingmar Kessler, Jan de Ruiter, and Alois Knoll. 2012a.  Modelling state of interaction from head poses for social human-robot interaction. In _Proceedings of the Gaze in Human-Robot Interaction Workshop held at the 7th ACM/IEEE International Conference on Human-Robot Interaction (HRI 2012)_.
  * Gaschler et al. (2012b) Andre Gaschler, Sören Jentzsch, Manuel Giuliani, Kerstin Huth, Jan de Ruiter, and Alois Knoll. 2012b.  Social behavior recognition using body posture and head pose for human-robot interaction. In _2012 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, 2128–2133.
  * Gergely et al. (2023) Anna Gergely, Anna Gábor, Márta Gácsi, Anna Kis, Kálmán Czeibert, József Topál, and Attila Andics. 2023\.  Dog brains are sensitive to infant-and dog-directed prosody.  _Communications Biology_ 6, 1 (2023), 859.
  * Holtzblatt et al. (2004) Karen Holtzblatt, Jessamyn Burns Wendell, and Shelley Wood. 2004.  _Rapid contextual design: a how-to guide to key techniques for user-centered design_.  Elsevier.
  * Korcsok et al. (2020) Beáta Korcsok, Tamás Faragó, Bence Ferdinandy, Ádám Miklósi, Péter Korondi, and Márta Gácsi. 2020\.  Artificial sounds following biological rules: A novel approach for non-verbal communication in HRI.  _Scientific reports_ 10, 1 (2020), 7080.
  * Ladd Jr (1978) D Robert Ladd Jr. 1978\.  Stylized intonation.  _Language_ 54, 3 (1978), 517–540.
  * Lunchick (2023) Paisley Lunchick. 2023\.  Teach your puppy these 5 basic cues.  https://www.akc.org/expert-advice/training/teach-your-puppy-these-5-basic-commands/
  * Lupetti et al. (2021) Maria Luce Lupetti, Cristina Zaga, and Nazli Cila. 2021\.  Designerly ways of knowing in HRI: Broadening the scope of design-oriented HRI through the concept of intermediate-level knowledge. In _Proceedings of the 2021 ACM/IEEE International Conference on Human-Robot Interaction_. 389–398.
  * Marge et al. (2022) Matthew Marge, Carol Espy-Wilson, Nigel G Ward, Abeer Alwan, Yoav Artzi, Mohit Bansal, Gil Blankenship, Joyce Chai, Hal Daumé III, Debadeepta Dey, et al. 2022\.  Spoken language interaction with robots: Recommendations for future research.  _Computer Speech & Language_ 71 (2022), 101255.
  * Mascitelli (2000) Ronald Mascitelli. 2000\.  From experience: harnessing tacit knowledge to achieve breakthrough innovation.  _Journal of Product Innovation Management: an International Publication of the Product Development & Management Association_ 17, 3 (2000), 179–193.
  * Riek (2012) Laurel D Riek. 2012\.  Wizard of oz studies in hri: a systematic review and new reporting guidelines.  _Journal of Human-Robot Interaction_ 1, 1 (2012), 119–136.
  * Schuller and Batliner (2013) Björn Schuller and Anton Batliner. 2013.  _Computational paralinguistics: emotion, affect and personality in speech and language processing_.  John Wiley & Sons.
  * Skantze (2017) Gabriel Skantze. 2017\.  Towards a general, continuous model of turn-taking in spoken dialogue using LSTM recurrent neural networks. In _Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue_. 220–230.
  * Ward (2019a) Nigel G Ward. 2019a.  _Prosodic patterns in English conversation_.  Cambridge University Press.
  * Ward (2019b) Nigel G. Ward. 2019b.  Survey Talk: Prosody Research and Applications: The State of the Art. In _Proc. Interspeech 2019_.
  * Ward and Levow (2021) Nigel G Ward and Gina-Anne Levow. 2021.  Computational Prosody Starter Bibliography.  (2021).
  * Zimmermann et al. (2013) Elke Zimmermann, Lisette Leliveld, and Simone Schehka. 2013\.  Toward the evolutionary roots of affective prosody in human acoustic communication: a comparative approach to mammalian voices.  _Evolution of emotional communication: from sounds in nonhuman mammals to speech and music in man_ 116 (2013), 132.