# The Multi-Event-Class Synchronization (MECS) Algorithm

Paolo Alborno  University of Genova ,  Gualtiero Volpe  University of Genova ,
Maurizio Mancini  University of Genova ,  Radoslaw Niewiadomski  University of
Genova ,  Stefano Piana  University of Genova and  Antonio Camurri  University
of Genova

###### Abstract.

Synchronization is a fundamental component of computational models of human
behavior, at both intra-personal and inter-personal level. Event
synchronization analysis was originally conceived with the aim of providing a
simple and robust method to measure synchronization between two time series.
In this paper we propose a novel method extending the state-of-the-art of the
event synchronization techniques: the Multi-Event-Class Synchronization (MECS)
algorithm. MECS measures the synchronization between relevant events belonging
to different event classes that are detected in multiple time series. Its
motivation emerged from the need to model non-verbal multimodal signals in
Human-Computer Interaction. Using MECS, synchronization can be computed
between events belonging to the same class (intra-class synchronization) or
between events belonging to different classes (inter-class synchronization).
In the paper we also show how our technique can deal with macro-events (i.e.,
sets of events satisfying constraints) and macro-classes (i.e., sets of
classes). In the last part of the paper, we apply the proposed method to two
types of data i) artificial and 2) real-world case study concerning analysis
of human multimodal behavior.

Event Synchronization, Intra-personal synchronization, Inter-personal
synchronization, Coordination, Event Classes

††journalvolume: 0††journalnumber: 0††article: 0††journalyear:
0††publicationmonth: 0††ccs: Computing methodologies Activity recognition and
understanding††ccs: Computing methodologies Model development and
analysis††ccs: Applied computing Arts and humanities

##  1\. Introduction

This paper presents _Multi-Event-Class Synchronization_ (_MECS_), a new
algorithm to measure the amount of synchronization between events detected in
two or more time series. MECS belongs to a family of Event Synchronization
(ES) techniques and it is inspired by the work of R. Q. Quiroga and colleagues
(Quiroga et al., 2002). The techniques to perform ES analysis are also known
under the name of Measures of spike train synchrony (Kreuz, 2011). ES analysis
is performed for measuring the degree of synchronization between events
occurring in a set of time series. The term “events” denotes a significant
behavior for a system.

With respect to other existing techniques, MECS brings substantial extensions
which allow to model a large set of real-life phenomena. First of all, it
deals with multiple classes of events. After grouping events into classes,
synchronization is computed within a class, i.e., between events belonging to
the same class (_intra-class synchronization_) and between classes, i.e.,
between events belonging to different classes (_inter-class synchronization_).
Additionally, events can be combined in _macro-events_ on which
synchronization is measured. Each _macro-event_ is an aggregation of the
events that satisfy some constraints. A relevant example of macro-event is a
sequence of events. Events and macro-events can be grouped in _macro-classes_
and synchronization can be computed within and between them.

While many of the existing ES algorithms (e.g., (Quiroga et al., 2002; Kreuz
et al., 2009) were developed in the context of brain signal analysis, MECS was
created with the purpose of studying multimodal human-human and human-machine
interaction. MECS can be applied to a large variety of problems and, in
particular, it can be used by human centered systems, multi-modal interfaces
for human-machine interaction, or to study multi-modal expressive behaviors of
individuals as well as social signals in groups. Indeed, intra-personal
synchronization between expressive behaviors of one or more body modalities is
an important cue of several emotion displays (Keltner, 1995; Niewiadomski et
al., 2011), synchronization between physiological signals and movement
kinematics allows one to distinguish between different qualities of human
full-body movement (Lussu et al., 2016). At the same time, interpersonal
synchronization of expressive behaviors in a group of people is an important
cue of group cohesion (Lakens and Stel, 2011; Hung and Gatica-Perez, 2010) and
soft-entrainment (Alborno et al., 2016).

This paper is organized as follows: in Section 2 we describe background and
present related work. Section 3 describes the MECS algorithm. In Section 4 we
discuss _macro-events_ and _macro-classes_. Section 5 presents a simple
implementation of the MECS algorithm with pseudo-code, then Section 6 provides
examples of the algorithm execution over a few synthetic sample signals.
Finally in Section 7, we discuss a concrete case study in the framework of
human behavior understanding: intra-personal synchronization between two
sensory modalities for analysis of expressive behavior.

Table 1. Comparison between existing event synchronization algorithms (Quiroga et al., 2002; Iqbal and Riek, 2016; Kreuz et al., 2009) and MECS. Reference | Number of | Macro
---|---|---
| time series | classes | events | classes
(Quiroga et al., 2002) | 2 | 1 | not handled | not handled
(Iqbal and Riek, 2016) | M | N | not handled | not handled
(Kreuz et al., 2009) | M | 1 | not handled | not handled
MECS | M | N | handled | handled

##  2\. Related work

Event Synchronization techniques measure synchronization and time-delay
patterns between two time series. An example of ES technique was proposed by
Quiroga et al. (Quiroga et al., 2002). Given two time series of $n$ samples
$x={x_{1},...,x_{n}}$ and $y={y_{1},...,y_{n}}$, $x_{i},y_{i}\in R$, events
are detected and the time instants $t^{x}_{i}$ and $t^{y}_{i}$, at which
events occur in time series $x$ and $y$ respectively, are computed.
Synchronization $Q_{\tau}$ between the detected events is measured as:

(1) $Q_{\tau}=\frac{c^{\tau}(y|x)+c^{\tau}(x|y)}{\sqrt{m_{x}m_{y}}}$

where $c^{\tau}(y|x)$ and $c^{\tau}(x|y)$ are, the number of times an event in
time series $y$ appears within a time interval defined by parameter $\tau$
after an event appears in time series $x$, and vice-versa. $m_{x}$ is the
number of events detected in time series $x$ and $m_{y}$ is the number of
events detected in time series $y$. $c(x|y)$ is computed as:

(2) ${c}^{\tau}(x|y)=\sum_{j=1}^{{m}_{x}}\sum_{i=1}^{{m}_{y}}{J}^{\tau}_{ij}$

where ${J}^{\tau}_{ij}$ is defined as follows:

(3) ${J}^{\tau}_{ij}=\begin{cases}1&\mbox{if
}0<{t_{i}}^{x}-{t_{j}}^{y}\leq{\tau}\\\ 1/2&\mbox{if
}{t_{i}}^{x}={t_{j}}^{y}\\\ 0&\mbox{otherwise}\end{cases}$

In particular, the second line of Equation 3 is needed, because when two
events occur simultaneously they will be counted 2 times by Quiroga’s
algorithm, a total contribution of 1 to the computation of $Q_{\tau}$. Also,
we highlight that the quantity $Q_{\tau}$ can be greater than 1.

Iqpal & Riek (Iqbal and Riek, 2016) proposed an extension of Quiroga’s
algorithm to deal with multiple types of events. In their approach, given two
time series $x$ and $y$ and the event type $E_{i},i=1...K$ they first compute
the synchronization index $Q_{\tau}(E_{i})$ for all events of type $E_{i}$
using the Equations 1 \- 3. Next, they compute synchronization of multiple
types of events between $x$ and $y$ as the average of $Q_{\tau}(E_{i})$,
weighted by the number of events of type $E_{i}$. In the last step, they
consider $M$ different time series $z_{1},...z_{M}$ and they compute their
pairwise and overall synchronization. In particular, the individual
synchronization of a given time series $z_{i}$ is the average of all the
pairwise synchronizations between $z_{i}$ and each time series $z_{j}:i\neq j$
that are beyond a certain fixed threshold $Q_{tresh}$. The overall
synchronization is the average of the products of the individual
synchronization indexes multiplied by their connectivity values, where the
connectivity value is the number of time series pairs having a pairwise
synchronization above the threshold $Q_{tresh}$, divided by the total number
of pairs.

Kreuz et al. (Kreuz et al., 2009) presented a multivariate extension to
measure the synchrony from the relative number of spike occurrences.

The Multi-Event Class Synchronization (MECS) algorithm we present in this
paper introduces new characteristics that are missing in works byQuiroga et
al., Iqpal & Riek, and Kreuz et al., as highlighted in Table 1. The table
provides a comparison between these algorithms in terms of: i) the maximum
number of input time series; ii) the maximum number of classes; iii) the
possibility of handling macro-events, and iv) macro-classes.

The MECS algorithm can compute the synchronization between $M=2$ time series
and a single event class ($N=1$). Differently from Quiroga et al., it provides
a normalized output in $[0,1]$ (see the above comment on the outputs of
Quiroga’s algorithm). It can also compute synchronization of $M>2$
multivariate time series, but, unlike Kreuz et al. which considers only single
event class ($N=1$), it manages multiple classes of events ($N>1$) within the
time series. Additionally to Iqpal & Riek, it manages the computation of
synchronization between events belonging to different classes (i.e., inter-
class synchronization).

Finally, differently from all the algorithms mentioned above, MECS introduces
the computation of synchronization between $N>1$ event classes over $M\geq 2$
time series, handling macro-events and macro classes.

##  3\. Multi-Event Class Synchronization

Multi-Event Class Synchronization (MECS) computes the amount of
synchronization between events occurring in a set of time series. Events may
belong to different classes: MECS computes (i) a separate synchronization
index for each class (_intra-class synchronization_), (ii) a synchronization
index for a specific aggregation of classes (_inter-class synchronization_),
and a global synchronization index for all classes.

To present MECS, we introduce an example scenario, which is helpful to explain
the major features of the algorithm. For example, suppose that we are
interested in measuring the level of motor coordination between the members of
$N$ users’ group performing a motor task (e.g., a fitness exercise). A measure
of coordination is obtained by evaluating the amount of synchronization
between the movements of the users. Let us consider:

  * •

A set $T$ of $N$ time series: $T\equiv{\\{TS_{1},...TS_{N}\\}}$

  * •

A set $E$ of $K$ event classes: $E\equiv{\\{E_{1},...,E_{K}\\}}$

Time series contain information about significant events in the data. In other
words, it describes occurrences of certain phenomena. Introducing the events
means to fit a continuous information (e.g., a velocity of a joint) to the
discrete domain. An event can be, for instance, a local maximum/minimum of the
continuous signal (e.g., joints velocity). Such events can be identified
automatically (see, e.g., (Alborno et al., 2016; Lussu et al., 2016)) or
annotated manually. Events can belong to several classes. Each class
$E_{1},...,E_{K}$ characterizes a different type of event, e.g., class $E_{1}$
might correspond to local maxima of $s$, while $E_{2}$ to local minima of the
same $s$.

Coming back to our example, the time series $TS_{1},...TS_{N}$ can be used
describe the motor activity of $N$ users (i.e., time series $TS_{j}$ models
the motor activity of user $j$) and contain events of any class
$E_{1},...,E_{K}$ while each class identifies a specific movement (for example
“step performed”, “object grabbed”, “object released”, and so on).

###  3.1. Intra-class synchronization

MECS relies its computation on the temporal distances between events. As in
(Quiroga et al., 2002; Kreuz, 2011), the computation consists of two steps: i)
the algorithm first detects events coincidences of two different time series
in a specific time interval (_coincidence detection_) and counts them, ii)
then the number of detected coincidences is normalized with respect to the
total number of possible coincidences (_normalization_). We associate to each
event $h$ of class $E_{k}\in E$, occurring in time series $TS_{n}\in T$, its
occurrence time:

(4) $\displaystyle t^{n,k}_{h}\qquad h=1,...,m_{n,k}$

where $m_{n,k}$ represents the total number of detected events of class
$E_{k}$ occurring in the $n$-th time series $TS_{n}$. For example,
$t^{2,3}_{4}$ represents the time at which the fourth event belonging to the
third class, i.e., $E_{3}$, occurred in $TS_{2}$. In the coincidences
detection phase, for each pair of time series $\langle TS_{i}\ ;\
TS_{j}\rangle$, the MECS algorithm computes the amount of coincidence
${c}_{k}(x,y)$, between an event $x$ detected on time series $TS_{i}$ and
another event $y$ (of the same class of $x$) detected on time series $TS_{j}$
(with $i\neq j$) by measuring the extent to which they are close in time
within a certain interval $\tau_{k}$ (_coincidence window_) that depends on
the class of the events. The temporal distance $d$ between events $x$ and $y$
is computed as:

(5) $d(x,y)=|{t^{i,k}_{x}}-{t^{j,k}_{y}}|$

The amount of coincidence $c_{k}$ between $x$ and $y$ is defined as follows:

(6) ${c}_{k}(x,y)=\begin{cases}1-\frac{d(x,y)}{\tau_{k}}&\mbox{if \quad}0\leq
d(x,y)\leq\tau_{k}\\\ 0&\mbox{otherwise}\end{cases}$

Different than (Quiroga et al., 2002; Kreuz, 2011), where coincidence is only
detected, in our algorithm it is also quantified, such that
${c}_{k}(x,y)\in[0,1]$. The dimension of the coincidence window $\tau_{k}$ is
either empirically chosen for a specific problem or it is automatically
calculated, e.g., for each pair of events $x$ and $y$ as proposed in (Kreuz,
2011), i.e.:

$\tau^{x,y}_{k}=\frac{1}{2}\dot{\min}\\{t^{i,k}_{x+1}-t^{i,k}_{x},t^{i,k}_{x}-t^{i,k}_{x-1},t^{j,k}_{y+1}-t^{j,k}_{y},t^{j,k}_{y}-t^{j,k}_{y-1}\\}$

For each class $E_{k}$, the overall coincidence $C_{k}(i|j)$ of all the events
of class $E_{k}$ in time series $TS_{i}$ with respect to the events of the
same class $E_{k}$ in time series $TS_{j}$ is computed as follows:

  * •

First, the average of the ${c}_{k}$ of each event $x$ in time series $TS_{i}$
with respect to all events in time series $TS_{j}$ is calculated;

  * •

Then the sum of the average of all the events in time series $TS_{i}$ is
taken.

That is:

(7)
${C}_{k}(i|j)=\sum\limits_{x=1}^{m_{i,k}}{\frac{1}{m_{j,k}}\bigg{[}\sum\limits_{y=1}^{m_{j,k}}{{c}_{k}(x,y)}\bigg{]}}$

Equation 7 shows that event $x$ in $TS_{i}$ can contribute to the overall
coincidence by being coincident (at different extents) with more than one
event in $TS_{j}$. Multiple coincidences are usually unwanted and the
computation of the coincidence window, as in Equation 3.1, is often performed
with the exact purpose of minimizing the likelihood of counting multiple
coincidences in one coincidence window $\tau_{k}$. Since MECS enables to
weight coincidences so that a perfect coincidence has a weight of 1.0, and the
amount of coincidence decreases along the coincidence window $\tau_{k}$, it
supports managing multiple coincidences that may indeed happen in some
application contexts. Analogously, the overall coincidence $C_{k}(j|i)$ of all
the events of class $E_{k}$ in time series $TS_{j}$ with respect to the events
of class $E_{k}$ in time series $TS_{i}$ is computed by taking the average
coincidence of each event $y$ in time series $TS_{j}$ with respect to all
events in time series $TS_{i}$ and then taking the sum of averages:

(8)
${C}_{k}(j|i)=\sum\limits_{y=1}^{m_{j,k}}{\frac{1}{m_{i,k}}\bigg{[}\sum\limits_{x=1}^{m_{i,k}}{{c}_{k}(y,x)}\bigg{]}}$

Pairwise synchronization of the events of class $E_{k}$ for the pair of time
series $\langle TS_{i}\ ;\ TS_{j}\rangle$ is computed as:

(9) ${S}_{k}(i,j)=\frac{{C}_{k}(i|j)+{C}_{k}(j|i)}{m_{i,k}+m_{j,k}}\quad
S_{k}(i,j)\in[0,1]$

Having defined the set $P\equiv\binom{T}{2}$ of all the 2-combinations of the
set $T$ (i.e., each element $p\in P$ is a distinct pair $\langle TS_{i}\ ;\
TS_{j}\rangle$, with $TS_{i}$, $TS_{j}\in T$, $i\neq j$), the overall
synchronization for the events of class $E_{k}$ is finally obtained as:

(10) $Q_{k}=\frac{1}{|P|}{\sum\limits_{p\in P}{}{S_{k}(i,j)}}\quad
Q_{k}\in[0,1]$

where the cardinality $|P|$ of set $P$ is given by the number of
2-combinations of $T$, that is:

(11) $|P|=\binom{N}{2}=\frac{N!}{2!(N-2)!}$

###  3.2. Global intra-class synchronization

To compute a global synchronization index $SI$ for the events of all classes,
we define the multi-class synchronization vector $\vec{Q}$ as:

(12) $\vec{Q}=[Q_{1},...,Q_{K}]$

$SI$ is obtained as a function of $\vec{Q}$, i.e., $SI=f(\vec{Q})$. A
straightforward choice for $f$ is the average over the $K$ components of
$\vec{Q}$. If event classes have e.g., different priorities, a set of weights
$\vec{W}=[W_{1},...,W_{K}]$ can be associated to each class of events and a
weighted synchronization index is computed as:

(13) $SI_{W}=f{(\vec{W},\vec{Q})}$

###  3.3. Inter-class synchronization

Given the set of event classes $E$, we may compute inter-class
synchronization, i.e., the synchronization between events that belong to the
different classes $E_{k1}$ and $E_{k2}$.

For each couple of time series $\langle TS_{i}\ ;\ TS_{j}\rangle$, $TS_{i}$,
$TS_{j}\in T$, $i\neq j$, the coincidence between an event $x$ found in the
first time series $TS_{i}$ and another event $y$ found in the second time
series $TS_{j}$ measured by releasing the constraint that they belong to the
same event class $E_{k}$, i.e., $x$ belongs to class $E_{k1}$ and $y$ to class
$E_{k2}$ ($E_{k1}$ and $E_{k2}$ $\in E$). The measure of how much events $x$
and $y$ are close in time is computed within a certain interval
${\tau_{k1,k2}}$ that may depend on the considered pair of classes of events.
The temporal distance $d$ between events $x$ and $y$ (see Equation 5) is
reformulated as:

(14) $d(x,y)=|{t^{i,k1}_{x}}-{t^{j,k2}_{y}}|$

Accordingly, the relative amount of coincidence $c_{k1,k2}$ (see Equation 6)
becomes:

(15) ${c}_{k1,k2}(x,y)=\begin{cases}1-\frac{d(x,y)}{\tau_{k1,k2}}&\mbox{if
\quad}0\leq d(x,y)\leq\tau_{k1,k2}\\\ 0&\mbox{otherwise}\end{cases}$

For a pair of classes $E_{k1}$ and $E_{k2}$, the overall coincidence
$C_{k1,k2}(i|j)$ of all the events of class $E_{k1}$ in time series $TS_{i}$
with respect to the events of class $E_{k2}$ in time series $TS_{j}$ and
analogously, the overall coincidence $C_{k1,k2}(j|i)$ of all the events of
class $E_{k1}$ in time series $TS_{j}$ with respect to the events of class
$E_{k2}$ in time series $TS_{i}$ are computed by:

(16)
${C}_{k1,k2}(i|j)=\sum\limits_{x=1}^{m_{i,k1}}{\frac{1}{m_{j,k2}}\bigg{[}\sum\limits_{y=1}^{m_{j,k2}}{{c}_{k1,k2}(x,y)}\bigg{]}}$

(17)
${C}_{k1,k2}(j|i)=\sum\limits_{y=1}^{m_{j,k2}}{\frac{1}{m_{i,k1}}\bigg{[}\sum\limits_{x=1}^{m_{i,k1}}{{c}_{k1,k2}(y,x)}\bigg{]}}$

Finally, inter-class pairwise synchronization of the events of class $E_{k1}$
and events of class $E_{k2}$, for the pair of time series $\langle TS_{i}\ ;\
TS_{j}\rangle$ is computed as:

(18)
${S}_{k1,k2}(i,j)=\frac{{C}_{k1,k2}(i|j)+{C}_{k1,k2}(j|i)}{m_{i,k1}+m_{j,k2}}$

and the overall synchronization for the pair of class $E_{k1}$ and $E_{k2}$
(refer to Equation 9):

(19) $Q_{k1,k2}=\frac{1}{|P|}{\sum\limits_{p\in P}{}{S_{k1,k2}(i,j)}}\quad
Q_{k1,k2}\in[0,1]$

where $P$ is the set of all the 2-combinations of the set $T$ (refer to
Equation 11), and ${S}_{k1,k2}(i,j)$ and $Q_{k1,k2}$ both belong to $[0,1]$.

##  4\. Macro classes and Macro events

Compared with previous algorithms ((Quiroga et al., 2002; Kreuz et al., 2009;
Iqbal and Riek, 2016), see Section 2) MECS is characterized by two important
extensions: Macro classes and Macro events. The first one introduces the
possibility to regroup the classes and compute the synchronization on
different levels of abstraction corresponding to a hierarchical organization
of the classes. The second extension permits to compute the synchronization
between aggregations of events belonging to different classes.

###  4.1. Macro Classes

Let’s define as $Pow(E)$ the power set of the event classes set $E$ minus the
empty set, i.e., $Pow(E)=\mathcal{P}(E)/\emptyset$.
$Pow(E)$ has cardinality $2^{K}-1$. For example, if
$E=\\{E_{1},E_{2},E_{3}\\}$, $Pow(E)$ will contain the following elements:

(20)
$\begin{split}Pow(E)=&\\{\\{E_{1}\\},\\{E_{2}\\},\\{E_{3}\\},\\{E_{1},E_{2}\\},\\\
&\\{E_{1},E_{3}\\},\\{E_{2},E_{3}\\},\\{E_{1},E_{2},E_{3}\\}\\}\end{split}$

We then define $\dot{E}\subseteq{Pow(E)}$ e.g.,
$\dot{E}=\\{\\{E_{1},E_{2}\\},\\{E_{1},E_{2},E_{3}\\}\\}$) and we consider
$\dot{E}$ as the new set of event classes, i.e., in the presented example,
$\dot{E_{1}}={\dot{E_{1}},\dot{E_{2}}}$ with $\dot{E_{1}}=\\{E_{1},E_{2}\\}$
and $\dot{E_{2}}=\\{E_{1},E_{2},E_{3}\\})$.

It is possible to take $2^{K}-1$ subsets of the original elements of $E$,
combining classes and merging them in macro classes. Each generated macro
class is actually a single class, or the combination of two or more classes of
$E$. MECS consider each item of each set in $\dot{E}$ as belonging to the same
class. Synchronization is computed using the set of equations explained in
Section 3.1 and Section 3.3 by using $\dot{E}$ as input event class set.

It is important to note that events that belong to one of the original $K$
classes can belong to more than one macro class e.g., events of class $E_{1}$
may belong to both $\dot{E_{1}}$ and $\dot{E_{2}}$ in $\dot{E}$.

Figure 1. The MECS algorithm computing the degree of synchronization of K
classes of events that have been extracted from N input signals.

###  4.2. Macro events

Events can be grouped into macro-events i.e., aggregations defined by a set of
constraints. An example of a macro-event is a sequence of events, where the
constraint to be satisfied is the order of occurrence of each event in the
sequence. By considering again $Pow(E)$, a sequence $S$ is defined as an
ordered $n$-tuple of elements (with repetitions) where each element is
referred by a sequence index: $(1,2,3,...,s-1,s)$ Starting from the elements
of $E$, examples of sequences are $S_{1}=\\{E_{2},E_{1},E_{3}\\}$,
$S_{2}=\\{E_{1},E_{1},E_{3},E_{1}\\}$ , $S_{3}=\\{E_{2},E_{1}\\}$ and so
forth.

Let’s define the following quantities:

  1. (1)

$S[i]$: i-th element of sequence $S$ that is an event class i.e., $S[i]\in E$,
$\forall i\in{1,...,s}$

  2. (2)

$t^{n,S[i]}_{e}$: the occurrence time of a generic event $e$ found in the time
series $TS_{n}$ and that belongs to the $i-th$ class of sequence $S$ i.e., any
$t^{n,S[i]}_{e}$ with $e\in 1,...,m_{S[i]}$.

  3. (3)

$IEI$: the Inter-Event Interval i.e. the maximum time allowed between two
events that belongs to two consecutive classes $S[i]$ and $S[i+1]$ of sequence
$S$ not to interrupt the sequence.

Then, within a time series $TS_{n}$, a particular sequence $S$ is detected if
the following three conditions are true:

$\forall i\in{1,...,s}$

  1. (1)

$t^{n,S[i+1]}_{e}>t^{n,S[i]}_{e}$

  2. (2)

$t^{n,S[i+1]}_{e}-t^{n,S[i]}_{e}\leq{IEI}$

  3. (3)

no other $t^{n,l}_{e}$ occurs in [$t^{n,S[i+1]}_{e}$, $t^{n,S[i]}_{e}$] where
$l\in{S}$

When a sequence $S$ is detected within a time series $TS_{n}$, it will be
treated as an event belonging to a new class named $E_{S}$. Since sequences
allows repetitions of the same elements, it is possible to define an infinite
number of sequences. Let us call $\dot{S}$ the set of all the sequences
defined starting by $E$.

The synchronization degree between sequences is computed using the set of
equations explained in Section 3.1 with $\dot{S}$ used as input event class
set.

##  5\. MECS Implementation

In this section we present an implementation of MECS algorithm. The schema
represented in Figure 1 provides a graphical representation of a sample MECS
application. Individual elements are shown with their interrelations.

For simplicity, we assume that all the input signals are being generated at
the same time and with a fixed frequency. Input signals are sampled and
streamed to the Discrete Events Generator module, that:

  * •

identifies the presence of events in the input streams and dispatches them
into classes (through event detection techniques).

  * •

generates $K$ discrete output streams $CH_{1},...,CH_{K}$ called Channels and
forward them to the Time Series Generators modules. Each channel $CH_{i}$
correspond to a single event class of the set $E$ (see Section 3).

Event detection and differentiation techniques are deliberately undefined
because they strictly depends on each specific application context. From the
The Discrete Events Generator events are sent to the Time Series Generators.
Each Time Series Generators module performs the following actions:

  * •

for each channel $CH_{i}$ fills a buffer $B_{i}$ with $buffDim$ samples taken
from the channel streams.

  * •

fills an internal matrix $M_{TS_{i}}$ ($K$ rows and $buffDim$ columns) with
the produced $K$ buffers, where $i\in{0,..,N}$.

  * •

forwards the $TS_{i}$ matrix to the MECS algorithm module.

Finally, before explaining the MECS algorithm module, let us first introduce a
set of auxiliary data structures:

  * •

$CH_{i}$: data streams used by the Discrete Events Generator that identifies
events, and dispatches them correctly, i.e., $CH_{i}$ contains events of class
$E_{i}$.

  * •

$Buffer$ $B$: represents a single portion of data. During the execution of the
algorithm, each channel stream is divided into buffers of size $buffDim$.

  * •

$M_{TS_{i}}$: matrix of channels and samples. Namely each $M_{TS_{i}}$ has $K$
rows and $buffDim$ columns. The value of each element of the matrix determines
the presence (value $\neq 0$) or absence (value $=0$) of an event.

  * •

$ECM(channel,ts)$ or Event Class Matrix: stores all the absolute positions of
all the detected events. Each element of the matrix is a list of positions.

  * •

$Sync(pair,channel)$ and $Tot_{Sync}(pair,channel)$ are internal data
structures used to store the values of ${C}_{k}(i|j)$ and ${S}_{k}(i,j)$.

The $ECM$, $Sync$ and $Tot_{Sync}$ data structure are re-initialized every
time a new buffer arrives.

Input: TODO mecs input

Output: TODO mecs output

for  _each new $Buffer$ $B$_ do

Init();

Compute();

Finalize();

$n_{buffers}$++;

end for

ALGORITHM 1 MECS computation Figure 2. Use of the Accumulator buffer
structure. Figure 3.  (A) Example of consecutive buffers with $buffDim$ = 8
and $n_{overlapped}$ = 7. (B) $Tau=4$, $n_{overlapped}=0\implies$ Merged
Buffer is bigger than the original buffer. Figure 4.  (C)
$Tau<n_{overlapped}\implies$ Merged Buffer has the same dimension of the
original buffer. (D) $Tau>n_{overlapped}\implies$ Merged Buffer is bigger than
the original buffer.

###  5.1. Initialization

In Algorithm 1, the body of the main routine of MECS is reported. The main
routine runs at every received buffer. To correctly compute the
synchronization vector, at each execution cycle, the algorithm stores a pre-
fixed number of samples taken from the last processed buffer. Such samples are
stored in a support data structure called Accumulator buffer. The dimension of
Accumulator buffer is computed according to $Tau$ and on the number of
overlapped samples $n_{overlapped}$. Possible cases are reported in Figures 3
and 4.

Input: TODO mecs input

Output: TODO mecs output

$MergBuff=createMergedBuffer(B)$;

for  _$k\in\\{CH_{1},...,CH_{K}\\}$_ do

for  _$ts\in\\{M_{TS_{1}},...,M_{TS_{N}}\\}$_ do

for  _all $sample\neq 0\in{MergBuff}$_ do

$abs_{Pos}=n_{buffers}*off_{set}+rel_{Pos}$;

$ECM(k,ts).insert(abs_{Pos})$;

end for

end for

end for

ALGORITHM 2 MECS: INIT

Next, the content of the Accumulator buffer is concatenated to the next input
buffer as shown in Figure 2 resulting in a Merged Buffer of size $mergDim$.

The initialization function _Init()_ fills the $ECM$ (Event Class Matrix) with
all the positions of all the events found in all the available channels.
Absolute positions $abs_{Pos}$ represents the occurrence timings of the events
in the whole period of execution. To compute the absolute position $abs_{Pos}$
of the detected events, the MECS algorithm uses the following quantities:

  * •

the number of received buffers $n_{buffers}$

  * •

the original dimension of the buffers $buffDim$

  * •

the dimension of the accumulator buffer $accDim$

  * •

the number of overlapping samples $n_{overlapped}$

  * •

the relative position of an event $relPos$ (the position of the sample in the
buffer $B$).

The absolute position of each event $abs_{Pos}$ is computed by using the
number of received buffers ($n_{buffers}$) and an $offset$ equals to the
difference between $mergDim$ and $n_{overlapped}$, as explained in Algorithm
2.

###  5.2. Execution

Input: TODO mecs input

Output: TODO mecs output

$MergBuff=createMergedBuffer(B)$;

for  _$k\in\\{CH_{1},...,CH_{K}\\}$_ do

forall  _$couples$ of time series $\langle tsi$,$tsj\rangle$ with $tsi\neq
tsj$_ do

forall  _$x\in ECM[k][tsi]$_ do

forall  _$y\in ECM[k][tsj]$_ do

$d$ = ComputeDist$(x,y)$;

$sync$ = $C_{k}(i|j)$;

$couple$ = $\langle tsi$,$tsj\rangle$;

$Sync[couple,k]$ += $sync$;

end forall

end forall

end forall

end for

ALGORITHM 3 MECS: COMPUTE

The Compute() routine calculates the distances between all the events found in
all the possible pairs of time series, using the absolute positions stored in
$ECM$, and saves the results in the $Sync(pair,channel)$ matrix. When this
routine completes its execution, the $Sync$ matrix stores the total
contribution to synchronization for each pair of time series $\langle
TS_{i},TS_{j}\rangle$ and for each event class $k$.

The Finalize() routine performs the following steps:

  * •

computes pairwise synchronization of the events of class $k$ for each pair of
time series $\langle TS_{i},TS_{j}\rangle$.

  * •

computes the overall synchronization for each class $k$ dividing pairwise
synchronizations ${S}_{k}$ by the set of all the 2-combinations of the
considered time series.

Input: TODO mecs input

Output: TODO mecs output

for  _$k\in\\{CH_{1},...,CH_{K}\\}$_ do

forall  _$couples$ of time series $\langle tsi,tsj\rangle$ with $tsi\neq tsj$_
do

$couple$ = $\langle min(tsi,tsj),max(tsi,tsj)\rangle$;

$Tot_{Sync}[couple,k]$ = ${S}_{k}(i,j)$;

end forall

end for

for  _$k\in\\{CH_{1},...,CH_{K}\\}$_ do

forall  _$couples$ of time series $\langle tsi,tsj\rangle$ with $tsi\neq tsj$_
do

$couple$ = $\langle min(tsi,tsj),max(tsi,tsj)\rangle$;

$Q[k]$ += $Tot_{Sync}[couple,k]$ / Comb$(N,2)$;

end forall

end for

ALGORITHM 4 MECS: FINALIZE

###  5.3. Computational complexity

The core procedure of MECS is defined in Algorithm 3, which is executed for
each input time series buffer. The size of the input data of the procedure is:
$N$ time series; $K$ channels per time series; $mergDim$ samples per channel.
Consequently, the complexity of the algorithm is:

(21) $K*(N*(N-1))/2)*mergDim^{2}$

In most applications $N,K\ll mergeDim$. Provided that, we conclude that the
complexity of the algorithm is $O(mergeDim^{5})$.

We tested the algorithm in the case study illustrated in Section 7 on a Intel
i7-6700k CPU @4 GhZ, 16 Gb, Windows 10 machine. With an input consisting of
$N=2$ time series with $K=4$ channels and $mergDim=10000$ samples the
algorithm took on average 6.8 seconds over 10 executions.

##  6\. Application on synthetic data

s We present two application examples of the MECS algorithm. We manually
construct some signals and events sequences, and we provide them as input to
the algorithm, reporting and commenting the corresponding output.

###  6.1. Inter-class synchronization between two time series

We assume that the input signal under investigation (shown in the top plot of
Figure 5) is the result of the composition of three signals:

  * •

a large sinusoidal signal with constant frequency (the second plot from top),

  * •

a smaller sinusoidal signal with decreasing frequency (the third plot from
top),

  * •

and a noise component (the fourth plot from top).

Figure 5. The input signal (highest plot) is the result of the composition of
its three components: a sine wave with constant frequency and amplitude
(second plot from top); a sine wave with constant amplitude and decreasing
frequency (third plot from top); a noise signal (lowest plot). Dots highlight
the peaks (automatically detected) of the four signals.

The aim of this example is to demonstrate how the MECS algorithm can be
exploited to find out the frequency of the main harmonic of a signal. The main
harmonic is the second signal shown from the top of Figure 5. To do that, we
propose to compute the synchronization of the input signal with a reference
signal corresponding to the input signal main harmonic (see Figure 6). We
first extract the peaks values of the three input signal components and the
reference signal: the peaks are highlighted as dots in Figure 5 and 6.
Afterwards, we create 2 time-series $TS_{1}$ and $TS_{2}$:

  * •

$TS_{1}$ contains the events of classes $E_{1}$, $E_{2}$, $E_{3}$, $E_{4}$
corresponding to the peaks of, respectively, the input signal and three
components;

  * •

$TS_{2}$ contains the events of class $E_{5}$ corresponding to the reference
signal peaks.

$TS_{1}$ and $TS_{2}$ are provided as input to the MECS algorithm, which
computes inter-class synchronization between all the pairs of classes in
$TS_{1}$ and $TS_{2}$ (that is, for each $i\in{1,2,3,4}$ it computes $Q$
between events of $E_{i}$ and $E_{5}$) and provides the values reported in
Figure 7 as output. In this example, the MECS algorithm is set up with a value
of $\tau=5$.

$0$$20$$40$$60$$80$$100$$120$$-1$$0$$1$$TS_{2}$ \- Class $E_{1}$ Figure 6. An
example reference signal. The frequency and amplitude are identical to the
first component of the input signal in Figure 5. Dots highlight the peaks of
the signal.

$0$$20$$40$$60$$80$$100$$120$$0$$1$Reference signal Vs. Input signal$Q_{1,5}$

$0$$20$$40$$60$$80$$100$$120$$0$$1$Reference signal Vs. Input signal constant
frequency component$Q_{2,5}$

$0$$20$$40$$60$$80$$100$$120$$0$$1$Reference signal Vs. Input signal
decreasing frequency component$Q_{3,5}$

$0$$20$$40$$60$$80$$100$$120$$0$$1$Reference signal Vs. Input signal noise
component$Q_{4,5}$

Figure 7. Synchronization results between the 3 signal components and the
input signal reported in Figure 5 and the reference signal reported in Figure
6. The four synchronization values are computed at the same time by applying
the MECS algorithm.

As illustrated in Figure 7, the synchronization between the input signal and
the reference signal (the first plot from top) exhibits a rhythmical (quasi-
periodic) pattern with a high constant amplitude. The same exists for the
first component, which is identical to the reference signal. Conversely, the
second and third components of the input signal, that is, the signal with a
variable frequency and the noise component, do not exhibit the same kind of
pattern.

###  6.2. Macro events synchronization between two time series

In this second example we apply MECS to two input time series $TS_{1}$ and
$TS_{2}$ consisting of events belonging to three classes (i.e., $E_{1}$,
$E_{2}$, $E_{3}$) (see Figure 8). Such events are represented within the time
series by the corresponding positive integer numbers $1,2,3$, while, when no
event is triggered, a value of zero is set. For example, $TS_{2}$ in Figure 8
starts with an event of class $E_{1}$, then no events are triggered for the
second sample of the time series, then an event of class $E_{2}$ is triggered,
and so on.

We set up the MECS algorithm to detect synchronization of macro events
(sequences) consisting of an event of class $E_{1}$, followed by an event of
class $E_{2}$, followed by an event of class $E_{3}$ (the concept of sequence
has been introduced in Section 4.2). We chose a value of $\tau$ is $20$. The
output of the algorithm, illustrated in Figure 9, is the amount of
synchronization between $S$ = $\\{E_{1},E_{2},E_{3}\\}$ events in the two
input time series. The Figure shows that the outputs are non-discrete values,
depending on the distance.

$0$$10$$20$$30$$40$$50$$0$$1$$2$$3$$TS_{1}$

$0$$10$$20$$30$$40$$50$$0$$1$$2$$3$$TS_{2}$

Figure 8. MECS applied to two time series. Macro events are triggered when the
following sequence of events $S$ = $\\{E_{1},E_{2},E_{3}\\}$ is detected (see
Section 4.2). The three events must occur in the specified order (in this case
any amount of samples with no events can appear between them, i.e.,
$IEI=\infty$). Detected macro events are highlighted in the figure with dashed
boxes enclosing each single event that contributes to the final sequence. For
example, the first macro event found in $TS_{1}$ (in the top left corner of
the figure) consists of the event of class $E_{1}$ followed by three samples
with no events, then the event of class $E_{2}$, another empty sample and
finally an event of class $E_{3}$. $0$$10$$20$$30$$40$$50$$0$$1$$Q_{S}1$
Figure 9. MECS algorithm output, between $TS_{1}$ and $TS_{2}$, with $\tau=20$
samples.

##  7\. A Case Study

We describe an example of application of MECS to measure multi-modal, intra-
personal synchronization between respiration phases and kinetic energy
captured while performing body movements characterized by different expressive
qualities.

Taking a breath is a physical action that can influence the body movements
performed at the same time. Similarly, body movements expressing abrupt
changes of velocity and acceleration can influence the respiration pattern.
Rhythm of respiration synchronizes with repetitive motor activities such as
running (Bernasconi and Kohl, 1993; Hoffmann et al., 2012), or rowing (Bateman
et al., 2006). Moreover, respiration plays an important role in learning
physical activities, such as yoga or tai-chi.

###  7.1. Definitions of expressive qualities

We analyze synchronization on multi-modal data of the dancers performing
different expressive qualities. During a performance, dancers are used to
display a large variety of expressive qualities. As in (Lussu et al., 2016),
we focus on movements displaying two particular expressive qualities: Fluidity
and Impulsivity.

A fluid movement is characterized by the following properties: (i) the
movements of every single body joint are smooth; (ii) energy is free to
propagate along the kinematic chains of the body (e.g., from head to trunk,
from shoulders to arms) according to a coordinated wave-like propagation
(Alborno et al., 2016; Piana et al., 2016).

An impulsive movement is characterized by a sudden and non-predictable change
of velocity, which is usually produced without exhibiting a preparation phase
(Niewiadomski et al., 2015). Examples of impulsive movements are: avoidance
movements (e.g., when hearing a sudden and unexpected noise) and movements
made to recover from a loss of balance. That is, quick and repetitive
movements are not impulsive.

While in classic dance (e.g., classic ballet) fluidity is a major expressive
quality, impulsive movements are more important for contemporary dance artists
(see for example the choreographies of Sagi Gross’s Dance
Company111http://www.grossdancecompany.com).

###  7.2. Hypothesis

Imagine a dancer mostly dancing fluidly and suddenly displaying one or more
very impulsive movements. Figure 10 shows two examples of multi-modal data
that may be captured in such a situation. Figure 10-A is an example of
impulsive movement, and Figure 10-B is a fluid movement performed by the same
dancer. The two signals refer to movement energy, computed as the kinetic
energy of the whole body, and to respiration, computed as the energy of the
audio signal captured by a microphone (that was located near to the dancer’s
mouth). Energy peaks of the audio respiration signal and those of the kinetic
energy are closer in Figure 10-A than in Figure 10-B. This is reasonable, as
we may expect that different modalities may be more synchronized at the time
of an impulse.

A B

Figure 10. Two parts of the same dance performances: A) impulsive movements
and B) fluid movements.

The respiration rhythm is interrupted by the impulse: a new respiration phase
starts and it is synchronized with the kinetic energy peak caused by a sudden
increase of velocity. Formally, we hypothesize that during an impulsive
movement there is a single strong peak in the respiration signal that appears
immediately after the beginning of the respiration phase and which is
synchronized with a similar peak in kinetic body energy. Such coincidence may
depend on the respiration phase and it is not observed for fluid movements.

###  7.3. Input

We applied the MECS algorithm on 90 seconds of a recordings of a female
professional dancer (see (piana) for more details about the recording setup).
The dancer was asked to move fluidly most of the time, and to perform several
impulsive movements in between. The data consist of audio of the respiration
captured using a wireless microphone (mono, 48kHz), two accelerometers (xOSC
222http://x-io.co.uk/x-osc) placed on the arms of the dancer, and 2 video
cameras (1280x720, at 50fps).

###  7.4. Features extraction

####  7.4.1. Respiration Energy (RE)

The audio stream was segmented in frames of 1920 samples. The instantaneous
energy of the audio signal was computed on each single frame using Root Mean
Square (RMS). Next, we extracted the envelope of the instantaneous audio
energy using an 8-frames buffer.

####  7.4.2. Kinetic Energy (KE)

Kinetic energy was computed from the data captured by the two accelerometers
placed on the dancer’s arms. Velocity was obtained by integrating the values
obtained from the accelerometers. Next, we computed the average kinetic energy
by taking the mean value of the kinetic energies obtained from the data coming
from the two accelerometers. The accelerometer signals and the kinetic energy
obtained thereof were sampled at 50fps. To keep synchronization with the audio
energy signal, every second value was taken.

###  7.5. Manual annotation

####  7.5.1. Expressive Quality Annotation (EQA)

An expert in expressive movement qualities annotated the beginning of each
segment of the video where impulsive or fluid movements can be perceived.
$EQA$ was used to distinguish peaks and respiration phases belonging to
segments of impulsive and fluid movements.

####  7.5.2. Respiration Phase Annotation (EX/IN)

Although the inspiration and expiration phases can be automatically extracted
(see as an example (Yahya and Faezipour, 2014)) from the audio signal, to
ensure high precision of the segmentation we performed manual annotation. The
audio signal was annotated by an expert who used the Audacity
software333http://www.audacityteam.org to assign the start and end time of
each occurrence of each inspiration and expiration phase detected in the audio
signal.

###  7.6. Applying MECS

Our aim is to understand if the synchronization between a single respiration
phase’s energy and the kinetic energy of the body, grows in correspondence of
impulsive behaviors given the two respiration phases.

Figure 11. Synchronization between respiration and kinetic energy: the four
time series of the extracted features before sequence detection.

In order to apply the MECS algorithm peaks, were extracted the kinetic energy
signal $KE$ and the respiration energy signal $RE$. Following the $EQA$
annotations, a set of $N=4$ time series $T=\\{TS_{1},TS_{2},TS_{3},TS_{4}\\}$
was obtained, such that:

  * •

$TS_{1}$ contains a value different than zero in correspondence of peaks in
the kinetic energy signal $KE$ during an fluid movement.

  * •

$TS_{2}$ contains a value different than zero in correspondence of:

    * –

peaks in the respiration energy signal $RE$,

    * –

the start time of an inspiration phase (i.e., an IN annotation),

    * –

the start time of an expiration phase (i.e., an EX annotation) during a fluid
movement.

  * •

$TS_{3}$ is the equivalent of $TS_{1}$ performed during an impulsive movement.

  * •

$TS_{4}$ is the equivalent of $TS_{2}$ performed during an impulsive movement.

The MECS algorithm was then applied to measure inter-class synchronization
between $K=3$ classes of events, i.e., $E=\\{E_{1},E_{2},E_{3}\\}$, in the
time series belonging to $T$. The three classes were defined as follows:

  * •

$E_{1}$ is the class of the events consisting of peaks in the kinetic energy
signal $KE$. These events are contained in time series $TS_{1}$ and $TS_{3}$.

  * •

$E_{2}$ is the class of the macro-events, in the respiration energy signal
$RE$, consisting of the following sequence $Seq=(IN,RE)$, i.e., the beginning
of an inspiration is nearly immediately followed by a peak in the respiration
energy signal $RE$. These sequences can be detected in time series $TS_{2}$
and $TS_{4}$.

  * •

$E_{3}$ is the class of the macro-events, in the respiration energy signal
$RE$, consisting of the following sequence $Seq=(EX,RE)$, i.e., the beginning
of an inspiration is nearly immediately followed by a peak in the respiration
energy signal $RE$. These sequences can be detected in time series $TS_{2}$
and $TS_{4}$.

To model the occurrence of a peak in the respiration energy signal $RE$ nearly
immediately after the beginning of a respiration phase $IN$ or $EX$, we tuned
the $IEI$ parameter in the conditions that events should satisfy for a
sequence to be detected (see Section 4.2).

We ran MECS to compute: $C_{1}=S_{1,2}(1,2)$, $C_{2}=S_{1,3}(1,2)$,
$C_{3}=S_{1,2}(3,4)$, $C_{4}=S_{1,3}(3,4)$.

MECS parameters were given the following values: $\tau_{k_{1},k_{2}}=15$
samples (18ms) for every pair $(k_{1},k_{2})$ of event classes, $IEI=25$
samples (50ms). For each condition $C_{h}$, time series $TS_{1}$, $TS_{2}$,
$TS_{3}$ and $TS_{4}$ were divided into $N_{b}$ buffers and, for each buffer,
the synchronization value $S_{k_{1},k_{2}}$ was computed, and finally it was
used to compute an overall synchronization score.

For each condition $C_{1}$ \- $C_{4}$, we count the number of times inter-
class pairwise synchronization $S_{k_{1},k_{2}}(i,j)>0$ (i.e., the number of
times at least a partial synchronization is detected) and we normalize such a
number by dividing it by the number of occurrences of the respiration phase
events related to (i.e., inspiration of $C_{1}$ and $C_{3}$, and expiration of
$C_{2}$ and $C_{4}$). That is, the final synchronization score for condition
$C_{h}$, $h=1...4$ is obtained as:

(22) $\displaystyle
Q_{C_{h}}=\frac{1}{Y}\sum\limits_{t=1}^{N_{b}}{\Theta(S_{k_{1},k_{2}}(i,j))}$

where: $i$ and $j$, and $k_{1}$ and $k_{2}$ are the indexes of the time series
$TS_{i}$ and $TS_{j}$ and belong to the event classes $E_{k_{1}}$ and
$E_{k_{2}}$ involved in condition $C_{h}$, respectively; $\Theta(z)$ is the
Heaviside function; $Y$ is the number of occurrences of the respiration phase
(inspiration or expiration) involved in condition $C_{h}$.

It is important to notice that the original ES algorithm is not powerful
enough to model these complex relationship between modalities. In detail, if
we want to analyze sequences of peaks in two modalities (respiration audio
energy and kinetic energy), which should be treated as a single event, if and
only if such peaks appear in a sufficiently “short” time window. The MECS
algorithm allows us to detect sequences and to analyze their development along
time. Moreover, MECS also allows us to distinguish two different classes of
events related to respiration, i.e., inspiration and expiration. In this way,
we are able to check whether the sequence of peaks in impulsive movements
happens for any respiration phase, or it is related to one of the two
respiration phases.

###  7.7. Results

The number of inhalations and exhalations in the analyzed trial was similar
(45 vs. 48). The number of inspiration intervals was lower for impulsive
movements than for fluid ones (12 vs. 33). When considering the results the
MECS algorithm provided for impulsive movements (conditions $C_{3}$ and
$C_{4}$), the total number of the synchronized events was much higher for the
expiration phases than for the inspiration phases (13 vs. 7), but the
normalized score for both respiration phases is similar with $Q_{C_{3}}=0.58$
and $Q_{C_{4}}=0.65$. When comparing these results with those obtained for
fluid segments the difference was big such that only few synchronized fluid
segments were observed for the inspiration phase with $Q_{C_{1}}=0.12$, and
even less for the expiration phase $Q_{C_{2}}=0.04$. Repeating the same
procedure with different values of $\tau_{k_{1},k_{2}}$ showed similar results
(see Figure 12).

Figure 12. Synchronization results for $\tau_{k_{1},k_{2}}$ equal to 10, 15,
and 20 samples (at 50fps, the correspond to 12, 18 and 24 milliseconds). The
same value is used for every pair of event classes $(k_{1},k_{2})$.

##  8\. Conclusion

In this paper, we presented Multi Event Class Synchronization (MECS), a novel
algorithm for event synchronization analysis. With respect to other techniques
existing in the literature, MECS introduces the possibility to measure
synchronization between different classes of events that are detected in
multiple time series, providing different synchronization measures (i.e., a
synchronization measure for each class, a global synchronization measure,
synchronization measures obtained by combining classes). Furthermore we
described how to operate on the event set, introducing the concept of macro
classes and macro events and we proposed a possible implementation. Finally,
we illustrated how MECS has been applied to artificially generated and real-
life data.

###### Acknowledgements.

This research has received funding from the European Union Horizon 2020
research and innovation programme.

## References

  * (1)
  * Alborno et al. (2016) Paolo Alborno, Stefano Piana, Maurizio Mancini, Radoslaw Niewiadomski, Gualtiero Volpe, and Antonio Camurri. 2016.  Analysis of Intrapersonal Synchronization in Full-Body Movements Displaying Different Expressive Qualities. In _Proceedings of the International Working Conference on Advanced Visual Interfaces_ _(AVI ’16)_. ACM, New York, NY, USA, 136–143.  https://doi.org/10.1145/2909132.2909262
  * Bateman et al. (2006) AH Bateman, AH McGregor, AMJ Bull, PMM Cashman, and RC Schroter. 2006.  Assessment of the timing of respiration during rowing and its relationship to spinal kinematics.  _Biology of Sport_ 23 (2006), 353–365.
  * Bernasconi and Kohl (1993) Paolo Bernasconi and Jana Kohl. 1993.  Analysis of co-ordination between breathing and exercise rhythms in man.  _J. Physiol_ 471 (1993), 693–706.
  * Hoffmann et al. (2012) Charles P. Hoffmann, G?rald Torregrosa, and Beno?t G. Bardy. 2012.  Sound Stabilizes Locomotor-Respiratory Coupling and Reduces Energy Cost.  _PLoS ONE_ 7, 9 (09 2012).
  * Hung and Gatica-Perez (2010) Hayley Hung and Daniel Gatica-Perez. 2010.  Estimating Cohesion in Small Groups Using Audio-Visual Nonverbal Behavior.  _IEEE Transactions on Multimedia_ 12, 6 (Oct 2010), 563–575.  https://doi.org/10.1109/TMM.2010.2055233
  * Iqbal and Riek (2016) Tariq Iqbal and Laurel D. Riek. 2016.  A Method for Automatic Detection of Psychomotor Entrainment.  _IEEE Transactions on Affective Computing_ 7, 1 (Jan 2016), 3–16.  https://doi.org/10.1109/TAFFC.2015.2445335
  * Keltner (1995) Dacher Keltner. 1995\.  Signs of appeasement: Evidence for the distinct displays of embarrassment, amusement, and shame.  _Journal of Personality and Social Psychology_ 68 (1995), 441–454.
  * Kreuz (2011) Thomas Kreuz. 2011\.  Measures of spike train synchrony.  _Scholarpedia_ 6, 10 (2011), 11934.
  * Kreuz et al. (2009) Thomas Kreuz, Daniel Chicharro, Ralph G Andrzejak, Julie S Haas, and Henry DI Abarbanel. 2009\.  Measuring multiple spike train synchrony.  _Journal of neuroscience methods_ 183, 2 (2009), 287–299.
  * Lakens and Stel (2011) Daniel Lakens and Marielle Stel. 2011.  If They Move in Sync, They Must Feel in Sync: Movement Synchrony Leads to Attributions of Rapport and Entitativity.  _Social Cognition_ 29 (2011), 1–14.
  * Lussu et al. (2016) Vincenzo Lussu, Radoslaw Niewiadomski, Gualtiero Volpe, and Antonio Camurri. 2016.  Using the Audio Respiration Signal for Multimodal Discrimination of Expressive Movement Qualities.  In _Human Behavior Understanding: 7th International Workshop, HBU 2016, Amsterdam, The Netherlands, October 16, 2016, Proceedings_ , Mohamed Chetouani, Jeffrey Cohn, and Albert Ali Salah (Eds.). Springer International Publishing, 102–115.
  * Niewiadomski et al. (2011) Radoslaw Niewiadomski, Sylwia Julia Hyniewska, and Catherine Pelachaud. 2011.  Constraint-Based Model for Synthesis of Multimodal Sequential Expressions of Emotions.  _Affective Computing, IEEE Transactions on_ 2, 3 (2011), 134–146.  https://doi.org/10.1109/T-AFFC.2011.5
  * Niewiadomski et al. (2015) Radoslaw Niewiadomski, Maurizio Mancini, Gualtiero Volpe, and Antonio Camurri. 2015.  Automated Detection of Impulsive Movements in HCI. In _Proceedings of the 11th Biannual Conference on Italian SIGCHI Chapter_ _(CHItaly 2015)_. ACM, New York, NY, USA, 166–169.
  * Piana et al. (2016) Stefano Piana, Paolo Coletta, Simone Ghisio, Radoslaw Niewiadomski, Maurizio Mancini, Roberto Sagoleo, Gualtiero Volpe, and Antonio Camurri. 2016.  Towards a Multimodal Repository of Expressive Movement Qualities in Dance. In _Proceedings of the 3rd International Symposium on Movement and Computing_ _(MOCO ’16)_. ACM, New York, NY, USA, Article 10, 8 pages.  https://doi.org/10.1145/2948910.2948931
  * Quiroga et al. (2002) Rodrigo Quian Quiroga, Thomas Kreuz, and Peter Grassberger. 2002\.  Event synchronization: a simple and fast method to measure synchronicity and time delay patterns.  _Physical review E_ 66, 4 (2002), 041904.
  * Yahya and Faezipour (2014) Omar Yahya and Miad Faezipour. 2014.  Automatic detection and classification of acoustic breathing cycles. In _Proceedings of the 2014 Zone 1 Conference of the American Society for Engineering Education_. 1–5.  https://doi.org/10.1109/ASEEZone1.2014.6820648