11institutetext: Process and Data Science, RWTH Aachen University, Aachen,
Germany
11email<EMAIL_ADDRESS>22institutetext: Fraunhofer Institute for Applied Information Technology
33institutetext: Information Systems Research, University of Freiburg,
Freiburg, Germany
33email<EMAIL_ADDRESS>
# A Framework for Extracting and Encoding Features from Object-Centric Event
Data

Jan Niklas Adams 11 Gyunam Park 11 Sergej Levich 33 Daniel Schuster 2211 Wil
M.P. van der Aalst 1122

###### Abstract

Traditional process mining techniques take event data as input where each
event is associated with exactly one object. An object represents the
instantiation of a process. Object-centric event data contain events
associated with multiple objects expressing the interaction of multiple
processes. As traditional process mining techniques assume events associated
with exactly one object, these techniques cannot be applied to object-centric
event data. To use traditional process mining techniques, the object-centric
event data are flattened by removing all object references but one. The
flattening process is lossy, leading to inaccurate features extracted from
flattened data. Furthermore, the graph-like structure of object-centric event
data is lost when flattening. In this paper, we introduce a general framework
for extracting and encoding features from object-centric event data. We
calculate features natively on the object-centric event data, leading to
accurate measures. Furthermore, we provide three encodings for these features:
tabular, sequential, and graph-based. While tabular and sequential encodings
have been heavily used in process mining, the graph-based encoding is a new
technique preserving the structure of the object-centric event data. We
provide six use cases: a visualization and a prediction use case for each of
the three encodings. We use explainable AI in the prediction use cases to show
the utility of both the object-centric features and the structure of the
sequential and graph-based encoding for a predictive model.

###### Keywords:

Object-Centric Process Mining Machine Learning Explainable AI.

##  1 Introduction

Figure 1: An object-centric event log and the underlying structure of events.
The left-hand side depicts the event log. Events may be associated with
multiple objects of different object types (here: Order and Item). The right-
hand side shows the graph of directly-follows relationships for the events
given by the objects. An event with multiple objects may have multiple
predecessor events.

Process mining [1] is a branch of computer science producing data-driven
insights and actions from event data generated by processes. These insights
are typically grouped into three categories: process discovery, conformance
checking, and enhancement. Process discovery techniques create process models
describing the possible paths of actions in a process. Conformance checking
techniques quantify and qualify the correspondence between a process model and
event data. Process enhancement techniques take an encoding of features of the
event data as input and deliver insights, predictions, or actions as output.
Such enhancement techniques include process performance analysis [20, 24],
prediction [10, 26, 29] or clustering of similar process executions [25].

Generally, process enhancement techniques encode features of event data in
either of two ways: as a table [18, 9] or as a set of sequences [12, 26, 19].
In a tabular encoding, each row corresponds to feature values for, e.g., an
event. This tabular encoding is used, for example, for regression, decision
trees, and feed-forward neural networks. However, each process execution
(also: case) is a timely ordered sequence of events. Therefore, summarizing
event data to tabular encoding removes the sequential structure of the event
data. Since this structure itself is meaningful, sequential encodings were
developed [19]. These encodings represent each process execution as a sequence
of feature values and are used for predictive models considering sequentially
encoded data, such as LSTMs [26], or to visualize the variant of the process
execution.

Traditional process mining builds on two central assumptions: Each event is
associated with exactly one object (the case) and each object is of the same
type. Each object is associated with a sequence of events. A traditional event
log, therefore, describes a collection of homogeneously typed, isolated event
sequences. This is a valid assumption when analyzing, e.g., the handling of
insurance claims. In this example, each object describes an instantiation of
the same type: an insurance claim. Events are associated to exactly one
insurance claim. However, real-life information systems often paint another
picture: Events may be related to multiple objects of different types [3, 4,
11, 28]. The most prominent example of information systems generating event
data with multiple associated objects are ERP systems. Objects in such systems
would correspond to, e.g., an order, different items of this order, and
invoices in an order-to-cash process. Consider the simplified example of an
order handling process depicted in Fig. 1. An event may be related to objects
of type order, item, or both. An event with multiple objects may have multiple
predecessor events. Therefore, the structure of an Object-Centric Event Log
(OCEL) resembles a graph, not a sequential structure as is assumed in
traditional process mining.

Figure 2: Flattening an object-centric event log (cf. Fig. 1) such that it can
be used for traditional process enhancement techniques. The event log is
transformed into sequences of a chosen case notion. Due to deficiency,
convergence and divergence, the features calculated on a flattened log might
be misleading, e.g., through missing events. Furthermore, the graph-like
structure of the original event log is lost.

This gap between OCELs and traditional process enhancement techniques is
currently bridged by flattening an event log [2], i.e., mapping an OCEL into
traditional event log format by enforcing a homogeneous, sequential structure.
This involves two steps: Choosing a case notion and duplicating events with
multiple objects of that notion. All objects not included in this case notion
are discarded. Flattening the event log of Fig. 1 is depicted in Fig. 2 for
three different case notions. The first two are case notions of a single
object type [2]. The third case notion is a composite case notion of co-
appearing orders and items, i.e., an order and all corresponding items. The
flattened event data may be used as input for traditional process enhancement
techniques.

However, flattening manipulates the information of the object-centric event
log. The problems related to flattening are deficiency (disappearing events)
[3], convergence (duplicated events) [2] and divergence (misleading directly-
follows relations) [28, 2]. We showcase divergence using an example. One might
use a composite case notion of order and item to flatten the event log (cf.
Fig. 2 Case Notion: Order & Item). All orders and items related through events
form one composite object, i.e., o1,i1,i2, and o2,i3. The events of these
objects are flattened to one sequence, introducing inaccurate precedence
constraints. E.g., events $e_{3}$ and $e_{4}$, which describe an item being
picked, are now sequentially ordered, indicating some order between them.
However, the original event data show that these two picking events are
independent. The same holds for the relationship of pick item and pay order:
The object-centric event data do not indicate any precedence constraint.
However, the sequential representation enforces one.

Figure 3: Our framework enables accurate feature extraction for object-centric
event data. Furthermore, we provide three encodings for object-centric
features: tabular, sequential, and graph-based. We present a visualization and
prediction use case for each encoding.

The three problems of flattening have major consequences on the quality of the
calculated features of the flattened OCEL: Due to missing events, duplicated
events, or wrong precedence constraints, many features deliver incorrect
results (cf. Sec. 4). Furthermore, the tabular or sequential encoding
constructed from these features does not preserve the graph-like structure of
the event data, removing important structural information. Therefore, features
for OCELs can not accurately be extracted and encoded.

To solve the previously mentioned problem, an approach is necessary that
calculates features natively on the object-centric event data and enables a
graph-based encoding preserving the actual structure of the event log. In this
paper, we introduce a general framework for extracting and encoding features
for object-centric event data (cf. Fig. 3), providing two contributions: 1) We
translate the computation of the features introduced in the framework of de
Leoni et. al [18] to the object-centric setting, providing accurate measures.
2) We provide three different encodings to represent the extracted features
for different algorithms and methods: tabular, sequential, and graph-based.
Using features and encoding, we provide six use cases. These use cases
showcase the generalizability of our framework to a plethora of different
tasks. We use one visualization and one prediction use case for each encoding.
In the prediction use cases, we depict how the different features and the
structure of the encodings are utilized by predictive models, leveraging on
explainable AI and SHAP values [21]. These contributions may be used as a
foundation for new algorithms, new visualizations, new machine learning
models, more accurate predictions, and more.

This paper is structured as follows. First, we discuss related work on feature
extraction and encoding in Sec. 2. We introduce object-centric event data and
process executions in Sec. 3. In Sec. 4, we provide an overview of native
feature calculation for object-centric event data. In Sec. 5, we define three
encodings for object-centric features. Sec. 6 depicts our six use cases for
features and their encodings. We conclude this paper in Sec. 7.

##  2 Related Work

Table 1: Process enhancement techniques and supporting frameworks.

| Feature Extraction | Feature Encoding |  Existing work
---|---|---|---
Object-centric | Flattened | Tabular | Sequential | Graph-based
(a) Process enhancement techniques | P1 |  | ✓ | ✓ |  |  | [9, 13]
P2 |  | ✓ |  | ✓ |  | [19, 12, 26]
P3 |  | ✓ |  |  | ✓ | [23, 27, 16]
(b) Frameworks | F1 |  | ✓ | ✓ |  |  | [18]
F2 |  | ✓ |  | ✓ |  | [7, 14]
This paper | ✓ |  | ✓ | ✓ | ✓ |

A plethora of process enhancement techniques exist in the literature,
including process performance analysis, predictive process monitoring, and
trace clustering [1]. Such techniques use encoded features extracted from an
event log as input. Table 1(a) shows three categories of techniques using
different feature extraction (i.e., feature extractions using 1. OCELs and 2.
flattened event logs) and encoding (i.e., 1. tabular, 2. sequential, and 3.
graph encoding) approaches with representative examples. First, P1 represents
the techniques using features extracted from flattened event logs and encoded
as tabular formats. For instance, van Dongen et al. [9] use tabular encoding
by transforming an event log into feature-outcome pairs to predict remaining
times using non-parametric regression. Also, in [13], an event log is encoded
into a tabular format with additional features on context, e.g., resource
availability, to predict processing times. Second, techniques in P2 also use
features based on flattened event logs but encoded as sequential formats.
Leontjeva et al. [19] propose _complex sequence encoding_ to encode an event
log to sequences to predict the outcome of an ongoing case. To predict the
next activity of an ongoing case, Evermann et al. [12] encode control-flow
features using _embedding_ techniques, whereas Tax et al. [26] use _one-hot
encoding_. Finally, P3 consists of techniques using features extracted from
flattened event logs and encoded as graph formats. Philipp et al. [23] encode
an event log to a graph where each node represents an activity, and each edge
indicates the relationship between activities. The graph is used to learn a
_Graph Neural Network (GNN)_ to predict process outcomes. Venugopal et al.
[27] extend [23] by annotating nodes with temporal features. They use GNNs to
predict the next activity and next timestamp of an event. Instead of
representing a node as an activity, Harl et al. [16] uses one-hot encoding of
an activity to represent a node to deploy _gated graph neural network_ that
provides the explainability based on _relevance score_.

Furthermore, to support the development of process enhancement techniques
using different feature extraction and encoding, several frameworks have been
proposed (cf. Table 1(b)). First, De Leoni [18] in F1 suggest a framework to
compute features using flattened event logs and encode them to tables. Second,
Becker et al. [7] and Di Francescomarino et al. [14] in F2 propose frameworks
for techniques for sequentially encoding extracted features. To the best of
our knowledge, no framework supporting graph encoding exists.

Despite the limitations of flattened event logs to extract misleading
features, no study has been conducted to develop process enhancement
techniques using features based on OCELs. In this work, we provide a framework
for extracting and encoding features based on OCELs, with the goal of
facilitating the development of object-centric process enhancement approaches.
Our proposed framework supports all existing encoding formats, i.e., tabular,
sequential, and graph, to be used for different algorithms and methods.

##  3 Object-Centric Event Data

Given a set $X$, the powerset $\mathcal{P}(X)$ denotes the set of all possible
subsets. A sequence $\sigma:\\{1,\ldots,n\\}\rightarrow X$ of length
$\mathit{len}(\sigma)=n$ assigns order to elements of $X$. We denote a
sequence with $\sigma=\langle x_{1},\ldots,x_{n}\rangle$ and the set of all
sequences over $X$ with $X^{*}$. We overload the notion $x\in\sigma$ to
express $x\in\mathit{range}(\sigma)$.

A graph is a tuple $G=(V,E)$ of nodes $V$ and edges $E\subseteq V\times V$.
The set of all subgraphs of $G$ is given by
$\mathit{sub}(G)=\\{(V^{\prime},(V^{\prime}\times V^{\prime})\cap E)\mid
V^{\prime}\subseteq V\\}$. A path connects two distinct nodes through edges.
The set of paths between two nodes $v,v^{\prime}\in V,v\neq v^{\prime}$ is
defined by
$\mathit{path}_{G}(v,v^{\prime})=\\{\langle(v,v_{1}),(v_{1},v_{2}),\ldots,(v_{k},v^{\prime})\rangle\in
E^{*}\\}$. Two distinct nodes are connected if the set of paths between them
is not empty $\mathit{path}_{G}(v,v^{\prime})\neq\emptyset$. The distance
between two nodes is the length of the shortest path
$\mathit{dist}_{G}(v,v^{\prime})=len(\sigma_{d})$ such that
$\sigma_{d}\in\mathit{path}_{G}(v,v^{\prime})\wedge\neg\exists_{\sigma_{d}^{\prime}\in\mathit{path}_{G}(v,v^{\prime})}\
\allowbreak len(\sigma_{d})\allowbreak>len(\sigma_{d}^{\prime})$. A graph
$G=(V,E)$ is connected iff a path exists between all edges
$\forall_{v,v^{\prime}\in V}\ v\neq
v^{\prime}\wedge\mathit{path}_{G}(v,v^{\prime})\neq\emptyset$. The set of
connected subgraphs of $G=(V,E)$ is defined as follows
$\mathit{consub}(G)=\\{G^{\prime}\in\mathit{sub}(G)\mid G^{\prime}\text{ is
connected}\\}$.

An event log is a collection of events associated with objects. Each event
contains an activity, describing the executed action, a start and complete
timestamp and additional attributes. Each object is associated to a sequence
of events.

######  Definition 1 (Event Log)

Let $\mathcal{E}$ be the universe of events, $\mathcal{O}$ be the universe of
objects, $\mathcal{OT}$ be the universe of object types, $\mathcal{A}$ be the
universe of activities, $\mathcal{C}$ be the universe of attributes and
$\mathcal{V}$ be the universe of attribute values. Let $A\subseteq\mathcal{A}$
be a set of activities and $C\subseteq\mathcal{C}$ be a set of attributes.
Each object is mapped to exactly one object type
$\pi_{type}:\mathcal{O}\rightarrow\mathcal{OT}$. An event log
$L=(E,O,OT,\pi_{\mathit{ct}},\pi_{\mathit{st}},\pi_{\mathit{trace}},\pi_{\mathit{act}},\pi_{\mathit{att}})$
is a tuple composed of

  * $\bullet$

events $E\subseteq\mathcal{E}$, objects $O\subseteq\mathcal{O}$, and object
types $OT\subseteq\mathcal{OT}$,

  * $\bullet$

two time mappings for the completion
$\pi_{\mathit{ct}}:E\rightarrow\mathbb{R}$ and the start
$\pi_{\mathit{st}}:E\rightarrow\mathbb{R}$ of an event such that
$\pi_{\mathit{st}}(e)\leq\pi_{\mathit{ct}}(e)$ for any $e\in E$,

  * $\bullet$

a mapping $\pi_{\mathit{trace}}:O\rightarrow E^{*}$ mapping each object to a
sequence of events such that $\forall_{o\in O}\
\pi_{\mathit{trace}}(o)=\langle
e_{1},\ldots,e_{n}\rangle\wedge\forall_{i\in\\{1,\ldots n-1\\}}\
\pi_{\mathit{ct}}(e_{i})\leq\pi_{\mathit{ct}}(e_{i+1})$,

  * $\bullet$

an activity mapping $\pi_{\mathit{act}}:E\rightarrow A$ and,

  * $\bullet$

an attribute mapping $\pi_{\mathit{att}}:E\times C\nrightarrow\mathcal{V}$.

The table in Fig. 1 depicts an example of an OCEL. A row corresponds to one
event. Sorting the events of an object in timely order, we retrieve the event
sequence for the object, e.g.,
$\pi_{\mathit{trace}}(\text{i3})=\langle\text{place
order},\allowbreak\text{pick item},\allowbreak\text{send
delivery},\allowbreak\text{delivery received}\rangle$. The relationships
between objects can be expressed in the form of a graph, connecting objects
that share events.

######  Definition 2 (Object Graph)

Let
$L=(E,O,OT,\pi_{\mathit{ct}},\pi_{\mathit{st}},\pi_{\mathit{trace}},\pi_{\mathit{act}},\pi_{\mathit{att}})$
be an event log. We denote the objects of an event $e{\in}E$ with
$\pi_{\mathit{obj}}(e){=}\\{o{\in}O\mid e{\in}\pi_{\mathit{trace}}(o)\\}$. The
object graph $\mathit{OG}_{L}{=}(O,I)$ is an undirected graph of nodes $O$ and
edges of object interactions $I{=}\\{\\{o,o^{\prime}\\}{\subseteq}O\mid
o{\neq}o^{\prime}\wedge\exists_{e\in E}\
\\{o,o^{\prime}\\}\subseteq\pi_{\mathit{obj}}(e)\\}$.

Objects which are directly or transitively connected in the object graph
depend on each other by sharing events. In traditional process mining, a
process execution (case) is the event sequence of one object. We use the
definitions of process executions [6] and generalize this notion such that a
process execution is the set of events for multiple, connected objects.

######  Definition 3 (Process Execution)

Let
$L=(E,O,OT,\pi_{\mathit{ct}},\pi_{\mathit{st}},\pi_{\mathit{trace}},\pi_{\mathit{act}},\allowbreak\pi_{\mathit{att}})$
be an event log and $\mathit{OG}_{L}=(O,I)$ be the corresponding object graph.
A process execution $p=(O^{\prime},E^{\prime})$ is a tuple of objects
$O^{\prime}\subseteq O$ and events $E^{\prime}\subseteq E$ such that
$e^{\prime}\in
E^{\prime}\Leftrightarrow\pi_{\mathit{obj}}(e^{\prime})\subseteq O^{\prime}$
and $O^{\prime}$ forms a connected subgraph in $\mathit{OG}_{L}$.

We define two techniques to extract process executions from an OCEL. These two
techniques are two out of many possible process execution extraction
techniques. The first technique extracts process executions based on the
connected components of the object graph. All transitively connected objects
form one process execution. This might lead to large executions for entangled
event logs. Therefore, we introduce the leading type extraction. A process
execution is constructed for each object of a chosen leading object type.
Connected objects are added to this process execution unless a connected
object of the same type has a lower distance to the leading object. This
limits executions in size but also removes dependencies.

######  Definition 4 (Execution Extraction)

Let
$L{=}(E,O,OT,\pi_{\mathit{ct}},\pi_{\mathit{st}},\pi_{\mathit{trace}},\allowbreak\pi_{\mathit{act}},\allowbreak\pi_{\mathit{att}})$
be an event log. An execution extraction
$\mathit{EX}{\subseteq}\mathit{consub}(\mathit{OG}_{L})$ retrieves connected
subgraphs from the object graph. A subgraph
$\mathit{ex}=(O^{\prime},I^{\prime})\in\mathit{EX}$ is mapped to a process
execution $f^{\mathit{extract}}(\mathit{ex},L)=(O^{\prime},E^{\prime})$ with
$E^{\prime}=\\{e\in E\mid O^{\prime}\cap\mathit{obj}(e)\neq\emptyset\\}$. We
define two extraction techniques:

  * $\bullet$

$\mathit{EX}_{\mathit{comp}}(L)=\\{G\in\mathit{consub}(\mathit{OG}_{L})\mid\neg\exists_{G^{\prime}\in\mathit{consub}(\mathit{OG}_{L})}\
G\in\mathit{sub}(G^{\prime})\\}$, and

  * $\bullet$

$\mathit{EX}_{\mathit{lead}}(L,\mathit{ot})=\\{G\in\mathit{lead\\_graphs}=\\{G^{\prime}=(O^{\prime},I^{\prime})\in\mathit{consub}(OG_{L})\mid\exists_{o\in
O^{\prime}}\ \pi_{\mathit{type}}(o)=ot\wedge\forall_{o^{\prime}\in
O^{\prime}}\ \neg\exists_{o^{\prime\prime}\in O^{\prime}}\
o^{\prime\prime}\neq
o^{\prime}\wedge\pi_{\mathit{type}}(o^{\prime\prime})=\pi_{\mathit{type}}(o^{\prime})\wedge\mathit{dist}_{G^{\prime}}(o,o^{\prime})>\mathit{dist}_{G^{\prime}}(o,o^{\prime\prime})\\}\mid\neg\exists_{G^{\prime\prime}\in\mathit{lead\\_graphs}}G\in\mathit{sub}(G^{\prime\prime})\\}$
for $\mathit{ot}\in OT$.

When looking at the example of Fig. 1, the process executions retrieved by
applying $\mathit{EX}_{\mathit{comp}}$ would be based on the connected
components of the object graph, i.e., $\\{o1,i1,i2\\}$ and $\\{o2,i3\\}$.
Using the leading type order, we would retrieve the same executions. Using
item as the leading type, we would retrieve $\\{i1,o1\\}$, $\\{i2,o1\\}$ and
$\\{i3,o2\\}$.

##  4 Object-Centric Features

This section deals with the problem resulting from flattening OCELs to apply
process enhancement techniques: Features are calculated on the manipulated,
flattened event data. Therefore, they might be inaccurate. We propose an
object-centric adaptation of the features introduced by the seminal machine
learning framework of de Leoni et al. [18]. We calculate them natively on the
OCEL. Furthermore, we provide several new features recently introduced in the
literature on object-centric process mining. A feature is, generally,
calculated for an event. It might describe a measure for the single event, in
relationship to its process executions, or the whole system.

######  Definition 5 (Features)

Let
$L=(E,O,OT,\pi_{\mathit{ct}},\allowbreak\pi_{\mathit{st}},\pi_{\mathit{trace}},\pi_{\mathit{act}},\pi_{\mathit{att}})$
be an event log and $\mathit{EX}{\subseteq}\mathit{consub}(\mathit{OG}_{L})$
be a set of extracted process executions. A feature
$f_{L}{:}E{\times}\mathit{EX}{\nrightarrow}\mathbb{R}$ maps an event and a
process execution onto a real number.

Figure 4: Overview of the features that can be extracted for event $e_{6}$.
These features are the object-centric adaptations of [18].

The primary need for adapting traditional feature calculation arises from two
main differentiations between object-centric and traditional event data:
First, each event can have multiple predeccesors/successors, one for each
object. Second, each event might have multiple objects of different types. The
computation of features that are depended on previous and following behavior
has to be adapted to the graph structure. The most obvious example are
preceding activities: In traditional feature extraction, there is only one
preceding activity for each event. In object-centric feature extraction, there
are multiple preceding activities, one for each object. The graph-structure as
well as the multiplicity of objects also enables the definition of new
features leveraging on the graph structure and object (type) associations.
Previous (i.e., all events that happened before the considered event in an
execution) and following events can be adapted in two ways: time-based (using
the event’s timestamp) and path-based (using path information of the graph).
We use a simple time-based adaptation. However, the graph-based adaptation
might give interesting new research directions.

An overview of the features collected from an object-centric adaptation of de
Leoni et al.’s framework [18] and features recently introduced in the
literature [3, 22] is depicted in Fig. 4. Similar to de Leoni et al., we group
features according to different perspectives: Control-Flow, Data-Flow,
Resource, Performance and Objects. We, now, discuss the different perspectives
and the adaptations that are necessary to apply them to the object-centric
setting. Table 2 provides a qualitative evaluation of the impact of flattening
on the resulting feature value: Features can be equal, they can be
misleading/incorrect after flattening, and not be available for flat event
data.

Table 2: Impact of flattening on calculated feature values. Calculating a
feature for an event on object-centric vs. flattened data can lead to correct
or misleading results. Some features only exist on object-centric event data.
Most features are misleading due to the graph structure and object
multiplicity.

Features | Impact of flattening | Only available for OCEL
---|---|---
Correct | Misleading
Control-Flow |  C1, C2, C3, C4 |  | ✓ |
C5 | ✓ |  |
Data |  D1, D2 |  | ✓ |
D3 | ✓ |  |
Resource | R1 |  | ✓ |
R2, R3 | ✓ |  |
Performance |  P1, P2, P3, P6, P10 |  | ✓ |
P4, P5, P7, P8 |  |  | ✓
P9 | ✓ |  |
Objects |  O1, O2, O3, O4, O5 |  |  | ✓

The main adaptations of the control-flow perspectives are concerned with the
switch from sequential to graph-like control-flow. Multiple preceding
activities (C2) as well as multiple current activities (C1) (endpoints of the
current execution graph) are possible. For previous and following activities
(C3, C4), we use a simple time-based adaptation.

The data-flow perspective needs slight adaptations for preceding
characteristic values (D2). Since there might be multiple preceding values,
these need to be aggregated. Previous characteristic values (D1) are adapted
on a time basis, and the characteristic value (D3) needs no adaptation.

The resources perspective’s features are mainly concerned with system-wide
measurements, such as the workload of the current resource (C1) or the total
system workload (C2). Therefore, this perspective remains mostly unaffected by
a move to object-centricity. Future research might investigate new features
derived from resource multiplicity per event.

The performance perspective has recently been studied for new object-centric
features [22]. Due to an event having multiple predecessors, the established
performance measures can be extended by several features expressing the time
for synchronization between objects (P5), the pooling time of an object type
(P7), or the lag between object types before the event (P8).

Finally, a new feature perspective concerning objects opens up. The paper
introducing the discovery of object-centric Petri nets [3] introduces some
basic features of the object perspective. For example, an event’s number of
objects (O5), the event’s number of objects of a specific type (O6), or the
current system’s total object count (O1). Investigations of additional
features in this perspective, e.g., quantifying the relationships between
objects through graph metrics on the object graph, might also be an
interesting research direction.

##  5 Feature Encodings

In this section, we tackle the absence of feature encodings that represent the
graph-like structure of object-centric event data. We extend the currently
used tabular and sequential encodings with a graph-based one and introduce all
three encodings formally. Together with the formal definition of each
encoding, we provide some common use cases, advantages, disadvantages and a
continuation of our running example from Fig. 1. As an example of extracted
features we choose the number of previous objects (O2), the synchronization
time (P5) and the remaining time (P3). The execution extraction for our
example is the connected components extraction $\mathit{EX}_{\mathit{comp}}$.
A tabular encoding is a common representation of data points used for many use
cases, such as regression analysis, clustering, different data mining tasks,
etc.

Figure 5: Example of tabular a), sequential b) and graph-based c) feature
encodings for the running example in Fig. 1. The graph-based encoding
preserves the structural information from the OCEL.

######  Definition 6 (Tabular Encoding)

Let
$L=(E,O,OT,\pi_{\mathit{ct}},\pi_{\mathit{st}},\pi_{\mathit{trace}},\pi_{\mathit{act}},\allowbreak\pi_{\mathit{att}})$
be an event log and $\mathit{EX}\subseteq\mathit{consub}(\mathit{OG}_{L})$ be
a set of process executions. Let $F_{L}\subseteq
E\times\mathit{EX}\nrightarrow\mathbb{R}$ be a set of features. The event
feature table is defined by $\mathit{tab}(e,f_{L})=f_{L}(e,\mathit{ex})$ for
all $e\in E$, $\mathit{ex}\in\mathit{EX}$ (rows) and all $f_{L}\in F_{L}$
(columns).

We depict an example of tabular encoding in Fig. 5 a). Such an encoding is
easily readable and versatile usable, however, the structural order
information of the event log is lost in the process of tabular encoding. A
sequential encoding is commonly used in sequence visualization, clustering,
classification or next value predictions (cf. Sec. 2).

######  Definition 7 (Sequential Encoding)

Let
$L=(E,O,OT,\pi_{\mathit{ct}},\pi_{\mathit{st}},\pi_{\mathit{trace}},\pi_{\mathit{act}},\allowbreak\pi_{\mathit{att}})$
be an event log and $\mathit{EX}\subseteq\mathit{consub}(\mathit{OG}_{L})$ be
a set of extracted process executions. Let
$F_{L}{=}\\{f_{L,1},\ldots,f_{L,m}\\}{\subseteq}E{\times}\mathit{EX}{\nrightarrow}\mathbb{R}$
be a set of features. The sequential encoding of an execution
$\mathit{ex}\in\mathit{EX}$ is defined by
$\mathit{seq}(\mathit{ex},F_{L})=\langle(f_{L,1}(e_{1},\mathit{ex}),\allowbreak\ldots,\allowbreak
f_{L,m}(e_{1},\mathit{ex})),\ldots,\allowbreak(f_{L,1}(e_{n},\mathit{ex}),\allowbreak\ldots,\allowbreak
f_{L,m}(e_{n},\mathit{ex}))\rangle$ with with
$(O^{\prime},\\{e_{1},\ldots,e_{n}\\})=f^{\mathit{extract}}(\mathit{ex},L)$
and
$\pi_{\mathit{ct}}(e_{1})\leq\cdots\leq\allowbreak\pi_{\mathit{ct}}(e_{n})$.

We depict a sequential encoding of the running example in Fig. 5 b). The
events for process executions are ordered according to the complete timestamp
of the event. The resulting sequence is attributed with the different feature
values for each event. This encoding respects the timely order of events.
However, it does not respect the true precedence constraints of the event log:
By merging all events into one sequence, some event pairs are forced into a
precedence relationships they did not exhibit in the event log (cf. Sec. 1). A
graph encoding of features may be used for extensive visualization, applying
graph algorithms or for utilizing graph neural networks [30].

######  Definition 8 (Graph Encoding)

Let
$L{=}(E,O,OT,\pi_{\mathit{ct}},\pi_{\mathit{st}},\pi_{\mathit{trace}},\pi_{\mathit{act}},\allowbreak\pi_{\mathit{att}})$
be an event log $\mathit{EX}\subseteq\mathit{consub}(\mathit{OG}_{L})$ be a
set of extracted process executions. Let
$F_{L}=\\{f_{L,1},\ldots,f_{L,m}\\}\subseteq
E\times\mathit{EX}\nrightarrow\mathbb{R}$ be a set of features. For an
extracted execution $\mathit{ex}\in\mathit{EX}$, the graph of the
corresponding process execution
$p=(O^{\prime},E^{\prime})=f^{\mathit{extract}}(\mathit{ex},L)$ is defined by
$G_{p}=(E^{\prime},K)$ with edges $K=\\{(e,e^{\prime})\in E^{\prime}\times
E^{\prime}\mid e\neq e^{\prime}\wedge o\in O^{\prime}\wedge\langle
e_{1},\ldots,e_{n}\rangle\in\pi_{\mathit{trace}}(o)\wedge e=e_{i}\wedge
e^{\prime}=e_{i+1}\wedge i\in\\{1,\ldots,n-1\\}\\})$. The graph encoding is
defined by $G_{\mathit{feat}}(p,F_{L})=(E^{\prime},K,l)$ with a node labeling
function $l(e)=\\{f_{L}(e,\mathit{ex})\mid f_{L}\in F_{L}\\}$ for any $e\in
E^{\prime}$.

An example of the graph-based feature encoding for our running example is
depicted in Fig. 5 c). Each process execution is associated with a graph. Each
node of the graph represents the feature values of an event.

##  6 Use Cases

Table 3: Results for the different models based on different encodings.

| Regression | LSTM | GNN
---|---|---|---
Baseline MAE | $0.7598$
Train MAE | $0.5101$ | $0.4717$ | $0.4460$
Validation MAE | NA | $0.4625$ | $0.4534$
Test MAE | $0.5087$ | $0.4568$ | $0.4497$

In this section, we evaluate our framework by providing six use cases. We
pursue two evaluation goals with this approach: First, we aim to showcase the
generalizability of the framework by providing a collection of common process
mining tasks the framework can be applied to. Second, we aim to showcase the
feature’s and encoding’s effectiveness in the use cases. Over the last years,
explainable AI has been increasingly employed to make predictive process
monitoring transparent [15, 17]. Through the use of SHAP [21] values, we are
able to quantify feature importance as well as structural importance of
sequential and graph-based encoding.

The use cases are split into two parts: three visualization and three
prediction use cases. We use a real-life loan application event log [8] as an
OCEL. An event can be related to an application and multiple loan offers as
objects. We use tabular, sequential, and graph-based encoding to gain insights
into the process through the visualization use cases. The prediction use cases
aim at predicting the remaining time of an event’s process execution (P3)
using three different techniques for the different encodings: regression
(tabular), LSTM neural networks (sequential), and GNNs (graph-based). We use
the same features for each encoding: Preceding activities (C2), average
previous requested amount (D1), the elapsed time (P2), and the previous number
of offers (O3). We use a 0.7/0.3 train/test split of the same events for each
model for comparability reasons. We set aside 20% of the training set as a
validation set. The performance is assessed using the _Mean Absolute Error
(MAE)_ of the normalized target variable. Furthermore, we provide a baseline
MAE achieved by predicting the average remaining time of the training set. The
summarized results are depicted in Table 3.

We provide an open-source python implementation of our
framework111https://github.com/ocpm/ocpa. The framework can be extended with
new features and adapted algorithms.

Figure 6: Time series describing two features over time: the weekly average
number of loan offers per event and the weekly average requested amount for
each application. Using this evaluation some initial insights can be
generated, e.g., the gradual increase in requested amount over time.

###  6.1 Tabular Encoding

#### Visualization

We split the event log into subsequent sublogs containing the events of one
week each. For each sublog, we extract the average requested amount (D3) and
the number of offers per event (O6). The resulting time series is depicted in
Fig. 6. We can observe the dynamics of the process over time, e.g., the
increase in the requested amount over time. Furthermore, we can observe that
the number of offers is stable, except for a few short spikes.

#### Prediction

We use a linear regression model to predict the remaining time based on the
tabular encoding (cf. Table 3). This is an object-centric adaption of use
cases [9, 13]. We generate the SHAP values, i.e., the impact of different
features on the individual model prediction, for 1000 predictions of the test
set. The resulting bee swarm plot is depicted in Fig. 7 (left side). Red
points indicate a high feature value. The more they are positioned to the
left, the more the feature value reduces the model’s prediction. Therefore,
the combination of color and position gives insights into the feature value’s
impact on the model output. We can, e.g., observe a high decreasing impact of
the existence of the Call activity in the preceding activities to the
predicted remaining time. One can also observe an impact of the new object-
centric feature of the number of previous objects of type offer on the
predicted remaining time: the more offers were previously recorded in a case,
the lower the predicted remaining time. In conclusion, the selected set of
object-centric feature adaptations yields valuable information for a
predictive model.

Figure 7: Left: Bee swarm plot of SHAP values for the regression model,
showing the aggregated importance of different features to the predictions.
Right: SHAP values of one LSTM prediction, visualized for the different
positions of the input sequence.  Figure 8: Sequential variant visualization
of a process execution enriched with the object information (blue =
application, orange = first and second offer).

###  6.2 Sequential Encoding

#### Visualization

We choose one specific process execution and extract the sequential encoding
for the current event’s activity (C5) and the event’s objects (O4) features.
The result is a variant enriched by object information, depicted in Fig. 8.
Even though such a visualization might have misleading causality information
for events between objects, one can already retrieve some valuable insight
into the intra-object order and the overall activities of an execution.

#### Prediction

We use a neural network with two 10-hidden-node LSTM layers to predict the
remaining time of the sequentially encoded features. We use subsequences of
length four (cf. Table 3 for results). This is an object-centric adaption of
use cases [26, 19]. The regression use case already covered the importance of
features for the prediction. Therefore, we focus on the importance of the
sequential encoding in this use case. We use SHAP values for each feature of
the four positions in the sequential encoding. The calculated feature impacts
for an individual prediction are depicted in Fig. 7 (right side). The more the
value diverges from zero, the higher the feature’s impact on the model’s
output. We observe features with high importance among all four positions of
the sequence. Therefore, the model utilizes the sequential encoding of the
features, showcasing its usefulness.

###  6.3 Graph Encoding

#### Visualization

Fig. 9 a) depicts the graph-based variant visualization retrieved from OC$\pi$
[5] of the same process execution as Fig. 8. Using the graph, one can place
concurrent events in two different lanes according to their objects, not
indicating any precedence between them. One can intuitively determine the
concurrent paths in the variant and the interaction of different objects. For
large process executions, this provides structured access to the control-flow
of the underlying process.

#### Prediction

We use the graph-based feature encoding as an input for a GNN. The GNN
contains two graph convolution layers. Each node in both layers has a size of
24. Input graphs are constrained to four nodes (cf. LSTM use case). We read
the graphs out by averaging over the convoluted values, summarizing to one
predicted remaining time (cf. Table 3 for results). This is an object-centric
adaptation of use cases [23, 27]. We adapt SHAP values to determine the
importance of graph edges to the predicted remaining time. Fig. 9 b) depicts
the calculated values for one graph instance. The more the value of an edge
diverges from zero, the higher its existence impacts the model’s prediction.
We observe substantially different values for all edges: While some edges have
a relatively low negative or positive impact on the model’s output, the
presence of other edges heavily impacts the predicted remaining time.
Therefore, the graph structure itself yields important information for
predicting the remaining time.

Figure 9: Use cases for the graph encoding: a) activities and objects of one
process execution. Shared events between objects are colored with multiple
colors. b) shows the importance of different edges of one instance graph when
predicting.

##  7 Conclusion

We introduced a general framework to extract and encode features from OCELs.
Currently, object-centric event data needs to be flattened to apply process
enhancement techniques to the data. This leads to inaccurate features.
Additionally, no feature encoding is available to express the graph-like
structure of object-centric event data. Our framework calculates features
natively on the object-centric event data, leading to accurate features.
Furthermore, we provide a graph-based encoding of the features, preserving the
underlying structure of an OCEL. We show the utility of the features and
encodings in six use cases, a visualization and prediction use case for each
of the three encodings. This framework lays a foundation for future machine
learning approaches utilizing object-centric event data and new algorithms
using our encodings as a basis.

We provide a collection of use cases showing the applicability of our
framework for extracting and encoding features. For each of our framework
steps, interesting future research directions are present: Which feature work
well with which encoding? What are the best prediction techniques for which
encoding? How to optimize existing network architectures to achieve maximum
results? Furthermore, investigations of new features derived from the graph
structure and object-multiplicity as well as further traditional features not
included in de Leoni et al.’s framework [18] is an interesting direction for
future research.

## References

  * [1] van der Aalst, W.M.P.: Process mining: Data science in action. Springer (2016)
  * [2] van der Aalst, W.M.P.: Object-centric process mining: Dealing with divergence and convergence in event data. In: SEFM. pp. 3–25. Springer (2019)
  * [3] van der Aalst, W.M.P., Berti, A.: Discovering object-centric Petri nets. Fundam. Informaticae 175(1-4), 1–40 (2020)
  * [4] Adams, J.N., van der Aalst, W.M.P.: Precision and fitness in object-centric process mining. In: ICPM. pp. 128–135. IEEE (2021)
  * [5] Adams, J.N., van der Aalst, W.M.P.: Oc$\pi$: Object-centric process insights. In: PETRI NETS. vol. 13288, pp. 139–150. Springer (2022)
  * [6] Adams, J.N., Schuster, D., Schmitz, S., Schuh, G., van der Aalst, W.M.P.: Defining cases and variants for object-centric event data. CoRR abs/2208.03235 (2022). https://doi.org/10.48550/arXiv.2208.03235, https://doi.org/10.48550/arXiv.2208.03235
  * [7] Becker, J., Breuker, D., Delfmann, P., Matzner, M.: Designing and implementing a framework for event-based predictive modelling of business processes. In: EMISA. pp. 71–84. GI (2014)
  * [8] van Dongen, B.: BPI Challenge 2017 (2017). https://doi.org/10.4121/uuid:5f3067df-f10b-45da-b98b-86ae4c7a310b
  * [9] van Dongen, B.F., Crooy, R.A., van der Aalst, W.M.P.: Cycle time prediction: When will this case finally be finished? In: OTM. pp. 319–336. Springer (2008)
  * [10] Ehrendorfer, M., Mangler, J., Rinderle-Ma, S.: Assessing the impact of context data on process outcomes during runtime. In: ICSOC. pp. 3–18. Springer (2021)
  * [11] Esser, S., Fahland, D.: Multi-dimensional event data in graph databases. J. Data Semant. 10(1-2), 109–141 (2021)
  * [12] Evermann, J., Rehse, J., Fettke, P.: Predicting process behaviour using deep learning. Decis. Support Syst. 100, 129–140 (2017)
  * [13] Folino, F., Guarascio, M., Pontieri, L.: Discovering context-aware models for predicting business process performances. In: OTM. pp. 287–304. Springer (2012)
  * [14] Francescomarino, C.D., Dumas, M., Federici, M., Ghidini, C., Maggi, F.M., Rizzi, W.: Predictive business process monitoring framework with hyperparameter optimization. In: CAiSE. pp. 361–376. Springer
  * [15] Galanti, R., Coma-Puig, B., de Leoni, M., Carmona, J., Navarin, N.: Explainable predictive process monitoring. In: ICPM. pp. 1–8. IEEE (2020)
  * [16] Harl, M., Weinzierl, S., Stierle, M., Matzner, M.: Explainable predictive business process monitoring using gated graph neural networks. Journal of Decision Systems 29(sup1), 312–327 (2020)
  * [17] Huang, T., Metzger, A., Pohl, K.: Counterfactual explanations for predictive business process monitoring. In: EMCIS. vol. 437, pp. 399–413. Springer (2021)
  * [18] de Leoni, M., van der Aalst, W.M.P., Dees, M.: A general process mining framework for correlating, predicting and clustering dynamic behavior based on event logs. Inf. Syst. 56, 235–257 (2016)
  * [19] Leontjeva, A., et al.: Complex symbolic sequence encodings for predictive monitoring of business processes. In: BPM. pp. 297–313. Springer (2015)
  * [20] Li, C., van Zelst, S.J., van der Aalst, W.M.P.: Stage-based process performance analysis. In: ICSOC Workshops. pp. 349–364. Springer (2020)
  * [21] Lundberg, S.M., Lee, S.: A unified approach to interpreting model predictions. In: NeurIPS. pp. 4765–4774 (2017)
  * [22] Park, G., Adams, J.N., van der Aalst, W.M.P.: Opera: Object-centric performance analysis (2022). https://doi.org/10.48550/ARXIV.2204.10662
  * [23] Philipp, P., et al.: Analysis of control flow graphs using graph convolutional neural networks. In: ISCMI. pp. 73–77 (2019)
  * [24] Rogge-Solti, A., Weske, M.: Prediction of remaining service execution time using stochastic petri nets with arbitrary firing delays. In: ICSOC. pp. 389–403. Springer (2013)
  * [25] Sun, Y., Bauer, B., Weidlich, M.: Compound trace clustering to generate accurate and simple sub-process models. In: ICSOC. pp. 175–190. Springer (2017)
  * [26] Tax, N., Verenich, I., Rosa, M.L., Dumas, M.: Predictive business process monitoring with LSTM neural networks. In: CAiSE. pp. 477–492. Springer (2017)
  * [27] Venugopal, I., Töllich, J., Fairbank, M., Scherp, A.: A comparison of deep-learning methods for analysing and predicting business processes. In: IJCNN. pp. 1–8 (2021)
  * [28] Waibel, P., Pfahlsberger, L., Revoredo, K., Mendling, J.: Causal process mining from relational databases with domain knowledge. CoRR abs/2202.08314 (2022)
  * [29] Wang, C., Cao, J.: Interval-based remaining time prediction for business processes. In: ICSOC. pp. 34–48. Springer (2021)
  * [30] Wu, Z., et al.: A comprehensive survey on graph neural networks. IEEE Trans. Neural Networks Learn. Syst. 32(1), 4–24 (2021)


# DALI: a large Dataset of synchronized Audio, LyrIcs and notes, automatically
created using teacher-student machine learning paradigm

###### Abstract

The goal of this paper is twofold. First, we introduce DALI, a large and rich
multimodal dataset containing 5358 audio tracks with their time-aligned vocal
melody notes and lyrics at four levels of granularity.

The second goal is to explain our methodology where dataset creation and
learning models interact using a teacher-student machine learning paradigm
that benefits each other. We start with a set of manual annotations of draft
time-aligned lyrics and notes made by non-expert users of Karaoke games. This
set comes without audio. Therefore, we need to find the corresponding audio
and adapt the annotations to it. To that end, we retrieve audio candidates
from the Web. Each candidate is then turned into a singing-voice probability
over time using a teacher, a deep convolutional neural network singing-voice
detection system (SVD), trained on cleaned data. Comparing the time-aligned
lyrics and the singing-voice probability, we detect matches and update the
time-alignment lyrics accordingly. From this, we obtain new audio sets. They
are then used to train new SVD students used to perform again the above
comparison. The process could be repeated iteratively. We show that this
allows to progressively improve the performances of our SVD and get better
audio-matching and alignment.

##  1 Introduction

Singing voice is one of the most important elements in popular music. It
combines its two main dimensions: melody and lyrics. Together, they tell
stories and convey emotions improving our listening experience. Singing voice
is usually the central element around which songs are composed. It adds a
linguistic dimension that complements the abstraction of the musical
instruments. The relationship between lyrics and music is both global (lyrics
topics are usually highly related to music genre) and local (it connects
specific musical parts with a concrete lexical meaning, and also defines the
structure of a song).

Despite its importance, singing voice has not received much attention from the
MIR community. It has only been introduced a few years ago as a standalone
topic[12, 17]. One of the most important factors that prevents its development
is the absence of large and good quality reference datasets. This problem also
exists in other MIR fields, nevertheless several solutions have been
proposed[3, 9]. Currently, researchers working in singing voice use small
designed dataset following different methodology[10]. Large datasets as the
one used in [13] are private and not accessible to the community.

The goal of this paper is to propose such a dataset and to describe the
methodology followed to construct it.

Figure 1: [Left part] The inputs of our dataset creation system are karaoke-
user annotations presented as a triple of {time (start + duration), musical-
notes, text}. [Right part] Our dataset creation system automatically finds the
corresponding full-audio track and aligned the vocal melody and the lyrics to
it. In this example, we illustrate the alignment for a small excerpts. We only
represent two levels of lyrics granularity: notes and lines.

###  1.1 Proposal

We present the DALI dataset: a large Dataset of synchronised Audio, LyrIcs and
notes that aims to stand as a reference for the singing voice community. It
contains 5358 songs (real music) each with – its audio in full-duration, – its
time-aligned lyrics and – its time-aligned notes (of the vocal melody). Lyrics
are described according to four levels of granularity: notes (and textual
information underlying a given note), words, lines and paragraphs. For each
song, we also provide additional multimodal information such as genre,
language, musician, album covers or links to video clips. The rest of this
paper focuses on our methodology for creating DALI. In Figure 1, we illustrate
the input and output of our dataset creation system. See Section 4 for more
details about the dataset itself.

The DALI dataset has been created automatically. Our approach consists in a
constant interaction between dataset creation and learning models where they
benefit from each other. We developed a system that acquires lyrics and notes
aligned in time and finds the corresponding audio tracks. The time-aligned
lyrics and notes come from Karaoke resources (see Section 3.1 for more
deatils). Here, non-expert users manually describe the lyrics of a song as a
sequence of annotations: time aligned notes with their associated textual
information. While this information is powerful it has two major problems: 1)
there is no information about the exact audio used for the annotation process
(only the song title and artist name which may lead to many different audio
versions), 2) even if the right audio is found, annotations may need to be
adjusted to fit the audio perfectly. In Section 3.2, we define how we retrieve
from the Web the possible audio candidates for each song. In Section 3.3, we
describe how we select the right audio among all the possible candidates and
how we automatically adapt the annotated time-alignment lyrics to this audio.
In order to do this, we propose a distance that measures the correspondence
between an audio track and a sequence of manual annotations. This distance is
also used to perform the necessary adaptations on the annotations to be
perfectly aligned with the audio. Our distance requires the audio to be
described as a singing voice probability sequence. This is computed using a
singing voice detection (SVD) system based on deep convolutional neural
network (ConvNet). The performance of our system highly depends on the
precision of the SVD. Our first version is trained on few but accurately-
labeled ground truths. While this system is sufficient to select the right
audio it is not to get the best alignment. To improve the SVD, in Section 3.4
we propose to use a teacher-student paradigm. Thanks to the first SVD system
(the teacher) we selected a first set of audio tracks and their corresponding
annotations. Using them, we train new SVD systems (the students). We show in
Section 3.4.1 that new SVD systems (the students) are better than the initial
one (the teacher). With this new version, we increase the quality and size of
the DALI dataset. Finally, we discuss our research in Section 5.

##  2 Related works

Table 1: Terms overview: definition of each term used in this paper. Term | Definition
---|---
Annotation | basic alignment unit as a triple of time (start + duration wrt $Fr$), musical-notes (with 0 = C3) and text.
A file with annotations | group of annotations that define the alignment of a particular song.
Offset_time as $O$ | it indicates the start of the annotations, its modifications moves all bock to the right or left.
Frame rate as $Fr$ | it controls the annotation grid size stretching or compressing its basic unit.
Annotation voice sequence as $avs(t)\in\\{0,1\\}$ | singing voice (SV) sequence extracted from karaoke-users annotations.
Predictions as $\hat{p}(t)\in[0,1]$ | SV probability sequence provided by our singing voice detection.
Labels | labels sequence of well known ground truths datasets checked by the MIR community.
Teacher | first SV detection (SVD) system trained on Labels.
Student | new SVD system trained on the $avs(t)$ for the subset of track for which $NCC(\hat{o},\hat{fr})\geq T_{corr}$.

We review previous works related to our work: singing voice detection methods
and the teacher-student paradigm.

Singing Voice detection. Most approaches share a common architecture. Short-
time observations are used to train a classifier that discriminates
observations (per frame) in vocal or non-vocal classes. The final stream of
predictions is then post-processed to reduce artifacts.

Early works explore classification techniques such as Support Vector Machines
(SVMs) [20, 16], Gaussian mixture model (GMM) [11] or multi-layer perceptron
(MLP)[4]. Other approaches also tried to use specific vocal traits such as
vibrato and tremolo [21] or to adapt speech recognition systems for the
particularities of singing voice [5]. Over the past few years, most works
focus on the use of deep learning techniques. For example, [24, 23] propose
the use of ConvNet combined with data augmentation techniques (to increase the
size of the training set) or trained on weakly labeled data (the data are only
labeled at the file level, not at the segment level). [13] also proposes the
use of CNN but with a Constant-Q input and a training on a very large private
datasets mined from Spotify resources. Some researchers suggest the use of
Recurrent Neural Networks (RNN)[15] or Long Short-Term Memory (LSTM) [14]. One
advantage of these models is that they directly model the decisions sequence
over time and no post-processing is needed. Other singing voice detection
systems are developed to be used as a pre-processing-step: for lyrics
transcription [17] or for source separation [25] trained then to obtain ideal
binary masks.

Teacher-student paradigm. Teacher-student learning paradigm [2, 28] has
appeared as a solution to overcome the problem of insufficient labeled
training data in MIR. Since manual labeling is a time-consuming tasks, the
teacher-student paradigm explores the use of unlabeled data for supervised
problems. The two main agents of this paradigm are: the ‘teacher’ and the
‘student’. The teacher is trained with labels of well known ground truths
datasets (often manually annotated). It is then used to automatically label
unlabeled data on a (usually) larger dataset. These new labels (the one given
by the teacher) are the ones used for training the student(s). Student(s)
indirectly acquire(s) the desired knowledge by mimicking the “teacher
behaviour”. This model has achived great results for tasks in speech
recognition [27] and multilingual models [8]. It has also been proved that
student(s) can achieve superior performances than the teacher [28, 8].

##  3 Singing voice dataset: creation

###  3.1 Karaoke resources

Outside the MIR community there are rich sources of information that can be
explored. One of these sources is Karaoke video games that fit exactly our
requirements. In these games, users have to sing along with the music to win
points according to their singing accuracy. To measure their accuracy, the
user melody is compared with a reference timing note (that has fine time and
frequency). Hence, large datasets of time-aligned note and lyrics exist.

Such datasets can be found as open-source. Nowadays, there are several active
and big karaoke open-source communities. In those, non-expert users exchange
text files containing lyrics and melody annotations. However there is no
further professional revision. Each file contains all the necessary
information to describe a song:

  * •

the sequence of triplets {time, musical-notes, text},

  * •

the offset_time (start of the sequence) and frame rate (annotation time-grid),

  * •

the song_title and the artist_name.

We refer to Table 1 for the definition of all the terms we use. These
annotations can be transformed to get the time and note frequencies as seen
Figure 1.

We were able to retrieve 13339 karaoke annotation files. Although this
information is outstanding for our community, it presents several problems
that have to be solved:

[.15cm]

Global.

When performing the annotation, users can choose the audio file they want. The
problem is that only the song_title and artist_name are provided. This
combination might refer to different audio versions (studio, radio edit, live,
remix, etc.). Consequently, we do not know which audio version has been used.
Annotations made for a version do not work for another. Besides, even if the
correct audio is known, annotations may not perfectly fit it. As a result
annotations must be adapted. This is done by modifying the provided
offset_time and frame rate. These issues are not problematic for karaoke-users
but critical To the automatic creation of a large dataset for MIR research.

Local.

It refers to errors due to fact the that users are non-professionals. It
covers local alignment problems of particular lyric blocks, text misspellings
or note mistakes.

In this paper we only focus on global problems leaving the local ones for
future works.

Figure 2: Architecture of our Singing Voice Detection system using a ConvNet.

WASABI  is a semantic database of song knowledge gathering metadata collected
from various music databases on the Web [18]. In order to benefit from the
richness of this database, we first linked each annotation file to Wasabi. To
that end, we connected a specific song_title and artist_name with all possible
corresponding audio versions (studio, radio, edit, live, remix, etc.). The
WASABI also provides lyrics in a text only annotations (grouped by lines and
paragraphs). Using the two lyrics representations (note-based annotations and
text only annotations), we created four levels of granularity: notes, words,
lines and paragraphs. Finally, WASABI also provides extra multimodal
information such as cover images, links to video clips, metadata, biography,
expert notes, etc.

###  3.2 Retrieving audio candidates

Our input is an annotation file connected to the WASABI database. This
database provides us with the different existing versions (studio, radio,
edit, live, remix, etc.) for a song_title and artist_name combination. Knowing
the possible versions, we then automatically query YouTube111We use
https://github.com/rg3/youtube-dl to get a set of audio candidates. We now
need to select among the set of audio candidates the one corresponding to the
annotation file.

###  3.3 Selecting the right audio from the candidate and adapting annotation
to it

Each audio candidate is compared to the reference annotation file. We do this
by measuring a distance between both and keeping the one with the largest
value.

Audio and annotations live in two different representation spaces that cannot
be directly compared. In order to find a proper distance, we need to transform
them to a common representation space. Two directions were studied:

[.15cm]

Annotations as audio.

We have explored lyrics synchronization techniques [10] but their complexity
and phonetic model limitations prevent us to use them. As annotations can be
transformed into musical notes, score alignment approaches [7, 26] seem a
natural choice. However, due to missing information in the corresponding score
(we only have the score of the vocal melody) these systems failed. We then
tried to reduce the audio to the vocal melody (using Melodia [22]) and then
align it to the vocal melody score but this also failed. Consequently, we did
not persist in this direction.

Audio as annotations.

The idea we develop in the remainder is the following. We convert the audio
track to a singing-voice probability $\hat{p}(t)$ over time $t$. This sequence
has value $\hat{p}(t)\rightarrow 1$ when voice is present at time $t$ and
$\hat{p}(t)\rightarrow 0$ otherwise. This probability is computed from the
audio signal using a Singing Voice Detection (SVD) system described below. We
name this function predictions. Similarly, the sequence of annotated triplets
{time, musical-notes, text} can be mapped to the same space: $avs(t)=1$ when a
vocal note exists at $t$ and $avs(t)=0$ otherwise. We name this function
annotation voice sequence.

Singing Voice Detection system. Our system is based on the deep Convolutionnal
Neural Network proposed by [24]. The audio signal is first converted to a
sequence of patches of 80 Log-Mel bands over 115 time frames. Figure 2 shows
the architecture of the network. The output of the system represents the
singing voice probability for the center time-frame of the patch. The network
is trained on binary target using cross-entropy loss-function, ADAMAX
optimizer, mini-batch of 128, and 10 epochs.

Cross-correlation. To compare audio and annotation, we simply compare the
functions $\hat{p}(t)$ and $avs(t)$. As explained before, the annotation files
also come with a proposed offset_time and frame rate. We denote them by $O$
and $Fr$ in the following. The alignment between $\hat{p}(t)$ and $avs(t)$
depends on the correctness of $O$ and $Fr$ values. We will search around $O$
and $Fr$ to find the best possible alignment. We denote by $o$ the correction
to be applied to $O$ and by $fr$ the best $Fr$. Our goals are to:

  1. 1.

find the value of $o$ and $fr$ that provides the best alignment between
$\hat{p}(t)$ and $avs(t)$,

  2. 2.

based on this best alignment, deciding if $\hat{p}(t)$ and $avs(t)$ actually
match each other and establishing if the match is good enough to be kept.

Figure 3: Singing Voice Dataset creation using a teacher-student paradigm.

Since we are interested in a global matching between $\hat{p}(t)\in[0,1]$ and
$avs(t)\in\\{0,1\\}$ we use the normalized cross-correlation (NCC) as
distance222Matches between $\hat{p}(t)$ and $avs(t)$ can also be found using
Dynamic Time Warping (DTW). However, we found its application not successfull
for our purpose. Indeed, DTW computes local warps that does not respect the
global structure of the user annotations. In addition, its score is not
normalized preventing its use for matches selection.:

$NCC(o,fr)=\frac{\sum_{t}avs_{fr}(t-o)\hat{p}(t)}{\sqrt{\sum_{t}avs_{fr}(t)^{2}}\sqrt{\sum_{t}\hat{p}(t)^{2}}}$

The NCC provides us directly with the best $\hat{o}$ value. This value
directly provides the necessary correction to be applied to $O$ to best align
both sequences.

To find the best value of $fr$ we compress or stretch annotation by changing
the grid size. This warp is constant and respect the annotation structure. We
denote it as $avs_{fr}(t)$. The optimal $fr$ value is computed using a brute
force approach, testing the values of $fr$ around the original $Fr$ in an
interval controlled by $\alpha$ (we use $\alpha=Fr*0.05$):

$\hat{fr},\hat{o}=\operatorname*{arg\,max}_{fr\in[Fr-\alpha,Fr+\alpha],o}NCC(o,fr)$

Our final score is given by $NCC(\hat{o},\hat{fr})$.

The audio is considered as good match the annotation if
$NCC(\hat{o},\hat{fr})\geq T_{corr}$. The value of $T_{corr}$ has been found
empirically to be $T_{corr}=0.8$. For a specific annotation, if several audio
candidate tracks have a value $NCC\geq T_{corr}$, we only keep the one with
the largest value. $T_{corr}=0.8$ is quite restrictive but even if we may
loose good pairs we ensure that those we keep are well aligned. When an audio
match is found, the annotations are adapted to it using $\hat{fr}$ and
$\hat{o}$.

Necessity to improve the Singing Voice Detection system. The score $NCC$
proposed above strongly depends on the quality of $\hat{p}(t)$ (the prediction
provided by the Singing Voice Detection (SVD) system). Small differences in
predictions lead to similar $NCC(\hat{o},\hat{fr})$ values but very different
alignments. While the predictions of the baseline SVD system are good enough
to select the correct audio candidates (although there are still quite a few
false negatives), it is not good enough to correctly estimate $\hat{fr}$ and
$\hat{o}$. As improving the SVD system the number of false negatives will be
reduced and we will also find better alignments. We hence need to improve our
SVD system.

The idea we propose below is to re-train the SVD system using the set of
candidates audio that match the annotations. This is a much larger training
set (around 2000) than the one used to train the baseline system (around 100).
We do this using a teacher-student paradigm.

###  3.4 Teacher-Student

Our goal is to improve our Singing Voice Detection (SVD) system. If it becomes
better, it will find better matches and align more precisely audio and
annotations. Consequently, we will obtain a better DALI dataset. This larger
dataset can then be used to train a new SVD system which again, can be used to
find more and better matches improving and increasing the DALI dataset. This
can be repeated iteratively. After our first iteration and using our best SVD
system, we reach 5358 songs in the DALI dataset.

We formulate this procedure as a Teacher-Student paradigm. The processing
steps of the whole Singing Voice Dataset creation is summarized in Figure 3.

[.15cm]

Upper left box.

We start from Karaoke resources that provide our set of annotation files. Each
annotation file defines a sequence of triplets {time, note, next} that we
convert to an annotation voice sequence $avs(t)$. For each annotation file, we
retrieve a set of audio candidates.

Upper right box.

We independently trained a first version of the SVD system (based on ConvNet)
using the training set of a ground truth labeled dataset as provided by the
Jamendo[20] or MedleyDB[6] datasets. We call this first version the teacher.

Upper middle part.

This teacher is then applied on each audio candidate to predict $\hat{p}(t)$.

Lower left box.

We measure the distance between $avs(t)$ and $\hat{p}(t)$ using our cross-
correlation method. It allows us to find the best audio candidate for an
annotation file and the best alignment parameters $\hat{fr},\hat{o}$.

Lower middle box.

We select the audio annotation pairs for which $NCC(\hat{o},\hat{fr})\geq
T_{corr}=0.8$. The set of selected audio tracks forms a new training set.

Lower right box.

This new set is then used to train a new SVD systems based on the same CNN
architecture. This new version is called the student. To this end, we need to
define the target $p$ to be minimized in the loss $\mathcal{L}(p,\hat{p})$.

There are three choices:

  * a)

we use as target $p$ the predicted value $\hat{p}$ given right by the teacher
(usual teacher-student paradigm).

  * b)

we use as target $p$ the value $avs$ corresponding to the annotations after
aligning them using $\hat{fr}$ and $\hat{o}$.

  * c)

a combination of both, keeping only these frames for which
$\hat{p}(t)=avs(t)$.

Up to now and since the $avs$ have been found more precise than the $\hat{p}$
we only investigated option b).

We compare in the following part the results obtained using different teachers
and students.

####  3.4.1 Validating the teacher-student pardigm

In this part, we demonstrate that the students trained on the new training-set
actually perform better than the teacher trained on the ground-truth label
dataset.

[.15cm]

Ground-truth datasets:

We use two ground-truth labels datasets: Jamendo[20] and MedleyDB[6]. We
created a third dataset by merging Jamendo and MedleyDB named as J+M. Each
dataset is split into a train and a test part using an artist filter (the same
artist cannot appear in both).

Teachers:

With each ground-truth datasets we trained a teacher using only the training
part. Once trained, each teacher is used to select the audio matches as
described in Section 3.3. As a result, we produce three new training sets.
They contains 2440, 2673 and 1596 items for the teacher J+M, Jamendo and
MedleyDB respectively. The intersection of the three sets (not presented here)
indicates that 89.8 % of the tracks selected using the MedleyDB teacher are
also present within the tracks selected using the J+M teacher or the Jamendo
teacher. Also, 91.4 % of the tracks selected using the Jamendo teacher are
within the tracks selected using the J+M teacher. It means that the three
teachers agree most of the time on selecting the audio candidates.

Students:

We train three students using the audio and the $avs$ value of the new
training sets. Even if there is a large audio files overlap within the
training sets, their alignment (and therefore the $avs$ value) is different.
The reason to this is that each teacher gets a different $\hat{p}$ which
results in different $\hat{fr},\hat{o}$ values.

####  3.4.2 Results

We evaluate the performances of the various teachers and students SVD systems
using the test parts of Jamendo (J_test) and MedleyDB (M_test). We measure the
quality of each SVD system using the frame accuracy i.e. average value over
all the tracks of the test set.

Results are indicated in Table 2. In this table, e.g. “Student
(Teacher_J_train) (2673)” refers to the student trained on the 2673 audio
candidates and the $avs$ values computed with the Teacher trained on Jamendo
train set.

Table 2: Performances of the teachers and students using the various datasets. Number of tracks in brackets. SVD system Test_set | J_test (16) | M_test (36)
---|---|---
Teacher_J_train (61) | 87% | 82%
Student (Teacher_J_train) (2673) | 82% | 82%
Teacher_M_train (98) | 76% | 85%
Student (Teacher_M_train) (1596) | 80% | 84%
Teacher_J+M_train (159) | 82% | 82%
Student (teacher_J+M_train) (2440) | 86% | 87%

[.15cm]

Performance of the teachers.

We first test the teachers. Teacher_J_train obtains the best results on J_test
(87%). Teacher_M_train obtains the best results on M_test (85%). In both
cases, since training and testing are performed on two parts of the same
dataset, they share similar audio characteristics. These results are
artificially high. To best demonstrate the generalization of the trained SVD
systems, we need to test them in a cross-dataset scenario, namely train and
test in different datasets.

Indeed, in this scenario the results are quite different. Applying
Teacher_J_train on M_test the results decreases down to 82% (a 5% drop).
Similarly when applying Teacher_M_train on J_test the results decreases down
to 76% (a 9% drop). Consequently, we can say that the teachers do not
generalize very well.

Lastly, the Teacher_J+M_train trained on J+M_train actually performs worse on
both J_test (82%) and M_test (82%) than their non-joined teacher (87% and
85%). These results are surprising and remain unexplained.

Performance of the students.

We now test the students. It is important to note that students are always
evaluated in a cross-dataset scenario since the DALI dataset (on which they
have been trained) does not contain any track from Jamendo or MedleyDB. Hence,
there is no possible overfitting for those. Our hypothesis is that students
achieve better the results than the teachers because they have seen more data.
Especially, we assume that their generalization to unseen data will be better.

This is true for the performances obtained with the student based on
Teacher_M_train . When applied to J_test, it reaches 80% which is higher than
the performances of the Teacher_M_train directly (76%).

This is also true for the performances computed with the student based on
Teacher_J+M_train. When applied either to J_test or M_test, it reaches 86.5%
(86% on Jamendo and 86% on MedleyDB) which is above the Teacher_J+M_train
(82%). Also, 86.5% is similar or above the results obtained with
Teacher_J_train on J_test (87%) and Teacher_M_train on M_test (85%). This is a
very interesting result that demonstrates the generalization of the student
system whichever data-set it is applied to. The student based on
Teacher_J+M_train is the one used for defining the final 5358 songs of the
DALI dataset.

However, the performances obtained with the student based on Teacher_J_train
applied to M_test (82%) do not improve over the direct use of the
Teacher_J_train (82%).

On alignment.

Not explained in this paper is the fact that the $\hat{fr}$ and $\hat{o}$
values computed with the students are much better (almost perfect) than the
ones obtained with the teacher. However, we cannot measure it precisely since
DALI dataset does not have ground-truth label annotations to that end. Indeed,
the goal of this paper is exactly to obtain such annotations automatically.

##  4 Singing voice Dataset: access

The DALI dataset can be downloaded at https://github.com/gabolsgabs/DALI.
There, we provide the detailed desciption of the dataset as well as all the
necessary information for using it. DALI is presented under the recommendation
made by [19] for the description of MIR corpora. The current version of DALI
is 1.0. Future updates will be detailed in the website.

##  5 Conclusion and future works

In this paper we introduced DALI, a large and rich multimodal dataset
containing 5358 audio tracks with their time-aligned vocal melody notes and
lyrics at four levels of granularity.

We explained our methodology where dataset creation and learning models
interact using a teacher-student paradigm benefiting one-another. From manual
karaoke user annotations of time-aligned lyrics and notes, we found a set of
matching audio candidates from the Web. To select and align the best
candidate, we compare the annotated vocal sequence (corresponding to the
lyrics) to the singing voice probability (obtained with a ConvNet). To improve
the latter (and therefore obtain a better selection and alignment) we applied
a teacher-student paradigm.

Through an experiment, we proved that the students outperform the teachers
notably in a cross-dataset scenario, when train-set and test-set are from
different datasets.

It is important to note that the results of the students are higher than the
teacher ones, even if they have been training on imperfect data. In our case,
we showed that, in the context of deep learning, it is better to have
imperfect but large dataset rather than small and perfect ones. However, other
works went in the opposite direction [1].

Future work. We have only performed the teacher-student iteration once. In
next works will use the results of the first student generations to train a
second student generations. This will define a new DALI dataset. We plan to
quantitative measure the quality of $\hat{o},\hat{fr}$ and to continue
exploring the alignments between note annotations and the audio. Currently, we
trained our student using as target $p=avs$, which do not transfer directly
the knowledge of the teacher. We will explore other possibilities of knowledge
transfer using other targets (points a) and c) in Section 3.4) as well as the
local problems describe at Section 3.1.

Acknowledgement. This research has received funding from the French National
Research Agency under the contract ANR-16-CE23-0017-01 (WASABI project).

## References

  * [1] Xavier Amatriain.  ”in machine learning, is more data always better than better algorithms?”.  https://bit.ly/2seQzj9.
  * [2] A. Ashok, N. Rhinehart, F. Beainy, and K. M. Kitani.  N2N learning: Network to network compression via policy gradient reinforcement learning.  CoRR, 2017.
  * [3] K. Benzi, M. Defferrard, P. Vandergheynst, and X. Bresson.  FMA: A dataset for music analysis.  CoRR, abs/1612.01840, 2016.
  * [4] A. Berenzweig, D. P. W. Ellis, and S. Lawrence.  Using voice segments to improve artist classification of music.  In AES 22, 2002.
  * [5] A. L. Berenzweig and D. P. W. Ellis.  Locating singing voice segments within music signals.  In WASPAA, pages 119–122, 2001.
  * [6] R. Bittner, J. Salamon, M. Tierney, M. Mauch, C. Cannam, and J. Bello.  Medleydb: A multitrack dataset for annotation-intensive mir research.  In ISMIR, 2014.
  * [7] A. Cont, D. Schwarz, N. Schnell, and C. Raphael.  Evaluation of Real-Time Audio-to-Score Alignment.  In ISMIR, Vienna, Austria, 2007.
  * [8] J. Cui, B. Kingsbury, B. Ramabhadran, G. Saon, T. Sercu, K. Audhkhasi, A. Sethy, M. Nussbaum-Thom, and A. Rosenberg.  Knowledge distillation across ensembles of multilingual models for low-resource languages.  In ICASSP, 2017.
  * [9] E. Fonseca, J. Pons, X. Favory, F. Font, D. Bogdanov, A. Ferraro, S. Oramas, A. Porter, and X. Serra.  Freesound datasets: A platform for the creation of open audio datasets.  In ISMIR, Suzhou, China, 2017.
  * [10] H. Fujihara and M. Goto.  Lyrics-to-Audio Alignment and its Application.  In Multimodal Music Processing, volume 3 of Dagstuhl Follow-Ups, pages 23–36. Dagstuhl, Germany, 2012.
  * [11] H. Fujihara, M. Goto, J. Ogata, and H. G. Okuno.  Lyricsynchronizer: Automatic synchronization system between musical audio signals and lyrics.  5(6):1252–1261, 2011.
  * [12] M. Goto.  Singing information processing.  In ICSP, pages 2431–2438, 2014.
  * [13] E. J. Humphrey, N. Montecchio, R. Bittner, A. Jansson, and T. Jehan.  Mining labeled data from web-scale collections for vocal activity detection in music.  In ISMIR, 2017.
  * [14] S. Leglaive, R. Hennequin, and R. Badeau.  Singing voice detection with deep recurrent neural networks.  In IEEE, editor, ICASSP, pages 121–125, Brisbane, Australia, 2015\.
  * [15] B. Lehner, G. Widmer, and S. Bock.  A low-latency, real-time-capable singing voice detection method with lstm recurrent neural networks.  In 2015 23rd European Signal Processing Conference (EUSIPCO), 2015\.
  * [16] M. Mauch, H. Fujihara, K. Yoshii, and M. Goto.  Timbre and melody features for the recognition of vocal activity and instrumental solos in polyphonic music.  In ISMIR 2011, pages 233–238, 2011.
  * [17] A. Mesaros.  Singing voice identification and lyrics transcription for music information retrieval invited paper.  In 7th Conference on Speech Technology and Human - Computer Dialogue (SpeD), pages 1–10, 2013.
  * [18] G. Meseguer-Brocal, G. Peeters, G. Pellerin, M. Buffa, E. Cabrio, C. Faron Zucker, A. Giboin, I. Mirbel, R. Hennequin, M. Moussallam, F. Piccoli, and T. Fillon.  WASABI: a Two Million Song Database Project with Audio and Cultural Metadata plus WebAudio enhanced Client Applications.  In Web Audio Conf., London, U.K., 2017. Queen Mary University of London.
  * [19] G. Peeters and K. Fort.  Towards a (better) Definition of Annotated MIR Corpora.  In ISMIR, Porto, Portugal, 2012.
  * [20] M. Ramona, G. Richard, and B. David.  Vocal detection in music with support vector machines.  In Proc. ICASSP ’08, 2008.
  * [21] L. Regnier and G. Peeters.  Singing Voice Detection in Music Tracks using Direct Voice Vibrato Detection.  In ICASSP, page 1, taipei, Taiwan, 2009.
  * [22] J. Salamon and E. Gómez.  Melody extraction from polyphonic music signals using pitch contour characteristics.  IEEE Transactions on Audio, Speech and Language Processing, 20:1759–1770, 2012.
  * [23] J. Schlüter.  Learning to pinpoint singing voice from weakly labeled examples.  In ISMIR, New York City, USA, 2016. ISMIR.
  * [24] J. Schlüter and T. Grill.  Exploring Data Augmentation for Improved Singing Voice Detection with Neural Networks.  In ISMIR 2015, Malaga, Spain, 2015.
  * [25] A. J. R. Simpson, G. Roma, and M. D. Plumbley.  Deep karaoke: Extracting vocals from musical mixtures using a convolutional deep neural network.  abs/1504.04658, 2015.
  * [26] F. Soulez, X. Rodet, and D. Schwarz.  Improving polyphonic and poly-instrumental music to score alignment.  In ISMIR, page 6, Baltimore, United States, 2003.
  * [27] S. Watanabe, T. Hori, J. Le Roux, and J. Hershey.  Student-teacher network learning with enhanced features.  In ICASSP, pages 5275–5279, 2017.
  * [28] C. Wu and A. Lerch.  Automatic drum transcription using the student-teacher learning paradigm with unlabeled music data.  In ISMIR, 2017.