# Line Segment Tracking: Improving the Phase 2 CMS High Level Trigger Tracking
with a Novel, Hardware-Agnostic Pattern Recognition Algorithm

E Vourliotis1a and P Chang2 P Elmer3 Y Gu1 J Guiang1 V Krutelyov1 B V Sathia
Narayanan1 G Niendorf4 M Reid4 M Silva2 A Rios Tascon3 M Tadel1 P Wittich4 A
Yagil1
on behalf of the CMS Collaboration  1University of California San Diego, CA,
US  2University of Florida, FL, US  3Princeton University, NJ, US  4Cornell
University, NY, US<EMAIL_ADDRESS>
###### Abstract

Charged particle reconstruction is one the most computationally heavy
components of the full event reconstruction of Large Hadron Collider (LHC)
experiments. Looking to the future, projections for the High Luminosity LHC
(HL-LHC) indicate a superlinear growth for required computing resources for
single-threaded CPU algorithms that surpass the computing resources that are
expected to be available. The combination of these facts creates the need for
efficient and computationally performant pattern recognition algorithms that
will be able to run in parallel and possibly on other hardware, such as GPUs,
given that these become more and more available in LHC experiments and high-
performance computing centres. Line Segment Tracking (LST) is a novel such
algorithm which has been developed to be fully parallelizable and hardware
agnostic. The latter is achieved through the usage of the Alpaka library. The
LST algorithm has been tested with the CMS central software as an external
package and has been used in the context of the CMS HL-LHC High Level Trigger
(HLT). When employing LST for pattern recognition in the HLT tracking, the
physics and timing performances are shown to improve with respect to the ones
utilizing the current pattern recognition algorithms. The latest results on
the usage of the LST algorithm within the CMS HL-LHC HLT are presented, along
with prospects for further improvements of the algorithm and its CMS central
software integration.

##  1 Motivation and the Line Segment Tracking Algorithm

The High Luminosity Large Hadron Collider (HL-LHC) is the planned upgrade of
the Large Hadron Collider (LHC) of CERN, with the target to collect data of
proton-proton collisions corresponding to an integrated luminosity of more
than 3000$\,\text{fb}^{-1}$ [1]. This can only be achieved by considerably
enhancing the instantaneous luminosity, which, in turn, implies a drastic
increase in the number of simultaneous collisions (pileup, PU). Because of
this, the computational complexity of event reconstruction is projected to
exceed the available computing resources, especially for the highly
combinatorial task of trajectory pattern recognition of charged particles.
This leads to both an increased timing, jeopardizing the ability to
reconstruct the data at the desired rate, and an increased cost due to the
higher demands for processing power.

To accommodate the HL-LHC conditions, the LHC experiments are planning a major
upgrade of their software and hardware infrastructure (Phase 2). The Line
Segment Tracking (LST) algorithm aims at improving and parallelizing on
Graphics Processing Units (GPUs) the charged hadron trajectory pattern
recognition of the Phase 2 CMS experiment [2]. It uses as input the hits of
the CMS Phase 2 outer tracker (OT) [3] and associates them to inner tracker
(IT) tracks, ultimately producing a collection of OT+IT and OT-only track
candidates. The early stages of the algorithm rely to a significant degree on
the characteristics of CMS Phase 2 OT, qualitatively shown in Fig. 1: one of
the key aspects of its design is that each layer comprises of 2 closely-spaced
silicon sensors. In this way, two hits are recorded on each layer and are
linked by the LST algorithm in a single pair of hits, called Mini Doublet
(MD). The MDs are useful in reducing the combinatorics for the trajectory
patterns and have the advantage of being locally reconstructed, which is
utilized by the LST algorithm to parallelize their creation. Another handle
for the reduction of the combinatorics is the definition of a lower
$p_{\mathrm{T}}$ threshold for track reconstruction by tuning the search
window for hit pairs. The current lower $p_{\mathrm{T}}$ threshold for LST is
set at 0.8$\,\mathrm{GeV}$.

Figure 1: A qualitative representation of the expected Phase-2 CMS tracker
geometry [3].

MDs serve as the elementary building blocks for the LST algorithm to create
tracks. Based on precomputed connection maps for modules in the IT and OT,
fulfilling geometric criteria that physical charged particle patterns obey,
two MDs are linked to create a Line Segment (LS). The selections used to
create LST objects are described in Ref. [4], and incorporate also machine
learning methods, as detailed in Ref. [5]. Two LSs with a common MD are
subsequently linked to form a T3, and two T3s with a common MD are linked to
form a T5. T3s and T5s are combined with IT tracks (pLSs) to create pT3s and
pT5s. Since the reconstruction of the above objects only requires local
information, it can be massively parallelized, as all objects of the same kind
can be created concurrently. Out of those objects, only a subset is propagated
to downstream algorithms to be made into full tracks:

  * •

pT5s, providing the majority of the efficiency.

  * •

pT3s, complementing the pT5 efficiency.

  * •

T5s, enabling the reconstruction of displaced tracks.

  * •

Unlinked pLSs, covering the track reconstruction at high $|\eta|$, where there
is no OT for the LST algorithm to create other objects.

The objects created by the LST algorithm are summarized in Fig. 2. Previously,
the LST algorithm performance had been demonstrated in the offline
reconstruction setup in Ref. [6]. It is worth noting that the LST algorithm is
written using the Alpaka abstraction framework [7, 8, 9], therefore it
seamlessly runs on multiple hardware devices.

(a) LS

(b) T3

(c) T5

(d) pT3

(e) pT5

Figure 2: A qualitative representation of the different objects created by the
LST algorithm [10].

##  2 Tracking in the Phase 2 CMS High Level Trigger

The High Level Trigger (HLT) is one of the two tiers of the system that
collect the events of interest for CMS. It consists of a farm of processors
running a version of the full event reconstruction software optimized for fast
processing, and reduces the event rate before data storage. As it is
responsible for the acquisition of the data as they are produced by the
(HL-)LHC, timing plays an important role for the algorithms it runs. Apart
from that, it needs to be general enough to cover for the majority of the
potential physics signals.

The pattern recognition of tracks is of major importance for the HLT, as it is
the basis of the reconstruction of most of the physics particles, while it
needs to be run at a disproportionately short time for the combinatorial
complexity of the task, especially at the harsh PU of HL-LHC. The CMS Phase 2
HLT uses the Combinatorial Kalman Filter (CKF) to reconstruct tracks with
$p_{\mathrm{T}}>0.9\,\mathrm{GeV}$ [11]. The track reconstruction is performed
in two stages (iterations) based on different sets of initial track
estimations (track seeding): the initial step produces tracks from pixel seeds
with at least 4 hits (quads) created by the Patatrack algorithm [12, 13],
while the highPtTriplet step produces tracks from pixel seeds with 3 hits
(triplets), created by the legacy pixel seeding algorithm [14]. It is worth
noting that the CKF algorithm used for the trajectory pattern recognition
(track building) in this configuration is inherently sequential, and is
implemented on Central Processing Unit (CPU). Once the built tracks, i.e. the
set of hits originating from the same track, have been identified, these
undergo a fitting procedure to extract the final track parameters, and are
selected with requirements based on those parameters (tracking ID). The
“highPurity” selection is applied, which provides a good balance between high
efficiency and low fake+duplicate rate for prompt tracks [14]. This baseline
configuration will be mentioned below as “Base CKF”.

The LST algorithm can be an ideal candidate to run at the CMS Phase 2 HLT. LST
allows for the parallel processing of track reconstruction on GPUs, hence
keeping the timing under control, while extending the physics acceptance of
the HLT to displaced tracks. In the following, a few potential configurations
for integrating the LST algorithm in the CMS Phase 2 HLT are documented. The
LST algorithm is utilized as a replacement for track building for the initial
step, using pixel seeds with at least 3 hits as pLSs. Since the highPurity
tracking ID has been optimized for prompt tracks, it is not applied to the LST
objects targeting displaced tracks (T5s). This leads to a high efficiency for
displaced tracks, without any significant increase in the fake and duplicate
rate. Finally, as mentioned above, LST cannot build tracks for $|\eta|\gtrsim
2.5$, as that region is outside of the OT acceptance (Fig. 1). As a result,
the CKF algorithm still needs to run in the highPtTriplet step to recover
efficiency in those high $|\eta|$ regions. Different configurations are being
used for the seeding of this “recovery iteration”: in the “LST with CKF on
Legacy Triplets” configuration, legacy triplets are used, in the “LST with CKF
on LST Quads” configuration only the quad LST pLSs are used, while in the “LST
with CKF on LST Quads+Triplets” configuration both the quad and triplet LST
pLSs are used. The last two configuration imply that the LST algorithm can be
used not only as a track building but also as a track seeding algorithm. All
of the configurations described above are summarized in a more condensed
format in Table 1.

Table 1: Summary of the HLT tracking sequence setup for each configuration
described in this note [10].

Iteration | Procedure | Base CKF | LST with CKF on | LST with CKF on | LST with CKF on
---|---|---|---|---|---
Legacy Triplets | LST Quads | LST Quads+Triplets
| Seeding | Patatrack quads | Patatrack quads + | Patatrack quads + | Patatrack quads +
Legacy Triplets | Legacy Triplets | Legacy Triplets
Initial | Building | CKF | LST | LST | LST
| Tracking ID | highPurity | highPurity (pT3, pT5, pLS) | highPurity (pT3, pT5) | highPurity (pT3, pT5)
None (T5) | None (T5) | None (T5)
| Seeding | Legacy triplets | Legacy triplets | LST pLS quads | LST pLS
quads+triplets
HighPt | Building | CKF | CKF | CKF | CKF
Triplet
| Tracking ID | highPurity | highPurity | highPurity | highPurity

##  3 Physics Performance and Throughput

This section is dedicated to the measurement of the physics and computational
performance of the CMS Phase 2 HLT configurations using LST for track
reconstruction. Three metrics are used for the physics performance:

  * •

Efficiency: The fraction of the matched simulated tracks from the hard-
scattering vertex.

  * •

Fake rate: The fraction of reconstructed tracks not matched to any simulated
track.

  * •

Duplicate rate: The fraction of reconstructed tracks matched to any simulated
track that is matched to multiple reconstructed tracks.

For the measurement of the efficiency, the simulated tracks from the hard-
scattering vertex are matched to the reconstructed tracks. A given simulated
track is considered a match to a reconstructed one if more than 75% of the
hits of the reconstructed track originate from the simulated track. For the
measurement of the fake and the duplicate rate, all simulated tracks are used
for the matching. In the following, any selections applied to the simulated or
reconstructed tracks (depending on the metric, as described above) are shown
on the plots. The radial distance and the z position of the production vertex
of the tracks are denoted as $\text{r}_{\text{vertex}}$ and
$\text{z}_{\text{vertex}}$ respectively. A simulated $t\bar{t}$ sample
produced with 200 PU for the upgraded Phase 2 detector geometry is used for
the measurements below.

Figure 3 shows the efficiency of the different configurations tested as a
function of the simulated track $p_{\mathrm{T}}$ (left) and
$\text{r}_{\text{vertex}}$ (right). It is obvious that the efficiency is lower
when using only quads in the recovery iteration, highlighting the importance
of triplets for track seeding in the current setup. When triplets are used,
the LST configurations reach an efficiency that is comparable to the one by
Base CKF, or even higher for $p_{\mathrm{T}}\lesssim 5\,\mathrm{GeV}$. The
efficiency as a function of $\text{r}_{\text{vertex}}$ demonstrates the fact
that any configuration using LST for track building allows for acceptance of
displaced tracks ($\text{r}_{\text{vertex}}\gtrsim 5~{}\text{cm}$). This
constitutes a completely new feature for the CMS HLT. Notably, the efficiency
drops at the radial distances roughly corresponding to tracker layers, with an
endpoint at $\sim\\!35~{}\text{cm}$, where less than 4 OT layers are
available, so no T5s can be built.

The fake rate is lower for any configuration using LST for track building.
Importantly, the right plot of Fig. 4 shows that most of the fake rate
reduction comes for tracks with $p_{\mathrm{T}}<10\,\mathrm{GeV}$. Given that
the bulk of tracks have low $p_{\mathrm{T}}$, this implies a significant
reduction of computational resources downstream, as less tracks need to be
processed. The left plot of Fig. 4 indicates a higher duplicate rate when the
recovery CKF iterations runs on legacy triplets, as both legacy triplets and
LST pLSs have the potential to reconstruct the same track. When only the LST
pLSs are used for track seeding, the overall duplicate rate is lower
throughout the whole $p_{\mathrm{T}}$ range.

Figure 3: The tracking efficiency is shown for Base CKF (blue), LST with CKF
on Legacy Triplet (red), LST with CKF on LST Quads (orange) and LST with CKF
on LST Quads+Triplets (purple) as a function of the simulated track
$p_{\mathrm{T}}$ (left) and $\text{r}_{\text{vertex}}$ (right) [10].

Figure 4: The tracking fake rate (left) and duplicate rate (right) are shown
for Base CKF (blue), LST with CKF on Legacy Triplet (red), LST with CKF on LST
Quads (orange) and LST with CKF on LST Quads+Triplets (purple) as a function
of the reconstructed track $p_{\mathrm{T}}$ [10].

Based on the computational performance of the current offline tracking
reconstruction, displaced tracking takes as much time as the prompt track
reconstruction. As displaced tracking is completely missing from the current
CMS Phase 2 HLT configuration, its addition would imply a 50% reduction of the
throughput. Table 2 shows the throughput of the tracking sequence for CMS
Phase 2 HLT configurations using LST, normalized to that of the Base CKF
configuration. The measurements were performed with 2 threads (for CPU),
pinned to 2 specific CPU cores, and 2 streams (for GPU) with local access to
the input. An AMD EPYC “Milan” 7763 CPU and an NVIDIA “Ampere” A30 PCIe GPU
were used. The results imply that a throughput reduction of at most 30% is
expected when running LST, hence including displaced tracking and improving
various performance metrics, as outlined above. The throughput reduction comes
only for the LST configuration that run single-threaded on CPU. When the LST
configurations are executed on GPU, the throughput is comparable with the Base
CKF one or even increases. It was observed that most of the slowdown for the
LST configurations is actually coming from the recovery iteration, which is
run with CKF.

Table 2: Throughput of the HLT tracking sequence for different configurations, normalized to that of the Base CKF one [10]. | LST with CKF on | LST with CKF on | LST with CKF on
---|---|---|---
Legacy Triplets | LST Quads | LST Quads+Triplets
LST on CPU | $0.72\pm 0.07$ | $0.86\pm 0.07$ | $0.70\pm 0.09$
Throughput / Base CKF
LST on GPU | $1.03\pm 0.09$ | $1.35\pm 0.12$ | $0.92\pm 0.09$
Throughput / Base CKF

##  4 Summary and Outlook

LST is a novel, hardware-agnostic pattern recognition algorithm, targeting
application to the CMS Phase 2 tracking. The algorithm can bring improvements
both to the physics performance, as it extends the acceptance of the current
tracking implementation to displaced tracks, and to the computational
performance, as it efficiently runs on GPUs, potentially increasing the event
reconstruction throughput. As such, it is a suitable candidate for performing
the charged particle trajectory pattern recognition at the CMS Phase 2 HLT,
where both the physics acceptance and timing considerations are of utmost
importance. This work presented the first exploratory integration of the LST
algorithm in the CMS Phase 2 HLT, demonstrating a lot of potential for the
future.

On top of the improvements showcased above, more developments are planned both
for the LST algorithm and for the CMS Phase 2 HLT configuration. The former
involve the creation of more objects, the integration of more machine learning
methods and the optimization of its implementation on CPU for LST, while the
latter revolve around the optimization of the usage of the Patatrack algorithm
for track seeding and the usage of the mkFit algorithm [15, 16] for the track
building of the recovery iteration.

## Acknowledgements

This work was supported by the U.S. National Science Foundation under
Cooperative Agreements OAC- 1836650, PHY-2323298, and PHY-2121686 and grant
PHY-2209443.

## References

  * [1] Aberle O et al 2020 High-Luminosity Large Hadron Collider (HL-LHC): Technical design report CERN-2020-010 URL https://cds.cern.ch/record/2749422
  * [2] CMS Collaboration 2008 Journal of Instrumentation 3 S08004
  * [3] CMS Collaboration 2017 The Phase-2 Upgrade of the CMS Tracker CMS-TDR-014 URL https://cds.cern.ch/record/2272264
  * [4] Chang P, Elmer P, Gu Y, Krutelyov V, Niendorf G, Reid M, Narayanan B V S, Tadel M, Vourliotis E, Wang B, Wittich P and Yagil A 2022 Journal of Physics: Conference Series 2375 012005
  * [5] CMS Collaboration 2023 Improved Performance of Line Segment Tracking Using Machine Learning CMS-DP-2023-075 URL https://cds.cern.ch/record/2872904
  * [6] CMS Collaboration 2023 Performance of Line Segment Tracking algorithm at HL-LHC CMS-DP-2023-019 URL https://cds.cern.ch/record/2857438
  * [7] Worpitz B 2015 Investigating performance portability of a highly scalable particle-in-cell simulation code on various multi-core architectures URL http://dx.doi.org/10.5281/zenodo.49768
  * [8] Zenker E, Worpitz B, Widera R, Huebl A, Juckeland G, Knüpfer A, Nagel W E and Bussmann M 2016 Alpaka - an abstraction library for parallel kernel acceleration (IEEE Computer Society) (Preprint 1602.08477) URL http://arxiv.org/abs/1602.08477
  * [9] Matthes A, Widera R, Zenker E, Worpitz B, Huebl A and Bussmann M 2017 Tuning and optimization for a variety of many-core architectures without changing a single line of implementation code using the alpaka library (Preprint 1706.10086) URL https://arxiv.org/abs/1706.10086
  * [10] CMS Collaboration 2024 Performance of the Line Segment Tracking Algorithm in the CMS Phase-2 High Level Trigger Tracking CMS-DP-2024-014 URL https://cds.cern.ch/record/2890677
  * [11] CMS Collaboration 2021 The Phase-2 Upgrade of the CMS Data Acquisition and High Level Trigger CMS-TDR-022 URL https://cds.cern.ch/record/2759072
  * [12] Bocci A, Innocente V, Kortelainen M, Pantaleo F and Rovere M 2020 Frontiers in Big Data 3
  * [13] CMS Collaboration 2022 Performance of Run-3 HLT Track Reconstruction CMS-DP-2022-014 URL https://cds.cern.ch/record/2814111
  * [14] CMS Collaboration 2014 Journal of Instrumentation 9 P10009
  * [15] Lantz S, McDermott K, Reid M, Riley D, Wittich P, Berkman S, Cerati G, Kortelainen M, Reinsvold A H, Elmer P, Wang B, Giannini L, Krutelyov V, Masciovecchio M, Tadel M, Würthwein F, Yagil A, Gravelle B, Norris B 2020 Journal of Instrumentation 15 P09030
  * [16] CMS Collaboration 2022 Performance of Run 3 track reconstruction with the mkFit algorithm CMS-DP-2022-018 URL https://cds.cern.ch/record/2814000


# SSM-Net: feature learning for Music Structure Analysis using a Self-
Similarity-Matrix based loss

###### Abstract

In this paper, we propose a new paradigm to learn audio features for Music
Structure Analysis (MSA). We train a deep encoder to learn features such that
the Self-Similarity-Matrix (SSM) resulting from those approximates a ground-
truth SSM. This is done by minimizing a loss between both SSMs. Since this
loss is differentiable w.r.t. its input features we can train the encoder in a
straightforward way. We successfully demonstrate the use of this training
paradigm using the Area Under the Curve ROC (AUC) on the RWC-Pop dataset.

##  1 Introduction

Music Structure Analysis (MSA) is the task aiming at identifying musical
segments that compose a music track (a.k.a. segment boundary estimation) and
possibly label them based on their similarity (a.k.a. segment labeling). Over
the years, systems for MSA have switched from

  * •

hand-crafted detection system (checker-board-kernel [1] or DTW [2]) applied to
hand-crafted audio features (MFCC or Chroma)

  * •

to deep learning detection system (boundary detection using ConvNet [3, 4, 5])
applied to hand-crafted audio features, and recently

  * •

to hand-crafted detection system (checker-board-kernel) applied to deep
learned features [6, 7].

Among the paradigms used to learn these features, metric learning using the
triplet loss [8] has been the most popular, either using unsupervised learning
[6] or using supervised learning [7]. In this paper, we propose a new paradigm
to learn these features, which is more straightforward and less-
computationally expensive (on a GPU Tesla P100-PCIE, training in about 1 hour
for our approach and 24 hours for [6]).

##  2 Proposal: SSM-Net

Our SSM-net system is illustrated in Figure 1. The inputs and architecture
(but not the loss) of our system are inspired by McCallum’s work [6] (but
largely simplified111We reduced the sampling rate of the features by a factor
8: McCallum divides each beat into 128 sub-beats while we only use 16 sub-
beats. We divided by a factor 2 the number of convolutional filters of each
layer and we removed the last two fully connected layers.).

Figure 1: SSM-net architecture. From left to right: input sequence
$\\{\mathbf{X}_{i}\\}$ of beat-synchronous CQT-patches, encoder $f^{\theta}$
applied to each $\mathbf{X}_{i}$, estimated SSM
$\mathbf{\hat{S}}_{ij}^{\theta}$ computed with embeddings
$\\{\mathbf{e}^{\theta}_{i}\\}=f^{\theta}(\\{\mathbf{X}_{i}\\})$, Loss
$\mathcal{L}^{\theta}$ computation. Figure 2: [Left] SSMs
$\mathbf{\hat{S}}_{ij}^{\theta}$ computed using embeddings
$\mathbf{e}^{\theta}$ obtained using (from left to right): cqt, mccallum-
biased, ssmnet and ground-truth SSM $\mathbf{S}_{ij}$, on track 2 from RWC-Pop
dataset. Loss $\mathcal{L}$ and AUC are indicated on top of each. [Right] Box-
plots of Loss $\mathcal{L}$ and AUC obtained for all tracks of RWC-Pop
dataset.

Input data $\\{\mathbf{X}_{i}\\}$. Each audio track is represented as a
temporal sequence of $T$ audio features $\mathbf{X}_{i}$ which we denote as
$\\{\mathbf{X}_{i}\\}_{i\in\\{1\ldots T\\}}$ or $\\{\mathbf{X}_{i}\\}$ for
short. $\\{\mathbf{X}_{i}\\}$ are beat-synchronized patches of Constant-Q-
Transform (CQT), each centered on a beat position $b_{i}$222The CQTs are
computed using librosa [9]. We used 72 log-frequencies ranging from C1 (31.70
Hz) to C7 (2093 Hz).. Each patch represents 4 successive beats333 The beat
positions $\\{b_{i}\\}$ are computed using madmom [10][11].. Each beat is
further sub-divided into 16 sub-beats. For this, the content of the CQTs
between two successive beats $b_{i-1}$ and $b_{i}$ is analyzed and
clustered444using constrained agglomerative clustering and median aggregation
as implemented in librosa.segment.subsegment. into 16. The inputs to our
network are therefore patches $\mathbf{X}_{i}$ of CQT, each of size (72
frequencies, 4*16 sub-beats) and centered on a beat $b_{i}$.

Network architecture $\mathbf{e}^{\theta}_{i}=f^{\theta}(\mathbf{X}_{i})$. The
architecture of our encoder $f^{\theta}$ is illustrated in Figure 1. It
comprises 3 consecutive blocks (L1, L2, L3) of a 2D convolution followed by a
SELU [12] activation, a 2D group normalization [13] with 32 channels and a 2D
max-pooling, The convolutional layers use a kernel size of (f=6, t=4)555f and
t denotes the frequency and time dimensions and the max-pooling layers use
respectively kernel sizes of (2, 4), (3, 4) and (3, 2). The output is then
passed to a single Fully-Connected (FC) layer of 128 units with a SELU
activation. The output is then L2-normalized and constitutes the embedding
$\mathbf{e}^{\theta}_{i}=f^{\theta}(\mathbf{X}_{i})$. $\theta$ denotes the set
of parameters to be trained (348.400 parameters). For comparison the original
McCallum [6] network has 1.280.768.

SSM-Net Loss. We apply the same encoder $f^{\theta}$ to each input
$\mathbf{X}_{i}$. We then obtain the corresponding sequence of embeddings
$\\{\mathbf{e}^{\theta}_{i}\\}_{i\in\\{1\ldots
T\\}}=f^{\theta}(\\{\mathbf{X}_{i}\\}_{i\in\\{1\ldots T\\}})$. We can then
easily construct an estimated SSM, $\mathbf{\hat{S}}_{ij}^{\theta}$, using a
distance/similarity $g$ function between all pairs of projections:

$\mathbf{\hat{S}}_{ij}^{\theta}=g(\mathbf{e}^{\theta}_{i}=f^{\theta}(\mathbf{X}_{i}),\mathbf{e}^{\theta}_{j}=f^{\theta}(\mathbf{X}_{j})),\;\;\;\forall
i,j$ (1)

$g$ is here a simple cosine-similarity which we scale to $[0,1]$:

$\mathbf{\hat{S}}_{ij}^{\theta}=1-\frac{1}{4}\lVert\mathbf{e}^{\theta}_{i}-\mathbf{e}^{\theta}_{j}\rVert_{2}^{2}\;\;\;\in[0,1]$
(2)

It is then possible to compare $\mathbf{\hat{S}}_{ij}^{\theta}$ to a ground-
truth binary SSM, $\mathbf{S}_{ij}$. We formulate this as a multi-class
problem (a set of $T^{2}$ binary classifications) and minimize the sum of
Binary-Cross-Entropy (BCE) losses. We compensate the class unbalancing by
using a weighting factor $\lambda$ computed as the percentage of 1 values in
$\mathbf{S}_{ij}$.

$\mathcal{L}^{\theta}\\!=\\!-\\!\sum_{i,j=1}^{T}\\!(1\\!-\lambda)\\!\left[\mathbf{S}_{ij}\\!\log(\\!\mathbf{\hat{S}}_{ij}^{\theta}\\!)\\!\right]\\!+\\!\lambda\\!\left[\\!(\\!1\\!-\\!\mathbf{S}_{ij}\\!)\\!\log\\!(\\!1\\!-\\!\mathbf{\hat{S}}_{ij}^{\theta}\\!)\\!\right]$
(3)

Since the computation of the SSM $\mathbf{\hat{S}}_{ij}^{\theta}$ is
differentiable w.r.t. to the embeddings $\\{\mathbf{e}^{\theta}_{i}\\}$, we
can compute $\frac{\partial\mathcal{L}^{\theta}}{\partial\theta}$

$\frac{\partial\mathcal{L}^{\theta}}{\partial\theta}=\sum_{i,j=1}^{T}\frac{\partial\mathcal{L}^{\theta}}{\partial\mathbf{\hat{S}}_{ij}^{\theta}}\left(\frac{\partial\mathbf{\hat{S}}_{ij}^{\theta}}{\partial\mathbf{e}^{\theta}_{i}}\frac{\partial\mathbf{e}^{\theta}_{i}}{\partial\theta}+\frac{\partial\mathbf{\hat{S}}_{ij}^{\theta}}{\partial\mathbf{e}^{\theta}_{j}}\frac{\partial\mathbf{e}^{\theta}_{j}}{\partial\theta}\right)$
(4)

Training. We minimize the loss using MADGRAD [14] with a learning rate of
$5\\!\times\\!10^{-4}$, a weight decay of $10^{-2}$ and early-stopping. The
mini-batch-size $m$ (here defined as the number of full-tracks) is set to 6.

Generating a ground-truth SSM $\mathbf{S}_{ij}$. To generate
$\mathbf{S}_{ij}$, we rely on the homogeneity assumption, i.e. we suppose that
all $t_{i}$ that fall within an annotated segment are identical since they
share the same label. If we denote by $\text{seg}(t_{i})$ the segment $t_{i}$
belongs to and by $\text{label}(\text{seg}(t_{i}))$ its label, we assign the
value $\mathbf{S}_{ij}=1$ if
$\text{label}(\text{seg}(t_{i}))=\text{label}(\text{seg}(t_{j}))$.

##  3 Evaluation

To evaluate the quality of the features independently of the choice of a
specific detection algorithm for MSA, we directly compare the ground-truth
$\mathbf{S}_{ij}$ and the $\mathbf{\hat{S}}_{ij}^{\theta}$ obtained using
various choices for $\mathbf{e}^{\theta}$. For each choice, we measure the
obtained Loss $\mathcal{L}$ (lower is better) and AUC (higher is better) . We
conside the following features $\mathbf{e}^{\theta}$:

  * •

cqt: the flattened CQT patches $\\{\mathbf{X}_{i}\\}$

  * •

convnet: the output of the un-trained (random weight) encoder $f^{\theta}$
applied to $\\{\mathbf{X}_{i}\\}$

  * •

ssmnet: the output of $f^{\theta}$ trained with SSM-Net

  * •

mccallum-normal/biased: the output of the same encoder $f^{\theta}$ but
trained using the two unsupervised metric learning approaches described in [6]

  * •

ssmnet-mccallum-normal/biased: same as for ssmnet but $f^{\theta}$ is pre-
trained using mccallum-normal/biased

To train our SSM-Net, , we used a sub-set of 695 tracks from the labeled
dataset Harmonix [15]. To train the unsupervised metric learning approach
described in [6], we used a large unlabeled dataset from YouTube of 26.000
tracks from various genres.The evaluation is performed on RWC-Pop [16] labeled
with AIST annotations [17]).

In Figure 2 [Left], we give an example of the SSM
$\mathbf{\hat{S}}_{ij}^{\theta}$ obtained using the embeddings
$\mathbf{e}^{\theta}$ learned by the most representative approaches. On this
example, ssmnet gives the $\mathbf{\hat{S}}_{ij}^{\theta}$ with the highest
contrast and the closest to the ground-truth. It gets a small
$\mathcal{L}$=0.15 and a high AUC=0.81.

In Figure 2 [Right], we represent the box-plots of $\mathcal{L}$ and AUC
considering all tracks of RWC-Pop. As one can see, the SSM-net approach leads
to the lowest $\mathcal{L}$. However McCallum leads to a higher AUC than SSM-
Net. We therefore combine the SSM-Net training with a McCallum pre-training.
This then leads to both a low $\mathcal{L}$ and a high AUC . This is the
approach we will develop in the future.

## References

  * [1] J. Foote, “Automatic audio segmentation using a measure of audio novelty,” in _Proc. of IEEE ICME (International Conference on Multimedia and Expo)_ , New York City, NY, USA, 2000.
  * [2] M. Müller, N. Jiang, and P. Grosche, “A robust fitness measure for capturing repetitions in music recordings with applications to audio thumbnailing,” _Audio, Speech and Language Processing, IEEE Transactions on_ , vol. 21, no. 3, pp. 531–543, 2013.
  * [3] K. Ullrich, J. Schlüter, and T. Grill, “Boundary Detection in Music Structure Analysis using Convolutional Neural Networks,” in _Proc. of ISMIR (International Society for Music Information Retrieval)_ , Taipei, Taiwan, 2014.
  * [4] T. Grill and J. Schlüter, “Music Boundary Detection Using Neural Networks on Combined Features and Two-Level Annotations,” in _Proc. of ISMIR (International Society for Music Information Retrieval)_ , Malaga, Spain, 2015.
  * [5] A. Cohen-Hadria and G. Peeters, “Music Structure Boundaries Estimation Using Multiple Self-Similarity Matrices as Input Depth of Convolutional Neural Networks,” in _AES International Conference on Semantic Audio_ , Erlangen, Germany, June, 22–24, 2017.
  * [6] M. C. McCallum, “Unsupervised Learning of Deep Features for Music Segmentation,” in _Proc. of IEEE ICASSP (International Conference on Acoustics, Speech, and Signal Processing)_ , Brighton, UK, May 2019.
  * [7] J.-C. Wang, J. B. L. Smith, W.-T. Lu, and X. Song, “Supervised metric learning for music structure features,” in _Proc. of ISMIR (International Society for Music Information Retrieval)_ , Online, November, 8–12 2021.
  * [8] F. Schroff, D. Kalenichenko, and J. Philbin, “FaceNet: A unified embedding for face recognition and clustering,” in _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , Jun. 2015, pp. 815–823, iSSN: 1063-6919.
  * [9] B. McFee, C. Raffel, D. Liang, D. P. Ellis, M. McVicar, E. Battenberg, and O. Nieto, “librosa: Audio and music signal analysis in python,” in _Proceedings of the 14th python in science conference_ , vol. 8, 2015.
  * [10] S. Böck and M. Schedl, “Enhanced beat tracking with context aware neural networks,” in _Proc. of DAFx (International Conference on Digital Audio Effects)_ , Paris, France, 2011\.
  * [11] S. Böck, F. Korzeniowski, J. Schlüter, F. Krebs, and G. Widmer, “madmom: a new Python Audio and Music Signal Processing Library,” in _Proceedings of the 24th ACM International Conference on Multimedia_ , Amsterdam, The Netherlands, 2016.
  * [12] G. Klambauer, T. Unterthiner, A. Mayr, and S. Hochreiter, “Self-normalizing neural networks,” in _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , ser. NIPS’17. Red Hook, NY, USA: Curran Associates Inc., 2017, pp. 972–981.
  * [13] Y. Wu and K. He, “Group normalization,” _International Journal of Computer Vision_ , vol. 128, pp. 742–755, 2019.
  * [14] A. Defazio and S. Jelassi, “Adaptivity without Compromise: A Momentumized, Adaptive, Dual Averaged Gradient Method for Stochastic Optimization,” _arXiv:2101.11075 [cs, math]_ , Apr. 2021. [Online]. Available: http://arxiv.org/abs/2101.11075
  * [15] O. Nieto, M. McCallum, M. E. P. Davies, A. Robertson, A. Stark, and E. Egozy, “The Harmonix Set: Beats, Downbeats, and Functional Segment Annotations of Western Popular Music,” in _Proc. of ISMIR (International Society for Music Information Retrieval)_ , Delft, The Netherlands, 2019.
  * [16] M. Goto, “Development of the RWC Music Database,” _Proc. of ICA (18th International Congress on Acoustics)_ , 2004.
  * [17] ——, “Aist annotation for the rwc music database,” in _Proc. of ISMIR (International Society for Music Information Retrieval)_ , Victoria, BC, Canada, 2006, pp. pp.359–360.

  *[MSA]: Music Structure Analysis
  *[SSM]: Self-Similarity-Matrix
  *[AUC]: Area Under the Curve ROC
  *[CQT]: Constant-Q-Transform
  *[FC]: Fully-Connected
  *[BCE]: Binary-Cross-Entropy


2023 July

, , , AND

# The Role of Communication and Reference Songs in the Mixing Process:
Insights from Professional Mix Engineers††thanks: To whom correspondence
should be addressed, e-mail<EMAIL_ADDRESS>
Soumya Sai Vanka1 Maryam Safi2 Jean-Baptiste Rolland2 György Fazekas1
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>1Centre for Digital Music, Queen Mary University
of London, London, UK
2Steinberg Media Technologies GmbH, Hamburg, Germany

###### Abstract

Effective music mixing requires technical and creative finesse, but clear
communication with the client is crucial. The mixing engineer must grasp the
client’s expectations, and preferences, and collaborate to achieve the desired
sound. The tacit agreement for the desired sound of the mix is often
established using guides like reference songs and demo mixes exchanged between
the artist and the engineer and sometimes verbalised using semantic terms.
This paper presents the findings of a two-phased exploratory study aimed at
understanding how professional mixing engineers interact with clients and use
their feedback to guide the mixing process. For phase one, semi-structured
interviews were conducted with five mixing engineers with the aim of gathering
insights about their communication strategies, creative processes, and
decision-making criteria. Based on the inferences from these interviews, an
online questionnaire was designed and administered to a larger group of 22
mixing engineers during the second phase. The results of this study shed light
on the importance of collaboration, empathy, and intention in the mixing
process, and can inform the development of smart multi-track mixing systems
that better support these practices. By highlighting the significance of these
findings, this paper contributes to the growing body of research on the
collaborative nature of music production and provides actionable
recommendations for the design and implementation of innovative mixing tools.

##  1 Introduction

###  1.1 Music Production and Mixing

Music production encompasses the entire life cycle of the creation of a song,
from idea conception to distribution. The process involves writing the score,
recording vocals and instruments, mixing the music, mastering the audio, and
finally distributing it [1] that influences the creation of different edits or
versions for different formats and media. Audio recordings are made in a
studio using digital audio workstations (DAW) and various other equipment. A
mixing engineer then combines these recordings using knowledge of signal
processing, sound, and music to produce a cohesive and well-balanced mix which
is further mastered and enhanced for distribution.

Music mixing is the process of combining various audio recordings to create a
polished, balanced, and artistically aligned finished product that elicits
feelings and tells a story [2]. The mixing engineer adjusts technical and
artistic elements, emphasising crucial components, balancing instrumentation,
and employing techniques like gain staging, equalisation, panning,
compression, and modulation effects to shape the sound [3].

###  1.2 Technological Advancements and the Recording Industry

Technological advancements have transformed the landscape of music production.
Multi-track recording, analogue synthesizers, DAWs, and software plugins have
given artists and producers greater control, resulting in more intricate and
detailed productions [4]. However, this has led to specialised roles such as
recording engineer, producer, mixing engineer and mastering engineer,
requiring extensive skills and knowledge [5]. The increased number of
stakeholders in the production process demands teamwork and effective
communication to achieve the desired goals [6].

In the past, professional music production required access to expensive
studios and equipment. However, affordable recording equipment and software
have democratised music production, allowing artists to create high-quality
music at home. This accessibility has led to a wider variety of styles and
genres, catering to different user groups [7, 8, 9]. Recent literature
identifies three user types: amateurs, who are typically newcomers to the
field without formal training [10], pro-ams, who are often skilled individuals
without full professional support [11, 12, 13], and professionals, who are
typically highly skilled paid engineers with industry expertise [9]. With
high-fidelity recordings and polished productions becoming the norm, artists
face pressure to create sonically impressive music [14]. Streaming platforms
and digital distribution have intensified competition, prompting musicians to
invest in complex production techniques for a unique and captivating sound.
Production quality plays a significant role in gaining traction, yet there
remains a gap between the accessibility of production tools and the expertise
required to use them effectively.

###  1.3 Intelligent Music Production and Automatic Mixing

Intelligent music production systems assist creators in navigating the complex
tasks of audio production. These systems provide insights, suggestions and
even full automation of control parameters in multitrack projects [15]. They
benefit both amateurs and professionals, offering assistance and streamlining
the process [16, 17].

Automatic mixing employs algorithms and intelligent systems to automate the
audio mixing process [18]. It adjusts key audio parameters such as volume,
panning, equalisation, compression, and spatial effects. Through the analysis
of audio signals and the application of predefined rules or machine learning
models, automatic mixing algorithms aim to create balanced and cohesive mixes.
Various approaches have been explored [17, 19], including knowledge-engineered
[20], machine learning-based [21], and deep learning-based methods [22, 23,
24, 25]. Most of these systems take raw tracks or stems as input and generate
a mix as output.

While these approaches provide solutions within limits, they often lack user
control to specify context and expectations. As music mixing allows for
multiple possible mixes from the same stems or tracks, context plays a crucial
role in determining the best fit for the client’s needs [26]. Professionals
and pro-ams have shown less interest in completely automatic services due to
the generic nature of these systems [27, 28]. They prefer technology that
allows for more control and contextual understanding to achieve desired
results [26].

###  1.4 Collaborative Practice-Informed Design for Smart Mixing Tools

Creating music involves multiple stakeholders who contribute their expertise
to shape the final product. Clear communication and understanding between
artists, recording engineers, mixing engineers, producers, and mastering
engineers are crucial to realising the artist’s vision. To design mixing
systems that facilitate effective collaboration, our work explores the
communication dynamics between artists and mixing engineers. These findings
can inform the development of future smart mixing tools.

During the mixing process, the mixing engineer employs various methods to
understand the artist’s expectations. The artist uses semantic terms, verbal
instructions, reference songs, and demo mixes to convey their needs. These
media allow for developing a tacit agreement between the client and the mixing
engineer about the context. The decisions made by the mixing engineer greatly
impact the clarity, balance, spatialisation, emotion, and dynamics of the
song. Collaborative work between the mixing engineer and artist is vital to
achieving the desired results while incorporating their respective identities
into the music. Our objective is to understand the communication process
enabling this vital synergy, and explore how this knowledge might influence
the design of smart mixing tools.

This study utilises semi-structured online interviews with five experienced
mix engineers, followed by a questionnaire-based study to validate the
findings. Through qualitative data analysis, we explore how clients express
their expectations and how mixing engineers interpret and derive insights from
these communication channels. The results of this study aim to inspire an
interaction-centred approach to designing mixing tools.

Figure 1: Themes and sub-themes extracted from the thematic analysis for the
interviews

##  2 Study Design and Methodology

The objective of this study is to investigate the exchange of ideas and
expectations between clients and mixing engineers during the song-mixing
process. We aimed to understand the informative cues and suggestions provided
by clients to convey their objectives for the mix, how mixing engineers
interpret and incorporate these cues into the mixing decisions, as well as how
mixing engineers assess the completion of a mix. To achieve these goals, we
implemented a multi-phase approach consisting of semi-structured interviews
and a structured web-based questionnaire.

###  2.1 Phase 1: Semi-Structured Interviews

Semi-structured interviews are qualitative research methods that involve
asking a pre-determined set of open questions, allowing for further
exploration of responses [29]. In this phase of the study, we conducted one-
to-one semi-structured interviews with professional mixing engineers to gain
insights into their experiences and perspectives.

####  2.1.1 Interview Structure

The interviews focused on exploring various aspects of communication,
expectation exchange, and collaboration between mixing engineers and artists
during the music-mixing process. Key themes explored during the interviews
included the interaction with clients, the role of demo mixes and reference
songs, the criteria for defining the completion of a mix, and the role of
creativity in the mixing process. The interviews were conducted through
recorded online video calls, and the recordings were later transcribed for
analysis purposes.

####  2.1.2 Participant Profile

To ensure the validity of the investigation and mitigate selectivity issues,
five professional mixing engineers were invited for interviews [30]. By
involving multiple experts in the field, we wanted to gain diverse
perspectives and minimize the risk of bias or individual preferences from
skewing the results. These engineers were chosen based on their expertise and
extensive experience in the field of music mixing. At the time of the
interviews, they all had more than five years of experience, possessed
advanced mixing skills, and actively worked on music projects for themselves
and other artists. Their qualifications included formal training, payment for
their work, and having their work published using traditional discographies
and music streaming platforms. Two of the interviewed engineers are female.
Three out of five engineers worked in Europe, one in Asia, and one in
America/Europe. Three of them specialise in pop, rock, and metal with the
other two mixing for jazz, ensemble, folk, classical and world music.

####  2.1.3 Data Analysis

The interview recordings were transcribed, and thematic analysis was conducted
using an inductive, iterative, and grounded approach [31] [32] in the MAXQDA
software 111https://www.maxqda.com/ . A series of codes were generated to
identify different themes across the interviews, which were refined over
several iterations. These themes were then used to inform the design of the
questionnaire. This involved identifying key themes, patterns, and categories
of information related to the exchange of ideas and expectations between the
client and the mixing engineer. The analysis process focused on extracting
meaningful insights and understanding the underlying factors influencing the
mixing process. Figure 1 provides a concise summary of the extracted themes
and sub-themes, highlighting their interconnectedness. The central theme
revolves around the critical role of communication and collaboration
throughout the mixing process. Furthermore, we have identified three distinct
stages where this communication and collaboration occur: prior to mixing,
during the mixing process, and upon completion of the mix. Figure 1
effectively illustrates how various sub-themes within different categories
intertwine, ultimately forming a comprehensive understanding of the subject.

###  2.2 Phase 2: Questionnaire-based Study

After the analysis of interviews, a structured web-based questionnaire was
designed to both validate and expand upon the insights gathered. The
qualitative analysis of the interview transcripts allowed us to structure the
inferences into questions that could be asked to a wider group of mixing
engineers for greater validity.

For example, one key inference from the interviews was that mixing engineers
often utilize reference songs from genres different from the ones they are
working on. To explore this further, we included specific questions in the
questionnaire to assess the frequency of this practice among a larger pool of
engineers. For example:

“Can you describe some cases where you would consider reference songs from
another Genre?”

“Consider a situation where you like how an instrument is processed in a song
from another genre [not the genre of the song you are working on] and you want
to adapt this to the instrument of your song.”

By including these questions, we aimed to gather quantifiable data and gain a
broader perspective on the communication and collaboration dynamics between
clients and mixing engineers in the song-mixing process. The questionnaire
took into account a diverse range of mixing engineers to ensure a more
comprehensive understanding of the topic, incorporating insights and themes
derived from the initial interviews.

Figure 2: Mixing Engineer’s profile for Phase 2 of the study

####  2.2.1 Questionnaire Structure

To ensure comparability across participants, we structured the questionnaire
in a standardised manner [33]. It consisted of three sections. The first
section collected general information about participants’ musical practices,
expertise and skills. The second section focused on standard practices, client
interaction, and the use of demo mixes and reference songs in the mixing
workflow. The third section explored the participants’ understanding of the
information provided by reference songs and demo mixes, as well as the role of
creativity in mixing practice. The questionnaire included 33 questions, which
were mostly multiple-choice or multi-select, with some optional long-answer
questions. It was hosted online using Microsoft Forms and took approximately
15-20 minutes to complete.

####  2.2.2 Participant Profile

The questionnaire was distributed among professional mixing engineers within
our personal network. A total of 22 participants with varied experiences took
part in the study. Figure 2 illustrates the experience levels of the
participants, with all of them having more than 3 years of production
experience. The majority of respondents identified their mixing skills as
advanced. They worked on a range of 1-10 to over 100 songs per year. The
participants consisted of professional or pro-am mixing engineers who worked
on their own music as well as projects for other artists. The participants
were based out of Asia, Europe, and America. About 95% of the participants
were male. About 50% of them worked in the pop, rock, electronic, and metal
genres, however, we also had engineers who specialised in classical, hip-hop,
jazz, folk, and world music in decreasing order.

Figure 3: Flow of communication and collaboration during the mixing process

####  2.2.3 Data Analysis

For the questionnaire data, a mixed methods approach was employed due to the
nature of the data collected. Qualitative data was processed using Thematic
Analysis [31, 32], while quantitative data was analysed using descriptive
statistical methods in Microsoft Excel. This approach provided a comprehensive
analysis of both qualitative and quantitative aspects of the data.

##  3 Results

The following section presents results from both study phases, categorised
into three stages: pre-mixing, during mixing, and mix completion. The findings
from both the interviews and the questionnaire are discussed together,
encompassing both quantitative data from the questionnaire and qualitative
insights from the interviews. An overview of this is shown in Figure 3. The
results of our research indicate a strong correlation between effective
communication and collaboration and the quality and alignment of the final
product.

###  3.1 Pre-Mixing

Effective communication with the client is an important part of the mixing
process. It is important for the mix engineer to understand the client’s
expectations and preferences, and to work with them to achieve the desired
sound. When asked what mixing means to them, engineers said,
“It’s all a thing about communication and trying to understand what the client
wants.”
“Their hopes, I would say.”
Engineers often set up an initial consultation session with the artist where
they try to get a sense of the artist’s vision for the mix. The session
involves discussions about the sound of individual elements of the mix as well
as the mix as a whole.

As seen in Figure 4, the artist uses different mediums to convey their
expectations. The artist may describe the desired sound using verbal
instructions, and semantic terms, provide reference songs, and offer a rough
mix (also called a demo mix) as a guide. Semantics are words, linguistic
units, or phrases applied to capture the meaning and interpretation of various
sonic elements of the mix. Previous work has explored the role of semantics in
music production and how it is used to describe the sound and expected
transformation for the sound [34, 35, 36].

Figure 4: Ways in which expectations about the mix are conveyed

“They give you some description of how they would like the mix to be. Yeah,
sometimes it’s just a verbal reference in terms of you know, hey have you
heard that song, maybe something on that space something in this space, or it
could just be like how we are having a chat … I need it very bright. So,
usually, we tend to think, then I try to understand from him what he means by
bright. Usually, it tends to be like he needs a lot of highs in the sound. You
know high frequencies.”

The use of the term “bright” to describe the sound is a common example of how
semantics can be applied to convey specific characteristics. In this context,
when the client describes the sound as “bright,” they are likely referring to
a sonic quality that emphasizes higher frequencies or has a sense of clarity
and brilliance.

Ultimately, the mix engineer should work closely with the artist to ensure
that the final mix meets their expectations and achieves the desired sound.

###  3.2 During Mixing

Mixing is a process of carving the sound of the final mix that emotes and
tells the intended story. The mix engineer begins by loading the raw tracks or
stems into the DAW and organising them in a way that is easy to navigate and
understand. This may involve labelling the tracks and grouping similar ones
together. Next, the engineer listens to the project and takes note of any
issues that need to be addressed. They may listen to the project multiple
times, as well as the rough mix and reference songs provided by the client to
get a sense of the desired direction for the mix. After this initial
preparation, the engineer uses the gathered direction to make decisions on the
technical aspects of mixing, such as gain staging, panning, and fixing any
frequency masking and noise or clicks. They then proceed to add more creative
elements such as reverb, delay, and distortion. All of these decisions are
made in alignment with the vision for the mix.

Figure 5: Situations when engineers import settings from the previous project

####  3.2.1 Inspiration from Previous Work

Mix engineers often have their own techniques and approaches to mixing. Figure
5 shows that some engineers prefer to start a project from scratch, while
others find it useful to borrow ideas or settings from past projects.

This is true in three cases:

  1. 1.

Artist-specific sonic preference: Some artists have a preference for a certain
sonic identity for elements in their mix. This could be unique to the artist’s
identity. In these cases, the artist might request the engineer to take
inspiration from previous songs/projects of the artist.
“For example, I was mixing a song for somebody and that artist really liked
the way her voice sounded in one particular mix. Next time a very different
song was done but, she wanted the vocals to sound just the way it was before.
So, I did turn back to what I had done and you know present it, and that
worked for that artist.”
This highlights the importance of understanding the client’s preferences and
using past projects as a reference to achieve their desired sound.

  2. 2.

Album-specific sonic preference: It is a practice to maintain the cohesiveness
of sound across an album. Hence, if an engineer is working on all the songs in
an album, they generally establish a sound for a song on the album and then
take inspiration from that for mixing the rest of the songs.
“It also depends, if I’m mixing just one song or a whole album and if I’m
mixing a whole album versus just like one song, I will typically ask the
artist for a song, that they’d like me to start with to establish the overall
sonic characteristics of the album. … their overall concept for the album
needs to be cohesive. So, I will, …mix one song, make sure it’s going in the
right direction, and then use that to lead me into every other song and start
to borrow from that first mix and bring things over other mixes and change
them accordingly.”
Importing settings from other tracks on the album can also be a time-saving
technique, especially if the tracks were recorded in the same acoustic space.
“So quite often when you’re mixing a whole album, the first thing you do is if
you got the same drum kit and it was all recorded in the same room. Yeah, just
start with the same drum sound on every song and then do the creative part.”

  3. 3.

Processing-specific sonic preference: Often mix engineers borrow sound
processing ideas from previous projects when they feel that the current mixing
project can benefit from a starting point as the sonic sound goal is similar
to something they have done previously.

It is important to remember, however, that every song is unique and may
require its own approach and treatment. In the coming subsections, we will
elaborate further on how elements like demo mixes and reference songs are used
to aid the process of mixing a song.

Directors |  Status |  Usage (in%) |  From client (in%)
---|---|---|---
Demo Mix |  Yes |  27 |  24
Sometimes |  50 |  38
No |  23 |  38
Reference Songs |  Yes |  45 |  14
Sometimes |  45 |  38
No |  9 |  48
Table 1: Likelihood of usage of various directors in the mixing process.

####  3.2.2 Demo Mix (Rough Mix)

Rough mix or demo mix are terms used to describe an unfinished coarse mix of a
song that is completed by adjusting gain and pan levels and basic processing
on some of the elements. Our studies (Table 1) showed that about 77% of the
participant mix engineers were likely to use demo mixes and more than 50% of
the time, they received a rough mix from the client. It is typically used to
give the mix engineer an idea of the artist’s vision for the mix and can be
helpful in providing a sense of direction for the mix as also presented in
Figure 6

Figure 6: Information derived from Demo mixes

“For me a demo, it’s not finished. It’s more like this is the start of how
we’re going to.”
“But a demo song as well could be really helpful for me to understand where
the thing should go.”
However, it’s important to note that the demo mix is often not perfect and may
require adjustments to get to the final mix.
“In 99.9% of the cases, it’s really bad. But most of the clients would give
you some sort of a demo or something that they make themselves so that you get
a direction of what they want to do.”
The rough mix is sometimes done by the recording engineer, producer or the
artist themselves. However, demo mixes are often rough and may not be fully
representative of the desired sound, so it is important for the mix engineer
to have a conversation with the client to understand their expectations and
preferences. The client might ask the engineer to stick with the balance,
panning, or processing on certain instruments or elements as shown in Figure
6.

“Usually I’m mixing from a rough mix as well. They do a rough mix and I
usually ask them the question very early. How important is this rough mix to
you? Is this part of your vision or do you want me to go in the opposite
direction and do something.”
“I will speak with the composer or the client you know what are you wanting me
to listen to from the demo mix? So, sometimes they will tell me that this is
the producer’s mix, so don’t change the balance of it. I want the balance to
be the same.”
The artist may ask the mix engineer to focus on certain elements of the rough
mix, such as balance or panning, or to preserve certain elements of the rough
mix.

####  3.2.3 Reference Songs

Apart from demo mixes, most mix engineers often also rely on reference songs
to guide them in the mixing process as shown in Table 1. A reference track is
a song from another artist to use as a benchmark for varying elements in your
own productions. Moreover, 90% of the engineers confirmed that they use more
than one reference song as seen from Figure 8.

#####  3.2.3.1 Directions from reference song

When asked what the use of reference songs is in the mixing workflow,
engineers described (as seen in Figure 9):

  1. 1.

Helped understand the client’s vision

  2. 2.

Get a sense of direction

  3. 3.

A pointer for the sound of the final mix: This may include understanding the
balance, panorama, and processing of certain instruments as shown in Figure 7.

Reference songs are commonly used by mix engineers as a direction for their
work.
“Most often a reference song acts as a direction sign on the road. It helps me
get a sense of expectation and direction.”
The reference song either brings out the same target emotion, has a similar
target semantic descriptor (for example the song sounds warm, bright, etc),
has the same target sonic and dynamic sound or has a similar compositional
structure. In addition to providing a sense of direction, reference songs can
also be used for A/B testing to see how far the engineer has progressed with
the mix, and to help composers and artists communicate their vision to the
engineer by presenting examples in the form of a reference song. However, it
is important to note that while reference songs are often chosen for their
ability to evoke a certain feeling or bring out a specific emotion, they are
typically not used to replicate the mix exactly as seen from a strong
disagreement in Figure 12. “I’m not specifically interested in my mix sounding
exactly the same. I’m interested in whether my mix feels the same.”

Figure 7: Information derived from the reference songs

#####  3.2.3.2 Finding a reference song

We also induced that about 50% of the time, clients send mixing engineers
reference songs. As seen in Figure 10, about 42% of the time, clients send
reference songs, however in case the client does not, engineers find them on
their own. Also, 78% of the engineers confirmed that once they find reference
songs, they pass it on to the client to confirm if the songs align with their
vision as seen from Figure 8. However, for certain purposes of usage like the
ones described below, engineers may not get feedback from the client.
“Depends on the purpose of the reference track. If the reference track is
aimed at the overall sound and artistic vision of the artist i.e. a genre-
specific reference track, then yes. If it’s related to being able to use a
reference track that I know has a well-balanced mix, then there’s no need to
discuss this with the artist as their judgment on that part of the process is
not always necessary.”
“It depends on why I’m using the reference. For overall vibe, balance, etc.
it’s always good to get a client’s input. More often than not I’m using a
reference for some smaller detail (e.g. exactly how two sounds interact in the
frequency spectrum), and in these cases, I don’t bother getting a client’s
input.”

Figure 8: Influence of the reference song in the mixing process Figure 9: The
use of reference songs in the mixing workflow Figure 10: How do you find a
reference song?

#####  3.2.3.3 Choosing a reference song

Choosing a reference song is an important part of the mixing process.
Reference songs are generally chosen based on the purpose they intend to
serve.

  1. 1.

Director for the entire mix: Reference songs can serve as a guide for the
engineer in order to achieve the desired sound and emotion for their mix.
“I’m definitely listening to the ambience and spatiality. I’m listening to
compression. How punchy is it? Or is it quite a sort of mellow in the overall
league? I’m not listening to specific EQs, but I’m listening to an overall EQ
curve.”
The referenced song should either have a similar sound to the final mix, which
could mean it has the same frequency profile, dynamics, and general
processing, or brings out the same emotion.
“It was you know very emotive and I had a reference in my mind that in this
particular song, it should sound that breezy or that lush. So, I went back to
that reference song in my head and looked it up and saw how is it sounding in
that song and then I got a few clues and then I try to incorporate that in my
song.”

  2. 2.

Director for a particular element in the mix: Reference songs can give
information about the processing or sound of one particular element or
instrument in the mix.
“If someone sent you a track it can mean so much because yeah, if you, if
you’re listening to something, then it’s very personal on the things you’re
going to check out. I mean, some people are only going to do it for the vocals
or for the whole song, or for structure. So, then I’m going to have to start a
talk about it I think.”

A reference song is often from the same genre and sounds similar to the final
expected mix, however, it is possible for it to be from another genre. This
holds true in two situations: i) when the reference gives information about
the processing of some specific element and not the mix as a whole, or ii)
when the artist simply wants to evoke a certain emotion and is not concerned
with genre.
“I always try to go to a song from the original genre or an album that reminds
me of that song. I got this kind of blues song which was really warm but they
didn’t give me any reference. I was listening to the rough mix and I was like
this reminds me of something. I went through my dance collection of CDs and I
picked up a couple of songs and sent them to the artists saying this rough mix
reminds me of these songs. Do you approve to use them as references?”

#####  3.2.3.4 Creativity

The process of mixing music is technical as well as creative. In our studies,
we asked engineers whether reference songs restrict their creativity, with the
majority (about 64% of the engineers) disagreeing, as indicated in Figure 8.
They emphasised that reference songs only provide a direction (a suggestion),
however, the goal can be achieved in multiple ways. Reference songs provide a
valuable means to grasp the client’s expectations. One engineer expressed that
the sounds and instruments in their mix might be different from those in the
reference, however, a reference might help them listen to the sound in the mix
that they might have otherwise ignored.

“I would put like about 20-30% of my own opinion into that. So, it doesn’t
really deviate too much from the actual reference, but then at the same time
there’s a bit of a character that is not the same as the thing that they bring
in the reference song.”
“For helping me inform specific decisions and resolve uncertainties and not a
creative limitation.”

#####  3.2.3.5 Feedback

The process of mixing requires a continuous dialogue between the mix engineer
and the client to ensure that the final product meets the client’s
expectations. One of the crucial elements of this dialogue is the feedback
that the client provides on the mix in progress. As the mix engineer works on
the track, they send different versions of the mix to the client for their
review, making necessary adjustments and tweaks based on the client’s
feedback. This iterative process allows the engineer to make sure that the mix
is on the right track, and that the client is happy with the direction it is
heading.
“I feel like it’s done and I can send it to the artist where if that’s the
first mix, I’m obviously happy with it if there is a revision.”

Figure 11: When do you know your mix is ready?

###  3.3 Mix Completion

Engineers agreed that the mix is ready whenever the mix sounds balanced, the
engineer is happy, and the client is satisfied as shown in Figure 11.

“Honestly, I just hit the play button and just sit back and I am looking for
anything that would surprise me and that’s a negative thing. I mean, if I just
sit back and listen to it and … it just sounds unnatural or something like
perhaps my attention …I know that it’s not finished.”
“I think it is done when the client says it’s done because in your head it’s
never done. That’s my experience. But, yes uh there comes a time a point in
the mix where you’re really happy.”
Engineers confirmed that often they compare the final mix against the given
reference songs. Our studies also show that most often engineers try to create
a mix that either sounds similar to the reference songs or evokes similar
feelings as shown in Figure 12.

“I feel it’s like a middle point between the two like obviously when I feel
like I’m happy with what I’ve done and I’ve got a reference I tried to compare
it to the reference and if I feel what I have done is totally different, I’m
like okay and I try to make it a bit closer”
“I’m not specifically interested in my mix sounding exactly the same. I’m
interested in whether my mix feels the same.”
“I mean when it sounds good to me, generally balanced, and it has enough
similarity to that reference.”
It is important to note that there isn’t a one-size-fits-all answer to when a
mix is ready. The decision ultimately depends on the individual engineer and
the specific project they are working on. Some may rely heavily on reference
songs to guide their mixing process, while others may be more focused on
creating a mix that evokes the desired emotion or feeling. The mix is
generally considered ready when it sounds smooth and balanced and evokes the
same feeling as the reference song. Ultimately, a mix is ready when it meets
the expectations of the engineer and the client and sounds good on a variety
of playback devices.

Figure 12: Comparing reference songs against mix

##  4 Discussion and Conclusion

Our investigation highlights the crucial role of communication and
collaboration in the mixing workflow. As mixing is a context-dependent task,
it is essential for the mixing engineer to understand the artist’s vision for
the song and make decisions aligned with it to create the desired mix. The
artist provides various mediums of information, including verbal instructions,
semantic terms, reference songs and demo mixes, to convey their expectations.
These directives provide a sense of direction to the mixing engineer,
facilitating their decision-making process.

Our findings also shed light on the information that mixing engineers gather
from each of these different directives, underscoring the dynamic nature of
collaboration in the mixing workflow. Previous research has emphasised the
importance of collaborative and co-creative smart mixing systems that promote
interaction, collaboration, and trust among professionals [27, 28, 26]). This
study further underscores the significance of incorporating context into the
mixing workflow and presents a preliminary approach for designing such
systems.

Context plays a crucial role in shaping mixing decisions and perceptions,
encompassing factors such as style, genre, knowledge, time and technology.
Moreover, context evolves dynamically throughout the song and the mixing
process [26]). Our research demonstrates that context in music mixing can be
incorporated through various means, including reference songs, demo mixes,
verbal instructions and semantic terms. However, each of these directives
presents its own set of challenges.

Previous studies on the utilisation of semantics to define mixing expectations
have highlighted a significant challenge in generalisation, as there is no
universally acknowledged and agreed-upon interpretation for specific semantic
terms [36]. Additionally, the semantic terms may hold varied meanings across
genres, production settings, and geographical regions [34]. Verbal
instructions provided in written form also require further examination. The
complex nature of the mixing process may not be fully captured by a few words,
making it difficult to represent the vision of the mix accurately.

In our work, we conducted a detailed investigation into the use of reference
songs and demo mixes as mediums of context. The results demonstrate that
reference songs have the ability to effectively convey the abstract nature of
mixing style, capturing the desired emotions and sonic qualities sought by the
artist. Similar findings have been observed in the music recording process
regarding the use of reference songs. Ongoing research exploring the use of
reference sound and reference songs for audio effects [37] and mixing style
transfer [38, 25] has shown promising success in capturing context and the
user’s intentions.

The future success of AI in music production depends on the ability of these
systems to support interaction. This interaction can be divided into two
phases: before the mixing process and after the mixing process. An AI-based
mixing system that enables smart mixing needs to be provided with context on
the input side. This context can be provided in the form of reference songs,
demo mixes, verbal instructions or semantic descriptors. Once the context is
provided, the system should be able to interpolate and generate a mix that
represents an understanding of the context. Furthermore, on the output side,
the system should allow for fine-tuning the mix to achieve the desired sound.
This is akin to iterating based on feedback provided in the collaborative
dynamics of mixing. The system should either provide means for alteration and
fine-tuning of the output or offer a way to receive feedback to further
improve the result in the desired direction.

In future work, our focus will be on investigating both subjective and
objective evaluation methods to assess the system’s effectiveness in capturing
and incorporating contextual information provided into the output.
Additionally, we aim to delve deeper into certain concepts that emerged in
this study but were not extensively explored. One such concept is the notion
of mixing style and its variations across different genres and artistic
styles.

Overall, our study underscores the importance of effective communication and
collaboration in the mixing process. It provides insights into the different
forms of communication employed by artists and the valuable information gained
by mixing engineers from these channels. By understanding and incorporating
these dynamics, future AI-based mixing systems can enhance the interaction and
collaboration between artists and engineers, expectantly leading to improved
outcomes in the music production process.

##  5 ACKNOWLEDGMENT

We express our sincere gratitude to the JAES reviewers for providing valuable
feedback on our work. We are truly appreciative of the contributions made by
all the mixing engineers and participants who participated in the interviews
and studies and shared their valuable knowledge with us. We extend our thanks
to Steinberg’s research and development team for their unwavering support and
honest feedback throughout this project.

This work is funded and supported by UK Research and Innovation [grant number
EP/S022694/1] and Steinberg Media Technologies GmbH under the AI and Music
postgraduate research program at the Centre for Digital Music, QMUL.

## References

  * [1] R. J. Burgess, _The art of music production: The theory and practice_ (Oxford University Press, 2013).
  * [2] M. Miller, _Mixing Music_ (Penguin, 2016).
  * [3] R. Izhaki, _Mixing audio: concepts, practices, and tools_ (Routledge, 2017).
  * [4] B. J. Hracs, “A creative industry in transition: The rise of digitally driven independent music production,” _Growth and Change_ , vol. 43, no. 3, pp. 442–461 (2012), https://doi.org/10.1111/j.1468-2257.2012.00593.x.
  * [5] R. J. Burgess, _The history of music production_ (Oxford University Press, 2014).
  * [6] R. Wilsmore and C. Johnson, _Coproduction: Collaboration in Music Production_ (Taylor & Francis, 2022).
  * [7] M. Sandler, D. De Roure, S. Benford, and K. Page, “Semantic web technology for new experiences throughout the music production-consumption chain,” presented at the _2019 International Workshop on Multilayer Music Representation and Processing (MMRP)_ , pp. 49–55 (2019).
  * [8] S. McGrath, A. Chamberlain, and S. Benford, “Making music together: an exploration of amateur and pro-am Grime music production,” presented at the _Proceedings of the Audio Mostly 2016_ , pp. 186–193 (2016).
  * [9] G. Bromham, “How can academic practice inform mix-craft?” in _Mixing Music_ , pp. 265–276 (Routledge, 2016).
  * [10] M. Hoare, S. Benford, R. Jones, and N. Milic-Frayling, “Coming in from the margins: amateur musicians in the online age,” presented at the _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_ , pp. 1295–1304 (2014).
  * [11] N. Prior, “New amateurs revisited: Popular music, digital technology, and the fate of cultural production,” in _Routledge handbook of cultural sociology_ , pp. 340–348 (Routledge, 2018).
  * [12] R. A. Stebbins, “The amateur: Two sociological definitions,” _Pacific Sociological Review_ , vol. 20, no. 4, pp. 582–606 (1977), https://doi.org/10.2307/1388717.
  * [13] C. Leadbeater and P. Miller, _The pro-am revolution: How enthusiasts are changing our society and economy_ (Demos London, 2004).
  * [14] M. Katz, _Capturing sound: How technology has changed music_ (Univ of California Press, 2010).
  * [15] R. Stables, J. D. Reiss, and B. De Man, _Intelligent Music Production_ (Focal Press, 2019).
  * [16] D. Moffat and M. B. Sandler, “Approaches in Intelligent Music Production,” _Arts_ , vol. 8, no. 5, p. 14 (2019 September), https://doi.org/10.3390/arts8040125.
  * [17] B. De Man, J. Reiss, and R. Stables, “Ten Years of Automatic Mixing,” presented at the _3rd Workshop on Intelligent Music Production_ (2017 09).
  * [18] E. Perez-Gonzalez and J. D. Reiss, “Automatic mixing,” _DAFX: Digital Audio Effects_ , pp. 523–549 (2011).
  * [19] C. J. Steinmetz, S. S. Vanka, M. A. Martínez Ramírez, and G. Bromham, _Deep Learning for Automatic Mixing_ (ISMIR, 2022 December), https://dl4am.github.io.
  * [20] B. De Man and J. D. Reiss, “A knowledge-engineered autonomous mixing system,” presented at the _Audio Engineering Society Convention 135_ (2013).
  * [21] D. Moffat and M. Sandler, “Machine learning multitrack gain mixing of drums,” presented at the _Audio Engineering Society Convention 147_ (2019).
  * [22] C. J. Steinmetz, _Learning to mix with neural audio effects in the waveform domain_ , Master’s thesis, Universitat Pompeu Fabra (2020 September).
  * [23] M. A. Martínez-Ramírez, W.-H. Liao, G. Fabbro, S. Uhlich, C. Nagashima, and Y. Mitsufuji, “Automatic music mixing with deep learning and out-of-domain data,” presented at the _ISMIR_ (2022).
  * [24] M. A. Martínez-Ramírez, D. Stoller, and D. Moffat, “A deep learning approach to intelligent drum mixing with the Wave-U-Net,” _Journal of the Audio Engineering Society_ (2021), https://doi.org/10.17743/jaes.2020.0031.
  * [25] J. Koo, M. A. Martínez-Ramírez, W.-H. Liao, S. Uhlich, K. Lee, and Y. Mitsufuji, “Music Mixing Style Transfer: A Contrastive Learning Approach to Disentangle Audio Effects,” _arXiv preprint arXiv:2211.02247_ (2022).
  * [26] M. N. Lefford, G. Bromham, G. Fazekas, and D. Moffat, “Context aware intelligent mixing systems,” presented at the _Audio Engineering Society_ (2021).
  * [27] S. S. Vanka, M. Safi, J.-B. Rolland, and G. Fazekas, “Adoption of AI Technology in the Music Mixing Workflow: An Investigation,” in _Audio Engineering Society_ (Helsinki, Finland) (2023 May).
  * [28] A. Tsiros and A. Palladini, “Towards a Human-Centric Design Framework for AI Assisted Music Production,” presented at the _Proceedings of the International Conference on New Interfaces for Musical Expression_ , pp. 399–404 (2020).
  * [29] H. Kallio, A.-M. Pietilä, M. Johnson, and M. Kangasniemi, “Systematic methodological review: developing a framework for a qualitative semi-structured interview guide,” _Journal of advanced nursing_ , vol. 72, no. 12, pp. 2954–2965 (2016), https://doi.org/10.1111/jan.13031.
  * [30] A. Bogner, B. Littig, and W. Menz, _Interviews mit Experten_ (Springer Fachmedien Wies- baden, 2006).
  * [31] V. Braun and V. Clarke, _Thematic analysis._ (American Psychological Association, 2012).
  * [32] V. Braun and V. Clarke, “Using thematic analysis in psychology,” _Qualitative research in psychology_ , vol. 3, no. 2, pp. 77–101 (2006), https://doi.org/10.1191/1478088706qp063oa.
  * [33] S. Brinkmann, “Unstructured and semi-structured interviewing,” _The Oxford handbook of qualitative research_ , vol. 2, pp. 277–299 (2014).
  * [34] R. Stables, S. Enderby, B. De Man, G. Fazekas, and J. D. Reiss, “SAFE: A system for extraction and retrieval of semantic audio descriptors,” presented at the _ISMIR 2014_ (2014).
  * [35] R. Stables, B. De Man, S. Enderby, J. D. Reiss, G. Fazekas, and T. Wilmering, “Semantic description of timbral transformations in music production,” presented at the _Proceedings of the 24th ACM international conference on Multimedia_ , pp. 337–341 (2016).
  * [36] D. Moffat, B. De Man, and J. D. Reiss, “Semantic Music Production: A Meta-Study,” _Journal of the Audio Engineering Society_ , vol. 70, no. 7/8, pp. 548–564 (2022), https://doi.org/10.17743/jaes.2022.0023.
  * [37] C. J. Steinmetz, N. J. Bryan, and J. D. Reiss, “Style transfer of audio effects with differentiable signal processing,” _arXiv preprint arXiv:2207.08759_ (2022), https://doi.org/10.17743/jaes.2022.0025.
  * [38] S. S. Vanka, J.-B. Rolland, and G. Fazekas, “Intelligent Music Production: Music Production Style Transfer and Analysis of Mix Similarity,” presented at the _DMRN-16_ (2021).

Soumya Sai Vankafigures/Figure_13.jpgSoumya Sai Vanka is a PhD student at
Queen Mary University of London’s Centre for Digital Music in the AI and Music
program, collaborating with Steinberg Media Technologies GmbH. She holds a
Gold medal in Physics (Bachelors[Hons]) from Sri Sathya Institute of Higher
Learning, India, and a master’s degree in Physics from Pondicherry University.
Soumya is also an accomplished saxophonist who trained in music production and
performed with various big bands. Her research focuses on expert-informed
style transfer in music mixing, with interests in self-supervised machine
learning, DDSP, intelligent music production, and human-centred design
approaches.

Maryam Safifigures/Figure_14.jpgMaryam Safi is an Audio Test Engineer in the
R&D team at Steinberg. Her work involves evaluating and supporting the
attainment of superior quality standards in various aspects of DAW software
development, such as existing audio effects plug-ins and prototypes of future
products. With an academic foundation in statistics and media technologies,
she is expanding her professional focus to audio quality in AI-based systems.
She supports research in a wide range of topics, including Music AI and the
integration of AI into audio plug-ins and mixing tools. Beyond her work,
Maryam serves as the Vice-Chair of the AES Germany Section. In 2020 she was
honoured with the Board of Governors award in recognition of exceptional
leadership of the AES Student Delegate Assembly. Jean-Baptiste
Rollandfigures/Figure_15.jpgJean-Baptiste Rolland is a Senior Software
Developer, contributing his expertise to Steinberg’s Research Team. In this
role, he focuses on practical research in audio analysis, bridging the gap
between theory and application within Steinberg’s product line. His work
encompasses a broad array of fields, including Digital Signal Processing,
Artificial Intelligence, and Software Engineering. He specializes in Music AI,
Audio Analysis, Audio Restoration, and Music Information Retrieval, with his
contributions resulting in features used, for example, in Cubase, Nuendo, and
Wavelab. Jean-Baptiste is also actively engaged in academic collaborations and
serves as an industry supervisor to students pursuing their academic work
(PhD, master’s degrees), helping to shape the future talent in this field.
György Fazekasfigures/Figure_16.jpgGyörgy Fazekas is a Senior Lecturer at the
Center for Digital Music, Queen Mary University of London. He holds a BSc,
MSc, and PhD degrees in Electronic Engineering. He is an investigator of
UKRI’s £6.5M Centre for Doctoral Training in Artificial Intelligence and Music
(AIM CDT) and he was QMUL’s Principal Investigator on the H2020funded Audio
Commons project. He was general chair of ACM’s Audio Mostly 2017 and papers
co-chair of the AES53rd International Conference on Semantic Audio and he
received the Citation Award of the AES. He published over 180 papers in the
fields of Music Information Retrieval, Semantic Web, Deep Learning, and
Semantic Audio.