# Searching For Music Mixing Graphs: A Pruning Approach

###### Abstract

Music mixing is _compositional_ — experts combine multiple audio processors to
achieve a cohesive mix from dry source tracks. We propose a method to reverse
engineer this process from the input and output audio. First, we create a
mixing console that applies all available processors to every chain. Then,
after the initial console parameter optimization, we alternate between
removing redundant processors and fine-tuning. We achieve this through
differentiable implementation of both processors and pruning. Consequently, we
find a sparse mixing graph that achieves nearly identical matching quality of
the full mixing console. We apply this procedure to dry-mix pairs from various
datasets and collect graphs that also can be used to train neural networks for
music mixing applications.

##  1 Introduction

#### Motivation

From a signal processing perspective, modern music is more than the mere sum
of source tracks. Mixing engineers combine and control multiple processors to
balance the sources in terms of loudness, frequency content, spatialization,
and much more. Many attempts have been made to uncover parts of this intricate
process. Some have gathered expert knowledge [1, 2] and built rule-based
systems [3, 4]. More recent work has adopted data-driven approaches. Neural
networks have been trained to map source tracks directly to a mix [5, 6] or to
estimate parameters of a fixed processing chain [7]. Yet, efforts to address
the compositional aspects of the music mixing, such as which processors to use
for each track, are still limited. One possible remedy is to consider a graph
representation whose nodes and edges are processors and connections between
them, respectively. In other words, each graph contains the essential
information about the mixing process. However, other than the dry source and
mixed audio, no public dataset provides such mixing graphs or related metadata
[8, 9, 10], which hinders this line of research. This is not surprising;
besides the cost of crowdsourcing, it is difficult to standardize the mixing
data from multiple engineers with different equipment. One recent work [11]
sidestepped this issue by creating synthetic graphs and using them for
training. However, this approach is not free from downsides. Neural networks
would suffer from poor generalization unless the synthetic data distribution
matches the real world. Similar data-related issues arise in different
domains, e.g., audio effect chain recognition [12, 13] and synthesizer sound
matching [14, 15, 16]. Furthermore, real-world multitrack mixes have a much
larger number of source tracks and graph sizes, making synthetic data
generation more challenging. Therefore, it is desirable to have a systematic,
reliable, and scalable method for collecting graphs. All these contexts lead
us to ask: _Can we find the mixing graphs solely from audio?_

$\mathcal{G},\mathcal{P}$$G_{\mathrm{c}}$$\mathbf{0}$$G_{\mathrm{p}},\mathbf{P}_{\mathrm{p}}$$\mathbf{P}_{\mathrm{c}}$$\hat{y}_{0}$$\hat{y}_{\mathrm{c}}$$\hat{y}_{\mathrm{p}}$${y}$$\tau$$\\!\\!\\!\mathrm{Parameter\>optimization}$$\\!\\!\\!\mathrm{Pruning}$$\mathcal{Y}$
$\\!\\!\\!\mathrm{Console\>parameters}$
$\\!\\!\\!\mathrm{Pruning\>search\>with\>parameters}$
$\\!\\!\\!\mathrm{Audio\>search\>space}$
$\\!\\!\\!\mathrm{Tolerance\>threshold\>range}$ Figure 1:  Music mixing graph
search via iterative pruning.

#### Problem definition

Precisely, for each song (piece) whose dry sources $s_{1},\cdots,s_{K}$ and
mix $y$ are available, we aim to find an audio processing graph $G$ and its
processor parameters $\mathbf{P}$ so that processing the dry sources
$s_{1},\cdots,s_{K}$ results in a mix $\hat{y}$ that closely matches the
original mix $y$. With a loss $L_{\mathrm{a}}$ that measures the match quality
on the mixture audio domain $\mathcal{Y}$ and regularization $L_{\mathrm{r}}$,
our objective can be written as follows,

$G^{*},\mathbf{P}^{*}=\operatorname*{arg\,min}_{G,\mathbf{P}}\big{[}L_{\mathrm{a}}(\hat{y},y)+L_{\mathrm{r}}(G,\mathbf{P})\big{]}.$
(1)

#### Contributions

(a) Full mixing console (before pruning)  (b) Pruned graph

Figure 2:  Finding the sparse graph $G_{\mathrm{p}}$ from the differentiable
mixing console $G_{\mathrm{c}}$. Initial letters in the nodes denote their
respective types. i: input, o: output, m: mix, e: equalizer, c: compressor, n:
noisegate, s: stereo imager, g: gain/panning, r: reverb, d: multitap delay.

One might want to explore the candidate graphs without any restriction.
However, this makes the problem ill-posed and underdetermined. The graph’s
combinatorial nature makes the search space $\mathcal{G}$ extremely large.
Furthermore, we have to find the processor parameters jointly. As a result,
numerous pairs of graphs and parameters can have similar match quality.
Therefore, it is desirable to add some restrictions, e.g., preferring
structures that are widely used by practitioners. To this end, we resort to
the following pruning-based search; see Figure 1 for a visual illustration.
Inspired by a recent work [17], we first create a so-called “mixing console”
$G_{\mathrm{c}}$ (see Figure 2a for an example). It applies a fixed processing
chain to each source. Then, it subgroups the outputs, applies the chain again,
and sums the processed subgroups to obtain a final mix $\hat{y}$. This
resembles the traditional hybrid mixing console [18]. Each chain comprises $7$
processors, including an equalizer, compressor, and multitap delay. We
implement all of them in a differentiable manner [19, 20, 21]. This allows
end-to-end optimization of all parameters $\mathbf{P}_{\mathrm{c}}$ with an
audio-domain loss $L_{\mathrm{a}}$ via gradient descent. After this initial
console training, we proceed to the pruning stage. Here, we search for a
maximally pruned graph $G_{\mathrm{p}}$ and its parameters
$\mathbf{P}_{\mathrm{p}}$ while maintaining the match quality of the mixing
console up to a certain tolerance $\tau$; this is shown as a circle centered
at $y$ in Figure 1. Also, see Figure 2b for an example pruned graph. We use
iterative pruning, alternating between the pruning and fine-tuning, i.e.,
optimization of the remaining parameters [22]. To collect graphs from multiple
songs, it is crucial to make the entire search efficient and fast. Pruning, in
particular, takes a significant amount of computation time; hence, we
investigate efficient and effective methods for pruning. During the pruning,
we need to find a subset of nodes that can be removed while not harming the
match quality. To achieve this, we view each processor’s “dry/wet” parameter
as an approximate importance score and use it to select the candidate nodes.
This approach gives $3$ variants of the pruning method with different trade-
offs between the computational cost and resulting sparsity. It also draws
connections to neural network pruning [23, 24] where the binary pruning
operation is relaxed to continuous weights. Note that casting the graph search
into pruning is a double-edged sword. The pruning only removes the processors
and does not consider all possible signal routings, reducing the search space
(from grey to colored regions in Figure 1). Consequently, it does not improve
the match quality over the mixing consoles. Nevertheless, the pruned graph
follows the real-world practice of selectively applying appropriate
processors. In other words, the sparsity is crucial for the graph’s
interpretability. Also, it keeps the search cost in a practical range, which
might be challenging with other alternatives [25, 26]. Our method serves as a
standalone reverse engineering algorithm [17], but it can also be used to
collect pseudo-label data to train neural networks for music mixing
applications. For example, we may extend existing methods for automatic mixing
[3, 4, 5, 6, 7, 27] and mixing style transfer [28] to output the graphs. This
allows end users to interpret and control the estimated outputs.

#### Data

We first report a list of datasets to which we can apply our method. For each
song, we need a pair of dry sources $s_{1},\cdots,s_{K}$ and a final mixture
$y$. Additionally, we use subgrouping information that describes how dry
tracks are grouped together. Therefore, we use the MedleyDB dataset [8, 9] as
it provides all of them. We also add the MixingSecrets library [10]. Since it
only provides the audio, we manually subgrouped each track based on its
instrument. Finally, we include our private dataset of Western music mixes
from multiple engineers (denoted as Internal). The resulting ensemble
comprises $1129$ songs ($188$, $472$, and $579$ songs for each respective
dataset). The number of dry tracks ranges from $1$ to $133$, and the number of
subgroups ranges from $1$ to $26$ (see Figure 6 for the statistics). Except
for the final pruned graph collection stage (Section 3.4), we use a random
subset for the evaluations (a total of $72$ songs, $24$ songs for each
dataset). Every signal is stereo and resampled to
$30$\mathrm{k}\mathrm{H}\mathrm{z}$$ sampling rate.

#### Supplementary materials

Refer to the following link for audio samples, pruned graphs, and appendices
that contain additional details: https://sh-lee97.github.io/grafx-prune.

##  2 Differentiable Processing on Graphs

An audio processing graph $G=(V,E)$ is assumed to be directed and acyclic ($V$
and $E$ denote the set of nodes and edges, respectively). Each node $v_{i}\in
V$ is either a processor or an auxiliary module and has its type $t_{i}$,
e.g., e for an equalizer. Each processor takes an audio $u_{i}$ and a
parameter vector $p_{i}$ as input and outputs a processed signal
$f_{i}(u_{i},p_{i})$. Then, we further mix the input and this processed result
with a “dry/wet” weight $w_{i}\in[0,1]$. Hence, the output $y_{i}$ of the
processor $v_{i}$ is given as follows,

$y_{i}=w_{i}f_{i}(u_{i},p_{i})+(1-w_{i})u_{i}.$ (2)

We have the following $3$ auxiliary modules:

  * •

Input — It outputs one of the dry sources $s_{k}$.

  * •

Mix — We ouput the sum of incomming signals.

  * •

Output — A sum of its inputs is considered as a final output $y$.

Each edge $e_{ij}\in E$ represents a “cable” that sends an output signal to
another node as input. Throughout the text, we denote an ordered collection
from multiple nodes with a boldface letter, e.g., $\mathbf{w}$ for a weight
vector, $\mathbf{S}$ for a source tensor, and $\mathbf{P}$ for a dictionary
with processor types as keys and their parameter tensors as values. Under this
notation, our task is to find $G$, $\mathbf{P}$, and $\mathbf{w}$ from
$\mathbf{S}$ and $y$.

###  2.1 Differentiable Implementation

Considering the music mixing, we use the following $7$ processors.

  * •

#### Gain/panning

We control both loudness and stereo panning of input audio by multiplying a
learnable scalar to each channel.

  * •

#### Stereo imager

We change the stereo width of the input by modifying the loudness of the side
channel (left minus right).

  * •

#### Equalizer

We use a finite impulse response (FIR) with a length of $2047$ to modify the
input’s magnitude response. The FIR is parameterized with its log magnitude
(thus $1024$ parameters). We apply inverse FFT of the magnitude with zero
phase, obtain a zero-centered FIR, and multiply it with a Hann window. We
apply the same FIR to both the left and right channels.

  * •

#### Reverb

We employ $2$ seconds of filtered noise as an impulse response for
reverberation. First, we create a $2$-channel uniform noise, where these
channels represent the mid and side. We filter the noise by multiplying an
element-wise $2$-channel magnitude mask to its short-time Fourier transform
(STFT), where the FFT sizes and hop lengths are $384$ and $192$, respectively.
This mask is constructed using the reverberation’s initial and decaying log
magnitudes. After the masking, we obtain the mid/side filtered noise via
inverse STFT, convert it to stereo, and perform channel-wise convolutions with
input to get an output.

  * •

#### Compressor

We use a slight variant of the recently proposed differentiable dynamic range
compressor [21]. First, we obtain the input’s smooth energy envelope. The
smoothing is typically done with a ballistics filter, but we instead use a
one-pole filter for speedup in GPU. Then, we compute the desired gain
reduction from the envelope and apply it to the input audio.

  * •

#### Noisegate

Except for the gain computation, its implementation is the same as the
compressor.

  * •

#### Multitap delay

For each (left and right) channel, we employ independent $2$ seconds of delay
effects with a single delay for every $100$\mathrm{m}\mathrm{s}$$ interval. To
optimize delay lengths using gradient descent, we employ surrogate complex
damped sinusoids [29]. Each sinusoid is converted to a delayed soft impulse
via inverse FFT. Its angular frequency represents a continuous relaxation of
the discrete delay length. Each delay is filtered with a length-$39$ FIR
equalizer to mimic the filtered echo effect [30].

#### Batched node processing

It is common to compute the graph output signal by processing each node one by
one [15, 19]. However, this severely bottlenecks the computation speed for
large mixing graphs. Therefore, we instead batch-process multiple nodes in
parallel. For the graph in Figure 2b, we can batch-process $1$ equalizer e,
$3$ noisegates n, and $5$ gain/pannings g sequentially. Then, we aggregate the
intermediate outputs to $2$ subgroup mixes m (also in parallel). This part is
identical to graph neural networks’ “message passing,” so we adopt their
implementations [31]. We repeat these parallel computations until we reach the
output node o. By doing so, we obtain the output faster; in this example, the
number of sequential processing is reduced from $15$ (one-by-one) to $8$
(optimal). We empirically found that up to $5.8\times$ speedup can be achieved
for the pruned graphs with a RTX3090 GPU.

###  2.2 Mixing Console

We construct a mixing console $G_{\mathrm{c}}$ as follows (see Figure 2a).

  1. (i)

We add an input node i for each source track.

  2. (ii)

We connect a serial chain (with a fixed order) of an equalizer e, compressor
c, noisegate n, stereo imager s, gain/panning g, multitap delay d, and reverb
r for each input.

  3. (iii)

We subgroup and sum the processed tracks with mix nodes m based on the
prepared subgrouping information.

  4. (iv)

We apply the same chain ecnsgdr to each mix output, then pass it to the output
node o (we omit the mix module here).

###  2.3 Optimization

Before exploring the pruning of each mixing console, as a sanity check, we
first evaluate its match performance. To investigate how much each processor
type contributes to the match quality, we start with a base graph, a mixing
console with no processors that simply sums all the inputs. Then, we add each
processor type one by one to the processor chain (see the first column of
Table 1). We optimize and evaluate all these preliminary graphs for each song.
For each graph, we train its parameters and weights simultaneously with an
audio-domain loss given as follows,

$L_{\mathrm{a}}=\alpha_{\mathrm{lr}}L_{\mathrm{lr}}+\alpha_{\mathrm{m}}L_{\mathrm{m}}+\alpha_{\mathrm{s}}L_{\mathrm{s}}$
(3)

where each term $L_{\mathrm{x}}$ is a variant of multi-resolution STFT loss
[32] ($\mathrm{x\in\\{lr,m,s\\}}$, $\mathrm{lr}$: left/right, $\mathrm{m}$:
mid, $\mathrm{s}$: side)

$L_{\mathrm{x}}=\sum_{i=1}^{I}\Bigg{[}\frac{\|\log
Y^{(i)}_{\mathrm{x}}-\log\hat{Y}^{(i)}_{\mathrm{x}}\|_{1}}{N}+\frac{\|Y^{(i)}_{\mathrm{x}}-\hat{Y}^{(i)}_{\mathrm{x}}\|_{F}}{\|Y^{(i)}_{\mathrm{x}}\|_{F}}\Bigg{]}.$
(4)

Here, $Y_{\mathrm{x}}^{(i)}$ and $\hat{Y}_{\mathrm{x}}^{(i)}$ denote the
$i^{\mathrm{th}}$ Mel spectrograms of the target and predicted mixture,
respectively. $N$, $\|\cdot\|_{1}$, and $\|\cdot\|_{F}$ denote the number of
frames, $l_{1}$ norm and Frobenius norm, respectively. We use FFT sizes of
$512$, $1024$, and $4096$, and hop sizes are $1/4$ of their respective FFT
sizes. The number of Mel filterbanks is set to $96$ for all scales. We apply
A-weighting before each STFT [33]. The per-channel loss weights are set to
$\alpha_{\mathrm{lr}}=0.5$, $\alpha_{\mathrm{m}}=0.25$, and
$\alpha_{\mathrm{s}}=0.25$. The implementation is based on auraloss [34]. We
further add a regularization that promotes gain-staging, a common practice of
audio engineers that keeps the total energy of input and output roughly the
same. This is achieved with the following loss:

$L_{\mathrm{g}}=\sum\nolimits_{v_{i}\in
V_{\mathrm{g}}}\left|\log\|f_{i}(u_{i})_{\mathrm{m}}\|_{2}-\log\|u_{i,\mathrm{m}}\|_{2}\right|$
(5)

where $(\cdot)_{\mathrm{m}}$ and $\|\cdot\|_{2}$ denote mid channel and
$l_{2}$ norm, respectively. We apply this regularization to a subset of
processors $V_{\mathrm{g}}\subset V$ that comprises all equalizers, reverbs,
and multitap delays. This allows us to (i) eliminate redundant gains that
these linear-time invariant (LTI) processors could create and (ii) restrict
the parameters to be in a reasonable range. Therefore, the total loss is given
as

$L(\mathbf{P},\mathbf{w})=L_{\mathrm{a}}(\mathbf{P},\mathbf{w})+\alpha_{\mathrm{g}}L_{\mathrm{g}}(\mathbf{P})$
(6)

where the gain-staging weight is set to $\alpha_{\mathrm{g}}=10^{-3}$. Here,
we used a slightly different notation from Equation 1 to emphasize what is
optimized. Each console is optimized for $12$\mathrm{k}$$ steps using AdamW
[35] with a $0.01$ learning rate. For each step, we random-sample a
$3.8$\mathrm{s}$$ region of dry sources $\mathbf{S}$ (thus the batch size is
$1$), compute the mix $\hat{y}$, and compare its last $2.8$\mathrm{s}$$ with
the corresponding ground-truth $y$. Note that the first second is used only
for the “warm-up" of the processors with long states such as compressors and
reverbs.

Table 1:  Matching performances of the mixing consoles using different processor type configurations.  |  | $L_{\mathrm{a}}$ | $L_{\mathrm{lr}}$ | $L_{\mathrm{m}}$ | $L_{\mathrm{s}}$
---|---|---|---|---|---
Base graph (sum of dry sources) | $19.7$ | $1.52$ | $1.46$ | $74.3$
\+ Gain/panning | ecnsgdr | $.689$ | $.686$ | $.634$ | $.752$
\+ Stereo imager | ecnsgdr | $.676$ | $.671$ | $.623$ | $.734$
\+ Equalizer | ecnsgdr | $.557$ | $.549$ | $.493$ | $.637$
\+ Reverb | ecnsgdr | $.481$ | $.471$ | $.457$ | $.523$
\+ Compressor | ecnsgdr | $.423$ | $.407$ | $.385$ | $.492$
\+ Noisegate | ecnsgdr | $.414$ | $.398$ | $.375$ | $.485$
\+ Multitap delay (full) | ecnsgdr | $.409$ | $.395$ | $.375$ | $.469$

###  2.4 Results

Table 1 reports the evaluation results that are calculated over the entire
song. First, the base graph results in an audio loss $L_{\mathrm{a}}$ of
$19.7$. The side-channel loss $L_{\mathrm{s}}$ is especially large as most
source tracks are close to mono while the target mixes have wide stereo
images. With the gain/pannings and stereo imagers, we can achieve “rough
mixes” with a loss of $0.676$. Then, we fill in the missing details with the
remaining processor types. Every type improves the match, and the full mixing
console reports a loss of $0.409$. Also, the top $5$ rows of Figure 7 show
mid/side log-magnitude STFTs of the target mixes, matches of the mixing
consoles, and their errors. We report the results with $3$, $4$, $6$, and $7$
types where the choice of processors and their order follow Table 1; see the
supplementary page for the results on other configurations and additional
songs. Again, we can observe that adding each type improves the match from the
spectrogram error plots. Furthermore, each song benefits more from different
types; for the song RockSteady, the multitap delays improve the match more
than the reverbs (Figure 7b), which is different from the average trend. Yet,
this is expected since the original mix heavily uses the delay effects.
Finally, we note that mixes from MixingSecrets are more challenging to match
than the others; it reports a mean audio loss of $0.545$, while MedleyDB and
Internal report $0.296$ and $0.385$, respectively.

##  3 Music Mixing Graph Search

Considering the full mixing console $G_{\mathrm{c}}$ as an upper bound in
terms of the matching performance, we want to find a sparser graph with a
similar match quality. We achieve this by pruning the console as much as
possible while keeping the loss increase up to a tolerance threshold $\tau$.
This objective can be written as

$\mathrm{minimize}\;\;|V_{\mathrm{p}}|\quad\mathrm{s.t.}\;\;\min
L_{\mathrm{a}}(G_{\mathrm{p}})\leq\min L_{\mathrm{a}}(G_{\mathrm{c}})+\tau$
(7)

where $V_{\mathrm{p}}$ and $|\cdot|$ denote the pruned graph’s node set and
its cardinality, respectively. We define the pruning as removal of the nodes
$V_{\mathrm{c}}\setminus V_{\mathrm{p}}$ and re-routing of their edges, in a
way that is equivalent to setting them to “bypass,” i.e., $w_{i}=0$ for
$v_{i}\in V_{\mathrm{c}}\setminus V_{\mathrm{p}}$. Also, $\min(\cdot)$
signifies that we are (ideally) interested in the optimized audio loss. We
only prune the processors, not the auxiliary nodes. Hence, we define a pruning
ratio $\rho$ as the number of pruned processors divided by the number of
processors in the initial console.

###  3.1 Iterative Pruning

Finding the optimal (sparsest) solution $V_{\mathrm{p}}^{*}$ is prohibitively
expensive. First, due to the interaction between the processors, we need a
combinatorial search. As such, we instead assume their independence and prune
the processors in a greedy manner. Following the iterative approach [22], we
gradually remove processors whenever the tolerance condition is satisfied.
Under this setup, we still need to fine-tune intermediate pruned graphs before
evaluating the tolerance condition. For reasonable computational complexity,
we simply omit this fine-tuning, paying the cost of possibly missing more
removable processors. Our method is summarized in Algorithm 1 (in the
following parentheses denote line numbers). First, we construct a mixing
console $G_{\mathrm{c}}=(V,E)$, optimize its parameters $\mathbf{P}$ and
dry/wet weights $\mathbf{w}$, and evaluate the loss (3-5). This validation
loss $L_{\mathrm{a}}^{\mathrm{min}}$ serves as a pruning threshold with the
tolerance $\tau$. Then, we alternate between pruning and fine-tuning, i.e.,
further optimization of the remaining parameters and weights (7-20). Each
pruning stage consists of multiple trials, which sample subsets of candidates
${V}_{\mathrm{cand}}$ from the set of remaining processors $V_{\mathrm{pool}}$
(10) and check whether they are removable (12). We keep the pruning if the
result satisfies the constraint or cancel it otherwise (12-15). We repeat this
process until the terminal condition (9) is satisfied. Implementation-wise, we
multiply binary masks, $\mathbf{m}$ and ${\mathbf{m}}_{\>\\!\mathrm{cand}}$,
to the weight vector $\mathbf{w}$ to mimic the pruning during the trials (11).
After that, we actually update the graph and remove the pruned processors’
parameters and weights for faster search (18). Sometimes, albeit rare, the
pruning can improve the match. In this case, we update the threshold (13).

Algorithm 1 Music mixing graph search with iterative pruning.

1:A mixing console $G_{\mathrm{c}}$, dry tracks $\mathbf{S}$, and mixture $y$

2:Pruned graph $G_{\mathrm{p}}$, parameters $\mathbf{P}$, and weights
$\mathbf{w}$

3:$\mathbf{P},\mathbf{w}\leftarrow\mathrm{Initialize}\>\\!(G_{\mathrm{c}})$

4:$\mathbf{P},\mathbf{w}\leftarrow\mathrm{Train}\>\\!(G_{\mathrm{c}},\mathbf{P},\mathbf{w},\mathbf{S},y)$

5:$L_{\mathrm{a}}^{\mathrm{min}}\leftarrow\mathrm{Evaluate}\>\\!(G_{\mathrm{c}},\mathbf{P},\mathbf{w},\mathbf{S},y)$

6:$G_{\mathrm{p}}\leftarrow G_{\mathrm{c}}$

7:for $n$ $\leftarrow$ $1$ to $N_{\mathrm{iter}}$ do

8:
$V_{\mathrm{pool}},\mathbf{m}\leftarrow\mathrm{GetAllProcessors}\>\\!(V),\mathbf{1}$

9: while $\mathrm{TryPrune}\>\\!(V_{\mathrm{pool}},\mathbf{w},\mathbf{m})$ do

10:
$V_{\mathrm{cand}},{\mathbf{m}}_{\>\\!\mathrm{cand}}\leftarrow\mathrm{SampleCandidate}\>\\!(V_{\mathrm{pool}},\mathbf{w})$

11:
$L_{\mathrm{a}}\leftarrow\mathrm{Evaluate}\>\\!(G_{\mathrm{p}},\mathbf{P},\mathbf{w}\odot\mathbf{m}\odot{\mathbf{m}}_{\>\\!\mathrm{cand}},\mathbf{S},y)$

12: if $L_{\mathrm{a}}<L_{\mathrm{a}}^{\mathrm{min}}+\tau$ then

13:
$L_{\mathrm{a}}^{\mathrm{min}}\leftarrow\min(L_{\mathrm{a}}^{\mathrm{min}},L_{\mathrm{a}})$

14: $\mathbf{m}\leftarrow\mathbf{m}\odot{\mathbf{m}}_{\>\\!\mathrm{cand}}$

15: end if

16:
$V_{\mathrm{pool}}=\mathrm{UpdatePool}\>\\!(V_{\mathrm{pool}},V_{\mathrm{cand}})$

17: end while

18:
$G_{\mathrm{p}},\mathbf{P},\mathbf{w}\leftarrow\mathrm{Prune}\>\\!(G_{\mathrm{p}},\mathbf{P},\mathbf{w},\mathbf{m})$

19:
$\mathbf{P},\mathbf{w}\leftarrow\mathrm{Train}\>\\!(G_{\mathrm{p}},\mathbf{P},\mathbf{w},\mathbf{S},y)$

20:end for

21:return $G_{\mathrm{p}},\mathbf{P},\mathbf{w}$

###  3.2 Candidate Sampling

The remaining design choices are choosing an appropriate candidate set
$V_{\mathrm{cand}}$ (10, 16) and deciding when to terminate the trials (9). We
explore the following $3$ approaches.

  * •

Brute-force — We random-sample every processor one by one, i.e.,
$|V_{\mathrm{cand}}|=1$. This granularity could achieve high sparsity, but
comes with a large computational cost.

  * •

Dry/wet — For efficient pruning, we need an informed guess of each node’s
importance. Intuitively, we can use each dry/wet weight $w_{i}$ as an
approximate importance. This observation leads to the following. For each
pruning iteration:

    1. (i)

We create a set of remaining processor types $T_{\mathrm{pool}}$. Next, we
count the number of processors of each type $t\in T_{\mathrm{pool}}$, denoted
as $N_{t}$.

    2. (ii)

For each trial, we sample a type $t\in T_{\mathrm{pool}}$ and choose the
smallest-weight processors of that type as candidates. The number of
candidates is set to $|V_{\mathrm{cand}}|=\max\;\\!(1,\lfloor
r_{t}N_{t}\rceil)$ where $r_{t}$ denotes the portion of the chosen processors
and is initialized to $0.1$ for every pruning iteration.

    3. (iii)

When the trial fails, we perform one of the followings. If
$|V_{\mathrm{cand}}|>1$, we halve the candidate set, i.e., $r_{t}\leftarrow
r_{t}/2$. Otherwise, i.e., if $|V_{\mathrm{cand}}|=1$, we finish the search of
this type by removing it from the pool as $T_{\mathrm{pool}}\leftarrow
T_{\mathrm{pool}}\setminus\\{t\\}$.

    4. (iv)

We iterate above two (ii)-(iii) until $T_{\mathrm{pool}}=\emptyset$.

This way, we can skip large-weight nodes and evaluate multiple candidates,
reducing the total number of trials. Note that if we set $r_{t}=0.5$, this
method is similar to the simple binary search. However, it can lead to over-
pruning of specific types sampled early in (ii). Hence, we set $r_{t}$ to a
more conservative value $0.1$.

  * •

Hybrid — Solely relying on the weight values could miss some processors that
can be pruned but have large weights. We mitigate this by combining the above
two, running the brute-force method for every $4^{\mathrm{th}}$ iteration.

By default, we use the hybrid method with tolerance $\tau=0.01$.

Figure 3:  Process of iterative pruning (hybrid, $\tau=0.01$). $24$ songs ($8$
songs per dataset) are shown; each color represents an individual song. The
upper and lower rows show the pruning ratios and mean dry/wet weights. The
yellow-shaded regions show the pruning phase.

###  3.3 Optimization

We use identical audio loss $L_{\mathrm{a}}$ and gain-staging regularization
$L_{\mathrm{g}}$. To promote sparsity, we add a weight regularization
$L_{\mathrm{p}}$, a $l_{1}$ norm of the weight $\mathbf{w}$. Hence, the full
objective is as follows,

$L(\mathbf{P},\mathbf{w})=L_{\mathrm{a}}(\mathbf{P},\mathbf{w})+\alpha_{\mathrm{g}}L_{\mathrm{g}}(\mathbf{P})+\alpha_{\mathrm{p}}L_{\mathrm{p}}(\mathbf{w}).$
(8)

We first train the console with $6$\mathrm{k}$$ steps. Then, we repeat
$N_{\mathrm{iter}}=12$ rounds of pruning, each with $0.5$\mathrm{k}$$-step
fine-tuning. As a result, the total number of optimization steps is the same
as the previous console training. During the first $4$\mathrm{k}$$ steps of
the pruning phase, we linearly increase the sparsity coefficient
$\alpha_{\mathrm{p}}$ from $0$ to $10^{-2}$. While we halved the full console
optimization steps, which could lead to increased loss, it is justified due to
the tight resource constraints. With a RTX3090 GPU, each song took about
$56$\mathrm{m}$$, $29$\mathrm{m}$$, and $36$\mathrm{m}$$ using the brute-
force, dry/wet, and hybrid methods, respectively.

###  3.4 Results

#### Pruning process

Figure 3 shows how the pruning progresses. Each graph’s sparsity increases
gradually while its weights adapt over time. This trend is different for
different processor types. The mean objective metrics are reported in Table 2.
The default setting reports an average audio loss $L_{\mathrm{a}}$ of $0.422$,
an $0.013$ increase from the full consoles, slightly exceeding the tolerance
$\tau=0.01$. This was expected due to the shorter full console training. The
average pruning ratio $\rho$ is $0.67$ and the equalizer and stereo imager are
the most and least remaining types ($0.46$ and $0.86$), respectively. We note
that MedleyDB and MixingSecrets report similar pruning ratios of $0.61$ and
$0.62$, respectively. However, the Internal graphs are more sparse; their
average pruning ratio is $0.77$.

Table 2: Pruning results with various candidate selection methods and tolerance $\tau$. The subscripts denote per-type pruning ratios.  | $\tau$ | $L_{\mathrm{a}}$ | $\rho$ | $\rho_{\text{\scriptsize$\texttt{g}$}}$ | $\rho_{\text{\scriptsize$\texttt{s}$}}$ | $\rho_{\text{\scriptsize$\texttt{e}$}}$ | $\rho_{\text{\scriptsize$\texttt{r}$}}$ | $\rho_{\text{\scriptsize$\texttt{c}$}}$ | $\rho_{\text{\scriptsize$\texttt{n}$}}$ | $\rho_{\text{\scriptsize$\texttt{d}$}}$
---|---|---|---|---|---|---|---|---|---|---
Mix console | $-$ | $.409$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
Brute-force | $.01$ | $.424$ | $.69$ | $.54$ | $.85$ | $.53$ | $.76$ | $.71$ | $.78$ | $.69$
Dry/wet | $.01$ | $.420$ | $.62$ | $.51$ | $.84$ | $.38$ | $.69$ | $.66$ | $.76$ | $.53$
Hybrid | $.001$ | $.411$ | $.49$ | $.35$ | $.76$ | $.27$ | $.53$ | $.57$ | $.62$ | $.34$
$.01$ | $.422$ | $.67$ | $.51$ | $.86$ | $.46$ | $.71$ | $.71$ | $.79$ | $.63$
$.1$ | $.499$ | $.87$ | $.73$ | $.94$ | $.81$ | $.90$ | $.85$ | $.91$ | $.92$
Figure 4:  Each node’s weight and loss increase when pruned.

(a) $\tau=0.001$
$\rho=0.45,L_{\mathrm{a}}=0.525$ (b) $\tau=0.01$
$\rho=0.74,L_{\mathrm{a}}=0.525$

(c) $\tau=0.1$
$\rho=0.85,L_{\mathrm{a}}=0.611$

Figure 5:  Pruning results (hybrid method) with various tolerances. Song:
TablaBreakbeatScience_RockSteady.

#### Sampling method comparison

Here, we fix the tolerance $\tau$ to $0.01$ and compare the candidate sampling
approaches; see Table 2. As expected, the brute-force method achieves the
highest sparsity, reporting a pruning ratio of $0.69$. Its average audio loss
is also the highest, $0.424$, an $0.015$ increase from the mixing console
result. The dry/wet method prunes the least with $0.62$, $7\%$ lower than the
brute-force method. However, its audio loss is the lowest, $0.420$, as more
processors remained. We can investigate the cause of this difference in
sparsity by analyzing the relationship between each dry/wet weight $w_{i}$ and
the loss increase $\Delta_{i}$ caused by pruning the processor $v_{i}$ defined
as follows,

$\Delta_{i}=L_{\mathrm{a}}(G\setminus\\{v_{i}\\})-L_{\mathrm{a}}(G).$ (9)

Figure 4 shows scatterplots for $2$ random-sampled songs, one for each song.
Each point $(w_{i},\Delta_{i})$ corresponds to each processor after the
initial console training. To maximize the sparsity using the dry/wet method, a
monotonic relationship between the weights $w_{i}$ and loss increases
$\Delta_{i}$ is desirable, which is unfortunately not the case. Yet, a
positive correlation exists, and this becomes more evident when we analyze the
relationship for each type separately, justifying the per-type candidate
selection. Still, we cannot completely remove the weakness of the dry/wet
method, leading us to the hybrid strategy as a compromise. We note that the
pruning methods are not only different in sparsity but also in trade-offs
between sparsity and match performance. By evaluating the methods with more
fine-grained tolerance settings ($7$ values from $0.001$ to $0.2$), we
observed that the brute-force method finds graphs with better matches even
with the same graph size, closely followed by the hybrid method; refer to the
supplementary page for the details.

#### Choice of tolerance

Finally, we analyze the effect of the value of tolerance $\tau$. Even with a
very low tolerance $\tau=0.001$, we can nearly halve the number of processors,
i.e., $\rho=0.49$. If we set the value too high, e.g., $\tau=0.1$, the
resulting graphs are highly sparse but degrade their matches
($L_{\mathrm{a}}=0.499$, i.e., $0.090$ increase). The default setting of
$\tau=0.01$ seems “just right,” balancing the match performance and graph
sparsity. We can verify this with the spectrogram errors (bottom $3$ rows for
each subplots; see Figure 7 and supplementary page). There is no noticeable
degradation from the full consoles to $\tau=0.001$ and $0.01$.

#### Case study

We report the pruning method’s behavior from observations of the individual
results.

  * •

Recall that, for the song RockSteady, there was no clear performance
improvement when we added the reverbs (Figure 7b). Hence, we can expect those
reverbs to be pruned with a moderate tolerance $\tau$. Figure 5 shows that
this is indeed the case; only $5/14$ reverbs are left when $\tau=0.001$ and
$0/14$ for $\tau=0.01$, which is much less than the average statistics (Table
2 and 3). When $\tau=0.1$, processors for the details get removed; only the
gain/pannings and equalizers remain. See captions in Figure 5 for the pruning
ratios and audio losses of the pruned graphs (the full console has an audio
loss of $0.523$).

  * •

The current pruning method fails to detect some redundant processors. In
Figure 5b, the bottom $2$ sources are processed with $3$ gain/pannings. Since
there is no nonlinear or time-varying processor between those, at least one
can be “absorbed” by the others. While this case can be handled with some
post-processing, it hints that we might have missed more sparse graphs.

  * •

Each pruning of the same song yields a slightly different graph. Pruning a
mixing console of GirlOnABridge multiple times resulted in graphs with the
number of processors from $19$ to $22$. This is because our iterative pruning
has a stochastic and greedy nature; candidates that were sampled early are
more likely to be pruned. Refer to the supplementary page for the pruned
graphs.

  * •

The pruning does not necessarily result in graphs that are close to the
maximum loss $L_{\mathrm{a}}(G_{\mathrm{c}})+\tau$. For RockSteady, pruning
with $\tau=0.01$ resulted in a loss of $0.525$, much lower than the threshold.
Interestingly, the $\tau=0.001$ case achieved the same loss in spite of a much
lower pruning ratio ($0.56$ versus $0.74$).

  * •

Processors for sources with short spans and low energy tend to get pruned as
their contributions to the audio loss are small. Yet, we found that this could
sometimes be perceptually noticeable.

Table 3:  Pruning results with the default setting on the full dataset.  | $L_{\mathrm{a}}$ | $\rho$ | $\rho_{\text{\scriptsize$\texttt{g}$}}$ | $\rho_{\text{\scriptsize$\texttt{s}$}}$ | $\rho_{\text{\scriptsize$\texttt{e}$}}$ | $\rho_{\text{\scriptsize$\texttt{r}$}}$ | $\rho_{\text{\scriptsize$\texttt{c}$}}$ | $\rho_{\text{\scriptsize$\texttt{n}$}}$ | $\rho_{\text{\scriptsize$\texttt{d}$}}$
---|---|---|---|---|---|---|---|---|---
MedleyDB | $.431$ | $.63$ | $.37$ | $.84$ | $.44$ | $.69$ | $.74$ | $.77$ | $.57$
MixingSecrets | $.625$ | $.64$ | $.50$ | $.87$ | $.33$ | $.64$ | $.63$ | $.80$ | $.69$
Internal | $.434$ | $.75$ | $.70$ | $.87$ | $.55$ | $.73$ | $.85$ | $.86$ | $.72$
Total | $.506$ | $.69$ | $.57$ | $.87$ | $.45$ | $.69$ | $.75$ | $.82$ | $.69$
Figure 6:  Statistics of the consoles and pruned graphs (full data). Each
dataset’s results are stacked to form the full histograms.

#### Full results

Finally, we pruned every song in the full dataset ensemble. Table 3 reports
the results. The overall trend follows the evaluation subset results but with
higher average audio loss ($0.509$ compared to the previous $0.422$). Figure 6
shows statistics of the $3$ datasets, initial mixing console graphs, and their
pruned versions. MedleyDB has the smallest number of source tracks, an average
of $17.6$. The Internal has the largest ($28.8$), closely followed by the
MixingSecrets ($27.9$). The Internal dataset also has more subgroups,
resulting in even larger mixing consoles. This is one potential cause of the
higher sparsity of its pruned graphs; more processors were initially used to
match the mix, and many of them were redundant. On average, $72.1$ processors
($108.5$ nodes) were remained for each song. Since each full mixing console
has an average of $247.6$ processors ($280.1$ nodes), we achieved a pruning
ratio of $0.692$.

##  4 Discussion

#### Summary

We started with a general formulation of retrieving mixing graphs from dry
sources and mix. Then, we posed restrictions to cast the search to the pruning
of mixing consoles, making it computationally feasible and obtaining more
interpretable graphs. Next, with additional assumptions, we derived the
iterative method that gradually removes negligible processors in a stochastic
and greedy manner. As a result, instead of finding the exact optimal, our
method gives (or “samples”) one of the close-to-optimal graphs. With the
differentiable processors and relaxation of the pruning with the dry/wet
weights, we optimized this objective via gradient descent. We explored $3$
methods to choose pruning candidates, comparing them in terms of their
computational cost and graph sparsity. The hybrid method gave a good
compromise, so we used it to gather over one thoudsand graph-audio pairs.

#### Future works

We list possible extensions of our method.

  * •

The choice of processors and their implementations directly affect the match
quality. Our setup, including the equalizer with zero-phase FIR and the reverb
based on STFT mask, was motivated by its simplicity and fast computation on a
GPU. However, other alternatives exist, e.g., parametric equalizer [20] and
artificial reverberation [36], that allow more efficient computation in CPU
and have compact parameterizations. Also, the spectrogram errors showed clear
temporal patterns (vertical stripes), indicating that the loudness dynamics
were not precisely matched. We suspect it is due to the ballistics
approximation error, as recently reported [37]. If so, we might need a more
sophisticated implementation of the compressor and noisegate. Also, the
current method does not support time-varying parameters (or “automation”),
which can cause audible errors. For example, we could not match fade-out,
i.e., a gradual decrease in track loudness. Finally, we can add other
processor types, e.g., saturation/distortion [38] or modulation effects [39].

  * •

We note several considerations to improve the current pruning method in terms
of sparsity, match quality, and interpretability. First, we can modify the
mixing console to reflect real-world practices more. For example, we can add
send and return loops with additional processor chains. Post-equalizers for
compressors and processors with multiple inputs or outputs (e.g., auxiliary
sidechain and crossover filter) are also commonly used. Second, to prevent the
pruning from harming the perceptual quality, the tolerance condition and the
objective function must be appropriately designed. We used a simple multi-
resolution STFT loss [32, 34], which has been reported to miss some perceptual
features [40, 41]. Hence, we might need an alternative objective as a remedy
[42]. Third, as discussed before, using average loss to determine the pruning
might be inappropriate. Lastly, to increase the sparsity, more advanced neural
network pruning techniques [23, 24] and domain-specific post-processing, e.g.,
merging LTI processors to a single processor with the combined effect, can be
applied.

  * •

We may relax the prior assumptions and restrictions on graph structures. This
will expand our search space and require different search methods other than
pruning. For example, allowing arbitrary processor order extends our framework
to different architecture search [25, 26]. A completely different approach
based on reinforcement learning could also be possible [43]. While all of
these are promising, balancing flexibility, match quality, and computation
cost will be the main challenge.

(a) Torres_NewSkin

(b) TablaBreakbeatScience_RockSteady

Figure 7:  Log-magnitude spectrograms of the matched mixes (odd columns) of
mixing consoles ($4$ center rows) and pruned graphs ($3$ bottom rows; in
$\mathrm{d}\mathrm{B}$). The even columns show the match errors.

## References

  * [1] P. D. Pestana and J. D. Reiss, “Intelligent audio production strategies informed by best practices,” 2014.
  * [2] F. Everardo, “Towards an automated multitrack mixing tool using answer set programming,” in _14th SMC Conf_ , 2017.
  * [3] E. Perez-Gonzalez and J. Reiss, “Automatic gain and fader control for live mixing,” in _IEEE WASPAA_ , 2009.
  * [4] B. De Man and J. D. Reiss, “A knowledge-engineered autonomous mixing system,” in _AES Convention 135_ , 2013.
  * [5] M. A. Martinez Ramirez _et al._ , “Automatic music mixing with deep learning and out-of-domain data,” in _ISMIR_ , 2022.
  * [6] D. Koszewski, T. Görne, G. Korvel, and B. Kostek, “Automatic music signal mixing system based on one-dimensional wave-u-net autoencoders,” _EURASIP Journal on Audio, Speech, and Music Processing_ , vol. 2023, no. 1, 2023.
  * [7] C. J. Steinmetz, J. Pons, S. Pascual, and J. Serrà, “Automatic multitrack mixing with a differentiable mixing console of neural audio effects,” in _IEEE ICASSP_ , 2021.
  * [8] R. M. Bittner, J. Salamon, M. Tierney, M. Mauch, C. Cannam, and J. P. Bello, “MedleyDB: A multitrack dataset for annotation-intensive mir research.” in _ISMIR_ , vol. 14, 2014.
  * [9] R. M. Bittner, J. Wilkins, H. Yip, and J. P. Bello, “MedleyDB 2.0: New data and a system for sustainable data collection,” _ISMIR LBD_ , 2016.
  * [10] M. Senior, _Mixing secrets for the small studio_ , 2018.
  * [11] S. Lee, J. Park, S. Paik, and K. Lee, “Blind estimation of audio processing graph,” in _IEEE ICASSP_ , 2023.
  * [12] C. Mitcheltree and H. Koike, “SerumRNN: Step by step audio VST effect programming,” in _Artificial Intelligence in Music, Sound, Art and Design_ , 2021.
  * [13] J. Guo and B. McFee, “Automatic recognition of cascaded guitar effects,” in _DAFx_ , 2023.
  * [14] N. Masuda and D. Saito, “Improving semi-supervised differentiable synthesizer sound matching for practical applications,” _IEEE/ACM TASLP_ , vol. 31, 2023.
  * [15] N. Uzrad _et al._ , “DiffMoog: a differentiable modular synthesizer for sound matching,” _arXiv:2401.12570_ , 2024.
  * [16] F. Caspe, A. McPherson, and M. Sandler, “DDX7: Differentiable FM synthesis of musical instrument sounds,” in _ISMIR_ , 2022.
  * [17] J. Colonel, “Music production behaviour modelling,” 2023.
  * [18] “The mixing console — split, inline and hybrids,” https://steemit.com/sound/@jamesub/the-mixing-console-split-inline-and-hybrids, accessed: 2024-02-26.
  * [19] J. Engel, L. H. Hantrakul, C. Gu, and A. Roberts, “DDSP: differentiable digital signal processing,” in _ICLR_ , 2020.
  * [20] S. Nercessian, “Neural parametric equalizer matching using differentiable biquads,” in _DAFx_ , 2020.
  * [21] C. J. Steinmetz, N. J. Bryan, and J. D. Reiss, “Style transfer of audio effects with differentiable signal processing,” _JAES_ , vol. 70, no. 9, 2022.
  * [22] G. Castellano, A. M. Fanelli, and M. Pelillo, “An iterative pruning algorithm for feedforward neural networks,” _IEEE transactions on Neural networks_ , vol. 8, no. 3, 1997.
  * [23] Y. He and L. Xiao, “Structured pruning for deep convolutional neural networks: A survey,” _arXiv:2303.00566_ , 2023.
  * [24] H. Cheng, M. Zhang, and J. Q. Shi, “A survey on deep neural network pruning-taxonomy, comparison, analysis, and recommendations,” _arXiv:2308.06767_ , 2023.
  * [25] H. Liu, K. Simonyan, and Y. Yang, “DARTS: Differentiable architecture search,” in _ICLR_ , 2019.
  * [26] Z. Ye, W. Xue, X. Tan, Q. Liu, and Y. Guo, “NAS-FM: Neural architecture search for tunable and interpretable sound synthesis based on frequency modulation,” _arXiv:2305.12868_ , 2023.
  * [27] C. J. Steinmetz, S. S. Vanka, M. A. Martínez-Ramírez, and G. Bromham, _Deep Learning for Automatic Mixing_. ISMIR, Dec. 2022.
  * [28] J. Koo _et al._ , “Music mixing style transfer: A contrastive learning approach to disentangle audio effects,” in _IEEE ICASSP_ , 2023.
  * [29] B. Hayes, C. Saitis, and G. Fazekas, “Sinusoidal frequency estimation by gradient descent,” in _IEEE ICASSP_ , 2023.
  * [30] U. Zölzer, Ed., _DAFX: Digital Audio Effects_ , 2nd ed., 2011.
  * [31] M. Fey and J. E. Lenssen, “Fast graph representation learning with pytorch geometric,” _arXiv:1903.02428_ , 2019.
  * [32] R. Yamamoto, E. Song, and J.-M. Kim, “Parallel wavegan: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram,” in _IEEE ICASSP_ , 2020.
  * [33] A. Wright and V. Välimäki, “Perceptual loss function for neural modeling of audio systems,” in _IEEE ICASSP_ , 2020.
  * [34] C. J. Steinmetz and J. D. Reiss, “auraloss: Audio focused loss functions in PyTorch,” in _Digital Music Research Network One-day Workshop (DMRN+15)_ , 2020.
  * [35] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” _arXiv:1711.05101_ , 2017.
  * [36] S. Lee, H.-S. Choi, and K. Lee, “Differentiable artificial reverberation,” _IEEE/ACM TASLP_ , vol. 30, 2022.
  * [37] C. J. Steinmetz, T. Walther, and J. D. Reiss, “High-fidelity noise reduction with differentiable signal processing,” in _AES Convention 155_ , 2023.
  * [38] J. T. Colonel, M. Comunità, and J. Reiss, “Reverse engineering memoryless distortion effects with differentiable waveshapers,” in _AES Convention 153_ , 2022.
  * [39] A. Carson, S. King, C. V. Botinhao, and S. Bilbao, “Differentiable grey-box modelling of phaser effects using frame-based spectral processing,” in _DAFx_ , 2023.
  * [40] B. Hayes, J. Shier, G. Fazekas, A. McPherson, and C. Saitis, “A review of differentiable digital signal processing for music & speech synthesis,” _Frontiers in Signal Process._ , 2023.
  * [41] J. Turian and M. Henry, “I’m sorry for your loss: Spectrally-based audio distances are bad at pitch,” in _“I Can’t Believe It’s Not Better!” NeurIPS workshop_ , 2020.
  * [42] C. Vahidi _et al._ , “Mesostructures: Beyond spectrogram loss in differentiable time–frequency analysis,” _JAES_ , 2023.
  * [43] J. You, B. Liu, Z. Ying, V. Pande, and J. Leskovec, “Graph convolutional policy network for goal-directed molecular graph generation,” _NeurIPS_ , 2018.
  * [44] M. A. Martínez-Ramírez, O. Wang, P. Smaragdis, and N. J. Bryan, “Differentiable signal processing with black-box audio effects,” in _IEEE ICASSP_ , 2021.
  * [45] A. Paszke _et al._ , “PyTorch: An imperative style, high-performance deep learning library,” _NeurIPS_ , 2019.
  * [46] T. Elsken, J. H. Metzen, and F. Hutter, “Neural architecture search: A survey,” _Journal of Machine Learning Research_ , vol. 20, no. 55, 2019.
  * [47] D. C. Elton, Z. Boukouvalas, M. D. Fuge, and P. W. Chung, “Deep learning for molecular design—a review of the state of the art,” _Molecular Systems Design & Engineering_, vol. 4, no. 4, 2019.
  * [48] B. Zoph and Q. V. Le, “Neural architecture search with reinforcement learning,” _arXiv preprint arXiv:1611.01578_ , 2016.

##  Appendix A Related Works

###  A.1 Composition of audio processors

Most audio processors are designed to modify some specific properties of their
input signals, e.g., magnitude response, loudness dynamics, and stereo width.
As such, combining multiple processors is a common practice to achieve the
full effect. Following the main text, we will use the terminology “graph” to
represent this composition, although some previous works considered simple
structures that allow a more compact form, e.g., a sequence. Now, we outline
the previous attempts that tried to estimate the processing graph or its
parameters from reference audio. These works differ in task and domain,
processors, graph structure, and estimation methods. For example, if the
references are dry sources and a wet mixture, this task becomes reverse
engineering [11, 12, 13, 17]. In terms of the prediction targets, some fixed
the graph and estimated only the parameters [7, 15, 17, 19, 44]. Others tried
to predict the graph [13] or both [11, 12, 26, 16]. Table 4 summarizes and
highlights such differences.

###  A.2 Differentiable signal processing

#### Differentiable processor

Exact implementation or approximation of processors in an automatic
differentiation framework, e.g., pytorch [45], enables parameter optimization
via gradient descent. Numerous efforts have been focused on converting
existing audio processors to their differentiable versions [17, 21, 38, 39,
20, 36]; refer to the recent review [40] and references therein for more
details. In many cases, these processors are combined with neural networks,
whose computation is done in GPU. Thus, converting the audio processors to be
“GPU-friendly” has been an active research topic. For example, for a linear
time-invariant (LTI) system with a recurrent structure, we can sample its
frequency response to approximate its infinite impulse response (IIR) instead
of directly running the recurrence; the former is faster than the latter [20,
36]. However, it is nontrivial to apply a similar trick to nonlinear,
recurrent, or time-varying processors. Typically, further simplifications and
approximations are employed, e.g., replacing the nonlinear recurrent part with
an IIR filter [21] or assuming frame-wise LTI to a linear time-varying system
[39]. Sometimes, we can only access input and output signals. In such a case,
one can approximate the gradients with finite difference methods [44, 37] or
use a pre-trained auxiliary neural network that mimics the processors [7]. In
the literature, these are also referred to as “differentiable;” hence, it is
rather an umbrella term encompassing all methods that obtain the output
signals or gradients within a reasonable amount of time. Nevertheless, our
work limits the focus to the implementations in the automatic differentiation
framework.

#### Audio processing graph

Now, consider a composition of multiple differentiable processors; the entire
graph remains differentiable due to the chain rule. However, the following
practical considerations remain. If we fix the processing graph prior to the
optimization and the graph size is relatively small, we can implement the
“differentiable graph” following the existing implementations [19, 15]. That
is, we compute every processor one by one in a pre-defined topological order.
However, we have the following additional requirements. First, the pruning
changes the graph during the optimization. Therefore, our implementation must
take a graph and its parameters along with the source signals as input
arguments for every forward pass. Note that this feature is also necessary
when training a neural network that predicts the parameters of any given graph
[11]. Second, the size of our graphs is much larger than the ones from
previous works [19, 15, 26, 11]. In this case, the one-by-one computation
severely bottlenecks the computation speed. Therefore, we derived a flexible
and efficient graph computation algorithm (i) that can take different graphs
for each forward pass as input and (ii) performs batched processing of
multiple processors within a graph, utilizing the parallelism of GPUs.
Finally, we note that other than the differentiation with respect to the input
signals $\mathbf{S}$ and parameters $\mathbf{P}$, one might be interested in
differentiation with respect to the graph structure $G$. The proposed pruning
method performs this to a limited extent; deletion of a node $v_{i}$ is a
binary operation that modifies the graph structure. We relaxed this to a
continuous dry/wet weight $w_{i}$ and optimized it with the audio loss
$L_{\mathrm{a}}$ and regularization $L_{\mathrm{p}}$.

###  A.3 Graph search

Several independent research efforts in various domains exist that search for
graphs that satisfy certain requirements. For example, neural architecture
search (NAS) aims to find a neural network architecture that achieves improved
performance [46]. In this case, the search space consists of graphs, with each
node (or edge) representing one of the primitive neural network layers. One
particularly relevant work to ours is a differentiable architecture search
(DARTS) [25], which relaxes the choice of each layer to a categorical
distribution and optimizes it via gradient descent. Theoretically, our method
can be naturally extended to this approach; we only need to change our $2$-way
choice (prune or not) to $(N+1)$-way (bypass or select one of $N$ processor
types). DARTS is clearly more flexible and general, allowing an arbitrary
order of processors. However, it also greatly increases the computational
cost, as we must compute all $N$ processors to compute their weight sum for
every node. For example, if we want to keep the mixing console structure and
allow arbitrary processor choices, the memory complexity becomes $O(N^{2})$
instead of the current $O(N)$. In other words, we must pay additional costs to
increase the size of the search space. This cost increase is especially
critical to us since we have to find a graph for every song. Another popular
related domain is the generation/design of molecules with desired chemical
properties [47]. One dominant approach for this task is to use reinforcement
learning (RL), which estimates each graph by making a sequence of decisions,
e.g., adding nodes and edges [43]. RL is an attractive choice since we can be
completely free with prior assumptions on graphs, and we can use arbitrary
quality measures that are not differentiable. We also note that RL can be used
for NAS [48]. However, applying RL to our task has a risk of obtaining
nontrivial mixing graphs that are difficult for practitioners to interpret; we
may need a soft regularization penalty that guides the generation process
towards familiar structures, e.g., ones like the pruned mixing consoles. Also,
it may need much larger computational resources to explore the search space
sufficiently.

##  Appendix B Dry/wet Pruning Algorithm

Algorithm 2 describes the details of the dry/wet method. For a simpler
description, we modified the initialization to include per-type node sets and
weights as in line 8-12. The termination condition is given in line 13. The
trial candidate sampling is implemented in line 14-15. The candidate pool
update is expanded to handle the trial successes and failures separately,
shown in line 20 and 22-27.

Table 4:  A brief summary and comparison of previous works on estimation of compositional audio signal processing.  [15] | _Task & domain_ |  Sound matching $[x]\to[\mathbf{P}]$. The synthesizer parameters $\mathbf{P}$ were estimated to match the reference (target) audio $x$.
---|---|---
| _Processors_ |  Oscillators, envelope generators, and filters that allow parameter modulation as an optional input.
| _Graph_ |  Any pre-defined directed acyclic graph (DAG). For example, a subtractive synthesizer that comprises $2$ oscillators, $1$ amplitude envelope, and $1$ lowpass filter were used in the experiments.
| _Method_ |  Trained a single neural backbone for the reference encoding, followed by multiple prediction heads for the parameters. Optimized with a parameter loss and spectral loss, where the latter is calculated with every intermediate output.
[16] | _Task & domain_ |  Sound matching $[x]\to[\mathbf{P}]$. A frequency-modulation (FM) synthesizer matches recordings of monophonic instruments (violin, flute, and trumpet). Estimates parameters of an operator graph that is empirically searched & selected.
| _Processors_ |  Differentiable sinusoidal oscillators, each used as a carrier or modulator, pre-defined frequencies. An additional FIR reverb is added to the FM graph output for post-processing.
| _Graph_ |  DAGs with at most $6$ operators. Different graphs for different target instruments.
| _Method_ |  Trained a convolutional neural network that estimates envelopes from the target loudness and pitch.
[26] | _Task & domain_ |  Sound matching $[x]\to[G,\mathbf{P}]$. Similar setup to the above [16] plus additional estimation of the operator graph $G$.
| _Processors_ |  Identical to [16], except for the frequency ratio that can be searched.
| _Graph_ |  A subgraph of a supergraph, which resembles a multi-layer perceptron (modulator layers followed by a carrier layer).
| _Method_ |  Trained a parameter estimator for the supergraph and found the appropriate subgraph $G$ with an evolutionary search.
[12] | _Task & domain_ |  Reverse engineering $[s,y]\to[G,\mathbf{P}]$ of an audio effect chain from a subtractive synthesizer (commercial plugin).
| _Processors_ |  $5$ audio effects: compressor, distortion, equalizer, phaser, and reverb. Non-differentiable implementations.
| _Graph_ |  Chain of audio effects generated with no duplicate types (therefore $32$ possible combinations) and random order.
| _Method_ |  Trained a next effect predictor and parameter estimator in a supervised (teacher-forcing) manner.
[13] | _Task & domain_ |  Blind estimation $[y]\to[G]$ and reverse engineering $[s,y]\to[G]$ of guitar effect chains.
| _Processors_ |  $13$ guitar effects, including non-linear processors, modulation effects, ambience effects, and equalizer filters.
| _Graph_ |  A chain of guitar effects. Maximum $5$ processors and a total of $221$ possible combinations.
| _Method_ |  Trained a convolutional neural network with synthetic data to predict the correct combination.
[7] | _Task & domain_ |  Automatic mixing $[\mathbf{S}]\to[\mathbf{P}]$. Estimated parameters of fixed processing chains from source tracks $(K\leq 16)$.
| _Processors_ |  $7$ differentiable processors, where $4$ (gain, polarity, fader, and panning) were implemented exactly. A combined effect of the remaining $3$ (equalizer, compressor, and reverb) was approximated with a single pre-trained neural network.
| _Graph_ |  Tree structure: applied a fixed chain of the $7$ processors for each track, and then summed the chain outputs altogether.
| _Method_ |  Trained a parameter estimator (convolutional neural network) with a spectrogram loss end-to-end.
[44] | _Task & domain_ |  Reverse engineering of music mastering $[s,y]\to[\mathbf{P}]$
| _Processors_ |  A multi-band compressor, graphic equalizer, and limiter. Gradient approximated with a finite difference method.
| _Graph_ |  A serial chain of the processors.
| _Method_ |  Optimized parameters with gradient descent.
[11] | _Task & domain_ |  Blind estimation $[y]\to[G,\mathbf{P}]$ and reverse engineering $[\mathbf{S},y]\to[G,\mathbf{P}]$. Estimates both graph and its parameters for singing voice effect $(K=1)$ or drum mixing $(K\leq 6)$.
| _Processors_ |  A total of $33$ processors, including linear filters, nonlinear filters, and control signal generators. Some processors are multiple-input multiple-output (MIMO), e.g., allowing auxiliary modulations. Non-differentiable implementations.
| _Graph_ |  Complex DAG; splits (e.g., multi-band processing) and merges (e.g., sum and modulation). $30$ processors max.
| _Method_ |  Trained a convolutional neural network-based reference encoder and a transformer variant for graph decoding and parameter estimation. Both were jointly trained via direct supervision of synthetic graphs (e.g., parameter loss).
[17] | _Task & domain_ |  Reverse engineering $[\mathbf{S},y]\to[\mathbf{P}]$ of music mixing. Estimated parameters of a fixed chain for each track.
| _Processors_ |  $6$ differentiable processors: gain, equalizer, compressor, distortion, panning, and reverb.
| _Graph_ |  A chain of $5$ processors (all above types except the reverb) for each dry track (any other DAG can also be used). The reverb is used for the mixed sum.
| _Method_ |  Parameters were optimized with spectrogram loss end-to-end via gradient descent.
Ours | _Task & domain_ |  Reverse engineering $[\mathbf{S},y]\to[G,\mathbf{P}]$ of music mixing. Estimated a chain of processors and their parameters for each track and submix where $K\leq 130$.
| _Processors_ |  $7$ differentiable processors: gain/panning, stereo imager, equalizer, reverb, compressor, noisegate, and delay.
| _Graph_ |  A tree of processing chains with a subgrouping structure (any other DAG can also be used). Processors can be omitted but should follow the fixed order.
| _Method_ |  Joint estimation of the soft masks (dry/wet weights) and processor parameters. Optimized with the spectrogram loss (and additional regularizations) end-to-end via gradient descent. Accompanied by hard pruning stages.

Table 5:  Per-dataset results of the mixing consoles with different processor type configurations.  |  | MedleyDB | MixingSecrets | Internal
---|---|---|---|---
|  | $L_{\mathrm{a}}$ | $L_{\mathrm{lr}}$ | $L_{\mathrm{m}}$ | $L_{\mathrm{s}}$ | $L_{\mathrm{a}}$ | $L_{\mathrm{lr}}$ | $L_{\mathrm{m}}$ | $L_{\mathrm{s}}$ | $L_{\mathrm{a}}$ | $L_{\mathrm{lr}}$ | $L_{\mathrm{m}}$ | $L_{\mathrm{s}}$
Base graph | ecnigdr | $50.7$ | $1.45$ | $1.42$ | $198$ | $7.30$ | $2.16$ | $2.02$ | $22.9$ | $1.12$ | $.951$ | $.940$ | $1.63$
\+ Gain/panning | ecnsgdr | $.550$ | $.583$ | $.485$ | $.550$ | $.876$ | $.856$ | $.819$ | $.973$ | $.642$ | $.619$ | $.597$ | $.734$
\+ Stereo imager | ecnsgdr | $.541$ | $.564$ | $.483$ | $.553$ | $.847$ | $.834$ | $.791$ | $.928$ | $.538$ | $.616$ | $.595$ | $.727$
\+ Equalizer | ecnsgdr | $.450$ | $.453$ | $.390$ | $.504$ | $.700$ | $.698$ | $.622$ | $.780$ | $.522$ | $.497$ | $.467$ | $.626$
\+ Reverb | ecnsgdr | $.368$ | $.361$ | $.360$ | $.390$ | $.614$ | $.601$ | $.579$ | $.674$ | $.463$ | $.451$ | $.432$ | $.517$
\+ Compressor | ecnsgdr | $.315$ | $.304$ | $.297$ | $.356$ | $.558$ | $.542$ | $.512$ | $.637$ | $.396$ | $.377$ | $.347$ | $.482$
\+ Noisegate | ecnsgdr | $.302$ | $.288$ | $.281$ | $.353$ | $.548$ | $.532$ | $.502$ | $.625$ | $.393$ | $.374$ | $.343$ | $.480$
\+ Multitap delay (full) | ecnsgdr | $.296$ | $.288$ | $.284$ | $.324$ | $.545$ | $.529$ | $.502$ | $.618$ | $.385$ | $.369$ | $.338$ | $.465$
Table 6:  Per-dataset results of the pruning. MC: mixing console. BF: brute-force, DW: dry/wet, and H: hybrid.  |  | MedleyDB | MixingSecrets | Internal
---|---|---|---|---
| $\tau$ | $L_{\mathrm{a}}$ | $\rho$ | $\rho_{\text{\scriptsize$\texttt{g}$}}$ | $\rho_{\text{\scriptsize$\texttt{s}$}}$ | $\rho_{\text{\scriptsize$\texttt{e}$}}$ | $\rho_{\text{\scriptsize$\texttt{r}$}}$ | $\rho_{\text{\scriptsize$\texttt{c}$}}$ | $\rho_{\text{\scriptsize$\texttt{n}$}}$ | $\rho_{\text{\scriptsize$\texttt{d}$}}$ | $L_{\mathrm{a}}$ | $\rho$ | $\rho_{\text{\scriptsize$\texttt{g}$}}$ | $\rho_{\text{\scriptsize$\texttt{s}$}}$ | $\rho_{\text{\scriptsize$\texttt{e}$}}$ | $\rho_{\text{\scriptsize$\texttt{r}$}}$ | $\rho_{\text{\scriptsize$\texttt{c}$}}$ | $\rho_{\text{\scriptsize$\texttt{n}$}}$ | $\rho_{\text{\scriptsize$\texttt{d}$}}$ | $L_{\mathrm{a}}$ | $\rho$ | $\rho_{\text{\scriptsize$\texttt{g}$}}$ | $\rho_{\text{\scriptsize$\texttt{s}$}}$ | $\rho_{\text{\scriptsize$\texttt{e}$}}$ | $\rho_{\text{\scriptsize$\texttt{r}$}}$ | $\rho_{\text{\scriptsize$\texttt{c}$}}$ | $\rho_{\text{\scriptsize$\texttt{n}$}}$ | $\rho_{\text{\scriptsize$\texttt{d}$}}$
MC | $-$ | $.296$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $.545$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $.385$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
BF | $.01$ | $.305$ | $.63$ | $.35$ | $.81$ | $.54$ | $.77$ | $.67$ | $.71$ | $.53$ | $.566$ | $.65$ | $.50$ | $.82$ | $.43$ | $.71$ | $.61$ | $.77$ | $.75$ | $.402$ | $.80$ | $.76$ | $.92$ | $.61$ | $.81$ | $.85$ | $.87$ | $.80$
DW | $.01$ | $.302$ | $.56$ | $.32$ | $.79$ | $.42$ | $.74$ | $.57$ | $.56$ | $.43$ | $.561$ | $.57$ | $.47$ | $.82$ | $.22$ | $.62$ | $.54$ | $.74$ | $.55$ | $.397$ | $.74$ | $.74$ | $.82$ | $.49$ | $.70$ | $.87$ | $.86$ | $.61$
H | $.001$ | $.295$ | $.44$ | $.22$ | $.73$ | $.26$ | $.61$ | $.50$ | $.54$ | $.24$ | $.550$ | $.43$ | $.35$ | $.76$ | $.13$ | $.42$ | $.43$ | $.55$ | $.38$ | $.388$ | $.59$ | $.48$ | $.77$ | $.40$ | $.57$ | $.78$ | $.77$ | $.39$
$.01$ | $.302$ | $.61$ | $.32$ | $.81$ | $.48$ | $.74$ | $.69$ | $.71$ | $.52$ | $.563$ | $.62$ | $.49$ | $.85$ | $.34$ | $.64$ | $.60$ | $.79$ | $.65$ | $.400$ | $.77$ | $.73$ | $.93$ | $.56$ | $.75$ | $.85$ | $.86$ | $.74$
$.1$ | $.375$ | $.83$ | $.59$ | $.93$ | $.83$ | $.92$ | $.80$ | $.84$ | $.90$ | $.648$ | $.84$ | $.70$ | $.91$ | $.75$ | $.86$ | $.83$ | $.93$ | $.91$ | $.474$ | $.93$ | $.90$ | $.99$ | $.84$ | $.92$ | $.93$ | $.95$ | $.96$
Algorithm 2 Music mixing graph search (dry/wet method).

1:A mixing console $G_{\mathrm{c}}$, dry tracks $\mathbf{S}$, and mixture $y$

2:Pruned graph $G_{\mathrm{p}}$, parameters $\mathbf{P}$, and weights
$\mathbf{w}$

3:$\mathbf{P},\mathbf{w}\leftarrow\mathrm{Initialize}\>\\!(G_{\mathrm{c}})$

4:$\mathbf{P},\mathbf{w}\leftarrow\mathrm{Train}\>\\!(G_{\mathrm{c}},\mathbf{P},\mathbf{w},\mathbf{S},y)$

5:$L_{\mathrm{a}}^{\mathrm{min}}\leftarrow\mathrm{Evaluate}\>\\!(G_{\mathrm{c}},\mathbf{P},\mathbf{w},\mathbf{S},y)$

6:$G_{\mathrm{p}}\leftarrow G_{\mathrm{c}}$

7:for $n$ $\leftarrow$ $1$ to $N_{\mathrm{iter}}$ do

8: $T_{\mathrm{pool}}\leftarrow\mathrm{GetProcessorTypeSet}\>\\!(V)$

9: for $t$ in $T_{\mathrm{pool}}$ do

10:
$V_{t},\mathbf{w}_{t}\leftarrow\mathrm{Filter}\>\\!(V,t),\mathrm{Filter}\>\\!(\mathbf{w},t)$

11: $N_{t},r_{t},\mathbf{m}_{t}\leftarrow|V_{t}|,0.1,\mathbf{1}$

12: end for

13: while $T_{\mathrm{pool}}\neq\emptyset$ do

14: $t\leftarrow\mathrm{SampleType}\>\\!(T_{\mathrm{pool}})$

15:
$\bar{V}_{t},\bar{\mathbf{m}}\leftarrow\mathrm{GetLeastWeightNodes}\>\\!(V_{t},\mathbf{w}_{t},\lfloor
N_{t}r_{t}\rceil)$

16:
$L_{\mathrm{a}}\leftarrow\mathrm{Evaluate}\>\\!(G_{\mathrm{p}},\mathbf{P},\mathbf{w}\odot\mathbf{m}\odot\bar{\mathbf{m}},\mathbf{S},y)$

17: if $L_{\mathrm{a}}<L_{\mathrm{a}}^{\mathrm{min}}+\tau$ then

18:
$L_{\mathrm{a}}^{\mathrm{min}}\leftarrow\min(L_{\mathrm{a}}^{\mathrm{min}},L_{\mathrm{a}})$

19: $\mathbf{m}\leftarrow\mathbf{m}\odot\bar{\mathbf{m}}$

20: $V_{t}\leftarrow V_{t}\setminus\bar{V}_{t}$

21: else

22: if $\lfloor N_{t}r_{t}\rceil>1$ then

23: $r_{t}\leftarrow r_{t}/2$

24: else

25: $T_{\mathrm{pool}}\leftarrow T_{\mathrm{pool}}\setminus\\{t\\}$

26: end if

27: end if

28: end while

29:
$G_{\mathrm{p}},\mathbf{P},\mathbf{w}\leftarrow\mathrm{Prune}\>\\!(G_{\mathrm{p}},\mathbf{P},\mathbf{w},\mathbf{m})$

30:
$\mathbf{P},\mathbf{w}\leftarrow\mathrm{Train}\>\\!(G_{\mathrm{p}},\mathbf{P},\mathbf{w},\mathbf{S},y)$

31:end for

32:return $G_{\mathrm{p}},\mathbf{P},\mathbf{w}$

Figure 8:  Loss increases from the mixing console and pruning ratios for
different pruning methods and tolerances.

##  Appendix C Supplementary Results

  * •

Table 5 and 6 report the per-dataset results on the mixing consoles and graph
pruning, respectively.

  * •

Figure 8 compares the pruning methods on $2$ random-sampled songs using $7$
tolerance settings from $0.001$ to $0.2$.

  * •

Figure 9 shows multiple graphs obtained by pruning the same console (song)
repeatedly.

  * •

Refer to Figure 10-12 for more pruned graphs obtained with the default setting
— hybrid method and $\tau=0.01$.

  * •

Figure 14-16 show more spectrogram plots.

|  |
---|---|---
|  |
(a)  (b)  (c)  (d)  (e)  (f)

Figure 9:  Each pruning run (default setting) yields a slightly different
graph. Song: EthanHein_GirlOnABridge.

(a) Internal_part1_65536 (b) DonCamilloChoir_TrudeTheBumblebee

Figure 10: Example pruned graphs (default setting). the number of tracks:
$K\leq 20$.

(a) LittleTybee_TheAlchemist (b) RaftMonk_Tiring

Figure 11: Example pruned graphs (default setting). the number of tracks:
$K>20$.

(a) Internal_part2_67692 (b) StevenClark_Bounty

Figure 12: Example pruned graphs (default setting). the number of tracks:
$K>20$ (continued).

(a) Torres_NewSkin (b) TablaBreakbeatScience_RockSteady

Figure 13: Matching of target music mixes with mixing consoles and their
pruned versions: MedleyDB dataset.

(a) MusicDelta_SwingJazz (b) ChrisJacoby_BoothShotLincoln

Figure 14: Matching of target music mixes with mixing consoles and their
pruned versions: MedleyDB dataset.

(a) HowlProject_IfIWereABell (b) IanDearden_TeraniaCreekWalking

Figure 15: Matching of target music mixes with mixing consoles and their
pruned versions: MixingSecrets dataset.

(a) Internal_66680 (b) Internal_67954

Figure 16: Matching of target music mixes with mixing consoles and their
pruned versions: Internal dataset.