Changhun Kim1 Joonhyung Park1 Hajin Shim1 Eunho Yang1,2

# SGEM: Test-Time Adaptation for Automatic Speech Recognition via Sequential-
Level Generalized Entropy Minimization

###### Abstract

Automatic speech recognition (ASR) models are frequently exposed to data
distribution shifts in many real-world scenarios, leading to erroneous
predictions. To tackle this issue, an existing test-time adaptation (TTA)
method has recently been proposed to adapt the pre-trained ASR model on
unlabeled test instances without source data. Despite decent performance gain,
this work relies solely on naive greedy decoding and performs adaptation
across timesteps at a frame level, which may not be optimal given the
sequential nature of the model output. Motivated by this, we propose a novel
TTA framework, dubbed SGEM, for general ASR models. To treat the sequential
output, SGEM first exploits beam search to explore candidate output logits and
selects the most plausible one. Then, it utilizes generalized entropy
minimization and negative sampling as unsupervised objectives to adapt the
model. SGEM achieves state-of-the-art performance for three mainstream ASR
models under various domain shifts.111Our code is available at
https://github.com/drumpt/SGEM.

Index Terms: automatic speech recognition, test-time adaptation, beam search,
entropy minimization, negative sampling

##  1 Introduction

While deep neural networks have achieved remarkable progress in a broad range
of areas, such as computer vision [1, 2], natural language processing [3, 4],
and speech processing [5, 6], these models are known to be susceptible to data
distribution shifts [7, 8]. This so-called domain shift problem readily occurs
in recent automatic speech recognition (ASR) models, which imposes challenges
in deploying ASR models to real-world applications. For example, utterances of
unseen speakers/words not exposed during training or utterances with
accidental background noise can be given at test time.

To tackle this domain shift problem for ASR models, many prior works have been
suggested, including data augmentation [9], feature alignment [10], domain
adversarial learning [11, 12], and knowledge distillation [13]. These works
mainly mitigate the domain shift problem under the unsupervised domain
adaptation (UDA) setting, where the source models are adapted to unlabeled
target domains. However, the UDA setting has several impractical assumptions
in real-world scenarios. First, it assumes that the source data is accessible,
which might be unavailable due to privacy/storage issues. Second, a pile of
target data has to be collected in advance. This is also unrealistic as it
requires substantial resources. Even worse, it restricts the generalization
capacity of the model only to the pre-collected target data, although the
target distribution can change arbitrarily at test time. Meanwhile, several
speaker adaptation methods [14, 15, 16] have demonstrated satisfactory
adaptation performance on variation in speakers during inference. However,
their restricted focus on speaker changes and reliance on prior knowledge of
test-time speakers impose limitations in effectively addressing arbitrary
domain shifts during test time.

Inspired by these limitations, a method called SUTA [17] was first proposed to
address an arbitrary domain shift problem under a more realistic setting -
test-time adaptation (TTA) - for ASR models as in other domains [18, 19, 20,
21, 22, 23, 24, 25, 26, 27]. Given an off-the-shelf ASR model pre-trained on
the source domain, TTA methods aim to adapt the model on-the-fly using
unlabeled instances from the target domain in test time without access to
source data. Inheriting the ideas of the TTA approaches in the computer vision
domain [19, 21, 22], SUTA shows decent performance in a single-utterance TTA
setting for the CTC-based ASR model [6].

However, directly adopting this approach to advanced ASR models [28, 29] could
not be optimal as SUTA was developed with the CTC-based models [30] in mind.
This is because, unlike the CTC-based models, which generate each output token
independently in a greedy manner, advanced ASR models are designed to work in
an autoregressive manner or typically utilize beam search decoding with an
external language model during the test phase. This indicates that output
logits acquired by greedy decoding may not adequately capture the output
distribution, and naively adapting with these logits at a frame level as in
[17] can cause undesirable behavior at the sequential level for general ASR
models.

In this paper, we propose a Sequential-Level Generalized Entropy Minimization
(SGEM) framework towards an effective TTA for general ASR models. To this end,
SGEM first explores candidate output logits and selects the most plausible one
using beam search to leverage the sequential nature of the output. Then, SGEM
leverages generalized entropy minimization loss and negative sampling loss as
auxiliary unsupervised objectives to adapt the model at a sequential level. We
validate SGEM for three representative ASR models on various datasets with
different distribution shifts, including unseen speakers/words and severe
background noise, and demonstrate that SGEM achieves state-of-the-art
performance in most settings. To the best of our knowledge, this is the first
work suggesting the TTA method for general ASR models.

Figure 1: The overall pipeline of the SGEM framework. Given a single utterance
$x$ in the target domain, we first acquire logits based on the most plausible
beam search output. Then, we utilize generalized entropy minimization and
negative sampling objectives with $x$ itself to adapt the ASR model
$f(\cdot|\theta)$ pre-trained on the source domain.

##  2 Proposed Method: SGEM

This section introduces SGEM, an effective TTA framework for general ASR
models. To this end, we describe the test-time adaptation setup for ASR models
in Section 2.1 and illustrate the core strategies of SGEM from Section 2.2 to
Section 2.5. Figure 1 depicts the overall pipeline of the proposed method.

###  2.1 Test-Time Adaptation Setup for ASR Models

We briefly explain general ASR models before formulating test-time adaptation
(TTA). Let $f(\cdot|{\theta})$ be an ASR model trained on the labeled source
domain $\mathcal{D}_{s}={\\{(x_{i}^{s},y_{i}^{s})\\}}_{i}$ in pairs of speech
and text, which takes a raw waveform $x$ and returns logits
$f(x|\theta)\in\mathbb{R}^{L\times C}$ for each timestep. Here, $L$ is the
number of timesteps, $C$ is the number of vocabulary classes, and $\theta$ is
the model parameter. It models the log joint probability $\log
p({\bm{y}}|x,\theta)$ of a candidate transcript
${\bm{y}}=(y_{1},\cdots,y_{L})$, in an autoregressive manner as follows:

$\displaystyle\log p({\bm{y}}|x,\theta):=\log
p_{\text{AM}}({\bm{y}}|x,\theta)+\lambda_{\text{LM}}\log
p_{\text{LM}}({\bm{y}})+Z$ $\displaystyle=\sum_{i=1}^{L}\log
p_{\text{AM}}(y_{i}|y_{<i},x,\theta)+\lambda_{\text{LM}}\log
p_{\text{LM}}(y_{i}|y_{<i})+Z,$

where $y_{i}\in\\{1,\cdots,C\\}$, $p_{\text{AM}}({\bm{y}}|x,\theta)$ is the
joint probability given by the model output $f(x|\theta)$,
$p_{\text{LM}}({\bm{y}})$ is the joint probability of an autoregressive
language model (LM), $\lambda_{\text{LM}}$ is a hyperparameter to control the
effect of the LM, and $Z$ is a normalizing constant. LM aims to boost the ASR
model to generate more faithful sentences. ASR decoding strategies approximate
the optimal solution ${\bm{y}}^{*}=\operatorname*{\arg\max}_{{\bm{y}}}\log
p({\bm{y}}|x,\theta)$.

TTA methods for an ASR model $f(\cdot|\theta)$ aim to adapt the model to the
unlabeled target speech domain $\mathcal{D}_{t}={\\{x_{i}^{t}\\}}_{i}$ without
access to $\mathcal{D}_{s}$. Specifically, we consider a single-utterance TTA
setting where we fine-tune the ASR model $f(\cdot|\theta)$ for each utterance
$x_{i}^{t}\in\mathcal{D}_{t}$ to get more precise output logits ${\log
p({\bm{y}}|x_{i}^{t},\theta)}$ with unsupervised objectives using only
$x_{i}^{t}$ itself. This single-utterance TTA setting is considerably
pragmatic regarding low latency without presuming that the test instances are
independent and identically distributed [17, 24, 27].

Table 1: Comparison of TTA performance, measured as word error rate (%), for three mainstream ASR models across 12 datasets with various types of domain shifts. The results are obtained using greedy decoding for inference. | Dataset | CH | TD | CV | VA | AC | AA | BA | CM | MU | NB | SD | TP | Avg.
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
CTC-based model | Unadapted | 31.2 | 13.2 | 36.9 | 14.5 | 28.1 | 40.9 | 66.6 | 49.8 | 50.4 | 119.2 | 19.2 | 26.2 | 41.4
SUTA | 25.0 | 12.0 | 31.4 | 11.8 | 17.7 | 31.3 | 55.2 | 39.4 | 39.7 | 113.0 | 15.0 | 17.8 | 34.1
SGEM | 24.7 | 12.0 | 31.1 | 11.6 | 17.3 | 30.7 | 53.1 | 38.5 | 38.6 | 110.5 | 14.8 | 17.5 | 33.4
Conformer | Unadapted | 28.7 | 15.1 | 36.8 | 17.4 | 18.8 | 44.8 | 74.3 | 45.7 | 56.0 | 122.1 | 20.8 | 36.9 | 43.1
SUTA | 25.2 | 13.4 | 32.4 | 14.7 | 14.5 | 39.8 | 73.3 | 38.4 | 48.7 | 125.5 | 16.4 | 28.8 | 39.3
SGEM | 24.5 | 13.3 | 31.6 | 14.6 | 14.4 | 38.5 | 70.4 | 38.7 | 48.5 | 120.9 | 16.8 | 28.9 | 38.4
Transducer | Unadapted | 11.8 | 7.2 | 12.9 | 6.5 | 14.1 | 20.4 | 31.0 | 29.7 | 31.3 | 74.6 | 12.7 | 16.2 | 22.4
SUTA | 10.3 | 6.8 | 12.1 | 5.5 | 12.0 | 18.5 | 28.3 | 26.7 | 28.7 | 74.6 | 11.7 | 14.7 | 20.8
SGEM | 9.9 | 6.6 | 12.0 | 5.2 | 11.6 | 18.0 | 27.5 | 26.0 | 28.0 | 76.5 | 11.5 | 14.3 | 20.6
|  |  |  |  |  |  |  |  |  |  |  |  |  |

###  2.2 Beam Search-Based Logit Acquisition

An existing TTA method for ASR models [17] exploits the greedy decoding
strategy without leveraging the external LM (i.e., $\lambda_{\text{LM}}=0$) to
get output logits for all timesteps. Also, it utilizes them with unsupervised
objectives such as entropy minimization [31] and minimum class confusion [32]
as if they are appropriate logits to adapt the ASR model. However, naively
using greedy decoding is proven defective [33] and can mislead the model to be
adapted on the wrong labels. Furthermore, this frame-level greedy adaptation
might be sub-optimal on the sequential output since it only considers the
joint probability of a sequence myopically over timesteps.

To this end, we exploit a novel logit acquisition strategy based on more
systematic beam search decoding. Given a beam width $B$, we find the most
plausible output sequence $\hat{{\bm{y}}}=(\hat{y}_{1},\cdots,\hat{y}_{L})$,
which approximates ${\bm{y}}^{*}$ using beam search [33]. Note that we do not
hold the logits of beam candidates in this step to reduce memory consumption.
Instead, the estimated sequence $\hat{{\bm{y}}}$ is passed to the model again
to acquire the $i$-th logit
$\bm{o}_{i}=(o_{i1},\cdots,o_{iC})\in\mathbbm{R}^{C}$ for all
$i\in\\{1,\cdots,L\\}$ where $o_{ij}=\log p(y_{i}=j|\hat{y}_{<i},x,\theta)$.
Our intuition behind the beam search-based logit acquisition is that
considering the logits obtained from beam search for adaptation naturally
aligned with how ASR models decode sentences, i.e., our approach can tailor
the model to adapt toward the actual yet accurate sentences generated by the
ASR models.

###  2.3 Generalized Entropy Minimization

While entropy minimization achieves decent performance on domain adaptation
tasks [19, 21, 22, 34] by reducing the uncertainty of predictions, and
extracting domain-invariant features on the target domain, we can further
improve this objective by adopting its generalized version, Rényi entropy, and
searching for more effective hyperparameter. For a discrete random variable
$X$, which takes values in $\\{1,\cdots,C\\}$, Rényi entropy
$\text{H}_{\alpha}(X)$ of order $\alpha$ with $\alpha\in(0,1)\cup(1,\infty)$
is defined as
$\text{H}_{\alpha}(X)=\frac{1}{1-\alpha}{\log\big{(}\sum_{j=1}^{C}\mathbb{P}(X=j)^{\alpha}\big{)}}$.
When $\alpha\rightarrow 1$ and $\alpha\rightarrow\infty$,
$\text{H}_{\alpha}(X)$ becomes Shannon entropy and cross-entropy with a
pseudo-label $\operatorname*{\arg\max}_{j}\mathbb{P}(X=j)$, respectively. For
a single-utterance TTA setting, we hypothesize that there exists an optimal
$\alpha\in(1,\infty)$ and define the generalized entropy loss as follows:

$\mathcal{L}_{\text{GEM}}=\frac{1}{L}\sum_{i=1}^{L}\frac{1}{1-\alpha}\log\Big{(}\sum_{j=1}^{C}p_{ij}^{\alpha}\Big{)},$
(1)

where
$p_{ij}=\frac{\exp(o_{ij}/T)}{{\sum_{j^{\prime}=1}^{C}{\exp(o_{ij^{\prime}}/T)}}}$
and $T$ is a temperature hyperparameter for preventing vanishing gradient. As
the blank token dominates all timesteps, we ignore timesteps with the highest
probability of blank token among all classes to alleviate the class imbalance
problem as in [17].

###  2.4 Negative Sampling

We further exploit negative sampling loss, originally adopted for semi-
supervised learning (SSL) in [35]. Negative sampling loss penalizes the
probabilities of low-confident classes, and Chen et al. [35] have shown that
adding it can further boost the performance of existing SSL algorithms. It can
be derived from the standard cross-entropy loss as follows. Given $L$ labeled
samples ${\\{(x_{i},y_{i})\\}}_{i=1}^{L}$, the standard cross-entropy loss is
defined as
$\mathcal{L}_{\text{CE}}=-\frac{1}{L}\sum_{i=1}^{L}\sum_{j=1}^{C}\mathbbm{1}_{[j=y_{i}]}\log
p_{ij}$. Note that $\sum_{j=1}^{C}\mathbbm{1}_{[j=y_{i}]}\log
p_{ij}=\log(1-\sum_{j\neq\text{$y_{i}$}}p_{ij})$. Since we do not know the
ground truth label $y_{i}$ for each $x_{i}$ in the unlabeled target domain, we
approximate $\mathcal{L}_{\text{CE}}$ with the negative sampling loss
$\mathcal{L}_{\text{NS}}$ defined as follows:

$\mathcal{L}_{\text{NS}}=-\frac{1}{L}\sum_{i=1}^{L}\log\Big{(}1-\sum_{j=1}^{C}\mathbbm{1}_{[p^{\prime}_{ij}<\tau]}p_{ij}\Big{)},$
(2)

where
$p_{ij}=\frac{\exp(o_{ij}/T)}{{\sum_{j^{\prime}=1}^{C}{\exp(o_{ij^{\prime}}/T)}}}$,
$p^{\prime}_{ij}=\frac{\exp(o_{ij})}{{\sum_{j^{\prime}=1}^{C}{\exp(o_{ij^{\prime}})}}}$
with a temperature hyperparameter $T$ for avoiding vanishing gradient, and
$\mathbbm{1}$ is an indicator function. $j$-th class of $x_{i}$ is considered
as a negative class when the probability $p^{\prime}_{ij}$ is less than a
threshold $\tau$. Without modification, Equation (2) can be interpreted in a
single-utterance TTA setting as penalizing probabilities of negative classes
at every timestep for a sequential output of length $L$.

###  2.5 Overall Framework

Our final unsupervised objective is the weighted sum of the generalized
entropy loss and the negative sampling loss as follows:

$\mathcal{L}=\mathcal{L}_{\text{GEM}}+\lambda_{\text{NS}}\mathcal{L}_{\text{NS}},$
(3)

where $\lambda_{\text{NS}}$ is negative sampling weight for balancing two
losses. For each utterance, we adapt the model for $N$ iterations in an
episodic manner where we newly reset the model to the pre-trained one to
preserve the knowledge from the source domain.

Table 2: Comparison of TTA performance, measured as word error rate (%), for the CTC-based model across 12 datasets with various domain shifts. The results are obtained using beam search decoding with an external language model for inference. Dataset | CH | TD | CV | VA | AC | AA | BA | CM | MU | NB | SD | TP | Avg.
---|---|---|---|---|---|---|---|---|---|---|---|---|---
Unadapted | 29.5 | 12.2 | 36.9 | 13.0 | 26.1 | 38.6 | 58.9 | 48.9 | 49.0 | 91.6 | 17.4 | 23.7 | 37.2
SUTA | 24.1 | 11.6 | 31.5 | 11.4 | 16.8 | 30.3 | 53.2 | 38.1 | 38.6 | 107.9 | 14.1 | 16.9 | 32.9
SGEM | 24.1 | 11.7 | 31.1 | 11.1 | 16.5 | 29.8 | 51.6 | 37.7 | 37.7 | 106.7 | 14.0 | 16.9 | 32.4
|  |  |  |  |  |  |  |  |  |  |  |  |

##  3 Experiments

###  3.1 Experimental Setup

Source ASR Models To verify the efficacy of SGEM, we evaluate it on three
mainstream ASR architectures: the CTC-based model [30], Conformer [36], and
Transducer [37]. More specifically, for the CTC-based model, we use wav2vec
2.0 [6] 222{https://huggingface.co/facebook/wav2vec2-base-960h} trained on the
LibriSpeech dataset [38]. For Conformer, we exploit Conformer-CTC [36]
333https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/stt_en_conformer_ctc_small_ls
trained on the LibriSpeech dataset. For Transducer, we adopt Conformer-
Transducer [28]
444https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/stt_en_conformer_transducer_small
trained on a composite NeMO ASRSET dataset, including the LibriSpeech dataset.
We utilize the external 4-gram language model
555https://huggingface.co/patrickvonplaten/wav2vec2-base-100h-with-lm for the
CTC-based model and Conformer.

Datasets We assess the performance of SGEM on multiple datasets under various
domain shift settings. To test SGEM under unseen speakers/words, we use the
test set of four datasets: CHiME-3 (CH) [39], TED-LIUM 2 (TD) [40], Common
Voice (CV) [41], and Valentini (VA) [42]. In addition, we validate SGEM under
accident background noise by injecting the following eight types of noises to
each utterance of in-domain LibriSpeech test-other dataset [38]: air
conditioner (AC), airport announcement (AA), babble (BA), copy machine (CM),
munching (MU), neighbors (NB), shutting door (SD), and typing (TP) with
$\text{SNR}=10$dB. For each type of noise, we randomly select one noise sample
from the MS-SNSD noise test set [43]. We also evaluate SGEM on L2-Arctic [44],
non-native English speech corpora, to verify SGEM under extreme
pronunciation/accent shifts. Specifically, we randomly select one speaker for
each first language.

Implementation Details Since the TTA setting has no validation set, we
optimize hyperparameters on the CH dataset for each model and apply them to
the other datasets. The best settings are as follows. For all models, we use
AdamW optimizer [45] and cosine annealing learning rate scheduler with
$\eta_{i}$ and $\eta_{f}$ for initial and final learning rates, respectively,
and set $(N,T,\tau)=(10,2.5,0.4/C)$ with vocabulary size $C$. We only train
feature extractors for the CTC-based model and encoders for the others.
Furthermore, we set
$(\eta_{i},\eta_{f},B,\lambda_{\text{LM}},\alpha,\lambda_{\text{NS}})=(4{\mspace{1.0mu}\cdot\mspace{1.0mu}}10^{-5},2{\mspace{1.0mu}\cdot\mspace{1.0mu}}10^{-5},5,0.3,1.5,1)$
for the CTC-based model,
$(4{\mspace{1.0mu}\cdot\mspace{1.0mu}}10^{-5},2{\mspace{1.0mu}\cdot\mspace{1.0mu}}10^{-5},5,0.3,1.25,2)$
for Conformer, and
$(4{\mspace{1.0mu}\cdot\mspace{1.0mu}}10^{-6},2{\mspace{1.0mu}\cdot\mspace{1.0mu}}10^{-6},3,0,1.25,0.5)$
for Transducer. All experiments are conducted on Nvidia TITAN Xp and GeForce
RTX 3090. Adaptation takes about 0.771 seconds for a 1-second utterance
averaged over three models.

###  3.2 Main Results

We compare the TTA performance of three mainstream ASR models, including the
CTC-based model, Conformer, and Transducer, across 12 datasets with various
domain shifts. Table 1 presents the word error rate (WER) of ASR model outputs
generated by the greedy search decoding method, following the evaluation
protocol used in the previous study [17]. Additionally, Table 2 showcases the
TTA performance for the CTC-based models using beam search decoding with
external LM. For both decoding methods, the ASR models with SGEM consistently
enhance the recognition accuracy of target utterances with an average word
error rate reduction of 15.6%, except for two cases on NB, where the
performance without adaptation is the best when using beam search decoding.
Furthermore, SGEM outperforms SUTA in terms of average WER across all 12
datasets for each of the three model architectures (CTC-based model: (greedy)
$34.1\%\rightarrow\textbf{33.4\%}$, (beam search)
$32.9\%\rightarrow\textbf{32.4\%}$ / Conformer:
$39.3\%\rightarrow\textbf{38.4\%}$ / Transducer:
$20.8\%\rightarrow\textbf{20.6\%}$). This indicates the superiority of our
unsupervised objectives as well as the logit acquisition method for adapting
sequential language outputs regardless of the decoding strategy.

###  3.3 Non-Native English Speech Corpora

To show the usability of SGEM at various domain shifts, we further analyze
SGEM on six different non-native English speech corpora. The result is
summarized in Table 3. As shown in Table 3, SGEM achieves the best results for
all corpora, outperforming the baseline. This implies the adaptability of SGEM
under extreme pronunciation/accent shifts, demonstrating its versatility in
practical situations with severe speaker shifts, such as globally used online
ASR systems.

Table 3: Comparison of TTA performance, measured as word error rate (%), for the CTC-based model on six non-native English speech corpora. Setting | Unadapted | SUTA | SGEM
---|---|---|---
Arabic | 32.5 | 27.1 | 26.5
Mandarin | 28.5 | 23.3 | 23.1
Hindi | 15.7 | 12.5 | 12.3
Korean | 23.3 | 19.7 | 19.5
Spanish | 35.7 | 29.8 | 29.3
Vietnamese | 18.5 | 15.7 | 15.4
Average | 25.7 | 21.4 | 21.0
|  |  |

###  3.4 Data Deficient Condition

It is commonly known that TTA methods fail under the data deficient condition
where the number of test instances is limited [27, 46]. This still holds in
the single-utterance TTA setting for ASR models, where the length of utterance
is short, so the number of output tokens is insufficient. To validate SGEM on
this harsh condition, we split the CH dataset according to utterance length
and evaluate SGEM with the CTC-based model on each split. As shown in Figure
2, SGEM performs best in every length interval. In addition, it is worth
noting that SGEM significantly outperforms the baseline for extremely short
utterances of less than 2 seconds, showing the superiority of our method in
real-world scenarios where short utterances are prevalent and negligible
latency is required.

Figure 2: Comparison of TTA performance, measured as relative word error rate
reduction (%), on different utterance lengths of CH dataset for the CTC-based
model.

###  3.5 Ablation Study

To validate the core components of SGEM: beam search-based logit acquisition
(BS, Section 2.2), generalized entropy minimization (GEM, Section 2.3), and
negative sampling (NS, Section 2.4), we conduct an ablation study for three
mainstream ASR models on CH dataset. As shown in Table 4, both generalized
entropy minimization and negative sampling alone achieve remarkable
performance gains for every model, indicating the efficacy of each component.
Meanwhile, substituting greedy search for beam search even with small beam
width (for all models) and without external LM (for Transducer) consistently
boosts the performance in all cases, showing the effectiveness of beam search-
based logit acquisition and implying that additional performance improvement
can be expected using larger beam sizes or language model if resources are
allowed.

Table 4: Ablation study of SGEM. BS | GEM | NS | CTC | Conformer | Transducer
---|---|---|---|---|---
✗ | ✗ | ✗ | 31.2 | 28.7 | 11.8
✗ | ✓ | ✗ | 24.9 | 24.7 | 10.0
✗ | ✗ | ✓ | 25.2 | 25.0 | 10.1
✗ | ✓ | ✓ | 24.8 | 24.7 | 10.0
✓ | ✗ | ✓ | 24.8 | 24.7 | 10.1
✓ | ✓ | ✓ | 24.7 | 24.5 | 9.9
|  |  |  |  |

##  4 Conclusion

We have suggested SGEM, an effective single-utterance TTA framework for
general ASR models. SGEM exploits beam search-based logit acquisition and
utilizes generalized entropy minimization and negative sampling objectives to
adapt the model at a sequential level. SGEM has achieved state-of-the-art
performance for the three mainstream ASR models across 12 datasets with
various domain shifts, including utterances of unseen speakers/words during
training and utterances with severe background noise. We have also verified
SGEM under harsh conditions, such as non-native English utterances with severe
pronunciation/accent shifts and the data deficient condition, and the efficacy
of each component of SGEM through an ablation study. SGEM sheds light on the
careful design of speech-specific components when devising test-time
adaptation methods for ASR models.

##  5 Acknowledgement

This work was supported by National Research Foundation of Korea (NRF) grants
(No.2018R1A5A1059921, No.2019R1C1C1009192) and Institute of Information &
communications Technology Planning & Evaluation (IITP) grant funded by the
Korea government (MSIT) (No.2019-0-00075, Artificial Intelligence Graduate
School Program(KAIST)).

## References

  * [1] K. He, X. Zhang, S. Ren, and J. Sun, ``Deep residual learning for image recognition,'' in _CVPR_ , 2016.
  * [2] L. B. Alexey Dosovitskiy and A. K. et al., ``An image is worth 16x16 words: Transformers for image recognition at scale,'' in _ICLR_ , 2021.
  * [3] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, ``Attention is all you need,'' _NeurIPS_ , 2017.
  * [4] N. R. Tom B. Brown, Benjamin Mann and M. S. et al., ``Language models are few-shot learners,'' in _NeurIPS_ , 2020.
  * [5] J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen, Y. Zhang, Y. Wang, R. Skerrv-Ryan _et al._ , ``Natural tts synthesis by conditioning wavenet on mel spectrogram predictions,'' in _ICASSP_ , 2018\.
  * [6] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, ``wav2vec 2.0: A framework for self-supervised learning of speech representations,'' in _NeurIPS_ , 2020\.
  * [7] E. Tzeng, J. Hoffman, K. Saenko, and T. Darrell, ``Adversarial discriminative domain adaptation,'' in _CVPR_ , 2017.
  * [8] Y. Ganin and V. Lempitsky, ``Unsupervised domain adaptation by backpropagation,'' in _ICML_ , 2015.
  * [9] W.-N. Hsu, Y. Zhang, and J. Glass, ``Unsupervised domain adaptation for robust speech recognition via variational autoencoder-based data augmentation,'' in _ASRU_ , 2017.
  * [10] W. Hou, J. Wang, X. Tan, T. Qin, and T. Shinozaki, ``Cross-domain speech recognition with unsupervised character-level distribution matching,'' in _INTERSPEECH_ , 2021.
  * [11] S. Sun, B. Zhang, L. Xie, and Y. Zhang, ``An unsupervised deep domain adaptation approach for robust speech recognition,'' in _Neurocomputing_ , 2017.
  * [12] S. Sun, C.-F. Yeh, M.-Y. Hwang, M. Ostendorf, and L. Xie, ``Domain adversarial training for accented speech recognition,'' in _ICASSP_ , 2018.
  * [13] V. Manohar, P. Ghahremani, D. Povey, and S. Khudanpur, ``A teacher-student learning approach for unsupervised domain adaptation of sequence-trained asr models,'' in _SLT_ , 2018.
  * [14] F. Seide, G. Li, X. Chen, and D. Yu, ``Feature engineering in context-dependent deep neural networks for conversational speech transcription,'' in _ASRU_ , 2011.
  * [15] K. Yao, D. Yu, F. Seide, H. Su, L. Deng, and Y. Gong, ``Adaptation of context-dependent deep neural networks for automatic speech recognition,'' in _SLT_ , 2012.
  * [16] D. Yu, K. Yao, H. Su, G. Li, and F. Seide, ``Kl-divergence regularized deep neural network adaptation for improved large vocabulary speech recognition,'' in _ICASSP_ , 2013.
  * [17] G.-T. Lin, S.-W. Li, and H.-y. Lee, ``Listen, adapt, better wer: Source-free single-utterance test-time adaptation for automatic speech recognition,'' in _INTERSPEECH_ , 2022.
  * [18] Y. Sun, X. Wang, Z. Liu, J. Miller, A. Efros, and M. Hardt, ``Test-time training with self-supervision for generalization under distribution shifts,'' in _ICML_ , 2020.
  * [19] D. Wang, E. Shelhamer, S. Liu, B. Olshausen, and T. Darrell, ``TENT: Fully test-time adaptation by entropy minimization,'' in _ICLR_ , 2021.
  * [20] Y. Liu, P. Kothari, B. van Delft, B. Bellot-Gurlet, T. Mordan, and A. Alahi, ``TTT++: When does self-supervised test-time training fail or thrive?'' in _NeurIPS_ , 2021.
  * [21] F. Fleuret _et al._ , ``Test time adaptation through perturbation robustness,'' in _NeurIPS DistShift Workshop_ , 2021.
  * [22] M. Zhang, S. Levine, and C. Finn, ``MEMO: Test time robustness via adaptation and augmentation,'' in _NeurIPS_ , 2022.
  * [23] D. Chen, D. Wang, T. Darrell, and S. Ebrahimi, ``Contrastive test-time adaptation,'' in _CVPR_ , 2022.
  * [24] Q. Wang, O. Fink, L. Van Gool, and D. Dai, ``Continual test-time domain adaptation,'' in _CVPR_ , 2022.
  * [25] A. Bartler, A. Bühler, F. Wiewel, M. Döbler, and B. Yang, ``MT3: Meta test-time training for self-supervised test-time adaption,'' in _AISTATS_ , 2022.
  * [26] S. Goyal, M. Sun, A. Raghunathan, and Z. Kolter, ``Test-time adaptation via conjugate pseudo-labels,'' in _NeurIPS_ , 2022.
  * [27] S. Niu, J. Wu, Y. Zhang, Z. Wen, Y. Chen, P. Zhao, and M. Tan, ``Towards stable test-time adaptation in dynamic wild world,'' in _ICLR_ , 2023.
  * [28] M. Burchi and V. Vielzeuf, ``Efficient conformer: Progressive downsampling and grouped attention for automatic speech recognition,'' in _ASRU_ , 2021.
  * [29] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, ``Robust speech recognition via large-scale weak supervision,'' in _ICML_ , 2022.
  * [30] A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, ``Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,'' in _ICML_ , 2006.
  * [31] Y. Grandvalet and Y. Bengio, ``Semi-supervised learning by entropy minimization,'' in _NIPS_ , 2004.
  * [32] Y. Jin, X. Wang, M. Long, and J. Wang, ``Minimum class confusion for versatile domain adaptation,'' in _ECCV_ , 2020.
  * [33] M. Freitag and Y. Al-Onaizan, ``Beam search strategies for neural machine translation,'' in _ACL_ , 2017.
  * [34] S. Khurana, N. Moritz, T. Hori, and J. Le Roux, ``Unsupervised domain adaptation for speech recognition via uncertainty driven self-training,'' in _ICASSP_ , 2021.
  * [35] J. Chen, V. Shah, and A. Kyrillidis, ``Negative sampling in semi-supervised learning,'' in _ICML_ , 2020.
  * [36] A. Gulati and Q. et al., ``Conformer: Convolution-augmented transformer for speech recognition,'' in _INTERSPEECH_ , 2020.
  * [37] A. Graves, ``Sequence transduction with recurrent neural networks,'' in _ICML_ , 2012.
  * [38] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, ``Librispeech: an asr corpus based on public domain audio books,'' in _ICASSP_ , 2015.
  * [39] J. Barker, R. Marxer, E. Vincent, and S. Watanabe, ``The third ‘chime’speech separation and recognition challenge: Dataset, task and baselines,'' in _ASRU_ , 2015.
  * [40] A. Rousseau, P. Deléglise, Y. Esteve _et al._ , ``Enhancing the ted-lium corpus with selected data for language modeling and more ted talks.'' in _LREC_ , 2014.
  * [41] R. Ardila, M. Branson, K. Davis, M. Henretty, M. Kohler, J. Meyer, R. Morais, L. Saunders, F. M. Tyers, and G. Weber, ``Common voice: A massively-multilingual speech corpus,'' in _LREC_ , 2020.
  * [42] C. Valentini-Botinhao _et al._ , ``Noisy speech database for training speech enhancement algorithms and tts models,'' _University of Edinburgh. School of Informatics. Centre for Speech Technology Research (CSTR)_ , 2017.
  * [43] C. K. Reddy, E. Beyrami, J. Pool, R. Cutler, S. Srinivasan, and J. Gehrke, ``A scalable noisy speech dataset and online subjective test framework,'' in _INTERSPEECH_ , 2019.
  * [44] G. Zhao, S. Sonsaat, A. Silpachai, I. Lucic, E. Chukharev-Hudilainen, J. Levis, and R. Gutierrez-Osuna, ``L2-arctic: A non-native english speech corpus,'' in _INTERSPEECH_ , 2018.
  * [45] I. Loshchilov and F. Hutter, ``Decoupled weight decay regularization,'' in _ICLR_ , 2017.
  * [46] J. Gao, J. Zhang, X. Liu, T. Darrell, E. Shelhamer, and D. Wang, ``Back to the source: Diffusion-driven test-time adaptation,'' in _ICML DyNN Workshop_ , 2022.


#  GSI Accelerator Physics Department

Impact of Field Errors of Quadrupole Units on Beam Losses at SIS100 Injection

Adrian<EMAIL_ADDRESS>
GSI Helmholtzzentrum für Schwerionenforschung GmbH, Darmstadt, Germany
22 September 2021
GSI Report: GSI-2021-00450

###### Abstract

_Abstract:_ This report discusses heavy-ion beam losses due to the nominal
transverse space charge and the field imperfections in SIS100 during
accumulation based on recent cold measurements of the first 10 series
quadrupole units. They are compared to the (stamped) First-of-Series
quadrupole unit #2.

##  1  Overview

The field quality of the steering and focusing magnet units in the SIS100 have
a strong impact on the maximally achievable intensities. As the SIS100
accumulates eight bunches over four injections from SIS18 during heavy-ion
beam accumulation, the first injected bunches circulate at the rigidity of
$18\text{\,}\mathrm{T}\text{\,}\mathrm{m}$ during a storage time of
$1\text{\,}\mathrm{s}$ before the ramp starts. For the reference heavy-ion
beam type Uranium-238 (charge state $28+$) this duration translates to 160’000
turns. During this time the bunches can be subject to emittance growth and
subsequent beam loss due to resonances driven by magnetic field imperfections.
Due to an effective space charge tune spread of up to $\Delta Q^{SC}_{y}=-0.3$
the resonances cover larger areas in the tune diagram.

##  2  Magnet Field Error Model

For this study we focused on the SIS100 heavy-ion fast extraction optics
working point, located in the tune quadrant $18.5<Q_{x,y}<19$. To obtain tune
figures in this quadrant, the $L\approx$1.3\text{\,}\mathrm{m}$$ long
superferric quadrupole magnets need to be powered at an integral strength of
around $K_{1}L=$0.28\text{\,}{\mathrm{m}}^{-1}$$. The thus required focusing
strength $K_{1}$ then determines the quadrupole gradient $\partial
B_{y}/\partial x$ at a given beam rigidity $B\rho$ via

$K_{1}=\frac{1}{B\rho}\,\frac{\partial B_{y}}{\partial x}\quad.$ (2.1)

At SIS100 injection with $B\rho=$18\text{\,}\mathrm{T}\text{\,}\mathrm{m}$$
the required quadrupole gradient hence amounts to

$\frac{\partial B_{y}}{\partial
x}=$18\text{\,}\mathrm{T}\text{\,}\mathrm{m}$\leavevmode\nobreak\
\frac{0.28}{$1.3\text{\,}\mathrm{m}^{2}$}=3.9\,\frac{$\mathrm{T}$}{$\mathrm{m}$}\quad.$
(2.2)

In the bench measurements, the most suitable setup reaching a comparable
gradient is at a current of $I=$900\text{\,}\mathrm{A}$$. The quadrupole field
error models in the following are based on the measured nonlinear multipole
components at this current. On the linear order, a normal gradient error of
rms amplitude
$b_{2}=$5\text{\,}\mathrm{u}\mathrm{n}\mathrm{i}\mathrm{t}\mathrm{s}$$
($1\text{\,}\mathrm{u}\mathrm{n}\mathrm{i}\mathrm{t}$ conventionally refers to
$1\text{\times}{10}^{-4}\text{\,}$) is assumed for all magnets [1] while the
skew component is neglected, $a_{2}=0$.

The field error model for the dipole magnet units is described in Ref. [2,
Section 2] and based on the bench measurements carried out by F. Kaether et
al.

###  2.1  First-of-Series Quadrupole Unit #2

The multipole measurements of the second First-of-Series (FoS) quadrupole unit
(the yoke of which has been stamped and not milled after assembly) [3] are
used to create a field error model for simulations henceforth referred to as
“FoS 2” model. The nonlinear part has been described in detail in Ref. [2,
Section 2]. Beam simulations demonstrate that the multipole errors beyond
dodecapolar order $b_{6}$ (with figures around or below
$0.5\text{\,}\mathrm{u}\mathrm{n}\mathrm{i}\mathrm{t}\mathrm{s}$) have no
effect on the losses with space charge during accumulation of heavy ions.

The black bars in Fig. 1(a) indicate the most relevant multipole components in
the FoS 2 model222Based on insights from magnet simulations, some figures from
the FoS 2 measurements have been considered lower than the expected average.
Thus, the following values have been slightly increased in the FoS 2 field
error model to remain on the conservative side, viz. $b_{3}=0.7$ (not $0.6$),
$b_{5}=1.0$ (not $0.9$), $a_{3}=1.2$ (not $0.5$) and $a_{6}=1.0$ (not $0.8$)..
The model contains no systematic components besides $b_{6}$, which is assumed
to amount to the measured
$6.9\text{\,}\mathrm{u}\mathrm{n}\mathrm{i}\mathrm{t}\mathrm{s}$. The
corresponding stochastic rms amplitude is taken as half the measured value,
$b_{6}=$3.45\text{\,}\mathrm{u}\mathrm{n}\mathrm{i}\mathrm{t}\mathrm{s}$$.

(a) First-of-Series #2 vs. ensemble (based on 10 series magnets), the
systematic component in $b_{6}$ is indicated by diamond markers. The assumed
$b_{2}$ component appears hatched.

(b) 10 series magnets, one bar marks the amplitude averaged along the magnet
length $L$ for the respective component. The grey area indicates an interval
of $1\text{\,}\mathrm{u}\mathrm{n}\mathrm{i}\mathrm{t}$ radius.

Figure 2.1: Rms amplitudes of multipole components for stochastic field error
models in simulations ($b_{2}$ is assumed to be
$5\text{\times}{10}^{-4}\text{\,}$, all others are based on measurements).

###  2.2  Ensemble of Series Quadrupole Units

Steady series production of the quadrupole units has commenced and cold bench
measurements of the first 10 units have provided a set of field
imperfections333The measurement protocols of the 10 cold quadrupole units bear
the CIDs 02000090630, 02000090647, 02000090654, 02000090661, 02000090678,
02000091330, 02000091354, 02000091361, 02000091378, 02000091385 in the
respectively tagged reports prepared by JINR Institute, Dubna.. Figure 1(b)
displays the measured multipole components, the bars each stand for one
quadrupole magnet (sorted by the respective report CID). In comparison to the
FoS 2 model from Fig. 1(a), the normal $b_{6}$ component has slightly grown to
values fluctuating around a mean of
$8.4\text{\,}\mathrm{u}\mathrm{n}\mathrm{i}\mathrm{t}\mathrm{s}$ while the
normal $b_{4}$ component has approximately halved and the skew $a_{4}$
component is strongly reduced for all 10 series quadrupole units. The grey
area marks an $\pm$$1\text{\,}\mathrm{u}\mathrm{n}\mathrm{i}\mathrm{t}$
interval to guide the eye.

The field error model for the beam dynamics simulations, which is based on
these 10 series unit measurements, is henceforth referred to as the _ensemble
model_. The stochastic multipole components determine the resonance driving
terms, they are estimated from the spread of individual magnets around the
systematic figure. The stochastic components of the ensemble model are
exhibited as red bars in Fig. 1(a). The average figures of all magnets for
each component give a hint on the corresponding systematic component. Except
for the aforementioned dodecapolar $\langle
b_{6}\rangle=$8.4\text{\,}\mathrm{u}\mathrm{n}\mathrm{i}\mathrm{t}\mathrm{s}$$,
all average figures lie below one unit. For a conservative estimate of the
resonance driving terms in the simulations, the systematic components apart
from $b_{6}$ are considered to be zero. The root mean square
$\sqrt{\sum_{i=1}^{10}[(b_{n})_{i}-\langle b_{n}\rangle]^{2}}$ then yields the
stochastic rms amplitude.

###  2.3  Comparison FoS #2 and Ensemble Models

Figure 1(a) depicts the stochastic rms amplitude figures of the two error
models. In comparison to the FoS 2 error model, the ensemble model features

  * •

a doubled normal sextupolar component ($\approx$ factor 1.8),

  * •

a halved normal octupolar component ($\approx$ factor 0.6),

  * •

strongly reduced normal dodecapolar and skew octupolar components.

##  3  Setup of Beam Dynamics Simulations

The full beam dynamics simulations of a single bunch in SIS100 with nonlinear
space charge, nonlinear thin-lens tracking and nonlinear RF bucket are carried
out using the open source CERN code SixTrackLib [4, 5, 6]. The full SIS100
lattice [7] is simulated based on the heavy-ion fast extraction tune settings,
where the working point is scanned in $0.01$ intervals across the tunes
$18.55\leq Q_{x,y}<19$. All beam and machine parameters are listed in Table
3.1.

Table 3.1: Parameters for Uranium-238 Beam Production in SIS100. Parameter | Value
---|---
Horizontal normalised rms emittance $\epsilon_{x}$ | $5.9\text{\,}\mathrm{m}\mathrm{m}\,\mathrm{m}\mathrm{r}\mathrm{a}\mathrm{d}$
Horizontal geometric KV emittance | $35\text{\,}\mathrm{m}\mathrm{m}\,\mathrm{m}\mathrm{r}\mathrm{a}\mathrm{d}$
Vertical normalised rms emittance $\epsilon_{y}$ | $2.5\text{\,}\mathrm{m}\mathrm{m}\,\mathrm{m}\mathrm{r}\mathrm{a}\mathrm{d}$
Vertical geometric KV emittance | $15\text{\,}\mathrm{m}\mathrm{m}\,\mathrm{m}\mathrm{r}\mathrm{a}\mathrm{d}$
Rms bunch length $\sigma_{z}$ | $13.2\text{\,}\mathrm{m}$
Rms momentum deviation $\sigma_{\Delta p/p_{0}}$ | $0.44\text{\times}{10}^{-3}\text{\,}$
Bunch intensity $N$ of U${}^{28+}_{238}$ | $0.625\text{\times}{10}^{11}\text{\,}$
Max. space charge tune shift $\Delta Q^{SC}_{y}$ | $-0.30$
Chromatic tune spread $Q^{\prime}_{x,y}\sigma_{\Delta p/p_{0}}$ | $0.02$
RF voltage (single-harmonic) $V_{RF}$ | $58.2\text{\,}\mathrm{kV}$
Harmonic $h$ | $10$
Kinetic energy |  $E_{kin}=$200\text{\,}\mathrm{MeV}$/$u
Relativistic $\beta$ factor | 0.568
Revolution frequency $f_{rev}$ | $157\text{\,}\mathrm{kHz}$

The simulations sample the bunch with 1000 macro-particles using an optics-
matched 6D Gaussian phase space distribution. This bunch is tracked through
the synchrotron elements and space charge nodes, 501 of which are placed in
intervals of slightly varying length along the ring. Using more space charge
nodes or more macro-particles does not significantly alter the simulation
results. Space charge is modelled as fixed (i.e. non-adaptive) frozen 3D
Gaussian field maps which remain constant throughout the simulation. The
transverse beam size used in the field maps at each space charge node follows
the computed local $\beta$\- and dispersion functions and is based on the
initial transverse emittances. The transverse nonlinear space charge force is
modulated with the longitudinal Gaussian beam profile. Evaluating the initial
tune footprint in the simulation gives a maximum space charge tune shift from
the bare working point in the vertical plane of $\Delta Q^{SC}_{y}=-0.3$ for
the particles in the centre of the bunch.

Without the multipole errors, the considered SIS100 lattice maintains an
unbroken six-fold symmetry with dipole and quadrupole magnet units each sliced
into 9 thin-lens slices. The $\beta$-beating effect of the longer radiation-
hardened quadrupole units is assumed to be fully corrected. To relate all non-
systematic resonances to the used field error model, the SIS100 model here
replaces the radiation-hardened quadrupole units by their shorter
superconducting counterpart. All dipole and quadrupole magnets around the ring
are assigned multipole error components following a Gaussian distributed
random sequence of stochastic rms amplitude (cut at 2 rms figures) as
described in the previous section. All 9 thin slices of each magnet are
assigned identical error figures. The random number sequence is ensured to be
identical across tunes and even separate simulation scenarios which are
compared in this report. The results based on the chosen random seed show a
conservative resonance behaviour when comparing among 10 different random
seeds.

Misalignment errors are not specifically taken into account in the simulations
carried out for this report. They (i.) result in feed-down for the multipole
components (only minor impact for reasonable rms orbit deviation
$\mathcal{O}($1\text{\,}\mathrm{mm}$)$) and (ii.) lead to diverging closed
orbit distortion close to integer tunes (potentially dangerous impact for
large rms orbit deviation, as beam loss can increase below the integer
$Q_{x}=19$ due to orbit deviations which move the beam into the aperture).

###  3.1  Interpretation of Results from Fixed Frozen vs. Self-consistent
Particle-in-cell Space Charge Models

It is important to note that the assumption of a fixed frozen space charge
model leads to a conservative estimate of the size of loss-free working point
areas: working points without beam loss in the fixed frozen case normally
remain loss-free in realistic simulations employing self-consistent 3D
particle-in-cell (PIC) algorithms. On the other hand, working points of low
but finite beam loss can potentially remain entirely free of beam loss in the
PIC case. Finally, significant beam loss figures predicted by fixed frozen
models tend to underestimate the more realistic figures from PIC, so that
absolute finite beam loss figures in the following tune diagram plots are not
necessarily meaningful (beyond qualitative indication of potentially dangerous
resonance lines). All in all, the assumption of fixed frozen field maps for
space charge enables fast and conservative prediction of loss-free working
point areas, the bounding contours of which might become larger (but usually
not smaller) when predicted by computationally expensive high-resolution PIC
simulations.

##  4  Simulation Results

For the FoS 2 field error model, the beam loss figures in the considered part
of the tune diagram are presented in Fig. 1(a). A good working point area is
located around $Q_{x},Q_{y}=(18.95,18.87)$ indicated by yellow colour and
enclosed by the black contour, inside of which the predicted beam loss across
the injection plateau remains below $5\%$.

The ensemble model based on the 10 series quadrupole units results in the beam
losses presented in the tune diagram in Fig. 1(b). The available low-loss tune
area is significantly enlarged towards the bottom in comparison to the FoS 2
model scenario.

(a) FoS 2 field error model.

(b) Ensemble field error model.

Figure 4.1: Beam loss tune diagrams during full $1\text{\,}\mathrm{s}$
injection plateau for different quadrupole nonlinear field error models.
Resonance lines up to octupolar order are displayed (solid: normal, dashed:
skew, grey: non-systematic, red: systematic).

###  4.1  Comments on Results from FoS 2 Field Error Model

The FoS 2 implied low-loss area is limited to the top by the coupling
line444The Montague (coupling) resonance $2Q_{y}-2Q_{x}$ is the reason for the
limit on top and it is intrinsically driven by the nonlinearities of the space
charge fields: for working points in the vicinity of the coupling line
$Q_{x}=Q_{y}$, the $\approx 2.5\times$ larger horizontal emittance is
exchanged with the vertical plane where the aperture is much more restrictive.
Therefore, the Montague resonance leads to strong beam loss around the
coupling line, which significantly exceeds the beam loss contribution from
skew quadrupole components $a_{2}$ within a reasonable range of tens of units
($a_{2}$ are not taken into account in the error models here though).
$Q_{x}=Q_{y}$ and to the right by the integer resonance $Q_{x}=19$. Both
limitations are either intrinsic (Montague) or a challenge for compensation.
The lower end of the low-loss area is limited by higher-order resonances. It
is here where the nonlinear multipole components become relevant by shrinking
the available tune space.

This is best shown by switching on higher-order multipole components one by
one: the dependence of the loss-free tune area extent on the included
multipole order $n$ (both normal $b_{n}$ and skew $a_{n}$ are included
simultaneously) is displayed in Fig. 4.2.

The blue area includes the field errors of both dipole and quadrupole magnets
up to sextupole order. Next, the turquoise area includes up to octupole order,
which is already almost congruent with the area enclosed by the dashed
reference line (which includes all orders of the field error model). Finally,
the orange area just adds the dodecapole $b_{6}$ component (which on its own
without taking into account $n=3,4$ can lead to finite beam loss on the order
of $1\%$ along the affiliated resonances) – $b_{6}$ explains the small
discrepancy between the turquoise area and the black dotted reference line.

In a nutshell, field imperfections of sextupolar and octupolar order are the
relevant driving terms which severely limit the available tune space to the
bottom.

Figure 4.2: Loss-free tune areas for various included field error orders $n$
(referring to both normal and skew component) in both dipole and quadrupole
magnet units (simulations for 20’000 turns).

###  4.2  Comments on Results from Ensemble Field Error Model

The octupole components $b_{4}$ and $a_{4}$ are both significantly larger in
the FoS 2 model. While $b_{4}$ can drive the normal $4Q_{y}=75$ resonance
located below the lower end of the low-loss working point area, the $a_{4}$
component can drive the skew $3Q_{x}-Q_{y}=38$ resonance which directly
crosses the low-loss area. It thus appears natural to identify the $a_{4}$
component as the main reason for the larger low-loss area in the ensemble
model case. This hypothesis is tested and confirmed by reintroducing a strong
$a_{4}$ component in the ensemble model: using the $a_{4}$ figure from the FoS
2 model while keeping all other $a_{n}$ and $b_{n}$ as in the ensemble model,
we have run another set of simulations for this tune quadrant. Indeed, these
simulation results predict again a reduced low-loss area of equivalent size as
based on the FoS 2 model.

###  4.3  Perfectly Linear Quadrupole Units

Finally we investigate the scenario of perfectly linear quadrupole units for
reference. The nonlinear field imperfections of the quadrupole units are set
to zero in the error model. Only the dipole unit field imperfections provide
driving terms for higher-order nonlinear resonances. These simulation results
are presented in Fig. 4.3. The tune diagram offers more low-loss areas.
Without resorting to careful sorting of single magnet units (with the goal to
cancel certain resonances), this picture can be understood as an optimum
situation with regard to the quadrupole unit quality – the dipole magnet field
quality is now set in stone with its completed unit production. The previously
identified working point region is of similar extent as in the ensemble model
case: this confirms satisfying field quality of the quadrupole unit series
production w.r.t. resonance-induced beam loss during accumulation.

Figure 4.3: Linear and nonlinear dipole unit field errors and only linear
gradient errors in quadrupole units. Beam loss tune diagram during full
injection plateau. Resonance lines up to quadrupolar order displayed.

##  5  Conclusion

Beam losses for heavy-ion bunches with the nominal space charge parameters
have been simulated for a storage time of $1\text{\,}\mathrm{s}$ corresponding
to the SIS100 injection plateau duration, considering two different scenarios
of the nonlinear field error model for the quadrupole units: the First-of-
Series #2 measurement based model (FoS 2) and the “ensemble” model based on
the recently measured first ten quadrupole units from the ongoing series
production. The ensemble model results in a larger acceptable low-loss working
point area in the tune diagram compared to the FoS 2 model. This discrepancy
has been traced to the significantly reduced skew octupolar $a_{4}$ component
in the series production quadrupole units. In general it has been found that
the low-loss working point area can be enlarged by minimising sextupolar and
octupolar field errors. All higher multipole orders are already sufficiently
weak and affiliated resonances remain relatively harmless across the injection
plateau for the studied nominal U28+ intensities.

## References

  * [1] K. Sugita  “"Considerations for Quadrupole Gradient Errors"”, Personal communication, 2020
  * [2] V. Kornilov et al.  “Beam quality and beam loss predictions with space charge for SIS100”  In _JINST_ 15.07, 2020, pp. P07020  DOI: 10.1088/1748-0221/15/07/P07020
  * [3] V. Maroussov  “"First-of-Series Quadrupole Measurements"”, Personal communication from Dec. 2, 2020. Original Report by A. Shemchuk, M. Shandov, from Feb. 16, 2018.
  * [4] M. Schwinzerl et al.  “Optimising and Extending a Single-particle Tracking Library for High Parallel Performance”  In _Proceedings of the 12th International Particle Accelerator Conference_ , 2021, pp. THPAB190
  * [5] R. De Maria et al.  “SixTrack V and runtime environment”  In _International Journal of Modern Physics A_ 34.36 World Scientific, 2019, pp. 1942035
  * [6] M. Schwinzerl  “SixTrackLib”  GitHub, https://github.com/SixTrack/sixtracklib/tree/6893b94f4e872a89869bcdecef91f15648b4cc21, 2020
  * [7] Y. El Hayek and D. Ondreka  “SIS100 optics”  GSI GitLab, https://git.gsi.de/sys/lattices/-/tree/c372947cfd39719efffda9fbce810fae189857fb, 2020


# Automatic music mixing with deep learning and out-of-domain data

###### Abstract

Music mixing traditionally involves recording instruments in the form of
clean, individual tracks and blending them into a final mixture using audio
effects and expert knowledge (e.g., a mixing engineer). The automation of
music production tasks has become an emerging field in recent years, where
rule-based methods and machine learning approaches have been explored.
Nevertheless, the lack of dry or clean instrument recordings limits the
performance of such models, which is still far from professional human-made
mixes. We explore whether we can use out-of-domain data such as wet or
processed multitrack music recordings and repurpose it to train supervised
deep learning models that can bridge the current gap in automatic mixing
quality. To achieve this we propose a novel data preprocessing method that
allows the models to perform automatic music mixing. We also redesigned a
listening test method for evaluating music mixing systems. We validate our
results through such subjective tests using highly experienced mixing
engineers as participants.

##  1 Introduction

Figure 1: Our method uses data preprocessing by computing average features
related to audio effects on an out-of-domain dataset (MSS data). This allows
normalization of wet stems and supervised training of an automatic mixer. At
inference, the same preprocessing is applied to dry data.

Music mixing is a highly cross-adaptive transformation since the processing of
an individual track depends on the content of all tracks involved. Apart from
artistic considerations, it typically tries to solve the problem of unmasking
by manipulating the dynamics, spatialisation, timbre or pitch of multitrack
recordings [1]. This manipulation is achieved through a set of linear and
nonlinear effects, which generally can be classified into five different
classes: gain, equalization (EQ), panning, dynamic range compression (DRC) and
artificial reverberation [2].

Several methods have been investigated to automatize this task [3]. For
example, rule-based systems [4, 5, 6], cross-adaptive audio mixing effects [7,
8] and data-driven methods [9, 10]. A black-box approach is introduced in [9],
where a Wave-U-Net is trained to perform automatic mixing of drums.
Conversely, neural proxies of audio effects are used as a fixed signal
processing path within a deep neural framework to perform automatic mixing of
songs [10]. However, the lack of dry or unprocessed multitrack data has
limited the performance of these deep learning approaches. Thus an unified
approach has yet to be found that achieves results close to or superior to the
quality of professional human-made mixes [11, 12]. It has been hypothesized
that the bottleneck of performance can be resolved with a large enough dataset
[10]. Nevertheless, collecting data is difficult, as it is unusual for
musicians and record labels to provide multitrack dry recordings.

In this work, we consider the use of out-of-domain data in conjunction with a
supervised deep learning approach to close such performance gap. To achieve
this, we propose a novel method that performs a data normalization or
augmentation procedure on each of the audio effect classes. Figure 1 depicts
our method. We train deep neural networks to perform automatic loudness, EQ,
panning, DRC and reverberation music mixing. Thus we present 1) a data
preprocessing step that allows training with out-of-domain data, 2) a new deep
learning architecture, 3) an exploration of stereo-invariant loss functions,
4) a design of a perceptual listening test targeting highly skilled
professionals, and 5) listening test results showing that our approach is
indistinguishable from professional human-made mixes. Audio samples and code
can be found at https://marco-martinez-sony.github.io/FxNorm-automix/.

##  2 Method

###  2.1 Effect normalization and augmentation

One of the main challenges of deep learning models for automatic mixing is the
lack of multitrack dry data. Since collecting a large dry multitrack dataset
is inherently difficult, we believe it is possible to reuse existing datasets,
such as music source separation (MSS) data. Music source separation has been
heavily researched in the last decade [13] and a significant effort has been
put into collecting training data, such as the well-known MUSDB18 dataset
[14]. However, direct application of such data is infeasible, since the
mixture is a summation of the wet stems, that is, mixing effects have already
been applied.

Several methods have been shown to be capable of reverse engineering the
effect parameters [15, 16, 17], however such approaches require the original
dry multitrack recordings and target mix. There are also data-driven methods
for removing effects [18, 19], although they only apply to specific effects
and are prone to adding sound artifacts.

Instead of removing audio effects, we propose to normalize each stem based on
audio features related to each class of audio effects. In this way, different
data features are scaled or transformed to make an equal contribution across
the dataset [20]. We propose normalization schemes for loudness, EQ, panning,
DRC and reverberation, thus ensuring that all stems have been normalized to
the same range of audio features. During training, we expect the models to
learn how to undo or denormalize the input stems and thus approximate the
original mix. At inference, we also normalize the real multitrack dry data
thus allowing the model to perform automatic music mixing. This Section
introduces how each effect is normalized and Section 2.2 the full data
preprocessing procedure.

Loudness—To normalize loudness, we independently compute the average loudness
$\mathcal{L}^{(k)}$ for the stem type $k$ as

$\mathcal{L}^{(k)}=\frac{1}{N}\sum_{i=1}^{N}\text{LUFS}(x^{(k)}_{i}),$ (1)

where LUFS is the integrated loudness level in dBFS in accordance to [21], $x$
is the $i$th stem waveform of $k$ type, e.g. $k$=vocals, and $N$ is the total
number of available songs. Then, based on $\mathcal{L}^{(k)}$, each stem is
loudness normalized using [22].

Equalization—EQ normalization is based on the average frequency magnitude
spectrum $\mathcal{F}^{(k)}$ as

$\mathcal{F}(\omega)^{(k)}=\frac{1}{N}\sum_{i=1}^{N}{\Gamma}(\omega)_{i}^{(k)},\quad{\Gamma}(\omega)^{(k)}=\frac{1}{M}\sum_{j=1}^{M}X(\omega)^{(k)}_{j},$
(2)

where $\Gamma^{(k)}$ corresponds to the individual stem mean frequency
magnitude, $\omega$ is the frequency index, $X^{(k)}$ is the magnitude of the
Short-Time Fourier Transform (STFT) of each $x^{(k)}$ and $M$ its total number
of frames.

We then proceed to normalize each stem by performing EQ matching with respect
to $\mathcal{F}^{(k)}$. EQ matching typically involves finding the optimal
filter settings by designing and applying a filter based on the difference
between the target and input frequency magnitudes [23]. The differential
frequency magnitude $\mathcal{F}(\omega)^{(k)}_{\text{diff}}$ is computed as

$\mathcal{F}(\omega)^{(k)}_{\text{diff}}=10^{\log_{10}(\mathcal{F}(\omega)^{(k)})-\log_{10}(\Gamma(\omega)^{(k)})}.$
(3)

${\mathcal{F}(\omega)^{(k)}_{\text{diff}}}$ is further smoothed with a
Savitzky-Golay filter [24]. An FIR filter is designed using the window method
[25], and applied using forward-backward filtering [26] to have a zero-phase
response. Since forward-back filtering squares the magnitude response, we
apply a square root to ${\mathcal{F}(\omega)^{(k)}_{\text{diff}}}$ before
designing the FIR filter.

Panning—The panning normalization is based on the Stereo Panning Spectrum [27,
28], where we use the average panning as a reference then re-pan accordingly.

We compute the left and right channel similarity measure $\Psi(\omega)$, to
approximate the panning gains assigned during mixing. We focus only in
amplitude panning thus we calculate $\Psi(\omega)$ using only frequency
magnitude as

$\Psi(\omega)=2\frac{X(\omega)_{{\mathrm{L}}}X(\omega)_{{\mathrm{R}}}}{X(\omega)_{{\mathrm{L}}}^{2}+X(\omega)_{{\mathrm{R}}}^{2}},$
(4)

where $X(\omega)_{{\mathrm{L}}}$ and $X(\omega)_{{\mathrm{R}}}$ correspond to
the left and right channel STFT magnitudes. $\Psi(\omega)$ denotes whether a
frequency bin $\omega_{0}$ is panned to the center
($\Psi(\omega)_{\omega_{0}}$=1), or whether is panned to either side
(0$\leq\Psi(\omega)_{\omega_{0}}<$1). The stereo side can be determined with
the difference $\Delta(\omega)$ between partial similarities
$\Psi(\omega)_{\mathrm{L}}$ and $\Psi(\omega)_{\mathrm{R}}$ as

$\Psi(\omega)_{\mathrm{L}}=\frac{X(\omega)_{{\mathrm{L}}}X(\omega)_{{\mathrm{R}}}}{X(\omega)_{{\mathrm{L}}}^{2}},\Psi(\omega)_{\mathrm{R}}=\frac{X(\omega)_{\mathrm{L}}X(\omega)_{{\mathrm{R}}}}{X(\omega)_{{\mathrm{R}}}^{2}},$
(5)

thus $\Delta(\omega)=\Psi(\omega)_{\mathrm{L}}-\Psi(\omega)_{\mathrm{R}}$,
where $\Delta(\omega)$>0 and $\Delta(\omega)$<0 correspond to signals panned
left or right, respectively. The panning gains, $\Phi(\omega)_{\mathrm{L}}$
and $\Phi(\omega)_{\mathrm{R}}$, then can be estimated by choosing a panning
law to approximate the panning coefficient $\alpha(\omega)$. We empirically
found that a linear panning law,
$\Phi(\omega)_{\mathrm{L}}$=1-$\alpha(\omega)$ and
$\Phi(\omega)_{\mathrm{R}}$=$\alpha(\omega)$, yields better approximations.
Thus we approximate the panning gains based on $\Delta(\omega)$ and assuming
$\Psi(\omega)$=2$\alpha(\omega)$. We compute the average similarity measure
$\mathcal{S}(\omega)^{(k)}$ across all $\Psi(\omega)^{(k)}$ computed from $N$
stems and their STFT frames $M$. $\mathcal{S}(\omega)^{(k)}$ is also further
smoothed with [24].

For each individual stem, re-panning is implemented by 1) computing
$\Psi(\omega)$ and $\Delta(\omega)$, 2) estimating $\Phi(\omega)_{\mathrm{L}}$
and $\Phi(\omega)_{\mathrm{R}}$, and 3) estimating the reference panning
gains, $\hat{\Phi}(\omega)_{\mathrm{L}}$ and
$\hat{\Phi}(\omega)_{\mathrm{R}}$, using $\mathcal{S}(\omega)^{(k)}$ and
$\Delta(\omega)$. Finally, we calculate a gain factor based on the ratio of
panning gains which we apply to $X(\omega)_{{\mathrm{L}}}$ and
$X(\omega)_{{\mathrm{R}}}$ as

$\hat{X}(\omega)_{\mathrm{L}}=\frac{\hat{\Phi}(\omega)_{\mathrm{L}}}{\Phi(\omega)_{\mathrm{L}}}{X}(\omega)_{\mathrm{L}},\hskip
0.6458pt\hat{X}(\omega)_{\mathrm{R}}=\frac{\hat{\Phi}(\omega)_{\mathrm{R}}}{\Phi(\omega)_{\mathrm{R}}}{X}(\omega)_{\mathrm{R}}.$
(6)

Panning normalization is performed per frame and the normalized audio is
obtained with the inverse STFT of $\hat{X}(\omega)_{\mathrm{L}}$ and
$\hat{X}(\omega)_{\mathrm{R}}$ with their original channel phase.

Dynamic Range Compression—DRC usually alters the transients of the input [29],
thus we base our DRC normalization on the onset peak levels which are linked
to such transient modification [30].

We first perform onset detection based on the High Frequency Content (HFC)
method [31, 32]. HFC emphasizes the energy variation that occurs in the upper
part of the spectrum which is typical of onsets [33]. We then select the
maximum peak for each of the detected onsets.

The average peak level $\mathcal{P}_{\mu}^{(k)}$ and peak level standard
deviation $\mathcal{P}_{\sigma}^{(k)}$ are defined as
${\frac{1}{N}\sum_{i}^{N}}{\mu}^{(k)}_{i}$ and
${\frac{1}{N}\sum_{i}^{N}{\sigma}^{(k)}_{i}}$, respectively. Where $\mu^{(k)}$
and $\sigma^{(k)}$ are the channel mean peak level and standard deviation in
dB, which for a robust estimate are calculated from the top 75th percentile of
detected peaks.

DRC normalization consists in upper bounding the peak levels of the audio.
Thus, if $\mu^{(k)}$ > ($\mathcal{P}_{\mu}^{(k)}$ \+
$\mathcal{P}_{\sigma}^{(k)}$), we apply a compressor to the input audio by
performing an incremental grid search of the ratio and threshold parameters
until $\mu^{(k)}$ < ($\mathcal{P}_{\mu}^{(k)}$ \+
$\mathcal{P}_{\sigma}^{(k)}$). Threshold is the level above which compression
starts, and ratio determines the amount of compression [29]. This is done with
fixed attack and release values, i.e. the start and stop timing settings.

Figure 2: Block diagram of the proposed model

Artificial Reverberation—Due to the inherent characteristics of reverberation
[29], an attempt to similarly normalize this effect is not trivial. Blind
estimation of reverberation features, such as reverberation time (RT) or
direct-to-reverberant ratio, is an open research area in itself, and
considering that most of the research has been done for speech signals [34,
35, 36], its application to music is beyond the scope of this paper.

We propose instead a data augmentation approach where we stochastically add
reverberation to already reverberated stems. Reverberation is added using a
mixing method called the ’Abbey Road reverb trick’ [37], where a chain of EQ
and reverb effects is applied as a send effect, i.e. a copy of the input is
processed and added to the input. Naively adding reverberation directly to the
input audio has the potential to clutter the low and low-mid frequencies. In
this manner, the process of learning how much reverberation is required for a
given mix is carried out by the network by learning to filter out the
additional reverberation that is present in the input stems.

###  2.2 Data preprocessing

Since most mixing effects are interrelated, applying the normalization methods
separately yields inaccurate results, e.g. EQ normalization modifies the
dynamics of the audio, thus modifying DRC features. To minimize this, we
perform effect preprocessing progressively and based in the following order:
EQ, DRC, panning and loudness. Since EQ and DRC normalization are done on a
per channel basis, panning normalization is applied after the above to avoid
further modification of the stereo features.

Thus, for each stem type $k$, average feature computation corresponds to 1)
calculate $\mathcal{F}(\omega)^{(k)}$ and EQ normalize, 2) calculate
$\mathcal{P}^{(k)}_{\mu}$ and $\mathcal{P}^{(k)}_{\sigma}$ and DRC normalize,
3) calculate $\mathcal{S}(\omega)^{(k)}$ and panning normalize, and 4)
calculate $\mathcal{L}^{(k)}$ and loudness normalize.

We apply our reverberation preprocessing method after all the average features
have been calculated. Considering that this procedure is based on a stochastic
data augmentation procedure, for consistency we treat the added reverberation
as noise for the normalization pipeline. The final data preprocessing method
corresponds to applying reverberation augmentation followed by EQ, DRC,
panning, and loudness normalization methods.

###  2.3 Architecture

We propose a new architecture based on [38] and [39]. It operates in the time-
domain and processes raw waveforms in a frame-wise manner. The model can be
divided into three parts: adaptive front-end, latent-space mixer and synthesis
back-end. The model is depicted in Figure 2 and its architecture is summarized
in Table 1.

The adaptive front-end is exactly the same as in [38], with the only
difference that now the input $\hat{x}$ corresponds to $K$ stereo tracks of
length $A$. In the front-end, time-domain convolutions are applied to the
input audio in order to learn a latent representation $\boldsymbol{Z}$ and a
filter bank which output feature map $\boldsymbol{X}_{1}$ corresponds to a
frequency band decomposition of $\hat{x}$. The latent-space mixer corresponds
to the temporal dilated convolutions (TCN) separator block of [39], and to
improve long-term dependencies learning, the TCN is followed by stacked
Bidirectional Long Short-Term Memory (BLSTM) layers. Contrary to [39], the
objective of the mixer is not to learn a mask for source separation, but
rather to learn a mixing mask $\hat{\boldsymbol{Z}}$.

The synthesis back-end uses $\hat{\boldsymbol{Z}}$ to modify
$\boldsymbol{X}_{1}$ based on the given mixing task and its design is
motivated by [38]. It upsamples the mixing mask using nearest neighbor
interpolation and via an element-wise multiplication applies it to each
filter-bank channel source of $\boldsymbol{X}_{1}$ at each time step. We
hypothesize that this frequency-based transformation is akin to the model
learning and applying a dynamic equalization effect [40] while also filtering
out the extra reverberant content of the normalized input stems.

The resulting feature map $\boldsymbol{X}_{4}$ is further modified via a
Squeeze-and-Excitation (SE) block [41] implemented as shown in [38]. The SE
layers scale the channel-wise information of $\boldsymbol{X}_{4}$ by applying
an adaptive gain $\boldsymbol{s}_{c}$, and consequently, learning a loudness
gain and panning transformation for each filter bank channel. The modified
frequency decomposition $\boldsymbol{X}_{5}$ is then reconstructed using a
non-trainable transposed convolution as shown in [38]. Finally, the resulting
$2K$ stereo tracks are channel-wise summed into a stereo output and a
hyperbolic tangent is used to avoid clipping. All convolutions use a stride of
1 to avoid ringing and filtering artifacts [42]. BLSTMs and SE layers are
applied to the filter dimension.

Table 1: Summarized architecture of the proposed model.

Layer | Output shape | Output
---|---|---
Input stems | ($2K$, $A$) | $\boldsymbol{\hat{x}}$
Conv1D | ($N$, $A$) | $\boldsymbol{X}_{1}$
Conv1D-Local | ($N$, $A$) | $\boldsymbol{X}_{2}$
MaxPooling | ($N$, $A/64$) | $\boldsymbol{Z}$
TCN | ($N$, $A/64$) | .
BLSTM | ($A/64$, $N$) | $\boldsymbol{\hat{Z}}$
Unpooling | ($N$, $A$) | $\boldsymbol{X}_{3}$
$\boldsymbol{X}_{3}\times\boldsymbol{X}_{1}$ | ($N$, $A$) | $\boldsymbol{X}_{4}$
SE (Abs) | ($A$, $N$) | .
SE (Global Avg) | (1, $N$) | .
SE (FC) | (1, $16N$) | .
SE (FC) | (1, $N$) | $\boldsymbol{s}_{c}$
$\boldsymbol{X}_{4}\times s_{c}$ | ($N$, $A$) | $\boldsymbol{{X}}_{5}$
Conv1D.T | ($2K$, $A$) | .
Summation | ($2$, $A$) | $\hat{y}$

###  2.4 Loss function

Based on the stereo-invariant loss function introduced by [10], we explore two
variations of such loss. First, due to the perceptual nature of the mixing
task and motivated by [43], we apply A-weighting pre-emphasis and low-pass FIR
filters ($\rho$) to the target and output audio frames $y$ and $\hat{y}$,
respectively. Then we compute the sum and difference signals
$y_{\text{sum}}$=$\rho(y_{\mathrm{L}})$+$\rho(y_{\mathrm{R}})$ and
$y_{\text{diff}}$=$\rho(y_{\mathrm{L}})$-$\rho(y_{\mathrm{R}})$.

The first loss follows closely [10] and is based on the Spectral Convergence
(SC), magnitude-normalized Frobenius norm, and the L1-norm spectral log-
magnitude (L1Log) as

$\begin{split}L_{\text{a}}=l_{\text{SC}}(Y_{\text{sum}},\hat{Y}_{\text{sum}})+l_{\text{L1Log}}(Y_{\text{sum}},\hat{Y}_{\text{sum}})\\\
+l_{\text{SC}}(Y_{\text{diff}},\hat{Y}_{\text{diff}})+l_{\text{L1Log}}(Y_{\text{diff}},\hat{Y}_{\text{diff}}),\end{split}$
(7)

where $Y$ and $\hat{Y}$ are the respective 4096-Fast Fourier Transform (FFT)
magnitude with a 25% hop size.

The second loss replaces the SC loss component with a less penalizing
normalization such as the widely used L2-norm on the spectral magnitude (L2),
defined as

$\begin{split}L_{\text{b}}=l_{\text{L2}}(Y_{\text{sum}},\hat{Y}_{\text{sum}})+l_{\text{L1Log}}(Y_{\text{sum}},\hat{Y}_{\text{sum}})\\\
+l_{\text{L2}}(Y_{\text{diff}},\hat{Y}_{\text{diff}})+l_{\text{L1Log}}(Y_{\text{diff}},\hat{Y}_{\text{diff}}).\end{split}$
(8)

We empirically found the perceptual-based pre-emphasis filter as vital when
modeling a highly perceptual task such as music mixing. We also found an
easier convergence during training when using a single frame-size loss rather
than a multi-resolution magnitude loss.

##  3 Experiments

###  3.1 Dataset

We first conduct experiments with a small dataset (S) which corresponds to the
MUSDB18 dataset [14]. This dataset consists of wet stems for vocals, drums,
bass and other ($K$=4), where the mixture is the summation of such stems.
There are a total of 150 songs, of which 86 are used for training and 14 and
50 for validation and testing purposes, respectively.

We also use a private large dataset (L) for music separation which corresponds
to 1,505 extra multitrack songs created in the same manner as MUSDB18. We
incorporate the MUSDB18 training and validation sets into the large dataset,
thus obtaining a total of 1,455+86=1,541 training songs and 50+14=64
validation songs. In both datasets, most of the songs correspond to mainstream
western music, with rock and pop as the predominant genres.

To validate our method, we use a private set of 18 dry multitrack songs, where
all songs have been produced by different musicians and mixing engineers. From
this data we grouped the respective dry stems without applying any effect in
the process.

###  3.2 Dataset preprocessing

We apply our preprocessing methods to both datasets independently. EQ
preprocessing is computed with a 65,536 point STFT with hop size of 25% and a
FIR filter of 1,001 taps. In order to avoid clipping, all audio stem channels
are loudness normalized to -30 dBFS prior to the EQ feature computation and
normalization.

For DRC preprocessing, to determine the timing settings of the compressor used
during the normalization, we averaged the values found in mixing engineering
best practices [8, 44]. Attack values correspond to 7.5, 10, 10 and 15 ms and
release values to 400, 180, 500 and 666 ms, for vocals, drums, bass and other,
respectively. For the incremental grid search we use ratio and threshold
values within {4, 20} and {-40, -10} db respectively. For the HFC computation
we use 128 mel bands of all stems with the exception of bass where we found
better onset detection with 16 mel bands. Prior to the DRC preprocessing, all
channels are peak normalized to -10 dB.

Panning preprocessing is done with a 2,048 point STFT with hop size of 50%.
For all stems, frequency bins are re-panned until 16 kHz, with the exception
of drums, which is re-panned until 16 kHz in order to avoid artifacts.

Reverberation augmentation is done by applying uniform random sampling on a
set of 130 different impulse responses (IR) whose RT is within {2, 4} seconds
(s). The EQ used prior to the reverberation corresponds to a low-shelf and
high-shelf with a fixed gain of -30 dB whose cutoff frequency is uniformly
sampled within {500, 700} Hz and {7, 10} kHz, respectively. At inference, to
simulate the reverberant characteristics of the training data, before applying
the aforementioned augmentation, a "pre-reverb" is added in the same manner
but from a different set of 400 IRs whose RT is within {1, 1.5}-s. A shorter
RT is chosen, as high reverberation levels has been shown to have a more
detrimental effect on subjective preference than low levels [45].
Reverberation augmentation is applied only to vocals and other stems.
Reverberation and DRC effects are implemented using the Python package from
[10].

###  3.3 Hyperparameters

The convolutional layers on the front-end have $N$=128 filters of size 64 and
128 respectively, and a 64-point window for max-pooling. In the mixer, the
bottleneck layer has 256 channels, the skip connection paths have 64 channels,
and the convolutional blocks have 128 channels of kernel size 3. We use 4
stacks of 6 convolutional layers with a dilation factor of 1,2,…,32. We
stacked 3 BLSTMs with a hidden feature map of 64 channels. The FC layers in
the SE block have 2048 (16$N$) and 128 channels respectively.

The input consists of 2$K$ channels, each channel consisting of $A$=10-s audio
frames at 44.1 kHz. The receptive field of the mixer is 505 samples, and
taking into account the pooling operation, the overall network has a receptive
field of 32,446 samples. Thus, the loss function is only calculated on a
8.52-s frame centered in the middle of the 10-s input. The network has in
total 2.7M parameters. As a baseline, we use the modified Wave-U-Net (WUN)
introduced by [9]. The same settings are used as in the original work which
yields a network of 2.5M parameters.

###  3.4 Training

Regarding on-the-fly data augmentation, we apply a randomization of the order
of the stereo channels, this is done consistently across the stems and the
target mix. For the proposed network we use the pretraining from [38]. We
train both models using a batch size of 4, an initial learning rate
$\beta$=0.001, and the following learning rate schedule; $\beta$ for 300
epochs, $\beta$/3 for 100 epochs, $\beta$/10 for 100 epochs, $\beta$/30 for 50
epochs, $\beta$/100 for 50 epochs and $\beta$/1000 for 25 epochs. An epoch
consists of 1600 batches, L2 norm of the gradient is clipped by 0.2 and we use
10-7 for weight decay regularization. We select the model with the lowest
validation loss.

##  4 Results & Analysis

We trained both networks with the preprocessed datasets S and L and the loss
functions $L_{a}$ and $L_{b}$, which yielded Ours-S-La, Ours-S-Lb, Ours-L-La
and Ours-L-Lb for our proposed network, and WUN-S-Lb and WUN-L-Lb for Wave-U-
Net. Convergence during training for WUN with $L_{a}$ was unsuccessful,
therefore it is excluded from the results.

###  4.1 Quantitative evaluation

To measure how close the output mixes are to the reference mixes, we use the
following audio features, spectral: centroid, bandwidth, contrast, flatness,
and roll-off [46]; panning: the Panning Root Mean Square (RMS) [27]; dynamic:
RMS level, dynamic spread and crest factor [47]; and LUFS loudness level [21].
All features are computed using a running mean of 0.5-s [27]. Table 2 shows
the results for the dry test set and the MUSDB18 test set. As expected, the
overall error values are larger for the dry test set. Also, the loudness,
dynamics, and panning values show a closer match to the reference mix when
compared to the Normalized mix, which is the sum of input stems after data
preprocessing. Spectral values do not match in the same way, which could
indicate that the generated mixes deviate in terms of EQ and reverberation.

Figure 3: Listening test boxplots and violin plots for all individual six
songs (S1 to S6) and all songs, respectively.

| dry test set | MUSDB18 test set
---|---|---
model |  spectral |  panning |  dynamic |  loudness |  spectral |  panning |  dynamic |  loudness
Normalized |  0.435 |  0.593 |  0.168 |  0.633 |  0.256 |  0.783 |  0.109 |  0.643
WUN-S-Lb |  0.527 |  0.215 |  0.082 |  0.094 |  0.250 |  0.250 |  0.078 |  0.097
Ours-S-La |  0.547 |  0.201 |  0.062 |  0.056 |  0.299 |  0.191 |  0.086 |  0.095
Ours-S-Lb |  0.427 |  0.207 |  0.063 |  0.061 |  0.276 |  0.212 |  0.085 |  0.084
WUN-L-Lb |  0.551 |  0.182 |  0.066 |  0.054 |  0.279 |  0.195 |  0.074 |  0.072
Ours-L-La |  0.590 |  0.191 |  0.055 |  0.091 |  0.312 |  0.593 |  0.168 |  0.105
Ours-L-Lb |  0.519 |  0.170 |  0.056 |  0.061 |  0.276 |  0.160 |  0.084 |  0.079

Table 2: Objective metrics correspond to the average mean absolute percentage
error by feature subgroup.

###  4.2 Listening Test

We designed a the listening test using the Web Audio Evaluation Tool [48] and
the APE interface [49]. The test is intended for professional mixing engineers
only and we ask participants to rate the mixes based on Production Value,
Clarity and Excitement as shown in [50]. Perceptual tests for mixing systems
[10, 9] often differ from MUSHRA [51] tests, as the reference sample is
omitted in order to encourage a direct comparison between mixes. However,
based on feedback from pilot tests, we decided to include the 4 dry stems in
the test as references. In this way, the ratings may reflect the quality of
transformation on each stem as applied by the mixing systems.

Fourteen participants with an average mixing engineering experience of 11.6
years took part in the test. In total there were six different songs and for
each song six different mixes were presented. Each mix was 25-s taken from the
chorus-to-verse transitions of the full mixes. The six mixes correspond to the
Ours-S-Lb, Ours-L-La, Ours-L-Lb, WUN-S-Lb and WUN-L-Lb models plus a
professional Human mix. Ours-S-La was omitted from the test to allow more
songs to be tested while avoiding listening fatigue [52]. A low-anchor was not
used as it has been shown to compress the other ratings at the higher end [1].
All mixes were loudness normalized to -$23$ dBFS [53].

Figure 3 shows the results of the listening test and Table 3 shows the
pairwise comparisons of mixes. For Production Value, there is no statistically
significant difference between the Human mixtures and the models that were
trained on the large dataset. For Clarity, Human mixes are rated lower than
Ours-L-La, Ours-L-Lb with a p-value of 0.008 and 0.013, respectively,
indicating that our models generate mixtures with less masking. For
Excitement, the null hypothesis is accepted, and therefore Human mixes are
considered no different than any other mixture system. Although Ours-L-Lb was
among the best among all criteria, the null hypothesis is also accepted when
compared to WUN-L-Lb. Thus, a further analysis is needed among both networks,
e.g. in terms of long-term learned dependencies and ability to generate full-
length coherent mixes.

In general, models trained with large datasets received the highest scores,
which could confirm that lack of data has been the bottleneck of data-driven
mixing systems. However, for Clarity and Excitement, the mixes by WUN-S-Lb are
not considered different from the rest, which might indicate that our data
preprocessing method is both effective for small and large datasets.

It should be noted that, in general, none of the mixes is consistently
considered as very good. This relates with other findings, where even
commercial mixes made by renown engineers are not rated as high [44, 1]. This
opens up a new research direction for evaluating mixing systems, as this type
of test is too difficult for inexperienced participants, and when experienced
participants do participate, they tend not to rate any mix as very good.

For individual songs, Human mixes have lower rates for Jazz and Dance-Pop. For
the latter, low ratings on all criteria may be due to highly compressed drums.
However, the high ratings in terms of Excitement and Production Value for the
Human J-Rock mix might be due to hard-panned guitars. In contrast, the models
are conservative when it comes to panning and are unlikely to hard-pan
sources, however a further analysis is required. Furthermore, although we show
the models successfully mixing multiple genres, an in-depth analysis of the
genre distribution of the dataset with ratings by genre is needed.

Production Value | WUN-S-Lb | Ours-S-Lb | WUN-L-Lb | Ours-L-La | Ours-L-Lb | Human
---|---|---|---|---|---|---
WUN-S-Lb | $\cdot$ | o | $\stackunder[1.2pt]{$\star$}{\rule{3.44444pt}{0.04303pt}}$ | o | $\stackunder[1.2pt]{$**$}{\rule{3.44444pt}{0.04303pt}}$ | $\stackunder[1.2pt]{$\star$}{\rule{3.44444pt}{0.04303pt}}$
Ours-S-Lb | o | $\cdot$ | o | o | $\stackunder[1.2pt]{$**$}{\rule{3.44444pt}{0.04303pt}}$ | $\stackunder[1.2pt]{$\star$}{\rule{3.44444pt}{0.04303pt}}$
WUN-L-Lb | $\accentset{\rule{2.41112pt}{0.03012pt}}{\star}$ | o | $\cdot$ | o | o | o
Ours-L-La | o | o | o | $\cdot$ | o | o
Ours-L-Lb | $\accentset{\rule{2.41112pt}{0.03012pt}}{**}$ | $\accentset{\rule{2.41112pt}{0.03012pt}}{**}$ | o | o | $\cdot$ | o
Human | $\accentset{\rule{2.41112pt}{0.03012pt}}{\star}$ | $\accentset{\rule{2.41112pt}{0.03012pt}}{\star}$ | o | o | o | $\cdot$

Clarity | WUN-S-Lb | Ours-S-Lb | WUN-L-Lb | Ours-L-La | Ours-L-Lb | Human
---|---|---|---|---|---|---
WUN-S-Lb | $\cdot$ | o | o | o | o | o
Ours-S-Lb | o | $\cdot$ | o | $\stackunder[1.2pt]{$*$}{\rule{3.44444pt}{0.04303pt}}$ | $\stackunder[1.2pt]{$**$}{\rule{3.44444pt}{0.04303pt}}$ | o
WUN-L-Lb | o | o | $\cdot$ | o | o | o
Ours-L-La | o | $\accentset{\rule{2.41112pt}{0.03012pt}}{*}$ | o | $\cdot$ | o | $\accentset{\rule{2.41112pt}{0.03012pt}}{*}$
Ours-L-Lb | o | $\accentset{\rule{2.41112pt}{0.03012pt}}{**}$ | o | o | $\cdot$ | $\accentset{\rule{2.41112pt}{0.03012pt}}{\star}$
Human | o | o | o | $\stackunder[1.2pt]{$*$}{\rule{3.44444pt}{0.04303pt}}$ | $\stackunder[1.2pt]{$\star$}{\rule{3.44444pt}{0.04303pt}}$ | $\cdot$

Excitement | WUN-S-Lb | Ours-S-Lb | WUN-L-Lb | Ours-L-La | Ours-L-Lb | Human
---|---|---|---|---|---|---
WUN-S-Lb | $\cdot$ | o | o | o | o | o
Ours-S-Lb | o | $\cdot$ | $\stackunder[1.2pt]{$*$}{\rule{3.44444pt}{0.04303pt}}$ | $\stackunder[1.2pt]{$\star$}{\rule{3.44444pt}{0.04303pt}}$ | $\stackunder[1.2pt]{$*$}{\rule{3.44444pt}{0.04303pt}}$ | o
WUN-L-Lb | o | $\accentset{\rule{2.41112pt}{0.03012pt}}{*}$ | $\cdot$ | o | o | o
Ours-L-La | o | $\accentset{\rule{2.41112pt}{0.03012pt}}{\star}$ | o | $\cdot$ | o | o
Ours-L-Lb | o | $\accentset{\rule{2.41112pt}{0.03012pt}}{*}$ | o | o | $\cdot$ | o
Human | o | o | o | o | o | $\cdot$

Table 3: Post hoc Mann-Whitney test results of pairwise comparison with
Bonferroni Correction. o > 0.05, $\star$ < 0.05, $*$ < 0.01, $**$ < 0.001.
E.g. when y-axis is compared to x-axis, $\bar{*}$ or
$\stackunder[1.2pt]{$*$}{\rule{3.44444pt}{0.04303pt}}$ indicate y-axis is
significantly better or worse than x-axis for a p-value < 0.01, respectively.

##  5 Conclusion

We present a novel data preprocessing approach that allows us to train deep
learning networks with existing wet or processed multitrack data by reusing
them to perform an automatic mixing task. During inference, we apply the same
data preprocesing to dry multitrack data and we tested the generated mixes via
objective and subjective tests. We introduced a new deep learning architecture
designed for the proposed task, a perceptual-based loss function along with a
redesigned listening test aimed at experienced mixing engineers. The results
indicate that our approach successfully achieves automatic loudness, EQ, DRC,
panning, and reverberation music mixing. Resulting mixes compared to
professional mixes scored higher in terms of Clarity and are indistinguishable
in terms of Production Value and Excitement. We believe that the proposed
preprocessing can be applied to other data-driven MIR tasks.

##  6 Acknowledgments

We would like to express special thanks to K. Gokan and S. Masui for their
valuable comments and all the mixing engineers from Sony Music Entertainment
Japan, Sony Europe and Queen Mary University of London who participated in the
listening test.

## References

  * [1] R. Stables, J. D. Reiss, and B. De Man, _Intelligent Music Production_. Focal Press, 2019.
  * [2] P. D. Pestana and J. D. Reiss, “Intelligent audio production strategies informed by best practices,” in _53rd Conference on Semantic Audio: Audio Engineering Society_ , January 2014.
  * [3] B. De Man, J. D. Reiss, and R. Stables, “Ten years of automatic mixing,” in _3rd AES Workshop on Intelligent Music Production_ , September 2017.
  * [4] G. Bocko, M. F. Bocko, D. Headlam, J. Lundberg, and G. Ren, “Automatic music production system employing probabilistic expert systems,” in _Audio Engineering Society Convention 129_. Audio Engineering Society, November 2010.
  * [5] B. De Man and J. D. Reiss, “A knowledge-engineered autonomous mixing system,” in _Audio Engineering Society Convention 135_. Audio Engineering Society, October 2013.
  * [6] F. Everardo, “Towards an automated multitrack mixing tool using answer set programming,” in _Proceedings of the 14th Sound and Music Computing Conference_ , Espoo, Finland, July 2017.
  * [7] J. D. Reiss, “Intelligent systems for mixing multichannel audio,” in _17th International Conference on Digital Signal Processing_. IEEE, 2011, pp. 1–6.
  * [8] P. Pestana, “Automatic mixing systems using adaptive digital audio effects,” Ph.D. dissertation, Universidade Católica Portuguesa, 2013.
  * [9] M. A. Martínez-Ramírez, D. Stoller, and D. Moffat, “A deep learning approach to intelligent drum mixing with the wave-u-net.” Journal of the Audio Engineering Society, 2021.
  * [10] C. J. Steinmetz, J. Pons, S. Pascual, and J. Serrà, “Automatic multitrack mixing with a differentiable mixing console of neural audio effects,” in _IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)_ , 2021.
  * [11] D. Moffat and M. B. Sandler, “Approaches in intelligent music production,” _Arts_ , vol. 8, no. 5, p. 14, September 2019. [Online]. Available: https://doi.org/10.3390/arts8040125
  * [12] T. Wilmering, D. Moffat, A. Milo, and M. B. Sandler, “A history of audio effects,” _Applied Sciences_ , vol. 10, no. 3, p. 791, January 2020. [Online]. Available: https://doi.org/10.3390/app10030791
  * [13] Y. Mitsufuji, G. Fabbro, S. Uhlich, F.-R. Stöter, A. Défossez, M. Kim, W. Choi, C.-Y. Yu, and K.-W. Cheuk, “Music demixing challenge 2021,” _Frontiers in Signal Processing_ , vol. 1, 2022. [Online]. Available: https://doi.org/10.3389/frsip.2021.808395
  * [14] Z. Rafii, A. Liutkus, F.-R. Stöter, S. I. Mimilakis, and R. Bittner, “The MUSDB18 corpus for music separation,” Dec. 2017. [Online]. Available: https://doi.org/10.5281/zenodo.1117372
  * [15] D. Barchiesi and J. Reiss, “Reverse engineering of a mix,” _Journal of the Audio Engineering Society_ , vol. 58, no. 7/8, pp. 563–576, 2010.
  * [16] D. Moffat and M. B. Sandler, “Automatic mixing level balancing enhanced through source interference identification,” in _Audio Engineering Society Convention 146_ , Dublin, Ireland, March 2019.
  * [17] J. T. Colonel and J. Reiss, “Reverse engineering of a recording mix with differentiable digital signal processing,” _The Journal of the Acoustical Society of America_ , vol. 150, no. 1, pp. 608–619, 2021.
  * [18] J. Imort, G. Fabbro, M. A. Martínez-Ramírez, S. Uhlich, Y. Koyama, and Y. Mitsufuji, “Distortion audio effects: Learning how to recover the clean signal,” in _23rd International Society for Music Information Retrieval Conference (ISMIR)_ , 2022.
  * [19] J. Koo, S. Paik, and K. Lee, “Reverb conversion of mixed vocal tracks using an end-to-end convolutional deep neural network,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2021.
  * [20] D. Singh and B. Singh, “Investigating the impact of data normalization on classification performance,” _Applied Soft Computing_ , vol. 97, 2020. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1568494619302947
  * [21] R. ITU-R, “ITU-R BS. 1770-2, algorithms to measure audio programme loudness and true-peak audio level,” _International Telecommunications Union, Geneva_ , 2011.
  * [22] C. J. Steinmetz and J. D. Reiss, “pyloudnorm: A simple yet flexible loudness meter in python,” in _Audio Engineering Society Convention 150_ , 2021.
  * [23] F. G. Germain, G. J. Mysore, and T. Fujioka, “Equalization matching of speech recordings in real-world environments,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2016.
  * [24] R. W. Schafer, “What is a savitzky-golay filter?” _IEEE Signal processing magazine_ , vol. 28, no. 4, pp. 111–117, 2011.
  * [25] P. Gaydecki, _Foundations of digital signal processing: theory, algorithms and hardware design_. Iet, 2004, vol. 15.
  * [26] J. O. Smith III, _Introduction to Digital Filters: with Audio Applications_. W3K Publishing, 2007, vol. 2.
  * [27] G. Tzanetakis, R. Jones, and K. McNally, “Stereo panning features for classifying recording production style.” in _8th International Society for Music Information Retrieval Conference (ISMIR)_ , 2007.
  * [28] C. Avendano, “Frequency-domain source identification and manipulation in stereo mixes for enhancement, suppression and re-panning applications,” in _IEEE Workshop on Applications of Signal Processing to Audio and Acoustics_ , 2003.
  * [29] U. Zölzer _et al._ , _DAFX-Digital audio effects_. John Wiley & Sons, 2002.
  * [30] J. P. Bello, L. Daudet, S. Abdallah, C. Duxbury, M. Davies, and M. B. Sandler, “A tutorial on onset detection in music signals,” _IEEE Transactions on speech and audio processing_ , vol. 13, no. 5, pp. 1035–1047, 2005.
  * [31] P. Masri, “Computer modelling of sound for transformation and synthesis of musical signals,” Ph.D. dissertation, University of Bristol, 1996.
  * [32] P. Brossier, Tintamar, E. Müller, N. Philippsen, T. Seaver, H. Fritz, cyclopsian, S. Alexander, J. Williams, J. Cowgill, and A. Cruz, “Aubio - a library for audio and music analysis,” February 2019. [Online]. Available: https://doi.org/10.5281/zenodo.2578765
  * [33] P. Brossier, “Automatic annotation of musical audio for interactive applications,” Ph.D. dissertation, Queen Mary University of London, 2006.
  * [34] J. Eaton, N. D. Gaubitch, A. H. Moore, and P. A. Naylor, “Estimation of room acoustic parameters: The ace challenge,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 24, no. 10, pp. 1681–1693, 2016\.
  * [35] H. Löllmann, E. Yilmaz, M. Jeub, and P. Vary, “An improved algorithm for blind reverberation time estimation,” in _Proceedings of International Workshop on Acoustic Echo and Noise Control (IWAENC)_ , 2010, pp. 1–4.
  * [36] S. Duangpummet, J. Karnjana, W. Kongprawechnon, and M. Unoki, “Blind estimation of room acoustic parameters and speech transmission index using mtf-based cnns,” in _29th European Signal Processing Conference (EUSIPCO)_ , 2021, pp. 181–185.
  * [37] P. White, “Processing reverb,” _Sound On Sound_ , vol. November 2020 Edition, 2022. [Online]. Available: https://www.soundonsound.com/techniques/processing-reverb
  * [38] M. A. Martínez-Ramírez, E. Benetos, and J. D. Reiss, “Deep learning for black-box modeling of audio effects,” _Applied Sciences_ , vol. 10, no. 2, p. 638, 2020. [Online]. Available: https://doi.org/10.3390/app10020638
  * [39] Y. Luo and N. Mesgarani, “Conv-tasnet: Surpassing ideal time–frequency magnitude masking for speech separation,” _IEEE/ACM transactions on audio, speech, and language processing_ , vol. 27, no. 8, pp. 1256–1266, 2019\.
  * [40] V. Välimäki and J. D. Reiss, “All about audio equalization: Solutions and frontiers,” _Applied Sciences_ , vol. 6, no. 5, p. 129, 2016.
  * [41] J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 7132–7141.
  * [42] J. Pons, S. Pascual, G. Cengarle, and J. Serrà, “Upsampling artifacts in neural audio synthesis,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2021.
  * [43] A. Wright and V. Välimäki, “Perceptual loss function for neural modeling of audio systems,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2020.
  * [44] B. De Man, “Towards a better understanding of mix engineering,” Ph.D. dissertation, Queen Mary University of London, 2017.
  * [45] B. De Man, K. McNally, and J. D. Reiss, “Perceptual evaluation and analysis of reverberation in multitrack music production,” _Journal of the Audio Engineering Society_ , 2017.
  * [46] G. Peeters, “A large set of audio features for sound description (similarity and classification) in the cuidado project,” _Analysis/Synthesis Team. IRCAM, Paris, France_ , vol. 54, no. 0, pp. 1–25, 2004.
  * [47] Z. Ma, B. De Man, P. D. Pestana, D. A. Black, and J. D. Reiss, “Intelligent multitrack dynamic range compression,” _Journal of the Audio Engineering Society_ , vol. 63, no. 6, pp. 412–426, 2015.
  * [48] N. Jillings, B. De Man, D. Moffat, J. D. Reiss, and R. Stables, “Web audio evaluation tool: A browser-based listening test environment,” in _International Sound and Music Computing Conference_ , Maynooth, Ireland, July 2015.
  * [49] B. De Man and J. D. Reiss, “APE: Audio perceptual evaluation toolbox for matlab,” in _Audio Engineering Society Convention 136_. Audio Engineering Society, 2014.
  * [50] P. D. Pestana and J. D. Reiss, “A cross-adaptive dynamic spectral panning technique.” in _17th International Conference on Digital Audio Effects (DAFx-14)_ , 2014.
  * [51] ITU-R BS. 1534-1, “Method for the subjective assessment of intermediate sound quality (MUSHRA),” _International Telecommunications Union, Geneva_ , 2001.
  * [52] R. Schatz, S. Egger, and K. Masuch, “The impact of test duration on user fatigue and reliability of subjective quality ratings,” _Journal of the Audio Engineering Society_ , vol. 60, no. 1/2, pp. 63–73, 2012.
  * [53] EBU, “EBU recommendation: Loudness normalisation and permitted maximum level of audio signals,” EBU-R128: Saconnex, 2014.