# Why do Angular Margin Losses work well for Semi-Supervised Anomalous Sound
Detection?

Kevin Wilkinghoff , , and Frank Kurth The authors are with Fraunhofer FKIE,
Fraunhoferstraße 20, 53343 Wachtberg, Germany (e-mail:
<EMAIL_ADDRESS>frank.kurth@fkie.fraunhofer.de).
0000-0003-4200-9129 0000-0002-9992-083X

###### Abstract

State-of-the-art anomalous sound detection systems often utilize angular
margin losses to learn suitable representations of acoustic data using an
auxiliary task, which usually is a supervised or self-supervised
classification task. The underlying idea is that, in order to solve this
auxiliary task, specific information about normal data needs to be captured in
the learned representations and that this information is also sufficient to
differentiate between normal and anomalous samples. Especially in noisy
conditions, discriminative models based on angular margin losses tend to
significantly outperform systems based on generative or one-class models. The
goal of this work is to investigate why using angular margin losses with
auxiliary tasks works well for detecting anomalous sounds. To this end, it is
shown, both theoretically and experimentally, that minimizing angular margin
losses also minimizes compactness loss while inherently preventing learning
trivial solutions. Furthermore, multiple experiments are conducted to show
that using a related classification task as an auxiliary task teaches the
model to learn representations suitable for detecting anomalous sounds in
noisy conditions. Among these experiments are performance evaluations,
visualizing the embedding space with t-SNE and visualizing the input
representations with respect to the anomaly score using randomized input
sampling for explanation.

###### Index Terms:

representation learning, anomaly detection, angular margin loss, compactness
loss, machine listening, domain generalization, explainable artificial
intelligence

##  I Introduction

Semi-supervised anomalous sound detection (ASD) is the task of reliably
detecting anomalous sounds while only having access to normal sounds for
training a model [1]. Since anomalies occur only rarely by definition and
usually are very diverse, collecting realistic anomalous samples for training
a system is much more difficult and thus more costly than collecting normal
data. Hence, a semi-supervised ASD setting is more realistic than a supervised
ASD setting, for which anomalous sounds are available for training, because it
substantially simplifies the data collection process. There are also
unsupervised ASD settings, for which the training dataset may also contain
anomalous samples and it is unknown whether a training sample is normal or
anomalous. But for many applications, it can be ensured that only normal
samples are collected for training and thus a semi-supervised setting can be
assumed.

ASD has many applications. Examples are machine condition monitoring [2, 3,
4], medical diagnosis [5, 6], bioacoustic monitoring [7, 8], intrusion
detection in smart home environments [9] and detecting crimes [10, 11] or
accidents [12, 13]. Furthermore, detecting anomalous samples can also be
understood as a subtask in acoustic open-set classification [14, 15, 16].
Throughout this work, we will use machine condition monitoring in domain-
shifted conditions as an application example [4]. Here, the audio signals may
contain one or several of the following three components: 1) normal machine
sounds, 2) anomalous machine sounds and 3) background noise consisting of a
mixture of many other sound events. The major difficulty of this ASD
application is that anomalous components of machine sounds can be very subtle
when being compared to the background noise making it difficult to reliably
detect anomalous signal components. Furthermore, machine sounds and background
noise can change substantially for different domain shifts, which we define as
alterations in the (acoustic) environment or changes in parameter settings of
the machines. The ASD system still needs to only detect anomalous signal
components without frequently raising false alarms caused by any domain shift.

There are several strategies to train an ASD system for machine condition
monitoring using only normal data. Among these strategies are generative
models such as autoencoders [17, 18, 19, 20, 21, 22] or normalizing flows [23,
24] that directly try to model the probablity distribution of normal data,
which is also called inlier modeling (IM) [3]. Another strategy is to use an
auxiliary task, usually a classification task, for training a model to learn
meaningful representations of the data (embeddings) that can be used to
identify anomalies. Possible auxiliary tasks for machine condition monitoring
are classifying between machine types [25, 26, 27, 28, 29] or, additionally,
between different machine states and noise settings [30, 31, 32, 33],
recognizing augmented and not augmented versions of normal data (self-
supervised learning) [25] or predicting the activity of machines [32]. Using
an auxiliary task to learn embeddings is also called outlier exposure (OE)
[34] because normal samples belonging to other classes than a target class can
be considered as proxy outliers [35]. Often an angular margin loss such as
SphereFace [36], CosFace [37] or ArcFace [38] is utilized for training an OE
model. Systems based on embeddings pre-trained on very large datasets [39, 40,
41] can be used, too. However, it has been shown that directly training a
system on the data yields better ASD results, even when only very limited
training data is available [42]. In addition, different strategies can be
combined by using an ensemble of multiple models [43, 44, 45].

Different strategies to train an ASD system have different strengths and
weaknesses. Using an auxiliary task for training relies on additional meta-
information to generate labels for a classification task whereas IM-based
models do not need any labels. Furthermore, autoencoders can localize
anomalies in the input space by visualizing an element-wise reconstruction
error as done in [21, 19]. However, training ASD models by using an auxiliary
task usually enhances their performance [46]. Even for IM-based models,
performance can be significantly improved when utilizing meta information such
as machine types. In [21] a class-conditioned autoencoder is used, in [44] not
only spectral features but also the machine ID is encoded and decoded, and in
[23] a normalizing flow is trained to assign lower likelihood to sounds of
other machines and a higher likelihood to sounds of the target machine. As
suspected in [32, 33], the most likely reason for the difference in
performance is that, as stated before, recordings for machine condition
monitoring are very noisy because of factory background noise. This is a
problem for IM-based models because they cannot tell the difference between
arbitrary sound events not emitted by a monitored machine and normal or
anomalous sounds emitted by the machine. Both are considered equally important
by the model. Moreover, anomalies present in these noisy audio recordings are
usually very subtle when being compared to the noise or other sound events
present in a recording making it even more difficult to detect potential
anomalies. When being trained with an auxiliary task, a model learns to ignore
noise, which can be assumed to be similar for all considered classes, and
therefore to isolate the target machine sound by ignoring the uninformative
background sound events. As a result, these models are more sensitive to
changes of the machine sounds and have better anomaly detection capabilities.

Localizing and visualizing frequencies or temporal regions of recordings that
are being considered anomalous is important for practical applications because
users can better understand the decisions of the ASD system ( explainable
artificial intelligence (xAI) [47]). Furthermore, this may help to find the
cause of mechanical failure and thus can simplify the maintenance process. As
stated before, autoencoders can easily localize anomalies by using an element-
wise reconstruction error. Additional investigations on visualizing and
explaining ASD decisions include showing that decisions of ASD systems for
machine condition monitoring largely rely on high-frequency information [48].
This has been visualized using LIME [49] applied to sounds (SLIME)[50].
Furthermore, uniform manifold approximation and projection (UMAP) [51] has
been used to visualize representations of the data such as stacked consecutive
frames of log magnitude spectrograms, log-mel magnitude spectrograms, or
openL3 embeddings [46].

The goal of this work is to explain why angular margin losses work well for
anomalous sound detection. To achieve this goal, the following contributions
are made: First and foremost, it is theoretically proven that, after
normalizing the embedding space, training an ASD model by minimizing an
angular margin loss using an auxiliary task can be considered as minimizing a
regularized one-class loss while being less affected by noise or non-target
sound events present in the data. Moreover, it is experimentally verified that
using an angular margin loss for training a model to discriminate between
classes of an auxiliary task also leads to better ASD performance and thus is
a better choice for an ASD task than minimizing a one-class loss such as an
intra-class (IC) compactness loss with a single or multiple classes. Last but
not least, a procedure for visualizing normal and anomalous regions of the
input representations based on randomized input sampling for explanation
(RISE) is presented. Using these visualizations, it is shown that normal and
anomalous sounds cannot be distinguished from the highly complex background
noise when training with a one-class loss. In contrast, when using an
auxiliary task with multiple classes the model learns to ignore noise and
isolate the targeted machine sound for monitoring their condition.

The paper is structured as follows: In Section II, various one-class losses
and angular margin losses are reviewed. Section III presents our main
theoretical results about the relation between these loss functions. Section
IV contains a description of the experimental setup and all experimental
evaluations consisting of performance evaluations, a comparison between losses
during training, visualizing normal and anomalous regions of input
representations as perceived by the system and visualizing the resulting
embedding spaces. Section V consists of the conclusions of this work. Last but
not least, the proofs of the results presented in III can be found in the
[sec:appendix]appendix.

##  II Loss Functions

In this section, a unified presentation and discussion of several loss
functions that are needed for presenting one of the main results of this work
in Sec. III will be given. The following notation will be used throughout the
paper: $X$ denotes the space of input data samples, $N\in\mathbb{N}$ the
number of classes defined for an auxiliary task and $D\in\mathbb{N}$ the
dimension of the embedding space.

###  II-A One-Class Classification Losses

When training a model for ASD while only having access to normal data i.e. a
single class, this is referred to as _one-class classification_ and is some
form of IM. The compactness loss [52], whose goal it is to project the data
into a hypersphere of minimum volume, will serve as a representative of losses
for one-class classification and is defined as follows.

######  Definition 1 (Compactness loss).

Let $Y\subset X$ be finite. Let $\mathcal{P}$ denote the power set, $\Phi$
denote the space of network architectures for extracting embeddings and
$W(\phi)$ denote the parameter space of $\phi\in\Phi$, i.e. ${\phi:X\times
W(\phi)\rightarrow\mathbb{R}^{D}}$. Then, the _compactness loss_ is defined as

$\displaystyle\mathcal{L}_{\text{comp}}:\mathcal{P}(X)\times\mathbb{R}^{D}\times\Phi\times
W\rightarrow\mathbb{R}_{+}$ (1)
$\displaystyle\mathcal{L}_{\text{comp}}(Y,c,\phi,w):=\frac{1}{\lvert
Y\rvert}\sum_{x\in Y}\lVert\phi(x,w)-c\rVert_{2}^{2}.$

The vector $c\in\mathbb{R}^{D}$ is called _center_.

After training, the (squared) Euclidean distance between the embedding of a
given sample and the center can be utilized as an anomaly score: A greater
distance indicates a higher likelihood for the sample to be anomalous. A
_trivial solution_ for minimizing the compactness loss with center
$c\in\mathbb{R}^{D}$ is a parameter setting $w_{c}\in W(\phi)$ such that
$\phi$ is the constant function ${\phi(x,w_{c})=c}$ for all ${x\in X}$. It is
of utmost importance to prevent that the model to be trained is able to learn
such a trivial solution. Otherwise it is impossible to differentiate between
normal and anomalous samples.

There are several strategies to prevent a model from learning a trivial
solution. First of all, it needs to be ensured that ${c\neq
c_{0}\in\mathbb{R}^{D}}$ where $c_{0}=\phi(x,w_{0})$ is defined as the output
of the network obtained by setting the weight parameters of model $\phi$ to
zero. This is because we have $\phi(x,w_{0})=c_{0}$ for all $x\in X$ as long
as the model uses only linear operators, e.g. dense or convolutional layers,
and all activation functions have zero as a fixed point, which is the case for
most commonly used activation functions. In [52], is has been shown that using
bias terms, bounded activation functions or a trainable center all enable the
model to learn a constant function and thus must also be avoided.

Another possibility to avoid trivial solutions is to impose additional tasks,
so-called _auxiliary tasks_ , not directly related to the ASD problem while
training. Autoencoders [53], which are trained to first encode and then decode
the input again and have many interesting applications by themselves such as
denoising data [54], can also be viewed as a way to regularize one-class
models. Here, the encoder is the one-class model mapping the input to an
embedding space. Learning a constant function is not a (trivial) solution for
the task because all necessary information for being able to completely
reconstruct the input needs to be encoded. However, noise including other
sound sources present in the input audio data needs to be encoded as well
because otherwise the input cannot be reconstructed. Therefore, the noise
heavily influences the embeddings and thus the embeddings can also be
considered noisy. Depending on the complexity of the noise, most information
contained in the embeddings is only related to the noise and not to the target
sound to be analyzed and thus detecting anomalies using an autoencoder may be
difficult. Moreover, in [52] it has been shown that using compactness loss,
even for clean datasets, outperforms commonly used autoencoder architectures
when detecting anomalies.

A second choice of an auxiliary task to prevent the model from learning a
constant function as a trivial solution is a classification task. Defining
multiple classes through an auxiliary task inherently prevents learning a
constant function as this would not be a (trivial) solution to the imposed
classification problem. In [55], an additional _descriptiveness loss_ is used
whose goal is to reduce inter-class similarity between classes of an
arbitrary, external multi-class dataset, which is only used to regularize the
one-class classification task. This is done by minimizing the standard
categorical cross-entropy (CCE) loss for classification on this additional
dataset as an auxiliary task. For each of the two tasks, another version of
the same network with identical structure and tied weights is used. During
training, both losses are jointly minimized using a weighted sum ensuring that
the so-called reference network associated with the compactness loss does not
learn a constant function because this would prevent the secondary network to
be able to classify correctly.

######  Remark.

The original definition of the compactness loss [52] also includes an
additional weight decay term. Such a weight decay term can be used to
complement any loss function and does not prevent the model from learning
trivial solutions as it is still possible that the model learns to map
everything to the center. Furthermore, all theoretical results presented in
this work are valid regardless of whether the weight decay term is included or
not. The proof of the main theorem can easily be modified to including the
weight decay term because it is just an additional additive term. Therefore,
we omitted this term in the theoretical investigations of this work for the
sake of simplicity while still using it in our experiments. However, we did
not notice any significant effect on the performance.

For the remainder of this work, we propose to normalize all representations in
the embedding space $\mathbb{R}^{D}$, meaning that ${\lVert
c\rVert_{2}=1=\lVert\phi(x,w)\rVert_{2}}$ for all $x\in X,w\in W(\phi)$ and
centers ${c\in\mathbb{R}^{D}}$. This can easily be achieved by dividing the
embeddings by their corresponding Euclidean norms. A normalization of the
embedding space essentially reduces the dimension by one as evident by using
stereographic projection. But doing so does not degrade the ASD performance
because the dimension of the embedding space usually is larger than it needs
to be.

Normalizing the embedding space has several advantages. Most importantly, the
initialization of the centers is substantially simplified. In high-dimensional
vector spaces i.i.d. random elements are almost surely approximately
orthogonal [56]. Hence, all class centers can be randomly initialized by
sampling from a uniform random distribution as also done in [33] and a careful
strategy for initializing the class centers is not needed. This does not cause
any problems e.g. by accidentally using class centers that are very similar to
each other in terms of cosine similarity whereas the corresponding acoustic
classes are very dissimilar or vice versa. Moreover, normalizing the centers
ensures that all centers are distributed equidistantly and sufficiently far
away from zero to avoid learning a trivial solution. Last but not least,
normalizing the embeddings may even prevent numerical issues while training
similar to when using batch normalization [57].

###  II-B Angular Margin Losses

We will review the definition of ArcFace [38] as a representative of angular
margin losses.

######  Definition 2 (ArcFace).

Let $Y\subset X$ be finite and ${l_{j}(x)\in\\{0,1\\}}$ denote the $j$th
component of the categorical class label function $l\in L$ where $L$ denotes
the space of all functions $l:X\rightarrow\\{0,1\\}^{N}$ with
$\sum_{j=1}^{N}l_{j}(x)=1$ for all $x\in X$. Let $\mathcal{P}$ denote the
power set, $\Phi$ denote the space of network architectures for extracting
embeddings and $W(\phi)$ denote the parameter space of $\phi\in\Phi$, thus
$\phi:X\times W(\phi)\rightarrow\mathbb{R}^{D}$. Let
$\operatorname*{smax}:\mathbb{R}^{N}\rightarrow[0,1]^{N}$ denote the softmax
function, i.e.

$\displaystyle\operatorname*{smax}(x)_{i}=\frac{\exp(x_{i})}{\sum_{j=1}^{N}\exp(x_{j})}.$
(2)

Then, the _ArcFace_ loss is defined as

$\displaystyle\mathcal{L}_{\text{ang}}:\mathcal{P}(X)\times\mathcal{P}(\mathbb{R}^{D})\times\Phi\times
W\times L\times\mathbb{R}_{+}\times[0,\frac{\pi}{2}]\rightarrow\mathbb{R}_{+}$
(3) $\displaystyle\mathcal{L}_{\text{ang}}(Y,C,\phi,w,l,s,m)$
$\displaystyle:=$ $\displaystyle-\frac{1}{\lvert Y\rvert}\sum_{x\in
Y}\sum_{j=1}^{N}l_{j}(x)\log(\operatorname*{smax}(s\cdot\cos_{\text{mar}}(\phi(x,w),c_{j},m)))$

where $\lvert C\rvert=N$ and, in this case,

$\displaystyle\operatorname*{smax}(s\cdot\cos_{\text{mar}}(\phi(x,w),c_{i},m))$
(4) $\displaystyle:=$
$\displaystyle\frac{\exp(s\cdot\cos_{\text{mar}}(\phi(x,w),c_{i},m))}{\sum_{j=1}^{N}\exp(s\cdot\cos_{\text{mar}}(\phi(x,w),c_{j},m\cdot
l_{j}(x))}$

with

$\displaystyle\cos_{\text{mar}}(x,y,m):=\cos(\arccos(\cos(x,y))+m)$ (5)

for cosine similarity

$\displaystyle\cos(x,y):=\frac{\langle x,y\rangle}{\lVert x\rVert_{2}\lVert
y\rVert_{2}}\in[-1,1].$ (6)

The vectors $c_{j}\in\mathbb{R}^{D}$ are called _class centers_ ,
$m\in[0,\frac{\pi}{2}]$ is called margin and $s\in\mathbb{R}_{+}$ is called
scale parameter.

######  Remark.

When using mixup [58] for data augmentation, the definition of the class label
function needs to be generalized to $l:X\rightarrow[0,1]^{N}$ with
$\sum_{j=1}^{N}l_{j}(x)=1$ for all $x\in X$. In the experimental evaluations
of this work, mixup will be used when training a model as this improves the
ASD performance [29]. Furthermore, the theoretical results presented in this
work still hold when using mixup but in the proofs only binary labels will be
used for the sake of simplicity.

In [59], it has been shown that the choice of both hyperparameters, the scale
parameter $s$ and the margin $m$, can have a significant impact on the
resulting performance. Strongly varying the magnitude of one of the individual
parameters has a similar effect on the sensitivity of the posterior
probabilities with respect to the angles as varying the other parameter. Both
a scale parameter that is too large and a margin that is too small lead to
very high posterior probabilities for the target class, approximately equal to
one, even for relatively large angles. Therefore, the loss function is
insensitive to changing the angle. A scale parameter that is too small limits
the maximum posterior probability of the target class that can be achieved.
Similarly, a margin that is too large also leads to relatively small posterior
probabilities. Thus, in both cases the model still tries to adapt its
parameters even when the angles are already small, which hinders convergence.
Due to the similar behavior of both parameters, a single appropriately chosen
parameter is sufficient for controlling the posterior probabilities and it has
even been shown that an adaptive scale parameter outperforms using two tuned
but fixed parameters. Therefore, we will assume that $s$ is adaptive as
specified for the AdaCos loss in [59] and set $m=0$, i.e.
$\cos_{\text{mar}}(x,y,0)=\cos(x,y)$ for the remainder of this work. Formally,
the definition of the AdaCos loss is the following.

######  Definition 3 (AdaCos).

Using the same notation as in Definition 2, let $Y^{(t)}\subset Y$ denote all
samples belonging to a mini-batch of size $B\in\mathbb{N}$, i.e. $\lvert
Y^{(t)}\rvert=B$. Let
${\theta_{x,i}:=\arccos(\cos(\phi(x,w),c_{i}))\in[0,\pi]}$ and the
_dynamically adaptive scale parameter_ $\tilde{s}^{(t)}\in\mathbb{R}_{+}$ at
training step $t\in\mathbb{N}_{0}$ be set to

$\displaystyle\tilde{s}^{(t)}:=\begin{cases}\sqrt{2}\cdot\log(N-1)&\text{if
}t=0\\\ \frac{\log
B_{\text{avg}}^{(t)}}{\cos\big{(}\min(\frac{\pi}{4},\theta_{\text{med}}^{(t)})\big{)}}&\text{else}\end{cases}$
(7)

where $\theta_{\text{med}}^{(t)}\in[0,\pi]$ denotes the median of all angles
$\theta_{x,i(x)}$ with $x\in X^{(t)}$ and $i(x)\in\\{1,...,N\\}$ such that
$l_{i}(x)=1$ and

$\displaystyle B_{\text{avg}}^{(t)}:=\frac{1}{B}\sum_{x\in
Y^{(t)}}\sum_{\begin{subarray}{c}j=1\\\ l_{j}(x)\neq
1\end{subarray}}^{N}\exp\big{(}\tilde{s}^{(t-1)}\cdot\cos(\phi(x,w),c_{j})\big{)}.$
(8)

Then, the _AdaCos_ loss is defined as

$\displaystyle\mathcal{L}_{\text{ada}}:\mathcal{P}(X)\times\mathcal{P}(\mathbb{R}^{D})\times\Phi\times
W\times L\rightarrow\mathbb{R}_{+}$ (9)
$\displaystyle\mathcal{L}_{\text{ada}}(Y,C,\phi,w,l):=\mathcal{L}_{\text{ang}}(Y,C,\phi,w,l,\tilde{s},0).$

######  Remark.

When using mixup [58] for data augmentation,
${\theta_{\text{med}}^{(t)}\in[0,\pi]}$ needs to be replaced with the median
of the mixed-up angles as specified in [29].

The AdaCos loss can also be extended to using multiple centers for each class,
called sub-clusters, instead of a single one. The idea of using these sub-
clusters is to allow the network to learn more complex distributions than a
normal distribution for each class enabling the model to have a more
differentiated view on the embeddings when using the cosine similarity as an
anomaly score. This has been shown to improve the ASD performance [29] and
thus helps to differentiate between normal and anomalous samples.

######  Definition 4 (Sub-cluster AdaCos).

Using the same notation as in Definitions 2 and 3, let
$C_{j}\in\mathcal{P}(\mathbb{R}^{D})$ with $\lvert C_{j}\rvert=M$ denote all
centers belonging to class $j\in\\{1,...,N\\}$. Let the _dynamically adaptive
scale parameter_ $\hat{s}^{(t)}\in\mathbb{R}_{+}$ at training step
$t\in\mathbb{N}_{0}$ be set to

$\displaystyle\hat{s}^{(t)}:=\begin{cases}\sqrt{2}\cdot\log(N\cdot
M-1)&\text{if }t=0\\\
\frac{f_{\text{max}}^{(t)}+\log\hat{B}_{\text{avg}}^{(t)}}{\cos\big{(}\min(\frac{\pi}{4},\theta_{\text{med}}^{(t)})\big{)}}&\text{else}\end{cases}$
(10)

with

$\displaystyle\hat{B}_{\text{avg}}^{(t)}:=\frac{1}{B}\sum_{x\in
Y^{(t)}}\sum_{j=1}^{N}\sum_{c\in
C_{j}}\exp\big{(}\hat{s}^{(t-1)}\cos(\phi(x,w),c)-f_{\text{max}}^{(t)}\big{)}$
(11)

and

$\displaystyle{f_{\text{max}}^{(t)}:=\max_{x\in
Y^{(t)}}\max_{j=1}^{N}\max_{c\in
C_{j}}\hat{s}^{(t-1)}\cdot\cos(\phi(x,w),c)}.$ (12)

Then, the _sub-cluster AdaCos_ loss is defined as

$\displaystyle\mathcal{L}_{\text{sc-
ada}}:\mathcal{P}(X)\times\mathcal{P}(\mathcal{P}(\mathbb{R}^{D}))\times\Phi\times
W\times L\rightarrow\mathbb{R}_{+}$ (13) $\displaystyle\mathcal{L}_{\text{sc-
ada}}(Y,C,\phi,w,l)$ $\displaystyle:=$ $\displaystyle-\frac{1}{\lvert
Y\rvert}\sum_{x\in
Y}\sum_{j=1}^{N}l_{j}(x)\log(\operatorname*{smax}(\hat{s}\cdot\cos(\phi(x,w),C_{j})))$

where $\lvert C\rvert=N$ and, in this case,

$\displaystyle\operatorname*{smax}(\hat{s}\cdot\cos(\phi(x,w),C_{j}))$ (14)
$\displaystyle:=$ $\displaystyle\sum_{c_{j}\in
C_{j}}\frac{\exp(\hat{s}\cdot\cos(\phi(x,w),c_{j}))}{\sum_{k=1}^{N}\sum_{c_{k}\in
C_{k}}\exp(\hat{s}\cdot\cos(\phi(x,w),c_{k}))}$

######  Remark.

As shown in [29], for the sub-cluster AdaCos loss as defined above mixup [58]
needs to be used. Otherwise, the dynamically adaptive scale parameter
$\hat{s}^{(t)}$ grows exponentially.

For the compactness loss, there is no benefit of using sub-clusters. The
reason is that an optimal solution of this sub-cluster compactness loss would
correspond to the mean of the sub-clusters or, in case all embeddings are
normalized, to its projection onto the unit sphere. Hence, there would be a
single global optimum and this sub-cluster compactness loss would behave as if
only a single sub-cluster is used. For the sub-cluster AdaCos loss, the
situation is completely different because the softmax function is applied to
all individual sub-clusters and the sum over the resulting scores is taken.
This makes the resulting softmax probability, and thus also the loss function,
symmetric with respect to the corresponding sub-clusters of an individual
class. Therefore, the loss is invariant to changing the position of an
embedding on the hypersphere as long as the sum of the distances to the sub-
clusters is the same. Hence, also the space of optimal solutions grows with
respect to the number of sub-clusters. However, due to the dependence on the
sub-clusters of the other classes caused by the softmax function, this
invariance is a simplification and the real situation is more complex.

##  III Relation between One-Class Losses and Angular Margin Losses

For the proof of the main theoretical result of this work, the following basic
identity is needed.

######  Lemma 5.

For $x,y\in\mathbb{R}^{D}$ with $\lVert x\rVert_{2}=\lVert y\rVert_{2}=1$, it
holds that

$\displaystyle\cos(x,y)=1-\frac{\lVert x-y\rVert_{2}^{2}}{2}.$ (15)

###### Proof.

See Appendix. ∎

######  Remark.

This lemma also shows that for normalized embeddings using Euclidean distance
and using cosine distance, which in this case is equal to the standard scalar
product, are equivalent for computing an anomaly score.

Now, the theorem itself follows.

######  Theorem 6.

Let $Y_{j}:=\\{x\in Y:l_{j}(x)=1\\}$. Then minimizing $\mathcal{L}_{\text{sc-
ada}}(Y,C,\phi,w,l)$ with gradient descent minimizes all IC compactness losses
with weighted gradients given by

$\displaystyle\frac{\hat{s}}{2}\sum_{i=1}^{N}\frac{1}{\lvert
Y_{i}\rvert}\sum_{x\in Y_{i}}\sum_{c_{i}\in
C_{i}}P(\tau(\phi(x,w))=c_{i}|\tau(\phi(x,w))\in C_{i})$ (16)
$\displaystyle\cdot$ $\displaystyle\frac{\partial}{\partial
w}\lVert\phi(x,w)-c_{i}\rVert_{2}^{2}$

while maximizing all inter-class compactness losses with weighted gradients
given by

$\displaystyle-\frac{\hat{s}}{2}\sum_{i=1}^{N}\frac{1}{\lvert
Y_{i}\rvert}\sum_{x\in Y_{i}}\sum_{k=1}^{N}\sum_{c_{k}\in
C_{k}}P(\tau(\phi(x,w))=c_{k})$ (17) $\displaystyle\cdot$
$\displaystyle\frac{\partial}{\partial w}\lVert\phi(x,w)-c_{k}\rVert_{2}^{2}$

where

$\displaystyle P(\tau(\phi(x,w))=c_{i}|\tau(\phi(x,w))\in C_{i})$ (18)
$\displaystyle:=$
$\displaystyle\frac{\exp(\hat{s}\cdot\cos(\phi(x,w),c_{i}))}{\sum_{c^{\prime}_{i}\in
C_{i}}\exp(\hat{s}\cdot\cos(\phi(x,w),c^{\prime}_{i}))}$

and

$\displaystyle P(\tau(\phi(x,w))=c_{k})$ (19) $\displaystyle:=$
$\displaystyle\frac{\exp(\hat{s}\cdot\cos(\phi(x,w),c_{k}))}{\sum_{k=1}^{N}\sum_{c^{\prime}_{k}\in
C_{k}}\exp(\hat{s}\cdot\cos(\phi(x,w),c^{\prime}_{k}))}$

with a cluster assignment function
$\tau:\mathbb{R}^{D}\rightarrow\mathbb{R}^{D}$ given by

$\displaystyle\tau(z,C)=\operatorname*{arg\,max}_{c\in C}\cos(z,c).$ (20)

###### Proof.

Let $x\in Y$, $\phi\in\Phi$ and $\hat{s}\in\mathbb{R}_{+}$ be fixed and
$i\in\\{1,...,N\\}$ such that ${l_{i}(x)=1}$ and $l_{j}(x)=0$ for $j\neq i$.
To simplify notation, define ${e(w,c):=\exp(\hat{s}\cdot\cos(\phi(x,w),c))}$.
Using Lemma 5, we see that

$\displaystyle\frac{\partial}{\partial w}\log\bigg{(}\sum_{c_{i}\in
C_{i}}e(w,c_{i})\bigg{)}$ $\displaystyle=$ $\displaystyle\frac{\sum_{c_{i}\in
C_{i}}e(w,c_{i})\cdot\hat{s}\cdot\frac{\partial}{\partial
w}\cos(\phi(x,w),c_{i}))}{\sum_{c^{\prime}_{i}\in C_{i}}e(w,c^{\prime}_{i})}$
$\displaystyle=$ $\displaystyle-\frac{\hat{s}}{2}\sum_{c_{i}\in
C_{i}}\frac{e(w,c_{i})\cdot\frac{\partial}{\partial
w}\lVert\phi(x,w)-c_{i}\rVert_{2}^{2}}{\sum_{c^{\prime}_{i}\in
C_{i}}e(w,c^{\prime}_{i})}$

and similarly

$\displaystyle\frac{\partial}{\partial
w}\log\bigg{(}\sum_{k=1}^{N}\sum_{c_{k}\in C_{k}}e(w,c_{k})\bigg{)}$
$\displaystyle=$ $\displaystyle\frac{\sum_{k=1}^{N}\sum_{c_{k}\in
C_{k}}e(w,c_{k})\cdot\hat{s}\cdot\frac{\partial}{\partial
w}\cos(\phi(x,w),c_{k}))}{\sum_{k=1}^{N}\sum_{c^{\prime}_{k}\in
C_{k}}e(w,c^{\prime}_{k})}$ $\displaystyle=$
$\displaystyle-\frac{\hat{s}}{2}\sum_{k=1}^{N}\sum_{c_{k}\in
C_{k}}\frac{e(w,c_{k})\cdot\frac{\partial}{\partial
w}\lVert\phi(x,w)-c_{k}\rVert_{2}^{2}}{\sum_{k=1}^{N}\sum_{c^{\prime}_{k}\in
C_{k}}e(w,c^{\prime}_{k})}$ $\displaystyle=$
$\displaystyle-\frac{\hat{s}}{2}\sum_{c_{i}\in
C_{i}}\frac{e(w,c_{i})\cdot\frac{\partial}{\partial
w}\lVert\phi(x,w)-c_{i}\rVert_{2}^{2}}{\sum_{k=1}^{N}\sum_{c^{\prime}_{k}\in
C_{k}}e(w,c^{\prime}_{k})}$
$\displaystyle-\frac{\hat{s}}{2}\sum_{\begin{subarray}{c}k=1\\\ k\neq
i\end{subarray}}^{N}\sum_{c_{k}\in
C_{k}}\frac{e(w,c_{k})\cdot\frac{\partial}{\partial
w}\lVert\phi(x,w)-c_{k}\rVert_{2}^{2}}{\sum_{k=1}^{N}\sum_{c^{\prime}_{k}\in
C_{k}}e(w,c^{\prime}_{k})}.$

Using both identities, we obtain

$\displaystyle\frac{\partial}{\partial
w}\sum_{j=1}^{N}l_{j}(x)\log(\operatorname*{smax}(\hat{s}\cdot\cos(\phi(x,w),C_{j})))$
$\displaystyle=$ $\displaystyle\frac{\partial}{\partial
w}\log\bigg{(}\sum_{c_{i}\in
C_{i}}\frac{e(w,c_{i})}{\sum_{k=1}^{N}\sum_{c_{k}\in
C_{k}}e(w,c_{k})}\bigg{)}$ $\displaystyle=$
$\displaystyle\frac{\partial}{\partial w}\log\bigg{(}\sum_{c_{i}\in
C_{i}}e(w,c_{i})\bigg{)}-\frac{\partial}{\partial
w}\log\bigg{(}\sum_{k=1}^{N}\sum_{c_{k}\in C_{k}}e(w,c_{k})\bigg{)}$
$\displaystyle=$ $\displaystyle-\frac{\hat{s}}{2}\sum_{c_{i}\in
C_{i}}\frac{e(w,c_{i})\cdot\frac{\partial}{\partial
w}\lVert\phi(x,w)-c_{i}\rVert_{2}^{2}}{\sum_{c^{\prime}_{i}\in
C_{i}}e(w,c^{\prime}_{i})}$ $\displaystyle+\frac{\hat{s}}{2}\sum_{c_{i}\in
C_{i}}\frac{e(w,c_{i})\cdot\frac{\partial}{\partial
w}\lVert\phi(x,w)-c_{i}\rVert_{2}^{2}}{\sum_{k=1}^{N}\sum_{c^{\prime}_{k}\in
C_{k}}e(w,c^{\prime}_{k})}$
$\displaystyle+\frac{\hat{s}}{2}\sum_{\begin{subarray}{c}k=1\\\ k\neq
i\end{subarray}}^{N}\sum_{c_{k}\in
C_{k}}\frac{e(w,c_{k})\cdot\frac{\partial}{\partial
w}\lVert\phi(x,w)-c_{k}\rVert_{2}^{2}}{\sum_{k=1}^{N}\sum_{c^{\prime}_{k}\in
C_{k}}e(w,c^{\prime}_{k})}$ $\displaystyle=$
$\displaystyle-\frac{\hat{s}}{2}\bigg{(}\sum_{c_{i}\in
C_{i}}e(w,c_{i})\cdot\frac{\partial}{\partial
w}\lVert\phi(x,w)-c_{i}\rVert_{2}^{2}$
$\displaystyle\cdot\bigg{(}\frac{1}{\sum_{c^{\prime}_{i}\in
C_{i}}e(w,c^{\prime}_{i})}-\frac{1}{\sum_{k=1}^{N}\sum_{c^{\prime}_{k}\in
C_{k}}e(w,c^{\prime}_{k})}\bigg{)}$
$\displaystyle-\sum_{\begin{subarray}{c}k=1\\\ k\neq
i\end{subarray}}^{N}\sum_{c_{k}\in
C_{k}}\frac{e(w,c_{k})\cdot\frac{\partial}{\partial
w}\lVert\phi(x,w)-c_{k}\rVert_{2}^{2}}{\sum_{k=1}^{N}\sum_{c^{\prime}_{k}\in
C_{k}}e(w,c^{\prime}_{k})}\bigg{)}$ $\displaystyle=$
$\displaystyle-\frac{\hat{s}}{2}\bigg{(}\sum_{c_{i}\in
C_{i}}e(w,c_{i})\cdot\frac{\partial}{\partial
w}\lVert\phi(x,w)-c_{i}\rVert_{2}^{2}$
$\displaystyle\cdot\bigg{(}\sum_{\begin{subarray}{c}k=1\\\ k\neq
i\end{subarray}}^{N}\sum_{c_{k}\in
C_{k}}\frac{e(w,c_{k})}{(\sum_{c^{\prime}_{i}\in
C_{i}}e(w,c^{\prime}_{i}))(\sum_{k=1}^{N}\sum_{c^{\prime}_{k}\in
C_{k}}e(w,c^{\prime}_{k}))}\bigg{)}$
$\displaystyle-\sum_{\begin{subarray}{c}k=1\\\ k\neq
i\end{subarray}}^{N}\sum_{c_{k}\in
C_{k}}\frac{e(w,c_{k})\cdot\frac{\partial}{\partial
w}\lVert\phi(x,w)-c_{k}\rVert_{2}^{2}}{\sum_{k=1}^{N}\sum_{c^{\prime}_{k}\in
C_{k}}e(w,c^{\prime}_{k})}\bigg{)}$ $\displaystyle=$
$\displaystyle-\frac{\hat{s}}{2}\sum_{\begin{subarray}{c}k=1\\\ k\neq
i\end{subarray}}^{N}\sum_{c_{k}\in
C_{k}}\frac{e(w,c_{k})}{\sum_{k=1}^{N}\sum_{c^{\prime}_{k}\in
C_{k}}e(w,c^{\prime}_{k})}$ $\displaystyle\cdot\bigg{(}\sum_{c_{i}\in
C_{i}}\frac{e(w,c_{i})}{\sum_{c^{\prime}_{i}\in
C_{i}}e(w,c^{\prime}_{i})}\cdot\frac{\partial}{\partial
w}\lVert\phi(x,w)-c_{i}\rVert_{2}^{2}$ $\displaystyle-\frac{\partial}{\partial
w}\lVert\phi(x,w)-c_{k}\rVert_{2}^{2}\bigg{)}$ $\displaystyle=$
$\displaystyle-\frac{\hat{s}}{2}\sum_{k=1}^{N}\sum_{c_{k}\in
C_{k}}\underbrace{\frac{e(w,c_{k})}{\sum_{k=1}^{N}\sum_{c^{\prime}_{k}\in
C_{k}}e(w,c^{\prime}_{k})}}_{=P(\tau(\phi(x,w))=c_{k})}$
$\displaystyle\cdot\sum_{c_{i}\in
C_{i}}\underbrace{\frac{e(w,c_{i})}{\sum_{c^{\prime}_{i}\in
C_{i}}e(w,c^{\prime}_{i})}}_{=P(\tau(\phi(x,w))=c_{i}|\tau(\phi(x,w))\in
C_{i})}$ $\displaystyle\cdot\bigg{(}\frac{\partial}{\partial
w}\lVert\phi(x,w)-c_{i}\rVert_{2}^{2}-\frac{\partial}{\partial
w}\lVert\phi(x,w)-c_{k}\rVert_{2}^{2}\bigg{)}$

where we used that

$\displaystyle\frac{1}{\sum_{c^{\prime}_{i}\in
C_{i}}e(w,c^{\prime}_{i})}-\frac{1}{\sum_{k=1}^{N}\sum_{c^{\prime}_{k}\in
C_{k}}e(w,c^{\prime}_{k})}$ $\displaystyle=$
$\displaystyle\frac{\sum_{k=1}^{N}\sum_{c_{k}\in
C_{k}}e(w,c_{k})-\sum_{c_{i}\in C_{i}}e(w,c_{i})}{(\sum_{c^{\prime}_{i}\in
C_{i}}e(w,c^{\prime}_{i}))(\sum_{k=1}^{N}\sum_{c^{\prime}_{k}\in
C_{k}}e(w,c^{\prime}_{k}))}$ $\displaystyle=$
$\displaystyle\sum_{\begin{subarray}{c}k=1\\\ k\neq
i\end{subarray}}^{N}\sum_{c_{k}\in
C_{k}}\frac{e(w,c_{k})}{(\sum_{c^{\prime}_{i}\in
C_{i}}e(w,c^{\prime}_{i}))(\sum_{k=1}^{N}\sum_{c^{\prime}_{k}\in
C_{k}}e(w,c^{\prime}_{k}))}.$

Now, summing over all samples $x\in Y$, normalizing with $\lvert Y\rvert$ and
taking the additive inverse yields the desired result.

When using mixup, the right hand side of the last equation needs to be
replaced with a weighted sum of two terms, each corresponding to one of the
two classes that are mixed-up, because there are
${i_{1},i_{2}\in\\{1,...,N\\}}$ such that ${l_{i_{1}}(x)\neq 0\neq
l_{i_{2}}(x)}$. Otherwise, the proof is exactly the same. In conclusion, the
proven result still holds for mixed-up samples but includes two similar terms
instead of one term. ∎

######  Corollary 7.

Minimizing $\mathcal{L}_{\text{ada}}(Y,C,\phi,w,l)$ with gradient descent is
equivalent to minimizing

$\displaystyle-\frac{\tilde{s}}{2}\sum_{\begin{subarray}{c}k=1\end{subarray}}^{N}\operatorname*{smax}(\hat{s}\cdot\cos(\phi(x,w),c_{k}))$
(21) $\displaystyle\cdot$ $\displaystyle\bigg{(}\frac{\partial}{\partial
w}\lVert\phi(x,w)-c_{i}\rVert_{2}^{2}-\frac{\partial}{\partial
w}\lVert\phi(x,w)-c_{k}\rVert_{2}^{2}\bigg{)}.$

###### Proof.

The proof of Theorem 6 does not depend on the exact structure of the
dynamically adaptive scale parameter and thus also holds for the standard
AdaCos loss by replacing $\hat{s}$ with $\tilde{s}$ and using only a single
sub-cluster for each class. ∎

$c_{1}$$c_{2}$

Figure 1: Illustration of IC compactness losses and the angular margin to be
ensured between the classes for ${D=2,N=2,M=1}$.

This theorem shows that using an angular margin loss such as the AdaCos loss
is essentially the same strategy as proposed in [55] and applied to ASD in
[27], i.e. using a compactness loss for increasing IC similarity, as defined
in Definition 1, and a so-called descriptiveness loss to decrease inter-class
similarity. However, there are differences between both approaches. When
minimizing an angular margin loss, inter-class compactness losses are used to
decrease inter-class similarity instead of a standard CCE loss. Second, when
using two loss functions one usually has to tune a weight parameter to create
a weighted sum of both loss terms, which is not needed for an angular margin
loss and impossible without access to anomalous samples. Furthermore, the
gradients belonging to individual samples are weighted with specific softmax
probabilities giving more emphasis the closer the sub-clusters are. As these
weights are non-uniform in general, this explicitly shows why using multiple
sub-clusters is not equivalent to using a single sub-cluster given by the
projection of the mean of the sub-clusters onto the hypersphere as it is the
case for an IC compactness loss with multiple sub-clusters. Last but not
least, an angular margin loss explicitly ensures a margin between classes, as
illustrated in Fig. 1, whereas a combination of compactness losses and a CCE
loss only implicitly does this by increasing intra-class similarity. Note
that, in [55], inter-class similarity is decreased on another dataset using
less relevant classes because only a single class is available on the target
dataset. Because of these differences, directly minimizing an angular margin
loss leads to a different solution than minimizing a combination of IC losses
and a descriptiveness loss.

Note that the IC compactness loss with multiple classes can also be considered
a prototypical loss [60] or angular prototypical loss [61] as used for few-
shot classification [62], which defines settings where only very few training
samples, called shots, are available for each class. The only difference
between these prototypical losses and an angular margin loss is that, for
prototypical losses, the center vectors are re-calculated as the means of
embeddings belonging to corresponding classes by using a so-called support set
during training while, for an angular margin loss, the class centers are fixed
or adaptable parameters of the network. Hence, this theorem also shows that
angular margin losses are a suitable choice for few-shot classification as
shown for open-set sound event classification [42] and few-shot keyword
spotting [63].

Choosing a classification task as an auxiliary task prevents learning a
constant function as a trivial solution. The reason is that, for such a
classification task, an optimal solution is a classifier that maps each sample
to its corresponding class center and thus corresponds to jointly learning
multiple trivial solutions, one for each class, instead of only learning a
constant function. As long as each anomalous sample belongs to a well-defined
normal class used during training, this optimal solution would yield
representations not suitable for detecting anomalies as they would not be
distinguishable from representations obtained with normal samples. However,
obtaining such a perfect classifier is much more difficult than learning a
constant mapping for a single class and thus training a single model to
classify between multiple classes already prevents trivial solutions as long
as the classification problem itself is not trivial e.g. by consisting of only
a single class. Still, in [33] it has been shown that the ASD performance can
be improved by applying the same three strategies as used for the compactness
loss [52], namely 1) not using bias terms, 2) not using bounded activation
functions and 3) not using trainable class centers. The most likely reason is
that these strategies prevent the model to learn trivial solutions, leading to
less informative embeddings, for individual classes that are easily
recognized.

##  IV Experimental Results

Using one-class losses and angular margin losses for ASD will now be compared
experimentally.

###  IV-A Dataset

TABLE I: Structure of the DCASE 2022 ASD dataset

number of recordings (per section) subset split source domain target domain
normal anomalous normal anomalous development training $990$ $0$ $10$ $0$
development test $50$ $50$ $50$ $50$ evaluation training $990$ $0$ $10$ $0$
evaluation test $50$ $50$ $50$ $50$

For most experiments conducted in this work, the DCASE 2022 ASD dataset [4] of
the task titled “Unsupervised Anomalous Sound Detection for Machine Condition
Monitoring Applying Domain Generalization Techniques” has been used. The
dataset consists of recordings of machine sounds with background factory
noise. Each recording has a single channel, a length of ten seconds and a
sampling rate of $16$ kHz and belongs to one of the seven machine types “fan”,
“gearbox”, “bearing”, “slide rail”, “valve” from MIMII DG [64] and “toy car”,
“toy train” from ToyADMOS2 [65]. For each machine type, there are six
different so-called sections each of which is dedicated to a specific type of
domain shift. A domain shift means that the characteristics of a machine sound
differ in some way between a source domain with many training samples and a
target domain with only few training samples. These shifts can be caused by
physical changes of the machines e.g. caused by replacing parts for
maintenance, or changes in the acoustical environment e.g. a different
background noise or using different recording devices. Ideally, the ASD system
is able to reliably detect anomalies despite these domain shifts without the
need for adapting the system (domain generalization [66]).

The dataset is divided into a development and an evaluation split each
containing recordings of $21$ sections, three for each machine type. For each
recording, information about the machine type and section are given. For the
training datasets, domain information (“source” or “target”) and additional
attribute information such as states of machine types or noise conditions are
given for each recording. For the test datasets, no domain information and no
additional attribute information are given. The exact structure of the dataset
can be found in Tab I. The task of an ASD system is to reliably detect
anomalous samples regardless of whether a sample belongs to a source or target
domain, i.e. using a single decision threshold for both domains of a section.

Some of the experiments have also been conducted on the DCASE2023 ASD dataset
[67, 68] belonging to the task “First-Shot Unsupervised Anomalous Sound
Detection for Machine Condition Monitoring”. Similar to the DCASE2022 ASD
dataset, this dataset is also aimed at domain generalization for ASD with the
following differences. First and foremost, the development and evaluation
split of the dataset contain different machine types. The development set
contains the same machine types as the DCASE2022 dataset, namely “fan”,
“gearbox”, “bearing”, “slide rail”, “valve” from MIMII DG [64] and “toy car”,
“toy train” from ToyADMOS2 [65]. The evaluation set contains seven completely
different machine types, namely “toy drone”, “n-scale toy train”, “vacuum”,
and “toy tank” from [69] and “bandsaw”, “grinder”, “shaker” from [64].
Furthermore, for each machine type there is only a single section. This lowers
the difficulty of the auxiliary classification task and thus makes it more
difficult to extract embeddings, which are sensitive to anomalous changes of
the target sounds.

For the DCASE ASD datasets, two performance measures are used to evaluate the
performance of individual ASD systems. One metric is the area under the  curve
(AUC), the other metric is the partial area under the  curve (pAUC) [70],
which is the AUC calculated over a low false positive rate ranging from $0$ to
$p$ with $p=0.1$ in this case. The pAUC is used as an additional metric
because decision thresholds for machine condition monitoring are usually set
to a value that gives a low number of false alarms and thus this area of the
ROC curve is of particular interest. Both are threshold-independent metrics
allowing a more objective comparison between different ASD systems than
threshold-dependent metrics [1, 71].

TABLE II: ASD performance obtained with different losses using different
auxiliary tasks. Harmonic means of all AUCs and pAUCs over all pre-defined
sections of the dataset are depicted in percent. Arithmetic mean and standard
deviation of the results over five independent trials are shown. Best results
in each column are highlighted with bold letters.

DCASE2022 development set loss classes of auxiliary task (number of classes)
source domain target domain both domains AUC pAUC AUC pAUC AUC pAUC IC
compactness loss (Def. 1) none ($1$) $56.4\pm 1.4$ $53.9\pm 0.6$ $53.6\pm 0.9$
$52.6\pm 0.3$ $55.1\pm 1.2$ $52.6\pm 0.4$ IC compactness loss (Def. 1) machine
types ($7$) $66.5\pm 2.9$ $60.6\pm 0.6$ $63.6\pm 2.2$ $57.1\pm 0.9$ $65.0\pm
1.7$ $57.8\pm 0.6$ IC compactness loss (Def. 1) machine types and sections
($42$) $77.6\pm 1.7$ $70.5\pm 0.9$ $75.3\pm 0.9$ $63.3\pm 0.8$ $76.4\pm 0.9$
$63.5\pm 0.6$ IC compactness loss (Def. 1) machine types and sections, models
trained individually ($1$) $50.0\pm 2.3$ $52.1\pm 0.6$ $51.7\pm 1.8$ $52.2\pm
0.4$ $51.8\pm 1.8$ $51.4\pm 0.4$ IC compactness loss (Def. 1) machine types,
sections and attribute information ($342$) $80.7\pm 1.9$ $73.7\pm 1.0$
$74.5\pm 0.9$ $62.1\pm 1.2$ $78.1\pm 0.8$ $63.3\pm 0.9$ IC compactness loss
(Def. 1) + CCE machine types, sections and attribute information ($342$)
$82.5\pm 0.7$ $75.2\pm 0.7$ $75.5\pm 0.6$ $61.2\pm 1.6$ $79.0\pm 0.6$ $64.8\pm
0.9$ AdaCos loss (Def. 3) machine types, sections and attribute information
($342$) $83.0\pm 1.3$ $75.2\pm 1.8$ $75.4\pm 1.0$ $60.9\pm 0.8$ $79.2\pm 0.9$
$64.3\pm 0.7$ sub-cluster AdaCos loss (Def. 4) machine types, sections and
attribute information ($342$) $\boldsymbol{84.2\pm 0.8}$ $\boldsymbol{76.5\pm
0.9}$ $\boldsymbol{78.5\pm 0.9}$ $\boldsymbol{62.5\pm 0.9}$
$\boldsymbol{81.4\pm 0.7}$ $\boldsymbol{66.6\pm 0.9}$ DCASE2022 evaluation set
loss classes of auxiliary task (number of classes) source domain target domain
both domains AUC pAUC AUC pAUC AUC pAUC IC compactness loss (Def. 1) none
($1$) $49.9\pm 0.8$ $50.6\pm 0.4$ $51.0\pm 0.4$ $51.0\pm 0.7$ $50.9\pm 0.5$
$50.3\pm 0.4$ IC compactness loss (Def. 1) machine types ($7$) $59.6\pm 1.3$
$56.9\pm 0.5$ $57.6\pm 1.8$ $53.8\pm 0.9$ $59.3\pm 1.5$ $54.6\pm 0.6$ IC
compactness loss (Def. 1) machine types and sections ($42$) $70.8\pm 1.2$
$62.1\pm 0.7$ $61.7\pm 0.8$ $55.4\pm 1.0$ $66.3\pm 0.6$ $56.5\pm 0.4$ IC
compactness loss (Def. 1) machine types and sections, models trained
individually ($1$) $52.9\pm 1.4$ $51.7\pm 0.5$ $54.5\pm 0.6$ $51.6\pm 0.3$
$54.2\pm 0.8$ $51.2\pm 0.3$ IC compactness loss (Def. 1) machine types,
sections and attribute information ($342$) $73.7\pm 0.5$ $63.4\pm 0.7$
$67.9\pm 1.0$ $57.8\pm 1.3$ $70.9\pm 0.6$ $58.5\pm 0.9$ IC compactness loss
(Def. 1) + CCE machine types, sections and attribute information ($342$)
$74.7\pm 0.7$ $64.9\pm 1.1$ $69.2\pm 0.7$ $59.8\pm 1.3$ $71.9\pm 0.6$ $59.5\pm
1.0$ AdaCos loss (Def. 3) machine types, sections and attribute information
($342$) $76.3\pm 1.0$ $\boldsymbol{66.0\pm 0.5}$ $\boldsymbol{69.9\pm 0.8}$
$\boldsymbol{59.9\pm 1.5}$ $73.2\pm 0.4$ $\boldsymbol{60.1\pm 0.9}$ sub-
cluster AdaCos loss (Def. 4) machine types, sections and attribute information
($342$) $\boldsymbol{76.8\pm 0.8}$ $65.8\pm 0.2$ $69.8\pm 0.5$ $59.7\pm 1.1$
$\boldsymbol{73.4\pm 0.5}$ $59.8\pm 0.8$ DCASE2023 development set loss
classes of auxiliary task (number of classes) source domain target domain both
domains AUC pAUC AUC pAUC AUC pAUC IC compactness loss (Def. 1) none ($1$)
$50.7\pm 3.5$ $52.6\pm 0.3$ $45.3\pm 1.9$ $50.1\pm 0.5$ $48.9\pm 1.4$ $50.9\pm
0.4$ IC compactness loss (Def. 1) machine types ($14$) $67.3\pm 2.7$ $63.0\pm
1.4$ $67.8\pm 1.2$ $58.6\pm 1.1$ $67.4\pm 1.4$ $59.4\pm 1.1$ IC compactness
loss (Def. 1) machine types, models trained individually ($1$) $46.7\pm 1.9$
$51.7\pm 0.6$ $45.9\pm 3.2$ $50.4\pm 0.8$ $47.6\pm 2.1$ $50.7\pm 0.6$ IC
compactness loss (Def. 1) machine types and attribute information ($186$)
$67.6\pm 2.5$ $61.6\pm 1.2$ $70.0\pm 2.4$ $56.4\pm 1.9$ $68.3\pm 1.9$ $57.1\pm
1.3$ IC compactness loss (Def. 1) + CCE machine types and attribute
information ($186$) $70.1\pm 1.5$ $63.3\pm 1.3$ $71.0\pm 1.3$ $55.5\pm 1.1$
$70.4\pm 1.0$ $56.7\pm 0.8$ AdaCos loss (Def. 3) machine types and attribute
information ($186$) $69.8\pm 1.5$ $62.8\pm 1.3$ $72.1\pm 1.2$ $55.4\pm 1.7$
$71.2\pm 0.7$ $56.8\pm 1.2$ sub-cluster AdaCos loss (Def. 4) machine types and
attribute information ($186$) $69.4\pm 1.5$ $61.4\pm 1.5$ $72.4\pm 1.6$
$55.3\pm 1.2$ $71.0\pm 1.2$ $56.3\pm 1.1$ DCASE2023 evaluation set loss
classes of auxiliary task (number of classes) source domain target domain both
domains AUC pAUC AUC pAUC AUC pAUC IC compactness loss (Def. 1) none ($1$)
$51.8\pm 2.1$ $51.4\pm 1.2$ $50.0\pm 1.9$ $50.5\pm 0.7$ $51.6\pm 0.9$ $50.8\pm
0.6$ IC compactness loss (Def. 1) machine types ($14$) $59.3\pm 1.9$ $54.4\pm
0.6$ $54.3\pm 2.1$ $51.2\pm 0.5$ $56.7\pm 1.2$ $52.0\pm 0.6$ IC compactness
loss (Def. 1) machine types, models trained individually ($1$) $51.3\pm 0.7$
$51.9\pm 0.7$ $54.7\pm 1.7$ $52.3\pm 0.8$ $53.2\pm 1.0$ $51.5\pm 0.6$ IC
compactness loss (Def. 1) machine types and attribute information ($186$)
$73.0\pm 1.9$ $62.1\pm 1.4$ $58.9\pm 2.7$ $55.1\pm 1.4$ $64.1\pm 1.8$ $55.6\pm
0.8$ IC compactness loss (Def. 1) + CCE machine types and attribute
information ($186$) $72.6\pm 1.4$ $62.5\pm 1.9$ $62.2\pm 2.6$ $56.2\pm 0.8$
$67.2\pm 0.7$ $58.0\pm 0.9$ AdaCos loss (Def. 3) machine types and attribute
information ($186$) $72.3\pm 1.7$ $62.1\pm 1.4$ $61.6\pm 3.1$ $56.4\pm 1.1$
$67.0\pm 1.5$ $57.4\pm 0.9$ sub-cluster AdaCos loss (Def. 4) machine types and
attribute information ($186$) $72.1\pm 1.9$ $61.3\pm 1.5$ $62.3\pm 2.7$
$56.0\pm 0.7$ $67.3\pm 1.3$ $57.4\pm 0.7$

###  IV-B System Description

The focus of this work is to explain why angular margin losses work well for
ASD. This requires using different loss functions for training an ASD system.
To this end, the conceptually simple state-of-the-art system presented in
[33], which only consists of a single model and uses the same settings for all
machine types, is utilized. For all experiments conducted in this work, only
the loss function used for training the system is altered. The system utilizes
a magnitude spectrogram as well as the whole magnitude spectrum as input
representations and uses two different convolutional sub-models for handling
these, resulting in two different embeddings. Then, both embeddings are
concatenated to obtain a single embedding and the sub-cluster AdaCos loss [29]
is applied with $16$ sub-clusters, which are initialized uniformly at random,
for training the model. For the magnitude spectrogram, temporal mean
normalization is applied to reduce the effect of different acoustic domains
and make both input feature representations a bit more different by removing
constant frequency information from the spectrograms. Furthermore, the model
does not use bias terms or trainable clusters as this improves the ASD
performance by avoiding trivial solutions as discussed before. The model is
trained for $10$ epochs with a batch size of $64$ using mixup [58] with a
uniform distribution for sampling the mixing coefficient and is implemented in
Tensorflow [72].

After training the model using an auxiliary classification task, embeddings
are extracted for the recordings. For each section of the dataset, k-means
with $k=16$ is applied to all normal training samples belonging to the source
domain of this section. The goal is to represent the distribution of the
normal embeddings and be able to compute an anomaly score by taking the
minimum cosine distance to the mean embeddings belonging to the same section
as a given test sample. Note that these means do not correspond to the sub-
clusters as some sub-clusters may not have been used by the network during
training. It is possible that the embeddings are clustered between the sub-
clusters due to the complex dependence between the sub-clusters of the other
classes. Still, it has been shown taking the same number of clusters usually
performs best [29]. Since there are only $10$ normal samples available for the
target domain, the minimum over the direct cosine distances to the
corresponding embeddings is used. As a last step, the minimum of the minimum
cosine distances belonging to both domains is used to have an ASD system that
generalizes to both domains. Hence, a higher anomaly score indicates anomalous
sounds whereas a smaller value indicates normal sounds. More details about the
system including a hyperlink to an open-source implementation can be found in
[33].

###  IV-C Performance Evaluations

Regardless of the loss function, training the ASD model without using
anomalous samples is not directly targeting the ASD performance but only
indirectly since the auxiliary task is aimed at obtaining embeddings suitable
for ASD. Although, there is a strong relation between the auxiliary and the
ASD task, as otherwise training an ASD model by using an auxiliary task would
not lead to usable representations, the actual ASD performance needs to be
evaluated experimentally and cannot be investigated theoretically because
there are no anomalous samples available during training. Therefore, the
resulting ASD performances obtained by minimizing both types of loss
functions, angular margin losses and one-class losses, using individual
auxiliary classification tasks will be evaluated first. Furthermore, a
combined loss consisting of the sum of the mean of the IC compactness losses
and an additional softmax layer with a CCE loss for classification, as
proposed in [55], is evaluated. The results can be found in Tab. II. Note that
it is also possible to divide the classification task into several different
classification tasks as for example one task for the machine type and other
ones for all or specific attributes [30, 31]. However, in our experience this
does not improve performance unless weights for the losses belonging to
different machine types are manually tuned to improve the ASD performance.
Since this requires access to anomalous samples, tuning these weights is
impossible in a truly semi-supervised setting.

It can be seen that for both datasets the ASD performance improves with the
number of classes being used for the auxiliary task. When using only a single
class for all data or for individual machine types and sections, the AUC is
close to $50\%$, which corresponds to randomly guessing whether a sample is
anomalous or not. The most likely reason for this is the factory background
noise contained in the recordings, which is highly diverse and contains many
sound sources other than the target machine. A model trained with a one-class
loss does not know the difference between the sound events emitted by the
machines to be monitored and any other sounds contained in the recordings. The
more complex (in terms of numbers of classes) the chosen auxiliary task is,
the more information needs to be captured inside the embeddings for solving
this task. Additionally, the background noise does not contain any helpful
information for learning to discriminate between the classes defined by the
auxiliary task assuming the noise is not class-specific. As a result, the
model learns to monitor specific frequencies or temporal patterns important
for specific machine types with specific settings and thus also learns to
ignore the background noise and to isolate sounds emitted by the targeted
machines. Furthermore, it can be observed that using an explicit
classification task improves performance on all dataset splits. Ensuring an
angular margin between the classes slightly improves the overall performance,
but not significantly, often leading to very similar results. The most likely
reason is that by increasing intra-class similarity implicitly introduces a
margin between different classes. Still, using an angular margin loss does not
have any drawbacks over using a compactness and a descriptiveness loss. As a
last observation, the sub-cluster AdaCos loss performs slightly better than
the AdaCos loss on the development split of the DCASE2022 dataset while
yielding a similar performance on the other dataset splits. A possible
explanation that there are no significant improvements on the DCASE2023
datasets when using an angular margin loss is that the auxiliary
classification task is not as difficult as for the DCASE2022 dataset because
there is only one section for each machine type. Slight improvements in
performance when using multiple sub-clusters for the AdaCos loss have been
observed on the DCASE2020 dataset [2] in [29]. In conclusion, an angular
margin loss for ASD in combination with an auxiliary classification task that
uses as many meaningful classes as possible is an excellent choice when
training an ASD system based on audio embeddings.

In the previous paragraph, we made the assumption that the noise is not class-
specific. However, if there is a single class with very specific noise that is
only present for this particular class or, even worse, if this is the case for
all classes, then an auxiliary classification task will very likely not
improve the results. The reason is that the model does not learn to closely
monitor the machine sound because also the background noise contains useful
information for discriminating between the classes. Therefore, assuming that
the noise is not class-specific is essential and intuitively makes sense for
machine condition monitoring as one would expect that at least some machines
share the same noise distribution when running in the same factory or acoustic
environment. Moreover, as shown in Theorem thm:ang_comp, minimizing an angular
margin loss using an auxiliary classification task also explicitly increases
intra-class similarity. Hence, even if the noise is class-specific and thus
the auxiliary classification task does not aid the ASD task, the performance
is still as least as good as when not using a classification task at all but
only minimizing the intra-class compactness losses and there should not be a
disadvantage.

###  IV-D Minimizing Compactness Loss by Minimizing an Angular Margin Loss

In Theorem 6, it has been shown that minimizing an angular margin loss also
minimizes all IC compactness losses and maximizes all inter-class compactness
losses. This fact is now verified experimentally by training a model using the
sub-cluster AdaCos loss while also monitoring all compactness losses. The
results are depicted in Fig. 2 and Fig. 3.

$20$$40$$60$$80$$100$0.1250.250.5124epoch loss train splits of development and
evaluation setsub-cluster AdaCos lossmean of intra-class compactness
lossesmean of inter-class compactness
losses$20$$40$$60$$80$$100$0.1250.250.5124epoch test split of development set
(normal data)

Figure 2: Different losses after each epoch when training by minimizing sub-
cluster AdaCos with a single sub-cluster per class and using mixup.

$20$$40$$60$$80$$100$0.1250.250.5124epoch loss train splits of development and
evaluation setAdaCos lossmean of intra-class compactness lossesmean of inter-
class compactness losses$20$$40$$60$$80$$100$0.1250.250.5124epoch test split
of development set (normal data)

Figure 3: Different losses after each epoch when training by minimizing AdaCos
and not using mixup.

Regardless of the dataset splits and regardless of using or not using mixup,
the angular margin loss and the mean of the IC compactness losses are
decreasing during training. The mean of the inter-class compactness loss is
constantly equal to $2$, even without training. The reason is that all sub-
cluster centers in this work are constant, randomly initialized and projected
to the unit sphere. Hence, By Lemma 5, a squared Euclidean distance of $2$
corresponds to an angle of $\frac{\pi}{2}$, i.e. orthogonality. The most
likely reason is that the randomly initialized center vectors are
approximately orthogonal with very high probability because of the high
dimension $D=256$ of the embedding space. Thus, samples that are similar to
the center of one class will be approximately orthogonal to the centers of the
other classes. Overall, this is exactly the expected behavior as predicted by
Theorem 6 and therefore verifies the theoretical results. Note that smaller
loss values do not correspond to a better ASD performance because minimizing
these losses only optimizes the performance for the auxiliary task, which is
not the same as the ASD task.

###  IV-E Visualizing Normal and Anomalous Regions in Input Representations as
Perceived by the System

(a) Spectrogram of an anomalous gearbox sound.

(b) Importance map of an anomalous gearbox sound when using sub-cluster
AdaCos.

(c) Importance map of an anomalous gearbox sound when using compactness loss.

(d) Spectrogram of a normal valve sound.

(e) Importance map of a normal valve sound when using sub-cluster AdaCos.

(f) Importance map of a normal valve sound when using compactness loss.

Figure 4: Log scaled Spectrograms (left column), importance maps obtained with
RISE when training with the sub-cluster AdaCos loss and classifying between
different machine types, sections and attribute information (middle column),
and importance maps obtained with RISE when training with an IC compactness
loss and no auxiliary classification task (right column) for two different
recordings belonging to the test split of the development set (rows). For the
importance maps, blue colors indicate normal regions and yellow colors
indicate regions that are found to be anomalous by the model. All subfigures
use individual color scales to improve visual appearance for differently
scaled importance maps and thus colors of different subfigures cannot be
compared to each other.

Figure 5: Visualizations of the test split of the development set in the
learned embedding space for different loss functions and auxiliary tasks using
t-SNE. Numbers in brackets denote the number of different classes used for the
auxiliary task.

To further investigate the effect of using an auxiliary task with multiple
classes, another experiment using RISE [73] is carried out. rise highlights
regions of the input representations that are considered normal or anomalous
by the ASD system. Our goal is to show that utilizing an auxiliary
classification task for training the system, as done when minimizing an
angular margin loss, enables the system to closely monitor specific machine
sounds by focusing on regions belonging to specific patterns of the input
data. Although the ASD performance is worse when only using spectrograms as
input representations [33], for these experiments a model using only
spectrograms as input has been trained. The reason is that these
representations are visually more appealing for the human eye than waveforms
or spectra and thus more suitable to visually highlight normal and anomalous
regions.

To visualize areas of the input representation responsible for a decision,
RISE masks random entries of the spectrograms using binary masks and evaluates
the ASD score using the masked spectrogram. This step is repeated for many
iterations. Then, the sum of the masks weighted with the corresponding ASD
scores is taken and normalized with the expected value of a random binary
mask, which depends on the chosen sampling distribution. The result is called
an _importance map_ and visualizes the impact of specific regions of a
spectrogram on the resulting anomaly score.

The problem is that the dimension of the spectrograms is very high because a
time dimension of $T=311$ and a frequency dimension of $F=513$ is used. Thus,
there are $2^{T\cdot F}=2^{159543}$ possible binary masks and thus RISE
requires clearly too many iterations. To significantly reduce the search space
from $2^{F\cdot T}$ to $2^{F+T}$, individual time and frequency masks are
randomly generated with a probability of $0.25$ for a time step or frequency
bin to be masked and both masks are combined by element-wise multiplication.
This restriction is not too severe because most sounds emitted by machines are
relatively stable over time with specific frequencies (e.g. fans), consist of
multiple stable sound events with on- and offsets (e.g. slide rails) or only
consist of short sound events over a wide frequency range with a specific
temporal structure (e.g. valves). For further reduction of the search space,
small binary masks are generated and then up-sampled and randomly cropped to
match the dimension of the spectrogram to be masked as proposed in [73]. More
concretely, we used time masks of size $20$ and frequency masks of size $34$
resulting in a search space of $2^{54}$, which is still very large but much
smaller than before. For generating a single importance map, $640,000$
iterations have been used.

Magnitude spectrograms (visualized in log scale) and corresponding importance
maps belonging to two different samples using i) a model trained with an IC
compactness loss without an auxiliary task, and ii) a model trained with the
sub-cluster AdaCos loss and an auxiliary task for classifying between
different machine types, sections and attribute information are depicted in
Fig. 4. For the depicted importance maps, blue colors indicate normal regions
and yellow colors indicate anomalous regions as perceived by the system. Note
that, since the system does not yield perfect results, these regions do not
need to really belong to normal and anomalous regions. As there are only
binary labels, indicating normal or anomalous samples, available for each
entire audio recording and we are no subject matter experts for machine
condition monitoring, we do not know which regions are normal or anomalous.
Still, for the purpose of showing that utilizing meta information when
training a model, as done by angular margin losses, helps the system to have a
better understanding of the structure of the data these plots are sufficient.
There are several observations to be made. Comparing the representations
depicted in Fig. 4b and 4e with the ones depicted in Fig. 4c and 4f, we
suggest that using sub-cluster AdaCos, i.e. Fig. 4b and 4e, more clearly shows
time and frequency structures at a resolution correlating with the structures
resp. acoustic events visible in the spectrograms depicted in Fig. 4a and 4d.

For the anomalous gearbox example (Fig. 4a), the importance map depicted in
Fig. 4b shows that specific frequencies are monitored and considered to be
normal or anomalous. Interestingly, the normal frequency regions (in blue) in
Fig. 4b exactly correspond to the frequencies containing high energy (Fig. 4a)
showing that the model expects a gearbox sound from this section to have high
energy in these regions. The frequencies that are considered most anomalous,
which mostly corresponds to the frequency range between the bottom two normal
frequency bands, only contain some energy. This indicates that a normal
machine sound should either contain no energy or much more energy for these
frequencies. In contrast to this, the importance map depicted in Fig. 4c does
not monitor specific frequencies and the only clearly visible structures are
two vertical lines indicating anomalous regions (in yellow). Although we
cannot guarantee that the regions in the spectrogram corresponding to these
vertical lines are not anomalous, at least visually there is no energy present
in these locations. Since the recordings of the machine sounds do not start
and end at the same fixed time steps, it does not make sense that the model
expects temporal patterns at exactly these time steps that are missing and to
thus consider such patterns to be anomalous. Therefore, it seems that these
structures are errors of the model.

The importance maps belonging to the normal valve example (Fig. 4d) show a
similar behavior but for temporal patterns in addition to specific
frequencies. Here, the main four normal vertical patterns in the importance
map shown in Fig. 4e correspond the four high energy patterns of the
spectrogram showing that the system views these temporal patterns as normal
for a valve sound. In contrast, the importance map depicted in Fig. 4f does
not show that the system has learned to detect these patterns and looks almost
random.

Overall, the depicted results add further confidence to the claim that
training a model with an auxiliary classification task with many classes
enables the model to learn much more meaningful embeddings, also leading to
much better capabilities for detecting anomalous sound events than a model
trained with only a single class.

###  IV-F Visualizing the Resulting Embedding Spaces Using t-SNE

TABLE III: Mean and standard deviation of the average Euclidean distance
between the t-SNE projections of each anomalous sample and the closest normal
sample over five trials for different losses and using different auxiliary
tasks.

loss classes of auxiliary task (number of classes) average distance IC
compactness loss none ($1$) $0.485\pm 0.007$ IC compactness loss machine types
($7$) $1.636\pm 0.037$ IC compactness loss machine types and sections ($42$)
$2.175\pm 0.075$ IC compactness loss machine types and sections, models
trained individually ($1$) $0.559\pm 0.002$ IC compactness loss machine types,
sections and attribute information ($342$) $2.646\pm 0.045$ sub-cluster AdaCos
loss machine types, sections and attribute information ($342$) $2.947\pm
0.022$

As a last experiment, the embedding spaces resulting from using different loss
functions and auxiliary tasks are visualized in Figure 5 using t-SNE [74].
Note that by Lemma 5 it does not matter whether t-SNE is evaluated with the
cosine distance or the Euclidean distance because both are equivalent when
determining the degree of similarity between samples on the unit sphere. It
can be seen that using more classes for the auxiliary task helps to separate
normal and anomalous samples (Fig. 5b,c,e,f). When only using a single class
(Fig. 5a) or individually trained models (Fig. 5d), there is no visual
difference between normal and anomalous samples. However, it can also be seen
that the model has not learned a trivial solution as the embedding spaces did
not collapse to a single fixed point, which would correspond to a uniformly
distributed t-SNE embedding space. Moreover, the ASD performance would be very
close to $50\%$ as normal and anomalous samples would be indistinguishable in
the embedding space. Both is not the case. Therefore, the applied
regularization strategies, namely not using trainable centers and not using
bias terms, work and a completely failed regularization is not the main
underlying problem. These visual impressions are verified by computing the
average Euclidean distance between each anomalous sample and the closest
normal sample in the t-SNE embedding space. The results can be found in Tab.
III and also agree with the performance results shown in Table II. Note that
the distance in the original embedding space is implicitly captured by the ASD
performance given in II because the anomaly score is computed by taking the
distance to the closest normal sample in the target domain and the closest
mean in the source domain. Again, the most likely explanation for the strong
differences between the embedding spaces in terms of ASD capabilities is that
using multiple classes enables the model to focus less on or even ignore the
background noise and isolate the targeted machine sounds. This helps the model
to more robustly detect deviations from normal machine sounds despite the
acoustically noisy recording conditions and thus results in better ASD
performance.

##  V Conclusions

In this work, it has been investigated why using angular margin losses works
well for semi-supervised ASD. To this end, it has been shown, both
theoretically and experimentally, that reducing an angular margin loss also
minimizes the IC compactness loss while simultaneously maximizing the inter-
class compactness loss. Therefore, angular margin losses in combination with
an auxiliary classification task can be viewed as regularized one-class losses
preventing the model to learn trivial solutions. In experiments conducted on
the DCASE 2022 ASD dataset for machine condition monitoring, it has been shown
that using an auxiliary task with as many meaningful classes as possible and
using an angular margin loss leads to significantly better ASD performance
than using a one-class loss such as the IC compactness loss. Furthermore, RISE
has been applied to create importance maps for different losses and t-SNE has
been used to visualize the resulting embedding spaces. All the conducted
experiments show that by using an angular margin the model used for extracting
the embeddings learns to monitor relevant frequency bins and learns machine-
specific temporal patterns. This enables the model to isolate machine sounds
and effectively ignore background noise present in the recording explaining
why angular margin losses with an auxiliary task are a good choice for
training an ASD system.

For future work, is is planned to investigate whether using auxiliary tasks
based on self-supervised learning to obtain suitable representations of the
data improves the resulting ASD performance. In addition, sophisticated
methods for visualizing anomalous regions of input representations should be
developed as being able to localize these regions is very useful for practical
applications and theoretical analysis of ASD systems.

## Acknowledgments

We would like to thank Paul M. Baggenstoss and Lukas Henneke as well as the
anonymous reviewers for their valuable comments that improved the quality of
this work.

## Proof of Lemma 5

Using only basic definitions, we obtain

$\displaystyle\lVert x-y\rVert_{2}^{2}$
$\displaystyle=\sum_{i=1}^{D}(x_{i}-y_{i})^{2}$
$\displaystyle=\sum_{i=1}^{D}x_{i}^{2}+\sum_{i=1}^{D}y_{i}^{2}-2\sum_{i=1}^{D}x_{i}y_{i}$
$\displaystyle=\lVert x\rVert_{2}^{2}+\lVert y\rVert_{2}^{2}-2\langle
x,y\rangle$ $\displaystyle=2\bigg{(}1-\frac{\langle x,y\rangle}{\lVert
x\rVert_{2}\lVert y\rVert_{2}}\bigg{)}$ $\displaystyle=2(1-\cos(x,y)),$

which finishes the proof. ∎

## References

  * [1] C. Aggarwal, _Outlier Analysis_ , 2nd ed. Springer, 2017.
  * [2] Y. Koizumi, Y. Kawaguchi, K. Imoto, T. Nakamura, Y. Nikaido, R. Tanabe, H. Purohit, K. Suefusa, T. Endo, M. Yasuda, and N. Harada, “Description and discussion on DCASE2020 challenge task2: Unsupervised anomalous sound detection for machine condition monitoring,” in _Detection and Classification of Acoustic Scenes and Events Workshop (DCASE)_ , 2020, pp. 81–85.
  * [3] Y. Kawaguchi, K. Imoto, Y. Koizumi, N. Harada, D. Niizumi, K. Dohi, R. Tanabe, H. Purohit, and T. Endo, “Description and discussion on DCASE 2021 challenge task 2: Unsupervised anomalous detection for machine condition monitoring under domain shifted conditions,” in _Detection and Classification of Acoustic Scenes and Events Workshop (DCASE)_ , 2021, pp. 186–190.
  * [4] K. Dohi, K. Imoto, N. Harada, D. Niizumi, Y. Koizumi, T. Nishida, H. Purohit, R. Tanabe, T. Endo, M. Yamamoto, and Y. Kawaguchi, “Description and discussion on DCASE 2022 challenge task 2: Unsupervised anomalous sound detection for machine condition monitoring applying domain generalization techniques,” in _7th Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE)_. Tampere University, 2022, pp. 26–30.
  * [5] S. N. Murthy and E. Agu, “Deep learning anomaly detection methods to passively detect COVID-19 from audio,” in _International Conference on Digital Health (ICDH)_. IEEE, 2021, pp. 114–121.
  * [6] T. Dissanayake, T. Fernando, S. Denman, S. Sridharan, H. Ghaemmaghami, and C. Fookes, “A robust interpretable deep learning classifier for heart anomaly detection without segmentation,” _IEEE J. Biomed. Health Informatics_ , vol. 25, no. 6, pp. 2162–2171, 2021.
  * [7] S. Ntalampiras and I. Potamitis, “Acoustic detection of unknown bird species and individuals,” _CAAI Transactions on Intelligence Technology_ , vol. 6, no. 3, pp. 291–300, 2021.
  * [8] T. Cejrowski and J. Szymanski, “Buzz-based honeybee colony fingerprint,” _Comput. Electron. Agric._ , vol. 191, p. 106489, 2021.
  * [9] C. Zieger, A. Brutti, and P. Svaizer, “Acoustic based surveillance system for intrusion detection,” in _Sixth International Conference on Advanced Video and Signal Based Surveillance (AVSS)_. IEEE, 2009, pp. 314–319.
  * [10] P. Foggia, N. Petkov, A. Saggese, N. Strisciuglio, and M. Vento, “Audio surveillance of roads: A system for detecting anomalous sounds,” _IEEE Trans. Intell. Transp. Syst._ , vol. 17, no. 1, pp. 279–288, 2016\.
  * [11] Y. Li, X. Li, Y. Zhang, M. Liu, and W. Wang, “Anomalous sound detection using deep audio representation and a BLSTM network for audio surveillance of roads,” _IEEE Access_ , vol. 6, pp. 58 043–58 055, 2018.
  * [12] G. Valenzise, L. Gerosa, M. Tagliasacchi, F. Antonacci, and A. Sarti, “Scream and gunshot detection and localization for audio-surveillance systems,” in _Fourth International Conference on Advanced Video and Signal Based Surveillance (AVSS)_. IEEE, 2007, pp. 21–26.
  * [13] T. Hayashi, T. Komatsu, R. Kondo, T. Toda, and K. Takeda, “Anomalous sound event detection based on wavenet,” in _26th European Signal Processing Conference (EUSIPCO)_. IEEE, 2018, pp. 2494–2498.
  * [14] S. Shon, N. Dehak, D. A. Reynolds, and J. R. Glass, “MCE 2018: The 1st multi-target speaker detection and identification challenge evaluation,” in _20th Annual Conference of the International Speech Communication Association (INTERSPEECH)_. ISCA, 2019, pp. 356–360.
  * [15] A. Mesaros, T. Heittola, and T. Virtanen, “Acoustic scene classification in DCASE 2019 challenge: Closed and open set classification and data mismatch setups,” in _Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE)_ , 2019, pp. 164–168.
  * [16] J. Naranjo-Alcazar, S. Perez-Castanos, P. Zuccarello, A. M. Torres, J. J. Lopez, F. J. Ferri, and M. Cobos, “An open-set recognition and few-shot learning dataset for audio event classification in domestic environments,” _Pattern Recognition Letters_ , 2022.
  * [17] E. Marchi, F. Vesperini, S. Squartini, and B. W. Schuller, “Deep recurrent neural network-based autoencoders for acoustic novelty detection,” _Comput. Intell. Neurosci._ , vol. 2017, pp. 4 694 860:1–4 694 860:14, 2017.
  * [18] Y. Koizumi, S. Saito, H. Uematsu, Y. Kawachi, and N. Harada, “Unsupervised detection of anomalous sound based on deep learning and the neyman-pearson lemma,” _IEEE ACM Trans. Audio Speech Lang. Process._ , vol. 27, no. 1, pp. 212–224, 2019.
  * [19] K. Suefusa, T. Nishida, H. Purohit, R. Tanabe, T. Endo, and Y. Kawaguchi, “Anomalous sound detection based on interpolation deep neural network,” in _International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 271–275.
  * [20] R. Giri, F. Cheng, K. Helwani, S. V. Tenneti, U. Isik, and A. Krishnaswamy, “Group masked autoencoder based density estimator for audio anomaly detection,” in _5th the Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE)_ , 2020, pp. 51–55.
  * [21] S. Kapka, “Id-conditioned auto-encoder for unsupervised anomaly detection,” in _5th the Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE)_ , 2020, pp. 71–75.
  * [22] G. Wichern, A. Chakrabarty, Z. Wang, and J. L. Roux, “Anomalous sound detection using attentive neural processes,” in _Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA)_. IEEE, 2021, pp. 186–190.
  * [23] K. Dohi, T. Endo, H. Purohit, R. Tanabe, and Y. Kawaguchi, “Flow-based self-supervised density estimation for anomalous sound detection,” in _International Conference on Acoustics, Speech and Signal Processing, ICASSP_. IEEE, 2021, pp. 336–340.
  * [24] K. Dohi, T. Endo, and Y. Kawaguchi, “Disentangling physical parameters for anomalous sound detection under domain shifts,” in _30th European Signal Processing Conference EUSIPCO_. IEEE, 2022, pp. 279–283.
  * [25] R. Giri, S. V. Tenneti, F. Cheng, K. Helwani, U. Isik, and A. Krishnaswamy, “Self-supervised classification for detecting anomalous sounds,” in _5th Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE)_ , 2020, pp. 46–50.
  * [26] J. A. Lopez, H. Lu, P. Lopez-Meyer, L. Nachman, G. Stemmer, and J. Huang, “A speaker recognition approach to anomaly detection,” in _5th the Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE)_ , 2020, pp. 96–99.
  * [27] T. Inoue, P. Vinayavekhin, S. Morikuni, S. Wang, T. H. Trong, D. Wood, M. Tatsubori, and R. Tachibana, “Detection of anomalous sounds for machine condition monitoring using classification confidence,” in _5th the Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE)_ , 2020, pp. 66–70.
  * [28] Q. Zhou, “ArcFace based sound mobilenets for DCASE 2020 task 2,” DCASE2020 Challenge, Tech. Rep., 2020.
  * [29] K. Wilkinghoff, “Sub-cluster AdaCos: Learning representations for anomalous sound detection,” in _International Joint Conference on Neural Networks (IJCNN)_. IEEE, 2021.
  * [30] ——, “Combining multiple distributions based on sub-cluster AdaCos for anomalous sound detection under domain shifted conditions,” in _Detection and Classification of Acoustic Scenes and Events Workshop (DCASE)_ , 2021, pp. 55–59.
  * [31] S. Venkatesh, G. Wichern, A. S. Subramanian, and J. L. Roux, “Improved domain generalization via disentangled multi-task learning in unsupervised anomalous sound detection,” in _7th Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE)_. Tampere University, 2022, pp. 196–200.
  * [32] T. Nishida, K. Dohi, T. Endo, M. Yamamoto, and Y. Kawaguchi, “Anomalous sound detection based on machine activity detection,” in _30th European Signal Processing Conference EUSIPCO_. IEEE, 2022, pp. 269–273.
  * [33] K. Wilkinghoff, “Design choices for learning embeddings from auxiliary tasks for domain generalization in anomalous sound detection,” in _International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2023.
  * [34] D. Hendrycks, M. Mazeika, and T. G. Dietterich, “Deep anomaly detection with outlier exposure,” in _7th International Conference on Learning Representations (ICLR)_. OpenReview.net, 2019.
  * [35] P. Primus, V. Haunschmid, P. Praher, and G. Widmer, “Anomalous sound detection as a simple binary classification problem with careful selection of proxy outlier examples,” in _5th the Workshop on Detection and Classification of Acoustic Scenes and Events 2020 (DCASE)_ , 2020, pp. 170–174.
  * [36] W. Liu, Y. Wen, Z. Yu, M. Li, B. Raj, and L. Song, “Sphereface: Deep hypersphere embedding for face recognition,” in _Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE Computer Society, 2017, pp. 6738–6746.
  * [37] H. Wang, Y. Wang, Z. Zhou, X. Ji, D. Gong, J. Zhou, Z. Li, and W. Liu, “Cosface: Large margin cosine loss for deep face recognition,” in _Conference on Computer Vision and Pattern Recognition (CVPR)_. Computer Vision Foundation / IEEE Computer Society, 2018, pp. 5265–5274.
  * [38] J. Deng, J. Guo, N. Xue, and S. Zafeiriou, “ArcFace: Additive angular margin loss for deep face recognition,” in _Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE, 2019, pp. 4690–4699.
  * [39] S. Grollmisch, D. Johnson, J. Abeßer, and H. Lukashevich, “IAEO3-combining OpenL3 embeddings and interpolation autoencoder for anomalous sound detection,” DCASE2020 Challenge, Tech. Rep., 2020.
  * [40] K. Wilkinghoff, “Using look, listen, and learn embeddings for detecting anomalous sounds in machine condition monitoring,” in _Detection and Classification of Acoustic Scenes and Events Workshop (DCASE)_ , 2020, pp. 215–219.
  * [41] R. Müller, F. Ritz, S. Illium, and C. Linnhoff-Popien, “Acoustic anomaly detection for machine sounds based on image transfer learning,” in _13th International Conference on Agents and Artificial Intelligence (ICAART)_. SCITEPRESS, 2021, pp. 49–56.
  * [42] K. Wilkinghoff and F. Fritz, “On using pre-trained embeddings for detecting anomalous sounds with limited training data,” in _31st European Signal Processing Conference (EUSIPCO)_. IEEE, 2023.
  * [43] J. A. Lopez, G. Stemmer, P. Lopez-Meyer, P. Singh, J. A. del Hoyo Ontiveros, and H. A. Cordourier, “Ensemble of complementary anomaly detectors under domain shifted conditions,” in _6th Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE)_ , 2021, pp. 11–15.
  * [44] I. Kuroyanagi, T. Hayashi, Y. Adachi, T. Yoshimura, K. Takeda, and T. Toda, “An ensemble approach to anomalous sound detection based on conformer-based autoencoder and binary classifier incorporated with metric learning,” in _6th Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE)_ , 2021, pp. 110–114.
  * [45] Y. Deng, A. Jiang, Y. Duan, J. Ma, X. Chen, J. Liu, P. Fan, C. Lu, and W. Zhang, “Ensemble of multiple anomalous sound detectors,” in _7th Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE)_. Tampere University, 2022, pp. 21–25.
  * [46] A. Fernandez and M. D. Plumbley, “Using UMAP to inspect audio data for unsupervised anomaly detection under domain-shift conditions,” in _6th Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE)_ , 2021, pp. 165–169.
  * [47] A. Holzinger, A. Saranti, C. Molnar, P. Biecek, and W. Samek, “Explainable AI methods - A brief overview,” in _xxAI - Beyond Explainable AI \- International Workshop, Held in Conjunction with ICML 2020_ , ser. Lecture Notes in Computer Science, vol. 13200. Springer, 2020, pp. 13–38.
  * [48] K. T. Mai, T. Davies, L. D. Griffin, and E. Benetos, “Explaining the decision of anomalous sound detectors,” in _7th Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE)_. Tampere University, 2022.
  * [49] M. T. Ribeiro, S. Singh, and C. Guestrin, “”Why should I trust you?”: Explaining the predictions of any classifier,” in _22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_. ACM, 2016, pp. 1135–1144.
  * [50] S. Mishra, B. L. Sturm, and S. Dixon, “Local interpretable model-agnostic explanations for music content analysis,” in _18th International Society for Music Information Retrieval Conference (ISMIR)_ , 2017, pp. 537–543.
  * [51] L. McInnes, J. Healy, and J. Melville, “UMAP: Uniform manifold approximation and projection for dimension reduction,” 2018. [Online]. Available: https://arxiv.org/abs/1802.03426
  * [52] L. Ruff, N. Görnitz, L. Deecke, S. A. Siddiqui, R. A. Vandermeulen, A. Binder, E. Müller, and M. Kloft, “Deep one-class classification,” in _35th International Conference on Machine Learning (ICML)_ , ser. Proceedings of Machine Learning Research, vol. 80. PMLR, 2018, pp. 4390–4399.
  * [53] G. E. Hinton and R. Salakhutdinov, “Reducing the dimensionality of data with neural networks,” _Science_ , vol. 313, no. 5786, pp. 504–507, 2006.
  * [54] P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio, and P. Manzagol, “Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion,” _J. Mach. Learn. Res._ , vol. 11, pp. 3371–3408, 2010.
  * [55] P. Perera and V. M. Patel, “Learning deep features for one-class classification,” _IEEE Transactions on Image Processing_ , vol. 28, no. 11, pp. 5450–5463, 2019.
  * [56] A. N. Gorban, I. Y. Tyukin, D. V. Prokhorov, and K. I. Sofeikov, “Approximation with random bases: Pro et contra,” _Information Sciences_ , vol. 364-365, pp. 129–145, 2016.
  * [57] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in _32nd International Conference on Machine Learning (ICML)_ , vol. 37, 2015, pp. 448–456.
  * [58] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “Mixup: Beyond empirical risk minimization,” in _6th International Conference on Learning Representations (ICLR)_ , 2018.
  * [59] X. Zhang, R. Zhao, Y. Qiao, X. Wang, and H. Li, “AdaCos: Adaptively scaling cosine logits for effectively learning deep face representations,” in _Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE, 2019, pp. 10 823–10 832.
  * [60] J. Snell, K. Swersky, and R. S. Zemel, “Prototypical networks for few-shot learning,” in _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems (NIPS)_ , 2017, pp. 4077–4087.
  * [61] J. S. Chung, J. Huh, S. Mun, M. Lee, H. Heo, S. Choe, C. Ham, S. Jung, B. Lee, and I. Han, “In defence of metric learning for speaker recognition,” in _21st Annual Conference of the International Speech Communication Association (Interspeech)_. ISCA, 2020, pp. 2977–2981.
  * [62] Y. Wang, Q. Yao, J. T. Kwok, and L. M. Ni, “Generalizing from a few examples: A survey on few-shot learning,” _ACM Comput. Surv._ , vol. 53, no. 3, pp. 63:1–63:34, 2021.
  * [63] K. Wilkinghoff and A. Cornaggia-Urrigshardt, “TACos: Learning temporally structured embeddings for few-shot keyword spotting with dynamic time warping,” 2023, arXiv:2305.10816.
  * [64] K. Dohi, T. Nishida, H. Purohit, R. Tanabe, T. Endo, M. Yamamoto, Y. Nikaido, and Y. Kawaguchi, “MIMII DG: Sound dataset for malfunctioning industrial machine investigation and inspection for domain generalization task,” in _DCASE_. Tampere University, 2022, pp. 31–35.
  * [65] N. Harada, D. Niizumi, D. Takeuchi, Y. Ohishi, M. Yasuda, and S. Saito, “ToyADMOS2: Another dataset of miniature-machine operating sounds for anomalous sound detection under domain shift conditions,” in _6th Detection and Classification of Acoustic Scenes and Events Workshop (DCASE)_. Tampere University, 2021, pp. 1–5.
  * [66] J. Wang, C. Lan, C. Liu, Y. Ouyang, and T. Qin, “Generalizing to unseen domains: A survey on domain generalization,” in _Thirtieth International Joint Conference on Artificial Intelligence (IJCAI)_. ijcai.org, 2021, pp. 4627–4635.
  * [67] K. Dohi, K. Imoto, N. Harada, D. Niizumi, Y. Koizumi, T. Nishida, H. Purohit, R. Tanabe, T. Endo, and Y. Kawaguchi, “Description and discussion on DCASE 2023 challenge task 2: First-shot unsupervised anomalous sound detection for machine condition monitoring,” in _8th Detection and Classification of Acoustic Scenes and Events Workshop (DCASE)_. Tampere University, 2023, pp. 31–35.
  * [68] N. Harada, D. Niizumi, D. Takeuchi, Y. Ohishi, and M. Yasuda, “First-shot anomaly detection for machine condition monitoring: A domain generalization baseline,” in _31st European Signal Processing Conference (EUSIPCO)_. IEEE, 2023.
  * [69] ——, “ToyADMOS2+: New toyadmos data and benchmark results of the first-shot anomalous sound event detection baseline,” in _8th Detection and Classification of Acoustic Scenes and Events Workshop (DCASE)_. Tampere University, 2023, pp. 41–45.
  * [70] D. K. McClish, “Analyzing a portion of the ROC curve,” _Medical decision making_ , vol. 9, no. 3, pp. 190–195, 1989.
  * [71] J. Ebbers, R. Haeb-Umbach, and R. Serizel, “Threshold independent evaluation of sound event detection scores,” in _International Conference on Acoustics, Speech and Signal Processing, (ICASSP)_. IEEE, 2022, pp. 1021–1025.
  * [72] M. Abadi _et al._ , “Tensorflow: A system for large-scale machine learning,” in _12th USENIX Symposium on Operating Systems Design and Implementation (OSDI)_ , 2016, pp. 265–283.
  * [73] V. Petsiuk, A. Das, and K. Saenko, “RISE: Randomized input sampling for explanation of black-box models,” in _British Machine Vision Conference (BMVC)_. BMVA Press, 2018, p. 151.
  * [74] L. Van der Maaten and G. Hinton, “Visualizing data using t-SNE,” _Journal of machine learning research_ , vol. 9, no. 11, pp. 2431––2456, 2008.

|  Kevin Wilkinghoff received his B.Sc. degree in Mathematics at the
University of Münster, Germany, and his M.Sc. degree in Computer Science at
the University of Bonn, Germany, in 2014 and 2017, respectively. Since 2017 he
is a research associate at Fraunhofer FKIE. Currently, he is working towards a
Ph.D. degree in Computer Science at the University of Bonn. His research
interests include anomaly detection, open-set classification and
representation learning for machine listening applications. In 2021, he
received the DCASE Best Paper Award.
---|---
|  Frank Kurth studied Computer Science and Mathematics at Bonn University,
Germany, where he recieved both a masters degree in Computer Science and the
degree of a doctor of natural sciences (Dr. rer. nat.) in 1997 and 1999,
respectively. From 1997-2007 he was with the Multimedia Signal Processing
group at Bonn University where he finished his Habilitation in Computer
Science in 2004 and was subsequently appointed apl. Professor in 2013. Since
2007 he is with Fraunhofer FKIE, Germany, where he currently heads a research
group focused on physical layer signal analysis in the area of communications.
His research interests include the application of pattern recognition and
machine learning techniques to audio, speech and communication signal
processing. Dr. Kurth has recieved the 2000 Dissertation Award of the German
Informatics Society (GI) and a 2000 Multimedia Award of the German Department
of Economy and Technology. He is co-author of more than 100 publications and
holds several patents. Dr. Kurth is a senior member of the IEEE.
---|---
  *[ASD]: anomalous sound detection
  *[IM]: inlier modeling
  *[OE]: outlier exposure
  *[xAI]: explainable artificial intelligence
  *[LIME]: local interpretable model-agnostic explanations
  *[SLIME]: sound   ()
  *[UMAP]: uniform manifold approximation and projection
  *[IC]: intra-class
  *[RISE]: randomized input sampling for explanation
  *[CCE]: categorical cross-entropy
  *[]:
  *[AUC]: area under the   () curve
  *[pAUC]: partial area under the  curve
  *[ROC]: receiver operating characteristic
  *[t-SNE]: t-distributed stochastic neighbor embedding