# Energy-Based Open-World Uncertainty Modeling
for Confidence Calibration

Yezhen Wang1 Bo Li111footnotemark: 1 Tong Che211footnotemark: 1 Kaiyang Zhou3
Ziwei Liu3 Dongsheng Li1

1Microsoft Research Asia 2MILA 3S-Lab, Nanyang Technological University Equal
contribution. Ordered by dice rolling. Correspondence to
<EMAIL_ADDRESS>
###### Abstract

Confidence calibration is of great importance to the reliability of decisions
made by machine learning systems. However, discriminative classifiers based on
deep neural networks are often criticized for producing overconfident
predictions that fail to reflect the true correctness likelihood of
classification accuracy. We argue that such an inability to model uncertainty
is mainly caused by the closed-world nature in softmax: a model trained by the
cross-entropy loss will be forced to classify input into one of $K$ pre-
defined categories with high probability. To address this problem, we for the
first time propose a novel $K$+1-way softmax formulation, which incorporates
the modeling of open-world uncertainty as the extra dimension. To unify the
learning of the original $K$-way classification task and the extra dimension
that models uncertainty, we 1) propose a novel energy-based objective
function, and moreover, 2) theoretically prove that optimizing such an
objective essentially forces the extra dimension to capture the marginal data
distribution. Extensive experiments show that our approach, Energy-based Open-
World Softmax (EOW-Softmax), is superior to existing state-of-the-art methods
in improving confidence calibration.

##  1 Introduction

Given the considerable success achieved so far by deep neural networks (DNNs),
one might be wondering if DNN-based systems can be readily deployed to solve
real-world problems. On the one hand, DNNs can achieve high accuracy if
trained with large-scale datasets [15]. But on the other hand, contemporary
DNNs are often criticized for producing overconfident predictions [14], which
fail to represent the true correctness likelihood of accuracy. This has raised
concerns over safety and reliability for using machine learning systems in
real-world scenarios. Having a confidence-calibrated system is critical. For
instance, in healthcare applications, the intelligence system should produce
low-confidence predictions when it is uncertain about the input—say they
differ significantly from the training data—so the decision-making process can
be transferred to human doctors for more accurate diagnosis and safer
handling. Research on confidence calibration for DNNs has received increasing
attention in recent years [14, 27, 19, 23, 27]. Since most classifiers are
based on softmax, a common practice to improve calibration is to insert a
temperature scaling parameter to the softmax function and adjust it in a
validation set [14]. Besides, methods like Smoothing labels [36, 28], which
essentially combines the one-hot ground-truth vector with a uniform
distribution, has also been shown effective in improving calibration.

Figure 1: Comparison between (a) the conventional softmax and (b) our proposed
Energy-based Open-World softmax (EOW-Softmax). Our new formulation introduces
an extra dimension to model uncertainty, which is supposed to produce high
scores when the input deviates from the training data distribution. In this
way, the original $K$ classification scores can be well calibrated.

However, most existing confidence calibration methods have overlooked the
underlying problem that causes neural network classifiers to generate
overconfident predictions, i.e. the inability to model _uncertainty_ in output
probabilities. We argue that the culprit for causing such a problem is the
closed-world nature in softmax [2, 34]. This is easy to understand: during
training the model is asked to classify input into one of $K$ pre-defined
categories with high probability (due to the cross-entropy loss), and as such,
the model has no choice but to assign one of the $K$ categories to any unseen
data, likely with high probability as well.

A potential countermeasure is to adopt a $K+1$-way formulation where the new
category can represent uncertainty about the input data. In this way, the $K$
classification scores might be better regularized, and hence better
calibrated. However, learning such a classifier is challenging as we do not
have access to those data with the $K+1$-th label, thus lacking supervision to
teach the network when to give low/high confidence. Furthermore, designing the
extra dimension is a non-trivial task as it is directly linked to the
formulation of the learning objective. It is also unclear how such a dimension
should be constructed, e.g., to design it as another logit produced by the
same network or an independent branch that regresses to uncertainty.

In this paper, we propose _Energy-based Open-World Softmax_ (EOW-Softmax), a
novel approach that introduces a $K+1$-way softmax based on energy functions
[25]. Specifically, the neural network classifier is designed to produce $K+1$
logits, where the first $K$ dimensions encode the scores for the original
$K$-way classification task, while the extra dimension aims to model open-
world uncertainty. See Figure 1 for a comparison between a model based on the
conventional softmax and that based on our EOW-Softmax. Besides, we resort to
an energy-based $K+1$-way classification objective function to unify the
learning of the $K$-way classification task and the uncertainty modeling.
Further more, we theoretically justify that optimizing the proposed objective
function essentially forces the summation of original $K$ softmax scores
($K+1$ scores in total) to be directly proportional to the marginal density
$p(x)$, hence explaining why our EOW-Softmax helps calibrate a model’s
confidence estimates.

The contributions of this paper are summarized as follows. 1) First, we
overcome the closed-world softmax problem by transforming the conventional
$K$-way softmax to a novel $K+1$-way formulation, where the extra dimension is
designed to model open-world uncertainty. 2) Second, a novel energy-based
objective function is developed to unify the learning of the original $K$-way
classification task and the uncertainty modeling. 3) A theoretical proof is
further provided to explain why our learning objective can help the network
capture uncertainty. 4) Finally, we conduct extensive experiments on standard
benchmark datasets to demonstrate that our method can lead to a better
calibrated model compared with other state-of-the-arts.

##  2 Related Works

Confidence Calibration With the emergence of deep learning technologies and
their wide successes, concerns over whether they are reliable to be deployed
in practice have also arisen. This is because researchers have found that
contemporary deep neural networks (DNNs) often produce overconfident
predictions [14], even on input images that are totally unrecognizable to
humans [32]. Many approaches for improving confidence calibration have been
developed. A widely used method is temperature scaling [14, 27, 19], which
inserts a scaling parameter to the softmax formulation (called ‘temperature’)
and adjusts it in a validation set with a goal to ‘soften’ the softmax
probabilities. Regularization methods, such as label smoothing [36] and Mixup
[37], have also been demonstrated effective in improving calibration. In
particular, label smoothing modifies the ground-truth labels by fusing them
with a uniform distribution, essentially forcing neural networks to produce
‘more flattened’ probabilities; whereas Mixup is a data augmentation method
that randomly mixes two instances at both the image and label space, with a
byproduct effect of improving calibration. Bayesian methods have also been
explored for calibration. For instance, Monte Carlo Dropout [7] applies
dropout in both training and testing; Deep Ensembles [23] uses as prediction
the output averaged over an ensemble of models. Adding adversarial
perturbations to the input has been found effective in smoothing the output
probabilities [23, 27]. In [26], a GAN model [11] is trained to generate out-
of-distribution (OOD) data and the classifier is encouraged to produce low-
confidence probabilities on these data. Such an idea has also been
investigated in [16] where adversarial perturbations are utilized to
synthesize OOD data. In [12], a Joint Energy-based Model (JEM) is proposed to
improve calibration by learning the joint distribution based on energy
functions [25]. A recent work [38] suggests that calibrating confidence across
multiple domains is beneficial to OOD generalization [42]. Studies on why
neural networks produce overconfident predictions have also been covered in
the literature. In [16], the authors suggest that ReLU neural networks are
essentially piecewise linear functions, thus explaining why OOD data can
easily cause softmax classifiers to generate highly confident output. In [30],
the authors identify that data variance and model curvature cause most
generative models to assign high density to OOD data. The authors in [34]
point out that the overconfidence issue is related to the closed-world
assumption in softmax, and design a distance-based one-vs-all (OvA) classifier
as the countermeasure.

Two works related to ours are JEM [12] and the OvA classifier [34]. Compared
with JEM, our approach is much easier to train because we only need to
optimize a _single_ classification objective to achieve both discriminative
classifier learning and generative modeling (see Sec. 3.3), while JEM has to
simultaneously optimize two separate objectives. Moreover, JEM has ignored the
closed-world softmax issue, which is addressed in this work with an augmented
softmax. Compared with the OvA classifier, our approach is significantly
different: we endow the classifier with the ability to model open-world
uncertainty, which is attributed to the extra dimension in softmax learned via
a novel energy-based objective function to capture the marginal data
distribution; in contrast, the OvA classifier converts the $K$-way
classification problem into multiple binary classification problems.

Energy-Based Models (EBMs) have been widely used in the area of generative
modeling [41, 8, 1, 4]. The basic idea in EBMs is to learn dependencies
between variables (e.g., images and labels) represented using energy
functions; and to assign low energies to correct configurations while give
high energies to incorrect ones [25]. However, training EBMs, especially on
high-dimensional data like images, has been notoriously hard due to sampling
issues [21, 13]. A widely used sampler is Stochastic Gradient Langevin
Dynamics (SGLD) [39], which injects noises to the parameter update and anneals
the stepsize during the course of training. Following prior work [33, 6, 12],
we also leverage SGLD to optimize our energy-based objective function.

Figure 2: Model architecture of our approach _EOW-Softmax_. The extra
dimension introduced in the augmented softmax (dashed) is learned using an
energy-based function to model open-world uncertainty, such that it can assign
high uncertainty scores to abnormal input far away from the training data
distribution, which in turn lower the classifier’s confidence on the original
$K$-way classification task. Note that the sampling in SGLD is performed on
the latent (feature) space rather than the input (image) space.

##  3 Methodology

According to [34], we argue that the culprit for causing the overconfidence
problem in most neural network classifiers’ output is the closed-world nature
in softmax. As a result, a model trained by the cross-entropy loss has to pick
one of $K$ pre-defined categories with high confidence. To overcome this
problem, we propose _Energy-based Open-World Softmax_ , or EOW-Softmax, to
regularize the $K$-way classification scores in such a way that allows their
confidence to be calibrated. The main idea in EOW-Softmax is to re-formulate
the original $K$-way classification task as a novel $K$+1-way classification
problem, where the extra dimension is designed to model _open-world
uncertainty_. To learn such a $K$+1-way classifier in an end-to-end manner, we
propose a novel energy-based objective function, which essentially forces the
extra dimension to be negatively correlated to the marginal data distribution.
In this way, the $K$ classification scores are automatically calibrated to be
less confident over input fallen beyond the training data distribution. See
Figure 2 for an overview of our model architecture. The rest of this section
are organized as follows. In Sec. 3.1, we provide a brief background on
energy-based models (EBMs), which are required by our approach to construct
the objective function. In Sec. 3.2, we discuss in detail the design of EOW-
Softmax. Sec. 3.3 gives a theoretical insight on why EOW-Softmax can help
calibrate a model’s confidence estimates.

###  3.1 A Brief Background on EBMs

The main building block in EBMs is an energy function
$E_{\theta}:\mathbb{R}^{D}\to\mathbb{R}$ (parameterized by $\theta$), which
aims to map a $D$-dimensional datapoint to a scalar.111When the input is an
image, $D$ can be understood as the length of the flattened tensor. The
learning is designed in such a way that $E_{\theta}$ can assign low energies
to observed configurations of variables while give high energies to unobserved
ones [25]. With $E_{\theta}$, any probability density $p(x)$ for
$x\in\mathbb{R}^{D}$ in an EBM can be written as

$p_{\theta}(x)=\frac{\exp(-E_{\theta}(x))}{Z(\theta)},$ (1)

where $Z(\theta)=\int_{x}\exp(-E_{\theta}(x))$ denotes the normalizing
constant, also known as the partition function. An EBM can be represented by
using any function as long as the function can generate a single scalar given
some input $x$. In this work, we assume $E_{\theta}$ is represented by a deep
neural network. People usually adopt gradient estimation to optimize an EBMs
[21, 13] and sample data from it by Markov Chain Monte Carlo (MCMC) methods
[9, 10, 18, 39].

###  3.2 Energy-Based Open-World Softmax

Open-World Softmax As discussed before, conventional softmax-based classifiers
lack the ability to model open-world uncertainty. To address this problem, we
design a neural network classifier to output probabilities on $K+1$ categories
(see Figure 2), with the $K$+1-th score representing open-world
uncertainty—the network should be able to produce high uncertainty scores to
abnormal input, which in turn can lower the confidence on the original $K$
categories’ prediction. Let $f_{\theta}:\mathbb{R}^{D}\to\mathbb{R}^{K+1}$ be
our neural network model (excluding the softmax layer), which produces $K+1$
logits, and $f_{\theta}(x)[i]$ the $i$-th logit given input $x$, with
$i\in\\{1,...,K,K+1\\}$. The output probabilities can then be obtained by
passing these $K+1$ logits to a softmax normalization layer, i.e.

$h_{\theta}(x)[i]=\frac{\exp(f_{\theta}(x)[i])}{\sum_{j=1}^{K+1}\exp(f_{\theta}(x)[j])},$
(2)

where $h_{\theta}$ is the combination of the neural network $f_{\theta}$ and
the softmax normalization layer.

Energy-Based Learning Objective Now the question is how to design a learning
objective that allows $h_{\theta}(x)[K+1]$ to encode uncertainty? Our idea
here is to associate the score of $h_{\theta}(x)[K+1]$ to the marginal data
distribution. Intuitively, when the input comes from within the training data
distribution $p(x)$, the model is supposed to be confident in its decision,
and therefore, $h_{\theta}(x)[K+1]$ should be low (conversely,
$\sum_{i=1}^{K}h_{\theta}(x)[i]$ should be high). If the input deviates from
the training data distribution, the model should become uncertain about
whether its decision is correct. In this case, $h_{\theta}(x)[K+1]$ should be
high to indicate a higher level of uncertainty, which naturally forces
$\sum_{i=1}^{K}h_{\theta}(x)[i]$ to stay low (due to the softmax
normalization). However, directly training $h_{\theta}(x)[K+1]$ to capture the
marginal distribution $p(x)$ (i.e. generative modeling) is difficult [30].
Instead, we propose a novel learning objective with the help of EBMs [25].
First, we define our energy function as

$E_{\theta}(x)=\log h_{\theta}(x)[K+1].$ (3)

Then, our energy-based objective function is defined as

$\min\limits_{\theta}\mathbb{E}_{p(x)}\bigg{[}-\log
h_{\theta}(x)[y]\bigg{]}+\lambda\mathbb{E}_{p_{\bar{\theta}}(x)}\bigg{[}-\log
h_{\theta}(x)[K+1]\bigg{]},$ (4)

where $\lambda>0$ is a hyper-parameter; the first term is the maximum log-
likelihood objective for the $K$-way classification task using the ground-
truth label $y$; the second term can also be seen as maximum log-likelihood
objective—for recognizing data sampled from $p_{\bar{\theta}}(x)$. Note that
$p_{\bar{\theta}}(x)$ denotes the model distribution with frozen parameters of
$p_{\theta}(x)$ of current iteration ($p_{\bar{\theta}}(x)$ will always be the
same as $p_{\theta}(x)$ but without the gradient calculation of $\theta$ since
parameter $\theta$ is frozen here and $\bar{\theta}$ should be regarded as a
constant in each updates). We will show later in Sec. 3.3 that optimizing Eq.
(4) can actually lead the summation of rest $K$ softmax scores of original
classes to be directly proportional to the marginal density $p(x)$, which in
turn can make the $K+1$-th softmax score be negatively correlated to $p(x)$.

SGLD-Based Optimization We approximate the expectation in the second term in
Eq. (4) using a sampler based on Stochastic Gradient Langevin Dynamics (SGLD)
[39]. Specifically, the SGLD sampling process follows

$z_{t+1}=z_{t}-\frac{\alpha}{2}\frac{\partial E_{\theta}(z_{t})}{\partial
z_{t}}+\sqrt{\alpha}\epsilon,\quad\epsilon\sim\mathcal{N}(0,I),$ (5)

where $t$ denotes the SGLD iteration, $\alpha$ the step-size, and $\epsilon$ a
random noise sampled from a normal distribution. In practice, $\alpha$ is
usually fixed as a constant.

Most SGLD-based methods draw samples from the image space. This forces the
information to flow through the entire neural network, which is
computationally expensive. Inspired by [4], we choose to draw samples from the
latent space (see Figure 2). Therefore, $z$ in Eq. (5) represents features
rather than an image. Such a design significantly accelerates the training
since the information only goes partially through the network model, which
allows much deeper architectures such as ResNet-101 [15] to fit into limited
resources. Moreover, the latent space is typically smoother than the image
space [3], which facilitates the estimate of gradients.

###  3.3 Theoretical Insight

In order to prove that our objective can force $h_{\theta}(x)[K+1]$ to be
negatively correlated with $p(x)$, i.e. representing uncertainty, we tend to
show that in theory, optimizing our objective in Eq. (4) is equivalent to
minimize the KL divergence between $p(x)$ and another EBM-modeled distribution
$q_{\theta}(x)$222It is worth noting that this $q_{\theta}(x)$ is not the
modeled EBMs $p_{\theta}(x)$ in Sec. 3.2., where $q_{\theta}(x)$ is defined by
energy function

$E^{\prime}_{\theta}(x)=-\log\sum\limits_{i=1}^{K}h_{\theta}(x)[i].$ (6)

To this end, we introduce an extra objective

$\min\limits_{\theta}\mathbb{E}_{p(x)}\bigg{[}-\log
h_{\theta}(x)[y]\bigg{]}+\mathbb{E}_{q_{\bar{\theta}}(x)}\bigg{[}\log\sum\limits_{i=1}^{K}h_{\theta}(x)[i]\bigg{]},$
(7)

Similar to Eq. (4), here $\bar{\theta}$ means the parameters are frozen. We
will show that optimizing Eq. (4) essentially optimize Eq. (7) and optimizing
Eq. (7) is an equivalent of $\min D_{KL}(p(x)||q_{\theta}(x))$ in following
Proposition 1 and Theorem 1, respectively.

######  Proposition 1.

Given two EBMs $p_{\bar{\theta}}(x)$ and $q_{\bar{\theta}}(x)$ with energy
functions defined in Eqs. (3) and (6) respectively, the optimization of Eq.
(4) is actually equivalent to optimize a combination of one $K$-way
classification objective and Eq. (7) with some suitable coefficient $\mu$.

###### Proof.

Since the first optimization term in Eq. (7) is identical with Eq. (4) as well
as the objective of maximum log-likelihood of $K$-way classification problems,
we only need to consider the second terms in both equations and prove that
they are equivalent to each other. Specifically, for the gradient of the
second term of Eq. (7), we have333The equality between Eqs. (9) and (10) holds
because
$q_{\bar{\theta}}(x)=q_{\theta}(x)=\frac{\sum\limits_{i=1}^{K}h_{\theta}(x)[i]}{Z^{\prime}(\theta)}$
and
$\frac{\partial}{\partial\theta}\log\sum\limits_{i=1}^{K}h_{\theta}(x)[i]=\frac{\frac{\partial}{\partial\theta}\sum\limits_{i=1}^{K}h_{\theta}(x)[i]}{\sum\limits_{i=1}^{K}h_{\theta}(x)[i]}$.

$\displaystyle\frac{\partial}{\partial\theta}\
\mathbb{E}_{q_{\bar{\theta}}(x)}\Big{[}\log\sum\limits_{i=1}^{K}h_{\theta}(x)[i]\Big{]}$
(8) $\displaystyle=\ $
$\displaystyle\int_{x}q_{\bar{\theta}}(x)\frac{\partial}{\partial\theta}\log\sum\limits_{i=1}^{K}h_{\theta}(x)[i]$
(9) $\displaystyle=\ $
$\displaystyle\int_{x}\frac{\sum\limits_{i=1}^{K}h_{\theta}(x)[i]}{Z^{\prime}(\theta)}\cdot\frac{\frac{\partial}{\partial\theta}\sum\limits_{i=1}^{K}h_{\theta}(x)[i]}{\sum\limits_{i=1}^{K}h_{\theta}(x)[i]}$
(10) $\displaystyle=\ $
$\displaystyle\frac{1}{Z^{\prime}(\theta)}\int_{x}\frac{\partial}{\partial\theta}\sum\limits_{i=1}^{K}h_{\theta}(x)[i]$
(11) $\displaystyle=\ $
$\displaystyle-\frac{1}{Z^{\prime}(\theta)}\int_{x}\frac{\partial}{\partial\theta}h_{\theta}(x)[K+1]$
(12) $\displaystyle=\ $
$\displaystyle-\frac{Z(\theta)}{Z^{\prime}(\theta)}\int_{x}\frac{h_{\theta}(x)[K+1]}{Z(\theta)}\cdot\frac{\frac{\partial}{\partial\theta}h_{\theta}(x)[K+1]}{h_{\theta}(x)[K+1]}$
(13) $\displaystyle=\ $
$\displaystyle-\frac{Z(\theta)}{Z^{\prime}(\theta)}\int_{x}p_{\bar{\theta}}(x)\frac{\partial}{\partial\theta}\log
h_{\theta}(x)[K+1],$ (14)

where $Z^{\prime}(\theta)$ and $Z(\theta)$ represents the partition functions
of $q_{\theta}(x)$ and $p_{\theta}(x)$ respectively. If we use $\mu$ to denote
$\frac{Z(\theta)}{Z^{\prime}(\theta)}$, we can restate the Eq. (14) as

$\mu\frac{\partial}{\partial\theta}\mathbb{E}_{p_{\bar{\theta}}(x)}\bigg{[}-\log
h_{\theta}(x)[K+1]\bigg{]},$

which is exactly the gradient of Eq. (4)’s second term. As a result, the
objective of Eq. (4) and of Eq. (7) are same under a suitable coefficient
$\mu$ in each iteration. Further more, remembering that the first term of Eq.
(4) is an equivalent of the objective of maximum log-likelihood of $K$-way
classification problems, we consequently conclude that optimizing our
objective of Eq. (4) essentially optimize Eq. (7) and a $K$-way classification
objective. ∎

According to Proposition 1, we know that our learning objective of Eq. (4) can
optimize the discriminative $K$-way classification objective and the
generative modeling objective in a unified discriminative objective, which has
never been explored in existing confidence calibration work.

Moreover, if we can further prove that optimizing Eq. (7) is de facto an
equivalent of minimizing the KL-divergence between $p(x)$ and the distribution
$q_{\theta}(x)$, then our objective of Eq. (4) could minimize this KL-
divergence either due to Proposition 1. To prove that, we need to recur to
Lemma 1 based on [21], which shows how to efficiently compute the gradient of
the KL divergence between the real distribution and an approximated
distribution modeled via EBMs.

######  Lemma 1.

Given a training dataset with data $\\{x\\}$ sampled from distribution $r(x)$,
and an energy model distribution $r_{\phi}(x)$ parameterized by $\phi$ and
associated to an energy function $E_{\phi}(x)$, the objective of minimizing
$D_{KL}(r(x)||r_{\phi}(x))$ can be optimized by descending the following
gradient w.r.t $\phi$,

$\mathop{\mathbb{E}}\limits_{x^{+}\sim r(x)}\Bigg{[}\frac{\partial
E_{\phi}(x^{+})}{\partial\phi}\Bigg{]}-\mathop{\mathbb{E}}\limits_{x^{-}\sim
r_{\phi}(x)}\Bigg{[}\frac{\partial E_{\phi}(x^{-})}{\partial\phi}\Bigg{]}.$
(15)

The proof can refer [21]. By descending the gradient in Eq. (15), the first
term decreases the energy of samples $x^{+}$ drawn from the data distribution,
while the second term increases the energy of samples $x^{-}$ drawn from the
energy model distribution. Based on Lemma 1, we can introduce following
theorem followed by a proof.

######  Theorem 1.

Let $p(x)$ denote the training data distribution, and $q_{\theta}(x)$ the
energy model distribution represented by the energy function
$E^{\prime}_{\theta}(x)$ defined in Eq. (6), we can achieve minimization of
$D_{KL}(p(x)||q_{\theta}(x))$ by optimizing Eq. (7).

###### Proof.

According to Lemma 1, the KL divergence between $p(x)$ and $q_{\theta}(x)$ can
be optimized by descending the gradient in Eq. (15). We can replace the need
of computing expectation over the parameterized density $q_{\theta}(x)$ in Eq.
(15) by fixing the parameters $\theta$, denoted by $q_{\bar{\theta}}(x)$
444only the numerical value of Eq. (15) matters in optimization process.. As
such, Eq. (15) is converted to

$\mathbb{E}_{p(x)}\Bigg{[}\frac{\partial
E^{\prime}_{\theta}(x)}{\partial\theta}\Bigg{]}-\mathbb{E}_{q_{\bar{\theta}}(x)}\Bigg{[}\frac{\partial
E^{\prime}_{\theta}(x)}{\partial\theta}\Bigg{]}.$ (16)

Now we can optimize $D_{KL}(p(x)||q_{\theta}(x))$ via an objective
$\mathbb{E}_{p(x)}\Big{[}E^{\prime}_{\theta}(x)\Big{]}-\mathbb{E}_{q_{\bar{\theta}}(x)}\Big{[}E^{\prime}_{\theta}(x)\Big{]}$
which holds a numerically same gradient with regard to $\theta$ as Eqs. (16)
and (15). Then we have555the inequality holds because of the fact $\forall
y\leq K,\ \ \sum\limits_{i=1}^{K}h_{\theta}(x)[i]\geq h_{\theta}(x)[y].$

$\displaystyle\mathbb{E}_{p(x)}\Bigg{[}E^{\prime}_{\theta}(x)\Bigg{]}-\mathbb{E}_{q_{\bar{\theta}}(x)}\Bigg{[}E^{\prime}_{\theta}(x)\Bigg{]}$
(17) $\displaystyle=$
$\displaystyle\mathbb{E}_{p(x)}\Bigg{[}-\log\sum\limits_{i=1}^{K}h_{\theta}(x)[i]\Bigg{]}+\mathbb{E}_{q_{\bar{\theta}}(x)}\Bigg{[}\log\sum\limits_{i=1}^{K}h_{\theta}(x)[i]\Bigg{]}$
$\displaystyle\leq$ $\displaystyle\mathbb{E}_{p(x)}\Bigg{[}-\log
h_{\theta}(x)[y]\Bigg{]}+\mathbb{E}_{q_{\bar{\theta}}(x)}\Bigg{[}\log\sum\limits_{i=1}^{K}h_{\theta}(x)[i]\Bigg{]}.$

Therefore, Eq. (7) is an upper bound of an equivalent variant of the KL-
divergence between $p(x)$ and $q_{\theta}(x)$. ∎

Combing Proposition 1 and Theorem 1, we can conclude that our objective of Eq.
(4) can minimize $D_{KL}(p(x)||q_{\theta}(x))$. Therefore, once our objective
converged, we obtain $p(x)\simeq
q_{\theta}(x)=\frac{\exp(-E^{\prime}_{\theta}(x))}{Z^{\prime}(\theta)}\propto\sum\limits_{i=1}^{K}h_{\theta}(x)[i]$,
which result in the summation of $K$ softmax scores of original classes to be
directly proportional to the marginal density $p(x)$ and in turn make the
$K+1$-th softmax score be negatively correlated to $p(x)$.

Table 1: Comparison between our approach EOW-Softmax and nine baselines on
four benchmark datasets. It is clear that our approach generally leads to a
better calibrated model than the baselines (lower ECE & NLL), while
maintaining the accuracy. $\uparrow$: the higher the better. $\downarrow$: the
lower the better.

Method | MNIST (MLP) | CIFAR10 (VGG11) | CIFAR100 (ResNet50) | Tiny-ImageNet (ResNet50)
---|---|---|---|---
Acc% $\uparrow$ |  ECE% $\downarrow$ |  NLL $\downarrow$ |  Acc% $\uparrow$ |  ECE% $\downarrow$ |  NLL $\downarrow$ |  Acc% $\uparrow$ |  ECE% $\downarrow$ |  NLL $\downarrow$ |  Acc% $\uparrow$ |  ECE% $\downarrow$ |  NLL $\downarrow$
Vanilla Training | 98.32 | 1.73 | 0.29 | 90.48 | 6.30 | 0.43 | 71.57 | 19.1 | 1.58 | 46.71 | 25.2 | 2.95
TrustScore [20] | 98.32 | 2.14 | 0.26 | 90.48 | 5.30 | 0.40 | 71.57 | 10.9 | 1.43 | 46.71 | 19.2 | 2.75
MC-Dropout [7] | 98.32 | 1.71 | 0.34 | 90.48 | 3.90 | 0.47 | 71.57 | 9.70 | 1.48 | 46.72 | 17.4 | 3.17
Label Smoothing [36] | 98.77 | 1.68 | 0.30 | 90.71 | 2.70 | 0.38 | 71.92 | 3.30 | 1.39 | 47.19 | 5.60 | 2.93
Mixup [37] | 98.83 | 1.74 | 0.24 | 90.59 | 3.30 | 0.37 | 71.85 | 2.90 | 1.44 | 46.89 | 6.80 | 2.66
JEM [12] | 97.23 | 1.56 | 0.21 | 90.36 | 3.30 | 0.34 | 70.28 | 2.46 | 1.31 | 45.97 | 5.42 | 2.47
OvA DM [34] | 96.67 | 1.78 | 0.27 | 89.56 | 3.55 | 0.37 | 70.11 | 3.58 | 1.40 | 45.55 | 4.22 | 2.50
Temperature Scaling [14] | 95.14 | 1.32 | 0.17 | 89.83 | 3.10 | 0.33 | 69.84 | 2.50 | 1.23 | 45.03 | 4.80 | 2.59
DBLE [40] | 98.69 | 0.97 | 0.12 | 90.92 | 1.50 | 0.29 | 71.03 | 1.10 | 1.09 | 46.45 | 3.60 | 2.38
EOW-Softmax (_ours_) | 98.91 | 0.88 | 0.15 | 90.24 | 1.57 | 0.25 | 71.33 | 1.08 | 1.03 | 46.97 | 3.45 | 2.22

Figure 3: Results on CIFAR-100-C. Each bar represents the mean ECE on 19
different corruptions, with the vertical line segment denoting the standard
deviation. In general, EOW-Softmax achieves the lowest ECE under five
different corruption levels.

##  4 Experiments

###  4.1 Experimental Setup

Settings We adopt three settings to evaluate our approach. 1) _Confidence
Calibration_ : This aims to evaluate the effectiveness of a method in
improving confidence calibration. Following [40], we use four datasets: MNIST
[24] (MLP), CIFAR-10 [22] (VGG11 [35]), CIFAR-100 [22] (ResNet50 [15]), and
Tiny-ImageNet [5] (ResNet50). Network architectures used for these datasets
are indicated in the parentheses. 2) _OOD Detection_ : A ResNet50 classifier
is trained on CIFAR-100 and tested on the combination of CIFAR-100’s test
split and an OOD dataset, i.e. CIFAR-10 and SVHN [31]. The goal for the
classifier is to assign low confidence to as many OOD samples as possible. The
accuracy is computed on predictions with confidence higher than a threshold.
3) _Robustness under Corruption_ : A classifier is trained on CIFAR-100. Its
calibration performance is evaluated on CIFAR-100-C [17], where the images are
perturbed by 19 different corruptions with five intensity levels.

Baselines We compare our approach with nine baseline methods: MC-Dropout [7],
Temperature Scaling [14], Mixup [37], Label Smoothing [36], TrustScore [20],
JEM [12], DBLE [40], and OvA DM [34].

Evaluation Metrics Following prior work [14], we use two metrics to assess how
well the confidence in a model’s predictions is calibrated, namely Expected
Calibration Error (ECE) [29] and Negative Log-Likelihood (NLL). The lower the
ECE/NLL, the better the calibration performance. Below we explain in detail
how these two metrics are calculated. ECE approximates the expectation of the
difference between accuracy and confidence (i.e. the likelihood on the
predicted label $\hat{y}$), which can reflect how well the prediction
confidence aligns with the true accuracy. Specifically, the confidence
estimates made on all test samples are partitioned into $L$ equally spaced
bins (following [14, 40, 34], $L=15$), and the difference between the average
confidence and accuracy within each bin $I_{l}$ is calculated,

$\displaystyle\text{ECE}=\sum\limits_{l=1}^{L}\frac{1}{N}|\sum\limits_{x\in
I_{l}}p(\hat{y}|x)-\sum\limits_{x\in I_{l}}1(\hat{y}=y)|,$ (18)

where $N$ denotes the total number of samples in the test set. NLL computes
the average negative log-likelihood on all test samples,

$\displaystyle\text{NLL}=-\frac{1}{N}\sum\limits_{m=1}^{N}\log
p(\hat{y}_{m}|x_{m}).$ (19)

Implementation Details We use the SGD optimizer with the learning rate of
1e-4, the momentum of 0.9, and the weight decay of 5e-4. The batch size is set
to 64. The number of epochs is 200. The learning rate is decayed by 0.1 at the
100-th and 150-th epoch, respectively. For SGLD, we use a constant step size
of 2 and a standard deviation of 1e-3 (see Eq (5)). The number of updates in
each SGLD round is set to 100. To ensure that the results are convincing, we
run each experiment 5 times with different random seeds and average their
results.

###  4.2 Main Results

Confidence Calibration We first evaluate the calibration performance on four
standard datasets, namely MNIST, CIFAR-10/100 and Tiny-ImageNet. The results
are shown in Table 1. In general, our approach EOW-Softmax obtains the best
overall calibration performance on most datasets. Comparing with Vanilla
Training, we observe that our EOW-Softmax achieves a similar test accuracy,
while significantly improves the calibration performance in terms of ECE,
especially on the three challenging datasets with natural
images—6.30%$\to$1.57% on CIFAR-10, 19.1%$\to$1.08% on CIFAR-100, and
25.2%$\to$3.45% on Tiny-ImageNet. These results strongly suggest that our
energy-based open-world uncertainty modeling has great potential for real-
world applications as it shows a good balance between test accuracy and
calibration performance. JEM and OvA DM are two related methods to ours. JEM
is based on joint distribution modeling while OvA DM transforms the
conventional softmax classifier to a distance-based one-vs-all classifier. The
comparison with these two methods shows that our approach is clearly better in
all metrics, which advocates our design of the $K$+1-way softmax for modeling
open-world uncertainty. Compared with the top-performing baselines, i.e.
Temperature Scaling and DBLE, our EOW-Softmax is highly competitive—it obtains
the best ECE on all datasets except CIFAR-10 where the performance is only
slightly worse than DBLE. Among these three methods, EOW-Softmax performs the
best in maintaining the original test accuracy, whereas Temperature Scaling
has to sacrifice the test accuracy in exchange for improvement on ECE. This is
because for fair comparison (all methods only have access to the training
set), Temperature Scaling has to separate a validation set out from the
original training set for tuning its scaling parameter, which reduces the
amount of training data.

Table 2: Test accuracy on the combination of in-distribution and OOD test set using ResNet50 trained on CIFAR-100. OOD dataset | Method | Probability threshold
---|---|---
0 | .25 | .5 | .75
CIFAR10 | Vanilla Training | 37.24 | 42.31 | 47.67 | 58.90
JEM [12] | 42.55 | 46.88 | 0.53 | 63.21
OvA DM [34] | 39.78 | 48.54 | 54.31 | 65.20
EOW-Softmax | 37.65 | 50.11 | 57.32 | 69.00
SVHN | Vanilla Training | 20.24 | 21.33 | 24.38 | 26.90
JEM [12] | 19.87 | 22.57 | 26.22 | 30.75
OvA DM [34] | 20.08 | 23.99 | 26.08 | 30.32
EOW-Softmax | 20.21 | 25.58 | 28.13 | 32.68

OOD Detection We follow [34] to simulate real-world scenarios by training a
classifier on CIFAR-100 and testing on the combination of CIFAR-100’s test set
and an OOD dataset (CIFAR-10/SVHN). The classifier is required to assign low
confidence to OOD samples such that their predictions can be rejected by a
pre-defined threshold. The results are reported in Table 2 where we compare
our approach with JEM and OvA DM (as these two methods are most related to
ours), as well as the vanilla training method. The probability threshold is
linearly increased from 0 to 0.75. Only predictions with confidence higher
than this threshold are kept. From the results, we observe that EOW-Softmax
outperforms all baselines with clear margins in different thresholds (except
0). This indicates that a classifier trained with EOW-Softmax suffers much
less from the overconfidence problem than the competitors, and is thus safer
to be deployed in practical applications.

Robustness under Corruption We also evaluate our approach on corrupted images,
i.e. CIFAR-100-C, and compare with JEM and OVA DM. Specifically, there are
five different intensities of corruption as defined in [17], each with 19
different corruption types. Under each intensity level of corruption, we test
a model’s ECE on images of all corruption types, and report their average
result and the standard deviation. The comparison is illustrated in Figure 3.
Overall, EOW-Softmax performs favorably against JEM and OVA DM, as well as the
vanilla training baseline. Though both EOW-Softmax and JEM are based on energy
models for generative modeling, JEM clearly has a larger variation in
performance among different corruption types. This is because JEM is more
difficult to train—it optimizes the log-likelihood of distribution
independently from the classifier learning. In contrast, the ‘generative
modeling’ in EOW-Softmax is seamlessly integrated into the $K$+1-way
classification task, which has a better training stability.

###  4.3 Ablation study

Hyper-parameter Recall that $\lambda$ in Eq. (4) balances between the standard
$K$-way classification loss and the energy-based loss for uncertainty
modeling. We experiment with different values (1, 0.1, and 0.01) on
CIFAR-10/100 to see the impact of this hyper-parameter. The results in Table 3
show that $\lambda=0.1$ leads to the best calibration performance (lowest ECE
values) on both datasets.

Where to Apply the SGLD Sampling? In our approach, SGLD sampling is applied to
the latent feature space rather than the pixel space as in most EBMs. To
justify this design, we experiment on CIFAR-100 with different variants of our
approach where the SGLD sampling is applied to different positions, including
the pixel space, stage-1, stage-2, and stage-3 (in the neural network). In
addition to the test accuracy and the ECE, we also report the training speed
(second per iteration) measured using a Tesla K80 GPU. Table 3 shows that
applying the SGLD sampling to the pixel space incurs huge computation
overhead, while shifting the sampling to the latent space significantly
improves the training speed without sacrificing the calibration performance.

Table 3: Ablation study on the impact of $\lambda$ in Eq. (4). Model | $\lambda$ | CIFAR10
---|---|---
Acc% $\uparrow$ |  ECE% $\downarrow$ |  NLL $\downarrow$
VGG11 | 1 | 89.81 | 2.11 | 0.28
0.1 | 90.24 | 1.57 | 0.25
0.01 | 90.11 | 3.37 | 0.36
Model | $\lambda$ | CIFAR100
Acc% $\uparrow$ |  ECE% $\downarrow$ |  NLL $\downarrow$
ResNet50 | 1 | 70.26 | 1.33 | 1.24
0.1 | 71.33 | 1.08 | 1.03
0.01 | 71.51 | 2.31 | 1.31
Table 4: Ablation on where to apply the SGLD sampling.  Position |  Acc% $\uparrow$ |  ECE% $\downarrow$ |  Second/Iter $\downarrow$
---|---|---|---
Pixel space | 73.94 | 1.89 | 25.15
Feature stage 1 | 73.21 | 2.09 | 10.31
Feature stage 2 | 73.48 | 1.73 | 5.21
Feature stage 3 | 73.05 | 2.17 | 2.37

##  5 Conclusion

This paper has addressed the closed-world problem in softmax that causes
neural network classifiers to produce overconfident predictions by introducing
an energy-based objective to model the open-world uncertainty.

Acknowledgement. This study is supported by NTU NAP and Microsoft Research Lab
Asia, and under the RIE2020 Industry Alignment Fund – Industry Collaboration
Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind
contribution from the industry partner(s).

## References

  * [1] Michael Arbel, Liang Zhou, and Arthur Gretton.  Generalized energy based models.  In ICLR, 2021.
  * [2] Abhijit Bendale and Terrance E Boult.  Towards open set deep networks.  In CVPR, 2016.
  * [3] Yoshua Bengio, Grégoire Mesnil, Yann N. Dauphin, and Salah Rifai.  Better mixing via deep representations.  CoRR, abs/1207.4404, 2012.
  * [4] Tong Che, Ruixiang Zhang, Jascha Sohl-Dickstein, Hugo Larochelle, Liam Paull, Yuan Cao, and Yoshua Bengio.  Your gan is secretly an energy-based model and you should use discriminator driven latent sampling.  arXiv preprint arXiv:2003.06060, 2020.
  * [5] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [6] Yilun Du and Igor Mordatch.  Implicit generation and modeling with energy based models.  In NeurIPS, 2019.
  * [7] Yarin Gal and Zoubin Ghahramani.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In international conference on machine learning, pages 1050–1059, 2016.
  * [8] Ruiqi Gao, Yang Song, Ben Poole, Ying Nian Wu, and Diederik P Kingma.  Learning energy-based models by diffusion recovery likelihood.  In ICLR, 2021.
  * [9] Charles J Geyer.  Markov chain monte carlo maximum likelihood.  1991\.
  * [10] Charles J Geyer.  Practical markov chain monte carlo.  Statistical science, pages 473–483, 1992.
  * [11] Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial networks.  In NeurIPS, 2014.
  * [12] Will Grathwohl, Kuan-Chieh Wang, Jörn-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky.  Your classifier is secretly an energy based model and you should treat it like one.  arXiv preprint arXiv:1912.03263, 2019.
  * [13] Will Sussman Grathwohl, Jacob Jin Kelly, Milad Hashemi, Mohammad Norouzi, Kevin Swersky, and David Duvenaud.  No {mcmc} for me: Amortized sampling for fast and stable training of energy-based models.  In ICLR, 2021.
  * [14] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger.  On calibration of modern neural networks.  In ICML, 2017.
  * [15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [16] Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf.  Why relu networks yield high-confidence predictions far away from the training data and how to mitigate the problem.  In CVPR, 2019.
  * [17] Dan Hendrycks and Thomas Dietterich.  Benchmarking neural network robustness to common corruptions and perturbations.  Proceedings of the International Conference on Learning Representations, 2019.
  * [18] Geoffrey E Hinton.  Training products of experts by minimizing contrastive divergence.  Neural computation, 14(8):1771–1800, 2002.
  * [19] Yen-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira.  Generalized odin: Detecting out-of-distribution image without learning from out-of-distribution data.  In CVPR, 2020.
  * [20] Heinrich Jiang, Been Kim, Melody Guan, and Maya Gupta.  To trust or not to trust a classifier.  In Advances in Neural Information Processing Systems, volume 31, 2018.
  * [21] Taesup Kim and Yoshua Bengio.  Deep directed generative models with energy-based probability estimation.  In ICLR-W, 2016.
  * [22] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * [23] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell.  Simple and scalable predictive uncertainty estimation using deep ensembles.  In NeurIPS, 2017.
  * [24] Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al.  Gradient-based learning applied to document recognition.  Proceedings of the IEEE, 86(11):2278–2324, 1998.
  * [25] Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and F Huang.  A tutorial on energy-based learning.  Predicting structured data, 1(0), 2006.
  * [26] Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin.  Training confidence-calibrated classifiers for detecting out-of-distribution samples.  In ICLR, 2018.
  * [27] Shiyu Liang, Yixuan Li, and Rayadurgam Srikant.  Enhancing the reliability of out-of-distribution image detection in neural networks.  In ICLR, 2018.
  * [28] Rafael Müller, Simon Kornblith, and Geoffrey Hinton.  When does label smoothing help?  In NeurIPS, 2019.
  * [29] Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht.  Obtaining well calibrated probabilities using bayesian binning.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
  * [30] Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan.  Do deep generative models know what they don’t know?  In ICLR, 2019.
  * [31] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng.  Reading digits in natural images with unsupervised feature learning.  In NeurIPS-W, 2011.
  * [32] Anh Mai Nguyen, Jason Yosinski, and Jeff Clune.  Deep neural networks are easily fooled: High confidence predictions for unrecognizable images.  CoRR, abs/1412.1897, 2014.
  * [33] Erik Nijkamp, Mitch Hill, Song-Chun Zhu, and Ying Nian Wu.  Learning non-convergent non-persistent short-run mcmc toward energy-based model.  arXiv preprint arXiv:1904.09770, 2019.
  * [34] Shreyas Padhy, Zachary Nado, Jie Ren, Jeremiah Liu, Jasper Snoek, and Balaji Lakshminarayanan.  Revisiting one-vs-all classifiers for predictive uncertainty and out-of-distribution detection in neural networks.  arXiv preprint arXiv:2007.05134, 2020.
  * [35] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [36] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818–2826, 2016.
  * [37] Sunil Thulasidasan, Gopinath Chennupati, Jeff Bilmes, Tanmoy Bhattacharya, and Sarah Michalak.  On mixup training: Improved calibration and predictive uncertainty for deep neural networks.  arXiv preprint arXiv:1905.11001, 2019.
  * [38] Yoav Wald, Amir Feder, Daniel Greenfeld, and Uri Shalit.  On calibration and out-of-domain generalization.  arXiv preprint arXiv:2102.10395, 2021.
  * [39] Max Welling and Yee W Teh.  Bayesian learning via stochastic gradient langevin dynamics.  In ICML, 2011.
  * [40] Chen Xing, Sercan Arik, Zizhao Zhang, and Tomas Pfister.  Distance-based learning from errors for confidence calibration.  In ICLR, 2020.
  * [41] Junbo Zhao, Michael Mathieu, and Yann LeCun.  Energy-based generative adversarial network.  In ICLR, 2017.
  * [42] Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy.  Domain generalization in vision: A survey.  arXiv preprint arXiv:2103.02503, 2021.


# Deep Learning Based EDM Subgenre Classification using Mel-Spectrogram and
Tempogram Features

Wei-Han Hsu, Bo-Yu Chen, and Yi-Hsuan Yang  Research Center for IT Innovation
Academia Sinica
Taipei, Taiwan
{ddmanddman, bernie40916<EMAIL_ADDRESS>
###### Abstract

Along with the evolution of music technology, a large number of styles, or
“subgenres,” of Electronic Dance Music (EDM) have emerged in recent years.
While the classification task of distinguishing between EDM and non-EDM has
been often studied in the context of music genre classification, little work
has been done on the more challenging EDM subgenre classification. The state-
of-art model is based on extremely randomized trees and could be improved by
deep learning methods. In this paper, we extend the state-of-art music auto-
tagging model “short-chunk CNN$+$Resnet” to EDM subgenre classification, with
the addition of two mid-level tempo-related feature representations, called
the Fourier tempogram and autocorrelation tempogram. And, we explore two
fusion strategies, early fusion and late fusion, to aggregate the two types of
tempograms. We evaluate the proposed models using a large dataset consisting
of 75,000 songs for 30 different EDM subgenres, and show that the adoption of
deep learning models and tempo features indeed leads to higher classification
accuracy.

###### Index Terms:

EDM subgenre classification, deep learning, feature fusion, tempogram,
convolutional neural network

##  I Introduction

Electronic Dance Music (EDM) is a kind of dance and club music. Disk Jocket
(DJs) usually need to classify EDM by their subgenres to get songs with
similar style for many purposes, such as for making the DJ set transitions
part. An automatic program for EDM subgenre classification can be useful for
human DJs [1]. It is also a fundamental building block towards realizing an
automatic AI DJ [2, 3].

The task _automatic EDM subgenre classification_ can be in general considered
as an instance of music auto-tagging problem [4]. Therefore, methodology-wise
we can base on research that has been done for general music auto-tagging and
classification. However, we note that, due to the similarity among the EDM
subgenres, EDM subgenre classification can sometimes be difficult even for
human DJs. The decision between genres can be fuzzy. For example, “tech-
house,” “deep-house,” and “progressive-house” music may sound fairly similar
as they are all “house” music.

We present in this paper a deep learning based approach to automatic EDM
subgenre classification, extending the recent work by Caparrini _et al._ [5],
which use a non-deep learning approach. Following their work, we compile our
training and test sets from Beatport (https://www.beatport.com/), a worldwide
principal source of music for DJs. The Beatport website assigns only a single
subgenre label to each song, so we can formulate the task as a multi-class
classification problem. Our dataset contains 30 different subgenres, each with
2,500 songs and hence 75,000 songs in total. We thus treat it as a 30-class
classification problem.

While the classifier used by Caparrini _et al._ [5] was based on extremely
randomized trees [6], a non-deep learning algorithm, our classifier is based
on the “short-chunk convolutional neural network (CNN)$+$Resnet” deep
architecture proposed by Won _et al._ [7], which represents the state-of-the-
art in music auto-tagging (see Figure 2). However, the original short-chunk
CNN model takes only the Mel-spectrograms as input for feature learning. While
this may be sufficient for classifying broader genre classes such as Pop,
Rock, Jazz, and EDM, it is unclear how it performs for a subgenre
classification task.

Figure 1: Boxplots of the tempo values (in beat-per-minute; or BPM) of
different EDM subgenres considered in our work.

In particular, we note that, among the 92 hand-crafted audio features employed
by Caparrini _et al._ [5], _tempo_ -related features were found to be the top-
four important features. By estimating the tempo values for all the songs in
our dataset using the algorithm of Grosche _et al._ [8], and plotting the
resulting distribution of tempo values for 100 randomly picked songs from each
genre, as shown in Figure 1, we can see that different EDM subgenres do prefer
different tempo values. For example, the tempo of “psy-trance” music is
usually around 140 beat-per-minute (BPM), while “deep-house” is usually around
120 BPM. Different subgenres also have different tempo ranges. For example,
the tempo of “jumpup-drum-and-bass” can vary widely from 90 to 170 BPM, while
the tempo values of “tech-house” typically fall within 120 to 130 BPM.

In light of this, we propose simple extensions of the short-chunk CNN$+$Resnet
model [7] to learn features from not only the Mel-spectrogram but also the
“tempogram” [8], a feature representation of the possible tempo values of each
short-time frame of a signal. We consider both the autocorrelation- and
Fourier-tempogram proposed by Grosche _et al._ [8], and experiment with both
an early-fusion architecture and a late-fusion architecture. Our experiment
shows that the Mel-spectrogram only baseline attains 55.4% song-level
accuracy, and that our best model incorporating additionally tempograms
achieves 60.6% accuracy. We find in particular salient improvement in
subgenres such as “future-house,” “leftfield-house-and-techno,” and “uplifing-
trance”.

While we cannot re-distribute the audio files of the data due to copyright
issues, we release our code and model checkpoints at https://github.com/mir-
aidj/EDM-subgenre-classifier.

##  II Related Work

Music genre classification is a well-researched task in the field of music
information retrieval (MIR), usually formulated as a multi-class
classification problem. For example, the most famous and widely-used dataset
in music genre classification, the GTZAN dataset [9], includes in total 1K
songs covering the following 10 genres: Blues, Classical, Country, Disco,
Hiphop, Jazz, Metal, Pop, Reggae and Rock. In contrast, relatively less
research has been done on music subgenre classification, which aims at
distinguishing between subgenres belonging to the same “super” genre such as
EDM and Jazz. We review three such existing work below.

EDM subgenre classification. Caparrini _et al._ [5] presented a non-deep
learning method for EDM subgenre classification. They compiled two different
collections of EDM subgenre data from Beatport, the first with 23 subgenres
and the second expanded version with 29 subgenres. They retrieved 100 songs
for each subgenre from the Beatport website for both datasets. Then, they
employed 92 hand-crafted audio features and tested several non-deep learning
based classifiers under 10-fold cross validation. For the first set, the
gradient tree boosting [10] performed the best, reaching 59.2% song-level
classification accuracy. For the second set, which is more challenging due to
the larger number of classes, the extremely randomised trees [6] performed the
best, reaching 48.2% accuracy. They also drew two confusion matrices for both
two sets, as well as the directed graph to show which subgenres were easily
misclassified. And, they measured the importance of each feature in their
tree-based classifier, finding that the top-four important features are all
related to tempo.

Figure 2: System diagram of the short-chunk CNN$+$Resnet model [7].

(a) (b) (c)

Figure 3: System diagram of the proposed architecture for (a) employing both
Mel-spectrograms and two different tempograms for music subgenre
classification; (b) zoom-in of the early-fusion module for fusing the two
tempograms; (c) the late-fusion variant. The ‘SCcnn’ in (a) denotes the stack
of 7 copies of feature extraction blocks (each with two 2-D convolutions)
shown in Figure 2.

Heavy metal subgenre classification. Tsatsishvili [11] investigated subgenre
classification for heavy metal music, also using non-deep learning methods.
The author compiled a dataset of 210 songs, comprising 30 songs for each of
the following 7 heavy metal subgenres: “black,” “death,” “melodic,” “death,”
“gothic,” “heavy,” “power” and “progressive.” Half the songs were used for
training and the other half for testing. Audio features employed in the
classifiers were automatically chosen from 200 hand-crafted features with
either a correlation-based feature selection method or a wrapper selection
method. The best classification accuracy was achieved by AdaBoost [12],
reaching 45.7%.

Jazz subgenre classification. Quinto _et al._ [13] employed deep learning
methods for Jazz subgenre classifcation, considering only three subgenres:
“acid-jazz,” “bebop,” and “swing/electroswing.” They considered a simpler
multi-layer perceptron (MLP) with 1–3 layers, as well as a more sophisticated
3-layer recurrent neural network employing long short-term memory (LSTM) with
32 neurons per layer. They employed Mel-frequency cepstral coefficients (MFCC)
as the input feature. Their dataset includes 254 minutes of “acid-jazz,” 141
minutes “bebop,” and 245 minutes “swing/electroswing.” 60% of them were used
for training and the rest for testing. Finally, they got 90% testing accuracy
with the LSTM classifier, and 79% with the MLP classifier. The accuracy was
fairly high, possibly because of the small number of classes.

Unlike music genre/subgenre classification, the music auto-tagging task [7,
14, 15, 16, 17, 18, 19, 20] is often formulated as a multi-label
classification problem, where a song can be labeled with multiple tags. The
performance of an auto-tagging model is usually evaluated in terms of metrics
such as PR-AUC and ROC-AUC [4]. Won _et al._ [7] experimented with a larger
number of different CNN-based models on three widely-used public datasets for
this task: MagnaTagATune (MTAT) [14], million song dataset (MSD) [21], and
MTG-Jamendo [19], finding that a particular architecture called “short-chunk
CNN$+$Resnet” performs in general the best. As it is straightforward to modify
this architecture for multi-class classification, we employ it as the backbone
architecture of our models in this work.

##  III Methods

###  III-A Dataset

Following Caparrini _et al._ [5], we build our dataset by crawling the audio
previews of songs and their corresponding class labels from Beatport.
Different from their work, we consider a larger set of classes, covering the
30 EDM subgenres listed in Figure 1, and a much larger collection of songs,
with consistently 2,500 songs per subgenre.111This Beatport dataset was
initially created in one of our prior works [24]. We split the dataset by the
ratio of 8:1:1 per subgenre to get the training, validation, and test sets.
Namely, the test set contains 250 tracks per subgenre. Every audio preview
made available by Beatport is 2 minutes long. We make them consistently mono
channel with 22,050 Hz sampling rate.

###  III-B Input Features: Mel-spectrogram and Tempograms

Instead of using hand-crafted features as done by Caparrini _et al._ [5], we
employ “raw” features as input to our deep neural network for feature
learning. The basic feature representation, also one of the most widely-used
one in MIR, is the Mel-spectrogram, a time-frequency representation that is
computed by applying the perceptually-motivated Mel filter bank to the
spectrogram of an audio waveform. With librosa [22], we compute the Mel-
spectrogram using a Hamming window of 2,048-sample long and 512-sample hop
length for the short-time Fourier transform (STFT), and 128 Mel filters.

We also employ the tempogram [8] as input feature, a “time-tempo”
representation that contains local tempo information for each frame of an
audio signal. Grosche _et al._ [8] proposed two types of tempogram: the
_Fourier tempogram_ and the _autocorrelation tempogram_. The former converts
frequency (Hz) to tempo (beat-per-minute; BPM), emphasizing the harmonics,
while the latter converts time-lag (seconds) to tempo, emphasizing instead the
subharmonics. We consider both here. The _Fourier tempogram_
222https://www.audiolabs-
erlangen.de/resources/MIR/FMP/C6/C6S2_TempogramFourier.html is computed by
firstly estimating from the audio waveform a “novelty curve” indicating note
onset candidates, and then computing the Fourier representation of the novelty
curve (not the original audio waveform) using STFT. The resulting
representation is assumed to capture local periodic patterns of the input
signal. Its frequency axis is finally mapped to a BPM axis. The
_autocorrelation tempogram_ ,333https://www.audiolabs-
erlangen.de/resources/MIR/FMP/C6/C6S2_TempogramAutocorrelation.html on the
other hand, is computed by calculating the local autocorrelation function for
different time lags from the also the novelty curve, and then converting the
time-lag to a linear BPM axis using interpolation and resampling. We also
employ librosa [22] to compute these two variants of the tempogram, with
512-sample hop size and a Hamming window of 2,048 samples. Figure 6 provides
examples of the Fourier tempogram and autocorrelation tempogram of songs in
our dataset.444Grosche _et al_ [8] also proposed the “cyclic tempogram,” where
tempi differing by a power of two are identified, but we do not use it here.

We use two different length per song to compute the Mel-spectrograms: the
30-second segment from 15s to 45s of a song, or the whole two minutes. For the
tempograms, we use only the segment a song from 15s to 45s for simplicity. The
size of a Fourier tempogram and an autocorrelation tempogram would be
193$\times$1,293 (BPM$\times$time) and 384$\times$1,292, respectively. Audio
is fed to our models using “chunks” (see below) of such fixed-length Mel-
spectrograms and tempograms. Every feature dimension is zscore-normalized.

###  III-C Short-chunk CNN with Resnet

We employ the short-chunk CNN$+$Resnet as the backbone architecture of our
models, as it has been shown to outperform competing models in music auto-
tagging across different benchmark datasets [7]. As depicted in Figure 2, this
model contains 7 copies of feature extraction layers, each comprising two 2-D
convolutional layers, two batch normalization layers (after convolution), and
one ReLU activate layer in between convolutions. The output of this stack of
layers goes through max pooling and then two dense layers and a final softmax
layer for classification. The name “short-chunk” stems from the fact that the
model is designed to take as input short segments of the Mel-spectrogram of
the size 128$\times$200, which in other words divides our Mel-spectrogram into
25 chunks (neglecting the last 168 frames). We similarly divide the tempograms
into 25 chunks. Following [7], we assume that the subgenre label of each chunk
is the same as the the subgenre label of the song the chunk comes from.

TABLE I: Chunk- and song-level testing accuracy of 30-class EDM subgenre classification of the evaluated short-chunk CNN-based models; the first two models use only the Mel-spectrograms, while the last two use both the Mel-spectrograms and tempograms | chunk-level | song-level
---|---|---
Mel-spectrogram only (30 sec) [7] | 46.1% | 50.4%
Mel-spectrogram only (120 sec) [7] | 46.1% | 55.4%
Fourier tempogram only (30 sec) | 32.0% | 34.9%
autocorrelation tempogram only (30 sec) | 28.3% | 31.2%
early-fusion | 53.4% | 60.3%
late-fusion | 53.3% | 60.6%

###  III-D Proposed Fusion Models

Figure 3(a) shows the architecture we propose to integrate the (chunk-level)
Mel-spectrogram, Fourier tempogram, and autocorrelation tempogram of a song
for classification. We firstly fuse the two tempograms into a combined
representation, and then concatenate it with the output of the feature
extraction blocks of the short-chunk CNN branch that deals with the Mel-
spectrogram. After feature concatenation, we use the same classification block
of the short-chunk CNN.

Figures 3(b) and (c) depict the two fusion strategies considered in our work
to combine the two tempograms. They both use four parallel 1-D convolutional
layers with different kernel sizes (3, 3, 5, 5) and strides (2, 3, 3, 5) for
feature extraction, a design that is inspired by the work of Pons _et al._
[23]. The outputs of the 1-D convolution layers are concatenated, mean-pooled,
futher processed with a 2-D convolution layer, and finally max-pooled to yield
a combined representation. The _early-fusion_ variant (Figure 3(b)) combines
the two tempograms at the very beginning with feature-wise concatenation,
while the _late-fusion_ variant (Figure 3(c)) combines them after the 1-D
convolutional layers.

Figure 4: Per subgenre testing classification accuracy of different models.

##  IV Experiments

We train the following deep models by using cross-entropy as the loss function
and Adam as the optimizer.

  * •

_Mel-spectrogram only_ : the short-chunk CNN$+$Resnet baseline [7] shown in
Figure 2.

  * •

_Proposed early- and late-fusion_ : the proposed models that use both the Mel-
spectrograms and the two tempograms, shown in Figure 3.

  * •

_Fourier tempogram or autocorrelation tempogram only_ : the ablated variants
that use only one of the tempograms, without using the Mel-spectrograms. This
is done with a simplified version of the fusion models, with four 1-D
convolutional layers and a 2-D convolutional layer for feature extraction, and
two dense layers for classification.

We set the batch size to 256, and train the models for 200 epochs with
learning rate $0.005$. We use 50% dropout before the last dense layer and
after the ReLU layer.

We report the chunk-level classification accuracy by taking the class with the
highest softmax-ed value as the prediction result. In addition, we also report
the song-level accuracy by a majority voting mechanism over the prediction
results of the short chunks of a song.

Table I shows that early-fusion and late-fusion models are both better than
Mel-spectrogram-input model in both chunk-level and song-level accuracy. The
late-fusion model performs overall the best, reaching 60.6% song-level testing
accuracy for our 30-class classification setting, which is much higher than
the 48.2% accuracy obtained by Caparrini _et al._ [5] for 29-class
classification using a non-deep learning method.This also stands as a $+$5 to
$+$10% performance gain relative to the Mel-spectrogram only baseline,
empirically validating the benefit of using tempo-related features for EDM
subgenre classification. Table I also shows that the Fourier tempogram only
model outperforms the autocorrelation tempogram counterpart.

(a) Confusion table of the short-chunk CNN baseline (120 sec)

(b) Confusion table of the proposed late-fusion model

Figure 5: The confusion matrices of the testing result of two evaluated
models. Each row shows whether the songs from a subgenre are misclassified to
other subgenres; those on the diagonal are correctly classified. As our test
set contains consistently 250 songs per subgenre, each row sums up to 250.

Figure 4 shows the per-genre result of these five models. We can see salient
performance improvement for genres such as “future-house,” “leftfield-house-
and-techno” and “uplifing-trance” for the proposed fusion models than the
baseline Mel-spectrogram only model.

Figure 5 shows the 30$\times$30 confusion matrices of the baseline model and
the proposed late-fusion model. We can see that the propose model still gets
confused for some pairs of subgenres, but less seriously so than the baseline
model.

Finally, Figure 6 visualizes the Mel-spectrograms and tempograms of four songs
from our test set. The songs shown in Figures 6(a) and (b) are both
“uplifting-trance” songs. They can both be correctly classified by our late-
fusion model, but the first song would be wrongly recognized as a “tech-
trance” song by the baseline model. We see that these two songs seem to have
fairly different patterns in the Mel-spectrograms, but similar patterns in the
autocorrelation tempograms. The baseline model does not work well for this
song, possibly because it only has access to the Mel-spectrograms. On the
other hand, the songs shown in Figures 6(c) and (d) are an “uplifing-trance”
song and a “tech-trance” song, respectively. Again, they can both be correctly
classified by our late-fusion model, but the first song in this pair (i.e.
(c)) would be wrongly regarded as a “tech-trance” song by the baseline model.
We see these two songs, despite they are associated with different subgenres,
have some similar local patterns in their Mel-spectrograms, which might have
caused confusion for the baseline model.

(a) (b) (c) (d)

Figure 6: The Mel-spectrogram, Fourier tempogram and autocorrelation tempogram
for (a) an “uplifting-trance” song misclassified as “tech-trance” by the
baseline model but correctly classified by our late-fusion model; (b) an
“uplifting-trance” song correctly classified by both the baseline model and
our fusion model; (c) another “uplifting-trance” song misclassified as “tech-
trance” by the baseline model but correctly classified by our late-fusion
model; (d) a “tech-trance” correctly classified by both the baseline model and
our fusion model.

##  V Conclusion

In the paper, we have presented a deep learning approach for EDM subgenre
classification, achieving 60.6% testing accuracy for 30-class classification
when using not only the Mel-spectrograms but also the tempograms as input
feature. We found that the proposed late-fusion model is 10% more accurate
than a Mel-spectrogram only deep learning baseline model. This may suggest
that, for a fine-grained classification problem such as EDM subgenre
classification, it is beneficial to consider multiple inputs for feature
learning.

We have a few ideas for future extension of this work. First, we would like to
explore different architectures to process the tempograms, such as the use of
recurrent layers or temporal convolutional layers [25]. Second, to incorporate
more features as input, including other mid-level features [26] as well as
high-level features [17, 24, 27]. Third, to employ other feature fusing
strategies. And finally, to employ metric-based learning method [28], which is
shown to be promising in recent work on music auto-tagging.

## References

  * [1] S. Sankalp, T. Baruah, S. Tiwari and S. Ganesh, “Intelligent classification of electronic music,” in _Proc. IEEE Int. Symposium on Signal Processing and Information Technology_ , pp. 31–35, 2014.
  * [2] Y.-S. Huang, S.-Y. Chou, and Y.-H. Yang, “DJnet: A dream for making an automatic DJ,” in _Proc. Int. Soc. Music Information Retrieval Conf._ , late-breaking and demo paper, 2017.
  * [3] B.-Y. Chen, W.-H. Hsu, W.-H. Liao, M. A. M. Ramírez, Y. Mitsufuji, and Y.-H. Yang, “Automatic DJ transitions with differentiable audio effects and generative adversarial networks,” in _arXiv preprint:2110.06525_ , 2021.
  * [4] J. Nam, K. Choi, J. Lee, S.-Y. Chou, and Y.-H. Yang, “Deep learning for audio-based music classification and tagging,” _IEEE Signal Processing Magazine_ , vol. 36, no. 1, pp. 41–51, Jan. 2019.
  * [5] A. Caparrini, J. Arroyo, L. Pérez-Molina, J. Sánchez-Hernández, “Automatic subgenre classification in an electronic dance music taxonomy,” _Journal of New Music Research_ , vol. 49, no. 3, pp. 269–284, 2020.
  * [6] P. Geurts, D, Ernst, L. Wehenkel, “Extremely randomized trees,” _Proc. Machine learning_ , pp. 3–42, 2006.
  * [7] M. Won, A. Ferraro, D. Bogdanov, X. Serra, “Evaluation of CNN-based automatic music tagging models,” in _Proc. Sound and Music Computing Conf._ , 2020.
  * [8] P. Grosche, M. Müller, and F. Kurth, “Cyclic tempogram: A midlevel tempo representation for music signals,” in _Proc. IEEE Int. Conf. Acoust., Speech, Signal Processing_ , pp. 5522–5525, 2010.
  * [9] G. Tzanetakis and P. Cook, “Musical genre classification of audio signals,” _IEEE Transactions on Speech and Audio Processing_ , vol. 10, no. 5, pp. 293–302, Jul. 2002. [Online] https://www.kaggle.com/andradaolteanu/gtzan-dataset-music-genre-classification.
  * [10] J. H. Friedman, “Stochastic gradient boosting,” _Proc. Computational statistics & data analysis_, vol. 38, no. 4, pp. 367–378, 2002.
  * [11] V. Tsatsishvili, “Automatic subgenre classification of heavy metal music,” University Of Jyväskylä, master thesis, 2011.
  * [12] J. Bergstra, N. Casagrande, D. Erhan, D. Eck, B. Kégl, “Aggregate features and adaboost for music classification,” _Proc. Machine learning_ , vol. 65, no. 2-3, pp. 473–484, 2006.
  * [13] R. J. M. Quinto, R. O. Atienza, and N. M. C. Tiglao, “Jazz music subgenre classification using deep learning,” in _Proc. IEEE Region 10 Conf._ , pp. 3111–3116, 2017.
  * [14] E. Law, K. West, M. I. Mandel, M. Bay, and J. S. Downie, “Evaluation of algorithms using games: The case of music tagging.” in _Proc. Int. Soc. Music Information Retrieval Conf._ , pp. 387-–392, 2009.
  * [15] D. Turnbull, L. Barrington, D. Torres and G. Lanckriet, “Semantic annotation and retrieval of music and sound effects,” _IEEE Transactions on Audio, Speech, and Language Processing_ , vol. 16, no. 2, pp. 467–476, Feb. 2008.
  * [16] J.-Y. Liu and Y.-H. Yang, “Event localization in music auto-tagging,” in _Proc. ACM Multimedia_ , pp. 1048–1057, 2016.
  * [17] J. Pons and X. Serra, “musicnn: Pre-trained convolutional neural networks for music audio tagging,” in _arXiv preprint:1803.01271_ , 2018.
  * [18] M. Won, S. Chun, O. Nieto Caballero, X. Serra, “Automatic music tagging with harmonic CNN,” in _Proc. Int. Soc. Music Information Retrieval Conf._ , 2019.
  * [19] D. Bogdanov, M. Won, P. Tovstogan, A. Porter, X. Serra, “The MTG-Jamendo dataset for automatic music tagging,” in _Proc. Machine Learning for Music Discovery Workshop_ , 2019.
  * [20] M. Won, S. Chun, and X. Serra, “Toward interpretable music tagging with self-attention,” _arXiv preprint:1906.04972_ , 2019.
  * [21] T. Bertin-Mahieux, D. P. Ellis, B. Whitman, and P. Lamere, “The million song dataset,” in _Proc. Int. Soc. Music Information Retrieval Conf._ , 2011.
  * [22] B. McFee, C. Raffel, D Liang, D. P. Ellis, M. McVicar, E. Battenberg, O. Nieto, “librosa: Audio and music signal analysis in Python,” in _Proc. Python in Science Conf._ , vol. 8, pp.18–25, 2015.
  * [23] J. Pons, O. Slizovskaia, and R. Gong, “Timbre analysis of music audio signals with convolutional neural networks,” in _Proc. European Signal Processing Conf._ , pp. 2744–2748, 2017.
  * [24] Y.-S. Huang, S.-Y. Chou and Y.-H. Yang, “Pop music highlighter: Marking the emotion keypoints,” _Transactions on International Society for Music Information Retrieval_ , vol. 1, no. 1, pp. 68–78, 2018.
  * [25] S. Bai, J. Z. Kolter, and V. Koltun, “An empirical evaluation of generic convolutional and recurrent networks for sequence modeling,” in _arXiv preprint:1909.06654_ , 2019.
  * [26] H. Foroughmand and G. Peeters, “Deep-Rhythm for global tempo estimation in music,” in _Proc. Int. Soc. Music Information Retrieval Conf._ , pp. 636–643, 2019.
  * [27] E. Zangerle, M. Vötter, R. Huber, and Y.-H. Yang, “Hit song prediction: Leveraging low- and high-level audio features,” in _Proc. Int. Soc. Music Information Retrieval Conf._ , pp. 319–326, 2019.
  * [28] M. Won, S. Oramas, O. Nieto, F. Gouyon, and X. Serra, “Multimodal metric learning for tag-based music retrieval,” in _Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing_ , pp. 591–595, 2021.