# Projected Belief Networks With Discriminative Alignment for Acoustic Event
Classification: Rivaling State of the Art CNNs

Paul M. Baggenstoss  Fraunhofer FKIE, Fraunhoferstrasse 20
53343 Wachtberg, Germany
Email<EMAIL_ADDRESS>
Kevin Wilkinghoff  Fraunhofer FKIE
Email<EMAIL_ADDRESS>
Felix Govaers  Fraunhofer FKIE
Email<EMAIL_ADDRESS>
Frank Kurth This work was supported jointly by the Office of Naval Research
Global and the Defense Advanced Research Projects Agency under Research Grant
- N62909-21-1-2024 Fraunhofer FKIE
Email<EMAIL_ADDRESS>

###### Abstract

The projected belief network (PBN) is a generative stochastic network with
tractable likelihood function based on a feed-forward neural network (FFNN).
The generative function operates by “backing up” through the FFNN. The PBN is
two networks in one, a FFNN that operates in the forward direction, and a
generative network that operates in the backward direction. Both networks co-
exist based on the same parameter set, have their own cost functions, and can
be separately or jointly trained. The PBN therefore has the potential to
possess the best qualities of both discriminative and generative classifiers.
To realize this potential, a separate PBN is trained on each class, maximizing
the generative likelihood function for the given class, while minimizing the
discriminative cost for the FFNN against “all other classes”. This technique,
called discriminative alignment (PBN-DA), aligns the contours of the
likelihood function to the decision boundaries and attains vastly improved
classifcation performance, rivaling that of state of the art discriminative
networks. The method may be further improved using a hidden Markov model (HMM)
as a component of the PBN, called PBN-DA-HMM. This paper provides a
comprehensive treatment of PBN, PBN-DA, and PBN-DA-HMM. In addition, the
results of two new classification experiments are provided. The first
experiment uses air-acoustic events, and the second uses underwater acoustic
data consisting of marine mammal calls. In both experiments, PBN-DA-HMM
attains comparable or better performance as a state of the art CNN, and
attains a factor of two error reduction when combined with the CNN.

###### Index Terms:

Bayesian classifier, generative model, PDF estimation, projected belief
network, acoustic event classification

##  I Introduction

###  I-A Generative vs. Discriminative Classification

Classification is probably the most important task in machine learning because
it involves making decisions between several possible ways to understand the
data. Until the advent of deep learning, this task was best solved by the
human mind in conjunction with human senses and perception. There remain two
distinct approaches to classification: the generative approach, and the
discriminative approach [1, 2]. Both approaches find parallels in human
perception and decision-making.

In the generative approach, an attempt is made to characterize each class
individually by estimating a set of patterns or parameters, then later compare
the data to each stored characterization. In statistical terms, the
probability distribution of the data is estimated for each class hypothesis,
and later the likelihood of the data is determined with respect to each
learned distribution estimate. In human perception, the generative approach
corresponds to recognizing something based on our memory of what the class
should look or sound like.

In the discriminative approach, an attempt is made to distinguish between the
existing class hypotheses, an approach that is inherently more efficient and
generally provides performance superior to the generative approach [2]. In
human perception, the discriminative approach corresponds to looking or
listening for characteristic clues that can distinguish between hypotheses.
For example, if we know that two identical twins differ only in a birthmark,
we look for the presence of this hard-to see characteristic.

The generative approach is inherently inefficient because it is blind to the
other class hypotheses, so does not know in advance which subtle clues it
needs, and must extract a complete detailed characterization of each class
hypothesis. As the data dimension increases, the amount of information
required to adequately characterize the data distribution rises exponentially,
outstripping what is available in the training data. This is the so-called
curse of dimensionality [2]. The generative approach becomes even more
difficult in noisy scenarios because generative models cannot distinguish
between the target signal and noise.

Despite these problems, generative classifiers are very useful, especially in
open-set classification problems where they can detect out-of-set events. In
addition, having a “second opinion” from a generative approach can result in
improved performance for a combined generative/discriminative approach.

###  I-B Attaining the best properties of both approaches

When combining classifiers, such as by additively combining the classifer
statistics (i.e. ensembling), it is important that the two methods (a) have
comparable performance and (b) are based on independent views of the data.
This is why it is important to create better-performing generative classifiers
to combine with purely discriminative ones. Therefore, it would be desireable
if generative classifiers could be constructed to utilize discriminative
information, analogous to how humans do, and thereby achieve improved
performance. Later, in the experiments, we will combine PBN with a
discriminative CNN, but this is not what we mean by a combined approach. This
is because the two methods are not able to benefit from each other during
training. Using PBN, it is possible to apply both the generative and
discriminative approaches simultaneously, that is to create a generative model
with discriminative capability. Even if this approach does not improve upon
state of the art discriminative classifiers, it could result in a classifier
that is better for combination with them.

In human perception, we use a combined generative and discriminative approach
if we give discriminative clues priority as we collect clues for the
generative approach. To use the example of the identical twins above, we would
add the presence of a birthmark to the list of features that describe each
class individually. So, conceptually, a combined approach is not hard to
understand. Does there exist a parallel idea in machine learning, in which a
given feature extractor (i.e. a given network) can be both generative and
discriminative?

Many methods have been proposed that either incorporate both generative and
discriminative ideas or achieve the goal of joint generative and
discriminative training [3, 4, 5, 6, 7, 8]. But these approaches concentrate
on training methods, and do not treat the problem at the fundamental level
that we are proposing.

###  I-C Discriminative and Generative functions in a single network

To have both discriminative and generative functions in a single network, the
network needs to operate in both the forward and backward directions. We will
call this a two-directional network, so as not to confuse the idea with bi-
directional networks111Bi-directional networks have some layers that operate
forward in time and some layers that operate backward in time.  [9] because it
can operate in the forward direction (i.e. as a feed-forward classifier
network), but also in the backward direction (as a generative network), using
the same network architecture and parameter set. We seek the following
properties of a two-directional network:

  1. 1.

The desired network topology must be dimension-reducing in order to serve also
as a classifier.

  2. 2.

The generative function should be complete, which means it should provide a
tractable likelikood function (LF) (i.e. log of the probability density
function) of the input data, and should be able to synthesize samples
corresponding to this LF.

  3. 3.

To be a truly dimension-reducing two-directional network, we propose that the
reverse path should constitute a right inverse operation. If the forward path
is written ${\bf z}=T({\bf x})$, where ${\bf x}\in\mathbb{R}^{N}$, ${\bf
z}\in\mathbb{R}^{M}$, $N>M$, and the right inverse is written ${\bf
x}=T^{-1}({\bf z})$, then we must have $T\left(T^{-1}({\bf z})\right)={\bf
z}$. This is important because or stated goal is that the discriminative
(forward path) and generative (backward path) should learn jointly. The right-
inverse property insures a tight connection between the two directions. In
fact, it should be clear that the right- inverse property can only be achieved
by a single network, i.e. not by separate networks that each operate in one
direction, because the reverse path is the inverse of the forward path.

There are a number of existing machine learning approaches that we can
consider.

Although they are not right-inverse networks, we discuss two related
approaches: auto-encoders and stacked restricted Boltzmann machines (RBMs).
The RBM consists of two back-to-back stochastic perceptron layers with tied
weights (the backward weight matrix is the transpose of the forward weight
matrix) [10]. Although the reverse path of the RBM is not a right inverse, it
uses the same weight matrix and is a generative network based on an elegant
theory, so is worth looking at. The input data (visible data) and the hidden
variables are jointly distributed according to the Gibbs distribution. The RBM
is trained using an approximate maximum likelihood approach called contrastive
divergence (CD) [11]. Once trained, either direction of the RBM can be
converted into a perceptron layer by replacing the stochastic perceptrons with
deterministic activation functions equal to their expected value. Multiple
deterministic RBM perceptron layers can then be cascaded together (in both
directions) to obtain a stacked RBM. The stacked RBM has also a statistical
interpretation and can be trained by a multi-layer version of CD called up-
down algorithm [10].

A stacked RBM is a multi-layer two-directional network without right-inverse,
and is identical to an auto-encoder with tied analysis and synthesis weights,
but trained differently. Note that it is not uncommon to tie the analysis and
synthesis weight matrices of an auto-encoder (i.e. so that their corresponding
linear operations are transposed versions of each other), which is possible
both for dense and convolutional layers [12]. Auto-encoders are in wide-spread
use because it is inherently useful to find a low-dimensional output feature
set (called the encoding or bottleneck features) with minimum dimension that
can be used to reconstruct the input data.

In addition to lacking a right-inverse, these networks are also not complete
generative models, because they lack a tractable LF. Even the stacked RBM,
which has a well-defined statistical model [10], cannot be tractably
marginalized to obtain the likelihood function of the input data. A
variational auto-encoder (VAE) is a type of auto-encoder [13, 14, 15], but the
LF must be approximated and it is not a right-inverse network.

Normalizing flows (NF) [16] provide a way of defining a probability
distribution by finding a 1:1 dimension-preserving transformation that maps
the input data to a well-defined distribution such as to independent
identically-distributed (iid) uniform or normal random variables. The
distribution of the input data is found using the change of variables theorem,
i.e. using the determinant of the transformation’s Jacobian matrix. The
forward path extracts the iid random variables, and the reverse path creates
synthetic input data. The problem with NF is that the forward path is not
dimension-reducing, so cannot be used as a discriminative classifier. That is
unfortunate because NF is truly two-directional and has a well-defined
likelihood function. In order to get around the requirement of dimension-
preserving transformations, a way to combine NF with variational autoencoders
(VAEs) called “SurVAE” has been proposed, [17]. However, for the case of
dimension-reducing deterministic transformations (surjections), SurVAE is the
same as and is a re-invention of PDF projection [18], which is the basis of
our proposed approach and will be reviewed in the next section.

###  I-D What’s New in This Paper

The approaches presented in this paper can all be found in various forms in
existing conference and journal publications [19, 20, 21, 22, 23, 24, 25, 26,
27, 28, 29]. However, there is no comprehensive and cohesive treatment to make
the idea set accessible to readers. This paper seeks to fill this void. The
experimental results are also new, as well as the way the concepts are
introduced, specifically in the context of a two-directional network.

##  II Review of PDF Projection

###  II-A Definition of PDF Projection

Before we introduce the projected belief network (PBN), we need to introduce
and review PDF projection, the principle upon which it is based. Consider a
fixed and differentiable dimension-reducing transformation ${\bf z}=T({\bf
x})$, where ${\bf x}\in\mathbb{X}\subseteq\mathbb{R}^{N}$, and ${\bf
z}\in\mathbb{R}^{M}$, where $M<N$, and whose $N\times M$ matrix of partial
derivatives is of full rank. Then, assuming a known or assumed feature
distribution $g({\bf z})$, one can construct a probability density function
(PDF) on the input data with support $\mathbb{X}$ given by

$G({\bf x})=\frac{p_{0,x}({\bf x})}{p_{0,x}({\bf z})}g({\bf z}),$ (1)

where $p_{0,x}({\bf x})$ is a prior distribution with support on $\mathbb{X}$
and $p_{0,x}({\bf z})$ is its mapping to $\mathbb{R}^{M}$ through $T({\bf
x})$. In our simplified notation, the argument of the distribution defines its
range of support, and the variable in the subscript defines the original range
where the distribution was defined. Therefore, $p_{0,x}({\bf z})$ is a
distribution over the range of ${\bf z}$, but is a mapping of a distribution
that was defined on $\mathbb{X}$. Note that $G({\bf x})$ is seen as a function
only of ${\bf x}$ since ${\bf z}$ is deterministically determined from ${\bf
x}$ using ${\bf z}=T({\bf x})$. In fact, it can be shown [19, 21] that $G({\bf
x})$ is a PDF (integrates to 1) and is a member of the set of PDFs that map to
$g({\bf z})$ through $T({\bf x})$. If $p_{0,x}({\bf x})$ is selected for
maximum entropy, then $G({\bf x})$ is unique for a given transformation, data
range $\mathbb{X}$, and a given $g({\bf z})$ (where “g” represents the “given”
feature distribution). One can think of $G({\bf x})$ as a constructed
distribution based on a set of parameters (i.e. the parameters of $T({\bf
x})$, which could be a neural network for example). Then, using the method of
maximum likelihood, the PDF of the input data can be estimated by training the
parameters of the transformation to maximize the mean of $\log G({\bf x})$
over a set of training data. This in fact results in a transformation that
extracts sufficient statistics and maximizes information [22] (see Section
II-E). We say that $G({\bf x})$ is the “projection” of $g({\bf z})$ back to
the input data range $\mathbb{X}$, i.e. a back-projection. We call the term
$J({\bf x})\stackrel{{\scriptstyle\mbox{\tiny$\Delta$}}}{{=}}{p_{0,x}({\bf
x})\over p_{0,x}({\bf z})}$ the “J-function” because in the special case of
dimension-preserving transformations (i.e. normalizing flows [16]), $J({\bf
x})$ is the determinant of the Jacobian of $T({\bf x})$. The log of the
J-function is analogous to the “likelihood contribution” found in [17].

###  II-B Data Generation

To generate data from $G({\bf x})$ in (1), one draws a sample ${\bf z}$ from
$g({\bf z})$, then draws a sample ${\bf x}$ from the level set ${\cal L}({\bf
z})$ defined by

${\cal L}({\bf z})=\\{{\bf x}\in\mathbb{X}|T({\bf x})={\bf z}\\},$ (2)

and weighted by the prior distribution $p_{0,x}({\bf x})$. In other words, we
sample from $p_{0,x}({\bf x})$ restricted and nomalized on ${\cal L}({\bf
z})$.

Depending on the transformation $T$ and prior distribution $p_{0,x}({\bf x})$,
sampling may be straight forward, or may require Markov Chain Monte Carlo
(MCMC) methods. PDF projection sampling for specific examples of
transformations are treated in [21, 23].

###  II-C Chain Rule

In order to implement (1), it is necessary to know $p_{0,x}({\bf z})$. For
complex non-linear transformations, it may be impossible to find a tractable
form for $p_{0,x}({\bf z})$. However, using the chain-rule, it is possible to
solve the problem by breaking the transformation into a cascade of simpler
transformations and applying the PDF projection idea recursively to stages of
a transformation. Consider a cascade of two transformations, ${\bf
y}=T_{1}({\bf x})$, and ${\bf z}=T_{2}({\bf y})$. Then, applying (1)
recursively,

$G({\bf x})=\frac{p_{0,x}({\bf x})}{p_{0,x}({\bf y})}\frac{p_{0,y}({\bf
y})}{p_{0,y}({\bf z})}g({\bf z}),$ (3)

which can be extended to any number of stages. To compute $\log G({\bf x})$,
one just accumulates the log-J function of the transformations. Data
generation is also cascaded, and is initiated by drawing a sample ${\bf z}$
from $g({\bf z})$.

###  II-D Multiple features

In recent years, end-to-end deep learning approaches are slowly making feature
extraction obsolete. But, generative classification approaches are rarely used
on unprocessed data. Some form of feature extraction, such as conversion to
time-frequency spectral features, is carried out before application of
generative models because this tends to reduce dimensionality and remove
nuisance information such as phase. However, data can sometimes have very
diverse character and just one feature extraction approach may not suffice.

But, the problem with using multiple feature extraction approaches in
generative models is that the definition of “input data” is not clear when
there are several feature sets. This problem is solved by PDF projection
because all feature extraction transformations can project back to the same
input data space.

There are a number of approaches to using PDF projection for this purpose. One
possible approach is to assign an individually optimized feature set to each
data class. But, using one feature extraction for each data class can be
problematic. For example, the existence of background noise in the data could
make a different feature extractor more “suitable”, increasing the likelihood
values for that feature/class combination and cause classification errors.
Therefore, it is safer and more effective to use the same set of feature
extraction approaches for all classes. Using PDF projection, the likelihood
function computed for each feature after back-projecting to the raw data can
be seen as a component of a mixture distribution [24, 25]. This brings much
more information to bear on the problem without increasing the dimensionality
of the generative models.

To add mathematical detail, consider $L$ feature extraction chains, resulting
in features ${\bf z}_{1},\ldots{\bf z}_{L}$, with feature probability density
functions $g_{1}({\bf z}_{1}),\ldots g_{L}({\bf z}_{2})$. Applying (1)
separately to each model, we arrive at the projected PDFs $G_{1}({\bf
x}),\ldots G_{L}({\bf x})$. The mixture distribution can be created as follows

$G({\bf x})=\sum_{i=1}^{L}\;\alpha_{i}\;G_{i}({\bf x}).$

Such a mixture model is then trained separately on each class, with the
weights $\\{\alpha_{i}\\}$ also considered as parameters to estimate.

However, a problem that arises when using multiple feature sets is caused by
the widely-used technique of overlapped, windowed segmentation of time-series.
While it is reasonbable to compare likelihood values for features extracted
from the same input data, it is not clear about how to resolve the differences
between two feature extractors operating on essentially different raw data.
When applying different overlapped segmentation prior to feature extraction,
the definition of “raw data” changes with the segmentation window-size. A
solution to this problem is provided by what we call Hanning-3 segmentation,
[30], however this it out of scope for this paper.

###  II-E Information Maximization by PDF Projection

Dimension-reduction is an important function in neural networks and machine
learning with many end-goals including classification, auto-encoding, and
feature embedding. When the end-goal is not known in advance, one can try to
maximize the information at the output. A widely-used criteria for maximizing
information transfer is the InfoMax principle, which seeks to maximize the
mutual information (MI) between the input data ${\bf x}$ and output data ${\bf
z}$ [31, 32, 33, 34]. The mutual information can be decomposed as

$I({\bf x},{\bf z})=H({\bf z})-H({\bf z}|{\bf x}),$

where $H({\bf z})$ and $H({\bf z}|{\bf x})$ are the entropy and conditional
entropy [32, 35]. For noiseless deterministic transformations, ${\bf z}$ is
known perfectly given ${\bf x}$, so $H({\bf z}|{\bf x})=0$ [32, 35]. Thus, the
task comes down to maximizing the entropy of the network output ${\bf z}$
[32]. But this is not very satisfactory as an information metric because
$H({\bf z})$ can be made as large as possible simply by scaling or applying
1:1 transformations which do not change the information content. To arrive at
meaningful results with InfoMax, one has to impose an arbitrary constraint,
such as variance or data range. A more satisfactory criterion was proposed in
[22]. The proposed approach is to quantify the separability of the input data
distribution $p({\bf x})$ from a maximum entropy (MaxEnt) reference
distribution $p_{0}({\bf x})$ through the Kullback-Leibler divergence (KLD),
written $D(p\|p_{0})$, where $D(p\|q)$ is the KLD :

$D(p\|q)\stackrel{{\scriptstyle\mbox{\tiny$\Delta$}}}{{=}}\mathbb{E}_{p}\left\\{\log\frac{p({\bf
x})}{q({\bf x})}\right\\}.$ (4)

A MaxEnt distribution $p_{0}({\bf x})$ is bland and featureless, so it makes
sense that the data distribution $p({\bf x})$ stands out relative to it. The
idea is that for maximum information flow, the separability of the two
distributions should not be reduced by the network, or in other words, we try
to maximize the divergence at the output, which is bounded by the divergence
at the input. It is shown in [22], that this same goal is achieved by PDF
projection when one estimates the distribution the input data by fitting
$G({\bf x})$ to $p({\bf x})$ using maximum likelihood and the available
training data:

$\max_{T}\left\\{\sum_{i}\log G({\bf x}_{i};T,g)\right\\},$ (5)

where we have added $T$ and $g$ as arguments to $G({\bf x})$ to make clear
that (1) depends on transformation $T$ and given feature density $g$. It is
interesting to note that as (5) is maximized over $T$, the true feature
distribution is in fact driven toward $g({\bf z})$. A side-benefit of this is
that if $g({\bf z})$ is some canonical distribution of (iid) output variables,
then the true feature distribution is driven toward $g({\bf z})$, i.e. $p({\bf
z})\longrightarrow g({\bf z})$. Therefore, (5) achieves three goals at the
same time :

  1. 1.

Maximization of information flow.

  2. 2.

PDF estimation, i.e. $G({\bf x}_{i};T,g)\longrightarrow p({\bf x})$.

  3. 3.

Generation of information-bearing statistics ${\bf z}$ with arbitrary desired
distribution as $p({\bf z})\longrightarrow g({\bf z})$.

##  III Projected Belief Network (PBN)

###  III-A Definition of PBN

When PDF projection is applied to a feed-forward neural network (FFNN) layer-
by-layer, this results in the projected belief network (PBN) [36]. The PBN is
a true two-directional network because the reverse path (sampling) is carried
out backwards through the FFNN and is a right-inverse.

###  III-B Mathematical Details of a PBN Layer

We concentrate on a single PBN layer, which corresponds to one perceptron
layer in a neural network. To illustrate the sampling process in one layer,
let ${\bf y}\in\mathbb{R}^{M}$ be the hidden variable at the output of a given
layer of a FFNN, and let ${{\bf x}\in\mathbb{R}^{N}}$ be the layer input,
where $N>M$. Let ${\bf y}=\lambda\left({\bf b}+{\bf W}^{\prime}{\bf
x}\right),$ where ${\bf W},{\bf b}$ are the layer weight matrix and bias, and
$\lambda(\;)$ is a strictly monotonic increasing (SMI) element-wise activation
function. We seek to sample ${\bf x}$ given ${\bf y}$. Because the SMI
activation function and bias are invertible, it is equivalent to sample ${\bf
x}$ given ${\bf z}$, where ${\bf z}={\bf W}^{\prime}{\bf x}.$ This is done by
drawing ${\bf x}$ randomly from the set of samples that map to ${\bf z}$,
weighted by prior distribution $p_{0,x}({\bf x})$. More precisely, ${\bf x}$
is drawn from level set (2), which in the case of a linear transformation is
the set (an affine subset and a manifold)

${\cal M}({\bf z})=\\{{\bf x}\in\mathbb{X}|{\bf W}^{\prime}{\bf x}={\bf
z}\\},$ (6)

with prior density $p_{0,x}({\bf x})$. This prior is selected according to the
principle of maximum entropy (MaxEnt) [37], so it has the highest possible
entropy subject to any constraints. These constraints include the range of the
variable ${\bf x}$, denoted by $\mathbb{X}$, determined by which activation
function was used in the up-stream layer, and some moment constraints that may
be necessary when $\mathbb{X}$ is unbounded. By definition, the drawn sample
will be a right-inverse (i.e. preimage) of ${\bf z}$.

Note that when discussing PBNs, we use the variable ${\bf x}$ to denote either
a layer input or a network input, depending on the context. The variable ${\bf
z}$ always refers to the output of the linear transformation, and the variable
${\bf y}$ always refers to the output of an activation function,

We consider three canonical data ranges that are common in machine mearning,
$\mathbb{R}^{N}$ : ${x_{i}\in(-\infty,\infty),\forall i}$, $\mathbb{P}^{N}$ :
${x_{i}\in[0,\infty),\forall i}$, and $\mathbb{U}^{N}$ :
${x_{i}\in[0,1],\forall i}$. In Table I, we list four useful combinations of
data range $\mathbb{X}$ and MaxEnt prior $p_{0,x}({\bf x})$. Note that when a
data range is defined that is unbounded, the entropy of a distribution with
support on that range can go to infinity. Therefore, to find a MaxEnt
distribution, it is necessary to place constraints on the moments, either
variance or mean. This is why the MaxEnt distributions are defined either with
a fixed variance (Gaussian or truncated Gaussian) or fixed mean (exponential).

TABLE I: MaxEnt priors and activation functions as a function of input data range. TG=“Trunc. Gauss.”. TED=“Trunc. Expon. Distr”. ${\cal N}\left(x\right)\stackrel{{\scriptstyle\mbox{\tiny$\Delta$}}}{{=}}\frac{e^{-x^{2}/2}}{\sqrt{2\pi}}$ and $\Phi\left(x\right)\stackrel{{\scriptstyle\mbox{\tiny$\Delta$}}}{{=}}\int_{u=-\infty}^{x}{\cal N}\left(u\right){\rm d}u.$ $\mathbb{X}$ | MaxEnt Prior | Activation
---|---|---
| $p_{0,x}({\bf x})$ | Name | $\lambda(\alpha)$ | Name
$\mathbb{R}^{N}$ | $\prod_{i=1}^{N}{\cal N}(x_{i})$ | Gaussian | $\alpha$ | linear
$\mathbb{P}^{N}$ | $\prod_{i=1}^{N}2{\cal N}(x_{i})$ | Trunc. Gauss. | $\alpha+\frac{{\cal N}(\alpha)}{\Phi(\alpha)}$ | TG
$\mathbb{P}^{N}$ | $\prod_{i=1}^{N}\frac{1}{2}e^{-x_{i}/2}$ | Expon. |  $\frac{2}{1-2\alpha}$, $\;\;\alpha<.5$ | Expon.
$\mathbb{U}^{N}$ |  1 | Uniform | $\frac{e^{\alpha}}{e^{\alpha}-1}-\frac{1}{\alpha}$ | TED

Additional details can be found in [26, 27].

The PDF of the input data of a PBN layer is given by the projected PDF (1),
which can be seen as an application of Bayes rule [21] $G({\bf x})=p_{m}({\bf
x}|{\bf z})g({\bf z})$, where $p_{m}({\bf x}|{\bf z})$ is the a posteriori
distribution, also equal to the J-function, i.e.

$p_{m}({\bf x}|{\bf z})=\frac{p_{0,x}({\bf x})}{p_{0,x}({\bf z})}=J({\bf x}).$
(7)

Note that $p_{m}({\bf x}|{\bf z})$ is a manifold distribution with support
only on a set of zero volume, the manifold (6). To see this, we integrate the
J-function on the manifold for fixed ${\bf z}$:

$\int_{{\bf x}\in{\cal M}({\bf z})}\;\frac{p_{0,x}({\bf x})}{p_{0,x}({\bf
z})}\;{\rm d}{\bf x}=\frac{1}{p_{0,x}({\bf z})}\;\int_{{\bf x}\in{\cal M}({\bf
z})}\;p_{0,x}({\bf x})\;{\rm d}{\bf x}.$ (8)

The denominator of (8) is independent of ${\bf x}$ on the manifold (6), and we
have the identity

$\int_{{\bf x}\in{\cal M}({\bf z})}\;p_{0,x}({\bf x})\;{\rm d}{\bf
x}=p_{0,x}({\bf z}).$

Therefore, $p_{m}({\bf x}|{\bf z})$ integrates to 1 on the manifold.

###  III-C Sampling a PBN

We now summarize the stochastic sampling of a multi-layer PBN. We assume that
the last layer of the network does not use an activation function (for
theoretical purposes we can ignore the activation functions because they are
assumed to be SMI and can be inverted). Let the network output be denoted by
${\bf z}$. Sampling proceeds as follows starting at the last (output) layer:

  1. 1.

Set the network layer counter to the last layer.

  2. 2.

We start by drawing a sample ${\bf z}$ from the network output feature
distribution $g({\bf z})$.

  3. 3.

Draw a sample from the manifold distribution $p_{m}({\bf x}|{\bf z})$, i.e.
draw a sample fom ${\cal M}({\bf z})$, weighted by the prior $p_{0,x}({\bf
x})$. How this is done depends on the prior distribution $p_{0,x}({\bf x})$,
which is selected from Table I. For the Gaussian case, drawing from
$p_{m}({\bf x}|{\bf z})$ is straight-forward, but for all the other cases, it
is necessary to find a starting solution, any member of ${\cal M}({\bf z})$,
then use a form of MCMC to draw a sample from set ${\cal M}({\bf z})$ weighted
by prior $p_{0,x}({\bf x})$ [23]. As an initial solution, one can either use a
linear programming solver, but a better starting point is the conditional mean
$\bar{{\bf x}}_{z}$ as given later in Section III-D. If no solution exists,
sampling has failed, return to step 1.

  4. 4.

If this current layer is the first network layer, then sampling is finished,
and ${\bf x}$ is a sample of the input data. If this is not the first layer,
then consider ${\bf x}$ to be the output ${\bf y}$ of the prevous layer’s
activation function. To proceed, work backward by inverting the activation
function and bias of the previous layer to obtain the previous layer’s linear
transformation output variable ${\bf z}$. Set layer counter to the previous
layer, and proceed with step 3.

###  III-D Asymptotic Form of PBN

As elegant as (7) may be, it is difficult to work with. For example,
integrating $p_{m}({\bf x}|{\bf z})$ to get the conditional mean,

$\bar{{\bf x}}_{z}=\mathbb{E}_{x|z}=\int_{{\bf x}\in{\cal M}({\bf z})}\;{\bf
x}\;p_{m}({\bf x}|{\bf z})\;{\rm d}{\bf x}$ (9)

only has a closed form in case of a Gaussian prior. This is a result of the
fact that the denominator $p_{0,x}({\bf z})$ is otherwise intractable.

Luckily, an “almost exact” approximation to $p_{0,x}({\bf z})$ can be found,
resulting in a simple functional form for $p_{m}({\bf x}|{\bf z})$ for which
the conditional mean can be found in closed form. Steven Kay [38] noted that
in cases like this (linear combinations of non-Gaussian random variables), the
moment generating function (MGF) of $p_{0,x}({\bf z})$ is in fact known
exactly, and can be inverted to find $p_{0,x}({\bf z})$ using the saddle point
approximation (SPA). The word “approximation” here is misleading because the
inversion of the MGF, is “almost” exact. Although it is only a first-order
approximation, the SPA integral has two important advantages: the integrand is
known exactly, and the integral is carried out at the saddle point. As $N$
becomes large, the integrand, as a result of the central limit theorem, takes
an approximately Gaussian shape at the saddle point, which is exploited by the
SPA to result in a solution for $p_{0,x}({\bf z})$ that is exact for all
practical purposes.

Note that ideally, $p_{m}({\bf x}|{\bf z})$ has support only on ${\cal M}({\bf
z})$, but the “almost exact” functional form for $p_{m}({\bf x}|{\bf z})$ ,
called the surrogate density [36, 26], is a proper distribution in its own
right, with support in all of $\mathbb{X}$. However, as $N$ becomes large, the
probability mass of the surrogate density converges around the manifold ${\cal
M}({\bf z})$ (see Section VII-A in [23]) and the resulting closed-form
expression for $\bar{{\bf x}}_{z}$ converges quickly to the true value [29].

The surrogate density as well as the prior $p_{0,x}({\bf x})$ for all cases of
$\mathbb{X}$ in Table I can be represented by a single exponential-form
density as follows

$p_{s}({\bf
x};\mbox{\boldmath$\alpha$},\alpha_{0},\beta)=\prod_{i=1}^{N}\;p_{e}(x_{i};\alpha_{i},\alpha_{0},\beta),$
(10)

where $p_{e}(x;\alpha,\alpha_{0},\beta)$ is a univariate distribution of the
exponential class

$\log p_{e}(x;\alpha,\alpha_{0},\beta)=(\alpha_{0}+\alpha)x+\beta x^{2}+\log
Z(\alpha,\alpha_{0},\beta),$ (11)

where $\mbox{\boldmath$\alpha$}=\\{\alpha_{1},\ldots\alpha_{M}\\}$, and $\log
Z(\alpha,\alpha_{0},\beta)$ is the the normalization factor necessary such
that $p_{e}(x;\alpha,\alpha_{0},\beta)$ integrates to 1. We also define the
corresponding theoretical activation function using the expected value of
$p_{e}(x;\alpha,\alpha_{0},\beta)$

$\lambda(\alpha,\alpha_{0},\beta)=\int_{x}\;x\;p_{e}(x;\alpha,\alpha_{0},\beta){\rm
d}x.$

In Table II, we provide $p_{e}(x;\alpha,\alpha_{0},\beta)$ for various choices
of $\mathbb{X}$, $\alpha_{0}$, and $\beta$. The prior is obtained for
$\mbox{\boldmath$\alpha$}={\bf 0}$

$p_{0,x}({\bf x})=p_{e}(x;{\bf 0},\alpha_{0},\beta).$

TABLE II: Extension of Table I showing $\alpha_{0}$, $\beta$, and $p_{e}(x;\alpha,\alpha_{0},\beta)$ for all cases.  $\mathbb{X}$ | $\alpha_{0}$ | $\beta$ | $p_{e}(x;\alpha,\alpha_{0},\beta)$ | Name
---|---|---|---|---
$\mathbb{R}^{N}$ | 0 | -.5 | ${\cal N}(x-\alpha)$ | Gaussian.
$\mathbb{P}^{N}$ | 0 | -.5 | $2{\cal N}(x-\alpha)$ | Trunc. Gauss. (TG)
$\mathbb{P}^{N}$ | -.5 | 0 | $\frac{(1-2\alpha)}{2}e^{-\frac{(1-2\alpha)}{2}x}$ | Expon.
$\mathbb{U}^{N}$ | 0 | 0 | $\left(\frac{\alpha}{e^{\alpha}-1}\right)\;e^{\alpha x}$ | Trunc. Expon. (TED)

By comparing Tabes I and II, it can be seen that the MaxEnt priors and
activation functions given in Table I are special cases of $p_{s}({\bf
x};\mbox{\boldmath$\alpha$},\alpha_{0},\beta)$ where
$\mbox{\boldmath$\alpha$}={\bf 0}$, and the activation functions are the
expected value as a function of $\alpha$. More precisely,

$p_{0,x}({\bf x})=p_{s}({\bf x};{\bf 0},\alpha_{0},\beta),$ (12)
$\lambda(\alpha)=\int_{x}\;x\;p_{e}(x;\alpha,\alpha_{0},\beta){\rm d}x,$ (13)

where $\alpha_{0}$ and $\beta$ are provided in Table II.

We restate the following theorem, first published in [26]:

###### Theorem 1

Let prior $p_{0,x}({\bf x})$ be written as (12) , having mean $\lambda({\bf
0})$ as given in (13). Then, the surrogate density for $p_{m}({\bf x}|{\bf
z})$ in (7) is $p_{s}({\bf x};{\bf W}{\bf h}_{z},\alpha_{0},\beta)$, where
${\bf h}_{z}$ is value of ${\bf h}$ that solves

${\bf W}^{\prime}\lambda\left({\bf W}{\bf h}\right)={\bf z}.$ (14)

Then, as $N$ becomes large, $p_{s}({\bf x};{\bf W}{\bf
h}_{z},\alpha_{0},\beta)\rightarrow p_{m}({\bf x}|{\bf z})$.

Furthermore, the mean of the surrogate density is asymptotically (for large
$N$) equal to the mean of $p_{m}({\bf x}|{\bf z})$ and is given by

$\bar{{\bf x}}_{z}=\lambda({\bf W}{\bf h}_{z}).$ (15)

For an outline of the proof, see [26].

In summary, the surrogate density converges to the posterior $p_{m}({\bf
x}|{\bf z})$, and so the mean of the surrogate density approaches the mean of
$p_{m}({\bf x}|{\bf z})$. This convergence occurs quickly as a function of
$N$, as has been demonstrated in certain cases (see Fig. 8 in [23]). It has
been shown that the error of the likelihood calculations when doing this are
negligible [36, 29]. The surrogate density mean $\bar{{\bf x}}_{z}$ given by
(15) enjoys numerous properties. As conditional mean estimator, it has many
well-known optimal properties [39]. Another special case of (15) corresponds
to autoregressive spectral estimation, which can be generalized for any linear
function of the spectrum, such as MaxEnt inversion of MEL band features [23].
A special case of (15) is mathematically the same as classical maximum entropy
image reconstruction [40, 41]. It can also be shown that ${\bf h}_{z}$ is the
maximum likelihood estimate of ${\bf h}$ under the likelihood function $p({\bf
x};{\bf h})=p_{s}({\bf x};{\bf W}{\bf h},\alpha_{0},\beta)$ [42].

As we have noted, the surrogate density has a close relationship to
$p_{0,x}({\bf z})$, with ${\bf h}_{z}$ being the saddlepoint for the SPA of
$p_{0,x}({\bf z})$. This follows from the fact that (14) is equivalent to the
saddle point equation for the distribution of the linear sum of independent
random variables (i.e. see , equation 25 in [38], on page 2245).

###  III-E Asymptotic Network and Deterministic PBN

The surrogate density can be seen as a layer in a generative network taking a
familiar form. We first simplify notation by defining the function
$\gamma({\bf h})={\bf W}^{\prime}\lambda\left({\bf W}{\bf h}\right).$ By
definition, $\gamma({\bf h}_{z})={\bf z}$. We also define the inverse : ${\bf
h}_{z}=\gamma^{-1}({\bf z}).$ The concept of $\gamma^{-1}({\bf z})$ is
illustrated in Figure 1 in which a two-layer network is shown. On the top of
the figure is the forward path, and on the bottom is the reconstruction path
that uses the asymptotic form of the network. We concentrate on just the first
layer by using the shortcut path “(layer bypass)”. Feature ${\bf z}$ is
converted to ${\bf h}_{z}$ through $\gamma^{-1}({\bf z})$, then multiplied by
${\bf W}$ to raise the dimension back to $N$, and finally passed through
activation function $\lambda(\;)$ to produce $\bar{{\bf x}}_{z}$. Optionally,
it can be passed to the generating distributions $p_{s}({\bf
x};\mbox{\boldmath$\alpha$},\alpha_{0},\beta)$ for stochastic generation,
indicated by the block “p” in the figure. According to the definition of
$\gamma^{-1}(\;)$, it is clear that ${\bf W}^{\prime}\bar{{\bf x}}_{z}={\bf
z},$ or in other words, the feature ${\bf z}$ is recovered exactly when
$\bar{{\bf x}}_{z}$ is processed by the forward path (i.e., a right-inverse).
The generative (reconstruction) path looks like the generative path of an
autoencoder with tied weights, except the activation function is replaced by
$\gamma^{-1}({\bf z})$.

Figure 1: Block diagram of 2-layer PBN in asymptotic form. Several versions of
PBN are shown in one figure, as implemented by dotted lines (see text).

This becomes more clear as we extend the network to two layers by not using
the shortcut path “(layer bypass)”. The forward path (top of figure) now
passes through a bias and activation function before linear transformation in
the second layer. The forward path is a standard multi-layer perceptron, or
“feed-forward neural network” (FFNN). The return path, although shown as a
separate network in the figure is tantamount to backing up through the FFNN.
With a second layer, something quite interesting happens in the reconstruction
path. Similar to the first layer, as we begin the reconstruction process by
applying the $\gamma^{-1}({\bf z})$ of the second layer to obtain ${\bf
h}_{z2}$, then apply linear transformation by matrix ${\bf W}_{2}$, we have
the choice of stochastic reconstruction through block “$p_{2}$” or
deterministic reconstruction using $\lambda_{2}(\;)$. The result is a
reconstruction of ${\bf x}_{2}$, the output of the first layer in the forward
path. It is a right-inverse reconstruction. To continue to the left, the
activation function and bias at the output of the first layer need to be
inverted in order to obtain a reconstruction of ${\bf z}$. But, for
deterministic reconstruction, this inversion of the activation function
cancels the application of $\lambda_{2}(\;)$ as long as the activation
function is the same. Therefore, using the reconstruction shortcut
“(activation bypass)” eliminates the need for inversion of the activation
function. For stochastic generation, the activation function is not used, so
the cancellation with its inverse does not occur, and this shortcut is not
possible.

When entirely using deterministic reconstruction and “(activation bypass)”,
the multi-layer reconstruction path looks like a perceptron network, but the
non-linearities are replaced by the functions $\gamma^{-1}({\bf z})$, which do
not operate element-wise. This type of network is called deterministic PBN
(D-PBN) [43, 44]. We have called this network a “network based on first
principles” [26] because it lends insight into how an auto-encoder would be
designed if it were not influenced by classical neural network topology.
Furthermore, the network is a right-inverse network, so is a two-way network.

###  III-F Solving the Saddle Point Equation and Sampling Failure

Solving (14) presents a computational challenge due to the need to repeatedly
invert matrices of the size $M\times M$, where $M$ is the output dimension of
a layer [38]. The problem can be avoided or mitigated by various methods [27],
some of which will be employed in the experiments.

Another potential problem with sampling a D-PBN is sampling failure. The
solution to (14) is not guaranteed to exist unless ${\bf z}={\bf
W}^{\prime}{\bf x}$ for some ${\bf x}$ in $\mathbb{X}$ [26]. When backing up
through a FFNN, this condition is only strictly met for the last layer (first
layer when working backwards), meaning that some input samples cannot be auto-
encoded. Luckily, the sampling efficiency, which is the fraction of samples
that succeed, can be driven close to 1 with proper initialization. Then
further training typically drives the sampling efficiency to one, resulting in
no sampling failures in the training set, or in the validation set [26, 28,
45].

Note that sampling a stochastic PBN is also subject to the problem of failed
samples and sampling efficiency, but training a stochastic PBN is not because
the likelihood function can be calculated for any sample. Therefore, a D-PBN
can be initialized by first training the network as a PBN, then later as a
D-PBN. A far more efficient way to initialize a D-PBN is to use the up-down
algorithm, which we mentioned in Section I-C. This can initialize a D-PBN with
sampling efficiency equal to or near 1. Then, D-PBN training can proceed if
the contributions of failed samples are weighted by 0. The sampling efficiency
then rises. To understand why, note that sampling occurs on a manifold (6),
the drawn sample of ${\bf x}$ is the conditional mean of ${\bf x}$ given ${\bf
z}$, which is also the weighted centroid of the manifold. This can be seen as
the “safest” solution, as far from the boundaries as possible, making it less
likely that a sample will fail. Sampling can also fail for some distorted or
unusual samples, which are poorly reconstructed anyway by other types of auto-
encoders.

###  III-G Generative Classifier Topologies using PBN

We now discuss ways to construct generative classifiers, and how these
topologies can be implemented using PBN, and which approach is preferred.
Consider data vector ${\bf x}$ and data class labels $i\in\\{1,2\ldots K\\},$
having prior class probabilities $P(i)$. From the discrete labels, we can
construct a label signal ${\bf y}$ that can take one of $K$ values ${\bf
y}\in\\{{\bf y}_{1},{\bf y}_{2}\ldots{\bf y}_{K}\\},$ suitable for processing
by a network, using one-hot encoding or other methods. We consider the
following three main approaches to constructing classifiers:

  1. 1.

Discriminative approach. In the discriminative approach, the task is to
directly estimate the conditional label probabilities $P(i|{\bf x})$ or the
posterior label signal densities $p({\bf y}|{\bf x})$. This is the standard
approach used in classifiers because it is direct and does not require
estimating probabilty distributions of the input data.

  2. 2.

Conditional distributions (generative). The simplest and most straight-forward
generative classifier topology is the so-called Bayesian classifier
constructed from the class-conditional data distributions $p({\bf x}|i)$,
which we also write as $p_{i}({\bf x})$. The label probabilities are
calculated as a second step using Bayes rule: $P(i|{\bf x})=\frac{p({\bf
x}|i)\;P(i)}{p({\bf x})},$ however the normalizing factor $p({\bf x})$ is
unnecessary if classification is implemented using

$\arg\max_{i}\\{p({\bf x}|i)\;P(i)\\}.$

Decision boundaries are formed by the comparison of likelihood function
values. To implement this with a PBN, we train a separate PBN on the training
data for each class to obtain $p({\bf x}|i)$. The advantage of training a
separate model on each class is the large class-selectivity that it can
provide. By training on each class separately, network weights become more
fine-tuned to a given class. In convolutional layers, convolutional kernels
take on patterns that act as basis functions for representing data of the
given class. This approach , however, has one very important disadvantage:
training a separate model on each class allows differences in initialization
and training to greatly influence the resulting models and cause imbalances,
which shift the decision boundaries, causing classification errors. There are
ways to mitigate this. For example in speaker recognition, GMM-UBM [46] uses a
single Gaussian mixture model (GMM) probability density estimate as a base
model (the universal background model - UBM) which is then modified for each
data class using just 1 step of the E-M algorithm. This eliminates the effects
of randomly initializing each model. The general idea can be used when
training PBNs. Class-dependent PBNs can be trained starting with a common
model trained on all classes, but the benefits of this are limited and it can
reduce class selectivity.

  3. 3.

Joint distribution (generative). The joint-distribution approach trains a
single model on the joint distribution of the class and label, $p({\bf x},{\bf
y})$, then classifies using

$\arg\max_{i}\\{p({\bf x},{\bf y}_{i})\\}.$

Although it avoids the model mismatch problem of the conditional distributions
approach, it has one important flaw. Generative models are only approximations
to the true distribution. In any approximation, compromises are made between
conflicting goals: model simplicity and model accuracy. Since we intend to use
joint distribution $p({\bf x},{\bf y})$ as a classifier, and there usually is
insufficient data to accurately estimate the true distribution, we must
compromise by using a simplified model. We hope that the compromise favors
high class selectivity, but we cannot guarantee this, nor can it be forced. In
short, the joint-distribution approach avoids the disadvantages of the
conditional distributions approach, but does not have its advantages, nor does
it have the advantages of the discriminative approach.

Furthermore, a problem also arises because ${\bf x}$ and ${\bf y}$ have
different forms, so the complete input data $\\{{\bf x},{\bf y}\\}$ is non-
homogeneous. This is especially true when ${\bf x}$ is multi-dimensional (i.e.
an image or spectrogram). It is not clear how to insert ${\bf y}$ into the
input image, or combine it with the image so that it can be meaningfully
processed by a convolutional network. This problem can be avoided by injecting
the label signal ${\bf y}$ into intermediate convolutional layers after some
down-sampling, or into dense (fully-connected) layers near the end of the
network. Examples of injecting labels into the data can be found in design of
class-dependent GANs [47] or in a deep belief network (DBN) [10].

The joint distribution approach has a further disadvantage when using PBNs. In
a PBN, a kind of vanishing-gradients problem arises with respect to the
labels. The higher the dimension a layer’s input data has, the greater the
effect of changing the layer weights has on the cost function. Therefore, the
labels will have less effect if they interact with weights at the network
output. As an aside, injecting label information in a PBN at multiple layers
is not justified theoretically because the likelihood function for such a
network may be intractable. In short, the joint distribution approach is
problematic for PBN and likely to suffer from poor class-selectivity in the
likelihood function.

To summarize, we prefer the conditional distributions approach for a PBN
classifier because it has the potential for high class selectivity as a result
of training networks individually on each class. But, we must seek ways to
avoid the problem of model imbalance and the resulting shift of decision
boundaries. This problem is solved by discriminative alignment.

##  IV Discriminative Alignment of PBN (PBN-DA)

For reasons mentioned in Section III-G, it is generally assumed that the
discriminative approach to classification is superior to the generative
approach. In short, estimating the class-dependent data distributions at high
dimension is much harder than predicting the class label [2, 1]. But, because
a PBN is a two-directional network one should be able to implement both
paradigms in one network. But how? There appears to be a contradiction:
discriminative classifiers are trained simultaneously on all data classes, but
the conditional distributions approach, which we have said we want to use (see
Section III-G), requires training separate networks on each class. A solution
to this dilemma was proposed by the method of discriminative alignment [29,
27, 48] in which each class-dependent network is trained simultaneously as a
generative model for the given class, but also as a discriminative model
against “all other classes”. This approach tends to “align” the network
weights, giving the generative model high selectivity against the other
classes, thus getting the best of both the generative and discriminative
approaches. By causing high selectivity against other classes, it mitigates
the problem of model mismatch because with high selectivity (i.e. high
likelihood function slope when moving in the direction of other classes), so
the decision boundaries will not shift much due to model mismatch.

To see this visually, we trained a simple PBN network on two-dimensional data
with two data classes. In Figure 2, on the top row, we see data from the two
classes (left), an intensity plot of the PBN likelihood function after
training on the first data class (center), and the corresponding likelihood
contours (right). As we would expect, the likelihood has a peak at the
location of the data. On the second row, the analogous plots are seen for a
network trained on the second data class. However, as can be seen by the
contour plots, there is not much selectivity against the other data class
because the slope is highest in the orthogonal direction. In short, the class-
selectivity appears to be poor.

We then re-trained the two PBNs with combined cost function. A classifier
(cross-entropy) cost component was added to the PBN log-likelihood cost,
giving the network the additional task of discriminating the two classes. In
rows 3 and 4 of the figure, we see the results. Interestingly, now the contour
plots show high selectivity against the other data class (i.e. high slope in
the direction that separates the two data classes). The contours have been
“discriminatively aligned”. Now, when classifying class 1 vs. class 2 using a
straight Bayesian likelihood classifier, the classification results will
resemble the properties of the discriminative classifier. In short, the best
qualities of both network types are realized. The method has shown very
promising results when compared to state of the art discriminative classifiers
[29, 27, 48].

Figure 2: From top to bottom: PBN trained on class 1 (red), PBN trained on
class 2 (blue), PBN trained on class 1 with discriminative alignment, PBN
trained on class 2 with discriminative alignment. Left column: input data,
center: likelihood surface, right: contour lines of likelihood surface.

###  IV-A Self-Combination of PBN-DA

As we have claimed, the PBN is a two-directional network, incorporating two
distinct network topologies in a single embodiment. In an $K$-class classifier
scenario, the PBN-DA consists of $K$ distinct networks, each trained on a
single class. The forward networks are trained as a classifier to discriminate
one class from all others, and the generative PBN likelihood function is
maximized over the training data of the same class. It follows, that one could
use two methods to construct a classifier from a trained PBN-DA (a) a straight
generative “conditional distributions” approach classifier based on comparing
the projected log-likelihood functions of the $K$ PBNs, and (b) a forward
classifier based on finding the maximum output statistic across all class
assumptions and all networks. We could therefore construct two distinct
classifiers from a single set of trained PBNs.

Alternatively, the discriminative influence can be integrated into and
combined with the generative classifier using a special output distribution.
Notice that the last term in the projected likelihood function for each PBN is
the output feature distribution, for example $g({\bf z})$ in (3). Let
$g_{m}({\bf y})$ be the output feature distribution for the network trained on
class $m$. Since $g_{m}({\bf y})$ is given, it can be constructed as an
indicator function for class $m$, so that it has a higher likelihood if ${\bf
y}$ equals the label signal for class $m$ (e.g. one-hot encoding), given by
${\bf y}_{m}$. For example,

$g_{m}({\bf y})=\delta[{\bf y}-{\bf y}_{m}].$ (16)

In the experiments, we approximated (16) with a TED ouput activation function
(See Table I) which, similar to sigmoid, has an output in the range $[0,1]$.
We assume the TED distribution

$g_{m}({\bf
y})=\prod_{i=1}^{K}\;\left(\frac{\alpha_{i,m}}{e^{\alpha_{i,m}}-1}\right)\;e^{\alpha_{i,m}y_{i}},$

where ${\bf y}=\\{y_{1}\ldots y_{K}\\}$, and $\alpha_{i,m}=\\{-C,\;i\neq
m,\;\;+C,i=m\\}.$ As can easily be verified, $g_{m}({\bf y})$ will have high
value when ${\bf y}$ is the one-hot encoding for class $m$, and a low value
otherwise. The constant $C$ can be used to control the relative importance of
the output statistic (i.e. the class labels) in the projected LF. This has an
effect similar to linear combining (ensembling) two classifier statistics (in
one network!), and it is possible to experimentally determine the optimal
value of $C$.

###  IV-B PBN-DA-HMM

The hidden Markov model (HMM) [49] was for a long time the state of the art in
acoustic event classification and modeling. The HMM owes its success to two
things:

  1. 1.

The Markov assumption, which allows modeling events with unknown start time,
and duration. The HMM is very forgiving against time-distorted or intermittent
time sequences, a common problem in real-world speech and acoustic events.

  2. 2.

The efficient forward algorithm for implementation of training and evaluation.

With the advent of convolutional neural networks (CNNs), the detection and
modeling of events with unknown temporal location and size was greatly
improved.

Despite the success of CNNs, it remains a possibility that the HMM is still
useful as a component of a PBN. The LF of a PBN is formed recursively from the
contributions of each layer according to the approach in equation (3). Suppose
a PBN is broken into two parts. Let the hidden variables coming out of the
first half be denoted by ${\bf h}$. Then, using PDF projection, the second
half of the network implements a PDF estimate ${\bf h}$, denoted by $G({\bf
h})$, e.g. equation (3). However, it can be that the dimension of ${\bf h}$ is
small enough that one can use well-known PDF estimation methods, such as HMM,
to replace, and possibly improve upon $G({\bf h})$. The Markovian assumption
exploited by the HMM, while over-simplified, provides an excellent compromise
between tractability and generative modeling accuracy. There are two
advantages to this, (a) the use of discriminative alignment in the pre-
training of the first part of the network lends discriminative information to
the features (class-selectivity), and (b) the use of HMM to replace the second
part of the network has the stated advantages of the Markov model.

To summarize the method of PBN-DA-HMM: we train a full PBN as PBN-DA on class
$m$, then when finished, shorten the network, tapping off the intermediate
features ${\bf h}_{m}$, and estimate the distribution $p_{m}({\bf h}_{m})$
using HMM. This is repeated for all classes $m$, and finally a class-specific
classifier is constructed using the method in Section II-D.

##  V Experiment 1: ESC50 Data Set

###  V-A Data Selection, Feature Selection, and Feature Extraction

The environmental sound classification data set (ESC50) [50] consists of $50$
data classes, with $40$ ten-second air-acoustic recordings in each data class.
The classes are diverse, and it is difficult to represent them well by one
feature extraction approach alone. As outlined in Section II, PDF projection
allows the use of multiple feature extraction approaches in a single common
generative model.

To this end, we attempted to match each data class with a feature set. This
was done by segmenting the time-series into overlapped Hanning-weighted
segments, then extracting log-MEL-band features. Time-series were then re-
synthesized by reconstructing the power spectrum from the log-MEL-band
features using maximum-entropy feature inversion [23, 26], as implemented by
$\bar{{\bf x}}_{z}$ in Section III-D, then reconstructing the complex-valued
FFT using random-phase, reconstructing the segment time-series by inverse-FFT,
Hanning-weighting and finally re-synthesizing time-series by using overlap-
add. The time-series were played back and compared acoustically with the
original. For almost all classes, a segment size and number of MEL bands could
be found to result in very good reconstruction, as determined subjectively by
ear. A large number of classes (23 classes) were well adapted to a segment
size of 768, 2/3 overlap, and 48 log-MEL-spaced bands. The 23 classes (classes
are numbered 0-49) were 0,1,6,9,10,11,13,16-19,21,23-25,27-30,36,45,47,49. The
features for each 10-second event consisted of 624 time samples, resulting in
a 624$\times$48 (time $\times$ freq) matrix.

As explained in Section II-D, PBN classifiers can be constructed using
multiple feature sets. But, this would require significantly more time and
processing resources and would detract from evaluating the benefits of
discriminative alignment together with HMM. Therefore, it was decided to put
off any multiple-feature-set experiments using all 50 classes to future work
and conduct a limited feasibility experiment on the 23-class subset using one
feature type.

In a previous publication [48], we reported the results of experiments for an
8-class subset of these same 23 classes. We now provide results for all 23
classes. There were then 23$\times$40 = 920 total events.

To partition the data, we used 4:1 random data holdout, selecting 10 testing
samples of the 40 samples of each class at random, and trained on the
remaining 30. We did this four times, independently. There were 230 testing
samples in each fold. The partitions were designated by letters A-D. For
reproducibility, we provide these features and data folds online [51].

###  V-B Network Architectures

We used two network architectures in the experiments, (a) the “PBN networks”,
which were trained separately on each of the 23 classes, and (b) a state of
the art “CNN classifier” trained jointly as a classifier on all classes.

PBN networks. We used a network similar to that used in [48]. The eight-layer
PBN network had three convolutional and 5 dense layers, ending with a
classifier layer of 23 neurons. The input data is 624$\times$48 (time $\times$
freq). Kernels in the three convolutional layers were 8 12$\times$16 kernels,
30 10$\times$5 kernels, and 120 21$\times$3 kernels respectively. Downsampling
was 3$\times$4, 3$\times$2 , and 3$\times$1\. Convolutional border modes were
“valid”. The dense layers had 512, 256, 128, 128, and 23 neurons. The last
layer is the cross-entropy classifier (output) layer. The output of the third
convolutional layer has dimension 16$\times$120, which is tapped off for HMM
processing. For the HMM, the data is seen as having 16 time steps and a
feature dimension of 120. The twenty-three class-dependent PBN networks used
linear activation at the output of the first 3 layers, in order to form a
Gaussian group [27], and thereby greatly reducing the required computation.
The remaining layers used the truncated Gaussian (TG) activation [52], similar
in behavior to softplus, not unlike leaky Relu [53], but continuous (see Table
I).

CNN classifier.

TABLE III: Modified ResNet architecture of the CNN.

layer name structure output size input BN (temporal axis) $624\times 48$ 2D
convolution $7\times 7$, stride$=(2,1)$ $312\times 48\times 16$ residual block
$\begin{pmatrix}3\times 3\\\ 3\times 3\end{pmatrix}\times 2$, stride$=1$
$155\times 46\times 16$ residual block $\begin{pmatrix}3\times 3\\\ 3\times
3\end{pmatrix}\times 2$, stride$=1$ $78\times 23\times 32$ residual block
$\begin{pmatrix}3\times 3\\\ 3\times 3\end{pmatrix}\times 2$, stride$=1$
$39\times 12\times 64$ residual block $\begin{pmatrix}3\times 3\\\ 3\times
3\end{pmatrix}\times 2$, stride$=1$ $20\times 6\times 128$ max pooling
$20\times 1$, stride$=1$ $1\times 6\times 128$ flatten BN $786$ dense
(embedding) linear $128$ sub-cluster AdaCos $4$ sub-clusters per class $23$

The CNN model consists of a modified ResNet architecture [54] with about $800$
k trainable parameters and is shown in Tab. III. It consists of an initial
temporal mean normalization operation, followed by a convolutional layer, 4
residual blocks, a temporal max-pooling operation and a dense layer with a
linear activation function. The residual blocks each consist of two
convolutional layers with kernels of size $3\times 3$, batch normalization, a
max pooling operation of size $2\times 2$ and use the ReLU activation
function. Futhermore, the whole network does not contain any bias terms. The
same architecture has been used successfully for the 8-class subset of ESC-50
[48], few-shot open-set acoustic event classification [55] and anomalous sound
detection [56]. The model learns to embed audio data onto a hypersphere of
fixed dimension and is trained by minimizing the angular margin loss sub-
cluster AdaCos [57]. This loss is an angular margin loss with an adaptive
scale parameter similar to AdaCos [58] but uses multiple instead of a single
centers for each class, called sub-clusters. In this work, we used an
embedding dimension of $128$ and $4$ randomly initialized sub-clusters for
each class that are not adapted during training. When training the model,
dropout with a probability of $50\%$ is applied to the hidden representations
before the last dense layer [59]. Additionally, two data augmentation
techniques were used to improve the performance of the CNN. As a first
technique, mixup [60] was applied to the input samples and their corresponding
classes using a mixing coefficient sampled from a uniform distribution.
Secondly, we applied random shifts up to $10\%$ of the temporal dimension of
the input representations (which was 640 time steps, so shifts were +/- 64
samples). The CNN is trained for $500$ epochs with a batch size of $8$ using
adam [61] and is implemented in Tensorflow [62]. To have a fair comparison, no
external data was used to train the CNN.

After training, each test sample is embedded into the embedding space by
applying the mapping learned by the CNN and projected to the unit sphere by
normalizing with respect to the Euclidean norm. Then, similarity scores for
each class are computed by taking the cosine similarity to the class-wise mean
embeddings of all normalized embeddings extracted from the training samples.
The class of the mean embedding with the shortest distance is used as the
classification result.

###  V-C Network Training and Initialization

A separate PBN was trained on each of the 23 classes using discriminative
alignment. About 1500 epochs were required. In addition to looking for
convergence of the PBN LF, we also sought to have zero training errors in the
discriminative task of “class $m$” against “all other classes”. The cross-
entropy discriminative cost function was scaled by 1000 before subtracting
from the LF.

These networks were then used as-is for PBN-DA, then shortened and used for
PBN-DA-HMM as described in Section IV-B. The networks were shortened by
tapping off the output of the third layer, with data shape 16$\times$120\. We
trained an HMM to estimate the probability distribution $p_{m}({\bf h}_{m})$
of this output map. We then added $\log p_{m}({\bf h}_{m})$ to the combined
log-J-function for the first 3 layers to obtain the projected input data LF
for PBN-DA-HMM.

For data augmentation during training, we used random circularly-indexed time
shifts with a maximum of $\pm 40$ time segments. Because the CNN and HMM were
significantly dependent on the random initialization, we always conducted
three trials, and averaged the results. Due to computational expense, it was
not practical to repeat the training of the PBN networks over multiple trials,
but the four-fold partioning provided adequate statistical diversity.

Note that the networks obtained by step (a) are used for PBN-DA in the
experiments, and from step (d) for PBN-DA-HMM.

###  V-D Computational Requirements for PBNs

Using PBN Toolkit software [45], with a GPU (NVIDIA GeForce GTX 1050 Ti), and
double pecision (float64) with a batch size of 230, we are able to compute one
epoch in 59 seconds. Double precision was needed to avoid problems during
inversion of matrices. It typically required about 1500 epochs to train each
of the 23 models, equating to about one day per class. With two computers, the
training was finished in less than 2 weeks. We used stochastic gradient
descent (or gradient ascent for LF), and ADAM optimization algorithm. Note
that when training one model, we use data from all classes, not just the
corresponding class. However, only data from the corresponding class is
applied to the gradient of the LF cost function, whereas data from all classes
is applied to the gradient of the discriminative cost function.

###  V-E Self-Combination of PBN-DA

We experimentally determined the optimal value of parameter $C$, as explained
in Section IV-A, by evaluating the number of classification errors for PBN-DA,
averaged over the four folds.

Figure 3: Mean errors for self-combination as a function of $C$.

As can be seen in Figure 3, there is a distinct minimum at about $C=2000$. The
value $C=2000$ was used in subsequent experiments for PBN-DA.

###  V-F Individual Results

TABLE IV: Number of errors (out of a total if 230) on each of the 4 data partitions for the various classifiers.  |  | Partition |
---|---|---|---
Algorithm | Trial | A | B | C | D | mean
CNN | 1 | 29 | 29 | 28 | 27 | 28.2
CNN | 2 | 36 | 27 | 23 | 23 | 27.2
CNN | 3 | 36 | 31 | 30 | 29 | 31.5
CNN | mean | 33.7 | 29 | 27 | 26.3 | 29
PBN-DA | n/a | 77 | 88 | 84 | 52 | 75.25
PBN-DA-HMM | 1 | 20 | 21 | 17 | 59 | 29.3
PBN-DA-HMM | 2 | 31 | 20 | 20 | 55 | 31.5
PBN-DA-HMM | 3 | 30 | 18 | 7 | 54 | 27.3
PBN-DA-HMM | mean | 27 | 19.7 | 14.7 | 56.0 | 29.3
CNN+PBN-DA | 1 | 30 | 27 | 27 | 28 | 28
CNN+PBN-DA | 2 | 27 | 26 | 24 | 24 | 25.2
CNN+PBN-DA | 3 | 28 | 26 | 25 | 27 | 26.5
CNN+PBN-DA | mean | 28.3 | 26.3 | 25.3 | 26.3 | 26.6
CNN+PBN-DA-HMM | 1 | 18 | 13 | 14 | 15 | 15
CNN+PBN-DA-HMM | 2 | 23 | 9 | 9 | 18 | 14.7
CNN+PBN-DA-HMM | 3 | 19 | 11 | 10 | 17 | 14.2
CNN+PBN-DA-HMM | mean | 20 | 11.0 | 11 | 16.7 | 14.7

The number of errors out of a totlal of 230 events in each of the 4 data
partitions are shown in Table IV for convolutional neural network (CNN), PBN-
DA, and PBN-DA-HMM classifiers. Three random trials are shown for all methods
except PBN-DA. Due to computational expense, the PBNs are trained just once.
For PBN-DA-HMM, three independent trials pertain just to the HMM.

First, it can be seen that the CNN classifier has significantly better
performance than PBN-DA. In previous publications [29, 27, 48], PBN-DA
competed more favorably with CNN, but it must be pointed out that in these
experments, CNN used more data augmentation during training (larger random
data shifts - 64 vs. 40 and mixup).

Second, it can be seen that CNN has virtually identical performance as PBN-DA-
HMM (29 versus 29.3 errors). This is remarkable, first because PBN-DA-HMM used
less data augmentation than CNN, but more significantly, because PBN-DA-HMM is
a generative conditional distributions classifier (See Section III-G). This is
no doubt a result of the incorporation of both generative and discriminative
approaches in a single network.

###  V-G Combined Results

It is well-known that combining the output of several classifiers usually
improves the classification performance, especially if the combined
classifiers (a) have comparable performance, and (b) they are based on
different methods or views of the data. When looking at the individual
performances in Table IV, it is clear that the stage is set for good
classifier combination.

Classifier combination (ensembling) results are shown as a function of the
linear combination factor for CNN with PBN-DA (in red) and CNN with PBN-DA-HMM
(in blue) in Figure 4, and are summarized in Table IV. Some interesting things
to note are that (a) PBN-DA-HMM works much better than PBN-DA, and combines
also much better with CNN, (b) PBN-DA-HMM performs better than CNN in three of
the four folds, and (c), that despite significantly worse performance in the
last fold, the combination still works exceptionally well. This is an
indication of the independence of the information being combined. It can also
be seen in the figure that the linear combination factor at which the minimum
number of errors is achieved is about the same for all four folds.

In Figure 5, the average of the four folds is seen, showing a factor of two
reduction in classification errors at the best point. In Table IV, we
summarize the classifier performance obtained by linear combination of CNN
with PBN-DA and with PBN-DA-HMM, which is obtained at the optimal combining
factor. This factor, about 6000, is held constant across the data folds222The
factor 6000 seems high, but this does not indicate anything about the relative
importance of the two statistics. It is only due to the wide difference in the
scale of the output statistics - log likelihood for PBN and cross-entropy
classifier cost for CNN..

Figure 4: Mean number of classification errors out of 230 samples, for each of
the four partitions when combining PBN-DA-HMM with CNN. Figure 5: Mean number
of classification errors out of 230 samples, averaged over the four folds when
combining PBN-DA-HMM with CNN.

###  V-H Reproducibility

Because we used a non-standard subset of ESC50 data, and non-standard data
partitions, we make the feature data available, as well as software and
instructions to reproduce the results in this paper as [45].

##  VI Experiment 2: Acoustic Trends Blue Fin Data Set

###  VI-A Data Description

The Australian Acoustic Trends Blue Fin data set [63] consists of acoustic
recordings from various hydrophones, along with a set of annotations of marine
mammal vocalizations that describe the bounding boxes of each vocalization
(start and end time, as well as start and end frequency). We collected 200
examples of from each of six call types, denoted by “BM-Ant-A ”, “BM.Ant-B ”,
“BM.Ant-Z ”, “Bm.D ”, “Bp-20Hz”, “Bp-Downsweep”. Example annotations are shown
in Figure 6 from class “Bp-Downsweep”. An attempt was made to select only
those annotations that were recognizeable. The first of the three samples in
the figure was deemed recognizable, and the last two were not and were
discarded. In order to achieve 200 samples per event, it was necessary to
relax the criteria somewhat. The data set is “dirty” in the sense that many
examples are weak and occur simultaneously with interfering noise and
interfering calls. A uniformly-sized time-window of 3072 samples at 250 Hz
sample rate (12 seconds) was extracted for each selected annotation.

Figure 6: Examples of annotations from Acoustic Trends Blue Fin Data Set of
type “Bp-Downsweep”.

For the current experiments, the time-series were converted to log-band energy
features with the following parameters: FFT size 384 with 128-sample shift
(2/3 overlap), 40 linear-spaced hanning-weighted frequency bands, resulting in
a $40\times 24$ feature map per event. Extracted .wav files and feature values
for the selected annotations have been made available online [64].

###  VI-B Networks

For the PBNs, a 5-layer network was used, consisting of two convolutional
layers, followed by three dense layers of 128, 32, and 6 nodes. The first
convolutional layer had 8 $7\times 13$ kernels with $2\times 4$ downsampling,
resulting in 8 output maps of $12\times 10$ (dimensions always given in
time$\times$freq). The second convolutional layer had 48 $4\times 10$ kernels
with $2\times 1$ downsampling, resulting in 48 output maps of $5\times 1$
(dimensions always given in time$\times$freq). Linear activation was used at
the output of the convolutional layers, and truncated Gaussian (TG) activation
function was used at the output of all dense layers (see Table I). One of the
primary reasons for the high computational load of PBN is the need to invert
matrices of dimenson $M\times M$, where $M$ is the total output dimension of a
network layer. One of the means of mitigating this is to use linear activation
functions in a few of the first network layers. These layers can then be
grouped together in what is called a Gaussian group with a low output
dimension [27]. We formed a Gaussian group from the first three layers.

For the benchmark CNN, the same network structure was used, but max-pooling
was used instead of downsampling, and TG activation functions were used at the
outputs of all layers.

###  VI-C Training

Six separate PBNs were trained (one on each data class) using discriminative
alignment and data augmentation of $+/-3$ samples random circular time shifts
and $+/-1$ sample random frequency shift. Vernier shifts (i.e. not quantized
to integer shifts) were accomplished by shifting the data in the frequency
domain using 2D-FFT. The self-combination factor $C=3$ was used during
training (see Section IV-A). For testing and evaluation, the value $C=300$ was
used. The cross-entropy classifier cost (against “all other classes”) was
multiplied by 2000 before being subtracted from the PBN log-likelihood value.
A learning rate of $2e^{-5}$ was used for 1500 epochs or until there were no
more training errors (in the target class vs “all other classes” FFNN
experiment).

For the benchmark CNN, we used drop-out regularization as well.

###  VI-D PBN-DA-HMM

To implement PBN-DA-HMM, we tapped the output of the second convolutional
layer, making a $5\times 48$ feature map, where the first dimension was time.
The PDF of this feature was estimated using a 4-state HMM, where each state
was represented by a Gaussian mixture (GMM) of 3 components. To prevent
covariance collapse, a value of 0.12 was added to the diagonal elements of the
GMM covariance matrices.

###  VI-E Results

Results are shown for PBN-DA (in red) and PBN-DA-HMM (in blue) in combination
with the benchmark CNN for all four data folds in Figure 7, and averaged over
the folds in Figure 8. Shown is total errors out of a total of 300 for each
data fold.

Figure 7: Results of PBN-DA vs. PBN-DA-HMM in combination with CNN for
Australian Blue Fin data. Shown is number of errors out of 300 for each of the
four data folds.  Figure 8: Results of PBN-DA vs. PBN-DA-HMM in combination
with CNN for Australian Blue Fin data. Shown is number of errors out of 300
averaged over the four data folds.

These results are strikingly similar to the first experiment, using a
completely different data set. As in the first experiment, PBN-DA performed
well, but not as well as the benchmark CNN, and resulted in significant error
reduction when combined with the CNN. Also, as in the first experiment, PBN-
DA-HMM performed exceedingly well, better than the benchmark CNN, and resulted
in a very significant 2:1 error reduction.

##  VII Conclusion and Future Work

In this paper, generative classifiers called PBN-DA and PBN-DA-HMM have been
described that combine both generative and discriminative methods in a single
network. Generative PBN networks are trained separately on each class using
discriminative alignment, ensuring that the generative models are selective
against the other data classes. Optionally, when (a) the input data has a time
dimension where convolutional layers can be applied, and (b) a suitable
reduced-dimension latent variable is present at the output of an intermediate
network layer, then the PBN-DA-HMM method can be applied. To do this, the PBNs
are shortened, tapping off at the suitable intermediate layer. The probability
density of these latent variables is estimated using HMMs, which leverage the
Markov assumption to create good probability density density estimates. When a
time-dimension is not avaibale, it should be possible to apply Gaussian
mixture model effectively.

Two separate experiments using vastly different acoustic data sets, confirm
that the approach does indeed attain some the best qualities of both
discriminative and generative approaches. In fact, it is seen that the
performance of PBN-DA-HMM is virtually identical to or better than a state of
the art CNN, and by linear combining with CNN attains a factor of two
reduction in error rate. The relatively high computational burden of PBN means
that the approach is suited to problems with high cost of errors, such as in
military and human safety applications.

Software implementations of PBN and related programs [51], [45] and data sets
for the current problems are made available online [64].

For future work, it is planned to extend the results to the full ESC50 data
set.

## References

  * [1] I. Goodfellow, Y. Bengio, A. Courville, and Y. Bengio, _Deep Learning_. Cambridge, MA: MIT press, 2016\.
  * [2] V. Vapnik, _The Nature of Statistical Learning_. Springer, 1999.
  * [3] Z. Zheng, X. Yang, Z. Yu, L. Zheng, Y. Yang, and J. Kautz, “Joint discriminative and generative learning for person re-identification,” in _CVPR 2019, Long Beach, CA_ , June 2019, pp. 2133–2142.
  * [4] J. Gordon and J. M. Hernández-Lobato, “Combining deep generative and discriminative models for bayesian semi-supervised learning,” _Pattern Recognition_ , vol. 100, 2020.
  * [5] Z. Tu, “Learning generative models via discriminative approaches,” in _2007 IEEE Conference on Computer Vision and Pattern Recognition_ , 2007, pp. 1–8.
  * [6] R. Raina, Y. Shen, A. Y. Ng, and A. McCallum, “Classification with hybrid generative/discriminative models,” in _NIPS 2003, Vancouver and Whistler British Columbia, Canada_ , December 2003.
  * [7] H. Liu and P. Abbeel, “Hybrid discriminative-generative training via contrastive learning,” _ArXiv_ , 2020. [Online]. Available: https://doi.org/10.48550/arXiv.2007.09070
  * [8] T. Jaakkola and D. Haussler, “Exploiting generative models in discriminative classifiers,” in _NIPS 1998, Denver_ , 1998.
  * [9] M. Schuster and K. Paliwal, “Bidirectional recurrent neural networks,” _IEEE Transactions on Signal Processing_ , vol. 45, no. 11, pp. 2673–2681, 1997.
  * [10] G. E. Hinton, S. Osindero, and Y.-W. Teh, “A fast learning algorithm for deep belief nets,” in _Neural Computation 2006_ , 2006.
  * [11] M. Welling, M. Rosen-Zvi, and G. Hinton, “Exponential family harmoniums with an application to information retrieval,” _Advances in neural information processing systems_ , 2004.
  * [12] P. Li and P. Nguyen, “On random deep weight-tied autoencoders: Exact asymptotic analysis, phase transitions, and implications to training,” _ICLR_ , 2019.
  * [13] S. Odaibo, “Tutorial: Deriving the standard variational autoencoder (vae) loss function,” 2019.
  * [14] C. Doersch, “Tutorial on variational autoencoders,” 2021. [Online]. Available: https://doi.org/10.48550/arXiv.1606.05908
  * [15] I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Lerchner, “Learning basic visual concepts with a constrained variational framework,” in _Proceedings of ICLR 201_ , 2017, pp. 595–597.
  * [16] I. Kobyzev, S. J. D. Prince, and M. A. Brubaker, “Normalizing flows: An introduction and review of current methods,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 43, no. 11, pp. 3964–3979, 2021.
  * [17] D. Nielsen, P. Jaini, E. Hoogeboom, O. Winther, and M. Welling, “Survae flows: Surjections to bridge the gap between vaes and flows,” in _NIPS 2020 (Virtual)_ , 2020.
  * [18] P. M. Baggenstoss and F. Govaers, “A comparison of pdf projection with normalizing flows and survae,” _arXiv_ , 2023.
  * [19] P. M. Baggenstoss, “The PDF projection theorem and the class-specific method,” _IEEE Trans Signal Processing_ , pp. 672–685, March 2003.
  * [20] ——, “Beyond moments: Extending the maximum entropy principle to feature distribution constraints,” _Entropy_ , vol. 20, no. 9, 2018. [Online]. Available: http://www.mdpi.com/1099-4300/20/9/650
  * [21] ——, “Maximum entropy PDF design using feature density constraints: Applications in signal processing,” _IEEE Trans. Signal Processing_ , vol. 63, no. 11, Jun. 2015.
  * [22] P. M. Baggenstoss and S. Kay, “Nonlinear dimension reduction by pdf estimation,” _IEEE Transactions on Signal Processing_ , 2022.
  * [23] P. M. Baggenstoss, “Uniform manifold sampling (UMS): Sampling the maximum entropy pdf,” _IEEE Transactions on Signal Processing_ , vol. 65, no. 9, pp. 2455–2470, May 2017.
  * [24] ——, “Class-specific model mixtures for the classification of time-series,” 2014.
  * [25] ——, “Class-specific model mixtures for the classification of acoustic time-series,” _IEEE Trans. AES_ , Aug. 2016.
  * [26] ——, “A neural network based on first principles,” in _ICASSP 2020, Barcelona (virtual)_ , Barcelona, Spain, Sep 2020.
  * [27] ——, “Using the projected belief network at high dimensions,” _Proceedings of EUSIPCO 2022, Belgrade_ , 2022.
  * [28] ——, “Applications of projected belief networks (PBN),” _Proceedings of EUSIPCO, A Corunã, Spain_ , 2019.
  * [29] ——, “Discriminative alignment of projected belief networks,” _IEEE Signal Processing Letters_ , Sep 2021.
  * [30] ——, “On the equivalence of hanning-weighted and overlapped analysis windows using different window sizes,” _IEEE Signal Processing Letters_ , vol. 19, no. 1, pp. 27–30, Jan 2012.
  * [31] Z. Zhu, S. Kay, and R. S. Raghavan, “Information-theoretic optimal radar waveform design,” _IEEE Signal Processing Letters_ , vol. 24, no. 3, pp. 274–278, 2017.
  * [32] J.-P. Nadal and N. Parga, “Nonlinear neurons in the low-noise limit: a factorial code maximizes information transfer,” _Network: Computation in Neural Systems_ , vol. 5, no. 4, pp. 565–581, 1994.
  * [33] D. Lin and X. Tang, “Conditional infomax learning: an integrated framework for feature extraction and fusion,” in _European conference on computer vision_. Springer, 2006, pp. 68–82.
  * [34] G. Deco and D. Obradovic, _An Information-Theoretic Approach to Neural Computing_. Springer, 1996.
  * [35] S. Kay, _Information-Theoretic Signal Processing and its Applications_. Sachuest Point Publishers, 2020.
  * [36] P. M. Baggenstoss, “On the duality between belief networks and feed-forward neural networks,” _IEEE Transactions on Neural Networks and Learning Systems_ , pp. 1–11, 2018.
  * [37] E. T. Jaynes, “Information theory and statistical mechanics i,” _Physical Review_ , p. 171–190, 1957.
  * [38] S. M. Kay, A. H. Nuttall, and P. M. Baggenstoss, “Multidimensional probability density function approximations for detection, classification, and model order selection,” _IEEE Transactions on Signal Processing_ , vol. 49, no. 10, pp. 2240–2252, Oct 2001.
  * [39] S. Kay, _Fundamentals of Statisticsl Signal Processing, Estimation Theory_. Prentice Hall, Upper Saddle River, New Jersey, USA, 1993.
  * [40] S. J. Wernecke and L. R. D’Addario, “Maximum entropy image reconstruction,” _IEEE Trans. Computers_ , vol. C-26, no. 4, pp. 351–364, 1977.
  * [41] G. Wei and H. Zhen-Ya, “A new algorithm for maximum entropy image reconstruction,” in _Proceedings of ICASSP-87_ , vol. 12, April 1987, pp. 595–597.
  * [42] O. Barndorff-Nielsen and D. R. Cox, “Edgeworth and saddle-point approximations with statistical applications,” _Journal of the Royal Statistical Society: Series B (Methodological)_ , vol. 41, no. 3, pp. 279–299, 1979.
  * [43] P. M. Baggenstoss, “Applications of projected belief networks (pbn),” in _Proceedings of EUSIPCO 2019_ , La Coruña, Spain, Sep 2019.
  * [44] ——, “Improved auto-encoding using deterministic projected belief networks and compound activation functions,” _Proceedings of EUSIPCO 2023, Helsinki_ , 2023.
  * [45] P. Baggenstoss, “PBN Toolkit,” accessed: 2023-10-28. [Online]. Available: http://class-specific.com/pbntk
  * [46] U. Bhattacharjee and K. Sarmah, “Gmm-ubm based speaker verification in multilingual environments,” _IJCSI International Journal of Computer Science Issues_ , vol. 9, no. 2, pp. 373–380, 2012.
  * [47] T. Miyato and M. Koyama, “cGANs with projection discriminator,” in _ICLR 2018_. IEEE, 2018.
  * [48] P. M. Baggenstoss and K. Wilkinghoff, “Novel generative classifier for acoustic events (accepted),” _Proceedings of EUSIPCO 2023, Helsinki_ , 2023\.
  * [49] L. R. Rabiner, “A tutorial on hidden Markov models and selected applications in speech recognition,” _Proceedings of the IEEE_ , vol. 77, no. 2, pp. 257–286, February 1989.
  * [50] K. J. Piczak, “ESC: Dataset for Environmental Sound Classification,” 2015. [Online]. Available: https://doi.org/10.7910/DVN/YDEPUT
  * [51] P. Baggenstoss, “CSF Toolkit,” accessed: 2021-10-28. [Online]. Available: http://class-specific.com/csftk
  * [52] ——, “New restricted Boltzmann machines and deep belief networks for audio classification,” _2021 ITG Speech Communication, Kiel (Virtual)_ , 2021\.
  * [53] A. L. Maas, A. Y. Hannun, and A. Y. Ng, “Rectifier nonlinearities improve neural network acoustic models,” in _30th International Conference on Machine Learning (ICML)_ , 2013.
  * [54] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Conference on Computer Vision and Pattern Recognition, CVPR_. IEEE, 2016, pp. 770–778.
  * [55] K. Wilkinghoff and F. Fritz, “On using pre-trained embeddings for detecting anomalous sounds with limited training data,” in _31st European Signal Processing Conference (EUSIPCO)_. IEEE, 2023.
  * [56] K. Wilkinghoff, “Design choices for learning embeddings from auxiliary tasks for domain generalization in anomalous sound detection,” in _International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2023.
  * [57] ——, “Sub-cluster AdaCos: Learning representations for anomalous sound detection,” in _International Joint Conference on Neural Networks (IJCNN)_. IEEE, 2021.
  * [58] X. Zhang, R. Zhao, Y. Qiao, X. Wang, and H. Li, “AdaCos: Adaptively scaling cosine logits for effectively learning deep face representations,” in _Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE, 2019, pp. 10 823–10 832.
  * [59] G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Improving neural networks by preventing co-adaptation of feature detectors,” _CoRR_ , vol. abs/1207.0580, 2012.
  * [60] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “Mixup: Beyond empirical risk minimization,” in _6th International Conference on Learning Representations (ICLR)_ , 2018.
  * [61] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in _3rd International Conference on Learning Representations (ICLR)_ , 2015\.
  * [62] M. Abadi _et al._ , “Tensorflow: A system for large-scale machine learning,” in _12th USENIX Symposium on Operating Systems Design and Implementation (OSDI)_ , 2016, pp. 265–283.
  * [63] B. Miller, K. Stafford, I. Van Opzeeland, D. Harris, F. Samaran, A. šIrović, S. Buchan, K. Findlay, N. Balcazar, S. Nieukirk, E. Leroy, M. Aulich, F. Shabangu, R. Dziak, W. Lee, and J. Hong, “An annotated library of underwater acoustic recordings for testing and training automated algorithms for detecting antarctic blue and fin whale sounds,” in _Australian Antarctic Data Centre_ , 2020. [Online]. Available: https://data.aad.gov.au/metadata/records/AcousticTrends_BlueFinLibrary
  * [64] P. Baggenstoss, “Selected events from acoustic trends blue fin data set,” accessed: 2023-10-28. [Online]. Available: http://class-specific.com/au6

  *[CNN]: convolutional neural network