# Label Uncertainty Modeling and Prediction for Speech Emotion Recognition
using _t_ -Distributions
††thanks: This work was supported by the Landesforschungsförderung Hamburg
(LFF-FV79), under the ”Mechanisms of Change in Dynamic Social Interaction”
project.

Navin Raj Prabhu  Signal Processing
Universität Hamburg
Hamburg, Germany
<EMAIL_ADDRESS>Nale Lehmann-Willenbrock  Industrial and
Organizational Psychology
Universität Hamburg
Hamburg, Germany
<EMAIL_ADDRESS>Timo Gerkmann  Signal Processing
Universität Hamburg
Hamburg, Germany
<EMAIL_ADDRESS>
###### Abstract

As different people perceive others’ emotional expressions differently, their
annotation in terms of arousal and valence are per se subjective. To address
this, these emotion annotations are typically collected by multiple annotators
and averaged across annotators in order to obtain labels for arousal and
valence. However, besides the average, also the uncertainty of a label is of
interest, and should also be modeled and predicted for automatic emotion
recognition. In the literature, for simplicity, label uncertainty modeling is
commonly approached with a Gaussian assumption on the collected annotations.
However, as the number of annotators is typically rather small due to resource
constraints, we argue that the Gaussian approach is a rather crude assumption.
In contrast, in this work we propose to model the label distribution using a
Student’s $t$-distribution which allows us to account for the number of
annotations available. With this model, we derive the corresponding Kullback-
Leibler divergence based loss function and use it to train an estimator for
the distribution of emotion labels, from which the mean and uncertainty can be
inferred. Through qualitative and quantitative analysis, we show the benefits
of the $t$-distribution over a Gaussian distribution. We validate our proposed
method on the AVEC’16 dataset. Results reveal that our $t$-distribution based
approach improves over the Gaussian approach with state-of-the-art uncertainty
modeling results in speech-based emotion recognition, along with an optimal
and even faster convergence.

###### Index Terms:

uncertainty, subjectivity, distribution learning, $t$-distribution, Bayesian
networks, speech emotion recognition

##  I Introduction

Emotions can be inner subjective experiences, but in order to become socially
relevant, they need to be expressed in social context (e.g., [1]). Therefore,
emotions are typically studied as emotional expressions that others
subjectively perceive and respond to [2]. A common theoretical backdrop for
analyzing emotions is the two-dimensional pleasure and arousal framework [3],
which describes emotional expressions in two continuous, bipolar, and
orthogonal dimensions: pleasure-displeasure (_valence_) and activation-
deactivation (_arousal_). One way in which emotions become expressed in social
interactions, and therefore accessible for social signal processing (SSP),
concerns speech signals. Speech emotion recognition (SER) research spans
roughly two decades [2], with ever improving state-of-the-art results. As a
consequence, affective sciences and SER has shown increasing prominence in
high-critical and socially relevant domains, e.g. health, security, and
employee well-being [2, 4, 5].

A crucial challenge when studying emotional expressions and trying to
establish a ground truth using the pleasure-arousal framework concerns the
significant degree of subjectivity surrounding the perceptions of these
expressions [2]. Commonly, majority voting [6] or evaluator-weighted mean
(EWE) [7] have been used as approximations to obtain ground-truth labels.
However, in the context of reliable real-world applications, it is required
for SER systems to not only model ground-truth labels but also account for
subjectivity based label uncertainty [2, 8].

Figure 1: Overview of the proposed architecture and loss
$\mathcal{L}_{\text{KL}}$.

In SER, label uncertainty has been approached using soft-labels [5], multi-
task learning (MTL) [9, 10], stochastic models [11, 12], and label
distribution learning [13, 14]. The subjective annotations of emotion creates
a label distribution which explains the uncertainty in emotions [5]. In this
light, label distribution learning techniques for label uncertainty in SER are
gaining research focus, with improved performances [13, 14]. However, a
problem with label distribution learning in SER is the limited annotations
available [14, 5], due to resource inefficient task of gaining more
annotations [5].

Emotion annotations as label distributions are usually modeled by making a
Gaussian assumption on them for mathematical convenience [13, 14]. However, a
Gaussian assumption with limited samples is not well justified [15, 16], as
the central limit theorem (CLT) which primarily backs Gaussian distributions
does not hold with insufficient numbers of samples [17]. Publicly available
SER datasets commonly comprise of only three to six annotations [18, 19, 20,
21, 22], and well agree that gaining more annotations is resource inefficient
[23, 5]. In this light, it is important for machine learning (ML) models to
account for the limited annotations and model the label distribution
accordingly. Alternately, Student’s $t$-distribution, also known as
$t$-distribution, is a probability distribution that also accounts for the
number of samples available while modeling [15]. Noting this, and the resource
constraints in gaining more annotations, the $t$-distribution becomes a more
appropriate choice for modeling emotion annotations.

In machine learning, two types of uncertainty can be distinguished. Label
uncertainty captures data inherent noise whereas model uncertainty accounts
for the uncertainty in model parameters [24]. Stochastic and probabilistic
models have mainly been deployed for uncertainty modeling [25, 26, 27]. Bayes
by Backpropagation (BBB) for Bayesian neural networks (BNN) [27] uses _simple
gradient updates_ to optimize weight distributions for _stochastic outputs_ ,
thereby are promising candidates for label distribution learning in SER.

In this paper, we propose to model emotion annotations as a $t$-distribution,
in contrast to a Gaussian assumption [13, 14]. To the best of our knowledge,
this is the first time the problem of limited emotion annotations is tackled
from an ML perspective, a common challenge in affective computing and SSP
[23]. For this, we adopt a BBB-based stochastic uncertainty model, as proposed
in [14], to include a $t$-distribution instead of a Gaussian. To this end, we
introduce a Kullback-Leibler (KL) divergence loss for label uncertainty that
quantifies distribution similarity between stochastic emotion predictions,
modeled as a Gaussian distribution, and _ground-truth emotion annotations_ ,
modeled as a _$t$ -distribution_. Subsequently, we present analyses to reveal
the benefits of using $t$-distribution over a Gaussian. Finally, we show that
the BBB-based uncertainty model trained on the proposed $t$-distribution based
KL-divergence loss can aptly capture label uncertainty with state-of-the-art
results, along with a robust loss curve.

##  II Related Work

###  II-A Ground-truth labels

To handle subjectivity in emotional expressions, annotations
$\\{y_{1},y_{2},..,y_{a}\\}$ for emotions are collected from $a$ annotators
[20, 22]. The _ground-truth label_ is then obtained as the mean $m$ over all
annotations from $a$ annotators [28, 29],

${m}=\dfrac{1}{a}\sum_{i=1}^{a}y_{i}.$ (1)

Alternatively, the EWE, which weights annotations with inter-annotator
correlations, has been proposed and referred to as the _gold-standard_
$\widetilde{m}$ [7]. Both $m$ and $\widetilde{m}$ based approximation of
ground-truth leads to loss of information on subjectivity [5].

Traditional SER approaches, given a raw audio sequence of $T$ frames
$\mathcal{X}=[x_{1},x_{2},...,x_{T}]$, aim to estimate either the $m_{t}$ or
$\widetilde{m}_{t}$ for each time frame $t\in[1,T]$, referred to as
$\widehat{m}_{t}$. The concordance correlation coefficient (CCC) has been
widely used as a loss function for this task [2]. For Pearson correlation $r$,
the CCC between $m$ and $\widehat{m}$, for $T$ frames, is formulated as

$\mathcal{L}_{\text{CCC}}(m)={\frac{2r\sigma_{m}\sigma_{\widehat{m}}}{\sigma_{m}^{2}+\sigma_{\widehat{m}}^{2}+(\mu_{m}-\mu_{\widehat{m}})^{2}}},$
(2)

where $\mu_{m}=\dfrac{1}{T}\sum_{t=1}^{T}m_{t}$,
$\sigma_{m}^{2}=\dfrac{1}{T}\sum_{t=1}^{T}(m_{t}-\mu_{m})^{2}$, and
$\mu_{\widehat{m}}$, $\sigma_{\widehat{m}}^{2}$ are obtained similarly for
$\widehat{m}$.

###  II-B Label uncertainty in SER

Alternative to exclusively modeling $m_{t}$ or $\widetilde{m}_{t}$, works have
attempted to model ground truth that also explains inter-annotator
disagreement, for example by means of soft labels [5] and entropy of
disagreement [30]. Fayek et al. [31] and Tarantino et al. [32] proposed to
learn soft labels instead of ${m}_{t}$ with improved performance. Steidl et
al. [30] quantified label uncertainty using the entropy measure, and trained a
model to minimize the difference in entropy between model outputs and
annotator disagreement. Sridhar et al. [5] proposed an auto-encoder based
learning technique to jointly model soft- and hard-labels of emotion
annotations, and subsequently estimating label uncertainty as the entropy on
soft-labels.

Label uncertainty has also been approached as a prediction task by estimating
either the moments of the distribution [9, 10] or the distribution in itself
[13, 14]. Han et al. [9, 10] used an MTL approach to model the unbiased
standard deviation $s$ of $a$ annotators as an auxiliary task,

$s=\sqrt{\dfrac{1}{a-1}\sum_{i=1}^{a}(y_{i}-m)^{2}}.$ (3)

Sridhar et al. [11] introduced a Monte-Carlo dropout model to obtain
uncertainty estimates from the distribution of stochastic outputs.
Foteinopoulou et al. [13] trained a MTL network using a KL divergence loss
that models emotion annotations as a uni-variate Gaussian with mean $m$ and
unknown variance. Raj Prabhu et al. [14] introduced a stochastic BNN and
trained them on Gaussian emotion annotations. Notwithstanding their improved
performances, in [13, 14] _Gaussian_ emotion annotations are assumed, despite
only having limited annotations. Apart from the apparent mathematical
incorrectness of this assumption, they are susceptible to unreliable $m$ and
$s$ for lower values of $a$ and sparsely distributed annotations [14].

###  II-C On distributions

A Gaussian distribution
$\widehat{\mathcal{Y}}\sim{\mathcal{N}}(\widehat{\mu},\widehat{\sigma}^{2})$
is a continuous probability distribution for a real-valued random variable
$y$, with general form of its probability density function [16]

$p(y\mid\widehat{\mu},\widehat{\sigma})={\frac{1}{\widehat{\sigma}{\sqrt{2\pi}}}}e^{-{\frac{1}{2}}\left({\frac{y-\widehat{\mu}}{\widehat{\sigma}}}\right)^{2}}.$
(4)

The parameters $\widehat{\mu}$ and $\widehat{\sigma}$ are the mean and
standard deviation of the distribution, respectively.

Due to its simplicity and intelligibility, Gaussian distributions are often
used to model random variables whose distribution are unknown [27, 13, 14].
Their importance is however backed by the CLT which only holds as the number
of observations of the random variable grows [17]. However, due to the
resource constraints in collecting annotations, in most human-behaviour
research [23] and in SER [20, 18, 19, 21], we do not have sufficient
annotations to assume a Gaussian distribution on them. As this is a common
challenge for reliable real-world applications, it is important for SER
algorithms to account for _limited_ annotations and model annotation
distributions accordingly. Kotz and Nadarajah [15], and, Bishop and Nasrabadi
[16], note that in scenarios of limited observations and samples the
$t$-distribution becomes more robust and realistic over a Gaussian.

Student’s $t$-distribution is a probability distribution that arises when
estimating the moments of a normally distributed population in _situations
where the sample size is small_ [15, 33], with the probability density
function given by [34, 35, 36],

$p(y\mid\nu,{{\mu}},{{\sigma}})=\frac{1}{\mathrm{B}(\frac{1}{2},\frac{\nu}{2})}\frac{1}{\sqrt{\nu{\sigma}^{2}}}\left(1+\frac{(y-{\mu})^{2}}{\nu{\sigma}^{2}}\right)^{-{\frac{\nu+1}{2}}},$
(5)

where $\nu$ denotes the degrees of freedom and B(., .) is the Beta function,
for Gamma function $\Gamma$, formulated as,

$B(i,j)=\dfrac{\Gamma(i)\,\Gamma(j)}{\Gamma(i+j)}.$ (6)

The density function (5) resembles the bell shape of a normally distributed
variable, except that it has heavier tails, meaning that it better captures
values that fall far from its mean [16, 15]. The degree of freedom $\nu$, also
known as the normality parameter, controls the normality of the distribution,
and is correlated with the ${\sigma}$ parameter [15, 16]. The standard
deviation ${\sigma}$ in (5) is scaled by $\nu$ and is formulated as

${\sigma}={{\sigma}\,\sqrt{\frac{\nu}{\nu-2}}{\text{ for }}}\nu>2.$ (7)

As $\nu$ increases, the $t$-distribution approaches the normal distribution
[36].

##  III Proposed $t$-distribution Label Uncertainty Model

To better represent subjectivity in annotations of emotional expressions, we
propose to estimate the _emotion annotation distribution_ $\mathcal{Y}_{t}$
for each frame $t$. For this, in contrast to a Gaussian assumption
$\mathcal{Y}_{t}\sim\mathcal{N}(m_{t},s_{t})$ [13, 14], we model the
annotations as a $t$-distribution with degrees of freedom $\nu$:

$\mathcal{Y}_{t}\sim\mathcal{N}(\nu,m_{t},s_{t}).$ (8)

Thus, the goal is to obtain an estimate $\widehat{\mathcal{Y}}_{t}$ of
$\mathcal{Y}_{t}$.

###  III-A Model architecture

We adopt an end-to-end architecture, initially proposed by Raj Prabhu et al.
[14], which uses a feature extractor [37] to learn temporal-paralinguistic
features from $x_{t}$, and a BBB-based uncertainty layer [27] to estimate
$\mathcal{Y}_{t}$. We include the $t$-distribution modeling as part of the
architecture, and the architecture proposed here can be seen in Figure 1.

Unlike a standard neuron which optimizes a deterministic weight $w$, the BBB-
based neuron learns a probability distribution on the weight
$P(w|\mathcal{D})$, parametrized by $\theta=(\mu_{w},\sigma_{w})$ using a
Gaussian $\mathcal{N}(\mu_{w},\sigma_{w})$, given the training data
$\mathcal{D}$ [27]. For an optimized $\theta$, the predictive distribution
$\widehat{\mathcal{Y}}_{t}$ for an audio frame $x_{t}$, is given by
$P(\widehat{y}_{t}|x_{t})=\mathbb{E}_{P(w|D)}[P(\widehat{y}_{t}|x_{t},w)]$,
where $\widehat{y}_{t}$ are realizations of $\widehat{\mathcal{Y}}_{t}$.
Stochastic outputs in BBB are achieved using multiple forward passes $n$ with
stochastically sampled weights $w$, thereby modeling
$\widehat{\mathcal{Y}}_{t}$ using the $n$ stochastic estimates. Following
[27], the BBB-based MLP is trained on the negative evidence lower bound
(ELBO),

$\mathcal{L}_{\text{BBB}}\approx\sum_{i=1}^{\textit{$n$}}\log
q(w^{(i)}|\theta)-\log P(w^{(i)})-\log P(D|w^{(i)}),$ (9)

where $q(w|\theta)$ is the variational posterior that minimizes the KL
divergence with the true Bayesian posterior, and $w^{(i)}$ is the $i^{th}$
sampled weight from $q(w|\theta)$. Finally, as suggested in [14], during
testing the uncertainty estimate $\widehat{s}_{t}$ is the standard deviation
of $\widehat{\mathcal{Y}}_{t}$, and mean estimate $\widehat{m}_{t}$ is the
realization $\widehat{y}_{t}$ obtained using the mean of the optimized weights
$\mu_{w}$.

###  III-B $t$-distribution label uncertainty loss derivation

To capture the label uncertainty, we derive a KL divergence based loss
function, between the Gaussian stochastic outputs $\widehat{\mathcal{Y}}$ and
the $t$-distribution ground-truth $\mathcal{Y}$. Note that assuming a Gaussian
distribution on the stochastic outputs $\widehat{\mathcal{Y}}$ is a fair
assumption as the number of stochastic outputs to model
$\widehat{\mathcal{Y}}$ can be controlled using $n$ in (9). As the number of
sample observations for a distribution approaches thirty a $t$-distribution
converges to a stable Gaussian [35, 36]. Noting this, we intend to choose a
$n$ greater than 30 and thereby assume $\widehat{\mathcal{Y}}$ to be Gaussian.
As a positive side effect, we result in deriving the KL divergence between a
Gaussian and a $t$-distribution, in contrast to between two $t$-distributions,
with the later involving mathematical complexities in calculating intractable
expectations for a loss function.

(a) $\widehat{\mathcal{Y}}_{t}\sim\mathcal{N}(0,0.5)$, and $\nu=6$

(b) $\widehat{\mathcal{Y}}_{t}\sim\mathcal{N}(0,1)$, and $\nu=6$

(c) $\widehat{\mathcal{Y}}_{t}\sim\mathcal{N}(0,1)$, and $\nu=12$

(d) $\widehat{\mathcal{Y}}_{t}\sim\mathcal{N}(0,1)$, and $\nu=30$

Figure 2: Analysis of the $t$-distribution based KL divergence
$\mathcal{L}_{KL}$ (13), in comparison with Gaussian $\mathcal{L}_{KL}$ (14).

For a Gaussian $\widehat{\mathcal{Y}}$ (see (4)), and a $t$-distributed
$\mathcal{Y}$ (see (5)), the $\mathcal{L}_{KL}$ is formulated as [38, 39],

$\mathcal{L}_{KL}=f_{KL}({\mathcal{Y}}_{t}\|\widehat{\mathcal{Y}}_{t})=H(\mathcal{Y}_{t},\widehat{\mathcal{Y}}_{t})-H({\mathcal{Y}}_{t}),$
(10)

where $\text{H}(.,.)$ is the cross-entropy between two distributions, and
$\text{H}(.)$ is the entropy of a distribution. Similar to [14], in (10), we
choose the true distribution $\mathcal{Y}_{t}$ to precede its estimate
$\widehat{\mathcal{Y}}_{t}$, promoting a mean-seeking approximation rather
than a mode-seeking one and capturing the full distribution [40].

The cross-entropy term $\text{H}(.,.)$ in (10), using (4), can be further
formulated as,

$\displaystyle\text{H}({\mathcal{Y}}_{t},\widehat{\mathcal{Y}}_{t})=-\int{\mathcal{Y}}_{t}(y)\,\,\log\,\widehat{\mathcal{Y}}_{t}(y)\,dy$
(11) $\displaystyle=$
$\displaystyle-\int\mathcal{Y}_{t}(y)\,\,\Big{[}\log\Big{(}\frac{1}{\sqrt{2\pi\widehat{\sigma}^{2}}}e^{-{\frac{1}{2}}\left({\frac{y-\widehat{\mu}}{\widehat{\sigma}}}\right)^{2}}\Big{)}\Big{]}dy$
$\displaystyle=$
$\displaystyle-\int\mathcal{Y}_{t}(y)\,\,\Big{[}-\frac{1}{2}\log(2\pi\widehat{\sigma}^{2})+\log\Big{(}e^{-{\frac{1}{2}}\left({\frac{y-\widehat{\mu}}{\widehat{\sigma}}}\right)^{2}}\Big{)}\Big{]}dy$
$\displaystyle=$
$\displaystyle\dfrac{1}{2}\log(2\pi\widehat{\sigma}^{2})+\int\mathcal{Y}_{t}(y)\,\,\Big{(}\frac{(y-\widehat{\mu})^{2}}{2\widehat{\sigma}^{2}}\Big{)}dy$
$\displaystyle=$
$\displaystyle\dfrac{1}{2}\log(2\pi\widehat{\sigma}^{2})\,+\,\frac{1}{2\widehat{\sigma}^{2}}\Big{[}\int\mathcal{Y}_{t}(y)y^{2}\,dy-2\widehat{\mu}\int\mathcal{Y}_{t}(y)\,y\,dy$
$\displaystyle\qquad\qquad\qquad\qquad\qquad\qquad+\,\widehat{\mu}^{2}\int\mathcal{Y}_{t}(y)\,dy\Big{]}$

Noting that $\int\mathcal{Y}_{t}(y)y^{2}\,dy={\mu}^{2}+{\sigma}^{2}$,
$\int\mathcal{Y}_{t}(y)\,y\,dy={\mu}$, and $\int\mathcal{Y}_{t}(y)\,dy=1$,
where ${\mu}$ and ${\sigma}$ are parameters of the $t$-distribution
$\mathcal{Y}_{t}$, $p(y\mid\nu,{\mu},{\sigma})$, the equation (11) becomes,

= 12 log(2π^σ^2) + 12^σ2[σ^2 + μ^2 - 2^μμ \+ ^μ^2 ] (12) = 12 log(2π^σ^2) +
σ2\+ (μ-^μ)22^σ2

Finally, using (12) in (10), our proposed KL divergence is

$\boxed{\mathcal{L}_{KL}=\frac{1}{2}\log(2\pi\widehat{\sigma}^{2})\,\,+\,\,\frac{{\sigma}^{2}\,+\,({\mu}-\widehat{\mu})^{2}}{2\widehat{\sigma}^{2}}-H({\mathcal{Y}}_{t}).}$
(13)

We implement (13) as a custom loss function using the pytorch package [41], by
extending the studentT sub-package111Code for the models and the loss
functions introduced are available at https://github.com/sp-uhh/label-
uncertainty-ser.

Similarly, as used in [14], the KL divergence between two Gaussians
$\mathcal{N}({\mu},\sigma^{2})$ and
$\mathcal{N}(\widehat{\mu},\widehat{\sigma}^{2})$ is given by [16, 41]

$\mathcal{L}_{KL}=\log\left(\frac{\widehat{\sigma}}{{\sigma}}\right)+\,\,\frac{{\sigma}^{2}\,+\,({\mu}-\widehat{\mu})^{2}}{2\widehat{\sigma}^{2}}-\frac{1}{2}.$
(14)

While the two loss-functions (14) and (13) have their second term in common,
two differences can be noted. Firstly, as (14) calculates the divergence
between two similar distributions, $\mathcal{Y}_{t}$ and
$\widehat{\mathcal{Y}}_{t}$, (14) includes the logarithm of the ratio between
the two Gaussian’s standard deviation in its formulation. However, in (13),
the deviations of $\widehat{\mathcal{Y}}_{t}$ and ${\mathcal{Y}}_{t}$ are
separately quantified using terms $\frac{1}{2}\log(2\pi\widehat{\sigma}^{2})$
and $H({\mathcal{Y}}_{t})$, respectively. Secondly, the number of annotators
is included in (13) by scaling ${\sigma}_{t}$ with normality factor $\nu$,
using (7). The implication of these differences, and the quantitative
differences between (14) and (13) are presented and analyzed in the following
section.

###  III-C $t$-distribution loss analysis

In contrast to [14], where the loss function is $\mathcal{L}_{KL}$ between two
Gaussians, and to [13], where $\mathcal{L}_{KL}$ is between a Gaussian and a
Dirac delta, our proposed loss (13) formulates the KL divergence between a
Gaussian and a $t$-distribution to capture label uncertainty when only limited
annotations are available. To validate the derivation and to further
understand the advantages of the $t$-distribution $\mathcal{L}_{KL}$ (13) over
the Gaussian $\mathcal{L}_{KL}$ (14), we plot the $\mathcal{L}_{KL}$ values as
a function of varying $\sigma$ of $\mathcal{Y}_{t}$, for (13) and (14). We
perform this analysis under four different scenarios, by varying parameters
$\widehat{\sigma}$ and ${\nu}$, i) Figure 2(a) for scenario
$\widehat{\sigma}=0.5$ and $\nu=6$, ii) Figure 2(b) for scenario
$\widehat{\sigma}=1.0$ and $\nu=6$, iii) Figure 2(c) for scenario
$\widehat{\sigma}=1.0$ and $\nu=12$, and, iv) Figure 2(d) for scenario
$\widehat{\sigma}=1.0$ and $\nu=30$.

From Figure 2, firstly, we see that $\mathcal{L}_{KL}$ behaves differently
when the ground-truth $\mathcal{Y}_{t}$ is modeled as a $t$-distribution (13),
in comparison to the Gaussian assumption (14). Specifically, from Figure 2(a),
for $\widehat{\sigma}=0.5$ and $\nu=6$, we see that the minimum
$\mathcal{L}_{KL}$ (13) is achieved only at ${\sigma}=0.61$, in contrast to
the Gaussian (14) $\widehat{\sigma}={\sigma}=0.5$. While the Gaussian attempts
exactly fitting the model to the ground-truth ${\sigma}=0.5$, the
$t$-distribution tries to fit on a more relaxed ${\sigma}=0.61$ by also
considering the reduced degree of freedom $\nu=6$. This behaviour is similar
to that observed during the confidence intervals calculation using a Gaussian
and $t$-distribution [42], where a $t$-distribution shows relaxation on
$\sigma$ with respect to $\nu$. Moreover, Bishop and Nasrabadi [16] associate
this relaxed ${\sigma}$ towards the increased robustness of the
$t$-distribution to outliers and sparse distributions.

Secondly, we note that the observed relaxation on ${\sigma}$ is dependent on
two factors, 1) the standard-deviation of the stochastic outputs
$\widehat{\sigma}$, and 2) the degree of freedom of the ground-truth $\nu$.
From figures 2(a) and 2(b), we see that, while $\nu$ is constant, the
relaxation on $\sigma$ _increases_ along with an increase in
$\widehat{\sigma}$. At $\widehat{\sigma}=0.5$ a relaxation of $0.11$ is made
by $t$-distribution (13) from $0.5$ to $0.61$, while a larger relaxation of
$0.22$ is made for $\widehat{\sigma}=1.0$. Similarly, from figures 2(c) and
2(d), we see that, while $\widehat{\sigma}$ is constant, as $\nu$ increases
the relaxation on $\sigma$ _decreases_. That is, the $t$-distribution (13)
starts behaving similar to that of the Gaussian, inline with literature that
states that as the degree of freedom $\nu$ of $t$-distribution increases, the
distribution converges into a Gaussian [35, 36, 15]. This is also inline with
our initial motivation behind using the $t$-distribution, which we expected to
account for the number of annotators $\nu$ while fitting on annotation
distribution $\mathcal{Y}$.

From a machine learning and SER perspective, from Figure 2, we note several
benefits that $t$-distribution loss term $\mathcal{L}_{KL}$ (13) brings forth
in-terms of label uncertainty modeling. Firstly, training on a
$t$-distribution based $\mathcal{L}_{KL}$ (13) leads to training on a relaxed
$s_{t}$, and thereby can lead to better capturing of the whole ground-truth
label distribution. Moreover, the resulting loss function is mathematically
more solid than a Gaussian assumption as in [13, 14], when less than _thirty_
annotations are available. Secondly, we note that the $t$-distribution
$\mathcal{L}_{KL}$ (13) values are always higher for lower values of $\sigma$
and $\widehat{\sigma}$, in all cases. This, in comparison to the Gaussian
$\mathcal{L}_{KL}$ (14), might lead to larger penalization of the model
through the $\mathcal{L}_{KL}$ loss, and may thereby promote better and
quicker convergence during training. Finally, the $t$-distribution
$\mathcal{L}_{KL}$ (13) also adapts to different datasets with different
number of annotators by considering the number of annotations available during
training.

###  III-D Training loss

The proposed uncertainty training loss is formulated as,

$\mathcal{L}=(1-\mathcal{L}_{\text{CCC}}(m))+\mathcal{L}_{\text{BBB}}+\mathcal{L}_{\text{KL}}.$
(15)

Intuitively, $\mathcal{L}_{\text{CCC}}(m)$ (2) optimizes for mean predictions
$m$, $\mathcal{L}_{\text{BBB}}$ (9) optimizes for BBB weight distributions,
and $\mathcal{L}_{\text{KL}}$ (13) optimizes for the label distribution
$\mathcal{Y}_{t}$ as a $t$-distribution.

##  IV Experimental Setup

###  IV-A Dataset

(a) For arousal.

(b) For valence.

Figure 3: Distribution of standard deviations $s_{t}$ in dataset [29]

To validate our model, we use the AVEC’16 [29] version of the RECOLA dataset
[20]. In this work, we only utilize the audio signals collected at 16 kHz,
from the multimodal signals recorded. The dataset consists of continuous
arousal and valence annotations by $a=6$ annotators at $40$ ms frame-rate. As
illustrated in Figure 3, in the AVEC’16 [29] dataset, arousal and valence
annotations are distributed on average with $\mu_{{m}}=0.01$ and
$\mu_{{m}}=0.11$, and $\mu_{s}=0.23$ and $\mu_{s}=0.14$, respectively, where
$\mu_{s}=\dfrac{1}{T}\sum_{t=1}^{T}s_{t}$. This reveals the significant level
of subjectivity present in the dataset, where $s_{t}$ distributions are heavy-
tailed with usually high $s_{t}$ and $\mu_{s}$. The dataset is divided into
speaker disjoint partitions for training, development and testing, with nine
$300$ s recordings each. As the annotations for the test partition are not
publicly available, all results are computed on the development partition.

###  IV-B Baselines

To evaluate the performance of our proposed approach, we use MTL- and BBB-
based uncertainty models [14]. From [10] we use the perception uncertainty
(_MTL PU_) and single-task models (_STL_), and from [14] the model uncertainty
(_MU_) and _label uncertainty_ (_MU $+$LU_) algorithms. Similar to [14], for a
fair comparison, we reimplemented the baselines from [10], thereby also
enabling us to compare the models in-terms of their $s$ estimates, which were
not presented in [10]. The method proposed in this work, $t$-distribution
based label uncertainty, will be called henceforth as _t-LU_ , for
convenience.

###  IV-C Choice of hyperparameters

The hyperparameters of the _feature extractor_ are fixed as suggested in [37],
similar to [43, 44]. The hyperparameters of the _uncertainty layer_ are
adopted from [14], who show state-of-the-art results in label uncertainty
modeling. For example, as _prior distribution_ $P(w)$ a simple Gaussian prior
with unit standard deviation $\mathcal{N}(0,1)$ was used. Similarly, the
_posterior distribution_ $P(w|D)$ is initialized using the same heuristics.

In this work, we assume a Gaussian on $\widehat{\mathcal{Y}}_{t}$, and noted
previously that $n\geq 30$ is required for the assumption to hold. In this
light, and keeping the time-complexity in mind, we fixed $n=30$. For training,
we use the Adam optimizer with learning rate $10^{-4}$. The batch size used
was 5, with a sequence length of 300 frames, $40$ ms each. All the models were
trained for a fixed 100 epochs. The best model is selected and used for
testing when best $\mathcal{L}$ (15) is observed on train partition.

TABLE I: Comparison on mean $m$, standard deviation $s$, and label
distribution estimations $\mathcal{Y}$, in terms of
$\mathcal{L}_{\text{ccc}}(m)$, $\mathcal{L}_{\text{ccc}}(s)$, and
$\mathcal{L}_{\text{KL}}$, respectively. Larger CCC indicates improved
performance as indicated by $\uparrow$. Lower KL indicates improved
performance as indicated by $\downarrow$. ** indicates that the respective
approach achieves statistically significant better results than _all_ other
approaches in comparison. * indicates that it achieves statistically
significant better results over _only some_ of the approaches in comparison.

(a) For arousal | $\mathcal{L}_{\text{ccc}}(m)\uparrow$ | $\mathcal{L}_{\text{ccc}}(s)\uparrow$ | $\mathcal{L}_{\text{KL}}\downarrow$
---|---|---|---
STL [10] | 0.7192 | - | -
MTL PU [10] | 0.7336 | 0.2861 | 0.7965
MU [14] | 0.7559 | 0.0764 | 0.6900
MU$+$LU [14] | 0.7437 | 0.3402 | 0.2576
_t_ -LU (proposed) | 0.7665** | 0.3752** | 0.2349**

(b) For valence | $\mathcal{L}_{\text{ccc}}(m)\uparrow$ | $\mathcal{L}_{\text{ccc}}(s)\uparrow$ | $\mathcal{L}_{\text{KL}}\downarrow$
---|---|---|---
STL [10] | 0.3878 | - | -
MTL PU [10] | 0.4163 | 0.0292 | 0.9981
MU [14] | 0.3248 | 0.0359 | 0.6334
MU$+$LU [14] | 0.2831 | 0.0422 | 0.4054
_t_ -LU (proposed) | 0.3768* | 0.0481* | 0.3914*

(a) For arousal.

(b) For valence.

Figure 4: Label distribution $\mathcal{Y}_{t}$ estimation results for a test
subject.

###  IV-D Validation measures

To validate the proposed method’s _mean_ and _standard deviation_ estimates,
we use $\mathcal{L}_{\text{CCC}}(m)$ and $\mathcal{L}_{\text{CCC}}(s)$
metrics, respectively, widely used in literature [37, 44, 10]. However,
$\mathcal{L}_{\text{CCC}}(m)$ and $\mathcal{L}_{\text{CCC}}(s)$ validates mean
and standard deviation estimates _separately_. To further jointly validate
mean and standard deviation estimates, as label distribution
$\widehat{\mathcal{Y}}_{t}$, we use the $\mathcal{L}_{\text{KL}}$ measure. A
similar measure is used in [14], but with a Gaussian assumption on
$\mathcal{Y}_{t}$ and hence can be biased. However, for a fair comparison, we
validate all the models in comparison using $\mathcal{L}_{\text{KL}}$ based on
their respective distribution assumptions on $\mathcal{Y}_{t}$, as the models
are trained in a similar fashion. The proposed _t-LU_ method is validated and
trained on the $t$-distribution $\mathcal{L}_{\text{KL}}$ (13), and the
baselines from [10] and [14] are validated and trained on Gaussian
$\mathcal{L}_{\text{KL}}$ (14). Nevertheless, from the experiments we also
noted that the proposed _t-LU_ performs better in-terms of both (13) and (14).

##  V Results and Discussion

Table I(b) shows the average performance of the baselines and the proposed
model, in terms of their mean $m$, standard deviation $s$, and distribution
$\widehat{\mathcal{Y}}_{t}$ estimations, $\mathcal{L}_{\text{CCC}}(m)$,
$\mathcal{L}_{\text{CCC}}(s)$ and $\mathcal{L}_{KL}$, respectively.
Comparisons with respect to $\mathcal{L}_{\text{CCC}}(s)$ and
$\mathcal{L}_{KL}$ are not presented for the STL as this algorithm does not
contain uncertainty modeling. Statistical significance is estimated using one-
tailed $t$-test on error distributions, asserting significance for _p_ -values
$\leq 0.05$, similar to [5].

###  V-A Comparison on mean estimates

In terms of mean estimates $\mathcal{L}_{\text{CCC}}(m)$ of _arousal_ , Table
I(a) shows that the proposed _t_ -LU model performs best in comparison with
the baselines, with statistical significance. While the _t_ -LU model achieves
an $\mathcal{L}_{\text{CCC}}(m)$ of $0.7665$, the BBB-based baselines, MU$+$LU
and MU, achieve $0.7437$ and $0.7559$, respectively. This also reveals the
superiority of the proposed $t$-distribution $\mathcal{L}_{KL}$ (13) over the
Gaussian $\mathcal{L}_{KL}$ (14), with _t_ -LU outperforming MU$+$LU.
Moreover, in [14], it was noted that training on KL loss _with the Gaussian
assumption_ (14) makes a compromise on $\mathcal{L}_{\text{CCC}}(m)$
performances with improving $\mathcal{L}_{\text{CCC}}(s)$. However, the
proposed _t_ -LU is free from this compromise with _t_ -LU outperforming MU.
Finally, we also note that the proposed _t_ -LU performs significantly better
than the MTL-based uncertainty baseline MTL PU, and also the STL which does
not model uncertainty.

In terms of mean estimates $\mathcal{L}_{\text{CCC}}(m)$ of _valence_ ,
concerning Table I(b), it is noted that the the proposed _t_ -LU performs
significantly better than the BBB-based models, MU$+$LU and MU. While the _t_
-LU model achieves an $\mathcal{L}_{\text{CCC}}(m)$ of $0.3768$, the BBB-based
baselines, MU$+$LU and MU, achieve $0.2831$ and $0.3248$, respectively.
Similar to arousal, for valence, we see that _t_ -LU is free from compromises
on $\mathcal{L}_{\text{CCC}}(m)$, as in [14]. However, the proposed _t_ -LU
does not improve over the MTL-based baselines, MTL PU and STL. It is a common
trend in SER literature that the audio modality inadequately explains the
valence dimension of emotion [43]. However, a probable explanation for this is
that the MTL-based architectures are generally better than the BBB-based, in-
terms of $\mathcal{L}_{\text{CCC}}(m)$ of valence. Results present by Han et
al. [10] also show similar behaviour where MTL-based architectures show
significant improvements _specifically in mean estimates of valence_ [10].

###  V-B Comparison on uncertainty estimates

While the proposed _t_ -LU achieves significantly improved results over the
baselines for mean estimates, especially for the arousal dimension, the main
goal of this paper is to aptly capture label uncertainty in emotions, in terms
of $\mathcal{L}_{\text{CCC}}(s)$ and $\mathcal{L}_{KL}$. Concerning the
uncertainty estimates in arousal, Table I(a) shows that the proposed _t_ -LU
achieves state-of-the-art results, with significant improvements over the
baselines. For instance, the MU$+$LU model, the best performing baseline,
achieves a $\mathcal{L}_{\text{CCC}}(s)$ and $\mathcal{L}_{KL}$ of $0.3402$
and $0.2576$, respectively, while _t_ -LU significantly improves by achieving
$0.3752$ and $0.2349$, respectively. This performance, in terms of both the
measures, explains the advantage of the $t$-distribution based KL loss term
(13) in label uncertainty modeling. The $t$-distribution $\mathcal{L}_{KL}$,
as seen in Figure 2, promotes the model to fit on a more relaxed $s_{t}$ and
penalizes more for tighter standard deviations, thereby leading to better
capturing the label distribution.

For valence, unlike the $\mathcal{L}_{\text{CCC}}(m)$ performance, Table I(b)
shows that the proposed _t_ -LU achieves improved performance in-terms of the
_uncertainty estimates_ , over all the baselines in comparison. It is further
noted that _t_ -LU improves with statistical significance over the MTL-based
baselines, however improves without statistical significance in comparison
with the BBB-based baselines. While the MU$+$LU, best performing baseline,
achieves a $\mathcal{L}_{\text{CCC}}(s)$ and $\mathcal{L}_{KL}$ of $0.0422$
and $0.405$, respectively, _t_ -LU improves by achieving $0.0481$ and
$0.3914$, respectively. However, the lack of statistical significance over
some baselines, as well as the generally rather low performance of all
approaches, could be owed to the common observation that speech inadequately
explains valence [43].

###  V-C Qualitative analysis on distribution estimation

To further validate the results, we plot the mean $m$ and standard deviation
$s$ of the estimated distributions for a test subject, seen for arousal in
Figure 4(a), and valence in Figure 4(b). Moreover, some parts of the plots are
boxed and numbered to note performance differences. For arousal, in Figure
4(a), further backing the results in Table I(a), the proposed $t$-LU model
better captures $m$ and $s$ of the annotation distribution, in comparison with
MU and MU$+$LU. For example, in the box numbered _2_ in Figure 4(a), the
proposed $t$-LU captures the whole annotation distribution $\mathcal{Y}_{t}$,
resembles the _Ground-truth - s_ best. This further highlights the robustness
of training on a relaxed $\sigma_{t}$ through a $t$-distribution. Moreover, in
box _2_ , we also note that $t$-LU, by best capturing $s$, also improves
notably in terms of the mean estimates $m$. This can be noted in all boxes _1_
-_4_ , where $t$-LU best captures the whole arousal annotation distribution,
by improving on both $s$ and $m$ estimations.

For valence, in Figure 4(b), backing results in Table I(b), we note that the
proposed $t$-LU improves significantly in-terms of the mean estimates $m$,
with only small improvements on standard deviation estimates $s$. This can be
seen for instance in box _1_ , where $t$-LU notably improves in terms of $m$
estimations, over MU and MU$+$LU, while only small improvements in terms of
$s$ estimations can be observed. Overall, we note that the proposed speech-
based uncertainty model better captures the annotation distribution of arousal
than that of valence.

###  V-D Analysis on training loss curve

(a) For arousal.

(b) For valence.

Figure 5: Loss curve comparison between Gaussian $\mathcal{L}_{KL}$ (14) and
proposed $t$-distribution $\mathcal{L}_{KL}$ (13).

To further study the advantages of the proposed $t$-distribution
$\mathcal{L}_{KL}$ (13) during the training phase, we compare the testing loss
curve of (13) with the Gaussian $\mathcal{L}_{KL}$ in MU$+$LU (14). The
comparison can be seen for arousal in Figure 5(a), and for valence in Figure
5(b).

For both the arousal and the valence dimension, Figure 5 illustrates two
crucial advantages of the proposed $t$-distribution $\mathcal{L}_{KL}$ (13) as
a loss term in the training phase. Firstly, we see that in the initial epochs,
before epoch 20, the proposed loss term converges quicker than the Gaussian
$\mathcal{L}_{KL}$ (14). This is the result of the proposed $\mathcal{L}_{KL}$
(13) loss term which penalizes more for lower $s_{t}$ values, in comparison to
the Gaussian $\mathcal{L}_{KL}$ (14), as seen in Section III-C, thereby
achieving faster convergence. Secondly, it is noted that during the later
epochs, after 60 epoch, the Gaussian $\mathcal{L}_{KL}$ (14) shows signs of
overfitting with increasing testing loss. However, at the same time, the
proposed $\mathcal{L}_{KL}$ (13) converges to the best minima during the later
epochs. The proposed $\mathcal{L}_{KL}$ achieves minima $\mathcal{L}_{KL}$ at
the epoch 85, with $\mathcal{L}_{KL}$ of 0.2349 for arousal and 0.3914 for
valence, while the Gaussian achieves a minima well before the later epochs, at
epoch 54, with $\mathcal{L}_{KL}$ of 0.2532 for arousal and 0.3958 for
valence. The proposed $\mathcal{L}_{KL}$ is free from overfitting in the later
stages of training and also learns the optima at this stage. This behaviour
can be attributed to the nature of the proposed $\mathcal{L}_{KL}$ which
promotes the model to learn a more relaxed $s_{t}$, as seen in Section III-C,
thereby introducing more regularization to the model, resulting in preventing
overfitting and converging on an improved $s_{t}$.

##  VI Conclusions

Label uncertainty modeling in emotion recognition is commonly approached by
assuming a Gaussian distribution on the ground-truth emotion annotations,
however with only limited annotations. In contrast, in this work, we assumed a
Student’s $t$-distribution on the ground-truth emotion annotations, which also
accounts for the number of annotations available. This is the first time in
literature an attempt was made to handle the limited emotion annotations
available, from a machine learning perspective. For this, we proposed and
derived a KL divergence based loss term that aims to capture emotion
annotation distribution _as a $t$-distribution_. The derived $t$-distribution
loss term is also mathematically more sound than the Gaussian assumption, for
limited annotations. Subsequently, we showed that the proposed
$t$-distribution loss term leads to training on a relaxed standard deviation,
which is adaptable with respect to the number of annotations available.
Moreover, we also validated our approach on a publicly available dataset.
Quantitative analysis of the results showed that the proposed $t$-distribution
loss term improves over the Gaussian assumption with state-of-the-art results
in mean and standard deviation estimations, in-terms of both the CCC and KL
divergence measures. Finally, we also showed, through the analysis of the loss
curves, that the proposed loss term leads to faster and improved convergence,
and is less prone to overfitting.

## References

  * [1] G. A. Van Kleef, “How emotions regulate social life: The emotions as social information (easi) model,” _Current directions in psychological science_ , vol. 18, no. 3, pp. 184–188, 2009.
  * [2] B. W. Schuller, “Speech emotion recognition: two decades in a nutshell, benchmarks, and ongoing trends,” _Communications of the ACM_ , vol. 61, no. 5, pp. 90–99, Apr. 2018.
  * [3] J. A. Russell, “A circumplex model of affect.” _Journal of personality and social psychology_ , vol. 39, no. 6, p. 1161, 1980.
  * [4] D. Dukes, K. Abrams, R. Adolphs, M. E. Ahmed, A. Beatty, K. C. Berridge, S. Broomhall, T. Brosch, J. J. Campos, Z. Clay _et al._ , “The rise of affectivism,” _Nature Human Behaviour_ , pp. 1–5, 2021.
  * [5] K. Sridhar, W.-C. Lin, and C. Busso, “Generative approach using soft-labels to learn uncertainty in predicting emotional attributes,” in _2021 9th International Conference on Affective Computing and Intelligent Interaction (ACII)_. IEEE, 2021, pp. 1–8.
  * [6] C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan, “Iemocap: Interactive emotional dyadic motion capture database,” _Language resources and evaluation_ , vol. 42, no. 4, pp. 335–359, 2008.
  * [7] M. Grimm and K. Kroschel, “Evaluation of natural emotions using self assessment manikins,” in _IEEE Workshop on Automatic Speech Recognition and Understanding, 2005._ IEEE, 2005, pp. 381–385.
  * [8] H. Gunes and B. Schuller, “Categorical and dimensional affect analysis in continuous input: Current trends and future directions,” _Image and Vision Computing_ , vol. 31, no. 2, pp. 120–136, 2013.
  * [9] J. Han, Z. Zhang, M. Schmitt, M. Pantic, and B. Schuller, “From hard to soft: Towards more human-like emotion recognition by modelling the perception uncertainty,” in _Proceedings of the 25th ACM Int. Conf. on Multimedia_ , 2017, pp. 890–897.
  * [10] J. Han, Z. Zhang, Z. Ren, and B. Schuller, “Exploring perception uncertainty for emotion recognition in dyadic conversation and music listening,” _Cognitive Computation_ , pp. 1–10, 2020.
  * [11] K. Sridhar and C. Busso, “Modeling uncertainty in predicting emotional attributes from spontaneous speech,” in _ICASSP - IEEE Int. Conf. on Acoustics, Speech and Signal Processing_ , 2020, pp. 8384–8388.
  * [12] M. Kumar, E. Sanchez, G. Tzimiropoulos, T. Giesbrecht, and M. Valstar, “Stochastic Process Regression for Cross-Cultural Speech Emotion Recognition,” in _Proc. Interspeech 2021_ , 2021, pp. 3390–3394.
  * [13] N. M. Foteinopoulou, C. Tzelepis, and I. Patras, “Estimating continuous affect with label uncertainty,” in _2021 9th International Conference on Affective Computing and Intelligent Interaction (ACII)_. IEEE, 2021, pp. 1–8.
  * [14] N. R. Prabhu, G. Carbajal, N. Lehmann-Willenbrock, and T. Gerkmann, “End-to-end label uncertainty modeling for speech-based arousal recognition using bayesian neural networks,” 2022, submitted to INTERSPEECH 2022, https://arxiv.org/abs/2110.03299.
  * [15] S. Kotz and S. Nadarajah, _Multivariate t-distributions and their applications_. Cambridge University Press, 2004.
  * [16] C. M. Bishop and N. M. Nasrabadi, _Pattern recognition and machine learning_. Springer, 2006, vol. 4, no. 4.
  * [17] H. Fischer, _A history of the central limit theorem: From classical to modern probability theory_. Springer Science & Business Media, 2010.
  * [18] L. Martinez-Lucas, M. Abdelwahab, and C. Busso, “The MSP-conversation corpus,” in _Interspeech 2020_ , Shanghai, China, October 2020, pp. 1823–1827.
  * [19] R. Lotfian and C. Busso, “Building naturalistic emotionally balanced speech corpus by retrieving emotional speech from existing podcast recordings,” _IEEE Transactions on Affective Computing_ , vol. 10, no. 4, pp. 471–483, October-December 2019.
  * [20] F. Ringeval, A. Sonderegger, J. Sauer, and D. Lalanne, “Introducing the RECOLA multimodal corpus of remote collaborative and affective interactions,” in _10th IEEE Int. Conf. and Workshops on Automatic Face and Gesture Recognition (FG)_ , 2013, pp. 1–8.
  * [21] J. Kossaifi, R. Walecki, Y. Panagakis, J. Shen, M. Schmitt, F. Ringeval, J. Han, V. Pandit, A. Toisoul, B. Schuller _et al._ , “Sewa db: A rich database for audio-visual emotion and sentiment research in the wild,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 43, no. 3, pp. 1022–1040, 2019.
  * [22] N. Raj Prabhu, C. Raman, and H. Hung, “Defining and Quantifying Conversation Quality in Spontaneous Interactions,” in _Comp. Pub. of 2020 Int. Conf. on Multimodal Interaction_ , 2020, pp. 196–205.
  * [23] D. Gatica-Perez, “Automatic nonverbal analysis of social interaction in small groups: A review,” _Image and Vision Computing_ , vol. 27, no. 12, pp. 1775–1787, 2009.
  * [24] R. Zheng, S. Zhang, L. Liu, Y. Luo, and M. Sun, “Uncertainty in bayesian deep label distribution learning,” _Applied Soft Computing_ , 2021.
  * [25] S. Kohl, B. Romera-Paredes, C. Meyer, J. De Fauw, J. R. Ledsam, K. Maier-Hein, S. Eslami, D. Jimenez Rezende, and O. Ronneberger, “A probabilistic u-net for segmentation of ambiguous images,” _Advances in Neural Information Processing Systems_ , vol. 31, 2018.
  * [26] M. Garnelo, D. Rosenbaum, C. Maddison, T. Ramalho, D. Saxton, M. Shanahan, Y. W. Teh, D. Rezende, and S. A. Eslami, “Conditional neural processes,” in _Int. Conf. on Machine Learning_. PMLR, 2018, pp. 1704–1713.
  * [27] C. Blundell, J. Cornebise, K. Kavukcuoglu, and D. Wierstra, “Weight uncertainty in neural network,” in _International Conference on Machine Learning_. PMLR, 2015, pp. 1613–1622.
  * [28] M. Abdelwahab and C. Busso, “Active learning for speech emotion recognition using deep neural network,” in _2019 8th Int. Conf. on Affective Computing and Intelligent Interaction (ACII)_. IEEE, 2019.
  * [29] M. Valstar, J. Gratch, B. Schuller, F. Ringeval, D. Lalanne, M. Torres Torres, S. Scherer, G. Stratou, R. Cowie, and M. Pantic, “AVEC 2016: Depression, Mood, and Emotion Recognition Workshop and Challenge,” in _Proceedings of the 6th International Workshop on Audio/Visual Emotion Challenge_ , ser. AVEC ’16. New York, NY, USA: Association for Computing Machinery, 2016, pp. 3–10.
  * [30] S. Steidl, M. Levit, A. Batliner, E. Noth, and H. Niemann, “”of all things the measure is man” automatic classification of emotions and inter-labeler consistency,” in _IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005._ , vol. 1. IEEE, 2005, pp. I–317.
  * [31] H. M. Fayek, M. Lech, and L. Cavedon, “Modeling subjectiveness in emotion recognition with deep neural networks: Ensembles vs soft labels,” in _2016 international joint conference on neural networks (IJCNN)_. IEEE, 2016, pp. 566–570.
  * [32] L. Tarantino, P. N. Garner, and A. Lazaridis, “Self-attention for speech emotion recognition,” in _Proc. Interspeech_ , 2019.
  * [33] R. E. Walpole, R. H. Myers, S. L. Myers, and K. Ye, “Probability & statistics for engineers & scientists. 2007,” _Pearson Cloth, p237_ , vol. 1, pp. 67–71, 2017.
  * [34] J. K. Kruschke, “Doing bayesian data analysis,” _Europe’s Journal of Psychology_ , vol. 7, pp. 778–779, 2010.
  * [35] C. Villa and S. G. Walker, “Objective prior for the number of degrees of freedom of at distribution,” _Bayesian Analysis_ , vol. 9, no. 1, pp. 197–220, 2014.
  * [36] C. Villa and F. J. Rubio, “Objective priors for the number of degrees of freedom of a multivariate t distribution and the t-copula,” _Computational Statistics & Data Analysis_, vol. 124, pp. 197–219, 2018\.
  * [37] P. Tzirakis, J. Zhang, and B. W. Schuller, “End-to-End Speech Emotion Recognition Using Deep Neural Networks,” in _ICASSP - Int. Conf. on Acoustics, Speech and Signal Processing_ , 2018, pp. 5089–5093.
  * [38] S. Kullback and R. A. Leibler, “On information and sufficiency,” _The annals of mathematical statistics_ , vol. 22, no. 1, pp. 79–86, 1951.
  * [39] K. P. Murphy, _Machine learning: a probabilistic perspective_. MIT press, 2012.
  * [40] I. Goodfellow, Y. Bengio, and A. Courville, _Deep Learning_ , 3rd ed., ser. 5\. The address of the publisher: MIT Press, 7 2016, vol. 4, ch. 3, pp. 51–77, http://www.deeplearningbook.org.
  * [41] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga _et al._ , “Pytorch: An imperative style, high-performance deep learning library,” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [42] D. Rees, “Essential statistics,” _American Statistician_ , vol. 55, no. 1, 2001, section 9.5.
  * [43] P. Tzirakis, J. Chen, S. Zafeiriou, and B. Schuller, “End-to-end multimodal affect recognition in real-world environments,” _Information Fusion_ , vol. 68, pp. 46–53, 2021.
  * [44] P. Tzirakis, A. Nguyen, S. Zafeiriou, and B. W. Schuller, “Speech emotion recognition using semantic information,” in _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2021, pp. 6279–6283.