[affiliation=1]JingyaoWu [affiliation=2]TingDang
[affiliation=1]VidhyasaharanSethu [affiliation=1]EliathambyAmbikairajah

# Dual-Constrained Dynamical Neural ODEs for Ambiguity-aware Continuous
Emotion Prediction

###### Abstract

There has been a significant focus on modelling emotion ambiguity in recent
years, with advancements made in representing emotions as distributions to
capture ambiguity. However, there has been comparatively less effort devoted
to the consideration of temporal dependencies in emotion distributions which
encodes ambiguity in perceived emotions that evolve smoothly over time.
Recognizing the benefits of using constrained dynamical neural ordinary
differential equations (CD-NODE) to model time series as dynamic processes, we
propose an ambiguity-aware dual-constrained Neural ODE approach to model the
dynamics of emotion distributions on arousal and valence. In our approach, we
utilize ODEs parameterised by neural networks to estimate the distribution
parameters, and we integrate additional constraints to restrict the range of
the system outputs to ensure the validity of predicted distributions. We
evaluated our proposed system on the publicly available RECOLA dataset and
observed very promising performance across a range of evaluation metrics.

###### keywords:

Speech continuous emotion recognition, inter-rater ambiguity, emotion
distribution, neural ODE

##  1 Introduction

Speech emotion recognition plays an important role in building the next
generation of human-machine interactions [1]. Dimensional representation,
describing emotions with continuous values on affect dimensions (e.g., arousal
and valence), allows for broader descriptions of the complexity and richness
of emotion, making it extensively utilised in the speech emotion recognition
research community [2, 3]. Typically, the continuous emotion labels (also
denoted as ratings) are collected from a group of human raters by asking for
their perceptual evaluations of the same speech or video recordings. This is
depicted in the top plot of Figure 1, where dashed lines represent the various
arousal ratings collected from 6 raters. The differences in perception,
referred to as _inter-rater ambiguity_ [4], reflect the subtlety of natural
emotions, and this ambiguity should be taken into consideration when
developing emotion recognition systems.

Until recently, researchers in affective computing have often treated this
ambiguity as noise and used the average ratings from multiple raters as a
`gold standard' [5, 4]. However, there is now growing recognition that inter-
rater ambiguity in emotion labels conveying information about the emotion
state and such ambiguity is being taken into account by treating
arousal/valence labels as distributions [6, 7, 8, 9]. This paradigm shift has
spurred the development of ambiguity-aware emotion prediction systems that
predict emotion states as distributions on the arousal/valence dimensions,
rather than conventional mean predictions [6, 7, 8, 9, 10, 11].

Additionally, emotions are dynamic processes that evolve continuously over
time [12, 13]. This dynamic nature applies not only to the emotional state
itself but also to the associated ambiguity [14]. Figure 1 illustrates this
idea of emotion dynamics with arousal ratings (coloured dashed lines on the
top plot) from each rater gradually changing over time. The associated emotion
ambiguity (grey shaded area) also follows a smooth evolving process leading to
the smoothly varying distributions (bottom plot) that encode the underlying
ambiguity. Nevertheless, among the limited works that model emotion ambiguity,
only a small subset of these systems have attempted to incorporate the
temporal dependencies amongst the distributions.

In this paper, we present a speech-based ambiguity-aware emotion prediction
system that explicitly incorporates the temporal dependencies of emotion
distributions. Leveraging the advantages of the recently proposed constrained
dynamical neural ordinary differential equation (CD-NODE) for modelling time
series with the ability to impose constraints on the smoothness of dynamic
process, and its effectiveness in modelling time-varying emotion ratings [15],
we develop a novel ambiguity-aware dual constrained CD-NODE system (denoted as
CD-NODEγ). This system predicts ambiguity-aware emotion states with time-
varying distributions along the arousal/valence dimensions, and automatically
learns how these emotion distributions evolve over time. Moreover, CD-NODEγ
integrates smoothness and range constraints to manage the rate of change and
the absolute scope of emotion distributions, respectively, further ensuring
the validity of the distributions.

Figure 1: Illustration of emotion dynamics. The top pane shows six different
annotations of the same speech segment over time in coloured lines, and the
bottom pane shows a series of distributions reflecting the ambiguous emotion
states.

##  2 Related work

Recent years have witnessed an increasing amount of effort on modelling and
predicting the ambiguity associated with emotion labels, often treated as
probability distributions [6, 14, 16]. However, the majority of existing
research primarily addresses the ambiguity in predictions while neglecting the
temporal dependencies, treating distributions at each time step as
independent. Among the few studies addressing temporal dependencies in emotion
distributions, initial methods modelled emotion labels using Gaussian Mixture
Models (GMMs) without considering temporal dynamics [6]. Subsequent
improvements incorporated Kalman filters to account for the temporal evolution
of GMM parameters [14]. However, this approach assumes that the temporal
dynamics follow a linear dynamical system. Zhang et al. proposed using
Bidirectional LSTM to estimate the means and standard deviations of Gaussian-
distributed labels [17], but the poor prediction in standard deviations
suggests the unsuitability of the predefined distribution types. While LSTM
learns temporal dependencies in the parameters, it lacks transparency and
flexibility in integrating constraints or prior knowledge on emotion dynamics.
More recent efforts, such as a system utilizing a Sequential Monte Carlo
approach, explored predicting time-varying emotion distributions with a non-
linear dynamical system, yet focused solely on the dynamics of distribution
means [7]. These approaches either operate in separate stages rather than end-
to-end, apply simplistic linear constraints or make specific and inappropriate
assumptions about distribution dynamics, or lack the flexibility in
incorporating constraints. On the other hand, the CD-NODE system captures the
temporal dynamics of emotions without imposing predefined assumptions on their
evolution, facilitating end-to-end modelling and allowing for incorporating
constraints, which has shown superior performance in continuous emotion
prediction [15]. However, it only predicts mean arousal/valence ratings and
neglects the associated ambiguity.

##  3 Overview of CD-NODE

Figure 2: Proposed CD-NODEγ for predicting Beta distributions. The speech
features $\mathbf{x}_{t_{n}}$ are fed into two neural networks $f_{1}$ and
$f_{2}$ which learn the dynamics of each Beta distribution. Rate constraints
$\phi_{i}$ are applied at the outputs of neural networks and range constraints
$\gamma_{i}$ are applied at the outputs of the ODE solvers.

The recently published CD-NODE was developed to model time series with
additional constraints on the dynamics of the predicted quantity. It models
the continuous emotion labels (i.e., arousal/valence ratings)
$y(t)=\\{y_{t_{0}},y_{t_{1}},\cdots,y_{t_{N}}\\}$ as a non-linear dynamical
process using ODE with governing function $f$ parameterized by neural networks
$\bm{\theta}$, and the dynamics in $y(t)$ is dependent on both the input
speech features
$\bm{x}(t)=\\{\bm{x}_{t_{0}},\bm{x}_{t_{1}},\cdots,\bm{x}_{t_{N}}\\}$ and
previous $y$. It is defined as:

$\frac{dy(t)}{dt}=\phi(f(\bm{x}(t),y(t);\bm{\theta})))\vspace{-0.5em}$ (1)

The function $\phi(\cdot)$ is an additional constraint to limit the rate of
change of $y(t)$, which controls the smoothness of $y(t)$ as:

$\vspace{-0.5em}\phi(\cdot)=\alpha*\tanh(\frac{1}{\alpha})$ (2)

where $\alpha$ is a hyperparameter that can be tuned based on the
characteristics of $y(t)$. A smaller $\alpha$ results in a smoother trajectory
of $y(t)$ and vice versa. The governing function $f$ can be parameterised by
any network structure that approximates the nonlinear function in ODE. The
final trajectory of $y(t)$ is obtained by solving an ODE initial value problem
as shown in Equation (3).

$y(t)=\mathrm{ODESolve}(f,\bm{\theta},y_{t_{0}},t_{0},\cdots,t_{N},\bm{x}(t))\\\
$ (3)

##  4 Proposed CD-NODEγ

While the original CD-NODE based system only predicts the mean arousal/valence
rating, this paper proposes an ambiguity-aware dual-constrained Neural ODE,
referred to as, CD-NODEγ, that predicts ambiguous arousal/valence states as
time-varying Beta distributions as suggested in [16] that Beta distributions
are highly suitable for modelling emotion ambiguity. Furthermore, in addition
to the explicit smoothness constraint which was a key feature of the CD-NODE,
we introduce a second constraint $\gamma$, which restricts the system output
to fall within a given range, which in turn helps ensure the validity of the
predicted distributions.

###  4.1 Model Structure

Assuming the arousal/valence at time $t_{n}$ is represented by a beta
distribution that is parameterised by $\bm{\psi}_{t_{n}}$, the aim of the
emotion prediction system is to predict
$\bm{\psi}(t)=\\{\bm{\psi}_{t_{0}},\bm{\psi}_{t_{1}},\cdots,\bm{\psi}_{t_{N}}\\}$
given a sequennce of input speech features
$\bm{x}(t)=\\{\bm{x}_{t_{0}},\bm{x}_{t_{1}},\cdots,\bm{x}_{t_{N}}\\}$. The
most likely emotion intensity and the associated ambiguity can then be
inferred from the predicted beta distribution, $\widehat{\bm{\psi}}_{t_{n}}$,
at each time step.

The Beta distribution is typically parameterised using two parameters, denoted
as $\bm{\psi}_{t_{n}}=\\{\psi^{1}_{t_{n}},\psi^{2}_{t_{n}}\\}$. This then
leads to a multi-task learning framework to model
$\\{\psi^{1}_{t_{n}},\psi^{2}_{t_{n}}\\}$:

$\small\frac{d\psi^{i}(t)}{dt}=\phi_{i}(f_{i}(\mathbf{x}(t),\psi^{i}(t);\bm{\theta}^{i})),i\in[1,2]$
(4)

where $i\in[1,2]$ corresponds to the two parameters, $f_{i}$ are the two
governing functions modelled as neural networks (blue blocks in Figure 2) with
the aim of learning to predict $\frac{d\psi^{i}(t)}{dt}$. They are each
constrained with their own smoothness constraint function $\phi_{i}$ (green
blocks in Figure 2).

Noting that the parameters of any beta distributions will have constraints to
ensure a resulting distribution is valid (for e.g., when using the mode and
concentration parameterisation of a beta distribution, the concentration must
be higher than $2$ to ensure a bell shaped distribution), we impose an
additional constraint function $\gamma(\cdot)$ to limit the range of
$\psi^{1}_{t_{n}}$ and $\psi^{2}_{t_{n}}$. Thus, the final outputs are
obtained as per equation (5).

$\small\psi^{i}(t)=\gamma_{i}(\mathrm{ODESolve}(f_{i},\bm{\theta}^{i},\psi^{i}_{t_{0}},t_{0},\cdots,t_{N},\bm{x}(t)))$
(5)

Finally, the loss function for the proposed multi-task learning framework is
taken as:

$\small L=\lambda_{1}L_{1}+\lambda_{2}L_{2}$ (6) $\small
L_{i}=1-\rho_{c}^{i}(\psi^{i}(t),\widehat{\psi}^{i}(t)),i\in[1,2]$ (7)

where $\lambda_{i}$ are scaling factors; and $L_{i}$ are loss functions for
$\psi^{i}(t)$, dependent on the concordance correlation coefficient (CCC)
$\rho_{c}^{i}$ between the predicted $\widehat{\psi}^{i}(t)$ and the ground
truth $\psi^{i}(t)$ [15]. The model aims to minimize $L_{i}$, i.e., maximizing
the CCC.

###  4.2 Constraint Function: $\gamma$

The initial stage of our approach involves identifying the most suitable
parameter pairs for the Beta distribution in our proposed CD-NONEγ. This is a
crucial step in ensuring that our model can effectively learn the
distributional properties of emotions. The most commonly used parameter pairs
include: shape parameters $\\{a,b\\}$, and $\\{mean(\mu),variance(\sigma)\\}$
and $\\{mode(w),concentration(k)\\}$ which indicate the central tendency
($\mu$ or $w$) and spread ($\sigma^{2}$ or $k$) of the distribution. We
observed that shape parameters $\\{a,b\\}$ are less interpretable compared to
$\\{\mu,\sigma\\}$ and $\\{w,k\\}$. Furthermore, when comparing
$\\{\mu,\sigma\\}$ to $\\{w,k\\}$, we observe that both $\mu$ and $\sigma$ are
both bounded to the interval $[0,1]$, while only $w$ is bound to a finite
interval with the only bound on $k$ being $k>2$. The lack of an upper bound to
$k$ leads to dramatic changes in its dynamics which makes a CD-NODEγ that
predicts $\\{w,k\\}$ harder to train. Therefore, we chose $\\{\mu,\sigma\\}$
to represent the Beta distributions. Consequently, we design the constraint
functions $\gamma_{i}$ as in equation (8) for both $\mu$ and $\sigma$ as:

$\displaystyle\gamma_{\mu}=pS(\cdot),\gamma_{\sigma}=qS(\cdot)$ (8a)
$\displaystyle 0<p\leq 1,0<q^{2}\leq
min(p^{2}\frac{1-p}{1+p},(1-p)^{2}\frac{p}{2-p})$ (8b)

where, $S(\cdot)$ denotes a sigmoid function and is used to constrain the
output to $[0,1]$, and $p$ and $q$ are adjustable constants that must satisfy
the relationship as per equation (8b) in order to ensure the Beta distribution
is bell shaped [18].

##  5 Experimental Settings

###  5.1 Dataset

The RECOLA dataset is one of the most commonly used multimodal corpus which
consists of 9.5 hours spontaneous dyadic conversation recordings in French
[19]. There are 9 five-minute utterances each in the training and development
sets, identical to the data partition in the AVEC challenge 2015 [20]. Since
the challenge test sets are not publicly available, the system is trained and
the hyperparameters are optimized using the training set and tested on the
development set as per standard practice [15, 7, 18, 20]. The dataset is
annotated by 6 human raters with continuous arousal and valence labels within
range $[-1,1]$, with sampling rate of 40ms.

###  5.2 Beta distribution parameter estimation

To obtain the ground truth beta distribution, we follow the same approach used
in [18] to estimate the Beta distribution parameters. The original ratings are
first mapped to range $x\in[0,1]$ via a linear transformation $y=0.4975x+0.5$
in order to perform the Beta fit. To compute the Beta parameter at each frame,
the ratings from neighbouring $F$ frames are concatenated as
$\widetilde{\bm{y}}=[\bm{y}_{n-F}:\bm{y}_{n+F}]$, $\bm{y}_{n}$ refers to the
labels from 6 raters at frame $n$ to avoid overfitting. $F$ is 6 and 1 for
arousal and valence respectively as suggested in [18]. Finally, the Beta
parameters $\bm{\psi}_{t_{n}}$ are obtained by Maximum A Posterior (MAP) as:

$\small\bm{\psi}_{t_{n}}=\operatorname*{arg\,max}_{\bm{\psi}}\prod_{m}P(y_{n,m}|\bm{\psi})P(\bm{\psi})\vspace{-2.5mm}$
(9)

where $y_{n,m}$ denotes the rating from the $m^{th}$ rater at frame $n$;
$p(y_{n,m}|\psi)$ refers to the posterior probability of $y_{n,m}$; and
$P(\psi)$ refers to the prior probability which is estimated with kernel
density estimation, the same as in [18].

###  5.3 System configurations

Features. The Bag-of-audio-words(BoAW) feature representations are extracted
with 100 clusters from 20-dimensional MFCCs using OpenXbow [21]. Detailed
explanation refers to [22]. 4 second delay compensation is applied to both
arousal and valence to compensate the annotation delay [23].

Network parameters. A fully connected (FC) layer with 64 neurons and $tanh$
activation function is first employed to transfer the BoAW features. The
following $f_{1}$ (i.e., $f_{\mu}$) and $f_{2}$(i.e., $f_{\sigma}$) for
modelling dynamics of $\mu$ and $\sigma$ are composed of three FC layers
respectively, with $tanh$ activation function applied for the first two layers
to enforce the nonlinearity, and the smoothing constraint applied to the last
FC layer (c.f. Equation (8)), with $\alpha_{u}=0.5$ [15] and
$\alpha_{\sigma}=10$ which is optimized within $\\{0.5,1,2,4,6,8,10,15\\}$ for
both arousal and valence. For the proposed constraints in Equations (2), we
chose $p=0.75$ for both arousal and valence as 95% of the ratings are within
the range [0,0.75], and $q=0.15$ satisfying the requirement in Equation (8).
The initial values $\psi^{i}_{t_{0}}$ in Equation (5) are the Beta
distribution parameters inferred from the labels of the first frame during the
training phase, and set to zeros during the test phase given emotions at the
beginning of any utterances tend to be zeros. We used the adjoint method with
Runge-Kutta 5 ODE solver [24]. The absolute and relative error tolerances of
the ODE solver were $10^{-13}$ and $10^{-7}$ based on preliminary empirical
analyses [15]. The number of model parameters is approximately 30k111Code:
https://github.com/JingyaoWU66/CD-NODE-gamma.

Baseline. Two state-of-art systems are compared: the Sequential Monte Carlo
(SMC) approach [7] and the LSTM based system [18]. The LSTM system follows the
same settings as in [18] which directly predicts Beta distribution parameters.
In our experiments, the sigmoid activation function is applied at the output
layer to limit the output range for a fair comparison. The SMC approach
predicts distributions that are not constrained to Beta distributions and we
employ the same settings as in [7].

Training. For all systems, Adam optimizer was used, and the initial learning
rate is optimized to 0.01 and 0.001 for $f_{\mu}$ and $f_{\sigma}$, with a
decaying ratio of 0.9. 60 and 30 epochs were tested for CD-NODE and LSTM
baseline. We fixed the seed numbers to ensure reproducibility when testing
different models. The training utterances were chunked into 4-second windows
(i.e., 100 frames) in each batch for training efficiency, while it was tested
with the entire utterance to match the practical scenarios. The scaling
factors in the loss function in Equation (6) were optimized within the range
[0,20] with step size 1 and $\lambda_{1}=1$ and $\lambda_{2}=10$ were
selected. The predictions were post-processed with a moving average filter
with 12 frames( 0.5s) window size. The training time is approximately 72h on
Apple M2 chip with 8 cores on 16GB CPU.

Evaluation Metric. It is suggested in [7] that at low ambiguity regions, a
well-predicted distribution should be centred at the ground truth mean;
whereas at high ambiguity regions, the mean is not representative of the
underlying emotions, and thus, the spread of the distributions is more
important. We evaluate the first aspect by comparing root mean squared error
(RMSE) between the predicted mean and ground truth mean at different ambiguity
regions [7]; and the second aspect using Concordance Correlation Coefficient
(CCC) between the predicted standard deviation (SD) and ground truth SD.

Table 1: CCC comparison of the predicted Beta distributions in terms of mean and standard deviations (SD). Systems | Arousal | Valence
---|---|---
| Mean | SD | Mean | SD
SMC [7] | 0.702 | 0.403 | 0.391 | 0.195
LSTM$-w-k$ [18] | 0.637 | 0.436 | 0.321 | 0.044
LSTM$-\mu-\sigma$ | 0.658 | 0.468 | 0.346 | 0.013
Proposed CD-NODEγ | 0.747 | 0.412 | 0.425 | 0.081

##  6 Results

###  6.1 Comparisons with state-of-the-art

The comparison to the state-of-the-art systems is reported in Table 1, in
terms of CCC between the predicted Beta distribution (i.e., mean $\mu$ and
standard deviation $\sigma$) and the ground truth Beta distributions (refer to
section 5.2) correspondingly. In addition to [18] that uses LSTM to predict
$w$ and $k$ of beta distribution (referred to as LSTM$-w-k$), we further
validate the system performance for predicting $\mu$ and $\sigma$ for a fair
comparison, named as LSTM$-\mu-\sigma$. The proposed system is also compared
to the existing SMC model which adopts non-parametric distribution [7]. It is
observed that the proposed model CD-NODEγ yields the highest CCC in terms of
mean predictions $\mu$ for both arousal and valence, with significance p-value
$<$ 0.001. Regarding the SD prediction, CD-NODEγ shows comparable performance
to the best system LSTM$-\mu-\sigma$ for arousal, and underperforms the best
system SMC for valence. This could be because the dynamics of SD change more
rapidly compared to the mean, which may require a more advanced ODE solver to
obtain a more accurate solution.

Figure 3: RMSE of the proposed CD-NODEγ and baselines. The standard deviation
(SD) range corresponding to each decile is shown on the x-axis.

Figure 3 presents the RMSE values between the predicted and ground truth means
across different ambiguity regions, defined based on deciles of the ground
truth standard deviation. As expected, the RMSE generally increases at higher
standard deviation deciles for both arousal and valence, indicating that mean
prediction errors increase with rising ambiguity (low inter-rater agreement).
Conversely, the systems perform better at low ambiguity regions (high inter-
rater agreement), which supports our hypothesis. Notably, our proposed system,
CD-NODEγ, consistently achieves the lowest RMSE compared to all baseline
systems for both arousal and valence. These results demonstrates the superior
performance of CD-NODE${\gamma}$ in modelling time-varying emotion
distributions, as it accurately predicts the emotion state at low ambiguity
regions while still reasonably predicting the level of ambiguity at both low
and high ambiguity regions. Although CD-NODEγ and LSTM$-\mu-\sigma$ achieved
similar performance on valence, it may be attributed to the inherent
difficulty in predicting valence using speech data.

###  6.2 Impact of range constraint

We further validate the effectiveness of the constraint functions in the CD-
NODEγ, by comparing the system performance without constraint (D-NODE), with
only the smoothness constraint (CD-NODE), and with both the smoothness and
range constraints (CD-NODEγ). The performance in terms of CCC in Table 2 shows
that CD-NODEγ achieves the highest CCC and D-NODE without constraint shows the
lowest CCC, indicating the advantages of adding constraints. More importantly,
CD-NODEγ yields 3.6% and 6.5% relative improvements over CD-NODE for mean and
SD of arousal, and 3.7% and 26.6% relative improvements for mean and SD of
valence. This further confirms the effectiveness of the proposed range
constraint $\gamma$.

RMSE is also reported in Figure 4 for the three systems. All three systems
show an increasing trend of RMSE with increasing SD decibels, aligning with
the assumption that prediction error is larger where inter-rater ambiguity is
high. Notably, CD-NODEγ consistently shows a smaller RMSE compared to the
other two systems, e.g., 6.3% and 3.3% relative decreases at lowest ambiguity
decile on arousal compared to D-NODE and CD-NODE respectively, further
validating the benefits of proposed constraints. Overall, the results suggest
that CD-NODEγ with the range constraint can capture the time-varying dynamics
of emotion distributions more effectively, which might be because the
constraint function eliminates the noisy predictions outside the possible
range of the predicted quantity.

Table 2: CCC comparison of the three systems: D-NODE without constraint, CD-NODE with rate constraint and CD-NODEγ with rate and range constraint. Systems | Arousal | Valence
---|---|---
| Mean | SD | Mean | SD
D-NODE | 0.703 | 0.359 | 0.397 | 0.058
CD-NODE | 0.721 | 0.387 | 0.410 | 0.064
Proposed CD-NODEγ | 0.747 | 0.412 | 0.425 | 0.081
Figure 4: RMSE of the proposed CD-NODEγ, and baselines.

##  7 Conclusion

In this paper, a multi-task learning framework is used to develop an
ambiguity-aware dual-constrained dynamic neural ODE (CD-NODEγ) system that
models ambiguous emotional states as time-varying Beta distributions. Our
proposed method takes into account the temporal dynamics of distributions and
incorporates a range constraint that restricts the outputs to better ensure
the validity of predicted emotion distribution, in addition to a smoothness
constraint. Experimental results, when compared to various baselines, show
that the proposed CD-NODEγ model outperforms them in modelling emotional
states and produces accurate predictions in low ambiguity regions and reliable
predictions of high ambiguity regions. Moreover, the outcomes acquired with
various constraint configurations indicate that the inclusion of a range
constraint enhances the CD-NODE's ability to learn the distribution dynamics.
The results highlight the benefits of utilising CD-NODEγ to model temporal
dynamics of distributions with explicit constraints, which opens up new
avenues for emotion ambiguity modelling.

## References

  * [1] Y. Wang, W. Song, W. Tao, A. Liotta, D. Yang, X. Li, S. Gao, Y. Sun, W. Ge, W. Zhang _et al._ , ``A systematic review on affective computing: Emotion models, databases, and recent advances,'' _Information Fusion_ , 2022.
  * [2] J. A. Russell, ``A circumplex model of affect.'' _Journal of personality and social psychology_ , vol. 39, no. 6, p. 1161, 1980.
  * [3] H. Gunes and M. Pantic, ``Automatic, dimensional and continuous emotion recognition,'' _International Journal of Synthetic Emotions (IJSE)_ , vol. 1, no. 1, pp. 68–99, 2010.
  * [4] V. Sethu, E. M. Provost, J. Epps, C. Busso, N. Cummins, and S. Narayanan, ``The ambiguous world of emotion representation,'' _arXiv preprint arXiv:1909.00360_ , 2019.
  * [5] H. Gunes and B. Schuller, ``Categorical and dimensional affect analysis in continuous input: Current trends and future directions,'' _Image and Vision Computing_ , vol. 31, no. 2, pp. 120–136, 2013.
  * [6] T. Dang, V. Sethu, J. Epps, and E. Ambikairajah, ``An investigation of emotion prediction uncertainty using gaussian mixture regression.'' in _INTERSPEECH_ , 2017, pp. 1248–1252.
  * [7] J. Wu, T. Dang, V. Sethu, and E. Ambikairajah, ``A novel sequential monte carlo framework for predicting ambiguous emotion states,'' in _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2022, pp. 8567–8571.
  * [8] T. Mani Kumar, E. Sanchez, G. Tzimiropoulos, T. Giesbrecht, and M. Valstar, ``Stochastic process regression for cross-cultural speech emotion recognition,'' _Proc. Interspeech 2021_ , pp. 3390–3394, 2021.
  * [9] B. Zhang, G. Essl, and E. Mower Provost, ``Predicting the distribution of emotion perception: capturing inter-rater variability,'' in _Proceedings of the 19th ACM International Conference on Multimodal Interaction_ , 2017, pp. 51–59.
  * [10] J. Han, Z. Zhang, Z. Ren, and B. Schuller, ``Exploring perception uncertainty for emotion recognition in dyadic conversation and music listening,'' _Cognitive Computation_ , vol. 13, pp. 231–240, 2021.
  * [11] J. Wu, T. Dang, V. Sethu, and E. Ambikairajah, ``Belief mismatch coefficient (bmc): A novel interpretable measure of prediction accuracy for ambiguous emotion states,'' in _2023 11th International Conference on Affective Computing and Intelligent Interaction (ACII)_. IEEE, 2023, pp. 1–8.
  * [12] P. Kuppens and P. Verduyn, ``Emotion dynamics,'' _Current Opinion in Psychology_ , vol. 17, pp. 22–26, 2017.
  * [13] S. P. Yadav, S. Zaidi, A. Mishra, and V. Yadav, ``Survey on machine learning in speech emotion recognition and vision systems using a recurrent neural network (rnn),'' _Archives of Computational Methods in Engineering_ , vol. 29, no. 3, pp. 1753–1770, 2022.
  * [14] T. Dang, V. Sethu, and E. Ambikairajah, ``Dynamic multi-rater gaussian mixture regression incorporating temporal dependencies of emotion uncertainty using kalman filters,'' in _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2018, pp. 4929–4933.
  * [15] T. Dang, A. Dimitriadis, J. Wu, V. Sethu, and E. Ambikairajah, ``Constrained dynamical neural ode for time series modelling: A case study on continuous emotion prediction,'' in _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2023, pp. 1–5.
  * [16] D. Bose, V. Sethu, and E. Ambikairajah, ``Parametric distributions to model numerical emotion labels,'' _Proc. Interspeech 2021_ , pp. 4498–4502, 2021.
  * [17] Z. Zhang, J. Han, E. Coutinho, and B. Schuller, ``Dynamic difficulty awareness training for continuous emotion prediction,'' _IEEE Transactions on Multimedia_ , vol. 21, no. 5, pp. 1289–1301, 2018.
  * [18] D. Bose, V. Sethu, and E. Ambikairajah, ``Continuous emotion ambiguity prediction: Modeling with beta distributions,'' _IEEE Transactions on Affective Computing_ , no. 01, pp. 1–12, 2024.
  * [19] F. Ringeval, A. Sonderegger, J. Sauer, and D. Lalanne, ``Introducing the recola multimodal corpus of remote collaborative and affective interactions,'' in _2013 10th IEEE international conference and workshops on automatic face and gesture recognition (FG)_. IEEE, 2013, pp. 1–8.
  * [20] F. Ringeval, B. Schuller, M. Valstar, R. Cowie, and M. Pantic, ``Avec 2015: The 5th international audio/visual emotion challenge and workshop,'' in _Proceedings of the 23rd ACM international conference on Multimedia_ , 2015, pp. 1335–1336.
  * [21] M. Schmitt and B. Schuller, ``Openxbow: introducing the passau open-source crossmodal bag-of-words toolkit,'' 2017.
  * [22] M. Schmitt, F. Ringeval, and B. W. Schuller, ``At the border of acoustics and linguistics: Bag-of-audio-words for the recognition of emotions in speech.'' in _Interspeech_ , 2016, pp. 495–499.
  * [23] Z. Huang, T. Dang, N. Cummins, B. Stasak, P. Le, V. Sethu, and J. Epps, ``An investigation of annotation delay compensation and output-associative fusion for multimodal continuous emotion prediction,'' in _Proceedings of the 5th International Workshop on Audio/Visual Emotion Challenge_ , 2015, pp. 41–48.
  * [24] R. T. Chen, Y. Rubanova, J. Bettencourt, and D. K. Duvenaud, ``Neural ordinary differential equations,'' _Advances in neural information processing systems_ , vol. 31, 2018.