# A Unified Transformer-based Network for Multimodal Emotion Recognition

Kamran Ali and Charles E. Hughes Kamran Ali and Charles E. Hughes are both
with the Department of Computer Science, University of Central Florida, USA,
Orlando, FL 32816.
E-mail<EMAIL_ADDRESS>Manuscript received April 19, 2005; revised August
26, 2015.

###### Abstract

The development of transformer-based models has resulted in significant
advances in addressing various vision and NLP-based research challenges.
However, the progress made in transformer-based methods has not been
effectively applied to biosensing research. This paper presents a novel
Unified Biosensor-Vision Multi-modal Transformer-based (UBVMT) method to
classify emotions in an arousal-valence space by combining a 2D representation
of an ECG/PPG signal with the face information. To achieve this goal, we first
investigate and compare the unimodal emotion recognition performance of three
image-based representations of the ECG/PPG signal. We then present our UBVMT
network which is trained to perform emotion recognition by combining the 2D
image-based representation of the ECG/PPG signal and the facial expression
features. Our unified transformer model consists of homogeneous transformer
blocks that take as an input the 2D representation of the ECG/PPG signal and
the corresponding face frame for emotion representation learning with minimal
modality-specific design. Our UBVMT model is trained by reconstructing masked
patches of video frames and 2D images of ECG/PPG signals, and contrastive
modeling to align face and ECG/PPG data. Extensive experiments on the MAHNOB-
HCI and DEAP datasets show that our Unified UBVMT-based model produces
comparable results to the state-of-the-art techniques.

###### Index Terms:

Emotion recognition, Transformers, Biosensors, Multi-model representation
learning.

##  1 Introduction

Over the past few years, there has been an increasing interest in adopting a
bio-sensing perspective for emotion analysis. Certainly, the popularity of
bio-sensing research extends beyond affective computing. Various fields such
as robotics [1], [2], health [3], [4], and virtual reality [5] have also
embraced bio-sensing as a valuable research tool. Bio-sensing systems,
including those that measure electrocardiogram (ECG), Photoplethysmography
(PPG), electroencephalogram (EEG), galvanic skin response (GSR), etc, have
been available for decades; however, their size and complexity have limited
their use to controlled laboratory settings and hospitals. The emergence of
wearable biosensing systems has fueled the recent interest in utilizing bio-
sensing systems for various applications by simplifying and speeding up data
collection.

Several research studies have demonstrated the feasibility of recognizing
human emotions through facial expressions captured in images and videos [6],
[7], [8]. Although facial expression recognition systems have shown remarkable
success on databases captured under controlled conditions, their performance
significantly degrades [9], [10] when they are applied to real-life
situations. The system’s unreliability in natural environments is primarily
due to various factors, such as varying head pose, illumination conditions,
and occlusion of different parts of the face. Additionally, recognizing
emotions from facial expressions may not be entirely dependable since they can
be easily concealed or manipulated. Furthermore, facial expressions can be
influenced by social and cultural differences, as human expressiveness varies
among individuals. In contrast, physiological signals offer a more accurate
reflection of emotions and their subtle changes. Therefore, multimodal emotion
recognition techniques that combine facial information with physiological data
have the potential to compensate for the drawbacks of unimodal methods and
achieve more precise recognition outcomes [11], [12].

Figure 1: The framework of UBVMT-based multimodal emotion recognition

Many emotion recognition techniques have been proposed in the past years
combining different modalities. For instance, in [13] Shang et al. recognized
emotions by fusing EEG, EOG, and EMG signals. Similarly, in [14], Miranda et
al. combined EEG, ECG, and GSR modalities to perform emotion analysis. In
[15], Koelstra et al. classified emotions along the valance and arousal axes
by leveraging face information and EEG signals. Most of these multimodal
emotion recognition methods combine different modalities with the EEG signal.
However, several research studies show that there is a strong correlation
between features extracted from the ECG signal and human emotions. For
instance, in [16], Hany et al. extracted ECG features that represent the
statistical distribution of dominant frequencies by applying spectrogram
analysis to classify emotions. Similarly, in [17], time and frequency domain
features are extracted from the ECG signal to perform emotion analysis.
Building on those approaches, this paper presents a novel multimodal emotion
recognition technique that fuses information from face video frames and
ECG/PPG data. To the best of our knowledge, this is the first study that
investigates the fusion of face information with the ECG/PPG signal to
classify emotions.

Many attempts have been made to create effective techniques for integrating
multimodal information. Initially, methods involved simply combining high-
level features from all modalities to predict outcomes (known as ”early
fusion”) or adding up decisions made by each unimodal system with learnable
weights (called ”late fusion”) to produce the final inference. While these
methods outperformed unimodal techniques to some extent, the lack of inter-
modality interactions during training limited the potential improvement. In
recent times, Transformers [18] have emerged as a highly successful approach
for multimodel representation learning. Due to the advances in attention
mechanisms and transformers, later studies have focused predominantly on using
these techniques to develop more sophisticated multimodal fusion methods [19],
[20], [21], [22], [23]. However, most of these transformer-based multimodal
emotion recognition techniques leverage video, audio, and text modalities to
learn joint emotion representation. In this paper, we present a unified
transformer-based model that employs ECG/PPG data along with face information
to recognize emotions. The main framework of the proposed method is shown in
Figure 1. Based on our literature survey, no previous work has investigated
the application of transformers in learning emotion representation by using
face and ECG/PPG modalities.

Inspired by the compact architecture of the recently published Textless
Vision-Language Transformer (TVLT) [22], we present a Unified Biosensor-Vision
Multimodal Transformer (UBVMT) network to recognize emotions by fusing face
and ECG/PPG data. We first investigated the effectiveness of various image-
based representations of ECG/PPG signals for biosensor-based unimodal emotion
recognition. Several previous methods have employed an image-based
representation of an ECG/PPG signal for heart rate estimation [24], [25],
[26], [27], [28]. For instance, in [24], Song et al. obtained a spatiotemporal
representation of a pulse signal in a time-delayed way by constructing a
Toeplitz matrix. The Toeplitz matrix is then converted into an image that is
fed to a CNN to estimate heart rate information. The simplicity of the
Toeplitz representation allows it to preserve both the morphological and
chronological details of the 1D pulse signal. As a result, the CNN can
accurately extract the correct HR values from input feature images.

In [25], average pooling is applied to various blocks within an ROI region.
The spatiotemporal maps are then generated by arranging these temporal
sequences into rows, which are then fed to a deep network to get HR values. In
[26], the sample mean sequences of the R, G, and B channels from the ROI of
videos are extracted, and Short-Time Fourier Transform (STFT) is applied to
construct the 2D time-frequency representations of the sequences. In [27], a
spatiotemporal map is first computed from the input video as a representation
of the BVP signal, and the spatiotemporal images are then mapped to BVP
signals using the generative adversarial learning technique. In [28], Khare et
al. converted the time-domain EEG biosignals into a time-frequency
representation by employing Smoothed Pseudo Wigner-Ville distribution (SPWVD).

Apart from the image-based representation of ECG signals for heart rate
estimation, some works [29] and [30] have extracted deep learning features
from the scalogram of biosensor signals to recognize cognitive load and
hypertension risk stratification, respectively. They converted the PPG signals
to Scalograms by using Continuous Wavelet Transform. Scalograms were used
instead of spectrograms as the image representation for PPG signals. This is
because scalograms provide better highlighting of the low-frequency or rapidly
changing frequency components of the signal as compared to spectrograms.

In this paper, we first investigate and compare the performance of three image
representations of the ECG/PPG signal for emotion recognition. More
specifically, we perform unimodal emotion recognition by converting the time
domain ECG/PPG signal into 1. spatiotemporal maps [24], 2. time-frequency
representation by employing Smoothed Pseudo Wigner-Ville distribution (SPWVD)
[28], and 3. Scalograms [29] [30]. We show that the best emotion recognition
performance is obtained by employing the scalogram representation of the ECG
and PPG signals. Therefore, our Unified Biosensor-Vision Multimodal
Transformer (UBVMT) network is trained and validated by using the scalogram
representation of the ECG/PPG signal along with the face images. The
effectiveness of the proposed UBVMT-based multimodal emotion recognition
technique is validated on two datasets: the MAHNOB-HCI [31] and DEAP [32]
datasets. Extensive experiments show that the proposed approach produces
comparable results to the state-of-the-art emotion recognition techniques.
Overall, the main contributions of this paper are as follows:

  * •

Comparison of the performance of three image-based representations, namely
spatiotemporal maps, time-frequency representation, and scalograms of the
ECG/PPG signal for emotion recognition, and evidence that scalograms better
represent the emotion features contained by the ECG/PPG signals.

  * •

A novel technique that employs transformer architecture for biosensor-based
multimodal emotion recognition.

  * •

The Unified Biosensor-Vision Multimodal Transformer (UBVMT) network consisting
of homogenous blocks that learn vision-and-biosensor emotion representation
with minimal modality-specific design, thus, making it compact and applicable
in real-time emotion analysis.

  * •

Experimental results showing that the proposed Unified Biosensor-Vision
Multimodal Transformer-based (UBVMT) technique learns effective multimodal
emotion representation by fusing face with ECG/PPG data.

  * •

A baseline for emotion recognition research created by the fusion of face data
with an ECG/PPG signal.

##  2 Related Work

Psychologically, human emotions can be characterized based on two main
frameworks: categorical or dimensional representations. In the categorical
framework of emotions, emotions are classified into distinct labels such as
joy, sadness, anger, happiness, fear, and so on. This approach considers
emotions as discrete categories rather than continuous dimensions. In
dimensional conceptualizations of emotions, a commonly used framework involves
representing emotions within a two-dimensional space. In this space, valence
is positioned along one axis, while arousal is positioned along the other
[33]. Valence, often positioned on the horizontal axis, signifies the extent
of pleasantness or unpleasantness. On the other hand, arousal, typically
placed on the vertical axis, represents the level of activation or energy
associated with the emotion. Within a two-dimensional (2D) space, as shown in
Figure 2, emotions are portrayed by their valence and arousal level. While the
categorical approach to emotions is conceptually straightforward, it faces
certain challenges. It struggles to represent compound emotions that do not
fit neatly into a single category, and it does not provide a means to quantify
the degree or intensity of an emotional state. Therefore, in this paper, a
novel multimodal emotion recognition technique is developed to classify
emotions along the valence-arousal axis.

###  2.1 Emotion Recognition with ECG/PPG Signals

Previous research shows that there is a strong correlation between ECG/PPG
signals and human emotion. In [34], Yu et al. converted the rPPG signal of an
input image into time-frequency domain spectrogram images to classify the
short-term emotions. In [35], Ismail et al. employed ECG and PPG signals to
develop an emotion recognition system, and compared the performance of both
signals for classifying emotions. Lee et al. in [36] investigated the ability
of PPG signals to recognize emotions along the arousal-valence axis. Sepúlveda
el al. in [37] performed emotion recognition from ECG signals using a wavelet
scattering algorithm. Emotion features of the ECG signal are extracted at
different time scales leveraging the wavelet scattering technique, and the
extracted features are then fed into different classifiers to perform emotion
analysis. Sarkar et al. in [38] exploited a self-supervised deep multi-task
learning framework to develop an emotion recognition algorithm using the ECG
signals. Mellouk et al. presented a rPPG signal-based emotion classification
method using a deep learning architecture, which combines a one-dimensional
convolution neural network (1DCNN) and a long short-term memory (LSTM) [39].
In [16], Ferdinando et al. extracted emotion features from ECG signals using
spectrogram analysis of intrinsic mode function after applying the bivariate
empirical mode decomposition to ECG. Similarly, in [17], Ferdinando et al.
derived heart rate variability (HRV) features from ECG signals to categorize
emotions in an arousal-valance space. In [29], Gasparini et al. used a
scalogram-based representation of a PPG signal to extract deep learning
features for the recognition of cognitive load. Gasparini et al. showed that
the deep learning features outperformed hand-crafted features to classify
cognitive load, especially by leveraging feature selection methods to avoid
the curse of dimensionality. Similarly, Liang et al. in [30], converted the
PPG signals into scalogram-based representations using continuous wavelet
transforms that are input to a deep learning method for the classification and
evaluation of hypertension.

Figure 2: Arousal valence emotion model

###  2.2 Emotion Recognition with Facial Images

Facial images and videos have been widely used by the research community to
recognize emotions. Mostly, facial information is used to analyze emotions by
using facial expression recognition techniques [40], [41], [42], [43], [44].
However, in this paper, a transformer-based technique is presented that
estimates the continuous dimensions related to emotions. Based on the
dimensional approach, affective behavior can be characterized using various
underlying continuous dimensions. These dimensions offer a more accurate
representation of the emotions individuals experience in their daily lives. As
noted in Figure 2, two widely utilized dimensions are valence, which reflects
the positivity or negativity of an emotional state, and arousal, which gauges
the intensity of emotional activation [45], [46], [47].

To estimate valence and arousal from facial images, Dimitrios et al. [48] used
a Convolutional and Recurrent (CNN-RNN) deep neural architecture to extract
emotion features for dimensional emotion recognition. Deng et al. [49] used
the ResNet-50 model for multi-task expression recognition and implemented the
teacher-student architecture to enhance the training data without labels. This
approach aims to address the imbalanced data distribution across different
tasks. In [50], Kuhnke et al. utilized facial landmarks to align the image and
eliminate conflicting data. They achieve this by leveraging the correlation
among different representations to generate pseudo-labels. In [51], the
features extracted from a 3D-CNN are inputted into the 3D VGG and 2D SENet-101
networks. Subsequently, Gated Recurrent Units (GRUs) are employed to enhance
the model’s ability to learn temporal features.

###  2.3 Multi-modal Emotion Recognition

There have been many studies focusing on the analysis of multimodal systems
for recognizing human emotions. These studies have demonstrated that, by
integrating information from multiple modalities, the performance of models
for recognizing emotions can be enhanced [52]. Therefore, in [53], Yin et al.
fused information from ECG signal with Electroencephalogram (EEG),
Electrooculography (EOG), Galvanic Skin Response (GSR), Electromyography
(EMG), skin temperature, blood volume, and respiration to recognize emotions.
Miranda et al. [14] recorded EEG, GRS, and ECG signals using wearable sensors
and combined EEG and GRS information with the ECG signal to categorize human
emotions. Soleymani et al. [31] employed Hidden Markov Models (HMMs) to
classify emotions by integrating ECG features with EEG, GSR, respiration, and
skin temperature data. In [54], Stamos et al. fused emotion features extracted
from the ECG signal with the features obtained from the EEG signal to perform
emotion analysis. Santamaria et al. [55] integrated the modalities of ECG and
GSR to recognize emotions by employing Deep Convolutional Network (DCNN).
Elalamy [56] performed emotion analysis using the spectrogram and recurrence
plots (RP) of ECG, EDA, and Photoplethysmography (PPG) signals individually,
and investigated the performance by combining this information for multi-modal
emotion recognition. Most of the aforementioned techniques fuse the ECG signal
with other modalities such as EEG, EMG, GSR, etc., for multimodal emotion
analysis. But, unlike the EEG signal, which has been combined with the facial
information as proposed in [15], [57], [58], the ECG signal has not been fused
with facial features to perform emotion recognition. In this paper, a Unified
Biosensor-Vision Multimodal Transformer-based (UBVMT) emotion recognition
technique is presented that integrates ECG/PPG data with facial information.

##  3 RESEARCH METHODS

This section discusses various 1D-to-2D transformation methods to extract
emotion features from 2D representation of ECG/PPG signals. We then present
our Unified Biosensor-Vision Multimodal Transformer (UBVMT) architecture that
leverages the 2D representation of ECG/PPG signals along with face information
to recognize emotions.

Figure 3: Unimodal emotion recognition

###  3.1 The 2D Representation of ECG/PGG Signals

In this paper, one of our goals is to find an effective method that can be
used to transform the 1D time domain signal of ECG/PPG data into an image-
based representation for emotion recognition. Therefore, in this section, we
investigate the effectiveness of three different 1D-to-2D transformation
techniques to perform 2D image-based emotion analysis using ECG/PPG data. More
specifically, we present a unimodal emotion recognition network using
ResNet-18 [62] by converting the time domain ECG and PPG signals into (1)
spatiotemporal maps [24], (2) time-frequency representation by employing
Smoothed Pseudo Wigner-Ville distribution (SPWVD) [28], and (3) Scalograms
[29], [30] as shown in Figure 3.

####  3.1.1 Spatiotemporal Representation of ECG/PPG signal

In this section, a novel emotion recognition technique using a spatiotemporal
representation of an ECG/PPG signal is presented. Rencheng et al. [24]
converted a 1D ECG and PPG signal into a 2D spatiotemporal map to estimate
heart rate information. It has been reported that the performance degradation
of the conventional PPG signals for heart rate estimation due to noise can be
overcome by employing deep learning techniques on the spatiotemporal maps of
the PPG signal [24]. Therefore, in this section, we apply the technique
proposed in [24] to convert the 1D ECG/PPG signal into a 2D spatiotemporal map
and perform emotion analysis using the spatiotemporal maps as input to the
classifier. The spatiotemporal feature maps can retain both the morphological
and chronological features of ECG/PPG signals.

Construction of ECG/PPG Spatiotemporal Map: The spatiotemporal feature map of
an ECG/PPG signal is constructed by creating a square Toeplitz matrix. Suppose
the 1D input signal $S=(s_{1},s_{2},...,s_{P})$ has P samples and P is an even
number. The first row of the matrix consists of $s_{1}$ to $s_{P/2}$ samples.
Similarly, the second row contains samples from the second point, i.e.,
$s_{2}$ to the $(P/2+1)th$ sample, and so on. Therefore, a square Toeplitz
matrix T with a size equal to $P/2$ is constructed as:

$T=\begin{bmatrix}s_{1}&s_{2}&...&s_{P/2}\\\ s_{2}&s_{3}&...&s_{P/2+1}\\\
.&.&.&.\\\ .&.&.&.\\\ .&.&.&.\\\ s_{P/2}&s_{P/2+1}&...&s_{P-1}\\\
\end{bmatrix}$

A gray image is constructed by converting the matrix T into an image
representation. The obtained gray image has a clear structure because the
input signal is quasiperiodic. The second row of Figure 4 shows the
spatiotemporal image representation of 1D signals. As it can be seen in Figure
4 that the vertical patterns preserve the period information of the 1D signal.
This suggests that the 2D Toeplitz representation can effectively capture and
portray the periodic nature of a 1D signal. The morphological and
chronological information of the 1D signal is preserved in this simple 2D
spatiotemporal representation. Therefore, emotion features can be extracted
from these 2D maps of the ECG/PPG signals. The spatiotemporal image
representations of ECG/PPG signals were then adjusted to the size of 224 × 224
× 3, and fed to a pre-trained ResNet-18 [62] to perform emotion analysis.

####  3.1.2 Time-frequency Representation of ECG/PPG Signal

To leverage the effectiveness of Convolutional Neural Networks (CNNs), Khare
et al. [28] converted 1D EEG signals into 2D maps using a Time–Frequency (TF)
representation for emotion recognition. More specifically, filtered EEG time-
domain signal is transformed into time, frequency, and amplitude
representation by employing Smoothed Pseudo-Wigner–Ville Distribution (SPWVD).
The transformation of time-domain signals into Time-Frequency (TF)
representation enables the preservation of signal information in the spectral
domain. TF map is a representation that combines time, frequency, and
amplitude in a spatial format simultaneously. Therefore, a time-frequency
representation of the ECG/PPG signal can also be obtained by using other time-
frequency analysis techniques such as Short-Time Fourier Transform (STFT),
Wigner–Ville distribution, and so on. The time-frequency representation
obtained by using STFT is known as a spectrogram. To perform Short-Time
Fourier Transform (STFT), certain parameters need to be defined, such as the
window size, shape, and sampling frequency. It is crucial to maintain a
consistent window length across the entire signal. However, the resulting
spectrogram from STFT often exhibits limited resolution due to the trade-off
between time and frequency localization. In contrast, the utilization of the
Wigner-Ville distribution for obtaining the time-frequency representation
results in the presence of cross-terms and attenuation for low frequencies. As
reported in [59], SPWVD offers enhanced time-frequency resolution, effectively
resolving the limitations associated with STFT. The challenges associated with
the Wigner-Ville distribution are tackled in SPWVD by incorporating a cross-
term reducing window in the frequency domain.

Figure 4: 2D representations of ECG/PPG signal

In this paper, one of our goals is to find an effective method that can be
used to transform the 1D time domain signal of ECG/PPG into an image-based
representation. Therefore, to address the aforementioned constraints of STFT
and the Wigner-Ville distribution, in this section, the technique of Smoothed
Pseudo-Wigner–Ville Distribution (SPWVD) is employed to convert time-domain
ECG/PPG signals into a time-frequency representation. The performance of a
SPWVD-based TF representation of ECG/PPG signals is then investigated for
emotion recognition.

Construction of ECG/PPG TF Map: SPWVD provides a straightforward depiction of
the localization of signal energy in both time and frequency domains. The
choice of the length and type of the cross-term reducing window in both the
time and frequency domains can be made independently. Due to the independent
selection of the length and type of the cross-term reducing window, SPWVD
exhibits favorable time-frequency cluster characteristics. The representation
of SPWVD in mathematical terms can be expressed as [59], [60].

$SPWVD(t,f)=\int_{t_{1}}\int_{f_{1}}u(t-t_{1})v(f-f_{1})W(t_{1}f_{1}),dt_{1}df_{1}$
(1)

$W(t,f)=\int_{-\infty}^{\infty}x(t+\frac{\tau}{2})x^{*}(t-\frac{\tau}{2})e^{-j2\pi
f\tau}d\tau$ (2)

where $u(t)$ represents the smoothing window in the time domain, $v(f)$
denotes the smoothing window in the frequency domain, $\tau$ corresponds to a
lag, and $x(t)$ is the input signal. Controlling the smoothing scales in both
the time and frequency domains is straightforward. The length of the windows
for $u(t)$ and $v(f)$ can be chosen independently. The TF representation of
time-domain ECG/PGG signals obtained by employing the SPWVD technique is shown
in the third row of Figure 4.

To mitigate cross-terms in both time and frequency domains, the Kaiser window
is employed. However, selecting a window size that is too small may lead to
diminished resolution, while overly large windows can significantly increase
the image size. Consequently, a medium-sized window with a length of 31, as
suggested in [28], is chosen. For efficient computation, the window size is
maintained at $2^{n}$-1, where n represents the number of bits. The TF
representations of time-domain ECG/PGG signals are then adjusted to the size
of 224 × 224 × 3, and fed to a pre-trained ResNet-18 [62] to perform emotion
recognition.

####  3.1.3 Scalogram of ECG/PPG Signal

Gasparini et al. [29] converted the monodimensional photoplethysmography (PPG)
data into a bidimensional representation, and applied a pre-trained CNN to
classify the different levels of the subject’s hypertension. More
specifically, the 1D PPG signals were converted into scalograms by leveraging
Continuous Wavelet Transform [61]. Gasparini et al. opted to utilize the
scalogram instead of the spectrogram as the image representation for PPG
signals. The reason for this choice is that scalograms offer enhanced
highlighting of low-frequency or rapidly changing frequency components in the
signal, compared to spectrograms. Similarly, in [30], Liang et al. converted
1D PPG signals into 2D scalograms by using Continuous Wavelet Transform to
classify hypertension. In this paper, we investigate the performance of
scalogram representation of ECG/PPG signals for deep learning-based emotion
recognition.

Construction of ECG/PPG scalograms: Continuous Wavelet Transform (CWT) has
been used for decades as a valuable technique for analyzing both time and
frequency information. In this paper, each segment of the ECG/PPG signal was
transformed into a time-frequency representation, known as a scalogram, using
the continuous wavelet transform method. A scalogram represents the absolute
values of the continuous wavelet transform coefficients of a signal, displayed
as a graph of time and frequency. In contrast to a spectrogram, a scalogram
provides improved capability in identifying the low-frequency or rapidly
changing frequency components of the signal. To convert the PPG signal into a
scalogram, we obtained the absolute values of the wavelet coefficients for
each signal segment using the analytic morse (3,60) wavelet. The scalogram
representation of ECG/PGG signals obtained by employing the CWT technique is
shown in the fourth row of Figure 4. The scalograms were then adjusted to the
size of 224 × 224 × 3, and fed to a pre-trained ResNet-18 [62] to perform
emotion recognition.

Figure 5: Multimodal emotion recognition

###  3.2 The Unified Transformer-based Multi-modal Network

In this section, we discuss our Unified Biosensor-Vision Multimodal
Transformer (UBVMT) network where homogeneous transformer blocks take 2D
representations of ECG/PPG signals and raw visual inputs for multi-modal
emotion representation learning with minimal modality-specific design. UBVMT
is trained by employing masked autoencoding [63] and contrastive modeling [22]
to learn effective emotion representation. The overall architecture of our
Unified Biosensor-Vision Multi-modal Transformer (UBVMT) network is shown in
Figure 5. The objective of masked autoencoding is to reconstruct masked
patches of 2D representations of ECG/PPG signals and face video frames, while
contrastive modeling is applied to align ECG/PPG and face information. We
argue that, due to the unified architecture of UBVMT, the computational
redundancy and complexity of our method is reduced as compared to conventional
transformer-based multi-modal emotion recognition techniques.

The input to the UBVMT is the integration of face patch embedding, ECG/PPG
patch embedding, and modality embedding. To obtain face embeddings, the face
image of $224\times 224\times 3$ pixels is divided into a list of $16\times
16$ sized patches. The pixel values of each patch are normalized, and a linear
projection layer is used to convert face patches into a 768-dimensional patch
embedding. For an input face image of size $224\times 224\times 3$, the size
of the resultant face embedding is $14\times 14$. The spatial embedding
involves incorporating spatial information for each input patch by adding a
distinct trainable vector to the height and width axes of the $14\times 14$
embeddings. For ECG/PPG embedding, the 2D representation of an ECG/PPG signal
is divided into patches, and a liner projection layer is applied on each patch
to obtain a 768-dimensional patch embedding. Similar to the face modality, a
patch size of $16\times 16$ is used, and trainable temporal and frequency
embeddings are utilized to denote the temporal and frequency information of
the patches.

####  3.2.1 The architecture of UBVMT

Similar to the transformer model proposed in [18], UBVMT has an encoder E
which has 12 layers (hidden size 768), and a decoder D with 8 layers (hidden
size 512). After pretraining, we only employ the encoder $E$ part of the
transformer and finetune it for emotion recognition.

####  3.2.2 Pretraining UBVMT

UBVMT is pre-trained by employing masked autoencoding [63] and contrastive
modeling [22] to learn effective emotion representation. The pretraining
objective function $\mathcal{L}$ is the weighted sum of masked autoencoding
loss $\mathcal{L_{M}}$ and the contrastive modeling loss $\mathcal{L_{C}}$:

$\mathcal{L}=\lambda_{M}\mathcal{L_{M}}+\lambda_{C}\mathcal{L_{C}}$ (3)

where $\lambda_{M}=0.4$ and $\lambda_{C}=1$.

Masked Autoencoding Objective Function: The main objective of masked
autoencoding is to learn effective unimodal representations in biosensor-and-
vision settings. This objective involves masking random patches of ECG/PPG
images and the face video frames, allowing us to reconstruct missing inputs
effectively. Specifically, we implement a random dropout mechanism on a
portion of ECG/PPG embedding $x^{B}$ and face embedding $x^{F}$. Subsequently,
we input the remaining patch embeddings to the encoder $E$. To generate the
input to the decoder $D$, we add the dropped embeddings as trainable vectors
labeled as [MASK] and position them in the same locations as the original
inputs (indicated by gray boxes in Figure 5. Additionally, the corresponding
positional, and frequency embeddings, which are separately parametrized, are
incorporated into the decoder input. The objective function of masked
autoencoding is a mean squared error between the reconstructed and original
ECG/PPG images and face video frames:

$\mathcal{L_{M}}=\frac{1}{N_{m}^{B}}\sum_{i\in\text{masked}}\|x_{i}^{B}-\hat{x}_{i}^{B}\|_{2}^{2}+\frac{1}{N_{m}^{F}}\sum_{j\in\text{masked}}\|x_{j}^{F}-\hat{x}_{j}^{F}\|_{2}^{2}$
(4)

where $N_{m}^{B}$ and $N_{m}^{F}$ are the number of masked patches for ECG/PPG
images and face video frames, respectively. Loss $\mathcal{L_{M}}$ is computed
only on masked patches. Note that the ECG/PPG and face output of the encoder
are fed to the decoder separately.

Contrastive Modeling Objective Function: The main objective of contrastive
modeling is to perform ECG/PPG-vision matching and learn an effective cross-
modal representation, as shown in Figure 5. For every face image, a positive
vision-ECG/PPG pair $(x^{F+};x^{B})$ is generated. Additionally, we form half
of the vision-ECG/PPG pairs within a batch as mismatched (negative) pairs
$(x^{F-};x^{B})$ by substituting the face images $x^{F+}$ with randomly
selected face images $x^{F-}$ from the training dataset.

Similar to previous multimodal transformers [22], [64], [65], [66], [67], we
incorporate a linear layer with sigmoid activation as the classification head.
This is applied to the encoder output of the first [CLS] token, resulting in
the matching probability $p$ by employing binary cross-entropy loss:

$\mathcal{L_{C}}=-y\log p$ (5)

where the value of y is set to 1 when the input vision-ECG/PPG pair
$(x^{F};x^{B})$ is positive (match), and 0 otherwise. Throughout the training
process, $\mathcal{L_{M}}$ and $\mathcal{L_{C}}$ are computed using separate
forward passes.

##  4 EXPERIMENTAL DETAILS

###  4.1 Database Description

The effectiveness of transformer models to learn multimodal representation is
improved by pretraining them on large datasets [65], [66], [67]. As such,
UBVMT is pre-trained on the large, comprehensive CMU-MOSEI [68] dataset.
Similar to [24], [25], [69], rPPG signals are extracted from the CMU-MOSEI
video clips using the MTTS-CAN [70] method. After pertaining, MAHNOB-HCI [31]
and DEAP [32] datasets are used for multimodal emotion analysis.

The CMU-MOSEI dataset comprises 23,454 movie review clips, totaling over 65.9
hours of YouTube video content. It features contributions from 1000 speakers
and encompasses 250 distinct topics. Videos having non-human faces and faces
with more than $80^{\circ}$ head rotations from the frontal position are
discarded from the dataset. From each video, multiple video clips of 1.1 secs
are extracted, with the number of clips being dependent on the length of each
video. For our case, 90,037 video clips are extracted from the entire CMU-
MOSEI dataset. These video clips are then fed to the MTTS-CAN [70], and only
the pulse signals from the MTTS-CAN are used to obtain the rPPG signals of
these video clips.

The MAHNOB-HCI [31] (Multimodal Human-Computer Interaction) dataset is a
widely used publicly available dataset designed for research in affective
computing and multimodal emotion recognition. The dataset includes 527 facial
video recordings of 27 participants engaged in various tasks and interactions,
while their physiological signals such as 32-channel electroencephalogram
(EEG), 3-channel electrocardiogram (ECG), 1-channel galvanic skin response
(GSR), and facial expressions are captured. The ECG signals were sampled at a
rate of 256 Hz. The ECG data is precisely synchronized and aligned with the
face video recordings. Similar to [71], only the EXG2 signal from the 3 ECG
channel system is extracted for emotion analysis.

The Database for Emotion Analysis using Physiological Signals (DEAP) [32] is a
widely used and publicly available database designed for research in emotion
analysis and affective computing. The DEAP database contains data from 32
participants, aged between 19 and 37 ($50\%$ female), who were recorded
watching 40 one-minute music videos. Each participant was asked to evaluate
each video by assigning values from 1 to 9 for arousal, valence, dominance,
like/dislike, and familiarity. Face video was recorded for 22 out of the 32
participants, and the proposed UBVMT method is evaluated using this particular
group of subjects. For each dataset, we conduct a subject-independent 10-fold
cross-validation evaluation.

###  4.2 Pre-processing Steps

The MAHNOB-HCI emotion elicitation data contains unstimulated baseline and
stimulated response ECG signals. Furthermore, the database includes a
synchronization signal that facilitates the separation of the two. Our
experiments exclusively utilized the ECG signals recorded during the
stimulated phase. To remove motion artifacts, the original signal is
subtracted by the smoothing signal [31]. Subsequently, similar to [16], [17] a
notch filter was applied at $60$ Hz to eliminate power line interference.
Baseline drift was mitigated by implementing a highpass filter at $0.4$ Hz.
Additionally, other noises were eliminated using a low-pass filter set at
$200$ Hz. Due to the variability in the length of the ECG signals, they are
partitioned into segments of 5 seconds each.

The PPG and rPPG signals from the DEAP and CMU-MOSEI datasets are preprocessed
and segmented into segments of pulses corresponding to 1.1 secs following
[36]. PPG signal can be affected by disturbances, such as movement at the
sensor attachment site. Consequently, PPG signals in the DEAP dataset contain
movement noise. Hence, before segmenting PPG signals into single pulses of 1.1
secs, a high-order polynomial (an order of 50 polynomials) is fitted to the
PPG signal. Subsequently, the fitted curve is subtracted from the original PPG
signal, effectively eliminating any movement noise. To partition the PPG
signal into single pulses of 1.1 secs, we first find the maximum or peaks of
the signal and then use the peak value as the center of the segmenting window.
The next pulse is segmented by moving the center of the segmenting window to
the next peak, and so on. The biosignal data vary from person to person; as
such, the PPG/rPPG signals are normalized to mitigate variations in PPG signal
size among individuals. We must exercise caution to retain the personal
characteristics of the signal, as they vary depending on the emotions. The PPG
signals are normalized by employing personal maximum and minimum [36]:

$\bar{z_{i}}=\frac{(z_{i}-min_{person})}{(max_{person}-min_{person})}\times\alpha$
(6)

where $z_{i}$ is the PPG signal, $\bar{z_{i}}$ is the normalized signal, and
$\alpha$ is set to $1000$ to normalize $z_{i}$ between $0$ to $1000$. After
extracting the signal segments and their corresponding video frames, we
balance the dataset by discarding some segments that contain neutral facial
expressions. We employ the off-the-shelf TER-GAN [6] FER model trained on the
in-the-wild AffectNet [10] dataset to detect segments containing non-neutral
facial expressions.

###  4.3 Unimodal Emotion Recognition Details

This section presents the experimental details of the unimodal emotion
recognition experiments that were performed to investigate and compare the
performance of the three 2D representations of ECG/PPG signals. These
experiments were conducted by inputting only the 2D image-based
representations of the ECG signal (in the case of MAHNOB-HCI dataset), and the
PPG signal (in the case of DEAP dataset) to a pre-trained ResNet-18 [62].
ResNet-18 was employed to classify emotions in an arousal-valence space using
both the MAHNOB-HCI and DEAP datasets. In the case of the MAHNOB-HCI dataset,
the arousal dimension consisted of three categories: ’calm’, ’medium’, and
’activated’, while the valence dimension included the categories ’unpleasant’,
’neutral’, and ’pleasant’, similar to the class distribution presented in
[17]. The arousal and valence classes in the DEAP dataset are annotated on a
1-9 scale. Therefore, following [36], the arousal and valence are split into
two binary classes based on a threshold of 5, indicating high and low arousal,
and high and low valence, respectively.

ResNet-18 is finetuned by using Adam [72] optimization algorithm with a
learning rate of 0.001, batch size of 128, and cross-entropy loss function.
The ResNet-18 is trained for 50 epochs, and 10-fold cross-validation is
employed for the evaluation of the 2D ECG/PPG representation-based unimodal
emotion recognition technique. The best emotion recognition performance is
obtained by converting 1D ECG and PPG signals into the 2D scalogram
representation. Further details and comparison of the performance of the three
2D representation techniques are discussed in section 5.1.

###  4.4 Multimodal Emotion Recognition Details

Multimodal emotion recognition is performed by applying our novel Unified
Biosensor-Vision Multi-modal Transformer (UBVMT) network for emotion
recognition in an arousal-valence space using the facial expression
information and 2D representation of ECG/PPG data from the MAHNOB-HCI and DEAP
datasets, respectively. As far as we are aware, we are the first ones to model
the co-relation between the face and ECG/PPG data for emotion recognition
employing a transformer network. For the extraction of the face information,
since the ECG and PPG signals are synchronized with the facial videos of the
participants in both datasets, we partitioned the videos into clips of 5 secs
for the MAHNOB-HCI dataset and 1.1 secs for the DEAP database. Subsequently,
following [7], we extract the first frame from each video clip, pass it to
MTCNN [77] to extract only the face region as a facial expression
representation, and input it to the UBVMT network along with the 2D
representation of the corresponding ECG/PPG signal. As mentioned in section
4.3, the best unimodal emotion recognition performance is obtained by
transforming the 1D ECG and PPG signals into a 2D scalogram representation.
Therefore, multimodal emotion analysis is performed by combining the facial
expression images with the scalograms of ECG and PPG signals using our
proposed UBVMT network.

UBVMT is trained using the Adam [72] optimizer, with a batch size of 4,
learning rate of 1e-4, and using a cosine schedule [73] with a decay rate set
at 0.001. To formulate the pretraining objective in equation 3, the values of
$\lambda_{M}$ and $\lambda_{C}$ are set to 0.4 and 1, respectively. Following
MAE [63] and TVLT [22], a random masking strategy is applied, where $75\%$ of
the face and ECG/PPG patches are randomly masked. After pretraining, the
encoder of UBVMT is detached, and a two-layer MLP is added on top of the
encoder representation for emotion analysis. The encoder plus the MLP layers
are fine-tuned using the Adam [72] optimizer, the learning rate of 1e-4, and a
decay rate of 0.001. All the training and validation processes are carried out
using dual NVIDIA Tesla V100 GPUs.

TABLE I: MAHNOB-HCI dataset: Performance comparison of 2D representations of ECG signal for unimodal emotion recognition. Method | Valance | Arousal
---|---|---
Spatio-temporal maps[24] | 37.62 | 41.04
SPWVD[28] | 38.39 | 42.75
Scalogram[29, 30] | 42.91 | 49.14
TABLE II: DEAP dataset: Performance comparison of 2D representations of PPG signal for unimodal emotion recognition. Method | Valance | Arousal
---|---|---
Spatio-temporal maps[24] | 60.26 | 62.71
SPWVD[28] | 64.03 | 65.19
Scalogram[29, 30] | 76.51 | 77.02

##  5 Results and Analysis

In this section, the experimental results of both the unimodal emotion
recognition and the multimodal emotion analysis using the proposed UBVMT
method are discussed in detail.

###  5.1 Results and Analysis of Unimodal Method

The unimodal emotion recognition performance of the three 2D representations
of ECG/PPG signals for MAHNOB-HCI and DEAP datasets are presented in Table I
and Table II as an average of the 10-fold cross-validation. As it can be seen,
the scalogram representation of both ECG and PPG signals outperforms the
emotion recognition performance of the spatiotemporal maps and the SPWVD-based
2D representation. The superior performance of scalogram representation is
because it is more effective at identifying the low-frequency or rapidly-
changing frequency components of the signal, as both ECG and PPG are low-
frequency signals. Similarly, the emotion recognition accuracy of SPWVD is
higher than the accuracy of spatio-temporal maps because SPWVD offers better
time-frequency resolution, and thus captures emotion features more effectively
than the spatio-temporal maps.

The performance of our unimodal emotion recognition method is also compared
with the state-of-the-art unimodal emotion recognition techniques on both the
MAHNOB-HCI and DEAP datasets, as presented in Table III. For the MAHNOB-HCI
dataset, we compare our result with the ECG-based emotion analysis technique
proposed by Ferdinando et al. [17] and rPPG-based method of Yu et al. [76]. In
[17], Ferdinando et al. extracted heart rate variability (HRV) features from
ECG signals to classify emotions. As it can be seen in Table III, our
scalogram-based method outperforms the emotion recognition technique proposed
in [17]. In [76], Yu et al. extracted ten-dimensional HRV features from rPPG
signals of MAHNOB-HCI video clips, and fed them to a support vector machine
for emotion classification. Table III shows that they achieve a higher
recognition accuracy in the case of valence, while our scalogram-based method
outperforms their method in arousal categorization. Similarly, in Table III,
the performance of our PPG-based emotion classification method is compared
with the state-of-the-art emotion analysis techniques using the DEAP database.
For the comparison of the techniques involving only the PPG signal, our 2D
scalogram-based method is compared with the PPG-based technique proposed by
Lee et al. [36] and various 2D representations of PPG signal employed by
Elalamy et al. [56]. Table III shows that our scalogram-based PPG technique is
more effective than the one-dimensional convolutional neural network-based (1D
CNN) emotion analysis method proposed by Lee et al. [36]. Similarly, our
technique produces higher recognition accuracy than the 2D spectrogram-based
method proposed by Elalamy et al. [56]. Also, our 2D scalogram representation
outperforms the emotion recognition performance of the Recurrence PLot-based
(RP) 2D representation technique used in [56]. Most of the physiological-based
emotion analysis methods employ EEG signals for emotion recognition more than
any other physiological signal due to its capability to capture emotion
features more effectively. Therefore, in Table III, we also compare our
unimodal method with the state-of-the-art unimodal emotion analysis techniques
using EEG signals. Table III shows that our scalogram-based PPG representation
produces higher recognition accuracy as compared to the EEG-based emotion
recognition techniques of Patras et al. [32], Chung et al. [74] and Campos et
al. [75]. These results imply that the scalogram-based 2D representation of
the PPG signal can be used as an alternative to the EEG signal. Obtaining and
utilizing EEG signals, the most prevalent bio-signal in emotion recognition,
can be inconvenient due to the high cost of EEG measurement devices and the
cumbersome measurement process, especially for participants from neurodiverse
populations such as kids with ASD.

TABLE III: Performance comparison of our unimodal emotion recognition method with the state-of-the-art techniques. Study |  Modality | Valance | Arousal
---|---|---|---
MAHNOB-HCI
Ferdinando et al. [17] |  ECG | 42.55 | 47.69
Yu et al. [76] |  HRV | 46.86 | 44.02
Our (Scalogram) |  ECG | 42.91 | 49.14
DEAP
Patras et al. [32] |  EEG | 62.00 | 57.60
Chung et al. [74] |  EEG | 70.90 | 70.10
Campos et al. [75] |  EEG | 73.14 | 73.06
Lee et al. [36] |  PPG | 75.30 | 76.20
Elalamy et al. [56] |  PPG(spectrogram) | 69.60 | 69.30
Elalamy et al. [56] |  PPG(Recurrence Plots) | 69.40 | 69.20
Our (Scalogram) |  PPG | 76.51 | 77.02
TABLE IV: Performance comparison of our multimodal emotion recognition method with the state-of-the-art techniques. Study |  Modality | Valance | Arousal
---|---|---|---
MAHNOB-HCI
Soleymani et al. [31] |  EEG and Eye Gaze | 76.10 | 67.70
Koelstra et al. [15] |  EEG and Face | 73.00 | 68.50
Our (UBVMT) |  ECG and Face | 50.01 | 83.84
DEAP
Yin et al. [53] |  EEG, ECG, EOG, GSR, EMG, Skin temperature, Blood volume, Respiration | 76.17 | 77.19
Shang et al. [13] |  EEG, EOG, EMG | 51.20 | 60.90
Siddharth et al. [7] |  EEG, PPG, GSR | 71.87 | 73.05
Siddharth et al. [7] |  EEG and Face | 73.94 | 74.13
Elalamy et al. [56] |  PPG, EDA | 69.90 | 69.70
Our (UBVMT) |  PPG and Face | 81.53 | 82.64
TABLE V: F1 scores of our multimodal emotion recognition method. Study |  Dataset | Valance | Arousal
---|---|---|---
UBVMT |  MAHNOB-HCI | 0.501 | 0.846
UBVMT |  DEAP | 0.815 | 0.829
Figure 6: The confusion matrices for (a). MAHNOB-HCI valence emotion
recognition, (b). MAHNOB-HCI Arousal emotion recognition, (c). DEAP valence
emotion recognition, and (d). DEAP arousal emotion recognition.

###  5.2 Results and Analysis of Multi-modal Method

The average of the 10-fold cross-validation accuracy of the UBVMT multimodal
emotion recognition method is shown in Table IV. However, since our method
provides a baseline for multimodal emotion recognition by fusing the ECG/PPG
signal with the face information, there are no state-of-the-art methods with
which we can compare our results. Nonetheless, as can be seen in Table IV that
the proposed method outperforms the multimodal techniques fusing EEG
information with other physiological signals, especially for the recognition
of arousal. For the MAHNOB-HCI dataset, we compare our results with the EEG-
based multimodal methods proposed by Soleymani et al. [31] and Koelstra et al.
[15]. Here again, the proposed method outperforms both of these techniques by
a large margin in the recognition of arousal by producing an average accuracy
of $83.84\%$. In contrast, the method proposed by Soleymani et al. [31]
achieves the highest accuracy of $76.10\%$ for the classification of the
valence class. We argue that the superior performance of our technique in
recognizing the arousal class is because UBVMT is pre-trained by employing an
rPPG signal, and it has been reported in past research that PPG signals are
more effective in categorizing arousal [29]. Therefore, given a large enough
ECG dataset to pre-train the UBVMT network, we posit that we can improve the
classification accuracy of our method for the valence class as well. For the
DEAP dataset, we compare our results with the multimodal techniques fusing PPG
information with other signals like EEG, EDA GSR, etc. As can be seen in Table
IV, our method outperforms the state-of-the-art multimodal emotion
classification techniques both in recognizing valence and arousal with average
accuracies of $81.53\%$ and $82.64\%$, respectively. It is interesting to note
that our bimodal method outperforms the multimodal technique proposed by Yin
et al. [53], where the information from EEG, ECG, EOG, GSR, EMG, Skin
temperature, blood volume, and respiration signals are fused to classify
emotions. Similarly, comparing our technique with the bimodal method proposed
by Siddharth et al. [7] where EEG and face information are fused for emotion
analysis, Table IV shows that the fusion of PPG and face information using
UBVMT is more effective in recognizing emotions. Our PPG-face method is also
more accurate in classifying emotions compared to the fusion of EEG, EOG, and
EMG signals [13]. In [56], similar to our approach, Elalamy et al. transformed
the 1D PPG and EDA signals into 2D representations and fuse them to categorize
emotions using deep networks. Table IV shows that our scalogram-based PPG
representation, when fused with the face data, produces higher recognition
accuracy by employing our proposed technique. The confusion matrices and the
F1 scores of the multimodal emotion recognition results are shown in Figure 6
and Table V, respectively.

##  6 Conclusion

In this paper, a novel approach called Unified Biosensor-Vision Multi-modal
Transformer-based (UBVMT) method is presented to classify emotions in an
arousal-valence space. The proposed technique combines a 2D representation of
an ECG/PPG signal with facial information for emotion recognition. Initially,
we investigated and compared the unimodal emotion recognition performance
using three image-based representations of the ECG/PPG signal. Then, the
Unified Biosensor-Vision Multi-modal Transformer-based network is used for
emotion recognition by combining the 2D image-based representation of the
ECG/PPG signal and facial information. Our unified transformer model comprises
homogeneous transformer blocks that take the 2D representation of the ECG/PPG
signal and associated face frame as input for emotion representation learning
with minimal modality-specific design. The UBVMT model is trained using a
reconstruction loss involving masked patches of video frames and 2D images of
ECG/PPG signals, along with contrastive modeling to align face and ECG/PPG
data. Extensive experiments on the MAHNOB-HCI and DEAP datasets demonstrate
that our Unified Biosensor-Vision Multi-modal Transformer-based model achieves
comparable results to state-of-the-art techniques.

## Acknowledgments

The authors would like to thank Mustansar Fiaz, Sachin Shah, Robinson Vasquez,
Lisa Dieker, Shaunn Smith. Rebecca Hines, Ilene Wilkins, Kate Ingraham,
Caitlyn Bukaty, Karyn Scott, Eric Imperiale, Wilbert Padilla, Maria Demesa for
the discussions and suggestions throughout this research. This research was
supported in part by grants from the National Science Foundation Grants
2114808, and from the U.S. Department of Education Grants H327S210005,
H327S200009. The lead author (KA) was also supported in part by the University
of Central Florida’s Preeminent Postdoctoral Program (P3). Any opinions,
findings, conclusions, or recommendations expressed in this material are those
of the authors and do not necessarily reflect the views of the sponsors.

## References

  * [1] LaFleur, Karl, Kaitlin Cassady, Alexander Doud, Kaleb Shades, Eitan Rogin, and Bin He. ”Quadcopter control in three-dimensional space using a noninvasive motor imagery-based brain–computer interface.” Journal of neural engineering 10, no. 4 (2013): 046003.
  * [2] Millán, Jd R., Frederic Renkens, Josep Mourino, and Wulfram Gerstner. ”Noninvasive brain-actuated control of a mobile robot by human EEG.” IEEE Transactions on biomedical Engineering 51, no. 6 (2004): 1026-1033.
  * [3] Bamidis, Panagiotis D., Christos Papadelis, Chrysoula Kourtidou-Papadeli, Costas Pappas, and Ana B. Vivas. ”Affective computing in the era of contemporary neurophysiology and health informatics.” Interacting with Computers 16, no. 4 (2004): 715-721.
  * [4] Coyle, Shirley, Yanzhe Wu, King-Tong Lau, Sarah Brady, Gordon Wallace, and Dermot Diamond. ”Bio-sensing textiles-wearable chemical biosensors for health monitoring.” In 4th International Workshop on Wearable and Implantable Body Sensor Networks (BSN 2007) March 26–28, 2007 RWTH Aachen University, Germany, pp. 35-39. Springer Berlin Heidelberg, 2007.
  * [5] Riva, Giuseppe, Fabrizia Mantovani, Claret Samantha Capideville, Alessandra Preziosa, Francesca Morganti, Daniela Villani, Andrea Gaggioli, Cristina Botella, and Mariano Alcañiz. ”Affective interactions using virtual reality: the link between presence and emotions.” Cyberpsychology and behavior 10, no. 1 (2007): 45-56.
  * [6] Ali, Kamran, and Charles E. Hughes. ”Facial expression recognition by using a disentangled identity-invariant expression representation.” In 2020 25th International Conference on Pattern Recognition (ICPR), pp. 9460-9467. IEEE, 2021.
  * [7] Jung, Tzyy-Ping, and Terrence J. Sejnowski. ”Utilizing deep learning towards multi-modal bio-sensing and vision-based affective computing.” IEEE Transactions on Affective Computing 13, no. 1 (2019): 96-107.
  * [8] Fard, Ali Pourramezan, and Mohammad H. Mahoor. ”Ad-corre: Adaptive correlation-based loss for facial expression recognition in the wild.” IEEE Access 10 (2022): 26756-26768.
  * [9] Kollias, Dimitrios, and Stefanos Zafeiriou. ”Exploiting multi-cnn features in cnn-rnn based dimensional emotion recognition on the omg in-the-wild dataset.” IEEE Transactions on Affective Computing 12, no. 3 (2020): 595-606.
  * [10] Mollahosseini, Ali, Behzad Hasani, and Mohammad H. Mahoor. ”Affectnet: A database for facial expression, valence, and arousal computing in the wild.” IEEE Transactions on Affective Computing 10, no. 1 (2017): 18-31.
  * [11] C. Mühl, B. Allison, A. Nijholt, G. Chanel, A survey of affective brain computer interfaces: principles, state-of-the-art, and challenges, Brain-Computer Interfaces 1 (2) (2014) 66–84.
  * [12] S. D’mello, J. Kory, A Review and Meta-Analysis of Multimodal Affect Detection .
  * [13] Wang, Dan, and Yi Shang. ”Modeling physiological data with deep belief networks.” International journal of information and education technology (IJIET) 3, no. 5 (2013): 505.
  * [14] Miranda, J., M. Khomami Abadi, N. Sebe, and I. Patras. ”Amigos: A dataset for mood, personality and affect research on individuals and groups.” IEEE T Affect Comput (2017): 02.
  * [15] Koelstra, Sander, and Ioannis Patras. ”Fusion of facial expressions and EEG for implicit affective tagging.” Image and Vision Computing 31, no. 2 (2013): 164-174.
  * [16] Ferdinando, Hany, Tapio Seppänen, and Esko Alasaarela. ”Comparing features from ECG pattern and HRV analysis for emotion recognition system.” In 2016 IEEE Conference on Computational Intelligence in Bioinformatics and Computational Biology (CIBCB), pp. 1-6. IEEE, 2016.
  * [17] Ferdinando, Hany, Liang Ye, Tapio Seppänen, and Esko Alasaarela. ”Emotion recognition by heart rate variability.” Australian Journal of Basic and Applied Science 8, no. 14 (2014): 50-55.
  * [18] Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. ”Attention is all you need.” Advances in neural information processing systems 30 (2017).
  * [19] Le, Hoai-Duy, Guee-Sang Lee, Soo-Hyung Kim, Seungwon Kim, and Hyung-Jeong Yang. ”Multi-Label Multimodal Emotion Recognition With Transformer-Based Fusion and Emotion-Level Representation Learning.” IEEE Access 11 (2023): 14742-14751.
  * [20] Mocanu, Bogdan, Ruxandra Tapu, and Titus Zaharia. ”Multimodal emotion recognition using cross modal audio-video fusion with attention and deep metric learning.” Image and Vision Computing (2023): 104676.
  * [21] Yu, Jianfei, Kai Chen, and Rui Xia. ”Hierarchical interactive multimodal transformer for aspect-based multimodal sentiment analysis.” IEEE Transactions on Affective Computing (2022).
  * [22] Tang, Zineng, Jaemin Cho, Yixin Nie, and Mohit Bansal. ”TVLT: Textless Vision-Language Transformer.” arXiv preprint arXiv:2209.14156 (2022).
  * [23] Wang, Zhe, Yongxiong Wang, Chuanfei Hu, Zhong Yin, and Yu Song. ”Transformers for EEG-based emotion recognition: A hierarchical spatial information learning model.” IEEE Sensors Journal 22, no. 5 (2022): 4359-4368.
  * [24] Song, Rencheng, Senle Zhang, Chang Li, Yunfei Zhang, Juan Cheng, and Xun Chen. ”Heart rate estimation from facial videos using a spatiotemporal representation with convolutional neural networks.” IEEE Transactions on Instrumentation and Measurement 69, no. 10 (2020): 7411-7421.
  * [25] Niu, Xuesong, Hu Han, Shiguang Shan, and Xilin Chen. ”Synrhythm: Learning a deep heart rate estimator from general to specific.” In 2018 24th International Conference on Pattern Recognition (ICPR), pp. 3580-3585. IEEE, 2018.
  * [26] Hsu, Gee-Sern, ArulMurugan Ambikapathi, and Ming-Shiang Chen. ”Deep learning with time-frequency representation for pulse estimation from facial videos.” In 2017 IEEE international joint conference on biometrics (IJCB), pp. 383-389. IEEE, 2017.
  * [27] Lu, Hao, Hu Han, and S. Kevin Zhou. ”Dual-gan: Joint bvp and noise modeling for remote physiological measurement.” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12404-12413. 2021.
  * [28] Khare, Smith K., and Varun Bajaj. ”Time–frequency representation and convolutional neural network-based emotion recognition.” IEEE transactions on neural networks and learning systems 32, no. 7 (2020): 2901-2909.
  * [29] Gasparini, Francesca, Alessandra Grossi, and Stefania Bandini. ”A deep learning approach to recognize cognitive load using ppg signals.” In The 14th PErvasive Technologies Related to Assistive Environments Conference, pp. 489-495. 2021.
  * [30] Liang, Yongbo, Zhencheng Chen, Rabab Ward, and Mohamed Elgendi. ”Photoplethysmography and deep learning: enhancing hypertension risk stratification.” Biosensors 8, no. 4 (2018): 101.
  * [31] Soleymani, Mohammad, Jeroen Lichtenauer, Thierry Pun, and Maja Pantic. ”A multimodal database for affect recognition and implicit tagging.” IEEE transactions on affective computing 3, no. 1 (2011): 42-55.
  * [32] Koelstra, Sander, Christian Muhl, Mohammad Soleymani, Jong-Seok Lee, Ashkan Yazdani, Touradj Ebrahimi, Thierry Pun, Anton Nijholt, and Ioannis Patras. ”Deap: A database for emotion analysis; using physiological signals.” IEEE transactions on affective computing 3, no. 1 (2011): 18-31.
  * [33] Russell, James A. ”A circumplex model of affect.” Journal of personality and social psychology 39, no. 6 (1980): 1161.
  * [34] Yu, Sung-Nien, Shao-Wei Wang, and Yu Ping Chang. ”Improving Distinguishability of Photoplethysmography in Emotion Recognition Using Deep Convolutional Generative Adversarial Networks.” IEEE Access 10 (2022): 119630-119640.
  * [35] Ismail, Sharifah Noor Masidayu Sayed, Nor Azlina Ab Aziz, and Siti Zainab Ibrahim. ”A comparison of emotion recognition system using electrocardiogram (ECG) and photoplethysmogram (PPG).” Journal of King Saud University-Computer and Information Sciences 34, no. 6 (2022): 3539-3558.
  * [36] Lee, M.S., Lee, Y.K., Pae, D.S., Lim, M.T., Kim, D.W. and Kang, T.K., 2019. Fast emotion recognition based on single pulse PPG signal with convolutional neural network. Applied Sciences, 9(16), p.3355.
  * [37] Sepúlveda, A., Castillo, F., Palma, C. and Rodriguez-Fernandez, M., 2021. Emotion recognition from ECG signals using wavelet scattering and machine learning. Applied Sciences, 11(11), p.4945.
  * [38] Sarkar, P. and Etemad, A., 2020. Self-supervised ECG representation learning for emotion recognition. IEEE Transactions on Affective Computing, 13(3), pp.1541-1554.
  * [39] Mellouk, W. and Handouzi, W., 2023. CNN-LSTM for automatic emotion recognition using contactless photoplythesmographic signals. Biomedical Signal Processing and Control, 85, p.104907.
  * [40] Ali, K. and Hughes, C.E., 2021, January. Facial expression recognition by using a disentangled identity-invariant expression representation. In 2020 25th International Conference on Pattern Recognition (ICPR) (pp. 9460-9467). IEEE.
  * [41] Li, H., Wang, N., Yang, X., Wang, X. and Gao, X., 2022. Towards semi-supervised deep facial expression recognition with an adaptive confidence margin. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 4166-4175).
  * [42] Wang, H., Li, B., Wu, S., Shen, S., Liu, F., Ding, S. and Zhou, A., 2023. Rethinking the Learning Paradigm for Dynamic Facial Expression Recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 17958-17968).
  * [43] Lo, L., Ruan, B.K., Shuai, H.H. and Cheng, W.H., 2023. Modeling Uncertainty for Low-Resolution Facial Expression Recognition. IEEE Transactions on Affective Computing.
  * [44] Ali, K. and Hughes, C.E., 2020. Face reenactment based facial expression recognition. In Advances in Visual Computing: 15th International Symposium, ISVC 2020, San Diego, CA, USA, October 5–7, 2020, Proceedings, Part I 15 (pp. 501-513). Springer International Publishing.
  * [45] Russell, J.A., 1980. A circumplex model of affect. Journal of personality and social psychology, 39(6), p.1161.
  * [46] Thayer, R.E., 1990. The biopsychology of mood and arousal. Oxford University Press.
  * [47] Gunes, H. and Pantic, M., 2010. Automatic, dimensional and continuous emotion recognition. International Journal of Synthetic Emotions (IJSE), 1(1), pp.68-99.
  * [48] Kollias, D. and Zafeiriou, S., 2018. A multi-component CNN-RNN approach for dimensional emotion recognition in-the-wild. arXiv preprint arXiv:1805.01452.
  * [49] Deng, D., Chen, Z. and Shi, B.E., 2020, November. Multitask emotion recognition with incomplete labels. In 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020) (pp. 592-599). IEEE.
  * [50] Kuhnke, F., Rumberg, L. and Ostermann, J., 2020, November. Two-stream aural-visual affect analysis in the wild. In 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020) (pp. 600-605). IEEE.
  * [51] Zhang, Y.H., Huang, R., Zeng, J. and Shan, S., 2020, November. M 3 f: Multi-modal continuous valence-arousal estimation in the wild. In 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020) (pp. 632-636). IEEE.
  * [52] Zeng, Z., Pantic, M. and Huang, T.S., 2009. Emotion recognition based on multimodal information. In Affective information processing (pp. 241-265). London: Springer London.
  * [53] Yin, Z., Zhao, M., Wang, Y., Yang, J. and Zhang, J., 2017. Recognition of emotions using multimodal physiological signals and an ensemble deep learning model. Computer methods and programs in biomedicine, 140, pp.93-110.
  * [54] Katsigiannis, S. and Ramzan, N., 2017. DREAMER: A database for emotion recognition through EEG and ECG signals from wireless low-cost off-the-shelf devices. IEEE journal of biomedical and health informatics, 22(1), pp.98-107.
  * [55] Santamaria-Granados, L., Munoz-Organero, M., Ramirez-Gonzalez, G., Abdulhay, E. and Arunkumar, N.J.I.A., 2018. Using deep convolutional neural network for emotion detection on a physiological signals dataset (AMIGOS). IEEE Access, 7, pp.57-67.
  * [56] Elalamy, R., Fanourakis, M. and Chanel, G., 2021, September. Multi-modal emotion recognition using recurrence plots and transfer learning on physiological signals. In 2021 9th International Conference on Affective Computing and Intelligent Interaction (ACII) (pp. 1-7). IEEE.
  * [57] Soleymani, M., Asghari-Esfeden, S., Fu, Y. and Pantic, M., 2015. Analysis of EEG signals and facial expressions for continuous emotion detection. IEEE Transactions on Affective Computing, 7(1), pp.17-28.
  * [58] Wang, S., Qu, J., Zhang, Y. and Zhang, Y., 2023. Multimodal Emotion Recognition From EEG Signals and Facial Expressions. IEEE Access, 11, pp.33061-33068.
  * [59] de Souza Neto, E.P., Custaud, M.A., Frutoso, J., Somody, L., Gharib, C. and Fortrat, J.O., 2001. Smoothed pseudo Wigner–Ville distribution as an alternative to Fourier transform in rats. Autonomic Neuroscience, 87(2-3), pp.258-267.
  * [60] Scholl, Stefan. ”Fourier, Gabor, Morlet or Wigner: comparison of time-frequency transforms.” arXiv preprint arXiv:2101.06707 (2021).
  * [61] Antoine, J-P., Pierre Carrette, Romain Murenzi, and Bernard Piette. ”Image analysis with two-dimensional continuous wavelet transform.” Signal processing 31, no. 3 (1993): 241-272.
  * [62] He, Kaiming, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. ”Deep residual learning for image recognition.” In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778. 2016.
  * [63] He, K., Chen, X., Xie, S., Li, Y., Dollár, P. and Girshick, R., 2022. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 16000-16009).
  * [64] Tan, Hao, and Mohit Bansal. ”Lxmert: Learning cross-modality encoder representations from transformers.” arXiv preprint arXiv:1908.07490 (2019).
  * [65] Lu, Jiasen, Dhruv Batra, Devi Parikh, and Stefan Lee. ”Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.” Advances in neural information processing systems 32 (2019).
  * [66] Kim, Wonjae, Bokyung Son, and Ildoo Kim. ”Vilt: Vision-and-language transformer without convolution or region supervision.” In International Conference on Machine Learning, pp. 5583-5594. PMLR, 2021.
  * [67] Chen, Yen-Chun, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. ”Uniter: Learning universal image-text representations.” (2019).
  * [68] Zadeh, AmirAli Bagher, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. ”Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph.” In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2236-2246. 2018.
  * [69] Wu, Yi-Chiao, Li-Wen Chiu, Chun-Chih Lai, Bing-Fei Wu, and Sunny SJ Lin. ”Recognizing, Fast and Slow: Complex Emotion Recognition with Facial Expression Detection and Remote Physiological Measurement.” IEEE Transactions on Affective Computing (2023).
  * [70] Liu, Xin, Josh Fromm, Shwetak Patel, and Daniel McDuff. ”Multi-task temporal shift attention networks for on-device contactless vitals measurement.” Advances in Neural Information Processing Systems 33 (2020): 19400-19411.
  * [71] Yu, Zitong, Wei Peng, Xiaobai Li, Xiaopeng Hong, and Guoying Zhao. ”Remote heart rate measurement from highly compressed facial videos: an end-to-end deep learning solution with video enhancement.” In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 151-160. 2019.
  * [72] Kingma, Diederik P., and Jimmy Ba. ”Adam: A method for stochastic optimization.” arXiv preprint arXiv:1412.6980 (2014).
  * [73] Ilya Loshchilov and Frank Hutter. ”Decoupled weight decay regularization”. In ICLR, 2019.
  * [74] Yoon, Hyun Joong, and Seong Youb Chung. ”EEG-based emotion estimation using Bayesian weighted-log-posterior function and perceptron convergence algorithm.” Computers in biology and medicine 43, no. 12 (2013): 2230-2237.
  * [75] Atkinson, John, and Daniel Campos. ”Improving BCI-based emotion recognition by combining EEG feature selection and kernel classifiers.” Expert Systems with Applications 47 (2016): 35-41.
  * [76] Yu, Zitong, Xiaobai Li, and Guoying Zhao. ”Remote photoplethysmograph signal measurement from facial videos using spatio-temporal networks.” arXiv preprint arXiv:1905.02419 (2019).
  * [77] Zhang, Kaipeng, Zhanpeng Zhang, Zhifeng Li, and Yu Qiao. ”Joint face detection and alignment using multitask cascaded convolutional networks.” IEEE signal processing letters 23, no. 10 (2016): 1499-1503.

|  Kamran Ali Kamran Ali is a postdoctoral research associate in the computer
science department at the University of Central Florida, Orlando, FL, 32816,
USA. His research interests include computer vision and machine learning. He
is the corresponding author of this article. Contact him at
<EMAIL_ADDRESS>
---|---
|  Charles E. Hughes Charles E. Hughes is a pegasus professor of Computer
Science at the University of Central Florida, Orlando, FL, 32816, USA. His
research interests include virtual learning environments, computer graphics,
machine learning, and visual programming systems. Contact him at
<EMAIL_ADDRESS>
---|---