# Mental Workload Estimation with Electroencephalogram Signals by Combining
Multi-Space Deep Models

Hong-Hai Nguyen, Ngumimi Karen Iyortsuun, Hyung-Jeong Yang, Guee-Sang Lee, and
Soo-Hyung Kim
Chonnam National University, Gwangju, 61186, South Korea
Corresponding author: Soo-Hyung Kim

###### Abstract

The human brain is in a continuous state of activity during both work and
rest. Mental activity is a daily process, and when the brain is overworked, it
can have negative effects on human health. In recent years, great attention
has been paid to early detection of mental health problems because it can help
prevent serious health problems and improve quality of life. Several signals
are used to assess mental state, but the electroencephalogram (EEG) is widely
used by researchers because of the large amount of information it provides
about the brain. This paper aims to classify mental workload into three states
and estimate continuum levels. Our method combines multiple dimensions of
space to achieve the best results for mental estimation. In the time domain
approach, we use Temporal Convolutional Networks, and in the frequency domain,
we propose a new architecture called the Multi-Dimensional Residual Block,
which combines residual blocks.

_K_ eywords Electroencephalogram $\cdot$ Mental Workload $\cdot$ Temporal
Convolutional Networks $\cdot$ Time-Frequency domains

##  1 INTRODUCTION

Mental workload (MWL) refers to brain activities, which are the number of
resources in the human brain. The level of the resource can be changed when
the human is thinking or performing a task. Because the mental workload
resource of a person is limited, the ability to process a lot of information
at once will be limited1. Humans will be bored if they do easy work when their
MWL is low and their MWL is high when they do complicated tasks. However, a
high MWL is not suitable for their health and can impair their abilities for
memory, communication, activity, etc. For jobs with high MWL, such as doctors,
soldiers and pilots, the brain sometimes has serious accidents. Furthermore,
understanding how the human brain works in daily activities and tasks is an
essential area of neuroergonomics research. Therefore, the MWL estimation
helps to observe and help people at work and evaluate their work system, which
can be improved in the future2. Because the EEG signal is a recording of the
biological activities of individual brain cells or a subset of brain cells
transmitted directly or indirectly through the cerebral cortex and scalp. EEG
signals that record brain activities reflect physiological and pathological
functions of the hemisphere or of the whole brain related to clinical
symptoms, supplementing diagnosis and monitoring treatment, called a clinical
electroencephalogram. By providing a reliable EEG can contribute to predicting
and preventing work-related risks. Diagnostic applications in MWL estimation
can be created to address various real-world issues in healthcare, education,
and smart traffic, among others.

Three metrics were used to measure MWL, namely subject, performance, and
physiological measures2-5. The traditional method to measure the context is
through a set of various questions, such as the Aeronautics and Space
Administration Task Load Index (NASA-TLX) or Assessment Technique (SWAT).
Performance measurement involves measuring the person’s performance during the
task with an increasing workload. Both subject and performance measures are
taken after the task is completed, making them prone to bias. On the contrary,
physiological measurements can continuously record information about the
workload and do not affect the performance of the main task. Therefore,
physiological signals can be used to evaluate the effectiveness of mental
workload.

Physiological measurement uses a variety of biosignals such as
electroencephalogram (EEG), Galvanic Skin Response (GSR), Heart Rate
Variability (HRV), Skin Temperature (Temp), R-R intervals (RR) and
electrocardiogram (ECG). Among such signals, EEG is widely used for the
estimation of MWL6 because the information capacity in EEG is higher than
other physiological signals7. However, most researchers do not release their
datasets, which makes it challenging to compare methods. Fortunately, Lim et
al.8 provided an open dataset with a large number of participants and MWL
levels. In this study, we conducted experiments on this dataset to estimate
the MWL. Our work comprises classification (low, middle and high) and
continuous levels estimation (our proposed) for MWL. Specifically, we proposed
a Multi-Dimensional Residual Block to learn frequency domain information, and
we combined time and frequency domain information based on multimodal fusion,
which significantly improved the results.

Figure 1: System overview: Combined time and frequency domains for mental
workload estimation.

Figure 2: Overview of converting EEG signals to a 3D representation.

##  2 RELATED WORK

Traditional Machine learning and Deep learning are widely used methods to
classify MWL. Handcrafted features usually combine with traditional methods,
while deep learning automatically extracts features for classification. Wang
et al.2 employed Support Vector Machine (SVM) to classify different levels of
MWL, including low workload (0-back) and high workload (1, 2, 3-back), as well
as distinguishing between different levels of workload such as 1-back and
2-back (80%) and 1-back and 3-back (84%). The authors extracted a range of
features based on statistical, signal power, morphological, and frequency
features for binary classification. Ladekar et al. 9 utilized K-nearest
neighbors (KNN) to classify four levels of cognitive load by decomposing the
EEG signal into ten subbands and combining it with ensemble subspace KNN.
Their method achieved an accuracy of 83.65% for four classes. However, the
authors performed the experiment on a small sample size of less than 2000,
limiting the generalizability of their findings. Gómez et al.1 used SVM, KNN,
and random forest (RF), which are combined with features extraction from
frequency bands, the energy and the entropy of the Discrete Wavelet Transform
(DWT), Hjorth parameters, Fractal dimension, Detrended fluctuation analysis,
and Lempel-Ziv complexity. Zammouri et al.10 used PSD and archived an accuracy
of $79\%$ in the theta band. The alpha band got $78\%$ of the average
accuracy. There are two levels of cognitive load used to classify. The authors
showed that theta and alpha powers decrease with the increasing level of
mental task in the central and posterior locations, respectively. The research
methods employ the Machine Learning technique for classification. To achieve
desirable outcomes, it necessitates the calculation and appropriate selection
of features. However, the drawback of these methods is that they do not
incorporate continuous information of the time or frequency domain, which rely
on handcrafted features.

Automatic feature extraction of the deep learning model is better than
handcrafted features. In their work, Yang et al.11 achieved a classification
accuracy of $92\%$ for MWL by using SDAE (Stacked Denoising Autoencoder) to
preserve local information in the EEG. They also applied an ensemble
classifier to classify binary MWL (low and high) by extracting PSD features
(theta, alpha, beta, and gamma). Qiao et al.12 proposed a Ternary Task
Convolutional Bidirectional Neural Turing Machine (TT-CBNTM) for the
classification of cognitive states, which are four levels of cognitive load.
The TT-CBNTM learns temporal information and preserves spectral and spatial
information of EEG. Moreover, the TT-CBNTM can effectively avoid overfitting.
Although the method gives quite high results (96.3%), the authors performed it
on a small sample (2600 samples). Kwak et al.13 achieved a binary
classification accuracy of $93.9\%$ using a combination of a three-dimensional
convolutional neural network (3D CNN) and a long short-term memory model
(LSTM). They obtained 3D EEG images from the 1D EEG signal, and the 3D-CNN
learned spatial and spectral information from these images, while the LSTM
extracted temporal attention from subframes of the EEG image. Deep learning
techniques have the capability to extract features automatically. However,
previous studies have predominantly used frequency domain information as input
for the models. As a result, the outcomes indicate that the potential of time-
domain information has not been fully utilized.

In the field of emotion and motor imagery, a 3D representation of the EEG
signal has been used. To maintain spatial information on the electrodes, Zhao
et al.14 converted the 2D array of EEG signals into 3D. They applied a multi-
branch 3D-CNN to classify motor imagery using the 3D representation, which can
be extended to other related areas such as cognitive load and seizure
detection. Similarly, Liu et al.15 utilized 3D representation to classify
motor stages while preserving spatial and temporal information, by passing the
3D representation through a three-branch 3D-CNN. Jia et al.16 used two 3D
representations (spectral feature and temporal feature) to complement the
significant difference between features. For EEG emotion recognition, a
spatial-spectral-temporal based attention 3D dense network was proposed, which
performed well on two datasets.

We propose a fusion of the time and frequency domains to leverage the richness
of information available in both spaces for the classification task. It is
noteworthy that most existing methods are focused on classification, and a
regression approach for the continuous estimation of MWL remains relatively
less explored. In this study, we adopt a regression framework for continuous
MWL estimation. Furthermore, we exploit the benefits of 3D representation to
facilitate the learning of complex relationships among the features extracted
from EEG signals.

##  3 PROPOSED METHOD

In this section, we present a deep learning model for estimating MWL based on
both time and frequency domain information. We describe the 3D (three-
dimensional) representation of PSD into 3D block as depicted in Figure 2.
Additionally, we propose a new architecture, the Multi-Dimensional Residual
Block (MDRB), illustrated in Figure 3. Finally, we provide an overview of our
method in Figure 1.

###  3.1 3D representation block

We define $S^{n}=[E_{1}^{n},E_{2}^{n},...,E_{W}^{n}]^{T}\in\mathbb{R}^{W\times
C}$ denote the $n^{th}$ sample, where $n\in{1,2,...N}$ and $N$ is the number
of samples. Here, $W$ represents the window size, $C$ represents the number of
channels in the EEG devices, and
$E_{i}^{n}=[e_{i}^{1},e_{i}^{2},...,e_{i}^{C}]\in\mathbb{R}^{C}$ is the EEG
signal at timestamp $i$, where $i\in{1,2,...,W}$. For analysis, we convert EEG
signals from the time domain ($S^{n}$) to the frequency domain using Welch’s
method to obtain the Power Spectral Density (PSD). Each frequency band carries
information about different brain activities and states. Therefore,
frequencies link changes in brain activity with various stages of brain waves.
Initially, Hans Berger identified two bands: alpha ($\alpha$) and beta
($\beta$). Subsequently, researchers introduced additional bands such as delta
($\delta$), theta ($\theta$), sigma ($\sigma$), and gamma ($\gamma$). In this
study, we extract six frequency bands: delta (0.5-4 Hz), theta (4-8Hz), alpha
(8-12Hz), sigma (12-16Hz), beta (16-30Hz), and gamma (30-45Hz). The extracted
frequency bands depend on the window length of the Fast Fourier Transform
(FFT). Therefore, the length of values in each band may differ, but the number
of channels remains the same (14 channels). We define PSD for the six bands as
follows:

$\centering\begin{split}P^{n}&=[P_{1\delta}^{n},P_{2\delta}^{n},...,P_{k\delta}^{n},P_{1\theta}^{n},P_{2\theta}^{n},...,P_{k\theta}^{n},P_{1\alpha}^{n},P_{2\alpha}^{n},...,P_{k\alpha}^{n},\\\
&P_{1\sigma}^{n},P_{2\sigma}^{n},...,P_{k\sigma}^{n},P_{1\beta}^{n},P_{2\beta}^{n},...,P_{k\beta}^{n},P_{1\gamma}^{n},P_{2\gamma}^{n},...,P_{k\gamma}^{n}]^{T}\\\
&\in\mathbb{R}^{K\times C}\end{split}\@add@centering$ (1)

where $K=\sum_{i=1}^{6}kb$ is the total length of frequency of the bands. $kb$
is the length of the frequency band $b$,
$b\in\\{\delta,\theta,\alpha,\sigma,\beta,\gamma\\}$,
$P_{ib}^{n}=[p_{ib}^{1},p_{ib}^{2},...,p_{ib}^{C}]\in\mathbb{R}^{C}$ is the
signal frequency band $b$ at the frequency $i^{th}$.

Zhao et al. 14 proposed a method for converting EEG data from 2D to 3D
representation. The maximum number of horizontal and vertical electrodes is
selected as the width and height of the matrix, respectively, based on the
electrode positions. Fourteen signal positions ($P_{ib}^{n}$) are then mapped
onto the 2D matrix with a size of $6\times 6$. If an electrode position is
empty, it is filled with a value of zero. The resulting 2D matrix is shown in
Equation 2.

$f(X_{j})=\begin{bmatrix}0&0&AF3_{j}&AF4_{j}&0&0\\\
0&F7_{j}&F3_{j}&F4_{j}&F8_{j}&0\\\ 0&0&FC5_{j}&FC6_{j}&0&0\\\
0&T7_{j}&0&0&T8_{j}&0\\\ 0&0&P7_{j}&P8_{j}&0&0\\\ 0&0&O1_{j}&O2_{j}&0&0\\\
\end{bmatrix}$ (2)

where $X_{j}$ is the signal frequency of the band at the frequency $j^{th}$.

The extracted frequency bands step is shown in Figure 2 as the Power Spectral
Density step. From each band, we convert 2D matrix to 3D block with the size
$6\times 6\times kb$. There are six blocks from six frequency bands (the 3D
Block step as Figure 2). We stack them as a larger 3D block (the 3D
representation step, illustrated in Figure 2). The height and width dimension
keeps relationships between channels, and the depth dimension maintains
information about the frequency and continuity of the bands. We define 3D
block as $M^{n}=[M_{1}^{n},M_{2}^{n},...,M_{K}^{n}]\in\mathbb{R}^{6\times
6\times K}$, where $M_{i}^{n}=f(P_{ib}^{n})$, $ib$ is the band $b$ at
frequency $i^{th}$.

###  3.2 MDRB: Multi-Dimensional Residual Block

We propose a Multi-Dimensional Residual Block (MDRB) to capture the
relationship between channels and frequency bands, as illustrated in Figure 3.
The input to the MDRB is a 3D block of size $6\times 6\times K$, where $K$ is
the depth of the 3D input. The MDRB has two branches. In the first branch, the
input is passed through a 3D convolution with a filter size of $h\times
w\times d$, where $h$, $w$, and $d$ are the height, width, and depth of the
filter, respectively. The aim is to learn information about the relationships
between channels and frequency simultaneously. In the second branch, the input
passes through two 3D convolutions. The first convolution extracts information
on the frequency bands using a filter size of $1\times 1\times d$. The second
convolution extracts the relationships between channels using a filter size of
$h\times w\times 1$. Finally, we compute the average of the outputs of both
branches to obtain information from every observation of both branches.

Figure 3: Multi-Dimensional Residual Block.

###  3.3 Combined time and frequency domain

Our proposed model aims to combine the time and frequency domains, as depicted
in Figure 1. To learn temporal information about the time domain, we utilize
Temporal Convolutional Networks. For the frequency domain, we have two
branches: PSD-64 ($B_{1}$) and PSD-128 ($B_{2}$), corresponding to two 3D
blocks extracted from different lengths of FFT, computed by Welch’s method.
Each branch has a different depth, $d^{B_{1}}_{in}$ and $d^{B_{2}}_{in}$,
respectively. We have PSD-64 ($B_{1}$) and PSD-128 ($B_{2}$) branches with the
lengths of FFT 64 and 128, respectively. For each branch, the 3D block input
passes 2 MDRBs, which have a different depth length of the MDRB, $d^{i}_{1}$,
$d^{i}_{2}$, respectively, where $i$ is the PSD branch,
$i\in\\{B_{1},B_{2}\\}$. The value of MDRB depth
(MDRB-$d^{i}_{j},j\in\\{1,2\\}$) depends on the depth of the input block
($d^{i}_{in}$). The MDRB-$d^{i}_{1}$ has a depth $d^{i}_{1}$ which is
calculated by getting the ceiling function of dividing by 2 as Equation 3,
$d^{i}_{2}$ is calculated as Equation 5.

$\begin{split}d^{i}_{1}&=\lceil\frac{d^{i}_{in}}{2}\rceil\end{split}$ (3)

$\begin{split}d^{i}_{out}&=\lfloor\frac{d^{i}_{in}+2\times pad-
dil\times(d^{i}_{1}-1)-1}{str}+1\rfloor\end{split}$ (4)

$\begin{split}d^{i}_{2}&=d^{i}_{in}-d^{i}_{out}\end{split}$ (5)

where $pad$ is padding ($pad=0$), $dil$ is dilation ($dil=1$), and $str$ is
convolution stride ($str=1$).

The Global Average Pooling was applied to the MDRB output for down sampling,
and then the fully connected layer was attached to obtain the prediction. The
mean of the predictions (3 branches) returns the prediction for the
classification and estimation levels of MWL.

##  4 EXPERIMENTS

###  4.1 Dataset

The STEW (Simultaneous Task EEG Workload) EEG database8 includes 48
participants. Each participant needs to participate in two tasks, the ’No’
task and the SIMKAP (Simultaneous capacity) task. For the ’No’ task,
participants do not perform any task and require a comfortable position with
their eyes open. For the second task, participants need to solve the SIMKAP
task, a test about a commercial psychological created by Schuhfried GmbH to
evaluate a person’s multitasking ability and stress. Both tasks were recorded
for 3 minutes for the EEG signal. The first and last 15 seconds have been
removed to eliminate the effects between tasks. A rating scale of 1-9 was used
to assess MWL levels for each participant.

The STEW dataset contains EEG data collected from the Emotiv headset. The
device established 128Hz for sampling frequency, 16-bit A/D resolution, and
fourteen channels (AF3, F7, F3, FC5, T7, P7, O1, O2, P8, T8, FC6, F4, F8, AF4)
by 10-20 international system.

###  4.2 Mental workload estimation

We conducted experiments for classification and prediction of continuous
levels. The classification task splits the rating scale into 3 classes of low
(lo) workload (1-3 rating), middle (mi) workload (4-6 rating), and high (hi)
workload (7-9 rating). The continuous level prediction task scales the MWL
levels into the range [0-1] based on the rating scale from 1 to 9. The window
length is 512 and the shift 128 for fourteen channels, which is also the
parameter (window and shift size) that the STEW data used for their
experiments. The dataset comprises forty-eight participants but ignores three
because there are not available ratings for these participants. Therefore, 36
participants (80%) were used for training and 9 (20%) were used for testing.

###  4.3 Experimental setup

We used the TensorFlow framework to train and test our method. Our proposed
method is trained in $150$ epochs and uses early stopping to obtain the best
model. The Adam optimizer was employed with a learning rate of $0.001$ for
classification and $0.0001$ for estimation of continuous levels. The TCNs17
applied with the kernel size is $2$, the number of filters used for
convolution layers is $128$, $2$ stacks of residual block were applied, and
the dilation list is $[1,2,4,8]$. The classification of three classes is a
multi-label classification, so the Softmax function was used. The Sigmoid
function was also utilized to get the output estimation for continuous levels
of MWL.

In this study, categorical cross-entropy loss is used for classification. The
loss function is calculated as follows:

$\begin{split}CE&=-\sum_{i=1}^{N}y_{i}.log(\hat{y}_{i})\end{split}$ (6)

And we used MSE (mean squared error) as a loss function for continuous levels
estimation.

$\begin{split}MSE=\frac{1}{N}\sum_{i=1}^{N}(y_{i}-\hat{y}_{i})^{2}\end{split}$
(7)

where $N$ is the number of samples, $y_{i}$ is the label of $i^{th}$ and
$\hat{y}_{i}$ is the value prediction of $i^{th}$.

We used the Concordance Correlation Coefficient $\mathcal{CCC}$ function to
evaluate for the estimation task. The $\mathcal{CCC}$ function is defined as:

$\begin{split}\mathcal{CCC}&=\frac{2\rho\sigma_{\hat{Y}}\sigma_{Y}}{\sigma_{\hat{Y}}^{2}+\sigma_{Y}^{2}+(\mu_{\hat{Y}}-\mu_{Y})^{2}}\end{split}$
(8)

where $\mu_{Y}$ is the mean of the label $Y$, $\mu_{\hat{Y}}$ is the mean of
the prediction $\hat{Y}$, $\sigma_{\hat{Y}}$ and $\sigma_{Y}$ are the
corresponding standard deviations, $\rho$ is the Pearson correlation
coefficient between $\hat{Y}$and $Y$.

###  4.4 Results

Table 1 presents the results of our proposed method. We have split the dataset
into an 8:2 ratio, following the methodology of Lim et al.8, to compare our
results with their best single trial. The comparison shows that our proposed
method outperforms Lim et al.8 with an accuracy of 74.98% compared to 69.0%.
We conducted separate and combined branch experiments, where the bold font
defines the best performance. We also conducted seven experiments for each
task that corresponds to the combination of 3 branches (Time, $B_{1}$, and
$B_{2}$). It is noteworthy that combining of the branches gives better results
than applying an individual branch. For the classification task, we obtained
the best results $74.98\%$ for combining time and multi-frequency ($B_{1}$ and
$B_{2}$) domain, which is better than the previous work8 by extracting
features of the delta, theta, alpha, and beta bands, $69.00\%$ ($28$ features
from PSD) and $69.20\%$ ($56$ features from PSD) obtained, respectively.
Furthermore, by combining the branch $B_{1}$ and branch $B_{2}$ without the
time branch, we obtained 70.79%, is better than Lim et al.8, 69.00%
respectively. It can be seen that the information from the frequency domain is
helpful for MWL classification by combination.

We used the Concordance Correlation Coefficient (CCC) metric to evaluate
continuous estimation in our study. It should be noted that our use of CCC as
a metric is relatively rare. Therefore, we recognize the need for further
clarification and explanation of the results to enhance the reader’s
comprehension. CCC is a statistical measure widely employed in machine
learning to evaluate the agreement between two measurement methods or
observers for continuous-level estimation tasks, such as stress or
valence/arousal estimation. Our proposed method achieved a CCC value of 0.629
for the continuous level estimation task. It is important to consider the
specific context and application when interpreting the CCC values as high or
low. A CCC value between 0.5 and 0.7 generally suggests moderate agreement,
while a value above 0.7 indicates strong agreement. Therefore, the CCC value
of 0.629 obtained in our study can be interpreted as demonstrating moderate
agreement.

Table 1: MWL estimation: The results compare with previous work by accuracy and $CCC$ where $Acc_{3}$ is the classification for 3 classes. | Feature | $Acc_{3}(\%)$ | $\mathcal{CCC}$
---|---|---|---
Lim et al.8 | 28 features from PSD | 69.00 | -
| 56 features from PSD | 69.20 | -
Ours | Time | 56.92 | 0.341
| $B_{1}$ | 51.10 | 0.524
| $B_{2}$ | 62.89 | 0.510
| Time + $B_{1}$ | 34.32 | 0.609
| Time + $B_{2}$ | 36.28 | 0.477
|  $B_{1}$ \+ $B_{2}$ | 70.79 | 0.525
| Time + $B_{1}$ \+ $B_{2}$ | 74.98 | 0.629
Table 2: The results for test set with change of shift on train set. Shift | 128 | 256 | 384 | 512
---|---|---|---|---
| $\mathcal{CCC}$ | $Acc(\%)$ | $\mathcal{CCC}$ | $Acc(\%)$ | $\mathcal{CCC}$ | $Acc(\%)$ | $\mathcal{CCC}$ | $Acc(\%)$
Time | 0.341 | 56.92 | 0.349 | 32.65 | 0.231 | 42.86 | 0.243 | 40.59
$B_{1}$ | 0.524 | 51.10 | 0.527 | 54.27 | 0.417 | 58.16 | 0.449 | 55.97
$B_{2}$ | 0.510 | 62.89 | 0.417 | 50.60 | 0.365 | 65.99 | 0.408 | 61.38
Time + $B_{1}$ | 0.609 | 34.32 | 0.53 | 27.78 | 0.302 | 27.70 | 0.310 | 27.44
Time + $B_{2}$ | 0.477 | 36.28 | 0.341 | 27.51 | 0.355 | 27.63 | 0.534 | 27.51
$B_{1}$ \+ $B_{2}$ | 0.525 | 70.79 | 0.303 | 51.25 | 0.285 | 50.30 | 0.352 | 62.43
Time + $B_{1}$ \+ $B_{2}$ | 0.629 | 74.98 | 0.514 | 49.66 | 0.446 | 65.61 | 0.315 | 47.51

In our experimentation, we also varied the shift size when processing the
training data. Specifically, we tried shift sizes of 128, 256, 384, and 512,
which correspond to window overlaps of 75%, 50%, 25%, and 0%, respectively.
The corresponding results are presented in Table 2. Our findings indicate that
a shift size of 128 results in better accuracy than the other shift sizes
(256, 384, and 512).

Table 3: Experiments of the MDRB if the branch $(1)$ or the branch $(2)$ is used. | Only branch $(1)$ | Only branch $(2)$ | Combine branch
---|---|---|---
Shift | $Acc_{3}(\%)$ | $\mathcal{CCC}$ | $Acc_{3}(\%)$ | $\mathcal{CCC}$ | $Acc_{3}(\%)$ | $\mathcal{CCC}$
128 | 48.56 | 0.024 | 56.16 | 0.323 | 74.98 | 0.629
256 | 43.84 | 0.194 | 53.17 | 0.110 | 49.66 | 0.514
384 | 57.56 | 0.320 | 46.03 | 0.090 | 65.61 | 0.446
512 | 50.98 | 0.094 | 59.18 | 0.003 | 47.51 | 0.315

We conducted experiments to compare the performance of using only one of the
two branches in the MDRB. The results are summarized in Table 3. We observed
that using both branches in combination yields the best result, with an
accuracy of 74.98% achieved with a shift value of 128. On the other hand,
using only branch $(1)$ or branch $(2)$ results in lower accuracy, with 57.56%
and 59.18%, respectively. For continuous labels, using an individual branch
only achieves half the accuracy of the combined branches (0.323 < 0.629),
indicating that combining information from both branches is effective. Branch
$(1)$ captures an overview of information on both channels and frequency,
while branch $(2)$ extracts detailed information in order of frequency and
then channels.

##  5 DISCUSSION

Our method utilizes both time and frequency domain information to extract
valuable features that contribute to accurate classification and regression
results. The experimental results demonstrate the effectiveness of combining
these two domains, achieving good classification accuracy (74.09%) and a
concordance correlation coefficient ($CCC$) of 0.629 for regression. Our
innovation lies in combining the time and frequency domains, which is an
improvement over the previous work that used separate domain experiments.
Additionally, our proposed method incorporates a Multi-dimensional Residual
Block (MDRB) to extract information on channel relationships and frequency
domain efficiently for the deep learning model. Furthermore, we conducted a
novel experiment on the regression problem, which yielded good results and has
the potential to contribute to the development of continuous-level mental
workload state activity prediction systems.

Although we have achieved good results in MWL estimation, we have not
thoroughly tested each channel or band. Our experiment involved using all EEG
channels (14 channels) without performing specific experiments on each channel
to evaluate their contributions to the MWL estimator process. In the frequency
domain, we extracted information from six channels (delta, theta, alpha,
sigma, beta, and gamma) and used them as a whole, but we did not perform
separate experiments for each channel to assess the importance of each band in
comparison to the effectiveness of combining them. These limitations need to
be addressed in future research.

##  6 CONCLUSION

This paper presents our experiments on the STEW dataset, where we propose a
method that addresses both the classification and the estimation of continuous
levels of MWL. Our proposed method can be applied to other EEG data problems
that involve labels, such as classification and estimation of continuous
levels. Additionally, we introduce a Multi-Dimensional Residual Block (MDRB),
which utilizes 3D residual blocks to learn channel relationships and
information bands of the frequency domain. To improve the performance of the
model, we combine the frequency and time domains. We use TCNs to learn
temporal information for the time domain and combine it with two branches of
the frequency domain. Our experimental results show that our proposed model
outperforms the baseline of the STEW dataset. We also demonstrate that our
multimodal fusion approach, based on multi-domain, leads to better results as
each domain contributes differently to the MWL estimation process. However, we
acknowledge the limitation of our model in classifying the middle stage of
mental workload, which we aim to improve in the future. We plan to conduct
experiments on other types of estimation, such as sleep stages, emotions,
depression, etc., for EEG data.

## ACKNOWLEDGMENTS

This research was supported by Basic Science Research Program through the
National Research Foundation of Korea (NRF) funded by the Ministry of
Education (NRF-2021R1I1A3A04036408) and also supported by Institute of
Information & communications Technology Planning & Evaluation (IITP) grant
funded by the Korea government (MSIT) (No.2021-0-02068, Artificial
Intelligence Innovation Hub).

## REFERENCES

  * [1] Gómez, L. C., Hervás, R., Gonzalez, I., and Villarreal, V. “Studying the generalisability of cognitive load measured with EEG,” Biomedical Signal Processing and Control., vol. 70, no. 103032, 2021. Doi: 10.1016/j.bspc.2021.103032.
  * [2] Wang, S., Gwizdka, J., and Chaovalitwongse, W. A. “Using wireless EEG signals to assess memory workload in the n-back task, ” IEEE Trans. on Human-Machine Syst.., vol. 46, no. 3, pp. 424-435, 2015. Doi:10.1109/THMS.2015.2476818.
  * [3] Chakladar, D. D., Dey, S., Roy, P. P., and Dogra, D. P, “EEG-based mental workload estimation using deep BLSTM-LSTM network and evolutionary algorithm,” Biomedical Signal Processing and Control., vol. 60, p. 101989, 2020. Doi:10.1016/j.bspc.2020.101989.
  * [4] Gwizdka, J., “Distribution of cognitive load in web search,” Journal of the American Society for Inf. Science and Tech.., vol. 61, no. 11, pp. 2167-2187, 2010. Doi:10.1002/asi.21385.
  * [5] Cegarra, J., and Chevalier, A., “The use of Tholos software for combining measures of mental workload: Toward theoretical and methodological improvements,” Behavior Research Methods, vol. 40, no. 4, pp. 988-1000, 2008. Doi:10.3758/BRM.40.4.988
  * [6] Taori, T. J., Gupta, S. S., Gajre, S. S., and Manthalkar, R. R., “Cognitive workload classification: Towards generalization through innovative pipeline interface using HMM,” Biomedical Signal Processing and Control., vol. 78, p. 104010, 2022. Doi:10.1016/j.bspc.2022.104010.
  * [7] Hogervorst, M. A., Brouwer, A. M., and Van Erp, J. B., “ Combining and comparing EEG, peripheral physiology and eye-related measures for the assessment of mental workload,”. Frontiers in neuro.., vol. 8, p. 322, 2014. Doi: 10.3389/fnins.2014.00322
  * [8] Lim, W. L., Sourina, O., and Wang, L. P., “STEW: Simultaneous task EEG workload data set,” IEEE Trans. on Neural Systems and Reha. Engi.,., vol. 26, no. 11, pp. 2106-2114, 2018. Doi: 10.1109/TNSRE.2018.2872924
  * [9] Ladekar, M. Y., Gupta, S. S., Joshi, Y. V., and Manthalkar, R. R., “EEG based visual cognitive workload analysis using multirate IIR filters,” Biomedical Signal Processing and Control., vol. 68, p. 102819, 2021. Doi: 10.1016/j.bspc.2021.102819.
  * [10] Zammouri, A., Moussa, A. A., and Mebrouk, Y, “Brain-computer interface for workload estimation: Assessment of mental efforts in learning processes,” Expert Systems with Applications., vol 112, pp. 138-147, 2018. Doi: 10.1016/j.eswa.2018.06.027.
  * [11] Yang, S., Yin, Z., Wang, Y., Zhang, W., Wang, Y., and Zhang, J., “Assessing cognitive mental workload via EEG signals and an ensemble deep learning classifier based on denoising autoencoders,” Computers in biology and medicine., vol. 109, pp. 159-170, 2019. Doi: 10.1016/j.compbiomed.2019.04.034.
  * [12] Qiao, W., and Bi, X., “Ternary-task convolutional bidirectional neural turing machine for assessment of EEG-based cognitive workload,” Biomedical Signal Processing and Control., vol. 57, p. 101745, 2020. Doi: 10.1016/j.bspc.2019.101745.
  * [13] Kwak, Y., Kong, K., Song, W. J., Min, B. K., and Kim, S. E., “Multilevel feature fusion with 3d convolutional neural network for eeg-based workload estimation,” IEEE access., vol. 8, pp. 16009-16021, 2020. Doi: 10.1109/ACCESS.2020.2966834.
  * [14] Zhao, X., Zhang, H., Zhu, G., You, F., Kuang, S., and Sun, L., “A multi-branch 3D convolutional neural network for EEG-based motor imagery classification,” IEEE trans. on neural systems and reha. engi., vol. 27, no. 10, pp. 2164-2177, 2019. Doi: 10.1109/TNSRE.2019.2938295.
  * [15] Liu, T., and Yang, D., “A three-branch 3D convolutional neural network for EEG-based different hand movement stages classification,” Scien. Reports., vol. 11, no. 1, pp. 1-13, 2021. Doi: 10.1038/s41598-021-89414-x.
  * [16] Jia, Z., Lin, Y., Cai, X., Chen, H., Gou, H., and Wang, J., “Sst-emotionnet: Spatial-spectral-temporal based attention 3d dense network for eeg emotion recognition,” in Proc. of the 28th ACM int. conf. on multimedia, 2020, pp. 2909-2917.
  * [17] Philippe Remy. “Temporal convolutional networks for keras”., [Online]. Available: https://github.com/philipperemy/keras-tcn, 2020.