# EEG based stress analysis using rhythm specific spectral feature for video
game play

Shidhartho Roy  ID<EMAIL_ADDRESS>Monira Islam<EMAIL_ADDRESS>Md. Salah Uddin Yusuf<EMAIL_ADDRESS>Nushrat Jahan
<EMAIL_ADDRESS>Department of Electrical and Electronic Engineering,
Khulna University of Engineering & Technology, Khulna-9203, Bangladesh

###### Abstract

### Background and Objective

For the emerging significance of mental stress, various research directives
have been established over time to better understand the causes of stress and
how to deal with it. In recent years, the rise of video gameplay is
unprecedented, further triggered by the lockdown imposed due to the COVID-19
pandemic. Several researchers and organizations have contributed to the
practical analysis of the impacts of such extended periods of gameplay, which
lacks coordinated studies to underline the outcomes and reflect those in
future game designing and public awareness about video gameplay.
Investigations have mostly focused on the “gameplay stress” based on physical
syndromes only. Some studies have analyzed the effects of video gameplay with
EEG, MRI, etc., without concentrating on the relaxation procedure after video
gameplay.

### Methods

This paper presents an end-to-end stress analysis for video gaming stimuli
using EEG. The PSD value of the Alpha and Beta bands is computed to calculate
the Beta-to-Alpha ratio (BAR). In this article, BAR is used to denote mental
stress. Subjects are chosen based on various factors such as gender, gameplay
experience, age, and BMI. EEG is recorded using Scan SynAmps2 Express
equipment. There are three types of video gameplay: strategic, puzzle, and
combinational. Relaxation is accomplished in this study by the use of music of
various pitches. Two types of regression analysis is done to mathematically
model stress and relaxation curve. Brain topography is rendered to indicate
the stressed and relaxed region of the brain.

### Results

In the relaxed state, the subjects have BAR $0.701$, which is considered the
baseline value. Non-gamer subjects have an average BAR of $2.403$ for 1 hour
of strategic video gameplay, whereas gamers have $2.218$ BAR concurrently.
After $12$ minutes of listening to low-pitch music, gamers achieved $0.709$
BAR, which is nearly the baseline value. In comparison to Quartic regression,
the $4PL$ symmetrical sigmoid function performs regression analysis with fewer
parameters and computational power.

### Conclusion

Non-gamers experience more stress than gamers, whereas strategic game creates
more stress on human brain. During gameplay, the beta band in the frontal
region is mostly activated. For relaxation, low pitch music is the most useful
media. Residual stress is evident in the frontal lobe when the subjects
experienced high pitch music. Quartic regression performs regression analysis
follow the relaxation curve more accurately compared to 4PL symmetrical
sigmoid function.

###### keywords:

Beta-Alpha ratio , EEG , Stress - relaxation modeling , Topography , Video
gameplay.

††journal: arXiv

##  1 Introduction

In the modern society, stress is a significant matter of concern. According to
WHO, loss of productivity and mental health disorders have globally cost a
significant capital [19]. When the human body is exposed to negative stimuli,
it responds with stress. Acute stress is the result of short-term stressful
events, which are typically indicated by transient physiological changes.
Acute stress lasting for a long time might turn into episodic stress. Chronic
stress, anxiety, and clinical depression can result from prolonged exposure to
stress factors or traumatic situations [21]. This paper mainly centers around
acute stress. To study acute stress, video gameplay has been employed as the
stimulus.

Video games (VG) are becoming increasingly popular among young people in
current culture, and VG are growing in popularity not just as a research tool
but also as an area of study. In addition, a plenty of research is focused on
the neurological and behavioral consequences of VG because of the expanding
popularity and universal applications, offering a comprehensive brain
correlation in recent decades. Moreover, the rising diversity of digital
technology, such as smartphones and tablets, has brought entertainment
software or VG into the mainstream. Thus, a significant portion of society
(more than 30% in tablets and 70% in smartphones) is exposed to these
technologies and may now, in some way, be termed as casual gamers [8].

Because of its high temporal resolution, non-invasiveness, and low cost,
electroencephalography (EEG) is frequently utilized in research, including
neural engineering, neuroscience, and biomedical engineering (e.g., brain-
computer interfaces, BCI [10], sleep analysis [18], stress analysis [32], and
seizure detection [7]). EEG has provided insight into the brain’s functioning
in many cognitive processes and is regarded as the building block of
functional signaling in the brain. Oscillations of various frequencies are
linked to various cognitive functions. Alpha oscillations, for example, are
linked to relaxation, whereas beta rhythms are linked to tension.

When it comes to relaxation, listening to music is unquestionably a top
contender. Music has a powerful impact on both the mind and the body. Faster
music can help us to concentrate and feel more alert. Upbeat music can boost
one’s mood and make one feel more optimistic about life. A slower tempo can
relax the mind and muscles, making a person feel soothed while letting go of
the day’s stress. This study uses three different types of music (low, medium,
and high pitch) to study relaxation.

In the mainstream media, both good and bad health claims associated with VG
are common. These comments are primarily unconfirmed and sensational, based on
’expert’ opinions, but without proof. Therefore, it is becoming increasingly
important to know the actual impacts of long-term exposure to video gaming in
a deeper technical context. The effect can be positive in the form of
cognitive, motivational, emotional, and social benefits [33] or can be
negative in the form of exposure to graphic violence, addiction, contribution
to obesity, and cardiometabolism [13]. For a long time, video gaming has been
on the increase. However, the growth in gaming was unprecedented during the
COVID pandemic. According to a gamesindustry.biz study, VG sales have surged
by $63$ percent in $50$ major areas since pre-pandemic time [34]. The home
quarantine compelled everyone to explore the world of gaming, resulting in an
$80\%$ increase in the number of new games downloaded. Under the pandemic, the
social infrastructure for individuals, particularly those in quarantine, has
evolved into various platforms [15]. The rise of gameplay in recent years is
evident from Fig. 1. In early 2020, when lockdown started, the duration of
video gameplay in hours has almost doubled from before.

Figure 1: The surge of gameplay hours watched since 2013. At the beginning of
the pandemic, the gameplay hours watched have almost doubled [29].

Some researchers have previously focused on the influence of VG, depending on
participants’ exposure to that particular game or studies, on the content
features of VG [14]. Some research specializes in the genres of VG with
prominent graphic violence (action, war, shooting, killing, etc.) On this
basis, a reasonable quantity of scientific documentation on the research of
violence in VG and violence acclimatization is used [16]. For decades, social,
political, and scientific considerations have been given to the link between
exposure to media violence and violent behavior. Extensive empirical data
suggest that exposure to media violence impacts aggressive behavior, abuse,
violent cognition, physiologic excitement, and teenage animosity [30]. Another
study has shown that playing some VG, particularly ”action” games, often
improves the cognitive function [16]. It has also given thought on how the
business may be better regulated to promote games for cognitive advancement.
Considerable research has been conducted on the stressful effects of video
gaming on the human brain [12]. Using EEG as a tool for study, Soysal et al.
distinguished the function of the brain for different games and the stress
level caused by them. It can differentiate between a relaxed and a stressed
individual [26]. Saputra et al. researched a crucial aspect of video gaming
and found that, in addition to the entertainment value of video gaming,
players get stressed when they experience frustration as a result of virtual
failure [24].

Figure 2: Paper organization structure. Table 1: EEG based stress analysis in the recent literature. Bio-medical signal type |  Notable Methodology |  Stimuli |  Subject |  Contribution |  Ref
---|---|---|---|---|---
EEG |  Stress-related hormone labeling, ML-based classification |  construction work |  Workers at Construction Sites |  Developed a procedure to recognize construction worker’s stress by applying supervised learning algorithms |  [11]
EEG |  Multiclass SVM with error correcting output code (ECOC) |  arithmetic task |  20 to 24 years old male right-handed adults |  The classification of stress with a precision of 94.79% shows the right prefrontal cortex dominance over mental stress. |  [2]
EEG |  Frequency analysis from average power |  Multiplayer Online Battle Arena Video Game |  18-25 years old 10 participants |  First time experienced participant have significant difference for alpha/beta ratio than experienced gamer. |  [24]
EEG |  Unity3D game engine, EMOTIV kit |  Video game |  ADHD patients |  EEG-based game that targets ADHD symptomatic individuals to augment their focus. |  [4]
EEG and fNIRS |  SVM classifier, decision fusion of fNIRS and EEG signals |  Montreal Imaging Stress Task (MIST) |  22 male, right-handed adults (age 22 ± 2 years) |  suggesting fNIRS+EEG as an efficient tool of stress analysis |  [3]
EEG, PPG and GSR |  Keystroke Analysis |  Video game |  22 subjects (9 female) having mean age 29.3 years |  Analysis of keystroke data gathered in the tests to suggest a new method of stress measurement |  [9]
EEG |  power spectral density (PSD) |  to prepare a talk on an unknown topic. |  28 participants |  Finding an appropriate EEG recording phase for classifying mental stress into many groups. |  [6]
EEG |  PSD, Wavelet transformation, topographic analysis |  Music of different genres. |  48 individuals were selected from 11 to 50 years of age |  Short length of music offers alleviation from tension, however the stress is growing when individuals have been exposed to music for a long period. |  [23]
EEG |  KNN, LDA, MLP |  TempleRun video game |  20 healthy subjects |  A game player’s competence level is classified on a consumer gaming device |  [5]
EEG |  CBA Ratio and the Statistical Analysis |  Music |  nine participants |  In a stressful scenario, a drop of brain alpha wave frequencies and a concurrent increase in the participant blood pressure and pulse is shown |  [20]
EEG |  Genetic Algorithm-Based Feature Selection |  40 music videos |  Thirty-two healthy participants |  classification performance is best when the proposed GA-based method was used. |  [25]
EEG, ECG |  Analog amplification in the front-end |  Colors and their names |  Seven healthy young male subjects |  Long-term daily monitoring is needed to evaluate chronic stress |  [1]
EEG |  PSD, beta/alpha ratio |  Language based sustained mental task |  52 subjects all aged from 20 to 23 |  Mother language creates the least stress. |  [32]
EEG |  Wavelet, PSD |  Language based music |  30 subjects all aged from 20 to 23 |  Bass frequency reveals more relaxation. |  [22]

Despite these researches, relatively few works are based on relaxing the
stress caused by video gaming. The motivation behind this study was to
resemble real-world scenarios as EEG stimuli and analyze video gameplay with
EEG. For this purpose, we conducted an open survey on the nature of video
gaming on various people. Among 214 participants, $41.35\%$ indicated they do
not do anything after gaming, while $38.67\%$ stated they like some music
after a prolonged gaming period. Because these two populations outnumber
others, we pooled them in our study. To select the video gaming period, we
asked our participants how long they play at a time. On average, they reported
that they play 55.6 minutes of VG. Some gamers also documented that they play
3 to 4 hours at a stretch, and this has been considered as outlier in this
study. The relaxation period of 12 minutes is also based on the survey.
Subjects were asked to choose how long they want to spend time after video
gameplay before starting another cognitive intensive work. The average
response time was 11.08 minutes.

Furthermore, a few studies have been discovered to develop a mathematical
model based on stress development from gaming and the relaxation via music of
various pitches. Our goal in this study is mental state analysis during
gameplay of various game genres, brain mapping of these states, mental state
analysis after playtime, relaxing via multiple forms of music, and
corresponding regression analysis. This is, to the best of our knowledge, the
first attempt to aggregate all of them 111After acceptance, code and dataset
will be made open access for further research in the link:
https://github.com/royShidhartho/Stress-analysis-video-gameplay. The rest of
the paper is organized as shown in Fig. 2.

##  2 Literature Review

From the very beginning of the research to examine the response of the brain
due to video gameplay, the effects of video gameplay using EEG has been
emphasized. Previously some of the studies have been focused on other
techniques rather than EEG. But EEG had good accuracy in examining brain
activity for video game playing. From 2017 to 2021, many developments took
place in this era, such as using fMRI, MRI, PPG, GSR, fNIRS, ECG, etc. But
accuracy was not good compared to EEG. Though ECG is the technique that is
used the second-highest for this purpose, it is actually a low-quality method.
Furthermore, it is difficult to eliminate noise with conventional filtering
methods for fNIRS. From the table 1 below, it is clear that the EEG technique
is best suited for examining the stress analysis in different time duration.

Figure 3: Block diagram of this experiment.

Another assessment observed from the literature review is that people aged 19
to 24 years are mostly impacted by video gaming. This circumstance has both
positive and negative impacts on society. Though video gameplay can improve
problem-solving skills, it is time-consuming. We had also observed that old
adults, as well as patients, are involved in playing gameplay. The peak age
ranged from 19 to 24 years and with increasing age, the observation and the
interest in video gaming becomes lower consequently.

##  3 Methodology

To achieve the objective of this experiment, several methods are used.
Subjects were first prepared after settling on a data acquisition protocol
using an EEG recording device EEG signal and pre-processing by both onboard
and digital devices. The power ratio was computed after extracting features
from the clean EEG signal. This experiment made use of a variety of games and
music. Following the experiment, brain mapping was created using the EEG
topography approach, and mathematical modeling was completed using regression
analysis. Fig. 3 illustrates the block diagram of this experiment.

###  3.1 Subject Preparation

Four focus groups are organized for this study ($FG_{1}$ – $FG_{4}$) and from
four to $17$ each ($N=45$). Participants were selected using an open survey
procedure based on their experience in gaming and their health condition. All
subjects are between the ages of 20 and 24 years (22.29 on average) and
university students. These are organized into four focus groups: two groups
are called ’Gamers’ who on average play games 10 hours a week or more. The
other two groups play games for less than 2 hour per week or do not play at
all. Both men and women participated in the gamer and non-gamer groups. The
proportion of women in non-gamers is 45.83%, compared to 19.04% in gamers.
There is also a lack of sleep disorders or other serious disorders in the
selected subjects. Furthermore, the subjects were unaware of the study’s
goals. In figure 4, the subject distribution is depicted. Another criteria for
selecting the subjects was to select them from a wide BMI spectrum. Eight of
the subjects in this study are obese (BMI 30.0 and above), 18 of them are
overweight (BMI 25.0-29.9), 12 of them are normal (BMI 18.5-24.9), and the
rest of them, 7, are underweight (below BMI 18.5).

Figure 4: Subject distribution of this study. Blue shades denote the male
subjects, and pink shade embodies the female subjects.

###  3.2 Stimuli preparation

VG is used as stimuli to create stress in this study. Based on game design,
all VG were divided into three groups. On one end of the scale, puzzle games
demand quick thinking, whereas strategy games need quick response and extended
pondering.

  1. 1.

Puzzle: Video puzzle games aim to solve riddles. Logic, pattern recognition,
sequence resolution, and word completion are only a few of the problem-solving
capabilities that these puzzles might test for. The player may have endless
time or endeavors to solve a puzzle or a time restriction, or it may make it
more challenging to do fundamental problems by having to do it in real-time.

Gameplay format: These games usually include rules in which players use a
grid, network, or other interactive areas to control game pieces. In order to
earn a triumph, the players must unravel clues so that they may then move to
the next level. Usually, completing every riddle will lead to a more
complicated task.

  2. 2.

Strategy: A video strategy game promotes strategic thinking and preparation to
win. It focuses on strategic, tactical, and sometimes logistical questions. In
this area, there are additional action games. Many games also feature economic
concerns and exploration. It is classified into four subtypes depending on
whether the game is in turn or in real-time, whether strategic or tactical—for
example, Rome: Total War, Dota 2, etc.

Gameplay format: In order to reduce opposing troops, a player must perform a
sequence of operations against one or more opponents. Victory is achieved
through a greater plan with less luck. The user receives a divine vision of
the game world, with most strategy VG offering them indirect authority over
the game units. Consequently, most strategy games involve a mixture of
tactical and strategic considerations and contain various fighting components.

  3. 3.

Combinational:  It accumulates both the puzzle and strategy type games. Such
as Grand Theft Auto, San andreas etc.

Gameplay format: A combination game implies a timed sequence of plays, each of
which leaves the opponent incapable or near impossible to block or otherwise
prevent the next succession.

Figure 5: Game classification considered for this study.

###  3.3 EEG data Acquisition

The Scan SynAmps2 Express equipment (Compumedics Ltd., VIC, Australia) was
used to acquire EEG signals. A wired EEG cap containing 32 Ag/AgCl electrodes,
comprising 30 EEG electrodes and two reference electrodes, was utilized to
acquire EEG data (opposite lateral mastoids). An adapted international 10–20
system was used to put the EEG electrodes. All electrodes and the skin had a
contact impedance of less than $5\ k\Omega$ . The EEG recordings were
amplified and digitized at 500 Hz using the Scan SynAmps2 Express device
(resolution: 16 bits). Scan 4.5 from Neuroscan is the data collecting tool. On
the computer, the raw data was stored as .cnt files. Figure 6 manifests the
the experiment electrode positions.

Figure 6: 10-20 system of electrode placement. Blue electrode denotes the
electrode positions used in this experiment.

The acquisition of data is divided into two streams – one during video
gameplay and one following gameplay. In the gameplay stage, the relaxed state
value was captured initially after 10 minutes of relaxation in experimental
set-up data. The data was collected during the games in an interval of 15
minutes. Data were recorded after the gameplay within a period of 3 minutes.
The workflow of data gathering for the experiment is displayed in the Fig. 7 .

Figure 7: Data acquisition protocol. The orange color palette denotes the
during gameplay phase, and the green color palette represents after gameplay
phase.

###  3.4 Feature extraction

The FFT is a sophisticated reversible mapping technique used to transfer a
signal from a time domain to a frequency domain. In the periodogram calculated
by the FFT technique, the PSD properties of artifact-free EEG data may be
evaluated. However, there is elevated volatility and low statistical precision
in the periodogram. Welch developed a modified periodogram averaging approach,
resulting in a reduction in PSD variance [31].

This process involves splitting time series data into segments, overlapping
them, and then conducting FFT, calculating and averaging the amplitude of each
segment. In this study, 72 minutes EEG signals were selected as samples, and
the window length of each 10 seconds was divided into four equal segments. The
time sequence was separated into $50\%$ for each sample of 10 seconds, and
four segments were overlapped. For each segment, FFT was used to calculate and
average the capacity of the overlapping FFT coefficients in all segments. This
calculation is done for all frequency ranges until, ultimately, the Alpha and
Beta PSD ($\mu V^{2}/Hz$) values have been generated. The PSD is manifested
according to Welch in the following equation:

$P_{d}(f)=\frac{1}{MU}\sum\limits_{n=0}^{M-1}\left|x_{d}(n)w(n)e^{-j2\pi
fn}\right|^{2},\ $ (1)

for $d\ =$ $1,\ 2,\ 3,...,\ L$ signal intervals, $x_{d}(n)$ is the sequence,
where M is the interval length. $w(n)$ represents the window data, whereas $U$
is the normalization factor for window power. $U$ is defined as [27],

$U=\frac{1}{M}\sum\limits_{n=0}^{M-1}\left|w(n)\right|^{2}.\ $ (2)

The Welch spectrum of power can be shown as the mean over these modified
periodograms:

$P_{Welch}(f)=\frac{1}{L}\sum\limits_{i=0}^{L-1}P_{d}(f),\ $ (3)

where $P_{Welch}$ is each interval’s periodogram of the EEG signal.

Rhythm power ratio The power ratio was calculated by dividing the absolute
power of one frequency band by the absolute power of another frequency band,
using the averaged PSD of each frequency band. Delta activity revealed the
deep resting condition of a person. It was not presumed to display substantial
activity in the face of stress, which is why this rhythm has not been studied.
By dividing the PSD of beta by the PSD of alpha, the beta/alpha rhythm power
ratio was determined. After that, it was evaluated with the ratio value of the
difference between the baseline and the stress session. The processes can also
be used to determine the Theta/Beta ratio and differentiate between relaxed
and stressed states.The path of data collecting, signal pre-processing,
feature extraction, and power ratio computation is depicted in Figure 3.

####  3.4.1 EEG topography

EEG topography is a sort of neuroimage including the geometrical design of
uniformly spaced sites of multiple EEG electrodes on the head. A certain
computer software maps the activity on the color screen or printer in many
tones of colors by sorting the number of actions (for example, blue might
represent low EEG amplitude, while yellow and red might represent larger
amplitudes). From EEG topography, we are exploring the residual stress on the
human brain. Residual stress is termed as the remaining stress after the
designed relaxation period for this study.

In mathematical interpolation techniques (calculation of intermediate values
on the values of their neighbors), the spatial locations between electrodes
are established to provide a smooth color gradation. This technique delivers
the position of rhythm, amplitude changes regarding the skull surface, and a
considerably more precise and realistic image.

Comparing topographies for pattern similarity in various settings can
demonstrate various thinking processes and so directly test cognitive
theories. Topographic analysis of variance (TANOVA) is a statistical test that
examines similarity topographies. In this topographic similarity measure,
called ”angle measure,” the topographical pattern similarity is measured by a
high dimensional angle between two topographies.

Multivariate topography patterns across all sensors are displayed as $\vec{A}$
and $\vec{B}$ vectors with a size equal to that of the two situations. The
following equation (Eq. 4) calculates an angle cosine of $\theta$ to quantify
the topographical similitude between the two scenarios.

$\cos\theta=\frac{\vec{A}.\vec{B}}{\left|\vec{A}\right|.\left|\vec{B}\right|},$
(4)

The cosine value is an index that shows the spatial correlation between two
circumstances in which the +1 value indicates the same patterns as the -1
value. In addition, because the magnitude of the response in the two
circumstances normalizes this index, it benefits that the magnitude of the
responses does not impact it.

####  3.4.2 Regression analysis and metrics

Two types of regression analysis is employed in this paper.

  * 1.

4PL Symmetrical sigmoid function This results in a more advanced model that is
better suited to many biological processes. The four-parameter logistic
regression is known as this model (4PL). It is very beneficial for dose-
response or receptor binding tests or comparative tests. It contains four
parameters to be evaluated to ’fit the curve,’ as the name suggests. The model
is designed for data making an S-shaped curve. The model’s equation is:

$y=d+\frac{a-d}{1+(\frac{x}{c})^{b}},$ (5)

where , a = the least value that can be achieved (i.e. what happens at relaxed
state.), d = the highest value that can be achieved (i.e. what happens after 1
hour of gameplay), c = inflection point (i.e. the point on the S shaped curve
halfway between a and d), and b = Hill’s slope of the curve (the steepness of
the curve at point c).

Fig. 8 denotes the structure of the 4PL Symmetrical sigmoid function.

Figure 8: Symmetrical sigmoid function.

  * 2.

Quartic Regression The quartic regression adapts to a data set with a quartic
function (a polynomial function with a degree of 4). The form of quartic
function is:

$y=a+bx+cx^{2}+dx^{3}+ex^{4}+fx^{5}$ (6)

The quartic function takes a range of forms with various inflection points
(when a function changes form) and zero to many roots (places where the graph
crosses the axis). If the f value exceeds 0, three distinct fundamental forms
can be present.

Metrics Two types of regression metrics are employed here to fit the stress
and relaxation curve for this experiment.

  * 1.

Coefficient of determination

A statistical measure of how close the data are to the correct regression line
is R-squared. It is also known as the determination coefficient or the
multiple regression coefficients. A linear model may explain a certain
proportion of the variation in a response variable:

$R^{2}=1-\frac{sum\ of\ the\ squares\ of\ residuals}{total\ sum\ of\ squares}$
(7)

A high $R^{2}$ number, in general, suggests that the model is a good fit for
the data, though fit interpretations vary depending on the context of study.
For example, a $R^{2}$ of 0.44 shows that 44 percent of the variation in the
outcome can be explained simply by predicting the outcome using the covariates
in the model. In certain subjects, such as the social sciences, that
proportion may be difficult to forecast; in others, such as the physical
sciences, one would expect $R^{2}$ to be considerably closer to 100%.

  * 2.

Akaike information criterion (AIC)

The AIC is a prediction error estimator and a comparable quality for a certain
number of data models. AIC determines the quality of each model concerning
each of the other models based on the collection of models for the data [17].
AIC offers a way to pick the model. The information theory is the basis of
AIC. When used to describe the process that produced the data, a statistic
model is virtually never accurate; hence, information is lost through the
usage of the model to describe the process. The AIC calculates the amount of
information lost by a particular model: the less information lost, the greater
the model’s quality [28].

$AIC=2k-2ln(\hat{L}),$ (8)

where $\hat{L}$ is the maximum value of the likelihood function for the model
and k is the number of estimated parameters in the model. AIC considers the
trade-off between model goodness of fit and model simplicity when assessing
the amount of information lost by a model. To put it another way, AIC
considers both the risks of overfitting and underfitting.

###  3.5 Relaxation analysis

Music is used as a relaxation media in this study. Pitch has been used to
categorize music in this research. Pitch is the characteristic that allows
people to perceive sounds as ”upper” or ”lower” in the sense of musical
melodies. It is a perceptual feature of sounds that allows them to be arranged
on a frequency-related scale[1]. Only sounds with a precise and stable
frequency that can be differentiated from noise can calculate pitch. Pitch is
a significant auditory attribute of musical tones, along with duration,
loudness, and timbre.

Pitch can be described as a frequency; however, it is a subjective
psychoacoustical aspect of sound rather than a completely objective physical
characteristic. Pitch has long been a critical topic in psychoacoustics, with
theories of sound encoding, processing, and perception in the auditory system
being created and tested. The frequency of vibration of the sound waves that
create them determines the pitch of sounds. A high pitch is defined as a
frequency of 880 hertz [Hz; cycles per second], while a low pitch is defined
as a frequency of 55 hertz [Hz; cycles per second]. The music is divided into
the following categories:

  * 1.

Low pitch music: below 200 Hz,

  * 2.

Medium pitch music: 200 to 600 Hz, and

  * 3.

High pitch music: above 600 Hz

To bring down the beta band activity in previous levels, subjects are divided
into 4 groups.

  * 1.

Group 1: No music was introduced to bring down the brain activity to normal.

  * 2.

Group 2: Low pitch music was introduced to bring down the brain activity to
normal.

  * 3.

Group 3: Medium pitch music was introduced to bring down the brain activity to
normal

  * 4.

Group 4: High pitch music was introduced to bring down the brain activity to
normal.

##  4 Experimental result analysis

The full experimental study is divided into three phases; stress level
observation during gaming, relaxation level determination after video gameplay
and mathematical modeling of brain stress and relaxation. After discussing all
the observations, the summary of the discussion’s are pointed out in
evaluation sub-section.

###  4.1 Stress analysis

Before going into the gameplay and data acquisition, the relaxed state alpha
and beta value are measured. On average, the subjects exhibit BAR as 0.701
(Alpha = 4.329, Beta = 3.034). This value from now on is considered as the
baseline value. When the gameplay starts, subjects have gone through 3
distinct set of VG; puzzle, combinational, and strategic. As the subjects are
broadly divided into 2 groups, gamer and non-gamer, each group’s data are
calculated in average. For data acquisition, 15 minutes interval is
considered.

Table 2: Change of mental states during gameplay, observed with BAR. Game Types |  Gamer type |  After 15 min |  Increase (%) |  After 30 min |  Increase (%) |  After 45 min |  Increase (%) |  After 1 Hour |  Increase (%)
---|---|---|---|---|---|---|---|---|---
Puzzle |  Gamer |  0.729 |  0.0384 |  0.874 |  0.1979 |  1.297 |  0.4595 |  1.541 |  0.5451
Non-Gamer |  0.862 |  0.1868 |  0.984 |  0.2876 |  1.542 |  0.5454 |  1.897 |  0.6305
Combi- national |  Gamer |  0.814 |  0.1388 |  0.917 |  0.2356 |  1.341 |  0.4773 |  1.797 |  0.6099
Non-Gamer |  1.084 |  0.3533 |  1.295 |  0.4587 |  1.742 |  0.5976 |  2.149 |  0.6738
Strategic |  Gamer |  1.173 |  0.4024 |  1.376 |  0.4906 |  1.862 |  0.6235 |  2.218 |  0.6839
Non-Gamer |  1.337 |  0.4757 |  1.426 |  0.5084 |  1.856 |  0.6223 |  2.403 |  0.7083

Overall, in all categories of games, Strategic games are more stressful to
subjects. By comparison of gamer and non-gamer, gamers are more likely to
adopt strategic games. In the case of puzzle-type games, non-gamer people are
more stressed than the gamer subject group. The difference is also more
significant here. As gamer group are more used to solve the puzzle patterns it
is less stressful to them. Same goes for strategic game also. But in the case
of strategic game the action / shooting scenes of games are almost have same
impact on both type of subjects. The difference is also been observed in the
topographic plot of brain activity. In the topographic plot, the blue region
indicates the dominance of alpha activity and the red region denotes the
dominance of beta activity. After only 15 minutes of gameplay, the frontal
lobe of non-gamers is indicating the dominance of beta band activity. This
frontal lobe is directly connected with concentration, planning, and motor
control. So, this topographic plot clearly indicates that non-gamers are more
impacted with puzzle gameplay.

Over the gameplay time, this trend is continued to grow. Also in the same
time, there is a medium hotspot of beta activity in the occipital lobe which
is responsible for vision. This hotspot indicates the pressure increase in
eyes for constant gameplay. And, also this region keeps growing over time.
Another interesting phenomenon happens after 45 minutes of gameplay. The
parietal lobe, responsible for body awareness, exhibits a mild hotspot
connecting the frontal and occipital lobe. All of these brain activities
clearly indicate that non-gamers show a greater beta activity over gamers. As
strategic type games create the maximum stress on subjects, so in case of
topographic representation only subject reaction on strategic game is depicted
in Fig. 9.

Figure 9: Topographic representation during gaming for strategic type
gameplay. The blue color palette denotes the dominance of alpha activity in
that region.

###  4.2 Music evoked relaxation analysis

####  4.2.1 Puzzle

In puzzle-type gameplay, the after-gameplay BAR was 1.541 for gamers and 1.897
for non-gamers. Among the four subject groups, the group exposed to low pitch
music for relaxation was more evident to bring down the beta band activity
faster than the other three groups. Within this group, the beta activity in
the frontal lobe and motor cortex was downed significantly faster than the
other three groups. Another interesting phenomenon happened in the group,
which is not exposed to no music. Their response to the brain relaxation
process was just below the medium pitch song-exposed group. It is evident from
the brain response that low pitch music relaxes the stressed brain faster than
high pitch music. Also, at the same time, high music relaxes the brain at a
much slower rate, and the residual beta-band activity is highest among all
three types of music. Another noticeable change in brain activity is the
creation of a mild hotspot in the temporal lobe. The change of mental state
for puzzle-type gameplay is shown in Table 3. The residual stress is evident
in the frontal lobe when the subjects experienced high pitch music.

Table 3: Change of mental states after gameplay (puzzle), observed with BAR. Music Types |  Gamer type |  Immediately After |  After 3 min |  After 6 min |  After 9 min |  After 12 min
---|---|---|---|---|---|---
Low pitch |  Gamer |  1.541 |  1.023 |  0.865 |  0.812 |  0.709
Non-Gamer |  1.897 |  1.137 |  0.912 |  0.856 |  0.716
Medium Pitch |  Gamer |  1.541 |  1.148 |  0.892 |  0.841 |  0.714
Non-Gamer |  1.897 |  1.239 |  1.026 |  0.913 |  0.768
High pitch |  Gamer |  1.541 |  1.267 |  0.914 |  0.862 |  0.736
Non-Gamer |  1.897 |  1.342 |  1.103 |  0.924 |  0.784
No music |  Gamer |  1.541 |  1.186 |  0.895 |  0.852 |  0.718
Non-Gamer |  1.897 |  1.289 |  1.078 |  0.917 |  0.796

####  4.2.2 Combinational

For combinational game types, the residual stress after 12 minutes of rest was
more than in puzzle-type gameplay. Table 4 documented the mental state change
for relaxation after combinational type gameplay. The relaxation curve after
combinational type gameplay is between the responses after puzzle and
strategic type gameplay. The mild hotspots of the frontal lobe are smaller
than strategic type gameplay (Fig. 10). Another notable observation is when
subjects were not experienced with any music. In this case, the response is
almost similar to the medium pitch music response.

Table 4: Change of mental states after gameplay (combinational), observed with BAR. Music Types |  Gamer type |  Immediately After |  After 3 min |  After 6 min |  After 9 min |  After 12 min
---|---|---|---|---|---|---
Low pitch |  Gamer |  1.797 |  1.434 |  0.968 |  0.873 |  0.713
Non-Gamer |  2.149 |  1.614 |  1.172 |  0.866 |  0.72
Medium Pitch |  Gamer |  1.797 |  1.349 |  0.98 |  0.851 |  0.718
Non-Gamer |  2.149 |  1.614 |  1.172 |  0.924 |  0.829
High pitch |  Gamer |  1.797 |  1.542 |  1.034 |  0.872 |  0.820
Non-Gamer |  2.149 |  1.824 |  1.197 |  0.935 |  0.805
No music |  Gamer |  1.797 |  1.479 |  0.971 |  0.807 |  0.724
Non-Gamer |  2.149 |  1.672 |  1.094 |  0.894 |  0.815

####  4.2.3 Strategic

For strategic type gameplay, low pitch music was helpful for both gamers and
non-gamers. Another interesting observation is that for non-gamers, high pitch
music and no music yields an almost similar response in the relaxation curve.
In Fig. 10 the phases of brain after 12 minutes for various cases are depicted
in topographic representation. A mild hotspot of beta activity in the frontal
lobe is present for non-gamers when they are experienced to medium pitch music
for relaxation. Nevertheless, the overall percentage of beta activity is much
higher in non-gamers who experienced high pitch music. When gamers have not
experienced any music, some mild activity zones are visible frontal and
occipital lobes. Overall, both for gamers and non-gamers, low pitch music
brings more relaxation than any other relaxation procedure (Table 5).

Figure 10: Topographic representation after gaming for strategic type gameplay. The red color palette denotes the dominance of beta activity in that region. Table 5: Change of mental states after gameplay (strategic), observed with BAR. Music Types |  Gamer type |  Immediately After |  After 3 min |  After 6 min |  After 9 min |  After 12 min
---|---|---|---|---|---|---
Low pitch |  Gamer |  2.218 |  1.451 |  1.145 |  0.814 |  0.717
Non-Gamer |  2.403 |  1.663 |  1.232 |  0.873 |  0.749
Medium Pitch |  Gamer |  2.218 |  1.71 |  1.205 |  0.917 |  0.723
Non-Gamer |  2.403 |  1.846 |  1.386 |  0.996 |  0.824
High pitch |  Gamer |  2.218 |  1.76 |  1.235 |  0.94 |  0.816
Non-Gamer |  2.403 |  1.999 |  1.49 |  1.008 |  0.827
No music |  Gamer |  2.218 |  1.75 |  1.296 |  0.951 |  0.741
Non-Gamer |  2.403 |  1.983 |  1.472 |  1.017 |  0.819
Figure 11: Brain relaxation after gameplay.

###  4.3 Regression based mathematical analysis

For the mathematical modelling of the entire experiment, two different phases
have been followed, such as:

  1. 1.

During gameplay: Tables 6 and 7 denote the parameters for 4PL symmetrical
sigmoid function and quartic regression, respectively. $R^{2}$ and AIC is used
to measure the goodness of the mathematical modeling for the respective
equations. In the later gameplay period, stress increase rate was much higher
than the earlier time.

  2. 2.

After gameplay: In Tables 8 and 9, the mathematical modeling for brain
relaxation is documented. In the first three minutes the rate of relaxation is
higher than the later segments.

In Fig. 12, the comparison between the two mathematical models is depicted. In
almost all cases, the quartic regression dominates over the 4PL symmetrical
sigmoid function. But this achievement has come with a price for quartic
regression. On average, 20-30% more computational power was required to model
quartic regression than the 4PL symmetrical sigmoid function.

Table 6: Parameters and metrics of mathematical modeling for during gameplay experiment (4PL symmetrical sigmoid function). Game type |  Gamer type | Coefficient | Metrics
---|---|---|---
a |  b |  c |  d |  $R^{2}$ |  AIC
Puzzle |  Gamer |  0.7113 |  5.0082 |  41.0507 |  1.6653 |  0.9996 |  -27.47
Non Gamer |  0.7701 |  4.5143 |  42.8531 |  2.1471 |  0.9887 |  -8.195
Combi- national |  Gamer |  0.7371 |  3.1548 |  62.6104 |  3.0142 |  0.9943 |  -12.68
Non Gamer |  0.7223 |  1.1433 |  7008 |  8794 |  0.9919 |  -8.719
Strategic |  Gamer |  0.7187 |  0.9913 |  2403 |  5339054 |  0.989 |  -6.724
Non Gamer |  0.7472 |  0.9616 |  290172 |  460270.4 |  0.9605 |  0.3441
Table 7: Parameters and metrics of mathematical modeling for during gameplay experiment (quartic regression). Game type |  Gamer type | Coefficient |  Metrics
---|---|---|---
a |  b |  c |  d |  e |  $R^{2}$
Puzzle |  Gamer |  0.701 |  0.01184 |  -0.00136 |  5.373E-5 |  -5.086E-7 |  1
Non Gamer |  0.701 |  0.04116 |  -0.00341 |  10.598E-5 |  -9.169E-7 |  1
Combi- national |  Gamer |  0.701 |  0.02556 |  -0.00202 |  6.227E-5 |  -5.103E-7 |  1
Non Gamer |  0.701 |  0.05173 |  -0.00268 |  7.081E-5 |  -5.629E-7 |  1
Strategic |  Gamer |  0.701 |  0.06878 |  -0.00379 |  9.874E-5 |  -7.942E-7 |  1
Non Gamer |  0.701 |  0.0989 |  -54541 |  0.000126 |  -9.152E-7 |  1
Figure 12: Residuals of cubic and original average data Table 8: After gameplay mathematical modeling with 4PL symmetrical sigmoid function. Game type |  Music type |  Gamer type | Coefficient | Metrics
---|---|---|---|---
a |  b |  c |  d |  $R^{2}$ |  AIC
Puzzle |  Low Pitch |  Gamer |  1.5410 |  0.3937 |  360.9098 |  -2.4249 |  0.9979 |  -20.95
Non Gamer |  1.8970 |  0.4967 |  14.7121 |  -0.5570 |  0.9973 |  -16.05
Medium Pitch |  Gamer |  1.5414 |  1.2729 |  4.2799 |  0.5133 |  0.9937 |  -15.37
Non Gamer |  1.8970 |  0.79044 |  5.1763 |  0.2516 |  0.9999 |  -36.46
High Pitch |  Gamer |  1.5412 |  3.2904 |  3.4865 |  0.8151 |  0.9989 |  -24.73
Non Gamer |  1.8969 |  0.8284 |  8.9388 |  -0.0215 |  0.0009 |  -28.03
No music |  Gamer |  1.5417 |  1.6380 |  3.9345 |  0.6141 |  0.9913 |  -13.77
Non Gamer |  1.8969 |  0.4707 |  922.3299 |  -7.7168 |  1 |  -40.25
Combi- national |  Low Pitch |  Gamer |  1.7987 |  2.2673 |  4.1514 |  0.6491 |  0.994 |  -12.53
Non Gamer |  2.1484 |  1.4688 |  5.8055 |  0.2179 |  0.9996 |  -23.86
Medium Pitch |  Gamer |  1.7975 |  1.6254 |  4.3045 |  0.5303 |  0.9984 |  -19.37
Non Gamer |  2.1486 |  1.7268 |  4.3938 |  0.5875 |  0.9997 |  -25.68
High Pitch |  Gamer |  1.7970 |  2.3198 |  3.3399 |  0.7732 |  0.9999 |  -35.66
Non Gamer |  2.1487 |  1.7094 |  4.4000 |  0.5887 |  0.9999 |  -31
No music |  Gamer |  1.7975 |  2.7664 |  4.1841 |  0.6739 |  0.9997 |  -27.28
Non Gamer |  2.1490 |  2.5295 |  3.9101 |  0.7385 |  1 |  -42.42
Strategic |  Low Pitch |  Gamer |  2.2175 |  0.8197 |  0.89861 |  -0.7128 |  0.9965 |  -12.21
Non Gamer |  2.4024 |  1.1049 |  7.2473 |  -0.2689 |  0.9988 |  -16.5
Medium Pitch |  Gamer |  2.2183 |  1.5853 |  5.8047 |  0.2552 |  0.9999 |  -30.82
Non Gamer |  2.4022 |  1.2444 |  9.5813 |  -0.4817 |  0.9995 |  -21.38
High Pitch |  Gamer |  2.2188 |  1.6827 |  5.9997 |  0.2723595 |  0.9998 |  -25.39
Non Gamer |  2.399616 |  1.754762 |  7.592323 |  0.03812683 |  0.9985 |  -15.49
No music |  Gamer |  2.217533 |  1.426625 |  7.804479 |  -0.06470811 |  0.9999 |  -28.79
Non Gamer |  2.400454 |  1.721276 |  7.344441 |  0.09479044 |  0.9989 |  -17.12
Table 9: After gameplay mathematical modeling with Quartic Regression function. Game type |  Music type |  Gamer type | Coefficient |  Metrics
---|---|---|---|---
a |  b |  c |  d |  e |  $R^{2}$
Puzzle |  Low Pitch |  Gamer |  1.541 |  -0.2693 |  0.0393 |  -0.0025 |  0.000051 |  1
Non Gamer |  1897 |  -0.3926 |  0.0558 |  -0.00331 |  5.81276E-5 |  1
Medium Pitch |  Gamer |  1.541 |  -0.1170 |  -0.01404 |  0.00367 |  -0.00018 |  1
Non Gamer |  1.897 |  -0.3543 |  0.0576 |  -0.0046 |  0.00014 |  1
High Pitch |  Gamer |  1.541 |  0.0203 |  -0.0599 |  0.0086 |  -0.00035 |  1
Non Gamer |  1.897 |  -0.2892 |  0.0459 |  -0.00415 |  0.00014 |  1
No music |  Gamer |  1.541 |  -0.0649 |  -0.0333 |  0.00598 |  -0.00027 |  1
Non Gamer |  1.897 |  -0.3355 |  0.0585 |  -0.0053 |  0.00017 |  1
Combi- national |  Low Pitch |  Gamer |  1.797 |  0.0247 |  -0.0784 |  0.0114 |  -0.00047 |  1
Non Gamer |  2.149 |  -0.1875 |  0.00181 |  0.00044 |  -9.774E-06 |  1
Medium Pitch |  Gamer |  1.797 |  -0.1109 |  -0.02518 |  0.00474 |  -0.00021 |  1
Non Gamer |  2.149 |  -0.1708 |  -0.0077 |  0.00194 |  -7.305E-5 |  1
High Pitch |  Gamer |  1.797 |  -0.11003 |  -0.0254 |  0.0047 |  -0.00019 |  1
Non Gamer |  2.149 |  -0.1661 |  -0.0104 |  0.0024 |  -9.465E-5 |  1
No music |  Gamer |  1.797 |  0.0514 |  -0.0808 |  0.0107 |  -0.00041 |  1
Non Gamer |  2.149 |  -0.0276 |  -0.069 |  0.0098 |  -0.00038 |  1
Strategic |  Low Pitch |  Gamer |  2.218 |  -0.4486 |  0.0906 |  -0.00989 |  0.00038 |  1
Non Gamer |  2.403 |  -0.3553 |  0.04918 |  0.00019 |  -0.00098 |  1
Medium Pitch |  Gamer |  2.218 |  -0.1179 |  -0.0289 |  0.0044 |  -0.00017 |  1
Non Gamer |  2.403 |  -0.21475 |  0.0129 |  -0.00127 |  6.121E-5 |  1
High Pitch |  Gamer |  2.218 |  -0.0715 |  -0.0428 |  0.0059 |  -0.00023 |  1
Non Gamer |  2.403 |  -0.11075 |  -0.008 |  -0.0001019 |  5.0926E-5 |  1
No music |  Gamer |  2.218 |  -0.1420278 |  -0.00801 |  0.00123 |  -3.549E-5 |  1
Non Gamer |  2.403 |  -0.1115 |  -0.01139 |  0.0006 |  1.852E-5 |  1

The strategic game creates more stress on the subjects of the three broad
categories of games discussed here. However, gamer subject groups adapted more
to strategic games because of their experience.

In the relaxation phase, the subject groups who experienced puzzle games
achieved less BAR than other groups. Even for non-gamers, this observation
stands out. We conclude that even when subjects were not experienced to puzzle
games, puzzles are commonplace in our lives. So, the brain seems to adapt too
quickly.

For mathematical modeling, during gameplay, the stress curve is very
predictable, resulting in $R^{2}$ value 1 for every case. However, when 4PL
symmetrical sigmoid functions were used to model relaxation curves, the
$R^{2}$ value drops to even 0.9913. Only in two cases, the $R^{2}$ value
attains 1. In quartic regression function, despite requiring much more
computational power, yields the $R^{2}$ value to 1.

##  5 Outcome of the Study

The key outcome of the study is delineated as follows:

  1. 1.

Non-gamers experience more stress than gamers,

  2. 2.

During gameplay, the beta band in the frontal region is mostly activated,

  3. 3.

Strategic game creates more stress on human brain,

  4. 4.

Low pitch music is the most useful media for relaxation,

  5. 5.

Residual stress is evident in the frontal lobe when the subjects experienced
high pitch music,

  6. 6.

During relaxation, the alpha band in the frontal region is mostly activated,

  7. 7.

4PL symmetrical sigmoid function performs regression analysis with minimum
parameters/ computational power,

  8. 8.

Quartic regression performs regression analysis follow the relaxation curve
more accurately compared to 4PL symmetrical sigmoid function.

##  6 Conclusion

The study of stress analysis on the human brain for video gaming and music-
evoked relaxation analysis is critical for understanding the human brain for
everyday activities. This study is, to the best of our knowledge, the first of
its kind. This paper presents a comprehensive stress analysis for video
gameplay from a wide range of perspectives. Stress was analyzed from the
rhythm-specific spectral feature, BAR, from EEG. Among all games, the
strategic game proves to be the most stressful. Non-gamers exhibit more stress
to video gameplay than gamers, demonstrating the adaptability of video
gameplay with stress. Music of various pitches was used to compare no music
after the gameplay for relaxation analysis. After video gameplay, low pitch
music proved to be the most effective media for relaxation. Regression
analysis was used to represent relaxation and stress mathematically better.
The most effective function for this is quartic regression. EEG topography has
been used to identify the activation hotspot within the brain for a specific
game or relaxation procedure. During gameplay, the frontal region is
particularly active. This work can be expanded to include other types of VG or
even other stressful activities. In addition, any relaxation technique other
than music can be used to normalize brain activity.

## Acknowledgements

None. No funding to declare.

## Declaration of Competing Interest

The authors have no conflict of interest to disclose.

## References

  * Ahn et al. [2019] Ahn, J.W., Ku, Y., Kim, H.C., 2019.  A novel wearable eeg and ecg recording system for stress assessment.  Sensors 19, 1991\.
  * Al-Shargie et al. [2018] Al-Shargie, F., Tang, T.B., Badruddin, N., Kiguchi, M., 2018\.  Towards multilevel mental stress assessment using svm with ecoc: an eeg approach.  Medical & biological engineering & computing 56, 125–136.
  * Al-Shargie et al. [2017] Al-Shargie, F., Tang, T.B., Kiguchi, M., 2017.  Stress assessment based on decision fusion of eeg and fnirs signals.  IEEE Access 5, 19889–19896.
  * Alchalcabi et al. [2017] Alchalcabi, A.E., Eddin, A.N., Shirmohammadi, S., 2017.  More attention, less deficit: Wearable eeg-based serious game for focus improvement, in: 2017 IEEE 5th international conference on serious games and applications for health (SeGAH), IEEE. pp. 1–8.
  * Anwar et al. [2018] Anwar, S.M., Saeed, S.M.U., Majid, M., Usman, S., Mehmood, C.A., Liu, W., 2018\.  A game player expertise level classification system using electroencephalography (eeg).  Applied Sciences 8, 18\.
  * Arsalan et al. [2019] Arsalan, A., Majid, M., Butt, A.R., Anwar, S.M., 2019\.  Classification of perceived mental stress using a commercially available eeg headband.  IEEE journal of biomedical and health informatics 23, 2257–2264.
  * Chen [2014] Chen, G., 2014.  Automatic eeg seizure detection using dual-tree complex wavelet-fourier features.  Expert Systems with Applications 41, 2391–2394.
  * Choi et al. [2020] Choi, E., Shin, S.H., Ryu, J.K., Jung, K.I., Kim, S.Y., Park, M.H., 2020.  Commercial video games and cognitive functions: video game genres and modulating factors of cognitive enhancement.  Behavioral and Brain Functions 16, 2.
  * Das et al. [2017] Das, D., Bhattacharjee, T., Datta, S., Choudhury, A.D., Das, P., Pal, A., 2017.  Classification and quantitative estimation of cognitive stress from in-game keystroke analysis using eeg and gsr, in: 2017 IEEE Life Sciences Conference (LSC), IEEE. pp. 286–291.
  * He et al. [2018] He, Y., Eguren, D., Azorín, J.M., Grossman, R.G., Luu, T.P., Contreras-Vidal, J.L., 2018\.  Brain–machine interfaces for controlling lower-limb powered robotic systems.  Journal of neural engineering 15, 021004.
  * Jebelli et al. [2018] Jebelli, H., Hwang, S., Lee, S., 2018.  Eeg-based workers’ stress recognition at construction sites.  Automation in Construction 93, 315–324.
  * Johnstone et al. [2020] Johnstone, S.J., Jiang, H., Sun, L., Rogers, J.M., Valderrama, J., Zhang, D., 2020\.  Development of frontal eeg differences between eyes-closed and eyes-open resting conditions in children: Data from a single-channel dry-sensor portable device.  Clinical EEG and Neuroscience , 1550059420946648\.
  * Kracht et al. [2020] Kracht, C.L., Joseph, E.D., Staiano, A.E., 2020.  Video games, obesity, and children.  Current obesity reports 9, 1–14.
  * Kühn et al. [2014] Kühn, S., Gleich, T., Lorenz, R.C., Lindenberger, U., Gallinat, J., 2014.  Playing super mario induces structural brain plasticity: gray matter changes resulting from training with a commercial video game.  Molecular psychiatry 19, 265–271.
  * Marston and Kowert [2020] Marston, H.R., Kowert, R., 2020\.  What role can videogames play in the covid-19 pandemic?  Emerald Open Research 2.
  * Martínez-Tejada et al. [2021] Martínez-Tejada, L., Puertas-González, A., Yoshimura, N., Koike, Y., 2021.  Exploring eeg characteristics to identify emotional reactions under videogame scenarios. brain sci. 2021, 11, 378.
  * McElreath [2018] McElreath, R., 2018.  Statistical rethinking: A Bayesian course with examples in R and Stan.  Chapman and Hall/CRC.
  * Motamedi-Fakhr et al. [2014] Motamedi-Fakhr, S., Moshrefi-Torbati, M., Hill, M., Hill, C.M., White, P.R., 2014.  Signal processing techniques applied to human sleep eeg signals—a review.  Biomedical Signal Processing and Control 10, 21–33.
  * Mozos et al. [2017] Mozos, O.M., Sandulescu, V., Andrews, S., Ellis, D., Bellotto, N., Dobrescu, R., Ferrandez, J.M., 2017.  Stress detection using wearable physiological and sociometric sensors.  International journal of neural systems 27, 1650041.
  * Paszkiel et al. [2020] Paszkiel, S., Dobrakowski, P., Łysiak, A., 2020.  The impact of different sounds on stress level in the context of eeg, cardiac measures and subjective stress level: A pilot study.  Brain Sciences 10, 728\.
  * Rai et al. [2012] Rai, D., Kosidou, K., Lundberg, M., Araya, R., Lewis, G., Magnusson, C., 2012\.  Psychological distress and risk of long-term disability: population-based longitudinal study.  J Epidemiol Community Health 66, 586–592.
  * Roy et al. [2020] Roy, S., Islam, M., Yusuf, M.S.U., Rohan, T.I., 2020.  Frequency impact analysis with music evoked stimulated potentials on human brain, in: Innovations in Electronics and Communication Engineering. Springer, Singapore, pp. 505–513.
  * Roy et al. [2019] Roy, S., Yusuf, M.S.U., Islam, M., Rohan, T.I., 2019\.  Age based mood swing analysis and brain mapping for music genre, in: 2019 5th International Conference on Advances in Electrical Engineering (ICAEE), IEEE. pp. 165–170.
  * Saputra and Iqbal [2017] Saputra, R., Iqbal, B.M., 2017\.  Stress emotion evaluation in multiplayer online battle arena (moba) video game related to gaming rules using electroencephalogram (eeg), in: Proceedings of the 2017 4th International Conference on Biomedical and Bioinformatics Engineering, pp. 74–77.
  * Shon et al. [2018] Shon, D., Im, K., Park, J.H., Lim, D.S., Jang, B., Kim, J.M., 2018.  Emotional stress state detection using genetic algorithm-based feature selection on eeg signals.  International Journal of environmental research and public health 15, 2461.
  * Soysal et al. [2020] Soysal, Ö.M., Kiran, F., Chen, J., 2020.  Quantifying brain activity state: Eeg analysis of background music in a serious game on attention of children, in: 2020 4th International Symposium on Multidisciplinary Studies and Innovative Technologies (ISMSIT), IEEE. pp. 1–7.
  * Subha et al. [2010] Subha, D.P., Joseph, P.K., Acharya, R., Lim, C.M., 2010\.  Eeg signal analysis: a survey.  Journal of medical systems 34, 195–212.
  * Taddy [2019] Taddy, M., 2019.  Business data science: Combining machine learning and economics to optimize, automate, and accelerate business decisions.  McGraw Hill Professional.
  * TWITCH [2021] TWITCH, 2021.  Twitch statistics & charts.
  * Wang et al. [2021] Wang, L., Ding, X., Zhang, W., Yang, S., 2021.  Differences in eeg microstate induced by gaming: A comparison between the gaming disorder individual, recreational game users and healthy controls.  IEEE Access 9, 32549–32558.
  * Welch [1967] Welch, P., 1967.  The use of fast fourier transform for the estimation of power spectra: a method based on time averaging over short, modified periodograms.  IEEE Transactions on audio and electroacoustics 15, 70–73.
  * Yusuf et al. [2019] Yusuf, M.S.U., Islam, M., Roy, S., 2019.  Stress identification during sustained mental task and brain relaxation modeling with beta/alpha band power ratio, in: 2019 4th International Conference on Electrical Information and Communication Technology (EICT), IEEE. pp. 1–5.
  * Zayeni et al. [2020] Zayeni, D., Raynaud, J.P., Revet, A., 2020.  Therapeutic and preventive use of video games in child and adolescent psychiatry: a systematic review.  Frontiers in psychiatry 11, 36\.
  * Zhu [2021] Zhu, L., 2021.  The psychology behind video games during covid-19 pandemic: A case study of animal crossing: New horizons.  Human Behavior and Emerging Technologies 3, 157–159.


# Robust DOA estimation from deep acoustic imaging

###### Abstract

Direction of arrival estimation (DoAE) aims at tracking a sound in azimuth and
elevation. Recent advancements include data-driven models with inputs derived
from ambisonics intensity vectors or correlations between channels in a
microphone array. A spherical intensity map (SIM), or acoustic image, is an
alternative input representation that remains underexplored. SIMs benefit from
high-resolution microphone arrays, yet most DoAE datasets use low-resolution
ones. Therefore, we first propose a super-resolution method to upsample low-
resolution microphones. Next, we benchmark DoAE models that use SIMs as input.
We arrive to a model that uses SIMs for DoAE estimation and outperforms a
baseline and a state-of-the-art model. Our study highlights the relevance of
acoustic imaging for DoAE tasks.

Index Terms—  spherical intensity maps, acoustic imaging, sound event
localization, super-resolution.

##  1 Introduction

The release of large-scale sound event localization and detection (SELD)
datasets [1, 2, 3] allowed for deep learning approaches for direction of
arrival estimation (DoAE) in favor of traditional beamforming [4]. Today, the
best-performing models use a combination of two inputs: 1) the intensity
vectors of first order ambisonics (FOA) and 2) the correlations between
channels in a tetrahedral microphone (4 channels) [5, 6]. However, a spherical
intensity map (SIM) is an alternative input representation that remains
unexplored.

A SIM is computed using delay-and-sum beamforming (DASB) [7]. DeepWave, an
acoustic imaging model [8], builds upon DASB by applying sparse coding [9] to
deblur a cleaner representation of spatial sound activity (see Figure 1). We
evaluated whether we could use DeepWave’s SIM to localize sound on the entire
LOCATA dataset [2], and we found that it is possible to achieve great
localization performance (see Table 1, top row). However, DeepWave assumes
high-resolution inputs, such as those captured by the EigenMike (32 channels)
[10] or Pyramic (48 channels) [11] arrays.

In this study we investigate whether high-resolution SIMs can be used to carry
out DoAE. Since most SELD datasets use low-resolution arrays, we first
demonstrate it is possible to upsample them using a super-resolution model.
Next, we benchmark models that use SIMs as input to carry out DoAE, and
compare against a baseline and open-source state-of-the-art (SoTA) model. Our
code and data are openly-
available.111https://github.com/adrianSRoman/DeepWaveDOA In summary, our
contributions are:

  1. 1.

A method to upsample the covariance between channels in a low-resolution
microphone to high-resolution.

  2. 2.

A benchmark of models that use SIMs for DoAE, evaluated on the LOCATA [2] and
STARSS23 [1] datasets.

Fig. 1: The acoustic image by DeepWave (top) and DASB (bottom) for two sound
sources. Dots denote ground truth.

##  2 Related work

Current DoAE models process the intensity vectors of FOA and correlations
between channels in a tetrahedral microphone [5, 6]. Significant ones include
the “Sound event localization and detection network” (SELDnet) [12],
considered to be a baseline for the task, as well as SoTA models that use
multi-head self-attention [5] and multi-task outputs [6].

A SIM is an acoustic image indicating the location of a sound source [13].
SIMs have not been assessed quantitatively as input for DoAE deep-learning
models. The DASB algorithm [7] is the most commonly-used method to compute
SIMs, but resulting images exhibit poor angular resolution [8]. DeepWave
achieves an equivalent SIM using a backprojection operation, and further
denoises it using deblurring,

$x^{(\ell)}=\tanh\Bigl{(}\underbrace{[\bar{\mathbf{B}}\circ\mathbf{B}]^{H}\text{vec}(\mathbf{\Sigma})}_{backprojection}+\underbrace{\mathbf{P}_{\theta}(\mathbf{L})x^{(\ell-1)}}_{deblurring}-\tau\Bigl{)},$
(1)

where $\mathbf{\Sigma}\in\mathbb{C}^{M\times M}$ is the instantaneous
covariance matrix of a microphone array with $M$ channels. Backprojection is
equivalent to DASB (sec. 5.2 in [14]). It maps
$\mathbf{\Sigma}\in\mathbb{C}^{M\times M}$ onto a uniformly-tiled SIM (i.e.
tesselated [15]), representing azimuth and elevation.
$\text{vec}(\mathbf{\Sigma})\in\mathbb{C}^{M^{2}}$ is a column-wise matrix-
flattening operator, $\mathbf{B}\in\mathbb{C}^{M\times N}$ is a trainable
matrix, and $\bar{\mathbf{B}}$ is its complex conjugate. $\circ$ is the
Khatri-Rao product and $H$ denotes a Hermitian matrix. Deblurring iterates
over an initial random spherical map $x^{(0)}\in\mathbb{R}^{N}$ using
graphical convolutions that clean it. $\mathbf{P}_{\theta}(\mathbf{L})$ is a
polynomial of the graph Laplacian $\mathbf{L}\in\mathbf{R}^{NxN}$, and
$\theta$ are graph convolution filters.

DeepWave applies these operations $L$ times with tanh for sparsity. The
trainable parameters are $\theta$, $\mathbf{B}$ and $\tau$ (a bias term), and
are shared across the $L$ iterations. In practice, $F$ of these operations
happen in parallel, one for each frequency band. DeepWave’s output acoustic
image (or SIM) has $N$ pixels, proportional to the number of microphone
channels.

##  3 Approach

###  3.1 DoAE from acoustic images via K-means clustering

First we optimized a DeepWave model to get a sense of maximum performance on
the LOCATA dataset (see Methods). This model processes a high-resolution
microphone. We apply K-means clustering over DeepWave’s SIMs to calculate the
DoA. For a 32ch input with $F=9$ and $L=5$, DeepWave generates a $N=242$
acoustic image with values ranging between 0 and 1. After arranging the $N$
pixels in a 2D space $N=A\times E$ for azimuth ($A$) and elevation ($E$) we
obtain $\mathbf{I}\in\mathbb{R}^{F\times A\times E}_{[0,1)}$. Next, we apply a
1D Tukey window along the elevation axis with a $0.8$ tapering factor to
remove artifacts closer to the poles. Finally, we run K-means clustering with
$K=3$ on the $15$ pixels with the maximal intensity (all others are clipped to
zero). This yields three centroids that represent DoA coordinates. To make
this algorithm robust, we apply two post-processing heuristics: (1) ignore
clusters where points are separated by more than $15^{\circ}$ from the
centroid, and (2) merge clusters with centroids within $15^{\circ}$ of each
other.

###  3.2 Upsampling $\mathbf{\Sigma}$ using super-resolution

DeepWave’s resolution is proportional to the number of microphone channels.
Lower-resolution microphones, such as 4 channel, are commonly used in SELD
datasets. We introduce a channel upsampling method derived from the Deep Back
Projection Network (DBPN) by Haris et al. [16], a computer vision super-
resolution model. Our complex-valued DBPN (CDBPN) upsamples
$\mathbf{\Sigma}\in\mathbb{C}^{4\times
4}\rightarrow\mathbf{\Sigma}\in\mathbb{C}^{32\times 32}$ (i.e. from a 4ch to a
32ch array; a factor of 8).

###  3.3 DeepWave SIMs as inputs for DoAE

The original DeepWave processes 100ms at a time and lacks temporal memory [8].
We also study the effect of adding a gated recurrent unit (GRU) on top. In
other words, DeepWave’s SIM becomes the GRU input (Figure 2). We train models
in an end-to-end fashion using the ADPIT loss, which makes this type of model
have the multi-ACCDOA representation to localize overlapping sound sources
[17].

##  4 Methodology

Fig. 2: Signal pipeline in the models we study.

###  4.1 Datasets

Evaluations with real recordings. We evaluate models on LOCATA because it was
recorded with an EigenMike (32ch) in a room where human actors moved while
speaking (the microphone also moves in some recordings) [2]. To apply CDBPN
upsampling we simulate a tetrahedral microphone (4ch) using the 6th, 10th,
22nd, and 26th EigenMike channels (following Adavane et al.’s [18] approach).
Metadata does not indicate a speaker’s gender, so annotations only have the
instantaneous DoA (every 100ms). Our study focuses on DoAE, so these
annotations are all we need (i.e. we do not classify sound events into
categories). We also evaluate on STARSS23 [1], which contains more rooms and
uses a 4ch microphone. We only use its DoA annotations, as we do not classify
events into specific classes.

Simulated recordings for training. To train models we simulate 32ch
recordings. Our simulated dataset was generated using the SpatialScaper
library [19]. We integrated two RIR databases that use the EigenMike. The
first one is METU-SPARG, collected at 240 points on a cubic grid in a
classroom [20], and the second one is ARNI [21], collected in the “variable
acoustics room” at Aalto University. We simulate soundscapes with speech from
the FSD50K dataset [22] spatialized in either of the two rooms. We generate a
total of 50min of data in the METU room and 10min in ARNI.

Models evaluated on STARSS23 are trained using the companion dataset with
simulated 4ch recordings using RIRs from 12 rooms [3]. We refer to it as
“DCASE-sim”.

Input | Model / operations |  $LE$ $\downarrow$ (std.) |  $LR$ $\uparrow$ (std.)
---|---|---|---
32ch | Backproj $\rightarrow$ Deblurring $\rightarrow$ K-means | 14.8o ($\pm 0.00$) | 99.20 ($\pm 0.00$)
Backproj $\rightarrow$ Deblurring $\rightarrow$ GRU | 20.9o ($\pm 3.63$) | 69.36 ($\pm 2.15$)
Backproj $\rightarrow$ GRU | 20.0o ($\pm 3.01$) | 70.46 ($\pm 4.14$)
Backproj $\rightarrow$ Conv2D $\rightarrow$ GRU | 15.96o ($\pm 2.51$) | 76.36 ($\pm 9.21$)
4ch $\rightarrow$ CDBPN $\rightarrow$ 32ch | Backproj $\rightarrow$ Deblurring $\rightarrow$ K-means | 27.10o ($\pm 0.00$) | 99.20 ($\pm 0.00$)
Backproj $\rightarrow$ Deblurring $\rightarrow$ GRU | 20.30o ($\pm 2.76$) | 72.33 ($\pm$ 0.04)
Backproj $\rightarrow$ GRU | 18.06o ($\pm 3.53$) | 70.96 ($\pm 3.82$)
Backproj $\rightarrow$ Conv2D $\rightarrow$ GRU | 17.42o ($\pm 0.73$) | 82.40 ($\pm 5.76$)
4ch | SELDnet23 | 16.8o ($\pm 0.45$) | 77.13 ($\pm 3.84$)
FOA | SELDnet23 | 22.43o ($\pm 3.80$) | 80.73 ($\pm 10.98$)
FOA + 4ch | EINV2 | 19.83o ($\pm 0.75$) | 80.66 ($\pm 7.26$)
Table 1: Localization error (LE) and recall (LR) on LOCATA. Scores reflect the
average across three experimental replications. Note that the EINV2 model uses
both tetrahedral microphone (4ch) and first order ambisonics (FOA) inputs.

###  4.2 Models, ablations, and baselines

The two main models we study are DeepWave+K-means and DeepWave+GRU. We carry
out an ablation study of the DeepWave+GRU model222the original DeepWave paper
has an ablation study without the GRU [8] to understand the role of the
deblurring operation given the additional GRU. Therefore we have three
variants: 1) Backproj $\rightarrow$ Deblur $\rightarrow$ GRU, 2) Backproj
$\rightarrow$ GRU, and 3) Backproj $\rightarrow$ Conv2D $\rightarrow$ GRU
(i.e. we replace deblurring with a 2D convolution layer)333The variant that
passes $\mathbf{\Sigma}$ directly to the GRU does not learn, independent of
whether CDBPN upsampling is used..

Our optimizations are gradual steps to enable DoAE using a DeepWave backbone.
DeepWave+K-means enables DoAE on SIMs, provided high-resolution array data.
CDBPN introduces low-resolution array upsampling. A GRU on top of a DeepWave
backbone allows end-to-end training using multi-ACCDOA representations. In
each variant DeepWave operations are key for DoAE.

Across datasets we compare performance against two models: SELDnet23, a
baseline for DoAE, and EINV2, the highest-ranked open-source model444acording
to the “DCASE” 2023 SELD challenge. Note these models were designed to carry
out SELD (i.e. classification in addition to DoAE). Also, EINV2 assumes data
augmentation, but the authors did not make this code available on their
repository555https://github.com/Jinbo-Hu/DCASE2022-TASK3. Therefore we use the
EINV2 without data augmentation.

###  4.3 Training procedure and evaluation metrics

CDBPN: We train CDPBN to upsample from 4ch to 32ch. We use METU soundscapes
for training and ARNI for validation. The input and target are the
instantaneous (100ms) 4ch $\mathbf{\Sigma}\in\mathbb{C}^{4\times 4\times F}$
and 32ch $\mathbf{\Sigma}\in\mathbb{C}^{32\times 32\times F}$, respectively.
We use the original DBPN code, but we use $F$ channels instead of RGB, and
train two models, one for the real part and another for the imaginary part of
$\mathbf{\Sigma}$. We freeze the optimal CDBPN and use it in experiments where
we upsample from 4ch to 32ch.

LOCATA experiments: We train models using the generated METU soundscapes, and
we validate using the ARNI ones. The final evaluation is carried out across
the entire LOCATA dataset. Frequency bands are nine total ($F=9$) and linearly
spaced from 1.5kHz to 4.5kHz. We evaluate DeepWave models with and without the
CDBPN upsampling.

STARSS23 experiments: STARSS23 comes divided in four directories (“dev-train-
tau”, “dev-train-sony”, “dev-test-tau” and “dev-test-sony”), each with a
unique set of rooms, and each room has its own sound sources (i.e. a set of
“actors” and sounding objects such as a guitar or household blender). We train
models using “DCASE-sim”, “dev-train-tau”, and “dev-train-sony”, and we
validate using “dev-test-tau”. The final evaluation is carried out on the
“unseen” rooms in “dev-test-sony”. Note that the optimal DeepWave model that
we evaluate on the STARSS23 dataset uses CDBPN since the STARSS23 and “DCASE-
sim” datasets only have 4ch.

Metrics. We evaluate Localization Error (LE), the radial difference between
predicted and true location per sound event, and Localization Recall (LR), the
true positive rate of instantaneous detections out of the total annotated
sound event instances [23]. We do not measure sound classification performance
since we focus on sound localization.

##  5 Results

Table 1 shows performance by the model variants on the LOCATA dataset and
compares against SELDnet23 and EINV2. DeepWave+K-means with 32ch a input
outperforms all models with low LE and high LR. We optimized this model to
give a sense of “ceiling” performance. Its counterpart using 4ch upsampled
with CDBPN shows a deteriorated LE (see Figure 3 bottom), but LR is same. We
enhanced CDBPN for upsampling $\mathbf{\Sigma}$; however, this does not
encompass SIM generation, thus posing challenges for K-means post-processing.

The models with Backprojection, Deblurring, and GRU show LE around 20o,
independent of whether the input is 32ch or upsampled from 4ch using CDBPN.
This indicates that the GRU is resilient to CDBPN distortions, and can improve
LE by ~7o compared to DeepWave+K-means.

The models that only use Backprojection and GRU further improve LE, indicating
that Deblurring is unnecessary given the GRU. This is consistent with the
result we discussed in the previous paragraph, where we note that the GRU is
resilient to CDBPN distortions. In other words, DeepWave’s Deblurring becomes
unnecessary when the GRU is added. Adding a Conv2D layer between
Backprojection and the GRU further improves performance. This makes sense, as
the Conv2D layer operates across the $F$ frequency “channels”, and passes the
GRU a representation with encoded information across frequency bands. In other
words, it saves the GRU the step of having to aggregate information across the
frequency axis.

Note that all GRU models show a deteriorated LR compared to the ones using
K-means. This could be caused by the temporal memory that the GRU adds, which
may come with a time constant to “react” in response to sounds appearing and
disappearing from the scene. In contrast, DeepWave+K-means operated on single
frames of audio without memory, and was optimized to “quickly” react to sound
activity.

Input | Model |  $LE$ $\downarrow$ |  $LR$ $\uparrow$
---|---|---|---
up32ch | Backproj $\rightarrow$ Conv $\rightarrow$ GRU∗ | 20.5o | 70.1
4ch | SELDnet23 | 23.3o | 82.3
FOA | SELDnet23 | 21.9o | 83.0
FOA + 4ch | EINV2 | 24.0o | 84.2
Table 2: Performance of SELDnet23, EINV2 and our best model on the STARSS23
“dev-test-sony” split. ∗Plus two MHSA layers after the GRU.

Next we evaluated the optimal model (CDBPN $\rightarrow$ Backproj
$\rightarrow$ Conv2D $\rightarrow$ GRU) on STARSS23. Compared to LOCATA,
STARSS23 contains more diverse sound sources and acoustic conditions. We found
optimal performance by adding two multi-headed self-attention (MHSA) layers
after the GRU [24]. Our model outperforms the LE performance of SELDnet
trained with FOA, as well as SELDnet trained with 4ch and EINV2. Reasons for
the deteriorated EINV2 performance compared to SELDnet23 include the smaller
validation set we used (“dev-test-tau”), and the test set being smaller and
more challenging (“dev-test-sony”). Note that EINV2 is known to have a female
speech LE “a lot higher than average” [6].

When it comes to CDBPN, we analyzed its performance in terms of the magnitude
and phase error. Figure 3 breaks down these results. The top matrices show the
input $\mathbf{\Sigma}_{4}\in\mathbb{C}^{4\times 4\times F}$, CDBPN output
$\mathbf{\hat{\Sigma}}_{32}\in\mathbb{C}^{32\times 32\times F}$ and target
$\mathbf{\Sigma}_{32}\in\mathbb{C}^{32\times 32\times F}$ (averaged across the
frequency axis) for magnitude (first row) and phase (second row). The bottom
plot shows the magnitude and phase error between
$\text{avg}(\mathbf{\hat{\Sigma}}_{32})$ and
$\text{avg}(\mathbf{\Sigma}_{32})$ as a function of frequency (16
logaritmically-spaced bands). Results indicate that upsampling
$\mathbf{\Sigma}$ at higher-frequency bands is more challenging for CDBPN.

Fig. 3: CDBPN upsampling of a single reverberant white noise source directly
facing the front of the microphone.

##  6 Conclusion and future work

We have studied SIMs for DoAE. We focused on DeepWave because of its high-
resolution SIM. We proposed a CDBPN model to upsample microphone array
information from 4ch to 32ch. This allows our optimal model (4ch $\rightarrow$
CDBPN $\rightarrow$ 32ch $\rightarrow$ Backproj $\rightarrow$ Conv2D
$\rightarrow$ GRU) to process existing SELD data with 4ch audio. The model can
be interpreted as the combination of DeepWave (Backprojection to generate a
SIM) and SELDnet23 (Conv2D + GRU) operations. Our systematic benchmark against
the SELDnet and EINV2 models on the LOCATA and STARSS dataset is an indicator
that our models can be generalized to other DoAE tasks.

Future work includes developing our model to also carry out sound event
classification. Furthermore, generalizing CDBPN work with arbitrary microphone
array shapes will be an important endeavor. For the time being, our results
demonstrate the advantages of using SIMs for DoAE, which could benefit
existing and future DoAE and SELD models.

##  7 Acknowledgements

This work is supported by the National Science Foundation grant no.
IIS-1955357. The authors thank the funding source and their grant
collaborators.

## References

  * [1] Kazuki Shimada, Archontis Politis, et al.,  “Starss23: An audio-visual dataset of spatial recordings of real scenes with spatiotemporal annotations of sound events,”  arXiv preprint 2306.09126, 2023.
  * [2] Heinrich W Löllmann, Christine Evers, Alexander Schmidt, Heinrich Mellmann, Hendrik Barfuss, Patrick A Naylor, and Walter Kellermann,  “The locata challenge data corpus for acoustic source localization and tracking,”  in 2018 IEEE 10th Sensor Array and Multichannel Signal Processing, 2018, pp. 410–414.
  * [3] Archontis Politis,  “[DCASE2022 Task 3] Synthetic SELD mixtures for baseline training,” Apr. 2022.
  * [4] Pierre-Amaury Grumiaux, Srđan Kitić, Laurent Girin, and Alexandre Guérin,  “A survey of sound source localization with deep learning methods,”  The Journal of the Acoustical Society of America, vol. 152, no. 1, pp. 107–151, 2022.
  * [5] Qing Wang, Jun Du, Hua-Xin Wu, Jia Pan, Feng Ma, and Chin-Hui Lee,  “A four-stage data augmentation approach to resnet-conformer based acoustic modeling for sound event localization and detection,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 31, pp. 1251–1264, 2023.
  * [6] Jinbo Hu, Yin Cao, Ming Wu, Qiuqiang Kong, Feiran Yang, Mark D. Plumbley, and Jun Yang,  “A track-wise ensemble event independent network for polyphonic sound event localization and detection,”  in IEEE ICASSP, 2022, pp. 9196–9200.
  * [7] Boaz Rafaely,  Fundamentals of spherical array processing, vol. 8,  Springer, 2015.
  * [8] Matthieu Simeoni, Sepand Kashani, Paul Hurley, and Martin Vetterli,  “Deepwave: a recurrent neural-network for real-time acoustic imaging,”  Advances In Neural Information Processing Systems, vol. 32, 2019.
  * [9] Karol Gregor and Yann LeCun,  “Learning fast approximations of sparse coding,”  in Proceedings of the 27th intl. conf. on machine learning, 2010, pp. 399–406.
  * [10] MH Acoustics,  “Eigenmike microphone array release notes (v17. 0),”  25 Summit Ave, Summit, NJ, USA, 2013.
  * [11] Robin Scheibler, Juan Azcarreta, René Beuchat, and Corentin Ferry,  “Pyramic: Full stack open microphone array architecture and dataset,”  in 2018 16th IWAENC. IEEE, 2018, pp. 226–230.
  * [12] Sharath Adavanne, Archontis Politis, Joonas Nikunen, and Tuomas Virtanen,  “Sound event localization and detection of overlapping sources using convolutional recurrent neural networks,”  IEEE Journal of Selected Topics in Signal Processing, vol. 13, no. 1, pp. 34–48, 2018.
  * [13] Ning Chu, José Picheral, Ali Mohammad-Djafari, and Nicolas Gac,  “A robust super-resolution approach with sparsity constraint in acoustic imaging,”  Applied Acoustics, vol. 76, pp. 197–208, 2014.
  * [14] Alle-Jan van der Veen and Stefan J Wijnholds,  “Signal processing tools for radio astronomy,”  in Handbook of Signal Processing Systms., pp. 421–463. Springer, 2013.
  * [15] Saksham Singh Kushwaha, Iran R Roman, and Juan P Bello,  “Analyzing the effect of equal-angle spatial discretization on sound event localization and detection.,”  in DCASE, 2022.
  * [16] Muhammad Haris, Gregory Shakhnarovich, and Norimichi Ukita,  “Deep back-projection networks for super-resolution,”  in Proceedings of the IEEE CVPR conference, 2018, pp. 1664–1673.
  * [17] Kazuki Shimada, Yuichiro Koyama, Shusuke Takahashi, Naoya Takahashi, Emiru Tsunoo, and Yuki Mitsufuji,  “Multi-accdoa: Localizing and detecting overlapping sounds from the same class with auxiliary duplicating permutation invariant training,”  in IEEE ICASSP. IEEE, 2022, pp. 316–320.
  * [18] Sharath Adavanne, Archontis Politis, and Tuomas Virtanen,  “Localization, detection and tracking of multiple moving sound sources with a convolutional recurrent network,”  arXiv preprint 1904.12769, 2019.
  * [19] Iran Roman, Chris Ick, Sivan Ding, Adrian Roman, Brian McFee, and Juan Pablo Bello,  “Spatial scaper: a library to simulate and augment soundscapes for sound event localization and detection in realistic rooms,”  in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024.
  * [20] Orhun Olgun and Huseyin Hacihabiboglu,  “METU SPARG Eigenmike em32 Acoustic Impulse Response Dataset v0.1.0,” Apr. 2019.
  * [21] Thomas McKenzie, Leo McCormack, and Christoph Hold,  “Dataset of spatial room impulse responses in a variable acoustics room for six degrees-of-freedom rendering and analysis,”  arXiv preprint 2111.11882, 2021.
  * [22] Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra,  “Fsd50k: an open dataset of human-labeled sound events,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 30, pp. 829–852, 2021.
  * [23] Archontis Politis, Annamaria Mesaros, et al.,  “Overview and evaluation of sound event localization and detection in dcase 2019,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 684–698, 2020.
  * [24] Parthasaarathy Sudarsanam, Archontis Politis, and Konstantinos Drossos,  “Assessment of self-attention on learned features for sound event localization and detection,”  arXiv preprint arXiv:2107.09388, 2021.