Ivar Ekeland, Roger Temam, Jeffrey Dean, David Grove, Craig Chambers, Kim B.
Bruce, and Elisa Bertino 11institutetext: Quaid-i-Azam University, Islamabad,
Pakistan22institutetext: Fudan University, Shanghai, China33institutetext:
Polytechnic University of Bari, Italy 44institutetext: Riphah International
University, Islamabad, Pakistan55institutetext: University of Manouba,
Manouba, Tunisia66institutetext: Taibah University, Medina, Saudi Arabia
77institutetext: Edinburgh Napier University, Edinburgh, UK

# Voting Classifier-based Intrusion Detection for IoT Networks

Muhammad Almas Khan 11 Muazzam A Khan 11 Shahid Latif 22 Awais Aziz Shah 33
Mujeeb Ur Rehman 44 Wadii Boulila 5566 Maha Driss 5566 Jawad Ahmad 77

###### Abstract

Internet of Things (IoT) is transforming human lives by paving the way for the
management of physical devices on the edge. These interconnected IoT objects
share data for remote accessibility and can be vulnerable to open attacks and
illegal access. Intrusion detection methods are commonly used for the
detection of such kinds of attacks but with these methods, the
performance/accuracy is not optimal. This work introduces a novel intrusion
detection approach based on an ensemble-based voting classifier that combines
multiple traditional classifiers as a base learner and gives the vote to the
predictions of the traditional classifier in order to get the final
prediction. To test the effectiveness of the proposed approach, experiments
are performed on a set of seven different IoT devices and tested for binary
attack classification and multi-class attack classification. The results
illustrate prominent accuracies on Global Positioning System (GPS) sensors and
weather sensors to 96% and 97% and for other machine learning algorithms to
85% and 87%, respectively. Furthermore, comparison with other traditional
machine learning methods validates the superiority of the proposed algorithm.

###### keywords:

IoT; Intrusion Detection; Machine Learning; Classification; Voting Classifier.

##  1 Introduction

Over the past few years, the Internet of Things (IoT) has witnessed rapid
growth and adaptability in every sector of life [1], [2], [3], [4], and [5].
Figure 1 shows the domain of IoT/IIoT in real life. With the rapid growth of
IoT, the security of the smart systems based on IoT is an important task.
Current literature reveals the importance of cyber-attack detection schemes
for smart systems [6, 7, 8, 9, 10]. History shows that an IoT device
compromised by an attacker resulted in a power blackout and 225000 people were
affected due to the lower quality of security mechanism [11]. IoT technology
has many features such as inter-dependency among devices, constrained,
diversity, etc [12]. Understanding theses features in detail help us in
securing the smart systems from upcoming threats. Out of these features
identified in [12], for example, the interdependence of IoT devices provide
less human involvement and takes a smart decision when one device implicitly
controls another within the network as they share data with each other. To
understand the inter-dependence, let us considers a simple scenario of smart
home devices, where the thermostat device senses the home temperature and
compares it with the threshold value. If the temperature is out of the
threshold stored in the thermostat, it tries to balance the environmental
temperature. In order to balance the environmental temperature it checks the
Air Condition (AC) smart plug. If the AC is unplugged, it automatically opens
the windows in order to stabilize the environmental temperature and allow
ventilation. Without proper security, this interdependence among IoT devices
provides an opportunity to intruders by compromising an IoT device they can
open the door or window of a facility. For instance, the code injection and
Man in the Middle (MitM) attack [13] can compromise an IoT device.

Figure 1: Internet of Things Applications

In code injection, the adversary injects malicious code to IoT devices and
performs alteration of the data. Similarly, in MitM attack the intruder
eavesdrops and tries to communicate messages between two nodes. Here, the
attacker can control the action of other node. To secure each IoT device
within the IoT network and make secure the interdependence among devices, this
paper evaluates the ensemble-based voting classifier for intrusion detection
in the IoT network. Ensemble-based voting classifier combined the traditional
ML algorithm and give voting to its prediction to get the final prediction by
the ensemble-based voting classifier. There are two types of voting e.g soft
voting and hard voting. A detailed mathematical description of the proposed
algorithm is given in section III. The performance of the conceived algorithm
is evaluated on the latest dataset which is a real representative of IoT
networks called Ton-IoT [14]. Ton-IoT [15] has been selected as the dataset in
our experiments after analyzing the available datasets in the current state of
the art representing a real IoT/IIoT network. The novel contributions of this
paper are:

  * •

Proposal of an ensemble-based model for attack classification.

  * •

Evaluation of the proposed model for binary and multi-attacks class IoT
dataset.

  * •

Comparison of the proposed model with traditional machine learning algorithms
using evaluation metrics such as accuracy, precision, recall, and F1-score.
115 F1-measure.

The remainder of the paper has been organized as follows: Section II presents
background and detailed discussion on the related work and existing threats in
IoT. The problem statement and proposed solution is discussed in section III.
Section IV describes the selection and pre-processing of the selected
datasets. Section V presents the cross-comparison between the existing state
of the art intrusion detection algorithms and our proposed methodology.
Finally, Section VI concludes the work and draws some potential future
directions.

##  2 Literature Review

IoT is an innovative technology for making the systems smart by automating the
working environment to reduce human involvement and improve system performance
[16]. Security in IoT is one of the main concerns because innovative types of
cyberattacks are emerging every day as IoT technology is rapidly evolving
[12]. Without security, these networks are vulnerable to attacks and a
favorite place for intruders. To attack and take control over the whole IoT
system, attackers usually compromise an IoT device and control indirectly the
other devices [11] & [17]. To prevent these attacks at early stages, one of
the possible ways is to detect the intrusion or malicious activity of an
attacker on the network. In this regard, Intrusion detection algorithms play
an important role in detecting such malicious activities within the networks.
Table I provides the summary of this section. In [18], 4 ensemble techniques
such as Boosted Trees, Bagged Trees, Subspace Discriminant, and RUSBoosted
Trees were implemented to propose RPL based network intrusion detection for
IoT networks. The proposed NIDS are evaluated on RPL-NIDDS17 dataset. To
prevent the malicious events in IoT networks, more specifically the botnet
attack against MQTT, HTTP, and DNS protocols, [19] proposed an ensemble-based
intrusion detection technique. Three machine learning algorithms namely DT,
NB, and ANN were combined to develop an adaptive boost ensemble method for
attack detection. The introduced scheme was evaluated on two datasets namely
UNSW-NB15 and NIMS botnet dataset.

Table 1: Intrusion detection models with evaluation on datasets and evaluation
metrics

Author & paper title | Model | Evaluation Dataset | Evaluation metrics
---|---|---|---
[18] | Ensemble learning | RPL-NIDDS17 | Accuracy, ROC
[19] | Adaptive boost |  | UNSW-NB15
---
NIMS
| Accuracy, False positive rate
---
Detection rate, ROC
[20] | Ensemble Voting |  | CIC-IDS2017
---
NSL-KDD
AWID
| Accuracy, Precision
---
Detection rate, F-measure
[21] | XGBoost | NSL-KDD |  | Accuracy, Precision
---
Recall, F-measure, ROC
[22] | XGBoost | UNSW-NB15 | Accuracy
[23] | Ensemble learning | AWID |  | Accuracy, Precision
---
Recall, F1-Measure
[15] |  | LR, LDA
---
kNN, RF, CART
NB, SVM, LSTM
Ton-IoT |  | Accuracy, Precision
---
Recall, F-measure

In [20], 2 step intrusion detection system were proposed. In the first step,
the proposed method selects optimal features and in the second step, they
combine machine learning algorithms namely C4.5, RF, and Random Forest by
Penalizing Attribute(RF PA) to develop a voting ensemble classifier. The
proposed method was evaluated on NSL-KDD, AWID, and CIC-IDS2017 datasets with
evaluation metrics Accuracy, Precision, Detection rate, F-measure. In [21]
Intrusion detection using XGBoost model proposed. The proposed model was
evaluated on NSL-KDD dataset with evaluation metrics accuracy, precision,
recall, and F1-measure. To detect intrusion on network-level [22] proposed
Extreme Gradient Boosting (XGBoost) based intrusion detection, the proposed
system was evaluated on the UNSW-NB15 with accuracy as an evaluation metric.
For supporting the narrowband and broadband IoT applications the wifi is
needed in some place, to protect the attacks on wifi, [23] proposed an
ensemble-based intrusion detection by evaluated on AWID dataset with
evaluation metrics accuracy, precision, recall, and F1-measure. In [24], Liu
et all studied 11 algorithms comprising 7 supervised and 3 unsupervised
algorithms to find the best solution for the detection of intrusion in the IoT
networks. The findings suggest XGBOOST as the best performer and Expectation-
Maximization (EM) from unsupervised algorithms. The performance of 11
algorithms was tested on NSL-KDD datasets with Accuracy, Matthews correlation
coefficient (MCC), and Area Under the Curve (AUC) as evaluation metrics.

However, the evaluation performed using the datasets [25] plays the main role
in intrusion detection. In [15], the authors proposed a new generation
telemetry dataset of IoT 4.0 which has been generated in a diverse attack
scenario. This dataset contains data of 7 IoT devices having real sensor
measurements. The performance of ML and DL algorithms is tested in which DT
and RF outperform when their evaluation metrics such as accuracy, precision,
recall, F-measure, etc were observed. However, the results show that the
performance of a single ML algorithm change as the data from the sensors
changes. Therefore ensemble-based learning is required in order to give
optimal performance in attack detection on every sensor. Designing an optimal
intrusion detection system requires a realistic dataset as mentioned before,
which is close to real-time scenarios [26, 27, 15]. In this research work,
several publically available datasets are explored and evaluated based on
several comparison metrics such as diverse attack scenarios, telemetry data of
IoT, and availability of separate datasets for each IoT object, etc. For the
designing and analyzing of an IDS, publically available datasets are KDDCUP9,
NSL-KDD, Labelled Wireless Sensor Network Data Repository (LWSNDR), UNSW-NB15,
Aegean Wi-Fi Intrusion Dataset (AWID), ISCX, UNW-IoT trace, UNSW-IoT, BoT-IoT,
RPL-NIDDS17, CICIDS2017, and Ton-IoT. A newly generated dataset known as
TON_IoT [15] is a publicly available for the evaluation of an IDS in IoT and
IIoT networks. This dataset has been generated in diverse attack scenarios and
data is gathered from the Telemetry data of IoT/IIoT services. The dataset
contains 7 IoT representing Fridge sensor, Garage door, GPS (Global
Positioning Sensor) sensor, Modbus, Weather, Motion light sensor, and
Thermostat. The data stored in these datasets are different from each other
and hence obtained dataset is from heterogeneous sources. All IoT devices do
not deal with the same type of data, For example, garage door IoT devices only
deal with ’ON’ or ’OFF’ representing the state of the door. Similarly, some
devices deal with real-valued numeric data. That’s why the performance of the
traditional ML algorithm does not remain the same with changes in the type of
data. Due to the aforementioned reason, we have combined a few ML algorithms
such as DT, NB, RF, and K-NN to propose an accurate classifier that could deal
with any type of data with optimal performance on a maximum number of devices
within IoT networks. Table 2 summarizes the publicly available dataset for
analysis of an IDS. One can see from this table that the Ton-IoT dataset
provides an accurate environment (based on different/separate IoT device data)
for an IDS for IoT devices.

Table 2: Comparison of publicly available datasets. Comparison Metrics Dataset |  KDDCUP99 |  NSL-KDD |  LWSNDR |  UNSW-NB15 |  AWID |  ISCX |  UNW-IoT trace |  UNSW-IoT |  BoT-IoT |  RPL-NIDDS17 |  CICIDS2017 |  Ton-IoT
---|---|---|---|---|---|---|---|---|---|---|---|---
Attack label | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | N/A | ✓ | ✓ | ✓ | ✓ | ✓
Diverse attack scenario | x | x | x | ✓ | ✓ | ✓ | x | ✓ | ✓ | ✓ | ✓ | ✓
Telemetry data of IoT | x | x | ✓ | x | x | x | x | x | x | x | x | ✓
Separate dataset for each IoT | x | x | x | x | x | x | x | x | x | x | x | ✓
Year |  1998 |  1998 |  2010 |  2015 |  2015 |  2017 |  2018 |  2019 |  2018 |  2018 |  2017 |  2020

##  3 Proposed Ensemble-based Intrusion Detection Scheme

The classifiers based on ensemble technique is one of the methods for
generating a powerful classifier that has higher classification accuracy as
compared to traditional ML classifiers. Due to hybrid in nature, ensemble-
based techniques generally perform better on most of the dataset and hence
give optimal results when compared to an individual traditional ML classifier.
However, it is tedious to find the best pair of machine learning classifiers
to the ensemble for particular data to get the best possible results [28]. In
this research work, we have combined DT, NB, RF, and kNN using a voting-based
technique. To understand fully the proposed method, we have outlined the
mathematical description in this section. To understand the proposed ensemble-
based classifier, preliminary mathematical knowledge has been explained in
this section. Prior to the mathematical details, we have highlighted an
ensemble-based voting classifier in Figure 2. $C1$, $C2$ … $C_{m}$ shows
classifiers while $P1$, $P2$, … $P_{m}$ show the predictions, respectively. To
show the superiority of an ensemble-based classifier, authors have discussed
different factors in [29, al2020ensemble, 30, 28, 31].

Figure 2: Ensemble-based voting Classifier

In a voting method for binary classification, the simple majority is used
while for multi-classification it is extended to plurality voting.
Mathematically ensemble-based voting is written as:

$\hat{y}=argmax_{i}\left[\sum_{1}^{m}W_{j}\chi_{A}(C_{j}(X)=i)\right]$ (1)

where the $\hat{y}$ is the prediction of ensemble model, $C_{j}$ is the
classifiers ensembles, $\chi_{A}$ is the characteristic functions, $A$ is the
unique label set, and $W_{j}$ represents weight with every classifier
predictions. If weight with every classifier is the same then the above
expression can be written as:

$\widehat{y}=\mod(C_{1}(x),C_{2}(x),C_{3}(x),...C_{m}(x))$ (2)

In Eq. 2, the $C_{i}(x)$ represents an individual classifier that predicts
labels for a sample $x$. For example, in a binary classification, the
mathematical expression representing the model can be written as:

$C(x)=\left[sign\sum_{j}^{m}C_{j}(x)\right]=\left\\{\begin{array}[]{ll}1&if\sum_{i}C_{j}(x)\geq
0\\\ -1&\mbox{otherwise}\\\ \end{array}\right.$ (3)

To further elaborate, the above description, lets suppose 5 base classifiers
in a binary classification using uniform weights (each base classifier has
equal weights). $C_{1}(x)\rightarrow 1$, $C_{2}(x)\rightarrow 1$,
$C_{3}(x)\rightarrow 1$, $C_{4}(x)\rightarrow 0$ $C_{5}(x)\rightarrow 0$.
$\hat{y}$= mod $\\{$1,1,1,0,0$\\}$ $\hat{y}$=1

In the above case, 3 base classifiers out of 5 classify sample $x$ as 1 and 2
classifiers are classified as 0 and then by majority voting the prediction of
ensemble model $\hat{y}$ using hard voting will be 1 because maximum
classifier is agreed on 1. To further show the superiority of ensemble and
voting classifiers, assume $n$ number of based classifiers in the ensemble
model. Let us consider that a single classifier has $\varepsilon$ error and
further assume that all classifiers are independent. Considering the
aforementioned assumptions, the probability error of an ensemble model is
represented as follows:

$P(y\geq
k)=\sum_{k}^{n}\binom{n}{k}\varepsilon^{k}(1-\varepsilon)^{n-k}=\varepsilon_{ensemble}$
(4)

##  4 Pre-processing of the selected datasets

This section provides the overview and pre-processing of the selected datasets
for the experimental analysis. Based on the cross-comparison of existing
datasets performed in Section II, the recently open available Ton-IoT [15]
dataset has been selected for the evaluation of the proposed intrusion
detection in IoT networks. This dataset contains 7 different IoT devices
namely fridge, motion light, garage door, thermostat, GPS sensor, Modbus, and
weather. The data of these datasets are stored separately in 7 CSV file
formats. Additionally, the data for the aforementioned heterogeneous IoT
devices have been plotted within this section. Figure 3 represents the
statistical analysis for the datasets of 2 IoT objects (fridge and garage
door) for normal and attack, respectively. Prior to evaluating the proposed
model, feature scaling and label encoding has been applied to some IoT data.
These datasets have real sensor measurements with low and high values. The
statistical analysis for Modbus & GPS tracker datasets is given in Figure 4
and the dataset statistics of the thermostat, weather sensor, and motion light
is shown in Figure 5 with normal and attacked data. The Light Motion sensor
dataset has ON/OFF status of lights and similarly, the garage dataset has
CLOSE/OPEN in their dataset. We have applied label encoding to prepare it to
be used in our proposed model. To evaluate the performance of the Ensemble-
based voting classifier on all IoT data, a python script has been implemented
to combine all 7 IoT datasets into one CSV file. Afterward, a median value is
chosen for filling the missing variables in the combined IoT dataset as
depicted in Figure 2. A recent study reveals that the use of median is less
exposed to outlier as compared to the use of mean value, [32].

Table 3: Seven IoT datasets statistics of attack and normal data

|  | Attacked data
---|---|---
IoT dataset |  Normal data | Password | scanning | XSS | DDos | Ransomeware | Injection | Backdoor
| Garage
---
Door
35000 | 5000 | 529 | 2042 | 5000 | 2902 | 5000 | 5000
| Fridge
---
Sensor
35000 | 5000 | n/a | 2042 | 5000 | 2902 | 5000 | 5000
| GPS
---
tracker
35000 | 5000 | 550 | 577 | 5000 | 2833 | 5000 | 5000
Modbus | 35000 | 5000 | 529 | 577 | n/a | n/a | 5000 | 5000
| Light
---
Motion
35000 | 5000 | 1775 | 449 | n/a | 2264 | 5000 | 5000
Weather | 35000 | 5000 | 529 | 866 | 5000 | 2865 | 5000 | 5000
Thermostat | 35000 | 5000 | 1775 | 449 | 5000 | 2264 | 5000 | 5000

Figure 3: Combined IoT dataset statistics

##  5 Results and Discussion

This section provides a cross-comparison of the reviewed and integrated
datasets in our proposed models with different combinations of DT-RF-kNN-NB,
DT-RF-NB, and DT-RF-kNN machine learning algorithms. Moreover, the conducted
analysis considers four evaluation metrics i.e., accuracy, precision, recall,
and F-measure. First, the binary classification for each dataset is given for
the combination of algorithms against evaluation matrices. Later, the binary
classification is done for combined IoT datasets.

Table 4: Comparison of the traditional ML algorithms and the proposed models.

|  Performance of the state of art machine learning |  Our proposed models with different combinations
---|---|---
Datasets |  Evaluation metrics | LR |  LDA | kNN | RF | CART | NB | SVM | LSTM |  | (DT-RF-
---
kNN-NB)
| (DT-RF-
---
NB)
| (DT-RF-
---
kNN)
| Accuracy | 0.57 | 0.77 | 0.99 | 0.97 | 0.97 | 0.50 | 0.81 | 1.00 | 1.00 | 1.00 | 1.00
| Precision | 0.34 | 0.79 | 0.99 | 0.97 | 0.97 | 0.53 | 0.86 | 1.00 |  1.00 | 1.00 | 1.00
| Recall | 0.58 | 0.77 | 0.99 | 0.97 | 0.97 | 0.51 | 0.82 | 1.00 | 1.00 | 1.00 | 1.00
Fridge sensor |  F-measure | 0.43 | 0.77 | 0.99 | 0.97 | 0.97 | 0.51 | 0.80 | 1.00 | 1.00 | 1.00 | 1.00
|  |
| Accuracy | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |  1.00 | 1.00 | 1.00
| Precision | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00
| Recall | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |  1.00 | 1.00 | 1.00
Garage door | F-measure | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00
|  |
| Accuracy | 0.86 | 0.86 | 0.88 | 0.85 | 0.84 | 0.84 | 0.86 | 0.88 |  0.96 | 0.95 | 0.96
| Precision | 0.88 | 0.88 | 0.89 | 0.85 | 0.85 | 0.86 | 0.88 | 0.89 | 0.96 | 0.96 | 0.96
| Recall | 0.86 | 0.86 | 0.88 | 0.85 | 0.85 | 0.85 | 0.87 | 0.88 |  0.96 | 0.96 | 0.96
GPS sensor | F-measure | 0.87 | 0.87 | 0.88 | 0.85 | 0.85 | 0.86 | 0.87 | 0.88 | 0.96 | 0.96 | 0.96
|  |
| Accuracy | 0.67 | 0.67 | 0.77 | 0.97 | 0.98 | 0.67 | 0.67 | 0.67 | 0.97 | 0.96 | 0.96
| Precision | 0.46 | 0.46 | 0.77 | 0.98 | 0.99 | 0.46 | 0.46 | 0.47 | 0.97 | 0.97 |  0.97
| Recall | 0.68 | 0.68 | 0.78 | 0.98 | 0.98 | 0.68 | 0.68 | 0.68 | 0.97 | 0.97 | 0.97
Modbus | F-measure | 0.55 | 0.55 | 0.77 | 0.98 | 0.99 | 0.55 | 0.55 | 0.55 | 0.97 |  0.97 | 0.97
|  |
| Accuracy | 0.58 | 0.58 | 0.54 | 0.58 | 0.58 | 0.58 | 0.58 | 0.59 | 0.58 | 0.58 | 0.58
| Precision | 0.34 | 0.34 | 0.34 | 0.34 | 0.34 | 0.34 | 0.34 | 0.35 |  0.34 | 0.35 | 0.34
| Recall | 0.59 | 0.59 | 0.59 | 0.59 | 0.59 | 0.59 | 0.59 | 0.59 | 0.59 | 0.59 | 0.58
Light motion | F-measure | 0.43 | 0.43 | 0.43 | 0.43 | 0.43 | 0.43 | 0.43 | 0.44 |  0.43 | 0.44 | 0.43
|  |
| Accuracy | 0.66 | 0.66 | 0.60 | 0.66 | 0.59 | 0.66 | 0.66 | 0.66 | 0.66 | 0.66 | 0.66
| Precision | 0.44 | 0.44 | 0.56 | 0.59 | 0.56 | 0.44 | 0.44 | 0.45 |  0.78 | 0.54 | 0.61
| Recall | 0.66 | 0.66 | 0.61 | 0.66 | 0.59 | 0.66 | 0.66 | 0.67 | 0.66 | 0.66 | 0.67
Thermostat |  F-measure | 0.53 | 0.53 | 0.57 | 0.53 | 0.57 | 0.53 | 0.53 | 0.54 | 0.53 | 0.53 | 0.65
|  |
| Accuracy | 0.58 | 0.60 | 0.81 | 0.84 | 0.87 | 0.59 | 0.63 | 0.82 |  0.97 | 0.97 | 0.98
| Precision | 0.60 | 0.59 | 0.81 | 0.84 | 0.88 | 0.72 | 0.68 | 0.82 | 0.98 | 0.97 | 0.98
| Recall | 0.59 | 0.60 | 0.81 | 0.84 | 0.87 | 0.59 | 0.63 | 0.81 |  0.97 | 0.97 | 0.98
Weather | F-measure | 0.53 | 0.53 | 0.81 | 0.84 | 0.87 | 0.67 | 0.55 | 0.80 | 0.97 | 0.97 | 0.98

The results for binary attack classification are discussed here in this
section. The dataset is classified using several traditional algorithms
including DT, RF, NB, and the proposed ensemble-based method. The obtained
results from the ML algorithms are depicted in Table 3. It is evident from
Table 3 that the proposed ensemble-based methods outperform other ML
algorithms in most cases. However, in the case of the Modbus and light motion
sensor, the accuracy and other accuracy parameters are comparably the same as
the ensemble method. Moreover, it is clear from Table 3 that a single
combination or a base classifier does not produce accurate results for all
sensors. Data of each sensor are combined into a single CSV file and several
experiments were performed. The results of a combined dataset are shown in
Table 4. One can see from Table 4 that ensemble-based classifier have optimum
accuracy, precision, recall, and F-measure in most cases. However, for the
CART-based classifier, we have almost similar results when compared to the
ensemble-based classifier. We have also performed experiments for multi-
classification and results are depicted in Table 5. It is clear from Table 5
that CART outperforms all other classifiers.

Table 5: Binary classification on combined IoT dataset

| Performance of the state of art machine learning | Our proposed models
---|---|---
Datasets |  Evaluation metrics | LR | LDA | kNN | RF | CART | NB | SVM | LSTM |  | (DT-RF-
---
kNN-NB)
| (DT-RF-
---
NB)
| (DT-RF-
---
kNN)
| Accuracy | 0.61 | 0.68 | 0.84 | 0.85 | 0.88 | 0.62 | 0.61 | 0.81 | 0.87 | 0.88 | 0.88
| Precision | 0.37 | 0.74 | 0.85 | 0.87 | 0.90 | 0.63 | 0.37 | 0.83 | 0.90 | 0.90 | 0.89
| Recall | 0.61 | 0.68 | 0.84 | 0.85 | 0.88 | 0.62 | 0.61 | 0.81 | 0.88 | 0.88 | 0.88
Combined IoT Dataset | F-measure | 0.46 | 0.62 | 0.84 | 0.85 | 0.88 | 0.51 | 0.46 | 0.80 | 0.87 | 0.88 | 0.88

Table 6: Multi class classification results on combined IoT dataset

| Performance of the state of art machine learning | Our proposed models
---|---|---
Datasets |  Evaluation metrics | LR | LDA | kNN | RF | CART | NB | SVM | LSTM |  | (DT-RF-
---
kNN-NB)
| (DT-RF-
---
NB)
| (DT-RF-
---
kNN)
| Accuracy | 0.61 | 0.62 | 0.72 | 0.71 | 0.77 | 0.54 | 0.60 | 0.68 | 0.76 | 0.75 | 0.75
| Precision | 0.37 | 0.74 | 0.85 | 0.87 | 0.90 | 0.63 | 0.37 | 0.83 | 0.75 | 0.74 | 0.74
| Recall | 0.61 | 0.68 | 0.84 | 0.85 | 0.88 | 0.62 | 0.61 | 0.76 | 0.75 | 0.75 | 0.75
Combined IoT Dataset | F-measure | 0.46 | 0.62 | 0.84 | 0.85 | 0.88 | 0.51 | 0.46 | 0.80 | 0.73 | 0.73 | 0.74

##  6 Conclusion

This paper presented a cross-comparison of the available datasets for
intrusion attack detection in the Internet of Things (IoT) domain and propose
a novel approach based on an ensemble-based voting classifier that combines
multiple traditional classifiers as a base learner and gives the vote to the
predictions of the traditional classifier in order to get the final
prediction. The performance of the proposed model has been compared with the
state-of-the-art intrusion detection algorithms available in the current
literature and a comparison has been drawn against the matrices of accuracy,
precision, recall, and f-measure. The outcome of the evaluation demonstrated
that the proposed strategy outperforms in most of the cases.

## References

  * [1] M. M. Islam, A. Rahaman, and M. R. Islam, “Development of smart healthcare monitoring system in iot environment,” _SN computer science_ , vol. 1, pp. 1–11, 2020.
  * [2] G. Mois, S. Folea, and T. Sanislav, “Analysis of three iot-based wireless sensors for environmental monitoring,” _IEEE Transactions on Instrumentation and Measurement_ , vol. 66, no. 8, pp. 2056–2064, 2017.
  * [3] A. Salam and S. Shah, “Urban underground infrastructure monitoring iot: the path loss analysis,” in _2019 IEEE 5th World Forum on Internet of Things (WF-IoT)_. IEEE, 2019, pp. 398–401.
  * [4] Y. Hajjaji, W. Boulila, I. R. Farah, I. Romdhani, and A. Hussain, “Big data and iot-based applications in smart environments: A systematic review,” _Computer Science Review_ , vol. 39, p. 100318, 2021.
  * [5] A.-R. Al-Ali, I. A. Zualkernan, M. Rashid, R. Gupta, and M. Alikarar, “A smart home energy management system using iot and big data analytics approach,” _IEEE Transactions on Consumer Electronics_ , vol. 63, no. 4, pp. 426–434, 2017.
  * [6] A. Churcher, R. Ullah, J. Ahmad, F. Masood, M. Gogate, F. Alqahtani, B. Nour, W. J. Buchanan _et al._ , “An experimental analysis of attack classification using machine learning in iot networks,” _Sensors_ , vol. 21, no. 2, p. 446, 2021.
  * [7] A. Shafique, J. Ahmed, W. Boulila, H. Ghandorh, J. Ahmad, and M. U. Rehman, “Detecting the security level of various cryptosystems using machine learning models,” _algorithms_ , vol. 1, p. 5, 2021.
  * [8] A. Ali, S. Shaukat, M. Tayyab, M. A. Khan, J. S. Khan, J. Ahmad _et al._ , “Network intrusion detection leveraging machine learning and feature selection,” in _2020 IEEE 17th International Conference on Smart Communities: Improving Quality of Life Using ICT, IoT and AI (HONET)_. IEEE, 2020, pp. 49–53.
  * [9] A. Qayyum, J. Ahmad, W. Boulila, S. Rubaiee, F. Masood, F. Khan, W. J. Buchanan _et al._ , “Chaos-based confusion and diffusion of image pixels using dynamic substitution,” _IEEE Access_ , vol. 8, pp. 140 876–140 895, 2020\.
  * [10] F. Masood, W. Boulila, J. Ahmad, S. Sankar, S. Rubaiee, W. J. Buchanan _et al._ , “A novel privacy approach of digital aerial images based on mersenne twister method with dna genetic encoding and chaos,” _Remote Sensing_ , vol. 12, no. 11, p. 1893, 2020.
  * [11] L. Robert, J. Michael, and C. Tim, “Analysis of the cyber attack on the ukrainian power grid,” _USA: Electricity Information Sharing and Analysis Centre (E-ISAC)_ , 2016.
  * [12] W. Zhou, Y. Jia, A. Peng, Y. Zhang, and P. Liu, “The effect of iot new features on security and privacy: New threats, existing solutions, and challenges yet to be solved,” _IEEE Internet of Things Journal_ , vol. 6, no. 2, pp. 1606–1616, 2018.
  * [13] M. Zolanvari, M. A. Teixeira, L. Gupta, K. M. Khan, and R. Jain, “Machine learning-based network vulnerability analysis of industrial internet of things,” _IEEE Internet of Things Journal_ , vol. 6, no. 4, pp. 6822–6834, 2019.
  * [14] N. Mustafa, “Ton-iot dataset,” https://cloudstor.aarnet.edu.au/plus/s/ds5zW91vdgjEj9i, (accessed 2020).
  * [15] A. Alsaedi, N. Moustafa, Z. Tari, A. Mahmood, and A. Anwar, “Ton_iot telemetry dataset: a new generation dataset of iot and iiot for data-driven intrusion detection systems,” _IEEE Access_ , vol. 8, pp. 165 130–165 150, 2020.
  * [16] S. Batool, N. A. Saqib, M. K. Khattack, and A. Hassan, “Identification of remote iot users using sensor data analytics,” in _Future of Information and Communication Conference_. Springer, 2019, pp. 328–337.
  * [17] G. Falco, C. Caldera, and H. Shrobe, “Iiot cybersecurity risk modeling for scada systems,” _IEEE Internet of Things Journal_ , vol. 5, no. 6, pp. 4486–4495, 2018.
  * [18] A. Verma and V. Ranga, “Elnids: Ensemble learning based network intrusion detection system for rpl based internet of things,” in _2019 4th International conference on Internet of Things: Smart innovation and usages (IoT-SIU)_. IEEE, 2019, pp. 1–6.
  * [19] N. Moustafa, B. Turnbull, and K.-K. R. Choo, “An ensemble intrusion detection technique based on proposed statistical flow features for protecting network traffic of internet of things,” _IEEE Internet of Things Journal_ , vol. 6, no. 3, pp. 4815–4830, 2018.
  * [20] Y. Zhou, G. Cheng, S. Jiang, and M. Dai, “Building an efficient intrusion detection system based on feature selection and ensemble classifier,” _Computer Networks_ , vol. 174, p. 107247, 2020.
  * [21] S. S. Dhaliwal, A.-A. Nahid, and R. Abbas, “Effective intrusion detection system using xgboost,” _Information_ , vol. 9, no. 7, p. 149, 2018.
  * [22] A. Bansal and S. Kaur, “Extreme gradient boosting based tuning for classification in intrusion detection systems,” in _International Conference on Advances in Computing and Data Sciences_. Springer, 2018, pp. 372–380.
  * [23] F. D. Vaca and Q. Niyaz, “An ensemble learning based wi-fi network intrusion detection system (wnids),” in _2018 IEEE 17th international symposium on network Computing and Applications (NCA)_. IEEE, 2018, pp. 1–5.
  * [24] R. Panigrahi and S. Borah, “A detailed analysis of cicids2017 dataset for designing intrusion detection systems,” _International Journal of Engineering & Technology_, vol. 7, no. 3.24, pp. 479–482, 2018.
  * [25] M. S. Amin, L. Hassan, A. A. Shah, U. Akbar, and H. A. Niaz, “Accelerated gpu based protein sequence alignment–an optimized database sequences approach,” _IJCSNS_ , vol. 17, no. 10, p. 231, 2017.
  * [26] J. Asharf, N. Moustafa, H. Khurshid, E. Debie, W. Haider, and A. Wahab, “A review of intrusion detection systems using machine and deep learning in internet of things: Challenges, solutions and future directions,” _Electronics_ , vol. 9, no. 7, p. 1177, 2020.
  * [27] S. Latif, Z. Zou, Z. Idrees, and J. Ahmad, “A novel attack detection scheme for the industrial internet of things using a lightweight random neural network,” _IEEE Access_ , vol. 8, pp. 89 337–89 350, 2020.
  * [28] A. Dogan and D. Birant, “A weighted majority voting ensemble approach for classification,” in _2019 4th International Conference on Computer Science and Engineering (UBMK)_. IEEE, 2019, pp. 1–6.
  * [29] W. Boulila, I. R. Farah, K. S. Ettabaa, B. Solaiman, and H. B. Ghézala, “Improving spatiotemporal change detection: A high level fusion approach for discovering uncertain knowledge from satellite image databases,” in _Icdm_ , vol. 9. Citeseer, 2009, pp. 222–227.
  * [30] S. Raschka, _Python Machine Learning: Unlock deeper insights into Machine Leaning with this vital guide to cutting-edge predictive analytics_. Packt Publishing, 2015, https://www.amazon.com/Python-Machine-Learning-Sebastian-Raschka-ebook/dp/B00YSILNL0(visited ).
  * [31] A. A. Aburomman and M. B. I. Reaz, “A novel svm-knn-pso ensemble method for intrusion detection system,” _Applied Soft Computing_ , vol. 38, pp. 360–372, 2016.
  * [32] A. Géron, _A. Géron, Hands-on Machine Learning with Scikit-Learn and TensorFlow:Concepts, Tools, and Techniques to Build Intelligent Systems._ USA: O’Reilly Media, 2017, https://www.amazon.com/Hands-Machine-Learning-Scikit-Learn-TensorFlow/dp/1491962291(visited ).


# A Comparative Study of Machine Learning Models Predicting Energetics of
Interacting Defects

Hao Yu
Department of Electrical and Computer Engineering
Boston University
Boston, MA 02215
<EMAIL_ADDRESS>

###### Abstract

Interacting defect systems are ubiquitous in materials under realistic
scenarios, yet gaining an atomic-level understanding of these systems from a
computational perspective is challenging - it often demands substantial
resources due to the necessity of employing supercell calculations. While
machine learning techniques have shown potential in accelerating materials
simulations, their application to systems involving interacting defects
remains relatively rare. In this work, we present a comparative study of three
different methods to predict the free energy change of systems with
interacting defects. We leveraging a limited dataset from Density Functional
Theory(DFT) calculations to assess the performance models using materials
descriptors, graph neural networks and cluster expansion. Our findings
indicate that the cluster expansion model can achieve precise energetics
predictions even with this limited dataset. Furthermore, with synthetic data
generate from cluster expansion model at near-DFT levels, we obtained enlarged
dataset to assess the data requirements for training accurate prediction
models using graph neural networks for systems featuring interacting defects.
A brief discussion of the computational cost for each method is provided at
the end. This research provide a preliminary evaluation of applying machine
learning techniques in imperfect surface systems.

##  1 Introduction

Defects in materials profoundly influence their properties. Understanding
defects interactions is essential for optimizing material properties.
Phenomena arising from defect interactions can occur from atomic-level to
mesoscale and modeling across the scales is computationally challenging.
Recently, machine learning potentials have shown success in accelerating
materials simulations for molecules and crystals[Noé et al., 2020]. However,
surface systems with a number of defects are rarely studied. Except defects
interactions are complex, simulations of surface systems are costly since
calculation for large supercells are needed [Sun and Ceder, 2013]. These
prevent the progress of studies on complex phenomena in surface systems.
To explore the potential of applying machine learning techniques to study
interacting defects on surfaces at atomic level, we provided as case study of
single-type defects on pure element crystal. Driven from possible applications
in real-life problem related to lithium battery[Liu and Lu, 2017], we focus on
different numbers of vacancies on (100) surface of lithium (Figure 1). We
defined the defect concentration on surface as

$\textrm{defect concentration}=\frac{\textrm{number of
vacancies}}{\textrm{sites of perfect crystal on surface}}\times 100\%,$ (1)

The prediction task is, given the structure of defects distribute on surface
to predict the free energy change $\Delta G$ when vacancies present on the
surface. This free energy change is defined as

$\Delta G=E_{\textrm{vac}}+nE_{\textrm{Li}}-E_{\textrm{Li slab}},$ (2)

where $E_{\textrm{vac}}$ is the ground state energy of lithium surface with
defects from Density Functional Theory calculation(DFT), $n$ is the number of
vacancies in this system and $E_{\textrm{Li}}$ is the chemical potential of
lithium (atomic energy of lithium in its most stable form), and $E_{\textrm{Li
slab}}$ is the ground state energy of a pristine lithium supercell.
Our initial data set calculated by DFT has 88 structures with concentration
from 6.25% to 50%. We split the training set and test set by concentration to
make the training set and test set dissimilar both in configurations and
concentrations. The training set contains structures with concentration less
than 40%, and there are 73 structures. The test set contains the rest 15
structures with concentration higher than 40%. We first study the model
performance on initial test data with three possible approaches. Cluster
expansion(CE) is the the approach we showed it can predict the target at high
accuracy. We further generate more surface systems with different defect
configurations and their energy using this cluster expansion model trained
with initial data, and use these configuration-energy pairs to test the
potential of applying graph neural network(GNN) in this type of systems.
Contributions of this work include (1) Qualitative reveal the limitation of
machine learning potentials for systems with interacting defects and provide a
model to alleviate; (2) Establish a computationally feasible way to test the
potential of GNN for surface with intereacting defects.

Figure 1: Example of lithium slab with defects on surface. Left: Top view is
looking at the surface with defects (represented by gray circles). Circles
filled with purple represent the surface lithium atoms, while circles with
half filled purple represent the lithium atoms in the second layer. Dashed
square indicate the unit cell. Right: Side view of a lithium slab. Surface
atoms represented with filled purple circles. Atoms at first 2 layers match
the ones on the left figure. Dashed square is unit cell box. We tried
different representations by keeping atoms with 2/4/6 layers in the structure.

##  2 Methodology

###  2.1 Atomic Representations

Within the machine learning framework of predicting property $y\in\mathbb{R}$
of a material system $(\mathbf{Z},\mathbf{r})$, where
$\mathbf{Z}\in\mathbb{R}^{N\times 3}$ is the matrix of atomic positions and
$\mathbf{r}\in\mathbb{N}^{N}$ is atomic numbers, it is assumed that the
property $y$ has contribution from each atom in the system[Bader, 1991]. It
can be formulated into regression task in two ways, (1) transform
$(\mathbf{Z},\mathbf{r})$ to descriptors and use ML algorithms learn from
these representations; (2) transform $(\mathbf{Z},\mathbf{r})$ as a graph
$\mathcal{G}$ with $N$ nodes and edges represented in adjacency matrix
$\mathbf{A}\in\mathbb{R}^{N\times N}$ and then apply graph neural network.
Coulomb matrix (CM), Smooth Overlap of Atomic Positions(SOAP) and Many-body
Tensor Representation (MBTR) are commonly used descriptors [Laakso et al.,
2023, Himanen et al., 2020]. Descriptors are encoded and limited within
domain-knowledge from experts, it is effective when training data is little.
Graphs are naturally good representation of atomic structures, and graph
neural networks are used for tasks where graph-structured data has inherent
relationships. To obtain a good GNN, large amount of data related to this task
is needed. We used the framework provided by the open catalyst
project[Chanussot* et al., 2021] conduct experiments.

###  2.2 Configurational Cluster Expansion

When the property of a material system is related to the configurational
degrees of freedom, configurational cluster expansion is a way to build a
model that can extrapolate predictions to unseen configurations. In our case,
The lithium surface systems we consider only differs at surface configurations
and there are $2^{N}$ configurations, $N$ is the sites of perfect crystal on
surface. This collection is denoted as
$\vec{\sigma}=(\sigma_{1},...,\sigma_{n},...,\sigma_{N})$ The energy
dependency with configurations in cluster expansion can be formulated with
$E(\mathbf{\vec{\sigma}})=\sum_{\alpha}V_{\alpha}\Phi_{\alpha}(\vec{\sigma}),$
where $\alpha$ refers to the point, pair, triplet, etc. clusters within the
crystal, $V_{\alpha}$ is the effective interaction strength of a cluster and
$\Phi_{\alpha}$ is the crystal basis function. We can obtain these quantities
when we construct a cluster expansion model provided few structures in the
configurational space[Ångqvist et al., 2019].

##  3 Results

Figure 2: Train and Test predictions with initial small data set. Table 1: Statistics of Model Predictions | MAE (Train/Test in $eV$)
---|---
Model Name/ Layers | 2 | 4 | 6
MBTR | 0.006/0.032 | 0.012/0.038 | 0.014/0.147
DimeNet | 0.020/0.117 | 0.028/0.138 | 0.117/0.219
Cluster Expansion | 0.007/0.021 | 0.007/0.024 | 0.007/0.024

The models were trained on systems with low defect concentrations and then
tested on systems with high defect concentrations. We explored 3 kinds
approaches: (1) descriptors + ML algorithms, (2) graph neural networks, (3)
cluster expansion. For the first two approaches, we presented the prediction
statistics of the most representative models, they are (1) MBTR[Huo and Rupp,
2022] with linear regression, (2) DimeNetGasteiger et al. [2022]. We tested
the best number of atoms to include in representation and the results in mean
absolute error(MAE) and coefficient of determination($R^{2}$) are in Table 1
and Figure 2. All the models training with representations building from 2
layers of atoms, which are surface and subsurface atoms, have the lowest test
MAE. Cluster expansion model can predict the energy with high concentration
defects most accurately and achieving negligible error compare to DFT level
accuracy. Linear regression model with MBTR also has low test MAE, however,
the prediction error is higher when the concentration of defect is higher.
DimeNet has the worst performance compare to the last two models, this is due
to the training data is limited. Since the most important factor that impact
the energy of this kind of extended system is surface configuration, when
building a prediction model, information about the atoms far from surface
makes less contribution in determining the free energy change of the system.
This match well with the results in Figure 1 comparing column-wise. Notably,
in graph neural network, if we represented the supercell structures as graphs
while only surfaces differ, the model will treat every structure the same.

Since cluster expansion model is superior in prediction accuracy and
uncertainty, we use this cluster expansion model to generate more target
energies with different surface configurations at low concentrations and
support the training of DimeNet. Using the energies predicted by cluster
expansion as surrogates, it bypass the need of computational cost for
thousands of supercell of DFT calculationsto test the potential of DimeNet
applied at systems with interacting defects. We trained DimeNet with 500 and
5,000 CE generated energies respectively and the results are in Figure 3 and
Table 2. As there are more training data points, DimeNet gains more expressive
representation. When training with 5,000 configuration, DimeNet is able to
predict as good as MBTR designed by domain expert. This finding motivate us to
do further studies on how DimeNet captures relationship between defects and
interactions by utilizing more low concentration defect configurations and
whether the representation generalize to high concentration systems.

Figure 3: Test performance of DimeNet training with more data provided by cluster expansion model Table 2: DimeNet train on more configurations Number of Training Configurations | 73 | 500 | 5000
---|---|---|---
Test MAE ($eV$) | 0.117 | 0.052 | 0.033

##  4 Conclusion

In this work, we explored three approaches to predict the free energy change
of surface systems with interacting defects with a small data set. While
materials descriptor with classical machine learning algorithm performs well
with limited data, cluster expansion is more accurate and less biased in
prediction. Within the scope of interacting defects scattered on surface,
cluster expansion model can serve as a means to provide training data at low
cost to survey the potential of applying deep neural networks in the task of
property prediction for surface with interacting defects. On the other hand,
it is worth considering the necessity of using machine learning with
descriptors or deep learning to study the defects interactions at surface only
when cluster expansion model performs well with purchasable cost to obtain
training data.

##  5 Disclaimer

This project was conducted independently during my early stage of graduate
studies and may not have received formal supervision. If you come across this
report, please feel free to contact me with any issues or suggestions you may
have.

## References

  * Noé et al. [2020] Frank Noé, Alexandre Tkatchenko, Klaus-Robert Müller, and Cecilia Clementi.  Machine learning for molecular simulation.  _Annual Review of Physical Chemistry_ , 71(1):361–390, 2020.  doi: 10.1146/annurev-physchem-042018-052331.  URL https://doi.org/10.1146/annurev-physchem-042018-052331.  PMID: 32092281.
  * Sun and Ceder [2013] Wenhao Sun and Gerbrand Ceder.  Efficient creation and convergence of surface slabs.  _Surface Science_ , 617:53–59, 2013.  ISSN 0039-6028.  doi: https://doi.org/10.1016/j.susc.2013.05.016.  URL https://www.sciencedirect.com/science/article/pii/S003960281300160X.
  * Liu and Lu [2017] Guangyu Liu and Wei Lu.  A model of concurrent lithium dendrite growth, sei growth, sei penetration and regrowth.  _Journal of The Electrochemical Society_ , 164(9):A1826, jun 2017.  doi: 10.1149/2.0381709jes.  URL https://dx.doi.org/10.1149/2.0381709jes.
  * Bader [1991] Richard F. W. Bader.  A quantum theory of molecular structure and its applications.  _Chemical Reviews_ , 91(5):893–928, 1991.  doi: 10.1021/cr00005a013.  URL https://doi.org/10.1021/cr00005a013.
  * Laakso et al. [2023] Jarno Laakso, Lauri Himanen, Henrietta Homm, Eiaki V. Morooka, Marc O. J. Jäger, Milica Todorović, and Patrick Rinke.  Updates to the DScribe library: New descriptors and derivatives.  _The Journal of Chemical Physics_ , 158(23):234802, 06 2023.  ISSN 0021-9606.  doi: 10.1063/5.0151031.  URL https://doi.org/10.1063/5.0151031.
  * Himanen et al. [2020] Lauri Himanen, Marc O.J. Jäger, Eiaki V. Morooka, Filippo Federici Canova, Yashasvi S. Ranawat, David Z. Gao, Patrick Rinke, and Adam S. Foster.  Dscribe: Library of descriptors for machine learning in materials science.  _Computer Physics Communications_ , 247:106949, 2020.  ISSN 0010-4655.  doi: https://doi.org/10.1016/j.cpc.2019.106949.  URL https://www.sciencedirect.com/science/article/pii/S0010465519303042.
  * Chanussot* et al. [2021] Lowik Chanussot*, Abhishek Das*, Siddharth Goyal*, Thibaut Lavril*, Muhammed Shuaibi*, Morgane Riviere, Kevin Tran, Javier Heras-Domingo, Caleb Ho, Weihua Hu, Aini Palizhati, Anuroop Sriram, Brandon Wood, Junwoong Yoon, Devi Parikh, C. Lawrence Zitnick, and Zachary Ulissi.  Open catalyst 2020 (oc20) dataset and community challenges.  _ACS Catalysis_ , 2021.  doi: 10.1021/acscatal.0c04525.
  * Ångqvist et al. [2019] Mattias Ångqvist, William A. Muñoz, J. Magnus Rahm, Erik Fransson, Céline Durniak, Piotr Rozyczko, Thomas H. Rod, and Paul Erhart.  Icet – a python library for constructing and sampling alloy cluster expansions.  _Advanced Theory and Simulations_ , 2(7):1900015, 2019.  doi: https://doi.org/10.1002/adts.201900015.  URL https://onlinelibrary.wiley.com/doi/abs/10.1002/adts.201900015.
  * Huo and Rupp [2022] Haoyan Huo and Matthias Rupp.  Unified representation of molecules and crystals for machine learning.  _Machine Learning: Science and Technology_ , 3(4):045017, nov 2022.  doi: 10.1088/2632-2153/aca005.  URL https://dx.doi.org/10.1088/2632-2153/aca005.
  * Gasteiger et al. [2022] Johannes Gasteiger, Janek Groß, and Stephan Günnemann.  Directional message passing for molecular graphs, 2022.


Copyright for this paper by its authors. Use permitted under Creative Commons
License Attribution 4.0 International (CC BY 4.0).

Italian Workshop on Artificial Intelligence for Human Machine Interaction
(AIxHMI 2023), November 06, 2023, Rome, Italy

[orcid=<EMAIL_ADDRESS>]

[orcid=0000-0002-9831-3210<EMAIL_ADDRESS>]

[orcid=0000-0001-7840-6228<EMAIL_ADDRESS>url=, ] [1]

[orcid=0000-0002-6279-6660<EMAIL_ADDRESS>url=https://mmsp.unimib.it/francesca-gasparini/, ]

[orcid=0000-0003-1308-8497<EMAIL_ADDRESS>url=https://mmsp.unimib.it/alessandra-grossi/, ]

[orcid=0000-0002-4405-8234<EMAIL_ADDRESS>url=https://mmsp.unimib.it/aurora-saibene/, ]

[1]Corresponding author.

# An experimental protocol to access immersiveness in video games

Marika Malaspina  Jessica Amianto Barbato  Marco Cremaschi  Francesca
Gasparini  Alessandra Grossi  Aurora Saibene  University of Milano-Bicocca,
Viale Sarca 336, 20126, Milano, Italy  NeuroMI, Milan Center for Neuroscience,
Piazza dell’Ateneo Nuovo 1, 20126, Milano, Italy

(2023)

###### Abstract

In the video game industry, great importance is given to the experience that
the user has while playing a game. In particular, this experience benefits
from the players’ perceived sense of being in the game or immersion. The level
of user immersion depends not only on the game’s content but also on how the
game is displayed, thus on its User Interface (UI) and the Head’s-Up Display
(HUD). Another factor influencing immersiveness that has been found in the
literature is the player’s expertise: the more experience the user has with a
specific game, the less they need information on the screen to be immersed in
the game.
Player’s level of immersion can be accessed by using both questionnaires of
their perceived experience and exploiting their behavioural and physiological
responses while playing the target game.
Therefore, in this paper, we propose an experimental protocol to access
immersiveness of gamers while playing a third-person shooter (Fortnite) with
UIs with a standard, a dietetic, and a proposed HUD. A subjective evaluation
of the immersion will be provided by completing the Immersive Experience
Questionnaire (IEQ), while objective indicators will be provided by face
tracking, behaviour and physiological responses analyses.
The ultimate goal of this study is to define guidelines for video game UI
development that can enhance the players’ immersion.

###### keywords:

Immersiveness Head’s-Up Displays Affective gaming Third-person shooter
PlayStation Fortnite

##  1 Introduction

In the gaming sector, immersion is an element considered an important
interaction experience [1]. In literature, the concept of immersion is
generally reconnected and defined through the concept of flow, settled by the
psychologist Mihály Csikszentmihalyi in 1975 [2, 3]. The author has described
flow as an _optimal experience_ , thus an experience which is achieved by
people when they enter into a state of euphoria, characterised by total
attention on what they are doing, by loss of sense of self, and by an
alteration of time perception.

While Csikszentmihalyi’s studies were performed on different types of people,
such as sportsmen and artists, lately, the concept has been translated to
other fields. In particular, considering video games, the literature presents
different opinions about the veracity overlap of the flow and immersion
concepts [1, 4], where immersion can be simply described in this case as “
_the sense of being in the game_ ”. This means that immersion can be
considered not only as a feeling of presence in the game but also as a real
cognitive statement whereby people are immersed in their activity (but not as
much as in flow) [5].

The subject of immersion in the video game context has prompted numerous
studies and projects. For example, in Sanders and Cairns’s work [6], video
game immersion is analysed considering the timing of the game session and the
use of music, while Christou [7] provides an analysis of the relationship
between immersion and appeal.

A significant aspect of this topic involves analysing this phenomenon starting
from the video game UI. The UI plays a crucial role as it is the point of
interaction between the player and the video game. It also acts as a filter
for the player’s game experience.

Among the works that focus on the study of immersion in video games by
considering their UIs, _Iacovides et al._ ’s one [8] compares the diegetic and
non-diegetic UI versions of the case study, which is _Battlefield 3_
111Electronic Arts Inc. (https://www.ea.com/en-
gb/games/battlefield/battlefield-3?setLocale=en-gb accessed September 21,
2023), and evaluates if both UIs lead to a positive game experience using
questionnaires. Afterwards, they examine the involvement experienced by
novices and experts for each of the proposed UI versions. Notice that, in this
case, for _expert_ is intended for a player using a specific game or a game
category for at least half an hour per week. The authors find that players
with expertise can be more immersed when the HUD is dietetic, i.e., when no
information is displayed.

In gaming, another aspect that can be investigated to analyse immersivity is
the players’ physiological response during the gameplay experience. As part of
the research on this topic, Lages [9] introduces the concept of _immersive
entertainment_ to identify all the entertainment applications devised for
immersive systems. The author presents eight factors influencing immersive
entertainment: (i) presence and user experience understanding, (ii) characters
and storytelling, (iii) interaction technique design, (iv) virtual environment
rendering, (v) physiological sensing and biofeedback introduction, (vi) social
experience improvement, (vii) user safeguarding and responsible design, and
(viii) content creation tool availability.

In particular, Lages points out for factor (v), on which we are particularly
interested, that there is currently (2021, year of publication) a vast use of
facial feature trackers, but that newer headsets integrating these sensors
with physiological ones, i.e., electroencephalogram (EEG), electrooculogram
(EOG), electromyogram (EMG), electrodermal activity (EDA), and
photoplethysmogram (PPG), are being developed, especially to detect users’
emotional states.

On this topic, Hughes and Jorda [10] present biosignals commonly used in
research on gaming, especially for rehabilitation purposes and/or targeting
people with disabilities. In particular, the authors find that mostly EMG,
EEG, EDA, eye tracking with EOG, and electrocardiogram (ECG) non-invasive
technologies are employed considering both user monitoring (passive mode) and
affective gaming (active mode). Considering the provided analyses, EMG is
involved in an active modality, and thus for game control or rehabilitation
exercises; EEG is efficient for emotion detection and neurological assessment
in a passive mode, while its signals can be used actively for game control and
virtual environment change; eye tracking and EOG provide information on the
user’s fatigue, the points of interest in the visualised content, and more
actively these signals can be used to control games; ECG allows stress
monitoring and the level of engagement, as well as changes of environment and
gameplay experience according to the monitored signals; these changes can be
also enabled by analysing the EDA signals, which are good indicators of
stress, panic, and other negative emotions.

Despite the extensive research conducted using physiological signals, their
implementation within the context of video gaming remains limited. Bian et al.
[11] propose a model based on the idea that flow is a state of moderate
arousal accompanied by a state of joy. For this reason, physiological
parameters related to these processes can predict the flow state. In the
proposed model, five indicators are included, i.e., facial EMG, cardiovascular
activity, EDA, respiratory activity, and EEG, accompanied by their features.
Notice that this model is also based on a previous study [12], which found the
connection between flow and heart rate variability using a driving simulator
and is applied in the context of Virtual Reality (VR).

Therefore, inspired by Iacovides et al. and Bian et al. works, we propose an
experimental protocol to evaluate immersion, considering the characteristics
of UIs in the _Fortnite_ 222Epic Games Inc. (https://store.epicgames.com/en-
US/p/fortnite?lang=en-US accessed September 19, 2023). video game, on the
level of expertise of Fortnite users as well as on their physiological
responses.

In particular, the IEQ introduced by Iacovides et al. will be used to evaluate
the perceived subjects’ level of immersion. Moreover, face tracking, EEG,
Galvanic Skin Response (GSR), PPG, and respiratory data will be collected for
a physiological evaluation of the phenomenon under analysis, following
guidelines made from Bian et al.’s physiological model.

This study aims to define guidelines for creating video game UIs that can
optimise the immersion of expert and novice players within the target video
game category, i.e., third person shooter (TPS). Thus, the research coming
from the proposed experimental protocol is intended to answer the following
research questions:

  * •

How can we discriminate different levels of immersiveness?

  * •

Do the physiological signals contribute to the interpretation of the levels of
immersiveness considering them separately or as an ensemble?

  * •

Does the players’ expertise influence the outcome, and in what measure?

  * •

Does the information present on the UI affect the players’ immersiveness?

The rest of the paper is organised as follows. Section 2 provides a brief
overview of the role of Artificial Intelligence (AI) in physiological signal
interpretation, especially in the fields of affective computing and gaming.
Section 3 and Section 4 report the experimental settings and protocols.
Finally, conclusions are drawn in Section 5.

##  2 The role of artificial intelligence

Video games elicit emotions in their players, influencing their sense of being
in the game [13]. Moreover, the player’s behaviour directly influences the
gameplay and game objectives.

Affective computing automatically recognises human emotions by studying
implicit measures derived from physiological sensors and facial expressions,
allowing close interaction between humans and machines [14, 15]. In fact,
affective models can be designed to exploit and translate the data collected
from the previously cited sensors (Section 1). This enables computer systems
to perceive and interpret human emotions and consequently provide intelligent,
empathetic, and responsive interactions [16].

Therefore, affective computing can be leveraged to dynamically change a video
game, adjusting the storyline by identifying a player’s emotional state and
actions. This particular application, which may also include game control by
means of physiological signals, is called affective gaming [17] and has as its
main aim the creation of more profound and immersive gaming experiences that
can evoke strong emotions.

A notable and recent example of this approach is Before Your
Eyes333https://www.beforeyoureyesgame.com/ accessed September 20, 2023., a
narrative-driven adventure game that takes players on an emotional journey
through the life of a character named Benjamin. Players navigate the story by
blinking their eyes, whose movements are tracked by a webcam or an eye-
tracking technology.

Besides being a powerful storytelling medium, the game becomes a means to
evoke genuine emotions. In particular, Before Your Eyes tries to elicit
empathy and understanding of Benjamin’s story in the players. This capacity
may represent a good starting point to drive social change and promote a more
compassionate society. Moreover, affective gaming can represent a potential
therapeutic tool, allowing users to explore and process their emotions.

Another important application of affective gaming can be related to the
Dynamic Difficulty Adjustment (DDA), which is a mechanism that pushes the
player to change the game’s difficulty level, considering their performance.
This operation may not be performed manually by a player during the gameplay
and could additionally demoralise the player, who may feel like is resorting
to this expedient to continue the game. Using information from different
sensors, it is possible to tune the game difficulty and maximise one of four
emotional responses: challenge, competence, flow, and valence [18].

Given these premises, affective computing and gaming will be considered in a
first phase to evaluate the players’ responses to the different UIs proposed
in the following experimental protocol, trying to assess users’ levels of
immersiveness and their perceived emotions. In a second phase, a model will be
devised to dynamically change the game UI according to the real-time perceived
immersion by the players, exploiting the outcome of the previous phase.

##  3 Experimental settings

This Section provides the information related to the experimental settings of
the proposed experimental protocol, which is divided into three main phases
(detailed in Section 4). Therefore, a brief description of each questionnaire,
physiological sensors, and gaming devices will be presented during the
research.

###  3.1 Questionnaires

The experimental design includes the implementation of a _preliminary
questionnaire_ , utilised during Phase 0 (Section 4.1), and a _pre-test_ and
an _IEQ questionnaire_ , employed during the Phases 1 (Section 4.2) and 2
(Section 4.3).

The initial survey analyses responses from expert gamers444Expert gamers are
people who play the shooter game category at least half an hour a week.,
focusing on their preferences for shooter games. The questionnaire prompts
users to identify the elements that require modification, relocation, or
exclusion, with the aim of enhancing the gaming experience.

The questionnaire comprises two parts. The first part is a demographic section
covering the participant’s age, gender, and gaming behaviour. The second part
presents various sections of the game frame and the complete game frame of the
case study, i.e., Fortnite. Participants are requested to comment on potential
improvements to individual sections that could enhance their overall gaming
experience. Figure 1 depicts an example of the game frame proposed to the
participant and its accompanying question: “Focusing on this game frame, do
you think that it is possible to modify, move, or delete some of its elements
to improve your gaming experience?”. Notice that the questionnaire is intended
for Italian speakers only, and its purpose is purely to gather feedback from
expert users, which will inform adjustments to the HUD in line with findings
from Nielsen’s [19] and Federoff’s [20] heuristics.

Figure 1: Preliminary questionnaire game frame example with accompanying
question: “Focusing on this game frame, do you think that it is possible to
modify, move, or delete some of its elements to improve your gaming
experience?”.

In Phases 1 and 2, other two questionnaires are employed. The first one is a
_pre-test_ questionnaire, which collects some demographics on the subjects and
their gaming expertise.

The second questionnaire, i.e., IEQ, surveys the subject’s video game
experience during the experimental phases. The IEQ, which was also used in the
gaming sector by _Iacovides et al._ [8], is a questionnaire that reveals the
general experience of immersivity [4]. IEQ has five dimensions, which are
analysed by the different proposed questions. In particular, these dimensions
reflect different aspects of an immersive experience, i.e., (i) cognitive
involvement, (ii) emotional involvement, (iii) real-world dissociation, (iv)
challenge, and (v) control. These five interrelated components are evoked by
31 questions, which have a 5 point-scale.

###  3.2 Face tracking and behaviour analysis

A GoPro HERO10 Black555GoPro Inc.
(https://gopro.com/en/us/shop/cameras/hero10-black/CHDHX-101-master.html
accessed September 19, 2023). is used to collect video recordings (5.3K60 +
4K120 video resolution) to be exploited for face tracking and behaviour
analysis. In fact, previous research [21] has demonstrated its ability to
provide data on detecting the participant’s facial behaviour. This feature
permits facial tracking and analyses of the user’s visual movements to
compensate for the lack of an eye-tracking device [4].

###  3.3 Physiological devices

This Section provides a brief overview of the technical characteristics of the
selected physiological devices. Figure 2 depicts the EEG (Section 3.3.1), the
GSR/PPG (Section 3.3.2), and the respiration (Section 3.3.3) devices.

Figure 2: Physiological devices that will be used to record the subjects’
physiological responses. From left to right, top to bottom: Unicorn, GSR/PPG
device, and respiration device.

####  3.3.1 Electroencephalographic device

The _Unicorn Brain Interface_ 666g.tec medical engineering GmbH
(https://www.unicorn-bi.com/ accessed August 16, 2023). (Unicorn) is used to
record the EEG signals. This interface comprises the acquisition software, the
EEG cap, a default eight electrode set (Fz, C{3, z, 4}, Pz, PO{7, 8}, and Oz
plus ground and reference sensors placed on the subjects’ mastoids), and a
Bluetooth dongle. This wearable device has the advantage of not relying on
wired connections and being characterised by hybrid sensing technologies,
i.e., the electrodes can be used with a dry or wet configuration.

Notice that all the available electrodes will be employed to capture neural
responses elicited by the game events, considering that Bian et al. suggest
the analyses on the alpha and beta frequency bands only without specifying
electrode positioning and that other works [22] show that electrodes from
different brain areas are selected when emotion recognition is involved. In
particular, Gannouni et al. [22] found that each emotion is coupled with a
specific frequency band and a specific electrode set covering the whole brain
surface.

Finally, notice that the Unicorn analog-to-digital converter has a resolution
of 24 bits and a sampling frequency of 250Hz. The Unicorn also presents 3-axis
accelerometer and gyroscope. The acceleration and gyroscope ranges are $\pm
8$g and $\pm 1000$°/s setting in x, y, and z directions. The bandwidth is
44.8Hz for the accelerometer while 41Hz for the gyroscope.

####  3.3.2 Galvanic skin response and photoplethysmogram

GSR and PPG signals are acquired with the _Shimmer3 GSR+ Unit_ 777Shimmer
(https://shimmersensing.com/product/shimmer3-gsr-unit/ accessed August 16,
2023). (Shimmer GSR). This device comprises two electrodes to monitor skin
conductivity and an optical pulse sensor to be placed on the earlobe.
Moreover, it is integrated with 3-axis accelerometer, gyroscope, magnetometer,
and altimeter. Both PPG and GSR data are acquired using a sampling frequency
of 128Hz.

As for the Unicorn, the Shimmer GSR is a wearable and wireless device based on
Bluetooth technology. The device measurement range is 10k-4.7M$\Omega$ (.2uS –
100uS) $\pm$ 10%. 22k-680k$\Omega$ (1.5-45uS) $\pm$ 3%, while the frequency
range is 15.9Hz. In the proposed experiment, the device will be placed on the
participant’s left hand with the two GSR electrodes attached respectively to
the index and pinkie fingers to reduce possible artefacts due to the use of
the joypad.

####  3.3.3 Respiration device

The _Shimmer3 EMG Unit_ 888Shimmer
(https://shimmersensing.com/product/shimmer3-emg-unit/ accessed August 16,
2023). (Shimmer EMG) records the respiration demodulation through ECG data.
The device consists of two pairs of electrodes placed on the midclavicular
lines to measure the respiration rate as charging impedance across the chest
[23]. In addition, an electrode is placed on the right side of the sternum as
a reference. A sampling frequency of 128 Hz is used to acquire the signals.

Notice that Shimmer3 EMG is also a non-invasive device that allows monitoring
and collecting data through Bluetooth wireless technology.

###  3.4 Gaming device and selected game

PlayStation®5999Sony Interactive Entertainment
(https://www.playstation.com/en-gb/ps5/?smcid=pdc%3Aen-
gb%3Aprimary%20nav%3Amsg-hardware%3Aps5 accessed September 19, 2023). has been
selected as the gaming device with its DualSense™wireless controller.

The selected TPS game is _Fortnite_. The authors of this paper selected this
game for two main reasons. Firstly, _Fortnite_ is a famous and appreciated
game, having more than 350 million registered users by the end of 2022101010
_everyeye.it_ (https://www.everyeye.it/notizie/giocatori-fortnite-totale-
risposta-612779.html accessed September 21, 2023).. Secondly, the HUD has a
very important role in the game. In fact, _Celia Hodent_ and her team devised
the HUD to be informative and non-intrusive, considering the whole user
experience. According to _Hodent_ 111111Celia Hodent
(https://celiahodent.com/understanding-the-success-of-fortnite-ux/ accessed
September 21, 2023)., information is always available to the players avoiding
taxing them with learning and memorising everything, thus avoiding an
excessive cognitive load.

##  4 Experimental Protocol

The experimental protocol is divided into three main phases of interface
definition, assessment of the immersiveness with expert players, and
assessment of usability with non-expert players.

In fact, Phase 0 is devoted to the definition of a HUD (from now on proposed
HUD) in-between the standard Fortnite UI and the one that does not report any
information on the screen to enhance the immersion of the player while
avoiding removing useful information to complete the game session.

Before briefly introducing the other phases, notice that the three UIs that
will be used in the experiment can be summarised as follows:

  * •

The UI with Fortnite standard HUD;

  * •

The UI without any information, thus without the HUD;

  * •

The UI created using gamers’ opinion and the heuristics present in the
literature [19, 20], called proposed HUD.

Notice that the first two interfaces are considered following the example of
_Iacovides et al._ [8]. In fact, _Iacovides et al._ compare these two types of
interfaces and find that expert users were more immersed when using the UI
without a HUD, and thus without information that would distract them. This
observation did not apply to non-expert players, who did not obtain any
increase in immersion probably due to their inexperience in the target game,
i.e., _Battlefield 3_.

Proceeding with the subsequent phases, Phase 1 is intended for expert Fortnite
players, i.e., Fortnite users that have played the game for at least two years
and one hour per week. This phase will assess the effectiveness of the
proposed HUD resulting from Phase 0, comparing it to the case where the
participants play the game using the non-diegetic Fortnite HUD. Instead, Phase
2 will evaluate the usability of the proposed HUD by non-expert Fortnite
players, i.e., participants who know how to use a PlayStation console but have
never played Fortnite. This is to understand how expertise influences the
definition of the HUD and how the HUD adapts to different users. Moreover,
immersiveness will also be accessed for non-expert players.

Physiological responses will be collected to evaluate the best interface type
for immersiveness optimisation considering the users’ experience while playing
the game.

Table 1 summarises the characteristics of each phase and provides information
on the participants’ inclusion criteria, the questionnaires that will be
provided, and the physiological data that will be recorded.

As a final remark, notice that players are left free to choose which hero to
play during the experiment, as outfits do not guarantee competitive advantages
within the game [24].

Table 1: Summary of the three phases of the proposed experimental protocol. Phase |  Description |  Inclusion criteria |  Methods |  Indicators
---|---|---|---|---
0 |  Interface definition |  Participant plays a TPS for at least 30 min per week |  Preliminary questionnaire plus Nielsen and Federoff’s heuristics |  none
1 |  Immersiveness assessment with expert participants |  Fortnite expert player aged 18-35 y.o., with a mimum of 2 years experience in Fortnite and playing to it at least 1 hour per week |  pre-test questionnaire and IEQ |  EEG, GSR, PPG, respiration, facial expression
2 |  Usability and immersiveness assessment with non-expert participants |  Fortnite non-expert player aged 18-35 y.o., using a PlayStation console with a joypad |  pre-test questionnaire and IEQ |  EEG, GSR, PPG, respiration, facial expression

###  4.1 Phase 0: interface definition

The aim of Phase 0 is to formulate a revised _Fornite_ interface with HUD
based on the responses obtained from the preliminary questionnaire detailed in
Section 3.1. The modifications will thus respond to the preferences of
experienced users and the requirements imposed by Nielsen [19] and Federoff’s
[20] heuristics. The proposed HUD will later be presented to expert and non-
expert players in Phase 1 and 2, respectively.

####  4.1.1 Inclusion and exclusion criteria

Only expert players will be involved in this definition phase. In this phase,
for _expert_ is intended for a user who plays shooter games for at least 30
minutes per week. Notice that in this phase, the user is not necessarily an
expert in Fortnite.

####  4.1.2 Materials and methods

The materials and methods used in this phase are:

  * •

The 10 usability heuristics of Nielsen [19];

  * •

The heuristics for the gaming sector identified and analysed by Federoff [20];

  * •

A preliminary questionnaire for the expert gamers121212Available at
https://docs.google.com/forms/d/e/1FAIpQLSfvCgCW9XxWfKekU5WOlxmAUwVEPqt-
oPBktEdfolcWVNdG4g/viewform?usp=sf_link.

_Unreal Engine_ ®131313Epic Games Inc. (https://www.unrealengine.com/en-US
accessed September 21, 2023). will be used to create the proposed HUD. Notice
that this tool is the same developed and used by _Epic Games_ , _Fortnite_
developer.

###  4.2 Phase 1: immersiveness with expert players

The main aim of Phase 1 is to evaluate the immersion of the proposed HUD
obtained in Phase 0 with respect to the case study dietetic HUD version.

In fact, expert _Fortnite_ players will have to play a game session with the
UI (i) without HUD and (ii) with the proposed HUD.

The experiment is conducted in a within-subject condition through two gaming
sessions carried out randomly by expert players.

So, it is a (2x1) experiment with _p_ =0.8 (the same as _Iacovides et al._
[8]), and the number of expert players needed is 34.

####  4.2.1 Inclusion and exclusion criteria

Expert users aged between 18 and 35 who have played Fortnite for at least two
years and play it an hour per week will be involved in Phase 1. Expert users
who have played shooter games but not Fortnite will be excluded.

####  4.2.2 Materials and methods

In Phase 1, a pre-test questionnaire141414Available at
https://docs.google.com/forms/d/e/1FAIpQLSeKRO65yJVSSzps5XrWL9PXB2YDJqg-9ZmuXR7UKzpc3P_4ng/viewform?usp=sf_link
and an IEQ151515provided by _Jennet et al._ [4] at
https://docs.google.com/forms/d/e/1FAIpQLSeRMgdk4_IMxZ-
LFcrj9le8bjRaclUHq1G80Y1t_OWoHKGmmw/viewform?usp=sf_link. after each game
sessions will be presented to the participants.

As for _Iacovides et al._ [8], participant expertise and the number of
interfaces (two) will be considered to understand the immersion level of the
expert during the game sessions.

Face tracking and player behaviour will be derived from the videos recorded by
means of _GoPro HERO10 Black_. Moreover, the following devices will be used to
exploit different physiological indicators, as suggested by _Bian et al._ ’s
framework [11]: Unicorn, Shimmer GSR, Shimmer EMG. Section 3.3 reports more
details on the devices.

###  4.3 Phase 2: usability with non-expert players

The main aim of Phase 2 is to evaluate the usability of the proposed HUD based
on the feedback of expert shooter gamers obtained in Phase 0 with respect to
the classic interface of the case study.

In fact, non-expert Fortnite players will have to play a game session with the
UI (i) with the standard HUD, and (ii) with the proposed HUD.

The experiment is conducted in a within-subject condition through two random
gaming sessions by non-expert players.

So, it is a (2x1) experiment with _p_ =0.8 (the same as _Iacovides et al._
[8]), and the number of non-expert players needed is 34.

Besides accessing the proposed HUD usability, the player level of immersion
will also be evaluated.

####  4.3.1 Inclusion and exclusion criteria

Non-expert users who have not played Fortnite aged between 18 and 35 will
participate in this experiment phase. These non-expert users are selected
among players who know how to play with a PlayStation console.

####  4.3.2 Materials and methods

As for Phase 1, the pre-test questionnaire and IEQ will be presented to the
participants. Moreover, participant expertise and the number of interfaces
(two) will be considered to understand the usability of the proposed HUD as
well as the immersion level of the non-expert player during the game sessions.

The same devices reported for Phase 1 will be used: _GoPro HERO10 Black_ for
face tracking and behaviour analysis, and Unicorn, Shimmer GSR, and Shimmer
EMG for physiological responses recording.

##  5 Conclusions

In this paper, a brief overview of the experimental protocol intended to
assess the immersiveness of expert and non-expert participants while playing a
third-person shooter is presented.

The experiment will be carried out by following literature works [8, 11] that
successfully assessed players’ immersion levels considering different HUDs and
physiological responses. Moreover, a HUD for the Fortnite UI will be proposed
based on expert shooter players’ opinions to enhance the gamers’
immersiveness.

Affective computing and gaming will be exploited to clearly understand users’
experience, behaviour, and levels of immersion by leveraging the information
captured by physiological sensors.

In future works, these AI applications will be further introduced to provide a
dynamic mechanism that can change the HUD in real-time depending on the
single-player responses.

AI

    Artificial Intelligence
DDA

    Dynamic Difficulty Adjustment
ECG

    electrocardiogram
EDA

    electrodermal activity
EEG

    electroencephalogram
EOG

    electrooculogram
EMG

    electromyogram
GSR

    Galvanic Skin Response
HUD

    Head’s-Up Display
IEQ

    Immersive Experience Questionnaire
PPG

    photoplethysmogram
TPS

    third person shooter
UI

    User Interface
VR

    Virtual Reality

## References

  * E Brown [2004] P. C. E Brown,  A grounded investigation of game immersion,  Extended Abstracts of Human Factors in Computing Systems (2004).
  * Csikszentmihalyi [1975] M. Csikszentmihalyi,  Beyond boredom and anxiety. san francisco,  CA, US: Jossey-Bass (1975).
  * Csikszentmihalyi [1990] M. Csikszentmihalyi, Flow: The Psychology of Optimal Experience, 1990.
  * C Jennet [2008] . A. W. C Jennet, A L Cox,  Measuring and defining the experience of immersion in games,  International Journal of Human-Computer Studies 66 (2008) 641–661.
  * M Taylor [2017] A. C. M Taylor,  Augmenting the hud: A mixed methods analysis on the impact of extending the game ui beyond the screen (2017).
  * Sanders T [2010] C. P. Sanders T,  Time perception, immersion and music in videogames,  Proceedings of HCI 2010 (HCI) (2010).
  * Christou [2014] G. Christou,  The interplay between immersion and appeal in video games,  Computers in human behavior 32 (2014) 92–100.
  * Iacovides et al. [2015] I. Iacovides, A. Cox, R. Kennedy, P. Cairns, C. Jennett,  Removing the hud: the impact of non-diegetic game elements and expertise on player involvement,  in: Proceedings of the 2015 Annual Symposium on Computer-Human Interaction in Play, 2015, pp. 13–22.
  * Lages [2021] W. S. Lages,  Opportunities and challenges in immersive entertainment,  Anais Estendidos do XX Simpósio Brasileiro de Jogos e Entretenimento Digital (2021) 1037–1040.
  * Hughes and Jorda [2021] A. Hughes, S. Jorda,  Applications of biological and physiological signals in commercial video gaming and game research: a review,  Frontiers in Computer Science 3 (2021) 557608.
  * Bian et al. [2016] Y. Bian, C. Yang, F. Gao, H. Li, S. Zhou, H. Li, X. Sun, X. Meng,  A framework for physiological indicators of flow in vr games: construction and preliminary evaluation,  Personal and Ubiquitous Computing 20 (2016) 821–832.
  * Tozman et al. [2015] T. Tozman, E. S. Magdas, H. G. MacDougall, R. Vollmeyer,  Understanding the psychophysiology of flow: A driving simulator experiment to investigate the relationship between flow and heart rate variability,  Computers in Human Behavior 52 (2015) 408–418.
  * Setiono et al. [2021] D. Setiono, D. Saputra, K. Putra, J. V. Moniaga, A. Chowanda,  Enhancing player experience in game with affective computing,  Procedia Computer Science 179 (2021) 781–788.
  * Picard [1999] R. W. Picard,  Affective computing for hci.,  in: HCI (1), Citeseer, 1999, pp. 829–833.
  * Marín-Morales et al. [2020] J. Marín-Morales, C. Llinares, J. Guixeres, M. Alcañiz,  Emotion recognition in immersive virtual reality: From statistics to affective computing,  Sensors 20 (2020) 5163.
  * Tao et al. [2006] J. Tao, T. Tan, R. Picard, Affective computing and intelligent interaction, volume 3784, Springer, 2006.
  * Kotsia et al. [2013] I. Kotsia, S. Zafeiriou, S. Fotopoulos,  Affective gaming: A comprehensive survey,  in: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, 2013, pp. 663–670.
  * Moon et al. [2022] J. Moon, Y. Choi, T. Park, J. Choi, J.-H. Hong, K.-J. Kim,  Diversifying dynamic difficulty adjustment agent by integrating player state models into monte-carlo tree search,  Expert Systems with Applications 205 (2022) 117677.
  * NielsenNorman-Group [2020] NielsenNorman-Group,  10 usability heuristics for user interface design (2020).
  * Federoff [2002] M. A. Federoff,  Heuristics and usability guidelines for the creation and evaluation of fun in video games (2002).
  * Wang et al. [2016] J. Wang, Y. Cheng, R. S. Feris,  Walk and learn: Facial attribute representation learning from egocentric video and contextual data,  in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  * Gannouni et al. [2021] S. Gannouni, A. Aledaily, K. Belwafi, H. Aboalsamh,  Emotion detection using electroencephalography signals and a zero-time windowing-based epoch estimation and relevant electrode identification,  Scientific Reports 11 (2021) 7071.
  * Redmond [2013] C. Redmond,  Trans-thoracic impedance measurements in patient monitoring,  EDN Network (2013).
  * Schöber and Stadtmann [2020] T. Schöber, G. Stadtmann,  Fortnite: The business model pattern behind the scene,  European University Viadrina Frankfurt (Oder) Department of Business Administration and Economics Discussion Paper (2020).

  *[UI]: User Interface
  *[HUD]: Head’s-Up Display
  *[IEQ]: Immersive Experience Questionnaire
  *[EEG]: electroencephalogram
  *[EOG]: electrooculogram
  *[EMG]: electromyogram
  *[EDA]: electrodermal activity
  *[PPG]: photoplethysmogram
  *[ECG]: electrocardiogram
  *[VR]: Virtual Reality
  *[GSR]: Galvanic Skin Response
  *[TPS]: third person shooter
  *[AI]: Artificial Intelligence
  *[DDA]: Dynamic Difficulty Adjustment