# SCORPION Cyber Range: Fully Customizable Cyberexercises, Gamification and
Learning Analytics to Train Cybersecurity Competencies

Pantaleone Nespoli, Mariano Albaladejo-González, José Antonio Pastor Valera,
José A. Ruipérez-Valiente, Joaquin Garcia-Alfaro, Félix Gómez Mármol The
authors are with the Department of Information and Communications Engineering,
University of Murcia, 30100 Murcia, Spain, and SAMOVAR, Télécom SudParis,
Institut Polytechnique de Paris, 91120 Palaiseau, France
<EMAIL_ADDRESS>mariano.albaladejog, japv, jruiperez,
<EMAIL_ADDRESS><EMAIL_ADDRESS>(Corresponding
author<EMAIL_ADDRESS>
###### Abstract

It is undeniable that we are witnessing an unprecedented digital revolution.
However, recent years have been characterized by the explosion of
cyberattacks, making cybercrime one of the most profitable businesses on the
planet. That is why training in cybersecurity is increasingly essential to
protect the assets of cyberspace. One of the most vital tools to train
cybersecurity competencies is the Cyber Range, a virtualized environment that
simulates realistic networks. The paper at hand introduces SCORPION, a fully
functional and virtualized Cyber Range, which manages the authoring and
automated deployment of scenarios. In addition, SCORPION includes several
elements to improve student motivation, such as a gamification system with
medals, points, or rankings, among other elements. Such a gamification system
includes an adaptive learning module that is able to adapt the cyberexercise
based on the users’ performance. Moreover, SCORPION leverages learning
analytics that collects and processes telemetric and biometric user data,
including heart rate through a smartwatch, which is available through a
dashboard for instructors. Finally, we developed a case study where SCORPION
obtained 82.10% in usability and 4.57 out of 5 in usefulness from the
viewpoint of a student and an instructor. The positive evaluation results are
promising, indicating that SCORPION can become an effective, motivating, and
advanced cybersecurity training tool to help fill current gaps in this
context.

Cyber Range, cybersecurity, gamification, educational technology

##  1 Introduction

Information and Communication Technologies (ICT) are significantly impacting
our daily lives, substantially improving our quality of life through access to
a multitude of online services and the use of products that use these
technologies. Therefore, it should be no surprise that individuals in our
society are becoming increasingly dependent on technology. Moreover, the new
advancements of Beyond 5G and 6G platforms are going to further drive this
digital revolution. In fact, thanks to these platforms, connectivity is
capable of reaching a speed never seen before with incredibly low latency.
Moreover, the rise of new disruptive technologies (e.g., Blockchain [1, 2])
and innovative paradigms (e.g., Internet of Things (IoT) [3]) have paved the
way for significant contributions from both industry and academia.

Nonetheless, the digital revolution also has a negative impact. In fact, these
technologies are also becoming one of the main targets of actors with
malignant purposes who attack them to cause operational damage and/or extract
economic benefits from the companies or users that use them [4]. That is, the
number of cyber attacks on ICTs has continued to grow dramatically in recent
years, making cybercrime one of the most profitable businesses for them.
Moreover, such attacks are becoming increasingly sophisticated, targeting
virtually any economic sector with a digitized infrastructure [5, 6].
Therefore, academia, as well as the public and private sectors, are actively
working to build robust and efficient security plans to deal with cyber
threats. This demand is also being met by specific educational programs in
cybersecurity matters and companies dedicated to training in these aspects
[7].

In this alarming scenario, it is easy to understand the importance of
equipping professionals from all fields with the cybersecurity and cyber
defense skills needed to defend assets belonging to cyberspace against
potential cyber threats. That is why Cyber Ranges represent one of the most
vital tools today to provide quality training for professionals with
particular and applied cybersecurity skills [8]. The National Institute of
Standards and Technology (NIST) defines Cyber Ranges as “interactive,
simulated representations of an organization’s local network, system, tools
and applications” [9]. In this sense, it is easy to think of Cyber Ranges as
virtualized platforms for training expert ICT infrastructure professionals
(e.g., system administrators) against cyber-attacks, to avoid possible
collateral damage as part of such training in real infrastructures.

Due to the above, in recent years, the profiles of potential users have been
diversifying, substantially expanding the range of environments appearing and
how to use them. This means that Cyber Ranges are used by different groups
such as students, militaries, ICT security professionals, government
employees, or professionals of Small and Medium Enterprises (SMEs), among
others. However, they share a common characteristic: the use of realistic
simulations of cybersecurity and cyber defense scenarios, thus learning to
make key decisions and defend real systems.

Nevertheless, training in these areas still presents several challenges. For
example, current Cyber Ranges present difficulty in providing realistic
scenarios for training or the impossibility of offering on-demand services
using virtual infrastructures with fundamental characteristics such as
reusability, adaptability and scalability. From a more educational
perspective, one of the major complications of current Cyber Ranges is the
loss of motivation and sense of boredom on the part of system users [10]. This
problem can be verified for various reasons, such as low interest in the
contents taught, apathy towards the subject if it is too easy, or, on the
contrary, inability to understand the content or complete exercises due to its
high difficulty, generating frustration.

Intending to solve the aforementioned challenges, this paper presents
SCORPION, a fully functional and virtualized Cyber Range for training in
cybersecurity and cyber defense capabilities. Based on a set of parameters,
our proposal is able to generate a series of random scenarios to perform
motivating and realistic cyber maneuvers. Specifically, SCORPION leverages a
scenario authoring tool that allows creating a series of dynamic and
parametrizable scenarios with random variables, obtaining greater flexibility
in providing cyberexercises to system users [11].

Moreover, another differentiating feature of our proposal is the use of a
learning analytics system, which leverages telemetric data traces and
biometric signals generated by the system users while solving the challenges
present in cyber maneuvers [12]. In this way, it is possible to assess key
competencies in dual environments (i.e., civilian/military), not only related
to cyber defense and cybersecurity content but also transversal skills such as
the ability to work under pressure.

In addition, SCORPION presents a powerful gamification module composed of a
vast number of components with the objective of improving users’ motivation in
solving the proposed challenges [13]. In particular, the cyberexercises
conducted in such scenarios are dynamic and adaptive to the learner, as
opposed to the static ones offered by current Cyber Ranges, moving education
in a personalized direction to the user’s needs.

To demonstrate its capabilities, SCORPION is evaluated through a case study to
validate its usability, usefulness, and utility regarding the 3 abovementioned
novelties (i.e., scenario authoring tool, learning analytics and gamification
module) from students’ and instructors’ perspectives. Specifically, a video
demonstration and a questionnaire are presented to a sample of teachers and
students from the Computer Science faculty at the University of Murcia. It is
worth mentioning that this sample has been selected since they represent
likely users of the Cyber Range platform.

The remainder of this article is as follows: Section 2 presents a review of
the main works related to Cyber Ranges. Then, Section 3 describes the
architecture of SCORPION, detailing its components. Section 4 presents a
number of use cases related to the proposed system. Section 5 exposes the
results of the evaluation of SCORPION from usability, accessibility and
utility viewpoints. Then, Section 6 provides an in-depth discussion of the
need for using the framework and its potential social impact. Finally, Section
7 concludes the article, proposing interesting future directions.

##  2 Related work

Recently, the demand for cybersecurity and cyber defense experts has risen
sharply in both the public and private sectors, and the trend is expected to
continue to grow over the next few years. Consequently, several Cyber Ranges
proposals have appeared in response to this demand, trying to cover different
needs. The papers in [14, 8, 15] review the state of the art with a focus on
Cyber Ranges, analyzing various characteristics and different perspectives,
including advantages, potential drawbacks, and possible future avenues.

At the international level, Cyber Range platforms are quite diffuse, both in
the private and public sectors. For example, Masaryk University (Czech
Republic) has been developing the KYPO Cyber Range platform since 2013 [16].
That platform is based on several years of experience in the use of cyberspace
in education, training and cyber defense exercises, including the Czech
technical cybersecurity exercises, the Cyber Czechs, which were organized in
collaboration with the Czech National Agency for Cybersecurity and Information
Security (NCISA). The platform has already been used for teaching students in
several courses at Masaryk University and for training cybersecurity
professionals in the energy sector. A very relevant feature of KYPO is that it
is a free and open software Cyber Range, which has allowed other institutions
to deploy this solution in their own facilities [17].

From an industrial viewpoint, Cyber Ranges have been proposed as training
platforms to train offensive and/or defensive practitioners’ skills by paying
a certain amount. In this sense, such partitioners are the final users of the
platform, being able to execute the prepared cyberexercises. One notorious
example is the Indra Cyber Range
Platform111https://cyberrange.indracompany.com/ (ICR) [18], which enables
agencies, public and private, to instill, assess and exercise Knowledge,
Skills, and Abilities (KSA) to professionals who need to integrate
cybersecurity in various contexts as part of their professional role, either
as a complementary or as a primary function. Another prominent platform is the
Airbus Cyber Range222https://www.cyber.airbus.com/, which has been developed
by Airbus [19]. Similar to the aforementioned cyber range, the Airbus Cyber
Range relies on virtualization technologies as the foundation for constructing
scenarios that incorporate physical networks and Operational Technology (OT)
components. Additionally, the Leonardo Cyber
Range333https://cybersecurity.leonardo.com/en/security-cyber/cyber-digital-
solutions/cyber-training is a versatile operational environment that strives
to generate authentic operational training scenarios utilizing state-of-the-
art technologies such as Infrastructure-as-Code provisioning, cloud
management, and software-defined networking [20]. Its primary objective is to
enable government agencies and critical infrastructure cybersecurity teams to
cope with complicated cyber threats and attacks targeting both Information
Technology (IT) and OT. Additionally, the platform enables the assessment of
new attack and defense techniques, verification of infrastructure management
protocols, and evaluation of protective measures for technological systems and
incident response methods.

With the goal of “democratizing cybersecurity training”, the authors in [21]
present CyTrONE, a cybersecurity training system that facilitates training
activities by offering an open source ecosystem444https://github.com/crond-
jaist/cytrone capable of automating the content generation and configuration
tasks. The authors claim that CyTrONE has three main advantages: i) improving
the accuracy of training configuration, ii) decreasing the overall
configuration time and cost, and iii) making training instantiated for many
users. In addition, the system is evaluated from two fundamental perspectives
(functionalities and performance), demonstrating its capabilities for
effective training.

Likewise, the Cyber Range Alpaca is proposed in [22]. In particular, this open
source framework 555https://github.com/StetsonMathCS/alpaca aims to build
training according to user-specified constraints. Unlike other Cyber Ranges,
Alpaca relies on a vulnerability database and a planning engine to simulate
exploits sequences that allow an attacker to reach a specific target. All
paths that meet user-specified constraints (e.g., the minimum number of steps
or use of a particular vulnerability) are found by Alpaca and collected in a
vulnerability network. This network shows all possible ways to exploit the
system. Once the lattice is found, a Cyber Range is built through automatic
scripting to instantiate a virtual machine containing all the vulnerabilities
that make up the lattice. Then, the authors present some initial use cases as
well as a first evaluation of the tool, specifying many future works with the
feeling that Alpaca is still in the development phase.

In addition, the Cyber Range THREAT-ARREST is described in [23]. As part of a
European project, THREAT-ARREST aims to provide an advanced training platform
that includes emulation, simulation, serious gaming and visualization
capabilities to adequately prepare users with different knowledge and
backgrounds to deal with both known and new cyber-attacks. Moreover, such a
platform adopts a model-based approach, in which cyber-attack preparedness and
training models are used to specify potential attacks, security controls to be
launched and tools that can be used. By doing so, the system provides the
opportunity to plan and drive the training process integrated with security
assurance mechanisms to ensure the relevance of the training.

Furthermore, the authors in [24] present AIT Cyber Range, which features a
flexible and scalable architecture developed with open-source technologies
(i.e., OpenStack, Terraform, and Ansible). Concretely, AIT Cyber Range
consists of four main components, that is, i) computing platform, ii)
infrastructure provisioning, iii) software provisioning, and iv) scenario
engine. Then, three cyberexercises categories are summarized (i.e., intra-
organizational, national, and international cyberexercises) with several teams
participating and different objectives to achieve. Even if the planning is
promising, one could say that the AIT Cyber Range is still in an early
development phase.

Moreover, the ICSTASY Cyber Range is described in [25]. Developed with the
Korean Agency for Defense Development, ICSTASY mainly focuses on training for
military users, even if the authors claim that it can support dual use.
Specifically, its system design focuses on four pillars, that is, i) the use
of template scenarios, ii) autonomous blue-red agents [26, 27], iii)
instructors’ visibility on the training sessions, and iv) automated post-
cyberexercise evaluation. Then, the authors show several graphical results of
the development for the proposed cyberexercise phases (i.e., preparation,
implementation, and evaluation), adding quite an extensive list of MITRE-
related attack techniques for their red-team agents.

Related work |  Open-source technologies |  Full automation |  Red/Blue support |  Random scenarios generation |  Cyberexercise adaptability |  Gamification module |  Learning analytics
---|---|---|---|---|---|---|---
Vykopal et al. [16] |  ✓ |  ✓ |  ✗ |  ✗ |  ✗ |  ✗ |  ✗
Beuran et al. [21] |  ✓ |  ✓ |  ✗ |  ✗ |  ✗ |  ✗ |  ✗
Eckroth et al. [22] |  ✓ |  $\approx$ |  ✗ |  ✗ |  ✗ |  ✗ |  ✗
Hatzivasilis et al. [23] |  ✗ |  ✓ |  $\approx$ |  ✗ |  $\approx$ |  ✓ |  ✗
Leitner et al. [24] |  ✓ |  ✓ |  $\approx$ |  ✗ |  ✗ |  ✗ |  ✗
Lee et al. [25] |  ✗ |  ✓ |  ✓ |  $\approx$ |  ✗ |  ✗ |  ✗
Our proposal |  $\approx$ |  ✓ |  ✗ |  ✓ |  ✓ |  ✓ |  ✓

Legend: ✓ Yes – ✗ No – $\approx$ Partially Table 1: Comparison of the related
works highlighting the main features. Commercial solutions (e.g., ICR, Airbus
and Leonardo Cyber Ranges) are excluded from the table, but mentioned in the
paper.

The projects mentioned above undoubtedly represent an essential step towards
the possibility of offering real training in cybersecurity and cyberdefense
for different categories of end users. However, some challenges still need to
be answered, hindering several potential advances, as summarized in Table 1.
In order to solve such challenges, our proposal aims to innovate concerning
previous Cyber Ranges solutions on the generation of training scenarios. In
particular, SCORPION provides a powerful scenario authoring tool that can
output parameterizable topologies using random variables, allowing for greater
flexibility and, consequently, positive repercussions on the part of users,
both instructors and students. Firstly, instructors are free to create more
flexible scenarios that are always different for each run, preventing it from
being repetitive. Secondly, from the students’ viewpoint, the randomization of
scenarios gives them access to a greater variety of challenges that will keep
them motivated, so they can practice similar skills in slightly different
simulations to lead to improved learning outcomes.

Additionally, the trainings (or cyberexercises) in the Cyber Range have
traditionally been static, with a series of parameters and/or events coded
within specific timelines, with no possibility of actually adapting to the
student’s capabilities. This limitation carries negative implications, i.e.,
it can generate boredom or frustration on the part of the learner if the
cyberexercise posed is too easy or too difficult, respectively. In this
aspect, our solution improves the state-of-the-art proposals by leveraging a
powerful gamification module to improve students’ motivation in solving the
faced cyberexercise. Specifically, several gamification elements are
introduced, including a component that is able to implement an adaptive system
that can adapt the resources available to students in training based on the
student’s performance in previously completed challenges. Thus, if the student
solves them effortlessly, the available resources will be decreased, and vice-
versa.

Last but not least, SCORPION features a learning analytics component that uses
telemetric traces generated by students and biometric signals captured by
smart wristbands. In this case, we generate specific performance metrics with
respect to the competencies acquired in the completed challenges, as well as
the ability to monitor the student’s performance in real time through a user-
friendly visualization dashboard.

##  3 Architecture

The architecture of SCORPION contemplates a division of users by roles, in our
case, into students, instructors, and administrators. The students are the
ones who carry out the cyberexercises, generating different telemetries during
the cyberexercise, and additionally, the biometric data. Besides, the students
can belong to different groups allowing participation as a team in
collaborative cyberexercises. On the other hand, the instructors are in charge
of designing and activating the cyberexercises and analyzing the different
learning analytics collected during them. Finally, an administrator role
focuses mainly on maintaining and managing the Cyber Range. As previously
mentioned, the SCORPION architecture stands out from the rest of the
aforementioned Cyber Ranges due to the incorporation of a scenarios authoring
tool for the instructor, able to output randomized and parameterizable network
topologies, a gamification system, including an adaptive cyberexercise module,
and a learning analytics component for processing and monitoring telemetric
and biometric data of the students (containing a stress detection module), as
shown in Figure 1. For this section onwards, we use the term cyberexercise
instance as one activation and execution of a cyberexercise with one or more
students.

Figure 1: Abstract view of SCORPION’s architecture

###  3.1 Front-end, software components and interconnections

To interact with the proposed framework, users connect to the front-end. This
application comprises several components or microservices, each in a Docker
container where they are executed independently, as shown in Figure 2. These
components are:

  * •

Nginx: this component acts as a reverse proxy to redirect requests to the
corresponding endpoint, allowing us to take advantage of the security and
performance features of the service. End-user requests are intercepted by this
component and, depending on the destination (/ or /ws), are sent to Gunicorn
or Daphne, respectively. In addition, it is also responsible for serving
static content (such as HTML templates or JS scripts) and multimedia (device
images, user avatars, etc.).

  * •

Gunicorn: it is in charge of serving the HTTP/HTTPS requests made by the users
of the platform.

  * •

Daphne: it serves WS/WSS requests addressed to WebSockets, since these always
start with the prefix “/ws”.

  * •

PostgreSQL: it is responsible for storing all the information necessary for
the operation of the application, such as users, scenario definitions,
challenges and cyberexercises, services, and templates, among others.

  * •

Redis: together with Django Channels and WebSockets, they are responsible for
providing the application with the ability to get real-time, asynchronous
updates without the need for the end user to refresh the browser page.

  * •

Celery: this component is in charge of processing and executing asynchronous
tasks in the background, such as deploying scenarios or releasing the
resources associated with a scenario when it is deleted. Besides, it works in
combination with Celery Beat, which is a scheduler for background periodic
tasks within the framework.

Figure 2: Architecture of SCORPION

###  3.2 Scenarios authoring tool

Figure 3: Component diagram of the random scenarios generator

Compared to the works analyzed in Section 2, the scenario authoring tool
represents the first differentiating factor. Particularly, the main objective
of this component is twofold. On the one side, it can generate random network
topologies at each iteration based on the internal logic described in this
section. Those ready-to-use topologies contain nodes fully equipped with
services to run cyberexercises (e.g., vulnerabilities, misconfigurations,
offensive tools, etc.). On the other side, it allows the platform instructors
to create an ad-hoc scenario starting from scratch.

The generation of network scenarios is carried out thanks to an additional
platform component, which has its own internal models, independent from the
rest, to be easily extensible and maintainable. The entire generation process
is depicted in Figure 3. Specifically, this component takes as input two
parameters: the environment and the complexity of the scenario to be
generated. The first refers to the environment whose network will be created,
e.g., smart home, industrial control system, etc. The latter, instead, refers
to the density of the generated network, e.g., the number of nodes, services,
the interconnections, etc. From these, the corresponding generation rule is
retrieved and, following the internal logic of the element, a JSON is
generated as output with the information of the specific scenario generated.

The internal logic of the generator consists of two loops, that is, one for
network entities (e.g., routers, switches, etc.) and one for final entities
(e.g., laptops, servers, etc.). The first loop randomly places the network
entities in the scenario topology. In particular, it creates a new network
entity and checks if any other network entity has already been generated. If
so, one of these previously created network entities is randomly taken and
connected to the one just created. This connection will be direct or indirect
(i.e., there will be a final entity between them) depending on the probability
of a direct connection between the two entities. To decide this, a random
number between 0 and 1 is generated. If this number is less than the
aforementioned probability, both network entities (the one just created and
the one randomly taken from those already created) will be directly connected.
Otherwise, if this random number is equal to or greater than such probability,
a new final entity is created that will act as a “bridge” between both network
entities since it will be connected to both of them. On the other hand, within
the second loop, only one new final entity has to be created in each iteration
and connected to a network entity randomly selected from those previously
created in the first loop.

Once all the necessary entities have been created, the generator proceeds to
perform the IP addressing of each of these entities, assigning to each network
entity a range of addresses and to each final entity connected to that network
entity a unique address within that subnet. Finally, the generator calls an
auxiliary function that is responsible, from all the previously generated
information (entities, IP addressing, etc.), to generate two JSON with the
nodes and links present in the topology, which are used in the front-end to
draw the topology for the end user.

Additionally, the JSON file is then parsed to generate the corresponding
Terraform file. Terraform is an infrastructure-as-code tool defined by
HashiCorp. In particular, it allows one to define resources and infrastructure
in declarative and readable configuration files, and it manages the lifecycle
of its infrastructure. In this sense, the generated topology definition is
portable among different virtualization providers, just changing a few labels.

Using the logic described above, the random scenario generator is able to
build models that represent realistic topologies of network entities and
endpoints that will be different at each iteration and modifiable to be
adapted to particular needs.

###  3.3 Learning analytics system

Another novelty feature of SCORPION is the module for learning analytics
processing. No other Cyber Range in the state of the art includes learning
analytics, which makes it impossible to assess and monitor students’
knowledge. Our learning analytics system includes two main data sources.
First, we collect trace data from students when interacting with the
cyberexercises, for example, the hints utilized, solutions attempts, or time
needed. SCORPION employs these trace data to generate learning analytics and
visualize them through different graphics. For each cyberexercise instance and
challenge, SCORPION shows the instructor the time needed, the points obtained,
the penalizations, the hint attempts, and the hint requests. The students also
have four summary learning analytics in their profiles, all computed by each
type of competence of the challenges and globally. These four analytics are
the total of points obtained in all cyberexercises, the total time in
cyberexercises, the proportion of points obtained in cyberexercises, and the
proportion of remaining time in cyberexercises.

Moreover, we also enable the reception of multimodal signals with the RESTful
API. The API is generic and allows different smart wristbands to send signals
from electrocardiograms, photoplethysmography (PPG), inter-beat distance, or
heart rate via Wi-Fi, and the stress detection module could use any of them.
The stress detection module divides these signals into windows of constant
size. For each window, this module extracts different heart rate features,
which are introduced into an anomaly detection model that classifies each of
the windows into stress and non-stress. Finally, SCORPION computes the
percentage of the stress windows of each challenge and shows the heart rate
and the stress percentage of the stress to the instructor in the live progress
graph in real-time.

###  3.4 Adaptive cyberexercises with gamification

The gamification and adaptive systems receive the learning analytics of the
students processed by the learning analytics processing and monitoring system
and use them to improve student motivation through the application of common
game elements in the context of Cyber Ranges. No other cyber range includes an
adaptive system with gamification elements. Therefore, there is also a lack of
attention to students’ motivation in Cyber Ranges that we solve in SCORPION.
The adaptive system seeks to keep the learner in an optimal challenge zone;
thus, he neither feels overwhelmed nor bored by the cyberexercise’s
difficulty. The adaptive system is also considered part of the gamification
system as it is a type of tool frequently found in games. Additionally, the
gamification system includes the following elements:

  * •

User profile and avatar: The user profile stores his role, avatar, username,
first name, last name, and email address. If the profile has the student role,
it also stores his points, different learning analytics, and medals gained.
All these data are only accessible to the user, instructors, and
administrators, so one student can not visualize the profiles of other users.

  * •

Points: Points are a bargaining chip to improve the student’s performance in
cyberexercises. Instructors designate the maximum number of points awarded for
each challenge of the cyberexercise, and the points obtained by the student
are calculated based on the resources consumed. Those resources are time,
hints, and flag attempts consumed (i.e., the challenge’s solution).

  * •

Medals: Medals are another reward for the students that aims to orient their
behaviors toward desired actions. For example, the Sherlock Holmes medal is
acquired for every ten challenges resolved of the Forensic competence,
encouraging the students to solve more challenges of this competence. Besides,
the students can achieve many times the same medal, and the design applied
also allow the addition of new medals to orient the student’s behavior to
other actions.

  * •

Learning analytics graphs: These graphs allow the visualization of learning
analytics, facilitating an analysis of student performance and generating
competitive motivation among them by comparing their learning analytics.
However, these general learning analytics graphs are only available to the
instructors, who decide whether to show them to all students during or at the
end of the cyberexercises. Figure 4 shows the graph that summarizes the solved
challenges of one cyberexercise instance and Figure 5 the mean score achieved
in multiple executions of one cyberexercise to the instructor.

Figure 4: Graph summarizing the solved challenges of one cyberexercise
instance Figure 5: Graph summarizing the mean score of multiple executions of
one cyberexercise

##  4 Use cases

This section describes the most relevant use cases of a complete workflow in
the proposed platform. We focus on these use cases, leaving out tasks such as
user management or platform configuration since they are similar to those in
other Cyber Ranges.

###  4.1 Creating a scenario

The first step in creating a cyberexercise by an instructor is the
construction of the scenario in which it takes place. To create the scenario,
the instructor has to specify the name, description, environment, and
complexity. From these last two parameters and taking the corresponding
generation rule (which contains a series of parameters that make the
generation different in each iteration, such as the maximum and the minimum
number of final entities and network entities), SCORPION generates a topology
that the instructor can customize each of the entities of this topology by
clicking on the node to be modified as shown in Figure 6. The instructor can
only modify the device type if the node corresponds to a network entity. On
the other hand, if a node is an end entity, the instructor can modify the
device type, change the template to be used by that entity, and add or remove
services and flags. Once the instructor has designed the scenario and it meets
his requirements, it is saved for later use in future cyberexercises.

Figure 6: Topology of a scenario with four final devices and two network
devices

###  4.2 Creating a challenge

In SCORPION, cyberexercises are conceived as a sequence of challenges deployed
in the same scenario. Therefore, the next step consists of creating the
challenges that will compose the cyberexercise. For each challenge, it is
necessary to establish the name, the evaluated competence, the objective
(attack or defend), the maximum time, the number of attempts, the hints, the
scenario on which it will be executed, a description, an explanation of the
solution, the challenge flag (the solution of the challenge), and the
difficulty which is automatically calculated according to the challenge score.
The available challenge resources will be reduced based on the student’s
performance on the previous challenge if the cyberexercise is adaptive

###  4.3 Creating a cyberexercise

To create a cyberexercise in the proposed framework, the instructor has to
select the scenario where the cyberexercise will take place. Based on the
selected scenario, the instructor has to indicate the sequence of challenges
among those available for that scenario. The selected challenge will establish
the competence, points, time, and difficulty. The time and the points are
calculated as the sum of the time and points of all their challenges. The
difficulty is calculated based on the points and the number of challenges. We
consider the initial competencies of binary, coding, crypto, enumeration,
steganography, forensics, lateral movement, and web, but the administrators
can add more through the configuration panel. Whether a cyberexercise is
composed of challenges of different competencies, the final competence is
miscellaneous. In addition, the cyberexercise has a name, a narrative, and
goals.

###  4.4 Activating a cyberexercise

After the cyberexercise has been created, the instructor can activate the
cyberexercise instance. During the first step in the activation, the
instructor indicates the students or groups participating in the cyberexercise
instance and specifies whether it will be collaborative and/or adaptive.
Figure 7 shows the activation of a collaborative adaptive cyberexercise.
Subsequently, the instructor assigns a scenario deployment to each student or
group. The generation of the scenario deployments can be done while the
cyberexercise is being activated, or they can have been deployed previously to
avoid the waiting time produced by virtualization and the reservation of
resources for the deployment generation. Then, the instructor can start, pause
and stop the cyberexercises and, on the other hand, visualize the students’
progress in real-time, as shown in Figure 8. This live progress graph shows
additional information when the instructor clicks on one of the student’s
challenges, indicating, among other data, the student’s stress level in a
challenge and the consumed resources. After finishing the cyberexercise, the
instructor can observe different graphs summarizing the cyberexercise
instance, such as the score of the students, the time consumed, and the
average heart rate of each student.

Figure 7: Activating a collaborative adaptive cyberexercise Figure 8: Live
progress graph

If the student wears a smart wristband with the SCORPION application, at least
five minutes before starting the cyberexercise, the student has to log in
through the wristband and activate the recording. This process aims to collect
a baseline of the student’s biometric data for the stress detection module.

Once the student joins the cyberexercise, he visualizes the visible entities
of the scenario topology, allowing the connection to these virtualized
entities. The cyberexercise view also shows the name of the challenge, its
description, the remaining time, and the maximum score. Furthermore, this view
allows the student to attempt flags and request hints. After finishing each
challenge, the student moves to the next challenge; if the cyberexercise is
adaptive, the resources available in each challenge will depend on his
performance in the previous challenges. Figure 9 displays the student view
during a cyberexercise. At the end of the cyberexercise, the student observes
a summary of his cyberexercise instance statistics, including his score, which
is calculated based on the resources consumed in each challenge.

Figure 9: Student view during a cyberexercise

###  4.5 Visualization of learning analytics

Instructors can analyze the cyberexercise learning analytics at three
aggregation levels:

  * •

Cyberexercise. The different metrics at this level allow the instructors to
analyze the overall performance of the students in that cyberexercise. For
example, an instructor may detect that the cyberexercise is too difficult for
the students and add more hints.

  * •

Cyberexercise instance. This aggregation level summarizes the performance of a
set of students in a specific cyberexercise instance. Sometimes instructors
are interested in analyzing the performance of a particular cyberexercise
instance, for example, because the cyberexercise instance is an exam or
because the general statistics of a cyberexercise include heterogeneous sets
of students who may have very different knowledge levels.

  * •

Challenge. When the cyberexercise instance is finished, the instructors can
analyze the final state of the live progress chart, allowing them to analyze
the different events of each challenge. At this level, an instructor may
observe whether there is a clue that leads the students too much to the
solution, the adaptations applied by the adaptive system, or if any of the
students have not managed the stress correctly in any of the challenges.

Figure 10: User profile statistics and his trophy gallery

Besides, the instructors can analyze the student’s overall and competency-
based performance in the student profile, including the medals gained. For
each competence, the total points, the total time, the proportion of points
obtained, and the proportion of remaining time are displayed. Figure 10 shows
these statistics. On the other hand, the students can visualize their learning
analytics for each cyberexercise instance and their overall and competency-
based performance.

##  5 Evaluation of SCORPION

We considered obtaining an external assessment of our Cyber Range essential.
The first objective was to determine if the platform was usable for students
and instructors. An unnecessarily complex platform is likely to be slightly
used or disused due to all the training required. Furthermore, the appearance
and how the elements are displayed in SCORPION could be intuitive for the
authors but not for external instructors and students. Therefore, we included
a System Usability Scale (SUS) questionnaire in our evaluation to assess the
usability of SCORPION.

Additionally, SCORPION includes different novelties in the state-of-the-art,
mainly, the parametrizable random scenario generator, the learning analytics
and the adaptive cyberexercises with gamification elements. However, suppose
these functionalities are considered useless. In that case, developing them
further is no point, and our Cyber Range would not stand out from the rest of
Cyber Ranges. Consequently, we included different questions to evaluate the
usefulness of the SCORPION’s functionalities.

We even considered the case study an excellent opportunity to discover what
features the users would stand out from SCORPION to leverage this
functionality and preserve it in new versions of our Cyber Range. On the other
hand, knowing things to be improved and receiving new ideas to continue
improving SCORPION was considered highly advantageous. Therefore we included
two open questions to collect what feature they would highlight of SCORPION
and potential improvements.

###  5.1 Participants

The participants were students or instructors of Computer Science at the
University of Murcia (UMU), Spain. A total of 31 subjects participated in the
evaluation. It is worth noting that SCORPION is focused on training
cybersecurity skills; therefore, users are instructors and students of some
kind of engineering. For this reason, it does not make sense to evaluate the
platform with students or instructors from other disciplines. The highest
educational level of 38.7% of our participants is a bachelor’s degree, 32.3% a
master’s degree, 16.1% a doctorate, and 12.9% a high school degree. The
volunteer roles are not mutually exclusive, as a volunteer can assume multiple
roles simultaneously. These roles include 71% researchers, 67.7% students,
22.6% instructors, 3.2% ITC professionals in companies, and 3.2% security
analysts. As for their skills in ITC, 58.1% self-assessed them with a score of
4, 32.2% with a 5 (the maximum score), and 9.7% with a 3; none assessed their
IT skills with scores of 2 and 1 (minimum). Regarding their cybersecurity
skills, 35.5% self-assessed them with a score of 4, 32.2% with a 3, 16.1% with
a 2, 9.7% with a 1 (the minimum score), and 6.5% with a 5 (the maximum score).

###  5.2 Methodology

The study was composed of a video demonstration of SCORPION and a
questionnaire. The demonstration lasted 16 minutes, showing the main
functionalities available in SCORPION for students and instructors. All the
workflow described in Section 4 was also shown. After the demonstration, the
volunteers filled out the questionnaire described in Table
LABEL:tab:questionnaire-questions. This questionnaire included:

  * •

The Spanish version of the SUS questionnaire [28] in a 5-item Likert Scale to
evaluate the usability of SCORPION.

  * •

Usefulness questionnaire with nine questions focused on functionalities for
instructors and five questions for students. Two of the questions should only
be answered by instructors or students, assessing the teaching and learning
capabilities of SCORPION.

  * •

Two open questions to collect which features they would like to highlight of
SCORPION and potential improvements.

###  5.3 Results

The achieved scores in the questionnaires are summarized in Table 2. In the
SUS questionnaire, SCORPION achieved an average score of 82.10 points over 100
and a standard deviation of 12.55. This is a very good result, showing that
SCORPION is usable, making students and instructors want to use it. Again, we
obtained excellent results in the usefulness evaluation, showing that the
different functionalities included in SCORPION are considered useful by
instructors and students. Therefore, the random scenario generator, the
adaptive cyberexercises, the learning analytic, and the inclusion of
gamification elements have been considered valuable, making our Cyber Range
stand out from the rest of the state-of-the-art Cyber Ranges.

Regarding the open question about the best features of SCORPION, 61.29%
highlighted the usability and intuitive nature of the front-end. In the first
open question, 19.35% of the participants highlighted the novelty
functionalities of SCORPION. This result indicates that usability is more
important for the participants than including novelty functionalities.
Therefore, in the development of Cyber Range, it is essential to include
usability questionnaires as done in SCORPION. Regarding possible improvements,
38.71% highlighted minor improvements over the functionalities implemented,
29.03% proposed new functionalities, 22.58% did not find possible
improvements, and 9.68% indicated functionalities they would like to see in
more detail in the case study video. These outcomes indicate that students
propose more minor improvements on existing functionalities than new
functionalities.

Question ID |  Score |  Number of questions
---|---|---
Average SUS |  82.10 |  10
Total usef-inst-scen |  9.55 |  2
Total usef-inst-crea-cyber |  8.68 |  2
Total usef-inst-act-cyber |  9.39 |  2
Total usef-inst-perf |  8.97 |  2
Average instructor usefulness |  4.57 |  8
Total usef-stud-real-cyber |  9.39 |  2
Total usef-stud-gam |  8.9 |  2
Average student usefulness |  4.57 |  4
Total usef-inst-cond |  4.57 |  1
Total usef-stud-cond |  4.73 |  1
Table 2: Scores obtained in the evaluation of SCORPION

##  6 Discussion

The developed Cyber Range architecture includes several modules that improve
upon the state of the art. Our analysis of three literature reviews on Cyber
Ranges [14, 8, 15] revealed that the possibilities for instructors to
configure their own scenarios by hand were not discussed in any of the
reviews. Therefore, we introduced an authoring tool that allows for the
creation of an infinite number of scenarios, which can be easily deployed in
the virtualization environment. The evaluation results contained six items
related to the cyberexercise authoring tool and obtained an average score of
4.6, thus we can conclude that respondents are in agreement and the tool has a
large potential for the creation of cyberexercises. This feature provides
great versatility and functionality, and can be scaled to other environments
as long as the necessary template entities are generated for the type of
scenario to be set up.

Moreover, our Cyber Range includes several significant novelties that have not
been fully explored in the literature. For instance, while gamification has
been identified as a powerful approach to improving student motivation [29],
its application in the context of cybersecurity has mainly been limited to
serious games [10]. We found only a few examples of Cyber Ranges that include
gamification features, such as the Mizzou Cyber Range [30]. To address this
gap, we introduced a large number of gamification elements in our Cyber Range,
including some that, to the best of our knowledge, have not been proposed
before, such as the adaptive system. In fact, our Cyber Range is the first to
consider adaptive learning functionalities, despite being considered one of
the key enabling technologies to provide personalized education to students
[31]. We included two items in the survey to evaluate the gamification
elements, and the respondents agreed with an average score of 4.45 that these
features can help improve the motivation of learners, which we consider a
success and the primary objective of gamification.

Another notable feature is the module for learning analytics processing, which
currently includes two main sources of data. First, we collect trace data from
students when interacting with the cyberexercises. Moreover, we also enable
the reception of multimodal signals with the RESTful API. While we have only
supported the reception of heart rate through a smartwatch and used this
signal for stress detection, this module can be easily extended to include a
greater number of signals from other devices and other cognitive abilities.
For example, we could extend it with the use of a brain-computer interface
(BCI) helmet to obtain the electroencephalogram (EEG) and use it to measure
concentration and activity levels. Such applications fall within the area of
evaluating capabilities through sensors to support training [32]. All these
data are processed to generate performance indicators and learning analytics,
which are then displayed in a visualization dashboard [33]. Instructors can
access the full information of all their students, which can enable the
possibility of providing personalized support and monitoring, while students
can access only their own performance metrics for self-awareness purposes.
These two survey items about learning analytics were also highly rated by
respondents with an average score of 4.49.

By unifying all of these features on a single platform, our Cyber Range
encompasses what is commonly known as the Learning Content Management System
(LCMS) that instructors interact with to create content, and the Learning
Management System (LMS), which is where students develop their learning [34].
Additionally, with the included learning analytics functionalities, numerous
metrics (even in real-time), the different gamification elements, and the
option to activate the adaptive system, our Cyber Range is an integrated
system that includes three cutting-edge educational technologies [35] unified
in a single environment for the training of professionals in cybersecurity.
Not only SCORPION has a large number of state-of-the-art functionalities, but
the tool is usable as a whole. SCORPION obtained a SUS score of 82.10 based on
the ten items of the questionnaire, which makes our results situated above the
10th percentile in terms of all the considered studies in the research
developed by Sauro [36]; thus, our usability results are remarkable. While we
have not found any other Cyber Ranges that have been evaluated with a SUS
score to compare with, a systematic review of the perceived usability of
educational technologies with SUS reported that the average score of Internet
platforms, that included LMSs, was a SUS score of 66.25 [37]; therefore, our
results are 16 points above the average SUS score reported in all the
educational technology literature.

Furthermore, the presented architecture is modular, scalable, and agnostic to
the virtualization environment where the scenarios will be deployed. This
means that the entire architecture could be reused by simply changing the
virtualization environment in the Terraform file generated by the scenario
generator system. Only a small adaptation would be required to generate the
new Terraform file in the appropriate format. This significantly facilitates
its reuse in other cybersecurity training environments, which have clear dual
applications (civil and military). Entities that may be interested in this
type of cybersecurity training are varied, including higher education
institutions, ICT professionals in companies, and units of security forces and
bodies working in cyber defense. Given the infrastructure we have and its easy
deployment through containers, it would be possible for them to reuse and
deploy it on their premises. However, it is also true that they may not have
the necessary personnel to maintain such technical infrastructure, so it could
also be of interest to provide these training services with Cyber Ranges as a
service, following a paradigm that we could define as Cyber Range as a Service
(CRaaS). This service could transparently offer practical cybersecurity
training and realistic scenarios, abstracting from all the complexity of
infrastructure, virtualization, as well as learning management and content
creation. In the future, we also plan to enable Learning Tools
Interoperability (LTI) IMS standard to enable SCORPION to be used as a third-
party provider of cybersecurity exercises [38]. This would enable a seamless
integration of SCORPION with other LMSs, such as Sakai, Moodle or Open edX,
greatly facilitating the teaching of integral cybersecurity courses with both
theoretical and practical contents.

One important limitation to highlight is that the Cyber Range described here
has not yet been tested in a real environment with a large number of students
and scenarios deployed simultaneously, so the exact technical needs and
possible scalability problems that may arise are not yet known. In addition,
there is still work that needs to be developed, such as capturing telemetry
from the machines in the deployed scenarios, supporting blue team operations,
or increasing the number of template entities available in the virtualized
template repository to improve scenario creation possibilities. Nevertheless,
given the importance of high-quality cybersecurity training today, the novel
contributions to the state of the art in the context of Cyber Ranges, and the
scarce supply of Cyber Ranges developed by Spanish entities (especially
universities), we consider that our Cyber Range makes a significant
contribution to literature and society.

##  7 Conclusion and future work

The rapid development of technology has led to the emergence of cybercrime as
a prominent threat to digital security. This type of crime involves the
sophisticated targeting of digital assets by highly skilled and organized
individuals and groups. Additionally, the ongoing Ukraine vs. Russia conflict
has demonstrated the increasing importance of cyberspace as another
battleground. Given this concerning context, the need for training
professionals and individuals in cybercompetences has become imperative. Cyber
Range platforms have emerged as an effective and efficient means of providing
hands-on training. However, the current state-of-the-art Cyber Range ecosystem
faces several challenges, including the static nature of cyberexercises and
the lack of adaptability to users’ capabilities.

In the paper at hand, we have presented an overview of our Cyber Range
architecture, which enables key and novel functionalities within the
literature. Some of the key points of the architecture are as follows: 1) a
scenario authoring tool, able to generate random and realist network
topologies, which is supported by a specific module that allows for
parameterization, randomization, and detailed configuration of such scenarios
and each of their nodes, 2) a gamification module that allows for the
deployment of various elements such as scores, medals, including adaptive
system that is capable of adapting the available resources of students in each
challenge of a cyberexercise based on their previous performance, and 3) a
learning component that collect telemetric and biometric students’ data and
generates numerous performance metrics in cyberexercises.

To demonstrate its capabilities, the efficacy of SCORPION is then assessed by
means of a case study in order to substantiate its usability and usefulness.
SCORPION obtained 82.10% in usability and 4.57 out of 5 in usefulness from the
viewpoint of a student and an instructor. Therefore, it has been demonstrated
that the platform is usable and that the functionalities that make SCORPION
stand out in the state-of-the-art are useful.

This Cyber Range opens up new opportunities for validation and research in the
future. Firstly, those related to the technical validation of the architecture
to analyze its scalability, effectiveness, and usability within real
experiences, whether in cybersecurity courses in higher education or other
centers interested in this training. These experiences will also generate
large data sets that will allow the analysis of students’ behavior with the
Cyber Range, facilitating research with real case studies to evaluate the
impact of the different modules of the Cyber Range. We plan to explore how to
offer these services to different interested actors so that they can consume
them transparently, without worrying about infrastructure or deployment
requirements. Finally, we also plan to extend many of these modules to
incorporate new functionalities, such as collecting additional biometric
signals or supporting the deployment of more services in the scenarios.

Overall, the Cyber Range architecture presented in this article represents a
significant contribution to the literature and society, given the importance
of high-quality cybersecurity training today. While there are some
limitations, such as the need for further testing in real-world scenarios, the
potential impact of this Cyber Range on cybersecurity education and research
is promising.

## Acknowledgment

This work has been partially funded by the SCORPION project (21661-PDC-21),
granted by the Seneca Foundation of the Region of Murcia, the grant
PID2021-122466OB-I00, by MCIN/AEI/10.13039/501100011033/FEDER, and the Spanish
Ministry of Universities linked to the European Union through the
NextGenerationEU program, under Margarita Salas postdoctoral fellowship
(172/MSJD/22).

## Declaration of competing interests

The authors declare that they have no known competing financial interests or
personal relationships that could have appeared to influence the work reported
in this paper.

## References

  * [1] J. V. Botello, A. P. Mesa, F. A. Rodríguez, D. Díaz-López, P. Nespoli, F. G. Mármol, BlockSIEM: Protecting smart city services through a Blockchain-based and distributed SIEM, Sensors 20 (16) (2020).  doi:10.3390/s20164636.
  * [2] A. Arooj, M. S. Farooq, T. Umer, Unfolding the blockchain era: Timeline, evolution, types and real-world applications, Journal of Network and Computer Applications 207 (2022) 103511.  doi:https://doi.org/10.1016/j.jnca.2022.103511.
URL https://www.sciencedirect.com/science/article/pii/S1084804522001527

  * [3] P. Nespoli, D. Useche Peláez, D. Díaz López, F. Gómez Mármol, COSMOS: Collaborative, Seamless and Adaptive Sentinel for the Internet of Things, Sensors 19 (7) (2019).  doi:10.3390/s19071492.
  * [4] I. Kotenko, I. Saenko, O. Lauta, Modeling the Impact of Cyber Attacks, Springer International Publishing, Cham, 2019, pp. 135–169.  doi:10.1007/978-3-319-77492-3_7.
  * [5] K. Huang, M. Siegel, S. Madnick, Systematically understanding the cyber attack business: A survey, ACM Comput. Surv. 51 (4) (jul 2018).  doi:10.1145/3199674.
  * [6] M. K. Hasan, A. A. Habib, Z. Shukur, F. Ibrahim, S. Islam, M. A. Razzaque, Review on cyber-physical and cyber-security system in smart grid: Standards, protocols, constraints, and recommendations, Journal of Network and Computer Applications 209 (2023) 103540.  doi:https://doi.org/10.1016/j.jnca.2022.103540.
URL https://www.sciencedirect.com/science/article/pii/S1084804522001813

  * [7] N. Chowdhury, V. Gkioulos, Cyber security training for critical infrastructure protection: A literature review, Computer Science Review 40 (2021) 100361.  doi:https://doi.org/10.1016/j.cosrev.2021.100361.
  * [8] N. Chouliaras, G. Kittes, I. Kantzavelou, L. Maglaras, G. Pantziou, M. A. Ferrag, Cyber ranges and testbeds for education, training, and research, Applied Sciences 11 (4) (2021) 1–23.  doi:10.3390/APP11041809.
  * [9] R. Petersen, D. Santos, M. C. Smith, K. A. Wetzel, G. Witte, Workforce Framework for Cybersecurity (NICE Framework), Tech. rep., NIST (2020).
URL
https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-181r1.pdf

  * [10] J.-N. Tioh, M. Mina, D. W. Jacobson, Cyber security training a survey of serious games in cyber security, in: 2017 IEEE Frontiers in Education Conference (FIE), IEEE, 2017, pp. 1–5.  doi:10.1109/FIE.2017.8190712.
  * [11] R. Beuran, K.-i. Chinen, Y. Tan, Y. Shinoda, Towards Effective Cybersecurity Education and Training, Research report (School of Information Science, Graduate School of Advanced Science and Technology, Japan Advanced Institute of Science and Technology) IS-RR-2016 (2016) 1–16.
  * [12] P. Blikstein, M. Worsley, Multimodal learning analytics and education data mining: using computational technologies to measure complex learning tasks, Journal of Learning Analytics 3 (2) (2016) 220–238.  doi:10.18608/jla.2016.32.11.
  * [13] J. Swacha, State of research on gamification in education: A bibliometric survey, Education Sciences 11 (2) (2021).  doi:10.3390/educsci11020069.
URL https://www.mdpi.com/2227-7102/11/2/69

  * [14] E. Ukwandu, M. A. B. Farah, H. Hindy, D. Brosset, D. Kavallieros, R. Atkinson, C. Tachtatzis, M. Bures, I. Andonovic, X. Bellekens, A review of cyber-ranges and test-beds: Current and future trends, Sensors (Switzerland) 20 (24) (2020) 1–36.  arXiv:2010.06850, doi:10.3390/s20247148.
  * [15] M. M. Yamin, B. Katt, V. Gkioulos, Cyber ranges and security testbeds: Scenarios, functions, tools and architecture, Computers and Security 88 (2020) 101636.  doi:10.1016/j.cose.2019.101636.
  * [16] J. Vykopal., R. Oslejsek., P. Celeda., M. Vizvary., D. Tovarnak., Kypo cyber range: Design and use cases, in: Proceedings of the 12th International Conference on Software Technologies - ICSOFT,, INSTICC, SciTePress, 2017, pp. 310–321.  doi:10.5220/0006428203100321.
  * [17] T. Lieskovan, J. Hajný, Building open source cyber range to teach cyber security, in: The 16th International Conference on Availability, Reliability and Security, ACM, 2021, pp. 1–11.  doi:10.1145/3465481.3469188.
  * [18] R. D. Medenou Choumanof, S. Llopis Sanchez, V. M. Calzado Mayo, M. Garcia Balufo, M. Páramo Castrillo, F. J. González Garrido, A. Luis Martinez, D. Nevado Catalán, A. Hu, D. S. Rodríguez-Bermejo, G. R. Pasqual de Riquelme, M. A. Sotelo Monge, A. Berardi, P. De Santis, F. Torelli, J. Maestre Vidal, Introducing the cysas-s3 dataset for operationalizing a mission-oriented cyber situational awareness, Sensors 22 (14) (2022).  doi:10.3390/s22145104.
URL https://www.mdpi.com/1424-8220/22/14/5104

  * [19] A. Bécue, E. Maia, L. Feeken, P. Borchers, I. Praça, A new concept of digital twin supporting optimization and resilience of factories of the future, Applied Sciences 10 (13) (2020).  doi:10.3390/app10134482.
URL https://www.mdpi.com/2076-3417/10/13/4482

  * [20] Leonardo cyber range, Tech. rep., Leonardo, Genova, Italy (2022).
  * [21] R. Beuran, C. Pham, D. Tang, K.-i. Chinen, Y. Tan, Y. Shinoda, Cytrone: An integrated cybersecurity training framework, Proceedings of the 3rd International Conference on Information Systems Security and Privacy (ICISSP 2017) (2017) 157–166.
URL https://ci.nii.ac.jp/naid/120007005345/en/

  * [22] J. Eckroth, K. Chen, H. Gatewood, B. Belna, Alpaca: Building dynamic cyber ranges with procedurally-generated vulnerability lattices, in: Proceedings of the 2019 ACM Southeast Conference, ACM SE ’19, Association for Computing Machinery, New York, NY, USA, 2019, p. 78–85.  doi:10.1145/3299815.3314438.
  * [23] G. Hatzivasilis, S. Ioannidis, M. Smyrlis, G. Spanoudakis, F. Frati, C. Braghin, E. Damiani, H. Koshutanski, G. Tsakirakis, T. Hildebrandt, L. Goeke, S. Pape, O. Blinder, M. Vinov, G. Leftheriotis, M. Kunc, F. Oikonomou, G. Magilo, V. Petrarolo, A. Chieti, R. Bordianu, The THREAT-ARREST cyber range platform, Proceedings of the 2021 IEEE International Conference on Cyber Security and Resilience, CSR 2021 (2021) 422–427doi:10.1109/CSR51186.2021.9527963.
  * [24] M. Leitner, M. Frank, W. Hotwagner, G. Langner, O. Maurhart, T. Pahi, L. Reuter, F. Skopik, P. Smith, M. Warum, Ait cyber range: Flexible cyber security environment for exercises, training and research, in: Proceedings of the European Interdisciplinary Cybersecurity Conference, EICC 2020, Association for Computing Machinery, New York, NY, USA, 2021, pp. 1–6.  doi:10.1145/3424954.3424959.
  * [25] D. Lee, D. Kim, C. Lee, M. K. Ahn, W. Lee, ICSTASY: An Integrated Cybersecurity Training System for Military Personnel, IEEE Access 10 (2022) 62232–62246.  doi:10.1109/ACCESS.2022.3182383.
  * [26] S. Y. Enoch, Z. Huang, C. Y. Moon, D. Lee, M. K. Ahn, D. S. Kim, HARMer: Cyber-Attacks Automation and Evaluation, IEEE Access 8 (2020) 129397–129414.  arXiv:2006.14352, doi:10.1109/ACCESS.2020.3009748.
  * [27] S. Y. Enoch, C. Y. Moon, D. Lee, M. K. Ahn, D. S. Kim, A practical framework for cyber defense generation, enforcement and evaluation, Computer Networks 208 (March) (2022) 108878.  doi:10.1016/j.comnet.2022.108878.
  * [28] M. D. R. Sevilla-Gonzalez, L. M. Loaeza, L. S. Lazaro-Carrera, B. B. Ramirez, A. V. Rodríguez, M. L. Peralta-Pedrero, P. Almeda-Valdes, Spanish version of the system usability scale for the assessment of electronic tools: Development and validation, JMIR Human Factors 7 (4) (2020) e21161.  doi:10.2196/21161.
  * [29] E. D. Mekler, F. Brühlmann, K. Opwis, A. N. Tuch, Do points, levels and leaderboards harm intrinsic motivation? an empirical analysis of common gamification elements, in: Proceedings of the First International Conference on gameful design, research, and applications, 2013, pp. 66–73.  doi:10.1145/2583008.2583017.
  * [30] K. B. Vekaria, P. Calyam, S. Wang, R. Payyavula, M. Rockey, N. Ahmed, Cyber range for research-inspired learning of “attack defense by pretense” principle and practice, IEEE Transactions on Learning Technologies 14 (3) (2021) 322–337.  doi:10.1109/TLT.2021.3091904.
  * [31] F. Martin, Y. Chen, R. L. Moore, C. D. Westine, Systematic review of adaptive learning research designs, context, strategies, and technologies from 2009 to 2018, Educational Technology Research and Development 68 (4) (2020) 1903–1929.  doi:10.1007/s11423-020-09793-2.
  * [32] J. Schneider, D. Börner, P. Van Rosmalen, M. Specht, Augmenting the senses: a review on sensor-based learning support, Sensors 15 (2) (2015) 4097–4133.  doi:10.3390/s150204097.
  * [33] J. A. Ruipérez-Valiente, P. J. Muñoz-Merino, D. Leony, C. D. Kloos, Alas-ka: A learning analytics extension for better understanding the learning process in the khan academy platform, Computers in Human Behavior 47 (2015) 139–148.  doi:10.1016/j.chb.2014.07.002.
  * [34] S. Ninoriya, P. Chawan, B. Meshram, Cms, lms and lcms for elearning, International Journal of Computer Science Issues (IJCSI) 8 (2) (2011) 644.
  * [35] M. J. Gomez, J. A. Ruipérez-Valiente, F. J. García Clemente, Analyzing trends and patterns across the educational technology communities using fontana framework, IEEE Access 10 (2022) 35336–35351.  doi:10.1109/ACCESS.2022.3163253.
  * [36] J. Sauro, Sustisfied? little-known system usability scale facts, User Experience Magazine 10 (3) (2011).
  * [37] P. Vlachogianni, N. Tselios, Perceived usability evaluation of educational technology using the system usability scale (sus): A systematic review, Journal of Research on Technology in Education 54 (3) (2022) 392–409.  doi:10.1080/15391523.2020.1867938.
  * [38] A. J. Sierra, Á. Martín-Rodríguez, T. Ariza, J. Muñoz-Calle, F. J. Fernández-Jiménez, Lti for interoperating e-assessment tools with lms, in: Methodologies and Intelligent Systems for Technology Enhanced Learning: 6th International Conference, Springer, 2016, pp. 173–181.  doi:10.1007/978-3-319-40165-2_18.

## Appendix A

The questions included in the evaluation of SCORPION are shown in Table
LABEL:tab:questionnaire-questions

Table 3: Questionnaire employed in the evaluation of SCORPION. The questions have been translated from Spanish to English Questio- nnaire |  Functionality evaluated |  Question |  Identifier
---|---|---|---
System Usability Scale (SUS) |  General usability |  I think that I would like to use this system frequently |  SUS-1
General usability |  I found the system unnecessarily complex |  SUS-2
General usability |  I thought the system was easy to use |  SUS-3
General usability |  I think that I would need the support of a technical person to be able to use this system |  SUS-4
General usability |  I found the various functions in this system were well integrated |  SUS-5
General usability |  I thought there was too much inconsistency in this system |  SUS-6
General usability |  I would imagine that most people would learn to use this system very quickly |  SUS-7
General usability |  I found the system very cumbersome to use |  SUS-8
General usability |  I felt very confident using the system |  SUS-9
General usability |  I needed to learn a lot of things before I could get going with this system |  SUS-10
Usefulness |  Instructor: Scenario generator |  I think the random scenario generator helps in scenario creation |  usef-inst-scen-1
Instructor: Scenario generator |  I think the interactive figure with the topology of the scenario generator facilitates scenario editing and creation |  usef-inst-scen-2
Instructor: challenges and cyberexercises |  I think the tool facilitates the creation of cyberexercises by decomposing them into reusable challenges |  usef-inst-crea-cyber-1
Instructor: challenges and cyberexercises |  I think the tool enables the creation of realistic cyberexercises |  usef-inst-crea-cyber-2
Instructor: activation of a cyberexercise |  I think the possibility of activating collaborative cyberexercises allows the development of teamwork capacity |  usef-inst-act-cyber-1
Instructor: activation of a cyberexercise |  I think the management of deployments through the platform makes activating a cyberexercise easier for the instructor |  usef-inst-act-cyber-2
Instructor: learning analytics |  I think that heart rate measurement and stress detection can be useful for measuring work capacity under pressure |  usef-inst-perf-1
Instructor: learning analytics |  I think that the statistics generated by the platform allow easy analysis of students’ performance in cyberexercises |  usef-inst-perf-2
Question for real professors |  Answer this question only if you are a higher education professor. How much do you agree with the following statement: “The use of SCORPION would improve my ability to teach cybersecurity competencies?” |  usef-inst-cond
Student: realization of a cyberexercise |  I think that during a cyberexercise, the student visualizes the information needed to solve the challenge, e.g., the scenario and description of the challenge |  usef-stud-real-cyber-1
Usefulness |  Student: realization of a cyberexercise |  I think that during the completion of the cyberexercise, the student visualizes the necessary information about available and consumed resources such as time, available flag attempts, or hints |  usef-stud-real-cyber-2
Student: gamification |  I think the adaptive system improves student motivation during cyberexercise execution |  usef-stud-gam-1
Student: gamification |  I think the medal and scoring system motivate the student to perform more cyberexercises |  usef-stud-gam-2
Question for real students |  Answer this question only if you are a computer engineering student or a similar degree in higher education. How much do you agree with the following statement: “Would using SCORPION improve your chances of acquiring cybersecurity skills?” |  usef-stud-cond
Opinion |  Open question |  What positive features would you highlight? |  OP-1
|  Open question |  What tool features do you think could be improved or be added? |  OP-2