# From Paper to Platform: Evolution of a Novel Learning Environment for
Tabletop Exercises

Valdemar Švábenský  0000-0001-8546-280X Masaryk UniversityFaculty of
InformaticsBrnoCzech Republic<EMAIL_ADDRESS>,  Jan Vykopal
0000-0002-3425-0951 Masaryk UniversityFaculty of InformaticsBrnoCzech Republic
<EMAIL_ADDRESS>,  Martin Horák  0000-0002-1835-6465 Masaryk
UniversityFaculty of InformaticsBrnoCzech Republic<EMAIL_ADDRESS>,
Martin Hofbauer  0009-0005-3998-9164 Masaryk UniversityFaculty of
InformaticsBrnoCzech Republic<EMAIL_ADDRESS>and  Pavel Čeleda
0000-0002-3338-2856 Masaryk UniversityFaculty of InformaticsBrnoCzech Republic
<EMAIL_ADDRESS>
(2024)

###### Abstract.

For undergraduate students of computing, learning to solve complex practical
problems in a team is an essential skill for their future careers. This skill
is needed in various fields, such as in cybersecurity and IT governance.
Tabletop exercises are an innovative teaching method used in practice for
training teams in incident response and evaluation of contingency plans.
However, tabletop exercises are not yet widely established in university
education. This paper presents data and teaching experience from a
cybersecurity course that introduces tabletop exercises in classrooms using a
novel technology: INJECT Exercise Platform (IXP), a web-based learning
environment for delivering and evaluating the exercises. This technology
substantially improves the prior practice, since tabletop exercises worldwide
have usually been conducted using pen and paper. Unlike in traditional
tabletop exercises, which are difficult to evaluate manually, IXP provides
insights into students’ behavior and learning based on automated analysis of
interaction data. We demonstrate IXP’s capabilities and evolution by comparing
exercise sessions hosted throughout three years at different stages of the
platform’s readiness. The analysis of student data is supplemented by the
discussion of the lessons learned from employing IXP in computing education
contexts. The data analytics enabled a detailed comparison of the teams’
performance and behavior. Instructors who consider innovating their classes
with tabletop exercises may use IXP and benefit from the insights in this
paper.

tabletop exercise, incident response, team collaboration, cybersecurity,
hands-on training, learning analytics, INJECT

††journalyear: 2024††copyright: rightsretained††conference: Proceedings of the
2024 Innovation and Technology in Computer Science Education V. 1; July 8–10,
2024; Milan, Italy††booktitle: Proceedings of the 2024 Innovation and
Technology in Computer Science Education V. 1 (ITiCSE 2024), July 8–10, 2024,
Milan, Italy††doi: 10.1145/3649217.3653639††isbn:
979-8-4007-0600-4/24/07††ccs: Social and professional topics Computing
education

##  1\. Introduction

Collaborative problem-solving of complex issues (CPSCI), such as the
resolution of incidents in large organizations that critically rely on
information technology (IT), is a core competency for the twenty-first-century
workforce (Fiore et al., 2018; CC2020 Task Force, 2020). However, many
university graduates lack necessary skills in these areas (Fiore et al.,
2018).

University students of applied computing (a target student demographic of this
paper) learn CPSCI in cybersecurity and IT governance courses, among others.
These courses cover topics like cyber incident response, emergency readiness,
information sharing, or contingency plan validation when managing an IT
infrastructure.

Computing educators found it difficult to provide students with practical
learning experience in such courses (Ottis, 2014). Thus, researchers and
instructors have been exploring innovative ways to teach these
interdisciplinary topics, which connect technological and human aspects of
computing, in an immersive and meaningful way (Ottis, 2014).

###  1.1. What Are Tabletop Exercises?

A tabletop exercise (TTX) is a type of a teaching activity designed to train
professional teams in incident response to a crisis situation (Grance et al.,
2006). The simulated crisis happens in the context of business operations in
an organization, e.g., a phishing attack on employees or malware infecting the
company infrastructure. The team members (exercise participants) hold various
roles in the organization, e.g., manager or cybersecurity incident responder
(Angafor et al., 2023). During the exercise, the team discusses which actions
to take to effectively respond to the emergency while following proper
protocols and regulations. The team discussions are facilitated by
instructors, who also present an exercise debriefing at the end.

TTXs are an effective educational tool, which enhance incident preparedness of
individuals, particularly their communication, coordination, and collaboration
(Angafor et al., 2023). In computing education, TTXs provide students with
realistic incident response experience and deepen their understanding of
related processes. TTXs are especially relevant for cybersecurity and
information security management courses and align with broader IT governance
courses (European Union Agency for Cybersecurity (2010), ENISA).

###  1.2. Problem Statement and Innovation

TTXs differ from technical hands-on exercises in an emulated computer
infrastructure, such as in a cyber range (Yamin et al., 2020) or a locally
virtualized learning environment (Vykopal et al., 2021). Instead, TTXs are
much more lightweight and do not dive into technical matters deeply. They are
traditionally conducted using pen and paper or simple online office
applications, such as Google Forms, to collect participant responses. The
advantage of this approach is its low cost and low barrier to entry. On the
other hand, the assessment of the participating teams has to be done manually
by the instructors, which is highly time-consuming. It takes days or even
weeks until the trainees can receive educational feedback, which diminishes
its effectiveness and decreases learning gains from the TTX.

We aim to transition the TTX format from this low-tech approach into INJECT
Exercise Platform (IXP): a novel, lightweight, open-source environment for
supporting the deployment and evaluation of TTXs. This represents a major
innovation that automates repetitive tasks for instructors, leaving them more
room for teaching. Since IXP automatically collects exercise data, it can
deliver pedagogical insights and feedback using the methods of learning
analytics. This paper shares our experience in deploying IXP in computing
classes and analyzes student data from these classes.

###  1.3. Goals and Scope of This Paper

We developed a novel TTX, which we deployed on three occasions (“runs”) with
three groups of learners. In the first run, we used only online Microsoft
Office (Microsoft, 2023) applications. In the second run, we used a simple
prototype of the TTX platform. Finally, the third run demonstrated a more
developed version of IXP. Our goal is to compare the student data and teacher
perspective on facilitating the TTX in these three different versions of the
learning environment. Specifically, this paper explores the following research
questions:

  1. (1)

What types of insights about the student behavior and learning can the
platform deliver to instructors?

  2. (2)

What is the instructors’ teaching experience when comparing the three exercise
runs?

##  2\. Related Work

As this paper focuses on transitioning TTXs from pen-and-paper format to a
software platform, we reviewed literature covering all three stages of the
transition: in-person pen-and-paper format, online pen-and-paper format, and
online platform for TTX delivery. We also review analytics of data from TTXs.
Based on our review of related work, we summarize the unique contributions of
our paper.

###  2.1. Pen-and-Paper Tabletop Exercises

Ottis (2014) described how to create lightweight TTXs for cybersecurity
education. The TTX detailed in the paper is in-person with two groups of
participants: “red” and “blue”. Red teams are in charge of creating the
exercise scenario from the attackers’ perspective. (A TTX scenario is an
outline of the sequence of events that drive the exercise and guide
participant discussion (U.S. Cybersecurity & Infrastructure Security Agency,
CISA).) Blue teams are responsible for handling the attack events. The paper
presents observations from eight such TTXs with 250 students in total.

Angafor et al. (2023) used Microsoft Teams to conduct an online pen-and-paper-
like TTX for 20 participants from an unnamed company. After the exercise, the
participants answered a survey about their awareness of attack mitigation
controls, as well as feedback on the completeness of controls currently used
in the company. The authors used descriptive statistics to analyze the survey.
However, the publication does not include any learning analytics.

Brilingaitė et al. (2017) conducted a TTX with students of IT and social
sciences. A custom software that hosted the exercise also logged data about
user actions. These actions include the number of messages and the number of
exercise events in different states of progress, which do not offer extensive
possibilities for analysis. The option of further analysis of exercise logs is
mentioned, but neither these logs nor the analysis are available.

###  2.2. Software for Tabletop Exercises

TTXs in the cybersecurity context are quite prominent (European Union Agency
for Cybersecurity (2012), ENISA; European Union Agency for Cybersecurity
(2015), ENISA), which leads to research and development of software solutions.
While there are companies, such as Privasec (Global, 2023) or Red Goat (Goat,
2023), that provide paid software for TTXs, open-source solutions exist as
well.

We discovered 16 open-source projects for TTXs (Vykopal et al., 2024) on
GitHub, out of which 11 contained software solutions. Most of them are simple
and specifically tailored for delivering just one specific exercise scenario.
An example is an application (Lewis, 2022) that presented the scenario in a
few sentences via a command-line interface and asked the participants to
discuss the possible solutions.

While some software solutions are more advanced than just presentations of
scenarios, they focus on the cybersecurity aspects of the exercise, as opposed
to more discussion-based problems. These solutions are also more technical. An
example is Ransomware Simulator (McKeown, 2022), which works as a reporting
tool for incidents, asking for the event ID, owner, summary, and response to
the event. The instructors can add the option of a simulated ransomware attack
launched at a specified time, locking participants out of the tool.

The open-source software solution that we consider to have the most features
is OpenEx (Filigran, 2023). This solution is not specifically tailored for
cybersecurity TTXs, and it allows to create and execute different scenarios.
Unlike other software we found, OpenEx records logs of participant
interactions within the scenario. This enables analyzing the data gathered
during the exercise; however, OpenEx does not implement such analysis. Another
downside of OpenEx is using real email infrastructure for participant
communication, which can lead to delays due to antispam or system outages.

###  2.3. Data Analytics in Tabletop Exercises

As of October 2023, searching “tabletop exercise” in the multi-faceted
citation database Scopus (Elsevier, 2023) returned 418 papers. While this
amount is non-negligible, the number of publications with learning analytics
of data from TTXs is low.

Mareš et al. (2023) conducted a TTX for 33 experts in cybersecurity and
related fields, such as law. The authors analyzed data from two surveys about
the participants’ behavior, performance, and workload handling. The first
survey was carried out immediately after the exercise, and the following
survey two weeks later. Higher performance of participants was significantly
correlated with their lower levels of perceived stress ($r=0.30$ to $0.37$,
$p=0.039$). However, this study does not include data about the participants’
learning.

Hsieh et al. (2023) compared fire safety knowledge acquisition between drill-
based and game-based learning. Although this study did not use the TTX format
as defined above, the game-based learning was carried out as a tabletop game.
The authors used t-tests to measure the knowledge gain of both groups. The
knowledge gain of the game-based group ($t=12.58$) was substantially larger
than for the other group ($t=6.14$), with $p<0.001$ for both statistics.

We did not find any other publication containing learning analytics of TTX
data. The only data currently being collected from TTXs are not focused on the
educational process, but on feedback on the exercise itself and its perceived
usefulness for the participants (Ottis, 2014; Kopustinskas et al., 2020; Ota
et al., 2022). These data are valuable for the exercise creators but do not
provide deeper insight into the TTX participant behavior.

###  2.4. Novel Contributions of This Paper

TTXs are suited for computing education, and some software solutions for
conducting TTXs exist. However, the existing research does not focus on TTX
participant learning behavior. The software solutions do not implement
analytics, other than descriptive. So, our work contributes to educators and
researchers with these inputs:

  * •

We propose an innovated TTX format (Section 3.1).

  * •

Since TTXs rarely use dedicated software for evaluation and in-depth analysis,
we develop a new learning environment that provides these functionalities
(Section 3.2).

  * •

Unlike traditional TTXs, in which the instructors analyze the exercise data
manually, we demonstrate the platform’s capabilities in automated data
collection and analysis. The data come from three runs of a novel TTX deployed
in an authentic teaching context (Section 4).

  * •

For instructors and practitioners, we share the practical lessons learned from
using the platform (Section 5).

  * •

Artifacts associated with this work are available (Section 6).

##  3\. Tabletop Exercise Delivery

We now define the key features of TTXs. The purpose of this section is
twofold: (1) to provide the background for our research study, which is
detailed in Section 4, and (2) to represent a contribution on its own by
defining the innovated TTX format and its properties.

###  3.1. Proposed Exercise Format

####  3.1.1. Participant Roles

Human participants in a TTX can have one of three roles. Designers prepare the
exercise and its scenario. Instructors facilitate the exercise by guiding the
participants and evaluate the exercise at its end. They may or may not be
different from designers. Trainees attend the exercise to improve their
skills. Trainees are grouped into teams that are independent of each other
(i.e., each team completes the same tasks in parallel). Each person may have a
different role in the team.

####  3.1.2. Components of the Exercise

An inject is a pre-scripted message, such as an email, provided to trainees
during the TTX. Its purpose is to move the scenario forward and prompt
additional actions. For example, it can inform the trainees about a data
breach in their company, requiring them to respond accordingly (Grance et al.,
2006).

A tool is a simplified simulated version of a real-world computer
application/service. Its purpose is to allow trainees to perform actions to
respond to injects. For example, trainees in a TTX do not use an actual email
client, but an “email” tool to send and receive in-exercise messages. Another
example is a “browser”, a tool that returns an in-exercise website based on
the provided exercise URL.

A milestone is a true/false condition that denotes whether a specific team
reached a certain important situation in the TTX scenario. Its purpose is to
track each team’s progress through the TTX. For example, it can mark that a
team used an email tool to respond to a query from a manager. The exercise
milestones can be completed in any order, and there is rarely a single correct
solution. These properties make the team assessment challenging.

####  3.1.3. Exercise Workflow

The exercise is driven by injects, which are either provided by the
instructor, or triggered automatically based on time (e.g., an inject is sent
after 20 minutes of the exercise) or based on milestone (in)completion. Some
injects may be provided to all teams at once; others are conditioned by the
team’s progress. This allows each team to progress through the scenario
independently of other teams. As a result, each team can progress at their own
pace, which removes the need for them to wait idle.

When a new inject arrives, trainees in a team discuss the situation to agree
on which action to take (e.g., which tool to use). Effective communication
under time pressure is a crucial component of TTXs. Trainees do not choose an
inject response from pre-defined options, but have to think of a unique open-
ended solution. If a team gets stuck, the instructor should help them by
asking guiding questions or providing a gentle hint via responding to the
team’s emails.

###  3.2. Exercise Platform

Figure 1. Trainees’ view of the INJECT Exercise Platform. In the left sidebar
(A), trainees see injects or email conversations. The middle pane (B) shows
injects, emails, and outcomes of tools, depending on the view chosen in the
left bar. In the right sidebar (C), trainees see all available tools. After
clicking on the tool, a dropdown menu (D) for the tool’s arguments appears.

Screenshot of the frontend of the Tabletop Exercise Platform.

We developed a novel learning environment called INJECT Exercise Platform
(IXP), which is an interactive web application for supporting the delivery and
evaluation of TTXs.

Designers can use the platform to instantiate an exercise definition, which
prescribes the exercise story, injects, available tools, and milestones. An
exercise definition is implemented as a set of structured text-based files (in
YAML format) that are both human- and machine-readable. It automates a
substantial portion of the TTX, since it provides trainees with tools and
inject response templates.

Instructors deploy an exercise definition in IXP when they want to host an
exercise. The definition is created only once, but thanks to the platform’s
automation capabilities, it can be deployed repeatedly under the same
conditions for different trainees. Compared to manually-hosted exercises, IXP
significantly reduces the workload and personnel requirements for TTX
delivery.

Trainees interact with the scenario through automated tools during the
exercise. This interaction moves the scenario forward and impacts the
simulated environment. Figure 1 shows the trainees’ view of the IXP to
demonstrate some of the interaction options.

###  3.3. Exercise Content Example

To illustrate the exercise format and the components of the platform, we now
describe an exercise that we developed for the IXP. This TTX was also selected
for our research study described in Section 4.

####  3.3.1. Learning Objectives

The TTX was based on a real cyber attack that happened at our university, in
order to provide the trainees with an authentic learning experience. The
learning objectives of the TTX are: (a) to perform cyber incident triage, (b)
to coordinate and execute incident response, and (c) to mitigate the impacts
of an incident on a large organization with multiple involved parties.

####  3.3.2. Description of the Story

In the TTX scenario, the trainees assume the role of the members of a Computer
Security Incident Response Team (CSIRT) of the university. At the beginning,
the trainees receive an initial inject: a report of a phishing email targeting
the university employees. Several employees have fallen victims to the attack
and submitted their login credentials to a fraudulent phishing website. As a
result, a malicious actor accessed sensitive information on the employees’
internal project server and also deleted important files, making the project
website unavailable. As the time progresses, the trainees receive more and
more injects in the form of emails from the affected employees, asking the
team to take swift action in response to the ongoing emergency.

####  3.3.3. Available Tools

The trainees can take numerous actions at each point of the TTX. They can
apply technical solutions (e.g., inspect network data or block traffic to/from
certain IP addresses) as well as take managerial/governance steps (e.g.,
notify the responsible persons according to the data protection law). Each
action (e.g., using a specific tool) or inaction (e.g., not responding to a
query quickly enough) may trigger another inject to propel the scenario
forward and place the trainees in the midst of another time-critical issue. To
support discussion in a team, only one person from the team can interact with
the tools in IXP at any given time, after the members consult and mutually
agree on their progress.

##  4\. Research Methods

We delivered the exercise described in Section 3 on three occasions,
throughout three years with the total of 91 university students of computing.
This section describes the design of our research study. The research
questions were posed in Section 1.3.

###  4.1. Course Context

The TTX is the culmination of the course titled Cybersecurity in an
Organization, which is taught at the Faculty of Informatics, Masaryk
University: a large, public university in Central Europe.

####  4.1.1. Learning Outcomes

The course graduates should understand the role and services of a CSIRT in an
organization. Specifically, the course covers knowledge and skills required
for the work role of Cyber Defense Incident Responder as defined by the NICE
Cybersecurity Workforce Framework (for Cybersecurity Careers and, NICCS).

####  4.1.2. Teaching Format

The course is offered once per academic year and spans a standard 13-week Fall
semester. It is taught in-person using a combination of flipped classroom
sessions, discussion, and homework assignments. The TTX at the course’s end
provides a hands-on learning experience with the knowledge and skills studied
throughout the semester. All teaching materials are written in English, but
the language of instruction is local (Czech).

####  4.1.3. Student Population

All students enrolled in the course were students of Faculty of Informatics,
and the vast majority pursued their degree in cybersecurity. The class size
was up to 42 students. Most students were undergraduates. In the latest
semester, we had 34 bachelor-level students (31 of which in the cybersecurity
degree program) and 7 master-level students.

###  4.2. Field Studies Setup and Participants

This paper analyzes data and experience from three groups of trainees who
completed the TTX. Table 1 summarizes the three training sessions as the IXP
readiness increased over three years.

Table 1. Information about the three TTX runs. Run | Date | Students (team division) | Platform
---|---|---|---
#1 | Nov 25, 2021 | 19 (5 teams of 3–4 people) | Documents
#2 | Nov 23, 2022 | 36 (9 teams of 4 people) | Prototype
#3 | Nov 22, 2023 | 36 (12 teams of 3 people) | IXP

To ensure a fair comparison, we attempted to keep the three TTX runs as
consistent and similar as possible. All three runs took place within the same
course at the same stage of the semester, on the same exercise, and with the
same core instructors (though with slightly different teaching assistants).
The modality of all runs was fully in-person. The duration of the exercise was
80–90 minutes.

Before each exercise run, the TTX was thoroughly tested during a dry run with
our colleagues and senior graduate students (not students of the course). The
purpose of the test run was to verify that the platform is ready for practical
usage and that the exercise can be meaningfully completed, and to fix any
errors or issues that could negatively impact the learning experience of
trainees.

The only substantial difference between the three runs is the subject of
examination in this paper – the TTX platform readiness.

  * •

Run 1 was an imitation of a pen-and-paper exercise using shared text-based
documents on Microsoft SharePoint.

  * •

Run 2 was deployed in the first prototype of the dedicated IXP developed as a
master’s thesis (Urban, 2023).

  * •

Run 3 featured the latest version of IXP, which was substantially improved by
a dedicated development team, mainly including bug fixes and a better user
experience.

###  4.3. Research Ethics and Data Privacy

This research did not require an approval from the university’s institutional
review board. All trainees receive their course points simply for active
participation. IXP does not store any personal information that could reveal
the trainees’ identity. The data exported for analysis are anonymous and
cannot be linked to specific individuals. The trainees were informed that
their anonymized exercise activity data may be used for educational research
purposes. Lastly, post-exercise surveys were voluntary and anonymous.

###  4.4. Exercise Data Collection

Since the Run 2, IXP provides transparent, automated collection of exercise
metadata and actions of trainees. The log records are stored in the standard
JSONL format (Ward, 2023), and each record has a uniform timestamp with
microsecond precision. Per each team, IXP gathers and categorizes the records
into four log files:

  * •

Which injects did the team receive (inject_categories.jsonl).

  * •

In-exercise email communication (emails.jsonl).

  * •

Actions performed using the in-exercise tools (action_logs.jsonl).

  * •

Reached milestones (milestones.jsonl).

The format of the logs was improved for the latest version of IXP deployed for
Run 3. To enable uniform data analysis, we automatically converted the logs
from Run 2 to this new format.

###  4.5. Exercise Data Analysis

To analyze the exercise data, we used a combination of a learning analytics
dashboard built into IXP and dedicated Python scripts. The scripts process the
logs after the TTX ends in order to provide additional analytics for assessing
the team performance more deeply, such as correct/incorrect tool usage. The
scripts also enable to evaluate the TTX as a whole, by looking at metrics like
time needed to reach individual milestones.

##  5\. Results and Their Discussion

We present and compare the results of data analyses from Run 2 and 3. Since
Run 1 was executed using text-based documents, it did not yield logs in the
above-described format. All observations are tied to implications for
computing educators.

###  5.1. Analysis of the TTX Data of Trainees

####  5.1.1. Run 2

The TTX had 14 defined milestones, which captured actions such as blocking
traffic from certain sources, communicating with the affected users, and
notifying responsible parties. The teams reached between 5 and 12 milestones,
with an average of 10 (71%). Only 2 out of the 9 teams scored below average,
indicating they may have benefited from an intervention by an instructor.

The TTX provided the teams with 7 possible tools. The team that reached the
fewest milestones also used the least amount of tools (6 times in total
compared to the overall team average of 20 occurrences of tool usage).
However, the team that reached the second smallest number of milestones had
the second largest number of tool usages. While other explanations are
possible, this may indicate that rare tool usage is associated with low
exercise completion, but frequent tool usage does not necessarily imply
success (the tools might not be used efficiently).

####  5.1.2. Run 3

In order to improve the granularity of capturing teams’ actions, we added 8
additional milestones to the TTX. When looking just at the 14 original
milestones, the teams reached between 4 and 11 of them, with an average of 8
(57%). Compared to Run 2, this lower ratio indicates that if an instructor
provided a post-exercise debrief, it might be a valuable learning experience
for all trainees. This debrief would inform the trainees about the additional
actions that could have been made but were missed.

Overall, the first reached milestone was to visit the compromised website. The
first team that reached this milestone did so in around 8 minutes from the TTX
start. This milestone was also the fastest to reach across all teams, in 15.5
minutes on average. However, the slowest team to achieve this milestone took
30 minutes. Instead, this team focused on four other milestones before,
prioritizing different aspects of the TTX compared to the vast majority of
teams. This can be an interesting observation for the instructor, showing
possibly alternative approaches to solving the in-exercise problems.
Regardless, this team took rather long to reach their first milestone, which
suggests they may have benefited from a hint or intervention.

The milestones that took the longest to complete encompassed the communication
with the simulated employees. Four teams took a little more than an hour to
address their stakeholders, and this step was completely overlooked by five
teams. Although this skill is non-technical, it is still an important part of
the cyber incident responders’ work role. Therefore, instructors can use these
insights from the TTX data to remind the learners about this responsibility or
revise the course content in this aspect.

The TTX provided the teams with 11 possible tools (additional 4 compared to
Run 2). A team used a tool between 10 and 46 times throughout the entire TTX,
with 31 uses on average (including repeated uses of the same tool). The
approaches of individual teams differed vastly: different teams used certain
tools more often and (almost) ignored other tools. For example, all teams used
the tool to block traffic incoming from a certain IP address, but only two-
thirds of the teams blocked traffic outgoing to the compromised website.

Finally, the improvement of IXP for Run 3 enabled to evaluate the syntactical
correctness of tool usage. When looking at these data, all tools have much
more correct rather than incorrect invocations, showing that all trainees
understood the tools’ interfaces. However, the DNS lookup tool has
substantially higher percentage of erroneous applications compared to other
tools. Within the errors that were not simply typos, there might have been
confusion among the trainees that could be addressed by the instructor.

Looking at the teams’ written communication, they engaged in 6 email threads
on average. The team that communicated the most (9 threads) reached the most
milestones, and vice versa, the team that communicated the least (3 threads)
reached almost the least number of milestones. This provides a teaching
opportunity if the instructor compares the differences between the teams,
showing that active communication is crucial while resolving a crisis.

####  5.1.3. Summary

The automatic collection and analysis of data provided by the IXP equips
instructors and researchers with valuable insights that would be difficult to
obtain otherwise, especially in the traditional pen-and-paper TTX format. By
adding more granularity to the milestones and enhancing the platform’s logging
capabilities, we were able to observe deeper insights in Run 3 compared to Run
2. These include difficult milestones and errors in tool usage.

###  5.2. Trainees’ Learning Experience in IXP

To complement the analysis of exercise logs, we present the results of a post-
exercise survey administered to all trainees after Run 3.

In the overall evaluation, 35 out of 36 learners considered the TTX scenario
realistic. A majority, 29 out of 36, found the TTX beneficial for practical
applications because it improved their understanding of incident handling. One
participant stated, “At the beginning, we were quite lost. There is just so
much difference between having a specified incident handling task and having
to figure out everything by yourself.” Additionally, 31 participants expressed
satisfaction with the ease of use of IXP to facilitate the exercise.

Our survey unveiled three pivotal insights for refining future exercises.
Primarily, we encountered challenges in effectively communicating which in-
exercise email addresses are trustworthy. Consequently, specific teams
refrained from accessing some exercise emails, deeming them potentially
malicious.

Secondly, trainees would like IXP’s email feature to resemble familiar
interfaces more. The current version can send and receive emails, but the
trainees expected more features, like auto-saving drafts or showing emails in
threads. The absence of such features led to communication delays, influencing
the learning experience.

Finally, some teams got stuck in various stages of the scenario. Instructors
were briefed to assist by sending exercise emails to guide these teams.
However, this approach proved challenging as instructors struggled to identify
the right moments for intervention. Given the continuous team discussions,
instructors found it hard to determine when it was appropriate to influence
the discussion.

###  5.3. Instructors’ Teaching Experience in IXP

During a focus group discussion hosted with the instructors after Run 3, the
instructors observed that enhancing IXP led to improving the following two key
aspects in the teaching practice:

  * •

Reliability: When using shared documents in Run 1, students sometimes
accidentally rewrote their past conversation, and instructors got confused
when working with multiple teams. A dedicated learning environment eliminates
these errors.

  * •

Involvement: Run 1 and 2 had fewer teams, almost exclusively with 4 people.
With the improvement of IXP for Run 3, we were able to have more teams, almost
exclusively with 3 people. This means that a single student got more
opportunities to speak in the team and to be actively involved in the
decision-making, improving their individual experience.

##  6\. Conclusion

Tabletop exercises are a promising method for innovating computing courses.
They enable students to exercise collaborative problem-solving in the context
of cybersecurity, IT governance, and other domains of applied informatics.

Introducing the INJECT Exercise Platform, a dedicated learning environment for
TTXs, alleviates many challenges that instructors face. For example, having a
platform to automate repetitive tasks, such as providing injects or outputs of
tools, enables instructors to focus on the exercise facilitation. The
automation capabilities of the learning environment also enable further
educational research.

We release the IXP as open-source software with an example exercise at
https://inject.muni.cz. The research data, Python scripts for data processing,
and complete results are also available at
https://gitlab.fi.muni.cz/inject/papers/2024-iticse-from-paper-to-platform.

###  6.1. Open Research Challenges

Currently, it is difficult to identify the right moments for intervention
during TTXs. A key challenge is how to determine when a team would benefit
from a hint, using insights from exercise data. For example, measuring
expected time to reach a milestone can imply how long the instructor should
wait before giving a team a hint. This would help teams to navigate through
scenario challenges.

Another limitation is that IXP does not yet support instructors in quickly
reacting to expected trainee responses. Therefore, future work can explore
machine learning and natural language processing techniques to evaluate the
similarity in the responses to injects between different teams. Then, the
platform can provide instructors with pre-defined responses that would suit
the trainees’ inputs.

Finally, future work should use the data to measure team performance (Amon et
al., 2019), evaluate students’ achievement of learning objectives, and
experimentally determine the effect of IXP on skill acquisition.

###### Acknowledgements.

This research was supported by the Open Calls for Security Research 2023–2029
(OPSEC) program granted by the Ministry of the Interior of the Czech Republic
under No. VK01030007 – Intelligent Tools for Planning, Conducting, and
Evaluating Tabletop Exercises.

## References

  * (1)
  * Amon et al. (2019) Mary Jean Amon, Hana Vrzakova, and Sidney K. D’Mello. 2019.  Beyond Dyadic Coordination: Multimodal Behavioral Irregularity in Triads Predicts Facets of Collaborative Problem Solving.  _Cognitive Science_ 43, 10 (2019), e12787.  https://doi.org/10.1111/cogs.12787
  * Angafor et al. (2023) Giddeon N. Angafor, Iryna Yevseyeva, and Leandros Maglaras. 2023.  Scenario-based incident response training: lessons learnt from conducting an experiential learning virtual incident response tabletop exercise.  _Information and Computer Security_ 31, 4 (2023), 404–426.  https://doi.org/10.1108/ICS-05-2022-0085
  * Brilingaitė et al. (2017) Agnė Brilingaitė, Linas Bukauskas, Virgilijus Krinickij, and Eduardas Kutka. 2017.  Environment for Cybersecurity Tabletop Exercises. In _11th European Conference on Games Based Learning (ECGBL 2017)_. Academic Conferences Ltd, Graz, Austria, 47–55.  https://www.researchgate.net/publication/320244434_Environment_for_Cybersecurity_Tabletop_Exercises
  * CC2020 Task Force (2020) CC2020 Task Force. 2020.  _Computing Curricula 2020: Paradigms for Global Computing Education_.  Association for Computing Machinery, New York, NY, USA.  https://doi.org/10.1145/3467967
  * Elsevier (2023) Elsevier. 2023.  Scopus.  Retrieved December 13, 2023 from https://www.scopus.com
  * European Union Agency for Cybersecurity (2010) (ENISA) European Union Agency for Cybersecurity (ENISA). 2010.  _Good Practice Guide for Incident Management_.  Technical Report. ENISA.  Retrieved December 13, 2023 from https://www.enisa.europa.eu/publications/good-practice-guide-for-incident-management
  * European Union Agency for Cybersecurity (2012) (ENISA) European Union Agency for Cybersecurity (ENISA). 2012.  National and International Cyber Security Exercises: Survey, Analysis & Recommendations.  Retrieved December 13, 2023 from https://www.enisa.europa.eu/publications/exercise-survey2012?v2=1
  * European Union Agency for Cybersecurity (2015) (ENISA) European Union Agency for Cybersecurity (ENISA). 2015.  Latest Report on National and International Cyber Security Exercises.  Retrieved December 13, 2023 from https://www.enisa.europa.eu/publications/latest-report-on-national-and-international-cyber-security-exercises
  * Filigran (2023) Filigran. 2023.  _OpenEx Platform_.  GitHub.  Retrieved December 13, 2023 from https://github.com/OpenEx-Platform/openex
  * Fiore et al. (2018) Stephen M Fiore, Arthur Graesser, and Samuel Greiff. 2018.  Collaborative problem-solving education for the twenty-first-century workforce.  _Nature human behaviour_ 2, 6 (2018), 367–369.  https://doi.org/10.1038/s41562-018-0363-y
  * for Cybersecurity Careers and (NICCS) National Initiative for Cybersecurity Careers and Studies (NICCS). 2020.  Incident Response.  Retrieved December 13, 2023 from https://niccs.cisa.gov/workforce-development/nice-framework/specialty-areas/incident-response
  * Global (2023) Privasec Global. 2023.  _Cybersecurity Table Top Exercise (TTX)_.  Privasec Global.  Retrieved December 13, 2023 from https://privasec.com/ttx-cybersecurity-tabletop-exercise/
  * Goat (2023) Red Goat. 2023.  Training, exercising and consultancy to help defend your organisation against cyber threats.  Retrieved December 13, 2023 from https://red-goat.com/
  * Grance et al. (2006) Tim Grance, Tamara Nolan, Kristin Burke, Rich Dudley, Gregory White, and Travis Good. 2006.  _Guide to Test, Training, and Exercise Programs for IT Plans and Capabilities_.  Technical Report. NIST.  https://doi.org/10.6028/NIST.SP.800-84
  * Hsieh et al. (2023) Hui-Wen Hsieh, Chia-Shan Wu, Chun-Chin Tsai, Yen-Chi Liao, Pin-Yu Chen, Hui-Ling Tseng, Mei-Zen Huang, and Mei-Fang Chen. 2023.  Comparing the effectiveness of board game-based and drill-based education programs in improving Taiwanese nurses' fire safety knowledge, attitudes, and behavior: A quasi-experimental study.  _Nurse Education Today_ 129 (Oct. 2023), 105919.  https://doi.org/10.1016/j.nedt.2023.105919
  * Kopustinskas et al. (2020) Vytis Kopustinskas, Bogdan Vamanu, Marcelo Masera, Rimantas Šikas, Julia Vainio, Romualdas Petkevičius, and Lawrence Walzer. 2020.  Tabletop Exercise as a Tool to Foster Resilience in the Energy Sector: Lessons Learned in the Baltic States. In _Proceedings of the 30th European Safety and Reliability Conference and 15th Probabilistic Safety Assessment and Management Conference_. Research Publishing Services, Venice, Italy, 255–262.  https://doi.org/10.3850/978-981-14-8593-0_4019-cd
  * Lewis (2022) Nathan Lewis. 2022.  _TableTop Exercise_.  GitHub.  Retrieved December 13, 2023 from https://github.com/GreeneGunnar14/TableTop_Exercise
  * Mareš et al. (2023) Miroslav Mareš, Roman Chytilek, Zuzana Špačková, Jakub Drmola, Lenka Hrbková, Petra Mlejnková, and Michal Tóth. 2023.  Assessment of performance during cybersecurity tabletop exercises.  _Security Journal_ (2023), 24 pages.  https://doi.org/10.1057/s41284-023-00391-4
  * McKeown (2022) Justin McKeown. 2022.  _Ransomware Simulator_.  GitHub.  Retrieved December 13, 2023 from https://github.com/justinmckeown/ransomware-simulator
  * Microsoft (2023) Microsoft. 2023.  Microsoft Office.  Retrieved December 13, 2023 from https://www.microsoft.com/en/microsoft-365/
  * Ota et al. (2022) Yuitaka Ota, Erika Mizuno, Tomomi Aoyama, Yoshihiro Hashimoto, Ichiro Koshijima, Haruna Asai, and Shiho Taniuchi. 2022.  Designing Framework for Tabletop Exercise to Promote Resilience Against Cyber Attacks.  In _14th International Symposium on Process Systems Engineering_. Computer Aided Chemical Engineering, Vol. 49. Elsevier, Kyoto, Japan, 1471–1476.  https://doi.org/10.1016/B978-0-323-85159-6.50245-1
  * Ottis (2014) Rain Ottis. 2014.  Light Weight Tabletop Exercise for Cybersecurity Education.  _Journal of Homeland Security and Emergency Management_ 11 (2014), 579–592.  Issue 4. https://doi.org/10.1515/jhsem-2014-0031
  * Urban (2023) Michal Urban. 2023.  _A Tool for Conducting Tabletop Exercises_.  Master thesis. Masaryk University, Faculty of Informatics.  https://is.muni.cz/th/xwkjh/?lang=en/ Supervisor: Jan Vykopal.
  * U.S. Cybersecurity & Infrastructure Security Agency (CISA) U.S. Cybersecurity & Infrastructure Security Agency (CISA) 2015.  _Communications-Specific Tabletop Exercise Methodology_.  U.S. Cybersecurity & Infrastructure Security Agency (CISA).  https://www.cisa.gov/sites/default/files/publications/CommunicationsSpecificTabletopExerciseMethodology_0.pdf
  * Vykopal et al. (2021) Jan Vykopal, Pavel Čeleda, Pavel Seda, Valdemar Švábenský, and Daniel Tovarňák. 2021.  Scalable Learning Environments for Teaching Cybersecurity Hands-on. In _Proceedings of the 51st IEEE Frontiers in Education Conference_ (Lincoln, NE, USA) _(FIE ’21)_. IEEE, New York, NY, USA, 1–9.  https://doi.org/10.1109/FIE49875.2021.9637180
  * Vykopal et al. (2024) Jan Vykopal, Pavel Čeleda, Valdemar Švábenský, Martin Hofbauer, and Martin Horák. 2024.  Research and Practice of Delivering Tabletop Exercises. In _Proceedings of the 29th Conference on Innovation and Technology in Computer Science Education_ _(ITiCSE ’24)_. Association for Computing Machinery, New York, NY, USA, 7 pages.  https://doi.org/10.1145/3649217.3653642
  * Ward (2023) Ian Ward. 2023.  Documentation for the JSON Lines text file format.  Retrieved December 13, 2023 from https://jsonlines.org/
  * Yamin et al. (2020) Muhammad Mudassar Yamin, Basel Katt, and Vasileios Gkioulos. 2020.  Cyber ranges and security testbeds: Scenarios, functions, tools and architecture.  _Computers & Security_ 88 (2020), 101636.  https://doi.org/10.1016/j.cose.2019.101636