# Introducing and Interfacing with Cybersecurity – A Cards Approach

Ryan Shah, Manuel Maarek, Shenando Stals, Lynne Baillie, Sheung Chi Chan,
Robert Stewart, Hans-Wolfgang Loidl and Olga Chatzifoti† Heriot-Watt
University, Edinburgh and Glasgow School of Art†UK<EMAIL_ADDRESS>
(2023)

###### Abstract.

Cybersecurity is an important topic which is often viewed as one that is
inaccessible due to steep learning curves and a perceived requirement of
needing specialist knowledge. With a constantly changing threat landscape,
practical solutions such as best-practices are employed, but the number of
critical cybersecurity-related incidents remains high. To address these
concerns, the National Cyber Security Centre published a Cybersecurity Body of
Knowledge (CyBOK) to provide a comprehensive information base used to advise
and underpin cybersecurity learning. Unfortunately, CyBOK contains over 1000
pages of in-depth material and may not be easy to navigate for novice
individuals. Furthermore, it does not allow for easy expression of various
cybersecurity scenarios that such individuals may be exposed to. As a solution
to these two issues, we propose the use of a playing cards format to provide
introductory cybersecurity knowledge that supports learning and discussion,
using CyBOK as the foundation for the technical content. Upon evaluation in
two user studies, we found that 80% of the participants agreed the cards
provided them with introductory knowledge of cybersecurity topics, and 70%
agreed the cards provided an interface for discussing topics and enabled them
to make links between attacks, vulnerabilities and defences.

security, playing cards, knowledge base, design

††copyright: acmcopyright††journalyear: 2023††doi:
XXXXXXX.XXXXXXX††conference: European Symposium on Usable Security; October
16–17, 2023; Copenhagen, Denmark††price: 15.00††isbn:
978-1-4503-XXXX-X/18/06††ccs: Security and privacy Human and societal aspects
of security and privacy††ccs: Applied computing Education

##  1\. Introduction

Cybersecurity remains a fundamental concern to users of computer systems, with
security often being overlooked due to its portrayal as a subject pertaining
to issues of perceived technical difficulty, steep learning curves and a
requirement of specialist knowledge and/or expertise (Harknett and Stever,
2009; Hoffman et al., 2011; Asen et al., 2019). While the security foundations
of computer-based systems have improved over time, limiting the potential for,
or mitigating the effects of, attacks arising from vulnerabilities, requires
the involvement of all users of these systems (e.g. the general population)
and is a necessary step to improve the understanding of cybersecurity (Adams
and Sasse, 1999). Moreover, the increasing complexity and diversity of the
threat landscape for cybersecurity (Wall, 2017; Kaloudi and Li, 2020; Bahizad,
2020) further substantiates the need for improving understanding of
cybersecurity. In the domain of software engineering, practical solutions to
achieve this include activities such as the documentation of vulnerabilities
of computer systems and updating respective knowledge bases. Open databases
such as the Common Vulnerabilities and Exposures (CVE) (cve, [n.d.]) and
Common Weakness Enumeration (CWE) (Martin et al., 2011), have played a pivotal
role in raising the awareness of known vulnerabilities such that appropriate
defensive measures can be developed or updated. While these reference
databases are well maintained, they may still appear complex to the general
population and may contribute to the already existing problems of
inaccessibility and specialist requirements that are pinned against the topic
of cybersecurity. Because of this, several knowledge bases have been developed
to inform and underpin cybersecurity education and training (Martin et al.,
2021; Newhouse et al., 2017; ncs, 2017), which aim to address these issues at
a high-school or higher-education level. Although they may be a useful
learning resource for providing key cybersecurity knowledge, their primary
purpose is to be used by those who are already knowledgeable in cybersecurity
to develop further curricula to teach those who may have little-to-no
knowledge of cybersecurity. Furthermore, among these knowledge bases, there
may be some key topics which are not covered and their format and density may
not be perceived as accessible to novice users. Thus, this may directly impact
one’s ability to understand key cybersecurity topics but also to make links
between these topics to capture real-world cybersecurity scenarios.
Ultimately, the weaknesses of existing solutions regarding limitations of
accessibility, steep learning curves and a perceived requirement of specialist
knowledge/expertise, must be ameliorated by a new solution that provides an
answer to the following research questions. Specifically, can a new solution:

  1. RQ1

Provide introductory cybersecurity knowledge to novice users?

  2. RQ2

Provide material for expressing interpretation and documentation of key
cybersecurity topics, which can support independent learning and self-
efficacy?

  3. RQ3

Act as an index for the CyBOK knowledge base which provides an interface for
discussion on key cybersecurity topics?

  4. RQ4

Provide links between key cybersecurity topics, allowing the generation of
concepts which can capture various cybersecurity scenarios?

In this paper, we provide an answer to these research questions by proposing
the use of a playing cards format as a medium to provide: introductory
knowledge of key cybersecurity topics, acting as an index for the CyBOK
knowledge base (cyb, [n.d.]b; Martin et al., 2021); support independent
learning and self-efficacy; and allow for links to be made between key
cybersecurity topics to capture real-world scenarios. The novelty of this work
is three-fold. We first present the design principles for the cybersecurity
cards to address these limitations. Second, we provide an evaluation of the
cards in a workshop with masters-level students to understand whether the
cards satisfy the aforementioned provisions. The output of this evaluation is
a second revised deck of the cybersecurity cards. Third, we carried out the
same workshop but with a different demographic to the first, with participants
at late primary and early secondary school level (ages ranging from 10 to 15
years old, mean 12.8 years).

The remainder of this paper is organised as follows. Section 2 provides
background and related work, as well as the selection procedure we applied to
the production of our cybersecurity cards using the CyBOK knowledge base and
the limitations of other approaches. The design principles applied to the
cybersecurity cards, as well as the initial implementation (Version 1), are
described in Section 3. An evaluation of Version 1 of the cards is provided in
Section 4. In Section 5, we present Version 2 of the cards as a result of the
findings from the first evaluation, as well as a further evaluation of the
second version of the cards in Section 6. In Section 7, we provide a
discussion of the results from both evaluations and the paper concludes in
Section 8.

##  2\. Background

The need for practical and easy-to-learn cybersecurity learning material is a
constant problem which stems from the evolving nature of cybersecurity and
computing technologies as the number of connected users and devices scales. In
recent years, the number of critical cybersecurity incidents have increased
significantly, correlating with increasing numbers of online users during the
Covid-19 pandemic, for example, as well as an increase in the adoption of
various connected computer systems in day-to-day activities. Among these
incidents, research shows that around 95% of cybersecurity breaches occur as a
result of human error (wef, 2020) and that organisations lack the
sophistication, interest and/or knowledge to handle these threats (cis, 2020;
sop, 2021).

It has been shown that those in cybersecurity careers require a set of skills,
involving the abilities to carry out various tasks at any time in non-
traditional environments, and adapt to the dynamic nature of these
environments (nic, 2015). In the domain of software engineering, basic
cybersecurity training such as password best-practices and multi-factor
authentication are employed for individuals to conform to, with the aim of
alleviating concerns and mitigating the potential for liabilities that arise
as a result of cybersecurity-related incidents (Walters and Matulich, 2011;
Alotaibi and Almagwashi, 2018; Sarginson, 2020). It has been identified that a
large number of Android applications contain security-related code snippets
copied and pasted from Stack Overflow, of which nearly 98% contained at least
one insecure code snippet (Fischer et al., 2017). The value of security
information depends strongly on its source (Rader and Wash, 2015) and
reputable information sources are only useful, so long as they are well-
understood and perceived as actionable (Thomas et al., 2019; Redmiles et al.,
2020). While sites such as StackOverflow are reputable for providing
actionable solutions, it is clear that the security of solutions are not well
understood. For novice individuals, such as those who write and/or deploy
software code without formal software engineering training, it may not be true
that they may fully comprehend the impact of not adhering to security best-
practices.

To address this, various curricula guidelines and knowledge frameworks have
been developed for cybersecurity, covering a range of fundamental topics
ranging from software and hardware security, to networks and cyber-physical
systems. The Joint Task Force (JTF) on Cybersecurity Education proposed a
draft of curricular guidance on cybersecurity to support educational efforts
in the USA (Bishop et al., 2017). They designed a framework model for a body
of knowledge that covers six knowledge areas which several concepts span over,
targeting specific disciplines and application areas that pertain to the
demographic of cybersecurity professionals. The National Initiative for
Cybersecurity Education (NICE) (Newhouse et al., 2017) is a cybersecurity
workforce framework, developed by NIST in the USA, which aims to provide a
foundation for describing and sharing information about knowledge, skills and
abilities in cybersecurity to strengthen an organisation’s cybersecurity. The
National Cyber Security Center (NCSC) in the UK proposed a Certified Master’s
Program that defines several pathways to address knowledge and skill gaps in
cybersecurity education, which describe what topics must be covered and to
what depth (ncs, 2017). While all these frameworks tend to agree on key
cybersecurity topics that must be understood, they only promote greater
emphasis on a subset of topics. For example, NICE covers a wide range of key
topics but gaps exist such as with topics related to cyber-physical systems
and human factors. The NCSC Certified Master’s Program does not place much
emphasis on attacks and defences, but in contrast focuses on key topics such
as software security.

The Cybersecurity Body of Knowledge (CyBOK) is a knowledge base developed by
University of Bristol funded by the NCSC. It was developed to encompass the
wide variety of topics within the field of cybersecurity and to show that it
also spans multiple disciplines. In practice, it has been successful in
providing a framework for NCSC certified degrees and academic/professional
training programmes (cyb, [n.d.]a). CyBOK is decomposed into 21 knowledge
areas (KAs) (as of version 1.1), each introduced by a reference document and a
set of topics presented as a branch of the overall Knowledge Tree (Figure 1)
(cyb, [n.d.]b). Each of these knowledge areas are organised into a hierarchy
of between 3 to 5 categories that present as a tree of topics. For each KA in
CyBOK, there are a number of chapters that form an encyclopedic collection of
knowledge of key concepts that are based on state-of-the-art academic
literature. These key concepts are known as Topics, with some Topics
decomposed further into a set of more specialised subjects (Sub-Topics). For
example, the category of Software Security in the Software and Platform
Security KA contains 4 overarching themes, split into 20 sub-topics (e.g.
structured output generation vulnerabilities), each of which describe further
specialised information (e.g. sql injection).

Figure 1.  Partial View of CyBOK 1.1 Knowledge Tree (CyBOK, 2021). The
knowledge areas and topics that are highlighted show the subset of the CyBOK
knowledge base that was selected due to the link to the domain of software
engineering.

It has been shown that, in comparison with other knowledge frameworks, CyBOK
covers a wider range of knowledge areas and does not have gaps that are
present within other frameworks (Hallett et al., 2018). While CyBOK
facilitates a body of knowledge which attributes to the production of material
for cybersecurity education and professional training, there are some
weaknesses which may render it an inaccessible resource to more novice
individuals such as those in the domain of software engineering who write or
deploy code with no formal software engineering training. First, the links
between meaning and relationships among topics and sub-topics vary across the
entire Knowledge Tree, which prevents easy expression of various cybersecurity
scenarios. Second, the material across the CyBOK knowledge base and its
indexing structure is not easy to traverse for novice users. Gonzalez et al.
(González-Tablas and Rashed, 2022) show that it would be difficult for novice
individuals to infer the links between various topics, given that some follow
either a single predominant theme or span several topics themselves.
Ultimately, to support novice users as well as those more experienced, key
cybersecurity knowledge provided by knowledge bases such as CyBOK require
adequate presentation that can facilitate independent learning whilst also
providing a suitable interface for discussion of various cybersecurity
scenarios to make the links between meaning and relationships among topics.

Aside from knowledge frameworks, cybersecurity information has also been
presented in other ways. Capture the Flag (CTF) activities provide a series of
competitive exercises used to find vulnerabilities in computer systems and
applications and have shown to be a valuable learning tool (Trickel et al.,
2017; Schreuders, 2022; Swann et al., 2021). Thomas et al. (Thomas et al.,
2019) propose the use of a collectible card game (CCG) as a means of teaching
cybersecurity to high school students, given the benefits of prevalence
culturally to all age groups (familiarity) and encouraging the understanding
of competitive strategy and mistake-making as a way of learning (Turkay et
al., 2012). Anvik et al. (Anvik et al., 2019) propose the use of a web-based
card game for learning programming and cybersecurity concepts, using simple
vocabulary to create ubiquitous learning experiences. Denning et al. (Denning
et al., 2013) propose the use of a tabletop card game, Control-Alt-Hack, with
the aim of providing awareness training for cybersecurity, arguing that
playing card games can provide a reachable foundation for providing digestible
cybersecurity information to large audiences. However, while these gamified
approaches show various levels of success, there are limitations. First, many
of these different approaches have different target personas and goals.
Second, card game approaches such as Control-Alt-Hack (Denning et al., 2013)
do not cover a broad range of key cybersecurity topics, such as those
identified by knowledge frameworks such as CyBOK, and do not adequately
highlight the links between vulnerabilities, attacks and defences.
Specifically, attacks are typically highlighted first, which does not help
users understand how attacks present themselves (opportunistic vulnerability
targeting) and how to protect against them. Third, while CTF activities, for
example, are beneficial in this aspect (Swann et al., 2021; Trickel et al.,
2017), a key disadvantage pertains to novice users wherein competitions rely
on technical expertise and the ability to traverse computer systems using
various command-line tools and other bespoke applications (Ford et al., 2017),
or requiring (at a minimum) a basic understanding of cybersecurity concepts in
order to progress in finding vulnerabilities (McDaniel et al., 2016).

##  3\. Cybersecurity Cards – Version 1

To answer the research questions presented in Section 1, we propose a novel
approach to represent key cybersecurity knowledge, utilising specially
designed playing cards. In recent years, it has been shown that specially
designed cards used as a tool for education attributes to positive outcomes in
the space of learning, attitudes and critical thinking skills (Kordaki and
Gousiou, 2016, 2017). With a large portion of the general population familiar
with card-based games, it is clear that a playing card format is a good
medium, both physically and digitally, with relatively intuitive logic if they
are designed well. The cards proposed in this work were developed by a team of
experts in the fields of cybersecurity, human-computer interaction and games
experts. A subset of the cards we propose in this section have been made
available for viewing and can be found
online111https://anonymous.4open.science/r/cybersecurity_cards-9F00/
(anonymous repository for double-blind reviewing purposes). The full set of
cards will be made publicly available under the CC BY-NC-SA Creative Commons
license after the research project ends in January 2024.

###  3.1. Focus of Knowledge

In order to establish a good grounding for an evaluation of a card-based
solution to address the proposed research questions, an important factor is to
determine what information the cards represent ($RQ1$). In this work, we
leverage the information found in the CyBOK knowledge base and produce the
cybersecurity cards to address the limitations of the knowledge base.
Specifically, the focus of knowledge we chose to take is in the domain of
software engineering and secure coding, given recent increases in threats
pertaining to this domain (Saxena et al., 2020; Pranggono and Arabo, 2021) and
the observation of finding people with skills in secure coding reported as the
most difficult task (WILLETTS, 2014). The topics that fall within the scope of
the focus of knowledge in this work were derived from a filtering and
selection procedure, where the CyBOK knowledge tree was traversed from
Knowledge Areas to Topics. Given that not all KAs in CyBOK are related to the
domain of software engineering, we selected topics that relate to this theme.
As highlighted in Figure 1, we chose the following knowledge trees to
traverse: Human Factors, Malware & Attack Technologies, Software Security, Web
& Mobile Security and the Secure Software Lifecycle. For each KA, we follow
each branch of the tree to the leaf nodes. For each of the leaves, we pick up
the topic and provide a better description of that name to create the first
version of the card. If there are duplicates or highly similar topics in the
CyBOK mapping, these are either merged together or split into distinct types.
For example, CyBOK’s information on Cross-Site Request Forgery is mapped to
two vulnerabilities: Inadequate Data Authenticity and Origin Verification and
Inadequate Session Expiration. Any leaves that fall within Topics that, for
example, may not follow a single predominant theme and do not relate to the
selected focus of knowledge are cut out. The category of Software & Platform
Security is directly related to the scope of this work whereas others are not,
such as Forensics which involves identifying and analysing data to support
legal proceedings. This process was carried out by two academic researchers in
cybersecurity who decided on the final first set of cards in consensus.

Figure 2. Organisation and Design of Cybersecurity Cards (Version 1). General
cards contain the icon of its corresponding class (1), with Detailed cards
containing an identifier code (2) which is used to make links between attacks,
defences and vulnerabilities (3). The arrows demonstrate the links between the
presented attacks, vulnerabilities and defences.

###  3.2. Design and Structure

The playing cards format is easy to handle in both a physical or digital
medium (Altice, 2014) and are designed to be handled individually or combined
into a set of cards (known as a deck). The deck of cards is first split into
three distinct classes: Vulnerability, Attack and Defence, following suit from
common threat modelling approaches and security catalogues that encompass the
vulnerability-attack-defence trichotomy (Kordy et al., 2014; Ferro et al.,
2021; Tatam et al., 2021; cve, [n.d.]; Martin et al., 2011). Within each of
these classes, the deck is further split into General and Detailed cards
(Figure 2). General cards represent a type of one of the classes (e.g. Code),
while Detailed cards provide lower-level, concrete examples that relate to the
General card. In terms of design (Figure 2), attack and vulnerability cards
are encased in a square border, with attack cards filled with a solid red
colour while vulnerability cards are signalled with a diagonal line to
separate white from red. The defence cards resemble an octogonal shape (akin
to a shield) which has a white fill colour. The diagonal separation of white
and red in vulnerability cards aims to highlight vulnerabilities as the
central focal point. General cards in the deck include: a title representing
the KA topic; a type represented by a symbol (see 1); and a description
related to the topic it represents. Detailed cards are assigned a unique
identifier in the form $a\\_Bi$ in the top right corner 2, where $a$ refers to
the class of the Detailed card, $B$ referring to the first letter of the
General card it is categorised under and $i$ acting as an index number in the
set of $N$ Detailed cards for the category $B$. The symbol for the Detailed
card’s category is made opaque in the center of the card, behind the
description of the Detailed card. Attack cards also contain a description of
the impact of the attack. Defence cards contain a target symbol to help
further identify the vulnerability it aims to protect against attacks.
Vulnerability cards also describe an attack vector with associated attack card
identifiers, as well as the consequence of the vulnerability with associated
defence card identifiers 3. The aim of the identifiers is to provide a means
to make links between key cybersecurity topics by allowing users to capture
various attack-defence scenarios that revolve around specific vulnerabilities
($RQ4$). In total, there are 124 cards in the deck which is composed of 30
vulnerability cards, 32 attack cards and 47 defence cards, each of which are
categorised under one of the 15 General cards.

####  3.2.1. Relationships Between Cards

Given that cybersecurity stems from conflict between attackers and defenders
targeting one or more vulnerabilities, creating a capturable many-to-many
relationship is essential when introducing cybersecurity concepts. Thus, the
cybersecurity cards should represent this relationship, where multiple attacks
can target multiple vulnerabilities that can, in turn, be mitigated or
countered by multiple defences. While CyBOK does contain information about
these relationships, it is hard to infer these from implicit references within
the reference material. Thus, the cards aim to act as an index for the CyBOK
knowledgebase by providing a means for presenting these implicit relationships
in a manner that supports independent learning and self-efficacy ($RQ2,RQ3$).
In Figure 2, we can see that the vulnerability card ”Blind Trust of User
Input” is linked to a set of identifier codes 3 for a number of related
attacks and defences. This example shows a link to the ”Command Injection” and
”Memory Thief” attacks ($RQ4$). The first attack involves the execution of
unauthorised commands to a system (which may be input by a user) and the
second involves the stealing of confidential information from memory. In
contrast, two defence examples are shown as links to the attacks and the
vulnerability, which include ”Code Assertions” and ”Input Sanitisation” which
involve monitoring code execution and sanitising user input to eliminate
malicious code or escape characters, respectively.

##  4\. Evaluation

The logical next step after designing the cybersecurity cards was to conduct a
user evaluation to see if the participants could use the cards, and whether
the cards provided a clear communication of these topics and an interface for
discussing them.

###  4.1. Methodology

Participants were recruited to take part in a workshop, advertised to
university students, where they interacted with the cybersecurity cards in
order to analyse and devise a cybersecurity scenario. Ethical approval was
granted by the University ethics committee for the workshop recruitment and
procedure before the study took place. A participant information sheet was
provided via a link in the online registration form, as well as being used to
obtain written consent from participants. Furthermore, the workshop was held
in Spring 2022 and we had a special clause added to state we would follow
local government guidelines on the Covid-19 protocol (cov, 2022) which
provided guidance on reducing risks from transmitting Covid-19, such as
staying home if participants had symptoms or tested positive for Covid-19. In
total, we managed to recruit 13 participants but only 11 had usable data. Upon
registration for the workshop, we found that the 11 participants were masters-
level students who were between 22 and 35 years old (mean 26.3 years). The
participants were enrolled in a conversion masters program in computer science
and came from either an engineering (6), mathematics (3), computing (1) or
biology (1) background. When asked if the participants had any prior
experience or skills with cybersecurity, only one participant mentioned they
had experienced anomaly detection but others stated they had no prior
experience or skills in cybersecurity. Further, when asked if they had any
experience or skills with secure coding, 8 of the participants said they had
no experience or skills with secure coding, 2 responded neutral and 1 said
they had some experience. We hypothesise that the low number of participants
to be attributed to the Covid-19 global pandemic leaving individuals with more
dynamic priorities and commitments.

Phase |  Activity | Rationale | Duration (min)
---|---|---|---
Prep |  Participant information sheet | Inform participants | 5
|  Informed consent form | Inform consent | 5
|  Demographic Questionnaire | Participant profiles | 5
Activity |  Introduction to workshop theme | Introduce theme | 10
|  Introduction to cybersecurity cards and experts roaming around the room providing help when needed | Familiarity with cards | 10
|  Theme exploration – devising attack and defence scenarios | Cybersecurity trichotomy | 20
Post |  Cybersecurity Questionnaire | Evaluate cards | 10
Table 1. Workshop Structure Showing Phases, Activities and Duration (min)

An overview of the workshop structure can be seen in Table 1. Participants
were given the information sheet, consent form and demographic questionnaire
to fill out in advance of the workshop, with copies of the consent form
brought on the day as back ups in case they forget to send or bring the
consent form. When they arrived they were split into three groups (consisting
of three or four participants). During the workshop, participants were first
introduced to the theme of the workshop which was Code Security – the practice
of developing software that embeds security and best-practices into the code.
After this, they were given an introduction to the deck of cybersecurity cards
where they had time to explore and familiarise themselves with the various
attacks, defences and vulnerabilities. Cybersecurity experts that were present
would roam around the room and were able to answer any questions participants
may have had when going through the deck. After this, they would use the
cybersecurity cards to find vulnerabilities and devise attack and defence
scenarios within the theme of the workshop. At the end of the workshop, each
participant was individually asked to fill in a self-assessment questionnaire
online (Appendix A) on how using the cards contributed to the themes described
in Section 4.2. The rationale behind this questionnaire was to collect both
quantitative and qualitative data regarding a participant’s experience of
interacting with the cybersecurity cards. The quantitative aspect of the
questionnaire (Q1) makes use of a 7-point Likert scale, ranging from strongly
disagree to strongly agree, to assess the performance and effort from
participants while using the cards. The qualitative aspect (Q2–5) aims to
understand motivations and thoughts behind using the cards. Question 2 and 4
gave participants a series of options (checkboxes), which were then coupled
with a follow-up question (Q3 and Q5 respectively) to further elaborate on
their choices. Two experienced postdoctoral researchers independently analysed
and coded the qualitative responses, grouping them into themes, which were
then discussed systematically and final agreements on codes were made in
consensus. This analytical technique has been applied successfully in various
bodies of work (Zade et al., 2018; Chinh et al., 2019). The data from
questionnaires is presented as italics in quotation marks, alongside a
participant ID (e.g. W1P1 for this first workshop) where relevant.

###  4.2. Results

Figure 3. Results of Workshop 1. This figure shows an overview of the
responses for Q1 in Appendix A. (1) Providing a knowledgebase = (a)–(d); (2)
Independent learning and self-efficacy = (e)–(f); (3) Interface for
discussion: (g)–(h).

An overview of the results from this evaluation can be seen in Figure 3. For
subsequent discussion, the results from the questionnaires will initially
follow three themes, noted by the title of each subsection.

####  4.2.1. Providing a Knowledge Base

The first theme aimed at determining whether our cybersecurity cards provided
users with introductory cybersecurity knowledge, whilst also supporting
learning in a well-documented manner. Providing a knowledge base looked into
four aspects: providing knowledge of individual concepts, cybersecurity
terminology, wider scope of topics, and the trichotomy of attacks, defences
and vulnerabilities (a–d) in Question 1 of the questionnaire in Appendix A).
Overall, we observed that our cards were able to effectively provide key
cybersecurity knowledge, adapted from the CyBOK knowledge base, in an
accessible manner. Interestingly, with regard to individual concepts and the
relationships between attacks, defences and vulnerabilities, three
participants strongly agreed with this. Furthermore, four participants also
strongly agreed that the cards were able to provide them with knowledge about
wider scope. For cybersecurity terminology, only two responded with somewhat
disagree, which we believe may be due to the text on cards using too much
technical terminology. It is clear that the cards have indeed achieved a
positive result regarding providing users with knowledge of fundamental
cybersecurity knowledge ($RQ1$), as well as providing links between topics
that allows the generation of concepts which capture the relationships between
attacks, vulnerabilities and defences in scenarios pertaining to the theme of
Code Security ($RQ4$).

####  4.2.2. Independent Learning and Self-Efficacy

With regard to independent learning and self-efficacy (e,f), we found that 8
participants agreed the cards enabled them to undertake independent learning
with three strongly agreeing. Furthermore, we found that 9 participants agreed
the cards provided them with access to key cybersecurity knowledge when one of
the rotating experts was not present in the group. In both cases, only a
single participant disagreed. Ultimately, the results demonstrate that our
cards approach provides a means for expressing interpretation and
documentation of key topics that supports independent learning and self-
efficacy ($RQ2$).

####  4.2.3. Providing an Interface for Discussion

The final theme looks at whether the cards provide users with an interface for
discussing key cybersecurity topics (g,h). Interestingly, we found that more
participants agreed that they could hold discussions on these topics with the
expert (9), in comparison with other participants in their group (7). However,
while both cases show a majority in agreement, when discussing topics with
others in their group more participants remained neutral as opposed to
disagreeing. While we see a positive outcome regarding the cards providing an
interface for the discussion of key cybersecurity topics ($RQ3$), it is
important to determine what was not clear and why this was the case, as well
as understanding how the cards could be improved.

####  4.2.4. Understanding Drawbacks Of Using The Cards

The next part of the questionnaire involves questions that were designed to
help uncover and understand any limitations of the cybersecurity cards, such
that improvements can be made to better fulfill the goals of the research
questions described in Section 3. The first step was to determine which
category or subset of the cybersecurity cards the participants did not use and
why this was the case. We found that the least used cards were the general and
detailed defence and vulnerability cards. The next question asked participants
why the cards they had selected were not used, one participant stated that
”general cards gave some idea about the content” (W1P6) and another stated
that they were only ”engaged in attack and defence” (W1P11). With this said,
however, most participants described that they had used all of the categories,
for example stating they had used ”at least one card [from] each” (W1P1) or
only ”concentrate[d] on a few cards” (W1P9). This suggests that while the
cards which were said to not have been used may not have been critical to
discussions they had about certain topics, they may have still been looked at
and thought about. We then asked participants about how the cards could be
improved. We found that 5 participants recorded that the number of cards in
the deck was too high. This may be due to the use of a physical medium and the
number of categories and types of each General card and is further suggested
by 3 other participants stating that there were too many types or categories.
Interestingly, 5 participants stated that both the color coding of the
types/categories was not clear, as well as the relationships between the
cards, with one participant describing that ”Threat cards [were] difficult to
handle/understand” (W1P7). This may also correspond with the feeling that the
number of cards was too high, but also a perceived difficulty of understanding
links between the cards and the terminology used. In contrast to this,
however, one participant mentioned that the ”Cards provides an entry point for
more detailed scenarios of cybersecurity and helps to create relationship
between attack and defense situations more clear” (W1P4). Interestingly, 3
participants recorded that the information on the cards were too abstract,
with no participants stating that they were too detailed. Thus, this helps
reinforce the understanding that the difficulty relating to understanding the
cards may be linked to a lack of clarity on the relationships between them.

##  5\. Cybersecurity Cards – Version 2

Upon review of the results from the evaluation questionnaire, the
cybersecurity cards were in places redesigned and the new deck is hereafter
referred to as Version 2. As well as providing the subset of the Version 1
cards for viewing, the equivalent subset for Version 2 of the cards have also
been made available and can be found
online222https://anonymous.4open.science/r/cybersecurity_cards-9F00/
(anonymous repository for double-blind reviewing purposes). The full set of
cards will be made publicly available under the CC BY-NC-SA Creative Commons
license after the research project ends in January 2024.

###  5.1. Structural Redesign

One of the limitations that required addressing was with regard to the number
of cards in the deck, which some participants felt was too high and ”difficult
to handle and understand (W1P7)”. In contrast with a standard deck of playing
cards used for the likes of Solitaire and Poker, which has 52 cards, Version 1
of our cards has more than double this amount (109) resulting in it being
perceived by participants as hard to handle. Thus, reducing the number of
cards will likely be better received, whilst still meeting our requirements.
In Version 2, the first major revision is reducing the number of cards in the
deck. The first decision made with regard to this was the removal of General
cards (types of attack, defence or vulnerability), given that participants
specified these as the least used cards in the deck. In Version 2, General
cards are replaced with a glossary (Appendix B) that provides a description
for each type, alongside a symbol associated with the type in the top left of
the card to help improve readability and visibility ( 4 in Figure 4). We also
merged some cards when distinctions were too specific, by identifying cards of
overlapping topics and a set of 2 or more attack similar attack cards were
merged by a more general card. For example, the Smudge Attack, Shoulder
Surfing and Social Engineering cards were merged to Social Engineering. This
reduces the number of cards in Version 2 to a total of 70 cards, made up of 20
attack, 20 vulnerability and 30 defence cards. By providing a glossary, users
can refer to this sheet as a form of guidance when looking at devising
particular cybersecurity scenarios. Furthermore, we also added two new classes
of vulnerability. First, Human is split into User and Management. The User
class captures the impact of vulnerabilities from careless or malicious users
(e.g. insider attackers), while Management captures bad management practices
such as poorly implemented security policies. Second, we provide a System
vulnerability type which relates to computer infrastructure, to better capture
attacks and defences that target the computer system.

Figure 4. Design Comparison Between Version 1 and 2. Version 1 identifier
codes at the top right of each card (1) has been replaced by icons
corresponding to its class and a unique class index number for each card (3).
Links between cards using codes (2) have been replaced with the corresponding
icons in vulnerabilities (4), and links between presented cards are
demonstrated using the arrows.

In terms of general design (Figure 4), the red background is made slightly
lighter making it easier on the eyes, with the symbols watermarked in the
background of the card also made lighter to maintain focus on the content of
the card whilst also still highlighting the card type. Further, the language
of the text in the cards has been improved with the aim of improving clarity
and reducing technical jargon that some participants had struggled with. For
example, “attack vectors” and “consequences” were replaced with “related
attacks” and “related defences” respectively. Finally, defence cards are now
more akin to a stop sign to aid with better distinguishing them from the
attack cards.

###  5.2. Relationship Clarification

The next revision made in Version 2 of the cards is a clarification of the
links between the cards to highlight relationships between various
vulnerabilities, attacks and defences, which was described to be inadequate.
Figure 4 shows a comparison between Version 1 and 2 of the cards. In Version
1, identifier codes in the top right corner were used to distinguish between
types of attacks, defences and vulnerabilities 1. The relationships between
attacks, defences and vulnerabilities is clarified by linking attack and
defence codes within the vulnerability cards 2. In Version 2, card identifiers
are replaced with a symbol and card number related to each type positioned in
the top left corner 4. The card number is the number of the card within a
type. For example, Command / Data Injection is card number 2 for the Injection
attack type. The links between vulnerabilities follow a similar approach to
Version 1, but with the codes replaced by pairs of symbols with ID numbers and
related vulnerabilities are detailed within attack and defence cards 3. Links
to vulnerabilities are given to encourage users to use intuition to make links
by learning from card content, rather than looking at explicit relationships.

##  6\. Evaluation – Version 2

To evaluate the second version of the cards, we make use of the same workshop
format and methodology (Table 1) as described in Section 4. In this second
workshop, we recruited 23 participants with ages ranging from 10 to 15 years
old (mean 12.8 years) from either senior primary school or early high school.
When asked to describe any experience and/or skills they may already have with
cybersecurity, one participant stated they know the basics, one stated that
passwords are important, one stated they know about hacking, and all other
participants saying they have either no experience (17) or are unsure (3).
When asked about experience or skills with coding, two participants stated
they had limited experience with Python programming, six stating they have
used Scratch or other block-based visual programming tools, with the remainder
(15) having no programming experience. Only 2 participants stated they had
experience with secure coding, with one neutral and the remainder either
disagreeing or strongly disagreeing with this. While the demographic of
participants in this second workshop differs from the first workshop, with
participants in this workshop being younger, their experience with
cybersecurity and code security remains similar and the activity within both
workshops are the same following the theme of Code Security. After the
workshop, the participants were given the same questionnaire to answer as
those received in the first workshop (Appendix A). For subsequent discussion,
we will discuss the results of the evaluation on the second version of the
cards on the same themes as the first version presented in Section 4.2. The
data from questionnaires is presented as italics in quotation marks, alongside
a participant ID (e.g. W2P6 for this second workshop) where relevant. An
overview of the results for the second evaluation can be seen in Figure 5.

Figure 5. Results of Workshop 2. This figure shows an overview of the
responses for Q1 in Appendix A.

###  6.1. Results

####  6.1.1. Providing a Knowledge Base

For the first theme of determining whether the cards provided introductory
cybersecurity knowledge that supports learning in a well-documented manner
($RQ1$), we look at the four items (a–d) in Question 1 of the questionnaire in
Appendix A. For individual concepts (a), 17 of the participants agreed that
the cards provided them with knowledge of cybersecurity concepts, terminology
and topics (6 somewhat agree, 4 agree and 7 strongly agree) with 3
participants scoring neutral and 3 somewhat disagreeing with this. For wider
scope (b), 17 participants agreed (3 somewhat agree, 6 agree and 8 strongly
agree) that the cards provided them with this knowledge, with 2 disagreeing
and 4 remaining neutral. For relationships between attacks, defences and
vulnerabilities (c), 16 agreed (6 somewhat agree, 3 agree and 7 strongly
agree) that the cards facilitated the understanding of the links between cards
that capture the vulnerability-attack-defence trichotomy that is present in
cybersecurity ($RQ4$), with 4 neutral and 3 disagreeing. Finally, with
regarding to cybersecurity terminology (d), 14 participants agreed (4 somewhat
agree, 4 agree and 6 strongly agree) that the cards provided them with
knowledge on terminology. Three participants disagreed with this and the
remaining 6 were neutral.

####  6.1.2. Independent Learning and Self-Efficacy

The theme of independent learning and self-efficacy relate to items (e,f) in
Question 1. The first item asked participants if the cards helped them
undertake independent learning of cybersecurity (e). We found that for the
second version of the cards, 11 participants agreed (2 somewhat agree, 4
agree, 5 strongly agree), 7 were neutral and 5 disagreeing (1 strongly
disagreeing). For the second item, we asked if the cards provided access to
cybersecurity knowledge without the presence of an expert. We found that 13
participants agreed (3 somewhat agree, 3 agree and 7 strongly agree), with 5
participants neutral to this and 5 disagreeing (1 strongly disagreeing). These
results show that the second version of our cards successfully supports
independent learning and self-efficacy ($RQ2$).

####  6.1.3. Providing an Interface for Discussion

The final theme explored relates to the cards providing an interface for
discussing key cybersecurity topics ($RQ3$) and relates to items (g, h) in
Question 1. The first question related to this theme asked participants if the
cards enabled them to discuss cybersecurity topics with an expert g. We found
that 14 participants agreed with this (3 somewhat agree, 5 agree and 6
strongly agree), with 6 participants remaining neutral to this and 3
disagreeing (1 strongly disagreeing). The final question in this theme asked
them if the cards enabled them to hold discussions on key cybersecurity topics
with others in their group h. We found that 14 participants agree with this (4
somewhat agree, 4 agree and 6 strongly agree), with 5 participants neutral and
4 disagreeing with this.

####  6.1.4. Understanding Drawbacks of Version 2

As done in the first workshop, the next three questions were designed to
gather an understanding of any limitations to the design of the cards and
whether there may be any improvements that can be made. In Question 2, we
asked participants whether there were any categories or subset of cards not
used by them in the workshop. The difference in this question compared to the
same questionnaire employed for evaluating Version 1 is due to the removal of
the General cards and that they are replaced with a Glossary. Thus, in
Appendix A, we refer to Question 2 (Version 2 Evaluation) which includes the
Glossary as an option, as well as cards within a specific category for each of
the classes. We found that the least used items were the Glossary (7
participants) and Race Condition (7) cards, with the next least used cards
being Injection (6), Memory (6), Mitigation (6) and Education (6). Looking at
Question 3 to understand why these types of cards (excluding the glossary) in
particular were not used, some participants mentioned that they “didn’t need
them” (W2P8), were “unnecessary” (W2P20) or “don’t know” (W2P9). This was
because some participants had decided to focus on a different aspect of
cybersecurity. With regard to the glossary, some participants stated that
“nobody reads the glossary” (W2P10), suggesting that this was not needed and
the cards alone for some participants were enough to facilitate learning and
understanding of key cybersecurity topics. Interestingly, some participants
had stated that they had “used all the cards” (W2P4, W2P6), with some others
giving a similar reason yet selecting many (if not all) checkboxes for all
cards. In Question 4, we asked participants if there were any other
improvements to the cards they would like to suggest. Most participants (14)
said “No” to this question, however one participant suggested to use “simple
wording”, which may correlate with why certain cards (e.g. Race Condition or
Memory attacks) were not used. As well as this, three participants suggested
to make the cards “easier to use” (W2P2, W2P14), however any suggestions as to
how they might think this could be achieved were not elaborated.

##  7\. Discussion

The aim of this research is to understand how a playing cards approach can
provide clear communication of key cybersecurity topics and an interface for
discussing them, leveraging the information from the CyBOK knowledge base. To
investigate this, we were guided by the research questions proposed in Section
3. Subsequent discussion will encompass the evaluations of both versions of
the cards and follows each of the research questions in turn.

### RQ1: Do the cybersecurity cards provide introductory cybersecurity
knowledge to novice users?

Cybersecurity is often overlooked as a subject due to issues such as perceived
technical difficulty, steep learning curves and a requirement of specialist
knowledge and/or expertise. In industry, for example, the lack of
cybersecurity professionals has been linked to a lack of practical
cybersecurity content within learning materials (Caldwell, 2013). While CyBOK
aims to rectify this learning gap, traversing the knowledge base and
understanding the material requires prior knowledge, as evidenced by the
primary usage of CyBOK in the development of higher education programmes
(Hallett et al., 2018). Ultimately, this means it may not be considered
accessible for novice individuals. In this work, we show that our
cybersecurity cards achieved a positive result with regard to providing
introductory knowledge of key cybersecurity topics to novice users.
Specifically, we found that 82% of participants from the first workshop in
higher education agreed with this (Section 4.2.1). In our second workshop
involving late-primary and secondary aged school children. We found that
around 70% of them also agreed with this (Section 6.1.1). Both of these
results are significant as participants described themselves initially as
having little-to-no cybersecurity experience.

### RQ2: Do the cybersecurity cards provide material for expressing
interpretation of key topics that supports independent learning and self-
efficacy?

One of the concerns surrounding cybersecurity relates to the preconception of
steep learning curves and a requirement of specialist knowledge and expertise
(Asen et al., 2019; Bahizad, 2020), which is a problem that is not adequately
managed by the CyBOK knowledge base. From our evaluation of the cards, we
found that most of our participants in higher education from the first
workshop agreed they were able to understand key cybersecurity topics
independently (Section 4.2.2). In the second workshop, we found that half of
the primary and secondary participants also agreed with this (Section 6.1.2).
It appears that removing the general cards and replacing them with a glossary
may have impacted this, backed up by statements from the participants saying
that nobody reads the glossary. In terms of self-efficacy, we found that
participants in the first workshop agreed that even without a cybersecurity
expert present in the group, they were able to access cybersecurity knowledge
solely using our cybersecurity cards. In the second workshop, we found that
more participants agreed on this compared to understanding topics
independently.

### RQ3: Do the cards act as an index for the CyBOK knowledge base, which
provides an interface for discussion on key cybersecurity topics?

CyBOK has been shown to lack depth of cybersecurity knowledge and is hard to
traverse, particularly in the aspect of practical experience such as
discussing key topics, which is essential for mastering cybersecurity skills
(Manson and Pike, 2014; Hallett et al., 2018). The use of playing card formats
as an alternative approach for learning cybersecurity topics has shown to be
successful, but are either designed for those with existing cybersecurity
knowledge (Swann et al., 2021; Trickel et al., 2017; Schreuders, 2022) or do
not leverage a peer-reviewed and well-established knowledge foundation such as
CyBOK (Denning et al., 2013; Anvik et al., 2019). This is important as the
value, actionability and perception of security information strongly depends
on the source (Rader and Wash, 2015; Redmiles et al., 2020). While in this
work we also propose the use of a playing cards format, we leverage a strong
information foundation and the intended use is not in the context of games
which other playing cards approaches are. Furthermore, our cards have a single
target persona of someone with little-to-no expertise in cybersecurity that
writes software code. The first version of our cybersecurity cards show that
the majority of participants agreed they were able to discuss key topics with
cybersecurity experts that periodically checked in on them, as well as with
other members in their group (Section 4.2.3). In Version 2, we found similar
results with more strongly agreeing compared to those in higher education
using the previous cards. While in both cases some participants disagreed with
this, this may be due to reasons such as simply not wanting to discuss topics
with the experts. Interestingly, in the second workshop, we found that more
participants disagreed that the cards enabled them to hold discussions with
others in their group. This could be due to a lack of interest in doing so, or
potentially they simply did not know they could do that. In this work, we
demonstrate that our playing cards approach can act as a suitable index for
CyBOK and provides an interface for discussing key cybersecurity topics for
even novice or non-technical users.

### RQ4: Do the cards provide links between key cybersecurity topics, allowing
for the capture of various scenarios?

In the evaluation questionnaire, we asked participants whether the cards
provided them with knowledge about the relationships between attacks, defences
and vulnerabilities (Section 4.2.1). We found that the majority of
participants agreed with this. This is further strengthened given that most
participants in the first workshop also agreed the cards promoted discussion
of key cybersecurity topics with both experts and other members in their
groups (Section 4.2.3), with similar results seen in the second workshop
(Section 6.1.3). While the CyBOK knowledge base does encapsulate the attack-
defence-vulnerability trichotomy, it is difficult to identify whether some
topics focus on a single predominant theme or whether it spans across various
themes (González-Tablas and Rashed, 2022). Furthermore, it has been shown that
because of this difficulty, these links could be identified from series of
keywords which are only meaningfully extracted via specialised algorithms such
as topic model analysis (González-Tablas and Rashed, 2022; Hallett et al.,
2018). With regard to other learning approaches for cybersecurity, Capture The
Flag (CTF) activities can aid with highlighting links between attacks,
defences and vulnerabilities by prioritising the focus of finding
vulnerabilities (Trickel et al., 2017; Swann et al., 2021; Schreuders, 2022).
However, the disadvantage to CTF approaches relates to requiring technical
expertise (e.g. using command-line tools) in order to progress in finding
vulnerabilities (McDaniel et al., 2016). In this work, we found that those
with little-to-no cybersecurity or coding expertise can use our cards to make
links across the cybersecurity trichotomy, whilst devising various
cybersecurity scenarios in the domain of code security.

###  7.1. Reported Limitations

From the evaluation of the first version of the cards, we identified some
limitations from participant responses. First, some participants felt the
overall number of cards was too high (124 cards in total) and General cards
were not used. In the second version, the deck size was reduced to improve
physical handling, whilst also adhering to the primary goals, with General
cards also replaced by a glossary. In the second workshop, we found that there
were no further suggestions to reduce the size of the deck. Second, the layout
and content of first version of the cards were described as difficult to go
through by participants in the first workshop. In version 2, we improved the
contrast of the red colour, as well as the layout of card elements to help
improve clarity, as well as improving the terminology used to describe each of
the cards. In the second workshop, none of the participants suggested any
further design changes. Finally, we believed the issue of readability in
Version 1 to form links from vulnerabilities was likely due to the identifier
codes and instead replaced them with a symbol and index number.

###  7.2. Future Work

A first point of future work would be to revisit the glossary, as some
participants indicated that the glossary was not useful and in some cases not
used at all. Given that the purpose of the glossary is to improve meaning and
uniformity in the usage of technical terminology, one potential solution to
improve our approach is to improve the content of the glossary (e.g. the
language). As well as this, the importance of the glossary can also be better
highlighted when using the cards. While the current approach makes links to
the glossary using the class symbol (e.g. attack) may be suitable, better
engagement with the glossary could be achieved through informing users of the
importance of the glossary when introducing the cards. In the case of digital
cards, hyperlinks within cards (e.g. via the symbol) to the glossary could be
made clear for improved accessiblity. With regard to the layout and written
content, the language used for each of the cards can be further refined. As
well as this, another design change can be to better visibly distinguish the
classes of cards. In previous work, it has been studied whether different
colours of warnings can affect one’s perception of risk (Leonard, 1999; Arenas
et al., 2014). For example, the same red color could be used for attack cards
as it is typically associated with danger and a higher perceived relative
amount of risk. An orange or amber colour is typically used for warnings and
the colour yellow for caution, which could be used to signal the vulnerability
cards. The defence cards could be represented with a blue colour. For example,
blue has represented defence in some areas of military, such as to signal a
friendly icon for NATO APP-6/A affiliation (Varga et al., 2019).

##  8\. Conclusion

Cybersecurity is a complex subject area that is constantly changing due to the
dynamic nature of vulnerabilities, attacks and defences. Many users who may
have little-to-no knowledge of cybersecurity are left vulnerable, with the key
question of whether best practices are truly understood remains unclear.
Existing knowledge bases such as CyBOK provide key cybersecurity information,
but are typically designed to support the development of educational curricula
or those with prior knowledge. In this work, we propose an approach leveraging
a playing cards format with the goals of introducing cybersecurity topics to
novice users, facilitating independent learning and understanding of the
various relationships found in the cybersecurity ecosystem. Upon evaluation,
we found that our approach was successful in achieving these goals for a wide
age group (10-35 years) of both non-technical users and those with some
experience. Using the data from this evaluation, we designed a second version
of these cards and further evaluated them and showing they still meet the
proposed requirements. Ultimately, our cybersecurity cards provide a
comprehensive and effective tool that allows novice individuals to gain
introductory knowledge of cybersecurity, while promoting understanding and
independent learning of cybersecurity.

## References

  * (1)
  * cve ([n.d.]) [n.d.].  Common Vulnerabilities and Exposures (CVE).  https://www.cve.org/ Accessed 19/07/2023.
  * cyb ([n.d.]a) [n.d.]a.  The Cyber Security Body of Knowledge (CyBOK) - Use Cases.  https://www.cybok.org/usecases/ Accessed 19/07/2023.
  * cyb ([n.d.]b) [n.d.]b.  The Cyber Security Body of Knowledge (CyBOK), Version 1.1.  https://www.cybok.org/ Accessed 19/07/2023.
  * nic (2015) 2015\.  NICCS Cybersecurity Workforce Planning.  https://niccs.us-cert.gov/careers/workforce-planning Accessed 19/07/2023.
  * ncs (2017) 2017\.  NCSC Certified Master’s in Cyber Security.  Accessed 19/07/2023.
  * wef (2020) 2020\.  After reading, writing and arithmetic, the 4th ’r’ of literacy is cyber-risk. World Economic Forum.  https://www.weforum.org/agenda/2020/12/cyber-risk-cyber-security-education Accessed 19/07/2023.
  * cis (2020) 2020\.  Securing What’s Now and What’s Next 20 Cybersecurity Considerations for 2020. Cisco.  https://www.cisco.com/c/dam/m/en_hk/ciscolive/2020-ciso-benchmark-cybersecurity-series.pdf Accessed 19/07/2023.
  * sop (2021) 2021\.  Ransomware Recovery Cost Reaches Nearly $2 Million, More Than Doubling in a Year.  _SOPHOS_ (Apr 2021).  https://www.sophos.com/en-us/press-office/press-releases/2021/04/ransomware-recovery-cost-reaches-nearly-dollar-2-million-more-than-doubling-in-a-year
  * cov (2022) 2022\.  Coronavirus (COVID-19): staying safe and protecting others.  https://www.gov.scot/publications/coronavirus-covid-19-staying-safe-and-protecting-others Accessed 19/07/2023.
  * Adams and Sasse (1999) Anne Adams and Martina Angela Sasse. 1999.  Users are not the enemy.  _Commun. ACM_ 42, 12 (1999), 40–46.
  * Alotaibi and Almagwashi (2018) Bashayer Alotaibi and Haya Almagwashi. 2018.  A review of BYOD security challenges, solutions and policy best practices. In _2018 1st International Conference on Computer Applications & Information Security (ICCAIS)_. IEEE, 1–6.
  * Altice (2014) Nathan Altice. 2014\.  The playing card platform.  _Analogue Game Studies_ 4, 2 (2014).
  * Anvik et al. (2019) John Anvik, Vincent Cote, and Jace Riehl. 2019.  Program wars: a card game for learning programming and cybersecurity concepts. In _Proceedings of the 50th ACM Technical Symposium on Computer Science Education_. 393–399.
  * Arenas et al. (2014) Lina María Arenas, Jolyon Troscianko, and Martin Stevens. 2014\.  Color contrast and stability as key elements for effective warning signals.  _Frontiers in Ecology and Evolution_ 2 (2014), 25.
  * Asen et al. (2019) Alex Asen, Walter Bohmayr, Stefan Deutscher, Marcial González, and David Mkrtchian. 2019\.  Are you spending enough on cybersecurity?. Boston Consulting Group.  Accessed 19/07/2023.
  * Bahizad (2020) Sahba Bahizad. 2020\.  Risks of Increase in the IoT Devices. In _2020 7th IEEE international conference on cyber security and cloud computing (CSCloud)/2020 6th IEEE international conference on edge computing and scalable cloud (EdgeCom)_. IEEE, 178–181.
  * Bishop et al. (2017) Matt Bishop, Diana Burley, Scott Buck, Joseph J Ekstrom, Lynn Futcher, David Gibson, Elizabeth K Hawthorne, Siddharth Kaza, Yair Levy, Herbert Mattord, et al. 2017\.  Cybersecurity curricular guidelines. In _Information Security Education for a Global Digital Society: 10th IFIP WG 11.8 World Conference, WISE 10, Rome, Italy, May 29-31, 2017, Proceedings 10_. Springer, 3–13.
  * Caldwell (2013) Tracey Caldwell. 2013\.  Plugging the cyber-security skills gap.  _Computer Fraud & Security_ 2013, 7 (2013), 5–10.
  * Chinh et al. (2019) Bonnie Chinh, Himanshu Zade, Abbas Ganji, and Cecilia Aragon. 2019.  Ways of qualitative coding: A case study of four strategies for resolving disagreements. In _Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems_. 1–6.
  * CyBOK (2021) CyBOK. 2021.  The Cyber Security Body of Knowledge v1.1, Web and Mobile Security Knowledge Tree.  https://www.cybok.org/media/downloads/Web_Mobile_Security_v1.0.1.pdf Accessed 19/07/2023.
  * Denning et al. (2013) Tamara Denning, Adam Lerner, Adam Shostack, and Tadayoshi Kohno. 2013.  Control-Alt-Hack: the design and evaluation of a card game for computer security awareness and education. In _Proceedings of the 2013 ACM SIGSAC conference on Computer & communications security_. 915–928.
  * Ferro et al. (2021) Lauren S Ferro, Andrea Marrella, and Tiziana Catarci. 2021\.  A Human Factor Approach to Threat Modeling. In _International Conference on Human-Computer Interaction_. Springer, 139–157.
  * Fischer et al. (2017) Felix Fischer, Konstantin Böttinger, Huang Xiao, Christian Stransky, Yasemin Acar, Michael Backes, and Sascha Fahl. 2017.  Stack overflow considered harmful? the impact of copy&paste on android application security. In _2017 IEEE Symposium on Security and Privacy (SP)_. IEEE, 121–136.
  * Ford et al. (2017) Vitaly Ford, Ambareen Siraj, Ada Haynes, and Eric Brown. 2017\.  Capture the flag unplugged: an offline cyber competition. In _Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education_. 225–230.
  * González-Tablas and Rashed (2022) Ana I González-Tablas and Mohammed Rashed. 2022.  Exploring CyBOK with Topic Modeling Techniques. In _Human Aspects of Information Security and Assurance: 16th IFIP WG 11.12 International Symposium, HAISA 2022, Mytilene, Lesbos, Greece, July 6–8, 2022, Proceedings_. Springer, 49–65.
  * Hallett et al. (2018) Joseph Hallett, Robert Larson, and Awais Rashid. 2018\.  Mirror, mirror, on the wall: What are we teaching them all? Characterising the focus of cybersecurity curricular frameworks. In _2018 USENIX Workshop on Advances in Security Education ( $\\{$ASE$\\}$ 18)_.
  * Harknett and Stever (2009) Richard J Harknett and James A Stever. 2009.  The cybersecurity triad: Government, private sector partners, and the engaged cybersecurity citizen.  _Journal of Homeland Security and Emergency Management_ 6, 1 (2009).
  * Hoffman et al. (2011) Lance J Hoffman, Diana Burley, and Costis Toregas. 2011\.  Thinking across stovepipes: Using a holistic development strategy to build the cybersecurity workforce.  _IEEE Security and Privacy_ 1 (2011), 13.
  * Kaloudi and Li (2020) Nektaria Kaloudi and Jingyue Li. 2020.  The ai-based cyber threat landscape: A survey.  _ACM Computing Surveys (CSUR)_ 53, 1 (2020), 1–34.
  * Kordaki and Gousiou (2016) Maria Kordaki and Anthi Gousiou. 2016.  Computer card games in computer science education: A 10-year review.  _Journal of Educational Technology & Society_ 19, 4 (2016), 11–21.
  * Kordaki and Gousiou (2017) Maria Kordaki and Anthi Gousiou. 2017.  Digital card games in education: A ten year systematic review.  _Computers & Education_ 109 (2017), 122–161.
  * Kordy et al. (2014) Barbara Kordy, Sjouke Mauw, Saša Radomirović, and Patrick Schweitzer. 2014.  Attack–defense trees.  _Journal of Logic and Computation_ 24, 1 (2014), 55–87.
  * Leonard (1999) S David Leonard. 1999\.  Does color of warnings affect risk perception?  _International Journal of Industrial Ergonomics_ 23, 5-6 (1999), 499–504.
  * Manson and Pike (2014) Daniel Manson and Ronald Pike. 2014.  The case for depth in cybersecurity education.  _ACM Inroads_ 5, 1 (2014), 47–52.
  * Martin et al. (2021) Andrew Martin, Awais Rashid, Howard Chivers, Steve Schneider, Emil Lupu, and George Danezis. 2021\.  _Introduction to CyBOK Knowledge Area Version 1.1._ Technical Report.
  * Martin et al. (2011) Bob Martin, Mason Brown, Alan Paller, Dennis Kirby, and S Christey. 2011.  Common Weakness Enumeration (CWE).  _SANS top_ 25 (2011).
  * McDaniel et al. (2016) Lucas McDaniel, Erik Talvi, and Brian Hay. 2016.  Capture the flag as cyber security introduction. In _2016 49th Hawaii International Conference on System Sciences (HICSS)_. IEEE, 5479–5486.
  * Newhouse et al. (2017) William Newhouse, Stephanie Keith, Benjamin Scribner, and Greg Witte. 2017.  National initiative for cybersecurity education (NICE) cybersecurity workforce framework.  _NIST special publication_ 800, 2017 (2017), 181\.
  * Pranggono and Arabo (2021) Bernardi Pranggono and Abdullahi Arabo. 2021.  COVID-19 pandemic cybersecurity issues.  _Internet Technology Letters_ 4, 2 (2021), e247.
  * Rader and Wash (2015) Emilee Rader and Rick Wash. 2015.  Identifying patterns in informal sources of security information.  _Journal of Cybersecurity_ 1, 1 (2015), 121–144.
  * Redmiles et al. (2020) Elissa M Redmiles, Noel Warford, Amritha Jayanti, Aravind Koneru, Sean Kross, Miraida Morales, Rock Stevens, and Michelle L Mazurek. 2020\.  A comprehensive quality evaluation of security and privacy advice on the web. In _29th USENIX Security Symposium (USENIX Security 20)_. 89–108.
  * Sarginson (2020) Nic Sarginson. 2020\.  Securing your remote workforce against new phishing attacks.  _Computer Fraud & Security_ 2020, 9 (2020), 9–12.
  * Saxena et al. (2020) Neetesh Saxena, Emma Hayes, Elisa Bertino, Patrick Ojo, Kim-Kwang Raymond Choo, and Pete Burnap. 2020.  Impact and key challenges of insider threats on organizations and critical businesses.  _Electronics_ 9, 9 (2020), 1460.
  * Schreuders (2022) Cliffe Schreuders. 2022\.  Complete index of CTF scenarios mapped to CyBOK.  https://github.com/cliffe/SecGen/blob/master/README-CyBOK-CTF-Scenarios-Indexed.md.  Accessed 19/07/2023.
  * Swann et al. (2021) Matthew Swann, Joseph Rose, Gueltoum Bendiab, Stavros Shiaeles, and Fudong Li. 2021\.  Open Source and Commercial Capture The Flag Cyber Security Learning Platforms-A Case Study. In _2021 IEEE International Conference on Cyber Security and Resilience (CSR)_. IEEE, 198–205.
  * Tatam et al. (2021) Matt Tatam, Bharanidharan Shanmugam, Sami Azam, and Krishnan Kannoorpatti. 2021.  A review of threat modelling approaches for APT-style attacks.  _Heliyon_ 7, 1 (2021), e05969.
  * Thomas et al. (2019) Michael K Thomas, Andria Shyjka, Skip Kumm, and Rigel Gjomemo. 2019\.  Educational design research for the development of a collectible card game for cybersecurity learning.  _Journal of Formative Design in Learning_ 3, 1 (2019), 27–38.
  * Trickel et al. (2017) Erik Trickel, Francesco Disperati, Eric Gustafson, Faezeh Kalantari, Mike Mabey, Naveen Tiwari, Yeganeh Safaei, Adam Doupé, and Giovanni Vigna. 2017.  Shell We Play A Game?$\\{$CTF-as-a-service$\\}$ for Security Education. In _2017 USENIX Workshop on Advances in Security Education (ASE 17)_.
  * Turkay et al. (2012) Selen Turkay, Sonam Adinolf, and Devayani Tirthali. 2012\.  Collectible card games as learning tools.  _Procedia-Social and Behavioral Sciences_ 46 (2012), 3701–3705.
  * Varga et al. (2019) Margaret Varga, Carsten Winkelholz, and Susan Träber-Burdin. 2019\.  An exploration of cyber symbology. In _2019 IEEE Symposium on Visualization for Cyber Security (VizSec)_. IEEE, 1–5.
  * Wall (2017) David S Wall. 2017\.  Crime, security and information communication technologies: The changing cybersecurity threat landscape and its implications for regulation and policing.  _Security and Information Communication Technologies: The Changing Cybersecurity Threat Landscape and Its Implications for Regulation and Policing (July 20, 2017)_ (2017).
  * Walters and Matulich (2011) Melissa Walters and Erika Matulich. 2011.  Assessing password threats: Implications for formulating university password policies.  _Journal of Technology Research_ 2 (2011), 1.
  * WILLETTS (2014) D WILLETTS. 2014\.  _Cyber security skills—business perspectives and govenment’s next steps_.  Technical Report. Tech. rep., HM Government.
  * Zade et al. (2018) Himanshu Zade, Margaret Drouhard, Bonnie Chinh, Lu Gan, and Cecilia Aragon. 2018.  Conceptualizing disagreement in qualitative coding. In _Proceedings of the 2018 CHI conference on human factors in computing systems_. 1–11.

##  Appendix A Workshop Evaluation Questionnaire

Question 1

Each item below is scored using a Likert-type scale (1 - 7): (1=Strongly
disagree, 2= disagree, 3= somewhat disagree, 4= don’t know/neutral, 5=
somewhat agree, 6= agree, 7= strongly agree).

The Cybersecurity cards that I have used during the workshop:

|  Strongly Disagree |  Disagree |  Somewhat Disagree |  Don’t Know / Neutral |  Somewhat Agree |  Agree |  Strongly Agree
---|---|---|---|---|---|---|---
(a) Provided me with knowledge about individual cybersecurity concepts |  $\bigcirc$ |  $\bigcirc$ |  $\bigcirc$ |  $\bigcirc$ |  $\bigcirc$ |  $\bigcirc$ |  $\bigcirc$
(b) Provided me with knowledge about the wide scope of cybersecurity concepts |  $\bigcirc$ |  $\bigcirc$ |  $\bigcirc$ |  $\bigcirc$ |  $\bigcirc$ |  $\bigcirc$ |  $\bigcirc$
(c) Provided me with knowledge about the relationship between cybersecurity concepts, such as the relationships between attacks, defences and vulnerabilities |  $\bigcirc$ |  $\bigcirc$ |  $\bigcirc$ |  $\bigcirc$ |  $\bigcirc$ |  $\bigcirc$ |  $\bigcirc$
(d) Provided me with knowledge about cybersecurity terminology |  $\bigcirc$ |  $\bigcirc$ |  $\bigcirc$ |  $\bigcirc$ |  $\bigcirc$ |  $\bigcirc$ |  $\bigcirc$
(e) Enabled me to undertake independent learning about cybersecurity |  $\bigcirc$ |  $\bigcirc$ |  $\bigcirc$ |  $\bigcirc$ |  $\bigcirc$ |  $\bigcirc$ |  $\bigcirc$
(f) Provided access to cybersecurity knowledge when the cybersecurity expert was not present |  $\bigcirc$ |  $\bigcirc$ |  $\bigcirc$ |  $\bigcirc$ |  $\bigcirc$ |  $\bigcirc$ |  $\bigcirc$
(g) Enabled me to discuss cybersecurity with the cybersecurity expert |  $\bigcirc$ |  $\bigcirc$ |  $\bigcirc$ |  $\bigcirc$ |  $\bigcirc$ |  $\bigcirc$ |  $\bigcirc$
(h) Enabled me to discuss cybersecurity with others (i.e. not including the cybersecurity expert |  $\bigcirc$ |  $\bigcirc$ |  $\bigcirc$ |  $\bigcirc$ |  $\bigcirc$ |  $\bigcirc$ |  $\bigcirc$

Question 2

Which category or subset of the cybersecurity cards did you not use, if any?
Please tick all the boxes that apply in your opinion:

  1. $\square$

General Attack Cards

  2. $\square$

General Vulnerability Cards

  3. $\square$

General Defence Target Cards

  4. $\square$

General Defence Cards

  5. $\square$

Detailed Attack Cards

  6. $\square$

Detailed Vulnerability Cards

  7. $\square$

Detailed Defence Cards

Question 2 (Version 2 Evaluation)

Which category or subset of the cybersecurity cards did you not use, if any?
Please tick all the boxes that apply in your opinion:

The logos in the Glossary indicate the specific categories of Attack, Defence
and Vulnerability cards.

  1. $\square$

Glossary

  2. $\square$

Attack cards (in general) (fully red cards)

  3. $\square$

Vulnerability cards (in general) (cards with half red/half white diagonal)

  4. $\square$

Defence cards (in general) (white cards with red border)

  5. $\square$

Attack – Injection cards

  6. $\square$

Attack – Memory cards

  7. $\square$

Attack – Race condition cards

  8. $\square$

Attack – Side channel cards

  9. $\square$

Attack – Authentication cards

  10. $\square$

Attack – Web cards

  11. $\square$

Attack – System cards

  12. $\square$

Attack – Human factors cards

  13. $\square$

Defence – Detection cards

  14. $\square$

Defence – Mitigation cards

  15. $\square$

Defence – Education cards

  16. $\square$

Defence – Prevention cards

  17. $\square$

Vulnerability – Code cards

  18. $\square$

Vulnerability – System cards

  19. $\square$

Vulnerability – Environment cards

  20. $\square$

Vulnerability – User cards

  21. $\square$

Vulnerability – Management cards

  22. $\square$

Other

Question 3

Looking at the types of cards you selected in the previous question, why did
you not use those cards?

Enter your answer

Question 4

How could the cybersecurity cards potentially be improved in your opinion?
Please tick all the boxes of the statements below you agree with:

  1. $\square$

The total number of cards was too high.

  2. $\square$

The total number of cards was too low.

  3. $\square$

There were too many different types/categories of cybersecurity cards.

  4. $\square$

The logos and icons were not clear.

  5. $\square$

The numbering of the cards was not clear.

  6. $\square$

The colour-coding of the different types/categories of cybersecurity cards was
not clear.

  7. $\square$

The relationships between the different cybersecurity cards was not clear.

  8. $\square$

The information provided by the cybersecurity cards was too abstract.

  9. $\square$

The information provided by the cybersecurity cards was too detailed.

  10. $\square$

The information provided by the cybersecurity cards was too technical or too
difficult to understand.

  11. $\square$

The Glossary overview of the different types of cybersecurity cards was not
clear. (Version 2)

  12. $\square$
  13. $\square$

Other

Question 5

Are there any other improvements to cybersecurity cards you would like to
suggest?

Enter your answer

##  Appendix B Version 2 Glossary