# The Role of Diversity in Cybersecurity Risk Analysis: An Experimental Plan

Katja Tuma  Vrije Universiteit Amsterdam<EMAIL_ADDRESS>and  Romy Van Der Lee
Vrije Universiteit Amsterdam<EMAIL_ADDRESS>
(2022)

###### Abstract.

Cybersecurity threat and risk analysis (RA) approaches are used to identify
and mitigate security risks early-on in the software development life-cycle.
Existing approaches automate only parts of the analysis procedure, leaving key
decisions in identification, feasibility and risk analysis, and quality
assessment to be determined by expert judgement. Therefore, in practice teams
of experts manually analyze the system design by holding brainstorming
workshops. Such decisions are made in face of uncertainties, leaving room for
biased judgement (e.g., preferential treatment of category of experts). Biased
decision making during the analysis may result in unequal contribution of
expertise, particularly since some diversity dimensions (i.e., gender) are
underrepresented in security teams. Beyond the work of risk perception of non-
technical threats, no existing work has empirically studied the role of
diversity in the risk analysis of technical artefacts. This paper proposes an
experimental plan for identifying the key diversity factors in RA.

secure design, threat modeling, risk analysis, cybersecurity, diversity

††journalyear: 2022††copyright: rightsretained††conference: Third Workshop on
Gender Equality, Diversity, and Inclusion in Software Engineering; May 20,
2022; Pittsburgh, PA, USA††booktitle: Third Workshop on Gender Equality,
Diversity, and Inclusion in Software Engineering (GE@ICSE’22), May 20, 2022,
Pittsburgh, PA, USA††doi: 10.1145/3524501.3527595††isbn:
978-1-4503-9294-5/22/05

##  1\. Introduction

Security-by-design techniques strive to avoid risks before the software is
actually developed to reduce the security incurred costs in later stages of
development. For instance, threat and risk analysis (RA, for short) can be
performed by adopting systematic techniques to scrutinize the software system
(e.g., STRIDE (Shostack, 2014), attack trees (Saini et al., 2008), CORAS (Lund
et al., 2010), to name a few). Due to development trends (Agile, DevOps),
analysis automation is a much needed direction for improvement (Cruzes et al.,
2018). However, many automation efforts automate parts of the analysis
procedure and rely on an extended manual modeling effort (Tuma et al., 2018).
Key decisions in the identification, feasibility and risk analysis, and
quality assessment are often left to the experts. In practice, teams of
experts often analyze the system design manually by holding brainstorming
workshops where they discuss potential risks and decide which risks will be
mitigated and how. Risk based decisions must be made in face of uncertainties
(Bier, 2020), leaving room for biased judgement (e.g., preferential treatment
of category of experts). In addition, the quality of analysis outcomes is
ultimately determined by (potentially biased) expert judgement (Jaspersen and
Montibeller, 2015; Brito and Dawson, 2020). For instance, evidence of
groupthink (Wang and Wagner, 2018) in safety analysis sessions suggests that
knowledge is not always contributed (equally) by all participants, but exact
variables that effect the technical quality of the analysis are not yet
understood. Further, the type of judgement bias present in risk-based decision
making and more generally the role of diversity has not yet been sufficiently
explored in RA (as discussed in the related work, Section 3).

In many engineering domains, there is a lack of security experts (Tuma et al.,
2021). In addition, security expert teams are not diverse, especially for what
concerns gender. According to a news post by the International Consortium of
Minority Cyber Professionals (non-profit) in August 2021 (Cyversity, 2021),
the cybersecurity workforce in the USA is currently only 14% female. Generally
speaking, there is a significant mismatch between the cultural notion of a
successful technical professional and the perceived skills that are needed for
success in the computer science domain. Wynn and Correll (Wynn and Correll,
2017) show that men are more likely than women to believe they possess the
stereotypical traits and skills of a successful tech employee. This mismatch
can explain the low technical confidence of young female professionals that
are deciding about their career path in computer science (Beyer et al., 2003).
We believe there is still untapped potential in empowering women to pursue
their career as security experts. This motivates us to gather empirical
evidence that could help in (i) raising more interest and confidence levels of
female students in cybersecurity topics, and (ii) encouraging organizations to
build gender balanced security teams.

The purpose of the laid out research track is to identify the diversity
dimensions (see Section 2) that play a role during RA of more sophisticated
technical artefacts. To simulate RA in the experimental setup, the ACM Code of
Ethics scenarios can be used as RA task when interacting with the
communication science population as opposed to traditional economics/social
experiments. Second, case studies with detailed information about the software
design (e.g., IoT home monitoring system used in (Tuma and Scandariato, 2018))
can be used as RA task when interacting with the computer science population.
The experimental conditions will differ depending on the measure and target
audience (as described in Section 5).

This paper discusses the challenges (see Section 4) and outlines the
experimental plan (see Section 5) for identifying the role of diversity in the
technical domain of RA. Our experimental plan is three-fold. First, we plan to
characterize the key differentiating factors in diversity dimensions (e.g.,
gender) that are present to RA. Second, we plan to evaluate their effects in a
non-technical population and technical population. Third, we plan to develop
and validate diversity interventions (given to participants in the form of
training) in realistic (technical) environments. We briefly mention the next
steps in the laid out plan and present the concluding remarks in Section 7.

##  2\. Diversity Dimensions in RA

Diversity processes are well understood in the social science disciplines,
such as, the role of diversity factors in group dynamics or effects of
diversity biases in organizational processes. In contrast to the technical
domain, studies have demonstrated how diversity (gender, nationality) can be
beneficial for decision making and progress, once effectively incorporated
(Nielsen et al., 2017; van der Lee and Ellemers, 2018; Van Dijk et al., 2012).

Gender. Gender is an individual’s own gender identity, which is typically, man
and woman, but can also be non-binary. Rodriguez et al. (Rodríguez-Pérez et
al., 2021) found evidence of bias against women in some software engineering
communities, and sometimes negative perceptions about women working in teams.
Thus, this is an interesting dimension to further investigate in the context
of cybersecurity.

Education. Education is an individuals’ achieved level and topic of
specialization (e.g., computer security vs AI) of academic studies. Risk-based
decisions have to be made in organizations by the managerial layers, who
typically have a good understanding of the product, but do not necessarily
posses the technical skills of security experts or engineers. Therefore, it is
interesting to investigate this dimension and include participants from a
different domain (e.g., with background in communication sciences). Education
turned out to be a non-significant variable in the study of the impact of
commercial Antivirus on people’s awareness of security incidents (Jardine,
2020). However, it is not clear whether this dimension has an impact in
performing a RA task.

Nationality & Race. Nationality is the country of origin, which is often
coupled with the culture and language that categorizes social groups. Race is
a social construct linked with individual’s physical characteristics such as
skin color and is used to categorize populations. Determining the effect of
racial (and nationality) bias in cybersecurity practices is to date an open
question. Thomas et al. (Thomas et al., 2018) conducted semi-structured
interviews with 14 Black women in computing and report that Black women
experienced isolation (though it is not clear whether due to gender or race).
But, few studies have focused on racial (and nationality) diversity in the
software engineering discipline (Rodríguez-Pérez et al., 2021).

Age & Seniority. Age and seniority (i.e., position in the organization and
years of experience) may have an impact on both the technical outcomes of a RA
task and on the judgment of outcomes quality (e.g., over-favoring results
submitted by senior analysts). For instance, a category of experts (e.g.,
junior vs senior) may underestimate the ease with which a mitigation can be
implemented in reality (Wright et al., 2002). This may hurt later-on when
infeasible mitigations have to be replaced with cheaper (and perhaps) less
insecure solutions. We remark that though this dimension is indeed relevant,
empirically observing seniority (in a realistic setting) requires involving
professionals. Our experimental plan is designed in an academic (controlled)
setting with student participants. Thus we do not attempt to observe this
dimension as part of this work.

##  3\. Related Work

There is an extensive account of existing RA techniques, their main
characteristics, information sources, and limitations. We refer the interested
reader to the literature reviews by Tuma et al. (Tuma et al., 2018) and Ling
et al. (Ling et al., 2020). For an overview of research on diversity theory
and the relevant social processes involved in organizations (e.g., group
dynamics) see studies (Nielsen et al., 2017; van der Lee and Ellemers, 2018;
Van Dijk et al., 2012).

###  3.1. Diversity Studies in Risk Analysis

Beyond the gender theory about different risk perception (Gustafsod, 1998),
there was little scientific inquiry in the computer science domain about
gender diversity (or other diversity processes) in risk analysis in the past
10 years. When considering non-technical scenarios (e.g., returning from work
at night or natural disasters) different risk perceptions between men and
women (and ethnic minorities) have been investigated, mainly in the USA but
also in Sweden (Olofsson and Rashid, 2011). While the studies conducted in the
USA suggest that white males perceived lower risk compared to women and
minorities, the same was not observed by Olofsson and Rashid for Sweden
(Olofsson and Rashid, 2011). A recent study (Jardine, 2020) investigated the
impact of commercial Antivirus on people’s awareness of security incidents
with 1000 respondents. They found that the only relevant variables were online
insecurity, risk aversion and gender (other non-significant variables were
education, age, experience with information systems). Other empirical studies
(Fenwick and Neal, 2001; Myaskovsky et al., 2005) claim a diversity based on
participation in simple games.

Thus, we discern that existing research has focused on measuring diversity
aspects in RA when participants consider non-technical artefacts while there
is still the need to study diversity factors when considering technical (e.g.,
cybersecurity threats and mitigations) artefacts.

###  3.2. Diversity Studies in Computer Science & Software Engineering

Rodriguez et al. (Rodríguez-Pérez et al., 2021) have recently conducted a
systematic literature review to identify the diversity aspects that have been
studied in software engineering and development. Gender has been the most
widely researched dimension (61% of the papers in (Rodríguez-Pérez et al.,
2021)), while others (nationality, age, race) are less explored. This does not
come as a surprise, considering that computer science, engineering, and
physics are less gender balanced compared to other STEM fields (e.g., biology)
(Cheryan et al., 2017). In addition, existing research has focused on studying
diversity in the context of open source communities and developers (e.g., see
the study on gender bias on GitHub (Imtiaz et al., 2019)), while other domains
are less explored. For instance the role of gender-diversity in the domain of
software architecture is unexplored (Razavian and Lago, 2015; Spichkova et
al., 2017). Some tools have been proposed to assess diversity dimensions of
software or software development processes (GenderMag, InclusiveMag, and AID
Tool, to name a few). However, most studies have focused on identifying bias
through case studies, rather than conducting experiments or proposing new
models, tools, or processes (Rodríguez-Pérez et al., 2021).

GenderMag (Vorvoreanu et al., 2019) is a method to inspect software (e.g.,
interface design) to identify gender-inclusiveness issues. InclusiveMag
(Mendez et al., 2019) is a generalization of GenderMag that can be used to
generate inclusiveness methods for a particular dimension of diversity. AID
Tool (Chatterjee et al., 2021) is an automation of the GenderMag method to
detect ”inclusivity bugs” which are software features that disproportionately
disadvantage particular groups of contributors in OSS. The tool was evaluated
on 20 GitHub projects.

##  4\. Challenges

This section outlines the challenges in establishing the role of diversity in
RA.

###  4.1. Evidence of Biased Judgement

The first challenge is capturing whether diversity dimensions have an effect
on biased judgment. To observe this we plan to measure the type (i.e., gender
bias, education bias, and nationality and race bias) and amount of bias the
analyst may be projecting when making (subjective) judgements about another
analyst’s work (report of outcomes). We are interested to observe the evidence
of biased judgement in studies with and without the technical artefact and in
different populations (i.e., computer science and communication science).
Measuring such bias is relevant considering that sometimes a preferential
(managerial) choice must be made between analysis reports of two different
analysts which may belong to different diversity groups (e.g., man vs woman).
Lastly, capturing the interplay between the evidence of biased judgment and
actual diversity effects on analysis outcomes is an unexplored direction. We
may find there are no actual differences in the outcomes produced, but there
is evidence of biased judgments (or vice-versa).

###  4.2. Evidence of Diversity Effects on Outcomes

Outcome |  Outcome type
---|---
Threat |  spoofing, tampering, repudiation, information disclosure, denial of service, elevation of privilege
Assumption |  domain, security
Attack surface |  physical, close-proximity, remote
Effected security objective |  confidentiality, integrity, availability, accountability
Threat impact |  financial, safety, operational, reputation, legal (incl. privacy)
Risk priority |  high, medium, low
Threat mitigation |  preventative, detective/reactive, corrective
Table 1. Type of RA outcomes investigated in this work

The second challenge is to investigate whether diversity dimensions have an
actual effect on the task of RA. To this aim, we will measure the effects of
diversity dimensions on the RA outcomes. Since the quality of analysis lacks a
formalised definition (e.g., often natural language is used to describe attack
scenarios and informal notations are used for modeling (Tuma et al., 2018)),
we will use measures that can be easily reproduced. Namely, we can observe how
diversity dimensions effect the type of analysis outcomes. Table 1 shows
various outcomes that are often included in the analysis report. In what
follows, we describe the outcomes and motivate their inclusion for
observation.

Threats. A malicious attacker (with means and motivations) poses a threat to
harm the desired security properties of system assets. As part of the analysis
outcomes, the identified threats (e.g., spoofing an authorized user) are
accompanied by a detailed description of the attack scenarios (e.g.,
CAPEC-656: Voice Phishing
111https://capec.mitre.org/data/definitions/656.html) that realize the threat.
We borrow the threat categories from the documentation of STRIDE, a threat
modeling methodology developed by Microsoft (Shostack, 2014).

Analyses conducted by experts tend to be more balanced in terms of the
categories of identified threats, whereas RA novices tend to report
disproportionately more tampering, denial of service and information
disclosure threats (Tuma and Scandariato, 2018; Tuma et al., 2021). We are
interested to observe whether category distribution patterns emerge for other
diversity dimensions.

Assumptions. Assumptions are statements about the system under analysis that
may or may not be true (Tuma et al., 2021). While system requirements and
constraints are typically explicitly defined and static, assumptions are often
implicit and dynamic in nature (i.e., they can be invalidated and modified as
the project evolves). Due to little support offered to the analysts regarding
the assumption definition, quality assessment, and management, a recent study
investigated the role of assumptions in context of STRIDE (Van Landuyt and
Joosen, 2021). Van Landuyt and Joosen (Van Landuyt and Joosen, 2021) find that
the majority of assumptions (created by students during STRIDE) were used to
either justify an existence of threats or are used to eliminate threats. In
(Van Landuyt and Joosen, 2021) a substantial subset (78%) of the assumptions
was in direct reference to security-related concepts (i.e., security
assumptions), however also domain assumptions (statements about component
functionalities) were made.

In the safety science domain, managerial over-reliance on routine technical
inspection was identified as a common underlying characteristic of human-
created disasters (Waring, 2015). In RA, a category of experts may make
assumptions about different properties of the system and similar over-reliance
on verifying assumptions could be present. We are interested to investigate
the effect of other diversity dimensions on the type of assumptions that are
made during RA.

Attack surface. Security analysts often rely on defining the attack surface
and the required attacker profile to exploit it to determine the feasibility
of an attack scenario. Intuitively, consider the feasibility of an information
disclosure threat on internal communication channels to expose intercepted
emails by a ’script kiddie’ with little computing resources and off-the-shelf
hacking tools. The attack may require gaining physical access to the
organizations internal network and breaking TLS encryption. In addition, the
’script kiddie’ may not have a motive to launch this attack. Determining
feasibility is subjective, domain-specific and not always obvious.

Giddens et al. (Giddens et al., 2020) found that when analyzing internal
threats, managers will consider male employees as exhibiting greater malicious
intentions (associated with computer abuse) compared to female employees).
Thus, it is interesting to measure whether diversity dimensions play a role in
identifying the core variables for determining threat feasibility.

Effected security objective. The attacker goal is to compromise a security
objective (confidentiality, integrity, availability, and accountability) of an
asset of value. Since spoofing and elevation of privilege threats may
compromise multiple security objectives (e.g., a lack of authentication due to
spoofing may result in loss of confidentiality, integrity, or availability),
we are also interested to observe possible patterns of the threatened security
objectives per diversity factor.

Threat impact. Given a successful attack, threat impact refers to the damage
it causes to the organization. Estimating the type (and amount) of impact is
useful for organizations to put the threat into perspective of the domain
(e.g., when safety-critical systems are analyze, the safety impact can not be
left unmitigated). Some impacts may be easier to estimate depending on the
background and previous education. For instance, for an engineer the number of
effected units may seem easier to determine compared to legal and financial
loss estimates, for which other department expertise may be needed (Cruzes et
al., 2018). Thus, we are interested to investigate whether diversity
dimensions (e.g., education) have an effect on the type of impacts that our
participants identify.

Risk priority. Since the number of identified threats quickly explodes in
realistic projects (Tuma et al., 2021), they must be prioritized based on
estimations of risk (e.g., the last step of STRIDE (Shostack, 2014)). The most
critical threats are red flags and consequently larger investments are made to
develop and test their mitigations. We refer to risk as a product of threat
probability and impact. How individuals assess risk priorities may be related
to their risk perception which is already well understood (Gustafsod, 1998).

Mitigations. Mitigations of a cybersecurity risk can be preventative (e.g.,
implementation of two-factor authentication), detective/reactive (e.g., using
intrusion detection and access revocation techniques) and corrective (such as
maintaining audit trails or restoring from a secure state). Multiple
strategies can be adopted to counter a cybersecurity threat, and the final
choice may depend on domain-related factors, as the cost of implementing the
mitigation has to be reasonable. A category of experts (e.g., man vs woman)
may underestimate the ease with which a mitigation is actually implemented, as
observed in (Wright et al., 2002). By doing so, they in fact contribute to the
actual mitigation of the threat. Thus, we are interested to observe how
diversity dimensions effect the type of mitigations that are identified during
the analysis.

###  4.3. Identification of countermeasures to identified effects

Because risk decisions must be made in face of uncertainties for which no
statistical bias is available (Bier, 2020), organizational issues play a major
role in which the final technical artefact is chosen (Pence and Mohaghegh,
2020) (i.e., security mitigations), and thus we must put in place
technical/organizational measures to counterbalance it. Our final challenge is
to identify:

  1. i

RA practices (e.g., methods, tools, processes borrowed from adjacent
disciplines) that help foster diversity, and

  2. ii

feasible and effective countermeasures for organizations to adopt.

Identifying diversity fostering practices could be partially achieved by
conducting a more systematic review of the existing literature. However, to
evaluate the identified countermeasures empirically more future efforts will
be needed.

##  5\. Experimental Plan

Study of diversity effects and their prevalence in RA. There is a fairly
complete account and understanding of the existing RA techniques, their main
characteristics, information sources, and limitations (Tuma et al., 2018; Ling
et al., 2020). On the other hand, research has shown, with regard to gender
diversity, that there are differences in the perceived competence of men and
women (van der Lee and Ellemers, 2018). In the context of academic funding
competitions, to illustrate this process, there is evidence that men and women
produce ideas of similar quality, yet women are underestimated for their
competence (Van der Lee and Ellemers, 2015). In other words, this implies that
women are likely to be equally competent in identifying security threats as
men, yet their assessments might not be sufficiently valued. Hence, this might
have far reaching consequences for the RA as this might result in an
underestimation of the security threats. It is thus pivotal to investigate
whether diversity processes are at play during RA.

RQ1. What diversity processes play a role during security threat analysis and
risk assessment of IT systems?

Experimentation and validation of diversity interventions in RA. Previous
studies have experimented with RA techniques with practitioner and student
participants (Tuma and Scandariato, 2018; Tuma et al., 2021; Labunets et al.,
2013; Labunets et al., 2017) measuring (among others) technique productivity,
analysis outcomes, perceived ease of use. But, diversity relevant measures,
controls and relevant training for RA techniques have not yet been defined for
such experiments (Rodríguez-Pérez et al., 2021). In contrast, Van der Lee and
Ellemers (van der Lee and Ellemers, 2018) have identified characteristics of
effective diversity interventions, and applied those to practical decision
making processes. One of the key features of diversity intervention
effectiveness is the degree to which they are applicable to the current
situation. In other words, whether participants can use the knowledge that
they gained (Homan et al., 2015). The interventions thus need to promote ways
in which analysts can use their diversity to better their decision making. A
factor that might hinder this process is the perceived objectivity of the
analysts, thereby paradoxically increasing rather than decreasing implicit
biases rooted in stereotypes (Uhlmann and Cohen, 2007). Hence, to strengthen
the quality of RA the diversity interventions must be grounded in theory and
tested in the context of RA.

RQ2. What experimental protocols, metrics, and controls capture and
effectively incorporate diversity in the context of RA of IT systems?

###  5.1. Hypotheses & Methods

Figure 1. Design of the experiment with two versions of the survey for
computer science (Comp. Sci) and communication science (Comm. Sci) populations

To answer the research questions RQ1 and RQ2, we formulate two type of
hypotheses. First, we pose three hypotheses about the equivalence of the
sample means for biased judgement (e.g., by using Two One-Sided T-Tests (TOST)
(Meyners, 2012)) in presence and absence of the technical artefact.

To test hypotheses $H_{1}-H_{3}$, we designed a survey experiment with two
populations, computer science BSc students and communication science BSc
students. The survey is targeting students taking courses taught by the
experimenters. Individual participation to the survey will be voluntary. The
survey will be based on a high-level scenario (such as the ACM ethics Malware
Disruption case 222https://ethics.acm.org/code-of-ethics/using-the-code/case-
malware-disruption/) and will require the participants to understand basic
principles of security and privacy. We adopted a blocked design for the
experiment with two versions of the survey which are distributed to both
populations, as shown in Figure 1. The first will include a high-level
executive summary of the RA outcomes (in security layman terms) including the
final recommendation (i.e., threat mitigation) and a fictitious persona
description of the RA analyst. The fictitious personas (also called vignettes)
can be used to measure biased judgements. The second survey version will
include the executive summary, the final RA recommendation, the fictitious
persona, and a technical appendix with the actual RA outcomes in full detail.
Table 2 shows the dependent variables used to measure the biased judgement.
The level of each vignette dimension (gender, seniority, and mitigation) is
randomly sampled and presented to the participant (similar to survey
experiment in (Hibshi et al., 2015)) in a context of the ACM Code of Ethics
scenario. The participant will be then asked to rate their trust in the
analyst and agreement with the suggestion of the analyst. Several control
questions will also follow. The independent variables are the participant
gender and education background.

Dimension |  Level
---|---
$\$Name$ |  $\bullet$ (male) Frank
|  $\bullet$ (female) Ana
$\$Seniority$ |  $\bullet$ (senior) Senior Analyst
|  $\bullet$ (junior) Junior Analyst
$\$Mitigation$ |  $\bullet$ (detective/reactive) coordinating the web browsers with blacklists which block any incoming traffic from server hosting malicious activity.
|  $\bullet$ (corrective) engineering a worm which targets behavior resembling
“hacker” activity on the effected server.
Table 2. The vignette dimensions and levels for the survey experiment designed
to measure bias in the judgement of RA outcomes.

$H_{1}:Comp.Sci=Comm.Sci$

First, we will measure whether the judgement bias (i.e., the type of bias and
the effect size) in the computer science population is equivalent to the
judgement bias in the communication science population in absence of the
technical artefact.

$H_{2}:Comm.Sci+Artefact=Comm.Sci$

If we find that the bias is in fact equivalent across the two domains, it is
interesting to also investigate the equivalence of judgement bias for the
communication science population with the technical artefact compared to the
communication science population without the technical artefact.

$H_{3}:Comp.Sci+Artefact<>Comp.Sci$

If we find that the bias is in fact equivalent within the non-technical
domain, we must (naturally) also find bias equivalence within the technical
domain. Therefore, it is interesting to investigate the equivalence of
judgement bias for the computer science population with the technical artefact
compared to the computer science population without the technical artefact. If
our statistical analysis does not support the equivalence hypothesis, we must
further investigate how the two samples differ. For instance, if compared to
the computer science population without the technical artefact the judgement
bias of the computer science population with the technical artefact is
greater, this is evidence of a systematic bias in the computer science
community.

Figure 2. Possible (?) diversity effects and expected absence of effects in RA
outcome type studies in presence of the technical artefact

Second, we pose hypotheses about the equivalence of the sample means for the
RA outcomes in presence of the technical artefact. Figure 2 depicts hypotheses
$H_{4}:H_{6}$. To test hypotheses $H_{4}-H6$ we will conduct controlled
experiments with computer science MSc students. Concretely, we will hand out
the documentation of a software system (e.g., requirements, component and
deployment diagrams etc.) to individual students. After a training session,
the students will be tasked to perform a RA following a prescribed technique
and hand-in the RA outcomes.

$H_{4}:Comp.SciF=Comp.SciM$

Regarding gender, we expect that the outcome types reported by women are
equivalent to the outcome types reported by men. Studies of risk perception
suggest that women perceive certain risks differently compared to men. Though
we do not foresee strong differences, we might find some effects when it comes
to risk priority.

$H_{5}:Comp.Sci1=Comp.Sci2$

Regarding education, we expect that the students of various specialization
tracks report equivalent outcome types for the same system under analysis. Our
population is computer science students, with some differences in the elective
courses and program choices (e.g., we plan to include students from various
master programs, such as IA, computer Security, and Software Engineering). We
may find some outliers in the proportions of domain vs security assumptions
made, depending on the previous completed courses or the master
specialization. For example, possibly AI students make less security
assumptions, compared to Computer Security students. Though we do not have
string convictions, we may find that previous courses may have an effect on
the threat impact (e.g., students with robotics and AI background may focus on
safety impacts). Similarly, we may find come differences in the type of
mitigations considered by students of different background.

$H_{6}:Comp.SciRa=Comp.SciRb$

We expect that the students of various race and nationality report
statistically equivalent outcome types. We have not found any research that
would suggest otherwise.

The findings from investigating $H1-H6$ will play a key role in developing
diversity interventions in the form of training. To validate the diversity
training, we plan to conduct comparative experiments in a similar controlled
setting, where we administer an additional diversity training to half of the
participants.

##  6\. Potential Threats to Validity

Small female population. There is typically around 20% (or less) female
students enrolled in computer science programs. We are aware of the validity
threats caused by an unbalanced population sample, which is omnipresent in all
gender diversity studies in STEM disciplines (Rodríguez-Pérez et al., 2021).
To partially mitigate this threat, we planned a sister experiment (with a less
technically demanding task) with a communication science population (see
Section 5). Second, we will rally female computer scientist students towards
participation through local steminist groups and similar community organized
channels.

The small female population in computer science is also the rationale for
organizing the controlled experiments with computer science population
(testing hypotheses $H_{4}-H6$) with individual students. A more realistic
setup (as was done in previous studies (Tuma and Scandariato, 2018)) would
include dividing participants into teams as in practice RA is performed in
teams of experts. But measuring the individual female contribution to the
outcomes inside gender-mixed teams becomes very hard and possibly not
reliable.

Generalizability of results to practitioners. We do not plan to include
practitioners experienced in RA to participate in our studies. This limits us
in observing the full complexity of the diversity effects (e.g., including
seniority) that are actually present in organizations where RA is routinely
performed. Still, studies have shown (Runeson, 2003; Höst et al., 2000; Salman
et al., 2015) that the differences between the performance of professionals
and graduate students are often limited. To extend our observations we plan to
conduct industrial cases studies in the future (e.g., similar to the study
performed by Wang and Wagner (Wang and Wagner, 2018)).

##  7\. Conclusion

This paper outlines an experimental plan to discover the role of diversity in
cybersecurity risk analysis. We present the relevant diversity dimensions that
will be observed and discuss the challenges of capturing evidence about biased
judgement during RA and about diversity effects on the actual RA outcomes. As
a first step, we propose to measure the key differentiating diversity factors
in RA by conducting survey experiments with a technical and non-technical
population. Second, we plan to conduct controlled experiments with the
computer science population to record diversity effects on the outcomes and to
validate diversity interventions. Finally, an interview study with
practitioners would be beneficial for identifying the most urgent diversity
dimensions to be addressed first with the interventions.

###### Acknowledgements.

We express our deepest appreciation to prof. Fabio Massacci for his insightful
suggestions for the experimental plan. This research is partially supported by
the Network Institute Academy Assistant program (NIAA) at the Vrije
Universiteit and by the H2020 AssureMOSS project that received funding from
the European Union’s Horizon 2020 research and innovation program under grant
agreement No 952647. This paper reflects only the author’s view and the
Commission is not responsible for any use that may be made of the information
contained therein.

## References

  * (1)
  * Beyer et al. (2003) Sylvia Beyer, Kristina Rynes, Julie Perrault, Kelly Hay, and Susan Haller. 2003.  Gender differences in computer science students. In _Proceedings of the 34th SIGCSE technical symposium on Computer science education_. 49–53.
  * Bier (2020) Vicki Bier. 2020\.  The Role of Decision Analysis in Risk Analysis: A Retrospective.  _Risk Analysis_ 40, S1 (2020), 2207–2217.
  * Brito and Dawson (2020) Mario P Brito and Ian GJ Dawson. 2020.  Predicting the validity of expert judgments in assessing the impact of risk mitigation through failure prevention and correction.  _Risk analysis_ 40, 10 (2020), 1928–1943.
  * Chatterjee et al. (2021) Amreeta Chatterjee, Mariam Guizani, Catherine Stevens, Jillian Emard, Mary Evelyn May, Margaret Burnett, and Iftekhar Ahmed. 2021\.  AID: An Automated Detector for Gender-Inclusivity Bugs in OSS Project Pages. In _2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)_. 1423–1435.  https://doi.org/10.1109/ICSE43902.2021.00128
  * Cheryan et al. (2017) Sapna Cheryan, Sianna A Ziegler, Amanda K Montoya, and Lily Jiang. 2017.  Why are some STEM fields more gender balanced than others?  _Psychological bulletin_ 143, 1 (2017), 1\.
  * Cruzes et al. (2018) Daniela Soares Cruzes, Martin Gilje Jaatun, Karin Bernsmed, and Inger Anne Tøndel. 2018\.  Challenges and experiences with applying Microsoft threat modeling in agile development projects. In _Proceedings of the Australasian Software Engineering Conference (ASWEC)_. IEEE, 111–120.
  * Cyversity (2021) Cyversity. 2021\.  The International Consortium of Minority Cyber Professionals (non-profit).  https://www.cyversity.org/.  (Accessed on 07/01/2022).
  * Fenwick and Neal (2001) Graham D Fenwick and Derrick J Neal. 2001.  Effect of gender composition on group performance.  _Gender, Work & Organization_ 8, 2 (2001), 205–225.
  * Giddens et al. (2020) Laurie Giddens, Laura C Amo, and Dianna Cichocki. 2020\.  Gender bias and the impact on managerial evaluation of insider security threats.  _Computers & Security_ 99 (2020), 102066.
  * Gustafsod (1998) Per E Gustafsod. 1998\.  Gender Differences in risk perception: Theoretical and methodological erspectives.  _Risk analysis_ 18, 6 (1998), 805–811.
  * Hibshi et al. (2015) Hanan Hibshi, Travis D Breaux, and Stephen B Broomell. 2015\.  Assessment of risk perception in security requirements composition. In _2015 IEEE 23rd International Requirements Engineering Conference (RE)_. IEEE, 146–155.
  * Homan et al. (2015) Astrid C Homan, Claudia Buengeler, Robert A Eckhoff, Wendy P van Ginkel, and Sven C Voelpel. 2015\.  The interplay of diversity training and diversity beliefs on team creativity in nationality diverse teams.  _Journal of Applied Psychology_ 100, 5 (2015), 1456\.
  * Höst et al. (2000) Martin Höst, Björn Regnell, and Claes Wohlin. 2000\.  Using students as subjects—a comparative study of students and professionals in lead-time impact assessment.  _Empirical Software Engineering_ 5, 3 (2000), 201–214.
  * Imtiaz et al. (2019) Nasif Imtiaz, Justin Middleton, Joymallya Chakraborty, Neill Robson, Gina Bai, and Emerson Murphy-Hill. 2019.  Investigating the effects of gender bias on GitHub. In _2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE)_. IEEE, 700–711.
  * Jardine (2020) Eric Jardine. 2020\.  The Case against Commercial Antivirus Software: Risk Homeostasis and Information Problems in Cybersecurity.  _Risk Analysis_ 40, 8 (2020), 1571–1588.
  * Jaspersen and Montibeller (2015) Johannes G Jaspersen and Gilberto Montibeller. 2015.  Probability elicitation under severe time pressure: A rank-based method.  _Risk Analysis_ 35, 7 (2015), 1317–1335.
  * Labunets et al. (2017) Katsiaryna Labunets, Fabio Massacci, and Federica Paci. 2017\.  On the equivalence between graphical and tabular representations for security risk assessment. In _International Working Conference on Requirements Engineering: Foundation for Software Quality_. Springer, 191–208.
  * Labunets et al. (2013) Katsiaryna Labunets, Fabio Massacci, Federica Paci, et al. 2013\.  An experimental comparison of two risk-based security methods. In _Proceedings of the International Symposium on Empirical Software Engineering and Measurement_. IEEE, 163–172.
  * Ling et al. (2020) Engla Ling, Robert Lagerström, and Mathias Ekstedt. 2020\.  A systematic literature review of information sources for threat modeling in the power systems domain. In _International Conference on Critical Information Infrastructures Security_. Springer, 47–58.
  * Lund et al. (2010) Mass Soldal Lund, Bjørnar Solhaug, and Ketil Stølen. 2010.  _Model-driven risk analysis: the CORAS approach_.  Springer Science & Business Media.
  * Mendez et al. (2019) Christopher Mendez, Lara Letaw, Margaret Burnett, Simone Stumpf, Anita Sarma, and Claudia Hilderbrand. 2019\.  From GenderMag to InclusiveMag: An inclusive design meta-method. In _2019 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)_. IEEE, 97–106.
  * Meyners (2012) Michael Meyners. 2012\.  Equivalence tests–A review.  _Food quality and preference_ 26, 2 (2012), 231–245.
  * Myaskovsky et al. (2005) Larissa Myaskovsky, Emily Unikel, and Mary Amanda Dew. 2005\.  Effects of gender diversity on performance and interpersonal behavior in small work groups.  _Sex Roles_ 52, 9 (2005), 645–657.
  * Nielsen et al. (2017) Mathias Wullum Nielsen, Sharla Alegria, Love Börjeson, Henry Etzkowitz, Holly J Falk-Krzesinski, Aparna Joshi, Erin Leahey, Laurel Smith-Doerr, Anita Williams Woolley, and Londa Schiebinger. 2017\.  Opinion: Gender diversity leads to better science.  _Proceedings of the National Academy of Sciences_ 114, 8 (2017), 1740–1742.
  * Olofsson and Rashid (2011) Anna Olofsson and Saman Rashid. 2011.  The white (male) effect and risk perception: can equality make a difference?  _Risk Analysis: An International Journal_ 31, 6 (2011), 1016–1032.
  * Pence and Mohaghegh (2020) Justin Pence and Zahra Mohaghegh. 2020.  A Discourse on the Incorporation of Organizational Factors into Probabilistic Risk Assessment: Key Questions and Categorical Review.  _Risk Analysis_ 40, 6 (2020), 1183–1211.
  * Razavian and Lago (2015) Maryam Razavian and Patricia Lago. 2015.  Feminine expertise in architecting teams.  _IEEE Software_ 33, 4 (2015), 64–71.
  * Rodríguez-Pérez et al. (2021) Gema Rodríguez-Pérez, Reza Nadri, and Meiyappan Nagappan. 2021\.  Perceived diversity in software engineering: a systematic literature review.  _Empirical Software Engineering_ 26, 5 (2021), 1–38.
  * Runeson (2003) Per Runeson. 2003\.  Using students as experiment subjects–an analysis on graduate and freshmen student data. In _Proceedings of the International Conference on Empirical Assessment in Software Engineering_. 95–102.
  * Saini et al. (2008) Vineet Saini, Qiang Duan, and Vamsi Paruchuri. 2008.  Threat modeling using attack trees.  _Journal of Computing Sciences in Colleges_ 23, 4 (2008), 124–131.
  * Salman et al. (2015) Iflaah Salman, Ayse Tosun Misirli, and Natalia Juristo. 2015\.  Are students representatives of professionals in software engineering experiments?. In _Proceedings of the International Conference on Software Engineering-Volume 1_. IEEE Press, 666–676.
  * Shostack (2014) Adam Shostack. 2014\.  _Threat Modeling: Designing for Security_.  John Wiley & Sons. 590 pages.
  * Spichkova et al. (2017) Maria Spichkova, Heinz Schmidt, and Catia Trubiani. 2017\.  Role of women in software architecture: an attempt at a systematic literature review. In _Proceedings of the 11th European Conference on Software Architecture: Companion Proceedings_. 31–34.
  * Thomas et al. (2018) Jakita O Thomas, Nicole Joseph, Arian Williams, Jamika Burge, et al. 2018\.  Speaking truth to power: Exploring the intersectional experiences of Black women in computing. In _2018 Research on Equity and Sustained Participation in Engineering, Computing, and Technology (RESPECT)_. IEEE, 1–8.
  * Tuma et al. (2018) Katja Tuma, Gül Calikli, and Riccardo Scandariato. 2018\.  Threat analysis of software systems: A systematic literature review.  _Journal of Systems and Software_ 144 (2018), 275–294.
  * Tuma et al. (2021) Katja Tuma, Christian Sandberg, Urban Thorsson, Mathias Widman, Thomas Herpel, and Riccardo Scandariato. 2021.  Finding security threats that matter: Two industrial case studies.  _Journal of Systems and Software_ 179 (2021), 111003.
  * Tuma and Scandariato (2018) Katja Tuma and Riccardo Scandariato. 2018.  Two Architectural Threat Analysis Techniques Compared. In _Proceedings of the European Conference on Software Architecture (ECSA)_. Springer, 347–363.
  * Uhlmann and Cohen (2007) Eric Luis Uhlmann and Geoffrey L Cohen. 2007.  “I think it, therefore it’s true”: Effects of self-perceived objectivity on hiring discrimination.  _Organizational Behavior and Human Decision Processes_ 104, 2 (2007), 207–223.
  * Van der Lee and Ellemers (2015) Romy Van der Lee and Naomi Ellemers. 2015.  Gender contributes to personal research funding success in The Netherlands.  _Proceedings of the National Academy of Sciences_ 112, 40 (2015), 12349–12353.
  * van der Lee and Ellemers (2018) Romy van der Lee and Naomi Ellemers. 2018.  Perceptions of gender inequality in academia: Reluctance to let go of individual merit ideology.  In _Belief Systems and the Perception of Reality_. Routledge, 63–78.
  * Van Dijk et al. (2012) Hans Van Dijk, Marloes L Van Engen, and Daan Van Knippenberg. 2012\.  Defying conventional wisdom: A meta-analytical examination of the differences between demographic and job-related diversity relationships with performance.  _Organizational Behavior and Human Decision Processes_ 119, 1 (2012), 38–53.
  * Van Landuyt and Joosen (2021) Dimitri Van Landuyt and Wouter Joosen. 2021.  A descriptive study of assumptions in STRIDE security threat modeling.  _Software and Systems Modeling_ (2021), 1–18.
  * Vorvoreanu et al. (2019) Mihaela Vorvoreanu, Lingyi Zhang, Yun-Han Huang, Claudia Hilderbrand, Zoe Steine-Hanson, and Margaret Burnett. 2019.  From gender biases to gender-inclusive design: An empirical investigation. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_. 1–14.
  * Wang and Wagner (2018) Yang Wang and Stefan Wagner. 2018.  On groupthink in safety analysis: An industrial case study. In _Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice_. 266–275.
  * Waring (2015) Alan Waring. 2015\.  Managerial and non-technical factors in the development of human-created disasters: A review and research agenda.  _Safety science_ 79 (2015), 254–267.
  * Wright et al. (2002) George Wright, Fergus Bolger, and Gene Rowe. 2002.  An empirical test of the relative validity of expert and lay judgments of risk.  _Risk Analysis: An International Journal_ 22, 6 (2002), 1107–1122.
  * Wynn and Correll (2017) Alison T Wynn and Shelley J Correll. 2017.  Gendered perceptions of cultural and skill alignment in technology companies.  _Social Sciences_ 6, 2 (2017), 45.