# A replication of a controlled experiment with two STRIDE variants

Winnie Mbaka<EMAIL_ADDRESS>Vrije Universiteit AmsterdamThe Netherlands and
Katja Tuma<EMAIL_ADDRESS>Vrije Universiteit AmsterdamThe Netherlands

###### Abstract.

To avoid costly security patching after software deployment, security-by-
design techniques (e.g., STRIDE threat analysis) are adopted in organizations
to root out security issues before the system is ever implemented. Despite the
global gap in cybersecurity workforce and the high manual effort required for
performing threat analysis, organizations are ramping up threat analysis
activities. However, past experimental results were inconclusive regarding
some performance indicators of threat analysis techniques thus practitioners
have little evidence for choosing the technique to adopt. To address this
issue, we replicated a controlled experiment with STRIDE. Our study was aimed
at measuring and comparing the performance indicators (productivity and
precision) of two STRIDE variants (element and interaction). We conclude the
paper by comparing our results to the original study.

STRIDE, Threat Analysis, Controlled Experiment, Empirical Software
Engineering, Replication

††ccs: Software and its engineering††ccs: General and reference Empirical
studies

Pre-print 0.5

##  1\. Introduction

Security-by-design techniques (Dougherty et al., 2009; Santos et al., 2017),
aim to forefront some security effort (to the design phase) and prevent costly
security fixes to software in production. For instance, threat analysis (Tatam
et al., 2021; Ling et al., 2020) is used to scrutinize the software
architecture for potential security issues. STRIDE (Shostack, 2014) is a
popular threat analysis technique developed by Microsoft. The outcomes of such
analyses include adopting mitigations early-on or planning architectural
refactoring to limit the use of external (possibly vulnerable) libraries and
to minimize the attack surface of the software system.

Indeed, organizations are ramping up their investments in performing
architectural threat and risk analysis (latest BSIMM study gathered data from
128 organizations and reports an increase by more than 65% (Group, 2021)).
However, previous research has shown that architectural threat analysis
requires a high manual effort (Scandariato et al., 2015) and demands the
involvement of security and domain experts (Cruzes et al., 2018). But this is
hampered by a significant shortage of the security workforce. This trend has
been reported across sectors and globally. According to CyberSeek, the United
States faced a shortage of about 314,000 (44% of the total, 716,000, employed
cybersecurity workforce at the time) security professionals in January 2019
(CyberSeek, 2019). Similar gap was observed for the EU security workforce
(Blažič, 2021).

Empirical evidence of threat analysis performance indicators is a crucial
piece of the puzzle. First, such evidence supports security experts in
understanding the trade-offs between the myriad of existing threat analysis
techniques (e.g., STRIDE (Shostack, 2014), CORAS (Lund et al., 2011), attack
trees (Saini et al., 2008), threat patterns (Abe et al., 2013), PASTA (Wolf et
al., 2021), etc.). Second, favourable performance indicators would result in
cost saving for organizations where security experts are scarce. For instance,
STRIDE can be performed using two techniques, per-element and per-interaction.
The documentation of STRIDE-per-interaction includes a larger threat mapping
table is considered to be more suitable for expert use and is “too complex to
use without a reference chart handy” (Shostack, 2014). Yet, past empirical
studies were inconclusive about some performance indicators. For instance,
past controlled experiments have neither found significant difference between
STRIDE variants (Tuma and Scandariato, 2018) for precision and productivity
(whereas a significant difference was measured for recall) nor conducted a
statistical analysis of equivalence.

To address the above problems we conducted a replication of a past controlled
experiment comparing STRIDE-per-element to STRIDE-per-interaction with 45
student participants. We have replicated the analysis protocol from (Tuma and
Scandariato, 2018) and adopted the same measures of success (i.e., precision
and productivity). Similar to the previous study, we found no significant
difference (or equivalence) between the treatment groups in precision and
productivity. In addition, we also observe similar variations in the
distribution of identified threats per category (more spoofing, tampering,
information disclosure and denial of service compared to repudiation and
elevation of privilege threats) for both treatment groups. In contrast to the
original study, we found that per interaction teams performed better (albeit,
not significantly) which negates the claim that this technique is more
suitable for expert analysts. Finally, we analyze and discuss the practice of
study replication (with human participants) in software engineering. The
statistical tests used to answer to our hypotheses can be observed in the
replication package 111https://doi.org/10.5281/zenodo.6513366.

The rest of this paper is organized as follows. Section 2 discusses the
related works and replication best practices. Section 3 describes the research
questions and methodology of the replication. In Section 4 we present the
results and discuss them in Section 5. Section 6 lists the threats to the
validity of this study. Finally, Section 7 gives the concluding remarks.

##  2\. Related Work

We positioned our contributions with respect to existing literature on
empirical studies of STRIDE and replication studies in the existing literature
on requirements engineering or software design analysis.

###  2.1. Empirical studies of STRIDE

In addition to the replicated study (Tuma and Scandariato, 2018), several
works have investigated STRIDE empirically.

Scandariato et al. (Scandariato et al., 2015) performed a descriptive analysis
measuring the productivity, precision, and recall of STRIDE in an academic
setting. Their study reports similar values for a version of STRIDE-per-
element, however the conditions of the descriptive study were different
compared to our controlled experiment, therefore the results can not be
directly compared. In comparison to (Scandariato et al., 2015), this study is
replicating a controlled experiment of two STRIDE techniques.

Two studies (Cruzes et al., 2018; Bernsmed and Jaatun, 2019) conducted case
studies investigating the challenges of STRIDE. Bernsmed et al. (Bernsmed and
Jaatun, 2019) conducted semi-structured interviews (with transcription code
analysis) with agile organizations and recorded the perceived challenges. The
authors report that practitioners see value in performing STRIDE despite the
high manual effort it requires. Other discovered challenges were related to
the lack of expertise by developers conducting the analysis, and
uncompatibility of systematic approaches with the Agile workflow.

Stevens et al. (Stevens et al., 2018) conducted a qualitative case study to
investigate the efficacy the Center of Gravity (CoG) technique in an
industrial setting. The CoG, originally conceived as a military strategy, is a
risk-first threat analysis technique but has not been extensively used to
analyze software security. The authors designed surveys and classroom sessions
and involved 25 practitioners in the study. Similar to other studies conducted
with experts, they report a very high accuracy of the participant results.

###  2.2. Replications

Due to the nature of most software engineering studies, drawing reliable
conclusions from a single study might be prone to errors (Shull et al., 2004).
Hence, researchers have resolved to using replication methods as a means of
tackling these limitations. According to (Miller, 2007) replication would
allow the experiment to amass enough “evidence” to be convincing (if
successful). Generally, SE replication studies are directly related to the
concept of applying similar experimental procedures as in the original study,
on a different participation pool. According to (Santos et al., 2019), this
process is aimed at generating new/raw data. However, it is crucial to
differentiate between two important concepts, replication and reproduction. In
the latter, reproduction, researchers follow replication procedures however,
their experimental analysis is performed on the original data with the aim of
getting the same results. To this end, reproduction deals with re-analysing
raw data obtained from baseline experiments while replications leads to
generation of new data that can be combined with other studies to provide
joint conclusions (Santos et al., 2019).

Labunets et al (Labunets et al., 2013) conducted a controlled experiment that
was replicated in (Labunets et al., 2017) using student participants to
compare two risk assessment methods, a visual method (CORAS) and a textual
method (SREP). The first study found that the visual method was more effective
for identifying threats than the textual one. However, the replicated
experiment showed that the two techniques were (statistically) equivalent in
terms of the quality of identified threats and security controls. Similar to
our work, the authors conduct a TOST analysis to determine statistical
equivalence between the treatments.

Jung et al. (Jung et al., 2013) performed a within-subject experiment to
compare two safety analysis techniques, Component Integrated Fault Trees (CFT)
and Fault Trees (FT), with 7 phd candidates and replicated the experiment with
11 practitioners. Concretely, the authors compare the two techniques in terms
of quality of results and perceived method consistence, clarity, and
maintainability. Similar to our work, the authors find no major differences in
terms of result quality, and some differences were observed in the perceived
measures. In contrast to our work, Jung et al. (Jung et al., 2013) compared
safety techniques and replicated the experiment with a different population
sample (i.e., practitioners).

Several studies have empirically compared (Opdahl and Sindre, 2009; Karpati et
al., 2011, 2012) and conducted replications (Jung et al., 2013; Riaz et al.,
2017; Rueda et al., 2020) requirement engineering techniques (e.g.,
requirements elicitation). For brevity, we direct the interested reader a
comprehensive review by Ambreen et al. (Ambreen et al., 2018).

Replication best practice. In order to get a more complete overview of the
current status of empirical research in software engineering, we conducted a
literature search on some of the major (empirical) software-engineering
venues. Our sample consisted of papers form ESEM, IST, TSE and other relevant
venues (e.g., RE, JSS, ESEJ, EASE, WICSA). We extracted the literature based
on a keyword search on Scopus which leaned 159 results in the domain of
computer science. We limited the search to papers investigating requirements
engineering or software architecture (design) approaches. We manually examined
each paper, with regards to the use of empirical research methods, recruitment
of participants (from academia or industry), subject sample size and
background, experimental processes (data collection, analysis and processing),
statistical tests and results (conclusion).

Table 1. Analysis of Replicated Studies. Keyword ‘same’ refers to the
replication maintaining the exact measures as the original study, ‘similar’ to
slight variations in the measures, ‘Different’ to major variations in the
measures, ‘extra steps’ to additional measures and processes being used, and
‘complements’ to support the results and introduce relevant additional
information.

Paper |  Research Methods (EX, QEX) | RQs | Subjects | Population size |  Population Background | Data Collection | Data Cleaning | Data Processing | Statistical Test | Conclusion |  Provides Replication Package
---|---|---|---|---|---|---|---|---|---|---|---
(Wang et al., 2018) |  EX | Similar | Academia | Similar |  Same | Same | Same | Same | Additional | Partly Supports |  Yes
(Abrahão and Insfran, 2017) |  EX | Same | Academia | Similar |  Different | Similar | Same | Same | Same | Partly Supports |  Yes
(Fagerholm et al., 2019) |  EX | Different | Both | Different |  Same | Same | Same | Same | Different | Supports |  Yes
(Riaz et al., 2017) |  QEX | Different | Academia | Different |  Similar | Similar | Extra Steps | Extra Steps | Same | Partly Supports |  No
(Gómez and Acuña, 2014) |  EX | Same | Academia | Similar |  same | Similar | - | Same | Additional | Partly Supports |  No
(Kosar et al., 2012) |  EX | Same | Academia | Different |  Same | Same | Same | Similar | Different | Supports |  Upon Request
(Wagner et al., 2019) |  EX | Similar | Academia | Different |  Same | Same | Same | Same | Different | Complements |  No
(Jung et al., 2013) |  EX | Same | Industry | Similar |  Different | Same | Same | Same | Same | Supports |  No
(Perez-Gonzalez et al., 2019) |  EX | Similar | Industry | Different |  Same | Same | Same | Same | - | Partly Supports |  No
(Fernández-Sáez et al., 2012) |  EX | Same | Academia | Different |  Same | Same | Same | Same | Additional | Non conclusive |  Yes
(Ferreira et al., 2022) |  EX | Same | Both | Different |  Same | Same | Same | Same | Same | Supports |  No
(Scanniello and Erra, 2014) |  EX | Same | Academia | Same |  Same | Extra Steps | Same | Same | Different | Supports |  Yes
(Scanniello et al., 2014) |  EX | Same | Academia | Different |  Same | Extra Steps | Same | Same | Same | Non conclusive |  No
(Javed and Zdun, 2014) |  EX | Same | Academia | Different |  Similar | Same | Same | Same | Same | Supports |  No
This work |  EX | Similar | Academia | Different |  Similar | Similar | Same | Extra Steps | Different | Partly contradicts |  Yes

The resulting analysis was compared to the relevant baseline study and
populated in Table 1. In total, we analysed 15 studies (which included
replication and family of experiments) and limited our analysis to papers
produced in the last 10 years. From the analysis we noticed a few trends.

We could observe two type of replications: close replications and
differentiated replications (Lindsay and Ehrenberg, 1993). Close replications
recreate the same conditions of the original study with the aim of determining
to what extent (if at all) the original study is replicable. On the other
hand, differentiated replications include deliberate variations of the
original conditions or major effects, which allows researchers to observe the
impact of treatment variation on the study outcomes. First, close replications
(Wang et al., 2018) (Abrahão and Insfran, 2017) (Riaz et al., 2017) (Gómez and
Acuña, 2014) (Perez-Gonzalez et al., 2019) can only partially
support/contradict the original study. Indeed, the issue with close
replication studies in software engineering was already observed by Lung et
al. (Lung et al., 2008). Instead, replications that go one step further
(Fagerholm et al., 2019) (Kosar et al., 2012) (Wagner et al., 2019) (Jung et
al., 2013) (Ferreira et al., 2022) (Scanniello and Erra, 2014) (Javed and
Zdun, 2014), by either changing the way they analyse (using statistical tests)
or analyze from different perspective tend to either support or contradict
completely.

Second, we noticed that, on average, less studies provide replication
packages. Availability of replication packages for experiments encourages
better replications and complementary studies (Shull et al., 2004). However,
availability of information packages and transfer are still an open question
in SE (Solari et al., 2018). Particularly for studies involving human
participants, researchers are often additionally constrained to share the
complete data set due to privacy concerns.

Third, most studies (73% of our analysed papers) use only academic
participants. Although (Labunets et al., 2013), (Svahnberg et al., 2008) agree
that students are well suited to perform empirical studies, (Falcão et al.,
2015) argues that the low proportion of professionals used in SE experiments
may reduce experiments realism by inhibiting the understanding of industrial
software processes. Fourth, many studies do not report important information
(such as participant reward, target population, hypothesis, and conclusion and
construct validity), as also observed in (Falcão and Soares, 2021).

While (Jedlitschka et al., 2008) outlines the guidelines for reporting
controlled experiments studies, the results shown in this research is a clear
indication that some aspects of reporting need improvement. However, we
understand that some limitations (space, number of pages) are often a
determining factor to the amount of information included in reports. None-the-
less, it is important to include necessary information to facilitate better
understanding and future replication efforts (either internal or external).

##  3\. Research Methods

This work is a replication study of the controlled experiment in (Tuma and
Scandariato, 2018). This replication follows the experimental procedure of the
original study. The primary goal of the present study is to examine whether
the hypotheses in the previous study are upheld given a different
participation pool. As a secondary goal we also aim to observe how experiments
with threat analysis techniques using human participants are replicated.

###  3.1. Research questions

Our main motivation for conducting this study is to provide a reference
direction for researchers interested in undertaking controlled experiments in
the field of threat modeling using STRIDE by measuring two dependent
variables, productivity and precision. According to (Abramo and D’Angelo,
2014) productivity is the quintessential indicator of efficiency in any
production system. Similarly, in this study, productivity is defined as the
measure of correctly identified output (True positive) produced within a
specified time-frame. In practice, the STRIDE threat modeling process is time-
consuming (Scandariato et al., 2015). Any organization or team that is new to
threat modeling may be overwhelmed with information on STRIDE application and
potentially miss out on important aspects to consider. Even experienced threat
modelers may find this method more time consuming and thus adversely affecting
productivity. However, (Krishnan, 2017) claims that providing a structured
approach to threat modeling, similar to the one presented in this paper, can
be used to mitigate time constraints. On the other hand, measuring precision,
defined as correctness of analysis, is crucial in maintaining high
productivity. To this end, there is a direct correlation between high
productivity and high precision and vice versa. For instance, It would be
impractical for a team to have high precision if they have a high number of
false positives (TP¡ FP). From a scientific perspective, both productivity and
precision obtained in the previous study will be compared to this replication
in an attempt at understanding how a different participation pool perceives
and handles similar tasks. The difference, or lack thereof, of statistical
significance in productivity and precision between the two studies will be
used inform the practicability of using non-experts in practical
implementation of STRIDE. To achieve the aforementioned objectives, this study
addresses the following research questions (RQs):

RQ1:Productivity. Are the means of productivity between treatment groups
equivalent?

The following null and alternative hypotheses were formulated:

$Hprod_{0}$: The productivity difference between the two group means is
outside the equivalence interval

$Hprod_{1}$: The productivity difference between the two group means is within
the equivalence interval

RQ2:Precision. Are the means of precision between treatment groups equivalent?

The following null and alternative hypotheses were formulated:

$Hprec_{0}$: The precision difference between the two group means is outside
the equivalence interval

$Hprec_{1}$: The precision difference between the two group means is within
the equivalence interval

###  3.2. Design of the replication

The following sub-sections will describe how the replication study was
conducted. We followed best practices for replicating experiments with human
participants (discussed in Section 2) and tried to stay as close as possible
to the original design. For instance, we have adapted a similar training and
used the resources published in the original study companion web-
site222https://sites.google.com/site/empiricalstudythreatanalysis/home?authuser=0.

####  3.2.1. Compared techniques

Threat modeling is a systematic approach performed by security analysts (and
domain experts) to identify and mitigate security threats early-on in the
design phase of the Software Development Life-cycle (SDL). It includes
eliciting the threats, finding and discussing the security attacks, and
prioritizing the identified threats to plan for their mitigations. In this
work, we investigate Microsoft’s threat modeling methodology, STRIDE
(Shostack, 2014). STRIDE is an acronym representing six different threat
categories which are used to help the analysts in findings the security
threats. The threat categories, spoofing, tampering, repudiation, information
disclosure, denial of service and elevation of privilege have been described
further in the documentation of STRIDE.

Figure 1. High Level DFD (Home monitoring System)

STRIDE is a model-based approach and uses the Data-Flow Diagram (DFD) to
represent the software system under analysis. A DFD is a graphical model
representing the sources of data, the data flows through the software system,
and where it is consumed (e.g., at rest). Thus, the DFD notation consists of
data flows, external entities, processes, data stores, and trust boundaries.
The latter are used to separate DFD elements depending on the level of
privilege. For instance, Figure 1 is a DFD representation of how information
moves around in a software-based system (in this case, the home monitoring
system). Analysts can perform the analysis with STRIDE using two variants,
per-element and per-interaction (Shostack, 2014). The main difference between
the variants is the strategy of visiting the diagram (see figure 1). For more
details see Appendix Appendix A: STRIDE-per-element vs per-interaction.

####  3.2.2. Description of the Case

The home monitoring system (HomeSys) is an automated surveillance system
designed for residential places. Its main objective is to enable the home-
owner to remotely monitor their property. One of the key features of HomeSys
is its ability to send notifications when a critical event occurs (eg. sudden
spike in temperature, or smoke). The architectural documentation of a HomeSys
consists of three component diagrams:

  * •

Context diagram

  * •

Gateway component diagram

  * •

Cloud component diagram

The context diagram consists of three external entities, the mote, sensor, and
actuator (as also depicted on the DFD in Figure 1), that communicates with the
system. The customer, can access the HomeSys cloud services (depicted as
process ‘HomeSys Cloud’ in Figure 1) via a browser or a mobile application.
Using their user profiles, customers can manage their HomeSys installation and
gain access to historical data on the dashboard. The Gateway component
(depicted as process ‘Gateway’ in Figure 1) is a hardware device that enables
automation of smart homes.

To properly understand the Home monitoring system, a detailed documentation
was made available to the participants.

####  3.2.3. Participants

All participants are Computer Science Master students enrolled in a course
taught by the experimenters. At the beginning of the course an entry survey
was conducted to help the experimenters understand participants’ background
and areas of expertise relevant to the study. Majority of the students had a
background in computer science with specific areas of expertise including
artificial intelligence, internet and web technologies, software engineering,
and computer systems. The remainder (20%) of the participants had a cyber-
security background. Most of them stated that they were new to secure design
techniques (e.g STRIDE, threat modeling, Data Flow Diagrams, misuse cases,
attack trees etc). Lastly, a large number of the participants rated their
familiarity with specific software design models (e.g sequence, component and
deployment diagrams) as being very limited. However, the course material also
included training lectures on relevant technical skills like STRIDE as a
threat modeling tool and components of a DFD.

####  3.2.4. Treatment

First, the students were randomly divided into two groups (A and B)
representing the two treatment groups. Group A was tasked with analysing the
HomeSys app using STRIDE Per Element while Group B was tasked to use STRIDE
per Interaction. Next, participants within each treatment group were randomly
assigned a partner.

####  3.2.5. Task

The experiment started after the participants watched all the necessary
training videos assigned to each treatment group. The participants were then
asked to complete three tasks:

  1. (1)

Create a detailed DFD (Figure 1 is an example of a high level DFD of the
HomeSys application)

  2. (2)

Identify all the possible security threats

  3. (3)

Map the threats to their element type

Table 2. CSV Threat Template

ThreadID | ElementName | ElementType | Category | Description | Assumption
---|---|---|---|---|---
1 | External Entity X | External Entity | Spoofing | …… | …..
2 | Process X | Process | Repudiation | ….. | …..

In the first task, participants were instructed to refer to the architectural
description of the HomeSys monitoring system and use UML diagrams to create
DFDs (similar to Figure 1 ). When participants were satisfied with their DFDs,
they began analysing it using the different STRIDE variant depending on the
treatment group. Using STRIDE per-element, participants analysed every element
independently (e.g., starting in the upper-left corner of Figure 1 ). To
facilitate this analysis, mapping element types to threat categories, STRIDE
also provides a mapping chart (see Figure 1). On the other hand, STRIDE per-
interaction group would visit every interaction in their DFD to elicit a
possible threat scenario. An example of an interaction using Figure 1 is when
an actuator sends a command to the gateway. For the third task, the
participants received a threat template (see Table 2) where they categorised
each threat based on its element name, type, threat category and a description
of how the threat might occur. Optionally, the students could state their
assumptions. After completion, the participants were required to upload a copy
of their DFD accompanied by a filled-in template with the identified security
threats.

Time taken to complete the task was captured using an online survey tool. The
start time was entered when the participants started the survey and ended
after the submission of the exit questionnaire.

####  3.2.6. Measures

To analyse the participants submissions, a ground truth was developed. The
process of creating a ground truth was divided into two sub-tasks. In the
first one, one experimenter analysed the ground truth from the previous
experiment (each unique DFD had a different ground truth in the previous
experiment) to consolidate a single list of threats that are applicable in the
HomeSys monitoring system. Secondly, the ground truth was discussed between
the authors to come up with a final draft. The final ground truth was used as
a guide to mark correct and incorrect threats. Correctly marked threats were
categorised as True positives while incorrect threats were considered false
positives. Unlike the previous experiment, we do not identify False negatives
and therefore recall was not analysed. Table 3 is a visual representation of
how threat classification was done. In the first row, the participants
reported threats were compared to the ground truth. The other rows checked the
consistency within the participants reported threats.

Table 3. Criteria for Threat Classification True Positive |  False Positive
---|---
Correct element name, type, and STRIDE category |  Correct element name, type, but wrong STRIDE category
Description matching threat category |  Mismatch of description and threat category
Assumptions matching the description |  Mismatch of assumption and description
Clearly stated description and assumptions |  Lack of either description or assumption or both

Two measures of success were used to determine the outcome of the experiment,
team productivity and precision.
Productivity- in measuring the productivity of each STRIDE variant, the sum of
each teams’ TP were was divided by the time taken (in hours) to complete tasks
(see tasks). The formulae for productivity is defined below.

$Prod=\frac{TP}{Hour}$

Precision- to determine the precision, the rates of True Positives and False
Positives were considered. Using both metrics, precision was calculated using
the formulae below.

$Prec=\frac{TP}{TP+FP}$

####  3.2.7. Execution

Table 4. List of all Available Materials

Element | Interaction | Both Groups
---|---|---
| Training video:
---
STRIDE-per-element
| Training video:
---
STRIDE-per-interaction
| Training Video:
---
HomeSys monitoring system
| STRIDE book:
---
Chapter three (element)
| STRIDE book:
---
Chapter three (interaction)
| HomeSys
---
Documentation

Training- Prior to the start of the experiment,the participants were given
access to comprehensive training videos the day before via a learning
management system and instructed to watch them. The training materials
included a description of the HomeSys monitoring system, a STRIDE training
video which was different depending on the group (group A received STRIDE per
Element). Additional documentation was also provided, Table 4 is a detailed
list of all materials provided to each treatment group.

Physical Labs- The experiment was conducted during a two hour class where the
different teams were allocated different labs depending on their treatment
group. Due to the large number of teams (22 pairs), the experimenters
separated the groups into four labs with each lab being supervised by either a
teaching assistant or the course lecturers. The participants were allowed to
ask the supervisors questions but team interactions were limited to partners.

Reports- All necessary metrics, such as duration taken to complete the task,
the group member who submitted the report, and the uploaded DFDs and CSV
templates were captured on Qualtrics. The last step in the experiment
consisted of an exit questionnaire which consisted of questions aimed at
gauging the participants experience, familiarity, confidence on their
performance, and general knowledge on the subject. The responses were captured
on a Likert scale.

##  4\. Results

In addition to the analysis performed in (Tuma and Scandariato, 2018), if we
can not find significant difference, we also check whether the location shift
between the two sample means in precision is statistically equivalent. For
both measures (productivity and precision) we performed a T-test equivalence
test to check whether the location shift between the two sample means in the
treatment groups is statistically equivalent. The delta was estimated by the
authors due to lack of larger samples from other similar studies that could be
used to empirically derive the delta estimate. For productivity, we consider a
delta of 0.5 correct threats per hour as equivalent, while for precision we
consider a delta of 5% as equivalent.

The outcome of the study will be presented in this section by answering the
research questions.

###  4.1. True Positives and False Positives

Table 5. Summary Observations in Per-Element Group

Element Group (ID) | Time Taken (hours) | TP | FP | Productivity | Precision
---|---|---|---|---|---
0 | 1.20 | 3 | 10 | 2.49 | 0.23
1 | 1.65 | 4 | 6 | 2.41 | 0.40
2 | 1.67 | 2 | 6 | 1.19 | 0.25
3 | 1.56 | 7 | 13 | 4.47 | 0.35
4 | 1.70 | 7 | 1 | 4.09 | 0.87
5 | 1.90 | 7 | 3 | 3.66 | 0.70
6 | 1.72 | 0 | 7 | 0.00 | 0.00
7 | 2.40 | 10 | 5 | 4.15 | 0.66
8 | 2.49 | 5 | 5 | 2.00 | 0.50
9 | 3.16 | 9 | 6 | 2.84 | 0.60
10 | 5.48 | 3 | 10 | 0.54 | 0.23
11 | 2.27 | 7 | 63 | 3.08 | 0.10
Mean | 2.27 | 5.33 | 11.25 | 2.5 | 0.40
Standard Deviation | 1.13 | 2.99 | 16.61 | 1.19 | 0.26

Table 6. Summary Observations in Per-interaction Group

Interaction Group (ID) | Time Taken (hours) | TP | FP | Productivity | Precision
---|---|---|---|---|---
0 | 1.93 | 9 | 5 | 4.65 | 0.64
1 | 0.92 | 10 | 30 | 10.81 | 0.25
2 | 2.35 | 1 | 5 | 0.42 | 0.16
3 | 1.94 | 8 | 4 | 4.10 | 0.66
4 | 1.90 | 9 | 2 | 4.71 | 0.81
5 | 1.91 | 8 | 4 | 4.17 | 0.66
6 | 1.92 | 4 | 27 | 2.07 | 0.12
7 | 2.02 | 0 | 11 | 0.00 | 0.00
8 | 2.08 | 9 | 5 | 4.30 | 0.64
9 | 2.07 | 1 | 2 | 0.48 | 0.33
Mean | 1.91 | 5.90 | 9.50 | 3.57 | 0.43
Standard Deviation | 0.37 | 3.95 | 10.34 | 3.17 | 0.28

Table 5 and 6 show a summary of all necessary observations in both groups
(Identified TP, FP, overall productivity, and precision for each team). The
average number of True positives identified by the per-Element group was 5.33
(STDEV= 2.99) with a corresponding average False positive of 11.25
(STDEV=16.61). The per-Interaction group reported an average of 5.90 (STDEV=
3.95) True Positives and 9.50 (STDEV=10.34) false positives. Figure 2 is a
representation of the average TP and FP of each threat category across the
treatment groups. To understand the high occurrence of FP in both treatment
groups, we further analysed the submitted threat templates. We discovered that
at least one team in each treatment group listed a huge number of threat
categories without a corresponding description, assumption, or both despite
repeated warning during the training to avoid making this mistake. According
to the criteria for threat classification presented on Table 3, such mistakes
automatically led to the classification of some threat categories as false
positives regardless of whether or not they were correctly matched to the
element name and element type. The authors concluded that the members of these
two teams lacked an understanding of what the task required them to do. While
the teams performed relatively better in identifying spoofing, information
disclosure, and denial of service threats, the high frequency of FPs were
attributed to the mistake mentioned above. However, a significant amount of
tampering threats identified had been matched with the wrong element type, a
common occurrence in threat analysis using non-experts (Shostack, 2008).
Generally, repudiation and elevation of privilege attacks were less identified
with an average of two attacks per team. For teams that listed more than two
repudiation or elevation of privilege attacks, they either lacked a threat
description, an assumption or both.

###  4.2. RQ1: Productivity

The study defined productivity as the number of correctly identified threats
(True Positives) per time taken to complete the task (in hours). True
positives were also used to represent the amount of work (effort) performed by
a team. The average time spent by the teams performing STRIDE per element is
2.27 h (STDEV=1.13), while STRIDE per interaction teams spent an average of
1.9 h (STDEV= 0.37) (not statistically significant).The overall productivity
(calculated as TP/Hrs) was dependant on the amount of correctly identified
threats. The higher the TP the higher the average productivity, and vice
versa. As depicted in Table 5, the average productivity in the per-element
group is 2.5 TP/h (STDEV= 1.19), and per-interaction teams is 3.57 TP/h
(STDEV=3.17). The equivalence test on productivity produced a 0.7159 p-value,
supporting the null hypothesis ($Hprod_{0}$). Therefore, we conclude that the
groups productivity is not equivalent.

###  4.3. RQ2: Precision

The study defined precision as the correctness of the analysis calculated as
TP/TP+FP. Overall, the average precision for each treatment group was 0.43
(STDEV= 0.28) for per-interaction and 0.40 (STDEV=0.26) for per-element.
Similar to the original study, there is no statistically significant
difference between the precision of both teams. However, we conducted an
equivalence test (p-value=0.3986), since the p-value is outside the
equivalence interval, supporting the null hypothesis ($Hprec_{0}$), we
concluded that the location shift between the two sample means of precision
from both treatment groups is statistically not equivalent. Figure 3 and 4
display precision for each threat category (median-yellow line, mean-green
triangle).

Figure 2. Precision Per Threat Category (per-element) Figure 3. Precision Per
Threat Category (per-element) Figure 4. Precision Per Threat Category (per-
interaction)

###  4.4. Exit Questionnaire

The final step in the experiment was conducting an exit questionnaire, meant
to gauge the students perceived complexity in performing the task. Their
responses were recorded using a predefined likert scale. The training
materials made available to the participants was instrumental in improving
their familiarity with the home monitoring system (HomeSys), evident from the
exit questionnaire. Both treatment groups found the process of building a DFD
neither challenging nor easy, which also reflected on the quality of the DFD
they uploaded. Analyzing the DFD and threat mapping using the different
variant charts was considered to be easy by both teams. However, participants
strongly agreed that the process was time consuming and tedious. There was a
slight difference in how much each treatment group liked their technique.
Majority of the teams in Per-element chose ”I do not like it” while Per-
interaction teams stated that they ”quite liked it”, which can be used to
explain their slightly better productivity.

##  5\. Discussion

This section reports analysis from comparing our results to the original
study.

Comparison of True Positives and False Positives. The original study (Tuma and
Scandariato, 2018) reported slightly better averages of TP from Per-element
group (14.3) compared to per-interaction (11.6). However, the average FPs
recorded were relatively lower for per-element (14.2) while per-interaction
had 15 FP. In comparison, our study reported lower averages across both teams
(per-element: 5.33 TP and 11.25 FP vs per-interaction: 5.90 TP and 9.50 FP).
Overall, the original study produced higher averages across both treatment
groups. We conclude this was a result of the original study having relatively
higher participants per team (4-5 students) while the replication had 2
students per team. However, analysis done on both studies show that there were
no significant differences between the average number of TP and FP across
treatment groups.

Comparison of productivity. As discussed in Section 4, the average time spent
performing the tasks in the replication was not statistically different,
similar to the original study from (Tuma and Scandariato, 2018). Replication
(per-element: 2.27 h, per-interaction: 1.9 h), original (per-element: 3.5 h,
per-interaction: 3.95 h). The resulting average productivity across both
studies were also not statistically different. Replication productivity (per-
element: 2.5 TP/h, per-interaction: 3.57 TP/h), original productivity (per-
element: 4.35 TP/h, per-interaction: 3.27 TP/h). It is clear that per-
interaction group in the replication study performed better than in the
original. Productivity is driven by the number of TP, compared to the original
study, the productivity was much lower in the replication. In contrast to the
past study, in the replication, per-interaction was slightly better (not
significant) in terms of productivity. A possible explanation for this is the
tendency of per-interaction teams to liking the technique more (as described
already in 4.4)

Comparison of precision. We also compared the precision of teams from (Tuma
and Scandariato, 2018) and our treatments groups. Similar to productivity, the
resulting precision measure by our replication was similar to the precision
measured in the past study. Replication precision (per-element: 0.40, per-
interaction: 0.43), original precision (per-element: 0.62, per-interaction:
0.58). Although the difference is not significant, per-interaction teams in
the replication study were slightly more precise than per-element (again, the
reverse is true for the original study).

Comparison of threat categories. To complete our analysis, we compared the
average number of identified threats per category across each treatment group
in the original and replication studies. On average, both treatment groups
identified more (TPs and FPs) spoofing, tampering, information disclosure and
denial of service compared to repudiation and elevation of privilege threats.
This trend was also observed in other studies (Tuma and Scandariato,
2018)(Scandariato et al., 2015). We concluded that such threats are easily
identifiable by participants regardless of the treatment group. On the other
hand, repudiation and elevation of privilege threats were less identified
(averagely lower TP). A possible explanation for this might be that students
lack a clear understanding of how vulnerabilities can be exploited to result
in repudiation or elevation of privilege.

##  6\. Threat to validity

This section will discuss the various limitations with the experiment that
might pose a threat to validity to the results.

Participant selection and sample size. The use of student participants as
opposed to experts might be considered a treat to validity due to their
limited knowledge of implementing industrial-level techniques. However,
(Labunets et al., 2013) states that student participants are well-suited for
empirical experiments. The sample size, 45 students, was relatively smaller
than in the original study (110 students). Since the experiment was conducted
in-person, the authors were satisfied with this sample size because it was
easy to manage given the Covid-19 mandates on social distancing in public
gatherings.

Participants’ Motivation. Participation in the experiment was part of the
learning objectives of the course, however only participation was graded (as a
pass or fail) and not their performance in the task of the experiment. This
was an important decision because alternatively, students tend to submit more
security threats (which results in many false positives) to maximize their
grade. Second, the students actually used the data they produced as part of an
assignment in the course. Hence the students were motivated to undertake the
study seriously.

Group Composition and Execution. Compared to the previous study whereby teams
were made up of 3-5 participants, our replication consisted of two members per
team with an exception of one team (3 members). The reduced team composition
might present an unequal distribution of knowledge of task or techniques used
in the experiment. However, we provided access to all training materials 24
hours prior to the experiment. Participants could access these materials at
anytime, even during the experiment. Additionally, there was at-least one
lecturer or teaching assistant present in the labs to answer any questions
that might arise. We remark that the time given to participants to perform the
task in future replications should be proportional to the system under
analysis (in our replication we allotted about 2h for the analysis, whereas
the original study allotted about 4h).

##  7\. Conclusion

This study focuses on threat modelling techniques by investigating performance
indicators (productivity and precision) of two STRIDE variants (Element and
Interaction). We conducted a replication of the original study (Tuma and
Scandariato, 2018) in a controlled experiment. Our participants are enrolled
Master’s students with a background in computer science and cyber-security.
The contribution of this work is as follows:

  1. (1)

A quantitative comparison of performance indicators of STRIDE by testing the
hypothesis of the original study on a different participation pool. From the
results, study found that per-interaction teams had better productivity and
precision compared to per-element while in the original study per-element
performed better in both measures.

  2. (2)

A comparative analysis of SE replicated studies with the aim of providing
suggestions for future replication designs. For the best practices of
replication, we analysed 15 technical papers obtained from some of the major
software-engineering venues, and documented the trends identified. We also
explained how these trends may be a hindrance to performing complete (and
successful) replications.

As future work, we plan to conduct similar studies with industry professionals
with (possibly) prior experience in threat analysis. In addition it would be
interesting to observe performance (and perceived performance) differences
between expert to non-expert populations. Finally, the target population of
our study (and related studies investigating STRIDE) was computer science
students (or professionals in other studies), however, such a sample is not
very diverse e.g., in terms of gender. We are interested to observe whether
diversity factors plays a role in security threat analysis and to what extent
biased decision making (if any) is more pronounced in the computer science
domain when compared to populations with different educational backgrounds
(e.g., with a primary background in social sciences).

## Acknowledgements

Witheld for anonymous submission

## References

  * (1)
  * Abe et al. (2013) Tatsuya Abe, Shinpei Hayashi, and Motoshi Saeki. 2013\.  Modeling security threat patterns to derive negative scenarios. In _Proceedings of the Asia-Pacific Software Engineering Conference (APSEC)_ , Vol. 1. IEEE, 58–66.
  * Abrahão and Insfran (2017) S. Abrahão and E. Insfran. 2017.  Evaluating software architecture evaluation methods: An internal replication.  _ACM International Conference Proceeding Series_ Part F128635 (2017), 144–153.  https://doi.org/10.1145/3084226.3084253
  * Abramo and D’Angelo (2014) Giovanni Abramo and Ciriaco Andrea D’Angelo. 2014\.  How do you define and measure research productivity?  _Scientometrics_ 101, 2 (2014), 1129–1144.
  * Ambreen et al. (2018) Talat Ambreen, Naveed Ikram, Muhammad Usman, and Mahmood Niazi. 2018.  Empirical research in requirements engineering: trends and opportunities.  _Requirements Engineering_ 23, 1 (2018), 63–95.
  * Bernsmed and Jaatun (2019) Karin Bernsmed and Martin Gilje Jaatun. 2019.  Threat modelling and agile software development: Identified practice in four Norwegian organisations. In _2019 International Conference on Cyber Security and Protection of Digital Services (Cyber Security)_. IEEE, 1–8.
  * Blažič (2021) Borka Jerman Blažič. 2021\.  Cybersecurity Skills in EU: New Educational Concept for Closing the Missing Workforce Gap.  In _Cybersecurity Threats with New Perspectives_. IntechOpen.
  * Cruzes et al. (2018) Daniela Soares Cruzes, Martin Gilje Jaatun, Karin Bernsmed, and Inger Anne Tøndel. 2018\.  Challenges and experiences with applying microsoft threat modeling in agile development projects. In _2018 25th Australasian Software Engineering Conference (ASWEC)_. IEEE, 111–120.
  * CyberSeek (2019) CyberSeek. 2019\.  _Cybersecurity Supply/Demand Heat Map_.  Retrieved April 21, 2022 from https://www.cyberseek.org/heatmap.html
  * Dougherty et al. (2009) Chad Dougherty, Kirk Sayre, Robert C Seacord, David Svoboda, and Kazuya Togashi. 2009.  _Secure Design Patterns_.  Technical Report. Carnegie-Mellon University Pittsburgh, Software Engineering Institute.
  * Fagerholm et al. (2019) F. Fagerholm, C. Becker, A. Chatzigeorgiou, S. Betz, L. Duboc, B. Penzenstadler, R. Mohanani, and C.C. Venters. 2019\.  Temporal Discounting in Software Engineering: A Replication Study.  _International Symposium on Empirical Software Engineering and Measurement_ 2019-Septemer (2019).  https://doi.org/10.1109/ESEM.2019.8870161
  * Falcão et al. (2015) L. Falcão, W. Ferreira, A. Borges, V. Nepomuceno, S. Soares, and M.T. Baldassare. 2015\.  An Analysis of Software Engineering Experiments Using Human Subjects.  _International Symposium on Empirical Software Engineering and Measurement_ 2015-November (2015), 128–131.  https://doi.org/10.1109/ESEM.2015.7321185
  * Falcão and Soares (2021) L. Falcão and S. Soares. 2021.  Human-Oriented Software Engineering Experiments: The Large Gap in Experiment Reports.  _ACM International Conference Proceeding Series_ (2021), 330–334.  https://doi.org/10.1145/3474624.3474649
  * Fernández-Sáez et al. (2012) A.M. Fernández-Sáez, M. Genero, and M.R.V. Chaudron. 2012\.  Does the level of detail of UML models affect the maintainability of source code?  _Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)_ 7167 LNCS (2012), 134–148.  https://doi.org/10.1007/978-3-642-29645-1_15
  * Ferreira et al. (2022) J.M. Ferreira, F. Rodriguez, A. Santos, O. Dieste, S.T. Acuna, and N. Juristo. 2022\.  Impact of Usability Mechanisms: A Family of Experiments on Efficiency, Effectiveness and User Satisfaction.  _IEEE Transactions on Software Engineering_ (2022).  https://doi.org/10.1109/TSE.2022.3149586
  * Gómez and Acuña (2014) Marta N Gómez and Silvia T Acuña. 2014.  A replicated quasi-experimental study on the influence of personality and team climate in software development.  _Empirical software engineering_ 19, 2 (2014), 343–377.
  * Group (2021) Synopsys Software Integrity Group. 2021\.  _Building Security In Maturity Model (BSIMM12)_.  Retrieved April 20, 2022 from https://www.bsimm.com
  * Javed and Zdun (2014) M.A. Javed and U. Zdun. 2014.  The supportive effect of traceability links in architecture-level software understanding: Two controlled experiments.  _Proceedings - Working IEEE/IFIP Conference on Software Architecture 2014, WICSA 2014_ (2014), 215–224.  https://doi.org/10.1109/WICSA.2014.43
  * Jedlitschka et al. (2008) Andreas Jedlitschka, Marcus Ciolkowski, and Dietmar Pfahl. 2008\.  Reporting experiments in software engineering.  In _Guide to advanced empirical software engineering_. Springer, 201–228.
  * Jung et al. (2013) J. Jung, K. Hoefig, D. Domis, A. Jedlitschka, and M. Hiller. 2013.  Experimental comparison of two safety analysis methods and its replication.  _International Symposium on Empirical Software Engineering and Measurement_ (2013), 223–232.  https://doi.org/10.1109/ESEM.2013.59
  * Karpati et al. (2011) Peter Karpati, Andreas L Opdahl, and Guttorm Sindre. 2011\.  Experimental comparison of misuse case maps with misuse cases and system architecture diagrams for eliciting security vulnerabilities and mitigations. In _Availability, Reliability and Security (ARES), 2011 Sixth International Conference on_. IEEE, 507–514.
  * Karpati et al. (2012) Peter Karpati, Guttorm Sindre, and Raimundas Matulevicius. 2012\.  Comparing misuse case and mal-activity diagrams for modelling social engineering attacks.  _International Journal of Secure Software Engineering (IJSSE)_ 3, 2 (2012), 54–73.
  * Kosar et al. (2012) Tomaž Kosar, Marjan Mernik, and Jeffrey C Carver. 2012\.  Program comprehension of domain-specific and general-purpose languages: comparison using a family of experiments.  _Empirical software engineering_ 17, 3 (2012), 276–304.
  * Krishnan (2017) Sriram Krishnan. 2017\.  A hybrid approach to threat modelling.  _URL https://blogs. sans. org/appsecstreetfighter/files/2017/03/A-Hybrid-Approach-to-Threat-Modelling. pdf,[Accessed on 10-Jul-2018]_ (2017).
  * Labunets et al. (2017) Katsiaryna Labunets, Fabio Massacci, and Federica Paci. 2017\.  On the equivalence between graphical and tabular representations for security risk assessment. In _International Working Conference on Requirements Engineering: Foundation for Software Quality_. Springer, 191–208.
  * Labunets et al. (2013) Katsiaryna Labunets, Fabio Massacci, Federica Paci, et al. 2013\.  An experimental comparison of two risk-based security methods. In _2013 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement_. IEEE, 163–172.
  * Lindsay and Ehrenberg (1993) R Murray Lindsay and Andrew SC Ehrenberg. 1993.  The design of replicated studies.  _The American Statistician_ 47, 3 (1993), 217–228.
  * Ling et al. (2020) Engla Ling, Robert Lagerström, and Mathias Ekstedt. 2020\.  A systematic literature review of information sources for threat modeling in the power systems domain. In _International Conference on Critical Information Infrastructures Security_. Springer, 47–58.
  * Lund et al. (2011) Mass Soldal Lund, Bjørnar Solhaug, and Ketil Stølen. 2011.  _Model-driven Risk Analysis: The Coras Approach_.  Springer.
  * Lung et al. (2008) J. Lung, J. Aranda, S. Easterbrook, and G. Wilson. 2008\.  On the difficulty of replicating human subjects studies in software engineering.  _Proceedings - International Conference on Software Engineering_ (2008), 191–200.  https://doi.org/10.1145/1368088.1368115
  * Miller (2007) J. Miller. 2007\.  Creating real value in software engineering experiments.  _Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)_ 4336 LNCS (2007), 38\.  https://doi.org/10.1007/978-3-540-71301-2_11
  * Opdahl and Sindre (2009) Andreas L Opdahl and Guttorm Sindre. 2009.  Experimental comparison of attack trees and misuse cases for security threat identification.  _Information and Software Technology_ 51, 5 (2009), 916–932.
  * Perez-Gonzalez et al. (2019) H.G. Perez-Gonzalez, A.S. Nunez-Varela, F.E. Martinez-Perez, F.E. Hernandez-Castro, F. Torres-Reyes, R. Juarez-Ramirez, K. Bauer, and C. Guerra-Garcia. 2019.  Exploring Software Design Skills of Students in different Stages of their Curriculum.  _Proceedings - 2019 7th International Conference in Software Engineering Research and Innovation, CONISOFT 2019_ (2019), 65–71.  https://doi.org/10.1109/CONISOFT.2019.00019
  * Riaz et al. (2017) M. Riaz, J. King, J. Slankas, L. Williams, F. Massacci, C. Quesada-López, and M. Jenkins. 2017.  Identifying the implied: Findings from three differentiated replications on the use of security requirements templates.  _Empirical Software Engineering_ 22, 4 (2017), 2127–2178.  https://doi.org/10.1007/s10664-016-9481-1
  * Rueda et al. (2020) S. Rueda, J.I. Panach, and D. Distante. 2020.  Requirements elicitation methods based on interviews in comparison: A family of experiments.  _Information and Software Technology_ 126 (2020).  https://doi.org/10.1016/j.infsof.2020.106361
  * Saini et al. (2008) Vineet Saini, Qiang Duan, and Vamsi Paruchuri. 2008.  Threat Modeling Using Attack Trees.  _Consortium for Computing Sciences in Colleges_ 23, 4 (2008), 124–131.
  * Santos et al. (2019) Adrian Santos, Sira Vegas, Markku Oivo, and Natalia Juristo. 2019.  A procedure and guidelines for analyzing groups of software engineering replications.  _IEEE Transactions on Software Engineering_ 47, 9 (2019), 1742–1763.
  * Santos et al. (2017) Joanna C. S. Santos, Katy Tarrit, and Mehdi Mirakhorli. 2017\.  A Catalog of Security Architecture Weaknesses. In _Proceedings of the International Conference on Software Architecture Workshops (ICSAW)_. IEEE Computer Society, 220–223.  https://doi.org/10.1109/ICSAW.2017.25
  * Scandariato et al. (2015) Riccardo Scandariato, Kim Wuyts, and Wouter Joosen. 2015\.  A descriptive study of Microsoft’s threat modeling technique.  _Requirements Engineering_ 20, 2 (2015), 163–180.
  * Scanniello and Erra (2014) G. Scanniello and U. Erra. 2014.  Distributed modeling of use case diagrams with a method based on think-pair-square: Results from two controlled experiments.  _Journal of Visual Languages and Computing_ 25, 4 (2014), 494–517.  https://doi.org/10.1016/j.jvlc.2014.03.002
  * Scanniello et al. (2014) G. Scanniello, M. Staron, H. Burden, and R. Heldal. 2014\.  On the effect of using SysML requirement diagrams to comprehend requirements: Results from two controlled experiments.  _ACM International Conference Proceeding Series_ (2014).  https://doi.org/10.1145/2601248.2601259
  * Shostack (2008) Adam Shostack. 2008\.  _Experiences Threat Modeling at Microsoft_.  http://ftp.informatik.rwth-aachen.de/Publications/CEUR-WS/Vol-413/paper12.pdf
  * Shostack (2014) Adam Shostack. 2014\.  _Threat modeling: Designing for security_.  John Wiley & Sons.
  * Shull et al. (2004) Forrest Shull, Manoel G Mendoncça, Victor Basili, Jeffrey Carver, José C Maldonado, Sandra Fabbri, Guilherme Horta Travassos, and Maria Cristina Ferreira. 2004.  Knowledge-sharing issues in experimental software engineering.  _Empirical Software Engineering_ 9, 1 (2004), 111–137.
  * Solari et al. (2018) Martín Solari, Sira Vegas, and Natalia Juristo. 2018\.  Content and structure of laboratory packages for software engineering experiments.  _Information and Software Technology_ 97 (2018), 64–79.
  * Stevens et al. (2018) R Stevens, D Votipka, and E.M. Redmiles. 2018.  The Battle for New York: A Case Study of Applied Digital Threat Modeling at the Enterprise Level. In _SEC’18: Proceedings of the 27th USENIX Conference on Security Symposium_. USENIX Association, 621–637.
  * Svahnberg et al. (2008) Mikael Svahnberg, Aybüke Aurum, and Claes Wohlin. 2008\.  Using students as subjects-an empirical evaluation. In _Proceedings of the Second ACM-IEEE international symposium on Empirical software engineering and measurement_. 288–290.
  * Tatam et al. (2021) Matt Tatam, Bharanidharan Shanmugam, Sami Azam, and Krishnan Kannoorpatti. 2021.  A review of threat modelling approaches for APT-style attacks.  _Heliyon_ 7, 1 (2021), e05969.
  * Tuma and Scandariato (2018) Katja Tuma and Riccardo Scandariato. 2018.  Two architectural threat analysis techniques compared. In _European Conference on Software Architecture_. Springer, 347–363.
  * Van Landuyt and Joosen (2021) Dimitri Van Landuyt and Wouter Joosen. 2021.  A descriptive study of assumptions in STRIDE security threat modeling.  _Software and Systems Modeling_ (2021), 1–18.
  * Wagner et al. (2019) S. Wagner, D.M. Fernández, M. Felderer, A. Vetrò, M. Kalinowski, R. Wieringa, D. Pfahl, T. Conte, M.-T. Christiansson, D. Greer, C. Lassenius, T. Männistö, M. Nayebi, M. Oivo, B. Penzenstadler, R. Prikladnicki, G. Ruhe, A. Schekelmann, S. Sen, R. Spínola, A. Tuzcu, J.L. De La Vara, and D. Winkler. 2019\.  Status quo in requirements engineering: A theory and a global family of surveys.  _ACM Transactions on Software Engineering and Methodology_ 28, 2 (2019).  https://doi.org/10.1145/3306607
  * Wang et al. (2018) Y. Wang, D.R. Degutis, and S. Wagner. 2018.  Speed up BDD for safety verification in agile development: A partially replicated controlled experiment.  _ACM International Conference Proceeding Series_ Part F147763 (2018).  https://doi.org/10.1145/3234152.3234181
  * Wolf et al. (2021) Andreas Wolf, Dimitrios Simopoulos, Luca D’Avino, and Patrick Schwaiger. 2021.  The PASTA threat model implementation in the IoT development life cycle.  _INFORMATIK 2020_ (2021).

## Appendix A: STRIDE-per-element vs per-interaction

In addition to DFD creation, analysts often make assumptions about the system
under analysis. Assumptions are statements (which may or may not be true)
about the domain, functionality and security of its’ components. They can be
identified when the diagram is built and they can arise during the analysis.
Landuyt (Van Landuyt and Joosen, 2021) defines threat assumptions as
information used to hypothesise certain system properties that are relevant to
the attack identified. Threat assumptions are important in threat analysis as
they can be used to justify threat existence and prioritise mitigation
efforts.

After the threats are elicited, analysts discuss attack scenarios and document
them with threat descriptions. A threat description contains concrete steps of
the identified security threats and how they may impact system components
associated to it. Finally, the threat can be prioritized according to
estimation of risk (this step is not investigated in our work).

STRIDE-per-Element. This technique is performed by analysing each component in
a DFD individually. To limit effects of threat explosion, the documentation of
STRIDE provides a threat to element mapping table. For each element type a
different subset of threat categories are suggested. For instance, for
external entities (such as “Sensor” in Figure 1), (Shostack, 2014) suggests
investigating spoofing and repudiation threats. STRIDE-per-element is
described as a simplified approach to identifying threats that can be easily
understood by the beginner (Shostack, 2014).

STRIDE-per-Interaction. However, threats can not be always discussed in a
vacuum as they are a result of some interaction in the system. An interaction
is a tuple of origin, destination, and the interaction itself (e.g., the
“Customer” sending “credentials” to the “HomeSys Cloud” in Figure 1). In
contrast to STRIDE-per-element, this technique is performed by systematically
visiting each interaction in a DFD. For instance, when an external entity,
such as “Sensor”, is passing information to a process, such as “Gateway” with
no logging in place, the Gateway is able to deny having received sensitive
information from the Sensor. In this case, (Shostack, 2014) suggests
investigating repudiation threats. Again, to limit the number of threats that
the analysts need to consider, a threat to interaction mapping table is used
(Shostack, 2014). STRIDE-per-interaction is considered to be more suitable for
expert use and is “too complex to use without a reference chart handy”
(Shostack, 2014).