[1]

Investigation

Conceptualization

[1]Corresponding author

# Common Privacy Weaknesses and Vulnerabilities in Software Applications

Pattaraporn Sangaroonsilp<EMAIL_ADDRESS>organization=School of
Computing and Information Technology, Faculty of Engineering and Information
Sciences, University of Wollongong, country=Australia  Hoa Khanh Dam
<EMAIL_ADDRESS>Aditya Ghose<EMAIL_ADDRESS>
###### Abstract

In this digital era, our privacy is under constant threat as our personal data
and traceable online/offline activities are frequently collected, processed
and transferred by many software applications. Privacy attacks are often
formed by exploiting vulnerabilities found in those software applications. The
Common Weakness Enumeration (CWE) and Common Vulnerabilities and Exposures
(CVE) systems are currently the main sources that software engineers rely on
for understanding and preventing publicly disclosed software vulnerabilities.
However, our study on all 922 weaknesses in the CWE and 156,537
vulnerabilities registered in the CVE to date has found a very small coverage
of privacy-related vulnerabilities in both systems, only 4.45% in CWE and 0.1%
in CVE. These also cover only a small number of areas of privacy threats that
have been raised in existing privacy software engineering research, privacy
regulations and frameworks, and industry sources. The actionable insights
generated from our study led to the introduction of 11 new common privacy
weaknesses to supplement the CWE system, making it become a source for both
security and privacy vulnerabilities.

###### keywords:

Privacy VulnerabilitiesThreatsSoftware engineering

##  1 Introduction

Technologies and digitalisation are rapidly emerging in our society. People
significantly rely on software applications and computing devices in their
daily lives, consequently leaving their traceable digital activities,
contributions and communications on those digital devices across the Internet
(Johnson, 2021b). Even when people are not using software, data about their
normal life activities may also be collected by software applications through
the ubiquity of IoT and GPS devices, surveillance cameras, face recognition
apps and so on. Thus, our privacy is under constant threat in this current
digital age. In fact, privacy invasions and attacks have been increasing
significantly in recent years (Statista, 2021; The Office of the Australian
Information Commissioner (2021), OAIC). For examples, a cyber crime in the
U.S. in 2018 exposed 471 million personal records, and a breach of a national
ID database in India leaked over 1.1 billion records including biometric
information (e.g., iris and fingerprint scans) (Johnson, 2021a). Those
incidents and threats raise an urgent need for privacy to be deeply integrated
into the development, testing and maintenance of software applications.

Although security and privacy are often discussed together, they are _not_ the
same (Bambauer, 2013). Security often refers to protection against the
unauthorised access to software applications and the data they collect and
store. On the other hand, privacy relates to protection of the individual
rights to their personally identifiable information in terms of how those
personal data are collected, used, protected, transferred, altered, disclosed
and destroyed (Scholz, 2015; ISO/IEC, 2011; Norton, 2021; Data Privacy
Manager, 2021; SNIA, 2021). For example, security controls are put in place to
ensure that only people with credentials have access a software application in
a hospital. However, if anyone with valid credentials can see patient health
records using this software, then privacy is not protected. This example
demonstrates that security can be achieved without privacy, however security
is an essential component for privacy protection.

Cyberattacks, either in the form of security or privacy attacks, are often
formed by exploiting _vulnerabilities_ or _weaknesses_ 111The two terms are
often interchangeable. Hereby, we will use vulnerabilities to refer to both of
them. found in software systems. For instance, the infamous WannaCry
ransomware attack exploited a vulnerability in Microsoft Windows systems,
while the Heartbleed vulnerability in OpenSSL has made millions of websites
and online platforms across the world vulnerable to cyberattacks. To prevent
similar attacks, efforts have been put into understanding and publicly
disclosing vulnerabilities so that developers can identify and fix them in
their software applications. These efforts have resulted in the widely-known
Common Weakness Enumeration (CWE), and Common Vulnerabilities and Exposures
(CVE) systems (The MITRE Corporation, 2021b, a).

However, there have been very little work (e.g., Antón and Earp, 2004; Yang et
al., 2013; Ma et al., 2013) in identifying privacy vulnerabilities. A system
which specifically records common privacy vulnerabilities does _not_ exist
yet. Thus, software developers often rely on the CWE and CVE systems to learn
about privacy-related weaknesses and vulnerabilities. However, it is not clear
to what extent privacy concerns are covered in those systems, and whether
privacy receives adequate attention (which it deserves). To answer these
questions, we have collected _all_ 922 weaknesses recorded in CWE and 156,537
records registered in CVE to date, filtered out non privacy-related records
and further analysed the shortlisted records that are privacy-related. We have
found only 41 and 157 privacy vulnerabilities in the CWE and CVE systems
respectively. The coverage of privacy-related vulnerabilities in both systems
is very limited, only 4.45% in CWE and 0.1% in CVE. (Contribution 1)

The next questions we aimed to explore are what privacy threats are covered in
those privacy-related vulnerabilities in the CWE/CVE systems and if they are
adequately cover the privacy threats raised in both research and practice. To
answer these questions, we have conducted an explanatory study on the privacy
engineering literature, privacy standards and frameworks (e.g., ISO/IEC
29100), regulations in different jurisdiction, including the European Union
General Data Protection Regulation (EU GDPR), California Consumer Privacy Act
(CCPA), Health Insurance Portability and Accountability Act (HIPAA), Gramm-
Leach Bliley Act (GLBA), the U.S. Privacy Act (USPA) and the Australian
Privacy Act (APA), and reputable industry sources (e.g., OWASP (Open Web
Application Security Project, 2020) and Norton (Stouffer, 2021)). This
explanatory study informed us to develop a taxonomy of common privacy threats
that have been raised in research and practice. The taxonomy is built upon the
existing well-known privacy threats taxonomy (Stallings, 2019). Multiple
raters/coders then examined all 41 and 157 privacy vulnerabilities in the CWE
and CVE systems, and mapped them to this taxonomy. The Cohen’s Kappa
coefficient, used to measure the inter-rater agreement, was obtained at 0.874
and 0.875 for the CWEs and CVEs respectively, an almost perfect agreement,
suggesting the strong reliability of the classification. We found that the
existing privacy weaknesses and vulnerabilities reported in the CWE and CVE
systems cover only 13 out of 24 common privacy vulnerabilities raised in
research and practice. Many important types of privacy weaknesses and
vulnerabilities are _not_ covered such as improper personal data collection,
use and transfer, allowing unauthorised actors to modify personal data,
processing personal data at third parties, and improper handling of user
privacy preferences and consent. (Contribution 2)

These actionable insights led to our proposal of 11 new common privacy
weaknesses to CWE222We chose CWE instead of CVE since CVEs specify unique
vulnerabilities detected in specific software systems and application, while
CWEs are at a more abstract, generic level.. These new CWE entries cover the
areas of privacy threats that have been raised in research and practice but do
not exist in CWE yet. To further confirm the relevance and validity of our
proposal, we extracted real code examples from software repositories that
match with the new CWEs. Our contribution follows the CWE’s true spirit of a
community-developed list, and will enhance the CWE system to serve as a common
language and baseline for identifying, mitigating and preventing not only
security but also privacy weaknesses and vulnerabilities. (Contribution 3)

The remainder of the paper is structured as follows. Section 2 discusses
related existing work in security and privacy vulnerabilities in software
applications. The identification of privacy-related vulnerabilities in CWE and
CVE is presented in Section 3. Section 4 discusses the taxonomy of privacy
threats and how the privacy-related vulnerabilities in CWE and CVE systems
cover those privacy threats. Section 5 presents a new common privacy weakness
proposal. Threats to validity is discussed in Section 6. Finally, we conclude
and discuss future work in Section 7.

##  2 Related Work

Several systems have been established to standardise the reporting process and
structure of common vulnerabilities (e.g., CWE (The MITRE Corporation, 2021b),
CVE (The MITRE Corporation, 2021a) and OWASP (Open Web Application Security
Project, 2020)). However, identifying the root causes of the reported
vulnerabilities is still a time-consuming and expertise-required process
(Gonzalez et al., 2019). Recent work employed information retrieval, data
mining, natural language processing, machine learning and deep learning
techniques to characterize vulnerabilities reported in CVE and CWE systems (Li
et al., 2017; Gonzalez et al., 2019; Liu et al., 2020). However, privacy
vulnerabilities were not addressed in those previous work.

Substantial research have been done for detecting software vulnerabilities in
software systems (Wang et al., 2010; Liu et al., 2012; Chernis and Verma,
2018; Lin et al., 2020; Chakraborty et al., 2021; Hanif et al., 2021). Early
vulnerability discovery approaches evolved from static analysis, fuzzing,
penetration testing and Vulnerability Discovery Models (VDMs) (Wang et al.,
2010; Liu et al., 2012). Later approaches applied more advanced techniques
such as machine learning, deep learning and neural networks to improve the
accuracy of vulnerability detection (Chernis and Verma, 2018; Lin et al.,
2020; Chakraborty et al., 2021; Dam et al., 2021; Hanif et al., 2021). These
approaches heavily focus on detecting security vulnerabilities in software
systems, while detection of privacy vulnerabilities were overlooked.

Recent work have also studied and used the CWE and CVE systems. For example,
the work in (Bhandari et al., 2021) collected CVE records with their
associated CWEs and code commits. The collected information was then analysed
to produce insightful metadata such as concerned programming language and
code-related metrics. This work can be applied in multiple applications
related to software maintenance such as automated vulnerability detection and
classification, vulnerability fixing patches analysis and program repair.
Galhardo et al. (2020) proposed a formulation to calculate the most dangerous
software errors in CWE. They used this formulation to identify the top 20 most
significant CWE records in 2019. Again, these prior work only focus on
security vulnerabilities.

Limited work has been done in identifying privacy vulnerabilities in software
systems (Antón and Earp, 2004; Ma et al., 2013; Yang et al., 2013)). Antón and
Earp (2004) proposed a taxonomy of privacy goals based on Internet privacy
policies. The study employed a content analysis through goal mining process to
extract privacy goals from 25 privacy policies in e-commerce industries. The
process consists of three steps: goals identification, classification and
refinement. These identified privacy goals were classified into privacy
protection and privacy vulnerability goals. The privacy vulnerability goal
addresses a set of information processing that may violate consumer privacy
(e.g., information monitoring, collection and transfer).

Yang et al. (2013) introduced a framework to detect privacy leakage in mobile
applications. This study identified several common privacy vulnerabilities in
Android such as unintended sensitive data transmission and local logging. Ma
et al. (2013) discussed a privacy vulnerability in mobile sensing networks
which collect mobility traces of people and vehicles (e.g., traffic
monitoring). Although these networks receive anonymous data, it was proven in
the study that these data can identify victims. These studies have confirmed
the occurrence of privacy vulnerabilities in multiple types of software
systems (e.g., web/mobile applications and sensing networks). However, most of
the existing studies only focused on security concerns when investigating
software vulnerabilities, thus overlooked privacy-related concerns in many
contexts.

A large number of existing studies have investigated methods to preserve and
protect privacy in software development. Several studies have proposed
approaches to derive privacy requirements from organisational goals
(Kalloniatis et al., 2008), data protection and privacy regulations (Breaux
and Antón, 2008; Mihaylov et al., 2016; Ayala-Rivera and Pasquale, 2018; Guo
et al., 2020) or privacy policies (Omoronyia et al., 2012; Massey et al.,
2013) to ensure that software systems comply with those restrictions and/or
constraints. Tschersich et al. (2011) and Yang et al. (2016) presented
frameworks for designing privacy-preserving architecture in software
development. Recent work have examined how organisations or service providers
have compiled with individual rights in data protection regulations. For
example, Kröger et al. (2020) studied on how iOS and Android app vendors
respond when users request to access their personal data. This request is a
mechanism to execute the right of access, one of the individual rights covered
in GDPR and other privacy regulations. The study found that only half of the
app vendors responded to the user access requests. This violates the right of
access as the users must be able to request and get responses to access their
personal data.

A number of studies have proposed taxonomies of privacy threats (e.g., Solove,
2006; Kotz, 2011; Alsamani and Lahza, 2018). Kotz (2011) developed a taxonomy
of privacy threats for _mhealth_ systems (mobile computing for personal health
monitoring). It emphasises on identity, access and disclosure threats of
patients and their personal health information. The threats in this taxonomy
can be caused by patients, internal parties and external parties. Alsamani and
Lahza (2018) proposed a taxonomy of security and privacy threats for IoT. The
taxonomy identifies the security and privacy threats posed by IoT objects
(e.g., sensors and cameras). It adopted four groups of threats developed by
Solove (2006) to analyse privacy concerns of IoT objects. These two studies
cover limited areas of existing software applications. One of the most well-
known privacy threat taxonomies was proposed by Solove (2006). This taxonomy
covers the harmful activities that can violate privacy of individuals.
However, the taxonomy does not focus on privacy threats in software systems.
Thus, we have adapted and extended this taxonomy to address privacy threats in
software engineering.

##  3 Identifying privacy-related vulnerabilities in CWE and CVE

Common Weakness Enumeration (CWE) and Common Vulnerabilities and Exposures
(CVE) are two well-known systems for publicly known weaknesses and
vulnerabilities in software and hardware. A vulnerability is defined as “a
weakness in an information system, system security procedures, internal
controls, or implementation that could be exploited or triggered by a threat
source” (Guitierrez et al., 2013).

The CWE system identifies common categories of flaws, bugs and other errors
found in software and hardware implementation, code, design or architecture
that could be vulnerable to attacks (The MITRE Corporation, 2021b). The CWE
system has three views: i) by research concepts, ii) by software development
and iii) by hardware design. Our study focuses on the research concepts and
software development views as they are related to software applications. A CWE
record consists of a description, relationships to other CWE records,
likelihood of exploit, demonstrative examples, observed examples, common
consequences, mitigations and other relevant attributes. The interested
parties can use this information to identify a weakness in their software
systems and applications. For example,
CWE-359333https://cwe.mitre.org/data/definitions/359.html describes a weakness
that exposes private personal information to an unauthorised actor (see Figure
1). This CWE record also provides demonstrative examples, one of which is a
code fragment that expose a user’s location. The CWE record also specifies
potential mitigations such as identifying and consulting with all relevant
privacy regulations regarding the processing of users’ location.

Figure 1: A screenshot highlights privacy-related information in CWE-359.
Figure 2: Privacy vulnerabilities identification process

CVE is a catalogue of cybersecurity vulnerabilities that may exist in software
products, applications and open libraries (e.g. Skype, Mozilla Firefox and
Android). These vulnerabilities and their details are reported by
organisations that have partnered with the CVE program. Once reported, the CVE
Numbering Authorities (CNAs) will be responsible for determining each
vulnerability record and assigning a unique identifier (i.e. CVE ID) to that
record. Each record describes the details of a vulnerability and specifies
affected version of a software. The CVE records are then publicly published
and accessible to interested parties (e.g. software developers, organisations
and researchers). These records specify unique vulnerabilities detected in
specific software systems and application, thus they are more specific
comparing to CWE records. For example,
CVE-2000-1243444https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2000-1243
refers to a privacy leak in version 3.04 of Dansie Shopping Cart in which
sensitive information such as user credentials were sent to an e-mail address
controlled by the product developers.

In the absence of a system which specifically records common privacy
vulnerabilities, software engineers and other interested parties often rely on
the CWE and CVE systems for privacy weaknesses and vulnerabilities (like the
one in Figure 1). However, both CWE and CVE target at cybersecurity, and
although security and privacy are often discussed together, they are not the
same. Security vulnerabilities are often exploited by unauthorised access to
perform malicious actions in software applications. By contrast, privacy
vulnerabilities may lead to violations of the individual rights to their
personally identifiable information in terms of how those personal data are
collected, used, protected, transferred, altered, disclosed and destroyed.
Hence, we have explored to what extent privacy concerns are covered in the CWE
and CVE systems, and whether privacy receives adequate attention which it
deserves.

###  3.1 Approach

The privacy vulnerability identification process (see Figure 2) consists of
the following steps: (i) obtaining the CWE and CVE lists, (ii) determining a
list of keywords and performing a keyword search, (iii) identifying privacy-
related criteria and annotating the CWE and CVE records, and (iv) performing
privacy vulnerability analysis.

To identify the privacy vulnerability in CWE and CVE lists, we first download
the whole CWE records in the research concepts and software development
views555The CWE data is available at https://cwe.mitre.org/data/downloads.html
and CVE records666The CVE data is available at
https://cve.mitre.org/data/downloads/index.html from their websites. In the
CWE list, the research concepts view contains 922 weaknesses. The software
development view contains 418 weaknesses, however all of them are also covered
in the research concepts view. Thus, we examined all the 922 weaknesses in the
CWE to date. The attributes include CWE-ID, name, weakness abstraction,
status, description, extended description, related weaknesses, weakness
ordinalities, applicable platforms, alternate terms, mode of introduction,
exploitation factors, likelihood of exploit, common consequences, detection
methods, potential mitigations, observed examples, functional areas, affected
resources and taxonomy mappings, related attack patterns and notes. The CVE
list contains 156,537 records to date, all of which were examined in our
study. We examined all of the following attributes in each CVE record: name
(i.e. CVE-ID), status, description, references, phases, votes and comments.

After obtaining both lists, we then performed keyword searches to filter out
the CWE and CVE records that do not have privacy related keywords. We used a
search function in Microsoft Excel to examine the keywords in those records. A
set of keywords consists of 37 words categorised into 4 groups as follows:

  * •

Group 1: general terms that relate to weaknesses and vulnerabilities in
privacy. The terms include privacy, violation, leak and leakage (4 keywords).
The term _privacy_ generally appears in the CWE and CVE that reported privacy
weaknesses and vulnerabilities. On top of that, we also include the terms
_violation, leak and leakage_ as they are alternatively used to express
concerns in the context of privacy (see
CWE-359777https://cwe.mitre.org/data/definitions/359.html for more details).

  * •

Group 2: terms used to refer to personal data or personally identifiable data.
They could be sometimes used interchangeably. The keywords in this group
include personal information, personal data, sensitive information, sensitive
data, private information, private personal information, personally
identifiable information, PII, protected health information, PHI, health
information and health data (12 keywords). These terms are used in regulations
(e.g., GDPR and HIPAA), standards (e.g., ISO/IEC 29100) and industry sources
(e.g., Norton, CWE and CVE).

  * •

Group 3: terms that relate to relevant privacy and data protection
regulations/standards/frameworks. In this group, we select a set of specific
well-known and widely-adopted data protection regulations and privacy
frameworks, which include General Data Protection Regulation, GDPR, California
Consumer Privacy Act, CCPA, Health Insurance Portability and Accountability
Act, HIPAA, Gramm-Leach Bliley Act, GLBA, Safe Harbor Privacy Framework,
ISO/IEC 29100 (10 keywords). In addition, we also include the general terms to
ensure that we cover unseen regulations and frameworks. The terms consist of
regulation, data protection, privacy act, privacy framework, privacy standard
(5 keywords).

  * •

Group 4: terms that are frequently seen in the privacy policies and literature
when discussing personal data protection and user privacy (e.g., Massey et
al., 2013, GDPR, CCPA and ISO/IEC 29100). These include right(s), consent, opt
in/opt-in, opt out/opt-out, preference and breach (6 keywords).

Table 1: Individual rights on privacy Rights |  Description |  Relevant data protection regulations/privacy acts
---|---|---
Right to be informed |  Individuals must be informed about the collection, use and processing of their personal data. For example, the software applications must inform the purposes of personal data collection and processing. In case the personal data is processed by other parties, the individuals must be informed to whom their personal data is disclosed/transferred. |  GDPR, CCPA, HIPAA, GLBA, APA
Right of access |  Individuals must be able to request to access and receive a copy of their personal data. |  GDPR, HIPAA, USPA, APA
Right to rectification |  Individuals must be able to request to correct, complete and/or make changes to their personal data. |  GDPR, HIPAA, USPA, APA
Right to erasure |  Individuals must be able to request to erase their personal data. |  GDPR, CCPA
Right to restrict of processing |  Individuals must be able to request to restrict of personal data processing. |  GDPR, HIPAA
Right to data portability |  Individuals must be able to request obtain, reuse, move, copy or transfer their personal data across different services for their own purposes. |  GDPR
Right to object |  Individuals must be able to object to the processing of their personal data in certain circumstances (e.g. direct marketing). |  GDPR, APA
Rights in relation to automated decision making and profiling |  Organisations must obtain consent from individuals when the software may process their personal data based on automated decision making and profiling. |  GDPR
Right to opt-out of sale |  Individuals must be able to request to opt-out if their personal data is sold by organisations. |  CCPA
Right to non-discrimination |  Individuals, who exercise their individual rights with respect to particular regulations, must be provided with the same quality and price of goods and services by businesses. |  CCPA
Right to request confidential communications |  Individuals must be able to request to change means or location for receiving communication of health information. |  HIPAA
Right to be protected against unwarranted invasion |  Personal data must be protected from invasion resulting from the collection, use, disclosure, maintenance and other processing. |  GDPR, USPA
Right to make a complaint |  Individuals must be able to lodge a complaint to relevant organisations or supervisory authority if their personal data is mishandled. |  GDPR, APA
Right to opt-out of sharing |  Individuals must be able to opt-out from sharing their personal data. |  GLBA
Right to not identifying yourself |  Individuals must have an option not identifying themselves in certain circumstances (e.g. using pseudonym). |  APA

We acquired 185 CWE and 1,088 CVE records that contain at least one of the
specified keywords. Next, we manually examined each of those records to
identify privacy vulnerabilities. A vulnerability is considered as privacy-
related if it satisfies one of the following criteria:

  1. 1.

A weakness or vulnerability involves with any processing of personal data
including, but not limited to, collection, use, storage, transfer, alteration,
erasure and disclosure. As personal data can be used on its own or in
conjunction with other information to identify or trace an individual (e.g.
name, identification number, social security number, user credentials, health
records, email addresses), the processing of personal data may affect
information and user privacy (Scholz, 2015; ISO/IEC, 2011; Data Privacy
Manager, 2021).

  2. 2.

A weakness or vulnerability which may lead to violations of the individual
rights (see Table 1). To extract the individual rights, we first selected a
range of well-established data protection and privacy regulations in different
domains such as governments, businesses, healthcare and finance (e.g., EU
GDPR, California Consumer Privacy Act (CCPA), Health Insurance Portability and
Accountability Act (HIPAA), Gramm-Leach Bliley Act (GLBA), the U.S. Privacy
Act (USPA) and the Australian Privacy Act (APA). These regulations have been
widely enacted in country- and regional-level, hence they are well respected
by organisations worldwide. The regulations consist of articles or sections
that explain provisions and details of legal constraints. In each regulation,
we went through each article to look for the individual rights of data
subjects/patients/consumers (i.e., the rights that data subjects must have in
order to control the processing of their personal data). Once we found the
individual right, we added and summarised its description into our list.
Several individual rights are fundamental for personal data and privacy
protection (e.g., right to be informed and right of access), hence they have
been found in several regulations.

We went through the shortlisted 185 CWE and 1,088 CVE records to determine the
vulnerabilities that meet the above criteria. In addition, we have found that
the National Vulnerability Database (NVD) had done some mapping between CVEs
and CWEs. Hence, once we have identified privacy-related CVEs, we used this
mapping and applied the above criteria to identify additional privacy-related
CWE records.

We also note that the CWE weaknesses are organised hierarchically where each
CWE weakness can have other CWE weaknesses as its parents or children. We
investigated whether privacy properties are inherited through this structure.
We discuss here CWE-200888https://cwe.mitre.org/data/definitions/200.html as
an example to demonstrate this case. CWE-200 describes a weakness that exposes
sensitive information to an unauthorised actor. This weakness has one parent
and eleven children. CWE-200 itself is classified as privacy-related, however
its parent and seven of its children are not privacy-related. Similarly,
CWE-201, one of the CWE-200 children, is privacy-related, but its child
CWE-598 is not privacy-related. We also further explored the rest of CWE-200
children, and found that privacy properties are not inherited through the
hierarchical structure. Thus, we did not take a hierarchical structure into
account when examining whether a weakness is privacy-related.

###  3.2 Results

We identified 41 and 157 privacy vulnerabilities in the CWE and CVE records
respectively. The first 28 privacy-related CWE records were found after the
keyword search and manual examination steps. The additional 13 privacy-related
CWE records were later added after being identified by the privacy-related CVE
records. They cover a wide range of privacy concerns in software applications
such as missing personal data protection, improper access control,
insufficient credentials protection, personal data exposures, unintentional
errors made by software developers and personal data attacks from external
attackers. These weaknesses and vulnerabilities are not only related to
security, but also affect privacy of individuals.

We discuss here a few examples of CWE and CVE records that were classified as
privacy vulnerabilities or weaknesses and refer the reader to (Sangaroonsilp
et al., 2021) for a full list of them.
CWE-359999https://cwe.mitre.org/data/definitions/359.html refers to exposure
of private personal information to an unauthorized actor. Private personal
information here includes social security numbers, geographical location,
financial data and health records. In addition, this CWE also mentions
relevant data protection regulations and privacy acts such as GDPR and CCPA.

Another example is CVE-2020-13702101010https://cve.mitre.org/cgi-
bin/cvename.cgi?name=CVE-2020-13702 which refers to the rolling proximity
identifier used in the Apple/Google exposure notification API beta through
2020-05-29. This vulnerability enables attackers to evade Bluetooth Smart
Privacy due to a secondary temporary UID through tracking individual device
movement using a Bluetooth LE discovery mechanism. This is a privacy
vulnerability since it concerns user’s location and relates to the processing
of user’s location. CVE-2021-21301111111https://cve.mitre.org/cgi-
bin/cvename.cgi?name=CVE-2021-21301 describes a vulnerability in Wire for iOS
(iPhone and iPad) before version 3.75. The application enables camera for
video capturing, although the users have disabled this service. This
vulnerability seriously violates user privacy as the users are not aware of
their camera being enabled, and the camera may capture and expose the users
and their environment.

CVE-2019-16522121212https://cve.mitre.org/cgi-
bin/cvename.cgi?name=CVE-2019-16522 refers to the cross-site scripting (XSS)
attacks occurred in WordPress. The eu-cookie-law plugin through 3.0.6 for
WordPress is vulnerable to cross-site scripting (XSS) attacks. This
vulnerability affects the cookie consent message which leads to the unclear
information provided to the users, including details regarding personal data
processing. CVE-2011-1717131313https://cve.mitre.org/cgi-
bin/cvename.cgi?name=CVE-2011-1717 describes a vulnerability occurred in Skype
for Android. The application stores sensitive user data without encryption in
the sqlite3 databases (e.g. user IDs, phone number and date of birth). This
CVE concerns the user data that can be used to identify an individual such as
phone number and date of birth. It also involves with the lack of personal
data protection stored in the database.

There are many cases where the reported vulnerabilities are _not_ specifically
privacy related. For example,
CWE-78141414https://cwe.mitre.org/data/definitions/78.html enables the
attacker to execute arbitrary commands on the operating systems, leading to
unauthorised access to operating systems. However, this is not specifically a
privacy vulnerability since it does not involve personal data or any
processing of personal data. Similarly,
CVE-2021-21466151515https://cve.mitre.org/cgi-
bin/cvename.cgi?name=CVE-2021-21466 describes a vulnerability that allows an
attacker to inject malicious code into certain versions of SAP Business
Warehouse and SAP BW/4HANA. The attacker could create a malicious ABAP report
which could be used to access data and disrupt the system functionalities.
This vulnerability can lead to Denial of Service. This is again a security
vulnerability rather than a privacy one since it does not directly related to
violations of the individual rights to their personally identifiable
information.

The coverage of privacy-related vulnerabilities in both CWE and CVE records is
quite limited: 4.45% in the CWE system and 0.1% in the CVE system.

##  4 Common privacy threats in software applications

This section investigates how those privacy-related vulnerabilities identified
in CWE and CVE (see Section 3) address common privacy threats in software
applications. We first discuss a taxonomy of common privacy threats that we
have developed based on an explanatory study of the literature. We then report
the threats that have not been covered by the existing privacy-related
vulnerabilities in CWE/CVE.

###  4.1 Explanatory study

To identify common privacy threats in practice, we performed an explanatory
study on the literature of the following three groups: existing privacy
software engineering research, well-established data protection regulations
and privacy frameworks, and additional reputable industry resources. The
details of this process are described below.

####  4.1.1 Privacy engineering research

Privacy engineering has attracted an emerging area of research in software
engineering (Gürses and Del Álamo, 2016). Our study in this area followed a
systematic literature review process proposed by Kitchenham (2007) to retrieve
relevant papers and conduct a literature survey. This process consists of
three phases: planning, papers selection and extracting & reporting (see
Figure 3). In the planning phase, we defined our study goal which is to
identify privacy threats addressed in the privacy engineering research. Our
research question is “What are privacy threats caused by software
developers/data processors/service providers/external parties that were
addressed in privacy engineering research?”. To achieve the study goal, we
developed a review protocol to determine the scope of our study. The protocol
consists of four tasks: (i) determining a search keyword and time scope, (ii)
selecting software engineering publication venues, (iii) determining exclusion
and inclusion criteria, and (iv) determining a set of questions to identify
the privacy threats in the papers. In the papers selection phase, we conducted
a search process and applied inclusion and exclusion to the retrieved papers.
Finally, we analysed and identified privacy threats in the included papers and
reported the results.

Figure 3: An overview process of privacy engineering research review.

##### Search keyword and time scope.

We used a search keyword _“privacy”_ to search in title, abstract and keywords
fields of papers in the selected publication venues. Only one search keyword
was used as we already performed a search in the specific software engineering
publication venues, thus we aimed to get all the papers that address privacy.
In addition, the papers from these venues are peer-reviewed by experienced
researchers in privacy and software engineering area, hence their significant
contributions and quality are well received by the research community. We also
determined to search for papers that were published in the past 20 years (2001
- 2020). This task was automatically done by the search functions in the
academic databases (i.e., IEEE Xplore, ACM Digital Library, ScienceDirect,
SpringerLink and Scopus). We have found 1,434 papers related to privacy during
the period of 2001 to 2020 (see Figure 4).

Figure 4: Number of papers included in each step.

##### Software engineering publication venues.

As privacy has also been widely addressed in other fields of study (e.g., law
and social sciences), we scope down the search to get a reasonable number of
papers in software engineering. We selected seventeen highly recognised
software engineering publication venues consisting of 7 conferences, 6
journals, 2 symposiums and 2 workshops. These venues focus on various
disciplines in software engineering, which make them proper candidates for
representing privacy in multiple areas. Many of these venues were included in
the existing papers that conducted systematic literature review in software
engineering research (e.g., Ebrahimi et al., 2019; Perera et al., 2020;
Bertolino et al., 2018). The full list of venues, along with the number of
papers found in each, is shown in Table 2.

##### Exclusion and inclusion criteria.

We manually performed an inclusion and exclusion task to ensure that the
papers retrieved from the automated search process satisfy our study scope. We
initially determined a set of inclusion and exclusion criteria to filter out
the papers that are irrelevant to our goal. The exclusion criteria (EC) are as
follows:

  * •

EC1: Papers that contain insufficient, incomplete or irrelevant information.
As we automatically exported the records of studies from their databases, we
found a number of records that contain journal/conference/workshop
introduction, information from program chairs, guest editorials, summaries of
keynotes and prefaces, which do not satisfy our study goal. Hence, these
records are excluded from our study.

  * •

EC2: Papers that are duplicate. We consider from the title of the papers. If
the papers have the exact same titles, other versions except the most recent
version of those studies are excluded.

  * •

EC3: Papers that are secondary or tertiary studies (e.g., existing systematic
literature review papers and systematic mapping studies) and posters are
excluded.

The following inclusion criteria (IC) are applied in the screening step:

  * •

IC1: The primary contribution of the papers is related to privacy.

  * •

IC2: The research contribution of the papers is related to software
development.

After applying the exclusion criteria, we excluded 209 out of 1,434 papers.
1,225 papers were passed to the next step. We then applied the inclusion
criteria to the abstracts of the papers (see Figure 5). If the papers satisfy
both inclusion criteria, they are included in our study. If the papers do not
satisfy one or both criteria, we exclude them from our study. Finally, 417
papers were included in our study161616The bibliographic data of those papers
are available in (Sangaroonsilp et al., 2021).

Figure 5: Inclusion criteria.

##### Privacy threats identification.

To identify privacy threats, we examine the research studies reported in those
papers, and shortlist the ones that focus specifically on privacy
vulnerabilities and attacks from this list. The papers were analysed by asking
the following questions:

  * •

What is a cause of privacy-related problem in software that has been raised in
the paper? This cause must involve with personal data and is harmful to data
subjects. This question aims to list all the privacy threats addressed in the
paper.

  * •

Is the identified cause caused by software developers, data
controllers/processors, organisations or external parties? This question helps
us focus on the privacy threats that are not caused by users. There are papers
that investigate user perceptions towards different privacy perspectives
(e.g., user perceptions of online behavioural advertising/smart home devices
and user confidence in using smartphones). We do not include the privacy
threats that are caused by users in this study as it is out of our scope.

We individually examined the selected papers and added privacy threats
identified in the papers into a threat list. During the investigation, if we
find a threat that is already included in our list, we annotate the paper with
the existing threat and proceed to the next paper, otherwise we add a new
threat into the list. Finally, we have found 179 papers that address privacy
vulnerabilities and attacks in software development (see Table 3). From these
studies, we have identified a range of privacy threats in various applications
(e.g., mobile/web applications, mobile sensors, smart home devices, network
communications and privacy policy compliance). These will be discussed in
details in Section 4.2. The rest of the papers either address the threats
caused by users or do not address privacy vulnerabilities or attacks in
software systems.

Table 2: Selected software engineering publication venues. Source title |  Acronym |  Type |  Count |  Included |  Threats found
---|---|---|---|---|---
ACM Transactions on Software Engineering and Methodology |  TOSEM |  Journal |  4 |  1 |  0
Empirical Software Engineering |  EMSE |  Journal |  6 |  3 |  0
IEEE Symposium on Security and Privacy |  SP |  Symposium |  330 |  64 |  9
IEEE Symposium on Security and Privacy Workshops |  SPW |  Workshop |  134 |  49 |  5
IEEE Transactions on Software Engineering |  TSE |  Journal |  22 |  7 |  1
Information and Software Technology |  IST |  Journal |  16 |  5 |  0
International Conference on Automated Software Engineering |  ASE |  Conference |  20 |  8 |  3
International Conference on Availability, Reliability and Security |  ARES |  Conference |  186 |  113 |  93
International Conference on Foundations of Software Engineering |  FSE |  Conference |  13 |  5 |  0
International Conference on Mining Software Repositories |  MSR |  Conference |  8 |  2 |  0
International Conference on Software Engineering |  ICSE |  Conference |  90 |  22 |  4
International Requirements Engineering Conference |  RE |  Conference |  68 |  25 |  2
International Symposium on Empirical Software Engineering and Measurement |  ESEM |  Conference |  5 |  1 |  0
International Workshop on Evolving Security and Privacy Requirements Engineering |  ESPRE |  Workshop |  32 |  7 |  0
Requirements Engineering |  REJ |  Journal |  19 |  10 |  2
Symposium on Usable Privacy and Security |  SOUPS |  Symposium |  104 |  77 |  47
Systems and Software |  JSS |  Journal |  377 |  18 |  13
Sum |  |  |  1,434 |  417 |  179
Table 3: Number of papers by threat subcategories. Threats |  #papers
---|---
Surveillance |  4
Identification |  10
Insecurity |  74
Secondary use |  13
Exclusion |  5
Breach of confidentiality |  3
Intrusions |  18
Compliance |  31
Multiple threats |  21
Not a vulnerability/an attack |  238
Total |  417

####  4.1.2 Privacy regulations and frameworks

We have studied 7 well-established data protection and privacy regulations and
frameworks (i.e., GDPR (Office Journal of the European Union, 2016), CCPA
(California State Legislature, 2018), HIPAA (Department of Health and Human
Services, 2013), GLBA (United State Congress, 1999), USPA (U.S. Department of
Justice, 1974), APA (Federal Register of Legislation, 2017) and ISO/IEC 29100
(ISO/IEC, 2011)). We have found that these regulations and frameworks focus
mainly on the privacy threats to the rights that allow the individuals to
control their personal data processing. They also raise a number of privacy
threats relating individuals being informed about the processing (e.g.,
collection, use and transfer) of their personal data in software applications.

####  4.1.3 Additional industry sources

We have also included a range of reputable industry sources (e.g., OWASP and
Norton) on this topic. These sources (e.g., Open Web Application Security
Project, 2020; Norton, 2021) cover the recent trends of privacy risks and
attacks. These sources are developed by well known communities and companies.
For example, OWASP identified 20 privacy risks in web applications (The OWASP
Foundation, 2015) such as leaking personal data or vulnerabilities that
involves stealing personal data through common cyberattacks such as cross-site
scripting and broken session management. Norton also identified a number of
social engineering cyberattacks that are related to privacy such as phishing
and keystroke logging attacks (Stouffer, 2021).

In the next section, we will discuss in details the common privacy threats
that we have found in our explanatory study of the literature.

###  4.2 A taxonomy of common privacy threats

We have built a taxonomy of common privacy threats upon the well-established
privacy threats taxonomy described in Stallings (2019). This taxonomy was
originally proposed by Solove (2006) which covers privacy of individuals
through the conception of laws. This taxonomy cover cases which are mainly
caused by physical activities which violate privacy of individuals (e.g., a
newspaper reports the name of a rape victim, or a company sells its members’
personal information despite promising not to do so) (Solove, 2006). This
taxonomy has been considered as one of the comprehensive taxonomies that
emphasise on privacy threats. Later, Stallings (2019) adapted Solove’s
taxonomy to the context of information systems. However, Stallings’s taxonomy
covers privacy threats in a generic and rather abstract level. Thus, we
tailored this taxonomy and refined it into a more concrete version that
addresses privacy threats in software engineering. The taxonomy consists of
four categories of privacy threats: information collection, information
processing, information dissemination and invasions (see Figure 6). Each of
these groups contains different subcategories covering relevant harmful
privacy threats. The yellow boxes in Figure 6 represent the categories and
subcategories included in the original taxonomy.

After identifying the privacy threats in the explanatory study, we classified
those threats into two groups: vulnerabilities and compliance. Privacy
vulnerabilities refer to technical issues, flaws or errors that lead to
privacy exploits in software applications and platforms. Compliance addresses
the privacy threats which are related to the individual rights and the
governance of personal data. We then expanded the Stallings’s taxonomy by
mapping the privacy threats in the vulnerabilities group into their relevant
subcategories in the original taxonomy (i.e., blue boxes in Figure 6).
However, the privacy threats in the compliance group have not been addressed
in any groups in the original taxonomy. Thus, we propose the compliance group
as an extension to the original taxonomy (see Figure 7). This group is
discussed in details in Section 4.2.5. The full taxonomy is available at
(Sangaroonsilp et al., 2021).

In our taxonomy, we classify twenty-four privacy vulnerabilities into seven
subcategories. In the classification process, we mapped a privacy threat in to
the most relevant subcategory based on the description of each subcategory
explained in Stallings (2019). We describe the categories, subcategories and
their relevant privacy threats below. We also include the samples of sources
where the privacy threats were raised or discussed.

Figure 6: Common privacy vulnerabilities in software applications

####  4.2.1 Information collection.

(Sources: Antón and Earp, 2004; Deng et al., 2011; Jana et al., 2013; Hasan et
al., 2020; Lebeck et al., 2018; De and Metayer, 2016; Drosatos et al., 2014;
Office Journal of the European Union, 2016; ISO/IEC, 2011; Department of
Health and Human Services, 2013; United State Congress, 1999; U.S. Department
of Justice, 1974; Federal Register of Legislation, 2017; The OWASP Foundation,
2015) This category concerns privacy threats that occur when collecting
personal data from individuals. The _surveillance_ subgroup covers
vulnerabilities existing in the way software collects personal data such as
watching individuals through cameras or CCTVs, listening to individuals, or
recording individuals’ activities. The privacy threat related to this subgroup
is caused when personal data is collected without consent or permissions from
individuals (e.g., via mobile sensors).

####  4.2.2 Information processing.

(Sources: Antón and Earp, 2004; Deng et al., 2011; Zhang et al., 2020;
Calandrino et al., 2011; Jana et al., 2013; Hasan et al., 2020; Omoronyia et
al., 2012; Some, 2019; Figueiredo et al., 2017; Venkatadri et al., 2018; Horbe
and Hotzendorfer, 2015; De and Metayer, 2016; Fisk et al., 2015; Barman et
al., 2015; Lin, 2016; Ahamed et al., 2009; Sicari et al., 2012; Yang et al.,
2016; Erola et al., 2011; Bilogrevic et al., 2011; Tuan Anh et al., 2014;
Siewe and Yang, 2016; Calciati et al., 2018; Drosatos et al., 2014; Zhang et
al., 2005; Castiglione et al., 2010; Tschersich et al., 2011; Rafiq et al.,
2017; Iyilade and Vassileva, 2014; Office Journal of the European Union, 2016;
ISO/IEC, 2011; Department of Health and Human Services, 2013; United State
Congress, 1999; Federal Register of Legislation, 2017; U.S. Department of
Justice, 1974; The OWASP Foundation, 2015) This category covers
vulnerabilities related to the use, storage, and manipulation of the collected
personal data. It contains four subgroups: identification, insecurity,
secondary use and exclusion. The _identification_ subcategory addresses a
privacy threat that aggregates personal data from various sources and use it
to identify individuals.

The vulnerabilities in _insecurity_ subcategory are caused by improper
protection and handling of personal data. There are multiple forms of this
vulnerability type such as lacking mechanisms to protect personal data,
allowing unauthorised actors (e.g., internal/external staff and attackers) to
access or modify personal data, or track individual users (e.g., user’s
locations, visiting history, etc.). Transferring personal data between
software applications without protection is also another form of this
vulnerability. Personal data is sometimes required to be processed at third
parties. This poses a privacy vulnerability since the third parties may apply
lower levels of personal data protection than the source does. In addition,
this subcategory refers to the vulnerabilities where mechanisms to protect
personal data are in place but they are not appropriate. Different types of
personal data require different methods/techniques and levels of protection.
Thus, appropriate protection mechanisms should be used to protect personal
data against potential risks. This vulnerability type is often refined into
improper techniques/methods and insufficient levels of protection (e.g., weak
encryption).

The _secondary use_ subcategory refers to the use of personal data for other
purposes without consent or not following user privacy preferences. Personal
data used or transferred without user permissions, or to an unintended
destination is a privacy vulnerability. The personal data used without
following user privacy preferences can also cause a privacy
vulnerability171717For example, CVE-2005-2512 reports a vulnerability which
could result in a privacy leak in mail.app in Mac OS 10.4.2 and earlier in
which remote images are loaded against the user’s preferences when an HTML
message is printed or forwarded.

The _exclusion_ subcategory refers to the failure to provide individuals with
notice and input for managing their personal data. These vulnerabilities
relate to consent which allows the users to express their agreement on the use
of their personal data in software applications. We note that consent handling
may seem to be a part of compliance, however this subcategory focuses on the
malfunctions that cause vulnerabilities in consent handling. User consent is
required when the processing of personal data is not required by laws. Users
should also be notified when the conditions on the consent are changed. In
addition, users should be allowed to modify or withdraw their consent. One
example of this privacy vulnerability is in mobile applications where users
allow a specific permission to an app, however the permission is overridden in
other apps without their consent (Calciati et al., 2018; Zhang et al., 2020).
Apart from user consent, user privacy preferences are also important. Privacy
preferences enable users to personalise how they prefer their personal data to
be managed. This privacy vulnerability type can be in two forms: privacy
preferences not provided to the users, and users not be able to modify their
privacy preferences.

####  4.2.3 Information dissemination.

(Sources: Zhang et al., 2020; Calciati et al., 2018; Jana et al., 2013; Lucia
et al., 2012; The OWASP Foundation, 2015) This category refers to the privacy
threats that lead to the revelation of personal data to public. The _breach of
confidentiality_ subcategory covers the vulnerabilities that cause personal
data leakage by those who are responsible for personal data processing (e.g.,
software developers, data analysts, etc.).

####  4.2.4 Invasions.

(Sources: Deng et al., 2011; Zeng et al., 2019; Chalhoub et al., 2020; The
OWASP Foundation, 2015; Stouffer, 2021) This category addresses attacks that
directly affect individuals. The _intrusions_ subcategory covers
vulnerabilities exploited by common privacy attacks in software applications.
There are typically four attacks: web applications, phishing, keystroke
logging and smart home devices. The attacks in web applications include, but
not limited to: injection, broken authentication/authorisation/session
management, security misconfigurations, cross-site request forgery (CSRF),
insecure direct object reference (IDOR), using components with known
vulnerabilities and invalidated redirects and forwards. Phishing attacks can
be used to steal personal data by sending malicious links to the users through
software applications, text messages and emails. These links bring the users
to fake pages or programs that ask for their confidential data. The users are
not aware of these malicious activities and end up giving away their personal
data that can identify themselves (e.g. address, personal identifiers and
financial data) to the attackers. Keystroke logging allows the attackers to
track and record the keys that users have input on their keyboards. The
attackers can then capture and steal personal data entered by users. Smart
home devices are also vulnerable to external attacks. The attackers can
intercept smart home systems to gain access to personal data.

####  4.2.5 Compliance

We used a bottom-up approach to construct a taxonomy for the compliance
category. The privacy threats that address the same concerns are grouped into
the same subcategory. We classify privacy compliance into the following three
subcategories. The first is _not complying with individual rights_ (sources:
Office Journal of the European Union, 2016; California State Legislature,
2018; Department of Health and Human Services, 2013; United State Congress,
1999; U.S. Department of Justice, 1974; Federal Register of Legislation, 2017;
ISO/IEC, 2011; Deng et al., 2011; Omoronyia et al., 2013; Bhatia and Breaux,
2018; Bhatia et al., 2016; Omoronyia et al., 2012; De and Metayer, 2016; Yu et
al., 2021; Mihaylov et al., 2016; The OWASP Foundation, 2015). There are 16
individual rights identified in our study (see Table 1).

The second is _not providing contact details of a responsible person_
(sources: Office Journal of the European Union, 2016; ISO/IEC, 2011). This
subcategory cover cases where software applications do not provide the contact
details of a responsible person or a representative who controls the
processing of their personal data. This is a privacy threat as the users do
not know whom to be contacted regarding their personal data processing. The
third is _improper personal data breach response_ (sources: Office Journal of
the European Union, 2016; ISO/IEC, 2011; Department of Health and Human
Services, 2013; The OWASP Foundation, 2015). When a breach occurs, a
responsible person must notify two key stakeholders: concerned individuals and
a supervisory authority. The privacy threat is raised if the responsible
person does not communicate the breach incident to the concerned users whose
personal data is leaked to the public and the supervisory authority who
monitors the personal data processing under the individual rights.

Figure 7: Common privacy compliance in software applications

Our taxonomy covers the existing vulnerabilities occurred in different
activities in software systems. These privacy vulnerabilities have been raised
in real world software applications and software development processes. The
taxonomy was also developed based on an existing comprehensive privacy threats
taxonomy, and validated with the common vulnerabilities reported in CWE and
CVE, as we will discuss in details in the next section.

###  4.3 Privacy threats covered in CWE/CVE

The next step in our study was to investigate how 41 privacy vulnerabilities
in CWE and 157 in CVE (see Section 3) address the taxonomy of common privacy
threats in Section 4.2. This process consists of the following steps. The
first two co-authors (hereafter referred to as the coders) _independently_
analysed each of the 41 privacy vulnerabilities in CWE, and classified it into
the most relevant privacy threat in the taxonomy. Of the 157 privacy
vulnerabilities in CVE, 112 were assigned to a specific CWE by the NVD. These
112 CVEs are automatically classified into the same privacy threat as their
associated CWE. The remaining 45 CVEs, which were not assigned a specific
CWE181818NVD used two special placeholder names for these: NVD-CWE-noinfo and
NVD-CWE-other., were manually classified by our coders. To facilitate the
classification step, each coder was provided with a Google Sheet form pre-
filled with the privacy vulnerabilities in CWE and CVE and the privacy threats
in the taxonomy. The privacy threats were prepared as a drop down list. The
coders then selected the vulnerability that is most relevant and best
described those CWE and CVE in their view.

We have also employed the Inter-Rater Reliability (IRR) assessment and
disagreement resolution processes to ensure the reliability of the
classifications. The Cohen’s Kappa coefficient is used to measure the inter-
rater agreement as it is a perfect measure for a multi-class classification
problem with two coders (Hallgren, 2012). Once the coders had completed the
classifications, the inter-rater agreement was computed. The Kappa agreement
values between the coders are 0.874 and 0.875 in CWE and CVE classifications
respectively, which both achieve _almost perfect agreement_ level (Viera and
Garrett, 2005). A disagreement resolution was conducted to resolve some small
classification conflicts (4 CWEs and 4 CVEs). The coders met, went through
them together, discussed and reclassified those vulnerabilities. Thus, the
final classification reached the maximum agreement between the coders.

####  4.3.1 Results

We have found that all the 41 CWEs and 157 CVEs together cover 13
vulnerabilities in the taxonomy. They are annotated with the corresponding
CWEs and CVEs in Figure 6. For brevity, we do not include the CVE numbers in
Figure 6, but they are provided in full in our replication package
(Sangaroonsilp et al., 2021). Exposing personal data to an unauthorised actor,
insufficient levels of protection, and personal data attacks are the top three
most addressed vulnerabilities in both CWE and CVE. Personal data protection
seems to attract a lot of attentions in CWE/CVE with more than half of the
CWEs (56.1%) and CVEs (59.87%) vulnerabilities reported, most of which (36.59%
in CWEs and 50.32% in CVEs) are related to exposing personal data to an
unauthorised actor. There are 19.51% in CWE and 15.92% in CVE reporting
vulnerabilities regarding personal data attacks.

There are four types of privacy vulnerabilities that are covered by both CWE
and CVE: exposing personal data to an unauthorised actor, insufficient levels
of personal data protection, improper methods/techniques for personal data
protection, and personal data attacks. There are several types of privacy
vulnerabilities that have been reported in CVE, but not in CWE, such as
allowing unauthorised actors to track individuals, not following user privacy
preferences, and not asking for user consent. 8.28% of the CVEs refer to those
types of privacy vulnerabilities, suggesting that those types of
vulnerabilities need to be added into the CWE system.

There are a number of areas that are not covered by the existing privacy
vulnerabilities in CWE and CVE (as highlighted in red outline in Figure 6).
For example, exclusion involves a range of privacy vulnerabilities in failing
to provide users with notice of user consent and input about their privacy
preferences. User consent and privacy preferences are two essential mechanisms
that enable users to control their personal data processing. These sources
(Antón and Earp, 2004; ISO/IEC, 2011; Department of Health and Human Services,
2013) confirm that user privacy is vulnerable if users are not presented with
options to specify or cannot modify their user privacy preferences. Similarly,
user privacy may be compromised if the users cannot modify or withdraw their
consent, or are not notified about any changes of consent (Office Journal of
the European Union, 2016; ISO/IEC, 2011; Department of Health and Human
Services, 2013; Federal Register of Legislation, 2017; The OWASP Foundation,
2015). However, none of them (except not asking for a user consent
vulnerability) are covered in existing CVE.

Other areas that are not well covered in existing CWEs and CVEs are privacy
vulnerabilities due to insecurity and secondary use. Processing personal data
at a third party is a risk since they may apply lower levels of personal data
protection, particularly mobile (Zhang et al., 2020) and web applications (The
OWASP Foundation, 2015). There are cases that user privacy is violated when
personal data is used for unspecified purposes or used or transferred without
permissions (Antón and Earp, 2004; Deng et al., 2011; Zhang et al., 2020; Jana
et al., 2013; Hasan et al., 2020; Rafiq et al., 2017; Lebeck et al., 2018;
Iyilade and Vassileva, 2014; De and Metayer, 2016; Fisk et al., 2015; Office
Journal of the European Union, 2016; ISO/IEC, 2011; Department of Health and
Human Services, 2013; United State Congress, 1999; U.S. Department of Justice,
1974; Federal Register of Legislation, 2017; The OWASP Foundation, 2015). In
addition, although allowing unauthorised actors to modify personal data and
collecting personal data without user consent/permissions are serious threats,
none of the existing privacy vulnerabilities in CWE and CVE cover this.

The existing privacy weaknesses and vulnerabilities reported in the CWE and
CVE systems cover only 13 out of 24 common privacy vulnerabilities raised in
research and practice.

##  5 New common privacy weaknesses

Our study has shown the gaps in the existing CWE and CVE systems in terms of
covering privacy vulnerabilities. To fill these gaps, we propose 11 new common
privacy weaknesses to be added to the CWE system. We focus on CWE instead of
CVE since CVE specifies unique vulnerabilities existing in specific software
systems and application, while CWEs are at the generic level, similar to our
taxonomy of privacy threats. We followed the CWE schema (CWE Team, 2021) to
define the new CWE entries which include attributes such as name, description,
mode of introduction, common consequence, detection method, potential
mitigation and demonstrative example. These attributes provide an overview of
a privacy weakness in terms of its causes and consequences, and mitigation
methods.

The new common privacy weaknesses that we propose address four groups of
threats: surveillance, insecurity, secondary use and exclusion. The weaknesses
in the insecurity group can be detected and resolved by implementing security
mechanisms to better protect personal data and user privacy. On the other
hand, the other groups of weaknesses require more attention from software
development teams on examining privacy constraints involved, and designing
relevant functions to respond to those constraints. The software development
team needs to determine relevant functions, takes privacy constraints into
consideration and regularly reviews existing functions to ensure that they do
not violate user privacy. For example, a user may want to change his/her user
preferences when a new feature is launched, or a system must notify its users
when there is a change to the policy that the users have given consent to.

Due to space limit, we present here two examples (see Table 4 and Table 5) of
the new CWE privacy weaknesses and refer to the readers to our replication
package (Sangaroonsilp et al., 2021) for the remaining newly proposed CWEs.
Table 4 shows a new common privacy weakness for not allowing a user to
withdraw his/her consent. This weakness belongs to the exclusion category.
Following the CWE schema, a short summary of the weakness is provided in its
name, while a detailed description is provided in the description section. The
mode of introduction briefly discusses how and when the weakness is
introduced, which in this case is the architecture and design phase.

Table 4: An example of a missing consent withdrawal weakness. Subcategory:
Exclusion
---
Class: Not allowing a user to withdraw his/her consent
Name: Missing consent withdrawal
Description: The software forces users to give consent before providing its
services. These services may include personal data processing. However, the
software does not a provide a function for users to withdraw their consent.
The users can only accept consent, but they cannot withdraw their consent when
they wish to. This vulnerability seriously violates user privacy.
Mode of introduction: Phase: Architecture and Design. This weakness is caused
by a missing privacy consideration about consent management and its related
processes, which leads to the missing consent withdrawal function.
Common consequence: The software violates user privacy by not allowing users
to express their agreement on the use of their personal data.
Detection method: Method: Manual analysis. Description: A consent management
page or window in a software does not show an icon or option to withdraw
consent.
Potential mitigations: Phase: Architecture and Design. Strategy: User consent
withdrawal consideration. Description: The software development team should
consider which points in the software that should provide a user an ability to
withdraw consent.
Demonstrative example: Issue MDL-62309 in Moodle reports that the users cannot
withdraw consent and cannot enter the site without giving consent. This issue
violates user privacy as the users should be able to freely withdraw consent.
Figure 8: An example of missing consent withdrawal function in Moodle (Issue
MDL-62309).

The common consequence section identifies a privacy property that is violated
and an effect that is caused by the weakness. The detection method section
describes different methods that the weakness can be detected in software. We
also propose methods to mitigate the weakness. It is noted that different
phases in software development may pose different privacy concerns. Finally,
we provide a demonstrative example of the new weaknesses by extracting code
fragments, issue reports and commits from real software repositories. For
example, Figure 8 shows a code fragment extracted from the Github commit
_ad5e213_
191919https://github.com/moodle/moodle/commit/ad5e2135c5d2ccd7f53a08fc0c66de66d431cfdf
in Issue MDL-62309202020https://tracker.moodle.org/browse/MDL-62309 of the
Moodle project. This example demonstrates the existence of the missing consent
withdrawal weakness in practice.

Table 5 presents a new common privacy weakness for collecting personal data
without user consent/permissions. This weakness belongs to improper personal
data collection category. Figure 9 shows a code fragment confirming the
existence of a missing consent check before collecting personal data weakness.
The code fragment is extracted from the Github commit _0b09df0_
212121https://github.com/HumanDynamics/rhythm-server/commit/
0b09df0fa0a35dd1fe8a6b2160fb4e68299574d4 of the HumanDynamic repository (Human
Dynamics, ).

Table 5: An example of a collecting personal data without user
consent/permissions weakness. Subcategory: Surveillance
---
Class: Collecting personal data without user consent/permissions
Name: Missing a consent check before collecting personal data
Description: The software does not check for a user consent prior to personal
data collection. This makes the software collects personal data that users
have not given consent to (e.g., location and speech).
Mode of introduction: Phase: Implementation. This weakness is caused by
missing a consent check before collecting personal data.
Common consequence: The software violates user privacy since users has not
given consent/permissions to collect their personal data.
Detection method: Method: Manual analysis. Description: Perform a code check
at points of personal data collection.
Potential mitigations: Phase: Implementation. Strategy: Check for user consent
before collecting data. Description: The software development team should
perform a consent check at every point that collects personal data in the
software.
Demonstrative example: A commit 0b09df0 in HumanDynamics repository collects
user speech without consent check.
Figure 9: An example of missing a consent check before collecting personal
data in HumanDynamics (Commit 0b09df0).

We propose 11 new common privacy weaknesses to be added to CWE. These will
significantly improve the coverage of privacy weaknesses and vulnerabilities
in CWE, and subsequently CVE.

##  6 Threats to validity

Our method for the extraction of privacy vulnerabilities in CWE and CVE using
keywords might not result in the complete list. However, we have used several
strategies to mitigate this threat such as determining the keywords based on
alternate terms described in CWE, and using the frequent terms identified in
the studies that performed a large-scale analysis in privacy policies and
considering general terms to cover unseen materials (e.g., regulation, data
protection and privacy standard). Our taxonomy of common privacy threats is
constructed and refined based on the existing privacy threats taxonomy. We
have put our best effort to ensure the comprehensiveness of the study by
examining popular software engineering publication venues, well-established
data protection regulations and privacy frameworks, and reputable industry
sources. However, we acknowledge that there might be other sources of other
privacy threats that we have not identified yet. We have carefully defined a
set of inclusion and exclusion criteria to select the most relevant papers so
that we got a reasonable number of papers to be examined individually. Future
work would involve expanding our explanatory study to increase the
generalisability of our taxonomy for common privacy threats. In addition,
classifying the privacy-related vulnerabilities in CWE and CVE into the common
privacy threats in the taxonomy involved subjective judgements. We have
applied several strategies (e.g., using multiple coders, applying inter-rater
reliability assessments and conducting disagreement resolution) to mitigate
this threat. Future work could explore the use of external subject matter
experts in these tasks.

##  7 Conclusions and future work

The increasing use of software applications in people’s daily lives has put
privacy under constant threat as personal data are collected, processed and
transferred by many software applications. In this paper, we performed a
number of studies on privacy vulnerabilities in software applications. Our
study on the Common Weakness Enumeration (CWE) and Common Vulnerabilities and
Exposures (CVE) systems found that the coverage of privacy-related
vulnerabilities in both systems is quite limited (4.45% in the CWE system and
0.1% in the CVE system).

We have also investigated on how those privacy-related vulnerabilities
identified in CWE and CVE address the common privacy threats in software
applications. To do so, we developed a taxonomy of common privacy threats
based on selected privacy engineering research, well-established data
protection regulations and privacy frameworks and industry resources. We have
found that only 13 out of 24 common privacy vulnerabilities in the taxonomy
are covered by the existing weaknesses and vulnerabilities reported in CWE and
CVE. The top three most addressed vulnerabilities are exposing personal data
to an unauthorised actor, insufficient levels of protection and personal data
attacks. Based on these actionable insights, we proposed 11 new common privacy
weaknesses to be added to the CWE system. We also mined code fragments from
real software repositories to confirm the existence of those privacy
weaknesses. These newly proposed weaknesses will significantly improve the
coverage of privacy weaknesses and vulnerabilities in CWE, and subsequently
CVE.

Future work involves expanding our taxonomy to cover additional common privacy
threats that may have raised or discussed in other sources that we have not
included in this study. We will also perform a study to characterise privacy
vulnerabilities in software applications. This will enable us to develop new
techniques and tools for automatically detecting privacy vulnerabilities in
software and suggesting fixes.

## References

  * Ahamed et al. (2009) Ahamed, S.I., Li, H., Talukder, N., Monjur, M., Hasan, C.S., 2009.  Design and implementation of S-MARKS: A secure middleware for pervasive computing applications.  Journal of Systems and Software 82, 1657–1677.  URL: https://dl.acm.org/doi/abs/10.1016/j.jss.2009.03.020, doi:10.1016/j.jss.2009.03.020.
  * Alsamani and Lahza (2018) Alsamani, B., Lahza, H., 2018\.  A taxonomy of IoT: Security and privacy threats, in: 2018 International Conference on Information and Computer Technologies, ICICT 2018, Institute of Electrical and Electronics Engineers Inc.. pp. 72–77.  doi:10.1109/INFOCT.2018.8356843.
  * Antón and Earp (2004) Antón, A.I., Earp, J.B., 2004\.  A requirements taxonomy for reducing Web site privacy vulnerabilities.  Requirements Engineering 9, 169–185.  doi:10.1007/s00766-003-0183-z.
  * Ayala-Rivera and Pasquale (2018) Ayala-Rivera, V., Pasquale, L., 2018\.  The grace period has ended: An approach to operationalize GDPR requirements, in: Proceedings - 2018 IEEE 26th International Requirements Engineering Conference, RE 2018, Institute of Electrical and Electronics Engineers Inc.. pp. 136–146.  doi:10.1109/RE.2018.00023.
  * Bambauer (2013) Bambauer, D.E., 2013.  Privacy Versus Security.  Journal of Criminal Law and Criminology 103\.  URL: https://scholarlycommons.law.northwestern.edu/jclc/vol103/iss3/2.
  * Barman et al. (2015) Barman, L., Elgraini, M.T., Raisaro, J.L., Hubaux, J.P., Ayday, E., 2015.  Privacy threats and practical solutions for genetic risk tests, in: Proceedings - 2015 IEEE Security and Privacy Workshops, SPW 2015, pp. 27–31.  doi:10.1109/SPW.2015.12.
  * Bertolino et al. (2018) Bertolino, A., Calabrò, A., Lonetti, F., Marchetti, E., Miranda, B., 2018.  A categorization scheme for software engineering conference papers and its application, in: Journal of Systems and Software, Elsevier. pp. 114–129.  doi:10.1016/j.jss.2017.11.048.
  * Bhandari et al. (2021) Bhandari, G.P., Naseer, A., Moonen, L., 2021.  CVEfixes: Automated Collection of Vulnerabilities and Their Fixes from Open-Source Software URL: https://doi.org/10.1145/3475960.3475985http://arxiv.org/abs/2107.08760%0Ahttp://dx.doi.org/10.1145/3475960.3475985, doi:10.1145/3475960.3475985, arXiv:2107.08760.
  * Bhatia and Breaux (2018) Bhatia, J., Breaux, T.D., 2018\.  Semantic incompleteness in privacy policy goals, in: Proceedings - 2018 IEEE 26th International Requirements Engineering Conference, RE 2018, pp. 159--169.  doi:10.1109/RE.2018.00025.
  * Bhatia et al. (2016) Bhatia, J., Breaux, T.D., Reidenberg, J.R., Norton, T.B., 2016\.  A Theory of Vagueness and Privacy Risk Perception, in: Proceedings - 2016 IEEE 24th International Requirements Engineering Conference, RE 2016, Institute of Electrical and Electronics Engineers Inc.. pp. 26--35.  doi:10.1109/RE.2016.20.
  * Bilogrevic et al. (2011) Bilogrevic, I., Jadliwala, M., Kumar, P., Walia, S.S., Hubaux, J.P., Aad, I., Niemi, V., 2011.  Meetings through the cloud: Privacy-preserving scheduling on mobile devices, in: Journal of Systems and Software, Elsevier Science Inc. PUB552 New York, NY, USA. pp. 1910--1927.  URL: https://dl.acm.org/doi/abs/10.1016/j.jss.2011.04.027, doi:10.1016/j.jss.2011.04.027.
  * Breaux and Antón (2008) Breaux, T.D., Antón, A.I., 2008\.  Analyzing regulatory rules for privacy and security requirements.  IEEE Transactions on Software Engineering 34, 5--20.  doi:10.1109/TSE.2007.70746.
  * Calandrino et al. (2011) Calandrino, J.A., Kilzer, A., Narayanan, A., Felten, E.W., Shmatikov, V., 2011.  "You might also like:" Privacy risks of collaborative filtering, in: Proceedings - IEEE Symposium on Security and Privacy, pp. 231--246.  doi:10.1109/SP.2011.40.
  * Calciati et al. (2018) Calciati, P., Kuznetsov, K., Bai, X., Gorla, A., 2018.  What did really change with the new release of the app?, in: Proceedings - International Conference on Software Engineering, pp. 142--152.  URL: https://doi.org/10.1145/3196398.3196449, doi:10.1145/3196398.3196449.
  * California State Legislature (2018) California State Legislature, 2018.  California Consumer Privacy Act of 2018.
  * Castiglione et al. (2010) Castiglione, A., De Santis, A., Soriente, C., 2010.  Security and privacy issues in the Portable Document Format.  Journal of Systems and Software 83, 1813--1822.  doi:10.1016/j.jss.2010.04.062.
  * Chakraborty et al. (2021) Chakraborty, S., Krishna, R., Ding, Y., Ray, B., 2021.  Deep Learning based Vulnerability Detection: Are We There Yet?  IEEE Transactions on Software Engineering TBD.  URL: https://git.io/Jf6IA., doi:10.1109/tse.2021.3087402, arXiv:2009.07235.
  * Chalhoub et al. (2020) Chalhoub, G., Flechais, I., Nthala, N., Abu-Salma, R., 2020\.  Innovation inaction or in action? The role of user experience in the security and privacy design of smart home cameras, in: Proceedings of the 16th Symposium on Usable Privacy and Security, SOUPS 2020, pp. 185--204.  URL: https://www.usenix.org/conference/soups2020/presentation/reinheimer.
  * Chernis and Verma (2018) Chernis, B., Verma, R., 2018\.  Machine learning methods for software vulnerability detection, in: IWSPA 2018 - Proceedings of the 4th ACM International Workshop on Security and Privacy Analytics, Co-located with CODASPY 2018, pp. 31--39.  URL: https://doi.org/10.1145/3180445., doi:10.1145/3180445.3180453.
  * CWE Team (2021) CWE Team, 2021.  CWE - Schema Documentation - Schema Version 6.5.
  * Dam et al. (2021) Dam, H.K., Tran, T., Pham, T., Ng, S.W., Grundy, J., Ghose, A., 2021.  Automatic Feature Learning for Predicting Vulnerable Software Components.  IEEE Transactions on Software Engineering 47, 67--85.  doi:10.1109/TSE.2018.2881961.
  * Data Privacy Manager (2021) Data Privacy Manager, 2021.  5 things you need to know about Data Privacy.  URL: https://dataprivacymanager.net/5-things-you-need-to-know-about-data-privacy/.
  * De and Metayer (2016) De, S.J., Metayer, D.L., 2016\.  Privacy Harm Analysis: A Case Study on Smart Grids, in: Proceedings - 2016 IEEE Symposium on Security and Privacy Workshops, SPW 2016, Institute of Electrical and Electronics Engineers Inc.. pp. 58--65.  doi:10.1109/SPW.2016.21.
  * Deng et al. (2011) Deng, M., Wuyts, K., Scandariato, R., Preneel, B., Joosen, W., 2011.  A privacy threat analysis framework: Supporting the elicitation and fulfillment of privacy requirements.  Requirements Engineering 16, 3--32.  URL: https://link.springer.com/article/10.1007/s00766-010-0115-7, doi:10.1007/s00766-010-0115-7.
  * Department of Health and Human Services (2013) Department of Health and Human Services, 2013\.  HIPAA administrative simplification regulation text.  URL: https://www.hhs.gov/sites/default/files/hipaa-simplification-201303.pdf.
  * Drosatos et al. (2014) Drosatos, G., Efraimidis, P.S., Athanasiadis, I.N., Stevens, M., D’Hondt, E., 2014.  Privacy-preserving computation of participatory noise maps in the cloud.  Journal of Systems and Software 92, 170--183.  doi:10.1016/j.jss.2014.01.035.
  * Ebrahimi et al. (2019) Ebrahimi, F., Tushev, M., Mahmoud, A., 2019.  Mobile App Privacy in Software Engineering Research: A Systematic Mapping Study.  IEEE TRANSACTIONS ON SOFTWARE ENGINEERING 14\.  URL: http://seel.cse.lsu.edu/data/TSE2019.xlsxhttp://arxiv.org/abs/1910.03622, arXiv:1910.03622.
  * Erola et al. (2011) Erola, A., Castell-Roca, J., Viejo, A., Mateo-Sanz, J.M., 2011\.  Exploiting social networks to provide privacy in personalized web search.  Journal of Systems and Software 84, 1734--1745.  URL: https://dl.acm.org/doi/abs/10.1016/j.jss.2011.05.009, doi:10.1016/j.jss.2011.05.009.
  * Federal Register of Legislation (2017) Federal Register of Legislation, 2017.  Privacy Act 1988.  URL: https://www.legislation.gov.au/Details/C2016C00979.
  * Figueiredo et al. (2017) Figueiredo, L.S., Molnar, D., Veanes, M., Livshits, B., 2017\.  Prepose: Privacy, security, and reliability for gesture-based programming, in: IEEE Security and Privacy, Institute of Electrical and Electronics Engineers Inc.. pp. 14--23.  doi:10.1109/MSP.2017.44.
  * Fisk et al. (2015) Fisk, G., Ardi, C., Pickett, N., Heidemann, J., Fisk, M., Papadopoulos, C., 2015\.  Privacy principles for sharing cyber security data, in: Proceedings - 2015 IEEE Security and Privacy Workshops, SPW 2015, Institute of Electrical and Electronics Engineers Inc.. pp. 193--197.  doi:10.1109/SPW.2015.23.
  * Galhardo et al. (2020) Galhardo, C.C., Mell, P., Bojanova, I., Gueye, A., 2020\.  Measurements of the Most Significant Software Security Weaknesses, in: ACM International Conference Proceeding Series, ACM. pp. 154--164.  URL: https://doi.org/10.1145/3427228.3427257, doi:10.1145/3427228.3427257.
  * Gonzalez et al. (2019) Gonzalez, D., Hastings, H., Mirakhorli, M., 2019.  Automated Characterization of Software Vulnerabilities, in: Proceedings - 2019 IEEE International Conference on Software Maintenance and Evolution, ICSME 2019, pp. 135--139.  URL: http://www.securityfocus.com/bid/99202,, doi:10.1109/ICSME.2019.00023, arXiv:1909.13693.
  * Guitierrez et al. (2013) Guitierrez, C.M., Jeffrey, W., Furlani, C.M., 2013.  FIPS PUB 200: Minimum Security Requirements for Federal Information and Information Systems.  Technical Report.  URL: http://csrc.nist.gov/publications/fips/fips200/FIPS-200-final-march.pdf.
  * Guo et al. (2020) Guo, H., Kafali, Ö., Jeukeng, A.L., Williams, L., Singh, M.P., 2020.  Çorba: crowdsourcing to obtain requirements from regulations and breaches.  Empirical Software Engineering 25, 532--561.  doi:10.1007/s10664-019-09753-2.
  * Gürses and Del Álamo (2016) Gürses, S., Del Álamo, J.M., 2016\.  Privacy Engineering: Shaping an Emerging Field of Research and Practice.  IEEE Security and Privacy 14, 40--46.  doi:10.1109/MSP.2016.37.
  * Hallgren (2012) Hallgren, K.A., 2012.  Computing inter-rater reliability for observational data: An overview and tutorial.  Tutorials in Quantitative Methods for Psychology 8, 23--34.  doi:10.20982/tqmp.09.2.p095.
  * Hanif et al. (2021) Hanif, H., Md Nasir, M.H.N., Ab Razak, M.F., Firdaus, A., Anuar, N.B., 2021.  The rise of software vulnerability: Taxonomy of software vulnerabilities detection and machine learning approaches.  Journal of Network and Computer Applications 179, 103009.  doi:10.1016/j.jnca.2021.103009.
  * Hasan et al. (2020) Hasan, R., Crandall, D., Fritz, M., Kapadia, A., 2020\.  Automatically detecting bystanders in photos to reduce privacy risks, in: Proceedings - IEEE Symposium on Security and Privacy, Institute of Electrical and Electronics Engineers Inc.. pp. 318--335.  doi:10.1109/SP40000.2020.00097.
  * Horbe and Hotzendorfer (2015) Horbe, R., Hotzendorfer, W., 2015\.  Privacy by design in federated identity management, in: Proceedings - 2015 IEEE Security and Privacy Workshops, SPW 2015, Institute of Electrical and Electronics Engineers Inc.. pp. 167--174.  doi:10.1109/SPW.2015.24.
  * (41) Human Dynamics, .  Human Dynamics Github Repository.  URL: https://github.com/HumanDynamics.
  * ISO/IEC (2011) ISO/IEC, 2011.  International Standard ISO/IEC 29100.  Technical Report. ISO/IEC.
  * Iyilade and Vassileva (2014) Iyilade, J., Vassileva, J., 2014\.  P2U: A privacy policy specification language for secondary data sharing and usage, in: Proceedings - IEEE Symposium on Security and Privacy, Institute of Electrical and Electronics Engineers Inc.. pp. 18--22.  doi:10.1109/SPW.2014.12.
  * Jana et al. (2013) Jana, S., Narayanan, A., Shmatikov, V., 2013.  A scanner darkly: Protecting user privacy from perceptual applications, in: Proceedings - IEEE Symposium on Security and Privacy, pp. 349--363.  doi:10.1109/SP.2013.31.
  * Johnson (2021a) Johnson, J., 2021a.  Cyber crime: biggest online data breaches as of 2020\.  URL: https://www.statista.com/statistics/290525/cyber-crime-biggest-online-data-breaches-worldwide/.
  * Johnson (2021b) Johnson, J., 2021b.  Internet users in the world 2021.  URL: https://www.statista.com/statistics/617136/digital-population-worldwide/.
  * Kalloniatis et al. (2008) Kalloniatis, C., Kavakli, E., Gritzalis, S., 2008.  Addressing privacy requirements in system design: The PriS method.  Requirements Engineering 13, 241--255.  doi:10.1007/s00766-008-0067-3.
  * Kitchenham (2007) Kitchenham, B., 2007.  Guidelines for performing systematic literature reviews in software engineering.  Technical report, Ver. 2.3 EBSE Technical Report. EBSE .
  * Kotz (2011) Kotz, D., 2011.  A threat taxonomy for mHealth privacy, in: 2011 3rd International Conference on Communication Systems and Networks, COMSNETS 2011.  doi:10.1109/COMSNETS.2011.5716518.
  * Kröger et al. (2020) Kröger, J.L., Lindemann, J., Herrmann, D., 2020.  How do app vendors respond to subject access requests?: A longitudinal privacy study on iOS and Android Apps, in: ARES ’20: Proceedings of the 15th International Conference on Availability, Reliability and Security, ACM, New York, NY, USA. pp. 1--10.  URL: https://doi.org/10.1145/3407023.3407057, doi:10.1145/3407023.3407057.
  * Lebeck et al. (2018) Lebeck, K., Ruth, K., Kohno, T., Roesner, F., 2018\.  Towards Security and Privacy for Multi-user Augmented Reality: Foundations with End Users, in: Proceedings - IEEE Symposium on Security and Privacy, Institute of Electrical and Electronics Engineers Inc.. pp. 392--408.  doi:10.1109/SP.2018.00051.
  * Li et al. (2017) Li, X., Chen, J., Lin, Z., Zhang, L., Wang, Z., Zhou, M., Xie, W., 2017.  A Mining Approach to Obtain the Software Vulnerability Characteristics, in: Proceedings - 5th International Conference on Advanced Cloud and Big Data, CBD 2017, Institute of Electrical and Electronics Engineers Inc.. pp. 296--301.  doi:10.1109/CBD.2017.58.
  * Lin (2016) Lin, C.Y., 2016.  A reversible data transform algorithm using integer transform for privacy-preserving data mining.  Journal of Systems and Software 117, 104--112.  doi:10.1016/j.jss.2016.02.005.
  * Lin et al. (2020) Lin, G., Wen, S., Han, Q.L., Zhang, J., Xiang, Y., 2020\.  Software Vulnerability Detection Using Deep Neural Networks: A Survey.  Proceedings of the IEEE 108, 1825--1848.  URL: https://www.ieee.org/publications/rights/index.html, doi:10.1109/JPROC.2020.2993293.
  * Liu et al. (2012) Liu, B., Shi, L., Cai, Z., Li, M., 2012.  Software vulnerability discovery techniques: A survey, in: Proceedings - 2012 4th International Conference on Multimedia and Security, MINES 2012, pp. 152--156.  doi:10.1109/MINES.2012.202.
  * Liu et al. (2020) Liu, S., Lin, G., Han, Q.L., Wen, S., Zhang, J., Xiang, Y., 2020.  DeepBalance: Deep-Learning and Fuzzy Oversampling for Vulnerability Detection.  IEEE Transactions on Fuzzy Systems 28, 1329--1343.  doi:10.1109/TFUZZ.2019.2958558.
  * Lucia et al. (2012) Lucia, Lo, D., Jiang, L., Budi, A., 2012.  Kb e-anonymity: Test data anonymization for evolving programs, in: 2012 27th IEEE/ACM International Conference on Automated Software Engineering, ASE 2012 - Proceedings, pp. 262--265.  doi:10.1145/2351676.2351718.
  * Ma et al. (2013) Ma, C.Y., Yau, D.K., Yip, N.K., Rao, N.S., 2013.  Privacy vulnerability of published anonymous mobility traces.  IEEE/ACM Transactions on Networking 21, 720--733.  doi:10.1109/TNET.2012.2208983.
  * Massey et al. (2013) Massey, A.K., Eisenstein, J., Anton, A.I., Swire, P.P., 2013\.  Automated text mining for requirements analysis of policy documents, in: 2013 21st IEEE International Requirements Engineering Conference, RE 2013 - Proceedings, IEEE Computer Society. pp. 4--13.  doi:10.1109/RE.2013.6636700.
  * Mihaylov et al. (2016) Mihaylov, B., Onea, L., Hansen, K.M., 2016.  Architecture-based regulatory compliance argumentation.  Journal of Systems and Software 119, 1--30.  doi:10.1016/j.jss.2016.04.057.
  * Norton (2021) Norton, 2021.  Why Your Online Privacy Matters.  URL: https://us.norton.com/internetsecurity-privacy-why-your-online-privacy-matters.html.
  * Office Journal of the European Union (2016) Office Journal of the European Union, 2016.  Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Da.  Technical Report.  URL: https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=OJ:L:2016:119:FULL, doi:10.1017/9781780688459.021.
  * Omoronyia et al. (2013) Omoronyia, I., Cavallaro, L., Salehie, M., Pasquale, L., Nuseibeh, B., 2013.  Engineering adaptive privacy: On the role of privacy awareness requirements, in: Proceedings - International Conference on Software Engineering, pp. 632--641.  doi:10.1109/ICSE.2013.6606609.
  * Omoronyia et al. (2012) Omoronyia, I., Pasquale, L., Salehie, M., Cavallaro, L., Doherty, G., Nuseibeh, B., 2012\.  Caprice: A tool for engineering adaptive privacy, in: 2012 27th IEEE/ACM International Conference on Automated Software Engineering, ASE 2012 - Proceedings, pp. 354--357.  doi:10.1145/2351676.2351745.
  * Open Web Application Security Project (2020) Open Web Application Security Project, 2020\.  About Us | The OWASP Foundation.  URL: https://owasp.org/about/.
  * Perera et al. (2020) Perera, H., Hussain, W., Whittle, J., Nurwidyantoro, A., Mougouei, D., Shams, R.A., Oliver, G., 2020.  A study on the prevalence of human values in software engineering publications 2015 - 2018, in: Proceedings - International Conference on Software Engineering, IEEE Computer Society. pp. 409--420.  URL: https://doi.org/10.1145/3377811.3380393, doi:10.1145/3377811.3380393, arXiv:1907.07874.
  * Rafiq et al. (2017) Rafiq, Y., Dickens, L., Russo, A., Bandara, A.K., Yang, M., Stuart, A., Levine, M., Calikli, G., Price, B.A., Nuseibeh, B., 2017\.  Learning to share: Engineering adaptive decision-support for online social networks, in: ASE 2017 - Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering, Institute of Electrical and Electronics Engineers Inc.. pp. 280--285.  doi:10.1109/ASE.2017.8115641.
  * Sangaroonsilp et al. (2021) Sangaroonsilp, P., Dam, H.K., Ghose, A., 2021.  Replication Package for Common Privacy Weaknesses and Vulnerabilities in Software Applications manuscript.  URL: https://bit.ly/JSS21CommonPrivacyVul.
  * Scholz (2015) Scholz, L.H., 2015.  Information Privacy and Data Security, in: Cardozo Law Review de Novo, 2015.  URL: https://papers.ssrn.com/abstract=2600495.
  * Sicari et al. (2012) Sicari, S., Grieco, L.A., Boggia, G., Coen-Porisini, A., 2012\.  DyDAP: A dynamic data aggregation scheme for privacy aware wireless sensor networks.  Journal of Systems and Software 85, 152--166.  doi:10.1016/j.jss.2011.07.043.
  * Siewe and Yang (2016) Siewe, F., Yang, H., 2016.  Privacy protection by typing in ubiquitous computing systems.  Journal of Systems and Software 120, 133--153.  doi:10.1016/j.jss.2016.07.037.
  * SNIA (2021) SNIA, 2021.  What is Data Privacy?  URL: https://www.snia.org/education/what-is-data-privacy.
  * Solove (2006) Solove, D.J., 2006.  A taxonomy of privacy.  doi:10.2307/40041279.
  * Some (2019) Some, D.F., 2019.  EmPoWeb: Empowering web applications with browser extensions, in: Proceedings - IEEE Symposium on Security and Privacy, Institute of Electrical and Electronics Engineers Inc.. pp. 227--245.  doi:10.1109/SP.2019.00058, arXiv:1901.03397.
  * Stallings (2019) Stallings, W., 2019.  Information Privacy Engineering and Privacy by Design: Understanding Privacy Threats, Technology, and Regulations Based on Standards and Best Practices.  Addison-Wesley Professional.
  * Statista (2021) Statista, 2021.  Annual number of data breaches and exposed records in the United States from 2005 to 2020.  URL: https://www.statista.com/statistics/273550/data-breaches-recorded-in-the-united-states-by-number-of-breaches-and-records-exposed/.
  * Stouffer (2021) Stouffer, C., 2021.  What is social engineering?  URL: https://us.norton.com/internetsecurity-emerging-threats-what-is-social-engineering.html.
  * The MITRE Corporation (2021a) The MITRE Corporation, 2021a.  About CVE.  URL: https://cve.mitre.org/about/index.html.
  * The MITRE Corporation (2021b) The MITRE Corporation, 2021b.  About CWE.  URL: https://cwe.mitre.org/about/index.html.
  * The Office of the Australian Information Commissioner (2021) (OAIC) The Office of the Australian Information Commissioner (OAIC), 2021.  Notifiable Data Breaches Report.  Technical Report December 2020.
  * The OWASP Foundation (2015) The OWASP Foundation, 2015.  OWASP Top 10 Privacy Risks Survey.
  * Tschersich et al. (2011) Tschersich, M., Kahl, C., Heim, S., Crane, S., Böttcher, K., Krontiris, I., Rannenberg, K., 2011.  Towards privacy-enhanced mobile communities - Architecture, concepts and user trials.  Journal of Systems and Software 84, 1947--1960.  URL: www.pepers.org, doi:10.1016/j.jss.2011.06.048.
  * Tuan Anh et al. (2014) Tuan Anh, D.T., Ganjoo, M., Braghin, S., Datta, A., 2014\.  Mosco: A privacy-aware middleware for mobile social computing.  Journal of Systems and Software 92, 20--31.  URL: https://www.infona.pl//resource/bwmeta1.element.elsevier-ef19f2ed-d759-30fe-b942-4e347e412634, doi:10.1016/j.jss.2013.11.1110.
  * United State Congress (1999) United State Congress, 1999.  GRAMM-LEACH-BLILEY ACT.  URL: https://www.govinfo.gov/content/pkg/PLAW-106publ102/pdf/PLAW-106publ102.pdf.
  * U.S. Department of Justice (1974) U.S. Department of Justice, 1974.  The Privacy Act of 1974.
  * Venkatadri et al. (2018) Venkatadri, G., Andreou, A., Liu, Y., Mislove, A., Gummadi, K.P., Loiseau, P., Goga, O., 2018.  Privacy Risks with Facebook’s PII-Based Targeting: Auditing a Data Broker’s Advertising Interface, in: Proceedings - IEEE Symposium on Security and Privacy, Institute of Electrical and Electronics Engineers Inc.. pp. 89--107.  doi:10.1109/SP.2018.00014.
  * Viera and Garrett (2005) Viera, A.J., Garrett, J.M., 2005\.  Understanding interobserver agreement: the kappa statistic.  Family Medicine 37, 360--363.
  * Wang et al. (2010) Wang, T., Wei, T., Gu, G., Zou, W., 2010.  TaintScope: A checksum-aware directed fuzzing tool for automatic software vulnerability detection, in: Proceedings - IEEE Symposium on Security and Privacy, pp. 497--512.  doi:10.1109/SP.2010.37.
  * Yang et al. (2016) Yang, G., Yu, J., Shen, W., Su, Q., Fu, Z., Hao, R., 2016.  Enabling public auditing for shared data in cloud storage supporting identity privacy and traceability.  Journal of Systems and Software 113, 130--139.  doi:10.1016/j.jss.2015.11.044.
  * Yang et al. (2013) Yang, Z., Yang, M., Zhang, Y., Gu, G., Ning, P., Wang, X.S., 2013.  AppIntent: Analyzing sensitive data transmission in Android for privacy leakage detection, in: Proceedings of the ACM Conference on Computer and Communications Security, pp. 1043--1054.  URL: http://dx.doi.org/10.1145/2508859.2516676, doi:10.1145/2508859.2516676.
  * Yu et al. (2021) Yu, L., Luo, X., Chen, J., Zhou, H., Zhang, T., Chang, H., Leung, H.K., 2021\.  PPChecker: Towards Accessing the Trustworthiness of Android Apps’ Privacy Policies.  IEEE Transactions on Software Engineering 47, 221--242.  doi:10.1109/TSE.2018.2886875.
  * Zeng et al. (2019) Zeng, E., Mare, S., Roesner, F., 2019.  End user security and privacy concerns with smart homes, in: Proceedings of the 13th Symposium on Usable Privacy and Security, SOUPS 2017, pp. 65--80.  URL: https://www.usenix.org/conference/soups2017/technical-sessions/presentation/acar.
  * Zhang et al. (2005) Zhang, N., Shi, Q., Merabti, M., 2005.  Revocation of privacy-enhanced public-key certificates.  Journal of Systems and Software 75, 205--214.  doi:10.1016/j.jss.2004.04.013.
  * Zhang et al. (2020) Zhang, X., Wang, X., Slavin, R., Breaux, T., Niu, J., 2020.  How does misconfiguration of analytic services compromise mobile privacy, in: Proceedings - International Conference on Software Engineering, pp. 1572--1583.  doi:10.1145/3377811.3380401.