# The efficacy potential of cyber security advice as presented in news
articles

Mark Quinlan  Aaron Ceross  Andrew Simpson

###### Abstract

Cyber security advice is a broad church: it is thematically expansive,
comprising expert texts, user-generated data consumed by individual users via
informal learning, and much in-between. While there is evidence that cyber
security news articles play a role in disseminating cyber security advice, the
nature and extent of that role are not clear. We present a corpus of cyber
security advice generated from mainstream news articles. The work was driven
by two research objectives. The first objective was to ascertain what kind of
actionable advice is being disseminated; the second was to explore ways of
determining the efficacy potential of news-mediated security advice. The
results show an increase in the generation of cyber security news articles,
together with increases in vocabulary complexity and reading difficulty. We
argue that these could present challenges for vulnerable users. We believe
that this corpus and the accompanying analysis have the potential to inform
future efforts to quantify and improve the efficacy potential of security
advice dissemination.

###### keywords:

Cyber Security , Human Computer Interaction , Text Analysis , News Corpus ,
Online Security , HCI

††journal: Interacting with Computers

[inst1]organization=Department of Computer Science, University of Oxford,
addressline=Wolfson Building, Parks Road, city=Oxford, postcode=OX1 3QD,
country=United Kingdom

News-mediated security advice has been sharply increasing since 2018.

Many cyber security news articles have a high level of specificity.

Subject-specific terminology within our security news articles is continuously
evolving.

Cyber security news is often of short length and low readability, with a
negative impact on efficacy potential.

The research indicates increasingly diversified interest in goal-specific
cyber security advice.

##  1 Introduction

The usable security field is now nearly 30 years old [1], although the
underlying concepts are much older. For example, Auguste Kerckhoffs
acknowledged the role of the user in successful security implementation as
early as 1883 when he published his seminal piece on military cryptography
[2]. Since then, users have become active participants in ensuring the
security of systems and important mediators of experts’ security advice —
which, in the context of this paper, we define as explicit instructions
intended to make recipients more secure, once implemented [3, 4, 5].

The Internet is awash with implicit and explicit security advice, disseminated
both by experts and by other users. Much of this advice has its limitations.
For example, while the underlying threats may be common, the advice provided
to deal with these threats can differ significantly, both in terms of wording
and implied level of urgency. Explicit security advice that is either too
abstract or too vague, or that assumes a high level of security knowledge or
expertise, can fail to serve as _usable advice_ , as characterised by
Kerckhoffs: “the system must be easy to use and must neither require stress of
mind nor the knowledge of a long set of rules” [2].

Unfortunately, because individuals face a wide range of cyber security threats
— including viruses [6], bot-nets [7], port-scanners [8], spyware [9, 10],
malware [9], stalkerware [11], and rootkits [9, 12] — it is difficult for them
to develop sufficient ‘knowledge’ of the ‘rules’ even when advice is presented
clearly and thoroughly. Furthermore, there is the question of what individual
users do with the advice they receive. They may, for example, reject advice
they deem too difficult to implement [13, 5] or delay implementing advice
because they underestimate the consequences of losing control of their data
[14]. These complications are exacerbated by the ongoing centralisation of
services [15] and heavy use of social media platforms [16], which may inspire
complacency through habituation while introducing novel risks. There is
evidence that some users register threats on a subconscious level, raising the
possibility that they will ignore novel threats. Such so-called ‘security
fatigue’ has been discussed by authors such as Furnell and Thomson [17].

Despite the significance of these problems, there have been relatively few
quantitative studies analysing the security advice literature available to
what we might term ‘everyday’ users. Notable examples of such studies include
that of Redmiles and colleagues [3], which explores how individual users
actualise and perceive their own cyber security capabilities (which often
derive from informally learnt advice), and that of Renaud and Dupuis [18],
which identifies large sub-concepts and classifications in the security advice
field. We endeavor to bridge these contributions by providing additional
context about the current security advice environment. Our research is guided
by the following research objectives (to which we return in Section 5):

  * 1.

RO1: What kind of informally learnt and actionable security advice most often
appears in news articles?

  * 2.

RO2: What is the efficacy potential of this security advice as consumed by an
individual user?

We describe the process we undertook to assemble a corpus of security advice
that reflects the advice presented to individual users daily within their
typical informal learning environment. The corpus spans a 24-month period and
was assembled from a data-set containing 15,422 (English language) news and
online magazine articles from North American (US and Canada) and UK-based
sources, as well as some historical data stretching back to 2000. We ascribed
broad classifications to specific advice, ascertained the dominant methods of
advice construction and dissemination, and analysed their potential efficacy
potential over time.111This corpus and its associated database are available
from https://github.com/Quinma/CSNP. Our hope is that this corpus will help to
lay the foundations for further work on quantifying and improving the efficacy
potential of security advice dissemination.

The remainder of this paper is organised as follows. In Section 2 we provide
the motivation for, and the background to, our contribution, and define the
terms of interest. In Section 3 we describe the process used to create the
data-set, and our data cleansing process that develops the data-set into the
corpus. In Section 4 we present our results and perform an initial analysis,
before, in Section 5, discussing the results in relation to our research
objectives. Section 6 considers limitations to the study and considers
possible future directions. We conclude the paper in Section 7.

##  2 Background and motivation

In this section we discuss the background to, and the motivation for, the work
described in this paper. We start (in Section 2.1) by contextualising the term
_security advice_. We define what security advice is and how security advice
is learnt. We then consider the individual user and attempt to define a
persona to represent this user (in the broadest possible sense) with a view to
capturing their media and information landscapes.

###  2.1 What is security advice?

The concept of _advice_ is difficult to define, partly because it pertains to
almost every discipline involving human behaviour [19, 20]. _Security advice_
is particularly complicated, as it derives from diverse disciplines, including
psychology [21, 22], medicine [23], and computer science.222Given that cyber
security is the discipline of concern in this paper, we shall refer to it
simply as ‘security’. Each of these disciplines has its own approaches to
security advice, and, as such, it would be foolish to attempt a comprehensive
review of this topic covering all of these fields. Rather, we make reference
to those contributions that helped to frame, motivate and scope the present
study.

In this paper, rather than taking a broad view of security advice, we limit
our concerns to what we term _professional security advice_. Çelen et al. [24]
describe the term _professional advice_ as “advice rendered by experts” (see
Section 2.2 for a consideration of experts), which is then disseminated to
individual users. Professional security advice is designed to alleviate
security challenges, but, because this must often occur through the mediation
of individual users, these instructions are characteristically _persuasive_
and typically _explicit_ , rather than _implicit_.

Explicit advice often takes the form of a verbal or written appeal, such as
_Keep your browser up to date at all times_. There are even different
varieties of appeals, as with _Keep your browser up to date at all times_ …
_or else your browser will get infected_ , which constitutes a _fear appeal_
[25, 26]. Implicit advice often takes the form of a threat message [18], such
as _Downloading a file from an untrusted source is risky_ [27].

Based on this distinction and the definitions prevalent in the literature [3,
4, 5, 13, 28, 29, 30], we define _security advice_ thus:

> A written instruction, provided by a trusted and professional source, with
> the explicit goal of enabling the recipient to be more secure once they
> execute the instruction.

###  2.2 The expert

Note that this definition depends on the intent of the advice, rather than the
outcome. This is because professional security advice cannot ensure the
success of its security recommendations as these are often developed based on
limited observational data [31]. For example, it is difficult to quantify
threat-related data, such as the chance or probable extent of a malicious
actor attack [32, 33, 34, 35]. Similarly, it is difficult to calculate the
loss of customer confidence following such threats [36].

Although the projected costs of hardware and/or software development to
alleviate security threats might be well established in particular cases,
their associated indirect costs and downstream issues are not [35]. Despite
these uncertainties, professional security advice must appear authoritative,
and this gap means that some professional advice may prove ineffective.
Furthermore, it is sometimes the case that some security artefacts are
outdated or redundant almost immediately due to the ever-improving armoury
tools and techniques available to malicious actors [37, 38, 39]. It is
unfortunately the case that this arms race or game of ‘whack-a-mole’ is the
environment within which most security advice is created and disseminated
[40].

Within the context of this paper, we believe it to be important to provide an
explicit definition of an expert. First, expertise as a concept can be divided
into two distinct categories: expertise as a function of What someone knows,
and expertise as a function of What someone does. In the former, we are
interested primarily in the expert’s epistemic knowledge of a particular
domain — in this case, their capacity to provide justifications for any given
range of ideas and knowledge [41].

Many organisations and public-facing institutions employ security experts that
provide insight and compose security advice based on considerable expertise
gained from education and/or industry experience [42]. Shortages of such
security experts are regularly reported [43, 44, 45, 46], and this has led to
an requirement for additional tools and techniques that can be used to aid
current security experts in their work [46, 47]. Shires [48] assessed the
difficulty in establishing a firm definition of cyber security experts; an
analysis of self-described practices within media highlighted a varied
perception of how they operate in terms of acquiring and disseminating
information. Frey et al. [49] reported upon a test in which participants
possessing several levels of cyber security knowledge were tested. The self-
identified security experts tended to achieve poor scores in the test: “they
tended to display a strong interest in looking up advanced technological
solutions rather than intelligence gathering” [49].

Within almost any field in which professional advice is rendered, two forces
of market are always at play. The first concerns reputation, in which
professionals are interested in how their advice is seen to be well-informed
by colleagues and recipients [50, 51, 52]; the second concerns competition,
which, within our context (as our advice is provided by experts working within
media settings), relates to how competition between the advice providers can
distort information before it reaches the intended recipient. For example,
professional advice can often take the shape of a contest, in which the
experts are evaluated on the basis of their opinions. (An example of this
phenomenon can be found within the financial markets, where the Wall Street
Journal Forecasting survey pits analysts and experts against each other and
provides rankings [53, 54].) Both of these kinds of influences have the
potential to alter the contents of any advice rendered. Given the
indeterminacy, incompleteness or sometimes faultiness of the data used to
generate expert advice, alongside an unknown mixture of experts with a
particular education and/or occupational background, we consider the security
expert who renders our advice to be an individual who creates cyber security
knowledge out of a mixture of epistemological and performative expertise.
Their backgrounds and motivations are otherwise opaque to us — as they may
well be to the reader of the news articles — and this forms a limitation on
how we may perceive expertise in this field.

Of course, individual users do not always get their advice from experts [24,
55, 56]. Indeed, some may not receive information from professional sources at
all [3, 13, 28, 57]. Such users may rely on information sourced from their
local environment, which we might characterise as naïve advice [58]. Contrary
to this label, naïve advice has certain efficiency-enhancing properties when
used in negotiations [59], public-good experiments [60], and certain types of
games [61].

Although not the focus of this paper, naïve advice must be taken into account
when we examine the recipients of security advice and how they respond to such
advice.

###  2.3 Who are the consumers of security advice?

Based on existing literature [62, 63, 64], we define consumers of security
advice as ‘individual users’ who own systems, devices, and/or services that
maintain internet connectivity. An individual user can be of any gender, age
group, or professional background. We include corporate users in this
definition because their learning habits extend beyond their formal corporate
learning environments — that is, they may engage in informal learning outside
of work environments [65].

A user may design their home environment to facilitate actions like activity
planning, online shopping, interpersonal communication or transmission of
sensitive information (such as medical data) [64]. Each technology set-up can
be extremely unique, akin to a fingerprint, making it difficult to assess the
risks and vulnerabilities relevant to a particular space [66, 64].

Given this variation in system complexity and user activity, each individual
user must assume a degree of responsibility for the continued maintenance and
integrity of their network — and, by extension, for the network overall [52,
67, 68]. Because networks are permeable, typical users may compromise their
own security _and_ the security of others by unwittingly granting system
access to malicious actors (for example, by downloading files without scanning
them [27]) or by failing to detect the presence of bot-nets in a slowly
running system (which can destabilise large swathes of the overall network)
[69].

At the same time, the complexity of the technology environment and the
diversity of online tasks makes it difficult for individual users to protect
their online assets. This, paired with the (perceived) complexity of security
precautions and the sheer variety of security advice and related decisions,
leads individual users to report low confidence in their own decisions and in
their capacity to secure their own domains [67, 70].

###  2.4 How do individuals consume security advice?

There are many ways in which individual users can encounter new security
advice, but most involve some degree of formal and informal learning [13, 18,
71, 72].

Formal learning occurs through structured courses in an online or in-person
classroom environment, usually followed by an assessment [73, 74]. For
example, the delivery of security awareness programmes such as SETA (Security
Education, Training and Awareness)333https://livlab.org/seta/ occurs within
organisations and includes classes that train employees to recognise threats.
These training programmes tend to focus on compliance with corporate policy
[75], and they evoke generic situational awareness [75], rather than providing
specific contexts and situations from which an individual user can learn.

Informal learning is unstructured, occurring outside of formal education
contexts and without direct targeted interaction with security experts [72].
Nevertheless, it is the primary way in which adults learn about the world
around them [76, 77]. As such, it is the main mode of learning considered in
this paper. Informal learning is usually triggered by some internal or
external impetus [76], and it occurs primarily when individuals choose to
actively seek out new ideas and advice.

Thus, for the purposes of this paper, we distinguish _informal learning_ from
_incidental learning_ [76, 78] on the basis of their differing intentionality:
_informal learning_ requires some kind of prior impetus and concerted effort,
whereas _incidental learning_ is often a by-product of carrying out another
task [76]. Despite being a conscious decision, informal learning is often
conducted haphazardly and influenced either by randomised chance [76] or by
the learning behaviours of others [78].

Many of the studies concerned with individual users’ security intentions frame
users’ situational awareness and knowledge as necessary conditions for
appropriate security decisions [18, 79]. Essentially, researchers assume that
individuals must know about the issue at hand before they can make a
reasonable decision. Thus, when an individual is faced with a security message
about a potential threat, their decision process could proceed in one of two
ways.

  1. 1.

First, if they already possess prior knowledge about the threat (and, more
importantly, about how to prevent it), they will take appropriate action. This
is a _threat control process_ [18].

  2. 2.

Second, if they do not possess prior awareness or knowledge, and therefore do
not know how to neutralise the stated threat, the security message may be
rejected. The individual user may instead act to control the psychological
fear generated by the message (rather than the practical threat implied by its
contents) [18, 79].

Individual users may initially accept security advice, but subsequently reject
it if they lack relevant coping strategies and actionable means to counteract
the threat, choosing to deal with the issue in some other way [3, 29, 63].
Arguably, then, the _efficacy potential_ of security advice depends on how
well suited it is to a given individual’s existing frame of reference. This
poses interesting problems for security advice that is disseminated to a broad
audience, as is the case with media-acquired advice. As such, we give
particular consideration to the role of the media.

###  2.5 The efficacy potential of advice

According to self-efficacy theory, individual users pass judgement on their
own ability to cope with a given situation, thus developing self-efficacy
beliefs for a specific domain. Based on these beliefs, individual users are
able to initiate and persevere with behavioural strategies that lead to
successful outcomes [80, 81] Self-efficacy in these cases tends to be a
generative capability that allows individual users to organise their skill-
sets and beliefs, which allows for an efficacy potential for these users [81].

What this means for cyber security advice is that, to enhance the efficacy
potential, researchers must enact strategies which help structure and direct
the behaviour of individual users towards goal setting, and measure the
progress towards this goal [82] and many usable-security studies have
investigated this, e.g. [3, 4, 83]. Furthermore, self-efficacy is closely
linked to motivation, with the level of self-efficacy needing to be higher in
order to correspond to the difficulty of the faced problem [81, 84]. As
already noted, cyber security is seen as both important and complex, yet the
motivation to enhance self-efficacy is limited (as explained in [5] and
explored through psychological and cultural means in [85]).

###  2.6 The role of the media in security advice consumption

There are many possible sources of security advice available to individual
users engaged in informal learning [56], including retailers and vendors of
security software and services [56, 72], online sources with varying levels of
expertise and credibility [72], governmental organisations such as NIST (in
the United States)444https://www.nist.gov/ and the National Cyber Security
Centre (in the United Kingdom)555https://www.ncsc.gov.uk/, professional media
services such as the BBC and the Associated Press, and online media
organisations such as Ars Technica666https://arstechnica.com/, which often
create and distribute security content.

The media and communications field has the greatest reach of all of these
sources. In 2017, Ruoti et al. [86] reported that individual users primarily
learnt about threats through four primary sources: advertisements, news
reports, television dramas, and movies. Subsequently, in 2018, Das et al. [30]
documented that news reports about threats (including cyber threats) were
among the most-shared stories between individuals. Resources such as news
outlets are particularly important for older users, especially when assessing
the severity of threats and the pertinence of advice [87]. Even fictional news
can influence individual decisions about security [88].

Additionally, the media and communications field is uniquely capable of
influencing public opinion _about_ security advice [86]. News sources
facilitate group-based consensus [89] and set the agenda for what is regarded
as an important topic, be it a presidential election [90] or an event such as
the 2017 Wannacry threat [91], which may be accompanied by security advice.
Given this unique influence, we accord special importance to media-acquired
advice in our research. Indeed, we would argue that researchers in the
security advice field have a duty to understand current practices in national
and international media communications.

Media sources are adept at controlling both of these factors, using various
strategies to prime the individual user and make them feel invested in the
given topic — regardless of whether it truly pertains to them. This ‘taste-
making’ function complements their primary advice-creation function. As such,
media sources act as ‘knowledge brokers’ within informal learning contexts,
facilitating the one-way delivery of information, concepts, and ideas from
professional sources to individual users [92, 93] and ultimately influencing
the opinions, actions, and personal development of the recipients of security
advice [94]. Consequently, the work described in this paper addresses both of
these elements, as we endeavour to analyse both how media sources magnify the
risk of certain security threats and their potentially associated mitigating
strategies (through the use of an ontology to compare our results to), and how
this security programming might gradually orient individual users’ perception
of security advice in general over time through a sentiment analysis.

Thus, at this point, it is worth reiterating the research objectives of
Section 1:

  * 1.

RO1: What kind of informally learnt and actionable security advice most often
appears in news articles?

  * 2.

RO2: What is the efficacy potential of this security advice as consumed by an
individual user?

##  3 Methodology

In this section we discuss the two elements that we utilised to obtain the
necessary data. The first element was a news-scraper, which was developed in
Python and was designed to extract complete articles from structured data
sources. The second element was a viable search methodology, which was
assembled from multiple components. We first give consideration to the news-
scraper.

###  3.1 The news-scraper

Web scraping is a technique that allows researchers to automate the capture of
online information. Scrapers are popular tools for digital research, and they
are often characterised as ‘outsider’ tools that can be used with freely
available online data — that is, data that does not require privileged access
[95]. To ensure that we had enough information to answer our research
objectives, we designed our news-scraper to collect as much data as possible
from our news sources. The tool’s basic functional requirements are shown in
Table 1; this gave rise to the abstract architecture depicted in Figure 1.

Table 1: Basic functional requirements for the news-scraper. Requirement |  Detail
---|---
1 |  The ability to systematically search for news articles within a set time frame utilising pre-set search queries.
2 |  The ability to extract the full content from news articles.
3 |  The ability to extract metadata, including publication date, author(s), titles, source names, and country of origin, for further analysis.

We utilised a news-aggregation API to filter content from a variety of
unstructured and structured news sources were consistent with our definition,
and we added functions to enable the complete capture of content, in
accordance with Requirements 2 and 3.

The captured data was then fed into a data-storage pipeline before being
converted into a flat-file database storage solution. Incoming data was merged
with existing records when required to avoid duplicate data.

###  3.2 The search terms

To fulfill Requirement 1, we followed the precedent of Schatz et al. [96], who
sought to derive a more precise definition of security by utilising Google
Trends to automatically collect the phrases that individuals were using to
search for security content. As this had to be accomplished from the
perspective of our individual user, this excluded the possibility of
replicating the work of Humayun et al. [97] who looked at primary studies
undertaken within academia. Instead, we followed the Systematic Mapping Study
protocol of Kosar et al. [98].

We defined a set of search and inclusion/exclusion criteria (for example,
Cybersecurity OR Cyber AND Security) and additional queries containing both
base search terms and queries derived from Google Trends (online OR advice OR
protection OR protect OR prevent OR preventative OR tips OR email OR social
network OR password OR hack OR hacked OR hacking). All search terms were
technology-agnostic — they did not include explicit references to products or
services.

We augmented the Google Trends queries with phrases pertaining to 20 cyber
security events that (1) had occurred in the previous 24 months and (2) had
been covered by at least 10 major English-language news outlets. The news-
scraper then carried out searches over a 24-month time span and returned all
results that included these terms within the title or body of the content.
Therefore, while not exhaustive, our corpus represents security advice as
accurately as possible within the confines of our scope.

Figure 1: An overview of the news-scraper tool.

###  3.3 Cleaning the data

First, we screened our results according to the inclusion/exclusion criteria.
These were defined as follows.

  * 1.

Must be a news or blog article that directly addresses at least one aspect of
cyber security/contain our search terminology directly. Blog articles were
limited to tutorials, editorials, tool demonstrations and discussion of
technical reports.

  * 2.

Must be written in English (due to the nature of our analysis methodology).

  * 3.

Must be accessible, and not hidden behind a paywall or other kind of lockout
mechanism (as in these cases only a few lines of text may have been
retrieved).

Any article found to be in breach of these criteria was excluded.

In this way, we reduced the initial pool of 16,876 usable articles from our
first cleaning process to 15,422 individual articles. For the remaining
articles our focus and technique were informed by recent work, such as that of
Satyapanich et al. [99], which describes the process for extracting semantic
information (such as people, places, and events) from security articles, and
that of Al Moubayed et al. [100], who used Bayesian topic modelling to ascribe
classifications to, and uncover trends in, security and criminal documents. We
prepared the corpus for analysis using common data pre-processing techniques.
We utilised tokenisation to break down the text, first into sentence units and
then into individual words. We then replaced uppercase text with lowercase
equivalents and removed punctuation. We lemmatised the corpus to standardise
the tense and to replace any third-person words with first-person variants.
Finally, we used a stemming technique to reduce words to their root form,
where appropriate [101].

###  3.4 Classifying the data

As we saw in Section 2.4, individuals require actionable elements within their
security advice. We therefore utilised an ontological framework to help us
classify and integrate the data collected from the sources queried by our
news-scraper. We chose our ontology according to our idea of efficacy
potential.

We required an ontology which, while evaluative in nature, could demonstrate
effectively that a particular level of security has been achieved. Therefore,
we conducted a non-exhaustive search for an ontology through work such as
ENISA’s IoT Security Standards Gap
Analysis777https://www.enisa.europa.eu/publications/
iot-security-standards-gap-analysis and a report by the UK’s Department for
Digital, Culture, Media and Sport, mapping security recommendations for
various audiences888https://www.gov.uk/government/publications/mapping-of-iot-
security-
recommendations-guidance-and-standards. We additionally looked at including
only those ontologies that have received frequent updates to include concepts
recent to our individual user such as edge computing, e.g. [102]. We found
that many ontologies were aimed at a technical or policy audienc and often
included several layers of abstraction within the work, used vaguely defined
terminology or simply did not include our original requirement of actionable
security advice.999For example, high-level policies for IT professionals and
business executives were considered to be out of scope.

Figure 2: An overview of the CIS-vectors ontological framework.

An ontology by the Center for Internet Security
(CIS)101010https://learn.cisecurity.org/cis-controls-download was chosen, as
we found it provided us with actionable security advice that can aid towards
self-efficacy. Although it was not intended for individual users, this
ontology does prioritise risk-based security and focuses on the practical
mitigation of these risks, and it accomplishes this by identifying and
utilising 20 domain-specific CIS-vectors that represent practical and
actionable remedies for security threats. Furthermore the CIS ontology has
been aligned with other ontological frameworks such as that of NIST to allow
for easier adoption by organisations and
projects.111111https://www.cisecurity.org/blog/v7-1-introduces-implementation-
groups-
cis-CIS-vectors/ A high-level version of the framework can be seen in Figure
2. The individual CIS-vectors are discussed in detail in Section 4.

The CIS ontological framework provides us with the requisite entity types and
properties which are ascribed to individual news articles within the corpus as
additional metadata. Thus, we are able to use the ontology to define the
entities, relations, and other factors that can be extracted from the corpus.
The ontology also serves to focus the corpus and to restrict our vision to the
research objectives, as the language utilised within security can range from
extremely specific to extremely ambiguous [103]. In many cases, this range can
make it difficult to apply an ontology to specific news articles within the
corpus.

###  3.5 Additional work to encompass null values from CIS-vectors

Given our search terminology, we observed that 6,134 of the 15,422 articles
(representing 36.3% of the total corpus) contained references to any of our
CIS-vectors. We performed a second pass on the corpus, introducing additional
syntactic variants of the terminology utilised within the CIS-vectors. For
example, we separated ‘malware defenses’ into ‘malware AND defences’, ‘malware
defence’, and ‘malware defense’ to correct for localisation issues.

The results of the second pass are illustrated in Figure 4.1. As can be seen,
the occurrence rate was subsequently 7,988 articles, or 51.7% of the corpus.
Each of these articles contained references to one or more CIS-vectors. For
the remaining 48.3% of the corpus, we performed a Latent Dirichlet Allocation
(LDA) analysis of these articles in order to generate further details, the
results of which are outlined in Section 4.2. LDA is a statistical modeling
tool that allows for the discovery of otherwise abstract topics within text
files. It provides us with both a topic-per-word and topic-per-document model.
To ensure the accurate selection of topic numbers and models, we followed the
methodologies proposed by Cao et al. [104] and Deveaud et al. [105].

##  4 Results

Figure 3: Articles published per day between January 2015 and December 2020.

In this section we give consideration to our results. First, we corroborate
the findings of Alagheband et al. [106], which indicated that coverage of
security topics in the New York Times has steadily increased over the last
decade. Figure 3 highlights this increase over time: the “vast terra incognita
of print” [107]. The data also exposed the sheer diversity of publishers,
ranging from traditional outlets such as the BBC news and CNN through to
specialty security blogs. Even so, we must acknowledge that this list is
inevitably incomplete, as our search methodology, while extensive, was non-
exhaustive, and it was limited to English-language media.

Next, we identify the prevalence and features of ‘ideal’ news articles in our
corpus and use this information to help answer our research objectives. An
ideal news article must contain a summary of the information that an
individual user requires (in this case, regarding security advice),
eliminating irrelevant and redundant information wherever possible [108]. To
determine the prevalence of such articles in our corpus, we first utilised our
CIS-vectors to ascertain how many of the articles contain content-specific
vocabulary that users may expect to find within these articles, and we
performed additional analysis on those articles that contained no such terms.
We then derived statistics pertaining to sentence length and vocabulary size,
which we then compared to third-party corpora (where available). Finally, we
utilised sentiment analysis as an efficacy potential measurement tool,
building on work by Kalra and Prasad [109], who used it for stock market
assessments. This was done to decipher any trends that could inform our
efficacy potential research question.

###  4.1 CIS-vector occurrences

Figure 4: The occurrence rate of CIS-vectors. Null values are excluded.

Figure 4 highlights the occurrences of our CIS-vectors in the corpus. The
most-used CIS-vectors were CIS-13 (Data protection), CIS-11 (Limitation and
control of network ports, protocols, and services), and CIS-2 (Inventory and
control of software assets). CIS-13 highlights the growing trend towards data
protection awareness and its relevance for individual users; it occurred 0.4
times per article, on average.

Delving deeper into the reasons for this expanding data protection coverage,
we find that, between 2018 and 2019, the most significant topics were related
to data breaches, data protection guidelines for individuals and organisations
(such as the EU’s General Data Protection Regulation (GDPR)121212https://gdpr-
info.eu/), and data privacy-related security advice for social media users. In
2020 there was a shift towards protecting health-related data in medical
contexts, with advice and threat messaging geared towards disease contact and
exposure tracing applications, such as those mentioned by Yasaka et al. [110].

Figure 5: A correlation plot, highlighting in particular the strong
correlation between CIS-4 and CIS-16.

CIS-11 indicates network security-related information and advice, and its
occurrence rate increased significantly between 2019 and the end of 2020. At
least one publication ([111]) notes a similar increase in interest. Again, we
found that most of this network security advice was related to privacy, and it
appeared in texts ranging from technical articles to installation guides for
the Tor Project. In many cases, these articles contained more difficult
vocabulary and technical terminology than the average publication.

CIS-2 pertains to software assets and their associated CIS-vectors, and it
proved to be one of the most diffuse topic. In our corpus, we found articles
linked to Internet of Things home security, smart grid and connected vehicle
software, and security issues that arise in connection with these devices and
services.

Correlations between the CIS-vectors are depicted in Figure 5. The
correlations were weak across the corpus, with one notable exception: the
correlation between CIS-16 (Account monitoring and control) and CIS-4
(Controlled use of administrative privileges). Though CIS-16 appeared more
frequently overall, tokens associated with both vectors appeared consistently
between articles.

###  4.2 Articles containing no CIS-vectors

Table 2 lists the most common topics that occurred in those articles that
featured no CIS-vectors from our classification (representing 48.2% of the
corpus). The topics were derived through LDA topic modelling, as described in
Section 3.5.

Table 2: The five most common topics in the non-CIS articles. Topic 3 | Topic 6 | Topic 9 | Topic 11 | Topic 14
---|---|---|---|---
security | safety | cyber | trump | police
internet | health | security | president | crime
system | recovery | attacks | election | cases
users | covid-19 | business | russia | issue
data | protection | threats | u.s. | cyber

We can see that, despite the absence of CIS-vectors, security is still a focal
point in these articles. In these cases, though, the focus is on national
(cyber) security (Topic 11), cyber crime (Topic 14), business threats (Topic
9), and health and safety issues related to cyber crime and security (Topic
6). Topic 3 embodies similar concepts as CIS-vectors CIS-13 (Data protection),
CIS-11 (Limitation and control of network ports, protocols, and services), and
CIS-2 (Inventory and control of software assets).

###  4.3 Sentence length and vocabulary size

We use sentence length, vocabulary size, and a selection of readability scores
as proxies for difficulty.

####  4.3.1 Sentence length

Figure 6: The distribution of sentences per article.

Sentence length is an often-utilised tool in the discovery of readability
within corpora [108, 112]. Figure 6 displays the average article length. The
mean article length was 9.92 sentences, and the median length was 10
sentences. We can compare to the work of Goldstein et al. [108] on the
automated summarisation of news articles, which led to a corpus of 1,000
Reuters articles with a (post-summarisation) average length of 23 sentences.
We can also compare this to the work of Lim et al. [112], whose smaller corpus
yielded an average of 14 sentences per article.

The gap between the publication of these comparators (1999 and 2018,
respectively) may suggest an overall decline in the length of news articles.
It also suggests that our corpus of security-specific news is on the shorter
side of the spectrum. This last point is, however, caveated by the fact that a
comparison with a more historical data and a wider potential variety of
possible sources would be needed to further confirm this finding.

####  4.3.2 Vocabulary

We estimated the vocabulary growth of the corpus using Heaps’ law [113], which
describes the relationship between tokens and types. This law states that a
vocabulary, expressed as $v$ unique word types, is proportional to the power
law of $n$, the number of tokens in an arbitrary text. The relation is
expressed as

$v=Kn^{\beta}$

Here, $K$ is a positive constant and $\beta$ lies between 0 and 1. In effect,
as a body of text increases, the potential to discover new distinct word types
decreases. In our corpus, we can see from Figure 7 that the vocabulary range
largely adheres to the predicted value (black line). This means that new
vocabulary terms are continually arising in the data, which could complicate
users’ informal learning.

Figure 7: A visualisation of Heaps’ law.

###  4.4 Readability scores

A readability index, such as the ones shown in Table 3, is an estimation of
how difficult a text is to read. In online environments, it is often measured
to assess click-through rates and user satisfaction [114]. Grinberg [115]
utilised it, alongside sentence length, to model user engagement with news
articles. As such, it is an interesting variable to consider when assessing
the efficacy potential of the texts in our corpus.

Table 3: A selection of readability metrics. Metric | Score
---|---
Flesch–Kincaid | 12.51631
Gunning–Fog Index | 16.02665
Coleman–Liau Index | 13.23249
SMOG | 14.55265
Automated Readability Index | 13
Average grade level | 13.86562

Readability is determined by measuring a text’s complexity, which is
approximated via quantifiable attributes such as word length, sentence length,
syllable count, and so on. The Flesch–Kincaid test [116] is one of the most
utilised readability tests, and it calculates readability by (1) dividing the
number of utilised words by the number of sentences and (2) dividing the
average number of syllables per word by the number of utilised words. The
scoring range starts at 100 for the easiest to read and descends to 0 for
unreadable texts. As an example, the combined Harry Potter novels have a score
of 72.83. Other frequently used systems include the Gunning–Fog Index [117],
which looks at sentence length and number of polysyllabic words; the
Coleman–Liau Index [118], which does not assess syllables; the Automated
Readability Index [119]; and the Simple Measure of Gobbledygook (or SMOG)
[120], which utilises a similar methodology as the Flesch–Kincaid, but from
sections within the text. All of these metrics utilise a 100–0 scoring system
and are broadly comparable with each another. As such, we employ all of them
in this study.

Table 3 highlights a selection of readability scores, all utilising the same
100–0 scoring scale. In Figure 8, we see the distribution of these scores. Our
p-value was less than the standard alpha value (set at 0.05). As we can see in
Figure 8, there is no significant deviation in scores across the distribution
of articles, which also indicates that the scores are statistically
significant. The scores suggest a high degree of requisite prior knowledge in
order to gain the most from reading the articles.

Figure 8: The distributions of our readability metrics.

###  4.5 Sentiment analysis

Figure 9: A visualisation of changing sentiment, depicting a slight increase
in negative sentiment along with a corresponding increase in article
generation.

Sentiment analysis is a group of text analysis techniques that allow for the
automatic derivation of sentiment (positive or negative) from large data-sets
[121]. Sentiment analysis is widely used across domains, from marketing [121]
to stock market analysis [109]. Previous sentiment analysis work within the
security field has focused on predicting cyber attacks or identifying
potential perpetrators, for example, by assessing sentiment in online hacker
forums [122]. The lexicons generated from these studies (for example, those
pertaining to sentiment within political analysis of sovereign cyber
capabilities) are of limited use within our work, as their terminology often
differs substantially from what could be construed as ‘security advice’ based
on our definition. As such, we utilised Latent Semantic Scaling (LSS), which
is a semi-supervised technique for scaling documents based on work by
Deerwester et al. [123]. It allows for a limited set of pre-generated seed
words, which are words embedded with a specific positive or negative value. To
produce our small library of seed words, we utilized the SENTPROP [124]
framework. We chose this framework as it combines word-vector embeddings with
a label propagation approach, which are well-known techniques to generate
seed-word libraries. Additionally, SENTPROP can generate accurate results with
smaller corpora. [124]. In our system, the overall sentiment of a news article
is correlated with the sentiments of individual words within that article,
thereby allowing for a sentiment polarity check.

The results of this sentiment analysis process can be seen in Figure 9. The
scores suggest an overall decrease in positive sentiment over the time period;
however, these results are not statistically significant, likely because (1)
the increase in published articles over the time period distorted the results
and (2) a high p-value deviated significantly across the standard alpha value
(set at 0.05).

##  5 Discussion

We now consider our results in the context of the research objectives of
Section 1. We focus on the results derived from our CIS framework and
associated vectors in Section 5.1, and ascertain how the readability,
vocabulary and sentiment of the corpus affects its efficacy potential in
Section 5.2.

###  5.1 What kind of informally learnt and actionable security advice most
often appears in news articles?

Three overarching themes prevail in our security corpus. The first is _data
protection_ (Theme 1), which is reflected in the strong focus on CIS-13 (Data
protection) and Topic 3 of our LDA analysis. The second is _physical and
digital security_ (Theme 2), which is supported by CIS-11 (Limitation and
control of network ports, protocols, and services), CIS-2 (Inventory and
control of software assets), and Topic 3 of the LDA analysis. The third is
_personal and collective safety_ (Theme 3) in the face of personal, business,
or sovereign threats to one’s security, which is supported by Topics 3, 11 and
14.

Table 4: An overview of the themes and supporting evidence. Theme | Supporting evidence
---|---
1\. Data protection | CIS-13, Topic 3
2\. Cyber-physical systems security | CIS-2, CIS-11, Topic 3
3\. Personal and collective safety | Topic 3, Topic 11, Topic 14

All of these themes represent a unique set of constructs and associated user
behaviours. For Themes 1 and 3, a significant driver for personal safety is
privacy: “the right of a party to maintain control over, and confidentiality
of, information about itself” [125]. Although privacy is a significant token
by itself (appearing 4,887 times in the corpus), further indirect references
to it suggest that it is the underlying motivation for a significant number of
data protection-related articles, be they in the realm of health data,
shopping data, or, more broadly, associated with the GDPR.

In Theme 2, personal safety entails the need for user intervention in faulty
systems, either because the system cannot determine the cause of a certain
threat or the appropriate corrective action to take, or, in some cases,
because the system itself is acting maliciously towards the user. These
articles were the most likely to contain directly actionable security advice,
and thus were the most efficacious for individual users.

Theme 3 also encapsulates threats to business and sovereignty. These articles
are unlikely to contain actionable security advice, but they can aid in the
creation of policy [126], which may then lead to actionable advice. These
articles may even influence public opinion regarding the (cyber) security of
national sovereignty, much like how terrorism news shaped national opinion and
policy, as seen in work by Gadarian [127]. This cycle of influence leads to
the creation of policies and legislation, such as the aforementioned GDPR,
which in turn influences public awareness of potential data security threats,
ultimately stimulating new forms of cyber offense and defensive capabilities.
These capabilities are then disseminated to individual users, potentially as a
form of security advice. Assessing future developments within these themes and
re-assessing their relevance periodically could provide a lens for evaluating
the past, current, and future impact of news media on security advice efficacy
potential.

###  5.2 What is the efficacy potential of this security advice as consumed by
an individual user?

Many of the articles an individual user may access for cyber security advice
may contain subject-specific vocabulary (such as that found within our
ontological framework). Given that (1) there is limited overlap between advice
sets within our ontological framework and (2) the average length of the
articles in our corpus (expressed as sentence length) is shorter than the
average length of comparator articles (see Section 4.3), there appears to be a
certain level of focus within the articles that could indicate efficacy
potential. However, we have also seen from ontological frameworks such as the
CIS-Control schema that these tools may not encompass all of the possible
security vectors within the current media environment. Furthermore, these
results must be qualified given our topic modelling methodology. Our
application of Heaps’ law highlights the growing vocabulary within our corpus,
demonstrating that the subject-specific terminology in news articles on
security advice is continuously evolving. This may point to an increasingly
diversified interest in security advice that is tailored to a specific,
predetermined goal. This encourages us to question the efficacy potential of
all-encompassing frameworks such as the CIS-Control schema.

The results of our readability tests and sentiment analysis may further
challenge the efficacy potential of current media-mediated security
dissemination. We find within our corpus a trend towards high reading
difficulty levels: ease of reading correlated with publication type, and news
articles ranked higher on all readability indices. As all five of our
assessment metrics reported statistically significant results with similar
distribution scores (see Table 3), we can confidently assert that just 3% of
our corpus was written at a U.S. school system 6th-grade level, which is
typically the recommended reading level for standard distributed materials
[128]. Most of the articles in this corpus require a reading level of a
typical college undergraduate.

Recalling that an individual user must have (1) a sense of certainty about the
content, (2) a personal interest in the content, and (3) sufficient ability to
deploy the content in order to feel sufficiently compelled to act on the
information, this threat control process could easily be derailed by the
continued divergence and growth of subject-specific vocabulary and dense
prose. Haney and Lutters [71] argue that there is a rejection threshold that
informs the maintenance of security in a rapidly evolving landscape, and they
maintain that individual users are approaching this threshold.

Security is not the only specialised field that deals with these dissemination
issues, and it may be helpful to observe the solutions pursued in other
contexts. For example, medical advice dissemination to the general public
(taken here as the equivalent of our ‘individual user’) also involves
communicating complicated concepts and extensive vocabulary to individuals who
have no relevant formal training on the subject. Britt et al. [129] found that
many readers stop reading medical texts if they gauge significant difficulty
within the first few sentences. Consequently, the American Medical Association
(AMA) and the U.S. Department of Health and Human Services (USDHHS) have set
explicit guidelines that require public-facing information to achieve a
U.S.-standardised readability level of 6th grade or below [128]. Extrapolating
these considerations to our own corpus, it would stand to reason that
increasing readability to a more generally accessible level could constitute a
cost-effective remedy.

Although the overall sentiment of the corpus would not suggest that users may
be being treated as an enemy (as, for example, was documented in Adams and
Sasse’s seminal 1999 paper [130]), it does appear that what we encountered
would not fulfil Kerckhoffs’ criterion for ease of use. Neither would we agree
that cyber security advice as portrayed in our corpus allows for self-efficacy
upon reading. Instead, an individual user must face security topics using a
multi-pronged approach, whereby self-efficacy is derived from multiple sources
of increasing complexity. If the cyber security field is to continue down the
path of increased specialisation, perhaps the time has come to recognise this
emerging reality and clarify — in a transparent fashion — the expectations
that are being placed on users.

##  6 Limitations and future work

The scope of this study was limited by the type and amount of information we
were able to acquire to build the corpus. In our case, this meant focusing on
English-language material, even though a preliminary search conducted before
implementation unearthed a rich catalogue of data in other languages. This
also means that our security topics, analysis, and findings likely exhibit
Anglo-Saxon bias. The technical tools utilised for the readability scores were
also designed for English-language articles. There is significant scope for
the enhancement of our search methodology, where for example users may only
utilise the first page of any search enquiry [131]. It is our hope that this
methodology be utilised to answer the same research objectives in other
languages and cultural contexts. Finally, although the CIS ontology proved
adequate for the scope of this project, the primary focus of CIS is
technologists. Although our search for a suitable ontology was non-exhaustive,
we believe the inclusion of CIS serves to highlight the need for further
development in user-focused cyber security ontologies which may serve as a
better basis from which to base a study such as ours.

We utilised automated methodologies in order to classify topics and measure
sentiment and reading difficulty, and the results are tempered by the
respective limitations of these methodologies. Moreover, our results represent
a specific snapshot in the security timeline; access to a larger historical
data-set would inevitably change the overall results, potentially yielding a
more statistically significant sentiment analysis.

Our approach to tackling the second research objective may limit the
usefulness of our conclusions. We approximated article efficacy potential by
using text analysis to predict user engagement, and we did not consider other
metrics that could have enhanced the findings. Traditionally speaking,
reading-difficulty assessments in laboratory settings involve comprehension
tests, eye tracking, and brain-imaging. Knowledge of how users interact with
our corpus in these terms would allow for a significantly richer analysis of
security advice efficacy potential.

The aforementioned limitations can, of course, be addressed in future research
that builds upon what is presented here — not least because our research
method (described in Section 3) allows for continuous data capture.
Furthermore, the data within this corpus could serve as the foundation for
further analysis of security advice dissemination. Because this corpus
contains a significant variety of sources, structural analysis of sentence
construction for threat messaging could reveal the rhetorical structure of
fear appeals, as per previous work in the field such as that of Renaud and
Dupuis [18]. A fear appeal is designed to motivate the reader to execute
security advice, and an in-depth analysis of its features could yield results
that would improve the efficacy potential of security advice dissemination.

The corpus itself could be augmented with social media data, which would add
the significant vector of digital naïve advice [58]. Bias within the articles
could be used as another indicator of efficacy potential via methods like that
presented by Lim et al. [112]. We believe that the results of this study can
provide a basis for further reflection on security advice dissemination, and
that it can stimulate a conversation about individual users’ learning
environment. Importantly, we hope that it serves as a point of departure for
future studies.

##  7 Conclusion

We have presented work on a corpus of security advice generated from
mainstream news articles as might be faced by individual users on a regular
basis. The work was oriented by two questions: (1) What kind of informally
learnt and actionable security advice most often appears in news articles? and
(2) What is the efficacy potential of this security advice as consumed by an
individual user?

We found that news-mediated security advice has been increasing since 2018,
and that many such news articles focus on specific security topics. This level
of focus may indicate efficacy potential. Additionally, we found that news-
mediated security advice is characterised by short article length and low
readability, making it difficult for many individual users to comprehend its
content. We found that the subject-specific terminology within our security
news articles is continuously evolving, potentially indicating increasingly
diversified interest in goal-specific security advice. Again, this may
increase the relative difficulty of acquiring and comprehending news-mediated
security advice, with an associated impact on efficacy potential. Our approach
involved using quantitative methods to yield qualitative findings. Our hope is
that this research can help lay the foundations for various means of
quantifying and improving the efficacy potential of security advice
dissemination.

## References

  * [1] M. Theofanos, Is usable security an oxymoron?, IEEE Computer 53 (2) (2020) 71–74.
URL https://doi.org/10.1109/MC.2019.2954075

  * [2] La cryptographie militaire., Journal des Sciences Militaires IX (1883) 5–38.
  * [3] E. M. Redmiles, S. Kross, M. L. Mazurek, How I learned to be secure: A census-representative survey of security advice sources and behavior, in: Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, CCS ’16, Association for Computing Machinery, New York, NY, USA, 2016, pp. 666–677.  doi:10.1145/2976749.2978307.
URL https://doi.org/10.1145/2976749.2978307

  * [4] S. L. Pfleeger, M. A. Sasse, A. Furnham, From weakest link to security hero: Transforming staff security behavior, Journal of Homeland Security and Emergency Management 11 (4) (2014) 489–510.  doi:10.1515/jhsem-2014-0035.
URL https://doi.org/10.1515/jhsem-2014-0035

  * [5] C. E. Herley, So long, and no thanks for the externalities: The rational rejection of security advice by users, in: Proceedings of the 2009 Workshop on New Security Paradigms Workshop, NSPW ’09, Association for Computing Machinery, New York, NY, USA, 2009, pp. 133–144.  doi:10.1145/1719030.1719050.
URL https://doi.org/10.1145/1719030.1719050

  * [6] M. N. Al-Mhiqani, R. Ahmad, W. Yassin, A. Hassan, Z. Z. Abidin, N. S. Ali, K. H. Abdulkareem, Cyber-security incidents: a review cases in cyber-physical systems, International Journal of Advanced Computer Science and Applications 9 (1) (2018) 499–508.  doi:10.14569/IJACSA.2018.090169.
URL https://doi.org/10.14569/IJACSA.2018.090169

  * [7] E. Bertino, N. Islam, Botnets and internet of things security, IEEE Computer 50 (2) (2017) 76–79.  doi:10.1109/MC.2017.62.
URL https://doi.org/10.1109/MC.2017.62

  * [8] H. N. Viet, Q. N. Van, L. L. T. Trang, N. Shone, Using deep learning model for network scanning detection, in: Proceedings of the 4th International Conference on Frontiers of Educational Technologies, 2018, pp. 117–121.  doi:10.1145/3233347.3233379.
URL https://doi.org/10.1145/3233347.3233379

  * [9] K. Sowndarajan, S. Binu, Android security issues and solutions, in: 2017 International Conference on Innovative Mechanisms for Industry Applications (ICIMIA), IEEE, 2017, pp. 686–689.  doi:10.1109/ICIMIA.2017.7975551.
URL https://doi.org/10.1109/ICIMIA.2017.7975551

  * [10] C. E. Herley, B. W. Keogh, A. M. Hulett, A. M. Marinescu, J. S. Williams, S. Nurilov, US patent 9,021,590: Spyware detection mechanism (April 28 2015).
  * [11] C. Khoo, K. Robertson, R. Deibert, Installing fear: A Canadian legal and policy analysis of using, developing, and selling smartphone spyware and stalkerware applications, University of Toronto Citizen Lab Report (120) (2019).
  * [12] D. Wang, Z. Zhang, P. Wang, J. Yan, X. Huang, Targeted online password guessing: An underestimated threat, in: Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, 2016, pp. 1242–1254.  doi:10.1145/2976749.2978339.
URL https://doi.org/10.1145/2976749.2978339

  * [13] E. Rader, R. Wash, B. Brooks, Stories as informal lessons about security, in: Proceedings of the Eighth Symposium on Usable Privacy and Security, ACM, 2012, p. 6.  doi:10.1145/2335356.2335364.
URL https://doi.org/10.1145/2335356.2335364

  * [14] L. Brandimarte, A. Acquisti, G. Loewenstein, Misplaced confidences: Privacy and the control paradox, Social Psychological and Personality Science 4 (3) (2013) 340–347.  doi:10.1177/1948550612455931.
URL https://doi.org/10.1177/1948550612455931

  * [15] S. B. Barnes, A privacy paradox: Social networking in the United States, First Monday 11 (9) (2006).  doi:10.5210/fm.v11i9.1394.
URL https://doi.org/10.5210/fm.v11i9.1394

  * [16] K. Thakur, T. Hayajneh, J. Tseng, Cyber-security in social media: Challenges and the way forward, IT Professional 21 (2) (2019) 41–49.
URL https://doi.org/10.1109/MITP.2018.2881373

  * [17] S. Furnell, K.-L. Thomson, Recognising and addressing ‘security fatigue’, Computer Fraud & Security 2009 (11) (2009) 7–11.  doi:10.1016/S1361-3723(09)70139-3.
URL https://doi.org/10.1016/S1361-3723(09)70139-3

  * [18] K. Renaud, M. Dupuis, Cyber-security fear appeals: Unexpectedly complicated, in: Proceedings of the New Security Paradigms Workshop, 2019, pp. 42–56.  doi:10.1145/3368860.3368864.
URL https://doi.org/10.1145/3368860.3368864

  * [19] M. Ottaviani, P. N. Sørensen, Professional advice, Journal of Economic Theory 126 (1) (2006) 120–142.
  * [20] S. Bonaccio, R. S. Dalal, Advice taking and decision-making: An integrative literature review, and implications for the organizational sciences, Organizational Behavior and Human Decision Processes 101 (2) (2006) 127–151.  doi:10.1016/j.obhdp.2006.07.001.
URL https://doi.org/10.1016/j.obhdp.2006.07.001

  * [21] B. K. Wiederhold, The role of psychology in enhancing cyber-security, Cyberpsychology, Behavior, and Social Networking 17 (3) (2014) 131–132.  doi:10.1089/cyber.2014.1502.
URL https://doi.org/10.1089/cyber.2014.1502

  * [22] R. C. Dreibelbis, J. Martin, M. D. Coovert, D. W. Dorsey, The looming cyber-security crisis and what it means for the practice of industrial and organizational psychology, Industrial and Organizational Psychology 11 (2) (2018) 346–365.  doi:10.1017/iop.2018.3.
URL https://doi.org/10.1017/iop.2018.3

  * [23] S. Yuan, A. Fernando, D. C. Klonoff, Standards for medical device cyber-security in 2018 (2018).  doi:10.1177/1932296818763634.
URL https://doi.org/10.1177/1932296818763634

  * [24] B. Çelen, S. Kariv, A. Schotter, An experimental test of advice and social learning, Management Science 56 (10) (2010) 1687–1701.  doi:10.1287/mnsc.1100.1228.
URL https://doi.org/10.1287/mnsc.1100.1228

  * [25] S. T. Lawson, S. K. Yeo, H. Yu, E. Greene, The cyber-doom effect: The impact of fear appeals in the us cyber-security debate, in: 2016 8th International Conference on Cyber Conflict (CyCon), IEEE, 2016, pp. 65–80.
  * [26] M. Bada, A. M. Sasse, J. R. C. Nurse, Cyber-security awareness campaigns: Why do they fail to change behaviour?, arXiv preprint arXiv:1901.02672 (2019).
  * [27] W. Fan, K. H. Yeung, Online social networks—paradise of computer viruses, Physica A: Statistical Mechanics and its Applications 390 (2) (2011) 189–197.  doi:10.1016/j.physa.2010.09.034.
URL https://doi.org/10.1016/j.physa.2010.09.034

  * [28] E. M. Redmiles, A. R. Malone, M. L. Mazurek, I think they’re trying to tell me something: Advice sources and selection for digital security, in: 2016 IEEE Symposium on Security and Privacy (SP), IEEE, 2016, pp. 272–288.
  * [29] I. Ion, R. Reeder, S. Consolvo, “… no one can hack my mind”: Comparing expert and non-expert security practices, in: Eleventh Symposium On Usable Privacy and Security (SOUPS), 2015, pp. 327–346.
  * [30] S. Das, J. Lo, L. Dabbish, J. I. Hong, Breaking! A typology of security and privacy news and how it’s shared, in: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, 2018, pp. 1–12, paper No. 1.  doi:10.1145/3173574.3173575.
URL https://doi.org/10.1145/3173574.3173575

  * [31] M. Abomhara, G. Køien, Cyber-security and the internet of things: vulnerabilities, threats, intruders and attacks, Journal of Cyber-Security and Mobility 4 (1) (2015) 65–88.  doi:10.13052/jcsm2245-1439.414.
URL https://doi.org/10.13052/jcsm2245-1439.414

  * [32] J. Tregear, Risk assessment, Information Security Technical Report 3 (6) (2001) 19–27.  doi:10.1016/S1363-4127(01)00304-1.
URL https://doi.org/10.1016/S1363-4127(01)00304-1

  * [33] B. Cashell, W. D. Jackson, M. Jickling, B. Webel, The economic impact of cyber-attacks, Congressional Research Service Documents, CRS RL32331 (Washington DC) (2004) 2.
  * [34] J. Jang-Jaccard, S. Nepal, A survey of emerging threats in cyber-security, Journal of Computer and System Sciences 80 (5) (2014) 973–993.  doi:10.1016/j.jcss.2014.02.005.
URL https://doi.org/10.1016/j.jcss.2014.02.005

  * [35] N. Wagner, C. Ş. Şahin, J. Pena, W. W. Streilein, Automatic generation of cyber architectures optimized for security, cost, and mission performance: A nature-inspired approach, in: Advances in Nature-Inspired Computing and Applications, Springer, 2019, pp. 1–25.
  * [36] A. D. Smith, Cybercriminal impacts on online business and consumer confidence, Online Information Review 28 (3) (2004) 224–234.  doi:10.1108/14684520410543670.
URL https://doi.org/10.1108/14684520410543670

  * [37] H. Chen, R. H. L. Chiang, V. C. Storey, Business intelligence and analytics: From big data to big impact, MIS Quarterly (2012) 1165–1188.
URL https://doi.org/10.2307/41703503

  * [38] Y. Wang, Y. Wang, J. Liu, Z. Huang, A network gene-based framework for detecting advanced persistent threats, in: 2014 Ninth International Conference on P2P, Parallel, Grid, Cloud and Internet Computing, IEEE, 2014, pp. 97–102.  doi:10.1109/3PGCIC.2014.41.
URL https://doi.org/10.1109/3PGCIC.2014.41

  * [39] P. Casas, F. Soro, J. Vanerio, G. Settanni, A. D’Alconzo, Network security and anomaly detection with big-dama, a big data analytics framework, in: 2017 IEEE 6th International Conference on Cloud Networking (CloudNet), IEEE, 2017, pp. 1–7.  doi:10.1109/CloudNet.2017.8071525.
URL https://doi.org/10.1109/CloudNet.2017.8071525

  * [40] R. von Solms, J. van Niekerk, From information security to cyber-security, Computers & Security 38 (2013) 97–102.  doi:10.1016/j.cose.2013.04.004.
URL https://doi.org/10.1016/j.cose.2013.04.004

  * [41] B. D. Weinstein, What is an expert?, Theoretical medicine 14 (1) (1993) 57–73.
  * [42] T. Caldwell, Plugging the cyber-security skills gap, Computer Fraud & Security 2013 (7) (2013) 5–10.
  * [43] J.-y. Park, An analysis on training curriculum for educating information security experts, Management & Information Systems Review 31 (1) (2012) 149–165.
  * [44] S. Miller, C. Wagner, U. Aickelin, J. M. Garibaldi, Modelling cyber-security experts’ decision making processes using aggregation operators, computers & security 62 (2016) 229–245.
  * [45] A. Šorgo, T. Bartol, D. Dolničar, B. Boh Podgornik, Attributes of digital natives as predictors of information literacy in higher education, British Journal of Educational Technology 48 (3) (2017) 749–767.
  * [46] T. Li, G. Convertino, R. K. Tayi, S. Kazerooni, What data should i protect? recommender and planning support for data security analysts, in: Proceedings of the 24th International Conference on Intelligent User Interfaces, 2019, pp. 286–297.
  * [47] K. Mindermann, Are easily usable security libraries possible and how should experts work together to create them?, in: Proceedings of the 9th international workshop on cooperative and human aspects of software engineering, 2016, pp. 62–63.
  * [48] J. Shires, Cyber-noir: Cyber-security and popular culture, Contemporary Security Policy 41 (1) (2020) 82–107.
  * [49] S. Frey, A. Rashid, P. Anthonysamy, M. Pinto-Albuquerque, S. A. Naqvi, The good, the bad and the ugly: a study of security decisions in a cyber-physical systems game, IEEE Transactions on Software Engineering 45 (5) (2017) 521–536.
  * [50] I. Ajzen, From intentions to actions: A theory of planned behavior, in: Action control, Springer, 1985, pp. 11–39.
  * [51] S. Lahlou, M. Langheinrich, C. Röcker, Privacy and trust issues with invisible computers, Communications of the ACM 48 (3) (2005) 59–60.
  * [52] S. L. Pfleeger, D. D. Caputo, Leveraging behavioral science to mitigate cyber-security risk, Computers & Security 31 (4) (2012) 597–611.
  * [53] U. Malmendier, D. Shanthikumar, Are small investors naive about incentives?, Journal of Financial Economics 85 (2) (2007) 457–489.
  * [54] Y. Guan, C. Li, H. Lu, M. Wong, Regulations and brain drain: Evidence from wall street star analysts’ career choices, Management Science, Forthcoming (2018).
  * [55] R. W. Reeder, I. Ion, S. Consolvo, 152 simple steps to stay safe online: Security advice for non-tech-savvy users, IEEE Security & Privacy 15 (5) (2017) 55–64.
  * [56] E. Rader, R. Wash, Identifying patterns in informal sources of security information, Journal of Cyber-Security 1 (1) (2015) 121–144.  doi:10.1093/cybsec/tyv008.
URL https://doi.org/10.1093/cybsec/tyv008

  * [57] E. M. Redmiles, S. Kross, M. L. Mazurek, How well do my results generalize? comparing security and privacy survey results from mturk, web, and telephone samples, in: 2019 IEEE Symposium on Security and Privacy (SP), Vol. 00, IEEE, 2019, pp. 227–244.  doi:10.1109/SP.2019.00014.
URL https://doi.org/10.1109/SP.2019.00014

  * [58] A. Schotter, Decision making with naive advice, American Economic Review 93 (2) (2003) 196–201.
  * [59] W. Steinel, A. E. Abele, C. K. W. De Dreu, Effects of experience and advice on process and performance in negotiations, Group Processes & Intergroup Relations 10 (4) (2007) 533–550.  doi:10.1177/1368430207081541.
URL https://doi.org/10.1177/1368430207081541

  * [60] A. Chaudhuri, Sustaining cooperation in laboratory public goods experiments: a selective survey of the literature, Experimental Economics 14 (1) (2011) 47–83.  doi:10.1007/s10683-010-9257-1.
URL https://doi.org/10.1007/s10683-010-9257-1

  * [61] X. J. Kuang, R. A. Weber, J. Dana, How effective is advice from interested parties?: An experimental test using a pure coordination game, Journal of Economic Behavior & Organization 62 (4) (2007) 591–604.
  * [62] G. R. Milne, L. I. Labrecque, C. Cromer, Toward an understanding of the online consumer’s risky behavior and protection practices, Journal of Consumer Affairs 43 (3) (2009) 449–473.  doi:10.1111/j.1745-6606.2009.01148.x.
URL https://doi.org/10.1111/j.1745-6606.2009.01148.x

  * [63] A. E. Howe, I. Ray, M. Roberts, M. Urbanska, Z. Byrne, The psychology of security for the home computer user, in: 2012 IEEE Symposium on Security and Privacy, IEEE, 2012, pp. 209–223.  doi:10.1109/SP.2012.23.
URL https://doi.org/10.1109/SP.2012.23

  * [64] N. Nthala, I. Flechais, “if it’s urgent or it is stopping me from doing something, then i might just go straight at it”: a study into home data security decisions, in: International Conference on Human Aspects of Information Security, Privacy, and Trust, Springer, 2017, pp. 123–142.
  * [65] J. Garrick, Informal learning in corporate workplaces, Human Resource Development Quarterly 9 (2) (1998) 129–144.
  * [66] Z. Byrne, J. Weidert, J. Liff, M. Horvath, C. Smith, A. Howe, I. Ray, Perceptions of internet threats: Behavioral intent to click again, in: Proceedings of the 27th Annual Conference of the Society for Industrial and Organizational Psychology, 2012, pp. 26–28.
  * [67] R. Shillair, S. R. Cotten, H.-Y. S. Tsai, S. Alhabash, R. LaRose, N. J. Rifon, Online safety begins with you and me: Convincing internet users to protect themselves, Computers in Human Behavior 48 (2015) 199–207.  doi:10.1016/j.chb.2015.01.046.
URL https://doi.org/10.1016/j.chb.2015.01.046

  * [68] R. West, The psychology of security, Communications of the ACM 51 (4) (2008) 34–40.  doi:10.1145/1330311.1330320.
URL https://doi.org/10.1145/1330311.1330320

  * [69] P. Burghouwt, M. Spruit, H. Sips, Towards detection of botnet communication through social media by monitoring user activity, in: International Conference on Information Systems Security, Springer, 2011, pp. 131–143.
  * [70] A. Al Hasib, Threats of online social networks, International Journal of Computer Science and Network Security (IJCSNS) 9 (11) (2009) 288–293.
  * [71] J. M. Haney, W. G. Lutters, ”it’s scary…it’s confusing…it’s dull”: How cyber-security advocates overcome negative perceptions of security, in: Fourteenth Symposium on Usable Privacy and Security (SOUPS), 2018, pp. 411–425.
URL https://www.usenix.org/conference/soups2018/presentation/haney-perceptions

  * [72] B. Stanton, M. F. Theofanos, S. S. Prettyman, S. Furman, Security fatigue, IT Professional 18 (5) (2016) 26–32.  doi:10.1109/MITP.2016.84.
URL https://doi.org/10.1109/MITP.2016.84

  * [73] S. D. Hight, The importance of a security, education, training and awareness program, november 2005, Security 27601 (2005) 1–5.
  * [74] A. Caballero, Security education, training, and awareness, in: Computer and information security handbook, Elsevier, 2017, pp. 497–505.  doi:10.1016/b978-0-12-803843-7.00033-8.
URL https://doi.org/10.1016/b978-0-12-803843-7.00033-8

  * [75] C. Lee, C. C. Lee, S. Kim, Understanding information security stress: Focusing on the type of information security compliance activity, Computers & Security 59 (2016) 60–70.  doi:10.1016/j.cose.2016.02.004.
URL https://doi.org/10.1016/j.cose.2016.02.004

  * [76] J. Malcolm, P. Hodkinson, H. Colley, The interrelationships between informal and formal learning, Journal of Workplace Learning (2003).  doi:10.1108/13665620310504783.
URL https://doi.org/10.1108/13665620310504783

  * [77] T. Ollis, Learning in social action: The informal and social learning dimensions of circumstantial and lifelong activists., Australian Journal of Adult Learning 51 (2) (2011) 248–268.
  * [78] G. Bull, A. Thompson, M. Searson, J. Garofalo, J. Park, C. Young, J. Lee, Connecting informal and formal learning experiences in the age of participatory media, Contemporary Issues in Technology and Teacher Education 8 (2) (2008) 100–107.
  * [79] A. Forget, S. Pearman, J. Thomas, A. Acquisti, N. Christin, L. F. Cranor, S. Egelman, M. Harbach, R. Telang, Do or do not, there is no try: user engagement may not improve security outcomes, in: Twelfth Symposium on Usable Privacy and Security (SOUPS), 2016, pp. 97–111.
  * [80] J. E. Maddux, Self-efficacy theory, in: Self-efficacy, adaptation, and adjustment, Springer, 1995, pp. 3–33.
  * [81] A. Bandura, W. H. Freeman, R. Lightsey, Self-efficacy: The exercise of control, Springer, 1999.
  * [82] A. M. Saks, B. E. Ashforth, Proactive socialization and behavioral self-management, Journal of Vocational behavior 48 (3) (1996) 301–323.
  * [83] M. Warner, Cyber-security: A pre-history, Intelligence and National Security 27 (5) (2012) 781–799.
  * [84] S. A. Stumpf, A. P. Brief, K. Hartman, Self-efficacy expectations and coping with career-related events, Journal of Vocational Behavior 31 (1) (1987) 91–108.
  * [85] T. Halevi, N. Memon, J. Lewis, P. Kumaraguru, S. Arora, N. Dagar, F. Aloul, J. Chen, Cultural and psychological factors in cyber-security, in: Proceedings of the 18th International Conference on Information Integration and Web-based Applications and Services, 2016, pp. 318–324.
  * [86] S. Ruoti, T. Monson, J. Wu, D. Zappala, K. Seamons, Weighing context and trade-offs: How suburban adults selected their online security posture, in: Proceedings of the Thirteenth Symposium on Usable Privacy and Security (SOUPS), 2017, pp. 211–228.
URL https://www.usenix.org/conference/soups2017/technical-
sessions/presentation/ruoti

  * [87] J. Nicholson, L. Coventry, P. Briggs, ”if it’s important it will be a headline” cyber-security information seeking in older adults, in: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 2019, pp. 1–11, paper No. 349.  doi:10.1145/3290605.3300579.
URL https://doi.org/10.1145/3290605.3300579

  * [88] K. R. Fulton, R. Gelles, A. McKay, R. Roberts, Y. Abdi, M. L. Mazurek, The effect of entertainment media on mental models of computer security, in: Proceedings of the Fifteenth Symposium on Usable Privacy and Security (SOUPS), 2019, pp. 79–95.
URL https://www.usenix.org/conference/soups2019/presentation/fulton

  * [89] H. D. Lasswell, The structure and function of communication in society, in: L. Bryson (Ed.), The Communication of Ideas, Harper and Row, 1948, pp. 37–51.
  * [90] M. E. McCombs, D. L. Shaw, The agenda-setting function of mass media, Public Opinion Quarterly 36 (2) (1972) 176–187.  doi:10.1086/267990.
URL https://doi.org/10.1086/267990

  * [91] N.-B. Schirrmacher, J. Ondrus, F. T. C. Tan, Towards a response to ransomware: Examining digital capabilities of the wannacry attack, in: PACIS, 2018, p. 210\.
  * [92] E. Wenger, Communities of practice: Learning as a social system, Systems Thinker 9 (5) (1998) 2–3.
URL https://thesystemsthinker.com/communities-of-practice-learning-as-a-
social-system/

  * [93] M. Meyer, The rise of the knowledge broker, Science Communication 32 (1) (2010) 118–127.  doi:10.1177/1075547009359797.
URL https://doi.org/10.1177/1075547009359797

  * [94] D. Contandriopoulos, M. Lemire, J.-L. Denis, É. Tremblay, Knowledge exchange processes in organizations and policy arenas: a narrative systematic review of the literature, The Milbank Quarterly 88 (4) (2010) 444–483.  doi:10.1111/j.1468-0009.2010.00608.x.
URL https://doi.org/10.1111/j.1468-0009.2010.00608.x

  * [95] N. Marres, E. Weltevrede, Scraping the social? issues in live social research, Journal of Cultural Economy 6 (3) (2013) 313–335.  doi:10.1080/17530350.2013.772070.
URL https://doi.org/10.1080/17530350.2013.772070

  * [96] D. Schatz, R. Bashroush, J. Wall, Towards a more representative definition of cyber-security, Journal of Digital Forensics, Security and Law 12 (2) (2017) 53–74.  doi:10.15394/jdfsl.2017.1476.
URL https://doi.org/10.15394/jdfsl.2017.1476

  * [97] M. Humayun, M. Niazi, N. Jhanjhi, M. Alshayeb, S. Mahmood, Cyber-security threats and vulnerabilities: a systematic mapping study, Arabian Journal for Science and Engineering 45 (4) (2020) 3171–3189.
  * [98] T. Kosar, S. Bohra, M. Mernik, Protocol of a systematic mapping study for domain-specific languages, Journal of Information and Software Technology 21 (C) (2016) 77–91.
  * [99] T. Satyapanich, T. Finin, F. Ferraro, Extracting rich semantic information about cyber-security events, in: 2019 IEEE International Conference on Big Data (Big Data), 2019, pp. 5034–5042.  doi:10.1109/BigData47090.2019.9006444.
URL https://doi.org/10.1109/BigData47090.2019.9006444

  * [100] N. Al Moubayed, D. Wall, A. S. McGough, Identifying changes in the cyber-security threat landscape using the LDA-web topic modelling data search engine, in: T. Tryfonas (Ed.), Human Aspects of Information Security, Privacy and Trust, Springer International Publishing, Cham, 2017, pp. 287–295, lecture Notes in Computer Science, vol 10292.
  * [101] M. F. Porter, An algorithm for suffix stripping, Program (2006).
  * [102] S. Piasecki, L. Urquhart, D. McAuley, Defence against the dark artefacts: Smart home cyber crimes and cyber-security standards, Computer Law & Security Review 42 (2021) 105542.
  * [103] J. Ruohonen, K. K. Kimppa, Updating the wassenaar debate once again: Surveillance, intrusion software, and ambiguity, Journal of Information Technology & Politics 16 (2) (2019) 169–186.
  * [104] J. Cao, T. Xia, J. Li, Y. Zhang, S. Tang, A density-based method for adaptive LDA model selection, Neurocomputing 72 (7–9) (2009) 1775–1781.  doi:10.1016/j.neucom.2008.06.011.
URL https://doi.org/10.1016/j.neucom.2008.06.011

  * [105] R. Deveaud, E. Sanjuan, P. Bellot, Accurate and effective latent concept modeling for ad hoc information retrieval, Document Numérique 17 (1) (2014) 61–84.  doi:10.3166/dn.17.1.61-84.
URL https://doi.org/10.3166/dn.17.1.61-84

  * [106] M. R. Alagheband, A. Mashatan, M. Zihayat, Time-based gap analysis of cyber-security trends in academic and digital media, ACM Transactions on Management Information Systems (TMIS) 11 (4) (2020) 1–20.  doi:10.1145/3389684.
URL https://doi.org/10.1145/3389684

  * [107] M. Taylor, M. Wolff, The Victorians since 1901: Histories, representations and revisions, Manchester University Press, 2004.
  * [108] J. Goldstein, M. Kantrowitz, V. Mittal, J. Carbonell, Summarizing text documents: Sentence selection and evaluation metrics, in: Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, 1999, pp. 121–128.  doi:10.1145/312624.312665.
URL https://doi.org/10.1145/312624.312665

  * [109] S. Kalra, J. S. Prasad, Efficacy of news sentiment for stock market prediction, in: 2019 International Conference on Machine Learning, Big Data, Cloud and Parallel Computing (COMITCon), IEEE, 2019, pp. 491–496.  doi:10.1109/COMITCon.2019.8862265.
URL https://doi.org/10.1109/COMITCon.2019.8862265

  * [110] T. M. Yasaka, B. M. Lehrich, R. Sahyouni, Peer-to-peer contact tracing: Development of a privacy-preserving smartphone app, JMIR mHealth and uHealth 8 (4) (2020) e18936.  doi:10.2196/18936.
URL https://doi.org/10.2196/18936

  * [111] A. M. Lindner, G. Pryciak, J. Elsner, Tor and the city: Msa-level correlates of interest in anonymous web browsing, Surveillance & Society 18 (4) (2020) 507–521.
URL https://ojs.library.queensu.ca/index.php/surveillance-and-
society/article/view/13235/9469

  * [112] S. Lim, A. Jatowt, M. Yoshikawa, Understanding characteristics of biased sentences in news articles, in: CIKM Workshops, 2018, pp. 121–128.
  * [113] H. S. Heaps, Information retrieval, computational and theoretical aspects, Academic Press, 1978.
  * [114] T. Kanungo, D. Orr, Predicting the readability of short web summaries, in: Proceedings of the Second ACM International Conference on Web Search and Data Mining, 2009, pp. 202–211.  doi:10.1145/1498759.1498827.
URL https://doi.org/10.1145/1498759.1498827

  * [115] N. Grinberg, Identifying modes of user engagement with online news and their relationship to information gain in text, in: Proceedings of the 2018 World Wide Web Conference, 2018, pp. 1745–1754.  doi:10.1145/3178876.3186180.
URL https://doi.org/10.1145/3178876.3186180

  * [116] R. Flesch, Flesch-kincaid readability test, Retrieved October 26 (3) (2007) 2007\.
  * [117] J. C. Roberts, R. H. Fletcher, S. W. Fletcher, Effects of peer review and editing on the readability of articles published in annals of internal medicine, JAMA 272 (2) (1994) 119–121.  doi:10.1001/jama.1994.03520020045012.
URL https://doi.org/10.1001/jama.1994.03520020045012

  * [118] M. Coleman, T. L. Liau, A computer readability formula designed for machine scoring, Journal of Applied Psychology 60 (2) (1975) 283–284.  doi:10.1037/h0076540.
URL https://doi.org/10.1037/h0076540

  * [119] R. J. Senter, E. A. Smith, Automated readability index, Tech. rep., AMRL-TR. Aerospace Medical Research Laboratories (1967).
URL https://apps.dtic.mil/dtic/tr/fulltext/u2/667273.pdf

  * [120] G. H. Mc Laughlin, Smog grading—a new readability formula, Journal of Reading 12 (8) (1969) 639–646.
URL https://www.jstor.org/stable/40011226

  * [121] D. M. E.-D. M. Hussein, A survey on sentiment analysis challenges, Journal of King Saud University - Engineering Sciences 30 (4) (2018) 330–338.  doi:10.1016/j.jksues.2016.04.002.
URL https://doi.org/10.1016/j.jksues.2016.04.002

  * [122] M. Macdonald, R. Frank, J. Mei, B. Monk, Identifying digital threats in a hacker web forum, in: Proceedings of the 2015 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining 2015, 2015, pp. 926–933.  doi:10.1145/2808797.2808878.
URL https://doi.org/10.1145/2808797.2808878

  * [123] S. Deerwester, S. T. Dumais, G. W. Furnas, T. K. Landauer, R. Harshman, Indexing by latent semantic analysis, Journal of the American society for information science 41 (6) (1990) 391–407.
  * [124] W. L. Hamilton, K. Clark, J. Leskovec, D. Jurafsky, Inducing domain-specific sentiment lexicons from unlabeled corpora, in: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Vol. 2016, NIH Public Access, 2016, pp. 595–605.
URL https://www.aclweb.org/anthology/D16-1057.pdf

  * [125] A. E. Oldehoeft, Foundations of a Security Policy for Use of the National Research and Educational Network, U.S. Department of Commerce, National Institute of Standards and Technology, 1992.
URL https://nvlpubs.nist.gov/nistpubs/Legacy/IR/nistir4734.pdf

  * [126] T. E. Cook, Governing with the news: The news media as a political institution, University of Chicago Press, 1998.
  * [127] S. K. Gadarian, The politics of threat: How terrorism news shapes foreign policy attitudes, The Journal of Politics 72 (2) (2010) 469–483.  doi:10.1017/s0022381609990910.
URL https://doi.org/10.1017/s0022381609990910

  * [128] A. Kher, S. Johnson, R. Griffith, Readability assessment of online patient education material on congestive heart failure, Advances in Preventive Medicine 2017 (2017).  doi:10.1155/2017/9780317.
URL https://doi.org/10.1155/2017/9780317

  * [129] R. K. Britt, W. B. Collins, K. Wilson, G. Linnemeier, A. M. Englebert, ehealth literacy and health behaviors affecting modern college students: A pilot study of issues identified by the american college health association, Journal of medical Internet research 19 (12) (2017) e392.
URL https://doi.org/10.2196/preprints.3100

  * [130] A. Adams, M. A. Sasse, Users are not the enemy, Communications of the ACM 42 (12) (1999) 40–46.  doi:10.1145/322796.322806.
URL https://doi.org/10.1145/322796.322806

  * [131] N. Höchstötter, D. Lewandowski, What users see–structures in search engine results pages, Information Sciences 179 (12) (2009) 1796–1812.