# Revisiting the Design Agenda for Privacy Notices and Security Warnings

Diana Korka  0000-0003-2019-1599<EMAIL_ADDRESS>University of
LausanneSwitzerland ,  Kavous Salehzadeh Niksirat  0000-0003-4438-3544
<EMAIL_ADDRESS>University of LausanneSwitzerland and  Mauro
Cherubini  0000-0002-1860-6110<EMAIL_ADDRESS>University of
LausanneSwitzerland

(2023; 16 April 2023)

###### Abstract.

System-generated user-facing notices, dialogs, and warnings in privacy and
security interventions present the opportunity to support users in making
informed decisions about identified risks. However, too often, they are
bypassed, ignored, and mindlessly clicked through, mainly in connection to the
well-studied effect of user fatigue and habituation. The contribution of this
position paper is to provide a summarized review of established and emergent
design dimensions and principles to limit such risk-prone behavior and to
identify three emergent research and design directions for privacy-enhancing
dialogs.

††copyright: none††journalyear: 2023††conference: Conference on Human Factors
in Computing Systems; April 23–28, 2023; Hamburg, Germany††booktitle: CHI
2023: Workshop on Privacy Interventions and Education (PIE): Encouraging
Privacy Protective Behavioral Change Online, April 28, 2023, Hamburg,
Germany††isbn: 978-1-4503-XXXX-X/18/06††ccs: Human Computer Interaction Usable
privacy and security††ccs: Human Computer Interaction Privacy notices††ccs:
Human Computer Interaction Security warnings

##  1\. Motivation

System-generated user-facing notices and dialogs in security and privacy
interventions are ubiquitous elements of human-computer interaction, in a
multitude of contexts such as websites, mobile applications, computers,
smartphones, wearables, privacy assistants or smart home devices. From a
design perspective, these interventions draw on recommendations of risk
communication theory and applicable legal requirements. Typically they inform
or warn users of existing privacy and security risks and present them with the
choice of opting out of that risk, or continuing should they have sufficient
evidence or should they choose to ignore warnings. They are, for example,
privacy notices informing on practices with users’ personal data and
requesting users’ consent, dissuasive privacy-related dialogs identifying
possible privacy breaches and offering alternatives for opting out, software
installation dialogs, warnings for “your connection is not private”, or warn-
and-continue dialogs for downloading/saving email attachments (see Figures 2
and 3 in Appendix B). They differ from interventions that design out the
identified risk (e.g., no personal data is collected at all) or those that
completely prevent users from taking the risky option (e.g., Internet service
providers blocking the use of specific IP addresses).

While such mechanisms for privacy and security interventions can be beneficial
by educating users to protect themselves against potential risks, numerous
experimental field and laboratory studies show that users routinely ignore
them, thus exposing themselves to the negative consequences of identified
risks (Dhamija et al., 2006; Ebert et al., 2021; Obar and Oeldorf-Hirsch,
2018; Schaub et al., 2017; Schechter et al., 2007; Wu et al., 2006; Böhme and
Köpsell, 2010). The observed behavior brings into question the effectiveness
of such interventions in everyday life. We reviewed the related literature to
identify design recommendations and unexplored research areas for supporting
user behavior. Particularly, we looked at three goals for rendering the design
more effective: (i) improve the informative and educational value of such
notices to help users understand the risk they expose themselves and/or others
to, (ii) strengthen their persuasion power such that users take the safe
option when justified, and (iii) limit habituation and inattentive click-
through behavior. Specifically, we sought to identify common guidelines to
inform the design of notices and dialogs,111Notices and dialogs can vary
greatly from one device to another. For example, for wearables, privacy
notices are often made available upon first use, and on an accompanying
website, as displaying such information on a small screen is not feasible.
because small differences in design have been shown to impact their
effectiveness significantly (Akhawe and Felt, 2013).

##  2\. Background

Privacy policies or notices are intended to help users make informed data
privacy decisions. Nevertheless, consumers often find themselves clueless when
presented with notices in the form of long lists of data practices as they are
written in highly technical and legalistic terms, vague to remain flexible,
and aiming primarily to demonstrate compliance to regulators (Böhme and
Köpsell, 2010; Schaub et al., 2017). Obar and Oeldorf-Hirsch (2018) found that
74% of the respondents to a large-scale survey clicked through and skipped
reading privacy policies, 97% agreed to them, and many failed to recognize
when some of these notices included problematic and harmful data practices.
Ebert et al. (2021) assessed the effectiveness of short and concise privacy
notices from a fitness tracking app by asking participants to recall data
practices and confirmed that users rarely read privacy notices when these are
hidden behind links.

Security warnings identify a risk to those exposed to it, afford users the
choice to avoid it, and explain the consequences of risk materialization
(Wogalter, 2006). Enhancing the effectiveness of security warnings is a well-
researched topic, yet risk-prone user behavior persists. Wu et al. (2006)
examined the usability of security toolbars in web browsers and showed them to
be ineffective in preventing phishing attacks. Similarly, for SSL (or Secure
Socket Layer) warnings, more than half of the participants in a lab study
ignored them, thus potentially exposing themselves to fraudulent websites
(Dhamija et al., 2006). In another study, 53% of participants ignored security
warnings on an online banking site, clicked through, and entered passwords,
thus potentially exposing themselves to man-in-the-middle or phishing attacks
(Schechter et al., 2007).

The potential benefits of enhancing the effectiveness of such mechanisms are
three-fold: (i) to improve their informative and educational function, (ii) to
enhance their persuasive power to orient users towards safer options, and
(iii) to address _habituation_ , also known as _“the repetition suppression
effect, for repeated exposure”_ (Anderson et al., 2014). Anderson et al.
(2014) used functional magnetic resonance imaging (fMRI) to measure
habituation in participants exposed to security warning messages. They showed
that repeated exposure leads to significantly reduced neural activity in the
brain’s visual processing center, and attention diminishes. The unconscious
automatic response tunes out learned repetitive stimuli to focus attention on
another task at hand. Response decay was already noticeable from the second
presentation and continued to increase linearly with the number of
repetitions; at the same time, the habituation response was found to be
significantly greater for security warnings than for general software
applications. Habituation can have a contamination effect across computer
systems. If users become accustomed to ignoring security warnings in one
context, they may apply the same behavior in another similar context. For
example, Sunshine et al. (2009) observed that participants remembered previous
security warnings and applied them to other websites, similar to Egelman et
al. (2008) who showed that participants who recognized a warning message from
a trusted website were significantly less likely to read a similar message
when it appeared on another website and thus also less likely to heed it.

All is not lost however, as attention recovers spontaneously after some time
of non-exposure to the habituation-provoking stimulus (Amran et al., 2018). In
addition, attention decay can be countered by _dishabituation_ interventions,
such as exposure to polymorphic warnings, with repeated changes in the message
appearance (Anderson et al., 2014).

##  3\. The Design Space for Privacy Notices and Security Warnings

Prior work identified a set of design dimensions, guidelines and principles,
by considering privacy notices and security warnings in isolation. We
summarize the design dimensions for privacy notices in Figure 1 in Appendix A,
the left hand side. Schaub et al. (2018) identify four dimensions for the
design space by reviewing the usability literature.

  * •

Timing. The moment the notice is triggered in the design flow, for example, at
setup, periodically, persistent, or on demand.

  * •

Channel. The means of providing the notice, for example, within the same
interaction platform of the system, or rather through a secondary channel, for
example, through video tutorials, or yet through a public channel, such a
privacy beacon, which broadcasts preferences to any nearby users.

  * •

Modality. The form the notice takes, for example visual, auditory, haptic,
machine-readable.

  * •

Control. The level of control afforded by the notice to the user, whether the
notice is in a take-it-or-leave-it form, whether it enables users to refine
their privacy preferences, and whether consent is required before accessing
the service.

Furthermore, the authors discuss recommendations for privacy notice design, in
connection to the four dimensions. For timing, two considerations stand out:
(i) seeking better integration of the notice into a system’s interaction flow
and (ii) considering that reading the notice conflicts with the user’s primary
task. For channel, the preference is, when possible, the direct (main) channel
(confirmed by Ebert et al. (2021)), providing the notice in the same platform,
application or device the user interacts with, thus avoiding context change,
to say a separate accompanying website with privacy terms and conditions. For
modality, a combination of, for example, text and image content can help
attract attention and better convey the message. In addition, further
consideration is given to reducing the level of reading difficulty of the
text, avoiding the use of specialized terms or jargon, and preferring a simple
and concise layout of notices over lengthy texts. For control, the
recommendation goes towards affording choice, similar to the pre-selected
checkboxes for the data types to be shared designed by Wang et al. (2013).

Feng et al. (2021) extend the design space for affording choice, where choice
is defined as communicating and providing access to actions that result in an
effective control over data practices. For example, they refine the Control
dimension of Schaub et al. (2018), to Type and Functionality of choices. Type
considers the comprehensiveness of choices: binary choice (e.g., opt-in or
opt-out), multiple choice (e.g., several opt-ins, for more granular data
practices), contextualized choice (e.g., before a particular risk occurs), and
more comprehensive rights-based choice (e.g., affording data portability, and
rectification). Functionality considers the way the system presents privacy
choices, the enforcement mechanism, and the feedback on privacy choice status.

Another consideration is that privacy notices tend to serve both the purpose
of demonstrating compliance with legal requirements222For example, General
Data Protection Regulation (GDPR) (The European Parliament and the Council of
the European Union, 2016) in Europe or California Consumer Privacy Act (CCPA)
(Office of the California Attorney General, 2018) in the US., and of
effectively informing users of current data practices. Thus, Schaub et al.
(2017) argue in favor of a new practice, complementing long, complex,
compliance-oriented privacy notices, with user-centric just-in-time, tailored,
multi-layered, and specific notices that are better integrated into the
interaction flow. We did not retain a specific recommendation on design to
reduce habituation to privacy notices.

For security messages, Wogalter (2006) provided design principles for several
aspects of a warning, including identifying standard components of a warning
message, standard text color related to the level of risk, background, and
symbols to be used in a signal word panel (with reference to ANSI
(Association, 2022)), message text format, font type, wording, pictorial
symbols, giving the warning with adequate time to enable the user to act,
recommending usability testing, and suggesting to change the appearance of the
warning to reduce habituation (i.e., polymorphic design). Furthermore, the
usable security team at Microsoft proposed a list of guiding principles for
the design of effective security warnings (Reeder et al., 2011) (summarized in
Figure 1 in Appendix A, the right-hand-side).

  * •

Necessary. A warning should only interrupt the interaction flow when a user
absolutely needs to be involved.

  * •

Explained. It should explain the decision the user needs to make and give them
helpful information for making that choice. Therefore, the warning should
identify the source application that triggered it, the process the user needs
to undergo to make the right decision, the risk consequences, the unique
knowledge that the user has to make the decision, the list of available
choices, and the information evidence to be used in the decision-making.

  * •

Actionable. It should only be presented if the user can realistically make a
better-informed choice.

  * •

Tested. It should undergo usability testing.

Several usability tests shed further insights on the design space. Few privacy
studies centered on how to make the information and educational content easier
to grasp. For example, Kelley et al. (2010); Reinhardt et al. (2021);
Kitkowska et al. (2022) leverage the advantages of a _compact and structured
layout_ to study the impact of different privacy policy formats, with exciting
prospects for nutrition-label tabular summaries, and for adding visual icons,
or other graphical representations to convey comparable information. Windl et
al. (2022) took on these recommendations to develop a tool that automatically
extracts and displays concise and comparable contextual privacy notices from
existing privacy notices, leading to better-informed behavior. Most
interventions focused on how to attract user attention to the information.
Affording more granular control over the data types shared, through separate
checkboxes, resulted in lower privacy permissions (Wang et al., 2013), and
incorporating interaction elements (e.g., swiping) improved attention (Karegar
et al., 2020). Others successfully explored nudges for privacy, in online
social media, by drawing attention to the audience of a future post or by
delaying the post briefly and allowing additional edits (Wang et al., 2014)
and by adding examples of the personal data surrendered when accepting
permissions for a mobile application (Harbach et al., 2014). For security,
proven nudges consisted of interactive attractors to the information evidence
to be used in decision-making (Bravo-Lillo et al., 2013), claiming the
presence of a human auditor in the loop to audit security decisions
(Brustoloni and Villamarín-Salomón, 2007), or persuading users to take the
safer decisions using more visually appealing design and through dissimulating
unsafe options behind “advanced” buttons, requiring additional interaction
steps (Felt et al., 2015). A few security interventions addressed habituation,
by varying the form in which users provide inputs (Brustoloni and Villamarín-
Salomón, 2007), or by animating warning messages (e.g., jiggle) and varying
the size of the warning icons (Anderson et al., 2015).

##  4\. An Emerging Research and Design Agenda

We identify future research and design directions for security and privacy
user-facing dialogs. These can apply to the growing number of initiatives to
improve the effectiveness of such interactions and reduce the extent to which
users ignore, bypass, or become habituated to them. We propose these ideas for
discussion at the workshop on Privacy Interventions and Education (PIE):
Encouraging Privacy Protective Behavioral Change Online, of the ACM CHI
Conference on Human Factors in Computing Systems (2023)333Workshop on Privacy
Interventions and Education (PIE): Encouraging Privacy Protective Behavioral
Change Online, ACM CHI Conference on Human Factors in Computing Systems, 23-28
April 2023, Hamburg, Germany
https://programs.sigchi.org/chi/2023/program/session/102064.

###  4.1. Leveraging Behavioral Theories and Dual-System Thinking

Most existing privacy- and security- dialogs support rational decision-making.
We ask, how can behavioral decision-making theories and dual-system thinking
be leveraged to underpin research on improved effectiveness?

While most previous efforts to enhance the effectiveness of interventions
focused on supporting users in rational decision-making, dual system thinking
theory (Kahneman, 2012) shows that human decision-making is only boundedly
rational, suffers from cognitive biases, and can be underpinned by emotions,
values, cognitive frames, and worldviews. Redmiles et al. (2018) measured that
rationality accounted for between 48 to 56% of the decisions in a behavioral
economics-based experiment, in which participants were required to make
security decisions with clearly identified benefits and costs. Arguably with
privacy notices, benefits and costs may not be immediately apparent.

A number of studies have begun to explore the design space in this area. For
example, Anderson et al. (2014) found that security warnings trigger more fear
and anxiety signals in the brain as compared to general software applications.
But it is not clear how these emotions mediate attention and habituation.
Cherubini et al. (2021) designed fear-, shame-, and empathy-arousing
dissuasive warnings in a context of multi-party privacy risks, informed by
dual-system theory and with the intention of provoking self-reflection.
Further research would be needed to understand the impact of such warnings in
the wild. Furthermore, some studies explored the potential of comic-based
design complements in attracting users’ attention to the privacy policy
content, with encouraging results (Tabassum et al., 2018; Kitkowska et al.,
2020). Kitkowska et al. (2020) used the Antecedents – Privacy Concerns –
Outcomes (APCO) decision-making model and dual-process theory to inform the
design of privacy notices and found that the affective state (positive or
negative) elicited by a privacy note, can have an indirect effect on the
decision to disclose information. Curiosity was also associated with improved
privacy comprehension. In addition, providing users with control over privacy
decisions leads to higher satisfaction when combined with curiosity-enhancing
cues. These few studies show the potential of further exploring emotion-
arousing design in attracting attention.

Analyzing simplified lines of reasoning for observing cues and arriving at
conclusions can be a valuable analytical tool. Identifying mental models can
help inform more effective designs that address common misconceptions or
knowledge gaps. For example, Bravo-Lillo et al. (2011) found that novices use
principally a website’s appearance to judge its trustworthiness. Another
methodological approach, participatory design can help understand users’
perspectives, and involve them in the design process (see, for example,
Salehzadeh Niksirat et al. (2021)). For example, after repeated exposure,
participants reported becoming habituated to the prompts of a multi-party
privacy mediation bot, but at the same time, familiarity helped them feel more
confident in interacting with it (Salehzadeh Niksirat et al., 2023).

###  4.2. Further Exploring the Design Space of Sustainable and Persuasive
Dialogs

We ask, what design elements can sustainably persuade users to make more
informed risk decisions? Under what conditions can users take on the habit of
making safe choices by default?

Given users’ limited technical understanding, it is difficult to inform them
thoroughly of the risk threat at hand. Perhaps behavior change theories can be
leveraged to guide users towards less risky choices. Several studies have
already started exploring the potential of nudges (Acquisti et al., 2017) to
enhance privacy (e.g., (Choe et al., 2013; Wang et al., 2014)) and security
interactions (e.g., (Bravo-Lillo et al., 2013; Felt et al., 2015)), with mixed
results for user satisfaction. Equally important to comprehend are dark
patterns (i.e., nudging users towards riskier choices)  (Nouwens et al.,
2020). Mapping out more comprehensively the design space of nudging
interventions would help, as they currently only target two aspects of the
design space: (i) increase the salience of information evidence to be
considered by users before risk decisions, (ii) frame risk-related decision
alternatives as main (safe) and secondary (unsafe).

Additionally, it is interesting to understand how standard privacy and
security dialogs compare in terms of compliance with self-determination theory
(SDT) (Ryan and Deci, 2000) by exploring aspects related to users’ autonomy
(i.e., feeling in control), competence (i.e., feeling competent to decide),
and relatedness (i.e., feeling connected to others), to evaluate to what
extent these interventions are capable of instilling a lasting behavioral
impact on users. Some studies (e.g., (Ertan et al., 2020)) have already
addressed the impact of social/peer influence in changing security-related
behavior. Going further, one could investigate, for example, how social
influence can be leveraged to enhance privacy notices and dialogs. And given
the influence of advice, under what conditions, if any, can users take on the
habit of by default selecting less risky actions (e.g., choosing not to
consent to share private data with websites).

More research could shed light on how to use design to instill behavior change
in privacy interventions. For example, when investigating alternatives for
privacy policies’ layout, Kelley et al. (2010) found that preserving a
standard tabular nutrition-label-inspired format was helpful because it
rendered the comparison across policies possible. We wonder whether, with
repeated exposure, users could take on the habit to read such policies because
the comparison is afforded.

###  4.3. Integrating the Design Space for Privacy and Security Dialogs

Finally, inspired by a study from Felt et al. (2015), we ask, can design
principles for security risk communications be used to inform the design of
privacy notices?

Felt et al. (2015) took into account how their newly designed security warning
fitted with the wider service context, and especially with the other warnings
generated by the platform (e.g., by considering the relative level of danger
of different security risks). Section 3 enumerates design features and
guidelines that were arrived at by considering privacy notices and security
warnings in isolation. We argue that comparing the design space for privacy
notices with that for security warnings can help identify potential gaps. Both
design spaces are underpinned by recommendations of risk communication theory
and have a good deal of elements in common. In addition, whereas for privacy
dialogs, only design dimensions and features are identified, for security
warnings, (voluntary) design principles are available. Some of the design
principles outlined for security warnings, may have not yet been fully
considered in the design space for privacy notices, and may be relevant when
designing privacy notices (e.g., highlighting the risk consequences and the
process the user should follow to make less risky choices). Figure 1 in
Appendix A attempts to compare the two design spaces and to identify common
areas of partial overlap for similar design features (identified with dotted
rectangles), as well as design features given prominence under security, but
which are not fully reflected in the privacy design space (identified with an
asterisk and in italics).

## References

  * (1)
  * Acquisti et al. (2017) Alessandro Acquisti, Idris Adjerid, Rebecca Balebako, Laura Brandimarte, Lorrie Faith Cranor, Saranga Komanduri, Pedro Giovanni Leon, Norman Sadeh, Florian Schaub, Manya Sleeper, Yang Wang, and Shomir Wilson. 2017\.  Nudges for Privacy and Security: Understanding and Assisting Users’ Choices Online.  _Comput. Surveys_ 50, 3 (Aug. 2017), 44:1–44:41.  https://doi.org/10.1145/3054926
  * Akhawe and Felt (2013) Devdatta Akhawe and Adrienne Porter Felt. 2013.  Alice in Warningland: A Large-Scale Field Study of Browser Security Warning Effectiveness. In _Proceedings of the 22nd USENIX Conference on Security_ _(SEC’13)_. USENIX Association, USA, 257–272.
  * Amran et al. (2018) Ammar Amran, Zarul Fitri Zaaba, and Manmeet Kaur Mahinderjit Singh. 2018\.  Habituation Effects in Computer Security Warning.  _Information Security Journal: A Global Perspective_ 27, 2 (March 2018), 119–131.  https://doi.org/10.1080/19393555.2018.1448492
  * Anderson et al. (2014) Bonnie Anderson, Anthony Vance, Brock Kirwan, David Eargle, and Seth Howard. 2014.  Users Aren’t (Necessarily) Lazy: Using NeuroIS to Explain Habituation to Security Warnings.  _ICIS 2014 Proceedings_ (Dec. 2014).
  * Anderson et al. (2015) Bonnie Brinton Anderson, C. Brock Kirwan, Jeffrey L. Jenkins, David Eargle, Seth Howard, and Anthony Vance. 2015.  How Polymorphic Warnings Reduce Habituation in the Brain: Insights from an fMRI Study. In _Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems_ _(CHI ’15)_. Association for Computing Machinery, New York, NY, USA, 2883–2892.  https://doi.org/10.1145/2702123.2702322
  * Association (2022) National Electrical Manufacturers Association. 2022.  ANSI Z535.3-2022 Criteria For Safety Symbols.  https://webstore.ansi.org/standards/nema/ansiz5352022-2475872.
  * Böhme and Köpsell (2010) Rainer Böhme and Stefan Köpsell. 2010.  Trained to Accept? A Field Experiment on Consent Dialogs. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_ _(CHI ’10)_. Association for Computing Machinery, New York, NY, USA, 2403–2406.  https://doi.org/10.1145/1753326.1753689
  * Bravo-Lillo et al. (2011) Cristian Bravo-Lillo, Lorrie Faith Cranor, Julie Downs, and Saranga Komanduri. 2011.  Bridging the Gap in Computer Security Warnings: A Mental Model Approach.  _IEEE Security & Privacy Magazine_ 9, 2 (March 2011), 18–26.  https://doi.org/10.1109/MSP.2010.198
  * Bravo-Lillo et al. (2013) Cristian Bravo-Lillo, Saranga Komanduri, Lorrie Faith Cranor, Robert W. Reeder, Manya Sleeper, Julie Downs, and Stuart Schechter. 2013\.  Your Attention Please: Designing Security-Decision UIs to Make Genuine Risks Harder to Ignore. In _Proceedings of the Ninth Symposium on Usable Privacy and Security_ _(SOUPS ’13)_. Association for Computing Machinery, New York, NY, USA, 1–12.  https://doi.org/10.1145/2501604.2501610
  * Brustoloni and Villamarín-Salomón (2007) José Carlos Brustoloni and Ricardo Villamarín-Salomón. 2007\.  Improving Security Decisions with Polymorphic and Audited Dialogs. In _Proceedings of the 3rd Symposium on Usable Privacy and Security_ _(SOUPS ’07)_. Association for Computing Machinery, New York, NY, USA, 76–85.  https://doi.org/10.1145/1280680.1280691
  * Cherubini et al. (2021) Mauro Cherubini, Kavous Salehzadeh Niksirat, Marc-Olivier Boldi, Henri Keopraseuth, Jose M. Such, and Kévin Huguenin. 2021.  When Forcing Collaboration Is the Most Sensible Choice: Desirability of Precautionary and Dissuasive Mechanisms to Manage Multiparty Privacy Conflicts.  _Proceedings of the ACM on Human-Computer Interaction_ 5, CSCW1 (April 2021), 53:1–53:36.  https://doi.org/10.1145/3449127
  * Choe et al. (2013) Eun Kyoung Choe, Jaeyeon Jung, Bongshin Lee, and Kristie Fisher. 2013.  Nudging People Away From Privacy-Invasive Mobile Apps Through Visual Framing.  (Sept. 2013).
  * Dhamija et al. (2006) Rachna Dhamija, J. D. Tygar, and Marti Hearst. 2006\.  Why Phishing Works. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_ _(CHI ’06)_. Association for Computing Machinery, New York, NY, USA, 581–590.  https://doi.org/10.1145/1124772.1124861
  * Ebert et al. (2021) Nico Ebert, Kurt Alexander Ackermann, and Björn Scheppler. 2021\.  Bolder Is Better: Raising User Awareness through Salient and Concise Privacy Notices. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ _(CHI ’21)_. Association for Computing Machinery, New York, NY, USA, 1–12.  https://doi.org/10.1145/3411764.3445516
  * Egelman et al. (2008) Serge Egelman, Lorrie Faith Cranor, and Jason Hong. 2008\.  You’ve Been Warned: An Empirical Study of the Effectiveness of Web Browser Phishing Warnings. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_ _(CHI ’08)_. Association for Computing Machinery, New York, NY, USA, 1065–1074.  https://doi.org/10.1145/1357054.1357219
  * Ertan et al. (2020) Amy Ertan, Georgia Crossland, Claude Heath, David Denny, and Rikke Jensen. 2020.  Cyber Security Behaviour In Organisations.  https://doi.org/10.48550/arXiv.2004.11768 arXiv:arXiv:2004.11768
  * Felt et al. (2015) Adrienne Porter Felt, Alex Ainslie, Robert W. Reeder, Sunny Consolvo, Somas Thyagaraja, Alan Bettes, Helen Harris, and Jeff Grimes. 2015\.  Improving SSL Warnings: Comprehension and Adherence. In _Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems_. ACM, Seoul Republic of Korea, 2893–2902.  https://doi.org/10.1145/2702123.2702442
  * Feng et al. (2021) Yuanyuan Feng, Yaxing Yao, and Norman Sadeh. 2021.  A Design Space for Privacy Choices: Towards Meaningful Privacy Control in the Internet of Things. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ _(CHI ’21)_. Association for Computing Machinery, New York, NY, USA, 1–16.  https://doi.org/10.1145/3411764.3445148
  * Harbach et al. (2014) Marian Harbach, Markus Hettig, Susanne Weber, and Matthew Smith. 2014.  Using Personal Examples to Improve Risk Communication for Security & Privacy Decisions. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_ _(CHI ’14)_. Association for Computing Machinery, New York, NY, USA, 2647–2656.  https://doi.org/10.1145/2556288.2556978
  * Kahneman (2012) Daniel Kahneman. 2012\.  _Thinking, Fast and Slow_.  Farrar, Straus and Giroux.
  * Karegar et al. (2020) Farzaneh Karegar, John Sören Pettersson, and Simone Fischer-Hübner. 2020.  The Dilemma of User Engagement in Privacy Notices: Effects of Interaction Modes and Habituation on User Attention.  _ACM Transactions on Privacy and Security_ 23, 1 (Feb. 2020), 5:1–5:38.  https://doi.org/10.1145/3372296
  * Kelley et al. (2010) Patrick Gage Kelley, Lucian Cesca, Joanna Bresee, and Lorrie Faith Cranor. 2010.  Standardizing Privacy Notices: An Online Study of the Nutrition Label Approach. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_ _(CHI ’10)_. Association for Computing Machinery, New York, NY, USA, 1573–1582.  https://doi.org/10.1145/1753326.1753561
  * Kitkowska et al. (2022) Agnieszka Kitkowska, Johan Högberg, and Erik Wästlund. 2022\.  Online Terms and Conditions: Improving User Engagement, Awareness, and Satisfaction through UI Design. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ _(CHI ’22)_. Association for Computing Machinery, New York, NY, USA, 1–22.  https://doi.org/10.1145/3491102.3517720
  * Kitkowska et al. (2020) Agnieszka Kitkowska, Mark Warner, Yefim Shulman, Erik Wästlund, and Leonardo A. Martucci. 2020\.  Enhancing Privacy through the Visual Design of Privacy Notices: Exploring the Interplay of Curiosity, Control and Affect. In _Proceedings of the Sixteenth USENIX Conference on Usable Privacy and Security_ _(SOUPS’20)_. USENIX Association, USA, 437–456.
  * Nouwens et al. (2020) Midas Nouwens, Ilaria Liccardi, Michael Veale, David Karger, and Lalana Kagal. 2020.  Dark Patterns after the GDPR: Scraping Consent Pop-ups and Demonstrating Their Influence. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_. 1–13.  https://doi.org/10.1145/3313831.3376321 arXiv:2001.02479 [cs]
  * Obar and Oeldorf-Hirsch (2018) Jonathan A. Obar and Anne Oeldorf-Hirsch. 2018.  The Biggest Lie on the Internet: Ignoring the Privacy Policies and Terms of Service Policies of Social Networking Services.  https://doi.org/10.2139/ssrn.2757465
  * Office of the California Attorney General (2018) Office of the California Attorney General. 2018.  California Consumer Privacy Act (CCPA).  https://oag.ca.gov/privacy/ccpa.
  * Redmiles et al. (2018) Elissa M. Redmiles, Michelle L. Mazurek, and John P. Dickerson. 2018.  Dancing Pigs or Externalities? Measuring the Rationality of Security Decisions. In _Proceedings of the 2018 ACM Conference on Economics and Computation_ _(EC ’18)_. Association for Computing Machinery, New York, NY, USA, 215–232.  https://doi.org/10.1145/3219166.3219185
  * Reeder et al. (2011) Rob Reeder, Ellen Cram Kowalczyk, and Adam Shostack. 2011\.  Poster: Helping Engineers Design NEAT Security Warnings.  _SOUPS 2011_ (2011).
  * Reinhardt et al. (2021) Daniel Reinhardt, Johannes Borchard, and Jörn Hurtienne. 2021\.  Visual Interactive Privacy Policy: The Better Choice?. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ _(CHI ’21)_. Association for Computing Machinery, New York, NY, USA, 1–12.  https://doi.org/10.1145/3411764.3445465
  * Ryan and Deci (2000) Richard M Ryan and Edward L Deci. 2000.  Self-Determination Theory and the Facilitation of Intrinsic Motivation, Social Development, and Well-Being.  _American Psychologist_ (2000).
  * Salehzadeh Niksirat et al. (2021) Kavous Salehzadeh Niksirat, Evanne Anthoine-Milhomme, Samuel Randin, Kévin Huguenin, and Mauro Cherubini. 2021.  I Thought You Were Okay: Participatory Design with Young Adults to Fight Multiparty Privacy Conflicts in Online Social Networks. In _Designing Interactive Systems Conference 2021_ _(DIS ’21)_. Association for Computing Machinery, New York, NY, USA, 104–124.  https://doi.org/10.1145/3461778.3462040
  * Salehzadeh Niksirat et al. (2023) Kavous Salehzadeh Niksirat, Diana Korka, Hamza Harkous, Kévin Huguenin, and Mauro Cherubini. 2023.  On the Potential of Mediation Chatbots for Mitigating Online Multiparty Privacy Conflicts - A Wizard-of-Oz Study.  _Proceedings of the ACM on Human-Computer Interaction (PACMHCI) / ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW)_ 7, CSCW1 (2023), 142:1–142:33.  https://doi.org/10.1145/3579618
  * Schaub et al. (2017) Florian Schaub, Rebecca Balebako, and Lorrie Faith Cranor. 2017\.  Designing Effective Privacy Notices and Controls.  _IEEE Internet Computing_ 21, 3 (May 2017), 70–77.  https://doi.org/10.1109/MIC.2017.75
  * Schaub et al. (2018) Florian Schaub, Rebecca Balebako, Adam L. Durity, and Lorrie Faith Cranor. 2018.  A Design Space for Effective Privacy Notices*.  In _The Cambridge Handbook of Consumer Privacy_ (first ed.), Evan Selinger, Jules Polonetsky, and Omer Tene (Eds.). Cambridge University Press, 365–393.  https://doi.org/10.1017/9781316831960.021
  * Schechter et al. (2007) Stuart E. Schechter, Rachna Dhamija, Andy Ozment, and Ian Fischer. 2007.  The Emperor’s New Security Indicators. In _Proceedings of the 2007 IEEE Symposium on Security and Privacy_ _(SP ’07)_. IEEE Computer Society, USA, 51–65.  https://doi.org/10.1109/SP.2007.35
  * Sunshine et al. (2009) Joshua Sunshine, Serge Egelman, Hazim Almuhimedi, Neha Atri, and Lorrie Faith Cranor. 2009\.  Crying Wolf: An Empirical Study of SSL Warning Effectiveness. In _Proceedings of the 18th Conference on USENIX Security Symposium_ _(SSYM’09)_. USENIX Association, USA, 399–416.
  * Tabassum et al. (2018) Madiha Tabassum, Abdulmajeed Alqhatani, Marran Aldossari, and Heather Richter Lipford. 2018\.  Increasing User Attention with a Comic-based Policy. In _Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems_ _(CHI ’18)_. Association for Computing Machinery, New York, NY, USA, 1–6.  https://doi.org/10.1145/3173574.3173774
  * The European Parliament and the Council of the European Union (2016) The European Parliament and the Council of the European Union. 2016.  Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the Protection of Natural Persons with Regard to the Processing of Personal Data and on the Free Movement of Such Data, and Repealing Directive 95/46/EC (General Data Protection Regulation).
  * Wang et al. (2013) Na Wang, Jens Grossklags, and Heng Xu. 2013.  An Online Experiment of Privacy Authorization Dialogues for Social Applications. In _Proceedings of the 2013 Conference on Computer Supported Cooperative Work_ _(CSCW ’13)_. Association for Computing Machinery, New York, NY, USA, 261–272.  https://doi.org/10.1145/2441776.2441807
  * Wang et al. (2014) Yang Wang, Pedro Giovanni Leon, Alessandro Acquisti, Lorrie Faith Cranor, Alain Forget, and Norman Sadeh. 2014.  A Field Trial of Privacy Nudges for Facebook. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_ _(CHI ’14)_. Association for Computing Machinery, New York, NY, USA, 2367–2376.  https://doi.org/10.1145/2556288.2557413
  * Windl et al. (2022) Maximiliane Windl, Niels Henze, Albrecht Schmidt, and Sebastian S. Feger. 2022.  Automating Contextual Privacy Policies: Design and Evaluation of a Production Tool for Digital Consumer Privacy Awareness. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ _(CHI ’22)_. Association for Computing Machinery, New York, NY, USA, 1–18.  https://doi.org/10.1145/3491102.3517688
  * Wogalter (2006) Michael Wogalter. 2006\.  Purposes and Scope of Warnings.  _Handbook of Warnings_ (Jan. 2006), 3–9.
  * Wu et al. (2006) Min Wu, Robert C. Miller, and Simson L. Garfinkel. 2006\.  Do Security Toolbars Actually Prevent Phishing Attacks?. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_. ACM, Montréal Québec Canada, 601–610.  https://doi.org/10.1145/1124772.1124863

##  Appendix A Appendix

Figure 1. Outline of design dimensions, principles and features for privacy
notices and security warnings

* refers to features from security warning design not fully reflected in the privacy notice design space and
** refers to a suggested separate dimension of the design space.

##  Appendix B Appendix

Figure 2. Mosaic of privacy notices examples
Figure 3. Mosaic of security warnings examples