# Rules Of Engagement: Levelling Up To Combat Unethical CUI Design

Thomas Mildner  University of BremenBremenGermany<EMAIL_ADDRESS>0000-0002-1712-0741  ,  Gian-Luca Savino  University of
St.GallenSt.GallenSwitzerland<EMAIL_ADDRESS>0000-0002-1233-234X ,
Philip Doyle  University College DublinDublinIreland
<EMAIL_ADDRESS>0000-0002-2686-8962 and  Rainer Malaka  University
of BremenBremenGremany<EMAIL_ADDRESS>0000-0001-6463-4828

(2022)

###### Abstract.

While a central goal of HCI has always been to create and develop interfaces
that are easy to use, a deeper focus has been set more recently on designing
interfaces more ethically. However, the exact meaning and measurement of
ethical design has yet to be established both within the CUI community and
among HCI researchers more broadly. In this provocation paper we propose a
simplified methodology to assess interfaces based on five dimensions taken
from prior research on so-called dark patterns. As a result, our approach
offers a numeric score to its users representing the manipulative nature of
evaluated interfaces. It is hoped that the approach - which draws a
distinction between persuasion and manipulative design, and focuses on how the
latter functions rather than how it manifests - will provide a viable way for
quantifying instances of unethical interface design that will prove useful to
researchers, regulators and potentially even users.

ethical design, dark patterns, conversational user interfaces, evaluation
methods and techniques

††journalyear: 2022††copyright: acmlicensed††conference: 4th Conference on
Conversational User Interfaces; July 26–28, 2022; Glasgow, United
Kingdom††booktitle: 4th Conference on Conversational User Interfaces (CUI
2022), July 26–28, 2022, Glasgow, United Kingdom††price: 15.00††doi:
10.1145/3543829.3544528††isbn: 978-1-4503-9739-1/22/07††booktitle: ††ccs:
Human-centered computing User interface design††ccs: Human-centered computing
User centered design††ccs: Human-centered computing HCI theory, concepts and
models

##  1\. Introduction

Although ethical interface design has been a concern in HCI for some time,
researchers are only beginning to formulate ways to understand, define and
measure occurrences of unethical interface design. In terms of attempts to
build knowledge around this issue, most progress can be found in literature on
dark patterns, which began just over a decade ago. In 2010, Brignull
introduced the term of dark patterns as “tricks used in websites and apps that
make you do things that you didn’t mean to” (Brignull et al., 2010).
Throughout the past decade, many examples of dark patterns have been
identified, generating a rich taxonomy of specific dark pattern types and dark
pattern strategies (Brignull et al., 2010; Bösch et al., 2016; Greenberg et
al., 2014; Conti and Sobiesk, 2010; Gray et al., 2018, 2020; Mathur et al.,
2019; Zagal et al., 2013). While this is a positive move in terms of better
understanding unethical interface design practises, the body of work suffers
somewhat from a lack of consensus, continuity, and clear definition about what
exactly is being examined across various different studies. This creates a
significant problem when trying to devise ways to protect users against their
potentially harmful effects, including efforts to help users identify
unethical design practises more easily, and attempts to regulate them.

This lack of consensus and continuity in how concepts are defined also echoes
a general problem in human-machine dialogue (HMD) research, which has been
highlighted in a number of recent reviews of research in the field (Clark et
al., 2019; Seaborn and Urakami, 2021a, b; Brüggemeier et al., 2020). The aim
of this provocation paper is to draw attention to the potential for learning
from the problems within these bodies of work and to further encourage efforts
to bring clarity to how we define and measure examples of unethical design in
CUIs. As a research community, we might then be able to establish more
effective ways of identifying and measuring unethical design practises in the
context of CUI development before they become as common and problematic as
they already in graphical interface interaction.

##  2\. Fundamentals: Persuasion & Manipulation

To a certain degree persuasion is a fundamental feature of design. Ideally,
the aim is to design objects that signify their potential uses and constraints
so people can interact with them as intuitively as possible. That is, we look
to create objects that gently nudge people toward using them in a certain way.
This is a common general understanding of design in HCI research, and echoes
widely known basic principles for ensuring alignment between a user’s mental
model and a system forwarded by Norman in ’The Design of Everyday Things’
(Norman, 2002). Yet, in recent years the notion of persuasive design has begun
to acquire negative connotations, largely due to a rise in design techniques
aimed at governing and exploiting peoples’ decision-making. Thaler and
Sunstein provide an example of this type of design technique when introducing
the term of Nudges (Leonard, 2008) to describe interventions that alternate
peoples’ decision making process in a predictable way, allowing design to be
used to navigate a users’ focus into a predefined direction or goal. However,
approaches of this nature, that take advantage of our cognitive biases, can be
used equally efficiently for benevolent or malevolent ends; as is evidenced by
the growth in research on dark patternsacross numerous domains.

This raises questions about how we define persuasive design, and indeed, how
we draw distinctions between designs regarded as persuasive versus
manipulative. Nudges have certainly been used in beneficial ways, including
improving peoples’ eating (Sarcona et al., 2017), healthcare (Zhao et al.,
2016), fitness (MEDICA, 2018), sleep (Hosszu et al., 2019) and relaxation
practices (Weekly et al., 2018). It would also be inaccurate to assume people
are naive to how these systems work, with the implicit nature of persuasive
cognitive approaches being both part of the appeal and part of the reason they
are effective (Madhusanka and Ramadass, 2021). However, they also share
fundamental similarities with designs used to encourage users to make
potentially harmful decisions; so-called dark patterns. They disarm a person
of their autonomy, albeit temporarily, by encouraging them to make choices
they might not have made otherwise. Therein lies our ethical obligation and
challenges as HCI researchers: to ensure we combat unethical acts of
manipulative interface design, whilst ensuring we continue to deliver
persuasive interface designs that improve the lives of people who use them. We
argue that establishing clear conceptual definitions is the first step toward
consistent identification and measurement of unethical interface design in
CUIs.

Based on established psychological definitions, we suggest drawing a
distinction between persuasive design and manipulative design. Here,
persuasion is defined as, _“an active attempt by one person to change another
person’s attitudes, beliefs, or emotions associated with some issue, person,
concept, or object”_ (of Psychology, 2022a). To bring this into context with
HCI research, we suggest a slight adaptation, defining persuasive design as,
_‘an active attempt to influence a person’s behaviours, attitudes, beliefs, or
emotions associated through interface design’_. On the other hand,
manipulation is described in psychology as, _‘behaviour designed to exploit,
control or otherwise influence others to one’s own advantage’_ (of Psychology,
2022b); or in the context of HCI, _‘designs aimed at exploiting, controlling
or otherwise influencing users to one’s own advantage’_. Our challenge in
bringing clarity to this distinction lies in developing ways to identify and
measure occurrences of unethical manipulative design, which will aid us in
defending the benefits of using psychological knowledge to improve people’s
lives through technology also.

##  3\. Understanding and Measuring Unethical Design: Lessons from Dark
Patterns

| Brignull
---
2010 (Brignull et al., 2010)
| Conti & Sobiesk
---
2010 (Conti and Sobiesk, 2010)
| Zagal et al.
---
2013 (Zagal et al., 2013)
| Greenberg et al.
---
2014 (Greenberg et al., 2014)
| Bösch et al.
---
2016 (Bösch et al., 2016)
| Gray et al.
---
2018 (Gray et al., 2018)
| Gray et al.
---
2020 (Gray et al., 2020)
| Mathur et al.
---
2019 (Mathur et al., 2019)
| · Trick Questions
---
· Sneak Into Basket
· Roach Motel
· Privacy Zuckering
· Confirmshaming
· Disguised Ads
| · Price Comparison
---
Prevention
· Misdirection
· Hidden Costs
· Bait and Switch
· Forced Continuity
· Friend Spam
| · Coercion
---
· Distraction
· Forced Work
| · Manipulating
---
Navigation
| · Restricting
---
Functionality
· Trick
· Confusion
· Exploiting Errors
· Interruption
· Obfuscation
· Shock
| · Grinding
---
· Impersonation
· Monetized Rivalries
· Pay to Skip
| · Playing by
---
Appointment
| · Pre-Delivered
---
Content
| · Social Pyramid
---
Schemes
| · Attention Grabber
---
· Bait and Switch
| · The Social Network
---
Of Proxemic Contracts
Or Unintended
Relationships
· Captive Audience
· We Never Forget
| · Disguised Data
---
Collection
| · Making Personal
---
Information Public
· The Milk Factor
| · Privacy Zuckering
---
| · Hidden Legalese
---
Stipulations
· Shadow User Profiles
· Bad Defaults
· Immortal Accounts
· Information Milking
· Forced Registration
| · Address Book
---
Leeching
| · Nagging
---
· Obstruction
· Sneaking
· Interface Interference
· Forced Action
| · Automating the User
---
· Two-Faced
· Controlling
· Entrapping
· Nickling-And-Diming
· Misrepresenting
| · Countdown Timers
---
| · Limited-time
---
Messages
| · High-demand
---
Messages
· Activity Notifications
· Confirmshaming
| · Testimonials
---
of Uncertain
Origins
· Hard to Cancel
· Visual Interference
· Low-stock Messages
· Hidden Subscriptions
· Pressured Selling
· Forced Enrollment

Table 1. This table lists 62 types of dark pattern described in prior
research.

[This table shows types of dark patterns described in prior research.]This
table presents a total of 62 types of dark patterns divided by their original
publication over 8 columns. These columns are as follows: 1\. Brignull et al.
· Trick Questions · Sneak Into Basket · Roach Motel · Privacy Zuckering ·
Confirmshaming · Disguised Ads · Price Comparison Prevention · Misdirection ·
Hidden Costs · Bait and Switch · Forced Continuity · Friend Spam

2\. Conti & Sobiesk · Coercion · Distraction · Forced Work · Manipulating
Navigation · Restricting Functionality · Trick · Confusion · Exploiting Errors
· Interruption · Obfuscation · Shock

3\. Zagal et al. · Grinding · Impersonation · Monetized Rivalries · Pay to
Skip · Playing by Appointment · Pre-Delivered Content · Social Pyramid Schemes

4\. Greenberg et al. · Attention Grabber · Bait and Switch · The Social
Network Of Proxemic Contracts Or Unintended Relationships · Captive Audience ·
We Never Forget · Disguised Data Collection · Making Personal Information
Public · The Milk Factor

5\. Bösch et al. · Privacy Zuckering · Hidden Legalese Stipulations · Shadow
User Profiles · Bad Defaults · Immortal Accounts · Information Milking ·
Forced Registration · Address Book Leeching

6\. Gray et al. · Nagging · Obstruction · Sneaking · Interface Interference ·
Forced Action

7\. Gray et al. · Automating the User · Two-Faced · Controlling · Entrapping ·
Nickling-And-Diming · Misrepresenting

8\. Mathur et al. · Countdown Timers · Limited-time Messages · High-demand
Messages · Activity Notifications · Confirmshaming · Testimonials of Uncertain
Origins · Hard to Cancel · Visual Interference · Low-stock Messages · Hidden
Subscriptions · Pressured Selling · Forced Enrollment

Research into unethical interface design practises began with examinations of
e-commerce websites by Brignull (Brignull et al., 2010). The work identified
twelve specific examples of unethical design aimed at inhibiting people’s
ability to make informed choices. These include, Sneak into Basket, Hidden
Costs, and Price Comparison Prevention, which all operate on the premise of
obscuring information and potentially misleading users into buying unwanted or
unnecessarily expensive products. Other dark patterns defined by the author
include, Forced Continuity, Privacy Zuckering, and Roach Motel, which are all
tactics aimed at forcing people to sign-up, or stay signed up for accounts and
services they might not require anymore. These examples are also used by
service providers to gain access to private data without fully informing users
why it is being collected, who has access to it, or even what it might be used
for. Since then, researchers have described and defined a plethora of other
examples, with a recent review of the literature from Mathur et al. (Mathur et
al., 2021) identifying 62 specific types of dark patterns. For readability,
Table 1 offers a complete overview. Understanding how these existing dark
patterns were established and how they are used to facilitate unethical design
in graphical user interfaces (GUIs) could allow us to prevent similar
developments in CUIs.

While it may seem like unethical GUI design has been a perennial problem for
HCI researchers, literature on dark patterns shows that these tactics and the
form they take develop and change over time through conscious efforts made by
interface design practitioners. Grey et al. (Gray et al., 2018; Gray and
Chivukula, 2019) identified multiple dark patterns as well as design
constraints of practitioners which lead to their creation. In a first study,
the authors analysed an image-based corpus to define five types of dark
patterns that practitioners engage in when developing manipulative designs
(Gray et al., 2018). Most of Gray at al.’s dark patterns include descriptions
from prior works. For example, the Obstruction dark pattern, used to make
processes unnecessarily difficult, incorporates Brignull’s Roach Motel, Price
Comparison Prevention, and Intermediate Currency (Brignull et al., 2010). In a
follow up to this work, Gray et al. also analysed 4775 user-generated posts of
the Reddit sub-forum r/assholedesign (Gray and Chivukula, 2019). Analysis
resulted in identifying six properties of “asshole designers”. The work is
particularly useful for understanding the origins of dark patterns and how
they emerge from constraints under which practitioners work.

While many of the previously described dark patterns are applicable to
different domains, research has also found domain specific examples. For
example, Zagal et al. identified seven dark patterns that related to video
game mechanics (Zagal et al., 2013). While certain patterns exploit a game’s
ecosystem of connected users, such as Social Pyramid Schemes and
Impersonation, others impact game-play experience like Grinding and Playing by
Appointment. Elsewhere, Greenberg et al. (Greenberg et al., 2014) consider
dark patterns in conjunction with proxemics theory (Hall, 1966). Identifying
nine types of dark pattern in total, the authors discuss interactions with
manipulative design in spatial environments. The Attention Grabber and
Disguised Data Collection dark patterns, for instance, could be used in the
design of digital billboards that exploit people’s proximity and personal data
to deliver personalised advertising.

Understanding the creation of dark patterns and analysing their occurrences
helps to close the “cultural lag” (Ogburn, 1922) where the creation of ethical
guidelines inevitably lags behind the release of novel dark patterns and even
novel technologies. However, recent attempts by the authors to apply the
aforementioned corpus of dark patterns in the domain of social media highlight
a central problem in this body of work: it is hallmarked by a high degree of
overlap between definitions, inconsistent terminology, and descriptions that
operate on different levels often without explicit acknowledgement. That is,
the difference between specific dark pattern designs and broader dark pattern
strategies is not always recognised. Attempts to apply taxonomies of dark
patterns also shows that while some established dark patterns types are
applicable to other domains, it is also likely that novel CUI specific dark
patterns types and strategies will need to be described and defined. Further,
while this is particularly problematic from a research perspective, we see an
even more urgent need to begin this work so we might also better protect
users.

##  4\. Ways to Combat Unethical Interface Design

With regards to technology, particularly online interfaces, studies have shown
that the burden to counteract dark patterns often falls on users (Di Geronimo
et al., 2020; Bongard-Blanchy et al., 2021). Although regulations, such as the
GDPR (Commission, 2016) or the CCPA (Legislature, 2018), aim to protect users
in online environments, the previously mentioned cultural lag (Ogburn, 1922)
means these efforts are struggling to counter all problematic designs
described under the umbrella of manipulative design. Indeed, one could argue
they have led to the creation of new dark patterns. The design of cookie
consent banners that favour ’accept all’ options over offering users greater
control, shows how design can be used to easily negate efforts to combat dark
patterns, and how current regulatory efforts fail to address the fundamental
nature of manipulative design. This has lead to designs exploiting cognitive
biases not covered by the GDPR, such as anchoring effects, to steer users into
giving consent that they might not have given were they provided with a
neutral choice (Gray et al., 2021; Mathur et al., 2019).

In January 2022, the European Union proposed a new article 13a, as part of the
Digital Service Act (Parliament, 2022), to address previous concerns. Offering
a generalised definition of problematic design that is closely worded to
Mathur et al’s. (Mathur et al., 2021) definition of dark patterns, article 13a
contains an extendable list of specific interface designs to be regulated.
However, we argue that this approach may lead to a similar problem seen in
attempts to apply dark pattern taxonomies across different interfaces and
domains. That is, the taxonomies quickly become very large, difficult to
maintain, and not always appropriate across interfaces types and use cases.
Further, creating an ever growing list of examples may also deepen this
problem over-complicating crucial efforts to combat unethical design.

| Mathur et al. 2019 (Mathur et al., 2019)
---
Dark Pattern Characteristics
Characteristic |  Question
Asymmetric |  Does the user interface design impose unequal weights or burdens on the available choices presented to the user in the interface?
Covert |  Is the effect of the user interface design choice hidden from the user?
Deceptive |  Does the user interface design induce false beliefs either through affirmative misstatements, misleading statements, or omissions?
Hides Information |  Does the user interface obscure or delay the presentation of necessary information to the user?
Restrictive |  Does the user interface restrict the set of choices available to users?

Table 2. This table lists the introductory questions Mathur et al. (2019)
(Mathur et al., 2019) gave for each dark pattern characteristic.

[Dark Pattern Characteristics]This table lists the introductory questions
Mathur et al. (2019) gave for each dark pattern attribute. The attributes and
their introductory questions are as follows: 1\. Asymmetric: Does the user
interface design impose unequal weights or burdens on the available choices
presented to the user in the interface?

2\. Covert: Is the effect of the user interface design choice hidden from the
user?

3\. Deceptive: Does the user interface restrict the set of choices available
to users?

4\. Hides Information: Does the user interface design induce false beliefs
either through affirmative misstatements, misleading statements, or omissions?

5\. Restrictive: Does the user interface obscure or delay the presentation of
necessary information to the user? [Dark Pattern Characteristics]This table
lists the introductory questions Mathur et al. (2019) gave for each dark
pattern attribute. The attributes and their introductory questions are as
follows: 1\. Asymmetric: Does the user interface design impose unequal weights
or burdens on the available choices presented to the user in the interface?

2\. Covert: Is the effect of the user interface design choice hidden from the
user?

3\. Deceptive: Does the user interface restrict the set of choices available
to users?

4\. Hides Information: Does the user interface design induce false beliefs
either through affirmative misstatements, misleading statements, or omissions?

5\. Restrictive: Does the user interface obscure or delay the presentation of
necessary information to the user?

##  5\. Levelling up to Unethical Interface Design

By categorising specific dark pattern types into five broad characteristics,
Mathur et al. (Mathur et al., 2019) offer an alternative and potential useful
avenue for combating manipulative designs. This higher level categorisation is
based on the cognitive biases specific designs are developed to exploit. By
relying on more fundamental concepts, which are much less interchangeable than
specific interface layouts, the approach offers a way to understand
manipulative design that focuses on their impact on users, in a broadly
applicable fashion, whilst remaining agnostic to the interface or domain in
question. By being based on cognitive biases, instead of mere dark pattern
definitions that stem from GUI domains, this model allows enhanced evaluation
outside its original scope, such as CUIs.

Figure 1. This figure demonstrates the derivation of a five-dimensional dark
pattern vector, which can further be reduced into a single digit value,
inspired by Mathur et al.’s five dark pattern characteristics and attributes
(Mathur et al., 2019). Although the example is based on a screenshot of a
mobile interface, it could be easily adapted for any conversational user
interface.

[This figure demonstrates the derivation of a five-dimensional dark pattern
vector, which can further be reduced into a single digit value, inspired by
Mathur et al.’s five dark pattern characteristics and attributes (Mathur et
al., 2019). Although the example is based on a screenshot of a mobile
interface, it could be easily adapted for any conversational user
interface.]The figure shows a five step procedure to get a dark pattern score
based on Mathur et al.’s five dark pattern characteristics. The first step
shows an illustration of a screenshot containing multiple dark patterns. In
the second step, these dark patterns are linked to each applying attribute
respectively. In this exemplar, the attributes asymmetry, covert, and
deception are recognised. The attributes information hiding and restriction
are not. Steps four and five are then deriving a score for

In an attempt to advance their model and differentiate between manipulative
and persuasive designs (i.e, between dark patterns and bright patterns), we
developed a technique that allows us to evaluate individual examples based on
characteristics that stem from the cognitive biases they target. We therefore
consider each of the five characteristic by asking a specific question to
assess impact on each of these dimensions, as seen in Table 2. By assigning
each a value from 1 (not at all) to 5 (extremely), we are able to evaluate
persuasive designs across the five dimensions, indicating the degree to which
they might be regarded as persuasive or manipulative. The benefit of this
approach is that we gain an overall score that can be used to determine the
degree to which a specific design is either persuasive or manipulative, whilst
identifying which dimensions of persuasion a manipulative design targets.
Depending on the context and situation in which a design is evaluated, the
score determining the degree of persuasive versus manipulative design can be
adapted by alternating the threshold set to identify what is acceptable design
depending on the score. Figure 1 visualises the steps of this process. By
providing this clearly defined and measurable conceptualisation of persuasive
and manipulative design, this model yields a certain duality. On one hand, we
might help regulators combat unethical interface design practices in a
consistent and broadly applicable fashion, whilst protecting the benefits of
persuasion. On the other, this approach could allow practitioners to evaluate
their own designs through user studies.

##  6\. Current Caveats To Keep In Mind

This provocation paper addresses unethical design in CUIs but discusses the
topic with the means of manipulation and exploitation of cognitive biases. We
understand a distinction between ethical design and measurement of unethical
practices. Yet, we argue that by learning which unethical practices are at
play, we are able to compare and understand practitioners’ strategies better
while promoting more conscious handling of design techniques that, in the
wrong context, may exploit cognitive biases harming the user. The currently
available amount of tools to assess the good or bad in design is limited while
a growing demand to evaluate interfaces ethically can be seen across
disciplines. By utilising knowledge about cognitive biases, and exploitation
thereof, we are able to understand malicious interface strategies better and
can further classify them to share new knowledge between research communities.
As a basis for this research, cognitive biases describe basic behaviour
traits, certain strategies and heuristics under which decisions are formed,
shared among all humans. Building on this existing knowledge, we aim to
establish a robust measurement that also allows for comparison of interfaces.

We acknowledge the early stage of this endeavour and are aware of current
limitations. Arguably, a numeric and finite score as a determiner for how
ethical an interface is may be appealing to different cohorts, whether in the
context of regulation or user interface design. As all ordinal scales,
however, the proposed approach can only represent a limited abstraction of
manipulative dimensions in an interface. Moreover, the questionnaire has not
yet been verified and thus it is uncertain how effective it will prove to be
in action. In this early stage, we rely on the five characteristics proposed
by Mathur et al. (Mathur et al., 2019). By rooting their characteristics in
cognitive bias research, they gain the advantage of being similarly effective
across domains. Still, it is questionable whether these exact five dimensions
are as effective in the context of CUIs when compared to their origin in GUIs.
This could easily be addressed in future research by investigating the
differences and unanimity between cognitive biases exploited throughout
different kinds of interface modalities. Further studies could then look at
the variety of described cognitive biases to identify exploitation across
domains to offer alternative sets of questions and target each modality
precisely. For example, where colourisation of certain buttons may promote
some choices over others in GUI contexts, in voice based CUIs colour of voice,
emotional speech, and pitch could be misused for similar deceptions.

##  7\. Conclusion

In this provocation piece, we argue that current attempts to account for
manipulative interface design are curtailed by a lack of clarity and
continuity around how concepts are defined; hampering efforts to measure and
combat their use. We also argue that there are valuable lessons to learn from
previous work on GUI dark patterns that might help us head off these problems
in the realm of CUIs. Indeed, the approach we suggest may prove useful across
multiple domains and interface contexts, and stands to benefit researchers,
regulators and potentially users also. Further, by aligning the
conceptualisation with how manipulative designs function, rather than how they
manifest, we hope to make it much more difficult for designers to
circumnavigate. Manipulative design that targets our cognitive biases
represents a real and pertinent danger to people, and difficulties faced, even
by knowledgeable users and regulators, highlight an urgent need to develop
quantifiable approaches that can be easily understood by a range of
stakeholders involved in the fight against unethical interface design
practices.

###### Acknowledgements.

The research of this project was partially supported by the Klaus Tschira
Stiftung gGmbH.

## References

  * (1)
  * Bongard-Blanchy et al. (2021) Kerstin Bongard-Blanchy, Arianna Rossi, Salvador Rivas, Sophie Doublet, Vincent Koenig, and Gabriele Lenzini. 2021.  ”I Am Definitely Manipulated, Even When I Am Aware of It. It’s Ridiculous!” - Dark Patterns from the End-User Perspective. In _Designing Interactive Systems Conference 2021_ (Virtual Event, USA) _(DIS ’21)_. Association for Computing Machinery, New York, NY, USA, 763–776.  https://doi.org/10.1145/3461778.3462086
  * Bösch et al. (2016) Christoph Bösch, Benjamin Erb, Frank Kargl, Henning Kopp, and Stefan Pfattheicher. 2016\.  Tales from the Dark Side: Privacy Dark Strategies and Privacy Dark Patterns.  _Proc. Priv. Enhancing Technol._ 2016, 4 (2016), 237–254.
  * Brignull et al. (2010) Harry Brignull, Marc Miquel, Jeremy Rosenberg, and James Offer. 2010.  Dark Patterns-User Interfaces Designed to Trick People.  http://darkpatterns.org/ visited on 18-02-2022.
  * Brüggemeier et al. (2020) Birgit Brüggemeier, Michael Breiter, Miriam Kurz, and Johanna Schiwy. 2020.  User Experience of Alexa when controlling music: comparison of face and construct validity of four questionnaires. In _Proceedings of the 2nd Conference on Conversational User Interfaces_. 1–9.
  * Clark et al. (2019) Leigh Clark, Philip Doyle, Diego Garaialde, Emer Gilmartin, Stephan Schlögl, Jens Edlund, Matthew Aylett, João Cabral, Cosmin Munteanu, Justin Edwards, et al. 2019\.  The state of speech in HCI: Trends, themes and challenges.  _Interacting with Computers_ 31, 4 (2019), 349–371.
  * Commission (2016) European Commission. 2016\.  GDPR-16 2016. Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation) (Text with EEA relevance).  https://ec.europa.eu/commission/sites/beta-political/files/data-protection-factsheet-changes_en.pdf
  * Conti and Sobiesk (2010) Gregory Conti and Edward Sobiesk. 2010.  Malicious interface design: exploiting the user. In _Proceedings of the 19th international conference on World wide web - WWW ’10_. ACM Press, Raleigh, North Carolina, USA, 271.  https://doi.org/10.1145/1772690.1772719
  * Di Geronimo et al. (2020) Linda Di Geronimo, Larissa Braz, Enrico Fregnan, Fabio Palomba, and Alberto Bacchelli. 2020.  UI Dark Patterns and Where to Find Them: A Study on Mobile Applications and User Perception. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_ (Honolulu, HI, USA) _(CHI ’20)_. Association for Computing Machinery, New York, NY, USA, 1–14.  https://doi.org/10.1145/3313831.3376600
  * Gray and Chivukula (2019) Colin M. Gray and Shruthi Sai Chivukula. 2019.  Ethical Mediation in UX Practice.  In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_. Association for Computing Machinery, New York, NY, USA, 1–11.  https://doi.org/10.1145/3290605.3300408
  * Gray et al. (2020) Colin M. Gray, Shruthi Sai Chivukula, and Ahreum Lee. 2020\.  _What Kind of Work Do ”Asshole Designers” Create? Describing Properties of Ethical Concern on Reddit_.  Association for Computing Machinery, New York, NY, USA, 61–73.  https://doi.org/10.1145/3357236.3395486
  * Gray et al. (2018) Colin M. Gray, Yubo Kou, Bryan Battles, Joseph Hoggatt, and Austin L. Toombs. 2018.  _The Dark (Patterns) Side of UX Design_.  Association for Computing Machinery, New York, NY, USA, 1–14.  https://doi.org/10.1145/3173574.3174108
  * Gray et al. (2021) Colin M. Gray, Cristiana Santos, Nataliia Bielova, Michael Toth, and Damian Clifford. 2021.  _Dark Patterns and the Legal Requirements of Consent Banners: An Interaction Criticism Perspective_.  Association for Computing Machinery, New York, NY, USA, pp. 1–18.  https://doi.org/10.1145/3411764.3445779
  * Greenberg et al. (2014) Saul Greenberg, Sebastian Boring, Jo Vermeulen, and Jakub Dostal. 2014.  _Dark Patterns in Proxemic Interactions: A Critical Perspective_.  Association for Computing Machinery, New York, NY, USA, 523–532.  https://doi.org/10.1145/2598510.2598541
  * Hall (1966) Edward T. Hall. 1966\.  _The hidden dimension_.  Doubleday, Garden City, N.Y.
  * Hosszu et al. (2019) Alexandra Hosszu, Daniel Rosner, and Michael Flaherty. 2019\.  Sleep Tracking Apps’ Design Choices: A Review. In _2019 22nd International Conference on Control Systems and Computer Science (CSCS)_. IEEE, 426–431.
  * Legislature (2018) California State Legislature. 2018\.  CCPA-18 2018. California Consumer Privacy Act of 2018 [1798.100 - 1798.199] (CCPA).  https://leginfo.legislature.ca.gov/faces/codes_displayText.xhtml?division=3.&part=4.&lawCode=CIV&title=1.81.5.
  * Leonard (2008) Thomas C. Leonard. 2008\.  Richard H. Thaler, Cass R. Sunstein, Nudge: Improving decisions about health, wealth, and happiness.  _Constitutional Political Economy_ 19, 4 (Dec. 2008), 356–360.  https://doi.org/10.1007/s10602-008-9056-2
  * Madhusanka and Ramadass (2021) BGDA Madhusanka and Sureswaran Ramadass. 2021.  Implicit intention communication for activities of daily living of elder/disabled people to improve well-being.  In _IoT in Healthcare and Ambient Assisted Living_. Springer, 325–342.
  * Mathur et al. (2019) Arunesh Mathur, Gunes Acar, Michael J. Friedman, Elena Lucherini, Jonathan Mayer, Marshini Chetty, and Arvind Narayanan. 2019\.  Dark Patterns at Scale.  _Proceedings of the ACM on Human-Computer Interaction_ 3, CSCW (nov 2019), 1–32.  https://doi.org/10.1145/3359183
  * Mathur et al. (2021) Arunesh Mathur, Mihir Kshirsagar, and Jonathan Mayer. 2021\.  _What Makes a Dark Pattern… Dark? Design Attributes, Normative Considerations, and Measurement Methods_.  Association for Computing Machinery, New York, NY, USA, pp. 1–18.  https://doi.org/10.1145/3411764.3445610
  * MEDICA (2018) EDIZIONI MINERVA MEDICA. 2018\.  Fitness mobile apps positively affect attitudes, perceived behavioral control and physical activities.  _The Journal of sports medicine and physical fitness_ (2018).
  * Norman (2002) Donald A. Norman. 2002\.  _The design of everyday things_.  Basic Books, [New York].  http://www.amazon.de/The-Design-Everyday-Things-Norman/dp/0465067107/ref=wl_it_dp_o_pC_S_nC?ie=UTF8&colid=151193SNGKJT9&coliid=I262V9ZRW8HR2C
  * of Psychology (2022a) APA Dictionary of Psychology. 2022a.  Manipulation.  https://dictionary.apa.org/persuasion visited on 18-02-2022.
  * of Psychology (2022b) APA Dictionary of Psychology. 2022b.  Manipulation.  https://dictionary.apa.org/manipulation visited on 18-02-2022.
  * Ogburn (1922) W.F. Ogburn. 1922\.  _Social Change with Respect to Culture and Original Nature_.  B.W. Huebsch, Incorporated, University of California.
  * Parliament (2022) European Parliament. 2022\.  Digital Servicess Act***I. European Parliament [A9-0356/2021].  https://www.europarl.europa.eu/doceo/document/TA-9-2022-0014_EN.html
  * Sarcona et al. (2017) Alessandra Sarcona, Laura Kovacs, Josephine Wright, and Christine Williams. 2017.  Differences in eating behavior, physical activity, and health-related lifestyle choices between users and nonusers of mobile health apps.  _American Journal of Health Education_ 48, 5 (2017), 298–305.
  * Seaborn and Urakami (2021a) Katie Seaborn and Jacqueline Urakami. 2021a.  _Measuring Voice UX Quantitatively: A Rapid Review_.  Association for Computing Machinery, New York, NY, USA.  https://doi.org/10.1145/3411763.3451712
  * Seaborn and Urakami (2021b) Katie Seaborn and Jacqueline Urakami. 2021b.  Measuring voice UX quantitatively: A rapid review. In _Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems_. 1–8.
  * Weekly et al. (2018) Taelyr Weekly, Nicole Walker, Jill Beck, Sean Akers, and Meaghann Weaver. 2018.  A review of apps for calming, relaxation, and mindfulness interventions for pediatric palliative care patients.  _Children_ 5, 2 (2018), 16.
  * Zagal et al. (2013) José P Zagal, Staffan Björk, and Chris Lewis. 2013\.  Dark Patterns in the Design of Games. In _Proceedings of the 8th International Conference on the Foundations of Digital Games (FDG 2013)_ (May 14-17). Society for the Advancement of the Science of Digital Games, Chania, Crete, Greece, 39–46.  http://www.fdg2013.org/program/papers.html
  * Zhao et al. (2016) Jing Zhao, Becky Freeman, Mu Li, et al. 2016\.  Can mobile phone apps influence people’s health behavior change? An evidence review.  _Journal of medical Internet research_ 18, 11 (2016), e5692.