# Invisible, Unreadable, and Inaudible Cookie Notices: An Evaluation of Cookie
Notices for Users with Visual Impairments

James M Clarke  University of Surrey<EMAIL_ADDRESS>&
<EMAIL_ADDRESS>Maryam Mehrnezhad  Royal Holloway University of London,
<EMAIL_ADDRESS>Ehsan Toreini  University of Surrey,
<EMAIL_ADDRESS>&<EMAIL_ADDRESS>
###### Abstract

This paper investigates the accessibility of cookie notices on websites for
users with visual impairments (VI) via a set of system studies on top UK
websites (n=46) and a user study (n=100). We use a set of methods and
tools–including accessibility testing tools, text-only browsers, and screen
readers, to perform our system studies. Our results demonstrate that the
majority of cookie notices on these websites have some form of accessibility
issues including contrast issues, not having headings, and not being read
aloud immediately when the page is loaded. We discuss how such practises
impact the user experience and privacy and provide a set of recommendations
for multiple stakeholders for more accessible websites and better privacy
practises for users with VIs. To complement our technical contribution we
conduct a user study and finding that people with VIs generally have a
negative view of cookie notices and believe our recommendations could help
their online experience. We also find a disparity in how users wish to respond
to cookie notices as apposed to how they do in reality.

##  1 Introduction

Visual impairment (VI) is a term used to describe any type of vision loss,
ranging from partial vision loss to someone who cannot see at all [1, 2].
People with VI have various types of assistive technologies (AT) available to
help them browse the internet [56], e.g., text-only browsers and screen
readers. Screen readers are installed on users’ computers or phones to read
information by outputting it as sound [49, 56, 65]. They work with the browser
and interpret the code that is used to build web pages [6]. Screen readers are
not capable of conveying visual and spatial information, such as layout and
images, to the user unless relevant meta-information is provided in the web
page code through _markups_. In addition, the text is only presented line by
line, making it harder to get an overview of the page [49]. This also makes it
more difficult to understand the relations of a website’s different parts and
identify navigation links. To ensure that AT can correctly interpret websites,
there are various accessibility standards, such as the Web Content
Accessibility Guidelines (WCAG) provided by the World Wide Web Consortium
(W3C) [63]. WCAG aims to provide a shared standard for web content
accessibility. The WCAG documents explain how to make web content more
accessible to disabled people. To be included in the WCAG, issues must impact
disabled people with greater effect than those without disabilities [64].

The majority of websites employ some type of tracking, using various
techniques such as cookies and fingerprinting [12]. There are two types of
cookies, functional and non-functional [30], with the most common use of non-
functional cookies being for personalised advertising [70, 55, 59]. A simple
method to counteract this type of tracking is to allow users to manage which
cookies are stored on their device [53]. With the implementation of the
General Data Protection Regulation (GDPR) in 2018, companies operating in the
EU and the UK and/or handling EU/UK citizens data need to choose a legal basis
to collect and process user data [48]. One of the most well-known of these is
cookie notices to gain consent from users [33]. Alongside the GDPR, the
ePrivacy Directive and the Information Commissioner Office (ICO) give specific
guidance on obtaining consent through cookie notices [42, 34].

Previous research has shown that individuals want to protect themselves from
online tracking [8, 47], though they are not always confident [8, 36].
Multiple studies have looked at how the function and presentation of cookie
notices differ [36, 34, 24, 61, 9]. Similarly, there are studies showing that
the designs of cookie notices can affect users’ interactions [61], including
through dark patterns [40]. Previous research has examined the effect of GDPR
and cookie notices on the number of cookies [30, 9, 24]. It has been shown
that there is a disparity between the requirement of data protection laws, the
practises of websites, and users behaviour regarding online tracking
protection [36].

Limited research has been conducted on privacy and VIs. Users with VIs have
previously been found to have concerns about being tracked online [25],
similar to others [8, 47]. There has also been research looking at VI and
online information credibility [6]. In the context of cookie notices and VI,
research is extremely sparse [52]. There are some reports on usability issues
with cookie notices while looking at the wider accessibility of websites [66].
To the best of our knowledge, there is no research on cookie notices and AT
where a comprehensive range of methods is utilised. Our research questions
include:

  * •

RQ1: How do websites and cookie notices comply with the web content
accessibility guidelines and the general data protection regulations? RQ1-a:
How do popular websites comply with the current accessibility guidelines
(e.g., WCAG) and the GDPR? RQ1-b: Does compliance necessarily mean good
privacy practices for VI users?

  * •

RQ2: Can the existing automated accessibility tools evaluate cookie notices?
RQ2-a: How do the current cookie notices score with the automated
accessibility tools (e.g., WAVE and Google Lighthouse)? RQ2-b: Does a high
score necessarily mean good practice for VI users?

  * •

RQ3: How do cookie notices interface with AT? RQ3-a: How does the mainstream
AT (e.g., text-only browsers and screen readers) interact with cookie notices?
RQ3-b: How do the current practises impact VI users’ privacy?

  * •

RQ4: What are the general perception and practice of VI users regarding cookie
notices? RQ4-a: What issues have VI users encountered with cookie notices?
RQ4-b: Who do participants believe is responsible for online accessibility?

This paper contributes to the body of knowledge via its system studies, user
studies, and the discussions and recommendations that we provide for improving
the online privacy of users with VIs. First, we provide a set of evaluation
methods based on the off-the-shelf tools for AT and for users with VI. This
enables us and other researchers to conduct system experiments and assess
websites and cookie notices for their accessibility. Second, using these
methods and tools, we run experiments on 46 popular UK websites (according to
Alexa) and report a wide range of accessibility issues with their cookie
notices. Table 1 presents an overview of our system studies. Third, we conduct
user studies with 100 UK participants who use AT and extract their perception,
practices, and preferences regarding cookies notices on websites. The results
of our systems studies as well as the user studies confirm that current
practises are far from ideal in protecting the privacy of users with VIs.
Finally, we discuss the impact of these practises on user privacy and provide
recommendations for web developers, AT designers, policymakers, and end users
to improve the privacy of real-world practises.

##  2 Background and Related Work

Differential Vulnerabilities recognise how different populations face
different types and degrees of security and privacy risks [44]. This
challenges the universalising tendencies that frame cybersecurity around an
abstract or generic user who either does not exist or is only a subset of
actual end users [11, 35]. This ties into social sciences research looking at
models of disability such as the Critical Realist Model [14]. Both of these
threads consider the real-world lived experiences of disabled people, as well
as their thoughts. With differential vulnerabilities considering how different
threats can arise for different user groups. Studying and evaluating the
privacy of users with VIs is challenging. The common range of privacy
assessment methods would not be directly useful here. Instead, such approaches
should be combined with accessibility assessment methods, as defined in the
accessible writing guidelines of the Association for Computing Machinery [21].

According to the Office of National Statistics in 2020, almost 11 million
adults with disabilities recently used the Internet in the UK [41]. A 2016
GOV.UK survey of 712 AT users found that 29% used a screen reader to browse
the Internet, while others used screen magnifiers, speech recognition or
readability software [18]. They also found several different screen readers
being used, the most popular being JAWS. WebAIM also found that JAWS was the
most popular screen reader with 53.7% of users using it as their primary
screen reader and NVDA was the second most popular with 30.7% of users using
it as their primary screen reader [68].

###  2.1 Users with VI and Privacy

Evaluating the accessibility of websites is possible through a number of
automatic and manual ways and through the use of a range of tools such as
screen readers and text-only browsers. For example, Southwell and Slater have
previously used the WCAG to evaluate university library finding aids [56].
They used an automated web-accessibility checker, WAVE 5.0 by WebAim, to
perform an initial assessment of each finding aid and then manually tested
each website using the WebbIE 3 Web browser, which is a non-graphical text-
only browser. They also used screen readers directed by keyboard navigation
including _System Access to GO_ and _NVDA_. When using the automated checker,
they found that most of the websites tested (58 of 65) had at least one
accessibility error. The most common errors were missing document language,
missing alternative text, missing form labels, and linked images missing
alternative text. They then used the non-graphical browser, finding only 68%
had headings that enabled navigation to another important section of the
document. Of those which had enough headings, they did not always have the
headings in proper sequential order, or were missing first-level headings.
Fewer sites offered links for navigation, 57% did, 43% did not, and 25% of the
sites lacked both headings and links for any kind of navigation. Using the
screen readers, they found that the main content of all 65 finding aids was
readable; this opposes the 89% error rate noted by the automatic checker.

Table 1: Overall view of our system studies  Method | (I) Cookie notice | (II) General | (III) Manual | (IV) Manual
---|---|---|---|---
| & Tracking | Automated | Testing via | Testing via
| Behaviour Evaluation | Accessibility Tools | Text-only Browser | Screen Readers
Tools | Google Chrome, Brave | WAVE, | WebbIE | JAWS, NVDA
|  | Google Lighthouse |  |
Website Accessibility | NA | Yes | Yes | Yes
Assessment |  |  |  |
Cookie Notice | Yes (General, Baseline) | Partial (Accessibility) | Yes (Accessibility) | Yes (Accessibility)
Assessment |  |  |  |

There is scarce research on the security and privacy of users with VI. Brulé
et al. analysed 178 papers on technologies designed for people with VI, with
the aim of facilitating and stimulating future research [5]. Inan et al.
surveyed 20 individuals who are visually impaired to investigate their
internet use and explore their cybersecurity challenges and concerns while
browsing the internet [25]. They found a number of problems, such as automatic
web page refreshing and missing or improper headings. In this study, the
possibility of someone tracking their internet activities was the highest-
rated concern. The authors suggest that it is important to guide the user to
enable security and privacy settings and to provide accessible software
solutions to protect and warn this marginalised group. Hayes et al. shadowed
and interviewed people with VI and their allies [23]. Finding that self-
perceptions can influence someone’s behaviour, which could have privacy and
security implications, such as hiding or concealing their disability due to
perceived stigma. Akter et al. studied 155 people with VIs privacy concerns
relating to camera-based AT [3]. Finding that users of these systems were more
concerned about the privacy of others, who may inadvertently be captured in
their images, than themselves. However, camera-based AT can create a lack of
personal security in the lives of the people they are trying to help. Previous
research reports that users with VIs often find it difficult to complete their
security task and that they had moderately high levels of concern about
cybersecurity [39]. Similarly, there are reports on the complications of
authentication methods such as passwords and two-factor authentication for
users with VIs [51]. An exploratory user study, conducted using semi-
structured in-person interviews with 14 people with VIs found that
participants were aware and concerned about privacy and security and faced a
variety of risks [2].

More relevant to this paper is the work of Schnell and Roy [52]. They
conducted an evaluation of a select group of 40 educational and financial
website cookie notices using WCAG [52]. Finding that even for users without
disabilities, there were challenges to accessing, understanding, and
processing privacy information. Also, finding that educational websites were
more accessible than financial websites, however, not all websites complied
with the WCAG’s criteria chosen for their testing. In contrast to this work,
we offer a comprehensive evaluation method to review website cookie notices
and apply our methods to a range of websites rather than only educational and
financial.

Although there have been a number of user studies looking generally at users
who are VI and security, to the best of our knowledge, there have been none
looking specifically at cookie notices. In this paper, we aim to address this
gap via a series of system studies and a dedicated user study with users who
have VIs, both focusing on cookie notices.

###  2.2 AT Regulations, Standards, and Tools

According to GDPR, cookie notices should be presented on all websites that use
cookies and should include opt-out options, as well as opt-in options without
highlighting the latter and including any privacy or cookie walls. Cookie
notices should be separated from other matters such as privacy policy and
terms and conditions, and the user should be able to opt-out of the previously
accepted cookie settings with the same ease as they gave the consent. Enabling
non-essential cookies before the user’s consent is a non-compliant practice
too. Based on Article 12 et seq. GDPR [16]: “The controller shall take
appropriate measures to provide any information referred to in Articles 13 and
14 and any communication under Articles 15 to 22 and 34 relating to processing
to the data subject in a concise, transparent, intelligible, and easily
accessible form, using clear and plain language, in particular for any
information specifically addressed to a child. The information shall be
provided in writing, or by other means, including, where appropriate, by
electronic means. When requested by the data subject, the information may be
provided orally, provided that the identity of the data subject is proven by
other means.” This article interprets as the data controller (i.e. web tracker
in the context of this paper), must inform every user about the nature of the
data to be collected and the purposes of such collection. Hence, websites need
to be fully compliant with the regulations and also offer usable practises to
comply with further requirements. The ambiguity of how practises should
include marginalised users has not been discussed widely, and only limited
examples are available. For example, in the verdict of an Italian case in
which the data controller was mandated to provide the information acoustically
for video surveillance [15].

There are many aspects to the real-world implementation of accessible web
technologies [43]. For instance, an accessible web design approach should
support enhancing the visual characteristics of the front–end design and
utilise a range of colours, while ensuring the contrast of the colours is
accessible to users who are visually impaired or colour–blind. Also, they need
to build an audio commentary for the page and the images. The interconnected
nature of web pages (as various resources fetched from different origins in
the page) could potentially increase the complexity of fully accessible web
design. To harmonise such practises, the W3C has provided a comprehensive list
of 167 tools to evaluate accessibility compatibility
measurements111w3.org/WAI/ER/tools/. They are implemented on a number of
platforms and technologies, some supporting cross-platform products. These
products include 20 support APIs, 14 authoring tool plugins, 45 browser
plugins, 19 command line tools, 25 desktop applications, 4 mobile
applications, and 90 online tools.

There are a number of standards and regulations worldwide to provide
accessibility requirements for the technologies to be considered as _publicly
presentable_ in a region, country, or regional-based regulations e.g.,
European, Italian, Irish, Israeli, Japanese, Korean, and US Federal Law,
platform (web accessibility frameworks), e.g., various versions of WCAG (2.1,
2.0 and 1.0), or file formats, e.g., EPUB standards.

Looking at the standards based on the support for VI, we can conclude that all
of them recognise such disabilities and provide a standardised set of
guidelines for the implementation of such support. In general, they offer
similar suggestions to mitigate such disabilities. For example, these
standards support VI by advising to provide a form of AT and non-visual access
to support visually impaired users, including the proprietary agent (in this
case, the dedicated hardware or special browsers), an audio description to
explain the important visual detail, the high contrast visualisation, adoption
of flash thresholds, magnification, reduction of the required field of vision,
and control of contrast, brightness, and intensity. Following these guidelines
can contribute towards meeting the minimum requirements for complying with
such regulations.

In our observations, most accessibility tools are based on W3C standard family
(WCAG 2.1 (85 tools out of 167), WCAG 2.0 (139 tools), WCAG 1.0 (46 tools)).
Moreover, some of them comply with country-specific regulations such as German
standards (21 tools), French standards (12 tools), Japanese standards (18
tools), EU standards (9 tools), US federal procurement (67 tools), Irish
National IT Accessibility Guidelines (16 tools), Israeli web accessibility
guidelines (7 tools), Italian accessibility legislation (11 tools), Korean
standards (1 tool). Finally, format-specific standards such as EPUB
accessibility 1.0 are only supported in 3 tools.

###  2.3 Online Tracking

Adopted in April 2016 and implemented in May 2018, the GDPR changed the rules
on online tracking and consent (including consent to cookies) [48, 33]. In
order to process personal data, companies must choose a legal basis for
processing. One of the most well-known is consent. Valid consent must be
freely given, specific, informed, unambiguous, explicit, revocable, given
prior to any data collection, and requested in a readable and accessible
manner [48]. The ePrivacy Directive (“ePD”, aka “cookie law”) [42], provides
supplementary rules to the GDPR. According to the ePD website, publishers must
rely on user consent when collecting and processing personal data using non-
mandatory (not strictly necessary for the services requested by the user)
cookies or other technologies [42]. This is in accordance with the guidance
given by the European Data Protection Board and the ICO [13, 27]. Various
studies (e.g., [61, 9, 34, 59, 36, 24]) exist on the implementation and
effectiveness of cookie notices, privacy banners, and tracking practises.
Examples of dark patterns include providing invalid consent, nudging the user,
making opting-out difficult, not providing the user with opting-out options
from previously accepted cookies settings, pre-enabling non-essential cookies,
and including trackers in the cookie notice itself. For example, the top 5
consent management platforms have been reported to use dark patterns and
implied consent [40].

There is a body of knowledge on the user dimensions of tracking, including
concerns and negative feelings of users about tracking [47], differences
between demographics such as gender and country [8], the disparity between
regulations, website practises and users’ limited knowledge for protecting
against tracking and their demand for more transparency and control [36, 54,
46, 37, 60]. What is lacking in the previous work is the measurement of the
current practises in the wild for web tracking notices for users with visually
impairments. In this paper, we aim to run experiments in order to fill this
gap.

##  3 Accessibility Evaluation Methodology

In this section, we present our methodology for the evaluation of the
websites.

Our assessment includes a number of different methods and tools including
automated accessibility testing tools, a non-graphical browser and screen
readers, as explained at length in this section. The overall design of our
experiments and the tools used in each part is presented in Table 1. We have
included a website analysis template in Appendix A.

All experiments took place between April and October 2022 on a laptop PC
running Windows with a screen size of 13.3 inches and a resolution of 3840 x
2160. Windows is the most commonly used desktop OS among screen reader users
according to the WebAIM 2021 survey [68]. As a case study, we use Alexa’s top
50 UK websites in April 2022. We selected this sample since GDPR is a regional
regulation (EU/UK). We looked at English websites for analysis in our fluent
language. Based on Alexa, the popular UK websites are comparable to others in
Europe, e.g., Germany and France. From this list, four websites are excluded
because they redirect to another website already on the list or are down.
_t.co_ is an example of a website that was excluded due to redirecting to
_twitter.com_ , however, both _amazon.com_ and _amazon.co.uk_ are retained.
The US version of the site (.com) does not contain a cookie notice, whereas
the UK version (.co.uk) does; therefore, it was important to keep both sites
on the list for comparison. These are just examples, and the full list is
presented in Table 8 (Appendix). The cookie notice experiments were conducted
by two researchers to ensure consistency. A researcher performed accessibility
testing twice with one specialist software/tool, by recording the results in
tables (Appendix). Due to the rounds of experiments taking place over the
course of six months, we believe this demonstrates stability in our results.

###  3.1 Cookie Notices

All 46 websites were visited using Google Chrome (Version 103.0.5060.134
(Official Build) (64-bit)) and Brave222brave.com (Version 1.41.99 Chromium:
103.0.5060.134 (Official Build) (64-bit)). Using Google Chrome without a
screen reader acts as a baseline and gave an example of how sighted users
would see the site and the cookie notice. Chrome is one of the most popular
browsers with the highest market share in 2022 [57]. Brave is a secure browser
that was created in 2016 by two former executives of Mozilla Corporation, the
company that makes the Firefox browser [32]. Brave comes with a feature called
Brave Shields built in, which includes several privacy-preserving features.
Brave adopts various privacy-enhancing techniques which are not possible at
the browser extension level (due to access restrictions and performance
limitations), making it a powerful tool to observe the tracking behaviours of
websites. It is commonly used for assessing the tracking behaviour of websites
on PC and mobile platforms [34, 36]. We completed these experiments before the
introduction of cookie notice blocking by Brave [58].

For each of the 46 websites, we open them in these two browsers and record the
location and control options given to users. When recording the details, we do
not interact with the website in any way, including not interacting with
notifications (e.g. requesting location permission, update notifications). To
ensure that no cookies had previously been cached, each website is viewed in a
new private or incognito window.

We also record which options are given to the user in the cookie notice
according to the categories suggested in similar work, e.g. [34, 36]. These
categories include: (i) _Agree or Reject_ : where two options are presented,
Agree (Agree, Accept, OK, Understand, etc.) or Reject (Reject, Decline, No,
etc.), with the same level of control (e.g., two buttons). These are further
categorised by which option is emphasised. (ii) _Agree or Settings_ : where
two options are presented, Agree or Settings (Options, Settings, Policy,
Manage, Learn more, etc.), again with the same level of control. Which are
further categorised by which option is emphasised. (iii) _Agree, Reject, or
Settings_ : where three options are presented; Agree, Reject, and Settings.
These are further categorised on the basis of which item is highlighted in the
notice. (iv) _No Notice_ : The website does not display a cookie notice.

###  3.2 Accessibility Evaluation

Our accessibility evaluation consists of two parts. First, we use automated
testing tools, which are designed to give developers an overview of how
accessible their website is [63]. This allows us to get an impression of the
overall accessibility of a website, and in some cases includes information
about the accessibility of the cookie notice. Second, we use software designed
for individuals with VI in the real world to assess the results of the
automated testing tools and to allow us to more specifically focus on cookie
notices. In this section, we explain these approaches.

####  3.2.1 Automated Accessibility Testing Tools

Websites are evaluated using two different automated accessibility testing
tools, WebAIM WAVE 5.0 Web Accessibility Evaluation Tool333wave.webaim.org/
and Google Lighthouse444developer.chrome.com/docs/lighthouse/overview/.

WAVE is an automated accessibility tool that we use to perform an initial
assessment of the conformance of each website to WCAG. WAVE generates a report
containing Errors, Alerts, Features, Structural elements, and ARIA landmarks.
Errors indicate issues that will impact certain users with disabilities, as
well as showing failures to meet the requirements of the WCAG. Whereas alerts
are elements which may cause accessibility issues, but need further manual
testing to determine this. Features are elements that can improve
accessibility if implemented correctly. Structural elements are HTML regions
and headings, and ARIA can be used to present important accessibility
information to people with disabilities. WAVE has been used in previous work,
e.g. Southwell and Slater, when evaluating university library finding aids
[56]. During their testing, they used WAVE to perform an initial evaluation of
the conformity of each finding aid to Section 508 and WCAG 2.0 guidelines. We
tested the web version of WAVE 5.0 in our preliminary testing and it did not
detect any cookie notices. Therefore, we use the browser extension
version555wave.webaim.org/extension/ for our experiments. The WAVE extension
evaluates the rendered version of the web page allowing dynamically generated
content to be evaluated [69], while the WAVE Web version may not be able to
apply all the scripting on the page. This is a possible reason for the cookie
notices not being displayed during our preliminary tests.

We use Google Lighthouse to give an overall accessibility score, as well as to
record specific problems with each website. Lighthouse is an open-source
automated testing tool, which can audit performance, accessibility, and more
[17]. We only test accessibility using the default (navigation) mode and while
representing a desktop device. We record the score out of 100 and the
individual issues with each website. This score is a weighted average of all
accessibility audits it performs, with weighting based on axe user impact
assessments [10]. The axe user impact assessments are composed of WCAG rules
with some supplementary rules added [26].

Both WAVE and Google Lighthouse give an overview of accessibility for a whole
website, however, WAVE also allowed us to view where specific problems
occurred.

Manual Testing via Text-only Browser: To complement and verify the results of
the testing tools, we apply a range of methods to manually assess the privacy
practises of these websites via their cookies notices. We visit all these
websites using WebbIE666webbie.org.uk/, a text-only browser for people with
VI. The WebbIE Ctrl-H and Ctrl-L commands are used to examine the heading and
links on a page. This approach has been used in similar work, e.g., work [56].
WebbIE was uninstalled and reinstalled for each round of testing, as it does
not have a private browsing mode or cookie manager. Through this method, we
examine how users navigate the page and if and how cookie notices are
displayed. We assign each website to one of the following categories: (i) _No
Headings_ : The website in general has no headings which can be used for
navigation. (ii) _Basic Headings_ : The website has some headings but there is
a limited amount which is not useful for navigation. (iii) _Full Headings_ :
The website has a number of headings that are useful for navigation.

Headings allow screen readers and other accessibility software to navigate
around a webpage. For example, WebbIE can move easily to different headings on
a website allowing for quicker navigation and locating key information, e.g. a
cookie notice. The categories above are derived from previous work [56], where
similar categories were used to evaluate the accessibility of library finding
aids. Similarly, we observe the website’s behaviour in presenting the cookie
notice and each website’s cookie notice was also put into the following
categories: _(i) Headings throughout_ : Headings are available throughout the
cookie notice. _(ii) Heading at the start_ : A heading is present at the start
of the notice, however, there were no other headings in the body of the
notice. _(iii) No headings_ : There are no headings present in the cookie
notice at all. _(iv) Notice missing_ : The cookie notice is not shown when
using WebbIE, however, one is present when using the graphical browsers. _(v)
No notice_ : The website does not have a cookie notice when viewed with the
graphical browser.

The _Headings throughout_ category for cookie notices is based on the _Full
Headings_ category for the website as a whole. Meaning that a user would be
able to navigate the cookie notice using heading-based navigation; this is
particularly useful for longer cookie notices as seen on some websites. The
_Heading at the start_ category is used to classify notices that only have a
heading at the start. This would allow for navigation to the notice itself but
means that a user would have to rely on a different type of navigation within
the notice, e.g. line-by-line or link-based navigation. Whereas _No headings_
would mean a user would not be able to use heading navigation at all within
the cookie notice and would have to rely on another form of navigation. In
some instances when viewed graphically, a website did display a cookie notice,
however, when using WebbIE one was not present. For this reason, we include a
_Notice missing_ category to signify this. Whereas with the _No notice_ , a
notice was not present on the website when viewed graphically. We included
different categories for headings (Basic, Full, and No headings) since we
found lengthy cookie notices on some websites (e.g., google.com,
facebook.com), however, headings are not always needed due to a number of
cookie notices being shorter.

####  3.2.2 Manual Testing via Screen Readers

In order to have more comprehensive and conclusive results, we also carry out
our experiments using screen readers to manually test each website. JAWS and
NVDA were chosen as the most popular according to WebAIM [68], 53.7% and
30.7%, respectively. We use these screen readers in conjunction with Google
Chrome as these are the most common combinations of screen reader and browser
[68], 32.5% and 16.0%, respectively. NVDA is a free OS-level screen reader
with support for popular applications such as web browsers, email clients,
music players, and office programmes. JAWS is another OS-level screen reader
that users need to purchase. For our experiments, we purchased a home licence
(£865 with the Authorisation USB Dongle). Both screen readers should have
similar reliability when parsing websites [67], however, they often parse
website code slightly differently [45]. It is for this reason that we use the
two most popular screen readers during our testing.

We categorise cookie notices based on the way these screen readers are able to
read them [62, 52]. Accordingly, each website’s cookie notice was given a pass
or fail for the following categories: _(1) Readable_ : The screen reader
software is able to read the cookie notice. _(2) Immediately Read_ : The
cookie notice is the first thing to be read from the page, excluding the page
title. _(3) Keyboard navigable_ : The cookie notice of a website is navigable
using a keyboard while using a screen reader. _(4) Link or button purpose_ :
The purpose of a link or button can be solely determined by the link or
button. _(5) Abbreviations are explained_ : All abbreviations are explained.
This was either in the cookie notice or the website offered some mechanism for
identifying the expanded form. _(6) Page titled_ : The page has a title that
describes its topic or purpose. _(7) Cookie notice titled_ : The cookie notice
has a title or heading which is readable by the screen reader software. _(8)
Headings useful for navigation_ : There are useful headings for navigation
present throughout the cookie notice. _(9) No timing_ : There is no timing for
reading the cookie notice.

The _Readable_ category is based upon WCAG 3.1, Readable, defined as “Make
text content readable and understandable” by WCAG [62], with the guideline
being used in previous work [52]. We created the _Immediately Read_ category
to show that a cookie notice is read close to the start of a web page. This is
important as a number of websites start tracking a user before they respond to
the notice, and therefore users must be able to respond to the notice at the
first given opportunity. Also, meaning that users do not have to actively
search the website for the notice to respond. The category _Link or button
purpose_ is based on WCAG 2.4.9, Link purpose (link only), which is defined as
“A mechanism is available to allow the purpose of each link to be identified
from the link text alone, except where the purpose of the link would be
ambiguous to users in general” [62]. _Abbreviations are explained_ is based
upon WCAG 3.1, Abbreviations, which W3C define as “A mechanism for identifying
the expanded form or meaning of abbreviations is available”. _Page titled_ is
also based on a WCAG, namely 2.4.2 Page Titled which is defined as “Web pages
have titles that describe topic or purpose”. We create another category based
on this called _Cookie notice titled_ , this is to judge if a cookie notice
can easily be navigated to. It also aligns with previous testing for headings,
as titles often consist of headings. Alongside this, we test for _Heading
useful for navigation_ , which is based on the previous heading testing with
WebbIE in cookie notices. It also aligns with WCAG 2.4.10 Section headings,
defined as “Section headings are used to organize the content”. We also define
the category _No timing_ which is based on WCAG 2.2.3 No timing. This is
defined as “Timing is not an essential part of the event or activity presented
by the content, except for non-interactive synchronized media and real-time
events”.

###  3.3 Limitations

To the best of our knowledge, this is the first work on the assessment of
cookie notices on a range of websites for users with VI. We simply chose to
test the 46 top websites in the UK (out of 50). In practise, the top Alexa
websites may not be the most popular websites for users with VI. However, we
could not find a formal report on popular websites for this group of users. We
acknowledge that this is a limited sample set and more research is required to
evaluate a larger number of websites. When testing websites, we only tested
the first page in our experiments. Although this is a common practise for the
privacy assessment of websites in general, it is not clear if all pages would
present the same information and produce the same output for AT. Further,
detailed work would be needed to explain how different web pages interact with
AT.

Previous research has demonstrated the usefulness of mobile technology for
people with VI, e.g. [19, 20]. However, due to the lack of research in this
area, we only generally focus on desktop web browsers, for which the majority
of the accessibility and AT tools and standards are also designed. Cross-
platform studies are left as future work.

##  4 Accessibility evaluation Results

Our results include (1) a general assessment of the cookie notices of the
websites and their tracking behaviour, and (2) an accessibility evaluation of
these websites and their cookie notices.

###  4.1 Cookie Notices and Tracking Behaviour

Cookie Notice Position: We observed that the majority of websites displayed a
cookie notice (n = 35 or 76.1%) when using Google Chrome. Of the positions, a
bottom overlay was the most common (n = 15 or 32.6%), followed by a middle
overlay (n = 7 or 15.2%). When using Brave, a higher number of web pages
displayed no notice (n = 15 or 32.6%). Other than this, the popularity of
categories is in the same order as that of Google Chrome. While there are some
papers (e.g. [61, 4]) looking at cookie notice positions and user engagement,
we could not find any for users with VI.

Cookie Notice Control Options: Of the options given when using Google Chrome,
Agree or Settings was the most common (17 or 37.0%). The most commonly
emphasised option along with Agree or Settings was Agree (13 or 28.3%). Table
2 describes the options presented to users in Chrome and Brave. These results
from Brave resemble those of Google Chrome; however, when using Brave, there
was a higher percentage of websites which displayed no notice. These results
are consistent with previous work (e.g., [34]) when cookie notices were
evaluated across platforms. The cookie notices of some websites are blocked
due to the notice itself being a tracker; resulting in it being blocked by
Brave.

Previous research studying GDPR compliance has focused on the following
requirements: consent must be explicit, accepting all is as easy as rejecting
all, and there are no pre-ticked boxes [40]. It has been shown that the pre-
selection of options can impact users’ choices when giving consent [61]. For
this reason and to respond to RQ1-a, in Table 2, we highlight categories that
are in violation of the above requirements and therefore in violation of GDPR.
As you can see, three categories (14 websites) comply with the above
requirements. However, we did not test them for additional GDPR compliance
items, such as opting out from previously accepted cookie notices with the
same ease of opting in.

Table 2: Cookie notices’ user control options in Chrome and Brave, as well as GDPR violations. | Emphasised | Browser | GDPR
---|---|---|---
Options | option | Chrome | Brave | violation
(i) Agree or Reject | None | 4 | 4 | No
| Agree | 4 | 4 | Yes
(ii) Agree | None | 4 | 4 | Yes
or Settings | Agree | 13 | 9 | Yes
(iii) Agree, Reject | None | 5 | 5 | No
or Settings | Agree & Reject | 5 | 5 | No
(iv) No Notice | 11 | 15 | Yes

Tracking Behaviour: We also observed these websites regarding their tracking
behaviour through Brave. Before interacting with the cookie notices only 3 of
the 46 websites (6.5%) did not have at least one item blocked by the Brave
Shields without any interaction with the cookie notice. The average number of
items blocked was 9, the maximum was 81, 11 of the websites had more than 10
items blocked, and 6 had more than 20. The majority of items blocked were in
the _trackers & ads_ category. Our results support similar work e.g., [34, 36,
33] reporting that the majority of websites start tracking the user regardless
of the presence of the cookie notice before any user interaction with it.

###  4.2 Automated Accessibility Testing Tools in Websites

WAVE ran on all but one website; when using it on _ebay.co.uk_ , the overlay
containing the results did not appear. Of the remaining sites, 42 (93.3%)
contained at least one accessibility error, with the average number of errors
being 18.98. Of the websites tested 35 (77.8%) contained at least one contrast
error. All websites tested contained at least one structural element with the
average being 84.02. Table 3 shows a summary of the results. We further break
these down into categories which could cause issues, e.g. errors, contrast
errors, and alerts, and those which could improve user experience, e.g.
features, structural elements, and ARIA.

Table 3: Summary of WAVE 5.0 test results for 46 websites. Criteria | no. of websites | Average no. of items
---|---|---
| with at least one | across websites
| item per criteria |
Cookie Notice | 33 | -
Errors | 42 | 18.98
Contrast Errors | 35 | 22.98
Alerts | 45 | 124
Features | 46 | 77.16
Structural Elements | 46 | 84.02
ARIA | 43 | 235.42

Errors are general issues that cause problems such as missing labels from HTML
code. While a contrast error would cause issues for someone with vision loss,
e.g., light text on a light background or dark text on a dark background.
Alerts are criteria that need further testing to establish if they hinder or
help accessibility. For example, for an image with long alternative text, a
long description could be needed to fully describe the image, or it may be
unjustified. Features are elements which work to improve a user’s experience.
For example, a form label is present and associated with a form control. This
is similar to structural elements such as headings and lists which also help
the user’s experience. ARIA is a set of roles and attributes that define ways
to make websites more accessible to people with VI. An example of an error
could be missing alternative text or a form control that does not have a
corresponding label. ARIA is only useful if implemented correctly such as when
an ‘aria-label’ or ‘aria-labelledby’ attribute is present which can be
interpreted by AT.

To complement this, we also note whether a cookie notice was present when
testing using WAVE. In some instances, we observed specific issues with the
cookie notice. The most common problems were with the low contrast between the
background of the cookie notice and the text, links, or buttons. Table 8
(Appendix) shows detailed results. Overall, the website with the lowest number
of issues was bbc.co.uk, with 0 errors, 0 contrast errors, 134 alerts, 23
features, 119 structural elements and 371 ARIA. There were multiple websites
with close to the same number of issues, namely xvideos.com, spankbang.com and
xnxx.com, all of which had between 165 and 176 items which would cause issues.

In addition, we used Google Lighthouse for the overall accessibility score of
the website. The average score was 89% (highest: 100%, lowest: 63%). Since
Google Lighthouse uses the axe user impact assessments, the overall score is
affected largely in the same way as individual WAVE tests. For example,
including a button that has an accessible name will improve the overall score
given to a web page. In general, these popular websites had a range of good
and poor accessibility practises when tested with these automated
accessibility tools. There were several websites we tested that achieved the
best possible Lighthouse score of 100%: bbc.co.uk, wikipedia.org, gov.uk,
paypal.com, microsoft.com, linkedin.com, and doubleclick.net. The lowest
score, 63%, was achieved by tiktok.com. which, according to Lighthouse, had a
number of labels and names missing, as well as navigation and contrast issues.
These scores for each website are shown in Table 8 (Appendix).

###  4.3 Manual Cookie Notice Accessibility Testing via AT Tools

Manual Testing via Text-only Browser: By using a text-only browser, we
performed an analysis on the overall accessibility of the websites and their
cookie notices. When using WebbIE, 27 of the 46 (58.7%) websites contained
_full headings_ which would be useful for navigation. With 7 (15.2%) of them
only having _basic headings_ and 12 (26.0%) containing _no headings_. The
inclusion of headings throughout the website does not directly impact privacy,
and was included in this analysis to give context to the accessibility of
cookie notices.

Percent of websites58.7%15.2%26.1%Headings at the start (13.0%)No headings
(17.4%)Notice missing (15.2%)No notice (13.0%)Headings throughout (2.2%)No
headings (4.4%)Notice missing (4.4%)No notice (4.4%)No headings (2.2%)Notice
missing (17.4%)No notice (6.5%)Full HeadingsBasic HeadingsNo Headings Figure
1: WebbIE accessibility testing; inner circle: the whole site, outer circle:
the cookie notice.

When observing the cookie notices, 17 (48.6%) of the 35 websites which
previously displayed a cookie notice did not display a cookie notice when
using WebbIE (_notice missing_). Furthermore, only 1 (2.9%) of the 35 websites
which had previously displayed a cookie notice website had _headings
throughout_ , and 6 (17.1%) had a _heading at the start_ of the notice. 11
(31.4%) of the websites’ cookie notices contained _no headings_ , although,
the majority of the websites which did not contain a notice did include links
to privacy and cookies. Regardless of the number of headings throughout the
website, we often found that cookie notices were missing. However, when a
website had full headings the cookie notice was more likely to have a heading
at the start. The results of these tests are shown in Figure 1, with the inner
circle representing the headings in the website as a whole and the outer
circle specifically looking at the cookie notice. The use of a heading at the
start of the cookie notice can make it easier to locate, due to this the lack
of headings seen in our testing could lead to problems. Headings inside the
notice can also make it easier to navigate within a cookie notice, especially
if it is lengthy, and therefore easier to make a decision.

Manual Testing via Screen Readers: When testing with NVDA, 29 (82.9%) of the
35 websites, which graphically included a cookie notice, contained a cookie
notice which could be read aloud. This result is higher than was expected
following the other test. However, it still means that 6 of the cookie notices
could not be read at all with NVDA. Of the cookie notices that could be read,
20 of the 35 (57.1%) were read aloud immediately when the website loaded.
Others were read aloud after other elements of the page had been read or had
to be specifically located to be read. 27 (77.1%) of the 35 cookie notices
were keyboard navigable, these were not always the same websites as those read
immediately. Therefore, this leaves 8 websites which users with VI may not be
able to navigate. In some cases, these cookie notices created keyboard traps
that the user would not be able to leave. Only 5 (14.3%) of the 35 cookie
notices contained a link or button that could be solely determined by the link
or button. Hence, without allowing time for the screen reader to output the
notice, the user may not understand what they are agreeing to.

Although all 46 (100%) websites contained a title, only 19 of the 35 (54.2%)
cookie notices contained a title. This means it would not be possible to
navigate to them using the heading, it could also make it more difficult to
search for the cookie notice. 35 of the 35 (100%) cookie notices did not have
any type of time limit on replying to the cookie notice. This is an excellent
result, meaning that users will have time to ingest the information and make a
decision. None of the 7 websites which contained abbreviations explained them,
meaning that if users are unfamiliar with these terms they may not understand
what they are consenting to. Also, none of the 35 websites’ privacy policies
contained headings which were useful for navigation, however, some did contain
different links. Due to this, it may be difficult to navigate the cookie
notices, which is particularly important for some of the longer cookie notices
we observed. We summarise these results in Table 4 (detailed results in Table
9 (Appendix)).

In comparison, JAWS enabled 34 (97.1%) of the 35 websites with a cookie notice
to be read out loud. Meaning all but one of the cookie notices could be read
aloud, which is a significantly better result than when using NVDA. Of these,
22 (62.9%) of the 35 were read aloud immediately when the website loaded,
which again is higher than when using NVDA. 29 of the 35 (82.9%) were keyboard
navigable, this is an improvement of 2 cookie notices over NVDA. The number of
cookie notices with a link or button that could be solely determined by the
link or button was also higher at 11 (31.4%) of the 35. All of the other
results were the same for JAWS as NVDA. These results are summarised in Table
4 and detailed results are available in Table 9 (Appendix). The reason for the
disparity in results is due to the fact that the screen readers parse webpages
differently, resulting in differing numbers of readable notices. This
underscores the importance (lack) of standardisation efforts.

Table 4: Number of websites which passed and failed each criterion of the manual testings via NVDA and JAWS. | NVDA | JAWS
---|---|---
Criteria | Pass | Fail | Pass | Fail
Readable | 29 | 6 | 34 | 1
Immediately read | 20 | 15 | 22 | 13
Keyboard navigable | 27 | 8 | 29 | 6
Link or button purpose | 5 | 30 | 11 | 24
Abbreviations are explained | 0 | 7 | 0 | 7
Page titled | 46 | 0 | 46 | 0
Cookie notice titled | 19 | 16 | 19 | 16
Headings useful for navigation | 0 | 35 | 2 | 33
No timing | 35 | 0 | 35 | 0

We identified poor practices on some of these websites. For instance, a news
website (dailymail.co.uk) read out adverts immediately before reading anything
else such as the navigation bar or the cookie notice. This is highlighted in
Figure 6 (Appendix). This is despite the fact that this website’s cookie
notice is displayed using a large portion of the website. Another example was
an online payment site (paypal.com), which read the body of the website aloud
before reading the cookie notice. This aligns with the cookie notice being
visually at the bottom of the page; however, this means that a user with VI
using a screen reader could easily miss the cookie notice. An example of the
visual representation of this notice and a scripted output of the website
while using JAWS is available in Figure 8 (Appendix). We highlight this
example, however, a similar output was common across multiple websites. One
social news website (reddit.com) was the only website with a cookie notice
which could not be read with either screen reader, even with intervention with
mouse input. Visually the cookie notice was located at the bottom of the
window, however, it could not be selected with the screen readers. A visual
example of the cookie notice is included in Figure 7 (Appendix). In contrast,
some of the websites presented the user with reasonable options when using a
screen reader. For instance, bbc.co.uk clearly presented the users with opt-in
and settings options. A scripted version of such output via NDVA screen reader
is provided in Figure 9 (Appendix).

##  5 User Study Methodology

In this Section, we explain the design of our online survey, data collection,
and analysis.

###  5.1 Questionnaire Design

When designing this survey, we followed the design principles for
questionnaires for people with VIs [28]. Specifically, we aimed to inform
participants about the topic of the survey before beginning the questionnaire,
we indicate the type of answer after each question, and we also start each
question with a consecutive number. We conducted an accessibility evaluation
before conducting the survey, using the tools mentioned above, during which we
did not find any issues.

Our questionnaire is made up of five sections—Internet and AT, Privacy-
enhancing technology usage, Cookie notices, Suggestions, and Demographics—with
the complete questionnaire included in Appendix B.

Internet and AT: After verifying the screening questions, we ask our
participants a few background questions about technology usage, such as what
devices and AT they use. We list different AT technologies including screen
readers, braille displays, text only browsers, magnification software, and
assistive browser extensions based on our research as well as allowing
participants to add additional items.

PETs usage: Next, after a brief explanation of PETs, we ask participants which
PETs they use listening to them according to the categorisation suggested in
[8], where the authors measure the correlation of people’s feelings about
tracking and their protective actions. These categories include: browsers do
not track, virtual private networks, private browsing, password manger,
privacy-focused web browsers, encrypted messaging apps, ad blockers, file
encryption tools, we additionally allowed participants to name other tools.

Cookie notices: We also ask our participants what they think cookie notices
are and what they think they are supposed to do. As well as how they feel
about cookie notices and how they interact with them. For the design of these
questions we followed [36, 8]. We ended this section by asking if they have
encountered issues with cookie notices, what they were, and why they think
they happen.

Suggestions: Finally, informed by the results of our website experiments, we
ask participants who they believe should be responsible for ensuring the
privacy and accessibility of websites. We also ask which of our suggestions
would help improve their experience online.

Demographics: We conclude by asking demographic questions.

###  5.2 Data Collection and Analysis

We conducted our user studies via Prolific Academic777prolific.co/ among UK
participants. We conducted one initial testing round of the survey with 10
participants, asking for feedback upon completion. We fixed minor typos and
made a few structural changes accordingly. At this stage and throughout
further participation we received no complaints relating to accessibility of
the questionnaire. We then distributed the questionnaire to a further 100
participants. We chose to use Prolific Academic to distribute the survey as
this user group is notoriously difficult to recruit, therefore using a paid
platform allowed us to gain this sample size. We rewarded participants at a
rate of £12 per hour, which is categorised as ‘Great’ by Prolific Academic.
This research received full approval from the University of Surrey’s Ethical
Committee before the research commenced.

Our method for processing the collected data is a mix of quantitative and
qualitative analysis. For our free-text style questions, we run thematic
analysis [7]; taking an inductive approach and allowing the data to determine
our themes. We are confident that adopting a deductive approach would have
yielded comparable themes. Two researchers conducted the thematic analysis
independently, and due to the small sample size all authors discussed and
agreed on these themes. Our research focused on exploring potential
differences between users with visual impairments and those of previous work,
allowing data to determine themes. For lengthy and complex responses multiple
themes were assigned to them. We also chose participant responses which
represent themes for inclusion in the paper.

###  5.3 Limitations

The interaction and intersection of online services and AT could be a complex
research topic to be investigated by user studies. To complement the technical
findings of this paper, we ran our studies on an online platform and through a
survey that provided us with self-reported responses which has its own
limitations. We plan to extend our user studies to one-to-one interviews as
well as focus groups to gain a deeper understanding of the implications of
this user group.

##  6 User Study Results

In this section, we present our findings of the user study. Our study was
completed by 100 participants who self certify as using AT and live in the UK.
Our participants occupy different jobs ranging from students, educators,
healthcare and social assistance to business, hospitality, and some not
working. 59 participants identify as male, 38 as female, 2 as non-binary, and
1 choosing not to say.

###  6.1 AT and PETs Usage

Half of the users surveyed use magnification software, 42 used a screen
reader, 22 use an assistive browser extension, with 9 not using any AT while
browsing the web, and other forms of AT having less than 15 users.
Participants use various online services, with the most popular being email
(98), shopping and e-commerce (93), and social media (90). Table 10 (Appendix)
gives an overview of the demographic questions.

In response to Q2.1, all but 3 participants use at least one of the PET
categories suggested, Figure 2 shows detailed results. The most popular
technology used was a password manager (67%), and the least popular was file
encryption tools (11%).

010203040506070Percent of participantsPassword managerAd blockersVirtual
Private Network (VPN)Manual opt-outPrivate browsingBrowsers do not
trackEncrypted messaging appsPrivacy-focused web browsersFile encryption
toolsNone Figure 2: Q2.1: Which of the following privacy enhancing
technologies do you use? (multiple choice).

We also asked about the ways these users learn about PETs. Participants
reported different ways including recommendation of friends / social contacts,
being informed at work / school, and news. Only 19% of participants said that
they learn about these PETs via the privacy/cookie policy of a website.

###  6.2 Cookie Notices

When we asked our participants about their understanding of a cookie notice
(Q3.1), they described it via different terms and we observed a few themes,
where one third of our participants mentioned ‘tracking’ with a negative tone.
For instance, P94 said: “It is a pop up that appears on virtually every
website I visit these days. Can be quite annoying since it collects data, but
I tend to reject the tracking cookies if possible”.

In response to Q3.2 on the feelings of the participants about cookie notices
(Table 5). Around half of our participants expressed negative feelings, one
third had neutral feelings, and around a quarter expressed positive feelings
regarding cookie notices. For instance, P9 said: “I don’t have any specific
feeling about them just something that’s there.” and P33 said “I don’t like
them, they are made difficult to understand on purpose, in order to make the
user click ”Accept”. They need to be made more simple.”

Table 5: Q3.2: How do you feel about cookie notices?  Category | Examples | N
---|---|---
Strongly negative | Don’t trust, Intrusive | 24
| Very bad, Frustrating |
Negative | Dislike, Don’t understand | 19
| Confusing |
Neutral | Okay, Not bothered | 31
| Don’t care |
Positive | Important, Essential | 26
| Useful |

In Q3.3 and Q3.4, we asked the participants how they interact with cookie
notices (Table 6). The responses varied across categories including agree,
decline, ignore, edit cookie settings, get rid of it, and use other PETs.
Except those who said they would agree to the cookie notice (47%), all the
other categories included the word “try” in some of the responses e.g., “try
to decline” and “try to edit the settings and say no.” Interestingly, 7% of
participants spoke about trying to get rid of the cookie notice in any
possible way by e.g., responding quickly. P46 said: “generally tick as little
as possible to view the page and also reject where I can if not[,] I have to
accept if the page [is needed]”. Where as P13 said “I try to reject them but
this can be very difficult- I find they are often deliberately set up to make
it impossible to read.”

Table 6: Q3.3: How do you interact with cookie notices?  Category | Examples | N
---|---|---
Agree | Accept, Say yes, Agree | 47
Decline | Reject, Reject, Cancel, Disagree | 34
Ignore | Ignore, Skip it | 8
Edit cookie settings | Edit cookie setting/notice | 7
Get rid of it | Make it go away, Respond quickly | 7
Use PET | Clear cookies later/regularly | 6

In response to more questions in this category (Q3.7 and Q3.8), we found a gap
between the actual handling of cookie notices vs their preferred way. For
instance, 20% of participants said they agreed to cookie notices in reality,
when they wanted to act differently. Figure 3 shows the differences for each
category.

01020304050Percent of participantsUnable to respondOtherIgnore cookie
noticeEditing the settingsDisagreeAgreeWishReality Figure 3: Q3.7: How would
you like to handle cookie notices? and Q3.8: How do you actually respond to
cookie notices?

###  6.3 Issues with Cookie Notices

For Q3.5, the majority (59%) of participants said they had not encountered
issues with cookie notices (Figure 7). The rest said they have experienced
issues regarding cookie display or settings or described negative feelings
such as frustration regarding them. P98 said that they had experienced “cookie
notices blocking content on the page that, if not blocked, I could read and
close the page without having to interact with the notice.” P50 said “some
websites make it a bit difficult to reject all cookies, it’ll open up another
page where you’ll have to individually select each tick box to reject.”

Table 7: Q3.5: Please describe in your own words what type of issues have you experienced with cookie notices? Category | Examples | N
---|---|---
None | Nothing, None, No problem | 59
Display problems | Too big, Loading, Can’t find, Can’t read | 14
Cookie settings problems | Difficult to reject, Forced to accept | 13
Negative feelings | Too many, Tired of disabling, Annoying | 9
Other | Cookies full, Tried to disable | 2

However, when presented with a list of possible problems in Q3.6, only 20%
said none. 79% of the participants said that they had experienced at least
one, the most common being unclear response options in a cookie notice and
being unable to leave a cookie notice. Detailed results for this question are
in Figure 4.

0510152025Percent of participantsUnclear response options in cookie
noticeUnable to leave cookie noticeUnable to answer cookie noticeNoneUnable to
find cookie noticeUnable to enter cookie noticeLack of headings for cookie
noticeCookie notice not being present via ATLow contrast cookie noticeCookie
basket was too full Figure 4: Q3.6: Which of the following issues have you
experienced?

In a follow-up question (Q3.9), we asked what is the potential reason when
participants cannot respond to a cookie notice. The responses of the
participants fell into two main categories: technical issues (37%) or
malicious behaviour (16%). Four participants explicitly mentioned issues with
AT e.g., P27 said that “Assistive technology may not be picking up a notice
that has been given.” For example, P15 said they believe its “because they’re
trying to force you to accept by pretending it’s broken?”

###  6.4 Suggestions

We asked our participants about the responsible stakeholders for accessibility
and security/privacy of web services. In this multiple-choice question,
several entities came up including: website developers (77%), policymakers
(48%), end-users (24%), accessibility evaluation designers (18%), and AT
designers (15%).

In addition, in response to Q4.2 in which we listed a set of recommendations
(based on our system studies), all participants thought that at least one of
our suggestions would help to improve user experience. For example, 79% of
participants believe accessibility-by-design in websites would help their
experience. Figure 5 shows the popularity of other recommendations. We discuss
these in Section 7 at length.

020406080Percent of participantsAccessibility-by-design in
websitesAccessibility testing by web designersInclusion of more
headingsImproving the related lawsImproving the related
specificationsDevelopment of more specific testing toolsEnd user’s engagement
with cookie noticesAccessibly testing of sections of websitesDesigning AT-
friendly PETs Figure 5: Q4.2: Which of these recommendation do you think would
help improve your experience online? (multiple choice)

##  7 Discussion

In this section, we discuss our results across our system studies and user
study.

###  7.1 Website Accessibility and User Privacy

In response to RQ2-a, we found that 93.3% of websites contained at least one
accessibility error and 77.8% contained at least one contrast error. This
means that most websites tested are not compliant with the WCAG success
criteria and, therefore, could be inaccessible, difficult for people with VI
to access, or cause access issues.

The most common error during our testing of cookie notices was low-contrast
buttons or links. The WCAG criteria 1.4.3 and 1.4.6 give guidance for
contrast, the minimum guidance is a contrast ratio of at least 4.5 to 1 with
enhanced guidance of a contrast ratio of at least 7 to 1 [62]. For the
websites that contained a contrast error, this means that they did not meet
the minimum guidance and, therefore, could make text difficult to read for
people with VI. Alongside this, we found a number of websites that had no
errors in their cookie notices but contained errors elsewhere on the page.
Suggesting that the overall accessibility landscape is inadequate, this aligns
with previous research, e.g. [22]. Our results align with previous work,
reporting that the majority of university library finding aids had at least
one accessibility error [56].

These errors with contrast could affect users who have vision loss but are not
fully blind. Due to this group of people being larger than those who are fully
blind, this result is concerning. Low contrast could cause users to miss
important links or become confused about where to give or reject consent. For
example, previous research has shown that higher contrasts between text and
background colour led to faster searching [31], as well as affecting reading
speed [50]. It has also been shown in a requirement survey that links can
cause usability issues for users with VI [71].

###  7.2 Cookie Notice Accessibility Issues

In response to RQ3-a, we have categorised cookie notice accessibility issues
including text-only browser issues, keyboard traps, and visual presentation of
cookie notices vs. screen reader output. We explain each category here.

Text-only browser issues: WebbIE was used to manually examine the heading
contained within a website and its cookie notice. We found that 58.7% of the
websites contained headings that could be useful for navigation, 15.2%
containing basic headings and 26.0% containing no headings at all. Only 2.9%
of websites contained headings throughout their cookie notice, with 17.1%
having a heading at the start. A number of cookie notices did not appear when
using WebbIE, this is most likely due to WebbIE being built using the
Microsoft Web Browser object which gives a program its own internal IE [29].
In June 2022, Microsoft officially ended support for IE for some OSs [38]. It
is therefore likely that web pages stopped supporting IE due to it being a
legacy browser, and this caused these websites not to work with WebbIE.

Keyboard traps: It was found that 77.1% of websites that contained cookie
notices were keyboard navigable when using NVDA. The most common problem found
was having to intervene and use a mouse, an option that is not feasible for
people with VI. There were two main times when a mouse was needed. Firstly, to
get NVDA to read the cookie notice, as some websites required the user to
click on the cookie notice to interact with it. The other issue was escaping
the cookie notice as there were websites that trapped the user in the cookie
notice. This directly contradicts the WCAG success criteria 2.1.2, which is
rated at level A. Whereas when using JAWS 82.9% of websites that contained
cookie notices were keyboard navigable. Due to how JAWS operates a higher
number of privacy policies could be read, with fewer of them creating a
keyboard trap. This is most likely due to how different screen readers handle
CSS code differently [67].

Visual presentation vs screen reader audio output: Only 14.3% of the cookie
notices contained buttons or links whose use could be determined solely by the
button or link when using NVDA. Whereas 31.4% of the cookie notices contained
buttons or links whose use could be determined solely by the button or link.
This difference was due to JAWS reading out alternative text associated with
buttons on some websites. An example of this is where a button might visually
only say _Accept all_ whereas when read aloud using JAWS it says “Accept the
use of cookies and other data for the purposes described”. This change gives
the user significantly more context on what the button does for them and
allows them to skip the reading of the cookie notice. However, it could be
argued that this could be done without the additional alternative text and,
therefore, benefit both users without and with VI. For example, the accept
button on a website simply reads _Accept all cookies_. Therefore, it was easy
to ascertain the function of the button, only from this text, without the need
for additional markup.

###  7.3 Reading Aloud the Cookie Notice

When using a screen reader, the content of the web page is spoken out loud in
a linear order, which may differ from the visual order on the screen [56].
When using WebbIE to view the web pages non-graphically, the cookie notices
were often not at the start of the web page. To combat this, screen readers
can also navigate using headings to jump to different sections. However, the
lack of headings at the start of cookie notices makes it difficult to locate
them when using this method. Screen readers can also search for content within
a web page [49], but without a clear starting heading this becomes difficult.
There were multiple websites where the cookie notice was not read aloud
immediately, and the cookie notice also did not include a heading. In these
examples, it would be difficult to navigate to the cookie notice, without
either knowing what you are searching for or visually identifying it.

When using NVDA with Chrome, 82.9% of cookie notices were read aloud, with
57.1% immediately after the website title was read. There were also websites
that read the cookie notice quickly after opening but not immediately; for
example, elements such as navigation bars were often read aloud before cookie
notices. For example, one website (ebay.co.uk) reads the title of the page,
then the navigation bar, and then the cookie notice. These were normally
websites that did not display the notice at the top when using a browser
graphically. In another example, graphically the cookie notice was at the
bottom, but was read after the heading of the website, and before the main
body. A possible reason for this is the hierarchy of the underlying HTML and
CSS code.

When using JAWS with Google Chrome, 97.1% of cookie notices could be read
aloud, with 62.9% being read aloud immediately. The main issue when a cookie
notice was not read immediately was that the user had to go through the whole
page to read the cookie notice. As we showed in the results section, once
loaded, these websites start collecting data at a large scale and even before
user interaction with the cookie notice. When the cookie notice is the last
item to be read to a VI user, it can easily distract the user from engaging
with it leading to missing the cookie notice altogether.

The results of our user study also confirm that cookie notices accessibility
issues are indeed associated with negative feelings (RQ4-a). They also
highlighted that there are a range of display issues with these cookie notices
such as “they can’t be read”. This contributed to the gap we identified
between the way that these users handle these cookie notices vs how they would
like to handle them (Figure 3).

###  7.4 Website vs Cookie Notice Accessibility

100% of websites contained a title, while only 51.4% of cookie notices
contained a title, which explained what it was. This result was the same for
both screen readers. This lack of titles makes it more difficult to use
headings to quickly navigate to the cookie notice. It is more of a problem
when the notice is not immediately read aloud and then the user has to
navigate to it. The lack of a title also means the user might miss the cookie
notice. None of the 6 websites which contained abbreviations explained them on
both screen readers. This lack of explanation affects the understanding of all
users and directly contradicts WCAG success criteria 3.1.4. Regardless of this
being a high-level success criterion, it is important in the context of cookie
notices. Adding some type of mechanism for understanding abbreviations when
they are used would help all users understand what they are agreeing to.

In response to RQ3-b, we summarise the impact of the issues we encountered on
users with VI. The fact that some cookie notices were missing when using the
text-only browser means that users would not be able to respond to them. This
also applies to the cookie notices that were not readable using screen
readers. Similarly, users could not consent when cookie notices are not being
read immediately, not including headings, and generally being difficult to
navigate. Such a practice might require users to apply additional effort to
specifically navigate to the notice. The lack of headings, structural
elements, and explanatory buttons within the cookie notice means that it could
take users with VI a longer time to respond to a cookie notice than other
users. All these issues mean that these users are less protected against
online tracking and cookies can be placed on their devices without any
possibility for the users to know or give consent.

##  8 Recommendations

In this section, we provide a set of recommendations and best practices for
different stakeholders.

Website developers: There are a number of ways for websites to have the
maximum compatibility with the tools and software used by people with VI. When
including a cookie notice, it should be close to the top of the document’s
code. This will allow screen readers and other accessibility tools to quickly
output this to the user. Developers could then use CSS code to change the
visual location, meaning that a screen reader would always be able to read it
aloud quickly. For example, developers may want visually to move it to the
lower left corner (on desktop) or the lower part of the screen (on mobile) to
improve the number of consent decisions for users without VI [61]. In
addition, developers should always include a heading at the start of important
content, whether this is a cookie notice or other important information. This
allows for ATs to easily and quickly navigate to this information. It also
allows users to quickly understand the content of the section they are about
to interact with and therefore if this information is useful to them.

To assess usability, developers should aim to use a multitude of tools. Tools
such as WAVE and Lighthouse are aimed at allowing developers to easily
evaluate a website. However, we showed in response to RQ2-b, they do not
always highlight the problems users may face and high scores do not
necessarily mean that a website is accessible. This is specifically the case
when it comes to cookie notices and potentially other PETs. Therefore, more
manual tests should be undertaken to find more nuanced issues with a website.
Such testing should be conducted in a comprehensive manner and by multiple VI
tools, since using a combination of such tools is a common practice for VI
users. The tools we suggest are WAVE to perform automated testing, followed up
by using a screen reader such as NVDA since it is free and relatively easy to
use.

Designers of accessibility evaluation tools: This research shows that
accessibility tools and software available do not automatically assess
websites for their privacy practises. The addition of the ability to test sub-
sections of a website for accessibility issues would make testing elements of
a website, such as a cookie notice, a simpler process. This would allow
testing just to focus on such elements. In addition, this will enable subsets
of development teams to test the accessibility of their work. In our testing
with automated tools, it was often not clear where the errors and alerts were
without further manual evaluation. However, this practise should not replace
the overall accessibility testing of the website, but would allow more focus
to be given to some areas of the website.

Furthermore, the creation of specific accessibility tools and tests for cookie
notices and other PETs would greatly improve real-world standard practises.
Such tools could not only test the accessibility of the cookie notices,
privacy policies, and other PETs, but also could evaluate the law compliance
across platforms e.g., software, websites, and mobile apps.

Policymakers: To respond to RQ1-b, we have performed both accessibility and
GDPR compliance analysis. Overall if all websites comply with WCAG, it would
benefit to all users, especially users with disabilities. The question of GDPR
compliance is a more complicated question in relation to users with VI. GDPR
bring many benefits for the privacy of users, however, in some cases, the
implementation of cookie notices has affected the overall accessibility of
websites. For this reason, we make the following recommendation to
policymakers.

The inclusion of specific guidelines for accessibility issues of user privacy
which align with those included in GDPR and the ePD would generally improve
the landscape. For example, guidelines on specific matters such as the length
of time after loading that a cookie notice should be read aloud, what should
be included in the content of the notices, and how should the options be
presented to the users.

Standardisation bodies can create comprehensive specifications for website
developers and dedicated privacy sections. For example, a W3C specification
which includes all the information that developers need to comply with legal
frameworks, such as GDPR or The California Consumer Privacy Act (CCPA), as
well as guidelines, such as WCAG. Such specifications can be also offered by
Google and Apple for app developers in order to improve the privacy of VI
users.

End users: Generally, we believe that the onus around this issue should not be
pushed onto end users, who are already a marginalised group. However, there
are still additional steps users with VI could take. End-users who are
concerned regarding cookie notices can manually search for them. All of the
browsers tested have a feature to search within websites. Though, such a
practise might not be needed in a near future due to the non-effective nature
of cookie notices on websites. Several papers have reported that cookie
notices are not practical and even when the user opts-out the websites still
track the users. Some of these cookie notices are trackers themselves [33]. As
a response, Brave has recently announced that it would automatically block
cookie notices altogether [58]. This option could work to improve the privacy
of users, along with the privacy-preserving nature of the Brave browser. Due
to the browser being based on chromium, it would likely be just as accessible
as Google Chrome. However, this remains an open research problem to be tackled
in the future.

In response to RQ4-b, we concluded that our participants believed that our set
of recommendations can improve their online experience and privacy. Figure 5
displays the popularity of each item where accessibility-by-design in websites
is rated top, followed by accessibility testing by web designers, inclusion of
more headings, improvement of related laws/specifications, development of more
specific testing tools, end user engagement with cookie notices, accessibility
testing of sections of websites (including cookie notices), and designing AT-
friendly PETs.

##  9 Conclusion

This paper investigated the interaction between ATs and cookie notices via a
set of system studies of 46 top UK websites and a user study of 100 users with
VI via Prolific Academic. We find that 22 of these websites had at least one
issue with the accessibility of their cookie notice when manually tested using
a screen reader. We also observed websites which did not have issues with
their cookie notices when using AT but did include issues such as low contrast
when viewing them graphically. These practises often created accessibility
issues when trying to read and respond to cookie notices. The results of our
user study revealed that users with VI overall have a negative view on cookie
notices. We also find that all users believe that at least one of our
recommendations would help improve their experience online.

In future work, we would like to conduct cross-platform studies looking at
mobile web browsers, mobile apps, and desktop web browsers and their
interaction with AT. We would also like to automate our methodology and run
large-scale system studies. We also would like to focus on the creation and
adaptation of dedicated accessibility testing tools for privacy matters and
compliance with the law.

## Acknowledgements

This research project has been granted ethical approval by the University of
Surrey’s ethics committee.

## References

  * [1] Adeyemi, I., Sanders, C., Ong, B. N., Howells, K., Quinlivan, L., Gorman, L., Giles, S., Amp, M., Monaghan, E., Naseem, S., Pearson, A., and Cheraghi-Sohi, S. Challenges and adaptations to public involvement with marginalised groups during the COVID-19 pandemic: Commentary with illustrative case studies in the context of patient safety research.  Research Involvement and Engagement 8, 1 (Apr. 2022), 13.
  * [2] Ahmed, T., Hoyle, R., Connelly, K., Crandall, D., and Kapadia, A. Privacy Concerns and Behaviors of People with Visual Impairments.  In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (Seoul Republic of Korea, Apr. 2015), ACM, pp. 3523–3532.
  * [3] Akter, T., Dosono, B., Ahmed, T., Kapadia, A., and Semaan, B. ”i am uncomfortable sharing what i can’t see”: Privacy concerns of the visually impaired with camera based assistive applications.  In 29th USENIX Security Symposium (USENIX Security 20) (online, Aug. 2020), USENIX Association, pp. 1929–1948.
  * [4] Bermejo Fernandez, C., Chatzopoulos, D., Papadopoulos, D., and Hui, P. This website uses nudging: Mturk workers’ behaviour on cookie consent notices.  Proceedings of the ACM on Human-Computer Interaction 5, CSCW2 (2021), 1–22.
  * [5] Brulé, E., Tomlinson, B. J., Metatla, O., Jouffrais, C., and Serrano, M. Review of quantitative empirical evaluations of technology for people with visual impairments.  In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (New York, NY, USA, 2020), CHI ’20, Association for Computing Machinery, pp. 1–14.
  * [6] Chandrashekar, S. Is hearing believing? Perception of online information credibility by screen reader users who are blind or visually impaired.  PhD thesis, University of Toronto Toronto, ON, 2010.
  * [7] Coopamootoo, K. P., and Groß, T. Why privacy is all but forgotten.  Proceedings on Privacy Enhancing Technologies 2017, 4 (2017), 97–118.
  * [8] Coopamootoo, K. P., Mehrnezhad, M., and Toreini, E. ”i feel invaded, annoyed, anxious and i may protect myself”: Individuals’ feelings about online tracking and their protective behaviour across gender and country.  In 31st USENIX Security Symposium (USENIX Security 22) (Boston, MA, Aug. 2022), USENIX Association, pp. 287–304.
  * [9] Degeling, M., Utz, C., Lentzsch, C., Hosseini, H., Schaub, F., and Holz, T. We value your privacy… now take some cookies.  Informatik Spektrum 42, 5 (2019), 345–346.
  * [10] Developers, C. Lighthouse accessibility scoring.  https://developer.chrome.com/docs/lighthouse/ accessibility/scoring/, 2019\.
  * [11] Egelman, S., and Peer, E. The Myth of the Average User: Improving Privacy and Security Systems through Individualization.  In Proceedings of the 2015 New Security Paradigms Workshop (Twente Netherlands, Sept. 2015), ACM, pp. 16–28.
  * [12] Englehardt, S., and Narayanan, A. Online Tracking: A 1-million-site Measurement and Analysis.  In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security (Vienna Austria, Oct. 2016), ACM, pp. 1388–1401.
  * [13] European Data Protection Board.  Guidelines on consent under regulation 2016/679.  https://ec.europa.eu/newsroom/article29/items/623051, adopted on 10 april 2018.
  * [14] Frauenberger, C. Disability and Technology: A Critical Realist Perspective.  In Proceedings of the 17th International ACM SIGACCESS Conference on Computers & Accessibility - ASSETS ’15 (Lisbon, Portugal, 2015), ACM Press, pp. 89–96.
  * [15] Garante per la protezione dei dati personali.  Video Surveillance ’ Decision.  https://garanteprivacy.it:443/web/guest/home/docweb/-/docweb-display/docweb/1734653, Apr. 2010.
  * [16] Gerl, A., and Meier, B. The layered privacy language art. 12–14 gdpr extension–privacy enhancing user interfaces.  Datenschutz und Datensicherheit-DuD 43, 12 (2019), 747–752.
  * [17] Google Developers.  Lighthouse Overview.  https://developer.chrome.com/docs/lighthouse/overview/, May 2022.
  * [18] GOV.UK.  Results of the 2016 GOV.UK assistive technology survey - Accessibility in government.  https://accessibility.blog.gov.uk/2016/11/01/results-of-the-2016-gov-uk-assistive-technology-survey/, Nov. 2016.
  * [19] Griffin-Shirley, N., Banda, D. R., Ajuwon, P. M., Cheon, J., Lee, J., Park, H. R., and Lyngdoh, S. N. A survey on the use of mobile applications for people who are visually impaired.  Journal of Visual Impairment & Blindness 111, 4 (2017), 307–323.
  * [20] Hakobyan, L., Lumsden, J., O’Sullivan, D., and Bartlett, H. Mobile assistive technologies for the visually impaired.  Survey of ophthalmology 58, 6 (2013), 513–528.
  * [21] Hanson, V. L., Cavender, A., and Trewin, S. Writing about accessibility.  Interactions 22, 6 (Oct. 2015), 62–65.
  * [22] Hanson, V. L., and Richards, J. T. Progress on website accessibility?  ACM Transactions on the Web (TWEB) 7, 1 (2013), 1–30.
  * [23] Hayes, J., Kaushik, S., Price, C. E., and Wang, Y. Cooperative privacy and security: Learning from people with visual impairments and their allies.  In Fifteenth Symposium on Usable Privacy and Security (SOUPS 2019) (Santa Clara, CA, Aug. 2019), USENIX Association, pp. 1–20.
  * [24] Hu, X., and Sastry, N. Characterising Third Party Cookie Usage in the EU after GDPR.  In Proceedings of the 10th ACM Conference on Web Science (Boston Massachusetts USA, June 2019), ACM, pp. 137–141.
  * [25] Inan, F. A., Namin, A. S., Pogrund, R. L., and Jones, K. S. Internet use and cybersecurity concerns of individuals with visual impairments.  Journal of Educational Technology & Society 19, 1 (2016), 28–40.
  * [26] Inc., D. S. Axe-core.  https://github.com/dequelabs/axe-core, Nov. 2022.
  * [27] Information Commissioner’s Office.  Guidance on the use of cookies and similar technologies.  https://ico.org.uk/for-organisations/guide-to-pecr/guidance-on-the-use-of-cookies-and-similar-technologies/, Feb. 2022.
  * [28] Kaczmirek, L., and Wolff, K. G. Survey Design for Visually Impaired and Blind People.  In Universal Acess in Human Computer Interaction. Coping with Diversity, C. Stephanidis, Ed., vol. 4554. Springer Berlin Heidelberg, Berlin, Heidelberg, 2007, pp. 374–381.
  * [29] King, A., Evans, G., and Blenkhorn, P. Blind people and the world wide web.  https://www.webbie.org.uk/webbie.html, 2004.
  * [30] Kretschmer, M., Pennekamp, J., and Wehrle, K. Cookie Banners and Privacy Policies: Measuring the Impact of the GDPR on the Web.  ACM Transactions on the Web 15, 4 (July 2021), 1–42.
  * [31] Ling, J., and Van Schaik, P. The effect of text and background colour on visual search of web pages.  Displays 23, 5 (2002), 223–230.
  * [32] Lund, B. The Brave browser: A monetary opportunity for libraries in the cryptoverse.  Library Hi Tech News 38, 6 (Jan. 2021), 15–16.
  * [33] Matte, C., Bielova, N., and Santos, C. Do cookie banners respect my choice? : Measuring legal compliance of banners from iab europe’s transparency and consent framework.  In 2020 IEEE Symposium on Security and Privacy (SP) (San Francisco, CA, USA, 2020), 2020 IEEE Symposium on Security and Privacy (SP), pp. 791–809.
  * [34] Mehrnezhad, M. A Cross-Platform Evaluation of Privacy Notices and Tracking Practices.  In 2020 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW) (Genoa, Italy, Sept. 2020), IEEE, pp. 97–106.
  * [35] Mehrnezhad, M., and Almeida, T. Caring for Intimate Data in Fertility Technologies.  In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama Japan, May 2021), ACM, pp. 1–11.
  * [36] Mehrnezhad, M., Coopamootoo, K., and Toreini, E. How Can and Would People Protect From Online Tracking?  Proceedings on Privacy Enhancing Technologies 2022, 1 (Jan. 2022), 105–125.
  * [37] Melicher, W., Sharif, M., Tan, J., Bauer, L., Christodorescu, M., and Leon, P. G. Preferences for Web Tracking.  Proceedings on Privacy Enhancing Technologies 2016, 2 (2016), 1–20.
  * [38] Microsoft.  Lifecycle FAQ - Internet Explorer and Microsoft Edge.  https://docs.microsoft.com/en-us/lifecycle/faq/internet-explorer-microsoft-edge, n.d.
  * [39] Napoli, D., Baig, K., Maqsood, S., and Chiasson, S. ” i’m literally just hoping this will $\\{$Work:’$\\}$’obstacles blocking the online security and privacy of users with visual disabilities.  In Seventeenth Symposium on Usable Privacy and Security (SOUPS 2021) (2021), pp. 263–280.
  * [40] Nouwens, M., Liccardi, I., Veale, M., Karger, D., and Kagal, L. Dark Patterns after the GDPR: Scraping Consent Pop-ups and Demonstrating their Influence.  In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu HI USA, Apr. 2020), ACM, pp. 1–13.
  * [41] Office for National Statistics.  Internet users, UK.  https://www.ons.gov.uk/businessindustryandtrade /itandinternetindustry/bulletins/internetusers/2020, 2020.
  * [42] Parliament, T. E., and the Council of the European Union.  Directive 2002/58/EC of the European Parliament and of the Council of 12 July 2002 concerning the processing of personal data and the protection of privacy in the electronic communications sector (Directive on privacy and electronic communications), July 2002.
  * [43] Pernice, K., Nielsen, J., Farrell, S., Mizobuchi, S., Ishida, N., Stover, U. A., Yohay, M., Franko, E., and Richardson, A. Usability guidelines for accessible web design.  Evidence–Based User Experience Research, Training, Consulting (2001).
  * [44] Pierce, J., Fox, S., Merrill, N., and Wong, R. Differential vulnerabilities and a diversity of tactics: What toolkits teach us about cybersecurity.  Proceedings of the ACM on Human-Computer Interaction 2, CSCW (2018), 1–24.
  * [45] PowerMapper Software.  Screen reader reliability.  https://www.powermapper.com/tests/screen-readers/, Sept. 2022.
  * [46] Pugliese, G., Riess, C., Gassmann, F., and Benenson, Z. Long-Term Observation on Browser Fingerprinting: Users’ Trackability and Perspective.  Proceedings on Privacy Enhancing Technologies 2020, 2 (Apr. 2020), 558–577.
  * [47] Rao, A., Schaub, F., and Sadeh, N. What do they know about me? Contents and Concerns of Online Behavioral Profiles, June 2015.
  * [48] Regulation, G. D. P. General Data Protection Regulation (GDPR) – Official Legal Text.  https://gdpr-info.eu/, 2016.
  * [49] Rotard, M., Taras, C., and Ertl, T. Tactile web browsing for blind people.  Multimedia Tools and Applications 37, 1 (Mar. 2008), 53–69.
  * [50] Rubin, G. S., and Legge, G. E. Psychophysics of reading. VI—The role of contrast in low vision.  Vision Research 29, 1 (Jan. 1989), 79–91.
  * [51] Schmeelk, S., and Petrie, H. Digital authentication for visually disabled people: Initial results of an online survey.  In Computers Helping People with Special Needs: 18th International Conference, ICCHP-AAATE 2022, Lecco, Italy, July 11–15, 2022, Proceedings, Part II (2022), Springer, pp. 41–50.
  * [52] Schnell, K., and Roy, K. Website privacy notification for the visually impaired.  In 2021 IEEE Symposium Series on Computational Intelligence (SSCI) (Orlando, FL, USA, 2021), 2021 IEEE Symposium Series on Computational Intelligence (SSCI), pp. 1–6.
  * [53] Seničar, V., Jerman-Blažič, B., and Klobučar, T. Privacy-Enhancing Technologies—approaches and development.  Computer Standards & Interfaces 25, 2 (May 2003), 147–158.
  * [54] Shirazi, F., and Volkamer, M. What deters jane from preventing identification and tracking on the web?  In Proceedings of the 13th Workshop on Privacy in the Electronic Society (New York, NY, USA, 2014), WPES ’14, Association for Computing Machinery, pp. 107–116.
  * [55] Sørensen, J., and Kosta, S. Before and After GDPR: The Changes in Third Party Presence at Public and Private European Websites.  In The World Wide Web Conference (New York, NY, USA, May 2019), WWW ’19, Association for Computing Machinery, pp. 1590–1600.
  * [56] Southwell, K. L., and Slater, J. An evaluation of finding aid accessibility for screen readers.  Information Technology and Libraries 32, 3 (2013), 34–46.
  * [57] Statcounter.  Browser Market Share Worldwide.  https://gs.statcounter.com/browser-market-share, n.d.
  * [58] the Brave Privacy Team.  Blocking annoying and privacy-harming cookie consent banners.  https://brave.com/privacy-updates/21-blocking-cookie-notices/, Sept. 2022\.
  * [59] Trevisan, M., Traverso, S., Bassi, E., and Mellia, M. 4 Years of EU Cookie Law: Results and Lessons Learned.  Proceedings on Privacy Enhancing Technologies 2019, 2 (Apr. 2019), 126–145.
  * [60] Ur, B., Leon, P. G., Cranor, L. F., Shay, R., and Wang, Y. Smart, useful, scary, creepy: Perceptions of online behavioral advertising.  In Proceedings of the Eighth Symposium on Usable Privacy and Security - SOUPS ’12 (Washington, D.C., 2012), ACM Press, p. 1.
  * [61] Utz, C., Degeling, M., Fahl, S., Schaub, F., and Holz, T. (Un)informed Consent: Studying GDPR Consent Notices in the Field.  In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security (London United Kingdom, Nov. 2019), ACM, pp. 973–990.
  * [62] W3.  How to Meet WCAG (Quickref Reference).  https://www.w3.org/WAI/WCAG21/quickref/, Oct. 2019.
  * [63] W3.  WCAG 2 Overview.  https://www.w3.org/WAI/standards-guidelines/wcag/, Mar. 2022.
  * [64] W3.  Understanding Conformance.  https://www.w3.org/WAI/WCAG21/Understanding/ conformance#levels, n.d.
  * [65] Wang, Y., and Price, C. E. Accessible privacy.  In Modern Socio-Technical Perspectives on Privacy. Springer, Cham, 2022, pp. 293–313.
  * [66] Washington, B. S., Feng, J. H., Ahram, T., and Falcão, C. Proper Implementation of Website Features Affecting the Use of Screen Readers, vol. 972 of Advances in Usability and User Experience.  Springer International Publishing, 2020.
  * [67] WebAIM.  Screen Readers and CSS: Are We Going Out of Style (and into Content)?  https://webaim.org/blog/screen-readers-and-css/, Aug. 2017.
  * [68] WebAIM.  WebAIM: Screen Reader User Survey #9 Results.  https://webaim.org/projects/screenreadersurvey9/, June 2021.
  * [69] WebAIM.  Wave chrome, firefox, and edge extensions.  https://wave.webaim.org/extension/, n.d.
  * [70] whotracksme.  GDPR - what happened?  https://whotracks.me/blog/gdpr-what-happened.html, 2018.
  * [71] Yu, W., Kuber, R., Murphy, E., Strain, P., and McAllister, G. A novel multimodal interface for improving visually impaired people’s web accessibility.  Virtual Reality 9, 2 (2006), 133–148.

##  Appendix A Website Analysis Template

For each website in our list, we followed these steps for our analysis:

  * •

Part I: Cookie Notice (baseline)

    * -

Step 1: Open Google Chrome incognito window and visit the homepage of the
website.

    * -

Step 2: Observe if there is a notice (cookie consent, privacy settings,
banner, etc.).

      * -

No: Write it in the file.

      * -

Yes: Observe the location and user control options e.g. OK, Accept, Yes,
Reject, No, More Options, Settings, Links to privacy-related pages, etc. Write
your observations in the file.

    * -

Step 3: Close the Google Chrome incognito window.

    * -

Step 4: Open Brave private window and visit the homepage of the website.

    * -

Step 5: Repeat Step - ‣ • ‣ A (cookie notice).

    * -

Step 6: Record the number of items blocked by the Brave Shields in the file.

    * -

Step 7: Close the Brave private window.

  * •

Part II: Automated Accessibility Testing Tools

    * -

Step 8: Open a new Google Chrome incognito window and visit the website’s
homepage.

    * -

Step 9: Click on WAVE extension to run the test.

    * -

Step 10: Record the number of each of the categories in the file.

    * -

Step 11: Close the Google Chrome incognito window.

    * -

Step 12: Open a new Google Chrome incognito window and visit the website’s
homepage.

    * -

Step 13: Open developer tools and navigate to the Lighthouse tab.

    * -

Step 14: Select Navigation Mode, A desktop device and the Accessibility
Categories.

    * -

Step 15: Analyse the page.

    * -

Step 16: Record the overall accessibility score and the number of each item
shown.

    * -

Step 17: Close the Google Chrome incognito window.

  * •

Part III: Manual Testing Via Text-only Browser

    * -

Step 18: Open WebbIE Browser and visit the homepage of the website.

    * -

Step 19: Record the number of headings for the website overall.

    * -

Step 20: Record the presence of a notice, and if so the presence of headings.

    * -

Step 21: Close WebbIE.

  * •

Part IV: Manual Testing via Screen Readers

    * -

Step 22 : Open NVDA screen reader.

    * -

Step 23: Open a new Google Chrome incognito window and visit the website’s
homepage.

    * -

Step 24: Allow screen reader to read website.

    * -

Step 25: Interact with the website using keyboard.

    * -

Step 26: Record pass/fail for categories in 3.2.3.

    * -

Step 27: Close the Google Chrome incognito window and screen reader.

    * -

Step 28: Open JAWS screen reader.

    * -

Step 29: Repeat Steps - ‣ • ‣ A-- ‣ • ‣ A.

Note that we performed two rounds of testing (with identical results). We
uninstalled and reinstalled WebbIE for each round since it does not support
cookie management.

##  Appendix B User Study Template

###  B.1 Screening validation

  * •

Do you live in the United Kingdom? (One answer is possible)

    * $\square$

Yes

    * $\square$

No

  * •

Do you use assistive technology? (One answer is possible)

    * $\square$

Yes

    * $\square$

No

###  B.2 Section 1: Internet and assistive technology

This section is about your internet usage and any assistive technology you may
utilise while browsing the web.

  * •

1.1: How do you describe your visual impairment? (Text input is possible)

  * •

1.2: Which devices do you use to access the internet? Please specify if other
(Several answers are possible)

    * $\square$

Personal Computer (Desktop or Laptop) Mobile Phone

    * $\square$

Tablet Computer

    * $\square$

Smart home devices

    * $\square$

Wearable devices Internet-enabled TVs Gaming consoles Public computers

    * $\square$

Other:

  * •

1.3: Which device are you using to complete this questionnaire? Please specify
if other (One answer possible)

    * $\square$

Personal Computer (Desktop or Laptop)

    * $\square$

Mobile Phone

    * $\square$

Tablet Computer

    * $\square$

Smart home devices

    * $\square$

Wearable devices

    * $\square$

Internet-enabled TVs

    * $\square$

Gaming consoles

    * $\square$

Public computers

    * $\square$

Other:

  * •

1.4: In an average week, how many hours do you use the internet? (One answer
is possible)

    * $\square$

Less then 1 hour 1-5 hours

    * $\square$

6-10 hours

    * $\square$

11-15 hours

    * $\square$

16-20 hours

    * $\square$

21-15 hours

    * $\square$

26-30 hours

    * $\square$

More than 30 hours

  * •

1.5: What assistive technology do you use when browsing the web? Please
specify if other. (Several answers are possible)

    * $\square$

Screen reader (Jaws, NVDA, Voice over or other)

    * $\square$

Braille Display

    * $\square$

Text Only Browser (WebbIE or other)

    * $\square$

Magnification software

    * $\square$

Assistive browser extension

    * $\square$

Alternative input devices

    * $\square$

None

  * •

1.6: Which of the following screen readers do you use? Please specify if
other. (Several answers are possible)

    * $\square$

JAWS

    * $\square$

NVDA

    * $\square$

VoiceOver

    * $\square$

Natural Reader

    * $\square$

Read&Write

    * $\square$

Narrator

    * $\square$

Talkback

    * $\square$

ChromeVox

    * $\square$

Orca

    * $\square$

I don’t use a screen reader

    * $\square$

Other:

  * •

1.7: How would you describe your level of expertise with a screen reader?
(Text input is possible)

  * •

1.8: Do you use plug-ins with a screen reader? (One answer is possible)

A plugin or add-on adds a specific feature or functionality to a screen
reader. Allowing users to customise their software and add the features they
need.

    * $\square$

Yes

    * $\square$

No

  * •

1.9: Which plugins do you use? (Text input is possible)

###  B.3 Section 2: Privacy enhancing technology usage

This section is about the measures you take to protect your privacy and
security while browsing the internet and Privacy Enhancing Technologies
(PETs). PETs are tools that can help protect your privacy online by limiting
the collection, use, and dissemination of your personal information.

  * •

2.1: Which of the following privacy enhancing technologies do you use? Please
specify if other. (Several answers are possible)

    * $\square$

Browsers do not track

    * $\square$

Virtual Private Network (VPN)

    * $\square$

Private browsing

    * $\square$

Password manager

    * $\square$

Manual cookie opt-out

    * $\square$

Privacy-focused web browsers

    * $\square$

Encrypted messaging apps

    * $\square$

Ad blockers

    * $\square$

File encryption tools

    * $\square$

None

    * $\square$

Other:

  * •

2.2: How did you learn about privacy enhancing technology? Please specify if
other. (Several answers are possible)

    * $\square$

Friend / social contact recommendation

    * $\square$

Work / school recommendation

    * $\square$

Privacy/cookie policy of a website

    * $\square$

News

    * $\square$

I don’t know

    * $\square$

I don’t use privacy enhancing technologies

    * $\square$

Other:

###  B.4 Section 3: Cookie notice

Cookie notices appear when a user lands on a website and informs them that the
website is using cookies (a data file) and other trackers that process
personal data, and that the user must make a choice whether they want their
personal data processes.

  * •

3.1: In your own words what is a cookie notice and what are they supposed to
do? (Text input is possible)

  * •

3.2: How do you feel about cookie notices? (Text input is possible)

  * •

3.3: How do you interact with cookie notices? (Text input is possible)

  * •

3.4: Have you encountered any issues with cookie notices? (One answer is
possible)

    * $\square$

Yes

    * $\square$

No

  * •

3.5: Please describe in your won words what type of issues have you
experienced with cookie notices? (Text input is possible)

  * •

3.6: Which of the following issues have you experienced? Please specify if
other (Several answers are possible)

    * $\square$

Cookie notice not being present via my assistive technology

    * $\square$

Unable to find cookie notice

    * $\square$

Unable to answer cookie notice

    * $\square$

Low contrast cookie notice

    * $\square$

Unclear response options in cookie notice

    * $\square$

Lack of headings for cookie notice

    * $\square$

Unable to leave cookie notice

    * $\square$

Unable to enter cookie notice

    * $\square$

Other:

  * •

3.7: How would you wish to handle cookie notices? Please specify if other.
(One answer is possible)

    * $\square$

Agree

    * $\square$

Disagree

    * $\square$

Editing the settings

    * $\square$

Ignore cookie notice

    * $\square$

Other:

  * •

3.8: How do you actually respond to cookie notices? Please specify if other.
(One answer is possible)

    * $\square$

Agree

    * $\square$

Disagree

    * $\square$

Editing the settings

    * $\square$

Ignore cookie notice

    * $\square$

Unable to respond to cookie notice

    * $\square$

Other:

  * •

3.9: If you are unable to respond to cookie notices, what do you think is the
reason? (Text input is possible)

###  B.5 Section 4: Suggestions

  * •

4.1: Who do you think should be responsible for ensuring the secure
accessibility of the Internet? Please specify if other. (Several answers are
possible)

    * $\square$

Website Developers

    * $\square$

Designers of Accessibility Evaluation Tools

    * $\square$

Policymakers

    * $\square$

Users of the internet

    * $\square$

Designers of Assistive Technologies

    * $\square$

Other:

  * •

4.2: Which of these recommendation do you think would help improve your
experience online? Please specify if other (Several answers are possible)

    * $\square$

Website designers should design websites with accessibility in mind

    * $\square$

Website designers should include more headings for useful information

    * $\square$

Website designers should complete more accessibility testing

    * $\square$

Evaluation tools should allow for testing of sections of a web page

    * $\square$

Specific testing tools for parts of a webpage (i.e. cookie notices or other
elements)

    * $\square$

The inclusion of laws specifying privacy and accessibility

    * $\square$

Specifications of achieving privacy and accessibility

    * $\square$

End users can searching for cookie notices

    * $\square$

End users can specific privacy protecting tools such as the Brave internet
browser

    * $\square$

Other:

###  B.6 Section 5: Demographic and background questions

  * •

5.1: What is your age? (One answer is possible)

    * $\square$

18 to 24

    * $\square$

25 to 34

    * $\square$

35 to 44

    * $\square$

45 to 54

    * $\square$

55 to 64

    * $\square$

65 or over

    * $\square$

Prefer not to say

  * •

5.2: What is your gender? (One answer is possible)

    * $\square$

Female

    * $\square$

Male

    * $\square$

Non-binary

    * $\square$

Prefer not to say

    * $\square$

Other

  * •

5.3: What is your highest level of education? (One answer is possible)

    * $\square$

Secondary education

    * $\square$

Post-secondary education

    * $\square$

Undergraduate education

    * $\square$

Graduate education

    * $\square$

Prefer not to say

  * •

5.4: What is your occupation? Please specify if other. (One answer is
possible)

    * $\square$

Healthcare and social assistance

    * $\square$

Education and training

    * $\square$

Sales and retail

    * $\square$

Administrative and support

    * $\square$

Manufacturing and production

    * $\square$

Information technology

    * $\square$

Business and finance

    * $\square$

Transportation and logistics

    * $\square$

Construction and trades

    * $\square$

Arts, entertainment, and media

    * $\square$

Prefer not to say

    * $\square$

Other

  * •

5.5: What services do you use online? Please specify if other. (Multiple
answers are possible)

    * $\square$

Email

    * $\square$

Social media

    * $\square$

Online shopping and e-commerce

    * $\square$

Video Streaming

    * $\square$

Music/audio streaming

    * $\square$

Online payment/banking

    * $\square$

File sharing and cloud storage

    * $\square$

Online travel booking

    * $\square$

Online education and e-learning

    * $\square$

Online commutation and collaboration services

    * $\square$

Other

##  Appendix C Detailed results of automated accessibility tools

Table 8: Detailed results of WAVE 5.0 and Google Lighthouse

Website |  Privacy Policy |  Errors |  Contrast Errors |  Alerts |  Features |  Structural Elements |  ARIA |  Lighthouse Score
---|---|---|---|---|---|---|---|---
google.com | ✓ | 1 | 10 | 4 | 5 | 7 | 350 | 97
youtube.com | ✓ | 26 | 1 | 70 | 92 | 66 | 676 | 89
yahoo.com | ✓ | 2 | 8 | 6 | 2 | 4 | 4 | 86
facebook.com | ✓ | 4 | 46 | 12 | 1 | 9 | 9 | 93
bbc.co.uk | ✓ | 0 | 0 | 134 | 23 | 119 | 371 | 100
amazon.co.uk | ✓ | 6 | 2 | 139 | 194 | 55 | 381 | 95
reddit.com | ✓ | 9 | 139 | 657 | 104 | 36 | 433 | 77
wikipedia.org | ✗ | 3 | 0 | 97 | 123 | 70 | 24 | 100
live.com | ✗ | 0 | 5 | 8 | 11 | 38 | 77 | 98
instagram.com | ✓ | 0 | 25 | 3 | 10 | 10 | 37 | 95
twitter.com | ✓ | 1 | 19 | 3 | 1 | 4 | 55 | 88
ebay.co.uk | ✗ | 0 | 0 | 0 | 0 | 0 | 0 | 93
dailymail.co.uk | ✓ | 75 | 31 | 890 | 628 | 497 | 61 | 74
bing.com | ✓ | 12 | 23 | 38 | 26 | 40 | 244 | 94
gov.uk | ✓ | 0 | 1 | 5 | 16 | 68 | 38 | 100
netflix.com | ✓ | 2 | 22 | 9 | 22 | 36 | 68 | 88
theguardian.com | ✗ | 17 | 59 | 131 | 102 | 290 | 624 | 79
pornhub.com | ✗ | 65 | 1 | 197 | 110 | 67 | 119 | 96
office.com | ✓ | 4 | 0 | 3 | 25 | 59 | 199 | 94
fandom.com | ✓ | 12 | 33 | 14 | 28 | 28 | 2 | 86
xvideos.com | ✓ | 104 | 64 | 158 | 2 | 22 | 1 | 88
paypal.com | ✓ | 12 | 33 | 14 | 28 | 28 | 2 | 100
microsoft.com | ✓ | 2 | 0 | 3 | 24 | 47 | 183 | 100
linkedin.com | ✓ | 27 | 0 | 0 | 9 | 32 | 174 | 100
xhamster.com | ✗ | 8 | 118 | 57 | 58 | 25 | 2 | 81
imdb.com | ✗ | 8 | 0 | 42 | 98 | 37 | 1552 | 88
duckduckgo.com | ✗ | 23 | 5 | 13 | 6 | 38 | 35 | 96
amazon.com | ✗ | 4 | 2 | 178 | 216 | 45 | 317 | 98
zoom.us | ✓ | 7 | 3 | 58 | 64 | 93 | 889 | 80
twitch.tv | ✓ | 71 | 0 | 150 | 130 | 48 | 300 | 86
amazonaws.com | ✓ | 4 | 14 | 140 | 228 | 377 | 87 | 86
tiktok.com | ✓ | 52 | 14 | 54 | 9 | 59 | 0 | 63
whatsapp.com | ✓ | 5 | 14 | 10 | 4 | 142 | 5 | 85
doubleclick.net | ✓ | 2 | 4 | 51 | 11 | 78 | 141 | 100
spankbang.com | ✗ | 16 | 160 | 500 | 190 | 36 | 0 | 72
sky.com | ✓ | 3 | 16 | 28 | 22 | 41 | 105 | 90
apple.com | ✗ | 11 | 0 | 15 | 47 | 62 | 217 | 92
rightmove.co.uk | ✓ | 3 | 7 | 68 | 12 | 48 | 55 | 87
booking.com | ✓ | 30 | 3 | 37 | 40 | 85 | 574 | 92
etsy.com | ✓ | 7 | 1 | 31 | 42 | 175 | 1682 | 73
indeed.com | ✓ | 2 | 0 | 8 | 20 | 25 | 134 | 90
msn.com | ✓ | 26 | 0 | 213 | 359 | 311 | 44 | 79
github.com | ✗ | 2 | 35 | 18 | 114 | 74 | 179 | 86
adobe.com | ✓ | 2 | 1 | 12 | 111 | 120 | 139 | 96
chaturbate.com | ✗ | 19 | 113 | 494 | 102 | 210 | 5 | 84
xnxx.com | ✓ | 165 | 0 | 808 | 3 | 20 | 0 | 97

##  Appendix D Detailed results of screen reader tests

Table 9: NVDA and JAWS results

| NVDA | JAWS
---|---|---
Website |  Readable |  Immediately |  Keyboard Navigable |  Link or button Purpose |  Abbreviations are explained |  Page Titled |  Cookie Notice Titled |  Headings useful for navigation |  No Timing |  Readable |  Immediately |  Keyboard Navigable |  Link or button Purpose |  Abbreviations are explained |  Page Titled |  Cookie Notice Titled |  Headings useful for navigation |  No Timing
google.com |  ✓ |  ✓ |  ✓ |  ✗ |  - |  ✓ |  ✗ |  ✗ |  ✓ |  ✓ |  ✓ |  ✓ |  ✗ |  - |  ✓ |  ✗ |  ✗ |  ✓
youtube.com |  ✓ |  ✓ |  ✓ |  ✗ |  - |  ✓ |  ✗ |  ✗ |  ✓ |  ✓ |  ✓ |  ✓ |  ✓ |  - |  ✓ |  ✗ |  ✗ |  ✓
yahoo.com |  ✓ |  ✓ |  ✓ |  ✗ |  ✗ |  ✓ |  ✓ |  ✗ |  ✓ |  ✓ |  ✓ |  ✓ |  ✗ |  ✗ |  ✓ |  ✓ |  ✗ |  ✓
facebook.com |  ✓ |  ✗ |  ✓ |  ✗ |  ✗ |  ✓ |  ✓ |  ✗ |  ✓ |  ✓ |  ✗ |  ✓ |  ✓ |  ✗ |  ✓ |  ✓ |  ✓ |  ✓
bbc.co.uk |  ✓ |  ✓ |  ✓ |  ✗ |  - |  ✓ |  ✓ |  ✗ |  ✓ |  ✓ |  ✓ |  ✓ |  ✗ |  - |  ✓ |  ✓ |  ✗ |  ✓
amazon.co.uk |  ✓ |  ✓ |  ✓ |  ✗ |  - |  ✓ |  ✓ |  ✗ |  ✓ |  ✓ |  ✗ |  ✓ |  ✓ |  - |  ✓ |  ✓ |  ✗ |  ✓
reddit.com |  ✗ |  ✗ |  ✗ |  ✗ |  - |  ✓ |  ✗ |  ✗ |  ✓ |  ✗ |  ✗ |  ✗ |  ✗ |  - |  ✓ |  ✗ |  ✗ |  ✓
wikipedia.org |  - |  - |  - |  - |  - |  ✓ |  - |  - |  - |  - |  - |  - |  - |  - |  ✓ |  - |  - |  -
live.com |  - |  - |  - |  - |  - |  ✓ |  - |  - |  - |  - |  - |  - |  - |  - |  ✓ |  - |  - |  -
instagram.com |  ✗ |  ✗ |  ✓ |  ✗ |  - |  ✓ |  ✓ |  ✗ |  ✓ |  ✓ |  ✗ |  ✓ |  ✓ |  - |  ✓ |  ✓ |  ✓ |  ✓
twitter.com |  ✓ |  ✓ |  ✓ |  ✓ |  - |  ✓ |  ✓ |  ✗ |  ✓ |  ✓ |  ✓ |  ✓ |  ✓ |  - |  ✓ |  ✓ |  ✗ |  ✓
ebay.co.uk |  ✓ |  ✗ |  ✓ |  ✗ |  ✗ |  ✓ |  ✗ |  ✗ |  ✓ |  ✓ |  ✗ |  ✓ |  ✓ |  ✗ |  ✓ |  ✗ |  ✗ |  ✓
dailymail.co.uk |  ✗ |  ✗ |  ✗ |  ✗ |  ✗ |  ✓ |  ✗ |  ✗ |  ✓ |  ✓ |  ✗ |  ✗ |  ✗ |  ✗ |  ✓ |  ✗ |  ✗ |  ✓
bing.com |  ✓ |  ✗ |  ✓ |  ✗ |  - |  ✓ |  ✗ |  ✗ |  ✓ |  ✓ |  ✗ |  ✓ |  ✗ |  - |  ✓ |  ✗ |  ✗ |  ✓
gov.uk |  ✓ |  ✓ |  ✓ |  ✓ |  - |  ✓ |  ✓ |  ✗ |  ✓ |  ✓ |  ✓ |  ✓ |  ✓ |  - |  ✓ |  ✓ |  ✗ |  ✓
netflix.com |  ✓ |  ✓ |  ✓ |  ✗ |  - |  ✓ |  ✗ |  ✗ |  ✓ |  ✓ |  ✓ |  ✓ |  ✗ |  - |  ✓ |  ✗ |  ✗ |  ✓
theguardian.com |  ✓ |  ✗ |  ✓ |  ✗ |  ✗ |  ✓ |  ✗ |  ✗ |  ✓ |  ✓ |  ✗ |  ✓ |  ✗ |  ✗ |  ✓ |  ✗ |  ✗ |  ✓
pornhub.com |  - |  - |  - |  - |  - |  ✓ |  - |  - |  - |  - |  - |  - |  - |  - |  ✓ |  - |  - |  -
office.com |  ✓ |  ✓ |  ✓ |  ✗ |  - |  ✓ |  ✗ |  ✗ |  ✓ |  ✓ |  ✓ |  ✓ |  ✗ |  - |  ✓ |  ✗ |  ✗ |  ✓
fandom.com |  ✓ |  ✗ |  ✗ |  ✗ |  ✗ |  ✓ |  ✓ |  ✗ |  ✓ |  ✓ |  ✗ |  ✗ |  ✗ |  ✗ |  ✓ |  ✓ |  ✗ |  ✓
xvideos.com |  ✗ |  ✗ |  ✗ |  ✗ |  - |  ✓ |  ✗ |  ✗ |  ✓ |  ✓ |  ✓ |  ✓ |  ✗ |  - |  ✓ |  ✗ |  ✗ |  ✓
paypal.com |  ✓ |  ✗ |  ✗ |  ✗ |  ✗ |  ✓ |  ✓ |  ✗ |  ✓ |  ✓ |  ✗ |  ✗ |  ✗ |  ✗ |  ✓ |  ✓ |  ✗ |  ✓
microsoft.com |  ✓ |  ✓ |  ✓ |  ✗ |  - |  ✓ |  ✗ |  ✗ |  ✓ |  ✓ |  ✓ |  ✓ |  ✗ |  - |  ✓ |  ✗ |  ✗ |  ✓
linkedin.com |  ✓ |  ✓ |  ✓ |  ✗ |  - |  ✓ |  ✓ |  ✗ |  ✓ |  ✓ |  ✓ |  ✓ |  ✗ |  - |  ✓ |  ✓ |  ✗ |  ✓
xhamster.com |  - |  - |  - |  - |  - |  ✓ |  - |  - |  - |  - |  - |  - |  - |  - |  ✓ |  - |  - |  -
imdb.com |  - |  - |  - |  - |  - |  ✓ |  - |  - |  - |  - |  - |  - |  - |  - |  ✓ |  - |  - |  -
duckduckgo.com |  - |  - |  - |  - |  - |  ✓ |  - |  - |  - |  - |  - |  - |  - |  - |  ✓ |  - |  - |  -
amazon.com |  - |  - |  - |  - |  - |  ✓ |  - |  - |  - |  - |  - |  - |  - |  - |  ✓ |  - |  - |  -
zoom.us |  ✓ |  ✓ |  ✓ |  ✗ |  - |  ✓ |  ✗ |  ✗ |  ✓ |  ✓ |  ✓ |  ✓ |  ✓ |  - |  ✓ |  ✗ |  ✗ |  ✓
twitch.tv |  ✓ |  ✗ |  ✓ |  ✓ |  - |  ✓ |  ✓ |  ✗ |  ✓ |  ✓ |  ✓ |  ✓ |  ✓ |  - |  ✓ |  ✓ |  ✗ |  ✓
amazonaws.com |  ✓ |  ✓ |  ✓ |  ✗ |  - |  ✓ |  ✓ |  ✗ |  ✓ |  ✓ |  ✓ |  ✓ |  ✓ |  - |  ✓ |  ✓ |  ✗ |  ✓
tiktok.com |  ✓ |  ✗ |  ✗ |  ✗ |  - |  ✓ |  ✓ |  ✗ |  ✓ |  ✓ |  ✗ |  ✗ |  ✗ |  - |  ✓ |  ✓ |  ✗ |  ✓
whatsapp.com |  ✓ |  ✓ |  ✓ |  ✗ |  - |  ✓ |  ✓ |  ✗ |  ✓ |  ✓ |  ✓ |  ✓ |  ✗ |  - |  ✓ |  ✓ |  ✗ |  ✓
doubleclick.net |  ✓ |  ✗ |  ✓ |  ✗ |  - |  ✓ |  ✗ |  ✗ |  ✓ |  ✓ |  ✓ |  ✗ |  ✗ |  - |  ✓ |  ✗ |  ✗ |  ✓
spankbang.com |  - |  - |  - |  - |  - |  ✓ |  - |  - |  - |  - |  - |  - |  - |  - |  ✓ |  - |  - |  -
sky.com |  ✓ |  ✓ |  ✓ |  ✗ |  - |  ✓ |  ✓ |  ✗ |  ✓ |  ✓ |  ✓ |  ✓ |  ✗ |  - |  ✓ |  ✓ |  ✗ |  ✓
apple.com |  - |  - |  - |  - |  - |  ✓ |  - |  - |  - |  - |  - |  - |  - |  - |  ✓ |  - |  - |  -
rightmove.co.uk |  ✓ |  ✓ |  ✓ |  ✓ |  - |  ✓ |  ✓ |  ✗ |  ✓ |  ✓ |  ✓ |  ✓ |  ✗ |  - |  ✓ |  ✓ |  ✗ |  ✓
booking.com |  ✓ |  ✓ |  ✓ |  ✗ |  - |  ✓ |  ✓ |  ✗ |  ✓ |  ✓ |  ✓ |  ✓ |  ✗ |  - |  ✓ |  ✓ |  ✗ |  ✓
etsy.com |  ✓ |  ✓ |  ✓ |  - |  - |  ✓ |  - |  - |  ✓ |  ✓ |  ✓ |  ✓ |  ✗ |  - |  ✓ |  - |  ✗ |  ✓
indeed.com |  ✓ |  ✓ |  ✓ |  ✓ |  - |  ✓ |  ✗ |  ✗ |  ✓ |  ✓ |  ✗ |  ✓ |  ✓ |  - |  ✓ |  ✗ |  ✗ |  ✓
msn.com |  ✓ |  ✓ |  ✓ |  ✗ |  - |  ✓ |  ✓ |  ✗ |  ✓ |  ✓ |  ✓ |  ✓ |  ✗ |  - |  ✓ |  ✓ |  ✗ |  ✓
github.com |  - |  - |  - |  - |  - |  ✓ |  - |  - |  - |  - |  - |  - |  - |  - |  ✓ |  - |  - |  -
adobe.com |  ✗ |  ✗ |  ✗ |  ✗ |  - |  ✓ |  ✗ |  ✗ |  ✓ |  ✓ |  ✗ |  ✓ |  ✗ |  - |  ✓ |  ✗ |  ✗ |  ✓
chaturbate.com |  - |  - |  - |  - |  - |  ✓ |  - |  - |  - |  - |  - |  - |  - |  - |  ✓ |  - |  - |  -
xnxx.com |  ✗ |  ✗ |  ✗ |  ✗ |  - |  ✓ |  ✗ |  ✗ |  ✓ |  ✓ |  ✓ |  ✓ |  ✗ |  - |  ✓ |  ✗ |  ✗ |  ✓

###  D.1 Answers to demographic questions

Table 10: Answers to demographic questions 1.2 Devices used | N | 1.3 To Answer | N | 1.4 Hours a Week | N | 1.5 AT | N
---|---|---|---|---|---|---|---
Personal Computer | 94 | Personal Computer | 71 | More than 30 hours | 28 | Magnification software | 50
Mobile Phone | 94 | Mobile Phone | 22 | 26-30 hours | 23 | Screen reader | 42
Tablet Computer | 47 | Tablet Computer | 5 | 6-10 hours | 14 | Assistive browser extension | 22
Gaming consoles | 36 | Other | 2 | 16-20 hours | 12 | Other | 14
Internet-enabled TV | 34 |  |  | 21-15 hours | 10 | None | 9
Smart home devices | 33 |  |  | 11-15 hours | 8 | Alternative input devices | 7
Wearable devices | 16 |  |  | 1-5 hours | 5 | Braille Display | 2
Public computers | 9 |  |  |  |  | Text Only Browser | 1
1.6 Screeen reader | N | 5.1 Age | N | 5.2 Gender | N | 5.3 Education | N
No screen reader | 40 | 25 to 34 | 33 | Male | 59 | Undergraduate | 37
VoiceOver | 18 | 45 to 54 | 20 | Female | 38 | Graduate | 26
Narrator | 12 | 18 to 24 | 18 | Non-binary | 2 | Post-secondary | 19
ChromeVox | 11 | 35 to 44 | 14 | Prefer not to say | 1 | Secondary | 15
JAWS | 10 | 55 to 64 | 12 |  |  | Prefer not to say | 3
Natural Reader | 10 | 65 or over | 2 |  |  |  |
Read&Write | 9 | Prefer not to say | 1 |  |  |  |
Talkback | 6 |  |  |  |  |  |
NVDA | 5 |  |  |  |  |  |
Other | 5 |  |  |  |  |  |
Orca | 1 |  |  |  |  |  |
5.5 Online Services | N | 1.7 Experience | N |  |  |  |
Email | 98 | Basic | 33 |  |  |  |
Shopping and e-commerce | 93 | Moderate | 29 |  |  |  |
Social media | 90 | None | 20 |  |  |  |
Payment/banking | 89 | Expert | 9 |  |  |  |
Video Streaming | 80 | Previous experience | 5 |  |  |  |
Music/audio streaming | 77 | Other | 4 |  |  |  |
Travel booking | 68 |  |  |  |  |  |
File sharing and cloud storage | 64 |  |  |  |  |  |
Education and e-learning | 62 |  |  |  |  |  |
Commutation and collaboration | 35 |  |  |  |  |  |

##  Appendix E Examples of websites, cookie notices and their outcome via
screen readers

---
(Enter webpage URL and press Enter Key)
NVDA: |  dailymail.co.uk selected
NVDA: |  UK home daily mail online
NVDA: |  Link, graphic online news, sport, celebrity, science and health stories
NVDA: |  List with 14 items
NVDA: |  Link home
NVDA: |  Link, news
NVDA: |  Link u.s
NVDA: |  [Advertisement read], [Same advertisement read]
NVDA: |  [Advertisement read], [Same advertisement read]
NVDA: |  [Advertisement read], [Same advertisement read]
NVDA: |  Discover the best black Friday deals, discover the best black Friday deals
(Down arrow key pressed)
NVDA: |  Link home
(Proceeds to read the navigation bar)
(Starts to read news items on homepage)
Figure 6: Top: Graphical representation of dailymail.co.uk with the
highlighted cookie notice at the bottom of the page. Bottom: Example
transcript via a screen reader. Figure 7: Visual representation of reddit.com
cookie notice. None of the two screen readers could audibly output the cookie
notice. They only read the body of the web page.
---
(Enter webpage URL and press Enter Key)
JAWS: |  A simple and safer way to pay and get paid, vertical bar, PayPal UK
JAWS: |  Page has three regions, 8 headings and 33 links
JAWS: |  A simple and safer way to pay and get paid, vertical bar, PayPal UK
JAWS: |  Link PayPal
JAWS: |  Navigation region, list of four items
(Reads aloud navigation bar)
(Navigate through body of site)
JAWS: |  If you accept cookies, we’ll use them to improve and customise your experience and enable our partners to show you personalised PayPal ads when you visit other sites.
JAWS: |  Link, manage cookies and learn more
JAWS: |  Accept button
JAWS: |  Decline button
Figure 8: Top: Graphical representation of paypal.com with highlighted cookie
notice at the bottom of the page. Bottom: Example transcript via a screen
reader.
---
(Enter webpage URL and press Enter Key)
NVDA: |  BBC.co.uk selected
NVDA: |  BBC Home
NVDA: |  Banner landmark, Let us know you agree to cookies, Heading level two
NVDA: |  Clickable banner landmark, We use, link, cookies to give you the best online experience. Please let us know if you agree to all of these cookies
NVDA: |  Button, Yes, I agree
NVDA: |  Link, No, take me to settings
NVDA: |  BBC Navigation landmark, BBC Homepage
(Proceeds to read the rest of the home page)
Figure 9: Top: Graphical representation of bbc.co.uk with highlighted cookie
notice at the top of the page. Bottom: Example transcript via a screen reader.