# Lessons in VCR Repair: Compliance of Android App Developers with the
California Consumer Privacy Act (CCPA)

Nikita Samarin,1,2 Shayna Kothari,1 Zaina Siyed,1 Oscar Bjorkman,1 Reena
Yuan,1 Primal Wijesekera,1,2 Noura Alomar,1 Jordan Fischer,1,3 Chris
Hoofnagle,1 and Serge Egelman1,2 nsamarin, shayna.kothari, zainasiyed, oscarb,
reenayuan, primal, nnalomar, jordan.fischer, choofnagle<EMAIL_ADDRESS>1UC Berkeley, 2ICSI, 3Drexel Kline School of LawBerkeleyCAUSA

###### Abstract.

The California Consumer Privacy Act (CCPA) provides California residents with
a range of enhanced privacy protections and rights. Our research investigated
the extent to which Android app developers comply with the provisions of the
CCPA that require them to provide consumers with accurate privacy notices and
respond to “verifiable consumer requests” (VCRs) by disclosing personal
information that they have collected, used, or shared about consumers for a
business or commercial purpose. We compared the actual network traffic of 109
apps that we believe must comply with the CCPA to the data that apps state
they collect in their privacy policies and the data contained in responses to
“right to know” requests that we submitted to the app’s developers. Of the 69
app developers who substantively replied to our requests, all but one provided
specific pieces of personal data (as opposed to only categorical information).
However, a significant percentage of apps collected information that was not
disclosed, including identifiers (55 apps, 80%), geolocation data (21 apps,
30%), and sensory data (18 apps, 26%) among other categories. We discuss
improvements to the CCPA that could help app developers comply with “right to
know” requests and other related regulations.

privacy, enhancing, technologies, compliance

††journalyear: 2023††journalvolume: 2023††journalnumber: X††doi:
XXXXXXX.XXXXXXX

##  1\. Introduction

On January 1, 2020, the California Consumer Privacy Act (CCPA) went into
effect. Modeled after the European Union’s General Data Protection Regulation
(GDPR), the CCPA is designed to increase the control of California consumers
over their personal information and offer stronger privacy protections than
those available to data subjects in the rest of the United States. Among other
provisions, the CCPA requires certain companies operating in California to
disclose their data collection and sharing practices and respond to consumers’
requests to access their personal information held by the company. This “right
to know” allows individuals to obtain information that belongs to them and
confirm that businesses comply with the data practices stated in their privacy
notices.

The required notice of data practices and the right to know what personal
information was collected by a business embody two crucial principles of data
protection: individual participation and openness (for Economic Co-operation
and Development, 1980). Businesses comply with these principles by posting
privacy policies and responding to “subject access requests” (SARs) from
consumers (known as “verifiable consumer requests” or “VCRs” under the CCPA).
Although these principles appear in other privacy frameworks, regulations such
as the GDPR and the CCPA define a stricter set of requirements and impose
heavier penalties for non-compliance than previous data privacy regimes. For
instance, the CCPA prescribes what businesses need to include in their privacy
notices and how they should respond to VCRs.

When implemented correctly, the “right to know” can greatly benefit consumers.
First, accurate information about data collection and sharing practices is
necessary to allow consumers to make informed decisions about whether and what
information to disclose to the business or whether to seek alternatives, if
necessary. Second, the ability to request data pertaining to oneself allows
consumers to amend inaccurate information held by the business (the right to
rectification) or transmit information to another business of their choosing
(the right to data portability). Awareness of the information held by the
business can also prompt consumers to request data relating to them be deleted
(the right to erasure) (Kemp, 2020) and lead to the adoption of other privacy-
enhancing technologies (PETs). As such, the right to know and other privacy
rights enabled by it serve to advance consumers’ informational self-
determination and increase their bargaining power in digital environments.

Unfortunately, scholarship has already identified shortcomings of other
privacy rights granted by the CCPA. For instance, Consumer Reports found that
consumers struggled to opt out of the sale of their personal information and
were at least “somewhat dissatisfied” with the processes they had to go
through 52% of the time (Mahoney, 2020). More recently, Nortwick and Wilson
(Van Nortwick and Wilson, 2022) found that many websites required to comply
with CCPA either failed to provide users with options to request not selling
their data to third parties or provided options that suffered from major
usability issues. Other studies have also found issues with similar privacy
laws enacted earlier, most notably the GDPR in Europe, including evidence of
non-compliance by app developers (Zimmeck et al., 2019) and personal
information leakage by abusing the right of access (Di Martino et al., 2019).
These shortcomings have to be addressed to ensure that the regulations’ stated
goal of furthering privacy protections for consumers is adequately fulfilled.

Although prior studies have focused on the impacts of the CCPA and the GDPR
(Veys et al., 2021; Di Martino et al., 2019, 2022; Kröger et al., 2020), we
were unable to find any empirical studies measuring the compliance of
businesses with the “right to know” requirements set by the CCPA, specifically
in the context of mobile applications (“apps”). We thus pose the following
research question: To what extent do Android app developers comply with the
provisions of the CCPA that require them to maintain accurate privacy notices
and respond to consumers’ access requests by disclosing personal information
that they have collected about them? We focus on mobile apps in large part
because they present inherent and unique privacy risks, as the devices they
are installed on accompany their users throughout their everyday lives and
provide access to a wide range of sensitive information, including
geolocation, health, and biometric data.

We examined the data practices of 160 top-ranked Android mobile app developers
from the U.S. Google Play Store, who we expected to meet the definition of a
“business” regulated under the CCPA and, thus, be required to comply with its
provisions. Due to ethical concerns, we focused only on the subset that
publicly posted information indicating they would be responsive to users’ CCPA
requests. We then submitted VCRs to these 109 companies by following the CCPA-
specific disclosures in their privacy policies, and compared their responses
with the actual data practices that we identified through static and dynamic
analysis of their mobile apps. We found that at least 39% of the apps shared
device-specific identifiers and at least 26% shared geolocation information
with third parties without disclosing it in response to our requests.
Furthermore, of the 69 app developers who substantially responded to our
requests, all but one disclosed the specific pieces of collected personal
information, but only 36% included the CCPA-required categories of third-party
data recipients in their responses.

The results of our work hold several important policy implications. We argue
that regulators—and, in particular, the newly-formed California Privacy
Protection Agency (CPPA)—should issue more guidance for developers to help
them better comply with the CCPA and its latest amendment, the California
Privacy Rights Act (CPRA). Such guidance should include examples of personal
information that can be collected from consumers’ mobile devices and emphasize
the legal obligations for developers who meet the definition of a “business”
regulated by CCPA. One such obligation is to provide accurate responses to
consumers’ VCRs; regulators should remind developers that they have to provide
all of the requested information, including the categories of personal
information and third parties, and ensure that the provided categories are
specific to the consumer in question.

##  2\. Background and Related Work

We provide an overview of the CCPA, including information about the required
notices and disclosures to consumers. We then highlight prior work that
investigated the accuracy of disclosures made in privacy policies, the
efficacy of subject access request mechanisms, and the potential privacy
violations that exist in online systems, including mobile apps and web-based
systems.

###  2.1. Overview of CCPA’s Requirements

The California Consumer Privacy Act (CCPA) is a state statute that was signed
into law in June 2018, becoming effective on January 1, 2020 and enforceable
on July 1 of the same year (California Consumer Privacy Act (2018), CCPA). The
CCPA secures a number of privacy rights for California consumers and imposes
new obligations on companies operating in California. In contrast to the EU’s
General Data Protection Regulation (GDPR), the CCPA only applies to for-profit
businesses that do business in California and meet any of the following
conditions (California Consumer Privacy Act (2018), CCPA; of the European
Parliament, 2016):

  * •

Have a gross annual revenue of over $25 million;

  * •

Buy, receive, or sell the personal information of 50,000 or more California
residents, households, or devices; or

  * •

Derive 50% or more of their annual revenue from selling California residents’
personal information.

Importantly, the CCPA grants consumers the right to be notified about the data
collection and sharing practices of a business and, after such collection has
taken place, the right to know the personal information that the business has
pertaining to them.

Notices to Consumers. The CCPA requires businesses to provide consumers with a
privacy policy and a notice at collection. The purpose of the privacy policy
is “to provide consumers with a comprehensive description of a business’s
online and offline practices regarding the collection, use, disclosure, and
sale of personal information and of the rights of consumers regarding their
personal information” (California Consumer Privacy Act (2018), CCPA). The CCPA
regulations require that the privacy policy is “posted online through a
conspicuous link using the word ‘privacy’ […] on the download or landing page
of a mobile application” and include the following information (California
Consumer Privacy Act (2018), CCPA):

  * •

Explanation that a consumer has the right to request that the business
disclose what personal information it collects, uses, discloses, and sells;

  * •

Instructions for submitting a verifiable consumer request;

  * •

Description of the process for verifying the consumer request, including
information the consumer must provide;

  * •

Categories of personal information the business has collected about consumers
in the preceding 12 months;

  * •

Categories of personal information, if any, that the business has disclosed or
sold in the preceding 12 months and, for each category, the categories of
third parties with whom the information was shared;

  * •

Categories of sources from which the personal information is collected; and

  * •

Business or commercial purpose for collecting or selling personal information.

In addition to the privacy policy, the CCPA requires businesses to provide
consumers “with timely notice, at or before the point of collection, about the
categories of personal information to be collected from them and the
[collection] purposes” in the form of a notice at collection (California
Consumer Privacy Act (2018), CCPA). Although businesses might choose to
maintain a separate notice at collection, they can also provide a link to the
section of the privacy policy containing the required information, as long as
the company presents the link at or before the collection of personal
information (California Consumer Privacy Act (2018), CCPA).

Verifiable Consumer Requests. The CCPA grants another fundamental privacy
right to California consumers, namely, the right to know the personal
information that a business has collected pertaining to them. Consumers can
exercise this right by submitting a “verifiable consumer request” (VCR). The
CCPA requires businesses to provide two or more designated methods for
submitting VCRs. Furthermore, businesses have 10 days to confirm the receipt
of the VCR and 45 days to complete the request, either by providing the
requested data or denying it. The CCPA allows businesses to extend the
timeline by up to an additional 45 days, provided they inform the requester of
the extension and its reasons.

As part of the VCR, consumers can request the same types of information that
is required to be in a privacy policy (see list above). However, unlike the
general data practices described in the privacy policy, the response to the
VCR has to be specific to the consumer making the request. Crucially, in
addition to the aforementioned information, a consumer can also request that
the business disclose specific pieces of personal information that it has
collected about the consumer. Unlike the GDPR (Perry and LaBruyere, 2023), the
CCPA does not require companies to disclose specific names of third parties
with whom they share the consumer’s personal information.

The CCPA regulations describe the steps that businesses must take to verify
the identity of the consumer submitting the VCR. Such verification is crucial
to ensure that the company does not disclose a consumer’s personal information
to an unauthorized party. Simultaneously, businesses need to carefully
consider the type and sensitivity of personal information to ensure that their
verification procedures do not prevent consumers from successfully exercising
their privacy rights. Furthermore, a business should avoid collecting
additional personal information solely for the purposes of identity
verification (unless absolutely necessary), it cannot impose fees for
verification, and should implement reasonable security measures to prevent
unauthorized disclosure of consumers’ personal information. If a business
maintains a password-protected account with the consumer, they can employ that
existing account’s authentication mechanisms to verify the consumer’s
identity. Otherwise, the business is required to verify the requester’s
identity to a “reasonable degree of certainty” by matching either two (before
disclosing categories of personal information) or three (before disclosing
specific pieces of personal information) data points provided by the consumer
with data points maintained by the business.

The CCPA defines a consumer as a California resident “however identified,
including by any unique identifier,”111Cal. Civil Code §1798.140(g). which
means that consumers need not use their real names to identify themselves when
making VCRs. That is, the CCPA allows consumers to use pseudonyms when
transacting with businesses and exercising their privacy rights, and does not
require that they divulge their legal names to make VCRs (i.e., for
verification, it only needs to match the personal information previously
collected by the business).

###  2.2. Comparison with the GDPR

The EU’s General Data Protection Regulation (GDPR), which went into effect on
May 25, 2018, is considered to be one of the most comprehensive data
protection laws to date (Burgess, 2020). Similar to the CCPA, the GDPR offers
strong privacy protections to individuals and imposes obligations on
businesses conducting business in Europe. In particular, the GDPR also
requires companies to disclose their data collection and sharing practices in
a privacy policy and respect individuals’ right to be informed and right of
access to personal information pertaining to them.

Despite the similarities in the rationale between the CCPA and GDPR, there are
also important differences with regard to the scope and application of
specific provisions (Jehl and Friel, 2018; DataGuidance and of Privacy FOrum,
2018):

  1. (1)

Personal Scope. The GDPR applies broadly to entities that establish the means
and purposes of the processing of Europeans’ personal information, covering
natural and legal persons, for-profit, non-profit, and public entities, small
and large organizations, irrespective of their size or revenue. On the other
hand, the CCPA only applies to for-profit businesses subject to the criteria
enumerated in Section 2.1.

  2. (2)

Material Scope. The CCPA excludes specific categories of personal information
from its scope of application covered by industry-specific federal privacy
laws, whereas the GDPR does not feature such exceptions. For instance, medical
information covered by the Health Insurance Portability and Accountability Act
(HIPAA) and financial information covered by the Gramm-Leach-Bliley (GLB) Act
are both outside of the scope of application of the CCPA.

  3. (3)

Required Notices. The CCPA requires covered businesses to disclose in their
privacy policies the categories of personal information collected, sold, or
disclosed for a business purpose in the preceding 12 months.

  4. (4)

Right of Access. The CCPA mandates that companies provide personal information
requested by the consumer under the right to know “in a portable and, to the
extent technically feasible, readily usable format that allows the consumer to
transmit this information to another entity without hindrance,” effectively
establishing the right to data portability. In contrast, the GDPR separates
the right of access and the right to data portability, which have their own
conditions.

  5. (5)

Procedures. The CCPA requires organizations to respond to consumers’ request
in 45 days starting with the receipt of the request, extendable once by an
additional 45 days. The GDPR requires covered entities to respond within one
month, extendable once by an additional two months.

  6. (6)

Penalties. The GDPR empowers competent data protection authorities to both
assess any violations of the law and directly issue fines to entities. In
contrast, the Attorney General of the State of California is responsible for
assessing violations of the CCPA and bringing civil actions against the
offending businesses to seek statutory damages in court.

The next section provides an overview of prior studies investigating the
efficacy of the right of access, primarily under the GDPR. We believe that
although the methodologies and general findings are applicable to our study,
the highlighted differences between the two data protection laws also
necessitate the present exploration of businesses’ compliance with the CCPA.

###  2.3. Efficacy of Subject Access Requests

Our work relates to prior studies that investigated how effective subject
access request (SAR) mechanisms are in helping data subjects exercise their
rights (Veys et al., 2021; Di Martino et al., 2019, 2022; Kröger et al., 2020;
Adhatarao et al., 2021; Alizadeh et al., 2020; Bufalieri et al., 2020; Urban
et al., 2018). In (Urban et al., 2019b), SARs were sent to 38 third-party
businesses in an effort to evaluate how they comply with Article 15 of the
GDPR, and the study showed that most failed to properly disclose all relevant
user data in their responses to the requests. Urban et al. (Urban et al.,
2018) sent SARs to 36 organizations and found that 58% delayed responding to
the requests. Kröger et al. (Kröger et al., 2020) sent similar requests to app
developers over a period of a few years and identified potential weaknesses in
the processes developers followed to handle and respond to such requests,
which continued to exist even after GDPR became enforceable. Similarly, the
results of sending SARs to businesses in (Ausloos and Dewitte, 2018)
highlighted the difficulty they experienced finding all data needed to respond
to the requests. The authors also emphasized the importance of using
automation whenever possible when responding to SARs and developing templates
that businesses can follow so that they can reach a state of “legal
certainty,” where they can be assured that they are in compliance with laws
that provide users with the right to access their data. Tolsdorf et al.
(Tolsdorf et al., 2021) identified data incompleteness and inconsistency
issues when evaluating the accuracy of information displayed in privacy
dashboards for a number of service providers.

Herrmann and Lindemann (Herrmann and Lindemann, 2016) observed that businesses
were more likely to respond to data deletion requests than subject access
requests, and identified websites that adopted SAR mechanisms that made them
vulnerable to revealing their users’ data in their responses to adversarial
data access requests. In a number of other studies, researchers further
examined how businesses’ SAR mechanisms can be used by adversaries to extract
subjects’ personal data through social engineering attacks (e.g.,
impersonation) (Di Martino et al., 2019, 2022; Pavur and Knerr, 2019; Cormack,
2016; Boniface et al., 2019; Cagnazzo et al., 2019). Di Martino et al. (Di
Martino et al., 2019) showed how these types of attacks can be mounted against
a number of organizations by relying on information that is available to the
public. In their follow-up work (Di Martino et al., 2022), they proposed
alternative approaches to authenticating data subjects that can help
businesses strengthen their SAR mechanisms by reducing the likelihood of
leaking subjects’ personal data when responding to data access requests made
by adversaries. Jordan et al. (Jordan et al., 2021) focused specifically on
addressing the problem of how organizations can respond to data access
requests that do not have corresponding user accounts.

While prior work has investigated organizations’ responses to SARs from a
number of different perspectives, we believe that the literature is yet to
paint a complete picture on the extent to which responses to SAR are
consistent with disclosures made in privacy policies and actual system
behaviors. Researchers investigated whether SAR processes are sufficiently
explained in privacy policies or aligned with the requirements of applicable
laws and compared privacy policy disclosures to responses to SARs (Urban et
al., 2019b; Boniface et al., 2019; Bufalieri et al., 2020; Bowyer et al.,
2022; Urban et al., 2018), but we are unaware of studies that compared
organizations’ responses to actual system behaviors. We systematically compare
information obtained from the three sources of information we considered:
privacy policies, responses to SARs and actual app behaviors.

Researchers also studied the usability of subject access request and deletion
mechanisms from a number of different angles, including the ease of initiating
the requests as well as the extent to which the content of the responses can
be understood by average users (Veys et al., 2021; Wei et al., 2020; Habib et
al., 2020; Bowyer et al., 2022; Urban et al., 2019a). After investigating
users’ awareness of their rights under the GDPR in (Kuebler-Wachendorff et
al., 2021), researchers found that users do not have sufficient understanding
of their “right to data portability.” Habib et al. (Habib et al., 2020)
uncovered challenges users experience with locating information related to how
to exercise their privacy rights and correctly using the privacy controls made
available to them by businesses. Veys et al. (Veys et al., 2021) observed how
real users interacted with the content of the responses obtained from
businesses after requesting to download their data. They found that most
responses are yet to be considered accessible to users and identified areas
where future improvements can be made to better align these responses with
user expectations (Veys et al., 2021). Urban et al. (Urban et al., 2019a)
highlighted the importance of improving the designs of current user-facing
tools provided by organizations to allow users to understand how their data is
used. After studying the extent to which responses to SARs submitted to
Twitter are empowering real users to understand how their data was used in ad
targeting, Wei et al. (Wei et al., 2020) similarly found content-related
issues that might negatively affect how understandable and readable ad
explanations are to users. Table 1 compares some of the key metrics of this
study with those of prior work.

Table 1. Comparison of Key Metrics with Related Work Study |  | Request
---
Count
| Response
---
Count
| GDPR or
---
CCPA?
| Policy
---
Analysis?
| App
---
Analysis?
This | 109 | 80 (73%) | CCPA | Yes | Yes
(Adhatarao et al., 2021) | 109 | 62 (57%) | GDPR | No | No
(Urban et al., 2018) | 36 | 32 (89%) | GDPR | Yes | No
(Cagnazzo et al., 2019) | 14 | 14 (100%) | GDPR | No | No
(Bufalieri et al., 2020) | 326 | 212 (65%) | GDPR | Yes | No
(Pavur and Knerr, 2019) | 150 | 112 (75%) | GDPR | No | No
(Kröger et al., 2020) | 225 |  | 43–58
---
(19–26%)
GDPR | No | No
(Ausloos and Dewitte, 2018) | 60 | 44 (73%) | — | Yes | No
(Di Martino et al., 2022) | 40 | 34 (85%) | GDPR | No | No
(Di Martino et al., 2019) | 55 | 51 (93%) | GDPR | No | No
(Urban et al., 2019b) | 38 | 16 (42%) | GDPR | Yes | No
(Herrmann and Lindemann, 2016) |  | 150 apps
---
120 sites
43% | GDPR | No | No

###  2.4. Analysis of Privacy Policy Disclosures

Others have focused on understanding apps’ and websites’ privacy practices by
analyzing disclosures made in privacy policies (Harkous et al., 2018; Andow et
al., 2020; Wang et al., 2018; Zimmeck et al., 2019, 2017). Some proposed
systems, such as POLICHECK (Andow et al., 2020), MAPS (Zimmeck et al., 2019)
and HPDROID (Fan et al., 2020), which automated the process of comparing
disclosures made in privacy policies about how user data is used, collected,
or shared with personal data transmissions observed as a result of performing
technical analyses (Wang et al., 2018; Andow et al., 2020; Zimmeck et al.,
2019, 2017; Slavin et al., 2016). The literature also proposed systems, such
as Polisis (Harkous et al., 2018), PI-Extract (Bui et al., 2021) and
PrivacyFlash (Zimmeck et al., 2021), which made it possible to transform
privacy policies into formats that are more understandable to users or auto-
generate policies that reflect actual app behaviors. Linden et al. (Linden et
al., 2018) found that disclosures made in privacy policies improved as a
result of GDPR enforcement, but that more improvements would have to be made
before they can be considered usable and transparent to users. Other recent
studies have also examined the accuracy of disclosures made in privacy
policies (Andow et al., 2019; Okoyomon et al., 2019; Wang et al., 2018).
Compared to prior studies, we follow a systematic approach to analyzing apps’
privacy policies by having coders answer a set of questions that are
reflective of the requirements of the CCPA.

###  2.5. Data Practices of Mobile Apps

Finally, our work also relates to prior studies that investigated potential
privacy violations in online systems, including mobile apps and websites
(Nguyen et al., 2021; Reyes et al., 2018; Han et al., 2020). To examine the
extent to which apps comply with privacy regulations, researchers relied on
static and dynamic app analysis tools to identify potential legal violations
at scale (Reyes et al., 2018; Han et al., 2020; Feal et al., 2020; Nguyen et
al., 2021; Ren et al., 2018; Jia et al., 2019). These studies identified a
range of deceptive data collection and transmission practices and highlighted
the need for stronger enforcement actions by regulators. While prior work has
examined apps’ level of compliance with privacy regulations by looking into
network flows, privacy policy disclosures, or responses to data subject access
requests, our investigation compares the data obtained across all three of
these sources to evaluate the effect of the CCPA on developers’ privacy
practices. We believe that our work, therefore, is crucial in evaluating the
overall efficacy of CCPA and its utility to mobile app users.

##  3\. Methodology

We aim to uncover contradictions between personal information…

  1. (1)

that we record being collected and transmitted by an app using dynamic and
static analysis;

  2. (2)

disclosed to us in response to a “right to know” request we made after using
the mobile app; and

  3. (3)

that the app developer claimed to collect in their app’s privacy policy.

The following sections cover each part of the study in more detail. We
additionally describe our procedure for selecting the Android apps that we
examined, as well as our procedure for testing the apps and submitting the
verifiable consumer requests.

###  3.1. Dataset

We focused on the 8 top-ranked Android mobile apps in the 20 Google Play Store
categories that have the highest number of cumulative app installs. Companies
developing these apps fall or can be reasonably inferred to fall under the
CCPA definition of a “business.”222A “business” includes mobile app developers
that are for-profit entities and conduct business in California (i.e., make
their applications available in California) and meet at least one of the
following three criteria: (1) collect the personal information of at least
50,000 consumers in California; (2) have an annual gross revenue in excess of
$25 million, or (3) derives 50 percent or more of its annual revenues from
selling California consumers’ personal information (CCPA, 1798.140(c)). We
selected only one mobile app (with the highest user install count) per
developer in order to have the ability to match the personal information
disclosed by the developer with the app that we tested and to examine a
broader range of developer practices for responding to VCRs.

Furthermore, we replaced certain apps that we were unable to test. This
included apps, for instance, that required business accounts, financial
information, or additional hardware devices. This selection procedure produced
a total of 160 unique apps, which we downloaded with their privacy policies in
November 2021.

It is important to note that, although our procedure was designed to select
developers that we expected to be covered by the CCPA, the resulting list was
only an approximation (i.e., we could not be sure that all of these developers
were actually subject to the CCPA), as we used the number of app installs to
gauge the total number of California consumers from whom an app may have
collected personal information. Nonetheless, we could not be sure, and as an
ethical matter, we did not want to waste people’s time by submitting VCRs to
organizations that were not required to respond to them. Thus, we further
limited our study to only those companies that explicitly mentioned CCPA in
their privacy policies. Under the FTC Act33315 U.S.C. §45. (and various other
state consumer protection laws), businesses in the U.S. are prohibited from
materially misrepresenting their practices to consumers. This includes making
false statements in privacy policies, which the FTC enforces (e.g.,
(Commission, 2021)). Thus, any business that states in their privacy policy
that they respond to CCPA VCRs must actually do so, regardless of whether or
not they are actually covered by the CCPA.

Two researchers from our team independently read the text of 160 privacy
policies to determine whether or not each contained references to the CCPA.
For cases without a majority consensus, a third researcher provided the tie-
breaking vote. Our analysis indicated that out of the selected 160 apps, 109
(68%) include CCPA-specific disclosures in their privacy policies (with
Krippendorff’s alpha = 0.81, indicating an acceptable level of inter-rater
agreement (Krippendorff, 2011)). For the remainder of this paper, our
discussion will focus primarily on these 109 apps.

###  3.2. App Analysis

We used an instrumented version of Android 9.0 (Pie) that monitored resource
accesses (e.g., access to Android APIs) and logged all network traffic,
regardless of the use of TLS. (Prior published work has applied a similar
approach (Reardon et al., 2019; Sanfilippo et al., 2020; Andow et al., 2020;
Han et al., 2020; Alomar and Egelman, 2022).) Because network traffic was
captured at the OS level (as opposed to using a proxy), we were still able to
observe and decrypt transmissions that were secured using certificate pinning.
Since the values of identifiers (and other personal information) were known
for each device, our tools automatically searched for various permutations in
the captured network traffic, including hashes (e.g., MD5, SHA-1, SHA-256,
etc.).

Using this instrumentation on Google Pixel 3a devices, we automatically
recorded decrypted network traffic, which included destinations (i.e.,
hostname, port, IP) and payloads. Decrypted traffic payloads included API
endpoints and key/value pairs. All network traffic was attributed to specific
apps and their SDKs, using a combination of kernel-level instrumentation to
attribute sockets to processes and stack inspection to identify specific SDKs.
A variety of open source tools for collecting network traffic can be used to
verify our results and, we believe, reproduce our findings from scratch (e.g.,
(Frida, 2022; OWASP, 2022)). While the instrumentation was specifically
written for Android Pie (9), which was released roughly three years prior to
our testing, millions of people still use Pie (e.g., at the time that we
conducted our study, roughly 20% of US Android users were using Pie or earlier
(statcounter, 2021)), many with CCPA rights. We also have no reason to believe
that the same app binaries would be more/less compliant under newer Android
versions.

#### Pseudonyms.

Similar to (Zang et al., 2015), we generated pseudonyms and other fictitious
values for different types of personal information covered by the CCPA to
facilitate the subsequent search for this data in the logs produced by app
testing and to improve the ecological validity of our study. Our motivation
behind using “fake” data was to reduce the number of confounding variables:
while all experimenters were California residents, if we used our real names
and identifiers, we would not know whether data received from CCPA VCRs was
collected by the company during the study period or before (or possibly from
other sources).

The CCPA defines a consumer as a California resident “however identified,
including by any unique identifier,”444Cal. Civ. Code §1798.140(g).,
therefore, the usage of fictitious data did not legally affect the requirement
of the companies to respond to our requests. This provision ensures that
companies that only collect pseudonyms are still subject to CCPA requests,
while also disincentivizing companies from collecting additional personal
information solely for the purpose of responding to requests. A physical
address (and email, phone number, etc.) can be fictitious, so long as they can
be used to identify the California consumer who is the data subject. Thus, the
use of pseudonyms both reduced confounding factors and was legally valid.

We produced pseudonymous data using random value generators, such as the
Random Lists (Random Lists, 2023) website and Faker Python package (joke2k,
2023). We obtained other types of personal information, including device
identifiers and geolocation data, directly from our test devices. We present
our data taxonomy in Appendix C, while Table 9 provides examples of personal
information that we used.

#### Testing Procedure.

We manually tested the selected 109 apps, each for approximately 15-20 minutes
using test phones with our instrumented version of the Android operating
system. We set up each test phone—to be used by an individual tester in
California—to use its own set of pseudonymous identifiers, such as the phone
number, email address, usernames, and other types of information. During each
test, we created a user account for the app (if applicable) and input the
predefined pseudonymous data corresponding to the specific test phone, as
described above. We later searched for the predefined data values within the
resulting test logs (which included captured network traffic), as well as
performed an open-ended search to see if the app transmitted other personal
data.

#### Data Recipients.

Apps can transmit data both to first- and third-party destinations in order to
deliver essential and non-essential functionality. Specifically, we might
observe an app transmit the same personal information only to domains
controlled by the app developer or to a combination of first- and third-party
endpoints.

First, we categorized the observed destination domains as either first- or
third-party for each tested app. Using the same approach as in (Trimananda et
al., 2022), we tokenized the destination domain and the app package name. We
then classified a specific domain as first party if its tokens appeared in the
app’s privacy policy URL or matched the package name’s tokens, otherwise, we
labeled the domain as third party. Next, we went over the resulting party
labels for each domain and manually corrected any mistakes. For each third-
party domain, we also obtained the effective second-level domains (eSLD) using
tldextract and used it to locate the entity that controls it using Crunchbase,
Netify, and other online resources. Two researchers from our team assigned a
category to each third-party domain using the information that we obtained
from our online search, which we then used to compare against the categories
of recipients in VCR responses and privacy policy disclosures.

The CCPA recognizes that a first party can either directly or indirectly
collect personal information.555Cal. Civ. Code §1798.130(a)(3)(A). As such,
the collection of personal information via third parties (either service
providers or third parties under the CCPA) still triggers the CCPA obligations
on the first party as if the app developer directly collected the personal
information itself. The liability of first parties for third-party app and
website data collection has been affirmed by People of the State of California
v. Sephora USA, Inc. (Court, 2022).

For this reason, we labeled each data point (e.g., for purposes of Table 4)
that we observed being captured and transmitted to a third-party domain (e.g.,
using SDKs, codebases, or other pieces of code in the app) as collected both
by the first party (i.e., the app developer) and the third party. We
categorized the data point as collected by the first party if the app
transmitted it only to domain(s) controlled by the app developer.

###  3.3. Verifiable Consumer Requests

For each tested app, we identified directions in its privacy policy for how to
submit a verifiable consumer request (VCR). To avoid abusing the time and
resources of developers who do not have to comply with the provisions of the
CCPA, we erred on the side of caution and only submitted verifiable consumer
requests to developers who explicitly referenced the CCPA in their privacy
documents.

As part of each request, we asked to obtain all types of information that a
business is required to provide under the CCPA in response to a consumer
request:

  1. (1)

specific pieces and categories of personal information requested, collected,
and shared by the app;

  2. (2)

categories of sources from which the personal information was collected;

  3. (3)

business or commercial purposes for collecting the personal information; and

  4. (4)

specific names and categories of third parties with whom the app developer
shared personal information.666CCPA does not require businesses to disclose
specific third parties, however, some app developers opt in their privacy
notices to provide that information upon request.

We submitted each request from the same pseudonymous email account that was
used to test the app. We employed email templates to ensure a level of
uniformity when, for instance, we submitted the initial requests, sending
follow-ups if the developer did not respond, asking for an alternative
identity authentication mechanism, etc. We provide the email templates that we
used to submit the requests and follow up with the developer in Appendix A.
Nevertheless, some developers still instructed us to use an alternative method
for submitting the request, such as a privacy management platform.

###  3.4. Privacy Policy Analysis

Additionally, we analyzed disclosures made in the privacy policies of tested
apps using a deductive approach to qualitative coding. Our codebook contains
codes for the collection and sharing of categories of personal information
taken from Cal. Civil Code 1798.140. One of the authors with experience
assisting companies in complying with the CCPA requirements developed the
codes for the categories of third parties. We include the resulting codebook,
code descriptions, and prompts in Appendix B.

As discussed previously, we first identified whether each policy contained
references to the CCPA using the following prompt: “Does this app developer
include disclosures that reference the CCPA, either as part of the general
privacy policy or as a standalone document?” We then analyzed each of the 109
privacy policies containing CCPA-specific information to identify information
about the developer’s data collection and sharing practices. In particular,
for each category of personal information defined under the CCPA (e.g.,
identifiers or geolocation), we examined whether an app developer collected or
disclosed each category and to which category of recipients.

At least two annotators from our team first independently located the relevant
privacy policies, and then used the prompts enumerated in Table 5 to locate
the disclosures that pertained to the collected and shared categories of
personal information and the categories of third parties. We then computed
Krippendorff’s $\alpha$ to evaluate the inter-rater reliability on a per-
question basis (Krippendorff, 2018). We resolved any divergences in our
responses using a majority vote or, if a majority was absent, a third
researcher independently provided the tie-breaking vote. After resolving the
disagreements, we obtained a list of categories of personal information and
recipients that we compared against our app analysis results.

###  3.5. Comparison

We compared these three data viewpoints to quantify the accuracy and
completeness of the information disclosed by the developers. We first compared
each specific piece of personal information that we observed being collected
and shared with the specific pieces of information disclosed by the developer
in the VCR, when applicable. In this case, we simply matched the values that
we observed being collected and shared with the values provided to us by the
developer. As mandated by the CCPA, we only accepted responses containing the
values (and not just the types of information) to be valid with respect to
disclosing the specific pieces of personal information.

Furthermore, we compared the categories of collected and shared information
and the categories of recipients that we observed during app testing with the
the same categories disclosed in the VCR and privacy policy. We only
considered the categories disclosed in the VCR responses to be valid if we
were able to sufficiently match them with the CCPA-defined categories of
personal information and to our categories of third parties. These categories
included common types of recipients that we observed across different app
privacy policies and VCR responses, such as advertising networks, marketing
partners, analytics providers, fraud and security, search engines, social
media networks, payment processors, customer support providers, storage and
infrastructure, affiliates, and law enforcement. We obtained the same
categories of personal information and third parties from the privacy policies
using the qualitative coding approach discussed previously. The CCPA-defined
categories of personal information as well as the categorization of our own
PII types are presented in Table 9.

Once we had obtained these categories, we identified the categories that the
developer had collected but not disclosed by looking at the difference between
categories that we observed during app testing and the categories provided by
the app developer in the privacy policy and VCR.

###  3.6. Ethics

We performed a study of institutional processes and did not collect data about
individuals (of Human Research Protections, 2022). As such, our IRB determined
that our study did not meet the legal definition of human subjects research,
and therefore declined to review it. We nonetheless spent over a year
deliberating how to conduct it ethically, including avoiding guessing whether
a company was subject to CCPA, not incurring costs by asking legal questions,
and making sure correspondence was not perceived as legal threats, ethical
issues that have come up for other researchers (Mayer, 2021). Instead, we
performed a measurement study of publicly-available services by exercising our
legal rights using the methods companies themselves prescribed.

We acknowledge that some companies may not have automated systems to process
CCPA requests, and therefore processing our VCRs may have imposed costs on
them. However, we believe that business’ interests in this regard are
outweighed by the public interest in understanding CCPA effectiveness. This is
also a straw man argument: all individuals who made CCPA requests for our
study were legitimately interested in learning about companies’ privacy
practices and made legally-valid requests to do so; that they additionally
followed a prescribed methodology and shared the results for research purposes
does not suddenly make the requests invalid or unethical. CCPA empowers
California residents with rights, which must be honored regardless of intent.

##  4\. Analysis

We present the results from submitting the VCRs, focusing on the methods
available to do so, the types of information required to initiate and verify
requests, and the percentage of developers who completed the requests, with an
emphasis on the disclosure of the CCPA-specific information, as enumerated in
Section 3.4. Furthermore, we compare the personal information provided to us
by the developers with our dynamic analysis of their Android apps.

###  4.1. Access Requests

We analyzed the 109 apps with CCPA-specific information in their privacy
policies. Whenever possible, we created an account with each app using an
email address created specifically for this study and unique to the testing
phone. As a result, we registered accounts with 91 (83%) apps.

The majority of developers (66%) provided at least two methods for submitting
the VCR. The most common method was by email, with 71 (65%) companies offering
it as an option. The next most common method was a dedicated VCR form or
portal offered by 42 (39%) companies. Notably, 15 of these companies relied on
OneTrust (Trust, 2021), a third-party suite of products that includes support
for SAR management, with the remaining 27 either relying on another third-
party provider or implementing their own solutions. We identified a number of
other methods for submitting requests, including a phone number (25%), contact
via customer support service (19%), physical mail (19%), account or in-app
privacy settings (15%) or through a Google Form (2%).

Table 2. Distribution of Methods for Submitting VCRs Name of Method | Count | Proportion
---|---|---
Email | 71 | 0.65
Company DSAR Portal | 27 | 0.25
Phone | 27 | 0.25
Customer Support Service | 21 | 0.19
Physical Mail | 21 | 0.19
OneTrust DSAR Portal | 15 | 0.14
Account Privacy Settings | 11 | 0.10
In-App Privacy Settings | 5 | 0.05
In-App Feedback Form | 3 | 0.03
Google Form | 2 | 0.02

Whenever possible, we submitted VCRs using email or a customer support
service. In these cases, our messages to the companies included a self-
attestation of California residence, as well as the pseudonyms and email
addresses associated with the phone used for testing the app. However, in 16
cases, the app developer directed us to use an alternative VCR submission
method other than the one we had chosen. Ultimately, we submitted 52 VCRs via
email and 6 VCRs via customer support or a feedback form out of the total 109
requests sent out. We submitted the remaining 51 requests either using a
provided VCR portal (34%) or within an app’s or account’s privacy settings
(13%).

When using a dedicated form, portal, or in-app privacy controls to submit the
VCR, we generally received a confirmation of the request within the same user
interface. For this reason, we focused on the 58 apps that required a free-
form request submission to see if the companies would confirm the receipt of
our request within the statutory 10 day period mandated by the CCPA. Out of
these 58 companies, 40 (69%) explicitly confirmed our request, whereas the
remaining 18 (31%) did not.

Companies also must verify the identity of consumers submitting VCRs to ensure
they do not inadvertently disclose personal information to someone
impersonating the data subject. Therefore, we also recorded information that
we provided or any authentication steps we performed to verify our VCR (Table
3). We implicitly verified the ownership of our email address in 52 instances,
when we made the request via email. For all other cases, 32 companies
requested email verification after submitting the request, typically by
clicking a link or providing a unique PIN sent to the testing email address.
Furthermore, 35 companies required us to successfully log into our accounts
either to submit or to verify the VCR.

App developers also requested specific pieces of personal information to match
against their records, either as part of the initial VCR submission process or
by following up with us after we submitted our requests. Most often,
developers asked us to provide some basic information about ourselves,
including, our email address (36 instances), full name (26), state (21), and
country of residence (15). Developers also requested technical information
that is not always easily accessible for smartphone users. In particular, we
were asked to provide the Android Advertising ID (AAID) in 5 cases, a company-
defined ‘device‘ or ‘user‘ ID in 5 cases, and our current IP address in 2
cases. Table 2 presents a breakdown of the different types of information or
actions required to verify the VCRs.

Some companies had more stringent requirements to complete their identity
verification, either at the moment or after submitting the VCR. Five companies
out of 109 required us to certify the accuracy of the provided information
under penalty of perjury and 4 required a signed affidavit that, in at least
one case, had to be notarized.777This is explicitly prohibited by regulations
(§999.323(d)). Furthermore, two companies requested proof of phone number
ownership by providing a recent mobile operator bill, another two asked for
photocopies of a government-issued ID, and one company outsourced identity
verification to the ID.me service, which allows an individual to verify
themselves either by providing a photocopy of their government-issued ID or
their phone number to allow a look-up with the mobile operator records.
Finally, one company asked us to “make [ourselves] available for a phone call
with a [redacted] customer service representative who will call from [their]
privacy line.” In these instances when we could not furnish such documents, we
requested an alternative verification method through logging into our account
and providing details of that login to the company, if applicable. The CCPA
regulations explicitly provide for such an alternative verification method for
account-holders. Two companies agreed, and allowed us to verify our identity
using the alternative verification method.

The majority of companies, namely 102 or 94%, did not ask for proof of our
California residency. Out of the remaining 7 app developers, three asked us to
provide proof of our address (e.g., a bank statement or a recent utility
bill), one requested a government-issued ID showing California residency
(e.g., a California driver’s licence), one asked us to sign a declaration of
California residency under the penalty of perjury, and the remaining two
requested California state residency verification via ID.me and the phone
call, as described previously.

Table 3. Methods or Information Required to Verify VCR Method or PII Type | Count
---|---
Email | 36
Account Authentication | 35
Email Authentication | 32
Full Name | 26
State of Residence | 21
App-specific Information | 18
Country of Residence | 15
Username | 9
Phone Number | 7
Postal Address | 6
Device or User ID | 5
Android Advertising ID (AAID) | 5
Certification w/ Penalty of Perjury | 5
Signed Affidavit | 4
Photocopy of a Government-issued ID | 3
Phone Authentication | 3
Current IP Address | 2
Date of Birth | 2
ID.me | 1
Call with a Company Representative | 1

###  4.2. Developer Responses

Out of the 109 requests that we sent out, we did not receive a response from
the developer in 21 (19%) cases. In these instances, the developer either did
not respond to the initial request or became unresponsive after a brief
interaction, for instance, after asking for verification. In all of these
cases, we followed up with the app developers at least once to confirm that
they were unresponsive.

We were unable to verify our identity to the company’s satisfaction in 5 (5%)
other cases, as we were unable to produce the requested documentation and the
company did not agree to use an alternative method. Finally, 3 (3%) developers
could not verify our identity to a sufficient degree and, thus, did not
respond with any personal information. We excluded these 29 cases from our
analysis of the responses and focused on the remaining 80 responses.

#### Human vs. Automated Responses

We first identified the proportion of companies employing automation when
responding to our VCRs. Similar to (Urban et al., 2019b), we labeled responses
that directly answered to our questions as “human.” In contrast, we marked
responses sent by a computer system (e.g., help desk ticketing software) or
containing only generic privacy-related information as “automated.” Out of 80
responses, we labeled 32 (40%) responses as “human” and the remaining 48 (60%)
as “automated.”

#### Follow-up Actions

We first examined the number of actions that the data subject would have to
perform to successfully receive a response to their VCR, and the amount of
time they would have to wait for the company to reply back. Across the 80
responses, we performed an average of 1.8 (±0.78, median = 1) actions to
obtain our VCR response, including submitting the request, passing identity
verification, following up with the developer, etc. The most actions that we
performed was 4. Additionally, it took us 14.86 (±18.86, median = 7) days on
average to receive responses to our VCRs, however, the average was skewed
heavily by developers who instantly replied back with the response (e.g., if
made through in-app account settings) and those that took extraordinarily
long, with the longest duration to complete the request of 76 days.

#### Composition of the Response

Out of these 80 companies, 69 (63%) provided data in response to our request,
8 (7%) replied that they held no data on us and the remaining 3 (3%) told us
to obtain the requested information directly from our account profile.

For the 69 companies that provided us data, we examined whether they provided
all types that a business is required to provide under the CCPA (Section 3.4).
All but one app developer provided us with specific pieces of information in
their responses. However, compliance with other parts of the CCPA’s right to
know was less uniform. For instance, only 24 (35%) companies provided the
categories of personal information collected from us, 18 (26%) provided the
categories of personal information disclosed or sold to third parties, 25
(36%) provided the categories of those third parties, 30 (43%) responded with
the business or commercial purpose for collecting or selling our personal
information, and 23 (33%) disclosed the sources, from which our information
was collected.

#### Compliance

The relatively high compliance with the request to provide specific pieces of
information is not surprising, as many app developers are likely using tools
to automatically respond to CCPA (and GDPR) requests by integrating with and
pulling data from their internal customer relationship management (CRM)
platforms. Furthermore, in most cases, even when an developer provided the
categories of collected or shared personal information or the categories of
third parties, sources, or purposes, these disclosures came directly from
their privacy policies. We mark these cases as valid disclosures, as we are
unable to verify whether those categories in fact apply to our case or not
from the developer’s response alone.

#### Response Format

The 69 companies that replied with the personal information collected about us
communicated this information to us in a number of ways, including 23 (33%)
companies that included the data directly in the email reply or as an email
attachment, 19 (28%) that provided the data as an attachment on the VCR
platform, and 12 (17%) that made it available from account or in-app privacy
settings. The remaining 15 companies used a variety of methods to transmit the
data to us, including, as a file shared with us via a cloud storage provider,
as a download link in the email reply, or via a message sent to us through a
customer support portal.

#### Security of the Process

We looked at the security mechanisms (if any) used by the developers of the 69
apps to securely communicate our personal information to us, beyond our email
provider’s access controls. At least 43 companies used an expiration time on
the download links or files that they shared with us, ranging anywhere from 24
hours to 90 days. However, in 4 of these cases, we verified that the files
remained accessible and downloadable even after the stated expiration time.
Additionally, 26 app developers relied on their standard account
authentication for access control, 2 used Gmail’s “confidential mode” and 3
relied on other access controls, such as those enforced by cloud storage
providers. Additionally, 16 companies required email verification to access
and download the file, while 9 secured the data file by setting a password to
open it, which they communicated separately to us.

#### Data Format

We looked at the format and characteristics of the 62 data files that
contained specific pieces of collected personal information. Developers
relayed the files using a number of formats, including CSV (27 instances),
JSON (18), PDF (12), Excel (11), and TXT (9). Only 6 companies presented the
same data using two different formats, whereas the remaining 56 either used a
single format or a combination of several comprising a single data record.

###  4.3. Comparison with App Analysis Results

We strive to not only to understand the process of submitting a VCR under the
CCPA, but also the accuracy of the data provided back to us. We first focus on
the 68 companies who replied with the specific pieces of personal information.
In this case, the response to the VCR included specific values that were
collected by the developers, therefore, we simply matched the values from the
VCR with the data that we observed being transmitted over the network.

Only 9 apps that provided us the specific pieces of personal information fully
disclosed the extent of their data collection practices. With respect to the
enumerated list of categories of personal information defined by the CCPA, we
observed the collection, but not the disclosure, of identifiers by 55 apps,
geolocation data by 21 apps, sensory data by 18 apps, customer record
information by 16 apps and, to a lesser extent, professional information in 4
cases, characteristics of protected classifications (e.g., gender or age) in 3
cases, and education information in one case.

Table 4. Counts of Apps that Collected but not Disclosed Various PII

Category | Subcategory | PII Name | #Apps | TLS # | 1st Party # | 3rd Party #
---|---|---|---|---|---|---
Identifiers | User | Username | 3 | 3 (100%) | 0 (0%) | 3 (100%)
| Network | IP Address | 23 | 21 (91.3%) | 9 (39.1%) | 20 (87%)
|  | Router MAC | 8 | 8 (100%) | 2 (25%) | 6 (75%)
|  | Router SSID | 8 | 7 (87.5%) | 3 (37.5%) | 6 (75%)
| Device | AAID | 49 | 44 (89.8%) | 32 (65.3%) | 43 (87.8%)
|  | Hardware ID | 1 | 1 (100%) | 0 (0%) | 1 (100%)
|  | IMEI | 3 | 3 (100%) | 2 (66.7%) | 1 (33.3%)
|  | IMSI | 2 | 2 (100%) | 1 (50%) | 1 (50%)
|  | SIM ID | 2 | 2 (100%) | 1 (50%) | 1 (50%)
|  | Wi-Fi MAC | 1 | 1 (100%) | 1 (100%) | 0 (0%)
|  | Fingerprint ID | 25 | 23 (92%) | 2 (8%) | 25 (100%)
| App | Identity ID | 16 | 15 (93.8%) | 1 (6.2%) | 16 (100%)
|  | App Fingerprint ID | 10 | 7 (70%) | 8 (80%) | 8 (80%)
|  | Android ID | 20 | 17 (85%) | 10 (50%) | 18 (90%)
Customer Records | Customer | Phone Number | 5 | 4 (80%) | 5 (100%) | 1 (20%)
| Contacts | Name | 2 | 2 (100%) | 2 (100%) | 0 (0%)
|  | Phone Number | 5 | 5 (100%) | 5 (100%) | 0 (0%)
| Residence | Street | 3 | 3 (100%) | 1 (33.3%) | 2 (66.7%)
|  | City | 5 | 4 (80%) | 4 (80%) | 3 (60%)
|  | County | 2 | 2 (100%) | 2 (100%) | 1 (50%)
|  | ZIP Code | 6 | 5 (83.3%) | 5 (83.3%) | 3 (50%)
Protected Classifications |  | Gender | 1 | 1 (100%) | 1 (100%) | 1 (100%)
|  | Date of Birth | 5 | 5 (100%) | 4 (80%) | 1 (20%)
Geolocation | Precise | GPS Coordinates | 13 | 10 (76.9%) | 10 (76.9%) | 10 (76.9%)
| Coarse | GPS Coordinates | 5 | 5 (100%) | 3 (60%) | 2 (40%)
|  | City | 15 | 14 (93.3%) | 8 (53.3%) | 11 (73.3%)
|  | County | 3 | 3 (100%) | 3 (100%) | 1 (33.3%)
|  | ZIP Code | 9 | 7 (77.8%) | 5 (55.6%) | 7 (77.8%)
Professional |  | Job | 2 | 1 (50%) | 2 (100%) | 1 (50%)
|  | Company | 3 | 3 (100%) | 3 (100%) | 0 (0%)
Education |  | University | 1 | 1 (100%) | 1 (100%) | 0 (0%)
Sensory Data |  | Sensor Readings | 22 | 22 (100%) | 1 (4.5%) | 22 (100%)

  * •

‘# Apps’ denotes the total number of apps that did not disclose the specific
PII out of a total of 80 apps that provided valid responses to VCRs.
Percentages denote the proportion out of the total number of apps that did not
disclose the specific PII.

In terms of the specific pieces of personal information, we observed the
collection, but not the disclosure, of device-specific identifiers, such as
the Android Advertising ID (AAID), by 51 apps, app-specific identifiers, such
as the Android ID, by 28 apps, coarse GPS coordinates (i.e., with a
granularity up to a certain neighborhood) by 4, ZIP code by 8, the name of the
city by 12 apps, precise GPS coordinates (i.e., that point to a specific
building) by 12, parts of postal address by 10, user’s phone number by 5,
information about a user’s contacts by 5 apps, and so on.

We examined the network transmission logs for the 8 apps developed by
companies that told us that they did not hold any data on us; only one
appeared to not actually collect any data. The remaining 7 collected data
across a range of CCPA-defined categories of personal information, in
particular, identifiers (7), geolocation (3), and sensory data (3). More
specifically, all 7 apps collected the AAID, 5 collected our IP address, and
one collected a device-identifying ID generated by the Branch.io SDK.
Furthermore, one of the apps collected, but did not disclose the collection of
precise GPS coordinates, and 3 apps collected coarse geolocation data that
pinpointed the specific city, neighborhood, or ZIP code, where the device was
physically located. Finally, 3 apps collected readings generated by the
device’s accelerometer, gyroscope, or magnetometer sensors.

Table 4 summarizes the undisclosed data collection that we observed across the
80 apps, for which we received a response, including information about the
usage of TLS encryption, as well as the number of apps that do not disclose
the categories of personal information shared with the first-party and third-
party domains. We note that our results provide a lower bound on the number of
pieces of collected-but-undisclosed personal information, as additional
personal information collected by the apps might not have been detected during
our analysis of the apps’ network traffic. We additionally present the top 20
third-party recipients of personal information, as well as the number of apps
that shared different categories of personal information with these entities
in Figure 1.

Figure 1. Top 20 Third-Party Data Recipients. Each number represents the
unique number of apps, from which the entity collected a specific category of
personal data. The first two columns display the number of unique apps sharing
any category of PII, and whether it was via TLS.

###  4.4. Privacy Policies

Finally, we analyzed the disclosures made in the privacy policies of tested
apps. For each of the 109 privacy policies containing CCPA-specific
information, multiple researchers from our team independently indicated which
categories of personal information were collected or disclosed by each
developer and to which category of recipients. Table 5 summarizes the number
of policies disclosing the collection and sharing of categories of personal
information, the categories of recipients, and the inter-rater reliability
scores.

Table 5. Categories Disclosed in Privacy Policies

Prompt | Category | Yes # | No # | $\alpha$
---|---|---|---|---
Does the privacy policy state that the app developer collects…. | Identifiers | 109 | 0 | —
Customer Records | 95 | 14 | 0.517
Protected Classifications | 63 | 46 | 0.663
Commercial Information | 78 | 31 | 0.596
Biometric Information | 12 | 97 | 0.714
| Network Activity
---
107 | 2 | 0.176
Geolocation Data | 97 | 12 | 0.616
Sensory Data | 63 | 46 | 0.373
Professional Information | 46 | 63 | 0.726
Education Information | 15 | 94 | 0.616
Inferences | 62 | 47 | 0.542
Does the privacy policy state that the app developer discloses or shares…. | Identifiers | 103 | 6 | $<0$
Customer Records | 81 | 28 | 0.183
Protected Classifications | 49 | 60 | 0.411
Commercial Information | 65 | 44 | 0.445
Biometric Information | 8 | 101 | 0.579
| Network Activity
---
98 | 11 | 0.099
Geolocation Data | 84 | 25 | 0.287
Sensory Data | 53 | 56 | 0.275
Professional Information | 29 | 80 | 0.625
Education Information | 15 | 94 | 0.605
Inferences | 58 | 51 | 0.434
Does the privacy policy state that the app developer shares personal information with… | Affiliates | 98 | 11 | 0.449
Advertising Networks | 97 | 12 | 0.356
Marketing | 86 | 23 | 0.517
Analytics | 101 | 8 | 0.275
Security and Fraud | 66 | 43 | 0.293
Payment Processors | 78 | 31 | 0.573
Customer Support | 55 | 54 | 0.596
Storage and Infrastructure | 59 | 50 | 0.637
Search Engines | 10 | 99 | 0.347
Social Media | 49 | 60 | 0.599
Order Fulfillment | 25 | 84 | 0.559
Law Enforcement | 106 | 3 | 0.234
Unspecified Partners | 78 | 31 | 0.042

  * •

Column ‘$\alpha$’ refers to Krippendorff’s alpha, a measure of inter-rater
reliability.

All 109 policies disclosed the collection of identifiers and only two did not
mention the collection of “Internet activity information,” which includes data
about app interactions. Additionally, 97 (89%) and 95 (87%) policies disclosed
the collection of geolocation data and customer records information,
respectively. The broad nature of these categories entails that most
developers collect and, frequently, share this information, particularly in
the context of mobile apps where technical identifiers, data from sensors, and
usage information can be used both to provide the required app functionality
and to track users. By the same token, users do not gain much by being
informed about the collection of these categories.

We also identified the categories of personal information that the developers
disclosed or sold, 888Cal. Civ. Code §1798.140(t)(1) broadly defines ‘selling’
as disclosing “a consumer’s personal information by the business to another
business or a third party for monetary or other valuable consideration,” i.e.,
even when no monetary exchange is involved. as well as the categories of
recipients of users’ personal information. Although the CCPA requires
companies to enumerate the recipients for each category of personal
information, in practice we found that only a small number of policies did so.
Therefore, we focused on locating the categories of recipients in the text of
policies irrespective of which personal information they received.

Table 6. Comparison of Observed and Disclosed Categories

Categories | Apps | Policies | VCRs
---|---|---|---
|  |  Disclosed |  Undisclosed |  Disclosed |  Undisclosed
Identifiers — Collection | 75 | 74 | 1 | 22 | 53
Identifiers — Sharing | 60 | 55 | 5 | 11 | 49
Customer Records — Collection | 59 | 56 | 3 | 12 | 47
Customer Records — Sharing | 16 | 13 | 3 | 2 | 14
Protected Classifications — Collection | 16 | 9 | 7 | 5 | 11
Protected Classifications — Sharing | 4 | 2 | 2 | 1 | 3
Geolocation Data — Collection | 38 | 36 | 2 | 6 | 32
Geolocation Data — Sharing | 23 | 20 | 3 | 3 | 20
Sensory Data — Collection | 22 | 15 | 7 | 0 | 22
Sensory Data — Sharing | 22 | 10 | 12 | 0 | 22
Professional Information — Collection | 12 | 5 | 7 | 3 | 9
Professional Information — Sharing | 1 | 0 | 1 | 0 | 1
Education Information — Collection | 8 | 2 | 6 | 1 | 7
Education Information — Sharing | 0 | 0 | 0 | 0 | 0
Affiliates or Subsidiaries | 3 | 3 | 0 | 0 | 3
Advertising Networks | 23 | 22 | 1 | 5 | 18
Marketing | 27 | 17 | 10 | 3 | 24
Analytics | 49 | 46 | 3 | 7 | 42
Security and Fraud | 3 | 3 | 0 | 0 | 3
Payment Processors | 2 | 2 | 0 | 0 | 2
Customer Support | 1 | 0 | 1 | 0 | 1
Storage and Infrastructure | 26 | 15 | 11 | 2 | 24
Search Engines | 5 | 0 | 5 | 0 | 5
Social Media | 35 | 15 | 20 | 1 | 34

  * •

‘Apps’ denotes the number of apps observed collecting or sharing a specific
category of PII, or the number of apps that transmitted some PII to a specific
third-party recipient, while ‘Disclosed’ indicates how many of these disclosed
that collection or sharing in a privacy policy or a VCR.

Unsurprisingly, we found that the most frequently collected categories of
personal information are also the most frequently shared. In particular, 103
(94%) policies disclosed the sharing of identifiers, 98 (90%) disclosed the
sharing of internet activity information, and 84 (77%) disclosed the sharing
of geolocation data. With respect to recipients, almost every privacy policy
(106 or 97%) stated that the company might share users’ personal information
with law enforcement, if legally compelled. We also observed analytics
providers (93% of policies), advertising networks (89%), and marketing
partners (79%) being disclosed as the stated recipients of personal
information from the apps’ users.

For many categories, we did not attain a significant level of inter-rater
reliability (Krippendorff’s $\alpha$ in Table 5). We attribute this result to
the broad nature of some categories. For instance, there is a significant
overlap between the ‘identifiers’ and ‘customer records’ categories.
Recipients of personal information also commonly fall into similar categories,
e.g., many companies that provide advertising also offer analytics and
marketing solutions. Finally, although some companies used the CCPA-defined
categories of personal information to describe their data collection and
sharing practices, others relied on their own categorizations, and the CCPA
does not define the categories of third-party recipients, further decreasing
the consistency between policies written by different developers.

We observed the highest inter-rater agreement regarding the collection of
professional or employment-related data (Krippendorff’s $\alpha$ = 0.726),
biometric data (0.714), and protected classifications (0.663). In general, a
Krippendorff’s alpha of .667 or higher is considered acceptable for drawing
tentative conclusions (Krippendorff, 2011).

#### Categories Comparison.

Finally, we compared the categories of personal information that we observed
being collected and the categories of recipients with the categories disclosed
by the developer in the VCR response and with the categories that we obtained
from analyzing the privacy policies. We present the results of this comparison
for the 80 apps that completed the VCR in Table 6. Compared to the VCR
responses, 25 (31%) privacy policies failed to fully inform us about all of
the categories of collected personal information, while only 17 (21%) did not
fully disclose the sharing of information to third parties.

##  5\. Discussion

Our results present several important implications for developers and policy
makers with respect to the process of submitting verifiable consumer requests
and ensuring accurate responses. We highlight the following areas for
improvement: determining CCPA applicability, the security of consumers’
personal information, and the usability, completeness, and accuracy of
developers’ responses.

###  5.1. Determining CCPA Applicability

Only 71% of selected apps included CCPA-specific disclosures in their privacy
policies. As a compromise between evaluating the compliance of popular apps
without burdening smaller developers that do not have to comply, we decided
only to submit VCRs to those who provided CCPA-specific information in their
privacy documents. However, this naturally limited the scope of our analysis
and also prompted us to consider how ordinary consumers could determine which
companies are covered by CCPA requirements.

We imagine that the only organizations that consumers could realistically
determine to conform to the CCPA’s definition of a “business” (see Section
2.1) are public companies that disclose revenues in earnings reports. However,
this severely limits the ability of consumers to determine whether a company
has to comply with the CCPA; even if everyone could easily read earnings
reports, fewer than 0.01% of companies in the U.S. are publicly traded (of
Economic Research, 2021). Companies with a large online presence can surpass
the data collection threshold if, for instance, they use cookies, other
tracking technologies, or even simply record technical information from users’
devices, such as IP addresses, but there is no way for consumers to know when
the threshold is met. This could be addressed by requiring all companies doing
business in California to state in their privacy policies whether they are
subject to the CCPA.

###  5.2. Authentication and Security

Our analysis also demonstrated that many app developers did not use any
identity verification mechanism beyond a proof of access to an email account;
other companies required copies of government-issued identity documents and
signed affidavits. Given different domains and company sizes, it is unlikely
that a one-size-fit-all authentication approach will work for all
organizations. However, we highlight several issues that we encountered and
propose solutions.

For apps that maintain user accounts, we suggest relying on existing
authentication mechanisms to submit requests and access the provided data. At
the very least, these companies should require a password to perform these
actions. Ideally, these companies would also require a second authentication
factor, such as a mobile push notification or a one-time password (OTP). App
developers should also notify users about VCR submissions using established
communication channels to help detect fraudulent requests.

Authentication is more difficult for developers that do not require the
creation of user accounts to access their apps. These companies should request
at least three (and possibly more) non-trivial pieces of user-specific
information to match against the data already held. In the case of mobile
apps, the developer could require the user to send the VCR via the app, such
that the request also contains device-specific information alongside the
requested user-specific information. However, developers should also provide
an option to submit VCRs via other means, as a user might have already
uninstalled the app or changed their device. If the company does not hold
sufficient information to verify the consumer to a reasonable degree, then
they should rightfully reject the request to avoid leaking consumers’ personal
information to unauthorized parties. Companies should also not request copies
of government-issued IDs for authentication, as most organizations would not
(and, ideally, should not) have access to unique ID numbers to match against;
information in photos, such as name or birthdate, can be easily digitally
altered.

Finally, once the developer successfully confirms the identity of the
consumer, they should take necessary precautions to secure access to and
transmission of consumer’s personal information. In addition to existing
authentication mechanisms and, ideally, two-factor authentication, developers
should employ TLS, use download links with a time expiration, and secure files
using a password set by the consumer beforehand. Although none of these
measures can fully prevent the leakage of personal information, they can
definitely increase the cost for attackers attempting to fraudulently gain
access to consumers’ sensitive information.

###  5.3. Usability, Completeness, and Accuracy

We also discovered that VCR responses from app developers noticeably varied in
their format and contents. For instance, although 97% of companies that
completed our requests provided specific pieces of personal information, that
proportion dropped to 35% for categories of third parties. Furthermore, only 7
companies provided a choice to receive the data either in a human-readable
(e.g., TXT) or a machine-readable format (e.g., JSON).

We believe that regulators should issue more guidance to businesses when it
comes to the logistics of providing personal information back to consumers.
Besides questions of authentication and security, regulators should provide
examples of categorizations that developers could use in responding to VCRs.
For instance, although the text of the CCPA mentions covered categories of
personal information, similar categories for third parties or sources of
collection are absent. Many businesses use CCPA-defined categories of personal
information in their policies and VCR responses and, thus, similar taxonomies
would be beneficial in other contexts. We believe that to achieve greater
transparency, the CCPA should also require companies to disclose names of
third parties with whom they share personal information, as opposed to only
requiring the categories to be disclosed.

With respect to the accuracy of responses containing specific pieces of
personal information, we discovered that developers would often collect but
not disclose identifiers, geolocation data, and sensory data. As is already
the case in newer versions of Android, developers should not be allowed to
collect persistent non-resettable identifiers from consumers’ phones, such as
hardware identifiers. Instead, developers and third-party libraries should
only gain access to dedicated, resettable identifiers, specifically, the
Android Advertising ID (AAID). Regulators should also remind developers that
device identifiers, even resettable ones, constitute personal information
under the CCPA and, therefore, have to be disclosed upon receipt of a
verifiable consumer request. Developers should also be reminded that the
collection of such identifiers increases their chance of becoming subject to
the CCPA once they reach the predefined data collection threshold. Providing
more examples to developers, especially in the context of mobile apps, could
help clarify what information and at which level of granularity constitutes
personal information under the CCPA.

Finally, the CCPA’s “right to know” encompasses two distinct privacy rights:
the right of access and the right to data portability. Although both rights
can provide access to personal information held by a business, they serve
different purposes. Whereas data provided under the right to data portability
should be easily imported or transmitted to another service, data provided
under the right of access should be comprehensible to the consumer to whom the
data pertains. As these two privacy rights are not differentiated under the
CCPA the same way they are, for instance, under the GDPR, businesses provide
responses mainly in the machine-readable formats that are easier to export,
such as JSON. However, such formats are unlikely to be easily usable by
ordinary consumers. We therefore argue that the CCPA could be enhanced by
differentiating between the two rights and by providing guidelines to
developers about the best practices and formats to use when responding to
requests under each of these rights.

##  6\. Limitations

We investigated the extent to which Android app developers comply with the
provisions of the CCPA that require them to disclose their data sharing
practices in privacy policies and in response to consumers’ access requests.
As our objective was to select apps that we reasonably inferred to fall under
the CCPA definition of a “business,” it is important to note that the
resulting sample of apps is not meant to be representative. Our results,
therefore, do not generalize to the entire population of Android apps and do
not necessarily provide insights about the data collection and sharing
behaviors of other apps.

As we previously explained in Section 3, we tested the apps and interacted
with developers using pseudonyms. We acknowledge that some companies may not
have automated systems to process CCPA-related requests, and therefore
processing our VCRs may have imposed costs on the employees responding to
requests. However, as in related studies (Ausloos and Dewitte, 2018; Herrmann
and Lindemann, 2016; Norris and L’Hoiry, 2017; Kröger et al., 2020; Urban et
al., 2019b; Wong and Henderson, 2019), we believe that our approach was
necessary to investigate the quality of the VCR responses under realistic
conditions and to mitigate research participation effects (McCambridge et al.,
2014). Furthermore, we believe that that business interests in this regard are
outweighed by the public interest in understanding the effectiveness of CCPA
rights and raising awareness around existing issues.

Finally, the developments in privacy regulation will necessitate further work
in understanding how changes in specific scopes and provisions translate into
differences in compliance of different businesses. In particular, most of the
provisions of the California Privacy Rights Act (CPRA) revising the CCPA will
become operative on January 1, 2023, with enforcement commencing on July 1,
2023. We believe that future work should continue examining the application of
and compliance with the new privacy regimes to guide the development of
further consumer data protection laws.

###### Acknowledgements.

This work was supported by the U.S. National Science Foundation (under grant
CNS-1817248), the National Security Agency (under contract H98230-18-D-0006),
the Center for Long-Term Cybersecurity (CLTC) at U.C. Berkeley, and by CITRIS
and the Banatao Institute at the University of California. We would like to
thank Brandie Nonnecke and Liam Webster for feedback, as well as Refjohürs
Lykkewe.

## References

  * (1)
  * Adhatarao et al. (2021) Supriya Adhatarao, Cédric Lauradoux, and Cristiana Santos. 2021.  Why IP-based Subject Access Requests Are Denied?  _arXiv preprint arXiv.2103.01019_ (2021), 15 pages.
  * Alizadeh et al. (2020) Fatemeh Alizadeh, Timo Jakobi, Alexander Boden, Gunnar Stevens, and Jens Boldt. 2020.  GDPR reality check-claiming and investigating personally identifiable data from companies. In _2020 IEEE European Symposium on Security and Privacy Workshops (EuroS &PW)_. IEEE, New York, NY, USA, 120–129.
  * Alomar and Egelman (2022) Noura Alomar and Serge Egelman. 2022.  Developers Say the Darnedest Things: Privacy Compliance Processes Followed by Developers of Child-Directed Apps.  _Proceedings on Privacy Enhancing Technologies (PoPETs)_ 2022, 4 (2022), 250–273.
  * Andow et al. (2019) Benjamin Andow, Samin Yaseer Mahmud, Wenyu Wang, Justin Whitaker, William Enck, Bradley Reaves, Kapil Singh, and Tao Xie. 2019.  PolicyLint: Investigating Internal Privacy Policy Contradictions on Google Play. In _28th USENIX security symposium (USENIX security 19)_. USENIX, Berkeley, CA, USA, 585–602.
  * Andow et al. (2020) Benjamin Andow, Samin Yaseer Mahmud, Justin Whitaker, William Enck, Bradley Reaves, Kapil Singh, and Serge Egelman. 2020\.  Actions Speak Louder than Words:Entity-Sensitive Privacy Policy and Data Flow Analysis with PoliCheck. In _29th USENIX Security Symposium (USENIX Security 20)_. USENIX, Berkeley, CA, USA, 985–1002.
  * Ausloos and Dewitte (2018) Jef Ausloos and Pierre Dewitte. 2018.  Shattering One-Way Mirrors. Data Subject Access Rights in Practice.  _Data Subject Access Rights in Practice (January 20, 2018). International Data Privacy Law_ 8, 1 (2018), 4–28.
  * Boniface et al. (2019) Coline Boniface, Imane Fouad, Nataliia Bielova, Cédric Lauradoux, and Cristiana Santos. 2019\.  Security analysis of subject access request procedures. In _Annual Privacy Forum_. Springer, Berlin, Germany, 182–209.
  * Bowyer et al. (2022) Alex Bowyer, Jack Holt, Josephine Go Jefferies, Rob Wilson, David Kirk, and Jan David Smeddinck. 2022\.  Human-GDPR Interaction: Practical Experiences of Accessing Personal Data. In _CHI Conference on Human Factors in Computing Systems_. ACM, New York, NY, USA, 1–19.
  * Bufalieri et al. (2020) Luca Bufalieri, Massimo La Morgia, Alessandro Mei, and Julinda Stefa. 2020.  GDPR: when the right to access personal data becomes a threat. In _2020 IEEE International Conference on Web Services (ICWS)_. IEEE, New York, NY, USA, 75–83.
  * Bui et al. (2021) Duc Bui, Kang G Shin, Jong-Min Choi, and Junbum Shin. 2021\.  Automated Extraction and Presentation of Data Practices in Privacy Policies.  _Proceedings on Privacy Enhancing Technologies (PoPETs)_ 2021, 2 (2021), 88–110.
  * Burgess (2020) Matt Burgess. 2020\.  _What is GDPR? The summary guide to GDPR compliance in the UK_.  Wired.  https://www.wired.co.uk/article/what-is-gdpr-uk-eu-legislation-compliance-summary-fines-2018.
  * Cagnazzo et al. (2019) Matteo Cagnazzo, Thorsten Holz, and Norbert Pohlmann. 2019\.  GDPiRated–stealing personal information on-and offline. In _European Symposium on Research in Computer Security_. Springer, Berlin, Germany, 367–386.
  * California Consumer Privacy Act (2018) (CCPA) California Consumer Privacy Act (CCPA). 2018\.  California Civil Code §1798.100 et seq..
  * Commission (2021) U.S. Federal Trade Commission. 2021\.  Flo Health, Inc.  https://www.ftc.gov/legal-library/browse/cases-proceedings/192-3133-flo-health-inc.
  * Cormack (2016) Andrew Cormack. 2016\.  Is the Subject Access Right Now Too Great a Threat to Privacy.  _Eur. Data Prot. L. Rev._ 2 (2016), 15.
  * Court (2022) California Superior Court. 2022\.  People of the State of California v. Sephora USA, Inc.,.  Case No. CGC-22-601380.
  * DataGuidance and of Privacy FOrum (2018) DataGuidance and Future of Privacy FOrum. 2018.  Comparing Privacy Laws: GDPR v. CCPA.  https://fpf.org/wp-content/uploads/2018/11/GDPR_CCPA_Comparison-Guide.pdf.
  * Di Martino et al. (2022) Mariano Di Martino, Isaac Meers, Peter Quax, Ken Andries, and Wim Lamotte. 2022.  Revisiting identification issues in GDPR ‘Right Of Access’ policies: a technical and longitudinal analysis.  _Proceedings on Privacy Enhancing Technologies (PoPETs)_ 2022, 2 (2022), 95–113.
  * Di Martino et al. (2019) Mariano Di Martino, Pieter Robyns, Winnie Weyts, Peter Quax, Wim Lamotte, and Ken Andries. 2019\.  Personal Information Leakage by Abusing the GDPR’Right of Access’. In _Fifteenth Symposium on Usable Privacy and Security (SOUPS 2019)_. USENIX, Berkeley, CA, USA, 371–385.
  * Fan et al. (2020) Ming Fan, Le Yu, Sen Chen, Hao Zhou, Xiapu Luo, Shuyue Li, Yang Liu, Jun Liu, and Ting Liu. 2020.  An empirical evaluation of GDPR compliance violations in Android mHealth apps. In _2020 IEEE 31st international symposium on software reliability engineering (ISSRE)_. IEEE, New York, NY, USA, 253–264.
  * Feal et al. (2020) Álvaro Feal, Paolo Calciati, Narseo Vallina-Rodriguez, Carmela Troncoso, Alessandra Gorla, et al. 2020\.  Angel or devil? a privacy study of mobile parental control apps.  _Proceedings on Privacy Enhancing Technologies (PoPETs)_ 2020, 2 (2020), 314–335.
  * for Economic Co-operation and Development (1980) Organization for Economic Co-operation and Development. 1980.  OECD guidelines on the protection of privacy and transborder flows of personal data.
  * Frida (2022) Frida. 2022.  https://frida.re/.
  * Habib et al. (2020) Hana Habib, Sarah Pearman, Jiamin Wang, Yixin Zou, Alessandro Acquisti, Lorrie Faith Cranor, Norman Sadeh, and Florian Schaub. 2020\.  ” It’s a scavenger hunt”: Usability of Websites’ Opt-Out and Data Deletion Choices. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_. ACM, New York, NY, USA, 1–12.
  * Han et al. (2020) Catherine Han, Irwin Reyes, Álvaro Feal, Joel Reardon, Primal Wijesekera, Narseo Vallina-Rodriguez, Amit Elazar, Kenneth A Bamberger, and Serge Egelman. 2020\.  The price is (not) right: Comparing privacy in free and paid apps.  _Proceedings on Privacy Enhancing Technologies (PoPETs)_ 2020, 3 (2020), 222–242.
  * Harkous et al. (2018) Hamza Harkous, Kassem Fawaz, Rémi Lebret, Florian Schaub, Kang G Shin, and Karl Aberer. 2018\.  Polisis: Automated analysis and presentation of privacy policies using deep learning. In _27th USENIX Security Symposium (USENIX Security 18)_. USENIX, Berkeley, CA, USA, 531–548.
  * Herrmann and Lindemann (2016) Dominik Herrmann and Jens Lindemann. 2016.  Obtaining personal data and asking for erasure: Do app vendors and website owners honour your privacy rights?  _arXiv preprint arXiv:1602.01804_ (2016), 1–15.
  * Jehl and Friel (2018) Laura Jehl and Alan Friel. 2018.  CCPA and GDPR Comparison Chart.  https://www.bakerlaw.com/webfiles/Privacy/2018/Articles/CCPA-GDPR-Chart.pdf.
  * Jia et al. (2019) Qiwei Jia, Lu Zhou, Huaxin Li, Ruoxu Yang, Suguo Du, and Haojin Zhu. 2019\.  Who leaks my privacy: Towards automatic and association detection with gdpr compliance. In _International Conference on Wireless Algorithms, Systems, and Applications_. Springer, Berlin, Germany, 137–148.
  * joke2k (2023) joke2k. 2023.  Faker.  https://pypi.org/project/Faker/.  Accessed: March 15, 2023.
  * Jordan et al. (2021) Scott Jordan, Yoshimichi Nakatsuka, Ercan Ozturk, Andrew Paverd, and Gene Tsudik. 2021.  Viceroy: Gdpr-/ccpa-compliant enforcement of verifiable accountless consumer requests.  _arXiv preprint arXiv:2105.06942_ (2021), 1–17.
  * Kemp (2020) Katharine Kemp. 2020\.  Concealed data practices and competition law: why privacy matters.  _European Competition Journal_ 16, 2-3 (2020), 628–672.
  * Krippendorff (2011) Klaus Krippendorff. 2011\.  Computing Krippendorff’s Alpha-Reliability.  https://repository.upenn.edu/asc_papers/43
  * Krippendorff (2018) Klaus Krippendorff. 2018\.  _Content analysis: An introduction to its methodology_.  Sage publications, Thousand Oaks, CA, USA.
  * Kröger et al. (2020) Jacob Leon Kröger, Jens Lindemann, and Dominik Herrmann. 2020\.  How do app vendors respond to subject access requests? A longitudinal privacy study on iOS and Android Apps. In _Proceedings of the 15th International Conference on Availability, Reliability and Security_. ACM, New York, NY, USA, 1–10.
  * Kuebler-Wachendorff et al. (2021) Sophie Kuebler-Wachendorff, Robert Luzsa, Johann Kranz, Stefan Mager, Emmanuel Syrmoudis, Susanne Mayr, and Jens Grossklags. 2021\.  The Right to Data Portability: conception, status quo, and future directions.  _Informatik Spektrum_ 44, 4 (2021), 264–272.
  * Linden et al. (2018) Thomas Linden, Rishabh Khandelwal, Hamza Harkous, and Kassem Fawaz. 2018.  The privacy policy landscape after the GDPR.  _arXiv preprint arXiv:1809.08396_ (2018), 1–18.
  * Mahoney (2020) Maureen Mahoney. 2020\.  _California Consumer Privacy Act: Are consumers’ digital rights protected_.  Technical Report. Consumer Reports Digital Lab.
  * Mayer (2021) Jonathan Mayer. 2021\.  Princeton-Radboud Study on Privacy Law Implementation.  https://privacystudy.cs.princeton.edu/.
  * McCambridge et al. (2014) Jim McCambridge, John Witton, and Diana R Elbourne. 2014\.  Systematic review of the Hawthorne effect: new concepts are needed to study research participation effects.  _Journal of clinical epidemiology_ 67, 3 (2014), 267–277.
  * Nguyen et al. (2021) Trung Tin Nguyen, Michael Backes, Ninja Marnau, and Ben Stock. 2021\.  Share First, Ask Later (or Never?) Studying Violations of GDPR’s Explicit Consent in Android Apps. In _30th USENIX Security Symposium (USENIX Security 21)_. USENIX, Berkeley, CA, USA, 3667–3684.
  * Norris and L’Hoiry (2017) Clive Norris and Xavier L’Hoiry. 2017.  Exercising Citizen Rights Under Surveillance Regimes in Europe–Meta-analysis of a Ten Country Study.  In _The Unaccountable State of Surveillance_. Springer, Berlin, Germany, 405–455.
  * of Economic Research (2021) Ntional Bureau of Economic Research. 2021\.  Why Are There So Few Public Companies in the U.S.?  https://www.nber.org/digest/sep15/why-are-there-so-few-public-companies-us.  Accessed: 2022-08-31.
  * of Human Research Protections (2022) Office of Human Research Protections. 2022\.  What is Human Subjects Research?  https://www.hhs.gov/ohrp/sites/default/files/OHRP-HHS-Learning-Module-Lesson2.pdf.
  * of the European Parliament (2016) Regulation (EU) 2016/679 of the European Parliament. 2016.  Retrieved 2022-01-28 from http://data.europa.eu/eli/reg/2016/679/oj/eng
  * Okoyomon et al. (2019) Ehimare Okoyomon, Nikita Samarin, Primal Wijesekera, Amit Elazari Bar On, Narseo Vallina-Rodriguez, Irwin Reyes, Álvaro Feal, Serge Egelman, et al. 2019\.  On the ridiculousness of notice and consent: Contradictions in app privacy policies. In _Workshop on Technology and Consumer Protection (ConPro 2019), in conjunction with the 39th IEEE Symposium on Security and Privacy_. IEEE, New York, NY, USA.
  * OWASP (2022) OWASP. 2022.  Zed Attack Proxy (ZAP).  https://www.zaproxy.org/.
  * Pavur and Knerr (2019) James Pavur and Casey Knerr. 2019.  Gdparrrrr: Using privacy laws to steal identities.  _arXiv preprint arXiv:1912.00731_ (2019), 1–10.
  * Perry and LaBruyere (2023) Benjamin William Perry and Rachel M. LaBruyere. 2023.  Who Has My Data? EU Court Rules GDPR Requires Disclosure of Data Recipient Identities, Not Just Categories, in Response to Data Subject Access Requests.  Lexology.  https://www.lexology.com/library/detail.aspx?g=5d58c7be-ca44-4249-b11b-1a68144dac24.
  * Random Lists (2023) Random Lists. 2023\.  Random Lists.  https://www.randomlists.com/.  Accessed: March 15, 2023.
  * Reardon et al. (2019) Joel Reardon, Álvaro Feal, Primal Wijesekera, Amit Elazari Bar On, Narseo Vallina-Rodriguez, and Serge Egelman. 2019.  50 ways to leak your data: An exploration of apps’ circumvention of the android permissions system. In _28th USENIX security symposium (USENIX security 19)_. USENIX, Berkeley, CA, USA, 603–620.
  * Ren et al. (2018) Jingjing Ren, Martina Lindorfer, Daniel J Dubois, Ashwin Rao, David Choffnes, and Narseo Vallina-Rodriguez. 2018.  A longitudinal study of pii leaks across android app versions. In _Proceedings of the 25th Network and Distributed System Security Symposium (NDSS 2018)_. Internet Society, Reston, VA, USA.
  * Reyes et al. (2018) Irwin Reyes, Primal Wijesekera, Joel Reardon, Amit Elazari Bar On, Abbas Razaghpanah, Narseo Vallina-Rodriguez, Serge Egelman, et al. 2018\.  “Won’t somebody think of the children?” examining COPPA compliance at scale.  _Proceedings on Privacy Enhancing Technologies (PoPETs)_ 2018, 3 (2018), 63–83.
  * Sanfilippo et al. (2020) Madelyn R. Sanfilippo, Yan Shvartzshnaider, Irwin Reyes, Helen Nissenbaum, and Serge Egelman. 2020\.  Disaster Privacy/Privacy Disaster.  _Journal of the Association for Information Science and Technology_ 71, 9 (2020), 1002–1014.  https://doi.org/10.1002/asi.24353 arXiv:https://asistdl.onlinelibrary.wiley.com/doi/pdf/10.1002/asi.24353
  * Slavin et al. (2016) Rocky Slavin, Xiaoyin Wang, Mitra Bokaei Hosseini, James Hester, Ram Krishnan, Jaspreet Bhatia, Travis D Breaux, and Jianwei Niu. 2016\.  Toward a framework for detecting privacy policy violations in android application code. In _Proceedings of the 38th International Conference on Software Engineering_. ACM, New York, NY, USA, 25–36.
  * statcounter (2021) statcounter. 2021\.  Mobile Android Version Market Share United States Of America.  https://gs.statcounter.com/android-version-market-share/mobile/united-states-of-america/2021.
  * Tolsdorf et al. (2021) Jan Tolsdorf, Michael Fischer, and Luigi Lo Iacono. 2021\.  A case study on the implementation of the right of access in privacy dashboards. In _Annual Privacy Forum_. Springer, Berlin, Germany, 23–46.
  * Trimananda et al. (2022) Rahmadi Trimananda, Hieu Le, Hao Cui, Janice Tran Ho, Anastasia Shuba, and Athina Markopoulou. 2022.  $\\{$OVRseen$\\}$: Auditing Network Traffic and Privacy Policies in Oculus $\\{$VR$\\}$. In _31st USENIX security symposium (USENIX security 22)_. USENIX, Berkeley, CA, USA, 3789–3806.
  * Trust (2021) One Trust. 2021\.  One Trust.  https://www.onetrust.com/.  Accessed: 2022-08-31.
  * Urban et al. (2019a) Tobias Urban, Martin Degeling, Thorsten Holz, and Norbert Pohlmann. 2019a.  ”Your hashed IP address: Ubuntu.” perspectives on transparency tools for online advertising. In _Proceedings of the 35th Annual Computer Security Applications Conference_. ACM, New York, NY, USA, 702–717.
  * Urban et al. (2018) Tobias Urban, Dennis Tatang, Martin Degeling, Thorsten Holz, and Norbert Pohlmann. 2018.  The unwanted sharing economy: An analysis of cookie syncing and user transparency under GDPR.  _arXiv preprint arXiv:1811.08660_ (2018), 1–26.
  * Urban et al. (2019b) Tobias Urban, Dennis Tatang, Martin Degeling, Thorsten Holz, and Norbert Pohlmann. 2019b.  A study on subject data access in online advertising after the GDPR.  In _Data Privacy Management, Cryptocurrencies and Blockchain Technology_. Springer, Berlin, Germany, 61–79.
  * Van Nortwick and Wilson (2022) Maggie Van Nortwick and Christo Wilson. 2022.  Setting the Bar Low: Are Websites Complying With the Minimum Requirements of the CCPA?  _Proceedings on Privacy Enhancing Technologies (PoPETs)_ 2022, 1 (2022), 608–628.
  * Veys et al. (2021) Sophie Veys, Daniel Serrano, Madison Stamos, Margot Herman, Nathan Reitinger, Michelle L Mazurek, and Blase Ur. 2021.  Pursuing Usable and Useful Data Downloads Under GDPR/CCPA Access Rights via Co-Design. In _Seventeenth Symposium on Usable Privacy and Security (SOUPS 2021)_. USENIX, Berkeley, CA, USA, 217–242.
  * Wang et al. (2018) Xiaoyin Wang, Xue Qin, Mitra Bokaei Hosseini, Rocky Slavin, Travis D Breaux, and Jianwei Niu. 2018\.  Guileak: Tracing privacy policy claims on user input data for android applications. In _Proceedings of the 40th International Conference on Software Engineering_. ACM, New York, NY, USA, 37–47.
  * Wei et al. (2020) Miranda Wei, Madison Stamos, Sophie Veys, Nathan Reitinger, Justin Goodman, Margot Herman, Dorota Filipczuk, Ben Weinshel, Michelle L Mazurek, and Blase Ur. 2020\.  What Twitter knows: Characterizing ad targeting practices, user perceptions, and ad explanations through users’ own Twitter data. In _29th USENIX Security Symposium (USENIX Security 20)_. USENIX, Berkeley, CA, USA, 145–162.
  * Wong and Henderson (2019) Janis Wong and Tristan Henderson. 2019.  The right to data portability in practice: exploring the implications of the technologically neutral GDPR.  _International Data Privacy Law_ 9, 3 (2019), 173–191.
  * Zang et al. (2015) Jinyan Zang, Krysta Dummit, James Graves, Paul Lisker, and Latanya Sweeney. 2015.  Who knows what about me? A survey of behind the scenes personal data sharing to third parties by mobile apps.  _Technology Science_ 30 (2015), 1–53.
  * Zimmeck et al. (2021) Sebastian Zimmeck, Rafael Goldstein, and David Baraka. 2021\.  PrivacyFlash Pro: Automating Privacy Policy Generation for Mobile Apps.. In _NDSS_. Internet Society, Reston, VA, USA, 18 pages.
  * Zimmeck et al. (2019) Sebastian Zimmeck, Peter Story, Daniel Smullen, Abhilasha Ravichander, Ziqi Wang, Joel R Reidenberg, N Cameron Russell, and Norman Sadeh. 2019.  MAPS: Scaling privacy compliance analysis to a million apps.  _Proceedings on Privacy Enhancing Technologies (PoPETs)_ 2019, 3 (2019), 66–86.
  * Zimmeck et al. (2017) Sebastian Zimmeck, Ziqi Wang, Lieyong Zou, Roger Iyengar, Bin Liu, Florian Schaub, Shomir Wilson, Norman M Sadeh, Steven M Bellovin, and Joel R Reidenberg. 2017.  Automated Analysis of Privacy Requirements for Mobile Apps.. In _NDSS_. Internet Society, Reston, VA, USA, 15 pages.

##  Appendix A VCR Email Templates

This appendix contains the email templates that we used to submit verifiable
consumer requests to app developers, as well as the conditions under which we
sent it. Note that we cited the provision Cal. Civil Code 1798.140 in the
template emails to direct the developers to the list of categories predefined
by the CCPA to facilitate their response and to improve the consistency of
categorization across different companies.

###  A.1. Initial Request

Email template used to initiate the VCR.

Dear Privacy Compliance Officer,

My name is [name]. I live in California and I am exercising my data access
rights under the California Consumer Privacy Act (CCPA) to obtain a copy of
the categories and the specific pieces of personal information that [company]
has collected about me.

I’m requesting a copy of any and all of the records you have pertaining to me
including (but not limited to):

  1. (1)

Specific pieces of personal information and any persistent identifiers that
you have collected about me including all information or content provided or
posted by me, any information you have collected about me, or any personal
information you have obtained or acquired about me from a third party business
or service provider;

  2. (2)

Categories of personal information you have collected about me pursuant to the
enumerated list of categories in Cal. Civil Code 1798.140(o);

  3. (3)

Categories of sources from which my personal information is collected;

  4. (4)

Categories of personal information that you have sold or disclosed for a
business purpose about me by each category of personal information enumerated
in Cal. Civil Code 1798.140(o);

  5. (5)

Third parties to whom my personal information was sold or disclosed for a
business purpose; and

  6. (6)

The business or commercial purpose for collecting or selling my personal
information.

I expect a confirmation of receipt within 10 business days and information
about how [company] will process my request, sent to this email address.
Please let me know if you need any more information from me as soon as
possible.

If you believe that you are not subject to the CCPA, please reply back as soon
as possible and let me know why you believe the CCPA does not apply in this
case.

Sincerely,

[Name]

###  A.2. Unable to Perform Request

Company has directed us to use an alternative method to submit VCR that does
not provide access to the full records.

Dear [Name of Privacy Compliance Officer],

Thank you for your reply. Unfortunately, the [alternative request method] that
you have directed me to use to submit my request does not allow me to fully
exercise my data access rights under the California Consumer Privacy Act.

Specifically, the [alternative request method] does not allow me to request a
copy of the following records you have pertaining to me:

(Select and include the appropriate ones in the email)

  1. (1)

Specific pieces of personal information and any persistent identifiers that
you have collected about me including all information or content provided or
posted by me, any information you have collected about me, or any personal
information you have obtained or acquired about me from a third party business
or service provider;

  2. (2)

Categories of personal information you have collected about me pursuant to the
enumerated list of categories in Cal. Civil Code 1798.140(o);

  3. (3)

Categories of sources from which my personal information is collected;

  4. (4)

Categories of personal information that you have sold or disclosed for a
business purpose about me by each category of personal information enumerated
in Cal. Civil Code 1798.140(o);

  5. (5)

Third parties to whom my personal information was sold or disclosed for a
business purpose; and

  6. (6)

The business or commercial purpose for collecting or selling my personal
information.

Please let me know how I should proceed as soon as possible.

Sincerely,

[Name]

###  A.3. Missing Information Request

Company responded to our VCR without providing all of the requested
information.

Dear [Name of Privacy Compliance Officer],

Thank you for your reply. Unfortunately, the copy of the records that I have
received does not contain all of the requested information. Specifically, I
have not received a copy of the following records you have pertaining to me:

(Select and include the appropriate ones in the email)

  1. (1)

Specific pieces of personal information and any persistent identifiers that
you have collected about me including all information or content provided or
posted by me, any information you have collected about me, or any personal
information you have obtained or acquired about me from a third party business
or service provider;

  2. (2)

Categories of personal information you have collected about me pursuant to the
enumerated list of categories in Cal. Civil Code 1798.140(o);

  3. (3)

Categories of sources from which my personal information is collected;

  4. (4)

Categories of personal information that you have sold or disclosed for a
business purpose about me by each category of personal information enumerated
in Cal. Civil Code 1798.140(o);

  5. (5)

Third parties to whom my personal information was sold or disclosed for a
business purpose; and

  6. (6)

The business or commercial purpose for collecting or selling my personal
information.

Please let me know how I should proceed as soon as possible.

Sincerely,

[Name]

###  A.4. Account Holder Verification Request

We created an account with the app and the developer required us to furnish
documentation to verify our identity that we could not provide.

Dear [Name of Privacy Compliance Officer],

Thank you for your reply. Unfortunately, I prefer not to provide the
information that you have requested to verify my identity, as I believe it to
be invasive and beyond the requirements of the CCPA.

As an account holder with [company], I would prefer verifying my identity
using existing authentication practices for my account per CCR § 999.324(a).
For your convenience, the [email address OR username] associated with my
account is [email address OR username].

Please let me know if you need any more information from me as soon as
possible.

Sincerely,

[Name]

###  A.5. Account Non-Holder Verification Request

We did not create an account with the app and the developer required us to
furnish documentation to verify our identity that we could not provide.

Dear [Name of Privacy Compliance Officer],

Thank you for your reply. Unfortunately, I prefer not to provide the
information that you have requested to verify my identity, as I believe it to
be invasive and beyond the requirements of the CCPA.

Instead, I would prefer verifying my identity by matching the following three
pieces of personally identifiable information that I have previously provided
to [company] per CCR § 999.325(b) and (c):

(Select and include the appropriate ones in the email)

  1. (1)

PII1 Type: PII1 Value

  2. (2)

PII2 Type: PII2 Value

  3. (3)

PII3 Type: PII3 Value

Please let me know if you need any more information from me as soon as
possible.

Sincerely,

[Name]

###  A.6. First Follow-Up

Company did not respond to our initial request in 10 business days.

Dear Privacy Compliance Officer,

My name is [name] and I am following up on a request I made on [date] to
access the personal information that [company] has collected about me. I was
expecting to receive a confirmation of receipt and information about how
[company] would process my request within 10 business days per 11 CCR §
999.313(a). For your convenience, my original request is as follows:

I’m requesting a copy of any and all of the records you have pertaining to me
including (but not limited to):

  1. (1)

Specific pieces of personal information and any persistent identifiers that
you have collected about me including all information or content provided or
posted by me, any information you have collected about me, or any personal
information you have obtained or acquired about me from a third party business
or service provider;

  2. (2)

Categories of personal information you have collected about me pursuant to the
enumerated list of categories in Cal. Civil Code 1798.140(o);

  3. (3)

Categories of sources from which my personal information is collected;

  4. (4)

Categories of personal information that you have sold or disclosed for a
business purpose about me by each category of personal information enumerated
in Cal. Civil Code 1798.140(o);

  5. (5)

Third parties to whom my personal information was sold or disclosed for a
business purpose; and

  6. (6)

The business or commercial purpose for collecting or selling my personal
information.

I expect a reply to this email address as soon as possible. If you believe
that you are not subject to the California Consumer Privacy Act (CCPA), please
reply back as soon as possible and let me know why you believe the CCPA does
not apply in this case.

Sincerely,

[Name]

###  A.7. Second Follow-Up

Company did not respond to our first follow-up email in 10 business days.

Dear Privacy Compliance Officer,

My name is [name] and I am following up on a request I originally made on
[date] to access the personal information that [company] has collected about
me. I have previously followed up about my request on [date], but I have not
heard back from you. I was expecting to receive a confirmation of receipt and
information about how [company] would process my request within 10 business
days per 11 CCR § 999.313(a). My original request is as follows:
I’m requesting a copy of any and all of the records you have pertaining to me
including (but not limited to):

  1. (1)

Specific pieces of personal information and any persistent identifiers that
you have collected about me including all information or content provided or
posted by me, any information you have collected about me, or any personal
information you have obtained or acquired about me from a third party business
or service provider;

  2. (2)

Categories of personal information you have collected about me pursuant to the
enumerated list of categories in Cal. Civil Code 1798.140(o);

  3. (3)

Categories of sources from which my personal information is collected;

  4. (4)

Categories of personal information that you have sold or disclosed for a
business purpose about me by each category of personal information enumerated
in Cal. Civil Code 1798.140(o);

  5. (5)

Third parties to whom my personal information was sold or disclosed for a
business purpose; and

  6. (6)

The business or commercial purpose for collecting or selling my personal
information.

I expect a reply to this email address as soon as possible. If you believe
that you are not subject to the California Consumer Privacy Act (CCPA), please
reply back as soon as possible and let me know why you believe the CCPA does
not apply in this case.

Sincerely,

[Name]

##  Appendix B Codebook

Tables 7 and 8 include the codebook that we used to perform a qualitative
analysis of disclosures in privacy policies. We use the categories of personal
information defined in Cal. Civil Code 1798.140 to represent the codes for the
collection and sharing in Table 7. Table 8 contains our codes for the
categories of third parties.

For each privacy policy, coders saw the following prompts:

  * •

Does this app developer include disclosures that reference the CCPA, either as
part of the general privacy policy or as a standalone document?

  * •

Does the privacy policy state that the app developer collects [PII Code]?

  * •

Does the privacy policy state that the app developer discloses or shares [PII
Code]?

  * •

Does the privacy policy state that the app developer shares personal
information with [Third Party Code]?

Table 7. Personally-identifiable Information (PII) Codes PII Code | Description
---|---
Identifiers |  | Real name, alias, postal address, unique perso-
---
nal identifier, online identifier, IP address, email
address, account name, social security number,
driver’s license number, passport number, or
other similar identifiers.
| Customer
---
Records
| Name, signature, social security number, physical
---
characteristics or description, address, telephone
number, passport number, driver’s license or state
identification card number, insurance policy num-
ber, education, employment, employment history,
bank account number, credit card number, debit
card number, or any other financial information,
medical or health insurance information.
| Characteristics of
---
Protected
Classifications
under California
or Federal Law
| Age, race, color, ancestry, national origin, citizen-
---
ship, religion or creed, marital status, medical
condition, physical or mental disability, sex, gen-
der, gender identity, gender expression, pregnan-
cy or childbirth and related medical conditions,
sexual orientation, veteran or military status,
genetic information (including familial genetic
information).
| Commercial
---
Information
| Records of personal property, products or servi-
---
ces purchased, obtained, or considered, or other
purchasing or consuming histories or tendencies.
| Biometric
---
Information
| Genetic, physiological, behavioral, and biological
---
characteristics, or activity patterns used to extract
a template or other identifier or identifying infor-
mation, such as, fingerprints, faceprints, and
voiceprints, iris or retina scans, keystroke, gait,
or other physical patterns, sleep, health, or exer-
cise data.
| Network
---
Activity
| Browsing history, search history, or information
---
regarding a consumer’s interaction with a website,
application, or advertisement.
| Geolocation
---
Data
| Information such as physical location or move-
---
ments.
| Sensory
---
Data
| Audio, electronic, visual, thermal, olfactory, or
---
similar information.
| Professional
---
Information
| Information such as current or past job history or
---
performance evaluations.
| Education
---
Information
| Education records directly related to a student
---
maintained by an educational institution or party
acting on its behalf, such as grades, transcripts,
class lists, student schedules, student identifica-
tion codes, student financial information, or stu-
dent disciplinary records.
Inferences |  | Consumer’s preferences, characteristics, psycho-
---
logical trends, predispositions, behavior, attitudes,
intelligence, abilities, or aptitudes.
Table 8. Third-Party Data Recipients Codes 3rd Party Code | Description
---|---
| Affiliated
---
Companies
| Companies related to the app developer through
---
ownership, such as when the app developer holds
a stake in the company (e.g., a subsidiary) or
when another third party controls both the com-
pany and the app developer.
| Advertising
---
Networks
| Connect advertisers to websites or apps (the
---
“publishers”) that want to host advertisements.
| Marketing
---
Providers
| Offer products, services, or other promotions to
---
the app’s users, for instance, by calling, texting or
emailing them with marketing messages.
| Analytics
---
Providers
| Capture data about the app’s audience in order to
---
identify unique users, track their interactions, and
record their behavior for the purpose of improving
the app, informing company strategy, or general
research.
| Security and
---
Fraud
| Provide tools, such as identity verification and
---
fraud detection, to prevent fraudulent activity,
improve app security, enforce terms of service,
and protect users and property.
| Payment
---
Processors
| Enable merchants to sell products and accept in-
---
app card payments.
| Customer
---
Support
| Provide tools to collect, organize, respond to, and
---
report on customer support requests tounder-
stand user needs, provide assistance, and
streamline communication.
| Storage and
---
Infrastructure
| Provide services, such as data hosting, cloud
---
storage, load balancing and other infrastructure
to optimize content delivery and performance.
| Search
---
Engines
| Collect, organize and enable the search for
---
content online, including information generated
by users interacting with the app or other users.
| Social Media
---
Platforms
| Provide technologies and means of communi-
---
cation, through which users create and share
information and ideas in online communities.
| Order
---
Fulfillment
Process orders and deliver products to customers.
| Law
---
Enforcement
| Sharing to comply with a legal obligation or a
---
request from regulators, courts, law enforcement,
and other governmental agencies.
| Unspecified
---
Partners
| Sharing with unspecified partners and
---
service providers.

##  Appendix C Data Taxonomy

Table 9 enumerates the 7 categories of personal information defined in the
CCPA relevant to this work, our subcategories, as well as the types and values
of personal information that we have predefined for each test device.

We generated pseudonymous data for User Identifiers, Customer Records,
Protected Classifications, Professional and Education Information using
publicly-available random value generators, such as those found on the Random
Lists 999https://www.randomlists.com/ website and the Faker
101010https://pypi.org/project/Faker/0.7.4/ Python package. We obtained other
types of personal information, including Device Identifiers and Geolocation
Data, directly from our test devices.

Table 9. Data Taxonomy

CCPA Category | Subcategory | Description | PII Types | Example Values
---|---|---|---|---
Identifiers | User | Identifiers set by the user | Usernames, email address, website | schneider90christopher19
| Network |  | Identifiers unique to
---
user’s network
IP Address, router MAC and SSID | 135.***.***.79, 48:**:**:**:**:06
| Device |  | Identifiers unique
---
to user’s device
| Android advertising ID (AAID),
---
hardware IDs, IMEI, IMSI, SIM ID,
Wi-Fi MAC, GSFID
| 97PAY11GN2, 359677097304580,
---
58:CB:52:8B:C8:66,
03140e43-9bb7-[…]
| App |  | Identifiers unique to
---
a single app
| Android ID, app fingerprint ID,
---
identity ID
| 7892f8834ddbf2df
---
1039977256339324001
| Customer
---
Records
Customer |  | Information about
---
the user
Name, phone number, height, weight |  | Christopher Schneider,
---
323-448-****
| Contacts |  | Information about
---
user’s contacts
Contact name, contact phone number | Scott Pratt, 415-200-****
| Residence |  | Information about
---
user’s general address
of residence
Street, city, county, ZIP Code | 957 Green Causeway, Los Angeles
| Protected
---
Classifications
— |  | Information protected
---
under the California and
U.S. federal laws
Gender, date of birth, age | Male, 20-May-1990
| Geolocation
---
Data
Precise |  | Locates a specific
---
building
| Precise longitude/latitude coordinates,
---
street name
*****
| Coarse |  | Does not locate a
---
specific building
| Coarse longitude/latitude coordinates,
---
city, county, ZIP Code
*****
Sensory Data | — |  | Audio, electronic, visual,
---
thermal, olfactory, or
similar information
| Accelerometer, gyroscope,
---
magnetometer readings
AK0991X, BMI160
| Professional
---
Information
— |  | Current or past job history or
---
performance evaluations
Job, company |  | Clinical Psychologist,
---
Williams and Davis
| Education
---
Information
— |  | Education records directly
---
related to a student
College | Villanova University

  * •

Some of the values have been redacted to preserve the privacy of researchers
to whom the data pertains.