# Toward the Cure of Privacy Policy Reading Phobia: Automated Generation of
Privacy Nutrition Labels From Privacy Policies

Shidong Pan11footnotemark: 122footnotemark: 2, Thong Hoang22footnotemark: 2,
Dawen Zhang11footnotemark: 122footnotemark: 2, Zhenchang Xing11footnotemark:
122footnotemark: 2, Xiwei Xu22footnotemark: 2, Qinghua Lu22footnotemark: 2,
and Mark Staples22footnotemark: 2 11footnotemark: 1School of Computing,
Australian National UniversityCanberra, Australia Shidong.Pan,
<EMAIL_ADDRESS>22footnotemark: 2Software Systems Research Group,
CSIRO’s Data61, AustraliaSydney, Australia James.Hoang, Dawen.Zhang, Xiwei.Xu,
Qinghua.Lu<EMAIL_ADDRESS>
###### Abstract.

Software applications have become an omnipresent part of modern society. The
consequent privacy policies of these applications play a significant role in
informing customers how their personal information is collected, stored, and
used. However, customers rarely read and often fail to understand privacy
policies because of the “Privacy Policy Reading Phobia” (PPRP). To tackle this
emerging challenge, we propose the first framework that can automatically
generate privacy nutrition labels from privacy policies. Based on our ground
truth applications about the Data Safety Report from the Google Play app
store, our framework achieves a 0.75 F1-score on generating first-party data
collection practices and an average of 0.93 F1-score on general security
practices. We also analyse the inconsistencies between ground truth and
curated privacy nutrition labels on the market, and our framework can detect
90.1% under-claim issues. Our framework demonstrates decent generalizability
across different privacy nutrition label formats, such as Google’s Data Safety
Report and Apple’s App Privacy Details.

Privacy Nutrition Labels, Privacy Policy, Large Language Model

††copyright: none

##  1\. Introduction

Software applications have become an omnipresent part of modern society. These
applications collect and use a tremendous amount of personal information from
customers to enhance user experiences. However, the collected data is often
misused for undisclosed actions, such as targeted advertising (Ullah et al.,
2020), price discrimination (Mikians et al., 2012), or gender discrimination
(Datta et al., 2014), which inevitably raise privacy concerns among the
customers. Privacy policies are one of the most common methods to inform
customers how their personal information is collected, stored, and used
(Caramujo and Da Silva, 2015; Yu et al., 2015; Perez et al., 2018; Kemp,
2020). Hence, privacy policies are a vital component of responsible technology
in any software application ecosystems.

Figure 1.  The left image is a nutrition facts label format published by the
U.S. Food & Drug Administration in 2016 (foo, 2016). The middle and right
images are screenshots of privacy nutrition labels from the Google Play app
store (goo, 2022) and Apple App Store (app, 2022a), respectively.

However, customers rarely read and often fail to understand privacy policies
(Obar and Oeldorf-Hirsch, 2020). The privacy policies of software applications
are often written in lengthy descriptions and overwhelming details, including
technical and legal terms, leaving customers unable to comprehend their
content. According to a recent survey conducted by The Washington
Times111https://www.washingtonpost.com/technology/2022/05/31/abolish-privacy-
policies/, 36% of interviewees have never read privacy policies, and only 9%
have always read privacy policies before agreeing to a software application’s
terms. We call this specific quasi-bibliophobia as “Privacy Policy Reading
Phobia” (PPRP). The three main reasons causing the PPRP problem are presented
as follows:

  * •

Privacy policies require their aspects, such as legal, technical, and social,
to be explained clearly and understandably (Blakkarly and Graham, 2022). As
the legal and technical statements need specific domain knowledge, the
software application providers may employ explicit expert language to describe
their information. As a result, customers who lack technical or legal
education may struggle to understand privacy policies. Some research studies
found that most privacy policies require at least a college-level education to
comprehend their content (Kincaid et al., 1975; Krumay and Klar, 2020),
leading customers to feel uneasy about spending time reading and understanding
the privacy policies.

  * •

Privacy policies often provide lengthy and detailed descriptions of data
practices in software applications, leading to information overload (Krumay
and Klar, 2020; Blakkarly and Graham, 2022). A recent study shows that the
privacy policies of 75 leading applications and websites contain an average of
4,000 words and require approximately 16 minutes to read completely (Blakkarly
and Graham, 2022). Moreover, the average reading time for a whole privacy
policy is only 73 seconds (Obar and Oeldorf-Hirsch, 2020); hence, it will be
challenging for customers to understand the complete content of the privacy
policy.

  * •

The “Privacy Paradox” refers to the phenomenon where individuals express
concerns about their privacy but, in practice, engage in behaviors that
compromise their own privacy (Barth and De Jong, 2017). It is commonly
attributed to that customers often have a confirmation bias (Nickerson, 1998;
Mercier, 2016; Lewicka, 1998). Specifically, as customers often lack awareness
of privacy protection laws, they tend to accept the software applications’
privacy policies because they believe that these applications are harmless
(Dommeyer and Gross, 2003; Xie and Karan, 2019; Tinggi et al., 2011).

As the key is that software applications are used by broad groups of people in
life and work, there is an urgent need to provide a more concise and
transparent form of privacy policies, thus helping customers to save time
reading and understanding these policies (Mulder, 2019; Mohan et al., 2019;
Tang et al., 2021). Concise, transparent and understandable language to
deliver privacy information are also commonly required in privacy regulations
such as GDPR (GDP, 2016) [Art. 12 (1)] and CCPA (CCP, 2018) [Regs
§999.308(a)(2)(d)]. Researchers have proposed privacy nutrition labels (Kelley
et al., 2009, 2010, 2013) to offer a structured and succinct way of informing
customers about how their information is collected and used in practice. Kelly
et al. (Kelley et al., 2009) defined privacy nutrition labels aiming to
improve the visual presentation and comprehensibility of privacy policies for
customers to mitigate the problem of lengthy and detailed policies.
Specifically, they designed privacy nutrition labels by employing a two-
dimensional grid layout with titles and bold texts to reduce the complexity of
privacy policies. Following this work, Kelly et al. (Kelley et al., 2010) also
found that standardized privacy nutrition labels may increase the speed of
comprehension and enjoyment of reading for customers. Later on, some research
work focuses on designing privacy nutrition labels for mobile applications
(Kelley et al., 2013) and Internet of Things (IoT) devices (Emami-Naeini et
al., 2020) to help customers decide whether they should install the software
applications or purchase the IoT devices.

Privacy nutrition labels aim to improve customers’ awareness and help them
easily learn and understand privacy practices from software applications. This
concept was originally inspired by the common nutrition labels for food.
Figure LABEL:fig:sfig1 is an example of the nutrition facts label published by
the U.S. Food & Drug Administration (foo, 2016), which demonstrates the
existence and amount of indicated nutrition elements. Moreover, the concept of
privacy nutrition labels has been put into practice. Recently, the Google Play
app store222https://techcrunch.com/2022/04/26/google-play-launches-its-own-
privacy-nutrition-labels-following-similar-effort-by-apple/ and Apple App
Store333https://techcrunch.com/2020/12/14/apple-launches-its-new-app-privacy-
labels-across-all-its-app-stores/ also required all applications to provide
their privacy nutrition labels to empower customers to learn about privacy
policies before using the applications (see Figure LABEL:fig:sfig2).

In recent years, many researchers have extensively investigated the problem of
privacy nutrition labels to assist customers in lowering their anxiety when
installing software applications (Ciocchetti, 2008; Kelley et al., 2009; Li et
al., 2022b; Cranor, 2012; Balebako et al., 2014). Specifically, they have
explored the variants of privacy nutrition labels in multiple dimensions to
improve their visual information. For example, Cranor (Cranor, 2012) pointed
out that privacy nutrition labels require some features, such as a
standardization format, machine-readable automation, or a simple layer.
Balebako et al. (Balebako et al., 2014) showed that there is a gap between
customers’ interpretation and expert opinions in understanding privacy
policies, leading developers to evaluate the usability of their software
applications before deploying them. Even though privacy nutrition labels have
been proven useful in the last decade, unlike food nutrition labels, there is
no widely recognised and officially promulgated standard for privacy nutrition
labels. Specifically, various software application platforms have different
taxonomies for the personal information involved during usage. Some
application stores (e.g., Google Play app store) also require to include
general privacy practices, such as the security measures during data transfer
or whether the software application provides certain data rights (e.g., the
Right to be Forgotten), in the privacy nutrition labels.

In this paper, we propose a framework for automatically generating privacy
nutrition labels from verbose privacy policies to assist software application
users in comprehending the privacy policies. Specifically, by subtly utilizing
the pre-trained Large Language Models (LLMs), our framework shows decent
adaptability to cope with different privacy nutrition label formats. Our
framework takes as input software applications’ privacy policies and platform-
specific privacy label formats to construct privacy nutrition labels that
satisfy the platform standards. Specifically, we first put privacy policies
into the document processing module and the context classification module to
produce privacy policy segments related to high-level data practices. These
segments and platform-specific privacy label formats are then further placed
into the label generation module, which is mainly driven by an LLM, to finally
generate privacy nutrition labels. Based on our ground truth dataset, the
framework achieves a 0.75 F1-score on first-party collection data practices, a
0.63 F1-score on third-party sharing data practices, and an average 0.93
F1-score on general security practices for Google’s Data Safety Report format.
We summarise and attribute the wrong cases to three main reasons: omnibus data
types, ambiguity caused by group-specific clauses, and under-performed context
classification. We further examine the inconsistencies between ground truth
and curated privacy nutrition labels. The results show that our framework
performs better for apps without privacy nutrition labels. Our examination
exposes that there could be around 12.6% of data practices of the first-party
collection are under-claimed. Our proposed framework can detect 90.1% of these
under-claimed issues. As for Apple’s App Privacy Details format, our framework
yields an average 0.70 F1-score for three data practice attributes, exhibiting
decent generalizability. Moreover, we explore an alternative framework design
that recursively employs another LLM to replace the first two modules that are
based on Natural Language Processing (NLP) techniques. We find that the new
strategy improves 13.1% F1-score on generating third-party collection data
practices, with approximately five times greater monetary cost. Overall, our
framework demonstrates decent performance in generating privacy nutrition
labels from privacy policies, which can assist customers in quickly and easily
understanding the privacy information contained in privacy policies.

Figure 2. An overview framework for generating privacy nutrition labels from
privacy policies.

The main contributions of our work are as follows:

  * •

To the best of our knowledge, we are the first to propose a novel framework
for automatically generating privacy nutrition labels from privacy policies.

  * •

Our framework achieves respectable performance on both general and specific
data practices for applications on the Google Play app store. Moreover, we
also find that there are inconsistencies between the curated and ground truth
privacy nutrition labels in those applications.

  * •

We employ our framework on Apple’s App Privacy Details format to demonstrate
its generability. The results show that our approach is applicable on multiple
platforms, such as the Google Play app store and the Apple App Store.

  * •

We also propose a fully Large Language Model-based framework for constructing
privacy nutrition labels from privacy policies. The results show that this
framework improves the performance of data practices involving data shared
with third-party. However, this framework requires much more monetary
investment.

The rest of this paper is organized as follows: Section 2 introduces the
motivation and structure of our proposed framework; Section 3 covers the
details of experimental setups, including our dataset. Section 4 describes the
results and findings of our research questions; Section 5 discusses internal
and external threats to the validity of our project; Section 6 summaries the
development of related work; and lastly, in Section 7, we present a conclusion
and an outlook to the future.

##  2\. Framework of Privacy Nutrition Labels Generations

In this section, we present the motivation behind our framework design. We
provide an overview of the proposed framework, which automatically generates
privacy nutrition labels from privacy policies. Additionally, we introduce the
three main modules of the framework in greater detail.

###  2.1. Motivation

Our framework takes privacy policies as input and employs NLP techniques and
pre-trained LLM, to automatically generate privacy nutrition labels for
privacy policy readers. There are four main reasons to motivate the
practicality of the proposed framework.

  * •

Nowadays, we have many existing tools that assist developers in automatically
generating their privacy policies across various domains, such as websites
(Liu et al., 2014; web, 2022), mobile applications (Yu et al., 2015; Zimmeck
et al., 2021; Iub, 2022; ter, 2022), and e-commerce (Spiekermann et al.,
2001). These tools also help developers easily comply with their privacy
policies with data and privacy protection laws. However, there is no tool to
automatically create privacy nutrition labels to help customers easily
understand privacy policies. As we have more privacy policies than privacy
nutrition labels and these labels are often built based on privacy policies
(Kelley et al., 2009, 2010, 2013), there is a need and an opportunity to take
advantage of available privacy policies to automatically construct privacy
nutrition labels to save time and effort for customers, relieving the pain of
reading privacy policies.

  * •

The idea of privacy nutrition labels has gained traction in recent years, with
several companies and organizations adopting their own versions. Even though
privacy nutrition labels have been proven useful in the last decade, unlike
food nutrition labels, there is no widely recognised and officially
promulgated standard for privacy nutrition labels. In addition, the laws and
regulations around data privacy emerge and evolve rapidly, making it intricate
to create a universal privacy nutrition labels that is meaningful and relevant
for all consumers for a long period of time. To respond to these dynamic
requirements, the automatic privacy nutrition labels generation framework is
expected to have sufficient adaptability.

  * •

Pre-trained LLMs (Floridi and Chiriatti, 2020; Min et al., 2021) have shown
their surprisingly strong ability on almost every task of NLP task. In
particular, the breakthrough and far-reaching of GPT-3 (Brown et al., 2020), a
pre-trained LLM, have fundamentally changed the development paradigm of deep
learning models (Min et al., 2021). One of the most significant improvement of
pre-trained LLMs is “One ring to rule them all.” Specifically, pre-trained
LLMs are trained on massive amounts of text data using unsupervised learning
techniques. This means that during training, the model learns to understand
the patterns and structures of language, including syntax, semantics, and
contextual relationships between words and phrases. These learned patterns and
structures are stored in the model’s weights and can be directly applied to a
wide range of natural language processing tasks. Its strong adaptability
finely matches our need; therefore, we integrate LLM into our framework to
automatically generate privacy nutrition labels from privacy policies.

  * •

Pre-trained LLMs have been employed in practice to solve many vital problems,
such as code generation (Narasimhan et al., 2021; Paik and Wang, 2021), bug
detection (Allamanis et al., 2021), or user interface design (Ellawela and
Lakmali, 2021). However, if a privacy policy is directly fed into these LLMs
to generate its privacy nutrition labels, there will be the following issues.
First, LLMs have a limited input length, which means they will cause errors
when the length of a given privacy policy is greater than the limited input
length. According to (Krumay and Klar, 2020), the average length for popular
apps and websites is approximately 4,000 words, while the input limitation for
GPT-3 is around 1,500 English words. Second, the privacy policies are often
lengthy, leading to longer processing times and requiring much more
computational resources without preprocessing them. Finally, based on our
preliminary results, we find that LLMs tend to return answers like, yes,
presence, or positive, for yes-or-no questions along with long context
prompts, which will harm the quality of generated privacy nutrition labels.

Above all, we propose a framework that basically combines traditional NLP
techniques and LLMs to generate privacy nutrition labels in the desired
format.

###  2.2. Framework overview

Figure 2 illustrates the overall framework for generating private nutrition
labels from privacy policies. The proposed framework takes a privacy policy
and platform-specific privacy label format as input. The privacy policy is fed
into a document processing module and a context classification module to
produce privacy policy segments related to data practices. A list of privacy
policy segments and platform-specific privacy label format is then further put
into a label generation module to generate privacy nutrition labels. The three
main modules of our framework are listed as follows:

  * •

Document processing module: This module takes information from the privacy
policy as input and outputs a list of privacy policy segments. Following a
previous work (Windl et al., 2022), we define that each segment may contain
one to four sentences.

  * •

Context classification module: This module takes as input a list of segments
and aims to produce a list of privacy policy segments related to data
practices categories. We follow the taxonomy proposed in (Wilson et al.,
2016).

  * •

Label generation module: This module takes a list of privacy policy segments,
their corresponding coarse data practices, and desired privacy label formats
as inputs and outputs privacy nutrition labels by mainly employing the Q&A
function in the natural language of pre-trained LLM.

In the following subsections, we explain the details of each module.

###  2.3. Document processing module

Researchers often collect privacy policies using URLs (Toch et al., 2010;
Apolinarski et al., 2015; Rahmouni et al., 2014). For mobile applications, we
can easily obtain privacy policy URLs by looking at their homepages on app
stores, such as the Google Play app store or the Apple app store. For
websites, privacy policy URLs are usually located at the top or bottom of
their homepages and contain some keywords, such as “privacy policy”, “privacy
notice”, and “legal terms”. We employ two Python libraries, i.e., Selenium
(Sel, 2022) and BeautifulSoup (bea, 2022), to capture these privacy policies
in HTML DOM format by visiting these URLs. We then further preprocess these
privacy policies and construct their segments. We briefly present these steps
in the following paragraphs.

Preprocessing. Following the previous work (Windl et al., 2022), we remove
some unnecessary elements, such as script tags, CSS instructions, and headers,
from privacy policy HTMLs as these elements omit actual privacy policy
content. We then employ a language-detection Python library (lan, 2022),
namely langdetect, to further remove texts in non-primary language from
privacy policies, because some websites contain the same privacy policies in
different languages. In the end, we only keep privacy policies written in
English in this study, but our approach is not limited to English.

Segmentation. As developers often write privacy policies in lengthy and
detailed texts to explain each requirement in data practices (Krumay and Klar,
2020; Blakkarly and Graham, 2022), there is a need to group similar sentences
of privacy policies into a segment to describe each specific requirement in
data practices. Inspired by the previous work on the segmentation method in
privacy policies (Windl et al., 2022), we first employ a Python natural
language processing library (Qi et al., 2020), namely Stanza, to tokenize a
privacy policy into multiple sentences. Second, we use a Python word embedding
library (Bojanowski et al., 2016), namely fastText, to train the word
embedding vector from the MAPS privacy policies datasets (Zimmeck et al.,
2019). After this step, each token in privacy policies is represented by a
word embedding vector. We then aggregate the word embedding vectors of each
token in a sentence to build the sentence embedding vector. Finally, we
calculate the cosine similarity between the sentence embedding vectors and
merge these sentences into a segment by using a model collected from (Windl et
al., 2022).

At the end of this module, we obtain a list of privacy policy segments.

###  2.4. Context classification module

The objective of this module is to produce privacy policy segments that are
highly related to data practices. We follow the previous work (Wilson et al.,
2016) to define 12 high-level categories for data practices. Specifically, the
high-level categories include “First-Party Collection/Use”, “Third-Party
Sharing/Collection”, “User Access, Edit and Deletion”, “Data Retention”, “Data
Security”, “International & Specific Audiences”, “Do Not Track”, “Policy
Change”, “User Choice/Control”, “Introductory/Generic”, “Practice not
covered”, and “Privacy contact information”.

We reuse the trained multi-label classification model mentioned in (Windl et
al., 2022). Specifically, the multi-label classification model contains a
convolutional neural network, including one convolutional layer, one pooling
layer, and two dense layers. Based on previous studies (Harkous et al., 2018;
Windl et al., 2022), we set the probability threshold of our classification
model at 0.5. Typically, if the predicted probability score of a privacy
segment is greater than 0.5, the privacy segment is labeled as the
corresponding data practice(s).

At the end of this module, we can obtain privacy policy segments that are
highly related to the categories of data practices.

###  2.5. Label generation module

The target of this module is to accurately and efficiently generate privacy
nutrition labels. The module takes privacy policy segments, their
corresponding data practices categories, and privacy label format as inputs
and outputs privacy nutrition labels by mainly employing the LLM, i.e., GPT-3.
We describe how we process inputs step-by-step in the following paragraphs.

Privacy nutrition labels format. To standardise the displayed information on
privacy nutrition labels, platforms usually have a set of requirements that
specifically define the information the developers need to provide. Similar to
the food nutrition label, which has different sections such as trace elements,
vitamins, and energy, privacy nutrition labels also explicitly categorise
essential information into various sections. And for each section, a set of
attributes needs to be fulfilled by the developers. For example, as we shown
in Table 1, Google stipulates three sections for the Data Safety Report:
first-party data collected, data shared with third-party, and general security
practices; and for each section, attributes, and their scopes are also
designated.

Mapping. After obtaining the privacy nutrition labels format, we map each
section with one or more high-level data practices categories discussed in the
context classification module. For example, [“first-party collection/use” and
first-party data collected]; [“user access, edit and deletion”, “data
retention” and RTBF]. By doing so, only related privacy policy segments will
be fed into the LLM.

LLM and prompt. The pre-trained LLM requires users to provide a prompt to
identify the context of a given question and the question itself for Q&A
completion. According to Floridi and Chiriatti (Floridi and Chiriatti, 2020),
the results of LLMs highly depend on the definition of the prompt, i.e.,
clear, reasonable, and appropriate. Moreover, some studies show that we could
amplify LLMs’ problem-solving capabilities by providing good prompts (Kojima
et al., 2022; gpt, 2022). For this reason, we aim to design our prompts based
on expected privacy nutrition labels format. As for the question design, LLMs
show stronger capability on open-ended question answering compared to closed-
ended questions (Hendrycks et al., 2020; gpt, 2022). However, the intuitive
open-ended Q&A inevitably requires more effort in processing and analysing
diverse answers. On privacy nutrition labels, to guarantee the uniformity of
displayed information, all attributes’ values are from a predefined universal
set, therefore we believe that close-ended Q&A fits our task better. To
alleviate the inherent weaknesses of LLM in answering closed-ended questions,
we design our questions in a yes-or-no format rather than closed-ended multi-
choice questions. As for the context, we directly use the related privacy
policy segments based on the mapping. Moreover, we also include the
application’s name displayed in the market in the context to improve the LLMs’
performance. By combining the question and its context as prompt, we then
iterate the question-answer process on all privacy nutrition label attributes
and collect answers. If the answer starts with “yes”, we count the value for
this attribute as presence or positive, otherwise absence or negative.
Detailed explanations of prompt design are presented in the supplementary
material.

At the end of this module, we can obtain the generated privacy nutrition
labels.

##  3\. Experimental Setup

In this section, we describe our dataset and evaluation metrics used to
estimate the performance of our framework. We also justify the selection of
the pre-trained LLM.

Table 1. google Data safety reports format on the Google Play app store. RTBF
stands for the Right To Be Forgotten.

First-party data collected
---
Attribute | Description | Attribute value
Data type | The data type that the app claims to collect. | 38 types
Data shared with third-party
Attribute | Description | Attribute values
Data type | The data type that the app claims to share. | 38 types
Security practices
Attribute | Description | Attribute values
Encryption | Data is encrypted in transit. | Yes/No
RTBF | You can request that data be deleted. | Yes/No

###  3.1. Dataset

To evaluate the performance of our framework, we need to construct the ground
truth of privacy nutrition labels. The Google Play app store is one of the
pioneering stores requiring its applications to provide privacy nutrition
labels (Li et al., 2022a). Thus, we collect Android mobile applications with
their privacy policies and privacy nutrition labels from the Google Play app
store. In the following paragraphs, we present the details of its privacy
nutrition labels format and our collection strategy.

Google Play app store and its Data Safety Report. Android mobile applications
are widely discussed and recognized as transparent and friendly to academic
research (Li et al., 2022a; Kumar et al., 2022). Although there are many
application markets for the Android platform, the Google Play app store is the
largest (with over 2.6 million applications) and most accessible app market
(app, 2022b). Moreover, the Google Play app store not only requires developers
to provide a privacy policy link for their applications but also the Data
Safety Report, which is a form of privacy nutrition labels. We summarize the
Data Safety Report format (privacy nutrition labels) in Table 1. The Data
Safety Report (goo, 2022) contains privacy information from three
perspectives. The first section is the first-party data collected, which
includes 38 data types. Similarly for the section for data shared with third-
party, which also includes 38 data types. For security practices, the
encryption and the Right To Be Forgotten (RTBF) attributes give information
about whether data is encrypted during transit and whether the users can
request their data to be deleted.

Apps selection. We randomly harvest around 10% of mobile applications from the
Google Play app store based on the AndroZoo collection (Allix et al., 2016).
We found that only 17,675 (6.6%) mobile applications provide a Data Safety
Report, whilst 84.3% of applications provide a privacy policy on their Google
Play app store homepages. We suspect that the Google Play app store has not
rigorously enforced the new requirements for privacy nutrition labels, which
further raises our concerns about the quality of existing privacy nutrition
labels. According to previous studies (Liu et al., 2018; Yu et al., 2018;
Zimmeck et al., 2019; Liu et al., 2021), we often have a large number of
mobile applications containing low-quality privacy policies; hence, we should
remove these privacy policies so we can accurately generate privacy nutrition
labels. To collect high-quality privacy policies from mobile applications, we
follow previous work (Zimmeck et al., 2019; Liu et al., 2021; Kumar et al.,
2022). Specifically, we first remove privacy policies that less than 200 words
(Kumar et al., 2022) or file size smaller than 2KB (Liu et al., 2021). Then we
refer to the rate of applications, the number of installations, popular
categories, etc., to select mobile applications having high-quality privacy
policies. In the end, our dataset collection contains 52 mobile applications.
Figure 3 describes the application category distribution of our dataset. We
then further extract their privacy policies and manually label their privacy
nutrition labels.

Table 2. The statistical analysis of our dataset. No. Privacy policies | 52
---|---
No. Segments | 7,659
No. Words | 202,932
Segments per Privacy policy | 151
Words per Segment | 27
Words per Privacy policy | 3,979
Figure 3. The application category distribution of our dataset.

Ground truth dataset. There has been no academic research done to ensure the
quality of privacy nutrition labels on the Google Play app store.
Additionally, according to a recent industrial report from Mozilla (See,
2023), nearly 80% of the apps they reviewed have some discrepancies between
the apps’ privacy policies and the information they claimed on Google’s Data
Safety Report. Moreover, only a small portion of applications have privacy
nutrition labels, leading to deep concern about their quality. For these
reasons, we need to create a ground truth dataset for privacy nutrition labels
as in the data safety report format to accurately evaluate our framework. We
recruited two Ph.D. students with privacy policy research backgrounds to read
the original privacy policies and label all attributes in each section
individually. For any disagreement, the students discussed and agreed on the
same answer, and if the disagreement persisted, a third author (a senior
researcher) joined the discussion to facilitate a resolution. Table 2 further
presents the statistical analysis of our collected privacy policies in terms
of the number of segments and the number of words.

###  3.2. Evaluation metrics

We employ classification evaluation metrics such as precision, recall, and
F1-score to estimate the performance of our framework. Specifically, we aim to
guarantee the uniformity of privacy nutrition labels, i.e., all attributes
come from a universal attribute set and all attributes’ values are also from
predefined universal attribute sets (see Table 1). We use the macro-average to
calculate the scores for each evaluation metric to prevent a bias in our
results.

###  3.3. Large language model selection

Language models have seen tremendous advancements in recent years, especially
with numerous large models such as GPT-3 (Brown et al., 2020), chatGPT (cha,
2023), GPT-4 (OpenAI, 2023), LLaMA (Touvron et al., 2023), and Bard (Goo,
2023). We select GPT-3 in this paper for the three following reasons: First,
GPT-3 is one of the earliest LLMs published in 2020, and basically, all
following LLMs are proved stronger than GPT-3 on all dimensions. Thus, we
believe various LLM selections will not negatively affect the fidelity and
performance of our framework. Second, GPT-3 has been the focus of extensive
research and analysis by the academic community for a significant period. This
indicates that there is a wealth of knowledge and resources available for
further analysis and investigation in the future. Third, some researchers have
proposed and proved that GPT-3 could be compressed into a much smaller model
while maintaining similar performance (Shi et al., 2022; Tao et al., 2022).
The slimming of LLMs enables the possibility that our framework is extended to
work offline for some specific IoT scenarios.

##  4\. Results and Discussion

Table 3. google Performance of our proposed framework on the ground truth
dataset (RQ1). RTBF stands for the Right To Be Forgotten.

| Prec. | Rec. | F1-score
---|---|---|---
First-party data collected | 0.772 | 0.797 | 0.751
Data shared with third-party | 0.668 | 0.753 | 0.632
Encryption | / | / | 0.942
RTBF | / | / | 0.923
Security practices | / | / | 0.932
(a)

| Prec. | Rec. | F1-score
---|---|---|---
First-party data collected | 0.773 | 0.814 | 0.758
Data shared with third-party | 0.689 | 0.781 | 0.655
Encryption | / | / | 0.942
RTBF | / | / | 0.923
Security practices | / | / | 0.932
(b)

In this section, we present and discuss the results of the following research
questions (RQs):

  * •

RQ1: What is the performance of the proposed framework?

  * •

RQ2: Are there any inconsistencies between the ground truth and the privacy
nutrition labels curated on the Google Play app store?

  * •

RQ3: How is the generalizability of the proposed framework?

  * •

RQ4: What is the performance of fully LLM-based design for the proposed
framework?

###  4.1. Performance of our framework (RQ1)

Our results from the ground truth dataset are shown in Table 3a. The results
show the overall first-party data collected is better than the data shared
with third-party. We find that privacy policies tend to clearly enumerate the
data practices for first-party, but indefinitely for third-party. This
observation is intuitively reasonable and conforms to conclusions from
previous studies (See, 2023; Li et al., 2022b, a). Furthermore, both
attributes in security practice achieve over 0.9 F1-score (accuracy) ,
demonstrating strong capability in the general data practices of our
framework. We further manually inspect and analyze the wrong cases,
summarising the following reasons behind them.

Omnibus data types. In law, clauses that include the term “other(s)” are often
referred to as “omnibus” or “catch-all” clauses. As for the Data Safety Report
format presented by the Google Play app store, we notice that there are seven
data types for first-party data collected and seven data types for data shared
with third-party include the term “other” in their names or definitions. For
example, Other Info which is defined as “Any other personal information, such
as date of birth, gender identity, veteran status, etc.”, or Other In-app
Messages which is represented as “Any other types of messages. For example,
instant messages or chat content.” The use of the omnibus data types can be
helpful in situations where the privacy nutrition labels format is intended to
be flexible or adaptable, as it allows for the inclusion of new or unforeseen
circumstances that may arise over time. However, these data types lead to
ambiguity or uncertainty about their scope and application to the customers.
They also raise the understanding difficulty of our framework. According to
Table 3b, the performance of our framework increases on all evaluation
metrics, i.e., precision, recall, and F1-score, if we exclude the 14 omnibus
data types.

Ambiguity caused by group-specific clauses. To be compatible and compliant
against regulation laws such as GDPR (GDP, 2016) and CCPA (CCP, 2018), privacy
policies should have some group-specific clauses that are only applicable to
certain groups of users based on residency or age. Therefore, privacy policies
commonly have group-specific clauses to respond to various regulatory
requirements, which brings ambiguity issues to our generation process. Since
the target user group is not specified, our model will be confused by some
contradictory group-specific clauses on the same privacy or data practice. For
example, many privacy policies mentioned that they do not collect or share any
data from underage users, while they still actively collect or share data from
adult users. In this case, it does not matter whether our model returns
presence or absence for a data practice; either should be counted as correct.
Therefore, group-specific privacy nutrition labels are expected to be
explicitly defined by the platform, and with the clear group-specific report
format definitions and requirements as the input, our model would exhibit
better performance.

Under-performed context classification. Another significant factor that limits
our model’s performance is the capability of the context classification
module. The average F1-score of this module used to predict the high-level
categories of data practices is 0.70. The worst class among the 12 high-level
data practices is “Data Retention”, with a 0.31 F1-score. For example, the app
Super Slime Simulator (Sup, 2023), a creative game app, we find the segment
including the following sentence, i.e., “You can request the deletion of the
Personal Information. …withdraw your consent to the processing of such
information, and disable its future collection by contacting us at…”, which is
incorrectly classified as “Third-party Sharing/Collection”, instead of “Data
Retention”. Consequently, this segment is not mapped with the attribute RTBF
in the security practices section, and is not fed into the label generation
module. As a result, the RTBF of this app is wrongly labeled as “absence.” We
manually put this segment into the label generation module, and our module
labels the segment as RTBF. Therefore, it is essential to improve the
classifier performance in the context classification module to further enhance
the performance of our proposed framework.

Finding 1: Our framework demonstrates decent performance on general data
practices, yielding 0.94 and 0.92 in terms of F1-score for encryption and
RTBF, respectively. As for specific data practices, such as first-party data
collected and data shared with third-party, our framework achieves 0.75 and
0.63 F1-scores, respectively.

###  4.2. Discrepancy between ground truth and curation on market (RQ2)

Table 4. google Performance of our proposed framework for apps without a data
safety report on the Google Play app store (RQ2). RTBF stands for the Right To
Be Forgotten.

| Prec. | Rec. | F1-score
---|---|---|---
First-party data collected | 0.794 | 0.866 | 0.808
Data shared with third-party | 0.688 | 0.786 | 0.675
Encryption | / | / | 1.000
RTBF | / | / | 1.000
Security practices | / | / | 1.000
(c)

| Prec. | Rec. | F1-score
---|---|---|---
First-party data collected | 0.798 | 0.885 | 0.811
Data shared with third-party | 0.688 | 0.872 | 0.683
Encryption | / | / | 1.000
RTBF | / | / | 1.000
Security practices | / | / | 1.000
(d)

In Section 3.1, we find that a substantial proportion of applications fail to
provide privacy nutrition labels on the Google Play app store. Thus, we take a
closer look at our dataset and notice that nine applications do not claim any
data practices, including five tool applications, two game applications, one
productivity application, and one communication application. However, based on
the ground truth dataset, these applications mention at least one data
practice in their privacy policies. For example, Easy
Booster444https://play.google.com/store/apps/details?id=easy.booster.clean.tool
is a tool application to make mobile phones run smoother and faster, with over
10 million installs. Its privacy policy contains the following phrase:
“Information Collection and Use … Log Data. We want to inform you that
whenever you use our Service … This Log Data may include information such as
your device Internet Protocol (“IP”) address, device name, operating system
version, the configuration of the app when utilizing our Service …”

Table 4a and Table 4b show the performance of the applications with and
without the 14 omnibus data types, respectively. The results indicate a
marginal improvement over the mean performance of the entire dataset, namely
0.05 (+7.9%) F1-score for first-party data collected, 0.04 (+6.8%) F1-score
for data shared with third-party, and 0.07 (+7.2%) F1-score for security
practices. Various privacy policy generation tools are available for software
applications, which include source code analysis-based tools (Yu et al., 2015,
2018; Zimmeck et al., 2021) and online automated privacy policy generators
that employ questionnaires (Iub, 2022; ter, 2022; web, 2022). However, there
are no existing tools that can assist developers in constructing privacy
nutrition labels from privacy policies. Our promising results suggest that our
framework can be applied to further assist them in automatically generating
privacy nutrition labels, especially for the data practices related to first-
party or shorter privacy policies.

Under-claimed data practices. Several previous studies have discussed and
identified common challenges developers when they construct privacy nutrition
label (Li et al., 2022a, b). Our findings are similar to theirs. Specifically,
except for the absence of privacy nutrition labels, we find that there are
many under-claimed data practices in the Data Safety Report curated on the
Google Play app store, compared to their privacy policies. For example, Match
Masters (Mat, 2023) is a tile-matching game app with over 10 million installs.
In their privacy policy, they wrote a long list of “Which Information do we
collect?”, including “Personal Data (optional)”, “Location”, “Sessions log”,
and “Mobile device model”. We are able to identify all corresponding first-
party data practices in its Data Safety Report except Location. Although there
are four conditions under which developers are exempt from disclosing data
accessed by an application as “collected” in the Data Safety Report, there is
no special explanation about “Location” in the privacy policy. Thus, we regard
the missing “Location” as an under-claim. In total, there are around 12.6% of
data practices, related to first-party data collected, are under-claimed,
excluding omnibus data types. The specific under-claim rates for each data
type are shown in Figure 4 as light gray bars. The top three under-claimed
data types are User Payment Info, Approximate Location, and User IDs.

Figure 4. For privacy nutrition labels curated on the market, the under-claim
rate of first-party data collected in percentage (as light grey bars) and the
not detected under-claim rate (as dark grey bars). Inclusive attributes are
removed. Attributes that are less than 10% under-claim rate are omitted.

Under-claiming detection. Under-claiming data practices in privacy information
undermines user trust, can lead to unintended disclosure of privacy
information, and may have legal consequences. To tackle this challenging
issue, we employ our framework for detecting under-claim issues. The results
show that our framework is able to identify 90.1% of these issues. Especially,
for data practices about basic personal information, such as User IDs, Name,
Email Address, Phone Number, and Address, our framework can successfully
notice 100% of under-claim issues (See Figure 4). Generally speaking,
individually verifying all under-claiming is too complicated to conduct
without specific legal knowledge, as many cases merely fit the exemption
conditions. Still, in addition to providing early reminders to the customers,
our framework is applicable to assist legal experts in cross-checking and
detecting the consistency between privacy policies and their privacy nutrition
labels.

Finding 2:  There are inconsistencies between the generated and ground truth
privacy nutrition labels. Specifically, around 17% of applications do not
provide privacy nutrition labels and for those applications, our framework can
achieve 0.81, 0.68, and 1.00 in terms of F1-score for first-party data
practices, third-party data practices, and security practices, respectively.
For the Data Safety Report curated on the Google Play app store, there are
12.6% under-claim data practices. Our framework can detect 90.1% of them.

###  4.3. Generalizability of proposed framework (RQ3)

Table 5. apple Performance of our proposed pipeline on Apple’s app privacy
details. (RQ3)

Attribute | Prec. | Rec. | F1-score
---|---|---|---
Data linked to you | 0.763 | 0.778 | 0.737
Data not linked to you | 0.769 | 0.799 | 0.687
Data used to track you | 0.769 | 0.701 | 0.648
Average | 0.767 | 0.759 | 0.691
(a)

Attribute | Prec. | Rec. | F1-score
---|---|---|---
Data linked to you | 0.782 | 0.771 | 0.756
Data not linked to you | 0.767 | 0.811 | 0.691
Data used to track you | 0.765 | 0.696 | 0.637
Average | 0.771 | 0.759 | 0.695
(b)

Table 6. apple Apple’s app privacy details format. Notably, the first two
attributes are not mutual exclusive.

Data type
---
Attribute | Description | Attribute Value
Data linked to you |  Data that is linked to your identity (via your account, device, or other details). | 13 categories
Data not linked to you |  Data that is not linked to your identity (via your account, device, or other details). | 13 categories
Data used to track you |  Data may be used to track you across apps and websites owned by other companies | 13 categories

To demonstrate the generalizability of our method, we further employ it to
generate privacy nutrition labels in Apple’s App Privacy Details format. The
Apple App Store is Apple’s official app market for mobile apps on its iOS and
iPadOS operating systems. Similar to Google’s Data Safety Report, Apple’s
version of privacy nutrition labels is called App Privacy Details. Developers
are required to provide this information to submit new applications and their
updated information to the Apple App Store. Compared to Data Safety Report in
the Google Play app store, there are two prominent format differences for
Apple’s App Privacy Details (See Table 6). First, App Privacy Details classify
all data practices into three categories: data linked to you, data not linked
to you, and data used to track you; Second, Apple’s App Privacy Details do not
have general security practices that are applicable to all data practices.

Among the 52 applications in the original ground truth dataset, 13
applications are not published on the Apple App Store and four applications
have various privacy policy links. We remove these applications, and there are
35 remaining applications. We employ our framework for these privacy policies’
applications to generate privacy nutrition lables as per Apple’s App Privacy
Details format. Table 5a shows the performance of our framework on the 35
applications. We also further exclude three applications that do not provide
privacy nutrition labels. Table 5b presents the results of our framework for
the remaining 32 applications.

The information included in Apple’s App Privacy Details is more challenging to
comprehend than the privacy policies. Especially when links between data
practices and users are unknown, the process of generating privacy nutrition
labels or drafting privacy policies is more challenging (Li et al., 2022b, a;
Zhang et al., 2022). The results of our evaluation demonstrate that the
framework has the best performance in generating privacy nutrition labels for
data linked to you, followed by data not linked to you and data used to track
you. This ranking aligns with the findings of previous studies and corresponds
to our intuitive sense of the relative difficulty of understanding these
attributes. Our framework achieves overall comparable results, i.e., 0.77
precision, 0.76 recall, and 0.70 F1-score, showing that the proposed framework
performs decent generalizabiltiy on generating privacy nutrition labels in
various formats.

Finding 3: Our framework exhibits decent generalizability by achieving an
overall 0.77 precision, 0.76 recall, and 0.70 F1-score on Apple’s App Privacy
Details format.

###  4.4. A trial of fully LLM-based solution (RQ4)

Table 7. google Performance of an alternative framework design based on
recursively employing LLMs (RQ4).

| Prec. | Rec. | F1-score
---|---|---|---
First-party collected | 0.757 | 0.787 | 0.744
Shared with third-party | 0.759 | 0.761 | 0.715
(c)

| Prec. | Rec. | F1-score
---|---|---|---
First-party collected | 0.755 | 0.811 | 0.751
Shared with third-party | 0.776 | 0.757 | 0.710
(d)

Chain-of-AI (CoAI) or Chain-of-Thought (CoT) has gained significant popularity
in recent several years as a means of creating more powerful and sophisticated
AI systems. By combining multiple AI models, especially LLMs, in a sequential
or interconnected manner, this approach has been shown to be effective in
natural language processing tasks, such as machine translation and chatbots
(Gulcehre et al., 2017; Schwenk and Koehn, 2008), amplifying LLMs through
recursive question-answering and debate (Mialon et al., 2023). However, some
arguments (Yang, 2023) suggest that for relatively simpler natural language
understanding (NLU) tasks, LLMs may not outperform traditional fine-tuned
models. As we discussed in Section 4.1, under-performed context classification
module limits the performance of framework. Inspired by those ideas, we
propose a fully LLM-based framework design to explore its performance.

Figure 5. An overview for the fully LLM-based framework.

Figure 5 shows an overview of our alternative framework. As we discussed in
Section 2, the document processing module and the context classification
module are implemented based on traditional NLP techniques, and the output is
privacy policy segments with their privacy practice labels. For alternative
design, we employ another LLM (LLM-1) to conduct the equivalent task.
Specifically, for every question that will be processed by the LLM-2 in the
label generation module, we employ LLM-1 to find its relevant paragraphs from
the privacy policy documents. Detailed explanations and prompt examples are
presented in the supplementary material.

Table 7 shows the performance of our alternative framework design. Notably,
given the approximately same LLM usage quota, we were only able to run around
20% of the dataset by random sampling. In addition, since security practices
have obtained an average over 0.9 F1-score, we skip them in this research
question. This new framework design achieves approximate performance on all
metrics for first-part data collected but improves 0.09 (+13.6%) and 0.08
(+13.1%) on precision and F1-score for data shared with third-party,
respectively. However, there is no noticeable increase for data practices
involving the first-party data collected. Compared to the original document
processing module and the context classification module based on traditional
NLP techniques, the extra LLM can better collect and perceive third-party
information that is usually scattered across the privacy policy document.

On the other hand, the cost of implementing this approach is a notable side
effect, since most mainstream LLMs are pay-to-use based on input length and we
recursively employ them to process long privacy policy documents. Based on our
observation, we coarsely estimate the monetary cost of fully LLM-based
solution at more than five times that of the original framework design.

Finding 4: A fully LLM-based framework does improve the performance for data
practices involving the data shared with third-party, with 0.09 (+13.6%) on
precision and 0.08 (+13.1%) on F1-score, respectively, but there is no
noticeable increase for data practices involving the first-party data
collected. Moreover, the fully LLM-based framework will result in
approximately five times greater monetary cost.

##  5\. Threats to Validity

Internal validity. Threats to internal validity refer to errors in our
experiments. To perform our experiments, we employ the power of GPT-3, a pre-
trained LLM to automatically generate privacy nutrition labels from privacy
policies. We follow OpenAI’s API to access GPT-3. We evaluate the performance
of our framework on the Google Play app store dataset. We use langdetect
Python library to detect English texts in privacy policies. We reuse the
trained multi-label classification model to identify privacy policy segments
that are related to data practices. We have rechecked the code and data.
However, there are still some remaining unknown errors.

External validity. Threats to external validity refer to the generalizability
of the study. In our experiments, we only employ applications from the Google
Play app store and the Apple App Store. This may be a threat to external
validity, as these applications may not be generalized beyond our
investigations. As these stores are widely popular in mobile applications and
require developers to provide high-quality privacy nutrition labels, we
believe that there is a minimal threat to external validity. In the future, we
plan to employ our framework on various datasets, such as websites and
e-commerce, to reduce the threats to external validity.

##  6\. Related Work

Previous researchers have been investigating the idea of privacy nutrition
labels from various perspectives. The iOS privacy nutrition label has been
empirically studied, from its market-ready status (Li et al., 2022a) to how it
is created (Li et al., 2022b). Emami-Naeini et al. (Emami-Naeini et al., 2019,
2021b) extensively studied the relationship between privacy attributes of IoT
devices and the consumer’s willingness to purchase, which showed privacy
attributes as a major concern of consumers. They further reported the design
space and experts’ opinions on the content of the privacy label for IoT
devices (Emami-Naeini et al., 2020). Based on their findings, a prototype
privacy label was proposed (Emami-Naeini et al., 2021a), aiming to inform
consumers when making purchase decisions.

Another set of related works explored the potential of automatically
generating privacy information. Some of the work focuses on generating privacy
policies from the source code of the programs, to achieve consistency between
the behavior of the software and the corresponding privacy policies. Yu et al.
(Yu et al., 2015, 2018) introduced a system called “AutoPPG” that scans the
codebase of the software and generates privacy policies by comparing the
function names and identifying the usage of privacy-related Android
programming interfaces. Another work by Zimmeck et al. (Zimmeck et al., 2021)
leveraged code signatures to extract privacy attributes of iOS apps from their
source code. Jain et al. (Jain et al., 2021) aimed to generate privacy
policies using a machine translation model piped with static analysis on the
permission-requiring code segments. From the perspective of the language
quality of generated privacy policies (Bateni and Dara, 2021), they proposed a
framework and compared the performance of LSTM and Bi-LSTM to train generative
models for privacy policy generation. Instead of focusing on privacy policies,
Gardner et al. (Gardner et al., 2022) identified data collection in the source
code, and interactively assisted developers to provide precise privacy labels
for their applications. Pan et al. (Pan et al., 2023) conducted a large-scale
empirical study of developer-oriented online Automated Privacy Policy
Generators (APPGs) for mobile apps.

Former works have unexplored the potential of automatically generating privacy
labels from privacy policies. Our paper aims to fill the gap between the
vastly available privacy policies and the newly required privacy nutrition
labels from application markets, such as the Google Play app store and the
Apple App Store. To the best of our knowledge, this framework is the first to
comprehend privacy policies and privacy label formats to generate privacy
nutrition labels, relieving customers’ pain when they try to read the privacy
policies. This approach has shown impressive performance and opened a new path
for this research direction.

##  7\. Conclusion and Future Work

The “Privacy Policy Reading Phobia” is widely recognized and discussed in
previous studies. In this paper, we propose a framework that can automatically
generate privacy nutrition labels from privacy policies, liberating software
application users from hunting for information through privacy policies. Our
framework achieves a 0.75 F1-score on generating first-party data collected, a
0.63 F1-score on generating data shared with third-party, and an average of
0.93 F1-score on general security practices for Google’s Data Safety Report.
We summarise three reasons undermining the performance of our framework,
including omnibus data types, ambiguity caused by group-specific clauses, and
under-performed context classification. We also examine the inconsistencies
between the ground truth and curated privacy nutrition labels. We find that
there could be around 12.6% data practices of the first-party data collected
are under-claimed and our framework can detect 90.1% of under-claimed issues.
Our framework also demonstrates decent generalizability, yielding an average
of 0.70 F1-score for Apple’s App Privacy Details. In addition, we explore a
fully LLM-based framework design, improving 13.1% F1-score on generating data
shared with third-party, but no noticeable increasing for first-party data
collected. Also, the alternative design requires approximately five times more
monetary cost. Lastly, we discuss the internal and external threats to the
validity of our project. By providing customers with easy-to-understand
summaries of privacy policies, companies can increase transparency and foster
trust with their user base. This is particularly important in the context of
data privacy, where consumers are increasingly concerned about the use and
protection of their personal information.

Based on our promising results and findings, we recognize two potential
directions for future research. First, we plan to implement this proposed
framework as an online tool or a web browser extension, so customers can
utilize this tool to generate the privacy nutrition label whenever they
encounter a privacy policy. We also aim to improve and optimize the efficiency
of our framework in the future. Second, we believe it is worthwhile to design
a wide-recognized, user-friendly, and regulation-compliance standard privacy
nutrition label format for websites. In addition, various privacy laws address
relatively different and few personal data types. Therefore, a region-wise or
regulation-wise privacy nutrition label format could be especially needed
soon. Above all, we believe our framework represents a step forward toward the
cure of “Privacy Policy Reading Phobia”, positively influencing users’
behavior and promoting privacy-conscious decision-making.

## References

  * (1)
  * foo (2016) 2016\.  _FDA Nutrition Facts Label_.  https://www.fda.gov/food/food-labeling-nutrition/changes-nutrition-facts-label Accessed: 2022-10-03.
  * GDP (2016) 2016\.  _General Data Protection Regulation (GDPR)_.  https://gdpr-info.eu/ Accessed: 2022-04-25.
  * CCP (2018) 2018\.  _California Consumer Privacy Act of 2018 (CCPA)_.  https://oag.ca.gov/privacy/ccpa Accessed: 2022-04-25.
  * gpt (2022) 2022\.  _Amplifying GPT-3 on closed-ended questions._ https://generative.ink/posts/amplifying-gpt-3-on-closed-ended-questions/ Accessed: 2022-10-03.
  * app (2022a) 2022a.  _App privacy details on the App Store_.  https://developer.apple.com/app-store/app-privacy-details/ Accessed: 2022-10-10.
  * app (2022b) 2022b.  _AppBrain_.  https://www.appbrain.com/ Accessed: 2022-08-26.
  * bea (2022) 2022\.  _BeautifulSoup_.  https://www.crummy.com/software/BeautifulSoup/ Accessed: 2022-10-01.
  * Iub (2022) 2022\.  _Iubenda_.  https://www.iubenda.com/en/ Accessed: 2022-03-28.
  * lan (2022) 2022\.  _langdetect_.  https://pypi.org/project/langdetect/ Accessed: 2022-10-01.
  * Sel (2022) 2022\.  _Selenium_.  https://www.selenium.dev/ Accessed: 2022-10-01.
  * ter (2022) 2022\.  _Termly_.  https://termly.io/ Accessed: 2022-03-28.
  * goo (2022) 2022\.  _Understand app privacy and security practices with Google Play’s Data safety section_.  https://support.google.com/googleplay/answer/11416267 Accessed: 2022-10-10.
  * web (2022) 2022\.  _Website Policies_.  https://www.websitepolicies.com/ Accessed: 2022-03-28.
  * Goo (2023) 2023\.  _Bard_.  https://bard.google.com/ Accessed: 2022-03-27.
  * cha (2023) 2023\.  _chatGPT_.  https://openai.com/blog/chatgpt Accessed: 2022-03-27.
  * Mat (2023) 2023\.  _Match Masters_.  https://play.google.com/store/apps/details?id=com.funtomic.matchmasters Accessed: 2023-03-24.
  * See (2023) 2023\.  _See No Evil: Loopholes in Google’s Data Safety Labels Keep Companies in the Clear and Consumers in the Dark_.  https://foundation.mozilla.org/en/campaigns/googles-data-safety-labels/ Accessed: 2022-03-27.
  * Sup (2023) 2023\.  _Super Slime Simulator: DIY Art_.  https://play.google.com/store/apps/details?id=com.dramaton.slime Accessed: 2023-03-24.
  * Allamanis et al. (2021) Miltiadis Allamanis, Henry Jackson-Flux, and Marc Brockschmidt. 2021\.  Self-supervised bug detection and repair.  _Advances in Neural Information Processing Systems_ 34 (2021), 27865–27876.
  * Allix et al. (2016) Kevin Allix, Tegawendé F. Bissyandé, Jacques Klein, and Yves Le Traon. 2016\.  AndroZoo: Collecting Millions of Android Apps for the Research Community. In _Proceedings of the 13th International Conference on Mining Software Repositories_ (Austin, Texas) _(MSR ’16)_. ACM, New York, NY, USA, 468–471.  https://doi.org/10.1145/2901739.2903508
  * Apolinarski et al. (2015) Wolfgang Apolinarski, Marcus Handte, and Pedro Jose Marron. 2015\.  Automating the generation of privacy policies for context-sharing applications. In _2015 International Conference on Intelligent Environments_. IEEE, 73–80.
  * Balebako et al. (2014) Rebecca Balebako, Richard Shay, and Lorrie Faith Cranor. 2014\.  Is your inseam a biometric? a case study on the role of usability studies in developing public policy.  _Proc. USEC_ 14 (2014).
  * Barth and De Jong (2017) Susanne Barth and Menno DT De Jong. 2017.  The privacy paradox–Investigating discrepancies between expressed privacy concerns and actual online behavior–A systematic literature review.  _Telematics and informatics_ 34, 7 (2017), 1038–1058.
  * Bateni and Dara (2021) Nastaran Bateni and Rozita Dara. 2021.  Automated generation of privacy policy using deep models. In _2021 IEEE International Symposium on Technology and Society (ISTAS)_. IEEE, 1–6.
  * Blakkarly and Graham (2022) Jarni Blakkarly and Daniel Graham. 2022.  _Privacy policy comparison reveals half have poor readability_.  https://www.choice.com.au/consumers-and-data/protecting-your-data/data-laws-and-regulation/articles/privacy-policy-comparison Accessed: 2022-05-03.
  * Bojanowski et al. (2016) Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2016.  Enriching Word Vectors with Subword Information.  _arXiv preprint arXiv:1607.04606_ (2016).
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners.  _Advances in neural information processing systems_ 33 (2020), 1877–1901.
  * Caramujo and Da Silva (2015) João Caramujo and Alberto Manuel Rodrigues Da Silva. 2015\.  Analyzing privacy policies based on a privacy-aware profile: The Facebook and LinkedIn case studies. In _2015 IEEE 17th Conference on Business Informatics_ , Vol. 1. IEEE, 77–84.
  * Ciocchetti (2008) Corey A Ciocchetti. 2008\.  The future of privacy policies: A privacy nutrition label filled with fair information practices.  _J. Marshall J. Computer & Info. L._ 26 (2008), 1.
  * Cranor (2012) Lorrie Faith Cranor. 2012\.  Necessary but not sufficient: Standardized mechanisms for privacy notice and choice.  _J. on Telecomm. & High Tech. L._ 10 (2012), 273.
  * Datta et al. (2014) Amit Datta, Michael Carl Tschantz, and Anupam Datta. 2014\.  Automated experiments on ad privacy settings: A tale of opacity, choice, and discrimination.  _arXiv preprint arXiv:1408.6491_ (2014).
  * Dommeyer and Gross (2003) Curt J Dommeyer and Barbara L Gross. 2003.  What consumers know and what they do: An investigation of consumer knowledge, awareness, and use of privacy protection strategies.  _Journal of Interactive Marketing_ 17, 2 (2003), 34–51.
  * Ellawela and Lakmali (2021) Chaveen Ellawela and KBN Lakmali. 2021.  A Review about Voice and UI Design Driven Approaches to Identify UI Elements and Generate UI Designs. In _2021 International Conference on Intelligent Technologies (CONIT)_. IEEE, 1–4.
  * Emami-Naeini et al. (2020) Pardis Emami-Naeini, Yuvraj Agarwal, Lorrie Faith Cranor, and Hanan Hibshi. 2020.  Ask the experts: What should be on an IoT privacy and security label?. In _2020 IEEE Symposium on Security and Privacy (SP)_. IEEE, 447–464.
  * Emami-Naeini et al. (2021a) Pardis Emami-Naeini, Janarth Dheenadhayalan, Yuvraj Agarwal, and Lorrie Faith Cranor. 2021a.  An Informative Security and Privacy “Nutrition” Label for Internet of Things Devices.  _IEEE Security & Privacy_ 20, 2 (2021), 31–39.
  * Emami-Naeini et al. (2021b) Pardis Emami-Naeini, Janarth Dheenadhayalan, Yuvraj Agarwal, and Lorrie Faith Cranor. 2021b.  Which privacy and security attributes most impact consumers’ risk perception and willingness to purchase IoT devices?. In _2021 IEEE Symposium on Security and Privacy (SP)_. IEEE, 519–536.
  * Emami-Naeini et al. (2019) Pardis Emami-Naeini, Henry Dixon, Yuvraj Agarwal, and Lorrie Faith Cranor. 2019.  Exploring how privacy and security factor into IoT device purchase behavior. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_. 1–12.
  * Floridi and Chiriatti (2020) Luciano Floridi and Massimo Chiriatti. 2020.  GPT-3: Its nature, scope, limits, and consequences.  _Minds and Machines_ 30, 4 (2020), 681–694.
  * Gardner et al. (2022) Jack Gardner, Yuanyuan Feng, Kayla Reiman, Zhi Lin, Akshath Jain, and Norman Sadeh. 2022\.  Helping mobile application developers create accurate privacy labels. In _2022 IEEE European Symposium on Security and Privacy Workshops (EuroS &PW)_. IEEE, 212–230.
  * Gulcehre et al. (2017) Caglar Gulcehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, and Yoshua Bengio. 2017.  On integrating a language model into neural machine translation.  _Computer Speech & Language_ 45 (2017), 137–148.
  * Harkous et al. (2018) Hamza Harkous, Kassem Fawaz, Rémi Lebret, Florian Schaub, Kang G Shin, and Karl Aberer. 2018\.  Polisis: Automated analysis and presentation of privacy policies using deep learning. In _27th USENIX Security Symposium (USENIX Security 18)_. 531–548.
  * Hendrycks et al. (2020) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020.  Measuring massive multitask language understanding.  _arXiv preprint arXiv:2009.03300_ (2020).
  * Jain et al. (2021) Vijayanta Jain, Sanonda Datta Gupta, Sepideh Ghanavati, and Sai Teja Peddinti. 2021.  Prigen: Towards automated translation of android applications’ code to privacy captions. In _International Conference on Research Challenges in Information Science_. Springer, 142–151.
  * Kelley et al. (2009) Patrick Gage Kelley, Joanna Bresee, Lorrie Faith Cranor, and Robert W Reeder. 2009.  A” nutrition label” for privacy. In _Proceedings of the 5th Symposium on Usable Privacy and Security_. 1–12.
  * Kelley et al. (2010) Patrick Gage Kelley, Lucian Cesca, Joanna Bresee, and Lorrie Faith Cranor. 2010.  Standardizing privacy notices: an online study of the nutrition label approach. In _Proceedings of the SIGCHI Conference on Human factors in Computing Systems_. 1573–1582.
  * Kelley et al. (2013) Patrick Gage Kelley, Lorrie Faith Cranor, and Norman Sadeh. 2013\.  Privacy as part of the app decision-making process. In _Proceedings of the SIGCHI conference on human factors in computing systems_. 3393–3402.
  * Kemp (2020) Katharine Kemp. 2020\.  Concealed data practices and competition law: why privacy matters.  _European Competition Journal_ 16, 2-3 (2020), 628–672.
  * Kincaid et al. (1975) J Peter Kincaid, Robert P Fishburne Jr, Richard L Rogers, and Brad S Chissom. 1975.  _Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel_.  Technical Report. Naval Technical Training Command Millington TN Research Branch.
  * Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022\.  Large Language Models are Zero-Shot Reasoners.  _arXiv preprint arXiv:2205.11916_ (2022).
  * Krumay and Klar (2020) Barbara Krumay and Jennifer Klar. 2020.  Readability of privacy policies. In _IFIP Annual Conference on Data and Applications Security and Privacy_. Springer, 388–399.
  * Kumar et al. (2022) Renuka Kumar, Apurva Virkud, Ram Sundara Raman, Atul Prakash, and Roya Ensafi. 2022.  A Large-scale Investigation into Geodifferences in Mobile Apps. In _31st USENIX Security Symposium (USENIX Security 22)_. USENIX Association, Boston, MA.  https://www.usenix.org/conference/usenixsecurity22/presentation/kumar
  * Lewicka (1998) Maria Lewicka. 1998\.  Confirmation bias.  In _Personal control in action_. Springer, 233–258.
  * Li et al. (2022b) Tianshi Li, Kayla Reiman, Yuvraj Agarwal, Lorrie Faith Cranor, and Jason I Hong. 2022b.  Understanding Challenges for Developers to Create Accurate Privacy Nutrition Labels. In _CHI Conference on Human Factors in Computing Systems_. 1–24.
  * Li et al. (2022a) Yucheng Li, Deyuan Chen, Tianshi Li, Yuvraj Agarwal, Lorrie Faith Cranor, and Jason I Hong. 2022a.  Understanding iOS Privacy Nutrition Labels: An Exploratory Large-Scale Analysis of App Store Data. In _CHI Conference on Human Factors in Computing Systems Extended Abstracts_. 1–7.
  * Liu et al. (2014) Fei Liu, Rohan Ramanath, Norman Sadeh, and Noah A Smith. 2014\.  A step towards usable privacy policy: Automatic alignment of privacy statements. In _Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers_. 884–894.
  * Liu et al. (2018) Frederick Liu, Shomir Wilson, Peter Story, Sebastian Zimmeck, and Norman Sadeh. 2018.  Towards automatic classification of privacy policy text.  _School of Computer Science Carnegie Mellon University_ (2018).
  * Liu et al. (2021) Shuang Liu, Baiyang Zhao, Renjie Guo, Guozhu Meng, Fan Zhang, and Meishan Zhang. 2021\.  Have You been Properly Notified? Automatic Compliance Analysis of Privacy Policy Text with GDPR Article 13. In _Proceedings of the Web Conference 2021_. 2154–2164.
  * Mercier (2016) Hugo Mercier. 2016\.  Confirmation bias–myside bias.  In _Cognitive illusions_. Psychology Press, 109–124.
  * Mialon et al. (2023) Grégoire Mialon, Roberto Dessì, Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi-Yu, Asli Celikyilmaz, et al. 2023\.  Augmented language models: a survey.  _arXiv preprint arXiv:2302.07842_ (2023).
  * Mikians et al. (2012) Jakub Mikians, László Gyarmati, Vijay Erramilli, and Nikolaos Laoutaris. 2012.  Detecting price and search discrimination on the internet. In _Proceedings of the 11th ACM workshop on hot topics in networks_. 79–84.
  * Min et al. (2021) Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz, Eneko Agirre, Ilana Heinz, and Dan Roth. 2021.  Recent advances in natural language processing via large pre-trained language models: A survey.  _arXiv preprint arXiv:2111.01243_ (2021).
  * Mohan et al. (2019) Jayashree Mohan, Melissa Wasserman, and Vijay Chidambaram. 2019\.  Analyzing GDPR compliance through the lens of privacy policy. In _Heterogeneous Data Management, Polystores, and Analytics for Healthcare: VLDB 2019 Workshops, Poly and DMAH, Los Angeles, CA, USA, August 30, 2019, Revised Selected Papers 5_. Springer, 82–95.
  * Mulder (2019) Trix Mulder. 2019\.  Health apps, their privacy policies and the GDPR.  _European Journal of Law and Technology_ (2019).
  * Narasimhan et al. (2021) Aishwarya Narasimhan, Krishna Prasad Agara Venkatesha Rao, et al. 2021\.  CGEMs: A Metric Model for Automatic Code Generation using GPT-3.  _arXiv preprint arXiv:2108.10168_ (2021).
  * Nickerson (1998) Raymond S Nickerson. 1998\.  Confirmation bias: A ubiquitous phenomenon in many guises.  _Review of general psychology_ 2, 2 (1998), 175–220.
  * Obar and Oeldorf-Hirsch (2020) Jonathan A Obar and Anne Oeldorf-Hirsch. 2020.  The biggest lie on the internet: Ignoring the privacy policies and terms of service policies of social networking services.  _Information, Communication & Society_ 23, 1 (2020), 128–147.
  * OpenAI (2023) OpenAI. 2023.  GPT-4 Technical Report.  arXiv:2303.08774 [cs.CL]
  * Paik and Wang (2021) Incheon Paik and Jun-Wei Wang. 2021.  Improving Text-to-Code Generation with Features of Code Graph on GPT-2.  _Electronics_ 10, 21 (2021), 2706.
  * Pan et al. (2023) Shidong Pan, Dawen Zhang, Mark Staples, Zhenchang Xing, Jieshan Chen, Xiwei Xu, and James Hoang. 2023.  A Large-scale Empirical Study of Online Automated Privacy Policy Generators for Mobile Apps.  _arXiv preprint arXiv:2305.03271_ (2023).
  * Perez et al. (2018) Alfredo J Perez, Sherali Zeadally, and Jonathan Cochran. 2018\.  A review and an empirical analysis of privacy policy and notices for consumer Internet of things.  _Security and Privacy_ 1, 3 (2018), e15.
  * Qi et al. (2020) Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020.  Stanza: A Python Natural Language Processing Toolkit for Many Human Languages. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_.  https://nlp.stanford.edu/pubs/qi2020stanza.pdf
  * Rahmouni et al. (2014) Hanene Boussi Rahmouni, Kamran Munir, Marco Casassa Mont, and Tony Solomonides. 2014.  Semantic generation of clouds privacy policies. In _International Conference on Cloud Computing and Services Science_. Springer, 15–30.
  * Schwenk and Koehn (2008) Holger Schwenk and Philipp Koehn. 2008.  Large and diverse language models for statistical machine translation. In _Proceedings of the Third International Joint Conference on Natural Language Processing: Volume-II_.
  * Shi et al. (2022) Jieke Shi, Zhou Yang, Bowen Xu, Hong Jin Kang, and David Lo. 2022.  Compressing Pre-trained Models of Code into 3 MB. In _37th IEEE/ACM International Conference on Automated Software Engineering_. 1–12.
  * Spiekermann et al. (2001) Sarah Spiekermann, Jens Grossklags, and Bettina Berendt. 2001\.  E-privacy in 2nd generation E-commerce: privacy preferences versus actual behavior. In _Proceedings of the 3rd ACM conference on Electronic Commerce_. 38–47.
  * Tang et al. (2021) Jenny Tang, Hannah Shoemaker, Ada Lerner, and Eleanor Birrell. 2021.  Defining privacy: How users interpret technical terms in privacy policies.  _Proceedings on Privacy Enhancing Technologies_ 2021, 3 (2021).
  * Tao et al. (2022) Chaofan Tao, Lu Hou, Wei Zhang, Lifeng Shang, Xin Jiang, Qun Liu, Ping Luo, and Ngai Wong. 2022.  Compression of generative pre-trained language models via quantization.  _arXiv preprint arXiv:2203.10705_ (2022).
  * Tinggi et al. (2011) Michael Tinggi, Shaharudin Jakpar, Ting Bick Chin, and Junaid M Shaikh. 2011.  Customers? Confidence and trust towards privacy policy: a conceptual research of hotel revenue management.  _International Journal of Revenue Management_ 5, 4 (2011), 350–368.
  * Toch et al. (2010) Eran Toch, Norman M Sadeh, and Jason Hong. 2010.  Generating default privacy policies for online social networks.  In _CHI’10 Extended Abstracts on Human Factors in Computing Systems_. 4243–4248.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023\.  LLaMA: Open and Efficient Foundation Language Models.  arXiv:2302.13971 [cs.CL]
  * Ullah et al. (2020) Imdad Ullah, Roksana Boreli, and Salil S Kanhere. 2020\.  Privacy in targeted advertising: A survey.  _arXiv preprint arXiv:2009.06861_ (2020).
  * Wilson et al. (2016) Shomir Wilson, Florian Schaub, Aswarth Abhilash Dara, Frederick Liu, Sushain Cherivirala, Pedro Giovanni Leon, Mads Schaarup Andersen, Sebastian Zimmeck, Kanthashree Mysore Sathyendra, N Cameron Russell, et al. 2016\.  The creation and analysis of a website privacy policy corpus. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. 1330–1340.
  * Windl et al. (2022) Maximiliane Windl, Niels Henze, Albrecht Schmidt, and Sebastian S Feger. 2022.  Automating Contextual Privacy Policies: Design and Evaluation of a Production Tool for Digital Consumer Privacy Awareness. In _CHI Conference on Human Factors in Computing Systems_. 1–18.
  * Xie and Karan (2019) Wenjing Xie and Kavita Karan. 2019.  Consumers’ privacy concern and privacy protection on social network sites in the era of big data: empirical evidence from college students.  _Journal of Interactive Advertising_ 19, 3 (2019), 187–201.
  * Yang (2023) Jingfeng Yang. 2023\.  _Why did all of the public reproduction of GPT-3 fail? In which tasks should we use GPT-3.5/ChatGPT?_ https://jingfengyang.github.io/gpt Accessed: 2022-03-27.
  * Yu et al. (2018) Le Yu, Xiapu Luo, Jiachi Chen, Hao Zhou, Tao Zhang, Henry Chang, and Hareton KN Leung. 2018.  Ppchecker: Towards accessing the trustworthiness of android apps’ privacy policies.  _IEEE Transactions on Software Engineering_ 47, 2 (2018), 221–242.
  * Yu et al. (2015) Le Yu, Tao Zhang, Xiapu Luo, and Lei Xue. 2015\.  Autoppg: Towards automatic generation of privacy policy for android applications. In _Proceedings of the 5th Annual ACM CCS Workshop on Security and Privacy in Smartphones and Mobile Devices_. 39–50.
  * Zhang et al. (2022) Shikun Zhang, Yuanyuan Feng, Yaxing Yao, Lorrie Faith Cranor, and Norman Sadeh. 2022.  How Usable Are iOS App Privacy Labels?  _UMBC Faculty Collection_ (2022).
  * Zimmeck et al. (2021) Sebastian Zimmeck, Rafael Goldstein, and David Baraka. 2021\.  PrivacyFlash Pro: Automating Privacy Policy Generation for Mobile Apps.. In _NDSS_.
  * Zimmeck et al. (2019) Sebastian Zimmeck, Peter Story, Daniel Smullen, Abhilasha Ravichander, Ziqi Wang, Joel R Reidenberg, N Cameron Russell, and Norman Sadeh. 2019.  Maps: Scaling privacy compliance analysis to a million apps.  _Proc. Priv. Enhancing Tech._ 2019 (2019), 66.