# No More Trade-Offs.
GPT and Fully Informative Privacy Policies

Przemysław Pałka1, The work of P. Pałka, between September 3 and October 3,
2023, was funded by and conducted within the frame of the Visiting Researcher
program at the Department of Legal Studies, University of Bologna, Italy. The
research prior to and following this period was funded by the National Science
Center, Poland, grant no. 2022/45/B/HS5/01419.  Marco Lippi2, Research partly
funded by PNRR - M4C2 - Investimento 1.3, Partenariato Esteso PE00000013 -
“FAIR - Future Artificial Intelligence Research” - Spoke 8 “Pervasive AI”,
funded by the European Commission under the NextGeneration EU programme.
Francesca Lagioia3,4,†, European Research Council (ERC) Project “CompuLaw”
(Grant Agreement No 833647) under the European Union’s Horizon 2020 research
and innovation programme.  Rūta Liepiņa3,‡ Giovanni Sartor3,4,†,‡

( 1 Jagiellonian University<EMAIL_ADDRESS>
2 University of Modena and Reggio Emilia
3 Alma Mater Studiorum University of Bologna
4 European University Institute
)

###### Abstract

The paper reports the results of an experiment aimed at testing to what extent
ChatGPT 3.5 and 4 is able to answer questions regarding privacy policies
designed in the new format that we propose. In a world of human-only
interpreters, there was a trade-off between comprehensiveness and
comprehensibility of privacy policies, leading to the actual policies not
containing enough information for users to learn anything meaningful. Having
shown that GPT performs relatively well with the new format, we provide
experimental evidence supporting our policy suggestion, namely that the law
should require fully comprehensive privacy policies, even if this means they
become less concise.

##  1 Introduction

Imagine you just listened to a song or ordered food using an app. Imagine
asking your smartphone: “Hey Skynet, could you tell me the names of all
corporations who will know that I just listened to Italodisco, or ordered
tofu, and list all the purposes for which they will use this data?”111These
are no (longer) trivial matters; in a ML-powered inference economy [33] such
data can be proxy for estimating one’s race [22] or political convictions
[12]. As a consumer, you have the right to know this [7, 16, 21]. Such
knowledge is a precondition for choosing whether and how to use various
services [29]. And yet, in 2023, asking such a question and receiving a
straightforward answer is not possible.

The reason behind this impossibility is not (only) technological. Due to the
spread of large language models (LLMs), machines are nowadays able to read and
comprehend natural language at a level that was unthinkable only a few years
ago [27]. Therefore, we posit that the reason is chiefly that the information
necessary to answer such questions is not publicly available in privacy
policies [26]. And that the law is partly to blame for that.

The law222In this paper we focus on EU law, i.e. the GDPR, and account for the
American law. requires privacy policies to be simultaneously comprehensive,
i.e., to contain information about all the data categories processed, purposes
of use, and their recipients; and comprehensible, i.e., to be concise, written
in simple language, and easy to understand. In the world of human-only
interpreters, there is a trade-off between the two: a shorter and simpler
policy would be less comprehensive, and vice versa. The comprehensibility
requirement has been advocated for in scholarship, calling for ever more
simplification [7, 13, 18, 37].

In this paper, we challenge the conventional wisdom and take the opposite
approach. We argue that the law should focus on requiring the privacy policies
to be fully comprehensive, even if the cost of comprehensiveness is lowering
comprehensibility of such documents for individual consumers. The reason is
simple: soon anyone could engage in simplification of, or information
retrieval from, long privacy policies, as such efforts can be increasingly
automated. However, for these summaries or answers to be meaningful, they need
to be based on comprehensive information about corporate practices concerning
consumers’ personal data.

This policy suggestion assumes the capacity of AI to process fully
comprehensive privacy policies. Hence, we decided to verify that premise. In
this paper, we report some preliminary results of our first experiment, aimed
at testing the extent to which tools like ChatGPT are already capable of
answering consumer questions based on fully comprehensive privacy policies
(and not the existing privacy policies, as tested in [15, 28, 34]). To do so,
we have written a mock privacy policy of “Orderoo Inc.,” a company similar to
DoorDash, UberEats, etc., in the format we propose as fully comprehensive. The
format is presented below in Appendix C, and the full privacy policy, with all
the experiment documentation, made available on GitHub.333
https://github.com/ruutaliepina/full-privacy

We design our experiment as a proof-of-concept across four different settings,
i.e., using short and long prompts, for ChatGPT 3.5 and 4. The results are
interesting and promising, as they clearly show how LLMs are able to digest
the information contained in long documents, and to answer complex questions
that need to connect several paragraphs. Nevertheless, we note how the
provided answers sometimes contain mistakes, which poses an additional
challenge from the perspective of computer science, to what extent the output
of these models can be further controlled and post-processed in order to make
reliable information available to the end user.

The paper proceeds in five steps. First, we discuss the related work, and
second, the legal background. Third, we describe what a fully comprehensive
privacy policy could look like. Fourth, we report the results of our
preliminary experiments. Finally, we discuss implications for further research
in computer science, law, and policy.

##  2 Related Work

Empirical scholars have shown that consumers do not read privacy policies [3,
24] and, when they do, they often misunderstand them [29]. One study estimated
that reading the privacy policies of all the websites one uses would take
hundreds of hours each year [23]. As a solution to this problem, numerous
researchers have suggested making privacy policies even shorter or easier to
understand, e.g., by using the so-called “privacy nutrition labels” [7, 13,
18, 37]. However, it has been also shown that even simplified privacy policies
have a close-to-zero effect on consumers’ understanding of corporate data
practices or their rights [6].

This might suggest that the problem with privacy policies is not (only) their
form but also their content. Put simply: privacy policies often do not contain
the information necessary for consumers – even the hypothetical ones with time
and expertise to read them – to learn how exactly their data will be processed
[10]. A recent study [26] highlighted how, by using vague terms for data
categories (“we collect data about your use of our service”) or purposes of
processing (“we use your data to improve our service”), many privacy policies
leave corporations the freedom to engage in data practices that consumers
cannot reasonably expect. The same study demonstrated that many privacy
policies do not explain what data will be used for what purpose or shared with
what recipient [26].

From a technical perspective, several studies have shown that natural language
processing (NLP) tools can be used to summarize privacy policies [40], answer
users’ questions [28], or represent the contents of the policies in a user-
friendly form [15]. However, the usefulness and the practical value of such
automated representation, summarizing, and question-answering will be only as
good and specific as the information actually included in the privacy
policies. As of today, the value of this information is limited.

The recent breakthroughs in LLMs have created new opportunities for analyzing
texts through simple prompt-based queries. The LLMs use transformer
architecture and have been trained on vast amounts of general data such as
webpages, books, articles and cover a wide range of topics [36]. Legal
language is highly specialised and normally general language models perform
poorly without additional domain specific fine-tuning [8]. While these
developments are fairly recent, some studies testing the abilities of LLMs in
legal analytics have already emerged [31]. It has been shown that GPT models
(3, 3.5, and 4) can answer the US [17] and Japanese [39] bar exam questions on
par with the examinees. These tests include various reasoning tasks such as
rule application, legal opinion drafting, and multiple choice questions. As
for privacy policy analysis, Tang et al. [34] experimented with ChatGPT and
GPT4 to improve information extraction from privacy policy relevant to the
GDPR – e.g., “how and why a service provider collects user information,” “how
long user information is stored.” The reported results of their PolicyGPT show
a significant increase in retrieval metrics as compared to the older models,
which used expert annotated datasets.

##  3 Legal Requirements for Privacy Policies: Comprehensiveness vs.
Comprehensibility

The law assigns the drafters of privacy policies a difficult task. On the one
hand, it requires the privacy policies to be comprehensive, i.e., contain all
the information relevant to a consumer pondering whether to use a service. On
the other, it demands privacy policies be comprehensible, i.e., written in
simple language and easy to understand for consumers.

In the United States, online consumer privacy is governed by the so-called
“notice and choice” model [30]. Within its logic, companies who wish to
collect and use personal data should make privacy policies available to
consumers (“notice”) who, in turn, can decide whether such data practices are
acceptable to them (“choice”). Whether companies actually live up to their
promises is policed by both contract law [5] and public enforcement by the
Federal Trade Commission [32] and state Attorneys General [9]. The latter can
also issue guidelines. For example, in 2014, then-AG of California, Kamala
Harris [1], stated that privacy policies “should provide a comprehensive
overview of […] practices regarding the collection, use, sharing and
protection of personally identifiable information” and recommended that
drafters of privacy policies “use plain, straightforward language. Avoid
technical or legal jargon. Use short sentences.”444 [1], pp 9-10.

In the European Union, the omnibus General Data Protection
Regulation555Regulation (EU) 2016/679, OJ L 119, 4.5.2016, p. 1–88,
hereinafter “the GDPR.” governs online consumer privacy, among other areas of
life [38]. It requires that data controllers provide consumers with detailed
information about their data practices,666The GDPR, arts. 13-14. yet do so in
“in a concise, transparent, intelligible and easily accessible form, using
clear and plain language.”777The GDPR, art. 12.1. The guidelines endorsed by
the European Data Protection Board888Art. 29 WP, Guidelines on transparency
under Regulation 2016/679, last Revised and Adopted on 11 April 2018,
available at: https://edpb.europa.eu/our-work-tools/our-
documents/guidelines/transparency. Hereinafter “The Transparency Guidelines.”
– a body comprised of the Supervisory Authorities from all the EU Member
States – provide some examples of how to square the two:

Poor Practice Examples

> “We may use your personal data to develop new services” (as it is unclear
> what the “services” are or how the data will help develop them);
>
> “We may use your personal data for research purposes” (as it is unclear what
> kind of “‘research” this refers to); and
>
> “We may use your personal data to offer personalised services” (as it is
> unclear what the “personalisation” entails).

Good Practice Examples

> “We will retain your shopping history and use details of the products you
> have previously purchased to make suggestions to you for other products
> which we believe you will also be interested in” (it is clear what types of
> data will be processed, that the data subject will be subject to targeted
> advertisements for products, that their data will be used to enable
> this).999The Transparency Guidelines, p. 9.

This looks clear and simple when presented as an isolated example. However,
when one considers the sheer volume of data processed by online companies
nowadays, the multitude of purposes for which different categories of data are
used, or the range of entities with whom data is shared, the comprehensive
privacy policy quickly becomes inflated. In the world of human-only readers,
there is an inherent trade-off between comprehensiveness and
comprehensibility. Admittedly, corporations have business interests not to
disclose too much, like trade secrets or good PR. However, by requiring that
privacy policies were simultaneously comprehensive and comprehensible, the law
has given the corporations an excuse. They can always defend themselves by
stating “but we simply tried to make the privacy policy shorter!” Fortunately,
human readers could soon be significantly assisted by AI.

##  4 The Legal Proposal: Fully Comprehensive Privacy Policies

Our policy suggestion aims to create an informational environment where AI’s
full potential in assisting consumers can be realized. We propose that the law
should require corporations to disclose fully comprehensive privacy policies,
ideally in a standardized form, and relax the comprehensibility requirements.
Such notices can be later processed by other parties to suit particular
consumers’ informational needs, potentially via automated means.

Under the GDPR’s transparency requirements (which we consider a good start)
the unit of information that must be disclosed is an act of processing of
personal data. An act of processing means any action that a data controller
takes regarding personal data.101010GDPR art. 4.1. Thus, transparency at the
stage in which personal data are collected requires the consumers to be
informed, among others, about the following:

  1. 1.

Categories of personal data concerned (e.g., an email address);

  2. 2.

Purpose of processing for which each category of data are intended (e.g.,
issuing receipts)

  3. 3.

Legal basis for each data processing111111Art. 6.1. GDPR lists six possible
bases for processing: (a) consent, (b) contractual necessity, (c) legal
obligation, (d) subject’s vital interest, (e) public task, or (f) controller’s
legitimate interest. In the analyzed context, (a), (b), (c) and (f) are
relevant. Whenever a data controller relies on (f), they should specify what
legitimate interest they have in mind, to enable the proportionality
assessment(Article 6(1)(f)). (e.g., contractual necessity)

  4. 4.

Storage period or, when it is not possible, the criteria to determine such a
period (e.g., till account closure), and with regard to each data and purpose

  5. 5.

Recipients or at least the categories of recipients ( e.g., a cloud provider),
as well as their

    1. (a)

Role (controller or processor)

    2. (b)

Purpose of sharing (e.g., data backup)

    3. (c)

Legal basis of sharing (e.g., legitimate interest)

In our empirical work on privacy policies [10, 11, 20, 26] we have not yet
encountered a single policy that is compliant with all these requirements.
This may be due to different reasons, one of them probably being that such a
document would not meet the concise and comprehensibility requirements. Note
how a single category of data (e.g., an email address) might be used for
several purposes, e.g., sending receipts, marketing information, serve as an
identifier, etc. Each of these purposes must be described as a separate unit
of information, as they might have different legal bases.

Such information can be represented in several different ways. On the one had,
there are various machine-readable formats [19, 35] (though scholarly attempts
at constructing those seem to have culminated about a decade ago), potentially
based on ontologies [14, 25]. On the other hand, there are human-readable
formats, i.e., texts or tables. However, the advances in NLP [41] contribute
to the blurring of the machine-/human-readable distinction.

In this paper, we propose two formats for fully comprehensive privacy
policies. First, we suggest creating a spreadsheet with 8 columns: (A)
category identifier, (B) data type, (C) source, (D) purpose, (E) purpose
explanation, (F) legal basis, (G) legal basis explanation, and (H) storage
period. The number of rows should equal the number of purposes of processing
for each data type, as these might have different legal bases and be stored
for different periods. In addition, a second sheet describes the sharing of
data with other parties, in a similar form. We illustrate how this could look
like in Appendix B. The value of this approach is in its clarity and logical
structure of all the data points without the added noise.

Second, exactly the same information can be represented as solid text. Here,
each paragraph corresponds to one data type and lists the source and all the
purposes with their legal bases. Then, it lists all the recipients of personal
data and identifies their role (controller or processor), the purpose of
sharing, and its legal basis. We illustrate how this could look like in
Appendix C. We conduct the experiment on the solid text version.

##  5 The Experimental Methodology

We run an experiment as a proof-of-concept of our methodology. Our goal is to
test to what extent the consumer-facing versions of ChatGPT 3.5 and 4 may
correctly answer user questions regarding the contents of fully comprehensive
privacy policies. To do so, we created a mock privacy policy of Orderoo Inc.,
a food delivery company similar to DoorDash and Uber Eats.121212Note that the
mock policy is in the fully comprehensive format but could be longer if we
accounted for more categories of data.

We identified the following six questions to be used for the test:

  1. Q1:

What data does Orderoo process about me?

  2. Q2:

For what purposes does Orderoo use my email address?

  3. Q3:

Who does Orderoo share my geolocation with?

  4. Q4:

What types of data are processed on the basis of consent, and for what
purposes?

  5. Q5:

What data does Orderoo share with Facebook?

  6. Q6:

Does Orderoo share my data with insurers?

The three initial questions are rather straightforward and aimed to test the
model’s ability to retrieve the relevant information and connect different
informative elements. The answer to Q1 can be found at the beginning of each
paragraph of the policy, while the answers to Q2 and Q3 are contained in a
single paragraph. The last three questions are more complicated. The answers
to Q4 and Q5 are spread throughout the entire document. Q6 requires a negative
answer, i.e., the policy does not mention any data sharing with insurers. The
latter is mostly aimed to test potential hallucinations.

As noted above, for the purpose of this work, we tested ChatGPT versions 3.5
and 4, on Sep 30-Oct 2, 2023, while located in Bologna, Italy. We defined two
prompts, one short and one long, the latter aimed at mitigating hallucinations
(i.e., making answers up) [2, 4], unwarranted simplification (i.e., listing
only some answers, preceded by phrases like “for example” or “such as”), and
unwarranted text generation (i.e., adding information that the user did not
ask for, e.g., advice that one should carefully read the laws, etc.). As an
example consider the following:

> Short prompt: What data does Orderoo process about me?

> Long prompt: What data does Orderoo process about me? In answering the
> question please rely solely on the information included in the text and not
> your knowledge from other sources; please read the document carefully and
> mention everything, do not omit any information included in the text; please
> do not shorten or simplify the answers by inserting elements like “for
> example”, “including” or “such as”, please limit your answer strictly to
> what I am asking about and refrain from giving me advice or informing me
> about things I have not asked about.

Hence, we considered four settings: (1) GPT 3.5, short prompt; (2) GPT 3.5,
long prompt; (3) GPT 4 short prompt; (4) GPT 4 long prompt. For each setting,
we repeated the test five times with two independent accounts, for a total of
ten runs. Each chat began with the same statement, i.e., Hi, I will copy-paste
a document here and then ask you some questions about its contents, is that
ok?, followed by the entire mock privacy policy of Orderoo Inc. For the
purpose of evaluation, we consider the following four cases: (i) correct
answer; (ii) hallucination; (iii) false positive (when the answer mentioned
information that clearly was taken from the document, though wrong); and (iv)
false negative (when the answer failed to mention something). We also decided,
whenever the answer was not correct, to prompt GPT again with a generic “redo
it” question (i.e. Are you sure? Please try again), thus recording whether
this led to improved performance or not. All the answers were manually
checked, and the whole set of conversations is available at the aforementioned
repository.

Our initial hypotheses were the following:

  1. H1:

There will be a problem with hallucination, especially when asking about
Facebook (as GPT “knows” a lot about it from other sources)

  2. H2:

There should be no problem with false positives, and false negatives should
occur sporadically, due to unwarranted simplification

  3. H3:

GPT-4 will do better than GPT-3.5 on all the questions

  4. H4:

Longer prompts will lead to better answers than shorter ones

##  6 Experimental Results and Discussion

| Q1 | Q2 | Q3 | Q4 | Q5 | Q6
---|---|---|---|---|---|---
GPT-3.5 (S) | 10 | 10 | 10 | 0 | 0 | 10
GPT-3.5 (L) | 10 | 10 | 10 | 0 | 0 | 10
GPT-4 (S) | 10 | 10 | 7 | 4 | 10 | 10
GPT-4 (L) | 10 | 10 | 7 | 2 | 10 | 10
Table 1: Results obtained by GPT-3.5 and GPT-4, with short (S) and long (L)
prompts, respectively. We report the number of correct answers out of 10 runs,
obtained with two different accounts (5 runs each). The complete list of
results is reported in Appendix A.

Table 1 reports the results obtained across the different settings. Notably,
in none of the settings did GPT manage to answer all six questions correctly.
However, several interesting observations can be made based on these results.
Significantly, results were not always consistent across the runs. For Q3 and
Q4, in fact, one of the two accounts obtained different answers when
questioning GPT-4 both with short and long prompts.

Regarding H1, none of the experiments reported hallucination. This suggests
that, when asked questions about a document uploaded by the user, the
hallucination problem is less pertinent than with open-ended questions that
rely on GPT’s “own knowledge.” Hence, H1 has been falsified.

Concerning H2 and H3, the only instances of false positives occurred in Q3,
with GPT-4 (though only for one account). The system listed one additional
entity with whom Orderoo allegedly shares geolocation, namely “Cloud711”. This
is the name of a company made up by us, mentioned repeatedly in the document,
so this was not a hallucination. Moreover, every data type other than
geolocalization is shared by Orderoo with Cloud711, according to the policy.
Thus, in this case, the system just wrongly interpreted the text. All other
runs got this correct. False negatives, instead, consistently appeared in Q3
and Q4 for GPT-3.5 (all 10 runs, both short and long prompts) and also, though
less frequently, for GPT-4 (6 and 8 cases for short and long prompts in Q4,
respectively; only once in Q3). Therefore, given all this evidence, also H2
and H3 have been partly falsified.

Remarkably, all runs correctly answered Q1 and Q2 and Q6 (where a negative
answer was correct). This suggests that having even longer privacy policies,
where additional sections are organized not only around data categories but
also around purposes and legal bases, could help. However, this would mean
that the documents will be significantly longer (more processing power needed)
and come with the additional need to ensure consistency throughout the
document. Surprisingly, the longer prompts had no effect on GPT 3.5. For GPT
4, instead, the results were interesting: the additional instruction led to
more false negatives. In this sense, H4 has been falsified.

The good performance of GPT-4 when answering questions about the fully
comprehensive privacy policy (if we consider the majority of cases, 5 correct
answers out of 6 questions) suggests that our policy proposal should be
seriously considered by lawmakers or even privacy-sensitive corporations
acting on their own motion. This proposal challenges the conventional wisdom
that privacy policies should be made shorter rather than longer and simpler
rather than more complex [7, 13, 18, 37]. However, for this reform to indeed
meaningfully affect the position of consumers, further research in computer
science is needed.

Some limitations with the use of pretrained systems should be noted.
Currently, ChatGPT free version limits the input text to approximately 3000
words. When arguing for more comprehensive privacy policies, the word limit
could create problems. For example, during the experiments, we also tested the
performance of the system by entering the privacy policy splitting it over two
messages. The results were significantly worse compared to the single input –
the answers were based on general information about privacy policies, not the
specific mock policy we provided. Moreover, GPT architecture remains opaque
and users are given limited control over the results they receive.

##  7 Conclusion and Future Work

In this paper, we suggested that making privacy policies fully comprehensive
is a pro-consumer move, provided that automated means for question answering
are available. We created a mock privacy policy, demonstrating what format we
have in mind, and tested the performance of Chat GPT 3.5 and 4, using both
short and long prompts. The novelty of this experiment lays in the creation
and use of a synthetic privacy policy to test the abilities of the new LLMs.
Even though the results where not perfect, the good performance of GPT 4
suggests that this is a promising path.

As part of experimental work, we plan to expand the mock privacy policy and
the number of documents to be assessed, test different LLMs, including open
source LLMs; explore the use of API to create further controls for the quality
of the answers, as well as, link the answers to the relevant portions of the
text.

## References

  * [1] Kamala D. Harris (Attorney General California Department of Justice).  Making your privacy practices public: Recommendations on developing a meaningful privacy policy.  Accessible at: https://oag.ca.gov/news/press-releases/attorney-general-kamala-d-harris-issues-guide-privacy-policies-and-do-not-track, 2014\.
  * [2] Razvan Azamfirei, Sapna R Kudchadkar, and James Fackler.  Large language models and the perils of their hallucinations.  Critical Care, 27(1):1–2, 2023.
  * [3] Yannis Bakos, Florencia Marotta-Wurgler, and David R Trossen.  Does anyone read the fine print? consumer attention to standard-form contracts.  The Journal of Legal Studies, 43(1):1–35, 2014.
  * [4] Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al.  A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity.  arXiv preprint arXiv:2302.04023, 2023.
  * [5] Oren Bar-Gill, Omri Ben-Shahar, and Florencia Marotta-Wurgler.  Searching for the common law: The quantitative approach of the restatement of consumer contracts.  The University of Chicago Law Review, pages 7–35, 2017.
  * [6] Omri Ben-Shahar and Adam Chilton.  Simplification of privacy disclosures: an experimental test.  The Journal of Legal Studies, 45(S2):S41–S67, 2016.
  * [7] Ryan Calo.  Against notice skepticism in privacy (and elsewhere).  Notre Dame L. Rev., 87:1027, 2011.
  * [8] Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Martin Katz, and Nikolaos Aletras.  LexGLUE: A benchmark dataset for legal language understanding in english.  arXiv preprint arXiv:2110.00976, 2021.
  * [9] Danielle Keats Citron.  The privacy policymaking of state attorneys general.  Notre Dame L. Rev., 92:747, 2016.
  * [10] Giuseppe Contissa, Koen Docter, Francesca Lagioia, Marco Lippi, Hans-W Micklitz, Przemysław Pałka, Giovanni Sartor, and Paolo Torroni.  Claudette meets GDPR: Automating the evaluation of privacy policies using artificial intelligence.  Available at SSRN 3208596, 2018.
  * [11] Giuseppe Contissa, Koen Docter, Francesca Lagioia, Marco Lippi, Hans-Wolfgang Micklitz, Przemyslaw Palka, Giovanni Sartor, Paolo Torroni, et al.  Automated processing of privacy policies under the EU General Data Protection Regulation.  Frontiers in Artificial Intelligence and Applications, 313:51–60, 2018.
  * [12] Daniel DellaPosta, Yongren Shi, and Michael Macy.  Why do liberals drink lattes?  American Journal of Sociology, 120(5):1473–1511, 2015.
  * [13] Pardis Emami-Naeini, Janarth Dheenadhayalan, Yuvraj Agarwal, and Lorrie Faith Cranor.  An informative security and privacy “nutrition” label for internet of things devices.  IEEE Security & Privacy, 20(2):31–39, 2021.
  * [14] Mohamad Gharib, Paolo Giorgini, and John Mylopoulos.  An ontology for privacy requirements via a systematic literature review.  Journal on Data Semantics, 9:123–149, 2020.
  * [15] Hamza Harkous, Kassem Fawaz, Rémi Lebret, Florian Schaub, Kang G Shin, and Karl Aberer.  Polisis: Automated analysis and presentation of privacy policies using deep learning.  In 27th USENIX Security Symposium (USENIX Security 18), pages 531–548, 2018.
  * [16] Margot E Kaminski.  The right to explanation, explained.  Berkeley Technology Law Journal, 34(1):189–218, 2019.
  * [17] Daniel Martin Katz, Michael James Bommarito, Shang Gao, and Pablo Arredondo.  GPT-4 passes the bar exam.  Available at SSRN 4389233, 2023.
  * [18] Patrick Gage Kelley, Joanna Bresee, Lorrie Faith Cranor, and Robert W Reeder.  A “nutrition label” for privacy.  In Proceedings of the 5th Symposium on Usable Privacy and Security, pages 1–12, 2009.
  * [19] Ponnurangam Kumaraguru, Lorrie Cranor, Jorge Lobo, and Seraphin Calo.  A survey of privacy policy languages.  In Workshop on Usable IT Security Management (USM 07): Proceedings of the 3rd Symposium on Usable Privacy and Security, ACM, 2007.
  * [20] R Liepina, H-W Micklitz, et al.  GDPR privacy policies in CLAUDETTE: Challenges of omission, context and multilingualism.  In CEUR Workshop Proceedings, volume 2385. CEUR-WS, 2019.
  * [21] René LP Mahieu, Jef Ausloos, et al.  Harnessing the collective potential of gdpr access rights: towards an ecology of transparency.  Internet Policy Review, 6, 2020.
  * [22] Shantal R Marshall and Laura P Naumann.  What’s your favorite music? Music preferences cue racial identity.  Journal of Research in Personality, 76:74–91, 2018.
  * [23] Aleecia M McDonald and Lorrie Faith Cranor.  The cost of reading privacy policies.  Isjlp, 4:543, 2008.
  * [24] Jonathan A Obar and Anne Oeldorf-Hirsch.  The biggest lie on the internet: Ignoring the privacy policies and terms of service policies of social networking services.  Information, Communication & Society, 23(1):128–147, 2020.
  * [25] Monica Palmirani, Michele Martoni, Arianna Rossi, Cesare Bartolini, and Livio Robaldo.  Pronto: Privacy ontology for legal reasoning.  In Electronic Government and the Information Systems Perspective: 7th International Conference, EGOVIS 2018, Regensburg, Germany, September 3–5, 2018, Proceedings 7, pages 139–152. Springer, 2018.
  * [26] Przemysław Pałka, Radosław Pałosz, and Katarzyna Wiśniewska.  Annotated privacy policies of 100 online platforms.  Mendeley Data, V1, 2023.
  * [27] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  OpenAI blog, 1(8):9, 2019.
  * [28] Abhilasha Ravichander, Alan W Black, Shomir Wilson, Thomas Norton, and Norman Sadeh.  Question answering for privacy policies: Combining computational and legal perspectives.  arXiv preprint arXiv:1911.00841, 2019.
  * [29] Joel R Reidenberg, Travis Breaux, Lorrie Faith Cranor, Brian French, Amanda Grannis, James T Graves, Fei Liu, Aleecia McDonald, Thomas B Norton, Rohan Ramanath, et al.  Disagreeable privacy policies: Mismatches between meaning and users’ understanding.  Berkeley Tech. LJ, 30:39, 2015.
  * [30] Neil Richards and Woodrow Hartzog.  The pathologies of digital consent.  Wash. UL Rev., 96:1461, 2018.
  * [31] Jaromir Savelka, Kevin D Ashley, Morgan A Gray, Hannes Westermann, and Huihui Xu.  Can GPT-4 support analysis of textual data in tasks requiring highly specialized domain expertise?  arXiv preprint arXiv:2306.13906, 2023.
  * [32] Daniel J Solove and Woodrow Hartzog.  The ftc and the new common law of privacy.  Colum. L. Rev., 114:583, 2014.
  * [33] Alicia Solow-Niederman.  Information privacy and the inference economy.  Nw. UL Rev., 117:357, 2022.
  * [34] Chenhao Tang, Zhengliang Liu, Chong Ma, Zihao Wu, Yiwei Li, Wei Liu, Dajiang Zhu, Quanzheng Li, Xiang Li, Tianming Liu, et al.  PolicyGPT: Automated analysis of privacy policies with large language models.  arXiv preprint arXiv:2309.10238, 2023.
  * [35] Inger Anne Tøndel and Åsmund Ahlmann Nyre.  Towards a similarity metric for comparing machine-readable privacy policies.  In Open Problems in Network Security: IFIP WG 11.4 International Workshop, iNetSec 2011, Lucerne, Switzerland, June 9, 2011, Revised Selected Papers, pages 89–103. Springer, 2012.
  * [36] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [37] Ari Ezra Waldman.  Privacy, notice, and design.  Stan. Tech. L. Rev., 21:74, 2018.
  * [38] Karen Yeung and Lee A Bygrave.  Demystifying the modernized european data protection regime: Cross-disciplinary insights from legal and regulatory governance scholarship.  Regulation & Governance, 16(1):137–155, 2022.
  * [39] Fangyi Yu, Lee Quartey, and Frank Schilder.  Exploring the effectiveness of prompt engineering for legal reasoning tasks.  In Findings of the Association for Computational Linguistics: ACL 2023, pages 13582–13596, 2023.
  * [40] Razieh Nokhbeh Zaeem, Rachel L German, and K Suzanne Barber.  Privacycheck: Automatic summarization of privacy policies using data mining.  ACM Transactions on Internet Technology (TOIT), 18(4):1–18, 2018\.
  * [41] Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun.  How does NLP benefit legal system: A summary of legal artificial intelligence.  arXiv preprint arXiv:2004.12158, 2020.

## APPENDIX

##  Appendix A Additional details on experimental results

We hereby report the complete list of the results obtained with the 10
different runs (5 for each of two accounts) across the four considered
settings.

| Account #1 | Account #2
---|---|---
| $r_{1}$ | $r_{2}$ | $r_{3}$ | $r_{4}$ | $r_{5}$ | $r_{1}$ | $r_{2}$ | $r_{3}$ | $r_{4}$ | $r_{5}$
Q1 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
Q2 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
Q3 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
Q4 | FN | FN | FN | FN | FN | FN | FN | FN | FN | FN
Q5 | FN | FN | FN | FN | FN | FN | FN | FN | FN | FN
Q6 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
Table 2: Results obtained in Setting 1, i.e., with GPT-3.5 with short prompts on the 10 overall runs (5 for two accounts each). A checkmark indicates a correct answer. FN indicates that the answer contains a false negative. | Account #1 | Account #2
---|---|---
| $r_{1}$ | $r_{2}$ | $r_{3}$ | $r_{4}$ | $r_{5}$ | $r_{1}$ | $r_{2}$ | $r_{3}$ | $r_{4}$ | $r_{5}$
Q1 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
Q2 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
Q3 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
Q4 | FN | FN | FN | FN | FN | FN | FN | FN | FN | FN
Q5 | FN | FN | FN | FN | FN | FN | FN | FN | FN | FN
Q6 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
Table 3: Results obtained in Setting 2, i.e., with GPT-3.5 with long prompts on the 10 overall runs (5 for two accounts each). A checkmark indicates a correct answer. FN indicates that the answer contains a false negative. | Account #1 | Account #2
---|---|---
| $r_{1}$ | $r_{2}$ | $r_{3}$ | $r_{4}$ | $r_{5}$ | $r_{1}$ | $r_{2}$ | $r_{3}$ | $r_{4}$ | $r_{5}$
Q1 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
Q2 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
Q3 | ✓ | ✓ | ✓ | ✓ | ✓ | FP | ✓ | ✓∗ | FP∗ | ✓
Q4 | FN | FN | FN | FN | FN | ✓ | ✓ | FN | ✓ | ✓
Q5 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
Q6 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
Table 4: Results obtained in Setting 3, i.e., with GPT-4 with short prompts on the 10 overall runs (5 for two accounts each). A checkmark indicates a correct answer. FN and FP indicate that the answer contains a false negative or false positive, respectively. ∗ indicates that the answer changed after the system was asked to rethink the answer.  | Account #1 | Account #2
---|---|---
| $r_{1}$ | $r_{2}$ | $r_{3}$ | $r_{4}$ | $r_{5}$ | $r_{1}$ | $r_{2}$ | $r_{3}$ | $r_{4}$ | $r_{5}$
Q1 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
Q2 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
Q3 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | FP | FP | FP | ✓
Q4 | FN | FN | FN | FN | FN | FN | FN | FN | ✓ | ✓
Q5 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
Q6 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
Table 5: Results obtained in Setting 4, i.e., with GPT-4 with short prompts on
the 10 overall runs (5 for two accounts each). A checkmark indicates a correct
answer. FN and FP indicate that the answer contains a false negative or false
positive, respectively. ∗ indicates that the answer changed after the system
was asked to rethink the answer.

##  Appendix B Mock privacy policy sample (spreadsheet)

Figure 1 presents a sample of the mock privacy policy. It includes two data
types (i.e., email address, name and surname) their specified sources,
purposes for processing, explanations of these purposes, legal basis and
further explanations, and information about storage.

Figure 1: Mock privacy policy

##  Appendix C A sample of the mock privacy policy (solid text)

THIS IS A MOCK POLICY OF “ORDEROO INC.” A COMPANY SIMILAR TO DELIVEROO/
DOORDASH /JUST EAT/ UBER EATS etc.

ORDEROO INC. PRIVACY POLICY

We process your personal data in the following way:

1\. Your email address. You provide us with your email address when
registering for the service. We use your email address for the following
purposes: unique identifier, it serves as a unique identifier allowing you to
set up and log in to your account (contractual necessity); account access, to
let you reset your password if you forget it (contractual necessity);
transaction-related-communication, to send you receipts of your orders (legal
obligation: to issue receipts, according to the Receipts Act); distribution of
own advertising, to send you advertisements of our own services, new
functionalities or new order options (legitimate interest: informing the
consumers about the available offers and features, and promoting them);
distribution of third-party marketing, to send you advertisements of vendors
selling their products on our site (legitimate interest: to subsidize the
price of the service with payments from the vendors we promote); tracking
transaction history, we keep it as a part of your order history data in case
it becomes necessary to reveal it to investigative authorities (legal
obligation: Accounting Act and Code of Criminal Procedure); we use the domain
name part of your email when profiling (see the separate section at the bottom
of the document). We share your email address with Cloud711 (processor), for
the purpose of data storage and backup, i.e., storing our IT operations on
their servers (legitimate interest: lowering the cost of operation and keeping
the data safe); Microsoft (processor) for the purpose of facilitating
communication, i.e., sending our own emails (legitimate interest in
outsourcing the operation of email servers and protocols); CoolAccountants
(processor) for the purpose of accounting, i.e., reviewing our financial
records and keeping them in order (legal obligation: Accounting Act);
FraudDetectors (processor) for the purpose of fraud detection (legitimate
interest: not becoming a victim of fraud). We do not share your email address
with recipients choosing their own purposes of processing (controllers). We
store your email for as long as you’re using our services, i.e., until you
delete your account, PLUS THREE MONTHS. For the purposes required by the
Receipts Act, Accounting Act, and the Code of Criminal Procedure, we store
your email address for five years after your last transaction.