# A Large-scale Empirical Study of Online Automated Privacy Policy Generators
for Mobile Apps

Shidong Pan<EMAIL_ADDRESS>Australian National University & CSIRO’s
Data61CanberraACTAustralia ,  Dawen Zhang<EMAIL_ADDRESS>CSIRO’s
Data61 & Australian National UniversitySydneyNSWAustralia ,  Mark Staples
<EMAIL_ADDRESS>CSIRO’s Data61SydneyNSWAustralia ,  Zhenchang
Xing<EMAIL_ADDRESS>CSIRO’s Data61 & Australian National
UniversityCanberraACTAustralia ,  Jieshan Chen<EMAIL_ADDRESS>CSIRO’s Data61SydneyNSWAustralia ,  Xiwei Xu<EMAIL_ADDRESS>CSIRO’s
Data61SydneyNSWAustralia and  Thong Hoang<EMAIL_ADDRESS>CSIRO’s
Data61SydneyNSWAustralia

###### Abstract.

Mobile phones and apps have become a ubiquitous part of digital life. There is
a large variety and volume of personal data sent to and used by mobile apps,
leading to various privacy issues. Privacy regulations protect and promote the
privacy of individuals by requiring mobile apps to provide a privacy policy
that explains what personal information is gathered and how these apps process
and safely keep this information. However, developers often do not have
sufficient legal knowledge to create such privacy policies. Online Automated
Privacy Policy Generators (APPGs) can create privacy policies, but their
quality and other characteristics can vary. In this paper, we conduct the
first large-scale, comprehensive empirical study of APPGs for mobile apps.
Specifically, we collected and analyzed 46,472 Android app privacy policies
from the Google Play Store and systematically evaluated 10 APPGs on multiple
dimensions. We reported analyses on how widely APPGs are used and whether
policies are consistent with app permissions. We found that nearly 20.1% of
privacy policies could be generated by APPGs and summarized the potential and
limitations of APPGs.

Privacy policy, Automated privacy policy generators, Mobile apps, GDPR

††copyright: none

##  1\. Introduction

Mobile phones and apps are now a ubiquitous part of digital life. There is a
large variety and volume of data collected and used by mobile apps, which
inevitably brings many privacy issues. Privacy policies inform users about
what, why and how their personal data are collected and used, and have become
an important element of responsible technology in mobile app ecosystems. They
form part of legal agreements for apps and services, and are required under
regulation in many jurisdictions, such as European General Data Protection
Regulation (GDPR) (GDP, 2018), California Consumer Privacy Act (CCPA) (CCP,
2018), Brazilian Law of General Data Protection (LGPD) (LGP, 2018), Australian
Privacy Principles (APP) (APP, 2019), and Chinese Personal Information
Protection Law (PIPL) (PIP, 2021). For example, APP Article 1.3(APP, 2019)
imposes an obligation on organisational entities to “have a clearly expressed
and up-to-date APP Privacy Policy about how the entity manages personal
information.” Privacy policies must be consistent with the data collected and
functions provided by service providers, and inadequate policies can create
significant business and legal problems.

Developing privacy policies is a complex process, demanding both the knowledge
of app features and corresponding legal requirements. Except for a few large
companies, most app developers do not have legal support and struggle to
prepare accurate privacy policies (Balebako and Cranor, 2014; Balebako et al.,
2014; Schaub et al., 2015; Li et al., 2022). To develop privacy policies,
developers may ad-hoc copy-paste-modify existing privacy policies. As part of
this big picture, app development by non-professional developers is fast
growing, especially with the trending no-code/low-code automated app
development tools (Oltrogge et al., 2018). To fill in the market need, Online
Automated Privacy Policy Generators (APPGs) can provide more automated
solutions and more systematic support for developers to create privacy
policies for their apps, rather than ad-hoc reusing. Most APPGs are
questionnaire-based tools, which work by asking app developers a series of
privacy-related questions about the app and using those answers to generate
privacy policies. APPGs can create privacy policies, but their quality and
other characteristics can vary and are not yet deeply understood.

While larger companies have greater resources and legal expertise to create
high quality privacy policies for their apps, most (citizen) developers are
less capable and have fewer legal resources (Balebako and Cranor, 2014; Li et
al., 2018). As we observe in this paper, many apps fail to provide a privacy
policy (Section 2.2.1), perfunctorily provide only a low quality privacy
policy (Section 2.2.2), or provide a privacy policy not in local language
(Section 2.2.3). APPGs are becoming an increasingly popular solution used by
developers, but developers are prone to be unaware of hidden issues in APPGs
(Zimmeck et al., 2021). Potential design flaws may reflect in the policies,
amplify vulnerabilities, violate users’ trust assumptions, and ultimately harm
end-users. So, assessing the capability and limitations of APPGs is necessary.
To better understand the scale of potential problems and their broader impact,
it is worthwhile to conduct a market penetration analysis on mainstream APPGs.

Many previous studies focus on privacy policies for websites (Levy and Gutwin,
2005; Wilson et al., 2016; Amos et al., 2021; Nokhbeh Zaeem and Barber, 2021;
Windl et al., 2022), for mobile apps (Yu et al., 2018a; Zimmeck et al., 2019;
Liu et al., 2021; Kumar et al., 2022), and some literature mixes up (Poplavska
et al., 2020; Sun and Xue, 2020). However, we have observed an important
difference between privacy policies for websites and mobile apps, which is
that mobile apps highly depend on device permissions to function normally.
Generally, we find that users are more vigilant when directly providing
personal information to either websites or mobile apps, but underestimate the
impact brought by one-time permission requests, which means that once a user
grants the request, apps can continuously collect critical personal
information without further user approval or consent. Therefore, it is
significant to examine whether developers correctly display the needed device
permissions in the generated privacy policies with APPGs for mobile apps.

Figure 1. Overview of data flow in the methodology for our empirical study.

Above all, to evaluate current APPGs and their use, we investigate the
following research questions:

RQ1: What are APPGs for mobile apps and what are the differences between them?
(Section 3)

RQ2: How many mobile apps’ privacy policies could be generated by APPGs? Which
APPGs could be more popular? (Section 4.2) and

RQ3: To what extent can developers correctly use APPGs to display the needed
device permissions in the generated privacy policy? (Section 4.3)

An overview of our methodology is shown in Fig. 1. We first collected a large-
scale dataset of privacy policies for mobile apps. Based on a random sample of
apps from the Google Play Store, we found that 15.7% of apps do not provide a
privacy policy link. We then successfully downloaded 46,472 privacy policies
(Crawled Privacy Policy Collection) and found that 22.3% of them are low
quality privacy policies. Moreover, 20.5% of these privacy policies fail to
provide a privacy policy in English, the first language in the target market.
Then we collected an initial set of APPGs by Google search. After removing
duplicates and filtering out irrelevant results, we successfully identified 10
APPGs for mobile apps as our research objects. We conducted an assessment of
these 10 APPGs on multiple dimensions, especially the recognition of the
extent of data use and the readability of the generated privacy policies.

We then generated 30 privacy policies (Generated Privacy Policy Collection)
using the 10 APPGs for three synthetic apps and conducted an analysis on
market penetration and permissions coverage. Our results show that many
(20.1%) app developers highly likely use APPGs to generate their privacy
policies. In addition, we found a significant portion of dangerous permissions
is missing in the generated policies of all APPGs. Our results also suggest
that inappropriate APPGs selection will limit developers’ ability to include
essential device permissions, and questionnaire-mode APPGs can help users
better incorporate these essential device permissions.

Contributions. To our knowledge, this is the first large-scale, comprehensive
empirical study on automated privacy policy generators for mobile apps.
Specifically, we make the following contributions:

  * •

We conducted a systematic analysis of the existence and quality of privacy
policies for 99,149 apps at the Google Play Store and collected a large-scale
privacy policy dataset containing 46,472 privacy policies.111Our dataset is
available at: https://github.com/APPG-APPG/APPG from all app categories.

  * •

We carried out a comprehensive assessment of 10 APPGs for mobile apps on
multiple dimensions, especially the recognition of the extent of data use and
readability of generated privacy policies. We also revealed the market
penetration and practicability of APPGs on device permissions. Our
observations, findings, and insights will benefit all stakeholders beyond just
app developers, including APPG providers and privacy regulators from the APPG
designing perspective and the legal perspective.

##  2\. Status Quo of Privacy Policy for Mobile Apps

Privacy policy plays an essential role in mobile apps ecosystems. It is not
only required by regulations in some jurisdictions, but is also increasingly
demanded by mobile app users or marketplaces. According to the App Privacy
section on the Apple App Store developers page (app, 2022a), a privacy policy
is required when submitting new apps or app updates.

“By adding the following links on your product page, you can help users easily
access your app’s privacy policy and manage their data in your app. Privacy
Policy (Required): The URL to your publicly accessible privacy policy.”

In the Google Play Console Help Center(goo, 2022a), to prepare an app for
review, developers are asked to add a privacy policy that includes involved
data practices:

“Adding a privacy policy to your app’s store listing helps provide
transparency about how you treat sensitive user and device data.”

Although many principles regulate the existence and quality of privacy
policies, we still discovered some critical problems in the mobile apps
market, as discussed in Section 2.2.

###  2.1. Crawled Privacy Policy Dataset

To better understand the status and quality of existing privacy policies, we
need a large-scale privacy policy dataset. However, existing datasets suffer
from various problems.

The differences between datasets for previous research and our research goals.
Existing datasets are either not focused on mobile apps, biased, small, or
outdated. First, some prior research focuses on privacy policies across all
platforms, including websites and mobile apps (Amos et al., 2021; Sun and Xue,
2020; Liu et al., 2018), while we only target mobile apps. Since they did not
include metadata (e.g., platforms) of the collected policies, we cannot filter
them out from other platforms. Second, some research only focus on apps of
certain groups and their collected datasets are not representative enough for
apps in the market. For example, one study (Kumar et al., 2022) manually
collected 5,684 “globally popular apps.” This selection criterion leads to two
critical problems: a) The top overall popular apps appear in a limited set of
categories, such as Social and Finance, which can drive out apps from less
popular categories such as Parenting or Libraries & Demo. b) These globally
popular apps do not distribute evenly across various app categories, which may
cause potential biased and unrepresentative findings. Third, some existing
datasets are small and can contain potential bias. For example, APP-350
(Zimmeck et al., 2019) only analysed 350 privacy policies, and another study
(Liu et al., 2021) conducted fine-grained analysis on only 304 privacy
policies.

Finally, due to the rapid iteration of apps, apps’ privacy policies, and
APPGs, some prior datasets can quickly become outdated.

As a summary, to make our findings comprehensive, generalizable, and up-to-
date, we need to collect the latest large-scale dataset that can assemble apps
of different popularity and from various app categories.

Google Play Store. It has been widely discussed and recognized that Android
mobile apps are more transparent and friendly to academic research compared to
iOS mobile apps (Habchi et al., 2017; Kollnig et al., 2021). There are many
app markets for the Android platform, but Google Play is the largest and the
most accessible app market, with over two million apps, according to AppBrain
(app, 2022). Thus, we only focus on the Google Play Store. Particularly, we
noticed some apps are specially designed for Android smartwatches instead of
mobile phones; therefore, we excluded apps belonging to categories named Watch
apps and Watch faces.

Our dataset. Above all, we collected a new large-scale privacy policy dataset
from existing apps on the Google Play Store. We collected app metadata from
AndroZoo (Allix et al., 2016), which is a large-scale and growing Android app
collection extracted from multiple sources, including the Google Play Store.
We removed duplicates and apps from other app markets and then randomly
sampled 268,500 apps as our initial app dataset, which is around 10% of the
whole app population. From this initial sample, we observed that some apps are
either invalid or not usable. These were mainly dummy apps or student apps
with a package size normally less than 10 KB, so we removed invalid apps like
these. We excluded unavailable apps that returned error messages on their
Google Play Store page. Apps can be unavailable for many reasons including
geodifferences (Kumar et al., 2022) caused by government censoring, or being
removed by Google because of disruptive adware, malware, restricted content,
or other reasons. After these exclusions, we were left with 99,194 usable apps
and their metadata. We then used “google play scraper” (goo, 2022b) to obtain
app information including privacy policy links, app categories showed,
required device permissions, etc. Based on the available privacy policy links
shown on app’s homepages, we further utilize the BeautifulSoup (bea, 2022) and
Selenium (Sel, 2022) python packages to download those websites. Eventually,
we obtained 46,472 privacy policies, and we refer to this privacy policy
dataset as “Crawled Privacy Policy Collection.” We conducted the above data
gathering process in March 2022.

###  2.2. Observations on Status Quo

In this section, we report an analysis of app privacy policies and identify
three common problems in existing apps: missing/unavailable privacy policies,
low quality privacy policies, and language non-localization problems. Although
similar problems were discussed in several previous studies (Zimmeck et al.,
2019; Kumar et al., 2022; Li et al., 2022), considering the rapid development
of markets, we believe it is worthwhile to revisit those critical issues.

####  2.2.1. Missing Links and Unavailable Privacy Policies

We observed that 15.7% (15,572/99,194) of apps do not provide a privacy policy
link on their Google Play Store homepage. One prior study (Zimmeck et al.,
2019) reported that this statistic was 49.1% in 2019. Seemingly, as more
regulation principles were launched, and more law enforcement efforts were
practiced, fewer apps provided a privacy policy. However, we found that 37.5%
(37,150/99,194) of privacy policy links lead to unavailable websites with
error messages such as “403 Forbidden” or “404 Not Found.” This problem could
be caused by potential reasons such as deliberately providing a dummy link, an
outage, or the removal of the website server. Hence, in addition to simply
requiring app developers to provide the privacy policy, the mobile app market
also needs to frequently and regularly verify the validity of provided privacy
policy links.

####  2.2.2. Low Quality Privacy Policies

Several previous studies (Liu et al., 2021; Zimmeck et al., 2019) reported
that it was not rare that some apps only provide a low quality privacy policy,
and we spotted a similar issue in our dataset. Firstly, we noticed that some
privacy policies do not contain meaningful privacy-related context, or are
just dummy websites. For example, “Bway App” is a free app that provides
football results, football statistics, trends, and match result predictions,
with over 10,000 installs; however, their privacy policy website (Bwa, 2022)
does not contain any content. Moreover, the file size of the crawled privacy
policy website is close to 0 KB. In addition, some are apparent “homemade”
privacy policies that are too general to include essential data practices. For
instance, “Easy Communication” is designed to help people with autism,
cerebral palsy, dyslexia, intellectual disability, and other special needs to
communicate easily, with over 1,000 installs. Its privacy policy (Eas, 2022)
only has 156 words, and ambiguously mentions permissions without any
elaborations. Based on our observations and previous work, we empirically set
the file size threshold as 2 KB (Liu et al., 2021), and the document length
threshold as 200 words (Kumar et al., 2022). If a crawled privacy policy does
not meet both criteria, it will be regarded as a low quality privacy policy.
We identified 22.3% (10,375/46,472) low quality privacy policies in total.

####  2.2.3. Language non-Localization Problem

Language localization is necessary to promote apps in a global market.
Although app UI and content may be translated, app developers may neglect the
privacy policy. We employed the Python package langdetect (lan, 2022) to
detect the language of crawled privacy policy documents and found 20.5%
(9,523/46,472) of apps do not provide an English privacy policy, although
these apps were released in markets where the primary language is English. We
observed this problem even with top apps from some large companies. For the
privacy policies of the top 1,000 most installed apps (all $>$ 500k installs)
in our dataset, there are still around 14.1% of them that do not provide an
English privacy policy. APPGs may be a solution to language non-localization
problems. Although currently only APPG #1 Iubenda supports multilingual
generation, some APPGs (#4, #7, and #9) have included placeholders for future
options for language selection.

##  3\. Online Automated Privacy Policy Generators for Mobile Apps (RQ1)

To answer RQ1, this section describes what are APPGs for mobile apps and what
are the differences between them? We first identified 10 Online APPGs for
mobile apps and presented an analysis of various characteristics.
Specifically, we focused on their recognition of the range of possible data
uses in apps and the readability analysis of their generated privacy policy
documents.

###  3.1. Collecting Online APPGs for Mobile Apps

To better understand the current status of APPGs, we identified popular APPGs
for mobile apps by using the Google search engine and exploring crowd-
knowledge platforms. In detail, we employed the Google search engine as the
tool to broadly search APPGs for mobile apps. We acted as a hypothetical
citizen developer who is trying to generate a privacy policy for their app. We
created an initial set of search terms, including “Apps Privacy Policy
Generators”, “Apps Privacy Policy Generators Online”, “Privacy Policy
Automated Generation”, “Google Play Store Privacy Policy Generator”, “GDPR
Policy Generators”. For each search keyword, we evaluated all entries on the
first result page, including advertised entries. In addition to direct results
from Google search, we also identified APPGs by using our Google search terms
again in app developer crowd-knowledge platforms including Stack Overflow
(sta, 2022), GeeksforGeeks (gee, 2022), and GitHub (git, 2022). To evaluate
search results, we manually identified and read highly related threads, and
collected the mentioned APPGs. After collecting an initial set of APPGs and
removing duplicates, we filtered out irrelevant APPGs based on the following
two criteria. First, we removed APPGs that only support privacy policy
generation for websites. Second, we also removed APPGs that do not actually
provide their proposed functions. For example, while some APPGs claim that
they are able to generate privacy policies for mobile apps, they do not
actually provide such a service. The process above identified 10 APPGs as
shown in Table 1. “Freeware” represents whether a generator needs monetary
investment to use. “Free” means the APPG provides a free version and “Paid”
means the APPG provides a premium version that unlocks more features.
“Registration” indicates whether a user needs to register an account to use
the APPG. All APPGs we collected are based on question answering and template
completion (‘boilerplate’), however, they have various different “Modes” to
handle information interactions with users. “User interface”(UI) mode means
that users need to select the relative data practices through an integrated
UI, then answer a few other questions to complete the privacy policy.
“Questionnaire” mode only requires users to complete a long questionnaire that
covers all potential data practices, and a privacy policy will be generated
depending on the answers. “Document” mode is common in legal consultation
websites. Users need to purchase access to a privacy policy document, then
answer a series of questions to complete the template.

Table 1. APPGs for mobile apps identified in this study.

# | Name | Freeware | Registration | Mode
---|---|---|---|---
1 | Iubenda (Iub, 2022) | Free&Paid | Required | User interface
2 |  | App Privacy Policy
---
Generator (App, 2022)
Free only | Not required | Questionnaire
3 | Termly (ter, 2022a) | Free&Paid | Required | Questionnaire
4 | Privacy Policies (pri, 2022a) | Free&Paid | Required | Questionnaire
5 | App Privacy Policy (app, 2022b) | Free only | Not required | Questionnaire
6 |  | Privacy Policy
---
Online(pri, 2022b)
Free only | Not required | Questionnaire
7 | Terms Feed (ter, 2022b) | Free&Paid | Required | Questionnaire
8 | Website Policies (web, 2022) | Free&Paid | Required | Questionnaire
9 | Free Privacy Policy (fre, 2022) | Free&Paid | Required | Questionnaire
10 | Lawpath (law, 2022) | Paid only | Required | Document

Notably, all analyses of APPGs in this paper include features in both free and
paid versions. Also, to dispel the inconsistencies caused by updates on
features or functions, we conducted all analyses of APPGs in March 2022, and
we did not find any major updates based on our daily basis observation.

###  3.2. Characterization

A comprehensive assessment is conducted on identified APPGs along 10 important
dimensions. The first five mainly target practicability, user friendliness,
and convenience from potential APPG users’ perspectives. The next three simply
indicate the legal compliance with privacy regulations. The last two focus on
the understandability of privacy policies. The results are shown in Table 2
and these 10 metrics are specifically defined as follows:

Table 2. The characterization of 10 APPGs on 10 different dimensions.

# and APPG Name |  Extent of Data Use |  Customizability |  User Instruction |  Complexity |  Publishing Support |  GDPR |  CCPA |  LGPD |  Multilingual Support |  Readability
---|---|---|---|---|---|---|---|---|---|---
#1 Iubenda | ● | ◐ | ● | ● | ● | ● | ● | ● | ● | ○
#2 App Privacy Policy Generator | ◐ | ○ | ○ | ○ | ◐ | ○ | ○ | ○ | ○ | ●
#3 Termly | ● | ◐ | ● | ● | ● | ● | ● | ○ | ○ | ●
#4 Privacy Policies | ◐ | ○ | ○ | ◐ | ● | ● | ● | ○ | ○ | ○
#5 App Privacy Policy | ○ | ○ | ○ | ○ | ● | ○ | ○ | ○ | ○ | ◐
#6 Privacy Policy Online | ○ | ○ | ○ | ○ | ● | ○ | ○ | ○ | ○ | ◐
#7 Terms Feed | ◐ | ◐ | ○ | ◐ | ● | ● | ● | ○ | ○ | ○
#8 Website Policies | ● | ● | ◐ | ● | ● | ● | ● | ○ | ○ | ○
#9 Free Privacy Policy | ◐ | ○ | ○ | ◐ | ● | ● | ● | ○ | ○ | ○
#10 Lawpath | ○ | ○ | ● | ○ | ◐ | ○ | ○ | ○ | ○ | ◐

  * •

○: low level / does not support; ◐: intermediate level / partially support; ●:
high level / fully support

Extent of Data Recognised. Mobile apps normally require data from direct user
input or device sensors. Some apps may also provide data to third-party
services, for revenue or to enhance the user experience. Following privacy
regulations such as GDPR and CCPA, app developers should include information
about all collected or shared data within their privacy policy. However, some
APPGs only recognise a limited range of data practices that should be included
in privacy policies. Since failing to include all data practices could be
regarded as a violation of regulations, this is an important characteristic
and is discussed in more detail in the next section (Section 3.3).
Tentatively, we use (●) to denote that the APPG recognises a wide range of
data use, (◐) for a smaller range of data use, and (○) for a few kinds of data
use.

Customizability. Customizability basically addresses two concerns: can users
add privacy practices that were not originally included in the APPG pipeline,
and do users have the freedom to add customized clauses in the generated
privacy policy document? The former can help developers to provide a more
specific and accurate privacy policy; the latter can more easily accommodate
developers who want to address additional concerns about users’ privacy. APPGs
are denoted as (●), (◐), and (○), as indications that they are positive for
both, one and zero of these questions, respectively.

Publishing Support. This dimension reflects how convenient it is to deploy the
generated privacy policy. Every APPG provides at least one of the following
publishing options: 1) a permanently hosted website link containing the
privacy policy; and 2) the generated privacy policy context in editable HTML
format. In section 2.2.1, we presented that a large amount of privacy policy
links in the market lead to inaccessible websites, and we believe that a
permanently hosted website link provided by APPGs can mitigate this problem to
some extent. (●) indicates the APPG provides both options, and (◐) means APPGs
only provide option 2.

User Instruction. The main target group of APPGs is expected to be developers
with little or no legal knowledge about privacy and data protection, therefore
it is crucial to provide sufficient user instructions to help correctly
utilize the APPG. Appeared user instructions can be generally categorised into
three types: providing further explanation of obscure questions, elaborating
terms by listing examples, and supporting users with an interactive help
center. APPGs are scored as (○) if they only provide one or nil occurrence of
the above user instructions, (◐) for two to five occurrences, and (●) for more
than five occurrences. Especially, APPG #1 Iubenda also provides an
introduction video and comprehensive documentation.

Complexity. This dimension indicates the general learning cost of APPG users,
and higher complexity reflects a higher cost. For questionnaire- and document-
mode APPGs, we found that the complexity is basically related to the number of
questions that users encountered through the generation process. Based on the
“Statistic summary” in Table 4, if the sum of maximum multiple-choice
questions and maximum completion is less than 10, the APPG is marked as (○),
if it is between 10 to 50, the APPG is marked as (◐); and if it is greater
than 50, the APPG is marked as (●). For UI-mode APPGs, users have to spend
much more time getting familiar with the UI and learning the operation
procedure, therefore, we manually set them as (●).

GDPR, CCPA, and LGPD. These three dimensions are binary, simply indicating
whether the APPG claims that they provide compliance with corresponding
privacy regulations. (●) denotes support and (○) denotes non-support. No APPG
claims compliance with APP (APP, 2019) and PIPL (PIP, 2021), so we omitted two
empty columns in the table.

Multilingual Support. In Section 2.2.3, we discussed how current privacy
policies suffer from the language non-localization problem. APPGs that provide
multi-language support should benefit both app developers and app users. We
use (●) to denote that APPG can generate privacy policies in more than one
language, otherwise (○).

Readability. This metric represents the difficulty level for app users to read
and understand the privacy policy documents. We specifically introduced the
computing method and meaning of readability scores in Section 3.4. We use (●)
to denote the readability score is greater or equal to 46, (◐) for 40 to 46,
and (○) for smaller than 40.

Since the assessment of apps in these dimensions involved minor subjective
judgments, the first two authors assessed the 10 APPGs individually. For any
disagreement at the table, they discussed and agreed on the same answer, and
if the disagreement persisted, a third author joined the discussion to
facilitate a resolution. As shown in Table 2, basically the level of
Complexity is positively associated with the level of Extent of Data Use,
Customizability, and User Instruction. Most of APPGs (8/10) exhibit complete
publishing support. However, only #1 Iubenda supports generating privacy
policies in languages other than English. As for compliance with privacy
regulations, more than half APPGs (6/10) claim that their privacy policies
conform to the GDPR and CCPA, and only #1 Iubenda provides support for LGPD.

###  3.3. Extent of Data Use Recognised by APPGs

In this section, we further investigate the extent of data, data use, and data
practices recognised by APPGs to support the generation of privacy policies.
We identify four major aspects: the app’s basic information, users’ personal
information, device permissions, and third-party services. The results are
shown in Table 4, where (●) means that the APPG explicitly recognises the
corresponding data practice, and (○) means that the APPG does not explicitly
recognise the corresponding data practice. Since it involves intensive manual
work, to avoid the effect caused by potential human error, the first two
authors’ information is in the table individually. For any disagreement, they
discuss and agree on the same answer.

App’s basic information. Apart from user names, providing at least one
communication channel to app users for any potential inquiries is commonly
required by privacy protection and regulation laws, and all the APPGs support
this with different levels of granularity.

User’s personal information. We manually separate this into two sub-
categories: general and sensitive personal information based on our
observations. Most APPGs recognise general personal information, and only some
APPGs (3/10) explicitly recognise sensitive information.

Device permissions. According to the Android developer’s guide (and, 2022),
there are nine dangerous permission groups, namely CALENDAR, CAMERA, CONTACTS,
LOCATION, MICROPHONE, PHONE, SENSORS, SMS, and STORAGE, on mobile phones. We
expect the collection and use of data related to these permissions to be
clearly declared in a privacy policy, therefore, they are set as row indices.
We found that over half of the APPGs (6/10) recognise CAMERA, CONTACTS, and
LOCATION, only two APPGs recognise all the permissions. Failing to correctly
state sensor permissions in the privacy policy could lead to serious privacy
issues, so we further investigated whether users can correctly utilize the
APPG to state all claimed sensor permissions in Section 4.3.

Third-party services. Third-party services have become a significant part of
mobile apps. Considering the diversity of service providers and popularity, we
selected 5 third-party services as the row indices. Overall, APPGs show decent
coverage among those third-party services.

Statistic summary. We counted the completion questions and multi-choice
questions that users would encounter for questionnaire and document-mode
APPGs, as a quasi indicator to approximately reflect the users’ learning cost.

As shown in Table 4, #1, #3, and #8 are the best in terms of the recognised
data use, they almost cover every data index listed in the table. #2, #4, #7,
and #9 decently cover users’ general personal information and third-party
services, but only a few or null users’ sensitive personal information and
device permissions. #5, #6, and #10 merely cover some app’s basic information
and failed to support the rest categories. Above all, users need to select
appropriate APPGs, otherwise, the unsupported data use will be missing in the
generated privacy policy.

###  3.4. Readability of Privacy Policies Generated by APPGs

While a long privacy policy can provide more comprehensive and detailed
descriptions of data practices in apps, adding unnecessary information to a
privacy policy may lead to information overload (Bawden and Robinson, 2009;
Waldman, 2020). Although APPGs can generate more concise privacy policies that
are more specifically tailored to the data and data practices relevant to each
app, this does not necessarily mean that those privacy policies are more
readable. The vast majority of readers of privacy policies are ordinary people
with little legal knowledge. Some regulations, i.e., GDPR (GDP, 2018), CCPA
(CCP, 2018), and LGPD (LGP, 2018), require mobile app developers to use clear
and understandable language in privacy statements (Voss, 2014; Krumay and
Klar, 2020). However, those high-level regulations do not give details for
this requirement. Therefore, from the privacy policy readers’ perspective, we
introduce the readability score as an indicator to measure the difficulty
level for readers (also the app users) to comprehend the privacy policy
document.

Specifically, we adopt the Flesch reading-ease Test (Kincaid et al., 1975):

$\displaystyle 206.835-1.015\left(\frac{\text{total words}}{\text{total
sentences}}\right)-84.6\left(\frac{\text{total syllables}}{\text{total
words}}\right)$

to evaluate the readability of generated privacy policy context. A higher
score indicates the content is easier to read and understand, the score scales
from 0 to 100. We calculate the readability scores of 30 privacy policy
documents in the Generated Privacy Policy Collection (see Section 4.1), and
compute the mean scores of each APPG. Furthermore, we select 12 leading mobile
apps with over one billion installs from various mother companies on the
Google Play Store (lea, 2022) and compute their average readability score as a
comparable baseline. Results are shown in Table 3.

Table 3. Comparison of APPGs’ readability scores.

# | APPG Name | Average Readability Score
---|---|---
1 | Iubenda | 37.3
2 | App Privacy Policy Generator | 49.3
3 | Termly | 46.0
4 | Privacy Policies | 37.7
5 | App Privacy Policy | 43.9
6 | Privacy Policy Online | 44.0
7 | Terms Feed | 37.7
8 | Website Policies | 36.7
9 | Free Privacy Policy | 37.7
10 | Lawpath | 41.0
/ | 12 leading apps | 46.0

Table 4. The breakdown of APPG’s recognition of data use. The first row of
each section is the number of APPGs as per Table 1. “Multiple-choice
questions” refers to whether developers select options from listed items, and
“Completion questions” means developers fill in blanks with plain text.
(Selecting a date is also regarded as a completion question.) Since some
questions sometimes unlock later questions, we use “Minimum” and “Maximum” to
reflect the lower and upper bounds.

App’s basic information | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10
---|---|---|---|---|---|---|---|---|---|---
Name of the app | ● | ● | ● | ● | ● | ● | ● | ● | ● | ●
Name of the developer | ● | ● | ● | ● | ○ | ○ | ● | ● | ● | ●
State/Country of the developer | ○ | ○ | ● | ● | ● | ● | ● | ○ | ● | ●
Physical address of the developer | ● | ● | ● | ● | ○ | ○ | ● | ● | ● | ●
Email address of the developer | ● | ● | ● | ● | ● | ● | ● | ● | ● | ●
Phone number of the developer | ○ | ● | ● | ● | ○ | ○ | ● | ○ | ● | ○
Customizable policy effective date | ● | ● | ● | ○ | ○ | ○ | ○ | ○ | ○ | ●
Users’ personal information | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10
Users’ general personal information |  |  |  |  |  |  |  |  |  |
Name | ● | ● | ● | ● | ○ | ○ | ● | ● | ● | ○
Age group* (adulthood or underage) | ● | ○ | ● | ○ | ○ | ○ | ○ | ● | ○ | ○
Phone numbers | ● | ● | ● | ● | ○ | ○ | ● | ● | ● | ○
Email address | ● | ● | ● | ● | ○ | ○ | ● | ● | ● | ○
Residential addresses | ● | ● | ● | ● | ○ | ○ | ● | ○ | ● | ○
Users’ sensitive personal information |  |  |  |  |  |  |  |  |  |
Health data | ● | ○ | ● | ○ | ○ | ○ | ○ | ● | ○ | ○
Biometric data | ● | ○ | ● | ○ | ○ | ○ | ○ | ● | ○ | ○
Gender data | ● | ○ | ● | ○ | ○ | ○ | ○ | ○ | ○ | ○
Information revealing race or ethnic origin | ○ | ○ | ● | ○ | ○ | ○ | ○ | ● | ○ | ○
Government identifiers (e.g. medical card number) | ● | ○ | ● | ○ | ○ | ○ | ○ | ● | ○ | ○
Device permissions | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10
Calendar permission | ● | ○ | ● | ○ | ○ | ○ | ○ | ○ | ○ | ○
Camera permission | ● | ○ | ● | ● | ○ | ○ | ● | ○ | ● | ○
Contacts permission | ● | ○ | ● | ● | ○ | ○ | ● | ● | ● | ○
Location permission | ● | ○ | ● | ● | ○ | ○ | ● | ● | ● | ○
Microphone permission | ● | ○ | ● | ○ | ○ | ○ | ○ | ○ | ○ | ○
Phone permission | ● | ○ | ● | ○ | ○ | ○ | ○ | ○ | ○ | ○
Sensor permission | ● | ○ | ● | ○ | ○ | ○ | ○ | ○ | ○ | ○
SMS permission | ● | ○ | ● | ○ | ○ | ○ | ○ | ○ | ○ | ○
Storage permission | ● | ○ | ● | ○ | ○ | ○ | ○ | ○ | ○ | ○
Third-party services | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10
Facebook account access | ● | ● | ● | ○ | ○ | ○ | ○ | ● | ○ | ○
Twitter account access | ● | ○ | ● | ○ | ○ | ○ | ○ | ● | ○ | ○
Google Analytics for Firebase | ● | ● | ● | ● | ○ | ○ | ● | ● | ● | ○
Flurry Analytics | ● | ● | ● | ● | ○ | ● | ● | ● | ● | ○
AdMob | ● | ● | ● | ● | ○ | ● | ● | ● | ● | ○
Statistic summary | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10
Minimum multiple-choice questions | / | 4 | 58 | 7 | 2 | 5 | 7 | 38 | 7 | 2
Maximum multiple-choice questions | / | 5 | 79 | 26 | 2 | 5 | 26 | 63 | 26 | 2
Minimum completion questions | / | 4 | 11 | 2 | 2 | 3 | 2 | 2 | 2 | 5
Maximum completion questions | / | 4 | 76 | 22 | 2 | 3 | 22 | 13 | 22 | 6

  * •

*If date of birth is included in the APPG, then we consider that Age group can be inferred.

All APPGs’ readability scores are between 30 and 50, which indicates, that
readers should have at least a college-level education background to easily
read the policies (Kincaid et al., 1975). APPG #2, namely App Privacy Policy
Generator, has the highest readability score and is the most frequently used
APPG on the current market, as discussed in Section 4.2. Moreover, compared to
the average level of 12 leading apps, generally, all APPGs have slightly
poorer readability scores, except # 2 App Privacy Policy Generator and # 3
Termly, with 49.3 and 46 readability scores, respectively.

##  4\. Market Penetration Analysis (RQ2) and Permissions Coverage Analysis
(RQ3)

###  4.1. Synthetic Apps and Generated Privacy Policy Collection

To identify whether the privacy policy could be generated by APPGs, we first
need to build the ground truth with self-generated privacy policies for each
APPG. As shown in Table 1, seven APPGs require users to register an account to
use the service, and six of them also require users to pay subscription fees
to unlock all features. We registered as required and paid subscription fees.
Synthetic apps are commonly used in similar empirical studies such as
(Oltrogge et al., 2018; Sun and Xue, 2020). We designed and tailored 3
synthetic apps specifically based on APPGs’ characteristics, functions, and
features as summarised in Table 2 and Table 4. This guarantees that the
boilerplates and pre-defined clauses of every APPG can be fully explored and
reflected in the generated privacy policies. The functions and features of
these synthetic apps are as follows:

Synthetic App 1. A toy-like app only collects the general personal information
listed in Table 4, without any data disclosed or transferred internationally.

Synthetic App 2. A social app enables people to interact and communicate with
others. Users need to create an account by providing their general personal
information, and the app requires all device permissions listed in Table 4
except SENSORS. The app also accesses the third-party services listed in Table
4. However, the developer of this app is based in the European Union, with
only European customers.

Synthetic App 3. A medical app diagnoses COVID-19 by uploading a recorded clip
of coughing. Users need to provide both general personal information and
sensitive personal information. Government identifiers could be used when
reporting potential positive results to authorities, along with other
sensitive personal data and the diagnosis result. The app needs microphone
permission and storage permission but does not involve any commercial third-
party services. The developers of this app are based in California and the
European Union, with users in both places.

These three synthetic apps are various in terms of functionality and
sophistication, and they decently cover all data use mentioned in Table 4.
Therefore, they are capable of being the boilerplate apps for APPGs’ ground
truth in the majority of cases. For each APPG, we create the same three custom
privacy policies to test whether a privacy policy is generated by one of
analysed APPGs. Given ten APPGs and three synthetic apps, we obtain 30 privacy
policies as “Generated Privacy Policy Collection”.

###  4.2. Market Penetration Analysis (RQ2)

We report a market penetration analysis to highlight the extent to which APPGs
are used and to provide insights about why some APPGs are more popular.
Specifically, we employ two methods to detect whether a privacy policy could
be created by one of these APPGs: fingerprint keyword searching and document
similarity comparison.

Figure 2. The number of privacy policies generated by APPGs, detected by FKS
(left), DSC (mid), and both (right).

Fingerprint Keyword Searching (FKS). As shown in Table 2, APPGs provide
different publishing features, including providing a direct link as a
permanent website host, or an editable HTML document. We observe that to
better advertise themselves, APPGs will normally embed their company names
into the generated privacy policy website address or the editable HTML
document, so we construct the fingerprint keyword set based on this feature.
In order to cultivate our fingerprint keyword set, we manually inspected these
APPGs and the generated privacy policy collection. For every candidate
fingerprint keyword, we manually verify the validity of the generated privacy
policy collection. Specifically, if a candidate fingerprint keyword can be
found in all three ground-truth privacy policies generated for synthetic apps,
we append it to the fingerprint keyword set. Then we employ the collected set
to perform fingerprint keyword searching in the crawled privacy policy
collection. For each privacy policy in our collection, if we find a match in
either its website link or its document context with one of the fingerprint
keywords, the policy is considered to be generated by the corresponding APPG.

Document Similarity Comparison (DSC). As we mention in Section 3.2, some APPGs
allow users to customize the content to some extent, and APPG users may also
further adapt or polish the initial generated privacy policy in editable HTML
format before publishing. Consequently, the fingerprint keywords, even
sentence-level segments, are not necessarily included in a generated app
privacy policy sometimes, and the searching-based method does not work
anymore. To tackle this problem, we design a semantic-based method with better
robustness, called document similarity comparison.

In natural language processing (NLP) tasks, researchers adapt the concept from
mathematics by treating documents as vectors, to obtain the similarity between
two documents. TF-IDF (Rajaraman and Ullman, 2011) is a common numerical
statistic designed to reflect the relative importance of words in a document
corpus, and thus it can be adapted to convert documents into vectors. By doing
so, the similarity between two documents can be represented as the similarity
of two vectors, and we choose cosine similarity as the most common one. If the
average similarity between a crawled privacy policy and three ground-truth
generated privacy policies is greater than a threshold $\beta$, we consider it
a match, i.e., could be created by the current APPG. If a privacy policy
matches more than one APPG, we take the one with the highest similarity. To
define the value of $\beta$, we first calculate the document similarities
between the three generated privacy policies of each APPG as an initial value.
We then randomly sampled 100 privacy policies that are generated by one of 10
APPGs based on fingerprint keyword searching and evaluate the similarities
compared to the three generated privacy policies of each APPG to fine-tune the
$\beta$. Based on our data, we empirically set $\beta=0.75$.

Table 5. Summary of market use of different APPGs.

Method | Market Occupancy | Intersection | Union
---|---|---|---
FKS | 6.6% (3,066) | 4.4% (2,042) | 20.1% (9,332)
DSC | 18.1% (8,425)

Results. We employ the proposed methods to provide an estimation of the market
penetration of APPGs. Among 46,472 valid privacy policy documents from apps at
the Google Play Store, as shown in Table 5, we find 6.6% (3,066) privacy
policies are highly likely generated by one of these APPGs by using
fingerprint keyword searching and 18.1% (8,425) by using document similarity
comparison. We also check the intersection size, which is 4.4% (2,042),
indicating that these two methods are complementary to each other. We further
compute the union of the previous two methods, which is 20.1% (9,332), as the
possible overall market occupancy ratio. Overall, the results show that APPGs
do play a considerable role in the current mobile app market.

Figure 2 presents the market occupation for each APPG. We can see that APPG #2
App Privacy Policy Generator is the most popular APPG, and it could be
attributed to its free of charge and registration, with intermediate
recognition of data use and best readability score. In addition, we notice
that popular APPGs generally have a low level/do not support or an
intermediate level/partially support on all dimensions. Users may prefer to
select easy-to-use tools rather than spend extra time learning how to master a
sophisticated single-use APPG. However, the ease of use of APPGs may come at
the cost of a potentially-higher risk to breach privacy regulations.
Additionally, #1 Iubenda and #5 App Privacy Policy also take decent market
shares. #10 Lawpath is the least used APPG in our data since its price to use
is $199 for one privacy policy document, which is the highest among all APPGs.

In Figure 3, the grey bars denote the approximate adoption rate of APPG for
each category and the black dots indicate the means of claimed dangerous
permissions from each category. Generally, it shows that market occupancy
ratios range from 10% to 40% for all categories. In addition, it reveals the
relation between the adoption rate and the average dangerous device
permissions, by app categories. Specifically, developers tend to more commonly
use APPGs to generate their privacy policies, for app categories with low
average device permissions such as Books & Reference and game categories. On
the contrary, for apps categories that are prone to require more device
permissions to function, for example, Health & Fitness, Shopping, and
Business, their privacy policies tend not to be created by APPGs.

Figure 3. Use in market by APPGs (grey bars) and the average device
permissions claimed (black dots), by app categories. The left and the right
y-axis ticks represent the market occupancy percentages and the amount of
average permissions, respectively. The dashed line is a linear regression
computed from the means of claimed dangerous permissions from each category.

###  4.3. Permissions Coverage Analysis (RQ3)

Device (sensor) permissions enable features of mobile apps and are normally
highly sensitive in terms of personal privacy. All mobile apps at the Google
Play Store are required to clearly list the permissions they intend to obtain
on their homepages; however, it is common that developers do not put all their
app’s permissions in the privacy policy (Zimmeck et al., 2019). In this
section, we report an analysis revealing gaps between device permissions
claimed by developers and the context displayed in their generated privacy
policies by employing APPGs.

App developers need to claim and list all required device permissions,
including nine dangerous permissions, on their app homepages. We take this
data as the ground truth for the following permissions coverage analysis, and
for each app, note the number of claimed permissions as $P_{\text{claimed}}$.
To scrutinize how APPGs treat those permissions in the generated context, for
each APPG, we use Synthetic App 1 to generate privacy policies and each
privacy policy includes only one dangerous permission. By comparing
differences, we locate and obtain the phrases/sentences each APPG uses to
declare each permission in the privacy policy. For each app, by searching and
counting the corresponding phrases/sentences in its privacy policy, we obtain
the number of displayed permissions and denoted $P_{\text{displayed}}$. For
each app that uses the APPG, its $P_{\text{recognised}}$ equals the recognised
dangerous permissions enumerated in Table 4. We introduce two metrics for the
extent to which APPGs cover claimed dangerous permissions in the generated
privacy policy, as follows:

$\displaystyle\text{Recognised
Coverage}\,(RC)=\frac{1}{n}\sum_{1}^{n}\frac{P_{\text{displayed}}}{P_{\text{recognised}}}$

A higher $RC$ indicates the app developers use the APPG more properly, despite
the limited amount of recognised permissions. And

$\displaystyle\text{Expected
Coverage}\,(EC)=\frac{1}{n}\sum_{1}^{n}\frac{P_{\text{displayed}}}{P_{\text{claimed}}}$

A higher $EC$ reflects that the APPG’s limited amount of recognised
permissions has less impact on displaying all claimed permissions in the
generated privacy policy. Both metrics can reach 100% if there are no over-
claimed permissions.

Table 6. Dangerous permissions coverage for privacy policies generated by APPGs. # | APPG Name |  | Recognised
---
Coverage
| Expected
---
Coverage
1 | Iubenda | 15.6% | 15.6%
2 | App PrivacyPolicy Generator | 0% | 0%
3 | Termly | 60.3% | 60.3%
4 | Privacy Policies | 85.6% | 40.1%
5 | App Privacy Policy | 0% | 0%
6 | Privacy Policy Online | 0% | 0%
7 | Terms Feed | 52.7% | 25.7%
8 | Website Policies | 60.0% | 11.8%
9 | Free Privacy Policy | 64.4% | 25.6%
10 | Lawpath | 0% | 0%

We use the intersection of privacy policies identified by both FKS and DSC
(2,042 privacy policies) because it is more certain that they are generated by
one of the APPGs. The results are shown in Table 6. Based on our observations
in Section 3.3, there are four APPGs (#2, #5, #6, #10) that do not support
permission declarations, in other words, $P_{\text{APPG recognised}}=0$ for
every app identified to use those APPGs. In addition, we find that
$P_{\text{displayed}}=0$ for four those apps as well, which means none of the
developers realised that the APPG they used did not cover the permissions they
claimed. For #1 and #3, since they both can recognise all nine device
permissions, their $RC$s are $EC$s respectively.

There is a significant portion of missing dangerous permissions for all APPGs
since all $EC$s are much smaller than 100%. Second, compared to #1’s and #3’s
consistency on $RC$s and $EC$s, for #4, #7, #8, and #9, their $RC$s are
greatly smaller than $EC$s, This shows the importance of APPGs selection
because inappropriate APPGs will not help developers include all claimed
dangerous permissions in their privacy policy. Third, #1 has the lowest $RC$,
but it has high level user instruction; whereas #4, #7, and #9 has the
relatively high $RC$s, but low level of user instruction (See Table 2). This
counter-intuitive result indicates that current user instructions are not
helpful enough to guide developers to correctly select permissions. In
addition, #1 is the only UI-mode APPG, with the lowest $RC$ (15.6%), and other
comparable APPGs’ whose $RC$ is over 50% are all questionnaire-mode. This
finding suggests that, compared to UI-mode, questionnaire-mode can better
guide users to include more claimed device permissions.

##  5\. Discussion and Future Work

Challenges and opportunities coexist in current APPG development. Based on our
observations and findings, we summarize some take-home messages for different
roles in the APPG ecosystem.

App developers/APPG users. While app developers may benefit from using APPGs
to create privacy policies more efficiently, they should be aware of APPGs’
limitations. As illustrated in Section 3, APPGs have different qualities. Some
do not directly target specific regulations, such as GDPR and CCPA, and some
do not recognise data practices, i.e., the declaration of personal
information, permissions, and third-party services used in mobile apps.
Moreover, generated privacy policies may not be reader-friendly. If users do
not understand an app’s privacy policy, this may affect their trust in the
app. App developers should make sure the privacy policies they provide, either
created by themselves, or generated by APPGs, are comprehensive and readable,
especially when they intend to collect sensitive personal information, require
dangerous device sensors permissions, or seek to comply with privacy
regulations.

APPG providers. Our analysis suggests APPG providers should work on improving
the recognised data use since the majority of APPGs on the market only provide
a very limited scope of personal information and device sensor permissions.
However, based on our market penetration analysis, we found that developers
would rather use a simpler APPG (e.g., #2) than a correct one. It is
worthwhile to handle this trade-off in APPGs’ designing process. Furthermore,
our results indicate current user instructions are not helpful enough, thus,
APPG providers should work to improve user instructions and provide intuitive
UIs to guide users to correctly use tools.

Privacy regulators. We found that the privacy policy of 20.1% apps is highly
likely generated by existing APPGs, which means APPGs can significantly
contribute to reducing policy absence and low quality privacy policies on app
markets. Our results show that the APPGs’ design flaws will limit the app
developers to include all essential device permissions in the policies, which
creates a risk of violating some requirements in current privacy regulations.
Thus, regulators are expected to realise the significance of this issue and be
responsible and responsive to this emerging market trend.

Future work. Based on our findings, we recognise two potential directions for
future research. First, Online Application Generators (OAG) can automate
mobile app development and are popular with citizen app developers (Oltrogge
et al., 2018). Combining APPG and OAG together may further reduce the level of
the technical and legal skills required for citizen developers to develop
safe, useful, and regulation-compliant apps. Additionally, incorporating the
treatment of privacy practices into the early app design phase may help to
improve the trustworthiness of privacy policies. Another promising application
is to provide a contextual privacy policy for mobile apps with the help of
APPGs. The standardized language and format in the generated policies can make
it effortless to obtain the mapping relation between device permissions and
the corresponding clauses in the privacy policy document. Thus, when the
device permission is active, users can be directed to the corresponding
segment in the privacy policy, and do not need to seek it through the whole
document. Last, different from existing code-centric research, we hope this
study could raise attention to low-code/no-code tools such as APPGs, as they
are increasingly employed in practice but much less explored.

##  6\. Related Work

Previous researchers (Wilson et al., 2016) created a corpus of 115 privacy
policies and 23K fine-grained data practice annotations, and revealed Internet
users’ preferences on privacy policy structure and complexity. Another study
(Liu et al., 2020) created a manually labeled corpus containing 167 privacy
policies, and reported the performance of 4 document classification models. A
study (Liu et al., 2018) discussed privacy policy classification with
supervised-learning models. In one paper (Zimmeck et al., 2019), the authors
conducted an empirical study of privacy compliance on a million apps and
created another dataset called APP-350 Corpus consisting of apps’ privacy
practices. Amos et al. (Amos et al., 2021) built a large-scale website privacy
policy dataset, spanning two decades and over 130k websites. They also
reported that compliance and readability became worse over the last twenty
years.

Some previous works (Yu et al., 2015, 2018b; Slavin, 2022; Zimmeck et al.,
2021) proposed privacy policy automation tools based on the source code of
mobile apps. Yu et al. (Yu et al., 2015, 2018b) developed a system named
“AutoPPG”, to automatically construct readable descriptions from the mobile
apps’ source code, to help create privacy policies on Android. PoliDriod
(Slavin, 2022) is an Android Studio plugin that can be used for the detection
of potential privacy policy misalignments between Android API methods and a
mobile privacy lexicon. Zimmeck et al. (Zimmeck et al., 2021) identified code
signatures that are mapped to privacy practices expressed in privacy policies
and proposed a privacy policy generator called PrivacyFlash that leverages
static analysis for iOS apps. Others (Bateni, 2021; Windl et al., 2022) have
discussed the automated generation of privacy policies by learning-based
methods.

The closest work is (Sun and Xue, 2020), which is an empirical study of APPGs’
quality assessment. However, we found some APPGs they selected do not support
mobile apps, leading to questionable validity of the following analysis. In
addition, their analysis stopped at a preliminary stage, lacking systematic
and comprehensive assessments on multiple dimensions. Last and most
importantly, their analysis and conclusions are completely based on authors’
manual observations, without leveraging the large-scale market information to
evaluate the usage of APPGs in practice.

##  7\. Conclusion

Online Automated Privacy Policy Generators are broadly used by developers of
mobile apps to create privacy policies that address regulatory requirements.
This paper reports the first large-scale empirical study to comprehensively
analyse the features, characteristics, and quality of APPGs on 10 dimensions.
Our market penetration analysis indicates that the privacy policy of 20.1%
apps could be generated by existing APPGs on Google Play Store, and #2 is the
most popular one. In addition, our device permissions coverage analysis
reveals that existing APPGs have significant issues with including all
essential device permissions in the generated privacy policy, and UI-mode
APPGs could make it worse. Our future work would include improving pipelines
to generate privacy policies and developing contextual privacy policies to
help users better understand privacy practices.

## References

  * (1)
  * LGP (2018) 2018\.  _Brazilian General Data Protection Law (LGPD)_.  http://www.planalto.gov.br/ccivil_03/_Ato2015-2018/2018/Lei/L13709.htm Retrieved: 2022-08-26.
  * CCP (2018) 2018\.  _California Consumer Privacy Act of 2018 (CCPA)_.  https://oag.ca.gov/privacy/ccpa Accessed: 2022-04-25.
  * GDP (2018) 2018\.  _General Data Protection Regulation (GDPR)_.  https://gdpr-info.eu/ Retrieved: 2022-04-25.
  * APP (2019) 2019\.  _Australian Privacy Principles (APP)_.  https://www.oaic.gov.au/privacy/australian-privacy-principles Accessed: 2022-05-03.
  * PIP (2021) 2021\.  _Personal Information Protection Law of the People’s Republic of China (PIPL)_.  http://www.npc.gov.cn/npc/c30834/202108/a8c4e3672c74491a80b53a172bb753fe.shtml Accessed: 2022-04-25.
  * and (2022) 2022\.  _Android developer guide_.  https://developer.android.com/guide/topics/permissions/overview#permission-groups Accessed: 2022-08-26.
  * goo (2022a) 2022a.  _Android Developers, Google Play Console Help Center_.  https://support.google.com/googleplay/android-developer/answer/9859455?hl=en#:~:text=The%20privacy%20policy%20must%2C%20together,you%20of%20what%20is%20required Accessed: 2022-04-30.
  * app (2022a) 2022a.  _App privacy details on the App Store_.  https://developer.apple.com/app-store/app-privacy-details/ Accessed: 2022-04-30.
  * app (2022b) 2022b.  _App Privacy Policy_.  https://www.app-privacy-policy.com/ Accessed: 2022-03-28.
  * App (2022) 2022\.  _App Privacy Policy Generator_.  https://app-privacy-policy-generator.firebaseapp.com/ Accessed: 2022-03-28.
  * app (2022) 2022\.  _AppBrain_.  https://www.appbrain.com/ Accessed: 2022-08-26.
  * bea (2022) 2022\.  _BeautifulSoup_.  https://www.crummy.com/software/BeautifulSoup/ Accessed: 2022-05-02.
  * fre (2022) 2022\.  _Free Privacy Policy_.  https://www.freeprivacypolicy.com/ Accessed: 2022-03-28.
  * gee (2022) 2022\.  _Geeks for Geeks_.  https://www.geeksforgeeks.org/ Accessed: 2022-08-26.
  * git (2022) 2022\.  _Github_.  https://github.com/ Accessed: 2022-08-26.
  * goo (2022b) 2022b.  _google-play-scapper_.  https://pypi.org/project/google-play-scraper/ Accessed: 2022-04-30.
  * Iub (2022) 2022\.  _Iubenda_.  https://www.iubenda.com/en/ Accessed: 2022-03-28.
  * lan (2022) 2022\.  _langdetect_.  https://pypi.org/project/langdetect/ Accessed: 2022-04-30.
  * law (2022) 2022\.  _Lawpath_.  https://lawpath.com.au/ Accessed: 2022-03-28.
  * lea (2022) 2022\.  _Leading apps Wikipedia_.  https://en.wikipedia.org/wiki/List_of_most-downloaded_Google_Play_applications Accessed: 2022-08-26.
  * pri (2022a) 2022a.  _Privacy Policies_.  https://www.privacypolicies.com/ Accessed: 2022-03-28.
  * Bwa (2022) 2022\.  _The privacy policy of “Bway App”_.  http://www.e-droid.net/privacy.php?ida=1702882&idl=en Accessed: 2022-03-28.
  * Eas (2022) 2022\.  _The privacy policy of “Easy Communication”_.  http://natarajanraman.in/?page_id=67 Accessed: 2022-03-28.
  * pri (2022b) 2022b.  _Privacy Policy Online_.  https://www.privacypolicyonline.com/ Accessed: 2022-03-28.
  * Sel (2022) 2022\.  _Selenium_.  https://www.selenium.dev/ Accessed: 2022-04-28.
  * sta (2022) 2022\.  _Stack Overflow_.  https://stackoverflow.com/ Accessed: 2022-08-26.
  * ter (2022a) 2022a.  _Termly_.  https://termly.io/ Accessed: 2022-03-28.
  * ter (2022b) 2022b.  _Terms Feed_.  https://app.termsfeed.com/ Accessed: 2022-03-28.
  * web (2022) 2022\.  _Website Policies_.  https://www.websitepolicies.com/ Accessed: 2022-03-28.
  * Allix et al. (2016) Kevin Allix, Tegawendé F. Bissyandé, Jacques Klein, and Yves Le Traon. 2016\.  AndroZoo: Collecting Millions of Android Apps for the Research Community. In _Proceedings of the 13th International Conference on Mining Software Repositories_ (Austin, Texas) _(MSR ’16)_. ACM, New York, NY, USA, 468–471.  https://doi.org/10.1145/2901739.2903508
  * Amos et al. (2021) Ryan Amos, Gunes Acar, Elena Lucherini, Mihir Kshirsagar, Arvind Narayanan, and Jonathan Mayer. 2021\.  Privacy policies over time: Curation and analysis of a million-document dataset. In _Proceedings of the Web Conference 2021_. 2165–2176.
  * Balebako and Cranor (2014) Rebecca Balebako and Lorrie Cranor. 2014.  Improving app privacy: Nudging app developers to protect user privacy.  _IEEE Security & Privacy_ 12, 4 (2014), 55–58.
  * Balebako et al. (2014) Rebecca Balebako, Abigail Marsh, Jialiu Lin, Jason I Hong, and Lorrie Faith Cranor. 2014.  The privacy and security behaviors of smartphone app developers.  (2014).
  * Bateni (2021) Nastaran Bateni. 2021\.  _Automated Methods for Comparison and Generation of Privacy Policies_.  Ph. D. Dissertation. University of Guelph.
  * Bawden and Robinson (2009) David Bawden and Lyn Robinson. 2009.  The dark side of information: overload, anxiety and other paradoxes and pathologies.  _Journal of information science_ 35, 2 (2009), 180–191.
  * Habchi et al. (2017) Sarra Habchi, Geoffrey Hecht, Romain Rouvoy, and Naouel Moha. 2017.  Code smells in ios apps: How do they compare to android?. In _2017 IEEE/ACM 4th International Conference on Mobile Software Engineering and Systems (MOBILESoft)_. IEEE, 110–121.
  * Kincaid et al. (1975) J Peter Kincaid, Robert P Fishburne Jr, Richard L Rogers, and Brad S Chissom. 1975.  _Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel_.  Technical Report. Naval Technical Training Command Millington TN Research Branch.
  * Kollnig et al. (2021) Konrad Kollnig, Anastasia Shuba, Reuben Binns, Max Van Kleek, and Nigel Shadbolt. 2021.  Are iPhones Really Better for Privacy? Comparative Study of iOS and Android Apps.  _arXiv preprint arXiv:2109.13722_ (2021).
  * Krumay and Klar (2020) Barbara Krumay and Jennifer Klar. 2020.  Readability of privacy policies. In _IFIP Annual Conference on Data and Applications Security and Privacy_. Springer, 388–399.
  * Kumar et al. (2022) Renuka Kumar, Apurva Virkud, Ram Sundara Raman, Atul Prakash, and Roya Ensafi. 2022.  A Large-scale Investigation into Geodifferences in Mobile Apps. In _31st USENIX Security Symposium (USENIX Security 22)_. USENIX Association, Boston, MA.  https://www.usenix.org/conference/usenixsecurity22/presentation/kumar
  * Levy and Gutwin (2005) Stephen E Levy and Carl Gutwin. 2005.  Improving understanding of website privacy policies with fine-grained policy anchors. In _Proceedings of the 14th international conference on World Wide Web_. 480–488.
  * Li et al. (2018) Tianshi Li, Yuvraj Agarwal, and Jason I Hong. 2018\.  Coconut: An IDE plugin for developing privacy-friendly apps.  _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ 2, 4 (2018), 1–35.
  * Li et al. (2022) Tianshi Li, Kayla Reiman, Yuvraj Agarwal, Lorrie Faith Cranor, and Jason I Hong. 2022.  Understanding Challenges for Developers to Create Accurate Privacy Nutrition Labels. In _CHI Conference on Human Factors in Computing Systems_. 1–24.
  * Liu et al. (2018) Frederick Liu, Shomir Wilson, Peter Story, Sebastian Zimmeck, and Norman Sadeh. 2018.  Towards automatic classification of privacy policy text.  _School of Computer Science Carnegie Mellon University_ (2018).
  * Liu et al. (2020) Shuang Liu, Renjie Guo, Baiyang Zhao, Tao Chen, and Meishan Zhang. 2020.  APPCorp: A corpus for Android privacy policy document structure analysis.  _arXiv preprint arXiv:2005.06945_ (2020).
  * Liu et al. (2021) Shuang Liu, Baiyang Zhao, Renjie Guo, Guozhu Meng, Fan Zhang, and Meishan Zhang. 2021\.  Have You been Properly Notified? Automatic Compliance Analysis of Privacy Policy Text with GDPR Article 13. In _Proceedings of the Web Conference 2021_. 2154–2164.
  * Nokhbeh Zaeem and Barber (2021) Razieh Nokhbeh Zaeem and K Suzanne Barber. 2021.  A large publicly available corpus of website privacy policies based on dmoz. In _Proceedings of the Eleventh ACM Conference on Data and Application Security and Privacy_. 143–148.
  * Oltrogge et al. (2018) Marten Oltrogge, Erik Derr, Christian Stransky, Yasemin Acar, Sascha Fahl, Christian Rossow, Giancarlo Pellegrino, Sven Bugiel, and Michael Backes. 2018.  The rise of the citizen developer: Assessing the security impact of online app generators. In _2018 IEEE Symposium on Security and Privacy (SP)_. IEEE, 634–647.
  * Poplavska et al. (2020) Ellen Poplavska, Thomas B Norton, Shormir Wilson, and Norman Sadeh. 2020.  From prescription to description: Mapping the gdpr to a privacy policy corpus annotation scheme. In _Legal Knowledge and Information Systems-JURIX 2020: 33rd Annual Conference_.
  * Rajaraman and Ullman (2011) Anand Rajaraman and Jeffrey David Ullman. 2011.  _Mining of massive datasets_.  Cambridge University Press.
  * Schaub et al. (2015) Florian Schaub, Rebecca Balebako, Adam L Durity, and Lorrie Faith Cranor. 2015.  A design space for effective privacy notices. In _Eleventh symposium on usable privacy and security (SOUPS 2015)_. 1–17.
  * Slavin (2022) Rocky Slavin. 2022\.  _PoliDroid-AS_.  https://github.com/rslavin/PoliDroid-AS Accessed: 2022-05-05.
  * Sun and Xue (2020) Ruoxi Sun and Minhui Xue. 2020.  Quality assessment of online automated privacy policy generators: an empirical study.  In _Proceedings of the Evaluation and Assessment in Software Engineering_. 270–275.
  * Voss (2014) W Gregory Voss. 2014\.  Looking at european union data protection law reform through a different prism: The proposed eu general data protection regulation two years later.  _Journal of Internet Law_ 17, 9 (2014).
  * Waldman (2020) Ari Ezra Waldman. 2020\.  Cognitive biases, dark patterns, and the ‘privacy paradox’.  _Current opinion in psychology_ 31 (2020), 105–109.
  * Wilson et al. (2016) Shomir Wilson, Florian Schaub, Aswarth Abhilash Dara, Frederick Liu, Sushain Cherivirala, Pedro Giovanni Leon, Mads Schaarup Andersen, Sebastian Zimmeck, Kanthashree Mysore Sathyendra, N Cameron Russell, et al. 2016\.  The creation and analysis of a website privacy policy corpus. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. 1330–1340.
  * Windl et al. (2022) Maximiliane Windl, Niels Henze, Albrecht Schmidt, and Sebastian S Feger. 2022.  Automating Contextual Privacy Policies: Design and Evaluation of a Production Tool for Digital Consumer Privacy Awareness. In _CHI Conference on Human Factors in Computing Systems_. 1–18.
  * Yu et al. (2018a) Le Yu, Xiapu Luo, Jiachi Chen, Hao Zhou, Tao Zhang, Henry Chang, and Hareton KN Leung. 2018a.  Ppchecker: Towards accessing the trustworthiness of android apps’ privacy policies.  _IEEE Transactions on Software Engineering_ 47, 2 (2018), 221–242.
  * Yu et al. (2018b) Le Yu, Xiapu Luo, Jiachi Chen, Hao Zhou, Tao Zhang, Henry Chang, and Hareton KN Leung. 2018b.  Ppchecker: Towards accessing the trustworthiness of android apps’ privacy policies.  _IEEE Transactions on Software Engineering_ 47, 2 (2018), 221–242.
  * Yu et al. (2015) Le Yu, Tao Zhang, Xiapu Luo, and Lei Xue. 2015\.  Autoppg: Towards automatic generation of privacy policy for android applications. In _Proceedings of the 5th Annual ACM CCS Workshop on Security and Privacy in Smartphones and Mobile Devices_. 39–50.
  * Zimmeck et al. (2021) Sebastian Zimmeck, Rafael Goldstein, and David Baraka. 2021\.  Privacyflash pro: automating privacy policy generation for mobile apps. In _28th Network and Distributed System Security Symposium (NDSS 2021). NDSS_.
  * Zimmeck et al. (2019) Sebastian Zimmeck, Peter Story, Daniel Smullen, Abhilasha Ravichander, Ziqi Wang, Joel R Reidenberg, N Cameron Russell, and Norman Sadeh. 2019.  Maps: Scaling privacy compliance analysis to a million apps.  _Proc. Priv. Enhancing Tech._ 2019 (2019), 66.