# Leveraging Privacy Profiles to Empower Users in the Digital Society

Davide Di Ruscio  Paola Inverardi
Patrizio Migliarini  Phuong T. Nguyen  Università degli studi dell’Aquila
L’Aquila, Italy
{davide.diruscio, paola.inverardi, patrizio.migliarini,
<EMAIL_ADDRESS>
###### Abstract

Privacy and ethics of citizens are at the core of the concerns raised by our
increasingly digital society. Profiling users is standard practice for
software applications triggering the need for users, also enforced by laws, to
properly manage privacy settings. Users need to manage software privacy
settings properly to protect personally identifiable information and express
personal ethical preferences. AI technologies that empower users in their
interaction with the digital world by reflecting their personal ethical
preferences can be key enablers of a trustworthy digital society. This paper
focuses on the privacy dimension and contributes a step in the above direction
through an empirical study on an existing dataset collected from the fitness
domain. The study aims to understand which set of questions and settings is
more appropriate to differentiate users according to their privacy
preferences. The experimental results reveal that a compact set of semantic-
driven questions (about general privacy preferences) helps distinguish users
better than a complex domain-dependent one (concerning the fitness domain).
Based on the study outcome, we design and implement a recommender system to
provide users with suitable recommendations with respect to privacy choices.
We then show that the proposed recommender system provides relevant settings
to users, obtaining high prediction accuracy.

###### keywords:

Privacy profiles, Clustering, Recommender systems

backgroundcolor=, commentstyle=, breakatwhitespace=false, keywordstyle=,
language=Java, stringstyle=, basicstyle=, showstringspaces=false
backgroundcolor=, commentstyle=, breakatwhitespace=false, keywordstyle=,
language=Java, stringstyle=, basicstyle=,frame=single, showstringspaces=false
backgroundcolor=, commentstyle=, breakatwhitespace=false, keywordstyle=,
language=Java, stringstyle=, basicstyle=, showstringspaces=false

##  1 Introduction

Privacy and ethics of citizens are at the core of the concerns raised by our
increasingly digital society. Profiling users is standard practice for
software applications triggering the need for users, also enforced by laws, to
properly manage privacy settings and moral preferences. This deals with the
way users give their consent to storing, sharing to third parties, as well as
disseminating sensitive personal information and express moral preferences
like, for example, ticking to pay a _decarbonization tax_. Mobile apps have
been becoming increasingly popular as they can provide users with a wide range
of functionalities. For different reasons, apps often require access to
intimate information about the users and hosting device, triggering privacy
concerns. This requires proper management of privacy, with the ultimate aim of
protecting users’ preferences as well as personally identifiable information.

In this paper, we focus on the privacy dimension of an ordinary user with
little technical knowledge of the privacy mechanisms of the digital systems
she seamlessly uses but with an evident moral character. While choosing strict
settings may help protect her data, this may prevent the complete availability
of the functionalities provided by the software. In contrast, loosening
privacy settings mitigates the restriction on functionalities, but it may come
with the price of compromising her data privacy. In this respect, Artificial
Intelligence (AI) technologies can empower the user in maintaining a
reasonable trade-off between accessibility and protection, and reflecting the
user privacy preferences can be the key enabler of a trustworthy digital
society.

Understanding the commonalities and differences among users based on profiles
has been among the main issues in data privacy research [1, 2, 3].
Categorizing profiles contributes to better identification of users’ behaviors
and supports administrators in comprehending privacy choices. At the same
time, personal profiles may enable the design of functionalities that help
users set privacy preferences of the digital technologies they use. Various
proposals to categorize or group end-users into clusters based on their
security or privacy attitudes/behaviors in specific domains have been made [4,
5]. Users’ preferences were analyzed in an extensive study [6] on permission
settings from real Android mobile users to recommend personalized default
settings. Sanchez _et al._ [7] analyzed user-privacy preferences in the
fitness domain employing a specifically designed questionnaire consisting of
both domain-specific and general questions to recommend personalized privacy
settings for the fitness apps.

Though a lot of achievements have been reported, as discussed in [8], we
believe that there is still the need to understand how to characterize user’s
privacy behavior in a general setting. Indeed privacy is a dimension of ethics
and should be part of the ethical profile of a user and driven by ethical
consideration rather than by contextual attitudes or practices in given
domains. For example, relying on the analysis of current or past users’
preference settings as in [6] does not guarantee a correspondence between what
users believe as their general privacy profile and what they actually (can) do
when setting privacy preferences. Moreover, data privacy awareness in the
digital society is only recently exiting the specialists’ fields (legal,
ethical, economic, social) to impact the wider society. The pandemic has also
dramatically advanced the penetration of digital technologies in the society
from market to education [9, 10]. This means that a large body of collected
data on privacy settings may not reflect the attitude and attention to privacy
that present users have and will have in the future.

In this work, we explore a different research direction by relying on the data
of the study in the fitness domain [7] that were collected by means of a
questionnaire and a simulator.111We thank Prof. Dr. Ilaria Torre, University
of Genoa (Italy) for providing us with the privacy dataset [7]. We analyze
both general and domain-specific questions with the aim of _(i)_ identifying
general questions that reflect moral attitudes of the users; and _(ii)_
recommending privacy preferences accordingly. Moreover, we design and
implement a recommender system [11] to provide users with suitable
recommendations with respect to privacy choices. The experimental results are
positively interesting, revealing that a compact set of general questions
helps distinguish users better than a more complex domain-dependent one. We
also show that the proposed recommender system provides relevant settings to
users, obtaining high prediction accuracy.

The main contributions of our work are summarized as follows.

  * •

We investigate which sets of (general) privacy questions are more relevant for
classifying users with respect to their privacy moral preferences.

  * •

By means of an empirical evaluation, we show that self-assessment about
privacy attitudes given by users does not reflect the way they act in
practice.

  * •

We develop PisaRec, a recommender system to provide suitable privacy settings
that reflect user preferences. This aims to help users relieve the burden of
setting privacy configurations when they go online.

We organize the paper into the following sections. In Section 2, we present a
motivating example and a categorization of privacy profiles. Section 3
describes the proposed approach which makes use of both unsupervised and
supervised learning to handle user profiles. The methods used to evaluate our
approach are detailed in Section 4. We report and analyze the experimental
results in Section 5. Discussion related to the limitations and threats to
validity are provided in Section 6. We review related work in Section 7.
Finally, Section 8 sketches future work and concludes the paper.

##  2 Background

The following example illustrates the need for personalized automated privacy
assistance that a user interacting with multiple systems at a time may
require. Then we briefly report the most relevant aspects for our research
taxonomies for privacy profiles proposed in the literature.

###  2.1 Motivating example

After a long day at work, Alice is at the subway station. After the pandemic
outbreak, she will meet pals at the cinema. She is on time but learns that she
cannot buy a ticket from the subway station attendant due to rigorous hygiene
regulations. In addition, vending machines are out of commission for contact-
less technology upgrades. Instead, a QR code and simple instructions to buy an
electronic ticket online are posted in front of the vending machines. Her
train is about to arrive, she opens her camera app and frames the QR code. The
site structure appears in a split second, but as Alice scrolls down to find
the ticket she needs, a popup asks for her privacy settings. Above a very long
list of radial button options about disclosing GPS position, information about
her mobile phone, consent to save various types of cookies on her device,
share her list of contacts, etc., she is presented with three buttons: _accept
all_ , _strictly necessary_ , _decline all_.

Alice is very concerned about her privacy, and when not strictly necessary for
the purpose she wants to perform, she does not wish to disclose private
information. Since the service she is asking for is simple as asking for a
one-ride ticket, she clicks _decline all_. The next page seems to load slowly,
images and structure are shown in a non-adaptive way, so she has to pinch-in
to zoom and scroll to read the text that informs her that a cryptographic key
used for her session management cannot be stored due to her preferences, so
the session is not secure also the page asks her to choose language, timezone,
type of device and the web browser she is using, payment options, etc. While
reading, Alice realizes that her train is about to arrive at the station. So,
she decides to click the back button on her browser, reload the page and click
_strictly necessary_ when prompted. The site then stays fast and steady,
adapted to the display of her device, prompting if she wants to take a one-
ride ticket or a full day one. Her mobile wallet handles the payment
instantly, and she receives her ticket just before the train comes. On time to
the cinema, Alice enjoys the film with her friends, soon forgetting the online
ticket purchase experience. Her preferences are saved on her phone, so she
will buy train tickets quickly and easily in the future. Alice does not know
that the _strictly necessary_ option, although excluding third-party tracking
and marketing, includes all alternatives that are strictly essential to all
services offered by the booking site, including – the lower price inter-city
ticket that requires GPS tracking, the discounted price for kids that requires
age disclosure, discount for army and state officials who must check other
installed mobile applications, as well as the train pass app to see if the
ticket is part of a booklet, etc.

Analogously to various studies notably Liu _et al._ [12], we believe that a
software technology should assist Alice in automatically selecting the options
that, on the one hand, are needed for what she wants to do and, on the other
hand, are compliant with her moral preferences.

In this work, we show that it is possible to protect users by first
understanding their privacy profiles, which can be automatically identified by
considering a small set of general and domain-independent questions that are
shown to be enough to reflect the user’s moral attitude. Thus, our approach is
to categorize personal privacy profiles from an ethical perspective [13].
Profiles can then be used to automate app and web settings, leveraging
recommender systems like in this paper or other technologies.

###  2.2 Categorizations of privacy profiles

Table 1: Privacy categories according to different taxonomies (Listed in chronological order). How important is privacy to you? |  NOTHING |  LITTLE |  QUITE |  VERY
---|---|---|---|---
Segmentation [1] |  Unconcerned |  Pragmatists |  Fundamentalist
Privacy Personas [14] |  Marginally concerned |  Amateurs |  Technicians |  Lazy Experts |  Fundamentalists
Philosophies [15] |  Fatalism |  Nothing to hide |  Something to hide |  Trade-off |  Personal Resp. |  Moral right
Privacy Clustering [7] |  Unconcerned |  Socially active |  Health-focused |  Minimal |  Anonymous (Strict)
Self-assessment [7] |  Conservative |  Unconcerned |  Fence-Sitter |  Advanced Users
Our proposed categorization |  INATTENTIVE |  INVOLVED/ATTENTIVE |  SOLICITOUS

Table 1 gives a summary of the most notable taxonomies of privacy categories.
Starting from the question: “ _How important is privacy to you?_ ” from left
to right of the table, an increasing level of privacy concerns is shown.
Westin [1] proposed the first categorization of user profiles with three
levels, i.e., _Unconcerned_ , _Pragmatists_ , and _Fundamentalist_. Since
then, there have been other studies that follow up and develop this initial
taxonomy. In particular, Dupre _et al._ [14] expanded it proposing five
categories: _Marginally concerned_ , _Amateurs_ , _Technicians_ , _Lazy
Experts_ , and _Moral right_. Schairer _et al._ [15] came up with even more,
i.e., six categories, where the answer _Little_ is split into _Nothing to
hide_ , and _Something to hide_ ; and _Quite_ is made of _Trade-off_ , and
_Personal Resp_. Recently, Sanchez _et al._ [7] proposed a more compact
categorization, where users are grouped into four categories, _Privacy
concerns_ , _Unconcerns_ , _Fence-Sitter_ , and _Advanced Users_. As it
appears from the table, category refinement happens in the middle category and
may depend on the application domain as well as on the amount of input data.
Further categories are contextual and can be obtained as specialization of
personal profiles based on the single user’s experience. Therefore, as
starting point three categories provide three clearly distinguishable moral
attitudes.

The categorization we propose names the three clusters as _Inattentive_ ,
_Attentive_ , and _Solicitous_. While _Inattentive_ means that users do not
care about privacy, _Solicitous_ corresponds to an opposite attitude, where
users are completely aware of privacy issues. The _Attentive_ category is
something in between, and covers both _Little_ and _Quite_ answers to the
question “ _How important is privacy to you?_ ”

##  3 Proposed Approach

Figure 1: The proposed approach.

Typically, users specify privacy preferences by directly interacting with the
privacy settings provided by the used software. Similar to other techniques
[13, 12, 8] we propose an approach that relies on a software layer that
automatically identifies privacy profiles and interacts with the user or the
software system to recommend privacy preferences accordingly.

Concerning what we present in this paper, the assisted selection phase of
privacy preferences started on training data consisting of general, domain-
specific, and app-specific answers given to the questions defined in [7] (see
“Full set of questions” in Figure 1). We have empirically analyzed the full
set of questions to identify the corresponding subset (consisting of general
questions) that is sufficient to automatically identify our three user privacy
profile categories, i.e., _Inattentive_ , _Attentive_ , and _Solicitous_ (see
activity 1 in Figure 1).

The automated creation of user privacy profiles phase (see activity 2 in
Figure 1) relies on an unsupervised clustering module, which can automatically
group users in the training data. The automated assignment of privacy profiles
to users phase (see 3) relies on a supervised classifier using a feed-forward
neural network to automatically assign to the given user the corresponding
privacy profile among one of those identified in 2. Finally, a recommender
system is used to further validate the activities 1 and 2, and to provide
users with privacy settings recommendations (see activity 4) according to the
privacy settings of other users belonging to privacy profiles as detected in
3.

Details on the results obtained from the performed Empirical Study are given
in the Experimental results section, whereas the activities 2, 3, and 4 are
described as follows.

###  3.1 Automated creation of user privacy profiles

To automatically create user privacy profiles, we employed a clustering
process by relying on the graph-based representation of users and privacy
settings as shown in Figure 2. This representation is also used by the
developed neural network for classifying users presented in Section 3.2.

Figure 2: Graph representation of users and privacy settings.

Each user $u$ is represented by a vector
$\phi=(\phi_{1},\phi_{2},..,\phi_{F})$, where $\phi_{i}$ is the weight of term
$s_{i}$, computed as the _term-frequency inverse document frequency_ value as
follows:

$\phi_{i}=f_{s_{i}}\times log(\frac{\left|P\right|}{a_{s_{i}}})$ (1)

The similarity between two users $u$ and $v$ is computed using their
corresponding feature vectors $\phi=\\{\phi_{i}\\}_{i=1,..,F}$ and
$\omega=\\{\omega_{j}\\}_{j=1,..,F}$ by means of the cosine similarity
function:

$sim(u,v)=\frac{\sum_{t=1}^{n}\phi_{t}\times\omega_{t}}{\sqrt{\sum_{t=1}^{n}(\phi_{t})^{2}}\times\sqrt{\sum_{t=1}^{n}(\omega_{t})^{2}}}$
(2)

where $n$ is the cardinality of all settings that were set to $1$ by both $u$
and $v$. Intuitively, $u$ and $v$ are characterized by using vectors in an
$n$-dimensional space, and Equation 2 measures the cosine of the angle between
them. As an example, in Fig. 2, we see that the two users $u_{2}$ and $u_{4}$
are similar since they both set two settings $s_{1}$ and $s_{3}$.

A set of $n$ users is grouped into $\kappa$ pre-defined number of clusters,
with the aim of maximizing both the similarity among instances within a single
cluster, and the dissimilarity among independent clusters. To this end, we
calculate the distance between every pair of users and feed as the input for
the clustering engine. The K-medoids algorithm [16] has been chosen to group
users into clusters due to its simplicity and efficiency.

In the clustering process, the distance scores, computed as
$d_{C}(u,v)=1-sim_{C}(u,v)$, are used to assign users to clusters. Initially,
a set of medoids (users) is generated randomly, then a medoid is selected as
the user in the cluster that has minimum average distance to all the other
users in the cluster. Afterwards, users are assigned to the cluster with the
closest medoid, using a greedy strategy [16].

###  3.2 Automated assignment of privacy profiles to users

Supervised learning algorithms can simulate humans’ learning activities,
mining knowledge from labeled data and performing predictions for unknown data
[17]. Among others, neural networks have been widely adopted in various
applications, including pattern recognitions [18], or forecasting [19]. A
feed-forward neural network consists of connected layers of neurons, where the
output of a layer is transferred to the next layer’s neurons, except for the
output layer.

Figure 3: A three-layer neural network.

We built a feed-forward neural network to classify users into different
privacy groups, using preferences as features. The network consists of three
layers explained as shown in Figure 3. The input layer has $L$ neurons, being
equal to the number of input settings, i.e., $X=(x_{1},x_{2},...,x_{L})$. The
middle layer consists of $M$ neurons, i.e., $H=(h_{1},h_{2},...,h_{M})$, $M$
can be configured during the evaluation. There are $\kappa$ neurons in the
output layer, corresponding to $\kappa$ output categories, i.e.,
$\hat{y}=(\hat{y}_{1},\hat{y}_{2},..,\hat{y}_{\kappa})$. The predicted value
$\hat{y}_{k}$ for neuron $k$ of the output layer is computed to minimize the
error between the real values and the predicted ones. As discussed in the
Experimental results section, the conceived neural network has played an
important role in the performed analysis, especially to understand to what
extent self-declared privacy profiles reflect the actual user category.

###  3.3 Privacy settings recommendation

We conceptualize PisaRec, a Privacy settings assistant running on top of a
Recommender system to provide users with suitable data protection
configurations. PisaRec works based on the assumption that _“if users of the
same privacy profile already share some common privacy settings, then they are
supposed to share additional similar settings”_ [20]. In this way, we utilize
the proposed graph-based representation to model the relationship among users
and use a collaborative-filtering algorithm [21] to recommend missing
settings. To feed as input for the recommendation engine, we adopt the _user-
item_ paradigm [22], in which each user corresponds to one row, a column
represents each setting. In this way, a cell in the matrix dictates the rating
given by a user to a setting. The two values 0 and 1 correspond to _deny_ and
_allow_ , respectively. An example of a user-setting matrix for the set of
four users and five settings is as follows: $u_{1}\ni s_{1},s_{2}$; $u_{2}\ni
s_{1},s_{3}$; $u_{3}\ni s_{1},s_{3},s_{4},s_{5}$; $u_{4}\ni
s_{1},s_{2},s_{4},s_{5}$. Accordingly, the user-item ratings matrix built to
model the occurrence of the settings is depicted in Figure 4.

Figure 4: A user-setting matrix.

The following collaborative-filtering formula is utilized [20] to predict the
inclusion of a setting $s_{i}$ for user $u$:

$r_{u,s_{i}}=\overline{r_{u}}+\frac{\sum_{v\in
topsim(u)}(r_{u,s_{i}}-\overline{r_{v}})\cdot sim(u,v)}{\sum_{v\in
topsim(u)}sim(u,v)}$ (3)

where $\overline{r_{u}}$ and $\overline{r_{v}}$ are the mean of the ratings of
$u$ and $v$, respectively; $v$ belongs to the set of _top-k_ most similar
users to $u$ or neighbour users, i.e., $topsim(u)$; $sim(u,v)$ is the
similarity between $u$ and a similar user $v$, computed using Equation 2.

The clusters obtained from the previous section allow us to identify users
with similar privacy preferences. Based on the obtained categorization, given
an input user, the neural network assigns the user to a specific category.
Afterward, we build a graph only for this category following the paradigm in
Figure 2. Such a sub-graph contains fewer nodes and edges than a full graph
for all categories, aiming to optimize the computation. On top of this,
PisaRec recommends missing settings to users. The outcome of the computation
is a ranked list of probable settings, and we select the top-N of them to
present as the final recommendations.

##  4 Evaluation

To study the proposed approach’s performance, we first introduce three
research questions. Afterward, we describe the dataset and metrics used in our
evaluation.

###  4.1 Research questions

The following research questions are considered to evaluate our proposed
approach.

  * •

RQ1: _How well does the users’ self-assessment reflect their privacy
category?_ As users in the considered dataset [23] have been allowed to self-
assess their privacy category, we examine if such a self-evaluation reflects
their real category.

  * •

RQ2: _Which sets of questions are relevant for assessing privacy concerns?_ We
are interested in finding the set of questions that can better distinguish
between user profiles. For this research question, we cluster the users with
different sets of features, and identify the one that brings the best
clustering solution. The aim is to find a set of privacy questions that better
represents the user profiles.

  * •

RQ3: _To which extent is PisaRec able to utilize the obtained categorization
in recommending relevant privacy settings to users?_ We investigate how well
the conceived recommender system learns from existing profiles, providing
users with additional configurations that reflect their preferences.

###  4.2 Dataset

We opted for an existing dataset that has been collected through a domain-
specific survey about the usage of a fitness app including user privacy
preferences [7]. As shown in Table 2, there are 444 data entries which have
been divided into three main groups as follows:

  * •

_Domain specific_ : This is the set of questions being explicitly related to
the fitness activity. There are a total of $202$ questions in this category.

  * •

_App related_ : These questions are about the use or setting of the app,
consisting of 113 questions.

  * •

_Generic_ : This set of questions consists of generic questions that are not
related to other groups. There are 129 generic questions in total.

Table 2: Summary of the dataset. Questions/Data | Alias |  Description | # entries
---|---|---|---
Domain specific | D |  Questions related to the specific domain (Fitness) | 202
D Subset 1 | DP1 |  Subset of the D set consisting of privacy relevant questions | 123
App related | A |  Questions related to the mobile application and the specific software context | 113
A Subset 1 | AP1 |  Subset of the A set consisting of privacy relevant questions | 65
A Subset 2 | AP2 |  Subset of the A set that includes only generalizable questions | 6
D + A Subset 0 | S0 |  Privacy related questions from the D and the A sets (DP1+AP1) | 188
Generic | G |  Generic questions not specifically related to the domain (fitness) or the application/software context (mobile app) | 129
G Privacy Subset 1 | GP1 |  Subset of the G set consisting of privacy relevant questions | 110
G Subset 1 | G1 |  Subset of the G set consisting of questions related to the disclosure of information about user’s identity with the app | 35
G Data 2 | G2 |  Data concerning the time spent by the users to answer the questionnaire, play with the simulator, and the sum of the two | 3
G Subset 3 | G3 |  Subset of the G set consisting of questions related to the user’s identity | 19
G Subset 4 | G4 |  Subset of the G set consisting of questions related to the disclosure of private information with the app | 56
G Subset 5 | G5 |  Subset of the G set consisting of questions related to the concerns about privacy | 16
Full dataset | DATA |  Data collected with the questionnaire and the simulator (D+A+G ) | 444

###  4.3 Evaluation metrics

  * •

Compactness. The metric measures how closely relevant the users within a
cluster are [24]. In this respect, a lower value represents a better
clustering solution and vice versa.

  * •

Silhouette. It measures how similar a user $u$ is to all the remaining users
of the same cluster [24], computed using the following formula:

$s(u)=\frac{(b(u)-a(u))}{max\\{a(u),b(u)\\}}$ (4)

where $a(u)$ is the mean distance between $u$ and the others, $b(u)$ is the
minimum mean distance. A silhouette value falls into the range [-1,..+1],
where a higher score means a better clustering solution.

Furthermore, we also use Precision, Recall, ROC curve and AUC to study the
performance of the proposed approach.

First, there are the following definitions: True positive (TP) is the settings
that match with ground-truth data; False positive (FP) is the recommended
settings but do not match with the ground-truth data; False negative (FN): the
settings that should be recommended, but they are excluded. Then, the metrics
are as follows:

  * •

Precision and Recall. Precision measures the fraction of the number of
settings properly classified to the total number of recommended items and
Recall (or true positive rate – TPR) is the ratio of the number of correctly
classified items to the total number of items in the ground-truth data. The
metrics are defined as follows:

$P=\frac{TP}{TP+FP}$

$R=\frac{TP}{TP+FN}=TPR$

  * •

False positive rate (FPR). This metric measures the ratio of the number of
items that are falsely classified into a category c, to the total number of
items that are either correctly not classified, or falsely classified into the
category:

$FPR=\frac{FP}{TN+FP}$

  * •

ROC curve and AUC. The relationship between FPR and TPR is sketched in a 2D
space, using a receiver operating characteristic (ROC) [25], which spans from
(0,0) to (1,1). An ROC close to the upper left corner represents a better
prediction performance.

##  5 Experimental results

This section reports and analyzes the experimental results by answering the
research questions introduced in Section 4.1.

###  5.1 RQ1: _How well does the users’ self-assessment reflect their privacy
category?_

In the dataset [7] considered in our evaluation, each user has assigned
themselves to one of the following four groups: _Privacy Conservative_ (Class
0), _Unconcerned_ (Class 1), _Fence-Sitter_ (Class 2), and _Advanced User_
(Class 3). We investigate if the self-assessment is consistent, i.e., if all
the users properly perceive their real privacy category. This is important
since a proper self-clustering can be utilized in additional profiling
activities.

Figure 5: ROC curves with generic questions.

We conducted evaluation using the conceived neural network as the classifier.
Such a technique has been successfully applied to classify various types of
data, e.g., text [26], chemical patterns [27], metamodels [28], to name a few.
Similarly, we use the privacy settings as features, and the labels specified
by humans to train the classifier. We opt for the ten-fold cross validation
technique [29], where the dataset is split into ten equal parts, and the
evaluation is done in ten rounds. The evaluation metrics are computed on the
test set, i.e., for each user the network predicts a label, which is then
compared with the self-assessed label to evaluate the performance. Finally,
ROC curves are sketched by combining the scores obtained from all the ten
folds.

Figure 6: ROC curves with domain specific questions.

Figure 5 and Figure 6 depict the ROC curves obtained from the classification
results for generic and domain specific questions. It is evident that the
classifier achieves very low prediction performance on both configurations. In
particular, the curves bend over the diagonal line, being close to a random
guess. Moreover, the AUC values of the four categories are always lower than
0.65. In other words, we encounter _negative results_ , where the neural
network fails to predict a proper category for a user. These results suggest
that there are noises in the training data [30], which could possibly be both
in the features and the labels.

To confirm the hypothesis, we measure the similarity between each user and all
the remaining others. Interestingly, we found out that 96.20% of the users
have very similar users in completely different self-assessed categories. This
demonstrates that while users share similar preferences, they classify
themselves differently, causing a low prediction performance for the neural
network.

Answer to RQ1. The self-assessment given by users does not reflect their real
privacy category: Users with highly similar settings perceive themselves as
completely different groups. In practice, this means administrators should not
rely on such a self-categorization, but have to perform privacy profiling on
their own.

###  5.2 RQ2: _Which sets of questions are relevant for assessing privacy
concerns?_

As seen in RQ1, the self-assessment given by users is not consistent, thus it
is necessary to find another way to group users into clusters. We performed
experiments on different subsets of the questionnaire to study the influence
of each set on the clustering results. The ultimate aim is to identify a set
of questions that helps classify users better. In particular, we are
interested in analyzing the following groups of questions:

  * •

_QS 1_: It is a set of question sets as follows: Domain specific (D); App
related (A); Generic (G) and their combination (i.e., D+A+G) named COM..
Furthermore, we also include the set G+AP2 where AP2 contains generalizable
questions like _“Do you believe the company providing this fitness tracker is
trustworthy in handling your information?”_ Indeed, this is to ask if the
company is trustworthy, and therefore we consider it as a general question.
_QS 1_ permits to compare compactness and silhouette performances between a
single set and combinations of all questions.

  * •

_QS 2_: It is a set of questions sets as follows: DP1, AP1, and GP1 that are
the subsets of D, A, and G consisting only of privacy relevant questions,
respectively. COM. is the union of the three subsets, i.e., COM. =
DP1+AP1+GP1. _QS 2_ permits to understand the actual influence of the privacy-
related questions.

  * •

_QS 3_: It consists of subsets of generic questions G defined as follows: G1
are the questions related to disclosure of information about user’s identity
with the app; G2 the questions related to the time spent by the user in
completing the survey; G3 the questions related to user’s identity; G4 the
questions related to disclosure of private information with the app; G5 the
questions related to concerns about privacy. COM. is the combination of all
the subsets, i.e., COM. = G1+G2+G3+G4+G5. _QS 3_ is to ascertain the influence
of the generic questions with respect to the overall set of questions.

|
---|---
|
|

(a) Compactness for QS1

(b) Silhouette for QS1

(c) Compactness for QS2

(d) Silhouette for QS2.

(e) Compactness for QS3

(f) Silhouette for QS3.

Figure 7: Compactness and silhouette scores.

We compute and report for each set the corresponding compactness and
silhouette scores. Figure 7(a), Figure 7(c), and Figure 7(e) report the
compactness scores computed for the three question sets.

As it can be seen in Figure 7(a), using A as input yields the most compact
clusters. In particular, most of the scores are smaller than 40. When domain
specific questions (D) are used as the features, we also obtain low
compactness scores, albeit being larger than using A. If only generic
questions, i.e., G, are utilized, worse clustering solutions are seen. When
comparing the results obtained by using G with those of using G+AP2, we can
see that adding AP2 to G contributes to a better clustering. Concerning QS2
where only privacy relevant questions are considered, we see that using domain
specific privacy relevant questions (DP1) allows us to gain the most
discriminative clusters. Using the subset of privacy relevant questions, i.e.,
AP1 is also beneficial to the clustering of user profiles.

For QS3, there are comparable clustering solutions when using the features
sets G1, G3, G4, and G5. The best clustering is obtained with G2.

The silhouette scores in Figure 7(b), Figure 7(d), and Figure 7(f) further
enforce the compactness ones. A is the feature set that achieves the best
silhouette for QS1. Adding AP2 to G helps achieve a better clustering
solution, compared to using only G.

Answer to RQ2. According to the performed evaluation, generic questions plus
generalizable ones (i.e., G+AP2) provide the best clustering solution.

###  5.3 RQ3: _To which extent is PisaRec able to utilize the obtained
categorization in recommending relevant privacy settings to users?_

An issue with clustering is whenever there is a new user to be classified, it
is necessary to re-run the whole process. This is a time consuming phase,
especially where there is a large number of users. Thus, we propose a more
feasible way to assign new users to clusters, avoiding repetitive clustering.

Figure 8: ROC curves, three categories.

Given that there is an existing categorization of user profiles, the feed-
forward neural network presented in Section 3.2 is used to classify a new user
into a suitable group. Once clusters have been obtained, we feed them as input
to train the neural network and perform the testing using the ten-fold cross-
validation procedure. It is worth mentioning that we use three clusters
instead of four as explained in Section 2.2.

The final performance measured by means of ROC curves is depicted in Figure 8.
In particular, the AUC values for Class $0$, Class $1$, and Class $2$ are
$0.83$, $0.85$, and $0.76$, respectively. The curves representing the three
classes reside near the upper left corner, implying a good prediction
performance. Overall, the curves and the AUC values demonstrate that the
obtained performance is much better compared to that before clustering in
Figure 5 and Figure 6. This suggests that properly clustering user profiles
can substantially increase the neural network’s prediction performance.

Next, we validate the performance of PisaRec as follows. We opted for the ten-
fold cross validation technique [29], where the dataset is split into ten
equal folds, and the evaluation is done in ten rounds. By each round, one fold
is utilized as testing, and the other nine folds are merged to create the
training data. In a testing fold, for each user, the features are split into
two parts, one part is fed as query, and the remaining part is removed to be
used as ground-truth data. The ratio of the number of settings used as query
to the total number of settings is called $\alpha$. This simulates a real
scenario, where the user already specified some settings, and the system is
expected to recommend the rest, corresponding to the ground-truth data. For
each user, PisaRec returns a ranked list of $N$ settings ($N$ is
configurable), and the evaluation metrics are computed on the test set as
follows. The recommended items are then compared with the ground-truth data to
evaluate the performance. Eventually, we average out the metrics obtained from
the testing folds to produce the final results.

We experiment with different configurations by varying $\alpha$, $k$: the
number of neighbor users used for the computation, $N$: the number of
recommended items. In particular, $\alpha=\\{0.1,0.3,0.5\\}$;
$k=\\{3,5,10,15\\}$; and $N$ is varied from 1 to 50, simulating a real-world
scenario where users have to set several settings. The precision-recall curves
are then sketched following these parameters.

Figure 9: Configuration C1.

As seen in Figure 9, when $\alpha=0.1$, i.e., only a small amount of data is
used as query, PisaRec recommends relevant settings to users, however with
considerably low precision and recall. For instance, when $k=3$, a maximum
precision of 0.52 is obtained and the maximum precision is $0.7$ when $k=15$.
Similarly, the recall scores are low, i.e., smaller than 0.4 by all the
configurations. Altogether, this implies a mediocre performance which is
understandable as the configuration with $\alpha=0.1$ corresponds to the case
where the user only specified a few settings, and the system has limited
context to recommend additional settings.

Figure 10: Configuration C2.

When we increase $\alpha$ to $0.3$, there is an improvement in both precision
and recall as in Figure 10, compared to the results obtained with $\alpha=0.1$
in Figure 9. Precision scores are always larger than 0.55 in the
configurations, with 0.80 being the maximum value. Similarly, we also see that
recall scores are gradually improved. For instance, a maximum recall of 0.35
is achieved with $k=3$, and the corresponding maximum for $k=15$ is $0.48$.

Figure 11: Configuration C3.

Such an improvement is more evident when $\alpha=0.50$, i.e., a half of the
settings is used as query. In Figure 11, apart from some outliers, most of the
precision scores are larger than 0.70, with 0.85 as the maximum value.
Compared to the previous configurations with $\alpha=0.1$ and $\alpha=0.3$,
the recall scores are also better, i.e., with a longer list of items, recall
increases substantially. In particular, a recall of $0.73$ is seen when $k=10$
and $k=15$.

Concerning the number of neighbors used for computing recommendations, i.e.,
$k$ (see Section 3.3 and Formula 3), by considering Figure 9, Figure 10, and
Figure 11 together, it is evident that adding more users for the computation
contributes to a better prediction performance. For instance, by increasing k
from $3$ to $5$, $10$, and $15$, we boost both precision and recall by all the
cut-off values $N$.

Altogether, the experimental results show that even if users perceive their
categories differently as shown in RQ1, once we have identified their right
privacy group PisaRec can exploit the categories to provide relevant settings
to users, though the considered dataset is pretty small. We anticipate that
its performance can be further enhanced, if there is more data for training.

Answer to RQ3. PisaRec recommends highly relevant settings to a user, though
there is limited amount of data available for training. The prediction
performance improves alongside the amount of data fed as input.

##  6 Discussion

This section provides discussion related to the possible extensions of our
work, as well as the threats to validity of our findings.

###  6.1 Extendability

Dataset. In our work, we utilized a small dataset for the evaluation. The
amount of training data may impact the performance of both the clustering and
classification phases. Moreover, as PisaRec is a collaborative-filtering
recommender system, its performance is heavily driven by the quality and
amount of data. We anticipate that we may need to calibrate the systems’
parameters to maintain both timing efficiency and effectiveness with more
data.

The unsupervised algorithm. In the scope of this paper, we used the K-Medoids
algorithm to cluster the user profiles. Such a technique has been chosen due
to its simplicity and effectiveness. In fact, several clustering algorithms
could be employed to categorize user profiles. Thus, the outcome of a
clustering solution depends heavily on the considered techniques. We plan to
extend our work by considering other clustering algorithms, such as CLARA
[31], or DBSCAN [32].

The supervised classifier. The neural network used to classify user profiles
may be suitable only for the considered dataset. For a different dataset, it
is necessary to find adequate network configurations employing an empirical
evaluation. For instance, the number of hidden layers, or the number of
neurons for each layer, should be considerably increased to deal with a larger
number of user profiles.

###  6.2 Threats to validity

We are aware of the existence of some threats that might harm the validity of
the performed experiments as they are presented as follows.

  * •

Threats to _construct validity_ are related to any factor that can compromise
the validity of the given observations. The main threat to construct validity
is related to the size of the analyzed data. The used dataset is indeed
relatively small but has the advantage of coming from a recent work [7] thus
reflecting users’ contemporary privacy behaviors. More extensive experiments
are under planning encompassing other ethical dimensions beyond privacy.

  * •

Concerning the threats to _internal validity_ , i.e., any confounding factor
that could influence our findings, we attempted to avoid any bias in the
automatic creation of user profiles and in the way we split the full data into
groups. We tried to mitigate this threat by semantically analyzing and double-
checking the clusters obtained by the proposed approach.

  * •

Concerning the threats to _external validity_ , they are related to the
generalizability of our results. This is about checking the adequacy of our
privacy profiles in other contexts, notably, in the traveling or IoT domains.
Generalizability is actually our initial driver for extracting privacy
profiles from general moral questions. Thus, further experimental evidence is
planned to support the reported paper results.

##  7 Related work

This section reviews the related work and their main characteristics to
position our approach in the current scenario for eliciting, profiling, and
predicting user privacy preferences.

###  7.1 Overview

The work presented in this paper has been done in the context of the design of
the EXOSOUL research project that aims at providing users with a personalized
software layer that mediates users’ interactions with the digital world
according to user’s ethics, including privacy preferences [33, 13].

According to various studies [34, 35], the vast majority of users do not
bother to read privacy agreements because of the excessive language and
confusing explanations [36, 37, 38, 39]; it is unreasonable also to expect
they will read them on a regularly basis [40]. Resignation from privacy
choices may also be a result of their dissatisfaction with the lack of options
and excessive complexity [41].

Privacy profiling is at the core of our work therefore, most related studies
are on user clustering, privacy profiling, and privacy preferences settings.
The more significant part of existing studies about privacy profiling develop
on the work of Westin [1]. Based on a series of privacy-related surveys, the
author established “ _Privacy Indexes_ ” for most of these polls to summarize
results, indicate trends in privacy concerns, and suggest a widely recognized
segmentation methodology of “ _Privacy Profiles._ ”The methodology he applied
classifies people into three categories: privacy fundamentalists, pragmatists,
and unconcerned. Because of the commercial nature of Westin’s surveys, the
methodology and the details of how privacy indexes were calculated are not
fully disclosed, so we rely on subsequent works [2] that deeply analyzed and
reported them.

Westin Segmentation, and particularly the pragmatism adherence of the
consumers were criticized by the work of Hoofnagle and Urban [42, 43]. Their
experimental work investigated customer expectations for privacy safeguards,
showing that many people believe they have greater protection than they really
do due to a lack of knowledge about corporate practices, privacy policies, and
data usage limits. Westin’s methodology has been applied, revised, and
expanded in several empirical studies on privacy that include collected data.
The categorizations that are most relevant to our research are reported in
Table 1.  Dupree _et al._ [14] analyzed data from surveys and participants
interviews, the authors identified five user clusters that emerge from end-
user behaviors, including _Fundamentalists_ , _Lazy Experts_ , _Technicians_ ,
_Amateurs_ and the _Marginally Concerned_. Schairer _et al._ [15] presented a
model of privacy disposition and its development based on qualitative research
on privacy considerations in the context of emerging health technologies. The
authors identified six clusters, including _Fatalism_ , _Nothing to hide_ ,
_Something to hide_ , _Tradeoff_ , _Personal responsibility_ , _Moral right_.
In the research proposed by Sanchez _et al._ [7] the authors presented the
results of a fitness-related simulation and questionnaire to classify users
according to their privacy-related preferences. They used two different sets
of labels for their clusters, one for the computed privacy-profile assignment
consisting of six groups and one for self-assessment they proposed to the
users consisting in four groups. The first clusters were labeled as:
_Unconcerned_ , _Socially Active_ , _Health-focused_ , _Minimal_ , _Anonymous_
, _Strict_. The second clusters were labeled as: _Privacy Conservative_ ,
_Unconcerned_ , _Fence-Sitter_ , _Advanced User_.

###  7.2 Profiling and Clusterization

Different approaches were used in recent works to create user profiling
starting from data collection and analysis. Lee and Kobsa [5] performed a
cluster analysis on online survey data composed of IoT scenarios and user
responses like reaction parameters. Because all parameters have either
categorical or ordinal values, the authors utilized K-modes, a variant of the
K-means clustering algorithm. Qin _et al._ [4] proposed a user’s preferences
prediction through clusterization of partial preference relations on the
MovieLens dataset, which is commonly used to test collaborative filtering
technology. According to Fernquist _et al._ [fernquist2017iot], users may also
be identified by their data profiles created by their device based on time and
events: researchers gathered information on how and when people use their
networked devices, recording the time period in which a user interacts or
transmits data and the specific place. For the sake of interpreting their
findings, the researchers took into account three distinct sorts of events:
voice calls, texts, and data transfers, as well as combinations of these. The
findings showed that the profiles studied may be used to identify the user.

###  7.3 Automating privacy settings

Concerning automating privacy preferences settings, the closest approach is by
Liu _et al._ [6, 8] and by the Personalized Privacy Assistant Project team
[23]. Their approach employs user categorizations that are obtained by mining
existing privacy settings in the app domain, complemented with an initial
dialogue with the user to select the appropriate profile. Our approach is also
based on privacy profiles. However, they are obtained by analyzing data
resulting from questions that relate to the user’s ethics and are not
concerned with any specific domain.

Wilson _et al._ [44] identified the impact of privacy profiles on the
preferences, sharing inclinations, and overall satisfaction levels of users of
location-sharing apps. Their findings demonstrate that privacy profiles for
location sharing settings can have a long-lasting impact on how users perceive
their privacy, even in the face of ongoing opportunities to reflect on the
sharing outcomes that result from their chosen settings. This implies that
attempts to simplify privacy settings should be performed with caution since
such simplicity may easily impact the people with whom the settings are
intended to interact and educate.

Brandimarte  _et al._ [45] investigated the concept that giving people a
greater sense of control over the release and access to private information –
even information that enables them to be personally identified – would improve
their willingness to provide sensitive information. If their desire to reveal
adequately rises, this control may, counter-intuitively, result in being more
slack.

In their research, participants in a publication were informed that a profile
comprising their information would be produced automatically and published
online once the website was finished. Other participants were informed that
only half of their profiles would be published online. The uncertain
publishing condition was designed to reduce participants’ sense of control
over the public distribution of their survey responses without actually
decreasing access by others. Their theories predicted that decreasing control
would limit the desire to reveal in the uncertain publishing condition,
notwithstanding lower external costs or hazards. According to the researchers’
results, if individuals behave in enough offsetting manner, devices supposed
to safeguard them might instead end up worsening the hazards they confront.

###  7.4 Surveys and regulations

Based on previous research in survey technique and related domains, Redmiles
_et al._ [46] provide a set of important recommendations for conducting self-
report usability studies. There are established criteria and suggestions for
collecting good quality self-report data in other sectors that depend on self-
report data, such as health and social sciences. We used this information as a
guideline for selecting and refining question groups.

As discussed by Emami-Naeini  _et al._ [47], surveys and interviews can be
administered with consolidated methodologies like the Delphi Method. This
method is “ _a method for the systematic solicitation and collection of
judgments on a particular topic through a set of carefully designed sequential
questionnaires interspersed with summarized information and feedback of
opinions derived from earlier responses_ ” [48]. Using a three-round Delphi
process, the authors conducted an expert elicitation study with 22 privacy and
security experts to identify the factors that experts believe are important
for consumers to consider when comparing the privacy and security of IoT
devices to inform their purchase decisions. The same methodology could be used
to elicit preferences from the users.

Considering the research theme, we took into account the General Data
Protection Regulation (GDPR) [49], the document that governs the storing,
processing, and use of personal data by the European Union (EU) as of May 25,
2018. Even if not based in the EU, the GDPR applies to all third parties that
operate in the EU market or access the data of EU citizens.

##  8 Conclusion and future work

This paper proposes a holistic approach consisting of both supervised and
unsupervised learning to identify privacy profiles. By finding a set of
questions suitable for assessing privacy profiles, we recommend relevant
privacy preferences to users. An empirical study on the proposed system using
a fitness dataset shows that generic questions are suitable for categorizing
user profiles and recommending privacy settings. For future work, besides
developing further experimental evidence supporting the results reported in
this paper, we will work in the direction of building user profiles that cover
other ethical dimensions beyond privacy. Last but not least, in the scope of
the Exosoul project, we will deploy the conceived techniques to analyze data
collected from users, studying the characteristics of users’ behaviors and
their attitudes in the digital world.

## Acknowledgements

The research described in this paper has been carried out as part of the
EXOSOUL project and also partially supported by the Centre of excellence EX-
EMERGE (Centre of EXcellence on Connected, Geo-localized and Cyber-secure
vehicles) of the University of L’Aquila, Italy.

## References

  * [1] A. F. Westin, Bibliography of surveys of the u.s. public, 1970-2003 (2003).
URL
https://web.archive.org/web/20051224000944/http://www.privacyexchange.org/iss/surveys/surveybibliography603.pdf

  * [2] P. Kumaraguru, L. F. Cranor, Privacy Indexes: A Survey of Westin’s Studies, Tech. Rep. CMU-ISRI-5-138, Institute for Software Research International, School of Computer Science, Carnegie Mellon University, Pittsburgh, PA (Dezember 2005).
  * [3] Y. Zhao, Y. Yu, Y. Li, G. Han, X. Du, Machine learning based privacy-preserving fair data trading in big data market, Information Sciences 478 (2019) 449–460.  doi:https://doi.org/10.1016/j.ins.2018.11.028.
URL https://www.sciencedirect.com/science/article/pii/S0020025518309174

  * [4] M. Qin, S. Buffett, M. W. Fleming, Predicting user preferences via similarity-based clustering, in: S. Bergler (Ed.), Advances in Artificial Intelligence, Springer Berlin Heidelberg, Berlin, Heidelberg, 2008, pp. 222–233.
  * [5] H. Lee, A. Kobsa, Understanding user privacy in internet of things environments, in: 2016 IEEE 3rd World Forum on Internet of Things (WF-IoT), 2016, pp. 407–412.  doi:10.1109/WF-IoT.2016.7845392.
  * [6] J. Lin, B. Liu, N. Sadeh, J. I. Hong, Modeling users’ mobile app privacy preferences: Restoring usability in a sea of permission settings, in: 10th Symposium On Usable Privacy and Security ($\\{$SOUPS$\\}$ 2014), 2014, pp. 199–212.
  * [7] O. R. Sanchez, I. Torre, Y. He, B. P. Knijnenburg, A recommendation approach for user privacy preferences in the fitness domain, User Modeling and User-Adapted Interaction 30 (3) (2020) 513–565.  doi:10.1007/s11257-019-09246-3.
URL https://doi.org/10.1007/s11257-019-09246-3

  * [8] B. Liu, Can Machine Learning Help People Configure Their Mobile App Privacy Settings? (1 2020).  doi:10.1184/R1/11591340.v1.
URL
https://kilthub.cmu.edu/articles/thesis/Can_Machine_Learning_Help_People_Configure_Their_Mobile_App_Privacy_Settings_/11591340

  * [9] McKinsey, How COVID-19 has pushed companies over the technology tipping point—and transformed business forever (https://mck.co/3trP4OV), 2020.
URL https://mck.co/3trP4OV

  * [10] OECD, OECD Digital Economy Outlook 2020, 2020.  doi:https://doi.org/https://doi.org/10.1787/bb167041-en.
URL https://www.oecd-ilibrary.org/content/publication/bb167041-en

  * [11] A. Pujahari, D. S. Sisodia, Modeling side information in preference relation based restricted boltzmann machine for recommender systems, Information Sciences 490 (2019) 126–145.  doi:https://doi.org/10.1016/j.ins.2019.03.064.
URL https://www.sciencedirect.com/science/article/pii/S0020025519302774

  * [12] B. Liu, M. S. Andersen, F. Schaub, H. Almuhimedi, S. Zhang, N. Sadeh, A. Acquisti, Y. Agarwal, Follow my recommendations: A personalized privacy assistant for mobile app permissions, in: Proceedings of the Twelfth USENIX Conference on Usable Privacy and Security, SOUPS ’16, USENIX Association, 2016, pp. 27–41, event-place: Denver, CO, USA.
  * [13] M. Autili, D. D. Ruscio, P. Inverardi, P. Pelliccione, M. Tivoli, A software exoskeleton to protect and support citizen’s ethics and privacy in the digital world, IEEE Access 7 (2019) 62011–62021.  doi:10.1109/ACCESS.2019.2916203.
URL https://doi.org/10.1109/ACCESS.2019.2916203

  * [14] J. L. Dupree, R. Devries, D. M. Berry, E. Lank, Privacy personas: Clustering users via attitudes and behaviors toward security practices, in: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, CHI ’16, Association for Computing Machinery, New York, NY, USA, 2016, p. 5228–5239.  doi:10.1145/2858036.2858214.
URL https://doi.org/10.1145/2858036.2858214

  * [15] C. E. Schairer, C. Cheung, C. Kseniya Rubanovich, M. Cho, L. F. Cranor, C. S. Bloss, Disposition toward privacy and information disclosure in the context of emerging health technologies, Journal of the American Medical Informatics Association 26 (7) (2019) 610–619.  arXiv:https://academic.oup.com/jamia/article-pdf/26/7/610/34151415/ocz010.pdf, doi:10.1093/jamia/ocz010.
URL https://doi.org/10.1093/jamia/ocz010

  * [16] H.-S. Park, C.-H. Jun, A simple and fast algorithm for k-medoids clustering, Expert Systems with Applications 36 (2, Part 2) (2009) 3336–3341.  doi:https://doi.org/10.1016/j.eswa.2008.01.039.
URL https://www.sciencedirect.com/science/article/pii/S095741740800081X

  * [17] K. Gurney, An Introduction to Neural Networks, Taylor and Francis, Inc., USA, 1997\.
  * [18] C. M. Bishop, Neural networks for pattern recognition, Oxford University Press, Inc., New York, NY, USA, 1995.
  * [19] G. Zhang, B. Eddy Patuwo, M. Y. Hu, Forecasting with artificial neural networks:: The state of the art, International Journal of Forecasting 14 (1) (1998) 35–62.
  * [20] J. B. Schafer, D. Frankowski, J. Herlocker, S. Sen, The adaptive web, Springer-Verlag, Berlin, Heidelberg, 2007, pp. 291–324.
  * [21] C. Aggarwal, Neighborhood-based collaborative filtering, in: Recommender systems: The textbook, Springer International Publishing, Cham, 2016, pp. 29–70.
  * [22] T. D. Noia, V. C. Ostuni, Recommender systems and linked open data, in: W. Faber, A. Paschke (Eds.), Reasoning Web. Web Logic Rules - 11th International Summer School 2015, Berlin, Germany, July 31 - August 4, 2015, Tutorial Lectures, Vol. 9203 of Lecture Notes in Computer Science, Springer, 2015, pp. 88–113.  doi:10.1007/978-3-319-21768-0\\_4.
URL https://doi.org/10.1007/978-3-319-21768-0_4

  * [23] N. Sadeh, B. Liu, A. Das, M. Degeling, F. Schaub, Personalized privacy assistant, uS Patent 10,956,586 (Mar. 23 2021).
  * [24] Y. Liu, Z. Li, H. Xiong, X. Gao, J. Wu, Understanding of internal clustering validation measures, in: Proceedings of the 2010 IEEE International Conference on Data Mining, ICDM ’10, IEEE Computer Society, USA, 2010, p. 911–916.  doi:10.1109/ICDM.2010.35.
URL https://doi.org/10.1109/ICDM.2010.35

  * [25] T. Fawcett, An introduction to roc analysis, Pattern Recogn. Lett. 27 (8) (2006) 861–874.  doi:10.1016/j.patrec.2005.10.010.
URL https://doi.org/10.1016/j.patrec.2005.10.010

  * [26] S. Minaee, N. Kalchbrenner, E. Cambria, N. Nikzad, M. Chenaghlu, J. Gao, Deep learning–based text classification: A comprehensive review, ACM Comput. Surv. 54 (3) (Apr. 2021).  doi:10.1145/3439726.
URL https://doi.org/10.1145/3439726

  * [27] J. A. Burns, G. M. Whitesides, Feed-forward neural networks in chemistry: Mathematical systems for classification and pattern recognition, Chem. Rev. 93 (1993) 2583–2601, 377.
  * [28] P. T. Nguyen, J. Di Rocco, D. Di Ruscio, A. Pierantonio, L. Iovino, Automated classification of metamodel repositories: A machine learning approach, in: 2019 ACM/IEEE 22nd International Conference on Model Driven Engineering Languages and Systems (MODELS), 2019, pp. 272–282.  doi:10.1109/MODELS.2019.00011.
  * [29] R. Kohavi, A study of cross-validation and bootstrap for accuracy estimation and model selection, in: Proceedings of the 14th international joint conference on artificial intelligence - volume 2, IJCAI’95, Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 1995, pp. 1137–1143.
  * [30] X. Zhu, X. Wu, Class noise vs. attribute noise: A quantitative study of their impacts, Artif. Intell. Rev. 22 (3) (2004) 177–210.
  * [31] L. Kaufman, P. J. Rousseeuw, Finding Groups in Data: An Introduction to Cluster Analysis., John Wiley, 1990.
  * [32] M. Ester, H.-P. Kriegel, J. Sander, X. Xu, A density-based algorithm for discovering clusters in large spatial databases with noise, in: Proceedings of the Second International Conference on Knowledge Discovery and Data Mining, KDD’96, AAAI Press, 1996, p. 226–231.
  * [33] Exoskeleton, The software exoskeleton.
URL https://exosoul.disim.univaq.it/

  * [34] G. R. Milne, M. J. Culnan, Strategies for reducing online privacy risks: Why consumers read (or don’t read) online privacy notices, Journal of interactive marketing 18 (3) (2004) 15–29.
  * [35] J. A. Obar, A. Oeldorf-Hirsch, The biggest lie on the internet: Ignoring the privacy policies and terms of service policies of social networking services, Information, Communication & Society 23 (1) (2020) 128–147.
  * [36] J. Bhatia, T. D. Breaux, J. R. Reidenberg, T. B. Norton, A theory of vagueness and privacy risk perception, in: 2016 IEEE 24th International Requirements Engineering Conference (RE), IEEE, 2016, pp. 26–35.
  * [37] C. Jensen, C. Potts, Privacy policies as decision-making tools: an evaluation of online privacy notices, in: Proceedings of the SIGCHI conference on Human Factors in Computing Systems, 2004, pp. 471–478.
  * [38] A. M. McDonald, R. W. Reeder, P. G. Kelley, L. F. Cranor, A comparative study of online privacy policies and formats, in: International Symposium on Privacy Enhancing Technologies Symposium, Springer, 2009, pp. 37–55.
  * [39] J. R. Reidenberg, T. Breaux, L. F. Cranor, B. French, A. Grannis, J. T. Graves, F. Liu, A. McDonald, T. B. Norton, R. Ramanath, Disagreeable privacy policies: Mismatches between meaning and users’ understanding, Berkeley Tech. LJ 30 (2015) 39.
  * [40] A. M. McDonald, L. F. Cranor, The cost of reading privacy policies, Isjlp 4 (2008) 543.
  * [41] J. Colnago, Y. Feng, T. Palanivel, S. Pearman, M. Ung, A. Acquisti, L. F. Cranor, N. Sadeh, Informing the design of a personalized privacy assistant for the internet of things, in: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 2020, pp. 1–13.
  * [42] C. J. Hoofnagle, J. M. Urban, Alan westin’s privacy homo economicus, Wake Forest L. Rev. 49 (2014) 261.
  * [43] J. M. Urban, C. J. Hoofnagle, The privacy pragmatic as privacy vulnerable, in: Symposium on Usable Privacy and Security (SOUPS 2014) Workshop on Privacy Personas and Segmentation (PPS), 2014.
  * [44] S. Wilson, J. Cranshaw, N. Sadeh, A. Acquisti, L. F. Cranor, J. Springfield, S. Y. Jeong, A. Balasubramanian, Privacy manipulation and acclimation in a location sharing application, in: Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing, 2013, pp. 549–558.
  * [45] L. Brandimarte, A. Acquisti, G. Loewenstein, Misplaced confidences: Privacy and the control paradox, Social psychological and personality science 4 (3) (2013) 340–347.
  * [46] E. M. Redmiles, Y. Acar, S. Fahl, M. L. Mazurek, A summary of survey methodology best practices for security and privacy researchers, Tech. rep. (2017).
  * [47] P. Emami-Naeini, Y. Agarwal, L. F. Cranor, H. Hibshi, Ask the experts: What should be on an iot privacy and security label?, in: 2020 IEEE Symposium on Security and Privacy (SP), IEEE, 2020, pp. 447–464.
  * [48] C. R. Atherton, Group techniques for program planning: A guide to nominal group and delphi processes. by andré l. delbecq, andrew h. van de ven, and david h. gustafson. glenview, ill.: Scott, foresman & co., 1975. 174 pp. $4.75$ paper and interpersonal conflict resolution. by alan c. filley (1976).
  * [49] T. E. Parliament, the Council of the European Union, Eu general data protection regulation (gdpr) - regulation eu 2016/679 of the european parliament and of the council of 27 april 2016, Official Journal of the European Union (2016).