# Detecting Features Concerning Privacy From App Reviews

Jianzhang Zhang1, Jinping Hua1, Yiyang Chen1, Nan Niu2, Chuang Liu1  1Alibaba
Business School, Hangzhou Normal University, Hangzhou, China  2Department of
Electrical Engineering & Computer Science, University of Cincinnati, USA
<EMAIL_ADDRESS>{chenyiyang<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Privacy requirements not only relate to legal compliance but also influence
user satisfaction. Massive rapidly increasing App reviews have been proved a
valuable requirements knowledge repository. Existing studies on App reviews
mining have made much effort to automatically extract various requirements
related information, e.g., feature request, bug report, and user opinions.
However, less attention has been paid to privacy requirements refinement based
on user reviews mining, which is beneficial for addressing users’ privacy
concern. In this work, we aim to detect privacy related features from App
reviews to facilitate software maintenance activities. To that end, we design
a semi-automatic framework to identify privacy related reviews from which App
features are extracted and mapped to those listed in App descriptions.
Firstly, we combine information retrieval and supervised text classification
to identify privacy related reviews. Then, we design a dependency parsing
method to extract App features from those privacy related reviews. Finally,
those automatically extracted features are matched with those manually
annotated ones in App descriptions based on phrase similarity. We
quantitatively evaluate the three components of our framework on the reviews
of Apps from multiple categories. For privacy related reviews identification,
Gradient Boosting classifier achieves the highest F1 score of 93.77% among
other competitive algorithms including deep learning ones. On App feature
extraction, our dependency parsing based method can achieve a recall of
85.63%, more than 20% higher than the baseline. For feature matching, the
phrase embedding cosine similarity shows the best matching result among four
types of similarity methods, obtaining an average accuracy of 57%. We finally
discuss the potential applications of our framework in detecting feature
problems that may cause privacy threats.

###### Index Terms:

privacy requirements, feature extraction, feature matching, App review mining,
requirements engineering

##  I Introduction

Ubiquitous software systems play a fundamental role in the area of digital
economy. Mobile devices, especially smartphones, enhanced with various
applications (Apps) have largely facilitated our daily life, e.g.
entertainment and social activities. The intelligent services of those smart
devices are mostly empowered by data collection and mining. The cost of data
breaches is record-setting, with the global average now at $4.35
million111IBM’s annual Cost of Data Breach Report for 2022:
https://www.ibm.com/security/data-breach. Privacy concerns have drawn wide
attention.

In software life cycle, privacy requirements are essential to compliance and
user satisfaction [1; 2]. Privacy requirements capture data subjects’ privacy
needs such as data minimization, confidentiality, user content awareness,
policy and consent compliance [3; 4]. Though high-level, such requirements are
typically treated as instances of requirements pertaining to compliance,
traceability, access control, verification or usability [5]. Pioneer studies
have been conducted for supporting privacy requirements management, e.g.,
formalization [6; 4] and elicitation [7; 8]. However, as the rise of the
participatory data economy, the system comes into contact with the hyper-
connected setting, a range of unanticipated privacy requirements only emerge a
posteriori [5]. Therefore, to ensure successful evolution, privacy
requirements can be elicited from a diversity of sources during maintenance
from the data-driven requirements engineering (RE) viewpoint [9].

App stores among other software repositories have been widely explored for RE
activities [10; 11]. User reviews in App stores are increasing rapidly over
time. The dynamic and user-generated characteristics of App reviews provide
the chance to mine them for software maintenance [12; 13]. Privacy
requirements are typically high-level non-functional requirements which place
implementation constraints on those functional ones [14]. Though previous
studies have explored user privacy perception through analyzing App reviews
[1; 15; 16], less attention has been paid to privacy requirements refinement
by mining features stated in privacy related reviews. We therefore go a step
further to mine those features related to user privacy concerns from App
reviews to inform software maintenance.

In this paper, we aim to detect features mentioned in App reviews expressing
privacy concerns and map them to the features listed in App descriptions to
facilitate feature refinement. To that end, we propose a semi-automatic
framework for detecting privacy concern related features as shown in Figure 1.
There are three types of input artifacts, where privacy policies are used to
help prepare the labeled data for privacy related reviews classifiers. Most
parts of our framework are automatic, which are implemented based on natural
language processing (NLP) techniques, including text matching, document
classification, and dependency parsing. Limited manual efforts are required to
build the review classification training set and annotate features appearing
in App descriptions. To evaluate to which extent our research goal is achieved
by the proposed framework, we specifically formulate the following three
research questions:

  * •

RQ1: How effectively do the learnt classifiers identify privacy related
reviews and which algorithm performs best among the selected classification
algorithms?

  * •

RQ2: How effectively does our syntactic parsing based method extract features
from privacy related reviews compared with the baseline relying on part of
speech (POS) and sentence patterns?

  * •

RQ3: How effectively does text matching technique match features mentioned in
privacy related reviews and those listed in App descriptions and which
similarity method performs best?

Through RQ1, we want to investigate if text classification technique can
accurately identify privacy related reviews from massive rapidly accumulating
reviews. Through RQ2, we attempt to investigate if syntactic patterns are more
flexible than a bundle of manually curated POS and sentence patterns and thus
can cover more features in privacy related reviews. RQ3 is used to validate if
text matching technique is able to accurately map privacy related features
mentioned by users to App features highlighted by software vendors.

Figure 1: The semi-automatic framework for detecting features concerning
privacy from App reviews

Overall, our contributions can be summarized as threefold:

  1. 1.

We propose a semi-automatic framework for detecting features concerning
privacy and validate its effectiveness and potential usage via comprehensive
experiments and discussion with illustrative examples.

  2. 2.

We provide a general privacy related review classifier with an F1 of 93.77%
based on Gradient Boosting algorithm, which is trained on reviews of Apps from
various categories.

  3. 3.

We design a dependency parsing based feature extraction method with a recall
of 85.63% and compare four different types of similarity methods for mapping
features in privacy related reviews to those in App descriptions.

The remainder of this article is structured as follows: Section II introduces
each component of our framework and the underlying techniques. Section III
presents the experimental settings including data, baselines, and evaluation
measures. Results analysis and threats to validity are provided in Section IV.
Section V discusses the potential usage of our mining results in feature
refinement. Related work is present in Section VI. Section VII concludes the
paper and forecasts several extension avenues in future.

##  II Framework

The framework in Figure 1 includes four sequential modules. The first module
identifies a set of privacy related reviews used as the input of the second
module which is responsible for extracting features. The third module maps
features mentioned in privacy related reviews to features in App descriptions,
whose results can provide information for software maintenance. We next
elaborate the first three modules and leave the potential applications
discussed in Section V.

###  II-A Privacy Related Reviews Identification

Previous empirical study reports that App reviews typically contain multiple
topics and the quality varies widely ranging from helpful advice to insulting
offenses [17]. In this component, we aim to identify privacy related reviews.
Specifically, we refer to privacy related reviews as those expressing user
privacy concerns emerging during using the Apps. Those concerns may involve
potential privacy threats or actual privacy invasions. For example, “Should
not have an app that tracks your location all the time” and “Listening with
the phone mic to find personalized content” are privacy related review while
“This app kills your battery faster NOT A JOKE” is an irrelevant one.

Supervised text classification has been proved to be effective for review-
level requirements information extraction [18]. We thus rely on supervised
classifiers trained on labeled data to identify privacy related reviews.
However, the preparation of labeled data is not a trivial task. Recent studies
show that there are less than 1% of reviews that express user privacy concerns
[15; 19]. It is not practical to randomly sample reviews from the extremely
imbalanced reviews pool to prepare training set, which needs very huge human
annotation efforts. Similar to previous work [16], we employ information
retrieval technique to select privacy related review candidates, which are
very likely expressing privacy concerns but need to be further manually judged
for preparing labeled data. What contrast our method from previous work [16]
are the query and ranking measure of the retrieval results. We employ privacy
policies of Apps as queries and rank the retrieval results based on document
similarity. [16] uses existing related taxonomies to build queries and ranks
the retrieval results according to the hit count of string matching. The
rationale under our different choices is threefold.

Firstly, the privacy policies cover and further detail those groups of
concepts contained in the privacy relevant taxonomies [20; 21] used by [16].
There are four groups of concepts in the taxonomy proposed by [21] from the
law viewpoint: information collection, information processing, information
dissemination, and invasion. Privacy policies are the means to enforce
accountability on data protection imposed by legal frameworks. They mainly
include four types of statements concerning users’ privacy, i.e., how to
collect, share, disclose, and retain user data [22]. There are two classes of
privacy requirements in the taxonomy proposed by [20]: protection goals and
vulnerabilities. This requirements taxonomy is derived from a set of Internet
privacy policies. Therefore, the privacy policies can be seen as specific
instances of those taxonomies but have more informative statements.

Secondly, compared with abstract taxonomies, there is less language gap
between privacy policies and user reviews. Privacy policies are “notices”
published to end users to inform how their personal data is protected.
Besides, privacy policies vary from Apps reflecting characteristics of
different Apps. Hence, the user-oriented and product-specific nature of
privacy policies can potentially mitigate the language gap between them and
user reviews talking about specific Apps.

Thirdly, string matching retrieval is mainly based on the hit counts of
keywords in the taxonomies regardless of the context and meaning of reviews,
which may omit much relevant results. However, document similarity, commonly
used in information retrieval [23], between privacy policies and reviews
suffers less from that problem.

Consequently, we employ the privacy policy of each App to retrieve privacy
related review candidates from its reviews pool. Text embedding is used to
encode documents as numerical vectors with a fixed dimension. Cosine
similarity is computed to rank reviews in a descending order. The top ranked
reviews are treated as privacy related review candidates. After manually
annotating the privacy related review candidates, the resulted labeled dataset
is used to train review classifiers based on machine learning algorithms. The
best performing classifier would be used to identify privacy related reviews
from the whole reviews pool.

###  II-B Feature Extraction From Privacy Related Reviews

A feature can be formally defined as any increment of functionality added to
an existing system [24]. In the context of App reviews and descriptions, we
specifically refer to a feature as a basic functional module available to
users following the previous practice [25; 26; 12]. Those features can be
described by text fragments, e.g., add a credit card, make a video call.

Though privacy requirements impose constraints on multiple aspects of a
software system, the functionality aspect among others is most easily
perceived by users. We thus surmise that users’ privacy concerns are generated
mainly during experiencing the functions of Apps. For example, personalized
content recommendation is a typical feature driven by personal data mining. We
therefore focus on extracting features mentioned in privacy related reviews.

Features are mostly captured by noun phrases and (verb, object) pairs either
in formal or informal requirements artifacts [27; 28; 25; 26]. Figure 2 shows
three privacy related review examples with features highlighted in colorful
bold fonts. Review 1 contains only features comprised of noun phrases while
Review 2 contains only those comprised of (verb, object) pairs. Both types of
features appear in Review 3.

Review 1: Why would Instagram need my financial records, health info and audio
files among others? Review 2: Its the most ridiculous system because it can be
triggered by literally anything (you liking too many pictures, following too
many people, refreshing your page more than a few times, etc). Review 3: I
closed the location access, but personalized advertisements still appear.
Figure 2: Example privacy related reviews

We employ dependency parsing to extract noun phrases and (verb, object) pairs
that potentially express App features. Dependency parsing is an NLP task of
extracting a dependency parse of a sentence that represents its grammatical
structure and defines the relationships between head words and dependent words
which modify those heads. NLP community has well studied this task in English
language and achieved desirable performance in parsing accuracy
recently222http://nlpprogress.com/english/dependency_parsing.html. Many well-
known NLP tools, e.g. spaCy, Stanford CoreNLP, have incorporated some state of
the art dependency parsing models into their pipelines.

Figure 3 shows the dependency parsing result of Review 3 in Figure 2.
Relations among the words are illustrated above the sentence with directed,
labeled arcs from heads to dependents. With the parsing result at hand, flat
phrases that have a noun as their heads are identified as noun phrases. For
example, “personalized advertisements” and “the location access” are
identified as noun phrases in Figure 3, where amod and compound denote an
adjectival modifier relation and a noun compound modifier relation separately.
(verb, object) pairs are extracted by combining words have an obj relation.
“closed the location access” is a (verb, pair) instance, where modifiers of
the object “access” are all extracted. Based on the dependency parsing
results, our feature extraction method can be formulated as Algorithm 1.

Figure 3: Example review sentence dependency parsing result Algorithm 1
Feature Extraction Based on Dependency Parsing

1:A list of review sentences $S$

2:A flat list of feature mentions $F$

3:Initialize $NPs\leftarrow[],Ps\leftarrow[]$

4:for each sentence $s\in S$ do

5: parsing result $doc\leftarrow Dependency\\_Parsing(s)$

6: for each token $t\in doc$ do

7: if $token.pos$ is noun then

8: $np\leftarrow[token.modifiers,token]$

9: $NPs$ include $np$

10: end if

11: if $token.dep\\_rel$ in [dobj, nsubjpass, pobj] then

12: $verb\\_list\leftarrow[]$, $obj\\_list\leftarrow[]$

13: if $token.dep\\_rel$ is pobj then

14: $verb\leftarrow token.head.head$

15: else

16: $verb\leftarrow token.head$

17: end if

18: $verb\\_list\leftarrow verb.conjuncts$

19: for each $term\in[token.conjuncts,token]$ do

20: $object\leftarrow[term.modifiers,term]$

21: $obj\\_list$ include $object$

22: end for

23: $Ps\leftarrow Pair\\_Combination(verb\\_list,obj\\_list)$

24: end if

25: end for

26: for each $np\in NPs$ do

27: for each $pair\in Ps$ do

28: if $np$ not in $pair$ then

29: $F$ include $np$

30: end if

31: end for

32: end for

33: $F$ include $Ps$

34: for each $f\\_candidate\in F$ do

35: if $Remove\\_Stopwords(f\\_candidate)=\phi$ then

36: $F$ exclude $f\\_candidate$

37: end if

38: end for

39:end for

The main process of Algorithm 1 involves three stages: identifying noun
phrases, extracting three types of (verb, object) pairs, and post-processing.
The input is a list of review sentences. The output is a list of features
mentioned in that review. We elaborate each stage as follows:

(1) Noun phrase extraction (lines 5 - 8): For each token with a noun POS tag,
a noun phrase compounded of that token and its modifiers would be added into
the noun phrase list $NPs$.

(2) (verb, object) pairs extraction (lines 9 - 22): We firstly identify tokens
that have one of three types of dependency relations with other words. dobj
and nsubjpass respectively denote direct object and passive nominal subject
relations. pobj represents the relation of object of a preposition. We then
aggregate the words having a coordinating conjunction relation with the
current token to construct a $obj\\_list$. $verb\\_list$ is constructed by
aggregating the words having a coordinating conjunction relation with the
current token’s head (i.e., a verb). Finally, we pair the items in these two
lists to generate the (verb, object) pair list $Ps$. For instance, we can get
a (verb, object) pair list {send pictures, receive pictures, send videos,
receive videos} from the review fragment send and receive pictures or videos.
Particularly, for the pobj relation, we identify verbs acting as the head of
that preposition which is the head of the current token. For example, the word
communicate in communicate with friends is exactly an instance of the head
verbs described above.

(3) Post-processing (lines 24 - 36): Noun phrases are removed if they are
nested in the (verb, object) pairs, which are judged via word indexes. Feature
phrases are also removed if they only contain stop words, e.g., the app. The
stop words set contains general English stop words and a few domain stop words
such as App names. We reserve the same feature mentions attached with
different indexes as the count information may inform analysts of the priority
of different features during maintenance.

###  II-C Privacy Related Features Mapping

The extracted features above are noisy and numerous. On the one hand, the
unsupervised extraction process relies on syntactic patterns without
considering phrase meaning, which may cover most feature mentions while
simultaneously include much noise. On the other hand, massive reviews would
lead to a large number of extracted feature mentions. These mentions are
Reviewsvarious in expression increasing the redundancy. For example, “clear
recently played” and “delete listening history” express the same feature.
Thus, it is ineffective to directly utilize the extraction results to inform
maintenance.

To provide more concise feature information and inform feature refinement, we
decide to map features in privacy related reviews to features listed in App
descriptions via matching these two kinds of features based on phrase
similarity. Typically, App descriptions highlight the main features with brief
and listed statements. Both developers and users share a common understanding
of the brief descriptions, potentially decreasing the language gap between
these two types of feature mentions. Additionally, software product
descriptions have been widely proved to be useful for various feature related
RE tasks, e.g., feature modeling [29; 30], feature recommendation [31], and
product line variability modeling [32].

Specifically, to ensure matching quality, we manually labeled the features in
App descriptions. Given an App, let $F$ and $F_{d}$ separately denote the
feature set generated from reviews and the description of that App. For each
automatically extracted feature $f\\_review$, we calculate the phrase
similarity between it and every feature in the App description. We map
$f\\_review$ to the feature $f$ in the App description, which is most similar
to $f\\_review$. This process can be formulated as follow:

$\displaystyle\max_{f\in F\\_d}$ $\displaystyle
Phrase\\_Sim(f\\_review,f),\forall f\\_review\in F$ (1)

We compare four different types of phrase similarity methods to investigate
their effectiveness in feature matching.

##  III Experimental Settings

In this section, we introduce the data, baseline methods and evaluation
measures involved in each component. All of the experimental data, codes, and
results are publicly available333https://figshare.com/s/a725a06d50b4105cd56e
for replication and extension purposes.

###  III-A App Reviews and Privacy Policies Collection

For the App reviews, we firstly select five popular Apps in different
categories according to the rank lists of four English speaking
markets444https://www.data.ai/en/apps/ios/top/. This selection ensures that
there are plenty of user reviews in English. The selected Apps include
Facebook, Instagram, Youtube, TikTok, and Spotify from categories covering
social networking, photo and video, entertainment, and music. Reviews of Apps
from diverse categories could help build a general privacy related review
classifier. We then collect user reviews of those Apps from the Apple App
Store. The posting time spans from 2020-01-01 to 2022-04-29. The total amount
of reviews is 1,547,074 and the summary statistics are shown in Table I.

TABLE I: App Reviews Summary Statistics App | U. S. | Canada | U. K. | Australia | Total
---|---|---|---|---|---
Instagram | 196,975 | 22,394 | 25,980 | 10,550 | 255,899
YouTube | 189,256 | 21,551 | 29,085 | 12,711 | 252,603
TikTok | 200,143 | 112,164 | 175,214 | 56,715 | 544,236
Spotify | 167,943 | 22,699 | 25,626 | 15,727 | 231,995
Facebook | 182,115 | 23,742 | 22,742 | 33,742 | 262,341

For the privacy policies, we collect each App’s privacy policy from its
official website and remove the “generic” statements [22] from the policy
document, e.g., how to contact with App vendors and how to notify policy
change.

###  III-B Privacy Related Review Candidates Retrieval

For encoding the reviews and policy documents, we employ the text embedding
technique due to its salient performance in various text retrieval tasks from
different domains [33]. Specifically, we use the
SentenceTransformers555https://www.sbert.net/, which is a Python framework for
state-of-the-art text embeddings and offers a collection of pre-trained models
tuned for various tasks. We compare four pre-trained embedding models from the
SentenceTransformers model repository for retrieving privacy related review
candidates, which are shown in Table II.

TABLE II: Pre-trained Models Selected for Reviews Retrieval Models | Descriptions
---|---
all-mpnet |  These models are trained on more than 1 billion training pairs and designed for the general purpose.
all-MiniLM
multi-qa-MiniLM |  These models are trained on 215M question-answer pairs and specifically trained for semantic search.
multi-qa-mpnet

###  III-C Privacy Related Reviews Classification Data and Methods

For building the classification training set, two software engineering
postgraduates are responsible for independently annotating the retrieved
privacy related review candidates. They are informed of which reviews are
privacy related ones with typical examples in a 30 minutes training session.
One postgraduate firstly annotates the top ranked 600 retrieved reviews of
each App. Another postgraduate then annotates the same reviews set without
knowing the labels. They are allowed to skip those uncertain ones. Finally,
they annotate 2,769 reviews in total together. The inter-annotator agreement,
computed using Cohen’s kappa coefficient, is 0.88, indicating “almost perfect
agreement” [34]. The first author further discusses with them to resolve the
disagreement. The final labeled data set is unbalanced including 1,040 privacy
related reviews and 1,729 negative samples. In particular, following previous
practice [35; 36], we employ the random under-sampling to achieve a balanced
class distribution. The ratio of training and testing sets is 8:2.

For learning the review classifiers, we compare three types of machine
learning algorithms, which are described in Table III. For the first four
models, we use the TF-IDF representations as input feature vectors. For
fastText and BERT, we use their pre-trained word
embeddings666https://fasttext.cc/docs/en/english-
vectors.html777https://huggingface.co/bert-base-uncased as input feature
vectors. In particular, BERT is attached with a bi-directional GRU layer whose
parameters are fine-tuned for reviews classification.

TABLE III: Algorithms Selected for Reviews Classification Algorithms | Descriptions
---|---
Logistic Regression |  These classic linear models are widely used as baselines in the text classification task.
Linear SVM
Random Forest |  These ensemble algorithms are strong competitors in various data competitions.
Gradient Boosting
fastText |  These deep neural network (DNN) models have achieved new state of the arts in many benchmarks.
BERT

###  III-D Feature Extraction Baselines

We select SAFE [26] as the baseline, which is a state of the art in feature
extraction from App reviews. It includes 18 POS patterns and 5 frequent
sentence patterns covering enumerations and conjunctions.

For our syntactic parsing based method, we implement the Algorithm 1 based on
the well-known NLP tool spaCy and Python programming.

###  III-E Phrase Similarity Methods for Feature Matching

For feature matching, a software engineering postgraduate manually labels the
features appearing in App descriptions and the first author double-checks the
results. Four types of phrase similarity methods are selected for matching
feature phrases, which are introduced in Table IV.

TABLE IV: Four Types of Phrase Similarity Methods Methods | Descriptions
---|---
Jaccard |  It measures the similarity between two phrases to see which words are shared and distinct.
TF-IDF + Cosine |  It vectorizes phrases based on frequency and the cosine similarity is computed between vectors.
WordNet-based [37] |  It combines word concept similarity derived from WordNet and word order similarity to compute phrase similarity. This method has been used by previous work [26] to match features in App reviews and descriptions.
Phrase embedding [33] |  It vectorizes phrases based on language models pre-trained on large scale data and the cosine similarity is computed between vectors.
+
Cosine

For the phrase embedding method, we choose the paraphrase-multilingual-MiniLM
model from the SentenceTransformer model repository to vectorize feature
phrases, which is specifically designed to find texts with identical or
similar meaning.

###  III-F Evaluation Measures

For evaluating the performance of reviews classification, feature extraction,
and feature matching, we employ the commonly used accuracy, precision, recall,
and F-measure. For the reviews retrieval, we additionally use Average
Precision (AP). A higher AP means that there are more relevant items in the
top positions of a rank list.

##  IV Experiment Results Analysis

In this section, to evaluate the effectiveness of our framework, we answer the
research questions by reporting and analyzing the experiment results.

###  IV-A Privacy Related Review Candidates Retrieval

To determine the document embedding model for reviews retrieval, we establish
a ground truth with a subset of Facebook reviews. Specifically, the Facebook’s
privacy policy is used as the query and a subset of 14,122 reviews is treated
as the documents set. We firstly use the document embedding multi-qa-MiniLM to
compute the cosine similarity between the query and each review. The reviews
are then sorted by the similarity scores in a descending order. The first 500
reviews are manually labeled with 1 (privacy related) or 0 (irrelevant). The
resulted ground truth includes 255 privacy related reviews and 245 irrelevant
ones.

The other three models are further evaluated on this ground truth. In other
words, the 500 reviews are re-sorted by the computed similarity scores based
on those models. The better a model performs, the more privacy related reviews
will come near top positions at the rank list. As more top ranked reviews are
included in the final retrieval result, both the numbers of privacy related
and irrelevant ones increase, which means that the precision decreases while
the recall increases. For trade-off consideration, we compute $F1@k$, where
$k$ is the size of the retrieval result. $F1@k$ is the harmonic mean of
$precision@k$ and $recall@k$. $precision@k$ is the fraction of privacy related
reviews among the retrieved reviews, i.e., $\frac{\\#\ of\ related\ ones}{k}$.
$recall@k$ is the fraction of privacy related reviews that were retrieved,
i.e., $\frac{\\#\ of\ related\ ones}{255}$.

Figure 4 shows the $F1@k$ of four different document embedding models. As the
$k$ increases, the $F1$ scores of all models firstly increase rapidly and then
become stable. In most cases (the $k$ values), the multi-qaMiniLM performs
better than others. Both all-MiniLM and multi-qa-mpnet have relatively lower
$F1$. all-mpnet performs better than all-MiniLM and multi-qa-mpnet when
$k<235$. The F1 scores of all models stabilize at about 67% as $k$ increases
near 500. The highest $F1$ 69.88% (recall 82.35%) is achieved by multi-
qaMiniLM at $k=346$. Table V shows the average precision of the top 100
retrieval results returned by different models.

Figure 4: Retrieval performance of different document embedding models TABLE V: Average Precision of Different Models (Top 100) Models | all-mpnet | all-MiniLM | multi-qa-MiniLM |  multi-qa-mpnet
---|---|---|---|---
AP | 94.83% | 93.86% | 96.80% |  88.87%

Based on the results in Figure 4 and Table V, we finally choose the document
embedding model multi-qa-MiniLM to retrieve privacy related review candidates.
Specifically, we vectorize the privacy policy and reviews of each App with
multi-qa-MiniLM embeddings and compute the cosine similarity between each
review and the corresponding privacy policy. The reviews of each App are
ranked by similarity scores in a descending order. The top ranked reviews are
treated as privacy related review candidates and are further annotated to
prepare classification labeled data. We note that the size of privacy related
review candidates is not necessarily fixed. It could be determined in line
with the subsequent task at hand. For instance, in our work, we choose the top
600 ranked reviews of each App to build the classification training set.

###  IV-B Privacy Related Reviews Classification

After building the labeled data set with the privacy related candidates as
described in Section III-C, we train and evaluate classifiers based on the
three selected types of algorithms.

Table VI reports performance of different classifiers in identifying privacy
related reviews. For precision, all classifiers have a score exceeding 93%.
BERT and two ensemble algorithms obtain a precision higher than 95%. The
highest precision of 97.41% is achieved by Gradient Boosting. For recall, only
Gradient Boosting and two DNN algorithms obtain scores exceeding 90% with the
same value of 90.38%. Random Forest has the lowest recall of 88.94%. When
taking the precision and recall into consideration together, all classifiers
achieve an F1 score higher than 91%. Gradient Boosting has the highest F1
score of 93.77%. Overall, the two linear algorithms, i.e., Logistic Regression
and linear SVM performs worse than the ensemble and DNN algorithms. Gradient
Boosting performs best among others with the highest precision, recall, and F1
score.

TABLE VI: Classification Performance of Different Classifiers Algorithms | Precision | Recall | F1
---|---|---|---
Logistic Regression | 93.00% | 89.42% | 91.18%
linear SVM | 93.47% | 89.42% | 91.40%
Random Forest | 95.36% | 88.94% | 92.04%
Gradient Boosting | 97.41% | 90.38% | 93.77%
fastText | 94.00% | 90.38% | 92.16%
BERT | 95.43% | 90.38% | 92.84%

We perform quantitative error analysis of the classification results of all
classifiers based on the confusion matrices. Due to the space limitation, only
the confusion matrix of Gradient Boosting is shown in Table VII. The analysis
indicates that the number of false negative instances is far more than the
number of false positive ones. This observation accords with the result in
Table VI that all classifiers obtain a higher precision compared with their
recall.

TABLE VII: The Confusion Matrix of Gradient Boosting | Prediction |
---|---|---
| Privacy Related | Irrelevant | Total
Ground Truth | Privacy Related | $203$ | $5$ | $208$
Irrelevant | $20$ | $188$ | $208$
| Total | $223$ | $193$ | $416$

We further qualitatively investigate the content of those error instances.
Most of false negative (FN) reviews mix privacy concerns with too much
irrelevant content. This is consistent with the previous finding by [17] that
App reviews typically contain diverse topics. False positive (FP) instances
usually include privacy indicating words, e.g., sell, information, but do not
express privacy concerns. Figure 5 shows these two types of error instances.

FN: Creepy Tracking and Censorship of wrong think. YouTube was good once upon
a time. Now they censor opinions they dont like. Track your usage and
manipulate your search results. They are by extension a malicious foreign
actor and should be handled with caution FP: Facebook is done. Too much
censorship, cant express yourself freely anymore. Fact checkers is everywhere
with info theyve got from unreliable sources. Facebook groups wont allow to
even make a post anymore unless its a for sell post overall everything has
went downhill. Figure 5: False negative and false positive prediction
instances

Applying the Gradient Boosting classifier in the whole reviews set, the
numbers of privacy related reviews identified for each App are shown as Table
VIII.

TABLE VIII: The number of extracted privacy related reviews for each App App | Instagram | YouTube | TikTok | Spotify |  Facebook
---|---|---|---|---|---
# of privacy reciews | 6,534 | 2,452 | 3,627 | 2,277 |  13,107

We thus can answer the RQ1 as follows: In our evaluation experiments, all
three types of selected classifiers can identify privacy related reviews with
a desirable accuracy, where the F1 scores are all higher than 88%. Among the
classifiers, Gradient Boosting performs best with an F1 of 92.12%. Error
analysis indicates that the noisy nature of App reviews accounts for the main
error predictions.

###  IV-C Feature Extraction From Privacy Related Reviews

For evaluation purpose, we randomly sample 120 reviews from the identified
privacy related reviews, 24 reviews for each App. Ground truth is built by
manually annotating features in these reviews. There are totally 341 features
in the ground truth. We apply both our syntactic parsing based method and the
baseline in the reviews sample set to extract features automatically. Table IX
presents the comparison results. The “Extraction” column shows the number of
extracted features and the “Correct” column shows the number of correct ones
in those extracted features.

TABLE IX: Feature Extraction Performance Comparison Method | Extraction | Correct | Precision | Recall | F2
---|---|---|---|---|---
SAFE | 834 | 210 | 25.18% | 61.58% | 47.77%
Ours | 925 | 292 | 31.57% | 85.63% | 63.78%

As can be seen from Table IX, our method outperforms the baseline in all three
accuracy measures. The number of correct features extracted by our method is
39% ($\approx(292-210)/210$) more than the baseline. That leads to our method
obtaining a much higher recall than the baseline with a margin of more than
24% ($\approx 85.63\%-61.58\%$). The precision of our method is also higher
than that of the baseline with a less boost than that in recall. While more
correct features are extracted by our method, some noise instances are also
included at the same time. We argue that the recall should be paid more
attention than the precision in this feature extraction task, which is also
emphasized by previous work [38]. Because much more effort is needed to find
the missing features compared with filtering out noise from the extraction
result, especially when the review analysis is conducted at scale. We
therefore set the positive real factor $\beta$ in F-measure formula to $2$,
which is commonly used to weigh recall higher than precision. Our method
achieves an F2 score of 63.78%, 16.01% ($=63.78\%-47.77\%$) higher than the
baseline. Overall, our method exhibits a much better feature extraction
quality and quantity with much higher precision and recall.

In what follows, we compare our syntactic parsing based method with the POS
and sentence patterns based baseline to demonstrate the superiority of ours.

  1. 1.

Our syntactic parsing based method can cover all the hand-crafted patterns of
the baseline method, which mainly relies on manual analysis of review samples
to identify frequent patterns denoting App features. Specifically, our noun
phrases and (verb, object) pairs can cover all of 18 POS patterns in [26]. The
processing of coordinating conjunction dependency relation also captures
enumerations and conjunctions, which are identified by five sentence patterns
in the baseline. For example, in Figure 6(a), our method firstly identifies
the (view, conditions) pair through the obj relation. Then another object news
is found via the conj relation. The complete obj_list {the latest traffic
conditions, special traffic news} is generated by combining each object and
its modifiers. Finally, the pair combination operation between verb_list and
obj_list results in the feature list {view the latest traffic conditions, view
special traffic news}.

  2. 2.

In contrast to the hand-crafted patterns derived from review samples, our
method is more flexible and general, because it considers the high-level
syntactic structure of the whole sentence besides the more specific POS
information. Thus our method can cover more feature patterns than the limited
number of patterns in the baseline. For example, in Figure 6(b), our method
can accurately extract requests access to my location via properly processing
the pobj relation. The baseline method only extracts an incomplete feature
mention requests access conforming to the POS pattern “Verb Noun”.

  3. 3.

Our method also identifies feature mentions contained in passive sentences by
the nsubjpass relation, which the baseline can not deal with. As shown in
Figure 6(c), the feature lock my account can be extracted by our method but
not the baseline.

(a) Feature conjunctions extraction based on cc relation

(b) Feature extraction from the complex (verb, object) pair

(c) Feature extraction from the passive sentence

Figure 6: Feature extraction based on dependency parsing

Therefore, the answer to RQ2 is as follows: In our experimental evaluations,
our dependency parsing based method can largely outperform the POS and
sentence patterns based baseline in extracting features from privacy related
reviews. Our method achieves a recall of 85.63% (24% higher than the baseline)
and an F2 score of 63.78% owing to its utilization of high level syntactic
structure information.

###  IV-D Privacy Related Features Mapping

For evaluation purpose, we randomly select 100 privacy related reviews for
each App. Our feature extraction method is applied to extract features from
those reviews. We then compute the pairwise similarity between features in
reviews and those in the corresponding App description. For each unique
feature extracted from reviews, we match it with the feature in App
descriptions that is most similar to it. After generating all matching pairs
based on phrase similarity, we then sort them by the similarity score in a
descending order. For each App, we manually evaluate the top 20 matching pairs
generated by each of the four phrase similarity methods. Two features are
treated as correctly matched if they express the same functionality or one
feature can be seen as a sub feature of another. For example, (watch some
music video, watch video) and (use subscription, subscribe) are correct
feature matching pairs, while (recommend video, watch video) is not a correct
one. Table X shows the matching accuracy, which is the percentage of correct
matching pairs in the top 20 ones.

TABLE X: Feature Matching Accuracy Based on Different Phrase Similarity Methods | Jaccard | TF-IDF | WordNet-based |  Phrase embedding
---|---|---|---|---
Instagram | 30% | 30% | 40% | 45%
YouTube | 50% | 35% | 35% | 55%
TikTok | 10% | 10% | 20% | 40%
Spotify | 60% | 55% | 60% | 80%
Facebook | 15% | 30% | 45% | 65%
Average | 33% | 32% | 40% | 57%

As can be seen from Table X, in all of the five Apps, phrase embedding cosine
similarity method consistently outperforms the other three methods. Its best
accuracy 80% is achieved on the Spotify. Jaccard similarity and TF-IDF cosine
similarity obtain similar lower accuracy scores. Because both of these two
methods only consider word surface forms but can not deal with synonyms with
different surface forms, e.g., picture and image. Jaccard similarity only
relies on the overlap of word surface forms while TF-IDF vectors depend on the
frequency of word surface forms. As the WordNet-based method [37] considers
both concept and word order similarity, it performs better than Jaccard and
TF-IDF by achieving an average accuracy of 40%. Although the WordNet-based
method processes synonyms, the WordNet lexicon has a quite limited vocabulary
compared with the massive App reviews, which contains much more informal and
newly emerging expressions [10]. This might potentially hurts the accuracy of
the WordNet-based method. We attribute the relatively higher accuracy of the
phrase embedding method to its underlying pre-trained language model that is
trained on large scale various text corpus. Table XI presents several correct
feature matching examples produced by the phrase embedding method for Spotify.
Though the phrase embedding method can encode a feature phrase as a single
vector, the cosine similarity of two vectors does not explicitly consider the
importance of different words composing the phrases. This might restrict the
accuracy of the phrase embedding method.

Besides the intrinsic limitations of those selected methods, we also attribute
the relatively low accuracy of all methods to the absence of some features in
descriptions. There are features that are mentioned in reviews but not listed
in the brief descriptions. Features in descriptions are mainly those major
features of Apps while the scope of features in reviews is typically not
limited.

TABLE XI: Correct Feature Matching Examples Produced by Phrase Embedding for Spotify Feature in Reviews | Feature in Descriptions | Scores
---|---|---
download music | download music | 1.00
listen to a song | listen to the song | 0.95
audio quality | get well sound quality | 0.83
increase the privacy of collaborative playlist | make collaborative playlist | 0.72
search many other song | find music | 0.72

Consequently, we can answer RQ3 as follows: In our experimental evaluations,
all four types of text matching methods without making task-specific
adaptations show relatively low average accuracy partly due to the absence of
some features in descriptions. The highest average accuracy 57% is achieved by
the phrase embedding method empowered by the large pre-trained language model.

###  IV-E Threats to Validity and Limitations

Threats to internal validity mainly come from the processes of manual
annotation and evaluation. To mitigate these, we conduct several peer-coding
based training sessions with illustrative examples to introduce what privacy
related reviews are, what can be seen as a feature, and in what cases two
features are treated as correctly matched. Another threat to internal validity
is the retrieval process of the privacy related review candidates which are
used to prepare classification labeled data. The privacy policies may not
cover all privacy concerns involved in user reviews. Moreover, the ground
truth used to evaluate the document embedding models is retrieved by multi-qa-
MiniLM, which also performs best in the final evaluation. This may bring bias
to the evaluation but can not be easily avoided in a start point of zero
labeled data. In addition, it is possible that there are better models than
multi-qa-MiniLM, which could rank more privacy related reviews in front of
irrelevant ones.

Threats to external validity mainly relates to the diversity of our dataset,
which may decrease the generalizability of our research findings. To mitigate
these, we carefully select several popular Apps from different common
categories according to the App rank list. Furthermore, our reviews data is
collected from Apple App Store. The limited number of App categories and
single review source might make the findings derived from our research not
directly extend to other App categories and review sources, e.g., Google Play.

A major limitation of our framework is that the feature matching module has a
relatively low accuracy. The evaluation results demonstrate that directly
applying general purpose NLP techniques to feature matching may not obtain
desirable performance. Therefore, domain and task-specific adaption of
existing NLP techniques [28; 39] are further needed to improve the feature
matching accuracy, which we leaves for the future extension.

##  V The Potential Usage of Our Framework

As our framework mainly focuses on extracting features from privacy related
reviews, we would like to detect those feature problems that might impose
privacy threats. Some usage scenarios with illustrative examples are described
as follows:

  * •

New feature release: For those frequently extracted features, if they can not
be properly mapped to features in descriptions (or the full functional
requirements if exists), the reviews containing them may express new feature
requests. For example, the review “I personally would like it a lot more if
you could block other users who are stalking you. I can block them on
everything else except this and it makes me feel really unsafe. PLEASE ADD A
BLOCKING FEATURE!!” requests a “block other users” feature, which could inform
developers of the next release version.

  * •

Feature issue report: For those frequently extracted features that can be
mapped to features in descriptions, the reviews containing them may report a
feature bug. For example, the review Not friends only. Dont get me wrong I
love tiktok, but when I post for my friends and change the privacy setting to
friends only, other ppl whom I do not follow still see and like my content.
Its really bugging me reports a potential bug for the feature “change the
privacy setting to friends only”.

  * •

Feature interaction problem detection: For the frequently co-occurring feature
set in same reviews, if the features in it can be mapped to those in
descriptions, the reviews containing the feature set might contain an
undesired feature interaction situation that brings about privacy concerns. In
contrast to the feature release and feature issues that have been studied
based on user reviews mining [12; 40], less attention has been paid to
exploring the feature interaction problem reported by user reviews. We thus
illustrate this potential and promising usage scenario below.

According to [41], a feature interaction is some way in which a feature or
features modify or influence another feature in defining overall system
behavior. Feature interaction might occur as a result of a conflict in
services’ goals [42]. Not all feature interactions are problematic, but
unexpected or adverse feature interactions may bring about uncertainties to
software systems, which forms the essence of what has become known as the
feature interaction problem [43]. Figure 7 presents a privacy related review
of Spotify with features highlighted in bold font. This example review
illustrates a potential privacy disclosure caused by the interaction of two
features play control and play music on Bluetooth speakers. Similar to the
feature interaction examples in [41], we elaborate the process of feature
interaction in Figure 7 as follow:

  * •

The App can be running in the foreground or background. The play control
feature enables users to start or pause the music playing. The feature play
music on Bluetooth speakers enables playing music on successfully paired and
connected devices.

  * •

However, when the phone is connected with Bluetooth speakers and the App is
running, the play control feature just immediately enables the Bluetooth
speakers broadcast the current music without asking for users’ allowance.

  * •

Consequently, all music will be broadcasted by speakers even those users treat
as private. The above feature interaction thus could potentially cause privacy
disclosure.

We then explain how our framework can assist in identifying the above feature
interaction. The review in Figure 7 can be identified by our classification
module. The feature mentions in that reviews can be extracted by our feature
extraction module. The feature mappings $\\{\textit{stop
playing},\textit{start playing}\\}\to\\{\textit{play control}\\}$ and
$\\{\textit{play music to my car radio}\\}\to\\{\textit{play music on
Bluetooth speakers}\\}$ can be conducted by our feature matching module with
better similarity method. We note that the App descriptions in our framework
can be easily replaced with the full functional requirements, which are
accessible to developers. This replacement can reduce the cases that features
in reviews can not be properly matched. The co-occurrence of the feature set
{play control, play music on Bluetooth speakers} could be mined by association
rules (currently not included in our framework) if they are frequently
mentioned in privacy related reviews.

The app will often times stop playing, or refuse to start playing. There are
privacy issues with it, and even when the app is not on or running in the
background it’ll still play music to my car radio and the only way to control
it is to turn off Bluetooth. Pausing will not work, shutting the app down will
not work. They also have no standards when it comes to what they allow to
broadcast. It’s not a horrible app, but there is so much room for improvement
Figure 7: Privacy Disclosure Caused by Feature Interaction

##  VI Related Work

###  VI-A Privacy Requirements Analysis

RE literature handles privacy requirements mainly from perspectives of
Compliance [44; 45], Access Control [22], Verification [46] and Usability [7;
15; 1; 47]. Usability research focuses on the evaluation and understanding of
user behaviours, needs, and motivations through observation techniques, and
analysis of usability problems of existing privacy solutions [5]. Our work
also falls into this perspective. The Usability perspective covers a wide
spectrum including user studies on privacy perceptions [1; 16], privacy
breaches in social media [15], and improvement of user awareness [48]. Among
those, [16] is most related to our work. Both approaches automatically extract
and analyze privacy related reviews at scale. In contrast to [16], we employ
syntactic parsing and phrase similarity to detect features mentioned in
reviews for facilitating feature refinement, while [16] combine clustering and
summarization techniques to identify dominant privacy themes from a user
perception perspective for supporting App stores and developers. Additionally,
though employing different data sources (Google Play v.s. Apple App Store),
our reviews classification results further verify and confirm their findings
that ensemble methods perform best in identifying privacy related reviews and
the precision of classifiers are typically higher than the recall.

###  VI-B App Reviews Mining

App reviews mining is another area closely relevant to our study. App stores
have long been treated as valuable repositories [25] and user reviews in it
also provide plenty of valuable requirements knowledge [17]. Previous efforts
have been conducted to distill feature request [11], collect bug reports [40],
and identify domain-specific topics [49]. Among those, [26] is most related to
our work, both approaches extract features from reviews and match them with
those in App descriptions. In contrast to [26], our dependency parsing based
feature extraction methods largely outperforms their POS and sentence patterns
based method. Our selected phrase embedding method also achieves a higher
average accuracy than the WordNet-based method used by [26] in feature
matching evaluation. Besides the general usage scenarios discussed in [26], we
particularly focus on detecting the potentially problematic features that may
lead to privacy threats. We also illustrate a new potential and promising
application scenario of our framework, i.e., feature interaction detection.

##  VII Conclusion and Future Work

In this work, we propose a semi-automatic framework to detect features
concerning privacy from App reviews. The framework employs text classification
to identify privacy related reviews, relies on dependency parsing to extract
features, and uses phrase similarity to match features in reviews with those
in descriptions. Quantitative evaluations demonstrate the effectiveness of our
framework and qualitative discussions highlights the potential usefulness of
our framework in feature refinement.

There are several valuable avenues to extend our framework. The first one is
to consider the importance of different words in feature phrases to improve
the matching accuracy. The second one is to extend the framework with
association rules and complex networks to support automatic feature
interaction detection. Finally, automatic test case generation with NLP could
be integrated into the framework to enhance its practicability.

## References

  * Hatamian et al. (2019) Majid Hatamian, Jetzabel Serna, and Kai Rannenberg.  Revealing the unrevealed: Mining smartphone users privacy perception on app markets.  _Computers & Security_, 83:332–353, 2019.
  * Canedo et al. (2022) Edna Dias Canedo, Ian Nery Bandeira, Angelica Toffano Seidel Calazans, Pedro Henrique Teixeira Costa, Emille Catarine Rodrigues Cançado, and Rodrigo Bonifácio.  Privacy requirements elicitation: a systematic literature review and perception analysis of it practitioners.  _Requirements Engineering_ , pages 1–18, 2022.
  * Deng et al. (2011) Mina Deng, Kim Wuyts, Riccardo Scandariato, Bart Preneel, and Wouter Joosen.  A privacy threat analysis framework: supporting the elicitation and fulfillment of privacy requirements.  _Requirements Engineering_ , 16(1):3–32, 2011\.
  * Gharib et al. (2021) Mohamad Gharib, Paolo Giorgini, and John Mylopoulos.  Copri v.2- a core ontology for privacy requirements.  _Data & Knowledge Engineering_, 133:101888, 2021.
  * Anthonysamy et al. (2017) Pauline Anthonysamy, Awais Rashid, and Ruzanna Chitchyan.  Privacy requirements: present & future.  In _2017 IEEE/ACM 39th international conference on software engineering: software engineering in society track (ICSE-SEIS)_ , pages 13–22. IEEE, 2017.
  * Oltramari et al. (2018) Alessandro Oltramari, Dhivya Piraviperumal, Florian Schaub, Shomir Wilson, Sushain Cherivirala, Thomas B Norton, N Cameron Russell, Peter Story, Joel Reidenberg, and Norman Sadeh.  Privonto: A semantic framework for the analysis of privacy policies.  _Semantic Web_ , 9(2):185–203, 2018.
  * Thomas et al. (2014) Keerthi Thomas, Arosha K Bandara, Blaine A Price, and Bashar Nuseibeh.  Distilling privacy requirements for mobile applications.  In _Proceedings of the 36th international conference on software engineering_ , pages 871–882, 2014.
  * Mai et al. (2018) Phu X Mai, Arda Goknil, Lwin Khin Shar, Fabrizio Pastore, Lionel C Briand, and Shaban Shaame.  Modeling security and privacy requirements: a use case-driven approach.  _Information and Software Technology_ , 100:165–182, 2018\.
  * Maalej et al. (2015) Walid Maalej, Maleknaz Nayebi, Timo Johann, and Guenther Ruhe.  Toward data-driven requirements engineering.  _IEEE software_ , 33(1):48–54, 2015.
  * Dąbrowski et al. (2022) Jacek Dąbrowski, Emmanuel Letier, Anna Perini, and Angelo Susi.  Analysing app reviews for software engineering: a systematic literature review.  _Empirical Software Engineering_ , 27(2):1–63, 2022.
  * Malgaonkar et al. (2022) Saurabh Malgaonkar, Sherlock A Licorish, and Bastin Tony Roy Savarimuthu.  Prioritizing user concerns in app reviews–a study of requests for new features, enhancements and bug fixes.  _Information and Software Technology_ , 144:106798, 2022\.
  * Zhang et al. (2019) Jianzhang Zhang, Yinglin Wang, and Tian Xie.  Software feature refinement prioritization based on online user review mining.  _Information and Software Technology_ , 108:30–34, 2019\.
  * Gao et al. (2021) Cuiyun Gao, Jichuan Zeng, Zhiyuan Wen, David Lo, Xin Xia, Irwin King, and Michael R Lyu.  Emerging app issue identification via online joint sentiment-topic tracing.  _IEEE Transactions on Software Engineering_ , 2021.
  * Jacobson et al. (1999) Ivar Jacobson, Grady Booch, and James Rumbaugh.  _The Unified Software Development Process_.  Addison-Wesley Longman Publishing Co., Inc., USA, 1999.  ISBN 0201571692.
  * Nguyen et al. (2019) Duc Cuong Nguyen, Erik Derr, Michael Backes, and Sven Bugiel.  Short text, large effect: Measuring the impact of user reviews on android app security & privacy.  In _2019 IEEE symposium on Security and Privacy (SP)_ , pages 555–569. IEEE, 2019.
  * Nema et al. (2022) Preksha Nema, Pauline Anthonysamy, Nina Taft, and Sai Teia Peddinti.  Analyzing user perspectives on mobile app privacy at scale.  In _2022 IEEE/ACM 44th International Conference on Software Engineering (ICSE)_ , pages 112–124. IEEE, 2022.
  * Pagano and Maalej (2013) Dennis Pagano and Walid Maalej.  User feedback in the appstore: An empirical study.  In _2013 21st IEEE international requirements engineering conference (RE)_ , pages 125–134. IEEE, 2013.
  * Maalej et al. (2016) Walid Maalej, Zijad Kurtanović, Hadeer Nabil, and Christoph Stanik.  On the automatic classification of app reviews.  _Requirements Engineering_ , 21(3):311–331, 2016\.
  * Tao et al. (2020) Chuanqi Tao, Hongjing Guo, and Zhiqiu Huang.  Identifying security issues for mobile applications based on user review summarization.  _Information and Software Technology_ , 122:106290, 2020\.
  * Antón and Earp (2004) Annie I Antón and Julia B Earp.  A requirements taxonomy for reducing web site privacy vulnerabilities.  _Requirements engineering_ , 9(3):169–185, 2004\.
  * Solove (2005) Daniel J Solove.  A taxonomy of privacy.  _U. Pa. l. Rev._ , 154:477, 2005.
  * Caramujo et al. (2019) João Caramujo, Alberto Rodrigues da Silva, Shaghayegh Monfared, André Ribeiro, Pável Calado, and Travis Breaux.  Rsl-il4privacy: a domain-specific language for the rigorous specification of privacy policies.  _Requirements Engineering_ , 24(1):1–26, 2019\.
  * Manning et al. (2008) Christopher D Manning, Prabhakar Raghavan, and Hinrich Schütze.  _Introduction to Information Retrieval_.  Cambridge University Press, 2008.
  * Bowen et al. (1989) Thomas F Bowen, FS Dworack, Ching-Hua Chow, Nancy Griffeth, Gary E Herman, and Y-J Lin.  The feature interaction problem in telecommunications systems.  In _Seventh International Conference on Software Engineering for Telecommunication Switching Systems, 1989. SETSS 89._ , pages 59–62. IET, 1989\.
  * Harman et al. (2012) Mark Harman, Yue Jia, and Yuanyuan Zhang.  App store mining and analysis: Msr for app stores.  In _2012 9th IEEE working conference on mining software repositories (MSR)_ , pages 108–111. IEEE, 2012.
  * Johann et al. (2017) Timo Johann, Christoph Stanik, Walid Maalej, et al.  Safe: A simple approach for feature extraction from app descriptions and app reviews.  In _2017 IEEE 25th international requirements engineering conference (RE)_ , pages 21–30. IEEE, 2017.
  * Niu et al. (2013) Nan Niu, Juha Savolainen, Zhendong Niu, Mingzhou Jin, and Jing-Ru C Cheng.  A systems approach to product line requirements reuse.  _IEEE Systems Journal_ , 8(3):827–836, 2013.
  * Arora et al. (2016) Chetan Arora, Mehrdad Sabetzadeh, Lionel Briand, and Frank Zimmer.  Automated extraction and clustering of requirements glossary terms.  _IEEE Transactions on Software Engineering_ , 43(10):918–945, 2016.
  * Davril et al. (2013) Jean-Marc Davril, Edouard Delfosse, Negar Hariri, Mathieu Acher, Jane Cleland-Huang, and Patrick Heymans.  Feature model extraction from large collections of informal product descriptions.  In _proceedings of the 2013 9th joint meeting on foundations of software engineering_ , pages 290–300, 2013.
  * Carbonnel et al. (2019a) Jessie Carbonnel, Marianne Huchard, and Clémentine Nebut.  Modelling equivalence classes of feature models with concept lattices to assist their extraction from product descriptions.  _Journal of Systems and Software_ , 152:1–23, 2019a.
  * Dumitru et al. (2011) Horatiu Dumitru, Marek Gibiec, Negar Hariri, Jane Cleland-Huang, Bamshad Mobasher, Carlos Castro-Herrera, and Mehdi Mirakhorli.  On-demand feature recommendations derived from mining public product descriptions.  In _Proceedings of the 33rd international conference on software engineering_ , pages 181–190, 2011.
  * Carbonnel et al. (2019b) Jessie Carbonnel, Marianne Huchard, and Clémentine Nebut.  Towards complex product line variability modelling: Mining relationships from non-boolean descriptions.  _Journal of Systems and Software_ , 156:341–360, 2019b.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, 11 2019\.
  * Landis and Koch (1977) J Richard Landis and Gary G Koch.  The measurement of observer agreement for categorical data.  _biometrics_ , pages 159–174, 1977.
  * Villarroel et al. (2016) Lorenzo Villarroel, Gabriele Bavota, Barbara Russo, Rocco Oliveto, and Massimiliano Di Penta.  Release planning of mobile apps based on user reviews.  In _2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE)_ , pages 14–24. IEEE, 2016.
  * Kurtanović and Maalej (2017) Zijad Kurtanović and Walid Maalej.  Automatically classifying functional and non-functional requirements using supervised machine learning.  In _2017 IEEE 25th International Requirements Engineering Conference (RE)_ , pages 490–495. IEEE, 2017.
  * Li et al. (2006) Yuhua Li, David McLean, Zuhair A Bandar, James D O’shea, and Keeley Crockett.  Sentence similarity based on semantic nets and corpus statistics.  _IEEE transactions on knowledge and data engineering_ , 18(8):1138–1150, 2006.
  * Berry (2017) Daniel M Berry.  Evaluation of tools for hairy requirements and software engineering tasks.  In _2017 IEEE 25th International Requirements Engineering Conference Workshops (REW)_ , pages 284–291. IEEE, 2017.
  * Zhao et al. (2021) Liping Zhao, Waad Alhoshan, Alessio Ferrari, Keletso J Letsholo, Muideen A Ajagbe, Erol-Valeriu Chioasca, and Riza T Batista-Navarro.  Natural language processing for requirements engineering: A systematic mapping study.  _ACM Computing Surveys (CSUR)_ , 54(3):1–41, 2021\.
  * Haering et al. (2021) Marlo Haering, Christoph Stanik, and Walid Maalej.  Automatically matching bug reports with related app reviews.  In _2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)_ , pages 970–981. IEEE, 2021.
  * Zave (1999) Pamela Zave.  Faq sheet on feature interaction, 1999.  URL https://www.pamelazave.com/faq.html.
  * Alfakeeh et al. (2022) Ahmed S Alfakeeh, Ali H Al-Bayatti, Francois Siewe, and Thar Baker.  Agent-based negotiation approach for feature interactions in smart home systems using calculus of the context-aware ambient.  _Transactions on Emerging Telecommunications Technologies_ , 33(2):e3808, 2022.
  * Cameron and Velthuijsen (1993) E Jane Cameron and Hugo Velthuijsen.  Feature interactions in telecommunications systems.  _IEEE Communications Magazine_ , 31(8):18–23, 1993.
  * Breaux and Antón (2008) Travis Breaux and Annie Antón.  Analyzing regulatory rules for privacy and security requirements.  _IEEE transactions on software engineering_ , 34(1):5–20, 2008.
  * Liao et al. (2020) Song Liao, Christin Wilson, Long Cheng, Hongxin Hu, and Huixing Deng.  Measuring the effectiveness of privacy policies for voice assistant applications.  In _Annual Computer Security Applications Conference_ , pages 856–869, 2020.
  * Fisler et al. (2005) Kathi Fisler, Shriram Krishnamurthi, Leo A Meyerovich, and Michael Carl Tschantz.  Verification and change-impact analysis of access-control policies.  In _Proceedings of the 27th international conference on Software engineering_ , pages 196–205, 2005.
  * Ebrahimi et al. (2021) Fahimeh Ebrahimi, Miroslav Tushev, and Anas Mahmoud.  Mobile app privacy in software engineering research: A systematic mapping study.  _Information and Software Technology_ , 133:106466, 2021\.
  * Wisniewski et al. (2017) Pamela J Wisniewski, Bart P Knijnenburg, and Heather Richter Lipford.  Making privacy personal: Profiling social network users to inform privacy education and nudging.  _International Journal of human-computer studies_ , 98:95–108, 2017.
  * Tushev et al. (2022) Miroslav Tushev, Fahimeh Ebrahimi, and Anas Mahmoud.  Domain-specific analysis of mobile app reviews using keyword-assisted topic models.  In _2022 IEEE/ACM 44th International Conference on Software Engineering (ICSE)_ , pages 762–773. IEEE, 2022.