∎

11institutetext: M. A. Hadi 22institutetext: Dept. of Computer Science
University of British Columbia
Kelowna, Canada
Tel.: +1 250 899 6971
22email<EMAIL_ADDRESS>33institutetext: F. H. Fard 44institutetext:
Dept. of Computer Science
University of British Columbia
Kelowna, Canada
44email<EMAIL_ADDRESS>
# Evaluating Pre-Trained Models for User Feedback Analysis in Software
Engineering: A Study on Classification of App-Reviews ††thanks: This research
is supported by a grant from Natural Sciences and Engineering Research Council
of Canada RGPIN-2019-05175.

Mohammad A Hadi  Fatemeh H Fard

(Received: date / Accepted: date)

###### Abstract

Context: Reviews of mobile apps on App Stores and on social media are valuable
resources for app developers. Analyzing app reviews have proved to be useful
for many areas of software engineering (e.g., requirement engineering,
testing, etc.). Existing approaches rely on manual curating of a labeled
dataset to classify app reviews automatically. In practice, new datasets must
be labeled as the classification purpose changes (e.g., identifying bugs
versus usability issues or sentiment), limiting the power of developed models
to classify new classes or issues. Employing models that can produce the same
or better result using less labeled data can eliminate the problem by reducing
this manual effort. Recent Pre-trained Transformer based Models (PTM) are
trained on large natural language corpora in an unsupervised manner to retain
contextual knowledge about the language and have found success in solving
similar Natural Language Processing (NLP) problems. However, the applicability
of PTMs has not yet been explored for issue classification from app reviews.

Objective: We explore the advantages of PTMs for app review classification
tasks by comparing them with existing models; we also examine the
transferability of PTMs by applying them in multiple settings.

Method: We select six datasets from the literature which contain manually
labeled app review from Google Play Store, Apple App Store, and Twitter data.
We empirically study the performance and time efficiency of PTMs compared to
Prior approaches. In addition, we evaluate the performance of the PTMs when
app reviews are incorporated in their pre-training (i.e., domain-specific
PTMs). We set up different studies to evaluate PTMs in multiple settings:
binary vs. multi-class classification, zero-shot classification (when new
labels are introduced to the model), multi-task setting, and classification of
reviews from different resources. In all cases, Micro and Macro Precision,
Recall, and F1 scores are used and the time required for training and
prediction with the models are also reported.

Results: Our results show that PTMs can classify the app issue with higher
scores. However, in the multi-resource setting, the Ensemble method and Deep
learning with word embedding score higher than PTMs. When PTMs are pre-trained
with app specific reviews (i.e. Custom PTMs), the models achieve the highest
scores among all in all settings, and the prediction times are also reduced;
with the highest scores belonging to customized PTMs that are pre-trained with
a larger number of app reviews.

Conclusion: For app issue classification, the PTMs are the best choices in
some settings, compared to Prior approaches; but they are not the best ones
for all the studied settings. It is worth considering the customized PTMs pre-
trained on app reviews when higher performance and lower prediction times are
required.

###### Keywords:

pre-trained transformer models app review classification registered report

##  1 Introduction

Mobile application (app) marketplaces, such as Apple App Store and Google Play
enable the users to rate and review apps hadi . Users express their usage
experience through writing reviews from different perspectives, such as the
app’s quality, performance, and functionality. The reviews provide a distinct
way for the application developers to acquire customer feedback luliang . The
developers monitor the reviews regularly to address users’ major concerns and
resolve the reported issues in the forthcoming app update. If the app is not
regularly updated addressing the issues, it will gradually lose its popularity
liu2015 ; islam2010 . Therefore, app reviews have been studied extensively
2020survey  and are shown to be resourceful for requirement engineering
luliang , release planning ciurumelea2017analyzing , software maintenance
al2020classification , change-file localization zhou2020TSE , and testing
grano2018testing . Different studies also focused on analyzing the reviews to
filter non-informative reviews arminer , identify bugs maalej , classify
usability and quality concerns luliang , and find security and privacy issues
besmer2020privacy .

Other than App Stores, mobile app users also express their opinions on social
media such as Twitter, which regularly reveal new information for the
developers stanik . Nonetheless, the manual extraction of informative feedback
from App Stores or social media is difficult, as there are many unrelated and
noisy user comments. Therefore, researchers have developed techniques,
including rule-based, machine learning, or deep learning approaches, for
automatic extraction of useful information from user reviews to help app
developers arminer ; gukim ; guzman ; stanik . There are many studies that
classify the reviews for different purposes, which range from identifying
problems, user inquiries, feature requests, and aspect evaluations (e.g.,
feature strengths, weaknesses, and performance) to sieving usability,
portability, and reliability stanik ; 2020survey ; luliang ;
ciurumelea2017analyzing ; maalej ; besmer2020privacy . These studies provide
app developers with a variety of serviceable information that facilitates
making informed decisions for planning the app updates. This extra information
is crucial for app developers as mobile applications and their updates get
released quite frequently over a short period to meet the market requirements;
so, it often proves challenging to identify and prune all the software defects
and bugs during the testing phase Joorabchi .

The topics of interest in app review classification are numerous and different
studies leverage various supervised machine learning techniques to extract
useful software engineering information from app reviews 2020survey . A main
requirement of supervised techniques is the availability of labeled datasets,
which needs to be done manually by field-experts stanik . Although app review
analysis is shown to be useful for software developers in practice, this
manual effort is expensive and seems to be a barrier in many studies
2020survey . Recent works try to alleviate the problem by incorporating semi-
supervised learning deocadez2017semi-supervised  or active learning
dhinakaran2018active-learning . However, with the emergence of new labels of
interest, e.g., privacy or security issues besmer2020privacy , developers must
label new datasets, which renders the previously curated datasets obsolete. In
addition, the distribution of the data collected from different platforms
(i.e., App Stores or social media) varies. Therefore, even though the same
labels are to be classified from two platforms, a model that has been trained
on one (e.g., Google Play) needs to be retrained to extract the same labels
from another (e.g., Twitter) to yield similar performance stanik . Whenever
the classification purpose (i.e., labels of interest) or platform changes, the
models require retraining on newly labeled datasets since they do not retain
the natural language properties learned from previous training, including the
linguistic features and the context of the trained labels maalej . Training
from scratch prevents the extensibility of the developed models for
classifying new issues in practice.

A more recently established and widely accepted practice in Natural Language
Processing (NLP) is using Pre-Trained Language Models (PTM) and then
transferring its learned knowledge to various downstream NLP tasks, such as
sentiment analysis, question answering, or classification qiu2020pre . In NLP,
PTMs (such as BERT) are large language models that are trained on large
natural language corpora using a deep neural network in an unsupervised manner
how-far-ptm . These models are then fine-tuned for various downstream tasks
using limited labeled datasets. As PTMs are trained on large general domain
corpora, they learn contextual linguistic information and eliminate the need
to train downstream task models from scratch wada2020medicalBERT . PTMs reduce
the amount of effort (i.e., new model development time per task) to build
models for each task separately, and they reduce the amount of required
labeled dataset roberta . Consequently, PTMs are used to transfer the learned
knowledge to a new domain or a new task, and in settings where a model has not
seen any example of the required task or interested label during training
(known as zero-shot learning) zero . Although PTMs are used extensively and
led to many advances in NLP, their applicability for software engineering is
barely known. Only a few studies exist that explore the use of PTMs for
sentiment analysis in software engineering how-far-ptm ; biswas2020achieving ,
or for tasks related to programming languages such as comment generation
Svyatkovskiy . But, to what extent PTMs can be applied for app review
classification is unknown.

Moreover, the PTMs that are trained on domain-specific data, show significant
improvements over general purpose PTMs such as BERT (i.e. PTMs that are
trained on general purpose data) for NLP tasks in fields such as science or
law sci-bert ; legal-bert . These domain-specific PTMs leverage unsupervised
training on a large domain-specific corpus to compensate for the lack of high-
quality, large-scale labeled data in the specified domains. Our problem of
interest, the classification of app reviews, aims to extract useful
information for software engineers/app developers, which are domain-specific
information. In addition, app reviews are short text, and many reviews are
noisy and unrelated, which can introduce more complications hadi . To what
extent the PTMs can be useful in the context of app review classification and
whether domain-specific PTMs trained on app reviews can improve the results is
unknown. Previously, the deep learning approaches used Convolutional Neural
Networks (CNN) to implement the transfer of static linguistic knowledge from
non-contextual word embedding, which has shown to have comparable results with
traditional machine learning approaches for classification of requirements
related app reviews stanik . In contrast, another study using CNN shows
improvements over previous approaches for app review classification
aslam2020convolutional . Though these studies do not use PTMs, they use a deep
learning approach which sheds doubt on whether PTMs can be useful.

Therefore, in this study, we aim to explore the benefits of PTMs compared to
the existing approaches for app review analysis, specifically the app review
classification tasks. We define the app review classification as the task of
extracting useful information from users’ feedback which can be related to app
requirements, release planning, and software maintenance. The extracted
information may help identify different aspects of the application, such as
feature requests, aspect evaluations (e.g., feature strength, feature
shortcoming, application performance), usability, portability, reliability,
energy consumption, problem reports, privacy, and security inquiries about the
application. Our goal is to investigate the accuracy and time efficiency of
PTMs for the app issue classification task over different selected datasets
from literature with various labels and multiple tasks (i.e., issue
classification and sentiment analysis of app reviews). Therefore, experiments
will be conducted in different settings: by fine-tuning the PTMs on different
sizes of the labeled dataset for downstream tasks, exploring the same PTMs for
multiple tasks in app review analysis, and finally, comparing the performance
of PTMs when they are trained on non-domain-specific dataset versus PTMs
trained on the domain-specific dataset. These experiments will provide
baselines on the applicability of PTMs for app review analysis, including the
cost of using them (in terms of required time for predictions) and their
capability to reduce the manual effort required to label large datasets.

The contributions of this study are as follows:

  * •

This is the first study that explores the applicability of PTMs for automatic
app issue classification tasks compared to the existing tools.

  * •

We conduct an extensive comparison between four PTMs and four existing
tools/approaches on six different app review datasets with different sizes and
labels.

  * •

We are the first to explore the performance of general versus domain-specific
pre-trained PTMs for app review classification.

  * •

This is the first empirical study to examine the accuracy and efficiency of
PTMs in four different settings: binary vs. multi-class classification, zero-
shot setting, multi-task setting, and setting in which training data is from
one resource (e.g., App Store) and the model is tested on data from another
platform (e.g., Twitter).

We explore the following research questions for our study:

RQ1: How accurate and efficient are the PTMs in the classification of app
reviews compared to the existing tools?  In this research question, we explore
how the current PTMs perform compared to the existing tools, including their
required time for training and prediction. The existing tools are based on
curated rules, feature engineered machine learning algorithms, and deep
learning models.

Findings: We find that different PTMs outperform Prior approaches by $\sim$3%
to $\sim$15% on all datasets. Additionally, the best-performing PTMs require
slightly more time for prediction, although the difference in prediction time
is negligible (+0.04s to +2.87s).

RQ2: How does the performance of the PTMs change when they are pre-trained on
app-review dataset, instead of a generic dataset (e.g. Wiki-documents, book
corpus)? The current PTMs are trained using general text scraped from the web.
In some studies, domain-specific models are trained e.g. LEGAL-BERT legal-bert
and medical BERT wada2020medicalBERT  which have been shown to improve the
performance compared to the non-domain-specific models. In RQ2, we intend to
explore the performance of a domain-specific pre-trained model when trained on
app reviews.

Findings: The PTMs trained from scratch on domain-specific data (i.e. adding
app-reviews to the model) performed better than out-of-the-box models (+0.8%
to +2.2% micro-F1 score). The Custom PTMs trained from scratch do not
fluctuate much in prediction times with respect to the readily available PTMs
for app issue classification. In addition, we find that incorporating a
greater number of app reviews in the pre-training can help PTMs produce up to
15.2% better micro-F1 score.

RQ3: How do the PTMs perform in the following settings?
(RQ3-1) Binary vs multi-class setting,
(RQ3-2) Zero-shot classification,
(RQ3-3) Multi-task setting (i.e. different app-review analysis tasks),
(RQ3-4) Classification of user-reviews collected from different resources
(i.e., Twitter, App Store).

The answers to this research question will help us understand the
applicability of PTMs and their performance in different settings (i), when
the classification is on two or more classes; (ii), when a labeled dataset is
not available; (iii), the task changes; and (iv) different resources are used.
In part (iv) we explore the transferability of the models as the distributions
of data on various platforms is different ruder2017transfer .

Findings: Both Prior and PTM models yield better performance for binary
classification tasks than for multi-class settings. PTMs are the best choice
for zero-shot classification settings. Custom PTMs improve the results of
their non-domain-specific models. For multi-task and multi-resource settings,
the Custom LARGE PTMs (i.e., PMTs pre-trained with 10 million app reviews)
performed better than readily available PTMs. RoBERTa-based Custom PTMs have
the best scores among all models. They also have lower prediction times in the
multi-resource setting than Prior approaches.

Deviations from the Registered Report. The following are the parts that we
have changed compared to the Registered Report.

  * •

In the multi-task setting (RQ3-3), we add experiments on another dataset, in
addition to the previous dataset mentioned in the Registered Report. This new
dataset is on app reviews, but classifies them as different categories (e.g.
game, family). This new dataset is added to assess the performance of these
models for a different classification task but in the same domain.

  * •

Another difference of this submission with the original plan in the Registered
Report is on the multi-resource setting (RQ3-4). Here, we add experiments on
the Prior approaches as well to examine their capabilities for this setting
and compare the results of PTMs against them.

Paper Organization.  The rest of the paper is organized as follows. We review
the related works in Section 2. Section 3 discusses the Study Overview,
including research design, datasets, selection of the approaches, and
experimental setup. In Section 4 we detail our approach to answer each of the
research questions and discuss the results in Section 5. Section 6 is
dedicated to the discussions. Threats to validity are in Section 7 and we
conclude the paper in Section 8.

##  2 Related Works

###  2.1 App Review Classification in Software Engineering

####  2.1.1 Topic Modeling Based Models

According to Silva et al. silva2021topic , topic modeling has been applied to
software engineering research, including app review classification. Among the
most frequent topic modeling techniques used to categorize app reviews, the
researchers found that Latent Dirichlet Allocation (LDA) lda  and LDA-based
techniques are the most common. AR-Miner is one of the initial works for
mining app reviews proposed by Chen et al. arminer , which uses topic modeling
to group the informative reviews and investigates categories of users’
discussions. Nayebi et al. nayebi2018app  examined how Twitter app reviews can
contribute to mobile app development and applied topic modeling and
crowdsourcing. Adaptive Online LDA was developed by Gao et al. to classify app
reviews based on users’ feedback on various versions of mobile apps
gao2018online . Consequently, Hadi and Fard developed Adaptive Online Biterm
Topic Model (AOBTM) hadi2020aobtm  to model topics of app reviews in different
categories adaptively, therefore, alleviating the sparsity problems in short-
texts, which considered the statistical data for multiple previous time
slices.

Topic modeling approach is used in other works gao2018online ; SRM-citer2 ;
yang2021tour ; wardhana2021aspect ; gao2018online  to categorize app reviews
for detecting emerging issues for developers to update their apps, identifying
fine-grained app features in the reviews to extract users’ sentiments,
prioritizing important user reviews for developers, and performing aspect-
based sentiment analysis of app reviews.

####  2.1.2 Machine Learning Based Approaches

Earlier research has focused on using machine learning approaches for app
review filtering. Chen et al. arminer  classified non-informative reviews by
training a classifier and categorizing them into informative and non-
informative groups. Using a regression model, Fu et al. making_sense  filter
out rating-inconsistent reviews that differ in sentiment from their ratings.
In gukim , Gu et al. identified software features and distinguish between
features that are evaluated and those that are requested, using Max-Entropy
nigam1999maxEnt . There are other studies that used machine learning
techniques for app issue classification: shah2018simple  and Rustam2020class
use lexical features and multi-text features to use in conjunction with
different machine learning models for app review classification task. Maalej
and Nabil maalej  empirically evaluated different feature extraction
techniques to work with machine learning models to automatically classify bug
reports and feature requests. Martin et al. EM-citer1  and Sarro et al.
sarro2015  have also explored the breadth of research done on App Store
analysis (i.e., feature life cycle, dormant feature identification) for
software engineering.

In guzman ; Triantafyllou2020how ; Messaoud2019multi ; fu2021machine ;
guo2019enhancing , researchers have used ensemble machine learning techniques
to classify app reviews into categories relevant to software evolution, to
identify more efficiently and earlier terms in reviews that could be
classified into specific topics, and to filter out useful feedback from users,
such as feature requests, bug descriptions, and requirement analysis.

####  2.1.3 Deep Learning Based Approaches

Using machine learning, stanik  demonstrated that they could still achieve
comparable results with deep learning when analyzing user feedback in English
and Italian into problem reports, inquiries, and irrelevant. Aslam et al.
proposed an approach for app reviews classification using non-textual
information to classify app reviews and exploiting deep learning technique
aslam2020convolutional . In mekala2021class , a BERT-based sequence classifier
was developed and validated to achieve a state-of-the-art average
classification accuracy (87%) for feedback analysis. From a large volume of
online product reviews, Qiao et al. qiao2020deep  proposed to apply a domain-
oriented approach to deep learning to discover the most critical users’ needs,
such as app product new features and bug reports. Henao et al.
henao2021transfer  investigated the prospect of transfer learning for the
classification of app reviews and found that monolingual BERT models
outperform existing baseline methods in the classification of English App
Reviews.

Other studies he2019detecting ; haering2021automatic ; Aralikatte2018fault ;
wang2020analyzing ; hadi ; zhao2019sentiment  also applied deep learning
techniques to different extents, such as detecting promotion attacks, matching
bug reports, detecting adversarial spam, analyzing energy-related reviews, and
requirement evolution predictions from app reviews.

###  2.2 Pre-trained Models

There are two generations of pre-trained models: static and contextual word
embeddings ptm_survey . The Global Vectors for Word Representation (Glove)
glove  and word2vec w2v  are examples of first-generation static word
embeddings. Software Engineering (SE) research has used word embeddings for
various tasks, including code retrieval code_retrieval , detecting incoherent
comments comment_coherence , specifying SE-relevant tweets sieve , and program
comprehension with graphs program_representation . These kinds of embeddings
are context-independent, where each word has only one embedding and cannot
change with different contexts. These embeddings only take the syntax of the
words in a sentence into account. Therefore, non-contextual embeddings cannot
capture semantic information and model polysemous words ptm_survey .

The second-generation pre-trained models learn context-sensitive word
representations and can be tailored to perform downstream tasks effectively.
LSTM lstm -based Universal Language Model Fine-tuning (ULMFiT) ulmfit  and
Embeddings from Language Models (ELMo) deep_context  have also been used for
different SE-related tasks, such as identifying ambiguous software
requirements ulmfit_citer  and sentiment analysis for software engineering
elmo_citer . Transformer architectures attention  have been used to build
several pre-trained Transformer-based models bert ; albert ; roberta ; xlnet
that have achieved state-of-the-art performance for a variety of downstream
tasks, such as sentiment classification of SE-related texts how-far-ptm  and
identifying error-prone software error_prone .

Other studies Rajpurkar2016sqaud ; Reddy2019cpqa ; yang2018hotpotqa ;
Zhang_retrospective ; ju_technical ; tu_select  have used PTMs for solving
Question-Answering problem. PTMs have also been extensively used the sentiment
analysis task Bataa_investigation ; sun2019bert ; xu2019bert ; Rietzler_adapt
; karimi_adversarial ; li2019exploiting ; wux_mask ; peters2017semi . Previous
studies liu2018efficient ; hakala2019biomedical ; edunov2019ptm  developed
Named Entity Recognition systems with the help of PTMs. Others
clinchant2019bert ; imamura2019recycling ; zhang2019hibert  studied how PTMs
can improve machine translation tasks. Other works include using PTMs for text
summarization liu2019text ; zhong2020extractive ; jin2020BERT  and adverserial
defense and attacks identification wallace2019universal ; sun_adv ; lil_bert ;
zhuc2020freelb ; liu_adversarial .

Though the studies that use PTMs or classify app reviews are numerous, there
is no previous work that investigates their capability for app issue
classification as we conduct in this work.

##  3 Study Overview

In this section, we first overview the design of the research, and then
provide details about the selected datasets, Prior approaches, and chosen pre-
trained models. We also discuss what metrics will be used to evaluate the
models, as well as explain the experimental setup.

###  3.1 Research Overview

Figure 1: Overview of Research Steps for RQ1

Figure 2: Overview of Research Steps for RQ2 (left) and RQ3 (right)

Figures 1 and 2 show the overall process of our study. First, we select 6
widely used datasets from the literature for app review classification. We
also combine these datasets and make another dataset, which has multiple
labels and is highly imbalanced ($D_{7}$). We choose four approaches from the
literature that we refer to as Prior approaches. Four pre-trained language
models are selected to evaluate their capabilities regarding the research
questions in this study. We evaluate all the Prior approaches and the PTMs for
the datasets. From the PTMs, the best performing ones are selected to train
Custom PTMs (CPTM) and assess them in RQ2 for all the seven datasets. In RQ3,
we evaluate the performance of the PTMs and Custom PTMs in four different
settings. The details of the datasets and selected approaches and PTMs are
explained in next sub-sections.

###  3.2 Datasets

App reviews are analyzed automatically for various purposes and reviews are
classified with different labels such as user concerns, feature requests,
feature modifications, bug reports, and usability analysis aobtm . In our
experiments, we examine the performance of PTMs for app issue classification.
Therefore, we select a diverse set of datasets from the literature as
described below. These datasets have various sizes, different labels, and
collected from different platforms (i.e., Google Play, Apple App Store,
Twitter), and therefore, can represent to what extent the PTMs can be useful
for app issue classification.

Dataset 1 ($D_{1}$) This dataset is procured by Gu and Kim gukim  and contains
reviews for 17 popular Android apps from Google Play in 16 most popular
categories, such as games and social. The authors have manually labeled 2,000
reviews for each app (34,000 in total) in five classes according to their
predefined rules: Aspect Evaluation (5,937), Praise (8,112), Feature Request
(2,323), Bug Report (2,338), and Others (15,290).

Dataset 2 ($D_{2}$) This dataset contains 6,406 app reviews from Google Play
and 10,364 tweets (sampled from 5 million tweets written in English) which are
labeled manually into three classes: Problem Report, Inquiry, and Irrelevant
stanik . They included Problem Report (1,437), Inquiry (1,100), and Irrelevant
(3,869) records from Google Play. The number of records in each of these
categories from Twitter data is 2,933, 1,405 and 6,026, respectively.

Dataset 3 ($D_{3}$) This dataset is provided by Lu and Liang. luliang .
Researchers selected two popular Apps, one from Apple App Store (iBooks in the
books category) and one from Google Play (WhatsApp in the communication
category). They sampled the raw reviews collected from each platform and
manually classified 2,000 reviews for each app (4,000 in total). The reviews
are classified into six categories: Usability (432), Reliability (587),
Portability (119), Performance (121), Feature Request (558), and Others
(2,183).

Dataset 4 ($D_{4}$) This dataset was procured by Maalej and Nabil maalej  and
contains 2,000 manually labeled reviews from random apps selected from top
apps in different categories (1,000 from Apple App Store and 1,000 from Google
Play), where half of the apps are paid and half of them are free apps. The
authors ensured that there are 200 reviews for each of the 1, 2, 3, 4, and 5
stars in each of the 1000-reviews datasets. In addition, they provide 2,400
more reviews which are manually labeled. These reviews are for selected 3
random Android apps and 3 iOS apps from the top 100 apps (400 reviews for each
app). This dataset contains 4,400 labeled reviews in total with four
categories: Bug Report (378), Feature Request (299), User Experience (737),
and Rating (2721).

Dataset 5 ($D_{5}$) This dataset is published by Guo et al. guo2020caspar  and
contains 1,500 app reviews, which are manually labeled as User Action (428),
App Problem (399), and Neither (673). The reviews are selected from over 5.8
million records for 151 apps from Apple App Store.

Dataset 6 ($D_{6}$) This dataset was procured by Guzman et al. guzman ; it
contains reviews of 3 apps from Apple App Store and 4 apps from Google Play.
The apps are popular and from diverse categories. They sampled 260 reviews per
app and five annotators labeled the records, summing to 1,820 reviews that are
manually labeled. This dataset includes seven categories, namely: Bug Report
(990), Feature Strength (644), Feature Shortcoming (1281), User request (404),
Praise (1703), Complaint (277), and Usage Scenario (593)111Note that adding
numbers in all categories will exceed the total number, because some reviews
belong to multiple groups. We will follow the steps in guzman  to calculate
the evaluation metrics for this dataset..

Reasons behind choosing these Datasets ($D_{1}$ to $D_{6}$): We considered
these datasets for their diverse characteristics from three perspectives,
other than having different categories. First, the considered datasets were
constructed from various repositories: Apple App Store, Google Play, and
Twitter. Second, the size of the datasets vary and they have different number
of records for each label. For example, Dataset 2 is around four times larger
than Dataset 5. Also, Dataset 2 contains approximately 2,000 reviews per
label, whereas Dataset 1 has around 400 reviews per label. The size
differences can provide insights about the ability of PTMs for different sizes
(total training set and the available data for each label). Third, except
Dataset 1 and Dataset 5 which are balanced dataset, the other four datasets
are imbalanced. These differences ensure that we explore the capability of
PTMs for different sizes, platforms, and in more realistic settings.

Merged Dataset ($D_{7}$) This dataset will be compiled by merging all the 6
datasets, $D_{1}$–$D_{6}$. This Merged Dataset has 55,933 app reviews with 16
labels. As some of the labels used in the datasets are the same or have
similar definitions, we have grouped some of them. For grouping the labels, we
have consulted the definitions of the classes from the studies that published
the datasets $D_{1}$–$D_{6}$. This dataset would be closer to practical
applications, where a lot of irrelevant data exists and multiple classes of
informative reviews are of interest to be extracted from this highly
imbalanced dataset. The labels in the Merged Dataset, the grouped labels from
$D_{1}$–$D_{6}$, and the number of app reviews per group are provided in Table
1.

Table 1: Description for Merged Dataset (D7) Labels in Merged Dataset |  Grouped Labels [Label Name: Dataset Initial (# of Reviews)] |  # App Reviews in Total
---|---|---
Performance |  Performance: D3 (121) |  121
Portability |  Portability: D3 (119) |  119
Usability |  Usability: D3 (432) |  432
Reliability |  Reliability: D3 (587) |  587
Usage Scenario |  Usage Scenario: D6 (593) |  593
Feature Strength |  Feature Strength: D6 (644) |  644
User Experience |  User Experience: D4 (737) |  737
Feature Shortcoming |  Feature Shortcoming: D6 (1,281) |  1,281
Inquiry |  Inquiry: D2 (1,100), User Action: D5 (428) |  1,528
Problem |  Problem Report: D2 (1,437), App Problem: D5 (399), Complaint: D6 (277) |  2,113
Rating |  Rating: D4 (2,721) |  2,721
Bug Report |  Bug Report: D1 (2,338), Bug Report: D4 (378), Bug Report: D6 (990) |  3,706
Feature Request |  Feature Request: D1 (2,323), Feature Request: D3 (558), Feature Request: D4 (299), User request: D6 (404) |  3,584
Aspect Evaluation |  Aspect Evaluation: D1 (5,937) |  5,937
Praise |  Praise: D1 (8,112), Praise: D6 (1703) |  9,815
Irrelevant |  Others: D1 (15,290), Irrelevant: D2 (3,869), Others: D3 (2,183), Neither: D5 (673) |  22,015
Total |  |  55,933

###  3.3 Prior approaches

In this section, we discuss the four widely-used approaches for app review
analysis and the reasons of choosing them in our experiments. We have
conducted literature review and reviewed analysis of app review surveys to
select these Prior approaches.

AR-Miner Chen et al. proposed App Review Miner (AR-Miner) arminer  which
extracts valuable information from user reviews. Provided a collection of user
reviews, AR-Miner first applies a pre-trained classifier that separates non-
informative reviews. Then, AR-Miner applies Latent Dirichlet Allocation (LDA)
over the informative reviews to chunk them into different groups for
prioritizing them by an efficient ranking model proposed by the authors. AR-
Miner has been widely used to mine app-issues from user-reviews
panichella2015can ; ARM-citer2 ; ARM-citer3 ; ARM-citer4 ; ARM-citer5 ; ARM-
citer6 . Therefore, it is considered as a prior approach in our experiments.

Reason behind choosing AR-Miner: AR-Miner is an unsupervised approach and its
performance could be evaluated against other models. We mainly choose these
techniques to evaluate the PTMs against unsupervised, supervised, ensemble
methods using classical machine learning approaches, and finally deep learning
technique that is enriched by contextual embedding. AR-Miner is the first
framework to accommodate application developers in mining informative topics
from a large volume of app reviews. This was the first effective attempt to
leverage an unsupervised Topic Modeling technique to extract edifying issues
from app reviews. Developers embraced AR-Miner for its ability to filter out
non-informative reviews and display the key user feedbacks in an intuitive,
concise manner. The effectiveness of AR-Miner was validated by conducting a
comprehensive set of experiments on user reviews of four Android apps.
Researchers compared the AR-Miner results against real app developers’
decisions in different studies panichella2015can ; ARM-citer2 ; ARM-citer3 ;
ARM-citer4 ; ARM-citer5 ; ARM-citer6 . They analyzed the advantages of AR-
Miner over manual inspection and other techniques used in a traditional
channel finkelstein ; johann ; moran ; sarro2015 . Based on the empirical
results, AR-Miner has proven to be effective and efficient in extracting
informative reviews.

SUR-Miner Gu et al. gukim  proposed Software User Review Miner (SUR-Miner),
which is a framework that summarizes users’ sentiments, opinions, and emotions
toward different aspects of an application. SUR-Miner parses aspect-opinion
pairs from reviews by considering their structures. For this purpose, it uses
pre-defined sentence
templates/patterns222https://guxd.github.io/srminer/appendix.html. Then, for
each review, it combines the sentiment of the sentences with its aspect-
opinion pairs. These are used in the final step to summarize the software
aspects. SUR-Miner generates reliable summaries and achieved high F1 score for
aspect-opinion identification, sentiment analysis, and app review
classification compared to the previous works SRM-citer1 ; SRM-citer2  by
using a simple but effective machine learning algorithm, Max-Entropy. It is
adopted in many studies SUR-citer3 ; palomba2017recommending ; SUR-citer4 ;
SUR-citer5 ; features  and therefore we choose it as a prior approach.

Reason behind choosing SUR-Miner: SUR-Miner is the first framework that fully
took advantage of user reviews’ monotonic structure and semantics; it also
defined sentence patterns to extract aspect opinion pairs from app-review.
Researchers carefully selected five distinct text features, specifically
tailored for app reviews, and leveraged these features to train a supervised
machine learning classifier. Based on the empirical evaluation, we choose this
machine learning approach as the terminal classes categorized by the SUR-Miner
model were shown to be significantly more accurate and precise than other
methods, demonstrating its effectiveness SRM-citer1 ; SRM-citer2 .

Ensemble Methods Guzman et al. systematically evaluated different sets of
ensemble methods and identified one for classifying user reviews guzman . They
selected machine learning techniques that were used for text classification,
and compared the performance of each algorithm individually, for app review
classification. They also studied the performance of these models when their
results are combined using ensemble methods. Ensemble methods are well known
machine learning techniques as they can enhance the prediction performance of
single classifiers, maintaining their strengths and reduce their
vulnerabilities. In this work, Logistic Regression, Naive Bayes, Support
Vector Machines, and Neural Network Classifier were grouped to vote for the
final prediction. This ensemble method outperformed the individual algorithms
with statistical significance. It was tested on a large dataset of seven
diverse apps, where it shows that in all cases, this ensemble method either
outperformed or matched its best baseline. Also, this method influenced
further research in the field of app review analysis EM-citer1 ; EM-citer2 ;
ciurumelea2017analyzing ; EM-citer4 ; EM-citer5 ; EM-citer6  and therefore is
used in our study.

Reason behind choosing Ensemble Methods: Different studies have attempted to
identify the best machine learning approach for app issue classification
without spending too much time perfecting custom-engineered features for
different issues maalejea ; augmenting ; luliang ; changelog ; features .
Guzman et al. guzman  first leveraged different machine learning algorithms
and combined their predictions to circumvent the problem of various machine
learning models being used for classification of issues from app reviews.
Their study showed that recall (an evaluation metric) improved significantly
after combining the predictions of Logistic Regression and Neural Networks.
Software developers and software evolution experts widely adopted this
approach to analyze user reviews and prioritize their tasks feedback ;
alsubaihin ; categorizing ; findingFeatures .

Deep Learning Model leveraging Non-contextual Word Embedding Stanik et al.
used a Deep Convolutional Neural Network (CNN) stanik  for classifying app
reviews. Their model contains an embedding layer and its weights are
initialized with a word embedding model, e.g. word2vec or FastText. This CNN
architecture outperforms the shallow Neural Networks classifying app reviews
in the determined groups, with a small margin. This method uses non-contextual
word embedding within a deep neural network and performed a transfer of
knowledge representation. This approach is one of the first studies leveraging
transfer learning and deep learning and is used in other studies for issue
classification task mcnn-citer1 ; mcnn-citer2 ; aslam2020convolutional , and
therefore, we choose it as a prior approach in our study.

Reason behind choosing this Deep Learning Approach: This study is a recent
work with the purpose of understanding and evaluating the extent to which deep
learning models could be used to categorize user feedback into different
predefined categories stanik . Based on this study, researchers determined
that the domain experts’ knowledge incorporation with traditional machine
learning model could achieve comparable results to those of the deep learning
approach due to the substantial performance improvements provided by using
simple yet powerful features in the traditional machine learning techniques.
Nevertheless, the developers and researchers widely adopted this deep learning
approach as the improvement of pre-trained word embeddings bring considerable
performance gain to the classifier in other domains yang2018 ; reimers2019 ;
ren2016 .

It is worth noting that the number of studies for app review classification is
numerous. We choose these four Prior approaches as they are well cited and
widely used in many other studies. More importantly, they use different
machine learning and deep learning approaches which can be representative of
multiple techniques used in other studies. Additionally, these models are open
sourced and are publicly available. The best of these four approaches in terms
of the performance metrics defined in Section 3.7 will be chosen as the
baseline.

We have already considered two more tools that we eliminated from our study:
AR-Doc and CLAP. AR-doc uses a parser based on a NLP Classifier, which uses a
set of 500 sentence-structures manually evaluated for four types of classes:
Information Giving, Information Seeking, Feature Request, and Problem
Discovery. To use AR-Doc, we will need to build different sets of sentence
structures for each dataset and their respective labels, which is infeasible.
In addition, we could not find any publicly available implementation of CLAP
and therefore unable to use CLAP333We contacted the authors of CLAP but did
not receive a response..

###  3.4 Pre-Trained Models (PTM)

In this section, we briefly discuss our choice of the four PTMs. All these
models have Transformer deep learning architecture which is based on attention
mechanism xlnet . As Transformer is currently the main architecture for PTMs
roberta , we choose the following Transformer PTMs: BERT, XLNet, RoBERTa, and
ALBERT. The same PTMs are also used for sentiment classification in software
engineering how-far-ptm .

BERT Devlin et al. bert  designed Bidirectional Encoder Representations from
Transformers (BERT) to learn contextual word representations from unlabeled
texts. Contextual word embeddings designate a word’s representation based on
its context by capturing applications of words across different contexts. BERT
employed a bidirectional encoder to learn the words’ contextual
representations by optimizing for Masked Language Model (MLM) and Next
Sentence Prediction (NSP) tasks. For MLM, 15% of all the tokens are replaced
with a masked token (i.e., [MASK]) beforehand, and the model is trained to
predict the masked words, based on the context provided by the non-masked
words. For NSP, the model takes sentence-pairs as input for learning to
predict whether a pair-match is correct or wrong. During training, 50% of the
inputs are true consequent pairs, while the other 50% are randomized non-
consequent sentence-pairs. Devlin et al. trained two versions: small-sized
BERTBASE and big-sized BERTLARGE. BERTBASE is a smaller model with 12 layers
and 110 million parameters. BERTLARGE has 24 layers and 340 million
parameters. BERTLARGE is more computationally expensive and consumes more
memory compared to BERTBASE. Thus, in this work, we will use BERTBASE.

Reason behind choosing BERT: BERT advanced the state-of-the-art for 11 NLP
tasks. It achieved an absolute improvement of 7.6% over the previous best
score on General Language Understanding Evaluation (GLUE)
benchmark444https://gluebenchmark.com/; it also achieved 93.2% accuracy on
SQuAD 1.1555Stanford Question Answering Dataset:
https://rajpurkar.github.io/SQuAD-explorer/explore/1.1/dev/, outperforming
human performance. As the tasks in these datasets require natural language
understanding and include classification tasks, we choose BERT.

XLNet XLNet xlnet  uses Auto-Regressive language modeling and Auto-Encoding.
The model is “generalized” because it captures the bi-directional context
using a mechanism called Permutation Language Modeling (PLM). XLNet integrates
auto-regressive models and bi-directional context modeling, yet overcoming the
disadvantages of BERT. PLM is the idea of capturing bidirectional context on
all permutations of terms present in an input sequence. XLNet discards the
one-directional linear modeling to maximize the log-likelihood over all
permutations of the sequence-terms. Each position is expected to learn
utilizing contextual information from the entire sequence, thereby capturing
the bidirectional context. No [MASK] is needed, and input data need not be
corrupted.

In addition, XLNET addresses capturing the dependency between masked
positions, which is neglected by BERT. Consider the sentence “New Delhi is a
city.” Then, input to BERT to be “[MASK] [MASK] is a city,” and the objective
of BERT would be predicting ”New” given ”is a city” and predicting ”Delhi”
given ”is a city”. In this objective, there is no dependency between learning
“New” and “Delhi.” So, BERT can result in a prediction like “New Gotham is a
city.” If we assume that the current permutation is [is, a, city, New, Delhi],
BERT would predict the tokens 4 and 5 independent of each other. Whereas,
XLNet, predicts in the order of the sequence. i.e., first predicts token 4 and
then predicts token 5: it computes the likelihood of ”New” given ”is a city”
plus the likelihood of ”Delhi” given ”New, is a city”.

Reason behind choosing XLNet: XLnet outperformed BERT on many NLP tasks; for 8
different tasks XLNet beat BERT by a substantial margin. This model achieved
best results for 18 NLP tasks, including sentiment classification and natural
language inference. As XLNET is outperforming other models on text
classification tasks, we choose it as one of the PTMs in our study.

RoBERTa Robustly optimized BERT approach (RoBERTa) outperformed all the state-
of-the-art benchmarks upon release roberta . Liu et al. modified BERT’s pre-
training steps that yield substantially better performance on all the
classification tasks. RoBERTa increased the amount of mini-batch sizes, data,
and training time to train the model. RoBERTa is also trained on dataset that
includes longer sequences than before. The masking pattern in RoBERTa was also
modified to be generated spontaneously.

Reason behind choosing RoBERTa: RoBERTa outperforms BERT on nine different NLP
tasks on the GLUE benchmark; it also equals or exceeds XLNet’s model in four
out of nine individual tasks. Based on these results, RoBERTa can present a
reasonable choice for PTM in our study.

ALBERT A Lite BERT (ALBERT) albert  applies three parameter reduction
techniques: Factorized embedding parameterization, Cross-layer parameter
sharing, and Inter-sentence coherence loss. In the first one, researchers
separated the hidden layers’ size from the input embeddings’ size (previously
of same sizes). They projected one-hot vectors to embedding and the hidden
space with lower dimensions; it increased the hidden layer-size without
significantly increasing the vocabulary embeddings’ parameter size. Second,
all the parameters across all layers are shared. The big-scale ALBERT model
has substantially fewer parameters than BERTLARGE. Finally, the NSP task is
swapped with Sentence-Order Prediction (SOP) loss which help ALBERT perform
better.

Reason behind choosing ALBERT: ALBERT uses parameter reduction techniques
which resulted in having 18 times less parameters than BERT. Its’ training
time is 1.7 faster and has negligible inferior performance than the original
BERTLARGE model. The much larger ALBERT architecture, which contains fewer
parameters than BERTLARGE, achieved higher F1 scores on the SQuAD 2.0 and the
GLUE benchmarks; it also achieved high accuracy on the ReAding Comprehension
from Examinations benchmark (RACE benchmark). Therefore, we choose ALBERT as
one of the PTMs in our experiments to evaluate its performance and inference
time for app issue classification, because ALBERT is shown to be faster than
other PTMs without sacrificing much performance.

For BERT, XLNET, RoBERTa and ALBERT, their base versions are used. Table 2
includes the names of the models used from the Huggingface Transformers
library huggingface  and default configurations.

Table 2: Details of PTMs Used in Our Study Architecture | Used Model | Parameters | Layers | Hidden | Heads
---|---|---|---|---|---
BERT | bert-base-cased | 110M | 12 | 768 | 12
RoBERTa | roberta-base | 125M | 12 | 768 | 12
XLNet | xlnet-base-cased | 110M | 12 | 768 | 12
ALBERT | albert-base-v1 | 11M | 12 | 768 | 12

###  3.5 Implementation

We are going to discuss the implementations of both Prior and PTM approaches
for the mentioned datasets.

####  3.5.1 Prior approaches

AR-Miner uses LDA lda  topic modeling algorithm to categorize text document
corpus into a predefined number of different related and coherent topics. For
app review classification, we hid the labels of the considered datasets and
presented the app reviews altogether as a text corpus, where each review
represented different documents. If a dataset has $n$ classes, we would
extract $n$ number of topics from the app review corpus; so, the documents
will be separated into $n$ number of segments. After the extraction of topics,
we looked into the associated labels for each review for assigning each
document cluster (topic) to a relevant class/label. The relevance was measured
by the prevalence of a class or label in a topic after extraction. Suppose,
the highest number of app reviews belong to label/class $x$ in a topic
(document cluster), we would understand that AR-Miner has labeled all
documents in that cluster with class $x$, where the app reviews that truly
belonged to class/label $x$ are regarded as the True Positives and the rest of
the reviews are regarded as False Positive.

SUR-Miner used Max-Entropy model to classify app reviews. This classifier
model considers all probability distributions empirically consistent with the
training data and chooses the distribution with the highest entropy.
Empirically consistent probability distribution with training data is one in
which an estimated frequency of occurrence of a class and a feature value is
equal to the actual frequency in the training data. Following the literature
gukim , we have implemented Generalized Iterative Scaling (GIS) as our scaling
method and performed Early Stopping to avoid over-fitting and to stop the
iterative solver from taking a long time to optimize over a large number of
feature weights. As mentioned in gukim , we have also adopted 2-4 Grams
(Character N-Gram) for the classification task.

Ensemble Approach adopted an ensemble of Logistic Regression classifier and
shallow Neural Network classifier, and applied the majority voting scheme to
combine the output of the classification approaches, by following the
literature guzman . We have concatenated TF-IDF vector representations of app
reviews, number of words in the review, number of characters in the review,
number of lower case characters, number of upper case characters to use as
features for our ensemble approach. We have used Adam optimizer and binary
cross entropy as our loss function in the shallow neural network.

Deep Learning approach with Word Embeddings followed stanik . For the neural
network’s input layer, text inputs must have a fixed size. This input size has
been fixed at 400 words, a measure we found suitable for all the app reviews
in all of the datasets we studied, and we identified no app reviews that
exceeded 400 words. In addition to the input layer, our network consists of an
embedding layer (pre-filled with a fast-text model), a 1D convolution layer, a
1D global max-pooling layer, a dense layer, and a final output layer with a
softmax activation. Tanh activation was used for the previous layers. The
embedding layer weights were frozen during training, leaving the trainable
parameters.

###  3.6 Experimental Setup

We will train all the Prior approaches and fine-tune all the PTMs on each of
the datasets separately. In the following, we provide the general information
about training the models using stratified k-fold cross validation, as well as
how PTMs will be trained and a classifier is added on top of each model. The
detailed process for each research question is provided in section 4.

To avoid introducing bias to the results due to probable differences in the
distribution of the training and test splits, following stanik , we will use
k-fold cross validation on each dataset $D_{i}$ separately, where $D_{i}$
represents one of the seven datasets in our study. We use k-fold cross
validation as a more rigorous approach than splitting the dataset into train
and test set once how-far-ptm . Also, this method is used previously for app
review classification in the software engineering domain micromacro . In the
k-fold cross validation, we split a dataset $D$ into $k$ equal size disjoint
parts/folds $D\textsuperscript{(1)},...,D\textsuperscript{(k)}$. We then build
$k$ classifiers $c\textsuperscript{(i)}$, each time using $k-1$ splits as
training and one part $D\textsuperscript{(i)}$ as test set. This way, each
split of the data can be used as test set once. As a result, we will have $k$
different test set performances. As the datasets in the study are imbalanced,
we will use an alternate option of cross validation called stratified k-fold
cross validation apple ; imbalancedLearning . In this method, the only
difference is that the distribution of the examples from each class in the
original dataset is preserved in each fold $D\textsuperscript{(i)}$. The
stratified k-fold cross validation is commonly used in machine learning
practices as it reduces the experimental variance. Therefore, when comparing
different methods, it is easier to identify the best method apple .  We
consider $k=5$ as this value has been shown empirically to have test error
rate estimates that do not suffer from high bias or high variance
james2013introduction . The value of $k=5$ is also used in previous app review
classification studies stanik . For the evaluation, we use metrics explained
in section 3.7. It is worth mentioning that we apply the stratified 5-fold
cross validation on each of the datasets $D_{1}$–$D_{7}$ separately. For
example, we split $D_{1}$ into 5 folds
$D_{1}\textsuperscript{(1)},...,D_{1}\textsuperscript{(5)}$ and build 5
classifiers for $D_{1}$. We then compute the evaluation metrics for $D_{1}$.
We continue this process for each of the datasets $D_{2}$–$D_{7}$ and report
the performance on each of them separately.

Note that the number of labels in the datasets $D_{1}-D_{6}$ ranges between
[3,7] and all the considered Prior approaches were designed for multi-class
classification. Therefore, we can retrain and evaluate them using the
stratified 5-fold cross validation. Among the Prior approaches, only AR-Miner
requires some adaptation. AR-Miner has three phases: classification of non-
informative reviews, topic modeling of informative reviews, and ranking of the
groups according to their relevance to developers’ applications. As all
datasets $D_{1}$–$D_{7}$ are labeled, we will directly utilize the second step
to group all the reviews (from both training and test set) in a dataset into
the corresponding number of labels available. Following the literature arminer
, we will count the training set reviews in a resulting group and use the
label with the majority count to annotate the group. This annotation indicates
the class that the reviews in the group belong to. Then, we will examine the
group’s test set reviews to compute the evaluation metrics.

For evaluating the PTMs, for each dataset, we will use the training set to
fine-tune the PTMs and then evaluate them on the test set. We will obtain all
the four PTMs that are pre-trained on general domain corpora from Hugging Face
library666https://huggingface.co/. To build a classifier, a feed-forward dense
layer and Soft-max activation function will be added on top of each model. We
follow how-far-ptm  and bert  to set the hyper-parameter values. In our work,
we set the batch size to 16 and Adam learning rate to $2e-5$. The models are
trained for 4 epochs and AdamW optimizer is used for all models.

We execute all the experiments on a Linux machine with Intel 2.21 GHz CPU and
16GB memory. For training PTMs from scratch, we use 2 $\times$ NVIDIA Tesla
V100 32GB to enhance the parallelization performance.

###  3.7 Evaluation Metrics

In this section, we describe the evaluation metrics. For all the
classifications, we will use three metrics: Precision (P), Recall (R), F1
score (F1), and their micro and macro average values. To answer the time
efficiency of the models, we will examine the training time for the Prior
approaches and the fine-tuning time for PTMs. We will also report the
prediction times and changes in time (increase or decrease of time compared to
a baseline).

Precision (P): Precision can be calculated by dividing the number of records
that their labels are correctly predicted by total number of predicted
observations in that class: $P=\frac{TP}{TP+FP}.$ Here, TP refers to the
number of records that their label is correctly predicted, and FP refers to
the the number of records falsely predicted to belong to this class. In multi-
class classification, for each group A, all the observations that belong to
other labels and are falsely predicted as group A are added to compute the FP.

Recall (R): For each group A, Recall can be calculated by dividing the number
of accurately predicted observations in A by the number of all observations
available in the corresponding class: $R=\frac{TP}{TP+FN}$ Here, FN is the
number of observations in class A which are falsely predicted as other labels.

F1 Score (F1): F1 score is the weighted average of Precision and Recall:

$\mbox{F1}=\frac{2\cdot(P\cdot R)}{P+R}$ (1)

As we have multi-class classification and the datasets are imbalanced, we use
micro-average and macro-average metrics as follows. Here, the micro-average
calculates the contribution of all records in all classes (therefore the
contribution of the class with the predominant number of records is taken into
account), whereas the macro-average is the average of the values for each
class (therefore, each class contributes equally in the final value). The
micro- and macro-averaged precision (P) are computed as:

$P_{micro}=\frac{\sum_{j=1}^{m}TP_{j}}{\sum_{j=1}^{m}TP_{j}+\sum_{j=1}^{m}FP_{j}}$
(2) $P_{macro}=\frac{\sum_{j=1}^{m}P_{j}}{m}$ (3)

$TP_{j}$ and $FP_{j}$ are the number of true positive and false positive
predictions for the $j$-th class, respectively. $P_{j}$ is the precision for
class $j$ and $m$ is the number of classes. Similarly, the micro- and macro-
average of Recall (R) and F1 score will be calculated and denoted as Rmicro,
Rmacro, F1micro, and F1macro.

As we are using stratified 5-fold cross validation, for each dataset $D$, we
will report averages of these metrics obtained from each of the k classifiers:

$F1\textsuperscript{avg}\textsubscript{micro}=1/k\sum_{n=1}^{k}F1\textsubscript{micro}\textsuperscript{(i)}$
(4)
$F1\textsuperscript{avg}\textsubscript{macro}=1/k\sum_{n=1}^{k}F1\textsubscript{macro}\textsuperscript{(i)}$
(5)

The $F1\textsubscript{micro}\textsuperscript{(i)}$ and
$F1\textsubscript{macro}\textsuperscript{(i)}$ refer to the test performance
of classifier $c\textsuperscript{(i)}$ on held out test set
$D\textsuperscript{(i)}$. Similarly, the Pavgmicro, Pavgmacro, Ravgmicro, and
Ravgmacro will be computed.

Following previous works how-far-ptm , if a model has higher values for both
F1avgmicro and F1avgmacro, we consider it to be better than other models.

Time: To compare the time efficiency, we will report the training time for
other approaches and fine-tuning time for PTMs, and the prediction time of all
the approaches. Prediction time is the time the model requires to process the
test data and predict labels. The time will be measured in seconds and will be
reported for each dataset. A model is considered to be more time efficient
than another if the prediction time is less; as training a model or fine-
tuning a PTM is a one-time process and need not be repeated.

Time Change in Percentage: We employ all four Prior approaches on all datasets
$D_{1}$–$D_{7}$ and choose their best as our baseline approach. In addition to
the Time mentioned above, we will also report the increase and decrease in
percentage for the studied PTMs with respect to the considered baseline for
the measured Time. The reason of reporting this change in percentage is that
time-duration depends on hardware configuration, and percentage change
provides a better understanding about the change in time for the readers.

##  4 Approach for Research Questions

The following steps will be taken to answer each research question.

###  4.1 Approach for Research Question 1

For RQ1, we identify the accuracy and efficiency of PTMs compared to Prior
approaches. For this RQ, all Prior approaches and PTMs will be trained and
tested on all of the datasets separately, as explained in section 3.6.  We
only consider the reviews (not tweets) from Dataset 2 for RQ1. For each
dataset and approach, all the evaluation metrics described in section 3.7 will
be reported. We highlight the best model for each dataset with highest
F1avgmicro and F1avgmacro. The lowest training/fine-tuning and prediction
times will also be highlighted for each dataset, along with the time change in
percentages. We denote the best performing PTM among others as PTM-X to pursue
experiments in other research questions. If no PTM achieves the best results
in terms of both micro- and macro- F1 score, the PTM that achieves the highest
F1avgmicro-score (denoted as PTM-XM1) and the PTM with the highest F1avgmacro-
score (denoted as PTM-XM2) will be selected to be studied in RQ2.

###  4.2 Approach for Research Question 2

In RQ2, we are interested to evaluate the performance of PTMs when they are
pre-trained on domain-specific corpus rather than non-domain-specific corpora.
In RQ2, we will use PTM-X (or PTM-X1 and PTM-X2) from RQ1 and all the seven
datasets. We can either pre-train PTM-X using just the domain-specific app
reviews or combine them with the general domain documents to pre-train PTM-X
from scratch. In the first approach, PTM-X with the same architecture will be
trained from scratch on app review sentences. But pre-training PTM-X on a
small number of domain-specific documents has its disadvantages; the model may
overfit our dataset and it can result in performance degradation in downstream
tasks. By pre-training PTMs using both general and domain-specific datasets,
we can avoid this problem. We use the approach of Wada et al.
wada2020medicalBERT , where we pre-train the PTM-X simultaneously with both
general and domain-specific documents. Following the literature
wada2020medicalBERT , we double the frequency of pre-training domain-specific
documents during the optimization on the Masked Language Modeling (MLM) task,
which is the task the ALBERT and RoBERTa models are trained on.

To accomplish this, we must increase the frequency of pre-training for MLM
over app reviews. The objective can be achieved by improving the vocabulary
representation of app reviews. Therefore, we use simultaneous pre-training
after the up-sampling introduced in wada2020medicalBERT . We create pre-
training instances from a set of corpora with different sizes to pre-train a
PTM, as described in Figure 3. Upsampling refers to the copying and pasting
the app reviews portion of the corpus until the predetermined ratio of app
reviews and general pre-training data is achieved; also, we have uniformly
mixed the book documents, wiki documents, and app reviews consecutively to
form the pre-training corpus. With this up-sampling technique, more instances
from the small app review corpus are used for MLM.

Figure 3: Creating Up-sampled dataset for PTM pre-training tasks

We will pre-train three different PTM-Xs, where we will respectively integrate
2.8 million, 5.6 million, and 10 million app reviews that we have collected
from Google Play, with the general domain documents from Wiki-texts777The
Wiki-texts dataset is open sourced at https://dumps.wikimedia.org/.. Our
collected dataset has app reviews for more than 2000 apps from different
categories. The dataset includes app_name, app_category, review, rating,
reply_text, and date. This will give us three versions of a domain-specific
pre-trained model which we refer to as Customized PTM-X (CPTM-X). We denote
the three sizes of the CPTM-X as CPTM-Xbase, CPTM-Xmedium, and CPTM-Xlarge,
respective to the number of the app review sentences integrated for pre-
training. Finally, we will fine-tune all the CPTM-Xs on the seven datasets
according to the steps and evaluation metrics explained in sections 3.6 and
3.7. The best CPTM-X will be used in RQ3.

###  4.3 Approach for Research Question 3

We have used both the readily available PTMs and our CPTM-X in different
settings for evaluating RQ3. We are considering four different settings to
determine the PTMs’ capacity regarding app review analysis.

####  4.3.1 Binary vs. Multi-Class Setting

Previous studies have shown that the pre-trained Transformer based models
deviate from their original performance when the classification task involves
multiple classes instead of binary classes adhikari2019docbert ; chang2019x .
Studies on app review classification also show that binary classification
yields better results Jha2019MiningNR ; maalejea . We are, therefore,
interested in evaluating the performance of the PTMs in both binary and multi-
class settings, especially, investigating the performance of CPTM-X. We will
investigate the accuracy and time efficiency of PTMs for binary classification
compared to the original multi-class settings. We convert all the seven
datasets into binary-class datasets by randomly choosing one class (i.e.
label) for each dataset and switching all the other labels in that dataset
into ”Others”. Then, we will use these modified datasets to regenerate the PTM
and CPTM results for the binary classification task on each dataset. The four
PTMs and CPTM-X will be used to record the evaluation metrics. All other steps
will remain similar to RQ1.

####  4.3.2 Zero-Shot Classification Setting

The Zero-Shot Learning (ZSL) that we will investigate in our study is the
recent approach in which we evaluate the models on fully-unseen labels. In
this setting, we assume that the system is never trained on any labeled data
for the task we are interested in, which is different from some ZSL settings
in which labels are partially seen by the model. The setting we choose here is
more realistic and can provide insights about the potential benefits of PTMs
in practice, where the developers might want to use the same model to derive
new unseen aspects of the dataset. Using PTMs for this kind of zero-shot
classification requires different training.

For ZSL, we will explore the method proposed by Yin et al. zero  in which
classification is considered as a Natural Language Inference (NLI) task. This
approach determines the compatibility of two distinct sequences by embedding
both sequences and labels into the same space. In NLI, a pair of sentences are
considered: “premise” and “hypothesis”; and the task is to predict whether the
“hypothesis” is an entailment of “premise” or not (contradiction). We follow
the steps in zero  to prepare the datasets and models and set up our study for
ZSL. We will use the reviews as “premise” and candidate labels as
“hypothesis”. For Example, for the app review “Please add a back button in the
armory page.” and its label “Feature Request”, we use the review as a premise
and the label as hypothesis. So, the NLI model can predict whether the
hypothesis is an entailment of the premise or not. This prediction will be
compared to the actual labels for the reviews to calculate the evaluation
metrics. If the NLI model correctly predicts the hypothesis as an entailment
of the premise, it is recorded as a True Positive. On the other hand, if the
NLI model predicts the hypothesis as contradiction or neutral to the premise,
it is recorded as a False Negative. If the NLI model predicts the hypothesis
as contradiction or neutral, where the hypothesis is actually an entailment,
we refer to this as False Positive (FP). On the other hand, if the NLI model
predicts the hypothesis as contradiction or neutral, where the hypothesis is
actually contradiction or neutral, it is regarded as True Negative (TN).

We fine-tune CPTM-X on NLI dataset888https://cims.nyu.edu/ sbowman/multinli/
and test it on the seven datasets to determine how well PTMs can classify
issues from app-reviews without being trained on them. Note that Multi-NLI
data does not contain app reviews.  In addition, we use RoBERTa-large-
nli999https://huggingface.co/joeddav/xlm-roberta-large-xnli, which is a
readily available fine-tuned model for zero-shot classification 101010We only
use the readily available PTMs for this study and avoid fine-tuning other PTMs
on Multi-NLI data in this setting, due to high computational expenses.. This
will provide insights about using a custom PTM trained on app reviews and the
PTM trained on general purpose corpora in a zero-shot setting.

####  4.3.3 Multi-Task Setting

Research has confirmed that when a new classification task (e.g. sentiment
classification) is introduced, a new model should be trained and the models
for another classification task (e.g. app issue classifiers) cannot be applied
for this new task stanik . In addition, the same model or classification
technique can have different performances on various analysis tasks. For
example, Hemmatian and Sohrabi found Naive Bayes (a probabilistic classifier)
to be performing better than Decision Tree (a non-probabilistic classifier)
for sentiment classification tasks hemmatian2019survey ; on the other hand,
Maalej et al. reported that using Bag-of-Word technique and Decision Tree
outperformed Naive Bayes for app issue classification task maalejea .
Therefore, we are interested in evaluating the performance of PTMs in both app
issue classification and sentiment analysis task settings. In sentiment
analysis, the task is to classify the sentiment of a given review into
positive, neutral, or negative polarities how-far-ptm , which is different
from the app issue classification. In this setting, we will evaluate the PTMs
and CPTM-X on the new task of sentiment classification for app reviews. We
fine-tune these models on the sentiment classification dataset and test them
for sentiment classification. This set up will help evaluating whether the
models trained for app issue classification can be applied for another task,
thus providing insights about the multi-task ability of the PTMs. For the
sentiment classification, we will use another
dataset111111https://www.kaggle.com/lava18/google-play-store-apps that
consists of 37,185 app reviews with three sentiment polarity labels: positive
(13,758), neutral (9,993), and negative (13,434).

We have gone beyond sentiment classification and implemented the PTMs for
category classification as well. The same Kaggle datasets also contains
category information for 20,955 app review. The number of user reviews in this
categories are: Family (18.9%), Games (50.3%), Tools (12.5%), Medical (10.7%),
and Business (7.6%). As it is an imbalanced dataset, we use stratified K-fold
cross validation for maintaining uniformity with other experiments. The
experimental settings and evaluation metrics we will use are similar to RQ1.
The results will determine how these models perform for a different
classification task in the same domain (app reviews).

####  4.3.4 Multiple Resources

The distribution of the data from multiple resources varies and models built
for App Stores are not suitable to classify user feedbacks from another
resource such as Twitter SUR-citer7 . In this setting, we will study the
accuracy and time efficiency of the PTMs for the classification of data
collected from multiple resources. For this experiment, Dataset 2 will be used
as it contains App Store reviews and Twitter data with the same labels. First,
we will fine-tune the PTM-X and CPTM-X on App Store reviews of Dataset 2 and
report the evaluation metrics (training steps are similar to RQ1) when tested
on Twitter data. In addition, we will fine-tune the PTM-X and CPTM-X on
Twitter part of Dataset 2 and report the results when tested on App Store
reviews. This way, we will gain insights about the PTMs’ capacity to classify
app reviews collected from different resources on which the PTMs were not
fine-tuned on. The evaluation metrics used here are explained in section 3.7.
To have a comparison, we also test the Prior approaches for this setting. From
the Prior approaches, AR-Miner does not require training. For the other
approaches, we train them on the App Store reviews of Dataset 2 and test them
on Tweets in Dataset 2.

##  5 Results

In this section, we present the results of our experiments for each of the
research questions. Note that we have computed all the scores (micro- and
macro- scores of P, R, F1) and the training and prediction times for all of
the models for each dataset. But due to the large number of experiments, we
bring all tables in the appendix. In the following, we only discuss the the
F1- scores in addition to the prediction time. The full results are shown in
Tables 3 and 4 for RQ1, Tables 5 and 6 for RQ2, Tables 7 and 8 for RQ3.1,
Table 9 for RQ3.2, Table 10 for RQ3.3, and Table 12 for RQ3.4 in the
appendices.

###  5.1 RQ1: Accuracy and Efficiency of PTMs Compared to Prior approaches

Figure 4: PTM and Prior Performance (micro-F1 score)

Fig 4 shows the micro-F1 scores for Prior approaches and PTMs, where the
performance of all models are shown for each dataset. Among the Prior
approaches, DL+WE is the best performing model, followed by Ensemble approach.
Only for $D_{5}$, which has the lowest number of app-reviews, the Ensemble
method has better scores than DL+WE. We conjecture that this low number of
reviews favored the Ensemble model to enhance the prediction performance of
participating classifiers. For the cumulative dataset (Dataset 7), DL+WE
improves the results of Ensemble Method, SUR-Miner and AR-Miner by $\sim$8%,
$\sim$20%, and $\sim$40%, respectively. So, we have selected DL+WE as our
baseline from Prior approaches.

Among the PTMs, on most datasets ALBERT and RoBERTa achieve the highest
F1-micro and F1-macro scores. The F1-micro results of these PTMs are higher
than the best Prior Approach and other PTMs for all datasets, except that BERT
outperforms other models for $D_{3}$. The Ensemble and DL+WE approaches can
match the performance of some worse performing PTMs in the respective Dataset,
such as DL+WE outperformed BERT on Dataset 2. The other Prior approaches
(i.e., AR-Miner and SUR-Miner) have significantly lower scores than the PTMs
for all Datasets. The performance of XLNET varies compared to Prior
approaches. For some datasets such as $D_{1}$, $D_{4}$, and $D_{5}$, it has
comparable or lower micro-F1 compared to Ensemble and DL+WE; but for $D_{7}$
it has higher scores. RoBERTa performs best on all metrics for Datasets 4 and
6, and yields best micro-average scores for $D_{5}$. These three datasets have
the least number of app reviews. On the other hand, ALBERT performs best on
Datasets $D_{1}$ and $D_{2}$, the two datasets with the highest number of app
reviews. For the cumulative dataset (Dataset 7), RoBERTa outperforms the other
PTMs on micro-scores, XLNET has the highest macro-scores, and the other PTMs
have relatively close results.

For $D_{3}$, the established order of PTM’s performance persists (RoBERTa
>ALBERT >XLNet), except for the performance yielded by BERT. BERT outperformed
the other three PTMs by a small margin for this dataset ($\sim$1.5% to
$\sim$4.5%). $D_{3}$ has 4,000 records with six labels, for which the ‘Other’
class has more than 2,000 records. In other words, the models learn about 5
classes from a small portion of reviews. When we further analyzed the
confusion matrices of ALBERT and RoBERTa for $D_{3}$, we find that the main
reason for dropping their performance in this dataset is that they predict the
records of the ’Other’ class wrong, but their prediction for the other five
classes are mostly correct. So, the False Negative predictions are higher and
this is also confirmed by having higher precision than recall for these models
in $D_{3}$. This is in contrast for $D_{1}$. Dataset 1 has a large number of
records (15,290) labeled as ’Other’ out of 34,000 records. The rest of this
dataset is labeled by 4 other classes. Therefore, in $D_{1}$, the number of
available reviews per label is much higher than the number of reviews for each
label in $D_{3}$, and the two PTMs perform better as they learn the features
for each class. That being said, still, ALBERT and RoBERTa have F1-micro
scores of over 85% and F1-macro of over 83% for $D_{1}$ and $D_{3}$. The score
of most models for $D_{6}$ is their lowest. This is also due to the specific
characteristics of this dataset, where it has the lowest number of records per
label.

As both RoBERTa and ALBERT have the highest or second highest scores in
maximum number of datasets, we choose these two PTM models as the best
performing ones to pursue with the next research questions. Here, these two
PTMs will be our PTM-X models and are chosen for training from scratch to
develop CPTM-X (Custom PTMs). We refer to these Custom PTMs as C-ALBERT and
C-RoBERTa.

Figure 5: PTM and Prior Prediction Time (seconds)

Fig 5 illustrates the prediction time for Prior and PTM approaches in seconds.
AR-Miner uses topic modeling to predict the labels/classes; so, the prediction
time is not reported for AR-Miner. From Fig 5, we can see that both Prior and
PTM approaches follow a similar trend, and their prediction times are related
to the size of the dataset, i.e., the larger the size of the dataset, the
longer the prediction time is.

Among the Prior approaches, our previously selected baseline DL+WE yielded the
least prediction time on 3 datasets ($D_{1}$, $D_{4}$, and $D_{7}$), whereas
SUR-Miner method’s prediction time is the least for rest of the datasets
($D_{2}$, $D_{3}$, $D_{5}$, and $D_{6}$). Among the PTMs, ALBERT consistently
yields the least prediction time for all the datasets. On the other hand, the
prediction time for XLNet is the highest for all datasets.

In the last column of Table 4 (Time-Diff (%)), we have included the percentage
change of the prediction times of PTMs with respect to the prediction time of
the Prior approach that has the lowest time. For example, for $D_{1}$, the
Time-Diff (%) for PTMs are calculated based on the prediction time of DL+WE;
and for $D_{2}$ it is calculated based on the prediction time of SUR-Miner.
For all datasets except $D_{7}$, the prediction time of PTMs compared to the
best Prior Approach increases by 0.49% to $\sim$500%. For $D_{7}$, BERT and
ALBERT predicted the classes $\sim 3\%$ and $\sim 22\%$, respectively, faster
than the fastest Prior approach (DL+WE). For $D_{7}$, RoBERTa increases the
time by 0.15% and XLNET increases the time by $\sim$89%.

Findings of RQ1 PTMs are generally superior to Prior approaches when
predicting classes with higher scores across all datasets. The two candidates
for the best PTMs we choose are ALBERT and RoBERTa, as they invariably perform
well across all dataset sizes. Based on the time taken for prediction, DL+WE
took the shortest amount of time among all Prior and PTM approaches, and
ALBERT has taken the shortest amount of time among the PTM approaches.

###  5.2 RQ2: Domain-Specific PTMs vs. General PTMs

Figure 6: Performance of Custom PTMs (micro F1 score)

Fig 6 shows the performance of C-ALBERT and C-RoBERTa after they are pre-
trained on domain-specific app-reviews along with general domain corpora. In
CPTM-X-Base, we have incorporated 2.8 million app reviews during pre-training.
For CPTM-X-Medium and CPTM-X-Large, we have increased the number of app
reviews during pre-training to 5.6 million and 10 million, respectively. From
the result, we can see that all the Custom PTMs performed better in all
datasets than their corresponding out-of-the-box PTMs. We also find that
increasing the number of app reviews in pre-training corpora helps boost the
performance of Custom PTMs.

ALBERT has lower F1-micro scores for datasets $D_{3}$ and $D_{4}$. After pre-
training with app reviews, we notice that all C-ALBERT models (i.e., C-ALBERT-
BASE, C-ALBERT-MEDIUM, and C-ALBERT-LARGE) have significantly better
predictions for both of these datasets ($\sim 5\%$ to $\sim 13\%$ improved
performance). Although this is not the case for $D_{6}$, all C-ALBERT models
still improve their performances. We relate this to the low number of reviews
per class in $D_{6}$. The Custom PTMs produce better predictions than their
readily available counterpart. C-RoBERTa models also have better performance,
yielding up to 15.2% increase in performance over RoBERTa. As seen in Fig 6,
C-RoBERTa improves RoBERTa’s F1-micro scores for datasets $D_{1}$–$D_{4}$,
where RoBERTa has scores of below 95. There is none to little fluctuation in
prediction time for all the Custom PTMs.

Findings of RQ2 Incorporating domain-specific data (i.e., app reviews) during
pre-training of the PTMs improves their performance for app review
classification, and slightly improves the prediction time of CPTMs compared to
PTMs. The performance of the models benefits from incorporating more domain-
specific data in the pre-training.

###  5.3 RQ3: Experimenting PTMs in Different Settings

####  5.3.1 RQ3-1: Binary vs. Multi-class Setting

We report the binary classification results in Tables 7 and 8 for RQ3-1. The
multi-class classification results of the PTMs and CPTMs are previously
reported in Table 3 to Table 6. The micro- and macro- scores were calculated
for accommodating the existence of multi-class in our datasets, which is no
longer the case after we converted each multi-class datasets into binary class
datasets.

The randomly selected classes for $D_{1}$ to $D_{7}$ are Bug-Report, Inquiry,
Reliability, Feature Requests, App Problem, Bug-Report, and Aspect Evaluation,
respectively. All the other classes in these datasets are labeled as Other or
Irrelevant.

From the results in Table 7 and Table 8 we observe that both Prior and PTM
approaches have performed better in binary classification compared to multi-
class classifications. The improvement for the Prior approaches ranges from 0
(for AR-Miner) to $\sim$3.2%. On the other hand, the improvement for the PTM
approaches ranges from 1 to $\sim$7.7%. All the PTM approaches yielded better
performance for binary classification than they did for multi-class
classifications and we observe that readily-available PTMs’ performance boost
is similar to the performance boost produced by the Custom PTMs.

####  5.3.2 RQ3-2: Zero-Shot Setting

We report Precision, Recall, and F1 score for zero-shot classification results
in Table 9. As in this setting, we are not fine-tuning our model on a portion
of our considered Datasets $D_{1}$ to $D_{7}$ (rather, we are fine-tuning our
PTMs on a separate NLI dataset), we no longer require cross validation over
multiple folds.

In this setting, we anticipate lower performance because our PTM models do not
have any exposure to any instances of the datasets to fine-tune on. In this
experiment, we use AR-Miner as the baseline to compare the results of this
setting for PTMs and CPTMs. AR-Miner is chosen as it is a topic modeling based
approach that does not require seeing the label beforehand. The other models
from Prior approaches are not considered here as all of them are supervised
machine learning or deep learning techniques that require training with the
labels beforehand. The C-RoBERTa-LARGE-NLI and C-ALBERT-LARGE-NLI are the
CPTMs that are pre-trained on 10 million app reviews (See RQ2). Overall, in
$D_{7}$, RoBERTa-LARGE-MNLI yielded best performance (71%). We observe that
RoBERTa-based models achieve higher scores compared to ALBERT-based PTMs.
Results of AR-Miner vary between 36% and 44% F1 score for the datasets, and
AR-Miner has closer results to ALBERT-NLI. For $D_{5}$, ALBERT-NLI yielded a
$\sim$37% F1 score, which is $\sim$5% worse than the performance of AR-Miner.
For Datasets $D_{1}$ to $D_{7}$, the PTMs yield on an average 45.875%,
44.425%, 43.125%, 43.225%, 46.45%, 38.8%, 42.725% less F1 score than their
counterparts in RQ1. All PTM approaches outperformed AR-Miner, the only prior
approach capable of zero-shot classification, except in $D_{5}$ for ALBERT-
NLI. The Custom PTMs in all cases improve the results by approximately 10 F1
scores in all datasets. Similar to previous RQs, the ALBERT-based model has a
lower prediction time.

####  5.3.3 RQ3-3: Multi-task Setting

In Tables 10 and 11, we report the micro and macro Precision, Recall, and F1
score and the training and prediction times for two classification tasks in
the app review domain: category classification and sentiment classification.

Among all the PTMs, C-ALBERT-LARGE yielded the best result for category
classification tasks with micro-average F1 scores of $\sim$91% and C-RoBERTa-
LARGE yielded the best result for sentiment classification tasks with micro-
average F1 scores of $\sim$93%, respectively.

BERT, ALBERT, RoBERTa, and XLNET performed 2.4%, 1.8%, 7.5%, and 4.3% worse
for category classification, respectively; they performed 1.1%, 0.1%, 5.5%,
and 0.9% worse for Sentiment classification task, respectively with respect to
their averaged micro-F1 scores over 7 datasets, in Table 4. C-ALBERT-BASE,
MEDIUM, and LARGE performed 4%, 2.7%, and 2.4% worse for category
classification, respectively, compared to their averaged micro-F1 scores over
7 datasets, in Table 4; they performed 2.8%, 3.4%, and 2.3% worse for the
sentiment classification task, respectively with respect to their F1 scores
averaged over 7 datasets, in Table 5. On the other hand, C-RoBERTa-BASE,
MEDIUM, and LARGE performed 4.1%, 5.9%, and 6.2% worse for category
classification, respectively; they performed 1.5%, 2.3%, and 3.3% worse for
the sentiment classification task, respectively with respect to their F1
scores averaged over 7 datasets, in Table 6. For both classification tasks, we
observe that the custom PTMs achieve higher scores compared to the PTMs,
showing that although the task is different (e.g., sentiment analysis vs. app
review classification), they still benefit from pre-training the models on the
same domain (app reviews).

####  5.3.4 RQ3-4: Multi-resource Setting

Dataset $D_{2}$ contains labeled app reviews from two different sources: App
Store and Twitter. In the previous results (Table 3 to 10), we have not
incorporated app reviews collected from Twitter. Here, we use the reviews from
both App Store and Twitter from $D_{2}$.

Training on App Store App-reviews and Testing on Twitter. In the first part of
this setting, we fine-tuned all the available approaches (except for AR-Miner)
on app reviews from App Store and tested them on app reviews collected from a
different resource, i.e., Twitter. The results are available in Table 12.
Being a topic modeling approach, AR-Miner does not require any supervised
training.

C-RoBERTa-LARGE achieves the best score among all the considered approaches
for this setting, in terms of F1 score (0.86). From the previous approaches,
DL+WE approach scored higher than the readily available PTMs, and all Custom
PTMs, except C-RoBERTa-LARGE. It also beats the C-ALBERT-LARGE by 1% F1 score.
The Ensemble method surpassed only readily available PTMs and closely matched
the performance of C-ALBERT-MEDIUM. AR-Miner and SUR-Miner have the lowest
scores among the Priors, and, we do not observe a big reduction in their
performance with respect to single-resource setting, in terms of F1 scores.
These approaches have scored ($\sim$5%) and ($\sim$2%) less in multi-resource
setting than their single-source counterparts.

Compared to the setting where PTMs are evaluated only on app reviews from App
Store, we observe $\sim$13% to $\sim$31% performance reduction for micro-F1
scores. This performance reduction is less for Custom-PTMs. Among the Custom
ALBERT models, the C-ALBERT-LARGE model produced the best micro-F1 score of
0.80 ($\sim$25% reduction) and C-ALBERT-BASE model produced the least micro-F1
score of 0.71 ($\sim$21% reduction). The C-RoBERTa group has less reduction
compared to C-ALBERT models, where the C-RoBERTa-LARGE produced the best
micro-F1 score of 0.86 ($\sim$15% reduction) and C-RoBERTa-BASE model produced
the least micro-F1 score of 0.77 ($\sim$15% reduction) among all C-RoBERTa
models.

Training on Twitter and Testing on App Store App-reviews. In the second part
of this setting, we trained/fine-tuned all the available approaches (except
for AR-Miner) on Twitter and tested them on app reviews collected from a
different resource, App Store. The results are available in Table 13.

C-RoBERTa-LARGE achieves the best score among all the considered approaches
for this setting, with an F1 score of 0.92. From the previous approaches, the
DL+WE approach scored higher than the readily available PTMs in this setting.
Here, it could not surpass the C-ALBERT-LARGE but beat the C-ALBERT-MEDIUM by
1% F1 score. The Ensemble method outperformed only readily available PTMs and
C-ALBERT-BASE from the Custom PTMs. AR-Miner and SUR-Miner have the lowest
scores among the Priors, which have scored ($\sim$10%) less and ($\sim$2%)
more in multi-resource setting than their single-source counterparts,
respectively. SUR-Miner beat the BERT in this setting by $\sim$4%.

Compared to the setting where PTMs are evaluated only on app reviews from App
Store, we observe $\sim$9% to $\sim$30% performance reduction for micro-F1
scores. This performance reduction is less for Custom-PTMs. Among the Custom
ALBERT models, the C-ALBERT-LARGE model produced the best micro-F1 score of
0.87 ($\sim$8% reduction), and C-ALBERT-BASE model produced the least micro-F1
score of 0.74 ($\sim$18% reduction). The C-RoBERTa group has less reduction
compared to C-ALBERT models, where the C-RoBERTa-LARGE produced the best
micro-F1 score of 0.92 ($\sim$5% reduction) and C-RoBERTa-BASE model produced
the least micro-F1 score of 0.78 ($\sim$14% reduction) among all C-RoBERTa
models. The smaller performance reduction for the models in this part,
compared to when models are trained on App Store data and tested on Twitter
data, could be related to the size of the training data. The dataset $D_{2}$
has a higher number of labeled tweets ($\sim$ 10K) than the app reviews from
App Store ($\sim$ 6.5K).

The prediction times of all Custom PTMs are lower than Prior approaches in
this setting.

Findings of RQ3 1. Both Prior and PTM models yield better performance for
binary classification tasks than they do for multi-class setting. 2. PTMs are
the best choice for zero-shot classification setting and Custom PTMs, though
having less score than the large RoBERTa-MNLI model, improve the results of
their non-domain specific models by approximately 10 F1 scores. 3. For multi-
task and multi-resource settings, the readily available PTMs and Custom PTMs
showed the same trend as demonstrated in RQ1 and RQ2, that is: Readily
Available PTM $<$ Custom-BASE $<$ Custom-MEDIUM $<$ Custom-LARGE. 4. RoBERTa-
based Custom PTMs have the best scores among all models. They also have lower
prediction times in the multi-resource setting compared to Prior approaches.

##  6 Discussions

###  6.1 Implications For Users

In the following, we discuss our findings and implications for the
users121212Here, users can be developers, practitioners, or researchers who
want to use the studied models to classify issues related to mobile apps from
user feedbacks..

Prior approaches: Deep learning with word embedding can be used for app issue
classification when one does not want to use PTMs. From the Prior approaches,
Deep Learning Technique leveraging Word Embedding (DL+WE) generated the best
results for predicting the correct classes of app reviews (RQ1), followed
closely by the Ensemble Method. DL+WE also took the least amount of time to
predict the classes, whereas SUR-Miner took the second least amount of time.
However, if no labels are available to train the mentioned models, AR-Miner
should be used, which utilizes topic modeling to separate the classes of app
reviews.

Prior approaches vs. PTMs: If the prediction time does not matter much but
high performance is required, PTMs should be used to classify app issues. From
the results of RQ1, we observed that all PTMs outperformed the Prior
approaches (except for 1 out of 28 cases) to predict classes accurately.
RoBERTa and ALBERT scored the highest F1 scores in most of the datasets among
D1 to D7. Compared with the fastest prior approach, ALBERT has the lowest, and
XLNET has the highest increase in prediction time. Consequently, it is worth
applying PTMs for app-review classification if the developer, researcher, or
software engineer can spare marginally additional time to achieve higher
prediction scores.

Prior approaches vs. PTMs: PTMs can be used for all settings when higher
performance is required, except for multi-resource setting, where the deep
learning model with word embedding has better scores. For the multi-class
classification (RQ1), PTMs perform significantly better than the Prior
approaches for most datasets. For Dataset $D_{2}$, the difference in the
scores among the two groups is not much, but when we consider $D_{7}$, the
results of PTMs are almost 10 F1 score higher than the best Prior Approach.
The prediction times also vary, but, some PTMs can predict the labels at
approximately the same time as the best Prior approaches (Ensemble and DL+WE).
For binary classification, the best Prior approaches achieve scores of below
90% and mostly in the low 80% F1 score. But, the F1 scores of PTMs are mainly
above 90% and the Custom PTMs have even higher scores. However, for multi-
resource setting (RQ3-4), the Ensemble and DL+WE have better F1 scores than
all PTMs, but not higher than CPTMs. This result is interesting, showing that
the learned knowledge about the app review domain for issue classification is
important in this setting, no matter if the model is trained on App Store data
and tested on Twitter data, or vice versa. This is confirmed by the obtained
results of the C-RoBERTa-LARGE, which have the highest scores among all models
in this setting.

Binary vs. Multi-class: When possible, the classification task should be
turned into a binary classification to achieve higher scores. All PTMs and
Prior approaches performed better in binary classification than in multiclass
classification. From Tables 7 and 8 , we observe that converting or trimming a
multi-class dataset to a binary class dataset help all the prior and PTM
approaches to yield better performance. Prior approaches are improved by 0 to
$\sim$3%, whereas the gain for the PTMs ranges from 1 to $\sim$7%. For binary
classification, all PTM approaches performed better than they did for multi-
class classifications, and the performance boost produced by readily-available
PTMs is proportionate to the performance boost produced by Custom PTMs.

Zero-shot: In a zero-shot setting, the Roberta-based Custom PTMs or RoBERTa-
LARGE-MNLI can be used to classify app issues. All PTMs can be implemented in
a Zero-Shot setting and yield adequate performance. The only Prior approach
that can be used directly in this setting is AR-Miner which is based on topic
modeling. All the readily available and Custom PTMs are fine-tuned on NLI
dataset for Inference classification task. Due to the fact that these PTM
models had not been exposed to any instances of the datasets in our study in
the zero-shot setting, they are expected to perform worse than their reported
performance in Tables 4, 5, and 6. The Custom-PTMs pre-trained with domain-
specific knowledge could not outperform the readily available RoBERTa-LARGE-
MNLI. This can be related to the dataset used for its training, which is a
much larger dataset than the NLI datasets we used to train our PTMs. Although
the CPTMs cannot beat the RoBERTa-LARGE-MNLI model, it is notable that these
CPTMs are smaller models and the RoBERTa-LARGE-MNLI model is a very large
language model; but still, the CPTMs we studied achieve significant
improvements compared to their general domain PTMs in this setting.

Custom PTMs vs. PTMs: Custom PTMs are the best models among all that should be
used to achieve better prediction scores with lower prediction times.
Customized PTMs that are pre-trained with the app review data have the best
performance in all settings: binary classification, multi-class
classification, multi-task, and multi-resource setting. Even for zero-shot
setting, they perform better than all the other models (except for RoBERTa-
LARGE-MNLI which is trained on a much larger NLI data). The benefit of using
Custom PTMs, specifically RoBERTa-based ones, compared to PTMs, is more
obvious for the multi-task and multi-resource setting in RQ3-3 and RQ3-4.
Interestingly, the PTMs do not perform well in the multi-task and multi-
resource setting, while CPTMs have the highest scores. The PTMs in these two
settings perform lower than the Ensemble and DL+WE models. But, the best CPTMs
achieve scores of above 90% for multi-task and 86% and 92% F1 score for multi-
resource. Other than achieving the best score, CPTMs have the lowest
prediction time among all models for multi-class classification, multi-task,
and multi-resource settings. They also achieve the lowest prediction time for
some of the datasets in binary classification. Our results for RQ3-4 confirm
that using over-the-shelf PTMs might not be the best option when classifying
issues for app reviews in all settings. PTMs can still have good performance
in binary and multi-class classification. But for harder scenarios (i.e.,
zero-shot, multi-task, and multi-resource), pre-training them with domain-
specific data increases their performance and reduces their prediction time.
Moreover, the more app review data is used in the pre-training of CPTMs, the
better their performances are.

###  6.2 Implications For Researchers

Training Custom PTMs: Research for incorporating app review data in the pre-
training of the current PTMs is needed. The inclusion of app reviews in the
pre-training corpus allows PTMs to classify user feedbacks with higher F1
scores. We have pre-trained three models of both ALBERT and RoBERTa with three
different sizes of corpora. This experiment found that more domain-specific
information during the pre-training helps the newly trained PTMs yield better
predictions of app reviews. C-ALBERT and C-RoBERTa models can improve the
ALBERT and RoBERTa scores for up to $\sim$13% and $\sim$14%, respectively.
Here, we followed the work of Wada et al. wada2020medicalBERT  to incorporate
app review data for pre-training of the PTMs. However, there are multiple
research on training the PTMs with the domain-specific data legal-bert ;
hakala2019biomedical . Other approaches can be investigated for domain-
specific models that achieve higher scores in the settings of RQ3.

Multi-class Classification and Zero-shot Settings: Research needed to build
models that perform better in multi-class and zero-shot setting. Our results
showed that the studied models perform better in a balanced binary
classification setting, rather than an imbalanced multi-class setting. This
suggests that more research is needed to develop models that perform better
when multiple labels are available and the dataset is imbalanced. Moreover,
researchers can work on a method to increase the performance of the models
when classifying with new labels that the model has not seen are required.

##  7 Threats to Validity

Internal Validity. A possible internal validity can be related to the obtained
results. To mitigate this threat, we use stratified k-fold cross-validation to
avoid the bias that might be introduced to the results by the test set. In
addition, to mitigate threats to the validity of the results, hyper-parameter
values are kept the same for all PTMs and Deep learning models in all the
fine-tuning steps. Also, we prevent overfitting by using early stopping and
higher dropout rate. We also keep the values of alpha and beta for LDA as
reported in AR-Miner. We run all the experiments on a single machine, and we
report the machine configuration to enforce the reproducibility of the
results. Furthermore, we consider the same metrics to compare the PTMs with
Prior approaches. Along with the generated raw duration, we also provide the
change of time in percentage as compared to the baseline. Our adopted stricter
micro and macro-metrics diminish the ambiguity while providing comparisons
among the considered approaches’ associated results.

Construct Validity. The selection of the Prior approaches and PTMs can pose a
validity threat to our study. We identified the four most common approaches as
priors by examining the highly practiced methods, tools, and techniques
employed by researchers and application developers. The considered approaches
include different Machine Learning algorithms, Ensemble methods, and Deep
Learning Approaches for app issue classification tasks and are selected after
conducting a literature review. The PTMs were adopted by following previous
study how-far-ptm  that conducted an empirical study on PTMs performance for
sentiment analysis in Software Engineering. Additionally, we pre-trained
transformer-based models by incorporating app-review-related datasets with the
previously used generic dataset to better understand the potential of domain-
specific PTMs (CPTM-X) for app issue classification. Another threat to the
study can be related to dataset $D_{7}$. This dataset is a merge of other
datasets and has labels borrowed from them. Although we consulted the
definitions of the labels from their publishers, there might be a chance that
the samples in one group of a dataset are closely related to the samples from
another label of another dataset, thus, affecting the results. We mitigate
this threat by combining labels from different datasets that have similar
names. We also used stratified k-fold cross validation to alleviate potential
threats.

External Validity. In this study, we empirically study the ability of the PTMs
in issue classification of app reviews. The app review classification mainly
focuses on extracting useful information in the context of software
engineering; which can be used by app developers for requirements engineering,
release planning, and other tasks as mentioned in the paper. Our study is
therefore limited to app review classification for software engineers and we
do not study PTMs for other purposes such as finding issues from business
perspectives tang2019systematicCommerce , product reviews
zhao2017productReview , and applications such as intention mining
huang2018intentionMining . We also do not study the summarization of relevant
issues from app reviews. However, we conduct our study on six different
datasets. In addition, by merging them, we run experiments on $D_{7}$ which
has multiple classes covering different aspects of apps useful for app
developers. Another threat in the generalizability of the results lies in
different tasks we considered. We mitigate this threat by studying the
sentiment classification and category classification of app reviews in the
multitask setting. Although PTMs might be useful for other tasks, we do not
study them in this work. To ensure the generalizability of our study in the
specified scope, we have incorporated datasets that include app reviews from
diverse app categories and from two platforms i.e., Google Play and Apple App
Store. The datasets have various sizes and different labels. In addition, we
experiment on issue classification from another platform, i.e. Twitter, which
has been shown to require different models than App Stores due to the
differences of the platforms, noise data, and user feedback.

##  8 Conclusion

We conducted an extensive exploratory study comparing app issue classification
tools and pre-trained Transformer-based models in various settings. We
conducted the experiments on six available datasets and a highly imbalanced
dataset, which is a combination of the six datasets. Domain-specific PTMs were
trained using different sizes of app review data we collected from Google Play
and these customized PTMs were also studied here. Our results confirm that
PTMs are achieving higher scores in binary and multi-class classification
compared to Prior approaches, but the over-the-shelf PTMs are not always the
best models to be used in all scenarios. Instead, CPTMs have the highest
scores and are able to perform better than other models in all settings.
Moreover, incorporating app specific data in the pre-training of PTMs reduces
the prediction time. One of the future directions of this research is
assessing domain-specific PTMs in other areas of app reviews and exploring
ways to increase performance in zero-shot setting.

## References

  * (1) Wei Liu, Ge Zhang, Jun Chen, Yuze Zou, and Wenchao Ding. 2015. A Measurement-based Study on Application Popularity in Android and iOS App Stores. In ¡i¿Proceedings of the 2015 Workshop on Mobile Big Data¡/i¿ (¡i¿Mobidata ’15¡/i¿). Association for Computing Machinery, New York, NY, USA, 13–18. DOI:https://doi.org/10.1145/2757384.2757392
  * (2) Islam, Dr. MD Rashedul and Mazumder, Tridib. (2010). Mobile application and its global impact. International Journal of Engineering and Technology. 10. 72-78.
  * (3) Anureet Kaur, Kulwant Kaur, Systematic literature review of mobile application development and testing effort estimation, Journal of King Saud University - Computer and Information Sciences, 2018, ISSN 1319-1578, https://doi.org/10.1016/j.jksuci.2018.11.002.
  * (4) Mengmeng Lu and Peng Liang. 2017. Automatic Classification of Non-Functional Requirements from Augmented App User Reviews. In Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering (EASE’17). Association for Computing Machinery, New York, NY, USA, 344–353. DOI:https://doi.org/10.1145/3084226.3084241
  * (5) A. Ciurumelea, A. Schaufelbühl, S. Panichella and H. C. Gall, ”Analyzing reviews and code of mobile apps for better release planning,” 2017 IEEE 24th International Conference on Software Analysis, Evolution and Reengineering (SANER), 2017, pp. 91-102, doi: 10.1109/SANER.2017.7884612.
  * (6) Al-Hawari, A., Najadat, H. and Shatnawi, R. Classification of application reviews into software maintenance tasks using data mining techniques. Software Qual J (2020). https://doi.org/10.1007/s11219-020-09529-8
  * (7) Zhou, Yu and Su, Yanqi and Chen, Taolue and Huang, Zhiqiu and Gall, Harald and Panichella, Sebastiano. (2019). User Review-Based Change File Localization for Mobile Applications.
  * (8) G. Grano, A. Ciurumelea, S. Panichella, F. Palomba and H. C. Gall, ”Exploring the integration of user feedback in automated testing of Android applications,” 2018 IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER), 2018, pp. 72-83, doi: 10.1109/SANER.2018.8330198.
  * (9) Ning Chen, Jialiu Lin, Steven C. H. Hoi, Xiaokui Xiao, and Boshen Zhang. 2014. AR-miner: mining informative reviews for developers from mobile app marketplace. In Proceedings of the 36th International Conference on Software Engineering (ICSE 2014). Association for Computing Machinery, New York, NY, USA, 767–778. DOI:https://doi.org/10.1145/2568225.2568263
  * (10) W. Maalej and H. Nabil, ”Bug report, feature request, or simply praise? On automatically classifying app reviews,” 2015 IEEE 23rd International Requirements Engineering Conference (RE), 2015, pp. 116-125, doi: 10.1109/RE.2015.7320414.
  * (11) Andrew R. Besmer and Jason Watson and M. Shane Banks, 2020. ”Investigating User Perceptions of Mobile App Privacy: An Analysis of User-Submitted App Reviews,” International Journal of Information Security and Privacy (IJISP), IGI Global, vol. 14(4), pages 74-91, October.
  * (12) Stanik, C., Haering, M., and Maalej, W. (2019, September). Classifying multilingual user feedback using traditional machine learning and deep learning. In 2019 IEEE 27th International Requirements Engineering Conference Workshops (REW) (pp. 220-226). IEEE.
  * (13) Roger Deocadez, Rachel Harrison, and Daniel Rodriguez. 2017. Preliminary Study on Applying Semi-Supervised Learning to App Store Analysis. In Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering (EASE’17). Association for Computing Machinery, New York, NY, USA, 320–323. DOI:https://doi.org/10.1145/3084226.3084285
  * (14) Dhinakaran, V. T., Pulle, R., Ajmeri, N., and Murukannaiah, P. K. (2018, August). App review analysis via active learning: reducing supervision effort without compromising classification accuracy. In 2018 IEEE 26th International Requirements Engineering Conference (RE) (pp. 170-181). IEEE.
  * (15) T. Zhang, B. Xu, F. Thung, S. A. Haryono, D. Lo and L. Jiang, ”Sentiment Analysis for Software Engineering: How Far Can Pre-trained Transformer Models Go?,” 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME), 2020, pp. 70-80, doi: 10.1109/ICSME46990.2020.00017.
  * (16) Devlin, J., Chang, M., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL-HLT.
  * (17) Wada, S., Takeda, T., Manabe, S., Konishi, S., Kamohara, J., and Matsumura, Y. (2020). A pre-training technique to localize medical BERT and enhance BioBERT. ArXiv, abs/2005.07202.
  * (18) Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. ArXiv, abs/1907.11692.
  * (19) Yin, W., Hay, J., and Roth, D. (2019). Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach. ArXiv, abs/1909.00161.
  * (20) E. Biswas, M. E. Karabulut, L. Pollock and K. Vijay-Shanker, ”Achieving Reliable Sentiment Analysis in the Software Engineering Domain using BERT,” 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME), 2020, pp. 162-173, doi: 10.1109/ICSME46990.2020.00025.
  * (21) Beltagy, I., Lo, K., and Cohan, A. (2019). SciBERT: A Pretrained Language Model for Scientific Text. EMNLP.
  * (22) Chalkidis, I., Fergadiotis, M., Malakasiotis, P., Aletras, N., and Androutsopoulos, I. (2020). LEGAL-BERT: The Muppets straight out of Law School. ArXiv, abs/2010.02559.
  * (23) N. Aslam, W. Y. Ramay, K. Xia and N. Sarwar, ”Convolutional Neural Network Based Classification of App Reviews,” in IEEE Access, vol. 8, pp. 185619-185628, 2020, doi: 10.1109/ACCESS.2020.3029634.
  * (24) Ruder, S., and Plank, B. (2017). Learning to select data for transfer learning with Bayesian Optimization. EMNLP.
  * (25) Mohammad Abdul Hadi and Fatemeh Hendijani Fard. 2020. ReviewViz: assisting developers perform empirical study on energy consumption related reviews for mobile applications. In ¡i¿Proceedings of the IEEE/ACM 7th International Conference on Mobile Software Engineering and Systems¡/i¿ (¡i¿MOBILESoft ’20¡/i¿). Association for Computing Machinery, New York, NY, USA, 27–30. DOI:https://doi.org/10.1145/3387905.3388605
  * (26) X. Gu and S. Kim, ”What Parts of Your Apps are Loved by Users?” (T),” 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE), 2015, pp. 760-770, doi: 10.1109/ASE.2015.57.
  * (27) E. Guzman, M. El-Haliby and B. Bruegge, ”Ensemble Methods for App Review Classification: An Approach for Software Evolution (N),” 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE), 2015, pp. 771-776, doi: 10.1109/ASE.2015.88.
  * (28) Joorabchi, M. E., Mesbah, A., and Kruchten, P. (2013, October). Real challenges in mobile app development. In 2013 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (pp. 15-24). IEEE.
  * (29) Qiu, X., Sun, T., Xu, Y., Shao, Y., Dai, N., and Huang, X. (2020). Pre-trained models for natural language processing: A survey. Science China Technological Sciences, 1-26.
  * (30) Svyatkovskiy, A., Deng, S. K., Fu, S., and Sundaresan, N. (2020, November). Intellicode compose: Code generation using transformer. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (pp. 1433-1443).
  * (31) C. Gao et al., ”Emerging App Issue Identification from User Feedback: Experience on WeChat,” 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), 2019, pp. 279-288, doi: 10.1109/ICSE-SEIP.2019.00040.
  * (32) Gao, C., Zeng, J., Lyu, M. R., and King, I. (2018, May). Online app review analysis for identifying emerging issues. In Proceedings of the 40th International Conference on Software Engineering (pp. 48-58).
  * (33) Hadi, M. A., and Fard, F. H. (2020, September). AOBTM: Adaptive Online Biterm Topic Modeling for Version Sensitive Short-texts Analysis. In 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME) (pp. 593-604). IEEE.
  * (34) Guo, H., and Singh, M. P. (2020, October). Caspar: extracting and synthesizing user stories of problems from app reviews. In 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE) (pp. 628-640). IEEE.
  * (35) Panichella, S., Di Sorbo, A., Guzman, E., Visaggio, C. A., Canfora, G., and Gall, H. C. (2015, September). How can i improve my app? classifying user reviews for software maintenance and evolution. In 2015 IEEE international conference on software maintenance and evolution (ICSME) (pp. 281-290). IEEE.
  * (36) Villarroel, L., Bavota, G., Russo, B., Oliveto, R., and Di Penta, M. (2016, May). Release planning of mobile apps based on user reviews. In 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE) (pp. 14-24). IEEE.
  * (37) Di Sorbo, A., Panichella, S., Alexandru, C. V., Shimagaki, J., Visaggio, C. A., Canfora, G., and Gall, H. C. (2016, November). What would users change in my app? summarizing app reviews for recommending software changes. In Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering (pp. 499-510).
  * (38) Bavota, G., Linares-Vasquez, M., Bernal-Cardenas, C. E., Di Penta, M., Oliveto, R., and Poshyvanyk, D. (2014). The impact of api change-and fault-proneness on the user ratings of android apps. IEEE Transactions on Software Engineering, 41(4), 384-407.
  * (39) Palomba, F., Linares-Vásquez, M., Bavota, G., Oliveto, R., Di Penta, M., Poshyvanyk, D., and De Lucia, A. (2015, September). User reviews matter! tracking crowdsourced reviews to support evolution of successful apps. In 2015 IEEE international conference on software maintenance and evolution (ICSME) (pp. 291-300). IEEE.
  * (40) Moran, K., Linares-Vásquez, M., Bernal-Cárdenas, C., Vendome, C., and Poshyvanyk, D. (2016, April). Automatically discovering, reporting and reproducing android application crashes. In 2016 IEEE international conference on software testing, verification and validation (icst) (pp. 33-44). IEEE.
  * (41) Finkelstein, A., Harman, M., Jia, Y., Martin, W., Sarro, F., and Zhang, Y. (2014). App store analysis: Mining app stores for relationships between customer, business and technical characteristics. RN, 14(10), 24.
  * (42) Johann, T., Stanik, C., and Maalej, W. (2017, September). Safe: A simple approach for feature extraction from app descriptions and app reviews. In 2017 IEEE 25th International Requirements Engineering Conference (RE) (pp. 21-30). IEEE.
  * (43) Moran, K., Linares-Vásquez, M., Bernal-Cárdenas, C., and Poshyvanyk, D. (2015, August). Auto-completing bug reports for android applications. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering (pp. 673-686).
  * (44) Sarro, F., Al-Subaihin, A. A., Harman, M., Jia, Y., Martin, W., and Zhang, Y. (2015, August). Feature lifecycles as they spread, migrate, remain, and die in app stores. In 2015 IEEE 23rd International Requirements Engineering Conference (RE) (pp. 76-85). IEEE.
  * (45) Yatani, K., Novati, M., Trusty, A., and Truong, K. N. (2011, May). Review spotlight: a user interface for summarizing user-generated reviews using adjective-noun word pairs. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 1541-1550).
  * (46) Guzman, E., and Maalej, W. (2014, August). How do users like this feature? a fine grained sentiment analysis of app reviews. In 2014 IEEE 22nd international requirements engineering conference (RE) (pp. 153-162). IEEE.
  * (47) Zhao, Y., Xu, X., and Wang, M. (2019). Predicting overall customer satisfaction: Big data evidence from hotel online textual reviews. International Journal of Hospitality Management, 76, 111-121.
  * (48) Palomba, F., Linares-Vásquez, M., Bavota, G., Oliveto, R., Di Penta, M., Poshyvanyk, D., and De Lucia, A. (2018). Crowdsourcing user reviews to support the evolution of mobile apps. Journal of Systems and Software, 137, 143-162.
  * (49) Di Sorbo, A., Panichella, S., Alexandru, C. V., Visaggio, C. A., and Canfora, G. (2017, May). SURF: summarizer of user reviews feedback. In 2017 IEEE/ACM 39th International Conference on Software Engineering Companion (ICSE-C) (pp. 55-58). IEEE.
  * (50) Palomba, F., Salza, P., Ciurumelea, A., Panichella, S., Gall, H., Ferrucci, F., and De Lucia, A. (2017, May). Recommending and localizing change requests for mobile apps based on user reviews. In 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE) (pp. 106-117). IEEE.
  * (51) Bakiu, E., and Guzman, E. (2017, September). Which feature is unusable? Detecting usability and user experience issues from user reviews. In 2017 IEEE 25th International Requirements Engineering Conference Workshops (REW) (pp. 182-187). IEEE.
  * (52) Martin, W., Sarro, F., Jia, Y., Zhang, Y., and Harman, M. (2016). A survey of app store analysis for software engineering. IEEE transactions on software engineering, 43(9), 817-847.
  * (53) Guzman, E., Alkadhi, R., and Seyff, N. (2016, September). A needle in a haystack: What do twitter users say about software?. In 2016 IEEE 24th international requirements engineering conference (RE) (pp. 96-105). IEEE.
  * (54) Guzman, E., Ibrahim, M., and Glinz, M. (2017, September). A little bird told me: Mining tweets for requirements and software evolution. In 2017 IEEE 25th International Requirements Engineering Conference (RE) (pp. 11-20). IEEE.
  * (55) Ali, M., Joorabchi, M. E., and Mesbah, A. (2017, May). Same app, different app stores: A comparative study. In 2017 IEEE/ACM 4th International Conference on Mobile Software Engineering and Systems (MOBILESoft) (pp. 79-90). IEEE.
  * (56) Scalabrino, S., Bavota, G., Russo, B., Di Penta, M., and Oliveto, R. (2017). Listening to the crowd for the release planning of mobile apps. IEEE Transactions on Software Engineering, 45(1), 68-86.
  * (57) Maalej, W., Kurtanović, Z., Nabil, H., and Stanik, C. (2016). On the automatic classification of app reviews. Requirements Engineering, 21(3), 311-331.
  * (58) Wang, C., Wang, T., Liang, P., Daneva, M., and van Sinderen, M. (2019, January). Augmenting App Review with App Changelogs: An Approach for App Review Classification. In SEKE (pp. 398-512).
  * (59) Wang, C., Zhang, F., Liang, P., Daneva, M., and van Sinderen, M. (2018, October). Can app changelogs improve requirements classification from app reviews? an exploratory study. In Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (pp. 1-4).
  * (60) Stade, M., Fotrousi, F., Seyff, N., and Albrecht, O. (2017, September). Feedback gathering from an industrial point of view. In 2017 IEEE 25th International Requirements Engineering Conference (RE) (pp. 71-79). IEEE.
  * (61) Al-Subaihin, A. A., Sarro, F., Black, S., Capra, L., and Harman, M. (2019). App store effects on software engineering practices. IEEE Transactions on Software Engineering, 47(2), 300-319.
  * (62) Prana, G. A. A., Treude, C., Thung, F., Atapattu, T., and Lo, D. (2019). Categorizing the content of github readme files. Empirical Software Engineering, 24(3), 1296-1327.
  * (63) Dabrowski, J., Letier, E., Perini, A., and Susi, A. (2019, March). Finding and analyzing app reviews related to specific features: A research preview. In International Working Conference on Requirements Engineering: Foundation for Software Quality (pp. 183-189). Springer, Cham.
  * (64) Singh, R. K., Sachan, M. K., and Patel, R. B. (2021). 360 degree view of cross-domain opinion classification: a survey. Artificial Intelligence Review, 54(2), 1385-1506.
  * (65) Stanik, C. (2020). Requirements Intelligence: On the Analysis of User Feedback (Doctoral dissertation, Staats-und Universitätsbibliothek Hamburg Carl von Ossietzky).
  * (66) Yang, X., Macdonald, C., and Ounis, I. (2018). Using word embeddings in twitter election classification. Information Retrieval Journal, 21(2), 183-207.
  * (67) Reimers, N., Schiller, B., Beck, T., Daxenberger, J., Stab, C., and Gurevych, I. (2019). Classification and clustering of arguments with contextualized word embeddings. arXiv preprint arXiv:1906.09821.
  * (68) Ren, Y., Zhang, Y., Zhang, M., and Ji, D. (2016, March). Improving twitter sentiment classification using topic-enriched multi-prototype word embeddings. In Thirtieth AAAI conference on artificial intelligence.
  * (69) Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R. R., and Le, Q. V. (2019). Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32.
  * (70) Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., and Soricut, R. (2019). Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942.
  * (71) Novielli, N., Girardi, D., and Lanubile, F. (2018, May). A benchmark study on sentiment analysis for software engineering research. In 2018 IEEE/ACM 15th International Conference on Mining Software Repositories (MSR) (pp. 364-375). IEEE.
  * (72) Forman, George, and Martin Scholz. ”Apples-to-apples in cross-validation studies: pitfalls in classifier performance measurement.” Acm Sigkdd Explorations Newsletter 12.1 (2010): 49-57.
  * (73) He, Haibo, and Yunqian Ma, eds. ”Imbalanced learning: foundations, algorithms, and applications.” (2013).
  * (74) James, Gareth, et al. An introduction to statistical learning. Vol. 112. New York: springer, 2013.
  * (75) Adhikari, Ashutosh, et al. ”Docbert: Bert for document classification.” arXiv preprint arXiv:1904.08398 (2019).
  * (76) Chang, Wei-Cheng, et al. ”X-bert: extreme multi-label text classification with using bidirectional encoder representations from transformers.” arXiv preprint arXiv:1905.02331 (2019).
  * (77) Jha, Nishant, and Anas Mahmoud. ”Mining non-functional requirements from app store reviews.” Empirical Software Engineering 24.6 (2019): 3659-3695.
  * (78) Hemmatian, Fatemeh, and Mohammad Karim Sohrabi. ”A survey on classification techniques for opinion mining and sentiment analysis.” Artificial Intelligence Review 52.3 (2019): 1495-1545.
  * (79) Guzman, Emitza, Rana Alkadhi, and Norbert Seyff. ”An exploratory study of Twitter messages about software applications.” Requirements Engineering 22.3 (2017): 387-412.
  * (80) Tang, Ailie KY. ”A systematic literature review and analysis on mobile apps in m-commerce: Implications for future research.” Electronic Commerce Research and Applications 37 (2019): 100885.
  * (81) Zhao, Wei, et al. ”Weakly-supervised deep embedding for product review sentiment analysis.” IEEE Transactions on Knowledge and Data Engineering 30.1 (2017): 185-197.
  * (82) Huang, Qiao, et al. ”Automating intention mining.” IEEE Transactions on Software Engineering 46.10 (2018): 1098-1119.
  * (83) M. A. Hadi and F. H. Fard, ”AOBTM: Adaptive Online Biterm Topic Modeling for Version Sensitive Short-texts Analysis,” 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME), 2020, pp. 593-604, doi: 10.1109/ICSME46990.2020.00062.
  * (84) T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, and J. Brew, “Huggingface’s transformers: State-of-the-art natural language processing,” ArXiv, vol. abs/1910.03771, 2019.
  * (85) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł. and Polosukhin, I., 2017. Attention is all you need. In Advances in neural information processing systems (pp. 5998-6008).
  * (86) Qiu, X., Sun, T., Xu, Y., Shao, Y., Dai, N. and Huang, X., 2020. Pre-trained models for natural language processing: A survey. Science China Technological Sciences, pp.1-26.
  * (87) Pennington, J., Socher, R. and Manning, C.D., 2014, October. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) (pp. 1532-1543).
  * (88) Cimasa, A., Corazza, A., Coviello, C. and Scanniello, G., 2019, August. Word embeddings for comment coherence. In 2019 45th Euromicro Conference on Software Engineering and Advanced Applications (SEAA) (pp. 244-251). IEEE.
  * (89) Sulistya, A., Prana, G.A.A., Sharma, A., Lo, D. and Treude, C., 2020. Sieve: Helping developers sift wheat from chaff via cross-platform analysis. Empirical Software Engineering, 25(1), pp.996-1030.
  * (90) Howard, J. and Ruder, S., 2018. Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146.
  * (91) Sarzynska-Wawer, J., Wawer, A., Pawlak, A., Szymanowska, J., Stefaniak, I., Jarkiewicz, M. and Okruszek, L., 2021. Detecting formal thought disorder by deep contextualized word representations. Psychiatry Research, 304, p.114135.
  * (92) Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q.V. and Salakhutdinov, R., 2019. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860.