itemize description enumerate

11institutetext: Wenhan Zhu 22institutetext: Michael W. Godfrey
33institutetext: Shane McIntosh 44institutetext: David R. Cheriton School of
Computer Science, University of waterloo, Waterloo, Canada
44email: {w65zhu, migod<EMAIL_ADDRESS>55institutetext:
Sebastian Proksch 66institutetext: Delft University of Technology, Delft,
Netherlands
66email<EMAIL_ADDRESS>77institutetext: Daniel M. German
88institutetext: Department of Computer Science, University of Victoria,
Victoria, Canada
88email<EMAIL_ADDRESS>99institutetext: Li Li 1010institutetext: School of
Software, Beihang University, Beijing, China
1010email<EMAIL_ADDRESS>
# What is an App Store?
The Software Engineering Perspective

Wenhan Zhu  Sebastian Proksch  Daniel M. German
Michael W. Godfrey  Li Li
Shane McIntosh

(Author pre-print copy. The final publication is available at Springer via:
https://doi.org/10.1007/s10664-023-10362-3)

###### Abstract

“App stores” are online software stores where end users may browse, purchase,
download, and install software applications. By far, the best known app stores
are associated with mobile platforms, such as Google Play for _Android_ and
Apple’s App Store for iOS. The ubiquity of smartphones has led to mobile app
stores becoming a touchstone experience of modern living. App stores have been
the subject of many empirical studies. However, most of this research has
concentrated on properties of the apps rather than the stores themselves.
Today, there is a rich diversity of app stores and these stores have largely
been overlooked by researchers: app stores exist on many distinctive
platforms, are aimed at different classes of users, and have different end-
goals beyond simply selling a standalone app to a smartphone user.

The goal of this paper is to survey and characterize the broader
dimensionality of app stores, and to explore how and why they influence
software development practices, such as system design and release management.
We begin by collecting a set of app store examples from web search queries. By
analyzing and curating the results, we derive a set of features common to app
stores. We then build a dimensional model of app stores based on these
features, and we fit each app store from our web search result set into this
model. Next, we performed unsupervised clustering to the app stores to find
their natural groupings. Our results suggest that app stores have become an
essential stakeholder in modern software development. They control the
distribution channel to end users and ensure that the applications are of
suitable quality; in turn, this leads to developers adhering to various store
guidelines when creating their applications. However, we found the app stores’
operational model could vary widely between stores, and this variability could
in turn affect the generalizability of existing understanding of app stores.

###### Keywords:

app store, software release, software distribution, empirical software
engineering

##  1 Introduction

The widespread proliferation of smartphones and other mobile devices in recent
years has in turn produced an immense demand for applications that run on
these platforms. In response, online “app stores” such as Google Play and
Apple’s App Store have emerged to facilitate the discovery, purchasing,
installation, and management of apps by users on their mobile devices. The
success of mobile app stores has enabled a new and more direct relationship
between app creators and users. The app store serves as a conduit between
software creators (often, developers) and their users, with some mediation
provided by the app store. The app store provides a “one-stop shopping”
experience for users, who can compare competing products and read reviews of
other users. The app store might also acts as a quality gatekeeper for the
platform, providing varying levels of guarantees about the apps, such as easy
installation and removal, expected functionality, and malware protection. To
the software creator, the app store provides a centralized marketplace for
their app, where potential users can find, purchase, and acquire the app
easily; the app store also relieves the developer from basic support problems
related to distribution and installation, since apps must be shown to install
easily during the required approval process. Indeed, one of the key side
effects of mobile app stores is that it has forced software developers to
streamline their release management practices and ensure hassle-free
deployment at the user’s end.

The success of mobile app stores has also led to the establishment of a
plethora of other kinds of app store, often for non-mobile platforms, serving
diverse kinds of user communities, offering different kinds of services, and
using a variety of monetization strategies. Many technical platforms now
operate in a _store-centric_ way: essential services and functionality are
provided by the platform while access to extensions/add-ons is offered only
through interaction with the app store. For instance, Google Play, the app
store, operates on top of the technical platform _Android_ , which provides
the runtime environment for the applications. When new technical platforms are
introduced, an app store is often expected to serve as a means to host and
deliver products to its users dixon2010home . Example technical platforms that
use app store-like approaches include Steam store.steam , GitHub Marketplace
store.github , the Chrome Web Store store.chrome , WordPress store.wordpress ,
AutoDesk store.autodesk , DockerHub store.dockerhub , Amazon Web Services
(AWS) store.aws , Homebrew store.homebrew , or Ubuntu Packages store.ubuntu .

For platforms that operate in this way, the app store is an essential part of
the platform’s design. For example, consider source code editors, such as
VSCode and IntelliJ. The tool itself — which we consider to be a technical
platform in this context — offers the essential functionality of a modern
source code editor; however, many additional services are available through
the associated app store that are not included by default. Thus, extensions
that allow for language-specific syntax highlighting or version control
integration must be added manually by the user through interaction with the
tool’s app store. We conjecture that the app store has fundamentally changed
how some classes of software systems are designed, from the overall ecosystem
architecture of the technical platform to the way in which add-ons are
engineered to fit within its instances.

In this work, we will explore the general space of app stores, and also
consider how app store-centric design can affect software development
practices. Previous research involving app stores has focused mainly on mobile
app stores, often concentrating on properties of the apps rather than
properties of the stores. For example, Harman et al. performed one of the
first major studies of app stores in 2012, focusing on the BlackBerry App
World harman2012app . However, concentrating the investigative scope so
narrowly may lead to claims that do not generalize well across the space of
all app stores. For example, Lin et al. found that reviews of games that
appeared in mobile app stores differed significantly from the reviews of the
same game that appeared within the Steam platform’s own app store
lin2019empirical . In our work, we aim to take a more holistic approach to
studying app stores by considering both mobile and non-mobile variants. In so
doing, we hope to create a more general model of app stores that fits this
broader space.

To achieve a holistic view, we start from the definition of an app store. A
precise definition of the term “app store” has been omitted in much of the
previous research in this area. Currently, Google Play and Apple’s App Store
dominate the market and are the main targets of research on app stores; in the
past, the BlackBerry App World and Microsoft’s Windows Phone Store were also
important players, but these stores are now defunct.111The Windows Phone Store
was absorbed into the broader Windows Store in 2015. Wikipedia recognizes
Electronic AppWrapper wiki.appwrapper  as the first true platform-specific
electronic marketplace for software applications, but the term became popular
when Apple introduced its App Store along with the iPhone 3G in 2008. Since
then, the term has largely come to refer to any centralized store for mobile
applications. We present our own working definition of the term “app store” in
Sec. 2.4.

The goal of this work is to survey and characterize the broader dimensionality
of app stores, and also to explore how and why they may feed back into
software development practices, such as release management. As a step toward
this goal, we focus on two research questions (RQs) that aim to explore the
space of app stores:

RQ1: _What fundamental features describe the space of app stores?_

To understand app stores, we first need a way to describe them. It would be
especially useful if this description framework would highlight the
similarities and differences of app stores. We start by collecting a set of
app store examples, and then extract from them a set of features that
illustrate important differences between them. We then expand this list of app
stores with search queries to derive a larger set of example stores. We
explicitly seek generalized web queries to broaden our search space beyond the
common two major mobile app stores of _Apple_ and _Google_. By combining the
web queries and the initial set of app stores, we selected a representative
set of app stores and extracted their features. In the end, we first surveyed
app stores and derived a feature-based model to describe them; we then
expanded the set of app stores through web queries; and finally, we extracted
features based on the model for a representative set of app stores.

RQ2: _Are there groups of stores that share similar features?_

Despite the ability to describe individual stores, it is also important to
understand the relationships between different stores. Having a understanding
of the natural groupings can help us gain insights into the understanding of
the generalizability of results gathered for different app stores. We perform
a _K-means_ macqueen1967some  clustering based on the extracted features of
the expanded set of app stores collected previously. The optimal $k$ value is
determined by the Silhouette method rousseeuw1987silhouettes . The clustering
results suggest that there are 8 groups in the expanded set of app stores. The
differences can be observed in the type of application offered, standalone or
extension, and/or type of operation, business or community-oriented.

In this study, we make several contributions towards a better understanding of
the app store ecosystem.

  * •

We identified a set of descriptive features that can be used to characterize
app stores.

  * •

We identified a set of 291 app stores and mapped 53 of them into the feature
space.

  * •

We identified 8 coherent groups of app stores based on the similarity of
features.

  * •

We discuss our insights on how the features and the diversity of app stores
can impact software engineering practices.

Overall, our study contributes towards a holistic view of app stores within
software engineering, which can form the basis for subsequent study of app
stores in general.

##  2 Background and Related Work

###  2.1 Early App Store Research

To date, research in this area has concentrated on a narrow set of app stores
that primarily involves mobile platforms. Harman et al. harman2012app
proposed app stores as a valid kind of software repository worthy of formal
study within the broader research area of mining software repositories; while
their work was not specific to mobile app stores, they used BlackBerry App
World as their canonical example. Ruiz et al. ruiz2012understanding  studied
the topic of reuse within app stores, focusing their work on Android
Marketplace.222Android Marketplace has since been re-branded as Google Play.
In both cases, these early works did not provide a formal definition of “app
store”, and tacitly used only app stores for mobile platforms in their
studies.

In their 2016 survey on app store research, Martin et al. martin2016survey
observed that studies have often focused on only a few specific app stores,
and have ignored comparisons between app stores. In a recent literature
survey, Dąbrowski et al. dkabrowski2022analysing  found the median number of
app stores studied to be 1, with the maximum being 3. We also note that
results from one app store study may not generalize to another store since the
two stores may differ in significant ways; for example, if a store does not
allow users to provide their own reviews of the apps within the store, app
creators will have to rely on other means to gain popularity and trust from
users, such as promotion outside of the app store. The same trend can be
observed in more specific app store topics such as app reviews; for example,
Lin et al. lin2019empirical  found that reviews of games within the Steam app
store can be dramatically different from reviews of the same game in mobile
app stores.

Existing work has yet to explore the full diversity of app stores,
concentrating on Google Play and Apple’s App Store, and largely ignoring those
such as Steam, AWS, and GitHub Marketplace that are not specific to mobile
platforms. With the heterogeneity of app stores and their typical uses, we
believe that the research in this area can be strengthened by expanding the
breadth to encompass a more diverse perspective on app stores; in turn, this
breadth can help to validate the generalizability of the study findings.

###  2.2 App Stores in Recent Software Engineering Research

To better understand the involvement of app stores in recent research, we
reviewed relevant recent papers from the two flagship software engineering
research conferences: the _ACM/IEEE International Conference on Software
Engineering_ (“ICSE”) and the _ACM SIGSOFT International Symposium on the
Foundations of Software Engineering_ (“FSE”) We used _Google Scholar_ to find
papers containing the keyword “app store” between January 2020 and April 2022
for the two conferences. We found a total of 34 such papers (listed in Table
1). After reading through all of them, we found that each paper fit into one
of two broad categories: mining software applications (20/34) and mining app
store artifacts (14/34). We note that our efforts do not constitute a
comprehensive literature survey; instead, our goal was to gain an overview of
how app stores are involved in recent research, and why app stores matter in
their context.

Table 1: Recent papers on app stores

Loc Paper Store Mining software applications ICSE ’21 Atvhunter: Reliable
version detection of third-party libraries for vulnerability identification in
android applications zhan2021atvhunter  Google Play ICSE ’20 How does
misconfiguration of analytic services compromise mobile privacy? zhang2020does
Google Play FSE ’21 Algebraic-datatype taint tracking, with applications to
understanding Android identifier leaks rahaman2021algebraic  Google Play FSE
’20 Code recommendation for exception handling nguyen2020code  Google Play FSE
’20 Static asynchronous component misuse detection for Android applications
pan2020static  F-Droid, Google Play, Wandoujia App Store ICSE ’21 Sustainable
Solving: Reducing The Memory Footprint of IFDS-Based Data Flow Analyses Using
Intelligent Garbage Collection arzt2021sustainable  Google Play ICSE ’22
DescribeCtx: Context-Aware Description Synthesis for Sensitive Behaviors in
Mobile Apps yao2022describectx  Google Play ICSE ’20 Time-travel testing of
android apps dong2020time  Google Play ICSE ’20 An empirical assessment of
security risks of global android banking apps chen2020empirical  Google Play,
APKMonk, and others ICSE ’21 Too Quiet in the Library: An Empirical Study of
Security Updates in Android Apps’ Native Code almanee2021too  Google Play ICSE
’20 Accessibility issues in android apps: state of affairs, sentiments, and
ways forward alshayban2020accessibility  Google Play ICSE ’21 Don’t do that!
hunting down visual design smells in complex uis against design guidelines
yang2021don  Android ICSE ’21 Identifying and characterizing silently-evolved
methods in the android API liu2021identifying  Google Play ICSE ’21 Layout and
image recognition driving cross-platform automated mobile testing yu2021layout
Apple’s App Store, Google Play FSE ’21 An empirical study of GUI widget
detection for industrial mobile games ye2021empirical  Android Games ICSE ’21
Fine with “1234”? An Analysis of SMS One-Time Password Randomness in Android
Apps ma2021fine  Google Play, Tencent Myapp ICSE ’21 IMGDroid: Detecting Image
Loading Defects in Android Applications song2021imgdroid  Android ICSE ’21
GUIGAN: Learning to Generate GUI Designs Using Generative Adversarial Networks
zhao2021guigan  Android ICSE ’20 Unblind your apps: Predicting natural-
language labels for mobile gui components by deep learning chen2020unblind
Google Play FSE ’21 Frontmatter: mining Android user interfaces at scale
kuznetsov2021frontmatter  Google Play Mining app store non-technical
attributes ICSE ’20 Schrödinger’s security: Opening the box on app developers’
security rationale van2020schrodinger  Apple’s App Store, Google Play ICSE ’20
Scalable statistical root cause analysis on app telemetry murali2021scalable
Facebook App ICSE ’21 An empirical assessment of global COVID-19 contact
tracing applications sun2021empirical  Android ICSE ’21 We’ll Fix It in Post:
What Do Bug Fixes in Video Game Update Notes Tell Us? truelove2021we  Steam
ICSE ’21 Automatically matching bug reports with related app reviews
haering2021automatically  Google Play ICSE ’21 Prioritize crowdsourced test
reports via deep screenshot understanding yu2021prioritize  Android ICSE ’21 A
first look at human values-violation in app reviews obie2021first  Google Play
ICSE ’21 Does culture matter? impact of individualism and uncertainty
avoidance on app reviews fischer2021does  Apple’s App Store ICSE ’21 COVID-19
vs social media apps: does privacy really matter? haggag2021covid  Google
Play, Apple’s App Store ICSE ’20 Society-oriented applications development:
Investigating users’ values from bangladeshi agriculture mobile applications
shams2020society  Google Play FSE ’21 Checking conformance of applications
against GUI policies zhang2021checking  Android ICSE ’21 Identifying key
features from app user reviews wu2021identifying  Apple’s App Store ICSE ’21
Champ: Characterizing undesired app behaviors from user comments based on
market policies hu2021champ  Google Play, Chinese android app stores ICSE ’20
Caspar: extracting and synthesizing user stories of problems from app reviews
guo2020caspar  Apple’s App Store

##### Mining software applications

App stores have been extensively used as a mining source of software
applications. In these papers, the major focus is often on another subject and
app stores provide a source where they can collect applications for either a
data source or verification dataset. For example, Zhan et al.
zhan2021atvhunter  proposed an approach to detect software vulnerabilities in
third-party libraries of _Android_ applications. They leveraged the app store
to collect a dataset to verify the effectiveness of their approach. In these
studies, the app store is both a convenient and practical source of data
collection. However, the involvement of app stores may not be necessary since
the purpose is to gather a dataset of application. In Yang et al.’s work
yang2021don , they leveraged _Android_ applications from an existing dataset
without the need to collect from an app store. We argue that the importance of
app stores in these types of studies is the selection criteria used by the
researchers to collect applications from app stores. These features can
include star ratings, total downloads, and app category.

##### Mining app store artifacts

In these studies, researchers focused on unique software artifacts that come
from the operation of the app stores. App stores have a much heavier
involvement in these studies compared to the previous group. App reviews is
the major software artifact the researchers focused on, where they leverage
the data to identify features of applications wu2021identifying , locating bug
reports haering2021automatically , and detect undesired app behaviors
hu2021champ . One interesting research practice we observed is where van der
Linden et al. van2020schrodinger  leveraged the developer contact information
shared on app stores to send out surveys related to security practices.

###  2.3 Store-Focused Research

As stated above, we found that most recent research involving app stores
focuses on the applications they offer rather than on studying the app stores
themselves; in particular, most research in the domain focuses on the
development of mobile applications. Meanwhile, a few papers have specifically
considered app stores and their effects on software engineering, but again
these works focus heavily on mobile app stores.

In a recent paper, Al-Subaihin et al. al-subaihin2021app  interviewed
developers about how app stores affect their software engineering tasks. They
found that developers often leverage the review section from similar
applications to help with understanding the expected user experience and
anticipated features. App stores also provides a kind of playground for
releasing beta version of apps to receive feedback from users. The built-in
communication channels also play a large role in informing development. The
interviews suggest that developers pay attention to viewing user requests in
app store via channels such as reviews and forums. The approval period of app
stores affects how developers plan their release. App stores introduce non-
technical challenges in the development process. Given the app store model of
release, app store-specific metrics, such as total number of downloads, are
considered highly important to developers.

Running an app store presents both technical and non-technical challenges to
the store owner. Technical challenges include verifying that each app will
install correctly, while non-technical challenges include ensuring that the
promotional information in the app’s product page adheres to store guidelines.
Wang et al. wang2018beyond  investigated several _Android_ app stores in China
and compared them to Google Play. Their study showed that these stores were
much less diligent in screening the apps they offered, with a significantly
higher presence of fake, cloned, and malicious apps than Google Play.

Jansen and Bloemendal surveyed the landscape of app stores from the
perspective of the business domain jansen2013defining . They selected 6 app
stores — 5 mobile stores and 1 _Windows_ store — at the time of publication
(2013), and investigated each store manually to find _features_ (i.e., those
actors can interact with) and _policies_ (i.e., rules, regulations and
governing processes that limit the functional reach of the features) from each
app store. The actors they define are the same as the three major stakeholders
of the app store model (i.e., the _store owner_ , _users_ , and _developers_).
Our study further contributes to the understanding of app stores. First, we
studied a significantly larger set of app stores: our methodology was focused
towards the identification of as many different types of stores as possible.
In total, we studied 53 stores in various domains including mobile, embedded
systems, computer games, application add-ons, and open source distributions
and packaging systems. Second, Jansen and Bloemendal studied app stores from
the perspective of a software business; for example, in their work they would
consider features and policies on whether users are able to generate affiliate
links to earn revenue through sharing applications. In contrast, our work
focuses on app stores in the perspective of their role in the software
engineering process.

In our study, we approach app stores from a broad landscape not limited to
mobile app stores. We focus on the similarity of features offered between
stores to understand their natural groupings and discuss the challenges in the
diversity of app stores.

###  2.4 Working Definition of an App Store

Previous researchers have often taken a casual approach to defining the term
“app store”, when a definition has been provided at all. For example, in their
survey paper, Martin et al. define an app store as “A collection of apps that
provides, for each app, at least one non-technical attribute”, with an app
defined as “An item of software that anyone with a suitable platform can
install without the need for technical expertise” martin2016survey . However,
we feel that this definition is too generous. For example, consider a static
website called Pat’s Apps that lists of a few of someone’s (Pat’s) favourite
applications together with their personalized ratings and reviews;
superficially, this would satisfy Martin et al.’s requirements as it is a
collection of apps together with Pat’s own reviews (which are non-technical
attributes). We feel that this kind of “store” is outside our scope of study
for several reasons: Pat’s software collection is not comprehensive, it is
unlikely that Pat provides any technical guarantees about quality of the apps,
and a passive list of apps on a web page does not constitute an automated
“store”.

Figure 1: Three major stakeholders of most app stores

Jansen and Bloemendal jansen2013defining  define app store as “An online
curated marketplace that allows developers to sell and distribute their
products to actors within one or more multi-sided software platform
ecosystems.” We note that this definition ignores that app stores are expected
to provide infrastructure for the deployment, installation, and maintenance of
the apps, which impacts the software development process. Their model also
ignores marketplaces that do not have payment mechanisms, such as the Google
Chrome Extensions store and the various open source apps stores, where all of
the software products may be free to download and install.

In our work, we seek to define an idea of app store beyond the well-known
mobile ones and with an emphasis on how their existence may affect the
software development cycle. Because we are focused on exploring the notion of
what app stores are, we formulate a working definition of the term; we did so
to provide clear inclusion/exclusion criteria for the candidate app stores
that we discover in Sec. 3.

Our working definition was influenced by considering the three major
stakeholders of the app store model: the _app creators_ who create and submit
applications to the store; the _app stores_ themselves, and the organizations
behind their operation who curate the app collection and coordinate both the
store and installation mechanisms; and the _end users_ who browse, download,
review, and update their applications through the app store (see Figure 1).

We thus arrived at the following working definition for _app store_ as an
online distribution mechanism that:

  1. 1.

offers access to a comprehensive collection of software or software-based
services (henceforth, “apps”) that augment an existing technical
infrastructure (i.e., the runtime environment),

  2. 2.

is curated, i.e., provides some level of guarantees about the apps, such as
ensuring basic functionality and freedom from malware, and

  3. 3.

provides an end-to-end automated “store” experience for end users, where

    1. (a)

the user can acquire the app directly through the store,

    2. (b)

users trigger store events, such as browsing, ordering, selecting options,
arranging payment, etc., and

    3. (c)

the installation process is coordinated automatically between the store and
the user’s own instance of the technical platform.

We can see that using this working definition, our Pat’s Apps example fails to
meet all three of our main criteria.

We note that our working definition above evolved during our investigations;
it represents our final group consensus on what is or is not an app store for
the purposes of doing the subsequent exploratory study. The steps by which the
representation is finalized are discussed in Sec. 3.1.2. For example, our
working definition implicitly includes _package managers_ such as the Debian-
Linux apt tool and Javascript’s NPM tool. It is true that package managers are
typically non-commercial, and so are “stores” only in a loose sense of the
term; furthermore, they usually lack a mechanism for easy user browsing of
apps and do not provide a facility for user reviews. However, at the same
time, they are a good fit conceptually: they tend to be comprehensive,
curated, and offer an automated user experience for selection and
installation. Furthermore, some package managers serve as the backend to a
more traditional store-like experience; for example, the Ubuntu Software
Center builds on a tool _aptitude_ , which interacts with software
repositories to provide a user experience similar to that of Google Play.

##  3 Research Methodology

Figure 2: Methodology overview: There are three main _stages_ , further broken
down into six _steps_.

To investigate the research questions, we designed a three-stage methodology
that is illustrated in Figure 2. The goal of the first two stages is to answer
RQ1, while the third stage addresses RQ2.

In the first stage (Step \footnotesize1⃝ and \footnotesize2⃝) we identified
our initial list of features using a small set of well-known app stores
(Apple’s App Store, Google Play, Steam etc.) In the second stage (Steps
\footnotesize3⃝, \footnotesize4⃝, and \footnotesize5⃝) we methodically
expanded our list to a conceptually wider ranging set of 53 app stores. We
then described these stores using the features identified in the first stage.
A major goal of this stage was to evaluate whether the set of available
features was sufficient to describe the characteristics of all these stores.
This set of features forms the answer to RQ1.

In the third stage (Step \footnotesize6⃝), we took advantage of the labeling
of the 53 stores. We used _K-means_ clustering analysis to identify groups of
stores that shared similar features. These groupings form the answer to RQ2.

We now describe our methodology in more detail.

###  3.1 Extracting Features Describing App Stores

Our basic assumption is that an app store can be categorized based on a finite
set of features. The features would correspond to traits of the app store
where they describe the distinguishing qualities or functional characteristics
of the app store. We encode these features as binary values, i.e., each store
_has_ or _does not have_ a given feature.

In order to identify such features, we first created a seeding set of
representative app stores. We started by enumerating well-known app stores
that we were aware of (Step \footnotesize1⃝). Once this set of representative
app stores was created, we used an iterative process to identify the features
that we felt best characterized these stores (Step \footnotesize2⃝). We then
used these features to describe each store.

#### 3.1.1 Stage 1: Identifying Features

First, each of the six authors was tasked with identifying representative
characteristics of five stores and the possible features for each. Each author
worked alone in this step; however, to seek better reliability as well as
encourage diverse opinions, each store was assigned to two authors. We list
the 15 stores that were assigned in this step with a short description in
Table 2. After that, all of the authors met as a group to discuss their
findings and further refine the proposed feature set.

In the subsequent iterations, the authors worked in pairs, and the pairings
were reassigned after each iteration (Step \footnotesize2⃝). In these
iterations, each author-pair was assigned a set of 2–3 app stores and was
asked to describe them using the current set of features; a key concern was to
evaluate whether the existing features were sufficient or needed refinement.
For each store, each author-pair analyzed both its store-front and its
documentation; in some cases, we could navigate the store as users but not as
developers, in these cases, we relied on the store’s supporting documentation.

After this step, the six authors discussed their findings as a group and
updated the set of features. The features were discussed in detail to ensure
that they were conceptually independent from each other. We also made sure
that each feature applied to at least one store to ensure that it was
relevant.

Our process leveraged ideas from the coding process of _Grounded theory_
walker2006grounded  to extract the features of app stores, and followed the
practice of open card sorting coxon1999sorting  to create the categorized
feature set. Similar to prior workadolph2011using ; hoda2012developing ;
masood2020agile , we followed practices of _Grounded theory_ ’s coding process
to extract the features— where we consider codes as a specific feature of app
store operation — and stopped when we reached saturation with no new features
added after a new round of describing app stores. Similar to prior
workvassallo2020developers ; chen2021maintenance ; wang2022demystifying , we
applied card sorting to the collected features so inter-related features are
grouped together. The authors formed a group in this process and discussed how
different features belong to the same conceptual group and stopped when
consensus was reached.

Table 2: Investigated stores for feature extraction

Store Description Google Play Store Google’s app store for Android Apple App
Store Store for Apple devices Samsung GalaxyApps Store specifically for
Samsung devices GitHub Marketplace Providing applications and services to
integrate with GitHub platform Atlassian Marketplace Providing applications
and services to integrate with various Atlassian products Homebrew Package
manager for MacOS MacPorts A package manager for MacOS Ubuntu Packages
Software repository for the Ubuntu Linux distribution, with a official front
end Ubuntu Software Center Steam Gaming focused app store running on multiple
operating systems (e.g., Windows, Linux) Nintendo EShop Provides applications
for Nintendo devices (e.g., Nintendo Switch, Nintendo 3DS) GoG Gaming focused
store focusing on providing DRM free games JetBrains Plugin Store Provides
plugins to enhance the behavior of JetBrains IDEs VSCode Marketplace Provides
plugins to enhance the editor Chrome Web Store Provides extensions to enhance
Chromium based web browsers AWS Marketplace Provides servers and cloud
services

#### 3.1.2 Stage 2: Expanding Our Set of App Stores and Further Evaluation and
Refinement the Features

Once we had agreed on the features, our next goal was to verify that these
features were capable of describing other app stores that were not part of the
initial seed, or if features were missing or needed refinement. We used a
common search engine, _Google_ , to expand our set of app stores in a
methodical manner (Step \footnotesize3⃝). To achieve the goal of including a
broad range of yet undiscovered app stores, we first derived general search
terms by combining synonyms for "app" and "store". More specifically, we have
built all possible combinations of the following terms to construct our search
queries:

First half

software, (extension -hair -lash), (addon OR add-on),
solution, plugin OR plug-in, install, app, package

Second half

repository, shop, ("app store" OR store), ("market place" OR marketplace),
manager

For example, a concrete query was created by combining app and ("app store" OR
store). For some queries, it was necessary to refine the term to avoid noise
in the results; for example, searching for the term extension would mainly
return results related to hair product or eye lashes. In total, with 8
synonyms for app and 5 synonyms for store we were able to create 40 unique
_Google_ search queries. We felt confident that these search terms were
representative when we found that the initial seed list had been exhaustively
covered.

Our _Google_ search was performed in November 2020. We queried and stored the
search results for each search query. Two authors classified each result as to
whether or not it corresponded to an app store. We devised two inclusion
criteria for this decision: 1) the store in question should offer software or
software-based services, and 2) the store in question should offer an end-to-
end experience for users (ordering, delivery, installation). We considered
only direct hits to the store (e.g., product page), and we explicitly excluded
results that contain only indirect references to a store, such as blog posts,
videos, or news. Any disagreements were resolved through discussion. However,
despite our initial effort of maintaining a clear set of inclusion criteria
for app stores, several corner cases became apparent during the labeling
process. The first two authors discussed these cases as they arose, and
continually updated the inclusion criteria throughout the labeling process. In
a few special cases no agreement could be reached, so another author acted as
a moderator and resolved the disagreement by a majority vote. Over time, the
inclusion criteria and features evolved and eventually reached a stable state
(in Step \footnotesize3⃝). Our final state of the inclusion/exclusion criteria
is presented as the working definition for app stores defined in Sec. 2.4.

The classification of search results was stopped when a new results page did
not contain any new links to app stores, or once all 10 retrieved pages were
analyzed. Initially, 586 URLs were examined by the first two authors until a
saturation of agreement was reached (90.7% agreement rate). The first author
continued to label the rest. In the end, a total of 1,600 URLs were labeled.
Multiple search results can refer to the same store; these duplicates were
detected and eliminated by using the root domain of the URL. The most common
duplicate references were found for the domains _google.com_ (61), _apple.com_
(22), and _microsoft.com_ (18). In the end, we found 291 stores. We note that
the exact number of unique stores may differ since two root domains can point
to the same store, _kodi.tv_ and _kodi.wiki_ , or the same root domain may
contain multiple stores, _chrome.google.com_ and _play.google.com_.

In the next step (Step \footnotesize5⃝), we constructed and labeled a set of
app stores based on our identified features from Step \footnotesize2⃝. We
began from the URLs labeled in the last step and selected the first three
occurring stores for each search term; this resulted in 104 URLs pointing to
48 unique stores. Two of the stores were could not be accessed by the authors:
ASRock App Shop requires physical hardware to use it, and PLCnext Store’s
website was unresponsive at the time of labeling. These stores were removed
from the list. In addition, we discussed several more stores that we felt
deserved explicit investigation: AWS, Flatpak, GoG, MacPorts, Nintendo eShop,
Steam, and Samsung’s Galaxy Store. These are the stores that the authors
investigated in Step \footnotesize2⃝ but did not show up in the first three
occurring results from the search terms. Meanwhile, the added stores all show
up in the list of 291 stores identified by all labeled URLs.

We thus selected and labeled a total of 53 app stores. This sample is non-
exhaustive, but we believe that our wide range of search queries has created a
representative sample of the population of app stores that enables our
experiments.

The first two authors proceeded to describe 12 app stores, selected as the
first from each search query, using the set of features. This was done to make
sure there was consistency in the interpretation and use of each feature.
After that, the first author labeled the remaining stores.

To check the applicability of our dimensions and the labeling guidelines, we
have measured the inter-rater agreement between two authors on the 12 stores.
We used the Cohen’s Kappa cohen1960coefficient  as a measurement for our
inter-rater agreement. The Cohen’s Kappa is widely used in software
engineering research perez2020systematic . We have reached an agreement of
86.3% with Cohen’s Kappa cohen1960coefficient  of 0.711). Our agreement based
on the Cohen’s Kappa is considered as a substantial lantz1996behavior  inter-
rater agreement suggesting a high confidence of agreement between the two
raters.

The outcomes of RQ1 were a list of features that describe the main
characteristics of app stores grouped by dimensions, and a set of 53 App
Stores, each labeled using these features.

###  3.2 Finding Natural Groupings of App Stores

With the outcomes of RQ1, we next performed a _K-means_ clustering analysis to
identify groups of similar stores. _K-means_ is a well known clustering
algorithm widely used in software engineering research pickerill2020phantom ;
khatibi2014flexible ; al2019empirical ; kuchta2018correctness . It groups
vectorized data points iteratively until $k$ centroids are formed. We used the
_K-means++_ implementation arthur2006k  to conduct the clustering process.

#### 3.2.1 Stage 3: Cluster Analysis

To identify related app stores, we decided to cluster them using the _K-means_
algorithm (Step \footnotesize6⃝).

To prepare our labels for the _K-means_ clustering process, we converted each
label of the feature to a binary value: 1 if the store has the feature, and 0
if it does not. Having binary-encoded data ensured that we do not suffer from
having categorical values that do not make sense in the scope of _K-means_.
However, performing _K-means_ on binary data can also be problematic, since
the initial centroids selected will be binary. To mitigate this issue, we
applied Principal Component Analysis (PCA) wold1987principal  to both reduce
the dimensional space and to produce a mapping in the continuous range. We
kept all principal components that explained a variance of at least 0.05.
Finally, we used the Silhouette method rousseeuw1987silhouettes  to determine
the best number of clusters within a range of 1 to 20. To identify the
features that best characterize each cluster, we have calculated the deviation
of each cluster _centroid_ (i.e, the _center_ of the cluster) from the
_centroid-of-centroids_ (_C_) over all clusters.

As an unsupervised method, the result of _K-means_ provides only the
clustering result with the stores in each cluster. We then further discussed
the results of the _K-means_ process and categorized the clusters by the
properties of the contained stores. Following our discussion and
categorization, we assigned groupings and names to each of the clusters.

##  4 Results

In this section, we present the results of our investigations into each of the
research questions. The results are organized based on the three stages
discussed in Sec. 3.

###  RQ1: _What fundamental features describe the space of app stores?_

#### Stage 1: Features characterizing app stores

As discussed in Sec. 3.1, we derived a set of features and organizational
categories that describe the set of studied app stores; the results of these
efforts are summarized in Table 3. We have modelled the features as a binary
representation; thus, each store either has or does not have this feature. We
note that for some categories, the features are mutually exclusive; for
example, in the category _Rights Management_ , a store can have either
_Creator managed DRM_ or _Store-enforced DRM_ , but not both. In other
categories, an app store may have several of the features within a given
category; for example, there may be several kinds of communication channels
between users, app creators, and the store owner for a given app store. We now
describe each high-level category in detail.

Table 3: Features for describing app stores

Feature Description Monetization The type of payment options directly offered
by the app store. Free Free as in in the product can be directly acquired One-
time payment A single payment needed for the product Seat-based subscription
The subscription is based on the number of products provided Time-based
subscription A payment is needed by a set time interval (e.g.,, monthy,
yearly) Resourced-based subscription A payment is needed by the amount of
resource used (e.g., API calls, CPU time) Micro-transaction Additional payment
can be collected based on additional feature offered in a product Custom
pricing (i.e., “Contact us for price”) The actual price is based on a per case
situation; this happens mostly in business-focused app stores Rights
Management* How does the store take care of DRM on the product provided.
Creator-managed DRM No DRM is offered by the store and is taken care of by the
creator Store-enforced DRM Store wide DRM for every product offered in the
store Do I need an account?* Whether it is possible to use the app store
without registration. Account required An account is required to use the store
No registration possible The store does not have an account system Some
features requires registration Some content of the store is locked behind an
account, but the store can be used without one. Product type The type of
product the store offers. Standalone apps The product operates by itself
Extension/add-ons to apps/hardware The product acts as a feature extension to
another application/hardware Service/resources The software product is a
service Package/library The product is not an end-user product, but offers
functionality to other products Target audience* The intended users of the app
store. General purpose The app store is intended to be used by everyone.
Domain-specific The app store have a specific focus and is very unlikely to be
used by a normal person Type of product creators The type of creators who
submits products to the app store. Business The creators mostly have a
commercial or business focus Community The creators are from the community
(e.g., open source developers) Intent of app store The reason why the app
store exists from the app stores’ perspective. Community building/support The
app store aims to serve a technical community Profit The app store aims to
earn money Centralization of product delivery The app store aims to provide a
way for customer to gather apps in a centralized way Expanding a platform
popularity/usefulness The app store aims to extend functionality from the
platform it is based on Role of intermediary The role app store play between
the creator of products and the customer of the app store. Embedded
advertisement API Provides an advertisement method for creators to take
advantage of CI/CD Offers continuous integration/continuous deployment for
creators Checks at run time Provide checks when apps installed from the app
store is ran Checks before making available to the customer Provide checks
when an app is submitted to the app store for quality reasons Composability*
The relationship between products provided in the app store. Independent The
products in the app store are unrelated to each other Vendor internal add-
on/extension/unlock Some products can be based on other products from the same
creator (e.g., game DLC, app feature packs) Package manager type of app
relationship A dependency relationship exists between products in the app
store Analytics The type of analytical data provided by the app store.
Sentiment and popularity ratings Information related to the popularity of a
product (e.g., downloads, score ratings) Marketing feedback Information
related to marketing for the creator (e.g., sales, conversion, retention)
Product usage data Information related to the usage of the product. (e.g.,
logging, user profiling) Communication channels The methods where different
parties of the app store can communicate with each other. Documentation
Information related to the operation of the store (e.g., instructions to
install applications) Product homepage A homepage for a specific product in
the app store Ratings Any form of rating customers can give to a product
(e.g., star, score, up/down vote) Written reviews (in text) A written viewer
where customers can write their experience to the product. Community forum A
forum like feature offered by the store where people can discuss things
related to the store/product. Support ticket A system where customers can
inquiry for support questions related to the product offered by the store.
Promotion/marketing The store offers a way to provide promotional/marketing
feature to the products in the app store (e.g., featured apps, top downloads
of the month).

_*:_ Categorical values are mutually exclusive; one and only one categorical
value in the dimension can apply to a given store.

##### Monetization

describes what, if any, payment options are provided to the user directly by
the store. If a product is offered free within the store, but requires an
activation key obtained elsewhere, we consider that the product is free. While
most of the options are self-explanatory, some may be less obvious. For
example, GitHub Marketplace offers _seat-based subscriptions_ where app
pricing is calculated by the number of installations made to individual
machines; usually, this occurs within the context of enterprise purchase.
Also, AWS offers _resource-based subscription_ where the price charged is
determined by the amount of resources — such as cloud storage and CPU time —
that are used during the execution of the service.

##### Rights Management

describes the Digital Rights Management (DRM) policy of the store; the values
describe whether the store uses a store-wide DRM feature. For example, for
Steam, all games have DRM encryption, whereas the F-Droid store contains only
open source apps, so there is no need for DRM.

##### Do I need an account?

describes whether a user can access and use the store without being registered
with the app store. We find that most stores are either _account required_
(e.g., Apple’s App Store) or _no registration possible_ (e.g., Snapcraft).
However, we also found that some stores can be used without an account for
some purposes, with other features requiring explicit registration; for
example, the Microsoft Store allows users to download free applications
without an account, but to purchase an app or leave a review, an account is
required.

##### Product type

describes the kinds of applications that are offered by the store. For
example, Google Play and Steam focus on _standalone apps_ , the VSCode
Marketplace store offers _add-ons_ to an existing tool, and AWS allows users
to “rent” web-based _resources and services_.

##### Target audience

describes the intended user base of the store. _General-purpose_ stores offer
products aimed at the broad general public of everyday technology users; this
includes stores such as Google Play, Steam, and the Chrome Web Store. _Domain-
specific_ stores, on the other hand, have a dedicated focus on a specialized
field; for example, Adobe Magento focuses on building e-commerce platforms.

##### Type of product creators

describes the typical focus of creators submitting applications to the store.
We distinguish between two groups of creators: those with a commercial or
business focus, and those with community focus such as open source developers.

##### Intent of app store

describes the perceived high-level goals of the app store. The values are
derived from the app stores’ own descriptions of their goals, often found in
_“About us”_ web pages. For example, both F-Droid and ApkPure are _Android_
app stores; however, F-Droid’s focus is to provide a location to download and
support FOSS software, while ApkPure’s goal is to provide a location for users
to be able to download _Android_ apps when Google Play may be unavailable.

##### Role of intermediary

describes the roles that the app store plays in mediating between the users
and creators; these are software engineering-related services that are mostly
independent of each other. For example, _checks at run time_ tracks if the app
store ensures that its products function correctly (e.g., Steam tracking game
stats). Also, _CI/CD_ indicates that the app store provides explicit support
for continuous integration and deployment of the apps, which may be linked to
specific development tools used by the creator.

##### Composability

describes the relationship between products offered by the store. App stores
of _independent_ composability offer products that have no relationship with
each other, such as Firefox Add-ons. _Vendor internal add-on/extension/unlock_
means that the products within the app store can be based on each other, but
only when they are from the same vendor, such as game DLC and micro-
transaction unlocks. _Package managers_ contain apps that can have complicated
dependency relationships regardless of the creator of the products, such as
the _Ubuntu_ package management tool apt.

##### Analytics

describes what kind of diagnostic information is provided by the store. We
distinguish between three kinds: _Sentiment and popularity ratings_ offer
user-based information related to store products, such as number of installs
in Home Assistant. _Marketing feedback_ tracks telemetry information for
creators on the performance of their product, such as GitHub Marketplace
tracking retention rate for their products for creators. _Product usage data_
details the observed usage of the products; for example, Steam tracks the
average number of hours users spend on each product.

##### Communication channels

tracks the types of methods the store directly offers for communications
between both users and creators. Since most stores offer a _product homepage_
for each of their products, the app creators are largely free to put any
information here. This means that if a creator wishes, they can put links to
other communication methods external to the store. We do not track such
information here since it is product dependent instead of store dependent.
While ratings and reviews/comments are often paired together, during our
exploration, we found cases where user ratings were permitted but user reviews
were not; thus, we have separate values for ratings and reviews. Communication
channels can take various forms with different variability, for example, some
stores allow responses for reviews. For this aspect, we stay at a high level
based on the functionality of the communication channels and consider the
variations as detailed implementation for each functionality.

#### Stage 2: Expanded collection of app stores and labeled set of
representative stores

In stage 1, we identified 53 store candidates. To provide the required data
for our experiments, two authors explored these stores to identify which of
the _fundamental features_ of the previous stage are true for each store. The
query results are summarized in Table 4, where we list the search term
construction keywords and the first 3 occurrence of stores by the search term.
For example, in search term constructed from (addon OR add-on) and ("market
place") OR marketplace, the first 3 occurrences are Google Play, PrestaShop,
and CS-Cart.

There are many app stores beyond Google Play and Apple’s App Store. These app
stores exhibit a diverse set of features.

Table 4: First three identified stores for each _Google_ query |  ("app store" OR store) |  ("market place" OR marketplace) |  shop |  repository |  manager
---|---|---|---|---|---
app |  Apple App Store, Google Play |  BigCommerce, Google Play, HubSpot |  Apple App Store |  F-Droid, Guardian Project, IzzyOnDroid |  Google Play
software |  Mac App Store |  MarketPlaceKit, Sellacious, CS-Cart |  $\phi$ |  $\phi$ |  $\phi$
(addon OR add-on) |  Mac App Store, Home Assistant, Firefox Add-ons |  Google Play, PrestaShop, CS-Cart |  PrestaShop, Chrome Web Store |  Kodi |  CurseForge, Ajour, Minion
(plugin OR plug-in) |  Google Play, SketchUcation, RICOH THETA |  WordPress, JetBrains |  Bukkit, Plugin Boutique |  WordPress, JetBrains |  Jenkins, JMeter, Autodesk
(extension -hair -lash) |  Chrome Web Store, Microsoft Edge |  VSCode Marketplace, Adobe Magento, Chrome Web Store |  Chrome Web Store |  TYPO3, GNOME SHELL |  Chrome Web Store
install |  Google Play, Apple App Store |  Google Play, Eclipse |  Apple App Store, Google Play, Microsoft Store |  Kodi, Home Assistant, DockerHub |  Google Play, APKPure, Daz3D
solution |  Mac App Store, Microsoft Store |  CS-Cart |  $\phi$ |  $\phi$ |  $\phi$
(software library -book) |  Microsoft Store |  VSCode Marketplace, QT Extensions, GitHub Marketplace |  $\phi$ |  $\phi$ |  $\phi$
package |  Apple App Store, Google Play, Snapcraft |  CS-Cart, concrete5 |  Google Play |  Packagist, PyPI, Ubuntu Packages |  Chocolatey, NPM, NuGet

###  RQ2: _Are there groups of stores that share similar features?_

Using the labeled data of the 53 stores, we were able to perform the _K-means_
cluster analysis that we have introduced in Sec. 3.2. With the number of
clusters guided by the Silhouette method to choose the best $K$ value for _K-
means_ , our clustering resulted in eight clusters.

Due to the nature of unsupervised methods, _K-means_ is able to identify only
the clusters and their members; no real-world meanings are extracted for why
the cluster members belong together. It is also important to note that the _K-
means_ algorithm performs _hard clustering_ ; that is, it creates a
partitioning of the stores into mutually exclusive groups that together span
the whole space. Thus each store will be assigned to the unique cluster that
the algorithm considers to best represent it. For this reason, the raw results
from _K-means_ should not be seen as authoritative, but rather as a vehicle
for identifying groups of stores with similar characteristics. Therefore, we
leverage the _K-means_ clustering and further examine the clusters in detail
to try to derive a human understandable categorization of the stores.

We start by analyzing the differences between clusters by analyzing the
definitive characteristics of each cluster. In Table 5, we show the details of
the top 10 features that deviate the most from the _C_. Column C contains the
_centroid-of-centroids_ with values for each feature. The remaining columns
represent each cluster by an index from 1 to 8. The values in these columns
represent the proportion of app stores in the cluster with a specific feature,
the mean, and the background color of each cell represent the deviation of the
particular cluster _centroid_ (i.e., difference between the centroid of this
cluster and the centroid-of-centroids for the feature). Each row corresponds
to a feature of the stores, which makes it easy to understand which features
are descriptive of a cluster.

Table 5: The 8 clusters found by the _K-means_ algorithm, with top deviated
features from the centroid of centroids (C). Each cell with a value represents
one of the ten most influential features of the corresponding cluster. The
number indicates the percentage of stores with the specific feature. The color
encodes whether stores in that cluster are less (magenta) or more (green)
likely to have the feature, compared to the centroid.

Cluster Index Features C 1 2 3 4 5 6 7 8 _Monetization_ Free 1.00 One-time
payment 0.35 0.00 0.00 0.00 1.00 Seat-based subscription 0.09 0.50 Time-based
subscription 0.30 0.75 0.00 0.86 Resource-based subscription 0.05 Micro-
transactions 0.11 0.86 Custom Pricing 0.01 _Rights Management_ Creator managed
DRM 0.72 0.25 1.00 0.14 Store-enforced DRM 0.27 0.75 0.86 _Do I need an
account to use the store_ Account Required 0.33 0.00 0.75 1.00 0.86 No
registration possible 0.35 1.00 0.00 0.00 0.00 1.00 Some features require
registration 0.30 1.00 1.00 _Product Type_ Standalone apps 0.42 0.00 1.00
Extension/add-ons to apps/hardware 0.68 0.33 0.00 Service/Resources 0.08
Package/Library 0.17 0.89 _Target audience_ General purpose 0.33 0.00 0.83
0.00 1.00 Domain-specific 0.67 1.00 0.17 1.00 0.00 _Type of product creators_
Business 0.67 0.22 0.00 1.00 Community 0.67 1.00 0.11 _Intent of app store_
Community building / support 0.52 1.00 1.00 0.00 0.11 Profit 0.38 0.00 0.00
0.00 0.78 0.00 1.00 Centralization of product delivery 0.84 Expanding the
platform 0.76 0.17 _Role of intermediary_ Embedded Advertisement API 0.16 0.71
CI/CD 0.05 Checks at run time 0.14 0.50 Quality/security checks 0.74 0.25
_Composability_ Independent 0.56 0.00 1.00 1.00 0.00 Vendor internal 0.15 1.00
Package manager type 0.19 1.00 _Analytics_ Sentiment and popularity ratings
0.73 0.00 0.33 Marking feedback 0.25 Product Usage data 0.33 _Communication
channels_ Documentation (wikis, FAQs) 0.81 0.25 Product homepage 0.97
Star/Score/Up/Downvote rating 0.57 0.11 1.00 1.00 0.00 1.00 0.00 Written
reviews (in text) 0.47 0.00 1.00 0.00 0.89 0.00 Community Forum 0.45 0.75 0.00
Support Ticket 0.35 Promotion/Marketing 0.71 0.25

The table only shows the top 10 deviations per cluster (i.e., _column_) to
focus on the most important contributors to each cluster. Since all features
are binary — each store has or does not have the feature— all values of the
centroid-of-centroids are between $[0,1]$; thus, a positive deviation (shown
with a green background) implies that the stores in the cluster are _more_
likely to have the attribute, and a negative deviation (shown with a magenta
background) implies that the stores are _less_ likely to have the attribute.

For example, for cluster 8 the most important contributor is _[Composability]
Vendor internal add-on/extension/unlock_ where the centroid of the cluster is
$1$. When comparing against the centroid-of-centroids (at $0.15$), the
deviation is at $0.85$; this implies that all stores in this cluster have this
feature. On the other hand, an example of negative deviation for cluster 1 is
the feature _[Composability] Independent_ with a centroid of $0$ indicating
that no stores in this cluster have this feature. Since the centroid-of-
centroids for this features is at $0.56$, this implies the deviation for
stores in this cluster is $-0.56$.

After the top characteristics that make each cluster distinctive had been
identified, we leveraged this information to name and describe each cluster
accordingly. Using the information from Table 5 which shows the defining
features of each cluster, we derived an organization of the clusters based on
several dimensions. The results are described in Table 6.

Table 6: List of stores and descriptions by cluster, with the example store that is closest to cluster centroid Type |  Stores in Cluster |  Example Store |  Cluster Description | Index
---|---|---|---|---
Extensions |  |  |  |
Commercial specialized |  Adobe Magento, AutoDesk, BigCommerce, GoG, HubSpot, Plugin Boutique, Presta Shop, SketchUcation, CS-Cart |  _Presta Shop_ offers addons to the ecommerce solution platform. |  Products in the stores are very domain specific. Creators are mostly business and their store front offers rating systems and written reviews. | 6
Community specialized |  Bukkit, CurseForge, DockerHub, Home Assistant, Jmeter, Kodi, Minion, VSCode Marketplace |  _Kodi_ add-on components offers extensions to the _Kodi_ entertainment center. |  These are community focused stores that offers free products to users. Stores also tailor to a specific domain. | 3
Community non-specialized |  Apkpure, Chrome Web Store, Eclipse Marketplace, Firefox Add-ons, Gnome, Wordpress |  _Wordpress_ offers free extensions for users using the wordpress platform. |  Products in these stores offers extensions to the platform. Essential operations do not need registration (e.g., installing apps). Products offered in the stores face a generic audience and are independent from each other. | 4
General |  |  |  |
Commercial |  AWS, Google Play Store, Microsoft Store, Nintendo eShop, Steam, Samsung Galaxy Store, Apple App Store |  _MicroSoft Store_ offers applications for the windows platform. |  Typical stores many people encounter everyday. They run for profit and offer vendor internal products supporting most monetization options. | 8
Community |  Chocolatey, F-Droid, Flatpak, Guardian Project Repository, IzzyOnDroid, Snapcraft |  _F-Droid_ is a free and open source software only _Android_ application store. |  These stores contain standalone free products only. Creators for the stores are mostly from the community and the products are majority open source. | 7
Package Manager |  Ajour, Jenkins, MacPorts, NPM, NuGet, Packagist, PyPI, Typo3, Ubuntu packages |  _Packagist_ is the main repository for _PHP_ packages. |  No account system is involved for these stores. Products are free and most in package style with inter-dependency relationships. Communication channels are also limited with ratings and reviews missing for most stores. | 1
Subscription oriented |  Github Marketplace, JetBrains, Qt Marketplace, concrete5 marketplace |  _Github Marketplace_ offers applications and actions to improve the workflow related to _git_ repositories hosted on _GitHub_ |  Often offers subscription services and supports DRM management by the store. Products are not standalone applications and either provide service or extends a platform. | 2
Other |  MarketPlaceKit, RICOH THETA, Sellacious, daz3D |  _Sellacious_ is a ecommerce platform and provides extensions to the platform. |  They do not have much communication channels offered. Rating and reviews do not exist in the stores. The stores mostly exists to distribute extensions centrally to the platform they are based on. | 5

One important dimension focuses on the type of application served by stores in
the cluster. We identified three major types of applications that
differentiate the clusters: _General_ , where the store offers stand-alone
programs that run without the need of specific software (aside from a specific
operating system, e.g., Google Play, AWS, Steam); _Extensions_ , where the
store offers extensions to a specific program or platform e.g., VSCode
Marketplace for _VSCode_ , Chrome Web Store for _Google Chrome_ ; and _Package
manager_ , where the store offers stand-alone programs, but also manages
dependency-relationships and requirements between different applications in
the store e.g., NPM, MacPorts, Ubuntu Packages. Another dimension in which
these clusters can be organized is whether they are Commercial (business-
oriented) or Community-managed (no money is involved).

App stores are not all alike. Intuitive groupings emerge naturally from the
data. Their differences can be due to the type of application they offer —
standalone or extensions — and their operational model, either business- or
community-oriented. We found that app stores in different groups of our
clustering have different properties, and these properties may have bearing on
empirical studies involving app stores.

##  5 Discussion

In this section, we discuss our findings regarding what we consider app stores
to be based on our clustering results, and we describe various research
opportunities involving the influence of app stores on software engineering
practices.

###  5.1 What Is an App Store?

The term _app store_ became popular largely through Apple’s App Store, which
launched in 2008 along with the _iPhone 3G_ app.store.introduce . Other online
software stores have also appeared and have had the term applied to them.
Originally, the term usually referred to stores of applications for mobile
devices, but we have found that today there is ample diversity of the type of
applications that app stores offer and in the features they provide to app
developers and users. App Stores are also dynamic: features are continually
being added, removed, and altered by store owners in response to changes in
their goals and feedback from their socio-technical environments. For example,
the Chrome Web Store initially introduced a built-in monetization option that
provided a mechanism for applications to receive payments from its users;
however, the store later decided to deprecate this monetization option
chrome.disable.payment  and suggested developers to switch to alternative
payment-handling options.

In our work, we have employed a working definition through our
inclusion/exclusion criteria for app stores to be included in our research.
However, due to the complexity, diversity, and constantly evolving nature of
app stores, we have decided not to attempt a firm, prescriptive definition of
the term. Instead, in the following paragraphs, we will discuss each of
several aspects of app stores in detail, and hope that in the future, a more
robust definition and operating model can emerge.

#### 5.1.1 Common Features of App Stores

Although we found significant diversity among the example app stores we
studied, we were able to identify a set of three common features that appear
to span the space of app stores.

##### Simple installation and updates of apps

An app store facilitates simple installation of a selected application, and
can also enable simple updating. For some stores, apps are expected to run on
the hardware of the client; in others, the app store provides and manages the
hardware where the app runs. In both cases, the app store frees the user from
worrying about the technical details of installation, including compatibility
with their specific hardware and software configuration, as well as the
installation of the app and its dependencies, if any. Typically, app stores
will also automate the installation of updates to the application, again
freeing the user from worrying about if they have the latest version of the
app with the latest features and bug fixes.

##### App exploration and discovery

App Stores provide mechanisms that allow users to find apps they might want to
use. In its simple form, this mechanism might be a search engine that returns
a list of apps that match a given set of keywords (such as homebrew, PyPI). In
the labeled app stores, 73% of stores provide some kind of aggregated
recommendations (e.g., advertisement and trends in WordPress), up to personal
recommendations that are based on other apps the user has installed before
(e.g., Apple’s App Store). User feedback via reviews (present in 47% of the
labeled app stores) and forums (present in 45% of the labeled app stores) can
provide further information to aid other users in identifying apps of possible
interest to them.

##### The app store guarantees the runtime environment

In practice, app stores often execute within a runtime environment (RTE), such
as an operating system (e.g., Google Play on _Android_) or an extensible
software application (e.g., Firefox Add-ons on _Firefox_). Many app stores
simply sit on top of the RTE, acting primarily as a gatekeeper for adding and
deleting apps. However, some app stores are more tightly integrated with the
RTE; in extreme cases, the app store can extend the RTE with the app store’s
own functionality and together provide an augmented RTE for the applications
managed through the app store. Steam is a good example for extending the RTE
with its own features; developers can integrate with many services offered by
Steam, such as an achievement system that offers players recognition when they
fulfill certain requirements in the game. Figure 3 illustrates the situation
where a product may integrate with additional store-added features to the RTE,
which in turn enriches the user experience of the store users. When _Product
B_ is offered in _App Store Y_ , it will not have the features provided by
_App Store X_.

Figure 3: Stores may offer optional extensions to the runtime environment for
applications

The app store ensures that apps are installed only when their runtime
requirements are satisfied. The process is often done through running checks
on apps submitted to the app store, which 74% of the labeled app stores
perform specifically. By specifying the runtime requirements, the assumption
for both the developer and the user is that if the application is installed —
implying that the requirements are satisfied — it is expected to run properly.
This is usually achieved by a software layer on top of the RTE, provided by
either the app store or the user. In its simplest form, this software layer is
responsible for installing and updating apps (see “Simple installation and
updates of apps” above). In some cases, this software layer might also include
a set of libraries that the apps can use to provide features specific to the
app store thus forming part of the RTE for the applications. These libraries
might range in purpose (domain specific, common GUI, resource management,
etc.). In extreme cases, this layer includes the operating system, as it is
the case with Apple’s App Store. However, checks during runtime is a very rare
feature, which only 14% of the labeled app stores provides.

Some hardware platforms have become so tightly integrated to the software
layer of the app store that they can be considered monolithic: the hardware is
rendered unusable without the app store. This is exemplified by the Apple’s
App Store, where one cannot use the hardware without first having an account
in the app store; even operating system upgrades are distributed via the
store.

This tight level of integration has clear benefits for all three stakeholders:
end users have fewer installation technical details to worry about; app
developers can be assured that users will be able to install their apps
without the need for technical support; and app store owners can strictly
manage who has access to the user’s RTE and how. However, such tight
integration is technically unnecessary and may even be undesirable. From a
software engineering perspective, such tight coupling could be seen as a
“design smell”, since the operating system and the app store layers address
fundamentally different concerns. Also, tight integration can create an
artificial barrier to competition, effectively establishing a quasi-monopoly
for the store owner; the store owner may assume the role of gatekeeper not
only for streamlining technical issues, but also for business reasons,
requiring a kind of toll to be paid by app developers for access to the store.
A recent initiative in the European Union eu.commission  aims to enable fair
competition by enforcing that ecosystems are opened up, which will likely also
allow the installation of alternative software layers for other app stores, a
term called side-loading. In contrast to the _Apple_ ’s tight control of the
operating system as part of its app store, _Android_ allows third-party app
store software (e.g., F-Droid store.fdroid ) to be installed in co-existence
with the system default (often Google Play).

As mentioned above, some stores distribute software that runs on hardware
owned by the App Store itself; in these cases, the RTE is fully managed and
controlled by the store. For example GitHub Marketplace and Atlassian
Marketplace offer applications that run on GitHub and Atlassian servers
respectively. In most cases, these applications are not deployed to the user’s
computers.

#### 5.1.2 Different Types of App Stores

While some features are broadly shared by all app stores, in Sec. 4, we
identified different groups of app stores based on their features. For stores
within the same group, they often share common features, whereas across
different groups, the stores tend to have less in common. We now discuss the
differences across the groups in detail.

##### Diversity in goals

As a platform focusing on delivering products to customers, the high-level
goal of one app store can be dramatically different from the other. Even app
stores providing software for the same underlying RTE can have radically
different purposes. For example, consider the app stores that run on
_Android_. Google Play is the _de facto_ store for _Android_ applications.
F-Droid store, on the other hand, offers only free and open source _Android_
applications, and APKPure offers multiple versions of the same software so the
user can decide which version they would like to install.

_Apple’s_ app store offers applications for all its RTEs: _MacOS_ (laptop and
desktops), _iOS_ (phones and tables), and the _Safari_ browser. In contrast,
_Google_ has different stores for _AndroidOS_ and for its web browser,
_Chrome_. The Microsoft Store sells hardware and apps for _Windows_ and
_XBox_. Alexa Skills offers skills that enhance the voice agent _Alexa_ ’s
capabilities.

In many program language ecosystems, the core language development (focusing
on the language features) and packaging system (focusing on extending the
functionality of the language) are led by separate organizations (e.g., NPM
org.npm  and JavaScript org.javascript ).

##### Diversity in business model

Another important difference we observed is between business-managed and
community-managed stores. In business-managed stores (with few exceptions), a
primary goal is to generate a profit. These stores provide a payment mechanism
between the app creator and the purchaser, with the store keeping a percentage
of any sales. These stores have to solve three key concerns: first,
implementing registration and authentication of users and developers; second,
some type of digital rights management, so only users who have acquired the
software can use it; and third, a payment mechanism e.g., subscription, one-
time payment, and advertisement.

Community-managed stores, on the other hand, are often run by volunteers, and
their features focus on facilitating not-for-profit product delivery from
developer to user. Many community stores offer limited community interactions
compared to business stores where customer feedback is important. For example,
in the Kodi store, add-ons have a web page (e.g., _The Movie Database Python_
kodi.tmdb ). This page provides information regarding installation of the add-
on, such as known compatibility concerns, download links, and installation
requirements. Meanwhile, most communication channels about the add-on are
hosted elsewhere; for example, installation and usage instructions, extended
descriptions, and screenshots can be found in the community forum instead.

It is important to note that the products contained in community-oriented
stores are not limited to open source software; some community-managed app
store policies often permit the distribution of proprietary software. In the
natural groupings we observed, no rights management is enforced from the store
side for Cluster 3; at the same time, most stores in Cluster 8 have some form
of rights management built-in to the store. For example, Homebrew permits apps
that are not open source if the apps are free to use; these apps might include
in-app purchases — such as an upgrade to a full-feature app — that are handled
outside of Homebrew.

###  5.2 Implications for the Main Participant Stakeholders

The results of our study includes an evidence-based detailed view of the broad
landscape of app stores. This view can help us improve the understanding of
the realities and potentialities of app stores in general. Meanwhile, the
results of our work can also benefit the different stakeholders involved with
app stores, including app creators, app stores themselves, users, and
researchers.

##### Application creators

Those who create applications — including those who design, develop, test, and
market apps — benefit from a holistic view of other stores that will allow
them identify potential new markets (stores where they can offer their
software) and to understand changing and emerging features that could
eventually come to their app store of choice. For new creators, this research
emphasizes that a software store has both technical requirements — such as the
use of a specific software development kit — and non-technical ones — such as
restrictions on what applications can do, approval processes and timelines —
and that these requirements vary significantly from one store to another.

##### App Stores

The overview presented herein provides a framework for comparison between app
stores, particularly those that operate on the same market, such as _Android_
application stores. It can also help promote wide adoption of features that
are not universal, such as communication channels between users and
developers.

##### Users

With the diversity in app stores, especially when multiple app stores are
competing in the same domain, it allows users the chose of where to acquire
their applications. This allows for more diversity for how the apps are
distributed and the user’s choice also affect the competition.

##### Researchers

As discussed in Sec. 2, most prior research has focused on the applications
offered in app stores, and there is a need for research that focuses on
studying the store themselves. This emphasis could aid researchers in
considering different points of view when conducting app store-centric
studies, and also suggest avenues of exploration concerning how the
development process is affected by the existence of app stores.

We describe this point in detail in the next sections.

###  5.3 App Store Features

In this section, we discuss how each of feature groups from Table 3 has been
addressed by current SE research and we suggest some possible future
directions.

##### Monetization

App development can be affected by their pricing strategy. For example
different software architecture to support a different system of monetization
(e.g., locking functionality behind microtransactions) pham2017paas . Studies
have shown a correlation between app features and pricing
finkelstein2017investigating ; harman2012app ; sarro2015feature . Moreover, in
many studies on apps alshayban2020accessibility ; aljedaani2019comparison ,
free apps and charged apps are often considered as different types of
applications. Future work could further explore how different monetization
options affect app development.

##### Rights management

Digital rights management is still an ongoing challenge in software
engineering. Existing studies have explored the options of implementing
different DRM systems to support developers lu2019blockchain ;
gaber2020privdrm . DRM can also add challenges in other development activities
such as complicating the testing procedure sung2019test  and affect
performance denuvo.performance . Often we can observe the store offering means
for providing and enforcing DRM. Because DRM is still a nascent technology
within software engineering, it remains an open area to explore for future
study and how app stores can play a role.

##### Account requirement

User identity enables telemetry of user behavior. An account system is also
the prerequisite of a store-wide DRM system as discussed in the previous
paragraph. Existing research has focused on how to leverage the user identity
information to create targeted recommender systems he2015mining  and also
investigated the concerns of privacy-related issues scoccia2022empirical . The
interest of developers (detailed tracing data) and users (privacy) are in
conflict, app stores that require user identification could prove to be an
excellent study subject for future research in that area.

##### Product type

Existing research has already shown different software engineering practices
based on the software product. For example, gaming development is very
different from traditional software development and open source development
murphy2014cowboys ; pascarella2018video . Research have shown that different
types of software can introduce specific challenges unique to them
lee2020empirical ; ibrahim2020too . Future research should better understand
how the product type affects user expectations and development practices, for
example, with respect to the delivery of software or the way creators and
users can interact.

##### Target audience

When an app developer decides on a specific app store to sell their app, they
are also effectively selecting for a specific type of user
subramanian2006empirical ; manotas2016empirical ; gholami2021should . Users of
a general-purpose store such as Google Play are different and much more
diverse than the user population guzman2018user  in very specialized stores,
such as the add-on store for a particular game. Research needs to understand
better which features are relevant in each specific context sun2021empirical ,
so the experience can be tailored to the concrete situation.

##### Type of product creators

Existing research has shown many differences between open source and
industrial software development pascarella2018video ; lee2020building . Some
studies have touched the aspect of release engineering in open source
development nayebi2017version , where developers would strategically select
which versions to release on the app store. However, we believe that there is
still room for more understanding in how targeting releases towards app stores
affects software development.

##### Intent of app store

While in most domains, there exists a dominant app store, we can also observe
situations where multiple app stores compete in the same domain (e.g., game
stores on PC, mobile app stores in China wang2018beyond ). In these
situations, users have a choice of which app store to use when the same
application is offered. In practice, some studies have explored how the high
level operation of app stores can affect the software delivery process
especially involving security concerns hu2021champ ; sun2021empirical .
Competition between app stores within the same domain remains largely
unstudied, as does how their operations can affect both developers and users.

##### Role of intermediary

App stores provide a platform for users and developers. Researchers have
explored how it affects software development processes such as testing and
release management nayebi2016release ; shen2017towards . There are many
opportunities for security ferreira2021containing  and quality assurance al-
subaihin2021app ; tang2019large  to be ensured on the app store side. Future
study can explore how the differences between apps managed through an app
store and apps that are not. For example, studying the difference between
open-source web extensions that are in and not in app stores.

##### Composability

Existing research has explored co-installability in the scope of package
manager systems vouillon2013software ; claes2015historical . However, we only
have limited understanding of co-installability for standalone applications in
an extension system. For example, if two standalone extensions were to modify
the same component of the underlying software, a potential incompatibility
could occur. Future research can explore this area by performing empirical
studies on existing systems to understand the issue of conflicts.

##### Analytics

App stores as the central hub between developers and users have access to rich
information useful for analytics. Previous studies have taken advantage of the
app store specific information to help software developers
mcmillan2012detecting ; martin2016causal ; maalej2019data . For example,
Ullmann et al. ullmann2022makes  leveraged records of rating statistics and
downloaded information to study the factors in developing successful video
games. Another study leveraged analytic information collected by the app store
to identify incompatible builds of application and physical devices
khalid2014prioritizing . Future work can explore what are the possible data to
collect and form analytics, and how can the analytic data be leveraged to help
developers and users.

##### Communication channels

Communication channels are the most studied area of app store features.
Specifically, there has been a heavy focus on app reviews, where researchers
have leveraged the information in app reviews to aid software development in
areas such as extracting/locating bug reports haering2021automatically ,
discover feature requests wu2021identifying  and collect user feedback
guo2020caspar . However, existing studies also suggest that the use of
communication channels in app stores are often multi-purpose
dkabrowski2022analysing . Researchers also find that some interaction
requirements between interested parties are relegated to other platforms such
as Twitter nayebi2017app . Future work can explore different types of
communication channels in their functionality and how they can integrate with
app stores. The corpora from communication channels are also rich information
sources where researchers can leverage to extract information about developer-
user interactions.

###  5.4 Research Opportunities Involving App Stores

App Stores are becoming the primary channel for software delivery and exert
considerable influence in many aspects of the software development process. A
previous study by Rosen and Shihab rosen2016mobile  on Stack Overflow
questions by mobile developers has shown that app delivery is one of the
biggest challenges developers face. Our results in Sec. 4 demonstrate that
there is a wide variety of types of stores, each with different features and
goals. Today, app stores encompass many kinds of applications, from games
running on the hardware of the user to add-ons for applications that run on
corporate servers such as _GitHub_. However, existing research often focuses
heavily on the applications offered inside app stores, especially those of the
two major mobile app stores. In the following paragraphs, we discuss several
research opportunities to study how app stores can affect software
development.

#### 5.4.1 App Stores as Actors in Software Development

##### App Stores affect the software product cycle

Researchers need to consider how and why app stores can affect the software
development life cycle. For example, we know that app stores can constrain and
sometimes even dictate software release processes. Some stores go beyond this
and exert a kind of socio-technical environmental pressure on other software
development practices, becoming a de facto stakeholder in app development.
Sometimes these environmental pressures are technical in nature, where the app
store might dictate the programming language or deployment platform/OS; some
app stores go further and create RTEs, software development kits (SDKs), and
user interface (UI) libraries that must be used by all app developers.
Sometimes these environmental pressures are non-technical in nature, such as
when the app store prescribes the kinds of application that is allowed in the
store. For example, _Microsoft_ recently announced that it will not permit app
developers to profit from open source applications.333See Update to 10.8.7
https://docs.microsoft.com/en-us/windows/uwp/publish/store-policies-change-
history When an app store operates in a manner such that it has control over
what kind of application to include, it creates a software ecosystem and as
such, it faces the same challenges that any other ecosystem has: how to
thrive. In particular, stores need to understand the needs of their developers
and users to retain existing ones and attract new ones. However, suggested by
what we have observed in Sec. 4, app stores are diverse with a large number of
features that characterize and differentiate between them. While stores are
experimenting and evolving, each action is likely to have an effect on the
ecosystems they formed, both positively and negatively. Thus, the impact of
app stores in the economy and their markets is worthy of further study.

##### An app may be offered in several app stores

Developers want to run their software on the platform that is provided or
supported by the store, and as such they must accept the requirements and
limitations that such a store may impose. This issue is compounded when the
app is being offered in more than one store, as the developers might have to
adapt their processes to different sets of requirements, some of which might
be conflicting. For instance, an app can be both available in F-Droid (in
Cluster 7) and Google Play (in Cluster 8). In Google Play, it is common for
applications to collect telemetry data to better understand typical user
behaviour; however, in F-Droid — an open source and privacy-oriented store —
such data collection is highly discouraged. Furthermore, developers must also
adapt to the features and limitations that a store provides regarding software
deployment, communication with users and — when they exist — the mechanism
available to profit from their software and to use digital rights management.
This is particularly interesting if the targeted app stores are in different
natural groupings. This introduces new areas of studies such as how store
policies propagate to applications over time, and how violations of store
policies can be detected automatically. Researchers have already begun to
investigate this topic through qualitative approaches to identify how
applications comply with specific policies that concern accessibility
alshayban2020accessibility  and human values obie2021first .

##### App stores strongly affect the release engineering process

App Stores are especially important in release engineering. Specifically, the
release process needs to consider how the application is to be packaged,
deployed, and updated. The heterogeneity of the platform provided by RTEs
might also affect the number of versions of the application that need to be
deployed, e.g., variety of target CPUs, different screen sizes and
orientations, and amount of available memory.

When an application is developed for multiple stores, it must effectively be
managed as a product line; this is because multiple deliverables must be
created, one for each platform-store combination wang2019characterizing .
Multiple deliverables can also help for telemetry reasons such as tracking the
installation source of the application ng2014android . The differences between
packaged versions might be as significant as requiring the source code to be
written in different programming languages, using different frameworks; also,
each store is likely to require different deployment processes.

For example, when cross-releasing browser add-ons, developers may have to
rewrite part of the functionality in _Swift_ /_Objective-C_ for better
integration with Safari (in the Apple’s App Store), while at the same time
maintaining a fully _JavaScript_ version for Chrome Web Store. Also, the
scheduling of release activities is often dictated by the release processes of
the stores. A previous study has showed that taking into consideration of app
review times is an important factor when planning releases al-subaihin2021app
. The app store standardizes, and often simplifies, the release engineering
processes for its store; but it also becomes a potential roadblock that might
delay or even reject a new release.

#### 5.4.2 The Challenge of Transferring Understanding Between Stores

As noted above, prior work has examined many aspects of app stores, yet the
app store itself has rarely been the focus of the research. In many studies,
the app store serves as a convenient collection of apps, and the research
focuses on mobile development concerns such as testing and bug localization.
Even when research focuses on the app store itself, the scope rarely extends
beyond Google Play and Apple’s App Store. Based on our observations, the
diversity of app stores in their operational goals, business models, delivery
channels, and feature sets can affect the generalizability of research
outcomes. For example, there have recently been many studies
dkabrowski2022analysing ; lin2019empirical ; wu2021identifying ;
haering2021automatically ; guo2020caspar ; obie2021first ; fischer2021does
that focus on app reviews. However, for an app store that does not have
reviews (e.g., Nintendo eShop) none of the findings and tools can be leveraged
(e.g., stores in Cluster 1, 5, and 7).

##### App Stores that have the same features may still differ significantly

Depending on the problem domain, the details of software development practices
can vary dramatically. For example, game development has been compared to both
more traditional industrial software development murphy2014cowboys  and to
open source software development pascarella2018video ; in both cases, the
development processes can differ greatly. We conjecture that the same may also
occur across app stores, where despite the same feature is being offered in
the different stores, the convention of using them could be different. As
mentioned above, one specific observation has been made between the gaming-
focused store Steam and mobile stores (e.g., Google Play) in Cluster 8, where
Lin et al. lin2019empirical  found that reviews across the platforms for the
same app were often quite different in tone. Such uncertainly invites future
research to validate their findings in one store to another to improve the
generalizability of the results, and also encourages replication studies to
verify existing results on other stores.

##### A feature not in the app store does not mean the functionality is
missing

While some app stores aim to provide a complete experience, where all
interactions from the developers and users are expected to be performed within
the store, some app stores export part of the work to other platforms. This
can even occur for common features that one might find essential. For
instance, starred reviews are universal in Cluster 2, 4, and 6 where typical
users leverage this information to decide whether an application is good;
starred reviews are uncommon for other stores in Cluster 1, 5, 7. The
specialized store may have some other metric to indicate popularity or
quality, such as total number of downloads, but the focus of the store is
often to offer a managed way of installation. Other features, such as
application support, are left to other platforms such as social media.
Research can further explore the integration between app stores and other
platforms.

##  6 Threats to Validity

##### Internal Validity

Our initial seeding of app stores comes from personal experience of app stores
by the authors of the paper. Personal bias could cause us to miss other types
of app stores. However, given the number of authors on this paper and our
initial effort to consider as many stores as possible, we feel that have
created a wide, deep, and collaborative “best effort”. When we labeled app
stores by their dimensions, it is a qualitative process. As with any
qualitative process, the results could be biased by the authors performing the
task. We tackled this issue by first labeling a few stores separately by all
authors and discussing the results until a consensus was achieved; thus, we
started with a set of “gold standard” labels. Then the labeling task was
delegated to two authors who continued to label the stores separately with a
portion of the store overlapping. The overlapping labels are then verified by
the _Cohen’s Kappa_ between the two authors to measure the agreement.

We leveraged the _K-means_ algorithm for the clustering process. We first
applied _PCA_ techniques to reduce the dimensions of the initial labeling and
provide an orthogonal basis to feed the _K-means_ clustering. When using other
clustering algorithms (e.g., _Mean-shift_ , _DBSCAN_), the clustering result
might change; while _K-means_ is widely adopted for clustering process in SE
research, by nature, determining the proper $k$ value is still a challenge. We
followed common best practice to use metrics (i.e., the _Silhouette_ method)
to determine the best value $k$. Despite our efforts, the output of the _K-
means_ clustering is not perfect. We mainly leveraged the _K-means_ clustering
as the first step to illustrate that app stores forms natural clusters which
are different from each other. Based on the _K-means_ output, we further
grouped the clusters into types based on our qualitative understanding of the
app store space.

##### External Validity

During the process of expanding app stores, we relied on the _Google Search
Engine_ to find web results based on keywords. The results of this step rely
on the capability of _Google_ and are subject to change over time as _Google_
updates its search algorithms. The order may also be affected by SEO
operations. Combining results from other search engines (e.g., _DuckDuckGo_ ,
_Bing_) can help to reduce the bias.

When we applied our inclusion criteria, 1) app stores must contain software
products and 2) should offer an end-to-end experience for users (ordering,
delivery, installation), we excluded stores that focus on digital assets that
are not software, such as a pure assets store that offers cosmetic
enhancements to desktop environments; we also excluded stores that offer
software products but in a way such that installation is completely managed by
users. An extreme example, would be the software section of _Amazon_ where
software is sold as an activation key which users would input to activate the
software that they need to install themselves. A more general inspection of
all means of distribution software can be performed to gain a broader
understanding of software distribution.

We relied on only publicly available information to label each store. So if
some functionality (e.g., analytics information) is not documented publicly,
we were unable to confirm whether the store has such functionality. We also
set a time limit to label each store so in case we were unable to find
information about the store, with each store receives the same amount of
attention.

One of the main challenges for reproducibility and replicability is that the
_Google Search_ results and app stores can change overtime. New app stores are
likely to emerge and existing app stores may introduce and remove features.
The focus of our study is not to establish an exhaustive catalog of app
stores, nor to study the historic evolution of a store. Our goal is to
establish a framework that can describe app stores and to understand whether
the operations of app stores follow different patterns. Based on the
granularity which we extracted features from app stores, we expect the
majority of the feature groups will remain stable over time. In the future, if
researchers would like to repeat our study, the labeling results may differ
due to updates in the app store. To mitigate this issue, we have included a
snapshot of all _Google Search_ results, and documented how we would perform
the labeling. So while the final labels may differ, by applying the same
process, a replication study would be possible with updated data.

##  7 Summary

In this paper, we have explored the idea of what an app store is and what
features make app stores unique from each other. We labeled a set of
representative stores, curated from web search queries, by their features to
study the natural groupings of the stores. Our analysis suggests that app
stores can differ in the type of product offered in the store, and whether the
store is business oriented or community oriented. These natural groupings of
the stores challenge the manner in which app store research has largely been
mobile focused. Previous studies have already shown empirical differences in
activities in mobile app stores and game stores lin2019empirical . Our study
further suggests that in the future, when we study app stores, we will need to
consider the generalizability of the results across app stores. Since one type
of app store may operate under different constraints than another kind,
results observed in one app store setting may not generalize to others.

## Conflict of Interests

The authors declared that they have no conflict of interest.

## Data Availability Statement

A dataset consists of the _Google_ query results and the app store labeling
results are available on _Zenodo_.444https://zenodo.org/record/7968192

###### Acknowledgements.

We would like to thank the attendees of the Shonan meeting
shonan.meeting.report  on “Release Engineering for Mobile Applications”, where
the paper’s idea was conceived. One of the authors has received funding from
the European Union’s Horizon 2020 research and innovation programme under
grant agreement number $825328$ (FASTEN).

## References

  * (1) C. Dixon, R. Mahajan, S. Agarwal, A. Brush, B. Lee, S. Saroiu, and V. Bahl, “The home needs an operating system (and an app store),” in SIGCOMM Workshop on Hot Topics in Networks, ACM, 2010.
  * (2) Valve, “Welcome to Steam.” https://store.steampowered.com/, 2022.  Accessed: Jun. 22 2022.
  * (3) GitHub, “GitHub Marketplace · to improve your workflow · GitHub.” https://github.com/marketplace?type=, 2022.  Accessed: Jun. 06 2022.
  * (4) Google, “Chrome Web Store - Extensions.” https://chrome.google.com/webstore/category/extensions, 2022.  Accessed: Jun. 22, 2022.
  * (5) WordPress, “WordPress Plugins | WordPress.org.” https://wordpress.org/plugins/, 2022.  Accessed: Jun. 22, 2022.
  * (6) Autodesk, “Autodesk App Store : Plugins, Add-ons for Autodesk software, AutoCAD, Revit, Inventor, 3ds Max, Maya ….” https://apps.autodesk.com/, 2022.  Accessed: Jun. 22, 2022.
  * (7) Docker, “Explore Docker’s Container Image Repository | Docker Hub.” https://hub.docker.com/search?q=, 2022.  Accessed: Jun. 22, 2022.
  * (8) Amazon, “AWS Marketplace: Homepage.” https://aws.amazon.com/marketplace/, 2022.  Accessed: Jun. 22, 2022.
  * (9) Rémi Prévost, Mike McQuaid, and Danielle Lalonde, “The Missing Package Manager for macOS (or Linux) — Homebrew.” https://brew.sh/, 2022.  Accessed: Jun. 22, 2022.
  * (10) Canonical, “Ubuntu Software Center in Launchpad.” https://launchpad.net/software-center, 2009.  Accessed: Jun. 22, 2022.
  * (11) M. Harman, Y. Jia, and Y. Zhang, “App store mining and analysis: MSR for App Stores,” in Int. Conf. on Mining Software Repositories, IEEE, 2012.
  * (12) D. Lin, C.-P. Bezemer, Y. Zou, and A. E. Hassan, “An empirical study of game reviews on the steam platform,” in Empirical Software Engineering, Springer, 2019.
  * (13) Wikipedia, “Electronic AppWrapper - Wikipedia.” https://en.wikipedia.org/wiki/Electronic_AppWrapper, 2022.  Accessed: Jun. 22, 2022.
  * (14) J. MacQueen et al., “Some methods for classification and analysis of multivariate observations,” in Proceedings of the fifth Berkeley symposium on mathematical statistics and probability, Oakland, CA, USA, 1967\.
  * (15) P. J. Rousseeuw, “Silhouettes: a graphical aid to the interpretation and validation of cluster analysis,” in Journal of computational and applied mathematics, Elsevier, 1987.
  * (16) I. J. M. Ruiz, M. Nagappan, B. Adams, and A. E. Hassan, “Understanding reuse in the android market,” in Int. Conf. on Program Comprehension, IEEE, 2012\.
  * (17) W. Martin, F. Sarro, Y. Jia, Y. Zhang, and M. Harman, “A survey of app store analysis for software engineering,” in Transactions on Software Engineering, IEEE, 2016.
  * (18) J. Dąbrowski, E. Letier, A. Perini, and A. Susi, “Analysing app reviews for software engineering: a systematic literature review,” in Empirical Software Engineering, Springer, 2022.
  * (19) X. Zhan, L. Fan, S. Chen, F. Wu, T. Liu, X. Luo, and Y. Liu, “Atvhunter: Reliable version detection of third-party libraries for vulnerability identification in android applications,” in Int. Conf. on Software Engineering, IEEE, 2021.
  * (20) X. Zhang, X. Wang, R. Slavin, T. Breaux, and J. Niu, “How does misconfiguration of analytic services compromise mobile privacy?,” in Int. Conf. on Software Engineering, IEEE, 2020.
  * (21) S. Rahaman, I. Neamtiu, and X. Yin, “Algebraic-datatype taint tracking, with applications to understanding Android identifier leaks,” in Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ACM, 2021.
  * (22) T. Nguyen, P. Vu, and T. Nguyen, “Code recommendation for exception handling,” in Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ACM, 2020\.
  * (23) L. Pan, B. Cui, H. Liu, J. Yan, S. Wang, J. Yan, and J. Zhang, “Static asynchronous component misuse detection for Android applications,” in Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ACM, 2020.
  * (24) S. Arzt, “Sustainable Solving: Reducing The Memory Footprint of IFDS-Based Data Flow Analyses Using Intelligent Garbage Collection,” in Int. Conf. on Software Engineering, IEEE, 2021.
  * (25) S. Yang, Y. Wang, Y. Yao, H. Wang, Y. F. Ye, and X. Xiao, “DescribeCtx: Context-Aware Description Synthesis for Sensitive Behaviors in Mobile Apps,” in Int. Conf. on Software Engineering, IEEE, 2022.
  * (26) Z. Dong, M. Böhme, L. Cojocaru, and A. Roychoudhury, “Time-travel testing of android apps,” in Int. Conf. on Software Engineering, IEEE, 2020.
  * (27) S. Chen, L. Fan, G. Meng, T. Su, M. Xue, Y. Xue, Y. Liu, and L. Xu, “An empirical assessment of security risks of global android banking apps,” in Int. Conf. on Software Engineering, IEEE, 2020.
  * (28) S. Almanee, A. Ünal, M. Payer, and J. Garcia, “Too Quiet in the Library: An Empirical Study of Security Updates in Android Apps’ Native Code,” in Int. Conf. on Software Engineering, IEEE, 2021.
  * (29) A. Alshayban, I. Ahmed, and S. Malek, “Accessibility issues in android apps: state of affairs, sentiments, and ways forward,” in Int. Conf. on Software Engineering, IEEE, 2020.
  * (30) B. Yang, Z. Xing, X. Xia, C. Chen, D. Ye, and S. Li, “Don’t do that! hunting down visual design smells in complex uis against design guidelines,” in Int. Conf. on Software Engineering, IEEE, 2021.
  * (31) P. Liu, L. Li, Y. Yan, M. Fazzini, and J. Grundy, “Identifying and characterizing silently-evolved methods in the android API,” in Int. Conf. on Software Engineering: Software Engineering in Practice, IEEE, 2021.
  * (32) S. Yu, C. Fang, Y. Yun, and Y. Feng, “Layout and image recognition driving cross-platform automated mobile testing,” in Int. Conf. on Software Engineering, IEEE, 2021.
  * (33) J. Ye, K. Chen, X. Xie, L. Ma, R. Huang, Y. Chen, Y. Xue, and J. Zhao, “An empirical study of GUI widget detection for industrial mobile games,” in Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ACM, 2021.
  * (34) S. Ma, J. Li, H. Kim, E. Bertino, S. Nepal, D. Ostry, and C. Sun, “Fine with “1234”? An Analysis of SMS One-Time Password Randomness in Android Apps,” in Int. Conf. on Software Engineering, IEEE, 2021.
  * (35) W. Song, M. Han, and J. Huang, “IMGDroid: Detecting Image Loading Defects in Android Applications,” in Int. Conf. on Software Engineering, IEEE, 2021\.
  * (36) T. Zhao, C. Chen, Y. Liu, and X. Zhu, “GUIGAN: Learning to Generate GUI Designs Using Generative Adversarial Networks,” in Int. Conf. on Software Engineering, IEEE, 2021.
  * (37) J. Chen, C. Chen, Z. Xing, X. Xu, L. Zhut, G. Li, and J. Wang, “Unblind your apps: Predicting natural-language labels for mobile gui components by deep learning,” in Int. Conf. on Software Engineering, IEEE, 2020.
  * (38) K. Kuznetsov, C. Fu, S. Gao, D. N. Jansen, L. Zhang, and A. Zeller, “Frontmatter: mining Android user interfaces at scale,” in Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ACM, 2021.
  * (39) D. Van Der Linden, P. Anthonysamy, B. Nuseibeh, T. T. Tun, M. Petre, M. Levine, J. Towse, and A. Rashid, “Schrödinger’s security: Opening the box on app developers’ security rationale,” in Int. Conf. on Software Engineering, IEEE, 2020.
  * (40) V. Murali, E. Yao, U. Mathur, and S. Chandra, “Scalable statistical root cause analysis on app telemetry,” in Int. Conf. on Software Engineering: Software Engineering in Practice, IEEE, 2021.
  * (41) R. Sun, W. Wang, M. Xue, G. Tyson, S. Camtepe, and D. C. Ranasinghe, “An empirical assessment of global COVID-19 contact tracing applications,” in Int. Conf. on Software Engineering, IEEE, 2021.
  * (42) A. Truelove, E. S. de Almeida, and I. Ahmed, “We’ll Fix It in Post: What Do Bug Fixes in Video Game Update Notes Tell Us?,” in Int. Conf. on Software Engineering, IEEE, 2021.
  * (43) M. Haering, C. Stanik, and W. Maalej, “Automatically matching bug reports with related app reviews,” in Int. Conf. on Software Engineering, IEEE, 2021.
  * (44) S. Yu, C. Fang, Z. Cao, X. Wang, T. Li, and Z. Chen, “Prioritize crowdsourced test reports via deep screenshot understanding,” in Int. Conf. on Software Engineering, IEEE, 2021.
  * (45) H. O. Obie, W. Hussain, X. Xia, J. Grundy, L. Li, B. Turhan, J. Whittle, and M. Shahin, “A first look at human values-violation in app reviews,” in Int. Conf. on Software Engineering: Software Engineering in Society, IEEE, 2021.
  * (46) R. A.-L. Fischer, R. Walczuch, and E. Guzman, “Does culture matter? impact of individualism and uncertainty avoidance on app reviews,” in Int. Conf. on Software Engineering: Software Engineering in Society, IEEE, 2021.
  * (47) O. Haggag, S. Haggag, J. Grundy, and M. Abdelrazek, “COVID-19 vs social media apps: does privacy really matter?,” in Int. Conf. on Software Engineering: Software Engineering in Society, IEEE, 2021.
  * (48) R. A. Shams, W. Hussain, G. Oliver, A. Nurwidyantoro, H. Perera, and J. Whittle, “Society-oriented applications development: Investigating users’ values from bangladeshi agriculture mobile applications,” in Int. Conf. on Software Engineering: Software Engineering in Society, IEEE, 2020\.
  * (49) Z. Zhang, Y. Feng, M. D. Ernst, S. Porst, and I. Dillig, “Checking conformance of applications against GUI policies,” in Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ACM, 2021.
  * (50) H. Wu, W. Deng, X. Niu, and C. Nie, “Identifying key features from app user reviews,” in Int. Conf. on Software Engineering, IEEE, 2021.
  * (51) Y. Hu, H. Wang, T. Ji, X. Xiao, X. Luo, P. Gao, and Y. Guo, “Champ: Characterizing undesired app behaviors from user comments based on market policies,” in Int. Conf. on Software Engineering, IEEE, 2021.
  * (52) H. Guo and M. P. Singh, “Caspar: extracting and synthesizing user stories of problems from app reviews,” in Int. Conf. on Software Engineering, IEEE, 2020.
  * (53) A. A. Al-Subaihin, F. Sarro, S. Black, L. Capra, and M. Harman, “App Store Effects on Software Engineering Practices,” in Transactions on Software Engineering, IEEE, 2021.
  * (54) H. Wang, Z. Liu, J. Liang, N. Vallina-Rodriguez, Y. Guo, L. Li, J. Tapiador, J. Cao, and G. Xu, “Beyond google play: A large-scale comparative study of chinese android app markets,” in Internet Measurement Conference 2018, 2018.
  * (55) S. Jansen and E. Bloemendal, “Defining app stores: The role of curated marketplaces in software ecosystems,” in ICSOB, Springer, 2013.
  * (56) D. Walker and F. Myrick, “Grounded theory: An exploration of process and procedure,” in Qualitative health research, Sage, 2006.
  * (57) A. P. M. Coxon et al., Sorting data: Collection and analysis.  Sage, 1999.
  * (58) S. Adolph, W. Hall, and P. Kruchten, “Using grounded theory to study the experience of software development,” in Empirical Software Engineering, Springer, 2011.
  * (59) R. Hoda, J. Noble, and S. Marshall, “Developing a grounded theory to explain the practices of self-organizing Agile teams,” in Empirical Software Engineering, Springer, 2012.
  * (60) Z. Masood, R. Hoda, and K. Blincoe, “How agile teams make self-assignment work: a grounded theory study,” in Empirical Software Engineering, Springer, 2020.
  * (61) C. Vassallo, S. Panichella, F. Palomba, S. Proksch, H. C. Gall, and A. Zaidman, “How developers engage with static analysis tools in different contexts,” in Empirical Software Engineering, Springer, 2020.
  * (62) J. Chen, X. Xia, D. Lo, J. Grundy, and X. Yang, “Maintenance-related concerns for post-deployed Ethereum smart contract development: issues, techniques, and future challenges,” in Empirical Software Engineering, Springer, 2021\.
  * (63) P. Wang, C. Brown, J. A. Jennings, and K. T. Stolee, “Demystifying regular expression bugs,” in Empirical Software Engineering, Springer, 2022.
  * (64) J. Cohen, “A coefficient of agreement for nominal scales,” in Educational and psychological measurement, Sage, 1960.
  * (65) J. Pérez, J. Díaz, J. Garcia-Martin, and B. Tabuenca, “Systematic literature reviews in software engineering—Enhancement of the study selection process using Cohen’s kappa statistic,” in Journal of Systems and Software, Elsevier, 2020.
  * (66) C. A. Lantz and E. Nebenzahl, “Behavior and interpretation of the $\kappa$ statistic: Resolution of the two paradoxes,” in Journal of clinical epidemiology, Elsevier, 1996.
  * (67) P. Pickerill, H. J. Jungen, M. Ochodek, M. Maćkowiak, and M. Staron, “Phantom: Curating github for engineered software projects using time-series clustering,” Empirical Software Engineering, 2020.
  * (68) V. Khatibi Bardsiri, D. N. A. Jawawi, S. Z. M. Hashim, and E. Khatibi, “A flexible method to estimate the software development effort based on the classification of projects and localization of comparisons,” Empirical Software Engineering, 2014.
  * (69) A. Al-Subaihin, F. Sarro, S. Black, and L. Capra, “Empirical comparison of text-based mobile apps similarity measurement techniques,” Empirical Software Engineering, 2019.
  * (70) T. Kuchta, T. Lutellier, E. Wong, L. Tan, and C. Cadar, “On the correctness of electronic documents: studying, finding, and localizing inconsistency bugs in PDF readers and files,” Empirical Software Engineering, 2018.
  * (71) D. Arthur and S. Vassilvitskii, “k-means++: The advantages of careful seeding,” tech. rep., Stanford, 2006.
  * (72) S. Wold, K. Esbensen, and P. Geladi, “Principal component analysis,” in Chemometrics and intelligent laboratory systems, Elsevier, 1987.
  * (73) Apple, “Apple Introduces the New iPhone 3G.” https://www.apple.com/ca/newsroom/2008/06/09Apple-Introduces-the-New-iPhone-3G/, 2008\.  Accessed: Jul. 17, 2022.
  * (74) Google, “Chrome Web Store payments deprecation.” https://developer.chrome.com/docs/webstore/cws-payments-deprecation/, 2022\.  Accessed: Mar. 16, 2022.
  * (75) E. Commission, “Digital Markets Act: Commission welcomes political agreement on rules to ensure fair and open digital markets.” https://ec.europa.eu/commission/presscorner/detail/en/IP_22_1978, 2022.  Accessed: Jul. 13, 2022.
  * (76) F-Droid, “F-Droid - Free and Open Source Android App Repository.” https://f-droid.org/, 2022.  Accessed: Oct. 02, 2022.
  * (77) npm, “npm About.” https://www.npmjs.com/about, 2022.  Accessed: Oct. 02, 2022.
  * (78) E. International, “TC39 - Specifying JavaScript..” https://tc39.es/, 2022\.  Accessed: Oct. 02, 2022.
  * (79) T. Kodi, “The Movie Database Python | Matrix | Addons | Kodi.” https://kodi.tv/addons/matrix/metadata.themoviedb.org.python, 2022.  Accessed: Jul. 13, 2022.
  * (80) V. V. H. Pham, X. Liu, X. Zheng, M. Fu, S. V. Deshpande, W. Xia, R. Zhou, and M. Abdelrazek, “PaaS-black or white: an investigation into software development model for building retail industry SaaS,” in Int. Conf. on Software Engineering Companion (ICSE-C), IEEE, 2017.
  * (81) A. Finkelstein, M. Harman, Y. Jia, W. Martin, F. Sarro, and Y. Zhang, “Investigating the relationship between price, rating, and popularity in the Blackberry World App Store,” Information and Software Technology, 2017\.
  * (82) F. Sarro, A. A. Al-Subaihin, M. Harman, Y. Jia, W. Martin, and Y. Zhang, “Feature lifecycles as they spread, migrate, remain, and die in app stores,” in Int. requirements engineering conference (RE), IEEE, 2015\.
  * (83) W. Aljedaani, M. Nagappan, B. Adams, and M. Godfrey, “A comparison of bugs across the ios and android platforms of two open source cross platform browser apps,” in Int. Conf. on Mobile Software Engineering and Systems, IEEE, 2019.
  * (84) Z. Lu, Y. Shi, R. Tao, and Z. Zhang, “Blockchain for digital rights management of design works,” in Int. Conf on Software Engineering and Service Science (ICSESS), IEEE, 2019.
  * (85) T. Gaber, A. Ahmed, and A. Mostafa, “Privdrm: A privacy-preserving secure digital right management system,” in Evaluation and Assessment in Software Engineering, ACM, 2020.
  * (86) A. Sung, S. Kim, Y. Kim, Y. Jang, and J. Kim, “Test automation and its limitations: a case study,” in Int. Conf. on Automated Software Engineering (ASE), IEEE, 2019.
  * (87) M. Lemon, “Two Point Hospital no longer uses Denuvo DRM.” https://www.vg247.com/two-point-hospital-no-longer-uses-denuvo-drm, 2018\.  Accessed: Mar. 31, 2023.
  * (88) X. He, W. Dai, G. Cao, R. Tang, M. Yuan, and Q. Yang, “Mining target users for online marketing based on app store data,” in Int. Conf. on Big Data (Big Data), IEEE, 2015.
  * (89) G. L. Scoccia, M. Autili, G. Stilo, and P. Inverardi, “An empirical study of privacy labels on the Apple iOS mobile app store,” in Int. Conf. on Mobile Software Engineering and Systems, 2022.
  * (90) E. Murphy-Hill, T. Zimmermann, and N. Nagappan, “Cowboys, ankle sprains, and keepers of quality: How is video game development different from software development?,” in Int. Conf. on Software Engineering, 2014.
  * (91) L. Pascarella, F. Palomba, M. Di Penta, and A. Bacchelli, “How is video game development different from software development in open source?,” in Int. Conf. on Mining Software Repositories, IEEE, 2018.
  * (92) D. Lee, G. K. Rajbahadur, D. Lin, M. Sayagh, C.-P. Bezemer, and A. E. Hassan, “An empirical study of the characteristics of popular Minecraft mods,” Empirical Software Engineering, 2020.
  * (93) M. H. Ibrahim, M. Sayagh, and A. E. Hassan, “Too many images on dockerhub! how different are images for the same system?,” Empirical Software Engineering, 2020.
  * (94) G. H. Subramanian, P. C. Pendharkar, and M. Wallace, “An empirical study of the effect of complexity, platform, and program type on software development effort of business applications,” Empirical Software Engineering, 2006\.
  * (95) I. Manotas, C. Bird, R. Zhang, D. Shepherd, C. Jaspan, C. Sadowski, L. Pollock, and J. Clause, “An empirical study of practitioners’ perspectives on green software engineering,” in Int. Conf. on Software Engineering, 2016.
  * (96) S. Gholami, H. Khazaei, and C.-P. Bezemer, “Should you upgrade official docker hub images in production environments?,” in Int. Conf. on Software Engineering: New Ideas and Emerging Results (ICSE-NIER), IEEE, 2021\.
  * (97) E. Guzman, L. Oliveira, Y. Steiner, L. C. Wagner, and M. Glinz, “User feedback in the app store: a cross-cultural study,” in Int. Conf. on Software Engineering: Software Engineering in Society, 2018.
  * (98) D. Lee, D. Lin, C.-P. Bezemer, and A. E. Hassan, “Building the perfect game–an empirical study of game modifications,” Empirical Software Engineering, 2020.
  * (99) M. Nayebi, H. Farahi, and G. Ruhe, “Which version should be released to app store?,” in Int. Symposium on Empirical Software Engineering and Measurement (ESEM), IEEE, 2017.
  * (100) M. Nayebi, B. Adams, and G. Ruhe, “Release Practices for Mobile Apps–What do Users and Developers Think?,” in Int. Conf. on software analysis, evolution, and reengineering (saner), IEEE, 2016.
  * (101) S. Shen, X. Lu, Z. Hu, and X. Liu, “Towards release strategy optimization for apps in Google Play,” in Proceedings of the 9th Asia-Pacific Symposium on Internetware, 2017.
  * (102) G. Ferreira, L. Jia, J. Sunshine, and C. Kästner, “Containing malicious package updates in npm with a lightweight permission system,” in Int. Conf. on Software Engineering (ICSE), IEEE, 2021.
  * (103) C. Tang, S. Chen, L. Fan, L. Xu, Y. Liu, Z. Tang, and L. Dou, “A large-scale empirical study on industrial fake apps,” in Int. Conf. on Software Engineering: Software Engineering in Practice (ICSE-SEIP), IEEE, 2019.
  * (104) J. Vouillon and R. D. Cosmo, “On software component co-installability,” Transactions on Software Engineering and Methodology (TOSEM), 2013.
  * (105) M. Claes, T. Mens, R. Di Cosmo, and J. Vouillon, “A historical analysis of Debian package incompatibilities,” in Int. Conf. on Mining Software Repositories, IEEE, 2015.
  * (106) C. McMillan, M. Grechanik, and D. Poshyvanyk, “Detecting similar software applications,” in Int. Conf. on Software Engineering (ICSE), IEEE, 2012\.
  * (107) W. Martin, F. Sarro, and M. Harman, “Causal impact analysis for app releases in google play,” in Int. Symposium on Foundations of software engineering, 2016.
  * (108) W. Maalej, M. Nayebi, and G. Ruhe, “Data-driven requirements engineering-an update,” in Int. Conf. on Software Engineering: Software Engineering in Practice (ICSE-SEIP), IEEE, 2019.
  * (109) G. C. Ullmann, C. Politowski, Y.-G. Guéhéneuc, and F. Petrillo, “What makes a game high-rated? towards factors of video game success,” in Int. ICSE Workshop on Games and Software Engineering: Engineering Fun, Inspiration, and Motivation, 2022.
  * (110) H. Khalid, M. Nagappan, E. Shihab, and A. E. Hassan, “Prioritizing the devices to test your app on: A case study of android game apps,” in Int. Symposium on Foundations of Software Engineering, 2014.
  * (111) M. Nayebi, H. Cho, H. Farrahi, and G. Ruhe, “App store mining is not enough,” in Int. Conf. on Software Engineering Companion (ICSE-C), IEEE, 2017.
  * (112) C. Rosen and E. Shihab, “What are mobile developers asking about? a large scale study using stack overflow,” in Empirical Software Engineering, Springer, 2016.
  * (113) H. Wang, X. Wang, and Y. Guo, “Characterizing the global mobile app developers: a large-scale empirical study,” in Int. Conf. on Mobile Software Engineering and Systems, IEEE, 2019.
  * (114) Y. Y. Ng, H. Zhou, Z. Ji, H. Luo, and Y. Dong, “Which Android app store can be trusted in China?,” in Computer Software and Applications Conference, IEEE, 2014.
  * (115) S. McIntosh, Y. Kamei, and M. Nagappan, Release Engineering for Mobile Applications — Communications of NII Shonan Meetings.  Springer, 2019.