# Towards Accurate Labeling of Android Apps for Reliable Malware Detection

Aleieldin Salem  Technische Universität München
Garching bei München, Germany
<EMAIL_ADDRESS>
###### Abstract

In training their newly-developed malware detection methods, researchers rely
on threshold-based labeling strategies that interpret the scan reports
provided by online platforms, such as VirusTotal. The dynamicity of this
platform renders those labeling strategies unsustainable over prolonged
periods, which leads to inaccurate labels. Using inaccurately labeled apps to
train and evaluate malware detection methods significantly undermines the
reliability of their results, leading to either dismissing otherwise promising
detection approaches or adopting intrinsically inadequate ones. The
infeasibility of generating accurate labels via manual analysis and the lack
of reliable alternatives force researchers to utilize VirusTotal to label
apps. In the paper, we tackle this issue in two manners. Firstly, we reveal
the aspects of VirusTotal's dynamicity and how they impact threshold-based
labeling strategies and provide actionable insights on how to use these
labeling strategies given VirusTotal's dynamicity reliably. Secondly, we
motivate the implementation of alternative platforms by (a) identifying
VirusTotal limitations that such platforms should avoid, and (b) proposing an
architecture of how such platforms can be constructed to mitigate VirusTotal's
limitations.

###### Index Terms:

Software Reliability; Android Security; Malware Detection; Machine Learning

##  I Introduction

The generation of reliable ground truths for malicious and benign applications
(hereafter apps) is fundamental for implementing and evaluating effective
malware detection methods. Assigning apps inaccurate labels (e.g., labeling
malicious apps as benign) might impact the reliability of studies that inspect
trends adopted by malicious apps, and, more importantly, might impede the
development of effective detection methods [1, 2, 3, 4]. Manual analysis and
labeling of apps is arguably the most reliable method to label apps. However,
it can neither cope with the frequent release of malware nor the requirement
of some detection methods (e.g., Machine Learning (ML)-based methods) of large
numbers of labeled apps for training and validation. Consequently, researchers
turn to online platforms, such as VirusTotal [5], to label apps in their
datasets as malicious and benign.

VirusTotal does not label apps as malicious and benign. Given the hash of an
app or its executable, the platform provides the scan results from about 60
different commercial antiviral software [6, 7, 8, 9, 10]. So, it is up to the
platform's user to decide upon strategies to interpret such information to
label apps as malicious and benign. Unfortunately, there are no standard
procedures for interpreting the scan results acquired from VirusTotal to label
apps, which leads researchers to use their intuitions and adopt ad hoc
threshold-based strategies to label the apps in the datasets used to train and
evaluate their detection methods. In essence, threshold-based labeling
strategies deem an app as malicious if the number of antiviral scanners
labeling the apps as malicious meets a certain threshold. For example, based
on VirusTotal's scan reports, Li et al. labeled the apps in their
_Piggybacking_ dataset as malicious if at least one scanner labeled them as
malicious [7], Pendlebury et al. labeled an app as malicious if four or more
scanners did so [11], and Wei et al. labeled apps in the _AMD_ dataset as
malicious if 50% or more of the total scanners labeled an app as such [9].

Some of the aforementioned threshold-based labeling strategies may indeed
accurately label apps better than others and should be standardized.
Nonetheless, researchers have found VirusTotal to be dynamic in terms of the
labels given by the scanners it uses [12, 3, 13], which affects threshold-
based labeling strategies as follows. Threshold values that used to yield the
most accurate labels might change in the future as VirusTotal changes the
scanners it includes in its scan reports. Using out-of-date or inaccurate
thresholds alters the distribution of malicious and benign apps in the same
dataset, effectively yielding different detection results as revealed by
recent results [11, 14]. On the one hand, researchers might dismiss promising
detection approaches, because they underperform on a dataset that utilizes a
labeling strategy that does not reflect the true nature of the apps in the
dataset. On the other hand, developers of inadequate detection methods might
get a false sense of confidence in the detection capabilities of their
detection methods because they perform well, albeit using an inaccurate
labeling strategy [15, 16].

Until a more stable alternative to VirusTotal is introduced, the research
community will continue to use VirusTotal to label apps using subjective
thresholds. So, the overarching objective of this paper is to provide the
research community with actionable insights about VirusTotal's dynamicity, its
limitations, and how to optimally interpret its scan reports to label apps
accurately using threshold-based labeling strategies. To achieve this
objective, we focus on Android apps as a case study and utilize four datasets
of 53K Android malicious and benign apps. Furthermore, based on the identified
limitations of VirusTotal, we provide a blueprint for a platform that
mitigates the limitations of VirusTotal and provides the research community
with a more stable, reliable alternative to VirusTotal.

The contributions of this paper, therefore, are:

  * •

The dynamicity of VirusTotal is common knowledge within the research community
and has been mentioned in previous research without, to the best of our
knowledge, being adequately discussed. In this paper, we reveal the details of
such dynamicity and how it manifests, how it projects an improper image of the
performance of otherwise competent scanners, and how it undermines the
performance of threshold-based labeling strategies over time (Section III-B).

  * •

We provide the research community with actionable insights about how to use
threshold-based labeling strategies to label Android apps in a manner that
copes with VirusTotal's dynamicity and limitations (Section III-C). We
demonstrate that the optimal thresholds that yield the best labeling
accuracies change over time and, hence, advise researchers to find the current
optimal threshold(s) to use prior to labeling apps in datasets they use to
train and evaluate malware detection methods.

  * •

There are voices within the research community that calls for the replacement
of VirusTotal. However, without a clear enumeration of the shortcomings of
VirusTotal, we risk implementing alternative labeling platforms that suffer
from the same shortcomings of VirusTotal. We identified four limitations of
VirusTotal that undermine its reliability and usefulness. Those limitations
are (a) frequent inclusion and exclusion of scanners in the scan reports of
apps, (b) using inadequate versions of scanners that are designed to detect
malicious apps for other platforms, (c) refraining from frequently and
automatically reanalyzing and re-scanning apps, and (d) denying access to the
history of scan reports.

  * •

Based on the identified limitations, we provide the community in Section IV
with a blueprint of how to build alternatives to VirusTotal that mitigate the
platform's limitations.

##  II Preliminaries

To motivate the need for our paper and its line of research, in this section,
we give an example of how the dynamicity of VirusTotal impacts the labeling
accuracy of threshold-based labeling strategies that are commonly used within
the research community. This example also demonstrates the impact of
inaccurate labeling on the reliability of malware detection methods and their
results. Based on this example, we postulate research questions meant to
reveal insights about VirusTotal's dynamicity and limitations and their impact
on the labeling accuracy of threshold-based labeling strategies.

###  II-A Motivating Example

In this example we focus on ML-based detection methods given their popularity
within the academic community [15, 8, 17, 10, 18]. Researchers devise new
techniques to extract features from Android apps and use those features to
train ML models. The trained models are evaluated by assessing their abilities
to recognize the malignancy of apps not used during the training process
(i.e., out-of-sample apps), by deeming them as malicious or benign. This
process requires datasets of Android Android Package (APK) archives that
researchers often acquire from online repositories, such as _AndroZoo_ [19] or
VirusTotal itself. To simulate this process, we downloaded a random collection
of 6,172 apps developed in between 2018 and 2019 from _AndroZoo_. We refer to
this dataset as _AndroZoo_ throughout this paper.

The acquired APK archives and their corresponding apps need to be labeled
either as malicious and benign or in terms of the malware families and types
they belong to [9, 2]. Since manually analyzing thousands of apps is
infeasible, researchers usually download the scan reports of apps in their
training datasets from VirusTotal and label them according to some labeling
strategy they deem accurate. The features extracted from the downloaded apps
are used alongside their labels to train ML models. As an example of an ML-
based Android malware detection method, we utilize a detection method that is
renowned in the research community and has been used by different researchers
as a benchmark [11], namely _Drebin_ [20]. In this example, we use different
threshold-based labeling strategies that have been utilized by researchers in
the past, viz. we use thresholds that deem any given app as malicious if the
number of scanners in its VirusTotal scan report deeming it malicious (i.e.,
_positives_) is at least one scanner [7], four scanners [11, 3], ten scanners
[6], 50% of scanners (i.e., $\frac{positives}{total}\geq 50\%$) [9], and the
strategy adopted by Arp et al. in [20]111An app is deemed as malicious if at
least two out of the following ten VirusTotal scanners label it as such: AVG,
Avira (formerly AntiVir), BitDefender, ClamAV, ESET-NOD32, F-Secure,
Kaspersky, McAfee, Panda, and Sophos.. We refer to those strategies as
vt$\geq$1, vt$\geq$4, vt$\geq$10, vt$\geq$50%, and drebin, respectively.

To test the ability of the trained _Drebin_ classifiers to classify out-of-
sample apps accurately, we use two small datasets that we refer to as _Hand-
Labeled_ 222http://tiny.cc/95bhaz and _Hand-Labeled 2019_
333http://tiny.cc/a7bhaz. Both datasets comprise 100 Android apps that were
downloaded from _AndroZoo_ and _manually analyzed and labeled_ , to acquire
reliable ground truth. The primary difference between both datasets is that
apps in the latter were developed in 2019. Furthermore, we ensured that apps
in both datasets do not overlap with apps in the _AndroZoo_ dataset.

2019-07-052019-07-192019-08-022019-08-162019-08-302019-09-132019-09-272019-10-112019-10-252019-11-08$\displaystyle
0.0$$\displaystyle 0.1$$\displaystyle 0.2$$\displaystyle 0.3$$\displaystyle
0.4$$\displaystyle 0.5$$\displaystyle 0.6$$\displaystyle 0.7$$\displaystyle
0.8$$\displaystyle 0.9$$\displaystyle 1.0$MCC ScoresScan
datesvt$\displaystyle\geq$1vt$\displaystyle\geq$4vt$\displaystyle\geq$10vt$\displaystyle\geq$50%drebin

(a) _Hand-Labeled_

2019-07-052019-07-192019-08-022019-08-162019-08-302019-09-132019-09-272019-10-112019-10-252019-11-08$\displaystyle
0.0$$\displaystyle 0.1$$\displaystyle 0.2$$\displaystyle 0.3$$\displaystyle
0.4$$\displaystyle 0.5$$\displaystyle 0.6$$\displaystyle 0.7$$\displaystyle
0.8$$\displaystyle 0.9$$\displaystyle 1.0$MCC ScoresScan
datesvt$\displaystyle\geq$1vt$\displaystyle\geq$4vt$\displaystyle\geq$10vt$\displaystyle\geq$50%drebin

(b) _Hand-Labeled 2019_

Figure 1: The MCC scores achieved by the _Drebin_ classifiers labeled using
different threshold-/ML-based labeling strategies against the _Hand-Labeled_
and _Hand-Labeled 2019_ dataset between July 5th, 2019 and November 8th, 2019.

In Figure 1, we plot the classification accuracy of the _Drebin_ classifiers
whose feature vectors were labeled using different threshold-based labeling
strategies over a period of four months. Each point on the X-axis refers to a
point in time in which we re-scanned all apps in the _AndroZoo_ , _Hand-
Labeled_ , and _Hand-Labeled 2019_ datasets on VirusTotal, downloaded their
up-to-date scan reports, re-trained the _Drebin_ classifiers using all the
different labeling strategies, and re-tested the trained classifiers. We use
the MCC score [21] to represent the classification accuracy of the _Drebin_
classifiers instead of conventional metrics, such as accuracy [22], that are
unable to capture or penalize bias towards certain classes in imbalanced
datasets. The MCC values range from -1 (i.e., all apps were misclassified) to
1 (i.e., perfect classification), with the value of 0 indicating a
classification ability similar to random classification.

We are not concerned with the absolute performance of the _Drebin_
classifiers. With this example, we wish to demonstrate two issues. Firstly,
despite all being utilized within previous research efforts, it appears that
some labeling strategies (e.g., vt$\geq$4), contribute to training _Drebin_
classifiers that generalize better to out-of-sample Android apps than other
strategies. That is using the exact same feature set and ML algorithm,
different labeling strategies significantly alters the performance of the same
detection method, which leads to two different answers to the question: _Is
the _Drebin_ methods a reliable Android malware detection method?_ A
researcher that opts to use the vt$\geq$50% labeling strategy will deem the
_Drebin_ method as unreliable and dismiss it, whereas one that uses the
vt$\geq$4 strategy might deem it as potentially reliable and continues to
refine it. In general, having multiple perspectives on the reliability of a
detection method might either force researchers to dismiss promising methods
that underperform during the evaluation, or adopt ones that are mediocre yet
perform well during the evaluation phase. Secondly, the MCC scores of the
_Drebin_ classifiers appear to fluctuate from one scan date to another,
although they are merely two weeks apart. So, depending on the scan date of
the VirusTotal scan reports used to label apps in the training and test
datasets, researchers might get different classification accuracies from their
detection methods. For example, during the period between September 13th,
2019, and October 11th, 2019, using the labeling strategy vt$\geq$1 led to
training _Drebin_ classifiers that performed better than other labeling
strategies on the _Hand-Labeled_ dataset.

###  II-B Research Questions

In the previous example, we learned that some threshold-based labeling
strategies contribute to training ML models that can classify out-of-sample
Android apps more accurately, despite the fact that all of the strategies that
we used in the example were devised by researchers and utilized within the
literature. Furthermore, we found that the labeling accuracy of most labeling
strategies seems to fluctuate over time, due to some unknown aspect of
VirusTotal's dynamicity. This fluctuation means that threshold-based labeling
strategies cannot be permanently and universally utilized to label Android
apps and train ML-based detection methods. In this paper, as mentioned in
Section I, we aim to provide the research community with actionable insights
about the impact of VirusTotal's dynamicity on the labeling strategies they
utilize to label Android apps used to train and evaluate their malware
detection methods, and how this dynamicity might impact the reliability of
their methods. We provide those insights by (1) identifying the aspects of
VirusTotal's dynamicity and how it manifests, (2) finding methods to
workaround such dynamicity given the lack of better alternative platforms to
VirusTotal and the infeasibility of manually-analyzing apps, (3) pinpointing
the limitations of VirusTotal that cannot be mitigated, and finally (4) using
the identified limitations to sketch a blueprint for a more reliable
alternative platform. We attempt to address these issues by answering the
following research questions:

  * RQ1

How does VirusTotal's dynamicity impact the performance of threshold-based
labeling strategies?

  * RQ2

Which labeling strategy should be used to label apps based on their VirusTotal
scan reports accurately?

  * RQ3

What are the limitations of VirusTotal and how can they be addressed?

##  III Threshold-Based Labeling Strategies

###  III-A Accuracy of Threshold-Based Labeling Strategies

In this section, we discuss the ability of different threshold-based labeling
strategies to accurately label apps in our test datasets, namely _Hand-
Labeled_ and _Hand-Labeled 2019_. Using the scan reports of apps in the
aforementioned datasets downloaded at different points in time, we label the
apps use different labeling strategies and compare them against the ground
truth we generated by manually analyzing those apps. In addition to the
labeling strategies we used in Section II-A, we use all thresholds between one
and ten scanners and the threshold of 25% of scanners.

2019-07-052019-07-192019-08-022019-08-162019-08-302019-09-132019-09-272019-10-112019-10-252019-11-08Scan
dates$\displaystyle 0.0$$\displaystyle 0.1$$\displaystyle 0.2$$\displaystyle
0.3$$\displaystyle 0.4$$\displaystyle 0.5$$\displaystyle 0.6$$\displaystyle
0.7$$\displaystyle 0.8$$\displaystyle 0.9$$\displaystyle 1.0$MCC
Scoresvt$\displaystyle\geq$1vt$\displaystyle\geq$2vt$\displaystyle\geq$3vt$\displaystyle\geq$4vt$\displaystyle\geq$5vt$\displaystyle\geq$6vt$\displaystyle\geq$7vt$\displaystyle\geq$8vt$\displaystyle\geq$9vt$\displaystyle\geq$10vt$\displaystyle\geq$25%vt$\displaystyle\geq$50%drebin

(a) _Hand-Labeled_

2019-07-052019-07-192019-08-022019-08-162019-08-302019-09-132019-09-272019-10-112019-10-252019-11-08Scan
dates$\displaystyle 0.0$$\displaystyle 0.1$$\displaystyle 0.2$$\displaystyle
0.3$$\displaystyle 0.4$$\displaystyle 0.5$$\displaystyle 0.6$$\displaystyle
0.7$$\displaystyle 0.8$$\displaystyle 0.9$$\displaystyle 1.0$MCC
Scoresvt$\displaystyle\geq$1vt$\displaystyle\geq$2vt$\displaystyle\geq$3vt$\displaystyle\geq$4vt$\displaystyle\geq$5vt$\displaystyle\geq$6vt$\displaystyle\geq$7vt$\displaystyle\geq$8vt$\displaystyle\geq$9vt$\displaystyle\geq$10vt$\displaystyle\geq$25%vt$\displaystyle\geq$50%drebin

(b) _Hand-Labeled 2019_

Figure 2: The labeling accuracy of different threshold-based labeling
strategies against apps in _Hand-Labeled_ and _Hand-Labeled 2019_ datasets
based on their VirusTotal scan reports downloaded between July 5th, 2019 and
November 8th, 2019. Accuracy is calculated in terms of the MCC of each
labeling strategy.

In Figure 2, we plot the performance of each labeling strategy on the _Hand-
Labeled_ and _Hand-Labeled 2019_ datasets between July 5th, 2019 and November
8th, 2019 in terms of the MCC score. We attempted to gain access to the
VirusTotal scan reports of apps in both datasets that pre-dates July.
Unfortunately, access to such reports is not available under academic licenses
and requires the purchase of costly commercial ones, which we consider the
first limitation of VirusTotal.

VirusTotal Limitation 1 VirusTotal does not grant access to academic
researchers to the history of scan reports of apps previously added and
scanned on the platform, even if such apps were added by the academic
community itself. In fact, we are not sure whether VirusTotal keeps or
discards the current scan reports of apps prior to rescanning apps.

One can notice that some threshold-based labeling strategies are more accurate
than others over time. Starting with the performance of vt$\geq$1, while the
labeling strategy managed to achieve a decent MCC score on apps in the _Hand-
Labeled_ dataset, its performance noticeably decreased against apps in the
newer _Hand-Labeled 2019_ dataset. Low threshold values, such as one or two
scanners, might result in false positives, especially against new apps whose
VirusTotal scan reports are not mature enough. In most cases, if an app has
one or two VirusTotal scanners deeming it as malicious, it is a case of a
subjective definition of malignancy. For example, we noticed that some
scanners such as Tencent consistently label any apps (e.g.,
ed23237e34ff47580a99ac70f35e84b32c05ab1d), that utilize _App Inventor_ 444App
Inventor is a visual programming environment maintained by MIT that enables
non-technical users to develop apps for Android [23]. as malicious apps
belonging to the A.gray.inventor.a malware family.

As for (vt$\geq$50%), pushing the threshold that high might prevent recently-
developed malicious apps and apps that belong to ambiguous malware types
(e.g., Adware) from being labeled as malicious, resulting in a high number of
false negatives. Similar to vt$\geq$1, the older the app and its VirusTotal
scan report, the better the performance of vt$\geq$50% and the newer the app,
the worse the performance, especially since the malicious apps were not deemed
labeled by enough VirusTotal scanners to make the 50% mark required by the
strategy to deem them as malicious successfully.

Another aspect of how the age of apps and, in turn, the maturity of their
VirusTotal scan reports impacts the performance of different threshold-based
labeling strategies can be seen in the proximity of different MCC lines in
Figure 2. In particular, in 2(a), the lines of almost all threshold-based
labeling strategies are close to one another and exhibit a relatively steady
performance (i.e., the performance does not noticeably fluctuate). However,
the MCC lines in 2(b) are more distributed across the figure and exhibit more
fluctuations in performance. For example, on the _Hand-Labeled 2019_ dataset,
the MCC score of drebin sharply decreased from a little above 0.3 on August
30th, 2019 to almost 0.0 on September 13th, 2019 only to sharply increase to
around 0.45 two weeks later. The reason behind the proximity in the case of
apps in the _Hand-Labeled_ dataset is that their _positives_ values are high
enough to accommodate threshold values up to at least 15 scanners, which is
represented by the vt$\geq$25% labeling strategy. The novelty of malicious
apps in the _Hand-Labeled 2019_ means that their _positives_ values are much
lower in comparison, which prevents thresholds higher than six scanners from
achieving high MCC scores.

2019-07-052019-07-192019-08-022019-08-162019-08-302019-09-132019-09-272019-10-112019-10-252019-11-08Scan
dates$\displaystyle 0$$\displaystyle 5$$\displaystyle 10$$\displaystyle
15$$\displaystyle 20$Median _positives_ _Hand-Labeled_ malicious _Hand-Labeled
2019_ malicious _Hand-Labeled_ benign _Hand-Labeled_ benign

(a) Mean _positives_

2019-07-052019-07-192019-08-022019-08-162019-08-302019-09-132019-09-272019-10-112019-10-252019-11-08Scan
dates$\displaystyle 0$$\displaystyle 5$$\displaystyle 10$$\displaystyle
15$$\displaystyle 20$Std. Deviation _positives_ _Hand-Labeled_ malicious
_Hand-Labeled 2019_ malicious _Hand-Labeled_ benign _Hand-Labeled_ benign

(b) Std. Deviation _positives_

2019-07-052019-07-192019-08-022019-08-162019-08-302019-09-132019-09-272019-10-112019-10-252019-11-08Scan
dates$\displaystyle 0$$\displaystyle 5$$\displaystyle 10$$\displaystyle
15$$\displaystyle 20$Median _positives_ _Hand-Labeled_ malicious _Hand-Labeled
2019_ malicious _Hand-Labeled_ benign _Hand-Labeled_ benign

(c) Median _positives_

Figure 3: The mean, standard deviation, and median of the _positives_
attributed found in scan reports of apps in the _Hand-Labeled_ and _Hand-
Labeled 2019_ datasets between July 5th, 2019 and November 8th, 2019.

To corroborate this argument, we calculated the mean, median, and standard
deviation of the _positives_ attribute for apps in both datasets over the same
period of time. The results in Figure 3 show that the _positives_ attribute of
malicious apps in the _Hand-Labeled_ dataset stays within the range of 15 to
20. Even with a standard deviation of ten scanners, the range of scanners
needed to label malicious apps in this dataset correctly remains between 7 and
20 scanners. As for the malicious apps in the _Hand-Labeled 2019_ dataset,
their _positives_ values have mean and median values around seven, ranging
between 2.89 scanners and 10.91 scanners. The benign apps in both datasets
have mean and median values that are almost zero with a negligible standard
deviation of at most one scanner. So, any threshold values above three are
guaranteed to avoid false positives resulting from deeming benign apps as
malicious. The aforementioned values of the _positives_ attribute allows the
labeling strategies using thresholds between three and six VirusTotal scanners
(i.e., vt$\geq$3, vt$\geq$4, vt$\geq$5, and vt$\geq$6), to outperform all
other threshold-based labeling strategies on both datasets in terms of the
MCC.

What we can conclude from this measurement is that the thresholds that result
in decent labeling accuracy differ from one dataset to another depending on
one the age of apps in the dataset and the maturity of their VirusTotal scan
reports. To generalize those thresholds to multiple datasets, one must possess
a diverse dataset that includes Android apps of different ages, which is a
concept we discuss in detail in Section III-C.

###  III-B Sensitivity to VirusTotal's Dynamicity

In the previous section, despite finding that a range of thresholds between
three and six yields the best MCC scores on the _Hand-Labeled_ and _Hand-
Labeled 2019_ datasets, we noticed that the performance of labeling strategies
utilizing these thresholds fluctuates at different points in time especially
against the latter dataset. This fluctuation is largely attributed to
VirusTotal's dynamicity and the immaturity of the scan reports of recently-
developed Android apps. In this section, we analyze (a) whether the reason
behind such fluctuation is indeed the dynamicity of VirusTotal, and (b) the
aspects of such dynamicity that cause this fluctuation. In this analysis, we
focus on the performance of threshold-based labeling strategies on the _Hand-
Labeled 2019_ dataset on two dates, namely September 27th, 2019 and November
8th, 2019.

We found that the labeling accuracy of threshold-based labeling strategies
using thresholds between three and six on benign apps in the _Hand-Labeled
2019_ dataset did not change, according to their specificity scores
($\frac{TN}{N}$). Focusing on their performance on the malicious apps, we
found that they respectively had recall scores ($\frac{TP}{P}$) of 0.7, 0.6,
and 0.6, 0.6 on November 8th, 2019 instead of 0.8, 0.8, 0.8, and 0.7 on
September 27th, 2019. Since the total number of malicious apps in this dataset
is ten, we can investigate the differences in _positives_ values in their scan
reports and the different VirusTotal scanners that deemed them malicious on
both dates. In Table I, we detail the change in the _positives_ values in
terms of the VirusTotal scanners that deemed the apps as malicious which (a)
were added to the scan reports on November 8th, 2019, (b) were removed from
the September 27th, 2019 scan reports, and (c) changed their verdicts between
both dates.

TABLE I: The evolution of _positives_ for apps in the _Hand-Labeled 2019_
dataset that we deemed malicious after manual analysis and a detailed view of
the VirusTotal scanners that were added/removed or changed their verdicts
between September 27th, 2019 and November 8th, 2019 and how that affected the
performance of vt$\geq$3, vt$\geq$4, vt$\geq$5, and vt$\geq$6. The check mark
(✓) depicts whether the threshold-based labeling strategy managed to detect
the malicious app on November 8th, 2019. App's SHA1 Hash _positives_
(September 27th, 2019) _positives_ (November 8th, 2019) Added Positives
Removed Positives Flipped to Positive Flipped to Negative vt$\geq$3 vt$\geq$4
vt$\geq$5 vt$\geq$6 8a9... 0 0 – – – – bd9... 5 3 – ESET-NOD32 Fortinet Ikarus
Cyren – ✓ 765... 7 7 – – Zillya Trustlook ✓ ✓ ✓ ✓ 5be... 12 13 Ikarus – – – ✓
✓ ✓ ✓ 6da... 7 6 – – – Trustlook ✓ ✓ ✓ ✓ c70... 13 13 – – Symantec Zoner ✓ ✓ ✓
✓ b9f... 10 12 Ikarus – AegisLab K7GW McAfee ✓ ✓ ✓ ✓ a0a... 8 11 Fortinet –
Zillya Zoner – ✓ ✓ ✓ ✓ 90e... 6 1 – ESET-NOD32 Fortinet Ikarus Yandex – Cyren
0d5... 1 1 – – – – Recall 0.70 0.60 0.60 0.60 MCC 0.82 $\approx$0.76
$\approx$0.76 $\approx$ 0.76

Three apps out of ten did not encounter any change in their _positives_
values. Nonetheless, one5557658f70ae6acccfa9f3e900f8ae689603cc19d0b of these
apps maintained the same value of _positives_ because two VirusTotal scanners
changed their verdicts, namely Zillya changed its verdict from benign to
malicious and Trustlook changed its verdict vice versa. However, VirusTotal
contributed to altering the performance of all labeling strategies by removing
scanners that correctly deemed two apps malicious from their scan reports. In
particular, the scanners ESET-NOD32, Fortinet, and Ikarus were removed from
one666bd97c85d38bd5bfc5e29b05b1a3a81b12949065a app's scan report, effectively
reducing its _positives_ value from five on September 27th, 2019 to only three
on November 8th, 2019; this prevented the vt$\geq$4, vt$\geq$5, and vt$\geq$6
labeling strategies from correctly deeming the app as malicious. The same
scanners along with Yandex were not included in the
second77790e6ac481fdd497f152234f1cd5bec6d40f50037 app's November 8th, 2019
scan report, which brought the value of _positives_ from six to one, putting
the app beyond the reach of all of the three aforementioned strategies.

Unfortunately, we cannot identify the reasons behind VirusTotal's decision to
alter the set of scanners it includes in an app's scan report. It is a
puzzling fact that VirusTotal added the Ikarus and Fortinet scanners to the
scan reports of some apps, whilst removing them from the reports of others.
The more confusing fact is that the removed scanners contain the ESET-NOD32
scanner, which we found to have correctly labeled the apps as malicious on
September 27th, 2019. This seemingly haphazard inclusion and exclusion of
scanners within periods as small as two months contribute to the frequent
change of the currently optimal range of thresholds. While such change does
not impact older apps, such as the ones in the _Hand-Labeled_ dataset, its
impact is noticeable on newer apps, such as the ones in the _Hand-Labeled
2019_ dataset. With this set of measurements, we unveil the first limitation
of VirusTotal, which has a direct impact on the performance of threshold-based
labeling strategies, viz.:

VirusTotal Limitation 2 VirusTotal changes the set of scanners it includes in
the scan reports of apps over time by including and excluding the verdicts of
scanners regardless of the quality of those verdicts.

During this analysis, we noticed that the VirusTotal version of some of the
renowned scanners, such as BitDefender and Panda, fail to recognize the
malignancy of any of the malicious apps in either dataset. Both of those
scanners continue to be given good reviews by users on the Google Play
marketplace and, more importantly, on platforms that assess the effectiveness
of antiviral software such as _AV-Test_ [24]. VirusTotal states that the
versions of scanners it uses "_may differ from commercial off-the-shelf
products. The [antiviral software] company decides the particular settings
with which the engine should run in VirusTotal_ " [5]. In fact, we found that
the version used by VirusTotal's for BitDefender, for instance, is 7.2,
whereas the versions available on Google Play have codes between 3.3 and 3.6.
The 7.2 version of BitDefender corresponds to a free edition version developed
for Windows-based malware that targets older versions of Windows, such as
Windows XP [25] and, hence, is inadequate to use to detect Android malware.
The positive reputation that BitDefender has in the market suggests that using
its adequate version (i.e., the one that is designed to detect Android
malware), would yield a detection performance better than the version on
VirusTotal. To verify this hypothesis, we downloaded and installed the latest
version of the BitDefender scanner from the Google Play marketplace, installed
it on an Android Virtual Device (AVD), and used it to scan the malicious apps
in both the _Hand-Labeled_ and _Hand-Labeled 2019_ datasets. We also
downloaded ten apps randomly sampled from the _AMD_ [9] dataset to test
BitDefender's accuracy. Unlike the results obtained from VirusTotal that the
scanner detected none of those malicious apps, we found that BitDefender
detects 56.5% of the malicious apps in the _Hand-Labeled_ dataset, 20% of
those in the _Hand-Labeled 2019_ dataset, and 70% of those in the sampled
_AMD_ dataset. Figuring out the reason why antiviral software companies opt to
provide VirusTotal with older, inadequate versions of their scanners is not in
the scope of this paper and, in fact, impossible to answer on behalf of
antiviral software companies. However, it leads us to identify the third
limitation of VirusTotal:

VirusTotal Limitation 3 VirusTotal may replace the versions of scanners with
inadequate ones that are not designed to detect Android malware presumably
based on the request of the scanner's vendor or managing firm.

###  III-C Finding the Optimal Threshold

In the previous section, we found that VirusTotal changes the set and versions
of scanners it includes in the scan reports of apps. This change impacts the
long term labeling accuracy of threshold-based labeling strategies. For
example, while the dynamicity of VirusTotal has caused the MCC score of
vt$\geq$4 to decrease against the _Hand-Labeled 2019_ dataset from 0.89 to
0.76 (i.e., a decrease of 14.61%), it did not have an impact on the MCC scores
of vt$\geq$2 yet caused the scores of vt$\geq$1 to decrease. Effectively, the
previously-discussed aspects of VirusTotal's dynamicity cause threshold-based
labeling strategies to trade places in terms of the most accurate ones. In
other words, at any given moment in time, a (different) subset of threshold-
based labeling strategies will depict the most accurate labeling strategies
(i.e., optimal thresholds). So, before labeling apps in their training and
evaluation datasets, researchers must identify the most accurate threshold(s)
at that particular point in time. Given a reference set of Android apps whose
ground truth is known, one straightforward method to identify the currently
accurate thresholds is to download the apps' latest VirusTotal scan reports,
compare the labeling accuracy of all thresholds between one and 60 (i.e.,
average total number of scanners), and choose the thresholds that yield the
best scores. In this section, we investigate the feasibility of this brute
force approach to identify the optimal thresholds of VirusTotal scanners at
any point in time.

Algorithm 1 An algorithm to find the current optimal threshold of VirusTotal
scanners to use in labeling Android apps.

1:procedure FindCurrentOptimalThreshold($A,\gamma$)

2:  _tmpResults_ = $\\{\\}$

3: for all $\alpha\in A$ do

4:  _response_ = VirusTotal.rescanApp($\alpha$)

5: if  _response_ == True then

6:  _report_ = VirusTotal.downloadReport($\alpha$)

7: if  _report_ != Null then

8:  _positives_ α = _report_["positives"]

9: for all $\sigma\in\\{1,2,3,...,60\\}$ do

10: if  _positives_ ${}_{\alpha}\geq\sigma$ then

11:  _label_ α = malicious

12: else

13:  _label_ α = benign

14:  _tmpResults_[vt$\geq\sigma$].append(_label_ α)

15:

16:  _bestThreshold_ = ""

17:  _bestScore_ = 0.0

18: for all $\sigma\in\\{1,2,3,...,60\\}$ do

19:  _currentScore_ = calculateScore(_tmpResults_[vt$\geq\sigma$], $\gamma$)

20: if _currentScore_ $\geq$ _bestScore_ then

21:  _bestScore_ = _currentScore_

22:  _bestThreshold_ = vt$\geq\sigma$ return _bestThreshold_ , _bestScore_

Algorithm 1 depicts a simple algorithm to find the current optimal threshold
of VirusTotal scanners that yields the most accurate labels. To assess the
quality of labels given by different threshold-based labeling strategies, this
algorithm requires the presence of a dataset ($A$) of pre-labeled Android
malicious and benign apps. As mentioned earlier in this paper, the most
reliable ground truth ($\gamma$) can be generated using manual analysis and
labeling of apps. Without such a reliable ground truth ($\gamma$) that acts as
a reference to compare against, one has to choose a subjective threshold
($\sigma$) that one believes represents the nature of apps in ($A$) and use it
as ground truth ($\gamma_{\sigma}$) If so, the only threshold that would
generate labels mimicking ($\gamma_{\sigma}$) would be ($\sigma$) itself.
Effectively, we end up with the same problem we are attempting to avoid,
namely that of choosing subjective thresholds based on personal views.

Relying on manual analysis already introduces infeasibility to the algorithm.
However, assuming the existence of pre-labeled apps, another problem arises.
As discussed earlier, the immaturity of newly-developed Android malware and
the dynamicity of VirusTotal lowers the values of the _positives_ attribute in
the scan reports of those apps which, in turn, lowers the thresholds needed to
label them accurately. Without access to newly-developed Android malware,
researchers risk choosing thresholds based on the scan reports of old
malicious apps, which are usually higher than the thresholds required to
detect new malware. For example, in Figure 2, if a researcher only has access
to the _Hand-Labeled_ dataset, on October 11th, 2019, they might opt to use
the drebin labeling strategy because it exhibits stable performance of high
MCC scores. However, this labeling strategy will perform much worse on newer
apps in the _Hand-Labeled 2019_ dataset (i.e., it does not generalize to newer
malicious apps). Consequently, researchers adopting this brute force approach
to finding the currently optimal thresholds need to continuously update their
reference datasets with newly-developed and discovered Android (malicious)
apps.

If the reference dataset ($A$) satisfies the previous condition, identifying
the currently optimal threshold can be performed as follows. For each app in
the dataset ($\alpha\in A$), the latest scan report of ($\alpha$) needs to be
acquired. Firstly, the user has to issue a rescan request to VirusTotal, which
takes around four minutes to complete (line 5) As discussed earlier, this
request can be issued using the platform's web interface or using the
Application Programming Interface (API) interface. In general, under the
academic license, a total of 20K requests can be issued per day. So, depending
on the size of the reference dataset ($A$), the process of rescanning all apps
might take several days or even months. Furthermore, we recently were
forbidden from issuing this type of request using our academic license. As of
the date of writing this paper, we are unaware of whether VirusTotal prevents
academic licenses from issuing this type of requests, or whether we are
encountering an individual technical difficulty. We consider the decision of
VirusTotal not to automatically and regularly rescan apps as another
limitation of the platform:

VirusTotal Limitation 4 VirusTotal does not rescan the apps it possesses on a
regular basis and delegates this task to manual requests issued by its users.
One direct consequence of this decision is prolonging the process of acquiring
up-to-date scan reports of apps.

In line 7, after the rescan requests are completed, researchers need to
download the up-to-date scan reports from VirusTotal. Similar to the rescan
API requests, download requests are limited to 20K requests per day, which
might add a few more days to the process. Between lines 8 and 23, the process
becomes straightforward. Using thresholds ($\sigma$) between one and 60, the
labels of apps in ($A$) are calculated and stored in a temporary structure
under the key vt$\geq\sigma$ (line 15). The stored labels are then compared
against the ground truth ($\gamma$), and a score is calculated, say MCC. The
threshold-based based strategy that yields the best score is returned to the
user.

##  IV An Alternative to VirusTotal

In this section, we discuss how the limitations we discussed above can be
addressed upon building alternative platforms. We do this by describing the
architecture of a hypothetical new platform, called _Eleda_ 888Eleda is one of
the mirror twins in Sharon Shinn’s novel _The Truth-Teller’s Tale_ , who is a
truth-teller incapable of telling lies, earning her the society’s
trustworthiness. With Eleda’s proposed design, we aspire to provide the
research community with an alternative to VirusTotal that is more stable and
reliable., that is designed to mitigate VirusTotal's limitations. An overview
of _Eleda_ 's modules and operations can be seen in Figure 4.

Data Acquisition. The first operation of _Eleda_ is to acquire APK archives of
Android malicious and benign apps to scan and analyze (step (1)). _AndroZoo_
[19] automates the process of crawling different app marketplaces and
continuously downloads their APK archives. Access to such a platform is
granted to researchers via an API key that can be used to download apps using
cURL. Using this module, _Eleda_ can frequently query _AndroZoo_ for newly-
crawled and downloaded apps, which is indicated using the looped arrow.

Figure 4: An overview of the modules and operations of the proposed VirusTotal
replacement, _Eleda_.

Scanner Acquisition and Update. This module is responsible for analyzing and
scanning the acquired apps. In step (2), the module retrieves the names of
antiviral scanners and queries app marketplaces, such as Google Play, for
their latest versions. This list can be manually populated at first to include
the list of VirusTotal scanners that are designed to detect Android malware
and are available on Google Play. As of March 2020, 38 ($\approx$63%) out of
around 60 scanners on VirusTotal are available on Google Play. After
downloading the latest version of each scanner in the list, _Eleda_ updates an
AVD that is used to scan APK archives (step (3)). The process of acquiring and
updating new versions of antiviral scanners from Google Play–and possibly
third-party Android app marketplaces–is meant to mitigate the third limitation
of VirusTotal, namely using inadequate scanners and scanner versions not
designed to scan Android apps. Moreover, using a pre-populated list of
scanners is meant to keep the set of antiviral scanners used to scan APK
archives constant (i.e., mitigating the second limitation).

App Scanning. To address the fourth limitation of VirusTotal that it only re-
scans Android apps upon request, in step (4), _Eleda_ 's App Scanning module
retrieves the set of APK archives available in the platform's app repository,
scans them using the AVD s set up by the Scanner Acquisition and Update
module, builds the latest scan reports of those apps, and stores them in
another repository (e.g., in JavaScript Object Notation (JSON) format). In
order to make the transition from VirusTotal to _Eleda_ seamless, the scan
reports can contain the same information contained in VirusTotal scan reports.
For example, static information about the app components can also be extracted
using analysis tools, such as _Androguard_ [26], and the API calls issued by
the app during runtime can also be monitored and recorded [27]. The frequency
of the re-scan operation can be set by the users of _Eleda_. Given that the
platform is expected to store a large number of APK archives, it need not re-
scan all apps at the same time. Instead, each app can be scanned every
constant interval (e.g., two weeks), starting from its initial acquisition
date.

User Interaction. The last operation of _Eleda_ is user interaction. In step
(6), the platform receives a query from a remote user in the format of a hash
of an Android app's APK or the archive itself. If the app is not already in
the platform's app repository, the App Scanning module can add the app's APK
archive to the repository. The APK archive is then scanned using the
platform's AVD s, and its scan report can be displayed to the user. _Eleda_
can mimic the design of VirusTotal by offering users to interact with the
platform using a web-based interface or using API-requests. However, to
address VirusTotal's first limitation, _Eleda_ can provide the users with all
scan reports of the queried app.

##  V Discussion

In this section, we discuss the insights we gained from our experiments, the
limitations of our work, and how we (plan to) address them.

Aspects of VirusTotal's dynamicity. In answering RQ1, we attempted to (a)
reveal the aspects of VirusTotal's dynamicity that are not clearly documented
within the literature, and (b) study the impact of such dynamicity on the
labeling performance of threshold-based labeling strategies. In Section III-A,
we found that VirusTotal regularly manipulates the set of scanners it includes
in the scan reports of apps by adding/removing scanners from such reports,
including ones that correctly label the apps. This leads to changing the
number of scanners that deem an app as malicious (i.e., _positives_), which is
what threshold-based labeling strategies hinge on to discern an app's
malignancy. While this change has a negligible impact on the labels of benign
apps, we found that malicious apps–especially those recently developed–suffer
the most from this frequent manipulation of scanners. So, to answer RQ1, the
regular manipulation of scan reports means that thresholds that proved to be
accurate at one point in time cannot be used over an extended period.

Optimally using VirusTotal to label apps. Given the aforementioned dynamicity
of VirusTotal, the concern of RQ2 is to find a workaround for the platform's
dynamicity to be able to label (Android) apps based on their scan reports. We
found in Section III-A that, at any point in time, there are thresholds that
provide the most accurate threshold-based labeling strategies. So, instead of
relying on a fixed threshold to label apps (e.g., four scanners), for
prolonged periods, researchers should identify the _current optimal_
thresholds based on the latest scan reports of the apps they wish to label
(i.e., after re-scanning the apps on VirusTotal). In Section III-C, we propose
an algorithm to identify such thresholds with the help of a diverse, pre-
labeled dataset of apps, and detail the possible challenges that might face
researchers adopting this algorithm.

VirusTotal's Limitations. The research community has long aspired to replace
VirusTotal with a more reliable alternative. However, without pinpointing the
shortcomings of VirusTotal, we risk constructing a platform that suffers from
the same limitations. So, it is imperative to identify those limitations,
which is the concern of RQ3. In this paper, we identified four limitations of
VirusTotal that jeopardizes its usefulness. First, the platform does not grant
access to the history of scans, effectively preventing researchers from
studying the performance of scanners over extended periods of time. Second,
the platform changes the set of scanners it uses to scan the same apps over
time, which undermines the sustainability of threshold-based labeling
strategies. Third, the platform uses scanners or versions of scanners that are
not suitable to detect Android malware. Fourth, the platform does not
automatically re-scan apps and relies on manually re-scanning apps either via
its web-interface or via remote API requests. In Section IV, we give an
example of how such limitations can be mitigated upon constructing an
alternative platform.

###  V-A Limitations and Threats to Validity

Internal validity is concerned with actions or factors that could influence
our results. In this paper, we relied on ground truth for the _Hand-Labeled_
and _Hand-Labeled 2019_ datasets that were generated after manually analyzing
their apps. The intrinsic subjectivity of manually deeming apps as malicious
can threaten the reliability of this ground truth. In the process of manually
labeling apps in those two datasets, we complemented our process of static and
dynamic analysis of apps with consulting the VirusTotal scan reports of those
apps. We opted to ignore the verdicts of VirusTotal's scanners vis-à-vis the
verdicts of a few apps that we clearly observed their malicious behavior
despite being labeled as benign by all scanners. In fact, it is not uncommon
for antiviral scanners to be oblivious to the malignancy of malicious apps.
For example, over the past three years, we tracked the verdicts given by
VirusTotal scanners to a repackaged, malicious
version999aa0d0f82c0a84b8dfc4ecda89a83f171cf675a9a of the K9 Mail open source
app [28] that has been developed by one of our students during a practical
course. Despite being a malicious app of type Ransom, the scanners continued
to unanimously deem the app as benign since February 8th, 2017, even after
analyzing and re-scanning the app. Another example is an
app10101066c16d79db25dc9d602617dae0485fa5ae6e54b2: A calculator app grafted
with a logic-based trigger that deletes user contacts only if the result of
the performed arithmetic operation is 50. that we repackaged three years ago;
the app continued to be labeled as benign by all scanners until only K7GW
recognized the app's malignancy in July 2019 and labeled it as a Trojan.

External validity focuses on the possibility of generalizing our results.
There are two main aspects of generalization in our case. Firstly, our results
are confined to two small datasets of Android apps. We chose to limit the size
of our datasets to ensure rigorous analysis of their codebases and runtime
behaviors in order to generate as reliable ground truth as possible. To
compensate for their small size, we randomly downloaded 100 apps from
_AndroZoo_ that should act as a random sample of Android apps without bias
towards malignancy, category, or marketplace. The second aspect of
generalization is the confinement to Android, which we adopted as a case study
in this paper. So, are our results transferrable to other domains (e.g.,
Windows)? Concerning the first and fourth limitations of VirusTotal, we found
that the platform implements the same policies regardless of the domain
researchers wish to focus on. As for the second and third limitations, using
ten randomly sampled Windows-based malicious apps that we acquired from
VirusTotal (e.g., 1111110b3beb60b80bba63154ab7491046528be0054e10), we found
that VirusTotal uses the same versions of scanners for apps belonging to
different domains. We also found that VirusTotal changes the set of scanners
it includes in the apps' scan reports at two different scan dates. So, it
seems that our insights might also generalize to domains beyond Android.

Reliability validity is concerned with the possibility of replicating our
findings. To replicate our results using the same datasets, we offer the scan
reports of apps in the _Hand-Labeled_ and _Hand-Labeled 2019_ datasets, the
_Drebin_ feature vectors used in Section II-A, and the tools we used to carry
out our measurements and experiments to the research community.

##  VI Related Work

We can categorize the insights and results of this paper into two categories.
Firstly, we study VirusTotal and its dynamicity, detail how this dynamicity
manifests itself, and demonstrate how it impacts threshold-based labeling
strategies commonly used within the research community. Secondly, based on our
findings from studying VirusTotal, we attempt provide the research community
with actionable insights about how to optimally utilize VirusTotal until a
valid alternative is implemented. In this context, we surveyed the literature
to find related work that fall under the categories of (a) studying
VirusTotal, and (b) using it for accurate labeling.

Studying VirusTotal. The research community has studied different aspects of
VirusTotal and its scanners. In [29], Mohaisen et al. inspected the relative
performance of VirusTotal scanners on a small sample of manually-inspected and
labeled Windows executables. The authors introduced four criteria, called
correctness, completeness, coverage, and consistency, to assess the labeling
capabilities of VirusTotal scanners and demonstrated the danger of relying on
VirusTotal scanners that do not meet such criteria. The main objective of this
study is, therefore, to shed light on the inconsistencies among VirusTotal
scanners on a small dataset. In [13], Mohaisen and Alrawi built on their
previous study and attempted to assess the detection rate, the correctness of
reported labels, and the consistency of detection of VirusTotal scanners
according to the aforementioned four criteria. They showed that in order to
obtain complete and correct (i.e., in comparison to ground truth) labels from
VirusTotal, one needs to utilize multiple independent scanners instead of
hinging on one or a few of them. Similarly, within the domain of Android
malware, Hurier et al. studied the scan reports of VirusTotal scanners to
identify the lack of consistency in labels assigned to the same app by
different scanners and proposed metrics to quantitatively describe such
inconsistencies [2]. More recently, Peng et al. [12] showed that VirusTotal
scanners exhibit similar inconsistencies upon deeming Uniform Resource Locator
(URL) as malicious and benign. The authors also showed that some VirusTotal
scanners are more correct than others, which requires a strategy to label such
URL s that does not treat all scanners equally. While these studies revealed
inconsistencies in the verdicts given by VirusTotal, they did not delve into
the possible reasons behind such inconsistences (i.e., whether they are indeed
due to the scanners' incompetences or due to VirusTotal's dynamicity). In this
paper, we build on the insights in [2, 13, 29] to highlight the aspects of
VirusTotal's dynamicity that impact the verdicts given by scanners and how
this impacts the performance of threshold-based labeling strategies and, in
turn, the reliability of malware detection methods built on top of their
labels.

Accurate Labeling Strategies. Aware of their sensitivity to VirusTotal's
dynamicity, researchers have attempted to replace threshold-based labeling
strategies with more sophisticated labeling strategies, primarily based on ML.
In [4], Kantchelian et al. used the VirusTotal scan reports of around 280K
binaries to build two ML-based techniques to aggregate the results of multiple
scanners into a single ground-truth label for every binary. In the first
technique, Kantchelian et al. assume that the ground truth of an app (i.e.,
malicious or benign), is unknown or _hidden_ , making the problem of
estimating this ground truth is that of unsupervised learning. Furthermore,
they assumed that the verdicts of more consistent, less erratic scanners are
more likely to be correlated with the correct, hidden ground truth than more
erratic scanners. Thus, more consistent scanners should have larger weights
associated with their verdicts. To estimate those weights and, hence, devise
an unsupervised ML-based labeling strategy, the authors used an Expectation
Maximization (EM) algorithm based on a Bayesian model to estimate those
models. The second technique devised by Kantchelian et al. is a supervised one
based on regularized logistic regression. However, the authors did not
describe the nature of the features they use to train such an algorithm. To
devise an automated method to label apps based on different verdicts given by
antiviral scanners, Sachdeva et al. [30] performed measurements to determine
the most correct VirusTotal scanners using scan reports of a total of 5K
malicious and benign apps. Using this information, they assign a weight to
each scanner that they use to calculate a malignancy score for apps based on
their VirusTotal scan reports. Depending on manually-defined thresholds, the
authors use this score to assign a confidence level of Safe, Suspicious, or
Highly Suspicious to test apps. The works in [4] and [30] seem to dismiss
threshold-based labeling strategies and go for sophisticated labeling methods
that may be difficult to comprehend and utilize. In our paper, we studied the
reasons behind the fluctuating performance of this type of labeling strategies
and found that, despite their simplicity, they can be effectively utilized to
accurately label Android apps based on their VirusTotal scan reports.
Furthermore, we provide the research community with an algorithm to workaround
VirusTotal's dynamicity and yield the current optimal thresholds that would
yield accurate labels.

##  VII Conclusion

The infeasibility of manually analyzing and labeling Android apps and the lack
of more stable alternatives forces the research community to use the online
platform, VirusTotal, to label apps they use in training and evaluating
malware detection methods. Although VirusTotal is known to be dynamic and
volatile, previous research neither delved into the aspects of the platform's
dynamicity, how it impacts the threshold-based labeling process, and how to
work around it nor implemented alternatives to VirusTotal.

With a focus on Android apps, in this paper, we studied VirusTotal to identify
how its alleged dynamicity manifests itself and how it impacts threshold-based
labeling strategies that are widely-adopted within the research community.
Using our findings, we provided a method that bypasses the aspects of
VirusTotal's dynamicity to find the thresholds of scanners that would yield
the best labeling accuracies and more effective and reliable malware detection
methods. Despite this method, we realize that VirusTotal's limitations
ultimately calls for the replacement of the online platform with more reliable
alternatives. In order not to implement alternative platforms that suffer from
the same shortcomings of VirusTotal, we discussed the four limitations of
VirusTotal that we identified through our measurements and analysis and how to
avoid them in building alternative platforms.

## References

  * [1] M. Hurier, G. Suarez-Tangil, S. K. Dash, T. F. Bissyandé, Y. L. Traon, J. Klein, and L. Cavallaro, ``Euphony: Harmonious unification of cacophonous anti-virus vendor labels for android malware,'' in Proceedings of the 14th International Conference on Mining Software Repositories. IEEE Press, 2017, pp. 425–435.
  * [2] M. Hurier, K. Allix, T. F. Bissyandé, J. Klein, and Y. Le Traon, ``On the lack of consensus in anti-virus decisions: Metrics and insights on building ground truths of android malware,'' in International Conference on Detection of Intrusions and Malware, and Vulnerability Assessment. Springer, 2016, pp. 142–162.
  * [3] B. Miller, A. Kantchelian, M. C. Tschantz, S. Afroz, R. Bachwani, R. Faizullabhoy, L. Huang, V. Shankar, T. Wu, G. Yiu et al., ``Reviewer integration and performance measurement for malware detection,'' in International Conference on Detection of Intrusions and Malware, and Vulnerability Assessment. Springer, 2016, pp. 122–141.
  * [4] A. Kantchelian, M. C. Tschantz, S. Afroz, B. Miller, V. Shankar, R. Bachwani, A. D. Joseph, and J. D. Tygar, ``Better malware ground truth: Techniques for weighting anti-virus vendor labels,'' in Proceedings of the 8th ACM Workshop on Artificial Intelligence and Security. ACM, 2015, pp. 45–56.
  * [5] VirusTotal. Virustotal. [Online]. Available: http://tiny.cc/xjbb7y (2019)
  * [6] H. Wang, Z. Liu, J. Liang, N. Vallina-Rodriguez, Y. Guo, L. Li, J. Tapiador, J. Cao, and G. Xu, ``Beyond google play: A large-scale comparative study of chinese android app markets,'' in Proceedings of the Internet Measurement Conference 2018. ACM, 2018, pp. 293–307.
  * [7] L. Li, D. Li, T. F. Bissyandé, J. Klein, Y. Le Traon, D. Lo, and L. Cavallaro, ``Understanding android app piggybacking: A systematic study of malicious code grafting,'' IEEE Transactions on Information Forensics and Security, vol. 12, no. 6, 2017, pp. 1269–1284.
  * [8] G. Suarez-Tangil, S. K. Dash, M. Ahmadi, J. Kinder, G. Giacinto, and L. Cavallaro, ``Droidsieve: Fast and accurate classification of obfuscated android malware,'' in Proceedings of the Seventh ACM on Conference on Data and Application Security and Privacy. ACM, 2017, pp. 309–320.
  * [9] F. Wei, Y. Li, S. Roy, X. Ou, and W. Zhou, ``Deep ground truth analysis of current android malware,'' in International Conference on Detection of Intrusions and Malware, and Vulnerability Assessment. Springer, 2017, pp. 252–276.
  * [10] W. Yang, D. Kong, T. Xie, and C. A. Gunter, ``Malware detection in adversarial settings: Exploiting feature evolutions and confusions in android apps,'' in Proceedings of the 33rd Annual Computer Security Applications Conference. ACM, 2017, pp. 288–302.
  * [11] R. J. J. K. Feargus Pendlebury, Fabio Pierazzi and L. Cavallaro, ``Tesseract: Eliminating experimental bias in malware classification across space and time,'' in 28th USENIX Security Symposium. Santa Clara, CA: USENIX Association, 2019.
  * [12] P. Peng, L. Yang, L. Song, and G. Wang, ``Opening the blackbox of virustotal: Analyzing online phishing scan engines,'' in Proceedings of the Internet Measurement Conference. ACM, 2019, pp. 478–485.
  * [13] A. Mohaisen and O. Alrawi, ``Av-meter: An evaluation of antivirus scans and labels,'' in International Conference on Detection of Intrusions and Malware, and Vulnerability Assessment. Springer, 2014, pp. 112–131.
  * [14] A. Salem and A. Pretschner, ``Poking the bear: Lessons learned from probing three android malware datasets,'' in Proceedings of the 1st International Workshop on Advances in Mobile App Analysis. ACM, 2018, pp. 19–24.
  * [15] F. Pendlebury, F. Pierazzi, R. Jordaney, J. Kinder, and L. Cavallaro, ``Enabling fair ml evaluations for security,'' in Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security. ACM, 2018, pp. 2264–2266.
  * [16] H. Sanders and J. Saxe, ``Garbage in, garbage out: how purportedly great ml models can be screwed up by bad data,'' Technical report, 2017.
  * [17] K. Tam, A. Feizollah, N. B. Anuar, R. Salleh, and L. Cavallaro, ``The evolution of android malware and android analysis techniques,'' ACM Computing Surveys (CSUR), vol. 49, no. 4, 2017, p. 76.
  * [18] S. Arshad, M. A. Shah, A. Khan, and M. Ahmed, ``Android malware detection & protection: a survey,'' International Journal of Advanced Computer Science and Applications, vol. 7, 2016, pp. 463–475.
  * [19] K. Allix, T. F. Bissyandé, J. Klein, and Y. Le Traon, ``Androzoo: Collecting millions of android apps for the research community,'' in Mining Software Repositories (MSR), 2016 IEEE/ACM 13th Working Conference on. IEEE, 2016, pp. 468–471.
  * [20] D. Arp, M. Spreitzenbarth, M. Hubner, H. Gascon, and K. Rieck, ``Drebin: Effective and explainable detection of android malware in your pocket.'' in NDSS, 2014.
  * [21] scikit learn. sklearn.metrics.matthews_corrcoef. [Online]. Available: http://tiny.cc/8gbb7y (2019)
  * [22] ——. sklearn.metrics.matthews_corrcoef. [Online]. Available: http://tiny.cc/x65xlz (2019)
  * [23] M. A. Inventor. About us - explore mit app inventor. [Online]. Available: http://tiny.cc/byp89y (2019)
  * [24] A.-T. T. I. I.-S. Institute. The best antivirus software for android. [Online]. Available: http://tiny.cc/1k66az (2019)
  * [25] P. Magazin. Bitdefender free edition. [Online]. Available: http://tiny.cc/8vx9az (2008)
  * [26] androguard. androguard: Reverse engineering, malware and goodware analysis of android applications … and more (ninja !). [Online]. Available: https://goo.gl/toC9dB (2018)
  * [27] J. N. Aleieldin Salem, Michael Hesse and A. Pretschner, ``Towards empirically assessing behavior stimulation approaches for android malware,'' in The 13th International Conference on Emerging Security Information, Systems and Technologies. International Academy, Research and Industry Association (IARIA), 2019.
  * [28] K-9 mail - advanced email for android. [Online]. Available: http://tiny.cc/1fbb7y (2019)
  * [29] A. Mohaisen, O. Alrawi, M. Larson, and D. McPherson, ``Towards a methodical evaluation of antivirus scans and labels,'' in International Workshop on Information Security Applications. Springer, 2013, pp. 231–241.
  * [30] S. Sachdeva, R. Jolivot, and W. Choensawat, ``Android malware classification based on mobile security framework,'' IAENG International Journal of Computer Science, vol. 45, no. 4, 2018.