# Online Malware Classification with System-Wide
System Calls in Cloud IaaS

Phillip Brown, Austin Brown and Maanak Gupta  Department of Computer Science
Tennessee Tech University
Cookeville, TN USA
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Mahmoud
Abdelsalam  Department of Computer Science
North Carolina A&T State University
Greensboro, NC, USA
<EMAIL_ADDRESS>
###### Abstract

Accurately classifying malware in an environment allows the creation of better
response and remediation strategies by cyber analysts. However, classifying
malware in a live environment is a difficult task due to the large number of
system data sources. Collecting statistics from these separate sources and
processing them together in a form that can be used by a machine learning
model is difficult. Fortunately, all of these resources are mediated by the
operating system’s kernel. User programs, malware included, interacts with
system resources by making requests to the kernel with system calls.
Collecting these system calls provide insight to the interaction with many
system resources in a single location. Feeding these system calls into a
performant model such as a random forest allows fast, accurate classification
in certain situations. In this paper, we evaluate the feasibility of using
system call sequences for online malware classification in both low-activity
and heavy-use Cloud IaaS. We collect system calls as they are received by the
kernel and take n-gram sequences of calls to use as features for tree-based
machine learning models. We discuss the performance of the models on baseline
systems with no extra running services and systems under heavy load and the
performance gap between them.

###### Index Terms:

Malware Classification, Cloud Computing Security, Dynamic Malware Analysis,
Machine Learning

##  I Introduction

The rapid infrastructure churn of modern cloud systems requires a fast,
scalable malware classification system that provides actionable intelligence
that can be used for rapid remediation. In this work, we attempt to design a
malware classification system that is able to run on features collected in
real time from a running, online Linux system. Our primary motivation is
collection performance: feature collection must not interfere with the
processes on the system that are required for business value. Our chosen
primary features, system calls, are more difficult to collect than simpler
summary statistics like performance metrics or network flows, but their
granularity and breadth of information are wonderful aids when classifying
malware [1].

Malware analysis is broadly divided into two disciplines: static and dynamic.
Static malware analysis deals with malware at rest. Static analysis inspects
malware files without executing them. This makes static analysis a safe method
of analysis because malware cannot damage an analysis machine or network.
Modern malware creation techniques can often evade static analysis by hiding
parts of the executable or disguising control flow. The ease of disguising
malicious code from static analysis is the greatest weakness of static
analysis.

Dynamic analysis studies malware during and after execution. Unlike static
analysis, the malware is actively executed and its effects on the system are
studied. Much more information about a piece of malware can be collected
during dynamic analysis. Since executable sections must be available for the
malware to run, dynamic analysis defeats packing strategies that would evade
static analysis. Further, network, memory, and file activity on the system can
be observed to investigate what parts of the system the malware is affecting.

Malware is often destructive so care must be taken to create a network where
damage that may be caused by the malware is reversible and contained to a
well-monitored and segmented part of the network. Malware may attempt to
detect if it is running in a sandbox or being monitored and alter its behavior
to avoid analysis. Careful design of the execution environment can reduce the
malware’s ability to detect that it is being inspected. Online dynamic
analysis is a specific dynamic technique where the malware is not run in a
simulated environment but rather a real, internet connected system. This
allows malware to connect to internet resources if those are required for its
operation. While this gives the best view into the behavior of the malware, it
is the most dangerous method of analysis, as giving malware access to the
internet potentially allows it to spread and infect vulnerable systems. We
have specifically designed our experiments and environments to leverage online
analysis to replicate real enterprise installations that may be targeted by
malware authors.

TABLE I: Related Works Compared

Reference | Analysis | Features | ML Technique | Domain | Platform
---|---|---|---|---|---
|  Classification |  Detection |  System Calls |  Other |  Traditional |  Deep Learn. |  Online |  Linux |  Android |  Windows
Canzanese et al. [2] |  |  |  |  |  |  |  |  |  |
Babenko and Kirillov [3] |  |  |  |  |  |  |  |  |  |
Canfora et al [4] |  |  |  |  |  |  |  |  |  |
Alsulami and Mancoridis [5] |  |  |  |  |  |  |  |  |  |
Das et al [6] |  |  |  |  |  |  |  |  |  |
Chandramohan et al [7] |  |  |  |  |  |  |  |  |  |
Dawson et al [8] |  |  |  |  |  |  |  |  |  |
Azmandian et al [9] |  |  |  |  |  |  |  |  |  |
Our approach |  |  |  |  |  |  |  |  |  |

00footnotetext: Experiments performed on a physical Android phone with network
access

##  II Motivation

We chose to perform malware classification over detection because it gives
greater insight into the threat and scale of possible damages for an
infection. Simple spam tools or adware may not dictate the same measure of
response that a crypto-locker or remote access Trojan would. Our specific
dynamic analysis implementation is intended to detect a compromised system by
inspecting system-wide features. While one process may be easier to classify,
doing classification for every process on a system would quickly become
impossible. Therefore, we observe features from the entire system at once.
This has the additional benefit of being able to detect the compromise of
existing benign processes on the system. ”File-less” malware is malicious code
that is somehow injected into the process of an already running process. This
avoids malware detection if the detection is observing only new processes or
has already flagged the compromised process as benign.

We chose to target online Linux systems because of the current and likely
future popularity for the GNU/Linux-based operating systems in cloud
computing. Cloud environments are increasingly popular for enterprise use
because of their scalability, price efficiency, and availability. As their
popularity increases, these systems are increasingly being targeted by malware
authors seeking to compromise them.

Our choice of model and features were also driven by practical considerations.
While modern neural models have shown great promise for security work, their
high resource requirements during training and the high number of samples
required to effectively train them makes them difficult to implement without
considerable infrastructure investment. Traditional machine learning models
require less training data and can be effectively trained on consumer-grade
CPU hardware, and their performance has been proven to be acceptable for
security tasks [10, 11].

The main contributions of this work are:

  * •

We evaluate the feasibility of using traditional machine learning models for
classifying malware from behavior in a custom whole-system system-call
sequence data set from simulated cloud IaaS systems.

  * •

We show this approach is effective for low-activity cloud systems, but is less
effective when systems are under heavy load and generating many system calls.

##  III Related Work

Malware detection and classification are wide and active disciplines. To
narrow the field of peers, we will compare our work to only those other works
that are attempting dynamic analysis. While there is a plethora of static
malware analysis research, static and dynamic analysis methods are distinct
enough that static analysis is not comparable to our work. Static analysis
analyzes the attributes of malware files while dynamic analysis is focused on
observing the behavior of a malware during its execution. We compare our work
to only those other works that are also observing malware behavior. We would
be particularly interested in comparing our work to other work where the
malware is not sand-boxed and is able to run as it would ”in the wild” on
compromised systems, but such research is understandably difficult to find due
to the dangers of running malware on live systems.

There is a large body of work on malware detection with various sources of
dynamic features collected at run-time. We can make useful comparisons here to
data collection and processing because these are largely similar between
detection and classification tasks. Reference [8] uses VM introspection tools
to collect system calls from a VM running below the context of the monitor
system. Reference [7] provides an insightful method to process raw system
calls into a format better suited for machine learning.

###  III-A System Call Collection

Reference [6] is perhaps the most similar in concept to our own, also
collecting Linux system calls. This work, however, is only concerned with
single isolated systems and does not classify malware, only detects it.
Reference [2] collects system calls for malware classification, but does so on
hosts running the Windows operating system that are not connected to the
internet.

Several malware detection works focus on the Android mobile operating system.
While Android applications are packaged and managed differently than mainline
GNU/Linux distributions, the similarities in the platforms make Android
security research valuable contributors to our own work. Reference [4] uses
system call sequences collected from Android program execution to detect
malware. Their approach uses n-grams of system calls and the counts of those
n-grams as features for a support vector machine. Similar to our work, the
authors in this work wanted a realistic environment for malware to execute. To
this end, they executed their malware and benign data sets on a physical
Android phone instead of using Android emulators.

##  IV Data Collection Methodology

###  IV-A Experiment Environment

A primary goal of this research is to design malware classification that works
in real environments. All of the malware experiments were carried out on real
Linux cloud servers with unrestricted internet access. The experiments were
hosted on an OpenStack111www.openstack.org instance graciously provided by the
University of Texas at San Antonio222www.utsa.edu. OpenStack provides a free
and powerful cloud infrastructure platform like those that are popular in
industry and as targets for malware authors. This OpenStack platform emulates
production cloud environments with full Linux systems. This provides some
important advantages for malware analysis over safer methods like sand-boxing
or emulation; the systems used in this research are just like those in the
wild that malware will be written to infect. These systems likely pass most
network and sandbox-detection evasion methods in malware so that the
potentially hidden behavior of the executable can be captured.

Figure 1: The OpenStack Environment where the malware was examined

Malware behavior collection experiments were performed in two stages across
the same environment. The environment, shown in Figure 1, is a number of
target virtual machines (VMs) running fully-patched Ubuntu 18.04 on OpenStack,
each with its own network connection and public IP address. To collect malware
behavioral data, each target machine was run for ten minutes. For the first
five minutes, the machine ran unaltered. After approximately five minutes
(some randomness was used to prevent malware from always executing at the same
time in the traces), the controller node copies in and executes a malware
executable. At ten minutes the target is destroyed and replaced by a new
target and the experiment continues. In the first stage of experiments, no
software other than what is required for the Ubuntu 18.04 image was running.
In the second stage, the server was running an Apache web server hosting a
WordPress site while simulated traffic was generated from the controller node.

To efficiently collect the system calls from the live systems, we needed a
kernel-level solution that did not interrupt process execution. The utility
`sysdig`333https://github.com/draios/sysdig provides this functionality.
System call collection is done in kernel at call time and is completely
transparent to the calling process. Performance overhead is low enough to be
acceptable in our experiments here, with no measured impact to application
response.

During test execution on the targets, `sysdig` was run to collect all system
calls sent to the kernel from all processes. The output of each system call
record was retrieved from the target machine, labeled with the hash of the
malware executable used, and stored for later use. Results were stored as-is
in the proprietary `sysdig` binary output file because of its compact size and
completeness.

TABLE II: All classes from the 4,180 final samples Class | Count | Used
---|---|---
trojan | 2299 |
virus | 616 |
backdoor | 382 |
rootkit | 253 |
miner | 226 |
grayware | 142 |
worm | 142 |
none | 87 |
ransomware | 21 |
downloader | 7 |
bot | 3 |
hoax | 2 |

###  IV-B Executable Sample Selection and Curation

A major share of the work behind this research was collecting and curating a
suitable malware set that provided good samples that ran in our selected
environment and had enough class diversity to perform useful classification.
Initially, the academic dataset from VirusTotal444https://www.virustotal.com/
was explored, but it was found to be poorly balanced. While the samples in the
VirusTotal dataset were appropriate for our systems and were functional, the
majority of them ($>$80%) were Mirai or Mirai variants. Additional samples
from MalShare and VirusShare555https://www.malshare.com/ and
https://virusshare.com/ were added to increase the numbers of other classes.
After removing duplicates and selecting only x86-64 binaries, from more than
10,000 total samples we selected 4,180 samples that were functional. These
classes are shown in Table II and those used in our experiments are check-
marked. These classes were sufficient as a base for our final class selection
described in Section V.

Figure 2: Behavior over time of the experiments

##  V Machine Learning Classification

###  V-A Classes

Our malware samples were not labeled by the providers. To determine a
reasonable classification, we examined the combined output of the various
engines behind VirusTotal with AVClass666https://github.com/malicialab/avclass
[12][13], a tool written to determine family and class labels from VirusTotal
scan results.

VirusTotal uses several engines and scanners to determine what a malware file
is. At the time of writing, VirusTotal claimed over 70 anti-virus scanners and
services.VirusTotal has an available API where an executable can be uploaded
and VirusTotal will distribute the sample to all of those engines and return a
report containing reports from each engine. These reports are not
standardized; each AV vendor engine has their own reporting syntax and
semantics. This creates a difficulty for determining accurate class labels for
a piece of malware since it may be reported as different classes by different
vendors, and each vendor may use a different term or label for the same class.
While this makes determining proper labels more difficult, it is not
impossible. Fortunately, AVClass can parse these VirusTotal API results and
determine proper class labels.

Semantics have a large influence on malware classification. The classes in
Table II are labels created to describe malware based on some behavior that it
exhibits. Some classes may have overlap in the behavior that they describe. In
our data, we find the classes `trojan`, `virus`, and `backdoor`. Since the
classification labels obtained from AVClass are sourced by consensus from many
sources, the class labels are particularly likely to differ among the various
engines based on the perspectives of those that wrote them. The `trojan` class
in particular was found by the authors of [13] to have often been used as a
’catch-all’ class when a better classification could not be made. This has
some impact on our classification work.

The classes in Figure II are the total set present in all the malware
selected. Obviously, the low-numbered classes are not usable for training a
machine learning model, so they are dropped along with the ’none’ class
samples that AVClass was not able to determine a class for. This left the
seven classes `trojan`, `virus`, `backdoor`, `rootkit`, `miner`, `grayware`,
and `worm`. We did not pursue family attribution in this work, as we found
that AVClass could not produce confident family labels for enough of our
dataset.

###  V-B Model Selection

In testing, we found random forests to be accurate classifiers that were easy
to train on our data. Other work has also found decision tree models to be
among the fastest and most accurate among traditional machine learning models
for security work [10][11]. The particular tree model we found to be most fast
and effective was
LightGBM777https://lightgbm.readthedocs.io/en/latest/index.html, a gradient-
boosting tree-based model.

TABLE III: Reduced set of 35 system calls read | write | creat
---|---|---
open | openat | unlink
chdir | access | utime
chmod | ftruncate | rename
getdents | fstat | fstat64
fadvise64 | execve | rt_sigaction
rt_sigprocmask | kill | tgkill
sched_yield | send | bind
connect | recvfrom | poll
epoll_create | select | ioctl
brk | mmap | mmap2
munmap | mprotect |

###  V-C Feature Processing

The raw features extracted from the malware experiments were the list of
system calls made to the kernel during our execution periods. We decided to
perform machine learning analysis using a random forest fed n-grams of system
calls from the sequence. This approach has been accurate [14] and showed good
and efficient results in our work.

There was some experimentation done to determine a proper feature space.
First, we did not consider every possible combination of calls but only those
that were present in all of our experiment runs. Second, to reduce the number
of n-grams calculated, we performed the experiments with a limited number of
system calls. Limiting the number of calls collected drastically reduces the
number of possible n-gram features. Some detail is lost with the discarded
calls of course, but careful selection of the retained calls still gives a
good view of system activity. Based on [6] and observations from our dataset,
we determined a short list of system calls that implicate security affecting
behavior such as file interaction or network communication. These calls are
listed in Table III.

Figure 3: Performance on the baseline experiment set with no background
services running

While malware injection happens at approximately the same time in each malware
experiment, there is a lot of uncertainty in how active the malware is. This
has bearing on how n-gram features are extracted. Each malware binary in our
final experiment set is confirmed run when it is injected, so features around
the midpoint of the execution time will likely contain malware activity. The
execution period before malware injection is also certain; the only activity
during this period is related to the the baseline services we installed on the
execution VMs. As time progresses past malware injection, however, behavior
becomes more uncertain. Figure 2 shows the phases of the experiments. Features
sampled farther and farther from malware injection may or may not actually
contain any malware behavior and there is no way to determine this in our
feature set.

We experimented with dividing the ten-minute experiment periods into time
slices to reduce the amount of time covered by each system call sequence
extraction. We tested model performance when all system calls from the 10
minutes were collected together and when the experiment was divided into
three, five, and ten slices of equal length. Dividing into ten slices of a
minute each was found to give a good balance of model efficacy and extraction
performance. Larger slices were less accurate due to the number of non-malware
calls increasing and smaller slices were less efficient because feature
extraction had to be performed many more times.

We also had to develop a strategy to handle the uncertainty of malware
behavior in the latter half of the experiments. The malware is on the system
at this time so this time period cannot be considered benign, but the malware
may not be active during the entire period and so may not show in the
collected features. Labeling that period as a malware class will likely lead
to poor classification both during training and classification. We withhold
these time slices in training and separate them during classification and
reporting.

TABLE IV: Classification performance per-slice for minute-long slices of 3-gram features on baseline data. The bolded row is the only slice in which malware is certainly active Minute | Label | Accuracy | Precision | Recall | F1
---|---|---|---|---|---
1 | Benign | 100.00 | NA | NA | NA
2 | Benign | 99.09 | NA | NA | NA
3 | Benign | 99.18 | NA | NA | NA
4 | Benign | 99.74 | NA | NA | NA
5 | Benign | 98.83 | NA | NA | NA
6 | Malicious | 94.30 | 0.98 | 0.90 | 0.95
7 | Malicious | 33.57 | 0.86 | 0.25 | 0.39
8 | Malicious | 55.54 | 0.83 | 0.30 | 0.40
9 | Malicious | 41.82 | 0.84 | 0.28 | 0.41
10 | Malicious | 64.48 | 0.82 | 0.34 | 0.44

##  VI Results

###  VI-A Result Methodology

A notable effort was made to determine the suitability of class labels in our
dataset. As mentioned in section V-A, [13] discussed the semantic meaning
behind certain anti-virus labeling with a focus on the `trojan` label. In
Figure 3, it can be observed that several samples belonging to other classes
are improperly classified as trojans. Our belief is that this is not due to a
lack of model performance, but due to the differences among anti-virus vendors
when selecting class labels. If a particular vendor cannot determine a proper
class for a piece of malware, it may default to assigning it to the trojan
class. Additionally, the described behavior of some of these classes is quite
similar. In particular, the classes `trojan`, `rootkit`, and `backdoor` may be
reasonably used to describe malware that establishes an illicit backdoor
connection to an infected machine. The final class name for samples from these
classes is somewhat arbitrarily determined by malware research vendors.

To measure our models’ performance, we use four evaluation888
$Accuracy=\frac{TP+TN}{TP+TN+FP+FN},Precision=\frac{TP}{TP+FP},\\\
Recall=\frac{TP}{TP+FN},F1-score=2\times\frac{Precision\times
Recall}{Precision+Recall}$ metrics: accuracy, precision, recall, and F1 score.
Our preferred measure of performance is the model’s F1 score. F1 measures the
relationship between precision and recall. Precision measures the number of
samples that correctly belong to the predicted class and recall measures how
many samples of a class were correctly assigned that class. The F1 score is
the harmonic mean of precision and recall.

Note that Tables IV and V report NA values for precision, recall, and F1 for
time slices that only have one class (benign) present, as having no true
positives and false positives or negatives yields zeros in the denominators of
the equations. Since we do not label samples in these slices as a positive
malicious class, these metrics are difficult to report.

TABLE V: Classification performance per-slice for minute-long slices of 3-gram features on application data. The slices after malware injection have been removed Minute | Label | Accuracy | Precision | Recall | F1
---|---|---|---|---|---
1 | Benign | 100.00 | NA | NA | NA
2 | Benign | 100.00 | NA | NA | NA
3 | Benign | 99.67 | NA | NA | NA
4 | Benign | 100.00 | NA | NA | NA
5 | Benign | 100.00 | NA | NA | NA
6 | Malicious | 74.92 | 0.45 | 0.27 | 0.30
TABLE VI: Detection (binary classification) results for both data sets Data set | Accuracy | Precision | Recall | F1
---|---|---|---|---
Baseline | 1.00 | 1.00 | 1.00 | 1.00
Application | 98.19 | .97 | .97 | .97

###  VI-B Results Discussed

Our experiments have determined that this approach is valid for determining
the class of a piece of malware on a quiet system, but the approach as
implemented is not as effective for classification of malware on a system
under heavy load.

The results in Table IV shows classification performance on the baseline
experiments where the malware is running on a system with no additional
configuration after installation. These results indicate that the model has
effectively learned to classify these samples based on patterns in their
system call usage. There is no malware running on the system until it is
injected at the half-way point, so all time slices prior to this point are
labeled benign. The time slice where malware is injected and all following
slices are labeled as the class to which the malware belongs. While time
slices after the inject are labeled as the injected malware class, the malware
is not certain to take any action during this period and may not leave any
evidence of its presence. The performance metrics for these slices are
therefore low. The majority of slices after the inject slice are identified by
the model as benign, indicating that the malware was likely not active.

Table V displays the results of classification for the data collected while a
web-server application was active and stress tested on the test machines; see
section IV-A for experiment details. Clearly the results are much worse. This
is attributable to the massive increase in the number of system calls observed
on the data sets collected during the baseline and application experiments.
The number of features doubles from the baseline to the application dataset.
The increase in system calls made by the web service increases the chance that
the system calls made by the malware will be dispersed through the benign
calls and difficult to pick out.

Despite the inability to accurately classify malware in the application
dataset, detection works well. Table VI displays the results when all
malicious classes are combined into a single `malicious` class. While the main
purpose of this work is to classify malware on these sequences, the results
here do indicate that there is enough feature difference to make conclusions
about malware behavior. Additional feature processing or model development may
accurately classify malware while system call intensive applications are
running on the system and is left for further work.

##  VII Conclusion and Future Work

In this work we described a system for attempting to determine the class of
malware infecting a system by observing the all system calls made to the
kernel. This approach is feature efficient because only a single feature feed
needs to be collected from the kernel instead of separate feeds from every
process. This work has shown to be effective at determining what class of
malware a system is infected with by observing all system calls made to the
kernel when other activity on the system is low. Future work should continue
feature testing and processing to identify ways to improve the performance on
systems with high amounts of other activity. The detection performance we
achieved does indicate that there is enough information in the system call
sequences to make some observations on malware activity, so this should be
explored further.

## Acknowledgement

This research is partially supported by NSF Grants 2025682 and 1565562 at TTU,
and 2150297 at NCAT.

## References

  * [1] R. J. Maulana and G. P. Kusuma, “Malware classification based on system call sequences using deep learning,” _Advances in Science, Technology and Engineering Systems Journal_ , vol. 5, no. 4, p. 207–216, 2020.
  * [2] R. Canzanese, M. Kam, and S. Mancoridis, “Toward an automatic, online behavioral malware classification system,” in _2013 IEEE 7th International Conference on Self-Adaptive and Self-Organizing Systems_ , Sep 2013, p. 111–120.
  * [3] L. Babenko and A. Kirillov, “Development of method for malware classification based on statistical methods and an extended set of system calls data,” in _Proceedings of the 11th International Conference on Security of Information and Networks_. ACM, Sep 2018, p. 1–6. [Online]. Available: https://dl.acm.org/doi/10.1145/3264437.3264478
  * [4] G. Canfora, E. Medvet, F. Mercaldo, and C. A. Visaggio, “Detecting android malware using sequences of system calls,” in _Proceedings of the 3rd International Workshop on Software Development Lifecycle for Mobile_ , ser. DeMobile 2015. Association for Computing Machinery, Aug 2015, p. 13–20. [Online]. Available: https://doi.org/10.1145/2804345.2804349
  * [5] B. Alsulami and S. Mancoridis, “Behavioral malware classification using convolutional recurrent neural networks,” in _2018 13th International Conference on Malicious and Unwanted Software (MALWARE)_ , Oct 2018, p. 103–111.
  * [6] S. Das, Y. Liu, W. Zhang, and M. Chandramohan, “Semantics-based online malware detection: Towards efficient real-time protection against malware,” _IEEE Transactions on Information Forensics and Security_ , vol. 11, no. 2, p. 289–302, Feb 2016.
  * [7] M. Chandramohan, H. B. K. Tan, L. C. Briand, L. K. Shar, and B. M. Padmanabhuni, “A scalable approach for malware detection through bounded feature space behavior modeling,” in _2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE)_ , Nov 2013, p. 312–322.
  * [8] J. A. Dawson, J. T. McDonald, L. Hively, T. R. Andel, M. Yampolskiy, and C. Hubbard, “Phase space detection of virtual machine cyber events through hypervisor-level system call analysis,” in _2018 1st International Conference on Data Intelligence and Security (ICDIS)_ , Apr 2018, p. 159–167.
  * [9] AzmandianFatemeh, MoffieMicha, AlshawabkehMalak, DyJennifer, AslamJaved, and KaeliDavid, “Virtual machine monitor-based lightweight intrusion detection,” _ACM SIGOPS Operating Systems Review_ , Jul 2011, pUB27 New York, NY, USA. [Online]. Available: https://dl.acm.org/doi/abs/10.1145/2007183.2007189
  * [10] A. S. A. Aziz, S. E.-O. Hanafi, and A. E. Hassanien, “Comparison of classification techniques applied for network intrusion detection and classification,” _Journal of Applied Logic_ , vol. 24, p. 109–118, Nov 2017\.
  * [11] M. C. Belavagi and B. Muniyal, “Performance evaluation of supervised machine learning algorithms for intrusion detection,” _Procedia Computer Science_ , vol. 89, p. 117–123, Jan 2016.
  * [12] S. Sebastián and J. Caballero, “Avclass2: Massive malware tag extraction from av labels,” _arXiv:2006.10615 [cs]_ , Oct 2020, arXiv: 2006.10615. [Online]. Available: http://arxiv.org/abs/2006.10615
  * [13] M. Sebastián, R. Rivera, P. Kotzias, and J. Caballero, _AVclass: A Tool for Massive Malware Labeling_ , ser. Lecture Notes in Computer Science. Springer International Publishing, 2016, vol. 9854, p. 230–253. [Online]. Available: http://link.springer.com/10.1007/978-3-319-45719-2_11
  * [14] D. Canali, A. Lanzi, D. Balzarotti, C. Kruegel, M. Christodorescu, and E. Kirda, “A quantitative study of accuracy in system call-based malware detection,” in _Proceedings of the 2012 International Symposium on Software Testing and Analysis - ISSTA 2012_. ACM Press, 2012, p. 122. [Online]. Available: http://dl.acm.org/citation.cfm?doid=2338965.2336768