[1]Department of Computer Science, San Jose State University

Research in the field of malware classification often relies on
machine learning models that are trained on high-level features,
such as opcodes, function calls, and control flow graphs.
Extracting such features is costly, since disassembly or
code execution is generally required. In this paper, we conduct
experiments to train and evaluate machine learning models for malware classification,
based on features that can be obtained without disassembly or execution of code.
Specifically, we visualize malware samples as images and employ image analysis
techniques. In this context, we focus on two machine learning models, namely,
Convolutional Neural Networks (CNN) and Extreme Learning Machines (ELM).
Surprisingly, we find that ELMs can achieve accuracies on par with CNNs,
yet ELM training requires less than 2%  of the time
needed to train a comparable CNN.

§ INTRODUCTION

Malware is software that is designed to disrupt or damage computer systems.
According to Symantec, more than $669$ million new
malware variants were detected in 2017, which was an increase of more
than 80% from 2016 <cit.>. Clearly, malware detection is a critical
task in computer security.

Commonly used malware detection techniques can be broadly categorized into anomaly-based
and signature-based. However, these strategies have some potential disadvantages.
Obfuscation can be used to evade
signature-based detection <cit.>, while anomaly-based detection is costly and
often yields an unacceptably high false positive rate <cit.>. Malware detection
based on machine learning models may overcome these weaknesses.

Successful machine learning based malware detection approaches have been
trained on high-level features such as opcode sequences, function calls, or control flow
graphs <cit.>.
However, extracting such high-level features can be costly; hence approaches that do not
require extensive pre-processing are preferred, provided that sufficient accuracy can be attained.
Specifically, we prefer features that can be obtained without disassembly or code execution.

In this research, we consider the Malimg dataset <cit.>,
which consists of malware samples that have been visualized as images.
To obtain these malware images, bytes from executable files are
trivially mapped to pixels to create grayscale
images <cit.>. The primary focus of our research is to
compare the classification performance of convolutional neural networks (CNN)
and extreme learning machines (ELM) for these malware images.

CNNs have been shown to perform well in image classification tasks <cit.>.
In comparison, ELMs are a somewhat less well-known technique,
particularly in the field of malware analysis.
ELMs are somewhat controversial, but experimental studies have
shown that they can produce acceptable predictive
performance in some tasks, and at a much lower training
cost, as compared to networks that are trained by backpropagation <cit.>.
This makes ELMs attractive for malware classification, particularly when considering the large
volume of new malware variants.

When testing CNN and ELM models on the Malimg dataset <cit.>,
we perform extensive parameter tuning to optimize the performance of each model. For ELMs we also
consider an ensemble approach, as previous work has shown that such ensembles can overcome
some of the instability issues that are inherent in ELMs <cit.>.

The remainder of this paper is organized as follows. Section <ref> discusses
relevant previous work. In Section <ref> we provide a summary of the
dataset used in this research, and we briefly discuss the machine learning techniques that are
used. In Section <ref>, we present and analyze the results of our experiments.
Finally, in Section <ref>, we summarize our results and consider possible
avenues for future work.

§ BACKGROUND

Due to the large volume of new malware, antivirus developers must
constantly upgrade their methods and algorithms. This makes accurate and rapid
detection of malware a critical topic in information security <cit.>.

Malware classification can be based on static or dynamic features, or some combination thereof.
Static features are those that are extracted from static files,
while dynamic features are extracted during code execution or emulation.
Static approaches often use features such as calls to external libraries, strings,
and byte sequences for classification <cit.>. Other static approaches
extract higher-level information from binaries, such as sequences of API calls <cit.> or opcode information <cit.>. Examples of dynamic features include resource usage
and the frequency of calls to specific kernel functions.

A considerable amount of previous work has been done on malware classification.
In this section, we discuss a few representative samples of malware classification
techniques that have appeared in literature.

§.§ Previous Work

Machine learning models for malware detection and classification are
trained on features or attributes extracted from executable files.
As previously mentioned,
examples of such features include
opcodes, API calls, control flow graphs, and many others. In many cases
the extraction of these features can be costly, so approaches using raw bytes
are preferred, if comparable accuracy can be obtained.
For example, byte $n$-grams have been successfully used as features.
As another example, it is possible to treat executable files
as images, and apply image analysis techniques.

An opcode-based approach for malware classification is
presented in <cit.>. The opcodes are obtained by
disassembling the corresponding  files.
The authors then use a technique for determining
the relevance of an opcode that is similar to the well-known tf-idf
(Term Frequency-Inverse Document Frequency) forumla, which is
a numerical statistic often used in text mining and search engines.
In this context,
opcode $n$-grams are considered as the “words.”
These authors also define a Weighted Term Frequency (WTF)
to measure of the relevance of each opcode. Although they obtain
satisfactory results, their method has a number of disadvantages,
including the costly step of disassembling executable files to extract opcodes.

Another opcode-based malware detection technique is proposed in <cit.>.
This work relies on an adjacency matrix based on pairs of consecutive opcodes.
The feature vector for each file is constructed by applying the “power iteration”
method to the opcode graph. They obtain reasonable results using this technique for
the problem of distinguishing malware from benign. However, the authors observe that
the power iteration method generally takes a significant amount of time to converge
to the dominant eigenvector.

In <cit.>, the authors divide each executable into non-overlapping byte substrings
and the frequency histogram of these substrings comprise the feature vectors.
They developed a data-mining method that is tested on parts of the file,
such as the header or text (code) section. The authors claim that their
technique improves detection rates for new malware, as compared to
traditional signature-based methods.

The method in <cit.> is also based on byte $n$-grams.
In this case, each $n$-gram is considered as a boolean attribute, which is
if the $n$-gram is present in an executable file and  otherwise. The top 500
$n$-grams are selected based on Information Gain (IG). The authors test a variety
of machine learning algorithms on these features (SVM, Naïve Bayes, and
Decision Tree) and obtain good results for detecting unknown malware instances.

The method proposed in <cit.> relies on $n$-grams as file signatures.
The authors employ the $k$-nearest neighbor algorithm and experiment with
different values of $n$ for the $n$-grams. They achieve their best results
using 4-grams.

In <cit.>, the authors propose a technique based on converting executable files to audio.
Specifically, they convert each byte of an executable into its corresponding
decimal value, which in turn represents a specific musical note in the MIDI
encoding scheme. Using this technique, each executable is converted into a MIDI file.

The MIDI file is then converted into an audio signal, and Music Information Retrieval
(MIR) techniques are used to extract audio features. Their method achieves an accuracy
in excess of 90% for distinguishing malware from benign,
based on $k$-nearest neighbor, AdaBoost, and Random Forest classifiers.

Extreme Learning Machines (ELM) have been applied to malware detection on
the Android platform in <cit.>. In this work, ELMs are trained on
static features, and the results are reasonably strong. In <cit.>,
the authors consider the effectiveness of High Performance Extreme Learning Machine (HP-ELM)
by varying the features and activation functions of the HP-ELM. Their experiments yield
a maximum accuracy of 95.92%.

A two layer ELM is applied to the malware detection problem in <cit.>.
A partially connected network is used between the input and the first hidden layer.
This layer is then aggregated with a fully connected network in the second layer.
The authors utilize an ensemble to improve the accuracy and robustness
of the system.

In <cit.>, the authors propose a method to directly
map bytes from an executable file to pixels to create a grayscale image. From the resulting
images, they observe that malware files belonging to the same family appear similar in
layout and texture. They achieve a classification accuracy of 98%
using $k$-nearest neighbor. The features considered are based on
so-called GIST descriptors, which summarize gradient information in an image.
One clear advantage of this method is that neither disassembly nor
code execution is required to obtain the features.

The research we present in this paper was motivated to some extent by image-based
analysis in <cit.>, as well as the ELM-based
thread in <cit.>.
In this paper, we focus on
the relative advantages and disadvantages of Convolutional Neural Networks (CNN)
and Extreme Learning Machines (ELM) when trained directly on pixel data,
rather than, say, high-level GIST descriptors. The features we consider are trivial
to extract. In addition, the ELM technique is extremely efficient to train,
and we obtain surprisingly strong results, as compared to CNNs.

Next, we provide details on relevant implementation and background topics. Then
in Section <ref> we present and discuss our experimental results.

§ IMPLEMENTATION

In this section, we give a summary of the malware families and the
dataset used in this research. Also, we briefly discuss the machine learning
techniques that we use in our experiments.

§.§ Dataset

We consider malware classification based on the Malimg
dataset <cit.>.
This dataset contains in excess of 9300 grayscale images
belonging to 25 different malware families. This dataset has been used
as the basis for several previous research papers, including <cit.>
and the
aforecited <cit.>,
among others.

Table <ref> lists the 25 families in the Malimg dataset
and the number of samples that belong to each family. Note that the Malimg dataset is
extremely imbalanced. For example, the largest family, , has  nearly
one-third of all of the samples and it is more than 36 times as large as the
least numerous family, , while the two largest families
comprise nearly half of the entire dataset.
This imbalance creates some
issues when attempting to classifying the samples.
We return to this imbalance issue in
Section <ref>.

Type of each malware family
Family     Samples     Family     Samples

125 	    		   184

116 	    		   123

1591 	    		   159

2949 	    	   136

198 	    	   142

106 	    		   158

146 	    		   80

200 	    	   128

177 	    	   132

162 	    			   408

381 	    		   97

431 	    		   800
213 	    Total				    9342

§.§.§ Visualization of Malware Files

In <cit.>, the authors describe the technique used
to convert malware executables into grayscale images. A given malware binary is read
as a vector of 8 bit unsigned integers and then organized into a 2-dimensional
array. This is visualized
as a grayscale image where each pixel is in the range 0 to 255,
where 0 is black and 255 is white. The width of the image is fixed and the height
is allowed to vary depending on the file size, as summarized in Table <ref>.

Image dimensions based on malware size
File Size     Width (pixels)

less than 10kB             32

10kB – 30kB	         64

30kB – 60kB	            128

60kB – 100kB	           256

100kB – 200kB	           384

200kB – 500kB           512

500kB – 1000kB           768

greater than 1000kB	            1024

Figures <ref> through <ref> show
show three different samples from each of three malware families.
Specifically, Figure <ref> has samples from the
family, while Figure <ref> shows
samples of  family, and Figure <ref>
contains samples of the  family.
These images indicate that there is considerable
intra-family similarity and inter-family differences,
at least with respect to these three specific malware families.
Intuitively, we would expect image-based analysis to perform
well on such data.

Adialer.C samples

Dialplatform.B samples

Dontovo.A samples

§.§ Classification Techniques

In this section, we describe the machine learning models considered in
this paper. First, we discuss Convolutional Neural Networks and
then we outline the main concepts behind Extreme Learning Machines.

§.§.§ Convolutional Neural Networks

Convolutional Neural Networks (CNN) draw inspiration from the workings and hierarchical
structure of the primary visual pathway of the brain. In the 1950s and 1960s,
experiments were conducted to understand how the brain perceives the
world visually <cit.>. It was discovered
that there are two basic types of visual neuron cells, namely,
simple cells (S cells) and complex cells (C cells). These cells activate
when they identify basic shapes such as lines in a fixed area and at a specific angle.

A CNN is a type of feed-forward neural network for processing data
that takes a 2-dimensional or higher-dimensional matrix as input. CNNs have
proven highly effective for image analysis. CNNs typically
include multiple convolutional layers, one or more pooling layers,
and a fully-connected output layer.
All of the CNN experimental results discussed in this
paper are based on models trained using Keras <cit.>.

Convolutional layers are the main building blocks of CNNs.
A discrete convolution is a sequence that is itself
a composition of two sequence—more precisely,
a discrete convolution is
computed as a sum of pointwise products. Let $c=x*y$
denote the convolution of sequences $x=(x_0,x_1.x_2,\ldots)$
and $y=(y_0,y_1.y_2,\ldots)$. Then $c_k$, the $k^{\thth}$
element of the convolution, is given by

%  (x*y)_{k} = \sum_{k=i+j} x_{i} y_{j} = \sum_{i=-\infty}^{\infty} x_{i} y_{k-i}
c_{k} = \sum_{k=i+j} x_{i} y_{j} = \sum_{i} x_{i} y_{k-i}

We can view this process as $x$ acting as a “filter” (or kernel) on the
sequence $y$ over a sliding window.
The concept of a discrete convolution
is easily extended to higher-dimensional data.
With CNNs, we generally treat color images as 3-dimensional arrays,
with the third dimension given by the red, green, and blue color planes
of the RGB encoding.

In a CNN, the coefficients of the convolutions (i.e., weights)
are learned via training, with this training typically completed
via the backpropagation algorithm. Convolutions serve to greatly reduce the
number of weights, as compared to a fully connected
network, and thus make training on images practical.
In addition, when applying CNNs to images, we obtain a high degree
of translation invariance, which is highly desirable in image
analysis, and serves to greatly reduces the overfitting that
would otherwise tend to occur.

The first convolutional layer of a CNN is applied to the input image.
This is illustrated in Figure <ref>, where
the input consists of an RGB image, which is treated as 3-dimensional
data. In this case, five filters are learned at the first layer—by randomly
initializing the weights of the filters, the CNN can learn
multiple filters at each layer. Thus, the next convolutional layer
can also be applied to higher-dimensional data.

After the first convolutional layer, subsequent convolutional layers
are applied to the output of previous convolutional layers. Hence,
we compute convolutions of convolutions.
At the first convolutional layer, the filters learn intuitive
features, such as edges and basic shapes, whereas the second convolutional
layer learns more abstract features, such as “texture.”
Convolutional layers learn progressively more abstract
features that ultimately can be used to distinguish, say,
a picture of a cat from that of a dog.

First convolutional layer with stack of 5 filters (RGB image)

Between each pair of convolutional layers, it is common to add a pooling layer.
The function of pooling is to reduce the dimensionality,
and thus reduce the training time. Pooling
might also improve various desirable properties,
such as translation invariance.
There are several types of pooling
functions, with max pooling being the most common.
Max pooling consists of simply selecting the maximum value over
a given filter block. Figure <ref> illustrates max pooling
over a 2-dimensional feature vector.

Max pooling layer ($2\times 2$)

Note that convolutional layers
typically use a stride of one, while pooling uses a stride
equal to the filter width, which serves to further increase the
downsampling effect of pooling. Recently, it has
become popular to use convolutional layers
with a stride greater than one in place of pooling layers.

§.§.§ Extreme Learning Machine

As with most aspects of ELMs, the origin of the technique
is somewhat controversial. The unfortunate terminology of
“Extreme Learning Machine” was apparently first used
in <cit.>. Regardless of the origin of the technique,
ELMs are essentially randomized feedforward neural networks that
effectively minimize the cost of training.

An ELM consists of a single layer of hidden nodes,
where the weights between inputs and hidden nodes are
randomly initialized and remain unchanged throughout training.
The weights that connect the hidden nodes to
the output are trained, but due to the simple structure of an ELM,
these weights can be determined by solving linear equations—more
precisely, by solving a linear regression problem. Since
no backpropagation is required, ELMs are far more efficient to train,
as compared to other neural network architectures. However, since
the weights in the hidden layer are not optimized, we will typically
require more weights in an ELM, which implies that the testing phase
may be somewhat more costly, as compared to a network
trained by backpropagation. Nevertheless, in applications
where models must be trained frequently, ELMs
can be competitive.

Architecture of an ELM model

Consider the ELM architecture shown in Figure <ref>,
where $X$ denotes the input layer, $H$ is the hidden layer and $Y$ is
the output layer. In this example, there are $N$ samples of the form $(x_i, y_i)$
for $i=1,2, \dots, N$, where $x_i=\rowvecccc{x_{i_1}}{x_{i_2}}{\dots}{x_{i_n}}^T$
is the feature vector for sample $i$
and $y_i = \rowvecccc{y_{i_1}}{y_{i_2}}{\dots}{y_{i_m}}^T$ are the output labels,
where $T$ indicates the transposition operation.
Then the input and output for the ELM are
as $X = \rowvecccc{x_1}{x_2}{\dots}{x_n}^T$
and $Y = \rowvecccc{y_1}{y_2}{\dots}{y_m}^T$, respectively.
In this example, the hidden layer $H$ has $\ell$ neurons.
We denote the activation function of the hidden layer as $g(x)$.

To train an ELM, we randomly selects the weight
matrix that connects the input layer $X$ to the
hidden layer $H$. We denote this randomly-assigned weight matrix
as $W = \rowvecccc{w_1}{w_2}{\dots}{w_{\ell}}$
where each $w_i$ is a column vector.
We also randomly select the bias
matrix $B = \rowvecccc{b_1}{b_2}{\dots}{b_{\ell}}$
for this same layer.
During the training phase, both $W$ and $B$ remain unchanged.

After $W$ and $B$ have been initialized,
the output of the hidden layer $H$ is given by

H=g(WX + B) .

The output of the ELM is denoted as $Y$ and is calculated as

where $\beta$ is the weight matrix for the output layer.

The values of the weights $\beta$ at the hidden layer
are learned via linear least squares,
and can be computed using $H^\dagger$,
the Moore-Penrose generalized inverse of $H$, as discussed below.
It is worth emphasizing that the only parameters that are learned in
the ELM are the elements of $\beta$.

Given that $Y$ is the desired output, a unique solution of the
system based on least squared error can be found as follows.
We denote the Moore-Penrose generalization inverse of $H$ as $H^\dagger$,
which is defined as

H^\dagger = \begin{cases}
(H^TH)^{-1}H^T  \text{ if $H^TH$ is nonsingular}\\
H^T(HH^T)^{-1}  \text{ if $HH^T$ is nonsingular}
\end{cases}

Then the desired solution $\beta$ is give by

\beta=H^\dagger Y

After calculating $\beta$, the training phase ends.
For each test sample $x$, the output $Y$ can be calculated as

Y=g\bigl(C(x)\bigr) \beta

where $C(x)$ is defined below.
The entire training process is extremely efficient, particularly in comparison
to the backpropagation technique that is typically used to train neural
networks <cit.>.

For the research reported in this paper,
we use the Python implementation of
ELMs given in <cit.>. This implementation
uses input activations
that are a weighted combination of two functions
that are referred to as
an “MLP” kernel and an “RBF” kernel—we employ
the same terminology here. The MLP kernel
is simply the linear operation

\mlpKernel (x) = Wx + B

where the weights $W$ and biases $B$
are randomly selected from a normal distribution.
This is the kernel function that is typically associated with
a standard ELM.

The RBF kernel is considerably more complex, and is based
on generalized radial basis functions as defined in <cit.>.

The details of this RBF kernel go beyond the scope of this paper;
see <cit.> for additional information and, in particular,
examples where this kernel is applied to train ELMs.
We use the notation $\rbfKernel(x)$ to represent
the RBF kernel. Also, it is worth noting that the RBF kernel
is much more costly to compute, and hence its use does somewhat negate
one of the major advantages of an ELM. We provide timing comparisons for these
kernels in Section <ref>.

The input activations are given by

\begin{equation}\label{eq:inputAct}
C(x) = \alpha \mlpKernel (x) + (1 - \alpha) \rbfKernel (x)
\end{equation}

where $0\leq\alpha\leq 1$ is a user-specified mixing parameter.
Note that for $\alpha=0$ we use only the MLP kernel $\mlpKernel(x)$
and for $\alpha=1$, only the RBF kernel $\rbfKernel(x)$ is used.

§ EXPERIMENTS AND RESULTS

In this section, we discuss the results of malware classification
experiments involving the Malimg dataset.
First we consider our CNN experiments,
then we discuss our ELM results. Finally, we compare the
results obtained using these two approaches.

§.§ CNN Experiments

For the experiments discussed in this section,
each CNN model was trained for 50 epochs with the
activation function. Various CNN architectures with different combinations of
convolution, pooling, and dense (fully-connected) layers are considered.
We also give results related to tuning of
the hyperparameter, including input image size, batch size, number of filters,
filter size, and pooling size.

§.§.§ One Convolutional Layer

Our one-convolutional layer models were trained on
different input image sizes ($32\times 32$, $64\times 64$,
and $128\times 128$ pixels) and different number of
filters ($32$  and $64$). Each of these CNN models uses $3\times 3$
filters, a fully connected layer with $128$ neurons,
and an output layer with $25$ neurons corresponding to the classes
in the Malimg dataset. A  activation function is used in each case.
Table <ref> summarizes our results
for these six cases.
We see that the larger image size of $128\times 128$ clearly outperforms
the smaller image sizes.

CNN models with one convolutional layer
Image Size     Filters     Accuracy

$32\times 32$                32		     0.8400

$32\times 32$               64		     0.8467

$64\times 64$               32		     0.9340

$64\times 64$               64		     0.9245

$128\times 128$               32		     0.9630

$128\times 128$               64		     0.9589

Classwise accuracies for the $128\times 128$
experiments in Table <ref>
are summarized in Figure <ref>.
We observe that
training these CNNs on images of size $128\times 128$ with $32$ filters
results in an accuracy of $96.3\%$.

The confusion matrix for this best case
is given in Figure <ref>. This model is able to achieve an accuracy of
above 90% for 16 out of the 25 classes in the dataset—only one family,
namely, , has an accuracy below 50%.

CNN with one convolutional layer ($128\times 128$ images)

Confusion matrix for best CNN model ($128\times 128$ images and $32$ filters)

The confusion matrix in Figure <ref> also shows which specific
malware families are frequently misclassified. For example, we see that $5$
of the $12$ samples of the family  are
misclassified as belonging to , which is not surprising,
given that they are both variants of the Swizzor trojan family. The visual similarity
between malware images from these families can be seen in
Figure <ref>.

Samples from  families

Another interesting misclassification case occurs for samples
of , which are all misclassified
as belonging to the family . The visual similarity between these
two families is also quite obvious, as can be seen in Figure <ref>.

Samples from a pair of similar families

We note that for all three image sizes considered, increasing the number of filters has
minimal effect on the one-convolutional layer CNN accuracy. This could be attributed
to the fact that images in the dataset are in grayscale and have relatively
simple structure, and hence only a small number of filters is required.
In contrast, there is a definite increase in the overall
accuracy when the size of the input
image is increased. However, there are some anomalies,
such as , which is predicted with $100\%$
accuracy for all models based on image size $32\times 32$,
but for $64\times 64$ and $128\times 128$ size images,
samples of this family are consistently misclassified as .

§.§.§ Two Convolutional Layers

This section briefly describes the experiments conducted with
CNN models with two convolutional layers. These models were trained
for similar cases of image size, filters, pooling layer, etc.,
as in the previous section. However, since we have two convolutional
layers, we experiment with different numbers of filters in these layers.
Table <ref> summarizes our results
for these two convolutional layer models.

CNN models with two convolutional layers
2*Image Size      2c|Number of Filters

First Layer     Second Layer

$32\times 32$                32	   32	     0.820

$32\times 32$                32	   64	     0.816

$32\times 32$               64	   32	     0.823

$32\times 32$               64	   64	     0.834

$64\times 64$                32	   32	     0.934

$64\times 64$                32	   64	     0.928

$64\times 64$                64	   32	     0.950

$64\times 64$                64	   64	     0.943

$128\times 128$            32	   32	     0.921

$128\times 128$            32	   64	      0.957

The best results in Table <ref> were obtained with
input images of size $128\times 128$ pixels
with $32$ and $64$ filter maps. This model achieved an overall accuracy
of $95.67\%$.
The classwise accuracies for this case
are summarized in Figure <ref>.

CNN with two convolutional layers,
$128\times 128$ images, and $(32,64)$ filters

Our best two-convolutional layer model in Table <ref>
is just slightly below the best one-layer model in Table <ref>.
Since training time is significantly increased for two-layer models,
and since a one-convolutional layer model performs best,
the one-layer model is the clearly the best among the CNN
architectures that we tested.

§.§ ELM Experiments

This section describes out ELM experiments and results.
Analogous to the CNN experiments discussed in the previous section,
we test a wide variety of ELM architectures. For ELMs, the number of parameters
are relatively few—we experiment with the number of neurons in the hidden layer
and the choice of activation functions.
For each combination of parameters,
we train and test 50 ELMs,
and evaluate the performance in terms of the average accuracy.
In addition, also consider ensembles
of ELMs, we perform experiments to analyze the stability of our models,
and we consider the effect of regularization—in the form of

§.§.§ Input Activation

In this section, we experiment with the input activation function
parameter $\alpha$, which appears in equation eq:inputAct.
Recall that the input activation function
is a linear combination of a so-called MLP kernel
and an RBF kernel, with $\alpha$ determining the relative weighting
of these two kernels.

We trained $50$ ELMs on images of size $64\times 64$, with $\alpha=1$.
For this choice of the parameter $\alpha$, only the highly-efficient
MLP kernel is used. We refer to this as pure MLP

At the hidden layer, we experiment with
the activation functions , , , , ,
and with number of neurons selected from

(128, 256, 512, 1024, 2048, 4096) .

Figure <ref> shows the average accuracies over 50 ELMs
with the specified parameters.

Average accuracy of 50 ELM models ($\alpha = 1$)

From the results in Figure <ref>,
is is clear that the  and  activation functions
consistently perform somewhat better than ,
, and . Moreover, the average accuracies
increase with more neurons in the hidden layer,
until a slight drop from $2048$ to $4096$ neurons.

Despite the fact that the weights $W$
and biases $B$ are selected at random, the ELMs
in these experiments perform surprisingly consistently.
The standard deviations in the testing accuracies
are given in Table <ref>,
and serve to confirm a high degree of consistency,
regardless of the activation function.

Standard deviation over 50 ELM models ($\alpha=1$)
2*Activation     6cNeurons

128     256     512     1024     2048     4096
tanh     0.014     0.013     0.010     0.008     0.096     0.007

relu     0.013     0.011     0.008     0.007     0.007     0.008

softlim     0.016     0.015     0.009     0.009     0.009     0.009

hardlim     0.016     0.011     0.008     0.007     0.008     0.007

multiquadric     0.012     0.011     0.007     0.008     0.005     0.006

In Figure <ref> we compare training and testing accuracies
as the number of neurons increases. In each of these experiments, we
let $\alpha=1$ and the results are an average of 50 ELMs.
We observe that the training accuracy reaches 1.0 with 4096 neurons,
and this holds for all of the activation functions tested.
However, in every case, the test (or validation) accuracy declines
at 4096 neurons, as compared to 2048 neurons. This is a clear sign
of the models overfitting when 4096 neurons are used.

Training/testing accuracy vs number of neurons ($\alpha=1$)

Next, we generate 50 ELMs with the same parameters as above, except
the we use $\alpha=0.5$ instead of $\alpha=1$. In this case
input activation is equally split between the MLP and RBF kernels.
Figure <ref> shows the average accuracies across
these 50 ELM models for the indicated parameters.

Average accuracy of 50 ELM models ($\alpha = 0.5$)

We see that the accuracies for the $\alpha=0.5$ case,
as summarized in Figure <ref>,
are virtually indistinguishable from the results for the $\alpha=1$
case as given in Figure <ref>. In both cases,
the  and  activation functions
consistently outperform , , and .

We also conducted experiments with a pure RBF kernel, i.e., $\alpha=0$,
but the results were no better than the $\alpha=0.5$ case. Furthermore, for
any $0\leq\alpha < 1.0$, the training time is much greater than for the $\alpha=0$
case. This is clearly illustrated in Figure <ref>,
where timings for $\alpha=0$ and $\alpha=0.5$ are given.

Training time for 50 ELMs

The $\alpha=1$ case is the most efficient and it
performs as well as the $\alpha=0.5$ case. We also find that 1024
neurons performs best, and that the
activation outperforms the other activations. Hence, we
have selected the parameters in Table <ref>
for all ELM experiments discussed below, unless explicitly stated otherwise.

ELM parameters

Activation     Mixing     Neurons
$\alpha=1$     $1024$

§.§.§ Majority Vote Ensembles

Ensemble learning consists of combining several models into one,
which can serve to decrease the variance and the bias, thus improving predictions.
In this section, we consider a ensemble technique consisting
of a majority vote of classifiers. Here, the individual classifiers are 50 ELMs
and, again, these are combined by taking a simple majority vote of their decisions.
Since we have 25 classes, it is possible that there could be ties in
the voting, in which case we would simply make a random selection
from among those receiving the most votes. However, we found
no case where there was a tie vote in any of our experiments.

Figure <ref> shows that with increasing number of neurons,
our ensemble accuracy consistently increases. Furthermore, the ensemble classifier
always outperforms the average case for the individual ELMs.

Ensemble classifier accuracy ($64\times 64$ images)

§.§.§ Dropouts

In a fully connected layer, each neuron is connected to all input nodes.
Dropouts are neurons that are ignored during the training phase—an example
appears in Figure <ref>. The neurons to be dropped are chosen at random with a
probability $p$, with typically only a small percentage of neurons
are dropped <cit.>.
Dropouts offer a simple form of regularization, and hence serve to reduce overfitting.

(a) ELM example

(b) ELM with two dropouts

Since ELMs have no backpropagation, we take a slightly different approach
to dropouts in our ELM experiments. We implement an algorithm that
connects each neuron in the hidden layer to four randomly selected
input nodes. This technique results
in a relatively large percentage of connections being lost.
Such an aggressive approach is needed with ELMs, since training
occurs in a single step, as opposed to multiple epochs
in backpropagation.

Figure <ref> gives our average and ensemble accuracies
across 50 ELMs, for both the fully connected and dropout cases.
Dropout ELMs consistently perform better than fully connected ELMs.
In addition, the corresponding ensemble classifiers
consistently outperform the corresponding non-ensemble classifiers.

Fully connected vs dropout ELMs ($64\times 64$ images)

§.§.§ One Dimensional Input

An executable file can be viewed as a sequence of bytes. When
executable files are converted to images, sequential patterns present
in the executables may be lost, depending on the image dimensions.
To preserve these sequential patterns, we consider a different technique for
generating ELM input feature vectors. Instead of treating the Malimg
samples as images, we consider each as a one-dimensional vector
by simply reading the pixels from left to right from top to bottom.
Since the images in the dataset are of varying dimensions,
these one-dimensional vector also differs in length.

We convert all of out one-dimensional vectors to a fixed length $N$
by simply averaging elements. Specifically,
suppose that a vector $V$ is of length $T$. If necessary, we pad $V$
so that it is evenly divisible by $N$. Let $V'$ be this new vector, and
let $T'$ be its length. Then we generate a vector $X$ of length $N$ where

x_i = \frac{v_i' + v_{i+1}' + \cdots + v_{i+N-1}'}{\ell}

where $\ell = T'/N$.
We refer the resulting $X$ as an $N\times 1$ vectors
to emphasize that it is 1-dimensional, as opposed to the
2-dimensional image from which it was derived.
These $N\times 1$ vectors serve as the features for all
experiments in this section.

Our first one-dimensional input experiment
consists of ensemble classifiers comprised of $50$ dropout ELMs.
The input to each ELM is a vector of dimension $512\times 1$.
We train one such ensemble ELM with $512$ neurons, another with $1024$,
and a third with $2048$ neurons. In each case, we use $\alpha = 1$
and  activation functions.
Figure <ref> (a) shows the average ensemble classifier accuracy,
and we see that the best accuracy is achieved for ELMs with 1024 neurons.

We consider an additional set of experiments using 1024 neurons and the same
parameters as in our previous experiment, but varying the input vector length.
Figure <ref> (b) shows increasing the input vector length beyond $512\times 1$
has a minimal effect on the accuracy.
Interestingly, we see little variability in these one-dimensional input length results.
This is in contrast to the CNN results on 2-dimensional images, where
the image size was a significant parameter.

(a) Accuracy vs neurons

(b) Accuracy vs dimension

Average ensemble accuracy

§.§.§ Weighted ELMs

The Malimg dataset is highly imbalanced, as can be seen from
the numbers in Table <ref>. In a classification task with such unequal
number of samples per class, predictions are naturally biased towards
the classes with the most data. To better account for imbalanced
class distributions, a weighted ELM is proposed in <cit.>,
which employs a weighted linear system for the solution of the output layer weights.
To compute the necessary weights, we let

S_j = \text{total number of samples in class } j

S = \sum S_j

where the sum is over all classes in the dataset.
Denote row $j$ of $H$ as $h_j$
and row $j$ of $Y$ as $y_j$
and let $c_j=\sqrt{S/S_j}$. Then we compute

h'_j = c_j h_j \mbox{\ \ and\ \ } y'_j = c_j y_j

and form the weighted matrices $H'$ and $Y'$
from these weighted rows. The weighted ELM is trained
as discussed in Section <ref>, except that
we use $H'$ and $Y'$ and place of $H$ and $Y$.

For our weighted ELM experiments, we consider an ensemble of 50 dropout ELMs.
The individual ELMs have $1024$ neurons, and we use $\alpha=1$
with  activation functions, and one-dimensional input
vectors of size $1024\times 1$. Figure <ref> compares the accuracy
of the weighted and unweighted ensemble classifiers. The average accuracy for the
unweighted model is 96.5%, while for the weighted model we obtain
a slight improvement at 97.7%.

Weighted ELM accuracy ($1024\times 1$ input)

§ DISCUSSION

Our best CNN model achieves an overall accuracy of 96.3%.
This CNN is based on an input image of size $128\times 128$, it uses
one convolutional layer with $32$ filters, each of size $3\times 3$,
a max-pooling layer with a filter of size $2\times 2$,
a dense layer of $128$ neurons, and $25$ neurons in the output layer.
In comparison, our best ELM-based model achieves an accuracy of 97.7%.
This ELM-based model consists of an ensemble classifier built
on 50 ELMs with dropouts, each of which
uses a class weighting technique, with $\alpha = 1$
and $1024$ neurons in the hidden layer. These ensembled ELMs
all use  activation functions and are trained on
one-dimensional input of size $1024\times 1$.
Figure <ref> gives
the classwise F1 score comparison for our best CNN and ELM models.

Classwise F1 scores for CNN and ELM

In addition to outperforming CNNs overall, our
ELM achieves equal or better F1 scores on 18 of the 25 classes.
It is interesting to note that the ELMs are also able to
predict at least some instances of all 25 families,
whereas the CNN sometimes misses an entire family.
For example, in Figure <ref>, we see
that the CNN misclassifies all instances of the  family,
while in stark contrast, the ELM is able to correctly classify all
samples from this particular family.

Finally, we consider the training efficiency of ELMs in comparison to CNNs.
In Figure <ref> we give timings for some of the architectures
discussed above. In each case, the time listed is for training 50 models of the
specified type.

Training time comparison of CNN and ELM

We see that standard ELMs with the MLP kernel (i.e., the $\alpha = 1.0$ case)
can be trained in about 2% of the time required to train a CNN having just a
single convolution layer. In comparison to 2-convolutional layers, we
can train a standard ELM in less than 1.5% of the time required for the CNN.
And, even with the computationally expensive RBF kernel, the ELM can be trained in
less than 30% of the time required by the fastest CNN architecture. Given
that we obtained our best results with an ELM using the MLP kernel, these
results show a clear advantage of ELMs in cases where a large number of models
are to be trained.

§ CONCLUSION

In this paper, we have compared CNNs and ELMs for image-based malware classification.
We considered a variety of different CNN architectures, varying the input image size,
number of convolutional layers, and number of filters. In another set of experiments
we tested the performance of a wide variety of ELM architectures and parameters,
using both two-dimensional images one-dimensional vectors derived from these images.

Although CNNs are widely used for image classification, we found that,
based on a substantial number of experiments,
ELMs were able to outperform CNNs on the well-known Malimg
dataset. The primary advantage of ELMs is faster training times
as compared to CNNs.
In our experiments, one CNN model typically required hours to train, while
one ELM could be trained in a few seconds. This makes ELMs
an option that should be considered for malware classification and
detection, due to the large volume of new malware and variants released each year.
ELMs have many detractors, but our experimental results strongly indicate
that they can be competitive in the malware domain, at least in some situations.

For future work, it would be interesting to consider malware classification
within broad types (trojan, worm, backdoor, etc.), rather than specific families.
Experiments could also be conducted on other malware datasets
and based on other techniques for converting executables to images. Our one-dimensional
feature vector experiments suggest that for ELMs, it might not be necessary to consider
images at all, and hence additional experiments with raw executable files would be worthwhile.
Finally, many additional combinations of parameters could be considered,
both for CNNs and ELMs.