# Holmes: An Efficient and Lightweight Semantic Based Anomalous Email Detector

Peilun Wu1 and Hui Guo4 Email<EMAIL_ADDRESS><EMAIL_ADDRESS>Data Security & Compliance, CDO Data & Cloud, PwC CN.1
School of Computer Science and Engineering, University of New South Wales
(UNSW)14

###### Abstract

Email threat is a serious issue for enterprise security. The threat can be in
various malicious forms, such as phishing, fraud, blackmail and
malvertisement. The traditional anti-spam gateway often maintains a greylist
to filter out unexpected emails based on suspicious vocabularies present in
the email’s subject and contents. However, this type of signature-based
approach cannot effectively discover novel and unknown suspicious emails that
utilize various evolving malicious payloads. To address the problem, in this
paper, we present Holmes, an efficient and lightweight semantic based engine
for anomalous email detection. Holmes can convert each email event log into a
sentence through word embedding and then identify abnormalities that deviate
from a historical baseline based on those translated sentences. We have
evaluated the performance of Holmes in a real-world enterprise environment,
where around 5,000 emails are sent/received each day. In our experiments,
Holmes shows a high capability to detect email threats, especially those that
cannot be handled by the enterprise anti-spam gateway. It is also demonstrated
through our experiment that Holmes can discover more concealed malicious
emails that are immune from several commercial detection tools.

###### Index Terms:

phishing detection, novelty detection, machine learning, intrusion detection,
fraud detection.

##  I Introduction

Though the instant messaging software, such as Facebook and WeChat, has gained
increasing popularity, the email service is still indispensable for
enterprises. Since the email service is a public-facing application, it can be
targeted by the hacker as an easy entrance to the internal network. Based on
our observations, fraud, malvertisement and spread-phishing are the main email
threats frequently received by enterprise users. These emails use deceptive
subjects to pretend and hide themselves. Usually, malware infected attachments
or malicious URLs are embedded in the email body to spoof recipients for
further action. Once the attachment is downloaded or a link is clicked, the
recipients’s system is compromised or the confidential information is leaked
[1].

To alleviate the problem, an enterprise often deploys some anti-spam gateways
to filter out unexpected emails. However, the associated techniques for spam
detection, such as greylist and subject analysis, cannot effectively discover
novel and unknown email threats that are elaborately constructed by utilizing
various current hot topics, such as COVID-19, US election. These unknown
threats can easily bypass the anti-spam gateway and successfully permeate the
target system, leading to a series of damaging consequences, such as
administrator account theft, database attack and financial blackmail.

In this paper, we introduce a novel artificial intelligence based anomalous
email detector, Holmes, that can effectively tackle the challenges mentioned
above. Holmes combines word embedding with novelty detection to discover
anomalous behaviours from a high volume of mirrored SMTP traffic in a large-
scale enterprise environment. To improve the result interpret-ability, we
trace the real source IP addresses of suspicious emails in line with their
geographical positions and further visualize the correlated relations in a
directed-force graph. Our contributions are summarized as follows:

  * •

We propose an efficient and lightweight semantic based anomalous email
detector, Holmes. Different from other detectors that usually require to
examine email bodies, Holmes can discover anomalies simply based on email
headers, which significantly reduces the cost of resource consumption and
avoids accessing email bodies (a sensitive security issue).

  * •

We exploit graph visualization to reveal the correlated relations of detected
suspicious emails and demonstrate that the attacker portrait (based on their
geographical positions) is in line with the cyber threat intelligence
provided.

  * •

We evaluate Holmes with a commercial anti-spam gateway deployed in a real-
world enterprise environment. Holmes not only can accurately detect those
email threats that have been blocked by the anti-spam gateway, but also can
discover a large number of email threats that have successfully escaped from
the gateway. We also compare Holmes with several commercial email detectors
offered by different security vendors in VirusTotal[2], which shows that
Holmes outperforms those detectors with a very high detection rate on the use
of threat hunting in the wild.

The remainder of the paper is structured as follows. We begin with a brief
discussion of some related work on email detection in Section II. We then in
Section III introduce the proposed semantic based anomalous email detector,
Holmes. In Section IV, we present our evaluation results of Holmes and several
commercial security products; a demonstration of how visualization can be used
to reconstruct the attack stories is also given in this section. The
enhancements on Holmes for the real world implementation is given in Section
V. The paper is concluded in Section VI. As an add-on section, we append some
extra discussions at the end of this paper.

##  II Preliminary Knowledge

Anomalous emails can be classified into external threats and internal threats
in accordance with MITRE ATT&CK Matrix [3]. External threats are the emails
sent from external sources, whereas the internal threats are the emails sent
from legitimate users within an organization but whose email accounts have
been stolen and used for the lateral movement attack. Most of previous
research mainly focuses on one specific threat type, such as URL-based lateral
phishing [4] or phishing web pages from search engine in a large-scale
cyberspace [5]. There are still many open questions and unsolved challenges
that need to be addressed holistically. Some issues and the existing solutions
are presented below.

No Built-In Authentication in SMTP. The lack of a native authentication
mechanism inside the SMTP service presents a security loophole to attackers.
Attackers can easily forge the email header by pretending to be someone the
recipient knows or from a business the recipient has a relationship with, so
as to spoof recipients and avoid spam block lists [6]. To address the problem,
several frameworks, such as SPF[7] (Sender Policy Framework), DKIM[8] (Domain
Key Identified Mail) and DMARC[9] (Domain-Based Message Authentication,
Reporting, and Conformance), have been developed to incorporate authentication
into the email system. However, these designs are still not very effective in
terms of implementation. When integrating authentication into the mail system
with a typical component-based software design, there are inconsistency issues
between the software components offered by different parties [10], such as the
incompatibility of mail forwarding servers, which allows numerous email
threats escape detection.

Lack of Sensitivity to Unknown Variations. The unreliability of SMTP leaves
email threats to have evolved into many variations, which are difficult to be
discovered by the traditional security products. We have evaluated several
malicious email detection modules within our internal security products that
use pattern matching of attack signatures for anomaly detection. None of them
can discover the crafted phishing emails that utilize business-related content
to pretend themselves look normal for evasion. We also have used the crafted
phishing samples collected from our real-world hunting to evaluate the
detection rate of 60+ typical detection engines in VirusTotal (Enterprise
Service). Nevertheless, the evaluation result also shows their low sensitivity
to unknown threats – in fact, all testing samples can successfully escape from
the detection of those engines. This kind of low ability of detecting unknown
attack variations has motivated the security community to turn to AI-based
methods for anomaly detection.

High False Positive Rate. The research on anomaly detection for cyber threat
hunting has been around for decades. The main concern on applying machine
learning for anomaly detection is the significant false positive rate (FPR).
Even though new designs are continuously proposed aiming for improvement[11,
12, 13], they were rarely evaluated in a real-world working environment, let
alone put into use in commercial systems.

High Cost and Performance Bottleneck. The imbalance between the cost of data
collection and the performance of algorithmic consumption is a significant
challenge for most of the AI-based detectors. Though the complexity of AI
computing algorithms has been constantly improved, most AI modules still
require large computing and storage resources. which makes the existing attack
detectors not easy to use and very slow to response attacks. Furthermore, the
detectors that use supervised machine learning require the labeled input data
records and often need to be retrained once their performance begins to
degrade, which also makes the machine learning ineffective for detection
automation.

Lack of Provenance Analysis. So far few detectors have considered to integrate
the provenance analysis within the detection mechanism. We believe provenance
analysis is an important and enabling component in malicious email detection.
Provenance analysis[14] can reveal the attack story and the detail of attacker
portrait behind the email, such as (1) where the email is from, (2) who the
real sender is, (3) how the malicious shellcode executes, (4) what the
potential correlations between malicious events are. The above information is
important for the security team to analyze the attack techniques, tactics and
procedures (TTPs) and further assist the security experts to identify
individual attackers or organizations.

Figure 1: Holmes’ workflow and demonstration

##  III Holmes - Anomalous Email Detector

To address the above challenges, we introduce an efficient and lightweight
semantic oriented anomalous email detector, Holmes, that can detect an email
attack by analyzing the sender-recipient relation, which is available in the
email header of SMTP.

Holmes is a threat hunting tool for the incident response and investigation.
It works on mirrored network traffic to inspect and report anomalies in the
bypass but do not block them directly. Holmes is used to assist the incident
response team to discover more concealed threats that can escape from the
detection of traditional anti-spam gateway and threaten the network security.

Holmes is a self-adaptive learning machine. It can learn historical SMTP
traffic from last 24 hours and detect anomalies that deviate from the baseline
of historical behaviour in the next 24 hours. Fig 1 shows the overall Holmes
workflow along with the demonstrations as how data/information is evolved in
the system. Holmes consists of four main functions: 1) Word Embedding, 2)
Novelty Detection, 3) Rareness Selection, and 4) Correlation Graph Analysis.
For each email, Holmes takes its header and converts it into a numeric
presentation through word embedding. The numeric data records are then input
to the machine learning unit (Novelty Detection). The unit generates a list of
novel emails, which are, in turn, processed by the rareness selection
procedure to narrow down the detection targets. The detected results are
finally presented in a human readable format and the correlations of the
related email attacks are also pictured with a graph.

Holmes is also a super lightweight and high-efficient detector, which was
originally written in Python with only 52 lines of code. Based on the run-time
analysis, Holmes can complete the entire detection in less than 73 seconds
with 127 MB memory consumption on around 700 MB datasets. In this section, we
would open-source the original code and detail the implementation of the main
detection functions in a hope to assist researchers to evaluate and reuse
Holmes in their future research.

###  III-A Word Embedding

Since the email textual header information cannot be directly used for machine
learning, how to effectively represent textual data to the machine
understandable is important. Most algorithm engineers use OneHotEncoder[15] or
OrdinalEncoder[16], or Bag-of-Words (BOW)[17], which can be simply implemented
by the open-sourced library Scikit-Learn. However, those methods are not able
to effectively maintain the data semantic correlations in either temporal or
in spatial dimension.

To address the problem, we use paragraph vector (Doc2Vec)[18] for the
conversion, as detailed by the Python code in Listing 1.

⬇

1 def Doc2Vec(self,feature):

2 ’’’

3 :description: convert textual SMTP header to vectors

4 :param feature: SMTP features

5 :return: word vectors

6 ’’’

7 # build a vocabulary for each feature

8 documents = [TaggedDocument(doc, [i]) for i, doc in enumerate(feature)]

9 # set vector size

10 model = Doc2Vec(vector_size=40, min_count=2, epochs=40)

11 # build model

12 model.build_vocab(documents)

13 # train the model

14 model.train(documents, total_examples=model.corpus_count,
epochs=model.epochs)

15 # return vectors

16 return model.docvecs.vectors_docs

Listing 1: Doc2Vec

Compared to other conversion methods, Doc2Vec is able to better keep the
semantics of the words or more formally the distances between the words, which
can be of variable-length ranging from sentences to documents. Doc2Vec is a
semi-supervised learning algorithm. Its input is unlabeled but what will be
learned is specified/supervised. In our code, the inputs are email headers and
what to be learned are the features in the header, as highlighted in blue in
the top-right block in Fig. 1. Besides some basic attributes such as subject,
header.from or user-agent, which are often forged by hackers, we design two
additional features that can also be used to help identify anomalies: the
direction of email (direction) and the country of source IP address
(srcIp.country). The code in Listing 1 converts each email event to a feature
vector, as illustrated in the right side middle boxes in Fig. 1. The feature
vectors are then used for novelty detection.

###  III-B Novelty Detection

Anomalous emails are usually unknown and novel. Their behaviors often deviate
from the trace of historical normal activities. We use Local Outlier Factor
(LOF)[19] to discover those emails, as given in Listing 2 and Listing 3.

⬇

1 def local_outlier_factor(self,train_feature,test_feature):

2 ’’’

3 :description: estimate the decision score for each event

4 :param train_feature: training data

5 :param test_feature: testing data

6 :return decision scores

7 ’’’

8 # initialize the LOF parameters (not sensitive)

9 LOF = LocalOutlierFactor(n_neighbors=20, novelty=True, contamination=0.5)

10 # fit training data

11 LOF.fit(train_feature)

12 # return a list of decision scores of the testing data

13 return list(LOF.decisilst:listing 2on_function(test_feature))

Listing 2: Local Outlier Factor (LOF)

The LOF algorithm shown in Listing 2 can learn the feature vectors of
historical emails (i.e. the train_feature dataset in the code) then provide
the outlier score for newly seen emails (from the test_feature dataset).

There are some compelling advantages of applying LOF for novelty detection:
(1) It allows to train learning model on the data with contamination; (2) It
has a low computing complexity and can be used for online-learning, hence
avoiding the performance degradation and the cost of retraining; (3) It is not
sensitive to fine-tuning, which is beneficial to the effectiveness and
stability of parametric learning.

⬇

1 def novelty_analysis(self,factor,test_feature):

2 ’’’

3 :description: filter novel events in accordance with the decision scores

4 :param factor: decision scores of LOF

5 :param test_feature: testing data

6 :return novel/unseen samples

7 ’’’

8 # initialize threshold as zero

9 threshold = 0

10 # initialize an empty list to store the index of outliers

11 outliers = []

12 # initialize an empty list to store novelties

13 novelties = []

14 # if negative values return the index of outliers

15 for score in factor:

16 if score < threshold:

17 outliers.append(factor.index(score))

18 # return novelties according to the index of outliers

19 for index in outliers:

20 novelties.append(test_feature[index])

21 return novelties

Listing 3: Novelty Analysis

The decision scores from the LOF code can be negative and positive. The
negative values indicate the abnormalities and the positive values indicate
the normal behaviours. We regard any vector with a score smaller than a
threshold is associated with an anomalous email, which is traced by the its
index in the dataset, as shown in Listing 3.

###  III-C Rareness Selection

If we consider the relation of sender and recipient in emails, anomalous
emails are often associated with a weak relation in that a hacker usually does
not send and reply to an email with the same recipient regularly. We therefore
can further narrow down the malicious emails based on the sender-recipient
relation – namely, those abnormal emails with a weak sender-receiver relation
will be selected as the final detection result, as described in Listing 4.

⬇

1 def rareness_selection(self,focus_data):

2 ’’’

3 :description: filter rare events from novel emails

4 :param data: unseen samples

5 :return rare emails

6 ’’’

7 # initialize an empty dict to store the relation

8 relation = {}

9 # initialize an empty list to store the rareness events

10 rareness = []

11 # initialize the features of a relation:

12 for each_data in focus_data:

13 src_ip = each_data[0]

14 mail_from = each_data[1]

15 mail_to = each_data[2]

16 direction = each_data[3]

17 # define the features of a relation

18 each_relation = src_ip + direction + mail_from + mail_to

19 # count the relation

20 relation.setdefault(each_relation, []).append(each_data)

21 # filter the rare events that are smaller than a fine-tuned threshold from
the relation dict

22 threshold = 2

23 for k, v in relation.items():

24 if len(v) <= threshold:

25 rareness.extend(v)

Listing 4: Rareness Selection

In our design, the relationship is measured based on the combination of a set
of email features: source IP (src_ip), direction, sender (mail_from), and
receiver (mail_to). For a strong sender-receiver relation, there should be
many emails of the same IP-direction-sender-receiver value. Therefore, we
count emails for different IP-direction-sender-receiver values and select
those that have a low count value (smaller than a threshold) as malicious
emails, where the threshold is often fine-tuned based on different network
environment by the operation team.

###  III-D Correlation Graph Analysis (CGA)

Most of prior research overlooked a problem: what is the relation within the
anomalies? Lack of an effective solution significantly increases the load of
security analysts, blurs the attacker portraits, and further makes the
provenance analysis difficult. To address the problem, we introduce a
correlation graph analysis (CGA) module to improve the clarity of attacker
portrait descriptions by correlating different anomalous events.

CGA is a directed-force graph and in our design, each node consists of the
selected header features: country, srcIp, sender and subject. The directed
graph enforces the nodes that have dense connections come closer but separates
the nodes if they do not or have sparse connections. The graph depicts the
similarity of different anomalies (such as the same srcIp, same subject or
same sender) and centralizes the cluster in line with their geographical
locations, hence significantly improving the interpret-ability of provenance
analysis.

The graph on the bottom right of Fig. 1 demonstrates the visualization result
of CGA, where two clusters (one in red and one in blue) highlight the
connected components that are centralized in accordance with the country of
srcIp. The blue cluster shows that the same malicious email but sent from
different sources, and the red cluster reveals the same source sends multiple
different malicious emails.

The CGA module can be used to generate active IOCs (Indicator of Compromise)
for the Cyber Threat Intelligence Platform, where we can match the similar or
same malicious incidents occurred to other customers based on the IOCs.

##  IV Evaluation

Holmes has been deployed in an enterprise environment, where it can read
mirrored SMTP records from the Elastic-Search (ES) server. In this section, we
first present some case studies on the malicious emails detected by Holmes and
then show the comparison result Holmes with other popular commercial email
detectors.

###  IV-A Case Studies

According to our monthly email system data, Holmes can discover around 1,000
anomalous emails each day. Among them, about 23% are truly malicious. And most
of the malicious emails contain either phishing links or malware infected
attachments. The rest are mainly spams and only a few are false positives.

Based on the detection results, we derive some malicious emails from our email
server, which were not blocked by the anti-spam gateway but have been
identified by our security analysts as the high risks, to reconstruct the
attack stories. Here, we would showcase some delicate crafted phishing emails
and describe their malicious behaviour in detail.

#### Case A: Fake DHL Delivery Message

Fig. 2(a) shows the execution flow of a malicious email that pretends DHL
service and plays following tricks: (1) The email uses a normally-seen subject
that is associated with an invoice document; (2) The sender information has
been modified as ‘DHL Express’, which can be implemented by some hacking
tools, such as swaks[20] or cobalt strike[21]; (3) The email includes an
attachment named invoice.doc, which is, in fact, a malicious Trojan document
that utilizes the CVE-2017-11882[22] vulnerability; (4) The email contains a
dedicated picture of DHL delivery service to spoof recipients.

In this attack scenario, an attacker who successfully exploited the
vulnerability could run arbitrary code in the context of the current user
(recipient). If the user is logged on with the administrative user rights, the
attacker could take control of the affected system. The attacker could then
install programs; view, change, or delete data; or create new accounts with
full user rights. As we can seen, users whose accounts are configured to have
fewer user rights on the system could be less impacted than users who operate
with administrative user rights.

Figure 2: Case Studies Figure 3: Detection Result of Holmes Compared with
Other Detectors in VirusTotal Enterprise Version

#### Case B: Fake IT Security Alert

Fig. 2(b) shows the execution flow of a malicious email that uses a deceptive
subject named “New Sign-in Attempt”, aiming to spoof recipients to change
their email account password. Once the recipient clicks the button of “Update
security settings”, the web page will be redirected to the phishing website:
<EMAIL_ADDRESS>which induces the
victim user to type in the username and password. The web page will, in the
end, return to the enterprise homepage that the victim user works.

On the hacker side, the back-end server will receive the event log of the
failed login attempts from the victim user, and then record the username and
password. Hence, the hacker can use the legitimate email account to sign in,
such as web page or email server, and can even further send an elaborately
crafted phishing email to a person who is the victim’s frequent contact, which
is hard to be detected by most security products.

#### Case C: Phishing from a Legitimated Website

The malicious execution flow shown in Fig. 2(c) is similar to the attack shown
in Fig 2(b) in that it also has a link embedded in the mail content for
phishing campaign. However, the phishing link
https://armonaoil.com/admin/images/npgtr/newsl/potcpanel[.]html is from a
legitimate website rather than from a personally created malicious website,
which indicates that the legitimate website has been compromised for the use
of darknet market111The darknet is most often used for illegal activities such
as black markets, illegal file sharing, and the exchanging of illegal goods or
services..

By further analysis, we found that the enterprise in this case indeed opened
the cPanel web hosting server for public access, which was vulnerable to the
brute-force attack and remote external control. Furthermore, we examined the
recent activities on some popular darknet markets and found that more than
3,000 sites of cPanel accesses were selling in the darknet market (raidforums)
since 2020-11-22. Hence, it can be confirmed that the email attack is a
phishing campaign caused by the third-party information leakage.

###  IV-B A Comparative Study

To evaluate the detection capability of Holmes, we compare it with some
commercial email detectors that are offered by six key security vendors in
VirusTotal: Microsoft, Tencent, Kaspersky, FireEye, McAfee and Qihoo-360. We
select 15 malicious emails as testing samples. They either contain a phishing
link or have a malware infected attachment. All the testing samples were
collected from the real-world threat hunting during the whole December month
in 2020, and these samples had bypassed the detection of the enterprise anti-
spam gateway and successfully detected by Holmes. More examples can be found
in Table I. The comparison is to demonstrate the proportion of highly-
concealed malicious emails, which the other commercial tools still cannot to
discover. The result shows the evidence and reason why we still need a
behavioural anomaly detection tool like Holmes for anomalous email detection.

The comparison table is given in Fig. 3, where the email subjects representing
the 15 malicious emails are listed in the first column and the rest columns
are the detection results from the commercial detectors and Holmes. A FALSE
value from a detector on a malicious email indicates that the detector failed
to identify the malicious email.

For the detectors of Microsoft, Kaspersky and FireEye, we can see a good
performance on detection of those malicious emails that contain malware
infected attachments. However, they fail to detect the malicious emails that
contain phishing links. Based on the further analysis by our security experts,
most of the phishing domains have been registered no more than three months
and some of them are even from legitimate known enterprises. Moreover, all the
phishing links include a specific URL to access the particular crafted
phishing web page under the domain name that is shortly expired in around
three days. Such a short-lived situation significantly increases the
difficulty of anomaly detection.

From the comparison table, we can also see that McAfee demonstrates a moderate
detection rate on the malicious emails that contain malware infected
attachments. Similar to Microsft, Kasperly and FireEye, McAfee also cannot
detect the malicious emails that contain a newly registered phishing link.

Compared to all above detectors, Tencent and Qihoo-360 have a low detection
rate. Among the 15 malicious emails, only two are detected by Tencent and
three by Qihoo 360.

We would clarify that, the detection engines used for the comparison are
supplied by the VirusTotal Enterprise Service. Since the version of the
detectors may not be the same used in their commercial products, we would
state that the comparison result cannot completely indicate the detection
capability of their latest versions in the commercial products.

TABLE I: Samples of Phishing Emails Detected by Holmes on 2021-10-26. Phishing
Emails Detected by Holmes on 2021-10-26
---
Subject: You have an outstanding payment. |  Subject: Please confirm your email account. |  Subject: Incoming mails has blocked!
|  |
Subject: Closure Of Mailbox Notice ! |  Subject: Security notice - Immediate action required. |  Subject: Security notice - Immediate action required.
|  |
Subject<EMAIL_ADDRESS>Protection |  Subject: Email Account Security Alert Request. |  Subject: Notification<EMAIL_ADDRESS>Disk is full.
|  |
Subject: [WARNING!!!]Your Email Account Is About To Be Terminated. |  Subject<EMAIL_ADDRESS>Password Update. |  Subject: One time verification.
|  |

##  V Enhancements in the Latest Implementation

After the first deployment to the enterprise environment, as mentioned in the
above section, Holmes has been upgraded with a few enhancements. In the latest
version of Holmes, we rebuild the code warehouse that makes Holmes more
efficient to discover anomalies in a much smaller rolling time window. The
improvement is achieved by moving the data query system from Elastic-Search
(ES) server to the real-time Kafka computing platform. The main difference
between ES and Kafka is the way the data is processed. ES uses batch
processing whereas Kafka uses stream processing, and the stream processing is
more timely and efficient. The advantage of using Kafka is that Holmes can
detect anomalies in less than one minute without the risk of server crash,
significantly reducing the computing consumption. Fig. 4 and Fig. 5
respectively shows the detection performance of Holmes in December 2020 and
the run-time performance improvement after switching batch processing to
stream processing. Furthermore, use of Kafka can also help our security
analysts to better schedule time for threat identification and improve the
efficiency of threat responses.

Figure 4: Detection Performance (December 2020) Figure 5: Stream Processing &
Batch Processing

##  VI Conclusion

In this paper, we introduce Holmes, a lightweight semantic based anomalous
email detector, which can effectively discover malicious emails in the real-
world cyber threat hunting. Holmes also demonstrates a viable solution that
successfully transfers AI technology to the cyber security field and makes an
excellent trade-off between the cost of algorithmic consumption and the
detection performance.

We measure the performance of Holmes, and compare its detection capability
with several well-known commercial detectors offered by the security companies
in VirusTotal. Our evaluation result shows that, on the use of cyber threat
hunting, Holmes significantly outperforms those commercial products in a range
of malicious attack scenarios, which demonstrates its practical values in the
commercial competition. Our current evaluation is based on one-month data. As
future work, we will extend our investigation on a large scale dataset that is
collected over a long test period and covers emails of different languages.

## References

  * [1] I. D. Foster, J. Larson, M. Masich, A. C. Snoeren, S. Savage, and K. Levchenko, “Security by any other name: On the effectiveness of provider based email security,” in Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, pp. 450–464, 2015.
  * [2] VirusTotal. "https://www.virustotal.com/gui/home/search".  (accessed: 11.25.2020).
  * [3] MITRE. "https://attack.mitre.org/matrices/enterprise/".  (accessed: 11.25.2020).
  * [4] G. Ho, A. Cidon, L. Gavish, M. Schweighauser, V. Paxson, S. Savage, G. M. Voelker, and D. Wagner, “Detecting and characterizing lateral phishing at scale,” in 28th $\\{$USENIX$\\}$ Security Symposium ($\\{$USENIX$\\}$ Security 19), pp. 1273–1290, 2019.
  * [5] C. Whittaker, B. Ryner, and M. Nazif, “Large-scale automatic classification of phishing pages,” 2010.
  * [6] Barracuda. "https://www.barracuda.com/glossary/email-spoofing".  (accessed: 11.25.2020).
  * [7] M. Wong and W. Schlitt, “Sender policy framework (spf) for authorizing use of domains in e-mail, version 1,” tech. rep., RFC 4408, April, 2006.
  * [8] E. Allman, J. Callas, M. Delany, M. Libbey, J. Fenton, and M. Thomas, “Domainkeys identified mail (dkim) signatures,” tech. rep., RFC 4871, May, 2007\.
  * [9] M. Kucherawy and E. Zwicky, “Domain-based message authentication, reporting, and conformance (dmarc),” ser. RFC7489, 2015.
  * [10] J. Chen, V. Paxson, and J. Jiang, “Composition kills: A case study of email sender authentication,” in 29th $\\{$USENIX$\\}$ Security Symposium ($\\{$USENIX$\\}$ Security 20), 2020.
  * [11] P. Wu, N. Moustafa, S. Yang, and H. Guo, “Densely connected residual network for attack recognition,” 19th IEEE International Conference on Trust, Security and Privacy in Computing and Communications (IEEE TrustCom), 2020.
  * [12] P. Wu, H. Guo, and N. Moustafa, “Pelican: A deep residual network for network intrusion detection,” in 50th Annual IEEE/IFIP International Conference on Dependable Systems and Networks Workshops (DSN-W), pp. 55–62, IEEE, 2020\.
  * [13] P. Wu and H. Guo, “Lunet: A deep neural network for network intrusion detection,” in IEEE Symposium Series on Computational Intelligence (SSCI), pp. 617–624, IEEE, 2019.
  * [14] W. U. Hassan, A. Bates, and D. Marino, “Tactical provenance analysis for endpoint detection and response systems,” in 2020 IEEE Symposium on Security and Privacy (SP), pp. 1172–1189, IEEE, 2020.
  * [15] I. U. Haq, I. Gondal, P. Vamplew, and S. Brown, “Categorical features transformation with compact one-hot encoder for fraud detection in distributed environment,” in Australasian Conference on Data Mining, pp. 69–80, Springer, 2018.
  * [16] P. Cerda, G. Varoquaux, and B. Kégl, “Similarity encoding for learning with dirty categorical variables,” Machine Learning, vol. 107, no. 8, pp. 1477–1494, 2018.
  * [17] Y. Zhang, R. Jin, and Z.-H. Zhou, “Understanding bag-of-words model: a statistical framework,” International Journal of Machine Learning and Cybernetics, vol. 1, no. 1-4, pp. 43–52, 2010.
  * [18] Q. Le and T. Mikolov, “Distributed representations of sentences and documents,” in International conference on machine learning, pp. 1188–1196, 2014.
  * [19] H.-P. Kriegel, P. Kröger, E. Schubert, and A. Zimek, “Loop: local outlier probabilities,” in Proceedings of the 18th ACM conference on Information and knowledge management, pp. 1649–1652, 2009.
  * [20] Swaks. "https://github.com/jetmore/swaks".  (accessed: 11.25.2020).
  * [21] CobaltStrike. "https://www.cobaltstrike.com/".  (accessed: 11.25.2020).
  * [22] CVE-2017-11882. "https://msrc.microsoft.com/update-guide/en-US/vulnerability/CVE-2017-11882".  (accessed: 11.25.2020).


# RRMSE Voting Regressor: A weighting function based improvement to ensemble
regression

Shikun Chen
Mathematics for Engineers, Institute for Technologies of Metals
University of Duisburg-Essen
Friedrich-Ebert-Str. 12, 47119 Duisburg, Germany
<EMAIL_ADDRESS>
Nguyen Manh Luc
<EMAIL_ADDRESS>

###### Abstract

This paper describes the RRMSE (Relative Root Mean Square Error) based weights
to weight the occurrences of predictive values before averaging for the
ensemble voting regression. The core idea behind ensemble regression is to
combine several base regression models in order to improve the prediction
performance in learning problems with a numeric continuous target variable.
The default weights setting for the ensemble voting regression is uniform
weights, and without domain knowledge of learning task, assigning weights for
predictions are impossible, which makes it very difficult to improve the
predictions. This work attempts to improve the prediction of voting regression
by implementing the RRMSE based weighting function. Experiments show that
RRMSE voting regressor produces significantly better predictions than other
state-of-the-art ensemble regression algorithms on six popular regression
learning datasets.

_K_ eywords RRMSE $\cdot$ Supervised Machine learning $\cdot$ Ensemble
Regression $\cdot$ Weighting Function

##  1 Introduction

Ensemble learning refers to a machine learning paradigm where multiple models
(often called "base learner") are trained to solve the same problem and
combined to gain better results, either in classification or regression
problems. The combination can be implemented by aggregating the output from
each model with two objectives: reducing the model error and maintaining its
generalization. Compared to a single model, the ensemble model brings more
robustness and accuracy to final predictions [1].

More formally, an ensemble $\mathcal{F}$ is composed of a set of predictors of
a function $f$ denoted as $\hat{f_{i}}$.

$\mathcal{F}=\\{\hat{f_{i}},i=1,\ldots,k\\}$ (1)

The resulting ensemble predictor is denoted as $\hat{f_{f}}$. In general, the
ensemble process can be divided into three steps [2]. It starts with ensemble
generation, which consists of selecting base learner to be aggregated. If the
base models are chosen using the same induction algorithm then the ensemble is
called homogeneous, otherwise it is called heterogeneous. The second step is
ensemble pruning, the ensemble model is pruned by eliminating some of the
models generated earlier. Finally, in the ensemble integration step, a
strategy to combine the base models is conducted. This strategy is then used
to obtain the final prediction of the ensemble for new datasets, based on the
predictions of the base learners [3]. More specifically, for regression
problems, ensemble integration is performed using a linear combination of the
predictions.

$\hat{f_{f}}(x)=\sum_{i=1}^{k}[h_{i}(x)\ast\hat{f_{i}}(x)]$ (2)

where $h_{i}(x)$ are the weighting functions. The integration approaches can
be divided into constant and non-constant weighting functions [4]. In the
first case, the $h_{i}(x)$ are constants, on the other hand, the weights vary
according to the feature values $x$.

This study introduces the RRMSE Voting Regressor, using RRMSE to determine the
weighting function for heterogeneous ensemble generation. The performance of
this new method is compared to other relevant ensemble regression algorithms.

##  2 Ensemble Learning for Regression

There are basically two groups ensemble methods that are usually
distinguished, the first one is averaging methods, the basic principle is to
build several learners independently and then to average their predictions. By
contrast, in boosting methods, base learners are built sequentially and one
tries to reduce the bias of the combined learner. Since this work focuses on
the weighting function for heterogeneous ensemble models, only the common
averaging methods are discussed below.

###  2.1 Bagging Regression

Bagging [5] is often considered as homogeneous base learners. Each model
learns the error produced by the previous model using a slightly different
subset of the training dataset. By introducing randomization into its
construction procedure and then making an ensemble out of it, bagging is able
to reduce the variance of a base learner [6]. Bagging is based on a
bootstrapping sampling techniques, which creates multiple sets of the original
training dataset with replacement. Each subset is of the same size and can be
used to train models in parallel. Figure 1 illustrates the workflow of the
bagging algorithm. In this work, a heterogeneous based bagging algorithm was
developed, so that it can be used for comparison with the RRMSE voting
regressor.

Figure 1: Bagging consists in fitting several base models on different
bootstrap samples and build an ensemble model that “average” the results of
these base learners.

###  2.2 Voting Regression

Voting regression is another widely used ensemble method. It combines
conceptually different regression models and return the average predicted
values. Unlike bagging, each base model of voting regression fits on the whole
dataset. The way these predictions are combined plays a key role in the final
predictions. Figure 2 shows the structure of voting regression, note that
every base leaner has its own weight.

###  2.3 Related Work

This section describes the related work in terms of the weighting function,
i.e., ensemble integration. Equation 2 shows that the ensemble integration is
done using a linear combination of the predictions. To be more precise, in the
case of constant weighting functions, $h_{i}(x)$ will be replaced by the
coefficient $\alpha_{i}$.

####  2.3.1 Constant Weighting Functions

Constant weight integration functions always use the same set of coefficients,
regardless of the input to the prediction. [7] propose the Basic Ensemble
Method (BEM), which uses as an estimator for the target function.

$\hat{f}_{BEM}(x)=f(x)-\frac{1}{k}\sum_{i=1}^{k}m_{i}(x)$ (3)

where

$m_{i}(x)=f(x)-\hat{f}_{i}(x)$ (4)

Figure 2: Voting consists in fitting several base models on whole dataset and
build an ensemble model that “average” the results of these base learners.

BEM assumes that the $m_{i}(x)$ are mutually independent with zero mean. To
solve this issue, [7] modify the estimator to GBM - Generalized Ensemble
Method.

$\hat{f}_{GEM}(x)=\sum_{i=1}^{k}[\alpha_{i}\ast\hat{f}_{i}(x)]=f(x)+\sum_{i=1}^{k}[\alpha_{i}\ast
m_{i}(x)]$ (5)

where

$\begin{split}\sum_{j=1}^{k}\alpha_{i}=1,\\\
\alpha_{i}=\frac{\sum_{j=1}^{k}C_{ij}^{-1}}{\sum_{l=1}^{k}\sum_{j=1}^{k}C_{ij}^{-1}},\\\
C_{ij}=E[m_{i}(x)\ast m_{j}(x)]\end{split}$ (6)

The disadvantage of this approach is the multi-collinearity problem, which
causes the confidence intervals for the $\alpha_{i}$ coefficients to be wide,
i.e., the estimators of the coefficients will have high variance [4], since it
is necessary to calculate the inverse matrix $C^{-1}$. To avoid multi-
collinearity problem, [8] embed the ensemble integration phase in the ensemble
selection phase. By selecting models from the pool to be included in the
ensemble as a substitute and using the simple average as the weighting
function, the $\alpha_{i}$ coefficients are implicitly calculated as the
number of times each model is selected relative to the total number of models
in the ensemble. [9] take the advantage of principal component regression
(PCR) to prevent the multi-collinearity problem. The PCR method determines the
principal components (PCs) and then selects the number of PCs to use. The
search of the number of PCs to apply is then based on the ordered PCs, and it
is a function of the variation.

####  2.3.2 Non-Constant Weighting Functions

The difference between a constant and a non-constant weighting function is
that the non-constant function uses different coefficients for prediction
depending on the input. The predefined coefficients based on prior domain
knowledge or areas of expertise for each model are called static. On the other
hand, in the dynamic approach, the $h_{i}(x)$ weights from equation 2 are
determined based on the performance of the base learners. Predictors are
usually selected by evaluating their performance against similar data from the
training set for a selected performance measure. [10] and [11] use k-nearest
neighbors with the Euclidean distance and weighted k-nearest neighbors to
choose predictors. Dynamic weighting (DW) [12] assigns a weight to each base
learner according to its localized performance in the group of k-nearest
neighbors and the final prediction is based on the weighted average of the
predictions of the associated models. In this work, a dynamic weighting voting
regressor (DWR) was implemented to compare its performance with the RRMSE
voting regressor.

##  3 RRMSE Voting Regressor

In this section, the definition of RRMSE (Relative Root Mean Squared Error)
will be provided firstly, and then the algorithm of RRMSE voting regressor
will be presented.

RRMSE is derived from RMSE (Root Mean Squared Error), which is defined as
follow:

$\text{RMSE}(f,\hat{f})=\sqrt{\frac{1}{n}\sum_{i=0}^{n-1}(f_{i}-\hat{f}_{i})^{2}}$
(7)

where $\hat{f}_{i}$ is the predicted value of the $i$-th sample, and $f_{i}$
is the corresponding actual value, $n$ is the number of samples, when divide
the difference between the predicted value and actual value
$f_{i}-\hat{f}_{i}$ by $\hat{f}_{i}$, then error will be considered
relatively, which means that each residual is scaled against actual value.

$\text{RRMSE}(f,\hat{f})=\sqrt{\frac{\frac{1}{n}\sum_{i=0}^{n-1}(f_{i}-\hat{f}_{i})^{2}}{\sum_{i=0}^{n-1}\hat{f}_{i}^{2}}}$
(8)

A good ensemble is the one with accurate predictors and making errors in
different parts of the input space. For the regression problem it is possible
to decompose the generalization error in different components, using RRMSE as
dynamic weighting function can guide the process to optimize the ensemble
integration.

A high-level description of the proposed approach is presented in Algorithm 1.
In the first phase, a constant number is calculated to avoid zero division.
Subsequently, the RRMSE of each based learner are evaluated base on equation
8. In the third phase, the weights are assigned inversely proportional to
their total sum. Finally, the final predictions on testing dataset combine the
individual predictions of chosen base learners utilizing the proposed
weighting function.

Input : K - Set of base learners.

X - Set of training features of size $n\times m$

T - Set of testing data

$y$ \- Set of training labels of size $n$

$y^{\prime}$ \- Set of predicted value on each base learner

Output : $\hat{y}$ \- Set of predicted values

Phase 1: calculate a constant to prevent zero division.

Init: $\text{avg}\leftarrow mean(y)$, $\text{sum}\leftarrow 0$

for  _$i$ to $n$_ do

$\text{sum}\mathrel{+}=|(y[i]-\text{avg})|$

end for

$\text{constant}\leftarrow\text{sum}\div n$

Phase 2: calculate set of RRMSE.

Init: $\text{result}\leftarrow 0$

foreach  _$i\in n$_ do

$\text{diff}\leftarrow y[i]-y^{\prime}[i]$ ;

$\text{denominator}\leftarrow y[i]+\text{constant}$ ;

$\text{result}\mathrel{+}=(\text{diff}\div\text{denominator})^{2}$

end foreach

$\text{rrmse}\leftarrow\sqrt{\text{result}\div n}$ Phase 3: assigning RRMSE
weights to each learner

Init: $\text{errors}\leftarrow\text{empty array
[]}\quad$W$\leftarrow\text{empty array []}$

foreach  _$k\in K$_ do

$\hat{y}_{k}\leftarrow\text{predict label using X}$ ;

calculate the constant and rrmse for each $k$ ;

append each rrmse to errors ;

end foreach

$\text{errorList}\leftarrow\text{inverse the element in errors}$;

$\text{sumError}\leftarrow\text{sum of all element in errorList}$;

foreach  _$j\in\text{errorList}$_ do

$\text{$W$.insert}($j$\div\text{sumError})$

end foreach

Phase 4: RRMSE weighted voting prediction

foreach  _$x\in T$_ do

for  _$i=1$ to K_ do

Apply regressor $K_{i}$ on $x$.

end for

Predict the label $\hat{y}$ of $x$ using

$\hat{y}=\sum_{i=1}^{k}[W_{i}(x)\ast K_{i}(x)]$

end foreach

Algorithm 1 RRMSE Voting Regressor

##  4 Experiments

To access the effectiveness of RRMSE voting regressor, experiments were
performed using six well known regression datasets, list in Table 1. Instances
and Features indicate the total number of data points and the number of
predictor variables. The Abalone dataset [13] consists of 8 physical
measurements, and the task is to predict the age of Abalone shells. The Car
dataset [13] contains 7 features and the predicted attribute is "mpg", which
means the fuel consumption per gallon. The Diamond dataset contains the prices
and other 9 attributes of 53940 diamonds. Airfoil dataset [13] obtained from a
series of aerodynamic and acoustic tests of two and three-dimensional airfoil
blade section conducted in an anechoic wind tunnel. The task of Airfoil is to
predict the scaled sound pressure level by using 5 input features. Dataset
Smart Grid Stability [13] includes the data of local stability analysis of the
4-node star system implementing Decentral Smart Grid Control concept. The
Elongation dataset has 17 chemical components as input attributes, the task is
to predict the elongation of steel.

To evaluate the performance of the RRMSE voting regressor, three state-of-the-
art ensemble regression algorithms were also introduced. The first one is
voting regressor with uniform weights (VRU), is implemented in the Scikit-
Learn library [14]. The other two are heterogeneous based bagging regressor
with uniform weights (BR) and dynamic weighting regressor (DWR) [12]. RRMSE,
BR and DWR were implemented by the author in Python.

In addition to the RMSE given in equation 7, to measure the model performance
for regression task, the following metrics are also presented. Mean absolute
error (MAE) is a loss metric corresponding to the expected value of the
absolute error loss. If $\hat{f}_{i}$ is the predicted value of the $i$-th
sample, and $f_{i}$ is the corresponding true value, $n$ is the number of
sample, the MAE estimated over $n$ is defined as:

$\text{MAE}(f,\hat{f})=\frac{1}{n}\sum_{i=0}^{n-1}|f_{i}-\hat{f}_{i}|$ (9)

Mean squared error (MSE): is a loss metric corresponding to the expected value
of the squared error. MSE estimated over $n$ is defined as:

$\text{MSE}(f,\hat{f})=\frac{1}{n}\sum_{i=0}^{n-1}(f_{i}-\hat{f}_{i})^{2}$
(10)

Finally, $R^{2}$ score represents the proportion of variance of $f$ that has
been explained by the independent variables in the model. It provides an
indication of fitting goodness and therefore a measure of how well unseen
samples are likely to be predicted by the model:

$R^{2}(f,\hat{f})=1-\dfrac{\sum\limits_{i=0}^{n-1}(f_{i}-\hat{f}_{i})^{2}}{\sum\limits_{i=0}^{n-1}(f_{i}-\bar{f})^{2}}$
(11)

Table 1: The datasets used in the experiments described in this paper and their properties Dataset | Domain | Instances | Features
---|---|---|---
Abalone | Zoology | 4177 | 8
Car | Automobile | 398 | 7
Diamond | Consumption | 53940 | 9
Airfoil | Aerodynamic | 1503 | 5
Smart Grid Stability | Electricity | 60000 | 12
Elongation | Metallurgy | 385 | 17

For each ensemble algorithm, the same set of base learners are chosen, they
are Linear Regression (LR), K-Nearest Neighbors Regression (KNN), Stochastic
Gradient Descent Regression (SGD) and Random Forest Regression (RF). These
four regression models are implemented in the Scikit-Learn library [14].
Meanwhile, $80\%$ dataset are split for training, and $20\%$ are used as
testing dataset.

##  5 Results

To show the performance of the RRMSE voting regressor and compare it with
other methods, the hyperparameters for all base learners are set to default
values. Table 2 shows the 4 regression metrics and their performances across
different testing datasets. The integer in braces after number indicates the
ranking for that specific dataset over the different algorithms, the number
after algorithm names indicate the average rank of that algorithm with respect
to all metrics (where a lower value indicates a better rank).

From Table 2 it can be seen that the RRMSE voting regressor performs better
than the other 3 algorithms over 5 datasets, except for Abalone dataset
(average rank 2). To assess the degree of difference between the performances
of all 4 algorithms, non-parametric statistical significance tests were
performed. [15] propose Friedman aligned rank test to compare different
algorithms among a set of different datasets. The significance test indicates
a difference among the algorithms with a significance level of $\alpha=0.05$.
Next, a pairwise post-hoc Friedman aligned rank test was performed to explore
which algorithm had differed significantly.

The comparisons between algorithms and the p-values from the pairwise post-hoc
Friedman aligned rank tests are presented in Table 3 which is partitioned in
two parts. A cell in the upper-diagonal of the table indicates the
win/lose/tie counts of the algorithm in the corresponding row with respect to
the algorithm in the corresponding column. This is included for a direct
comparison between algorithm pairs. For instance. RRMSE voting regressor has
better results than VRU in 5 of the datasets and worse in 1 dataset. The lower
diagonal part of the Table 3 shows the p-values of the post-hoc Friedman
aligned rank tests. The asterisk (*) indicates if the comparison was found to
be significantly different. The different levels of significance ($\alpha$)
are defined by the number of asterisks, where * means $\alpha=0.10$, ** means
$\alpha=0.05$ and *** indicates $\alpha=0.01$.

It can be seen that RRMSE voting regressor performed better over all datasets
than DWR with all three significance levels, and better than BR with a
significance level of 0.05. The null hypothesis could be rejected with all
three significance level for VRU.

Table 2: Four regression metrics illustrating performance of the different algorithms compared in this study across the different testing datasets used. Number in brackets indicates relative ranking of the algorithm. Dataset | Methods | MAE | MSE | RMSE | $R^{2}$
---|---|---|---|---|---
Abalone | RRMSE (2) | 1.5102 (2) | 4.7140 (2) | 2.1712 (2) | 0.5645 (2)
| BR (3) | 1.5290 (3) | 4.7995 (3) | 2.1908 (3) | 0.5566 (3)
| VRU (1) | 1.4949 (1) | 4.6174 (1) | 2.1488 (1) | 0.5735 (1)
| DWR (4) | 1.5853 (4) | 4.8962 (4) | 2.2127 (4) | 0.5477 (4)
Car | RRMSE (1) | 1.6758 (1) | 4.7219 (1) | 2.1730 (1) | 0.9122 (1)
| BR (3) | 1.9065 (3) | 6.0156 (3) | 2.4527 (3) | 0.8881 (3)
| VRU (2) | 1.8426 (2) | 5.4789 (2) | 2.3407 (2) | 0.8981 (2)
| DWR (4) | 2.4460 (4) | 9.6812 (4) | 3.1115 (4) | 0.8199 (4)
Diamond | RRMSE (1) | 301.8838 (1) | 330903 (1) | 575.24 (1) | 0.9792 (1)
| BR (3) | 449.6199 (3) | 615979 (3) | 784.65 (3) | 0.9613 (3)
| VRU (2) | 437.4929 (2) | 571903 (2) | 756.24 (2) | 0.9640 (2)
| DWR (4) | 718.8416 (4) | 1473423 (4) | 1213.84 (4) | 0.9073 (4)
Airfoil | RRMSE (1) | 1.8090 (1) | 5.3482 (1) | 2.3126 (1) | 0.8932 (1)
| BR (3) | 3.1896 (3) | 16.0696 (3) | 4.0074 (3) | 0.6794 (3)
| VRU (2) | 3.0898 (2) | 14.9672 (2) | 3.8688 (2) | 0.7012 (2)
| DWR (4) | 3.6852 (4) | 22.9774 (4) | 4.7935 (4) | 0.5414 (4)
Smart Grid Stability | RRMSE (1) | 0.0075 (1) | 0.000095 (1) | 0.0098 (1) | 0.9287 (1)
| BR (3) | 0.0116 (3) | 0.0002 (3) | 0.0146 (3) | 0.8414 (3)
| VRU (2) | 0.0112 (2) | 0.0001 (2) | 0.0140 (2) | 0.8541 (2)
| DWR (4) | 0.0174 (4) | 0.0004 (4) | 0.0219 (4) | 0.6425 (4)
Elongation | RRMSE (1) | 0.8219 (1) | 1.1775 (1) | 1.0851 (1) | 0.6271 (1)
| BR (2.75) | 0.8524 (2) | 1.3431 (3) | 1.1589 (3) | 0.5746 (3)
| VRU (2.25) | 0.8531 (3) | 1.3174 (2) | 1.1478 (2) | 0.5828 (2)
| DWR (4) | 0.9984 (4) | 1.6482 (4) | 1.2838 (4) | 0.4780 (4)

In summary, the results indicate that RRMSE voting regressor was able to
perform better than BR and DWR. Although it was found not be very different
than VRU, based on the test, RRMSE voting regressor achieved the best average
rank.

##  6 Summary and Conclusion

In this paper, a new weighted voting ensemble algorithm for the regression
problem, entitled RRMSE voting regressor, was proposed. The weighting function
for base learners is assigned based on the inverse proportion of their sum of
RRMSE. Statistical significance tests show that the proposed method was able
to perform significantly better than BR and DWR, it seems to also be better
than VRU, but only marginally.

The results in Section 5 show that the RRMSE voting regressor did improve the
performance of ensemble regression and therefore it would now be interesting
to further explore RRMSE voting regressor with the focus of improving the
predictions.

There are many alternatives for base learner, and the setting of
hyperparameters for the individual base learner could also be enormous. The
results motivate the author to extend the method in the direction of better
base learner selection, using Random Search or Bayesian Optimization to
improve the prediction quality of base learners.

Table 3: Significance test on metric MAE, MSE, RMSE and $R^{2}$
Upper diagonal: win/lose/tie. Lower diagonal: post-hoc Friedman Aligned Rank
Test p-values: Significance levels: $\ast$ : $\alpha=0.1$, $\ast\ast$ : $\alpha=0.05$, $\ast\ast\ast$ : $\alpha=0.01$ MAE | RRMSE | BR | VRU | DWR
---|---|---|---|---
RRMSE |  | 6/0/0 | 5/0/1 | 6/0/0
BR | 0.2529 |  | 1/5/0 | 6/0/0
VRU | 0.4142 | 0.7439 |  | 6/0/0
DWR | *** 0.0011 | ** 0.0337 | ** 0.0143 |
MSE | RRMSE | BR | VRU | DWR
RRMSE |  | 6/0/0 | 5/0/1 | 6/0/0
BR | 0.1779 |  | 0/6/0 | 6/0/0
VRU | 0.4379 | 0.5676 |  | 6/0/0
DWR | *** 0.0010 | ** 0.0550 | ** 0.0127 |
RMSE | RRMSE | BR | VRU | DWR
RRMSE |  | 6/0/0 | 5/0/1 | 6/0/0
BR | 0.1779 |  | 0/6/0 | 6/0/0
VRU | 0.4379 | 0.5676 |  | 6/0/0
DWR | *** 0.0010 | ** 0.0550 | ** 0.0127 |
$R^{2}$ | RRMSE | BR | VRU | DWR
RRMSE |  | 6/0/0 | 5/0/1 | 6/0/0
BR | ** 0.04122 |  | 0/6/0 | 6/0/0
VRU | 0.2884 | 0.3271 |  | 6/0/0
DWR | *** 0.00008 | ** 0.0603 | *** 0.00426 |

## Acknowledgments

This research has received no external funding.

## References

  * [1] Nicolás García-Pedrajas, César Hervás-Martínez, and Domingo Ortiz-Boyer.  Cooperative coevolution of artificial neural network ensembles for pattern classification.  IEEE transactions on evolutionary computation, 9(3):271–302, 2005\.
  * [2] Fabio Roli, Giorgio Giacinto, and Gianni Vernazza.  Methods for designing multiple classifier systems.  In International Workshop on Multiple Classifier Systems, pages 78–87. Springer, 2001.
  * [3] Joao Mendes-Moreira, Carlos Soares, Alípio Mário Jorge, and Jorge Freire De Sousa.  Ensemble approaches for regression: A survey.  Acm computing surveys (csur), 45(1):1–40, 2012.
  * [4] Christopher John Merz.  Classification and regression by combining models.  University of California, Irvine, 1998.
  * [5] Leo Breiman.  Bagging predictors.  Machine learning, 24(2):123–140, 1996.
  * [6] Bambang Parmanto, Paul W Munro, and Howard R Doyle.  Reducing variance of committee prediction with resampling techniques.  Connection Science, 8(3-4):405–426, 1996.
  * [7] MP Perrone and LN Cooper.  ªwhen networks disagree: Ensemble methods for hybrid neural networks, º neural networks for speech and image processing.  Chapman-Hall, 1993.
  * [8] Rich Caruana, Alexandru Niculescu-Mizil, Geoff Crew, and Alex Ksikes.  Ensemble selection from libraries of models.  In Proceedings of the twenty-first international conference on Machine learning, page 18, 2004.
  * [9] Christopher J Merz and Michael J Pazzani.  A principal components approach to combining regression estimates.  Machine learning, 36(1):9–32, 1999.
  * [10] Kevin Woods, W. Philip Kegelmeyer, and Kevin Bowyer.  Combination of multiple classifiers using local accuracy estimates.  IEEE transactions on pattern analysis and machine intelligence, 19(4):405–410, 1997.
  * [11] Seppo Puuronen, Vagan Terziyan, and Alexey Tsymbal.  A dynamic integration algorithm for an ensemble of classifiers.  In International symposium on methodologies for intelligent systems, pages 592–600. Springer, 1999.
  * [12] Niall Rooney, David W. Patterson, Sarabjot S. Anand, and Alexey Tsymbal.  Dynamic integration of regression models.  In Multiple Classifier Systems, 2004.
  * [13] Dheeru Dua and Casey Graff.  UCI machine learning repository, 2017.
  * [14] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay.  Scikit-learn: Machine learning in Python.  Journal of Machine Learning Research, 12:2825–2830, 2011.
  * [15] Salvador García, Alberto Fernández, Julián Luengo, and Francisco Herrera.  Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: Experimental analysis of power.  Information sciences, 180(10):2044–2064, 2010.