Monitoring Security of Enterprise Hosts via DNS Data Analysis
Jawad Ahmed
A dissertation submitted in fulfillment
of the requirements for the degree of
Doctor of Philosophy

School of Electrical Engineering and Telecommunications
The University of New South Wales
June 2021

See pages 1 of ./abstract.jpg

See pages 1 of ./dec.jpg

##

To my family

## Abstract

Enterprise Networks are growing in scale and complexity, with heterogeneous
connected assets needing to be secured in different ways. Nevertheless,
virtually all connected assets use the Domain Name System (DNS) for address
resolution, and DNS has thus become a convenient vehicle for attackers to
covertly perform Command and Control (C&C) communication, data theft, and
service disruption across a wide range of assets. Enterprise security
appliances that monitor network traffic typically allow all DNS traffic
through as it is vital for accessing any web service; they may at best match
against a database of known malicious patterns, and are therefore ineffective
against zero-day attacks. This thesis focuses on three high-impact cyber-
attacks that leverage DNS, specifically data exfiltration, malware C&C
communication, and service disruption. Using big data (over 10B packets) of
DNS network traffic collected from a University campus and a Government
research organization over a 6-month period, we illustrate the anatomy of
these attacks, train machines for automatically detecting such attacks, and
evaluate their efficacy in the field. The contributions of this thesis are
three-fold:

  * •

Our first contribution tackles data exfiltration using DNS. We analyze
outgoing DNS queries to identify many stateless attributes such as the number
of characters, the number of labels, and the entropy of the domain name to
distinguish malicious data exfiltration queries from legitimate ones. We train
our machines using ground-truth obtained from a public list of top 10K
legitimate domains and empirically validate and tune our models to achieve
over 98% accuracy in correctly distinguish legitimate DNS queries from
malicious ones, the latter coming from known malware domains as well as
synthetically generated using popular DNS exfiltration tools.

  * •

Our second contribution tackles malware C&C communication using DNS. We
analyze DNS outgoing queries to identify more than twenty families of DGA
(Domain Generation Algorithm)-enabled malware when communicating with their
C&C servers. We identify attributes of network traffic which commences
following the resolution of a DGA-based DNS query. We train three protocol-
specific one-class classifier models, for HTTP, HTTPS and UDP flows, using
public packet traces of known malware. We develop a monitoring system which
uses reactive rules to automatically and selectively mirror TCP/UDP flows
(between internal hosts and malware servers) pertinent to DGA queries for
diagnosis by the trained models. We deploy our system in the field and
evaluate its performance to show that it flags more than 2000 internal assets
as potentially infected, generating more than a million suspicious flows of
which more than 97% are verified to be malicious by an off-the-shelf intrusion
detection system.

  * •

Our third contribution studies the use of DNS for service disruption. We
analyze incoming DNS messages, with a specific focus on non-existent (NXD) DNS
responses, to distinguish benign from malicious NXDs. We highlight two attack
scenarios based on their requested domain names. Using NXD behavioral
attributes of internal hosts, we develop multi-staged iForest classification
models to detect internal hosts that are launching service disruption attacks.
We show how our models are able to detect infected hosts which generate high-
volume as well as low-volume distributed NXD-based attacks on public resolvers
and/or authoritative name servers with an accuracy of over 99% in correctly
classifying legitimate hosts.

Our work shines a light on a very important vector in enterprise security and
equips the enterprise network operator with the means to detect and block
sophisticated attackers who use DNS as a vehicle for malware C&C
communication, data exfiltration, and service disruption.

## List of Publications

During the course of this thesis project, a number of publications have been
made based on the work presented here and are listed below for reference.

Journal Publications

  1. 1.

J. Ahmed, H. Habibi Gharakheili, and V. Sivaraman, “Learning-Based Detection
of Malicious Hosts by Analyzing Non-Existent DNS Responses,” (under review at
IEEE Globecom 2022) (Outcome of Chapter 5)

  2. 2.

J. Ahmed, H. Habibi Gharakheili, C. Russell and V. Sivaraman, “Automatic
Detection of DGA-Enabled Malware Using SDN and Traffic Behavioral Modeling,”
in IEEE Transactions on Network Science and Engineering TNSE, May 2022, doi:
10.1109/TNSE.2022.3173591 (Outcome of Chapter 4)

  3. 3.

J. Ahmed, H. Habibi Gharakheili, Q. Raza, C. Russell and V. Sivaraman,
“Monitoring Enterprise DNS Queries for Detecting Data Exfiltration From
Internal Hosts," in IEEE Transactions on Network and Service Management, vol.
17, no. 1, pp. 265-279, March 2020, doi: 10.1109/TNSM.2019.2940735. (Outcome
of Chapter 3)

Conference Publications

  1. 4.

J. Ahmed, H. Habibi Gharakheili, Q. Raza, C. Russell and V. Sivaraman, “Real-
Time Detection of DNS Exfiltration and Tunneling from Enterprise Networks,"
2019 IFIP/IEEE Symposium on Integrated Network and Service Management (IM),
Arlington, VA, USA, 2019, pp. 649-653. (Outcome of Chapter 3)

  2. 5.

J. Ahmed, H. Habibi Gharakheili, Q. Raza, C. Russell and V. Sivaraman, “Demo
Abstract: A Tool to Detect and Visualize Malicious DNS Queries for Enterprise
Networks," 2019 IFIP/IEEE Symposium on Integrated Network and Service
Management (IM), Arlington, VA, USA, 2019, pp. 729-730. (Outcome of Chapter 3)

## Acknowledgment

In the name of Allah Almighty, the most beneficent and the most merciful. All
praise to Him, who bestowed upon me the wisdom and strength to accomplish this
task gracefully. I owe acknowledgments and thanks to a number of people for
their guidance, support, and prayers that enabled me to conclude this work
successfully.

First and foremost, I would like to express my appreciation to my supervisor
Vijay Sivaraman, had it not been for his constant guidance, instructions,
encouragement, and occasional timely reminders, I would certainly never have
successfully completed this thesis. His support has been extremely valuable.
At the end of this long collaboration, I have got nothing but a lot of respect
for him. The second person I owe a great deal of thanks and respect to is my
joint supervisor, Hassan Habibi Gharakheili, for his insightful and
enlightening comments and opinions; he has been a reliable source of
information and yet a meticulous critic. His expertise played a significant
part in shaping the ideas presented in this thesis. We had many in-depth
technical discussions which were especially valuable for my thesis. I strongly
feel that this journey would not have been possible without the support and
nurturing of Vijay and Hassan. I would like to thank my supervisor at
CSIRO/Data61 Craig Russell for his support and comments on my research work.

I would also like to express my great appreciation to the University of New
South Wales (UNSW) and CSIRO/Data 61 for providing all the resources and a
pleasant atmosphere for conducting research. Without their support, I would
not have been able to embark on this.

I would also like to thank my colleagues Minzhao, Ayyoob, Tara, Arunan,
Chinaei, Iresha, Sharat, Arman, Lance, Qasim, and Asaf, Thank you all for your
fruitful discussions and unwavering support that made this journey a wonderful
one.

I have been extremely fortunate to have wonderful friends who were
instrumental in making my journey to this point a memorable and pleasurable
one. Tariq, Ali, Saad, Talal, Awais, Atif, Haq, Ihsan, Kaleem, Zubair, Akhtar,
Bilal, Ahmed, Ali Raza, Usman, Saqib, Azhar, Yasir Rizwan, Yasir Malik, Badar,
Babar and Sir Taj, thank you for being there for me.

I cannot thank enough my parents for all their selfless love. I would like to
acknowledge the numerous sacrifices they have made for me starting from the
day I was born and continuing to date. I would have been nothing without their
love and compassionate support. I would also like to thank my brother and my
sisters, for their love and affection. Last, but certainly not least, I am
very grateful to my wife and my kids who have been very patient and supportive
in the time I have been away from them. Needless to say without my wife’s
sacrifices, I would not have been able to complete work on my thesis
successfully.

###### Contents

  1. Abstract
  2. List of Publications
  3. Acknowledgment
  4. 1 Introduction
    1. 1.1 Problem Statement
      1. 1.1.1 Data theft using Domain Name Queries
      2. 1.1.2 Malware Command and Control Communication using DGAs
      3. 1.1.3 Service Disruption using Random Domain Names
    2. 1.2 Research Contributions
    3. 1.3 Thesis Organization
  5. 2 Security of DNS Infrastructure
    1. 2.1 DNS Infrastructure: Vulnerabilities and Challenges
      1. 2.1.1 DNS Hierarchy
      2. 2.1.2 Steps for DNS Resolution in an Enterprise Network
      3. 2.1.3 Vulnerabilities and Challenges of Existing DNS Architecture
    2. 2.2 DNS-Based Attacks
      1. 2.2.1 Non Volumetric Attacks
      2. 2.2.2 Volumetric attacks
    3. 2.3 Detection Systems in Cyber-Security
      1. 2.3.1 Knowledge-Based Detection
      2. 2.3.2 Machine Learning-Based Detection: Anomaly Detection
      3. 2.3.3 Limitations of Existing Detection Mechanisms
    4. 2.4 Review on Current State-of-the-Art DNS security
      1. 2.4.1 Monitoring DNS Queries of Enterprise Hosts for DNS Exfiltration
      2. 2.4.2 Behavioral Analysis of DGA-Enabled Malware of Enterprise Hosts
      3. 2.4.3 Identifying Malicious Hosts by Analyzing DNS NXDs
    5. 2.5 Research Gaps in Prior Works
      1. 2.5.1 Novelty of Our Approach
    6. 2.6 Conclusion
  6. 3 Monitoring DNS Queries for Detecting Data Exfiltration
    1. 3.1 Introduction
    2. 3.2 DNS Queries of Enterprise Hosts: Data Collection and Attributes Extraction
      1. 3.2.1 Our Dataset
      2. 3.2.2 Query Name Attributes Engineering
        1. 3.2.2.1 Count of Characters
        2. 3.2.2.2 Entropy
        3. 3.2.2.3 Labels
    3. 3.3 Detection of Anomalous Queries
      1. 3.3.1 Machine Training
      2. 3.3.2 Algorithms and Tuning Parameters
    4. 3.4 Performance Evaluation
      1. 3.4.1 Performance Metrics
      2. 3.4.2 Evaluating Models using Known DNS Exfiltration Data
      3. 3.4.3 Comparing Multi-Class Classifier with One-Class Classifier
      4. 3.4.4 Malicious DNS Queries from Enterprise Networks
    5. 3.5 Real-Time Deployment in Campus Network
      1. 3.5.1 Visualizing Detection of Malicious DNS Queries
    6. 3.6 Conclusion
  7. 4 Automatic Detection of DGA-Enabled Malware
    1. 4.1 Introduction
    2. 4.2 Analyzing Network Traffic Data
      1. 4.2.1 Our Datasets
      2. 4.2.2 DGA-Fueled Malware Families
      3. 4.2.3 Daily Activity Pattern of DGA-Based Domains
      4. 4.2.4 Infected Enterprise Hosts
      5. 4.2.5 Network Behavior of DGA-Fueled Malware
    3. 4.3 Modeling and Mirroring Traffic of Suspicious Malware Servers
      1. 4.3.1 Modeling Traffic Behavior of Malware
        1. 4.3.1.1 Attributes and Classifiers for Malware Flows
        2. 4.3.1.2 Model Training
        3. 4.3.1.3 Model Validation
        4. 4.3.1.4 Models Testing
      2. 4.3.2 Dynamic Traffic Selection using SDN
    4. 4.4 Evaluation Results
      1. 4.4.1 SDN-selected DGA Flows
      2. 4.4.2 Diagnosing DGA Flows
      3. 4.4.3 Infected Hosts Initiating Malicious DGA Flows
      4. 4.4.4 Comparing Our Diagnosis Models with Zeek IDS
    5. 4.5 Conclusion
  8. 5 Learning-Based Detection of Malicious Hosts by Analyzing Non-Existent DNS Responses
    1. 5.1 Introduction
    2. 5.2 Analyzing Two Variants of DNS Random Subdomain Attacks in Our Campus Network
      1. 5.2.1 Attack Scenarios
        1. 5.2.1.1 Attack on Authoritative DNS Servers
        2. 5.2.1.2 Attack on Open Resolvers
      2. 5.2.2 Drawback of Using a Threshold for NXDs Attack Detection
    3. 5.3 Multi-Staged Machine Learning Architecture
      1. 5.3.1 System Design
      2. 5.3.2 Model Training
    4. 5.4 Performance Evaluation
      1. 5.4.1 Performance of Fine-Grained Model
      2. 5.4.2 Performance of Coarse-Grained Model
      3. 5.4.3 Discussion
    5. 5.5 Conclusion
  9. 6 Conclusions and Future Work
    1. 6.1 Conclusions
    2. 6.2 Future Work

###### List of Figures

  1. 2.1 Key concepts covered in this chapter
  2. 2.2 Domain Name Space.
  3. 2.3 DNS Architecture.
  4. 2.4 DNS Attack Categories.
  5. 2.5 DNS Exfiltration Attack - Basics.
  6. 3.1 Number of queries per unique primary domain, over a week (Rsch: 397K, Univ: 1.1M).
  7. 3.2 CCDF of reputation rank: (a) Research institute, and (b) University campus.
  8. 3.3 Real-time number of queries.
  9. 3.4 CCDF of number of characters in query name for: (a) Research institute, and (b) University campus.
  10. 3.5 Scatter density map of numerical fraction of characters vs. total length of query name for: (a) Research institute, and (b) University campus.
  11. 3.6 Scatter density map of upper-case fraction of characters vs. total length of query name for: (a) Research institute, and (b) University campus.
  12. 3.7 CCDF of entropy of query name for: (a) Research institute, and (b) University campus.
  13. 3.8 CCDF of length of labels in query name for: (a) Research institute, and (b) University campus.
  14. 3.4.1 Performance Metrics
  15. 3.10 Number of DNS queries for top malicious domains over a day.
  16. 3.11 Web-UI of our real-time DNS exfiltration and tunneling detector.
  17. 3.12 Filtering queries for a specific primary domain name.
  18. 3.13 Filtering queries with a minimum anomaly score.
  19. 4.1 Aggregate load: (a) packet rate, and (b) bit rate, during peak hour (2pm-3pm) of a weekday (31-May-2019).
  20. 4.2 A weekly trace of DNS queries count: total versus DGA-based queries (from 25-Nov-2019 to 1-Dec-2019) – blue circles highlight representative points to illustrate that count of DGA-enabled queries is at least three orders of magnitude less than count of total DNS queries.
  21. 4.3 Time-trace of DGA-based DNS queries across 75 days (between 16-Sep-2019 and 1-Dec-2019) – green squares represent no DGA queries, red circles represent more than 15K DGA queries, and the black pentagon highlights DGA queries daily count peaks at 30K.
  22. 4.4 Hourly histogram of: (a) total DNS queries, and (b) DGA-based DNS queries, across 75 days (between 16-Sep-2019 and 1-Dec-2019).
  23. 4.5 Time-trace of daily DNS queries count for various DGA-enabled malware families: (a) ModPack, (b) Suppobox, and (c) Ramnit.
  24. 4.6 CCDF of TTL value in DGA-related DNS responses across representative malware families.
  25. 4.7 Communication pattern of a suspicious host with its C&C server of Ramnit family: (a) sequence of HTTPS flows, and (b) sequence of packets in a selected HTTPS flow.
  26. 4.8 Time trace of packets (outgoing/incoming) in a selected HTTP flow.
  27. 4.9  Clusters of K-means model for HTTP flows.
  28. 4.3.1.3 Model Validation
  29. 4.11 System architecture of our detection system.
  30. 4.12 CCDF of number of suspicious flows per host.
  31. 4.13 CCDF of malicious fraction of flows per host in mix-malicious-benign category.
  32. 4.14 Time trace of active malicious flows (between infected internal hosts and malware servers) during the 50-day trial for: (a) aggregate of all campus hosts, and (b) most active infected end-host (top) and a NAT gateway (bottom).
  33. 4.15 Delay between DGA-based DNS responses and commencement of their subsequent TCP/UDP flow.
  34. 4.16 CCDF of attributes: (a) packet count, and (b) duration, of suspicious flows.
  35. 4.17 Daily number of reactive flow entries installed by the SDN controller into the SDN switch, during the 50-day trial.
  36. 5.1 Daily trace of incoming DNS response with errors (typo mistakes: benign.)
  37. 5.2 Daily trace of incoming DNS response with errors (Disposable domains from Sophos antivirus: benign.)
  38. 5.3 Visual illustration of our data collection and various entities identified.
  39. 5.4 Timetrace of incoming NX DNS responses.
  40. 5.5 Time trace of an infected host in our dataset - a sudden rise in NXD responses observed with in an hour only.
  41. 5.6 Zoomed-in time trace of the suspicious host during the hour of interest (8th January 2020 1:30 - 2:30pm).
  42. 5.7 CCDF of number of occurrences of unique NXDs per FQDN for the suspicious host.
  43. 5.8 Time trace of an infected host.
  44. 5.2.1.2 Attack on Open Resolvers
  45. 5.10 Overview of our proposed scheme.
  46. 5.11 Stack plot of number of incoming DNS responses in our campus network over a day (26th Nov 2019).
  47. 5.12 Stack plot of number of incoming DNS responses of a suspicious host over a day (26th Nov 2019).
  48. 5.13 CCDF of number of occurrences of NXDs per host.

###### List of Tables

  1. 3.1 Summary of our dataset.
  2. 3.2 A sample list of malicious and normal DNS queries with unusual length.
  3. 3.3 Number of characters in FQDN for selected domains in our dataset.
  4. 3.4 Entropy value for a sample list of query names.
  5. 3.5 Labels pattern in query names for selected domains.
  6. 3.6 Summary of additional dataset (days 8-14) used for evaluation.
  7. 3.7 Detection accuracy of ground-truth instances after tuning.
  8. 3.8 Anomaly detection for Research institute.
  9. 3.9 Anomaly detection for University campus.
  10. 3.10 Performance of our machine for trusted domains.
  11. 3.11 Anomaly detection combined with whitelisting for Research institute.
  12. 3.12 Anomaly detection combined with whitelisting for University campus.
  13. 3.13 Avg. anomaly score for research institute.
  14. 3.14 Avg. anomaly score for University campus.
  15. 3.15 Avg. time complexity of our scheme.
  16. 3.16 Samples of malicious queries (DET) along with their attributes detected/undetected by the model of the research institute.
  17. 3.17 Anomaly score of queries publicly reported as DNS Exfiltration.
  18. 3.18 Detecting wild malicious DNS queries from two enterprises.
  19. 4.1 DGA-related domain families found in the campus network (from 16-Sep-2019 to 1-Dec-2019).
  20. 4.2 Top ten most frequently used DGA domain names found in the campus network.
  21. 4.3 Enterprise hosts making DGA-enabled queries.
  22. 4.4 DGA-enabled malware and infected campus hosts found by analysis of one-hour PCAP of full campus traffic.
  23. 4.5 Distribution ($\mu$ and $\sigma$) of attributes value for malicious flows in CTU-13 dataset.
  24. 4.7 Accuracy of HTTP models in correctly detecting malicious and benign flows.
  25. 4.8 Accuracy of HTTPS models in correctly detecting malicious and benign flows.
  26. 4.9 Accuracy of UDP models in correctly detecting malicious and benign flows.
  27. 4.10 Distribution of malware families among suspicious flows, selected and mirrored by our SDN system.
  28. 4.11 Results of testing suspicious flows against their corresponding EiF models.
  29. 5.1 Sample of FQDNs used by suspicious host.
  30. 5.2 Anomaly detection by fine-grained model.
  31. 5.3 Anomaly detection by coarse-grained model.

## Acronyms

C&C

    Command and Control
DDoS

    Distributed Denial-of-Service
DGA

    Domain Generation Algorithm
DNS

    Domain Name System
FTP

    File Transfer Protocol
IP

    Internet Protocol
NXD

    Non-eXistent-Domains
SDN

    Software Defined Networking
SSH

    Secure Shell Protocol
TCP

    Transmission Control Protocol
UDP

    User Datagram Protocol

## Chapter 1Introduction

[2] minitocopen

Cyber attacks are becoming more frequent and sophisticated. As a result,
enterprise networks constantly face the threat of valuable and sensitive data
being stolen by cyber-attackers. Sophisticated attackers are increasingly
exploiting the DNS service for malicious activities such as exfiltrating data
as well as maintaining tunneled command and control communications for
malware. This is because DNS traffic is usually allowed to pass through
enterprise firewalls without deep inspection or state maintenance, thereby
providing a covert channel for attackers to encode low volumes of data without
fear of detection. Similarly, attackers are also exploiting DNS for service
disruption such as to launch Distributed Denial-of-Service (DDoS) attacks on
authoritative DNS servers and/or open resolvers - The victim is bombarded with
random DNS to utilize their available resources. A famous example of this
attack is Mirai attack on Dyn DNS architecture (in 2016). Unfortunately,
Network operators lack the methods and tools to determine whether
outgoing/incoming DNS traffic of an enterprise network is legitimate or cyber-
breached.

###  1.1 Problem Statement

DNS is an essential protocol for the Internet that is used to resolve the
domain name like www.domain.com to its corresponding Internet Protocol (IP)
address, e.g. 10.0.0.2. A host needs to ask its local recursive DNS resolver
to resolve a domain. The DNS data is commonly collected at the recursive DNS
resolvers. The DNS protocol restricts the length of the domain name (for
outbound queries) to 255 bytes containing letters, digits, and hyphens. Also,
since the DNS protocol is used mostly over the User Datagram Protocol (UDP),
there is no guarantee that queries will be replied based on their order of
arrival.

####  1.1.1 Data theft using Domain Name Queries

One way for the attacker to exploit DNS is to register a domain (e.g.,
foo.com) so that the attacker’s malware in a host victim can then encode
valuable private information (such as credit card numbers, login passwords or
intellectual property) into a DNS request of the form arbitrary-
string.foo.com. This DNS request gets forwarded by resolvers in the global
domain name system to the authoritative server for the foo.com domain (under
the attacker’s control), which in turn sends a response to the host victim.
This provides the attacker with a low-rate but covert two-way communication
channel between a host victim and their C&C center.

Interestingly, enterprise firewalls are typically configured to allow all
packets on port 53 (used by DNS) since DNS is such a crucial service for
virtually all applications. Some firewalls do offer enhanced DNS protection.
Still, they require deep packet inspection of DNS messages to identify the
covert channel and then isolate domains that contain encoded data. The
significant resources needed for this capability [1] and the resulting impact
on firewall forwarding performance usually results in enterprise network
operators disabling such features. This ability to transit firewalls gives
attackers a covert channel, albeit a low-rate one, by which to exfiltrate
private data and to maintain communication with malware by tunneling other
protocols (e.g., Secure Shell Protocol (SSH), File Transfer Protocol (FTP) to
command-and-control centers. As one example, the remote access trojan DNS
Messenger [2] discovered in 2017 used DNS queries and responses to execute
malicious powerShell commands on compromised hosts.

####  1.1.2 Malware Command and Control Communication using DGAs

Malware-infected machines, forming a botnet, are typically managed remotely by
an adversary (aka botmaster) via a C&C channel. Cyber-criminals primarily use
a botnet for malicious activities such as stealing sensitive information,
disseminating spam, or launching denial-of-service attacks. Therefore, law
enforcement agencies routinely perform takedown operations on the blacklisted
C&C servers [3], disrupting their botnet activities. In response to these
efforts, botmasters have developed innovative approaches to protect their
infrastructure. The use of DGAs is one of the most effective techniques that
has gained increasing popularity [4].

Domain Generation Algorithm (DGA) make use of a “seed” (a random number that
is accessible to both the botmaster and the malware agent on infected hosts)
to generate a large number of custom domain names. Developing numerous time-
dependent domain names and registering only the relevant one(s) “just shortly”
before an attack allows a botnet to shift their C&C domains on the fly and
remain invisible for longer [5]. The botmaster waits for the malware to
successfully resolve a Domain Name System (DNS) query for the registered
domain, enabling the C&C communications to take place. Note that even if a C&C
server is taken offline or blacklisted, this process can simply be restarted
and a new server can come online. To date, more than 80 collections of DGA
domains (each corresponding to a malware family) have been recorded by
DGArchive [6] and are publicly available.

####  1.1.3 Service Disruption using Random Domain Names

glsdns works in such a way that if a query is being asked from the DNS
authoritative name server or open resolver, it is an obligation on them to
answer it even if the query is non-existent on their ecosystem. Non-existent
domains are of two types: (a) Popular search engines and Anti-viruses utilize
random domains to convey the one-time signal to their servers known as
disposable domains (eg,“elb.amazonaws.com.cn”, “cloudfront.net”, and
‘avts.mcafee.com”) - Benign domains also contain typographical mistakes for
example a user accidentally write “googel.com” instead of “google.com”; (b)
Malicious NXDs to launch a type of DDoS attack, DNS water torture attack [7]
also known as random subdomain attack by dynamically generating random strings
as the prefix of a victim domain. The DNS Water Torture Attack is a type of
DDoS attack on DNS servers. This attack affects both authoritative servers and
open/recursive resolvers, but mainly it targets the former.

Cyber actors use bots (compromised devices) to send many randomly generated
domain names on their victim servers. The queried domain names relate to the
primary domain governed by its authoritative name server to tell the IP
address of that particular domain. Due to the high number of requests during
the attack, the victim authoritative servers and/or the recursive resolvers
may have slow responses to the queries being asked or potentially become
unavailable. Although the problem has been well understood over the last
decade, it is mostly dealt with from the perspective of the victim server by
identifying the malicious queries. Therefore, we see this as an opportunity to
detect potentially infected hosts of an enterprise that initiate non-existent
queries to the outside world.

This thesis is structured into four parts to address the above-pointed issues.
First, we review the current literature based on DNS security. We discuss the
available solutions and identify the challenges and limitations in the state-
of-the-art. We then develop tools and models to detect DNS-based attacks in
enterprise networks. After a thorough analysis of real DNS traffic of our
campus network, we extract numerous meaningful attributes that can distinguish
malicious from legitimate queries. We highlight the prevalence and activity
pattern of more than twenty families of DGA-enabled malware across internal
hosts. We develop a monitoring system that uses Software Defined Networking
(SDN) reactive rules to automatically and selectively mirror Transmission
Control Protocol (TCP)/UDP flows pertinent to DGA queries (between internal
hosts and malware servers) for diagnosis by the trained models. Finally, we
draw insights into the use of DNS for service disruption.

###  1.2 Research Contributions

In the context DNS network security, the following can be considered as
significant contributions made by this research:

  * 1)

Our first contribution tackles data exfiltration using DNS. We analyze
outgoing DNS queries to identify many stateless attributes such as the number
of characters, the number of labels, and the entropy of the domain name to
distinguish malicious data exfiltration queries from legitimate ones. We also
develop, tune, and train a machine-learning algorithm to detect anomalies in
DNS queries using a benign dataset of top-rank primary domains. To achieve
this, we have used 14 days-worth of DNS traffic from each organization. We
then implement our scheme on live 10 Gbps traffic streams from the network
borders of the two organizations, inject more than three million malicious DNS
queries generated by two exfiltration tools, and show that our solution can
identify them with high accuracy. We compare our solution with the two-class
classifier used in prior work. (Chapter 3)

  * 2)

Our second contribution tackles malware C&C communication using DNS. We
analyze DNS outgoing queries to identify more than twenty families of DGA-
enabled malware when communicating with their C&C servers. We draw insights
into the behavioral profile of DGA-enabled malware flows when communicating
with C&C servers by analyzing a Packet Capture (PCAP) trace (3.2B packets)
collected during the peak hour from our campus network. We identify malware
traffic attributes and train three specialized one-class classifier models
using behavioral attributes of malware HTTP, HTTPS, and UDP flows obtained
from a public dataset. We develop a monitoring system that uses SDN reactive
rules to automatically and selectively mirror TCP/UDP flows pertinent to DGA
queries (between internal hosts and malware servers) for diagnosis by the
trained models. (Chapter 4)

  * 3)

Our third contribution studies the use of DNS for service disruption. We
analyse incoming DNS messages, with a specific focus on Non-eXistent-Domains
(NXD) DNS responses, to distinguish benign from malicious NXD s. We highlight
two attack scenarios based on their requested domain names. Using NXD
behavioral attributes of internal hosts, we develop multi-staged iForest
classification models to detect internal hosts that are launching service
disruption attacks. We show how our models can detect infected hosts which
generate high-volume as well as low-volume distributed NXD-based attacks on
public resolvers and/or authoritative name servers with an accuracy of over
99% incorrectly classifying legitimate hosts. (Chapter 5)

For each of the schemes developed above, we demonstrate the feasibility and
efficacy of our methods through the implementation in our campus network via
simulation and practical results. We believe that our research will play an
important role in securing the internal hosts of any enterprise network using
its DNS traffic.

###  1.3 Thesis Organization

The rest of this thesis is organized as follows. First, chapter 2 surveys the
landscape of DNS based network security and highlights the shortcomings in the
current state-of-the-art, and discusses that there is still a need for
thorough research based on DNS security in enterprise networks. In Chapter 3,
we propose real-time detection of data exfiltration in enterprise networks.
Chapter 4 gives the extensive data analysis of our campus network to draw
insights into the prevalence of DGA domains. It also discusses how to identify
the infected hosts that are communicating with C&C servers by selectively
mirrored their traffic using SDN. In Chapter 5, we propose an architecture
using multi-staged iForest classification models to detect internal hosts that
are launching service disruption attacks. Finally, we conclude the thesis in
Chapter 6 with pointers to directions for future work.

## Chapter 2Security of Enterprise DNS Infrastructure

With the exponential growth of networking devices in enterprise networks,
cyber-attacks are becoming increasingly sophisticated and intense. DNS is an
essential protocol used by every networking device for address resolution.
Thus, attackers have used DNS to perform malicious activities such as data
theft, C&C communication, and launch DDoS attacks covertly. According to
Forescout Research Labs [8], new DNS vulnerabilities have the potential to
impact millions of devices. This chapter will give a thorough background of
DNS infrastructure, vulnerabilities, and the existing detection mechanisms.
Furthermore, we will identify the existing gaps in the current literature and
why our research is essential to address some of the existing gaps.

Figure 2.1: Key concepts covered in this chapter

###  2.1 DNS Infrastructure: Vulnerabilities and Challenges

Before going into DNS security details, we first discuss the domain name space
and steps involved in DNS address resolution. We then discuss the
vulnerabilities and challenges face by the DNS infrastructure. Fig. 2.1 gives
an overview of key topics covered in this chapter.

####  2.1.1 DNS Hierarchy

The basic function of DNS is to translate a human-readable domain name into an
IP address and vice versa. To achieve this, DNS uses a distributed
hierarchical domain server system as shown in Fig 2.2. It consists of a tree-
based structure with a maximum tree depth of 128. A tree is divided into
zones. On the top, we have a root domain containing all top-level domains
(TLDs) or zones. The top-level domain is further divided into domain names
such as google or yahoo, which we labeled as the second-level domain. Each
domain is divided into subdomains, also known as the third-level domain, such
as docs or scholar. Finally, we have hostnames such as ftp or www as
highlighted in Fig 2.2.

Figure 2.2: Domain Name Space.

####  2.1.2 Steps for DNS Resolution in an Enterprise Network

This section gives all the necessary steps for DNS resolution in an enterprise
network. Large enterprises usually focus on the security of their DNS
infrastructure. Therefore they use recursive resolvers and their authoritative
DNS servers. Fig. 2.3 illustrates a big picture of how a client obtains the IP
address for a web server to get connected with the server within an enterprise
network. The first step is that the client sends a DNS request of
www.scholar.google.com to the recursive resolver of the enterprise. The
recursive resolver has a local cache memory where it saves the responses with
IP addresses for later use. It checks the current DNS request in its cache and
forwards the request to the root server if the domain name is not appeared in
its cache (step 2). The root servers have the resource details of all the top-
level domain servers, such as in this case, it will provide the IP address of
the .com server (step 3). The recursive resolver then contacts the com server
for IP resolution of the second-level domain (SLD) i.e., google (step 4). The
TLD server (.com in this scenario) sends back the IP address of the name
server of google (step 5). The recursive resolver then requests the IP address
of www.scholar.google.com from the Google name server (step 6). The google.com
server responds to the query of recursive resolver with the IP address of
https://www.scholar.google.com (step 7). Finally, the recursive resolver
provides the IP address to the client as a response (step 8), and the client
gets connected to the www.scholar.google.com server.

Figure 2.3: DNS Architecture.

####  2.1.3 Vulnerabilities and Challenges of Existing DNS Architecture

Some enterprises do not restrict their clients and let them configure their
own devices, such as configure the IP address of their local DNS resolver and
change it to any other publicly available open DNS resolver. Similarly, there
is an increase in using personal devices due to the advancement in the
Internet of Things (IoT) i.e., the concept of Bring Your Own Device (BYOD)
that is making a security loophole as far as DNS security infrastructure is
concerned. The attackers take advantage by using DNS as a vehicle to perform
malicious activities covertly.

DNS servers and open resolvers can cache most domain names and their IP
addresses and respond to anyone who may ask for the particular domain name.
However, DNS caches can be easily spoofed and manipulated by the attackers
with the help of Man-in-the-Middle (MitM) attacks such as DNS hijacking and
DNS cache poisoning attacks (explained briefly in the next section).

To resolve some of the aforementioned issues, Internet Engineering Task Force
(IETF) introduced the domain name system security extensions (DNSSEC) in 2005
[9] to enhance the data authentication and data integrity of existing DNS
architecture using public-key cryptography (digital signing) for
authentication. Similarly, DNS over TLS (DoT) [10] and DNS over HTTPS (DoH)
[11] have been introduced by IETF in 2016 and 2018 respectively to encrypt the
traditional DNS communication by using encrypted protocols [12]. However, this
can incur an increased overhead (4 times more bytes in any DNS query or
response) as compared to the traditional unencrypted DNS traffic [13].
Moreover, the signing and verifying operations in DNSSEC take valuable CPU
time and memory. Due to these drawbacks, the adoption rate of these encrypted
DNS technologies is fairly low in enterprise networks.

###  2.2 DNS-Based Attacks

The world of the Internet and networking is exposed to an overwhelming number
of DNS-based cyber-attacks and threats. In Fig. 2.4, we have classified DNS
attacks into two main categories; i.e., non-volumetric attacks and volumetric
attacks. We will discuss the anatomy of these attacks below.

####  2.2.1 Non Volumetric Attacks

This section lists the non-volumetric attacks on DNS protocol that can affect
any enterprise network with financial and reputational damages.

Domain Hijacking: Domain hijacking is a well-known type of DNS-based MitM
attack in which queries are incorrectly resolved by the attackers by
redirecting them to malicious websites. Attackers become successful in this
attack type by: (a) installing malware on the victim’s PC or routers and
changing the DNS server information to redirect the DNS requests attacker’s
control sites; (b) eavesdropping on the local network and sending the spoofed
response to the legitimate DNS request coming from the victim. In this
scenario, when the legitimate response arrives, it is ignored by the local DNS
server.

Another way to hijack a domain is to collect personal information about the
actual owner of the domain to impersonate him and convince the domain
registrar to update the information or transfer the domain to another
registrar they control.

DNS Cache Poisoning Attack: DNS cache poisoning attack aka DNS spoofing is
another example of MitM attack that follows a similar concept as that of DNS
hijacking i.e., to redirect the DNS requests to attackers controlled sites or
DNS servers. The anatomy of cache poisoning attack is to store incorrect and
fake information against the legitimate domain names to trap innocent users
into malicious activities. As a result, the internet traffic goes to the wrong
places until the cached data is fixed.

Figure 2.4: DNS Attack Categories. Figure 2.5: DNS Exfiltration Attack -
Basics.

DNS Exfiltration Attack: The anatomy of the DNS exfiltration attack is shown
in Fig.2.5. We see two hosts, one is benign, and the other is compromised with
malware (such as backdoor). The malware then encodes the personal information
(such as credit card data) in the subdomain of an attacker’s controlled
primary domain (e.g., maliciousDomain.com). The request then goes all the way
from the local resolver of the organization to the attacker’s domain. The
attacker then decodes the encoded sensitive information and uses it for
criminal activities.

DNS Domain Generation Algorithms (DGAs): DGA is a technique used by attackers
to generate new domain names (based on a predefined algorithm) and IP
addresses for malware’s C&C servers. It uses a “seed” (a random number that is
accessible to both the botmaster and the malware agent on infected hosts) to
generate a large number of custom domain names. Generating numerous time-
dependent domain names and registering only the relevant one(s) “just shortly”
before an attack allows a botnet to shift their C&C domains on the fly and
remain invisible for longer [5]. Some common examples of DGA-based malware
include “Gameover”, “Suppobox”, “Ramnit” and “Virut”.

The botmaster waits for the malware to successfully resolve a Domain Name
System (DNS) query for the registered domain, enabling the C&C communications
to take place. Note that even if a C&C server is taken offline or blacklisted,
this process can simply be restarted, and a new server can come online. To
date, more than 80 collections of DGA domains (each corresponding to a malware
family) have been recorded by DGArchive [6] and are publicly available.

####  2.2.2 Volumetric attacks

In DNS data flooding attacks, a common example is the Denial of Service (DoS)
attack that is a simple and powerful method to consume the available resources
of a victim having the intentions to deny its services to legitimate users
[14]. Most of the attacks on the Internet are performed by some of the hidden
software. This software is commonly known as bots [15] [16] [17].

Phantom Domain Attack: Phantom domain attack is another interesting yet
powerful attack that covertly uses DNS to target the victim DNS resolvers
[18]. The anatomy of the phantom attack is that it uses phantom domains that
are being controlled by the attackers, which provide a slow response or no
response at all. The bots are sending thousands of requests to the victim DNS
resolvers to respond to these phantom domains. The DNS server consumes
resources while waiting for responses, eventually exhausting all the available
compute and memory resources.

Random Subdomain Attack: Random subdomain attack aka DNS Water Torture attack
or Slow Drip attack is a type of DDoS attack on authoritative DNS servers
and/or open resolvers in which the victim is bombarded with random non-
existent domain names (NXD) from attackers controlled machines such as bots.
It requests subdomains or hosts that do not exist, which consumes memory and
processing resources- eventually leading to degraded performance or failure.
Mirai botnet is an example of this attack on Dyn DNS architecture in 2016 [7,
19].

DNS Reflection and Amplification Attack: DNS reflection and amplification
attack is a major type of volumetric attack which makes use of the open DNS
resolvers over the Internet to take part in the attack against a specific
victim [20]. The idea is to spoof the identity of the actual attacker so that
the responses can go to the spoofed IP address (real victim) and increase the
magnitude of the attack by asking for more than one type of DNS response which
leads to increased response packet size. Malicious actors send thousands of
spoofed DNS queries to the open DNS resolvers, which respond with the large-
sized responses directed to the victim’s IP address. The victim can be a
stand-alone PC or an authoritative name server which can be overwhelmed by the
unsolicited responses - can cause slow performance or outage depending on the
available resources of the victim.

###  2.3 Detection Systems in Cyber-Security

Intrusion detection systems are the computer tools that are used to enhance
the security of computer networks. IDS plays an essential role in identifying
the network traffic and classifying it as benign or malicious. In addition,
they can identify the intrusions from an external source as well as the
internal intrusions. Detection systems are broadly classified into two main
categories: Knowledge-based/Misuse-based detection and Anomaly-based
detection.

####  2.3.1 Knowledge-Based Detection

The working principle of Knowledge-based (Misuse-based) detection is that it
only classifies the attacks or the malicious content, and the traffic pattern
is previously known. It is also known as signature-based detection because
they depend on the signatures of the attacks provided by the expert in network
security. Snort [21] and Bro [22] are the two open-source signature-based IDS
that are widely used for this specific purpose. The main advantage of using
this type of intrusion detection is that it detects the known attacks without
overwhelming the system with false-positive alarms [23]. However, to detect
the latest attacks on the network, these systems require a continuous feed of
updates in systems. Due to the increase in categories of attacks, it is
challenging to update signatures of all attacks and vulnerabilities in the
network. Another disadvantage of this approach is that it burns many resources
in terms of updating the systems periodically. Thus, there is a definite need
for scalable intrusion detection systems to perform better for high-speed
networks.

####  2.3.2 Machine Learning-Based Detection: Anomaly Detection

Big data analytics has become a hot topic in recent times in the field of
industry and academia. Data has been increasing exponentially with time. Big
data comes into practice where the data sets are extensive and are difficult
to examine by traditional data processing methods [24]. To analyze the big
data, we use the tools and algorithms of Machine Learning (ML).

The anomaly-based detection approach somehow bridges the gap of misuse-based
detection by capturing the attacks that are not seen before and without using
an attack signature. The essential criteria to find the anomaly is that the
system is fed with normal traffic. Now, the system is aware of the behavior of
normal traffic. So, it raises the alarm whenever it observes any traffic that
diverges from this normal behavior. The main drawback of this technique is the
high number of false alarms in some anomaly-based NIDS. The framework in [25]
was designed to investigate the botnets in the network [14]. The authors in
[25] presented a novel system to detect these bots in the overall network with
the help of machine learning techniques by looking at the high-level
information about the network traffic. The framework is based on a learning-
based approach that automatically generates the bot detection models.

####  2.3.3 Limitations of Existing Detection Mechanisms

ML security applications range from identifying malicious activity within a
network to predicting the attack at any period by observing the anomalies in
the network traffic. The bulk of the research falls on one of two sides,
offline analysis on a labeled dataset with multiple ML algorithms [26] [27] or
real-time analysis on lab-grown data with a single ML algorithm [28]. The
trade-off in this spectrum is between computation and timeliness. The former
approach emphasizes mass data extraction (Deep Packet Inspection) but is not
feasible due to the global trend towards end-to-end encryption [29]. Attempts
at using purely flow-level information have yielded mixed results depending on
how the data was captured. [30].

Decision tree classifiers are not suitable for large volumes of benign
traffic. Most of the research works conducted in this area use disproportional
datasets [31]. A typical dataset is the KDD99 which has an attack-to-traffic
ratio of 4:1, which is acknowledged as a significant limitation by the
research community [32]. The advantage of using neural network methods is that
they are not limited to known attacks or signatures [33]. These trained models
can classify unknown attacks with credible success [34]. The two major
disadvantages of this model are the need for large volumes of labeled data and
time, and the predictions are computationally expensive relative to other
models.

###  2.4 Review on Current State-of-the-Art DNS security

This section will focus on the current state-of-the-art DNS security,
specifically command-and-control (C&C) communication, data theft, and service
disruption across a wide range of assets.

####  2.4.1 Monitoring DNS Queries of Enterprise Hosts for DNS Exfiltration

Malware’s Perspective: From a security standpoint, the DNS protocol is an
excellent covert channel. According to IDC 2021 Global DNS threat report, DNS-
based malware is ranked second among commonly used attack vectors following
phishing attacks [35]. Due to its crucial internet role, misconfiguration of
the DNS can lead to network disconnects. Therefore, security policies rarely
restrict it (e.g., allowing resolutions only to specific domain names). In
addition, the DNS protocol is often less monitored than other Internet
protocols (e.g., HTTP, FTP, and mail transfer protocols). It follows that the
use of the DNS protocol as a covert channel has been a part of previous cyber
campaigns, including a 56M credit cards theft from Home Depot in 2014 [36],
and a 25k credit cards theft from Sally Beauty [37]. During the Covid-19
pandemic, financial firms have been affected the most by DNS-based attacks
with an average of over $1 Million [35].

In an enterprise network, the users within the network can still make the DNS
queries out, which can cause the data exfiltration over the DNS. For example,
FireEye has discovered the new malware named Multigrain that has stolen the
credit card data from the point of sale systems and send that information by
exfiltrating data over the DNS [38]. BernhardPOS, NewposThings, and
FrameworkPOS are some other examples.

Feederbot [39] is one from the malware family which has used DNS for sending
its command and control messages to the botmaster [40]. Morto worm [41] has
been used by the attackers for sending the malware commands that were using
the DNS protocol via sending the DNS queries with the Query type of TXT. Wekby
pisloader [42]is also from the bad actors that are using their DNS server to
send and receive the command and control commands to and from the compromised
hosts. The information is encoded in the DNS requests. A remote access trojan
named DNSMessenger [43] was discovered in 2017, which uses DNS queries and
responses to execute the malicious PowerShell commands on compromised hosts.
It is a sophisticated attack as it does not require any file writing to the
compromised hosts. The technique uses DNS TXT records to extract the
PowerShell commands saved as DNS TXT records remotely.

Malicious Domains’ Perspective: DNS traffic has been analyzed to identify
malicious network activities [44, 45]. Studies in [46, 47] survey the
available research literature on the misuse of DNS protocol for various
attacks. Common malicious activities that utilize DNS include command and
control (C$\&$C) traffic tunneled over DNS channel, circulating spam messages,
transferring credit card numbers (or other sensitive information), and hosting
scams and phishing websites [48, 4]. Therefore, it is important to profile and
detect these malicious activities. Over the last decade, there has been an
increasing amount of works [49, 50, 51, 52, 53] on identifying these malicious
activities mostly related to C$\&$C communications [40, 54] and phishing [55].

Machine Learning Based Detection of DNS Exfiltration and Tunneling:

Researchers have used three categories of methods for detecting DNS
exfiltration and tunneling, namely statistical-based techniques [56, 57, 58,
59, 60], supervised multi-class classification [61, 62, 63, 64], and
unsupervised one-class classification [65, 66].

Work in [56] proposed a method to find maximum information that can be encoded
in a sub-domain portion of a DNS query name to detect whether the query
contains encoded data or not. The authors used an information-theoretic
approach, namely the use of Kolmogorov complexity. The authors established an
upper bound on the volume of surreptitious communication by investigating
inter-query time and query record type. In [57], authors employed mutual
information and principal component analysis for dimensionality reduction
based on consecutive DNS request and response sizes. In [58, 59, 60], authors
have proposed DNS tunnel detection using character frequency analysis.
However, the detection criteria are based on the threshold value for which
attackers can go undetected easily.

In [61, 62] the authors employed a supervised learning-based model with
logistic regression to classify queries as either normal or exfiltration.
Buczak et al. [63] used the Random Forest algorithm for the two-class
classification of benign and malicious DNS queries. Similarly, Samuel et al.
[64] proposed a model to detect malicious DNS query names (generated by
malware-infected machines) using Random Forest. However, attributes used in
prior works to train the model are stateful (such as tracking the inter-
arrival time of DNS packets or the frequency of query type) or require both
DNS query and response messages (such as response length) [61, 63]. Also, this
body of work essentially trains a model with both benign and malicious
instances (i.e., a two-class classifier) and the accuracy of detecting
malicious queries dropped when a new family of attack is introduced (e.g.,
model accuracy varied from 27% to 75% depending upon model parameters in
[63]).

Deep Learning Based Detection of DNS Exfiltration

Over the past two years, the interest of researchers on DNS exfiltration has
shifted from ML-based detection systems to deep learning-based detection
systems [67, 68, 69, 70]. Work in [70] employs a combination of Convolutional
Neural Network (CNN) and Long Short-Term Memory (LSTM). For LSTM, authors use
one hidden layer and they assumed that the first 128 bytes/characters contain
the actual message information; therefore, they set the length of the hidden
layer to 128. Similarly, for CNN, the authors used three convolutional layers,
two max-pooling layers, and one softmax layer. Their model is trained on two
classes of FQDN, i.e., benign and malicious. Our model instead is trained on
attributes of benign FQDNs, resulting in a one-class classifier. The key
difference is its way of detecting anomalies and ability to detect zero-day
attacks (i.e., it flags all anomalous domain names that deviate from the
attributes of the benign domain names). Although deep learning-based
approaches give a high percentage of accuracy, they require a huge amount of
data (typically more than a million instances) for improved performance.
Furthermore, Deep Learning-based methods demand an increased computing
resources and add more complexity to operational networks [71, 72].

####  2.4.2 Behavioral Analysis of DGA-Enabled Malware of Enterprise Hosts

Malware behavioral analysis has been widely studied by many researchers [73,
74, 75, 48, 4, 25, 76] using different tools and techniques [77]. The most
relevant works to ours can be divided into three categories: (a) detection of
malicious traffic based on unusual DNS queries (predicting the presence of DGA
domains) [78, 48, 4, 79, 64, 76, 80, 81, 82, 83, 84], (b) network behavioral
analysis of known malware and botnet by inspecting their traffic data and/or
metadata [85, 86, 87, 25, 74, 75], (c) use of SDN and/or programmable
networking in detecting cyber-attacks [88, 89, 90, 91].

Malicious DNS Queries: DNS traffic has been analyzed to identify malicious
network activities [45, 48, 92]. Over the past decade, there has been an
increasing number of works [49, 50, 51, 52] on detecting malicious network
activities mostly related to DNS exfiltration, DNS tunneling, and C$\&$C
communications [40, 54, 44, 93, 94].

In the past, blacklists were used to detect C&C communications between servers
and infected hosts. However, blacklisting has been defeated by attackers since
they migrated from a static domain mapping to the use of algorithmically
generated domain names. In response to this change, researchers have attempted
to automatically detect DGA domains using statistical modeling of DNS traffic
[78, 48, 4, 79, 64, 76], or machine/deep learning techniques [80, 81, 82, 83,
84, 95]. Antonakakis et al. [4] develop a clustering-based method for
detecting new (unknown) algorithmically generated domains (AGD) as well as
classifying known AGDs using supervised learning. The authors evaluated their
proposed solution in a large ISP network and found several new families
(unseen before) of DGAs, operating on the network. They used statistical
attributes including entropy and n-grams measures as well as structural
attributes such as length and label count in the domain name, extracted from
NXD (non-existent domains) responses on a per-host basis. Schuppen et al. [64]
employ manually-engineered features to train a binary classifier using Random
Forest algorithm to determine whether domains in NXDomain-failed DNS queries
(i.e., queries to non-existent domain names) are benign or malicious. It
remains unclear how this model performs in classifying unseen malicious
domains. We note that generating a rich training dataset of benign NXDs is
nontrivial, and a labeled benign dataset may get polluted by some new (but
unknown) malicious NXDs.

Malware and Botnet Behavioral Analysis: Flow-based analysis has been used to
detect malware and botnet traffic. However, monitoring high-rate network
traffic in large and complex enterprise environments is difficult and
computationally expensive. Works in [85, 86, 87, 25, 96] develop flow-based
features like packet count and distribution of packet length by analyzing
every packet of the network to model the patterns in encrypted traffic.
Similarly, Anderson et al. [74, 75] employed supervised learning algorithms to
classify malware and benign traffic. Their model was trained by a variety of
host-level attributes, including packet size, flows inter-arrival, DNS query
(e.g., TLD, TTL, domain rank in Alexa), HTTP data (such as server code,
content type, Accept-language, and location), and TLS data (such as TLS cipher
suites and TLS extension) that are measured over a while.

Our behavioral modeling approach differs from prior works in three ways: (a)
we only monitor the behavior of selected flows pertinent to certain servers
(resolved by DGA responses) instead of monitoring all traffic of every host on
the network, (b) we choose to extract statistical attributes of flows
(encrypted and unencrypted) to train our models, without the need for
inspecting payloads like HTTP content type or TLS handshake cipher keys, and
(c) our models are built by one-class classification algorithms and hence
become sensitive to changes in any attribute while multi-class models become
sensitive to changes in only discriminative attributes. Also, it is essential
to note that we only use a database of known DGA domains and check domain
queries against this database in real-time instead of classifying a domain as
benign or malicious. Our objective is to detect infected hosts of enterprise
networks by monitoring their selected flows.

Programmable Networking for Network Security:

Software-Defined Networking (Programmable Networks) is an evolving paradigm to
manage the networks in a better way. SDN is a technology that decoupled the
control plane with a data plane (i.e. switches and routers). Now the network
designs are not vendor-centric that was the case in traditional networks.
Then, the network operators had to configure every device with low-level
commands individually.

Programmable networking has recently gained popularity among researchers,
specifically for network security use-cases [88, 89, 90, 91]. Gupta et al.
[88] develop scalable telemetry (i.e., partitioning different types of traffic
such as TCP and ICMP) that can be used to collect and analyze the network
traffic in real-time using a programmable data-plane. Furthermore, the authors
show that their approach reduces the workload of the overall system to be able
to operate at a line rate. Similarly, Zhang et al. [91] proposed an approach
that uses P4 programmable switches for a better defense mechanism against DDoS
attacks. The authors considered the case of volumetric DDoS attacks. They
provided the defense strategies in a modular fashion that can be adopted for
each network and can be used for new defense strategies other than just the
DDoS. Although these prior works primarily leverage the programmability
features in the data plane, we instead employ the programmable control-plane
available by Openflow-based SDN to dynamically select suspicious flows for
diagnosis by trained machine learning models.

In a relevant work, [97], Ceron et al. developed an automated system for
offline malware analysis, recording the network behavior of a given malware in
a controlled sandbox environment orchestrated by an SDN controller. A known
host on their sandbox is infected by malware (from a set), and the SDN
controller inspects every packet from/to this infected host for taking
required actions like rate-limiting, blocking, or re-configuring the topology,
upon finding certain patterns in the packet payload (i.e., regex signature) or
headers (e.g., contacting specific IP address and/or TCP/UDP port numbers).

####  2.4.3 Identifying Malicious Hosts by Analyzing DNS NXDs

Identifying Malicious Queries: To resolve the NXD attacks, specifically water
torture attacks, researchers come up with significant countermeasures such as
rate limiting and IP address blocking. However, it can severely affect the
legitimate users since the malicious queries can forward through the open
resolvers or ISP cache servers. Therefore, the IP address of the cache server
will be blocked, and the legitimate users will not have access to the ISP
cache server. Similarly, if the volume of queries exceeds the limit set in
rate-limiting, it will block all queries, even from legitimate users.
Researchers [98, 99] have also examined domain names to detect the malicious
queries by focusing on NXD error responses only. Kazato et al. [98] predicted
whether a domain name included random words via a score calculated by
comparing bigrams of domain names of malicious domains and those of benign
domain names.

Identifying Attack on DNS Servers: A group of researchers [100, 99] have
identified NXD attacks on DNS servers by setting the threshold on the number
of non-existent domains. The approaches can be fruitful for heavy volume
attacks (such as bursty data) - moreover, choosing a threshold value would be
challenging. However, this approach does not work efficiently for the
lightweight and distributed NXDs, bypassing the threshold-based security
systems.

###  2.5 Research Gaps in Prior Works

A considerable number of previous research investigations have focused on the
detection of malicious domains [49, 50, 51, 52]. The techniques used are
mostly based on passive DNS analysis. Leyla et al. [50] came up with the
EXPOSURE system to detect the suspicious domains by extracting 15 features
from the DNS traffic. The features are divided into four main categories i.e,
DNS query-name based, time-based, answer based and TTL based. The authors have
claimed that their system has been validated in a real-world dataset
containing 100 billion DNS requests. The system captures the misused
suspicious domains taking part in botnet command and control and spamming.
Manos et al. [51] proposed a similar method by analyzing the passive DNS
queries. They gave NOTOS as a name to their approach, a dynamic reputation
system for the DNS domain names. They have extracted 41 features grouped into
two main categories i.e, network-based features and zone-based features. The
main idea is that malicious domains have unique characteristics that can
easily be separated from legitimate domains based on their extracted features.
Such approaches, however, have failed to address the information theft over
the DNS queries, and it does not detect the malicious domains in real-time.

Manos et al. [48] proposed another system called Kopis, which is operating at
the upper DNS hierarchy i.e, the authoritative name servers and Top-level
domain servers, in contrast to NOTOS and EXPOSURE, which were operated at the
local recursive DNS servers. This approach has enhanced the visibility of the
DNS messages. The same process is followed by [4] to capture the DNS traffic.
However, the aim is to identify the DGA (Domain Generation Algorithms) bots by
focusing only on the DNS queries with name error responses i.e, NXDomain
responses. Although extensive research has been carried out on malware
detection over DNS protocol at the upper DNS hierarchy, the features
considered may not be that effective at the network level.

We believe that a two-class classification approach (i.e., signature-based) is
insufficient to address new and increasing types of attacks. Also, obtaining
“ground truth” on a diverse set of malicious instances to train the classifier
is difficult [101]. The authors of [65] employed unsupervised machine learning
algorithms (i.e., one-class support vector machine and k-means) to detect DNS
tunneling. Their primary focus was to identify infected mobile devices using
stateful attributes, including the time between a DNS query and its
corresponding response and the size of individual devices’ DNS query/response.
In [102], Homem et al. benchmarked the performance of four algorithms (multi-
class decision trees, support vector machine, K-nearest neighbors, and neural
networks) in identifying tunneled traffic (e.g., HTTP, HTTPS, and FTP) over
DNS. The authors used only three attributes of DNS packets, including the size
of IP packet, length of query name, and entropy of query name. Similar to our
approach, Nadler et al. [66] proposed an anomaly-based solution to detect low
throughput data exfiltration over DNS. This work evaluated the performance of
isolation forest and support vector machine learning algorithms. However, the
authors maintain several attributes for each primary domain over the last $n$
hours (e.g., rate of A and AAAA records, the average length of query name).
This makes it difficult to detect malicious queries in real-time.

####  2.5.1 Novelty of Our Approach

To the best of our knowledge, our work in chapter 3 is the first that presents
a thorough analysis of attributes for query names from operational enterprise
networks. Our focus is on attributes of fully qualified domain names that can
be extracted in “real-time”, without a need for states (i.e., “stateless”) –
we assume that DNS traffic is not encrypted over TLS. We also provide
fascinating insights into the practical considerations of such a detection
scheme. Our scheme can be extended by collecting states only for those hosts
that generate anomalous queries and ultimately mitigate malicious DNS
tunneling/exfiltration – such mitigation is beyond the scope of this research.

Our work in chapter 4 develops an automatic SDN-based system for detecting
malware-infected hosts in real-time by relying more on the behavioral activity
profile of “selected flows” rather than the content of packets. Further, our
SDN switch does not send any network packets to the controller ( allowing the
solution to scale to high rates). Instead, packets that need to be inspected
in the software are sent as copies on a separate interface of the switch, to
which a software inspection engine is attached.

In chapter 5, we have developed a monitoring system at the source side that
can detect the internal hosts generating high volume NXD attacks as well as
distributed NXD attacks by using multi-level machine learning algorithms.
Existing research focuses on detecting heavy volume NXD attacks at the
reception side i.e., the authoritative name server or the queried domain
(primary domain). Instead, our work is to detect the enterprise hosts
generating attack traffic to other networks to protect our campus network’s
reputation.

###  2.6 Conclusion

Domain Name System (DNS) is an essential application layer protocol used by
every Internet-connected device to obtain the IP address of web servers.
However, due to its prevalence and importance, DNS traffic is rarely inspected
by intrusion detection systems, making this protocol vulnerable to various
cyber-attacks. In this chapter, we have studied and characterized DNS-based
attacks. We have also discussed the detection mechanisms and the underlying
challenges and gaps in the existing research. We investigate some of these
critical research problems in the rest of this thesis, beginning with data
theft in enterprise networks.

## Chapter 3Monitoring Enterprise DNS Queries for Detecting Data Exfiltration
from Internal Hosts

In the previous chapter, we highlighted the importance of DNS security in
enterprise networks. In this chapter, we analyze DNS data to draw insights
into the characteristics of exfiltrated domains and then develop a machine
learning-based method to detect the exfiltrated domains in real-time in two
enterprise networks. Parts of this chapter have been published in [44, 94,
93].

###  3.1 Introduction

Cyber-criminals have exploited DNS to maintain covert communication channels
with compromised hosts. The resulting damages can be huge, amounting to
several million dollars in a single attack [103]. Based on a recent DNS
security survey of Infoblox [104], 46 percent of the businesses of North
America and Europe have exploited by DNS exfiltration, and about 45 percent
affected by DNS tunneling. Several high-profile data exfiltration breaches
have been reported since 2014: the Sally Beauty breach (a theft of 25K credit
cards) [37] and FrameworkPOS malware (a theft of 56M credit cards from Home
Depot) [36] in 2014, BernhardPOS malware [105] in 2015, MULTIGRAIN malware
[38] in 2016, Win32.Backdoor.Denis [106] in 2017, and UDPoS Malware [107] in
2018. In addition, there have been several DNS tunneling incidents in which
malware actors used their DNS servers to send and receive the command and
control commands to and from compromised hosts. Examples include Feederbot
[39], and botmaster [40], Morto worm [41], and Wekby pisloader [42].

This chapter develops and validates a mechanism for real-time detection of DNS
exfiltration and tunneling in two operational networks – a large University
and a mid-sized Government Research Institute. Our first contribution is to
collect and conduct a thorough analysis of real DNS traffic from the two
organizations over several days and extract stateless attributes of DNS
messages, such as length, entropy, dots, numerics, uppercase characters, and
the number of labels, that can distinguish malicious from legitimate queries.
Our second contribution is to develop, tune, and train a machine-learning
algorithm to detect anomalous DNS queries based on the above attributes using
a known dataset of benign domains as ground truth based on 14 days worth of
DNS data from the two organizations. For our third contribution, we implement
our scheme on live 10 Gbps traffic streams from the network borders of the two
organizations, inject more than three million malicious DNS queries generated
using two exfiltration tools (our customized tool and an open-source tool) and
show that our scheme can identify such malicious activity with high accuracy.
We also show our one-class classifier outperforms an existing two-class
classifier in detecting unknown DNS exfiltration attacks. We draw insights
into anomalous DNS queries detected by our models, looking into their anomaly
scores, tracking query counts in real-time, the number of enterprise hosts
querying them, and investigating the TTL/Type fields of their corresponding
responses. We make our tools and datasets available to the public to
facilitate further Research into this area.

###  3.2 DNS Queries of Enterprise Hosts: Data Collection and Attributes
Extraction

In this section, we first analyze the characteristics of DNS traffic (with a
specific focus on query names) collected from the border of two enterprise
networks, a medium-sized research institute and a large University campus. In
both instances, the IT department of the enterprise provisioned a full mirror
(both inbound and outbound) of their Internet traffic (each on a 10 Gbps
interface) to our data collection system from their border routers (outside of
the firewall). We obtained appropriate ethics clearances for this study (UNSW
Human Research Ethics Advisory Panel approval number HC17499, and CSIRO Data61
Ethics approval number 115/17). We extracted DNS packets from each enterprise
Internet traffic stream in real-time by configuring rules to match
incoming/outgoing IPv4 and IPv6 UDP packets on port 53 in an OpenFlow switch.
The study here considers data collected over one week from 30-07-2018 to
05-08-2018.

####  3.2.1 Our Dataset

Table 3.1 shows a summary of our dataset from each organization. We captured a
total of 249M and 589M DNS packets from the border of the two networks and
stored them in daily CSV files – each row in our dataset represents a
timestamped DNS packet including headers and payload. The data shows that 17%
of total DNS traffic is carried over IPv6 packets in both networks. Also, more
than a third of our records correspond to outgoing DNS queries generated by
enterprise hosts – i.e., 86.9M and 221M in the Research and University
networks, respectively. We note that our dataset also contains queries for
unqualified domain names (i.e., 900K and 1.5M respectively in the Research and
University networks) that are discarded in our analysis – we use the cleaned
dataset. Unqualified query names contain no delimiting dots (e.g.,
“top_10_banks_offering_attractive_home”) or their top-level-domain is pure
numeric (e.g., “129.178”). After removing unqualified names, outgoing DNS
queries in total span respectively 2.2M and 6.2M distinct fully qualified
domain names (FQDN).

Table 3.1: Summary of our dataset.

Research University Total DNS packets 249M 589M IPv4 DNS packets 206M 489M
IPv6 DNS packets 43M 100M DNS queries 142M 341M DNS responses 107M 248M Total
Outgoing DNS queries 86.9M 221M Outgoing DNS queries (IPv4) 69.7M 177M
Outgoing DNS queries (IPv6) 17.2M 44M Outgoing DNS queries (only qualified)
86M 219.5M Unique query names (FQDN) 2.2M 6.2M Unique primary domains 397K
1.1M

These FQDNs are rooted themselves in 397K and 1.1M distinct primary domains
(i.e., one level under “com” or “co.uk”). Fig. 3.1 shows the number of queries
for each unique primary domain over the entire dataset, ordered from most
queried on the left, to least queried on the right. There is a small number of
domains on the left that predominate with very high query counts, followed by
a long-tail of domains, all of which receive a fairly small number of queries
(i.e., less than 1000 over a week). It is seen that the top 4K (out of 397K)
and 9K (out of 1.1M) domains respectively in the research institute and the
University comprise the head in their respective curve. For example, only
three domains namely “akamaiedge.net”, “in-addr.arpa”, and “akadns.net”
contribute to 15% of total queries generated by University hosts. In the
research network, on the other hand, top three domains of “kaspersky-
labs.com”, “kas-labs.com”, and “in-addr.arpa” contribute to 17% of total
queries. We note that queries for “in-addr.arpa” correspond to reverse DNS
lookups which are commonly used by email servers to check and see if the
message came from a valid server. Many email servers will reject messages from
any server that does not support reverse lookups since spammers typically use
invalid IP addresses.

Figure 3.1: Number of queries per unique primary domain, over a week (Rsch:
397K, Univ: 1.1M).

In terms of queries “reputation”, we used Majestic dataset [108] which is free
and updates on a daily basis – Majestic is a reverse search engine that
computes the number and strength of links to a domain (it is a measure of
trust instead of traffic estimates)[109, 110]. To get a sense of reputation
and probability of typical ranks, we show in Fig. 3.2 the complementary
cumulative distribution function (CCDF) of the reputation rank for primary
domains queried in both organizations. We can see that 44% of total queries,
in both organizations, are not listed in the top 1M domains of Majestic
domains ranking (i.e., CSV dataset released on 7-Aug-2018). Also, only 32% and
34% of queries in each network are among the top 10K most popular domains. In
our Majestic dataset, “google.com", “facebook.com, and “youtube.com" are top
three ranked domains respectively.

(a) Research institute. (b) University campus.

Figure 3.2: CCDF of reputation rank: (a) Research institute, and (b)
University campus. Figure 3.3: Real-time number of queries.

Considering the load of DNS queries generated by enterprise hosts, shown in
Fig. 3.3, we see that the number of packet-per-sec in the research network
varies between 50 to 400 depending on the day of week and peak/off-peak hours.
For the University network, on the other hand, a larger variation is observed
– i.e., 150 to more than 800 pps.

Table 3.2: A sample list of malicious and normal DNS queries with unusual
length.

Query name (FQDN) Security
6e517f3.grp10.ping.adm.cdd2e9cde9fee9cdc8.cdd0e8e9c8fce9d2e9fecdc4.c597f097ce87c5d3.ns.a23-33-37-54-deploy-
akamaitechnologies.com Malicious
708001701462b7fae70d0a28432920436f70797269676874.20313938352d32303031204d696372.6f736f667420436f72702e0d0a0d0a0.433a5c54454d503e.cspg.pw
Malicious PzMnPiosOD4nOCwuOzomPS4nNjovPS8uOzsnNCstODkjOCwoMwAA.29a.de
Malicious
bwzm133h9gb3pp9s6l3mu7r73sh.arm2513pu79r9.1z19e1bgm1hwu8z6u2.9rzlkhbvi45gaag52t3rqtqd2t.p2gliv6gklwzvvlt2jp1z6li7v.avqs.mcafee.com
Normal
0.19.6ce.71c.444.25.41.0.0.0.4.27.0.0.0.0.0.0.0.0.0.9efc95e03d7f3a4ae446ecd0d049e5ae9e016ee33703c9cb3506cad4bbd98bc.b.f.00.s.sophosxl.net
Normal p4-ces3lawazdkbw-
qlrq5qalxdt7tycq-385202-i1-v6exp3.ds.metric.gstatic.com Normal
_ldap._tcp.AWS._sites.dc._msdcs.AD.us-east-1.ec2-utilities.amazonaws.com
Normal

####  3.2.2 Query Name Attributes Engineering

We now look at the attributes of the query name (FQDN) in each DNS query
generated by enterprise hosts that are relevant to differentiating benign and
malicious DNS queries traffic. Our aim is to use only “Stateless” attributes
which can be derived from individual DNS query packets, independent of time-
series characteristics of queried domains or hosts DNS activity – there is no
overhead in computing these attributes in real-time. Our attributes are
inspired by various prior works (referred against each attribute).

According to RFC 1035 [111], the total length of a domain name (dots included)
is restricted to 255 characters, and domain names are represented as a
sequence of “labels” separated by dots. The maximum length of a label is 63
characters. It has been shown that DNS can be used for malicious purposes in
the form of DNS tunneling or exfiltration in which valuable information (e.g.,
credentials, credit card, or control messages) is embedded in the sub-domain
portion of a query name. Malware applications typically embed stolen data
[112] into the subdomain part of a DNS query for a domain where the name
server is under control of an attacker. A DNS query for “exfiltrated-
data.example.com” would be forwarded to the name server of “example.com”,
which would record “exfiltrated-data” and decode and decrypt the sensitive
information from that subdomain field.

Table 3.2 lists samples of malicious [105, 113, 2] and benign query names with
“unusual” length and string pattern. For example, the top two malicious query
names in this list respectively contain 129 and 136 characters. We note that
the sub-domain portion of these query names comprises random-looking strings
with a significant number of upper-case and numerical characters, and is
fairly long. For example, the second malicious query name from the top (i.e.,
for “cspg.pw") contains 38 numeric characters (i.e., 28%), and the third
malicious query name (i.e., for “29a.de") contains 38 numeric characters
(i.e., 28%) contains 23 uppercase letters (i.e., 39%). Given these
observations, we define our attributes by three main categories namely
characters count, entropy (an indication of randomness) of string, and length
of discrete labels in the query name.

(a) Research institute.

(b) University campus.

Figure 3.4: CCDF of number of characters in query name for: (a) Research
institute, and (b) University campus.

(a) Research institute.

(b) University campus.

Figure 3.5: Scatter density map of numerical fraction of characters vs. total
length of query name for: (a) Research institute, and (b) University campus.

#####  3.2.2.1 Count of Characters

The total number of characters is an important attribute since more characters
imply that the query name probably carries embedded information for an outside
host. In Fig. 3.4, we plot the distribution of character count for query names
in our dataset to understand the typical value of these attributes.

Total count of characters in FQDN:[102] We can see that more than 99% of host
queries in both organizations contain less than 80 characters, as shown by
black cross markers in Fig. 3.4. Only a very small fraction of query names
(i.e., about 0.3%) are really long, each with more than 100 characters. It is
important to note that antivirus tools tend to exchange legitimate data (i.e.,
for signature lookup) over DNS [66]. For example, in Table 3.2 the first two
“normal” query names correspond to “McAfee” and “Sophos” antivirus.
Interestingly, primary domains “mcafee.com" with 1.9M queries (average query
length of 84 characters), and “sophosxl.net" with 145K queries (average query
length of 106 characters) are among top ten frequent domains seen in our
dataset from the Research institute and the University network respectively as
shown in Table 3.3. Since the exfiltrated (or Command & Control) message is
carried by the sub-domain portion of an FQDN, we use the count of characters
in sub-domain [61] as our second attribute.

(a) Research institute.

(b) University campus.

Figure 3.6: Scatter density map of upper-case fraction of characters vs. total
length of query name for: (a) Research institute, and (b) University campus.
Table 3.3: Number of characters in FQDN for selected domains in our dataset.

primary domain # FQDN # unique FQDN frac. Numerical (%) frac. Uppercase (%)
avg. Length mcafee.com 1.9M 571K 39.4 0.31e-3 84.01 sophosxl.net 145K 41K 47.5
0.11 106.7 spotify.com 84K 819 7 0 41.7 cnr.io 121K 113K 19.97 70.08 209.8
e5.sk 66K 131 13.8 0.15 129.06

Additionally, we use the count of uppercase characters[62] and count of
numerical characters[62] in a query name to determine if it is benign or
malicious. This is because the fraction of uppercase and numerical characters
becomes high in encrypted/ encoded data [62] – however, not all encrypted data
is malicious. In Fig. 3.4, it is seen that only about 1% of all queries in
each organization contain more than 30 numerical characters. Unsurprisingly,
the upper-case character is very rare in domain names generated by hosts in
both enterprise networks – at least 98% of queries contain no upper-case
character, and less than 0.2% of queries contain more than 10 capitals.

To better understand the distribution of various characters in query names, we
plot the scatter density maps of total characters count versus numerical
fraction in Fig. 3.5, and total characters count versus uppercase fraction in
Fig. 3.6 – dark red areas depict higher density of points and dark blue areas
highlight the lower density of points. In Fig. 3.5, it can be seen that the
numerical fraction of a query name typically stays below 20% (mostly less than
10%) when the query name has less than 60 letters (i.e., dark red area on the
left bottom of plots). Interestingly, for the Research Institute shown in Fig.
3.5a, we observe a crowded region around 40% of numerical letters when the
FQDN length is between 60 to 80 characters. In Fig. 3.6, we see that the
fraction of uppercase letters is below 10% for short query names (i.e., less
than 80 characters for the Research Institute and less than 50 characters for
the University), and it tends to zero when query names get longer.

#####  3.2.2.2 Entropy

Random (“not-readable”) sub-domains are common in DNS exfiltration/tunneling
queries due to use of encryption and/or encoding [66]. Entropy [102] is a
measure to determine the degree of non-readability (or strength of encryption)
and uncertainty in a string. We use Shannon entropy [114] which takes a
discrete random variable $X$ as input (i.e., DNS query name in our case), and
mathematically is given by:

${H(X)=-\sum_{k=1}^{N}P(x_{k})\log_{2}P(x_{k})}$ (3.1)

where $P(x_{k})$ is the probability of the k-th symbol (i.e., lower-
case/upper-case letter, numerical, dot, or hyphen) in the input string $X$
containing various characters where $N$ is the total number of unique
characters. We note that only specific letters can be used in a valid DNS
query name [111] (i.e., 52 alphabetic and ten numeric characters, a hyphen,
and dot, thus $N=$ 64). This means that the entropy value of a query name will
take a value between 0 and $\log_{2}(64)=6$ [115]. Table 3.4 shows the entropy
value for a sample list of query names, both benign and malicious. For
example, the entropy of a simple query name such as “www.google.com" equals to
$2.84$, and it gets a higher value for a more random string such as the last
entry in Table 3.4, a query for “googleapis.com" whose entropy value is
$5.27$. We also observe that the entropy value of malicious queries
(highlighted in bold text) varies and is not necessarily higher than that of
benign queries. In Fig. 3.7, we plot the CCDF of entropy for all FQDNs queried
by hosts of the two organizations during a week. It can be seen that the
entropy value for more than 90% of query names is less than $4$ in both
networks, and having an entropy greater than $5$ is less likely (i.e., lower
than $0.1$%).

Table 3.4: Entropy value for a sample list of query names.

Query name (FQDN) Entropy www.google.com 2.84
202.135.201.205.23000000000012.sb-adfe2ko9.senderbase.org 3.75
708001701462b7fae70d0a28432920436f70797269676874.20313938352d32303031204d696372.6f736f667420436f72702e0d0a0d0a0.433a5c54454d503e.cspg.pw
3.92
0.19.6ce.71c.444.25.41.0.0.0.4.27.0.0.0.0.0.0.0.0.0.9efc95e03d7f3a4ae446ecd0d049e5ae9e016ee33703c9cb3506cad4bbd98bc.b.f.00.s.sophosxl.net
3.98
6e517f3.grp10.ping.adm.cdd2e9cde9fee9cdc8.cdd0e8e9c8fce9d2e9fecdc4.c597f097ce87c5d3.ns.a23-33-37-54-deploy-
akamaitechnologies.com 4.50
PzMnPiosOD4nOCwuOzomPS4nNjovPS8uOzsnNCstODkjOCwoMwAA.29a.de 4.59
f4a55fc3f30keaayaayqivpqaggkbqggudp6hm-yacnusej1525121392-sonar.xy.fbcdn.net
4.78 DIYNBPRYA0K5CVUWA.ns1.logitech-usa.com 4.86
0ca7d.1.288.WYB52Q2ZPIU2SEUTDDDGEJDQFAO6F2C53AVC6IVAZZLR2PJHEWQWRFG6Z2NPQ3J.CQ4888.1d19d9c4.cnr.io
5.10
X2AR6GEQVHCSMXKFUNVIZU67PVMD5EF3N74E4TLOEOYK47WEXKMQ.hash.rocketeer.ct.googleapis.com
5.27

(a) Research institute.

(b) University campus.

Figure 3.7: CCDF of entropy of query name for: (a) Research institute, and (b)
University campus.

#####  3.2.2.3 Labels

This category comprises two attributes of labels inside a FQDN. For example,
in the query name “www.scholar.google.com", there are four labels separated by
dots. We use the number of labels [63] as our sixth attribute. This is because
DNS exfiltration/tunneling traffic tends to use certain patterns of labels in
their query names. Table 3.5 shows the label patterns for five selected
domains in our dataset from the Research institute network. We abstract a
label pattern by an array (samples are shown in the second column) whose
elements indicate the length (i.e., character count) of the corresponding
label in the query name – e.g., the pattern for “www.scholar.google.com" is
represented by (3,7,6,3). We see that queries for each of the primary domains,
listed in Table 3.5, appear in various number of patterns – the primary domain
is obtained by combining the top level domain (TLD) and the second level
domain (2LD) (e.g., in “www.scholar.google.com", the primary domain is
“google.com"). For example, the domain “sophosxl.net" is queried by 2208
distinct label patterns during one-week period of our dataset, and each
pattern is seen in 66 queries on average. For “e5.sk" domain, on the other
hand, we observe only 10 unique patterns, each repeats more than 600 times.

Table 3.5: Labels pattern in query names for selected domains.

primary domain sample patterns # unique patterns avg # queries / pattern
sophosxl.net (1, 63, 63, 36, 16, 1, 2, 1, 8, 3) 2208 66 (1, 63, 63, 18, 40, 1,
2, 1, 8, 3) mcafee.com (3, 11, 7, 4, 4, 4, 3, 1, 26, 4, 6, 3) 316 6K (3, 11,
1, 4, 4, 4, 3, 1, 26, 4, 6, 3) spotify.com (48, 48, 48, 48, 16, 2, 7, 3) 43
1.9K (23, 2, 7, 3) cnr.io (5, 1, 3, 63, 63, 63, 30, 8, 3, 2) 46 2.6K (5, 1, 3,
63, 63, 63, 8, 8, 3, 2) e5.sk (63, 63, 63, 24, 1, 1, 2, 2) 10 660 (63, 63, 18,
1, 1, 2, 2)

(a) Research institute.

(b) University campus.

Figure 3.8: CCDF of length of labels in query name for: (a) Research
institute, and (b) University campus.

Another interesting observation is that queries for three domains namely
“sophosxl.net", “cnr.io", and “e5.sk" have several labels with 63 characters
(i.e., the max limit according to RFC), whereas queries for “spotify.com" and
“mcafee.com" do not use label length greater than 48 and 26 characters
respectively. For our last two attributes, we use maximum label length [63]
and average label length [63] in a query name. Fig. 3.8 depicts the CCDF of
the longest, and the average label length for FQDNs observed in the two
organizations. It is seen that for 90% of queries, their longest label does
not exceed 20 characters, and their average label length is ten characters (or
less) in both networks. On the other hand, only about 1% of queries have the
longest label of more than 40 characters in the Research and the University
networks, respectively.

Summary: Our main achievement in this section is to identify and capture eight
attributes (from the query name section of each outgoing DNS request packet)
that collectively have strong predictive power in determining whether the
query name is normal or malicious. The attributes include: (1) Total count of
characters in FQDN, (2) count of characters in sub-domain, (3) count of
uppercase characters, (4) count of numerical characters, (5) entropy, (6)
number of labels, (7) maximum label length, and (8) average label length.

Table 3.6: Summary of additional dataset (days 8-14) used for evaluation.

Research University Total Outgoing DNS queries 79.6M 228M Outgoing DNS queries
(IPv4) 62.6M 182M Outgoing DNS queries (IPv6) 17.0M 46M Outgoing DNS queries
(only qualified) 78.8M 217M Unique query names (FQDN) 2.1M 6.1M Unique primary
domains 382K 1.15M

###  3.3 Detection of Anomalous Queries

We now develop a machine learning technique to determine if a DNS query of an
enterprise host is normal or not (i.e., “anomaly detection”). By training a
model with only normal query names we aim to detect new/unknown malicious
attacks (i.e., anomalous queries) which can be missed by the two-class
classifier. The machine is invoked with the eight attributes of each DNS query
explained in the previous section. To validate the efficacy of our models, in
this section, we extend our dataset by including additional records (78.8 M
and 217 M qualified DNS queries from the Research Institute and the University
campus networks, respectively) collected over the one week of 6-Aug-2018 to
12-Aug-2018 - a summary of this additional dataset (i.e., days 8-14) is shown
in Table 3.6. In total, we analyze 14 days worth of DNS queries from the two
enterprises.

####  3.3.1 Machine Training

We train our anomaly detection machine with benign data from four days of our
dataset – we keep the remaining ten days worth of data for testing. Ground
truth of benign domains in the literature is primarily drawn from highly
ranked popular domains [47]. For example, Alexa top-ranked domains are
commonly used – Alexa no longer publishes free top one million sites. However,
we note that Alexa ranking is based on the browsing behavior of Internet users
(i.e., estimate of global traffic to a domain). As a result, some malicious
domains may appear among top K Alexa domains due to a burst of requests from a
high number of infected clients querying them [116]. We, therefore, use an
alternative, Majestic Million [108] that releases a free dataset of top 1M
domains and updates it daily. Majestic ranks sites by the number of subnets
linking to that site – it is a measure of trust instead of traffic estimates
[109, 110]. For the benign training instances, we only use the top 10,000
primary domains in the Majestic list. We also include FQDNs for “sophosxl.net”
domain which is not among the top 10K Majestic dataset – the Majestic dataset
is used as a reference of domain reputation to determine whether a queried
domain is benign or not.

####  3.3.2 Algorithms and Tuning Parameters

The objective is to maximize the detection of anomalous queries while reducing
the rate of false alarms (i.e., incorrectly detecting a normal query as
anomalous or vice versa). Many supervised machine-learning algorithms for
detecting anomalies such as one-class SVM and Replicator Neural Network suffer
from high false alarms. They are optimized for profiling the inlier behavior
rather than detecting anomalies. We employ “Isolation Forest (iForest)” [117]
which is an effective algorithm in detecting anomalous instances in high-
dimensional datasets with minimal memory and time complexities.

The iForest algorithm [117] works based on the concept of isolation without
employing any distance or density measure. This algorithm aims to isolate test
instances by randomly selecting a feature and then randomly selecting a split
value from a range (within min and max obtained from training) values of the
selected feature. Then, the score is calculated as the number of conditions
(path length) to check for isolating a test instance. Note that isolating
normal instances require more conditions. The process is repeated several
times to avoid issues due to randomness, and the average path length is
calculated and normalized.

Algorithm Tuning: We used scikit-learn and its APIs, an open-source machine-
learning package written in Python, to train and test our machine. We have
used three tuning parameters for iForest during the training phase, namely the
number of trees (n_estimators), height limit of trees (max_samples), and
contamination rate. We tune the value of each parameter while fixing the other
two parameters and validate the accuracy of our machine for both benign and
malicious instances (that we have the ground truth)in both organizations. The
default value for the number of trees is 100, the height limit of trees is set
to “auto” (implying 8 given the size of our dataset), and the contamination
rate is 10%.

To tune the algorithm, we require ground truth for both benign and malicious
instances. Our ground-truth for benign instances are chosen based on the top
10K domains of the Majestic list (§3.2.1)) – we have 1.7 M instances for the
research organization and 4.8 M for the university campus network). For the
ground-truth of malicious instances, we generated DNS exfiltration queries
with our open-source tool, forked from an open-source project called “DNS
Exfiltration Toolkit” (DET) [118]. We ran our tool on a machine inside the
University network that exfiltrates the content of a CSV file containing 1000
samples of random credit card details (obtained from [119]) to an
authoritative name server under our control located in the Research network.
DET employs AES-256 encryption and uses two tuning parameters: the max length
of the query name (i.e., 30 to 218 characters) and the max length of labels
(i.e., 30 to 63 characters) to diversify our synthetic malicious queries. We
generated a total of 1.4M exfiltration queries that are publicly available at
[120] in the form of a CSV file.

Table 3.7: Detection accuracy of ground-truth instances after tuning.

Benign Malicious Research Institute $98.44$% $95.07$% University Campus
$97.99$% $98.49$%

We found that setting the number of trees equal to 2 results in high accuracy
of more than $91$% for benign and $63$% for malicious instances – increasing
this parameter does not enhance the accuracy but increases the model size and
prediction time. Having fixed the number of trees to 2 and the contamination
rate to 10%, we varied the height of trees from 1 to 20. The detection
performance rises by increasing the height limit of trees and gets stabilized
at the value of 18 with the best accuracy of more than $90$% and $98$% for
ground-truth benign and malicious instances, respectively. We then fixed the
number of trees to 2 and the height limit of isolation trees to 18 to quantify
the impact of contamination rate. Decreasing the contamination rate from 10%
to 2% improved the performance of our model for both organizations as shown in
Table 3.7, with the accuracy of more than $97$% for benign instances and more
than $95$% for malicious instances.

To summarize, we found the optimal value of tuning parameters equal to 2, 18,
and 2% respectively for the number of trees, the height limit of trees, and
the contamination rate. Furthermore, for optimal tuning parameters, the
iForest algorithm sets the threshold value of anomaly score to $0.54$,
distinguishing normal and anomalous instances.

Table 3.10 shows the performance of our machine (after tunning) for selected
benign instances – for cross-validation. It can be seen that the rate of false
alarms is mostly less than $5$% in both organizations, though we see a higher
false rate (i.e., more than $10$%) for “in-addr.arpa" and “sophosxl.net"
domains in the University network. We found out that the attributes of some of
FQDNs of primary domains “in-addr.arpa" and “sophosxl.net" are similar to
those of exfiltration FQDNs (possibly benign DNS exfiltration). In the next
section, we will pre-filter instances for these domains that are highly
trusted (i.e., certainly benign) without passing them to the anomaly detection
machine.

Table 3.8: Anomaly detection for Research institute.

Input Output Days 1-4 Days 5-14 Benign domains (top 10K) normal $98.44$%
$98.35$% anomalous 1.56% 1.65% Others (beyond top 10K) normal $78.43$%
$77.35$% anomalous $21.57$% $22.65$%

Table 3.9: Anomaly detection for University campus.

Input Output Days 1-4 Days 5-14 Benign domains (top 10K) normal $97.99$%
$97.83$% anomalous 2.01% 2.17% Others (beyond top 10K) normal $70.57$%
$63.38$% anomalous $29.43$% $36.62$%

Table 3.10: Performance of our machine for trusted domains.

Research institute University campus primary domain normal anomalous Avg.
query length false-rate (%) normal anomalous Avg. query length false-rate (%)
akadns.net 2.6M 24K 38 0.91 7.6M 191K 38 2.4 googleapis.com 165K 1.6K 76 0.96
526K 15K 76 2.7 gstatic.com 207K 362 69 0.17 835K 986 76 0.11 in-addr.arpa
3.7M 49K 26 1.32 9.2M 1.1M 26 10.7 mcafee.com 1.9M 735 84 0.03 635K 13K 88
2.01 onmicrosoft.com 22K 1.6K 51 6.55 201K 1537 53 0.75 senderbase.org 1.1M
14K 66 1.32 2.2M 2816 66 0.12 sophosxl.net 138K 6.5K 103 4.44 2.5M 394K 119
13.7 spamhaus.org 12K 597 31 4.7 947K 7.7K 32 0.81 spotify.com 579 31 45 5.08
468K 1.2K 168 0.25 Top 100 domains 7.9M 135K 20 1.68 24M 351K 20 1.41 (e.g.,
google, apple)

###  3.4 Performance Evaluation

In this section, we evaluate the efficacy of our scheme by: (a) cross-
validating and testing the accuracy of the trained model for benign instances
and quantifying the performance in real-time on live 10 Gbps traffic streams
from the two organizations, (b) testing the detection rate for malicious DNS
queries that we generate using our customized tool (i.e., DET [118]) and an
open-source tool (i.e., Iodine [121]), (c) comparing our one-class classifier
with a two-class classifier, and (d) drawing insights into the top three
anomalous domains for which malicious DNS queries are made in the Research and
University networks. Note that our proposed approach is generic and hence can
be readily used in different organizations, but the model needs to be trained
by the specific data of each organization.

####  3.4.1 Performance Metrics

We begin with three performance metrics, namely accuracy, anomaly score, and
responsiveness of our models.

Accuracy: As mentioned in the previous section, we trained our model with
benign instances from 4 days’ worth of our data (i.e., Days 1-4), and tested
with all instances from Days 5-14 in addition to remaining instances from Days
1-4 that were not used for training (i.e., “Others”). Tables 3.8 and 3.9 show
the rate of detection (i.e., normal versus anomalous) for the benign and
Others instances in the two networks – instances in the Benign category are
among the top 10K of the Majestic ranking list, and instances in the Others
category are beyond 10K. It can be seen that $98$% of benign instances are
correctly detected as normal during both cross-validation (i.e., Days 1-4) and
testing (i.e., Days 5-14) phases. We note that our machine raises a false
alarm for about $2$% of benign domains, as highlighted in bold text.

To address this, we populate a whitelist of domains that are highly trusted.
Our whitelist comprises only the top 100 domains from the Majestic ranking
dataset (e.g., “google.com”, “bbc.com”, “amazonaws.com”) as well as popular
legitimate (e.g., “akadns.net”, “in-addr.arpa”, “spotify.com”) and security
services (e.g., “spamhaus.org”, “senderbase.org”). Note that these security
services are using disposable domains (i.e., “single-time use”) for the
purpose of signaling over DNS queries (e.g.,
“0.0.0.0.1.0.0.4e.135jg5e1pd7s4735ftrqweufm5.avqs.mcafee.com” [122]).

Table 3.11: Anomaly detection combined with whitelisting for Research
institute.

Input Output Days 1-4 Days 5-14 Benign domains (top 10K) normal $98.92$%
$99.20$% anomalous 1.08% 0.80% Others (beyond top 10K) normal $83.50$%
$87.48$% anomalous $16.50$% $12.52$%

Table 3.12: Anomaly detection combined with whitelisting for University
campus.

Input Output Days 1-4 Days 5-14 Benign domains (top 10K) normal $98.92$%
$98.90$% anomalous 1.08% 1.10% Others (beyond top 10K) normal $90.98$%
$81.74$% anomalous $9.02$% $18.26$%

Employing whitelisted domains would slightly enhance detection. Our refined
results are shown in Tables 3.11 and 3.12. We can see a slight reduction in
the rate of false alarms for benign domains – it is now capped at $1.20$% for
both networks, as highlighted in bold text. We note there are a total of 10K
(out of 923K) and 15K (out of 1.4M) false alarms for benign instances in the
Research and University network, respectively. Although the rate of false
positives is about 1%, network operators can further reduce this by employing
off-the-shelf intrusion detection systems (IDSs) such as Zeek on flagged
instances. With their signatures of known malicious traffic, Tools like Zeek
can filter out those flagged instances that are indeed benign. Note that
quantifying the rate of false alarms for detected anomalies under “Others” is
non-trivial due to the lack of ground-truth labels.

Table 3.13: Avg. anomaly score for research institute.

Input Output Days 1-4 Days 5-14 Benign domains normal $0.36$ $0.36$ anomalous
$0.59$ $0.61$ Others normal $0.44$ $0.43$ anomalous $0.64$ $0.65$

Table 3.14: Avg. anomaly score for University campus.

Input Output Days 1-4 Days 5-14 Benign domains normal $0.39$ $0.39$ anomalous
$0.57$ $0.58$ Others normal $0.43$ $0.43$ anomalous $0.63$ $0.62$

Anomaly Score: Anomaly detection algorithms use this score to determine if an
instance is classified as normal or abnormal. For the iForest algorithm, the
anomaly score varies from 0 to 1, where 0 means purely normal and 1 indicates
a definite anomaly. A value of an anomaly score of less than 0.5 is reasonable
enough to be interpreted as normal [117].

Tables 3.13 and 3.14 show the average anomaly score (i.e., normal versus
anomalous) for the benign and Others instances in the two networks. It can be
seen that the average anomaly score of benign instances during the cross-
validation phase (i.e., Days 1-4) is $0.36$ and $0.39$ which is well below the
threshold value of $0.54$ (obtained during model tuning in §3.3.2) for the
Research Institute and University networks respectively. Similarly, the
average anomaly score of benign instances during the testing phase (i.e., Days
5-14) is $0.36$ and $0.40$ for the Research Institute and University networks
respectively.

Table 3.15: Avg. time complexity of our scheme.

extracting attributes 54 $\mu$sec detecting anomalies 746 $\mu$sec Total time
per each query name 800 $\mu$sec

Responsiveness: In terms of responsiveness, we have quantified the average
time for extracting eight attributes and anomaly detection (via running
prediction against the trained model) by testing more than 300 million DNS
queries in our dataset from the two enterprise networks. Our attributes
extraction and anomaly detection engines run on a virtual machine using 4 CPU
cores, 6GB of memory, and storage of 50GB. As shown in Table 3.15, on average
it takes $800$ $\mu$sec to determine if a DNS query is normal or not. This
indicates that our scheme can process approximately 1250 DNS queries per
second, well above the actual rate of DNS queries in both organizations where
the peak value is 800 DNS queries per second, as shown in Fig. 3.3.

(a) DET tool. Figure 3.9: Attributes of DNS exfiltration query names of: (a)
DET tool, and (b) Iodine tool, detected vs. undetected by the University
model.

(b) Iodine tool.

####  3.4.2 Evaluating Models using Known DNS Exfiltration Data

In this subsection, we evaluate the efficacy of our detection scheme using DNS
exfiltration data (i.e., ground-truth) including two large sets generated by
our customized DET tool and the open-source Iodine tool, and a small set
collected from publicly reported real malicious DNS queries.

Our DET Tool: We showed previously in Table 3.7 that our models for the
Research Institute and the University campus respectively were able to
correctly detect $95.07$% and $98.49$% of exfiltration queries (generated by
our DET tool) as anomalous instances.

In Fig. 3.9a, we show the value of attributes for detected instances (blue
octagon on the left) versus undetected instances (red octagon on the right)
using the model generated from data of the university campus. Even though
undetected instances were shorter both in total length and average label
length, it is important to note that there is a fair overlap of value range
comparing detected (i.e., classified as anomalous) with undetected instances
(i.e., classified as normal) across all attributes, suggesting that attributes
collectively would determine a fairly accurate output of our model. To explain
it further, we look at the attributes of two pairs of FQDNs generated from our
DET tool (one classified as normal and one classified as anomalous by the
model of the research institute), as shown in Table 3.16 – normal and
anomalous classified FQDNs are shown in bold and italic fonts respectively.
This is because we have obfuscated the actual primary domain used in the DET
tool for privacy reasons. For each pair, we investigate distinguishing factors
given some identical (or close) attributes, highlighted in bold in Table 3.16.
We see six common attributes (character count, numerical character count,
number of dots, maximum label length, average length of labels, and sub-domain
character count). However, entropy and upper-case attributes have relatively
larger values in the detected instance (i.e., italic text). Moving to the
second example, where entropy, number of dots, and upper-case characters count
are very close in two instances, the query length becomes an essential factor
for the model detecting or missing a malicious instance.

Table 3.16: Samples of malicious queries (DET) along with their attributes
detected/undetected by the model of the research institute.

Sample FQDNs # chars entropy # numeric # dots # upp. max label. avg label #
chars subdom. s6wIrxk.363937356263363563663038333865.maliciousDomain.com 58
3.98 31 3 2 30 13 39
ClBQxLW.656661356534343938623539393265.maliciousDomain.com 58 4.24 30 3 6 30
13 39 J8tngo1.53061393230646235636634326137656436.maliciousDomain.com 63 4.15
37 3 2 35 15 44
dZrlKkg.645a726c4b6b677c217c337c217c3732613830656235373830.3333386331643631393837.maliciousDomain.com
101 4.15 64 4 3 50 19 82

Iodine Tool: To further evaluate the efficacy of our scheme, we used the
Iodine tool [121] to generate an additional dataset of malicious DNS queries.
First, we exfiltrated the same CSV file of 1000 samples of random credit card
details similar to our DET tool. It took approximately $8$ seconds to transfer
the entire CSV file. Next, we wrote a Python script to repeat the process with
a delay (between runs) uniformly distributed between 20 and 40 seconds. We ran
the script for three days. As a result, we captured more than 2.2 million
unique instances – our Iodine dataset is also made publicly available [120].
Note that, unlike our customized DET tool, we only used iodine with its
default settings (i.e., no variation of parameters in DNS queries). When this
dataset was tested with our iForest model of the University campus, except for
275 instances all others were correctly detected as anomalous. The average
anomaly score was $0.86$ and $0.49$ for correctly and incorrectly classified
instances, respectively. We also tested against the model of the Research
Institute. We found that a small number of malicious instances (1837 out of
2.2 million) were missed – the average anomaly score was $0.67$ and $0.45$
(lower scores than the university model) for correctly and incorrectly
classified instances, respectively.

In Fig. 3.9b, we show the value of attributes for Iodine instances (detected
versus undetected) when tested against the university in the same way as we
did in Fig. 3.9a for DET instances. We can see that undetected instances (red
octagon on the right) have fewer numerical characters in their query name – 2
to 10 numerical chars versus 21 to 242 in detected instances. Additionally, it
is observed that missed instances are relatively short (total chars count of
34-42), with a few uppercase chars (up to 8), and contain short labels
(average about 5). Note that the length of DNS queries generated by iodine is
typically longer (average of 207 chars). Still, we intentionally diversified
the query length (30 to 218 with an average of 64) in our custom DET tool,
resulting in a slightly higher percentage of missed instances.

Table 3.17: Anomaly score of queries publicly reported as DNS Exfiltration.

Known Malicious FQDN Anomaly Score
708001701462b7fae70d0a28432920436f70797269676874.20313938352d32303031204d696372.6f736f667420436f72702e0d0a0d0a0.433a5c54454d503e.cspg.pw
0.75
9ad9ca2.grp10.tt1.dcd2fed0d2fefecdc8d2c4c8c8fecdde.e3e29f9a9ff9cbc79fdae3fcc4
d2c8c4cdd0feded295e9e9e9e9e9e9feea.e9e9e9e9e9e9e9e9e9e9e9e9e9e
9e9e9e9e9e9e9e9e9e9e9e9e9e9e9e9e9.e9e9e9e9e9e9e9e9e9e9.ns.a23-33-37-54-deploy-
akamaitechnologies.com 0.70
PzMnPiosOD4nOCwuOzomPS4nNjovPS8uOzsnNCstODkjOCwoMwAA.29a.de 0.68
9ad9ca2.grp10.tt2.dcc8c8d0c8fccdd2fcd0dcdec8c8cdc8.e6dcc8c8d0c8fccdd2fcd0dcdec8c8cdc8e9dcdcdec8ded2feded0d2c8fc.ns.a23-33-37-54-deploy-
akamaitechnologies.com 0.67 ZTEZGKDFA0KNGUCQI.ns1.logitech-usa.com 0.65
WQPKBPRYA0IVDUQWI.ns1.logitech-usa.com 0.65 QRBJBPRYA0JBKUGVI.ns1.logitech-
usa.com 0.65 SXLXBPRYA0IVDUKTI.ns1.logitech-usa.com 0.65
SXLXBPRYA0IVDUKTI.ns1.logitech-usa.com 0.65
6e517f3.grp10.ping.adm.cdd2e9cde9fee9cdc8.cdd0e8e9c8fce9d2e9fecdc4.c597f097ce87c5d3.ns.a23-33-37-54-deploy-
akamaitechnologies.com 0.59 RoyNGBDVIAA0.0ffice36o.com 0.58
iucCGJDVIBDSNF3GK000.0ffice36o.com 0.58 viLxGJDVIBJAIMQGQ000.0ffice36o.com
0.58 gLtAGJDVIAJAKZXWY000.0ffice36o.com 0.58
TwGHGJDVIATVNVSSA000.0ffice36o.com 0.58 1QMUGJDVIA3JNYQGI000.0ffice36o.com
0.57 t0qIGBDVIAI0.0ffice36o.com 0.57

Real malicious DNS queries: Additionally, we tested $17$ samples of DNS
queries from known real malware reported on various forums [105, 113, 2, 123].
The top ten instances correspond to POS malware, and the bottom eight
instances were recently found as part of a new attack targeting networks of a
private airline company [123]. Our trained model was able to detect all of
them as abnormal instances. Table 3.17 gives the anomaly score of these known
malicious domains. It can be seen that values are well above the average
anomaly score of benign instances shown in Table 3.13 and 3.14.

####  3.4.3 Comparing Multi-Class Classifier with One-Class Classifier

Existing proposals have predominantly used stateful attributes (e.g.,
mean/variance of time-interval between a pair of DNS query/response, frequency
of A, AAAA, TXT types resource records, or time between two DNS responses from
a given domain) along with a combination of stateless attributes to detect
data theft over DNS protocol that is fundamentally different from our approach
(i.e., using stateless attributes only). Hence, we cannot compare the
performance of our model with prior research work. However, we compare the
efficacy of our proposed one-class classifier (i.e., iForest anomaly detector)
with a two-class classification model (i.e., Random Forest in [124, 63]) using
the stateless attributes considered in this chapter.

Table 3.18: Detecting wild malicious DNS queries from two enterprises.

Anomalous domains Daily avg. # queries Daily avg. # enterprise hosts Distinct
types Avg. TTL (sec) Research Institute imrworldwide.com 20.9K 12 A [64%],
AAAA [36%] 1250 cnr.io 17.4K 4 TXT [99.9%], AAAA [0.01%] 0 1rx.io 2.5 K 10
AAAA [58.9%], A [41.1%] 45 2o7.net 1.2K 8 A[53.4%], AAAA [46.6%] 144
University Campus imrworldwide.com 80.4K 203 A[98%], AAAA [1.8%], CNAME[0.02%]
1781 360.cn 69K 122 A[97.5%], AAAA [2.5%] 3139 adlooxtracking.com 3K 51 A
[88.1%], AAAA [11.9%] 903

We build a new dataset for two-class classification using 4-days worth of top
10K majestic million as benign (same as for the iForest model) and 1.4 million
records generated by our DET tool as malicious instances. We split this
dataset to 60% for training and 40% for testing. The accuracy of validation
(on training data) and testing is about 99% and 97% respectively, with the
default parameters of the Random Forest classifier. Also, we obtain the
confidence-level of the Random Forest model to assess its confidence in making
decision for test instances (benign or malicious). The model displays an
average confidence of 99.9% for correctly classified benign instances – this
measure is 76% for misclassified benign instances. Similarly, when we present
malicious queries from the DET tool, the average confidence is 99.9% and 70%
for correct and incorrect predictions, respectively.

To further evaluate the efficacy of Random Forest, we tested instances from
the Iodine dataset (not used in model training). The model correctly detected
only 0.001% of Iodine instances (2543 out of 2.2M) as malicious. Therefore, we
tuned three parameters of the model, namely the number of trees (1 to 200),
the number of selected attributes for each tree (1 to 8), and depth of the
tree (1 to 20). We used the same method described in IV and found the optimal
values equal to 8, 5, and 9 respectively for the number of trees, the number
of attributes, the depth. With these optimized parameters, the rate of
correctly detecting Iodine queries as malicious improved to 54.8%. However,
the average confidence of the tuned model for correctly and incorrectly
classified instances was 75% and 52% respectively – the model confidence
becomes lower than its performance for the trained DET instances.

Moreover, we presented the 17 samples of publicly reported malicious DNS
queries (listed in Table 3.17) to the tuned Random Forest model and found that
13 out of 17 instances were misclassified. This again proves that multi-class
classifiers display poor resilience to morphed attacks that deviate from known
attack signatures.

####  3.4.4 Malicious DNS Queries from Enterprise Networks

We now look at real DNS queries of the two enterprises that were detected as
anomalies in §3.4.1. Focusing on “Others" in Tables 3.11 and 3.12 (i.e.,
domains that are not among top 10K Majestic ranking list), we see about $12$%
and $18$% of instances in the Research and University networks respectively
that are flagged as anomalous DNS queries. We have further analyzed these
instances to gain insights into their primary domains. We found that only a
few primary domains contribute more than $80$% of anomalous instances in both
organizations. Table 3.18 lists these top domains. Interestingly, domain
“imrworldwide.com” is seen in both the Research and University networks – this
domain is known spyware that tracks web activity of victim hosts [125]. We
also see “adlooxtracking.com” the third top domain detected in the University
network which is a notorious domain that redirects web users to
phishing/unsafe webpages, resulting in freeware downloads [126]. In addition,
our model detects suspicious domains “2o7.net”, “cnr.io”, “1rx.io”, and
“360.cn” with a fairly significant number of queries. However, we cannot
verify that they are malware or spyware – this may need further investigations
into end-hosts that generate these queries.

Insights: To better understand these DNS queries detected as malicious, we
have further analyzed their corresponding DNS responses – note that DNS
responses are exclusively used in this section for drawing further insights
into anomalous queries. As mentioned above, Table 3.18 lists the top malicious
primary domains along with their statistics, including the daily average
number of DNS queries generated for each domain, the daily average number of
enterprise hosts querying for each domain, distinct DNS types with their
distribution, and the average of TTL values (specified in their corresponding
response). For the research institute, we can see that “imrworldwide.com” is
queried more than 20,000 times a day (on average) and only 12 unique hosts
(i.e., IP addresses) make these queries. Analyzing these IP addresses, we
found (by reverse lookup) that five of them are recursive resolvers of the
research institute – having recursive resolvers as querying hosts is also
observed for other anomalous domains. Focusing on seven hosts which are
regular clients, four were found actively making anomalous queries on four
days, while the other three hosts do not display malicious behavior over the
rest of the week of our analysis (though they are present on the network).
This observation suggests that those four regular hosts are possibly infected
by malware or spyware. Moreover, we found that of those four regular clients,
three generated queries to all top malicious domains, except “cnr.io” over the
entire week in the research network. Consistently generating anomalous queries
over the week is seen in two regular clients for “1rx.io”, and one regular
client for “2o7.net”.

Similarly, for the university campus, we see on average 80,000 daily queries
for “imrworldwide.com” from an average of 203 unique enterprise hosts. By
reverse lookup of host IP addresses, we found that seven of them are recursive
resolvers and the remaining 196 hosts are regular clients – 150 of these
clients consistently send queries for “imrworldwide.com” during the entire
week. Interestingly 130 of them fall under one subnet of size /24.
Furthermore, we found that a total of 290 university hosts generate queries
for at least one of the top three malicious domains (i.e., “imrworldwide.com”
or “adlooxtracking.com” or “360.cn”) – of these hosts, 35 make queries for all
of these top three malicious domains. By reverse lookup we discovered that 6
of them are recursive resolvers and the remaining 29 hosts are from the same
subnet of size /24, indicating that this particular subnet might be infected
by malware or spyware.

We further investigated the type field in DNS queries for these frequent
malicious domains. “A”-type and “AAAA”-type records map domains to IPv4 and
IPv6 addresses respectively. Our first observation by looking at distinct
types of anomalous queries in Table 3.18 is that there is a much greater
percentage use of IPv6 in the Research Institute than in the University
network. Secondly, we observe that “TXT” strongly dominates the type of DNS
queries for “cnr.io” in the Research Institute which clearly indicates a data
exfiltration/tunneling over DNS [127]. Note that sophisticated attackers tend
to use other types (i.e., “A”, “AAAA”, “CNAME”, “NS” and “MX” instead of
“TXT") to hide their malicious activities over DNS.

DNS responses contain a Time-To-Live (TTL) field in seconds, indicating the
duration of a DNS resource record to be cached on the host machine. According
to RFC 1033 [128], it is important to set an appropriate TTL value since very
low values result in overloading the DNS server, and very high values may
limit the flexibility of changing resource records in real-time. According to
RFC 1912 [129], it is recommended to set the TTL value between one to five
days. But, CDN (Content Distribution Network) services tend to use smaller
TTLs for fast reaction to dynamic resource changes. Unfortunately, malicious
entities also use small TTLs for minimizing their footprints and becoming more
resistant against DNS blacklisting [50]. In Table 3.18, we compute the average
TTL for each of the top malicious domains. We observe that malicious domains
use relatively smaller TTLs (i.e., less than an hour), for example it is set
to 0 in all of the DNS responses for “cnr.io”. Another example is “1rx.io” for
which the average TTL is 45 seconds.

(a) Research institute.

(b) University campus.

Figure 3.10: Number of DNS queries for top malicious domains over a day.

We plot in Fig. 3.10 the query count (computed every 10 minutes) of top
anomalous primary domains on day 6-Aug-2018, as an example – missing points in
this figure correspond to zero query count over those 10-min epochs. Note that
the mean and standard deviation of the anomaly score is shown next to each
domain name in the legend. Our first observation is that the query count for
all primary domains is higher during working hours (i.e., increasing an order
of magnitude at about 8 am, staying at a certain level, and falling back at
about 5 pm), though the primary domain “cnr.io” in Fig. 3.10a displays a
fairly consistent pattern of query count over a day (except one spike at
around 2 pm).

Finally, looking at the anomaly score of queries for these selected malicious
domains (as shown in the legend of Fig. 3.10), the primary domain cnr.io has
the largest mean value $0.62$ (the closer to $1$ means more anomalous). The
average score for other malicious domains in both networks varies between
$0.52$ to $0.56$ which is well above the average score for benign instances
(i.e., less than $0.40$) reported in Table 3.13 and 3.14. We acknowledge that
the anomaly score is just an indicator, that can be used to flag suspicious
domains. The network operators can then perform further investigation into
these identified domains.

###  3.5 Real-Time Deployment in Campus Network

This section presents our web-tool to access and visualize real-time detection
of malicious DNS queries for an enterprise network of a large university
campus in Sydney, Australia. We showcase how to visualize our real-time
learning-based detection engine operational on 10 Gbps traffic streams from
the network border of the university campus.

Figure 3.11: Web-UI of our real-time DNS exfiltration and tunneling detector.
Figure 3.12: Filtering queries for a specific primary domain name. Figure
3.13: Filtering queries with a minimum anomaly score.

####  3.5.1 Visualizing Detection of Malicious DNS Queries

We developed a tool to provide an intuitive user-interface for real-time
monitoring of outgoing DNS queries in enterprise networks using ReactJS. Our
web-tool is publicly available at [120].

This tool allows the user to visualize timestamped queries in real-time at
various time scales, including last 5 sec, 1 min, 10 min, 30 min, and 60 min,
as shown in Fig. 3.11. It also shows selected attributes for each query and
the output of our model. Records marked by a red cross correspond to DNS
queries detected as malicious by our machine learning model, while green ticks
mark benign queries – for privacy reasons, we do not show the FQDNs of live
traffic in our web-tool. The “’domain rank” shows the reputation of the
primary domain obtained from Majestic list [44] – if a primary domain is not
found in top million Majestic list then its rank is shown empty, for example
tribdss.com in Fig. 3.11.

The user can enter a specific primary domain name, filtering all records (in
real-time) associated with that primary domain. For example, in Fig. 3.12 the
user has selected google.com, the tool is showing the results specific to this
filtered domain. Furthermore, our tool is capable of filtering query records
based on anomaly score (computed by our model), as shown in Fig. 3.13 which
shows all queries with the anomaly score of 0.60 or more – anomaly score
varies from 0 to 1, where 0 is least anomalous and 1 is the most anomalous.
This value is calculated by the machine learning algorithm used while
classifying a query name. Additionally, our tool has a function to
pause/resume the stream. If the user notices a malicious domain in real-time,
then (s)he can pause the visualization stream to analyze the attributes of the
malicious domain.

###  3.6 Conclusion

Enterprise networks are potential targets of cyber-attackers for stealing
valuable and sensitive data over DNS channels. We have developed and validated
a mechanism for real-time detection of DNS exfiltration and tunneling from
enterprise networks. By analyzing DNS traffic from two organizations, we have
identified attributes of DNS query names that can be extracted efficiently in
real-time distinguishing legitimate from malicious queries. We then developed,
tuned, and trained a machine-learning algorithm to detect anomalies in DNS
queries using a known dataset of benign domains as ground truth. Lastly, we
evaluated the efficacy of our scheme on live 10 Gbps traffic streams from the
borders of two enterprise campus networks by injecting more than three million
malicious DNS queries via DET and Iodine tools – our tools and datasets are
publicly available. We showed that our solution outperforms the two-class
classifier in detecting new malicious DNS queries. We have drawn insights into
anomalous DNS queries by their anomaly scores, the trace of query count over
time, enterprise hosts querying them, and TTL and Type fields of their
corresponding responses. Lastly, we demonstrated our web tool for visualizing
our learning based on the detection of anomalous DNS queries (DNS exfiltration
and tunneling from enterprise networks) operational in our campus network.

## Chapter 4Automatic Detection of DGA-Enabled Malware Using SDN and Traffic
Behavioral Modeling

In the previous chapter, we analyzed outgoing DNS queries to tackle the data
exfiltration over DNS. This chapter combines Software Defined Networking (SDN)
and machine learning to develop an accurate, cost-effective, and scalable
system for detecting infected hosts communicating with external C&C servers,
subsequent to the resolution of DGA query names. Our solution dynamically
selects network flows for diagnosis by trained models in real-time and relies
more on the behavioral traffic profile than packet content. Parts of this
chapter are published in IEEE Transactions on Network Science and Engineering
(TNSE) [130].

Our contributions are threefold: (1) We analyze full DNS traffic collected
from the border of our university campus network (2.4B records for 75 days) to
highlight the prevalence and activity pattern of more than twenty families of
DGA-enabled malware across internal hosts. We draw insights into the