CADeSH: Collaborative Anomaly Detection
for Smart Homes

Yair Meidan,
Dan Avraham, Hanan Libhaber, and
Asaf Shabtai

Manuscript received April 1, 2022; revised July 6, 2022; accepted July 25, 2022.
This project received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 830927.
All authors are with the Department of Software and Information Systems Engineering, Ben-Gurion University of the Negev, Beer-Sheva, Israel. Email: {yairme, danavra<EMAIL_ADDRESS><EMAIL_ADDRESS>Copyright (c) 2022 IEEE. Personal use of this material is permitted. However, permission to use this material for any other purposes must be obtained from the IEEE by sending a request to<EMAIL_ADDRESS>
IEEE Internet of Things Journal, Vol. XX, No. XX, August 2022
Meidan et al.: CADeSH: Collaborative Anomaly Detection for Smart Homes

0000–0000/00$00.00  2021 IEEE

Although home IoT (Internet of Things) devices are typically plain and task oriented, the context of their daily use may affect their traffic patterns. That is, a given IoT device will probably not generate the exact same traffic data when operated by different people in different environments and when connected to different networks with different topologies and communication components. For this reason, anomaly-based intrusion detection systems tend to suffer from a high false positive rate (FPR). To overcome this, we propose a two-step collaborative anomaly detection method which first uses an autoencoder to differentiate frequent (`benign') and infrequent (possibly `malicious') traffic flows. Clustering is then used to analyze only the infrequent flows and classify them as either known ('rare yet benign') or unknown (`malicious'). Our method is collaborative, in that (1) normal behaviors are characterized more robustly, as they take into account a variety of user interactions and network topologies, and (2) several features are computed based on a pool of identical devices rather than just the inspected device.

We evaluated our method empirically, using 21 days of real-world traffic data that emanated from eight identical IoT devices deployed on various networks, one of which was located in our controlled lab where we implemented two popular IoT-related cyber-attacks. Our collaborative anomaly detection method achieved a macro-average area under the precision-recall curve of 0.841, an F1 score of 0.929, and an FPR of only 0.014. These promising results were obtained by using labeled traffic data from our lab as the test set, while training the models on the traffic of devices deployed outside the lab, and thus demonstrate a high level of generalizability. In addition to its high generalizability and promising performance, our proposed method also offers benefits such as privacy preservation, resource savings, and model poisoning mitigation. On top of that, as a contribution to the scientific community, our novel dataset is available online.

Internet of Things (IoT), Collaborative Anomaly Detection, IoT Attack Detection, Autoencoders, Clustering, Botnets, Distributed Denial-of-Service (DDoS), Cryptomining.

§ INTRODUCTION

In recent years, numerous cyber-attacks involving Internet of things (IoT) devices have taken place all over the world <cit.>. In the smart home sector <cit.>, these attacks include the recruitment of IoT devices to botnets <cit.> for the execution of distributed denial-of-service (DDoS) attacks <cit.> and the utilization of IoT devices for cryptocurrency mining <cit.>. Given the consistent increase in the number of IoT devices deployed <cit.>, the yearly increase in the rate of IoT malware development <cit.> (some of which is even open-sourced <cit.>), and the stealthiness of IoT-related cyber-attacks <cit.>, it is clear why new attack detection methods are needed for risk mitigation.

As a means of detecting IoT-related cyber-attacks, anomaly detection in network traffic data has been studied extensively <cit.>. However, the following shortcomings are typically associated with existing methods of this kind:

* Limited generalizability. In many cases, e.g., <cit.>, model training is limited, in that just one device from each IoT model is deployed in only one network. Hence, the transferability of the trained anomaly detector to other (identical) IoT devices cannot be guaranteed. This is especially true if the monitored IoT devices are deployed in other networks and other users interact with them.

* Indifference to the effect of user interaction. As noted in <cit.>, the traffic patterns of IoT devices can be heavily influenced by a device's interaction with the user(s), other devices, and the environment. Accordingly, an IoT device's  `normal' traffic behavior captured from an arbitrary location (where, e.g., a smart streaming device user relies on just a single streaming service source and always manually changes the channel) might not generalize well to other locations (where, e.g., users rely on numerous TV content sources for their streaming device and frequently use the voice assistant to change the channel).

* Excessive false alarms. Typically, anomaly detection methods only distinguish between normal and anomalous activities (i.e., a binary class label). However, when anomaly detection is used for attack detection, non-malicious anomalies could be mistakenly identified as cyber-attacks and thus lead to an excessive number of false alarms and unnecessary (automated) risk-mitigation responses. In other words, there are also legitimate reasons for traffic anomalies, such as infrequent user activities (booting a webcam or zooming in/out), or contextual changes (e.g., domestic smart motion detectors may send alerts more frequently during holidays or pandemic-related lockdowns, when families spend more time at home than usual).

To address these shortcomings, in this research we propose incorporating three enhancements for anomaly detection, especially when it is used for IoT-related attack detection in smart homes. The first enhancement is data sharing for collaborative model training. The idea is that instead of characterizing the normal traffic behavior using just a single device from a single environment, the traffic data of multiple identical IoT devices is shared so that the training data represents a larger variety of user interactions, network topologies, bandwidths, etc. With data sharing, network- and interaction-related traffic variations can be captured, contributing to the generalization of benign traffic behaviors. Once deployed, when inspecting the first local occurrence of a pattern which represents, e.g., a webcam zooming in, an anomaly detector  trained this way is less likely to raise a false alarm, because a similar pattern was probably already captured globally in at least one other environment. The second proposed enhancement is collaborative feature engineering, meaning that some features are calculated using a pool of identical devices rather than just the inspected device. For example, if an inspected device communicates with a destination IP address it never contacted before, the suspicion that such communication is malicious would decrease (leading to fewer false alarms) if other devices have also communicated with the same destination IP address recently. Of course, this kind of feature engineering also necessitates data sharing. The third enhancement we propose is fine-tuning and extending the set of possible class labels. Specifically, instead of dichotomously labeling each instance as either normal or abnormal (equivalent to benign or malicious, respectively, in attack detection), we consider three types of behavior and for each of them we define a corresponding attack detection-related class label, as follows:

* Frequent (`benign'). A limited <cit.> set of typical predefined operations which result in a limited number of normal traffic patterns. For instance, frequent IoT operations include sensing motion, detecting smoke, and sending video.

* Known (`rare yet benign'). A set of operations which occur much less frequently but are still considered normal, e.g., a webcam booting or zooming in/out. In the context of attack detection, a known event that is labeled as an anomaly is considered a false positive (FP). In our method, both frequent and known behaviors are eventually treated the same way, i.e, as an indication of a benign activity.

* Unknown (`malicious'). Extremely anomalous events which are highly indicative of the occurrence of malicious activity like a cyber-attack.

In this work, we propose CADeSH, a novel Collaborative Anomaly Detection method for IoT attack detection in Smart Homes. CADeSH is based on collaboration among IoT devices of the same model, and as such it offers several important benefits:

* Feasibility and privacy preservation. A key benefit in using CADeSH is that instead of relying on raw network traffic data, it makes use of metadata in IPFIX form (Internet Protocol Flow Information Export) <cit.> which (1) is already natively supported by most routers <cit.>, and (2) is considered more privacy preserving <cit.> and less resource consuming <cit.> than packet-level traffic analysis approaches, such as deep packet inspection (DPI).

* Efficiency. Multiple identical IoT devices are monitored simultaneously, so model retraining can be performed more frequently, based on shorter data collection periods.

* Quality assurance of the training data. CADeSH includes a collaborative mechanism in which not all of the monitored traffic is used for training, but rather only instances that are widely acknowledged (i.e., by multiple identical devices) as benign and representative are used. Consequently, the likelihood of model poisoning decreases.

* Robustness of the trained models. Although IoT devices are typically designed to execute a relatively small set of predefined operations <cit.>, the context of their use (in different weather conditions, by multiple people of different ages, etc.) might contribute to variability in a device's normal network traffic <cit.>. Collaborative anomaly detection allows such traffic variations to be captured and thus contributes to the model's generalization.

In addition to the benefits listed above, we make the following contributions:

* To improve the model's performance, we propose data enrichment using two additional (i.e., non-IPFIX) data sources: DNS requests and the Webroot API <cit.>.

* For evaluation, in addition to the benign and malicious data captured in our lab, we also relied on real-world data collected from five (real) home networks over a period of 21 days. This is the entire duration of network traffic data collection. The data was provided to us by a company which provides cyber-defense services for smart homes. To enable reproducible research and facilitate future advancements in research on collaborative IoT anomaly detection, we made our (anonymized) dataset publicly available. We are not aware of other similar publicly available online datasets.

Objectives of prior research on anomaly and attack detection, mainly in IoT settings

Ref.          t]@c@IoT-
related     1cObjective
<cit.>         Identify DDoS attacks as soon as they are launched from IoT devices that were recruited to a botnet
<cit.>                 Create an IoT anomaly detection service
<cit.>              Create an anomaly detection IoT system for smart cities
<cit.>            Review machine learning (ML) and deep learning (DL) based intrusion detection systems (IDSs) used in IoT settings
<cit.>               Overcome certain limitations of the Density-Based Spatial Clustering of Applications with Noise (DBSCAN) algorithm
<cit.>             Compare several ML algorithms for attack and anomaly detection in IoT systems
<cit.>         Create an IDS for low-powered and resource-constrained IoT devices, using unsupervised learning
<cit.>         Create a real-time, self-training, easily deployed, anomaly-based DDoS detection system for IoT networks
<cit.>              Detect compromised IoT devices without sharing data
<cit.>                       Improve the effectiveness of traditional ML IDSs on low-frequency attacks in high-dimensional networks
<cit.>         Propose a federated-based approach for the detection of botnet attacks, using on-device decentralized traffic data
<cit.>         Create an anomaly detector for IoT network communication
<cit.>           Create a DL-based botnet attack detector for compromised IoT devices
<cit.>                Create a software-defined network (SDN) based security mechanism to detect and mitigate DDoS attacks on IoT networks
<cit.>         Detect cyber-attacks based on user behavior
<cit.>                                              Create an anomaly-based DDoS attack detector, using autoencoders (AEs)
paper         Improve the performance of IoT-related cyber-attack detection,  using a two-step collaborative anomaly detection method

§ RESEARCH OBJECTIVE, SCOPE, AND ASSUMPTIONS

§.§ Objective and scope

The main objective of this research is to develop an efficient and generalizable collaborative method for the detection of various attacks involving IoT devices, while minimizing the false positive rate (FPR).

Given the large variety of (1) IoT application domains <cit.> and (2) viable IoT-related attacks <cit.>, we focus on the smart home domain and propose a means of detecting two common malicious activities, namely the recruitment of devices to IoT botnets (for later execution of DDoS attacks) and unauthorized cryptocurrency mining (see Section <ref>).

§.§ Assumptions

Considering the applicability of our proposed method, we note that collaborative attack detection (training and application) requires capturing and processing network traffic data from multiple IoT devices connected to multiple home networks. To enable this, we assume that the IoT devices are continuously monitored and their network traffic data is systematically collected into a central server. In real-world settings, this mechanism of data capturing, followed by model training and application, can be performed by Internet service providers (ISPs) as an additional countermeasure <cit.> against IoT malware or by third-party security providers. Another assumption we make in this research is that IoT devices of the same model produce network traffic data which is relatively similar to one another because of the common intended functionality, however these traffic patterns might vary due to either (a) differences in user interaction and network structure or (b) cyber-attacks.

§ BACKGROUND AND RELATED WORK

§.§ Attack detection and anomaly detection in the IoT

In recent years, traffic-based IoT attack detection methods have become more prevalent <cit.>. Many of the existing methods for IoT attack detection <cit.> rely on anomaly detection. In contrast to supervised methods <cit.>, which necessitate high-quality and sufficiently-large labeled datasets for training that cover both benign traffic and known attacks, unsupervised methods typically only require benign traffic (which is relatively easy to collect), and often, they can also detect zero-day attacks <cit.>. The basic assumption in relying on anomaly detection for attack detection is that the accumulation of traffic data from an uninfected IoT device during normal user interaction enables normal traffic patterns of the device to be captured. Then, after modeling the normal traffic patterns using various unsupervised machine learning algorithms, extreme deviations from the norm seen in the future could indicate malicious activity.

As can be seen in Table <ref>, numerous past studies utilized unsupervised machine learning algorithms for IoT attack detection. Among them, the authors of <cit.> proposed training an anomaly detector for each device separately. As part of their method, outbound packets trigger the extraction of behavioral snapshots of the network, to which an anomaly detection algorithm is applied to detect DDoS attacks when they are launched from IoT devices compromised by botnets (Mirai and BASHLITE). The study compared the anomaly detection performance when using the Isolation Forest (IF), Local Outlying Factor (LOF), One-Class Support Vector Machines (OCSVM) algorithms and autoencoders (AEs) and found that the latter obtained the best detection results. Similarly, the authors of <cit.> also proposed the use of AEs to differentiate benign and malicious traffic (generated by DDoS attacks), based on a flow of packets. Unlike those two studies, in this research (a) we train the anomaly detector using data from multiple networks (not just one), which increases generalizability, (b) in our method, two algorithms (not just one) are applied and three class labels are considered (rather than a binary class label), and (c) the FPR is reduced by adding the rare yet benign class label, which although considered abnormal, triggers no risk mitigating response. Unlike <cit.>, our data was captured from multiple real home networks and not from just one network in a controlled lab. Like <cit.>, we make our unique dataset publicly available. In the subsections that follow, we further compare our research to existing work, focusing on aspects such as the algorithmic complexity and flow, the breadth of data sources for model training, and the availability of the experimental data for reproducible research.

Algorithms in prior research on anomaly and attack detection, mainly in IoT settings

1c|Algorithm(s) used
<cit.>     AE
<cit.>             $k$-means, BIRCH (Balanced Iterative Reducing and Clustering using Hierarchies)
<cit.>          ExtraTrees, Random Forest                Collaborative     Data
t]@l@Boruta, Firefly, Davies-Bouldin index-based $k$-medoid, DBSCAN,
kernel-based locality sensitive hashing, $k$-distance graphs
<cit.>         Random forest
<cit.>     OPTICS (Ordering Points to Identify the Clustering Structure)                                  Collaborative     Data
<cit.>     t]@l@ARIMA (Autoregressive Integrated Moving Average), MLP (Multilayer Perceptron),
LSTM (Long Short-Term Memory)
Collaborative     Data
<cit.>          GRU (Gated Recurrent Unit)                                    Federated         Weights
<cit.>        $k$-means, mean shift, random forest
<cit.>     AE                                       Federated     Weights
<cit.>     AE                                       Collaborative     Data
<cit.>       DNN (Deep Neural Network)                                   Collaborative     Data
<cit.>            MCOD (Multi-Level Outlier Detection), MLP                                Collaborative     Data
<cit.>     Non-ML                                   Collaborative     Data
<cit.>            AE, Decision Tree, PCA (Principal Component Analysis), IF, OCSVM

<cit.>          Multinomial logistic regression                                      Federated         Weights
<cit.>            AE                                       Federated     Weights
This paper     AE, $k$-means     Collaborative     Data

§.§ Algorithmic complexity and flow

In multiple studies on IoT attack detection (e.g., <cit.>), only one algorithm was utilized. In contrast, we suggest a two-step method, where within each step a different algorithm is used (as further described and evaluated in Sections <ref> and <ref>):

* AE - to discern between frequent (`benign') and infrequent (possibly `malicious') traffic

* $k$-means - to further analyze only the infrequent traffic and discern between known (`rare yet benign') and unknown (`malicious') traffic to reduce the FPR

Considering the existing methods (see Table <ref>) that combine more than one algorithm, the research presented in <cit.> suggests feature extraction using clustering analysis, with a combination of $k$-means and mean shift, followed by a random forest classifier, to detect attacks. Another method <cit.> combines two clustering techniques: First, based on the packet interarrival time (IAT), clusters are built using $k$-means, and the abnormality value is calculated by BIRCH clustering. As opposed to our method, which was evaluated using flow-level data from multiple real IoT devices connected to real smart homes, this method was evaluated using emulated data from seven virtual state layer (VSL) services and three emulated IoT sites. Clustering was also employed in a more recent IoT-related study <cit.>, however in that research the aim was to perform fingerprint-based classification rather than anomaly detection for attack detection.

Datasets used in prior research on anomaly/attack detection, mainly in IoT settings

of data
of devices/models
of instances
of instances
of features
1c|Class labels
of time
9 IoT devices
t]@l@Benign/attacks related
to Mirai and BASHLITE
1 week

Virtual state layer
t]@l@7 VSL services,
3 emulated IoT sites
t]@l@Normal, DoS, data type
probing, malicious control,
malicious operation,
scan, spying, wrong setup

24 hours

1 hour

3* <cit.>     Generated
Audit logs (DARPA)

Flow     5,000,002

DoS, privilage esccalation, probing
7 weeks

University honeypots

Attack or not     2 months

Yahoo (S5)
Time series

Outliers and change-points

Virtual state layer
t]@l@7 VSL services,
3 emulated IoT sites
t]@l@Normal, DoS, data type
probing, malicious control,
malicious operation,
scan, spying, wrong setup
24 hours

2* <cit.>
30 IoT devices

9     Benign only

20 days

5 IoT simulated scenarios
5 scenarios

6* <cit.>
Smart home
9 IoT devices
Time window

Smart home
28 IoT devices

DDoS attack
6 months

Smart hospital
4 medical IoT devices

DDoS attack
5 minutes

Network telescope
1 network telescope

DDoS attack traces

7 booters

DDoS attack traces

DDoS attack traces

3* <cit.>
33 IoT devices
Anomalous Y/N
165 hours

Smart homes
14 IoT devices
Anomalous Y/N
2,352 hours

5 IoT devices
Anomalous Y/N
84 hours

Laboratory (ISCX 2012)
Devices in a testbed     Flow

7 days

Smart home
4 IoT devices
TCP flow
1 day

2* <cit.>
Smart home
9 IoT devices
Time window

5 IoT simulated scenarios
5 scenarios

Laboratory     1 simulation tool

28 IoT devices     Sensor data

Anomalous Y/N
6 months

3* <cit.>
Simulation (SYNT)     1 simulation tool

Generated     Lab (UNB)
12 computers

5 days

Real     US-Japan lines (MAWI)
5 sample points

Normal     12 months

t]@l@5 Smart homes,
8 identical IoT devices
t]@l@`Assumed benign,' 'rare yet
benign,' 'malicious' (crypto-
mining, Nmap scanning)
21 days

§.§ Collaborative anomaly detection

Although in many cases <cit.>, traffic-based IoT anomaly detection methods demonstrate excellent performance in detecting cyber-attacks (i.e., high recall), such methods also tend to suffer from a non-negligible FPR. One reason for this commonly seen drawback is that traffic anomalies can also be generated for non-malicious activities, such as infrequent (yet benign) user activities and contextual changes. Although infrequent, such actions should not trigger unnecessary risk-mitigating countermeasures for allegedly-detected cyber-attacks. To address this issue and reduce the FPR, we propose training the anomaly detector in a collaborative manner via data sharing while enriching the data with collaboratively engineered features. As part of our method, traffic data is collected from multiple IoT devices of the same model (i.e., many identical instances of an IoT product (of the same version), such as streamer.Amazon.Fire_TV_Gen_3 <cit.>), which are connected to various home networks and operated by different people. The motivation for training the anomaly detector in a collaborative manner using shared data is that the inherent richness and heterogeneity of such data allows the capturing of enough examples of similar local abnormalities which are rare yet benign, whereby globally they would be identified by the anomaly detector as sufficiently-frequent normal patterns.

Collaborative intrusion detection systems (CIDSs) are network intrusion detection systems (NIDSs) whose data is collected from various monitors that function as sensors (disparate IoT devices in our use case); these systems are comprised of several units that perform intrusion detection on the data collected from the sensors <cit.>. There are a few main CIDS architectures. The first is referred to as a centralized CIDS (data sharing) in which the data is sent to a central unit where the model performs computations on the data. Another architecture is referred to as a decentralized CIDS where there are several computation units located between the sensors and the model that deal with data aggregation and preprocessing. Distributed CIDSs are a form of collaboration architecture in which there is no central component and the analysis is done within the data monitors.

Many of the prior studies on (IoT-related) CIDSs are listed in Table <ref>. The authors of <cit.> based their method on the OPTICS algorithm for clustering (and cluster membership), and evaluated their method using IoT datasets to detect malicious traffic. In <cit.>, the authors presented a CIDS with multiple sparse AEs, one for each IoT device type. To evaluate their method, they used a dataset from one smart home with four different devices, however they did not publish their dataset, so their results cannot be reproduced. Two additional CIDSs, in which the proposed method is multi-staged (similar to our approach), are <cit.> and <cit.>. The former combines ARIMA, MLP, and LSTM to detect DDoS attacks, while the authors of the latter proposed a multi-stage CIDS which also combines active learning, however the experiments in both of these studies were conducted using non-IoT datasets.

In this research, similar to some of the studies presented in Table <ref>, we chose to implement the CIDS via data sharing, using a centralized architecture. The main reason for this choice is that in our use case (see Subsection <ref>), the traffic data of the monitored IoT devices is already collected in a central cloud server, and then it is preprocessed and analyzed in another existing server, incurring little overhead. Moreover, we do not assume a minimal level of processing power of the monitored IoT devices or home routers, or the users' consent to access their devices and use them for storage and/or computation; therefore, we prefer not to rely on the monitored IoT devices but rather on the readily-available central servers. As acknowledged by the authors of <cit.>, in reality, the communication and computational resources of the end nodes are typically limited and there may also be a time delay.

Given our preference for collaboration in the anomaly detection model's training, we could have opted for federated learning (FL) <cit.>, which was previously used for IoT security purposes <cit.>. As part of the FL collaborative approach, the sensors train (local) models and share their parameters (e.g., AE weights <cit.>) instead of sharing traffic data, mainly for privacy preservation and to limit the overhead of communication and storage <cit.>. However, in real-world scenarios, the deployment of FL on (typically resource-constrained) IoT devices might be infeasible due to memory, computing power, and energy consumption limitations <cit.>; such FL deployment would also necessitate obtaining the users' consent to remotely access their IoT devices to install the security software, which may also be infeasible. Moreover, FL often introduces a trade-off between privacy preservation and model performance in terms of accuracy <cit.>, efficiency <cit.>, and/or convergence <cit.>. Given the above, in this paper, as we wish to develop a feasible and high-performing CIDS mechanism for IoT device anomaly detection, we prefer the centralized collaborative approach over FL.

§.§ Existing public datasets for IoT attack detection

The availability of a scientific dataset is important for both (a) reproducing and validating related research, and (b) quantitatively evaluating future methods. In the NIDS domain, it was recently noted <cit.> that there is a dearth of datasets. This is also seen in the subdomain of IoT-related NIDSs, where there is a shortage of publicly available datasets, particularly high-quality datasets that (a) contain data captured over a long period of time and are (b) sufficiently representative of real-world settings, (c) properly labeled with present-day attacks, (d) comprised of data from multiple real devices and locations, such that they can be reliably used to evaluate novel methods for CIDSs. Table <ref> provides a description of the datasets used in prior research on (IoT) anomaly and attack detection. As can be seen in the table, not all of the datasets have been made publicly available. Of those that are publicly available, in some cases the research addresses IoT anomaly or attack detection but uses non-IoT data for evaluation <cit.>. The data of other datasets was captured in a period of just minutes <cit.> or hours <cit.>. Other studies used data which is simulated <cit.>, synthetic <cit.>, or generated in a lab <cit.>, sometimes using Raspberry PIs <cit.> instead of real IoT devices. Even if the data was captured from real-world settings, usually the dataset includes the data from just a small number of IoT devices <cit.> or a honeypot <cit.>, preventing the proper evaluation of a CIDS. With regard to the empirical evaluation of collaborative and FL approaches, another major limitation of existing datasets is that in most cases, the separation into nodes is only superficial. That is, traffic data is collected from just one IoT device deployed in one location and is later split using various strategies <cit.>. These data splits mimic nodes that perform collaboration, however the data splits fail to represent the diversity of identical devices which are deployed separately and interacted with by different users.

In comparison, in this research we utilize (and make publicly available) a novel dataset whose data was collected from eight identical IoT devices, deployed in six disparate networks, five of which are real home networks where the data is assumed benign, and one of which is in our controlled lab's network where we could implement IoT-related attacks and accurately label the traffic accordingly. In all of the homes the same IoT model was monitored, while different people in different surroundings and network settings freely interacted with the same IoT model. This allows us to quantitatively evaluate the proposed CADeSH method and set a benchmark for future (IoT-related, collaborative) NIDSs. In addition, our unique dataset covers a period of 21 days, which is relatively long time period compared to that of the datasets listed in Table <ref>. Another important aspect of our dataset is that unlike most datasets used thus far (some of which were published), it is not comprised of packet-level raw traffic data but rather is comprised mainly of IPFIX-level aggregated metadata, which results in lower communication/storage overhead while preserving privacy. We note that in <cit.>, a large and diverse IPFIX-based dataset was collected from an IoT testbed, however it (1) only contains benign data, limiting attack detection evaluation, (2) emanates from a single network only, such that collaborative learning can not be properly evaluated, and (3) reflects a limited frequency of human interaction, which is performed by just the lab staff, as opposed to the ongoing and natural real-world interaction with a diverse range of people in multiple disparate home networks reflected in our dataset.

Further details regarding the acquisition and properties of our novel dataset are provided in Section <ref>.

Our two-step method for collaborative IoT anomaly detection

§ PROPOSED METHOD

Overall, as illustrated in Fig. <ref>, the method we propose for (collaborative) IoT anomaly/attack detection in smart homes consists of two main steps. We refer to each of those steps as a filter, as some flows continue to the next step for further analysis while others are assigned a final label. Our method is trained (and is later used for inference) based on shared flow-level traffic data $F^m$ that is collected from multiple identical model $m$ IoT devices $D^m: \{ d^m_1, d^m_2,\dots ,d^m_n\}$, deployed in disparate home networks and interacted with by different users. As in any anomaly-based IDS, during training we rely on data which is assumed to be mostly benign so that a profile of the normal traffic patterns can be learned. To be on the safe side, our method includes also data selection, cleansing, and sanitization stages prior to training, as described in Subsection <ref>. Later, during inference, each outbound traffic flow $f^m$ is first labeled by $Filter_1^m$ as frequent (i.e., `benign' in terms of attack detection) or infrequent (i.e., somewhat anomalous, but not necessarily `malicious'). Afterwards, if labeled as infrequent by $Filter_1^m$, the flow is further labeled by $Filter_2^m$ as known behavior (i.e., `rare yet benign') or unknown (i.e., highly anomalous, presumably `malicious').

To implement $Filter_1^m$ and $Filter_2^m$, we propose (and empirically evaluate, see Sections <ref> and <ref>) the use of AEs and clusters, respectively. The reason for the former choice is that AEs have demonstrated effectiveness in binary IoT anomaly detection <cit.> and therefore could also facilitate the distinction between frequent and infrequent behaviors. Regarding the latter choice, the intuition is that rare yet benign traffic data can be divided into multiple clusters which represent various infrequent activities (slight anomalies such as a webcam booting or zooming in/out, etc.). A traffic flow $f^m$ which is too far from any of the rare yet benign clusters is considered a strong anomaly, presumably representing a cyber-attack.

To describe our method in further detail, we use the following notation:

$m$: An IoT model, defined <cit.> as the combination of type, manufacturer, and model number. For instance, webcam.D_Link.CS_930LB1 <cit.> and webcam.D_Link.DCS_933L <cit.> are two distinct IoT models that share the same type and manufacturer.

$D^m$: A set of \(n\) identical IoT devices $\{ d^m_1, d^m_2, \dots ,d^m_n\}$ defined by their MAC addresses, all of which are instances of model \(m\).

$f_i^m$: A single traffic flow that emerged from device \(d^m_i\), enriched with features from additional data sources such as DNS requests or the Webroot API; interchangeably referred to in this paper as an IPFIX, flow, record, or instance.

$F_i^m$: A set of \(x\) traffic flows $\{ f^m_{i1}, \dots ,f^m_{ix}\}$ that emerged from \(d^m_i\).

$F^m$: A superset of \(F^m_i\), containing flows from all of the devices in \(D^m\). We partition $F^m$ chronologically into three mutually exclusive parts: $F^m_{training}$ for training the filters, $F^m_{validation}$ for model optimization and anomaly threshold calibration, and $F^m_{test}$ for anomaly/attack detection evaluation.

$Filter_1^m$: The first anomaly detector, trained to determine whether a flow \(f^m_i\) represents a frequent or infrequent activity of device \(d^m_i\).

$Filter_2^m$: The second (subsequent) anomaly detector, trained to determine whether \(f^m_i\), which was already labeled by $Filter_1^m$ as infrequent, represents a known or unknown activity of device \(d^m_i\).

$k^*$: The optimal number of clusters in $Filter_2^m$, assuming $k$-means is used.

$MSE^m_f$: The mean squared error (i.e., the AE's reconstruction error), obtained by applying $Filter_1^m$ to a flow $f^m$. Since $Filter_1^m$ is trained using only assumed-benign data, a high $MSE^m_f$ value indicates a severe abnormality as the AE fails to reconstruct $f^m$.

$epochs_{max}$: A parameter to limit the number of epochs during which the AE's weights in $Filter_1^m$ are updated.

$delta_{min}$: A parameter to limit the duration of $Filter_1^m$'s training and decrease the chances that $Filter_1^m$ overfits the training data. $delta$ is the difference between the current and previous $MSE^m$ which is calculated using $F^m_{validation}$.

$patience_{max}$: An upper bound for the number of subsequent AE training epochs where $delta<delta_{min}$ (i.e., no substantial decrease in MSE is obtained). Reaching $patience_{max}$ is an indication of model convergence, and it is used to stop the AE's training even if $epochs_{max}$ has not been reached.

$TH^m_{frequent}$: Frequency threshold on $MSE^m$. Any (future or test) flow $f^m$ is labeled as infrequent if its reconstruction error $MSE_f^m$ exceeds $TH^m_{frequent}$, or as frequent otherwise. We propose setting $TH^m_{frequent}$ as a certain percentile (denoted as $pctl_{frequent}$) of $MSE^m$ on $F^m_{validation}$, e.g., the $60^{th}$ or $70^{th}$ percentile.

$\hat{f^m_{frequent}}$: The label assigned to a flow $f^m$ by $Filter_1^m$; it could be either frequent or infrequent. In our empirical evaluation, we denote the subsets of flows which were labeled infrequent by $Filter_1^m$ as $F^{m,\:infrq}_{training}$, $F^{m,\:infrq}_{validation}$, and $F^{m,\:infrq}_{test}$.

$TH^m_{known}$: A set of $k^*$ distance thresholds $th^m_{known}$, one for each cluster in $Filter_2^m$. Any (future or test) flow $f^m$ which was labeled as infrequent by $Filter_1^m$ is labeled by $Filter_2^m$ as unknown if its distance $d^{f,\:cluster}$ from the trained cluster to which it is assigned exceeds the related $th^m_{known}$ or as rare yet benign otherwise. We propose setting the cluster-specific thresholds in $TH^m_{known}$ as a certain percentile (denoted as $pctl_{known}$) of the related cluster distances $D^{cluster}$ on $F^{m,\:infrq}_{validation}$, e.g., the $95^{th}$, $98^{th}$, or $100^{th}$ percentile.

$\hat{f^m_{known}}$: The label assigned to a flow $f^m$ by $Filter_2^m$; could be either known (`rare yet benign') or unknown (`malicious').

§.§ $Filter_1^m$: frequent behavior or not

For a given IoT model $m$, the first step of CADeSH handles the training, optimization, and application of an anomaly detector $Filter^m_1$, which labels each outbound flow $f^m_i$ from an IoT device $d^m_i$ as either frequent or infrequent. As described in Algorithm <ref>, $Filter^m_1$ is trained and optimized using $F^m_{training}$ and $F^m_{validation}$, respectively, which are collected from multiple identical IoT devices $D^m$ deployed in multiple domestic networks and thus are regarded as highly representative of the typical traffic data associated with $m$. Since we do not have the actual ground truth label (i.e., `benign' or `malicious') for each flow $f^m_i$ gathered from outside our controlled lab, we assume that most of the flows in $F^m$ are benign and represent normal activity of the IoT devices of the respective model $m$. To increase the probability of using benign flows, we perform data sanitization (see Subsection <ref>). The weights of the AE underlying $Filter^m_1$ are updated using $F^m_{training}$ with MSE serving as the loss function. Training is stopped once the number of epochs exceeds $epochs_{max}$ or when the conditions for early stopping <cit.> are met, namely a sequence of epochs without a significant decrease of the MSE.

Input: $F^m_{training}$, $F^m_{validation}$, $epochs_{max}$, $delta_{min}$, $patience_{max}$

Output: $Filter^m_{1}$
[1]
$epoch \gets 1$
$epoch\_low\_MSE \gets 1$
$delta \gets \infty$
$MSE_{previous} \gets 0$
$epoch < epochs_{max}$ & $delta > delta_{min}$ & $epoch\_low\_MSE<patience_{max}$
$Filter^m_{1} \gets filter^m_{1}.update\_weights(F^m_{training})$
$MSE_{current} \gets filter^m_{1}.calc.\_MSE(F^m_{validation})$
$delta \gets MSE_{current} - MSE_{previous}$
$MSE_{previous} \gets MSE_{current}$
$epoch \gets epoch + 1$
$epoch\_low\_MSE \gets epoch\_low\_MSE+1$

Train the AE-based $Filter^m_1$

Input: $Filter^m_{1}$, $F^m_{validation}$, $pctl_{frequent}$

Output: $TH^m_{frequent}$
[1]
$MSE\_{validation} \gets []$
$f^m$ in $F^m_{validation}$
$MSE^m_f \gets filter^m_{1}.calc.\_MSE(f^m)$
$MSE\_{validation} \gets MSE\_{validation}.append(MSE^m_f)$
$MSE\_{validation} \gets MSE\_{validation}.sort()$
$TH^m_{frequent} \gets MSE\_{validation}.calc.\_percentile(pctl_{frequent})$

Set a frequency threshold for $Filter_1^m$

Algorithm <ref> is used to train an anomaly detector $Filter^m_1$ which captures assumed-benign activity of model $m$'s IoT devices. After $Filter^m_1$ has been trained, a decision regarding how to label each future or test $f^m$ ('frequent' or `infrequent') must be made. In order to set the frequency threshold $TH^m_{frequent}$, as described in Algorithm <ref>, we apply the AE-based $Filter^m_1$ to $F^m_{validation}$ and observe the distribution of the MSE. The higher the MSE value, the less frequent (more abnormal) $f^m$ is considered to be. We set the value of $TH^m_{frequent}$ as a certain percentile of $MSE^m_f$'s distribution on $F^m_{validation}$, e.g., the $60^{th}$ or $70^{th}$ percentile. This number reflects our expectation, based on findings from prior research <cit.>, that for IoT devices, a large portion of the normal traffic data is relatively repetitive, stable, and predictable, and thus its MSE should be rather low.

Eventually, as described in Algorithm <ref>, the label $\hat{f^m_{frequent}}$ of a given test flow is determined by comparing its $MSE^m_f$ (calculated using $Filter^m_1$) with the threshold $th^m_{frequent}$. A flow whose $MSE^m_f<th^m_{frequent}$ is labeled as frequent (i.e., no suspicion of any malicious activity), while a flow whose $MSE^m_f \ge th^m_{frequent}$ is temporarily labeled as infrequent and then inspected by $Filter^m_2$ to assign a final label, either known (rare yet benign) or unknown (malicious).

Input: $f^m$, $Filter^m_{1}$, $th^m_{frequent}$

Output: $\hat{f^m_{frequent}}$
[1]
$MSE^m_f \gets filter^m_{1}.calc.\_MSE(f^m)$
$MSE^m_f < th^m_{frequent}$ $\hat{f^m_{frequent}} \gets True$
$\hat{f^m_{frequent}} \gets False$

Determine whether a flow is frequent or not using $Filter_1^m$

§.§ $Filter_2^m$: known behavior or not

This component of our method only handles the flows that were previously classified by $Filter_1^m$ as infrequent and determines whether their abnormality is extreme enough to be considered a completely unknown behavior (presumably indicating a malicious activity) or is a known/legitimate behavior, i.e., rare yet benign. In this component, we propose (and empirically evaluate) the use of a clustering algorithm such as $k$-means <cit.>, assuming that the trained clusters in $Filter_2^m$ can represent various rare yet benign traffic patterns of an IoT model $m$.

Input: $F^{m,\:infrq}_{training}$, $k_{min}$, $k_{max}$

Output: $Filter^m_{2}$
[1]
$Avg\_Sil.\_Scores\gets []$
$k_{temp}$ in range($k_{min}$,$\:k_{max}$)
$clust.\_assignments_{temp} \gets k\_means.fit\_predict(F^{m,\:infrq}_{training},\:k_{temp})$
$Avg\_Sil.\_Score_{temp} \gets calc\_Avg\_Sil.\_Score(clust.\_assignments_{temp})$
$Avg\_Sil.\_Scores \gets Avg\_Sil.\_Scores.append(Avg\_Sil.\_Score_{temp})$
$k^* \gets arg\_max(Avg\_Sil.\_Scores)$
$Filter^m_{2} \gets k\_means.fit(F^{m,\:infrq}_{training},\:k^*)$

Train the ($k$-means) clustering-based $Filter^m_2$

To train and optimize the clustering model which implements $Filter_2^m$, we use $F^{m,\:infrq}_{training}$, which is a subset of $F^m_{training}$ where $MSE^m \ge TH^m_{frequent}$ (i.e., training instances previously classified as infrequent by $Filter_1^m$). This allows the model to focus only on data which is not entirely normal, such that better separation can be learned between real (malicious) anomalies and mediocre (rare yet benign) anomalies. As outlined in Algorithm <ref>, we find the optimal number of clusters $k^*$ by training a $k$-means model for each $k$ from the range $[k_{min}, ..., k_{max}]$ and selecting $k$ which maximizes the average silhouette score <cit.>.

Input: $Filter^m_{2}$, $F^{m,\:infrq}_{validation}$, $pctl_{known}$

Output: $TH^m_{known}$
[1]
$TH^m_{known} \gets []$
$cluster$ in $Filter^m_{2}$
$D^{cluster} \gets []$
$f$ in $F^{m,\:infrq}_{validation}$
$f \in cluster$ $d^{f,\:cluster} \gets distance(f,\:cluster_{center})$
$D^{cluster} \gets D^{cluster}.append(d^{f,\:cluster})$
$D^{cluster} \gets D^{cluster}.sort()$
$th^{m,\:cluster}_{known} \gets D^{cluster}.calc.\_percentile(pctl_{known})$
$TH^m_{known} \gets TH^m_{known}.append(th^m_{known})$

Set distance thresholds for $Filter_2^m$

Number of monitoring days, device locations, monitored devices, and IPFIXs for each label across our data partitions

1c|Data          Number of           1c|Device          Number of             4cNumber of IPFIXs
1c|partition     monitoring days     1c|location(s)     monitored devices     1c|Total     1c|Assumed benign     1c|Being scanned by Nmap     1cIs executing cryptomining
$F^m_{training}$                                    13                           2*5 X Non-lab                      2*7             1r|133,351           1r|133,351                    1r|
1-2 5-8
$F^m_{validation}$                                  3                                                                                1r|39,481             1r|39,481                      1r|
$F^m_{test}$                                        5                            1 X Lab only                                      1                              1r|27,255             1r|22,472                      1r|3,080                              1,703

Then, to set a maximum distance threshold $TH^m_{known}$ for each of the trained $k^*$ clusters, beyond which a (future or test) instance is considered abnormal enough to be labeled as unknown (malicious), we use Algorithm <ref>: We assign each (infrequent though assumed-benign) instance $f$ from $F^{m,\:infrq}_{validation}$ to its nearest cluster using $Filter_2^m$ and observe the distribution of distances $D^{cluster}$ between each instance and the center of its assigned cluster. For each cluster, the abnormality threshold $th^m_{known}$ is set to be a certain percentile of that cluster's distance distribution, e.g., the $99^{th}$ or $95^{th}$ percentile of $D^{cluster}$. The rationale for doing so is that the distribution of cluster distances on $F^{m,\:infrq}_{validation}$ should be representative of infrequent though assumed-benign data, so that extremely high distances ought to raise suspicion concerning malicious activities.

Input: $f$, $Filter^m_{2}$, $th^m_{known}$

Output: $\hat{f^m_{known}}$
[1]
$cluster \gets arg\_min(distance(f,\:cluster_{center}))$
$d^{f,\:cluster} \gets distance(f,\:cluster_{center})$
$d^{f,\:cluster} < th^m_{known}$ $\hat{f^m_{known}} \gets True$
$\hat{f^m_{known}} \gets False$

Determine whether a flow is known or not using $Filter_2^m$

Eventually, as outlined in Algorithm <ref>, a final decision for infrequent test instances is made based on (1) assigning them to the nearest cluster and (2) comparing them to their cluster-specific distance threshold $th^m_{known}$.

§ EVALUATION METHOD

§.§ Experimental setup

We were motivated to conduct this research by a company that provides cyber-defense services for smart homes, and this company provided the data used in our experiments. To protect the company's subscribers against various IoT-related attacks, flow-level traffic data is continuously captured from the monitored home networks, stored in a cloud server, and then augmented and analyzed centrally to enable risk-mitigating responses. As can be seen in Fig. <ref>, in each home network various devices (both IoT and non-IoT devices) are connected to the Internet via a router. This router sends the traffic data to a designated cloud storage server, from which data is sent to another server for analysis. Note that our lab network has the exact same setup as the home networks. The only difference is that in our lab we connected a laptop (denoted as `Attacker' in Fig. <ref>), which was used to implement cyber-attacks, to the network (see Subsection <ref>).

Our experimental setup

The abovementioned cyber-defense company kindly shared the following data with us: network traffic flows which were collected during a period of 21 days (see Table <ref>), plus the related DNS requests and responses and reputation intelligence of the destination IP addresses, produced by Webroot. To evaluate our IoT anomaly/attack detection method, we implemented the cyber-attacks during that 21 day period, such that the company's network traffic flows were collected at the same time, using the same setup as the other (home) networks. In our lab, as described next, we merged the various data sources, added a number of engineered features, and formed a single dataset for experimentation. For this process of data merging, engineering, preprocessing, and analysis, we used a Windows 2016 server with a 2GHz E5-2620 CPU and 256GB RAM.

§.§ Dataset

Instances. Each instance in our dataset represents an outbound network traffic flow (in the form of an IPFIX <cit.>), enriched using additional data sources. An IPFIX is a 5-tuple aggregation of the raw traffic data (source IP address, destination IP address, source port, destination port, and IP protocol). Most of the IPFIXs in our dataset were collected from real home networks by the company we cooperated with, and the remaining flows were collected in our lab, using the same setup used by the company's home subscribers. In our experiments, we concentrated on the IoT model, streamer.Amazon.Fire_TV_Gen_3 <cit.>, because it was deployed in five monitored home networks and we could empirically evaluate our collaborative algorithm and its generalizability to a sixth network (in our lab).

Each instance is comprised of several feature groups (Appendix <ref> contains a list of the feature names in each feature group, as well as brief descriptions):

* Raw IPFIX features, such as IP protocol identifier, number and size of packets in a flow, destination network, flow end reason, etc.

* Features calculated using the raw features of the same IPFIX, such as flow IAT.

* DNS-related features, such as the percentage of numerical characters in the requested domain, etc.

* Reputation intelligence features <cit.>, such as the reputation status of the destination IP address, which denotes whether the destination is likely to be associated with phishing attacks, spam, mobile threats, etc.

* `Time-based' features (which partially resemble the stateful features used in <cit.>), calculated using the raw features of the preceding IPFIXs of the other devices of the same IoT model. These two `time-based' features count the number of times during the previous complete hour in which any of the other devices contacted (1) the network of the current IPFIX's destination IP address and (2) its destination port, as another means of collaboration for anomaly/attack detection. The rationale is that an outbound flow whose destination network and/or port were not recently contacted by the counterparts of the inspected device could indicate an anomalous or even malicious behavior, e.g., communication with the C&C (command and control) server of a botnet <cit.>.

§.§ Implemented attacks and corresponding labels

In our lab, we infected streamer.Amazon.Fire_TV_Gen_3 with a cryptominer <cit.> and executed cryptomining from this device. To imitate a scanning activity typically performed by some botnets, we also scanned the network using Nmap (Network Mapper), as in <cit.>. In accordance, we labeled these malicious activities as (1) `is executing cryptomining,' or (2) `being scanned by Nmap.' All of the remaining IPFIXs captured in our lab or on the home networks were labeled as `assumed benign,' mainly because we could not clearly substantiate the ground truth.

§.§ Data merging, enrichment, preprocessing, and partitioning

The first stage of turning the raw data into a dataset ready for experimentation was selecting the outbound IPFIXs in which the source was the monitored streamer.Amazon.Fire_TV_Gen_3 devices (as opposed to inbound IPFIXs where these devices were the destination) and combining the selected outbound IPFIXs with their DNS requests and responses and their destinations' reputation intelligence data. Then we enriched the merged data by extracting features such as flow duration and IAT, DNS-related features, and (collaborative) time-based features.  Afterwards, as part of preliminary data preprocessing, we removed IPFIXs whose source MAC address or IAT were missing, removed the first two recorded hours of traffic data (because its time-based features could not be calculated properly), and in cases in which the time-based features were missing values, we input a value of zero.

After constructing the preliminary dataset and performing preprocessing, we labeled and partitioned it chronologically into three mutually-exclusive subsets, namely $F^m_{training}$, $F^m_{validation}$, and $F^m_{test}$ (13, 3, and 5 days, respectively, 21 days altogether). The total number of IPFIXs for each label in these data partitions is provided in Table <ref>. Then, as a final stage of preparing the dataset for experimentation, we performed several data selection and cleansing operations, mostly (1) ensuring that $F^m_{test}$ only includes instances captured in our controlled lab (and thus could reliably be used to evaluate anomaly/attack detection performance), while (2) ensuring that $F^m_{training}$ and $F^m_{validation}$ only include instances that were captured in another network where streamer.Amazon.Fire_TV_Gen_3 was deployed. This data selection strategy enabled us to evaluate the generalizability of our collaborative method and answer the question: Can an anomaly detector, which was trained and optimized using shared traffic data from multiple identical IoT devices deployed in various other networks, effectively detect IoT-related cyber-attacks that involve an identical IoT device that was not represented in $F^m_{training}$ or $F^m_{validation}$? Additional data cleansing operations included (3) keeping only external communications (i.e., communication sent outside each device's network) <cit.>; (4) keeping only assumed-benign flows in $F^m_{training}$ and $F^m_{validation}$, as opposed to $F^m_{test}$ which contains both malicious and assumed-benign flows; and (5) sanitizing $F^m_{training}$ <cit.> in order to avoid model poisoning <cit.> by removing extreme outliers, i.e., excluding training flows whose destination ports are too scarce, collaboratively considering all of the IoT devices in $F^m_{trn}$.

The entire process of data partitioning and analytical flow is illustrated in Fig. <ref>. To remove ambiguity and enable research reproducibility, we share the final cleansed and preprocessed dataset with the public, anonymized and ready for experimentation.

Data partitioning and analytical flow for evaluation of our method

Metrics used to evaluate our method

1c|Metric                      1c|Abbreviation             1c|Calculation                                   1cCoverage
True positives                                     TP                                            $\#(actual = malicious,\:predicted = `malicious')$                   9*t]@l@A given
False positives                                    FP                                            $\#(actual = benign,\:predicted = `malicious')$
True negatives                                     TN                                            $\#(actual = benign,\:predicted = `benign')$
False negatives                                    FN                                            $\#(actual = malicious,\:predicted = `benign')$
False positive rate                              FPR                                           $\frac{FP}{(FP+TN)}$
Precision                                                                                $\frac{TP}{(TP+FP)}$
Recall                                                                                $\frac{TP}{(TP+FN)}$
F1 score                                          F1                                            $2 \cdot \frac{precision \cdot recall}{precision + recall}$
Area under the precision-recall curve             AUPRC
Macro-average metric         $\frac{\sum <metric>}{\#(attack\:scenarios)}$     Multiple attack scenarios

§.§ Hyperparameter tuning

In our experiments we sought to optimize the following five hyperparameters that we suspected would have an effect on our method's performance, as follows:

* Destination IP addresses have been used extensively in past research <cit.> for IoT device fingerprinting, since IoT devices regularly communicate with a limited set of (their manufacturers') servers. However, when one-hot encoding the destination IP address, dimensionality and sparsity are likely to increase and thus harm model performance. Moreover, while a domain name typically remains unchanged, the associated IP addresses can change, so the model generalizability might decrease over time. To address these challenges, in our experiments we tested whether completely disregarding the IP addresses improves performance. We also tested the effect of using (and then one-hot encoding) only the prefixes of the destination IP addresses, which indicate the networks of the hosts <cit.>.

* Different transformations of numerical features were previously found <cit.> to have varying levels of impact on model performance, depending on the dataset and algorithm selected. In our experiments we chose to compare the use of numerical features as is vs. log-transforming them.

* As noted in Subsection <ref>, we expect most of the normal outbound flows of an IoT device (presumably around $\frac{2}{3}$) to be relatively stable. So, in order to set $Filter^m_1$'s threshold $TH^m_{frequent}$ we compared between choosing the $60^{th}$ or the $70^{th}$ percentile of the MSE using $F^m_{validation}$. Lower thresholds are expected to increase the number of FPs using $F^m_{test}$, while higher values are likely to increase the number of FNs.

* Having accumulated a large variety of features in the dataset, we decided to compare four options of the feature set to be used to implement the cluster-based $Filter_2^m$: (1) all of the available features, (2) a manually selected feature subset which is comprised of the `immediate suspects,' namely octet_delta_count, avg_packet_size, flow_duration_milliseconds, same_dest_ip_count_pool, and same_dest_port_count_pool (see Appendix <ref> for feature descriptions), (3) a PCA transformation of the original features, and (4) the hidden layer of the AE underlying $Filter_1^m$, as in <cit.> and <cit.>.

* In $Filter_2^m$, the distance of a flow $f$ from the cluster to which it is assigned is used as a measure of abnormality. In our experiments we compared two approaches for calculating this distance: (1) the raw Euclidean distance between $f$ and the cluster centroid, and (2) the normalized Euclidean distance, i.e., the number of standard deviations from the mean. In order to restrict these distance measures to the range [0, 1], as in probability, we eventually compared $tanh(raw\:Euclidean\:distance)$ with $tanh(normalized\:Euclidean\:distance)$.

§ RESULTS

In this section, we present the results of the hyperparameter tuning process, providing the hyperparameter values that resulted in the best performance using $F^m_{test}$. Then, using the best-performing combination of hyperparameter values, we present the results obtained by $Filter^m_1$ and $Filter^m_2$ (Subsections <ref> and <ref>, respectively) using $F^m_{test}$. In Subsection <ref>, we summarize the overall empirical results of our two-step method and compare these results to those of classical one-step anomaly detection algorithms, and in Subsection <ref>, we analyze the sensitivity of our method to a high-importance training consideration, namely the quantity of most recent IPFIXs present in the training set. Throughout this section we evaluate our results using the performance metrics defined in Table <ref>.

§.§ Hyperparameter tuning results

Table <ref> lists the five hyperparameters tuned throughout our experiments (described in Subsection <ref>) and indicates the value that performed best using $F^m_{test}$ in terms of the macro-average AUPRC (area under the precision-recall curve). We denote this best-performing combination as $hyp^*$. We chose to use the macro-average, because our dataset is imbalanced with respect to the malicious labels. We reached $hyp^*$ after conducting an exhaustive grid search, where the duration of a single end-to-end experiment was approximately $48\pm24$ minutes ($Mean \pm St.Dev.$), using the experimental setup described in Subsection <ref>.

Hyperparameter tuning performed in our experiments

1c|Hyperparameter                                          1c|Evaluated values     Performed
2*t]@l@Treatment of the
destination IP address                                Remove completely
Use only the prefix (network)
2*Treatment of numeric features                                          Log transform
Use as is
2*t]@l@MSE threshold between `frequent'
and `infrequent' ($Filter_1^m$)             $60^{th}$ percentile using $F^m_{validation}$
$70^{th}$ percentile using $F^m_{validation}$
4*t]@l@Features used for clustering
($Filter_2^m$)                                    All available features
AE's hidden layer
Manually selected feature subset
2*t]@l@Cluster distance to differentiate between
`known' and `unknown' ($Filter_2^m$)     Centroid distance
Normalized distance from the mean

Effect of the various tested hyperparameters on the macro-average AUPRC using $F^m_{test}$, estimated through 95% confidence intervals (CIs) and ANOVA

Based on the grid search we conducted, the macro-average AUPRC using $F^m_{test}$ was $0.286\pm0.206$, out of which the highest value (0.841) was attained by using the $hyp^*$ values presented in Table <ref>. Overall, as can be seen in Fig. <ref>, four of the five hyperparameters tuned had no significant effect on the  macro-average AUPRC using $F^m_{test}$. The fifth hyperparameter was the clustering feature treatment. An ANOVA test found a significant difference in means ($P-value=0.0$ (see Fig. <ref>)). This ANOVA test was followed by pairwise Tukey-HSD post-hoc tests which found that using the hidden AE layer is inferior to using all of the original features ($P-value=0.034$) and to the manually-predefined subset of the original features ($P-value=0.011$).

MSE (loss) convergence during training using $hyp^*$

§.§ Results for $Filter^m_1$

We used the Keras library <cit.> to implement $Filter_1^m$ as an AE with three hidden layers, i.e., a total of five dense fully-connected layers, including the input and output layers. We set the number of neurons in those layers to be 100-50-25-50-100% of the input dimension. ReLU was used in all of the layers, except for the output layer, where we used a sigmoid activation function. We used a batch size of 64 and the Adam optimizer <cit.>. Although we allowed $epochs_{max}=200$, in most cases the training of $Filter_1^m$ stopped much earlier: $37.8 \pm 21.1$  epochs due to meeting the early stopping criteria: $delta_{min}=0.00001,\:patience_{max}=5$. The rapid convergence when using $hyp^*$ can be seen in Fig. <ref>.

MSE distribution across partitions

1c|Percentile     $MSE^{training}$     $MSE^{validation}$     $MSE^{test}_{benign}$     $MSE^{test}_{malicious}$
Min.                       0.0000                    0.0000                      0.0000                       0.0010
25%                       0.0002                    0.0002                      0.0002                       0.0546
50%                       0.0004                    0.0003                      0.0004                       0.0565
75%                       0.0011                    0.0010                      0.0226                       0.0565
Max.                       0.1801                    0.1802                      0.1960                       0.1984

The results presented in Table <ref> show that for the most part, the MSE of the trained AE-based $Filter^m_1$ is very low for benign IPFIXs, indicating the ability of $Filter^m_1$ to accurately reconstruct benign inputs. Moreover, the MSE on benign test records is almost identical to the MSE on benign training and validation records. This indicates the ability of $Filter^m_1$ to generalize to data on which it was not trained. It is also worth noting that the $25^{th}$ and $50^{th}$ percentiles of $MSE_{malicious}^{test}$'s distribution are much larger than the respective values on $MSE_{benign}^{test}$ by more than two orders of magnitude. This bolsters the very basic assumption behind anomaly-based NIDSs, namely that there is an association between (network traffic) data anomalies and malicious activity.

To differentiate between a frequent and infrequent IPFIX, we set the threshold $TH^m_{frequent}$ as the $60^{th}$ percentile of the MSE's distribution on $F^m_{validation}$. Using this threshold, as can be seen in Table <ref>, $Filter^m_1$ did not produce any FNs on $F^m_{test}$, meaning that none of the actually malicious IPFIXs were mistakenly labeled by $Filter_1^m$ as frequent. The 10,489 IPFIXs labeled as infrequent, which could actually be `rare yet benign' or `malicious', proceeded to $Filter^m_2$ for further labeling as known or unknown, respectively.

Number of predicted vs. actual labels for $Filter_1^m$ and $Filter_2^m$ using $F^m_{test}$ and $hyp^*$

1c||3*Actual     6cPredicted
1c||                                     3c||$Filter_1^m$          3c$Filter_2^m$
Assumed benign                                            1r|11,983     10,489     22,472     10,174           315             10,489
Being scanned by Nmap                                     1r|-          3,080      3,080      48               3,032           3,080
Is executing cryptomining                                 1r|-          1,703      1,703      -                1,703           1,703

§.§ Results for $Filter^m_2$

Determination of
k based on the average silhouette score using $F^{m,\:infrq}_{validation}$

To implement $Filter_2^m$, we used $k$-means clustering and found that within the range of [$k_{min}=2,\:k_{max}=20$] the number of clusters that maximizes the average silhouette score <cit.> using $F^{m,\:infrq}_{training}$ is $k^*=15$ (see Fig. <ref>). To differentiate between a known and unknown behavior, we utilized the distance of the inspected IPFIX from the nearest cluster. To align this metric with a probability expression, we considered $tanh(cluster-distance) \approx p(abnormality)$. The distribution of the raw distances from the $k^*=15$ cluster centers is illustrated in Fig. <ref> for each cluster separately (ranging up to $\sim1.7$ using $F^{m,\:infrq}_{validation}$).

Distribution of the distance from each cluster in $F^{m,\:infrq}_{validation}$

In contrast, as shown in Fig. <ref>, the distribution of $tanh(cluster-distance)$ using $F^m_{test}$ is bounded to within the range of $[0,\:1]$, where it can also be seen that a threshold of $0.75$ offers a good separation between most assumed-benign IPFIXs and malicious IPFIXs.

Tanh(distance) in $F^m_{test}$ to differentiate between known and unknown IPFIXs

Performance of the KPIs as a function of the threshold on tanh(Euclidean distance from cluster centers) in $Filter_2^m$ for two attack scenarios: The IoT device is (a) being scanned by Nmap, and (b) executing cryptomining

§.§ Overall results and benchmarking

The KPIs which summarize the overall performance of our method using $F^m_{test}$ and $hyp^*$ are presented in Table <ref>. The bottom row of the table shows that the sequential use of $Filter_1^m$ and $Filter_2^m$ achieves a 0.841 macro-average AUPRC. In contrast to this KPI, which considers a multitude of threshold values on $tanh(cluster-distance)$ in $Filter^m_2$, the top four rows relate only to one threshold, denoted as $TH^m_{known}$. As can be seen in Figs. <ref> and <ref>, this threshold has a substantial impact on the precision, recall, F1 score, and FPR for the detection of Nmap scanning and cryptomining execution, respectively. Setting $TH^m_{known}=0.75$ yields a 0.929 macro-average F1 score. The recall is also very high (it is actually perfect for cryptomining execution detection), while the FPR has a relatively low value of 0.014 for both of the evaluated attacks.

Summary of our method's KPIs using $F^m_{test}$ and $hyp^*$

1c|Metric     Being scanned
by Nmap     Is executing
cryptomining     Macro-
average     $TH^m_{known}$
Precision     0.906     0.844     0.875     4*0.75
Recall        0.984     1.000     0.992
F1 score            0.944     0.915     0.929

FPR           0.014     0.014     0.014

AUPRC         0.827     0.855     0.841          1l

For benchmarking, we compared our two-step method with a variety of classical one-step algorithms widely used for anomaly and novelty detection in general <cit.>, and for IoT attack detection in particular <cit.>. Two of these algorithms are the building blocks of CADeSH (namely, AE and $k$-means clustering, which we used to implement $Filter^m_1$ and $Filter^m_2$, respectively). The remaining three are OCSVM <cit.>, LOF <cit.>, and IF <cit.>. This algorithm comparison (summarized in Table <ref>) takes into account both attack detection performance (in terms of AUPRC) and the upper bound of computational complexity (denoted using the big O notation). As can be seen in Table <ref>, on macro-average, our two-step CADeSH method outperformed all of the one-step algorithms in terms of the AUPRC. That is, although $k$-means and OCSVM performed better than CADeSH in the detection of Nmap scanning, they performed much worse in detecting cryptomining execution, and vice versa; the AE performed better than CADeSH in the detection of cryptomining execution, however it performed (much) worse in detecting Nmap scanning. In terms of computational complexity, we note that CADeSH is relatively expensive. Still, it is our opinion that attack detection performance is much more crucial in real-life scenarios. Moreover, when solutions like parallel execution using, e.g., graphics processing units (GPUs) are widely available, the actual time differences between the algorithms become less and less significant.

Comparison of our two-step method's with various one-step novelty detection algorithms

1c||                    3c||Attack detection AUPRC using $F^m_{test}$ and $hyp^*$                                                                                2cComputational complexity <cit.>
1c||Method     1c|Being scanned by Nmap     1c|Is executing cryptomining     Macro-average     1c|Upper bound                                   1cTerms
CADeSH                                   1c|0.827                              1c|0.855                                  0.841             1l|$O(k\operatorname{-}Means)+O(AE)$                                       6*c]@l@$d$: number of dimensions (features)
$k$: number of clusters to train
$r$: number of records
$I$: number of iterations until convergence
$t$: number of trees in the ensemble
$s$: sub-sampling size
k-Means                                  1c|0.970                     1c|0.167                                  0.569                      1l|$O(d \times k \times r \times I)$
AE                                       1c|0.176                              1c|0.917                         0.547                      1l|$O(d^2)$ for each layer
OCSVM                                    1c|0.885                              1c|0.129                                  0.507                      1l|$O(r \times d^2)$ for non-linear kernels
LOF                                      1c|0.590                              1c|0.240                                  0.415                      1l|$O(d^2)$
IF                                       1c|0.338                              1c|0.089                                  0.214                      1l|$O(t \times s \times log(s)$)

Sensitivity of CADeSH to the quantity of most recent training data

§.§ Sensitivity analysis

In our experiments, we also explored an issue that might need to be dealt with when considering real-world application of the proposed method, namely the quantity of training data required for (re-)training and how the value selected affects CADeSH's performance. The results of this experiment using $hyp^{*}$ and $F^m_{test}$ are presented in Fig. <ref>, where the macro-average precision, recall and F1 score are presented as a function of the number of most recent IPFIXs used for model training. In each iteration of this experiment, the selected IPFIXs were chronologically divided into a training and validation set, 80% and 20% of the data, respectively.

As can be seen in Fig. <ref>, although the best performance is attained using the entire training data (a total of 172,832 IPFIXs), an almost-identical performance level is attained using less than a quarter of this quantity (40,000 IPFIXs). On the one hand, smaller quantities of data might not represent enough human interactions and behaviors that are captured in the network traffic data, thus they lead to model under-fitting. On the other hand, in most cases larger quantities of training data lead to model over-fitting. A possible explanation is that these 40,000 IPFIXs (equivalent to approximately four days of traffic capturing) were the closest to the test set, such that they best represent the most recent normal activity based on which anomalies are detected. For example, if a given domain that is often contacted by model-$m$ IoT devices via DNS requests is not associated with a static IP address, then outdated data (from, e.g., over a week ago) might confuse the model.

§ DISCUSSION

Despite its promising attack detection performance, our method has the following limitations. First, it requires the same data collection setup (hardware and/or software) across all of the monitored networks for efficient and reliable data sharing, and thus it might not be easy to scale. In our use case, the company we cooperated with supplied a piece of hardware to home subscribers as a third-party cyber-security service. This hardware enabled both the service of continuous smart home monitoring for malicious activities, and the standardized collection of data which can be shared and used for model training; however, it is possible that in the future ISPs would offer such a service themselves and preinstall the necessary software on the home routers they supply, instead of an additional hardware. Second, our method requires correct IoT model identification prior to anomaly detection, in order to enable the training and deployment of an anomaly detector for each IoT model separately; to this end, there are already promising research results regarding IoT identification based on IPFIX records, even behind a NAT <cit.> (which is a typical setting in home networks). Third, as noted by <cit.>, behavioral changes are not rare in the IoT, which could have an effect on the profile of normal network patterns. This might necessitate relatively frequent model retraining, depending on the degree to which the traffic patterns of each monitored IoT model $m$ are sensitive to behavioral changes. Yet, as described in Subsection <ref>, only a small amount of (recent) training data is sufficient to produce a high-performing anomaly detector. Retraining might also be required following software updates, something which has yet to be empirically evaluated.

§ CONCLUSION AND FUTURE WORK

In this work, we proposed CADeSH, a novel collaborative two-step anomaly detection method. Using this method, an ISP can detect anomalous behaviors, which are indicative of malicious activities on specific IoT devices, locally (e.g., in a smart home). We evaluated our method empirically using real-world data from seven distinct streamer.Amazon.Fire_TV_Gen_3 devices deployed in the networks of five real homes and interacted with naturally by different people over a period of 21 days. In our experiments we trained the anomaly detection models using data from those home networks (for which we did not have ground truth labels) and then tested the models using an identical device deployed in our controlled lab and carefully labeled this device's outbound traffic as `assumed benign,' `being scanned by Nmap' (imitating a botnet scan), or `is executing cryptomining.' These two well-known IoT-related cyber-attacks were detected by our method with a recall value of 0.929 (on macro-average), precision of 0.875, and an FPR as low as 0.014. A grid search conducted for hyperparameter tuning, followed by sensitivity analysis, revealed that training data of just 40,000 IPFIXs was enough to obtain excellent performance in this study.

Although in this work we focused on detecting anomalous behaviors locally, in the future our method could also be leveraged by an ISP to detect abnormal nationwide trends. In other words, an IPFIX that is labeled as highly abnormal (i.e., representing a behavior which is both infrequent and unknown) is a strong indication of local malicious activity. In comparison, a multitude of such abnormalities identified simultaneously on many IoT devices from a sufficiently large number of networks is likely to indicate the propagation of a cyber-attack, e.g., botnet scanning or DDoS execution. We plan to design and evaluate this in our future work. Further extensions to this research include the evaluation of additional IoT device models, additional IoT-related attack scenarios, and additional algorithms and configurations for implementing $Filter^m_1$ and $Filter^m_2$, i.e., more complicated than a five-layer AE and $k$-means clustering.

§ ACKNOWLEDGMENT

The authors would like to thank Dominik Breitenbacher, Tar Wolfson, Itamar Biton, Julia Tulisov, and Liam Habani for their valuable contributions to this research project.

[]Yair Meidan is a Ph.D. candidate in the Department of Software and Information Systems Engineering (SISE) at Ben-Gurion University of the Negev (BGU). His research interests include machine learning and IoT security. Contact him at<EMAIL_ADDRESS>
[]Dan Avraham is an M.Sc. student in the SISE Department at BGU. His research interests include machine learning and cyber-security. Contact him at<EMAIL_ADDRESS>
[]Hanan Libhaber is an M.Sc. student in the SISE Department at BGU. His research interests include machine learning and cyber-security. Contact him at<EMAIL_ADDRESS>
[]Asaf Shabtai is a professor in the SISE Department at BGU. His research interests include computer and network security, and machine learning. He received a Ph.D. in information systems from BGU. Contact him at<EMAIL_ADDRESS>
§.§ Features in our publicly available dataset

The following is a brief description of the features in our publicly available dataset:

* IPFIX identifiers

* device_id: An identifier of the inspected device from which this flow emanated (0-6 represent the devices deployed in the home networks, 7 represents the device in our lab)

* flow_start_day: The day of the first packet of this flow

* flow_start_hour: The hour of the first packet of this flow

* flow_start_minute: The minute of the first packet of this flow

* flow_start_second: The second of the first packet of this flow

* flow_start_millisecond: The millisecond of the first packet of this flow

* source_network_id: An identifier of the home network from which this flow emanated (0-4 represent the home networks, 5 represents the network in our lab)

* Raw IPFIX features <cit.>

* protocol_identifier: The protocol number in the IP packet header, which identifies the IP packet payload type (6 represents TCP, 17 represents UDP)

* flow_duration_milliseconds: The difference in time [milliseconds] between the first observed packet of this flow and the last observed packet of this flow

* octet_delta_count: The number of octets sent in this flow, including IP header(s) and IP payload

* packet_delta_count: The number of packets sent in this flow

* avg_packet_size: The average size of packets in this flow, calculated as $octet\_delta\_count / packet\_delta\_count$

* flow_end_reason: The reason for this flow's termination, e.g., idle timeout, TCP FIN flag, etc.

* tcp_control_bits: The TCP control bits observed for the packets of this flow (SYN, ACK, FIN, PSH, RST, URG), encoded as a bit field (1 if present in any of this flow's packets, 0 otherwise)

* Features calculated using the raw features of the same IPFIX

* network_class_of_destination_IP_address: Determines which part of the IP address belongs to the network prefix and which part belongs to the host suffix (see <cit.>)

* network_prefix_of_destination_IP_address_anonimized: An anonimized prefix of the destination IP address, which indicates <cit.> the network of the host

* inter_arrival_time_milliseconds: The time [milliseconds] between the previous flow's start time and this flow's start time, calculated for this device_id as $flow\_start_{(t)}-flow\_start_{(t-1)}$

* Reputation intelligence features <cit.>

* reputation_status: A flag indicating whether the destination IP address is suspected of being involved in any of the following categories of malicious activity: spam, windows exploits, web attacks, botnets, scanners, DoS, reputation, phishing, proxy, mobile threats, or tor proxy.

* `Time-based' features (calculated using the full complete hour preceding flow_start)

* same_dest_port_count_pool: A collaborative feature indicating the number of times in which the destination port was contacted by any of the monitored devices (identified by their device_id) within the said time window

* same_dest_IP_count_pool: A collaborative feature indicating the number of times in which network_prefix_of_destination_IP_address_anonimized was contacted by any of the monitored devices within the given time window

* DNS-related features

* has_DNS_request_from_pool: A (collaborative) flag indicating whether a preceding DNS request that can be associated with this flow was captured

* DNS_host_percentage_of_numerical_chars_from_pool: Percentage of numerical characters in the requested DNS host

* Experiment-specific features

* actual_label: The actual label of this flow (either `assumed benign' or any of the attacks we implemented in our lab)

* partition: The subset to which this flow belonged in our experiments (either training, validation, or test)

§.§ Nomenclature

The following is a list of the acronyms we used in this paper (presented in alphabetical order), as well as their definitions.

ARIMAAutoregressive Integrated Moving Average
AUPRCArea under the precision-recall curve
BIRCHBalanced Iterative Reducing and Clustering using Hierarchies
C&CCommand and control
CADeSH(The name of our proposed method) Collaborative Anomaly Detection for IoT attack detection in Smart Homes
CIDSCollaborative intrusion detection system
DBSCANDensity-Based Spatial Clustering of Applications with Noise
DDoSDistributed denial-of-service
DLDeep learning
DNNDeep Neural Network
DPIDeep packet inspection
FLFederated learning
FPFalse positive
FPRFalse positive rate GPUGraphics processing unit
GRUGated Recurrent Unit
IATInterarrival time
IDSIntrusion detection system
IFIsolation Forest
IoTInternet of things
IPFIXInternet Protocol Flow Information Export
ISPInternet service provider
LOFLocal Outlying Factor
LSTMLong Short-Term Memory
MCODMulti-Level Outlier Detection
MLMachine learning
MLPMultilayer Perceptron
MSEMean squared error
NIDSNetwork intrusion detection system
NmapNetwork Mapper
OCSVMOne-Class Support Vector Machines
OPTICSOrdering Points to Identify the Clustering Structure
PCAPrincipal Component Analysis
SDNSoftware defined network
VSLVirtual state layer