# CyberSpec: Intelligent Behavioral Fingerprinting to Detect Attacks on
Crowdsensing Spectrum Sensors

Alberto Huertas Celdrán∗1, Pedro Miguel Sánchez Sánchez2, Gérôme Bovet3,
Gregorio Martínez Pérez2, and Burkhard Stiller1 ∗Corresponding author.1Alberto
Huertas Celdrán and Burkhard Stiller are with the Communication Systems Group
(CSG) at the Department of Informatics (IfI), University of Zurich UZH, 8050
Zürich, Switzerland (e-mail<EMAIL_ADDRESS>stiller@ifi.uzh.ch).2Pedro
Miguel Sánchez Sánchez and Gregorio Martínez Pérez are with the Department of
Information and Communications Engineering, University of Murcia, 30100
Murcia, Spain<EMAIL_ADDRESS>gregorio@um.es).3Gérôme Bovet is
with the Cyber-Defence Campus within armasuisse Science & Technology, 3602
Thun, Switzerland (gerome.bovet@armasuisse.ch).

###### Abstract

Integrated sensing and communication (ISAC) is a novel paradigm using
crowdsensing spectrum sensors to help with the management of spectrum
scarcity. However, well-known vulnerabilities of resource-constrained spectrum
sensors and the possibility of being manipulated by users with physical access
complicate their protection against spectrum sensing data falsification (SSDF)
attacks. Most recent literature suggests using behavioral fingerprinting and
Machine/Deep Learning (ML/DL) for improving similar cybersecurity issues.
Nevertheless, the applicability of these techniques in resource-constrained
devices, the impact of attacks affecting spectrum data integrity, and the
performance and scalability of models suitable for heterogeneous sensors types
are still open challenges. To improve limitations, this work presents seven
SSDF attacks affecting spectrum sensors and introduces CyberSpec, an ML/DL-
oriented framework using device behavioral fingerprinting to detect anomalies
produced by SSDF attacks affecting resource-constrained spectrum sensors.
CyberSpec has been implemented and validated in ElectroSense, a real
crowdsensing RF monitoring platform where several configurations of the
proposed SSDF attacks have been executed in different sensors. A pool of
experiments with different unsupervised ML/DL-based models has demonstrated
the suitability of CyberSpec detecting the previous attacks within an
acceptable timeframe.

###### Index Terms:

Crowdsensing Spectrum Sensors, SSDF Attacks, Behavioral Fingerprinting,
Anomaly Detection, ML/DL.

##  I Introduction

The growth of wireless IoT devices is increasing the demand for radio
frequency (RF) spectrum [1]. Nowadays, while some RF bands are congested (as
the ISM band, used by Wi-Fi, Bluetooth, and other standards), others are
under-occupied. Consequently, interference occurring in overcrowded RF bands
degrade service quality due to the necessity of re-transmissions and the
reduction of effective data rates. This situation promotes crowdsensing
spectrum sensors as one of the most attractive and valuable solution for
integrated sensing and communication (ISAC). Crowdsensing RF monitoring
platforms, like ElectroSense [2], sense the RF spectrum occupancy and allow
Cognitive Radio Networks (CRNs) to assign under-occupied bands to particular
devices, balancing and optimizing the RF spectrum usage. Furthermore, these
platforms are also critical when (i) intercepting illegal communications
occurring in licensed bands; (ii) detecting cyberattacks, jamming or
interfering critical transmissions; or (iii) classifying transmission
standards and technologies.

All these previous tasks highly rely on sensed RF spectrum data, a critical
asset that must be protected to ensure trusted services. This protection is a
challenging task, since Primary User Emulation Attack (PUEA) and Spectrum
Sensing Data Falsification Attack (SSDF) are two of the most well-known and
impactful attack families affecting spectrum sensing [3]. Moreover, this
situation becomes even more challenging in the crowdsensing paradigm, where
resource-constrained spectrum sensors show well-known and exploitable
vulnerabilities, apart from being easily manipulated by users with physical
access to them [4].

Behavioral fingerprinting turned out to be one of the most promising
approaches to detect cyberattacks [5]. In such a context, heterogeneous data
sources, like hardware events, system calls, or resource usage, can be
monitored to create precise device behavioral fingerprints [6]. These
fingerprints are evaluated to detect anomalies by different techniques, being
Machine and Deep Learning (ML and DL) the most promising approaches nowadays.
However, the following main open challenges appear when combining
fingerprinting and ML/DL techniques with spectrum sensing: (i) there is no
formal definition of the behavior and impact of SSDF attacks affecting the
integrity of spectrum data, (ii) there is no solution measuring the
performance and suitability of intelligent behavioral fingerprinting in
resource-constrained spectrum sensors, (iii) data sources and events precisely
modeling the normal and under attack behavior of spectrum sensors have not
been studied and specified, and (iv) there is no analysis of the detection
performance and scalability of ML and DL-based solutions considering the
behavior of groups of similar and different spectrum sensors.

To improve these challenges, the main contributions of this work include:

  * •

The definition and deployment of seven novel SSDF attacks (Repeat, Mimic,
Confusion, Noise, Spoof, Freeze, and Delay) affecting spectrum sensors. These
attacks are categorized into three main families focused on: (i) simulating
non-existing communications, (ii) hiding cyberattacks or illegal
transmissions, and (iii) performing both actions in parallel.

  * •

The design and complete implementation of CyberSpec, an ML/DL-oriented
framework that uses intelligent behavioral fingerprinting to detect anomalies
in resource-constrained spectrum sensors affected by SSDF attacks. Events
belonging to the device virtual memory, file system, CPU, network, scheduler,
device drivers and random numbers generation are monitored periodically to
create behavior fingerprints.

  * •

The deployment of CyberSpec in a real IoT-based crowdsensing RF monitoring
platform called ElectroSense [2]. Nine Raspberry Pis 3 and 4, acting as
sensors, were infected with different configurations of the seven SSDF attacks
to later measure the CyberSpec detection performance and time.

  * •

The analysis of a pool of experiments to detect normal and under-attack
behaviors using ML/DL models per (i) individual sensors, (ii) families of
sensors excluding a different number of devices from training, and (ii)
combinations of families. The obtained results show that five of the seven
SSDF attacks are almost perfectly detected in the three experiments. Device-
type and global ML/DL models provide similar detection performance to
individual models. Finally, when the 15%, 33%, and 50% of devices are excluded
from training, device-type models perform acceptably for most excluded devices
(80-70% TPR and 100% TNR for five attacks).

The remainder of this article is organized as follows. Section II reviews
attacks affecting spectrum data and behavior fingerprinting to detect
different cyberattacks. While Section III introduces the basics of spectrum
sensing, Section IV presents seven SSDF attacks affecting sensors. Section V
introduces CyberSpec, an ML/DL-oriented framework detecting anomalies. Section
VI outlines the deployment and performance of CyberSpec in ElectroSense.
Finally, Section VII draws conclusions and next steps.

##  II Related Work

Based on the review of existing cyberattacks affecting spectrum sensing,
solutions using device behavioral fingerprinting to detect cyberattacks are
analyzed.

###  II-A Cyberattacks Affecting Spectrum Sensing

Spectrum sensing is recognized as the basic functionality of Cognitive Radio
Networks (CRNs), and it is vulnerable to different cyberattacks. In [7], a
series of malicious behaviors are categorized into (i) misbehaving, (ii)
selfish, (iii) cheating, or (iv) malicious. Misbehaving consists of breaking
the rules established by the network or spectrum sensing platform. Selfish
behavior is to keep the network resources for the attacker’s benefit. A
cheating behavior occurs when the attacker provides fake spectrum information
to increase his/her quality of service (QoS). Finally, malicious behavior is
when the attacker targets the spectrum sensing to degrade the other nodes QoS
and the network efficiency [8].

Some of these malicious behaviors can be shown by different cyberattacks
affecting the sensing phase of spectrum monitors. This is the case of Primary
User Emulation Attack (PUEA) and Spectrum Sensing Data Falsification (SSDF)
[8]. PUEA [9] simulate transmissions of primary users (PU) over licensed
spectrum bands. On the other hand, SSDF attacks [10] send fake sensing data to
the sensing platform and combine misbehaving, cheating, and selfish behaviors.
Apart from these two attacks, others can be launched as a result of PUEA
and/or SSDF [3]. Denial-of-Service (DoS) attacks are one of them, where the
malicious spectrum sensor emulates a transmission from a PU to force secondary
users (SU) to vacate the spectrum. The DoS attack results in degrading the QoS
of SU [11].

Despite the progress done in the literature regarding SSDF attacks affecting
RF monitoring platforms, there is a lack of work detailing particular
behaviors of different SSDF attacks and their impacts on the spectrum data.

###  II-B Cyberattack Detection Through Behavioral Fingerprinting

The number of existing works applying behavioral fingerprinting in IoT devices
from the host perspective (as this work does) is reduced, but one of them is
HADES-IoT [12]. HADES-IoT is a host-based anomaly detection system for
different Linux-based IoT devices that creates white lists of benign system
calls. A 100% effectiveness is reported when evaluating the system with
different IoT malware affecting seven different IoT devices. Also, in the
field of fingerprinting and IoT, the authors of [13] propose a layered and
adaptive AD solution for IoT that uses Hierarchical Edge Computing (HEC) and
three DL models with different levels of complexity. IoT devices, edge
servers, and cloud architectures are the layers of the HEC where the models
are deployed and evaluated depending on contextual information. A pool of
experiments demonstrates a reduced detection delay while maintaining
acceptable detection accuracy. Finally, DAIMD [14] defines a hybrid scheme
that monitors memory, network, virtual file system, process, and system calls
of devices to detect both well-known and zero-day attacks. A convolution
neural network (CNN) model is trained to classify samples into benign and
malicious ones successfully.

System calls, resource usage, or Hardware Performance Counters (HPC) are
widely used to create behavioral fingerprints and detect cyberattacks
affecting different devices. From the system calls perspective, VizMal [15]
detects Android malware by creating color boxes that represent software
execution time windows. The color of each box refers to the maliciousness
level, and the size is the number of system calls executed during the time
window. A Long Short-Term Memory (LSTM) Neural Network trained with samples
labeled as malware and non-malware provides promising detection results.
VMGuard [16] is another ML-oriented security system that uses system calls to
detect malware affecting Virtual Machines (VM) in cloud scenarios. VMGuard
monitors the system calls of VMs to create a ‘Bag of n-grams (BonG)’
integrated with the Term Frequency-Inverse Document Frequency (TF-IDF) method.
During the evaluation process, Random Forest classifies different attacks
successfully.

TABLE I: Comparison of Solutions Using Behavioral Fingerprinting to Detect Cyberattacks Solution |  Year |  Scenario |  Data Source |  Attack |  Technique |  Performance
---|---|---|---|---|---|---
HADES-IoT [12] |  2019 |  IoT Devices (Linux) |  Syscalls |  IoT Malware |  White List |  1 Accuracy
DAIMD [14] |  2020 |  IoT Devices |  Memory, Network, File System, Processes, and Syscalls |  IoT Malware |  CNN |  0.998 Accuracy
VizMal [15] |  2020 |  Smartphone (Android) |  Syscalls |  Android Malware |  LSTM |  0.098 FPR
VMGuard [16] |  2020 |  Cloud Architectures (VMs) |  Syscalls |  Malware (UNM Dataset) |  Random Forest |  0.93 - 0.99 Accuracy
Ravichandiran et al. [17] |  2018 |  Cloud Architecture (Micro-services) |  Resource Usage (CPU) |  DoS |  Statistical Models |  1 Accuracy
RADS [18] |  2018 |  Cloud Data Center (VMs) |  Resource Usage (CPU) |  DDoS and Cryptominers |  One Class Classification |  0.90 - 0.95 Accuracy
Karl et al. [19] |  2018 |  Embedded Systems |  HPCs |  Malicious Intrusions |  LSTM and HMM |  0.98 - 1 Accuracy
CyberSpec (this work) |  2021 |  IoT Spectrum Sensors (Raspberry Pi) |  Linux kernel events |  SSDF Attacks |  Autoencoder |  0.60 - 1 TPR, 0.90 - 0.98 TNR

Resource usage is another well-known and widely used data source to create
behavioral fingerprints. In this context, the solution presented in [17]
detects Denial-of-Service (DoS) attacks in cloud scenarios by considering CPU
usage. CPU usage statistics of micro-services running on cloud architectures
are monitored to later detect DoS attacks using auto-regressive statistical
models. RADS [18] is another system that monitors resource usage to detect
DDoS attacks in cloud data centers. RADS uses one class classification
algorithm and time series analysis to obtain 90-95% accuracy with a low false-
positive rate of 0-3% when detecting anomalies produced by DDoS and
cryptomining attacks. Finally, HPCs are considered in [19], where authors
propose an ML/DL-oriented solution that detects deviations on normal behaviors
produced by malware affecting embedded systems with a repetitive
functionality. The validation reports an accuracy higher than 95% with Hidden
Markov Models (HMM) and LSTM Neural Networks.

TABLE I compares the most representative aspects of related work. In
conclusion of this analysis, most of the solutions are designed for powerful
devices, being useless for resource-constrained ones due to the monitored data
sources, used techniques and attacks. Despite HADES-IoT and DAIMD work for
IoT, they do not evaluate their suitability or performance when detecting SSDF
attacks in spectrum sensors (having a particular functionality that affects
the device behavior). Finally, none of the analyzed works studied the
detection performance of ML/DL models combining behaviors of different types
of devices. Due to the previous facts, more work aligned with SSDF attacks
affecting RF sensing platforms and ML/DL-based detection frameworks is needed.

##  III Crowdsensing Spectrum Sensors & Cybersecurity Issues

Crowdsensing spectrum sensors are resource-constrained devices, such as a
Raspberry Pi, equipped with Software-defined Radio (SDR) kits. Typically,
spectrum sensors implement a process in charge of (i) dividing the spectrum
into fixed-size segments, and (ii) scanning cyclically the segments composing
the whole RF spectrum, from the lower frequency supported by the sensor to the
higher. This RF monitoring process enables the collection of different data
types per segment, such as Power Spectrum Data (PSD), indicating the spectrum
occupancy of each particular segment. After that, the collected data is sent
periodically to a central platform, in charge of processing and analyzing it
to provide services in charge of (i) optimizing RF occupancy, (i) intercepting
illegal communications over licensed bands, and (iii) detecting cyberattacks
jamming, interfering legit communications.

The trustworthiness of the previous services depends on the integrity of PSD
collected by IoT spectrum sensors. However, the nature of crowdsensing RF
monitoring platforms, relying on resource-constrained spectrum sensors
vulnerable to cyberattacks and manipulations of dishonest users, makes
spectrum sensors vulnerable to SSDF attacks. In such a context, and as
explained in Section II, there is a literature gap in terms of defining and
implementing SSDF attacks having different behaviors and impacts on IoT
spectrum sensors.

##  IV SSDF Attacks

Seven novel SSDF attacks showing differences in terms of PSD impact are
presented in this section. The proposed SSDF attacks are categorized into
three main families: Transmission Simulation, Transmission Hiding, and Hybrid.
Transmission Simulation attacks modify the PSD of spectrum segments to
simulate non-existing wireless transmissions, since the goal of these attacks
is to keep spectrum segments (those where non-existing transmissions are
simulated) under-occupied to be used by the attacker. Secondly, they overload
the remainder spectrum segments (where real transmissions are assigned) to
increase interference and downgrade the Quality-of-Service (QoS). Transmission
Hiding attacks are focused on disrupting services that detect cyberattacks or
unauthorized transmission by hiding them. Finally, Hybrid attacks can cover
the previous two families, simulating fictitious transmissions, hiding illegal
ones, or doing both in parallel. Figure 1 shows the attacks belonging to each
family.

Figure 1: Taxonomy of Proposed SSDF Attacks

###  IV-A Hybrid SSDF Attacks

Attacks belonging to this family can be used either to simulate transmissions
or hide them. The family is composed of the following three attacks: Repeat,
Mimic, and Confusion.

Repeat attacks copy in a particular moment of time the PSD of a selected
spectrum segment and continuously replicates it in the segments targeted by
the attack. More in detail and as can be seen in Figure 2, first, the attacker
selects the RF segment that wants to be replicated (Source_seg) and the
spectrum segments whose PSD are going to be manipulated (SegA). Depending on
the PSD values of Source_seg, the attack will hide or simulate transmissions.
After that, it creates a file (File) to save PSD values of Source_seg. At this
point, the RF spectrum is continuously scanned, as indicated in Section III.
During the first scanning of the RF spectrum, the PSD values of Source_seg are
stored in File (only once). In the next RF scanning cycles, the attack
modifies the PSD values of the SegA with the File content.

Figure 2: Diagram Flow of Repeat Attacks

Mimic attacks are an evolution of Repeat, being the creation of new PSD copies
per RF spectrum the main difference of them. In particular, as seen in Figure
3, this attack defines two sets of spectrum segments (SegS are the segments
whose PSDs are going to be replicated, and SegA the replaced segments) and
creates one empty file (FileS) per SegS. After that, the scanning process
starts, and for each RF spectrum cycle the attack stores the PSD values of
SegS in FileS. Once the files contain the PSD values of SegS, the attack
substitutes the PSD of the SegA with the files content. The number of RF
segments of SegS and SegA must be equal, and depending on the SegA occupancy
the attack simulates or hides transmissions.

Figure 3: Diagram Flow of Mimic Attacks

Confusion attacks pretend to exchange the occupancy levels of two or more RF
segments. The attack can be used to hide a transmission, to simulate an non-
existing one, or for both at the same time, creating confusion. Figure 4 shows
how the attack starts defining the sets of spectrum segments that are going to
be exchanged (SegX and SegY), and creating two files (FileX and FileY) with
PSD values. After that, the monitoring process initiates the sequential RF
scanning. When the scanning process senses the occupancy of a segment
belonging to the SegX, its PSD values are stored in FileX, and replaced with
the content of FileY. Similarly, when the monitor senses an RF segment
belonging to SegY, it saves its PSD in FileY and replaces the segment
occupancy with FileX content.

Figure 4: Diagram Flow of Confusion Attacks

###  IV-B Transmission Simulation SSDF Attacks

Attacks of this family are in charge of modifying the RF spectrum occupancy
data to simulate fake transmissions. This family is composed of two attacks:
Noise and Spoof.

Noise attacks focus on adding random noise to the occupancy level of a set of
RF spectrum segments affected by the attack. Figure 5 shows the life-cycle of
this attack, where the attacker starts defining the spectrum segments (SegA)
that will be attacked and the intensity of the noise. After that, the RF
monitoring process starts. When those segments affected are scanned, the
attack generates random numbers and adds them to their PSD values. This
process is periodically repeated in each RF scanning cycle for all segments
affected.

Figure 5: Diagram Flow of Noise Attacks

Spoof attacks are an evolution of mimic attacks. The main difference is that
spoof adds random noise to the occupancy level of source spectrum segments to
replicate a transmission but adding differences to complicate its detection.
As seen in Figure 6, the attacker defines the sets of source and attacked
spectrum segments (SegS and SegA), creates a file per spectrum segment
belonging to SegS (FileS), and defines the intensity of random noise. After
that, the RF monitoring process starts. If the current segment belongs to
SegS, one random value per PSD is generated, added to the PSD of the current
segment, and stored in FileS. When the segment is within the set of the
attacked segments (SegA), the occupancy of the current segment is replaced
with the content of FileS. This process is repeated for the whole RF spectrum
and across time.

Figure 6: Diagram Flow of Spoof Attacks

###  IV-C Transmission Hiding SSDF Attacks

Freeze and Delay are the attacks belonging to this family and pretend to hide
illegal or unauthorized transmissions.

Freeze attacks copy the PSD values of one or more spectrum segments in a given
moment and replicate them across the time. This type of attack hides any
transmission starting after the screenshot generation and substitution. As
seen in Figure 7, the attacker begins defining the spectrum segments affected
by the attack (SegA) and creates an empty file, where the PSD values will be
stored. After that, the cyclic RF scanning starts. The first time (first scan)
the monitor senses the occupancy of a segment belonging to SegA, its PSD
values are stored in FileA. In successive iterations of the scanning process,
when the monitor senses a segment belonging to SegA, its PSD values will be
replaced with the PSDs previously saved in FileA for that particular segment.

Figure 7: Diagram Flow of Freeze Attacks

Delay attacks are an evolution of Freeze, and their main goal is to send the
RF monitoring platform obsolete or non-updated PSDs of affected spectrum
segments. The main difference between Delay and Freeze is that Freeze always
sends the same occupancy level for each affected segment, while Delay keeps a
sliding time window to send different outdated PSD. In this sense, Figure 8
shows that the attack starts defining the set of affected spectrum segments
(SegA), the time of delay to provide the segments occupancy (DelA), and
creates an empty file per affected segment (FileA). After that, the RF
spectrum scanning starts. If the sensed spectrum segments belong to the set of
SegA, its PSD values are stored in FileA. Later, it is checked if the time
window of delay is over. If so, the segment PSD values are replaced with the
oldest content of FileA, and the content is deleted from the file. In
contrast, when DelA is not over, the attack stores the segment occupancy
values in FileA, but the segment occupancy is not altered. It happens when the
attack starts and until the selected delay is reached.

Figure 8: Diagram Flow of Delay Attacks

##  V CyberSpec Framework

The CyberSpec framework considers device behavioral fingerprinting to detect
anomalies produced by SSDF attacks affecting resource-constrained spectrum
monitors. For that, the following two modules compose the flexible framework:

  * •

Behavior Fingerprinting. Monitor the internal behavior of the resource-
constrained spectrum sensor from different data sources to create behavioral
fingerprints.

  * •

Cyberattacks Detection. Train ML and DL algorithms with the device behavior
fingerprints. Moreover, it evaluates the trained models with the real-time
behavior of the sensors to detect anomalies produced by cyberattacks.

Figure 9 shows the two modules of the proposed framework as well as their
components, which are explained in this section. CyberSpec is fully deployed
in a distributed way, where the spectrum sensor hosts the Behavior
Fingerprinting module, and the Cyberattacks Detection is deployed on a server
(but it is suitable to be deployed in the sensors).

Figure 9: Design of the CyberSpec Framework

###  V-A Behavioral Fingerprinting

This module monitors the behavior of resource-constrained spectrum sensors in
periodic and configurable time windows to create behavior fingerprints that
are sent to the Cyberattack Detection module. To achieve this functionality,
the Data Acquisition component monitors all events shown in TABLE II (step 1
in Figure 9). The criteria to select the event families and events relies on
covering the most diverse and relevant hardware and software characteristics
of resource-constrained spectrum sensors, such as Raspberry Pi models. These
events are monitored periodically using configurable time windows (details are
provided in Section VI). In particular, the Perf Linux command is used for
monitoring purposes. Once the events are measured, their values are saved in a
behavioral vector containing a timestamp. Finally, the Communication component
sends the behavioral vector to the Cyberattacks Detection module (step 2 in
Figure 9).

###  V-B Cyberattacks Detection

The Cyberattacks Detection module hosts two processes. The first is performed
offline and consists of training unsupervised ML/DL models with the normal
behavior of the spectrum sensor. Once it is done, an online process
periodically evaluates the real-time behavior of the sensors with the trained
models to detect anomalies produced by cyberattacks.

The offline process is executed before the online one and it is composed of
the following four components: Dataset Generation, Data Curation, Algorithm
Selection, and Model Training. The Dataset Generation periodically receives
data vectors modeling the normal (benign) behavior of each resource-
constrained spectrum sensor and creates a dataset for each device. The dataset
creation task is configurable, but around ten days are required to build a
stable behavioral fingerprint. It is critical to ensure that no attacks affect
the device during this time period and contextual noise is minimized. Once the
dataset is created, the Data Curation component performs several tasks. The
first is to remove noisy vectors, features with constant values, features
providing no relevant information, highly correlated features, and features
with different data distributions across the time and between sensors. After
performing these tasks, the resulting features are shown in TABLE II in blue.
Finally, other important tasks of the Data Curation process are i) splitting
the datasets into training (72%), validation (18%) and testing (10%), and ii)
normalizing the values of the features. Once the final list of features is
decided, the Algorithm Selection component selects several unsupervised ML/DL
algorithms (explained in Section VI) to be trained by the framework offline
process. After that, the Model Training component receives the selected
algorithms and feeds them with the training set to build the models. Finally,
the performance of each model is evaluated with the validation set, and the
models obtaining the best scores are sent to the online process (cf. step 3 in
Figure 9).

TABLE II: Features, Categorized into Event Families, Used by CyberSpec to Detect Attacks on IoT Spectrum Sensors. The Final Selected Features are Shown in Blue. Family |  |  Features |
---|---|---|---
Network |  tcp:tcp_destroy_sock tcp:tcp_probe udp:udp_fail_queue_rcv_skb net:net_dev_queue |  net:net_dev_xmit qdisc:qdisc_dequeue skb:consume_skb skb:kfree_skb |  skb:skb_copy_datagram_iovec sock:inet_sock_set_state fib:fib_table_lookup
Virtual Memory |  vwriteback:global_dirty_state writeback:sb_clear_inode_writeback writeback:wbc_writepage writeback:writeback_dirty_inode writeback:writeback_dirty_inode_enqueue writeback:writeback_dirty_page writeback:writeback_mark_inode_dirty |  writeback:writeback_pages_written writeback:writeback_single_inode writeback:writeback_write_inode writeback:writeback_written kmem:kfree kmem:kmalloc kmem:kmem_cache_alloc |  kmem:kmem_cache_free kmem:mm_page_alloc kmem:mm_page_alloc_zone_locked kmem:mm_page_free kmem:mm_page_pcpu_drain page-faults pagemap:mm_lru_insertion
File Systems |  jbd2:jbd2_handle_start jbd2:jbd2_start_commit block:block_bio_backmerge block:block_bio_remap |  block:block_dirty_buffer block:block_getrq block:block_touch_buffer block:block_unplug |  cachefiles:cachefiles_create cachefiles:cachefiles_lookup cachefiles:cachefiles_mark_active filemap:mm_filemap_add_to_page_cache
Scheduler |  sched:sched_process_exec sched:sched_process_free sched:sched_process_wait sched:sched_switch |  signal:signal_deliver signal:signal_generate task:task_newtask cpu-migrations |  cs alarmtimer:alarmtimer_fired alarmtimer:alarmtimer_start
CPU |  clk:clk_set_rate rpm:rpm_resume |  rpm:rpm_suspend |  ipi:ipi_raise
Device Drivers |  irq:irq_handler_entry mmc:mmc_request_start |  preemptirq:irq_enable gpio:gpio_value |  dma_fence:dma_fence_init
Random Numbers |  random:get_random_bytes |  random:mix_pool_bytes_nolock |  random:urandom_read

The online process is periodically executed to detect cyberattacks affecting
the behavior of resource-constrained spectrum sensors. With that goal in mind,
when the Model Evaluation component has the models it evaluates the received
periodic behavior vectors (step 4 in Figure 9). Each evaluation predicts if
the current behavior vector is abnormal, which means that the spectrum sensor
is infected.

##  VI Validation Scenario & Experiments

ElectroSense [2] is the crowdsensing RF spectrum monitoring platform selected
to validate the CyberSpec framework and to measure its performance, when
detecting anomalies produced by the SSDF attacks of Section IV.

As validation scenario, this work has deployed a set of Raspberry Pis equipped
with SDR kits and the ElectroSense software sensing the RF spectrum. Sensors
monitor the PSD values of frequency bands segments ranging from 20 MHz to 1.6
GHz in blocks of 2.4MHz. This process takes about 50 s. The following
Raspberry Pi acting as ElectroSense sensors have been considered:

  * •

Six Raspberry Pis 3 Model B with Quad Core 1.2 GHz Broadcom BCM2837 64-bit
CPU, and 1 GB of RAM.

  * •

Three Raspberry Pis 4 with Quad Core 1.5 GHz Broadcom BCM2711 64-bit CPU, and
2 GB of SDRAM.

These nine spectrum sensors are connected to the Internet through the next
four local area networks, which are deployed in different geographical
locations.

  * •

LAN_1: One Raspberry Pi 3.

  * •

LAN_2: One Raspberry Pi 4.

  * •

LAN_3: Two Raspberry Pis 3 & one Raspberry Pi 4.

  * •

LAN_4: Three Raspberry Pis 3 & one Raspberry Pi 4.

As indicated in Section V, each sensor implements the Behavior Fingerprinting
module of CyberSpec to monitor and send its behavior. The computational and
time costs of the module are shown in TABLE III. In particular, the module
monitors the sensor behavior in time windows of 50 s and requires 6.8 s to
pre-process the events. 50 s is the minimum monitoring time required to
capture all SSDF attacks affecting any RF segment band since it is the time
needed by the sensors to scan the spectrum. In addition and for simplicity,
the Cyberattack Detection module, in charge of creating the datasets, training
and evaluating ML/DL models, is deployed on a trusted server.

TABLE III: Behavior Fingerprinting Module Cost for Raspberry Pis 3 and 4 CPU | Memory | Storage | Monitoring | Processing
---|---|---|---|---
0.5-2% 1 Core with 8% peaks | 900 kB | 7.8 kB | 50 s | 6.8 s

In such a scenario, the normal behavior of each Raspberry Pi was monitored for
eight days, and a dataset per device was created. After that, several
configurations of the seven attacks defined in Section IV were sequentially
executed in each Raspberry Pi for two hours, creating one dataset per attack
configuration and device. Attacks configurations differ in the affected
spectrum bandwidth to simulate and hide heterogeneous wireless communications
standards. In particular, the selected configurations of attacks affect
transmissions technologies such as WiFi (from 20 MHz to 160 MHz depending on
the 802.11 version), Bluetooth (2 MHz), 3G (200 kHz), and SOS (20 kHz). At
this point, it is worth mentioning that each Raspberry Pi has been infected
with only one attack at the same time. As a summary, the following datasets
were created per sensor.

  * •

One dataset with normal behavior (192 h of monitoring).

  * •

Forty-two datasets with malicious behavior (2h of monitoring each one). In
detail, six datasets with a different bandwidth configuration (20 kHz, 200
kHz, 2 MHz, 20 MHz, 80 MHz, and 160 MHz) per attack (Noise, Spoof, Repeat,
Confusion, Mimic, Freeze, and Delay).

TABLE IV: Anomaly Detection Algorithms and Hyper-parameters Tested & Selected Algorithm |  Hyperparameters tested |  Hyperparameters selected
---|---|---
Autoencoder |  $layers\in[1,3],neurons\\_layer\in[10,60]$ |  $layers=1,neurons=40$
LOF |  $n\\_neighbors\in[3,25]$ |  $n\\_neighbors=15$
OC-SVM |  $gamma\in[0.001,100]$ $kernel\in\\{^{\prime}rbf^{\prime},^{\prime}linear^{\prime},^{\prime}sigmoid^{\prime},^{\prime}poly^{\prime}\\}$ $degree\in[2,5](only\ poly\ kernel)$ |  $kernel=^{\prime}rbf^{\prime},gamma=0.001$
IF |  $Number\\_of\\_trees\in[50,1000]$ |  $Number\\_of\\_trees=150$
COPOD |  - |  -

###  VI-A Experiments

This section proposes the next three experiments to evaluate the performance
of CyberSpec when detecting the previous configurations of SSDF attacks (see
Section IV).

  * •

Exp_1: Performance analysis of individual models per Raspberry Pi.

  * •

Exp_2: Performance analysis of models per type of device, with different
amounts of Raspberry Pis 3 used during training and evaluation.

  * •

Exp_3: Performance analysis of global models combining the behavior of all
Raspberry Pis 3 and 4.

To perform the three experiments, a common methodology has been followed. It
starts choosing the datasets modeling the normal behavior of the selected
Raspberry Pis (different for each experiment). After that, the pipeline
indicated in Section V is followed to prepare the data for the ML/DL phase.
Next, a set of unsupervised anomaly detection algorithms are selected. In
particular, Autoencoder, Isolation Forest (IF), Copula-Based Outlier Detection
(COPOD), Local Outlier Factor (LOF), and One-Class Support Vector Machine (OC-
SVM) with a diverse set of hyper-parameters are considered at this stage (see
TABLE IV). The next step is to adjust the threshold to detect anomalies by
using a well-known statistical approach. In this sense, the Interquartile Rule
[20] is applied to find outliers. This technique calculates the quartiles and
the interquartile range (IQR) in the scores of the training instances and
defines two thresholds calculated as Q1 - 1.5*IQR and Q3 + 1.5*IQR. Then, any
score below or over these values is treated as an anomaly. In the case of the
Autoencoder, the Mean Squared Error (MSE) of the input reconstruction
(difference between model output and input in each feature) was used as a
score, and in the case of LOF, IF, COPOD, and OC-SVM, the raw algorithm
outputs were used. After that, a final selection of the hyper-parameters and
the threshold is made using the validation dataset (see TABLE IV). At this
point, it is important to mention that the hyper-parameters selection for the
three experiments has been made using the configuration of Exp_3, due to the
minimum impact of having individuals per experiment and the number of models
(more than 100 trained across all experiments). Finally, the detection
performance of each algorithm is evaluated with the normal and the behavior
under-attack of each device (in each experiment, different data setups are
used for training and testing). The metrics used to compare the detection
performance are the TNR (True Negative Rate) and TPR (True Positive Rate). TNR
indicates the number of non-anomalies found in the normal or benign behavior,
while TPR is used to evaluate malicious behavior and provides the number of
detected anomalies.

####  VI-A1 Exp_1: Individual models

This experiment employs the nine Raspberry Pis (six Raspberry Pis 3 and three
Raspberry Pis 4) and their training datasets (modeling the 72% of their normal
behaviors) to train the five selected algorithms (Autoencoder, IF, LOF, COPOD,
and OC-SVM) per device (45 ML/DL models in total). After that, the individual
models of each device have been evaluated with (i) the testing dataset
(modeling the 10% of normal behavior) of that device, and (ii) the 42 datasets
modeling the different configurations of the SSDF attacks affecting that
device.

For each algorithm, Figure 10 shows the TNR mean and standard deviation of
each individual model algorithm when they are evaluated with normal behavior.
As can be seen, IF is the model obtaining the best average performance for all
devices (95% TNR). At this point, it is important to mention that since the
evaluated datasets do not contain anomalies generated by SSDF attacks, the TPR
metric is not considered when detecting normal behavior.

Figure 10: Average TNR Performance of Individual ML/DL Models When Detecting
Normal Behavior

In addition, to choose the algorithm providing the best performance is needed
to analyze the performance when detecting SSDF attacks. In this sense, Figure
13 shows the mean TPR (for the nine devices) of the five individual ML/DL
models.

Figure 11: Average TPR Performance of Individual ML/DL Models When Detecting
SSDF Attacks

As can be seen in Figure 13, in general, Autoencoder, LOF, and OC-SVM are the
models providing the best performance (100% TPR) for all configurations of
noise, spoof, confusion, mimic and delay). Repeat and Delay attacks deserve
special consideration since they are not adequately detected until the
affected bandwidth reaches 80 and 160 MHz (only by Autoencoder). It is due to
the selected features do not monitor events produced by file read operations
(internal behavior of these two attacks, as shown in Section IV). In this
sense, when the affected bandwidth is sufficient (80 and 160 MHz), the attack
impacts the rest of the monitored events, and these variations are slightly
detected.

####  VI-A2 Exp_2: Device-type models

Since individual models per device might be cumbersome to maintain in dynamic
platforms, such as ElectroSense with a significant number of new devices per
month, this experiment evaluates the performance of models per device family
or type (one model for Raspberry Pis 3 and another for Raspberry Pis 4). In
particular, it analyzes the fact of having device-type models (i) trained and
evaluated with the normal behavior of all devices, and (ii) trained with
different numbers of devices and evaluated with the rest.

Regarding the first setup, the training datasets (modeling the 72% of the
normal behavior) of the 100% of Raspberry Pis 3 (six devices) are combined to
feed the five selected algorithms per device family. After that, the testing
datasets of the six Raspberry Pis 3 (modeling the 10% of their normal
behavior) are concatenated and evaluated. This process is repeated for the
three Raspberry Pis 4. Figure 12 shows the TNR for each algorithm and device
type (light green for Raspberry Pis 3 and blue for Raspberry Pis 4). As can be
seen, for both families of devices, the performance is similar (being
Autoencoder the one obtaining greater differences for both families with 0.85%
and 0.93% TNR), showing that the proposed features and framework are suitable
for Raspberry Pis 3 and 4. In addition, comparing these results with the
obtained by the individual models of the previous experiment, the differences
are minimal as well.

Figure 12: TNR Performance of Device-type ML/DL Models for Raspberry Pi 3 and
4 Recognizing Normal Behavior

Regarding the detection of attacks, Figure 11 shows the mean TPR of the
Raspberry Pi 3 & 4 family models when detecting anomalies contained in the 42
malicious datasets per device (252 datasets of Raspberry Pis 3, and 126 of
Raspberry Pis 4). These results are similar to those shown in Figure 13. In
particular, all configurations of noise, spoof, confusion, mimic, and delay
are detected by Autoencoder, LOF, and OC-SVM. These results show a promising
path to have models per family of devices, reducing the number of models in
the platform and, therefore, the maintenance cost.

(a)  (b)

Figure 13: Average TPR Performance of device-type ML/DL Models When Detecting
SSDF Attacks

Once the suitability of device type models is demonstrated, the second setup
of this experiment evaluates the scalability of the proposed solution when new
devices appear in the platform (unseen during the training phase). In this
sense, different tests have been performed excluding the 15%, 33%, 50%, 66%,
and 85% of Raspberry Pis 3 (1, 2, 3, 4, and 5 devices, respectively) from
training. For each test, all combinations of devices have been performed to
show robust results. In other words, and as an example, when one device is
excluded from the training phase, all combinations (6 in total) have been
considered to train the models. At this point, it is important to mention that
only Raspberry Pis 3 have been selected due to the number of available
devices. As in the previous experiment, 72% of the normal behavior of the
selected devices has been used for training and 100% of the normal behavior of
excluded devices for evaluation. Figure 14 shows the TNR mean and standard
deviation of Autoencoder, LOF, and OC-SVM. Only these three algorithms are
shown in the figure do to they are the ones obtaining 100% TPR for all
configurations of noise, spoof, confusion, mimic, and delay (in this
experiment test and the previous ones).

Figure 14: Average TNR Performance of Device-type ML/DL Models Trained with
Different Numbers of Raspberry Pis 3 When Detecting Normal Behavior

As can be seen in Figure 14, regardless of the number of excluded devices, OC-
SVM is the model providing the best averaged TNR. In addition, and as
expected, the higher the number of devices excluded from the training, the
lower the TNR. This is due to the fact of having independent data with
different distributions (also known as non-IID) per device. These different
data distributions could be influenced by external factors such as network
bandwidth, traffic, device temperature, or processes. In any case, OC-SVM
obtains good TNR averages (80%, 75%, and 70%) when 85%, 66%, and 50% of
devices are used only during training. Figure 14 also shows that the OC-SVM
models work very well for the majority of devices (high TNR mean) but bad for
a few of them (high standard deviation). Furthermore, when more than 50% of
devices are excluded from training, the detection performance of OC-SVM and
all models drops to 50%. In conclusion, the scalability results obtained by
OC-SVM are promising, but more Raspberry Pis 3 connected to the same and
different networks are required to confirm the positive trend.

####  VI-A3 Exp_3: Global model

This last experiment consists of training a global model that combines the
normal behavior of all Raspberry Pis 3 and 4. For that, it combines the nine
datasets modeling the 72% of the devices normal behavior, and the samples of
each dataset were balanced for the training process of the five selected
algorithms. Finally, the resulting models were evaluated with both normal
(testing dataset of each device with 10% of samples) and under attack
behaviors of all devices.

Figure 15 shows the TNR of global models (light green bars), models-type for
Raspberry Pi 3 (blue bars) and Raspberry Pis 4 (orange bars), and individual
models for a Raspberry Pi 4 (gray bars) when detecting normal behaviors. As
shown, there are no great differences in the TNR of the three experiments,
indicating that models combining the normal behavior of Raspberry Pis 3 and 4
perform reasonably well when detecting normal behaviors. Regarding the
detection of anomalies, the performance of the five ML/DL algorithms was
evaluated. Still, for the sake of simplicity, TABLE V shows the TPR obtained
by OC-SVM, the one providing the best ratio of TNR/TPR performance. As it can
be seen, spoof, confusion, mimic, and delay are perfectly detected in all
their configurations. The noise attack was perfectly detected almost for all
configurations, excluding the less impactful one (20kHz). Finally, both Repeat
and Freeze were not adequately detected, as in the previous two experiments.
It is important to mention that these results have been calculated by
computing the average mean of the TPR score provided by each Raspberry Pi 3
and 4.

Figure 15: Comparative of the TPR Performance of Different ML/DL Models When Detecting Normal Behavior TABLE V: Average TPR of OC-SVM Global Model When Detecting SSFD Attacks Attack | 20 kHz | 200 kHz | 2 MHz | 80 MHz | 160 MHz
---|---|---|---|---|---
noise | 61% | 99% | 100% | 100% | 100%
spoof | 99% | 100% | 100% | 100% | 100%
repeat | 6% | 29% | 26% | 18% | 26%
confusion | 100% | 100% | 100% | 100% | 100%
mimic | 100% | 100% | 100% | 100% | 100%
freeze | 7% | 24% | 30% | 28% | 32%
delay | 100% | 100% | 100% | 100% | 100%

Finally, TABLE VI shows the time needed by the server (equipped with an Intel
i7-5930K CPU @ 3.50GHz, 3 NVIDIA GTX1080 GPUs and 96 GB RAM) to train and
evaluate each model for the global experiment (worst-case scenario because the
individual and device-type experiments need less data for training). The
testing time is the most important because it affects the detection time of
CyberSpec, which is the sum of the monitoring time (50 s), the time to pre-
process the monitored events and create the features (6.8 s), and the time to
evaluate the features (2.75 s worst-case scenario). In summary, CyberSpec
takes less than 60 s to detect attacks since the model training process can be
done offline, and the communication time between the sensors and the server
where models are evaluated is minimal.

TABLE VI: Time Needed to Train and Test Global Models | Autoencoder | IF | LOF | COPOD | OC-SVM
---|---|---|---|---|---
Train | 97.82 s | 10.99 s | 58.27 s | 6.08 s | 770.39 s
Test | 0.03 s | 0.04 s | 0.02 s | 0.02 s | 0.01 s

##  VII Summary, Conclusions, and Future Work

This work introduced and formally described the behavior of seven Spectrum
Sensing Data Falsification (SSDF) attacks (Repeat, Mimic, Confusion, Noise,
Spoof, Freeze, and Delay) affecting crowdsensing spectrum sensors. The second
contribution was the design and implementation of CyberSpec, an ML/DL-oriented
framework that monitors internal events of IoT devices, such as network,
virtual memory, files system, scheduler, system calls, CPU, device drivers,
and random numbers, to detect anomalies produced by cyberattacks. The
performance of CyberSpec has been evaluated in a real crowdsensing spectrum
monitoring platform called ElectroSense, where nine Raspberry Pis 3 and 4
acting as spectrum sensors were infected with the previous SSDF attacks. Three
experiments with different unsupervised ML/DL models were performed to
evaluate the CyberSpec detection capabilities. Individual ML/DL models per
device, models per family of devices (excluding different % of devices from
training), and global models combining all devices have provided a promising
performance when detecting the normal behavior of six Raspberry Pis 3 and
three Raspberry Pis 4. Five (Noise, Spoof, Confusion, Mimic, and Delay) of the
seven analyzed attacks are almost perfectly detected (100% TPR) by
Autoencoder, LOF, and OC-SVM in the three experiments. The fact of having
models combining normal behaviors of different devices does not reduce the
detection performance significantly. Moreover, looking at the scalability of
device-type models, when the 15%, 33%, and 50% of devices are excluded from
training, they perform relatively well for most excluded devices (80-70% TPR
and 100% TNR for five attacks). Repeat and Freeze are not correctly detected
because CyberSpec does not monitor read file operations, and the behavior of
these attacks relies on that. Finally, the CyberSpec framework needs less than
60 s to detect normal/under-attack behavior while keeping a reduced
computational cost in the Raspberry Pis.

With the goal of giving an answer to the open challenges depicted in Section
I, the main conclusions after performing the experiments and analyzing the
results are (i) the proposed features and individual ML/DL models per device
are appropriate to detect precisely both normal behaviors and anomalies
produced by attacks affecting IoT devices; (ii) global ML/DL models (per
family of devices and combining two families) show a promising detection
performance in scenarios with a reduced number of ElectroSense sensors, but
more experiments are needed to evaluate its scalability when the number of
devices increases; and (iii) additional behavioral events and features are
required to properly detect attacks, such as Repeat and Freeze, having a
slight impact on the internal behavior of the ElectroSense sensors.

Future work will evaluate the performance of privacy-preserving mechanisms.
Solutions like the proposed in this work, where global models are trained with
the behavior of different devices, are not suitable for privacy-preserving
scenarios where federated learning can be a solution to train models while
preserving data sensitiveness. Furthermore, supervised algorithms will be
investigated to classify cyberattacks in different families. Finally, adding
new event families will lead to the improvement of the detection accuracy,
especially in attacks such as Repeat and Freeze.

## Acknowledgment

This work has been partially supported by (a) the Swiss Federal Office for
Defense Procurement (armasuisse) with the CyberSpec (CYD-C-2020003) and
TREASURE (R-3210/047-31) projects and (b) the University of Zürich UZH.

## References

  * [1] K. Riad, T. Huang, and L. Ke.  A Dynamic and Hierarchical Access Control for IoT in Multi-authority Cloud Storage.  Journal of Network and Computer Applications, 160:102633, 2020.
  * [2] S. Rajendran, R. Calvo-Palomino, M. Fuchs, B. Van den Bergh, H. Cordobes, D. Giustiniano, S. Pollin, and V. Lenders.  Electrosense: Open and Big Spectrum Data.  IEEE Communications Magazine, 56(1):210–217, 2018.
  * [3] J. Li, Z. Feng, Z. Feng, and P. Zhang.  A survey of security issues in Cognitive Radio Networks.  China Communications, 12(3):132–150, 2015.
  * [4] F.a Meneghello, M. Calore, D. Zucchetto, M. Polese, and A. Zanella.  IoT: Internet of Threats? A Survey of Practical Security Vulnerabilities in Real IoT Devices.  IEEE Internet of Things Journal, 6(5):8182–8201, 2019.
  * [5] B. Bezawada, M. Bachani, J. Peterson, H. Shirazi, I. Ray, and I. Ray.  Behavioral fingerprinting of iot devices.  In Proceedings of the 2018 Workshop on Attacks and Solutions in Hardware Security, page 41–50, 2018.
  * [6] P. M. Sánchez Sánchez, J. M. Jorquera Valero, A. Huertas Celdrán, G. Bovet, M. Gil Pérez, and G. Martínez Pérez.  A Survey on Device Behavior Fingerprinting: Data Sources, Techniques, Application Scenarios, and Datasets.  IEEE Communications Surveys Tutorials, 23(2):1048–1077, 2021.
  * [7] S. Alrabaee, M. Khasawneh, A. Agarwal, N. Goel, and M. Zaman.  Applications architectures and protocol design issues for cognitive radio networks: a survey.  International Journal of Wireless and Mobile Computing, 7(5):415–427, 2014.
  * [8] M. Khasawneh and A. Agarwal.  A collaborative approach for monitoring nodes behavior during spectrum sensing to mitigate multiple attacks in cognitive radio networks.  Security and Communication Networks, 2017, 2017.
  * [9] C. V. Vivekanand and K. B. Bagan.  Secure Distance Based Improved Leach Routing to Prevent Puea in Cognitive Radio Network.  Wireless Personal Communications, 113(4):1823–1837, 2020.
  * [10] K. Yadav, S. Dhar Roy, and S. Kundu.  Defense against spectrum sensing data falsification attacker in cognitive radio networks.  Wireless Personal Communications, pages 1–14, 2020.
  * [11] V. Zlomislić, K. Fertalj, and V. Sruk.  Denial of service attacks: An overview.  In 2014 9th Iberian Conference on Information Systems and Technologies (CISTI), pages 1–6, 2014.
  * [12] D. Breitenbacher, I. Homoliak, Y. L. Aung, N. O. Tippenhauer, and Y. Elovici.  HADES-IoT: A Practical Host-Based Anomaly Detection System for IoT Devices.  In Proceedings of the 2019 ACM Asia Conference on Computer and Communications Security, pages 479–484, 2019.
  * [13] M. V. Ngo, H. Chaouchi, T. Luo, and T. Q. S. Quek.  Adaptive Anomaly Detection for IoT Data in Hierarchical Edge Computing, 2020.
  * [14] J. Jeon, J. Hyuk Park, and Y. Jeong.  Dynamic Analysis for IoT Malware Detection with Convolution Neural Network Model.  IEEE Access, 8:96899–96911, 2020.
  * [15] A. De Lorenzo, F. Martinelli, E. Medvet, F.o Mercaldo, and A. Santone.  Visualizing the Outcome of Dynamic Analysis of Android Malware with VizMal.  Journal of Information Security and Applications, 50:102423, 2020\.
  * [16] P. Mishra, V. Varadharajan, E. S. Pilli, and U. Tupakula.  VMGuard: A VMI-Based Security Architecture for Intrusion Detection in Cloud Environment.  IEEE Transactions on Cloud Computing, 8(3):957–971, 2020.
  * [17] R. Ravichandiran, H. Bannazadeh, and A. Leon-Garcia.  Anomaly Detection using Resource Behaviour Analysis for Autoscaling systems.  In 2018 4th IEEE Conference on Network Softwarization and Workshops (NetSoft), pages 192–196, 2018.
  * [18] S. Barbhuiya, Z. Papazachos, P. Kilpatrick, and D. S. Nikolopoulos.  RADS: Real-time Anomaly Detection System for Cloud Data Centres, 2018\.
  * [19] K. Ott and R. Mahapatra.  Hardware Performance Counters for Embedded Software Anomaly Detection.  In 2018 IEEE 16th Intl Conf on Dependable, Autonomic and Secure Computing, 16th Intl Conf on Pervasive Intelligence and Computing, pages 528–535, 2018.
  * [20] H. P. Vinutha, B. Poornima, and B. M. Sagar.  Detection of Outliers Using Interquartile Range Technique from Intrusion Dataset.  In Information and Decision Sciences, pages 511–518, 2018.

|  Alberto Huertas Celdrán received the MSc and PhD degrees in Computer
Science from the University of Murcia, Spain. He is currently a postdoctoral
fellow at the Communication Systems Group CSG, Department of Informatics IfI
at the University of Zurich UZH. His scientific interests include medical
cyber-physical systems (MCPS), brain-computer interfaces (BCI), cybersecurity,
data privacy, continuous authentication, semantic technology, context-aware
systems, and computer networks.
---|---
|  Pedro M. Sánchez Sánchez received the MSc degree in computer science from
the University of Murcia, Spain. He is currently pursuing his PhD in computer
science at University of Murcia. His research interests are focused on
continuous authentication, networks, 5G, cybersecurity and the application of
machine learning and deep learning to the previous fields.
---|---
|  Gérôme Bovet is the head of data science for the Swiss Department of
Defense, where he leads a research team and a portfolio of about 30 projects.
His work focuses on Machine and Deep Learning approaches applied to cyber-
defense use cases, with an emphasis on anomaly detection, adversarial and
collaborative learning. He received his PhD in networks and systems from
Telecom ParisTech, France, in 2015.
---|---
|  Gregorio Martinez Pérez is Full Professor in the Department of Information
and Communications Engineering of the University of Murcia, Spain. His
scientific activity is mainly devoted to cybersecurity and networking, also
working on the design and autonomic monitoring of real-time and critical
applications and systems. He is working on different national (14 in the last
decade) and European IST research projects (11 in the last decade) related to
these topics, being Principal Investigator in most of them. He has published
160+ papers in national and international conference proceedings, magazines
and journals.
---|---
|  Burkhard Stiller received the Informatik-Diplom (MSc) degree in Computer
Science and the Dr. rer.- nat. (PhD) degree from the University of Karlsruhe,
Germany, in 1990 and 1994, respectively. He was with the Computer Lab,
University of Cambridge, U.K, ETH Zürich, Switzerland, and the University of
Federal Armed Forces Munich, Germany. Since 2004 he chairs the Communication
Systems Group CSG, Department of Informatics IfI, University of Zürich UZH,
Switzerland as a Full Professor. Besides being a member of the editorial board
of the IEEE Transactions on Network and Service Management, Springer’s Journal
of Network and Systems Management, and the KICS’ Journal of Communications and
Networks, Burkhard is past Editor in-Chief of Elsevier’s Computer Networks
journal. His main research interests are published in well over 300 research
papers and include systems with a fully decentralized control (blockchains,
clouds, peer-to-peer), network and service management (economic management),
Internet-of-Things (security of constrained devices, LoRa), and
telecommunication economics (charging and accounting).
---|---


# Robust Estimation of Causal Heteroscedastic Noise Models

Quang-Duy Tran Corresponding author<EMAIL_ADDRESS>Artificial
Intelligence Institute (A2I2),
Deakin University, Australia Bao Duong22footnotemark: 2 Phuoc
Nguyen22footnotemark: 2 Thin Nguyen22footnotemark: 2

###### Abstract

Distinguishing the cause and effect from bivariate observational data is the
foundational problem that finds applications in many scientific disciplines.
One solution to this problem is assuming that cause and effect are generated
from a structural causal model, enabling identification of the causal
direction after estimating the model in each direction. The heteroscedastic
noise model is a type of structural causal model where the cause can
contribute to both the mean and variance of the noise. Current methods for
estimating heteroscedastic noise models choose the Gaussian likelihood as the
optimization objective which can be suboptimal and unstable when the data has
a non-Gaussian distribution. To address this limitation, we propose a novel
approach to estimating this model with Student’s $t$-distribution, which is
known for its robustness in accounting for sampling variability with smaller
sample sizes and extreme values without significantly altering the overall
distribution shape. This adaptability is beneficial for capturing the
parameters of the noise distribution in heteroscedastic noise models. Our
empirical evaluations demonstrate that our estimators are more robust and
achieve better overall performance across synthetic and real benchmarks.

##  1 Introduction

The aim of the causal discovery is to uncover the underlying causal
relationships of the data. This task is relevant in various scientific
disciplines such as biology, economics, and sociology [17]. One foundational
challenge in causal discovery involves identifying the cause and effect
between two variables $X$ and $Y$. Randomized controlled trials (RCT) are
considered the ideal solution for determining causal relationships,
particularly in medicine [6]. However, RCTs require active intervention in
variables and observation of corresponding feedback, making them resource-
intensive and sometimes ethically impractical. To overcome the limitations of
RCT, studying observational data becomes a more challenging yet necessary
approach for identifying causal relationships.

__

Figure 1: Heteroscedastic noise models estimated with Gaussian distribution
and Student’s $t$-distribution for the likelihood $p\left(Y\mid X\right)$ on
pair 82 of the LS-s dataset. The Gaussian estimation fails to capture the
regions with small sample sizes and low variances, which causes suboptimal
fits in these regions. In contrast, estimating with a more robust Student’s
$t$-distribution allows the model to better capture the distribution of the
noise. Accordingly, our estimation method with $t$-distribution can correctly
retrieve the residual $N_{Y}=\frac{Y-f\left(X\right)}{g\left(X\right)}$ that
is independent of the cause $X$.

With solely observational data, determining the causal direction between $X$
and $Y$ requires prior assumptions of the data-generating process, which is
commonly represented by structural causal models (SCMs) [17]. In general, an
SCM includes a cause variable $X$, an independent noise term $N_{Y}$
($N_{Y}\operatorname{\perp\\!\\!\\!\perp}X$), and an effect variable $Y$
generated from $X$ and $N_{Y}$ via a function $f$. By restricting the
formulation of $f$, the identifiability or the ability to identify the causal
direction from observational data has been intensively studied. The earliest
and the most comprehensively examined model is the branch of additive noise
models (ANMs), where the noise term is added after applying the function $f$:
$Y:=f\left(X\right)+N_{Y}$. This model is proven to be identifiable when $f$
is linear and $N$ has a non-Gaussian distribution [24] or when $f$ is non-
linear [7]. As the cause only affects the mean of the noise, ANMs can be
estimated easily and there exist consistent estimators for these models [14].
Post non-linear noise models (PNLs) are more complex than ANMs where an
additional non-linear invertible function $g$ is applied after the noise
addition: $Y:=g\left(f\left(X\right)+N_{Y}\right)$. Various works have also
been conducted to study the identifiability and estimation of PNL models [31,
32]. Heteroscedastic noise models (HNMs) emerge as generalizations of ANMs. In
HNMs, the cause appears in not only the additive form but also in the
multiplicative form: $Y:=f\left(X\right)+g\left(X\right)N_{Y}$. In addition to
identifiability conditions [12, 26, 9], several methods for predicting the
causal direction between two variables with HNMs [27, 12, 26, 30, 9] have been
proposed.

Current estimators for HNMs use Gaussian likelihood as the optimization
objective [9]. The estimation is achieved by modeling the mean
$\mu\left(X\right)$ and the variance $\sigma^{2}\left(X\right)$ and maximizing
the Gaussian log-likelihood $\log p\left(Y\mid
X\right):=\log\mathcal{N}\left(Y\mid\mu\left(X\right),\sigma^{2}\left(X\right)\right)$.
Although Gaussian distribution is a common choice for the likelihood, this
light-tailed distribution is sensitive to small sample sizes and extreme
values. Figure 1 demonstrates an example where a region has a small sample
size and low variances, the model fitted with Gaussian likelihood cannot
capture the variance correctly. In this study, we propose ROCHE—a RObust
estimation approach for Causal HEteroscedastic noise models with Student’s
$t$-distribution using neural networks. The $t$-distribution is a heavy-tailed
distribution that generalizes the Gaussian distribution with an additional
degree of freedom parameter $\nu$. The degree of freedom $\nu$ controls the
heaviness of the tails which improves the adaptability of the model estimation
without changing the general shape of the distribution significantly. With the
robustness in model estimation, our proposed approach achieves better
performance on most bivariate causal discovery benchmarks with higher
stability compared to models estimated with Gaussian likelihood and other
related approaches.

Contributions. The key contributions of this study can be summarized as
follows:

  1. 1.

We propose a robust estimation approach for identifying the causal direction
of heteroscedastic noise models with Student’s $t$-distribution. This approach
addresses the limitations of using Gaussian likelihood and provides a more
robust alternative for modeling the noise distribution.

  2. 2.

We design a framework that incorporates neural networks to robustly estimate
the $t$-distribution likelihood. We introduce essential constraints that are
specifically tailored for the bivariate causal discovery task, ensuring
accurate and reliable estimation results.

  3. 3.

We demonstrate the effectiveness of our approach by comparing it with related
approaches for both homoscedastic and heteroscedastic noise models. We
evaluate our method on 13 commonly used bivariate causal discovery benchmarks,
showcasing its superior performance and robustness.

##  2 Related Works

Although the bivariate causal discovery or cause-effect inference from
observational data is fundamental and well-defined, this task still attracts a
considerable amount of research attention. Compared to multivariate settings,
the bivariate setting is more limited in identifying information such as
global information about the causal structures. The recent related literature
aiming to infer the cause-effect relationship between two variables can be
categorized into three branches which will be presented below.

#### Methods based on cause-effect asymmetry

The first branch of approaches explores the assumptions of identifiable
structural causal models (SCMs), where the causal models in the anti-causal
directions are proven to be non-existent. Scores for quantifying this
asymmetry in model existence are designed for each type of SCM. Prevalent
scoring choices for determining the causal direction are the log-likelihood
and non-parametric independent test scores such as the Hilbert-Schmidt
Independence Criterion (HSIC) [5] or mutual information between the estimated
residual and the cause. Various approaches in this branch has been extended to
handle multivariate data. In homoscedastic/additive noise models where the
cause only is assumed to contribute only to the means, mean regression methods
are being used in many methods for estimating the models [24, 7, 19, 20, 18,
1]. After the estimation step, the maximum likelihood is employed for
inferring the cause and effect in CAM [1]. Correspondingly, RESIT [20] also
executes the mean estimation, but an additional step of independence testing
is performed subsequently.

For heteroscedastic noise models, many works aiming to quantify the asymmetry
have been introduced. CAREFL [12] assumes the invertibility of the functions
with the Gaussianity of the noises and estimates the models with
autoregressive normalizing flows [8]. The likelihood of the affine
autoregressive flows is then computed on both training and test data to
conclude the cause-effect direction. HECI [30] divides the assumed cause $X$
in the several bins and, in each bin, the model is assumed to have a
homoscedastic noise form. The BIC score is computed in each bin, and the
correct causal direction is the one that will minimize this score. GRCI [26]
regresses the mean and the mean absolute deviation (MAD) with a leave-one-out
cross-validation approach. The residual is computed from the estimated mean
and MAD, and the independence between the residual and the corresponding cause
is assessed by the mutual information. A consistent estimator and a neural
network-based estimator for maximizing the Gaussian log-likelihood are
presented in LOCI [9]. Besides utilizing the log-likelihood as a criterion,
from the mean and the variance parameters of the Gaussian distribution, the
residual can be recovered for the next step of testing the independence with
HSIC. As our proposed approach belongs to this group, ROCHE follows the same
procedure for predicting the causal direction as previous methods.

#### Methods based on independence between cause and mechanism

The second branch of methods relies on the postulate of the independence
between the cause and the mechanism. If the causal direction between $X$ and
$Y$ is $X\rightarrow Y$, the marginal distribution of the cause
$P\left(X\right)$ is independent of the conditional distribution of the effect
given the cause $P\left(Y\mid X\right)$ and not vice versa. IGCI [10]
considers the case with low noise levels and formularizes this independence in
terms of information geometry for distinguishing the cause and effect with
relative entropy distances. CDCI [3] is based on the assumption that the
conditional distribution of the effect given the cause is invariant in shape.
Another interpretation of this independence in the causal mechanism is the sum
of the Kolmogorov complexities [13] of $P_{X}$ and $P_{Y\mid X}$ [11, 15]. The
minimum description length (MDL) is adopted for approximating the Kolmogorov
complexity as this complexity is not computable. QCCD [27] also follows this
interpretation with non-parametric quantile regression and code length
computation for distinguishing the cause and effect.

####

##  3 Preliminaries

###  3.1 Heteroscedastic Noise Models & Causal-Effect Inference

(Heteroscedastic Noise Model [9]). Given a random variable $X$ as the cause
and a random variable $N_{Y}$ as the noise that is independent to $X$
($N_{Y}\operatorname{\perp\\!\\!\\!\perp}X$), a heteroscedastic noise model
or, in different terms, a location-scale noise model generates the effect
variable $Y$ with the following formulation:

(3.1) $Y:=f\left(X\right)+g\left(X\right)N_{Y},$

where $f:\mathcal{X}\rightarrow\mathbb{R}$ and
$g:\mathcal{X}\rightarrow\mathbb{R}_{+}$.

Without loss of generality, for easier estimation, the mean of the noise
$\mathbb{E}\left[N_{Y}\right]$ is assumed to be equal to $0$. The functions
$f\left(X\right)$ and $g\left(X\right)$ contribute to the mean and standard
deviation of the noise. To estimate $f\left(X\right)$ and $g\left(X\right)$,
we can estimate the mean and the standard deviation of $p\left(Y\mid
X\right)$. There are two approaches to estimating the noise term $N_{Y}$ in
HNM. GRCI [26] uses leave-one-out cross-validation to learn the models for
regressing the mean and the mean absolute deviation (MAD). GRCI estimates MAD
instead of the standard deviation as the MAD can be estimated directly and
have smaller estimation errors. However, the use of cross-validation,
especially the leave-one-out method, has high a computational cost. LOCI [9],
on the other hand, models the mean and the variance and uses the Gaussian log-
likelihood
$\log\mathcal{N}\left(Y\mid\mu\left(X\right),\sigma^{2}\left(X\right)\right)$
as the maximization objective. This approach allows the mean and the variance
can be optimized simultaneously, but the assumption of the Gaussian likelihood
is more restrictive compared to GRCI’s approach.

###  3.2 Student’s $t$-Distribution

Student’s $t$-distribution is a distribution that is considered as the
generalization of Gaussian distribution. Belonging to the location-scale
family as the Gaussian family, the $t$-distribution’s parameters also include
a location $\mu$ and a scale $\tau^{2}$. In addition to the location and
scale, the third parameter of the $t$-distribution is the degree of freedom
$\nu$ ($\nu>0$). The probability density function (PDF) of the
$t$-distribution in defined as

(3.2)
$t\left(x\mid\mu,\tau^{2},\nu\right)=\frac{\Gamma\left(\frac{\nu+1}{2}\right)}{\Gamma\left(\frac{\nu}{2}\right)\sqrt{\pi\nu\tau^{2}}}\left(1+\frac{1}{\nu}\frac{\left(x-\mu\right)^{2}}{\tau^{2}}\right)^{-\frac{\nu+1}{2}},$

where $\Gamma\left(z\right)=\int_{0}^{\infty}t^{z-1}\exp\left(-t\right)dt$. As
derived from Eq. (3.2), $t$-distribution is a heavy-tailed distribution where
the heaviness of the tails is controlled by the degree of freedom $\nu$. As
$\nu\rightarrow\infty$, the $t$-distribution will become the Gaussian
distribution. If $\nu>1$, the expected value of the random variable $X\sim
t\left(\mu,\tau^{2},\nu\right)$ is defined as

(3.3) $\mathbb{E}\left[X\right]=\mu,$

and when $\nu>2$, the variance of $X$ is definite and is calculated from both
the scale $\tau^{2}$ and the degree of freedom $\nu$ as follows

(3.4) $\operatorname{Var}\left[X\right]=\tau^{2}\frac{\nu}{\nu-2}.$

Having an additional parameter $\nu$ controlling the variance allows the
$t$-distribution to handle sampling variability and extreme outliers by
adapting the shape of the distribution flexibly (see Figure 1 for an example).
Therefore, the $t$-distribution is more robust than the Gaussian distribution
while maintaining the ability to approximate the Gaussian distribution by
using a high degree of freedom. Due to these advantages, this distribution can
be applied in place of the Gaussian distribution improve the robustness such
as in the Student-$t$ processes [23, 29] as alternatives for Gaussian
processes, reliable variance networks estimation [2], or Student-$t$
variational autoencoder [28] for more robust density estimation. Following
these intuitions, we choose the $t$-distribution for the likelihood in our
approach to achieve a robust estimation of the heteroscedastic noise models.

##  4 ROCHE: Robust Estimation of Causal Heteroscedastic Noise Models

###  4.1 Parameters Estimation with the $t$-Distribution

Similar to previous approaches [26, 9], we need to estimate the mean and the
standard deviation to retrieve the independent residual. We estimate these
parameters via the likelihood of the assumed effect $Y$ given the assumed
cause $X$. The $t$-distribution probability density function in Eq. (3.2) is
chosen for this likelihood as follows

(4.5) $p\left(Y\mid
X\right)=t\left(Y\mid\mu\left(X\right),\tau^{2}\left(X\right),\nu\left(X\right)\right),$

and this likelihood is modeled using a neural network. Inspired by the fact
that the $t$-distribution is a Gaussian distribution with the variance
distributed as an inverse gamma distribution with two parameters $a=\nu/2$ and
$b=\nu\tau^{2}/2$. Instead of modeling the scale function
$\tau^{2}\left(X\right)$ and the degree of freedom function
$\nu\left(X\right)$ separately, we model two alternative parameter functions
$a\left(X\right)$ and $b\left(X\right)$. The degree of freedom $\nu$ and the
scale $\tau^{2}$ are then computed as

(4.6) $\displaystyle\nu\left(X\right)=2a\left(X\right),$ (4.7)
$\displaystyle\tau^{2}\left(X\right)=\frac{b\left(X\right)}{a\left(X\right)}.$

By doing so, the scale $\tau^{2}$ depends on the degree of freedom $\nu$, and
will adapt accordingly to the change in the degree of freedom. Since the
degree of freedom must be greater than $2$ for the variance to exist and the
scale of the $t$-distribution must be greater than $0$, the constraints
$a\left(X\right)>1$ and $b\left(X\right)>0$ are applied when modeling.

With this model, the neural network is optimized by minimizing the negative
log-likelihood as the loss function

(4.8) $\displaystyle\mathcal{L}_{\text{NLL}}$
$\displaystyle=-\frac{1}{n}\sum_{i=1}^{n}\log
t\left(y_{i}\mid\mu\left(x_{i}\right),\tau^{2}\left(x_{i}\right),\nu\left(x_{i}\right)\right)$

where $n$ is the number of samples and $\left(x_{i},y_{i}\right)$ is the
$i$-th sample from the dataset. There could be many models that can be fitted
with this loss function. The most preferable models are the one with higher
degrees of freedom which approach the Gaussian distribution. To implement this
preference, in addition to the negative log-likelihood loss, we propose an
additional constraint loss for the degree of freedom $\nu\left(\cdot\right)$
as follows

(4.9) $\displaystyle\mathcal{L}_{\text{constraint}}$
$\displaystyle=\frac{1}{n}\sum_{i=1}^{n}\left(\frac{\nu\left(x_{i}\right)}{\nu\left(x_{i}\right)-2}\right)^{2}$

This constraint is chosen from the $t$-distribution’s variance in Eq. (3.4),
where the variance is scaled from the distribution’s scale parameter
$\tau^{2}$ with a factor of $\nu/\left(\nu-2\right)$. The value of the
constraint will decrease and approach $1$ when all the values of
$\nu\left(x_{i}\right)$ increase. Penalizing this constraint will also reduce
the scaling term of the variance in Eq. (3.4), which makes the variance become
more stable by having values closer to $\tau^{2}$ instead of exploding to
infinity. Moreover, as the parameter $\tau^{2}$ in our approach is modeled
with Eq. (4.7), raising the value of $\nu$ which corresponds to $a$ will also
decrease the value of $\tau^{2}$. This behavior can avoid the underfitting
scenario where the location parameter $\mu$ is poorly predicted but the scale
is increased to account for the high error ranges. This scenario also happens
to the Gaussian log-likelihood where many approaches, such as $\beta$-NLL [22]
or Faithful Heteroscedastic [25], has been proposed to have more reliable mean
regression by modifying the Gaussian log-likelihood function, but these
methods need to have a trade-off between the mean accuracy and the log-
likelihood. We do not have this trade-off since the constraint term is a
detached module and our method still tries to find the models with the highest
log-likelihood.

###  4.2 Residual Estimation & Cause-Effect Inference

After estimating heteroscedastic noise models in Eq. (3.1), the residual can
be computed from the estimands as follows

(4.10)
$\tilde{N}_{Y}=\frac{Y-\tilde{f}\left(X\right)}{\tilde{g}\left(X\right)},$

where $\tilde{N}_{Y}$, $\tilde{f}\left(X\right)$ and $\tilde{g}\left(X\right)$
denote the estimated $N_{Y}$, $f\left(X\right)$, and $g\left(X\right)$. In our
case with the $t$-distribution, the function $f\left(X\right)$ corresponds to
$\mathbb{E}\left[Y\mid X\right]=\mu\left(X\right)$ and $g^{2}\left(X\right)$
corresponds to $\operatorname{Var}\left[Y\mid
X\right]=\tau^{2}\left(X\right)\frac{\nu\left(X\right)}{\nu\left(X\right)-2}$.
After learning the model, the estimated residual can be determined from the
estimated parameters of the $t$-distribution as follows

(4.11) $\displaystyle\tilde{N}_{Y}$
$\displaystyle=\frac{Y-\tilde{\mu}\left(X\right)}{\sqrt{\tilde{\tau^{2}}\left(X\right)\frac{\tilde{\nu}\left(X\right)}{\tilde{\nu}\left(X\right)-2}}}$

where $\tilde{\mu}$, $\tilde{\tau^{2}}$, $\tilde{\nu}$, $\tilde{a}$, and
$\tilde{b}$ are the parameters estimated with the $t$-distribution likelihood.

As the noise term $N_{Y}$ and the cause $X$ are assumed to be independent, the
dependence between the estimated noise $\tilde{N}_{Y}$ and the cause $X$
quantified using any non-parametric independent test can be utilized to
support the cause-effect inference step. In previous approaches, GRCI [26]
estimates the the independence between $\tilde{N}_{Y}$ and $X$ via the mutual
information, whereas LOCI [9] uses the Hilbert-Schmidt independence criterion
(HSIC) [5]. The Gaussian log-likelihood is also proposed in LOCI [9] as a
score for quantifying the causal direction. However, using Gaussian likelihood
scoring can cause model misspecification when the distribution of the noise is
non-Gaussian, especially in heteroscedastic noise models [21]. A more suitable
choice should be using the quantified dependence between $\tilde{N}_{Y}$ and
$X$. In this work, we choose the HSIC score [5] for easier comparison with
LOCI [9], which chooses the Gaussian likelihood for estimating the
heteroscedastic noise models. Higher HSIC score indicates that two variables
have a higher degree of independence.

##  5 Experiments

__

Figure 2: Performance on synthetic additive (AN & AN-s), location-scale (LS &
LS-s), and multiplicative (MN-U) noise models. Higher accuracy, AUDRC, and
AUROC are preferable. Our ROCHE approach is compared with methods for
heteroscedastic noise models: LOCI [9], GRCI [26], HECI [30], and QCCD [27],
and methods not designed for heteroscedasticity: CAM [1], RESIT [20], CGNN
[4], and ICGI [10]. Our ROCHE approach achieves perfect results on all 5
benchmark datasets. Figure 3: Performance on SIM, SIM-c, SIM-ln, and SIM-G
datasets. Higher accuracy, AUDRC, and AUROC are preferable. Our ROCHE approach
is compared with methods for heteroscedastic noise models: LOCI [9], GRCI
[26], HECI [30], and QCCD [27], and methods not designed for
heteroscedasticity: CAM [1], RESIT [20], CGNN [4], and ICGI [10]. ROCHE
acquires comparable and stable results on most datasets and accomplishes
better results in comparison with the Gaussian-likelihood-based LOCI.
Regression-based methods with subsequent independence tests including ROCHE,
LOCI, GRCI, and RESIT have better performance in the occurrence of
confounders.

###  5.1 Experimental Settings

In this section, we compare our ROCHE approach with related bivariate causal
discovery methods designed for heteroscedastic noise models and homoscedastic
noise models to demonstrate the effectiveness of our approach.

#### Baselines

Evaluated state-of-the-art causal discovery methods for heteroscedastic noise
models including QCCD [27], HECI [30], GRCI [26], and LOCI [9]. Among these
approaches, the most related method to our approach is LOCI which employs the
Gaussian likelihood for estimating the location and scale of the model. For
LOCI, we use the neural network version with HSIC as the score for easier
comparisons. For methods that are designed for homoscedastic or additive noise
models, we use CAM [1] and RESIT [20]. For other baselines that do not assume
heteroscedastic noise models, we consider ICGI [10] with Gaussian reference
measures and CGNN [4].

#### Datasets

We use both synthetic and real causal discovery benchmarks to evaluate our
approach.

The first group of chosen benchmarks is the five synthetic datasets from [27]
including AN, AN-s, LS, LS-s, and MN-U. These five datasets are generated from
structural causal models that are heteroscedastic (or location-scale) noise
models (LS and LS-s datasets) and special cases of heteroscedastic noise
models consisting of homoscedastic additive noise models (AN and AN-s
datasets) and multiplicative noise models (MN-U). The datasets denoted with
“-s” and the MN-U dataset use invertible sigmoid-type functions for the
generative functions which can make the identification of the causal direction
more complicated.

In the second group of benchmarks, we consider more difficult synthetic
datasets comprising the SIM, SIM-c, SIM-ln, and SIM-G from [16] and the Multi
and Net datasets from [4], the Cha dataset from [6]. The four SIM, SIM-c, SIM-
ln, SIM-G datasets are simulated with the functions
$X:=f_{X}\left(N_{X}\right)$ and $Y:=f_{Y}\left(X,N_{Y}\right)$ in the cases
without a confounder (in SIM, SIM-ln, and SIM-G) and the functions
$Z:=f_{Z}\left(N_{Z}\right)$, $X:=f_{X}\left(N_{X},N_{Z}\right)$, and
$Y:=f_{Y}\left(X,N_{Y},N_{Z}\right)$ in the cases with a confounder $Z$ (in
SIM-c). In the SIM-ln dataset, low levels of noise are applied in the models,
and the SIM-G dataset has approximations of Gaussian distributions for the
cause $X$ and approximately Gaussian non-linear additive noise generative
models. The Multi dataset is generated with pre-additive noises
($Y:=f\left(X+N_{Y}\right)$), post-additive noises (similar to conventional
additive noise models), pre-multiplicative noise ($Y:=f\left(X\times
N_{Y}\right)$), and post-multiplicative noise ($Y:=f\left(X\right)\times
N_{Y}$). The pairs in the Net dataset are generated with neural networks with
random distribution for $X$, such as exponential, gamma, log-normal, or
Laplace distribution. The 300 continuous variable pairs in the Cha benchmark
are chosen from the ChaLearn Cause-Effect Pairs Challenge [6].

For a real-world benchmark, we choose the Tübingen cause-effect pairs [16] for
comparison. This dataset contains pairs of cause(s) and effect(s) gathered
from many sources with diverse domains. Each pair in this benchmark has a
corresponding weight to scale down the results of pairs having similar
properties. For that reason, the evaluation results for this benchmark are
also weighted according to these weights.

#### Metrics

The cause-effect inference results are assessed with three common metrics in
bivariate causal discovery literature consisting of the accuracy score, the
area under the decision rate curve (AUDRC), and the area under the receiver
operating characteristic curve (AUROC). The accuracy score is the simplest
evaluation metric which computes the percentage of the pairs in the datasets
whose causal directions are correctly predicted. The AUDRC criterion is
proposed in [9] for quantifying the extent to which decision confidence aligns
with accuracy. Having higher AUDRC means that the methods will predict the
correct causal directions if the confidence of the scores is high, and the
incorrect cases are caused by lower confidence of the predicted score. The
last chosen metric is the AUROC score. This score indicates the ability of the
methods to predict correctly without intermingling between the two causal
directions. To compensate for the class imbalance when computing the AUROC
score, we augment the results by adding an additional result for the reversed
direction with a reversed label for each result acquired.

__

Figure 4: Performance on Multi, Net, and Cha datasets. Higher accuracy, AUDRC,
and AUROC are preferable. Our ROCHE approach is compared with methods for
heteroscedastic noise models: LOCI [9], GRCI [26], HECI [30], and QCCD [27],
and methods not designed for heteroscedasticity: CAM [1], RESIT [20], CGNN
[4], and ICGI [10]. ROCHE acquires the best results on the Net dataset and the
second highest scores on Multi and datasets.

__

Figure 5: Performance on Tübingen cause-effect pairs datasets. Higher
accuracy, AUDRC, and AUROC are preferable. Our ROCHE approach is compared with
methods for heteroscedastic noise models: LOCI [9], GRCI [26], HECI [30], and
QCCD [27], and methods not designed for heteroscedasticity: CAM [1], RESIT
[20], CGNN [4], and ICGI [10]. ROCHE provides promising results and performs
significantly better than LOCI with Gaussian likelihood estimation. In
general, methods with heteroscedasticity have better performance compared to
methods that do not assume heteroscedastic noise models.

###  5.2 Performance on Synthetic Additive, Location-Scale, and Multiplicative
Noise Datasets

Figure 2 illustrates the results of all considered approaches on the AN, AN-s,
LS, LS-s, and MN-U datasets. Our ROCHE approach achieves the perfect results
in all of these datasets which demonstrate our model’s ability to estimate
effectively synthesized heteroscedastic noise data, and special cases
including homoscedastic/additive noise and multiplicative noise data. LOCI,
which uses neural networks with Gaussian likelihood estimation, has some
misspecified cases in LS and LS-s datasets. One case where the causal
direction is wrongly predicted has already been presented in Figure 1. The
reason for these misspecified pairs is due to the suboptimal fit of the
variance in regions with low sample densities. Other methods designed for
heteroscedasticity, which are GRCI, HECI, and QCCD, have some pairs
misidentified, especially when the generative functions are invertible on
AN-s, LS-s, and MN-U datasets. CAM and RESIT achieve perfect scores on the
assumption-satisfied additive noise datasets but as the datasets become more
difficult these methods cannot perform stably. With no assumption in the
structural model, CGNN performs well on AN and LS datasets. However, this
approach is also affected by invertible generative functions. Despite not
being designed for heteroscedastic noise models, IGCI has nearly perfect
results in all cases.

###  5.3 Performance on More Complex Synthetic Datasets

The results for the 7 remaining simulated datasets are presented in Figures 3
and 4. These results show that our approach can adapt better to diverse model
configurations and noise distributions compared to other approaches. On SIM,
SIM-ln, and SIM-G datasets, our ROCHE’s results are the best ones in the group
of methods with heteroscedastic design. On confounding cases in the SIM-c
dataset, our method is more robust and retrieve the causal directions more
accurately than other current models. In most cases, the Student’s
$t$-distribution estimation approach in ROCHE provides better results compared
to the Gaussian estimation in LOCI and the conditional mean and condition mean
absolute deviation estimation in GRCI. Noticeably, regression-based methods
that regress the residuals and use independence testing for determining causal
direction, such as ROCHE, LOCI, GRCI, and RESIT, have demonstrated superior
performance in the scenario when confounding variables are present in the
SIM-c dataset. Despite having an assumption of low noise levels, IGCI has the
lowest distinguishing performance on the SIM-ln dataset. Our approach obtains
the second highest results on the Multi and Cha datasets and the highest
results on the Net dataset. QCCD predicts decently on the Net and SIM-ln
datasets. But on other datasets, the results are less stable with nearly
random predictions on the Cha dataset. Although HECI has the best scores on
the Multi datasets, this method does not perform as well on the remaining
datasets. Homoscedastic methods—CAM and RESIT have high variances in the
results among different benchmarks with even sub-standard results on the Multi
and Cha datasets. CGNN also exhibits high discrepancy between datasets with
decent results in most datasets, except for the SIM and SIM-c benchmarks. On
this group of complex benchmarks, our approach has the most stable performance
across these diverse benchmark datasets.

Table 1: Overall performance on 13 benchmarks. Reported results are the
mean$\pm$std of the Accuracy, AUDRC, and AUROC across the benchmarks. Higher
Accuracy, AUDRC, and AUROC are preferable. The best results are presented in
bold style. Our ROCHE approach achieves the best overall results with high
stability, which demonstrates the robustness of our method.

Method | Accuracy (%) $\uparrow$ | AUDRC (%) $\uparrow$ | AUROC (%) $\uparrow$
---|---|---|---
ROCHE (Ours) | $\mathbf{87.69\pm 10.23}$ | $\mathbf{92.86\pm\phantom{0}8.16}$ | $\mathbf{92.02\pm\phantom{0}7.90}$
LOCI [9] | $84.24\pm 11.43$ | $89.04\pm 13.42$ | $88.58\pm 11.66$
GRCI [26] | $83.20\pm\phantom{0}9.44$ | $89.73\pm 10.19$ | $88.86\pm\phantom{0}8.70$
HECI [30] | $65.25\pm 18.17$ | $73.34\pm 21.19$ | $69.60\pm 20.97$
QCCD [27] | $78.21\pm 16.61$ | $85.77\pm 13.70$ | $82.69\pm 15.79$
CAM [1] | $72.42\pm 21.23$ | $73.89\pm 21.97$ | $72.77\pm 22.33$
RESIT [20] | $64.45\pm 30.48$ | $68.36\pm 30.05$ | $67.08\pm 29.96$
CGNN [4] | $73.89\pm 15.43$ | $83.35\pm 14.48$ | $79.87\pm 15.55$
IGCI [10] | $73.69\pm 22.98$ | $77.26\pm 24.04$ | $76.00\pm 23.54$

###  5.4 Performance on Real-World Data

In addition to synthetic datasets, we also evaluate our method on the real-
world Tübingen cause-effect pairs dataset. The accuracy, AUDRC, and AUROC
results on this benchmark are depicted in Figure 5. Utilizing the more robust
$t$-distribution for the likelihood in ROCHE instead of the Gaussian
distribution in LOCI contributes to a huge performance gain on the Tübingen
benchmark. Our approach accomplishes promising results with the second highest
in accuracy and the third highest in other evaluation metrics on this dataset
which are satisfactory and comparable to other methods. The lower robustness
of the Gaussian likelihood estimation on real data affects the performance of
LOCI substantially and leads to the method’s lowest AUDRC and AUROC in this
benchmark. On the more complicated Tübingen dataset, methods targeted
heteroscedastic noise models provide better predictions compared to non-
heteroscedastic baselines, especially in the case of CAM and RESIT.

###  5.5 Overall Performance across All Benchmarks

We summarize the overall performance across 13 benchmarks in Table 1 to have a
complete evaluation for all methods. In general, our method has the best
average results in all three evaluation metrics with the percentage values of
accuracy, AUDRC, and AUROC being $87.69\%$, $92.86\%$, and $92.02\%$
respectively. In addition, ROCHE is more robust in comparison with remaining
methods with margins of errors less than $10\%$ in AUDRC and AUROC criteria.
The standard deviations of our approach have the second lowest value of
$10.23\%$ for the accuracy score, which is slightly higher than the best value
of $9.44\%$ from GRCI, and the lowest values of $8.16\%$ and $7.90\%$ for
AUDRC and AUROC scores respectively.

##  6 Conclusion

In this work, we have presented ROCHE, a robust estimation approach for
bivariate causal discovery with heteroscedastic noise models. Instead of
relying on the Gaussian distribution, as done in previous approaches which may
not accurately capture the model parameters, we adopt the more robust
Student’s $t$-distribution. This choice allows us to account for sampling
variability and extreme values more effectively. We propose a framework for
modeling the parameters using the $t$-distribution along with necessary
constraints to ensure the resulting estimators are suitable for subsequent
cause-effect inference tasks. The effectiveness of ROCHE has been demonstrated
through comprehensive experiments, yielding promising results and showcasing
the best overall performance across 13 benchmark datasets. Furthermore, our
approach exhibits robustness, as evidenced by the lowest overall margins of
errors across various evaluation metrics. In future work, we aim to extend our
approach to handle multivariate heteroscedastic noise models, thus enhancing
its applicability to real-world data scenarios.

## References

  * [1] P. Bühlmann, J. Peters, and J. Ernest, CAM: Causal additive models, high-dimensional order search and penalized regression, The Annals of Statistics, (2014), pp. 2526–2556.
  * [2] N. S. Detlefson, M. Jørgensen, and S. Hauberg, Reliable training and estimation of variance networks, in Advances in Neural Information Processing Systems, vol. 32, 2019.
  * [3] B. Duong and T. Nguyen, Bivariate causal discovery via conditional divergence, in Proceedings of the Conference on Causal Learning and Reasoning, 2022, pp. 236–252.
  * [4] O. Goudet, D. Kalainathan, P. Caillou, I. Guyon, D. Lopez-Paz, and M. Sebag, Learning functional causal models with generative neural networks, Explainable and Interpretable Models in Computer Vision and Machine Learning, (2018), p. 39.
  * [5] A. Gretton, O. Bousquet, A. Smola, and B. Schölkopf, Measuring statistical dependence with Hilbert-Schmidt norms, in Proceedings of the International Conference on Algorithmic Learning Theory, 2005, pp. 63–77.
  * [6] I. Guyon, A. Statnikov, and B. B. Batu, Cause effect pairs in machine learning, Springer, 2019.
  * [7] P. Hoyer, D. Janzing, J. M. Mooij, J. Peters, and B. Schölkopf, Nonlinear causal discovery with additive noise models, in Advances in Neural Information Processing Systems, vol. 21, 2008.
  * [8] C.-W. Huang, D. Krueger, A. Lacoste, and A. Courville, Neural autoregressive flows, in Proceedings of the International Conference on Machine Learning, 2018, pp. 2078–2087.
  * [9] A. Immer, C. Schultheiss, J. E. Vogt, B. Schölkopf, P. Bühlmann, and A. Marx, On the identifiability and estimation of causal location-scale noise models, in Proceedings of the International Conference on Machine Learning, 2023.
  * [10] D. Janzing, J. Mooij, K. Zhang, J. Lemeire, J. Zscheischler, P. Daniušis, B. Steudel, and B. Schölkopf, Information-geometric approach to inferring causal directions, Artificial Intelligence, 182 (2012), pp. 1–31.
  * [11] D. Janzing and B. Schölkopf, Causal inference using the algorithmic Markov condition, IEEE Transactions on Information Theory, 56 (2010), pp. 5168–5194.
  * [12] I. Khemakhem, R. Monti, R. Leech, and A. Hyvarinen, Causal autoregressive flows, in Proceedings of the International Conference on Artificial Intelligence and Statistics, 2021, pp. 3520–3528.
  * [13] A. N. Kolmogorov, Three approaches to the quantitative definition of complexity, Problems in Information Transmission, 1 (1965), pp. 3–11.
  * [14] S. Kpotufe, E. Sgouritsa, D. Janzing, and B. Schölkopf, Consistency of causal inference under the additive noise model, in Proceedings of the International Conference on Machine Learning, 2014, pp. 478–486.
  * [15] A. Marx and J. Vreeken, Telling cause from effect using MDL-based local and global regression, in Proceedings of the IEEE International Conference on Data Mining, 2017, pp. 307–316.
  * [16] J. M. Mooij, J. Peters, D. Janzing, J. Zscheischler, and B. Schölkopf, Distinguishing cause from effect using observational data: methods and benchmarks, The Journal of Machine Learning Research, 17 (2016), pp. 1103–1204.
  * [17] J. Pearl, Causality, Cambridge University Press, 2009.
  * [18] J. Peters and P. Bühlmann, Identifiability of Gaussian structural equation models with equal error variances, Biometrika, 101 (2014), pp. 219–228.
  * [19] J. Peters, J. M. Mooij, D. Janzing, and B. Schölkopf, Identifiability of causal graphs using functional models, in Proceedings of the Conference on Uncertainty in Artificial Intelligence, 2011, pp. 589–598.
  * [20] , Causal discovery with continuous additive noise models, The Journal of Machine Learning Research, 15 (2014), pp. 2009–2053.
  * [21] C. Schultheiss and P. Bühlmann, On the pitfalls of Gaussian likelihood scoring for causal discovery, Journal of Causal Inference, 11 (2023), p. 20220068.
  * [22] M. Seitzer, A. Tavakoli, D. Antic, and G. Martius, On the pitfalls of heteroscedastic uncertainty estimation with probabilistic neural networks, in Proceedings of the International Conference on Learning Representations, 2022.
  * [23] A. Shah, A. Wilson, and Z. Ghahramani, Student-t processes as alternatives to Gaussian processes, in Proceedings of the International Conference on Artificial Intelligence and Statistics, 2014, pp. 877–885.
  * [24] S. Shimizu, P. O. Hoyer, A. Hyvärinen, A. Kerminen, and M. Jordan, A linear non-Gaussian acyclic model for causal discovery., Journal of Machine Learning Research, 7 (2006).
  * [25] A. Stirn, H. Wessels, M. Schertzer, L. Pereira, N. Sanjana, and D. Knowles, Faithful heteroscedastic regression with neural networks, in Proceedings of the International Conference on Artificial Intelligence and Statistics, 2023, pp. 5593–5613.
  * [26] E. V. Strobl and T. A. Lasko, Identifying patient-specific root causes with the heteroscedastic noise model, arXiv preprint arXiv:2205.13085, (2022).
  * [27] N. Tagasovska, V. Chavez-Demoulin, and T. Vatter, Distinguishing cause from effect using quantiles: Bivariate quantile causal discovery, in Proceedings of the International Conference on Machine Learning, 2020, pp. 9311–9323.
  * [28] H. Takahashi, T. Iwata, Y. Yamanaka, M. Yamada, and S. Yagi, Student-t variational autoencoder for robust density estimation, in Proceedings of the International Joint Conference on Artificial Intelligence, 2018, pp. 2696–2702.
  * [29] Q. Tang, L. Niu, Y. Wang, T. Dai, W. An, J. Cai, and S.-T. Xia, Student-t process regression with Student-t likelihood, in Proceedings of the International Joint Conference on Artificial Intelligence, 2017, pp. 2822–2828.
  * [30] S. Xu, O. A. Mian, A. Marx, and J. Vreeken, Inferring cause and effect in the presence of heteroscedastic noise, in Proceedings of the International Conference on Machine Learning, 2022, pp. 24615–24630.
  * [31] K. Zhang and A. Hyvärinen, On the identifiability of the post-nonlinear causal model, in Proceedings of the Conference on Uncertainty in Artificial Intelligence, 2009, pp. 647–655.
  * [32] K. Zhang, Z. Wang, J. Zhang, and B. Schölkopf, On estimation of functional causal models: general results and application to the post-nonlinear causal model, ACM Transactions on Intelligent Systems and Technology, 7 (2015), pp. 1–22.