# End-to-End Intelligent Framework for Rockfall Detection

Thanasis Zoumpekas1∗<EMAIL_ADDRESS>Anna Puig1<EMAIL_ADDRESS>Maria Salamó1<EMAIL_ADDRESS>David García-Sellés2<EMAIL_ADDRESS>Laura
Blanco Nuñez3,4<EMAIL_ADDRESS>Marta Guinau2<EMAIL_ADDRESS>1WAI Research
Group, IMUB and UBICS Institutes, Department of Mathematics and Computer
Science, University of Barcelona, Barcelona, Spain  2RISKNAT Research Group,
Geomodels Institute, Department of Earth and Ocean Dynamics, University of
Barcelona, Barcelona, Spain  3GGAC Research Group, Geomodels Institute,
Department of Earth and Ocean Dynamics, University of Barcelona, Barcelona,
Spain  4Anufra, Soil and Water Consulting, Barcelona, Spain

###### Abstract

Rockfall detection is a crucial procedure in the field of geology, which helps
to reduce the associated risks. Currently, geologists identify rockfall events
almost manually utilizing point cloud and imagery data obtained from different
caption devices such as Terrestrial Laser Scanner or digital cameras. Multi-
temporal comparison of the point clouds obtained with these techniques
requires a tedious visual inspection to identify rockfall events which implies
inaccuracies that depend on several factors such as human expertise and the
sensibility of the sensors. This paper addresses this issue and provides an
intelligent framework for rockfall event detection for any individual working
in the intersection of the geology domain and decision support systems. The
development of such an analysis framework poses significant research
challenges and justifies intensive experimental analysis. In particular, we
propose an intelligent system that utilizes multiple machine learning
algorithms to detect rockfall clusters of point cloud data. Due to the
extremely imbalanced nature of the problem, a plethora of state-of-the-art
resampling techniques accompanied by multiple models and feature selection
procedures are being investigated. Various machine learning pipeline
combinations have been benchmarked and compared applying well-known metrics to
be incorporated into our system. Specifically, we developed statistical and
machine learning techniques and applied them to analyze point cloud data
extracted from Terrestrial Laser Scanner in two distinct case studies,
involving different geological contexts: the basaltic cliff of Castellfollit
de la Roca and the conglomerate Montserrat Massif, both located in Spain. Our
experimental data suggest that some of the above-mentioned machine learning
pipelines can be utilized to detect rockfall incidents on mountain walls, with
experimentally proven accuracy.

###### keywords:

Machine Learning , Imbalanced Classification , Rockfall Monitoring , Geology ,
Expert Systems

##  1 Introduction

In the field of geology, one crucial task is rockfall detection, which helps
to reduce the risk of future hazards (Robbins et al. [2021]). Recently,
technologies that are able to characterize the geometrical properties of rock
slopes and cliffs have emerged (Robiati et al. [2019]). As a result, the
possibility of detecting changes in a cliff with high precision has been
increased greatly.

Currently, geoscientists utilize specific methodologies in order to detect
changes in rock slopes mainly by comparing measurements at different points in
time (DiFrancesco et al. [2020]). However, the problem with such techniques is
that they rely greatly on the sensitivity of the sensor that captures the data
or even the measurement tool. Consequently, due to potential measurement
errors, the users need to examine case by case analyzing the nature of the
change in the cliff, i.e. vegetation, edge effect, noise or random objects.
For instance, it is often difficult to distinguish rock detachments from other
changes optically in point cloud or imagery data.

The issues presented above justify the need for a new automated intelligent
system designed to detect rockfalls. For this reason, in this paper, we
propose an intelligent framework for rockfall detection. At the time of
writing, a thorough search for such a system has failed to yield results. In
addition, we consider our study to be capable of elucidating the field towards
the implementation of a general machine learning classification framework able
to handle cases with extremely imbalanced data. The level of data imbalance in
this study is significant. This is justified in the later sections of this
article since there is a low number of rockfall labelled instances. Depending
on the nature of the scenery, the vegetation or edge effects can be numerous
compared to rockfall events.

Therefore, our contributions are three-fold: (i) a full intelligent framework
for rockfall detection handling imbalanced data; (ii) a study involving real
data from two case studies, which shows the efficacy and the efficiency of the
developed system; and (iii) a web-based rockfall detection system. Last but
not least, our framework could also be incorporated as a component in a
geological decision support system. Our study could also be utilized as a
baseline methodology and a detection accuracy benchmark for future related
experimental analyses.

In this article, we review several of the intelligent methods dealing with
rockfall and landslide detection, we implement the associated machine learning
models accompanied by various resampling strategies to handle the imbalanced
nature of the study, and apply them on point cloud data to identify rockfall
events. Two well differentiated geological environments have been selected as
case studies. The first one is located at the Montserrat Massif (Barcelona,
Spain), which corresponds to a fractured conglomerate cliff, called Degotalls.
The second case study involves data from the basaltic lava flow cliff of
Castellfollit de la Roca (Girona, Spain). The acquired data are pre-processed
and used as input in the learning phase of our intelligent framework. We
systematically compare various models in order to select the most accurate for
each case study, to be used in our prototype system for achieving effective
rockfall event identifications.

The paper is organized as follows. Section 2 contains an overview of related
efforts and Section 3 summarizes the design specifications and the
implementation of the developed framework, while also providing the background
material required for understanding the whole procedure of detecting rockfall
events. In Section 4, we present the experimental analysis of the considered
intelligent process, based on popular performance metrics for imbalanced
classification analysis, and the prototype web-based Rockfall Detection
System. Section 5 includes the final observations of our experiments. Finally,
Section 6 contains our concluding remarks and possible issues for further
exploration.

##  2 Related Work

Various research studies have been conducted on the intersection of the
geology and machine learning domains. Over recent years, a number of machine
learning applications have emerged in the geoscience field and there is a
growing enthusiasm for intelligent methods (Dramsch [2020]). However, the
geoscience domain presents new and special challenges for machine learning
algorithms and methodologies, because of the combinations of geoscience
properties encountered in each specific case. In addition, there is an open
need for novel machine learning research and automatic intelligent analysis,
especially in the geosciences field as highlighted by (Karpatne et al.
[2019]). The automation of intelligent pipelines performing specific, special
analysis will bring significant advances in both the geoscience and machine
learning domains, especially in the task of detecting objects and events,
estimating and long-term forecasting for geoscience variables and in mining
relationships in geoscience data.

The detection of deformations and rockfall events have been studied in the
literature for a couple of decades (Crosta & Agliardi [2003], Carrara [1983]).
The number of scientific publications on the use of modern and advanced
sensors such as TLS in rockfall and landslide studies has increased
considerably in the last years. However, further investigation on the
development of new methodologies using such sensors is required in order to
address and improve the current time consuming analytical methods due to the
high volume of data that these sensors produce (Abellán et al. [2010, 2014],
Lague et al. [2013], Oppikofer et al. [2009], Jaboyedoff et al. [2012]).
Research studies on the identification of landslide and rockfall events and on
general classification tasks in the geology domain present variations
regarding their input data, feature inventories, methodologies and in general
their specific objectives.

Regarding the input data, we could categorize the related research studies
into three broad categories: (i) the imagery data studies; (ii) point cloud
studies; and (iii) other sensory data studies. Considering imagery data
studies, remote-sensing techniques are considered to be among the most
important for landslide event detection and monitoring. Multiple efforts by
researchers working in the remote sensing field applied in geology appear in
the literature, utilizing intelligent methods on high-resolution Earth
Observation imagery data (Prakash et al. [2020]). Imagery data seem to be a
popular data source utilized in numerous studies. Recently published articles,
such as (Xiao et al. [2018], Fanos et al. [2018], Pham et al. [2018], Bandura
et al. [2019], Ghorbanzadeh et al. [2019], Wang et al. [2020a], Bui et al.
[2020]) use images as their main data source. In addition, researchers
involved in (James & Robson [2012]) constructed 3D images using a multi-view
stereo algorithm and consumer-grade cameras.

On the other hand, the use of TLS devices to extract point cloud data for the
categorization of landslide kinematics is significant. Also, research studies
show that it seems to be necessary to integrate point cloud data from various
sources, namely TLS combined with Airborne Laser Scanner (ALS) or Structure
from Motion (SfM) methods combined with TLS, in order to overcome the
limitations of each individual technique (Lissak et al. [2020]). Point clouds
are heavily utilized by various researchers, mainly because they carry
detailed, high quality information. Specifically, recent studies, including
(Mayr et al. [2018], Weidner et al. [2019, 2020b], Kong et al. [2020], Huu
Phuong [2020], Fanos et al. [2020], Weidner et al. [2020a], Wang et al.
[2020b]) use point clouds as their main data source for their analysis.
Moreover, advanced research studies use both images and point clouds
portraying a speciality and differentiate themselves from the majority of the
related studies (Fanos & Pradhan [2019], Bernsteiner et al. [2020], Loghin et
al. [2020]). Furthermore, Salvini et al. [2013] use images and TLS point cloud
data to model slopes. Last but not least, Hemalatha et al. [2019] utilize data
from wireless sensor networks and machine learning algorithms in order to
monitor and forecast landslides in real time.

Several researchers have compiled a brief selection of the features used for
landslide and rockfall identification. These features can be categorized in
five groups, namely morphological, hydrological, geological, land cover
features and features obtained from other sources, like rainfall intensity,
according to Prakash et al. [2020]. Pure geological features like the
lithology of a studied field or geo-structural information appear in older
studies such as (Agliardi et al. [2001]). Recently, various investigators have
used explanatory variables obtained from simple cartographic operations on
geological data. Commonly, these variables form land cover features determined
from distances to faults, rivers, and roads, which are calculated using basic
spatial operations in the GIS software (Reichenbach et al. [2018]). In
addition, Wang et al. [2020a] use combined morphological predictors, namely
elevation, aspect, curvature, slope, with hydrological ones like wetness index
and rainfall intensity data. Similarly, Micheletti et al. [2014] utilize
morphological and hydrological features adding multiple curvature
measurements, such as plan and surface curvature in order to explain the
mechanisms of landslides and to confirm the state of vegetation, roads and, in
general, geometric deformations.

Furthermore, there are variations regarding the methodologies utilized in
rockfall and landslide identification and forecasting, ranging from methods
using traditional geology and statistics to more intelligent ones using
machine learning approaches such as deep learning and neural networks.
Multiple efforts have been presented introducing analysis frameworks able to
deal with landslide and rockfall events.

Traditional geology is mainly utilized in research articles published more
than 10 years ago, such as (Abellán et al. [2009]) although recent papers
still investigate the application of such methods (Robiati et al. [2019]).
Lague et al. [2013] proposed a method, which compares and combines two
different sets of point clouds obtained from a TLS device. According to their
methodology, two different time frames of point cloud data are combined and
later compared in order to identify whether the input data form deformation or
rockfall event. The aforementioned method uses traditional statistics and
lacks of generalization performance.

Machine learning based methodological workflows are the most used techniques,
as there are various research articles utilizing them. Recent research studies
such as (Weidner et al. [2019], Hemalatha et al. [2019], Weidner et al.
[2020b], Kong et al. [2020], Huu Phuong [2020], Bernsteiner et al. [2020],
Loghin et al. [2020], Weidner et al. [2020a], Wang et al. [2020b]) utilize
mainly machine learning algorithms for rock-slope and landslide monitoring and
analysis. More elaborate machine learning methods outperform the majority,
examples include Mayr et al. [2017], who developed a landslide monitoring
approach for TLS point cloud data, which integrates a specialized machine
learning classification with topological rules in an object-based analysis
framework.

Deep learning and neural networks are currently utilized in fewer studies,
such as (Shi et al. [2019], Chen et al. [2015]). However, advanced neural
network architectures using specialized loss functions are being employed to
tackle specific issues in the field such as data imbalance in landslide
analysis, showing important advances in terms of performance (Prakash et al.
[2020]). Also, Xiao et al. [2018] present a landslide susceptibility
assessment framework based on deep learning algorithms using multi-source
imagery data. A wide assortment of recently published research studies
combines and compares machine learning, neural networks and ensemble based
techniques in order to achieve higher performance in terms of accuracy and
generalisability (Bui et al. [2020], Xiao et al. [2018], Fanos et al. [2018],
Fanos & Pradhan [2019], Fanos et al. [2020], Pham et al. [2018], Ghorbanzadeh
et al. [2019], Wang & Jia [2019], Bandura et al. [2019], Wang et al. [2020a],
Fanos et al. [2020]).

Landslide and rockfall identification are two tasks that present significant
imbalanced data issues. The data imbalanced issue has been addressed in
multiple studies. Prakash et al. [2020] approach the imbalanced learning task
by utilizing sophisticated loss functions in the training phase and data
augmentation techniques. Stumpf & Kerle [2011] proposed a machine learning
based analysis framework with an iterative scheme to handle class imbalance.
Zhao et al. [2020] tackled minor data imbalance issues in predicting landslide
susceptibility by utilizing a voting system and the random processing of
samples with a random forest algorithm.

In a more recent framework, Fanos et al. [2020] proposed combining ALS and TLS
data with GIS and introducing a hybrid ensemble model and a 3D kinematic
rockfall forecasting model able to deal with rockfall hazard assessment,
achieving promising accuracy. However, they did not address the imbalanced
nature of the study or deal with imbalanced data. Our proposal considers a
different machine learning approach that also deals with the problem of
imbalanced data using different types of data sources. Specifically, we
propose an intelligent analysis framework and rockfall detection decision
support software, that incorporate various processing stages considered to be
essential for such problems, namely clustering, resampling, model
parameterization and feature selection. Regarding the input data sources, we
consider only point cloud data acquired by a TLS device from two distinct
geological contexts, with features derived mostly from the morphology of the
terrain.

##  3 End-to-End Machine Learning Framework: Design and Implementation

This section introduces our developed end-to-end machine learning framework
for detecting rockfall events. First, we explain the basic procedure of
detecting rockfalls. Then, we analytically present our proposed framework.

###  3.1 Background on Detecting Rockfalls

One of the most common processes in the detection of rockfall events on
mountain cliffs and slopes is explained in (DiFrancesco et al. [2020]) and
depicted in Fig 1, specifically in steps (a) - (e). The methodology implies
capturing periodical measurements of the same cliff face at different time-
frames with a Terrestrial Laser Scanner (TLS), denoted as step (a) in Fig 1.
TLS is a measuring device which offers the ability to collect dense point-
clouds of objects. It also provides high-precision and high-accuracy data and
is widely used in the geology domain (Williams et al. [2011]). After the
capture, the procedure continues with the detection of changes in the surface
of the cliff from point cloud data comparison, which is mostly performed with
a technique called m3c2 (Lague et al. [2013]), denoted as step (b). As a
result, a new point cloud is obtained containing the metric distances between
the compared dense clouds. A clustering algorithm is then applied in step (c)
following the change detection step outputting a set of clusters, which are
aggregations of points with a significant distance value. These clusters allow
for the management of subsets of point clouds with specific topological
properties. A statistical analysis then generates a set of features (step (d))
to be evaluated manually, through a visual inspection with expert criteria, to
determine whether they are rockfalls or random noise (step (e)). In most
cases, samples, i.e. the clusters, happen to be random noise or even
measurement errors by the scanner itself.

The current semi-manual classification task of the TLS point cloud data for
the detection of rockfall events presents two main challenges. The first
concerns the sensibility of the sensor because in some cases the detection of
movements is smaller than the margin of error of the device. The second
concerns the process of distinguishing rock movement events from other kinds
of events, such as the movement of the sensor between measurements, the
appearance of vegetation or even random noise. The use of clustering
techniques helps to mitigate the aforementioned issues.

Machine learning algorithms present an interesting and currently widely
accepted solution for the automatic classification of TLS point cloud data for
the detection of rockfall events. However, rockfall detection is considered to
be a highly imbalanced classification task, due to the rarity of a rockfall
event in a relative dataset. Moreover, clustered point cloud data present
patterns that are not easily distinguishable, while having high
dimensionality, due to the considerable number of features. In addition,
automating the above-mentioned process seems to be an interesting solution for
an individual working in the geology domain. There is an open need for a
consistent and concise intelligent rockfall identification framework, as
detailed by (Karpatne et al. [2019]). In Section 3.2, we propose a developed
machine learning framework, which represents step (f) in Fig 1.

Figure 1: Procedure for detecting rockfalls.

###  3.2 Framework Overview

An outline of our proposal can be seen in Fig 3, where we synthesize our
framework by providing its methodology workflow, and in Fig 2, where we
present its component-based system architecture.

First, we collect point cloud data in different time frames (Fig 2 (a)) using
the TLS device and store it in a database (Fig 2 (b)). Next, we pre-process it
(Fig 2 (c)) in order to be fed in our analysis framework. Then, we
statistically analyse the input data, obtaining descriptive statistics to
elucidate the field regarding the data. The obtained dataset is then stored
and normalized. Subsequently, to prevent issues of imbalance, we define our
resampling module, portrayed in Fig 2 (d), which balances the normalized data
(Fig 2 (e)) to be used as input for machine learning algorithms (Fig 2 (f)).
Several intelligent machine learning pipelines are utilized to detect rockfall
events and to identify the most significant features (Fig 2 (g)). Finally, a
reporting component is used in order to display the results in a convenient
and illustrative manner, as depicted in Fig 2 (h).

Regarding our methodology, we initially feed normalized point cloud data in
our intelligent framework as depicted in Fig 3 (a). Then, in the resampling
process (Fig 3 (b)), an assortment of several resampling strategies
accompanied by multiple machine learning models are investigated in order to
handle this imbalanced classification task. The three best resamplers are then
selected and paired with a variety of models in the model selection and
parametrization stage (Fig 3 (c)). The completion of the hyper-
parameterization stage follows the feature selection phase (Fig 3 (d)), in
which the best parameterized models of each model variant accompanied by each
one of the three best resamplers are experimentally evaluated using different
numbers of features. The output of the aforementioned stage consists of
multiple properly parameterized machine learning pipelines (Fig 3 (e)). These
pipelines are then statistically compared in order to decide on the best
intelligent pipeline to be used in our prototype system (Fig 3 (f)), which is
able to produce detection results (Fig 3 (g)) and rank feature importance (Fig
3 (h)) with promising accuracy. In the following sub-sections, we analytically
present our framework and methodological approach.

Figure 2: Component-based system architecture. Figure 3: Intelligent framework
- methodology workflow.

###  3.3 Data Collection

The data used in this study has been freely provided by the
RISKNAT111http://www.ub.edu/risknat/ research group and the
GEOMODELS222http://www.ub.edu/geomodels/ Research Institute, which belong to
the University of Barcelona.

In this article, we utilize two distinct case studies. Regarding the first
case study, we use point cloud data measured from the Degotalls cliff in
Montserrat Massif, located in Barcelona, Spain. The dataset consists of
clustered TLS point cloud data aggregated from temporal point cloud
measurements, from 2007 to 2020 in eight nearly regular time steps. The second
case study includes clustered TLS point cloud data measured from the cliff of
Castellfollit de la Roca, located in Girona, Spain, from 2008 and 2012. We
would like to highlight that the two landscapes are very different in terms of
their geological and imagery properties.

Actually, in each case study, the data is composed of a set of statistics
computed from the point cloud data, using a technique based on the m3c2
methodology (Lague et al. [2013]), which creates clusters of points, as
described in Section 3.1, including several numerical features that are used
in the proposed workflow to identify rockfalls. Then, the aforementioned
statistical set is fed in our analysis framework. The Degotalls data consist
of 6004 instances, i.e. clusters, and has 37 distinct numerical features,
while the Castellfollit data consist of 10371 instances and has 31 numerical
features. The aforementioned features include the coordinates of the point
clouds and other statistically computed values. A list of these variables with
a brief explanation can be found in S1 Appendix.

###  3.4 Data Analysis

This section provides an exploratory data analysis of the point cloud data.
This analysis is fundamental in predictive analytics because it reveals not
only the existence of missing data but also elucidates the landscape of the
whole data collection phase. Specifically, in tables 4 and 5 in S2 Appendix,
we present the summary statistics for all the data features used in the
Degotalls and Castellfollit case studies respectively, providing basic
information for our subsequent analysis. For clarification purposes, we chose
to use an abbreviation for each variable of the data. The complete variable
names followed by a brief explanation can be found in S1 Appendix.

The classification labels of the clusters are determined by the event that
causes the change in the surface of the cliff. Rockfall events are denoted as
”Candidate”. Also, we call ”Precursor” the events in which the rock presents a
small movement prior to rockfall. In addition, the vegetation of the cliffs is
denoted as ”Vegetation” and the unknown or human-based events as ”Unknow”. The
artifacts due to the edge effects or the TLS noise are denoted as
”Limit_effect”.

Observing the graphs in Fig 4, it is clear that the data in each case study
are considered highly imbalanced. In the Degotalls case study (Fig 4 (a)), we
have only 65 rockfall candidate samples compared to the total of 6004 samples.
On the other hand, the Castellfollit case study (Fig 4 (c)) includes only 38
compared to the total of 10371 samples. Please note, that the labelling
process is done by expert geoscientists with a visual inspection of the 6004
and the 10371 clusters.

###  3.5 Data Preprocessing

The data analysis in Section 3.4 shows that the input data contain values of a
varying range of significant size, so normalization should be conducted in
order to assist the optimization algorithms to converge faster and the machine
learning models to achieve the best performance (Singh & Singh [2019]). We
first normalize the data using z-score normalization technique as proposed in
(Singh & Singh [2019]), which is defined in equation 1.

$X_{norm}=\frac{X-mean(X)}{std(X)},$ (1)

where $X$ denotes the feature array and $X_{norm}$ is the normalized $X$ as
resulting from the subtraction of its mean ($mean(X)$) and division by its
standard deviation ($std(X)$).

The normalized clustered data of point clouds are used for the training of our
machine learning models. Then, a vector ($v$) is constructed that models the
target label naively as follows. If the cluster is considered a rockfall
event, which can also be denoted as rockfall candidate event, then we set a
positive label ($v_{i}=1$), and if not we assign a negative label ($v_{i}=0$).
A summary of cluster labels is depicted in Figure 4, where sub-figures (a) and
(c) show the distribution of cluster labels before and figures (b) and (d)
after encoding in the Degotalls and Castellfollit case studies respectively.

Please note that the rockfall candidates belong in the class where $v_{i}=1$
and the various ”Limit_effect”, ”Unknow”, ”Vegetation”, ”Precursor” cases
belong in the class where $v_{i}=0$.

(a) Cluster labels - Degotalls case study.

(b) Encoded cluster labels for classification task - Degotalls case study.

(c) Cluster labels - Castellfollit case study.

(d) Encoded cluster labels for classification task - Castellfollit case study.

Figure 4: Target labels summary. Y-axis is in log-scale for comparison
purposes. The actual values are included as text above each bar.

###  3.6 Resampling Techniques

Rockfall events are characterized by their rarity and thus our task is
considered to be an imbalanced classification task. This is also justified by
Fig 4, where the candidate clusters for rockfall represent roughly 1% and 0.4%
percent of the total cases in Degotalls and Castellfollit respectively. There
are two well known approaches to balance the data instances, namely
undersampling and oversampling methods (Hernandez et al. [2013]).
Undersampling removes elements from the majority class, while oversampling
creates synthetic samples from the minority class. There are also ensemble
methods that undersample the majority class and oversample the minority one
simultaneously (Junsomboon & Phienthrakul [2017]). Below, we briefly describe
each one of the resamplers examined in this research. Finally, we conclude by
utilizing the three best approaches in our subsequent analysis phases,
according to their performance.

Undersampling Methods

Naively, the simplest technique to perform undersampling is by doing it
randomly, i.e. by removing individual samples belonging to the majority class
without any heuristic extraction. More elaborate techniques use some kind of
heuristic processing in order to remove elements that are informationally
irrelevant to the whole dataset. Some of the aforementioned methods, aggregate
distinct samples to undersample the data using well-known clustering
techniques, such as (Yen & Lee [2009]).

In this study, we utilize two undersampling methodologies based on the K-means
clustering algorithm. Specifically, we use two of the methods proposed by Yen
& Lee [2009] utilizing the K-means with the number of clusters equal to the
number of samples belonging to minority class. The first methodology generates
centroids based on K-means clustering and undersamples the majority class by
replacing a cluster of majority samples with the cluster centroid of the
K-Means algorithm (denoted Cluster Centroids in this writing). The second one
generates centers and undersamples the majority class by replacing a cluster
of majority samples with the cluster center of the K-Means algorithm (denoted
Cluster Representatives).

Oversampling Methods

Oversampling techniques perform the balancing of the dataset by focusing on
the minority classes. Various oversampling approaches are described in the
literature (Kovács [2019]). One of the mostly utilized oversampling
techniques, namely SMOTE, generates synthetic samples by combining features of
the nearest neighbours of the minority class (Chawla et al. [2002]). Numerous
variants of the aforementioned algorithm appear in the literature (Fernández
et al. [2018]). Kovács [2019] presented a detailed empirical comparison of
multiple variants of minority oversampling techniques involving a wide
assortment of imbalanced datasets for evaluation purposes. In this study, we
focus on some of them, specifically the ones that we utilized for the purpose
of this research.

The ADASYN oversampler uses a weighted density distribution $r_{x}$ for
different minority class examples according to their level of difficulty in
learning, thereby deciding the number of synthetic samples to be generated for
a particular point (He et al. [2008]). In contrast, SMOTE uses a uniform
weight for all minority points. The major difference between SMOTE and ADASYN
is the difference in the generation of synthetic sample points for minority
data points.

Barua et al. [2013] proposed a proximity weighted synthetic oversampling
technique (ProWSyn) that generates effective weight values for the minority
data samples based on the sample’s proximity information. Proximity, in this
case, is defined as the distance between the sample and the boundary (Barua et
al. [2013]). Another extension of SMOTE through a new component, an iterative
ensemble-based noise filter called Iterative-Partitioning Filter (IPF), is
named SMOTE-IPF. SMOTE-IPF is considered to overcome the problems produced by
noisy and borderline examples in imbalanced datasets as presented and
explained in (Sáez et al. [2015]).

SMOTE algorithm ignores data distribution and density information which is
significant in order to synthesize minority examples correctly. The SMOBD
algorithm tackles the aforementioned issues and introduces the effective
elimination of the influence of noise (Cao & Wang [2011]). In addition, the
Assembled-SMOTE method implements oversampling by taking into consideration
data distribution information in order to avoid the occurrence of overlapping
between classes (Zhou et al. [2013]).

Other methods, such as the Lee algorithm, generate synthetic samples and
decide whether to reject or accept a generated sample by considering their
location, i.e. the nearest neighbors of a synthetic sample (Lee et al.
[2015]). Batista et al. [2004] proposed an oversampling method call SMOTE-
TomekLinks, which is basically a SMOTE algorithm with the additional
application of Tomek links to the oversampled training set as a data cleaning
technique. Thus, the aforementioned technique improves the oversampling
accuracy by removing examples from both classes. Tomek links are introduced in
(Tomek [1976]) and can be defined as follows:

> Given two examples $E_{i}$ and $E_{j}$ belonging to different classes, and
> $d(E_{i},E_{j})$ is the distance between $E_{i}$ and $E_{j}$. A
> $(E_{i},E_{j})$ pair is called a Tomek link if there is not an example
> $E_{l}$, such that $d(E_{i},E_{l})<d(E_{i},E_{j})$ or
> $d(E_{j},E_{l})<d(E_{i},E_{j})$.

Furthermore, the CCR algorithm combines cleaning the decision border around
minority objects with guided synthetic oversampling to properly detect the
minority class examples (Koziarski & Wozniak [2017]). Moreover, the G-SMOTE
method introduced hybrid oversampling, by the notion of doing partially
sampling guided by the hidden patterns obtained from minority class and
randomization. Highly skewed data distributions are handled by simultaneous
over and undersampling (Sandhan & Choi [2014]). LVQ-SMOTE uses a combination
of the SMOTE ovesampler and the feature codebooks obtained by the learning
vector quantization in order to generate synthetic samples which occupy more
feature space than the other SMOTE variants (Nakamura et al. [2013]). Also, a
comparable performance can be obtained using the Polynom-fit SMOTE, which
oversamples the minority class using polynomial fitting functions (Gazzah &
Amara [2008]).

Moreover, the research study of (Stefanowski & Wilk [2008]) proposed a method
for selective pre-processing, called SPIDER that combines filtering and
oversampling of imbalanced data. Also, SWIM algorithm generates synthetic
samples that have the same Mahalanobis distance from the majority class as the
currently known minority data samples (Sharma et al. [2018]).

In this study, we examine and experiment with the sixteen resamplers explained
above, namely Cluster Centroids, Cluster Representatives, SMOTE, ProWSyn,
SMOTE-IPF, SWIM, SMOBD, Lee, ADASYN, Assembled-SMOTE, SMOTE-TomekLinks, CCR,
G-SMOTE, LVQ-SMOTE, Polynom-fit-SMOTE and SPIDER.

###  3.7 Models

This section describes the machine learning models trained and evaluated in
the rockfall detection task. In addition, we provide basic information
regarding the models utilized. However, a complete explanation and analysis of
the algorithmic aspects of each model is out of the scope of this paper. In
the following, there is a list containing all the models examined including
their hyper-parameters. For clarification purposes, we denote in parenthesis
the abbreviation for each model name to be used in this analysis.

Linear Discriminant Analysis (LDA) is a linear decision boundary classifier,
whose boundary is generated by fitting class conditional densities to the
input data. This particular model fits a Gaussian density to each class. All
classes are assumed to share the same covariance matrix (Hastie et al.
[2009]). We experiment with solvers such as Singular Value Decomposition
(SVD), Least Squares Solution (LSQR) and Eigenvalue Decomposition (EIGEN), and
with automatic or no shrinkage parameters.

Quadratic Discriminant Analysis (QDA) is a classifier that classifies objects
based on a quadratic decision boundary that is generated by fitting class
conditional densities to the input data. This particular model, like the
Linear Discriminant Analysis, fits a Gaussian density to each class. All
classes are assumed to share the same covariance matrix (Hastie et al.
[2009]). We utilize the default configuration of this classifier.

K-Nearest Neighbors Classifier (KNN) is a memory-based algorithm. Naively,
given a query point $x_{0}$, we find the $k$ training points
$x_{(r)},r=1,...,k$ closest in distance to $x_{0}$, and then classify it using
a majority vote policy among the $k$ neighbors (Hastie et al. [2009]). We
experiment using various numbers of nearest neighbors, such as 1, 3, 5 and 9
neighbors.

Gaussian Naive Bayes (GNB) assumes that given a class $G=j$, the features
$X_{k}$ are independent: $f_{j}(X)=\prod_{k=1}^{p}f_{jk}(X_{k})$ (Hastie et
al. [2009]). Gaussian Naive Bayes supports continuous valued features and
models each as conforming to a Gaussian (normal) distribution. We utilize the
default parameters for this specific classifier.

Decision Tree Classifier (DT) partitions the feature space into a set of
rectangles, and then fits a simple model on each one. Tree-based classifiers
are conceptually simple yet powerful (Hastie et al. [2009]). For this
particular classifier, we experiment with two strategies for splitting each
node, namely the best split and random split. Regarding the criterion for
splitting or alternatively the function to measure the quality of a split, we
experiment with Gini impurity (D’Ambrosio & Tutore [2011]) and information
gain metrics.

AdaBoost Classifier (AdaB) is a meta-estimator that initially fits a
classifier on the original input data and then fits additional copies of the
classifier on the same data but conditionally. It fits the additional copies
where the weights of incorrectly classified instances are adjusted such that
subsequent classifiers focus more on difficult cases (Freund & Schapire
[1997], Zhu et al. [2009]). Multiple numbers of estimators are investigated,
such as 10, 50, 100 and 500. The base estimator is chosen to be a simple
decision tree classifier with a max tree depth of 5.

Random Forest Classifier (RF) is a meta estimator that fits multiple decision
tree classifiers on various sub-samples of the original input data and then
uses averaging methods to improve the predictive accuracy and control over-
fitting issue (Hastie et al. [2009]). Multiple numbers of trees are
investigated, such as 10, 100, 500, 1000. Regarding the function to measure
the quality of a split of a tree-node, we investigate the Gini impurity and
the information gain.

Support Vector Classifier (SVC) algorithm constructs a hyperplane or set of
hyperplanes in a high-dimensional space, which can be used for classification.
Support vectors define the margins of the hyperplanes and are found after an
optimization process involving an objective function regularized by an error
term and a constraint. The decision boundaries used by Support Vector Machines
(SVMs) can be linear or non-linear using kernel functions. Different kernel
functions can be specified for the decision function (Awad et al. [2015]). In
this particular case, we experiment with common kernels such as polynomial
with multiple distinct degree values ranging from 2 to 8, a radial-basis
function, and sigmoid. Regarding the regularization parameter, we have used
multiple values, such as 0.1, 1, 10 and 100.

Extra Trees Classifier (ET) is a meta estimator that fits multiple randomized
decision trees on various sub-samples of the original input data and uses
averaging methods to improve predictive accuracy and control over-fitting
issue (Geurts et al. [2006]). Different numbers of estimators are
investigated, namely 10, 100, 500 and 1000. Also, the function to measure the
quality of a split of a tree-node is set to be the Gini impurity (D’Ambrosio &
Tutore [2011]) or the information gain.

XGBoost Classifier (XGB) is a scalable end-to-end ensemble machine learning
framework. It is actually an implementation of gradient boosted decision trees
designed in such way as to achieve state-of-the-art results while remaining
fast and efficient (Chen & Guestrin [2016]). Different values of boosting
learning rate are examined, ranging from $10^{-1}$ to $10^{-4}$. In addition,
various numbers of gradient boosted trees are used such as 10, 50 and 100,
together with a variety of boosters such as gbtree, which is a version of a
regression tree as a weak learner, gblinear, which uses generalized linear
regression with L1 and L2 shrinkage, and dart, which drops trees in such way
as to reduce the over-fitting issue.

Multi-Layer Perceptron (MLP) Classifier is a class of feed-forward artificial
neural networks (Murtagh [1991]). Alternatively, the MLP classifier can be
considered to be a shallow deep neural network. The investigated hyper-
parameters are hidden layer sizes of 50, 100, 150 and 200 and activation
functions for the hidden layers, namely rectified linear unit (ReLu)
$f(x)=max(0,x)$, hyperbolic tangent (TanH) $f(x)=tanh(x)$, and logistic
sigmoid $f(x)=\frac{1}{1+exp(-x)}$. Besides, three optimizers are tested
namely LBFGS, which is an optimizer of the family of quasi-Newton methods, the
stochastic gradient descent (SGD) and Adam, which is a stochastic gradient-
based optimizer.

###  3.8 Feature Selection

In this phase of the analysis framework, see Fig 3 (d), the best selected and
properly parameterized models of each model variant accompanied with each one
of the three best resamplers are used as inputs in a grid search algorithm
aiming to select the exact number of features that achieve the highest
performance. The criterion used for the univariate feature selection is mutual
information between two random variables. Mutual information is a non-negative
value that measures the dependency between the variables (Kraskov et al.
[2004]). According to the aforementioned methodology, we keep only the $k$
highest scoring features, with $k$ ranging from 5 to 37 (all features) in the
Degotalls case study and from 5 to 31 (all features) in the Castellfollit case
study.

###  3.9 Pipeline Selection

The input of this stage consists of the multiple machine learning pipelines,
each properly parameterized with the best number of features, as shown in Fig
3 (e). These pipelines are statistically compared in order to conclude with
the best intelligent pipeline to be used in our prototype system. The
statistical comparison of the various pipelines is conducted by applying the
Friedman, a non-parametric statistical test, and Nemenyi, a post-hoc test as
described in (Demšar [2006]). We run every pipeline $n=10$ times using 10-fold
cross-validation. In each run, we rank the pipelines from $1$ to $n_{models}$.
By averaging the ranks table to get the averaged ranks for each pipeline, we
can conclude that two models are significantly different if their average
ranks differ by at least the critical difference. The critical difference is
computed with the following formula:

$CD=q_{\alpha}\sqrt{\frac{n_{models}*(n_{models}-1)}{6*n}},$ (2)

where $q_{\alpha}$ denotes the critical value of the two tailed Nemenyi test,
which depends on the chosen $\alpha$ level of statistical significance and the
number of models or in this case the pipelines, $n_{models}$.

###  3.10 Feature Importance

In this phase of our framework, see Fig 3 (h), we calculate and visualize the
feature importance of the input dataset. This step provides significant
insights in terms of the interpretability and explainability of the outcome
produced by the utilized algorithm. Thus, we utilize a generalized feature
importance measurement, the permutation feature importance as introduced and
described in (Breiman [2001]). The computation of permutation feature
importance is done by first calculating a baseline metric, defined by a
performance scoring function, and evaluated on a dataset $X$. Next, a feature
from the validation set is permuted and the metric is evaluated again.
Finally, the permutation importance is defined as the difference between the
baseline metric and the metric obtained from permutating the aforementioned
feature.

In our study, we utilize an extensive permutation feature importance test in
order to get robust results and insights regarding the most significant
features utilized by the classifier. Specifically, we use the most statistical
significant classifier as concluded from the computations in the Pipeline
Selection stage of our framework, depicted in Fig 3 (e), with a 10-fold cross
validation technique on input data and 100 feature permutations in each fold.
Then, we average all the values obtained and express them in percentage values
for interpretability purposes. In addition, we use K-means clustering
algorithm to group the feature importances obtained in order to get a better
view of the most important features and their neighbors.

##  4 Experimental Analysis

This section presents the experimental analysis of our designed intelligent
framework, based on well-known performance metrics. First, we explain the
accuracy metric and then we present evaluation tables for each stage of the
framework. Additionally, an ablation study is conducted in order to justify
the addition of each component to the final prototype implementation.

###  4.1 Performance Metrics

Our aim is to design our models for a classification task, in which a false
negative is usually more disastrous than a false positive for preliminary
rockfall detection. The nature of our problem is imbalanced as discussed in
Section 3.1. For this reason, we utilize a special metric called balanced
accuracy, as introduced and explained in (Brodersen et al. [2010]). The
authors of the aforementioned study define balanced accuracy as a performance
metric for imbalanced classification tasks. Balanced accuracy can be naively
defined as the average accuracy obtained on either class.

Based on a confusion matrix

actual
value |  | Prediction outcome |
---|---|---|---
| p | n | total
p′ |  True Positive (TP) |  False Negative (FN) | P′
n′ |  False Positive (FP) |  True Negative (TN) | N′
total | P | N |

the balanced accuracy is given by:

$Acc_{b}=\frac{1}{2}\left(\frac{TP}{TP+FN}+\frac{TN}{TN+FP}\right).$ (3)

###  4.2 Experimental Evaluation

In this section, we experimentally compare the various machine learning
models, which we presented and briefly analyzed in Section 3.7. We first
present the performance evaluation of the various resamplers in Table 1 and
then a thorough evaluation of all the models used in the model selection and
parameterization and feature selection phases in Table 2. The entire
evaluation process is based on the balanced accuracy metric, denoted in
equation 3, using a stratified 10-fold cross-validation.

#### Resamplers

Table 1 displays the average performance of each resampler in the Degotalls
(a) and Castellfollit (b) case studies. The depicted values are the average
performances of all machine learning models evaluated with a stratified
10-fold cross validation procedure, without prior parameterization, paired
with each resampler. We average the balanced accuracy metric values resampler-
wise, in order to select the three best resamplers to proceed to the model
selection and parameterization stage. In both case studies, the Cluster
Centroids resampler achieves the highest balanced accuracy while being the
most robust method. It achieves a $\overline{Acc_{b}}$ of 0.89 with 3.83%
error and 0.82 with 7.78% error in the Degotalls and Castellfollit case
studies respectively. Observing the two sub-tables in Table 1, the results
obtained in both cases do not seem to differ much in terms of ranking.

Table 1: $Acc_{b}$ metric summary for each resampling method.
$\overline{Acc_{b}}$ denotes the average value of $Acc_{b}$ of the 10-fold
cross validation. With dark gray we denote the best, with lighter gray the
second best and with pale gray the third best.

(a) Degotalls case study.

Resampling Method | $\overline{Acc_{b}}$ | Error (%)
---|---|---
Cluster Centroids | 0.89 | 3.83
ProWSyn | 0.87 | 5.34
SMOTE-IPF | 0.87 | 5.56
SWIM | 0.86 | 5.65
SMOBD | 0.86 | 6.29
Lee | 0.86 | 5.48
ADASYN | 0.86 | 6.28
Assembled-SMOTE | 0.86 | 6.16
SMOTE | 0.86 | 5.85
SMOTE-TomekLinks | 0.86 | 6.56
CCR | 0.85 | 7.91
G-SMOTE | 0.85 | 7.15
LVQ-SMOTE | 0.84 | 6.20
Polynom-fit-SMOTE | 0.83 | 7.56
SPIDER | 0.80 | 6.29
Cluster Representatives | 0.78 | 7.67

(b) Castellfollit case study.

Resampling Method | $\overline{Acc_{b}}$ | Error (%)
---|---|---
Cluster Centroids | 0.82 | 7.78
ProWSyn | 0.71 | 16.94
CCR | 0.70 | 15.88
SWIM | 0.70 | 14.90
Lee | 0.68 | 17.52
SMOTE | 0.68 | 18.13
LVQ-SMOTE | 0.68 | 18.33
ADASYN | 0.68 | 18.50
SMOBD | 0.68 | 18.86
SMOTE-IPF | 0.68 | 17.76
SMOTE-TomekLinks | 0.68 | 17.33
Assembled-SMOTE | 0.67 | 18.08
G_SMOTE | 0.66 | 19.83
Polynom-fit-SMOTE | 0.65 | 18.00
SPIDER | 0.58 | 17.29
Cluster Representatives | 0.56 | 18.28

#### Model Selection

Table 2 shows the performance evaluation of the model selection and
parameterization phase in the Degotalls (a) and Castellfollit (b) case
studies. Specifically, it displays only the six best methods for each case
study. For clarification purposes, the full tables are included in Tables 6
and 7 respectively in the S2 Appendix. For the selection of the best hyper-
parameters of each model, a grid search algorithm is utilized using a
stratified 10-fold cross-validation procedure for each available combination
of parameters. Please note that the depicted values in the above-mentioned
table are the balanced accuracy measurements of another stratified 10-fold
cross-validation procedure on the dataset utilizing only the best properly
configured models that resulted from the aforementioned grid search. It is
clear that in the Degotalls case study, the XGBoost classifier parameterized
the best, accompanied by the SMOTE-IPF oversampler, noted as XGB-SMOTE_IPF in
Table 2 (a), and performed better than all the other models while remaining
robust. It achieves a $\overline{Acc_{b}}$ of 0.94 with a 3.83% error score.
Regarding the best hyper-parameters of the XGBoost classifier, a linear
booster, namely gblinear, with a learning rate of 0.1 with 50 estimators, was
chosen. On the other hand, in the Castellfollit case study, the Linear
Discriminant Analysis classifier paired with the ProWSyn oversampler, noted as
LDA-ProWSyn in Table 2 (b), appears to be the most robust technique in terms
of balanced accuracy, achieving a $\overline{Acc_{b}}$ of 0.93 with 0.36% of
error, using Eigenvalue Decomposition as the solver and with an automatic
shrinkage parameter.

#### Feature Selection

In addition, Table 2 also displays the performance results of the feature
selection phase in the Degotalls (a) and Castellfollit (b) case studies, in
which each properly parameterized model accompanied by a resampler is
evaluated using the best number of features over a stratified 10-fold cross-
validation process. In the first case study, the XGBoost classifier
accompanied by the SMOTE-IPF oversampler with 35 features seems to be the best
performing model, while in the second best model is the XGBoost classifier
paired with ProWSyn with 30 features, as displayed in Table 2.

Generalizing, we could say that in both case studies the top performing
methods are approximately the same utilizing around the same percentage of
features compared to the total available number of features in each case.

Table 2: $Acc_{b}$ metric summary. $\overline{Acc_{b}}$ denotes the average
value of $Acc_{b}$ of the 10-fold cross validation. With dark gray we denote
the best model in terms of accuracy and then robustness and with lighter gray
the second best for each phase. Features column display the number of features
utilized by each algorithm to achieve this score.

(a) Degotalls case study.

Method | Model Parameterization | Feature Selection
---|---|---
| $\overline{Acc_{b}}$ | Error (%) | $\overline{Acc_{b}}$ | Error (%) | Features
XGB-SMOTE_IPF | 0.94 | 3.83 | 0.95 | 3.78 | 35
MLP-SMOTE_IPF | 0.94 | 5.54 | 0.95 | 5.64 | 35
KNN-ClusterCentroids | 0.94 | 4.29 | 0.95 | 4.09 | 36
XGB-ProWSyn | 0.94 | 3.84 | 0.95 | 3.68 | 35
SVC-ProWSyn | 0.94 | 5.68 | 0.95 | 4.14 | 17
LDA-SMOTE_IPF | 0.94 | 4.13 | 0.95 | 4.07 | 31

(b) Castellfollit case study.

Method | Model Parameterization | Feature Selection
---|---|---
| $\overline{Acc_{b}}$ | Error (%) | $\overline{Acc_{b}}$ | Error (%) | Features
XGB-ProWSyn | 0.93 | 8.11 | 0.94 | 6.92 | 30
LDA-ProWSyn | 0.93 | 0.36 | 0.93 | 3.98 | 30
MLP-CCR | 0.93 | 8.21 | 0.93 | 8.18 | 30
LDA-CCR | 0.93 | 0.54 | 0.93 | 4.06 | 29
XGB-CCR | 0.92 | 8.11 | 0.92 | 8.17 | 30
SVC-ClusterCentroids | 0.91 | 8.18 | 0.91 | 8.26 | 30

#### Pipeline Selection

In Fig 5, there is an illustration of the Friedman and Nemenyi techniques,
which are computed as described in Section 3.9, for the Degotalls (a) and
Castellfollit (b) case studies. This process provides enough evidence to
identify which of the methods are more statistically significant than the
others. We use a critical distance corresponding to 95% statistical
significance. Please note that between two algorithmic approaches there is a
statistical significance when the lines do not overlap. The best algorithm is
the one with the minimum rank value.

Fig 5 (a) shows that the XGBoost classifier properly parameterized and paired
with the SMOTE-IPF oversampler with 35 features performs significantly better
than the majority of the machine learning pipelines examined. In addition, the
MLP classifier paired with the SMOTE-IPF undersampler with 35 features could
also be an acceptable solution.

Regarding the second case study in Fig 5 (b), the XGBoost classifier properly
parameterized and paired with the ProWSyn oversampler with 30 features is the
best machine learning pipeline by a significant margin. Furthermore, the MLP
classifier accompanied by the CCR oversampler with 30 features could also be
an alternative acceptable solution.

(a) Degotalls case study.

(b) Castellfollit case study.

Figure 5: Nemenyi test - scores of model pipelines.

#### Feature Importance

In Fig 6, there is a visualization of permutation feature importance
technique, as described and explained in Section 3.10. Please note that the
colorization refers to the distinct clusters obtained by the K-means
algorithm. For the calculation of the importance of feature in the Degotalls
case study, we use the XGBoost classifier paired with the SMOTE-IPF
oversampler with 35 features, which we concluded performed better than the
majority of all other methods tested, while for the Castellfollit case study,
we utilize the XGBoost classifier paired with the ProWSyn oversampler with 30
features.

Observing the sub-figures (a, b) in Fig 6, we can conclude that the most
important features for each case study appear to be the same. Moreover, the
highly important features for rockfall event detection are the coordinates of
the point clouds.

(a) Degotalls case study.

(b) Castellfollit case study.

Figure 6: Feature Importance. Orange color denotes high importance, blue and
red denote mid and low importance respectively.

###  4.3 Ablation Study

An ablation study is crucial for the development of an intelligent system,
because it helps to understand the contribution of each component to the
overall system. Table 3 portrays how the balanced accuracy metric is increased
with the addition of each proposed component in our analysis framework, in the
Degotalls (a) and Castellfollit (b) case studies. It shows the increment of
the aforementioned metric across all models ($\overline{Acc_{b}}$) and the
best performing model ($Acc_{b}^{best}$) in each phase. The baseline is
considered a state in which all the models are trained and evaluated without
prior resampling of data, parameterization and feature selection. It is clear
in both case studies that the addition of each component provides an
interesting performance increment, improving the accuracy and robustness of
the utilized method.

Table 3: Ablation study - results based on $Acc_{b}$ metric.

(a) Degotalls case study.

Method | $\overline{Acc_{b}}$ | Error (%) | $Acc_{b}^{best}$
---|---|---|---
Baseline | 0.78 | 7.65 | 0.84
+Resampling | 0.85 | 3.25 | 0.89
+Model Parameterization | 0.89 | 4.65 | 0.94
+Feature Selection | 0.91 | 4.45 | 0.95

(b) Castellfollit case study.

Method | $\overline{Acc_{b}}$ | Error (%) | $Acc_{b}^{best}$
---|---|---|---
Baseline | 0.56 | 18.04 | 0.80
+Resampling | 0.68 | 8.21 | 0.82
+Model Parameterization | 0.79 | 15.93 | 0.93
+Feature Selection | 0.82 | 11.75 | 0.94

###  4.4 Prototype Implementation

In this section, we present the prototype implementation of a web-based system
appropriate for rockfall detection. The final system utilizes the best
intelligent pipeline according to our performance evaluation.

In order to demonstrate the effectiveness of our intelligent system, we have
developed a simple application that detects whether the input point cloud data
are considered candidates for a rockfall event. The architecture of our
prototype system for visualizing the predictions can be found in Fig 7. In the
front-end, displayed in Fig 8, there is a simple but properly designed table,
which is used to source input feature values and a button that triggers the
initialization of the detection procedure. In the back-end, we normalize the
data entered by the users, if needed, and generate a prediction using our
intelligent framework, which is the pre-trained and loaded machine learning
pipeline. Finally, the results are displayed to the user in a fast and
accurate manner.

Figure 7: Rockfall Detection - Web Application Architecture. Figure 8:
Rockfall Detection - Web Application.

The web application itself can be found at this GitHub repository. In
addition, we have developed our models and conducted our experiments,
utilizing Python programming language. In particular, for building the models,
we mainly use the Scikit-learn Python library (Pedregosa et al. [2011]). For
visualization purposes, we use the Matplotlib library (Hunter [2007]).
Regarding the web application, we use Flask (The Pallets Projects [2010]), a
micro web framework.

##  5 Final Observations

In this section, we provide the final observations of our experiments and
analysis. In the particular case studies, we conclude that some of the
resampling methods and machine learning models examined could be used in order
to form intelligent pipelines to detect rockfall events. In the case of the
Degotalls mountain wall TLS data, we come to the conclusion that the XGBoost
classifier using a linear booster, namely gblinear, with a learning rate of
0.1 and 50 estimators utilizing 35 features of the input dataset accompanied
by the SMOTE-IPF resampler, achieved the best outcome, with a robust balanced
accuracy score of 95%. Furthermore, in the Castellfollit case study, we
conclude that the XGBoost classifier with a linear booster, a learning rate of
0.1 and 100 estimators utilizing 30 features accompanied by the ProWSyn
resampler achieved the best balanced accuracy score of 94%.

Intuitively, we could say that the second-order gradients and the advanced
regularization of the XGBoost algorithm during the learning phase, help to
identify better the data relations. In addition, using the gblinear booster,
the algorithm builds multiple regularized linear models to later incorporate
them in a generalized linear model with advanced regularization. It seems that
learning inner linear models and then additively producing a generalized
linear model facilitates the achievement of a higher accuracy than the other
algorithms in both case studies while remaining fast. Additionally, the
Castellfollit case study appears to be more difficult than the Degotalls case
study in terms of model learning, because of its data imbalance of 0.4%
compared to 1% in the Degotalls study. This can also be justified by the fact
that the XGBoost algorithm utilized twice as many estimators in Castellfollit
than Degotalls.

Moreover, the ablation study, portrayed in Table 3 justifies the need for each
component of our system. The increment in balanced accuracy metric seems to be
significant with the addition of the special framework stages, namely
resampling, model selection and parameterization and feature selection.
Specifically, the most significant addition appears to be the resampling
module of our system, displaying an increment in accuracy from the baseline of
approximately 9% and 21% in the Degotalls and Castellfollit case studies
respectively. Additionally, in general, an ablation study provides an
interesting view of how the developed intelligent system behaves by adding
additional components. This is a good practice to identify the performance
gains of a developed system.

In order to provide further evidence for our developed framework, we visualize
the predicted point cloud data along with the original data of the Degotalls
case study, as displayed in Fig 9. We chose to display the former
visualization because it includes four incorrectly classified rockfall events
representing roughly 6% of the total rockfall events in the dataset, as
depicted in Fig 4 (a). For clarification purposes, in the Castellfollit case
study, all rockfall events in the initial dataset were correctly classified by
the XGBoost classifier using 30 features paired with the ProWSyn resampler,
which is the best selected pipeline, as discussed above.

Figure 9: Point Clouds Visualization - Degotalls case study. We denote the
correctly and incorrectly classified instances with yellow and blue colors
respectively. We further highlight them in green and circle, the events that
are initially rockfall events but were incorrectly classified as not rockfall.

##  6 Conclusion and Future Work

Rockfall events are considered to be a great hazard in multiple regions across
the world. People working in the geology domain put a lot of effort into
identifying such events. However, rockfall event detection utilizing data from
remote sensors such as LiDAR or commercial cameras still need specialized
analysis tools that can improve the accuracy and efficiency of the distinction
of such events from other changes triggered by edge effects, data noise,
vegetation, etc. We believe that an intelligent decision assisting tool would
facilitate geologists work. However, the development of such a tool seems to
be a challenging task, due to the inherent nature of the data.

In this research, an extensive analysis of several algorithmic approaches to
the detection of the rockfall events has been presented. In particular, we
focus on the development of an end-to-end machine learning framework able to
analyze clustered point cloud data from a TLS device. In the presented
framework, due to the imbalanced nature of the rockfall event detection, we
implement various resampling methodologies, parameterize several machine
learning models and create intelligent pipelines to tackle the abovementioned
task.

We experimentally evaluate our framework on two case studies, involving data
from TLS measurements of a cliff at the Degotalls, Montserrat Massif in
Barcelona and the cliff at the Castellfollit de la Roca in Girona, both
located in Spain. We analytically present and analyze our design and
experimentally evaluate our approach. These efforts elucidate the elements of
the whole procedure needed to develop the backbone of an effective, machine
learning based, rockfall detection system. We reveal the possibility of
developing a practical system which is useful for rockfall detection.

Our study offers the opportunity to elucidate certain important issues and
provide specific scientific and technological solutions at the intersection of
geology and machine learning. There is significant potential for further
additional research regarding the identification of rockfalls. Additionally,
raw point cloud data carry an enormous amount of information and utilizing
such information effectively and efficiently remains a challenging task. For
this, a system utilizing raw point cloud data as input is under construction,
in which the proposed methodology is used combined with various added machine
learning and neural network based methodologies in order to detect rockfall
events directly from temporal point cloud data, showing good potential.
Analyzing raw point cloud data without any grouping techniques using
sophisticated deep learning architectures may bring advances in rockfall
detection, because of the complete utilization of the inner topological
features of the data.

## Acknowledgements

This project has received funding from the European Union’s Horizon 2020
research and innovation programme under the Marie Skłodowska-Curie grant
agreement No 860843. In addition, the authors want to acknowledge the support
from PROMONTEC (CGL2017-84720-R AEI/FEDER, UE) and SALTEC CGL2017-85532-P
(AEI/FEDER, UE) projects, founded by the Spanish MINEICO, and AGAUR project
2016 DI 069 (Agència de Gestió d’Ajuts Universitaris i de Recerca). Data from
Castellfollit de la Roca were acquired with the support of the Spanish
Ministry of Science and Education (pre-doctoral grant 2004-1852) and funded by
the Natural Park of the Garrotxa Volcanic Field (PNZVG) and the following
projects: MEC CGL2006-06596 (DALMASA), TopoIberia CSD2006-0004/Consolider-
Ingenio2010, MEC CGL2010-18609 (NUTESA). Data from Montserrat Massif were
funded by the Institut Cartogràfic i Geològic de Catalunya (ICGC). Anna Puig
and Maria Salamó also acknowledge Generalitat de Catalunya, for its support
under project 2017-SGR-341.

## References

  * Abellán et al. [2010] Abellán, A., Calvet, J., Vilaplana, J. M., & Blanchard, J. (2010).  Detection and spatial prediction of rockfalls by means of terrestrial laser scanner monitoring.  Geomorphology, 119, 162–171. doi:10.1016/j.geomorph.2010.03.016.
  * Abellán et al. [2009] Abellán, A., Jaboyedoff, M., Oppikofer, T., & Vilaplana, J. M. (2009).  Detection of millimetric deformation using a terrestrial laser scanner: experiment and application to a rockfall event.  Natural Hazards and Earth System Sciences, 9, 365–372. URL: https://nhess.copernicus.org/articles/9/365/2009/. doi:10.5194/nhess-9-365-2009.
  * Abellán et al. [2014] Abellán, A., Oppikofer, T., Jaboyedoff, M., Rosser, N. J., Lim, M., & Lato, M. J. (2014).  Terrestrial laser scanning of rock slope instabilities.  Earth Surface Processes and Landforms, 39, 80–97. URL: https://onlinelibrary.wiley.com/doi/full/10.1002/esp.3493https://onlinelibrary.wiley.com/doi/abs/10.1002/esp.3493%****␣manuscript.bbl␣Line␣50␣****https://onlinelibrary.wiley.com/doi/10.1002/esp.3493http://doi.wiley.com/10.1002/esp.3493. doi:10.1002/esp.3493.
  * Agliardi et al. [2001] Agliardi, F., Crosta, G., & Zanchi, A. (2001).  Structural constraints on deep-seated slope deformation kinematics.  Engineering Geology, 59, 83–102. URL: https://linkinghub.elsevier.com/retrieve/pii/S0013795200000661. doi:10.1016/S0013-7952(00)00066-1.
  * Awad et al. [2015] Awad, M., Khanna, R., Awad, M., & Khanna, R. (2015).  Support Vector Machines for Classification.  In Efficient Learning Machines (pp. 39–66).  Apress.  URL: https://link.springer.com/chapter/10.1007/978-1-4302-5990-9_3. doi:10.1007/978-1-4302-5990-9{\\_}3.
  * Bandura et al. [2019] Bandura, L., Halpert, A. D., & Zhang, Z. (2019).  Machine learning in the interpreter’s toolbox: Unsupervised, supervised, and deep learning applications.  In 2018 SEG International Exposition and Annual Meeting, SEG 2018 (pp. 4633–4637).  Society of Exploration Geophysicists.  URL: https://library.seg.org/doi/abs/10.1190/segam2018-2997015.1. doi:10.1190/segam2018-2997015.1.
  * Barua et al. [2013] Barua, S., Islam, M. M., & Murase, K. (2013).  ProWSyn: Proximity weighted synthetic oversampling technique for imbalanced data set learning.  In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (pp. 317–328).  Springer, Berlin, Heidelberg volume 7819 LNAI.  URL: https://link.springer.com/chapter/10.1007/978-3-642-37456-2_27. doi:10.1007/978-3-642-37456-2{\\_}27.
  * Batista et al. [2004] Batista, G. E. A. P. A., Prati, R. C., & Monard, M. C. (2004).  A study of the behavior of several methods for balancing machine learning training data.  ACM SIGKDD Explorations Newsletter, 6, 20–29. URL: https://dl.acm.org/doi/10.1145/1007730.1007735. doi:10.1145/1007730.1007735.
  * Bernsteiner et al. [2020] Bernsteiner, H., Broåová, N., Eischeid, I., Hamer, A., Haselberger, S., Huber, M., Kollert, A., Vandyk, T. M., & Pirotti, F. (2020).  Machine learning for classification of an eroding scarp surface using terrestrial photogrammetry with nir and rgb imagery.  In ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences (pp. 431–437).  Copernicus GmbH volume 5.  doi:10.5194/isprs-Annals-V-3-2020-431-2020.
  * Breiman [2001] Breiman, L. (2001).  Random forests.  Machine Learning, 45, 5–32. URL: https://link.springer.com/article/10.1023/A:1010933404324. doi:10.1023/A:1010933404324.
  * Brodersen et al. [2010] Brodersen, K. H., Ong, C. S., Stephan, K. E., & Buhmann, J. M. (2010).  The balanced accuracy and its posterior distribution.  In Proceedings - International Conference on Pattern Recognition (pp. 3121–3124).  doi:10.1109/ICPR.2010.764.
  * Bui et al. [2020] Bui, D. T., Tsangaratos, P., Nguyen, V. T., Liem, N. V., & Trinh, P. T. (2020).  Comparing the prediction performance of a Deep Learning Neural Network model with conventional machine learning models in landslide susceptibility assessment.  Catena, 188, 104426\. doi:10.1016/j.catena.2019.104426.
  * Cao & Wang [2011] Cao, Q., & Wang, S. (2011).  Applying over-sampling technique based on data density and cost-sensitive SVM to imbalanced learning.  In Proceedings - 2011 4th International Conference on Information Management, Innovation Management and Industrial Engineering, ICIII 2011 (pp. 543–548).  volume 2.  doi:10.1109/ICIII.2011.276.
  * Carrara [1983] Carrara, A. (1983).  Multivariate models for landslide hazard evaluation.  Journal of the International Association for Mathematical Geology, 15, 403–426. URL: https://link.springer.com/article/10.1007/BF01031290. doi:10.1007/BF01031290.
  * Chawla et al. [2002] Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002).  SMOTE: Synthetic minority over-sampling technique.  Journal of Artificial Intelligence Research, 16, 321–357. URL: https://www.jair.org/index.php/jair/article/view/10302. doi:10.1613/jair.953.
  * Chen et al. [2015] Chen, J., Zeng, Z., Jiang, P., & Tang, H. (2015).  Deformation prediction of landslide based on functional network.  Neurocomputing, 149, 151–157. URL: https://linkinghub.elsevier.com/retrieve/pii/S0925231214009497. doi:10.1016/j.neucom.2013.10.044.
  * Chen & Guestrin [2016] Chen, T., & Guestrin, C. (2016).  XGBoost: A scalable tree boosting system.  In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 785–794).  Association for Computing Machinery volume 13-17-August-2016.  URL: https://github.com/dmlc/xgboost. doi:10.1145/2939672.2939785.
  * Crosta & Agliardi [2003] Crosta, G. B., & Agliardi, F. (2003).  Failure forecast for large rock slides by surface displacement measurements.  Canadian Geotechnical Journal, 40, 176–191. URL: https://cdnsciencepub.com/doi/abs/10.1139/t02-085http://www.nrcresearchpress.com/doi/10.1139/t02-085. doi:10.1139/t02-085.
  * D’Ambrosio & Tutore [2011] D’Ambrosio, A., & Tutore, V. A. (2011).  Conditional classification trees by weighting the gini impurity measure.  In Studies in Classification, Data Analysis, and Knowledge Organization (pp. 273–280).  Springer, Berlin, Heidelberg.  URL: https://link.springer.com/chapter/10.1007/978-3-642-11363-5_31. doi:10.1007/978-3-642-11363-5{\\_}31.
  * Demšar [2006] Demšar, J. (2006).  Statistical Comparisons of Classifiers over Multiple Data Sets.  Technical Report.
  * DiFrancesco et al. [2020] DiFrancesco, P. M., Bonneau, D., & Hutchinson, D. J. (2020).  The implications of M3C2 projection diameter on 3D semi-automated rockfall extraction from sequential terrestrial laser scanning point clouds.  Remote Sensing, 12, 1885. URL: https://www.mdpi.com/2072-4292/12/11/1885. doi:10.3390/rs12111885.
  * Dramsch [2020] Dramsch, J. S. (2020).  70 years of machine learning in geoscience in review.  In Advances in Geophysics (pp. 1–55).  Academic Press Inc. volume 61.  URL: /pmc/articles/PMC7500415/?report=abstracthttps://www.ncbi.nlm.nih.gov/pmc/articles/PMC7500415/. doi:10.1016/bs.agph.2020.08.002.
  * Fanos & Pradhan [2019] Fanos, A. M., & Pradhan, B. (2019).  A Novel Hybrid Machine Learning-Based Model for Rockfall Source Identification in Presence of Other Landslide Types Using LiDAR and GIS.  Earth Systems and Environment, 3, 491–506. URL: https://link.springer.com/article/10.1007/s41748-019-00114-z. doi:10.1007/s41748-019-00114-z.
  * Fanos et al. [2020] Fanos, A. M., Pradhan, B., Alamri, A., & Lee, C.-W. (2020).  Machine Learning-Based and 3D Kinematic Models for Rockfall Hazard Assessment Using LiDAR Data and GIS.  Remote Sensing, 12, 1755. URL: https://www.mdpi.com/2072-4292/12/11/1755. doi:10.3390/rs12111755.
  * Fanos et al. [2018] Fanos, A. M., Pradhan, B., Mansor, S., Yusoff, Z. M., & Abdullah, A. F. b. (2018).  A hybrid model using machine learning methods and GIS for potential rockfall source identification from airborne laser scanning data.  Landslides, 15, 1833–1850. URL: https://link.springer.com/article/10.1007/s10346-018-0990-4. doi:10.1007/s10346-018-0990-4.
  * Fernández et al. [2018] Fernández, A., García, S., Herrera, F., & Chawla, N. V. (2018).  SMOTE for Learning from Imbalanced Data: Progress and Challenges, Marking the 15-year Anniversary.  URL: https://www.jair.org/index.php/jair/article/view/11192. doi:10.1613/jair.1.11192.
  * Freund & Schapire [1997] Freund, Y., & Schapire, R. E. (1997).  A Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting.  Journal of Computer and System Sciences, 55, 119–139. URL: https://linkinghub.elsevier.com/retrieve/pii/S002200009791504X. doi:10.1006/jcss.1997.1504.
  * Gazzah & Amara [2008] Gazzah, S., & Amara, N. E. B. (2008).  New oversampling approaches based on polynomial fitting for imbalanced data sets.  In DAS 2008 - Proceedings of the 8th IAPR International Workshop on Document Analysis Systems (pp. 677–684).  doi:10.1109/DAS.2008.74.
  * Geurts et al. [2006] Geurts, P., Ernst, D., & Wehenkel, L. (2006).  Extremely randomized trees.  Machine Learning, 63, 3–42. URL: https://link.springer.com/article/10.1007/s10994-006-6226-1. doi:10.1007/s10994-006-6226-1.
  * Ghorbanzadeh et al. [2019] Ghorbanzadeh, O., Blaschke, T., Gholamnia, K., Meena, S. R., Tiede, D., & Aryal, J. (2019).  Evaluation of different machine learning methods and deep-learning convolutional neural networks for landslide detection.  Remote Sensing, 11, 196. URL: http://www.mdpi.com/2072-4292/11/2/196. doi:10.3390/rs11020196.
  * Hastie et al. [2009] Hastie, T., Tibshirani, R., & Friedman, J. (2009).  The Elements of Statistical Learning.  Springer Series in Statistics.  New York, NY: Springer New York.  URL: http://link.springer.com/10.1007/978-0-387-84858-7. doi:10.1007/978-0-387-84858-7.
  * He et al. [2008] He, H., Bai, Y., Garcia, E. A., & Li, S. (2008).  ADASYN: Adaptive synthetic sampling approach for imbalanced learning.  In Proceedings of the International Joint Conference on Neural Networks (pp. 1322–1328).  doi:10.1109/IJCNN.2008.4633969.
  * Hemalatha et al. [2019] Hemalatha, T., Ramesh, M. V., & Rangan, V. P. (2019).  Effective and accelerated forewarning of landslides using wireless sensor networks and machine learning.  IEEE Sensors Journal, 19, 9964–9975. doi:10.1109/JSEN.2019.2928358.
  * Hernandez et al. [2013] Hernandez, J., Carrasco-Ochoa, J. A., & Martínez-Trinidad, J. F. (2013).  An empirical study of oversampling and undersampling for instance selection methods on imbalance datasets.  In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (pp. 262–269).  Springer, Berlin, Heidelberg volume 8258 LNCS.  URL: http://ccc.inaoep.mx. doi:10.1007/978-3-642-41822-8{\\_}33.
  * Hunter [2007] Hunter, J. D. (2007).  Matplotlib: A 2D Graphics Environment.  Computing in Science & Engineering, 9, 90–95. URL: http://ieeexplore.ieee.org/document/4160265/. doi:10.1109/MCSE.2007.55.
  * Huu Phuong [2020] Huu Phuong, N. T. (2020).  Lidar Point Cloud Classification Using Expectation Maximization Algorithm.  International Journal of Computer Science and Information Technology, 12, 1–13. doi:10.5121/ijcsit.2020.12201.
  * Jaboyedoff et al. [2012] Jaboyedoff, M., Oppikofer, T., Abellán, A., Derron, M.-H., Loye, A., Metzger, R., & Pedrazzini, A. (2012).  Use of LIDAR in landslide investigations: a review.  Natural Hazards, 61, 5–28. URL: https://link.springer.com/article/10.1007/s11069-010-9634-2http://link.springer.com/10.1007/s11069-010-9634-2. doi:10.1007/s11069-010-9634-2.
  * James & Robson [2012] James, M. R., & Robson, S. (2012).  Straightforward reconstruction of 3D surfaces and topography with a camera: Accuracy and geoscience application.  Journal of Geophysical Research: Earth Surface, 117. URL: https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/2011JF002289https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/2011JF002289https://agupubs.onlinelibrary.wiley.com/doi/10.1029/2011JF002289. doi:10.1029/2011JF002289.
  * Junsomboon & Phienthrakul [2017] Junsomboon, N., & Phienthrakul, T. (2017).  Combining over-sampling and under-sampling techniques for imbalance dataset.  In ACM International Conference Proceeding Series (pp. 243–247).  New York, New York, USA: Association for Computing Machinery volume Part F128357.  URL: http://dl.acm.org/citation.cfm?doid=3055635.3056643. doi:10.1145/3055635.3056643.
  * Karpatne et al. [2019] Karpatne, A., Ebert-Uphoff, I., Ravela, S., Babaie, H. A., & Kumar, V. (2019).  Machine Learning for the Geosciences: Challenges and Opportunities.  IEEE Transactions on Knowledge and Data Engineering, 31, 1544–1554. doi:10.1109/TKDE.2018.2861006.
  * Kong et al. [2020] Kong, D., Wu, F., & Saroglou, C. (2020).  Automatic identification and characterization of discontinuities in rock masses from 3D point clouds.  Engineering Geology, 265, 105442. doi:10.1016/j.enggeo.2019.105442.
  * Kovács [2019] Kovács, G. (2019).  An empirical comparison and evaluation of minority oversampling techniques on a large number of imbalanced datasets.  Applied Soft Computing Journal, 83, 105662. doi:10.1016/j.asoc.2019.105662.
  * Koziarski & Wozniak [2017] Koziarski, M., & Wozniak, M. (2017).  CCR: A combined cleaning and resampling algorithm for imbalanced data classification.  International Journal of Applied Mathematics and Computer Science, 27, 727–736. URL: https://content.sciendo.com/view/journals/amcs/27/4/article-p727.xml. doi:10.1515/amcs-2017-0050.
  * Kraskov et al. [2004] Kraskov, A., Stögbauer, H., & Grassberger, P. (2004).  Estimating mutual information.  Physical Review E - Statistical Physics, Plasmas, Fluids, and Related Interdisciplinary Topics, 69, 16. URL: https://journals.aps.org/pre/abstract/10.1103/PhysRevE.69.066138. doi:10.1103/PhysRevE.69.066138.
  * Lague et al. [2013] Lague, D., Brodu, N., & Leroux, J. (2013).  Accurate 3D comparison of complex topography with terrestrial laser scanner: Application to the Rangitikei canyon (N-Z).  ISPRS Journal of Photogrammetry and Remote Sensing, 82, 10–26. doi:10.1016/j.isprsjprs.2013.04.009.
  * Lee et al. [2015] Lee, J., Kim, N. R., & Lee, J. H. (2015).  An over-sampling technique with rejection for imbalanced class learning.  In ACM IMCOM 2015 - Proceedings (pp. 1–6).  New York, New York, USA: Association for Computing Machinery, Inc.  URL: http://dl.acm.org/citation.cfm?doid=2701126.2701181. doi:10.1145/2701126.2701181.
  * Lissak et al. [2020] Lissak, C., Bartsch, A., De Michele, M., Gomez, C., Maquaire, O., Raucoules, D., & Roulland, T. (2020).  Remote Sensing for Assessing Landslides and Associated Hazards.  URL: https://doi.org/10.1007/s10712-020-09609-1. doi:10.1007/s10712-020-09609-1.
  * Loghin et al. [2020] Loghin, A. M., Pfeifer, N., & Otepka-Schremmer, J. (2020).  Supervised Classification and Its Repeatability for Point Clouds from Dense Vhr Tri-Stereo Satellite Image Matching Using Machine Learning.  In ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences (pp. 525–532).  volume 5.  URL: https://doi.org/10.5194/isprs-annals-V-2-2020-525-2020. doi:10.5194/isprs-annals-V-2-2020-525-2020.
  * Mayr et al. [2017] Mayr, A., Rutzinger, M., Bremer, M., Oude Elberink, S., Stumpf, F., & Geitner, C. (2017).  Object-based classification of terrestrial laser scanning point clouds for landslide monitoring.  Photogrammetric Record, 32, 377–397. doi:10.1111/phor.12215.
  * Mayr et al. [2018] Mayr, A., Rutzinger, M., & Geitner, C. (2018).  Multitemporal analysis of objects in 3D point clouds for landslide monitoring.  In International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences - ISPRS Archives (pp. 691–697).  volume 42.  URL: https://doi.org/10.5194/isprs-archives-XLII-2-691-2018. doi:10.5194/isprs-archives-XLII-2-691-2018.
  * Micheletti et al. [2014] Micheletti, N., Foresti, L., Robert, S., Leuenberger, M., Pedrazzini, A., Jaboyedoff, M., & Kanevski, M. (2014).  Machine Learning Feature Selection Methods for Landslide Susceptibility Mapping.  Mathematical Geosciences, 46, 33–57. URL: https://link.springer.com/article/10.1007/s11004-013-9511-0http://link.springer.com/10.1007/s11004-013-9511-0. doi:10.1007/s11004-013-9511-0.
  * Murtagh [1991] Murtagh, F. (1991).  Multilayer perceptrons for classification and regression.  Neurocomputing, 2, 183–197. doi:10.1016/0925-2312(91)90023-5.
  * Nakamura et al. [2013] Nakamura, M., Kajiwara, Y., Otsuka, A., & Kimura, H. (2013).  LVQ-SMOTE - Learning Vector Quantization based Synthetic Minority Over-sampling Technique for biomedical data.  BioData Mining, 6, 16. URL: https://biodatamining.biomedcentral.com/articles/10.1186/1756-0381-6-16. doi:10.1186/1756-0381-6-16.
  * Oppikofer et al. [2009] Oppikofer, T., Jaboyedoff, M., Blikra, L., Derron, M.-H., & Metzger, R. (2009).  Characterization and monitoring of the Åknes rockslide using terrestrial laser scanning.  Natural Hazards and Earth System Sciences, 9, 1003–1019. URL: https://nhess.copernicus.org/articles/9/1003/2009/. doi:10.5194/nhess-9-1003-2009.
  * Pedregosa et al. [2011] Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., & Duchesnay, E. (2011).  Scikit-learn: Machine Learning in Python.  Journal of Machine Learning Research, 12, 2825–2830. URL: http://jmlr.org/papers/v12/pedregosa11a.html.
  * Pham et al. [2018] Pham, B. T., Shirzadi, A., Tien Bui, D., Prakash, I., & Dholakia, M. B. (2018).  A hybrid machine learning ensemble approach based on a Radial Basis Function neural network and Rotation Forest for landslide susceptibility modeling: A case study in the Himalayan area, India.  International Journal of Sediment Research, 33, 157–170. doi:10.1016/j.ijsrc.2017.09.008.
  * Prakash et al. [2020] Prakash, N., Manconi, A., & Loew, S. (2020).  Mapping landslides on EO data: Performance of deep learning models vs. Traditional machine learning models.  Remote Sensing, 12, 346. URL: https://www.mdpi.com/2072-4292/12/3/346. doi:10.3390/rs12030346.
  * Reichenbach et al. [2018] Reichenbach, P., Rossi, M., Malamud, B. D., Mihir, M., & Guzzetti, F. (2018).  A review of statistically-based landslide susceptibility models.  Earth-Science Reviews, 180, 60–91. URL: https://linkinghub.elsevier.com/retrieve/pii/S0012825217305652. doi:10.1016/j.earscirev.2018.03.001.
  * Robbins et al. [2021] Robbins, B., Stephens, I., & Marcuson, W. (2021).  Geotechnical Engineering.  In Encyclopedia of Geology (pp. 377–392).  Elsevier.  URL: https://linkinghub.elsevier.com/retrieve/pii/B9780124095489125084. doi:10.1016/B978-0-12-409548-9.12508-4.
  * Robiati et al. [2019] Robiati, Eyre, Vanneschi, Francioni, Venn, & Coggan (2019).  Application of Remote Sensing Data for Evaluation of Rockfall Potential within a Quarry Slope.  ISPRS International Journal of Geo-Information, 8, 367. URL: https://www.mdpi.com/2220-9964/8/9/367. doi:10.3390/ijgi8090367.
  * Sáez et al. [2015] Sáez, J. A., Luengo, J., Stefanowski, J., & Herrera, F. (2015).  SMOTE-IPF: Addressing the noisy and borderline examples problem in imbalanced classification by a re-sampling method with filtering.  Information Sciences, 291, 184–203. doi:10.1016/j.ins.2014.08.051.
  * Salvini et al. [2013] Salvini, R., Francioni, M., Riccucci, S., Bonciani, F., & Callegari, I. (2013).  Photogrammetry and laser scanning for analyzing slope stability and rock fall runout along the Domodossola-Iselle railway, the Italian Alps.  Geomorphology, 185, 110–122. doi:10.1016/j.geomorph.2012.12.020.
  * Sandhan & Choi [2014] Sandhan, T., & Choi, J. Y. (2014).  Handling imbalanced datasets by partially guided hybrid sampling for pattern recognition.  In Proceedings - International Conference on Pattern Recognition (pp. 1449–1453).  Institute of Electrical and Electronics Engineers Inc.  doi:10.1109/ICPR.2014.258.
  * Sharma et al. [2018] Sharma, S., Bellinger, C., Krawczyk, B., Zaiane, O., & Japkowicz, N. (2018).  Synthetic Oversampling with the Majority Class: A New Perspective on Handling Extreme Imbalance.  In Proceedings - IEEE International Conference on Data Mining, ICDM (pp. 447–456).  Institute of Electrical and Electronics Engineers Inc. volume 2018-Novem.  doi:10.1109/ICDM.2018.00060.
  * Shi et al. [2019] Shi, M., Sun, W., Zhang, T., Liu, Y., Wang, S., & Song, X. (2019).  Geology prediction based on operation data of tbm: Comparison between deep neural network and soft computing methods.  In 1st International Conference on Industrial Artificial Intelligence, IAI 2019.  Institute of Electrical and Electronics Engineers Inc.  URL: http://arxiv.org/abs/1809.06688http://dx.doi.org/10.1109/ICIAI.2019.8850794. doi:10.1109/ICIAI.2019.8850794.
  * Singh & Singh [2019] Singh, D., & Singh, B. (2019).  Investigating the impact of data normalization on classification performance.  Applied Soft Computing Journal, 97, 105524. doi:10.1016/j.asoc.2019.105524.
  * Stefanowski & Wilk [2008] Stefanowski, J., & Wilk, S. (2008).  Selective pre-processing of imbalanced data for improving classification performance.  In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (pp. 283–292).  Springer, Berlin, Heidelberg volume 5182 LNCS.  URL: https://link.springer.com/chapter/10.1007/978-3-540-85836-2_27. doi:10.1007/978-3-540-85836-2{\\_}27.
  * Stumpf & Kerle [2011] Stumpf, A., & Kerle, N. (2011).  Object-oriented mapping of landslides using Random Forests.  Remote Sensing of Environment, 115, 2564–2577. URL: https://linkinghub.elsevier.com/retrieve/pii/S0034425711001969. doi:10.1016/j.rse.2011.05.013.
  * The Pallets Projects [2010] The Pallets Projects (2010).  Flask — The Pallets Projects.  URL: https://palletsprojects.com/p/flask/.
  * Tomek [1976] Tomek, I. (1976).  Two Modifications of CNN.  IEEE Transactions on Systems, Man, and Cybernetics, SMC-6, 769–772. URL: http://ieeexplore.ieee.org/document/4309452/. doi:10.1109/TSMC.1976.4309452.
  * Wang et al. [2020a] Wang, H., Zhang, L., Yin, K., Luo, H., & Li, J. (2020a).  Landslide identification using machine learning.  Geoscience Frontiers, . doi:10.1016/j.gsf.2020.02.012.
  * Wang et al. [2020b] Wang, Y., Huang, J., & Tang, H. (2020b).  Automatic identification of the critical slip surface of slopes.  Engineering Geology, 273, 105672. doi:10.1016/j.enggeo.2020.105672.
  * Wang & Jia [2019] Wang, Z., & Jia, K. (2019).  Frustum ConvNet: Sliding Frustums to Aggregate Local Point-Wise Features for Amodal.  In IEEE International Conference on Intelligent Robots and Systems (pp. 1742–1749).  URL: http://arxiv.org/abs/1903.01864. doi:10.1109/IROS40897.2019.8968513.
  * Weidner et al. [2019] Weidner, L., Walton, G., & Kromer, R. (2019).  Classification methods for point clouds in rock slope monitoring: A novel machine learning approach and comparative analysis.  Engineering Geology, 263. doi:10.1016/j.enggeo.2019.105326.
  * Weidner et al. [2020a] Weidner, L., Walton, G., & Kromer, R. (2020a).  Automated Rock Slope Material Classification Using Machine Learning.  In 54th U.S. Rock Mechanics/Geomechanics Symposium.  American Rock Mechanics Association.
  * Weidner et al. [2020b] Weidner, L., Walton, G., & Kromer, R. (2020b).  Generalization considerations and solutions for point cloud hillslope classifiers.  Geomorphology, 354, 107039. doi:10.1016/j.geomorph.2020.107039.
  * Williams et al. [2011] Williams, R., Brasington, J., Vericat, D., Hicks, M., Labrosse, F., & Neal, M. (2011).  Monitoring braided river change using terrestrial laser scanning and optical bathymetric mapping.  In Developments in Earth Surface Processes (pp. 507–532).  Elsevier B.V. volume 15.  doi:10.1016/B978-0-444-53446-0.00020-3.
  * Xiao et al. [2018] Xiao, L., Zhang, Y., & Peng, G. (2018).  Landslide susceptibility assessment using integrated deep learning algorithm along the china-nepal highway.  Sensors (Switzerland), 18. URL: /pmc/articles/PMC6308679/?report=abstracthttps://www.ncbi.nlm.nih.gov/pmc/articles/PMC6308679/https://pubmed.ncbi.nlm.nih.gov/30558225/. doi:10.3390/s18124436.
  * Yen & Lee [2009] Yen, S. J., & Lee, Y. S. (2009).  Cluster-based under-sampling approaches for imbalanced data distributions.  Expert Systems with Applications, 36, 5718–5727. doi:10.1016/j.eswa.2008.06.108.
  * Zhao et al. [2020] Zhao, L., Wu, X., Niu, R., Wang, Y., & Zhang, K. (2020).  Using the rotation and random forest models of ensemble learning to predict landslide susceptibility.  Geomatics, Natural Hazards and Risk, 11, 1542–1564. URL: https://www.tandfonline.com/doi/full/10.1080/19475705.2020.1803421. doi:10.1080/19475705.2020.1803421.
  * Zhou et al. [2013] Zhou, B., Yang, C., Guo, H., & Hu, J. (2013).  A quasi-linear SVM combined with assembled SMOTE for imbalanced data classification.  In Proceedings of the International Joint Conference on Neural Networks.  doi:10.1109/IJCNN.2013.6707035.
  * Zhu et al. [2009] Zhu, J., Zou, H., Rosset, S., & Hastie, T. (2009).  Multi-class AdaBoost.  Technical Report.

## Supporting Information

##### S1 Appendix

For clarification purposes, we present the full names of the abbreviations
used for the data features considered in our study.

x, y, z: The coordinates of the cluster in Cartesian format.

n_points: Number of points in the cluster.

n_order: Number of the order of the cluster.

volume: Total volume of the cluster.

positive_volume: Positive volume of the cluster, volume of material moving
away from the observer (backward motion).

negative_volume: Negative volume, volume of material displaced toward the
observer (advance motion).

area: The area of the surface orthogonal to the TLS.

class_n_mean: Mean of the points classified as $n$. 0 is noise, 1 is advance
and 2 is waste of volume (associated to rockfall).

class_n_sigma: Standard deviation of the points classified as $n$. 0 is noise,
1 is advance and 2 is waste of volume (associated to rockfall).

class_all: Mean of the points classified as any of the classes.

n_orientations_o: Number of different orientations in the cluster at time $t$.

Intensity_mean_o: TLS intensity reflection mean at time $t$.

Intensity_sigma_o: TLS intensity reflection standard deviation at time $t$.

Intensity_mean_d: TLS intensity reflection mean at time $t+1$.

Intensity_sigma_d: TLS intensity reflection standard deviation at time $t+1$.

correlation_i: Correlation index between difference and distance between
points.

orientation_mean_o: Orientation dispersion of reference points.

slope_mean_o: Average slope of the reference points.

orientation_mean_d: Orientation dispersion of compared points.

slope_mean_d: Average slope of the compared points.

coplanararity_i_mean_o: Average of the coplanarity indices of the reference
points.

coplanararity_i_sigma_o: Variance of coplanarity of reference points.

colinearity_i_mean_o: Average of the collinearity Indices of the reference
points.

colinearity_i_sigma_o: Variance of collinearity of reference points.

coplanararity_i_mean_d: Average of the coplanarity indices of the compared
points.

coplanararity_i_sigma_d: Variance of coplanarity with comparative points.

colinearity_i_mean_d: Average of the collinearity indices of the compared
points.

colinearity_i_sigma_d: Variance of collinearity of compared points.

angles_mean: Mean of the angles between reference and compared point.

angles_sigma: Variance of angles between reference and comparative points.

##### S2 Appendix

In this section, we present the long tables including all the statistics and
results, considered in our study.

Feature | mean | std | min | 25% | 50% | 75% | max
---|---|---|---|---|---|---|---
x | 13.08 | 37.21 | -56.00 | -20.10 | 12.52 | 45.39 | 141.47
y | 229.02 | 28.26 | 170.07 | 205.60 | 225.09 | 250.26 | 320.02
z | 32.89 | 31.10 | -24.31 | 6.64 | 31.21 | 54.33 | 110.87
n_points | 16246.50 | 9036.31 | 16.00 | 8497.00 | 16302.50 | 24167.50 | 31700.00
n_order | 52.23 | 153.54 | 10.00 | 13.00 | 20.00 | 40.00 | 4702.00
volume | 0.08 | 0.27 | 0.00 | 0.01 | 0.02 | 0.06 | 9.05
positive_volume | 0.03 | 0.14 | 0.00 | 0.00 | 0.01 | 0.02 | 5.18
negative_volume | 0.05 | 0.16 | 0.00 | 0.00 | 0.01 | 0.04 | 3.86
area | 0.13 | 0.38 | 0.00 | 0.02 | 0.05 | 0.12 | 13.37
class_1_mean | 1.40 | 0.28 | 1.00 | 1.17 | 1.36 | 1.60 | 2.00
class_1_sigma | 0.37 | 0.16 | 0.00 | 0.30 | 0.43 | 0.49 | 0.50
class_2_mean | 25.65 | 21.41 | 0.00 | 8.92 | 19.25 | 39.08 | 88.60
class_2_sigma | 15.72 | 6.77 | 0.00 | 12.05 | 16.42 | 20.42 | 37.15
class_3_mean | 45.79 | 27.60 | 0.00 | 23.83 | 44.85 | 66.74 | 100.00
class_3_sigma | 21.06 | 10.11 | 0.00 | 14.05 | 22.15 | 28.87 | 46.04
class_4_mean | 28.54 | 23.30 | 0.00 | 10.00 | 23.39 | 41.95 | 100.00
class_4_sigma | 19.68 | 10.87 | 0.00 | 11.21 | 20.77 | 28.33 | 46.74
n_orientations_o | 41.54 | 26.04 | 8.02 | 21.44 | 26.66 | 67.40 | 133.70
intensity_mean_o | 102.09 | 49.64 | 20.94 | 63.27 | 82.51 | 137.15 | 251.53
intensity_sigma_o | 26.85 | 10.77 | 6.28 | 18.51 | 24.63 | 33.61 | 84.52
intensity_mean_d | 99.32 | 50.42 | 21.60 | 58.90 | 79.29 | 138.03 | 254.17
intensity_sigma_d | 25.96 | 11.44 | 1.86 | 17.11 | 24.00 | 33.63 | 94.14
correlation_i | -0.37 | 0.26 | -0.97 | -0.56 | -0.38 | -0.19 | 0.69
orientation_mean_o | 171.08 | 84.41 | 0.21 | 119.70 | 179.36 | 216.08 | 359.65
slope_mean_o | 50.27 | 21.07 | 0.24 | 37.27 | 53.62 | 66.03 | 89.99
orientation_mean_d | 174.20 | 83.88 | 0.01 | 122.05 | 181.33 | 219.56 | 359.82
slope_mean_d | 48.40 | 22.19 | 0.46 | 32.62 | 51.29 | 65.64 | 89.93
coplanararity_i_mean_o | 1.97 | 0.99 | 0.42 | 1.24 | 1.67 | 2.49 | 6.39
coplanararity_i_sigma_o | 0.29 | 0.15 | 0.02 | 0.19 | 0.26 | 0.35 | 1.95
colinearity_i_mean_o | 0.82 | 0.97 | 0.01 | 0.19 | 0.52 | 1.08 | 13.74
colinearity_i_sigma_o | 0.82 | 1.59 | 0.00 | 0.10 | 0.32 | 0.92 | 40.19
coplanararity_i_mean_d | 2.10 | 1.01 | 0.51 | 1.35 | 1.79 | 2.62 | 6.28
coplanararity_i_sigma_d | 0.32 | 0.17 | 0.01 | 0.20 | 0.29 | 0.40 | 2.04
colinearity_i_mean_d | 0.80 | 0.95 | 0.01 | 0.20 | 0.55 | 1.07 | 19.31
colinearity_i_sigma_d | 0.79 | 1.78 | 0.00 | 0.10 | 0.34 | 0.87 | 54.60
angles_mean | 0.36 | 0.30 | 0.00 | 0.10 | 0.29 | 0.54 | 1.53
angles_sigma | 0.19 | 0.15 | 0.00 | 0.05 | 0.17 | 0.31 | 0.66

Table 4: Summary data statistics - Degotalls case study. The 25%, 50% and 75%
denote the 25th, 50th, and 75th percentiles respectively.

Feature | mean | std | min | 25% | 50% | 75% | max
---|---|---|---|---|---|---|---
x | 61.85 | 53.80 | -50.94 | 18.06 | 65.13 | 109.43 | 153.41
y | 181.01 | 29.96 | 116.34 | 155.68 | 196.49 | 205.79 | 230.92
z | 24.31 | 11.93 | -3.58 | 14.21 | 23.33 | 34.12 | 52.67
n_points | 24287.94 | 13713.35 | 14.00 | 12297.50 | 24464.00 | 35866.50 | 47950.00
n_order | 80.41 | 1673.35 | 7.00 | 8.00 | 12.00 | 24.00 | 138158.00
area | 0.16 | 0.91 | 0.00 | 0.01 | 0.02 | 0.09 | 38.21
class_1_mean | 1.45 | 0.39 | 1.00 | 1.05 | 1.40 | 1.86 | 2.00
class_1_sigma | 0.24 | 0.20 | 0.00 | 0.00 | 0.29 | 0.44 | 0.50
class_2_mean | 40.06 | 25.75 | 0.00 | 18.64 | 39.43 | 61.00 | 91.86
class_2_sigma | 16.15 | 7.97 | 0.00 | 10.43 | 17.18 | 22.44 | 39.50
class_3_mean | 32.28 | 28.71 | 0.00 | 4.43 | 26.57 | 54.98 | 100.00
class_3_sigma | 14.94 | 11.04 | 0.00 | 5.08 | 14.71 | 23.35 | 46.47
class_4_mean | 27.65 | 29.43 | 0.00 | 1.26 | 16.56 | 48.65 | 100.00
class_4_sigma | 13.66 | 11.73 | 0.00 | 2.52 | 11.87 | 23.04 | 47.50
intensity_mean_o | 118.27 | 46.45 | 0.00 | 85.50 | 115.44 | 146.50 | 255.00
intensity_sigma_o | 31.69 | 14.75 | 0.00 | 20.66 | 29.90 | 40.71 | 106.77
intensity_mean_d | 98.77 | 43.72 | 0.00 | 65.38 | 94.00 | 125.44 | 255.00
intensity_sigma_d | 34.47 | 16.99 | 0.00 | 22.02 | 32.79 | 44.83 | 114.96
correlation_i | -0.40 | 0.29 | -0.97 | -0.63 | -0.42 | -0.20 | 0.73
orientation_mean_o | 156.17 | 97.21 | 0.07 | 57.08 | 179.20 | 221.43 | 359.97
slope_mean_o | 55.38 | 26.81 | 0.16 | 34.86 | 64.08 | 78.22 | 89.88
coplanararity_i_mean_o | 2.43 | 0.89 | 0.51 | 1.78 | 2.39 | 2.94 | 7.29
coplanararity_i_sigma_o | 0.38 | 0.22 | 0.02 | 0.23 | 0.33 | 0.47 | 1.91
colinearity_i_mean_o | 0.52 | 0.82 | 0.01 | 0.09 | 0.22 | 0.63 | 14.93
colinearity_i_sigma_o | 0.51 | 1.39 | 0.00 | 0.05 | 0.14 | 0.47 | 46.18
coplanararity_i_mean_d | 2.57 | 0.86 | 0.00 | 2.04 | 2.54 | 3.06 | 10.42
coplanararity_i_sigma_d | 0.57 | 0.47 | 0.00 | 0.26 | 0.44 | 0.74 | 5.05
colinearity_i_mean_d | 0.47 | 0.66 | 0.00 | 0.09 | 0.24 | 0.63 | 12.65
colinearity_i_sigma_d | 0.49 | 1.04 | 0.00 | 0.05 | 0.16 | 0.54 | 29.23
angles_mean | 0.43 | 0.36 | 0.00 | 0.11 | 0.32 | 0.73 | 1.54
angles_sigma | 0.20 | 0.16 | 0.00 | 0.06 | 0.16 | 0.34 | 0.74

Table 5: Summary data statistics - Castellfollit case study. The 25%, 50% and 75% denote the 25th, 50th, and 75th percentiles respectively. Method | Model Parameterization | Feature Selection
---|---|---
| $\overline{Acc_{b}}$ | Error (%) | $\overline{Acc_{b}}$ | Error (%) | Features
XGB-SMOTE_IPF | 0.94 | 3.83 | 0.95 | 3.78 | 35
MLP-SMOTE_IPF | 0.94 | 5.54 | 0.95 | 5.64 | 35
KNN-ClusterCentroids | 0.94 | 4.29 | 0.95 | 4.09 | 36
XGB-ProWSyn | 0.94 | 3.84 | 0.95 | 3.68 | 35
SVC-ProWSyn | 0.94 | 5.68 | 0.95 | 4.14 | 17
LDA-SMOTE_IPF | 0.94 | 4.13 | 0.95 | 4.07 | 31
MLP-ProWSyn | 0.94 | 4.07 | 0.95 | 4.09 | 31
LDA-ProWSyn | 0.94 | 4.08 | 0.94 | 4.11 | 31
LDA-ClusterCentroids | 0.93 | 3.87 | 0.94 | 3.74 | 36
MLP-ClusterCentroids | 0.93 | 4.05 | 0.94 | 4.00 | 29
SVC-ClusterCentroids | 0.93 | 4.30 | 0.94 | 3.99 | 33
XGB-ClusterCentroids | 0.92 | 3.99 | 0.93 | 4.08 | 35
ET-ClusterCentroids | 0.92 | 4.01 | 0.93 | 3.94 | 32
KNN-ProWSyn | 0.92 | 5.19 | 0.93 | 5.31 | 28
SVC-SMOTE_IPF | 0.92 | 5.26 | 0.93 | 3.94 | 35
ET-ProWSyn | 0.90 | 7.44 | 0.91 | 7.31 | 31
RF-ClusterCentroids | 0.91 | 3.94 | 0.91 | 4.00 | 33
KNN-SMOTE_IPF | 0.91 | 5.21 | 0.91 | 7.27 | 26
AdaB-ProWSyn | 0.89 | 8.29 | 0.90 | 8.20 | 33
QDA-ClusterCentroids | 0.89 | 5.56 | 0.90 | 5.52 | 31
GNB-ClusterCentroids | 0.88 | 6.07 | 0.90 | 6.76 | 6
GNB-SMOTE_IPF | 0.88 | 7.02 | 0.90 | 5.53 | 6
GNB-ProWSyn | 0.87 | 7.84 | 0.89 | 5.56 | 6
RF-ProWSyn | 0.86 | 8.67 | 0.89 | 6.68 | 35
QDA-ProWSyn | 0.85 | 9.20 | 0.89 | 6.57 | 5
AdaB-SMOTE_IPF | 0.85 | 9.07 | 0.88 | 9.27 | 36
AdaB-ClusterCentroids | 0.86 | 9.49 | 0.87 | 7.74 | 5
QDA-SMOTE_IPF | 0.85 | 9.23 | 0.87 | 5.90 | 7
DT-ClusterCentroids | 0.80 | 6.89 | 0.85 | 6.87 | 30
ET-SMOTE_IPF | 0.84 | 9.01 | 0.84 | 9.04 | 32
DT-SMOTE_IPF | 0.83 | 11.06 | 0.84 | 10.67 | 29
RF-SMOTE_IPF | 0.83 | 10.51 | 0.83 | 9.25 | 23
DT-ProWSyn | 0.80 | 9.92 | 0.80 | 10.06 | 32
Table 6: $Acc_{b}$ metric summary - Degotalls case study. $\overline{Acc_{b}}$ denotes the average value of $Acc_{b}$ of the 10-fold cross validation. Features column display the number of features utilized by each algorithm to achieve this score. Method | Model Parameterization | Feature Selection
---|---|---
| $\overline{Acc_{b}}$ | Error (%) | $\overline{Acc_{b}}$ | Error (%) | Features
XGB-ProWSyn | 0.93 | 8.11 | 0.94 | 6.92 | 30
LDA-ProWSyn | 0.93 | 0.36 | 0.93 | 3.98 | 30
MLP-CCR | 0.93 | 8.21 | 0.93 | 8.18 | 30
XGB-CCR | 0.92 | 8.11 | 0.92 | 8.17 | 30
SVC-ClusterCentroids | 0.91 | 8.18 | 0.91 | 8.26 | 30
LDA-CCR | 0.93 | 0.54 | 0.93 | 4.06 | 29
SVC-CCR | 0.90 | 12.93 | 0.90 | 12.89 | 30
MLP-ProWSyn | 0.87 | 11.45 | 0.90 | 9.38 | 25
SVC-ProWSyn | 0.90 | 7.13 | 0.90 | 7.10 | 30
KNN-ProWSyn | 0.90 | 12.75 | 0.90 | 12.90 | 30
LDA-ClusterCentroids | 0.89 | 8.23 | 0.89 | 8.25 | 28
ET-ClusterCentroids | 0.89 | 4.27 | 0.89 | 5.67 | 30
MLP-ClusterCentroids | 0.90 | 8.84 | 0.90 | 9.75 | 29
GNB-CCR | 0.88 | 5.75 | 0.88 | 5.90 | 30
QDA-ClusterCentroids | 0.75 | 12.30 | 0.87 | 5.95 | 19
XGB-ClusterCentroids | 0.90 | 4.37 | 0.90 | 9.66 | 25
KNN-ClusterCentroids | 0.86 | 9.68 | 0.86 | 10.15 | 26
GNB-ProWSyn | 0.86 | 11.35 | 0.86 | 11.42 | 28
RF-ClusterCentroids | 0.86 | 8.41 | 0.86 | 9.71 | 28
QDA-CCR | 0.72 | 8.66 | 0.83 | 6.30 | 11
QDA-ProWSyn | 0.51 | 7.35 | 0.82 | 13.70 | 8
GNB-ClusterCentroids | 0.75 | 8.20 | 0.79 | 6.36 | 15
RF-ProWSyn | 0.66 | 16.78 | 0.75 | 16.99 | 8
DT-CCR | 0.64 | 16.36 | 0.74 | 15.81 | 23
ET-ProWSyn | 0.68 | 11.95 | 0.73 | 14.61 | 6
DT-ClusterCentroids | 0.72 | 11.02 | 0.72 | 16.14 | 22
AdaB-ClusterCentroids | 0.71 | 16.78 | 0.71 | 14.14 | 21
DT-ProWSyn | 0.61 | 18.15 | 0.71 | 21.42 | 7
AdaB-ProWSyn | 0.68 | 17.96 | 0.70 | 17.87 | 29
ET-CCR | 0.68 | 16.04 | 0.68 | 16.19 | 28
KNN-CCR | 0.63 | 16.35 | 0.67 | 8.45 | 28
AdaB-CCR | 0.62 | 16.74 | 0.64 | 15.62 | 27
RF-CCR | 0.57 | 14.47 | 0.59 | 16.67 | 30
Table 7: $Acc_{b}$ metric summary - Castellfollit case study.
$\overline{Acc_{b}}$ denotes the average value of $Acc_{b}$ of the 10-fold
cross validation. Features column display the number of features utilized by
each algorithm to achieve this score.