∎

11institutetext: V. Cerqueira (✉) 22institutetext: Faculdade de Engenharia da
Universidade do Porto, Porto, Portugal
LIACC, Porto, Portugal
22email<EMAIL_ADDRESS>33institutetext: Luís Torgo 44institutetext:
Dalhousie University, Halifax, Canada

# Exceedance Probability Forecasting via Regression: A Case Study of
Significant Wave Height Prediction

Vitor Cerqueira  Luis Torgo

(Received: date / Accepted: date)

###### Abstract

Significant wave height forecasting is a key problem in ocean data analytics.
This task affects several maritime operations, such as managing the passage of
vessels or estimating the energy production from waves. In this work, we focus
on the prediction of extreme values of significant wave height that can cause
coastal disasters. This task is framed as an exceedance probability
forecasting problem. Accordingly, we aim to estimate the probability that the
significant wave height will exceed a predefined critical threshold. This
problem is usually solved using a probabilistic binary classification model or
an ensemble of forecasts. Instead, we propose a novel approach based on point
forecasting. Computing both type of forecasts (binary probabilities and point
forecasts) can be useful for decision-makers. While a probabilistic binary
forecast streamlines information for end-users concerning exceedance events,
the point forecasts can provide additional insights into the upcoming future
dynamics. The proposed solution works by assuming that the point forecasts
follow a distribution with the location parameter equal to that forecast.
Then, we convert these point forecasts into exceedance probability estimates
using the cumulative distribution function. We carried out experiments using
data from a smart buoy placed on the coast of Halifax, Canada. The results
suggest that the proposed methodology is better than state-of-the-art
approaches for exceedance probability forecasting.

###### Keywords:

Exceedance Probability Forecasting Time Series Forecasting Time series
Significant Wave Height

##  1 Introduction

Significant wave height (SWH) forecasting is a key problem in ocean data
analytics, and several methods have been developed for tackling it
huang1998empirical ; duan2016hybrid ; ali2020near . Forecasting the ocean
conditions is valuable for multiple operations, such as vessel performance. A
decision support system based on these forecasts can optimize ship performance
in operations such as selecting the best routes, speed, or heading
guo2023evaluating . Moreover, the passage of vessels requires a minimum depth
of water for their movement. The occurrence of extreme values of SWH reduces
the depth of water and this minimum may not be met.

Another motivation for forecasting SWH is related to renewable energy, where
forecasts are used to estimate energy production. Predicting impending large
values of SWH is important for coastal disaster prevention and protection of
wave energy converters, which should be shut down to prevent their damage
li2012wave . In summary, accurate SWH predictions can increase the performance
of vessels, reduce costs, and improve the reliability of ports.

In this work, we focus on the prediction of large values of SWH. We frame this
problem as an exceedance probability forecasting task. Exceedance probability
forecasting denotes the process of estimating the probability that a time
series will exceed a predefined threshold in a predefined future period. This
task is usually relevant in domains where extreme values (i.e., the tail of
the distribution) are highly relevant, such as earthquakes, and hurricanes
kunreuther2002risk ; frangopol2014prognosis .

Figure 1 shows an hourly SWH time series. The red horizontal dashed line
represents the threshold above which a specified exceedance event occurs. The
objective is to predict these events early in time.

Figure 1: Significant wave height time series with hourly granularity. The red
horizontal dashed line represents the exceeding threshold.

Exceedance forecasting is typically formalized as a binary classification
problem, where the target variable denotes whether or not the threshold will
be exceeded. Ensemble-based forecasting (regression) approaches can also be
applied, in which the exceedance probability is estimated according to the
ratio of individual predictions that exceed the threshold. Both approaches
often model these events using lagged features, i.e., an auto-regressive type
of modelling taylor2017probabilistic .

In this paper, we propose a novel approach to exceedance probabilistic
forecasting. The method is based on the numeric point forecasts, for example,
produced by an auto-regressive model. The proposed method does not rely on an
ensemble-based forecasting model, though it could be one. Essentially, we
assume that a given forecast is modeled using a continuous distribution. In
this case, we resort to a Weibull distribution following previous works on
modeling SWH data muraleedharan2007modified . Then, we use the cumulative
distribution function (CDF) of this distribution to estimate the exceedance
probability. The point forecast represents the location parameter of the
distribution, while the remaining parameters (e.g. scale) are estimated using
the training data.

Using a forecasting model instead of a classifier to obtain exceedance
probability estimates is useful because of the additional information. Binary
probabilities concerning exceedance events streamline the information being
given to the end-user. Coupling this information with the numeric forecasts
for upcoming observations offers extra information. This extra information
might be valuable for understanding the data dynamics. For example, when the
data approaches a threshold value, the projected trajectory based on point
forecasts offers valuable insights into the upcoming changes in the variable.

We apply the methodology in a case study related to SWH forecasting. The
results suggest that coupling a forecasting model with the proposed mechanism
based on the CDF is better for exceedance probability forecasting relative to
a classification method or an ensemble-based approach.

In summary, the contributions of this work are the following:

  * •

A novel approach for estimating time series exceedance probability based on
point forecasting and the CDF;

  * •

The application of this method to estimate the exceedance probability of SWH,
which is important for managing maritime operations.

The code for the methods used in the paper is publicly
available111https://github.com/vcerqueira/exceedance_wave. The rest of the
paper is organized as follows. In Section 2, we overview the related work. In
Section 3, we define the predictive problem, followed by Section 4, where the
proposed method is formalized. The case study used in this work is described
in Section 5. The experiments carried out to validate the proposed method are
presented in Section 6. We discuss the results and conclude the paper in
Section 7.

##  2 Related Work

This section provides an overview of previous works related to this paper. We
start by reviewing the literature on SHW forecasting (Section 2.1). Then, we
describe approaches for probabilistic forecasting focusing on exceedance
probability estimation (Section 2.2).

###  2.1 Significant Wave Height Forecasting

Ali et al. ali2020near  analyze buoy data from eastern Australia. They apply
several regression approaches, including a model tree and multivariate
adaptive regression splines, to forecast short-term (30 minutes in advance)
values of SWH. They conclude that a hybrid model, optimized using a method
based on least squares, provided the best performance.

Duan et al. duan2016hybrid  model to forecast SWH time series several hours in
advance. Unlike a short-term forecast, a horizon of a few hours is important
to ensure the safety of maritime operations, e.g. passage of vessels. They
apply the empirical mode decomposition (huang1998empirical ) to decompose the
time series into several simpler components, which are then modeled using an
auto-regressive approach. This decomposition leads to better forecasting
performance relative to an auto-regressive approach without this process.

Shamshirband et al. shamshirband2020prediction  compared different regression
algorithms for significant wave height forecasting, including artificial
neural networks, support vector regression, and extreme learning machines
huang2006extreme . They concluded that all methods perform comparably, though
the extreme learning machine model showed a slight advantage.

Several authors attempt to apply different neural network architectures for
SWH predictions, including vanilla recurrent neural networks
sadeghifar2017coastal , feedforward neural networks londhe2006one ;
jain2007real , gated recurrent units wang2021forecasting , LSTMs
wang2021forecasting ; abdullah2022significant ; bethel4153300assessing , or
graph neural networks chen2021significant . A comparison of these
architectures is not available. However, there is a tendency for recurrent
architectures, which is expected due to their capabilities for modeling
sequential data. In our work, we use a deep feedforward neural network, among
other regression methods. We found that, in our case study, this architecture
performed better than a recurrent one. Some architectures based on a multi-
layer perceptron architecture show a competitive forecasting performance with
transformers and recurrent-based architectures challu2023nhits .

Dixit et al. dixit2015removing  also model SWH using a neural network.
However, instead of using an auto-regressive type of modeling, they apply a
discrete wavelet transform as a preprocessing step. They claim that this step
helps remove prediction lag, thereby improving forecasting performance for SWH
multistep ahead forecasting.

Fasuyi et al. fasuyi2020machine  and Kuppili kuppili2021forecasting  also
model smart buoy data to interpolate data across different buoys located in
distinct locations. When a buoy is shut down for maintenance, the data from
other nearby buoys can be used to interpolate the missing values of that buoy.

The works mentioned so far are similar to each other in the sense that all
attempt to forecast the future values or interpolate from near buoys, of SWH
time series using machine learning regression models. It is difficult to
compare the performance of models across different works due to the usage of
different data and performance estimation methods. However, larger forecasting
horizons lead to poorer forecasting performance, as expected.

###  2.2 Exceedance Probability Forecasting

Conveying the uncertainty around forecasts is critical for better decision-
making. In standard time series forecasting tasks, uncertainty quantification
is usually carried out with prediction intervals khosravi2011comprehensive ,
or probabilistic forecasting gneiting2014probabilistic . When modeling binary
events, it is common to tacke this problem using a classification approach
taylor2016using . An instance of a binary event is exceedance, which, as
mentioned before, refers to when a time series exceeds a predefined threshold.

Slud et al. slud1994partial  apply logistic regression to model the exceedance
probability of rainfall. In their model, the lagged observations of the time
series are used as predictor variables in an auto-regressive fashion.

Taylor et al. taylor2016using  use an exceedance probability approach for
financial risk management. They predict whether the returns of financial
assets will exceed a threshold. This is accomplished by using a method called
CARL (conditional auto-regressive logit), which is based on logistic
regression. Later, CARL was extended to handle problems involving multiple
thresholds by Taylor et al. taylor2017probabilistic . The extended model was
used to predict wind ramp events, where the tails on both sides of the
distribution are important to predict.

The alternative to using a classification approach to estimate the exceedance
probability is to use an ensemble of forecasts. Accordingly, the probability
is computed as the ratio of forecasts above the threshold. This type of
approach is commonly used in fields such as meteorology lewis2005roots , where
different forecasts are obtained by considering different initial conditions
ferro2007comparing . Toth et al. toth2003probability  provide a comprehensive
description of ensemble-based approaches listing their main properties.
Estimating the exceedance probability for different thresholds enables the
construction of exceedance probability curves kunreuther2002risk , which may
be helpful for better decision-making.

Exceedance probability forecasting is related to extreme value prediction. The
goal of forecasting extreme values is to accurately predict the numeric value
of extreme observations in time series. For example, Polson et al.
polson2020deep  forecast energy load spikes using a deep learning approach. On
the other hand, in exceedance probability, the goal is simplified to predict
whether or not a given threshold is exceeded, i.e., the magnitude of
exceedance is not a major concern.

##  3 Problem Definition

This section formalizes the predictive task addressed in this work. We start
by defining SWH forecasting problems (Section 3.1), followed by the
formalization of exceedance probability forecasting (Section 3.2).

###  3.1 Time Series Forecasting

A univariate time series represents a temporal sequence of values
$Y=\\{y_{1},y_{2},\dots,$ $y_{n}\\}$, where
$y_{i}\in\mathcal{Y}\subset\mathbb{R}$ is the value of $Y$ at time $i$ and $n$
is the length of $Y$. The general goal behind forecasting is to predict the
value of the upcoming observations of the time series,
$y_{n+1},\ldots,y_{n+h}$, given the past observations, where $h$ denotes the
forecasting horizon.

We formalize the problem of time series forecasting based on an auto-
regressive strategy. Accordingly, observations of a time series are modeled
based on their recent lags. We construct a set of observations of the form
($X$, $y$) using time delay embedding according to the Takens theorem
Takens1981 . In each observation, the value of $y_{i}$ is modelled based on
the most recent $q$ known values: $X_{i}=\\{y_{i-1},y_{i-2},\dots,y_{i-q}\\}$,
where $y_{i}\in\mathcal{Y}\subset\mathbb{R}$, which represents the observation
we want to predict, $X_{i}\in\mathcal{X}\subset\mathbb{R}^{q}$ represents the
$i$-th embedding vector. For each horizon, the goal is to build a regression
model $f$ that can be written as $y_{i}=f(X_{i},Z_{i})$, where $Z_{i}$
represents additional covariates known at the $i$-th instance. In effect, we
address the multi-step ahead forecasting problem with a direct method
taieb2012review . We build $h$ models, one for predicting each future step
$\\{1,\dots,h\\}$. The case study in this work is based on a multivariate time
series that is described in Section 5.1. In this case, the covariates $Z$ are
explanatory time series related to SWH, including wind speed or sea surface
temperature. Each of the explanatory time series is also represented at each
time step $i$ with an embedding vector based on the past recent values. This
leads to an auto-regressive distributed lags modeling technique
nkoro2016autoregressive .

###  3.2 Exceedance Probability Forecasting

The probability of exceedance $p_{i}$ denotes the probability that the value
of a time series will exceed a predefined threshold $\tau$ in a given instant
$i$. Typically, $p_{i}$ is modelled by resorting to a binary target variable
$b_{i}$, which can be formalized as follows:

$b_{i}=\begin{cases}1&\text{if }y_{i}\geq\tau,\\\
0&\text{otherwise}.\end{cases}$ (1)

Essentially $b_{i}$ takes the value of 1 if the threshold $\tau$ is exceeded
is a future period $i$, or 0 otherwise. This naturally leads to a
classification problem where the goal is to build a model $g$ of the form
$b_{i}=g(X_{i},Z_{i})$. Accordingly, $p_{i}$ represents the conditional
expectation of $b_{i}$ which we estimate based on Bernoulli density. Logistic
regression is commonly used to this effect taylor2016using .

Since the exceedance events are rare, the number of cases for the positive
class (i.e. when $b_{i}=1$) is significantly smaller than the number of cases
for the negative class ($b_{i}=0$). This leads to an imbalanced classification
problem, which hinders the learning process of algorithms branco2016survey .

The alternative to using a classification model is to use a forecasting
ensemble $F$ composed of a set of $M$ models:
$F=\\{f_{1},f_{2},\dots,f_{M}\\}$. These models can be combined for
forecasting using the uniformly weighted average as follows:

$\hat{y}_{i}=\frac{1}{M}\sum_{k=1}^{M}f_{k}(X_{i},Z_{i})$

We can also use the ensemble $F$ to estimate the exceedance probability
according to the ratio of ensemble members that forecast a value that exceeds
the threshold $\tau$:

$p_{i}=\frac{1}{M}\sum_{k=1}^{M}I(f_{k}(X_{i},Z_{i})\geq\tau)$

where $I$ denotes the indicator function. Hereafter, we will refer to this
approach as an ensemble-based direct method for exceedance probability.
Ensemble-based approaches are commonly used in some fields related to
environmental science, for example, meteorology or hydrology
toth2003probability .

##  4 Methodology

The goal of this paper is to tackle exceedance probability forecasting
problems using a case study concerning SWH prediction. Since large values of
SWH are important to anticipate, we design an approach to estimate the
probability of their occurrence. To our knowledge, this is the first attempt
at framing SWH forecasting as an exceedance probability estimation task. In
this section, we describe the proposed approach for estimating the exceedance
probability (Section 4.1). We also list the main advantages of the method
relative to state-of-the-art approaches (Section 4.2).

###  4.1 Exceedance Probability via the CDF

The predictive task is to estimate $p$, the probability that the SWH will
exceed a specified threshold $\tau$. We tackle this task using a forecasting
model that produces point forecasts. The point forecasts, in addition to the
exceedance probability estimates, can provide valuable information about the
dynamics of the SWH. In effect, building a model capable of both types of
forecasts is desirable for operators.

This work introduces a novel approach for exceedance probability forecasting
based on a time series forecasting model. The proposed solution works in two
main steps:

  * •

Building a forecasting model and use it to obtain point forecasts

  * •

Converting point forecasts into exceedance probability estimates using the CDF

####  4.1.1 Building a forecasting model

We build a forecasting model $f$ using auto-regression as formalized in
Section 3.1. The value of future observations are modeled based on their past
lags. However, it is important to remark that the proposed methodology is
agnostic to the underlying forecasting model.

The proposed method leverages the numeric predictions produced by the
forecasting model regarding the value of future observations. Here, we denote
the point forecast for the $i$-th observation as $\hat{y}_{i}$.

####  4.1.2 Computing exceedance probability estimates

We assume that $\hat{y}_{i}$ can be modeled according to a right-skewed
Weibull distribution with a location parameter equal to the forecast
$\hat{y}_{i}$. The scale $\beta$ and shape $\alpha$ parameters are estimated
using the training data. We use the Weibull distribution as it provides an
adequate fit to SWH data muraleedharan2007modified . For other case studies, a
different distribution may be more appropriate. In these conditions, we can
estimate $p_{i}$, the exceedance probability at the $i$-th time step, using
the corresponding CDF. This is done as follows:

$p_{i}=1-F(\tau;\hat{y}_{i},\beta,\alpha)$ (2)

Where $F$ denotes the CDF of the Weibull distribution. When evaluated at the
threshold $\tau$, the CDF represents the probability that the respective
random variable will take a value less than or equal to $\tau$. In effect, we
subtract $F(\tau;\hat{y}_{i},\beta,\alpha)$ from $1$ to obtain the probability
that the random variable will exceed $\tau$. This process is described in
algorithm 1.

Input : $f$: Forecasting model

: $\tau$: Exceedance threshold

: $y_{train}$: Training data

Output : $p_{i}$

$\beta,\alpha\leftarrow$ $weibull.fit(y_{train})$ // Fit the Weibull
distribution using the training data and get its parameters

$\hat{y}_{i}\leftarrow$ $f$($\\{y_{i-1},y_{i-2},\dots,y_{i-q}\\}$) // Compute
the forecast for the $i$-th time-step

$p_{i}\leftarrow$ $1-F(\tau;\hat{y}_{i},\beta,\alpha)$// Estimate the
exceedance probability according to the CDF

Return $p_{i}$

Algorithm 1 Computing the exceedance probability estimate using point
forecasts

###  4.2 Methodological Advantages

Aside from performance considerations, which we will delve into in Section 6,
the proposed method may be preferable relative to a classifier for two main
reasons. The first reason is integration because the same forecasting model
can be used to predict both the upcoming values of SWH and to estimate
exceedance probability. The exceedance probability estimate simplifies the
information being given to the end user, which may help a more inexperienced
operator in their decision-making. Yet, complementary numeric forecast offers
more information, which might be valuable for more experienced operators. For
example, if the forecasts are near the threshold value, the predicted
trajectory offers insights into how the variable is expected to change.

The second reason is threshold flexibility: a classification approach fixes
the threshold during training, and it cannot be changed during inference.
Conversely, a regression model follows a lazy approach regarding the
threshold. Therefore, in a given instant we can use the same model to estimate
the exceedance probability for different thresholds – this enables the
plotting of exceedance probability curves. These advantages apply to an
ensemble-based forecasting approach. However, the high computational costs and
lack of transparency often preclude the application of these strategies
lakkaraju2016interpretable ; cerqueira2021model .

##  5 Case Study

In this section, we present the case study we use in the experiments. First,
we describe the dataset and the respective source (Section 5.1). Then, we
explain the preprocessing steps carried out before applying the predictive
models to the time series (Section 5.2).

###  5.1 Data Set

We conducted our study using data collected from the Environment and Climate
Change Canada Buoy (ECCC Buoy). This buoy is located 13 kilometers away in the
open sea close to Halifax, Canada. The data from this buoy was previously used
for an interpolation task by Fasuyi et al. fasuyi2020machine , as we described
in Section 2.1.

The time series spans from 11-02-2000 15:00:00 to 01-04-2020 11:00:00 and
includes the following variables:

  * •

SWH: Significant wave height (in meters) which represents the variable we want
to predict;

  * •

PWP: Peak wave period (in seconds), which is the wave period carrying the
maximum amount of energy;

  * •

CWH: Maximum zero crossing wave height (in meters);

  * •

WDIR: Wind direction (in degrees);

  * •

WSPD: Horizontal wind speed (in meters per second);

  * •

GSPD: Gust wind speed (in meters per second);

  * •

DRYT: Buoy dry bulb temperature (in degrees Celsius);

  * •

SSTP: Approximated sea surface temperature (in degree Celsius).

All variables of the time series are regularly sampled at an hourly frequency.
Figure 2 shows the SWH time series. For visualization purposes, we only plot
the response variable (SWH). The total number of observations is 125.890.
However, there is a considerable number of missing observations, especially in
the period from February 2016 to October 2019.

Figure 2: SWH time series collected with the ECCC buoy.

In Figure 3, we show a histogram that illustrates the distribution of the SWH
time series. The distribution is right-skewed where the heavy tail represents
the large SWH sea state we attempt to predict.

Figure 3: Histogram of the SWH time series distribution, which shows a
positive skewness Figure 4: Partial auto-correlation structure of the SWH time
series.

Figure 4 shows the partial auto-correlation scores of the SWH time series up
to 24 lags. The result shows that the variations in SWH can be explained with
the first few lags.

###  5.2 Data Preparation

As exploratory data analysis, in Figure 5 we show the Pearson correlation of
each pair of variables in the data set. Overall, the main variable SWH is
highly correlated with CHW (0.91), and moderately correlated with the wind and
gust speed variables (WSPD and GSPD) with values 0.41 and 0.42, respectively.

Figure 5: Heatmap showing the pairwise correlation across all variables in the
data set.

We constructed a set of observations for training the predictive models
according to the formalization in Section 3.1. The target variable $y$
represents the value of SWH observed $h$ steps (hours) ahead in the future.
The respective binary target variable $b$ is computed according to whether $y$
takes a value above $\tau$ in $h$ hours. In order to measure the
predictability of exceeding events in different time frames, we analyzed
multiple forecasting horizons. Specifically, we experiment with a horizon from
1 to 24 hours in advance. For example, with $h=6$ we attempt to predict
whether the threshold will be exceeded in 6 hours.

In terms of the threshold $\tau$, we adopt a data-driven approach to set this
value. Specifically, we set the threshold to be the 99th percentile of the
available SWH time series. We follow this approach, as opposed to a
judgemental one (e.g. based on some domain expert advice), because it can be
generalized to different data sets or domains. In effect, at each iteration of
the cross-validation procedure (which is detailed in the next section), we
compute $\tau$ using the training data. The average threshold across all
iterations was $3.17$ meters.

Regarding the predictor variables, we follow the procedure described in
Section 3. We apply time delay embedding to all variables in the time series
with an embedding size (number of lags) of 6. This means that, at a given
instant, $y$ (or $b$) is modeled based on auto-regression using the past 6
values of each of the eight variables described above. We also added the day
of the year in the explanatory variables to account for yearly seasonal
effects. This leads to a total of 49 explanatory variables.

We also performed a data validation step. The objective is to predict
exceedance events. Therefore, if exceedance is occurring in the explanatory
variables ($X$) it means that the event is already happening and prediction is
not necessary at that moment. In this context, we remove the instances where
exceedance is occurring in the explanatory variables because the model will
not be used in this case.

##  6 Experiments

This section describes the experiments carried out to validate the proposed
solution. These experiments cover the following research questions:

  1. 1.

RQ1: Which approach is better for SWH forecasting?;

  2. 2.

RQ2: How does the proposed method based on the CDF perform for estimating the
exceedance probability relative to state-of-the-art approaches?;

  3. 3.

RQ3: What is the impact of the forecasting horizon on the results obtained?

###  6.1 Experimental Design

We carry out a Monte Carlo cross-validation procedure to evaluate the
performance of models picard1984cross . This estimation method, which is also
referred to as repeated holdout, has been shown to provide competitive
forecasting performance estimates with univariate time series
cerqueira2020evaluating . Monte Carlo cross-validation is applied with 10
folds. The training and test sizes in each fold are 50% and 20% of the size of
the available data, respectively. In each fold, a point is randomly selected
from the available data (constrained by the size of training and testing
data). This point then represents the end of the training set, and the start
of the testing set cerqueira2020evaluating .

We evaluate the exceedance probability performance of each approach using the
area under the ROC curve (AUC) and log loss as evaluation metrics. Besides, we
also use typical regression metrics to measure forecasting performance, i.e.,
the performance for predicting the numeric value of future observations.
Specifically, we use the mean absolute error (MAE), mean absolute percentage
error (MAPE), coefficient of determination ($R^{2}$), and root mean squared
error (RMSE).

###  6.2 Methods

In this subsection, we overview the approaches used to estimate the exceedance
probability. These can be broadly split into classification methods (Section
6.2.1) and forecasting (regression) methods (Section 6.2.2).

####  6.2.1 Classification Methods

We apply the following three classification methods for estimating exceedance
probability:

  * •

Random Forest Classifier (RFC): A standard binary classifier built using a
Random Forest breiman2001random ;

  * •

Logistic Regression (LR): Another standard classifier but created using the
Logistic Regression algorithm. We specifically test this method because it is
a popular one for exceedance probability tasks (c.f. Section 2.2);

  * •

RFC+SMOTE: Exceedance typically involves imbalanced problems because the
exceeding events are rare. We created a variant of RFC, which attempts to cope
with this issue using a resampling method. Resampling methods are popular
approaches for dealing with the class imbalance problem. In effect, we test a
classifier that is fit after the training data is pre-processed with the SMOTE
resampling method by Chawla et al. chawla2002smote ;

####  6.2.2 Forecasting Methods

We test four different regression approaches. Namely, a Random Forest
regression method (RFR), a LASSO linear model, and a deep feed-forward neural
network (NN). These are three widely used regression algorithms for
forecasting. The fourth approach is a heterogeneous regression ensemble (HRE).
An ensemble is referred to as heterogeneous, as opposed to homogeneous, if it
is comprised of base models trained with different learning algorithms. The
following learning algorithms were used to train the base models of HRE: a
rule-based model based on the Cubist algorithm, a bagging ensemble of decision
trees (Bagging), ridge regression (Ridge), elastic-net (ElasticNet), LASSO,
multivariate adaptive regression splines (MARS), light gradient boosting
optimized using random search (LGBM); extra trees (ExtraTree), Adaboost
(AdaBoost), Random Forests (RFR), partial least squares regression (PLS),
principal components regression (PCR), nearest neighbors (KNN), and a deep
feedforward neural network (NN).

We applied different parameter settings and the total number of initial models
is 50. The complete list of parameters is shown in Table 1. Using a validation
set we trimmed the ensemble by removing half of the models with worse
performance. The trimming process has been shown to improve the forecasting
performance of ensembles jose2008simple ; cerqueira2019arbitrage . Usually, a
dynamic combination rule improves the performance of forecasting ensembles
relative to a static one. Notwithstanding, the simple average is a solid
benchmark according to Cerqueira et al. cerqueira2019arbitrage .

The neural network architecture contains 2 hidden layers with 32 and 16 units,
respectively. Dropout was also included after each hidden layer with a drop
rate of 0.2. All units followed a rectified linear unit activation function.
Training occurred during 30 epochs. We optimized mean squared error using
validation data and the Adam optimizer.

Table 1: Summary of the learning algorithms ID | Algorithm | Parameter | Value
---|---|---|---
MARS | Multivar. A. R. Splines | Degree | {1, 3}
No. terms | {10, 20}
|  | Threshold | {0.01, 0.001}
LASSO | LASSO Regression | L1 alpha | {0.25, 0.5,
0.75, 1}
Ridge | Ridge Regression | L2 alpha | {0.25, 0.5,
0.75, 1}
ElasticNet | ElasticNet Regression | Default | -
KNN | k-Nearest Neighbors | No. of Neighbors | {1, 5, 10}
Weights | {Uniform,
Distance}
RFR | Random Forest Regression | No. trees | {50, 100}
Max depth | {default, 3}
ExtraTree | Extra Trees Regression | No. trees | {50, 100}
Max depth | {default, 3}
Bagging | Bagging Regression | Base estimator | {Decision Stump,
Decision Tree}
|  | No. estimators | {50, 100}
RBR | Rule-based Regression | No. iterations | {1, 5}
NN | Deep Feedforward Neural Network | No. hidden layers | {2}
LGBM | LightGBM | Booster | {DART,
GOSS, GBDT}
AdaBoost | AdaBoost Regression | Base estimator | {Decision Stump,
Decision Tree}
Learning rate | {0.3, 0.7}
PCR | Principal Comp. Regr. | No. of components | {2, 5}
PLS | Partial Least Regr. | No. of components | {2, 5}
OMP | Orthogonal Matching Pursuit | Default | -

We apply the regression approaches in two different ways to obtain the
estimates for exceedance probability. The first is a Direct (abbreviated as D)
approach: the exceedance probability $p$ is computed according to the ratio of
ensemble members which predicts a value above the threshold $\tau$. The Direct
approach is only valid for ensembles, in this case, HRE (a heterogeneous
ensemble) and RFR (a homogeneous ensemble). The second approach is using the
CDF (CDF) according to our proposed method described in Section 4. We denote
which approach is used by appending the name to the regression model. For
example, HRE+CDF represents the heterogeneous ensemble applied with the CDF
for estimating exceedance probability.

###  6.3 Results

Figure 6: Sample of the forecasts and exceedance probabilities produced by the
model.

As a preliminary analysis, Figure 6 shows a sample of the forecasts and
exceedance probability estimates produced by the method NN+CDF for a lead time
of 6 hours. The model can capture different exceedance events during this
period.

In this subsection, we analyze the results of the experiments. First, we
analyze the performance of each regression method (RFR, LASSO, HRE, and NN)
for forecasting the future values of the time series (RQ1). The results are
shown in Figure 7, which illustrates the distribution of performance of each
method across the forecasting horizon with four metrics: MAE, MAPE, $R^{2}$,
and RMSE. For all metrics, NN and HRE show the best performance, which is
comparable with each other. These methods are followed by RFR. The linear
model LASSO underperforms relative to the others.

Figure 7: Distribution of forecasting performance across the forecasting
horizon for each regression approach by different metrics.

We then evaluated all methods according to their ability to estimate
exceedance probability. Figure 8 shows the distribution of AUC for each method
across the forecasting horizons (up to 24 hours). The best performing models
are NN+CDF, RFR+CDF and HRE+CDF, respectively. This rank is consistent with
how these models performed for forecasting based on the regression-based
scores. The results indicate that coupling a forecasting model with the
mechanism based on the CDF for estimating exceedance probability provides a
better performance relative to binary classification strategies (RFC,
RFC+SMOTE, and LR) and ensemble-based approaches (RFR+D and HRE+D).

Figure 8: Distribution of the AUC for each method across the forecasting
horizons.

Another interesting result to note is that HRE+D provides poor estimates
regarding exceedance probability, while RFR+D is competitive with
classification methods. This outcome suggests that building an ensemble with
weak models (RFR is an ensemble of decision trees) is better than an ensemble
of strong models, such as HRE.

Pre-processing the training data with a resampling method (SMOTE) improved the
average AUC of the RFC, though the difference is small. The linear model LR is
comparable with the Random Forests classifiers. On the other hand, LASSO+CDF
underperforms relative to the other methods, except for HRE+D. Overall, these
results answer the research question RQ2, regarding the competitiveness of the
proposed approach in terms of performance.

Figure 9: AUC for each method across the forecasting horizon. Classification
methods (RFC+SMOTE, RFC, and LR) are represented with long dashed, ensemble
direct methods (RFR+D, and HRE+D) are represented as dashed lines and CDF-
based methods are denoted with solid lines.

Figure 9 shows the AUC score for each method across the forecasting horizons.
There is a generalized tendency for decreasing scores as the forecasting
horizon increases (RQ3). This is expected as predicting further into the
future represents a more difficult task.

Figure 10: Results from the Bayesian correlated t-test. Probability of NN+CDF
losing, drawing, or winning significantly against the respective method when
considering a ROPE of 1%.

We carried out a Bayesian analysis to assess the significance of the results
using the Bayesian correlated t-test benavoli2017time . This test is used to
compare pairs of predictive models across the forecasting horizon. In this
case, we compare NN+CDF with all remaining methods. We define the region of
practical equivalence (ROPE) for the Bayes correlated t-test to be the
interval [-1%, 1%]. This means that the performance of the two methods under
comparison is considered equivalent if their percentage difference is within
this interval. The results are presented in Figure 10, which shows the
probability of NN+CDF winning in blue, drawing in green (results within the
ROPE), or losing in red against each remaining method. NN+CDF outperforms
other approaches, except for HRE+CDF. Notwithstanding, the ensemble HRE+CDF
also follows the proposed approach for exceedance probability forecasting.

##  7 Discussion

The main conclusion from the experiments is that the proposed method provides
better exceedance probability estimates relative to state-of-the-art
alternatives for a case study concerning SWH forecasting. This outcome has an
important practical impact because users (e.g. decision-makers or analysts
responsible for maritime operations) can rely on a forecasting model to
predict not only the value of future observations but also quantify the
probability of an exceedance event (in this case, a large SWH state).

A second important conclusion is that a deep learning neural network provided
the best results when coupled with the proposed approach. Besides, a
heterogeneous regression ensemble (HRE), whose members are trained with
several different learning algorithms, did not provide better exceedance
probability estimations than a Random Forest (RFR).

While exceedance probability forecasting usually refers to binary events
(whether or not the exceedance occurs), multiple thresholds can also be
defined. An example of this is the triple barrier method used in quantitative
trading, e.g. de2020machine . Stock market traders may be interested in
predicting buy, sell, or hold signals according to price movements. Exceeding
a positive threshold in terms of predicted price returns can be used as a buy
signal; the opposite (predicted price returns below a negative threshold) can
represent a sell trigger. If none of the thresholds is met, then the trader
should hold the current position.

We transformed the time series into a trainable format using time delay
embedding on each variable, as we formalized in Section 3. During our
experiments, we also tested a differencing operation to stabilize the mean,
but this process did not impact performance. The data shows a non-constant
variance over time where the dispersion is higher in the winter months (c.f.
Figure 2). We tested the application of a Box-Cox method or a log
transformation to stabilize the variance. However, this process also did not
improve performance. A possible explanation is that changes in variance are
correlated with changes in the mean level, and the latter are captured by the
model.

An important part of the proposed method is the choice of distribution. The
data shows a clear positive skew and previous works have modeled this type of
data using distributions such as Weibull or Rayleigh muraleedharan2007modified
. However, the most appropriate distribution depends on the input data.

##  8 Conclusions

This paper tackles SWH forecasting problems, which is a relevant topic in
ocean data analytics. We are particularly interested in predicting extreme
values of SWH, and frame this task as an exceedance probability estimation
problem.

We proposed a new method to estimate exceedance probability using the point
forecasts produced by an auto-regressive forecasting model. This approach is
an alternative to using a binary classification model or an ensemble-based
approach. The proposed solution involves transforming the point forecasts into
exceedance probabilities based on the CDF. In this particular case study, we
resort to the Weibull distribution, but the best distribution depends on the
input data.

The experiments carried out suggest that the proposed approach, coupled with a
forecasting model based on deep learning, leads to better performance than a
binary probabilistic classifier or an ensemble of forecasts. In future work,
we will generalize the framework to other application domains besides SWH
forecasting.

## Acknowledgments

The work of L. Torgo was undertaken, in part, thanks to funding from the
Canada Research Chairs program. We regret to inform that L. Torgo passed away
during the development of this article. We would like to express our
condolences to their family and colleagues.

## References

  * (1) Abdullah, F., Ningsih, N., Al-Khan, T.: Significant wave height forecasting using long short-term memory neural network in indonesian waters.  Journal of Ocean Engineering and Marine Energy 8(2), 183–192 (2022)
  * (2) Ali, M., Prasad, R., Xiang, Y., Deo, R.C.: Near real-time significant wave height forecasting with hybridized multiple linear regression algorithms.  Renewable and Sustainable Energy Reviews 132, 110003 (2020)
  * (3) Benavoli, A., Corani, G., Demšar, J., Zaffalon, M.: Time for a change: a tutorial for comparing multiple classifiers through bayesian analysis.  The Journal of Machine Learning Research 18(1), 2653–2688 (2017)
  * (4) Bethel, B.J., Dong, C., Zhou, S., Sun, W., Bao, Y.: Assessing long short-term memory network significant wave height forecast efficacy in the caribbean sea and atlantic ocean.  Available at SSRN 4153300
  * (5) Branco, P., Torgo, L., Ribeiro, R.P.: A survey of predictive modeling on imbalanced domains.  ACM Computing Surveys (CSUR) 49(2), 1–50 (2016)
  * (6) Breiman, L.: Random forests.  Machine learning 45(1), 5–32 (2001)
  * (7) Cerqueira, V., Torgo, L., Mozetič, I.: Evaluating time series forecasting models: An empirical study on performance estimation methods.  Machine Learning 109(11), 1997–2028 (2020)
  * (8) Cerqueira, V., Torgo, L., Pinto, F., Soares, C.: Arbitrage of forecasting experts.  Machine Learning 108(6), 913–944 (2019)
  * (9) Cerqueira, V., Torgo, L., Soares, C., Bifet, A.: Model compression for dynamic forecast combination.  arXiv preprint arXiv:2104.01830 (2021)
  * (10) Challu, C., Olivares, K.G., Oreshkin, B.N., Ramirez, F.G., Canseco, M.M., Dubrawski, A.: Nhits: Neural hierarchical interpolation for time series forecasting.  In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, pp. 6989–6997 (2023)
  * (11) Chawla, N.V., Bowyer, K.W., Hall, L.O., Kegelmeyer, W.P.: Smote: synthetic minority over-sampling technique.  Journal of artificial intelligence research 16, 321–357 (2002)
  * (12) Chen, D., Liu, F., Zhang, Z., Lu, X., Li, Z.: Significant wave height prediction based on wavelet graph neural network.  In: 2021 IEEE 4th International Conference on Big Data and Artificial Intelligence (BDAI), pp. 80–85. IEEE (2021)
  * (13) Dixit, P., Londhe, S., Dandawate, Y.: Removing prediction lag in wave height forecasting using neuro-wavelet modeling technique.  Ocean Engineering 93, 74–83 (2015)
  * (14) Duan, W.y., Huang, L.m., Han, Y., Huang, D.t.: A hybrid emd-ar model for nonlinear and non-stationary wave forecasting.  Journal of Zhejiang University-SCIENCE A 17(2), 115–129 (2016)
  * (15) Fasuyi, J.W., Newport, J., Whidden, C.: A machine learning redundancy model for the herring cove smart buoy.  Journal of Ocean Technology 15(3) (2020)
  * (16) Ferro, C.A.: Comparing probabilistic forecasting systems with the brier score.  Weather and Forecasting 22(5), 1076–1088 (2007)
  * (17) Frangopol, D., Kim, S.: Prognosis and life-cycle assessment based on shm information.  In: Sensor technologies for civil infrastructures, pp. 145–171. Elsevier (2014)
  * (18) Gneiting, T., Katzfuss, M.: Probabilistic forecasting.  Annual Review of Statistics and Its Application 1, 125–151 (2014)
  * (19) Guo, B., Gupta, P., Steen, S., Tvete, H.A.: Evaluating vessel technical performance index using physics-based and data-driven approach.  Ocean Engineering 286, 115402 (2023)
  * (20) Huang, G.B., Zhu, Q.Y., Siew, C.K.: Extreme learning machine: theory and applications.  Neurocomputing 70(1-3), 489–501 (2006)
  * (21) Huang, N.E., Shen, Z., Long, S.R., Wu, M.C., Shih, H.H., Zheng, Q., Yen, N.C., Tung, C.C., Liu, H.H.: The empirical mode decomposition and the hilbert spectrum for nonlinear and non-stationary time series analysis.  Proceedings of the Royal Society of London. Series A: mathematical, physical and engineering sciences 454(1971), 903–995 (1998)
  * (22) Jain, P., Deo, M.: Real-time wave forecasts off the western indian coast.  Applied ocean research 29(1-2), 72–79 (2007)
  * (23) Jose, V.R.R., Winkler, R.L.: Simple robust averages of forecasts: Some empirical results.  International journal of forecasting 24(1), 163–169 (2008)
  * (24) Khosravi, A., Nahavandi, S., Creighton, D., Atiya, A.F.: Comprehensive review of neural network-based prediction intervals and new advances.  IEEE Transactions on neural networks 22(9), 1341–1356 (2011)
  * (25) Kunreuther, H.: Risk analysis and risk management in an uncertain world 1.  Risk Analysis: An International Journal 22(4), 655–664 (2002)
  * (26) Kuppili, A.S.: Forecasting meteorological variables and anticipating climatic aberrations of an oceanic buoy using a neighbour buoy (2021)
  * (27) Lakkaraju, H., Bach, S.H., Leskovec, J.: Interpretable decision sets: A joint framework for description and prediction.  In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1675–1684 (2016)
  * (28) Lewis, J.M.: Roots of ensemble forecasting.  Monthly weather review 133(7), 1865–1885 (2005)
  * (29) Li, G., Weiss, G., Mueller, M., Townley, S., Belmont, M.R.: Wave energy converter control by wave prediction and dynamic programming.  Renewable Energy 48, 392–403 (2012)
  * (30) Londhe, S., Panchang, V.: One-day wave forecasts based on artificial neural networks.  Journal of Atmospheric and oceanic Technology 23(11), 1593–1603 (2006)
  * (31) Muraleedharan, G., Rao, A., Kurup, P., Nair, N.U., Sinha, M.: Modified weibull distribution for maximum and significant wave height simulation and prediction.  Coastal Engineering 54(8), 630–638 (2007)
  * (32) Nkoro, E., Uko, A.K., et al.: Autoregressive distributed lag (ardl) cointegration technique: application and interpretation.  Journal of Statistical and Econometric methods 5(4), 63–91 (2016)
  * (33) Picard, R.R., Cook, R.D.: Cross-validation of regression models.  Journal of the American Statistical Association 79(387), 575–583 (1984)
  * (34) Polson, M., Sokolov, V.: Deep learning for energy markets.  Applied Stochastic Models in Business and Industry 36(1), 195–209 (2020)
  * (35) de Prado, M.M.L.: Machine learning for asset managers.  Cambridge University Press (2020)
  * (36) Sadeghifar, T., Nouri Motlagh, M., Torabi Azad, M., Mohammad Mahdizadeh, M.: Coastal wave height prediction using recurrent neural networks (rnns) in the south caspian sea.  Marine Geodesy 40(6), 454–465 (2017)
  * (37) Shamshirband, S., Mosavi, A., Rabczuk, T., Nabipour, N., Chau, K.w.: Prediction of significant wave height; comparison between nested grid numerical model, and machine learning models of artificial neural networks, extreme learning and support vector machines.  Engineering Applications of Computational Fluid Mechanics 14(1), 805–817 (2020)
  * (38) Slud, E., Kedem, B.: Partial likelihood analysis of logistic regression and autoregression.  Statistica Sinica pp. 89–106 (1994)
  * (39) Taieb, S.B., Bontempi, G., Atiya, A.F., Sorjamaa, A.: A review and comparison of strategies for multi-step ahead time series forecasting based on the nn5 forecasting competition.  Expert systems with applications 39(8), 7067–7083 (2012)
  * (40) Takens, F.: Dynamical Systems and Turbulence, Warwick 1980: Proceedings of a Symposium Held at the University of Warwick 1979/80, chap. Detecting strange attractors in turbulence, pp. 366–381.  Springer Berlin Heidelberg, Berlin, Heidelberg (1981)
  * (41) Taylor, J.W.: Probabilistic forecasting of wind power ramp events using autoregressive logit models.  European Journal of Operational Research 259(2), 703–712 (2017)
  * (42) Taylor, J.W., Yu, K.: Using auto-regressive logit models to forecast the exceedance probability for financial risk management.  Journal of the Royal Statistical Society: Series A (Statistics in Society) 179(4), 1069–1092 (2016)
  * (43) Toth, Z., Talagrand, O., Candille, G., Zhu, Y.: Probability and ensemble forecasts.  Forecast verification: A practitioner’s guide in atmospheric science 137, 163 (2003)
  * (44) Wang, J., Wang, Y., Yang, J.: Forecasting of significant wave height based on gated recurrent unit network in the taiwan strait and its adjacent waters.  Water 13(1), 86 (2021)