# Probabilistic forecasts of wind power generation in regions with complex
topography using deep learning methods: An Arctic case

Odin Foldvik Eikeland
Department of Physics and Technology
UiT-the Arctic University of Norway
9037 Tromsø, Norway
<EMAIL_ADDRESS>
&Finn Dag Hovem
Ishavskraft Power Company
9024 Tromsø, Norway
&Tom Eirik Olsen
Ishavskraft Power Company
9024 Tromsø, Norway
&Matteo Chiesa
Department of Physics and Technology
UiT-the Arctic University of Norway
9037 Tromsø, Norway
<EMAIL_ADDRESS>
&Filippo Maria Bianchi
Department of Mathematics and Statistics
UiT-the Arctic University of Norway
9037 Tromsø, Norway
<EMAIL_ADDRESS>
Corresponding author

###### Abstract

The energy market relies on forecasting capabilities of both demand and power
generation that need to be kept in dynamic balance. Today, when it comes to
renewable energy generation, such decisions are increasingly made in a
liberalized electricity market environment, where future power generation must
be offered through contracts and auction mechanisms, hence based on forecasts.
The increased share of highly intermittent power generation from renewable
energy sources increases the uncertainty about the expected future power
generation. Point forecast does not account for such uncertainties. To account
for these uncertainties, it is possible to make probabilistic forecasts.

This work first presents important concepts and approaches concerning
probabilistic forecasts with deep learning. Then, deep learning models are
used to make probabilistic forecasts of day-ahead power generation from a wind
power plant located in Northern Norway. The performance in terms of obtained
quality of the prediction intervals is compared for different deep learning
models and sets of covariates.

The findings show that the accuracy of the predictions improves when
historical data on measured weather and numerical weather predictions (NWPs)
were included as exogenous variables. This allows the model to auto-correct
systematic biases in the NWPs using the historical measurement data. Using
only NWPs, or only measured weather as exogenous variables, worse prediction
performances were obtained.

_K_ eywords Energy analytics $\cdot$ Probabilistic forecasting $\cdot$ Wind
power electricity generation $\cdot$ Deep learning

##  1 Introduction

Making accurate predictions is of fundamental importance in the energy market
where decisions are based on expectations about the future. Today, when it
comes to renewable energy generation, such decisions are increasingly made in
a liberalized electricity market environment, where future power generation
must be offered through contracts and auction mechanisms, hence based on
forecasts [18, 12, 25]. Since renewable energy sources (RES) are to eventually
participate in market mechanisms under the same rules as conventional fossil-
fuel-based generators, mismatches between contracted generation and actual
deliveries may induce financial penalties [21]. Indeed, the energy production
from RES can be predicted with limited accuracy. This, in addition to
uncertainties in market prices, yield uncertain market returns. However, in
market environments under such high levels of uncertainty, the relevant
stakeholders may make better decisions if they are given the best possible
estimates about the future.

Forecasts in their most common form are to predict the next value that is most
likely to occur. Forecasts in such form are called a _point_ forecast. The
main objective of making point forecasts is to train a model to predict a
certain point in the future, and hopefully, the actual value will eventually
be the same (i.e., a perfect prediction). However, it is not reasonable to
expect that a model can perform 100% accurate predictions. The models will
therefore predict the most likely value based on what it has learned through
the information that is given. However, when predicting the most likely value
(or point) in the future, one does not consider the uncertainties in this
forecast (how sure are we that this will be the next value). For the
stakeholders that work in the energy market with a high degree of uncertainty,
it is of interest to measure the uncertainty in a given forecast. For
instance, the power production from solar photovoltaic (PV) or wind power is
highly intermittent and dependent on multiple features with a complex and non-
linear nature (weather, power market, human activity) [2, 6, 35, 18]. Making
accurate point predictions from these technologies is almost impossible,
especially when considering short-term forecasts that range from 12-hour to
36-hour ahead, which is the day-ahead market the energy trading companies must
relate to [6, 28]. Therefore, it is of fundamental importance to be able to
predict the _distribution_ of the expected outcome to find a certain interval
of possible outcomes. Probabilistic forecasts predict such distributions where
the interval of possible outcomes creates a prediction interval (PI).
Probabilistic forecasts will allow market participants to consider the
uncertainties in a given prediction.

The increased share of RES technologies with fluctuating generation in the
electricity market, and the rapid development within machine learning (ML),
have motivated the development of research concerning probabilistic forecasts
in energy applications with deep learning. Efforts have been done to construct
deep learning models that create probabilistic forecasts of expected
generation from technologies such as PV and wind power [12, 29, 35, 30, 13].
Similar approaches have been used to make probabilistic forecasts of expected
weather (such as wind speed), traffic, energy load, and spot prices in the
electricity market [34, 26, 36, 6]. The majority have used different deep
learning architectures such as Convolutional Neural Networks (CNNs) or
Recurrent Neural Networks (RNNs) where different loss functions are optimized
to compute predicted distributions. In addition, some works are proposing
methodologies where original Neural Networks (NNs) are modified for a specific
task to make probabilistic forecasts [1, 28, 30]. In [20] the authors compared
deep learning models to make probabilistic forecasts on different datasets and
showed that wind power generation was the most challenging one to predict with
high accuracy due to the highly non-linear feature and randomness in power
generation. Despite a large number of works on probabilistic forecasts with
deep learning techniques in energy applications, there is a lack of research
comparing the effect of using exogenous variables to obtain the best
prediction performance of wind power generation.

The electricity generation from wind power systems is directly affected by the
amount of wind that hits the turbines, where the amount of power that is
generated follows a wind-to-power conversion process, or the _power curve_
[38]. Therefore, the majority of work uses numerical weather predictions
(NWPs) as inputs to forecast the expected wind power generation in the short-
term range (6-72 hours ahead) [23, 37, 10, 33]. When predicting wind power
generation, a large part of the forecast error will therefore directly come
from the NWPs. Predicting short-term weather (especially wind speed) is a
complex task due to its non-linear and fluctuating nature [14, 39, 31].
Consequently, this gives large sources of error for making accurate short-term
predictions of wind power generation. The wind power case study considered in
this work is in a region with complex terrain where it is large weather
variations within small distances, making it increasingly difficult to have
accurate NWPs available. In this work, we address the issue of inaccurate NWPs
by constructing different configurations of datasets that could account for
such failure sources. We perform prediction experiments using different sets
of exogenous variables are compared to make probabilistic forecasts of wind
power generation with deep learning models. The exogenous variables considered
are the NWP (wind speed + wind direction) and measured weather (wind speed +
wind direction) from instruments installed locally in the wind power plant.
The wind power plant studied in this work lies in a region with a complex
topography in Arctic weather conditions that potentially have a larger source
for failure in the NWPs compared to other regions. Therefore it is a
motivation in this work to use deep learning models that are trained to
account for possible failure sources in weather predictions by using the
measured weather to correct for such failures. In the end, we compare the
accuracy of the probabilistic forecasts for different sets of covariates and
discover the variables that should be included to compute high-quality PIs.

This paper is structured as follows. In Section 2, an introduction to the
field of probabilistic forecasts is provided. Section 3 presents a review of
relevant research in the field of probabilistic forecast within energy
analytics. In section 4, the wind power plant case study is presented. Section
5 presents the methodology for making probabilistic forecasts. In Section 6
the results are presented in terms of the obtained quality of the PIs.
Conclusions are given in Section 7.

##  2 The concept of prediction intervals

Although the field of probabilistic forecasts is based on well-known concepts
in statistics, and there exists a vast amount of existing research on this
field, some concepts have been shown to be confused and misused [11]. To avoid
this, we start by defining and explaining some relevant concepts that will be
used through the study.

When making probabilistic forecasts, the goal is to make a PI that considers
the uncertainties in the predictions. The PI is an estimate of an interval in
which the future observation will fall with a certain probability. For
instance, for a 95% PI, there should be a 95% probability that the next value
will fall within the lower and upper bounds.

The generic form for computing the upper and lower bound of PIs is

$P(Y_{n}\in C(X_{n}))\geq 1-\alpha,$ (1)

where $Y_{n}$ is the response variable, $C(X_{n})$ is the confidence interval
centered on the covariate $X_{n}$, and $\alpha$ is the significance level. For
a 95% PI, $\alpha$ is 0.05.

PIs can either be of low quality, or of high quality. A high-quality PI will
be sharp and well-calibrated (the PI contains the desired proportion of the
samples $1-\alpha$). A low-quality PI is too wide or is miscalibrated (the
empirical quantiles do not match the theoretical ones).

Examples of high-quality and low-quality PIs is given in Fig. 1.

Figure 1: Examples of PI with high (uppermost figure) and low quality. The red
dots represent the observed values, the blue dots is the point predictions.
The yellow line represent the PI, which in this case is the 95% prediction
interval.

Here, we see an example of a point-and probabilistic forecast. The red dots
represent the observed values, the blue dots are the point predictions. The
yellow line represents the PI, which in this case is the 95% prediction
interval. Here, it is clear that the high-quality PI (uppermost Figure)
fulfills two important criteria when computing PIs. Sharpness and calibration.
The boundaries of the PI should be sharp to convey useful information about
the uncertainity in the predictions. In addition, the PI should be calibrated,
which means that for a 95% PI, 95% of the observed values should fall within
the PI. Here, the PI contains exactly 95% of the observed values, and hence we
can trust that there is a 95% probability that the observed values will fall
within this PI.

The middle and lowermost examples on contrary, show examples of low-quality
PIs. The example in the middle shows a PI with high sharpness, but it is not
calibrated as it only contains 80% of the observed values, which is not
adequate in a 95% PI. The lowermost exampled shows a calibrated PI as it
contains 95% of the observed values. However, it is not sharp, and PIs that
are too wide do not convey any useful information about the variation in the
observed values.

##  3 Related works on probabilistic forecasting with deep learning

There exists a vast amount of former research on probabilistic forecast in
different applications, and it is outside the scope to review all of them in
this work. Therefore, former works that have particularly focused on the
application of probabilistic forecast with deep learning within energy
applications are reviewed here. In the following, a few examples of some
former popular reviews that have high relevance for energy applications is
provided.
In [11], a comprehensive review on probabilistic forecast of the electric load
was conducted. The authors offered a tutorial review and presented notable
techniques, methodologies, and evaluation methods that could be valuable for
researchers and practitioners in the area of load forecasting. To improve the
field of probabilistic forecasts, they underlined the need to invest in
additional research, and they state the importance of considering emerging
technologies and energy policies in the probabilistic load forecasting
process.
Another relevant review was performed by [38]. The authors discuss the
challenge of making accurate predictions of the highly intermittent
electricity generation from wind power. They discuss the value of
probabilistic forecasts in such tasks as it could provide additional
quantitative information on the uncertainty. For actors that work with
decision-making in an uncertain environment (for instance power traders),
information about uncertainties in decisions is of interest. In this work, the
authors presented a review of state-of-the-art methods and new developments
within wind power probabilistic forecasting. They discuss different forecast
methods and classified them into different categories in terms of uncertainty
representation. Finally, they summarized requirements and the overall
framework of uncertainty forecasting evaluation.

Notably, both these two famous reviews are quite mature (2016 and 2014), and
the field of probabilistic forecasting with deep learning in energy analytics
has vastly developed in recent years. A recent review and comparison study was
performed by Mashlakov et al. [20]. Here, the recent developments in the field
of probabilistic forecasting, multivariate models, and multi-horizon time
series forecasting were reviewed. The authors empirically evaluate the
performance of novel deep learning models for predicting wind power, solar
power, electricity load, and wholesale electricity price for intraday and day-
ahead time horizons. They reviewed the performances of both point and
probabilistic forecast approaches. This comprehensive comparison work could
serve as a reference point for the quantitative evaluation of deep learning
models for probabilistic multivariate energy forecasting in power systems.

In addition to the above-mentioned reviews, Tab. 1 present an overview of some
recent research works which have contributed to developing the field of
probabilistic forecasting with deep learning in energy applications.

Table 1: Overview of different works Model | Year | Network | Forecasts | Metrics | Dataset
---|---|---|---|---|---
CNNs |  |  |  |  |
[29] | 2019 | CNN | Point, Prob. |  | MAPE,RMSE
---
QL
PV power
[7] | 2020 | TCN | Prob, Point | QL, NRMSE | Retailers
[12] | 2020 | CNN | Prob. |  | RMSE,MAPE
---
PCIP, PINAW
PV power
RNNs |  |  |  |  |
[8] | 2019 | RNN | Prob | QL,NRMSE |  | demand,traffic
---
finance
[26] | 2020 | RNN | Prob. | QL, NRMSE |  | demand, traffic,
---
electricity
[24] | 2021 | RNN | Prob | CRPS, MSE | traffic, energy
[4] | 2020 | NN library | Prob | QL | energy, PV
Modified NNs |  |  |  |  |
[28] | 2018 | Bi-LSTM | Prob | RMSE,QL | Power market
[34] | 2020 | BiGRU | Point, Prob. | RMSE, QL | Wind speed
[35] | 2020 | IDMDN | Prob | NRMSE | Wind power
[1] | 2020 | GRU+CNN | Point, Prob | RMSE, CRPS | Wind speed
[36] | 2018 | iQRNN | Prob. | QL,WS | Energy load
[32] | 2021 | PCFM | Point, Prob |  | RMSE, MAE
---
PINAW, PCIP
Wind speed
[17] | 2019 | WT + CNN | Point, Prob |  | RMSE, MAE, MAPE
---
PINAW, PCIP, CWC
Wind speed
[40] | 2018 | NARX | Point, Prob |  | RMSE, MAE
---
PINAW, PCIP
Wind speed
[41] | 2019 | LSTM + GPR | Point, Prob | RMSE, MAE, CRPS | Wind speed
Bayesian methods |  |  |  |  |
[6] | 2019 | Bayesian | Prob. |  | RMSE,MAPE
---
CRPS
Energy prices
[18] | 2020 | STNN | Point, Prob. | RMSE, CRPS | Wind speed
[3] | 2020 | Bayesian NNs | Point, Prob | RMSE, CPRS | Energy load
[15] | 2019 | Calibrated | Prob. | MAPE, QL | UCI datasets
Ensemble methods |  |  |  |  |
[30] | 2017 | WT + CNN | Prob | RMSE, PICP | Wind power
[2] | 2016 | 7 ML models | Prob |  | RMSE,MAE
---
QL
Solar power
[13] | 2021 | SEFMGPR | Prob | RMSE, R2 | Wind power
[16] | 2020 | EPS | Prob | CRPS | Solar irradiance
[27] | 2020 | MDE | Prob | QL | Wind power

All works in Tab. 1 have tested the performance of their proposed methodology
on energy-related datasets, and the prediction performances are presented with
popular probabilistic metrics such as the QL, CRPS, PCIP, and PINAW (some of
these metrics are defined and used in the result section in this work). The
methodologies involve either using CNNs, RNNs, or some modifications of them.
In addition, some work has proposed methodologies where they ensemble multiple
NNs together to make probabilistic forecasts. Most works have applied the
frequentist approach where the main aim is to optimize a cost function (such
as the quantile loss), but some works have also proposed a Bayesian approach
to make probabilistic forecasts with deep learning. Noteworthy, among all
works, there is no specific model or method that seems to consistently achieve
the best results, apart from the DeepAR model which has shown promising
results in several works [20, 26, 4].

In a recent review and comparison work by [20], the authors highlighted the
need of testing more automated deep learning models to progress the research
within the field of probabilistic forecasts in energy applications. The
authors suggested using open source libraries for modeling, such as the
GluonTS toolkit [4]. The aim of the GluonTS library developed by [4] is to
provide a flexible tool for probabilistic time series modeling with deep
learning-based models. Motivated by the suggestion in [20], we implement deep
learning models from the GluonTS library to make probabilistic forecasts of
wind power generation.

##  4 Wind power plant case study

###  4.1 The wind power plant located in Northern Norway

In this work, the power generation from a wind farm in the region of Northern
Norway is predicted. This wind farm consists of 18 turbines with a maximum
power generation capacity of 3 MW, giving a maximum capacity of 54 MW for this
power plant. In Fig. 2, an altitude map is created for the region of the wind
power plant.

Figure 2: Altitude map where the different colors indicate the altitude level.
Green color is altitudes between 75 and 230 MASL. The highest mountains around
in the region close to the wind park is up to 1032 MASL. The 18 turbines in
the wind farm are marked in red dots.

The different colors in the altitude map indicate the interval in Meter Above
Sea Level (MASL). Here, the green color is all altitudes between 75 and 230
MASL. The highest elevation in the mountains to the right has an altitude
between 632 and 1,032 MASL. The red dots are the geographical positions of the
different turbines. Most of the turbines are approximately at the same
altitude below 230 MASL. The two turbines at the rightmost positions are
slightly below an altitude of 75 MASL.

Such a complex topography where the altitude varies from 0 to above 1000 MASL
within small regions, is a typical phenomenon for the terrain in Northern
Norway. This terrain has a huge impact on the local weather variation, and
this particular wind power plant is highly affected by local variations in the
wind. The owner of this wind power plant reported that the leftmost turbine
produced 25% less energy than the rightmost turbine during the year 2020. The
reason is that the turbines located to the left are shielded by the large
mountains (indicated by the white color in the altitude map) which reduce the
amount of wind that hits the turbine. On the other hand, the turbines located
to the right are close to the ocean where there are few objectives that could
reduce the wind. This power plant is a perfect example of where it can be
large differences in production between the turbines. This can increase the
difficulties in making predictions for the whole park, as the total power
output from the park will be even more fluctuating compared to wind farms in
flat regions where the weather conditions are more equal throughout the whole
farm.

###  4.2 Dataset and and technical properties of wind turbines

The data available from the wind park is the historical measured power
generation, wind speed, and wind direction measured on each turbine from the
year 2020. The measured wind speed and wind direction variables are collected
from the weather stations mounted on the nacelle on the turbine that is
located in the middle of the wind park. The data are in a 1-hour resolution
and are received from the Troms Kraft Power company, the owner of the wind
power plant. This gives a dataset of totally 8,784 samples. In addition to
measured data, the NWP from the AROME-Arctic model is collected. This model is
developed by the meteorological institute of Norway (MET) and provides weather
forecasts with a spatial resolution of 2.5 kilometers and a temporal
resolution of 1 hour. Similar to the weather measurements, the predicted wind
speed and wind direction variables are collected from the AROME-Arctic model.
Using only wind speed and wind direction as additional input variables to the
prediction experiments has been shown to provide the most accurate forecasts
in former wind power forecast literature [23]. This is due to the fact that
wind power generation is directly affected by the amount of wind that hits the
turbine blades. Hereafter, the weather data is referred to the wind speed and
wind direction. Fig. 3 illustrate a map with the AROME-Arctic simulated wind
speed for a randomly selected hour on the 1st of March in 2020. The colors in
the map represent the simulated wind speed in each cell (spatial resolution of
2.5 kilometers). It is clear that the turbines in the wind power park are
distributed over two different cells, and there are differences in the wind
speed for the two cells.

Figure 3: The Arome Arctic weather simulation map where the red dots represent
the position of each wind power turbine. Each square represent the spatial
resolution of 2.5$\times$2.5 kilometers. 12 turbines and 6 turbines are
located in the leftmost and rightmost cell, respectively

Fig. 3 show large differences in wind speed within a few kilometers, ranging
from zero wind (blue color in southernmost part) to almost 30 m/s wind (red
color in northernmost part). The AROME-Arctic weather forecast for this
particular example shows that the rightmost part of the wind power plant has a
wind of 5.5 m/s, and the leftmost part has a wind of 4.9 m/s. To consider
possible local differences in wind speed and wind direction, the weather data
from both cells in the AROME-Arctic weather simulation map is included as
exogenous variables.

Before proceeding to the prediction methodology, some technical features of
wind power generation must be highlighted. Wind power generation is a (RES)
technology that has a highly intermittent power generation due to the strong
dependency on weather conditions. In addition, the power generation from the
wind turbines is dependent on the _power curve_. The power curve from one of
the turbines in the wind farm analyzed here is given in Fig. 4.

Figure 4: Power curve from a wind turbine in the wind farm studied in this
work with the cut-in, rated and cut-off wind speed marked. The zero production
above 4 and below 25 m/s wind represent periods where the turbine is shut down
due to turbine failure or maintenance

As seen in Fig. 4, the wind turbine has zero production between 0 m/s to 3/4
m/s. At the cut-in wind speed around 4 m/s, the power generation increases
towards a wind speed of 12-13 m/s. At this wind speed, the power production
reaches the maximum limit (or the rated power) of 3MW around and produces at
maximum it suddenly drops towards zero in the cut-off interval around 25 m/s.
The reason for this drop is due to safety. If the blades at the turbine rotate
too quickly, it can damage the equipment. Therefore, the turbines are forced
to stop.

##  5 Methodology

###  5.1 Dataset configuration, preliminary analyses and deep learning models

In this work, the aim is to predict the day-ahead wind power generation. The
day-ahead market closes at 12:00 where the market participants must submit
their final bids to the electricity market about the expected amount of power
generation the next day. Therefore, in this work, the forecast horizon is 36
hours to consider all hours in the next day (12 hours + 24 hours).

First, in order to identify the time varying patterns in the time series, some
statistical analyses are performed by computing the autocorrelation function
(ACF) and the partial autocorrelation function (PACF). Fig. 5 shows the ACF
and PACF graphs for the time series of the wind power generation and wind
speed in the year of 2020.

Figure 5: ACF and PACF of the time series on wind power generation and wind
speed in the year of 2020. The correlation outside the standard deviations are
correlations and not a statistical fluke.

The ACF and PACF plots show no repetitive patterns, which indicate lack of
seasonality in the time series. Interestingly, the time series show short-term
dependencies as the ACF show correlations outside the 95% confidence interval
for all time lags (depicted as a blue area in Fig. 5). The values outside of
the blue cone are very likely actual correlations. The PACF plots do not show
any strong correlations except from the first three lags, and the correlations
in the ACF plot after lag 3, are indirect correlations that can be explained
by the first three time steps. Noteworthy, the ACF and PACF plots capture only
linear dependencies in the wind power generation and wind speed, but there
could be non-linear dependencies in the time series in addition. The deep
learning models used in this work provide the capability to capture such non-
linear relationships.

The correlation in the ACF and PACF plots shows that historical data on wind
power generation should be used as input to the deep learning models to make
predictions of the expected power generation. In addition, as wind power
generation is directly affected by the amount of wind that hits the turbines,
it is of interest to investigate which variables should be included (or
excluded) as covariates to potentially improve the prediction performance. The
exogenous variables included in addition to the historical data on wind power
generation are the historical measurement data on weather conditions and the
NWP from the AROME-Arctic model. The measured weather is included as input due
to the complex terrain in the region of the wind farm, and the weather model
with a spatial resolution of 2.5 km might be too coarse to capture local
variations in the wind. Therefore, the information from measured wind from the
nacelle could provide important additional information.

The prediction experiments are summarized as; 1) Use historical measurement
data on power generation with historical measurement data on weather, and the
36 hour ahead NWP as exogenous variables to predict the future power
generation. 2) Use historical measurement data on power generation, where only
measured weather data is used as exogenous variables. 3) Use historical
measurement data on power generation, where only the 36-hour ahead NWP data is
used as exogenous variables.

Figure 6: The historical data on wind power generation (P) togheter with
historical data on measured weather (MW) and predicted weather (PW) to predict
the future wind power generation (target prediction).

In Fig. 6 the example on the first prediction experiment where both measured
weather (MW) and predicted weather (PW) is used as additional inputs to
predict the future power generation is illustrated. In prediction experiments
2) or 3), either PW or MW is removed as input.

To make predictions with the different covariates, the DeepAR model proposed
by [26] is selected as it has outperformed other models in several recent
works on probabilistic forecasts with deep learning [20, 26, 4]. The inputs to
the DeepAR models are the time series values (historical wind power
generation) until $t-1$ and potential covariates (such as measured and
predicted weather) at time $t$.The covariates and the time series values are
thereafter concatenated before being fed into the internal units, which can
either be LSTM or GRU layers. The output from the internal units is fed into
two different linear layers. One for computing the mean, and one for computing
the standard deviation. When computing the standard deviation, the linear
layer is fed into a SoftPlus layer to ensure positive values. In the end, the
computed mean and standard deviation are used as input to a Gaussian
likelihood model where predictive samples are generated. During training, the
model learns by maximizing a Gaussian log-likelihood function and is optimized
via stochastic gradient descent with respect to the model parameters. The
DeepAR model is described more in detail in [26].

In former works the default DeepAR configuration with internal LSTM units has
been used [26, 20, 4]. In this work, the DeepAR model is tested with Gated
Recurrent Units (GRU) [5] in addition to LSTM as internal units.

In this work, two versions of a persistence model are served as benchmarks for
the DeepAR model. The persistence model assumes that the wind power generation
at a certain future time will be the same as it is when the forecast is made
(for instance, if the generation is 50MW when the prediction is made, the
persistence model assumes that the generation the next hour also is 50MW). The
persistence model has been a popular model to use as a benchmark as it is a
simple method to implement, and are often a difficult method to outperform,
especially in the range of 1-6 hour ahead [23]. In addition, a modified
version of the persistence model is served as a benchmark model in this work.
The authors in [22] proposed a modified persistence model for predicting wind
power generation. This model is a combination of the mean of the time series,
where the future value is weighted as a function of the correlation between
the current and average power, respectively. In prediction experiments with a
forecast horizon above 3 hours, the modified persistence model has been shown
to outperform the original one [22].

###  5.2 Prediction strategy

To train and evaluate the models, the time series is split into a training set
(85%) and a test set (15%). The training set is further divided into training
(80%) and validation (20%). The training set is used to fit the model
parameters by minimizing the prediction loss, and the validation set is used
to find the optimal configuration of the hyperparameters. Each model is set to
train on 500 epochs, and to avoid potential overfitting during training, an
early stopping rate is introduced. A grid search was performed to find the
optimal hyperparameter configuration for each model. We selected the
configuration that resulted in the highest accuracy on the test set. The
search space for the selected hyperparameters and the optimum configuration
for each model is presented in Tab. 2.

Table 2: Details of the hyperparameter search space and optimal configurations for different models Parameter | Search space | Optimal DeepAR${}_{\text{GRU}}$ | Optimal DeepAR${}_{\text{LSTM}}$
---|---|---|---
Rolling window length | 36, 72 | 36 | 36
Layers | 1, 2, 3 | 2 | 2
Hidden units | 32, 64 | 32 | 64
Dropout rate | 0.0, 0.1, 0.2 | 0.0 | 0.2
Learning rate |  $10^{-4}$,$10^{-3}$,$10^{-2}$ | $10^{-2}$ | $10^{-3}$

The searched hyperparameters include the length of the rolling window, which
is the number of time-points that the model gets to see before making the
prediction, number of hidden units, number of layers, dropout rate, and
learning rate. The rest of the hyperparameters were not fitted, and the
default values were used.

##  6 Experimental evaluation and results

###  6.1 Evaluation metrics

When making point predictions, the main aim is to minimize the discrepancy
between the predicted and true output, respectively. For the purpose of
probabilistic forecasts, it is more complicated. Here one wants to minimize
the loss between the predicted and true value for a specific quantile level
(or percentile which are the upper and lower bound of the PI), but at the same
time, one wants to have a PI that contains the true outcome. In addition, one
does not want to have a PI that is too wide as it will contain no useful
information, so the PI should be as sharp as possible, but still contain the
true values. In the following, some popular metrics that will be used in the
result section to provide the skill score of both point-and probabilistic
forecasts are described.

A widely used metric to evaluate the performance of _point forecasts_ is the
root mean squared error (RMSE) and the Normalized RMSE (NRMSE). The RMSE is
defined as

$RMSE=\sqrt{\frac{1}{n}\sum_{i=1}^{n}(\hat{y}_{i}-y_{i})^{2}},$ (2)

where $\hat{y}$ and $y$ are the predicted and true values. The RMSE measures
the discrepancy between the predicted values and observed values at time $i$,
over $n$ number of observations. The NRMSE relates the RMSE to the observed
average value in the observation period and is defined as:

$NRMSE=\frac{RMSE}{\overline{y}},$ (3)

where $\overline{y}$ is the average of the observed values in the time series.

Another popular method to evaluate the accuracy of point forecasts are the
mean absolute percentage error (MAPE). MAPE is defined as:

$MAPE=\frac{100}{n}\sum_{i=1}^{n}|\frac{y_{i}-\hat{y}_{i}}{y_{i}}|,$ (4)

The common feature of all these metrics (RMSE, NRMSE, and MAPE) is that the
lower value, the higher accuracy.

The purpose of _probabilistic forecasts_ is to have a PI that fulfills the
calibration and sharpness criteria. In this section, some common metrics that
are used to describe the performance of the PIs in terms of the sharpness and
calibration criteria are presented.

The pinball-loss (PL), or the quantile-loss (QL) function is a common metric
to measure the performance of the PI. For each quantile level, the PL function
returns a value that can be interpreted as the accuracy of a quantile
forecasting model. Let $q$ be the target quantile, $y$ the real value, and
$\hat{y}$ the quantile forecast, then the PL for quantile $q$ can be written
as:

$PL_{q}(y,\hat{y})=\left\\{\begin{array}[]{lr}(y-\hat{y})q&y\geq\hat{y}\\\
(\hat{y}-y)(1-q)&\hat{y}>y\end{array}\right.$

The PL function penalizes the forecast if the model is over or under-
predicting depending on the quantile level that is computed. The large
quantile level will be more penalized for under-predicting than a low quantile
level. Similarly, a low quantile level will be more penalized for over-
predicting than a large quantile level. This makes sense as in the high
quantile level case, one expects most of the observed values to be smaller
than the predictions, and at the low quantile level one expects most of the
values to be above the predicted values. Similar to the point forecast
metrics, the lower PL, the more accurate the quantile forecast is. If
computing the PL over a set of different quantiles levels, the final quantile
loss result will be the average of all levels (often denoted as QLm).

In [26], the authors evluated the sharpness, or the quantile risk for
different quantile levels by considering the normalized sum, wQL of quantile
losses. The wQL for a quantile $q$ is computed as all pinball losses divided
by the sum of true output:

$wQL_{q}(y,\hat{y})=2\frac{\sum_{i}PL_{q}(y_{i},\hat{y}_{i})}{\sum_{i}y_{i}}$
(5)

A low wQL indicate a sharper PI.

Besides sharpness, it is important to compute how calibrated the PI is. A
popular calibration metric for probabilistic forecasts is the Prediction
Interval Coverage Probability (PICP) [12]. The PICP is employed to compute the
probability that the true outcome is within the PIs. The PCIP is defined as

$PCIP=\frac{1}{n}\sum_{i=1}^{n}u_{i},$ (6)

where $n$ is the total number of samples. When the true output is within the
upper and lower bound, $u_{i}=1$, otherwise $u_{i}=0$. To obtain a well-
calibrated PI, the coverage should be close as possible to the PI that is
specified. For instance, for a 95% PI, the PCIP should be 0.95.

The final metric that is used to measure the quality of the PIs is the Mean
Scaled Interval Score (MSIS). This metric was used as the preferred one in the
M4 forecasting competition, where 100,000 time series and 61 forecasting
methods were compared [19]. The MSIS score was proposed by [9] and evaluate
the performances of the generated PIs as

$MSIS=\frac{1}{h}\times\frac{\sum_{t=n+1}^{n+h}(U_{t}-L_{t})+\frac{2}{\alpha}(L_{t}-Y_{t})\mathbbm{1}Y_{t}<L_{t}+\frac{2}{\alpha}(Y_{t}-U_{t})\mathbbm{1}Y_{t}>U_{t}}{\frac{1}{n-m}\sum_{t=m+1}^{n}|Y-Y_{t-m}|},$
(7)

where the lower and upper bounds of the PI are denoted by $L_{t}$ and $U_{t}$,
respectively. $Y_{t}$ is the future observed values, $h$ is the forecast
horizon, and $\mathbbm{1}$ is the indicator function (1 if $Y_{t}$ is within
the PI and 0 otherwise). Here $\alpha$ is the significance level, and for a
95% PI, $\alpha$ is set to 0.05. The MSIS metric deals with the sharpness and
calibration criteria. It both penalize wide PIs (since $U_{t}$ and $L_{t}$
will be large), and penalize non-coverage. Here, a lower MSIS score indicates
a better PI in terms of sharpness and calibration.

###  6.2 Results and discussion

In Tab. 3 the results are given. In this work, the 95% PI is computed as it is
a widely used choice for economic, financial, and energy-related forecasting
applications [19]. The point prediction scores for predicting the 36-hour
ahead power generation is given in terms of NRMSE and MAPE (eq. (7) and (8)).
For probabilistic forecasts, the scores are given in terms of MSIS, PCIP, and
the wQL (eq. (9)-(11)). In Tab. 3, the mean wQL for quantile level 0.025 and
0.975 is computed.

Table 3: 36-hour ahead prediction scores with different dataset configurations
Configuration 1: Measured and predicted weather (wind speed + wind direction)
---
Model | NRMSE | MAPE | MSIS | PCIP2.5 | PCIP97.5 | mean wQL
DeepAR${}_{\text{GRU}}$ | 0.17 | 0.16 | 3.56 | 0.00 | 1.00 | 0.028
DeepAR${}_{\text{LSTM}}$ | 0.16 | 0.15 | 2.53 | 0.027 | 0.972 | 0.020
Configuration 2: Measured weather (wind speed + wind direction)
DeepAR${}_{\text{GRU}}$ | 0.45 | 0.41 | 10.73 | 0.00 | 0.55 | 0.084
DeepAR${}_{\text{LSTM}}$ | 0.40 | 0.35 | 10.16 | 0.00 | 0.55 | 0.080
Configuration 3: Predicted weather (wind speed + wind direction)
DeepAR${}_{\text{GRU}}$ | 0.24 | 0.21 | 5.32 | 0.00 | 1.00 | 0.042
DeepAR${}_{\text{LSTM}}$ | 0.29 | 0.25 | 4.03 | 0.00 | 0.972 | 0.032
Benchmark models
Persistence | 1.06 | 6.17 | - | - | - | -
Modified Persistence | 0.79 | 5.09 | - | - | - | -

Tab. 3 show that the DeepAR model with LSTM units resulted in the best
accuracy for all experiments. The Persistence models obtained the worst
prediction performance. The probabilistic scores are not reported for the
persistence models as they do not provide the capability to make probabilistic
forecasts.

The best performance was obtained with the dataset configuration where both
measured and predicted weather was included as exogenous variables. With this
configuration, the DeepAR${}_{\text{LSTM}}$ model computed the sharpest PI as
the mean wQL and the MSIS are low. In addition, this model computed a PI that
is well-calibrated as the PCIP is close to the quantile levels that are
specified. For instance, for PCIP2.5, the computed PI resulted in a PCIP of
2.7%, which indicates that it is a probability of 2.7% that the true outcome
is below this quantile level. The PCIP for the upper boundary is also close to
the specified quantile level.

Using only measured weather as exogenous variables resulted in the worst
prediction performance for both models. The computed PI for this configuration
is neither sharp nor calibrated, as the MSIS and wQL are large and the
PCIP97.5 value is 0.55. This shows that the true outcome is below the upper PI
boundaries only 55% of the time, which is not adequate in a 95% PI interval.

When the NWP is used as the only exogenous variable, acceptable prediction
performances were obtained for the day-ahead predictions. However, indicated
by the MSIS and wQL scores, the PI is less sharp in this case. In addition,
the PI is not fully calibrated for both models as there is zero probability
that the true outcome is below the lower PI boundary.

The results indicate that combining historical measurement data and NWPs helps
improve the day-ahead prediction of wind power generation. This shows that
adding historical data on measured weather allows the DeepAR model to auto-
correct systematic biases in the NWPs.

In Fig. 7 the 36-hour ahead predictions with the best DeepAR${}_{\text{LSTM}}$
model in terms of calibration and sharpness is provided (configuration 1 with
measurements and NWPs combined). In addition, the DeepAR${}_{\text{LSTM}}$ for
configuration 2 and 3 is shown. These are two examples of PI results that are
miscalibrated and have low sharpness. In the following illustration, the 50%
PI and the median prediction are included.

Figure 7: 36 hour ahead predictions of wind power generation. The uppermost
Figure show the DeepAR${}_{\text{LSTM}}$ model where the measured wind and NWP
are included as exogenous variables. The Figure in the middle shows the
DeepAR${}_{\text{LSTM}}$ model for configuration 2 where only measured weather
is used. This shows a low degree of sharpness and are miscalibrated. The
lowermost Figure shows the result from the DeepAR${}_{\text{LSTM}}$ model
where only the predicted wind is used as exogenous variables. This result show
a well calibrated PI, but low degree of sharpness as the PI boundaries is wide
compared to the first graph, and provide less useful information about the
variation in expected output.

The red line represents observed values, while the green line represents the
median value (or point forecast) of the probabilistic forecast. It is seldom a
perfect match between the predicted point forecast and the actual value. This
is not surprising as making accurate point forecasts of the day-ahead wind
power generation is a very difficult problem. The green nuances in the graphs
represent the different PIs. The uppermost graph in Fig. 7 show the resulting
prediction with the DeepAR${}_{\text{LSTM}}$ model using configuration 1. In
this prediction, the 95% PI indicated by the bright green color shows that the
actual measurements fall within the interval approximately 95% of the time and
therefore show a well-calibrated PI. The observed values fall outside the
interval at some points in the period between 12:00 and 18:00 the 30-Dec. This
is acceptable, as given by the 95% requirement, some values may fall outside
the range. The rest of the time the observed values are within the 95% PI. The
graph also shows that the model is more confident at the beginning of the
prediction period as the PI boundaries are very sharp, but the uncertainties
increase as longer ahead in the future the predictions are made.

On contrary, the graph in the middle shows the DeepAR${}_{\text{LSTM}}$ model
for the configuration using only measured wind. This shows a low degree of
sharpness and is not calibrated as several observations are outside the PI
boundaries. The lowermost graph shows the result from the
DeepAR${}_{\text{LSTM}}$ model where only the predicted wind is used as
exogenous variables. This result shows a well-calibrated PI as most
observations are within the boundaries, but the PI is less informative as is
it wide and consequently provides less useful information about the variations
in output.

Due to technical limitations, the wind power plant can maximum produce 54 MW
(18 turbines $\times$3 MW). However, all models in Fig. 7 compute a PI that
has upper boundaries that exceed this value. This is not possible due to the
theoretical maximum of 54 MW, and one can be sure that there is zero
probability that the total wind power generation will exceed this value.
Therefore, all values above 54 MW in the upper boundary of the computed 95%
PIs can be replaced by the theoretical maximum power generation. In Fig. 8 the
modified 95% PI of the DeepAR${}_{\text{LSTM}}$ with the best prediction
performance is provided. Here all values above the theoretical maximum are
replaced by the theoretical maximum of 54MW.

Figure 8: 36 hour ahead probabilistic forecast where the upper boundaries of
PI are adjusted with respect to the theoretical maximum power generation of 54
MW.

Fig. 8 show that replacing the upper PI boundaries with the theoretical
maximum power output from the wind farm generates a PI that is much sharper
and is very accurate at several time steps. This indicates that when making a
probabilistic forecast of power output from wind farms, knowledge regarding
technical limitations could contribute to achieving even more accurate PIs.
The scores of the original and adjusted 95% PI for the
DeepAR${}_{\text{LSTM}}$ for configuration 1 is given in Tab. 4.

Table 4: Probabilistic performance scores with the original and adjusted 95% PI | Original 95% PI | Adjusted 95% PI
---|---|---
Model | MSIS | mean wQL | MSIS | mean wQL
DeepAR${}_{\text{LSTM}}$ | 2.53 | 0.020 | 1.16 | 0.013

From Tab. 4 the MSIS and mean wQL show that the adjusted PI is sharper than
the original PI. The PCIP for the upper and lower boundaries are the same for
both PIs and are not reported.

##  7 Conclusions

This work tackled the problem of making accurate forecasts of RES technologies
that has a highly intermittent nature in electricity generation. The increased
share of such technologies in the electricity market increases the
uncertainties about expected future power generation. To account for such
uncertainties, deep learning models can be applied to make probabilistic
forecasts. In this work, a day-ahead probabilistic forecast was performed
using three different dataset configurations. The deep learning model selected
was the DeepAR model which has shown promise in several tasks related to
probabilistic forecasts in energy applications. In this work, the DeepAR model
was used with different internal units. The prediction performances obtained
for each model on the different configurations were compared in terms of
obtained quality on the PI.

The DeepAR model with LSTM units obtained the most accurate prediction
performance for all experiments. Among the different configurations, the best
performance in terms of sharpness and calibration was obtained when both
historical data on measured weather (wind direction + wind speed) _and_ the
NWPs (wind direction + wind speed) were used as exogenous variables. When
using only the predicted weather as exogenous variables, worse results were
obtained. This could be due to the highly complex topography of the region
where the wind farm is located, which makes it increasingly difficult to make
accurate weather forecasts. Using the historical measured wind allows the deep
learning model to auto-correct the potential failure sources in the NWPs.

The proposed methodology shows the importance of using measurements data to
account for systematic biases in the weather forecasts. As RES technologies
are directly affected by the current weather, minimizing the failure sources
from weather predictions is something that could provide more accurate
forecasts of expected power generation. In the end, using the information
about technical limitations of the wind power plant to adjust the upper
boundary resulted in a much sharper PI. This illustrates the importance of
using physical knowledge about the power system in addition to machine
learning techniques.

## Acknowledgments

O.F.E, M.C, and F.M.B acknowledge the support from the research project
“Transformation to a Renewable & Smart Rural Power System Community (RENEW)”,
connected to the Arctic Centre for Sustainable Energy (ARC) at UiT-the Arctic
University of Norway through Grant No. 310026.

## References

  * [1] M. Afrasiabi, M. Mohammadi, M. Rastegar, and S. Afrasiabi.  Advanced deep learning approach for probabilistic wind speed forecasting.  IEEE Transactions on Industrial Informatics, 17(1):720–727, 2020\.
  * [2] A. Ahmed Mohammed and Z. Aung.  Ensemble learning approach for probabilistic forecasting of solar power generation.  Energies, 9(12):1017, 2016.
  * [3] M. Al-Gabalawy, N. S. Hosny, and A. R. Adly.  Probabilistic forecasting for energy time series considering uncertainties based on deep learning algorithms.  Electric Power Systems Research, 196:107216, 2021.
  * [4] A. Alexandrov, K. Benidis, M. Bohlke-Schneider, V. Flunkert, J. Gasthaus, T. Januschowski, D. C. Maddix, S. S. Rangapuram, D. Salinas, J. Schulz, et al.  Gluonts: Probabilistic and neural time series modeling in python.  J. Mach. Learn. Res., 21(116):1–6, 2020.
  * [5] F. M. Bianchi, E. Maiorino, M. C. Kampffmeyer, A. Rizzi, and R. Jenssen.  Recurrent neural networks for short-term load forecasting: an overview and comparative analysis.  2017\.
  * [6] A. Brusaferri, M. Matteucci, P. Portolani, and A. Vitali.  Bayesian deep learning based method for probabilistic forecast of day-ahead electricity prices.  Applied Energy, 250:1158–1175, 2019.
  * [7] Y. Chen, Y. Kang, Y. Chen, and Z. Wang.  Probabilistic forecasting with temporal convolutional neural network.  Neurocomputing, 399:491–501, 2020.
  * [8] J. Gasthaus, K. Benidis, Y. Wang, S. S. Rangapuram, D. Salinas, V. Flunkert, and T. Januschowski.  Probabilistic forecasting with spline quantile function rnns.  In The 22nd international conference on artificial intelligence and statistics, pages 1901–1910. PMLR, 2019.
  * [9] T. Gneiting and A. E. Raftery.  Strictly proper scoring rules, prediction, and estimation.  Journal of the American statistical Association, 102(477):359–378, 2007.
  * [10] K. Higashiyama, Y. Fujimoto, and Y. Hayashi.  Feature extraction of nwp data for wind power forecasting using 3d-convolutional neural networks.  Energy Procedia, 155:350–358, 2018.
  * [11] T. Hong and S. Fan.  Probabilistic electric load forecasting: A tutorial review.  International Journal of Forecasting, 32(3):914–938, 2016.
  * [12] Q. Huang and S. Wei.  Improved quantile convolutional neural network with two-stage training for daily-ahead probabilistic forecasting of photovoltaic power.  Energy Conversion and Management, 220:113085, 2020.
  * [13] H. Jin, L. Shi, X. Chen, B. Qian, B. Yang, and H. Jin.  Probabilistic wind power forecasting using selective ensemble of finite mixture gaussian process regression models.  Renewable Energy, 174:1–18, 2021.
  * [14] R. G. Kavasseri and K. Seetharaman.  Day-ahead wind speed forecasting using f-arima models.  Renewable Energy, 34(5):1388–1393, 2009.
  * [15] V. Kuleshov, N. Fenner, and S. Ermon.  Accurate uncertainties for deep learning using calibrated regression.  In International Conference on Machine Learning, pages 2796–2804. PMLR, 2018.
  * [16] J. L. G. La Salle, J. Badosa, M. David, P. Pinson, and P. Lauret.  Added-value of ensemble prediction system on the quality of solar irradiance probabilistic forecasts.  Renewable Energy, 162:1321–1339, 2020.
  * [17] H. Liu, Z. Duan, C. Chen, and H. Wu.  A novel two-stage deep learning wind speed forecasting method with adaptive multiple error corrections and bivariate dirichlet process mixture model.  Energy Conversion and Management, 199:111975, 2019.
  * [18] Y. Liu, H. Qin, Z. Zhang, S. Pei, Z. Jiang, Z. Feng, and J. Zhou.  Probabilistic spatiotemporal wind speed forecasting based on a variational bayesian deep learning model.  Applied Energy, 260:114259, 2020.
  * [19] S. Makridakis, E. Spiliotis, and V. Assimakopoulos.  The m4 competition: 100,000 time series and 61 forecasting methods.  International Journal of Forecasting, 36(1):54–74, 2020.
  * [20] A. Mashlakov, T. Kuronen, L. Lensu, A. Kaarna, and S. Honkapuro.  Assessing the performance of deep learning models for multivariate probabilistic energy forecasting.  Applied Energy, 285:116405, 2021.
  * [21] N. Mazzi and P. Pinson.  Wind power in electricity markets and the value of forecasting.  In Renewable Energy Forecasting, pages 259–278. Elsevier, 2017\.
  * [22] C. Moehrlen.  Uncertainty in wind energy forecasting.  PhD thesis, University College Cork, 2004.
  * [23] C. Monteiro, R. Bessa, V. Miranda, A. Botterud, J. Wang, G. Conzelmann, et al.  Wind power forecasting: State-of-the-art 2009.  Technical report, Argonne National Lab.(ANL), Argonne, IL (United States), 2009.
  * [24] N. Nguyen and B. Quanz.  Temporal latent auto-encoder: A method for probabilistic multivariate time series forecasting.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 9117–9125, 2021.
  * [25] O. Ruhnau, P. Hennig, and R. Madlener.  Economic implications of forecasting electricity generation from variable renewable energy sources.  Renewable Energy, 161:1318–1327, 2020.
  * [26] D. Salinas, V. Flunkert, J. Gasthaus, and T. Januschowski.  Deepar: Probabilistic forecasting with autoregressive recurrent networks.  International Journal of Forecasting, 36(3):1181–1191, 2020.
  * [27] M. Sun, C. Feng, and J. Zhang.  Multi-distribution ensemble probabilistic wind power forecasting.  Renewable Energy, 148:135–149, 2020.
  * [28] J.-F. Toubeau, J. Bottieau, F. Vallée, and Z. De Grève.  Deep learning-based multivariate probabilistic forecasting for short-term scheduling in power markets.  IEEE Transactions on Power Systems, 34(2):1203–1215, 2018.
  * [29] H. Wang, H. Yi, J. Peng, G. Wang, Y. Liu, H. Jiang, and W. Liu.  Deterministic and probabilistic forecasting of photovoltaic power based on deep convolutional neural network.  Energy conversion and management, 153:409–422, 2017.
  * [30] H.-z. Wang, G.-q. Li, G.-b. Wang, J.-c. Peng, H. Jiang, and Y.-t. Liu.  Deep learning based ensemble approach for probabilistic wind power forecasting.  Applied energy, 188:56–70, 2017.
  * [31] J. Wang, Y. Song, F. Liu, and R. Hou.  Analysis and application of forecasting models in wind power integration: A review of multi-step-ahead wind speed forecasting models.  Renewable and Sustainable Energy Reviews, 60:960–981, 2016.
  * [32] J. Wang, S. Wang, and Z. Li.  Wind speed deterministic forecasting and probabilistic interval forecasting approach based on deep learning, modified tunicate swarm algorithm, and quantile regression.  Renewable Energy, 179:1246–1261, 2021.
  * [33] X. Wang, P. Guo, and X. Huang.  A review of wind power forecasting models.  Energy procedia, 12:770–778, 2011.
  * [34] L. Xiang, J. Li, A. Hu, and Y. Zhang.  Deterministic and probabilistic multi-step forecasting for short-term wind speed based on secondary decomposition and a deep learning method.  Energy Conversion and Management, 220:113098, 2020.
  * [35] H. Zhang, Y. Liu, J. Yan, S. Han, L. Li, and Q. Long.  Improved deep mixture density network for regional wind power probabilistic forecasting.  IEEE Transactions on Power Systems, 35(4):2549–2560, 2020.
  * [36] W. Zhang, H. Quan, and D. Srinivasan.  An improved quantile regression neural network for probabilistic load forecasting.  IEEE Transactions on Smart Grid, 10(4):4425–4434, 2018.
  * [37] Y. Zhang, Y. Li, and G. Zhang.  Short-term wind power forecasting approach based on seq2seq model using nwp data.  Energy, 213:118371, 2020.
  * [38] Y. Zhang, J. Wang, and X. Wang.  Review on probabilistic forecasting of wind power generation.  Renewable and Sustainable Energy Reviews, 32:255–270, 2014.
  * [39] W. Zhao, Y.-M. Wei, and Z. Su.  One day ahead wind speed forecasting: A resampling-based approach.  Applied Energy, 178:886–901, 2016.
  * [40] X. Zhao, J. Liu, D. Yu, and J. Chang.  One-day-ahead probabilistic wind speed forecast based on optimized numerical weather prediction data.  Energy Conversion and Management, 164:560–569, 2018.
  * [41] S. Zhu, X. Yuan, Z. Xu, X. Luo, and H. Zhang.  Gaussian mixture model coupled recurrent neural networks for wind speed interval forecast.  Energy Conversion and Management, 198:111772, 2019.


# Spherical Feature Pyramid Networks For Semantic Segmentation

Varun Anand, Thomas Walker, Pavlos Andreadis
<EMAIL_ADDRESS>
University of Edinburgh

###### Abstract

Semantic segmentation for spherical data is a challenging problem in machine
learning since conventional planar approaches require projecting the spherical
image to the Euclidean plane. Representing the signal on a fundamentally
different topology introduces edges and distortions which impact network
performance. Recently, graph-based approaches have bypassed these challenges
to attain significant improvements by representing the signal on a spherical
mesh. Current approaches to spherical segmentation exclusively use variants of
the UNet architecture, meaning more successful planar architectures remain
unexplored. Inspired by the success of feature pyramid networks (FPNs) in
planar image segmentation, we leverage the pyramidal hierarchy of graph-based
spherical CNNs to design spherical FPNs ($S^{2}$FPN). Our $S^{2}$FPN models
show consistent improvements over spherical UNets, whilst using fewer
parameters. On the Stanford 2D-3D-S dataset, our models achieve state-of-the-
art performance with an mIOU of 48.75, an improvement of 3.75 IoU points over
the previous best spherical CNN.

## Introduction

In recent years spherical image data has become increasingly common due to
omnidirectional imaging and LiDAR sensors used in autonomous vehicles (Geiger
et al. 2013; Chang et al. 2017). However, spherical data is encountered across
all disciplines as atomic charge distributions (Hoja et al. 2021), brain
activity (Feng et al. 2018), climate patterns (Neale 2012), and the cosmic
microwave background (Drake and Wright 2020; Defferrard et al. 2020). The task
of semantic segmentation frequently appears in these spherical domains,
perhaps most notably in the visual systems employed by autonomous vehicles.
For planar segmentation, feature pyramids are an established component of
state-of-the-art segmentation models (Tan, Pang, and Le 2019; Lin et al. 2017;
Yuan et al. 2020; Zhu et al. 2021). We introduce a Feature Pyramid Network
(FPN) for spherical images. As part of this design, we present an improved
scheme to transition spherical signals between pyramid levels, and perform an
ablation study over these design choices.

Early attempts to utilize deep learning models for segmenting spherical images
involved mapping them to the Euclidean plane and using well-studied planar,
convolutional neural networks (CNN) (Su and Grauman 2017; Kim, De Campos, and
Hilton 2016; Fan et al. 2017; Yu et al. 2016). Almost exclusively, these
methods use the equirectangular projection, mapping the latitude and longitude
of the spherical images to Euclidean grid coordinates. However, this mapping
introduces polar discontinuities, greatly distorting objects in a scene
according to their proximity to the poles (Cohen et al. 2018). Objects close
to the top and bottom edge of the projection have their scale exaggerated, a
distortion which planar segmentation models are known to be sensitive to (Yuan
et al. 2020; Lin et al. 2017). In fact, the introduction of boundaries by
planar representations is known to harm classification accuracy of close-by
pixels (Yuan et al. 2020).

Collectively, these drawbacks have motivated a branch of literature aimed at
generalizing successful planar deep learning techniques to operate natively on
the sphere. The primary effort has been directed towards designing
convolutions that directly consume spherical signals without the need for
destructive projections. One of the biggest challenges is the lack of a
perfectly symmetrical grid to discretize spherical signals (Cohen et al. 2018;
Gorski et al. 2005). Without a spatially consistent notion of a ‘pixel’, there
exists no clear way to define a traditional spatial convolution operation.
Further still, there is no canonical choice of filter orientation on the
sphere. This results in undesirable path-dependent convolutions, since spatial
kernels will change their orientation depending on the choice of path taken
during convolution (Cohen et al. 2019). The seminal work on fully spherical
CNNs (Cohen et al. 2018) led to a line of research on spectral methods. These
approaches can bypass the aforementioned challenges by breaking the signal
down into spherical harmonics and performing convolution in frequency space as
a spectral dot product. However, the repeated computation of Fourier
transforms and their inverses are very computationally expensive and scale
poorly to high resolution images (Cohen et al. 2018; Esteves et al. 2018).

Recently, the works of (Jiang et al. 2019; Defferrard et al. 2020; Shen et al.
2022) bypass these challenges by sampling the spherical signal onto the nodes
of an icosahedral graph, an approximation of a sphere of varying “levels” (in
analogy to image resolution, in the planar case, see figure 1). Two of these
methods parameterize spherical convolutions as a linear combination of partial
differential operators (PDOs). These PDO-based models are efficient and have
achieved competitive results on benchmark segmentation data sets (Jiang et al.
2019; Shen et al. 2022).

We continue the trend of generalizing successful planar techniques to the
spherical domain. In planar segmentation models, the importance of multi-scale
features is long established, even before the success of deep CNNs (Dalal and
Triggs 2005; Bay, Tuytelaars, and Gool 2006). Lin et al. (2017) introduced
Feature Pyramid Networks (FPNs), that have now become common-place in state-
of-the-art networks (Zhu et al. 2021; Tan, Pang, and Le 2019; Yuan et al.
2020; Lin et al. 2017). FPNs leverage the intrinsic receptive field hierarchy
of a CNN to extract semantic feature maps across various scales, which enable
them to detect objects of varying shapes and sizes. For our approach, we
leverage the varying receptive field of PDOs on an icosahedral graph to design
Spherical Feature Pyramid Networks for graph-based spherical models.

Our contributions are as follows:

  1. 1.

We design spherical feature pyramid networks and present results for a range
of model complexities. Our best model improves the state-of-the-art on
Stanford 2D-3D-S (Armeni et al. 2017) by a significant margin.

  2. 2.

We present optimal pooling and up-sampling routines for constructing
icosahedral feature pyramids.

Figure 1: Progressive refinement of icosahedral mesh from $\ell=0$ to
$\ell=5$.

## Related Work

### Graph-based Spherical CNNs

Despite the significant progress in making spectral approaches more scalable
(Kondor, Lin, and Trivedi 2018; Cobb et al. 2021; McEwen, Wallis, and Mavor-
Parker 2022), graph-based spherical CNNs still emerge as the most efficient
approach to processing spherical signals. However, a notable cost of
approximating the sphere as graph is the diminished ability to facilitate
rotational equivariance. The most recent research efforts have been in
attaining more uniform spherical samplings, and designing rotationally
equivariant convolutions (Perraudin et al. 2019; Defferrard et al. 2020; Shen
et al. 2022). Hence, these works focus on the signal domain and the layer-wise
operations, not considering improvements to the architectural design they
employ for the task of segmentation. Our work is orthogonal to surrounding
research, and builds on the PDO-based approach introduced by (Jiang et al.
2019) to utilize feature pyramids for segmentation.

### Multi-scale Features

Since objects in the real world have different shapes and sizes, it is
desirable for networks to be able to extract multi-scale features that capture
information across all object scales. Even before the success of deep CNNs,
the literature on planar image segmentation and object detection has a long
history of methods using Gaussian image pyramids (Anderson et al. 1984) to
identify objects across scales. These image pyramids were combined with hand-
engineered features (Dalal and Triggs 2005; Lowe 2004) to achieve state-of-
the-art results at the time. Though CNNs have replaced engineered features,
the need for multi-scale features still remains. Early CNN-based architectures
(Sermanet et al. 2013; Liu et al. 2016) used the pyramidal feature hierarchy
of a CNN to do object detection. However, these networks weight semantic
information unequally across the levels of the pyramid, and perform poorly on
small objects. Feature Pyramid Networks (FPN) (Lin et al. 2017) was the
seminal work to introduce a top-down pathway, to extract high-level semantic
information uniformly at multiple scales. FPNs simultaneously leverage a CNNs
receptive field and semantic hierarchies efficiently, and have been widely
adopted in succesful planar object detection and segmentation systems
(Kirillov et al. 2019), (Tan, Pang, and Le 2019).

## Preliminaries

In this section we provide an overview of the core mathematical components of
graph-based spherical CNNs which use PDOs.

### Icosahedral Mesh

We define our signal domain, the icosahedral spherical mesh. As originally
proposed by (Baumgardner and Frederickson 1985), we can accurately discretize
a sphere by recursively applying a subdivision routine initialized on an
icosahedron. In each iteration we add vertices at the midpoints between each
pre-existing vertex, and then re-project then to be unit distance to the
origin. Fully connecting the new vertices subdivides each original face into
four new triangles. Following this scheme we can naturally define upsampling
and downsampling algorithms which are analogous to different resolutions in
planar images, see figure 1. We refer to the original icosahedron as a level-0
($\ell=0$) mesh. Each iteration of this subdivision routine increases the
level $\ell+=1$. The number of vertices $n_{v}$ scales with level $\ell$
according to,

$n_{v}=10\cdot 4^{\ell}+2.$ (1)

Our input and output spherical signals are level-5 meshes, with the minimum
level-0 mesh at the lowest level of the network.

### MeshConv Operation

We apply the ”MeshConv” operation as defined in (Jiang et al. 2019). Given the
spherical signal $F$, and a partial differential operator (PDO) kernel
$G_{\theta}$, we parameterize a convolution at each vertex as,

$F\ast
G_{\theta}=\theta_{0}IF+\theta_{1}\nabla_{x}F+\theta_{2}\nabla_{y}F+\theta_{3}\nabla^{2}F,$
(2)

where $I$ is the identity operator and $\nabla_{x}$ and $\nabla_{y}$ are
gradients in the east-west and north-south directions respectively.
Collectively, the constituent PDOs capture diffusion properties of the signal
at each vertex. First and second differential operators, as well as the
Laplacian operator, and computed using the Libigl library (Jacobson, Panozzo
et al. 2018), and follow from results in discrete differential geometry (Crane
2015). Namely, we represent scalar functions on a mesh as a piece-wise linear
function with values defined at each mesh vertex:

$f(\mathbf{x})\approx\sum\limits_{i=1}^{n}\phi_{i}(\mathbf{x})\,f_{i},$ (3)

where $\phi_{i}$ is a piece-wise linear basis function defined on the mesh.
For each triangle $\phi_{i}$ is a linear function which is one only at vertex
$\textbf{x}=v_{i}$ and zero otherwise.

For future discussion, note that with this prescription we are making the
choice to perform bilinear interpolation to find the function values between
the vertices of the spherical signal. We can consider gradients of piecewise
linear functions as simply sums of gradients of the hat functions:

$\nabla
f(\mathbf{x})\approx\nabla\sum\limits_{i=1}^{n}\phi_{i}(\mathbf{x})\,f_{i}=\sum\limits_{i=1}^{n}\nabla\phi_{i}(\mathbf{x})\,f_{i}.$
(4)

Around a given vertex $v_{i}$, these basis gradients $\nabla\phi_{i}$ are zero
everywhere on the mesh except the surrounding faces which contain $v_{i}$. For
the spherical gradient signal itself, we sum the gradients on these faces
weighted by each face’s area. Following (Jiang et al. 2019), at each vertex we
take the dot product of this gradient with north-south and east-west basis
vector fields, separating these components for individual parameterization
within the PDO kernel.

Within this framework, the signal’s Laplacian can similarly be computed using
the cotangent formulation of Laplace-Beltrami operator,

$\Delta
f(v_{i})=\frac{1}{2A_{i}}\sum\limits_{{v_{j}\in\mathcal{N}(v_{i})}}{(\cot\alpha_{ij}+\cot\beta_{ij})(f(v_{j})-f(v_{i}))},$
(5)

where $\mathcal{N}(v_{i})$ is the set of vertices in the 1-ring neighborhood
of $v_{i}$, $A_{i}$ is cell area of vertex $v_{j}$ and
$\alpha_{ij},\beta_{ij}$ are referred to as the cotangent angles.

Figure 2: The scale of the the one-ring neighbourhood on level-1 (red),
level-2 (yellow), level-3 (green) and level-4 (blue) meshes, displayed on a
level-4 icosahedral signal.

Note that these computations use information from the one-ring neighbourhood
of each vertex to compute features. The size of this neighbourhood is larger
at lower levels of the mesh (see Figure 2). This property enables MeshConv
operating at lower mesh levels to have a larger receptive field on the input
spherical signal. We leverage this receptive field hierarchy to construct
Spherical Feature Pyramid Networks ($S^{2}$FPNs).

## Methodology

In this section, we describe the architecture of our $S^{2}$FPN model,
followed by a description of our design choices and their impact on the
receptive field of the models.

### Feature Pyramid Networks

Our spherical FPN consists of three stages which we detail below. Figure 3
provides an overview of our overall architecture.

Encoder: We use a modified version of the encoder used by the Spherical UNet
(Jiang et al. 2019). Starting with a level-5 mesh, we apply bottlenecked
ResBlocks similar to until the lowest mesh level of the model. Each ResBlock
consists of a MeshConv operation sandwiched between standard 1x1 convolutions,
with average pooling applied after the MeshConv. Our model differs from the
UNet in terms of the order of operations and the choice of downsampling. This
is explained further in section Up/Down-sampling Spherical Signals. The number
of channels is set to 32 at level-5 and is doubled on each transition to a
lower level mesh. At level-0, the channel width is capped at 512 (equal to
level-1), to avoid a drastic increase in parameters. Such a design results in
a semantic hierarchy, with lower mesh levels extracting higher-level semantic
information, at coarser resolutions.

Pyramid: To construct our feature pyramid we follow (Lin et al. 2017) and add
a 256-channel top-down pathway with lateral connections from the encoder. This
pathway upsamples features from the lowest level of the encoder, building
semantically rich feature maps across all scales. Each upsampling block
consists of bilinear upsampling, followed by addition with feature maps of the
same level from the encoder (Upsamp Block). The lateral connections use a 1x1
convolution to match the number of channels in the pyramid. By keeping the
number of channels fixed across all pyramid levels, we treat each scale
equivalently, weighting semantic information equally across all scales.

Head: While the design of the encoder and the pyramid are generic and can be
used for any task, we use a classification head designed for semantic
segmentation following Kirillov et al. (2019). Starting from the lowest mesh
level, every pyramid feature map is upsampled using bilinear upsampling
followed by MeshConv, to the output mesh level (CrossUpSamp). This results in
a set of level-5 feature maps which are element-wise summed and passed through
a final MeshConv layer for prediction. We set the the number of channels to be
128 across all the head feature maps.

Finally, all our MeshConv operations are followed by batch normalization and
ReLU. Our pyramid differs from Lin et al. (2017), who empirically found non-
linearities to not have an effect on their task. We found batch normalization
to be necessary to prevent explosions in MeshConv’s differential operators.

Figure 3: Overview of the L0:5 model used for the Stanford 2D-3D-S dataset.
The numbers represent the number of channels at that level.

### Up/Down-sampling Spherical Signals

Between levels of the feature pyramid the spherical image is up/down-sampled.
We diverge from the seminal work on PDO-based spherical CNNs (Jiang et al.
2019) in how we compute new vertex values when transitioning between levels,
as well as the order of operations in our “Up/Down-Samp” blocks.

#### Down-sampling

(Jiang et al. 2019) downsample by sampling the signal values only at the nodes
shared by the subsequent, lower-level graph. Instead, we average pool vertex
values in a 1-ring neighborhood around these shared vertices to ensure no
information is lost. By taking information from a 1-ring neighbourhood, we
also increase the receptive field of vertices at each level.

#### Up-sampling

In each iteration of the icosahedral sub-division routine, additional vertices
are added at the midpoints between existing vertices. In the seminal model
proposed by (Jiang et al. 2019), new vertices are assigned a value of zero,
essentially “zero-padding” the shared vertices of the existing, lower-level,
signal. This approach introduces artificial edges, since every vertex in the
mesh is now surrounded by a ring of black vertices (Figure 4a). The MeshConv
operator is particularly sensitive to these artifacts, since the constituent
differentials act as edge detectors. Accordingly, we bilinearly interpolate
the signal at parent vertices in order to compute the new midpoint vertex
values.

#### Order Of Operations

Finally, note that as part of the up/down-sampling block, (Jiang et al. 2019)
apply a final MeshConv after performing up/down-sampling. In our approach,
this order of operations is swapped such that convolution is performed prior
to transitioning between graph levels.

These design choices have a direct effect on the receptive field of the
MeshConv operation at each level, and therefore the receptive hierarchy our
FPN is designed to exploit. Our choice of ordering and average pooling
maintains the same receptive field as the seminal work, but without losing
information in down-sampling. Alternatively, by keeping the original ordering
with average pooling, the receptive field is larger by a one-ring neighborhood
of the previous level, see figure 5. This will increase the receptive field at
each level of the encoder, which may not be beneficial for finding smaller
scale features. For this reason, we perform an ablation study over these
design choices to empirically determine the optimal up/down-sampling procedure
for spherical FPNs.

(a) Padded upsampling.

(b) Bilinear upsampling.

(c) MeshConv applied to padded upsampling.

(d) MeshConv applied to bilinear upsampling.

Figure 4: Figure a) and b) show the results of upsampling a level-3 signal to
level-4 with zero-padded and bilinear upsampling respectively. Figures c) and
d) show the output of a single MeshConv operation, with all parameters set to
1. We can see unwanted “dotted” artifacts present in the zero-padded example.

(a)

(b)

Figure 5: The receptive field across the downsampling block depends on the
order of MeshConv and pooling operations. Swapping the order of operations
changes the receptive field from a 3-ring to a 2-ring neighbourhood.

## Experiments

### Stanford 2D-3D-S Experiments

We follow Jiang et al. (2019) and analyse our models on the Stanford 2D3DS
spherical image dataset, which contains 1413 equirectangular images of indoor
scenes, with RGBD channels and semantic labels corresponding to 13 classes. We
use the pre-processed data provided by Jiang et al. (2019) which samples the
original images at the latitude-longitudes of the spherical mesh vertex
positions spherical signal. The input RGB-D channels are interpolated using
bilinear interpolation, and semantic labels are acquired using nearest-
neighbor interpolation. Model performance is measured using two standard
metrics - pixel-wise accuracy, and mean intersection-over-union (mIoU),

We test a range of models, varying the maximum depth of the feature pyramid.
Besides simply providing models of varying complexity, we motivate this based
on the extreme spatial coarseness of the lowest level meshes. Planar FPNs (Lin
et al. 2017), which operate on established image recognition benchmark
datasets such as ImageNet (Deng et al. 2009) and MS COCO (Lin et al. 2014),
cap their lowest pyramid level to feature maps with a resolution of $20\times
15$, or 300 pixels. In contrast, the icosahedral mesh at the lowest level-0
consists of a mere 12 vertices, roughly corresponding to a $3\times 4$ 2D
image. At this extreme of spatial coarseness, the potential gains from
including increasingly lower pyramid levels may be non-trivial. Objects in the
scene may not have the excessively large features which are best represented
at this resolution. For this reason, we experiment with a range of pyramid
depths.

Experimental Setup: We test spherical FPNs with maximum level of 5, and
minimum level in {0,1,2,3}. All models use the bilinear upsampling and average
pooling blocks defined previously. For all experiments we use the Adam
optimizer to train our networks for 100 epochs, with an initial learning rate
of 0.01 and a step decay of 0.9 every 20 epochs. We use a batch size of 16 for
all models except the L0:5 FPN, for which we use a batch size of 8 due to
memory restrictions. All models were trained on Google Cloud using an NVIDIA
Tesla T4. The results are shown in Table 1

Figure 6: Mean IoU against the number of parameters for spherical FPNs (cyan
diamond) of varying minimum levels.

Model | Parameters | Accuracy | mIoU
---|---|---|---
(Jiang et al. 2019) | 5.18M | 54.65 | 0.3829
(Shen et al. 2022) | 0.86M | 60.4 | 0.4460
(Shakerinava and Ravanbakhsh 2021) | 2.31M | 62.5 | 0.45
L3:5 $S^{2}$FPN | 1.53M | 61.80 | 0.4512
L2:5 $S^{2}$FPN | 2.53M | 63.83 | 0.4814
L1:5 $S^{2}$FPN | 4.12M | 64.66 | 0.48.21
L0:5 $S^{2}$FPN | 7.01M | 64.73 | 0.4875

Table 1: Results for FPN experiments with varied pyramid depth. L$x$:5 refers
to a minimum level-$x$ mesh

Results and Discussion: All four FPN models achieve state-of-the-art
performance, with the smallest L3:5 FPN using $1.5\times$ fewer parameters.
The L2:5 model demonstrates substantial gains of 3.14 mIoU points over
(Shakerinava and Ravanbakhsh 2021), whilst using a comparable number of
parameters to. Our deepest FPN achieves 48.75 mIOU, setting a new state-of-
the-art for graph-based spherical CNNs on this dataset. The per-class mIoU
scores are shown in Table 2. Collectively, Our $S^{2}$FPNs achieve the best
performance across all the classes except door, where they fall short by 0.007
mIoU points.

| Model | Parameters | Mean | beam | board | bookcase | ceiling | chair | clutter | column | door | floor | sofa | table | wall | window
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Planar | UNet | - | 0.3587 | 0.0853 | 0.2721 | 0.3072 | 0.78757 | 0.3531 | 0.2883 | 0.0487 | 0.3377 | 0.8911 | 0.0817 | 0.3851 | 0.5878 | 0.2392
FCN8s | - | 0.3560 | 0.0572 | 0.3139 | 0.2894 | 0.7981 | 0.3623 | 0.2973 | 0.0353 | 0.4081 | 0.8884 | 0.0263 | 0.3809 | 0.5849 | 0.1859
Spherical | (Jiang et al. 2019) | 5.18M | 0.3829 | 0.0869 | 0.3268 | 0.3344 | 0.8216 | 0.4197 | 0.2562 | 0.1012 | 0.4159 | 0.8702 | 0.0763 | 0.4170 | 0.6167 | 0.2349
(Shen et al. 2022) | 0.86M | 0.446 | 0.114 | 0.433 | 0.382 | 0.839 | 0.503 | 0.313 | 0.124 | 0.484 | 0.90 | 0.181 | 0.495 | 0.659 | 0.371
(Shakerinava and Ravanbakhsh 2021) | 2.31M | 0.450 | - | - | - | - | - | - | - | - | - | - | - | - | -
L3:5 $S^{2}$FPN | 1.53M | 0.451 | 0.140 | 0.391 | 0.395 | 0.848 | 0.536 | 0.326 | 0.142 | 0.470 | 0.890 | 0.171 | 0.512 | 0.658 | 0.387
L2:5 $S^{2}$FPN | 2.53M | 0.481 | 0.126 | 0.509 | 0.423 | 0.850 | 0.563 | 0.350 | 0.149 | 0.500 | 0.892 | 0.240 | 0.543 | 0.686 | 0.426
L1:5 $S^{2}$FPN | 4.12M | 0.482 | 0.128 | 0.499 | 0.436 | 0.849 | 0.546 | 0.352 | 0.171 | 0.511 | 0.893 | 0.238 | 0.538 | 0.692 | 0.415
L0:5 $S^{2}$FPN | 7.01M | 0.487 | 0.09 | 0.505 | 0.443 | 0.842 | 0.581 | 0.352 | 0.158 | 0.535 | 0.893 | 0.237 | 0.558 | 0.691 | 0.451

Table 2: Per-class mIoU on Stanford 2D-3D-S dataset

### Ablation Study

We perform an ablation study over our choice of down-sampling and up-sampling
operations, as well as the order of MeshConv operation relative to down-
sampling. We consider two types of down-sampling to a lower-level mesh,
average pooling vertex values to compute new vertices (“average”), or simply
taking the values of vertices shared with the lower mesh (“drop”), as in Jiang
et al. (2019). For up-sampling, we test bilinear interpolation (“bilinear”)
and having new vertices take a value of zero (“zero-pad”). Finally, the
condition “swapped” refers to whether the order of MeshConv and downsampling
operation is swapped relative to Jiang et al. (2019). If the order is swapped,
MeshConv is applied before downsampling, as in 5a). All models used a L3:5
architecture, and were tested using an identical experimental set up to the
previous section.

Receptive Field | Swapped | Up-Sampling | Down-Sampling | Mean Acc (%) | mIoU
---|---|---|---|---|---
2-ring | ✗ | zero-pad | drop | 59.5 | 0.422
1-ring | ✓ | zero-pad | drop | 59.9 | 0.418
1-ring | ✓ | bilinear | drop | 60.3 | 0.426
2-ring | ✓ | zero-pad | average | 61.2 | 0.440
3-ring | ✗ | bilinear | average | 60.5 | 0.442
2-ring | ✓ | bilinear | average | 61.8 | 0.451

Table 3: Ablation study over up/down-sampling design choices for a L3:5 FPN.

Results and Discussion: The use of bilinear up-sampling and average pooling is
seen to improve the model both independently and in conjunction. Swapping the
order of MeshConv and down-sampling operation is seen to harm performance for
the model using ”drop” down-sampling, but improve the model when applied to a
model with average pooling. The reason for this requires more experimentation,
but could be due to the fact that both swapping the order and using “drop”
down-sampling reduce the receptive field on the previous level mesh. More
precisely, used together, they have a receptive field nine times as small as
that of average pooling and the original ordering. Combined with using the
shallowest L3:5 model, this significantly reduces receptive field on the input
signal and may be harming the network performance.

### ClimateNet

We also evaluate our method on the task proposed by (Mudigonda et al. 2017),
the segmentation of climate events from a 20-year run of the Community
Atmospheric Model v5 (CAM5) (Neale 2012). We use the data preprocessed by
(Jiang et al. 2019), which consists of spherical signals sampled onto a
level-5 icosahedral grid. Each map consists of 16 channels of measurements
such temperature, wind, humidity, and pressure. The training, validation, and
test set size is 43917, 6275, and 12549, respectively. The task is to use
these climate measurements to segment Atmospheric Rivers (AR) and Tropical
Cyclones (TC). The labels are heavily unbalanced with 0.1% TC, 2.2% AR, and
97.7% background (BG) pixels, and so we use a weighted cross-entropy loss.
Please see (Mudigonda et al. 2017) for information on ground truth label
production. Since the model proposed by (Jiang et al. 2019) is our most direct
comparison, we reduce the number of channels in our $S^{2}$FPN model (see 3)
by a factor of 4, in order to have a comparable number of parameters.

For all experiments we train for 50 epochs using the Adam optimizer, with an
initial learning rate of 0.001, and a step decay of 0.4 every 20 epochs. We
use a batch size of 128, training all models on a NVIDIA Tesla T4.

Model | Parameters | Mean Acc (%) | mAP
---|---|---|---
(Jiang et al. 2019) | 330K | 94.95 | 0.3841
(Defferrard et al. 2020) | 52M | 88.70 | 0.9138
(Shakerinava and Ravanbakhsh 2021) | 2.31M | 99.30 | 0.9520
L1:5 $S^{2}$FPN | 262K | 93.43 | 0.5448
L0:5 $S^{2}$FPN | 444K | 94.10 | 0.5132

Table 4: Results on ClimateNet

We omit comparisons to (Defferrard et al. 2020) and (Shakerinava and
Ravanbakhsh 2021) since their parameter scales are vastly different ($52$M,
and $2.3$M respectively). Compared to Spherical UNet, our FPN models show
substantial improvements in terms of mAP. The L1:5 model performs better than
the larger L0:5 model, perhaps indicating the effect of diminishing returns on
extremely coarse meshes.

## Conclusion

At present, there is not a universally accepted approach for performing
convolution on a sphere. Generalising convolutions to work on arbitrary shapes
and unstructured grids is an active research topic. However, the importance of
multi-scale features is generic. In theory, feature pyramids can be
constructed using any network with semantic and receptive field hierarchies.
In this work, we generalise the idea of feature pyramids to the spherical
domain and propose an alternative model to UNet for spherical image
segmentation. In addition, motivated by their influence on the receptive field
hierarchy of the network, we present improved pooling and up-sampling schemes
and measure their respective contributions through an ablation study. Our work
demonstrates that FPNs are a powerful domain-agnostic architecture, which
successfully generalize to a PDO-based spherical CNNs, attaining state-of-the-
art results.

## Acknowledgments

We would like to thank Chiyu ”Max” Jiang for his valuable comments on the
up/down-sampling methods, and for their work on pre-processing both ClimateNet
and Stanford 2D-3D-S datasets.

## References

  * Anderson et al. (1984) Anderson, C. H.; Bergen, J. R.; Burt, P. J.; and Ogden, J. M. 1984.  Pyramid Methods in Image Processing.
  * Armeni et al. (2017) Armeni, I.; Sax, A.; Zamir, A. R.; and Savarese, S. 2017.  Joint 2D-3D-Semantic Data for Indoor Scene Understanding.  _ArXiv e-prints_.
  * Baumgardner and Frederickson (1985) Baumgardner, J. R.; and Frederickson, P. O. 1985.  Icosahedral Discretization of the Two-Sphere.  _SIAM Journal on Numerical Analysis_ , 22(6): 1107–1115.
  * Bay, Tuytelaars, and Gool (2006) Bay, H.; Tuytelaars, T.; and Gool, L. V. 2006.  SURF: Speeded Up Robust Features.  _ECCV_.
  * Chang et al. (2017) Chang, A.; Dai, A.; Funkhouser, T.; Halber, M.; Nießner, M.; Savva, M.; Song, S.; Zeng, A.; and Zhang, Y. 2017.  Matterport3D: Learning from RGB-D Data in Indoor Environments.
  * Cobb et al. (2021) Cobb, O. J.; Wallis, C. G. R.; Mavor-Parker, A. N.; Marignier, A.; Price, M. A.; d’Avezac, M.; and McEwen, J. D. 2021.  Efficient Generalized Spherical CNNs.  arXiv:2010.11661.
  * Cohen et al. (2018) Cohen, T. S.; Geiger, M.; Koehler, J.; and Welling, M. 2018.  Spherical CNNs.  arXiv:1801.10130.
  * Cohen et al. (2019) Cohen, T. S.; Weiler, M.; Kicanaoglu, B.; and Welling, M. 2019.  Gauge Equivariant Convolutional Networks and the Icosahedral CNN.  arXiv:1902.04615.
  * Crane (2015) Crane, K. 2015.  _Discrete Differential Geometry: An applied introduction_.
  * Dalal and Triggs (2005) Dalal, N.; and Triggs, B. 2005.  Histograms of oriented gradients for human detection.  In _2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05)_ , volume 1, 886–893 vol. 1.
  * Defferrard et al. (2020) Defferrard, M.; Milani, M.; Gusset, F.; and Perraudin, N. 2020.  DeepSphere: a graph-based spherical CNN.  arXiv:2012.15000.
  * Deng et al. (2009) Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei-Fei, L. 2009.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , 248–255. Ieee.
  * Drake and Wright (2020) Drake, K. P.; and Wright, G. B. 2020.  A fast and accurate algorithm for spherical harmonic analysis on HEALPix grids with applications to the cosmic microwave background radiation.  _Journal of Computational Physics_ , 416: 109544.
  * Esteves et al. (2018) Esteves, C.; Allen-Blanchette, C.; Makadia, A.; and Daniilidis, K. 2018.  Learning SO(3) Equivariant Representations with Spherical CNNs.  arXiv:1711.06721.
  * Fan et al. (2017) Fan, C.-L.; Lee, J.; Lo, W.-C.; Huang, C.-Y.; Chen, K.-T.; and Hsu, C.-H. 2017.  Fixation prediction for 360 video streaming in head-mounted virtual reality.  In _Proceedings of the 27th Workshop on Network and Operating Systems Support for Digital Audio and Video_ , 67–72.
  * Feng et al. (2018) Feng, X.; Yang, J.; Laine, A. F.; and Angelini, E. D. 2018.  Discriminative analysis of the human cortex using spherical CNNs - a study on Alzheimer’s disease diagnosis.
  * Geiger et al. (2013) Geiger, A.; Lenz, P.; Stiller, C.; and Urtasun, R. 2013.  Vision meets robotics: The KITTI dataset.  _The International Journal of Robotics Research_ , 32(11): 1231–1237.
  * Gorski et al. (2005) Gorski, K. M.; Hivon, E.; Banday, A. J.; Wandelt, B. D.; Hansen, F. K.; Reinecke, M.; and Bartelmann, M. 2005.  HEALPix: A Framework for High‐Resolution Discretization and Fast Analysis of Data Distributed on the Sphere.  _The Astrophysical Journal_ , 622(2): 759–771.
  * Hoja et al. (2021) Hoja, J.; Sandonas, L. M.; Ernst, B. G.; Vazquez-Mayagoitia, A.; DiStasio, R. A.; and Tkatchenko, A. 2021.  QM7-X, a comprehensive dataset of quantum-mechanical properties spanning the chemical space of small organic molecules.  _Scientific Data_ , 8(1).
  * Jacobson, Panozzo et al. (2018) Jacobson, A.; Panozzo, D.; et al. 2018.  libigl: A simple C++ geometry processing library.  Https://libigl.github.io/.
  * Jiang et al. (2019) Jiang, C. M.; Huang, J.; Kashinath, K.; Prabhat; Marcus, P.; and Niessner, M. 2019\.  Spherical CNNs on Unstructured Grids.  arXiv:1901.02039.
  * Kim, De Campos, and Hilton (2016) Kim, H.; De Campos, T.; and Hilton, A. 2016.  Room layout estimation with object and material attributes information using a spherical camera.  In _2016 Fourth International Conference on 3D Vision (3DV)_ , 519–527. IEEE.
  * Kirillov et al. (2019) Kirillov, A.; Girshick, R.; He, K.; and Dollár, P. 2019.  Panoptic Feature Pyramid Networks.  arXiv:1901.02446.
  * Kondor, Lin, and Trivedi (2018) Kondor, R.; Lin, Z.; and Trivedi, S. 2018.  Clebsch-Gordan Nets: a Fully Fourier Space Spherical Convolutional Neural Network.  arXiv:1806.09231.
  * Lin et al. (2017) Lin, T.-Y.; Dollár, P.; Girshick, R.; He, K.; Hariharan, B.; and Belongie, S. 2017\.  Feature Pyramid Networks for Object Detection.  arXiv:1612.03144.
  * Lin et al. (2014) Lin, T.-Y.; Maire, M.; Belongie, S.; Bourdev, L.; Girshick, R.; Hays, J.; Perona, P.; Ramanan, D.; Zitnick, C. L.; and Dollár, P. 2014.  Microsoft COCO: Common Objects in Context.  Cite arxiv:1405.0312Comment: 1) updated annotation pipeline description and figures; 2) added new section describing datasets splits; 3) updated author list.
  * Liu et al. (2016) Liu, W.; Anguelov, D.; Erhan, D.; Szegedy, C.; Reed, S.; Fu, C.-Y.; and Berg, A. C. 2016.  SSD: Single Shot MultiBox Detector.  In _Computer Vision – ECCV 2016_ , 21–37. Springer International Publishing.
  * Lowe (2004) Lowe, D. G. 2004.  Distinctive Image Features from Scale-Invariant Keypoints.  _International Journal of Computer Vision_ , 60(2): 91–110.
  * McEwen, Wallis, and Mavor-Parker (2022) McEwen, J. D.; Wallis, C. G. R.; and Mavor-Parker, A. N. 2022.  Scattering Networks on the Sphere for Scalable and Rotationally Equivariant Spherical CNNs.  arXiv:2102.02828.
  * Mudigonda et al. (2017) Mudigonda, M.; Kim, S.; Mahesh, A.; Kahou, S. E.; Kashinath, K.; Williams, D. N.; Michalski, V.; O’brien, T.; and Prabhat, M. 2017.  Segmenting and Tracking Extreme Climate Events using Neural Networks.
  * Neale (2012) Neale, e. a., R. B. 2012.  Description of the NCAR Community Atmosphere Model (CAM 5.0).
  * Perraudin et al. (2019) Perraudin, N.; Defferrard, M.; Kacprzak, T.; and Sgier, R. 2019.  DeepSphere: Efficient spherical convolutional neural network with HEALPix sampling for cosmological applications.  _Astronomy and Computing_ , 27: 130–146.
  * Sermanet et al. (2013) Sermanet, P.; Eigen, D.; Zhang, X.; Mathieu, M.; Fergus, R.; and LeCun, Y. 2013\.  OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks.
  * Shakerinava and Ravanbakhsh (2021) Shakerinava, M.; and Ravanbakhsh, S. 2021.  Equivariant Networks for Pixelized Spheres.
  * Shen et al. (2022) Shen, Z.; Shen, T.; Lin, Z.; and Ma, J. 2022.  PDO-eS2CNNs: Partial Differential Operator Based Equivariant Spherical CNNs.  arXiv:2104.03584.
  * Su and Grauman (2017) Su, Y.-C.; and Grauman, K. 2017.  Learning spherical convolution for fast features from 360 imagery.  _Advances in Neural Information Processing Systems_ , 30.
  * Tan, Pang, and Le (2019) Tan, M.; Pang, R.; and Le, Q. V. 2019.  EfficientDet: Scalable and Efficient Object Detection.
  * Yu et al. (2016) Yu, Q.; Szegedy, C.; Stumpe, M. C.; Yatziv, L.; Shet, V.; Ibarz, J.; and Arnoud, S. 2016.  Large Scale Business Discovery from Street Level Imagery.  arXiv:1512.05430.
  * Yuan et al. (2020) Yuan, J.; Deng, Z.; Wang, S.; and Lui, Z. 2020.  Multi Receptive Field Network for Semantic Segmentation.  _CoRR_ , abs/2011.08577.
  * Zhu et al. (2021) Zhu, X.; Lyu, S.; Wang, X.; and Zhao, Q. 2021.  TPH-YOLOv5: Improved YOLOv5 Based on Transformer Prediction Head for Object Detection on Drone-captured Scenarios.