# Predicting vehicles parking behaviour in shared premises for aggregated EV
electricity demand response programs

Vinicius Monteiro de Lira Institute of Information Science and Technologies,
Italian National Research Council, Pisa, Italy,
<EMAIL_ADDRESS>Fabiano Pallonetto School of Business,
Maynooth University, Kildare, Ireland<EMAIL_ADDRESS>Lorenzo
Gabrielli Institute of Information Science and Technologies, Italian National
Research Council, Pisa, Italy<EMAIL_ADDRESS>Chiara Renso
Institute of Information Science and Technologies, Italian National Research
Council, Pisa, Italy<EMAIL_ADDRESS>
###### Abstract

The global electric car sales in 2020 continued to exceed the expectations
climbing to over 3 millions and reaching a market share of over 4%. However,
uncertainty of generation caused by higher penetration of renewable energies
and the advent of Electrical Vehicles (EV) with their additional electricity
demand could cause strains to the power system, both at distribution and
transmission levels. Demand response aggregation and load control will enable
greater grid stability and greater penetration of renewable energies into the
grid. The present work fits this context in supporting charging optimization
for EV in parking premises assuming a incumbent high penetration of EVs in the
system. We propose a methodology to predict an estimation of the parking
duration in shared parking premises with the objective of estimating the
energy requirement of a specific parking lot, evaluate optimal EVs charging
schedule and integrate the scheduling into a smart controller. We formalize
the prediction problem as a supervised machine learning task to predict the
duration of the parking event before the car leaves the slot. This predicted
duration feeds the energy management system that will allocate the power over
the duration reducing the overall peak electricity demand. We structure our
experiments inspired by two research questions aiming to discover the accuracy
of the proposed machine learning approach and the most relevant features for
the prediction models. We experiment different algorithms and features
combination for 4 datasets from 2 different campus facilities in Italy and
Brazil. Using both contextual and time of the day features, the overall
results of the models shows an higher accuracy compared to a statistical
analysis based on frequency, indicating a viable route for the development of
accurate predictors for sharing parking premises energy management systems.

##  1 Introduction

The advent of Electrical Vehicles (EV) are in increasing spreading in our
society. According to MCkinsley
report111https://www.mckinsey.com/industries/automotive-and-assembly/our-
insights/mckinsey-electric-vehicle-index-europe-cushions-a-global-plunge-in-
ev-sales# in our society EV sales rose 65 percent from 2017 to 2018 and Europe
has seen the strongest growth in EVs. According to the report ”The European
Union’s new emissions standard—95 grams of carbon dioxide per kilometer for
passenger cars—could also boost EV sales because it stipulates that 95 percent
of the fleet must meet this standard in 2020 and 100 percent in 2021”. A race
for larger batteries among manufacturer is leading the current EV technology,
and going forward it appears that as batteries technology improve, they are
going to replace motor fuel vehicles. The concerns as we move to EVs is that,
firstly, there will not be enough charge points to meet consumer demand and,
secondly, this additional load on the electricity grid will cause partial and
total failure of specific electrical plant due to overloading. One of the
foreseen solutions is the use of demand response aggregation and load control
[18, 13]. Demand response aggregators work with businesses to build a
portfolio of load (or micro-generation) of which they control as per
requirements of the electricity balancing market. Aggregators take payments
from the the electricity market and pass these payments to their clients
taking a service charge [11].

Although system services are provided to the power system via conventional
power stations, this same service is not available at distribution level. The
growth in demand aggregators will enable greater grid stability and greater
penetration of renewable energies into the grid [21].

EV energy storage capacity provides an excellent non-critical portfolio of
loads which could be dynamically controlled to offer system services as well
as organised in a fashion to minimise the cost of charging and reducing
electricity demand peaks. Given the well documented concerns of EV penetration
in the distribution system, demand response aggregators could provide an
excellent control topology to employ an algorithm to safeguard and optimise
the distribution network [1]. The present work fits this context supporting
optimization for EV charging and assuming a incumbent high penetration of EVs
in the system. We propose a methodology to predict an estimation of the
parking duration in shared parking premises. This is essential for estimating
the energy requirement of a specific parking lot, evaluate optimal EVs
charging schedule and integrate the scheduling into a smart controller.

The specific behaviour of parking lots of campuses refereed to EV charge is
peculiar since it substantially differs from the general parking lots
available in the streets. In campus-like facilities (Universities, large
industries, etc) we can observe regular patterns of parking behaviour that
mainly include staff working hours besides a part of other visitor [9]. This
can be an advantage when trying to predict general behavioral patterns of
parking habits and thus reach an optimal recharge plan for EVs.

The current work aims to provide an overall design of a smart charging energy
management system to optimally integrate the distributed energy systems and
EVs into the power grid by developing a parking prediction module to estimate
the vehicles’ parking time using machine learning algorithms.

Given this context, the specific objective of this work is to predict the
duration of each parking event in a campus-like parking lot, where the parking
event is the actual parking action of a car in a slot. The parking duration
prediction is needed to combine this information with an energy optimization
algorithm that balances the energy allocation to the recharge stations with
the objective for the vehicles of leaving the parking slot with a minimum
charge level. While the charge optimization problem is out of the scope of
this work, we now focus on the parking duration prediction. However, we
believe that the recharge optimization context is necessary to fully
understand the motivation of tackling this specific prediction problem.

We formalize the prediction problem as a supervised machine learning task
that, given a parking event at a given time, tries to predict the duration of
the parking event. The reason for this event-based formulation is to be able
to feed the energy management system with the duration prediction each time a
car is parked. This will allow the energy management system to decide when to
start the actual charge based on the prediction. For example, assuming several
car arrive at 9 AM, if we start charging without a control system, the cars
start charging causing a peak of electricity demand. On the contrary, if we
predict that a given car stay parked e.g. 8 hours, then we can delay the
charge of this car to a later moment trying to flatten the peaks. We structure
our experiments inspired by two research questions: RQ 1: How accurately a
supervised machine learning approach can predict the duration of a parking
event in a campus-like parking lot? and RQ 2. What are the most relevant
features for the prediction models? We experiment different algorithms and
features combination into 5 datasets from 2 different campus facilities in
Italy and Brazil. We show that using both contextual and time of the day
features, the overall results of the models shows an higher accuracy compared
to a statistical analysis based on frequency, indicating a viable route for
the development of accurate predictors for sharing parking premises energy
management systems.

Structure of the paper follows. Section 2 discusses some related works an how
this approach differentiates from them while Section 3 illustrates the design
of the proposed Energy Management System for optimal charging prediction in
parking premises. Section 4 introduces the problem definition and clarifies
the prediction problem formulation. Section 5 reports the details of the
experimental evaluation and Section 6. Finally, Section 7 draws the
conclusions and envisage some future works.

##  2 Related Works

Solutions for EV optimal routes have been drafted in the logistic sectors,
[27, 17] accounting for charging time [15]. However, the current work
addresses a different problem that is at the crossroad of two different
topics: predicting parking occupancy and predicting EV battery charging. We
report below the most relevant related works concerned these two aspects.

###  2.1 Parking availability prediction

Most of the recent works in the literature focuses on predicting which parking
lot will be free at the arrival of the car. This is motivated by the challenge
of finding a parking space in urban areas. It has been
reported222https://www.accessmagazine.org/spring-2011/free-parking-free-
markets/ that 30% of traffic congestion is caused by travelling for finding
parking spaces, bringing unnecessary energy consumption and environmental
pollution. Works in this area include the off-street (parking slots in private
areas) and in-street variants (slots in the streets). A pioneering paper by
[5] proposes the real-time availability forecast algorithm to predict parking
facility availability in real time using combined current (on-line) and
historical information. This work uses an algorithm operating with mixed real
and simulated information.

In the recent paper [16] authors develop a prediction model based on Naive
Bayes and machine learning methods like decision tree, random forest, and
regression analysis for building the prediction model of parking occupancy and
therefore predict the subsequent parking availability combining a matching-
based allocation strategy to assign users to selected parking spaces.

Deep learning to predict parking occupancy is proposed by many papers in the
literature. Paper [22] adopts a deep learning model for predicting block-level
parking occupancy 30 min in advance in paper. The model takes multi-source
data as input, e.g., parking, traffic and weather.

[12] proposes a Convolutional Neural Network (CNN) model for block-level
parking occupancy prediction extracting spatial relations of traffic flow
combined with a LSTM autoencoder to capture temporal correlations. Clustering
is also considered in the Clustering Augmented Learning Method (CALM) to learn
deep feature representations of spatio-temporal data obtained using the
proposed embedding.

Paper [25] also proposes a deep-learning approach called Du-Parking. This
approach models temporal closeness, period and current general influence
employing long short-term memory (LSTM) to model the temporal closeness and
period. This approach learns to dynamically aggregate the output to estimate
the final parking availability of given parking lot.

Compared to the above approaches the novelty of our method lies in the fact
that we do not aim at predicting the next free slot neither to suggest the
driver where to park. On the contrary, we predict how long the car will stay
parked computed at the specific time a car start the parking before leaving
the slot, which is a different problem.

###  2.2 EV charging profile predictions

In [6] the authors explore the best machine learning techniques for the
prediction of EV charging behaviour using ensemble approach based on the data
sparsity/entropy ratio. The work illustrates how such a methodology can
increase the accuracy of the prediction. However, the algorithm could be
difficult to scale to shared premises because it utilises only the data from
EV charging stations which represents only a small percentage of the premise.

Related works such as [26] are focusing on determining the optimal charging
start time assuming fixed parking time for two different group categories, day
parking and overnight parking. This approach on user behaviour has been used
to simplify the optimisation, but the data of a shared premise are more
dynamic and so the established categories do not reflect a real scenario.

In [14], researchers developed an optmisation algorithm for optimal EV battery
charging to reduce peak load at feeder level. Such an approach could
facilitate the penetration of EV charging points in shared premises. However
it does not take into account parking time prediction while it could be useful
for the scheduling of energy consumption.

Compared to the above approaches our proposal is novel since we plan to embed
this prediction task into an optimization algorithm of optimal charging
prediction and this combination, at the best of our knowledge, this is not yet
present the literature.

##  3 Energy management system with prediction of the electricity demand in
parking area

High penetration of EVs in shared parking lots is a likely scenario on common
premises. While parking is a necessary service for offices, commercial, and
residential buildings, EVs on parking site could become an asset for balancing
the local power distribution network, providing ancillary services and
increasing the grid’s energy flexibility [24]. An Energy Management System for
managing shared parking premises energy consumption will be effective only if
the aggregated parking occupancy behaviour is predicted with sufficient
accuracy. Specifically, the depicted energy management system controls the
electricity demand evaluating local and power system variables, such as
renewable energy production, demand response price signals and building
consumption. Furthermore, it should assess the potential flexibility of the
aggregated parked EVs to estimate and optimise the area’s energy demand
accurately.

Figure 1: The parking occupancy prediction module is embedded in the energy
management system controller for the prediction of the electricity demand used
by the parking area. In yellow the parking prediction modules proposed in the
present paper

Figure 3 illustrates an overall design of a distributed energy management
system. The inputs data points of the system are demand response price
signals, sensors data from buildings, weather forecast and images from the
parking camera. The systems produces an optimal predictive schedule that
balances the energy consumption while optimizing vehicle charging. The system
objective function is to reduce the overall electricity cost and to minimise
the Peak Average Ratio (PAR), defined as:

$PAR={{\max_{t\in{\cal
T}}\sum_{i=1}^{N}l_{i}^{t}}\over{{{1}\over{T}}\sum_{i=1}^{N}E_{i}}}$ (1)

where $l_{1},…,l_{N}$ are the load demand vectors for vehicle and $E_{i}$ is
the predetermined electricity demand to charge the battery to the full
capacity. The classical problem assumes that the vehicles will be parked on
the premises long enough to be fully charged. In reality, each vehicle’s
parking time is variable, and it depends on several factors such as peak
traffic, user habits, and weather conditions. Therefore, the aggregated
prediction of the parking time of vehicles is necessary to properly optimize
the energy demand of the site.

In this paper we focus on a part of the energy management system dedicated to
the parking occupancy prediction based on historical data, indicated in light
orange in Figure 1. We present a machine learning approach for forecasting the
duration of the stay of cars at the parking lot. The energy management system
uses this information for computing the optimal changing scheduling,
minimising electricity cost and PAR. The current work focuses on predicting
the parking time without making any assumption on the state of charge of the
battery. In a distributed energy management system, the state of charge of
each vehicle is acquired as soon as the car is plugged to the charging
station. Therefore, in the current work assumption on the state of charge was
not critical. The objective function of the system is to maximise the energy
supply to the car within the parking time reducing the overall peak demand.

##  4 The parking duration prediction problem

The objective of our approach is to exploit historical data on parking usage
and additional contextual data like weather conditions and parking lot
occupancy levels, to predict the duration of a parking slot occupancy.
Differently from many state of the art approaches that want to predict if a
giving parking lot will be free in a next period of time [5, 16], here we
focus on the prediction of the temporal duration of the occupancy of a car in
a slot. We recall that our approach, to be suitably integrated with an Energy
Management System, focuses on specific parking context that we call of shared
premises (e.g. parking lots of universities, workplaces, supermarkets, etc),
not focusing on fee-based street parking.

Overall, our approach can be applied to any parking environment where there is
a tendency for the car to stay parked a minimum amount of time and where the
electrical charge system of the parking lots can be integrated into the
controller of the Energy Management System. we also recall that our approach
is driver-profile agnostic since we do not rely on driver-profile data to
perform the predictions. This is because, for privacy reasons, the driver
profile is generally not available. We expect that having such profile the
predictions will become more accurate.

It is worth noticing that the parking behaviour in a campus-like facility
reflects a different parking behaviour compared to fare-based streets parking
lots. In campus-like parking, the parking duration is expected to be longer
than on street parking, since these premises are used by people parking to go
to work, or study or perform an activity with a minimal temporal duration,
while a fare-based parking in a street is generally affected by the parking
fees that tend to encourage the reduction of the parking duration.
Furthermore, the premises have usually a controlled access and the energy
management system can therefore optimise the electricity supply based on the
parking occupancy, while is not always true in fare-based street parking.

###  4.1 Problem formulation

Given a parking area managed by an Energy Management System, a car parking
event represents an event where a driver parks at a given timestamp in one of
the available slots. The vehicle stays parked for a certain temporal duration
until it leaves the slot. It is assumed that the vehicle can be charged while
parked. The charging time can start as soon as the car arrives, or can start
later on, or again, can start, interrupt and start again.

Having the prediction of the parking duration when a vehicle arrives at the
parking is essential to properly schedule the starting of the charge avoiding
energy usage peaks.

######  Definition 1 (Parking Slot).

Given a parking area in shared premises, we define a parking slot $s$ as a
tuple $s=<id,x,y>$, where $id$ represents the parking slot identifier, and $x$
and $y$ represent its spatial coordinates in the parking area. A parking slot
is the actual place where drivers park their cars. In our application
scenario, each slot can be equipped with a charge station where the car can be
recharged. The set of all the parking slots form the parking area.

######  Definition 2 (Car Parking Event).

We define a car parking event $e$ as a tuple $e=<s_{id},t_{start},d_{e}>$,
where $s_{id}$ represents the parking slot identifier where the car is parked,
$t_{start}$ represents the timestamp indicating when a car has started the
parking and $d_{e}$ is the temporal duration of the car park until it leaves
the slot.

We want to predict the parking duration $d_{e}$ of a car parking event $e$,
given a $slot_{id}$ and the parking event starting time $t_{start}$. This
prediction is modelled as a classification problem where the objective is to
assign, for each car parking event $e$, a class representing the predicted
duration interval. More formally, we have the following definition of the
problem.

######  Definition 3 (Car Parking Duration Prediction Problem).

Given a parking event $e$ where it is known the slot identifier $s\\_{id}$ and
the start time $t_{start}$ but not duration $d_{e}$, we want to define a
function $f(s\\_{id},t_{start})=c$ where the class $c$ represents a temporal
interval such that $d_{e}\in c$.

We can observe that our target variable $c$ represents ordinal categories. An
ordinal variable is a categorical variable, where there is a clear ordering of
the categories. For example, our variable could assume ordinal categories
like: short, medium or long duration. In the next section, we introduce the
details of the Machine Learning (ML) approach to solve the Car Parking
Duration Prediction Problem.

###  4.2 Predicting parking duration with Machine Learning

We propose to use supervised machine learning approaches to predict the
parking duration based on an historical dataset of car parking events and
contextual features.

The learning task is based on a three types of features: _single event-
related_ , _spatial_ and _contextual_ features. The event-related features are
detailed in Section 4.2.1 and represent the features that we can extract
directly from the sets of parking events like the time of the parking event or
the weather conditions. The spatial features are based on the location of the
parking slots inside the car parking area, while the contextual features
representing the occupancy of the different zones of the parking area. In
Sections 4.2.2 and 4.3 we explain how we have extracted some spatial and
contextual features that are used in our predictive models. All these features
are combined to feed the proposed supervised machine learning algorithms
(Section 5).

####  4.2.1 Extracting event-related features

Recalling Definition 2, a car parking event is defined by the tuple
$e=<s_{id},t_{start},d_{e}>$ and given $s_{id}$ and $t_{start}$ we want to
predict the temporal duration $d_{e}$. From the timestamp $t_{start}$, we
derive three features: the day of week $dw$, hour of the day $h$, and the
minutes $m$ rounded to 5 minutes. The motivation of these temporal features is
to enable the predictive model to learn the correlation between the time when
the car parks and the relative parking temporal duration. We also include in
this category of features the weather condition $wr$ at the moment of the car
parking event starts, $t_{start}$, using this as extra information to feed the
predictive models.

####  4.2.2 Extracting spatial features from the parking slots locations

Many studies have been made toward the understating of parking behavior and
the mechanism of people’s parking decisions [28, 3]. It has been observed that
some spatial aspects can bias the occupancy of a parking lot. For example,
usually drivers prefer to choose the parking areas close to the destination
[19], because they prefer to spend less time walking. However, this is not the
unique criteria. Some other branch of research with focus on sustainability
have investigated the impact of the trees in parking lots [23]. Trees provide
welcoming shade in the parking area, especially on hot days. Indeed, in the
real world, there are drivers that may prefer to park the car close to the
main entrance of the supermarket, while others may prefer to park in the
shading areas avoiding the light of the sun during the summer.

Motivated by these aspects, our approach focus on the spatial distribution of
the parking slots. For this reason, we split the whole parking lot into
smaller areas using different clustering approaches. Then, we include these
spatial features in our predictive models to learn if a parking area can
correlate with the slot occupancy duration.

More formally, given a set of parking slots $(s_{0},s_{1},...,s_{n})\in S$, we
use the spatial coordinates of each $s_{i}$ to create the spatial clusters
$(a_{1},a_{2},..,a_{k})$ of parking slots, where $k\leq n$. We will see in
section 5 that we have used two clustering algorithms for this task: DBScan
([8]) and K-Means ([20]). Thus, when training our predictive models over the
the dataset of historical car parking events we add as input feature a
representation of the cluster where the parking slot $s_{i}$ belongs to.

In Figure 2, we illustrate the spatial features. At the top, we can see images
from three different car parking lots. At the bottom, the dots represent the
pixel coordinate of the parking slots. The colors of each dots is a
representation of the spatial features which indicates the cluster (sub-area)
of each parking slot.

Figure 2: Illustration of the spatial features using the k-means algorithm to
split the parking lot into four spatial clusters.

###  4.3 Extracting contextual features

Another aspect that we investigate for the parking duration prediction is the
context. In our case the context is represented by the status of occupancy of
the slots in the spatial clusters and relationship of this occupancy with the
duration of a given parking event.

Specifically we want to discover if the occupancy status of an area (e.g 100%,
means totally full, while 0% totally empty) where a driver parks has
relationship with the parking duration.

The contextual features therefore represent the status of occupancy of the
different areas (i.e. the spatial clusters) of the parking. In other words, to
predict the duration of a given car parking event $e$, we also consider as
input feature the level of occupancy of the spatial clusters
$(a_{1},a_{2},..,a_{k})$ at the time of the event.

More formally, we define a function called $occupancy$ that given a
$slot_{id}$ and the timestamp $t_{start}$ of a parking event $e$, creates a
vector $(o_{a_{1}},o_{a_{2}},..,o_{a_{k}})$ by computing the occupancy level
of all parking spatial clusters $(a_{1},a_{2},..,a_{k})$ at time $t_{start}$.
The occupancy status ${o_{a_{j}}}$ of each spatial cluster $a_{j}$ is
basically the ratio between the number of occupied slots at the timestamp
$t_{start}$ and the number of slots in that cluster.

In Figure 3, we illustrate the contextual occupancy features. At the top, we
can see the spatial distribution of the slots from three different parking
lots. The color represents the spatial cluster of each slot. At the bottom, we
have the occupancy status in percent of the spatial clusters
$a_{1},a_{2},a_{3},$ and $a_{4}$ at a given time $t$.

Figure 3: Illustration of the contextual occupancy features representing the
occupancy status of the spatial clusters $a_{1},a_{2},a_{3},$ and $a_{4}$ at
time $e_{i}.t_{start}$.

To summarize, in this section we introduced three new categories of features:
event-related, spatial and contextual. The event-related features are
extracted from the parking event, the spatial features are computed by using
cluster techniques over the spatial distribution of the parking slots while
the contextual features are obtained by computing the occupancy status of the
spatial clusters.

The overall idea is to investigate how to train the predictive models using
different information that might have a predictive power on the parking
duration. In the next section we detail the experimental setting and results
on exploiting these features in a machine learning task for predicting the
parking duration of a given event.

##  5 Experimental evaluation

In this section we evaluated the proposed approach for predicting the parking
duration by exploiting historical parking data and specifically three kinds of
features (see Definition 4.1). For a proper evaluation of the method we
formulate different research questions (RQs) that we test in various scenarios
by exploiting four real world and publicly available parking datasets.

###  5.1 Research Questions

The research questions driving our experiments are the following:

RQ 1: How accurately a supervised machine learning approach can predict the
duration of a parking event in a campus-like parking lot? This research
question guides our first experiments. Here, we compare the performance
results of our machine learning based approach against several baselines. We
also investigate different machine learning approaches to tackle this problem
as a supervised task: Classification, Ordinal Regression, and Regression. We
use different features including the description of the occupancy of the
parking lot at the time the parking event starts and the spatial distribution
of the parking areas.

RQ 2. What are the most relevant features for the prediction models? We
consolidate our studies investigating which are the most promising features to
estimate car parking duration. We have performed a deep investigation on how
the machine learning algorithms use the provided features to train the models.

###  5.2 Experimental Setup

Datasets. We selected two public datasets of parking occupancy in campus-like
parking lots: PKlot [7] and CNRPark [2]. Both datasets contain the occupancy
information detected by video cameras for each slots of parking areas of two
academic institutions: the research area of the National Research Council of
Pisa333http://www.area.pi.cnr.it, in Italy and the parking area of the two
Brazilian universities. In both cases the whole parking lot is split in
different parking areas with a variable number of parking slots. In both
datasets, a car parking event occurs when a car parks in a parking slot of the
area. In this case, the event starts at the timestamp of the frame that
detects a car in the slot. The car parking event ends at the timestamp of the
frame showing: (1) an empty parking slot, or (2) a different car parked in the
same slot. The duration of the parking event is then computed as the
difference of the timestamps of the two image frames, the start and the end.
The CNRPark dataset contains images collected from November 2015 to February
2016 for a total of 23 monitored days. The parking lot has been monitored by 9
cameras covering the parking area of the Pisa National Research Council south
parking area. The meteorological conditions at the moment of the frame capture
has also been collected. This dataset contains a total of 4081 frames and
144,965 photos. In Table 1 we depict an excerpt of the CNRPark raw data.

Date | Time | Slot | Occupancy Bit | Occupancy String | Weather
---|---|---|---|---|---
12/11/2015 | 07:15 | 275 | 0 | free | SUNNY
12/11/2015 | 07:45 | 275 | 0 | free | SUNNY
12/11/2015 | 08:15 | 275 | 0 | free | SUNNY
12/11/2015 | 08:45 | 275 | 0 | free | SUNNY
12/11/2015 | 09:15 | 275 | 1 | busy | SUNNY
12/11/2015 | 09:45 | 275 | 1 | busy | SUNNY
12/11/2015 | 10:15 | 275 | 1 | busy | SUNNY
12/11/2015 | 10:45 | 275 | 1 | busy | SUNNY
12/11/2015 | 11:15 | 275 | 1 | busy | SUNNY
12/11/2015 | 11:45 | 275 | 1 | busy | SUNNY
Table 1: An extract of the CNR parking dataset

The PKlot dataset contains the occupancy information for each slot of the
parking areas of two academic institutions: (1) the Federal University of
Parana (UFPR) and (2) the Pontifical Catholic University of Parana (PUCPR),
both located in Curitiba, Brazil. The dataset includes a total of three
different parking lots represented by PUCPR, UFPR04, and UFPR05. The occupancy
information is detected by a number of cameras taking images of the parking
slots and detecting the change of the car or the slot becoming empty. This
dataset contains 12.417 images captured in three different parking areas with
different weather conditions for a total of 168 slots in the period between 11
September 2012 and 16 April 2013. Specifically, dataset PUCPR has 100 parking
slots, UFPR04 has 28 and UFPR05 has 45 slots. PKLot is larger than CNRPark and
contains images spanning across months.

Weekday | CNRPark | PUCPR | UFPR05 | UFPR04
---|---|---|---|---
Monday | 2 | 3 | 4 | 2
Tuesday | 2 | 8 | 6 | 5
Wednesday | 2 | 5 | 4 | 4
Thursday | 5 | 5 | 4 | 4
Friday | 6 | 7 | 5 | 5
Saturday | 4 | 5 | 5 | 4
Sunday | 2 | 4 | 2 | 1
Table 2: Number of monitored days in the parking lots datasets

Figure 4: Distribution of occupancy of the parking lots of CNR (left) and
PKLot (right)

Figure 5: Arrivals distribution at the parking lots of CNR (left) and PKLot
(right)

Data Cleaning. A detailed analysis on the image frames reveals the presence of
missing data for some hour of the day (e.g. due to a broken device or during
night hours due to the lack of infrared vision). Table 2 shows the total
number of days of the weeks monitored in each dataset. We notice that the data
collection is not uniform for each day of the week and the there is a tendency
to have lesser data in the week ends. When a parking event is starting or
ending during the missing temporal interval, it has been flagged as partial
and filtered out to avoid training inconsistencies. Additionally, we filter
out the days when the number of detected slots is lower than 50% of the total
slots. In total, after the data cleaning CNRPark counted 3552 parking events,
PUCPR 4291 parking events, UFPR04 1204 parking events and UFPR04 2148 parking
events.

Target Classes. We have considered two different scenarios of classes for the
predictive variable (i.e. the car parking event duration): (a) Lower
sensitivity, with longer time intervals having a total of 3 classes with
discrete values in minutes: $Short\leq 60,60<Mid\leq 240,Long>240$; and (b)
Higher sensitivity having shorter time intervals with a total of 6 classes:
$Short1,\leq 30,30>Short2\leq 60,60<Mid1\leq 120,120<Mid2\leq
240,240<Long1\leq 480,$ and $Long2>480$. With these two scenarios, we want to
illustrate applications requirements with different sensitivity for the
predictive variable. Tables 3 and 4 show the normalized distribution of the
car parking events for the low and high sensitivity scenarios respectively.
Both tables also report, for each dataset, the entropy calculated according to
the frequency values of the classes. We use the entropy as a measure of
disorder or randomness (uncertainty) of our target variable.

Training Approaches. Given the ordinal characteristic of our target variable,
we have explored three supervised approaches to train the predictive models
over the training set. For each approach, the best model is selected taking
into account the average results over the 5 folds of validation. These
approaches are: (a) Classification: the training is performed without taking
into account the order of the classes and the selected model is the one with
highest micro-fscore; (b) Regression: the training is executed to reduce the
mean square error (MAE) of the predicted values, therefore the model with
lowest error is selected; (c) Ordinal Regression (Ord): the approach proposed
in [10] takes into account the order of the classes. The method converts the
original ordinal class problem into a series of binary classification problems
that encode the ordering of the original classes. This approach also aims at
reducing the error and therefore selects the model with lowest mean square
error (MAE).

| Train |  | Test
---|---|---|---
| Short | Mid | Long | Entropy |  | Short | Mid | Long | Entropy
CNRPark | 0.09 | 0.31 | 0.60 | 0.81 |  | 0.13 | 0.36 | 0.51 | 0.89
PUCPR | 0.41 | 0.28 | 0.31 | 0.99 |  | 0.35 | 0.32 | 0.33 | 1.00
UFPR04 | 0.52 | 0.28 | 0.20 | 0.92 |  | 0.44 | 0.28 | 0.28 | 0.98
UFPR05 | 0.44 | 0.26 | 0.30 | 0.98 |  | 0.54 | 0.32 | 0.14 | 0.89

Table 3: Normalized distribution of the car parking events for the low
sensitivity scenario in the train and the test datasets. For each dataset we
also report the entropy calculated according to the frequency values of the
classes

| Short1 | Short2 | Mid1 | Mid2 | Long1 | Long2 | Entropy
---|---|---|---|---|---|---|---
| Train/Test | Train/Test | Train/Test | Train/Test | Train/Test | Train/Test | Train/Test
CNRPark | 0.06/0.10 | 0.04/0.03 | 0.09/0.10 | 0.22/0.26 | 0.53/0.57 | 0.07/0.04 | 0.76/0.78
PUCPR | 0.38/0.32 | 0.03/0.03 | 0.08/0.10 | 0.21/0.27 | 0.20/0.16 | 0.11/0.18 | 0.87/0.78
UFPR04 | 0.46/0.38 | 0.07/0.05 | 0.10/0.12 | 0.17/0.17 | 0.15/0.17 | 0.05/0.11 | 0.84/0.90
UFPR05 | 0.38/0.44 | 0.06/0.10 | 0.08/0.09 | 0.18/0.22 | 0.19/0.11 | 0.11/0.03 | 0.90/0.83

Table 4: Normalized distribution of the car parking events for the high
sensitivity scenario for the train and the test datasets respectively. For
each dataset we also report the entropy calculated according to the frequency
values of the classes

Algorithms. For the Classification and Regression tasks we used the following
algorithms: Random Forest (RF), XGBoosting (XGB), AdaBoosting (AB), Logistic
Regression (LR) and Support Vector Machine (SVM). For the Ordinal Regression
task we selected: Random Forest (RF), XGBoosting (XGB), AdaBoosting (AB) and
Logistic Regression (LR) To compute the spatial features, we have used the
K-means and the DBScan clustering algorithms. For all algorithms, we used the
implementation available in the scikit-learn library444https://scikit-
learn.org/.

Features. The following features are extracted and used to feed the ML
algorithms. The event-related features, as defined in Section 4.2.1, include
hour of the day $h$, time stamp minutes $m$, day of week $dw$, slot id $s$,
and weather condition $wr$; the spatial features, as defined in Section 4.2.2,
include the spatial cluster id $spt$; the occupancy features, as defined in
Section 4.3, include the spatial cluster occupancy $ocy$. We use different
feature combinations to train the models, specifically:

  1. 1.

Single event-related feature. We train the model using only one event-related
feature.

  2. 2.

All event-related features together. We train the model using all single
event-related features at once. We refer to $all$ when we use all the event-
related features to train the ML model.

For both cases, we perform two further combinations: using and not using the
spatial and occupancy features to feed the models.

Hyper parameters. We use a grid search to tune the hyperparameters of the
algorithms [4]. Specifically: For XGB, AB and RF, we vary the number of trees
in the range of {50, 100, 150}, while maximum tree depth vary in the ranges of
{2,3, or until all leaves are pure}, respectively; For SVM, we use the RBF
kernel with $\gamma$ varying in {0.0001, 0.001, 0.01}. For the LR, we have
used two different class weight parameter {balanced and uniform}, while the
multi class parameter changing between {auto, ovr (for binary
classification)}. For the K-means, the $k$ varies in the ranges of {2,
3,4,5,6}. While for the DBScan ranges are {50, 75, 100, 125, 150} and {2,3,4}
for the $\epsilon$ and minimum sample, respectively.

Baselines. To be able to evaluate the performance of our approach we have used
the following baselines: (a) Random: randomly choose a class; (b) Longest
Class: always select the longest interval; (c) Shortest Interval: always
choose the shortest interval; (d) Majority Class: always choose the class with
highest frequency in the training data. Furthermore, for each training
approach, we also use specific baselines. For the classification and ordinal
approaches, we use (e) Gaussian Naive Bayes (GNB) and (f) Multinomial Naive
Bayes (MNB) as additional baseline algorithms. While for regression, we
compare with the (f) Linear Regression (LN). Naive Bayes and Linear Regression
are both simple ML models with high bias. They are used here as baselines
given their easy interpretation.

ML model training process. For each dataset, we split the car parking events
into train and test with 0.8 and 0.2 ratio respectively without shuffle the
data. To avoid data leakage, we ordered the car parking events using their
timestamps before split. When training the models on the training data, we use
a stratified cross-validation with 5 folds. After the training, for each
algorithm, the best configuration of hyper-parameters is used to retrain the
model using the whole training data and then assess its performance now using
the test set.

Evaluation metrics. To evaluate the experiment results we have used the
following measures: micro f1-score ($F1_{micro}$), macro f1-score
($F1_{macro}$) and mean absolute error (MAE). We recall that the F1 score (see
equation 3) is a weighted average of the precision (see equation 4) and recall
(see equation 5), where best value is 1 and worst is 0. The micro f1-score is
a metric where we compute an F1 score counting the total true positives, false
negatives and false positives. The macro f1-score is a metric that treats all
classes equally, then it does not take label imbalance into account. Indeed,
the macro-average computes the metric independently for each class and then
take the average, hence treating all classes equally, whereas the micro-
average will aggregate the contributions of all classes to compute the average
metric. These measures give some clues about the precision and recall of the
models on predicting the true positives. By using the MAE as in equation 2 we
want to have a more interpretative measure of our regression models since it
computes the average error of the predictions values $(\hat{y}_{i})$ compared
to the real values $(y_{i})$. The idea is to investigate how close or how far
are the predictions respect to the real values. For example, given a $Short2$
parking event, a model that predicts it as $Mid1$ is better than a model that
predicts it as $Long2$ therefore in this case the error is lower. For all
experiments, we consider the MAE obtained over the test set as comparison
criteria between the models.

$MAE=(\frac{1}{n})\sum_{i=1}^{n}\left|\hat{y_{i}}-y_{i}\right|$ (2)

$f1=2.\>\frac{precision\>.\>recall}{precision+recall}$ (3)

$precision=\>\frac{TP\>}{TP+FP}$ (4)

$recall=\>\frac{TP\>}{TP+FN}$ (5)

###  5.3 RQ 1. Accuracy of ML in predicting parking duration

In this section we address RQ1 - studying the accuracy of our car parking
event duration prediction models. At this first study, we analyse the
performance of each ML approach ($Ordinal$, $Classification,$ and
$Regression$) when predicting parking events duration. Tables 5 and 6 report
the MAE, micro f1-score and macro f1-score of the models for the two different
setups of classes intervals: Lower sensitivity and Higher sensitivity,
respectively. The MAE was used as comparison criteria to select the best
models. For each dataset and ML approach pairs, the tables indicate the
strongest baseline and report the results of the best ML models for two set of
features: (a) using only event-related features, represented as
$Alg_{\\{h,m,dw,s,wr,all\\}}$, having no spatial and occupancy features; and
(b) using event-related features with spatial and occupancy features,
represented as $Alg^{\\{spt,ocy\\}}$. Both tables also report the improvement
in percentage achieved by the ML models over the baselines. We highlight in
bold the best MAE result per dataset.

From the results, we can observe that the ML models overcome the baselines in
all the datasets, for all the training approaches. Specially the ensemble
trees models (RF and XGB) show the best results in most of the training
approaches with XGB showing the best performance. For all datasets, when using
a $Regression$ approach, we observe that the most robust baseline is the
Linear Regression (LN), whereas for the approaches $Classification$ and
$Ordinal$ the strongest baseline is the Gaussian Naive Bayes (GNB). For the
Lower sensitivity scenario reported in Table 5, when predicting over the PUCPR
test set, the best MAE performance (0.316) is reached by using an $Ordinal$
approach with the XGB algorithm trained with all the event-related, spatial
and contextual features. This result represents an improvement of +46.3%
respect to its strongest baseline (GNB). In turn, for the Highest sensitivity
scenario, as reported in Table 6, the best MAE is 0.578 and this is the model
with lowest error for the CNR dataset obtained by two ML models: $Ordinal$
with RF using all the event-related features, and $Classification$ with XGB
fed with all features, including spatial and contextual. Moreover, we observe
that the use of the spatial and occupancy features in most of the cases (16
out of 24) has improved the performance of the ML models.

When analysing the improvement of the model performance respect to the
baselines, we observe that the highest gains were reported for the datasets
with highest entropy (i.e. high uncertainty or randomness). This happened
specially for the dataset PUCPR, where the ML models obtained an average gain
higher than 43%. Furthermore, in a deeper analysis, we also computed the
$R^{2}$ of the Linear Regression model (LN) for all the datasets. We observed
that the PUCPR dataset has the lowest $R^{2}$ values (0.004 and 0.003 for the
lower and higher sensitivity scenarios respectively, against 0.35 and 0.32
computed for the CNR dataset), demonstrating the difficulty of linear models
to learn the parking event duration patterns for this parking lot. Altogether,
these analysis show an consistent advantage in the use of low bias ML models
such as the XGB to predict parking event duration. This is due to the implicit
randomness and non-linearity of such events.

Dataset | Approach | Baseline | Algorithm | $MAE$ | $F1_{micro}$ | $F1_{macro}$
---|---|---|---|---|---|---
CNR | Classification | GNBh | $RF_{h}$ | 0.345 (+5.8%) | 0.714 (+7.6%) | 0.597 (+32.2%)
XGB${}_{all}^{ocy,spt}$ | 0.342 (+6.5%) | 0.714 (+7.6%) | 0.606 (+34.2%)
Ordinal | GNBh | $RF_{h}$ | 0.342 (+11.6%)* | 0.703 (+9.4%) | 0.591 (+9.6%)
XGB${}_{h}^{ocy,spt}$ |  0.336 (+13.1%) | 0.716 (+11.4%) | 0.605 (+12.2%)
Regression | LNall |  SVMall | 0.417 (+11.6%) | 0.696 (+3.3%) | 0.586 (+13.8%)
SVM${}_{all}^{ocy}$ | 0.397 (+13.2%) | 0.689 (+2.3%) | 0.581 (+13.0%)
PUCPR | Classification | GNBh |  XGBh | 0.341 (+42.0%) | 0.702 (+21.8%) | 0.701 (+49.8%)
XGB${}_{all}^{ocy,spt}$ |  0.316 (+46.3%) | 0.735 (+27.5%) | 0.73 (+55.9%)
Ordinal | GNBh |  XGBall | 0.336 (+42.8%) | 0.717 (+24.4%) | 0.712 (+52.1%)
RF${}_{all}^{ocy,spt}$ | 0.317 (+46.1%) | 0.726 (+26.0%) | 0.717 (+53.2%)
Regression | LNall |  RFall | 0.39 (+43.2%) | 0.693 (+74.0%) | 0.692 (+132.1%)
RF${}_{all}^{ocy,spt}$ | 0.4 (+41.8%) | 0.692 (+73.7%) | 0.691 (+131.6%)
UFPR04 | Classification | GNBh |  RFh | 0.556 (+3.6%)* | 0.568 (+3.0%) | 0.504 (+9.8%)
XGB${}_{all}^{ocy,spt}$ | 0.51 (+11.5%)* | 0.593 (+7.5%) | 0.547 (+19.2%)
Ordinal | GNBh |  XGBall | 0.531 (+7.9%) | 0.573 (+3.8%) | 0.54 (+17.7%)
XGB${}_{all}^{ocy,spt}$ |  0.506 (+12.2%)* | 0.556 (+0.7%) | 0.525 (+14.3%)
Regression | LNall |  XGBall | 0.574 (+9.7%) | 0.494 (+9.2%) | 0.502 (+14.9%)
XGB${}_{all}^{ocy,spt}$ | 0.571 (+10.1%) | 0.481 (+6.4%) | 0.478 (+9.2%)
UFPR05 | Classification | GNBall |  RFh |  0.340 (+16.6%) | 0.702 (+8.2%) | 0.645 (+13.4%)
XGB${}_{h}^{ocy,spt}$ | 0.379 (+6.9%) | 0.672 (+3.6%) | 0.6 (+5.6%)
Ordinal | GNBall |  RFh |  0.340 (+15.1%) | 0.702 (+7.1%) | 0.645 (+11.9%)
XGB${}_{all}^{ocy,spt}$ | 0.351 (+12.2%) | 0.695 (+6.0%) | 0.636 (+10.3%)
Regression | LNall |  RFh | 0.463 (+11.0%) | 0.6 (+17.8%) | 0.592 (+61.0%)
XGB${}_{all}^{ocy,spt}$ | 0.456 (+12.4%) | 0.593 (+16.4%) | 0.571 (+55.4%)

Table 5: Car parking event duration prediction results for the lower
sensitivity scenario comparing the top 1 results of each approach:
Distribution, Classification, Ordinal Regression and Regression. All the
improvement against the respective baseline present statistical significant
differences on the residuals compared to the best baseline (Dependent t-test
for paired samples’ test with 95% confidence interval), except those indicated
with *.

Dataset | Approach | Baseline | Algorithm | $MAE$ | $F1_{micro}$ | $F1_{macro}$
---|---|---|---|---|---|---
CNR | Classification | GNBh |  XGBall | 0.603 (+3.6%) | 0.65 (+10.3%) | 0.553 (+84.8%)
XGB${}_{all}^{ocy,spt}$ |  0.578 (+7.6%) | 0.664 (+12.7%) | 0.563 (+88.3%)
Ordinal | GNBh |  RFall |  0.578 (+7.8%) | 0.655 (+16.8%) | 0.468 (+56.8%)
RF${}_{all}^{ocy,spt}$ | 0.580 (+7.6%) | 0.668 (+19.0%) | 0.567 (+89.7%)
Regression | LNall |  SVMh | 0.648 (+12.8%) | 0.594 (+27.9%) | 0.365 (+42.2%)
SVM${}_{h}^{ocy}$ | 0.654 (+12.0%) | 0.588 (+26.7%) | 0.361 (+40.7%)
PUCPR | Classification | GNBh |  XGBall | 0.703 (+48.9%) | 0.64 (+53.6%) | 0.44 (+145.5%)
XGB${}_{all}^{ocy,spt}$ |  0.672 (+51.2%) | 0.65 (+55.9%) | 0.437 (+143.8%)
Ordinal | GNBh |  RFh | 0.723 (+44.3%) | 0.633 (+18.8%) | 0.421 (+80.9%)
XGB${}_{all}^{ocy,spt}$ | 0.693 (+46.6%) | 0.657 (+23.1%) | 0.452 (+94.0%)
Regression | LNall |  RFall | 0.781 (+48.7%) | 0.511 (+109.0%) | 0.431 (+208.6%)
XGB${}_{all}^{ocy,spt}$ | 0.833 (+45.2%) | 0.473 (+93.3%) | 0.404 (+189.8%)
UFPR04 | Classification | GNBall |  RFh | 1.348 (+11.0%) | 0.423 (+3.0%) | 0.178 (+-8.6%)
XGB${}_{all}^{ocy,spt}$ |  1.166 (+23.0%) | 0.486 (+18.2%) | 0.292 (+50.2%)
Ordinal | GNBh |  XGBall | 1.249 (+9.3%) | 0.465 (+4.7%) | 0.232 (+17.5%)
XGB${}_{all}^{ocy,spt}$ | 1.178 (+14.5%) | 0.461 (+3.7%) | 0.283 (+43.4%)
Regression | LNall |  XGBall | 1.252 (+9.3%) | 0.191 (+53.3%) | 0.167 (+42.0%)
XGB${}_{h}^{ocy,spt}$ | 1.246 (+9.7%) | 0.183 (+46.7%) | 0.186 (+58.0%)
UFPR05 | Classification | GNBall |  RFh |  0.854 (+23.4%) | 0.567 (+25.1%) | 0.322 (+45.4%)
XGB${}_{h}^{ocy,spt}$ | 0.977 (+12.3%) | 0.535 (+17.9%) | 0.313 (+41.4%)
Ordinal | GNBall |  RFh |  0.854 (+16.6%) | 0.567 (+8.0%) | 0.322 (+29.7%)
XGB${}_{h}^{ocy,spt}$ | 0.93 (+9.1%) | 0.558 (+6.2%) | 0.322 (+29.8%)
Regression | LNall |  XGBall | 0.987 (+13.3%) | 0.323 (+40.4%) | 0.232 (+41.9%)
XGB${}_{h}^{ocy,spt}$ | 0.975 (+14.4%) | 0.346 (+50.5%) | 0.284 (+74.1%)

Table 6: Car parking event duration prediction results for the higher
sensitivity scenario. Comparing the top-1 results of each approach:
Distribution, Classification, Ordinal Regression and Regression. All the
improvements against the respective baseline present statistical significant
differences on the residuals compared to the best baseline (Dependent t-test
for paired samples’ test with 95% confidence interval).

###  5.4 RQ 2. Measuring Features importance

In section, addressing Research Question N. 2, we study the feature importance
scores of the best models for each dataset and sensitivity scenario. Feature
importance scores can provide insights of how the ML models works and what can
be further improved. The relative scores can highlight which features are most
relevant for the model to predict the target values, and the converse, which
features are the least relevant.

Tables 7 and 8 show the relative importance score of the best predictor of
each dataset in the two scenarios. These measures allows us to discover what
are the most predictive features in our datasets. The higher is the feature’s
importance, the more informative the feature is for the prediction of the
parking event duration. In most of the datasets, the hour of the day $h$ is
the most important feature, indicating a relevant relationship between the
time that the parking event happens and its duration. Indeed, in the UFPR05
dataset, for both the lower and higher sensitivity scenarios, the best model
was achieved by only using the hour of the day $h$ as a feature. The day of
the week $dw$ has shown low relevance having importance values close to zero.
This means that no relevant changes concerning how the parking events happen
along the week’s days have been observed in our datasets. When analysing the
features with spatial properties $s$ indicating the parking slot and the
spatial cluster $spt$, we can observe that the ML models cannot learn
individual slot patterns in general. Indeed, the feature $s$, that
discriminates the single parking slot, has shown very low relevance having
most of the values close to zero. This means that the models did not find
distinguishable patterns over individual slots. These results could be
potentially different in the case of parking lots where the parking slots are
exclusively assigned to specific drivers. Differently, for the experiments
considering the spatial cluster $spt$, the ML model assigns a relevant weight
to this feature. This means that the models were able to learn and use the
dynamic of the parking events within the group of slots (i.e. areas of the
parking lot) to predict the duration of parking events.

Dataset | Approach | Algorithm | h | tm | dw | s | wr | spt | ocy
---|---|---|---|---|---|---|---|---|---
CNRPark | Ordinal |  XGB${}_{h}^{ocy,spt}$ | 0.547 | 0.000 | 0.000 | 0.000 | 0.000 | 0.230 | 0.223
PUCPR | Classification |  XGB${}_{all}^{ocy,spt}$ | 0.467 | 0.045 | 0.072 | 0.041 | 0.098 | 0.077 | 0.200
UFPR04 | Ordinal |  XGB${}_{all}^{ocy,spt}$ | 0.181 | 0.059 | 0.063 | 0.055 | 0.242 | 0.110 | 0.290
UFPR05 | Ordinal |  RFh | 1.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000

Table 7: Relative importance of the variables for the lower sensitivity
scenario

Dataset | Approach | Algorithm | h | tm | dw | s | wr | spt | ocy
---|---|---|---|---|---|---|---|---|---
CNRPark | Classification |  XGB${}_{all}^{ocy,spt}$ | 0.410 | 0.064 | 0.054 | 0.036 | 0.081 | 0.186 | 0.170
PUCPR | Classification |  XGB${}_{all}^{ocy,spt}$ | 0.284 | 0.031 | 0.038 | 0.027 | 0.077 | 0.132 | 0.410
UFPR04 | Classification |  XGB${}_{all}^{ocy,spt}$ | 0.157 | 0.059 | 0.049 | 0.043 | 0.231 | 0.166 | 0.294
UFPR05 | Classification |  RFh | 1.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000

Table 8: Relative importance of the variables in the higher sensitivity
scenario

Overall, for all the datasets, the occupancy feature $ocy$ has shown
significant importance, having the highest straight for the PUCPR dataset in
the higher sensitivity scenario. However, it has to be noted that the
occupancy feature $ocy$ is computed from the parking event starting time,
therefore, a further study is needed to to analyse the correlation between
$ocy$ and $h$. In regression analysis, we use the term multicollinearity to
indicate a model including multiple features that are correlated, not to the
target variable, but also to each other. The plot in Figure 6 helps us in this
analysis. The plot shows, for each dataset, the occupancy status in percentage
for four different spatial clusters defined by the algorithm K-means. We can
observe that there is no strict correlation between the hour of the day and
the occupancy of the parking lot areas. We see that the occupancy increases
from beginning of the day, then it drops at lunchtime and then again late in
the afternoon. These drops in occupancy over the day clearly show a non-
redundant relationship between the $h$ and $ocy$, evidencing then that their
effects on the ML models are actually distinguishable.

Figure 6: Visualization of the occupancy features values over the time of the
day of four spatial clusters defined by using the Kmeans algorithm. The plot
help us to show the non-perfect linear relationship between the time and the
occupancy of the parking.

##  6 Discussions

The use of EV batteries as flexible energy storage opens up several research
questions on integrating charging vehicles with the electrification of
buildings and users’ transport patterns. At the core of these challenges there
is the development of smart energy management systems. An energy management
system (EMS) monitors, controls, and optimises the distributed resources’
operation to achieve specific operational and economic objectives (e.g.,
minimise costs). As discussed in Section 3, the proposed energy management
system facilitates EVs’ integration as movable V2G assets. Such a system
requires implementing a parking occupancy prediction module based on
historical data, identifying driver behaviours, and using machine learning
algorithms for forecasting the parking time for each parking slot. The
analysis of various machine learning algorithms across four different parking
datasets to predict parking behaviours allowed us to establish a methodology
to develop the energy management system’s parking prediction module. Figure 7
shows the accuracy and the mean average error of the three machine learning
approaches tested against the four datasets covering the lower sensitivity
categories (3 categories). For this analysis, we use the prediction values of
the best models reported in Tables 5 and 6. In such a case, the identification
of the best prediction algorithm is not straightforward. The compared results
highlight each dataset’s limitation, providing an insight into the data
quality and consistency of behaviour. The events of the CNR dataset result in
the most difficult to predict, especially in the short term parking category.
On the opposite, UPFR05 looks more predictable among the four.

Both classification and regression algorithms produce a similar performance
for long term parking; however, classification is more accurate on the short
term forecasting while regression has an overall lower mean average error in
the medium range. The parking prediction module based on classification could
provide a better user experience to drivers because accurate identification of
short-term parking will force the controller to guarantee a higher energy
share to short-term park events. However, it will reduce the peak shaving
capabilities of the parking area. On the other hand, a regression model could
facilitate demand response measures because of forecasted parking events
shifted towards long parking time.

Figure 8 shows the predictive algorithms’ results on the higher sensitivity
categories (6 categories). The results confirm the challenges in forecasting
short term parking events for the CNR dataset while it confirms the best
prediction performance for the UPFR05 data. In this context, both
classification and ordinal approaches are the most accurate for short term
parking events ($<$30 mins), while regression is the most accurate for long
term parking events ($>$ 8hr). In the latter case, the ordinal approach is the
more accurate for three datasets over four, while predicting UPFR04 shows a
high percentage of errors. All the approaches result in low accuracy for
categories with a smaller number of events, such as categories 2 (30-60 min)
and 3 (1hr-2hr). Such a lower score depends on the limited amount of data
available for training. In comparison, high-frequency events are most likely
to be correctly predicted, as illustrated in the CNR dataset, category 5 (4-8
hrs) and in UPFR04, UPFR05 and PCUPR for category 1 ($<$30 min). Overall, the
lower sensitivity models are more accurate that to be integrated in an
optimisation module for energy management systems. The high sensitivity
predictions result suffering from low accuracy and high mean average error
that could lead to system malfunctions and uncertainty. Therefore, to
integrate the parking prediction module in an energy management system,
further improvement of the forecasts are necessary, especially for high
sensitivity experiments. Additional data could be used to improve the
prediction’s accuracy, such as higher picture resolution that could better
identify users by reading the licence plates or identify unique marks
(stickers, internal objects or scratches). Additionally, other data sources
can be employed to forecast the number of car at an aggregated level and
compare it with similar works.

Figure 7: For lower sensitivity (3 intervals) and for the four datasets
analysed, comparison of the accuracy and MAE for the prediction Figure 8: For
higher sensitivity (6 intervals) and for the four datasets analysed,
comparison of the accuracy and MAE for the prediction

The current work aims to provide an overall design of a smart charging energy
management system to optimally integrate the distributed energy systems and
EVs into the power grid by developing a parking prediction module to estimate
the vehicles’ parking time using machine learning algorithms. The proposed
system can capture EVs users’ aggregated uncertain behaviour to obtain an
optimised solution for both the capital expenditures (CAPEX) and operational
expenditures (OPEX) at the network’s planning and operation phases. CAPEX can
be minimised by optimising and distributing distributed energy resources and
charging stations for electric vehicles under economic and social constraints.
At the same time, optimal power flow solutions considering technical
constraints can lead to OPEX minimisation.

##  7 Conclusions and Future Works

The provision of energy system services plays a critical role for the power
system as renewable energy sources are displacing conventional generation, and
system demand is soaring due to the electrification of transport and thermal
loads. The provision of energy services using demand response programs is an
area of great potential for system operators; however, there remain several
challenges in proving the effectiveness of such schemes integrated with the
transport systems. The growing penetration of EVs with larger battery size
challenges the distribution network’s capacity, and it is becoming a threat to
the grid’s reliability. Nevertheless, suppose electric vehicles enable V2G
operations, and parking premises provide a data harvesting infrastructure to
produce accurate parking slot predictions. In that case, such a mobile storage
capacity can become an asset to improve grid reliability and resiliency and
used to defer capital upgrade costs.

Future works include the use of anonymised user profiles to reach more
accurate predictions based on the single user habits, as well as having more
dense and richer datasets to improve the accuracy of the models. Another
direction is the proper integration of the prediction module into an Energy
Management System as illustrated above in Section 3. Specifically, when a car
starts its parking at a parking slot (detected by a camera or by the
connection to the charging station) the prediction of the parking duration
will feed, together with the energy models and other contextual data, the
model predictive controller that will allocated the energy to the charge
station in the slots. This will foster new research directions like for
example a combined model of parking duration prediction and energy
optimization models to solve the power supply-demand balancing problem by
aggregating EVs, renewable energy systems, energy storage, and demand response
at distribution level. Additionally, the energy management system equipped
with the module for the parking prediction can trade the expected energy
consumption in the electricity market reducing the overall energy cost.

## Acknowledgment

The work is supported by the ERA-NET Smart Energy System, Sustainable Energy
Authority Ireland and Italian Ministry of Research with project N.
ENSGPLUSREGSYS18_00013. This publication has emanated from research conducted
with the financial support of of the EVCHIP project under grant agreement
19/RDD/579, EVCHIP.

## References

  * [1] Ata Ajoulabadi, Sajad Najafi Ravadanegh, and Behnam Mohammadi-Ivatloo.  Flexible scheduling of reconfigurable microgrid-based distribution networks considering demand response program.  Energy, 196:117024, 2020.
  * [2] G. Amato, F. Carrara, F. Falchi, C. Gennaro, and C. Vairo.  Car parking occupancy detection using smart camera networks and deep learning.  In 2016 IEEE Symposium on Computers and Communication (ISCC), pages 1212–1217, 2016.
  * [3] Gonzalo Antolín, Angel Ibeas, Borja Alonso, and Luigi dell’Olio.  Modelling parking behaviour considering users heterogeneities.  Transport Policy, 67:23–30, 2018.
  * [4] James S Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl.  Algorithms for hyper-parameter optimization.  In Advances in neural information processing systems, pages 2546–2554, 2011.
  * [5] Felix Caicedo, Carola Blazquez, and Pablo Miranda.  Prediction of parking space availability in real time.  Expert Systems with Applications, 39(8):7281 – 7290, 2012.
  * [6] Yu-Wei Chung, Behnam Khaki, Tianyi Li, Chicheng Chu, and Rajit Gadh.  Ensemble machine learning-based algorithm for electric vehicle user behavior prediction.  Applied Energy, 254:113732, 2019.
  * [7] Paulo RL De Almeida, Luiz S Oliveira, Alceu S Britto Jr, Eunelson J Silva Jr, and Alessandro L Koerich.  Pklot–a robust dataset for parking lot classification.  Expert Systems with Applications, 42(11):4937–4949, 2015.
  * [8] Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu.  A density-based algorithm for discovering clusters in large spatial databases with noise.  In Proceedings of the Second International Conference on Knowledge Discovery and Data Mining (KDD-96). AAAI Press, 1996.
  * [9] Duda-Wiertel, U. and Szarata, A.  The analysis of transport-related behaviours of drivers in highly occupied paid parking zones.  Advances in Transportation Studies, Volume 47, 2019
  * [10] Eibe Frank and Mark Hall.  A simple approach to ordinal classification.  In European Conference on Machine Learning, pages 145–156. Springer, 2001.
  * [11] Clark W Gellings.  The smart grid: enabling energy efficiency and demand response.  CRC press, 2020.
  * [12] S. S. Ghosal, A. Bani, A. Amrouss, and I. El Hallaoui.  A deep learning approach to predict parking occupancy using cluster augmented learning method.  In 2019 International Conference on Data Mining Workshops (ICDMW), pages 581–586, 2019.
  * [13] Estefanía Hervas-blasco, Emilio Navarro-Peris, Mattia De Rosa, and José M. Corberán.  Potential fuel saving in a powertrain derived from the recovery of the main energy losses for a long haul european mission.  Energy Conversion and Management, 150:485–499, 2017.
  * [14] M. S. Islam, N. Mithulananthan, and K. Y. Lee.  Suitability of pv and battery storage in ev charging at business premises.  IEEE Transactions on Power Systems, 33(4):4382–4396, 2018.
  * [15] Sašo Karakatič.  Optimizing nonlinear charging times of electric vehicle routing with genetic algorithm.  Expert Systems with Applications, 164:114039, 2021.
  * [16] Eric Hsueh-Chan Lu and Chen-Hao Liao.  Prediction-based parking allocation framework in urban environments.  International Journal of Geographical Information Science, 34(9):1873–1901, 2020.
  * [17] Ji Lu, Yuning Chen, Jin-Kao Hao, and Renjie He.  The time-dependent electric vehicle routing problem: Model and solution.  Expert Systems with Applications, 161:113593, 2020.
  * [18] Xiaoxing Lu, Kangping Li, Hanchen Xu, Fei Wang, Zhenyu Zhou, and Yagang Zhang.  Fundamentals and business model for resource aggregator of demand response in electricity markets.  Energy, 204:117885, 2020.
  * [19] Xiaolong Ma, Xiaoduan Sun, Yulong He, and Yixin Chen.  Parking choice behavior investigation: A case study at beijing lama temple.  Procedia-Social and Behavioral Sciences, 96:2635–2642, 2013.
  * [20] J. B. MacQueen.  Some methods for classification and analysis of multivariate observations.  In Proceedings of 5-th Berkeley Symposium on Mathematical Statistics and Probability, 1967.
  * [21] Madeleine McPherson and Brady Stoll.  Demand response for variable renewable energy integration: A proposed approach and its impacts.  Energy, 197:117205, 2020.
  * [22] Zhenyu Mei, Wei Zhang, Lihui Zhang, and Dianhai Wang.  Real-time multistep prediction of public parking spaces based on fourier transform–least squares support vector regression.  Journal of Intelligent Transportation Systems, 24(1):68–80, 2020\.
  * [23] Dragan D Milošević, Ivana V Bajšanski, and Stevan M Savić.  Influence of changing trees locations on thermal comfort on street parking lot and footways.  Urban forestry & urban greening, 23:113–124, 2017.
  * [24] Pallonetto, Fabiano and Galvani, Marta and Torti, Agostino and Vantini, Simone  A Framework for Analysis and Expansion of Public Charging Infrastructure under Fast Penetration of Electric Vehicles  World Electric Vehicle Journal, Vol 11, N 1, 2020
  * [25] Yuecheng Rong, Zhimian Xu, Ruibo Yan, and Xu Ma.  Du-parking: Spatio-temporal big data tells you realtime parking availability.  In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’18, page 646–654, New York, NY, USA, 2018. Association for Computing Machinery.
  * [26] Yu Wu, Alexandre Ravey, Daniela Chrenko, and Abdellatif Miraoui.  Demand side energy management of ev charging stations by approximate dynamic programming.  Energy Conversion and Management, 196:878 – 890, 2019.
  * [27] Shuai Zhang, Mingzhou Chen, Wenyu Zhang, and Xiaoyu Zhuang.  Fuzzy optimization model for electric vehicle routing problem with time windows and recharging stations.  Expert Systems with Applications, 145:113123, 2020.
  * [28] Fang Zong, Ping Yu, Jinjun Tang, and Xiao Sun.  Understanding parking decisions with structural equation modeling.  Physica A: Statistical Mechanics and its Applications, 523:408 – 417, 2019.


# Fitting the BumpHunter test statistic distribution and global p-value
estimation

Louis Vaslin1, Samuel Calvet1, Vincent Barra2, and Julien Donini1

( 1LPC, Université Clermont Auvergne, CNRS/IN2P3, Clermont-Ferrand; France
2Université Clermont Auvergne, CNRS, LIMOS, UMR 6158, Clermont-Ferrand; France
November 14, 2022 )

###### Abstract

In high Energy Physics, it is common to look for a localized deviation in data
with respect to a given reference. For this task, the well known BumpHunter
algorithm allows for a model-independent deviation search with the advantage
of estimating a global p-value to account for the Look Elsewhere Effect.
However, this method relies on the generation and scan of thousands of pseudo-
data histograms sampled from the reference background. Thus, accurately
calculating a global significance of $5\sigma$ requires a lot of computing
resources. In order to speed this process and improve the algorithm, we
propose in this paper a solution to estimate the global p-value using a more
reasonable number of pseudo-data histograms. This method uses a functional
form inspired by similar statistical problems to fit the test statistic
distribution. We have found that this alternative method allows to evaluate
the global significance with a precision about 5% up to the $5\sigma$
discovery threshold.

## Introduction

BumpHunter [1] is a well known algorithm that searches for a localized
deviation in a data distribution with respect to a reference background and
assesses its local and global statistical significance. Such an algorithm is
useful in various domains, including in the search for new resonances in High
Energy Physics (HEP) [2, 3]. In order to provide a public and practical
implementation of the BumpHunter algorithm, a Python version named
pyBumpHunter has been recently developed [4]. This implementation provides
several extensions to the original algorithm proposed in [1], including
signal-injection sensitivity test, 2D BumpHunter, side-band normalization and
multi-channel combinations.

The features proposed in pyBumpHunter can be applied to perform a model
independent signal search. However, as discussed in [5], the way the global-
value is computed could be improved. Indeed, BumpHunter relies on the sampling
of a large number of background-only distributions to compute the global
p-value. With such a method, the number of pseudo-data distributions required
to estimate a global significance around $5\sigma$ becomes very high (several
millions), which makes it quite time consuming.

In order to solve this issue, we propose a solution to calculate high global
significances at a lower cost using a fit of the BumpHunter test statistics
distribution. This procedure has been studied in order to determine any
potential bias on the global significance. The code used to produce the
results presented in this paper is available on GitHub [6].

##  1 The BumpHunter algorithm

The BumpHunter algorithm searches for a deviation in an observed data
histogram with respect to a reference model by scanning the distribution with
several variable-width windows. Local p-value are calculated for all tested
interval positions (and widths), and the interval with the smallest p-value is
retained as the one which presents the most significant deviation with respect
to the reference. The selected local p-value $p$ is transformed into a test
statistic value $t$ with:

$t=-\ln(p).$ (1)

The procedure is repeated on thousands of pseudo-data histograms generated by
smearing the bin content of the reference with a Poisson law. This gives a
distribution of local p-values that corresponds to the minimum p-values that
were selected for each pseudo-data histogram. Following equation 1, this
distribution is transformed into a test statistic distribution that will be
used to compute the global p-value and significance. This procedure is used to
account for the Look Elsewhere Effect [7].

The problem arises when the test statistic value associated with observed data
is higher than all the test statistic values associated with the background-
only pseudo-data. In this case the global p-value will be exactly 0, which
corresponds to an infinite global significance. In addition, the smallest non-
null global p-value that can be computed with this method is equal to the
inverse of the total number of generated pseudo-data. Therefore in order to
reach $5\sigma$ global significance with a sufficient number of pseudo-data
histograms that have a test statistic value greater than the one observed in
data, it is necessary to produce and scan millions of pseudo-data histograms.

##  2 Fitting the BumpHunter test statistic distribution

In order to reduce the number of pseudo-data distributions required to compute
the global p-value, one solution is to fit the test statistic distribution
with a function. This function can then be used to extrapolate the
distribution to higher values, allowing to reach smaller p-values (and higher
significances). A potentially suitable function can be found using the
“p-value hacking” method proposed in [8].

In this work, we consider the following problem: a test is performed $m$ times
on a population of $n$ individuals. For each of these tests, a p-value is
assessed, resulting in a distribution of $m$ p-values. Then, in order to
retain an unique p-value for this population only the minimum p-value among
these $m$ tests is considered. This procedure can be repeated many times on
different populations, each of them being assigned a minimum p-value. In the
end, a distribution of minimum p-values is obtained. The analytic form of this
distribution $\varphi$ is given by [8]:

$\displaystyle\varphi(p;p_{M})={}$ $\displaystyle
m\,e^{\mathrm{erfc}^{-1}(2p_{M})\left(2\mathrm{erfc}^{-1}(2p)-\mathrm{erfc}^{-1}(2p_{M})\right)}$
(2)
$\displaystyle\left(1-\frac{1}{2}\mathrm{erfc}\\!\left(\mathrm{erfc}^{-1}(2p)-\mathrm{erfc}^{-1}(2p_{M})\right)\right)^{m-1}$

with $p$ the minimum p-value, $m$ the number of tests, $p_{M}$ the true median
of the p-value distribution and erfc the complementary error function. Here
the denomination “true median” comes from the fact that, in such a procedure,
the p-value associated with each population is biased since the most favorable
occurrence is chosen in a set of repeated tests. Thus, this effect has to be
accounted for when computing the median of the distribution.

One can relate the local p-value distribution obtained with BumpHunter with
the problem treated in [8]. In the case of BumpHunter, the $n$ individuals of
the population can be identified to the number of events in the tested
intervals. As for the $m$ tests from which the minimum local p-value is
chosen, they correspond to all the different intervals that were tested for a
given pseudo-data histogram. Then, the test is repeated on different
populations, or in the case of BumpHunter, on different pseudo-data
histograms. With this equivalence established, it is reasonable to suppose
that the formula given in equation 2 could be used to fit the BumpHunter
(minimum) local p-value distribution. Equation 1 can then be used to transform
the minimum p-value distribution $\varphi(p)$ into the BumpHunter test
statistic distribution $\Phi(t)$.

Assuming this procedure would give a suitable analytical function to fit the
BumpHunter test statistics distributions, there are two aspects that should be
evaluated. The first one concerns the number of individuals $n$ that compose
the tested population. In the development presented in [8], this number is
assumed to be high enough to allow the use of the Central Limit Theorem, so
that equation 2 could be derived. In the case of BumpHunter, the relevant
number corresponds to the number of individuals that fall in the tested
intervals. As this number of events depends on the available statistics as
well as on the shape of the scanned distribution, there are cases where the
limit condition is not verified, for example when considering an exponentially
falling background with low statistics in the tail.

Another potential limitation comes from the $m$ tests from which the minimum
p-value is chosen. In the problem treated in [8], the author states from the
beginning that the $m$ tests are independent. In the case of BumpHunter, this
condition is usually not verified. The only case where the condition stands is
when the width of the scan window is equal to one bin. In any other cases, the
tested intervals will overlap such as to induce correlations between the
different tests, making them not fully independent. For these two reasons,
studies are required in order to verify if the solution proposed in [8]
applies correctly to the case of BumpHunter or if any potential biases should
be taken into account. These studies are the object of the next section.

##  3 Test of the fit procedure and bias evaluation

In order to test the behaviour of the BumpHunter test statistic fit, the
following procedure has been established :

  * •

A data histogram and a reference histogram are generated by sampling from the
same probability density function.

  * •

A low-statistics test statistic distribution is produced using $5\times
10^{4}$ pseudo-data histograms. The global p-value $p_{fit}$ is evaluated by,
first, fitting the test statistic distribution using equation 2 and then
integrating this function for different thresholds of test statistics values
($t_{data}$).

  * •

A high-statistics test statistic distribution is produced using, this time,
$2\times 10^{7}$ pseudo-data histograms. The higher statistics allows to
compute a global p-value $p_{BH}$ directly from the $t$ distribution without
relying on a fit. Again, this p-value will be evaluated for different
thresholds of data test statistics values.

  * •

A bias factor ($R$) on the global significance is defined as the ratio between
the global significance calculated using the high-statistics test distribution
($\sigma_{BH}$) and the one calculated using the low-statistics distribution
fit function ($\sigma_{fit}$) :

$R=\frac{\sigma_{BH}}{\sigma_{fit}}$ (3)

In order to cover multiple cases, this procedure is applied with different
background shapes and statistics. For the first test, we use an uniformly
distributed background with $10^{5}$ events in the range $[0,35]$ and 20 bins
of equal width. This example covers the case where the available statistics in
each bin is high enough ($n>30$). The width of the scan window is set to 1 bin
to ensure that tested intervals don’t overlap with each other, making all
intervals independent. A second test is performed in the same setup, but using
an exponentially distributed background with 1k events generated in the range
[0, 35] with the same binning as previously (the exponential has a scale of
3.5). Thus, the case where some intervals have a low statistic, in the tail of
the distribution, can be studied. Finally, a third test is performed using the
same uniform distribution as in the first test, but with a scan window width
set to 5 bins. This way, there are up to 4 overlapping bins between intervals
and this example serves to study a case where the $m$ tests from which the
minimum p-value is selected are not fully independent.

Figure 1 shows the results obtained in the first, say optimal, case. The left
panel shows the test statistics distribution obtained from the $5\times
10^{4}$ pseudo-experiment (blue histogram) together with the fitted function
(orange line). The fit is of good quality, with a $\chi^{2}/ndf$ test value of
1.45. The panel on the right shows the evolution of the global significance,
as function of $t_{data}$, when it is calculated either with the fit function
(solid orange) or by using the direct high-statistics method (dashed blue).
The two curves are very close and can not be distinguished by eye. The bias
ratio as a function of $t_{data}$ confirms that the global significances are
compatible within 1%. The variations that can be observed when $t_{data}>15$
are due to the low statistic available in the tail of the test statistic
distribution used to compute the global p-value using the direct method.

Figure 1: Results of the BumpHunter test statistic fit in the ideal case. The
left plot shows the test statistic histogram associated to pseudo-data with
the fitted function. The top plots show the evolution of the global
significance obtained with the direct method and using the fit as function of
$t_{data}$ together with the bias ratio.

Figure 2 shows the results obtained when the statistics is low in the tail of
the scanned histograms. The quality of the fit shown on the left panel is
quite poor, indeed, the test statistic distribution is not smooth anymore and
presents numerous spikes. In fact, when the available statistic in the
selected interval is low, there are very few possible values that the local
p-value can take compared to the first case. Nonetheless, the right panel
shows that the evolution of the global significance obtained using the fit
function closely follows the one obtained with the direct method. The ratio
confirms that the bias induced by the fit on the global significance is within
4% for all the tested values of $t_{data}$. Only the last points corresponding
to $t_{data}$ greater than 16 diverge because of the available statistics for
the direct method.

Figure 2: Results of the BumpHunter test statistic fit in the case when the
statistic is low in the tail of the scanned distributions. The left plot shows
the test statistic histogram associated to pseudo-data with the fitted
function. The right plot shows the evolution of the global significance
obtained with the direct method and using the fit as function of $t_{data}$
together with the bias ratio.

Figure 3 illustrates the case when the tested intervals overlap. The left
panel shows that the fit is of good quality, with a $\chi^{2}/ndf$ value of
1.06, and the test statistic distribution is as smooth as in the first case.
However, the right panel shows that the global significance obtained using the
fit function diverge from the ones obtained with the direct method when
$t_{data}$ is greater than 9. This tendency is confirmed when computing the
bias ratio. This bias increases with $t_{data}$ and reaches up to 2% for
$t_{data}=13$. For higher values, the global significance computed using the
direct method shows significant variations due the reasons discussed in the
previous cases.

Figure 3: Results of the BumpHunter test statistic fit in the case when the
tested intervals overlap. The left plot shows the test statistic histogram
associated to pseudo-data with the fitted function. The right plot shows the
evolution of the global significance obtained with the direct method and using
the fit as function of $t_{data}$ together with the bias ratio.

In addition to the three cases presented here, other tests have been performed
using different scan window widths, as well as reference background
distributions of different statistics and binning. The results have shown that
the bias induced by the fit on the global significance stays below 5% in any
case. We have also seen that the shape of the background distribution doesn’t
affect much the shape of the test statistic distribution as long as the
statistic available in the scanned distribution is high enough.

## Conclusion

We have defined a procedure that allows to compute the global p-value by
fitting the BumpHunter test statistics distribution. This procedure is
relevant when the global significance becomes too high to be computed
efficiently using the direct method originally defined in [1]. We could verify
that, when all the validity conditions of the fit function are met, there is
no bias on the global significance induced by the fit procedure. We could also
verify that, in the cases where these conditions are not met, the bias on the
global significance doesn’t go over 5% up to significance of $5\sigma$. Other
tests using alternative background distribution with different binning and
multiple scan window widths tend to confirm this statement. This encouraging
result motivates the implementation of an automatic fit procedure in the
future releases of pyBumpHunter.

## References

  * [1] Georgios Choudalakis.  On hypothesis testing, trials factor, hypertests and the BumpHunter.  In PHYSTAT 2011, Jan 2011.
  * [2] ATLAS Collaboration.  Search for new resonances in mass distributions of jet pairs using 139 fb-1 of pp collisions at $\sqrt{\mathrm{s}}$ = 13 TeV with the ATLAS detector.  Journal of High Energy Physics, 2020(3):145, Mar 2020.
  * [3] ATLAS Collaboration.  Search for $t\overline{t}$ resonances in fully hadronic final states in pp collisions at $\sqrt{s}$= 13 TeV with the ATLAS detector.  Journal of High Energy Physics, 2020(10):61, Oct 2020.
  * [4] Louis Vaslin.  GitHub \- scikit-hep/pyBumpHunter at v0.4.0, Apr 2022.  v0.4.0 release on 2022-04-04.
  * [5] Louis Vaslin, Samuel Calvet, Vincent Barra, and Julien Donini.  pyBumpHunter: A model independent bump hunting tool in python for high energy physics analyses, 2022.  arXiv:2208.14760 [hep-ex].
  * [6] Louis Vaslin.  GitHub - lovaslin/pyBH-fit, May 2022.  Last commit on 2022-10-19.
  * [7] Eilam Gross and Ofer Vitells.  Trial factors for the look elsewhere effect in high energy physics.  The European Physical Journal C, 70:525–530, 2010.
  * [8] Nassim Nicholas Taleb.  A short note on p-value hacking, 2016.  arXiv:1603.07532.