11institutetext: Faculdade de Engenharia da Universidade do Porto, Porto,
Portugal
11email<EMAIL_ADDRESS>
11email<EMAIL_ADDRESS>22institutetext: Laboratory for
Artificial Intelligence and Computer Science (LIACC), Portugal
33institutetext: Fraunhofer Portugal AICOS, Portugal
33email<EMAIL_ADDRESS>
# Meta-learning and Data Augmentation for Stress Testing Forecasting
Models††thanks: This work was partially funded by projects AISym4Med
(101095387) supported by Horizon Europe Cluster 1: Health, ConnectedHealth
(n.º 46858), supported by Competitiveness and Internationalisation Operational
Programme (POCI) and Lisbon Regional Operational Programme (LISBOA 2020),
under the PORTUGAL 2020 Partnership Agreement, through the European Regional
Development Fund (ERDF) and NextGenAI - Center for Responsible AI
(2022-C05i0102-02), supported by IAPMEI, and also by FCT plurianual funding
for 2020-2023 of LIACC (UIDB/00027/2020 UIDP/00027/2020)

Ricardo Inácio 1122 0009-0008-6435-5245 Vitor Cerqueira 1122
0000-0002-9694-8423 Marília Barandas 33 0000-0002-9445-4809 Carlos Soares
112233 0000-0003-4549-8917

###### Abstract

The effectiveness of univariate forecasting models is often hampered by
conditions that cause them stress. A model is considered to be under stress if
it shows a negative behaviour, such as higher-than-usual errors or increased
uncertainty. Understanding the factors that cause stress to forecasting models
is important to improve their reliability, transparency, and utility. This
paper addresses this problem by contributing with a novel framework called
MAST (Meta-learning and data Augmentation for Stress Testing). The proposed
approach aims to model and characterize stress in univariate time series
forecasting models, focusing on conditions where they exhibit large errors. In
particular, MAST is a meta-learning approach that predicts the probability
that a given model will perform poorly on a given time series based on a set
of statistical time series features. MAST also encompasses a novel data
augmentation technique based on oversampling to improve the metadata
concerning stress. We conducted experiments using three benchmark datasets
that contain a total of 49.794 time series to validate the performance of
MAST. The results suggest that the proposed approach is able to identify
conditions that lead to large errors. The method and experiments are publicly
available in a repository.

###### Keywords:

Time series Forecasting Stress testing Meta-learning Data augmentation

##  1 Introduction

Time series forecasting is a relevant problem in various application domains,
such as industry, healthcare, or finance. Accurate forecasts help reduce
uncertainty about the future and foster a data-driven decision-making within
organizations [18]. However, the effective application of forecasting models
is sometimes hampered by stress conditions that reduce their performance and
reliability.

In machine learning, stress denotes conditions where a model exhibits a
negative behaviour, such as higher-than-usual errors or increased uncertainty.
Stress factors result in unreliable predictions, causing end-users to distrust
models. In the context of forecasting, stress can stem from factors such as i)
data difficulty problems, for instance out-of-distribution samples or missing
data, or ii) an inadequate inductive bias where the model fails to capture
relevant patterns in specific time series. Stress testing aims to evaluate the
reliability of machine learning approaches by identifying, modelling, or
simulating challenging scenarios. This process provides actionable insights
for improving models and decision-making by end-users.

Figure 1 illustrates the forecasting performance of a model across several
univariate time series. It shows the distribution of SMAPE (symmetric mean
absolute percentage error) incurred by a forecasting model across 1428 time
series. The distribution is right-skewed, where the long-tail denotes time
series where the model is under stress and exhibits larger errors. We will
provide further details on this figure in Section 4.

Figure 1: Forecasting performance of a model across several univariate time
series according to SMAPE.

The goal of this paper is to stress test a given forecasting model.
Particularly, identify the conditions that lead to a large error in a given
time series. Understanding these factors is an essential step to improve the
understanding of the inner workings of forecasting models. Ultimately, this
knowledge promotes a more reliable and transparent use of forecasting systems
by enabling end-users to adapt their decisions under the identified stress
conditions. We accomplish this goal by proposing a new approach for modelling
stress in univariate time series forecasting models called MAST (short for
Meta-learning and data Augmentation for Stress Testing).

Meta-learning involves using machine learning to improve the performance of
machine learning models, with tasks such as algorithm selection [6] or dynamic
model combination [9]. MAST uses meta-learning to model the occurrence of
large errors based on structural properties of time series (e.g., seasonal
strength). These structural properties are computed using time series feature
extraction methods [28, 4] and include statistics such as the average value or
standard deviation.

Large errors (relative to the typical performance of a model) represent a
small fraction of the dataset (c.f. Figure 1). A limited number of samples
complicates the process of modelling these instances. In this context, we
frame the above meta-learning problem as an imbalanced learning task.
Imbalanced domain learning is the field of machine learning devoted to
modelling datasets where the target distribution is skewed, which means most
observations belong to a single class. However, the minority class is often
the most relevant, which, in our cases, represents the cases where a model
incurs a large error. Resampling techniques are typically used to tackle class
imbalance in binary classification problems. We explore resampling methods
(e.g., SMOTE [10]) to generate synthetic samples concerning large errors. We
hypothesize that the augmented dataset improves the performance of the meta-
learning model.

We validate the proposed approach in three benchmark datasets that comprise a
total of 49.794 time series. The experiments suggest that our method is able
to identify the conditions that lead to large errors by a forecasting model.
Using Shapley values [23], we provide concrete examples about the conditions
where a given model performs poorly. The proposed method and experiments are
available online 111https://github.com/ricardoinaciopt/mast.

In summary, the contributions of this paper are the following:

  * •

A novel methodology for stress testing forecasting models based on meta-
learning and data augmentation called MAST;

  * •

A data augmentation technique based on resampling strategies that creates
synthetic observations concerning large forecasting errors;

  * •

A set of extensive experiments that validate the proposed method in benchmark
datasets, including usage examples based on Shapley values.

##  2 Literature Review

This section lays out the foundations of our research. We start by defining
the problem of univariate time series forecasting (Section 2.1), focusing on
machine learning approaches trained in a global manner. Then, we describe
several methods for data augmentation in Section 2.2. We highlight practices
designed for time series and approaches tailored for tackling class imbalance.
We also list several frameworks for time series feature extraction (Section
2.3). Finally, we briefly overview the relevance of stress testing and its
scope within responsible artificial intelligence (Section 2.4).

###  2.1 Problem definition

####  2.1.1 Univariate time series forecasting.

A univariate time series is a time-ordered sequence of observations. It can be
defined as $Y=\\{y_{1},y_{2},\dots,$ $y_{t}\\}$, where $y_{i}\in\mathbb{R}$ is
the numeric value of $Y$ collected at time $i$ and $t$ is the length of $Y$.
In this work, we aim at predicting the value of the next $h$ observations of a
given univariate time series, $y_{t+1},\ldots,y_{t+h}$, based on historical
data.

We address the forecasting task using machine learning approaches and an auto-
regressive methodology. We use sliding windows based on time delay embedding
to reconstruct a time series $Y$ for supervised learning. Each observation of
a time series is modelled as a function of its past $p$ lags [5]. Essentially,
we build a dataset $D=\\{<X_{i},y_{i}>\\}^{t}_{i=p+1}$ where $y_{i}$
represents the $i$-th observation and $X_{i}\in\mathbb{R}^{p}$ is the $i$-th
corresponding set of $p$ lags: $X_{i}=\\{y_{i-1},y_{i-2},\dots,y_{i-p}\\}$.
The goal is to build a regression model $f$ with the form $y_{i}=f(X_{i})$.

####  2.1.2 Global Forecasting Models.

We focus on forecasting problems that involve a collection of multiple
univariate time series: $\mathcal{Y}=\\{Y_{1},Y_{2},\dots,Y_{n}\\}$, where
$Y_{j}$ is the $j$-th time series and $n$ is the number of time series in the
collection.

Local forecasting methods build an individual model for each time series in a
collection $\mathcal{Y}$. Classical techniques, such as ARIMA [17], tend to
follow this approach. In contrast, recent machine learning methods applied to
forecasting tend to follow a global approach and build a model for a given
collection using the historical data of all available time series. Global
forecasting models have outperformed local ones in benchmark datasets and
competitions [15].

Using several time series to train a forecasting model has been shown to lead
to better forecasting performance [15, 8]. In practice, this is achieved by
transforming each time series using the approach described in Section 2.1.1.
Then, a global model is trained on the concatenation of the resulting datasets
$\mathcal{D}=\\{D_{1},D_{2},\dots,D_{n}\\}$, where $D_{j}$ is the dataset
built for time series $Y_{j}$. A global forecasting approach often requires
appropriate data preprocessing steps such as transformations, which we detail
in Section 4.

###  2.2 Data Augmentation

Data augmentation techniques are useful when a large or diverse enough dataset
is not available for training a model [3]. For time series, there are various
methods to create synthetic time series. These range from simple methods such
as jittering or flipping the original time series [31], to more sophisticated
approaches such as seasonal decomposition [12] or generative models [20].

Most time series data augmentation techniques aim to create synthetic time
series that preserve the properties of the original ones. There are also
approaches that attempt to create time series samples in parts of the data
space with limited examples. For instance, Cerqueira et al. [7] leverage
resampling techniques such as SMOTE [10] to increase the sample size of
particular time series within a collection $\mathcal{Y}$. SMOTE randomly
selects instances from the minority class (in the case of Cerqueira et al.
[7], individual time series in a collection), and interpolates them based on
their nearest neighbours to create new samples. While resampling methods are
typically used to mitigate class imbalance issues in classification, Cerqueira
et al. [7] apply them in forecasting problems to improve the global-local
trade-off [19].

Overall, the ultimate goal of existing time series data augmentation
techniques is to obtain an enriched dataset that leads to better forecasting
performance [8]. In this work, we leverage these methods to improve meta-
learning and stress testing a forecasting model. We explore oversampling
strategies, to generate synthetic samples in observations where a forecasting
model shows a large error, increasing the overall representation of such
instances. We focus specifically on SMOTE and ADASYN as they are well-
established methods in the literature of imbalanced domain learning [27].

###  2.3 Time Series Feature Extraction

Extracting features from time series has been shown to improve performance in
different tasks, such as forecasting [21] or classification [4]. Several
frameworks have been proposed for this task. Examples include the works by
Prudêncio and Ludermir [29], Lemke et al. [21], Montero-Manso et al. [28],
Barandas et al. [4], or Christ et al. [11].

In the context of time series forecasting, feature-based modelling is
typically used in meta-learning approaches for model selection and
combination. This process involves summarising each time series in a
collection into a set of features. Then, a metamodel is trained to predict the
most appropriate learning algorithm based on these features. For a new query
time series, this metamodel predicts which model, or set of models, should be
used for forecasting. In the benchmark M4 forecasting competition, Montero-
Manso et al. [28] applied this approach and ranked second place, outperforming
several classical and machine learning approaches. Besides forecasting, time
series feature extraction is a common approach for time series classification
tasks. In these, the label associated with a given time series is modelled
based on a set of features [4].

In this work, we use the tsfeatures framework for time series feature
extraction [28]. As mentioned above, Montero-Manso et al. [28] developed a
competitive feature-based forecasting approach using this framework. Overall,
tsfeatures contains a set of 42 features, including entropy, linearity,
lumpiness, or trend. We refer the reader to the work by Montero-Manso et al.
[28] for a comprehensive description of these.

###  2.4 Stress Testing

Building reliable and robust predictive models is a key step towards the
responsible use of machine learning [14]. This task is challenging because
real-world data is susceptible to stress conditions that undermine the
performance of models. These conditions can be caused by issues such as out-
of-distribution data, missing values, or an inadequate inductive bias
concerning some parts of the input space.

Identifying the conditions in which a model is under stress and is prone to
under-perform is a crucial aspect for supporting the adoption of machine
learning systems, especially in sensitive domains such as healthcare. Stress
testing is an emerging topic that addresses this problem [13]. The goal of
stress testing is to evaluate the reliability of machine learning approaches
by identifying, modelling and simulating challenging scenarios.

Stress testing was recently pioneered in the context of computer vision in the
work by Cunha et al. [13]. They propose an extension of generative adversarial
networks called GASTeN (Generative Adversarial Stress Testing Network) that
aims to create synthetic samples with two key properties: i) realistic and ii)
challenging, where a given classifier serves a prediction with low confidence.
Their framework enables the simulation of a diverse set of conditions where a
model is not reliable, which provides useful insights for end-users. While
they focus on generating samples that are challenging for a given model, we
adopt a meta-learning approach to model and identify the weaknesses of a
model. To our knowledge, this is the first work that explores meta-learning
for stress testing.

Similar to us, Rožanec et al. [30] also address the problem of identifying
poor forecasts made by a global model, leveraging anomaly detection methods to
this effect. However, they focus on managing the global-local trade-off and
improving forecasting performance.

##  3 Stress Testing Forecasting Models using Meta-learning

This section formalizes MAST, the proposed approach for modelling the stress
based on meta-learning. We address univariate time series forecasting problems
using datasets involving multiple time series. We tackle this task using an
autoregressive approach, as defined in Section 2.1. In this work, MAST
involves using meta-learning for modelling the performance of a forecasting
model based on time series features and data augmentation. The workflow of
MAST, which is illustrated in Figure 2, is split into two stages: i) a
development stage and; ii) an inference stage.

Figure 2: Workflow behind MAST, which is split into a development stage and an
inference stage. In the development stage, we conduct performance estimation,
feature extraction, data augmentation, and meta-learning. Then, the resulting
metamodel is applied during the inference stage to predict whether a
forecasting model will incur a large error.

###  3.1 Development stage

In the development stage, we aim at building a metamodel that predicts the
probability that a time series will induce a large error on a forecasting
model. This goal is split into four steps:

  1. 1.

Performance estimation: Conducting a training plus evaluation cycle to
estimate the performance of a forecasting model using a collection of time
series;

  2. 2.

Feature extraction: Using the training data from step 1., summarize each time
series into a set of features;

  3. 3.

Data augmentation: Create synthetic time series samples to enrich the meta-
dataset concerning large errors;

  4. 4.

Meta-learning: Build a metamodel that predicts the performance of a
forecasting model based on time series features.

####  3.1.1 Performance estimation.

We conduct a procedure to estimate the performance of a forecasting model on
each time series in the available collection $\mathcal{Y}$. Let $\mathcal{D}$
denote the dataset built from $\mathcal{Y}$, which is prepared for auto-
regression. We split $\mathcal{D}$ into training and testing sets,
$\mathcal{D}_{\text{train}}$ and $\mathcal{D}_{\text{test}}$, and fit a model
$f$ using $\mathcal{D}_{\text{train}}$. The performance of $f$ is evaluated on
$\mathcal{D}_{\text{test}}$, resulting in error scores according to SMAPE
metric. This leads to a set of performance estimates
$E=\\{e_{1},e_{2},\dots,e_{n}\\}$, where $e_{i}$ is the performance estimate
of $f$ for time series $Y_{i}$.

####  3.1.2 Feature extraction and meta-dataset.

We summarise the training set of each time series $Y\in\mathcal{Y}$ using
feature extraction. Let $g$ denote a feature extraction function that
transforms a time series $Y_{i}$ into a set of features
$Z_{i}=\\{z_{i,1},z_{i,2},\dots,z_{i,m}$}, where $m$ is the number of features
and $z_{i,k}$ is the $k$-th feature of $Y_{i}$. This transformation is written
as $Z_{i}=g(Y_{i})$. There are several approaches in the literature for the
function $g$, including TSFEL [4], tsfeatures [28], or catch22 [22]. These
involve several statistical operations, such as the mean or standard deviation
of the input time series. While the methodology is applicable to any of these
approaches, we focus on tsfeatures [28], as we mentioned in Section 2.3.

The results from the performance estimation, $E$, and feature extraction $Z$
are concatenated into a meta-dataset
$\mathcal{D}^{\prime}=\\{<Z_{i},e_{i}>\\}^{n}_{i=1}$. Each element in
$\mathcal{D}^{\prime}$ includes the features of a given time series, along
with the error score of a given forecasting model $f$ on that data. Note that
the features are extracted using the training data, while the error is
computed on the test data. This process preserves the temporal order of data
and avoids data leakage.

Our goal is to use the meta-dataset to develop a metamodel $f^{\prime}$ that
classifies where the forecasting model $f$ will incur a large error in a given
time series. We define a binary variable for large errors based on the error
scores $E$, which is determined as follows:

$b_{i}=\begin{cases}1&\text{if }e_{i}>\tau,\\\ 0&\text{otherwise}.\end{cases}$
(1)

For a given time series, $b_{i}$ takes the value of 1 if $e_{i}$ is above
$\tau$, which represents the threshold above which an error is considered
large. The threshold $\tau$ can be defined by domain expertise or in a data-
driven way, e.g. using percentiles. In effect, the meta-dataset
$\mathcal{D}^{\prime}=\\{<Z_{i},b_{i}>\\}^{n}_{i=1}$ involves a binary
classification task. The goal is to model upcoming large errors ($b$) based on
time series features ($Z$) extracted from historical data.

####  3.1.3 Data augmentation.

We work under the assumption that large errors are rare. Thus, the
distribution of $b$ is imbalanced, with $b=1$ being the minority class. The
meta-class imbalance poses an additional challenge for learning an adequate
model for large errors. We tackle this issue by framing the task as an
imbalanced domain learning problem [27], and use resampling approaches for
data augmentation.

We employ an oversampling algorithm (e.g. SMOTE [10]) to create new
($Z_{i,synthetic}$, $b_{i,synthetic}$) samples concerning large errors
($b=1$). The idea is to simulate conditions that lead to large errors based on
the existing data.

####  3.1.4 Meta-learning

The created synthetic samples are represented in a dataset
$\mathcal{D}^{\prime}_{synthetic}$, which is concatenated with the original
dataset $\mathcal{D}^{\prime}$ to augment it. Subsequently, a metamodel
$f^{\prime}$ is trained based on the combined meta-dataset
$\mathcal{D}^{\prime}\cup\mathcal{D}^{\prime}_{synthetic}$. The output of the
metamodel $\hat{b}_{i}=f^{\prime}(Z_{i})$ provides information about the
probability that the forecasting model $f$ will incur a large error on
$Y_{i}$.

###  3.2 Inference stage

In the inference stage, we use the metamodel $f^{\prime}$ to predict the
probability that $f$ will incur a large error for a query time series
$Y_{query}$. We conduct this process as follows:

  1. 1.

Feature extraction: Transform $Y_{query}$ into a set of features $Z_{query}$

  2. 2.

Compute the probability of a large error using the metamodel:
$\hat{b}=f^{\prime}(Z_{query})$

The information about the probability of a large error can be used in various
ways, for example deferring the forecasts by not accepting the predictions
made by $f$. Another possibility is to use the metamodel output for a
descriptive analysis by characterizing the feature space in conditions with
high probability of a large error.

##  4 Experiments

We carried out extensive experiments to evaluate the proposed method. The
experiments focus on the following research questions:

  * •

RQ1: How does the base forecasting model compare with benchmark approaches?

  * •

RQ2: Is the metamodel able to predict large errors by a given forecasting
model?

  * •

RQ3: What is the impact of data augmentation using resampling in the
performance of the metamodel?

  * •

RQ4: How does the performance of the metamodel vary with different values of
$\tau$?

Besides these, we also present an example of how the results of the metamodel
can be used to understand the conditions where a forecasting model is expected
to perform poorly.

###  4.1 Data

We use the following datasets in the experiments: M3 Monthly [24], M4 Monthly
[25], and Tourism Monthly [2]. These datasets were part of different
forecasting competitions and represent long-standing univariate time series
forecasting benchmarks. In the interest of conciseness, we focus on time
series with a monthly sampling frequency. Table 1 provides a brief summary of
the data.

Table 1: Description of the datasets, including number of time series, number of observations, and average value length. | # Time series | # Observations | Avg. length
---|---|---|---
M3 Monthly | 1428 | 167562 | 117
M4 Monthly | 48000 | 11246411 | 234
Tourism Monthly | 366 | 109280 | 299

Both the forecasting horizon ($h$) and the number of lags ($p$) are set to 12,
which represents one seasonal cycle (one year) of data.

###  4.2 Experimental Design

####  4.2.1 Performance estimation.

The data partitioning process is illustrated in Figure 3. We split the
available data into training and testing sets for performance estimation. The
test set is composed of the last $h$ observations of each time series in
$\mathcal{Y}$. The training set is further split into a development set and a
validation set, where the latter is composed of the last $h$ observations of
each time series before the test set.

Figure 3: Data preparation step, where for each time series, a split equal to
the forecasting horizon (h) is done at the end of the data, to define the test
set, assigning the rest to the train set. Another split with the same size is
done at the end of the latter, defining the validation set, assigning the
remaining to the development set.

We carry out the methodology for building the metamodel using the development
set and the validation set (c.f. Figure 2). Using the training set, we re-
train the forecasting model and carry out the inference stage described in
Section 3.2. Finally, the performance of the forecasting model and the
metamodel are evaluated using the test set. For each time series, we evaluate
the forecasting performance of $f$. Using those results, we evaluate the
ability of the metamodel $f^{\prime}$ to predict large errors. We evaluate
forecasting performance using SMAPE, a benchmark metric [25]. Concerning the
metamodel, we resort to AUC (area under the ROC curve) to evaluate its
performance.

####  4.2.2 Forecasting model.

The forecasting model is built using a lightgbm (LGBM) regression algorithm.
This approach has been shown to provide state-of-the-art forecasting
performance. For instance, it is the algorithm used in the winning solution of
the M5 forecasting competition [26]. Following Bandara et al. [3], we train a
global forecasting model for each dataset listed in Table 4.1 with the
approach described in Section 2.1. We resorted to the mlforecast
implementation of lightgbm called AutoLightGBM. This methods automatically
conducts hyperparameter tuning using the optuna framework [1], which uses
Bayesian optimization. Specifically, the following parameters are optimized:
learning rate, number of leaves, max depth, and number of estimators. We also
include the seasonal naive method in the experiments as a benchmark. This
approach forecasts the upcoming values of a time series using the last known
observation of the same period.

The metamodel is also trained using a lightgbm algorithm. We use an objective
function tailored for binary classification and optimize the hyperparameters
via grid search. Moreover, we use the area under the ROC curve (AUC) as the
evaluation metric for the metamodel. While AutoLightGBM provides an off-the-
shelf approach for building optimized models based on lightgbm, it is only
available for forecasting. Thus, for the binary metamodel classifier, we
optimize lightgbm using grid search.

####  4.2.3 Parameter setting.

We define large errors in a data-driven way using percentiles. The threshold
$\tau$ is set to the percentile 90 of the SMAPE performance estimates obtained
in the validation set. We work under the assumption that the forecasting model
typically performs arbitrarily well, except for rare instances. Nonetheless,
we carry out a sensitivity analysis to analyse the effect of the threshold on
the performance of the metamodel.

As mentioned before, we leverage the tsfeatures framework for time series
feature extraction [28]. Concerning data augmentation, we test two popular
oversampling algorithms: SMOTE [10], and ADASYN [16].

###  4.3 Results

Figure 1 shows the distribution of SMAPE scores by the forecasting model
described in Section 4.2.2 across all the time series in the M3 dataset. We
start by validating the performance of the forecasting model by comparing it
with a seasonal naive baseline. While we aim at identifying the weaknesses of
a forecasting model, ensuring that it outperforms a baseline is important for
its applicability. The results are reported in Table 2, which show the SMAPE
scores of each approach in each dataset. The developed model outperforms
seasonal naive in all cases, which validates its forecasting performance
(RQ1).

Table 2: Forecasting performance of each method using SMAPE. Bold font denotes the best approach. | Seasonal Naive | LGBM
---|---|---
M3 Monthly | 7.98% | 7.28%
M4 Monthly | 7.17% | 6.58%
Tourism Monthly | 9.15% | 8.50%
Table 3: metamodel performance in each dataset according to AUC. Bold font denotes the best method in each dataset. | ADASYN | SMOTE | No Augmentation
---|---|---|---
M3 Monthly | 0.728 | 0.678 | 0.516
M4 Monthly | 0.713 | 0.709 | 0.566
Tourism Monthly | 0.787 | 0.761 | 0.730

The results concerning the metamodel are reported in Table 3. The metamodel is
able to predict large errors with a reasonable performance (RQ2). Overall, it
performs better when coupled with data augmentation using ADASYN. When no data
augmentation is applied, the proposed method struggles in two of the datasets
(M3 and M4), which emphasizes the importance of the synthetic samples (RQ3).

We conducted a sensitivity analysis to assess the impact of the threshold for
large errors ($\tau$) in the results (RQ4). The results are shown in Figure 4,
which illustrates the AUC for increasing values of $\tau$ for each dataset and
approach. Overall, increasing the value of $\tau$ leads to worse performance.
In all three datasets, the metamodel trained on augmented metadata shows
better performance than a variant without data augmentation.

Figure 4: Analysis of the impact of error percentile threshold in metamodel
performance, measured in AUC. For each time series collection, the performance
for the base and augmented metamodels is compared regarding six levels of
percentile error threshold.

###  4.4 Metamodel explanations

The results presented in the previous section provide evidence that MAST is
able to detect large errors. We can leverage Shapley values [23] to explore
how the time series features affect the output of the metamodel. For
conciseness, this analysis is focused on the M3 dataset and on the five most
important features on this task, according to the metamodel. These are the
following (most important first): linearity, lumpiness, hw_beta, spike, and
hw_gamma. Again, we refer to the work by Montero-Manso et al. [28] for a
description of these222Also available here:
https://htmlpreview.github.io/?https://github.com/robjhyndman/M4metalearning/blob/master/docs/M4_methodology.html#features.

Figure 5 shows a summary plot of the Shapley values for the 5 most relevant
features for the M3 dataset. This plot shows interesting insights about the
performance of the forecasting model based on the predictions of the
metamodel. For example, high linearity values reduces the probability of
classifying a time series as large error inducing. This suggests that the
forecasting model may struggle in time series with a low linearity value.

Figure 5: Shapley values for the 5 most relevant features for the M3 dataset’s
metamodel, indicating which ones contribute positively and negatively to its
output. Values to the right of the vertical line indicate a positive
influence, while the ones at the left indicate a negative one. The colour
indicates the magnitude of the features’ value.

##  5 Discussion

This work addresses the problem of evaluating the robustness of univariate
time series forecasting models to time series with varying characteristics. We
use meta-learning and data augmentation to model the relationship between
large errors and the structural properties of time series. The proposed
approach was validated using benchmark datasets composed of monthly time
series. The results indicate that our approach is able to identify conditions
that lead to large errors. The conditions identified by the metamodel can be
explored using techniques such as Shapley values.

Overall, the experiments suggest that using meta-learning to model large
errors is a promising approach for improving the transparency and reliability
of forecasting models. We plan to extend the proposed method in future work,
including the following aspects:

  * •

modelling stress in the form of high uncertainty, as opposed (or a complement)
to stress in terms of large errors;

  * •

Model the stress of algorithms instead of trained models to understand which
methods should be used for a given time series dataset;

  * •

Test different data augmentation approaches, such as moving blocks
bootstrapping [3].

  * •

Apply the developed approach in a wider range of time series, including
different sampling frequencies.

##  6 Final Remarks

modelling the stress of forecasting models, such as large errors, is important
to improve their transparency and reliability. This paper introduces a novel
framework for modelling and describing the stress in univariate time series
forecasting models based on meta-learning and data augmentation.

We validated our approach using benchmark datasets that contain multiple
univariate time series. The developed metamodel is able to relate large errors
with structural properties of time series, such as linearity or spike. In
future work, we plan to extend our approach for different time series and
stress scenarios. The code and data is publicly available333c.f. footnote 4.

## References

  * [1] Akiba, T., Sano, S., Yanase, T., Ohta, T., Koyama, M.: Optuna: A next-generation hyperparameter optimization framework. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2019)
  * [2] Athanasopoulos, G., Hyndman, R.J., Song, H., Wu, D.C.: The tourism forecasting competition. International Journal of Forecasting 27(3), 822–844 (2011), special Section 1: Forecasting with Artificial Neural Networks and Computational Intelligence Special Section 2: Tourism Forecasting
  * [3] Bandara, K., Hewamalage, H., Liu, Y.H., Kang, Y., Bergmeir, C.: Improving the accuracy of global forecasting models using time series data augmentation. Pattern Recognition 120, 108148 (2021)
  * [4] Barandas, M., Folgado, D., Fernandes, L., Santos, S., Abreu, M., Bota, P., Liu, H., Schultz, T., Gamboa, H.: Tsfel: Time series feature extraction library. SoftwareX 11, 100456 (2020)
  * [5] Bontempi, G., Ben Taieb, S., Le Borgne, Y.A.: Machine learning strategies for time series forecasting. Business Intelligence: Second European Summer School, eBISS 2012, Brussels, Belgium, July 15-21, 2012, Tutorial Lectures 2 pp. 62–77 (2013)
  * [6] Brazdil, P., van Rijn, J.N., Soares, C., Vanschoren, J.: Metalearning approaches for algorithm selection ii. In: Metalearning: Applications to Automated Machine Learning and Data Mining, pp. 77–102. Springer (2022)
  * [7] Cerqueira, V., Moniz, N., Inácio, R., Soares, C.: Time series data augmentation as an imbalanced learning problem. arXiv preprint arXiv:2404.18537 (2024)
  * [8] Cerqueira, V., Santos, M., Baghoussi, Y., Soares, C.: On-the-fly data augmentation for forecasting with deep learning. arXiv preprint arXiv:2404.16918 (2024)
  * [9] Cerqueira, V., Torgo, L., Pinto, F., Soares, C.: Arbitrage of forecasting experts. Machine Learning 108, 913–944 (2019)
  * [10] Chawla, N.V., Bowyer, K.W., Hall, L.O., Kegelmeyer, W.P.: Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research 16, 321–357 (2002)
  * [11] Christ, M., Kempa-Liehr, A.W., Feindt, M.: Distributed and parallel time series feature extraction for industrial big data applications. arXiv preprint arXiv:1610.07717 (2016)
  * [12] Cleveland, R.B., Cleveland, W.S., McRae, J.E., Terpenning, I., et al.: Stl: A seasonal-trend decomposition. J. Off. Stat 6(1), 3–73 (1990)
  * [13] Cunha, L., Soares, C., Restivo, A., Teixeira, L.F.: Gasten: Generative adversarial stress test networks. In: Crémilleux, B., Hess, S., Nijssen, S. (eds.) Advances in Intelligent Data Analysis XXI. pp. 91–102. Springer Nature Switzerland, Cham (2023)
  * [14] Freiesleben, T., Grote, T.: Beyond generalization: a theory of robustness in machine learning. Synthese 202(4), 109 (2023)
  * [15] Godahewa, R., Bandara, K., Webb, G.I., Smyl, S., Bergmeir, C.: Ensembles of localised models for time series forecasting. Knowledge-Based Systems 233, 107518 (2021)
  * [16] He, H., Bai, Y., Garcia, E.A., Li, S.: Adasyn: Adaptive synthetic sampling approach for imbalanced learning. In: 2008 IEEE international joint conference on neural networks (IEEE world congress on computational intelligence). pp. 1322–1328. IEEE (2008)
  * [17] Hyndman, R.J., Athanasopoulos, G.: Forecasting: principles and practice. OTexts (2018)
  * [18] Jain, C.L.: Answers to your forecasting questions. The Journal of Business Forecasting 36(1), 3 (2017)
  * [19] Januschowski, T., Gasthaus, J., Wang, Y., Salinas, D., Flunkert, V., Bohlke-Schneider, M., Callot, L.: Criteria for classifying forecasting methods. International Journal of Forecasting 36(1), 167–177 (2020)
  * [20] Kang, Y., Hyndman, R.J., Li, F.: Gratis: Generating time series with diverse and controllable characteristics. Statistical Analysis and Data Mining: The ASA Data Science Journal 13(4), 354–376 (2020)
  * [21] Lemke, C., Gabrys, B.: Meta-learning for time series forecasting and forecast combination. Neurocomputing 73(10-12), 2006–2016 (2010)
  * [22] Lubba, C.H., Sethi, S.S., Knaute, P., Schultz, S.R., Fulcher, B.D., Jones, N.S.: catch22: Canonical time-series characteristics. Data Mining and Knowledge Discovery 33(6), 1821–1852 (2019)
  * [23] Lundberg, S.M., Lee, S.I.: A unified approach to interpreting model predictions. Advances in neural information processing systems 30 (2017)
  * [24] Makridakis, S., Hibon, M.: The m3-competition: results, conclusions and implications. International Journal of Forecasting 16(4), 451–476 (2000), the M3- Competition
  * [25] Makridakis, S., Spiliotis, E., Assimakopoulos, V.: The m4 competition: 100,000 time series and 61 forecasting methods. International Journal of Forecasting 36(1), 54–74 (2020)
  * [26] Makridakis, S., Spiliotis, E., Assimakopoulos, V.: M5 accuracy competition: Results, findings, and conclusions. International Journal of Forecasting 38(4), 1346–1364 (2022)
  * [27] Moniz, N., Cerqueira, V.: Automated imbalanced classification via meta-learning. Expert Systems with Applications 178, 115011 (2021)
  * [28] Montero-Manso, P., Athanasopoulos, G., Hyndman, R.J., Talagala, T.S.: Fforma: Feature-based forecast model averaging. International Journal of Forecasting 36(1), 86–92 (2020)
  * [29] Prudêncio, R.B., Ludermir, T.B.: Meta-learning approaches to selecting time series models. Neurocomputing 61, 121–137 (2004)
  * [30] Rožanec, J., Trajkova, E., Kenda, K., Fortuna, B., Mladenić, D.: Explaining bad forecasts in global time series models. Applied Sciences 11(19) (2021)
  * [31] Wen, Q., Sun, L., Yang, F., Song, X., Gao, J., Wang, X., Xu, H.: Time series data augmentation for deep learning: A survey. arXiv preprint arXiv:2002.12478 (2020)