# A Quantitative Exploration of Natural Language Processing Applications for
Electricity Demand Analysis

Yun Bai, Simon Camal, Andrea Michiorri The authors are with the Centre for
Processes, Renewable Energies and Energy Systems (PERSEE), MINES Paris - PSL
University, Sophia Antipolis, France, e-mail: (yun.bai@minesparis.psl.eu).

###### Abstract

The relationship between electricity demand and weather has been established
for a long time and is one of the cornerstones in load prediction for
operation and planning, along with behavioral and social aspects such as
calendars or significant events. This paper explores how and why the social
information contained in the news can be used better to understand aggregate
population behaviour in terms of energy demand. The work is done through
experiments analysing the impact of predicting features extracted from
national news on day-ahead electric demand prediction. The results are
compared to a benchmark model trained exclusively on the calendar and
meteorological information. Experimental results showed that the best-
performing model reduced the official standard errors around 4%, 11%, and 10%
in terms of RMSE, MAE, and SMAPE. The best-performing methods are: word
frequency identified COVID-19-related keywords; topic distribution that
identified news on the pandemic and internal politics; global word embeddings
that identified news about international conflicts. This study brings a new
perspective to traditional electricity demand analysis and confirms the
feasibility of improving its predictions with unstructured information
contained in texts, with potential consequences in sociology and economics.

###### Keywords:

Electricity demand forecasting; Natural language processing; Population
behaviour; Social events

##  I Introduction

###  I-A Context

Known dependencies characterise electricity demand to economic activity (e.g.,
working or non-working days) and weather (e.g., low or high temperatures). The
impact of significant social events, such as major sports competitions, has
also been identified along with the recent COVID-19 pandemic. This work aims
to analyze unstructured information present in news’ textual content to find
relationships between social events and electricity demand using the
techniques of Natural Language Processing (NLP) and numerical predictions.

###  I-B State of the art

The field of text-based forecasting is relatively new and is being explored
deeply by researchers, with a visible acceleration after 2010 and a general
interest in price predictions. It is possible to identify three influential
milestones relevant to this research topic.

Firstly, an earlier expression of the idea could be traced to [1] where the
fundamental concepts of text-based forecasting were suggested, and tests were
carried out for movie revenues forecasting with the n-grams, part-of-speech
n-grams, and the dependency relations from online movie reviews. The study
showed an improvement in prediction performance and considered the mechanisms
behind the improvement among future developments, aiming to understand why
specific words are linked with the predicted variable.

Secondly, it is worth mentioning that the authors of [2] pointed out a series
of important conclusions: i) as in forecasting more generally, complex models
or methods were not necessarily more successful than simpler ones; ii) fully
automatic methods benefitted from replicability, speed, and ease of updating
but with the downside of missing more subtle shades of meaning.

Finally, in [3], textual data were used for improving electricity demand
prediction using weather reports and tweets, specifically in the context of
the sudden demand changes caused by COVID-19 lockdowns in France and Italy.
This study revealed that calendar and meteorological information extracted
from the text was a beneficial supplement in the absence of these data
sources. The paper also showed that the sudden relevance of words related to
‘remote working’ in the public discourse was strongly correlated to changes in
electricity demand.

Independently from these three milestones, the text-based forecasting approach
has been tested in several fields such as bankruptcy and fraud [4], stock
prices [5], psychological disturbances in college students [6], demand for
taxi rides [7], short term apartment rentals [8], street crime [9], COVID-19
evolution [10], tourism demand from online searches [11], and crude oil prices
[12].

The works above show that in terms of NLP methods, sentiment analysis and
topic modeling are well-established for extracting textual information, and
word embedding, also word vectorization, acts as an upstream task for NLP
applications such as text classification. The main techniques applied are
summarised following.

Sentiment analysis covers different aspects as introduced in [13]. Polarity
analysis portrays the sentiment tendency within a sentence. For supervised
methods, sentences are labeled as either negative-positive binary or negative-
neutral-positive ternary categories. Subjectivity analysis quantifies the
amount of personal opinion carried in a sentence. The higher proportion of
personal opinion in a sentence, the more subjective it is. Emotion analysis is
commonly used in social media analysis, such as Twitter and Weibo, to label
sentences with multiple human emotions such as happy, angry, sad, disgusted,
and scared [14, 15]. Besides that, researchers also apply advanced neural
network models and word embeddings to complete the model classification task
[16, 17, 18].

Topic analysis. Topics are abstract expressions of text contents and topic
analysis is a technique to discover the hidden semantic structures from the
text collections. [19]. Essentially, topic analysis is a dimensionality
reduction method. In the NLP field, bag-of-words provides vector
representations of text statistically, but it suffers from sparsity. Combined
with Principal Component Analysis (PCA), [20] developed Latent Semantic
Analysis (LSA), which represents documents as denser vectors and yields
principal components, or topics, that express deeper semantic content.
Subsequently, [21] built on this foundation and studied the Latent Dirichlet
Allocation (LDA), which is currently a popular topic model, and we will
describe it in Section II-A3.

Word vectorization is a technique aiming at mapping words to vectors in the
same space. Word vectors are closer in distance when they are similar in
lexical meaning. Earlier word vectorization methods were frequency-based, such
as one-hot encoding, index encoding, and Term Frequency-Inverse Document
Frequency (TF-IDF) [22]. Among the recent developments, Global Vectors (GloVe)
and Word2Vec models are based on the local terms co-occurrence [23, 24]. Also,
other advanced models would pay more attention to the words for prediction,
thus resolving sentence ambiguities and inferring word meaning. The attention
mechanism spawned the Transformer method, on which pre-trained language
models, such as BERT, are based [25]. These word embeddings are prediction-
based, obtain a higher-level representation of the text probability
distribution, and reduce the dimensionality of word vectors, yet frequency-
based methods struggle to do so.

###  I-C Hypotheses and objectives

The field of NLP application for forecasting, particularly electricity demand
forecasting, is at its beginning, and this work aims to explore the
possibilities and limitations of the approach. In particular, we consider two
following hypotheses:

  1. 1.

Except for weather and economic activity, electricity demand is also
influenced by social factors visible in the news.

  2. 2.

It is possible to quantify textual information and use it in practice via NLP.

The objectives of this paper are summarised as follows:

  1. 1.

To verify _IF_ it is possible to extract valuable information from news to
improve electricity demand forecasting.

  2. 2.

To explore _HOW_ to best treat textual information and develop a complete
forecasting chain that integrates text and other structured data.

  3. 3.

To understand _WHY_ we have improved performance and uncover the mechanisms of
this approach.

###  I-D Structure of the paper

In this paper, after Section I where the problem is introduced with the
context, state of the art, and a clarification of the contributions to
knowledge, the methodology is presented in Section II with an overview of the
NLP and forecasting techniques. Evaluation metrics and model explanations
techniques are employed along with the Case study Section II-E. Results are
shown in Section III followed by a discussion in Section IV, and conclusions
are drawn in Section V.

##  II Methodology

An overview of the workflow for this study is described and visually shown in
Figure 1. Firstly, electricity load, meteorology and economic activity, and
news data are acquired in modules A, B, and C. Then numerical and textual data
are pre-processed in modules D and E. In the case of numeric data, time series
are cleaned and synchronised for i) the target variable (aggregated
electricity demand), ii) ambient temperature, and iii) calendar features
(holidays and weekends). To build the benchmark model in module F, we used the
features from D, including lags, calendars, and temperatures. The first group
of calendar features is represented by the day of the week and day of the
year, embedded through their sine and cosine, reflecting the multiple
seasonality in electric demand. Additional variables account for weekends and
holidays.

Then, textual inputs are pre-processed to create different numerical input
features. Subsection II-A shows the detailed textual features. It should be
noted that some textual features are redundant for forecasting, and they are
filtered out by the Granger test as suggested in [26], which happens in module
E. The text-based forecasting model is built in module G with inputs from E
and D. Finally, the results of the two models are evaluated in H with an
analysis of errors and the explanation from the global, local, and causality
aspects.

Figure 1: The forecasting framework of this research.

###  II-A Preprocessing for the text-based model

This section describes the methods used in Module E in 1 to extract numerical
features to be fed to the following machine learning prediction model from the
raw textual data. For textual data, the pre-processing includes splitting a
sentence into words; converting all letters to lowercase; removing stopwords,
words with less than three letters, and all numbers. All news is finally
transformed into word lists.

This study borrowed three NLP techniques: sentiment and subjectivity analysis,
topic modeling, and text vectorisation, to capture sentiment scores, topic
distributions, and word embeddings in the news.

####  II-A1 Simple statistics

Count features include 27 features counted daily. For each text, the number of
words, sentences, unique words, non-stopping words, the average number of
sentences, and the average number of words for all the sentences each day are
calculated. Finally, we made two categorical features for the proportion of
news in the 18 sections on the BBC website (e.g., Asia, Business, UK Politics,
…).

Word frequencies analysis consists of the words in text after stop-words and
non-words have been removed. To reduce the number of words, only the most
relevant has been selected. Considering the different volumes of titles (_T_),
descriptions (_D_), and text bodies (_B_), we set separate thresholds
$\sigma_{T}=200$, $\sigma_{D}=400$, $\sigma_{B}=5000$, and only words that
exceeded this threshold were included. This resulted in the selection of
respectively 456, 329, and 550 words for _T_ , _D_ , and _B_.

####  II-A2 Sentiment and subjectivity analysis

This analysis is performed with the library TextBlob from Python’s Natural
Language ToolKit (NLTK) [27], widely used in sentiment analysis and is
particularly suitable for corpora without manual labeling [28, 29, 30]. This
algorithm calculates a score between [-1, 1] for each word according to its
negative or positive meaning. In addition, it calculates a score between [0,
1] according to its subjectivity, considering the modifying effect of
adjectives and adverbs. In this study, a distribution of sentiment and
subjectivity is calculated for each piece of news. This distribution is then
discretised in five quintiles $[0,0.2)$, $[0.2,0.4)$, $[0.4,0.6)$,
$[0.6,0.8)$, $[0.8,1]$. They also computed the maximum, minimum, average, and
standard deviation for 18 sentiment features.

####  II-A3 Topic distribution

Topic distribution is analysed through LDA model, with the objective to obtain
a probabilistic estimation of the belonging of each news article to a specific
topic. LDA is a classical unsupervised topic model that mimics human writing
by assuming a text-generation process. The LDA model assists in extracting
several topics from mass texts and gives the probability distribution of each
text under these topics. Based on this, the average topic probability
distribution for all daily news is easily obtained. The probability value
under each topic reflects how widespread the daily news is on that topic and
is a noteworthy feature. According to the topic number selection method in
[31], we set the ideal number of topics as $\kappa$ and obtained
$\kappa_{T}=87$, $\kappa_{D}=100$, $\kappa_{B}=69$ for each text type.

####  II-A4 Text vectorisation

This step is to obtain a vectorial representation of the text according to its
content for the other texts in the corpus through the library GloVe described
in [23]. Thanks to the pre-trained word embeddings based on a large corpus,
the current GloVe not only contains rich global and local semantic information
but also facilitates our application on a new corpus without repeating the
time-consuming training work.

High-dimensional textual features are extracted and expressed in this work
from the textual dataset by transforming words into 100-dimensional vectors
with GloVe. A text vector is obtained by averaging all the word vectors in
this text. The position of each element in the text vector is an axis in the
high-dimensional space. We averaged all the text vectors within a day to get
the features.

####  II-A5 Granger test

The method mentioned above can produce numerous features, totaling 2026 in the
configuration. However, it is necessary to prevent some of the features from
positively affecting the quality of the forecast. Therefore the Granger test
is done preventively before the training of the prediction model in module E.

The Granger test is a measure to test whether a stationary time series _X_
contributes to the forecasting of parameter _Y_ [32]. It is based on the
following autoregressive model:

$\displaystyle y_{t}=\theta_{0}+\sum_{i=1}^{T}\theta_{i}y_{t-i}$
$\displaystyle+\sum_{i=1}^{T}\phi_{i}x_{t-i}+\epsilon_{t}$ (1) $\displaystyle
E(\epsilon)$ $\displaystyle=0,$

where $\theta_{i}$ and $\phi_{i}$ are the lag coefficients of _X_ and _Y_ ,
and _T_ is a chosen lag order, which in this paper is 30. The null hypothesis
is

$H_{0}:\phi_{1}=\phi_{2}=...=\phi_{T}=0,$ (2)

i.e. that the lagged terms of _X_ are independent of _Y_. This can be rejected
when $p<0.05$.

As shown in [26, 33], the Granger test assists in the selection of text
features that are relevant to the forecasting target. It acts as a
dimensionality reduction for the text features. In these cases, the Granger
test is usually performed unilaterally, but we do not want _Y_ Granger cause
_X_. Therefore, the experiments in this paper include a bilateral Granger
test, i.e., requiring $p_{2}<0.05$ for the first test and $p_{1}\geq 0.05$ for
the second.

Note that the Granger test used for initial feature filtering is not an actual
causality test, as it involves correlations between lagged and predicted
values, and correlations do not necessarily lead to causality. We will discuss
the causality of text features and electricity load in more detail in Section
IV-C.

###  II-B Forecasting with textual features

Due to the nature of the dataset, the news was available per day, and it was
not possible to distinguish an intra-day order. Furthermore, the news may
refer to events of the previous days. It was then decided to frame the problem
as a forecasting problem for the day ahead. This section describes the
forecasting algorithm used in Block F and G. The idea is to verify if textual-
based features measured in day _d-1_ can provide additional explanation to the
behaviour of the demand in day _d+1_ , considering a prediction calculated in
day _d+1_ in a valuable time to take decisions (for example for trading or
scheduling).

After an initial comparison with different models, such as Support Vector
Regression and Multilayer Perceptron, the ExtraTrees algorithm has been
selected because of its higher performance (known in the case of relatively
small tabular datasets) and flexibility. ExtraTrees is an ensemble learning
method within the decision tree paradigm. Like Random Forest, ExtraTrees
creates many decision trees during the training but randomly samples each
tree. The features in the trees are also randomly selected by splitting values
without using the criterion of optimizing localization, which enables
ExtraTrees to achieve faster computational speed and membership diversity
[34]. Current applications of ExtraTrees have emerged in the field of
electricity load forecasting, for example, the peak load forecasting [35],
day-ahead load demand forecasting case from Spain [36], and medium- and long-
term load forecasting [37].

###  II-C Evaluation

The two regression models in blocks F and G are evaluated according to the
following criteria. Firstly, deterministic evaluation metrics are calculated
to compare the performance of the algorithm and the baseline quantitatively.
Then an analysis is carried out to explain the relationship between the
features and demand.

Root Mean Square Error (RMSE), Mean Absolute Error (MAE), and Symmetric Mean
Absolute Percentage Error (SMAPE) are used in this study, and they are
calculated as follows:

$RMSE=\sqrt{\frac{1}{H}\sum_{i=1}^{H}(y_{i}-\hat{y_{i}})^{2}},$ (3)

$MAE=\frac{1}{H}\sum_{i=1}^{H}\|y_{i}-\hat{y_{i}}\|,$ (4)

$SMAPE=\frac{100\%}{H}\sum_{i=1}^{H}\frac{\|y_{i}-\hat{y_{i}}\|}{\|y_{i}+\hat{y_{i}}\|/2},$
(5)

where _H_ is the forecasting horizons and $H=48$ for the half-hourly data in
our case. $y_{i}$ and $\hat{y_{i}}$ are truth and forecased loads at time _i_.

These metrics are calculated for each time step of the test dataset, but they
are then averaged over a whole yearly period and noted as
$\overline{rmse}$,$\overline{mae}$, and $\overline{smape}$.

###  II-D Models explaination

Previous research has confirmed that well-selected text features enhance
forecasting, yet it is often difficult to explain this improvement deeply.
Explainability is necessary to shed light on the behaviour of the trained
machine learning models, which otherwise would be completely black boxes. This
paper attempts to explore the mechanisms by which text features enhance
forecasting in terms of _i)_ global, _ii)_ local, and _iii)_ causality.

Global explainability is analysed through Pearson correlation coefficients. It
will not be discussed here due to its popularity. Local explainability is
analised through the Local Interpretable Model-agnostic Explanation (LIME),
whilst casuality is analysed through Double Machine Learning (Double ML). Both
methods are described in the following lines.

####  II-D1 Local explainability

LIME targets a sample of the original data and generates a new, normally
distributed local dataset using the current sample [38, 39]. After that, a
simple surrogate model, such as linear regression, is used to fit the new
dataset, yielding a locally interpretable perspective on the data under
perturbation. We can interpret how features in the current local affect the
forecasting by viewing the coefficients of the features in the linear model.

####  II-D2 Causation

Previous analysis, such as the Pearson correlation or the LIME model, verified
the correlation between the candidate features and electric demand.
Nevertheless, positive or negative correlations between variables may be the
result of coincidence, and it is known that correlation does not imply
causation [40]. For this reason, a test is proposed to verify the causality
between the features and the target. The so-called causality is the effect of
a particular feature of interest (Treatment, _T_) on the predictions
(Outcomes, _Y_), provided that the rest of the features remain constant
(Confounders, _X_). The Double ML method helps find the causality between
variables and predictions based on the following partially linear model [41]:

$Y=f(X)+\rho(X)T+\epsilon\quad E(\epsilon|X,T)=0,$ (6)

where $f(X)$ a train model and $\rho(X)$ the treatment effect. Although
$\rho(X)$ is regarded as a function by some studies [42], we treat it as a
constant $\rho(X)=\tau$ for simplicity. The next step is to explore the effect
of the text feature _T_ on the output _Y_ while maintaining the rest of the
features _X_ constant. In this way, it is possible to observe whether, or to
what extent, _T_ causes a change in _Y_. To estimate $\rho(X)$, the formula 6
is rewritten in residualised form:

$Y-\hat{Y}=\tau(T-\hat{T})+\delta\quad E(\delta|X,T)=0,$ (7)

in which $\hat{Y}=g(X)$ and $\hat{T}=h(X)$ are forecasts of _Y_ and _T_.
$g(X)$ and $h(X)$ are nuisance functions that can be replaced by many machine
learning methods. In this case, we set $f(X)$, $g(X)$, and $h(X)$ for all
ExtraTrees regressors. The treatment effect $\tau$ can then be obtained with
Ordinary Least Squares (OLS).

###  II-E Datasets

This work uses three datasets covering five years between June 2016 and May
2021. The first four years are used as a training set and the last year as a
test set. Aggregated electric demand for the UK is obtained from the ENTSOe
transparency platform [43] along with the official day-ahead forecasts.
Historical Bank holidays and daily temperatures for the city of London have
been taken from commercial websites. According to [44], we used the observed
rather than predicted temperatures for the convenience of reproducibility.

Previous studies have mainly used keywords for external texts to filter news
related to the forecasting domain [33, 45, 12]. However, this paper proposes
to use the entire volume of news from British Broadcasting Corporation (BBC)
to explore the impact of broader social events on electricity load
forecasting. Over 80,000 news items were collected, thanks to the repository
[46], which archives [47].

##  III Results

###  III-A Benchmark model

A prediction model based on ExtraTrees Regression (ETR) has been trained on
the basic datasets. Grid search and five-fold cross-validation were used to
find the optimal parameters and avoid overfitting. The performance of the
benchmark model is summarised in Table I, in which different combinations of
features are tested: demand $\mathcal{D}$, calendar features $\mathcal{C}$ and
temperature $\mathcal{T}$. These are compared with the official forecasts
obtained from the ENTSOe Transparency Platform $\mathcal{ENTSOe}$.

TABLE I: Benchmark model performance Features |  $\overline{rmse}$(MW) |  $\overline{mae}$(MW) |  $\overline{smape}$(%)
---|---|---|---
$\mathcal{ENTSOe}$ | 2800.50 | 2544.86 | 7.65
$\mathcal{D}$ | 2983.05±5.35 | 2539.25±5.14 | 7.75±0.01
$\mathcal{D}+\mathcal{C}$ | 2896.49±4.81 | 2468.48±4.22 | 7.56±0.01
$\mathcal{D}+\mathcal{T}$ | 2938.40±2.81 | 2488.93±2.61 | 7.59±0.01
$\mathcal{D}+\mathcal{C}+\mathcal{T}$ | 2800.77±4.84 | 2374.07±4.39 | 7.29±0.01

As expected, combining these inputs produces the best performance with
moderate gains concerning the official forecasts over the three deterministic
metrics. However, holidays do not provide improvements. The best improvements
range in the region of 200MW in $\overline{mae}$ and 4% in $\overline{smape}$.

###  III-B Textual features enhanced model

####  III-B1 Impact of textual features

In this subsection, a new model is trained with all the features used in the
benchmark model ($\mathcal{D}$, $\mathcal{C}$ and $\mathcal{T}$), and the
features extracted from the textual dataset, here divided in terms of Title
(T), Description (D) and Body (B). The textual features $\mathcal{F}_{t}$
explored in the step are: Count Features ($\mathcal{CF}$), Words Frequencies
($\mathcal{WF}$), Sentiment ($\mathcal{SE}$), Topic Distributions
($\mathcal{TD}$), and GloVe Word Embeddings ($\mathcal{GWE}$) from _T_ , _D_ ,
and _B_. The experimental results are presented in Table II, along with the
original and selected feature numbers after the Granger test in each group.

TABLE II: Forecasting errors when adding $\mathcal{F}_{t}$ into model
$\mathcal{D+C+T}$

Text Type | $\mathcal{F}_{t}$ | #Original $\mathcal{F}_{t}$ | #Selected $\mathcal{F}_{t}$ |  $\overline{rmse}$(MW) |  $\overline{mae}$(MW) |  $\overline{smape}$(%)
---|---|---|---|---|---|---
$\mathcal{D+C+T}$ | —- | 0 | 0 | 2800.77±4.84 | 2374.07±4.39 | 7.29±0.01
_T_ | $\mathcal{CF}$ | 27 | 1 | 2799.65±3.73 | 2372.63±3.38 | 7.29±0.01
$\mathcal{WF}$ | 456 | 32 | 2702.23±4.95** | 2283.52±4.24** | 6.98±0.01**
$\mathcal{SE}$ | 18 | 2 | 2803.73±6.15 | 2376.81±5.82 | 7.30±0.02
$\mathcal{TD}$ | 87 | 6 | 2806.27±5.35 | 2377.14±5.08 | 7.31±0.01
$\mathcal{GWE}$ | 100 | 5 | 2795.80±4.34 | 2367.49±2.88 | 7.27±0.01
_D_ | $\mathcal{CF}$ | 27 | 0 | 2798.81±4.85 | 2371.95±4.51 | 7.29±0.01
$\mathcal{WF}$ | 329 | 25 | 2760.65±6.85* | 2342.98±6.16* | 7.14±0.02*
$\mathcal{SE}$ | 18 | 0 | 2798.81±4.85 | 2371.95±4.51 | 7.29±0.01
$\mathcal{TD}$ | 100 | 5 | 2805.32±5.46 | 2379.75±4.96 | 7.31±0.01
$\mathcal{GWE}$ | 100 | 2 | 2803.58±5.26 | 2375.26±4.84 | 7.30±0.01
_B_ | $\mathcal{CF}$ | 27 | 0 | 2798.81±4.85 | 2371.95±4.51 | 7.29±0.01
$\mathcal{WF}$ | 550 | 10 | 2751.40±4.38* | 2330.54±3.95* | 7.16±0.01*
$\mathcal{SE}$ | 18 | 1 | 2788.45±4.61** | 2360.75±4.33** | 7.26±0.01**
$\mathcal{TD}$ | 69 | 2 | 2747.51±5.60** | 2323.50±5.36** | 7.12±0.02**
$\mathcal{GWE}$ | 100 | 3 | 2749.97±2.31** | 2327.66±2.43** | 7.16±0.01**

  * •

# the counting numbers, * better than $\mathcal{D+C+T}$, ** better than
$\mathcal{D+C+T}$ and best in _T_ , _D_ , and _B_.

The results in Table II show that 6 of the 15 sets of experiments outperform
$\mathcal{D+C+T}$. This reflects that some text features are beneficial for
load forecasting. Among these text features, $\mathcal{WF}$ in all three text
types reduces forecasting errors, especially in _T_. In addition,
$\mathcal{SE}$, $\mathcal{TD}$ and $\mathcal{GWE}$ from _B_ also improved
$\mathcal{D+C+T}$ and are better than those in _T_ and _D_. From now on these
best-performing features are renamed as follows: $\mathcal{WF}$ from _T_ , _D_
, and _B_ are renamed as $\mathcal{WF}_{T}$, $\mathcal{WF}_{D}$, and
$\mathcal{WF}_{B}$, whilst the other beneficial $\mathcal{F}_{t}$s are
$\mathcal{SE}_{B}$, $\mathcal{TD}_{B}$, and $\mathcal{GWE}_{B}$.

####  III-B2 Features combination

The performance improvement with different textual features combination is
tested. This is done because it is expected that the different textual
features have considerable overlap, coming from similar texts, especially if
considering the information contained in _T_ , _D_ , and _B_. The combinations
considered are $\mathcal{M}_{0}:=\mathcal{WF}_{T}$,
$\mathcal{M}_{1}:=\mathcal{WF}_{T}+\mathcal{WF}_{D}+\mathcal{WF}_{B}$,
$\mathcal{M}_{2}:=\mathcal{WF}_{T}+\mathcal{SE}_{B}$,
$\mathcal{M}_{3}:=\mathcal{WF}_{T}+\mathcal{TD}_{B}$,
$\mathcal{M}_{4}:=\mathcal{WF}_{T}+\mathcal{GWE}_{B}$,
$\mathcal{M}_{5}:=\mathcal{WF}_{T}+\mathcal{SE}_{B}+\mathcal{TD}_{B}$,
$\mathcal{M}_{6}:=\mathcal{WF}_{T}+\mathcal{SE}_{B}+\mathcal{GWE}_{B}$,
$\mathcal{M}_{7}:=\mathcal{WF}_{T}+\mathcal{TD}_{B}+\mathcal{GWE}_{B}$,
$\mathcal{M}_{8}:=\mathcal{WF}_{T}+\mathcal{SE}_{B}+\mathcal{TD}_{B}+\mathcal{GWE}_{B}$.

For these combinations, results are plotted in Figure 2. Here it is possible
to see how by combining $\mathcal{WF}_{D}$ and $\mathcal{WF}_{B}$ with
$\mathcal{WF}_{T}$, the forecasting errors increase. This shows that
$\mathcal{WF}_{T}$ is sufficient for forecasting in terms of word frequency.

$\mathcal{M}_{2}$ shows the addition of $\mathcal{SE}_{B}$ to reduce the
forecasting errors. $\mathcal{GWE}_{B}$ in $\mathcal{M}_{4}$ reduce the error
spread in the box plot. The combination $\mathcal{M}_{6}$, obtained from
$\mathcal{SE}_{B}$ and $\mathcal{GWE}_{B}$, brings together the advantages of
both and has the best performance, which is then used for further analysis.

Figure 2: Errors box plot for the models with textual features combination.
The left subplot is for the $\overline{rmse}$(MW), the middle one for the
$\overline{mae}$(MW), and the right one for the $\overline{smape}$(%).

####  III-B3 Errors Analysis

Errors are analysed according to different hours and day types. This analysis
is carried out on the model trained with the $\mathcal{M}_{6}$ input
combination. In Figure 3, it is possible to see the difference in terms of
$\overline{rmse}$, $\overline{mae}$ and $\overline{smape}$ between the
benchmark model and the advanced model with textual features for different
hours of the day. The performance improvement is generally more remarkable in
the first and last hours of the day, usually characterised by more giant ramps
in demand. In Table III, the same analysis is presented comparing the
performance on weekdays and weekends. The error is more significant on
weekends, probably because of the lower absolute value of the demand, and the
advanced model increases its advantage on weekends.

Figure 3: $\overline{rmse}$, $\overline{mae}$, and $\overline{smape}$ on different hours. The dashed and solid lines are forecasting without and with textual features. TABLE III: $\overline{rmse}$, $\overline{mae}$, and $\overline{smape}$ on weekdays and weekends. Day type |  $\overline{rmse}$(MW) |  $\overline{rmse}$(MW)-T | $\overline{mae}(MW)$ |  $\overline{mae}$(MW)-T |  $\overline{smape}$(%) |  $\overline{smape}$(%)-T
---|---|---|---|---|---|---
Weekdays | 2846.73 | 2762.44 | 2392.79 | 2313.22 | 7.23 | 6.95
Weekends | 2623.65 | 2431.61 | 2268.61 | 2089.53 | 7.29 | 6.70

####  III-B4 DM-test

The Diebold-Mariano (DM) test was applied to the forecasts across models to
evaluate the differences in forecast accuracy statistically. The null
hypothesis $\mathcal{H}_{0}$ is that there is no significant difference
between the two models. The alternative hypothesis is that one model is better
than another, given the one-sided situation. With a p-value less than 0.05, we
can infer a better model. We used four models for the DM-test: the ETR
$\mathcal{D}$, benchmark $\mathcal{D+C+T}$ in Table I, $\mathcal{M}_{0}$ and
$\mathcal{M}_{6}$ in Figure 2.

Table IV shows the p-values of DM-test for the four models. The bolded
p-values are less than 0.05 where we reject the null hypothesis and take the
model in the column better than the one in the row. For example, when
comparing the model $\mathcal{D+C+T}$ and $\mathcal{M}_{0}$, the p-value is
0.0404 and less than 0.05. So we found that there is a statistically
significant difference between the forecasting accuracy of model
$\mathcal{M}_{0}$ and $\mathcal{D+C+T}$, and $\mathcal{M}_{0}$ is superior to
$\mathcal{D+C+T}$.

TABLE IV: P-values of DM-test results

| $\mathcal{D}$ | $\mathcal{D+C+T}$ | $\mathcal{M}_{0}$ | $\mathcal{M}_{6}$
---|---|---|---|---
$\mathcal{D}$ | 1.0000 | 0.0000** | 0.0000** | 0.0000**
$\mathcal{D+C+T}$ | 0.9999 | 1.0000 | 0.0404* | 0.0181*
$\mathcal{M}_{0}$ | 0.9999 | 0.9596 | 1.0000 | 0.0322*
$\mathcal{M}_{6}$ | 0.9999 | 0.9818 | 0.9678 | 1.0000

  * 1

* for $0.01<p<0.05$, and ** for $0<p\leq 0.01$

##  IV Discussion

In the following subsections IV-A, IV-B, and IV-C, we would explain the
relationships between textual features and electricity load, from the global,
local, and causality views.

###  IV-A Global correlations

Although model $\mathcal{M}_{6}$ suggested that the combination of
$\mathcal{WF}_{T}$, $\mathcal{SE}_{B}$, and $\mathcal{GWE}_{B}$ is the best-
performed, we still list all the beneficial textual feature groups:
$\mathcal{WF}_{T}$, $\mathcal{WF}_{D}$, $\mathcal{WF}_{B}$,
$\mathcal{SE}_{B}$, $\mathcal{TD}_{B}$, $\mathcal{GWE}_{B}$. The detailed
descriptions are in Table V.

TABLE V: Textual features description Group | Features
---|---
$\mathcal{WF}_{T}$ |  Andy, Murray, newspaper, headlines, rules, Ireland, year, NHS, staff, Glasgow, family, home, European, mark, updates, Paris, say, elections, premier, hit, bomb, second, funeral, talks, Spain, budget, driver, care, sorry, Scotland, job, coronavirus
$\mathcal{WF}_{D}$ |  following, around, Ireland, MPS, least, away, reach, schools, wife, shows, weeks, help, figures, days, lead, Wales, security, hit, outside, Scotland, Monday, leaders, restrictions, pandemic, coronavirus
$\mathcal{WF}_{B}$ | coming, power, city, inside, job, ahead, social, strong, return, war
$\mathcal{SE}_{B}$ | the minimum subjectivity value
$\mathcal{TD}_{B}$ |  Topic-5 (Politics related to Ireland): Ireland, Northern, Irish, Belfast, DUP, Foster, republic, border, Sinn, Neill
Topic-18 (Coronavirus): covid, coronavirus, pandemic, cummings, Downing,
street, question, adviser, Johnson, questions
$\mathcal{GWE}_{B}$ |  Dim-9 (Weapons): Hossein, warhead, Gangnam, interceptor, missiles, bomb, enriched, clerical, Quds, Ballistic
Dim-51 (transportation): Persia, Ibn, Arriva, Transpennine, Merseyrail, fax,
Mesopotamia, BBBofC, Crosscountry,Daren
Dim-69 (Military): ang, corps, Muhammadu, commandant, army, commander,
graduated, ante, military, Buhari

We then measured the correlation between textual features and loads. Due to a
large number of words in $\mathcal{WF}$, we only mention six of the top 3 with
the strongest positive and negative correlations overall. Besides that,
$\mathcal{SE}_{B}$, $\mathcal{TD}_{B}$, and $\mathcal{GWE}_{B}$ were included.
Figure 4 illustrates the correlations scenarios over hours on different
seasons, weekdays, and weekends.

Figure 4: Pearson coefficients for textual features and load. Colored grids
with $p<0.05$ stand for significant Pearson correlations. Blue ones are with a
positive correlation between the feature and the hour. The red ones stand for
the negative correlations.

Generally, Figure 4 presents more obvious correlations in spring, summer, and
weekends. Except for seasons correlations, driver-T shows a positive
correlation with load in the dawn and early morning in autumn and winter. The
correlation is more pronounced for mps-D in winter. The three words with
negative correlations in Figure 4 are all related to coronavirus. They show
strong negative correlations with load in the spring and summer, with
decreasing correlations in the subsequent seasons. The coronavirus-related
Topic-18 shows similar regularity. We also noticed that the peaks differ in
spring (daytime) and summer (evening). In addition, social sentiment
correlated higher in winter. Among the dimensional features of GloVe, only
Dim-51, train transportation in the UK, presents a positive correlation with
the load in spring and summer and is more noticeable on weekends. This
correlation is relatively strong in the daytime in spring, and there is a peak
from hour 19h to 22h in summer.

###  IV-B Local correlations

Figure 5 lists two days with the most negative (2021-02-20) and positive
(2021-01-22) coefficients of driver-T, for example. Each subplot contains the
textual features (y-axis) and their coefficients (x-axis) of the LIME model.
The word frequencies of coronavirus-T, coronavirus-D, and pandemic-D serve
negative roles on both days. At the same time, the coefficients of the other
features vary, which is reasonable from a local view.

Figure 5: Feature coefficients from LIME model

###  IV-C Causality effects

We tested the causality effect of the text features from the subsection IV-A
and IV-B separately concerning each of the 48 half-hours of the coming day’s
load. We kept the treatment affects $\tau$s in Formula 7 corresponding to
$p<0.05$, set the rest to 0 and plotted the $\tau$ distribution of each
feature over the day, as shown in Figure 6.

Figure 6: Causality effects from the Double ML in half-hours. The x-axis is
the causality treatment effects, and the y-axis is the textual features.

According to [48], the causality effect around 10% to 20% is significant. In
Figure 6, the coefficients are mainly gathered around 0, indicating a weak
causality effect. We also notice that the causality distributions for some
features are flat. For example, pandemic-D, which shows a negative causality
effect stretched to -30%, is evidence that the word frequency of ‘pandemic’ in
news descriptions causes loads forecasting negatively. Topic-18, also related
to coronavirus, shows a relatively strong negative correlation with load in
Figure 4. However, in Figure 6, the coefficients for Topic-18 all stack up
around 0. We, therefore, cannot conclude that there is a robust causal
relationship between Topic-18 and load forecasting. This fact again confirms
that correlation and causality are different aspects.

###  IV-D Considerations

This work proceeds by starting with essential forecasting of the electricity
load from the UK with the calendar and temperature features, continuing to
explore the textual features that are helpful for forecasting. Followed by
some statistical tests to show the significant improvement and analysis of the
explanations in key findings.

There is still room for some discussion. The causality effect distributions in
Figure 6 show that several textual features have multiple modes, where some
modes are centered on coefficients with absolute value $\geq 10\%$. For
instance, a negative mode is identified in Dim-9, showing that there exist
conditions that this dimension (related to tensions in the Middle East and
assumed consequences on the oil market) is predicted to have a negative causal
impact on electricity demand. This illustrates the potential of such text-
based features to enrich predictive analysis applied to forecasting total load
at the national or regional level. Another example is the Dim-69: at the
beginning, it was included among the features selected through the Granger
test. However, a second verification, related to the military, suggests a weak
relation with electric load in the UK and is difficult to observe on the day
ahead time scale. It is then considered that: i) this feature has, in general,
a weak correlation with the load in every hour of the day, and every season or
day category (as seen in Figure 4, and ii) its causality score in Figure 6 is
dense around zero. These facts suggest that Dim-69 has been identified
initially because of spurious correlation and that further analysis and human
evaluation are necessary to understand the importance of every feature to be
fed to the forecast model.

Generally, a low Pearson correlation corresponds to low causality, but some
cases show high correlation and low causality, such as Topic-18. This may be
due to coincidence, and we have no evidence that Topic-18 caused the change in
electricity load.

The study depends on the dataset used, containing the period of the COVID-19
pandemic in the UK. This rare event influences the results. In the other case,
it is good to verify that the method proposed identified the keywords related
to the COVID-19 pandemic, which show the most relevance for load variation.

It is possible now to provide answers to the three main questions listed in
Section I-C. The first research question asked _IF_ it is possible to extract
valuable information from news in order to improve electricity demand
prediction. The answer is yes, and this can be seen in several results of this
work, such as in Table II, which shows the improvements of the forecasting
algorithm designed on regression, temperature, and calendar information when
adding features extracted from textual contents. Also, Figure 4 shows the
correlation at different hours, seasons, and day types of the most relevant
textual features. Moreover, the text-based model improves the official
standard by around 4%, 11%, and 10% in terms of RMSE, MAE, and SMAPE.

The second question was _HOW_ to treat textual information in order to extract
valuable features to improve demand prediction. We would answer the question
with the best performance textual features presented in the experiments: word
frequency counting, sentiment scores, and global word embeddings. In
particular, the last method is expected to be more robust to new keywords
relative to new concepts or events that the public has not yet experienced.

The third research question posed was to understand _WHY_ the improved
performance was observed and to explain the phenomena identified. We would
explain some key findings in the word frequency analysis. Firstly, the effect
of the recent COVID-19 pandemic appears as keywords. It is thought that the
mechanism identified by the algorithm is related to the reduced demand due to
lockdowns enforced in the years 2020-21. These keywords show obvious negative
correlations and causality effects. Secondly, the news related to Northern
Ireland politics impacts the electricity demand in Topic-5. The interpretation
is that these are a symptom of more generic political instability or may
result from economic problems that cause a demand reduction. Thirdly, the
tension in the Middle East, identified by Dim-9 mainly related to weapons. It
is hypothesized that international tension in the oil-rich region may impact
the economy and hence the electric load.

The work also has some limitations: Firstly, this work avoids exploring
further consequences of social sciences, economics, or energy policy findings.
The focus is kept on testing the main hypotheses and producing results that
can be explored further. Secondly, the work is carried out on national
aggregated electric demand, which is a parameter with a relatively stable
pattern. Therefore day-ahead forecast errors are low, and it is more difficult
to identify the effect of the additional data tested.

##  V Conclusions

This paper studied the link between unstructured textual information in news
and electricity demand. The overall methodology can be summarised as follows:
firstly, textual information in the news is converted into numerical time
series, including count features, word frequencies, sentiment scores, topic
distributions, and word embeddings according to different methods of TextBlob,
LDA model, and GloVe. Secondly, after the Granger test aimed at removing
spurious correlations, the rest features are fed to an existing load
forecasting algorithm working with known predictors such as calendar
information and temperature values. Finally, the performance is compared, and
the inputs are analysed to understand the mechanism of the news affecting
electricity load.

The study was carried out on the datasets of news and electricity demand
related to the United Kingdom for 2016-2021. In general, reduced performance
improvements in the region of 4%, 11%, and 10% in RMSE, MAE, and SMAPE are
observed. The best-performing method is a feature combination model with word
frequency from news titles, sentiment scores and GloVe word embeddings from
news text bodies. These features identified keywords relative to the COVID-19
pandemic, the minimum subjectivity of public sentiments, and international
conflicts.

This study, far from closing the subject, opens a new series of questions to
be treated in further research. For example, the results must be replicated on
other datasets (news and load), possibly in different countries and with
different spatial resolutions, to reduce the average effect present in
national electricity demand. Since the effect of some social events has a more
prolonged impact, it is better to replicate the study on longer horizons.
Other methods for data analysis and NLP are worth trying; examples are testing
n-grams instead of single keywords and using more complicated deep networks.
Probabilistic forecasting is another research scenario to benchmark the
performance of our proposed approach against other metrics, such as sharpness,
CRPS, and reliability, and to test the method against other challenges, such
as forecasting extreme loads. It is interesting to understand if the
relationship discovered between the textual data and the electricity demand
can also be explained through other variables, such as economic or criminal
activity; Finally, more fine-grained experiments should focus only on the
situations where the existing methods produce higher errors to verify if
social aspects help reduce significant or potentially more influential
forecast errors.

## Acknowledgement

The author Yun BAI was supported by the program of the China Scholarship
Council (CSC Nos. 202106020064).

## References

  * [1] M. Joshi, D. Das, K. Gimpel, and N. A. Smith, “Movie reviews and revenues: An experiment in text regression,” in _Human language technologies: The 2010 annual conference of the North American chapter of the Association for Computational Linguistics_ , 2010, pp. 293–296.
  * [2] M. P. Clements and U. Fritsche, “Text-based data and forecasting: Editor’s introduction,” pp. 1476–1477, 2020.
  * [3] D. Obst, “Textual data and transfer learning for time series forecasting,” Ph.D. dissertation, Aix-Marseille, 2021.
  * [4] M. Cecchini, H. Aytug, G. J. Koehler, and P. Pathak, “Making words work: Using financial text as a predictor of financial events,” _Decision support systems_ , vol. 50, no. 1, pp. 164–175, 2010.
  * [5] R. P. Schumaker and H. Chen, “A discrete stock price prediction engine based on financial news,” _Computer_ , vol. 43, no. 1, pp. 51–56, 2010.
  * [6] P. Resnik, A. Garron, and R. Resnik, “Using topic modeling to improve prediction of neuroticism and depression in college students,” in _Proceedings of the 2013 conference on empirical methods in natural language processing_ , 2013, pp. 1348–1353.
  * [7] F. Rodrigues, I. Markou, and F. C. Pereira, “Combining time-series and textual data for taxi demand prediction in event areas: A deep learning approach,” _Information Fusion_ , vol. 49, pp. 120–129, 2019.
  * [8] N. Peng, K. Li, and Y. Qin, “Leveraging multi-modality data to airbnb price prediction,” in _2020 2nd International Conference on Economic Management and Model Engineering (ICEMME)_. IEEE, 2020, pp. 1066–1071.
  * [9] Y. Zhang, P. Siriaraya, Y. Kawai, and A. Jatowt, “Analysis of street crime predictors in web open data,” _Journal of Intelligent Information Systems_ , vol. 55, no. 3, pp. 535–559, 2020.
  * [10] X. Zhang, H. Saleh, E. M. Younis, R. Sahal, and A. A. Ali, “Predicting coronavirus pandemic in real-time using machine learning and big data streaming system,” _Complexity_ , vol. 2020, 2020.
  * [11] Y. Huang, W. Huang, S. Yan, H. Wang, and J. Yan, “The research on the forecast of tourism demand based on baidu search index–taking beijing as an example,” in _2021 13th International Conference on Intelligent Human-Machine Systems and Cybernetics (IHMSC)_. IEEE, 2021, pp. 91–94.
  * [12] Y. Bai, X. Li, H. Yu, and S. Jia, “Crude oil price forecasting incorporating news text,” _International Journal of Forecasting_ , vol. 38, no. 1, pp. 367–383, 2022.
  * [13] F. Bravo-Marquez, M. Mendoza, and B. Poblete, “Combining strengths, emotions and polarities for boosting twitter sentiment analysis,” in _Proceedings of the second international workshop on issues of sentiment discovery and opinion mining_ , 2013, pp. 1–9.
  * [14] K. Sailunaz and R. Alhajj, “Emotion and sentiment analysis from twitter text,” _Journal of Computational Science_ , vol. 36, p. 101003, 2019.
  * [15] B. Xue, C. Fu, and Z. Shaobin, “A study on sentiment computing and classification of sina weibo with word2vec,” in _2014 IEEE International Congress on Big Data_. IEEE, 2014, pp. 358–363.
  * [16] D. Tang, B. Qin, X. Feng, and T. Liu, “Effective lstms for target-dependent sentiment classification,” _arXiv preprint arXiv:1512.01100_ , 2015.
  * [17] D. Tang, B. Qin, and T. Liu, “Document modeling with gated recurrent neural network for sentiment classification,” in _Proceedings of the 2015 conference on empirical methods in natural language processing_ , 2015, pp. 1422–1432.
  * [18] Y. Wang, M. Huang, X. Zhu, and L. Zhao, “Attention-based lstm for aspect-level sentiment classification,” in _Proceedings of the 2016 conference on empirical methods in natural language processing_ , 2016, pp. 606–615.
  * [19] C. H. Papadimitriou, H. Tamaki, P. Raghavan, and S. Vempala, “Latent semantic indexing: A probabilistic analysis,” in _Proceedings of the seventeenth ACM SIGACT-SIGMOD-SIGART symposium on Principles of database systems_ , 1998, pp. 159–168.
  * [20] S. Deerwester, S. T. Dumais, G. W. Furnas, T. K. Landauer, and R. Harshman, “Indexing by latent semantic analysis,” _Journal of the American society for information science_ , vol. 41, no. 6, pp. 391–407, 1990.
  * [21] D. M. Blei, A. Y. Ng, and M. I. Jordan, “Latent dirichlet allocation,” _Journal of machine Learning research_ , vol. 3, no. Jan, pp. 993–1022, 2003\.
  * [22] T. Joachims, “A probabilistic analysis of the rocchio algorithm with tfidf for text categorization.” Carnegie-mellon univ pittsburgh pa dept of computer science, Tech. Rep., 1996.
  * [23] J. Pennington, R. Socher, and C. D. Manning, “Glove: Global vectors for word representation,” in _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_ , 2014, pp. 1532–1543.
  * [24] T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient estimation of word representations in vector space,” _arXiv preprint arXiv:1301.3781_ , 2013\.
  * [25] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [26] Z. Zhou, J. Zhao, and K. Xu, “Can online emotions predict the stock market in china?” in _International conference on web information systems engineering_. Springer, 2016, pp. 328–342.
  * [27] A. Tom, N. Joel, B. Steven _et al._ , “Nltk documentation,” https://www.nltk.org.
  * [28] C. Kaur and A. Sharma, “Twitter sentiment analysis on coronavirus using textblob,” EasyChair, Tech. Rep., 2020.
  * [29] R. Bose, P. Aithal, and S. Roy, “Sentiment analysis on the basis of tweeter comments of application of drugs by customary language toolkit and textblob opinions of distinct countries,” _Int. J_ , vol. 8, 2020.
  * [30] A. A. Chaudhri, S. Saranya, and S. Dubey, “Implementation paper on analyzing covid-19 vaccines on twitter dataset using tweepy and text blob,” _Annals of the Romanian Society for Cell Biology_ , pp. 8393–8396, 2021.
  * [31] X. Li, Y. Bai, and Y. Kang, “Exploring the social influence of the kaggle virtual community on the m5 competition,” _International Journal of Forecasting_ , vol. 38, no. 4, pp. 1507–1518, 2022.
  * [32] C. W. Granger, “Investigating causal relations by econometric models and cross-spectral methods,” _Econometrica: journal of the Econometric Society_ , pp. 424–438, 1969.
  * [33] X. Li, W. Shang, and S. Wang, “Text-based crude oil price forecasting: A deep learning approach,” _International Journal of Forecasting_ , vol. 35, no. 4, pp. 1548–1560, 2019.
  * [34] P. Geurts, D. Ernst, and L. Wehenkel, “Extremely randomized trees,” _Machine learning_ , vol. 63, no. 1, pp. 3–42, 2006.
  * [35] D.-H. Kim, E.-K. Lee, and N. B. S. Qureshi, “Peak-load forecasting for small industries: A machine learning approach,” _Sustainability_ , vol. 12, no. 16, p. 6539, 2020.
  * [36] R. Porteiro, S. Nesmachnow, and L. Hernández-Callejo, “Short term load forecasting of industrial electricity using machine learning,” in _Ibero-American Congress of Smart Cities_. Springer, 2019, pp. 146–161.
  * [37] W. Xiang, P. Xu, J. Fang, Q. Zhao, Z. Gu, and Q. Zhang, “Multi-dimensional data-based medium-and long-term power-load forecasting using double-layer catboost,” _Energy Reports_ , vol. 8, pp. 8511–8522, 2022.
  * [38] M. T. Ribeiro, S. Singh, and C. Guestrin, ““ why should i trust you?” explaining the predictions of any classifier,” in _Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining_ , 2016, pp. 1135–1144.
  * [39] D. Slack, S. Hilgard, E. Jia, S. Singh, and H. Lakkaraju, “Fooling lime and shap: Adversarial attacks on post hoc explanation methods,” in _Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society_ , 2020, pp. 180–186.
  * [40] J. Aldrich, “Correlations genuine and spurious in pearson and yule,” _Statistical science_ , pp. 364–376, 1995.
  * [41] V. Chernozhukov, D. Chetverikov, M. Demirer, E. Duflo, C. Hansen, W. Newey, and J. Robins, “Double/debiased machine learning for treatment and structural parameters,” 2018.
  * [42] V. Chernozhukov, M. Goldman, V. Semenova, and M. Taddy, “Orthogonal machine learning for demand estimation: High dimensional causal inference in dynamic panels,” _arXiv_ , pp. arXiv–1712, 2017.
  * [43] “Entso-e. transparency platform,” https://transparency.entsoe.eu.
  * [44] D. Obst, J. De Vilmarest, and Y. Goude, “Adaptive methods for short-term electricity load forecasting during covid-19 lockdown in france,” _IEEE transactions on power systems_ , vol. 36, no. 5, pp. 4754–4763, 2021.
  * [45] B. Wu, L. Wang, S.-X. Lv, and Y.-R. Zeng, “Effective crude oil price forecasting using new text-based and big-data-driven model,” _Measurement_ , vol. 168, p. 108468, 2021.
  * [46] L. Chang, D. Pedrno, and M. Ambrosio, “News-crawler,” https://github.com/LuChang-CS/news-crawler.
  * [47] S. Matthew, “Bbc news front page archive,” https://dracos.co.uk/made/bbc-news-archive/archive.php.
  * [48] V. Chernozhukov, H. Kasahara, and P. Schrimpf, “Causal impact of masks, policies, behavior on early covid-19 pandemic in the us,” _Journal of econometrics_ , vol. 220, no. 1, pp. 23–62, 2021.


$f:{\mathcal{X}}\rightarrow\mathbb{R}^{|\mathcal{A}|}$ which takes as input
the feature vectors $x_{i}$ and outputs a vector of softmax probabilities over
labels, i.e. the $a$-th component of the vector $f(x)$, denoted as
$(f(x))_{a}$ corresponds to the estimated probability
$\mathbb{P}(A^{{\textup{gt}}}=a\mid X=x)$.

Next, we use $f$ to define the ground truth behaviour policy,

$\pi^{b}(a\mid x)=(f(x))_{a}.$

For the target policies, we use $f$ to define a parametric class of target
policies using a trained classifier
$f:{\mathcal{X}}\rightarrow\mathbb{R}^{|\mathcal{A}|}$.

$\pi^{\alpha^{\ast}}(a\mid
x)=\alpha^{\ast}\cdot{\mathbbm{1}}(a=\arg\max_{a^{\prime}\in\mathcal{A}}(f(x))_{a^{\prime}})+\frac{1-\alpha^{\ast}}{|\mathcal{A}|},$

where $\alpha^{\ast}\in[0,1]$. A value of $\alpha^{\ast}$ close to 1 leads to
a near-deterministic and well-performing policy. As $\alpha^{\ast}$ decreases,
the policy gets increasingly worse and ‘noisy’. In this experiment, we
consider target policies $\pi^{*}=\pi^{\alpha^{\ast}}$ for
$\alpha^{\ast}\in\\{0.0,0.2,0.4,\dots,1.0\\}$.

Using the behaviour policy defined above, we generate the contextual bandits
data described with training and evaluation datasets of sizes $m$ and $n$
respectively.

##### Estimation of behaviour policy $\widehat{\pi}^{b}$ and marginal ratio
$\hat{w}(y)$

We do not assume that the behaviour policy $\pi^{b}$ is known, and therefore
estimate it using training data. To estimate the behaviour policy
$\widehat{\pi}^{b}$, we train a random forest classifier using the training
data. This estimate of behaviour policy is used for all the baselines in our
experiment. Since the reward is binary, we can estimate the marginal ratios
$\hat{w}(y)=\mathbb{E}_{\pi^{b}}[\hat{\rho}(A,X)\mid Y=y]$ by directly
estimating the sample mean of $\hat{\rho}(A,X)$ for datapoints with $Y=y$. We
re-use the $m$ training datapoints to estimate this sample mean.

##### Baselines

We compare our estimator with Direct Method (DM), IPW and DR estimators. In
addition, we also consider Switch-DR [5] and DR with Optimistic Shrinkage
(DRos) [17]. To estimate $\hat{q}(x,a)$ for DM and DR estimators, we use
random forest classifiers (since reward $Y$ is binary). Moreover, because of
the binary nature of $Y$, the alternative method of estimating MR yields the
same estimator as the original method, therefore we do not consider the two
separately here. Additionally, in this experiment, we do not include MIPS (or
G-MIPS) baseline, as there is no natural informative embedding $E$ of the
action $A$.

####  F.3.1 Results

For this experiment, we compute the results over 10 different sets of logged
data replicated with different seeds. Figures 11 \- 17 show the results
corresponding to each baseline for the different datasets. It can be seen that
across all datasets, the MR achieves the smallest MSE with increasing
evaluation data size $n$. Moreover, across all datasets, MR attains the
minimum MSE with relatively small number of evaluation data ($n\leq 100$).

Unlike the experiments in Section 5.1, we observe that the KL-divergence
between target and behaviour policy decreases as $\alpha^{\ast}$ increases
(see Figure 9). Therefore, as $\alpha^{\ast}$ increases the shift between
target and behaviour policies decreases. Figures 11 \- 16 show that as
$\alpha^{\ast}$ increases, the difference between the MSE, squared bias and
variance of MR and the other baselines decreases. This confirms our findings
from earlier experiments that MR performs especially better than the other
baselines when the difference between behaviour and target policies is large.

Moreover, the figures also include results with increasing number of training
data $m$. It can be seen that MR out-performs the baselines even when the
number of training data $m$ is small ($m=100$). Moreover, the relative
advantage of MR improves with increasing $m$.

Figure 9: KL divergence $D_{\textup{KL}}(\pi^{b}\,||\,\pi^{*})$ with
increasing $\alpha^{\ast}$ for the classification data experiments. Here, we
only include the results for a specific choice of parameters for the Letter
dataset. We observe similar results for other datasets and parameter choices.

(a) $d=1000$, $n=10$, $\alpha^{\ast}=0.8$

(b) $d=1000$, $n=200$, $\alpha^{\ast}=0.8$

(c) $d=1000$, $n=800$, $\alpha^{\ast}=0.8$

Figure 10: MSE with varying number of training data $m$ for different choices
of parameters.

(a) Results with varying $n$ for $\alpha^{\ast}=0.2$ and $m=1000$

(b) Results with varying $\alpha^{\ast}$ for $m=n=1000$

(c) Results with varying $m$ for $n=1000$ and $\alpha^{\ast}=0.6$

Figure 11: Results for OptDigits dataset

(a) Results with varying $n$ for $\alpha^{\ast}=0.2$ and $m=1000$

(b) Results with varying $\alpha^{\ast}$ for $m=n=1000$

(c) Results with varying $m$ for $\alpha^{\ast}=0.6$ and $n=1000$

Figure 12: Results for PenDigits dataset

(a) Results with varying $n$ for $\alpha^{\ast}=0.2$ and $m=1000$

(b) Results with varying $\alpha^{\ast}$ for $n=1000$

(c) Results with varying $m$ for $\alpha^{\ast}=0.6$ and $n=1000$

Figure 13: Results for SatImage dataset

(a) Results with varying $n$ for $\alpha^{\ast}=0.2$ and $m=1000$

(b) Results with varying $\alpha^{\ast}$ for $m=n=1000$

(c) Results with varying $m$ for $\alpha^{\ast}=0.6$ and $n=1000$

Figure 14: Results for Letter dataset

(a) Results with varying $n$ for $\alpha^{\ast}=0.2$ and $m=1000$

(b) Results with varying $\alpha^{\ast}$ for $m=n=1000$

(c) Results with varying $m$ for $\alpha^{\ast}=0.6$ and $n=1000$

Figure 15: Results for Mnist dataset

(a) Results with varying $n$ for $\alpha^{\ast}=0.2$ and $m=500$

(b) Results with varying $\alpha^{\ast}$ for $n=500$ and $m=1000$

(c) Results with varying $m$ for $\alpha^{\ast}=0.6$ and $n=500$

Figure 16: Results for Digits dataset. Note that compared to other datasets we
consider smaller maximum dataset sizes $m,n$ here as the total number of
available datapoints was 1797.

(a) Results with varying $n$ for $\alpha^{\ast}=0.4$ and $m=2000$

(b) Results with varying $\alpha^{\ast}$ for $n=100$ and $m=2000$

(c) Results with varying $m$ for $\alpha^{\ast}=0.4$ and $n=100$

Figure 17: Results for CIFAR-100 dataset.

###  F.4 Application to Average Treatment Effect (ATE) estimation

In this subsection, we provide additional details for our experiment applying
MR to the problem of ATE estimation presented in the main text. We begin by
describing the dataset being used in this experiment.

##### Twins dataset

We use the Twins dataset as studied by [40], which comprises data from twin
births in the USA between 1989-1991. The treatment $a=1$ corresponds to being
born the heavier twin and the outcome $Y$ corresponds to the mortality of each
of the twins in their first year of life. Since the data includes records for
both twins, their outcomes would be considered as the two potential outcomes.
Specifically, $Y(1)$ corresponds to the mortality of the heavier twin (and
likewise for $Y(0)$). Closely following the methodology of [40], we only chose
twins which are the same sex and weigh less than 2kgs. This provides us with a
dataset of 11984 pairs of twins.

The mortality rate for the lighter twin is 18.9% and for the heavier twin is
16.4%, leading to the ATE value being $\theta_{\textup{ATE}}=-2.5\%$. For each
twin-pair we obtained 46 covariates relating to the parents, the pregnancy and
birth.

##### Treatment assignment

To simulate an observational study, we selectively hide one of the two twins
by defining the treatment variable $A$ which depends on the feature
_GESTAT10_. This feature, which takes integer values from 0 to 9, is obtained
by grouping the number of gestation weeks prior to birth into 10 groups. Then
we sample actions $A$ as follows,

$A\mid X\sim\textup{Bern}(Z/10),$

where $Z$ is _GESTAT10_ , and $X$ are all the 46 features corresponding to a
twin pair (including _GESTAT10_).

Using the treatment assignments defined above, we generate the observational
data by selectively hiding one of the two twins from each pair. Next, we
randomly split this dataset into training and evaluation datasets of sizes $m$
and $n$ respectively. In this experiment, we consider $m=5000$ training
datapoints.

##### Baselines

Recall that ATE estimation can be formulated as the difference between off-
policy values of deterministic policies $\pi^{(1)}\coloneqq{\mathbbm{1}}(A=1)$
and $\pi^{(0)}\coloneqq{\mathbbm{1}}(A=0)$. Therefore, any OPE estimator can
be applied to ATE estimation. In this experiment, we compare our estimator
against the baselines considered in our OPE experiments in Section F.3. This
includes the Direct Method (DM), IPW and DR estimators as well as Switch-DR
[5] and DR with Optimistic Shrinkage (DRos) [17]. To estimate $\hat{q}(x,a)$
for DM and DR estimators, we use multi-layer perceptrons (MLP) trained on the
$m$ training datapoints. Additionally, we estimate the behaviour policy
$\widehat{\pi}^{b}$ using random forest classifier trained on the full
training dataset.

Since the outcome in this experiment is binary, we estimate the weights
$w(y)=\mathbb{E}_{\pi^{b}}[\hat{\rho}(A,X)\mid Y=y]$ directly by estimating
the sample mean of $\hat{\rho}(A,X)$ for datapoints with $Y=y$. This means
that the alternative method of estimating MR yields the same value as the
default method. We therefore do not consider these estimators separately.
Additionally, since there is no natural embedding $R$ of the covariate-action
space which satisfies the conditional dependence Assumption D.1, we do not
consider the G-MIPS (or MIPS) estimator either.

##### Performance metric

For our evaluation, we consider the absolute error in ATE estimation,
$\epsilon_{\textup{ATE}}$, defined as:

$\epsilon_{\textup{ATE}}\coloneqq|\hat{\theta}^{(n)}_{\textup{ATE}}-\theta_{\textup{ATE}}|.$

Here, $\hat{\theta}^{(n)}_{\textup{ATE}}$ denotes the value of the ATE
estimated using $n$ evaluation datapoints. For example, for the IPW estimator,
the $\hat{\theta}^{(n)}_{\textup{ATE}}$ can be written as:

$\hat{\theta}^{(n)}_{\textup{ATE}}=\widehat{\textup{ATE}}_{\textup{IPW}}=\frac{1}{n}\sum_{i=1}^{n}\left(\frac{{\mathbbm{1}}(a_{i}=1)-{\mathbbm{1}}(a_{i}=0)}{\widehat{\pi}^{b}(a_{i}\mid
x_{i})}\right)\,y_{i}.$

All results for this experiment are provided in the main text.

###  F.5 Additional synthetic data experiments

(a) $d=1000$, $n_{a}=100$, $\alpha^{\ast}=0.4$.

(b) $d=10000$, $n_{a}=100$, $\alpha^{\ast}=0.4$.

Figure 18: Results with varying size of evaluation dataset $n$.

(a) $d=1000$, $n_{a}=100$, $n=100$.

(b) $d=10000$, $n_{a}=100$, $n=100$.

Figure 19: Results with varying $\alpha^{\ast}$.

(a) $n_{a}=100$, $n=100$, $\alpha^{\ast}=0.4$.

(b) $n_{a}=500$, $n=100$, $\alpha^{\ast}=0.4$.

Figure 20: Results with varying context dimensions $d$.

(a) $d=100$, $n=100$, $\alpha^{\ast}=0.2$.

(b) $d=100$, $n=100$, $\alpha^{\ast}=0.4$.

Figure 21: Results with varying number of actions $n_{a}$.

In addition to the synthetic data experiments provided in Section 5.1, we also
consider an additional synthetic data setup to obtain further empirical
evidence in favour of the MR estimator, and also compare it against the
generalised version of the MIPS estimator (described as G-MIPS in Appendix D).
Here, we use a similar setup to [14] (albeit without action embeddings $E$)
where the $d$-dimensional context vectors $x$ are sampled from a standard
normal distribution. Likewise, the action space is finite and comprises of
$n_{a}$ actions, i.e. $\mathcal{A}=\\{0,\dots,n_{a}-1\\}$, with $n_{a}$ taking
a range of different values. The reward function is defined as follows:

##### Reward function

The expected reward $q(x,a)\coloneqq\mathbb{E}[Y\mid x,a]$ for these
experiments is defined as follows:

$q(x,a)=\sin\left(a\cdot||x||_{2}\right).$

The reward $Y$ is obtained by adding a normal noise random variable to
$q(x,a)$

$Y=q(X,A)+\epsilon,$

where $\epsilon\sim\mathcal{N}(0,0.01)$. Here, it can be seen that conditional
on $R=(||X||_{2},A)$, the reward $Y$ does not depend on $(X,A)$, i.e., the
embedding $R$ satisfies the conditional independence assumption
$Y\perp\\!\\!\\!\perp(X,A)\mid R$.

##### Behaviour and target policies

We first define a behaviour policy by applying softmax function to $q(x,a)$ as

$\pi^{b}(a\mid
x)=\frac{\exp{(q(x,a))}}{\sum_{a^{\prime}\in\mathcal{A}}\exp{(q(x,a^{\prime}))}}.$

Just like in Section 5.1, to investigate the effect of increasing policy
shift, we define a class of policies,

$\pi^{\alpha^{\ast}}(a|x)=\alpha^{\ast}\,{\mathbbm{1}}(a=\arg\max_{a^{\prime}\in\mathcal{A}}q(x,a^{\prime}))+\frac{1-\alpha^{\ast}}{|\mathcal{A}|}\quad\textup{where}\quad
q(x,a)\coloneqq\mathbb{E}[Y\mid X=x,A=a],$

where $\alpha^{\ast}\in[0,1]$ allows us to control the shift between $\pi^{b}$
and $\pi^{*}$. Again, the shift between $\pi^{b}$ and $\pi^{*}$ increases as
$\alpha^{\ast}\rightarrow 1$. Using the ground truth behaviour policy
$\pi^{b}$, we generate a dataset which is split into training and evaluation
datasets of sizes $m$ and $n$ respectively.

In Figures 18 \- 21, we present the results for this experimental setup for
different choices of paramater configurations.

##### Estimation of behaviour policy $\widehat{\pi}^{b}$ and marginal ratio
$\hat{w}(y)$

For the MR estimator, we estimate the behaviour policy using a random forest
classifier trained on 50% of the training data and use the rest of the
training data to estimate the marginal ratios $\hat{w}(y)$ using multi-layer
perceptrons (MLP). Moreover, for a fair comparison we use a different
behaviour policy estimate $\widehat{\pi}^{b}$ for all other baselines which is
trained on the entire training data.

##### Additional Baselines

In addition to the baselines considered in the main text (Section 5.1), we
also consider Switch-DR [5] and DR with Optimistic Shrinkage (DRos) [17]. In
addition, we also include the results for MR estimated using the alternative
method (‘MR (alt)’) outlined in Section F.1.1. For the G-MIPS estimator
(defined in Appendix D) considered here, we use $R=(a,||x||_{2})$222It is easy
to see that in our setup, the embedding $R=(a,||x||_{2})$ satisfies the
conditional independence assumption $Y\perp\\!\\!\\!\perp(X,A)\mid R$ needed
for G-MIPS estimator to be unbiased. To estimate $\hat{q}(x,a)$ for DM and DR
estimators, we use multi-layer perceptrons (MLPs).

####  F.5.1 Results

For this experiment, the results are computed over 10 different sets of logged
data replicated with different seeds, and in Figures 18 \- 21 we use a total
of $m=5000$ training data.

##### Varying $n$

Figure 18 shows that MR outperforms the other baselines, in terms of MSE and
squared bias, when the number of evaluation data $n\leq 1000$. Additionally,
we observe that in this experiment, MR esitmated using alternative methods, MR
(alt), yields better results than the original method of estimating MR.
Moreover, while the variance of DM is lower than that of MR, the DM method has
a high bias and consequently a high MSE.

##### Varying $\alpha^{\ast}$

Figure 19 shows the results with increasing policy shift. It can be seen that
overall MR methods achieve the smallest MSE with increasing policy shift.
Moreover, the difference between MSE and variance of MR and IPW/DR methods
increases with increasing policy shift, showing that MR performs especially
better than these baselines when the difference between behaviour and target
policies is large.

##### Varying $d$ and $n_{a}$

Figures 20 and 21 show that MR outperforms the other baselines as the context
dimensions and/or number of actions increase. In fact, Figure 21 shows that MR
is significantly robust to increasing action space, whereas baselines like IPW
and DR perform poorly in large action spaces.

##### Varying $m$

Figure 22 shows the results with increasing number of training data $m$. We
again observe that the MR methods ‘MR’ and ‘MR (alt)’ outperforms the other
baselines in terms of the MSE and squared bias even when the number of
training data is low. Moreover, the variance of both the MR estimators
continues to improve with increasing number of training data.

Unlike our experimental results in Section F.2, ‘MR (alt)’ performs better
than the original MR estimator overall. This shows that one of these two
methods is not better than the other consistently in all cases, and their
relative performance depends on the dataset under consideration.

(a) $d=5000$, $n=100$, $n_{a}=10$, $\alpha^{\ast}=0.2$.

(b) $d=5000$, $n=100$, $n_{a}=10$, $\alpha^{\ast}=0.4$.

Figure 22: Results with varying number of training data $m$.

###  F.6 Self-normalised MR estimator

(a) $d=10000$, $n=200$, $n_{a}=20$, $m=5000$.

(b) $d=5000$, $n=200$, $n_{a}=20$, $m=1000$.

(c) $d=10000$, $n=200$, $n_{a}=20$, $m=5000$.

Figure 23: Results for self-normalised estimators with varying target policy
shift $\alpha^{\ast}$ for synthetic data setup considered in Section 5.1.
Here, “SN” denotes self-normalised estimators.

Self-normalization trick has been used in practice to reduce the variance in
off-policy estimators [30]. This technique is also applicable to the MR
estimator, and leads to the self-normalized MR estimator (denoted as
$\theta_{\textup{SNMR}}$) defined as follows:

$\theta_{\textup{SNMR}}\coloneqq\sum_{i=1}^{n}\frac{w(Y_{i})}{\sum_{j=1}^{n}w(Y_{j})}\,Y_{i}.$

We conducted experiments to investigate the effect of self-normalisation on
the performance of the IPW, DR and MR estimators. Figure 23 shows results for
three different choices of parameter configurations. Overall, we observe that
in all settings, the MR and self-normalised MR (SNMR) estimator outperform all
other baselines including the self-normalised IPW and DR estimators (denoted
as SNIPW and SNDR respectively). Moreover, in some settings, where the
importance ratios achieve very high values, self-normalisation can reduce the
variance and MSE of the corresponding estimator (for example, Figure 23(b)).
However, we also observe cases in which self-normalization does not
significantly change the results (Figure 23(a)), or may even slightly worsen
the MSE of the estimators (Figure 23(c)).