# BuildingsBench: A Large-Scale Dataset of 900K Buildings and Benchmark for
Short-Term Load Forecasting

Patrick Emami, Abhijeet Sahu, Peter Graf
National Renewable Energy Lab
{Patrick.Emami, Abhijeet.Sahu<EMAIL_ADDRESS>

###### Abstract

Short-term forecasting of residential and commercial building energy
consumption is widely used in power systems and continues to grow in
importance. Data-driven short-term load forecasting (STLF), although
promising, has suffered from a lack of open, large-scale datasets with high
building diversity. This has hindered exploring the pretrain-then-finetune
paradigm for STLF. To help address this, we present BuildingsBench, which
consists of 1) Buildings-900K, a large-scale dataset of 900K simulated
buildings representing the U.S. building stock, and 2) an evaluation platform
with over 1,900 real residential and commercial buildings from 7 open
datasets. BuildingsBench benchmarks two under-explored tasks: zero-shot STLF,
where a pretrained model is evaluated on unseen buildings without fine-tuning,
and transfer learning, where a pretrained model is fine-tuned on a target
building. The main finding of our benchmark analysis is that synthetically
pretrained models generalize surprisingly well to real commercial buildings.
An exploration of the effect of increasing dataset size and diversity on zero-
shot commercial building performance reveals a power-law with diminishing
returns. We also show that fine-tuning pretrained models on real commercial
and residential buildings improves performance for a majority of target
buildings. We hope that BuildingsBench encourages and facilitates future
research on generalizable STLF. All datasets and code can be accessed from
https://github.com/NREL/BuildingsBench.

##  1 Introduction

Figure 1: BuildingsBench gallery. Top row: commercial buildings (farthest left
is simulated commercial Buildings-900K data). Second and third rows are
residential buildings (farthest left in the second row is simulated
residential Buildings-900K data).

Residential and commercial buildings in the United States are responsible for
about 40% of energy consumption and 35% of greenhouse gas emissions [10].
Globally, these estimates are respectively 30% and 27% [21]. Building energy
demand forecasting plays a part in reducing global emissions by helping to
decarbonize the building sector.

Short-term load forecasting (STLF), which typically ranges from hour-ahead to
a few days ahead, plays a multitude of critical roles. STLF can help match
shifting energy supply with customer demand as well as aid energy markets with
accurately setting prices based on forecasted supply/demand [18]. Accurate
forecasts can be directly used by reinforcement learning [40] and model
predictive control [1, 9] for optimal building energy management.

However, STLF remains a challenging problem as energy demand can fluctuate
heavily due to a variety of unobserved and exogenous factors As such, data-
driven methods have risen in popularity to address STLF [2, 46]. Interest in
these techniques has also been spurred by a rise in deployments of advanced
sensors (i.e., smart meters) that record building energy consumption. However,
the number of instrumented buildings with publicly released data remains low.
Moreover, it is typical for multiple years of historical data to be reserved
for training and validation, with trained models tested on a single held-out
year _for the same building_ [26]. This incurs a lengthy data collection
period per building that is not scalable. There is thus a lack of sufficiently
large publicly available datasets (Table 1) which has hindered the
investigation of large-scale pretraining (i.e., one model trained on every
building [7, 17]). Recent successes of large-scale pretraining outside of
language and vision include pretraining on one million hours of audio for
speech recognition [47], suggesting its promise for STLF.

In this work, we introduce BuildingsBench, an open source platform for large-
scale pretraining and for benchmarking zero-shot STLF [7, 17] and transfer
learning for STLF [43, 14]. In zero-shot STLF, models forecast loads for
unseen target buildings _without any fine-tuning_. This drastically reduces
time-to-deployment on newly instrumented buildings. In transfer learning, a
pretrained model is fine-tuned on a target building assuming a limited yet
realistic amount of data (e.g., 6 months).

As part of BuildingsBench, we introduce Buildings-900K, a dataset of nearly
one million _simulated_ time series, which approaches the scale of datasets in
natural language processing and computer vision. This data is sourced from the
NREL End-Use Load Profiles (EULP) database [41]. The EULP is a multi-year
multi-institution effort to create a statistically representative database of
the entire U.S. building stock’s energy consumption via careful calibration
and validation of physics-based building simulations. We also provide an
evaluation suite that combines 7 open datasets totalling over 1,900 _real_
buildings (Table 2). Example time series from the simulated and real datasets
are shown in Figure 1. Uniquely, BuildingsBench has simulated and real
building energy consumption time series spanning a wide range of geographic
locations, years, and types of both residential and commercial buildings.

Our platform automates the evaluation of a variety of simple and advanced
baselines on the two tasks. Our results on zero-shot STLF reveals
synthetically pretrained models achieve strong performance on real commercial
buildings. We observe a smaller distribution shift between simulated and real
commercial buildings than residential buildings. We also show that pretrained
models can further improve performance by fine-tuning on real commercial and
residential building data. Buildings-900K also enables studying large-scale
pretraining of transformers on geographical time series. The utility of
transformers for forecasting has recently been questioned [44], but a lack of
sufficiently large public time series datasets has made investigating this
challenging. Our main finding is a power-law scaling with diminishing returns
between dataset scale (size and diversity) and generalization (for commercial
buildings). We expect that BuildingsBench will facilitate research on new
modeling techniques and large-scale datasets for generalizable STLF.

To summarize, we contribute 1) Buildings-900K, a simulated dataset for large-
scale pretraining, 2) a platform for benchmarking zero-shot STLF and transfer
learning, and 3) valuable insights on model pretraining for STLF.

##  2 Short-term Load Forecasting

The BuildingsBench benchmark considers the following univariate forecasting
problem. Given $H$ past load values $x_{t-H:t}$ and $H+T$ covariates
$z_{t-H:t+T}$ (defined in Section 3.1), we aim to predict the conditional
distribution for $T$ unobserved future load values $y_{t+1:t+T}$:

$p(y_{t+1},\dots,y_{t+T}\mid x_{t-H},\dots,x_{t},z_{t-H},\dots,z_{t+T}).$ (1)

We consider a day-ahead STLF scenario with $H=168$ hours (one week) and $T=24$
hours. A primary goal of BuildingsBench is to study and evaluate STLF models
which learn a single set of parameters $\theta$ shared by all buildings for
the distribution in Eq. 1. We use a probabilistic formulation for our
benchmark since applications of STLF increasingly require uncertainty
estimates, such as planning and scheduling of renewable energy sources for
buildings [18].

##  3 The Buildings-900K Dataset

In this section, we introduce our dataset for pretraining STLF models.

Simulated data source: Our dataset is sourced from the NREL EULP [41]
database. The EULP provides 15-minute resolution appliance-level consumption
simulated with EnergyPlus [8, 27] for a base set of 900K building models (550K
residential and 350K commercial) spread across all climate regions in the U.S.
It aims to provide a statistically representative picture of the entire U.S.
building stock at various aggregation levels (county, state, etc.) and under
various electrification scenarios. The building simulations were extensively
calibrated over a three year period with advanced metering data ($\sim$ 2.3
million meters), data from 11 utilities, as well as other public/private
datasets related to energy usage. Socio-economic building characteristics,
including location based on Public Use Microdata Area (PUMA, $\sim$2400 PUMAs
in the U.S.—see inset), are sampled from distributions generated from U.S.
Census survey responses. Please see Wilson et al. [41] for the complete
description.

Processing and storage: To create Buildings-900K, we extract 900K total energy
consumption time series (in kilowatt-hours (kWh)) from each of the non-
upgraded buildings in the 2021 version of the EULP. To promote accessibility
of our dataset, we also aggregate the 15-minute resolution to hourly to reduce
the size. This data requires about 110 GB to store, significantly less than
the entire EULP (70+ TB). We store all buildings within each PUMA in a single
Parquet file for each of the two available years (2018 and an aggregated
“typical meteorological year” [41]) and building types
(residential/commercial), which amounts to 9,600 Parquet files. Each file has
a column for the timestamp and a column for each building’s energy consumption
(8,760 rows per file). Processing the EULP to extract this data took $\sim$3
days with Apache PySpark on a 96-core AWS cloud instance.

Pretraining splits and loading: A validation set is created by withholding the
final two weeks of 2018. The test set consists of buildings in four PUMAs that
are withheld from both training and validation. All splits use a 24-hour
sliding-window to extract 192-hour load sub-sequences. Since shuffling large
datasets of sub-sequences is computationally demanding, our platform provides
a custom PyTorch [30] Dataset that creates an index file to map a shuffled
list of integers to a sub-sequence. Each line of the index file is accessible
in $O(1)$ time with Python’s seek function. Apache PyArrow is used to
efficiently load the indexed building’s time series into memory.

Hosting and licensing: Buildings-900K is hosted by the Open Energy Data
Initiative and is available for download with a permissive CC-4.0 license
(link available in App. A).

###  3.1 Feature extraction

Beyond the load time series, we extract covariates on-the-fly to condition
forecasts on:

  * •

Calendar features—day of the week, day of the year, and hour of the day—are
generated from the timestamp, which we then cyclically encode with sine and
cosine.

  * •

The latitude and longitude coordinate of the centroid of the building’s PUMA
(using metadata provided by the EULP) for encoding geospatial information.

  * •

A binary feature for building type (residential (0) or commercial (1)).

##  4 BuildingsBench Evaluation Platform

Table 1: Comparing popular building energy consumption datasets to our STLF
dataset.

# buildings Open access Residential Commercial Total hours # Sites Pecan
Street [36] 1,000 ✗ ✓ ✗ ? ? Electricity [37] 370 ✓ ✗ ✓ $\sim$12.9M 1 Building
D.G.P. 2 [25] 1,636 ✓ ✗ ✓ $\sim$53.6M 19 Low Carbon London [28] 5,567 ✓ ✓ ✗
$\sim$97.5M 1 Buildings-900K (_simulated_) 900,000 ✓ ✓ ✓ $\sim$15B 2400

Table 2: BuildingsBench day-ahead STLF evaluation data. | Residential | Commercial | Years spanned | # Sites | Total days
---|---|---|---|---|---
Buildings-900K test (_simulated_) | 915 | 565 | TMY, 2018 | 4 | 1.1M
BuildingsBench (_real_) | 953 | 970 | 2007–2018 | 10 | 1.2M

BuildingsBench provides a open source software platform for evaluating models
on zero-shot and transfer learning for STLF using a collection of _real
building datasets_. To avoid confusion, in our analysis (Sec. 5) we will use
the name BuildingsBench to refer _only to the real building evaluation data_
and explicitly state whether the results are for the Buildings-900K test set
(simulated) or for BuildingsBench (real) (Table 2). We now describe each task
in more detail.

Zero-shot STLF: For this task, pretrained models are asked to provide day-
ahead forecasts for unseen simulated and real buildings. Our use of “zero-
shot” here refers to a setting where only one week of historical data is
presumed available, and is thus insufficient for training or fine-tuning a
deep neural network. All available years are used for each test building.

Transfer learning: Our transfer learning task assumes 6 months of metered data
has been collected for each target building, about $\sim$180 training samples.
Fine-tuned models are then tasked with day-ahead forecasting for the next 6
months using a 24-hour sliding window.

###  4.1 Real building datasets

Here, we briefly describe the real building data used for evaluating the two
tasks and defer additional details to App. C. All together, our real building
benchmark has over 1,900 buildings and 1.2M days of energy usage.

Electricity [37]: 370 commercial buildings in Portugal with data spanning
2011–2014.

Building Data Genome (BDG) Project 2 [25]: 1,636 commercial buildings across
19 sites in 2016 and 2017. We include buildings from 4 U.S. sites (Panther,
Fox, Bear, Rat).

Low Carbon London [28]: Energy consumption meter readings from 5,567 London,
UK households between 2011–2014. To keep the overall number of residential and
commercial buildings roughly the same, we keep a random sample of 713
buildings.

SMART [5]: Meter readings for 7 homes in western Massachusetts, U.S., between
2014–2016.

IDEAL [31]: Electricity meter data from 255 homes in Edinburgh, UK, between
2017–2018.

Individual household electric power consumption [15]: Energy consumption from
a single household in Sceaux, Paris between 2007–2010.

Borealis [4]: 6-second load measurement recorded for 30 homes in Waterloo, ON
in 2011–2012.

Processing and storage: We resample each time series to hourly if the data is
sub-hourly. Smart meter time series typically have missing values, sometimes
extending over weeks or months. Buildings with more than 10% of the data
missing were not included. For included buildings, we linearly interpolate
missing values, and if values are missing for a span greater than 1 week, we
fill with zeros. We also provide an option to exclude buildings with a max
hourly consumption $>$ 1.3 MW (17 from BDG-2 and 61 from Electricity) to keep
the range of consumption values similar between Buildings-900K and
BuildingsBench test data, as extrapolation is not our focus. Each annual
consumption time series per building is stored as a CSV file.

Hosting and licensing: We host the processed versions of each dataset along
with the original permissive licenses alongside Buildings-900K under the same
CC-4.0 license. Our code is open sourced under a BSD-3 license.

###  4.2 Evaluation metrics

We primarily evaluate task performance with two metrics, the normalized root
mean square error (NRMSE) and the ranked probability score (RPS). The NRMSE is
widely used as it captures the ability to predict the correct load shape. For
a target building with $M$ days of load time series,

$NRMSE:=100\times\frac{1}{\bar{y}}\sqrt{\frac{1}{24M}\sum_{j=1,i=1}^{M,24}(y_{i,j}-\hat{y}_{i,j})^{2}},$
(2)

where $\hat{y}$ is the predicted load, $y$ is the actual load, and $\bar{y}$
is the average actual load over all $M$ days. In the appendix, we also report
the normalized mean absolute error and normalized mean bias error (see
descriptions in App. D).

The RPS is a well-known metric for uncertainty quantification in probabilistic
forecasting [13]. It compares the squared error between two cumulative
distribution functions (CDFs), the predicted CDF and the observation
represented as a CDF. Define the indicator function $\mathbf{1}_{x\leq y}$ as
1 if the actual load $x$ is $\leq y$ and 0 otherwise. The continuous RPS for a
predicted CDF $\hat{F}$ for the load at hour $i$ is

$RPS:=\int_{0}^{\infty}(\hat{F}_{i}(y)-\mathbf{1}_{y_{i}\leq y})^{2}dy.$ (3)

Our platform implements the closed form RPS for Gaussian CDFs as well as a
discrete RPS for categorical distributions (used by baselines that discretize
the positive real line for token-based load forecasting, see Sec. 4.3). These
are formally defined in App. D.

###  4.3 Baselines

Various baselines are implemented and benchmarked in the BuildingsBench
platform. For zero-shot STLF, we pretrain a representative time series
transformer [42] on Buildings-900K. Transformers have recently gained
significant interest for STLF [17, 20, 33, 45]. The model is described in
brief here and with more detail in App. E.

Transformer (Gaussian): This model is the original encoder-decoder transformer
[39] re-purposed for autoregressive time series forecasting as proposed in Wu
et al. [42]. It predicts a Gaussian distribution at time $t+i$ conditioned on
the past week and the previous $i-1$ predictions. We train it to minimize the
Gaussian negative log-likelihood loss. Since our demand time series are highly
non-Gaussian (periods of low consumption interspersed with bursts), the loss
diverged during training when we used standard scaling to normalize the data.
Using a Box-Cox power transformation [6] removes this instability. To compute
the RPS, we approximate a Gaussian in the unscaled space by backprojecting the
scaled standard deviation (see App. D for approximation details).

Transformer (Tokens): We also implement a transformer variant that uses
_tokenization_ to predict discrete load tokens. This baseline allows us to
test whether quantizing loads into tokens is beneficial for large-scale
pretraining. We also found that a comparison between tokenization and Gaussian
time series transformers was missing from the literature. The model is trained
to predict a categorical distribution over a vocabulary of load tokens by
minimizing a multi-class cross-entropy loss. To quantize loads, we use a
simple strategy—faiss-gpu [22] KMeans clustering with $K$ = 8,192 fit to the
Buildings-900K training set. We merge clusters $<$10 Watts apart to obtain a
compact vocabulary size of 2,274 tokens. See App. E for analysis on our
tokenizer design.

Each transformer is pretrained jointly on residential and commercial buildings
at three different sizes: Transformer-S (3M params), Transformer-M (17M
params), and Transformer-L (160M params). Models are trained on 1 billion load
hours (162,760 gradient steps with a batch size of 256) before early stopping.
Full discussion on hyperparameters and compute are in App. E and G.

We also benchmark simple persistence on both the zero-shot STLF and transfer
learning tasks:

Previous Day: For scenarios where the load conditions change relatively
slowly, the previous day’s load is a strong baseline for day-ahead STLF [38].

Previous Week: The 24-hour load profile on the same day from the previous week
is used [17].

Ensemble: This persistence baseline computes a Gaussian distribution for each
predicted hour $t+i$ whose mean is the average load at hour $t+i$ over the
past 7 days:

$\hat{\mu}=\frac{1}{7}\sum_{j=1}^{7}x_{(t+i-24j)};\hskip
10.00002ptp(y_{t+i}|x_{t-H:t}):=\mathcal{N}\Biggl{(}\hat{\mu},\sqrt{\frac{1}{7}\sum_{j=1}^{7}(x_{(t+i-24j)}-\hat{\mu})^{2}}\Biggr{)}.$
(4)

In STLF, outperforming persistence is an indicator that a model is producing
meaningful forecasts. Standard forecasting baselines are benchmarked on the
transfer learning task by training directly on target building data:

LightGBM: The light gradient-boosting machine (LightGBM) [23] is a popular
decision-tree-based algorithm suitable for STLF [29]. We use the multi-step
forecasting implementation from skforecast [3] with 100 estimators and no max
depth.

Linear regression, DLinear: Inspired by Zeng et al. [44], we benchmark a
linear direct multi-step forecaster that regresses all 24 future values as a
weighted sum of the past 168 values. We also implement DLinear which
decomposes the time series with a moving average kernel, applies a linear
layer to each component, and then sums the two to get the final prediction.

##  5 Benchmark results

Table 3: Zero-shot STLF results. Median accuracy (NRMSE) and ranked
probability score (RPS). Lower is better. No fine-tuning is performed on any
test building. Residential NRMSEs larger than 100% occur when the
normalization factor—the average consumption per hour—is small (e.g.,
$\sim$0.5 kWh) and thus is also smaller than the RMSE.

Buildings-900K-Test (_simulated_) BuildingsBench (_real_) Commercial
Residential Commercial Residential NRMSE (%) RPS NRMSE (%) RPS NRMSE (%) RPS
NRMSE (%) RPS Persistence Ensemble 32.73 1.13 54.33 0.179 17.17 5.39 80.11
0.067 Previous Day Persistence 34.53 - 58.84 - 17.41 - 102.44 - Previous Week
Persistence 31.76 - 73.39 - 19.96 - 103.51 - Transformer (Tokens) 14.08 0.482
43.94 0.138 14.82 4.81 101.7 0.072 Transformer (Gaussian) 16.62 0.613 44.76
0.150 13.86 5.15 83.87 0.082

Table 4: BuildingsBench transfer learning results. We show median accuracy
(NRMSE) and ranked probability score (RPS). For statistical robustness, we
compute average probability of improving NRMSE due to fine-tuning: $P(X<Y)$ :=
$\frac{100}{N}\sum_{i=1}^{N}\mathbf{1}_{[X_{i}<Y_{i}]}$ where $X_{i}$ is the
pretrained + fine-tuned NRMSE for building $i$ and $Y_{i}$ is the pretrained
NRMSE. Improvement due to fine-tuning.

Commercial buildings Residential buildings NRMSE ($\%$) RPS $P(X<Y)$ NRMSE
($\%$) RPS $P(X<Y)$ Not pretrained + Not fine-tuned Persistence Ensemble 17.41
5.16 - 80.13 0.058 - Previous Day Persistence 16.98 - - 101.78 - - Previous
Week Persistence 18.93 - - 104.38 - - Not pretrained + Fine-tuned Linear
regression 47.43 - - 102.17 - - DLinear 39.22 - - 98.56 - - Transformer
(Tokens) 44.62 27.18 - 108.61 5.76 - Transformer (Gaussian) 39.26 13.24 -
94.25 0.080 - LightGBM 16.63 - - 83.97 - - Pretrained + Not fine-tuned
Transformer (Tokens) 14.46 3.98 - 98.34 0.062 - Transformer (Gaussian) 13.65
4.07 - 85.83 0.077 - Pretrained + Fine-tuned Transformer (Tokens) 13.86
(-0.60) 3.73 (-0.254) 77.5 95.55 (-2.79) 0.060 (-0.002) 77 Transformer
(Gaussian) 13.36 (-0.30) 3.50 (-0.570) 84 82.17 (-3.66) 0.061 (-0.016) 83

Here, we analyze our baselines on the two benchmark tasks guided by the
following questions:

  1. 1.

Can models pretrained on Buildings-900K generalize to real buildings?

  2. 2.

Does fine-tuning a pretrained model on limited data from a target building
lead to improved performance?

  3. 3.

How does the number of pretraining buildings affect zero-shot generalization?

  4. 4.

How does the size of the pretrained model affect zero-shot generalization?

###  5.1 Zero-shot STLF

For zero-shot STLF, models are evaluated on day-ahead forecasting for all
unseen simulated and real buildings _without fine-tuning._ Results for
Transformer-L and other baselines are shown in Table 3 aggregated by the
median over all buildings. Due to space constraints, performance profiles over
all buildings and per-dataset metrics with 95% stratified bootstrap CIs are
reported in App. H and I.

On unseen simulated buildings, the pretrained transformers outperform the
persistence baselines in accuracy and uncertainty quantification. The average
change in accuracy from simulated to real test buildings for the transformers
was _better_ (-1% NRMSE) for commercial and _worse_ (+48% NRMSE) for
residential. The best transformer outperforms the best persistence method
(Persistence Ensemble) in zero-shot accuracy on real commercial buildings. The
Persistence Ensemble has better accuracy than the pretrained models on real
residential buildings. The RPS is sensitive to the load magnitude making it
difficult to evaluate the sim-to-real gap, but the relative RPS across
baselines follows a similar trend as accuracy (see Fig. 2 for visualizations).
These results suggest synthetic pretraining is viable for commercial zero-shot
STLF—see Sec. 6.2 for more discussion on residential buildings.

###  5.2 Transfer Learning

This task evaluates fine-tuning on a single target building for which 6 months
of data has been collected. Our fine-tuning protocol is to train for a max of
25 epochs using the first 5 months and to use the last month for early
stopping with a patience of 2 epochs. To ease the computational burden of
fine-tuning a model on each building in the benchmark, we randomly sample 100
residential and 100 commercial buildings and fine-tune separately on each.
Table 4 displays the results aggregated by the median over all buildings. We
fine-tune both pretrained Transformer-L and randomly initialized transformers.
We also report performance for the pretrained model without fine-tuning.

Our results indicate that fine-tuning the pretrained transformers on limited
data is likely to improve the performance on the target building. We found
that fine-tuning all layers of both pretrained models was necessary—only fine-
tuning the model’s last layer lead to slightly decreased performance (see App.
K) Encouragingly, the best overall fine-tuned model (Transformer (Gaussian))
beats LightGBM, a strong baseline in the low-data regime, on both commercial
and residential buildings.

(a)

(b)

(c)

(d)

Figure 2: Forecast uncertainty. Ground truth time series are truncated to
previous 24 hours for visibility. Light blue lines are 10 samples from the
predicted distribution. a-b) Successful commercial building forecasts. c-d)
Failed residential building forecasts.

###  5.3 Empirical Scaling Laws

(a) Zero-shot NRMSE

(b) Zero-shot RPS

(c) Zero-shot NRMSE

(d) Zero-shot RPS

Figure 3: Empirical scaling laws for commercial buildings. Intervals are 95%
stratified bootstrap CIs. Residential results are in Appendix J. a-b) Dataset
scale vs. zero-shot performance. The trends appear to be power-laws with
diminishing returns. c-d) Model size vs. zero-shot performance.

Pretraining buildings: To characterize how size and diversity of the dataset
impacts generalization, we trained the Transformer-M (17M) models on 1K, 10K,
and 100K simulated buildings. We adjust the training hyperparameters so all
models take similar numbers of gradient steps. The Transformer (Gaussian)
model accuracy on commercial buildings roughly follows a power-law scaling,
while RPS slightly decreases between $10^{5}$ and $10^{6}$ buildings (Fig.
3(a) and Fig. 3(b)). The Transformer (Tokens) model’s NRMSE and RPS both
exhibit power-law scaling with diminishing returns. This suggests naively
increasing the dataset size may not result in large improvements. Performance
remains constant for residential buildings (see Fig. 6 in Appendix J), which
we attribute to a stronger sim-to-real distribution shift compared to
commercial buildings.

Model size: We also compare transformers of sizes S, M, and L. The NRMSE and
RPS improves from S to M but performance plateaus or decreases from the M to L
models (Fig. 3(c) and Fig. 3(d)). We suspect that auto-correlation in the time
series causes the largest models to overfit despite using sliding windows to
extract training samples and aggressive early stopping. The good performance
of the smallest Transformer (Tokens) model is likely due to training on
quantized load values. Quantization helps generative models efficiently learn
important structure in the data and ignore negligible information [34] (our
tokenizer achieves a $\sim$63% dataset compression rate). Although, this comes
at a cost of lower accuracy (Fig. 3(c)). We observe model performance on
residential buildings is roughly constant across sizes (Fig. 6 in Appendix J).

##  6 Discussion

###  6.1 Findings

Pretraining on Buildings-900K leads to good zero-shot STLF performance on real
commercial buildings. More investigation is needed to achieve similar results
for residential buildings (Sec. 6.2).

Buildings-900K pretraining + finetuning on real buildings improves STLF for
both commercial and residential buildings. Our best pretrained + finetuned
baseline outperforms LightGBM in the BuildingsBench transfer learning task.
Finetuning all model layers appears necessary to mitigate negative transfer
caused by the distribution shift between simulated pretraining and real target
data.

We observe an approximate power-law relationship with diminishing returns
between dataset scale and zero-shot STLF for commercial buildings. Performance
plateaus as model size increases, possibly due to overfitting and distribution
shifts.

Pretraining on tokenized building loads obtains better zero-shot uncertainty
quantification but lower accuracy. Tokenization also enables smaller
pretrained transformers to achieve better zero-shot STLF task performance and
stable training.

Pretraining with geospatial coordinates slightly improves generalization: Due
to space constraints, the results of an ablation where the model ignores the
building’s latitude and longitude are provided in App. K. Briefly,
improvements in accuracy were modest (0.1 - 1% NRMSE).

###  6.2 Residential STLF Challenges

Residential loads are inherently more uncertain and variable than commercial
loads because they are more sensitive to occupant behaviour and changes in
weather (e.g., temperature and humidity) [35]. This is evidenced by the
relatively large persistence NRMSEs (80.11%-103.51%) on BuildingsBench
residential loads compared to commercial loads (17.17%-19.96%). However,
rather than decreasing the value of BuildingsBench, our work enables exploring
directions such as the inclusion of weather covariates, multi-variate
formulations of STLF, and benchmarking of advanced approaches [19].

###  6.3 Limitations

While we expect BuildingsBench to stimulate research on generalizable STLF,
our framework has limitations. First, pretraining on imperfect simulated data
is fundamentally limited if deploying on real buildings is the goal. Mixing
synthetic and real data is an interesting direction to explore. Second, the
training and evaluation data is mainly representative of building energy
consumption in the northwestern hemisphere. Expanding our framework to include
data from other global regions is needed to comprehensively evaluate
generalization. Third, due to limited time, we only pretrained transformer
baselines only Buildings-900K. We will maintain a leaderboard for
BuildingsBench in our code repository for new results.

##  7 Conclusions & Future Work

In this work, we introduce BuildingsBench, which consists of a large-scale
simulated dataset and a collection of real building datasets for benchmarking
zero-shot STLF and transfer learning. Upon comparing the performance of
pretrained transformers against persistence and traditional ML-based
forecasting baselines, we observe promising results for commercial buildings
and identify areas of improvement for residential buildings.

We plan to maintain and extend BuildingsBench in the following ways. As
relevant EULP data becomes newly available we will release updated versions of
Buildings-900K. To ground improvements on benchmark tasks in a concrete
downstream application, we will look into adding a reinforcement learning task
to the benchmark for building HVAC control using STLF. Overall, we hope that
BuildingsBench will facilitate research for communities applying ML to the
built environment as well as those conducting foundational studies on large-
scale pretraining and finetuning for time series.

## Acknowledgments and Disclosure of Funding

This work was authored by the National Renewable Energy Laboratory (NREL),
operated by Alliance for Sustainable Energy, LLC, for the U.S. Department of
Energy (DOE) under Contract No. DE-AC36-08GO28308. This work was supported by
the Laboratory Directed Research and Development (LDRD) Program at NREL. The
views expressed in the article do not necessarily represent the views of the
DOE or the U.S. Government. The U.S. Government retains and the publisher, by
accepting the article for publication, acknowledges that the U.S. Government
retains a nonexclusive, paid-up, irrevocable, worldwide license to publish or
reproduce the published form of this work, or allow others to do so, for U.S.
Government purposes. The research was performed using computational resources
sponsored by the Department of Energy’s Office of Energy Efficiency and
Renewable Energy and located at the National Renewable Energy Laboratory.

The authors would like to thank Cong Feng and Alex Rybchuk for helping revise
a draft version of this manuscript, as well as Dave Biagioni for providing
valuable insights.

## References

  * Afram & Janabi-Sharifi [2014] Afram, A. and Janabi-Sharifi, F.  Theory and applications of hvac control systems–a review of model predictive control (mpc).  _Building and Environment_ , 72:343–355, 2014.
  * Amasyali & El-Gohary [2018] Amasyali, K. and El-Gohary, N. M.  A review of data-driven building energy consumption prediction studies.  _Renewable and Sustainable Energy Reviews_ , 81:1192–1205, 2018.  ISSN 13640321.  doi: 10.1016/j.rser.2017.04.095.  URL https://linkinghub.elsevier.com/retrieve/pii/S1364032117306093.
  * Amat Rodrigo & Escobar Ortiz [2023] Amat Rodrigo, J. and Escobar Ortiz, J.  skforecast, 5 2023.  URL https://github.com/JoaquinAmatRodrigo/skforecast.
  * Ardakanian et al. [2020] Ardakanian, O., Huo, S., and Keshav, S.  6-second load measurement dataset, 2020.  URL https://doi.org/10.5683/SP2/R4SVBF.
  * Barker et al. [2012] Barker, S., Mishra, A., Irwin, D., Cecchet, E., Shenoy, P., Albrecht, J., et al.  Smart*: An open data set and tools for enabling research in sustainable homes.  _SustKDD, August_ , 111(112):108, 2012.
  * Box & Cox [1964] Box, G. E. and Cox, D. R.  An analysis of transformations.  _Journal of the Royal Statistical Society: Series B (Methodological)_ , 26(2):211–243, 1964.
  * Chadoulos et al. [2021] Chadoulos, S., Koutsopoulos, I., and Polyzos, G. C.  One model fits all: Individualized household energy demand forecasting with a single deep learning model.  In _Proceedings of the Twelfth ACM International Conference on Future Energy Systems_ , pp. 466–474. ACM, 2021.  ISBN 978-1-4503-8333-2.  doi: 10.1145/3447555.3466587.  URL https://dl.acm.org/doi/10.1145/3447555.3466587.
  * Crawley et al. [2001] Crawley, D. B., Lawrie, L. K., Winkelmann, F. C., Buhl, W. F., Huang, Y. J., Pedersen, C. O., Strand, R. K., Liesen, R. J., Fisher, D. E., Witte, M. J., et al.  Energyplus: creating a new-generation building energy simulation program.  _Energy and buildings_ , 33(4):319–331, 2001\.
  * Drgoňa et al. [2020] Drgoňa, J., Arroyo, J., Figueroa, I. C., Blum, D., Arendt, K., Kim, D., Ollé, E. P., Oravec, J., Wetter, M., Vrabie, D. L., et al.  All you need to know about model predictive control for buildings.  _Annual Reviews in Control_ , 50:190–232, 2020.
  * EIA [2023] EIA.  Energy information administration may 2023 monthly energy review.  2023\.  URL https://www.eia.gov/totalenergy/data/monthly/archive/00352305.pdf.
  * Gage [1994] Gage, P.  A new algorithm for data compression.  _C Users Journal_ , 12(2):23–38, 1994.
  * Gebru et al. [2021] Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Iii, H. D., and Crawford, K.  Datasheets for datasets.  _Communications of the ACM_ , 64(12):86–92, 2021\.
  * Gneiting & Katzfuss [2014] Gneiting, T. and Katzfuss, M.  Probabilistic forecasting.  _Annual Review of Statistics and Its Application_ , 1:125–151, 2014.
  * He et al. [2022] He, Y., Luo, F., and Ranzi, G.  Transferrable model-agnostic meta-learning for short-term household load forecasting with limited training data.  _IEEE Transactions on Power Systems_ , 37(4):3177–3180, 2022.  doi: 10.1109/TPWRS.2022.3169389.
  * Hebrail & Berard [2012] Hebrail, G. and Berard, A.  Individual household electric power consumption.  UCI Machine Learning Repository, 2012.  DOI: 10.24432/C58K54.
  * Hendrycks & Gimpel [2016] Hendrycks, D. and Gimpel, K.  Bridging nonlinearities and stochastic regularizers with gaussian error linear units.  _arXiv preprint arXiv:1606.08415_ , 2016.
  * Hertel et al. [2022] Hertel, M., Ott, S., Neumann, O., Schäfer, B., Mikut, R., and Hagenmeyer, V.  Transformer neural networks for building load forecasting.  _Tackling Climate Change with Machine Learning: workshop at NeurIPS 2022_ , 2022.
  * Hong & Fan [2016] Hong, T. and Fan, S.  Probabilistic electric load forecasting: A tutorial review.  _International Journal of Forecasting_ , 32(3):914–938, 2016.  ISSN 01692070.  doi: 10.1016/j.ijforecast.2015.11.011.  URL https://linkinghub.elsevier.com/retrieve/pii/S0169207015001508.
  * Hong et al. [2014] Hong, T., Pinson, P., and Fan, S.  Global energy forecasting competition 2012.  _International Journal of Forecasting_ , 30(2):357–363, 2014.  ISSN 0169-2070.  doi: https://doi.org/10.1016/j.ijforecast.2013.07.001.  URL https://www.sciencedirect.com/science/article/pii/S0169207013000745.
  * Huy et al. [2022] Huy, P. C., Minh, N. Q., Tien, N. D., and Anh, T. T. Q.  Short-term electricity load forecasting based on temporal fusion transformer model.  _IEEE Access_ , 10:106296–106304, 2022.  doi: 10.1109/ACCESS.2022.3211941.  URL https://ieeexplore.ieee.org/document/9910162/.
  * IEA [2022] IEA.  Buildings.  2022\.  URL https://www.iea.org/reports/buildings.
  * Johnson et al. [2019] Johnson, J., Douze, M., and Jégou, H.  Billion-scale similarity search with GPUs.  _IEEE Transactions on Big Data_ , 7(3):535–547, 2019.
  * Ke et al. [2017] Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., and Liu, T.-Y.  Lightgbm: A highly efficient gradient boosting decision tree.  _Advances in neural information processing systems_ , 30, 2017.
  * Loshchilov & Hutter [2019] Loshchilov, I. and Hutter, F.  Decoupled weight decay regularization.  In _International Conference on Learning Representations_ , 2019.
  * Miller et al. [2020] Miller, C., Kathirgamanathan, A., Picchetti, B., Arjunan, P., Park, J. Y., Nagy, Z., Raftery, P., Hobson, B. W., Shi, Z., and Meggers, F.  The building data genome project 2, energy meter data from the ASHRAE great energy predictor III competition.  _Scientific Data_ , 7(1):368, 2020.  ISSN 2052-4463.  doi: 10.1038/s41597-020-00712-x.  URL https://www.nature.com/articles/s41597-020-00712-x.
  * Mocanu et al. [2016] Mocanu, E., Nguyen, P. H., Gibescu, M., and Kling, W. L.  Deep learning for estimating building energy consumption.  _Sustainable Energy, Grids and Networks_ , 6:91–99, 2016\.
  * National Renewable Energy Lab [2023] National Renewable Energy Lab.  Energyplus.  2023\.  URL https://energyplus.net/.
  * Networks [2018] Networks, U. P.  Smartmeter energy consumption data in london households, 2018.  URL https://data.london.gov.uk/dataset/smartmeter-energy-use-data-in-london-households.
  * Park et al. [2021] Park, S., Jung, S., Jung, S., Rho, S., and Hwang, E.  Sliding window-based lightgbm model for electric load forecasting using anomaly repair.  _The Journal of Supercomputing_ , 77(11):12857–12878, Nov 2021.  ISSN 1573-0484.  doi: 10.1007/s11227-021-03787-4.  URL https://doi.org/10.1007/s11227-021-03787-4.
  * Paszke et al. [2019] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S.  PyTorch: An Imperative Style, High-Performance Deep Learning Library.  In _Advances in Neural Information Processing Systems 32_ , pp. 8024–8035. Curran Associates, Inc., 2019.  URL http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf.
  * Pullinger et al. [2021] Pullinger, M., Kilgour, J., Goddard, N., Berliner, N., Webb, L., Dzikovska, M., Lovell, H., Mann, J., Sutton, C., Webb, J., et al.  The ideal household energy dataset, electricity, gas, contextual sensor data and survey data for 255 uk homes.  _Scientific Data_ , 8(1):146, 2021.
  * Radford et al. [2018] Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., et al.  Improving language understanding by generative pre-training.  2018\.
  * Rathnayaka et al. [2022] Rathnayaka, P., Moraliyage, H., Mills, N., De Silva, D., and Jennings, A.  Specialist vs generalist: A transformer architecture for global forecasting energy time series.  In _2022 15th International Conference on Human System Interaction (HSI)_ , pp. 1–5. IEEE, 2022.
  * Razavi et al. [2019] Razavi, A., Van den Oord, A., and Vinyals, O.  Generating diverse high-fidelity images with vq-vae-2.  _Advances in neural information processing systems_ , 32, 2019.
  * Stephen et al. [2015] Stephen, B., Tang, X., Harvey, P. R., Galloway, S., and Jennett, K. I.  Incorporating practice theory in sub-profile models for short term aggregated residential load forecasting.  _IEEE Transactions on Smart Grid_ , 8(4):1591–1598, 2015.
  * Street [2021] Street, P.  Pecan street dataport, 2021.  URL https://www.pecanstreet.org/dataport.
  * Trindade [2015] Trindade, A.  ElectricityLoadDiagrams20112014.  UCI Machine Learning Repository, 2015.  DOI: 10.24432/C58C86.
  * Tsakoumis et al. [2002] Tsakoumis, A., Vladov, S., and Mladenov, V.  Daily load forecasting based on previous day load.  In _6th Seminar on Neural Network Applications in Electrical Engineering_ , pp. 83–86, 2002.  doi: 10.1109/NEUREL.2002.1057973.
  * Vaswani et al. [2017] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I.  Attention is all you need.  _arXiv:1706.03762 [cs]_ , 2017.  URL http://arxiv.org/abs/1706.03762.
  * Wei et al. [2017] Wei, T., Wang, Y., and Zhu, Q.  Deep reinforcement learning for building hvac control.  In _Proceedings of the 54th annual design automation conference 2017_ , pp. 1–6, 2017.
  * Wilson et al. [2021] Wilson, E. J., Parker, A., Fontanini, A., Present, E., Reyna, J. L., Adhikari, R., Bianchi, C., CaraDonna, C., Dahlhausen, M., Kim, J., et al.  End-use load profiles for the us building stock: Methodology and results of model calibration, validation, and uncertainty quantification.  Technical report, National Renewable Energy Lab (NREL), 2021.  URL https://www.nrel.gov/docs/fy22osti/80889.pdf.
  * Wu et al. [2020] Wu, N., Green, B., Ben, X., and O’Banion, S.  Deep transformer models for time series forecasting: The influenza prevalence case.  _arXiv:2001.08317_ , 2020.  URL http://arxiv.org/abs/2001.08317.
  * Xu & Meng [2020] Xu, X. and Meng, Z.  A hybrid transfer learning model for short-term electric load forecasting.  _Electrical Engineering_ , 102(3):1371–1381, 2020\.  doi: 10.1007/s00202-020-00930-x.  URL http://link.springer.com/10.1007/s00202-020-00930-x.
  * Zeng et al. [2022] Zeng, A., Chen, M., Zhang, L., and Xu, Q.  Are transformers effective for time series forecasting?  _arXiv preprint arXiv:2205.13504_ , 2022.
  * Zhang et al. [2022a] Zhang, G., Wei, C., Jing, C., and Wang, Y.  Short-term electrical load forecasting based on time augmented transformer.  _International Journal of Computational Intelligence Systems_ , 15(1):67, 2022a.
  * Zhang et al. [2021] Zhang, L., Wen, J., Li, Y., Chen, J., Ye, Y., Fu, Y., and Livingood, W.  A review of machine learning in building load prediction.  _Applied Energy_ , 285:116452, 2021.  ISSN 03062619.  doi: 10.1016/j.apenergy.2021.116452.  URL https://linkinghub.elsevier.com/retrieve/pii/S0306261921000209.
  * Zhang et al. [2022b] Zhang, Y., Park, D. S., Han, W., Qin, J., Gulati, A., Shor, J., Jansen, A., Xu, Y., Huang, Y., Wang, S., et al.  Bigssl: Exploring the frontier of large-scale semi-supervised learning for automatic speech recognition.  _IEEE Journal of Selected Topics in Signal Processing_ , 16(6):1519–1532, 2022b.

## Appendix

## Table of Contents

  * •

Appendix A: Details of hosting, licensing, and maintenance

  * •

Appendix B: Dataset datasheet

  * •

Appendix C: BuildingsBench benchmark datasets details

  * •

Appendix D: Details of evaluation metrics

  * •

Appendix E: Time series transformer details

  * •

Appendix F: Training details

  * •

Appendix G: Compute details

  * •

Appendix H: Benchmark performance profiles

  * •

Appendix I: Per-dataset results

  * •

Appendix J: Detailed residential building results

  * •

Appendix K: Ablation studies

  * •

Appendix L: Author responsibility statement

  * •

Appendix M: Additional qualitative results

##  Appendix A Hosting, Licensing, and Maintenance

Our datasets and code are available via the following links:

  * •

Github: https://github.com/NREL/BuildingsBench

  * •

Documentation: https://nrel.github.io/BuildingsBench

  * •

Datasets and Website: https://data.openei.org/submissions/5859

  * •

Tutorials: https://nrel.github.io/BuildingsBench/tutorials/

  * •

Dataset DOI: https://doi.org/10.25984/1986147

As described in Sec. 3 and Sec. 4, Buildings-900K and the BuildingsBench
benchmark datasets are available for download under a CC-4.0 license and our
code is available under a BSD 3-Clause license. We ensure the long-term
availability and maintenance of the data by hosting it on the Open Energy Data
Initiative (OEDI) platform (https://data.openei.org/about). The OEDI is a
centralized repository for storing energy-related datasets derived from U.S.
Department of Energy projects.

##  Appendix B Datasheet for Buildings-900K

Questions from Datasheet for Datasets (v8) [12].

###  B.1 Motivation

Q: For what purpose was the dataset created?

This dataset was created to research large-scale pretraining of models for
short-term load forecasting (STLF). It specifically addresses a lack of
appropriately sized and diverse datasets for pretraining STLF models.
Buildings-900K, which consists of simulated building energy consumption time
series, is derived from the National Renewable Energy Lab (NREL) End-Use Load
Profiles (EULP) database. We emphasize that the EULP was _not_ originally
developed for studying STLF. Rather, it was developed as a general resource to
"…help electric utilities, grid operators, manufacturers, government entities,
and research organizations make critical decisions about prioritizing research
and development, utility resource and distribution system planning, and state
and local energy planning and regulation." [41].

Q: Who created the dataset (e.g., which team, research group) and on behalf of
which entity (e.g., company, institution, organization)?

Researchers at NREL created Buildings-900K. Consent from the NREL developers
of the EULP project was obtained in writing to extract, process, and
redistribute a subset of the EULP. The EULP is a multi-year multi-instiutional
collaboration lead by NREL and is publicly available for download with a
permissive CC-4.0 license.

Q: Who funded the creation of the dataset?

This dataset was developed with funding provided by Laboratory Directed
Research and Development (LDRD) at NREL.

###  B.2 Composition

Q: What do the instances that comprise the dataset represent (e.g., documents,
photos, people, countries)?

Annual hourly energy consumption of EnergyPlus-simulated buildings.

Q: How many instances are there in total (of each type, if appropriate)?

There are $\sim$1.8 million annual time series instances (900K buildings
$\times$ 2 years).

Q: Does the dataset contain all possible instances or is it a sample (not
necessarily random) of instances from a larger set?

Buildings-900K consists of a subset of the EULP database. Specifically, we
extract only the out.site_energy.total.energy_consumption time series from
each upgrade=0 building in the 2021 version for weather-years AMY2018 and
TMY3. The EULP contains additional end-use time series for each building, the
EnergyPlus XML model files, the EnergyPlus weather files, and a variety of
additional simulations under various electrification scenarios. We did not
include these additional data in the initial version of Buildings-900K,
although we may consider adding them in the future.

Q: What data does each instance consist of?

Each instance is stored in a Parquet file and has timestamp information and
load values (in kWh). We also have spatial metadata that maps each building to
a U.S. county.

Q: Is any information missing from individual instances?

No.

Q: Are relationships between individual instances made explicit (e.g., users’
movie ratings, social network links)?

Each building has a unique identifier and is assigned to a county. That county
(or Public Use Microdata Area—PUMA) has its own unique identifier. Each PUMA
contains many buildings that are geographically close to each other.

Q: Are there recommended data splits (e.g., training, development/validation,
testing)?

  * •

For test: We withhold all buildings in 4 counties from both AMY2018 and TMY3
weather-years.

  * •

For validation: We withhold dates 2018-12-17 to 2018-12-31 from all buildings
in the training set to use as held-out validation data (e.g., for early
stopping).

  * •

For pretraining: All remaining data is used for pretraining.

Q: Are there any errors, sources of noise, or redundancies in the dataset?

No.

Q: Is the dataset self-contained, or does it link to or otherwise rely on
external resources (e.g., websites, tweets, other datasets)?

It is self-contained.

Q: Does the dataset contain data that might be considered confidential (e.g.,
data that is protected by legal privilege or by doctor-patient
confidentiality, data that includes the content of individuals’ non-public
communications)?

No. All simulated building models are created by sampling from conditional
distributions over attributes calibrated to real data.

Q: Does the dataset contain data that, if viewed directly, might be offensive,
insulting, threatening, or might otherwise cause anxiety?

No.

Q: Does the dataset relate to people?

No.

###  B.3 Collection process

Q: How was the data associated with each instance acquired?

Buildings-900K instances are the outputs of extensively calibrated and
validated EnergyPlus simulations. These simulations were run by the team
developing the EULP on high performance computing resources at NREL.

Q: What mechanisms or procedures were used to collect the data (e.g., hardware
apparatus or sensor, manual human curation, software program, software API)?

See above.

Q: If the dataset is a sample from a larger set, what was the sampling
strategy (e.g., deterministic, probabilistic with specific sampling
probabilities)?

Buildings-900K consists of a hand-selected subset of the EULP database.

Q: Who was involved in the data collection process (e.g., students,
crowdworkers, contractors) and how were they compensated (e.g., how much were
crowdworkers paid)?

N/A

Q: Over what timeframe was the data collected?

Buildings-900K consists of simulated data with representative weather from
2018.

Q: Were any ethical review processes conducted (e.g., by an institutional
review board)?

No.

###  B.4 Preprocessing/cleaning/labeling

Q: Was any preprocessing/cleaning/labeling of the data done (e.g.,
discretization or bucketing, tokenization, part-of-speech tagging, SIFT
feature extraction, removal of instances, processing of missing values)?

The steps we followed to create Buildings-900K are as follows:

1\. Select the out.site_energy.total.energy_consumption time series from each
AMY2018 and TMY3 building.

2\. Aggregate the 15-minute resolution energy consumption to hourly.

3\. Join the time series for all buildings in the same PUMA into a single
Parquet table.

4\. Save each PUMA-level Parquet table (compressed with Snappy).

Q: Was the “raw” data saved in addition to the preprocessed/cleaned/labeled
data (e.g., to support unanticipated future uses)?

The raw data (the EULP) is publicly accessible for download
https://data.openei.org/s3_viewer?bucket=oedi-data-lake&prefix=nrel-pds-
building-stock.

Q: Is the software used to preprocess/clean/label the instances available?

Yes, at https://github.com/NREL/BuildingsBench.

###  B.5 Uses

Q: Has the dataset been used for any tasks already?

In this work, we evaluate pretrained models on zero-shot STLF and transfer
learning for STLF (fine-tuning pretrained models on limited data from a target
building).

Q: Is there a repository that links to any or all papers or systems that use
the dataset?

We will add links to papers or systems that use the dataset to
https://data.openei.org/submissions/5859 and
https://nrel.github.io/BuildingsBench.

Q: What (other) tasks could the dataset be used for?

Pretrained models can sometimes serve as general-purpose feature extractors.
These features can be used for a wide range of tasks, including:

  * •

Predicting energy consumption of a building based on its characteristics.

  * •

Classifying user behavior (e.g., whether a building is currently occupied)
based on their energy consumption.

  * •

Classifying building type based on energy consumption.

  * •

Load disaggregation, that is, predicting the energy consumption of individual
appliances based on the total energy consumption of a building.

Among others, potentially. We did not explore these tasks in this paper.

Q: Is there anything about the composition of the dataset or the way it was
collected and preprocessed/cleaned/labeled that might impact future uses?

Buildings-900K is (in its current state) representative of buildings in the
United States. While we find evidence that models pretrained on Buildings-900K
generalize to commercial buildings in other countries in the northwestern
hemispheres, such as the UK and Canada, it may not be representative of
building energy consumption patterns in other regions of the world.

Q: Are there tasks for which the dataset should not be used?

None that the authors are currently aware of.

###  B.6 Distribution

Q: Will the dataset be distributed to third parties outside of the entity
(e.g., company, institution, organization) on behalf of which the dataset was
created?

Yes.

Q: How will the dataset will be distributed (e.g., tarball on website, API,
GitHub)?

It will be publicly available for download on the Open Energy Data Initiative
(OEDI) website at https://data.openei.org/submissions/5859.

Q: ill the dataset be distributed under a copyright or other intellectual
property (IP) license, and/or under applicable terms of use (ToU)?

CC-4.0.

Q: Have any third parties imposed IP-based or other restrictions on the data
associated with the instances?

No.

Q: Do any export controls or other regulatory restrictions apply to the
dataset or to individual instances?

No.

###  B.7 Maintenance

Q: Who is supporting/hosting/maintaining the dataset?

Code is hosted and publicly available on Github. NREL will maintain the code
and datasets. The data is hosted on the Open Energy Data Initiative site,
which is a centralized repository for energy-related datasets derived from
U.S. Department of Energy projects.

Q: How can the owner/curator/manager of the dataset be contacted (e.g., email
address)?

The lead maintainer at NREL is Patrick Emami (Patrick.Emami@nrel.gov).

Q: Will the dataset be updated (e.g., to correct labeling errors, add new
instances, delete instances)?

Yes. A version naming system will be used to indicate updated versions. We are
also maintaining a GitHub repository for the associated benchmark
(BuildingsBench) where notifications of updates will be posted.

Q: Will older versions of the dataset continue to be
supported/hosted/maintained?

Yes, older versions will remain available on the OEDI website.

Q: If others want to extend/augment/build on/contribute to the dataset, is
there a mechanism for them to do so?

Not officially, but our benchmark code is open source and pull requests are
welcome.

##  Appendix C BuildingsBench Datasets

###  C.1 ElectricityLoadDiagrams20112014

This dataset is available from the UCI Machine Learning
Repository111https://archive.ics.uci.edu/dataset/321/electricityloaddiagrams20112014
under a CC-4.0 license. It contains 15-minute consumption time series (in kW)
for 370 clients in Portugal from 2011 to 2014. The magnitudes of the loads are
much larger than residential homes, thus, we consider these time series as
commercial buildings. After filtering for missing data, we kept 359 buildings.
We included this dataset in BuildingsBench because of its permissive license,
popular use in machine-learning-based time series forecasting studies [44],
and size.

###  C.2 The Building Data Genome Project 2

This dataset is available222https://github.com/buds-lab/building-data-genome-
project-2/ under an MIT License. It consists of measurements from 3,053 meters
from 1,636 commercial buildings over two years (2016 and 2017). One or more
meters per building measured the total electrical, heating and cooling water,
steam, solar energy, water, and irrigation usage. This dataset was curated by
the _Building Data Genome Project_ , a consortium of academics and
practitioners. We use the whole building electricity meter measurements from
Bear, Fox, Panther and Rat sites, totalling 611 buildings (from the CSV file
electricity_cleaned.csv). This dataset was included in BuildingsBench because
of its permissive license, size, and diversity.

###  C.3 Low Carbon London

This dataset is available333https://data.london.gov.uk/dataset/smartmeter-
energy-use-data-in-london-households under a CC-4.0 license. It consists of
consumption measurements from 5,567 London households that participated in a
project led by UK Power Networks between November 2011 and February 2014. The
dataset consists of half-hourly consumption (in kWh), a unique household
identifier, and timestamps. Due to the large number of households included in
the dataset, we randomly subsampled 713 of them (so as to keep the overall
number of residential and commercial buildings in the benchmark approximately
similar). We included this dataset because of its permissive license and size.

###  C.4 SMART

This dataset is available online unaccompanied by a specific
license444https://traces.cs.umass.edu/index.php/smart/smart. It contains meter
measurements of aggregate electricity data for 7 residential homes (which are
called Home A,B,C,D,F,G,H) collected by researchers at UMass Amherst. After
filtering out 2 homes due to missing data, we were left with Home B (years
2014-2016), Home C (2014-2016), Home D (2016), Home F (2014-2016), and Home G
(2016). We included this dataset in BuildingsBench due to its accessibility
and because the homes are located in the U.S. (Western Massachusetts), are are
the simulated homes in Buildings-900K.

###  C.5 IDEAL

This dataset is available555https://datashare.ed.ac.uk/handle/10283/3647 under
a CC-4.0 license. It has meter readings for electric and gas usage, as well as
room temperature, humidity, and boiler readings from 255 UK homes. We use the
total home electricity measurements. The measurements were collected by
researchers at the University of Edinburgh. After our filtering process, we
kept 219 homes. This dataset was included in BuildingsBench due to its
permissive license and size.

###  C.6 Individual household electric power consumption (Sceaux)

This dataset is available via
Kaggle666https://www.kaggle.com/datasets/uciml/electric-power-consumption-
data-set under a DbCL-1.0 license. It contains consumption measurements from a
home in Sceaux, France between December 2006 and November 2010. It provides 7
types of meter readings at one minute resolution with timestamps. We use the
global active power readings. This dataset was included in BuildingsBench due
to its permissive license and due to its popularity both in the literature
[26] and on Kaggle.

###  C.7 Borealis

This dataset is
available777https://borealisdata.ca/dataset.xhtml?persistentId=doi:10.5683/SP2/R4SVBF
under a CC0-1.0 license. It is 6-second resolution measurements of total real
power consumption for 25 households in Waterloo, ON for at least three months
between 2011 and 2012. After filtering out homes with excessive missing data,
we kept 15 of them. The data was collected by researchers at the University of
Waterloo. We included this dataset in BuildingsBench due to its permissive
license and size.

##  Appendix D Evaluation Metrics

###  D.1 Additional quantitative metrics: NMAE, NMBE

NMAE: Similar to the NRMSE, the normalized mean absolute error (NMAE) measures
the ability of a model to predict the correct load shape. However, it is less
sensitive to large errors than the NRMSE. For a building with $M$ days of load
time series,

$NMAE:=100\times\frac{1}{\bar{y}}\left[\frac{1}{24M}\sum_{j=1,i=1}^{M,24}|y_{i,j}-\hat{y}_{i,j}|\right].$
(5)

NMBE: The normalized mean bias error (NMBE) is informative about a model’s
tendency to over or under-estimate building loads, which is practically useful
to measure. However, a model that tends to over-estimate or under-estimate the
load by an equal amount can achieve an NMBE close to zero (i.e., positive and
negative errors cancel). Therefore, it is not a useful indicator of
_accuracy_. For a building with $M$ days of load time series,

$NMBE:=100\times\frac{1}{\bar{y}}\left[\frac{1}{24M}\sum_{j=1,i=1}^{M,24}(y_{i,j}-\hat{y}_{i,j})\right].$
(6)

We visualize performance profiles and dataset-specific results for all metrics
in App. H and App. I, respectively.

###  D.2 Categorical ranked probability score

Here, we define the ranked probability score (RPS) for categorical
distributions. This is used to compute the RPS for our time series transformer
variant that predicts a categorical distribution over a vocabulary of
$|\mathcal{V}|$ discretized load tokens. Each load token is the centroid of a
cluster, where clusters are estimated via KMeans. See App. E for more details
about the tokenizer.

In what follows, we assume that the vocabulary of load tokens is sorted in
increasing order by value, that is, $0\leq
v_{0}<v_{1}<\dots<v_{|\mathcal{V}|-1}$, $v_{i}\in\mathcal{V}$.

Let $p(\hat{Y}_{i,j})$ be the predicted categorical distribution over
$\mathcal{V}$ for hour $i$ on day $j$. We assume this is a normalized
distribution, which is accomplished in practice by computing the softmax of
the model’s predicted logits. The corresponding CDF for $p(\hat{Y}_{i,j})$ is
$F(\hat{Y}_{i,j})$, which is given by the cumulative sum of the
$|\mathcal{V}|$ sorted entries of $p(\hat{Y}_{i,j})$. Then, let $F(Y_{i,j})$
be the CDF of the ground truth discretized load. If the ground truth category
is $k\in\\{0,\dots,|\mathcal{V}|-1\\}$, the CDF $F(Y_{i,j})$ is 0 up to index
$k-1$ and 1 thereafter (resembling a step function that goes from 0 to 1 at
ground truth load index $k$). Finally, define $\bar{v}[k]$ as the difference
between the maximum load value and the minimum load value assigned to load
token $v[k]$ by KMeans (recall, load tokens are centroids of 1D KMeans
clusters).

Then, the categorical RPS at hour $i$ for a building with $M$ days of load
time series is

$CatRPS_{i}:=\frac{1}{M}\sum_{j=1}^{M}\sum_{k=0}^{|\mathcal{V}|-1}(F(\hat{Y}_{i,j})[k]-F(Y_{i,j})[k])^{2}\bar{v}[k].$
(7)

###  D.3 Gaussian ranked probability score

The _continuous_ ranked probability score for Gaussian distributions can be
evaluated in closed form. In particular, let $\hat{\mu}_{i,j}$ and
$\hat{\sigma}_{i,j}$ be a predicted mean and standard deviation, and $y_{i,j}$
be the ground truth value. Define
$z_{i,j}=\frac{y_{i,j}-\hat{\mu}_{i,j}}{\hat{\sigma}_{i,j}}$. Then, the
Gaussian (C)RPS at hour $i$ for a building with $M$ days of load time series
is

$\displaystyle
GaussCRPS_{i}:=\frac{1}{M}\sum_{j=1}^{M}\hat{\sigma}_{i,j}\Biggl{[}z_{i,j}\Biggl{(}2\underbrace{\Biggl{(}\frac{1+\operatorname{erf}(z_{i,j})}{2\sqrt{2}}\Biggr{)}}_{\text{Gaussian
CDF}}-1\Biggr{)}+2\underbrace{\Biggl{(}\frac{\exp{\frac{-z_{i,j}^{2}}{2}}}{\sqrt{2\pi}}\Biggr{)}}_{\text{Gaussian
PDF}}-\frac{1}{\sqrt{\pi}}\Biggr{]}.$ (8)

Gaussian approximation of the inverse Box-Cox transform: Our time series
transformer variant that predicts a Gaussian distribution for each hour $i$ in
the day-ahead forecast uses a Box-Cox power transformation to normalize the
load values. In App. K, we show that standard scaling was insufficient to
remove training instabilities caused by the non-Gaussian load time series.
Since the Gaussian distribution is learned in the _Box-Cox transformed space_
, we cannot directly use the GaussCRPS metric to evaluate uncertainty
quantification _in the original un-scaled space_ (i.e., with units of kWh).
Our current solution, introduced here, is to compute a Gaussian distribution
in the original un-scaled space that closely approximates the distribution
given by the inverse Box-Cox transformation (for a reasonable range of
standard deviations). We leave exploring advanced non-approximate solutions
for future work.

First, we define the Box-Cox transformation of a random variable $X$

$Y=f_{\lambda}(X):=\begin{cases}\frac{X^{\lambda}-1}{\lambda}\hskip
10.00002pt\text{if }\lambda\neq 0\\\ \ln(X)\hskip 10.00002pt\text{if
}\lambda=0,\end{cases}$ (9)

and its inverse

$X=f_{\lambda}^{-1}(Y):=\begin{cases}(Y\lambda+1)^{\frac{1}{\lambda}}\hskip
10.00002pt&\text{if }\lambda\neq 0\\\ \exp(Y)\hskip 10.00002pt&\text{if
}\lambda=0.\end{cases}$ (10)

We estimate a Gaussian distribution in the un-scaled space that _approximates_
the (power-normal) distribution of the samples inverted with
$f_{\lambda}^{-1}$. We take the inverse of the predicted mean with
$f_{\lambda}^{-1}(\hat{\mu}_{i,j})$. Then, the standard deviation is estimated
with:

$\displaystyle\hat{\sigma}^{+}_{i,j}$
$\displaystyle=f_{\lambda}^{-1}(\hat{\mu}_{i,j}+\hat{\sigma}_{i,j})-f_{\lambda}^{-1}(\hat{\mu}_{i,j})$
(11) $\displaystyle\hat{\sigma}^{-}_{i,j}$
$\displaystyle=f_{\lambda}^{-1}(\hat{\mu}_{i,j})-f_{\lambda}^{-1}(\hat{\mu}_{i,j}-\hat{\sigma}_{i,j})$
(12) $\displaystyle\tilde{\sigma}_{i,j}$
$\displaystyle\approx\frac{\hat{\sigma}^{+}_{i,j}+\hat{\sigma}^{-}_{i,j}}{2}.$
(13)

The Gaussian distribution in the un-scaled space is thus
$\mathcal{N}(f_{\lambda}^{-1}(\hat{\mu}_{i,j}),\tilde{\sigma}_{i,j})$. This
Gaussian approximates the power-normal given by the inverse Box-Cox transform
for reasonable values of $\tilde{\sigma}_{i,j}$ (Fig. 4). However, when the
model is very uncertain about its prediction and thus the standard deviation
in the scaled space is large, the power-normal in the un-scaled space is
heavily skewed-right, making the Gaussian a poor approximation (Fig. 4(c)).

(a)

(b)

(c)

Figure 4: Gaussian approximation of the inverse Box-Cox. Visualization of the
Gaussian distribution
$\mathcal{N}(f^{-1}(\hat{\mu}_{i,j}),\tilde{\sigma}_{i,j})$ (orange) in the
un-scaled space of loads (in kWh) for computing the RPS when using Box-Cox
scaling. For reasonably large standard deviations $\tilde{\sigma}$ in the un-
scaled space, this Gaussian is a reasonable approximation of the power-normal
distribution given by the inverse Box-Cox (blue). c) When the model is highly
uncertain so that $\tilde{\sigma}$ is large, the power-normal is extremely
right-skewed. In this case, the Gaussian approximation is inaccurate.

##  Appendix E Time Series Transformer Details

Table 5: Transformer architecture hyperparameters.

Predicted Distribution # Layers # Heads Embedding Dim MLP Dim # Params
Transformer-S Categorical 2 4 256 512 3.3M Transformer-M Categorical 3 8 512
1024 17.2M Transformer-L Categorical 12 12 768 2048 160.7M Transformer-S
Gaussian 2 4 256 512 2.6M Transformer-M Gaussian 3 8 512 1024 15.8M
Transformer-L Gaussian 12 12 768 2048 160.7M

In this section, we provide additional details about the time series
transformer architecture used in this work. We use the original encoder-
decoder model from Vaswani et al. [39] as implemented in the PyTorch
nn.Transformer module. This autoregressive model is trained to predict the
load at each time step with teacher forcing. Following Wu et al. [42], we use
masking in the decoder to prevent the attention from peeking at the target
value at each time step. The first value in the input sequence passed to the
decoder is the last value of the input sequence passed to the encoder. The
Gaussian time series transformer uses a linear layer (with parameters shared
across time steps) to transform the high-dimensional output of the decoder
into a mean and standard deviation at each time step. The time series
transformer trained on tokenized load values uses a linear layer to transform
the decoder outputs into $|\mathcal{V}|$ = 2,274 logits for the cross-entropy
loss, also shared across time steps.

The architecture hyperparameters for the -L, -M, and -S models are shown in
Table 5. The largest model has 12 layers, 12 heads, and an internal dimension
of 768 selected to match Radford et al. [32], but a smaller feedforward
dimension of 2048 to keep the number of parameters at approximately $10^{8}$.
We downscale the architecture hyperparameters per variant to reduce the number
of parameters by roughly an order of magnitude. Similar to Radford et al.
[32], all models use GELU [16] activations and $\mathcal{N}(0,0.02)$ weight
initialization.

Model input encoding: Let $s$ be a scaling factor given by Embedding Dim$/256$
(e.g., $s$ = 3 for Transformer-L). The model inputs are encoded, concatenated,
and passed to the transformer encoder as follows:

  * •

day of year, day of week, hour of day: These are encoded into a 2-dim space
with $\sin(\pi x)$ and $\cos(\pi x)$ then projected with a linear layer to
$32s$-dim.

  * •

latitude, longitude: These scalars are each projected to $32s$-dim space with
linear layers.

  * •

building type: A linear embedding layer is used to learn a projection for each
of residential and commercial building types to $32s$-dim space.

  * •

continuous loads: When the load is a continuous scalar, it is projected to
$64s$-dim space with a linear layer.

  * •

discrete load tokens: When the load is a discrete token, a linear embedding
layer is used to learn a projection for each of the $|\mathcal{V}|$ tokens to
$64s$-dim space.

These various embeddings are concatenated into a feature vector matching the
Embedding Dim of the model for each time step.

###  E.1 Load tokenizer

Algorithm 1 Load tokenizer

// Inputs: loads ((n,1) array), $K$ initial centroids, $\tau$ = 0.01 threshold
(kWh)

kmeans = faiss.Kmeans($K$)

kmeans.train(loads)

// Merge step. Initialize state.

sorted_centroids = sort(kmeans.centroids)

current_centroid = sorted_centroids[0]

merged_centroids = []

temp_centroids = [current_centroid]

// Iterate over sorted centroids in increasing order

for $i=1$ to $K$ do

// The next centroid is less than $\tau$ away, merge it

if sorted_centroids[$i$] - current_centroid $<\tau$ then

temp_centroids.append(sorted_centroids[$i$])

else

merged_centroids.append(mean(temp_centroids))

temp_centroids = [sorted_centroids[$i$]]

current_centroid = sorted_centroids[$i$] // Add the last centroid to the new
list of merged centroids

merged_centroids.append(mean(temp_centroids))

return kmeans, merged_centroids

We explore quantizing load values into a vocabulary of discrete tokens to
closely mimic the application of transformers to natural language. The design
of the tokenizer is detailed here. We use simple KMeans clustering with a
basic merging strategy inspired by Byte Pair Encoding [11] which merges
clusters that are overly close to each other. We leave exploring other simple
and sophisticated tokenization strategies for future work.

The tokenizer uses KMeans clustering on a random subsample of load values from
the Buildings-900K training set to fit $K$ clusters (the size of the subsample
is chosen internally by faiss-gpu). The merge step is as follows. First, we
sort the $K$ cluster centroids in increasing order and then iterate over them.
For any cluster centroid $i$, if there are successive centroids that are less
than $\tau=0.01$ kWh greater than $i$, we replace all of these centroids with
a new centroid equal to their average. We provide pseudocode in Alg. 1. To
tokenize a continuous load value, we first look up the index of the original
centroid it belongs to, then use this index to look up the corresponding index
in the list of merged centroids. See Fig. 5 for a visual comparison of three
different vocabularies produced by the tokenizer, and App. K.1 for ablation
study results on the choice of $K$ and the merging step.

(a) $|\mathcal{V}|$ = 2,274 ($K$ = 8,192)

(b) $|\mathcal{V}|$ = 8,192 (No merging)

(c) $|\mathcal{V}|$ = 344 ($K$ = 512)

Figure 5: Load token frequency. We visualize histograms of token frequencies
from a subsample of Buildings-900K. a) After merging $K$ = 8,192 clusters into
2,274, there are few tokens that appear often (for loads near zero), with most
other tokens having moderate usage. b) Without merging, most tokens correspond
to small load values and appear to have roughly equal usage. c) When
$|\mathcal{V}|$ is small (344), there are few tokens for large load values.
The mean absolute quantization error for commercial buildings is approximately
0.34 kWh whereas for $|\mathcal{V}|$ = 2,274 it is 0.04 kWh.

##  Appendix F Training Details

Pretraining: We aimed to adopt hyperparameters from well-known transformer
architectures and thus did not spend much time optimizing them ourselves. The
transformers are trained with a batch size of 256 and AdamW [24] using
$\beta_{1}$ = 0.9, $\beta_{2}$ = 0.98, $\epsilon$ = 1e-9, and weight decay of
0.01 following Radford et al. [32]. The learning rate uses a linear warmup of
10K steps up to 6e-5 followed by cosine decay to zero. Models are trained for
162,760 steps, or about 1 billion load hours, after which the validation loss
no longer decreased.

Fine-tuning: We fine-tune all layers of the transformers (see an ablation
study on this in App. K.4) with a lower learning rate of 1e-6. We allow a
maximum of 25 fine-tuning epochs with an early stopping patience of 2.

##  Appendix G Compute Details

  * •

Buildings-900K processing: Creating the Buildings-900K dataset by processing a
subset of the EULP was accomplished with a 96-core AWS EC2 instance in 3-4
days.

  * •

Model pretraining: The largest model variant, the Transformer-L, was trained
on 1 billion hours from Buildings-900K with one NVIDIA 40GB A100 GPU in 24
hours. Each mini-batch of size 256 has 256 $\times$ 24 = 6,144 hours of loads.
Smaller model variants train faster, in about 18 hours. We use PyTorch
automatic mixed precision training with gradient scaling, which is crucial for
achieving fast training times.

  * •

Zero-shot STLF: The amount of time to run each benchmark task varies depending
on model inference speed. This task uses all available years for every
building in the benchmark. For the largest model variant, it takes about 2-3
hours on one NVIDIA 40GB A100. By contrast, the persistence baselines run in
well under one hour.

  * •

Transfer learning: Repeating the fine-tuning process for every building in the
benchmark can be computationally demanding. For example, we sub-sample 200
total buildings (100 residential and 100 commercial) to use for this task, and
this takes 1.5 - 3.5 hours to complete on one NVIDIA 40GB A100. We estimate it
would take over 15 hours to include all buildings in this task. Models that
are fine-tuned from frozen pretrained weights finish this task faster, as we
stop the fine-tuning process with early stopping using a patience of 2 epochs.
However, we allow baselines that are fine-tuned from randomly initialized
weights a maximum of 100 fine-tuning epochs per building. Fine-tuning
typically does not end early due to early stopping, and these runs tend to
occupy the higher end of the time range.

##  Appendix H Performance Profiles

Performance profiles plot the fraction of all buildings with performance
greater than a threshold $\tau$ for a range of threshold values. Here we show
performance profiles for commercial (Fig. 10, Fig. 11) and residential (Fig.
12, Fig. 13) BuildingsBench buildings for the NRMSE, NMAE, NMBE, and RPS
metrics. For reference, we also plot the performance profile of the
Persistence Ensemble baseline. The methods that estimate the forecast with a
Gaussian distribution (the Persistence Ensemble and Transformer (Gaussian))
achieve noticeably better NMBE performance than Transformer (Tokens).

##  Appendix I Benchmark Dataset Results

Per-dataset zero-shot accuracy results are visualized in Fig. 14.

##  Appendix J Residential Empirical Scaling Laws

(a) Zero-shot NRMSE

(b) Zero-shot RPS

(c) Zero-shot NRMSE

(d) Zero-shot RPS

Figure 6: Residential building empirical scaling laws. Model performance is
independent with respect to the considered dataset and model sizes. Intervals
are 95% stratified bootstrap CIs for the median. a-b) Dataset size vs. zero-
shot performance. c-d) Model size vs. zero-shot performance.

The residential building results for the experiment where various models are
trained under increasing dataset scale (size and number of unique pretraining
buildings) and model size are shown in Fig. 6. Clearly, the accuracy and
uncertainty quantification performance is independent with respect to the
considered dataset and model sizes. This suggests that our current modeling
approach for residential buildings is unable to take advantage of additional
data and model parameters. As discussed in Sec. 6.2, alternative modeling
approaches that consider auxiliary inputs might be more successful for large-
scale pretraining of residential building STLF. We believe this is an
interesting future research direction.

##  Appendix K Ablation Studies

Table 6: Ablation study results on the zero-shot STLF task. Median accuracy
(NRMSE) and ranked probability score (RPS). Results are for Transformer-L
(Tokens). We ablate the use of the lat,lon covariate and the size of the token
vocabulary $|\mathcal{V}|$. The vocabulary with 8,192 tokens does not use
merging (note the high RPS for residential buildings).

Buildings-900K-Test (_simulated_) BuildingsBench (_real_) Commercial
Residential Commercial Residential $|\mathcal{V}|$ Lat,Lon NRMSE (%) RPS NRMSE
(%) RPS NRMSE (%) RPS NRMSE (%) RPS 2,274 ✗ 16.28 0.540 45.99 0.147 14.91 4.84
102.60 0.072 344 ✓ 13.37 0.427 42.40 0.132 15.24 4.52 102.31 0.072 8,192 ✓
14.80 0.513 45.35 0.502 15.09 4.87 105.6 0.780 2,274 ✓ 14.08 0.482 43.94 0.138
14.82 4.81 101.7 0.072

We explore the following aspects of the transformer models in this section:

  * •

App. K.1) The size of the vocabulary and the impact of the merging step in the
load tokenizer.

  * •

App. K.2) Using standard scaling vs. Box-Cox scaling for training the
Transformer (Gaussian) model.

  * •

App. K.3) Impact of the lat, lon covariate on generalization.

  * •

App. K.4) Fine-tuning all layers vs. only the last layer for transfer
learning.

###  K.1 Load Tokenizer

Our experiments are conducted with a merged vocabulary of size $|\mathcal{V}|$
= 2,274 ($K$ = 8,192). To investigate the impact of using a smaller
vocabulary, we trained the Transformer-L model with a merged vocabulary of
size $|\mathcal{V}|$ = 344 ($K$ = 512) (Fig. 5(c)). We also trained a model
_without merging_ , thus, $|\mathcal{V}|$ = 8,192 ($K$ = 8,192) (Fig. 5(b)).
The results in Table 6 indicate that when the vocabulary is too small
($|\mathcal{V}|$ = 344), the accuracy on real buildings deteriorates. Without
merging ($|\mathcal{V}|$ = 8,192), the RPS on real residential buildings is
significantly worse. Nearly all tokens correspond to small load values, which
causes the model to allocate probability to a large number of tokens whose
load values are very close to the target value.

###  K.2 Gaussian data normalization

As can be seen in Fig. 7, standard normalizing the data was not sufficient to
prevent training instability (large spikes in the loss) due to the highly non-
Gaussian load values. By contrast, Box-Cox scaling was successful at
stabilizing training.

Figure 7: Box-Cox scaling vs. standard scaling training stability. Standard
normalizing the load values did not prevent the Transformer (Gaussian) model
from suffering from training instability. Box-Cox scaling was successful at
achieving stable training.

###  K.3 Latitude, longitude covariate

In our experiments, the transformer models have a latitude, longitude
covariate that roughly localizes where a building is in the world. The result
of training Transformer-L (Tokens) without this covariate are in Table 6. We
observe a negligible drop in zero-shot accuracy on real buildings, but a
significant loss in accuracy on the _simulated_ Buildings-900K-Test set. The
buildings in the Buildings-900K-Test set are from held-out PUMAs (i.e.,
counties) with never-before-seen latitude, longitude coordinates. As the
majority of real buildings are located in non-U.S. out-of-distribution
locations, the latitude and longitude covariate appears to not be particularly
helpful for generalizing to these buildings.

###  K.4 What layers should get fine-tuned?

Table 7: Ablation study results on transfer learning task. Median accuracy
(NRMSE) and ranked probability score (RPS). For statistical robustness, we
compute average probability of improving NRMSE due to fine-tuning: $P(X<Y)$ :=
$\frac{100}{N}\sum_{i=1}^{N}\mathbf{1}_{[X_{i}<Y_{i}]}$ where $X_{i}$ is the
pretrained + fine-tuned NRMSE for building $i$ and $Y_{i}$ is the pretrained
NRMSE. We compare fine-tuning only the last layer and fine-tuning all layers.

Commercial buildings Residential buildings NRMSE ($\%$) RPS $P(X<Y)$ NRMSE
($\%$) RPS $P(X<Y)$ Fine-tune last layer Transformer (Tokens) 14.48 3.95 54.5
98.23 0.062 66 Transformer (Gaussian) 13.45 3.82 66 83.89 0.072 66 Fine-tune
all layers Transformer (Tokens) 13.86 3.73 77.5 95.55 0.060 77 Transformer
(Gaussian) 13.36 3.50 84 82.17 0.061 83

For our main transfer learning benchmark results, we fine-tune all layers of
the transformers. Here, we compare the performance of these models with fine-
tuning only the last layer. Table 7 shows that when fine-tuning only the last
linear layer (that output logits or Gaussian parameters), the performance
suffers, with the Transformer (Tokens) model affected the most. Our intuition
is that since the pretraining data is synthetic, fine-tuning on real data may
require updating all layers. We believe a deeper investigation into transfer
learning paradigms for time series is an important topic of study.

##  Appendix L Author Responsibility Statement

The authors confirm that they bear all responsibility in case of any violation
of rights during the collection of the data or other work, and will take
appropriate action when needed, e.g. to remove data with such issues. The
authors also confirm the licenses provided with the data and code associated
with this work.

##  Appendix M Additional Plots

We provide additional qualitative results here (Fig. 8 and Fig. 9).

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

Figure 8: Qualitative results for Transformer-L (Gaussian). Ground truth time
series are truncated to previous 24 hours for visibility. Light blue lines are
10 samples from the predicted distribution.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

Figure 9: Qualitative results for Transformer-L (Tokens). Ground truth time
series are truncated to previous 24 hours for visibility. Light blue lines are
10 samples from the predicted distribution.

(a)

(b)

Figure 10: Real commercial building zero-shot NRMSE and NMAE performance
profiles with 95% boostrap CIs.

(a)

(b)

Figure 11: Real commercial building zero-shot NMBE and RPS performance
profiles with 95% boostrap CIs.

(a)

(b)

Figure 12: Real residential building zero-shot NRMSE and NMAE performance
profiles with 95% boostrap CIs.

(a)

(b)

Figure 13: Real residential building zero-shot NMBE and RPS performance
profiles with 95% boostrap CIs.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

Figure 14: Per-dataset median zero-shot accuracy with 95% boostrap CIs. Lower
is better.