# Transformer Training Strategies for Forecasting Multiple Load Time Series

Matthias Hertel  Maximilian Beichter  Institute for Automation and Applied
Informatics
Karlsruhe Institute of Technology  Benedikt Heidrich  Oliver Neumann  Benjamin
Schäfer  Ralf Mikut  Veit Hagenmeyer

###### Abstract

In the smart grid of the future, accurate load forecasts on the level of
individual clients can help to balance supply and demand locally and to
prevent grid outages. While the number of monitored clients will increase with
the ongoing smart meter rollout, the amount of data per client will always be
limited. We evaluate whether a Transformer load forecasting model benefits
from a transfer learning strategy, where a global univariate model is trained
on the load time series from multiple clients. In experiments with two
datasets containing load time series from several hundred clients, we find
that the global training strategy is superior to the multivariate and local
training strategies used in related work. On average, the global training
strategy results in 21.8% and 12.8% lower forecasting errors than the two
other strategies, measured across forecasting horizons from one day to one
month into the future. A comparison to linear models, multi-layer perceptrons
and LSTMs shows that Transformers are effective for load forecasting when they
are trained with the global training strategy.

## Introduction

Climate change is one of the biggest challenges facing humanity, with the risk
of dramatic consequences if certain limits of warming are exceeded [1]. To
mitigate climate change, the energy system must be decarbonized. A difficulty
in decarbonization is that renewable energy supply fluctuates depending on the
weather. However, supply and demand must be balanced in the grid at every
moment to prevent outages [2]. In addition, with the ongoing decentralization
of the renewable energy supply and the installation of large consumers, such
as electric vehicle chargers and heat pumps, low-voltage grids are expected to
reach their limits [3]. Thus, to balance the grid and to avoid congestions,
advanced operation and control mechanisms must be installed in the smart grid
of the future [4, 5]. This requires accurate forecasts on various aggregation
levels, up to fine-grained low-voltage level load forecasts [5, 6]. Such fine-
grained load forecasts can be used for demand-side management, energy
management systems, distribution grid state estimation, grid management,
storage optimization, peer-to-peer trading, peak shaving, smart electrical
vehicle charging, dispatchable feeders, provision of feedback to customers,
anomaly detection and intervention evaluation [5, 7, 8, 9, 10]. Moreover, the
aggregation of fine-grained load forecasts can result in a more accurate
forecast of the aggregated load [11].

With the smart meter rollout, fine-grained electrical load data will become
available for an increasing number of clients. In such a scenario where load
time series from multiple clients are available, different model training
strategies are possible. The goal of our work is to compare training
strategies for the Transformer [12], which was recently used for load
forecasting [13, 14, 15, 16, 17, 18].

Figure 1: The three training strategies, with models depicted as networks. An
example with three load time series, four days input and one day output is
shown. (a) Multivariate: one model processes all load time series
simultaneously; (b) local: separate models (blue, orange, green) process each
load time series; (c) global: one model (black) processes all load time series
one at a time.

### Task definition

We address the following multiple load time series forecasting problem: At a
time step $t$, given the history of the electrical load of $C$ clients
$x_{0}^{c},...x_{t}^{c}$ with $1\leq c\leq C$, the goal is to predict the next
$h$ electrical load values $x_{t+1}^{c},...,x_{t+h}^{c}$ for all clients
$1\leq c\leq C$, where $h$ is called the forecast horizon.

### Contribution

We compare three training strategies for the Transformer in a scenario with
multiple load time series. The training strategies are depicted in Figure 1.

  1. 1.

A _multivariate_ model training strategy, where a single model gets all load
time series as input and forecasts all load time series simultaneously.

  2. 2.

A _local_ model training strategy, where a separate univariate111By
’univariate’ we mean models which produce a forecast for a single time series.
We still call models ’univariate’ when they have multiple input variables,
such as exogenous time and calendar features. model is trained for each load
time series.

  3. 3.

A _global_ model training strategy, where a generalized univariate model is
used to forecast each load time series separately.

We compare our models with the models from related work [19, 20, 21, 22], as
well as with multiple baselines. In particular, we compare with the linear
models used in [23], to figure out if Transformers are effective for load
forecasting and which training strategy is the most promising one.

### Paper structure

First, we describe the Related work. Then, the Transformer architecture and
the training strategies are described in the Approach. This is followed by the
Experimental setup, Results and a Discussion. Finally, the paper concludes
with the Conclusion and future work.

## Related work

This section first presents related work on long time series forecasting and
load forecasting with Transformers. Most of the load forecasting literature
uses local models, but few works use global models, which are presented next.
The global training strategy can be understood as a transfer learning
technique. We therefore discuss transfer learning in the field of load
forecasting at the end of this section.

As Transformer are often used for long time series forecasting with up to one
month horizon, various extensions to the Transformer architecture exist that
aim to reduce the time and space complexity. This is done by the Informer
using ProbSparse self-attention [19], by the Autoformer using auto-correlation
[20], by the FEDformer using frequency enhanced decomposition [21] and by
PatchTST using patching [22]. The proposed models are multivariate or local,
except for the global PatchTST [22]. The experiments in these works are
conducted on six datasets from different domains, including one load
forecasting dataset, which we also use in our experiments (see section
Datasets). A global linear model called LTSF-Linear [23] gives better results
than the aforementioned multivariate Transformers. Parallel to our work,
global Transformers were shown to beat the aforementioned multivariate
Transformers [24]. However, this work does not optimize the model’s lookback
size and therefore achieves sub-optimal results. PatchTST [22] is a global
Transformer with patched inputs and is superior to LTSF-Linear [23] on the six
datasets.

Transformer architectures for short-term load forecasting are designed to use
external calendar and weather features [25, 26]. An evaluation of different
architectures is undertaken in [14]. Further work modifies the architecture
for multi-energy load forecasting [25]. Upstream decompositions are used to
improve the forecast quality [27]. These models are not compared on a common
benchmark dataset, but evaluated on different datasets on city or national
level. There, usually only one load time series is available, which only
allows for local models. Furthermore, the models are not compared to the
Transformer architectures for long time series.

Global load forecasting models are already used with convolutional neural
networks [8] and N-BEATS [9]. A mixture between a multivariate and a global
model is investigated in [28], where a single recurrent neural network (RNN)
model is trained on randomly pooled subsets of the time series. Some works
cluster the time series and then train global or multivariate models for each
cluster [29, 30]. PatchTST [22] is a global Transformer with patched inputs.
We compare to this approach in our experiments.

The authors of [31] and [32] present current literature on transfer learning
in the domain of energy systems. They define a taxonomy of transfer learning
methods and discuss different strategies of using transfer learning with
buildings from different domains. Two works [33, 34] use transfer learning by
pre-training and fine-tuning Transformers. Transferability from one building
to another is tested in [33], and from one district to another in [34]. In
contrast to these works, our transfer learning approach is to train a
generalized model on the data from many clients, without fine-tuning for a
target time series.

## Approach

We use an encoder-decoder Transformer [12] as a load forecasting model. This
model architecture has self-attention and cross-attention as its main . and
was initially used for machine translation. It was used as a forecasting model
in [35] and later adopted for load forecasting [13, 14, 15]. We use the model
implementation from [14].

The encoder gets $L$ vectors as input, which represent the last $L$ time
steps, where $L$ is called the lookback size. Each input vector consists of
one (in the case of local and global models) or $C$ (in the case of
multivariate models) load values, and nine additional time and calendar
features. The features are the hour of the day, the day of the week and the
month (all cyclically encoded with a sine and a cosine function), whether it
is a workday, whether it is a holiday and whether the next day is a workday
(all binary features). The input to the decoder consists of $h$ vectors, which
represent the following $h$ time steps for which a forecast will be made. In
the decoder input, the load values are set to zero, so that each value is
forecasted independently from the previous forecasted values, allowing for a
direct multi-step forecast instead of generating all values iteratively. The
input vectors to the encoder and the decoder are first fed through linear
layers to increase the dimensionality to the hidden dimension of the model
$d_{\mathrm{model}}$. Both the encoder and the decoder consist of multiple
layers with eight self-attention heads and the decoder layers have eight
additional masked cross-attention heads. Finally, a linear layer transforms
the $h$ decoder output vectors into a forecast with $h\times 1$ (for local and
global models) or $h\times C$ (for multivariate models) values. We varied the
number of encoder and decoder layers and the hidden dimension
$d_{\mathrm{model}}$, and found three layers with $d_{\mathrm{model}}=128$ to
give the best results. The full model architecture is shown in Figure 2.

Figure 2: Architecture of the Transformer forecasting model. The input and
output dimensions differ for the multivariate model and the local and global
models. The shown dimensions refer to the Electricity dataset with 321
clients.

### Training strategies

We compare multivariate, local and global Transformers. The training
strategies are depicted in Figure 1 and are further explained in the
following. Details on the inputs, outputs, number of models and training data
size for each training strategy are given in Table 1.

  * •

_Multivariate training strategy:_ In the input to the model, each time step is
represented by a vector of size $C+f$, where $C$ is the number of load time
series and $f$ is the number of calendar features. The model forecasts $C$
values for the next $h$ time steps, i.e. its output consists of $h$ vectors of
size $C$. A single model is used to forecast all time series simultaneously.

  * •

_Local training strategy:_ Local models get only one time series as input and
generate a forecast for this time series. In the input, each time step is
represented by a vector with $f+1$ entries for the $f$ calendar features and
the electrical load value. $C$ separate models are trained for the $C$ time
series, each using the training data from one time series.

  * •

_Global training strategy:_ The global approach is a single model that
generalizes for all load time series. The model gets one load time series as
input and generates a forecast for that load time series. In contrast to the
local models, only one global model is trained on samples from all load time
series, and this model is used to forecast all load time series. This results
in $C$ times as many training data for the global model as for a local model.
To generate forecasts for all $C$ time series, the global model is used $C$
times with the history of one load time series as input.

Table 1: Training strategy details for the Electricity dataset with 321 load time series, 2.1 years training data and nine time and calendar features. For the local models, training data is the amount of training data per model. Training strategy | models | input size | output size | training data
---|---|---|---|---
Multivariate | 1 | $L\times 330$ | $h\times 321$ | 2.1 years
Local | 321 | $L\times 10$ | $h\times 1$ | 2.1 years
Global | 1 | $L\times 10$ | $h\times 1$ | 321 $*$ 2.1 years

## Experimental setup

### Datasets

As recommended in recent literature reviews on load forecasting [5, 11, 36],
we conduct experiments on multiple datasets, namely the Electricity and the
Ausgrid solar home datasets. For both datasets we make a temporal split and
use the first 70% of each time series for training, the next 10% for
validation, and the last 20% for testing, as in related work [20, 21, 22, 23].

The Electricity dataset222https://github.com/laiguokun/multivariate-time-
series-data is published in [37] and used in related work on long-term
forecasting [19, 20, 21, 22, 23]. It is a subset of the UCI Electricity Load
Diagrams
dataset333https://archive.ics.uci.edu/ml/datasets/ElectricityLoadDiagrams20112014
first presented in [38], only containing the time series without missing
values. The dataset contains hourly electrical load data from 321 clients of a
Portuguese energy supplier. The clients are from different economic sectors,
including offices, factories, supermarkets, hotels, restaurants, among others
[38]. The time series range from 2012 to 2014.

The Ausgrid solar home dataset444https://www.ausgrid.com.au/Industry/Our-
Research/Data-to-share/Solar-home-electricity-data contains solar generation
and electrical load data from 300 clients555We use 299 of the clients because
one client had missing data. of an Australian energy supplier. The clients are
private houses with rooftop solar systems. The time series range from July
2010 to June 2013. We only use the electrical load data transformed into
hourly resolution.

### Comparison methods

We compare our models with models from related work [19, 20, 21, 22, 23], as
well as with a persistence baseline, linear regression models, multi-layer
perceptrons and long short-term memory networks.

  * •

_Models from related work:_ For Informer [19], Autoformer [20], FEDformer
[21], PatchTST [22] and LTSF-Linear [23], we take the results reported in the
publications where applicable, and run the code published with the papers
otherwise. All parameters except for the forecast horizon are left unchanged.

  * •

_Persistence baseline:_ The persistence baseline takes the value from one week
before the predicted hour as a forecast for the 24 hours and 96 hours
horizons, and the value from one month before the predicted hour as the 720
hours forecast.

  * •

_Linear regression:_ For each load time series, we train a linear regression
model with $h$ outputs. The input consists of the last 336 load values and the
nine time and calendar features for the current hour when the prediction is
made (see Approach for a description of the features). The main difference to
LTSF-Linear [23] is that the linear regression models are local models, but
LTSF-Linear is a global model. Furthermore, the two approaches use different
training algorithms and LTSF-Linear does not use time and calendar features.

  * •

_Multi-layer perceptron (MLP):_ As for the linear regression, we train a local
MLP for each load time series. The MLPs get the last 168 load values and the
nine time and calendar features of the current hour as input. Using more than
168 load values as input did not improve the results. Each MLP has two hidden
layers with ReLU activation [39] and 1024 neurons per layer.

  * •

_Long short-term memory (LSTM):_ We train multivariate, local and global LSTM
[40] models. We use the same architecture as in [41], consisting of two LSTM
layers with 20 units each and a linear prediction layer. Using larger models
did not improve the results.

### Training details

All models are trained with the AdamW optimizer [42] using the mean squared
error loss. We use a batch size of 128 and a learning rate of 0.0001 with 1000
warm-up steps and cosine decay with $\gamma=0.8$. When testing different
lookback sizes $L$, we find one week to be optimal for the multivariate
Transformer and the local Transformers. For the global Transformer, the
results improve with increasing lookback size until $L=336$ (two weeks), and
stay almost the same for $L=720$ (one month). For Transformer models with two
weeks input and one month output, the batch size has to be reduced to 64 due
to the quadratic memory consumption of the model. For the multivariate
Transformer, the batch size is set to 32 as in related work [19, 20, 21]. The
validation error is evaluated every 10,000 training steps and at the end of
every epoch. We use early stopping to end the training when no more
improvement on the validation set is seen for ten evaluations. For the MLPs,
the initial learning rate is set to 0.001 and decayed with $\gamma=0.5$ after
every epoch.

### Metric

As in related work [19, 20, 21, 22, 23], every load time series is
standardized by subtracting its mean and dividing by its standard deviation
and the metrics are computed on these standardized time series. For every hour
$t\in T_{\mathrm{test}}$ in the test set, a forecasting model predicts the
next $h$ hourly loads
$\hat{y}_{t}^{c}=\hat{y}_{t,t+1}^{c},...,\hat{y}_{t,t+h}^{c}$ for time series
$c$. Then, the mean absolute error (MAE) between the predictions
$\hat{y}^{c}=\\{\hat{y}_{i}^{c}\ \forall\ i\in T_{\mathrm{test}}$} and the
ground truth $y^{c}=y_{1}^{c},...,y_{T_{\mathrm{test}}}^{c}$ is computed. As
the final result, the MAE averaged across all $C$ load time series, the
$T_{\mathrm{test}}$ evaluation time points and the $h$ forecasting steps is
reported.

$\mathrm{MAE(y,\hat{y}})=\dfrac{1}{C\cdot|T_{\mathrm{test}}|\cdot
h}\sum_{c=1}^{C}\sum_{t\in
T_{\mathrm{test}}}\sum_{i=1}^{h}|y_{t+i}^{c}-\hat{y}_{t,t+i}^{c}|$

The mean squared error (MSE) is computed analogously, using the squared
residuals instead of the absolute residuals.

## Results

### Forecast accuracy

Table 2 shows the MAE results on the two datasets666The MSE results show a
similar pattern and can be found in appendix A.. On the Electricity dataset,
the global Transformer is the best model for the 24 hours horizon, and
PatchTST is the best model for longer horizons. On the Ausgrid solar home
dataset, PatchTST is the best model for all three horizons. The global
Transformer beats the local Transformers and the multivariate Transformer
across all tested horizons. On average, it reduces the error by 21.8% compared
to the multivariate Transformer and by 12.8% compared to the local
Transformers. Compared to the best local model, the linear regression, it
reduces the error by 2.9%. Compared to the best multivariate model, FEDformer,
it reduces the error by 15.4%. All multivariate models, including Informer,
Autoformer, FEDformer and the multivariate Transformer, perform poorly and do
not beat the persistence baseline with a lag of one week. The local linear
regression models are slightly better than the global linear model, LTSF-
Linear, on the Electricity dataset, but it is vice versa on the Ausgrid solar
home dataset. The MLP is in five out of six cases a bit worse than the linear
regression, with a 1.5% larger error on average. The local LSTMs are better
than the local Transformers, but the Transformer is better as a multivariate
model and as a global model (except for the one month horizon on the
Electricity dataset). The forecast errors are lower on the Electricity dataset
than on the Ausgrid dataset which is a more fine-grained dataset containing
single private houses.

### Computational cost

The training times are given in Table 3. The local Transformer models need by
far the longest time to train. Their training time increases sharply with
longer forecast horizons. The multivariate Transformer trains fast and is even
faster than the MLPs for short horizons. Training a global Transformer is much
faster than training the many local Transformers but takes longer than the
linear regression, MLP and the multivariate Transformer. The LSTM trains
always faster than the Transformer with the same training strategy.

Table 2: MAE results on the two datasets, with 24, 96 and 720 hours forecast
horizon. MV = multivariate, L = local, G = global. The best results are
highlighted in bold and the best results per training strategy are highlighted
in italic.

Model strat- input Electricity Ausgrid egy (days) 24h 96h 720h 24h 96h 720h
Informer [19] MV 4 0.399 0.407 0.450 0.582 0.607 0.645 Autoformer [20] MV 4
0.289 0.317 0.361 0.579 0.569 0.592 FEDformer [21] MV 4 0.284 0.297 0.343
0.560 0.566 0.609 LSTM MV 7 0.400 0.402 0.407 0.611 0.618 0.613 Transformer MV
7 0.366 0.384 0.382 0.584 0.586 0.576 Persistence L - 0.279 0.279 0.447 0.647
0.647 0.717 Linear regression L 14 0.203 0.233 0.296 0.496 0.524 0.565 MLP L 7
0.199 0.236 0.308 0.499 0.532 0.567 LSTM L 7 0.263 0.283 0.337 0.517 0.541
0.573 Transformer L 7 0.256 0.289 0.354 0.535 0.563 0.583 LTSF-Linear [23] G
14 0.209 0.237 0.301 0.490 0.515 0.553 PatchTST [22] G 14 0.190 0.222 0.290
0.468 0.494 0.522 LSTM G 7 0.207 0.239 0.302 0.491 0.525 0.559 Transformer G
14 0.184 0.225 0.312 0.482 0.514 0.533

Table 3: Training times in hours, measured on a machine with a Nvidia 3090 RTX
GPU.

Model Electricity Ausgrid 24h 96h 720h 24h 96h 720h Linear regression (local)
0.02 0.03 0.08 0.02 0.03 0.07 MLP (local) 0.42 0.42 0.36 0.40 0.39 0.39 LSTM
(multivariate) 0.06 0.08 0.30 0.03 0.04 0.10 LSTM (local) 8.25 7.61 7.20 4.27
3.55 3.49 LSTM (global) 1.09 0.82 0.98 1.11 0.84 0.71 Transformer
(multivariate) 0.19 0.23 0.88 0.10 0.09 0.39 Transformer (local) 14.20 16.82
102.09 8.33 9.74 62.53 Transformer (global) 3.42 2.00 9.86 3.85 2.85 6.77

## Discussion

#### Best Transformer training strategy:

On the two datasets, the global Transformer is superior to the multivariate
and local Transformers. We hypothesize that this is a result of the larger
number of training samples for the global model (see Table 1). The Transformer
benefits from more training data, even if the training data comes from
different sources. The multivariate models on the other hand are prone to
overfitting.

#### Best Transformer architecture:

PatchTST is the best model in five out of six cases. However, the difference
to the global Transformer is small. This shows that the success of PatchTST is
mainly a result of its global training strategy. Its improvement upon the
global Transformer can be due to the patching mechanism, a better
hyperparameter configuration, or the encoder-only architecture. Among the
multivariate models, Autoformer [20] and FEDformer [21] give better results
than the multivariate Transformer. It remains an open question whether these
architectures are also better global models than the standard Transformer and
PatchTST [22]. Another promising architecture is the Temporal Fusion
Transformer [26]. In previous work with just one aggregated time series, the
Informer [19] also gave better results than the Transformer [14].

#### Comparison with the state of the art:

The global Transformer achieves a better result for short-term forecasting on
the Electricity dataset than related work [19, 20, 21, 22, 23], and achieves
close results to the best results from PatchTST [22] for longer horizons and
on the Ausgrid solar home dataset. However, to establish a state of the art
for short-term and medium-term load forecasting, a comparison to other
forecasting models must be undertaken, including models that are not based on
the Transformer architecture and that are more sophisticated than our
baselines. Using weather data could improve the forecasts, because some
electrical load patterns, such as the usage of electrical heating, are
weather-dependent. Weather features could affect which model gives the best
results, because some models might be better in capturing these dependencies
than others.

#### Linear models:

As in related work [23], we observe that linear models are strong baselines.
The linear regression is in five out of six cases the best local model and
only outperformed by the local MLP for the one day horizon on the Electricity
dataset. No general answer can be given on whether the local linear regression
models are better or the global LTSF-Linear is better, because each variant is
better on one dataset.

#### Task complexity:

For longer horizons, the global Transformer’s performance compared to the
linear models deteriorates. This can be due to the increasing complexity when
the model forecasts many values simultaneously. We chose a direct multi-step
forecasting model because good results were achieved with this procedure
before [22, 23]. However, other multi-step forecasting procedures, such as
iterative single-step and iterative multi-step forecasting [43, 44], could be
beneficial for long-term forecasting because they reduce the number of
forecasted values per model run.

#### Transfer learning:

According to the definition of transfer learning in [31], the global training
strategy can be seen as a transfer learning method, because the model must
transfer knowledge between different types of buildings with different
consumption patterns. Pre-training on other tasks than forecasting or on less
similar data from domains other than electricity, as well as fine-tuning for a
time series of interest, could improve the results. An advantage of the global
model is that it can be applied to new time series without retraining. In [15]
it was shown that the Transformer generalizes better to new time series than
other approaches, but the forecasts are still better when training data from
the target time series is available.

#### Other forecasting tasks:

The Transformer model and the different training strategies are not designed
for load forecasting in particular, but can also be applied to other
forecasting tasks. We hypothesize that the global training strategy can also
be beneficial for other datasets containing multiple time series with similar
patterns.

## Conclusion and future work

We compare three Transformer training strategies for load forecasting on two
datasets with multiple years of data for multiple hundred clients. We show
that the multivariate training strategy used in related work on forecasting
with Transformers [19, 20, 21] is not optimal, and it is better to use a
global model instead. This shows that the right training strategy is crucial
to get good results from a Transformer. Our approach achieves better results
than related work [19, 20, 21], and comes close to the best results from
PatchTST [22]. In particular, our approach gives better results than the
linear models from [23] for one day to four days forecasting horizons, which
shows that, with the right training strategy, Transformers are effective for
load forecasting. However, simple linear models give decent results for both
short-term and medium-term horizons and train much faster than the
Transformers.

In the future, more sophisticated Transformer architectures could be tested
with the global training strategy. A comparison to other forecasting methods
could be undertaken, and weather data could be incorporated into the models to
see how it affects the results. Experiments with other datasets and varying
amounts of training data could show under which circumstances the global
Transformer model is better than other approaches. Additionally, transfer
learning from other tasks and datasets could be tested. Future work could
experiment with different datasets with varying amounts of data to see how
much training data is needed for the global model to surpass the local models.
A compromise between local and global models could be established by first
clustering similar time series and then training one global model per cluster.
The cluster-specific models would have less training data than the global
model, but could benefit from the training data being more similar.
Potentially, the global training strategy could also be beneficial for other
forecasting tasks than load forecasting.

## Acknowledgement

This project is funded by the Helmholtz Association’s Initiative and
Networking Fund through Helmholtz AI, the Helmholtz Association under the
Program “Energy System Design”, and the German Research Foundation (DFG) as
part of the Research Training Group 2153 “Energy Status Data: Informatics
Methods for its Collection, Analysis and Exploitation”.

## Availability of data and materials

See section Datasets for the sources of the public datasets. Code is available
on GitHub via https://github.com/KIT-IAI/transformer-training-strategies.

## Author’s contributions

MH: Conceptualisation, Investigation, Methodology, Software, Validation,
Visualisation, Writing - original draft. MB: Writing - original draft, Writing
- review and editing. BH, ON: Writing - review and editing. BS, RM, VH:
Funding acquisition, Supervision, Writing - review and editing.

## Competing interests

The authors declare that they have no competing interests.

## References

  * [1] H.-O. Pörtner, D. C. Roberts, H. Adams, C. Adler, P. Aldunce, E. Ali, R. A. Begum, R. Betts, R. B. Kerr, R. Biesbroek, et al., “Climate change 2022: Impacts, adaptation and vulnerability,” IPCC Geneva, Switzerland, 2022.
  * [2] J. Machowski, J. Bialek, J. R. Bumby, and J. Bumby, “Power system dynamics and stability,” John Wiley & Sons, 1997.
  * [3] H. K. Çakmak and V. Hagenmeyer, “Using open data for modeling and simulation of the all electrical society in eASiMOV,” in 2022 Open Source Modelling and Simulation of Energy Systems (OSMSES), 2022.
  * [4] S. D. Ramchurn, P. Vytelingum, A. Rogers, and N. R. Jennings, “Putting the ’smarts’ into the smart grid: a grand challenge for artificial intelligence,” Communications of the ACM, vol. 55, no. 4, pp. 86–97, 2012\.
  * [5] S. Haben, S. Arora, G. Giasemidis, M. Voss, and D. V. Greetham, “Review of low voltage load forecasting: Methods, applications, and recommendations,” Applied Energy, vol. 304, p. 117798, 2021.
  * [6] J. Á. G. Ordiano, S. Waczowicz, V. Hagenmeyer, and R. Mikut, “Energy forecasting tools and services,” WIREs Data Mining Knowl. Discov., vol. 8, no. 2, 2018.
  * [7] B. Yildiz, J. I. Bilbao, J. Dore, and A. B. Sproul, “Recent advances in the analysis of residential electricity consumption and applications of smart meter data,” Applied Energy, vol. 208, pp. 402–427, 2017.
  * [8] M. Voß, C. Bender-Saebelkampf, and S. Albayrak, “Residential short-term load forecasting using convolutional neural networks,” in 2018 IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (SmartGridComm), pp. 1–6, 2018.
  * [9] M. Grabner, Y. Wang, Q. Wen, B. Blažič, and V. Štruc, “A global modeling framework for load forecasting in distribution networks,” IEEE Transactions on Smart Grid (Early Access), 2023.
  * [10] D. Werling, B. Heidrich, H. K. Çakmak, and V. Hagenmeyer, “Towards line-restricted dispatchable feeders using probabilistic forecasts for PV-dominated low-voltage distribution grids,” in Proceedings of the Thirteenth ACM International Conference on Future Energy Systems, pp. 395–400, 2022.
  * [11] T. Hong, P. Pinson, Y. Wang, R. Weron, D. Yang, and H. Zareipour, “Energy forecasting: A review and outlook,” IEEE Open Access Journal of Power and Energy, vol. 7, pp. 376–388, 2020.
  * [12] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in NIPS, pp. 5998–6008, 2017.
  * [13] G. Zhang, C. Wei, C. Jing, and Y. Wang, “Short-term electrical load forecasting based on time augmented transformer,” Int. J. Comput. Intell. Syst., vol. 15, no. 1, p. 67, 2022.
  * [14] M. Hertel, S. Ott, B. Schäfer, R. Mikut, V. Hagenmeyer, and O. Neumann, “Evaluation of transformer architectures for electrical load time-series forecasting,” in Proceedings 32. Workshop Computational Intelligence, 2022\.
  * [15] M. Hertel, S. Ott, B. Schäfer, R. Mikut, V. Hagenmeyer, and O. Neumann, “Transformer neural networks for building load forecasting,” in Tackling Climate Change with Machine Learning: workshop at NeurIPS 2022, 2022\.
  * [16] Y. Cao, Z. Dang, F. Wu, X. Xu, and F. Zhou, “Probabilistic electricity demand forecasting with transformer-guided state space model,” in 2022 IEEE 5th International Conference on Automation, Electronics and Electrical Engineering (AUTEEE), pp. 964–969, IEEE, 2022.
  * [17] E. Giacomazzi, F. Haag, and K. Hopf, “Short-term electricity load forecasting using the temporal fusion transformer: Effect of grid hierarchies and data sources,” arXiv preprint arXiv:2305.10559, 2023.
  * [18] P. C. Huy, N. Q. Minh, N. D. Tien, and T. T. Q. Anh, “Short-term electricity load forecasting based on temporal fusion transformer model,” IEEE Access, vol. 10, pp. 106296–106304, 2022.
  * [19] H. Zhou, S. Zhang, J. Peng, S. Zhang, J. Li, H. Xiong, and W. Zhang, “Informer: Beyond efficient transformer for long sequence time-series forecasting,” in AAAI, pp. 11106–11115, 2021.
  * [20] H. Wu, J. Xu, J. Wang, and M. Long, “Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting,” in NeurIPS, pp. 22419–22430, 2021.
  * [21] T. Zhou, Z. Ma, Q. Wen, X. Wang, L. Sun, and R. Jin, “FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting,” in International Conference on Machine Learning, pp. 27268–27286, 2022.
  * [22] Y. Nie, N. H. Nguyen, P. Sinthong, and J. Kalagnanam, “A time series is worth 64 words: Long-term forecasting with transformers,” arXiv:2211.14730, 2022\.
  * [23] A. Zeng, M. Chen, L. Zhang, and Q. Xu, “Are transformers effective for time series forecasting?,” arXiv:2205.13504, 2022.
  * [24] W. M. J. Murphy and K. Chen, “Univariate vs multivariate time series forecasting with transformers,” 2023.
  * [25] C. Wang, Y. Wang, Z. Ding, T. Zheng, J. Hu, and K. Zhang, “A transformer-based method of multienergy load forecasting in integrated energy system,” IEEE Transactions on Smart Grid, vol. 13, no. 4, pp. 2703–2714, 2022.
  * [26] P. C. Huy, N. Q. Minh, N. D. Tien, and T. T. Q. Anh, “Short-term electricity load forecasting based on temporal fusion transformer model,” IEEE Access, vol. 10, pp. 106296–106304, 2022.
  * [27] P. Ran, K. Dong, X. Liu, and J. Wang, “Short-term load forecasting based on CEEMDAN and transformer,” Electric Power Systems Research, vol. 214, p. 108885, 2023.
  * [28] H. Shi, M. Xu, and R. Li, “Deep learning for household load forecasting—a novel pooling deep RNN,” IEEE Transactions on Smart Grid, vol. 9, no. 5, pp. 5271–5280, 2017.
  * [29] F. Han, T. Pu, M. Li, and G. Taylor, “Short-term forecasting of individual residential load based on deep learning and k-means clustering,” CSEE Journal of Power and Energy Systems, vol. 7, no. 2, pp. 261–269, 2020.
  * [30] E. Yang and C.-H. Youn, “Individual load forecasting for multi-customers with distribution-aware temporal pooling,” in IEEE INFOCOM 2021-IEEE Conference on Computer Communications, pp. 1–10, 2021.
  * [31] G. Pinto, Z. Wang, A. Roy, T. Hong, and A. Capozzoli, “Transfer learning for smart buildings: A critical review of algorithms, applications, and future perspectives,” Advances in Applied Energy, p. 100084, 2022.
  * [32] Y. Himeur, M. Elnour, F. Fadli, N. Meskin, I. Petri, Y. Rezgui, F. Bensaali, and A. Amira, “Next-generation energy systems for sustainable smart cities: Roles of transfer learning,” Sustainable Cities and Society, vol. 85, p. 104059, 2022.
  * [33] M. Nawar, M. Shomer, S. Faddel, and H. Gong, “Transfer learning in deep learning models for building load forecasting: Case of limited data,” arXiv:2301.10663, 2023.
  * [34] J. Gao, W. Hu, D. Zhang, and Y. Chen, “TgDLF2.0: Theory-guided deep-learning for electrical load forecasting via transformer and transfer learning,” arXiv:2210.02448, 2022.
  * [35] N. Wu, B. Green, X. Ben, and S. O’Banion, “Deep transformer models for time series forecasting: The influenza prevalence case,” arXiv preprint arXiv:2001.08317, 2020.
  * [36] F. vom Scheidt, H. Medinová, N. Ludwig, B. Richter, P. Staudt, and C. Weinhardt, “Data analytics in the electricity sector–a quantitative and qualitative literature review,” Energy and AI, vol. 1, p. 100009, 2020\.
  * [37] G. Lai, W.-C. Chang, Y. Yang, and H. Liu, “Modeling long-and short-term temporal patterns with deep neural networks,” in The 41st international ACM SIGIR conference on research & development in information retrieval, pp. 95–104, 2018.
  * [38] F. Rodrigues and A. Trindade, “Load forecasting through functional clustering and ensemble learning,” Knowledge and Information Systems, vol. 57, no. 1, pp. 229–244, 2018.
  * [39] “A gentle introduction to the rectified linear unit (ReLU).” https://machinelearningmastery.com/rectified-linear-activation-function-for-deep-learning-neural-networks/.  Accessed: 2023-04-28.
  * [40] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural computation, vol. 9, no. 8, pp. 1735–1780, 1997.
  * [41] W. Kong, Z. Y. Dong, Y. Jia, D. J. Hill, Y. Xu, and Y. Zhang, “Short-term residential load forecasting based on lstm recurrent neural network,” IEEE transactions on smart grid, vol. 10, no. 1, pp. 841–851, 2017.
  * [42] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019.
  * [43] N. H. An and D. T. Anh, “Comparison of strategies for multi-step-ahead prediction of time series using neural network,” in 2015 International Conference on Advanced Computing and Applications (ACOMP), pp. 142–149, 2015\.
  * [44] D. Sahoo, N. Sood, U. Rani, G. Abraham, V. Dutt, and A. Dileep, “Comparative analysis of multi-step time-series forecasting for network load dataset,” in 2020 11th International Conference on Computing, Communication and Networking Technologies (ICCCNT), pp. 1–7, 2020.

##  Appendix A Additional results

The mean squared errors for the same models as in Table 2 are given in Table
4.

Table 4: MSE results on the two datasets, with 24, 96 and 720 hours forecast
horizon. The best results are highlighted in bold and the best results per
training strategy are highlighted in italic.

Model strat- input Electricity Ausgrid egy (days) 24h 96h 720h 24h 96h 720h
Informer [19] MV 4 0.305 0.320 0.384 0.853 0.870 0.891 Autoformer [20] MV 4
0.166 0.201 0.254 0.732 0.754 0.800 FEDformer [21] MV 4 0.164 0.183 0.231
0.707 0.741 0.798 LSTM MV 7 0.311 0.314 0.329 0.841 0.851 0.835 Transformer MV
7 0.267 0.292 0.300 0.853 0.841 0.800 Persistence L - 0.214 0.214 0.490 1.163
1.163 1.353 Linear regression L 14 0.103 0.133 0.194 0.604 0.659 0.725 MLP L 7
0.097 0.135 0.212 0.627 0.693 0.749 LSTM L 7 0.146 0.169 0.234 0.647 0.693
0.751 Transformer L 7 0.144 0.186 0.271 0.721 0.776 0.802 LTSF-Linear [23] G
14 0.110 0.140 0.203 0.598 0.647 0.705 PatchTST [22] G 14 0.094 0.129 0.197
0.576 0.641 0.704 LSTM G 7 0.106 0.139 0.202 0.603 0.667 0.719 Transformer G
14 0.090 0.127 0.219 0.599 0.665 0.716


# Learning to Estimate RIS-Aided mmWave Channels

Jiguang He, Henk Wymeersch, Marco Di Renzo, and Markku Juntti J. He and M.
Juntti are with Centre for Wireless Communications, FI-90014, University of
Oulu, Finland. H. Wymeersch is with Department of Electrical Engineering,
Chalmers University of Technology, Gothenburg, Sweden. M. Di Renzo is with
Université Paris-Saclay, CNRS, CentraleSupélec, Laboratoire des Signaux et
Systèmes, 3 Rue Joliot-Curie, 91192 Gif-sur-Yvette, France.This work is
supported by Horizon 2020, European Union’s Framework Programme for Research
and Innovation, under grant agreement no. 871464 (ARIADNE). This work is also
partially supported by the Academy of Finland 6Genesis Flagship (grant
318927), as well as the H2020 RISE-6G project under grant agreement number
101017011.

###### Abstract

Inspired by the remarkable learning and prediction performance of deep neural
networks (DNNs), we apply one special type of DNN framework, known as model-
driven deep unfolding neural network, to reconfigurable intelligent surface
(RIS)-aided millimeter wave (mmWave) single-input multiple-output (SIMO)
systems. We focus on uplink cascaded channel estimation, where known and fixed
base station combining and RIS phase control matrices are considered for
collecting observations. To boost the estimation performance and reduce the
training overhead, the inherent channel sparsity of mmWave channels is
leveraged in the deep unfolding method. It is verified that the proposed deep
unfolding network architecture can outperform the least squares (LS) method
with a relatively smaller training overhead and online computational
complexity.

###### Index Terms:

Deep unfolding, reconfigurable intelligent surface, cascaded channel
estimation, deep neural network.

##  I Introduction

Reconfigurable intelligent surfaces (RISs) have recently been introduced for
enhanced energy efficiency (EE), spectrum efficiency (SE), positioning
accuracy, as well as network/physical-layer security [1, 2, 3, 4]. The RIS,
either being passive, active, or a hybrid combination of the former two, is
used to smartly control the radio propagation environment, by virtue of multi-
function capabilities, e.g., reflection, refraction, diffraction, scattering,
and even absorption [5]. In the literature, the RIS is commonly used as an
intelligent reflector, which breaks the well-known law of reflection [6], to
mitigate the blockage effect and expand the connectivity range, especially for
millimeter wave (mmWave) and terahertz (THz) communications.

However, the RIS is commonly assumed to be nearly-passive. That is, no power
amplifiers and baseband processing units are available at the RIS. Besides, in
order to compensate for the additional path loss in the reflection route via
the RIS, a large number of RIS elements must be employed. These make the
channel state information (CSI) acquisition difficult to tackle in practice,
since the channel estimation (CE) can only be performed either at the base
station (BS) or at the mobile station (MS). As RIS phase control and joint
active and passive beamforming are sensitive to the CSI accuracy, the full
potential of the RIS cannot be achieved when the CE is poorly performed.
Therefore, accurate yet efficient CE methods for the two individual channels
(i.e., the MS-RIS and RIS-BS channels) or the cascaded channel are of vital
importance. In our previous works, we took advantage of the inherent channel
sparsity and rank-deficiency features of the mmWave multiple-input multiple-
output (MIMO) channels and we applied the iterative reweighted method and the
atomic norm minimization (ANM) method for estimating the channel parameters of
RIS-aided mmWave MIMO systems [7, 8]. In another recent work [9], sparse
matrix factorization and matrix completion were exploited in a sequential
manner to facilitate the CE process. These works fall into the category of
conventional model-based approaches, which suit only for a small- or medium-
sized RIS, BS, and MS.

As the number of RIS elements and BS/MS antennas continues to grow (this is an
inevitable trend at high frequency bands, e.g., mmWave and THz), more training
overhead to obtain adequate CE performance, within the channel coherence time,
via conventional model-based methods is required. Besides, the associated
computational complexity will become inevitably high. These considerations
motivate the application of data-driven or hybrid approaches for CE in RIS-
aided communications [10]. In [11], a convolutional neural network (CNN) was
considered for RIS CE in a multi-user scenario. Therein, however, each user
needs to first estimate its own channel and then map the estimate to the
corresponding ground truth channel, which naturally increases the
computational complexity. Therefore, in the present paper, we resort to a
model-driven deep unfolding approach, which has already been used in MIMO
detection and sparse signal recovery [12, 13], for estimating the cascaded
channel in RIS-aided mmWave single-input multiple-output (SIMO) systems. To
the best of the authors’ knowledge, the concept of deep unfolding has not yet
been considered for RIS CE.

Deep unfolding mimics the operation of conventional (projected) gradient
descent schemes, and it is capable of directly mapping the received pilot
signals to the cascaded channel. Its computationally intensive training
process can be executed offline and the online implementation/prediction phase
only entails low-complexity calculations, e.g., matrix multiplications and
additions, and element-wise operations. In this study, specifically, the rank-
deficiency property of the cascaded channel is explicitly considered in the
deep unfolding framework. It is verified that the deep unfolding scheme can
outperform least squares (LS) estimation in terms of normalized mean square
error (NMSE) with a smaller training overhead and a reduced online
computational complexity.

Notations: A bold lowercase letter ${\mathbf{a}}$ denotes a vector, and a bold
capital letter ${\mathbf{A}}$ denotes a matrix. $(\cdot)^{\mathsf{T}}$ and
$(\cdot)^{\mathsf{H}}$ denote the matrix or vector transpose and Hermitian
transpose, respectively. $\mathrm{diag}({\mathbf{a}})$ denotes a square
diagonal matrix with the entries of ${\mathbf{a}}$ on its diagonal,
$\mathbb{E}[\cdot]$ is the expectation operator, $\mathbf{0}$ denotes the all-
zero vector, ${\mathbf{I}}_{M}$ denotes the $M\times M$ identity matrix, and
$j=\sqrt{-1}$. $\|\cdot\|_{\mathrm{F}}$ and $\|\cdot\|_{*}$ denote the
Frobenius norm and nuclear norm of a matrix, respectively, and $\|\cdot\|_{2}$
denotes the Euclidean norm of a vector.

##  II System Model

Figure 1: A typical scenario for maintaining the connectivity by deploying an
RIS in a mmWave SIMO network.

We consider a nearly-passive RIS-aided mmWave SIMO network, where the MS
communicates with the BS via the RIS, as shown in Fig. 1. The BS and RIS are
equipped with multiple antennas and with nearly-passive scattering elements,
respectively, while the MS is equipped with a single antenna. We further
assume that the direct MS-BS channel is blocked.111When the direct MS-BS
channel exists, we can estimate it by setting the RIS into an absorption mode.
Then, we can estimate the cascaded channel by subtracting the direct MS-BS
channel. The RIS-BS channel, which is denoted as
${\mathbf{H}}_{1}\in\mathbb{C}^{M\times N}$ with $M$ and $N$ being the number
of antennas and scattering elements at the BS and RIS, respectively, can be
written as

${\mathbf{H}}_{1}=\sum\limits_{i=1}^{L_{1}}g_{1,i}\boldsymbol{\alpha}(\phi_{1,i})\boldsymbol{\alpha}^{\mathsf{H}}(\theta_{1,i})={\mathbf{A}}(\boldsymbol{\phi}_{1})\mathrm{diag}({\mathbf{g}}_{1}){\mathbf{A}}^{\mathsf{H}}(\boldsymbol{\theta}_{1}),$
(1)

where $L_{1}\ll\mathrm{min}\\{M,N\\}$ is the number of channel paths,
including one line-of-sight (LoS) path (with $i=1$) and multiple non-line-of-
sight (NLoS) paths (with $i>1$), and $g_{1,i}\in\mathbb{C},$
$\theta_{1,i}\in\mathbb{R}$, and $\phi_{1,i}\in\mathbb{R}$ denote the
propagation path gain, the angle of departure (AoD), and the angle of arrival
(AoA) associated with the $i$th propagation path. The array response vector
$\boldsymbol{\alpha}(\phi_{1,i})\triangleq[1,\;\;e^{j\pi\sin(\phi_{1,i})},\ldots,e^{j(M-1)\pi\sin(\phi_{1,i})}]^{\mathsf{T}}\in\mathbb{C}^{M\times
1}$ is obtained by assuming half-wavelength inter-element spacing, and
$\boldsymbol{\alpha}(\theta_{1,i})\in\mathbb{C}^{N\times 1}$ can be formulated
in the same manner. We assume that
$g_{1,1}\sim\mathcal{CN}(0,\sigma_{\text{LoS}}^{2})$, and define the vectors
$\tilde{{\mathbf{g}}}_{1}\triangleq[g_{1,2},\ldots,g_{1,L_{1}}]^{\mathsf{T}}\sim\mathcal{CN}(\mathbf{0},\sigma_{\text{NLoS}}^{2}{\mathbf{I}}_{L_{1}-1})$,
${\mathbf{g}}_{1}\triangleq[g_{1,1},\tilde{{\mathbf{g}}}_{1}^{\mathsf{T}}]^{\mathsf{T}}$,
$\boldsymbol{\phi}_{1}\triangleq[\phi_{1,1},\ldots,\phi_{1,L_{1}}]^{\mathsf{T}}$,
$\boldsymbol{\theta}_{1}\triangleq[\theta_{1,1},\ldots,\theta_{1,L_{1}}]^{\mathsf{T}}$,
${\mathbf{A}}(\boldsymbol{\phi}_{1})\triangleq[\boldsymbol{\alpha}(\phi_{1,1}),\ldots,\boldsymbol{\alpha}(\phi_{1,L_{1}})]$,
and
${\mathbf{A}}(\boldsymbol{\theta}_{1})\triangleq[\boldsymbol{\alpha}(\theta_{1,1}),\ldots,\boldsymbol{\alpha}(\theta_{1,L_{1}})]$.

Similarly, the MS-RIS channel ${\mathbf{h}}_{2}\in\mathbb{C}^{N\times 1}$ can
be written as

${\mathbf{h}}_{2}=\sum\limits_{i=1}^{L_{2}}g_{2,i}\boldsymbol{\alpha}(\phi_{2,i})={\mathbf{A}}(\boldsymbol{\phi}_{2}){\mathbf{g}}_{2}.$
(2)

All the channel parameters in ${\mathbf{h}}_{2}$ are defined as those in
${\mathbf{H}}_{1}$. We also assume
$g_{2,1}\sim\mathcal{CN}(0,\sigma_{\text{LoS}}^{2})$,
$\tilde{{\mathbf{g}}}_{2}\triangleq[g_{2,2},\ldots,g_{2,L_{2}}]^{\mathsf{T}}\sim\mathcal{CN}(\mathbf{0},\sigma_{\text{NLoS}}^{2}{\mathbf{I}}_{L_{2}-1})$,
and define
${\mathbf{g}}_{2}\triangleq[g_{2,1},\tilde{{\mathbf{g}}}_{2}^{\mathsf{T}}]^{\mathsf{T}}$
and
${\mathbf{A}}(\boldsymbol{\phi}_{2})\triangleq[\boldsymbol{\alpha}(\phi_{2,1}),\ldots,\boldsymbol{\alpha}(\phi_{2,L_{2}})]$
with $L_{2}\ll N$.

The end-to-end uplink MS-RIS-BS channel (including the effect of the RIS) can
be written as

${\mathbf{h}}={\mathbf{H}}_{1}\boldsymbol{\Omega}{\mathbf{h}}_{2}={\mathbf{H}}_{1}\mathrm{diag}({\mathbf{h}}_{2})\boldsymbol{\omega},$
(3)

where $\boldsymbol{\Omega}=\mathrm{diag}(\boldsymbol{\omega})$ is the RIS
phase control matrix, with
$\boldsymbol{\omega}=[\omega_{1},\ldots,\omega_{N}]^{\mathsf{T}}$ and
$|\omega_{i}|=1$ for $\forall i$ [1]. It is worth noting that an RIS may be
composed of an array of amplifiers with increased power consumption and
hardware complexity [14]. In this case, the unit-modulus constraint on each
element $\omega_{i}$ can be relaxed. In the present paper, we are interested
in low-cost and low-complexity implementations of RISs, we hence focus on RISs
that can control only the phase response.

In (3), ${\mathbf{H}}_{1}\mathrm{diag}({\mathbf{h}}_{2})$ is referred to as
the cascaded channel. By knowing it, we can optimize the design of the RIS
phase control matrix and BS beamforming/combining vector. Let us define it as
${\mathbf{H}}_{c}\in\mathbb{C}^{M\times N}$, i.e.,

${\mathbf{H}}_{c}={\mathbf{H}}_{1}\mathrm{diag}({\mathbf{h}}_{2}).$ (4)

Based on the considered assumptions, we have
$\mathrm{rank}({\mathbf{H}}_{1})=L_{1}$ and
$\mathrm{rank}(\mathrm{diag}({\mathbf{h}}_{2}))=N$. Thus
$\mathrm{rank}({\mathbf{H}}_{c})\leq\mathrm{min}\\{\mathrm{rank}({\mathbf{H}}_{1}),\\\
\mathrm{rank}(\mathrm{diag}({\mathbf{h}}_{2}))\\}=L_{1}$. The inherent channel
sparsity (represented by the rank deficiency of the cascaded channel) can be
applied in order to enable an efficient yet accurate CE of (4).

##  III Learning to Estimate

In this section, we first introduce the channel sounding procedure, and then
describe the optimization problem formulation for recovering the cascaded
channel by using conventional (model-based) methods. Finally, we describe the
model-driven deep unfolding method for estimating the rank-deficient cascaded
channel.

###  III-A Channel Sounding

During the sounding process, pilot signals are sent from the MS to the BS via
the RIS. A different RIS phase control matrix is considered for each channel
use while the combining matrix at the BS is fixed. The received signal at
channel use $k$, for $k=1,\ldots,K$, can be written as

${\mathbf{y}}[k]={\mathbf{W}}^{\mathsf{H}}[k]{\mathbf{H}}_{1}\boldsymbol{\Omega}[k]{\mathbf{h}}_{2}s[k]+{\mathbf{W}}^{\mathsf{H}}[k]{\mathbf{n}}[k],$
(5)

where ${\mathbf{W}}[k]\in\mathbb{C}^{M\times N_{\text{W}}}$ is the combining
matrix at the BS with $N_{\text{W}}$ denoting the number of columns,222We
consider an analog combining matrix, which is a suitable choice for fulfilling
the requirements of reduced-complexity hybrid precoding architectures commonly
assumed for mmWave MIMO transceivers. When $N_{\text{W}}>N_{\text{RF}}$, with
$N_{\text{RF}}$ being the number of radio frequency (RF) chains at the BS, we
need $K\lceil N_{\text{W}}/N_{\text{RF}}\rceil$ channel uses to complete the
sounding process. Otherwise, $K$ channel uses are sufficient. $s[k]$ is the
pilot signal sent by the MS, and
${\mathbf{n}}[k]\sim\mathcal{CN}(0,\sigma^{2})$ is the additive white Gaussian
noise (AWGN) at the BS.

The received signal ${\mathbf{y}}[k]$ in (5) can be reformulated as

${\mathbf{y}}[k]={\mathbf{W}}^{\mathsf{H}}[k]{\mathbf{H}}_{c}\boldsymbol{\omega}[k]s[k]+{\mathbf{W}}^{\mathsf{H}}[k]{\mathbf{n}}[k],$
(6)

where $\boldsymbol{\Omega}[k]=\mathrm{diag}(\boldsymbol{\omega}[k])$.

Without loss of generality, we assume $s[1]=s[K]=1$ and
${\mathbf{W}}[1]={\mathbf{W}}[K]={\mathbf{W}}$. The received signals
${\mathbf{Y}}=[{\mathbf{y}}[1],\ldots,{\mathbf{y}}[K]]$ can be rewritten as

${\mathbf{Y}}={\mathbf{W}}^{\mathsf{H}}{\mathbf{H}}_{c}\bar{\boldsymbol{\Omega}}+{\mathbf{W}}^{\mathsf{H}}{\mathbf{N}},$
(7)

where
$\bar{\boldsymbol{\Omega}}=[\boldsymbol{\omega}[1],\ldots,\boldsymbol{\omega}[K]]$
and ${\mathbf{N}}=[{\mathbf{n}}[1],\ldots,{\mathbf{n}}[K]]$. An additional
vectorization step is considered for all the terms in (7), resulting in

${\mathbf{y}}=(\bar{\boldsymbol{\Omega}}^{\mathsf{T}}\otimes{\mathbf{W}}^{\mathsf{H}}){\mathbf{h}}_{c}+{\mathbf{n}},$
(8)

where
${\mathbf{y}}=\mathrm{vec}({\mathbf{Y}}),{\mathbf{h}}_{c}=\mathrm{vec}({\mathbf{H}}_{c})$,
and ${\mathbf{n}}=\mathrm{vec}({\mathbf{W}}^{\mathsf{H}}{\mathbf{N}})$. Based
on the vectorized received signal ${\mathbf{y}}$, we need to estimate the
vectorized cascaded channel ${\mathbf{h}}_{c}$. Let us define
$\boldsymbol{\Psi}=\bar{\boldsymbol{\Omega}}^{\mathsf{T}}\otimes{\mathbf{W}}^{\mathsf{H}}$.
Then, (7) can be simplified as
${\mathbf{y}}=\boldsymbol{\Psi}{\mathbf{h}}_{c}+{\mathbf{n}}$. Based on the
obtained signal model, the objective of this paper is to extract
${\mathbf{h}}_{c}$ from the noisy received signal ${\mathbf{y}}$ by assuming
that the matrix $\boldsymbol{\Psi}$ is known. This is accomplished by using
the deep unfolding method, which is detailed next.

###  III-B Optimization Problem Formulation

In order to recover ${\mathbf{h}}_{c}$ from the noisy observation
${\mathbf{y}}$, we formulate the following regularized optimization problem

$\hat{{\mathbf{h}}}_{c}=\arg\min_{{\mathbf{h}}_{c}}\|{\mathbf{y}}-\boldsymbol{\Psi}{\mathbf{h}}_{c}\|_{2}^{2}+\lambda\mathrm{rank}({\mathbf{H}}_{c}),$
(9)

which takes into consideration the rank deficiency of the cascaded channel
${\mathbf{H}}_{c}$ and the impact of noise [15]. The regularization parameter
$\lambda>0$ is introduced to control the tradeoff between the data fitting and
the rank of the cascaded channel. The optimization problem in (9) can be
further reformulated as

$\hat{{\mathbf{h}}}_{c}=\arg\min_{{\mathbf{h}}_{c}}\|{\mathbf{y}}-\boldsymbol{\Psi}{\mathbf{h}}_{c}\|_{2}^{2}+\lambda\|{\mathbf{H}}_{c}\|_{*},$
(10)

which is obtained by relaxing $\mathrm{rank}({\mathbf{H}}_{c})$ with its
nuclear norm, i.e., $\|{\mathbf{H}}_{c}\|_{*}$. This is a convenient approach
because $\mathrm{rank}({\mathbf{H}}_{c})$ is a noncovex function of
${\mathbf{H}}_{c}$. In addition,
$\|{\mathbf{H}}_{c}\|_{\mathrm{F}}\leq\|{\mathbf{H}}_{c}\|_{*}\leq\sqrt{r}\|{\mathbf{H}}_{c}\|_{\mathrm{F}}$
with $r\geq 1$. Notably, when $\mathrm{rank}({\mathbf{H}}_{c})=1$ (e.g.,
${\mathbf{H}}_{1}$ has only the LoS path), we have
$\|{\mathbf{H}}_{c}\|_{*}=\|{\mathbf{H}}_{c}\|_{\mathrm{F}}$. Also, the
singular values of ${\mathbf{H}}_{c}$ have a high probability to fulfill the
following condition: $\sigma_{1}\gg\sigma_{2}>\cdots>\sigma_{L_{1}}$ with
$\sigma_{i}$ being the $i$th largest singular value of ${\mathbf{H}}_{c}$,
which results in
$\|{\mathbf{H}}_{c}\|_{*}\approx\|{\mathbf{H}}_{c}\|_{\mathrm{F}}$. Thus, we
further replace $\mathrm{rank}({\mathbf{H}}_{c})$ in (10) with
$\|{\mathbf{H}}_{c}\|_{\mathrm{F}}$, i.e., $\|{\mathbf{h}}_{c}\|_{2}$, which
yields

$\hat{{\mathbf{h}}}_{c}=\arg\min_{{\mathbf{h}}_{c}}\|{\mathbf{y}}-\boldsymbol{\Psi}{\mathbf{h}}_{c}\|_{2}^{2}+\lambda\|{\mathbf{h}}_{c}\|_{2}.$
(11)

To accurately solve this optimization problem, we need to carefully choose the
regularization parameter $\lambda$. The optimal value of $\lambda$ is,
however, difficult to obtain. As a reference, $\lambda$ is chosen equal to
$4\sigma^{2}\sqrt{\frac{MN(M+N)\log(M+N)}{N_{\text{W}}K}}$ [15].

###  III-C Model-Driven Deep Unfolding

Deep unfolding is a deep neural network framework that mimics the conventional
gradient descent method. The difference lies in that deep unfolding is able to
learn from a large amount of (synthetic) data with enhanced performance and
reduced online implementation complexity and number of iterations. Typically,
this is exemplified in a reduced number of layers in the deep unfolding
network. When solving the optimization problem in (11) by using the
(conventional) gradient descent method, we iteratively update
${\mathbf{h}}_{c}^{(i)}$, with $(i)$ denoting the iteration index, as follows

${\mathbf{h}}_{c}^{(i)}={\mathbf{h}}_{c}^{(i-1)}-\beta\nabla
f({\mathbf{h}}_{c}^{(i-1)}),$ (12)

where $0<\beta<1$ is the step size and $\nabla f({\mathbf{h}}_{c}^{(i-1)})$ is
the gradient of
$f({\mathbf{h}}_{c})=\|{\mathbf{y}}-\boldsymbol{\Psi}{\mathbf{h}}_{c}\|_{2}^{2}+\lambda\|{\mathbf{h}}_{c}\|_{2}$
evaluated at ${\mathbf{h}}_{c}^{(i-1)}$, which can be expressed as

$\nabla
f({\mathbf{h}}_{c}^{(i-1)})=(\boldsymbol{\Psi}^{\mathsf{H}}\boldsymbol{\Psi}{\mathbf{h}}_{c}^{(i-1)}-\boldsymbol{\Psi}^{\mathsf{H}}{\mathbf{y}})+\lambda{\mathbf{h}}_{c}^{(i-1)}/\|{\mathbf{h}}_{c}^{(i-1)}\|_{2},$
(13)

when $\lambda$ is fixed. The initial value, i.e, ${\mathbf{h}}_{c}^{(0)}$, can
be set equal to the all-zero vector. In this case, the denominator
$\|{\mathbf{h}}_{c}^{(i-1)}\|_{2}$ in the last term of (13) needs to be
modified as $\|{\mathbf{h}}_{c}^{(i-1)}\|_{2}+\epsilon$ with $\epsilon>0$ when
$i=1$.

Substituting (13) into (12), we obtain

${\mathbf{h}}_{c}^{(i)}={\mathbf{h}}_{c}^{(i-1)}-\beta\boldsymbol{\Psi}^{\mathsf{H}}\boldsymbol{\Psi}{\mathbf{h}}_{c}^{(i-1)}+\beta\boldsymbol{\Psi}^{\mathsf{H}}{\mathbf{y}}-\beta\lambda{\mathbf{h}}_{c}^{(i-1)}/\|{\mathbf{h}}_{c}^{(i-1)}\|_{2}.$
(14)

The Gram matrix $\boldsymbol{\Psi}^{\mathsf{H}}\boldsymbol{\Psi}$, the
compressed statistics $\boldsymbol{\Psi}^{\mathsf{H}}{\mathbf{y}}$, and
${\mathbf{h}}_{c}^{(i-1)}$, are needed to apply the gradient descent
algorithm. Therefore, these three terms constitute the input variables of the
deep unfolding neural network model. To be specific, the $i$th layer of the
deep unfolding model for mimicking the gradient descent iteration in (14) is
introduced in Fig. 2. The accuracy and convergence speed of the channel
estimate in (14) highly depends on the specific choice of the step size
$\beta$ and the regularization parameter $\lambda$. In the considered deep
unfolding model, these parameters are learnable parameters that are
automatically determined during the data-driven training phase. To this end,
we introduce three generalized learnable parameters
$\delta_{1}^{(i)}\in[-1,\;0]$, $\delta_{2}^{(i)}\in[0,\;1]$, and
$\delta_{3}^{(i)}\in[-1,\;0]$, for the $i$th layer of the deep unfolding model
in Fig. 2. In (14), more specifically, $-\beta$ and $\beta$ in the second and
third term are replaced by $\delta_{1}^{(i)}$ and $\delta_{2}^{(i)}$,
respectively, and the product $-\beta\lambda/\|{\mathbf{h}}_{c}^{(i-1)}\|_{2}$
in the last term is enfolded in the learnable parameter $\delta_{3}^{(i)}$. As
shown in Fig. 2, in order to further enhance the prediction capabilities of
the deep unfolding network model, we serially concatenate the term
${\mathbf{h}}_{c}^{(i-1)}+\delta_{1}^{(i)}\boldsymbol{\Psi}^{\mathsf{H}}\boldsymbol{\Psi}{\mathbf{h}}_{c}^{(i-1)}+\delta_{2}^{(i)}\boldsymbol{\Psi}^{\mathsf{H}}{\mathbf{y}}+\delta_{3}^{(i)}{\mathbf{h}}_{c}^{(i-1)}$
with a learnable weight matrix ${\mathbf{M}}^{(i)}$, a bias vector
${\mathbf{b}}^{(i)}$, and a non-linear activation function. The complete deep
unfolding network model is illustrated in Fig. 3 and it comprises $L$ layers
from Fig. 2. In particular, the observations
$\\{\boldsymbol{\Psi}^{\mathsf{H}}\boldsymbol{\Psi},\boldsymbol{\Psi}^{\mathsf{H}}{\mathbf{y}}\\}$
are input to all the layers of the deep unfolding network model.

Figure 2: The $i$th layer of the deep unfolding network model for estimating
the cascaded channel vector. Figure 3: Complete deep unfolding network model
for channel estimation, which comprises $L$ layers from Fig. 2

##  IV Numerical Results

In this section, we evaluate the performance of the proposed deep unfolding
network model against two benchmark schemes: (i) the LS estimator and (ii) the
direct solution of (10) by using the CVX toolbox. We evaluate the impact of
the training overhead, the training SNR, the number of paths, and the angular
parameter distribution. As far as the RIS phase control matrices as concerned,
their diagonal elements are set equal to the column vectors of a discrete
Fourier transform (DFT) matrix. A set of orthonormal vectors are considered
for ${\mathbf{W}}$, e.g., the normalized column vectors from a DFT matrix. It
is worth mentioning that we transform the data from the complex-valued domain
to the real-valued domain before applying the deep unfolding network model.
The channel ${\mathbf{h}}_{c}^{(0)}$ is set equal to the all-zero vector. In
the first $L-1$ layers, we use relu activation functions, while no activation
function is applied in the last layer. The loss function during the training
phase is the NMSE between the output cascaded channel vector and the ground-
truth cascaded channel vector. The Adam algorithm is used for training, whose
learning rate is $0.001$ during the first $20$ epochs and it is halved in the
remaining epochs, and the batch size is $64$. We use $1e5$ samples for
training and $1e4$ samples for testing. The rest of the parameter setup is
given in Table I.

###  IV-A Impact of the Training Overhead

Fig. 4 shows the NMSE for a $1\times 16$ SIMO system (i.e., $M=16$) by using
the proposed deep unfolding approach as a function of the training overhead.
During the training phase, the SNR is $\gamma=1/\sigma^{2}=20$ dB. From Fig.
4, we see that the proposed scheme with $K=24$ channel uses for training
outperforms the LS estimator even if the latter uses a longer training phase
with $K=32$. Also, as expected, the higher the training overhead is, the lower
the NMSE of the proposed scheme is. It is worth noting that the proposed deep
unfolding method outperforms the numerical solution of (10) by using CVX. This
is attributed to the learning capability of the deep unfolded network through
the learnable parameters introduced in Fig. 2.

TABLE I: Parameter Setup. Parameter | Value
---|---
$L_{1}$ | $\\{1,2,3\\}$
$L_{2}$ | $\\{1,2,3\\}$
$\sigma_{\text{LoS}}^{2}$ | $1$
$\sigma_{\text{NLoS}}^{2}$ | $0.01$
$M$ | $16$
$N$ | $32$
$L$ | $15$
$\sin(\boldsymbol{\theta}_{1}),\sin(\boldsymbol{\phi}_{1}),\sin(\boldsymbol{\phi}_{2})$ | $\mathcal{U}[0,\;1]$
$K$ | $\\{24,28\\}$
$N_{\text{W}}$ | $8$
Figure 4: Impact of training overhead on estimating the cascaded channel for a
$1\times 16$ SIMO system with $N=32$ and $L_{1}=L_{2}=1$. Deep unfolding vs.
LS estimation and direct solution of (10).

###  IV-B Impact of the Number of Paths

In this subsection, we study the impact of the number of paths, which is
increased from one to two and three. The SNR for training is $\gamma=20$ dB
and $K=28$. The corresponding NMSE is shown in Fig. 5. When the number of
paths increases, the rank of the cascaded channel increases accordingly. In
this case, Fig. 5 shows that the NMSE increases as the number of paths
increases. In other words, by keeping the training overhead fixed, the
proposed deep unfolding method benefits from the sparsity of the channel,
i.e., the number of paths is small.

Figure 5: Impact of the number of paths on estimating the cascaded channel for
a $1\times 16$ SIMO system with $N=32$.

###  IV-C Impact of the Training SNR

In this subsection, we study the impact of the training SNR on the estimation
performance. Two SNR values, i.e., $\gamma=0$ dB, $\gamma=20$ dB, and an SNR
that varies in the set $\gamma\in\\{0,5,10,15,20\\}$ dB are considered. The
corresponding NMSE is shown in Fig. 6. We observe that the proposed deep
unfolding method provides the best NMSE when it is trained at a high SNR
except if the operating SNR during the test phase is too low.

Figure 6: Impact of the training SNR on estimating the cascaded channel for a
$1\times 16$ SIMO system with $N=32$.

###  IV-D Impact of the Angular Parameter Distribution

In this subsection, we evaluate the impact of the angular parameter
distribution when estimating the cascaded channel. The corresponding NMSE is
shown in Fig. 7. We observe that the NMSE decreases when the range of
$\sin(\boldsymbol{\theta}_{1}),\sin(\boldsymbol{\phi}_{1}),\sin(\boldsymbol{\phi}_{2})$
decreases. In other words, the estimation accuracy of the proposed deep
unfolding method increases when the individual channels are subject to a
reduced variability.

Figure 7: Impact of the angular parameter distribution on estimating the
cascaded channel for a $1\times 16$ SIMO system with $N=32$.

##  V Conclusions

In this letter, we have introduced a deep unfolding model for efficiently
estimating the end-to-end RIS channel in mmWave SIMO systems. With the aid of
simulation results, we have shown that the proposed approach can outperform
two benchmark schemes based on the LS method and a CVX-based numerical
solution of the channel estimation problem. The impact of the number of paths,
the training SNR, and the angular parameter distribution on the estimation
accuracy has been investigated. In addition, the proposed deep unfolding
network model has a low online prediction complexity, since it requires the
computation of vector matrix multiplications and additions. On the other hand,
the LS estimation methods usually require a matrix inversion.

Possible generalizations of the present work include the channel estimation
problem in RIS-aided multiuser MIMO systems, and the joint optimization of the
active and passive beamforming at the BS and at the RIS, respectively.

## References

  * [1] Q. Wu and R. Zhang, “Beamforming optimization for wireless network aided by intelligent reflecting surface with discrete phase shifts,” _IEEE Trans. Commun._ , pp. 1–1, 2019.
  * [2] C. Liaskos, A. Tsioliaridou, A. Pitsillides, S. Ioannidis, and I. F. Akyildiz, “Using any surface to realize a new paradigm for wireless communications,” _Communications of the ACM_ , vol. 61, pp. 30–33, 2018.
  * [3] M. Di Renzo et al., “Smart radio environments empowered by reconfigurable AI meta-surfaces: an idea whose time has come,” _EURASIP J. Wireless Comm. and Networking_ , vol. 2019, p. 129, 2019.
  * [4] H. Wymeersch, J. He, B. Denis, A. Clemente, and M. Juntti, “Radio localization and mapping with reconfigurable intelligent surfaces: Challenges, opportunities, and research directions,” _IEEE Veh. Technol. Mag._ , vol. 15, no. 4, pp. 52–61, 2020.
  * [5] M. Di Renzo et al., “Smart radio environments empowered by reconfigurable intelligent surfaces: How it works, state of research, and the road ahead,” _IEEE J. Sel. Areas Commun._ , vol. 38, no. 11, pp. 2450–2525, 2020.
  * [6] E. Basar et al., “Wireless communications through reconfigurable intelligent surfaces,” _IEEE Access_ , vol. 7, pp. 116 753–116 773, 2019\.
  * [7] J. He, M. Leinonen, H. Wymeersch, and M. Juntti, “Channel estimation for RIS-aided mmWave MIMO channels,” in _proc. IEEE Global Communications Conference_ , 2020, pp. 1–6.
  * [8] J. He, H. Wymeersch, and M. Juntti, “Channel estimation for RIS-aided mmWave MIMO systems via atomic norm minimization,” _IEEE Trans. Wireless Commun._ , pp. 1–1, 2021.
  * [9] Z. He and X. Yuan, “Cascaded channel estimation for large intelligent metasurface assisted massive MIMO,” _IEEE Wireless Commun. Lett._ , vol. 9, no. 2, pp. 210–214, Feb 2020.
  * [10] A. Zappone, M. Di Renzo, and M. Debbah, “Wireless networks design in the era of deep learning: Model-based, AI-based, or both?” _IEEE Trans. Commun._ , vol. 67, no. 10, pp. 7331–7376, 2019.
  * [11] A. M. Elbir, A. Papazafeiropoulos, P. Kourtessis, and S. Chatzinotas, “Deep channel learning for large intelligent surfaces aided mm-wave massive MIMO systems,” _IEEE Wireless Commun. Lett._ , vol. 9, no. 9, pp. 1447–1451, 2020.
  * [12] N. Samuel, T. Diskin, and A. Wiesel, “Learning to detect,” _IEEE Trans. Signal Process._ , vol. 67, no. 10, pp. 2554–2564, 2019.
  * [13] K. Gregor and Y. LeCun, “Learning fast approximations of sparse coding,” in _proc. international conference on machine learning_ , 2010, pp. 399–406.
  * [14] N. Landsberg and E. Socher, “A low-power 28-nm CMOS FD-SOI reflection amplifier for an active F-band reflectarray,” _IEEE Transactions on Microwave Theory and Techniques_ , vol. 65, no. 10, pp. 3910–3921, 2017.
  * [15] Y. Chen and Y. Chi, “Harnessing structures in big data via guaranteed low-rank matrix estimation: Recent theory and fast algorithms via convex and nonconvex optimization,” _IEEE Signal Process. Mag._ , vol. 35, no. 4, pp. 14–31, 2018.