#  KNOWLEDGE TRANSFER FROM LARGE-SCALE PRETRAINED LANGUAGE MODELS TO END-TO-
END SPEECH RECOGNIZERS

###### Abstract

End-to-end speech recognition is a promising technology for enabling compact
automatic speech recognition (ASR) systems since it can unify the acoustic and
language model into a single neural network. However, as a drawback, training
of end-to-end speech recognizers always requires transcribed utterances. Since
end-to-end models are also known to be severely data hungry, this constraint
is crucial especially because obtaining transcribed utterances is costly and
can possibly be impractical or impossible. This paper proposes a method for
alleviating this issue by transferring knowledge from a language model neural
network that can be pretrained with text-only data. Specifically, this paper
attempts to transfer semantic knowledge acquired in embedding vectors of
large-scale language models. Since embedding vectors can be assumed as
implicit representations of linguistic information such as part-of-speech,
intent, and so on, those are also expected to be useful modeling cues for ASR
decoders. This paper extends two types of ASR decoders, attention-based
decoders and neural transducers, by modifying training loss functions to
include embedding prediction terms. The proposed systems were shown to be
effective for error rate reduction without incurring extra computational costs
in the decoding phase.

Index Terms—  Automatic speech recognition, knowledge transfer, multi-task
learning, pretrained language models, BERT

##  1 Introduction

End-to-end techniques are known to be beneficial for realizing compact models
for automatic speech recognition (ASR). Since the compact models are
advantageous both in terms of interface responsiveness and energy consumption,
an accuracy improvement in end-to-end ASR is a crucial step for expanding the
availability of speech recognition technologies in, for example mobile devices
[1].

One of the most promising approaches for improving end-to-end ASR accuracy is
introducing additional training data since end-to-end models typically require
large amount of training data for mitigating over-fitting. However, obtaining
labeled training data is often costly compared to obtaining unpaired audio-
only and/or text-only data. End-to-end systems model speech recognition as one
unified model as opposed to a conventional system that factors the problem
into several models and separates the language model (LM) component from the
acoustic model (AM) component. Hence, training from text-only data can only be
realized in the form of an external LM which leads to additional complexity
and run-time costs.

Recently, a series of important methods for utilizing unpaired audio-only data
is proposed as pretraining methods of the end-to-end models [2, 3, 4]. Those
methods pretrain the encoder part of end-to-end ASR models by introducing a
surrogate objective function based on mutual information between instantaneous
feature representation and global context representation. Since those
pretraining processes can be performed with audio-only data, those methods are
promising for relaxing the limitation due to availability of paired training
data.

Previous work has shown an effective way to make use of unpaired audio data,
the work here focus on a way to make use of unpaired text data without raising
the model size and inference complexity. There are several methods for
integrating external language models trained on text-only data into end-to-end
speech recognition [5]. However, since language models are usually very large
in terms of model size and computational complexity, the use of external LMs
makes the model less compact and raises the computation cost. Recent language
models, such as bidirectional encoder representations from transformers (BERT)
[6], involve typically several hundred millions of parameters, and integrating
it is computationally prohibitive for small devices even though the language
model performs well and is in that respect desirable.

In this paper, learning from text-only data is achieved by designing multi-
task learning that performs knowledge distillation [7]. Applications of
transfer learning do not require any additional computational cost in the
recognition phase, and thus it is advantageous for models being deployed to
small devices. There are several recent attempts to distill knowledge from
external models into end-to-end speech recognizers [8, 9, 10, 11]. The prior
studies that are most related to our approach are based on distillation from a
masked language model [10, 11]. For transferring knowledge from a pretrained
BERT model, [10] minimizes the KL-divergence between the token posterior
distribution from the BERT masked language model and the ASR model. In
contrast, our approach focuses on using embedding vectors rather than
posterior distribution of the masked LM. As suggested in the experiments in
the original BERT paper [6], embedding vectors are believed to be a compact
representation containing richer information compared to the posterior
distributions of tokens. On the other hand, knowledge transfer from embedding
vectors is shown to be effective for improving performance of non-auto-
regressive speech recognizers [11]. In this paper, the effectiveness of
knowledge transfer from embedding vectors is verified with attention- [12] and
transducer-based [13] speech recognition models. Since the transducer-based
approach is popular in ASR for computationally limited devices as described in
[1], it is important to analyze the effectiveness of knowledge transfer for
such models.

##  2 Knowledge distillation from embeddings

This section describes methods to enhance existing end-to-end ASR neural
networks to multitask models that predict precomputed token-embedding vectors
in addition to the token distributions. In this section, each element in the
training dataset is denoted as a triple
$(\bm{\mathrm{X}},\bm{y},\bm{\mathrm{E}})$ of the feature vector sequence
$\bm{\mathrm{X}}$, token sequence $\bm{y}=(y_{1},y_{2},\cdots)$, and the
sequence of token embedding vectors
$\bm{\mathrm{E}}=(\bm{e}_{1},\bm{e}_{2},\cdots)$. Each token $y_{i}\in V$
represents a subword token, typically a word-piece [14], of the transcription.
The embedding vectors $\bm{\mathrm{E}}$ and the token sequences $\bm{y}$ are
related by a word embedding function
$\mathcal{M}:V^{N}\to(\mathbb{R}^{D^{{\mathtt{Emb}}}})^{N}$ as
$\bm{\mathrm{E}}=\mathcal{M}(\bm{y})$, and therefore the numbers of word
tokens and the token-embedding vectors in the same triple are the same. In
this paper, $\mathcal{M}$ is represented in a pretrained neural network, and
its parameters are not fine-tuned. Specifically, in the experimental section
below, we used BERT embeddings as $\mathcal{M}$.

###  2.1 Attention-based decoder with auxiliary regression

Fig. 1: Block diagrams of (a) attention-based model with a regression net, and
(b) transducer-based model with a regression net.

An attention-based auto-regressive decoder predicts the probability
distribution of the next token $y_{i}$ using the current decoder state
$\bm{\phi}(\bm{\mathrm{X}},\bm{y}_{1:i-1})$ as,

$p(y_{i}\mid\bm{\mathrm{X}},\bm{y}_{1:i-1})\propto\exp\left(\bm{\lambda}_{y_{i}}^{\mathsf{T}}\bm{\phi}(\bm{\mathrm{X}},\bm{y}_{1:i-1})\right).$
(1)

Here, $\bm{\lambda}_{y_{i}}$ is the parameter vector used for computing a
logit for $y_{i}$, and
$\bm{\phi}(\bm{\mathrm{X}},\bm{y}_{1:i-1})\in\mathbb{R}^{D}$ is a vector
referred to as the “pre-softmax activation”. Typically, $\bm{\phi}$ is
equipped with an attention mechanism to summarize the encoded representation
of $\bm{\mathrm{X}}$.

A cross-entropy loss is defined by using the above prediction as,

$L^{{\mathtt{XEnt}}}(\bm{\mathrm{X}},\bm{y})=-\sum_{i}\log
p(y_{i}\mid\bm{\mathrm{X}},\bm{y}_{1:i-1}).$ (2)

The conventional training for end-to-end models with an attention-based
decoder minimizes the expectation of this objective function using a gradient-
based method such as Adam [15].

In this paper, aiming at transferring knowledge from token-embedding vectors
to the decoder state vector $\bm{\phi}$, a regression neural network
$\mathcal{R}:\mathbb{R}^{D}\to\mathbb{R}^{D^{{\mathtt{Emb}}}}$ is introduced.
With this additional component, the following loss function
$L^{{\mathtt{Emb}}}$ based on the distances between the estimated embeddings
and the precomputed embeddings $\bm{e}_{i}$ is introduced as an auxiliary loss
function to be minimized.

$L^{{\mathtt{Emb}}}(\bm{\mathrm{X}},\bm{y},\bm{\mathrm{E}})=\sum_{i}d(\mathcal{R}(\bm{\phi}(\bm{\mathrm{X}},\bm{y}_{1:i-1})),\bm{e}_{i}),$
(3)

where $d$ is a differentiable distance function such as Lp distances.

With the above auxiliary loss function, the loss function to be minimized in
training can be defined as,

$L^{{\mathtt{XEnt}}}(\bm{\mathrm{X}},\bm{y})+\sigma
L^{{\mathtt{Emb}}}(\bm{\mathrm{X}},\bm{y},\bm{\mathrm{E}}).$ (4)

Here, $\sigma$ is a hyper-parameter that controls the relative importance of
the auxiliary task. Setting $\sigma=0$ reduces the training process to the
conventional cross-entropy on the token probabilities alone.

Fig 1-(a) depicts neural net architecture for implementing training with this
loss function. The “Regression Net” and “Pretrained Embedding Model”
components in the figure can be pruned in the recognition phase. Therefore,
the total computational complexity at the recognition phase is not increased
by this modification.

###  2.2 Transducer-based decoder with auxiliary regression

Neural transducers (also known as recurrent neural network transducers (RNN-
Ts)) [13] model a probability distribution of alignment variables
$\bm{z}=(z_{1},z_{2},\cdots)\in(V\cup\left\\{\varphi\right\\})^{T+N}$ where
$T$ and $N$ are the lengths of a feature vector and a word sequence,
respectively, and $\varphi$ is a blank symbol. The token sequence $\bm{y}$ is
considered to be extracted by using a blank removal function $\mathcal{B}$
that removes $\varphi$ from the alignment sequence $\bm{z}$. Therefore, the
predictive distribution of the token sequences can be expressed by
marginalizing the alignment variables, as follows:

$p(\bm{y}\mid\bm{\mathrm{X}})=\sum_{\bm{z}^{\prime}}\mathbf{1}\left[\mathcal{B}(\bm{z}^{\prime})=\bm{y}\right]p(\bm{z}^{\prime}\mid\bm{\mathrm{X}}).$
(5)

Here, $\mathbf{1}\left[P\right]$ is an indicator function whose value is $1$
if the predicate $P$ is true, and $0$ otherwise. It should be noted that
$\mathbf{1}\left[\mathcal{B}(\bm{z}^{\prime})=\bm{y}\right]$ can be viewed as
a hard-coded model of $p(\bm{y}\mid\bm{z}^{\prime})$.

The probability distribution over the alignment $p(\bm{z}\mid\bm{\mathrm{X}})$
is computed depending on outputs of two neural networks, the transcription and
prediction networks. The transcription network computes feature vectors
$\bm{\phi}_{t}$ for each frame $t$. The prediction network computes the
representation from the prefix of a (hypothesis) token sequence. In the
training phase with teacher forcing, the prediction network computes output
vectors $\bm{\psi}_{i}$ for each token $y_{i}$ using the token prefix
$\bm{y}_{1:i-1}$. In this paper, $\bm{\phi}_{t}$ and $\bm{\psi}_{i}$ are
called acoustic and language feature vectors.

Using those two feature vectors, the joint network $\mathcal{J}$ computes the
probability distribution over the alignment variable $\bm{z}$ as,

$p(z_{n}=k\mid\bm{z}_{1:n-1},\bm{\mathrm{X}},\bm{y})=\mathcal{J}_{k}(\bm{\phi}_{1+\tau(\bm{z}_{1:n-1})},\bm{\psi}_{1+\iota(\bm{z}_{1:n-1})}),$
(6)

where $\mathcal{J}_{k}(.)$ is the $k$-th element of the output vector of the
joint network, $\tau(\bm{z}_{1:n-1})$ and $\iota(\bm{z}_{1:n-1})$ are the
numbers of blank $\varphi$ and non-blank symbols, respectively, in the prefix
$\bm{z}_{1:n-1}$.

#### Joint regression network

Here, similar to attention-based decoder modeling, a regression network
$\mathcal{R}$ is introduced to predict token-embedding vectors
$\bm{\mathrm{E}}$. The auxiliary loss can be defined as an expectation of
distances as,

$L^{{\mathtt{Emb}}}(\bm{\mathrm{X}},\bm{y},\bm{\mathrm{E}})=\sum_{i}\left\langle
d(\mathcal{R}(\bm{\phi}_{t},\bm{\psi}_{i}),\bm{e}_{i})\right\rangle_{q_{i}(t\mid\bm{\mathrm{X}},\bm{y})}.$
(7)

It should be noted that the regression network here takes two input vectors
similar to the joint network. The alignment probability
$q_{i}(t\mid\bm{\mathrm{X}},\bm{y})$ is a probability of $t$-th acoustic
feature vector being consumed after processing the prefix $\bm{y}_{1:i-1}$,
and can be expressed as,

$\displaystyle
q_{i}(t\mid\bm{\mathrm{X}},\bm{y})\stackrel{{\scriptstyle\mathrm{def}}}{{=}}$
(8)
$\displaystyle\sum_{\bm{z}^{\prime}}\sum_{n}\mathbf{1}\left[\iota(\bm{z}^{\prime}_{1:n})=i-1\land\tau(\bm{z}^{\prime}_{1:n})=t-1\right]p(\bm{z}^{\prime}\mid\bm{\mathrm{X}},\bm{y}).$

The expectation over this posterior distribution can be computed by reusing
the results of the forward-backward algorithm used in the computation of the
conventional transducer loss function. It should be noted that gradient
information was not propagated through this $q$ function in the experimental
section below, to ensure that auxiliary loss does not affect to alignment
computation.

Fig 1-(b) depicts an example neural net architecture for this training. The
expectation in Eq. (7) can efficiently be computed by reusing the alignment
variable (specifically, forward and backward score) from the forward-backward
computation in the main transducer loss.

#### Token-synchronous regression network

As an alternative to the loss function described above, we investigated
another loss that has a single regression result per token.

This variant approximates the expectation of the loss functions in Eq. (7) by
using the expectation of the acoustic features as,

$L^{{\mathtt{Emb}}}(\bm{\mathrm{X}},\bm{y},\bm{\mathrm{E}})\simeq\sum_{i}d(\mathcal{R}(\left\langle\bm{\phi}_{t}\right\rangle_{q_{i}(t\mid\bm{\mathrm{X}},\bm{y})},\bm{\psi}_{i}),\bm{e}_{i}).$
(9)

This approximated loss is equivalent to Eq. (7) when an L2 distance metric and
a linear network are used as $d$ and $\mathcal{R}$, respectively.

This approximation reduces the training memory consumption regarding the
regression network from $O(TN)$ to $O(N)$. Furthermore, the regression network
in this architecture is trained to make a single prediction per token whereas
the joint regression network in Eq. (7) makes multiple predictions minimizing
the expectation of the loss. Therefore, this approximation is also beneficial
as the token-wise predictions can be helpful for subsequent processing of the
ASR results.

##  3 Experiments

We verified the effectiveness of the proposed methods by training models on
the LibriSpeech dataset [16] (960h of transcribed speech), and using a BERT
[6] language model pretrained with the BooksCorpus (800M words) [17] and
Wikipedia text data (2500M words).

###  3.1 Experimental Setup

The attention-based model we used was based on bi-directional long short-term
memory (BLSTMs). The configurations for the BLSTM encoder and decoder were set
to be identical with those of [18]. The transducer-based model employed a
Conformer encoder [19]. The encoder consisted of 17 Conformer blocks, and each
block was equipped with an 8-head dot-attention, 512-channel 1d-convolution,
and 2048-dim feed-forward module. The prediction network in the transducer-
based model was an LSTM consisting of a 640-dimensional (uni-directional) long
short-term memory (LSTM) layer and a 128-dimensional token-embedding layer.
The joint network in the transducer-based model was a feed-forward network
with a 640-dim hidden layer with $\tanh$-activation.

SpecAugment [20] was applied in the transducer-based systems. The numbers of
time- and frequency-masks were set to 10 and 2, respectively. The lengths of
the time- and frequency-masks were set to $0.05T$ and $27$, respectively,
where $T$ is the length of the input feature vector sequence. Both for the
attention and transducer-based models, Adam [15] was used as the optimization
method. The batch sizes were set to $512$ and $2048$ for the attention and
transducer-based models, respectively. The optimization results of transducer-
based systems was smoothed by computing exponential moving average (EMA) of
the parameter trajectory. The decay rate for EMA was set to $0.9999$. For each
training configuration, the best checkpoint was chosen based on the WERs
observed on the devother partition.

The weight for the auxiliary task ($\sigma$ in Eq. (4)) was varied for
analyzing the effect of the auxiliary task. For attention-based models, the
weights were selected from $\sigma\in\\{2^{0},2^{-1},2^{-2},2^{-3},2^{-4}\\}$,
and for transducer-based models, the weights were selected from
$\sigma\in\\{10^{-1},10^{-2},10^{-3},10^{-4}\\}$. As it is mentioned above,
setting $\sigma=0$ is equivalent with omitting the auxiliary task, and was
considered as a baseline setting.

The distance metric used in the auxiliary loss function was chosen as the L1
distance between the predicted and target vectors normalized (divided) by the
dimensionality of the embedding vectors ($D^{{\mathtt{Emb}}}=768$). The
regression network $\mathcal{R}$ was defined by a single affine
transformation. For the transducer-based decoder, we first used a token-
synchronous regression module, i.e. the loss function in Eq. (9). As described
in the previous section, Eqs. (9) and (7) are identical if we use an L2
distance and a linear regression network. We used an L1 distance in this
experiment and expected a subtle difference which we analyzed further in the
results below.

The pretrained BERT model we used is identical with one available on TFHub
[21]. This BERT embedding module consists of 12 transformer blocks with 768
hidden units and 12 self-attention heads. For generating the target embedding
vector sequences for training, the transcriptions in the training data were
tokenized and fed into the BERT embedding module. In this experiment, since we
adopted BERT tokens using 30k word-piece vocabulary, the ASR models were also
trained with this tokenization method. Since word-piece models used in the
baseline ASR model did not match those used for BERT modeling, we also
investigated the effect of using word-piece models developed for BERT in ASR
111We kept special tokens for BERT modeling, such as [CLS] and [SEP] as ASR
output target sequence..

###  3.2 Main results

Table 1 shows the word error rates (WERs) of the baseline and the proposed
training methods applied to attention-based models. By comparing “Baseline”
and “BERT tok.”, we observe that the use of a different tokenization strategy
did not degrade the word error rates significantly. By increasing, the
auxiliary task weight $\sigma$, we observed a word error rate reduction in all
the test sets. The best configuration for all the test sets is obtained when
$\sigma$ was set to $2^{-2}$ where the relative error rate reduction was
-16.7% and -10.6% for the testclean and testother datasets. By increasing the
auxiliary weight further, we observed a performance degradation.

LAS |  | dev | test
---|---|---|---
| $\sigma$ | clean | other | clean | other
Baseline | – | 4.5 | 13.5 | 4.7 | 13.7
BERT tok. | 0 | 4.6 | 13.2 | 4.8 | 14.1
+BERT reg. | 0.0625 | 4.2 | 12.1 | 4.3 | 13.2
| 0.125 | 4.0 | 11.8 | 4.2 | 12.7
| 0.25 | 3.9 | 11.6 | 4.0 | 12.6
| 0.5 | 4.1 | 12.3 | 4.2 | 13.2
| 1.0 | 4.2 | 13.0 | 4.2 | 13.8
Table 1: WERs of attention-based models: ”BERT tok.” denotes a system with the
baseline training method and word-piece model from pretrained BERT model.
”+BERT reg.” denotes the systems with auxiliary regression task with varying
task weights.

Table 2 shows the word error rates of the baseline and the proposed training
methods applied to transducer-based models. The best configuration for
devclean is obtained when $\sigma$ was set to $10^{-2}$ where the relative
error rate reduction was -3.7% and -11.3% for testclean and testother
datasets.

ConformerS |  | dev | test
---|---|---|---
| $\sigma$ | clean | other | clean | other
Baseline | – | 2.6 | 6.8 | 2.7 | 6.3
BERT tok. | 0 | 2.6 | 5.7 | 2.7 | 6.2
+BERT reg. | 0.0001 | 2.5 | 5.5 | 2.6 | 5.6
| 0.001 | 2.4 | 5.2 | 2.7 | 5.7
| 0.01 | 2.4 | 5.3 | 2.6 | 5.5
| 0.1 | 2.4 | 5.3 | 2.8 | 5.9
Table 2: WERs of transducers (see also Table 1 for legends.)

From both applications, we confirmed that the multitask learning with an
auxiliary regression to the word embeddings were beneficial for transferring
semantic knowledge from the pretrained BERT language model. Since setting the
auxiliary weight $\sigma$ to zero converges to the single task model, we
observed WER convergence to the baseline model by reducing $\sigma$. We also
observed subtle differences due to the tokenization method we employed. The
WordPiece models we took from the BERT model performed better with our
Conformer models. This might be due to the fact that the Conformer has more
expressive ability and the larger inventory of tokens in BERT-based
tokenization worked well with this setting.

###  3.3 Comparison between token-synchronous regression and joint regression
net

In the previous section, we verified the effectiveness of our method applied
to transducer-based decoders using token-synchronized regression shown in Eq.
(9). Even though the difference between the two variants proposed in Section
2.2 is subtle, they are not identical in this experiment since an L1 distance
is used as the distance metric.

ConformerS |  | dev | test
---|---|---|---
|  Best $\sigma$ | clean | other | clean | other
BERT tok. | 0 | 2.6 | 5.7 | 2.7 | 6.2
Token-sync. regr. | 0.01 | 2.4 | 5.3 | 2.6 | 5.5
Joint regr. | 0.01 | 2.5 | 5.1 | 2.7 | 5.8
Table 3: Comparison between token-synchronized regression and joint regression
networks.

Table 3 summarizes the results of the model with joint regression and with
token-synchronized regression networks. The best auxiliary weight was chosen
based on the results on devother. From the table, we could not observe clear
performance difference between the loss functions compared (Eqs. (7) and (9)).

As mentioned in Section 2.2, the token-synchronized regression network has
qualitative advantages. Therefore, in this setting, we confirmed that the use
of token-synchronized regression network was more relevant. However, the
accuracy difference might be larger if we were to use more complicated
regression network than a linear regression network.

###  3.4 Results with pretrained acoustic encoders

As an additional experiment, we combined the proposed training method with a
large-scale pretrained acoustic encoder. Application for a model like that is
most promising as it opens the door for end-to-end models to use both unpaired
audio and text data. Specifically, we evaluated the proposed methods with
Conformer-based wav2vec-2.0 pretraining method [4]. The pretrained encoder and
the decoder setting were identical with those of “Conformer-XL” in [4]
pretrained with LibriLight dataset (60k hours). As in the previous
experiments, the hyper-parameters were chosen to minimize the word-error rates
over the devother test set.

Pretrained-ConfXL |  | dev | test
---|---|---|---
|  Best $\sigma$ | clean | other | clean | other
Baseline | 0 | 1.77 | 3.47 | 1.75 | 3.56
+BERT reg. | 0.01 | 1.73 | 3.40 | 1.80 | 3.51
Table 4: WERs with pretrained acoustic encoders

Table 4 shows the WERs of the proposed training methods combined with the
pretrained encoder. We confirmed that the proposed method could further reduce
the word error rate even from the strong baseline with a pretrained encoders.
However, the advantage is relatively small and this suggest that a part of
semantic information might already be captured in wav2vec-2.0 training even
without using a transcription. Future work will focus on improving the
configuration for this type of combination.

##  4 Conclusion

This paper proposed a simple method to utilize text-only data and pretrained
word embedding models for enhancing performance of speech recognition without
increasing the model size. The proposed method is based on multi-task learning
consisting of a main speech recognition task and an auxiliary word-embedding
regression task. The multi-task learning method is applied to both attention
and transducer-based end-to-end speech recognizers. We confirmed that, with
including the auxiliary task as our methods propose, we were able to improve
the transcription accuracies of the resulting systems. For attention-based
models, the error reductions were 16.7% and 10.6% for the testclean and
testother datasets, respectively. For transducer-based models, the error
reductions were 3.7% and 11.3% for the testclean and testother datasets,
respectively.

Future work will focus on the usability of token-embeddings, obtained as a by-
product of this model, for down-stream natural language processing tasks. Even
though we confirmed that a training process with the proposed method can
reduce the distance metric between the estimated and precomputed token-
embedding vectors, it is not clear if the estimated vectors are beneficial
for, for example, an utterance classification task.

## References

  * [1] Yanzhang He, Tara N Sainath, Rohit Prabhavalkar, Ian McGraw, Raziel Alvarez, Ding Zhao, David Rybach, Anjuli Kannan, Yonghui Wu, Ruoming Pang, et al.,  “Streaming end-to-end speech recognition for mobile devices,”  in Proc. ICASSP, 2019, pp. 6381–6385.
  * [2] Aäron van den Oord, Yazhe Li, and Oriol Vinyals,  “Representation learning with contrastive predictive coding,”  arXiv preprint arXiv:1807.03748, 2018.
  * [3] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli,  “wav2vec 2.0: A framework for self-supervised learning of speech representations,”  in Proc. NeurIPS, 2020.
  * [4] Yu Zhang, James Qin, Daniel S Park, Wei Han, Chung-Cheng Chiu, Ruoming Pang, Quoc V Le, and Yonghui Wu,  “Pushing the limits of semi-supervised learning for automatic speech recognition,”  arXiv preprint arXiv:2010.10504, 2020.
  * [5] Shubham Toshniwal, Anjuli Kannan, Chung-Cheng Chiu, Yonghui Wu, Tara N Sainath, and Karen Livescu,  “A comparison of techniques for language model integration in encoder-decoder speech recognition,”  in Proc. IEEE Workshop on SLT, 2018, pp. 369–375.
  * [6] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova,  “BERT: Pre-training of deep bidirectional transformers for language understanding,”  in Proc. NAACL-HLT, Vol. 1, 2019, pp. 4171–4186.
  * [7] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean,  “Distilling the knowledge in a neural network,”  arXiv preprint arXiv:1503.02531, 2015.
  * [8] Gakuto Kurata and George Saon,  “Knowledge distillation from offline to streaming RNN transducer for end-to-end speech recognition,”  in Proc. INTERSPEECH, 2020, pp. 2117–2121.
  * [9] Sankaran Panchapagesan, Daniel S Park, Chung-Cheng Chiu, Yuan Shangguan, Qiao Liang, and Alexander Gruenstein,  “Efficient knowledge distillation for RNN-transducer models,”  in Proc. ICASSP, 2021, pp. 5639–5643.
  * [10] Hayato Futami, Hirofumi Inaguma, Sei Ueno, Masato Mimura, Shinsuke Sakai, and Tatsuya Kawahara,  “Distilling the knowledge of BERT for sequence-to-sequence ASR,”  in Proc. INTERSPEECH. 2020, pp. 3635–3639, ISCA.
  * [11] Ye Bai, Jiangyan Yi, Jianhua Tao, Zhengkun Tian, Zhengqi Wen, and Shuai Zhang,  “Fast end-to-end speech recognition via non-autoregressive models and cross-modal knowledge transferring from BERT,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 1897–1911, 2021.
  * [12] William Chan, Navdeep Jaitly, Quoc Le, and Oriol Vinyals,  “Listen, attend and spell: A neural network for large vocabulary conversational speech recognition,”  in Proc. ICASSP, 2016, pp. 4960–4964.
  * [13] Alex Graves,  “Sequence transduction with recurrent neural networks,”  in Proc. ICML Representation Learning Workshop, 2012.
  * [14] Mike Schuster and Kaisuke Nakajima,  “Japanese and Korean voice search,”  in Proc. ICASSP, 2012, pp. 5149–5152.
  * [15] Diederik P Kingma and Jimmy Ba,  “Adam: A method for stochastic optimization,”  arXiv preprint arXiv:1412.6980, 2014.
  * [16] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur,  “Librispeech: An asr corpus based on public domain audio books,”  in Proc. ICASSP, 2015, pp. 5206–5210.
  * [17] Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler,  “Aligning books and movies: Towards story-like visual explanations by watching movies and reading books,”  in Proc. ICCV, 2015, pp. 19–27.
  * [18] Kazuki Irie, Rohit Prabhavalkar, Anjuli Kannan, Antoine Bruguier, David Rybach, and Patrick Nguyen,  “On the choice of modeling unit for sequence-to-sequence speech recognition,”  in Proc. INTERSPEECH, 2019, pp. 3800–3804.
  * [19] Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang,  “Conformer: Convolution-augmented transformer for speech recognition,”  in Proc. INTERSPEECH, 2020, pp. 5036–5040.
  * [20] Daniel S. Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D. Cubuk, and Quoc V. Le,  “SpecAugment: A simple data augmentation method for automatic speech recognition,”  in Proc. INTERSPEECH, 2019, pp. 2613–2617.
  * [21] “TensorFlow Hub: `bert_en_uncased_l-12_h-768_a-12`,” https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/4, accessed on Sep. 27, 2021.


# Multi-market Energy Optimization with Renewables via Reinforcement Learning

Lucien Werner, Peeyush Kumar

##  1 Introduction

The decarbonziation of the bulk electric power system, whose generation mix
still mainly consists of gas and coal-fired thermal generation, is mainly
being driven by the rapidly falling costs of utility-scale wind and solar.
These variable generation sources produce no emissions when producing energy
but unlike conventional thermal power plants, they are not controllable—also
called dispatchable—and therefore can not modulate their output at will. In
some sense, nature determines the supply trajectory over time.

At low penetrations (e.g., 5-20%), variable renewable energy (VRE) can be
accommodated by the existing grid. However at the high penetrations needed for
a low-carbon grid, the variability and unpredictability of solar and wind pose
major challenges to a reliable power supply that is able to instantaneously
meet electrical demand. Even momentary supply-demand imbalances in electrical
grids can cause cascading blackouts and thus reliability is the major
challenge for a future grid powered mainly by renewables.

Energy storage in the form of batteries or pumped hydro has been widely
studied as a well to smooth out the fluctuations in VRE production. Co-
locating energy storage with renewable generation is a way for renewable
generation to make firm commitments in energy markets. This is important as it
reduces the amount of uncertainty and variability that the renewable generator
introduces into the balance of the power grid.

The operators of combined renewable plus storage plants seek to optimize the
revenue they obtain from energy markets. This will often include long-term
contracts with penalties for non-delivery (like PPAs), day-ahead and real-time
markets, as well as fast-timescale markets for frequency regulation. In this
work we study the combination of long-term and real-time markets, although the
modeling and optimization framework can be extended to include additional or
faster-timescale markets.

###  1.1 Contributions

This work contributes to two principle areas: energy market optimization and
reinforcement learning. The main novelty of the work is a stochastic
optimization framework for performing multi-market energy optimization. This
is the task of computing optimal energy dispatches in several, coupled energy
markets under uncertainty. Examples of such markets are long-term forward
contracts like power purchase agreements (PPAs) and capacity markets, as well
as more real-time scheduling markets like the day-ahead, real-time, and
frequency regulation markets. Each of these markets has a distinct transaction
timescales, which ranges from decades to seconds. The uncertainty arises from
the stochasticity of energy production by renewables (ultimately caused by
weather) and the unpredictability of short-timescale market prices.

Applying reinforcement learning to solve stochastic energy optimization
problems in an area of growing interest in the research community. However,
several challenges remain: specifically, learning safe policies that respect
physical and hard constraints and managing variability in exogenous
information during policy learning. In order to derive practical control
algorithms with an RL approach, both of these issues must be surmounted.

The challenges motivate the following research agenda:

  * •

Form multi-objective stochastic optimization framework for modeling multi-
market energy scheduling problem;

  * •

Guarantee hard constraint satisfaction for power systems;

  * •

Handle variability in exogenous data for prices and renewable generation.

###  1.2 Related Work

Reinforcement learning has increasingly been applied to complex control and
sequential decision-making problems in uncertain environments Arulkumaran et
al. (2017). Researchers have also explored applying RL in the power systems
context for problems such as voltage control, frequency control, and energy
scheduling. Zhang et al. (2019); Yang et al. (2020b). It is widely recognized
that safety and scalability and key issues that arise for algorithms for power
system applications Chen et al. (2021b). In general, RL policies are neither
guaranteed to satisfy real-world system constraints during training nor during
run-time. Our work directly addresses this challenge.

Existing works that focus on deep reinforcement learning for energy storage
management include Shang et al. (2020); Wang and Zhang (2018); Oh and Wang
(2020); Cao et al. (2020); Yang et al. (2020a); Mbuwir et al. (2017). None of
these approaches considers a general modeling framework that can accommodate
both storage and renewables for arbitrary systems.

Recent research on constraint satisfaction for power systems has two notable
contributions: Chen et al. (2021a) and Zheng et al. (2021). The former uses
differentiable convex layers to project actions onto a feasible set as part of
the policy learning. The latter takes a similar approach but proposes a novel
projection layer in the neural network that does not require computing the
solution map of a large convex optimization problem. The approach we take in
this paper is complementary to these proposals and exploits similar properties
of the projection problem (such as control-affine-ness) to incorporate the
projection step into the system dynamics.

Finally, Powell (2014); Powell and Meisel (2015); Powell (2021) form a body of
literature that focuses on multi-objective stochastic optimization for
multiple timescales. The solution approaches utilize classical dynamic
programming or approximate dynamic programming methods which either not
suitable for the continuous state-action space environment of our problem or
uses discretization schemes that are intractably large.

##  2 A Primer on Electricity Markets

Electricity markets are complex and varied, with regional differences in terms
of regulatory structures and market design but they share some common
similarities. For instance in the United States at a high level, these markets
can be divided into two primary types: regulated and deregulated markets
Litvinov et al. (2019).

In traditionally regulated markets, utilities are vertically integrated and
operate as monopolies within their territories, meaning that they own the
electricity generators and power lines, and customers only have the option to
buy power from them. To prevent overcharging, state regulators oversee how
these utilities set electricity prices, which are determined based on the
utility’s operating and investment costs as well as a "fair" rate of return on
those investments. These rates must be approved by the state’s public
utilities commission. Utilities in these regions must also seek state approval
for power plant investments, a process that often requires demonstrating the
necessity of proposed investments through an integrated resource planning
(IRP) process. This approach to regulation and monopolization means that
customers bear the risk of investments as utilities can recover their costs
through rates, regardless of how power plants perform.

Even though these vertically integrated utilities generate their own
electricity, they can trade with other utilities during times of need,
especially when it’s cheaper to purchase power than to generate it using their
own facilities. This is common in the western and southeastern United States,
and such transactions are regulated by the Federal Energy Regulatory
Commission (FERC).

Deregulated markets, on the other hand, began emerging in the 1990s when many
US states decided to create competition and lower costs Hogan (2016). This
restructuring required electric utilities to sell their generating assets,
leading to the creation of independent energy suppliers. The utilities
retained their transmission and distribution assets, which remain regulated In
these areas, retail deregulation allows customers to select their own electric
supplier, introducing competition for retail electricity prices. However,
transmission and distribution services are still provided by the local utility
company. Contracts with independent suppliers can have both pros and cons for
consumers, and only the generation portion of a customer’s utility bill is set
competitively.

In contrast to regulated states that plan for investment, deregulated states
use markets to determine which power plants are necessary for electricity
generation. Wholesale deregulation led to the creation of centralized
wholesale Cervigni and Perekhodtsev (2013) markets in which power is sold by
generators and bought by entities that sell it to consumers. Investment risks
in power plants fall to the electric suppliers and not to customers in
deregulated markets.

Following deregulation, Regional Transmission Organizations (RTOs) replaced
utilities as grid operators and became the operators of wholesale markets for
electricity. Many of these RTOs operate markets that span multiple states and
are therefore regulated by FERC, with the exception of the Texas RTO, the
Electric Reliability Council of Texas (ERCOT).

In deregulated retail utilities, electricity is purchased at market-determined
wholesale prices and sold to customers at similarly determined retail prices.
RTOs typically run three kinds of markets that determine these wholesale
prices: energy markets, capacity markets, and ancillary services markets.

Energy Markets: The main product in these markets is electricity itself.
Generators offer their available capacity for each hour of the day, while
utilities and other entities bid on the capacity they need to serve their
customers. The market operator, typically a Regional Transmission Organization
(RTO), then matches supply with demand and determines the price for each hour.
Capacity Markets: These markets operate in parallel to energy markets and
involve the trade of generation capacity, rather than energy itself. The
purpose of capacity markets is to ensure that there is enough power generation
capacity available to meet peak demand. They provide a means for generators to
be compensated for their availability, regardless of whether they are actually
generating electricity. The rules for capacity markets vary by region, but
they generally involve auctions in which generators bid their available
capacity, and utilities and other entities pay for the capacity they need to
ensure reliability Fabra (2018).
Ancillary Services Markets: These markets involve the trade of services that
are essential to maintaining the reliability and security of the power grid,
but that are not captured in the energy or capacity markets. Examples of
ancillary services include frequency regulation, voltage control, and
operating reserves. Like energy and capacity, ancillary services are typically
coordinated by RTOs, and prices are determined based on supply and demand
Helman et al. (2009).

Figure 1: Electricity Markets

Electricity markets operate on various timescales, with day-ahead markets
facilitating transactions based on the next day’s forecasted load, while real-
time markets adjust to meet the actual demand for electricity, which can vary
minute to minute. Capacity markets, plan for peak electricity demand years in
advance, often through an auction system Tang et al. (2022). Figure 1 display
the multiple timescales of these electricity markets.

##  3 Problem Formulation

In this section we cast the multi-market energy optimization problem as a
stochastic optimization problem over policies. In order to accommodate the
uncertainty in the exogenous prices and VRE production, we consider searching
for an optimal policy (of system state and information) rather than seeking an
optimal sequence of decisions, as is done in deterministic optimization.

First we pose the problem as a Markov decision process (MDP) and detail the
models for the energy system and the environment (including exogenous
processes). In Section 4 we present the method for solving the stochastic
optimization problem derived from the MDP.

We consider a discrete, finite time horizon of length $T$ indexed by $t$:

$t\in\\{1,\ldots,T\\}.$

We assume that the state of the system and the information state (exogenous
and endogenous) are unchanged over the duration of interval $t$. The length of
interval $t$ is determined by the granularity of the input data; for example
this could be 5 mins if nodal price data is used or 1 hr if exchange market
data is used.

The system is modeled as a microgrid attached to the bulk power grid at a
single bus. This microgrid includes renewable generation, energy storage, and
conventional controllable loads or generation. This framework could describe a
variety of realistic situations, including a hybrid renewable-storage power
plant, a commercial building with rooftop solar and aggregated controllable
electric vehicles, or an islandable microgrid.

Following the notation from Powell and Meisel (2015) we represent the
controllable state of the system—which includes the battery state-of-charge
(SOC), the renewable energy injections, the injections from controllable
loads/generation, and the commitments to both markets–with the vector
$X_{t}$111In general $X_{t}$ is a function.. The exogenous variables are real-
time prices and maximum renewable energy production and are collected in the
random vector $W_{t}$. The available actions are given by the vector function
$U_{t}$; actions include the injections (charge/discharge) from storage.
Finally the system transition function, which is deterministic given realized
uncertainty $W_{t}$, is given by

$X_{t+1}=f(X_{t},U_{t},W_{t}).$

We will detail all of these components subsequently in this section.

###  3.1 Environment Model

A key feature of our model is state and action constraints. Such constraints
are crucial in the context of power systems, as there are hard physical
constraints like max charge/discharge for energy storage that cannot be
violated as well as physical laws like conservation of power (at the PCC)
that—if violated—would render a policy meaningless.

Now we present the notation and constraints for all of the system components.

####  3.1.1 Energy storage

There $N_{B}$ storage devices, indexed with $b\in\\{1,\ldots,N_{B}\\}$.222We
will use the terms storage and battery interchangeably, even though the
storage device will be modeled to allow for more diverse storage technologies
such as pumped hydro, compressed gas, and mechanical storage. The state-of-
charge (SOC) of each storage device, which represents how much energy in Watt-
hours (Wh) the device holds at time $t$, is denoted
$X_{b,t}\in\mathbb{R}_{+}$. The vector of SOC values over the entire time
horizon is denoted $\mathbf{x}_{b}$.333The boldface notation generally
represents vectors of decisions. Although the state is in general a function
we will sometime use the vector notation for convenience. The SOC of each
storage resource is assumed to be bounded in each time period between positive
lower/upper bounds:

$\underline{x}_{b,t}\leq X_{b,t}\leq\overline{x}_{b,t}$ (1)

Note that when $\underline{x}_{b,t}=\overline{x}_{b,t}$ for some $t$, the SOC
is fixed. This can be used to enforce initial or terminal SOC constraints.

Charge/discharge actions change the SOC of the storage device. We model charge
and discharge actions separately, using $U^{+}_{b,t}\in\mathbb{R}_{+}$ for the
former and $U^{-}_{b,t}\in\mathbb{R}_{+}$ for the latter. Both charge and
discharge values are restricted to be positive. Vectors of charge/discharge
decisions over the entire time horizon are given by $\mathbf{u}_{b}^{+}$ and
$\mathbf{u}_{b}^{-}$. Charging limits are assumed in each time period:

$\displaystyle\underline{u}^{+}_{b,t}\leq u^{+}_{b,t}$
$\displaystyle\leq\overline{u}^{+}_{b,t}$ (2)
$\displaystyle\underline{u}^{-}_{b,t}\leq u^{-}_{b,t}$
$\displaystyle\leq\overline{u}^{-}_{b,t}$ (3)

The SOC and charge/discharge actions are related through the following SOC
update equation from time $t$ to time $t+1$:

$x_{b,t+1}=x_{b}+f_{b}^{+}(u^{+}_{b,t})-f_{b}^{-}(u^{-}_{b,t})$ (4)

The functions $f^{+}$ and $f^{-}$ map to positive values and represent the
possibly non-linear dependence of the SOC on the charge/discharge actions. A
simple version of these functions is

$\displaystyle f_{b}^{+}(u^{+}_{b,t})$ $\displaystyle=\eta^{+}_{b}u^{+}_{b,t}$
$\displaystyle f_{b}^{-}(u^{-}_{b,t})$ $\displaystyle=\eta^{-}_{b}u^{-}_{b,t}$

where $\eta_{b}^{\pm}$ are constant charge/discharge efficiencies. In general,
$f^{\pm}$ can be non-linear functions of both $u$ and $x$. Also notice that
(4) does not preclude simultaneous charging and discharging. This may be
desirable or even optimal in some context (e.g., for pumped hydro storage) but
can be undesirable for chemical storage. Nazir and Almassalkhi (2021), Garifi
et al. (2020) discuss conditions under which simultaneous charging and
discharging can occur.

We also model a cost of charge/discharge for each energy storage resource
$C_{b}$. In general, the cost function can depend explicitely on the SOC and
the particular charge/discharge actions, as well as any of the previously
realized actions. Therefore we will represent the cost function with the
notation

$C_{b,t}(\mathbf{x}_{b}[0:t],\mathbf{u}_{b}^{+}[0:t],\mathbf{u}_{b}^{-}[0:t])$

where $\mathbf{x}_{b}[0:t]$ is the sub-vector of SOC values up to time $t$.

The difficulty in modeling storage devices comes from the forms of $C_{b,t}$
and $f_{b}^{\pm}$. These functions are non-linear and in the case of the cost
function, depends on the entire charging profile rather than just a discrete
charging action. Our model allows either the typical linear simplifications or
black-box simulators that ingest inputs and return a cost. In Section 5 we
discuss the particular choices we make in the construction of our simulator.

####  3.1.2 Renewable Generation

There are $N_{R}$ renewable generators, indexed with
$r\in\\{1,\ldots,\mathbb{N}_{R}\\}$. The variable $x_{r,t}\in\mathbb{R}_{+}$
represents the power (W) that generator $r$ produces in time $t$. The vector
of generation of the time horizon is denoted $\mathbf{x}_{r}$. The power from
the generator can be curtailed but cannot be increased beyond the available
power. The max available power in any period is upper bounded by the nameplate
capacity. Therefore we impose the following constraints on $x_{r,t}$:

$0\leq x_{r,t}\leq\overline{x}_{r,t}\leq x_{r}^{\mathrm{nameplate}}$

Given the fast ramping capabilities of renewable generation, no ramping
constraints are placed on generation quantities between successive time
periods. Further, we assume that there can be curtailment costs but no
variable operation costs.

####  3.1.3 Controllable loads and generation

There are $N_{C}$ controllable loads or generators, indexed with
$i\in\\{1,\ldots,N_{R}\\}$. These are simply power injections (either positive
or negative) that can be dispatched by the controller. The variable
$x_{i,t}\in\mathbb{R}$ represent the power in watts that generator/load $i$
produces/consumes in time $t$. The vector of generation over the time horizion
is denoted $\mathbf{x}_{i}$. The following constraints are imposed on
$x_{i,t}$:

$\underline{x}_{i,t}\leq x_{i,t}\leq\overline{x}_{i,t}$

Fixed injections can be specified by setting
$\underline{x}_{i,t}=\overline{x}_{i,t}$.

####  3.1.4 System constraints

In the system model, the microgrid is connected to the bulk power grid at a
single point of common coupling (PCC). Therefore the total injection at this
point must satisfy the laws of powerflow. In particular,

$x_{t}^{\mathrm{PCC}}=\sum_{b}(u_{b,t}^{-}-u_{b,t}^{+})+\sum_{r}x_{r,t}+\sum_{i}x_{i,t}$

By convention we assume that $x_{t}^{\mathrm{PCC}}>0$ represents an outflow
into the grid and $x_{t}^{\mathrm{PCC}}<0$ represents an inflow.

For the multi-market optimization, $x_{t}^{\mathrm{PCC}}$ is divided between
real-time and long-term quantities. However, this is just an accounting issue.
The producer will decide in each time what fraction of $x_{t}^{\mathrm{PCC}}$
to commit to each market. However, the purpose of the RL policy we seek to
learn in this work is to optimize this tradeoff, and therefore the power
balance equation is

$x_{t}^{\mathrm{LT}}+x_{t}^{\mathrm{RT}}=\sum_{b}(u_{b,t}^{-}-u_{b,t}^{+})+\sum_{r}x_{r,t}+\sum_{i}x_{i,t}$

Note that microgrid network losses and congestion are not considered in this
formulation, mainly to simplify the exposition. Conceptually these are simple
to include in a linearized form, with a more involved linear equality
constraint for power balance and linear inequality constraints for power flow
constraints being added. It is a reasonable assumption for a small microgrid
whose devices are connected to a common busbar to be a copperplate (i.e., no
congestion, no loss) network.

###  3.2 Uncertainty

The principle challenge in solving the stochastic optimal control problem
posed in the previous section is accommodating the uncertainty of the
exogenous variables. There are two uncertain exogenous processes: the maximum
available renewable generation $\overline{x}_{r,t}\ \forall r$ and the real-
time price $\pi_{t}^{\mathrm{RT}}$. Neither of these random variables is
stationary and their values at a given $t$ are dependent on some underlying
non-stationary process.

We discuss each source of uncertainty in greater detail next.

####  3.2.1 Renewable generation

The power produced by each renewable (e.g., solar or wind) generator in the
microgrid is fundamentally uncertain. In the case of solar, the generation
profile over the course of a day generally follows a nominal trajectory,
peaking at midday and 0 at night, with fluctuations due to atmospheric
conditions like clouds. Solar availability is also determined by the time of
year, although the effect of the season on power output can be computed
precisely. Thus, solar power can generally be forecast well with the exception
of sometimes sharp dips in power due to local effects such as passing clouds
or shadows. On the other hand, wind power is significantly less predictable
and variable. Nonetheless wind can still be forecast in the short term with
reasonable accuracy. For more details see (Kumar et al., 2021). Figure 2 shows
sample generation profiles for wind and solar at a location in California.

Figure 2: Sample solar and wind generation profiles. Solar is scaled between 0
and 1.

####  3.2.2 Real-time prices

The real-time price that the producer faces is generated by the system
operator (e.g., ISO or RTO) every time the real-time market is cleared, which
is typically 5 to 15 min intervals. This price is derived from a system-wide
security constrained economic dispatch problem that can take into account
system operating conditions like network congestion and losses as well as load
and generation from all participants in the network. There is usually a price-
taking assumption on producers, where they are assumed to not be able to
manipulate the LMP. Although this assumption is not realistic in practice and
in transmission constrained zones/nodes, individual generators may be able to
increase the price through strategic withholding, Navid and Rosenwald (2012)
we do not consider this possibility in our work. Strategic behavior on the
part of generators is a broad topic that is the subject of much attention from
the ISOs. Manipulative bidding is monitored by ISOs and there are typically
penalties for participants that do so.

We consider two types of uncertain price in the work. The first is an exchange
price, which does not have a locational component, and can be found in
European-style balancing markets, e.g., Nordpool. Exchange prices typically
exhibit daily periodicity and track the demand in the market. An example is
given in Figure 3.

Figure 3: Nordpool hourly price over 1 week.

A second type of price is a locational marginal price (LMP), which includes
congestion and loss components. The LMP is specific to a particular location
(node) in the power network. By assumption, the producer in this work faces a
single LMP as the PCC for the microgrid is connected to a single network bus.
The variability in the LMP is significantly more challenging than that
exhivited by the exchange price. This is due to binding congestion constraints
that cause spikiness or discontinuity in the price. Further, LMPs do not enjoy
the spatial averaging and smoothing present in exchange prices. An example of
the LMP is given in Figure 4

Figure 4: 5-min LMP at a CAISO node over the course of one day. Price spikes
up to the price cap of approx $1000/MWh are likely due to temporary ramp-
induced shortages.

###  3.3 Markov Decision Process

We represent the evolution of the energy system through time as a Markov
Decision Process (MDP). The MDP is the following 5-tuple:

$(\mathcal{X},\mathcal{U},f,R,W_{t})$

Consider $\mathcal{X}_{t}\subseteq\mathcal{X}$, $\mathcal{X}_{t}(W_{t})$ is
the time-varying state space, which depends deterministically on the
realization of the exogenous variable.

$X_{t}\in\mathcal{X}_{t}(W_{t})$

Analogously we assume that there are possibly time-varying action constraints
on $U_{t}$:

$U_{t}\in\mathcal{U}_{t}$

The transition dynamics are given by

$X_{t+1}=f(X_{t},U_{t},W_{t})$ (5)

The reward function is the profit that the operator obtains in each time for
delivering energy into the two markets:

$R_{t}(X_{t},U_{t})=P^{\mathrm{RT}}(X_{t})+P^{\mathrm{LT}}(X_{t}^{\mathrm{LT}})-C(X_{t},U_{t})$
(6)

The first two terms of $R_{t}$ are the profit/loss functions for the real-time
and long term markets and the last term is the cost function, which has terms
for storage degradation and renewable curtailment. The exact forms of these
functions was discussed in Section 3.1.

Note that the way we have defined the MDP explicitly depends on the exogenous
uncertainty $W_{t}$. The state space, reward, and transition dynamics are
deterministic functions given a realization of $W_{t}$.

We assume that the randomness is from a distribution $\Omega$. In order to
have a physically feasible system, the support of $\Omega$ is assumed to be
finite. In addition, we have state and action constraints:

The reinforcement learning problem is to learn an (optimal) policy $\pi\in\Pi$
to determine the action $U_{t}$ in each time. That is,

$\displaystyle\arg\max_{\pi\in\Pi}\quad$
$\displaystyle\sum_{t=0}^{T}\mathbb{E}_{W_{t}}\left[R_{t}(X_{t},U_{t}^{\pi}(X_{t}))\right]$
s.t. $\displaystyle X_{t+1}=f(X_{t},U_{t}^{\pi}(X_{t}),W_{t})\quad\forall t$
$\displaystyle X_{t}\in\mathcal{X}_{t}(W_{t})\quad\forall t$ $\displaystyle
U_{t}^{\pi}(X_{t})\in\mathcal{U}_{t}\quad\forall t$

In the next section we discuss the algorithm we use to solve this stochastic
optimization problem and find optimal $\pi^{*}\in\Pi$.

##  4 Reinforcement Learning Algorithm

We first give some background on how the reinforcement learning algorithm
interacts with the power system environment described in the previous section
and then give details on the particular gradient-based policy method we
exploit.

Figure 5: Interaction during learning between system environment and RL
policy.

The key aspect of our RL architecture is that the action $U_{t}$ returned by
the policy is projected onto the feasible state-action space within the
environment as part of the transition dynamics $f$. This is possible because
the transition dynamics are assumed to be control-affine; that is affine in
the action $U_{t}$.

###  4.1 Proximal Policy Optimization (PPO)

Proximal Policy Optimization (PPO) is a type of Reinforcement Learning
algorithm that seeks to optimize policy functions in a more sample-efficient
and stable way compared to other policy gradient methods. Traditional policy
gradient methods make significant changes to the policy at each update, which
can lead to unstable learning and poor performance. PPO addresses this by
limiting the changes to the policy during each update, ensuring the new policy
does not deviate significantly from the old policy. This is achieved using a
specialized objective function that penalizes drastic changes. PPO, therefore,
balances exploration of the environment and exploitation of the current
policy, leading to reliable and efficient learning, which has made it a
popular choice for a variety of complex tasks in diverse domains, see Schulman
et al. (2017). We will add more details about the specific adaptation of PPO
for this case.

##  5 Experiments

To make our problem setting accessible to a wide array of standard RL
algorithms, we implement the energy environment described in this section as
an OpenAI gym environment Brockman et al. (2016).

###  5.1 Case Study

We implement a small microgrid with a renewable generator (i.e., solar and
wind) and two batteries: one with a large capacity but slow charge/discharge
and one with a small capacity and fast charge/discharge. The results of the
training are shown in Figure 6.

Figure 6: Time series of system evolution over 1 week at hourly intervals. The
top two panes show the revenue the operator receives over time, the next two
panes show the exchange and long-term market commitment quantities,
respectively. The next two show the charge/discharge rates of the two
batteries and their states-of-charge. Finally, the last two show the exogenous
data; in particular exchange price fluctuations and the solar generation
profiles.

##  6 Smoothing out the projection step

Due to the impact of the projection step on the stability of the learning, we
consider simplifying it and consolidating the environment dynamics.
Specifically, we have

  * •

A single battery with identical charge/discharge efficiencies

  * •

A single renewable generator that cannot be curtailed

  * •

No penalties for violating long-term commitment (only price arbitrage)

We therefore consider the following state:

$x_{t}=[x^{\mathrm{r}}_{t},\pi^{\mathrm{RT}}_{t},\pi^{\mathrm{LT}}_{t},x^{\mathrm{SOC}}_{t}]$

and action

$u_{t}=[u^{\mathrm{b}}_{t},u^{\mathrm{LT}}_{t}]$

The state space constraints are

$0\leq x^{\mathrm{r}}_{t}\leq x^{\mathrm{cap}}$
$\pi^{\mathrm{RT}}_{t},\pi^{\mathrm{LT}}_{t}\geq 0$ $0\leq
x^{\mathrm{SOC}}_{t}\leq 1$

The action space constraints are

$u^{\mathrm{min}}\leq u^{\mathrm{b}}_{t}\leq u^{\mathrm{max}}$
$u^{\mathrm{LT}}_{t}\geq 0$

We can use a hyperbolic tangent to squeeze close or half-closed ranges on the
action space.

Notice that we do not include the real-time commitment in the action space
because it is implicitly determined by the other variables:

$u^{\mathrm{RT}}_{t}=x^{\mathrm{r}}_{t}+u^{\mathrm{b}}_{t}-u^{\mathrm{LT}}_{t}$

How do we enforce $u^{\mathrm{RT}}_{t}\geq 0$? This is problematic because we
need to use the current state to constrain the action.

In addition, the current SOC depends on the previous SOC which enforces
additional constraints on $u^{\mathrm{b}}_{t}$.

Appendix

##  Appendix A Offline problem formulation

Our system model has the following components at a single node:

  * •

$B$ batteries

  * •

$R$ renewable generators

  * •

$N$ conventional dispatchable generators

  * •

Exogenous exchange price $\pi^{\mathrm{ex}}$. Settlement is real-time unit
pricing.

  * •

Pre-existing long-term commitment trajectory

  * •

We assume that the frequency of the exchange market is the LCD for all other
processes. Dispatch decisions (i.e., charge/discharge, curtailment) are made
at the frequency of the exchange market

  * •

We care to optimize over a time horizon $t=1,\ldots,T$

  * •

Inverter model for renewables?

  * •

Do we formulate the problem via state and action spaces?

  * •

It may sometimes be advantageous to curtail, for example when the
$\pi^{\mathrm{ex}}$ is negative. Question: would it then just be advantageous
to store energy in the battery?

We need to define

  1. 1.

Battery model (system of equations)

  2. 2.

Penalty structure for long term commitment

  3. 3.

Overall cost function

  4. 4.

Representation of uncertain parameters

$\displaystyle\underset{x_{t}^{\mathrm{ex}},x_{t}^{\mathrm{LT}},x_{r,t},x_{i,t},e_{b,t}^{\pm},s_{b,t}}{\max}\quad$
$\displaystyle\sum_{t=0}^{T}f_{t}^{\mathrm{ex}}(x_{t}^{\mathrm{ex}};\pi^{\mathrm{ex}}_{t})+f_{t}^{\mathrm{LT}}(x_{t}^{\mathrm{LT}};\pi^{\mathrm{LT}}_{t},y_{t})-\sum_{r}f^{\mathrm{curtail}}_{t}(x_{r,t},\overline{x}_{r,t})$
(7a) s.t. $\displaystyle
x_{t}^{\mathrm{ex}}+x_{t}^{\mathrm{LT}}=\sum_{r=1}^{R}x_{r,t}+\sum_{b=1}^{B}(e_{b,t}^{-}-e_{b,t}^{+})-\sum_{i=1}^{N}x_{i,t}\quad$
$\displaystyle\forall t$ (7b) $\displaystyle 0\leq x_{t}^{\mathrm{LT}}\leq
y_{t}\quad$ $\displaystyle\forall t$ (7c) $\displaystyle 0\leq
x_{t}^{\mathrm{ex}}\quad$ $\displaystyle\forall t$ (7d)
$\displaystyle\underline{x}_{r,t}\leq x_{r,t}\leq\overline{x}_{r,t}\quad$
$\displaystyle\forall r,\ \forall t$ (7e)
$\displaystyle\underline{x}_{i,t}\leq x_{i,t}\leq\overline{x}_{i,t}\quad$
$\displaystyle\forall i,\ \forall t$ (7f)
$\displaystyle\underline{e}^{+}_{b,t}\leq
e^{+}_{b,t}\leq\overline{e}^{+}_{b,t}\quad$ $\displaystyle\forall b,\ \forall
t$ (7g) $\displaystyle\underline{e}^{-}_{b,t}\leq
e^{-}_{b,t}\leq\overline{e}^{-}_{b,t}\quad$ $\displaystyle\forall b,\ \forall
t$ (7h) $\displaystyle e_{b,t}^{+}e_{b,t}^{-}=0\quad$ $\displaystyle\forall
b,\ \forall t$ (7i) $\displaystyle\underline{s}_{b,t}\leq
s_{b,t}\leq\overline{s}_{b,t}$ $\displaystyle\forall b,\ \forall t$ (7j)
$\displaystyle
s_{b,t+1}=s_{b,t}+\eta_{b}^{+}e_{b,t}^{+}-\eta_{b}^{-}e_{b,t}^{-}\quad$
$\displaystyle\forall b,\ \forall t$ (7k)

Remarks:

  1. 1.

The stochastic parameters are $\pi^{\mathrm{ex}}_{t}$ and
$\overline{x}_{r,t}$. The remaining parameters are given.

  2. 2.

Constraint (1b) is power balance.

  3. 3.

The exchange market has a reward function of the form

$f^{\mathrm{ex}}(z;\pi)=\pi z$ (8)

where $z\in\mathbb{R}$ and $\pi\in\mathbb{R}_{+}$. There is no cost associated
with generating.

  4. 4.

The long-term market has a reward function of the form
$f^{\mathrm{LT}}:\\{z|z\leq\overline{z}\\}\rightarrow\mathbb{R}$

$f^{\mathrm{LT}}(z;\pi,\overline{z})=\begin{cases}\pi
z,\quad&z=\overline{z}\\\ g(z),\quad&z<\overline{z}\end{cases}$ (9)

where $g(z)$ is some monotonically increasing function; e.g., constant,
linear, stepwise. Question: is $f^{\mathrm{LT}}$ convex? Like an inverted RElu
or some discontinuous function.

  5. 5.

An example of the curtailment penalty function is

$f_{t}^{\mathrm{curtail}}(z,\overline{z})=a(z-\overline{z})^{2}$ (10)

##  Appendix B MPC Formulation

  * •

$H$ be the horizon length for the MPC problem.

  * •

The state is the level of generation/demand for all generators and ESRs, as
well at the SOC for the ESRs. The action will be changes in the state
variables. We therefore assume some initial value for the state variables.

  * •

The state $X_{t}$ is comprised of the physical controllable state $R_{t}$ and
the information state $I_{t}$ [cite Powell part 2]

$\displaystyle R_{t}$
$\displaystyle=(x_{b,t},x_{r,t},x_{i,t},x_{t}^{\mathrm{ex}},x_{t}^{\mathrm{LT}})$
$\displaystyle I_{t}$ $\displaystyle=(\pi^{\mathrm{LT}},y_{t},q_{t})$

where $q_{t}$ are the forecasts computed at time $t$. Then

$X_{t}=(R_{t},I_{t})$ (11)

The definition of state is ambiguous… It is unclear how to distinguish certain
information $I_{t}$ from uncertain (exogenous) information $W_{t}$. For
example, we assume $x_{i,t}$ to be known over the whole horizon. But we
somehow need to reflect them in our state update.

  * •

The action (aka decision) is

$U_{t}=(u_{b,t}^{\pm},u_{r,t},u_{i,t},u_{t}^{\mathrm{ex}},u_{t}^{\mathrm{LT}})$
(12)

Each action must have a corresponding partner in $R_{t}$.

  * •

Exogenous information is

$W_{t}=(\pi^{\mathrm{ex}},\overline{x}_{r,t})$ (13)

Exogenous information is information that becomes first known between $t-1$
and $t$. This distinguishes $W_{t}$ from $I_{t}$ in the state. $W_{t}$ is
assumed to be uncertain.

  * •

The cost function is

$\displaystyle C_{t}(X_{t},U_{t},W_{t})$
$\displaystyle=f_{t}^{\mathrm{ex}}(x_{t}^{\mathrm{ex}};\pi^{\mathrm{ex}}_{t})+f_{t}^{\mathrm{LT}}(x_{t}^{\mathrm{LT}};\pi^{\mathrm{LT}}_{t},y_{t})-\sum_{r}f^{\mathrm{curtail}}_{t}(x_{r,t},\overline{x}_{r,t})$

$\displaystyle\underset{u_{r,t},u_{i,t},e_{b,t}^{\pm},u_{t}^{\mathrm{ex}},u_{t}^{\mathrm{LT}}}{\max}\quad$
$\displaystyle\sum_{t=t^{\prime}}^{t^{\prime}+H}f_{t}^{\mathrm{ex}}(x_{t}^{\mathrm{ex}};\pi^{\mathrm{ex}}_{t})+f_{t}^{\mathrm{LT}}(x_{t}^{\mathrm{LT}};\pi^{\mathrm{LT}}_{t},y_{t})-\sum_{r}f^{\mathrm{curtail}}_{t}(x_{r,t},\overline{x}_{r,t})$
(14a) s.t. $\displaystyle x_{r,t+1}=x_{r,t}+u_{r,t}\quad$
$\displaystyle\forall r,\forall t$ (14b) $\displaystyle
x_{i,t+1}=x_{i,t}+u_{i,t}\quad$ $\displaystyle\forall r,\forall i,\forall t$
(14c) $\displaystyle
x^{\mathrm{ex}}_{t+1}=x^{\mathrm{ex}}_{t}+u^{\mathrm{ex}}_{t}\quad$
$\displaystyle\forall t$ (14d) $\displaystyle
x^{\mathrm{LT}}_{t+1}=x^{\mathrm{LT}}_{t}+u^{\mathrm{LT}}_{t}\quad$
$\displaystyle\forall t$ (14e) $\displaystyle
x_{b,t+1}=x_{b,t}+\eta_{b}^{+}u_{b,t}^{+}-\eta_{b}^{-}u_{b,t}^{-}\quad$
$\displaystyle\forall b,\ \forall t$ (14f) $\displaystyle
u_{t}^{\mathrm{ex}}+u_{t}^{\mathrm{LT}}=\sum_{r=1}^{R}u_{r,t}+\sum_{b=1}^{B}(u_{b,t}^{-}-u_{b,t}^{+})-\sum_{i=1}^{N}u_{i,t}\quad$
$\displaystyle\forall t$ (14g) $\displaystyle 0\leq x_{t}^{\mathrm{ex}}\quad$
$\displaystyle\forall t$ (14h) $\displaystyle 0\leq x_{t}^{\mathrm{LT}}\leq
y_{t}\quad$ $\displaystyle\forall t$ (14i)
$\displaystyle\underline{x}_{b,t}\leq x_{b,t}\leq\overline{x}_{b,t}$
$\displaystyle\forall b,\ \forall t$ (14j)
$\displaystyle\underline{x}_{r,t}\leq x_{r,t}\leq\overline{x}_{r,t}\quad$
$\displaystyle\forall r,\ \forall t$ (14k)
$\displaystyle\underline{x}_{i,t}\leq x_{i,t}\leq\overline{x}_{i,t}\quad$
$\displaystyle\forall i,\ \forall t$ (14l)
$\displaystyle\underline{u}^{+}_{b,t}\leq
u^{+}_{b,t}\leq\overline{u}^{+}_{b,t}\quad$ $\displaystyle\forall b,\ \forall
t$ (14m) $\displaystyle\underline{u}^{-}_{b,t}\leq
u^{-}_{b,t}\leq\overline{u}^{-}_{b,t}\quad$ $\displaystyle\forall b,\ \forall
t$ (14n) $\displaystyle\underline{u}_{i,t}\leq
u_{i,t}\leq\overline{u}_{i,t}\quad$ $\displaystyle\forall i,\ \forall t$ (14o)
$\displaystyle u_{b,t}^{+}u_{b,t}^{-}=0\quad$ $\displaystyle\forall b,\
\forall t$ (14p) $\displaystyle x_{r,t^{\prime}}=x_{r,t^{\prime}-1}\quad$
$\displaystyle\forall r$ (14q) $\displaystyle
x_{i,t^{\prime}}=x_{i,t^{\prime}-1}\quad$ $\displaystyle\forall i$ (14r)
$\displaystyle x_{b,t^{\prime}}=x_{b,t^{\prime}-1}\quad$ $\displaystyle\forall
b$ (14s)

Aside: need to specify terminal constraints such that the transition equation
doesn’t violate any constraints?

The above problem can be expressed in canonical MPC form:

At each time $t^{\prime}$, solve

$\displaystyle\max_{\begin{subarray}{c}X_{t^{\prime}+1},\ldots,X_{t^{\prime}+H}\\\
U_{t^{\prime}},\ldots,U_{t^{\prime}+H-1}\end{subarray}}\quad$
$\displaystyle\sum_{t=t^{\prime}}^{t^{\prime}+H}C_{t}(X_{t},U_{t},W_{t})$ s.t.
$\displaystyle X_{t^{\prime}+1}=f_{t}(X_{t},U_{t},W_{t}),\quad\forall
t=t^{\prime},\ldots,t^{\prime}+H-1$ $\displaystyle
X_{t}\in\mathcal{X}_{t}\quad\forall t=t^{\prime},\ldots,t^{\prime}+H$
$\displaystyle U_{t}\in\mathcal{U}_{t}\quad\forall
t=t^{\prime},\ldots,t^{\prime}+H$ $\displaystyle
W_{t}\in\mathcal{W}_{t}\quad\forall t=t^{\prime},\ldots,t^{\prime}+H$
$\displaystyle X_{t^{\prime}}=X_{t^{\prime}-1}$

##  Appendix C Stochastic Optimization Formulation

An equivalent Markov-style decision problem can be formulated:

$\displaystyle\max_{\pi\in\Pi}\quad$
$\displaystyle\mathbb{E}^{\pi}\sum_{t=0}^{T}C_{t}(X_{t},U_{t}^{\pi}(X_{t}))$
(15)

where the state transition dynamics are

$X_{t+1}=f_{t}(X_{t},U_{t}^{\pi}(X_{t}))$

The choice of policy $\pi$ is tbd. If we desire a deterministic lookahead
policy with forecasts for the uncertain information, then $U_{t}^{\pi}(X_{t})$
is given by the argmin of the MPC problem from above.

Question: does one of the other policy classes better suit this problem?

##  Appendix D Literature Review

In this section, we review the existing approaches to battery and smart grid
optimization, highlight the leading methods.

We organize the scope of the approaches taken to tackle the energy scheduling
problem with uncertainty into two main categories: (1) stochastic optimization
and (2) reinforcement learning. Each of these classes will be refined further.

Lastly, before starting the review, we state the benchmark performance metric:
the ex-post (aka posterior, in-hindsight, offline, etc.) optimal. This is when
the problem is solved for the entire time horizon using all available
information and realizations of uncertainty. In other words, the God solution.

## References

  * Arulkumaran et al. [2017] K. Arulkumaran, M. P. Deisenroth, M. Brundage, and A. A. Bharath.  Deep reinforcement learning: A brief survey.  _IEEE Signal Processing Magazine_ , 34(6):26–38, 2017.
  * Brockman et al. [2016] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba.  Openai gym.  _arXiv preprint arXiv:1606.01540_ , 2016.
  * Cao et al. [2020] J. Cao, D. Harrold, Z. Fan, T. Morstyn, D. Healey, and K. Li.  Deep Reinforcement Learning-Based Energy Storage Arbitrage With Accurate Lithium-Ion Battery Degradation Model.  _IEEE Transactions on Smart Grid_ , 11(5):4513–4521, Sept. 2020.  ISSN 1949-3061.  doi: 10.1109/TSG.2020.2986333.
  * Cervigni and Perekhodtsev [2013] G. Cervigni and D. Perekhodtsev.  Wholesale electricity markets.  In _The Economics of Electricity Markets_ , pages 18–66. Edward Elgar Publishing, 2013.
  * Chen et al. [2021a] B. Chen, P. Donti, K. Baker, J. Z. Kolter, and M. Berges.  Enforcing policy feasibility constraints through differentiable projection for energy optimization.  _arXiv preprint arXiv:2105.08881_ , 2021a.
  * Chen et al. [2021b] X. Chen, G. Qu, Y. Tang, S. Low, and N. Li.  Reinforcement learning for decision-making and control in power systems: Tutorial, review, and vision.  _arXiv preprint arXiv:2102.01168_ , 2021b.
  * Fabra [2018] N. Fabra.  A primer on capacity mechanisms.  _Energy Economics_ , 75:323–335, 2018.
  * Garifi et al. [2020] K. Garifi, K. Baker, D. Christensen, and B. Touri.  Convex relaxation of grid-connected energy storage system models with complementarity constraints in dc opf.  _IEEE Transactions on Smart Grid_ , 11(5):4070–4079, 2020.
  * Helman et al. [2009] U. Helman, H. Singh, and P. Sotkiewicz.  Rtos, regional electricity markets, and climate policy.  _Generating Electricity in a Carbon-Constrained World_ , pages 527–563, 2009.
  * Hogan [2016] W. W. Hogan.  Virtual bidding and electricity market design.  _The Electricity Journal_ , 29(5):33–47, 2016\.
  * Kumar et al. [2021] P. Kumar, R. Chandra, C. Bansal, S. Kalyanaraman, T. Ganu, and M. Grant.  Micro-climate prediction-multi scale encoder-decoder based deep learning framework.  In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_, pages 3128–3138, 2021.
  * Litvinov et al. [2019] E. Litvinov, F. Zhao, and T. Zheng.  Electricity markets in the united states: Power industry restructuring processes for the present and future.  _IEEE Power and Energy Magazine_ , 17(1):32–42, 2019.
  * Mbuwir et al. [2017] B. V. Mbuwir, F. Ruelens, F. Spiessens, and G. Deconinck.  Battery energy management in a microgrid using batch reinforcement learning.  _Energies_ , 10(11):1846, 2017.
  * Navid and Rosenwald [2012] N. Navid and G. Rosenwald.  Market solutions for managing ramp flexibility with high penetration of renewable resource.  _IEEE Transactions on Sustainable Energy_ , 3(4):784–790, 2012.
  * Nazir and Almassalkhi [2021] N. Nazir and M. Almassalkhi.  Guaranteeing a physically realizable battery dispatch without charge-discharge complementarity constraints.  _IEEE Transactions on Smart Grid_ , 2021.
  * Oh and Wang [2020] E. Oh and H. Wang.  Reinforcement-Learning-Based Energy Storage System Operation Strategies to Manage Wind Power Forecast Uncertainty.  _IEEE Access_ , 8:20965–20976, 2020.  ISSN 2169-3536.  doi: 10.1109/ACCESS.2020.2968841.
  * Powell [2014] W. B. Powell.  Clearing the jungle of stochastic optimization.  In _Bridging data and decisions_ , pages 109–137. Informs, 2014.
  * Powell [2021] W. B. Powell.  Reinforcement learning and stochastic optimization, 2021.
  * Powell and Meisel [2015] W. B. Powell and S. Meisel.  Tutorial on stochastic optimization in energy—part ii: An energy storage illustration.  _IEEE Transactions on Power Systems_ , 31(2):1468–1475, 2015.
  * Schulman et al. [2017] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ , 2017.
  * Shang et al. [2020] Y. Shang, W. Wu, J. Guo, Z. Ma, W. Sheng, Z. Lv, and C. Fu.  Stochastic dispatch of energy storage in microgrids: An augmented reinforcement learning approach.  _Applied Energy_ , 261:114423, Mar. 2020.  ISSN 0306-2619.  doi: 10.1016/j.apenergy.2019.114423.  URL https://www.sciencedirect.com/science/article/pii/S0306261919321105.
  * Tang et al. [2022] Z. Tang, J. Liu, and P. Zeng.  A multi-timescale operation model for hybrid energy storage system in electricity markets.  _International Journal of Electrical Power & Energy Systems_, 138:107907, 2022.
  * Wang and Zhang [2018] H. Wang and B. Zhang.  Energy storage arbitrage in real-time markets via reinforcement learning.  In _2018 IEEE Power and Energy Society General Meeting, PESGM 2018_ , page 8586321. IEEE, Institute of Electrical and Electronics Engineers, 2018.  doi: 10.1109/PESGM.2018.8586321.  URL https://research.monash.edu/en/publications/energy-storage-arbitrage-in-real-time-markets-via-reinforcement-l.
  * Yang et al. [2020a] J. J. Yang, M. Yang, M. X. Wang, P. J. Du, and Y. X. Yu.  A deep reinforcement learning method for managing wind farm uncertainties through energy storage system control and external reserve purchasing.  _International Journal of Electrical Power & Energy Systems_, 119:105928, July 2020a.  ISSN 0142-0615.  doi: 10.1016/j.ijepes.2020.105928.  URL https://www.sciencedirect.com/science/article/pii/S0142061519312505.
  * Yang et al. [2020b] T. Yang, L. Zhao, W. Li, and A. Y. Zomaya.  Reinforcement learning in sustainable energy and electric systems: A survey.  _Annual Reviews in Control_ , 49:145–163, 2020b.
  * Zhang et al. [2019] Z. Zhang, D. Zhang, and R. C. Qiu.  Deep reinforcement learning for power system applications: An overview.  _CSEE Journal of Power and Energy Systems_ , 6(1):213–225, 2019.
  * Zheng et al. [2021] L. Zheng, Y. Shi, L. J. Ratliff, and B. Zhang.  Safe reinforcement learning of control-affine systems with vertex networks.  In _Learning for Dynamics and Control_ , pages 336–347. PMLR, 2021\.