xxxxx, xxxxx, xxxx

Jacques de Chalendar

# On Incorporating Forecasts into Linear State Space Model Markov Decision
Processes

Jacques A. de Chalendar1 and Peter W. Glynn2 1Department of Energy Resources
Engineering, Stanford University, Stanford CA 94305-2205, USA.
2Department of Management Sciences & Engineering, Stanford University,
Stanford CA 94305-2205, USA<EMAIL_ADDRESS>
###### Abstract

Weather forecast information will very likely find increasing application in
the control of future energy systems. In this paper, we introduce an augmented
state space model formulation with linear dynamics, within which one can
incorporate forecast information that is dynamically revealed alongside the
evolution of the underlying state variable. We use the martingale model for
forecast evolution (MMFE) to enforce the necessary consistency properties that
must govern the joint evolution of forecasts with the underlying state. The
formulation also generates jointly Markovian dynamics that give rise to Markov
decision processes (MDPs) that remain computationally tractable. This paper is
the first to enforce MMFE consistency requirements within an MDP formulation
that preserves tractability.

###### keywords:

Markov Decision Processes, weather forecasts, forecast models, stochastic
control of energy systems

##  1 Introduction

Forecasts are ubiquitous in energy system control problems and there is reason
to believe their importance will only grow, e.g. in the fast changing electric
sector. This is especially true for forecasts that provide weather-related
information, as weather patterns have a strong impact on energy demand and
increasingly on (renewable) energy production. The meteorological community
has made significant progress in that field over the past decades and can now
offer several advantages over purely statistical models [1]. In a recent
review of forecasting for renewable energy [2], the authors note a rise in
demand for probabilistic forecasting and that the typical renewable energy
production use cases for weather forecasts correspond to timescales of weeks
to years, or hours to a day ahead.

Much deeper transformations are in store once very high penetrations of
renewables are reached. A growing number of applications will require storing
electricity with durations from 10 to 100 hours [3, 4, 5]. Wind generation is
one example for which low availability levels can be observed for several
consecutive days. The need for longer duration storage will appear more
clearly with penetrations of > 70% wind and solar generation on a regional
grid – see e.g. Figure 1 from [5]. As soon as longer duration storage becomes
available, we will correspondingly need new energy management strategies. At
these timescales, conditioning on forecasts, in particular for weather
variables, will very likely have large impacts on problems of decision making
under uncertainty.

Using forecast information in the context of control problems is a difficult
general problem that implicitly appears in many real-life applications. In
sequential decision problems, it is often the case that exogenous forecast
information is presented to the controller at regular intervals. Given the key
role of Markov decision processes (MDPs) in the computation of optimal
policies in such settings, a full accounting of the impact of future forecast
information requires introducing the forecasts into the Markov state variable,
thereby leading to potential high-dimensional state representations. Another
fundamental issue relates to the fact that the forecasts should be
“compatible” with the state variable that is being forecast (e.g. weather), so
that the forecasted state variable (i.e. the state variable for which
forecasts are available) and the forecasts themselves should exhibit self-
consistent dynamics. To gain some appreciation for this issue, note that the
$s$-period forecast must contain information that implicitly “peeks” $s$
periods into the future of the underlying state space model, so that the
$s$-period forecast implicitly constrains the dynamics of the underlying model
over the next $s$ periods. These constraints need to be built into the joint
dynamics in such a way that the Markov structure is preserved. The
preservation of Markov structure is critical if we wish to be able to compute
optimal policies via the use of MDP-based theory and algorithms.

In this paper, we utilize the martingale model for forecast evolution (MMFE)
as a vehicle for imposing the appropriate mathematical consistency between the
dynamics of the forecasts and the forecasted state variables. The MMFE
framework was introduced and developed by [6, 7, 8] and has since been
utilized extensively by the inventory control and supply chain management
community (see e.g [9] and the references therein). Applications of the MMFE
and studies on the impact of forecasts on decision making can also be found in
the energy community, e.g. for hydraulic reservoir management [10] or wind
energy integration [11, 12].

To our knowledge, this paper is the first to rigorously introduce forecast
model consistency into MDPs, specifically in the context of linear state space
models. This work gives us the first principled and mathematically consistent
framework for the incorporation of forecasts into MDPs in the setting of state
space models with linear dynamics, and uses no ad hoc elements to add forecast
information into the MDP setting. Linear state space models are widely applied
across many disciplines, and can even represent the linearized dynamics
associated with nonlinear structure [13].

We note that forecasts must depend on a richer information filtration than
that associated with the forecasted state variable, because an optimal MDP
policy computed from the forecasted state variable already fully utilizes all
the information associated with the forecasted state variable’s filtration. In
our setting, the extra information that enters the forecasts is the
meteorological data available to the forecasters that is unobserved by the
energy system manager. Thus, a key contribution of our paper is the
development of an MMFE framework in which one can rigorously discuss, via the
use of the language of $\sigma$-algebras, the different ways in which forecast
information can be incorporated into an MDP framework. In our carefully chosen
formulation, each of these different approaches for incorporating MMFE
forecasts leads to a different, but computationally tractable, MDP.

Our first new MDP (Section 4) incorporates the “static” forecast information
that is available to the decision-maker at the beginning of the decision
horizon, and leads to an MDP that has the same state space as for the
forecasted state variable, but with transition probabilities that are non-
stationary as a consequence of the initial set of forecasts. The model has the
property that when one conditions the future dynamics of the forecasted state
variable on the forecast information available at the beginning of the
decision horizon, the Markov structure is preserved with no need to increase
the dimensionality of the state representation. In Section 5, we develop an
MDP in which the forecasts are dynamically updated over time, along with the
forecasted state variable. Thus, this formulation explicitly models the
additional forecasting information that is revealed to the decision-maker over
time. In this dynamic forecasting formulation, one needs to expand the state
space of the MDP to incorporate the forecast evolution, but the MDP has
stationary transition probabilities. Our final MDP is a formulation in which
new $r$-period lookahead forecasts are made available to the decision-maker
over time, in addition to an extended set of static forecasts that provides
forecast information more than $r$ periods into the future. This MDP that
combines both static and dynamic forecasts is introduced in Section 6, and
leads to both an enlarged state space and non-stationary transition
probabilities.

An alternative means of utilizing the availability of forecasts in the control
setting is to apply the ideas of model predictive control (MPC). MPC has
become a standard tool for many industrial applications and provides a
practical way of dealing with forecasts [14, 15, 16]. In this approach, one
uses the forecasts available to solve a sequence of MDP formulations over
time. At each decision epoch, a conventional MDP that incorporates the
forecasts available is solved, the optimal first period action is taken, and
this process is repeated at the next decision epoch. In particular, the MDPs
that are used by MPC at each decision epoch do not explicitly model, within
the MDP, the fact that the decision-maker will have available a new set of
forecasts at each future decision epoch within the decision horizon associated
with the MDP. This is also the case in adaptations of standard MPC to the
setting where the forecasts contain probabilistic information [16]. In
contrast, the MDPs introduced by this paper model the fact that the forecasts
are continually “refreshed” over the MDP’s decision horizon, and do so via a
formulation that preserves the computational tractability of the MDP.

In Section 7, we introduce a simple energy system control model that controls
interior building temperatures in an external weather environment for which
forecasts are available. In the presence of a quadratic cost structure, we are
able to use the existing linear-quadratic stochastic control theory to compute
the optimal value associated with MDPs for the energy system in which no
forecast information is available and also the optimal value for the dynamic
forecasting MDP of Section 5. This allows us to analyze the degree of
improvement that can be obtained by incorporating dynamic forecast information
into the MDP formulation in the context of our simple energy system example.
Section 8 concludes the paper with a discussion of additional research
questions that this paper motivates.

##  2 MDP’s with no forecasts

In this section, we review the basic MDP framework that can be used when
making sequential decisions involving an energy system that is affected by the
weather. Our formulation here does not take advantage of any forecast
information that may be available. We model the dynamics in discrete time, and
take the view that the weather variables $W_{n}$ at time $n$ can be
represented by an $\mathbb{R}^{d}$-valued random variable (rv). Given our MDP
modeling perspective, we assume that $(W_{n}\mathrel{\mathop{\mathchar
58\relax}}n\in\mathbb{Z})$ is a stationary $\mathbb{R}^{d}$-valued stochastic
process that enjoys the Markov property, so that

$W_{n+1}=f(W_{n},Z_{n+1})$ (1)

for $n\in\mathbb{Z}$, where $(Z_{n}\mathrel{\mathop{\mathchar
58\relax}}n\in\mathbb{Z})$ is an independent and identically distributed (iid)
sequence of $\mathbb{R}^{m_{1}}$-valued rv’s and $f$ is a (deterministic)
mapping from $\mathbb{R}^{d}\times\mathbb{R}^{m_{1}}$ into $\mathbb{R}^{d}$.
The stationarity is intended here to simplify the exposition and is (at best)
only approximately valid in the weather setting. For example, in examining
daily weather records, it may be that such time series look approximately
stationary over time scales of (say) one month. Given that our decision
horizon is typically much shorter than a month, the stationarity assumption
will often be a reasonable one in practice.

For the energy system’s control, we model its state evolution via an
$\mathbb{R}^{l}$-valued sequence $(X_{n}\mathrel{\mathop{\mathchar
58\relax}}n\in\mathbb{Z})$ for which

$X_{n+1}=\phi(X_{n},A_{n},W_{n+1},V_{n+1})$ (2)

for $n\in\mathbb{Z}$, where $(V_{n}\mathrel{\mathop{\mathchar
58\relax}}n\in\mathbb{Z})$ is an iid sequence of $\mathbb{R}^{m_{2}}$-valued
rv’s independent of the $Z_{j}$’s, $A_{n}$ is an $\mathcal{A}$-valued action
taken at time $n$, and $\phi$ is a deterministic mapping. The action $A_{n}$
must be adapted to the history
$\mathcal{F}_{n}\triangleq\sigma((W_{j},X_{j})\mathrel{\mathop{\mathchar
58\relax}}j\leq n)$, so that it can depend only on previously observed values
of the weather and control system state. The joint dynamics 1 and 2 assume
(reasonably) that the weather affects the control system dynamics, but not
vice-versa.

We now describe the dynamic program (DP) backwards dynamic recursion that is
commonly used to compute the optimal $A^{*}_{j}$’s, when optimizing the
control of such an energy system over a finite horizon $[0,t+1)$. Throughout
this paper, we take $n=0$ as the time at which the sequence of control actions
will be computed. Suppose that our goal is to minimize the total expected cost
of running the energy system over $[0,t+1)$, namely

$\mathbb{E}\left[\sum_{j=0}^{t}c(X_{j},A_{j},W_{j+1})|X_{0}=x,W_{0}=w\right]$
(3)

over all $F_{j}$-adapted controls $(A_{j}\mathrel{\mathop{\mathchar
58\relax}}0\leq j\leq t)$. Here $c(X_{j},A_{j},W_{j+1})$ represents the one-
period cost for running the energy system over $[j,j+1)$. For an
(appropriately integrable) function $h$ with domain
$\mathbb{R}^{l}\times\mathbb{R}^{d}$, define the operator

$(P_{a}h)(x,w)=\int_{\mathbb{R}^{m_{1}}}\int_{\mathbb{R}^{m_{2}}}h(\phi(x,a,f(w,z),v),f(w,z))\mathbb{P}(Z_{1}\in
dz)\mathbb{P}(V_{1}\in dv)$ (4)

and set

$\tilde{c}(x,a,w)=\int_{\mathbb{R}^{m_{1}}}c(x,a,f(w,z))\mathbb{P}(Z_{1}\in
dz)$ (5)

The DP value functions $(v_{i}(\cdot)\mathrel{\mathop{\mathchar
58\relax}}0\leq i\leq t)$ are then computed via the recursion

$v_{i}(x,w)=\min_{a\in\mathcal{A}}\left[\tilde{c}(x,a,w)+(P_{a}v_{i+1})(x,w)\right]$
(6)

for $0\leq i<t$, subject to the terminal condition

$v_{t}(x,w)=\min_{a\in\mathcal{A}}\tilde{c}(x,a,w).$ (7)

Assuming that $v_{t},v_{t-1},\cdots,v_{0}$ are recursively computed via 6 and
7 , we then select $a^{*}_{i}(x,w)$ ($a^{*}_{t}(x,w)$) as any minimizer
(assumed to exist) of the right-hand side of 6 (7), and put
$A_{i}^{*}=a_{i}^{*}(X_{i},W_{i})$ for $0\leq i\leq t$. Under suitable
integrability hypotheses, it is well known that
$(A_{i}^{*}\mathrel{\mathop{\mathchar 58\relax}}0\leq i\leq t)$ is then the
desired cost-minimizing adapted optimal control, see e.g. [17].

##  3 The mathematical structure of forecasts

In order to build forecast information into the Markov model of Section 2, we
review the mathematical structure of forecasts, so that we can ensure that the
model combining Markovian state dynamics and forecast information respects the
appropriate mathematical constraints. To this end, we assume that
$\mathbb{E}||W_{n}||^{2}<\infty$ (where $||\cdot||$ is the Euclidian norm). We
model the (point) forecast $F_{n|k}$ of $W_{n}$ available at time $k\leq n$ as
the rv $\mathbb{E}[W_{n}|\mathcal{G}_{k}]$, where $\mathcal{G}_{k}$ is a
$\sigma$-algebra representing the information available to the forecaster.
Since weather forecasters have available vastly more weather information than
does the energy system manager, we expect that $\mathcal{G}_{k}$ represents a
strictly richer “information set” than
$\mathcal{F}^{W}_{k}\triangleq\sigma(W_{j}\mathrel{\mathop{\mathchar
58\relax}}j\leq k)$. Consequently, we require $\mathcal{G}_{k}$ to be strictly
larger than $\mathcal{F}^{W}_{k}$. In fact, if
$\mathcal{G}_{k}=\mathcal{F}^{W}_{k}$, the availability of forecasts will
offer no advantage over the optimal control
$(A^{*}_{j}\mathrel{\mathop{\mathchar 58\relax}}0\leq j\leq t)$ computed in
Section 2, since that policy is already guaranteed to be optimal over all
$\mathcal{F}^{W}_{k}$-adapted policies.

We note that $F_{n|n}=W_{n}$ and that the tower property of conditional
expectation implies that

$\begin{split}\mathbb{E}[F_{n|k+1}|\mathcal{G}_{k}]&=\mathbb{E}\left[\mathbb{E}[W_{n}|\mathcal{G}_{k+1}]|\mathcal{G}_{k}\right]\\\
&=\mathbb{E}[W_{n}|\mathcal{G}_{k}]\\\ &=F_{n|k},\end{split}$ (8)

so that $(F_{n|k}\mathrel{\mathop{\mathchar 58\relax}}k\leq n)$ is a
martingale (in $k$) adapted to the $\mathcal{G}_{k}$’s for each fixed
$n\in\mathbb{Z}$. For $k\leq n$, let

$D_{n|k}=F_{n|k}-F_{n|k-1}$ (9)

be the $k$’th martingale difference associated with the martingale
$(F_{n|k}\mathrel{\mathop{\mathchar 58\relax}}k\leq n)$. The square
integrability of the $W_{n}$’s implies that $D_{n|k}D_{m|j}$ is integrable and

$\mathbb{E}[D_{n|k}D_{m|j}|\mathcal{G}_{j}]=D_{m|j}\mathbb{E}\left[\mathbb{E}[D_{n|k}|\mathcal{G}_{k}]|\mathcal{G}_{j}\right]=0$
(10)

for $j\leq k$, so that

$\mathbb{E}D_{n|k}D_{m|j}=0$ (11)

for $j\neq k$ and $n\geq k,m\geq j$. This orthogonality of $D_{n|k}$ and
$D_{m|j}$ is a key property of such martingale differences. As was discussed
in the Introduction, the fact that such martingale structure is a reasonable
requirement to impose on forecasts has been noted previously [6, 7, 8, 12].

##  4 MDP’s incorporating a static forecast

We now wish to build a tractable model under which
$(W_{k}\mathrel{\mathop{\mathchar 58\relax}}0\leq k\leq t+1)$ evolves over the
decision horizon, conditional on the forecasts
$(F_{n|0}\mathrel{\mathop{\mathchar 58\relax}}n\geq 0)$ available at the
outset of the decision interval.

To this end, let
$\mathcal{K}_{n}\triangleq\sigma(F_{m+j|m}\mathrel{\mathop{\mathchar
58\relax}}j\in\mathbb{Z}_{+},m\leq n)$ denote the $\sigma$-algebra associated
with the forecasts collected by time $n$, and note that
$\mathcal{K}_{n}\subseteq\mathcal{G}_{n}$, the $\sigma$-algebra associated
with all the information observed by the forecaster by time $n$. We now wish
to construct an MDP formulation appropriate to decision-making by the energy
system manager when she has access to the information available both in
$\mathcal{F}_{n}$ and $\mathcal{K}_{0}$. In other words, her decision at time
$n$ must be $\mathcal{F}_{n}\vee\mathcal{K}_{0}$ adapted, where
$\mathcal{B}_{1}\vee\mathcal{B}_{2}\vee\dots\vee\mathcal{B}_{l}$ is our
notation for the smallest $\sigma$-algebra containing
$\mathcal{B}_{1},\mathcal{B}_{2},\dots,\mathcal{B}_{l}$. We call this a static
forecast formulation, since the decision maker only uses the forecasts
available at time 0 in making decisions.

In particular, we shall build a model under which the (conditional on
$\mathcal{G}_{0}$) Markov property

$\mathbb{P}(W_{n+1}\in\cdot|\mathcal{G}_{0},W_{j}\mathrel{\mathop{\mathchar
58\relax}}j\leq n)=\mathbb{P}(W_{n+1}\in\cdot|\mathcal{K}_{0},W_{n})$ (12)

holds for $0\leq n\leq t$. This ensures that

$\mathbb{P}(W_{n+1}\in\cdot|\mathcal{K}_{0},W_{j}\mathrel{\mathop{\mathchar
58\relax}}0\leq j\leq n)=\mathbb{P}(W_{n+1}\in\cdot|\mathcal{K}_{0},W_{n}).$
(13)

We will now formulate a flexible model that satisfies both the ordinary Markov
property (as expressed through the recursion 1) and the conditional Markov
property (as expressed through 12). In particular, we now specialize the
stochastic recursion 1 to a linear state space model of the form

$W_{n+1}=GW_{n}+Z_{n+1}$ (14)

where $G$ is a deterministic $d\times d$ matrix having spectral radius less
than 1, and for which the $Z_{i}$’s are iid $\mathbb{R}^{d}$-valued rv’s for
which $\mathbb{E}||Z_{1}||^{2}<\infty$. Then,
$\mathbb{E}W_{n}=(I-G)^{-1}\mathbb{E}Z_{1}$ for $n\in\mathbb{Z}$. Let
$\tilde{W}_{n}=W_{n}-\mathbb{E}W_{0}$ and
$\tilde{Z}_{n}=Z_{n}-\mathbb{E}Z_{1}$ and note that 14 implies that

$\tilde{W}_{n+1}=G\tilde{W}_{n}+\tilde{Z}_{n+1}$ (15)

for $n\in\mathbb{Z}$. We further assume that for each $n\in\mathbb{Z}$, we can
write $\tilde{Z}_{n}$ in the form

$\tilde{Z}_{n}=\sum_{j=0}^{\infty}\epsilon_{n}(n-j),$ (16)

where the sum in 16 is assumed to converge a.s. and in mean square. The family
of rv’s $(\epsilon_{n}(j),j\leq n,n\in\mathbb{Z})$ is assumed to satisfy:

  1. A1.

The collection $(\epsilon_{n}(n-j)\mathrel{\mathop{\mathchar
58\relax}}n\in\mathbb{Z},j\in\mathbb{Z}_{+})$ is a family of independent mean
zero square integrable rv’s.

  2. A2.

$\epsilon_{n}(n-j)\stackrel{{\scriptstyle\mathcal{D}}}{{=}}\epsilon_{0}(-j)$,
for $n\in\mathbb{Z}$ (where $\stackrel{{\scriptstyle\mathcal{D}}}{{=}}$ means
“equality in distribution”).

### Remark

The $\epsilon_{n}(k)$ disturbance models the information gathered by the
forecaster at time $k$ that is relevant to the forecast for time $n$. In view
of this interpretation, it is natural that we then “model” the
$\sigma$-algebra $\mathcal{G}_{n}$ of Section 3 as
$\mathcal{G}_{n}\triangleq\sigma(\epsilon_{m}(j)\mathrel{\mathop{\mathchar
58\relax}}j\leq n,m\geq j)$ in the context of this state space model. In this
case $\mathcal{G}_{n}=\mathcal{K}_{n}$, as we will see later in this section,
although we note that in general $\mathcal{G}_{n}$ could be strictly richer
than $\mathcal{K}_{n}$. We further note that A2 implies that the distribution
for $\epsilon_{n}(k)$ only depends on $n-k$. A1 and A2 ensure that
$(\tilde{Z}_{n}\mathrel{\mathop{\mathchar 58\relax}}n\in\mathbb{Z})$ is an iid
sequence of mean zero square integrable rv’s.

If we set
$\mathcal{H}_{n}\triangleq\sigma(\epsilon_{n}(m-j)\mathrel{\mathop{\mathchar
58\relax}}m\leq n,j\in\mathbb{Z}_{+})$, we note that
$\mathcal{F}_{n}^{W}\triangleq\sigma(W_{j}\mathrel{\mathop{\mathchar
58\relax}}j\leq n)\subseteq\mathcal{H}_{n}$ and that the independence of
$(\epsilon_{n+1}(n+1-j)\mathrel{\mathop{\mathchar
58\relax}}j\in\mathbb{Z}_{+})$ from $\mathcal{H}_{n}$ ensures that

$\mathbb{P}(W_{n+1}\in\cdot|\mathcal{H}_{n})=\mathbb{P}(W_{n+1}\in\cdot|W_{n}).$
(17)

It follows that the policy $(A_{n}^{*}\mathrel{\mathop{\mathchar
58\relax}}0\leq n\leq t)$ computed in Section 2 is optimal not only over the
$\mathcal{F}_{n}$-adapted policies but also over the
$\mathcal{H}_{n}\vee\mathcal{F}_{n}$-adapted policies.

Furthermore, for $k\leq n$,

$\begin{split}W_{n}&=\mathbb{E}W_{0}+G^{n-k}\tilde{W}_{k}+\sum_{i=0}^{n-k-1}G^{i}\tilde{Z}_{n-i}\\\
&=\mathbb{E}W_{0}+G^{n-k}\tilde{W}_{k}+\sum_{i=0}^{n-k-1}G^{i}\sum_{j=0}^{\infty}\epsilon_{n-i}(n-i-j)\\\
&=\mathbb{E}W_{0}+G^{n-k}\tilde{W}_{k}+\sum_{i=0}^{n-k-1}G^{i}\sum_{r=-\infty}^{n-k-1}\epsilon_{n-i}(r)\\\
&=\mathbb{E}W_{0}+G^{n-k}\tilde{W}_{k}+\sum_{r=-\infty}^{k}\sum_{i=0}^{n-k-1}G^{i}\epsilon_{n-i}(r)+\sum_{r=k+1}^{n}\sum_{i=0}^{n-r}G^{i}\epsilon_{n-i}(r).\end{split}$
(18)

We recall that $\mathcal{G}_{n}$ contains the sequence of rv’s
$(\epsilon_{m}(j)\mathrel{\mathop{\mathchar 58\relax}}j\leq n,m>n)$ that are
independent of $\mathcal{H}_{n}$ (and hence $\mathcal{F}_{n}$). This
represents the additional information available to the forecaster about the
weather in future time periods that goes beyond the predictive information
present in observing $W_{n}$ that is locally available to the energy system
manager. Figures 1 and 2 illustrate the differences in the information sets
$\mathcal{H}_{n}$ and $\mathcal{G}_{n}$.

$\begin{matrix}\leavevmode\hbox to7.5pt{\vbox
to9pt{\pgfpicture\makeatletter\hbox{\hskip 3.75pt\lower-4.5pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-3.75pt}{-2.5pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$\ddots$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}&&&&\\\
\cdots&\epsilon_{-1}(-1)&&&\\\ \cdots&\epsilon_{0}(-1)&\epsilon_{0}(0)&&\\\
&\vdots&\vdots&\ddots&\\\
\cdots&\epsilon_{n-1}(-1)&\epsilon_{n-1}(0)&\cdots&\epsilon_{n-1}(n-1)\\\
\leavevmode\hbox to7.5pt{\vbox to9pt{\pgfpicture\makeatletter\hbox{\hskip
3.75pt\lower-4.5pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-3.75pt}{-2.5pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$\cdots$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}&\leavevmode\hbox
to25.75pt{\vbox to10pt{\pgfpicture\makeatletter\hbox{\hskip
12.87683pt\lower-5.0pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-12.87683pt}{-2.5pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$\epsilon_{n}(-1)$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}&\epsilon_{n}(0)&\cdots&\epsilon_{n}(n-1)&\leavevmode\hbox
to21.2pt{\vbox to10pt{\pgfpicture\makeatletter\hbox{\hskip
10.60025pt\lower-5.0pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-10.60025pt}{-2.5pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$\epsilon_{n}(n)$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}\\\
\cdots&\epsilon_{n+1}(-1)&\epsilon_{n+1}(0)&\cdots&\epsilon_{n+1}(n-1)&\epsilon_{n+1}(n)&\epsilon_{n+1}(n+1)\\\
&\vdots&\vdots&&\vdots&\vdots&\vdots\end{matrix}$

Figure 1: Weather-related information set associated with $\mathcal{H}_{n}$.

$\begin{matrix}\leavevmode\hbox to7.5pt{\vbox
to9pt{\pgfpicture\makeatletter\hbox{\hskip 3.75pt\lower-4.5pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-3.75pt}{-2.5pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$\ddots$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}&&&&\\\
\cdots&\epsilon_{-1}(-1)&&&\\\ \cdots&\epsilon_{0}(-1)&\epsilon_{0}(0)&&\\\
&\vdots&\vdots&\ddots&\\\
\cdots&\epsilon_{n-1}(-1)&\epsilon_{n-1}(0)&\cdots&\epsilon_{n-1}(n-1)\\\
\cdots&\epsilon_{n}(-1)&\epsilon_{n}(0)&\cdots&\epsilon_{n}(n-1)&\leavevmode\hbox
to21.2pt{\vbox to10pt{\pgfpicture\makeatletter\hbox{\hskip
10.60025pt\lower-5.0pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-10.60025pt}{-2.5pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$\epsilon_{n}(n)$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}\\\
\cdots&\epsilon_{n+1}(-1)&\epsilon_{n+1}(0)&\cdots&\epsilon_{n+1}(n-1)&\epsilon_{n+1}(n)\leavevmode\hbox
to1pt{\vbox to1pt{\pgfpicture\makeatletter\hbox{\hskip 0.5pt\lower-0.5pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{0.0pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}&\epsilon_{n+1}(n+1)\\\
\leavevmode\hbox to1pt{\vbox to1pt{\pgfpicture\makeatletter\hbox{\hskip
0.5pt\lower-0.5pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{0.0pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}&\vdots&\vdots&&\vdots&\leavevmode\hbox
to7.5pt{\vbox to9pt{\pgfpicture\makeatletter\hbox{\hskip
3.75pt\lower-4.5pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-3.75pt}{-2.5pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$\vdots$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}&\vdots\end{matrix}$

Figure 2: Weather-related information set associated with $\mathcal{G}_{n}$.

We then note that 18 implies that for $n\geq k$,

$\begin{split}F_{n|k}&=\mathbb{E}[W_{n}|\mathcal{G}_{k}]\\\
&=\mathbb{E}W_{0}+G^{n-k}\tilde{W}_{k}+\sum_{r=-\infty}^{k}\sum_{i=0}^{n-k-1}G^{i}\epsilon_{n-i}(r),\end{split}$
(19)

and the corresponding martingale differences are given by

$D_{n|k}=\sum_{i=0}^{n-k}G^{i}\epsilon_{n-i}(k).$ (20)

In addition, we note that 19 implies that

$F_{n+1|k}-\mathbb{E}W_{0}=G\left(G^{n-k}\tilde{W}_{k}+\sum_{r=-\infty}^{k}\sum_{i=0}^{n-k-1}G^{i}\epsilon_{n-i}(r)\right)+\sum_{r=-\infty}^{k}\epsilon_{n+1}(r)$
(21)

so that

$F_{n+1|k}-\mathbb{E}W_{0}=G(F_{n|k}-\mathbb{E}W_{0})+\sum_{r=-\infty}^{k}\epsilon_{n+1}(r).$
(22)

Since $(I-G)\mathbb{E}W_{0}=\mathbb{E}Z_{0}$, it follows that

$F_{n+1|k}=GF_{n|k}+Y_{n+1}(\mathcal{G}_{k})$ (23)

where

$Y_{n+1}(\mathcal{G}_{k})\triangleq\mathbb{E}Z_{0}+\sum_{r=-\infty}^{k}\epsilon_{n+1}(r).$
(24)

As a consequence of 23, we see that the “forward forecasts” from time $k$ are
correlated, and form (for each $k$) their own state space model with
independent (but not identically distributed) “noise” rv’s
$(Y_{n}(\mathcal{G}_{k})\mathrel{\mathop{\mathchar 58\relax}}n>k)$,
initialized at $F_{k|k}=W_{k}$. Such correlation in the forward forecasts is
clearly desirable from a modeling perspective.

We now turn to the conditional dynamics of $(W_{n}\mathrel{\mathop{\mathchar
58\relax}}n\geq k)$, conditional on $\mathcal{G}_{k}$. Define the
$W_{n}(\mathcal{G}_{k})$’s via

$\mathbb{P}\left((W_{n}(\mathcal{G}_{k})\mathrel{\mathop{\mathchar
58\relax}}n\geq
k)\in\cdot\right)=\mathbb{P}\left((W_{n}\mathrel{\mathop{\mathchar
58\relax}}n\geq k)\in\cdot|\mathcal{G}_{k}\right).$ (25)

The relations 18 and 19 imply that

$\begin{split}W_{n+1}(\mathcal{G}_{k})-F_{n+1|k}&=\sum_{r=k+1}^{n+1}\sum_{i=0}^{n+1-r}G^{i}\epsilon_{n+1-i}(r)\\\
&=G\sum_{r=k+1}^{n}\sum_{j=0}^{n-r}G^{j}\epsilon_{n-j}(r)+\sum_{r=k+1}^{n+1}\epsilon_{n+1}(r)\\\
&=G(W_{n}(\mathcal{G}_{k})-F_{n|k})+\sum_{r=k+1}^{n+1}\epsilon_{n+1}(r)\end{split}$
(26)

It follows that for $n\geq k$,

$W_{n+1}(\mathcal{G}_{k})=GW_{n}(\mathcal{G}_{k})+Z_{n+1}(\mathcal{G}_{k}),$
(27)

where

$Z_{n+1}(\mathcal{G}_{k})=F_{n+1|k}-GF_{n|k}+\sum_{r=k+1}^{n+1}\epsilon_{n+1}(r).$
(28)

Consequently, $(W_{n}(\mathcal{G}_{k})\mathrel{\mathop{\mathchar
58\relax}}n\geq k)$ is (conditional on $\mathcal{G}_{k}$) a Markov chain that
is a linear state space model driven by a sequence
$(Z_{n}(\mathcal{G}_{k})\mathrel{\mathop{\mathchar 58\relax}}n>k)$ of
conditionally independent (but non-identically distributed) rv’s. For a given
$k$, the variance of the $Z_{n}(\mathcal{G}_{k})$ sequence (conditional on
$\mathcal{G}_{k}$) increases with $n$, so the “uncertainty plume”
correspondingly grows with time, as one would expect.

With 27 in hand, we can now modify the value function recursion of Section 2
so as to compute the optimal policy when the energy system decision maker has
available at time $n\geq 0$ the information present in
$\mathcal{F}_{n}\vee\mathcal{K}_{0}$, the smallest $\sigma$-algebra containing
both $\mathcal{F}_{n}$ and the forecasts collected up to time $0$ by the
manager. The structure of our model implies that the policy that is optimal
over $\mathcal{F}_{n}\vee\mathcal{H}_{n}\vee\mathcal{G}_{0}$-adapted policies
is actually $\mathcal{F}_{n}\vee\mathcal{K}_{0}$-measurable. Figure 3
illustrates the weather-related information set associated with
$\mathcal{H}_{n}\vee\mathcal{G}_{0}$.

$\begin{matrix}\leavevmode\hbox to7.5pt{\vbox
to9pt{\pgfpicture\makeatletter\hbox{\hskip 3.75pt\lower-4.5pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-3.75pt}{-2.5pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$\ddots$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}&&&&\\\
\cdots&\epsilon_{-1}(-1)&&&\\\ \cdots&\epsilon_{0}(-1)&\epsilon_{0}(0)&&\\\
&\vdots&\vdots&\ddots&\\\
\cdots&\epsilon_{n-1}(-1)&\epsilon_{n-1}(0)&\cdots&\epsilon_{n-1}(n-1)\\\
\cdots&\epsilon_{n}(-1)&\epsilon_{n}(0)&\cdots&\epsilon_{n}(n-1)&\leavevmode\hbox
to21.2pt{\vbox to10pt{\pgfpicture\makeatletter\hbox{\hskip
10.60025pt\lower-5.0pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-10.60025pt}{-2.5pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$\epsilon_{n}(n)$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}\\\
\cdots&\epsilon_{n+1}(-1)&\leavevmode\hbox to27.35pt{\vbox
to10pt{\pgfpicture\makeatletter\hbox{\hskip 13.67686pt\lower-5.0pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-13.67686pt}{-2.5pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$\epsilon_{n+1}(0)$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}&\cdots&\epsilon_{n+1}(n-1)&\epsilon_{n+1}(n)&\epsilon_{n+1}(n+1)\\\
\leavevmode\hbox to1pt{\vbox to1pt{\pgfpicture\makeatletter\hbox{\hskip
0.5pt\lower-0.5pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{0.0pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}&\vdots&\leavevmode\hbox
to7.5pt{\vbox to9pt{\pgfpicture\makeatletter\hbox{\hskip
3.75pt\lower-4.5pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-3.75pt}{-2.5pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$\vdots$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}&&\vdots&\vdots&\vdots\end{matrix}$

Figure 3: Weather-related information set associated with
$\mathcal{H}_{n}\vee\mathcal{G}_{0}$.

Our goal is to minimize

$\sum_{j=0}^{t}\mathbb{E}[c(x_{j},A_{j},W_{j+1})|\mathcal{G}_{0},X_{0}]$ (29)

over all $\mathcal{F}_{n}\vee\mathcal{H}_{n}\vee\mathcal{G}_{0}$-adapted
policies $(A_{j}\mathrel{\mathop{\mathchar 58\relax}}0\leq j\leq t)$. For
$1\leq i\leq t$, define the operator $P_{a,i}(\mathcal{G}_{0})$ via

$(\mathbb{P}_{a,i}(\mathcal{G}_{0})h)(x,w)=\int_{\mathbb{R}^{d}}\int_{\mathbb{R}^{m_{2}}}h(\phi(x,a,Gw+z,v),Gw+z)\mathbb{P}(Z_{i}(\mathcal{G}_{0})\in
dz)\mathbb{P}(v_{1}\in dv)$ (30)

and set

$\tilde{c}_{i}(\mathcal{G}_{0},x,a,w)\int_{\mathbb{R}^{d}}c(x,a,Gw+z)\mathbb{P}(Z_{i}(\mathcal{G}_{0})\in
dz)$ (31)

for $1\leq i\leq t+1$. The appropriate value function backwards recursion in
this setting is then given by

$v_{i}(x,w)=\min_{a\in\mathcal{A}}[\tilde{c}_{i+1}(\mathcal{G}_{0},x,a,w)+(P_{a,i+1}(\mathcal{G}_{0})v_{i+1})(x,w)]$
(32)

for $0\leq i<t$, subject to the terminal condition

$v_{t}(x,w)=\min_{a\in\mathcal{A}}\tilde{c}_{t+1}(\mathcal{G}_{0},x,a,w).$
(33)

As in Section 2, an optimal
$\mathcal{F}_{n}\vee\mathcal{H}_{n}\vee\mathcal{G}_{0}$-adapted policy is then
given by $A_{i}^{*}=a_{i}^{*}(X_{i},W_{i})$ for $0\leq i\leq t$, where
$a_{i}^{*}(x,w)$ is any minimizer of the right-hand side of 32 and 33.

##  5 MDP’s incorporating a dynamic forecast

In this section, we discuss how our energy system manager should modify her
decision-making when she has access to a new set of forecasts each day. More
precisely, suppose that at each time $k$ through the decision horizon, the
decision maker receives the forecasts $(F_{n|k}\mathrel{\mathop{\mathchar
58\relax}}n\geq k)$ prior to making the decision for that period. Now, the
decision made at time $k$ can depend on both $W_{k}$ and the $F_{n|k}$’s. In
particular, the decision can now be
$\mathcal{F}_{k}\vee\mathcal{K}_{k}$-adapted. Since there is more information
about $(W_{n}\mathrel{\mathop{\mathchar 58\relax}}n\geq k)$ available when one
uses the forecasts, this will typically modify the optimal control relative to
the previously discussed formulations of Sections 2 and 4. Since the forecasts
used by the decision maker are constantly updated as $k$ increases, we refer
to this setting as a dynamic forecast formulation.

Let $\vec{F}_{n}=(F_{n+j|n}\mathrel{\mathop{\mathchar
58\relax}}j\in\mathbb{Z}_{+})$ be the entire set of forward forecasts issued
at time $n$ (and computed from the history $\mathcal{G}_{n}$). Recall that
$W_{n}=F_{n|n}$. We claim that the infinite-dimensional process
$(\vec{F}_{n}\mathrel{\mathop{\mathchar 58\relax}}n\in\mathbb{Z})$ is a Markov
chain. To see this, observe that 20 implies that

$\begin{split}F_{n+1+j|n+1}&=F_{n+j+1|n}+D_{n+j+1|n+1}\\\
&=F_{n+j+1|n}+\sum_{i=0}^{j}G^{i}\epsilon_{n+1+j-i}(n+1)\end{split}$ (34)

for $j\geq 0$. Since the collection of rv’s
$(\epsilon_{n+1+j}(n+1)\mathrel{\mathop{\mathchar 58\relax}}j\geq 0)$ is
independent of $\mathcal{G}_{n}$, it follows that
$(\vec{F}_{n}\mathrel{\mathop{\mathchar 58\relax}}n\in\mathbb{Z})$ is a Markov
chain. One important and related characteristic of our model is that the
Markov chain can be initialized from an arbitrary set of values. This means
that our model is consistent with any set of forecast values specified at time
$0$.

Of course, we cannot effectively compute optimal policies with a Markov chain
having an infinite dimensional state space. So, we need to truncate the set of
forecasts that we use within our formulation in order to generate a finite
dimensional Markov state variable. In particular, suppose that
$\mathcal{G}_{n,r}$ is the smallest $\sigma$-algebra containing both
$\mathcal{H}_{n}$ and the $\sigma$-algebra
$\sigma(\epsilon_{n+j}(k)\mathrel{\mathop{\mathchar 58\relax}}k\leq n,1\leq
j\leq r)$, so that it contains only the forecaster’s information about the $r$
future forecasts for periods $n+1,\cdots,n+r$, in addition to the information
associated with $\mathcal{H}_{n}$. Figure 4 illustrates the weather-related
information set associated with $\mathcal{G}_{n,r}$. We note that
$\mathcal{G}_{n}\supseteq\mathcal{G}_{n,r}$ and that for $1\leq j\leq r$,
$F_{n+j|n}=\mathbb{E}[W_{n+j}|\mathcal{G}_{n}]$ is a function only of rv’s
associated with $\mathcal{G}_{n,r}$, and hence is
$\mathcal{G}_{n,r}$-measurable.

Using the information associated with $\mathcal{G}_{n,r}$, we can use the
recursion in 34 for $0\leq j<r$. For $j=r$, we can use 23 to expand
$F_{n+1+r|n}$, which yields the recursion

$\begin{split}F_{n+1+r|n+1}&=F_{n+1+r|n}+D_{n+1+r|n+1}\\\
&=GF_{n+r|n}+\mathbb{E}Z_{0}+\sum_{j=-\infty}^{n}\epsilon_{n+1+r}(j)+\sum_{i=0}^{r}G^{i}\epsilon_{n+1+r-i}(n+1).\end{split}$
(35)

$\begin{matrix}\leavevmode\hbox to7.5pt{\vbox
to9pt{\pgfpicture\makeatletter\hbox{\hskip 3.75pt\lower-4.5pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-3.75pt}{-2.5pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$\ddots$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}&&&&\\\
\cdots&\epsilon_{-1}(-1)&&&\\\ \cdots&\epsilon_{0}(-1)&\epsilon_{0}(0)&&\\\
&\vdots&\vdots&\ddots&\\\
\cdots&\epsilon_{n-1}(-1)&\epsilon_{n-1}(0)&\cdots&\epsilon_{n-1}(n-1)\\\
\cdots&\epsilon_{n}(-1)&\epsilon_{n}(0)&\cdots&\epsilon_{n}(n-1)&\epsilon_{n}(n)\leavevmode\hbox
to1pt{\vbox to1pt{\pgfpicture\makeatletter\hbox{\hskip 0.5pt\lower-0.5pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{0.0pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}\\\
\cdots&\epsilon_{n+1}(-1)&\epsilon_{n+1}(0)&\cdots&\epsilon_{n+1}(n-1)&\epsilon_{n+1}(n)&\epsilon_{n+1}(n+1)\\\
&\vdots&\vdots&&\vdots&\vdots&\vdots\\\ \leavevmode\hbox to7.5pt{\vbox
to9pt{\pgfpicture\makeatletter\hbox{\hskip 3.75pt\lower-4.5pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-3.75pt}{-2.5pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$\cdots$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}&\leavevmode\hbox
to32.79pt{\vbox to10pt{\pgfpicture\makeatletter\hbox{\hskip
16.39563pt\lower-5.0pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-16.39563pt}{-2.5pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$\epsilon_{n+r}(-1)$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}&\epsilon_{n+r}(0)&\cdots&\epsilon_{n+r}(n-1)&\epsilon_{n+r}(n)\leavevmode\hbox
to1pt{\vbox to1pt{\pgfpicture\makeatletter\hbox{\hskip 0.5pt\lower-0.5pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{0.0pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}&\epsilon_{n+r}(n+1)&\cdots\\\
\cdots&{\epsilon_{n+r+1}(-1)}&\epsilon_{n+r+1}(0)&\cdots&\epsilon_{n+r+1}(n-1)&\epsilon_{n+r+1}(n)&\epsilon_{n+r+1}(n+1)&\cdots\\\
&\vdots&\vdots&&\vdots&\vdots&\vdots\end{matrix}$

Figure 4: Weather-related information set associated with $\mathcal{G}_{n,r}$.

As a result,
$\vec{F}_{n+1,r}\triangleq(F_{n+1+j|n+1}\mathrel{\mathop{\mathchar
58\relax}}1\leq j\leq r)$ is a linear function of $\vec{F}_{n,r}$ and a
collection of rv’s
$(\epsilon_{n+1+i}(n+1),\epsilon_{n+1+r}(j)\mathrel{\mathop{\mathchar
58\relax}}1\leq i\leq r,j\leq n)$ that are independent of $\mathcal{G}_{n,r}$.
It follows that $(\vec{F}_{n,r}\mathrel{\mathop{\mathchar
58\relax}}n\in\mathbb{Z})$ is an $rd$-dimensional Markov chain. It is also
easily seen that it is a Markov chain with stationary transition
probabilities. Furthermore, $W_{n+1}=F_{n+1|n+1}$ is a simple stochastic
function of $\vec{F}_{n,r}$, specifically

$W_{n+1}=F_{n+1|n}+\epsilon_{n+1}(n+1),$ (36)

so that it can easily be generated from $\vec{F}_{n,r}$ simultaneously with
$\vec{F}_{n+1,r}$. We can now turn to the computation of the optimal policy in
this setting. In particular, we seek the
$\mathcal{F}_{n}\vee\mathcal{G}_{n,r}$-adapted policy that minimizes

$\sum_{j=0}^{t}\mathbb{E}[c(X_{j},A_{j},W_{j+1})|X_{0},\vec{F}_{0,r}]$ (37)

over all $\mathcal{F}_{n}\vee\mathcal{G}_{n,r}$-adapted policies
$(A_{n}\mathrel{\mathop{\mathchar 58\relax}}0\leq n\leq t)$. Define the
operator $P_{a}$ (acting on integrable functions $h$) via

$(P_{a}h)(x,\vec{f})=\mathbb{E}[h(\phi(x,a,W_{1},V_{1}),\vec{F}_{1,r})|\vec{F}_{0,r}=\vec{f})]$
(38)

and let

$\tilde{c}(x,a,\vec{f})=\mathbb{E}[c(x,a,W_{1})|\vec{F}_{0,r}=\vec{f})].$ (39)

We can then compute the associated value functions for this formulation via
the backwards recursion

$v_{i}(x,\vec{f})=\min_{a\in\mathcal{A}}[\tilde{c}(x,a,\vec{f})+(P_{a}v_{i+1})(x,\vec{f})]$
(40)

for $0\leq i<t$, subject to the terminal condition

$v_{t}(x,\vec{f})=\min_{a\in\mathcal{A}}\tilde{c}(x,a,\vec{f}).$ (41)

The optimal $\mathcal{F}_{n}\vee\mathcal{G}_{n,r}$-adapted action $A_{n}^{*}$
to be taken in period $n$ is then given by
$A_{n}^{*}=a_{n}^{*}(X_{n},\vec{F}_{n,r})$, where $a_{n}^{*}(x,\vec{f})$ is
the minimizer of the right-hand side of 40 or 41 corresponding to
$v_{n}(x,\vec{f})$ for $0\leq n\leq t$.

### Remark

We note that the use of the reduced Markov state variable $\vec{F}_{n,r}$ for
the weather variables (as opposed to using the state variable
$(W_{n},\vec{F}_{n,r})$) is possible only because we made the modeling
decision in Section 2 to express the control state recursion in the form

$X_{n+1}=\phi(X_{n},A_{n},W_{n+1},V_{n+1})$ (42)

and cost $c(X_{m},A_{n},W_{n+1})$ in terms of $W_{n+1}$ rather than $W_{n}$.
If we had instead modeled the control state evolution via

$X_{n+1}=\phi(X_{n},A_{n},W_{n},V_{n+1})$ (43)

and/or cost $c(X_{m},A_{n},W_{n})$, then the decision maker at time $n$ would
need to know $W_{n}$, and $W_{n}$ would then need to be added to the Markov
state variable for the weather. Since either choice, $W_{n}$ or $W_{n+1}$, is
typically reasonable from a modeling viewpoint, we choose to use $W_{n+1}$ in
order to obtain this state reduction.

##  6 MDP’s incorporating both static and dynamic forecasts

For computational tractability, the value of $r$ used in $\vec{F}_{n,r}$ will
typically need to be small. But weather forecasters will typically provide
forward forecasts over a much larger number of periods. In order to
(partially) account for these longer range forecasts (without expanding our
state description for the MDP), we now build a formulation that takes into
account all the forward forecasts that are present in $\mathcal{G}_{0}$ (i.e.
the static forecasts that are available at time $0$), as well as the dynamic
forecasts associated with $\mathcal{G}_{n,r}$ for $1\leq n\leq t$. Thus, in
this formulation, the decision maker at time $n$ has access to
$X_{n},F_{n+1|n},\cdots,F_{n+r|n}$ and $F_{j|0}$ for $j\geq 1$. Figure 5
illustrates the weather-related information set corresponding to
$\mathcal{G}_{0}\vee\mathcal{G}_{n,r}$.

$\begin{matrix}\leavevmode\hbox to7.5pt{\vbox
to9pt{\pgfpicture\makeatletter\hbox{\hskip 3.75pt\lower-4.5pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-3.75pt}{-2.5pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$\ddots$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}&&&&\\\
\cdots&\epsilon_{-1}(-1)&&&\\\ \cdots&\epsilon_{0}(-1)&\epsilon_{0}(0)&&\\\
&\vdots&\vdots&\ddots&\\\
\cdots&\epsilon_{n-1}(-1)&\epsilon_{n-1}(0)&\cdots&\epsilon_{n-1}(n-1)\\\
\cdots&\epsilon_{n}(-1)&\epsilon_{n}(0)&\cdots&\epsilon_{n}(n-1)&\epsilon_{n}(n)\leavevmode\hbox
to1pt{\vbox to1pt{\pgfpicture\makeatletter\hbox{\hskip 0.5pt\lower-0.5pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{0.0pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}\\\
\cdots&\epsilon_{n+1}(-1)&\epsilon_{n+1}(0)&\cdots&\epsilon_{n+1}(n-1)&\epsilon_{n+1}(n)&\epsilon_{n+1}(n+1)\\\
&\vdots&\vdots&&\vdots&\vdots&\vdots\\\
\cdots&\epsilon_{n+r}(-1)&\epsilon_{n+r}(0)&\cdots&\epsilon_{n+r}(n-1)&\epsilon_{n+r}(n)\leavevmode\hbox
to1pt{\vbox to1pt{\pgfpicture\makeatletter\hbox{\hskip 0.5pt\lower-0.5pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{0.0pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}&\epsilon_{n+r}(n+1)&\cdots\\\
\cdots&\epsilon_{n+r+1}(-1)&\leavevmode\hbox to34.39pt{\vbox
to10pt{\pgfpicture\makeatletter\hbox{\hskip 17.19566pt\lower-5.0pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-17.19566pt}{-2.5pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\epsilon_{n+r+1}(0)$}} }}\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}&\cdots&\epsilon_{n+r+1}(n-1)&\epsilon_{n+r+1}(n)\leavevmode\hbox
to1pt{\vbox to1pt{\pgfpicture\makeatletter\hbox{\hskip 0.5pt\lower-0.5pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{0.0pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}&\epsilon_{n+r+1}(n+1)&\cdots\\\
\leavevmode\hbox to1pt{\vbox to1pt{\pgfpicture\makeatletter\hbox{\hskip
0.5pt\lower-0.5pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{0.0pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}&\vdots&\vdots\leavevmode\hbox
to1pt{\vbox to1pt{\pgfpicture\makeatletter\hbox{\hskip 0.5pt\lower-0.5pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{0.0pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}&&\vdots&\vdots&\vdots\end{matrix}$

Figure 5: Weather-related information set associated with
$\mathcal{G}_{0}\vee\mathcal{G}_{n,r}$.

We now turn to the conditional dynamics of
$(\vec{F}_{n,r}\mathrel{\mathop{\mathchar 58\relax}}n\geq 0)$, conditional on
$\mathcal{G}_{0}$. Define the $\vec{F}_{n,r}(\mathcal{G}_{0})$’s via

$\mathbb{P}((\vec{F}_{n,r}(\mathcal{G}_{0})\mathrel{\mathop{\mathchar
58\relax}}n\geq
0)\in\cdot)=\mathbb{P}((\vec{F}_{n,r}\mathrel{\mathop{\mathchar
58\relax}}n\geq 0)\in\cdot|\mathcal{G}_{0})$ (44)

Because the martingale difference terms on the right-hand side of 34 are
independent of $\mathcal{G}_{0}$,

$F_{n+1+j|n+1}(\mathcal{G}_{0})=F_{n+1+j|n}(\mathcal{G}_{0})+\beta_{n+1,j}(\mathcal{G}_{0})$
(45)

for $0\leq j<r$, where $\beta_{n+1,j}(\mathcal{G}_{0})$ is independent of
$\mathcal{G}_{0}\vee\mathcal{G}_{n,r}$ and

$\beta_{n+1,j}(\mathcal{G}_{0})\stackrel{{\scriptstyle
D}}{{=}}\sum_{i=0}^{j}G^{i}\epsilon_{n+1+j-i}(n+1)$ (46)

On the other hand, the right-hand side of 35 contains terms that are
$\mathcal{G}_{0}$-measurable. In particular,

$F_{n+1+r|n+1}(\mathcal{G}_{0})=GF_{n+r|n}(\mathcal{G}_{0})+\mathbb{E}Z_{0}+\beta_{n+1,r}(\mathcal{G}_{0})$
(47)

where

$\beta_{n+1,r}(\mathcal{G}_{0})=\sum_{j=-\infty}^{0}\epsilon_{n+1+r}(j)+\Lambda_{n+1,r}$
(48)

where $\sum_{j=-\infty}^{0}\epsilon_{n+1+r}(j)$ is
$\mathcal{G}_{0}$-measurable and $\Lambda_{n+1,r}$ is independent of
$\mathcal{G}_{0}\vee\mathcal{G}_{n,r}$, where

$\Lambda_{n+1,r}\stackrel{{\scriptstyle
D}}{{=}}\sum_{j=1}^{n}\epsilon_{n+1+r}(j)+\sum_{i=0}^{r}G^{i}\epsilon_{n+1+r-i}(n+1).$
(49)

Since $\vec{F}_{n+1,r}(\mathcal{G}_{0})$ can be expressed as a function of
$\vec{F}_{n,r}(\mathcal{G}_{0})$ and a family of rv’s
$\beta_{n+1}(\mathcal{G}_{0})\triangleq(\beta_{n+1,j}(\mathcal{G}_{0})\mathrel{\mathop{\mathchar
58\relax}}1\leq j\leq r)$ that are independent of $\mathcal{H}_{n}^{0}$, it
follows that $(\vec{F}_{n,r}(\mathcal{G}_{0})\mathrel{\mathop{\mathchar
58\relax}}n\geq 0)$ is (conditional on $\mathcal{G}_{0}$) a Markov chain.
However, as with the Markov chain of Section 4, the conditioning on
$\mathcal{G}_{0}$ makes this a Markov chain with non-stationary transition
probabilities; see 48 in particular. Furthermore, as in Section 4, the
variance of $\beta_{n+1,r}(\mathcal{G}_{0})$ (conditional on
$\mathcal{G}_{0}$) increases in $n$, so that the “uncertainty plume” increases
over time.

We now turn to the computation of a policy
$(A_{n}^{*}\mathrel{\mathop{\mathchar 58\relax}}0\leq n\leq t)$ that minimizes

$\mathbb{E}[\sum_{j=0}^{t}c(X_{j},A_{j},W_{j+1})|X_{0},\mathcal{G}_{0}]$ (50)

over all policies $(A_{n}\mathrel{\mathop{\mathchar 58\relax}}0\leq n\leq t)$
that are $\mathcal{H}_{n}^{0}$-adapted. For
$\vec{f}=(f_{1},f_{2},\cdots,f_{r})$, define the operator
$P_{a,i}(\mathcal{G}_{0})$ (acting on integrable functions $h$) via

$\begin{split}(P_{a,i}(\mathcal{G}_{0})&h)(x,\vec{f})=\int_{\mathbb{R}^{(r+1)d}}\int_{\mathbb{R}^{m_{2}}}h(\phi(x,a,w,v),y_{1},y_{2},\dots,y_{r})\mathbb{P}(f_{1}+\beta_{i,0}(\mathcal{G}_{0})\in
dw,\\\ &f_{j+1}+\beta_{i,j}(\mathcal{G}_{0})\in dy_{j},1\leq j\leq
r,Gf_{r}+\mathbb{E}Z_{0}+\beta_{i,r}(\mathcal{G}_{0})\in
dy_{r})\mathbb{P}(V_{1}\in dv)\end{split}$ (51)

and set

$\tilde{c}_{i}(\mathcal{G}_{0},x,a,\vec{f})\int_{\mathbb{R}^{d}}c(x,a,w)\mathbb{P}(f_{1}+\beta_{i,0}(\mathcal{G}_{0})\in
dw)$ (52)

for $1\leq i\leq t+1$. As in Section 5, the value function recursion takes the
form

$v_{i}(x,\vec{f})=\min_{a}[\tilde{c}_{i+1}(\mathcal{G}_{0},x,a,\vec{f})+(P_{a,i}(\mathcal{G}_{0})v_{i+1})(x,\vec{f})]$
(53)

for $1\leq i<t$, with terminal condition

$v_{t}(x,\vec{f})=\min_{a}\tilde{c}_{t}(\mathcal{G}_{0},x,a,\vec{f}).$ (54)

Again, the optimal $\mathcal{H}_{n}^{0}$-adapted policy is then given by
$A_{i}^{*}=a_{i}^{*}(X_{i},\vec{F}_{i})$ for $0\leq i\leq t$, where
$a_{i}^{*}(x,\vec{f})$ is any minimizer of the right-hand side of 53 and 54.

##  7 An energy control system example

In this section, we illustrate some of our theory in the setting of a simple
energy control system example. In particular, we let $W_{n}$ represent the
ambient outdoors temperature at the beginning of period $n$ at the site of the
energy system that is under control. We assume that
$(W_{n}\mathrel{\mathop{\mathchar 58\relax}}n\in\mathbb{Z})$ is a real-valued
Markov chain corresponding to a first order autoregressive process, so that

$W_{n+1}=gW_{n}+Z_{n+1}$ (55)

for $n\in\mathbb{Z}$, where $g\in(0,1)$ and the $Z_{i}$’s are iid with
$\mathbb{E}Z_{0}^{2}<\infty$. To help interpret $g$, we note that
$\text{corr}(W_{j},W_{j+n})=g^{n}$, so that the number of periods for the
correlation to decay to $0.1$ is approximately $\log(0.1)/\log g$.

We now describe our simplified energy control system corresponding to heating
and cooling a building. We assume that the difference $\Delta_{n}(\triangleq
X_{n}-W_{n})$ between the internal ($X_{n}$) and external ($W_{n}$)
temperatures is “mean reverting”, so that the $\Delta_{n}$’s satisfy their own
first order autoregression. In particular, in the absence of control,

$\Delta_{n+1}=\rho\Delta_{n}+V_{n+1},$ (56)

where the $V_{j}$’s are iid and independent of the $Z_{k}$’s with
$\mathbb{E}V_{0}^{2}<\infty$. We expect the building to equilibrate more
rapidly than does the outdoors temperature, so we expect $\rho\in(0,g)$.
Substituting 55 into 56, we find that in the presence of the control $A_{n}$,

$X_{n+1}=(g-\rho)W_{n}+\rho X_{n}+Z_{n+1}+V_{n+1}+A_{n}$ (57)

for $n\in\mathbb{Z}$.

We now wish to take advantage of the powerful toolset that is available when
our state space model has a quadratic cost structure. We assume that our goal
is to minimize the expected infinite horizon discounted cost given by

$\mathbb{E}\sum_{j=0}^{\infty}\alpha^{j}\left[(X_{j}-\tau)^{2}+\kappa
A_{j}^{2}\right]$ (58)

over all $\mathcal{F}_{j}$-adapted controls, where $\kappa>0$,
$\alpha\in(0,1)$ is the discount factor, and $\tau$ is the reference
temperature to which we are trying to steer the system. To incorporate $\tau$
into the linear/quadratic formulation, we add $Y_{j}$ as a state variable for
which $Y_{j}=Y_{j-1}$ for $j\in\mathbb{Z}$. Furthermore, we let
$\tilde{W}_{j}=W_{j}-(1-g)^{-1}\mathbb{E}Z_{0}$,
$\tilde{Z}_{j}=Z_{j}-\mathbb{E}Z_{0}$, $\tilde{X}_{j}=X_{j}-\tau_{0}$,
$\tau_{0}=(1-g)^{-1}\mathbb{E}Z_{0}+(1-\rho)^{-1}\mathbb{E}V_{0}$ and
$\tilde{V}_{j}=V_{j}-\mathbb{E}V_{0}$, and rewrite 55 and 57 in terms of the
mean zero “noise” rv’s $\tilde{Z}_{n+1}$ and $\tilde{V}_{n+1}$:

$\begin{split}\tilde{W}_{n+1}&=g\tilde{W}_{n}+\tilde{Z}_{n+1}\\\
\tilde{X}_{n+1}&=(g-\rho)\tilde{W}_{n}+\rho\tilde{X}_{n}+\tilde{Z}_{n+1}+\tilde{V}_{n+1}+A_{n}.\end{split}$
(59)

Furthermore, we can express $X_{n}-\tau$ as $\tilde{X}_{n}-Y_{n}$, where we
take $Y_{0}=\tau-\tau_{0}$.

Set $\chi_{n}=(\tilde{W}_{n},\tilde{X}_{n},Y_{n})^{T}$ and
$\xi_{n}=(\tilde{Z}_{n},\tilde{Z}_{n}+\tilde{V}_{n},0)^{T}$, and observe that
we can express our control system dynamics as

$\chi_{n+1}=A\chi_{n}+BA_{n}+\xi_{n+1}$ (60)

for $n\geq 0$, where

$A=\begin{pmatrix}g&0&0\\\ g-\rho&\rho&0\\\ 0&0&1\end{pmatrix},\qquad
B=\begin{pmatrix}0\\\ 1\\\ 0\end{pmatrix}.$ (61)

The objective 58 can then be re-expressed as

$\mathbb{E}\sum_{j=0}^{\infty}\alpha^{j}[\chi_{j}^{T}Q\chi_{j}+A_{j}^{T}RA_{j}],$
(62)

where

$Q=\begin{pmatrix}0&0&0\\\ 0&1&-1\\\ 0&-1&1\end{pmatrix},\qquad R=\kappa.$
(63)

We observe that this model does not satisfy the standard controllabillity
hypothesis that is commonly used within the literature on state space models
with quadratic costs (in particular, the $Y_{j}$’s are not controllable).
Nevertheless, the special problem structure here allows us to follow the
approach on p. 231-233 of [18] to obtain the solution of this stochastic
control problem in closed form.

In particular, define the optimal return operator $\mathcal{T}$ (defined on
suitably integrable functions $h$) via

$(\mathcal{T}h)(\vec{z})=\min_{a}[\vec{z}^{T}Q\vec{z}+a^{2}\kappa+\alpha\mathbb{E}h(A\vec{z}+Ba+\xi_{1})]$
(64)

for $\vec{z}=(w,x,y)^{T}$ and note that this stochastic control problem
corresponds to a positive dynamic program; see p. 214 of [18]. Hence, if
$v_{0}(\vec{x})\equiv 0$, it follows that $\mathcal{T}v_{0}\geq v_{0}$,
thereby implying that
$v_{k}\equiv\mathcal{T}^{k}v_{0}\geq\mathcal{T}^{k-1}v_{0}=v_{k-1}$, so that
the $v_{k}$’s are functions that are monotone increasing in $k$. Hence,

$v_{\infty}=\lim_{k\to\infty}v_{k}$ (65)

exists. Furthermore, if $h_{0}(\vec{z})\equiv 0$ and

$h_{k+1}(\vec{z})=\vec{z}^{T}Q\vec{z}+\alpha\mathbb{E}h_{k}(A\vec{z}+\xi)$
(66)

for $k\geq 0$, $v_{k}\leq h_{k}$ for $k\geq 0$. As for the $v_{k}$’s,
$(h_{k}\mathrel{\mathop{\mathchar 58\relax}}k\geq 0)$ is also a monotone
sequence so that $h_{k}\to h_{\infty}$. The limit $h_{\infty}$ is the value
function corresponding to the policy in which $A_{k}=0$ for $k\geq 0$. Since
$|g|<1$ and $|\rho|<1$, the associated stochastic dynamical system is stable
and $h_{\infty}$ is finite-valued. We can therefore conclude that $v_{\infty}$
is finite-valued.

We further note that if $J=(J(i,k)\mathrel{\mathop{\mathchar 58\relax}}1\leq
i,k\leq 3)$ is a symmetric non-negative definite matrix, the scalar $\alpha
B^{T}JB+R=\alpha J(2,2)+\kappa>0$ (since the diagonal entries of such a matrix
must be non-negative), so that $\alpha B^{T}JB+R$ is guaranteed to be non-
singular. As a result, the matrix recursion

$K_{j+1}=A^{T}(\alpha K_{j}-\alpha^{2}K_{j}B(\alpha
B^{T}K_{j}B+R)^{-1}B^{T}K_{j})A+Q,$ (67)

subject to $K_{0}=Q$, is well-defined, and

$v_{j+1}(\vec{z})=\vec{z}^{T}K_{j}\vec{z}+\sum_{i=0}^{j-1}\alpha^{j-i}\mathbb{E}\xi_{1}^{T}K_{i}\xi_{1};$
(68)

see p. 231 of [18]. By following the argument on p. 156 of [19], we can
conclude that there exists a finite-valued non-negative definite matrix
$K_{\infty}=(K_{\infty}(i,k)\mathrel{\mathop{\mathchar 58\relax}}1\leq i,k\leq
3)$ for which $K_{j}\to K_{\infty}$ as $j\to\infty$. Taking limits in 67, we
find that $K_{\infty}$ satisfies the matrix Ricatti equation

$K_{\infty}=A^{T}(\alpha K_{\infty}-\alpha^{2}K_{\infty}B(\alpha
B^{T}K_{\infty}B+R)^{-1}B^{T}K_{\infty})A+Q.$ (69)

Furthermore, as seen from p. 232 of [18], we conclude that the optimal value
function for the control problem is

$v_{\infty}(\vec{z})=\vec{z}^{T}K_{\infty}\vec{z}+\frac{\alpha}{1-\alpha}\mathbb{E}\xi_{1}^{T}K_{\infty}\xi_{1},$
(70)

and the associated optimal action $A^{*}_{j}$ to be taken at time $j$ is

$\begin{split}A^{*}_{j}&=-\alpha(\alpha
B^{T}K_{\infty}B+\kappa)^{-1}B^{T}K_{\infty}A\chi_{j}\\\ &=-\alpha(\alpha
K_{\infty}(2,2)+\kappa)^{-1}[K_{\infty}(2,1)g\tilde{W}_{j}+K_{\infty}(2,2)((g-\rho)W_{j}+\rho\tilde{X}_{j})\\\
&+K_{\infty}(3,3)[\tau-\tau_{0}]].\end{split}$ (71)

We now turn to the analyzing exactly the same MDP when the dynamic forecasts
of Section 5 are incorporated into the problem. To simplify our exposition, we
set $r=2$, so that our energy system manager has access to the forecasts
$F_{n+1|n}$ and $F_{n+2|n}$ (in addition to $W_{n}$ and $X_{n}$) at the time
that the decision at time $n$ is taken. Put
$\tilde{F}_{n+i|n}=F_{n+i|n}-\mathbb{E}Z_{0}/(1-g)$, for $i=1,2$ and
$n\in\mathbb{Z}$. If
$\underaccent{\tilde}{\chi}_{n}=(\tilde{W}_{n},\tilde{F}_{n+1|n},\tilde{F}_{n+2|n},\tilde{X}_{n},Y_{n})^{T}$,
Section 5’s discussion establishes that

$\underaccent{\tilde}{\chi}_{n+1}=\mathcal{A}\underaccent{\tilde}{\chi}_{n}+\mathcal{B}A_{n}+\underaccent{\tilde}{\xi}_{n+1},$
(72)

where

$\mathcal{A}=\begin{pmatrix}0&1&0&0&0\\\ 0&0&1&0&0\\\ 0&0&g&0&0\\\
-\rho&1&0&\rho&0\\\
0&0&0&0&1\end{pmatrix},\qquad\mathcal{B}=\begin{pmatrix}0\\\ 0\\\ 0\\\ 1\\\
0\end{pmatrix},$ (73)

and

$\underaccent{\tilde}{\xi}_{n}=\begin{pmatrix}\epsilon_{n+1}(n+1)\\\
\epsilon_{n+2}(n+1)+g\epsilon_{n+1}(n+1)\\\
\epsilon_{n+3}(n+1)+g\epsilon_{n+2}(n+1)+g^{2}\epsilon_{n+1}(n+1)+\sum_{j=-\infty}^{n}\epsilon_{n+3}(j)\\\
\epsilon_{n+1}(n+1)+\tilde{V}_{n+1}\\\ 0\end{pmatrix}.$ (74)

We can also re-express the objective 58 in terms of the
$\underaccent{\tilde}{\chi}_{j}$’s, namely

$\mathbb{E}\sum_{j=0}^{\infty}\alpha^{j}\left[\underaccent{\tilde}{\chi}_{j}^{T}\mathcal{Q}\underaccent{\tilde}{\chi}_{j}+A_{j}RA_{j}\right]$
(75)

where

$\mathcal{Q}=\begin{pmatrix}0&0&0&0&0\\\ 0&0&0&0&0\\\ 0&0&0&0&0\\\
0&0&0&1&-1\\\ 0&0&0&-1&1\end{pmatrix}.$ (76)

We now assume a specific probability structure on the $\epsilon_{n+j}(n)$’s
and $V_{n}$’s, namely that the $\epsilon_{n+j}(n)$’s and $V_{n}$’s are
independent normally distributed rv’s with
$\text{var}\,\epsilon_{n+j}(n)=\sigma^{2}\gamma^{2j}$ and
$\text{var}\,V_{n}\triangleq\sigma_{V}^{2}$, where $\gamma\in(0,1)$. With this
choice, the $\underaccent{\tilde}{\xi}_{n}$’s are iid multivariate normally
distributed random vectors with common matrix covariance $C$ defined by

$C=\begin{pmatrix}\sigma^{2}&\sigma^{2}g&\sigma^{2}g^{2}&\sigma^{2}&0\\\
\sigma^{2}g&\sigma^{2}(\gamma^{2}+g^{2})&\sigma^{2}(\gamma^{2}g+g^{3})&\sigma^{2}g&0\\\
\sigma^{2}g^{2}&\sigma^{2}(\gamma^{2}g+g^{3})&\sigma^{2}(\gamma^{4}+\gamma^{2}g^{2}+g^{4}+\frac{\gamma^{6}}{1-\gamma^{2}})&\sigma^{2}g^{2}&0\\\
\sigma^{2}&\sigma^{2}g&\sigma^{2}g^{2}&\sigma^{2}+\sigma_{V}^{2}&0\\\
0&0&0&0&0\end{pmatrix}.$ (77)

The solution of the stochastic control problem defined by the quadratic
objective 75 subject to the state specific dynamics 72 follows the same lines
as for the control system 60 and 62 without forecasts. In particular, there
exists a solution to the matrix Ricatti equation

$\mathcal{K}_{\infty}=\mathcal{A}^{T}(\alpha\mathcal{K}_{\infty}-\alpha^{2}\mathcal{K}_{\infty}\mathcal{B}(\alpha\mathcal{K}_{\infty}(4,4)+\kappa)^{-1}\mathcal{B}^{T}\mathcal{K}_{\infty})\mathcal{A}+\mathcal{Q}$
(78)

where $\mathcal{K}_{\infty}(4,4)$ is the $(4,4)$ entry of
$\mathcal{K}_{\infty}$ and the optimal value function
$\underaccent{\tilde}{v}_{n}$ for this MDP is given by

$\underaccent{\tilde}{v}_{\infty}(\underaccent{\tilde}{\Vec{z}})=\underaccent{\tilde}{\Vec{z}}^{T}\mathcal{K}_{\infty}\underaccent{\tilde}{\Vec{z}}-\frac{\alpha}{1-\alpha}\mathbb{E}\underaccent{\tilde}{\xi}^{T}\mathcal{K}_{\infty}\underaccent{\tilde}{\xi}$
(79)

where $\underaccent{\tilde}{\Vec{z}}=(w,f_{1},f_{2},x,y)^{T}$. Furthermore,
the optimal action adapted to $\mathcal{G}_{n,r}$ is given by

$A^{*}_{n}=-\alpha(\alpha\mathcal{K}_{\infty}(4,4)+\kappa)^{-1}\mathcal{B}\mathcal{K}_{\infty}\mathcal{A}\underaccent{\tilde}{\chi}_{n}.$
(80)

To compare the two optimal controls, we note that the system is controlled
only over $[0,\infty)$ and uncontrolled over $(-\infty,0)$. Hence, the
distribution of the control system at time $0$ is given by the distribution of
$((\tilde{W}_{0},\tilde{X}_{0})\mathrel{\mathop{\mathchar 58\relax}}n\leq 0)$
for the system without forecasts and by the distribution of
$(\underaccent{\tilde}{\chi}_{0}\mathrel{\mathop{\mathchar 58\relax}}n\leq 0)$
for the system with forecasts.

The expected cost for the system with the optimal $\mathcal{F}_{n}$-adapted
policy is therefore
$\mathbb{E}v_{\infty}(\tilde{W}_{0},\tilde{X}_{0},\tau-\tau_{0})$, while the
expected cost for the optimal $\mathcal{G}_{n,r}$-adapted policy is
$\mathbb{E}\underaccent{\tilde}{v}_{\infty}(\underaccent{\tilde}{\chi}_{0})$.
To explicitly compute the distribution of $(\tilde{W}_{0},\tilde{X}_{0})$, we
note that the uncontrolled system is Gaussian with zero mean, so the
distribution of $(\tilde{W}_{0},\tilde{X}_{0})$ is completely determined by
its covariance structure. Because the $W_{j}$’s and $\Delta_{j}$’s are
independent first order autoregressive processes,
$\text{var}\,\tilde{W}_{0}=(1-g^{2})^{-1}\text{var}\,Z_{0}$ and
$\text{var}\,\tilde{\Delta}_{0}=(1-\rho^{2})^{-1}\text{var}\,V_{0}$. Since
$X_{0}=W_{0}+\Delta_{0}$, it follows that
$\text{var}\,\tilde{X}_{0}=(1-g^{2})^{-1}\text{var}\,Z_{0}+(1-\rho^{2})^{-1}\text{var}\,V_{0}$
and
$\text{cov}\,(\tilde{W}_{0},\tilde{X}_{0})=(1-g^{2})^{-1}\text{var}\,Z_{0}$.

It follows that

$\begin{split}\mathbb{E}v_{\infty}&\left(\tilde{W}_{0},\tilde{X}_{0},\tau-(1-g)^{-1}\mathbb{E}Z_{0}-(1-\rho)^{-1}\mathbb{E}V_{0}\right)\\\
&=K_{\infty}(1,1)\frac{\text{var}\,Z_{0}}{1-g^{2}}+2K_{\infty}(1,2)\frac{\text{var}\,Z_{0}}{1-g^{2}}+K_{\infty}(2,2)\left(\frac{\text{var}\,Z_{0}}{1-g^{2}}+\frac{\text{var}\,V_{0}}{1-\rho^{2}}\right)\\\
&+K_{\infty}(3,3)\left(\tau-(1-g)^{-1}\mathbb{E}Z_{0}-(1-\rho)^{-1}\mathbb{E}V_{0}\right)^{2}\\\
&+\frac{\alpha}{1-\alpha}\left[K_{\infty}(1,1)\text{var}\,Z_{0}+2K_{\infty}(1,2)\text{var}\,Z_{0}+K_{\infty}(2,2)(\text{var}\,Z_{0}+\text{var}\,V_{0})\right]\end{split}$
(81)

We turn next to the evaluation of
$\mathbb{E}\underaccent{\tilde}{v}_{\infty}(\underaccent{\tilde}{\chi}_{0})$.
We write
$\underaccent{\tilde}{\chi}_{n}^{T}=(\underaccent{\tilde}{\chi}_{n}(1),\cdots,\underaccent{\tilde}{\chi}_{n}(5))$
and
$\underaccent{\tilde}{\xi}_{n}^{T}=(\underaccent{\tilde}{\xi}_{n}(1),\cdots,\underaccent{\tilde}{\xi}_{n}(5))$.
With this notation in hand, we can write

$\mathbb{E}\underaccent{\tilde}{v}_{\infty}(\underaccent{\tilde}{\chi}_{0})=\sum_{i,j=1}^{5}\mathcal{K}_{\infty}(i,j)[\mathbb{E}\underaccent{\tilde}{\chi}_{0}(i)\underaccent{\tilde}{\chi}_{0}(j)+C(i,j)],$
(82)

where $C(i,j)$ is the $(i,j)$’th entry of the covariance matrix $C$. As for
$\mathbb{E}\underaccent{\tilde}{\chi}_{0}(i)\underaccent{\tilde}{\chi}_{0}(j)$,
note that
$\mathbb{E}\underaccent{\tilde}{\chi}_{0}(i)\underaccent{\tilde}{\chi}_{0}(5)=0$
for $1\leq i\leq 4$, and

$\mathbb{E}\underaccent{\tilde}{\chi}_{0}(5)^{2}=\left(\tau-\frac{\mathbb{E}Z_{0}}{1-g}-\frac{\mathbb{E}V_{0}}{1-\rho}\right)^{2}.$
(83)

We again note that $X_{0}=W_{0}+\Delta_{0}$, so that
$\mathbb{E}\underaccent{\tilde}{\chi}_{0}(1)^{2}=\text{var}\,\tilde{W}_{0}=(1-g^{2})^{-1}\text{var}\,Z_{0}$,
$\mathbb{E}\underaccent{\tilde}{\chi}_{0}(4)^{2}=\text{var}\,\tilde{X}_{0}=(1-g^{2})^{-1}\text{var}\,Z_{0}+(1-\rho^{2})^{-1}\text{var}\,V_{0}$
and
$\mathbb{E}\underaccent{\tilde}{\chi}_{0}(i)\underaccent{\tilde}{\chi}_{0}(4)=\mathbb{E}\underaccent{\tilde}{\chi}_{0}(i)(W_{0}+\Delta_{0})=\mathbb{E}\underaccent{\tilde}{\chi}_{0}(i)W_{0}=\mathbb{E}\underaccent{\tilde}{\chi}_{0}(i)\underaccent{\tilde}{\chi}_{0}(1)$
for $1\leq i\leq 3$. Also, since $(\tilde{F}_{j+2|j}\mathrel{\mathop{\mathchar
58\relax}}j\leq 0)$ is a first order autoregressive process,
$\mathbb{E}\underaccent{\tilde}{\chi}_{0}^{2}(3)=\text{var}\,\tilde{F}_{2|0}=C(3,3)/(1-g^{2})$.
To compute a closed form for the remaining entries of
$(\mathbb{E}\underaccent{\tilde}{\chi}_{0}(i)\underaccent{\tilde}{\chi}_{0}(j)\mathrel{\mathop{\mathchar
58\relax}}1\leq i,j\leq 5)$, we note that

$\underaccent{\tilde}{\chi}_{0}\stackrel{{\scriptstyle\mathcal{D}}}{{=}}\mathcal{A}\underaccent{\tilde}{\chi}_{0}+\underaccent{\tilde}{\xi}_{1}.$
(84)

For example, we have
$\underaccent{\tilde}{\chi}_{0}(2)\stackrel{{\scriptstyle\mathcal{D}}}{{=}}\underaccent{\tilde}{\chi}_{0}(3)+\underaccent{\tilde}{\xi}_{0}(2)$,
so that

$\mathbb{E}\underaccent{\tilde}{\chi}_{0}(2)^{2}=\mathbb{E}\underaccent{\tilde}{\chi}_{0}(3)^{2}+C(2,2)=\frac{C(3,3)}{1-g^{2}}+C(2,2).$
(85)

Similarly, we find that

$\begin{split}\mathbb{E}\underaccent{\tilde}{\chi}_{0}(1)\underaccent{\tilde}{\chi}_{0}(2)&=\frac{gC(3,3)}{1-g^{2}}+C(2,3)+C(1,2),\\\
\mathbb{E}\underaccent{\tilde}{\chi}_{0}(1)\underaccent{\tilde}{\chi}_{0}(3)&=\frac{g^{2}C(3,3)}{1-g^{2}}+C(2,3)+C(1,3),\\\
\mathbb{E}\underaccent{\tilde}{\chi}_{0}(2)\underaccent{\tilde}{\chi}_{0}(3)&=\frac{gC(3,3)}{1-g^{2}}+C(2,3).\end{split}$
(86)

As an alternative to using the closed forms 81 and 82 to compute
$\mathbb{E}v_{\infty}(\tilde{W}_{0},\tilde{X}_{0},\tau-\tau_{0})$ and
$\mathbb{E}\underaccent{\tilde}{v}_{\infty}(\underaccent{\tilde}{\chi}_{0})$,
an iterative approach can be used to compute the covariance matrices; this may
be preferable for some applications. Given the recursion
$\chi_{n+1}=A\chi_{n}+\xi_{n+1}$,

$\chi_{n}\stackrel{{\scriptstyle\mathcal{D}}}{{=}}\sum_{k=0}^{\infty}A^{k}\xi_{k}.$
(87)

It follows that the covariance matrix for $\chi_{n}$ satisfies

$\Lambda=A\Lambda A^{T}+\Sigma_{\xi},$ (88)

subject to $\Lambda_{0}=0$ [20].

Implementations for both the closed-form and the iterative approaches are
provided in the case of our energy control system example in the online
repository for this paper [21]. Figure 6 shows the percentage reduction in
cost from using dynamic forecasts, namely

$D\triangleq\frac{\mathbb{E}v_{\infty}\left(\tilde{W}_{0},\tilde{X}_{0},\tau-\tau_{0}\right)-\mathbb{E}\underaccent{\tilde}{v}_{\infty}(\underaccent{\tilde}{\chi}_{0})}{\mathbb{E}v_{\infty}\left(\tilde{W}_{0},\tilde{X}_{0},\tau-\tau_{0}\right)}*100.$
(89)

Contour plots are used to explore $D$’s dependence as a function of selected
pairs of parameters for the system. In the top right plot of Figure 6, the
improvement increases as $\gamma$ grows closer to 1. As $\gamma$ grows,
previous terms in the $\epsilon_{n}(k)$ sequence play a larger role so the
value of forecasts increases. On the other hand, as $g$ grows closer to 1, the
dependence across time of the $W_{n}$ sequence grows, and so the value of
forecasts decreases. On the top right plot, the value of forecasts increases
as the dependence of $X_{n+1}$ on $X_{n}$ grows (with $\rho$). In the bottom
left plot, the value of forecasts increases with the noise in the weather
sequence $W_{n}$ (controlled by $\sigma^{2}$) but decreases with the noise in
the building temperature sequence $X_{n}$ (controlled by $\sigma_{V}^{2}$).
Finally, the bottom right plot shows a symmetry in the improvement with
respect to the control setpoint $\tau$ around $\tau_{0}$. We note that
$\tau_{0}=\mathbb{E}X_{0}$, the building temperature in the absence of
control. The value of forecasts decreases when the target setpoint is farther
from $\tau_{0}$: as the weight of the action in the value function grows, the
relative value of forecasts weakens.

Figure 6: Relative improvement $D$ (%) in 89 from using dynamic forecasts
versus no forecasts for the energy system control example of Section 7.
Contour plots are shown for $D$ as a function of pairs of system parameters.
In the bottom right plot, a dashed line is drawn at
$\tau_{0}=(1-g)^{-1}\mathbb{E}Z_{0}+(1-\rho)^{-1}\mathbb{E}V_{0}$. Unless
otherwise specified, values used for parameters are $\mathbb{E}W_{0}=80F$,
$\mathbb{E}V_{0}=2F$, $\mathbb{E}W_{0}=80F$, $\gamma=.95$, $\alpha=.9$,
$g=.6$, $\rho=.3$, $\tau=74F$, $\sigma^{2}=\sigma_{V}^{2}=1$.

##  8 Conclusion

In this work we introduced the first principled and mathematically consistent
framework for the incorporation of forecasts into MDPs in the setting of state
space models with linear dynamics, using no ad hoc elements to add forecast
information into the MDP setting. In this framework, we discussed the
different ways in which forecast information can be incorporated (static,
dynamic, static and dynamic together). Through an illustrative energy system
control example, we provided a numerical comparison of the optimal value
functions for the setting with no forecasts to the setting with dynamic
forecasts.

The introduction of this framework opens the door to several theoretical and
applied research questions, e.g. on how the quality of forecasts affects
control methods in different disciplines and in different applications.
Potential theoretical research directions include extensions to periodic
Markov chains (e.g. to model time-of-day effects), non-stationary Markov
chains, forecast updates that are not synchronized with decisions epochs, and
Markov chains with nonlinear dynamics.

JAC and PWG conceived of the study and wrote the manuscript.

The authors declare that they have no competing interests.

Funding for this research was supported by a State Grid Graduate Student
Fellowship through the Stanford Bits & Watts initiative and by Total SE.

The authors thank the members of the Stanford Cooler project, in particular
Professor Sally Benson and Rui Yan, for valuable insights and discussions, as
well as the two anonymous referees whose comments helped us to significantly
improve our work.

## References

  * [1] Bauer P, Thorpe A, Brunet G. 2015 The quiet revolution of numerical weather prediction.  _Nature_ 525, 47–55.
  * [2] Sweeney C, Bessa RJ, Browell J, Pinson P. 2020 The future of forecasting for renewable energy.  _Wiley Interdisciplinary Reviews: Energy and Environment_ 9, e365.
  * [3] Shaner MR, Davis SJ, Lewis NS, Caldeira K. 2018 Geophysical constraints on the reliability of solar and wind power in the United States.  _Energy & Environmental Science_ 11, 914–925.
  * [4] Ziegler MS, Mueller JM, Pereira GD, Song J, Ferrara M, Chiang YM, Trancik JE. 2019 Storage requirements and costs of shaping renewable energy toward grid decarbonization.  _Joule_ 3, 2134–2153.
  * [5] Albertus P, Manser JS, Litzelman S. 2020 Long-duration electricity storage applications, economics, and technologies.  _Joule_ 4, 21–32.
  * [6] Hausman WH. 1969 Sequential decision problems: A model to exploit existing forecasters.  _Management Science_ 16, B–93.
  * [7] Graves SC, Meal HC, Dasu S, Qui Y. 1986 Two-stage production planning in a dynamic environment.  In _Multi-stage Production Planning and Inventory Control_ , pp. 9–43. Springer.
  * [8] Heath DC, Jackson PL. 1994 Modeling the evolution of demand forecasts ith application to safety stock analysis in production/distribution systems.  _IIE Transactions_ 26, 17–30.
  * [9] Altug MS, Muharremoglu A. 2011 Inventory management with advance supply information.  _International Journal of Production Economics_ 129, 302–313.
  * [10] Zhao T, Zhao J, Yang D, Wang H. 2013 Generalized martingale model of the uncertainty evolution of streamflow forecasts.  _Advances in water resources_ 57, 41–51.
  * [11] Bitar EY, Rajagopal R, Khargonekar PP, Poolla K, Varaiya P. 2012 Bringing wind energy to market.  _IEEE Transactions on Power Systems_ 27, 1225–1235.
  * [12] Nair J, Adlakha S, Wierman A. 2014 Energy procurement strategies in the presence of intermittent sources.  In _The 2014 ACM International Conference on Measurement and Modeling of Computer Systems_ , pp. 85–97.
  * [13] Callier FM, Desoer CA. 2012 _Linear system theory_.  Springer Science & Business Media.
  * [14] Qin SJ, Badgwell TA. 2003 A survey of industrial model predictive control technology.  _Control Engineering Practice_ 11, 733–764.
  * [15] Mayne DQ. 2014 Model predictive control: Recent developments and future promise.  _Automatica_ 50, 2967–2986.
  * [16] Mesbah A. 2016 Stochastic model predictive control: An overview and perspectives for future research.  _IEEE Control Systems Magazine_ 36, 30–44.
  * [17] Bertsekas DP, Shreve S. 2004 _Stochastic Optimal Control: The Discrete-time Case_.  Athena Scientific Belmont, MA.
  * [18] Bertsekas DP. 2012 _Dynamic Programming and Optimal Control_ , volume 2.  Athena Scientific Belmont, MA.
  * [19] Bertsekas DP. 2005 _Dynamic Programming and Optimal Control_ , volume 1.  Athena Scientific Belmont, MA.
  * [20] de Jong R.  Lecture 6: vector autoregression.  Accessed January 19, 2021. https://www.asc.ohio-state.edu/de-jong.8/note6.pdf
  * [21] de Chalendar J. 2021.  On incorporating forecasts into linear state space model markov decision processes (supplemental code).  URL to be provided on publication.