# Forecasting with Neuro-Dynamic Programming

Pedro Afonso Fernandes111ORCID: https://orcid.org/0000-0001-5762-5157.
Correspondence: Universidade Católica Portuguesa, Católica Lisbon School of
Business & Economics, Palma de Cima, Building 5, 4th floor, Room 5430,
1649-023 Lisboa, Portugal. Email<EMAIL_ADDRESS>URL: https://paf.com.pt.

Universidade Católica Portuguesa
Católica Lisbon School of Business & Economics
Católica Lisbon Research Unit in Business & Economics (CUBE)
Católica Lisbon Forecasting Lab (NECEP)
Portugal

###### Abstract

Economic forecasting is concerned with the estimation of some variable like
gross domestic product (GDP) in the next period given a set of variables that
describes the current situation or state of the economy, including industrial
production, retail trade turnover or economic confidence. Neuro-dynamic
programming (NDP) provides tools to deal with forecasting and other sequential
problems with such high-dimensional states spaces. Whereas conventional
forecasting methods penalises the difference (or loss) between predicted and
actual outcomes, NDP favours the difference between temporally successive
predictions, following an interactive and trial-and-error approach. Past data
provides a guidance to train the models, but in a different way from ordinary
least squares (OLS) and other supervised learning methods, signalling the
adjustment costs between sequential states. We found that it is possible to
train a GDP forecasting model with data concerned with other countries that
performs better than models trained with past data from the tested country
(Portugal). In addition, we found that non-linear architectures to approximate
the value function of a sequential problem, namely, neural networks can
perform better than a simple linear architecture, lowering the out-of-sample
mean absolute forecast error (MAE) by 32% from an OLS model.

_Keywords:_ time series; forecasting; neuro-dynamic programming; reinforcement
learning; temporal differences.

JEL codes: C22; C45; C52; C53.

##  1 Introduction

Economic forecasting is concerned with the estimation of some variable of
interest like gross domestic product (GDP) in the next period given a set of
variables that describes the current situation or state of the economy. This
set can include known values of the variable of interest or other covariates
that describe the environment faced by economic agents. Forecasting is then
finding the optimal value of the parameters of some (typically) linear model
by minimising a loss function that measures the difference between the
observed and the predicted values of the variable of interest.

Thus, forecasting is concerned with sequential problems where some cost is
minimised given the current and future states of the economy. _Dynamic
programming_ (DP) is concerned with this kind of multistage decision processes
that follow the _principle of optimality_ , that is, whatever the initial
state and initial decision are, the remaining decisions must constitute an
optimal policy with regard to the state resulting form the first decision
[Bellman and Dreyfus, 1962]. Here, the _state_ is a finite-dimensional object
that, from the point of view of current and future costs, completely
summarises the current situation faced by economic agents or decision makers
[Ljungqvist and Sargent, 2018, section 1.4].

States must be kept in low dimension in order to solve optimally the
functional (Bellman) equation of a dynamic problem. Nevertheless, DP has a
wide range of applications, namely, in economics following the dynamic
recursive theories formulated by Stockey and Lucas [1989] and Ljungqvist and
Sargent [2018]. A good example is the savings problem where a representative
household maximises the expected utility of the current and future consumption
subject to a sequence of budget constraints that relates labour income,
financial wealth and consumption. The state space of this model could be kept
in a low discrete dimension, but complex models may require an approximation
to the return function that expresses the optimal long run value of some state
using, namely, quadratic forms, resulting in the so-called _optimal linear
regulator problem_. Even these models require point estimates of their
parameters [Mannor et al., 2007], which constitute a practical limitation of
DP, namely, in the field of economics.

Originally formulated by Bertsekas and Tsitsiklis [1996], _neuro-dynamic
programming_ (NDP), also known as _approximate_ DP or _reinforcement learning_
(in machine learning literature), provides a set of methods and tools to deal
with high-dimensional state spaces and challenging DP problems. It can deal
also with problems where a mathematical model is unavailable or hard to
construct using model-free methods. Thus, NDP is primarily concerned with
approximating the return function of a dynamic problem using linear or
nonlinear functional forms. Within this framework, optimality is relaxed in
order to solve complex and computationally intensive DP problems, where
regular approaches are not applicable or are too costly [Bertsekas, 2011].

Simultaneously, NDP makes use of ideas from artificial intelligence involving
simulation-based algorithms and neural networks [van Roy et al., 1997].
Reinforcement learning (RL) algorithms learn what to do so as to maximise some
reward (or minimise some cost), having in mind a long-term objective
[Szepesvári, 2010]. The learner is not told which actions to take, but instead
must discover which actions yield the most reward (or least cost) by trying
them in an interactive and trial-and-error process [Sutton and Barto, 2018].

The objective of this paper is to test the viability of using NDP/RL methods
and tools to approximate the GDP level with the state of the economy described
by several variables, possibly using big data. These variables include readily
available data that can be used to forecast GDP in the current quarter like
industrial production, retail sales, exports or economic confidence. Past data
concerned with GDP provides a guidance to train the models, but in a different
way from ordinary least squares and other supervised learning methods,
signalling the adjustment costs between sequential states. Our models were
trained using quarterly data from European countries and tested in Portugal.
We found that a neural network architecture with tensor product can perform
quite well, namely, during COVID-19 times.

The paper is organised as follow: firstly, we provide a review of selected,
relevant literature in section 2; then, in section 3, we describe succinctly
two approaches to train NDP models using linear and nonlinear architectures,
as well as the dynamic model here adopted to forecast GDP; a brief description
of the data set and its treatment is made in section 4; section 5 presents the
main findings, followed by a final conclusion in section 6.

##  2 Literature

This paper covers a topic with contributions from the fields of forecasting,
economics, dynamic programming and reinforcement learning. Here we track only
the most important references.

Dynamic programming has a wide range of applications in economics including
the savings problem, economic growth, job search, business cycles, olipoly
equilibrium or recursive contracts [Ljungqvist and Sargent, 2018].
Particularly important for the present research is the problem of a
competitive firm that maximises the inter-temporal value of its production
with an adjustment cost of the rate of output: in order to obtain a greater
return (or value) in the future, the firm must invest (or set aside) a part of
its current production incurring a quadratic cost, following a scheme like the
one originally proposed by Lucas and Prescott [1971]. The trade-off between
immediate costs and future returns gives the firm the incentive to forecast
the output market price as far as the investment (or change of output)
decision is concerned.

This problem can be solved optimally by assuming that the firm’s return
function is quadratic on the state variables, namely, the current level of
output. However, we have to know in advance the values of the model’s
parameters to get that kind of solution. Here, our goal is to estimate those
parameters directly from data following the temporal difference (TD) method
proposed by Sutton [1988]. Whereas conventional forecasting methods penalises
the difference (or loss) between predicted and actual outcomes, that method is
guided by the difference between temporally successive predictions.

Our goal is also concerned with using other architectures than linear-
quadratic ones for the return function, namely, neural networks. The
approximation of Bellman’s value and policy functions with neural networks was
originally proposed by Bertsekas and Tsitsiklis [1996] with lately
developments described by Bertsekas [2011, 2020]. Two broad approaches are
available in neuro-dynamic programming (NDP): the direct estimation of the
parameters using temporal differences or its indirect estimation using a least
squares scheme on a projected Bellman equation.

NDP applications include parking, maintenance and repair [Bertsekas and
Tsitsiklis, 1996], retailer inventory management [van Roy et al., 1997] and
dynamic catalog mailing policies [Mannor et al., 2007]. In addition, NDP is
being applied with great success in machines that play masterly complex games
like backgammon [Tesauro, 1995], Go [Silver et al., 2016] or chess [Thrun,
1995, McIlroy-Young et al., 2020]. Most advanced algorithms attained a
superhuman performance by _tabula rasa_ learning from games of self-play
[Silver et al., 2017]. Economic interpretations of algorithms that play board
games can be found in Igami [2020].

The approximation of functions with neural networks is a recent research topic
in economics with a seminal contribution in Duffy and McNelis [2001] who
approximated the conditional expectation function in the Euler equation of a
stochastic growth model. In fact, heterogeneous-agents models and other high-
dimensional problems cannot be solved using classic dynamic programming due to
the ”curse of dimensionality” [Bellman and Dreyfus, 1962]. In this scope,
Maliar et al. [2021] introduce a unified deep learning method that solves
complex dynamic economic models by casting them into nonlinear regression
equations. They use neural networks to perform model reduction and to handle
multicollinearity. Kahou et al. [2021] propose a new method for solving high-
dimensional dynamic programming problems and recursive competitive equilibria
with a large number of heterogeneous agents using deep learning and exploring
symmetry. Kase et al. [2022] take advantage of the scalability of neural
networks to estimate nonlinear heterogeneous-agents models with likelihood
methods. Azinovic et al. [2022] uses neural networks to compute approximate
functional rational expectations equilibria of economic models featuring a
significant amount of heterogeneity, uncertainty and binding restrictions,
namely, a large-scale overlapping generations (OLG) model.

In a broad sense, this paper is about the application of reinforcement
learning (RL) methods to economic forecasting. RL is founded on learning from
interaction with the environment [Sutton and Barto, 2018]. Guided by an
immediate reward (or cost), an agent may find an optimal long-run policy by
exploit the most advantageous actions and explore new, potentially better
actions. RL develops the optimal control (or DP) framework by introducing
computer science techniques like deep neural networks. Complete surveys of RL
applications in economics and finance can be found in Charpentier et al.
[2021] and Meng and Khushi [2019].

The use of NDP/RL to forecast GDP is a relatively new topic in literature, we
few applications such the ensembling approach of Li et al. [2022] that combine
three deep neural networks to forecast the regional GDP of China.

##  3 Framework

###  3.1 Dynamic programming

Dynamic programming is concerned with sequential problems where the state of a
discrete-time dynamic system evolves according to given transition
probabilities that depend on some control variable $u$ [Bertsekas and
Tsitsiklis, 1996, chapter 1]. Given an initial state $i$, we choose the
control $u$ to minimise either the immediate cost $g(i,u,j)$ associated with
the transition from $i$ to the next state $j$ or the optimal cost-to-go of
$j$, denoted by $J^{*}(j)$, which is the expected cost of all subsequent
periods starting from $j$. The objective of DP is to calculate the _optimal
cost-to-go function_ $J^{*}$ by iterating on the following Bellman equation:

$J^{*}(i)=\min_{u}E\left[g(i,u,j)+\alpha J^{*}(j)|i,u\right],$ (1)

for all $i$, where $\alpha\in(0,1)$ is a discount factor. The time-invariant
rule $\mu(i)$ that maps each state $i$ to the optimal control $u$ that attains
the minimum in equation (1) is called a _policy_ or _feedback control policy_.
It can be computed either simultaneously with the optimal cost-to-go $J^{*}$
or in real time by minimising the right-and-side of the Bellman equation.

###  3.2 Approximate dynamic programming

In many application, the number of states and controls becomes prohibitively
large in terms of computing time. Neuro-dynamic or approximate dynamic
programming deals with this ”curse of dimensionality” [Bellman and Dreyfus,
1962] by approximating the optimal cost-to-go function $J^{*}$ through a
neural network or linear architecture $\tilde{J}(i,r)$, where $r$ is a
parameter/weight vector [Bertsekas and Tsitsiklis, 1996, Bertsekas, 2011].
Once $r$ is determined, it yields a sub-optimal control at any state $i$
through the one-step look-ahead minimisation:

$\tilde{\mu}(i)=\arg\min_{u}E\left[g(i,u,j)+\alpha\tilde{J}(j,r)|i,u\right].$
(2)

The function $\tilde{J}$ is called the _scoring function_ or _approximate
cost-to-go function_ , and the value $\tilde{J}(j,r)$ is called the _score_ or
_approximate cost-to-go_ of state $j$ [Bertsekas and Tsitsiklis, 1996]. It is
a _compact representation_ in the sense that it reduces the description of $j$
to a few relevant features whose weights are the parameters $r$. For example,
in computer chess, the state is the current board position described by
features like the midgame/endgame material point values, material imbalances,
mobility and trapped pieces, pawn structure, king safety, outposts, bishop
pair, and other evaluation patterns [Bertsekas, 2011, Silver et al., 2017].

###  3.3 Neural network

The scoring function $\tilde{J}$ can be approximated by a nonlinear
architecture, namely, a _feedforward neural network_ (FFNN). Also know as
_deep neural network_ , _artificial neural network_ or _multilayer perceptron_
, it is a simple model where several linear combinations of inputs are passed
through nonlinear activation functions called _nodes_ [Taddy, 2019, chapter
10]. A set of nodes is called a _layer_ , and the output of ones layer’s node
becomes the input of the next layer in multilayer architectures. Finally, the
output of the last layer is combined linearly to produce a cost-to-go.

For a state $i$ represented by a _feature vector_
$x(i)=(x_{1}(i),\dots,x_{q}(i))^{\prime}$, the first layer of a FFNN is
composed by $s$ nodes $h_{k}(i)$ such that [Bertsekas and Tsitsiklis, 1996,
chapter 3]:

$h_{k}(i)=\sigma\left(\sum_{l=1}^{q}{r_{kl}}x_{l}(i)\right),k=1,\dots,s,$ (3)

where $\sigma(\cdot)$ is a nonlinear _activation function_ specified in
advance. Early implementations of FFNN typically favoured the logistic
function:

$\sigma(\xi)=\frac{1}{1+e^{-\xi}}=\frac{e^{\xi}}{e^{\xi}+1}.$ (4)

Nowadays, the most common activation function is the ReLU (rectified linear
unit):

$\sigma(\xi)=\textrm{max}(0,\xi),$ (5)

because it can be computed and stored more efficiently than the logistic
function and other options. The nonlinear nature of the activation function is
fundamental to capture complex nonlinearities and interaction effects [James
et al., 2021, chapter 10].

The nodes given by equation (3) can be used as input of a second layer whose
output can be passed to a third layer, and so on. Following Bertsekas and
Tsitsiklis [1996, chapter 3], we adopt a FFNN with a single layer whose final
output is given by

$\tilde{J}(i,r)=\sum_{k=1}^{s}{r_{k}h_{k}(i)}=\sum_{k=1}^{s}{r_{k}\sigma\left(\sum_{l=1}^{q}{r_{kl}}x_{l}(i)\right)},$
(6)

where the parameter vector $r$ is composed by the weights $r_{k}$ and $r_{kl}$
for $k=1,\dots,s$ and $l=1,\dots,q$. These coefficients can be estimated from
training data using _temporal difference learning_ (TD). Introduced by Sutton
[1988], TD deals with multi-step prediction problems where a future outcome is
predicted by updating the _weights_ of the model from the changes or _errors_
in successive predictions rather than from the overall error between
predictions and the final outcome. Thus, our simple FFNN can be trained using
the algorithm 1 [Szepesvári, 2010, section 2.2] [Bertsekas, 2011, section
6.2]:

Algorithm 1 TD(0) with nonlinear cost-to-go approximation (neural network)

1:$\delta\leftarrow\tilde{J}(i,r)-\alpha\tilde{J}(j,r)-g(i,u,j)$

2:$r_{k}\leftarrow r_{k}-\gamma\delta h_{k}(i)$

3:$r_{kl}\leftarrow r_{kl}-\gamma\delta r_{k}\nabla_{r_{kl}}h_{k}(i)$

4:return $r$

where $\gamma$ is a small non-negative number, called _step-size_ , that
decreases with time, and the gradient $\nabla_{r_{kl}}h_{k}(i)$ is the partial
derivative of the node $k$ with respect to the weight $r_{kl}$. Using the ReLU
function (5), this derivative is simply $x_{l}(i)$ if the weighted sum of
features $\sum_{l=1}^{q}{r_{kl}}x_{l}(i)$ is positive and zero otherwise. The
features $x_{l}(i)$ can be constructed from states individual components or
attributes using the tensor product [Szepesvári, 2010, section 2.2].

###  3.4 Linear architecture

Alternatively, the approximate cost-to-go function $\tilde{J}$ may adopt the
linear form

$\tilde{J}(i,r)=\sum_{l=1}^{q}{r_{l}x_{l}(i)},$ (7)

where $r=(r_{1},\dots,r_{q})^{\prime}$ and the features $x_{l}(i)$ are _basis
functions_ because they form a linear basis of the set of approximate
functions [Sutton and Barto, 2018, section 9.4]. The weights $r_{l}$ can be
estimated with temporal differences using the algorithm 2.

Algorithm 2 TD(0) with linear cost-to-go approximation

1:$\delta\leftarrow\tilde{J}(i,r)-\alpha\tilde{J}(j,r)-g(i,u,j)$

2:$r_{l}\leftarrow r_{l}-\gamma\delta x_{l}(i)$

3:return $r$

###  3.5 Dynamic model

In our simple model, output (GDP) is adjusted with a quadratic cost. The
objective of a social planner is to minimise this immediate cost, as well as
the future costs of adjustment. Thus, the cost-to-go function $J$ reflects the
adjustment path of the economy as a function of several state variables that
describe the economic environment. The Bellman equation of the model is

$J(i)=\min_{u}E\left[u^{2}+\alpha J(j)\right],$ (8)

where $i$ is the state of the economy in the current quarter, $u$ is the
quarter-over-quarter change of output and $j$ is the state in the next
quarter. This model is basically a simplified version of the competitive firm
model of Ljungqvist and Sargent [2018, section 7.2] adapted from Lucas and
Prescott [1971].

##  4 Data

The cost-to-go function $J$ in equation (8) was approximated using the two
architectures and algorithms described in sections 3.3 and 3.4. NDP/RL
algorithms can converge slowly, thus requiring long batches with several
observations to produce accurate approximations. Economic time series are
typically too short for that purpose. Anyway, it is possible to consider
multiple series concerned with several countries. Thus, we trained our models
with a panel of 26 countries from the European Union, reserving for testing
the data concerned with Portugal. Data covered the period from the first
quarter of 2000 to the fourth quarter of 2023 with 95 transitions between
adjacent quarters for each country, and a total of 2470 records in the
training data set.

Besides GDP, data covered the state variables indicated in table 1 that
describe the economic environment in each country, covering production and
turnover (industry, construction and services), international trade (exports
and imports) and the confidence of economic agents measured by the Economic
Sentiment Indicator (ESI). These variables are readily available and can be
used to assess the current situation of the economy right before the
dissemination of official macroeconomic aggregates from the national quarterly
accounts (NQA), supporting a _nowcasting_ exercise [Baffigi et al., 2004,
Giannone et al., 2008].

Table 1: List of state variables

(@,2.5cm,0.5cm)lcc—c—cccccc— Variable & Source Transformation
Industrial production index Eurostat $(x-min)/(max-min)$
Production in construction Eurostat $(x-min)/(max-min)$
Retail trade turnover Eurostat $(x-min)/(max-min)$
Exports of goods Eurostat $(x-min)/(max-min)$
Imports of goods Eurostat $(x-min)/(max-min)$
Economic Sentiment Indicator (ESI) EC $(x-min)/(max-min)$

Each observation was regularised by subtracting the minimum value and dividing
by the range, that is, the difference between the maximum and minimum values.
This operation was performed for each indicator individually and for each
country separately in order to assure the comparability between records.
Regularisation is a critical procedure to estimate, namely, the neural network
weights using the temporal difference algorithm 1.

Data concerned with Portugal covers also the period 2000Q1-2023Q4. The
observations up to 2014 (first 15 years) were used to train a benchmark model,
that is, an ordinary least squares (OLS) regression of GDP on the variables
listed in table 1. Then, the predictive performance of the models was tested
using the data for Portugal from the first quarter of 2015 to the fourth
quarter of 2023.

##  5 Findings

Table 2 presents the out-of-sample (OOS) mean absolute error (MAE) and root
mean squared error (RMSE) for the models previously described, that is, a
linear regression (benchmark), a neural network and a linear architecture,
both estimated with temporal differences. The response was the regularised
Portuguese GDP and the OOS errors were calculated for the testing period
between the first quarter of 2015 and the fourth quarter of 2023.

Table 2: OOS errors by model (regularised GDP level, Portugal, 2015Q1-2023Q4) Model | MAE | RMSE
---|---|---
Linear regression (OLS) | 0.0661 | 0.0832
TD(0) with neural network | 0.0453 | 0.0597
TD(0) with linear architecture | 0.1070 | 0.1281

Firstly, we found that an OLS model of GDP level on industrial production,
construction output, retail trade, international trade and economic sentiment
can perform quite well. In fact, this model provides a MAE of only 6.6% of the
range between the maximum and minimum levels of GDP for the testing period.
The RMSE is about 8.3% of that range.

Secondly, a neural network trained with data for the European Union countries
(excluding Portugal) can perform even better, lowering that MAE by 32% to 4.5%
and the RMSE by 28% to 5.9%. In fact, the non-linear nature of this
architecture might capture well the dramatic drop in GDP occurred during the
”Great Lockdown” of the spring of 2020, as suggested by figure 1, lowering the
OOS errors for the testing period. In addition, figure 2 reveals that TD with
neural network was quite effective during the COVID-19 times (since 2020) in
mitigating the cumulative absolute OOS error.

Figure 1: Forecasts of the regularised GDP level by different models and
testing data (Portugal, 2015Q1-2023Q4) Figure 2: Cumulative absolute OOS error
of the regularised GDP level by model (Portugal, 2015Q1-2023Q4)

Besides, the neural network performs better than a linear architecture
estimated with temporal differences too. The last model is the worst with a
MAE of 10.7% of the range between the maximum and minimum levels of GDP, and a
RMSE of 12.8% of that range. Thus, a simple least squares model estimated with
a restricted dataset (for Portugal) can perform better than a linear model
estimated with TD over a full dataset (for the EU countries except Portugal),
but so well than a neural network trained with the last dataset.

##  6 Conclusion

NDP/RL algorithms proved to achieve remarkable results using generic or even
simulated data, not directly connected with the predicted object, namely in
the field of board games. Here, we found that it is possible to train a model
with data concerned with other countries that performs quite well, even better
than models trained with past data from the tested country. In fact, time
series could be too short to train effectively forecasting models, and some
advantage may be obtained by using panel data from other countries with
algorithms that learn by trial-and-error, guided by the change of output as an
adjustment cost.

In addition, we found that non-linear architectures to approximate the value
function of a sequential problem, namely, neural networks with tensor product
can perform better than a simple linear architecture as far as the prediction
of GDP is concerned, despite the good achievement obtained by a linear model
estimated with ordinary least squares using past data for the region of
interest.

## Acknowledgments

This work was supported by Fundação para a Ciência e Tecnologia (FCT), Lisbon,
Portugal under a doctorate auxiliary researcher grant with the reference CUBE-
PhD-CEEC/1 from Católica Lisbon Research Unit in Business & Economics
(UID/GES/00407/2020).

## Abbreviations

The following abbreviations are used in this paper:

DP | Dynamic Programming
---|---
EC | European Commission
ESI | Economic Sentiment Indicator
FCT | Portuguese Foundation for Science and Technology
FFNN | Feedforward Neural Network
GDP | Gross Domestic Product
MAE | Mean Absolute Error
NDP | Neuro-Dynamic Programming
NQA | National Quarterly Accounts
OLG | Overlapping Generations
OLS | Ordinary Least Squares
OOS | Out-Of-Sample
ReLU | Rectified Linear Unit
RL | Reinforcement Learning
RMSE | Root Mean Squared Error
TD | Temporal Difference

## References

  * Azinovic et al. [2022] M. Azinovic, L. Gaegauf, and S. Scheidegger.  Deep Equilibrium Nets.  _International Economic Review_ , 63(4):1471–1525, 11 2022.  ISSN 0020-6598.  doi: 10.1111/iere.12575.
  * Baffigi et al. [2004] A. Baffigi, R. Golinelli, and G. Parigi.  Bridge models to forecast the euro area GDP.  _International Journal of Forecasting_ , 20(3):447–460, 7 2004.  ISSN 01692070.  doi: 10.1016/S0169-2070(03)00067-0.
  * Bellman and Dreyfus [1962] R. E. Bellman and S. E. Dreyfus.  _Applied Dynamic Programming_.  Princeton University Press, Princeton, N.J., 1962.  URL https://www.rand.org/pubs/reports/R352.html.
  * Bertsekas [2011] D. P. Bertsekas.  Approximate Dynamic Programming (online chapter).  In _Dynamic Programming and Optimal Control, Volume II_ , chapter 6. Athena Scientific, Belmont, Massachusetts, 3rd edition, 2011.  URL https://www.mit.edu/~dimitrib/dpbook.html.
  * Bertsekas [2020] D. P. Bertsekas.  _Rollout, Policy Iteration, and Distributed Reinforcement Learning_.  Athena Scientific, Belmont, Massachusetts, 2020.
  * Bertsekas and Tsitsiklis [1996] D. P. Bertsekas and J. N. Tsitsiklis.  _Neuro-Dynamic Programming_.  Athena Scientific, Belmont, Massachusetts, 1996.
  * Charpentier et al. [2021] A. Charpentier, R. Élie, and C. Remlinger.  Reinforcement Learning in Economics and Finance.  _Computational Economics_ , 2021.  ISSN 15729974.  doi: 10.1007/s10614-021-10119-4.
  * Duffy and McNelis [2001] J. Duffy and P. D. McNelis.  Approximating and simulating the stochastic growth model: Parameterized expectations, neural networks, and the genetic algorithm.  _Journal of Economic Dynamics and Control_ , 25(9):1273–1303, 9 2001.  ISSN 01651889.  doi: 10.1016/S0165-1889(99)00077-9.
  * Giannone et al. [2008] D. Giannone, L. Reichlin, and D. Small.  Nowcasting: The real-time informational content of macroeconomic data.  _Journal of Monetary Economics_ , 55:665–676, 2008.
  * Igami [2020] M. Igami.  Artificial Intelligence as Structural Estimation: Economic Interpretations of Deep Blue, Bonanza, and AlphaGo.  _The Econometrics Journal_ , 23(3):1–24, 2020\.  doi: 10.1093/ectj/utaa005.
  * James et al. [2021] G. James, D. Witten, T. Hastie, and R. Tibshirani.  _An Introduction to Statistical Learning with Applications in R_.  Springer, 2nd edition, 2021.  URL https://www.statlearning.com/.
  * Kahou et al. [2021] M. E. Kahou, J. Fernández-Villaverde, J. Perla, and A. Sood.  Exploiting Symmetry in High-Dimensional Dynamic Programming.  _NBER Working Paper_ , 28981:1–52, 7 2021.  doi: 10.3386/w28981.
  * Kase et al. [2022] H. Kase, L. Melosi, and M. Rottner.  Estimating Nonlinear Heterogeneous Agents Models with Neural Networks.  _Federal Reserve Bank of Chicago Working Paper_ , 26:1–55, 2022.  doi: 10.21033/wp-2022-26.
  * Li et al. [2022] Q. Li, C. Yu, and G. Yan.  A New Multipredictor Ensemble Decision Framework Based on Deep Reinforcement Learning for Regional GDP Prediction.  _IEEE Access_ , 10:45266–45279, 2022.  ISSN 2169-3536.  doi: 10.1109/ACCESS.2022.3170905.
  * Ljungqvist and Sargent [2018] L. Ljungqvist and T. J. Sargent.  _Recursive Macroeconomic Theory_.  The MIT Press, Cambridge, MA, 4th edition, 2018.
  * Lucas and Prescott [1971] R. E. Lucas and E. C. Prescott.  Investment under Uncertainty.  _Econometrica_ , 39(5):659–681, 1971.
  * Maliar et al. [2021] L. Maliar, S. Maliar, and P. Winant.  Deep learning for solving dynamic economic models.  _Journal of Monetary Economics_ , 122:76–101, 9 2021.  ISSN 03043932.  doi: 10.1016/j.jmoneco.2021.07.004.
  * Mannor et al. [2007] S. Mannor, D. Simester, P. Sun, and J. N. Tsitsiklis.  Bias and Variance Approximation in Value Function Estimates.  _Management Science_ , 53(2):308–322, 2 2007\.  ISSN 0025-1909.  doi: 10.1287/mnsc.1060.0614.
  * McIlroy-Young et al. [2020] R. McIlroy-Young, S. Sen, J. Kleinberg, and A. Anderson.  Aligning Superhuman AI with Human Behavior: Chess as a Model System.  _arXiv preprint_ , 6 2020.  doi: 10.1145/3394486.3403219.
  * Meng and Khushi [2019] T. L. Meng and M. Khushi.  Reinforcement Learning in Financial Markets.  _Data_ , 4(3):110, 7 2019.  ISSN 2306-5729.  doi: 10.3390/data4030110.
  * Silver et al. [2016] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, T. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis.  Mastering the game of Go with deep neural networks and tree search.  _Nature_ , 529(7587):484–489, 1 2016.  ISSN 0028-0836.  doi: 10.1038/nature16961.
  * Silver et al. [2017] D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, T. Lillicrap, K. Simonyan, and D. Hassabis.  Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm.  _arXiv preprint_ , 12 2017.  doi: 10.48550/arXiv.1712.01815.
  * Stockey and Lucas [1989] N. L. Stockey and R. E. Lucas.  _Recursive Methods in Economic Dynamics_.  Harvard University Press, 1989.
  * Sutton [1988] R. S. Sutton.  Learning to predict by the methods of temporal differences.  _Machine Learning_ , 3(1):9–44, 8 1988.  ISSN 0885-6125.  doi: 10.1007/BF00115009.
  * Sutton and Barto [2018] R. S. Sutton and A. G. Barto.  _Reinforcement Learning: An Introduction_.  The MIT Press, Cambridge, Massachusetts and London, England, 2nd edition, 2018.  URL http://incompleteideas.net/book/the-book-2nd.html.
  * Szepesvári [2010] C. Szepesvári.  _Algorithms for Reinforcement Learning_.  Morgan and Claypool Publishers, San Francisco, California, 2010.  URL https://sites.ualberta.ca/~szepesva/rlbook.html.
  * Taddy [2019] M. Taddy.  _Business Data Science: Combining Machine Learning and Economics to Optimize, Automate and Accelerate Business Decisions_.  McGraw-Hill, New York, USA, 2019.
  * Tesauro [1995] G. Tesauro.  Temporal difference learning and TD-Gammon.  _Communications of the ACM_ , 38(3):58–68, 3 1995\.  ISSN 0001-0782.  doi: 10.1145/203330.203343.
  * Thrun [1995] S. Thrun.  Learning to Play the Game of Chess.  In G. Tesauro, D. Touretzky, and T. Leen, editors, _Advances in Neural Information Processing Systems (NIPS) 7_ , Cambridge, MA, 1995. The MIT Press.  URL http://robots.stanford.edu/papers/thrun.nips7.neuro-chess.html.
  * van Roy et al. [1997] B. van Roy, D. P. Bertsekas, Y. Lee, and J. N. Tsitsiklis.  A Neuro-Dynamic Programming Approach to Retail Inventory Management.  In _Proceedings of the 36th Conference on Decision and Control_ , San Diego, California USA, 12 1997.