# Exploratory Control with Tsallis Entropy for Latent Factor Models 111The
authors wish to thank participants on the OMI Machine Learning and
Quantitative Finance Workshop, Research in Options 2022, AMS-EMS-SMF 2022,
IMSI workshop on Advances in Optimal Decision Making under Uncertainty, the
Bachelier World Congress 2022, and the Université Paris 1 Panthéon-Sorbonne
Finance & Modeling Webinar.

Ryan Donnelly<EMAIL_ADDRESS>Sebastian Jaimungal
<EMAIL_ADDRESS>Department of Mathematics, King’s College
London,
Strand, London, WC2R 2LS, United Kingdom  Department of Statistical Sciences,
University of Toronto,
Toronto, Ontario, Canada, M5G 1Z5

###### Abstract

We study optimal control in models with latent factors where the agent
controls the distribution over actions, rather than actions themselves, in
both discrete and continuous time. To encourage exploration of the state
space, we reward exploration with Tsallis Entropy and derive the optimal
distribution over states –- which we prove is $q$-Gaussian distributed with
location characterized through the solution of an FBS$\Delta$E and FBSDE in
discrete and continuous time, respectively. We discuss the relation between
the solutions of the optimal exploration problems and the standard dynamic
optimal control solution. Finally, we develop the optimal policy in a model-
agnostic setting along the lines of soft $Q$-learning. The approach may be
applied in, e.g., developing more robust statistical arbitrage trading
strategies.

###### keywords:

stochastic control, exploratory control, entropy regularization, reinforcement
learning

††journal: TBAt2t2footnotetext: SJ would like to acknowledge support from the
Natural Sciences and Engineering Research Council of Canada (grants
RGPIN-2018-05705 and RGPAS-2018-522715).

##  1 Introduction

Reinforcement Learning (RL) is an important technique in optimization which
has seen many recent developments. At the heart of RL is a trade-off between
acting in a manner which is believed to be optimal and deliberately acting
suboptimally with the purpose of better learning about the environment. This
is often referred to in RL as exploitation vs. exploration, and it allows
algorithms to hone in on an optimal strategy over repeated actions without a
priori specifying a model for the system’s dynamics.

In some situations it is possible to propose a simplistic model which is to be
improved upon through exploration. In this paper, we consider a dynamic
optimization problem where the agent desires to explore the state and control
space by implementing actions which may not be optimal according to the
proposed model. In this situation, repeatedly acting optimally according to
the model will result in repeated observations of the system along state
trajectories that result from the original model. If instead the agent
implements a different action, then she may generate observations in the state
and control space that may be used to propose a more precise model of the
system, yielding improved future optimization.

Deliberately acting suboptimally comes with opportunity costs, therefore,
exploratory actions should be accompanied with incentives. We represent this
incentive through two modifications of a standard dynamic optimal control
problem using an approach similar to Wang et al. (2020) (see also Guo et al.
(2022) and Firoozi and Jaimungal (2022) for extensions in the mean field game
context). First, the agent is allowed to randomize the action taken at any
point in time, specifying the distribution of this random action. Second, the
performance criterion is altered to reward the agent based on a measure of
randomness corresponding to their selected distribution. We take this reward
to be the Tsallis entropy, first introduced in Tsallis (1988) (see also
Tsallis (2011)), of the agent’s action distribution which generalizes the
Shannon entropy. We show that this generalization modifies the optimal
distribution over actions to be $q$-Gaussian distributed rather than Gaussian,
as is the case with Shannon entropy introduced in Wang et al. (2020).

In addition to applications in RL, exploratory control has been applied in
other contexts. For example, in Gao et al. (2022) the randomization of
controls is intended to emulate simulated annealing when optimizing non-convex
functions, rather than used as a method for generating observations of
infrequently visited states. In Tang et al. (2021) the properties of the
associated exploratory Hamilton-Jacobi-Bellman (HJB) equations are studied as
is the rate of convergence of the optimal exploratory control as exploration
goes to zero. Entropy regularization has also been employed in the context of
Markov decision processes, as in Neu et al. (2017) and Geist et al. (2019), or
other areas related to optimization such as stochastic games as in Savas et
al. (2019), Guan et al. (2020), and Hao et al. (2022).

Here, we primarily focus on dynamic control in discrete-time, in contrast to
those above who study the continuous time case. One primary rationale for
working in discrete-time is provide a more precise probabilistic description
of the controlled dynamics with randomized actions. Moreover, we incorporate
an unobserved latent factor into the environment dynamics to investigate how
the exploratory control responds to imperfect observations of the system. We
also study the continuous time version of the exploratory problem with latent
factors and Tsallis entropy. We proceed to show that the solution to the
(discrete- and continuous-time) exploratory control problem can be written in
terms of the feedback form of the optimal control in the standard (non-
randomized) control problem, and that the solution to the continuous-time
problem may act as an approximation to the solution in discrete-time. This is
useful in cases where the continuous-time problem has a tractable,
implementable, solution, rather than undertaking the additional effort of
solving a system of equations numerically in the discrete-time formulation.

After investigating how the exploratory control and its approximations behave
in discrete and continuous time, we next illustrate how the Tsallis entropy
may be incorporated into the performance criterion for (soft) $Q$-learning
(Ziebart, 2010; Fox et al., 2015). We show that the $Q$-function when
evaluated on a distribution of actions are tied to those where the
$Q$-function is evaluated on a Dirac mass of actions. This approach
incentivizes further exploration over the “greedy” $Q$-learning algorithms
which always select the action that is currently thought to be highest
performing, and provides better performance compared to taking random
arbitrary actions for the purposes of learning.

The remainder of the paper is organized as follows. In Section 2, we propose a
general linear-quadratic model with a latent factor, introduce exploratory
controls, and solve for the optimal control distribution in feedback form. In
Section 3, we demonstrate the behaviour of the optimal control distribution,
in particular how it depends on the parameters which determine the exploration
reward. In Section 4, we formulate an analogous continuous-time model which is
solved in feedback form. In Section 5, we show that the continuous-time model
acts as an approximation to the discrete time model. Section 6 discusses the
extension to $Q$-learning and Section 7 concludes. Longer proofs are contained
in the appendix.

##  2 Discrete-Time Model

We first introduce the dynamics and performance criterion for a standard
control problem. Then we propose the model which generalizes the dynamics and
performance criterion to allow for exploratory control processes. In the
discrete-time formulation we work with a linear-quadratic framework because
this makes it easier to establish the relation between the optimal standard
control and the optimal exploratory control, as well as the relation between
the discrete-time and continuous-time solutions.

###  2.1 Standard Control Model

Let $T>0$ be a fixed time and divide the interval $[0,T]$ into $N$ equally
sized subintervals of length $\Delta t=T/N$ with endpoints labeled
$0=t_{0}<t_{1}<\cdots<t_{N}=T$. Let
$(\Omega,\mathcal{G},\\{\mathcal{G}_{n}\\}_{n=0}^{N},{\mathbb{P}})$ be a
filtered probability space, and let stochastic processes
$A=\\{A_{n}\\}_{n=0}^{N-1}$ and $M=\\{M_{n}\\}_{n=0}^{N}$ be adapted to the
filtration $\\{\mathcal{G}_{n}\\}_{n=0}^{N}$, where $M$ is a martingale
independent of $A$ with $M_{0}=0$. Additionally, we assume
${\mathbb{E}}\left[\sum_{n=0}^{N-1}A_{n}^{2}\,\Delta t\right]<\infty$ and
${\mathbb{E}}\left[M_{n}^{2}\right]<\infty$.

Let a state process denoted by $Y=\\{Y_{n}\\}_{n=0}^{N}$ satisfy

$\displaystyle Y_{n}$ $\displaystyle=Y_{0}+\sum_{i=0}^{n-1}A_{i}\,\Delta
t+M_{n}\,,$ (1)

with $Y_{0}$ constant. The controlled state process denoted
$X^{\nu}=\\{X^{\nu}_{n}\\}_{n=0}^{N}$ satisfies

$\displaystyle X^{\nu}_{n}$
$\displaystyle=Y_{n}+\sum_{i=0}^{n-1}\gamma_{i}\,\nu_{i}\,\Delta t\,,$ (2)

with $\gamma=\\{\gamma_{n}\\}_{n=0}^{N-1}$ is a sequence of constants and
$\nu=\\{\nu_{n}\\}_{n=0}^{N-1}$ the control process. The performance criterion
for the standard control model is

$\displaystyle J^{\nu}(X_{0})$
$\displaystyle={\mathbb{E}}\biggl{[}-B\,(X^{\nu}_{N})^{2}+\sum_{n=0}^{N-1}(D_{n}\,X^{\nu}_{n}\,\nu_{n}-C_{n}\,(X^{\nu}_{n})^{2}-K_{n}\,\nu_{n}^{2})\,\Delta
t\biggr{]}\,,$ (3)

where $C=\\{C_{n}\\}_{n=0}^{N-1}$, $D=\\{D_{n}\\}_{n=0}^{N-1}$, and
$K=\\{K_{n}\\}_{n=0}^{N-1}$ are sequence of constants and $B$ is a constant.
We assume that all parameters in the model are chosen such that the resulting
functional (3) is concave with respect to the control process $\nu$.

We assume that an agent choosing the control $\nu$ is not able to directly
observe the processes $A$ or $M$, but is able to observe the process $Y$.
Thus, the control $\nu$ must be adapted to the filtration ${\mathcal{F}}$
generated by $Y$, which satisfies ${\mathcal{F}}_{n}\subset\mathcal{G}_{n}$
for each $n$. This makes the processes $A$ and $M$ latent factors and, through
observations of the trajectory of $Y$, may only be approximately estimated.
With the stated partial information setup in mind, we define the agent’s
dynamic value process to be

$\displaystyle H^{(0)}_{n}$
$\displaystyle=\sup_{\nu\in\mathcal{A}}{\mathbb{E}}\biggl{[}-B\,(X^{\nu}_{N})^{2}+\sum_{i=n}^{N-1}(D_{i}\,X^{\nu}_{i}\,\nu_{i}-C_{i}\,(X^{\nu}_{i})^{2}-K_{i}\,\nu_{i}^{2})\,\Delta
t\,\biggr{|}\,{\mathcal{F}}_{n}\biggr{]}\,,$ (4)

where $\mathcal{A}$ is the set of admissible strategies given by

$\displaystyle\mathcal{A}$
$\displaystyle=\biggl{\\{}\nu:\nu_{n}\in{\mathcal{F}}_{n}\mbox{ and
}{\mathbb{E}}\biggl{[}\sum_{n=0}^{N-1}\nu_{n}^{2}\biggr{]}<\infty\biggr{\\}}\,.$
(5)

To apply standard dynamic programming techniques to obtain the optimal
control, it is useful to rewrite the process $Y$ in (1) in terms of only
${\mathcal{F}}_{n}$-adapted processes. To this end, let
$\widehat{A}_{n}={\mathbb{E}}[A_{n}\,|\,{\mathcal{F}}_{n}]$ and
$\widehat{M}_{n}=\sum_{i=0}^{n-1}Y_{i+1}-Y_{i}-\widehat{A}_{i}\,\Delta t$.
Then we have

$\displaystyle Y_{n}$
$\displaystyle=Y_{0}+\sum_{i=0}^{n-1}\widehat{A}_{i}\,\Delta
t+\widehat{M}_{n}\,,$ (6)

and $\widehat{M}$ is easily seen to be a martingale which is
${\mathcal{F}}_{n}$-adapted. An application of the dynamic programming
principle (DPP) yields that the optimal control is given by

$\displaystyle\nu^{*}_{n}$
$\displaystyle=\frac{{\mathbb{E}}[h^{(1)}_{n+1}\,|\,{\mathcal{F}}_{n}]\,\gamma_{n}+2\,h^{(2)}_{n+1}\,\widehat{A}_{n}\,\gamma_{n}\,\Delta
t+(2\,h^{(2)}_{n+1}\,\gamma_{n}+D_{n})\,X_{n}}{2\,(K_{n}-h^{(2)}_{n+1}\,\gamma_{n}^{2}\,\Delta
t)}\,,$ (7)

where the ${\mathcal{F}}_{n}$-adapted process $h^{(1)}$ and the deterministic
sequence $h^{(2)}$ satisfy the backward stochastic difference equation
(BS$\Delta$E)

$\displaystyle\begin{split}h^{(1)}_{n}=&\;{\mathbb{E}}[h^{(1)}_{n+1}|{\mathcal{F}}_{n}]+2\,h^{(2)}_{n+1}\,\widehat{A}_{n}\,\Delta
t\\\
&\;+\frac{({\mathbb{E}}[h^{(1)}_{n+1}|{\mathcal{F}}_{n}]\,\gamma_{n}+2\,h^{(2)}_{n+1}\,\widehat{A}_{n}\,\gamma_{n}\,\Delta
t)\,(2\,h^{(2)}_{n+1}\,\gamma_{n}+D_{n})}{2\,(K_{n}-h^{(2)}_{n+1}\,\gamma_{n}^{2}\,\Delta
t)}\,\Delta t\,,\end{split}$ (8a) $\displaystyle h^{(2)}_{n}=$
$\displaystyle\;h^{(2)}_{n+1}-C_{n}\,\Delta
t+\frac{(2\,h^{(2)}_{n+1}\,\gamma_{n}+D_{n})^{2}}{4\,(K_{n}-h^{(2)}_{n+1}\,\gamma_{n}^{2}\,\Delta
t)}\,\Delta t\,,$ (8b)

subject to the terminal conditions $h^{(1)}_{N}=0$ and $h^{(2)}_{N}=-B$. The
details of this computation are omitted, but they are similar to the
derivation of the optimal exploratory control which are provided in Appendix A

###  2.2 Control Model with Exploration

In this section, we modify the dynamics of the standard control problem to
allow for exploratory control processes, and we also modify the performance
criterion to incorporate an exploration reward. The exploration reward will
incentivize the agent to choose a randomized control at each time point, even
when conditioning on all information available up to the time when the control
is implemented. The randomization of the implemented control will also require
a modification to the information structure of the problem. It this
modification of the information structure to incorporate randomized actions
that makes the continuous time counterpart somewhat poorly specified, but the
discrete-time problem precisely specified. To this end, let
$\\{u_{n}\\}_{n=0}^{N-1}$ be a sequence of independent $\mathcal{U}(0,1)$
random variables, also independent from both $A$ and $M$, and denote the
agent’s control by $\pi=\\{\pi_{n}\\}_{n=0}^{N-1}$,
$\pi_{n}|_{{\mathcal{F}}_{n}}\in{\mathcal{P}}({\mathds{R}})$ where
${\mathcal{P}}({\mathds{R}})$ is the space of probability measures on
${\mathds{R}}$. Hence, $\pi$ is a non-negative random field that integrates to
unity. The dynamics of the previous section are modified to

$\displaystyle F^{\pi}_{n}(x)$
$\displaystyle=\int_{-\infty}^{x}\pi_{n}(\nu)\,d\nu\,,$ (9)
$\displaystyle\nu^{\pi}_{n}$ $\displaystyle=(F^{\pi}_{n})^{-1}(u_{n})\,,$ (10)
$\displaystyle X_{n}^{\pi}$
$\displaystyle=Y_{n}+\sum_{i=0}^{n-1}\gamma_{i}\,\nu^{\pi}_{i}\,\Delta t\,.$
(11)

We extend the filtration that the agent’s adapts their strategy to so that she
may observe the realizations of her past control. Specifically, the we denote
the $\sigma$-algebra ${\mathcal{F}}^{u}_{n}=\sigma\\{(Y_{k},u_{i}):k\leq
n,i<n\\}$, and the set admissible exploratory controls $\mathcal{A}^{r}$ is
defined to be ${\mathcal{F}}^{u}_{n}$-adapted random fields such that

$\displaystyle\int_{-\infty}^{\infty}\pi_{n}(\nu)\,d\nu$ $\displaystyle=1\,,$
$\displaystyle{\mathbb{E}}\left[\sum_{n=0}^{N-1}(\nu^{\pi}_{n})^{2}\,\Delta
t\right]$ $\displaystyle<\infty\,,$ $\displaystyle\pi_{n}(\nu)\geq 0\,.$ (12)

By defining the $\sigma$-algebra ${\mathcal{F}}^{u}_{n}$ such that
$u_{n}\notin{\mathcal{F}}^{u}_{n}$ and $u_{n}\in{\mathcal{F}}^{u}_{n+1}$, at
time $n$ the agent does not know what her control $\nu_{n}$ will be, but she
controls the distribution of this random quantity. In light of Equations
(9)-(10), the distribution of $\nu_{n}$ conditional on ${\mathcal{F}}^{u}_{n}$
has density $\pi_{n}$ and in particular,

$\displaystyle{\mathbb{E}}\left[\nu^{\pi}_{n}\,|\,{\mathcal{F}}^{u}_{n}\right]$
$\displaystyle=\int_{-\infty}^{\infty}\nu\,\pi_{n}(\nu)\,d\nu\,,$ (13)
$\displaystyle{\mathbb{E}}\left[(\nu_{n}^{\pi})^{2}\,|\,{\mathcal{F}}^{u}_{n}\right]$
$\displaystyle=\int_{-\infty}^{\infty}\nu^{2}\,\pi_{n}(\nu)\,d\nu\,.$ (14)

Further, conditional on ${\mathcal{F}}^{u}_{n+1}$ at the next time step, she
is able to observe $u_{n}$ and therefore $\nu^{\pi}_{n}$ as her implemented
control has become realized.

We now modify the performance criterion from (3) to reward exploration in the
choice of control. This reward comes in the form of either the Shannon or
Tsallis entropy (see Tsallis (1988)) of the distribution of her control so
that the new performance criterion is

$\displaystyle J^{\pi}(X_{0})$
$\displaystyle={\mathbb{E}}\biggl{[}-B\,(X^{\pi}_{N})^{2}+\sum_{n=0}^{N-1}(D_{n}\,X^{\pi}_{n}\,\nu^{\pi}_{n}-C_{n}\,(X^{\pi}_{n})^{2}-K_{n}\,(\nu^{\pi}_{n})^{2})\,\Delta
t+\lambda\,S_{q}[\pi_{n}]\,\Delta t\biggr{]}\,,$ (15)

where the parameter $\lambda>0$ controls the magnitude of the reward for
exploration, and the quantity $S_{q}\left[\pi_{n}\right]$ is given by

$\displaystyle S_{q}[\pi_{n}]$
$\displaystyle=\left\\{\begin{array}[]{cc}\displaystyle\frac{1}{q-1}\biggl{(}1-\int_{-\infty}^{\infty}(\pi_{n}(\nu))^{q}\,d\nu\biggr{)}\,,&q>1/3,q\neq
1\\\\[10.00002pt]
\displaystyle-\int_{-\infty}^{\infty}\pi_{n}(\nu)\,\log(\pi_{n}(\nu))\,d\nu\,,&q=1\end{array}\right.\,.$
(18)

For a fixed action distribution $\pi_{n}$, we have $\lim_{q\rightarrow
1}S_{q}[\pi_{n}]=S_{1}[\pi_{n}]$ which is why we reduce to the Shannon entropy
in that limit. The bound $q>1/3$ is necessary so that the resulting optimal
densities have finite variance.

###  2.3 Optimal Exploratory Control

In this section, we obtain the optimal exploratory control $\pi^{*}$ and
discuss the relation between this sequence of conditional densities and the
optimal control in the standard framework. We apply the DPP to the dynamic
value process in the exploratory model (whose analogous without exploration is
(4)) given by

$\begin{split}H^{(\lambda)}_{n}=&\;\sup_{\pi\in\mathcal{A}^{r}}{\mathbb{E}}\biggl{[}-B\,(X^{\pi}_{N})^{2}+\sum_{i=n}^{N-1}(D_{i}\,X^{\pi}_{i}\,\nu^{\pi}_{i}-C_{i}\,(X^{\pi}_{i})^{2}-K_{i}\,(\nu^{\pi}_{i})^{2})\,\Delta
t\\\ &\qquad\qquad+\lambda\,S_{q}[\pi_{i}]\,\Delta
t\;\biggr{|}\;{\mathcal{F}}^{u}_{n}\biggr{]}\,.\end{split}$ (19)

######  Proposition 1 (Discrete-time optimal exploratory control)

Let $h^{(1)}$ and $h^{(2)}$ satisfy the BS$\Delta$E in Equation (8) and assume
$K_{n}-h^{(2)}_{n+1}\,\gamma_{n}^{2}\,\Delta t>0$ for each $n$. Then, the
optimal control density is given by

$\displaystyle\pi_{n}^{*}(\nu)$
$\displaystyle=\left\\{\begin{array}[]{cc}\displaystyle\biggl{(}\frac{q-1}{\lambda\,q}\biggr{)}^{\frac{1}{q-1}}\biggl{(}\psi_{n}-\frac{\lambda}{2\,\varsigma_{n}^{2}}\,(\nu-\mu_{n})^{2}\biggr{)}_{+}^{\frac{1}{q-1}}\,,&q>1\\\\[10.00002pt]
\displaystyle\frac{1}{\sqrt{2\,\pi\,\varsigma_{n}^{2}}}\,\exp\biggl{\\{}-\frac{(\nu-\mu_{n})^{2}}{2\,\varsigma_{n}^{2}}\biggr{\\}}\,,&q=1\\\\[10.00002pt]
\displaystyle\biggl{(}\frac{1-q}{\lambda\,q}\biggr{)}^{\frac{1}{q-1}}\biggl{(}\psi_{n}+\frac{\lambda}{2\,\varsigma_{n}^{2}}\,(\nu-\mu_{n})^{2}\biggr{)}^{\frac{1}{q-1}}\,,&1/3<q<1\end{array}\right.$
(23)

where

$\displaystyle\mu_{n}$
$\displaystyle=\displaystyle\frac{{\mathbb{E}}[h^{(1)}_{n+1}|{\mathcal{F}}_{n}]\,\gamma_{n}+2\,h^{(2)}_{n+1}\,\widehat{A}_{n}\,\gamma_{n}\,\Delta
t+(2\,h^{(2)}_{n+1}\,\gamma_{n}+D_{n})\,X^{\pi^{*}}_{n}}{2\,(K_{n}-h^{(2)}_{n+1}\,\gamma_{n}^{2}\,\Delta
t)}\,,$ (24) $\displaystyle\varsigma_{n}^{2}$
$\displaystyle=\frac{\lambda}{2\,(K_{n}-h^{(2)}_{n+1}\,\gamma_{n}^{2}\,\Delta
t)}\,,$ (25) $\displaystyle\psi_{n}$
$\displaystyle=\left\\{\begin{array}[]{cc}\displaystyle\Biggl{[}\frac{\Gamma(\frac{1}{q-1}+\frac{3}{2})}{\sqrt{\pi}\,\Gamma(\frac{1}{q-1}+1)}\biggl{(}\frac{\lambda\,q}{q-1}\biggr{)}^{\frac{1}{q-1}}\sqrt{K_{n}-h^{(2)}_{n+1}\,\gamma_{n}^{2}\,\Delta
t}\Biggr{]}^{\frac{1}{\frac{1}{q-1}+\frac{1}{2}}}\,,&q>1\\\\[10.00002pt]
\displaystyle\Biggl{[}\frac{\Gamma(\frac{1}{1-q})}{\sqrt{\pi}\,\Gamma(\frac{1}{1-q}-\frac{1}{2})}\biggl{(}\frac{\lambda\,q}{1-q}\biggr{)}^{\frac{1}{q-1}}\sqrt{K_{n}-h^{(2)}_{n+1}\,\gamma_{n}^{2}\,\Delta
t}\Biggr{]}^{\frac{1}{\frac{1}{q-1}+\frac{1}{2}}}\,,&1/3<q<1\end{array}\right.$
(28)

  * Proof

See Appendix A.

The optimal exploratory control in Proposition 1 includes a location parameter
denoted by $\mu_{n}$. This location parameter has identical feedback form to
the optimal control in the standard framework given in Equation (7). Thus, as
the $q$-Gaussian distribution is symmetric, at each point in time, the
exploratory control is chosen such that it is centered around what would be
optimal in the standard approach. When $q=1$, the optimal density is Gaussian,
the variance of the random control is directly proportional to the exploration
reward parameter $\lambda$, which is expected because there is more incentive
to deviate farther from the standard optimal control. When $q\neq 1$ the
densities come from the class of $q$-Gaussian distributions which have
distinctly different behaviour depending on whether $q$ is greater than or
less than $1$. When $q>1$, the density has compact support centered at
$\mu_{n}$ so that the density is only positive for
$|\nu-\mu_{n}|<\sqrt{\psi_{n}/(K_{n}-h^{(2)}_{n+1}\,\gamma_{n}^{2}\,\Delta
t)}$, whereas with $q<1$ the optimal distribution has unbounded support with
fat tails. For $q\neq 1$ it is difficult to see directly how the variance of
the random controls depends on the reward parameter $\lambda$ and the entropy
parameter $q$, but these are plotted in Figure 1.

Figure 1: The left panel shows the centered optimal density from (23) for
various values of $q$ with $\lambda=1$. When $q=1$ the density is Gaussian,
otherwise it has bounded support depending on the parameter $\psi_{n}$. The
middle panel shows the parameter $\psi_{n}$ as a function of $q$ for various
values of $\lambda$. The right panel shows the variance corresponding to the
density in (23). We set the quantity
$K_{n}-h^{(2)}_{n+1}\,\gamma_{n}^{2}\,\Delta t=0.1$ in all cases.

In Figure 1 we show examples of the density in (23) for various values of $q$.
The shrinking support as $q$ increases with $q>1$ is clear, and stems from
$\psi_{n}$ being decreasing in $q$ in this domain (shown in the middle panel).
The right panel shows the variance of the optimal density as $q$ and $\lambda$
vary. As expected, for all values of $q$ the variance is increasing with
respect to $\lambda$. This is due to the higher reward for exploration,
however, the dependence is sub-linear and less pronounced when $q$ is large.
The variance also decreases with increasing $q$, and is to be expected,
because as $q$ increases the distribution becomes less heavy tailed and
eventually (for $q>1$) has bounded support with shrinking support as $q$
increases.

##  3 Discrete-Time Numerical Experiments

In this section, we demonstrate, through simulations222We use the method given
in Thistleton et al. (2007) to generate $q$-Gaussian deviates., how the
optimal control distribution behaves and investigate how it depends on the
exploration parameter $\lambda$ and the Tsallis entropy parameter $q$. Doing
so requires specifying the dynamics of the latent factor process $A$ and
unaffected state process $Y$. To keep computations tractable, to obtain the
solution to the BS$\Delta$E in (8a), we specify the dynamics of $Y$ and $A$ to
be

$\displaystyle Y_{n}$ $\displaystyle=Y_{0}+\sum_{i=0}^{n-1}A_{i}\,\Delta
t+\sigma\,W^{(1)}_{t_{n}}\,,$ (29) $\displaystyle A_{n+1}$
$\displaystyle=e^{-\kappa\,\Delta t}\,A_{n}+\eta\,\Delta W^{(2)}_{t_{n}}\,,$
(30)

where $W^{(1)}$ and $W^{(2)}$ are independent Brownian motions (sampled at the
discrete times $\\{t_{0},\dots,t_{N}\\}$) and $\Delta
W^{(2)}_{t_{n}}:=W^{(2)}_{t_{n}}-W^{(2)}_{t_{n-1}}$, and independent of
$A_{0}\stackrel{{\scriptstyle{\mathbb{P}}}}{{\sim}}\mathcal{N}(0,\Sigma_{0})$.
Hence, $A$ may be viewed as a discretely sampled Ornstein-Uhlenbeck (OU)
process and $Y$ a discretely observed diffusion with stochastic drift given by
$A$. With this specification of $A$ and $Y$, standard filtering results (see,
e.g., Theorem 13.4 in Liptser and Shiryaev (2013)) show that the filtered
process $\widehat{A}$ is given by

$\displaystyle\widehat{A}_{n+1}$ $\displaystyle=e^{-\kappa\,\Delta
t}\,\widehat{A}_{n}+\frac{e^{-\kappa\,\Delta
t}\,\Sigma_{n}}{\sigma^{2}+\Sigma_{n}\,\Delta
t}\,(Y_{n+1}-Y_{n}-\widehat{A}_{n}\,\Delta t)\,,$
$\displaystyle\widehat{A}_{0}$ $\displaystyle={\mathbb{E}}[A_{0}]\,,$ (31)

where the deterministic sequence $\Sigma$ represents the estimation error of
$\widehat{A}$ and is given recursively by

$\displaystyle\Sigma_{n+1}$ $\displaystyle=e^{-2\,\kappa\,\Delta
t}\,\Sigma_{n}+\eta^{2}\,\Delta t-\frac{e^{-2\,\kappa\,\Delta
t}\,\Sigma_{n}^{2}\,\Delta t}{\sigma^{2}+\Sigma_{n}\,\Delta t}\,,\qquad
n=0,1,\dots,N-1.$

Linearity of the dynamics of $\widehat{A}$ allow us to represent $h^{(1)}$,
the solution to (8a), in terms of the solution to a deterministic difference
equation.

###### Proposition 2

Let $\widehat{A}$ be given by (31) and let $h^{(2)}$ be the solution to (8b).
Then the solution to (8a) is given by

$\displaystyle h^{(1)}_{n}$ $\displaystyle=\phi_{n}\,\widehat{A}_{n}\,,$ (32)

where $\phi$ is the solution to the backwards difference equation

$\displaystyle\phi_{n}$ $\displaystyle=a_{n}\,\Delta t+(1+b_{n}\,\Delta
t)\,e^{-\kappa\,\Delta t}\,\phi_{n+1}\,,$ $\displaystyle\phi_{N}$
$\displaystyle=0\,,$ (33)

with

$\displaystyle a_{n}$ $\displaystyle=2\,h^{(2)}_{n+1}\,(1+b_{n}\,\Delta t)\,,$
$\displaystyle b_{n}$
$\displaystyle=\frac{\gamma_{n}\,(2\,h^{(2)}_{n+1}\,\gamma_{n}+D_{n})}{2\,(K_{n}-h^{(2)}_{n+1}\,\gamma_{n}^{2}\,\Delta
t)}\,.$

  * Proof

The recursion (33) follows from direct substitution of (32) into (8a) and
using ${\mathbb{E}}[\widehat{A}_{n+1}|{\mathcal{F}}_{n}]=e^{-\kappa\,\Delta
t}\,\widehat{A}_{n}$ from (31).

For the dynamics given in (29) and (30), computing the optimal exploratory
control reduces to solving the backward difference equations (8b) and (33)
which is carried out numerically. Given the solutions $h^{(1)}$ and $h^{(2)}$
to (8), the optimal exploratory control in Proposition 1 can be implemented
and demonstrated. For this purpose we set the sequences
$C=\\{C_{n}\\}_{n=0}^{N-1}$, $D=\\{D_{n}\\}_{n=0}^{N-1}$,
$K=\\{K_{n}\\}_{n=0}^{N-1}$, and $\gamma=\\{\gamma_{n}\\}_{n=0}^{N-1}$ to be
constant.

Figure 2: Sample paths for various values of Tsallis entropy parameter $q$
(left) and exploration reward parameter $\lambda$ (right). Other parameter
values are $B=1$, $C=1$, $D=1$, $K=0.1$, $\gamma=1$, $N=10$, $\sigma=0.2$,
$\kappa=1$, $\eta=2$, $\widehat{A}_{0}=0$, $\Sigma_{0}=1$. In the left panel
we use $\lambda=0.5$ and in the right panel we use $q=2$. The black curve
represents the state path using the classical optimal control in (7).

In Figure 2 we show sample paths when implementing the optimal exploratory
control. In the left panel we observe that larger values of the Tsallis
entropy parameter $q$ typically result in the state path being closer to the
one corresponding to the classical optimal control (black curve). This is
expected, because as $q$ increases, the $q$-Gaussian distribution becomes more
concentrated, as in the discussions surrounding Figure 1. The figure also
shows that deviation of the paths from the classical optimal path increases as
$\lambda$ increases – this also expected behaviour due to the increased reward
for exploration and deviating from the classical optimizer.

##  4 Continuous-Time Model

Similar to Section 2, but now in continuous time, we present the dynamics and
performance criterion for a standard control problem so that when we solve the
problem with exploration we may directly compare the solutions.

###  4.1 Standard Control Model

Let $T>0$ be fixed, let
$(\Omega,\mathcal{G},\\{\mathcal{G}_{t}\\}_{t\in[0,T]},{\mathbb{P}})$ be a
filtered probability space, and let $A=\\{A_{t}\\}_{t\in[0,T]}$ and
$W=\\{W_{t}\\}_{t\in[0,T]}$ be adapted to the filtration
$\\{\mathcal{G}_{t}\\}_{t\in[0,T]}$, where $W$ is a Brownian motion
independent of $A$ and with ${\mathbb{E}}[\int_{0}^{T}A_{t}^{2}\,dt]<\infty$.

The unaffected state process is denoted by $Y=\\{Y_{t}\\}_{t\in[0,T]}$ and has
dynamics given by

$\displaystyle dY_{t}$ $\displaystyle=A_{t}\,dt+\sigma\,dW^{(1)}_{t}\,,$ (34)

with $Y_{0}$ and $\sigma>0$ constant. The controlled state process is denoted
by $X^{\nu}=\\{X^{\nu}_{t}\\}_{t\in[0,T]}$ and satisfies

$\displaystyle X_{t}^{\nu}$
$\displaystyle=Y_{t}+\int_{0}^{t}\gamma_{u}\,\nu_{u}\,du\,,$ (35)

where $\gamma=\\{\gamma_{t}\\}_{t\in[0,T]}$ is deterministic and
$\nu=\\{\nu_{t}\\}_{t\in[0,T]}$ is the control process. The performance
criterion in the standard control model is

$\displaystyle J^{\nu}(X_{0})$
$\displaystyle={\mathbb{E}}\biggl{[}\;g(X_{T}^{\nu})+\int_{0}^{T}\left(D_{t}\,X_{t}^{\nu}\,\nu_{t}-C_{t}\,(X_{t}^{\nu})^{2}-K_{t}\,\nu_{t}^{2}\right)dt\;\biggr{]}\,,$
(36)

where $C=\\{C_{t}\\}_{t\in[0,T]}$, $D=\\{D_{t}\\}_{t\in[0,T]}$, and
$K=\\{K_{t}\\}_{t\in[0,T]}$ are deterministic. As in discrete-time, admissible
controls are processes that are adapted to the filtration
$\\{{\mathcal{F}}_{t}\\}_{t\in[0,T]}$ generated by $Y$. It proves useful to
rewrite the dynamics of $Y$ by standard results of Kalman filtering (see for
example Fujisaki et al. (1972)) as

$\displaystyle dY_{t}$
$\displaystyle=\widehat{A}_{t}\,dt+\sigma\,d\widehat{W}^{(1)}_{t}\,,$ (37)

where $\widehat{A}_{t}={\mathbb{E}}[\,A_{t}\,|\,{\mathcal{F}}_{t}\,]$ and
$\widehat{W}$ is a Brownian motion with respect to the filtration
$\\{{\mathcal{F}}_{t}\\}_{t\in[0,T]}$. We assume that the process
$\widehat{A}$ has the Markov property with respect to the filtration
$\\{{\mathcal{F}}_{t}\\}_{t\in[0,T]}$ which allows us to more easily take a
Hamilton-Jacobi-Bellman (HJB) approach to classifying the optimal control.
With this in mind, the corresponding dynamic value function $H^{(0)}$
satisfies

$\displaystyle H^{(0)}(t,X_{t},\widehat{A}_{t})$
$\displaystyle=\sup_{\nu\in\mathcal{A}}{\mathbb{E}}\biggl{[}\;g(X_{T}^{\nu})+\int_{t}^{T}\left(D_{u}\,X_{u}^{\nu}\,\nu_{u}-C_{u}\,(X_{u}^{\nu})^{2}-K_{u}\,\nu_{u}^{2}\right)du\,\biggl{|}\,{\mathcal{F}}_{t}\;\biggr{]}\,,$
(38)

where the set of admissible strategies is given by

$\displaystyle\mathcal{A}$
$\displaystyle=\biggl{\\{}\nu:\nu_{t}\in{\mathcal{F}}_{t}\mbox{ and
}{\mathbb{E}}\biggl{[}\int_{0}^{T}\nu_{t}^{2}\,dt\biggr{]}<\infty\biggr{\\}}\,.$
(39)

Standard techniques show that the dynamic value function satisfies the HJB
equation and terminal condition

$\begin{split}\partial_{t}H^{(0)}+\mathcal{L}^{\widehat{A}}H^{(0)}+\widehat{A}\,\partial_{x}H^{(0)}+\tfrac{1}{2}\,\sigma^{2}\,\partial_{xx}H^{(0)}-C_{t}\,x^{2}+\frac{(\gamma_{t}\,\partial_{x}H^{(0)}+D_{t}\,x)^{2}}{4\,K_{t}}&=0\,,\\\
H^{(0)}(T,x,\widehat{A})&=g(x)\,,\end{split}$ (40)

and that the optimal control in feedback form is given by

$\displaystyle\nu^{*}(t,X_{t},\widehat{A}_{t})$
$\displaystyle=\frac{\gamma_{t}\,\partial_{x}H^{(0)}(t,X_{t},\widehat{A}_{t})+D_{t}\,X_{t}}{2\,K_{t}}\,.$
(41)

###  4.2 Control Model with Exploration

In continuous-time when the control is implemented continuously, there is not
a proper notion of a delayed observation of the control which necessitated the
introduction of a sequence of independent uniform variables and a modification
of the agent’s filtration in Section 2.2. Instead, we consider a framework
similar to other works with exploratory controls, such as Wang et al. (2020),
Guo et al. (2022), and Firoozi and Jaimungal (2022). Thus, we replace the
controlled process $X^{\nu}$ in the previous section with its exploratory
control counterpart

$\displaystyle X_{t}^{\pi}$
$\displaystyle=Y_{t}+\int_{0}^{t}\int_{-\infty}^{\infty}\gamma_{u}\,\nu\,\pi_{u}(\nu)\,d\nu\,du\,,$
(42)

where $\pi=\\{\pi_{t}\\}_{t\in[0,T]}$ is a random field adapted to
$\\{{\mathcal{F}}_{t}\\}_{t\in[0,T]}$ and acts as the control. To provide an
incentive for exploration, the performance criterion is modified to include an
entropic reward:

$\displaystyle J^{\pi}(X_{0})$
$\displaystyle={\mathbb{E}}\biggl{[}g(X_{T}^{\pi})+\int_{0}^{T}\int_{-\infty}^{\infty}\left(D_{t}\,X_{t}^{\pi}\,\nu-
C_{t}\,(X_{t}^{\pi})^{2}-K_{t}\,\nu^{2}\right)\,\pi_{t}(\nu)\,d\nu\,dt+\lambda\int_{0}^{T}S_{q}[\pi_{t}]\,dt\biggr{]}\,,$

where $S_{q}[\pi]$ is given by (18). The corresponding dynamic value function,
denoted by $H^{(\lambda)}$, satisfies

$\displaystyle\begin{split}H^{(\lambda)}(t,X_{t},\widehat{A}_{t})=\sup_{\pi\in\mathcal{A}^{r}}{\mathbb{E}}\biggl{[}\;&g(X_{T}^{\pi})+\int_{t}^{T}\int_{-\infty}^{\infty}(D_{u}\,X_{u}^{\pi}\,\nu-
C_{u}\,(X_{u}^{\pi})^{2}-K_{u}\,\nu^{2})\,\pi_{u}(\nu)\,d\nu\,du\\\
&+\lambda\int_{t}^{T}S_{q}[\pi_{u}]\,du\,\biggl{|}\,{\mathcal{F}}_{t}\;\biggr{]}\,,\end{split}$

where the set of admissible exploratory controls is

$\displaystyle\mathcal{A}^{r}$
$\displaystyle=\biggl{\\{}\pi:\pi_{t}\in{\mathcal{F}}_{t}\,,\,\pi_{t}\geq
0\,,\,\int_{-\infty}^{\infty}\pi_{t}(\nu)\,d\nu=1\,,\mbox{ and
}{\mathbb{E}}\biggl{[}\int_{0}^{T}\int_{-\infty}^{\infty}\nu^{2}\,\pi_{t}(\nu)\,d\nu\,dt\biggr{]}<\infty\biggr{\\}}\,.$

The following proposition gives the optimal exploratory control analogous to
the discrete time setting of Proposition 1, and shows that the exploratory
control solution may be written in terms of the standard control problem given
in (41). Additionally, we draw a relationship between the exploratory value
function $H^{(\lambda)}$ and the standard value function $H^{(0)}$.

######  Proposition 3 (Continuous-time optimal exploratory control)

The dynamic value function $H^{(\lambda)}$ is equal to

$\displaystyle H^{(\lambda)}(t,x,\widehat{A})$
$\displaystyle=H^{(0)}(t,x,\widehat{A})-\int_{t}^{T}\alpha_{q}(u)\,du\,,$ (43)

where $H^{(0)}$ is the dynamic value function of the standard control problem
and where

$\displaystyle\alpha_{q}(t)$
$\displaystyle=\left\\{\begin{array}[]{cc}-\frac{\lambda}{q-1}+\frac{\psi_{t}\,(q+1)}{3\,q-1}\,,&q>1\\\\[10.00002pt]
\lambda\,\log(\sqrt{\frac{\pi\,\lambda}{K_{t}}})\,,&q=1\\\\[10.00002pt]
-\frac{\lambda}{q-1}-\frac{\psi_{t}\,(q+1)}{3\,q-1}\,,&1/3<q<1,\\\\[10.00002pt]
\end{array}\right.$ (47)

and

$\displaystyle\psi_{t}$
$\displaystyle=\left\\{\begin{array}[]{cc}\Biggl{[}\frac{\Gamma(\frac{1}{q-1}+\frac{3}{2})}{\sqrt{\pi}\,\Gamma(\frac{1}{q-1}+1)}\biggl{(}\frac{\lambda\,q}{q-1}\biggr{)}^{\frac{1}{q-1}}\sqrt{K_{t}}\Biggr{]}^{\frac{1}{\frac{1}{q-1}+\frac{1}{2}}}\,,&q>1\\\\[10.00002pt]
\Biggl{[}\frac{\Gamma(\frac{1}{1-q})}{\sqrt{\pi}\,\Gamma(\frac{1}{1-q}-\frac{1}{2})}\biggl{(}\frac{\lambda\,q}{1-q}\biggr{)}^{\frac{1}{q-1}}\sqrt{K_{t}}\Biggr{]}^{\frac{1}{\frac{1}{q-1}+\frac{1}{2}}}\,,&1/3<q<1.\end{array}\right.\,.$
(50)

Furthermore, the optimal control density is given by

$\displaystyle\pi_{t}^{*}(\nu)$
$\displaystyle=\left\\{\begin{array}[]{cc}\biggl{(}\frac{q-1}{\lambda\,q}\biggr{)}^{\frac{1}{q-1}}\biggl{(}\psi_{t}-\frac{\lambda}{2\,\varsigma_{t}^{2}}\,(\nu-\mu_{t})^{2}\biggr{)}_{+}^{\frac{1}{q-1}}\,,&q>1\\\\[10.00002pt]
\frac{1}{\sqrt{2\,\pi\,\varsigma_{t}^{2}}}\,\exp\biggl{\\{}-\frac{(\nu-\mu_{t})^{2}}{2\,\varsigma_{t}^{2}}\biggr{\\}}\,,&q=1\\\\[10.00002pt]
\biggl{(}\frac{1-q}{\lambda\,q}\biggr{)}^{\frac{1}{q-1}}\biggl{(}\psi_{t}+\frac{\lambda}{2\,\varsigma_{t}^{2}}\,(\nu-\mu_{t})^{2}\biggr{)}^{\frac{1}{q-1}}\,,&1/3<q<1,\\\
\end{array}\right.$ (54)

where

$\displaystyle\mu_{t}$
$\displaystyle=\frac{\gamma_{t}\,\partial_{x}H^{(0)}(t,X_{t}^{\pi^{*}},\widehat{A}_{t})+D_{t}\,X_{t}^{\pi^{*}}}{2\,K_{t}}\,,$
(55) $\displaystyle\varsigma_{t}^{2}$
$\displaystyle=\frac{\lambda}{2\,K_{t}}\,.$ (56)

  * Proof

See Appendix B.

##  5 Discrete and Continuous Relation

In this section we demonstrate two relationships between the discrete- and
continuous-time formulations of exploration. The first is the convergence of
the discrete-time solution to the continuous-time solution, and the second is
that we may use some terms which appear in the continuous-time solution to
construct approximations to the discrete time solution, which may be desirable
because the continuous-time counterparts are generally easier to evaluate in
closed form. At the heart of these relationships is the fact that the
discrete-time system (8) converges to the solution of a continuous-time
analogue, which we now show before proceeding to numerical experiments.

To this end, we specify $g(x)=-B\,x^{2}$, the same terminal reward as in the
discrete-time formulation. This also makes the continuous time solution
explicitly tractable. With this terminal payoff the corresponding HJB equation
can be further simplified with the ansatz
$H^{(\lambda)}(t,x,\widehat{A})=f_{0}(t,\widehat{A})+f_{1}(t,\widehat{A})\,x+f_{2}(t)\,x^{2}$.
Substituting this form back into the HJB equation for $H^{(\lambda)}$ (see
(81) from the proof of Proposition 3) and grouping terms by powers of $x$
gives the three coupled equations

$\displaystyle\partial_{t}f_{0}+\mathcal{L}^{\widehat{A}}f_{0}+\widehat{A}\,f_{1}+\sigma^{2}\,f_{2}+\frac{\gamma_{t}^{2}\,f_{1}^{2}}{4\,K_{t}}-\alpha_{q}$
$\displaystyle=0\,,$ $\displaystyle f_{0}(T,\widehat{A})$ $\displaystyle=0\,,$
$\displaystyle\partial_{t}f_{1}+\mathcal{L}^{\widehat{A}}f_{1}+2\,\widehat{A}\,f_{2}+\frac{\gamma_{t}\,f_{1}\,(2\,\gamma_{t}\,f_{2}+D_{t})}{2\,K_{t}}$
$\displaystyle=0\,,$ $\displaystyle f_{1}(T,\widehat{A})$ $\displaystyle=0\,,$
$\displaystyle\partial_{t}f_{2}-C_{t}+\frac{(2\,\gamma_{t}\,f_{2}+D_{t})^{2}}{4\,K_{t}}$
$\displaystyle=0\,,$ $\displaystyle f_{2}(T)$ $\displaystyle=-B\,.$

Defining the processes $h^{(1)}$ and $h^{(2)}$ by

$\displaystyle h^{(1)}_{t}=f_{1}(t,\widehat{A}_{t})\qquad\text{and}\qquad
h^{(2)}_{t}=f_{2}(t)\,,$

we see that $h^{(1)}$ and $h^{(2)}$ satisfy the system of BSDEs

$\begin{split}dh^{(1)}_{t}&=-\biggl{(}2\,\widehat{A}_{t}\,h_{t}^{(2)}+\frac{\gamma_{t}\,h_{t}^{(1)}\,(2\,\gamma_{t}\,h_{t}^{(2)}+D_{t})}{2\,K_{t}}\biggr{)}\,dt+dZ_{t}\,,\\\
dh^{(2)}_{t}&=-\biggl{(}\frac{(2\,\gamma_{t}\,h_{t}^{(2)}+D_{t})^{2}}{4\,K_{t}}-C_{t}\biggr{)}\,dt\,.\end{split}$
(57)

with terminal conditions $h^{(1)}_{T}=0$ and $h^{(2)}_{T}=-B$, and where $Z$
is a martingale. Inspection of the BS$\Delta$E (8) in the formal limit $\Delta
t\rightarrow 0$ shows a similarity to the BSDE (57). This form of the value
function also reduces (55) to

$\displaystyle\mu_{t}$
$\displaystyle=\frac{\gamma_{t}\,h^{(1)}_{t}+(2\,\gamma_{t}\,h^{(2)}_{t}+D_{t})\,X_{t}^{\pi^{*}}}{2\,K_{t}}\,.$
(58)

Note again the formal similarity to (24) in the limit $\Delta t\rightarrow 0$.

Below, we use known results about finite difference methods for BSDEs and
dependence upon a parameter of solutions to BSDEs to establish that for
sufficiently small $\Delta t$, the solution to (57) will approximate the
solution to (8). First we recognize that different occurrences of $\Delta t$
in (8) play different roles. Equation (8) is rewritten with some occurrences
of $\Delta t$ replaced by a positive parameter $\beta$ and becomes

$\begin{split}h^{(1,\beta)}_{n}&={\mathbb{E}}[h^{(1,\beta)}_{n+1}|{\mathcal{F}}_{n}]+2\,h^{(2,\beta)}_{n+1}\,\widehat{A}_{n}\,\Delta
t+\frac{({\mathbb{E}}[h^{(1,\beta)}_{n+1}|{\mathcal{F}}_{n}]\,\gamma_{n}+2\,h^{(2,\beta)}_{n+1}\,\widehat{A}_{n}\,\gamma_{n}\,\beta)\,(2\,h^{(2,\beta)}_{n+1}\,\gamma_{n}+D_{n})}{2\,(K_{n}-h^{(2,\beta)}_{n+1}\,\gamma_{n}^{2}\,\beta)}\,\Delta
t\,,\\\ h^{(2,\beta)}_{n}&=h^{(2,\beta)}_{n+1}-C_{n}\,\Delta
t+\frac{(2\,h^{(2,\beta)}_{n+1}\,\gamma_{n}+D_{n})^{2}}{4\,(K_{n}-h^{(2,\beta)}_{n+1}\,\gamma_{n}^{2}\,\beta)}\,\Delta
t\,,\end{split}$ (59)

where the terminal conditions are still given by $h^{(1,\beta)}_{N}=0$ and
$h^{(2,\beta)}_{N}=-B$. When $\beta=\Delta t$, (59) coincides with (8). In
(59), the occurrences of $\Delta t$ correspond to a finite-difference method
approximating a derivative, and $\beta$ which has replaced other appearances
of $\Delta t$ is merely a parameter of this equation. For a fixed value of
$\beta$, the corresponding BSDE is given by

$\begin{split}dh^{(1,\beta)}_{t}&=-\biggl{(}2\,\widehat{A}_{t}\,h_{t}^{(2,\beta)}+\frac{(\gamma_{t}\,h_{t}^{(1,\beta)}+2\,h^{(2,\beta)}_{t}\,\widehat{A}_{t}\,\gamma_{t}\,\beta)\,(2\,\gamma_{t}\,h_{t}^{(2,\beta)}+D_{t})}{2\,(K_{t}-h^{(2,\beta)}_{t}\,\gamma_{t}^{2}\,\beta)}\biggr{)}\,dt+dZ_{t}\,,\\\
dh^{(2,\beta)}_{t}&=-\biggl{(}\frac{(2\,\gamma_{t}\,h_{t}^{(2,\beta)}+D_{t})^{2}}{4\,(K_{t}-h^{(2,\beta)}_{t}\,\gamma_{t}^{2}\,\beta)}-C_{t}\biggr{)}\,dt\,.\end{split}$
(60)

For a fixed value of $\beta>0$, (59) is an Euler scheme discretization of
(60), and as such we have convergence of the discrete-time solution to the
continuous-time solution (see for example Gobet and Labart (2007)). Treating
$(h^{(1,\beta)},h^{(2,\beta)})$ as a family of solutions depending on the
parameter $\beta$, there is convergence to $(h^{(1)},h^{(2)})$, the solution
of (57), as $\beta\rightarrow 0$ (see for example El Karoui et al. (1997)).

The implementation of the continuous-time solution in the following sections
requires a specification of a process $A$, and subsequently a computation of
the projection $\widehat{A}$. With the goal of relating the discrete-time
solution to the continuous-time solution, we use the continuous analogue of
(30) for the dynamics of $A$, namely a mean-reverting OU process satisfying

$\displaystyle dA_{t}$
$\displaystyle=-\kappa\,A_{t}\,dt+\eta\,dW^{(2)}_{t}\,,$ (61)

where $W^{(2)}$ is a Brownian motion independent of $W^{(1)}$ appearing in
(34), and with $A_{0}$ having Gaussian distribution with variance
$\Sigma_{0}$. By standard filtering results (see Theorem 12.7 in Liptser and
Shiryaev (2013)), the dynamics of the filtered process are given by

$\displaystyle d\widehat{A}_{t}$
$\displaystyle=-\kappa\,\widehat{A}_{t}\,dt+\frac{\Sigma_{t}}{\sigma^{2}}\,(dY_{t}-\widehat{A}_{t}\,dt)\,,$
$\displaystyle\widehat{A}_{0}$ $\displaystyle={\mathbb{E}}[A_{0}]\,,$ (62)

where $\Sigma_{t}$ represents the estimation error of $\widehat{A}$ and is
given by

$\begin{split}\Sigma_{t}&=\frac{(1+\alpha^{+}\,\Sigma_{0})\,e^{\xi\,t}-(1+\alpha^{-}\,\Sigma_{0})\,e^{-\xi\,t}}{-\alpha^{-}\,(1+\alpha^{+}\,\Sigma_{0})\,e^{\xi\,t}+\alpha^{+}\,(1+\alpha^{-}\,\Sigma_{0})\,e^{-\xi\,t}}\,,\\\\[10.00002pt]
\alpha^{\pm}&=\frac{-\sigma\,\kappa\pm\sqrt{\sigma^{2}\,\kappa^{2}+\eta^{2}}}{\sigma\,\eta^{2}}\,,\\\\[10.00002pt]
\xi&=\frac{\sqrt{\sigma^{2}\,\kappa^{2}+\eta^{2}}}{\sigma}\,.\end{split}$ (63)

With this specification of $A$ in mind, we may also write the solution of (57)
in closed form when the processes $C$, $D$, $K$, and $\gamma$ are constant.
The solution, which can be checked by direct substitution, is given by

$\begin{split}h^{(1)}_{t}&=\frac{\widehat{A}_{t}}{\gamma}\,\biggl{(}\frac{\psi^{-}\,(2\sqrt{C\,K}-D)\,(e^{-\kappa\,(T-t)}-e^{-\omega\,(T-t)})}{(\omega-\kappa)\,(\psi^{-}\,e^{-\omega\,(T-t)}+\psi^{+}\,e^{\,\omega\,(T-t)})}+\frac{\psi^{+}\,(2\sqrt{C\,K}+D)\,(e^{-\kappa\,(T-t)}-e^{\omega\,(T-t)})}{(\omega+\kappa)\,(\psi^{-}\,e^{-\omega\,(T-t)}+\psi^{+}\,e^{\,\omega\,(T-t)})}\biggr{)}\\\\[10.00002pt]
h^{(2)}_{t}&=\frac{\sqrt{C\,K}}{\gamma}\,\frac{\psi^{-}\,e^{-\omega\,(T-t)}-\psi^{+}\,e^{\,\omega\,(T-t)}}{\psi^{-}\,e^{-\omega\,(T-t)}+\psi^{+}\,e^{\,\omega\,(T-t)}}-\frac{D}{2\,\gamma}\\\\[10.00002pt]
\psi^{\pm}&=\sqrt{2\,C}\pm\sqrt{\frac{2}{K}}\,\gamma\,B\mp\frac{D}{\sqrt{2\,K}}\\\\[10.00002pt]
\omega&=\gamma\,\sqrt{\frac{C}{K}}\end{split}$ (64)

###  5.1 Discrete Convergence

Here, we discuss the behaviour of the discrete-time solution as $\Delta
t\rightarrow 0$ as it relates to the continuous-time solution. First we
inspect the continuous-time solution given in Proposition 3 and the dynamics
of $X^{\pi}$ given in (42), which reduce to

$\displaystyle
X_{t}^{\pi^{*}}=Y_{t}+\int_{0}^{t}\int_{-\infty}^{\infty}\gamma_{u}\,\nu\,\pi^{*}_{u}(\nu)\,d\nu\,du$
$\displaystyle=Y_{t}+\int_{0}^{t}\gamma_{u}\,\mu_{u}\,du$
$\displaystyle=Y_{t}+\int_{0}^{t}\gamma_{u}\,\nu_{u}^{*}\,du=X_{t}^{\nu^{*}}\,,$
(65)

where $\nu^{*}$ is the optimal classical control given in (41) and
$X^{\nu^{*}}$ is the resulting controlled process. Thus, when implementing the
optimal exploratory control in continuous-time, the resulting state process
coincides with that of the classical control problem.

varying $\lambda$

varying $q$

Figure 3: Convergence of discrete-time paths to continuous-time path. The top
row of panels uses $q=2$ and the bottom row uses $\lambda=0.5$. The left
column has $T/\Delta t=10$, the middle column $T/\Delta t=200$, and the right
column $T/\Delta t=10,000$. Other parameter values are $B=1$, $C=1$, $D=1$,
$K=0.1$, $\gamma=1$, $\sigma=0.2$, $\kappa=1$, $\eta=2$, $\widehat{A}_{0}=0$,
$\Sigma_{0}=1$. In each panel, the black curve represents the optimal state
path for the continuous-time problem (either the classical or exploratory
version which result in the same path).

In each panel of Figure 3, the black curve shows the optimal trajectory in
continuous-time for one particular outcome of the latent processes $A$ and
$W^{(1)}$. We also show trajectories in discrete-time when the control is
drawn from the optimal exploratory distribution in (23) for various values of
the exploration reward parameter, $\lambda$, and the Tsallis entropy
parameter, $q$. Each column of panels corresponds to a different granularity
of time discretization. We see that as $\Delta t\rightarrow 0$ the paths of
the state process under the optimal exploratory control approach the path of
the optimal state process in continuous-time. This effect is essentially due
to the law of large numbers applied to the sum of exploratory controls in
(11). As the number of terms in this sum approaches infinity, the randomness
attributed to the sequence of independent uniforms $\\{u_{n}\\}_{n=0}^{N-1}$
disappears. We are still left with an element of randomness contributed by the
conditional distributions $\\{F_{n}^{\pi}\\}_{n=0}^{N-1}$ which yields a
stochastic drift component to the process, as well as the Brownian innovations
inherent in the unaffected process $Y$. This figure also shows that for values
of the Tsallis entropy parameter $q$ close to the lower bound of $1/3$ the
fat-tailed behaviour of the optimal distribution can result in significant
deviations from the optimal standard path (blue curve in bottom left and
bottom middle panels), but for fixed $q$ this deviation will always disappear
as $\Delta t\rightarrow 0$.

###  5.2 Discrete Solution Approximation

In this section we use the solution of (57) in continuous-time to act as an
approximation of the corresponding processes which appear in the discrete-time
optimal exploratory control of Proposition 1. The complicated form of (8)
along with it’s discrete nature means numerical methods must be used to solve
the backward system, as was done to demonstrate its implementation in Section
3. As seen in (64), the solution (57) may be solved in closed form, thus
better lending itself for more efficient implementation.

To this end, let $\tilde{\pi}=\\{\pi_{n}\\}_{n=0}^{N-1}$ be a sequence of
conditional densities given by (23), except with all occurrences of
$h^{(1)}_{n}$ and $h^{(2)}_{n}$ replaced with $h^{(1)}_{t_{n}}$ and
$h^{(2)}_{t_{n}}$ respectively333We have abused notation by referring to
solutions to both the BS$\Delta$E and BSDE by $h^{(i)}$, but it should be
clear from context that $h^{(i)}_{t_{n}}$ refers to the solution of the BSDE
evaluated at time $t_{n}=n\,\Delta t$, whereas $h^{(i)}_{n}$ is the solution
of the BS$\Delta$E evaluated at the $n^{th}$ step.. In Figure 4 we show paths
resulting from the optimal control density along with this approximation.
Unsurprisingly, the error between these processes reduces as $\Delta t$
decreases. Moreover, the error is typically smallest towards the endpoints of
the time interval $[0,T]$. Both the optimal and approximate $h^{(i)}$
processes have the same terminal condition, and errors in computing an
approximation to the optimal control accumulate as we move backwards in time,
hence the largest discrepancies in the control process occur near $t=0$. Since
the state process begins at the same constant regardless of whether an
approximation is being used or not, the error increases from zero rapidly near
$t=0$. However, the performance criterion having a terminal reward of the form
$-B\,(X^{\pi}_{N})^{2}$ implies that as time approaches $T$ the controlled
state process has a soft target of $0$, regardless of whether the implemented
control is optimal or an approximation. As both the optimal process and the
approximation have the same target, they tend to come closer together again as
the end of the horizon approaches.

Figure 4: Approximation of discrete-time solution using closed form
expressions relevant to continuous-time solution. In the top row,
discretization is given by $T/\Delta t=10$, and in the bottom row it is
$T/\Delta t=100$. The left column shows a realization of both the optimal
discrete-time strategy $X_{n}^{\pi^{*}}$ (solid curves) and the approximation
$X_{n}^{\tilde{\pi}}$ (dotted curves) resulting from implementing a strategy
with $h^{(i)}_{n}$ replaced by $h^{(i)}_{t_{n}}$ in Proposition 1. The right
column shows the difference between these processes. Other parameter values
are $B=1$, $C=1$, $D=1$, $K=0.1$, $\gamma=1$, $\sigma=0.2$, $\kappa=1$,
$\eta=2$, $\widehat{A}_{0}=0$, $\Sigma_{0}=1$, $q=1.2$.

##  6 Extension to $Q$-Learning

In this section, we adapt the Tsallis entropy exploration reward to a model-
agnostic reinforcement learning setting – the so-called $Q$-learning paradigm.
As such, this section adopts notation commonly used with Markov decision
processes (MDPs) and $Q$-learning. For an overview see, e.g., Jaimungal
(2022).

In a similar fashion to Sections 2.2 and 4.2, we add a Tsallis entropy term to
the classical agent’s performance criterion to incentivize exploration within
the $Q$-learning framework. In this section, we work on discrete state and
actions space denoted ${\mathcal{X}}$ and ${\mathcal{A}}$, respectively.
Moreover, the agent chooses the distribution over actions conditional on
state, denoted $\pi(\nu|X_{n})$ – here, as in soft $Q$-learning (See e.g.,
Ziebart (2010); Fox et al. (2015)), we call $\pi$ the policy. The agent’s
Tsallis regularized performance criterion is defined as

$\displaystyle V^{\pi}(x)$
$\displaystyle={\mathbb{E}}^{{\mathbb{P}}_{\pi}}\left[\left.\sum_{n=1}^{\infty}\zeta^{n-1}\,(r_{n}+\lambda\,S_{q}[\pi_{n}(\cdot|X_{n})])\,\right|\,X_{1}=x\,\right],$

where $\lambda>0$ is the exploration reward parameter, $\zeta\in(0,1)$ is a
discount factor, $S_{q}$ is the discrete analogue of the Tsallis entropy in
Equation (18) given by

$\displaystyle S_{q}\left[\pi(\cdot|x)\right]$
$\displaystyle=\left\\{\begin{array}[]{cc}\displaystyle\frac{1}{q-1}\biggl{(}1-\sum_{\nu\in{\mathcal{A}}}\left(\pi(\nu|x)\right)^{q}\biggr{)}\,,&q>0,\,q\neq
1\\\\[20.00003pt]
\displaystyle-\sum_{\nu\in{\mathcal{A}}}\pi_{n}(\nu|x)\,\log(\pi(\nu|x))\,,&q=1,\end{array}\right.$
(68)

and the probability measure ${\mathbb{P}}_{\pi}$ is the one induced by the
state transition probabilities
${\mathbb{P}}_{\pi}(X_{n+1}=x^{\prime}\,|\,X=x)=\sum_{a\in{\mathcal{A}}}{\mathbb{P}}(X_{n+1}=x^{\prime}\,|\,\nu=a,X=x)\pi(a\,|\,x)$.
Defining the Tsallis regularized value function
$V^{*}(x):=\sup_{\pi(\cdot|x)\in{\mathcal{P}}({\mathcal{A}})}V^{\pi}(x)$,
where ${\mathcal{P}}({\mathcal{A}})$ denotes the space of probability measure
on ${\mathcal{A}}$, the Tsallis regularized value function also satisfies a
DPP which now takes the form

$\displaystyle V^{*}(x)$
$\displaystyle=\max_{\pi}{\mathbb{E}}^{{\mathbb{P}}_{\pi}}\biggl{[}r(x,X^{\prime},\nu)+\zeta\,V^{*}(X^{\prime})+\lambda\,S_{q}[\pi(\cdot|x)]\biggr{]}\,,$

where $(X^{\prime},\nu)$ has distribution
${\mathbb{P}}_{\pi}(X^{\prime}=x^{\prime},\nu=a)={\mathbb{P}}(X_{n+1}=x^{\prime}|\nu=a,X=x)\pi(a\,|\,x)$.
Next, define the state-policy free energy function, also called the soft
$Q$-function or $G$ function, as the value of taking an arbitrary policy
(distribution over actions) $\pi\in{\mathcal{P}}$, for the current action, and
then subsequently follow the, as yet unknown, optimal policy $\pi^{*}$. Then,
the DPP implies that

$\displaystyle Q(x;\pi):=$
$\displaystyle\,{\mathbb{E}}^{{\mathbb{P}}_{\pi}}\biggl{[}r(x,X^{\prime},\nu)+\zeta\,V^{*}(X^{\prime})+\lambda\,S_{q}[\pi(\cdot|x)]\biggr{]}$
(69) $\displaystyle=$
$\displaystyle\,{\mathbb{E}}^{{\mathbb{P}}_{\pi}}\biggl{[}r(x,X^{\prime},\nu)+\zeta\,\max_{\pi^{\prime}\in{\mathcal{P}}}Q^{*}(X^{\prime};\pi^{\prime})+\lambda\,S_{q}[\pi(\cdot|x)]\biggr{]}\,,$

As the $Q$-function is defined for an arbitrary $\pi\in{\mathcal{P}}$, it may
be evaluated on a point mass $\pi=\delta_{a}$ at a particular action
$a\in{\mathcal{A}}$. Evaluating the $Q$-function with this particular choice
leads to the classical (no exploration) $Q$-function and such a choice will be
useful for the representation of the optimal exploratory strategy below.

For $q\neq 1$, expanding the expectation in $\eqref{eqn:Q_explore}$ yields

$\displaystyle Q(x;\pi)$
$\displaystyle=\sum_{x^{\prime}\in{\mathcal{X}}}\sum_{\nu\in{\mathcal{A}}}\biggl{(}r(x,x^{\prime},\nu)+\zeta\,V^{*}(x^{\prime})\biggr{)}\,{\mathbb{P}}(x^{\prime}|x,\nu)\,\pi(\nu|x)+\frac{\lambda}{q-1}\biggl{(}1-\sum_{\nu\in{\mathcal{A}}}(\pi(\nu|x))^{q}\biggr{)}\,.$
(70)

Evaluating (70) at $\pi=\delta_{a}$ for a particular $a\in{\mathcal{A}}$ gives

$\displaystyle Q(x;\delta_{a})$
$\displaystyle=\sum_{x^{\prime}}\biggl{(}r(x,x^{\prime},a)+\zeta\,V^{*}(x^{\prime})\biggr{)}\,{\mathbb{P}}(x^{\prime}|x,a)\,,$
(71)

thus (70) may be rewritten as

$\displaystyle Q(x;\pi)$
$\displaystyle=\sum_{\nu\in{\mathcal{A}}}Q(x;\delta_{\nu})\,\pi(\nu|x)+\frac{\lambda}{q-1}\biggl{(}1-\sum_{\nu\in{\mathcal{A}}}(\pi(\nu|x))^{q}\biggr{)}\,.$
(72)

Maximizing $Q(x;\pi)$ over $\pi$, similar to the proofs of Propositions 1 and
3, gives the optimal distribution as

$\displaystyle\pi^{*}(\nu|x)$
$\displaystyle=\left\\{\begin{array}[]{cc}\biggl{(}\frac{q-1}{\lambda\,q}\biggr{)}^{\frac{1}{q-1}}\,\biggl{[}\psi(x)+Q(x;\delta_{\nu})\biggr{]}^{\frac{1}{q-1}}_{+}\,,&q>1,\\\\[10.00002pt]
\psi(x)\,\exp\biggl{\\{}\frac{Q(x;\delta_{\nu})}{\lambda}\biggr{\\}}\,,&q=1,\\\\[10.00002pt]
\biggl{(}\frac{1-q}{\lambda\,q}\biggr{)}^{\frac{1}{q-1}}\,\biggl{[}\psi(x)-Q(x;\delta_{\nu})\biggr{]}^{\frac{1}{q-1}}\,,&0<q<1,\end{array}\right.$
(76)

where $\psi(x)$ is a constant (for each $x\in{\mathcal{X}}$) chosen so that
$\sum_{\nu\in{\mathcal{A}}}\pi^{*}(\nu|x)=1$. The case $q=1$ reduces to the
so-called “Boltzman” policies obtained in Fox et al. (2015).

From the form of $\pi^{*}$ in (76) we see that the mode of the optimal policy
is the value of $\nu$ which maximizes $Q(x;\delta_{\nu})$, similar to the
result in Propositions 1 and 3 where the location parameter of $\pi^{*}$ has
the same feedback form as the optimal control $\nu^{*}$ of the standard
problem without exploration. Further, for $q>1$, some actions may be assigned
zero probability by $\pi^{*}$, similar to the behaviour seen with $q>1$ in
equations (23) and (54) where the optimal distribution has compact support.
Unlike the result in those equations, however, the exclusion of some actions
is not guaranteed. If the values of $Q(x;\delta_{\nu})$ do not vary
significantly with respect to $\nu$ or if $\lambda$ is relatively large, then
the policy in (76) will assign positive probability to each action. When $q<1$
all actions are assigned strictly positive probability, similar to (23) and
(54) where the optimal policy has support on all of $\mathbb{R}$.

This optimal strategy may be used to develop a iterative $Q$-learning paradigm
along the lines of Todorov and Li (2005) (see also the discussion on iLQG in
Jaimungal (2022)) but is outside the scope of this paper.

##  7 Conclusions

We developed a grounded approach for exploratory controls in discrete-time
with latent factors, and illuminated the need to carefully specify the
information structure in specifying randomized controls. We also introduced
Tsallis entropy as a reward for exploration so that the agent has the ability
to further tailor the distribution of their randomized actions, generalizing
beyond the Gaussian distribution to the class of $q$-Gaussian distributions.
In particular, the agent may restrict their randomized actions to a compact
set to cap the riskiness associated with a single action or may implement a
distribution with fat tails to exhibit more exploration, depending on the
specific problem setting. A comparison of the optimal exploratory control in
discrete-time with that of an analogous model in continuous-time shows that
the more tractable continuous-time solution may be used as an approximation to
the more realistic discrete-time formulation. Finally, we show how these
techniques may be adapted in the context of $Q$-learning so that the
implemented actions employ more reasonable exploration rather than being
maximally greedy.

Future work includes using our framework in a continual learning setting,
where an agent first makes an approximation of environment that falls within
our assumptions, then explores using our optimal control, and then updates the
model with the new information they have received from the environment, re-
optimise using our optimal exploratory control, and repeat. Further directions
left open include how to generalize these results to the multi-agent case
and/or mean-field game settings. As well, as applying the approach in real
world settings on data.

## Appendix

###  A Proof of Proposition 1

The dynamic programming principle for $H^{(\lambda)}$ in (19) takes the form

$\displaystyle H^{(\lambda)}_{n}$
$\displaystyle=\sup_{\pi_{n}\in\mathcal{A}^{r}_{n}}{\mathbb{E}}\biggl{[}H^{(\lambda)}_{n+1}+(D_{n}\,X_{n}\,\nu_{n}-C_{n}\,X_{n}^{2}-K_{n}\,\nu_{n}^{2})\,\Delta
t+\lambda\,S_{q}[\pi_{n}]\,\Delta t\biggl{|}{\mathcal{F}}^{u}_{n}\biggr{]}\,,$
(77)

where we have suppressed dependence of each process on $\pi_{n}$ to simplify
notation. We make the ansatz
$H^{(\lambda)}_{n}=h^{(0)}_{n}+h^{(1)}_{n}\,X_{n}+h^{(2)}_{n}\,X_{n}^{2}$
where $h^{(0)}$ and $h^{(1)}$ are ${\mathcal{F}}_{n}$-adapted (the filtration
generated by $Y$) and $h^{(2)}$ is deterministic. Substituting this for
$H^{(\lambda)}$ on both sides of the dynamic programming principle, using
(11), (13), and (14), and recalling that the sequence
$\\{u_{n}\\}_{n=0}^{N-1}$ is independent of
$\\{{\mathcal{F}}_{n}\\}_{n=0}^{N}$ yields

$\displaystyle h^{(0)}_{n}+h^{(1)}_{n}\,X_{n}+h^{(2)}_{n}\,X_{n}^{2}$
$\displaystyle\hskip
28.45274pt={\mathbb{E}}[h^{(0)}_{n+1}|{\mathcal{F}}^{n}]+{\mathbb{E}}[h^{(1)}_{n+1}|{\mathcal{F}}_{n}]\,(X_{n}+\widehat{A}_{n}\Delta
t)+{\mathbb{E}}[h^{(1)}_{n+1}\,\Delta\widehat{M}_{n}|{\mathcal{F}}_{n}]$
$\displaystyle\hskip
42.67912pt+h^{(2)}_{n+1}\,(X_{n}^{2}+\widehat{A}_{n}^{2}\,(\Delta
t)^{2}+\widehat{\sigma}^{2}_{n}\,\Delta t+2\,X_{n}\,\widehat{A}_{n}\,\Delta
t)-C_{n}\,X_{n}^{2}\,\Delta t$ $\displaystyle\hskip
42.67912pt+\sup_{\pi_{n}\in\mathcal{A}^{r}_{n}}\biggl{\\{}\biggl{(}{\mathbb{E}}[h^{(1)}_{n+1}|{\mathcal{F}}_{n}]\,\gamma_{n}+2\,h^{(2)}_{n+1}\,\widehat{A}_{n}\,\gamma_{n}\,\Delta
t+(2\,h^{(2)}_{n+1}\,\gamma_{n}+D_{n})\,X_{n}\biggr{)}\int_{-\infty}^{\infty}\nu\,\pi_{n}(\nu)\,d\nu$
$\displaystyle\hskip 42.67912pt-(K_{n}-h^{(2)}_{n+1}\,\gamma_{n}^{2}\,\Delta
t)\int_{-\infty}^{\infty}\nu^{2}\,\pi_{n}(\nu)\,d\nu+\lambda\,S_{q}[\pi_{n}]\biggr{\\}}\,\Delta
t\,,$

where $\widehat{\sigma}^{2}_{n}\,\Delta t={\mathbb{E}}[(\Delta
M_{n})^{2}|{\mathcal{F}}_{n}]$. By completing the square of the terms which
depend on $\nu$ within integrals, this can be rewritten as

$\begin{split}&h^{(0)}_{n}+h^{(1)}_{n}\,X_{n}+h^{(2)}_{n}\,X_{n}^{2}\\\
&\hskip
42.67912pt={\mathbb{E}}[h^{(0)}_{n+1}|{\mathcal{F}}_{n}]+{\mathbb{E}}[h^{(1)}_{n+1}|{\mathcal{F}}_{n}]\,(X_{n}+\widehat{A}_{n}\Delta
t)+{\mathbb{E}}[h^{(1)}_{n+1}\,\Delta\widehat{M}_{n}|{\mathcal{F}}_{n}]\\\
&\hskip 56.9055pt+h^{(2)}_{n+1}\,(X_{n}^{2}+\widehat{A}_{n}^{2}\,(\Delta
t)^{2}+\widehat{\sigma}^{2}_{n}\,\Delta t+2\,X_{n}\,\widehat{A}_{n}\,\Delta
t)-C_{n}\,X_{n}^{2}\,\Delta t+\frac{L_{n}^{2}}{4\,Q_{n}}\,\Delta t\\\ &\hskip
56.9055pt+\sup_{\pi_{n}\in\mathcal{A}^{r}_{n}}\biggl{\\{}\int_{-\infty}^{\infty}-Q_{n}\,\biggl{(}\nu-\frac{L_{n}}{2\,Q_{n}}\biggr{)}^{2}\,\pi_{n}(\nu)\,d\nu+\lambda\,S_{q}[\pi_{n}]\biggr{\\}}\,\Delta
t\,,\end{split}$ (78)

where

$\displaystyle L_{n}$
$\displaystyle={\mathbb{E}}[h^{(1)}_{n+1}|{\mathcal{F}}_{n}]\,\gamma_{n}+2\,h^{(2)}_{n+1}\,\widehat{A}_{n}\,\gamma_{n}\,\Delta
t+(2\,h^{(2)}_{n+1}\,\gamma_{n}+D_{n})\,X_{n}$ $\displaystyle Q_{n}$
$\displaystyle=K_{n}-h^{(2)}_{n+1}\,\gamma_{n}^{2}\,\Delta t\,.$

We provide the details of the maximization with respect to $\pi_{n}$ for
$q>1$, the case of $1/3<q\leq 1$ is essentially the same. We introduce a
scalar Lagrange multiplier $\psi_{n}$ to enforce the constraint
$\int_{-\infty}^{\infty}\pi_{n}(\nu)\,d\nu=1$, and a KKT multiplier (which is
a function of $\nu$) $\zeta_{n}$ to enforce the constraint $\pi_{n}(\nu)\geq
0$. The corresponding Lagrangian is

$\displaystyle\mathcal{L}(\pi_{n};\psi_{n},\zeta_{n})$
$\displaystyle=\int_{-\infty}^{\infty}-Q_{n}\,\biggl{(}\nu-\frac{L_{n}}{2\,Q_{n}}\biggr{)}^{2}\,\pi_{n}(\nu)\,d\nu+\frac{\lambda}{q-1}\biggl{(}1-\int_{-\infty}^{\infty}(\pi_{n}(\nu))^{q}\,d\nu\biggr{)}$
$\displaystyle\hskip
42.67912pt+\psi_{n}\biggl{(}\int_{-\infty}^{\infty}\pi_{n}(\nu)\,d\nu-1\biggr{)}+\int_{-\infty}^{\infty}\zeta_{n}(\nu)\,\pi_{n}(\nu)\,d\nu\,.$

Computing first order conditions for $\pi_{n}$ and then solving for
$\pi_{n}(\nu)$ with the non-negativity restriction in place yields

$\displaystyle\pi^{*}_{n}(\nu)$
$\displaystyle=\biggl{(}\frac{(q-1)\,\psi_{n}}{\lambda\,q}\biggr{)}^{\frac{1}{q-1}}\,\biggl{(}1-\frac{Q_{n}}{\psi_{n}}\,\biggl{(}\nu-\frac{L_{n}}{2\,Q_{n}}\biggr{)}^{2}\biggr{)}_{+}^{\frac{1}{q-1}}\,,$
(79)

and the KKT multiplier is given by

$\displaystyle\zeta_{n}(\nu)$
$\displaystyle=\biggl{(}Q_{n}\,(\nu-\frac{L_{n}}{2\,Q_{n}})^{2}-\psi_{n}\biggr{)}_{+}\,.$
(80)

The value of the Lagrange multiplier $\psi_{n}$ is chosen so that the integral
of the expression in (79) is equal to $1$. A simple substitution allows this
to be written as

$\displaystyle\int_{-\infty}^{\infty}\pi_{n}(\nu)\,d\nu$
$\displaystyle=\biggl{(}\frac{(q-1)}{\lambda\,q}\biggr{)}^{\frac{1}{q-1}}\,\frac{\psi_{n}^{\frac{1}{q-1}+\frac{1}{2}}}{\sqrt{Q_{n}}}\,\int_{-1}^{1}(1-x^{2})^{\frac{1}{q-1}}\,dx$
$\displaystyle 1$
$\displaystyle=\biggl{(}\frac{(q-1)}{\lambda\,q}\biggr{)}^{\frac{1}{q-1}}\,\frac{\psi_{n}^{\frac{1}{q-1}+\frac{1}{2}}}{\sqrt{Q_{n}}}\,\frac{\sqrt{\pi}\,\Gamma(\frac{1}{q-1}+1)}{\Gamma(\frac{1}{q-1}+\frac{3}{2})}\,,$

where this integral can be found in Gradshteyn and Ryzhik (2014) Section
3.251. Solving for $\psi_{n}$ gives

$\displaystyle\psi_{n}$
$\displaystyle=\Biggl{[}\frac{\Gamma(\frac{1}{q-1}+\frac{3}{2})}{\sqrt{\pi}\,\Gamma(\frac{1}{q-1}+1)}\biggl{(}\frac{\lambda\,q}{q-1}\biggr{)}^{\frac{1}{q-1}}\sqrt{K_{n}-h^{(2)}_{n+1}\,\gamma_{n}^{2}\,\Delta
t}\Biggr{]}^{\frac{1}{\frac{1}{q-1}+\frac{1}{2}}}\,,$

as in the statement of the proposition. Substituting the maximizer
$\pi^{*}_{n}$ from (79) into the supremum term in (78) and evaluating the
integrals using Gradshteyn and Ryzhik (2014) Section 3.251 gives

$\begin{split}&\sup_{\pi_{n}\in\mathcal{A}^{r}_{n}}\biggl{\\{}\int_{-\infty}^{\infty}-Q_{n}\,\biggl{(}\nu-\frac{L_{n}}{2\,Q_{n}}\biggr{)}^{2}\,\pi_{n}(\nu)\,d\nu+\lambda\,S_{q}[\pi_{n}]\biggr{\\}}\\\
&\hskip
42.67912pt=\frac{\lambda}{q-1}-\biggl{(}\frac{q-1}{\lambda\,q}\biggr{)}^{\frac{1}{q-1}}\frac{\psi_{n}^{\frac{1}{q-1}+\frac{3}{2}}\,\sqrt{\pi}}{\sqrt{Q_{n}}\,\Gamma(\frac{1}{q-1}+\frac{5}{2})}\biggl{(}\frac{1}{2}\,\Gamma\biggl{(}\frac{1}{q-1}+1\biggr{)}+\frac{q-1}{\lambda\,q}\,\Gamma\biggl{(}\frac{1}{q-1}+2\biggr{)}\biggr{)}\,.\end{split}$

Note that the above expression does not depend on $X_{n}$, so by grouping
terms in (78) by powers of $X_{n}$ we are left with the recursion equations

$\displaystyle h^{(0)}_{n}$
$\displaystyle={\mathbb{E}}[h^{(0)}_{n+1}|{\mathcal{F}}_{n}]+{\mathbb{E}}[h^{(1)}_{n+1}|{\mathcal{F}}_{n}]\,\widehat{A}_{n}\,\Delta
t+{\mathbb{E}}[h_{n+1}^{(1)}\,\Delta\widehat{M}_{n}|{\mathcal{F}}_{n}]$
$\displaystyle\hskip
19.91692pt+h^{(2)}_{n+1}\,(\widehat{\sigma}_{n}^{2}+\widehat{A}_{n}^{2}\,\Delta
t)\,\Delta
t+\frac{({\mathbb{E}}[h^{(1)}_{n+1}|{\mathcal{F}}_{n}]\,\gamma_{n}+2\,h^{(2)}_{n+1}\,\widehat{A}_{n}\,\gamma_{n}\,\Delta
t)^{2}}{4\,(K_{n}-h^{(2)}_{n+1}\,\gamma_{n}^{2}\,\Delta t)}\,\Delta t$
$\displaystyle\hskip 39.83385pt+\lambda\,p\,\Delta
t-\biggl{(}p+\frac{1}{2}\biggr{)}\,\biggl{(}\frac{q-1}{\lambda\,q}\biggr{)}^{p}\frac{\sqrt{\pi}\,\psi_{n}^{p+\frac{3}{2}}}{\sqrt{K_{n}-h^{(2)}_{n+1}\,\gamma_{n}^{2}\,\Delta
t}}\frac{\Gamma(p+1)}{\Gamma(p+\frac{5}{2})}\,\Delta t\,,$ $\displaystyle
h^{(1)}_{n}$
$\displaystyle={\mathbb{E}}[h^{(1)}_{n+1}|{\mathcal{F}}_{n}]+2\,h^{(2)}_{n+1}\,\widehat{A}_{n}\,\Delta
t+\frac{({\mathbb{E}}[h^{(1)}_{n+1}|{\mathcal{F}}_{n}]\,\gamma_{n}+2\,h^{(2)}_{n+1}\,\widehat{A}_{n}\,\gamma_{n}\,\Delta
t)(2\,h^{(2)}_{n+1}\,\gamma_{n}+D_{n})}{2\,(K_{n}-h^{(2)}_{n+1}\,\gamma_{n}^{2}\,\Delta
t)}\Delta t\,,$ $\displaystyle h^{(2)}_{n}$
$\displaystyle=h^{(2)}_{n+1}-C_{n}\,\Delta
t+\frac{(2\,h^{(2)}_{n+1}\,\gamma_{n}+D_{n})^{2}}{4\,(K_{n}-h^{(2)}_{n+1}\,\gamma_{n}^{2}\,\Delta
t)}\,\Delta t\,,$

where $p=\frac{1}{q-1}$ and the terminal conditions are given by
$h^{(0)}_{N}=h^{(1)}_{N}=0$ and $h^{(2)}_{N}=-B$. The optimizer $\pi^{*}_{n}$
does not depend on the process $h^{(0)}$, so we require only the recursion
equations for $h^{(1)}$ and $h^{(2)}$, which are identical to those in
equation (8) as in the statement of the proposition.

###  B Proof of Proposition 3

The HJB equation with terminal condition associated with the dynamic value
function $H^{(\lambda)}$ is

$\begin{split}\partial_{t}H^{(\lambda)}+\mathcal{L}^{\widehat{A}}H^{(\lambda)}+\widehat{A}\,\partial_{x}H^{(\lambda)}+\frac{1}{2}\,\sigma^{2}\,\partial_{xx}H^{(\lambda)}-C_{t}\,x^{2}\hskip
128.0374pt\\\
+\sup_{\pi_{t}\in\mathcal{A}^{r}_{t}}\biggl{\\{}(\gamma_{t}\,\partial_{x}H^{(\lambda)}+D_{t}\,x)\int_{-\infty}^{\infty}\nu\,\pi_{t}(\nu)\,d\nu-
K_{t}\int_{-\infty}^{\infty}\nu^{2}\,\pi_{t}(\nu)\,d\nu+\lambda\,S_{q}[\pi_{t}]\biggr{\\}}&=0\,,\\\
H^{(\lambda)}(T,x,\widehat{A})&=g(x)\,.\end{split}$ (81)

The maximization with respect to $\pi_{t}$ is essentially identical to that in
the proof of Proposition 1, and the resulting optimal density in feedback form
is

$\displaystyle\pi_{t}(\nu)$
$\displaystyle=\left\\{\begin{array}[]{cc}\biggl{(}\frac{q-1}{\lambda\,q}\biggr{)}^{\frac{1}{q-1}}\biggl{(}\psi_{t}-\frac{\lambda}{2\,\varsigma_{t}^{2}}\,(\nu-\mu(t,x,\widehat{A}))^{2}\biggr{)}_{+}^{\frac{1}{q-1}}\,,&q>1\\\
\frac{1}{\sqrt{2\,\pi\,\varsigma_{t}^{2}}}\,\exp\biggl{\\{}-\frac{(\nu-\mu(t,x,\widehat{A})^{2}}{2\,\varsigma_{t}^{2}}\biggr{\\}}\,,&q=1\\\
\biggl{(}\frac{1-q}{\lambda\,q}\biggr{)}^{\frac{1}{q-1}}\biggl{(}\psi_{n}+\frac{\lambda}{2\,\varsigma_{t}^{2}}\,(\nu-\mu(t,x,\widehat{A}))^{2}\biggr{)}^{\frac{1}{q-1}}\,,&1/3<q<1\\\
\end{array}\right.\,,$ (85)

where

$\displaystyle\mu(t,x,\widehat{A})$
$\displaystyle=\frac{\gamma_{t}\,\partial_{x}H^{(\lambda)}(t,x,\widehat{A})+D_{t}\,x}{2\,K_{t}}\,,$
(86) $\displaystyle\varsigma_{t}^{2}$
$\displaystyle=\frac{\lambda}{2\,K_{t}}\,,$ (87) $\displaystyle\psi_{t}$
$\displaystyle=\left\\{\begin{array}[]{cc}\Biggl{[}\frac{\Gamma(\frac{1}{q-1}+\frac{3}{2})}{\sqrt{\pi}\,\Gamma(\frac{1}{q-1}+1)}\biggl{(}\frac{\lambda\,q}{q-1}\biggr{)}^{\frac{1}{q-1}}\sqrt{K_{t}}\Biggr{]}^{\frac{1}{\frac{1}{q-1}+\frac{1}{2}}}\,,&q>1\\\
\Biggl{[}\frac{\Gamma(\frac{1}{1-q})}{\sqrt{\pi}\,\Gamma(\frac{1}{1-q}-\frac{1}{2})}\biggl{(}\frac{\lambda\,q}{1-q}\biggr{)}^{\frac{1}{q-1}}\sqrt{K_{t}}\Biggr{]}^{\frac{1}{\frac{1}{q-1}+\frac{1}{2}}}\,,&1/3<q<1\end{array}\right.\,.$
(90)

Resubstituting the optimal density from (85) into (81) gives

$\begin{split}\partial_{t}H^{(\lambda)}+\mathcal{L}^{\widehat{A}}H^{(\lambda)}+\widehat{A}\,\partial_{x}H^{(\lambda)}+\frac{1}{2}\,\sigma^{2}\,\partial_{xx}H^{(\lambda)}-C_{t}\,x^{2}+\frac{(\gamma_{t}\,\partial_{x}H^{(\lambda)}+D_{t}\,x)^{2}}{4\,K_{t}}-\alpha_{q}&=0\,,\\\
H^{(\lambda)}(T,x,\widehat{A})&=g(x)\,,\end{split}$ (91)

where $\alpha_{q}$ is a function only of $t$ given by

$\displaystyle\alpha_{q}(t)$
$\displaystyle=\left\\{\begin{array}[]{cc}-\frac{\lambda}{q-1}+\frac{\psi_{t}\,(q+1)}{3\,q-1}\,,&q>1\\\
\lambda\,\log(\sqrt{\frac{\pi\,\lambda}{K_{t}}})\,,&q=1\\\
-\frac{\lambda}{q-1}-\frac{\psi_{t}\,(q+1)}{3\,q-1}\,,&1/3<q<1\\\
\end{array}\right.\,.$ (95)

Equation (91) and its terminal condition is equivalent to (40) except for the
appearance of the term $\alpha_{q}$, and so we have

$\displaystyle H^{(\lambda)}(t,x,\widehat{A})$
$\displaystyle=H^{(0)}(t,x,\widehat{A})-\int_{t}^{T}\alpha_{q}(u)\,du\,,$

and the feedback form of the term $\mu$ in (86) in the optimal density reduces
to that in the statement of the proposition.

## References

  * El Karoui et al. (1997) El Karoui, N., S. Peng, and M. C. Quenez (1997).  Backward stochastic differential equations in finance.  Mathematical finance 7(1), 1–71.
  * Firoozi and Jaimungal (2022) Firoozi, D. and S. Jaimungal (2022).  Exploratory lqg mean field games with entropy regularization.  Automatica 139, 110177.
  * Fox et al. (2015) Fox, R., A. Pakman, and N. Tishby (2015).  Taming the noise in reinforcement learning via soft updates.  arXiv preprint arXiv:1512.08562.
  * Fujisaki et al. (1972) Fujisaki, M., G. Kallianpur, and H. Kunita (1972).  Stochastic differential equations for the non linear filtering problem.  Osaka Journal of Mathematics 9(1), 19–40.
  * Gao et al. (2022) Gao, X., Z. Q. Xu, and X. Y. Zhou (2022).  State-dependent temperature control for langevin diffusions.  SIAM Journal on Control and Optimization 60(3), 1250–1268.
  * Geist et al. (2019) Geist, M., B. Scherrer, and O. Pietquin (2019).  A theory of regularized markov decision processes.  In International Conference on Machine Learning, pp. 2160–2169. PMLR.
  * Gobet and Labart (2007) Gobet, E. and C. Labart (2007).  Error expansion for the discretization of backward stochastic differential equations.  Stochastic processes and their applications 117(7), 803–829.
  * Gradshteyn and Ryzhik (2014) Gradshteyn, I. S. and I. M. Ryzhik (2014).  Table of integrals, series, and products.  Academic press.
  * Guan et al. (2020) Guan, Y., Q. Zhang, and P. Tsiotras (2020).  Learning nash equilibria in zero-sum stochastic games via entropy-regularized policy approximation.  arXiv preprint arXiv:2009.00162.
  * Guo et al. (2022) Guo, X., R. Xu, and T. Zariphopoulou (2022).  Entropy regularization for mean field games with learning.  Mathematics of Operations Research.
  * Hao et al. (2022) Hao, D., D. Zhang, Q. Shi, and K. Li (2022).  Entropy regularized actor-critic based multi-agent deep reinforcement learning for stochastic games.  Information Sciences.
  * Jaimungal (2022) Jaimungal, S. (2022).  Reinforcement learning and stochastic optimisation.  Finance and Stochastics 26(1), 103–129.
  * Liptser and Shiryaev (2013) Liptser, R. and A. N. Shiryaev (2013).  Statistics of Random Processes: II. Applications, Volume 6.  Springer Science & Business Media.
  * Neu et al. (2017) Neu, G., A. Jonsson, and V. Gómez (2017).  A unified view of entropy-regularized markov decision processes.  arXiv preprint arXiv:1705.07798.
  * Savas et al. (2019) Savas, Y., M. Ahmadi, T. Tanaka, and U. Topcu (2019).  Entropy-regularized stochastic games.  In 2019 IEEE 58th Conference on Decision and Control (CDC), pp. 5955–5962. IEEE.
  * Tang et al. (2021) Tang, W., P. Y. Zhang, and X. Y. Zhou (2021).  Exploratory hjb equations and their convergence.  arXiv preprint arXiv:2109.10269.
  * Thistleton et al. (2007) Thistleton, W. J., J. A. Marsh, K. Nelson, and C. Tsallis (2007).  Generalized box–müller method for generating $q$-gaussian random deviates.  IEEE transactions on information theory 53(12), 4805–4810.
  * Todorov and Li (2005) Todorov, E. and W. Li (2005).  A generalized iterative lqg method for locally-optimal feedback control of constrained nonlinear stochastic systems.  In Proceedings of the 2005, American Control Conference, 2005., pp. 300–306. IEEE.
  * Tsallis (1988) Tsallis, C. (1988).  Possible generalization of boltzmann-gibbs statistics.  Journal of statistical physics 52(1), 479–487.
  * Tsallis (2011) Tsallis, C. (2011).  The nonadditive entropy sq and its applications in physics and elsewhere: Some remarks.  Entropy 13(10), 1765–1804.
  * Wang et al. (2020) Wang, H., T. Zariphopoulou, and X. Y. Zhou (2020).  Reinforcement learning in continuous time and space: A stochastic control approach.  J. Mach. Learn. Res. 21(198), 1–34.
  * Ziebart (2010) Ziebart, B. D. (2010).  Modeling purposeful adaptive behavior with the principle of maximum causal entropy.  Carnegie Mellon University.