# An Approach to Duality in Nonlinear Filtering

Jin-Won Kim, Amirhossein Taghvaei, Prashant G. Mehta and Sean P. Meyn
Financial support from the ARO grant W911NF1810334 and the NSF CMMI award
1761622 is gratefully acknowledged. J-W. Kim, A. Taghvaei and P. G. Mehta are
with the Coordinated Science Laboratory and the Department of Mechanical
Science and Engineering at the University of Illinois at Urbana-Champaign
(UIUC); S. P. Meyn is with the Department of Electrical and Computer
Engineering at the University of Florida at Gainesville; Corresponding email:
<EMAIL_ADDRESS>
###### Abstract

This paper revisits the question of duality between minimum variance
estimation and optimal control first described for the linear Gaussian case in
the celebrated paper of Kalman and Bucy. A duality result is established for
nonlinear filtering, mirroring closely the original Kalman-Bucy duality of
control and estimation for linear systems. The result for the finite state-
space continuous time Markov chain is presented. It’s solution is used to
derive the classical Wonham filter.

##  I Introduction

In Kalman’s celebrated paper with Bucy, it is shown that the problem of
optimal estimation is dual to an optimal control problem [1]. A striking
example of the dual relationship is that, with the time arrow reversed, the
dynamic Riccati equation (DRE) of the optimal control is the same as the
covariance update equation of the Kalman filter. The relationship is useful,
e.g., to derive results on asymptotic stability of the linear filter based on
asymptotic properties of the solution of the DRE [2].

A nonlinear extension of the minimum variance estimator has been considered to
be a harder problem. In literature, it has been noted that: i) the dual
relationship between the DRE of the LQ optimal control and the covariance
update equation of the Kalman filter is not consistent with the interpretation
of the negative log-posterior as a value function; and ii) some of the linear
algebraic operations, e.g., the use of matrix transpose to define the dual
system, are not applicable to nonlinear systems [3, 4]. For these reasons, the
original duality of Kalman-Bucy is seen as an LQG artifact that does not
generalize [3].

In this paper, a nonlinear extension of the minimum variance estimation is
presented for the special case of a Markov process in continuous time, on a
finite state-space. The dual system is a backward ordinary differential
equation. An optimal control objective is formulated whose solution yields the
minimum variance estimator. Using the elementary method of change of control,
the formula for the optimal control is obtained and used to derive the
classical Wonham filter.

The outline of the paper is as follows: classical duality is reviewed in Sec.
II, and the new dual optimal control problem for the finite case is described
in Sec. III. Its solution leading to the Wonham filter is presented in Sec.
IV.

##  II Background on classical duality

Linear Gaussian filtering model: Specified by the linear stochastic
differential equation (SDE):

Signal
$\displaystyle\,\mathrm{d}X_{t}=A^{\top}X_{t}\,\mathrm{d}t+\,\mathrm{d}B_{t}$
Observation
$\displaystyle\,\mathrm{d}Z_{t}=H^{\top}X_{t}\,\mathrm{d}t+\,\mathrm{d}W_{t}$

where $X_{t}\in\mathbb{R}^{d}$ is the state at time $t$,
$Z_{t}\in\mathbb{R}^{m}$ is the observation, $A$, $H$ are matrices of
appropriate dimension, and $B$, $W$ are mutually independent Wiener processes
(w.p.) taking values in $\mathbb{R}^{d}$ and $\mathbb{R}^{m}$, respectively.
The covariance matrices associated with $B$ and $W$ are denoted by ${Q}$ and
${R}$, respectively. The initial condition $X_{0}$ is drawn from a Gaussian
distribution $\mathcal{N}(\hat{x}_{0},\Sigma_{0})$, independent of $B$ or $W$.
It is assumed that the noise covariance matrix is non-singular, ${R}\succ 0$.

Minimum-variance estimator: Consider the problem of constructing a minimum
variance estimator for the random variable $f^{\top}X_{T}$, at some fixed time
$T$, where $f\in\mathbb{R}^{d}$ is an arbitrary, known vector.

Given the observations $\\{Z_{t}:t\in[0,T]\\}$, the following linear structure
for the optimal estimator is assumed:

$S_{T}=y_{0}^{\top}\hat{x}_{0}-\int_{0}^{T}u_{t}^{\top}\,\mathrm{d}Z_{t}$

where $y_{0}\in\mathbb{R}^{d}$ is constructed below, and the input
$u=\\{u_{t}:t\in[0,T]\\}$ is chosen to solve the optimization problem,

$\min_{u}\;\;{\sf E}(|S_{T}-f^{\top}X_{T}|^{2})$

The solution $S^{*}_{T}$ coincides with the minimum-variance estimator of
$f^{\top}X_{T}$.

This stochastic optimization problem is converted to a deterministic optimal
control problem via duality.

Dual optimal control problem:

$\displaystyle\mathop{\text{Minimize}}_{u}$ $\displaystyle
J(u)={\mathchoice{\genfrac{}{}{}{1}{1}{2}}{\genfrac{}{}{}{2}{1}{2}}{\genfrac{}{}{}{3}{1}{2}}{\genfrac{}{}{}{4}{1}{2}}}\
y_{0}^{\top}\Sigma_{0}y_{0}+\int_{0}^{T}{\mathchoice{\genfrac{}{}{}{1}{1}{2}}{\genfrac{}{}{}{2}{1}{2}}{\genfrac{}{}{}{3}{1}{2}}{\genfrac{}{}{}{4}{1}{2}}}u_{t}^{\top}Ru_{t}+{\mathchoice{\genfrac{}{}{}{1}{1}{2}}{\genfrac{}{}{}{2}{1}{2}}{\genfrac{}{}{}{3}{1}{2}}{\genfrac{}{}{}{4}{1}{2}}}y_{t}^{\top}Qy_{t}\,\mathrm{d}t$
Subject to
$\displaystyle\frac{\,\mathrm{d}y_{t}}{\,\mathrm{d}t}=-Ay_{t}-Hu_{t},\quad
y_{T}=f$

The process $\\{y_{t}:t\in[0,T]\\}$ is referred to as the dual process. The
solution of the optimal control problem yields the optimal control input,
along with the vector $y_{0}$ that determines the minimum-variance estimator
$S^{*}_{T}$.

The Kalman filter is obtained by expressing $\\{S^{*}_{t}(f):t\geq 0,\
f\in\mathbb{R}^{d}\\}$ as the solution to a linear SDE [5, Ch. 7].

##  III Duality for Nonlinear Filtering:
The Finite State space Case

Nonlinear filtering model: The finite state-space filtering problem is
considered, in which the state-space is the canonical basis
$\mathbb{S}=\\{e_{1},e_{2},\ldots,e_{d}\\}$ in $\mathbb{R}^{d}$.

The Markovian state process $X=\\{X_{t}:t\in[0,T]\\}$ evolves in continuous
time, taking values in $\mathbb{S}$. This and the observation process
$Z=\\{Z_{t}:t\in[0,T]\\}$ are modeled by the SDE,

Signal
$\displaystyle\;\;\,\mathrm{d}X_{t}=A^{\top}X_{t}\,\mathrm{d}t+\,\mathrm{d}B_{t}$
(1a) Observation
$\displaystyle\;\;\,\mathrm{d}Z_{t}=H^{\top}X_{t}\,\mathrm{d}t+\,\mathrm{d}W_{t}$
(1b)

where $A\in\mathbb{R}^{d\times d}$ is the rate matrix,
$H\in\mathbb{R}^{d\times m}$, $W$ is an $m$-dimensional w.p. with covariance
$R\succ 0$. $B=\\{B_{t}:t\in[0,T]\\}$ is defined by

$B_{t}=X_{t}-\int_{0}^{t}A^{\top}X_{\tau}\,\mathrm{d}\tau$

and it is a martingale since $A$ is the generator of the Markov process. The
initial distribution for $X_{0}$ is denoted $\pi_{0}\in{\cal P}(\mathbb{S})$
where ${\cal P}(\mathbb{S})$ denotes the probability simplex in
$\mathbb{R}^{d}$. It is assumed that $X$, $W$ are mutually independent.

The linear observation model is chosen without loss of generality: for any
function $h\;\colon\;\mathbb{S}\to\mathbb{R}$ we have $h(x)=H^{\top}x$ for
$x\in\mathbb{S}$, with $H_{i}=h(e_{i})$.

Two filtrations are required in this work: ${\cal F}=\\{{\cal F}_{t}:t\geq
0\\}$ and ${\cal Z}=\\{{\cal Z}_{t}:t\geq 0\\}$ where

${\cal F}_{t}:=\sigma(X_{\tau},W_{\tau}:0\leq\tau\leq t)\,,\quad{\cal
Z}_{t}=\sigma(Z_{\tau}:0\leq\tau\leq t)$

Let $C_{\cal Z}^{p}$ denote the family of $\mathbb{R}^{p}$-valued, continuous,
and ${\cal Z}$-adapted functions of time (the superscript “$p$” is omitted in
the special case $p=1$).

The filtering problem is to compute the posterior distribution ${\sf
P}(X_{t}\in\,\cdot\,\mid{\cal Z}_{t})$ [6]. The solution is derived here
through duality, very much like in the classical linear setting.

The dual system: A backward ordinary differential equation (ODE) on
$\mathbb{R}^{d}$,

$\frac{\,\mathrm{d}Y_{t}}{\,\mathrm{d}t}=-AY_{t}-HU_{t},\quad Y_{T}=f$ (2)

whose solution is

$Y_{t}=e^{A(T-t)}f+\int_{t}^{T}e^{A(\tau-t)}HU_{\tau}\,\mathrm{d}\tau\,,\quad
0\leq t\leq T$

An optimal control problem is posed for the dual system (2) whose solution
yields the nonlinear filter. This requires some restrictions on the class of
control inputs. The set of admissible control inputs is defined as follows:

$\mathcal{U}:=\left\\{U_{t}={\sf K}_{t}^{\top}Y_{t}+V_{t}:{\sf K}\in C_{\cal
Z}^{d\times m},\;V\in C_{\cal Z}^{m},\;t\in[0,T]\right\\}$ (3)

We denote $U=\\{U_{t}:t\in[0,T]\\}$, ${\sf K}=\\{{\sf K}_{t}:t\in[0,T]\\}$ and
$V=\\{V_{t}:t\in[0,T]\\}$. By construction, ${\sf K}$ and $V$ and ${\cal
Z}$-adapted processes but $U$ may not be ${\cal Z}$-adapted because of the
backward nature of the ODE (2).

The following proposition provides explicit representations for the solution
of the backward ODE (2). Its proof appears in Appendix -B.

###### Proposition 1

Consider the backward ODE (2) with control input $U_{t}={\sf
K}_{t}^{\top}Y_{t}+V_{t}$ where $\\{{\sf K}_{t}:t\in[0,T]\\}$ and
$\\{V_{t}:t\in[0,T]\\}$ are given ${\cal Z}$-adapted processes. Then there
exist ${\cal Z}$-adapted processes
$\\{\Phi_{t},\eta_{t},\kappa_{t},\gamma_{t}:t\in[0,T]\\}$, and $Y_{0}\in{\cal
Z}_{T}$, such that for each $t\in[0,T]$,

$Y_{t}=\Phi_{t}Y_{0}+\eta_{t},\quad U_{t}=\kappa_{t}^{\top}Y_{0}+\gamma_{t}$

This proposition is used to define stochastic integral being used throughout
the paper which is illustrated in the Appendix -C.

Minimum-variance estimator: The problem of interest is precisely as in the
linear Gaussian case: given a fixed time $T>0$, and $f\in\mathbb{R}^{d}$, the
goal is to obtain a representation for the minimum variance estimator for the
random variable $f^{\top}X_{T}$.

Given observations $Z=\\{Z_{t}:0\leq t\leq T\\}$ defined according to the
model (1b), the following linear structure for the estimator will be
justified:

$S_{T}=Y_{0}^{\top}\pi_{0}-\int_{0}^{T}U_{t}^{\top}\,\mathrm{d}Z_{t}$ (4)

The vector $Y_{0}$ is obtained from the solution to (2).

The optimal control input is chosen as the solution to the optimization
problem:

$\min_{U\in\mathcal{U}}\;\;{\sf E}[|S_{T}-f^{\top}X_{T}|^{2}]$

Justification for the form (4) is provided through the formulation of the dual
control problem.

###### Remark 1

The stochastic integral $\int_{0}^{T}U_{t}^{\top}\,\mathrm{d}Z_{t}$ in (4) is
defined as a forward integral. Formally, for a given admissible choice of
${\cal Z}$-adapted processes ${\sf K}$ and $V$, upon using the representation
in Prop. 1,

$\int_{0}^{T}U_{t}^{\top}\,\mathrm{d}Z_{t}=Y_{0}^{\top}\int_{0}^{T}\kappa_{t}\,\mathrm{d}Z_{t}+\int_{0}^{T}\gamma_{t}^{\top}\,\mathrm{d}Z_{t}$

where $\\{\kappa_{t}:t\in[0,T]\\}$, $\\{\gamma_{t}:t\in[0,T]\\}$ are adapted
processes and therefore the associated integrals are well-defined as standard
Itô-integrals. A self-contained background on interpreting stochastic
integrals for the non-adapted processes considered in this paper appears in
Appendix -C.

Dual optimal control problem:

$\displaystyle\mathop{\text{Min}}_{U\in\mathcal{U}}\ \ J(U)={\sf
E}\;\Big{(}{\mathchoice{\genfrac{}{}{}{1}{1}{2}}{\genfrac{}{}{}{2}{1}{2}}{\genfrac{}{}{}{3}{1}{2}}{\genfrac{}{}{}{4}{1}{2}}}|Y_{0}^{\top}X_{0}-Y_{0}^{\top}\pi_{0}|^{2}+\int_{0}^{T}{\mathchoice{\genfrac{}{}{}{1}{1}{2}}{\genfrac{}{}{}{2}{1}{2}}{\genfrac{}{}{}{3}{1}{2}}{\genfrac{}{}{}{4}{1}{2}}}U_{t}^{\top}RU_{t}\,\mathrm{d}t$
$\displaystyle\quad+\int_{0}^{T}{\mathchoice{\genfrac{}{}{}{1}{1}{2}}{\genfrac{}{}{}{2}{1}{2}}{\genfrac{}{}{}{3}{1}{2}}{\genfrac{}{}{}{4}{1}{2}}}Y_{t}^{\top}\,\mathrm{d}\langle
X,X^{\top}\rangle_{t}Y_{t}+{\cal E}_{t}U_{t}^{\top}\,\mathrm{d}W_{t}+{\cal
E}_{t}Y_{t}^{\top}\,\mathrm{d}B_{t}\Big{)}$ (5a) $\displaystyle\text{Subject
to}\ \ \frac{\,\mathrm{d}Y_{t}}{\,\mathrm{d}t}=-AY_{t}-HU_{t},\quad Y_{T}=f$
(5b)

where $\langle X,X^{\top}\rangle$ denotes the quadratic variation of the
Markov process $X$, and the error process ${\cal E}=\\{{\cal
E}_{t}:t\in[0,T]\\}$ is defined as follows:

${\cal
E}_{t}:=Y_{0}^{\top}(X_{0}-\pi_{0})+\int_{0}^{t}U_{\tau}^{\top}\,\mathrm{d}W_{\tau}+\int_{0}^{t}Y_{\tau}^{\top}\,\mathrm{d}B_{\tau}$
(6)

As in Remark 1, the four stochastic integrals appearing above are defined also
as forward integrals (see Appendix -C).

The relationship between the optimal control objective $J(\cdot)$ and the
minimum variance objective (III) is illustrated in the following proposition.
The proof appears in the Appendix -D.

###### Proposition 2

Consider the state-observation model (1), the linear estimator (4) and the
dual optimal control problem (5). For any arbitrary choice of an admissible
control input,

$J(U)={\mathchoice{\genfrac{}{}{}{1}{1}{2}}{\genfrac{}{}{}{2}{1}{2}}{\genfrac{}{}{}{3}{1}{2}}{\genfrac{}{}{}{4}{1}{2}}}{\sf
E}[|S_{T}-f^{\top}X_{T}|^{2}]$

This provides a justification for the objective function (5a) and moreover
shows that $J(U)\geq 0$ for any admissible control.

###### Remark 2

Consider a deterministic control input of the form
$U_{t}=k_{t}^{\top}Y_{t}+v_{t}$ where $\\{k_{t}\\}$, $\\{v_{t}\\}$ are
deterministic functions of time (in particular, they do not depend upon the
observations). Such a control is trivially admissible. In this case,
$\\{Y_{t}\\}$ is a deterministic function of time and the error process ${\cal
E}$ is a ${\cal F}$-martingale. Consequently,

${\sf E}\Big{(}\int_{0}^{T}{\cal E}_{t}U_{t}^{\top}\,\mathrm{d}W_{t}+{\cal
E}_{t}Y_{t}^{\top}\,\mathrm{d}B_{t}\Big{)}=0$

and the objective function in (5a) simplifies to

$J(U)={\mathchoice{\genfrac{}{}{}{1}{1}{2}}{\genfrac{}{}{}{2}{1}{2}}{\genfrac{}{}{}{3}{1}{2}}{\genfrac{}{}{}{4}{1}{2}}}Y_{0}^{\top}\Sigma_{0}Y_{0}+\int_{0}^{T}{\mathchoice{\genfrac{}{}{}{1}{1}{2}}{\genfrac{}{}{}{2}{1}{2}}{\genfrac{}{}{}{3}{1}{2}}{\genfrac{}{}{}{4}{1}{2}}}U_{t}^{\top}RU_{t}+{\mathchoice{\genfrac{}{}{}{1}{1}{2}}{\genfrac{}{}{}{2}{1}{2}}{\genfrac{}{}{}{3}{1}{2}}{\genfrac{}{}{}{4}{1}{2}}}Y_{t}^{\top}{\sf
E}(Q(X_{t}))Y_{t}\,\mathrm{d}t$

where $\Sigma_{0}:={\sf E}((X_{0}-\pi_{0})(X_{0}-\pi_{0})^{\top})$ and
$Q(\cdot)$ is a $\mathbb{S}\to\mathbb{R}^{d\times d}$ map defined as follows:

$Q(e_{i}):=\sum_{j\neq i}A_{ij}(e_{j}-e_{i})(e_{j}-e_{i})^{\top},\quad
i=1,\ldots,d$

The resulting problem is a deterministic LQ problem whose optimal solution
$\\{U_{t}^{*}:t\in[0,T]\\}$ will (in general) yield a sub-optimal estimate
$S_{T}^{*}$ using (4). The general problem considered here is much tougher
because ${\cal E}$ is not a ${\cal F}$-martingale: Under arbitrary admissible
controls, it is not even adapted to this filtration.

We have now set the stage to derive the nonlinear filter via the solution to
the dual optimal control problem.

##  IV Derivation of the Nonlinear Filter

Recall that an admissible input has the form $U_{t}={\sf
K}_{t}^{\top}Y_{t}+V_{t}$ where $t\in[0,T]$. The goal is to obtain a formula
for the gain process ${\sf K}=\\{{\sf K}_{t}:t\in[0,T]\\}$ such that the best
choice of $V=\\{V_{t}:t\in[0,T]\\}$ is zero.

This choice of input class can be regarded as an instance of the method of
“change of control” because $V$ represents the new variable for control [6,
Ch. 3.1].

If $V_{t}\equiv 0$ then $\bar{Y}=\\{\bar{Y}_{t}:t\in[0,T]\\}$ solves the
backward ODE

$\frac{\,\mathrm{d}\bar{Y}_{t}}{\,\mathrm{d}t}=-A\bar{Y}_{t}-H{\sf
K}_{t}^{\top}\bar{Y}_{t},\quad\bar{Y}_{T}=f$

and the associated control is denoted $\bar{U}_{t}={\sf
K}_{t}^{\top}\bar{Y}_{t}$ for $t\in[0,T]$. With an arbitrary $V$, the solution
is expressed

$\displaystyle Y_{t}=\bar{Y}_{t}+\tilde{Y}_{t},\quad
U_{t}=\bar{U}_{t}+\tilde{U}_{t}$

where $\tilde{Y}=\\{\tilde{Y}_{t}:t\in[0,T]\\}$ also solves a backward ODE:

$\frac{\,\mathrm{d}\tilde{Y}_{t}}{\,\mathrm{d}t}=-A\tilde{Y}_{t}-H{\sf
K}_{t}^{\top}\tilde{Y}_{t}-HV_{t},\quad\tilde{Y}_{T}=0$ (7)

with $\tilde{U}_{t}={\sf K}_{t}^{\top}\tilde{Y}_{t}+V_{t}$ for $t\in[0,T]$.

The error term is analogously split as ${\cal E}_{t}=\bar{{\cal
E}}_{t}+\tilde{{\cal E}}_{t}$, with

$\displaystyle\bar{{\cal E}}_{t}$
$\displaystyle=\bar{Y}_{0}(X_{0}-\pi_{0})+\int_{0}^{t}\bar{U}_{\tau}^{\top}\,\mathrm{d}W_{\tau}+\int_{0}^{t}\bar{Y}_{\tau}^{\top}\,\mathrm{d}B_{\tau}$
$\displaystyle\tilde{{\cal E}}_{t}$
$\displaystyle=\tilde{Y}_{0}(X_{0}-\pi_{0})+\int_{0}^{t}\tilde{U}_{\tau}^{\top}\,\mathrm{d}W_{\tau}+\int_{0}^{t}\tilde{Y}_{\tau}^{\top}\,\mathrm{d}B_{\tau}$

The optimal gain is described in the following theorem.

###### Theorem 1

Consider the optimal control problem (5). For any non-zero $V\in C_{\cal
Z}^{m}$,

$J(U)\geq J(\bar{U})$

where the optimal gain is defined as following:

$\displaystyle\,\mathrm{d}\bar{\pi}_{t}=A^{\top}\bar{\pi}_{t}\,\mathrm{d}t-{\sf
K}_{t}^{\top}(\,\mathrm{d}Z_{t}-H^{\top}\bar{\pi}_{t}\,\mathrm{d}t),\quad\bar{\pi}_{0}=\pi_{0}$
(8a) $\displaystyle{\sf K}_{t}=-{\sf
E}\big{(}(X_{t}-\bar{\pi}_{t})(X_{t}-\bar{\pi}_{t})^{\top}|{\cal
Z}_{t}\big{)}HR^{-1},\quad t\in[0,T]$ (8b)

###  IV-A Proof of Thm. 1

It is simple calculation to see that

$J(U)=J(\bar{U})+J(\tilde{U})+{\sf E}(\mathcal{C})$

where the cross-term $\mathcal{C}$ is defined by

$\displaystyle\mathcal{C}$
$\displaystyle=\underbrace{\tilde{Y}_{0}^{\top}(X_{0}-\pi_{0})(X_{0}-\pi_{0})^{\top}\bar{Y}_{0}}_{\text{term
(i)}}$
$\displaystyle+\underbrace{\int_{0}^{T}\tilde{U}_{t}^{\top}R\bar{U}_{t}\,\mathrm{d}t+\tilde{Y}_{t}^{\top}\,\mathrm{d}\langle
X,X^{\top}\rangle_{t}\bar{Y}_{t}}_{\text{term (ii)}}$
$\displaystyle+\underbrace{\int_{0}^{T}(\tilde{{\cal
E}}_{t}\bar{U}_{t}^{\top}+\bar{{\cal
E}}_{t}\tilde{U}_{t}^{\top})\,\mathrm{d}W_{t}+\int_{0}^{T}(\tilde{{\cal
E}}_{t}\bar{Y}_{t}^{\top}+\bar{{\cal
E}}_{t}\tilde{Y}_{t}^{\top})\,\mathrm{d}B_{t}}_{\text{term (iii)}}$

The strategy now is to choose ${\sf K}$ such that ${\sf E}(\mathcal{C})=0$ for
all possible choices of ${\cal Z}$-adapted $V$.

Term (i): A standard technique of optimal control theory dictates that the
terminal condition term be expressed as an integral by introducing a dual
variable. Towards this goal, we introduce a vector-valued stochastic process
$\bar{\pi}=\\{\bar{\pi}_{t}:t\in[0,T]\\}$ with $\bar{\pi}_{0}=\pi_{0}$ (the
prior). At this point of time, we only require that $\bar{\pi}$ is a ${\cal
Z}$-adapted process. The dynamics of this process will be defined later.

Using the process $\bar{\pi}$, together with the requirement (7) that
$\tilde{Y}_{T}=0$, we obtain

$\tilde{Y}_{0}^{\top}(\pi_{0}-X_{0})(\pi_{0}-X_{0})^{\top}\bar{Y}_{0}=-\int_{0}^{T}\,\mathrm{d}\big{(}\tilde{Y}_{t}^{\top}(\bar{\pi}_{t}-X_{t})(\bar{\pi}_{t}-X_{t})^{\top}\bar{Y}_{t}\big{)}$

The differential is evaluated by an application of the product formula:111See
Appendix -C for a justification of the product formula for the class of (non-
adapted) stochastic processes arising in this paper.

$\displaystyle\,\mathrm{d}$
$\displaystyle\big{(}\tilde{Y}_{t}^{\top}(\bar{\pi}_{t}-X_{t})(\bar{\pi}_{t}-X_{t})^{\top}\bar{Y}_{t}\big{)}$
$\displaystyle=$
$\displaystyle\,\tilde{Y}_{t}^{\top}\Big{\\{}\big{(}\,\mathrm{d}\bar{\pi}_{t}-A^{\top}\bar{\pi}_{t}\,\mathrm{d}t+{\sf
K}_{t}H^{\top}(X_{t}-\bar{\pi}_{t})\,\mathrm{d}t-\,\mathrm{d}B_{t}\big{)}(\bar{\pi}_{t}-X_{t})^{\top}$
$\displaystyle+(\bar{\pi}_{t}-X_{t})\big{(}\,\mathrm{d}\bar{\pi}_{t}-A^{\top}\bar{\pi}_{t}\,\mathrm{d}t+{\sf
K}_{t}H^{\top}(X_{t}-\bar{\pi}_{t})\,\mathrm{d}t-\,\mathrm{d}B_{t}\big{)}^{\top}$
$\displaystyle+\,\mathrm{d}\langle(\bar{\pi}-X),(\bar{\pi}-X)^{\top}\rangle_{t}\Big{\\}}\bar{Y}_{t}-V_{t}^{\top}H^{\top}(X_{t}-\bar{\pi}_{t})(X_{t}-\bar{\pi}_{t})^{\top}\bar{Y}_{t}\,\mathrm{d}t$

where $\langle(\bar{\pi}-X),(\bar{\pi}-X)^{\top}\rangle$ denotes the quadratic
variation of the process $\bar{\pi}-X$. It is noted that each of the term in
the integral is a quadratic either in $\tilde{Y}_{t}$ and $\bar{Y}_{t}$ or in
$V_{t}$ and $\bar{Y}_{t}$.

Term (ii): The second term is expressed as:

$\displaystyle\int_{0}^{T}$
$\displaystyle\tilde{U}_{t}^{\top}R\bar{U}_{t}\,\mathrm{d}t+\tilde{Y}_{t}^{\top}\,\mathrm{d}\langle
X,X^{\top}\rangle_{t}\bar{Y}_{t}$
$\displaystyle=\int_{0}^{T}\Big{(}\;\tilde{Y}_{t}^{\top}\big{(}{\sf
K}_{t}R{\sf K}_{t}^{\top}\,\mathrm{d}t+\,\mathrm{d}\langle
X,X^{\top}\rangle_{t}\big{)}\bar{Y}_{t}+V_{t}^{\top}R{\sf
K}_{t}^{\top}\bar{Y}_{t}\,\mathrm{d}t\Big{)}$

Term (iii): It remains to tackle the two stochastic integrals involving the
error processes. We begin by recalling (6):

$\displaystyle{\cal E}_{t}$
$\displaystyle=Y_{0}^{\top}(X_{0}-\pi_{0})+\int_{0}^{t}U_{\tau}^{\top}\,\mathrm{d}W_{\tau}+\int_{0}^{t}Y_{\tau}^{\top}\,\mathrm{d}B_{\tau}$

Proceeding as in term (i), the process $\bar{\pi}$ is again used to express
the terminal condition term $Y_{0}^{\top}(\pi_{0}-X_{0})$ as an integral. Once
again, using the product rule

$\displaystyle\,\mathrm{d}\big{(}{Y}_{t}^{\top}(X_{t}-\bar{\pi}_{t})\big{)}=-Y_{t}^{\top}\big{(}\,\mathrm{d}\bar{\pi}_{t}$
$\displaystyle-A^{\top}\bar{\pi}_{t}\,\mathrm{d}t+{\sf
K}_{t}H^{\top}(X_{t}-\bar{\pi}_{t})\,\mathrm{d}t\big{)}$
$\displaystyle+Y_{t}^{\top}\,\mathrm{d}B_{t}-V_{t}^{\top}H^{\top}(X_{t}-\bar{\pi}_{t})\,\mathrm{d}t$

Therefore,

$\displaystyle{\cal E}_{t}=$ $\displaystyle
Y_{0}^{\top}(X_{0}-\pi_{0})+\int_{0}^{t}U_{\tau}^{\top}\,\mathrm{d}W_{\tau}+\int_{0}^{t}Y_{\tau}^{\top}\,\mathrm{d}B_{\tau}$
$\displaystyle=$ $\displaystyle
Y_{t}^{\top}(X_{t}-\bar{\pi}_{t})+\int_{0}^{t}V_{\tau}^{\top}(\,\mathrm{d}W_{\tau}+H^{\top}(X_{\tau}-\bar{\pi}_{\tau})\,\mathrm{d}\tau)$
$\displaystyle+\int_{0}^{t}Y_{\tau}^{\top}\big{(}\,\mathrm{d}\bar{\pi}_{\tau}-A^{\top}\pi_{\tau}\,\mathrm{d}\tau+{\sf
K}_{\tau}(\,\mathrm{d}W_{\tau}+H^{\top}(X_{\tau}-\bar{\pi}_{\tau})\,\mathrm{d}\tau)\big{)}$

In order to reduce the notational burden, the following differential notation
is adopted for the ${\cal Z}$-adapted stochastic processes
${\bar{I}}=\\{{\bar{I}}_{t}:t\in[0,T]\\}$ and
$\mathcal{L}=\\{\mathcal{L}_{t}:t\in[0,T]\\}$:

$\displaystyle\,\mathrm{d}{\bar{I}}_{t}$
$\displaystyle:=\,\mathrm{d}Z_{t}-H^{\top}\bar{\pi}_{t}\,\mathrm{d}t$
$\displaystyle\,\mathrm{d}\mathcal{L}_{t}$
$\displaystyle:=\,\mathrm{d}\bar{\pi}_{t}-A^{\top}\bar{\pi}_{t}\,\mathrm{d}t+{\sf
K}_{t}\,\mathrm{d}{\bar{I}}_{t}$

The notation is used to express the error succinctly as

${\cal
E}_{t}=Y_{t}^{\top}(\bar{\pi}_{t}-X_{t})-\int_{0}^{t}Y_{\tau}^{\top}\,\mathrm{d}\mathcal{L}_{\tau}+\int_{0}^{t}V_{\tau}^{\top}\,\mathrm{d}{\bar{I}}_{\tau}$

In particular, upon splitting ${\cal E}_{t}=\bar{{\cal E}}_{t}+\tilde{{\cal
E}}_{t}$, we have

$\displaystyle\bar{{\cal E}}_{t}$
$\displaystyle=\bar{Y}_{t}^{\top}(X_{t}-\bar{\pi}_{t})+\int_{0}^{t}\bar{Y}_{\tau}^{\top}\,\mathrm{d}\mathcal{L}_{\tau}$
$\displaystyle\tilde{{\cal E}}_{t}$
$\displaystyle=\tilde{Y}_{t}^{\top}(X_{t}-\bar{\pi}_{t})+\int_{0}^{t}\tilde{Y}_{\tau}^{\top}\,\mathrm{d}\mathcal{L}_{\tau}+\int_{0}^{t}V_{\tau}^{\top}\,\mathrm{d}{\bar{I}}_{\tau}$

We thus obtain a useful expression for term (iii):

$\displaystyle\int_{0}^{T}\tilde{{\cal E}}_{t}\bar{U}_{t}^{\top}+\bar{{\cal
E}}_{t}\tilde{U}_{t}^{\top}\,\mathrm{d}W_{t}+\int_{0}^{T}(\tilde{{\cal
E}}_{t}\bar{Y}_{t}^{\top}+\bar{{\cal
E}}_{t}\tilde{Y}_{t}^{\top})\,\mathrm{d}B_{t}$
$\displaystyle=\int_{0}^{T}\Big{\\{}\tilde{Y}_{t}^{\top}(X_{t}-\bar{\pi}_{t})\bar{Y}_{t}^{\top}{\sf
K}_{t}+\bar{Y}_{t}^{\top}(X_{t}-\bar{\pi}_{t})\tilde{Y}_{t}^{\top}{\sf
K}_{t}+\big{(}\int_{0}^{t}\tilde{Y}_{\tau}^{\top}\,\mathrm{d}\mathcal{L}_{\tau}\big{)}\bar{Y}_{t}^{\top}{\sf
K}_{t}$
$\displaystyle\quad\quad+\big{(}\int_{0}^{t}\bar{Y}_{\tau}^{\top}\,\mathrm{d}\mathcal{L}_{\tau}\big{)}\tilde{Y}_{t}^{\top}{\sf
K}_{t}+\big{(}\int_{0}^{t}V_{\tau}^{\top}\,\mathrm{d}{\bar{I}}_{\tau}\big{)}\bar{Y}_{t}^{\top}{\sf
K}_{t}+V_{t}^{\top}\bar{Y}_{t}^{\top}(X_{t}-\bar{\pi}_{t})$
$\displaystyle\quad\quad+V_{t}^{\top}\big{(}\int_{0}^{t}\bar{Y}_{\tau}^{\top}\,\mathrm{d}\mathcal{L}_{\tau}\big{)}\Big{\\}}\,\mathrm{d}W_{t}$
$\displaystyle\quad+\int_{0}^{T}\Big{\\{}\tilde{Y}_{t}^{\top}(X_{t}-\bar{\pi}_{t})\bar{Y}_{t}^{\top}+\bar{Y}_{t}^{\top}(X_{t}-\bar{\pi}_{t})\tilde{Y}_{t}^{\top}+\big{(}\int_{0}^{t}\tilde{Y}_{\tau}^{\top}\,\mathrm{d}\mathcal{L}_{\tau}\big{)}\bar{Y}_{t}^{\top}$
$\displaystyle\quad\quad+\big{(}\int_{0}^{t}\bar{Y}_{\tau}^{\top}\,\mathrm{d}\mathcal{L}_{\tau}\big{)}\tilde{Y}_{t}^{\top}+\big{(}\int_{0}^{t}V_{\tau}^{\top}\,\mathrm{d}{\bar{I}}_{\tau}\big{)}\bar{Y}_{t}^{\top}\Big{\\}}\,\mathrm{d}B_{t}$

This concludes our program of expressing each of three terms in $\mathcal{C}$
as an integral with sub-terms containing $\bar{Y}_{t},\tilde{Y}_{t},V_{t}$.
Now, every sub-term is a quadratic of one of the two types:

  1. 1.

The type 1 quadratic sub-terms contain $\bar{Y}_{t}$ and $\tilde{Y}_{t}$. An
example of this type of quadratic is $\tilde{Y}_{t}^{\top}{\sf K}_{t}R{\sf
K}_{t}^{\top}\bar{Y}_{t}$ in the term (ii).

  2. 2.

The type 2 quadratic sub-terms contain $\bar{Y}_{t}$ and $V_{t}$. An example
of this is $V_{t}{\sf K}_{t}^{\top}\bar{Y}_{t}$ in the term (ii).

We express $\mathcal{C}=\mathcal{C}_{1}+\mathcal{C}_{2}$, where
$\mathcal{C}_{1}$ contains only the quadratic sub-terms of type 1 and
$\mathcal{C}_{2}$ contains only the quadratic sub-terms of type 2. Upon
collecting terms, we obtain

$\displaystyle\mathcal{C}_{1}$
$\displaystyle=\int_{0}^{T}\tilde{Y}_{t}^{\top}({\sf K}_{t}R{\sf
K}_{t}^{\top}\,\mathrm{d}t+\,\mathrm{d}\langle
X,X^{\top}\rangle_{t})\bar{Y}_{t}-\tilde{Y}_{t}^{\top}\,\mathrm{d}\big{\langle}(\bar{\pi}-X_{t}),(\bar{\pi}-X_{t})^{\top}\big{\rangle}_{t}\bar{Y}_{t}$
$\displaystyle+\int_{0}^{T}\Big{(}\tilde{Y}_{t}^{\top}(\bar{\pi}_{t}-X_{t})\bar{Y}_{t}^{\top}+\bar{Y}_{t}^{\top}(\bar{\pi}_{t}-X_{t})\tilde{Y}_{t}^{\top}\Big{)}\,\mathrm{d}\mathcal{L}_{t}$
$\displaystyle+\int_{0}^{T}\Big{(}\big{(}\int_{0}^{t}\bar{Y}_{\tau}^{\top}\,\mathrm{d}\mathcal{L}_{\tau}\big{)}\tilde{Y}_{t}^{\top}{\sf
K}_{t}+\big{(}\int_{0}^{t}\tilde{Y}_{\tau}^{\top}\,\mathrm{d}\mathcal{L}_{\tau}\big{)}\bar{Y}_{t}^{\top}{\sf
K}_{t}\Big{)}\,\mathrm{d}W_{t}$
$\displaystyle+\int_{0}^{T}\Big{(}\big{(}\int_{0}^{t}\bar{Y}_{\tau}^{\top}\,\mathrm{d}\mathcal{L}_{\tau}\big{)}\tilde{Y}_{t}^{\top}+\big{(}\int_{0}^{t}\tilde{Y}_{\tau}^{\top}\,\mathrm{d}\mathcal{L}_{\tau}\big{)}\bar{Y}_{t}^{\top}\Big{)}\,\mathrm{d}B_{t}$

and

$\displaystyle\mathcal{C}_{2}=\int_{0}^{T}V_{t}^{\top}\big{(}R{\sf
K}_{t}^{\top}+H^{\top}(X_{t}-\bar{\pi}_{t})(X_{t}-\bar{\pi}_{t})^{\top}\big{)}\bar{Y}_{t}\,\mathrm{d}t$
$\displaystyle+\int_{0}^{T}\Big{\\{}V_{t}^{\top}(X_{t}-\bar{\pi}_{t})^{\top}\bar{Y}_{t}+V_{t}^{\top}\big{(}\int_{0}^{t}\bar{Y}_{\tau}^{\top}\,\mathrm{d}\mathcal{L}_{\tau}\big{)}+\big{(}\int_{0}^{t}V_{\tau}^{\top}\,\mathrm{d}{\bar{I}}_{\tau}\big{)}\bar{Y}_{t}^{\top}{\sf
K}_{t}\Big{\\}}\,\mathrm{d}W_{t}$
$\displaystyle\quad\quad+\int_{0}^{T}\big{(}\int_{0}^{t}V_{\tau}^{\top}\,\mathrm{d}{\bar{I}}_{\tau}\big{)}\bar{Y}_{t}^{\top}\,\mathrm{d}B_{t}$

In order to have ${\sf E}(\mathcal{C})={\sf E}(\mathcal{C}_{1})+{\sf
E}(\mathcal{C}_{2})=0$ for all possible choices of $\bar{Y},\tilde{Y}$ and for
all possible choices of $\bar{Y},V$, we follow the following 2-step procedure:

  1. 1.

In Step 1, we obtain an equation for $\bar{\pi}$ by setting

${\sf E}(\mathcal{C}_{1})=0,\quad\text{a.s.}$

  2. 2.

Given $\bar{\pi}$ from Step 1, we next derive a formula for the optimal gain
${\sf K}$ by imposing the requirement

${\sf E}(\mathcal{C}_{2})=0,\quad\forall V\in C_{\cal Z}^{m}$

The 2-step procedure is inspired by the analogous procedure in classical LQ
theory where the step 1 is used to derive the Ricatti equation and the step 2
is used to derive the formula for the optimal feedback gain; cf., [6, Ch.
7.3.1].

Step 1: By inspection, we find that upon setting

$\,\mathrm{d}\bar{\pi}_{t}=A^{\top}\bar{\pi}_{t}\,\mathrm{d}t-{\sf
K}_{t}\,\mathrm{d}{\bar{I}}_{t}\,,\quad\bar{\pi}_{0}=\pi_{0}$ (9)

which is as presented in the theorem statement (8a), we have
$\,\mathrm{d}\mathcal{L}_{t}\equiv 0$, and $\mathcal{C}_{1}$ reduces to

$\displaystyle\mathcal{C}_{1}=\int_{0}^{T}$
$\displaystyle\tilde{Y}_{t}^{\top}({\sf K}_{t}R{\sf
K}_{t}^{\top}\,\mathrm{d}t+\,\mathrm{d}\langle
X,X^{\top}\rangle_{t})\bar{Y}_{t}$ $\displaystyle-$
$\displaystyle\tilde{Y}_{t}^{\top}\,\mathrm{d}\big{\langle}(\bar{\pi}-X),(\bar{\pi}-X)^{\top}\big{\rangle}_{t}\bar{Y}_{t}$

It is an easy calculation to compute the quadratic variation

$\,\mathrm{d}\langle(\bar{\pi}-X),(\bar{\pi}-X)^{\top}\big{\rangle}_{t}={\sf
K}_{t}R{\sf K}_{t}^{\top}\,\mathrm{d}t+\,\mathrm{d}\langle
X,X^{\top}\rangle_{t}$

and therefore, upon defining the dynamics of $\bar{\pi}$ according to (9),

$\mathcal{C}_{1}=0\quad\text{a.s.}$

This is true for any choice of ${\cal Z}$-adapted gain process ${\sf K}$.

Among the consequences are the following pretty representations for the error
processes:

$\displaystyle\bar{{\cal E}}_{t}$
$\displaystyle=\bar{Y}_{t}^{\top}(X_{t}-\bar{\pi}_{t})+\int_{0}^{t}\bar{Y}_{\tau}^{\top}\,\mathrm{d}\mathcal{L}_{\tau}=\bar{Y}_{t}^{\top}(X_{t}-\bar{\pi}_{t})$
(10)

and similarly,

$\displaystyle\tilde{{\cal E}}_{t}$
$\displaystyle=\tilde{Y}_{t}^{\top}(X_{t}-\bar{\pi}_{t})+\int_{0}^{t}V_{\tau}\,\mathrm{d}{\bar{I}}_{\tau}$

These expressions also hold for any ${\cal Z}$-adapted ${\sf K}$.

Step 2: A formula for the gain ${\sf K}=\\{{\sf K}_{t}:t\in[0,T]\\}$ is
obtained by enforcing ${\sf E}[\mathcal{C}_{2}]=0$.

We first carry out some simplifications. It is straightforward calculation
that, with $\bar{\pi}$ defined according to (9), the integrand of
$\mathcal{C}_{2}$ is a perfect differential:

$\mathcal{C}_{2}=\int_{0}^{T}\,\mathrm{d}\big{(}\bar{{\cal
E}}_{t}\int_{0}^{t}V_{\tau}^{\top}\,\mathrm{d}{\bar{I}}_{\tau}\big{)}=\bar{{\cal
E}}_{T}\int_{0}^{T}V_{t}^{\top}\,\mathrm{d}{\bar{I}}_{t}$ (11)

The following orthogonality condition is thus obtained upon using the
representation (10) for $\bar{{\cal E}}_{T}$:

$f^{\top}{\sf
E}\Big{(}(\bar{\pi}_{T}-X_{T})\int_{0}^{T}V_{t}^{\top}\,\mathrm{d}{\bar{I}}_{t}\Big{)}={\sf
E}(\mathcal{C}_{2})=0$

Since the function $f$ is arbitrary, we must have

${\sf
E}\Big{(}(X_{T}-\bar{\pi}_{T})\int_{0}^{T}V_{t}^{\top}\,\mathrm{d}{\bar{I}}_{t}\Big{)}=0$

To obtain the formula for ${\sf K}$, the expression inside the expectation is
written as an integral—essentially by reversing the steps in finding the
perfect differential. This yields

$\displaystyle{\sf E}\Big{(}\int_{0}^{T}\big{(}{\sf K}_{t}$ $\displaystyle
R+(X_{t}-\bar{\pi}_{t})(X_{t}-\bar{\pi}_{t})^{\top}H\big{)}V_{t}\,\mathrm{d}t\Big{)}$
(12) $\displaystyle-\;\;{\sf
E}\Big{(}\int_{0}^{T}\big{(}\int_{0}^{t}V_{\tau}^{\top}\,\mathrm{d}{\bar{I}}_{\tau}\big{)}(\,\mathrm{d}\bar{\pi}_{t}-A^{\top}X_{t}\,\mathrm{d}t)\Big{)}=0$

For the equation to hold for arbitrary choices of $V$ and $\bar{I}$ (which is
unrelated to the choice of $V$), the two terms should both be zero:

$\displaystyle{\sf E}\Big{(}\int_{0}^{T}\big{(}{\sf
K}_{t}R+(X_{t}-\bar{\pi}_{t})(X_{t}-\bar{\pi}_{t})^{\top}H\big{)}V_{t}\,\mathrm{d}t\Big{)}$
$\displaystyle=0$ (13) $\displaystyle{\sf
E}\Big{(}\int_{0}^{T}\big{(}\int_{0}^{t}V_{\tau}^{\top}\,\mathrm{d}{\bar{I}}_{\tau}\big{)}(\,\mathrm{d}\bar{\pi}_{t}-A^{\top}X_{t}\,\mathrm{d}t)\Big{)}$
$\displaystyle=0$ (14)

The formula for the optimal ${\sf K}$ is obtained by solving (13). Using the
tower property of conditional expectation, because $V_{t}$ and ${\sf K}_{t}$
are both ${\cal Z}_{t}$-measurable, we have

$E\Big{(}\int_{0}^{T}({\sf K}_{t}R+{\sf
E}((X_{t}-\bar{\pi}_{t})(X_{t}-\bar{\pi}_{t})^{\top}H\mid{\cal
Z}_{t}))V_{t}\,\mathrm{d}t\Big{)}=0$

Since $V$ is an arbitrary ${\cal Z}$-adapted function, ${\sf K}_{t}$ is
uniquely determined on $L^{2}$ space:

${\sf K}_{t}=-{\sf
E}((X_{t}-\bar{\pi}_{t})(X_{t}-\bar{\pi}_{t})^{\top}H\mid{\cal
Z}_{t})R^{-1},\quad t\in[0,T]$

This gives the formula for the optimal gain ${\sf K}$.

###### Remark 3

Using the optimal gain, the equation (9) for $\bar{\pi}$ becomes

$\,\mathrm{d}\bar{\pi}_{t}=A^{\top}\bar{\pi}_{t}\,\mathrm{d}t+{\sf
E}[(X_{t}-\bar{\pi}_{t})(X_{t}-\bar{\pi}_{t})^{\top}H\mid{\cal
Z}_{t}]R^{-1}\,\mathrm{d}{\bar{I}}_{t},\quad\bar{\pi}_{0}=\pi_{0}$

The equation is not closed because we do not know ${\sf E}(X_{t}\mid{\cal
Z}_{t})=:\pi_{t}$.

One could consider closing the equation by assuming a certainty equivalence
principle that $\pi=\bar{\pi}$. In that case,

${\sf E}((X_{t}-\bar{\pi}_{t})(X_{t}-\bar{\pi}_{t})^{\top}H\mid{\cal
Z}_{t})=\text{diag}(\pi_{t})(H-{\pi}_{t}^{\top}H)^{\top}$

where $\text{diag}(\pi_{t})$ is a diagonal matrix whose diagonal entries are
the elements of the vector $\pi_{t}$, and one obtains the equation

$\,\mathrm{d}{\pi}_{t}=A^{\top}{\pi}_{t}\,\mathrm{d}t+\text{diag}(\pi_{t})(H-{\pi}_{t}^{\top}H)^{\top}R^{-1}\,\mathrm{d}I_{t},\quad{\pi}_{0}=\pi_{0}$

where $\,\mathrm{d}I_{t}=\,\mathrm{d}Z_{t}-H^{\top}\pi_{t}\,\mathrm{d}t$. This
is the equation for the Wonham filter.

## References

  * [1] R. E. Kalman and R. S. Bucy, “New results in linear filtering and prediction theory,” _Journal of basic engineering_ , vol. 83, no. 1, pp. 95–108, 1961\.
  * [2] D. Ocone and E. Pardoux, “Asymptotic stability of the optimal filter with respect to its initial condition,” _SIAM Journal on Control and Optimization_ , vol. 34, no. 1, pp. 226–243, 1996.
  * [3] E. Todorov, “Optimal control theory,” _Bayesian brain: probabilistic approaches to neural coding_ , pp. 269–298, 2006.
  * [4] E. Todorov, “General duality between optimal control and estimation,” in _2008 47th IEEE Conference on Decision and Control_ , Dec 2008, pp. 4286–4292.
  * [5] K. J. Åström, _Introduction to Stochastic Control Theory_. Academic Press, 1970.
  * [6] A. Bensoussan, _Estimation and Control of Dynamical Systems_. Springer, 2018, vol. 48.

###  -B Proof of Proposition 1

For a given affine control law $U_{t}={\sf K}_{t}^{\top}Y_{t}+V_{t}$, the ODE
(2) is a linear system with random coefficients:

$\frac{\,\mathrm{d}Y_{t}}{\,\mathrm{d}t}=-(A+H{\sf
K}_{t}^{\top})Y_{t}-HV_{t},\quad Y_{T}=f$ (15)

It admits a unique solution $Y:[0,T]\rightarrow\mathbb{R}^{d}$. Now, because
$\\{{\sf K}_{t},V_{t};t\in[0,T]\\}$ are ${\cal Z}$-adapted and $Y_{T}=f$ is
deterministic, the solution $Y_{0}$ at time $t=0$ is a ${\cal Z}_{T}$-adapted
random vector.

For $t\geq\tau$, the state transition matrix $\Phi(t,\tau)$ is defined as the
solution to the matrix ODE

$\frac{\,\mathrm{d}}{\,\mathrm{d}t}\Phi(t,\tau)=-(A+H{\sf
K}_{t}^{\top})\Phi(t,\tau),\quad\Phi(\tau,\tau)=I$ (16)

A solution of (15) is given by

$Y_{t}=\Phi(t\,;0)Y_{0}-\int_{0}^{t}\Phi(t\,;\tau)HV_{\tau}\,\mathrm{d}\tau=:\Phi_{t}Y_{0}+\eta_{t}$

Similarly, $\displaystyle U_{t}={\sf
K}_{t}^{\top}Y_{t}+V_{t}={\underbrace{(\Phi_{t}^{\top}{\sf
K}_{t})}_{\kappa_{t}}}^{\top}Y_{0}+\underbrace{({\sf
K}_{t}^{\top}\eta_{t}+V_{t})}_{\gamma_{t}}$.

###  -C Stochastic integrals

Recall the filtrations: ${\cal
F}_{t}:=\sigma(X_{0},B_{\tau},W_{\tau}:0\leq\tau\leq t)$ and ${\cal
Z}_{t}=\sigma(Z_{\tau}:\tau\in[0,t])$, $t\in[0,T]$. There are two types of
stochastic processes:

  1. 1.

Adapted stochastic processes: $W,\>B,\>X\in{\cal F}$ and
$Z,\>\bar{\pi},\>\bar{I},\>{\sf K},\>V\in{\cal Z}$.

  2. 2.

Non-adapted stochastic processes: $Y,\>U,\>{\cal E}$, and their optimal and
perturbed counterparts, $\bar{Y},\>\bar{U},\>\bar{{\cal E}}$ and
$\tilde{Y},\>\tilde{U},\>\tilde{{\cal E}}$, respectively.

Now, allowing only for admissible control inputs from $\mathcal{U}$ (see (3)),
a generic stochastic process considered in this paper is expressed as
$\phi_{t}=F^{\top}\xi_{t}+\alpha_{t}$, where $F\in{\cal Z}_{T}$ and
$\xi_{t},\alpha_{t}\in{\cal F}_{t}$ for each $t$ (Prop. 1 and Prop. 4).

###### Definition 1

Consider two stochastic processes $\phi_{t}=F^{\top}\xi_{t}+\alpha_{t}$ and
$\psi_{t}=G^{\top}\zeta_{t}+\beta_{t}$, where
$\xi_{t},\alpha_{t},\zeta_{t},\beta_{t}\in{\cal F}_{t}$ are piecewise
continuous functions of time $t$ with at most finitely many jumps and $F,\;G$
are bounded random vectors. Consider a partition
$\Pi^{N}_{[0,t]}=\\{0=t_{0}<t_{1}<\ldots<t_{N}=t\\}$ with
$\Delta:=\displaystyle\max_{i}(t_{i}-t_{i-1})$. Then,

$\displaystyle\int_{0}^{t}\phi_{\tau}\,\mathrm{d}\psi_{\tau}$
$\displaystyle:=\lim_{\Delta\rightarrow
0}\sum_{i=1}^{N}\phi_{t_{i-1}}(\psi_{t_{i}}-\psi_{t_{i-1}})$
$\displaystyle\langle\phi,\psi\rangle_{t}$
$\displaystyle:=\lim_{\Delta\rightarrow
0}\sum_{i=1}^{N}(\phi_{t_{i}}-\phi_{t_{i-1}})(\psi_{t_{i}}-\psi_{t_{i-1}})$

provided the respective limits exist in $L^{2}$.

###### Proposition 3

Consider the two stochastic processes $\\{\phi_{t},\psi_{t}\\}$ as defined in
Defn. 1. Then

$\displaystyle\int_{0}^{t}\phi_{\tau}\,\mathrm{d}\psi_{\tau}\;\
\stackrel{{\scriptstyle L^{2}}}{{=}}\ $
$\displaystyle\;F^{\top}\Big{(}\int_{0}^{t}\xi_{\tau}\,\mathrm{d}\zeta_{\tau}^{\top}\Big{)}G+F^{\top}\Big{(}\int_{0}^{t}\xi_{\tau}\,\mathrm{d}\beta_{\tau}\Big{)}$
$\displaystyle\quad\quad+\;\;G^{\top}\Big{(}\int_{0}^{t}\alpha_{\tau}\,\mathrm{d}\zeta_{\tau}\Big{)}+\int_{0}^{t}\alpha_{\tau}\,\mathrm{d}\beta_{\tau}$
$\displaystyle\langle\phi,\psi\rangle_{t}\;\ \stackrel{{\scriptstyle
L^{2}}}{{=}}\ $
$\displaystyle\;F^{\top}\langle\xi,\zeta^{\top}\rangle_{t}G+F^{\top}\langle\xi,\beta\rangle_{t}+\langle\alpha,\zeta^{\top}\rangle_{t}G+\langle\alpha,\beta\rangle_{t}$

where the integrals on the right-hand side are standard Itô-integrals.
Moreover, the following Itô product formula holds:

$\phi_{t}\psi_{t}-\phi_{0}\psi_{0}=\int_{0}^{t}\phi_{\tau}\,\mathrm{d}\psi_{\tau}+\int_{0}^{t}\psi_{\tau}\,\mathrm{d}\phi_{\tau}+\langle\phi,\psi\rangle_{t}$

###### Proof:

The pre-limit is evaluated as

$\displaystyle\sum_{i=1}^{N}$
$\displaystyle\phi_{t_{i-1}}(\psi_{t_{i}}-\psi_{t_{i-1}})$ $\displaystyle=$
$\displaystyle
F^{\top}\Big{(}\sum_{i=1}^{N}\xi_{t_{i-1}}(\zeta_{t_{i}}^{\top}-\zeta_{t_{i-1}}^{\top})\Big{)}G+F^{\top}\sum_{i=1}^{N}\xi_{t_{i-1}}(\beta_{t_{i}}-\beta_{t_{i-1}})$
$\displaystyle\quad+\;G^{\top}\sum_{i=1}^{N}\alpha_{t_{i-1}}(\zeta_{t_{i}}-\zeta_{t_{i-1}})+\sum_{i=1}^{N}\alpha_{t_{i-1}}(\beta_{t_{i}}-\beta_{t_{i-1}})$

The result is obtained upon letting $\Delta\to 0$. For example,

$\lim_{\Delta\to
0}\;\;\sum_{i=1}^{N}\xi_{t_{i-1}}(\zeta_{t_{i}}^{\top}-\zeta_{t_{i-1}}^{\top})\;\;\
\stackrel{{\scriptstyle L^{2}}}{{=}}\
\;\;\int_{0}^{t}\xi_{\tau}\,\mathrm{d}\zeta_{\tau}^{\top}$

and therefore, because $F,\;G$ are bounded,

$\lim_{\Delta\to
0}\;\;F^{\top}\Big{(}\sum_{i=1}^{N}\xi_{t_{i-1}}(\zeta_{t_{i}}^{\top}-\zeta_{t_{i-1}}^{\top})\Big{)}G\;\;\
\stackrel{{\scriptstyle L^{2}}}{{=}}\
\;\;F^{\top}\Big{(}\int_{0}^{t}\xi_{\tau}\,\mathrm{d}\zeta_{\tau}^{\top}\Big{)}G$

The calculation for the cross variation is analogous. The product rule is
proved by using the following identity (which holds for arbitrary stochastic
processes):

$\displaystyle(\phi_{t_{i}}\psi_{t_{i}}-\phi_{t_{i-1}}\psi_{t_{i-1}})=$
$\displaystyle\phi_{t_{i-1}}(\psi_{t_{i}}-\psi_{t_{i-1}})+\psi_{t_{i-1}}(\phi_{t_{i}}-\phi_{t_{i-1}})$
$\displaystyle+(\phi_{t_{i}}-\phi_{t_{i-1}})(\psi_{t_{i}}-\psi_{t_{i-1}})$

Summing over $i$ and taking the limit as $\Delta\to 0$ yields the result.

###### Remark 4

The product rule is the only type of Itô formula used in the various proofs in
this paper. This is because of the linear quadratic nature of the optimal
control problem in finite-state-space settings. The following differential
notation is frequently used:

$\,\mathrm{d}(\phi_{t}\psi_{t})=\phi_{t}\,\mathrm{d}\psi_{t}+\psi_{t}\,\mathrm{d}\phi_{t}+\,\mathrm{d}\langle\phi,\psi\rangle_{t}$
(17)

###  -D Proof of Proposition 2

The following identity is established in this section for any admissible
control:

$J(U)={\mathchoice{\genfrac{}{}{}{1}{1}{2}}{\genfrac{}{}{}{2}{1}{2}}{\genfrac{}{}{}{3}{1}{2}}{\genfrac{}{}{}{4}{1}{2}}}{\sf
E}[|S_{T}-f^{\top}X_{T}|^{2}]$

The lefhand-side is the optimal control objective as defined in (5a). The
righthand-side is the mean-squared error. Recall that $S_{T}$ is the linear
estimate as defined by (4), $f\in\mathbb{R}^{d}$ is deterministic, and $X_{T}$
is the hidden state at time $T$.

The approach is to use the dual ODE (2) to express the mean-squared error as
an integral. The product formula (17) is used to obtain

$\displaystyle\,\mathrm{d}(Y_{t}^{\top}X_{t})$
$\displaystyle=\,\mathrm{d}Y_{t}^{\top}X_{t}+Y_{t}^{\top}\,\mathrm{d}X_{t}+\,\mathrm{d}\langle
Y^{\top}\\!\\!\\!,\;X\rangle_{t}$
$\displaystyle=(-Y_{t}^{\top}A^{\top}-U_{t}^{\top}H^{\top})X_{t}\,\mathrm{d}t+Y_{t}^{\top}(A^{\top}X_{t}\,\mathrm{d}t+\,\mathrm{d}B_{t})$
$\displaystyle=-U_{t}^{\top}H^{\top}X_{t}\,\mathrm{d}t+Y_{t}^{\top}\,\mathrm{d}B_{t}$

which is shorthand for the integral equation

$Y_{T}^{\top}X_{T}=Y_{0}^{\top}X_{0}+\int_{0}^{T}U_{t}^{\top}H^{\top}X_{t}\,\mathrm{d}t+Y_{t}^{\top}\,\mathrm{d}B_{t}$

With $Y_{T}=f$, upon subtracting this equation from (4),

$f^{\top}X_{T}-S_{T}=(Y_{0}^{\top}X_{0}-Y_{0}^{\top}\pi_{0})+\int_{0}^{T}U_{t}^{\top}\,\mathrm{d}W_{t}+Y_{t}^{\top}\,\mathrm{d}B_{t}$

With the definition of the error process ${\cal E}_{t}$ in (6), the left-hand
side is identified: $f^{\top}X_{T}-S_{T}={\cal E}_{T}$.

The product formula (17) is then used to obtain

${\mathchoice{\genfrac{}{}{}{1}{1}{2}}{\genfrac{}{}{}{2}{1}{2}}{\genfrac{}{}{}{3}{1}{2}}{\genfrac{}{}{}{4}{1}{2}}}{\cal
E}_{T}^{2}={\mathchoice{\genfrac{}{}{}{1}{1}{2}}{\genfrac{}{}{}{2}{1}{2}}{\genfrac{}{}{}{3}{1}{2}}{\genfrac{}{}{}{4}{1}{2}}}{\cal
E}_{0}^{2}+\int_{0}^{T}{\cal E}_{t}\,\mathrm{d}{\cal
E}_{t}+{\mathchoice{\genfrac{}{}{}{1}{1}{2}}{\genfrac{}{}{}{2}{1}{2}}{\genfrac{}{}{}{3}{1}{2}}{\genfrac{}{}{}{4}{1}{2}}}\langle{\cal
E},{\cal E}\rangle_{T}$

The integral form (5a) of the objective function follows from evaluating each
of the terms as summarized in the following.

###### Proposition 4

Consider the error process ${\cal E}=\\{{\cal E}_{t}:t\in[0,T]\\}$ defined in
(6). Suppose $U=\\{U_{t}:t\in[0,T]\\}$ is any admissible control. Then

$\displaystyle{\cal E}_{0}^{2}$
$\displaystyle=|Y_{0}^{\top}X_{0}-Y_{0}^{\top}\pi_{0}|^{2}$
$\displaystyle\int_{0}^{T}{\cal E}_{t}\,\mathrm{d}{\cal E}_{t}$
$\displaystyle=\int_{0}^{T}{\cal
E}_{t}U_{t}^{\top}\,\mathrm{d}W_{t}+\int_{0}^{T}{\cal
E}_{t}Y_{t}^{\top}\,\mathrm{d}B_{t}$ $\displaystyle\langle{\cal E},{\cal
E}\rangle_{T}$
$\displaystyle=\int_{0}^{T}U_{t}^{2}\,\mathrm{d}t+Y_{t}^{\top}\,\mathrm{d}\langle
X,X\rangle_{t}Y_{t}$

The proof is the direct application of Prop. 1 and Prop. 3.