Canonical Analysis of Gravity with Dynamical Determinant of Metric-General
Case

J. Klusoň 111Email addresses<EMAIL_ADDRESS>(J. Klusoň)

Department of Theoretical Physics and Astrophysics, Faculty of Science,
Masaryk University, Kotlářská 2, 611 37, Brno, Czech Republic

We analyse general form of theory with the dynamical determinant of metric. We
show that due to the presence of general function of determinant that
multiplies scalar curvature Hamiltonian constraint is either second class
constraint or it is necessary to impose condition of transversality on
parameters of spatial diffeomorphism.

##  1 Introduction and Summary

Theories with restricted diffeomorphism invariance attached great interest
recently as an alternative solution of the cosmological constant problem.
Famous example of such theories are unimodular theories of gravity where the
determinant of the metric is fixed [1, 2, 3, 4]. Clearly if we demand that the
action is invariant under change of coordinates we find that it is not
invariant under full diffeomorphism but instead under restricted one which
preserves determinant of the metric. The restriction imposed on allowed
diffeomorphism transformations has non-trivial consequences on the canonical
structure of theory as was shown for example in [5, 6, 7, 8].

More precisely, General Relativity is invariant under general diffeomorphism
(Diff) transformations. On the other hand it was shown long time ago in [12]
that symmetry group for a consistent description of the massless graviton can
be maximal subgroup of Diff that is known as TDiff since the parameter that
characterizes the infinitesimal form of diffeomorphism transformation is
transverse one which means that it obeys the condition

$\partial_{\mu}\xi^{\mu}=0\ .$ (1)

As we argued above characteristic property of theories invariant under
restricted diffeomorphism is the fact that determinant of metric transforms as
scalar under diffeomorphism transformation rather than as scalar density. Then
there is a natural question whether it is possible to consider theories where
determinant of metric appears as ordinary scalar. Such a model was firstly
discussed in [9] and the most general form of such theory was presented in
[10]. The quantum behaviour of such a theory was studied in [10] and
confrontation of this theory with observation was analyzed in [11].

The fact that the most general form of TDiff invariant theory contains kinetic
term for determinant makes it also very interesting from the Hamiltonian point
of view. In our previous paper [14] we studied special form of this action
where the function in front of the scalar curvature is equal to square root of
the determinant of metric. We found corresponding Hamiltonian and identified
primary and secondary constraints. We also showed that Hamiltonian constraint
is tertiary constraint which arises from the requirement of the preservation
of secondary constraints. Then we calculated Poisson brackets between these
constraints and we showed that they closed. In other words the Hamiltonian
constraint is the first class constraint too.

In this paper we would like to study the most general form of TDiff invariant
action when we consider general function of determinant of metric in front of
the scalar curvature in the action. It turns out that the presence of this
general function has crucial impact on the consistency of theory. In more
details, in order to proceed to the canonical formalism we should implement
$3+1$ decomposition of scalar curvature, for review see [15]. Then presence of
terms proportional to covariant derivatives in this expansion has crucial
impact on the canonical structure of theory as we show in the bulk of this
paper. More precisely, we find Hamiltonian that contains kinetic terms for
spatial components of metric and for lapse while there are only $n-1$ primary
constraints (in $n-$dimensional space-time) corresponding to the vanishing
conjugate momenta to shift functions. Then the requirement of preservation of
the primary constraints leads to an emergence of $n-1$ secondary constraints
which are spatial diffeomorphism constraints. We calculate Poisson brackets
between these constraints and canonical variables. Finally the requirement of
the preservation of these constraints leads to the Hamiltonian constraint. In
other words, the Hamiltonian for TDiff invariant gravity is again given as sum
of constraints. The crucial difference is in the fact that when we calculate
Poisson bracket between Hamiltonian constraints we find that it is not equal
to linear combinations of spatial diffeomorphism constraints. Rather we find
that it is equal to some complicated functions of canonical momenta. This
result have two interpretations. The first one is based on the presumption
that time evolution of the system is governed by the original Hamiltonian with
primary constraints included only. In this case the requirement of the
preservation of Hamiltonian constraint would implies an existence of
additional constraint. Then we should again study time evolution of this
quaternary constraint and so on. We should then expect an infinite number of
constraints to be generated which is clearly non-consistent.

The second interpretation is based on an existence of extended Hamiltonian
which is defined as Hamiltonian where all constraints are included [16]. In
this case the fact that Poisson bracket between smeared form of Hamiltonian
constraints does not vanish on constraint surface implies that Hamiltonian
constraint is second class constraint with themselves. This is not so
surprising as it sounds if we recognize that Hamiltonian constraint
corresponds to $\infty^{n-1}$ constraints defined in $\mathbf{R}^{n-1}$.
Moreover, similar situation occurs in case of non-projectable Hořava-Lifshitz
gravity [17], where it was shown in case when lapse is space-dependent that
the theory is inconsistent [18, 19, 20] in the sense that the Hamiltonian
constraints are second class constraints among themselves. The situation was
solved by including appropriate kinetic term for lapse [21], see also [22],
however it is not clear whether similar procedure can be applied for the model
studied in this paper.

As we wrote above requirement of the preservation of spatial diffeomorphism
constraints led to an emergence of Hamiltonian constraint however this is not
single possibility. We show that the requirement of the preservation of
spatial diffeomorphism constraint can be solved by imposing restriction on the
parameters of spatial diffeomorphism, namely $\partial_{i}\xi^{i}=0$ where
$i=1,\dots,n$. Note that this condition can be imposed since $\mathcal{H}_{T}$
is scalar not scalar density which is sharp difference with respect to general
relativity case. Further, the spatial diffeomorphism transverse condition
implies that there is no Hamiltonian constraint and hence all problems that
were mentioned in previous paragraph are eliminated. On the other hand if we
combine TDiff condition $\partial_{\mu}\xi^{\mu}=0$ with
$\partial_{i}\xi^{i}=0$ we find that $\xi^{0}$ does not depend on time. In
some way this can be interpreted as satisfactory result since there is no
Hamiltonian constraint on the other hand it suggests that Hamiltonian
formalism cannot reproduce TDiff constraint $\partial_{\mu}\xi^{\mu}=0$.

Let us outline our results. We found Hamiltonian for general TDiff invariant
theory which is characterized by two general functions $F(g),G(g)$ where no
restrictions on the form of these functions were imposed. We identified
primary and secondary constraints and we argued that the requirement of the
preservation of secondary constraint leads either to the spatial
diffeomorphism transverse condition or to an emergence of tertiary Hamiltonian
constraint. We argued that the second possibility leads to the theory with not
well defined canonical structure. On the other hand imposing the first
possibility leads to consistent theory but with absent time reparametrization.
We mean that both these problems deserve further investigation and hence
theories invariant under TDiff can serve as good laboratories for the
application of canonical formalism on more complicated theories than the fully
diffeomorphism invariant ones.

This paper is organized as follows. In the next section (2) we introduce
theory with dynamical determinant of metric and find corresponding
Hamiltonian. Then in section (3) we study stability of constraints and discuss
conditions when the theory is well defined.

##  2 General TDiff-Invariant Action

In this section we introduce basic formulation of gravity with time dependent
metric. This action has a form [10]

$\displaystyle S=\int d^{n}x\mathcal{L}\
,\quad\mathcal{L}=\frac{1}{\kappa}F(\sqrt{-g})\sqrt{-g}[R(g)+\frac{G(\sqrt{-g})}{\sqrt{-g}g^{2}}\partial_{\mu}g^{\mu\nu}\partial_{\nu}g]\
,\quad$ $\displaystyle g\equiv\det g\ ,\quad\kappa=16\pi\ ,$ (2)

where $F$ and $G$ are general functions of $\sqrt{-g}$. Note that we work in
$n$-dimensional space-time with metric signature $(-,+,\dots,+)$ and
$\mu,\nu=0,1,\dots,n$. Since under general transformations
$x^{\prime\mu}=x^{\mu}+\xi^{\mu}(x)$ the metric tensor transforms as

$g^{\prime}_{\mu\nu}(x^{\prime})=g_{\mu\nu}(x)-g_{\mu\sigma}(x)\partial_{\nu}\xi^{\sigma}(x)-\partial_{\mu}\xi^{\sigma}g_{\sigma\nu}(x)$
(3)

it is clear that the determinant transforms as

$g^{\prime}(x^{\prime})=g(x)-\partial_{\mu}\xi^{\mu}g(x)\ .$ (4)

Then clearly the action (2) is not invariant under full diffeomorphism but
instead under restricted ones where parameters $\xi^{\mu}$ obey the conditions

$\partial_{\mu}\xi^{\mu}=0\ .$ (5)

Due to the presence of kinetic term for $g$ it is clearly very interesting to
find Hamiltonian form of the action (2). Note that the special case of the
action (2) with $F(\sqrt{-g})=1$ was studied recently in [14]. In this paper
we would like to relax this condition and consider general function $F$. We
will see that it will lead to important consequences.

To proceed to the canonical formulation we use the well known $3+1$ formalism
that is the fundamental ingredient of the Hamiltonian formalism of any theory
of gravity 222For recent review, see [15].. We consider $n$ dimensional
manifold $\mathcal{M}$ with the coordinates $x^{\mu}\ ,\mu=0,\dots,n-1$ and
where $x^{\mu}=(t,\mathbf{x})\ ,\mathbf{x}=(x^{1},x^{2},\dots,x^{n-1})$. We
presume that this space-time is endowed with the metric $g_{\mu\nu}(x^{\rho})$
with signature $(-,+,\dots,+)$. Suppose that $\mathcal{M}$ can be foliated by
a family of space-like surfaces $\Sigma_{t}$ defined by
$t=x^{0}=\mathrm{const}$. Let $h_{ij},i,j=1,2,\dots,n-1$ denotes the metric on
$\Sigma_{t}$ with inverse $h^{ij}$ so that $h_{ij}h^{jk}=\delta_{i}^{k}$. We
further introduce the operator $\nabla_{i}$ that is covariant derivative
defined with the metric $h_{ij}$. We also define the lapse function
$N=1/\sqrt{-g^{00}}$ and the shift function $N^{i}=-g^{0i}/g^{00}$. In terms
of these variables we write the components of the metric $g_{\mu\nu}$ as

$\displaystyle g_{00}=-N^{2}+N_{i}h^{ij}N_{j}\ ,\quad g_{0i}=N_{i}\ ,\quad
g_{ij}=h_{ij}\ ,$ $\displaystyle g^{00}=-\frac{1}{N^{2}}\ ,\quad
g^{0i}=\frac{N^{i}}{N^{2}}\ ,\quad g^{ij}=h^{ij}-\frac{N^{i}N^{j}}{N^{2}}\ .$

and hence $g=-N^{2}\det h_{ij}$. We further have following decomposition of
$R$ in the form

$\displaystyle
R=K^{ij}K_{ij}-K^{2}+r+2\tilde{\nabla}_{\mu}[\tilde{n}^{\mu}K]-\frac{2}{N}\nabla_{i}\nabla^{i}N\
,$ $\displaystyle
K_{ij}=\frac{1}{2N}(\partial_{t}h_{ij}-\nabla_{i}N_{j}-\nabla_{j}N_{i})\
,\quad n^{0}=\sqrt{-g^{00}}\ ,\quad n^{i}=-\frac{g^{0i}}{\sqrt{-g^{00}}}\ ,$

and where $r$ is scalar curvature defined with $h_{ij}$ and where
$\tilde{\nabla}_{\mu}$ is covariant derivative compatible with $g_{\mu\nu}$ so
that $\tilde{\nabla}_{\mu}g_{\rho\sigma}=0$ while $\nabla_{i}$ is covariant
derivative compatible with the metric $h_{ij}$. Note that we can also write

$\displaystyle\tilde{\nabla}_{\mu}[n^{\mu}K]=\frac{1}{\sqrt{-g}}\partial_{\mu}[\sqrt{-g}n^{\mu}K]\
,$
$\displaystyle\frac{1}{g^{2}}\partial_{\mu}gg^{\mu\nu}\partial_{\nu}g=-\frac{1}{g^{2}}\frac{1}{N^{2}}(\partial_{0}g-N^{i}\partial_{i}g)^{2}+\frac{1}{g^{2}}h^{ij}\partial_{i}g\partial_{j}g\
.$

As in our previous paper [14] we perform following manipulation with the
kinetic term for determinant of metric

$\displaystyle\frac{1}{Ng}(\partial_{0}g-N^{i}\partial_{i}g)=\frac{2}{N^{2}}\partial_{0}N+2\partial_{i}(\frac{N^{i}}{N})+2K_{ij}h^{ji}\
$

and also

$\partial_{0}h_{ij}h^{ij}=2NK_{ij}h^{ji}+2\nabla_{i}N_{j}h^{ji}\ .$ (10)

Then the action (2) has the form

$\displaystyle S=\frac{1}{\kappa}\int
d^{n}x[F\sqrt{-g}K_{ij}\mathcal{G}^{ijkl}K_{kl}+\sqrt{-g}Fr+2F\partial_{\mu}(N\sqrt{h}n^{\mu}K)-2F\sqrt{h}\nabla_{i}\nabla^{i}N-$
$\displaystyle-\frac{G}{g^{2}N^{2}}(\partial_{0}g-N^{i}\partial_{i}g)^{2}+\frac{G}{g^{2}}h^{ij}\partial_{i}g\partial_{j}g]=$
$\displaystyle=\frac{1}{\kappa}\int
d^{n}x\left[F\sqrt{-g}K_{ij}\mathcal{G}^{ijkl}K_{kl}+F\sqrt{-g}r+\frac{FG}{g^{2}}h^{ij}\partial_{i}g\partial_{j}g+\right.$
$\displaystyle-4F^{\prime}g\sqrt{-g}\left(\frac{1}{N^{2}}\partial_{0}N+\partial_{i}(\frac{N^{i}}{N})+K\right)K-$
$\displaystyle\left.-4FG\left(\frac{1}{N^{2}}\partial_{0}N+2\partial_{i}(\frac{N^{i}}{N})+K\right)^{2}-2F\nabla_{i}\nabla^{i}\sqrt{-g}\right]\
,$

where $\mathcal{G}^{ijkl}$ is defined as

$\mathcal{G}^{ijkl}=\frac{1}{2}(h^{ik}h^{jl}+h^{il}h^{jk})-h^{ij}h^{kl}\ .$
(12)

The crucial point of the canonical analysis is that the action (2) can be
rewritten into following form

$\displaystyle S=\frac{1}{\kappa}\int
d^{n}x[\sqrt{-g}FK_{ij}\mathcal{M}^{ijkl}K_{kl}-4FG\tilde{X}^{2}+F\sqrt{-g}r+\frac{FG}{g^{2}}h^{ij}\partial_{i}g\partial_{j}g-2F\nabla_{i}\nabla^{i}\sqrt{-g}]\
,$

where we defined $\tilde{X}$ as

$\tilde{X}=\frac{\partial_{0}N}{N^{2}}+\partial_{i}(\frac{N^{i}}{N})+\left(1+\frac{1}{2}\frac{F^{\prime}g\sqrt{-g}}{FG}\right)K\
\quad$ (14)

and metric $\mathcal{M}^{ijkl}$ in the form

$\mathcal{M}^{ijkl}=\mathcal{G}^{ijkl}-\frac{F^{\prime
2}g^{3}}{F^{2}G\sqrt{-g}}h^{ij}h^{kl}=\frac{1}{2}(h^{ik}h^{jl}+h^{il}h^{jk})-(1+\frac{F^{\prime
2}g^{3}}{F^{2}G\sqrt{-g}})h^{ij}h^{kl}\ .$ (15)

From the action (2) we get conjugate momenta

$\displaystyle\pi^{ij}=\frac{\sqrt{h}F}{\kappa}\mathcal{M}^{ijkl}K_{kl}-\frac{8}{\kappa}FG\tilde{X}(1+\frac{1}{2}\frac{F^{\prime}g\sqrt{-g}}{FG})\frac{1}{2N}h^{ij}\
,$
$\displaystyle\pi_{N}=\frac{\partial\mathcal{L}}{\partial\partial_{0}N}=-\frac{8}{\kappa}FG\frac{1}{N^{2}}\tilde{X}\
.$

Using these relations we obtain Hamiltonian density in the form

$\displaystyle\mathcal{H}=\pi^{ij}\partial_{0}h_{ij}+\pi_{N}\partial_{0}N-\mathcal{L}=$
$\displaystyle=\frac{1}{\kappa}\sqrt{-g}FK_{ij}\mathcal{M}^{ijkl}K_{kl}-\frac{4}{\kappa}FG\tilde{X}^{2}-\sqrt{-g}Fr-$
$\displaystyle-\frac{FG}{g^{2}}h^{ij}\partial_{i}g\partial_{j}g+2F\nabla_{i}\nabla^{i}\sqrt{-g}+N^{i}\mathcal{H}_{i}\
,$

where we implicitly used integration by parts to introduce $\mathcal{H}_{i}$
as

$\mathcal{H}_{i}=-2\nabla_{j}\pi^{jk}h_{ki}+\partial_{i}\pi_{N}N+2\pi_{N}\partial_{i}N\
.$ (18)

It is clear that we have to express Hamiltonian density in terms of canonical
variables $\pi^{ij},\pi_{N}$ instead of $K_{ij}$ and $\partial_{0}N$. To do
this we use relation between $\tilde{X}$ and $\pi_{N}$ given in (2) to replace
$\tilde{X}$ in the definition of $\pi^{ij}$ to get

$\displaystyle\pi^{ij}-\frac{1}{2}(1+\frac{1}{2}\frac{F^{\prime}g\sqrt{-g}}{FG})\pi_{N}Nh^{ij}=\frac{\sqrt{h}}{\kappa}F\mathcal{M}^{ijkl}K_{kl}\
.$

In this paper we will presume that the matrix $\mathcal{M}^{ijkl}$ is non-
singular with inverse matrix $\mathcal{M}_{ijkl}$ that obeys the relation

$\mathcal{M}_{ijkl}\mathcal{M}^{klmn}=\frac{1}{2}(\delta_{i}^{m}\delta_{j}^{n}+\delta_{i}^{n}\delta_{j}^{m})\
.$ (20)

With this presumption we can express $K_{ij}$ from (2) as

$\displaystyle K_{ij}=\frac{\kappa}{\sqrt{h}F}\mathcal{M}_{ijkl}\Pi^{kl}\
,\quad\Pi^{ij}=\pi^{ij}-\frac{1}{2}(1+\frac{1}{2}\frac{F^{\prime}g\sqrt{-g}}{FG})\pi_{N}Nh^{ij}\
.$

Before we continue further we determine explicit form of the matrix
$\mathcal{M}_{ijkl}$. Let us presume that it has the form

$\mathcal{M}_{ijkl}=\frac{1}{2}(h_{ik}h_{jl}+h_{il}h_{jk})-\mathbf{D}h_{ij}h_{kl}\
.$ (22)

Then from (20) we can determine $\mathbf{D}$ to be equal to

$\displaystyle\mathbf{D}=\frac{(1+\frac{F^{\prime
2}g^{3}}{F^{2}G\sqrt{-g}})}{(n-1)(1+\frac{F^{\prime
2}g^{3}}{F^{2}G\sqrt{-g}})-1}\ .$

Note that for $F^{\prime}=0$, $\mathbf{D}=-\frac{1}{n-2}$ and hence
$\mathcal{M}_{ijkl}=\mathcal{G}_{ijkl}=\frac{1}{2}(h_{ik}h_{jl}+h_{il}h_{jk})-\frac{1}{n-2}h_{ij}h_{kl}$
that is well known form of de Witt inverse metric. Then using (2) in (2) we
obtain final form of Hamiltonian as function of canonical variables

$\displaystyle H=\int d^{n-1}\mathbf{x}\mathcal{H}\ ,$
$\displaystyle\mathcal{H}=\frac{\kappa}{\sqrt{h}F}N\Pi^{ij}\mathcal{M}_{ijkl}\Pi^{kl}-\frac{\kappa}{16}\frac{1}{FG}N^{4}\pi_{N}^{2}-N\sqrt{h}Fr-\frac{FG}{g^{2}}h^{ij}\partial_{i}g\partial_{j}g+$
$\displaystyle+2F\nabla_{i}\nabla^{i}\sqrt{-g}+N^{i}\mathcal{H}_{i}\equiv\mathcal{H}_{T}+N^{i}\mathcal{H}_{i}\
.$

This is final form of Hamiltonian for TDiff invariant gravity.

##  3 Stability of Constraints

Since original Lagrangian does not contain time derivative of shift functions
$N^{i}$ it is clear that corresponding conjugate momenta are absent.
Alternatively we say that there are primary constraints of theory

$\pi_{i}(\mathbf{x})\approx 0\
,\quad\left\\{N^{i}(\mathbf{x}),\pi_{j}(\mathbf{y})\right\\}=\delta^{i}_{j}\delta(\mathbf{x}-\mathbf{y})\
.$ (25)

By definition the constraint is stable if it is preserved during time
evolution of the system

$\partial_{t}\pi_{i}(\mathbf{x})=\left\\{\pi_{i}(\mathbf{x}),H\right\\}=-\mathcal{H}_{i}(\mathbf{x})\approx
0$ (26)

using Poisson brackets given in (25). We see that the requirement of the
preservation of constraint $\pi_{i}\approx 0$ implies secondary constraints

$\mathcal{H}_{i}(\mathbf{x})\approx 0\ .$ (27)

For further purposes we introduce its smeared form defined as

$\mathbf{T}_{S}(\xi)=\int d^{n-1}\mathbf{x}\xi^{i}\mathcal{H}_{i}$ (28)

and calculate Poisson brackets between $\mathbf{T}_{S}(\xi)$ and fundamental
fields

$\displaystyle\left\\{\mathbf{T}_{S}(\xi),h_{ij}\right\\}=-\partial_{m}h_{ij}\xi^{m}-\partial_{i}\xi^{m}h_{mj}-h_{im}\partial_{j}\xi^{m}\
,$
$\displaystyle\left\\{\mathbf{T}_{S}(\xi),N\right\\}=-\xi^{i}\partial_{i}N+\partial_{i}\xi^{i}N\
,$
$\displaystyle\left\\{\mathbf{T}_{S}(\xi),\pi^{ij}\right\\}=-\partial_{m}(\xi^{m}\pi^{ij})+\partial_{m}\xi^{i}\pi^{mj}+\pi^{im}\partial_{m}\xi^{j}\
,$
$\displaystyle\left\\{\mathbf{T}_{S}(\xi),\pi_{N}\right\\}=-\partial_{i}\pi_{N}\xi^{i}-2\pi_{N}\partial_{i}\xi^{i}\
,$
$\displaystyle\left\\{\mathbf{T}_{S}(\xi)h\right\\}=-\partial_{m}h\xi^{m}-2\partial_{i}\xi^{i}h\
,$
$\displaystyle\left\\{\mathbf{T}_{S}(\xi),N^{2}h\right\\}=-\xi^{m}\partial_{m}(N^{2}h)\
,$ $\displaystyle\left\\{\mathbf{T}_{S}(\xi),r\right\\}=-\xi^{m}\partial_{m}r\
.$

Using these basic Poisson brackets we also get

$\displaystyle\left\\{\mathbf{T}_{S}(\xi^{i}),g\right\\}=\left\\{\mathbf{T}_{S}(\xi^{i}),N\sqrt{g}\right\\}=-\xi^{m}\partial_{m}g\
,$
$\displaystyle\left\\{\mathbf{T}_{S}(\xi^{i}),\Pi^{ij}\right\\}=-\partial_{m}\xi^{m}\Pi^{ij}-\xi^{m}\partial_{m}\Pi^{ij}+\partial_{m}\xi^{i}\Pi^{mj}+\Pi^{im}\partial_{m}\xi^{j}\
.$

The first line in (3) gives an important result that says that $g$ behaves as
scalar under spatial diffeomorphism transformations. Further, the expression
on the second line in (3) suggests that $N$ behaves as tensor density of
weight $-1$. This fact is supported by following observation. Since
$\sqrt{-g}=Nh$ behaves as scalar under spatial diffeomorphism it is natural to
demand that

$\nabla_{i}(\sqrt{-g})=\nabla_{i}(N\sqrt{h})=\partial_{i}(\sqrt{h}N)\ .$ (31)

On the other hand from definition of covariant derivative ($\nabla_{i}h=0$) we
get

$\displaystyle\nabla_{i}N=\frac{1}{\sqrt{h}}(\partial_{i}(N\sqrt{h}))=\frac{1}{\sqrt{h}}(\partial_{i}N\sqrt{h}+N\partial_{i}\sqrt{h})=\partial_{i}N+\Gamma^{k}_{ki}N$

which is consistent with the Poisson bracket between $\mathbf{T}_{S}(\xi)$ and
$N$. Then we can write

$\displaystyle
F\nabla_{i}\nabla^{i}\sqrt{-g}=F\nabla_{i}[h^{ij}\partial_{j}\sqrt{-g}]=F\frac{1}{\sqrt{h}}\partial_{i}[\sqrt{h}h^{ij}\partial_{j}\sqrt{-g}]$

and hence

$\displaystyle\left\\{\mathbf{T}_{S}(\xi),F\nabla_{i}\nabla^{i}\sqrt{-g}\right\\}=-\xi^{m}\partial_{m}\left[\frac{F}{\sqrt{h}}\partial_{i}[\sqrt{h}h^{ij}\partial_{j}\sqrt{-g}]\right]\
.$

Now collecting all terms together we get that

$\displaystyle\left\\{\mathbf{T}_{S}(\xi),\mathcal{H}_{T}\right\\}=-\xi^{m}\partial_{m}\mathcal{H}_{T}\
$

that shows that $\mathcal{H}_{T}$ behaves as scalar under spatial
diffeomorphism transformations. Note that this is an important difference with
respect to the Hamiltonian constraint in General relativity which is scalar
density. Now we can study time evolution of the constraint
$\mathcal{H}_{i}\approx 0$ or its smeared form. Note that the time evolution
of constraint $\mathbf{T}_{S}(\xi)$ has the form

$\displaystyle\partial_{t}\mathbf{T}_{S}(\xi)=\left\\{\mathbf{T}_{S}(\xi),H_{T}\right\\}=\int
d\mathbf{y}\left\\{\mathbf{T}_{S}(\xi),\mathcal{H}(\mathbf{y})\right\\}+\left\\{\mathbf{T}_{S}(\xi),\mathbf{T}_{S}(N^{j})\right\\}=$
$\displaystyle=-\int
d\mathbf{y}\xi^{m}\partial_{m}\mathcal{H}_{T}+\mathbf{T}_{S}((\xi^{j}\partial_{j}N^{i}-N^{j}\partial_{j}\xi^{i}))$

using the fact that

$\displaystyle\left\\{\mathbf{T}_{S}(\xi^{i}),\mathbf{T}_{S}(N^{j})\right\\}=\int
d^{n-1}\mathbf{x}(\xi^{j}\partial_{j}N^{i}-N^{j}\partial_{j}\xi^{i})\mathcal{H}_{i}=\mathbf{T}_{S}((\xi^{j}\partial_{j}N^{i}-N^{j}\partial_{j}\xi^{i}))\
.$

We see that the constraint $\mathcal{H}_{i}\approx 0$ is preserved on
condition when following expression vanishes 333It is instructive to compare
this result with the situation when $\mathcal{H}_{T}$ is scalar density when
we get $\int
d^{n-1}\mathbf{x}(-\xi^{m}\partial_{m}\mathcal{H}_{T}-\mathcal{H}_{T}\partial_{m}\xi^{m})=-\int
d^{n-1}\mathbf{x}\partial_{m}(\xi^{m}\mathcal{H}_{T})$ which vanishes with
appropriate boundary conditions. In other words requirement of the
preservation of constraint $\mathbf{T}_{S}(\xi)$ does not generate new
constraint in case when $\mathcal{H}_{T}$ is scalar density.

$\int d^{n-1}\mathbf{x}\xi^{m}\partial_{m}\mathcal{H}_{T}=0\ .$ (38)

There are two possibilities how to make this expression zero. The first one is
to perform integration by parts and then we obtain condition on the
diffeomorphism parameters in the form

$\partial_{m}\xi^{m}=0\ .$ (39)

The second possibility is to impose tertiary constraint

$\mathcal{H}_{T}\approx 0\ .$ (40)

We will discuss each condition individually.

###  3.1 Condition $\partial_{m}\xi^{m}=0$

Imposing the condition $\partial_{m}\xi^{m}=0$ implies that there are no
constraint equivalent to the Hamiltonian one. This fact implies that there is
an additional physical mode with respect to General relativity which is $N$
and conjugate momentum. However there is a tension with the Lagrangian
formalism where we showed that the action is invariant under restricted space-
time diffeomorphism $\partial_{\mu}\xi^{\mu}=0$. In fact, if we combine this
condition with $\partial_{m}\xi^{m}=0$ we obtain that

$\partial_{0}\xi^{0}=0\ $ (41)

this implies that $\xi^{0}=\xi^{0}(\mathbf{x})$. On the other hand time
reparametrization is gauge symmetry in case when $\xi^{0}$ depends on time as
well. In other word the condition that $\xi^{0}$ does not depend on $t$ is
consistent with the absence of the Hamiltonian constraint.

###  3.2 Imposing Constraint $\mathcal{H}_{T}\approx 0$

Let us consider the second possibility which is introducing another constraint
$\mathcal{H}_{T}\approx 0$. Then we should clearly check whether this
constraint is preserved during the time evolution of the system. To do this we
introduce smeared form of this constraint

$\mathbf{T}_{T}(X)=\int d\mathbf{x}X\sqrt{h}\mathcal{H}_{T}\
,\quad\mathbf{T}_{T}(Y)=\int d\mathbf{y}Y\sqrt{h}\mathcal{H}_{T}\ ,$ (42)

where $X$ and $Y$ are ordinary test functions and where we include factor
$\sqrt{h}$ into definition of these constraints since $\mathcal{H}_{T}$ is
scalar. In principle it is straightforward to calculate Poisson bracket
between these constraints (42) however the result is very complicated. In
order to demonstrate it let us calculate following expression

$\displaystyle\left\\{\int
d^{n-1}\mathbf{x}X\sqrt{h}\frac{\kappa}{\sqrt{h}F}N\Pi^{ij}\mathcal{M}_{ijkl}\Pi^{kl},-\frac{1}{\kappa}\int
d^{n-1}\mathbf{y}\sqrt{h}N\sqrt{h}Fr\right\\}+$
$\displaystyle+\left\\{-\frac{1}{\kappa}\int
d^{n-1}\mathbf{x}X\sqrt{h}N\sqrt{h}Fr,\int
d^{n-1}\mathbf{y}Y\sqrt{h}\frac{\kappa}{\sqrt{h}F}\Pi^{ij}\mathcal{M}_{ijkl}\Pi^{kl}\right\\}=$
$\displaystyle=-4\int
d^{n-1}\mathbf{x}(Y\nabla_{m}X-X\nabla_{m}Y)N^{2}h\Pi^{mn}\frac{\nabla_{n}F}{F}+$
$\displaystyle+4\int
d^{n-1}\mathbf{x}(Y\nabla_{m}X-X\nabla_{m}Y)N^{2}hh^{mn}\frac{\nabla_{n}F}{F}\Pi(1-\mathbf{D}(n-2))+$
$\displaystyle+2\int
d^{n-1}\mathbf{x}(Y\nabla_{m}X-X\nabla_{m}Y)N^{2}h\nabla_{n}\Pi^{mn}-$
$\displaystyle-2\int
d^{n-1}\mathbf{x}(Y\nabla_{m}X-X\nabla_{m}Y)h^{mn}N^{2}h\nabla_{n}(\Pi(1-\mathbf{D}(n-2)))$

using this formula

$\frac{\delta r(\mathbf{x})}{\delta
h_{ij}(\mathbf{y})}=-r^{ij}(\mathbf{x})\delta(\mathbf{x}-\mathbf{y})+\nabla^{i}\nabla^{j}\delta(\mathbf{x}-\mathbf{y})-h^{ij}(\mathbf{x})\nabla_{k}\nabla^{k}\delta(\mathbf{x}-\mathbf{y})\
$ (44)

and Poisson brackets

$\displaystyle\left\\{\Pi^{ij}(\mathbf{x}),h_{kl}(\mathbf{y})\right\\}=-\frac{1}{2}(\delta^{i}_{k}\delta^{j}_{l}+\delta^{i}_{l}\delta^{j}_{k})\delta(\mathbf{x}-\mathbf{y})\
,$
$\displaystyle\left\\{\Pi^{ij}(\mathbf{x}),g(\mathbf{y})\right\\}=\frac{1}{2}\frac{F^{\prime}g^{2}\sqrt{-g}}{FG}h^{ij}\delta(\mathbf{x}-\mathbf{y})\
.$

The result given in (3.2) demonstrates the crucial point of the calculations
of Poisson brackets between smeared forms of $\mathcal{H}_{T}$. We see that
there are additional terms with respect to the similar calculations performed
in case of general relativity. The first one is proportional to derivative of
$F$. The next two ones are proportional to $1-\mathbf{D}(n-2)$ which are
generally space-time dependent. Note that this expression is zero in case of
General Relativity when $\mathbf{D}=\frac{1}{n-2}$. In the similar way we
should proceed with the calculations of remaining Poisson brackets and we find
that these terms do not cancel each other. Instead we get following schematic
result

$\left\\{\mathbf{T}_{T}(X),\mathbf{T}_{T}(Y)\right\\}=\int
d^{n-1}\mathbf{x}(X\partial_{m}Y-Y\partial_{m}X)\mathbf{M}^{m}\ ,$ (46)

where $\mathbf{M}^{m}$ is complicated expression that does not vanish on the
constraint surface. This result has following important consequence for the
time evolution of the constraint $\mathcal{H}_{T}$

$\displaystyle\partial_{t}\mathcal{H}_{T}(\mathbf{z})=\left\\{\mathcal{H}_{T}(\mathbf{z}),H\right\\}\approx\int
d^{n-1}\mathbf{y}\left\\{\mathcal{H}_{T}(\mathbf{z}),\mathcal{H}_{T}(\mathbf{y})\right\\}=$
$\displaystyle=\frac{2}{\sqrt{h}}\partial_{m}[\frac{1}{\sqrt{h}}]\mathbf{M}^{m}+\frac{1}{h}\partial_{m}\mathbf{M}^{m}\
.$

We see that the requirement of the preservation of the constraint
$\mathcal{H}_{T}\approx 0$ implies an additional constraint
$\mathcal{H}^{II}\equiv\frac{2}{\sqrt{h}}\partial_{m}[\frac{1}{\sqrt{h}}]\mathbf{M}^{m}+\frac{1}{h}\partial_{m}\mathbf{M}^{m}\approx
0$. However then for consistency of theory we should again demand that the
constraint $\mathcal{H}^{II}$ is preserved under time evolution with possible
new constraint generated. In this way we should continue further and again
check requirement of preservation of this constraint and so on. Of course,
there is a possibility to consider extended form of Hamiltonian with all
constrains included. In this case the tertiary constraint $\mathcal{H}_{T}$
should be add to the Hamiltonian multiplied by Lagrange multiplier
$\lambda^{II}$. Then the consistency of theory would show that
$\mathcal{H}_{T}$ is second class constraints with $\mathcal{H}_{T}$. Similar
situation occurs in case of non-projectable Hořava Lifshitz gravity with space
dependent lapse [18, 19, 20]. Note that this result is possible in principle
due to the fact that $\mathcal{H}_{T}(\mathbf{x})$ in fact corresponds to
$\infty^{n-1}$ constraints. However then the standard counting degrees of
freedom would imply that Hamiltonian constraint $\mathcal{H}_{T}\approx$
eliminates only one degree of freedom which is inconsistent. In case of non-
projectable Hořava-Lifshitz gravity this problem was solved by extension of
original gravity with specific term so that the theory possesses two second
class constraints making the right counting of physical degrees of freedom.
However we mean that such a procedure cannot be applied in case of TDiff
invariant gravity and hence it seems that TDiff invariant gravity does not
have well defined canonical structure.

In conclusion we would like to stress that in the special case when $F=1$ the
theory has well defined Hamiltonian structure as was shown in our previous
paper [14]. Further, it is possible that the theory has again well defined
structure when the matrix $\mathcal{M}^{ijkl}$ as was defined in our paper is
non-invertible. An example such a theory is Weyl transverse gravity [23]. The
canonical analysis of this theory is currently under study.

Acknowledgement:

This work is supported by the grant “Dualitites and higher order derivatives”
(GA23-06498S) from the Czech Science Foundation (GACR).

## References

  * [1] W. Buchmuller and N. Dragon, _“Einstein Gravity From Restricted Coordinate Invariance,”_ Phys. Lett. B 207 (1988), 292-294 doi:10.1016/0370-2693(88)90577-1
  * [2] M. Henneaux and C. Teitelboim, _“The Cosmological Constant and General Covariance,”_ Phys. Lett. B 222 (1989), 195-199 doi:10.1016/0370-2693(89)91251-3
  * [3] K. V. Kuchar, _“Does an unspecified cosmological constant solve the problem of time in quantum gravity?,”_ Phys. Rev. D 43 (1991), 3332-3344 doi:10.1103/PhysRevD.43.3332
  * [4] W. G. Unruh, _“A Unimodular Theory of Canonical Quantum Gravity,”_ Phys. Rev. D 40 (1989), 1048 doi:10.1103/PhysRevD.40.1048
  * [5] I. Y. Karataeva and S. L. Lyakhovich, _“Gauge symmetry of unimodular gravity in Hamiltonian formalism,”_ Phys. Rev. D 105 (2022) no.12, 124006 doi:10.1103/PhysRevD.105.124006 [arXiv:2203.06620 [hep-th]].
  * [6] R. Bufalo and M. Oksanen, _“Canonical structure and extra mode of generalized unimodular gravity,”_ Phys. Rev. D 97 (2018) no.4, 044014 doi:10.1103/PhysRevD.97.044014 [arXiv:1712.09535 [hep-th]].
  * [7] R. Bufalo, M. Oksanen and A. Tureanu, _“How unimodular gravity theories differ from general relativity at quantum level,”_ Eur. Phys. J. C 75 (2015) no.10, 477 doi:10.1140/epjc/s10052-015-3683-3 [arXiv:1505.04978 [hep-th]].
  * [8] J. Kluson, _“Canonical Analysis of Unimodular Gravity,”_ Phys. Rev. D 91 (2015) no.6, 064058 doi:10.1103/PhysRevD.91.064058 [arXiv:1409.8014 [hep-th]].
  * [9] E. Alvarez, D. Blas, J. Garriga and E. Verdaguer, _“Transverse Fierz-Pauli symmetry,”_ Nucl. Phys. B 756 (2006), 148-170 doi:10.1016/j.nuclphysb.2006.08.003 [arXiv:hep-th/0606019 [hep-th]].
  * [10] J. J. Lopez-Villarejo, _“TransverseDiff gravity is to scalar-tensor as unimodular gravity is to General Relativity,”_ JCAP 11 (2011), 002 doi:10.1088/1475-7516/2011/11/002 [arXiv:1009.1023 [hep-th]].
  * [11] E. Alvarez, A. F. Faedo and J. J. Lopez-Villarejo, _“Transverse gravity versus observations,”_ JCAP 07 (2009), 002 doi:10.1088/1475-7516/2009/07/002 [arXiv:0904.3298 [hep-th]].
  * [12] J. J. van der Bij, H. van Dam and Y. J. Ng, _“The Exchange of Massless Spin Two Particles,”_ Physica A 116 (1982), 307-320 doi:10.1016/0378-4371(82)90247-3
  * [13] E. Alvarez, A. F. Faedo and J. J. Lopez-Villarejo, _“Ultraviolet behavior of transverse gravity,”_ JHEP 10 (2008), 023 doi:10.1088/1126-6708/2008/10/023 [arXiv:0807.1293 [hep-th]].
  * [14] J. Kluson, _“Note About Canonical Formalism for Gravity with Dynamical Determinant of Metric,”_ [arXiv:2306.16746 [gr-qc]].
  * [15] E. Gourgoulhon, _“3+1 formalism and bases of numerical relativity,”_ [arXiv:gr-qc/0703035 [gr-qc]].
  * [16] M. Henneaux and C. Teitelboim, _“Quantization of gauge systems,”_
  * [17] P. Horava, _“Quantum Gravity at a Lifshitz Point,”_ Phys. Rev. D 79 (2009), 084008 doi:10.1103/PhysRevD.79.084008 [arXiv:0901.3775 [hep-th]].
  * [18] D. Blas, O. Pujolas and S. Sibiryakov, _“On the Extra Mode and Inconsistency of Horava Gravity,”_ JHEP 10 (2009), 029 doi:10.1088/1126-6708/2009/10/029 [arXiv:0906.3046 [hep-th]].
  * [19] M. Li and Y. Pang, _“A Trouble with Hořava-Lifshitz Gravity,”_ JHEP 08 (2009), 015 doi:10.1088/1126-6708/2009/08/015 [arXiv:0905.2751 [hep-th]].
  * [20] M. Henneaux, A. Kleinschmidt and G. Lucena Gómez, _“A dynamical inconsistency of Horava gravity,”_ Phys. Rev. D 81 (2010), 064002 doi:10.1103/PhysRevD.81.064002 [arXiv:0912.0399 [hep-th]].
  * [21] D. Blas, O. Pujolas and S. Sibiryakov, _“Consistent Extension of Horava Gravity,”_ Phys. Rev. Lett. 104 (2010), 181302 doi:10.1103/PhysRevLett.104.181302 [arXiv:0909.3525 [hep-th]].
  * [22] J. Kluson, _“Note About Hamiltonian Formalism of Healthy Extended Horava-Lifshitz Gravity,”_ JHEP 07 (2010), 038 doi:10.1007/JHEP07(2010)038 [arXiv:1004.3428 [hep-th]].
  * [23] I. Oda, _“Classical Weyl Transverse Gravity,”_ Eur. Phys. J. C 77 (2017) no.5, 284 doi:10.1140/epjc/s10052-017-4843-4 [arXiv:1610.05441 [hep-th]].


[3]

#  Deep differentiable reinforcement learning and optimal trading

Thibault<EMAIL_ADDRESS>222The author thanks Reda Messikh,
Stéphane Daul and Rémy Cottet for usefull discussions on the topic and
feedbacks on the manuscript. 333The opinions expressed in this article are
solely those of the author.

Abstract

In this article we introduce the differentiable reinforcement learning
framework. It is based on the fact that in many reinforcement learning
applications, the environment reward and transition functions are not black
boxes but known differentiable functions. Incorporating deep learning in this
framework we find more accurate and stable solutions than more generic actor
critic algorithms. We apply this deep differentiable reinforcement learning
(DDRL) algorithm to the problem of optimal trading strategies in various
environments where the market dynamics are known. Thanks to the stability of
this method, we are able to efficiently find optimal strategies for complex
multi-scale market models and for a wide range of environment parameters. This
makes it applicable to real life financial signals and portfolio optimization
where the expected return has multiple time scales. In the case of a slow and
a fast alpha signal, we find that the optimal trading strategy consists in
using the fast signal to time the trades associated to the slow signal.

Keywords: Reinforcement learning, Optimal trading, Differentiable, Optimal
control, Dynamic programming, Deep learning, Multi-scale signals.

##  1 Introduction

Reinforcement learning is a field of machine learning which aims at finding
optimal sequences of actions in order to achieve a multi-step task or maximize
a cumulative reward. Its combination with deep learning, see [LBH15], called
deep reinforcement learning (DRL), see [Fra+18], has recently had a huge
success in addressing sequential decision problems. For example, it
outperforms the best humans at many games, see [Mni+15], and has state of the
art applications in many real life applications such as robotics, see
[Lil+15], and self-driving cars, see [Pan+17].

Quantitative portfolio management is the use of mathematical and statistical
tools to devise ‘optimal’ portfolios of financial assets. Historically, this
field mostly focused on getting the best expected return (or alpha in the
following) with the smallest possible risk as in the mean variance framework
of Markowitz, see [Mar68]. However, this does not take into account the
practically paramount role of trading costs: An investor needs to control his
turnover if he does not want to spend his alpha in trading costs, see [FIM12].
This implies that the current choice of portfolio weights does not only impact
the portfolio returns over the next short period but also the future investor
positioning. This naturally puts us in the setting of reinforcement learning
where the current actions (trades) have an impact on the next reward as well
as on the future state of the environment which itself has an impact on future
rewards, see [SB18].

Over the last decade, the use in finance of modern machine learning techniques
such as deep learning has seen a huge increase to: price options, see [HMT19],
predict returns, see [GKX18] and model risks, see [GKX19]. In spite of the
challenges inherent to financial data such as the structural low signal to
noise ratio and non stationarity, adding non linearity and interactions
between usual features seems to add value compared to more classical linear
models, see [DJN21]. Deep reinforcement learning has also been extensively
applied to directly build optimal strategies from real financial data, see
[Den+16] and [Con+20], by minimizing some loss of the portfolio performance.
These approaches have the particularity that they do not split the portfolio
construction process in two steps: First modeling and estimating the dynamics
of the market (which is typically a data science issue with no mathematically
perfect solution). Second finding the optimal strategy in this model (which is
an engineering problem with a mathematically well defined solution). This
however makes it harder for an investor to understand his portfolio and what
criteria explain the asset weights.

An alternative approach taken in [Cha+20], which we follow here, is to assume
that the market is well modeled and its parameters are known. In this model,
we can then use deep reinforcement learning to find the optimal strategy. The
advantage of their approach is that one can always simulate as much data as
one needs and evaluate strategies as precisely as necessary. They use the
state of the art Deep Deterministic Policy Gradient (DDPG). This algorithm
tends to be hard to train and requires much engineering, meta-parameters
tweaking and to be trained on different starting points to be well fitted in
practice, see [FHM18]. Even then, it is only applied to a very simple $AR(1)$
alpha model and when more complex models are given as input, the strategies
obtained do not converge towards the optimal ones.

In particular, in this article, we are interested in devising optimal trading
strategies in environments where the alpha process has a multi-time scale
behavior, see [GP13] and [Boy+17]. Indeed, in practice, investors have access
to return predictors with different reversion time scales. For example, Value,
Momentum, see [AMP13], and Quality, see [AFP19], can be considered as slow
signals, indeed if a stock has a good Quality signal it will most likely
remain good in the coming months. On the other hand, price reversal, see
[BS91], and analyst recommendations, see [Jai+21], are much faster signals
with shorter memory. We tried to apply the DDPG algorithm to such multi-scale
trading environments however, even after many attempts, we did not manage to
make it converge to optimal strategies.

To solve this issue, we introduce the differentiable reinforcement learning
framework. It leverages on the fact that in many optimal control problems, the
underlying dynamics of the environment are known. This enables us to precisely
anticipate the future impact of our actions and to directly maximize the
cumulative expected reward. Using such knowledge is alluded to in [Cha+20].
Looking for action functions which take the form of a feed forward neural
network (FFNN) we devise an algorithm which derives optimal trading
strategies. Furthermore we thoroughly analyze and describe the interactions
between the slow and fast signals in the optimal action. We find that the slow
signal is more correlated to the portfolio weights and the fast signal is more
correlated to the trades. In other words, the fast signal is used to time the
trades associated to the slow signal.

Our contribution in this paper is twofold: First we introduce the
differentiable reinforcement learning framework where the underlying
environment dynamics are known. Combining it with deep learning, this enables
us to efficiently find the optimal action function. Second we apply this
algorithm to optimal trading environments where the expected return of the
assets has a multi-scale behavior (with a fast and slow component). We
describe the optimal strategies obtained in this setting and show that the
fast signal is used to time the execution of the trades associated to the slow
signal in an optimal fashion.

This paper is organized as follows: In Section 2, we introduce the
differentiable reinforcement learning framework and a generic algorithm to
find optimal action functions approximated by a deep neural network. In
Section 3, we show how generic (multi-scale) optimal trading problems fit into
the context of differentiable reinforcement learning. In Section 4, we exhibit
and describe the properties of optimal trading strategies. We conclude on how
to apply this in practice in Section 5.

##  2 Differentiable reinforcement learning

###  2.1 The reinforcement learning framework

In the standard reinforcement learning setup, see [SB18], a learning agent is
interacting with his environment in discrete time in the following way: At
each time $t$, he sees the current sate of the environment $s_{t}$ (in some
settings the agent only sees a noised version of the environment but this is
irrelevant in what follows) and chooses an action $a_{t}$. The environment
then samples a reward $r_{t}$ to the agent and shows it the next state
$s_{t+1}$. The aim of the agent is to find a sequence of actions which
maximize the expectation of the cumulative reward up to time $T$ which is
defined as

$\textrm{CR}_{T}=\sum_{i=0}^{T-1}r_{i}.$

What makes the problem challenging is that both the reward and the next state
(and thus future rewards) depend on the actions of the agent which thus have
short term and long term impact.

A noteworthy point is that as opposed to the more classical optimal control
setting, see [Ber12], in the reinforcement learning setting, the underlying
dynamics of the environment are not necessarily known by the agent. From his
point of view, the environment is simply a black box from which we can sample
state transitions and rewards by giving it actions.

Until recently, the choice of an optimal action function was only solvable
either in very specific settings where a closed form solution exists or where
both the action and the state spaces are finite and rather small, see [WD92].
In the last decade, the combination of ideas from optimal control with the
flexibility of deep learning has enabled astonishing progress in the context
of self-driving cars, see [Pan+17], games, see [Mni+15] and robotics, see
[Lil+15]. First for discrete action spaces, see [Mni+15], and even more
recently for continuous and potentially high dimensional action spaces. For
example [Lil+15] introduced the Deep Deterministic Policy Gradient (DDPG)
algorithm which jointly learns the optimal action and the value function by
approximating them with deep neural networks and using the Bellman equation.

However, even if progress is made every year in this direction, see [FHM18],
this kind of methods called actor critic are still hard to train. This is
especially the case when the expected cumulative reward is flat around the
optimal action where they require much tweaking of the meta-parameters and
engineering of the algorithm, see [IS15] and [Hen+18]. Even then, the results
are far from perfect and need to be trained on many learning paths with
different starting points to get closer to the optimal solution, see [Cha+20].

In the next paragraph, we introduce the differentiable reinforcement learning
setting. It uses the fact that in many applications such as the one of
[Cha+20], the underlying dynamics of the environment are known. Moreover, in a
sense that we will make precise, these dynamics are differentiable. In this
setting, we devise a conceptually simple algorithm which finds optimal action
functions that are far more accurate and stable than the ones obtained from
generic reinforcement learning methods.

Applied to the problem of optimal trading strategies, it enables us to
optimize more complex market models with a large number of meta-parameters.

###  2.2 The differentiable reinforcement learning framework

We define the differentiable reinforcement learning framework as follows:

  * •

The next state $s_{t+1}$ is a known differentiable transition function $\Phi$
of the current state $s_{t}$, the chosen action $a_{t}$ and a random variable
$U_{t+1}$ that we can sample from

$s_{t+1}=\Phi(s_{t},a_{t},U_{t+1}).$

  * •

The reward $r_{t}$ is a known differentiable function $\Psi$ of the current
state $s_{t}$, the chosen action $a_{t}$ and a random variable $V_{t}$ that we
can sample from

$r_{t}=\Psi(s_{t},a_{t},V_{t}).$

In this framework, the variables $U$ and $V$ fully capture the randomness of
the environment.

The aim of optimal control problems is to find an action function $A$ such
that

$a_{t}=A(s_{t})$

which maximizes the expectation of the long term cumulative reward.

Note that applications that take as input real world data instead of simulated
ones, do not make it into this framework. However, when working with known
environments that we can simulate we are mostly in this setting.

###  2.3 The cumulative reward function

An important property of this framework is that given an action function $A$
the cumulative reward $\textrm{CR}_{T}$ of the strategy is only a function of
$A$, $s_{0}$, $(V_{t})_{0\leq t\leq T-1}$ and $(U_{t})_{0\leq t<T-1}$:

$\textrm{CR}_{T}=\textrm{CR}_{T}(A,s_{0},(V_{t})_{0\leq t\leq
T-1},(U_{t})_{0\leq t<T-1}).$

For example, the rewards for T=1, 2 and 3 are shown below.

  * •

$\textrm{CR}_{1}=\psi(s_{0},A(s_{0}),V_{0}).$

  * •

$\textrm{CR}_{2}=\textrm{CR}_{1}+\psi(\phi(s_{0},A(s_{0}),U_{0}),A(\phi(s_{0},A(s_{0}),U_{0})),V_{1}).$

  * •

$\textrm{CR}_{3}=\textrm{CR}_{2}+\psi(\phi(\phi(s_{0},A(s_{0}),U_{0}),A(\phi(s_{0},A(s_{0}),U_{0})),U_{1}),A(\phi(\phi(s_{0},A(s_{0}),U_{0}),\\\
A(\phi(s_{0},A(s_{0}),U_{0})),U_{1})),V_{2}).$

Of course for $T$ large, this function is quite deep in the sense that it is a
composition of hundreds of base functions. However, we will see that we are
able to efficiently minimize its expectation with respect to the action
function for large $T$ (up to at least 100 which is more than enough for our
applications).

###  2.4 Deep learning application

In order to numerically find the action function which maximizes the
expectation of the cumulative reward up to a given time $T$, we need to
restrict it to a family of parametrized action functions, with parameters
denoted $\theta$:

$A(s)=F(s;\theta).$

If we now assume that the values of $A$ at state $s$ is differentiable with
respect to these parameters $\theta$. Then the cumulative reward samples are
also differentiable functions of the parameters $\theta$. To numerically
compute the associated derivatives of these deep composition of functions, we
use automatic differentiation (AD) and backpropagation through the Pytorch
package, see [Pas+17]. A stochastic gradient descent (SGD) like algorithm can
then be used to find the action parameters which minimize the empirical sample
average of the cumulative reward.

A flexible extensively studied and numerically convenient parametrization of
the action function is the dense feed forward neural network or multi-layer
perceptron, see [Sch15] for the definition. In what follows, unless stated
otherwise, we take a neural network with two hidden layers with 300 hidden
neurons each and $ReLU$ activation functions so that the model has around
10’000 parameters.

###  2.5 Sampling and empirical average

We fit $A$ (or equivalently $\theta$) by minimizing a loss function defined as
the opposite of the empirical average of the cumulative reward on $N$
independent samples of initial states $s_{0}^{i}$ and randomness
$(U^{i}_{t})_{t\leq T}$ and $(V^{i}_{t})_{t\leq T}$:

$\min_{\theta}-\sum_{i=1}^{N}\textrm{CR}_{T}(F(\cdot,\theta),s^{i}_{0},(V^{i}_{t})_{0\leq
t\leq T},(U^{i}_{t})_{0\leq t\leq T-1}).$

Note that the sampling distribution of $s_{0}^{i}$ does not matter too much.
What is important is that the sample covers the states that we are interested
in.

In what follows, unless stated otherwise, we minimize the above loss function
taking $N=10$ million samples with the Adam algorithm, see [KB14], with 1024
mini-batch sizes and the default parameters of the Pytorch library: lr=0.001
and betas=(0.9, 0.999), see [Pas+17]. We perform 50 epochs on the entire
sample size. At each epoch, we remove 10% of the learning rate to improve
convergence.

The complexity of the model is

$\\#\\{\text{model parameters}\\}\times T\times N\times\text{epochs}.$

On a a server using 16 2.9 GHz Xeon Platinium 8268 CPUs it takes around 10
hours to train the model (this time is weakly dependent on the environment
complexity). Note however that for the simplest environments, we get a very
good solution after the first epoch.

###  2.6 Making environment parameters variable

To make it usable in practice in a quantitative strategy, we need to be able
to apply this to potentially thousands of stocks with different environment
parameters (costs, spread, max weights, …). It would not be efficient to fit a
model for each set of parameters. One easy way to solve this issue is to
consider these environment parameters as static (that is non time varying)
parts of the state. We therefore only need to train one model which takes as
input the state and the environment parameters to simultaneously find the
optimal action for a continuum of models.

More formally denote $\zeta$ an environment parameter:

$s_{t+1}=\Phi(s_{t},a_{t},U_{t+1};\zeta)\text{ and
}r_{t}=\Psi(s_{t},a_{t},V_{t};\zeta).$

Then defining the new state as $s^{\prime}_{t}=(s_{t},\zeta)$, we are in the
same context as before without environment parameters:

$s^{\prime}_{t+1}=\Phi^{\prime}(s^{\prime}_{t},a_{t},U_{t+1})\text{ and
}r_{t}=\Psi^{\prime}(s^{\prime}_{t},a_{t},V_{t}).$

We can thus find the optimal action as a function of $s_{t}$ and $\zeta$.

As for initial states, the environment parameter sampling does not matter too
much and must simply cover the range of environments that one is interested
in.

##  3 Conception of optimal trading strategies

###  3.1 Optimal investment and reinforcement learning

As discussed in the introduction, when choosing his weights the investor must
not only consider his immediate reward but also how he will be positioned in
the future. As stated in [GP13]: ‘A good hockey player plays where the puck
is. A great hockey player plays where the puck is going to be’.

Three papers which effectively tackle this issue are: [GP13], where the
authors find a simple intuitive solution in the simple case where the
transaction costs are quadratic and related to the covariance matrix in a
strong way. [Boy+17], where the authors use convex optimization in a multi-
period setting to find a heuristic solution by assuming that the future alpha
trajectory is deterministic. As we will see, this assumption yields sub-
optimal strategies when the alpha is stochastic. [De ̵+12], where the authors
use optimal control theory to find the optimal weight trajectory when costs
are linear.

Another branch of application of deep reinforcement learning in finance is
illustrated in [Con+20]. Instead of splitting the model estimation and the
conception of an optimal strategy in this model in two steps, it consists in
giving as input to a large carefully engineered neural network the raw
features and expect as output optimal weights. See [KR20] for a recent review
of the applications on reinforcement learning in quantitative finance.

In this article, we follow the more classical road which consists in working
in two steps:

  * •

First modeling (and estimating) the expected return (for example using
supervised machine learning as in [GKX18]), risk and costs related to our
assets.

  * •

Then find the optimal strategy under in this model.

The advantage of this method is that it makes it easier to: decompose our
performance, test our assumptions, have margin of errors on our alpha
estimations and make active decisions. Moreover, in this setting, the strategy
conception step is not anymore a data estimation problem (since we can always
simulate more data from the model) but an optimization problem whose tentative
solutions can be precisely evaluated.

We are thus in the footsteps of [Cha+20]. However we want to work with more
complex alpha models. In such models, we did not manage to make our action
functions converge with the DDPG algorithm. In particular, we want to consider
models where our alpha process has (at least) two time scales: a slow and a
fast. To illustrate these alphas, assume that the slow alpha has a half-life
of one year (think Value, Momentum, Quality, see [AMP13] and [AFP19]) and the
fast alpha has a half-life of one month (think short term price reversal,
analyst revisions, sector momentum, see [BS91] and [Jai+21]).

The kind of non-trivial properties that we hope to find is that for reasonable
model parameters, the fast alpha times the execution of the slow alpha.
Indeed, assume that we expect a stock to perform well in the next year but not
in the next month. If we do not have the stock in our portfolio, we will wait
a month before buying it. If we have it in our portfolio, it might not be
worth to sell it and to buy it again in a month because of transaction costs.

###  3.2 Reward and objective

As in [Cha+20], we will consider environments where the reward writes as a
combination of a return term equal to the expected return times the position,
a risk term and a cost term:

$r_{t}=w_{t}\alpha_{t}-\text{Risk}(w_{t})-\text{Cost}(w_{t},lw_{t})$

where $\alpha$ is the expected stock return over the next period, $w$ is the
weight to be chosen and $lw$ is the last weight before the choice of the new
weight.

In this case, the state is the alpha and the last weight:

$s_{t}=(\alpha_{t},lw_{t})$

and the action is the next weight $w_{t}$.

In what follows, the action has a trivial effect on the next state: The last
dimension of the next state is equal to the action. However, we can imagine
trading environments where the action has a more complex effect on the alphas
at different scales which is equivalent to the problem introduced in [CGL17]
where market impact is transient and the kernel is approximated by a
combination of exponential functions.

###  3.3 Trading environments description

In this paragraph, we describe the different trading environments that we will
test. We give examples of alpha processes, risk functions and cost functions.

Note that all the combinations of models defined below enter the
differentiable reinforcement learning framework.

####  3.3.1 Alpha modeling

Let us begin by defining the two alpha dynamics that we will consider: mono-
scale and multi-scale.

##### Mono-scale alpha

In this model, as in [Cha+20], the alpha of the stock is an observable $AR(1)$
process (with $\rho_{\alpha}=0.9$ and $\sigma_{\alpha}=1$ for example):

$\alpha_{t}=\rho_{\alpha}\alpha_{t-1}+\sigma_{\alpha}N^{\alpha}_{t}$

where $N^{\alpha}_{t}$ are independent standard normal variables.

##### Two-scale alpha

In this model, the alpha of the stock is a combination of two (independent)
observable $AR(1)$ processes, a fast (F) and a slow (S) (with
$\rho_{\alpha^{S}}=0.9$ , $\rho_{\alpha^{F}}=0$, $\sigma_{\alpha^{S}}=1$ and
$\sigma_{\alpha^{F}}=3$ for example):

$\alpha^{S}_{t}=\rho_{\alpha^{S}}\alpha^{S}_{t-1}+\sigma_{\alpha^{S}}N^{\alpha^{S}}_{t}$
$\alpha^{F}_{t}=\rho_{\alpha^{F}}\alpha^{F}_{t-1}+\sigma_{\alpha^{F}}N^{\alpha^{F}}_{t}$
$\alpha_{t}=\alpha^{S}_{t}+\alpha^{F}_{t}$

where $N^{\alpha^{S}}_{t}$ and $N^{\alpha^{F}}_{t}$ are independent standard
normal variables.

This model will enable us to take into account the interaction between fast
and slow return predictors and the last weight in the optimal action.

####  3.3.2 Risk modeling

Let us now state the two risk terms that we will consider: $L_{2}$ and max
weight

#####  $L_{2}$ risk

This risk term is equal to a risk aversion times the square of the position:

$\text{Risk}(w_{t})=\frac{\lambda}{2}w_{t}^{2}.$

##### Max weight as a risk

As explained in [Cha+20], we can impose a max weight constraint as a
penalization of weights above a threshold $M$ (or below $-M$). Here we choose
a $ReLU$ penalization:

$\text{Risk}(w_{t})=K(|w_{t}|-M)_{+}$

where $K$ is large so that the optimal weight is never above $M$.

####  3.3.3 Cost modeling

Let us now state the two cost terms that we will consider: $L_{2}$ and
$L_{1}$.

#####  $L_{2}$ cost

As in [GP13], the cost term can be modeled as the square of the turnover. This
enables to have closed form solutions but is not very realistic:

$\text{Cost}(w_{t},lw_{t})=C\times|w_{t}-lw_{t}|^{2}.$

#####  $L_{1}$ cost

To take into account spread costs, we can model the cost as proportional to
the turnover:

$\text{Cost}(w_{t},lw_{t})=S\times|w_{t}-lw_{t}|.$

Note that this function is not strictly speaking differentiable. However, it
is differentiable almost everywhere. We can thus apply SGD like methods to
minimize the corresponding loss (in the same way that we can apply SGD to
minimize the parameters of neural networks with $ReLU$ activation functions).

##  4 Numerical results

In this section, we fit and describe the optimal action function in five
trading environments:

  * •

An environment with a mono-scale alpha, a $L_{1}$ cost and a $L_{2}$ risk to
show that we recover the result of [Cha+20] and study the behavior of our
algorithm.

  * •

An environment with a mono-scale alpha, a $L_{1}$ cost and a max weight risk
to show that we recover the result of [Cha+20] in a more numerically difficult
case where the optimal action is not a continuous function of alpha.

  * •

An environment with a mono-scale alpha with a variable alpha volatility, a
$L_{1}$ cost and a $L_{2}$ risk. With this environment we want to prove that
our algorithm can find the optimal strategy for a continuum of environment
parameters with only one fit and study the impact of the alpha variability on
the optimal strategy.

  * •

An environment with a multi-scale alpha, a $L_{1}$ cost and a $L_{2}$ risk to
study the impact of the fast alpha on the optimal strategy.

  * •

An environment with a multi-scale alpha, a $L_{1}$ cost with a variable spread
and a $L_{2}$ risk to study the impact of the spread on the effect of the fast
alpha on the optimal strategy.

###  4.1 Mono-scale alpha

To study the convergence of the algorithm towards the optimal action we begin
by a simple environment where:

  * •

The alpha is mono-scale with persistence parameter $\rho_{\alpha}=0.9$ and
volatility $\sigma_{\alpha}=1$.

  * •

There is a $L_{1}$ cost term with spread $S=4$.

  * •

There is a $L_{2}$ risk term with risk aversion $\lambda=1$.

The parameters are chosen so that we are in the regime where the optimal
weight has the same form as the alpha trajectory without trading at every
period.

####  4.1.1 Properties of the optimal action

The optimal weight is a function of the last weight and the alpha:

$a_{t}=A(\alpha_{t},lw_{t}).$

We represent it by plotting for a few last weight values the optimal weight as
a function of alpha in Figure 1.

Figure 1: Optimal weight as a function of the current alpha for different last
weights and the target weight which in this case is the alpha.

Note that in this model, if there is no spread costs, the optimal weight is
the one which maximizes the current reward $r_{t}$ which is
$w^{tgt}_{t}=\alpha_{t}/\lambda$. In the current case where $\lambda=1$,
$\alpha$ can thus be seen as a target weight. We find as in [De ̵+12] that the
optimal strategy has the same functional form as in the single period case,
see [DP20], and consists in not trading if the alpha is within a threshold
around the last weight and to get to this threshold if the alpha is outside of
it.

Figure 2: Upper and lower trading bounds as a functon of the last weight.

These thresholds are analysed theoretically in [De ̵+12]. We plot them in
Figure 2. The upper (resp. lower) bound is defined as the smallest (resp.
largest) alpha such that the optimal weight is strictly above (resp. below)
the last weight to which we subtract the last weight. We find that except for
a zero last weight, this threshold is asymmetric: for positive (resp.
negative) last weight, the upper bound in larger (resp. smaller) than minus
the lower bound. For high (resp. low) last weights, the lower (resp.upper)
bound tends to zero. This means that for a high (resp. low) last weight, the
optimal strategy is to sell (resp. buy) as soon as the alpha/target weight is
below (resp. above) the last weight.

We plot in Figure 3 a sample trajectory of the alpha and of the associated
optimal weight. We see that when the alpha gets too far from the weight,
trading happens.

Figure 3: Sample of the trajectory of the alpha signal and optimal weight.

####  4.1.2 Convergence speed

To showcase the stability of the convergence of the algorithm we plot in
Figure 4 the evolution of the average reward (normalized to one for the
optimal strategy) through learning paths over the first epoch of the
optimization.

Figure 4: Evolution of the reward over 5 training paths. The x axis
corresponds to the number of mini-batches (each having 1024 samples).

We find that the randomness of the starting point and of the sampling does not
affect the obtained strategy as it is sometimes the case when fitting deep
neural networks. We thus do not need to train multiple agents and take the
best.

####  4.1.3 Impact of the horizon $T$

In this paragraph, we study the impact of the horizon $T$ on the obtained
strategy. Intuitively, this horizon should be longer than the ‘memory’ of the
model. For example, here the autocorrelation of the alpha process behaves as
$(\rho_{\alpha})^{t}$ and thus, the half-life of the model is
$-\log(2)/\log(\rho_{\alpha})\sim 6.5$. Hence, one should take a horizon of at
least 10. Plotting in Figure 7 the long term reward as a function of the
horizon, this is roughly what we find: The reward (and the associated
strategy) converges around $T=5$. Recall that to be conservative and allow for
longer memory models, we take $T=50$.

Figure 5: Average long term reward as a function of the horizon $T$.

###  4.2 Max weight risk

An environment which is slightly harder to solve numerically, because of the
discontinuity of the optimal action as a function of alpha, is the following:

  * •

As before, the alpha is mono-scale with persistence parameter
$\rho_{\alpha}=0.9$ and volatility $\sigma_{\alpha}=1$.

  * •

There is a $L_{1}$ cost term with spread $S=4$.

  * •

There is a max weight risk term with max weight $M=3$ and penalization $K=10$
(but no $L_{2}$ risk aversion).

We plot in Figure 6 the optimal weight as a function of alpha for a few last
weights as above. We find as in [Cha+20] that the optimal solution is to get
to the max weight (resp. minus the max weight) if the alpha is greater (resp.
lower) than a limit and not to trade between. The optimal action as a function
of alpha is thus discontinuous. Although the fit is less good than before
(because a neural network needs large parameters to get large derivatives) it
still captures the global shape of the optimal strategy. Note that if we
increase the number of epochs, we get closer and closer to the perfect
discontinuous real solution.

Figure 6: Optimal weight as a function of the current alpha for different last
weights.

###  4.3 Alpha volatility impact

To show the ability of our model to treat environment parameters as static
state dimensions, we train a meta-model which finds the optimal action as a
function of the last weight, the alpha signal and the alpha volatility:

$a_{t}=A(\alpha_{t},lw_{t},\sigma_{\alpha}).$

To do this, we take the same model as in the first environment except that we
sample the alpha volatility uniformly over $[0,4]$ and treat is as a static
(that is non time varying) state dimension.

In Figure 7 we plot the optimal action function as a function of the alpha for
a last weight equals to zero and for different values of alpha volatility.

Figure 7: Optimal weight as a function of the current alpha for different
alpha volatility and last weight equals zero.

Interestingly, we see that the optimal strategy strongly depends on the alpha
variability. It makes sense that the non trading region increases with the
alpha volatility. Indeed, if the alpha volatility is large, it is more likely
that the alpha will change sign shortly and one will need to revert his
trades. The framework of [Boy+17] which assumes that future alpha is
deterministic equal to its expectation corresponds to zero. However, the
expectation of the future alpha is not enough. The uncertainty around this
expectation plays an important role.

###  4.4 Two-scale alpha

In this paragraph, we introduce the first multi-scale alpha environment. To
our knowledge, outside of the case of $L_{2}$ transaction costs, the study of
optimal strategies for a multi-scale alpha signals has never been done.

  * •

The alpha is two-scale with persistence parameters $\rho_{\alpha^{S}}=0.9$ and
$\rho_{\alpha^{F}}=0$ and volatilities $\sigma_{\alpha^{S}}=1$ and
$\sigma_{\alpha^{F}}=3$.

  * •

There is a $L_{1}$ cost term with spread $S=4$.

  * •

There is a $L_{2}$ risk term with risk aversion $\lambda=1$.

The parameters are chosen so that we are in a regime where the optimal weight
has the same form as the alpha trajectory without trading at every period. The
fast alpha magnitude is chosen such that, as it is the case in practice for
large portfolios, it is rarely worth doing a round trip to exploit the fast
alpha. There is however room to use the fast alpha to time the execution of
the slow alpha.

The optimal action is a function of the last weight, the slow alpha and the
fast alpha:

$a_{t}=A(\alpha^{F}_{t},\alpha^{S}_{t},lw_{t}).$

To show the properties of the optimal strategy, we begin by fixing the fast
alpha to zero and as before to plot in Figure 8 the optimal weight as a
function of the slow alpha for different starting weight.

Figure 8: Optimal weight as a function of the slow alpha for different
starting weight and fast alpha equals zero.

We find the same behavior as in the first environment without the fast alpha.
Let us now study the impact of the fast alpha on the optimal strategy. To do
this, in Figure 9, we fix the last weight equals to zero and plot as a heat-
map the optimal weight as a function of the slow alpha and the short term (or
total) alpha defined as the sum of the slow and fast alphas.

We see that this heat-map is split into three areas:

  * •

On the bottom left, the optimal action is to sell (more and more as you get
lower and lefter).

  * •

On the top right, the optimal action is to buy.

  * •

In the middle, there is a no trading region.

If the short term alpha is higher (resp. lower) than 2 spreads (resp. minus 2
spreads) we always buy (even if we expect to revert the trade next month). If
the the short term alpha is higher (resp. lower) than zero we never sell
(resp. buy) but prefer to wait next month.

Figure 9: Heat-map of the optimal weight as a function of the slow and fast
alphas for the last weight equals zero.

Looking at sample trajectories of the slow and fast alphas and of the optimal
weight in Figure 10, we see that the main long term driver of the weight is
the slow alpha and the fast alpha does not seem to matter much. However,
zooming in, we see that the fast alpha is used to time the trades
corresponding to the slow alpha. Indeed, looking at the two blue ovals on
Figure 11 below, we see that at some points the slow alpha goes up but the
optimal weight does not follow before the fast alpha confirms that it is worth
trading now and not to wait. This is a quite intuitive result: If you know
that a stock is going to perform well next year but not next month, you wait
before buying it. However, if you already have the stock then you do not
necessarily sell it.

Figure 10: Sample trajectories of the slow and fast alphas and of the optimal
weight. Figure 11: Zoom on sample trajectories of the slow and fast alphas and
of the optimal weight.

We confirm this by computing the empirical correlations between alphas,
optimal weights and trades (defined as $w_{t}-lw_{t}$). We find that although
the slow alpha is strongly correlated to the optimal weight (88%), trades are
more correlated to the fast alpha (50%) than to the slow alpha (26%).

| Slow alpha | Fast alpha | Optimal weight | Trade
---|---|---|---|---
Slow alpha | 1.00 |  |  |
Fast alpha | 0.00 | 1.00 |  |
Optimal weight | 0.88 | 0.17 | 1.00 |
Trade | 0.26 | 0.50 | 0.17 | 1.00
Table 1: Correlation matrix between weights, trades and slow and fast alphas.
We see that although the weights are driven by the slow alpha, the trades are
driven by the fast alpha.

###  4.5 Spread impact

To study the impact of the spread value on the effect of the fast signal in
the optimal strategy, we consider the same environment as in the previous
paragraph except that we treat the spread as a static state variable. We
sample the spread uniformly over [0, 6].

In Figure 12 we plot the no trading zones for 4 spread values.

Figure 12: No trading regions as a function of the slow and short term alpha
for different spread values: 0 (gray), 1 (green), 2 (purple), 3 (blue) and 4
(red). On the left of dashed lines, we sell, on the right of full lines we buy
and between the two we do not trade.

As expected, the higher the spread the larger the no trading zone and when the
spread is zero the no trading zone is empty. We also find that as the spread
decreases, conditionally on the short term alpha, the slow alpha matters less
and less and only the short term alpha is important. When the spread is zero,
the contour lines are fully vertical and the slow alpha does not matter.

##  5 Conclusion

In this article, we introduced the differentiable reinforcement learning
framework where the environment is not a black box but its transition and
reward functions are differentiable functions of the state, the action and
some randomness that can be sampled. We noted that many well studied
reinforcement learning and optimal control problems fall into this framework.
Using this environment knowledge, we proposed a simple algorithm to devise
optimal action functions by writing them as deep neural networks. The
stability and speed of this algorithm allows us to work in complex continuous
multidimensional state and action environments. We can also simultaneously
find a solution for many environments with one training by considering model
parameters as a static state variables.

We applied this framework to devising sequential optimal trading strategies.
We started by reproducing some well known strategies when the alpha process
has only one time scale. We then defined and tackled environments where the
expected return is a combination of a slow and fast component and studied the
interactions of these two components with the current weight. We found that
for reasonable parameters, the fast alpha is used to time the trades
corresponding to the slow alpha. We also studied the impact of the model
parameters such as the alpha volatility and the spread on the optimal
strategy.

Going further, being able to have access to the optimal strategy for any model
parameters allows us to extend this by combining these strategies over
thousands of stocks. One way to do this is to proceed iteratively by solving
the problems for all stocks independently and then to include risk and
constraint terms through penalization of the alphas of stocks. Another
research direction is to consider more complex models with even more alpha
time scales and transient market impact to apply deep learning to get
numerical solutions to the extensively studied optimal execution problems.

## References

  * [AFP19] Clifford S Asness, Andrea Frazzini and Lasse Heje Pedersen  “Quality minus junk”  In _Review of Accounting Studies_ 24.1 Springer, 2019, pp. 34–112
  * [AMP13] Clifford S Asness, Tobias J Moskowitz and Lasse Heje Pedersen  “Value and momentum everywhere”  In _The Journal of Finance_ 68.3 Wiley Online Library, 2013, pp. 929–985
  * [Ber12] Dimitri Bertsekas  “Dynamic programming and optimal control: Volume I”  Athena scientific, 2012
  * [Boy+17] Stephen Boyd et al.  “Multi-period trading via convex optimization”  In _arXiv preprint arXiv:1705.00109_ , 2017
  * [BS91] Marc Bremer and Richard J Sweeney  “The reversal of large stock-price decreases”  In _The Journal of Finance_ 46.2 Wiley Online Library, 1991, pp. 747–754
  * [CGL17] Gianbiagio Curato, Jim Gatheral and Fabrizio Lillo  “Optimal execution with non-linear transient market impact”  In _Quantitative Finance_ 17.1 Taylor & Francis, 2017, pp. 41–54
  * [Cha+20] Ayman Chaouki et al.  “Deep deterministic portfolio optimization”  In _The Journal of Finance and Data Science_ 6 Elsevier, 2020, pp. 16–30
  * [Con+20] Lin William Cong, Ke Tang, Jingyuan Wang and Yang Zhang  “AlphaPortfolio for investment and economically interpretable AI”  In _SSRN, https://papers. ssrn. com/sol3/papers. cfm_ , 2020
  * [De ̵+12] Joachim De Lataillade, Cyril Deremble, Marc Potters and Jean-Philippe Bouchaud  “Optimal trading with linear costs”  In _arXiv preprint arXiv:1203.5957_ , 2012
  * [Den+16] Yue Deng et al.  “Deep direct reinforcement learning for financial signal representation and trading”  In _IEEE transactions on neural networks and learning systems_ 28.3 IEEE, 2016, pp. 653–664
  * [DJN21] Stéphane Daul, Thibault Jaisson and Alexandra Nagy  “Performance attribution of machine learning methods for stock return prediction”  In _forthcoming_ , 2021
  * [DP20] Philip H Dybvig and Luca Pezzo  “Mean-variance portfolio rebalancing with transaction costs”  In _Available at SSRN 3373329_ , 2020
  * [FHM18] Scott Fujimoto, Herke Hoof and David Meger  “Addressing function approximation error in actor-critic methods”  In _International Conference on Machine Learning_ , 2018, pp. 1587–1596  PMLR
  * [FIM12] Andrea Frazzini, Ronen Israel and Tobias J Moskowitz  “Trading costs of asset pricing anomalies”  In _Fama-Miller Working Paper, Chicago Booth Research Paper_ , 2012
  * [Fra+18] Vincent François-Lavet et al.  “An introduction to deep reinforcement learning”  In _arXiv preprint arXiv:1811.12560_ , 2018
  * [GKX18] Shihao Gu, Bryan Kelly and Dacheng Xiu  “Empirical asset pricing via machine learning”, 2018
  * [GKX19] Shihao Gu, Bryan T Kelly and Dacheng Xiu  “Autoencoder asset pricing models”  Yale ICF Working Paper, 2019
  * [GP13] Nicolae Gârleanu and Lasse Heje Pedersen  “Dynamic trading with predictable returns and transaction costs”  In _The Journal of Finance_ 68.6 Wiley Online Library, 2013, pp. 2309–2340
  * [Hen+18] Peter Henderson et al.  “Deep reinforcement learning that matters”  In _Proceedings of the AAAI conference on artificial intelligence_ 32.1, 2018
  * [HMT19] Blanka Horvath, Aitor Muguruza and Mehdi Tomas  “Deep learning volatility”  In _Available at SSRN 3322085_ , 2019
  * [IS15] Sergey Ioffe and Christian Szegedy  “Batch normalization: Accelerating deep network training by reducing internal covariate shift”  In _International conference on machine learning_ , 2015, pp. 448–456  PMLR
  * [Jai+21] Thibault Jaisson, Laurent Nguyen, Reda Messikh and Gabriele Susinno  “The predictive power of analysts and their impact on prices”  In _Available at SSRN 3973080_ , 2021
  * [KB14] Diederik P Kingma and Jimmy Ba  “Adam: A method for stochastic optimization”  In _arXiv preprint arXiv:1412.6980_ , 2014
  * [KR20] Petter N Kolm and Gordon Ritter  “Modern perspectives on reinforcement learning in finance”  In _Modern Perspectives on Reinforcement Learning in Finance (September 6, 2019). The Journal of Machine Learning in Finance_ 1.1, 2020
  * [LBH15] Yann LeCun, Yoshua Bengio and Geoffrey Hinton  “Deep learning”  In _nature_ 521.7553 Nature Publishing Group, 2015, pp. 436–444
  * [Lil+15] Timothy P Lillicrap et al.  “Continuous control with deep reinforcement learning”  In _arXiv preprint arXiv:1509.02971_ , 2015
  * [Mar68] Harry M Markowitz  “Portfolio selection”  Yale university press, 1968
  * [Mni+15] Volodymyr Mnih et al.  “Human-level control through deep reinforcement learning”  In _nature_ 518.7540 Nature Publishing Group, 2015, pp. 529–533
  * [Pan+17] Xinlei Pan, Yurong You, Ziyan Wang and Cewu Lu  “Virtual to real reinforcement learning for autonomous driving”  In _arXiv preprint arXiv:1704.03952_ , 2017
  * [Pas+17] Adam Paszke et al.  “Automatic differentiation in PyTorch”  In _NIPS-W_ , 2017
  * [SB18] Richard S Sutton and Andrew G Barto  “Reinforcement learning: An introduction”  MIT press, 2018
  * [Sch15] Jürgen Schmidhuber  “Deep learning in neural networks: An overview”  In _Neural networks_ 61 Elsevier, 2015, pp. 85–117
  * [WD92] Christopher JCH Watkins and Peter Dayan  “Q-learning”  In _Machine learning_ 8.3-4 Springer, 1992, pp. 279–292


# Integrating Visuo-tactile Sensing with Haptic Feedback for Teleoperated
Robot Manipulation

Noah Becker21, Erik Gattung21, Kay Hansel1, Tim Schneider1, Yaonan Zhu5,
Yasuhisa Hasegawa5, and Jan Peters1-4 This research received funding from
CHIRON by DFG (PE 2315/8-1), the European Union’s Horizon Europe programme
under Grant Agreement No. 101135959 (project ARISE), BMWSB ZukunftBau (no.
10.08.18.7-21.34), and Aristotle by BMBF; 2Equally contributing authors;
1Computer Science Department, Technische Universität Darmstadt, Germany,
2German Research Center for AI (DFKI), 3Hessian Centre for Artificial
Intelligence (Hessian.AI), 4Centre for Cognitive Science (CogSci).
{noah.becker, erik.gattung} @stud.tu-darmstadt.de, {kay.hansel,
tim.schneider1, jan.peters} @tu-darmstadt.de,
5Department of Micro-Nano Mechanical Science and Engineering, Nagoya
University, Japan<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Telerobotics enables humans to overcome spatial constraints and allows them to
physically interact with the environment in remote locations. However, the
sensory feedback provided by the system to the operator is often purely
visual, limiting the operator’s dexterity in manipulation tasks. In this work,
we address this issue by equipping the robot’s end-effector with high-
resolution visuotactile GelSight sensors. Using low-cost MANUS-Gloves, we
provide the operator with haptic feedback about forces acting at the points of
contact in the form of vibration signals. We propose two different methods for
estimating these forces; one based on estimating the movement of markers on
the sensor surface and one deep-learning approach. Additionally, we integrate
our system into a virtual-reality teleoperation pipeline in which a human
operator controls both arms of a Tiago robot while receiving visual and haptic
feedback. We believe that integrating haptic feedback is a crucial step for
dexterous manipulation in teleoperated robotic systems.

###### Index Terms:

Visuo-Tactile, Haptic-feedback, Teleoperation, Human-robot interaction

##  I INTRODUCTION

Teleoperation, the remote control of robots, enables humans to overcome
spatial constraints and physically interact with the environment in remote
locations [1, 2]. Recently, contributions such as the Nimbro system [3, 4],
the ALOHA system [5], or the mobile ALOHA system [6] attracted a lot of
attention in the field. A common challenge lies in providing feedback that is
not easily conveyed through images, like temperature, surface structure, or
current grip force. Haptic feedback has been looked to as a solution to this
problem, providing additional information through touch instead of sight [7,
8]. This information is particularly relevant in our context, as we are aiming
for an intuitive teleoperation system designed for people with limited
technical knowledge.

In this work, we integrated vibrotactile feedback into a virtual reality (VR)
teleoperation pipeline to improve performance in object manipulation tasks.
This addition extends the visual feedback received from the VR system by
providing tactile sensations. The user operates a dual-arm manipulation robot,
benefiting from both visual and tactile cues to improve control and precision.
Vibration feedback, also called vibrotactile feedback, is one of the most
commonly used types of tactile feedback [9, 10, 11, 12]; used due to its low
profile, cost, and power consumption. To obtain haptic information about the
objects to be manipulated, we attached visuotactile sensors to the robot’s
grippers. We use these sensors to detect the force exerted when interacting
with objects. This information is crucial to ensure that objects are neither
dropped nor crushed when manipulating them. The force data is then converted
into a vibration intensity and transmitted to the user as vibrotactile
feedback. In this way, the user receives tactile feedback in real time,
providing a more intuitive sense of control when gripping or touching objects.
While gripping force could also be measured with simple force sensors,
visuotactile sensors can provide much richer information about the points of
contact, such as object texture [13, 14], object shape [15], or the presence
of slip [16]. In future work, we plan to feed back such information to the
user in addition to force feedback.

In the following, we provide an overview of the developed teleoperation system
and detail methods for visuo-tactile force estimation. We present initial
evaluations validating the proof of concept, followed by an outlook on future
research directions and potential applications.

Figure 1: The Teleoperation setup. Left - Tiago Robot equipped with tactile
GelSight sensor. Right - Comparison of soft and hard grasps of modeling
compound. Figure 2: Overview of different components of the proposed system.
(a) -  The haptic feedback pipeline. (b) -  The teleoperation pipeline. (c)
top - The dot-matrix gel with/without an external force from a cylindrical
object. (c) bottom - The corresponding optical motion flow visualised in 3D
overlaid by upscaled vectors of the motion flow.

##  II SYSTEM AND METHOD

We integrate haptic feedback into a VR teleoperation pipeline, as shown in
Figure 2. The user wears a VR headset from [17] providing a first-person view
through a stereo camera [18] mounted on the robot’s head. This setup enables
intuitive control of a dual-arm manipulation robot such as [19], akin to
systems described in works [20, 21]. For haptic feedback, we use two Prime X
Haptic gloves from MANUS [22], equipped with flex sensors and vibration motors
for each finger. The flex sensors detect the user’s finger pose, a gesture
that the robot’s grippers mimic. Additionally, a VR tracker from [17] on the
glove tracks the user’s hand pose and enables the robot to replicate arm
motions.

The haptic feedback pipeline uses a GelSight Mini [23] with a dot-matrix-gel
mounted to the robot’s gripper to sense contact with an object. The GelSight
streams the video of its gel to a transmitter node, which converts the images
to a haptic signal. This haptic signal gets forwarded to the gloves as
vibration feedback. We propose two different methods for generating the haptic
signal: either the Lucas-Kanade optical motion flow method [24] or a neural
network [25] trained to estimate the force acting on the gel.

For the first method, we track dots on the gel by analyzing small windows
around each dot across consecutive frames. We then calculate spatial gradients
of image intensity within these windows and relates them to intensity changes
over time, solving for each dot’s motion vector. This results in the estimated
motion of each dot as a 2D vector. We sum the magnitude of all motion flow
vectors to estimate the total force across the whole sensor. Previous research
has shown that motion flow is an adequate approximation of force [23, 26].

The neural network we used was proposed by [25], using a similar process as
described in [27]. It takes the image of a GelSight sensor with a dot matrix
and outputs shear force and normal force as a single vector. The magnitude of
this vector is used as the total force.

We apply a threshold to the total force calculated by either algorithm to
prevent the glove from vibrating due to noise or arm movements. In addition,
the values are normalized, and a log scale is applied so that the haptic
feedback range for smaller forces is larger, assisting with fine control.
These values are sent to the Manus glove, which controls the haptic motors to
vibrate proportional to the total force. The maximum vibration is reached when
the robot presses its fingers together with the maximum possible force.
Communication between system components is done using the Robot Operating
System (ROS) [28] and shared memory to minimize latency.

##  III EVALUATION

To evaluate our system’s strengths and weaknesses, we conducted a small
preliminary user study with 7 participants. In this study, participants were
tasked with using our teleoperation system to pick up a plasticine ball as
gently as possible without dropping it. We conducted trials both with and
without haptic feedback and using balls of various sizes. The extent of the
ball’s deformation served as a quantitative metric for the users’ ability to
execute tasks requiring precise force control. Preliminary results show that,
on average, ball deformation was reduced by 48%

For the qualitative analysis, we use the NASA Task Load Index (TLX) [29],
evaluating the subjective workload of the task described above. Preliminary
results of the test trial show that users perceive the haptic feedback as
enhancing performance, although the increased feedback while performing fine
control using haptic feedback also requires more effort to handle. Further
studies, both on quantitative and qualitative measurements, are still ongoing.

##  IV Conclusion

In this work, we developed a teleoperation system in which users get haptic
force feedback through vibration. Our preliminary user study indicates that
this haptic feedback improves dexterity and perceived performance.

In future work, we plan to utilize the rich data of visuotactile sensors to
provide additional haptic feedback to the user, such as shear forces, slip,
and texture, further reducing the reliance on visual cues. Another exciting
avenue for further research is to utilize these sensors in shared control,
e.g., by automatically adjusting gripping strength based on tactile feedback.

## References

  * [1] T. B. Sheridan, _Telerobotics, automation, and human supervisory control_. MIT press, 1992.
  * [2] ——, “Teleoperation, telerobotics and telepresence: A progress report,” _Control Engineering Practice_ , vol. 3, no. 2, pp. 205–214, 1995.
  * [3] M. Schwarz, C. Lenz, A. Rochow, M. Schreiber, and S. Behnke, “Nimbro avatar: Interactive immersive telepresence with force-feedback telemanipulation,” in _2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2021, pp. 5312–5319.
  * [4] C. Lenz, M. Schwarz, A. Rochow, B. Pätzold, R. Memmesheimer, M. Schreiber, and S. Behnke, “Nimbro wins ana avatar xprize immersive telepresence competition: Human-centric evaluation and lessons learned,” _International Journal of Social Robotics_ , pp. 1–25, 2023.
  * [5] T. Z. Zhao, V. Kumar, S. Levine, and C. Finn, “Learning fine-grained bimanual manipulation with low-cost hardware,” _arXiv preprint arXiv:2304.13705_ , 2023.
  * [6] Z. Fu, T. Z. Zhao, and C. Finn, “Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation,” _arXiv preprint arXiv:2401.02117_ , 2024.
  * [7] R. Muradore, P. Fiorini _et al._ , “A review of bilateral teleoperation algorithms,” _Acta Polytechnica Hungarica_ , vol. 13, no. 1, pp. 191–208, 2016.
  * [8] G. Niemeyer, C. Preusche, S. Stramigioli, and D. Lee, “Telerobotics,” _Springer handbook of robotics_ , pp. 1085–1108, 2016.
  * [9] J. Martínez, A. García, M. Oliver, J. P. Molina, and P. González, “Identifying virtual 3d geometric shapes with a vibrotactile glove,” _IEEE computer graphics and applications_ , vol. 36, no. 1, pp. 42–51, 2014\.
  * [10] L. Fritsche, F. Unverzag, J. Peters, and R. Calandra, “First-person tele-operation of a humanoid robot,” in _2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids)_. IEEE, 2015, pp. 997–1002.
  * [11] P. Weber, E. Rueckert, R. Calandra, J. Peters, and P. Beckerle, “A low-cost sensor glove with vibrotactile feedback and multiple finger joint and hand motion sensing for human-robot interaction,” in _2016 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN)_ , 2016, pp. 99–104.
  * [12] S. Tachi, “Telexistence: Enabling humans to be virtually ubiquitous,” _IEEE Computer Graphics and Applications_ , vol. 36, no. 1, pp. 8–14, 2016\.
  * [13] A. Böhm, T. Schneider, B. Belousov, A. Kshirsagar, L. Lin, K. Doerschner, K. Drewing, C. A. Rothkopf, and J. Peters, “What matters for active texture recognition with vision-based tactile sensors,” in _International conference on robotics and automation_ , 2024.
  * [14] W. Yuan, Y. Mo, S. Wang, and E. H. Adelson, “Active Clothing Material Perception Using Tactile Sensing and Deep Learning,” in _International conference on robotics and automation_ , 2018.
  * [15] S. Suresh, Z. Si, J. G. Mangelson, W. Yuan, and M. Kaess, “Shapemap 3-d: Efficient shape mapping through dense touch and vision,” in _International conference on robotics and automation_ , 2022.
  * [16] N. Funk, E. Helmut, G. Chalvatzaki, R. Calandra, and J. Peters, “Evetac: An event-based optical tactile sensor for robotic manipulation,” 2023.
  * [17] H. Corporation, “Vive deutschland,” 2024, accessed Mar. 20, 2024. [Online]. Available: https://www.vive.com/de/
  * [18] Stereolabs, “Zed mini stereo camera,” 2024, accessed Mar. 20, 2024. [Online]. Available: https://store.stereolabs.com/en-de/products/zed-mini
  * [19] P. Robotics, “Tiago++, the robot you need for bi-manual tasks,” 2019, accessed Mar. 14, 2024. [Online]. Available: https://blog.pal-robotics.com/tiago-bi-manual-robot-research/
  * [20] Y. Zhu, S. Nazirjonov, B. Jiang, J. Colan, T. Aoyama, Y. Hasegawa, B. Belousov, K. Hansel, and J. Peters, “Visual tactile sensor based force estimation for position-force teleoperation,” in _2022 IEEE International Conference on Cyborg and Bionic Systems (CBS)_. IEEE, 2023, pp. 49–52.
  * [21] Y. Zhu, B. Jiang, Q. Chen, T. Aoyama, and Y. Hasegawa, “A shared control framework for enhanced grasping performance in teleoperation,” _IEEE Access_ , 2023.
  * [22] M. Meta, “Prime x haptic vr,” 2023, accessed Mar. 15, 2024. [Online]. Available: https://www.manus-meta.com/products/prime-x-haptic
  * [23] W. Yuan, S. Dong, and E. H. Adelson, “Gelsight: High-resolution robot tactile sensors for estimating geometry and force,” _Sensors_ , vol. 17, no. 12, p. 2762, 2017.
  * [24] B. D. Lucas and T. Kanade, “An iterative image registration technique with an application to stereo vision,” in _IJCAI’81: 7th international joint conference on Artificial intelligence_ , vol. 2, 1981, pp. 674–679.
  * [25] E. Helmut, L. Dziarski, N. Funk, B. Belousov, and J. Peters, “Visuotactile shear force estimation,” 2023, unpublished manuscript.
  * [26] W. Yuan, R. Li, M. A. Srinivasan, and E. H. Adelson, “Measurement of shear and slip with a gelsight tactile sensor,” in _2015 IEEE International Conference on Robotics and Automation (ICRA)_ , 2015, pp. 304–311.
  * [27] k. Funk, P. O. Müller, B. Belousov, A. Savchenko, R. Findeisen, and J. Peters, “High-resolution pixelwise contact area and normal force estimation for the gelsight mini visuotactile sensor using neural networks,” in _Embracing Contacts-Workshop at ICRA 2023_ , 2023.
  * [28] Stanford Artificial Intelligence Laboratory et al., “Robotic operating system.” [Online]. Available: https://www.ros.org
  * [29] S. G. Hart and L. E. Staveland, “Development of nasa-tlx (task load index): Results of empirical and theoretical research,” in _Advances in psychology_ , vol. 52. North Holland, 1988, pp. 139–183.