$\displaystyle\qquad\hphantom{=}{}+4\frac{\theta_{1}^{\prime}(2v_{2},\tau)}{\theta_{1}(2v_{2},\tau)}\frac{\theta_{1}^{\prime\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}-4\frac{\theta_{1}^{\prime}(2v_{2},\tau)}{\theta_{1}(2v_{2},\tau)}\frac{\theta_{1}^{\prime^{2}}(v_{2},\tau)}{\theta_{1}^{2}(v_{2},\tau)}.$
(A.7)

###### Proof.

Differentiating the identity with respect to $v_{2}$ we obtain (A.7). ∎

Computing $\boldsymbol{g^{12}}$:

$\displaystyle
g^{12}=\big{(}t^{2}\big{)}^{2}\left[-\frac{\theta_{1}^{\prime\prime\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}+\frac{\theta_{1}^{\prime}(v_{2},\tau)\theta_{1}^{\prime\prime}(v_{2},\tau)}{\theta_{1}^{2}(v_{2},\tau)}-2\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\frac{\theta_{1}^{\prime\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}+2\frac{\theta_{1}^{\prime^{3}}(v_{2},\tau)}{\theta_{1}^{3}(v_{2},\tau)}\right]$
$\displaystyle\hphantom{g^{12}=}{}+\big{(}t^{2}\big{)}^{2}\left[2\frac{\theta_{1}^{\prime}(2v_{2},\tau)}{\theta_{1}(2v_{2},\tau)}\frac{\theta_{1}^{\prime\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}-2\frac{\theta_{1}^{\prime}(2v_{2},\tau)}{\theta_{1}(2v_{2},\tau)}\frac{\theta_{1}^{\prime^{2}}(v_{2},\tau)}{\theta_{1}^{2}(v_{2},\tau)}+4\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\frac{\theta_{1}^{\prime\prime}(2v_{2},\tau)}{\theta_{1}(2v_{2},\tau)}\right]$
$\displaystyle\hphantom{g^{12}=}{}+\big{(}t^{2}\big{)}^{2}\left[-4\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\frac{\theta_{1}^{\prime^{2}}(2v_{2},\tau)}{\theta_{1}^{2}(2v_{2},\tau)}\right].$

Applying (A.7), we get

$\displaystyle g^{12}=-2\pi{\rm
i}\big{(}t^{2}\big{)}^{2}\left[\frac{\partial^{2}}{\partial
v_{2}\partial\tau}\left(\\!\log\left(\frac{\theta_{1}^{\prime}(0,\tau)}{\theta_{1}(2v_{2},\tau)}\right)\right)\right].$

Computing $\boldsymbol{g^{11}}$:

$\displaystyle g^{11}=\frac{1}{2}\left(\frac{\partial t^{1}}{\partial
v_{0}}\right)^{2}-\frac{1}{2}\left(\frac{\partial t^{1}}{\partial
v_{2}}\right)^{2}+2\frac{\partial t^{1}}{\partial u}\frac{\partial
t^{1}}{\partial\tau}=\frac{1}{2}\left(\frac{\partial t^{1}}{\partial
v_{0}}\right)^{2}-\frac{1}{2}\left(\frac{\partial t^{1}}{\partial
v_{2}}\right)^{2}-4\pi{\rm i}t^{1}\frac{\partial t^{1}}{\partial\tau}.$

Computing $\frac{1}{2}\big{(}\frac{\partial t^{1}}{\partial
v_{0}}\big{)}^{2}$, $\frac{1}{2}\big{(}\frac{\partial t^{1}}{\partial
v_{2}}\big{)}^{2}$ and $-4\pi{\rm i}t^{1}\frac{\partial t^{1}}{\partial\tau}$:

To simplify the computation let us define

$\displaystyle
A:=\frac{\theta_{1}^{2}(v_{0},\tau)}{\theta_{1}^{2}(v_{2},\tau)}{\rm
e}^{-2\pi{\rm i}u}.$

Then,

$\displaystyle\frac{1}{2}\left(\frac{\partial t^{1}}{\partial
v_{0}}\right)^{2}=2\frac{\theta_{1}^{\prime^{2}}(v_{0},\tau)}{\theta_{1}^{2}(v_{0},\tau)}A^{2}+4A\frac{\theta_{1}^{\prime}(v_{0},\tau)}{\theta_{1}(v_{0},\tau)}\frac{\partial
t^{2}}{\partial
v_{0}}\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}+2\left(\frac{\partial
t^{2}}{\partial
v_{0}}\right)^{2}\frac{\theta_{1}^{\prime^{2}}(v_{2},\tau)}{\theta_{1}^{2}(v_{2},\tau)},$
$\displaystyle-\frac{1}{2}\left(\frac{\partial t^{1}}{\partial
v_{2}}\right)^{2}\\!=-2\frac{\theta_{1}^{\prime^{2}}(v_{2}\tau)}{\theta_{1}^{2}(v_{2},\tau)}A^{2}+2A\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\\!\left[2\frac{\partial
t^{2}}{\partial
v_{2}}\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}+2t^{2}\\!\left[\frac{\theta_{1}^{\prime\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}-\frac{\theta_{1}^{\prime^{2}}(v_{2},\tau)}{\theta_{1}^{2}(v_{2},\tau)}\right]\right]$
$\displaystyle\hphantom{\frac{1}{2}\left(\frac{\partial t^{1}}{\partial
v_{0}}\right)^{2}=}-2\left(\frac{\partial t^{2}}{\partial
v_{2}}\right)^{2}\frac{\theta_{1}^{\prime^{2}}(v_{2},\tau)}{\theta_{1}^{2}(v_{2},\tau)}-4t^{2}\frac{\partial
t^{2}}{\partial
v_{2}}\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\left[\frac{\theta_{1}^{\prime\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}-\frac{\theta_{1}^{\prime^{2}}(v_{2},\tau)}{\theta_{1}^{2}(v_{2},\tau)}\right]$
$\displaystyle\hphantom{\frac{1}{2}\left(\frac{\partial t^{1}}{\partial
v_{0}}\right)^{2}=}-2\big{(}t^{2}\big{)}^{2}\left[\frac{\theta_{1}^{\prime\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}-\frac{\theta_{1}^{\prime^{2}}(v_{2},\tau)}{\theta_{1}^{2}(v_{2},\tau)}\right]^{2},$
$\displaystyle-4\pi{\rm i}t^{1}\frac{\partial t^{1}}{\partial\tau}=-8\pi{\rm
i}A^{2}\left[\frac{\partial_{\tau}\theta_{1}(v_{0},\tau)}{\theta_{1}(v_{0},\tau)}-\frac{\partial_{\tau}\theta_{1}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\right]-8\pi{\rm
i}A\frac{\partial
t^{2}}{\partial\tau}\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}$
$\displaystyle\hphantom{-4\pi{\rm i}t^{1}\frac{\partial
t^{1}}{\partial\tau}=}{}-8\pi{\rm
i}At^{2}\frac{\partial}{\partial\tau}\left(\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\right)-16\pi{\rm
i}At^{2}\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\left[\frac{\partial_{\tau}\theta_{1}(v_{0},\tau)}{\theta_{1}(v_{0},\tau)}{}-\frac{\partial_{\tau}\theta_{1}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\right]$
$\displaystyle\hphantom{-4\pi{\rm i}t^{1}\frac{\partial
t^{1}}{\partial\tau}=}-16\pi{\rm i}t^{2}\frac{\partial
t^{2}}{\partial\tau}\frac{\theta_{1}^{\prime^{2}}(v_{2},\tau)}{\theta_{1}^{2}(v_{2},\tau)}-16\pi{\rm
i}\big{(}t^{2}\big{)}^{2}\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\frac{\partial}{\partial\tau}\left(\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\right).$

Then, we have

$\displaystyle g^{11}=(1)+(2)+(3)+(4)+(5),$

where

$\displaystyle(1)=A^{2}\left[2\frac{\theta_{1}^{\prime^{2}}(v_{0},\tau)}{\theta_{1}^{2}(v_{0},\tau)}-2\frac{\theta_{1}^{\prime^{2}}(v_{2}\tau)}{\theta_{1}^{2}(v_{2},\tau)}-8\pi{\rm
i}\left[\frac{\partial_{\tau}\theta_{1}(v_{0},\tau)}{\theta_{1}(v_{0},\tau)}-\frac{\partial_{\tau}\theta_{1}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\right]\right]$
$\displaystyle\hphantom{(1)}=A^{2}\left[2\frac{\theta_{1}^{\prime^{2}}(v_{0},\tau)}{\theta_{1}^{2}(v_{0},\tau)}-2\frac{\theta_{1}^{\prime^{2}}(v_{2}\tau)}{\theta_{1}^{2}(v_{2},\tau)}-2\frac{\theta_{1}^{\prime\prime}(v_{0},\tau)}{\theta_{1}(v_{0},\tau)}+2\frac{\theta_{1}^{\prime\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\right]$
$\displaystyle\hphantom{(1)}=2A^{2}[\wp(v_{0})-\wp(v_{2})]=2\frac{16\omega^{4}}{[\wp(v_{0})-\wp(v_{2})]^{2}}[\wp(v_{0})-\wp(v_{2})]=32\frac{\omega^{4}}{\wp(v_{0})-\wp(v_{2})},$
$\displaystyle(2)=-8\pi{\rm
i}t^{2}A\frac{\partial}{\partial\tau}\\!\left(\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\right)\\!+2At^{2}\frac{\theta_{1}^{\prime^{2}}(v_{2},\tau)}{\theta_{1}^{2}(v_{2},\tau)}\\!\left[2\frac{\theta_{1}^{\prime}(v_{0},\tau)}{\theta_{1}(v_{0},\tau)}\\!\left[\frac{\theta_{1}^{\prime}(v_{0}\\!-\\!v_{2},\tau)}{\theta_{1}(v_{0}\\!-\\!v_{2},\tau)}\\!+\\!\frac{\theta_{1}^{\prime}(v_{0}\\!+\\!v_{2},\tau)}{\theta_{1}(v_{0}\\!+\\!v_{2},\tau)}\right]\right]$
$\displaystyle\hphantom{(2)=}+2At^{2}\frac{\theta_{1}^{\prime^{2}}(v_{2},\tau)}{\theta_{1}^{2}(v_{2},\tau)}\left[2\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\left[\frac{-\theta_{1}^{\prime}(v_{0}-v_{2},\tau)}{\theta_{1}(v_{0}-v_{2},\tau)}+\frac{\theta_{1}^{\prime}(v_{0}+v_{2},\tau)}{\theta_{1}(v_{0}+v_{2},\tau)}-2\frac{\theta_{1}^{\prime}(2v_{2},\tau)}{\theta_{1}(2v_{2},\tau)}\right]\right]$
$\displaystyle\hphantom{(2)=}+2At^{2}\frac{\theta_{1}^{\prime^{2}}(v_{2},\tau)}{\theta_{1}^{2}(v_{2},\tau)}\left[2\left[\frac{\theta_{1}^{\prime\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}-\frac{\theta_{1}^{\prime^{2}}(v_{2},\tau)}{\theta_{1}^{2}(v_{2},\tau)}\right]-8\pi{\rm
i}\left[\frac{\partial_{\tau}\theta_{1}(v_{0},\tau)}{\theta_{1}(v_{0},\tau)}-\frac{\partial_{\tau}\theta_{1}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\right]\right]$
$\displaystyle\hphantom{(2)=}+\\!2At^{2}\frac{\theta_{1}^{\prime^{2}}(v_{2},\tau)}{\theta_{1}^{2}(v_{2},\tau)}\\!\\!\left[\\!-4\pi{\rm
i}\\!\left[\frac{\partial_{\tau}\theta_{1}(v_{0}\\!+\\!v_{2},\tau)}{\theta_{1}(v_{0}\\!+\\!v_{2},\tau)}\\!+\\!\frac{\partial_{\tau}\theta_{1}(v_{0}\\!-\\!v_{2},\tau)}{\theta_{1}(v_{0}\\!-\\!v_{2},\tau)}\\!-\\!\frac{\partial_{\tau}\theta_{1}(2v_{2},\tau)}{\theta_{1}(2v_{2},\tau)}\\!-\\!\frac{\partial_{\tau}\theta_{1}^{\prime}(0,\tau)}{\theta_{1}^{\prime}(0,\tau)}\\!\right]\\!\right]\\!.$

Using (A.2),

$\displaystyle
2\frac{\theta_{1}^{\prime\prime}(v_{0},\tau)}{\theta_{1}(v_{0},\tau)}+2\frac{\theta_{1}^{\prime\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}-4\pi{\rm
i}\frac{\partial}{\partial\tau}\left(\log\left(\frac{\theta_{1}^{\prime}(0,\tau)}{\theta(v_{0}-v_{2},\tau)}\right)\right)-4\pi{\rm
i}\frac{\partial}{\partial\tau}\left(\log\left(\frac{\theta_{1}^{\prime}(0,\tau)}{\theta(v_{0}+v_{2},\tau)}\right)\right)$
$\displaystyle\qquad{}=2\frac{\theta_{1}^{\prime}(v_{0},\tau)}{\theta_{1}(v_{0},\tau)}\left(\frac{\theta_{1}^{\prime}(v_{0}+v_{2},\tau)}{\theta_{1}(v_{0}+v_{2},\tau)}+\frac{\theta_{1}^{\prime}(v_{0}-v_{2},\tau)}{\theta_{1}(v_{0}-v_{2},\tau)}\right)$
$\displaystyle\qquad\phantom{=}+2\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\left(-\frac{\theta_{1}^{\prime}(v_{0}-v_{2},\tau)}{\theta_{1}(v_{0}-v_{2},\tau)}+\frac{\theta_{1}^{\prime}(v_{0}+v_{2},\tau)}{\theta_{1}(v_{0}+v_{2},\tau)}\right),$
$\displaystyle(2)=-8\pi{\rm
i}t^{2}A\frac{\partial}{\partial\tau}\left(\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\right)+2At^{2}\frac{\theta_{1}^{\prime^{2}}(v_{2},\tau)}{\theta_{1}^{2}(v_{2},\tau)}\left[-4\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\frac{\theta_{1}^{\prime}(2v_{2},\tau)}{\theta_{1}(2v_{2},\tau)}\right]$
$\displaystyle\phantom{(2)=}+2At^{2}\frac{\theta_{1}^{\prime^{2}}(v_{2},\tau)}{\theta_{1}^{2}(v_{2},\tau)}\left[2\left[\frac{\theta_{1}^{\prime\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}-\frac{\theta_{1}^{\prime^{2}}(v_{2},\tau)}{\theta_{1}^{2}(v_{2},\tau)}\right]+4\frac{\theta_{1}^{\prime\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\right]$
$\displaystyle\hphantom{(2)=}+2At^{2}\frac{\theta_{1}^{\prime^{2}}(v_{2},\tau)}{\theta_{1}^{2}(v_{2},\tau)}\left[-4\pi{\rm
i}\frac{\partial_{\tau}\theta_{1}^{\prime}(0,\tau)}{\theta_{1}^{\prime}(0,\tau)}+4\pi{\rm
i}\frac{\partial_{\tau}\theta_{1}(2v_{2},\tau)}{\theta_{1}(2v_{2},\tau)}\right].$

Using again (A.2),

$\displaystyle(2)=-8\pi{\rm
i}t^{2}A\frac{\partial}{\partial\tau}\left(\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\right)+8At^{2}\frac{\theta_{1}^{\prime^{2}}(v_{2},\tau)}{\theta_{1}^{2}(v_{2},\tau)}\left[\frac{\theta_{1}^{\prime\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}-\frac{\theta_{1}^{\prime^{2}}(v_{2},\tau)}{\theta_{1}^{2}(v_{2},\tau)}\right],$
$\displaystyle(3)=4\frac{\theta_{1}^{\prime^{2}}(v_{2},\tau)}{\theta_{1}^{2}(v_{2},\tau)}\left[\frac{1}{2}\left(\frac{\partial
t^{2}}{\partial v_{0}}\right)^{2}-\frac{1}{2}\left(\frac{\partial
t^{2}}{\partial v_{2}}\right)^{2}-4\pi{\rm i}t^{2}\frac{\partial
t^{2}}{\partial\tau}\right]$
$\displaystyle\hphantom{(3)}=8\frac{\theta_{1}^{\prime^{2}}(v_{2},\tau)}{\theta_{1}^{2}(v_{2},\tau)}\big{(}t^{2}\big{)}^{2}\left[\frac{\theta_{1}^{\prime\prime}(2v_{2},\tau)}{\theta_{1}(2v_{2},\tau)}-\frac{\theta_{1}^{\prime^{2}}(2v_{2},\tau)}{\theta_{1}^{2}(2v_{2},\tau)}\right],$
$\displaystyle(4)=-2\big{(}t^{2}\big{)}^{2}\left[\frac{\partial}{\partial
v_{2}}\left(\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\right)\right]^{2}-16\pi{\rm
i}\big{(}t^{2}\big{)}^{2}\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\frac{\partial}{\partial\tau}\left(\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\right),$
$\displaystyle(5)=-4\big{(}t^{2}\big{)}\frac{\partial t^{2}}{\partial
v_{2}}\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\frac{\partial}{\partial
v_{2}}\left(\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\right)$
$\displaystyle\hphantom{(5)}=-4\big{(}t^{2}\big{)}^{2}\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\frac{\partial}{\partial
v_{2}}\left(\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\right)\left[\frac{-\theta_{1}^{\prime}(v_{0}-v_{2},\tau)}{\theta_{1}(v_{0}-v_{2},\tau)}+\frac{\theta_{1}^{\prime}(v_{0}+v_{2},\tau)}{\theta_{1}(v_{0}+v_{2},\tau)}-2\frac{\theta_{1}^{\prime}(2v_{2},\tau)}{\theta_{1}(2v_{2},\tau)}\right]$
$\displaystyle\hphantom{(5)}=-4\big{(}t^{2}\big{)}^{2}\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\frac{\partial}{\partial
v_{2}}\left(\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\right)\left[\frac{-\theta_{1}^{\prime}(v_{0}-v_{2},\tau)}{\theta_{1}(v_{0}-v_{2},\tau)}+\frac{\theta_{1}^{\prime}(v_{0}+v_{2},\tau)}{\theta_{1}(v_{0}+v_{2},\tau)}-2\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\right]$
$\displaystyle\hphantom{(5)=}-4\big{(}t^{2}\big{)}^{2}\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\frac{\partial}{\partial
v_{2}}\left(\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\right)\left[2\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}-2\frac{\theta_{1}^{\prime}(2v_{2},\tau)}{\theta_{1}(2v_{2},\tau)}\right]$
$\displaystyle\hphantom{(5)}=-4\big{(}t^{2}\big{)}A\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\frac{\partial}{\partial
v_{2}}\left(\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\right)$
$\displaystyle\hphantom{(5)=}-4\big{(}t^{2}\big{)}^{2}\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\frac{\partial}{\partial
v_{2}}\left(\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\right)\left[2\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}-2\frac{\theta_{1}^{\prime}(2v_{2},\tau)}{\theta_{1}(2v_{2},\tau)}\right].$

Summing $(2)$ and $(5)$

$\displaystyle(2)+(5)=-8\pi{\rm
i}t^{2}A\frac{\partial}{\partial\tau}\left(\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\right)+8At^{2}\frac{\theta_{1}^{\prime^{2}}(v_{2},\tau)}{\theta_{1}^{2}(v_{2},\tau)}\left[\frac{\theta_{1}^{\prime\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}-\frac{\theta_{1}^{\prime^{2}}(v_{2},\tau)}{\theta_{1}^{2}(v_{2},\tau)}\right]$
$\displaystyle\hphantom{(2)+(5)=}-4\big{(}t^{2}\big{)}A\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\frac{\partial}{\partial
v_{2}}\left(\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\right)$
$\displaystyle\hphantom{(2)+(5)=}-4\big{(}t^{2}\big{)}^{2}\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\frac{\partial}{\partial
v_{2}}\left(\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\right)\left[2\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}-2\frac{\theta_{1}^{\prime}(2v_{2},\tau)}{\theta_{1}(2v_{2},\tau)}\right]$
$\displaystyle\hphantom{(2)+(5)}=-4\big{(}t^{2}\big{)}^{2}\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\frac{\partial}{\partial
v_{2}}\left(\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\right)\left[2\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}-2\frac{\theta_{1}^{\prime}(2v_{2},\tau)}{\theta_{1}(2v_{2},\tau)}\right]$
$\displaystyle\hphantom{(2)+(5)=}+At^{2}\left[{-}\,2\frac{\theta_{1}^{\prime\prime\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}+6\frac{\theta_{1}^{\prime}(v_{2},\tau)\theta_{1}^{\prime\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}-4\frac{\theta_{1}^{\prime^{3}}(v_{2},\tau)}{\theta^{3}_{1}(v_{2},\tau)}\right]$
$\displaystyle\hphantom{(2)+(5)}=-4\big{(}t^{2}\big{)}^{2}\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\frac{\partial}{\partial
v_{2}}\left(\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\right)\left[2\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}-2\frac{\theta_{1}^{\prime}(2v_{2},\tau)}{\theta_{1}(2v_{2},\tau)}\right]+2At^{2}\wp^{\prime}(v_{2})$
$\displaystyle\hphantom{(2)+(5)}=-4\big{(}t^{2}\big{)}^{2}\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\frac{\partial}{\partial
v_{2}}\\!\left(\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\right)\\!\left[2\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}-2\frac{\theta_{1}^{\prime}(2v_{2},\tau)}{\theta_{1}(2v_{2},\tau)}\right]\\!-32\frac{\omega^{4}}{\wp(v_{0})\\!-\\!\wp(v_{2})}.$

Summing $(1)$ and $(2)+(5)$

$\displaystyle(1)+(2)+(5)=-4\big{(}t^{2}\big{)}^{2}\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\frac{\partial}{\partial
v_{2}}\left(\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\right)\left[2\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}-2\frac{\theta_{1}^{\prime}(2v_{2},\tau)}{\theta_{1}(2v_{2},\tau)}\right].$

From the above results, we find

$\displaystyle g^{11}=$ $\displaystyle(1)+(2)+(5)+(3)+(4)$ $\displaystyle=$
$\displaystyle-4\big{(}t^{2}\big{)}^{2}\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\frac{\partial}{\partial
v_{2}}\left(\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\right)\left[2\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}-2\frac{\theta_{1}^{\prime}(2v_{2},\tau)}{\theta_{1}(2v_{2},\tau)}\right]$
$\displaystyle+8\frac{\theta_{1}^{\prime^{2}}(v_{2},\tau)}{\theta_{1}^{2}(v_{2},\tau)}\big{(}t^{2}\big{)}^{2}\left[\frac{\theta_{1}^{\prime\prime}(2v_{2},\tau)}{\theta_{1}(2v_{2},\tau)}-\frac{\theta_{1}^{\prime^{2}}(2v_{2},\tau)}{\theta_{1}^{2}(2v_{2},\tau)}\right]$
$\displaystyle-2\big{(}t^{2}\big{)}^{2}\left[\frac{\partial}{\partial
v_{2}}\left(\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\right)\right]^{2}-16\pi{\rm
i}\big{(}t^{2}\big{)}^{2}\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\frac{\partial}{\partial\tau}\left(\frac{\theta_{1}^{\prime}(v_{2},\tau)}{\theta_{1}(v_{2},\tau)}\right).$

Summarizing, we have proved the identities (3.7) and (3.8).

### Acknowledgements

I am grateful to Professor Boris Dubrovin for proposing this problem, for his
remarkable advises and guidance. I would like also to thanks Professors Davide
Guzzetti and Marco Bertola for helpful discussions, and guidance of this
paper. In addition, I thank the anonymous referees for their valuable comments
and remarks, that have helped to improve the paper.

## References

  * [1] Almeida G.F., Differential geometry of orbit space of extended affine Jacobi group $A_{n}$, arXiv:2004.01780.
  * [2] Bertola M., Frobenius manifold structure on orbit space of Jacobi groups. I, Differential Geom. Appl. 13 (2000), 19–41.
  * [3] Bertola M., Frobenius manifold structure on orbit space of Jacobi groups. II, Differential Geom. Appl. 13 (2000), 213–233.
  * [4] Bourbaki N., Lie groups and Lie algebras, Chapters 4–6, Elements of Mathematics (Berlin), Springer-Verlag, Berlin, 2002.
  * [5] Cutimanco M., Shramchenko V., Explicit examples of Hurwitz Frobenius manifolds in genus one, J. Math. Phys. 61 (2020), 013501, 20 pages.
  * [6] Dubrovin B., Geometry of $2$D topological field theories, in Integrable Systems and Quantum Groups (Montecatini Terme, 1993), Lecture Notes in Math., Vol. 1620, Springer, Berlin, 1996, 120–348, arXiv:hep-th/9407018.
  * [7] Dubrovin B., Differential geometry of the space of orbits of a Coxeter group, in Surveys in Differential Geometry: Integrable Systems, Surv. Differ. Geom., Vol. 4, Int. Press, Boston, MA, 1998, 181–211, arXiv:hep-th/9303152.
  * [8] Dubrovin B., Hamiltonian perturbations of hyperbolic PDEs: from classification results to the properties of solutions, in New Trends in Mathematical Physics, Springer, Dordrecht, 2009, 231–276.
  * [9] Dubrovin B., Strachan I.A.B., Zhang Y., Zuo D., Extended affine Weyl groups of BCD-type: their Frobenius manifolds and Landau–Ginzburg superpotentials, Adv. Math. 351 (2019), 897–946, arXiv:1510.08690.
  * [10] Dubrovin B., Zhang Y., Extended affine Weyl groups and Frobenius manifolds, Compositio Math. 111 (1998), 167–219, arXiv:hep-th/9611200.
  * [11] Dubrovin B., Zhang Y., Normal forms of hierarchies of integrable PDEs, Frobenius manifolds and Gromov–Witten invariants, arXiv:math.DG/0108160.
  * [12] Eichler M., Zagier D., The theory of Jacobi forms, Progress in Mathematics, Vol. 55, Birkhäuser Boston, Inc., Boston, MA, 1985.
  * [13] Ferapontov E.V., Pavlov M.V., Xue L., Second-order integrable Lagrangians and WDVV equations, arXiv:2007.03768.
  * [14] Hertling C., Multiplication on the tangent bundle, arXiv:math.AG/9910116.
  * [15] Pavlov M.V., Tsarev S.P., Tri-Hamiltonian structures of Egorov systems of hydrodynamic type, Funct. Anal. Appl. 37 (2003), 32–45.
  * [16] Romano S., 4-dimensional Frobenius manifolds and Painleve’ VI, Math. Ann. 360 (2014), 715–751, arXiv:1209.3959.
  * [17] Romano S., Frobenius structures on double Hurwitz spaces, Int. Math. Res. Not. 2015 (2015), 538–577, arXiv:1210.2312.
  * [18] Saito K., Yano T., Sekiguchi J., On a certain generator system of the ring of invariants of a finite reflection group, Comm. Algebra 8 (1980), 373–408.
  * [19] Shramchenko V., Deformations of Frobenius structures on Hurwitz spaces, Int. Math. Res. Not. 2005 (2005), 339–387, arXiv:math-ph/0408026.
  * [20] Strachan I.A.B., Frobenius submanifolds, J. Geom. Phys. 38 (2001), 285–307, arXiv:math.DG/9912081.
  * [21] Strachan I.A.B., Frobenius manifolds: natural submanifolds and induced bi-Hamiltonian structures, Differential Geom. Appl. 20 (2004), 67–99, arXiv:math.DG/0201039.
  * [22] Whittaker E.T., Watson G.N., A course of modern analysis, Cambridge Mathematical Library, Cambridge University Press, Cambridge, 1996.
  * [23] Wirthmüller K., Root systems and Jacobi forms, Compositio Math. 82 (1992), 293–354.
  * [24] Zuo D., Frobenius manifolds and a new class of extended affine Weyl groups of A-type, Lett. Math. Phys. 110 (2020), 1903–1940, arXiv:1905.09470.


# Neuro-Symbolic Approaches for Context-Aware Human Activity Recognition

Luca Arrotta<EMAIL_ADDRESS>0000-0001-5207-566X ,  Gabriele Civitarese
<EMAIL_ADDRESS>0000-0002-8247-2524 and  Claudio Bettini
<EMAIL_ADDRESS>0000-0002-1727-7650 University of MilanVia Celoria,
18MilanItaly

(2023)

###### Abstract.

Deep Learning models are a standard solution for sensor-based Human Activity
Recognition (HAR), but their deployment is often limited by labeled data
scarcity and model’s opacity. Neuro-Symbolic AI (NeSy) provides an interesting
research direction to mitigate these issues by infusing knowledge about
context information into HAR deep learning classifiers. However, existing NeSy
methods for context-aware HAR require computationally expensive symbolic
reasoners during classification, making them less suitable for deployment on
resource-constrained devices (e.g., mobile devices). Additionally, NeSy
approaches for context-aware HAR have never been evaluated on in-the-wild
datasets, and their generalization capabilities in real-world scenarios are
questionable. In this work, we propose a novel approach based on a semantic
loss function that infuses knowledge constraints in the HAR model during the
training phase, avoiding symbolic reasoning during classification. Our results
on scripted and in-the-wild datasets show the impact of different semantic
loss functions in outperforming a purely data-driven model. We also compare
our solution with existing NeSy methods and analyze each approach’s strengths
and weaknesses. Our semantic loss remains the only NeSy solution that can be
deployed as a single DNN without the need for symbolic reasoning modules,
reaching recognition rates close (and better in some cases) to existing
approaches.

human activity recognition, neuro-symbolic, knowledge infusion, context-
awareness, knowledge-based reasoning

††journalyear: 2023††ccs: Human-centered computing Mobile computing††ccs:
Human-centered computing Mobile devices

##  1\. Introduction

The sensor-based Human Activity Recognition (HAR) research area is dominated
by solutions based on purely data-driven Deep Learning (DL) models (Wang et
al., 2019; Chen et al., 2021). While DL-based solutions are very effective,
they still have some open research issues that limit their deployment in real-
world scenarios. Among the major problems, there are labeled data scarcity
(Abdallah et al., 2018) and the lack of transparency of the activity models
(Atzmueller et al., 2018).

In the literature, purely knowledge-based approaches have been proposed to
tackle both problems (Gayathri et al., 2017). Symbolic methods rely on domain
knowledge (e.g., based on common-sense knowledge) to model constraints between
sensor events and activities. The sensor data stream is then matched with
symbolic rules to identify the most likely activities according to knowledge.
Purely knowledge-based methods have two advantages: 1) they do not require
labeled data, and 2) they are based on human-readable formalisms that make
them interpretable and transparent. However, these approaches are too rigid
since it is unlikely that logic constraints can cover all the possible
patterns related to activity execution. Moreover, they are not suitable for
sensors that generate continuous values (e.g., accelerometer) since raw data
can not be mapped to a clear semantic.

In the general machine learning community, Neuro-Symbolic AI (NeSy) methods
are emerging to combine the strengths of data-driven and knowledge-based
methods (Hitzler and Sarker, 2022). The idea of NeSy methods is to enhance DL
models through domain knowledge. The potential advantages of NeSy are many.
First, it may significantly improve the recognition rate by driving the
classification with domain constraints. This may be especially true when only
a limited amount of labeled data is available; hence, those constraints can
not be learned directly from data. For the same reason, the use of domain
knowledge can potentially improve the classification of those cases out of the
training set distribution samples. Moreover, DL models enhanced through domain
knowledge have the potential of being more interpretable and transparent,
since their decisions are also influenced by the knowledge model (Li et al.,
2022).

This work focuses on sensor-based HAR on mobile/wearable devices (e.g.,
smartphones, smartwatches). While the majority of existing works in this field
only focus on inertial sensors, we also consider high-level context data
(e.g., semantic position, weather) as also proposed by a few research groups
(Riboni and Bettini, 2011; Cao et al., 2018; Asim et al., 2020). This research
area is usually referred to as Context-Aware Human Activity Recognition. In
the literature, a few NeSy approaches for context-aware HAR have been proposed
by applying knowledge-based reasoning on high-level context data (Arrotta et
al., 2022b). To the best of our knowledge, such approaches have never been
evaluated on public in-the-wild datasets, but only on small datasets acquired
in a scripted fashion.

Moreover, the existing approaches in the literature involve symbolic reasoning
during both training and classification. In real-world deployments, where the
DL model is deployed on resource-constrained devices (e.g., mobile/wearable
devices), the adoption of symbolic reasoning during classification is not
desirable since it is too computationally demanding. Empirical experiments in
the literature show that running symbolic reasoning on mobile devices is up to
$150$ times slower than on machines with higher resources (e.g., servers)
(Bobed et al., 2015). In the HAR domain, the work in (Bettini et al., 2020)
reports that context-aware ontological reasoning on mobile devices takes on
average $1.3$ seconds for each data sample. Since samples are collected with
high periodicity (e.g., a few seconds), such approaches may be inefficient in
terms of computational resources.

In this work, we propose a novel NeSy method for Context-Aware HAR on mobile
devices. Our approach is based on a custom loss function that combines a
standard classification loss and a novel semantic loss function based on
symbolic reasoning. Our semantic loss drives the activity model to classify
activities considering both raw sensor data patterns and high-level knowledge
constraints. Indeed, after the training phase, the classifier internally
encodes such constraints, that are exploited to classify activities without
requiring symbolic reasoning.

Our experimental evaluation on scripted and in-the-wild datasets show that our
method based on a semantic loss outperforms in terms of recognition rate a
classic $DL$ approach based on a standard classification loss. We also
compared our approach with two alternative NeSy strategies that use symbolic
reasoning during classification, showing that our semantic loss often reaches
recognition rates close (and sometimes better) to such state-of-the-art
methods, while avoiding the significant cost of performing symbolic reasoning
during inference. Hence, we believe that our semantic loss reaches a good
trade-off between efficiency and recognition rate.

To summarize, our contributions are the following:

  * •

We formalize the NeSy Context-Aware HAR research problem and reformulate
existing solutions using our notation.

  * •

We propose a novel NeSy solution for Context-Aware HAR, based on a semantic
loss function that does not require symbolic reasoning after training.

  * •

We performed an extensive evaluation on scripted and in-the-wild datasets,
comparing our solution with two existing NeSy methods that require symbolic
reasoning during classification. Our results show that, especially considering
in-the-wild settings, our semantic loss reaches recognition rates that are
often close (and sometimes better) than the ones of the other approaches.

##  2\. Related work

Most of the works proposed in the literature for sensor-based HAR on
mobile/wearable devices rely on supervised Deep Learning (DL) methods (Wang et
al., 2019; Chen et al., 2021). The combination of inertial and high-level
context data has the potential to significantly improve the recognition rate
compared to considering only inertial sensors as proposed by the majority of
the works (Saguna et al., 2013).

Despite their success, existing DL solutions require a large amount of labeled
data during the learning process. Unfortunately, the annotation process is
error-prone, expensive, time-consuming, and tedious, especially considering
large amounts of data. Moreover, the inner mechanisms of deep learning
classifiers are opaque, thus not allowing humans to understand the rationale
behind each model’s prediction.

To mitigate the data scarcity problem, the HAR research community investigated
transfer learning, unsupervised learning, and semi-supervised learning
approaches (Chen et al., 2021). Transfer learning methods usually take
advantage of models trained on a source domain with a significant amount of
labeled data. Such pre-trained models are then fine-tuned in a target domain
using a small amount of labeled samples (Sanabria et al., 2021; Soleimani and
Nazerfard, 2021). On the other hand, semi-supervised approaches for HAR rely
on small labeled datasets to initialize the model, which is then incrementally
updated by leveraging the unlabeled data stream (Abdallah et al., 2018). Semi-
supervised methods for HAR include self-learning, co-learning, active
learning, and label propagation. Finally, unsupervised learning strategies can
be used to derive activity clusters from the available large amounts of
unlabeled data (Kwon et al., 2014; Jain et al., 2022; Hiremath et al., 2022).

In general, the major drawback of the above-mentioned solutions is that they
are not conceived to be interpretable. Hence, they can be considered black
boxes. In the literature, the majority of interpretable models for HAR are not
based on deep learning but on less effective inherently interpretable models
(Atzmueller et al., 2018; Guesgen, 2020). While a few works consider deep
models (Arrotta et al., 2022a), they do not take into account data scarcity.

Neuro-Symbolic AI (NeSy) integrates neural and symbolic AI architectures to
combine their abilities to perform learning and knowledge-based reasoning
(Kamruzzaman Sarker et al., 2021). This combination improves the capability of
the deep learning classifier to learn from smaller amounts of training data,
and, at the same time, it also increases its interpretability (Gaur et al.,
2021). While most NeSy methods have been proposed for computer vision and NLP
applications, only a few NeSy methods exist for HAR. Considering HAR in smart-
home environments, the domain knowledge can be used to derive an initial
activity model that is subsequently adapted to the user’s habits through data-
driven strategies (Sukor et al., 2019). In (Azkune and Almeida, 2018),
unsupervised methods are used to extract frequent patterns from unlabeled
data. These patterns are then associated with the corresponding activities
through domain knowledge. However, while these approaches are effective on
smart-home environmental sensors, they cannot be applied to the inertial
sensors data provided by mobile and wearable devices, which are the focus of
this work.

Considering context-aware HAR on mobile devices, (Bettini et al., 2020)
proposed to use domain knowledge on high-level context data to refine the
predictions of an activity classifier trained on inertial sensors data.
Finally, a recent work proposes the infusion of domain knowledge on context
data into the deep learning classifier during both the training and inference
phases (Arrotta et al., 2022b). However, these approaches rely on ontological
reasoning during classification, which may be critical for the deployment of
resource-constrained mobile devices. More details about existing NeSy context-
aware HAR approaches will be presented in Section 3.3.

To the best of our knowledge, this is the first work that proposes a NeSy
solution for HAR with the following characteristics: a) it infuses knowledge
directly inside the DNN activity model during training, and b) it does not
require symbolic reasoning during classification.

##  3\. Preliminaries

In this section, we formalize context-aware HAR and we formulate the NeSy
Context-Aware HAR problem. Moreover, we take advantage of this formalization
to re-formulate existing NeSy strategies for Context-Aware HAR.

###  3.1. Context-Aware Human Activity Recognition

Let $D_{u}$ be the dataset of raw sensor data collected from the mobile
devices (e.g., smartphone, smartwatch) of a user $u$. Given a set of users
$U=\\{u_{1},\dots,u_{n}\\}$, let $D^{\star}=\\{D_{u_{1}},\dots,D_{u_{n}}\\}$
be the set of the datasets from all the users. Let $A=\\{a_{1},\dots,a_{k}\\}$
be the set of considered activities. The dataset $D^{\star}$ is associated
with a set of annotations $L$ that describes the activities performed by each
user $u$. Each annotation $l\in L$ is a tuple $l=\langle
u,a,t_{s},t_{e}\rangle$ where $a$ is a label identifying the activity actually
performed by $u$ during the time interval $[t_{s},t_{e}]$. Each user dataset
$D_{u}$ is partitioned in a set of non-overlapping fixed-length windows
$W_{u}=\\{w_{1},\dots,w_{q}\\}$ with each window including $z$ seconds of
consecutive raw sensor data of $D_{u}$.

In this work, we use the notion of _context_ as a specific high-level
situation that occurs in the environment surrounding and including the user
while sensor data are being acquired (e.g., it is raining, location is a park,
current speed is high). Let $C=\langle C_{1},\dots,C_{p}\rangle$ be a set of
possible contexts that are meaningful for the application domain.

Considering the recognition of physical activities (e.g., walking, sitting on
transport), each window $w$ can be partitioned into $w^{I}$ and $w^{C}$.
$w^{I}$ are sensor data that cannot be used to derive high-level contexts
(e.g., data from inertial sensors). On the other hand, $w^{C}$ are those raw
sensor data that can be used to derive high-level contexts in $C$ through
reasoning and/or abstraction. More specifically, given a window $w=\langle
w^{C},w^{I}\rangle$, let $ca(w^{C})$ be a Context Aggregation function that
derives all the contexts $C^{w}\subset C$ that are true during $w$ based on
$w^{C}$. This function can rely on simple rules, available services, or
context-aware middlewares (Henricksen et al., 2005). For instance, the
geographical coordinates provided by the location service of the user’s
smartphone can be used to derive her semantic location (e.g., at home, in a
public park) by querying a dedicated web service.

###### Definition 3.1 (Context-aware HAR).

Given a dataset $D^{\star}$ and the annotations set $L$, the problem of
context-aware Human Activity Recognition is to provide to an unseen tuple
$\langle w^{I},C^{w}\rangle$, derived from a sensor data window $w$ from user
$u$, the probability distribution $P=\langle p_{1},\dots,p_{k}\rangle$, where
$p_{i}$ is the probability that $u$ performed the activity $a_{i}$ in contexts
$C^{w}$, with $\sum_{i=1}^{k}p_{i}=1$.

###  3.2. Neuro-Symbolic Context-Aware HAR

The context-aware HAR problem could be tackled by using purely data-driven
models where context data are simply used as input. However, based on a set of
contexts $C$, it is possible to build a knowledge model $K$ that encodes
relationships between the activities in $A$ and the contexts in $C$. For
instance, according to common-sense knowledge, the activity lying is less
likely performed in outdoor settings, like a public park, when it is raining.
Note that $K$ can be built in several different ways: by domain experts using
common-sense knowledge on HAR, re-using existing knowledge bases (e.g.,
ontologies), or considering semi-automatic approaches in charge of extracting
knowledge from external sources (e.g., text, images, and videos from the web).

Given a knowledge model $K$ and a set of contexts $C^{w}$, let $SR(K,C^{w})$
be a symbolic reasoning function that outputs the set of activities
$A^{\star}$ that are consistent with $C^{w}$ according to the constraints in
$K$.

###### Definition 3.2 (Neuro-Symbolic Context-Aware HAR model).

A Neuro-Symbolic Context-Aware Human Activity Recognition model combines a
deep learning model $DNN$ and the symbolic reasoning function $SR()$ to solve
the context-aware HAR problem.

Figure 1 graphically illustrate the high-level structure of NeSy Context-Aware
HAR. As we will describe in Sections 3.3 and 4, the $DNN$ and the $SR()$
modules can be combined in different ways, based on the specific NeSy context-
aware HAR technique.

Figure 1. The neuro-symbolic context-aware HAR approach

###  3.3. Revisiting Existing Neuro-Symbolic Approaches

In this section, we re-formulate existing Neuro-Symbolic AI (NeSy) approaches
with the notation introduced in sections 3.1 and 3.2 to compare them with our
novel NeSy approach in an appropriate way. In particular, we consider two
state-of-the-art approaches for NeSy HAR: context refinement and symbolic
features.

####  3.3.1. Context refinement

The goal of the context refinement method (Bettini et al., 2020) is to revise
a posteriori the $DNN$ predictions through the HAR knowledge encoded into $K$.
As shown in Figure 2, the $DNN$ is trained with the cross-entropy loss
function $\mathcal{L}_{cross}$, which penalizes misclassifications on the
training data. During classification, the output of the $SR()$ function is
used to exclude from the probability distributions derived by $DNN$ on a
specific input those activities that are unlikely to be the correct
predictions considering the current context of the user.

Figure 2. The context refinement neuro-symbolic approach

More formally, given a probability distribution $P=\langle
p_{1},\dots,p_{k}\rangle$ emitted by $DNN$ on a tuple $\langle
w^{I},C^{w}\rangle$, and the set $A^{\star}$ of context-consistent activities
provided by $SR(K,C^{w})$, the probability $p_{i}$ is discarded from $P$ if
$a_{i}\notin A^{\star}$, and $P$ is normalized to obtain again a probability
distribution over the context-consistent activities of $A^{\star}$.

The objective of context refinement is to correct a significant number of
wrong decisions made by $DNN$, thus increasing its recognition rate. At the
same time, it ensures that each classified activity is consistent with the
surrounding context of the user. However, computing $SR()$ is strictly
required during the inference phase of $DNN$, and its high computational cost
complicates the deployment of the context refinement method on mobile devices.
Moreover, if $K$ encodes rigid constraints about the relationships between
contexts and activities, context refinement would discard activities that are
occasionally performed in unusual context scenarios (e.g., running at the
mall).

####  3.3.2. Symbolic features

The objective of the symbolic features method (Arrotta et al., 2022b) is to
directly incorporate the knowledge encoded in $K$ into $DNN$, not only at the
inference phase, but also during the learning process. Hence, the symbolic
features method allows the $DNN$ also to learn the correlations between input
data and context-consistent activities. As depicted in Figure 3,

Figure 3. The symbolic features neuro-symbolic approach

the information about the context-consistent activities provided by $SR$ is
used to generate symbolic features that are infused within the hidden layers
of $DNN$ through a dedicated layer named knowledge infusion layer. More
formally, given an input tuple $\langle w^{I},C^{w}\rangle$, and the set
$A^{\star}$ of context-consistent activities provided by $SR(K,C^{w})$, the
symbolic features consist of a vector $f_{s}$ in which the $i$-th element is
$1$ if $a_{i}\in A^{\star}$, $0$ otherwise. Given the sequence of $DNN$’s
layers $\ell_{1},\dots,\ell_{m}$, and the symbolic features $f_{s}$ generated
through $SR$, the symbolic features method adds to $DNN$ a knowledge infusion
layer $\ell_{ki}$. This layer receives as input the symbolic features $f_{s}$
and the features automatically extracted by a $DNN$’s hidden layer $\ell_{j}$
with $1<j<m$. Then, $\ell_{ki}$ concatenates in the latent space the features
received as input and generates a novel feature vector that is provided to the
next layer $\ell_{j+1}$. Also in this case, the $DNN$ is trained through the
cross-entropy loss function $\mathcal{L}_{cross}$.

This methodology is less rigid than context refinement since domain knowledge
is infused into the data-driven model instead of being used to strictly
discard those activities that are context-inconsistent according to $K$. On
the other hand, similarly to context refinement, the main problem of symbolic
features is that it is challenging to deploy it on mobile devices since
symbolic features must also be inferred during classification.

##  4\. Knowledge Infusion through Semantic Loss

The existing methods introduced in Section 3.3 require running symbolic
reasoning during classification. As already discussed in the introduction, it
is well-known that such approaches are not suitable for resource-constrained
devices like mobile/wearable devices (Bobed et al., 2015; Bettini et al.,
2020). In this section, we present our novel approach that is named knowledge
infusion through semantic loss (or semantic loss for short). Our method
generates an activity classifier encoding knowledge-based constraints without
requiring symbolic reasoning during the inference phase. Hence, a model based
on semantic loss can be trained offline on a cloud-based server and then
deployed on the users’ mobile/wearable devices to locally perform real-time
activity recognition efficiently.

###  4.1. Methodology

In the following, we describe the mechanisms of our semantic loss approach. As
depicted in Figure 4,

Figure 4. Our neuro-symbolic approach based on semantic loss

the goal of semantic loss is to exploit the knowledge $K$ to guide the
learning process of $DNN$ through a specifically designed loss function. As in
the symbolic features method, $DNN$ still learns the correlations between
context-consistent activities and input data. At the same time, since no
additional features are infused into $DNN$, the use of $K$ and $SR$ during
classification is not required, thus solving one of the main limits of the
existing solutions.

Specifically, the loss function
$\mathcal{L}=\mathcal{L}_{cross}+\alpha\mathcal{L}_{semantic}$ that guides the
training process of $DNN$ is a combination of the cross-entropy loss function
$\mathcal{L}_{cross}$ with a semantic loss function $\mathcal{L}_{semantic}$.
Consistently with other works in the DL literature (Chen et al., 2019; Wu et
al., 2019), $\alpha$ is a trade-off parameter in charge of balancing the
different loss terms. In particular, $\mathcal{L}_{semantic}$ determines how
much the $DNN$’s output satisfies the constraints about the HAR domain encoded
into $K$. More formally, given a probability distribution $P=\langle
p_{1},\dots,p_{k}\rangle$ emitted by $DNN$ on a tuple $\langle
w^{I},C^{w}\rangle$, and the set $A^{\star}$ of context-consistent activities
provided by $SR(K,C^{w})$, we denote with $\hat{p}\in P$ the maximum
probability value of $P$, and $\hat{a}\in A$ its corresponding activity. In
the following, we describe five alternative semantic loss functions we
designed and tested for this work.

  1. (1)

The AllConsistentActs (All) semantic loss focuses on the whole probability
distribution $P$ emitted by $DNN$. Intuitively, given $P$, this semantic loss
has the objective of training the network to maximize the sum of the
probability values in $P$ that correspond to the context-consistent activities
in $A^{\star}$. Hence, we would expect that $DNN$ learns to emit non-zero
probabilities only for context-consistent activities during classification.
Equation 1 formally defines the All semantic loss:

(1) $\mathcal{L}_{semanticAll}(P,A^{\star})=1-\sum_{i}p_{i}\forall i\mid
a_{i}\in A^{\star}$

A potential drawback of this strategy is that, since it aggregates probability
values with a sum, different combinations of these values may lead to the same
penalty. Hence, the resulting penalties could be poorly informative for $DNN$
to properly learn knowledge constraints. For this reason, the following
alternative semantic losses only focus on the most likely activity $\hat{a}$.

  2. (2)

The MinusProb-Prob (-PP) semantic loss aims at associating low probability
values with context-inconsistent activities and higher probability values with
context-consistent activities. In particular, context-inconsistent predictions
are penalized by their probability value. On the other hand, the penalty of
context-consistent activities is inversely proportional to their probability.
More formally,

(2) $\mathcal{L}_{semantic-
PP}(\hat{p},\hat{a},A^{\star})=\begin{cases}1-\hat{p}&\text{if }\hat{a}\in
A^{\star}\\\ \hat{p}&\text{otherwise}\end{cases}$

However, a potential drawback of this strategy is that penalty values for
consistent activities with relatively low probability values are similar to
penalty values for context-inconsistent activities with relatively high
probability values.

  3. (3)

The goal of the Zero-One (01) semantic loss is to maximize the differences
between penalties of context-consistent and context-inconsistent activities.
Specifically,

(3)
$\mathcal{L}_{semantic01}(\hat{p},\hat{a},A^{\star})=\begin{cases}0&\text{if
}\hat{a}\in A^{\star}\\\ 1&\text{otherwise}\end{cases}$

The following strategies are refined versions of the 01 loss.

  4. (4)

The MinusProb-One (-P1) semantic loss aims at improving the confidence of
$DNN$ on context-consistent predictions. Indeed, while the penalty for
context-inconsistent activities is fixed, the penalty for context-consistent
activities is inversely proportional to the corresponding probability value.
Hence, context-consistent activities with low probabilities values are
penalized as well. More formally,

(4)
$\mathcal{L}_{semantic-P1}(\hat{p},\hat{a},A^{\star})=\begin{cases}1-\hat{p}&\text{if
}\hat{a}\in A^{\star}\\\ 1&\text{otherwise}\end{cases}$

  5. (5)

Finally, the idea of the Zero-Prob (0P) semantic loss is that context-
consistent activities should not be penalized, while context-inconsistent
activities should be penalized directly proportionally to their associated
probability values. Hence, $DNN$ should better learn that the higher the
probability values of context-inconsistent activities, the higher the penalty.
Therefore, 0P aims at reducing the probability values on context-inconsistent
activities. More formally,

(5)
$\mathcal{L}_{semantic0P}(\hat{p},\hat{a},A^{\star})=\begin{cases}0&\text{if
}\hat{a}\in A^{\star}\\\ \hat{p}&\text{otherwise}\end{cases}$

##  5\. Experimental evaluation

In this section, we describe the experimental evaluation that we carried out
to assess the quality of our method based on semantic loss presented in
Section 4, compared to the state-of-the-art NeSy approaches introduced in
Section 3.3. First, we introduce the two datasets that we considered in this
work. Then we describe our experimental setup: how we pre-processed the
datasets, the models used and the evaluation methodology adopted. Finally, we
present the results of our evaluation.

###  5.1. Datasets

The evaluation of context-aware HAR approaches requires datasets including
both inertial sensor data and contextual information. However, there are a few
publicly available datasets with such characteristics. Existing NeSy
approaches for context-aware HAR have been evaluated only on scripted and non-
public datasets (Bettini et al., 2020). In this work, we consider a scripted
dataset that we collected in a parallel work and a publicly available in-the-
wild dataset, both including sensor and context data.

####  5.1.1. DOMINO

DOMINO (Arrotta et al., 2023) is a HAR dataset we collected as parallel
research in our research lab. DOMINO includes several context-dependent
activities monitored through mobile devices that collected both inertial
sensor data and high-level context data.

In particular, DOMINO includes data from $25$ subjects wearing a smartwatch on
their dominant hand’s wrist and a smartphone in their pocket. Raw sensor data
have been collected from the inertial sensors (accelerometer, gyroscope, and
magnetometer) installed on both these mobile devices. At the same time, the
dataset also includes high-level context data collected by combining public
web services and the smartphone’s built-in sensors. The measurements of the
barometer and the GPS of the smartphone were discretized to provide
information about the users’ height and speed variations. Moreover, the
dataset incorporates the output of the following web services: (1) Google’s
Places API provided the semantic places closest to the user; from this
information, it was also derived the presence of the user in an indoor or an
outdoor environment; (2) OpenWeatherMap provided current local weather
conditions (e.g., sunny), while (3) Transitland provided transportation routes
and stops close to the user; the combination of this information with location
data was used to derive whether the user was following a public transportation
route.

DOMINO was acquired in a scripted fashion: the volunteers were asked to
perform a sequence of indoor/outdoor activities, but they were not told how to
execute them. Also, the volunteers were monitored by the research staff during
data acquisition. As a consequence, the variability of context situations is
limited. Overall, DOMINO contains almost $9$ hours of labeled data ($\approx
350$ activities instances), including $14$ different types of activities:
walking, running, standing, lying, sitting, stairs up, stairs down, elevator
up, elevator down, cycling, moving by car, sitting on transport, standing on
transport and brushing teeth.

####  5.1.2. ExtraSensory

ExtraSensory (Vaizman et al., 2017) is a public dataset for context and
activity recognition. It includes inertial and context data collected in the
wild from mobile devices of up to $60$ users. Inertial data were collected
through each user’s personal smartphone (including both iOS and Android
devices) and from a smartwatch provided by the researchers. More specifically,
the dataset includes raw data measured by the accelerometer, the gyroscope,
and the magnetometer of the smartphone, and raw data collected by the
accelerometer of the smartwatch. Besides providing raw sensor data,
ExtraSensory also provides data as handcrafted feature vectors ($138$
features) extracted from the raw measurements collected through inertial and
other smartphone sensors (e.g., microphone, luminosity sensor) in 20-second
time windows.

Overall, ExtraSensory contains about $300k$ minutes of labeled data, including
$51$ different labels self-reported by the users and encoding both high-level
context information (e.g., at home, with friends, phone in bag, phone is
charging) and performed activities (e.g., sitting, bicycling).

Since it has been collected in the wild, different research groups in the HAR
community used ExtraSensory to assess the generalization capabilities of
activity recognition frameworks in real-world scenarios (Cruciani et al.,
2020; Tarafdar and Bose, 2021). Due to the complexity of the dataset, existing
HAR methods evaluated on ExtraSensory achieved low recognition rates. For
instance, by considering as input the raw inertial measurements provided by
the accelerometer and the gyroscope of the smartphones, the CNN-based method
proposed in (Cruciani et al., 2020) reached an average macro f1 score of
$\approx 0.53$, only considering $4$ target activity classes: idle (lying or
sitting), walking, running, and cycling. In another work, by considering the
handcrafted features of ExtraSensory, an AdaBoost classifier reaches $\approx
0.63$ of average macro f1 score on $5$ target activities (i.e., walking,
standing, sitting, exercise, and sleeping) (Tarafdar and Bose, 2021). Hence,
this dataset represents a challenging benchmark.

###  5.2. Experimental Setup

In the following, we describe our experimental setup.

####  5.2.1. Data pre-processing

Consistently with existing works proposing NeSy approaches for Context-Aware
HAR (Bettini et al., 2020), for both datasets, we segmented sensor data into
non-overlapping windows of $k=4$ seconds. In the following, we describe the
specific pre-processing steps we adopted for each dataset.

##### DOMINO

Considering DOMINO, we planned to recognize all the $14$ different available
activities, by considering the raw inertial measurements collected by the
accelerometer and the gyroscope of the smartphone and the smartwatch.
Moreover, in our experiments, we considered $6$ different context information
types: the presence of the user in indoor/outdoor locations, her semantic
place (e.g., home, office, gym, bar), her discretized speed (i.e., null, low,
medium, high), her proximity to public transportation routes, her discretized
height variation (i.e., negative, null, positive), and the weather conditions
(e.g., sunny, rainy). Table 1 shows the number of samples involved during our
experiments for each activity class of DOMINO.

Table 1. Number of samples for each activity class in DOMINO Activity | Number of samples
---|---
Brushing teeth | 163
Cycling | 323
Elevator down | 171
Elevator up | 110
Lying | 387
Moving by car | 188
Running | 334
Sitting | 1764
Sitting on transport | 213
Stairs down | 266
Stairs up | 187
Standing | 1875
Standing on transport | 297
Walking | 1378
Total | 7656

##### ExtraSensory

Considering ExtraSensory, we planned to recognize $7$ different activities:
bicycling, lying down, moving by car, on transport, sitting, standing, and
walking. Specifically, for the activity class walking we consider those
samples labeled as walking and/or strolling in the original dataset. For
moving by car, we consider samples labeled with in a car, car driver, and/or
car passenger, even when coupled with the label sitting. Finally, we labeled
as on transport those samples originally labeled with sitting or standing
coupled with the label on a bus.

Before conducting our experiments, we performed some steps of data cleaning.
First of all, we considered only those samples including inertial measurements
recorded from the accelerometer and the gyroscope of the smartphone and from
the accelerometer of the smartwatch. Indeed, for some users of ExtraSensory,
gyroscope data from smartphones are not available. Moreover, not all of the
dataset’s users wore the smartwatch during data collection. Then, based on the
available self-reported labels, we discarded the data collected while the
smartphone’s user was in a bag, or on a table. Indeed, we considered only
phone positions that have been commonly considered in the literature (i.e., in
the pocket and in hand). Finally, since the labels of ExtraSensory were self-
reported by the users involved in the data collection, we discarded samples
that we considered unreliable, due to the fact that they included self-
reported labels not consistent with the recorded data. For instance, we
discard segmentation windows including positive speed values but labeled with
static physical activities like lying. As another example, we discarded those
samples simultaneously labeled with in a car and at home. Table 2 shows the
number of samples for each activity class of ExtraSensory after data cleaning.
Note that, after our data cleaning process, we considered data overall from
$31$ subjects.

Table 2. Number of samples for each activity class in ExtraSensory Activity | Number of samples
---|---
Bicycling | 2920
Lying down | 3055
Moving by car | 2150
On transport | 610
Sitting | 23905
Standing | 14280
Walking | 11230
Total | 58150

As inertial sensor data, we considered the raw data measured from the
accelerometer and the gyroscope of the smartphone and from the accelerometer
of the smartwatch.

Regarding context data, we considered the ones that can be easily derived from
sensors of mobile/wearable devices. For instance, we considered as input
context data the information about the user’s semantic place (e.g., at the
beach) since it could be derived by combining localization data and external
web services, but not the position of the user’s smartphone (i.e., in the
pocket, in hand). In some cases, we discretized available information: for
instance, the speed values observed thanks to the GPS were discretized into
null/low/medium/high speed. Other high-level context information was obtained
by directly considering available data, like audio level, light level, screen
brightness, battery plugged AC/USB, battery charging, on the phone, ringer
mode normal/silent/vibrate, and the time of the day (e.g., Time 0-6, Time
18-24). Moreover, we relied on the self-reported label on a bus, assuming that
similar information could be derived by combining GPS data and web services
like Transitland, as we did in DOMINO. Finally, we considered the semantic
locations self-reported by the subjects (i.e., home, workplace, school, gym,
restaurant, shopping, bar, beach). As already mentioned, semantic location
information can be derived, for instance, by combining location coordinates
data with Google’s Places API.

####  5.2.2. $DNN$’s architecture

The $DNN$ we used for our experiments receives as input three separate inputs
for each segmentation window: a) the smartphone’s inertial sensors data, b)
the smartwatch’s inertial sensors data, and c) the one-hot encoded high-level
context data111Note that, we did not include raw context data as input since
it is intuitively easier to learn correlations between activities and high-
level context (e.g., semantic place, discretized speed) rather than between
activities and raw context (e.g., geographical coordinates, raw speed
measurements)..

Similarly to existing works, we rely on convolutional neural networks to
capture spatiotemporal dependencies of sensor data (Zhao et al., 2017; Ronao
and Cho, 2016; Ha et al., 2015; Yang et al., 2015). Even though more
sophisticated networks have been proposed in the literature, in this work we
use a simple solution to focus on the contribution of knowledge. The exact
structure of our own CNN model has been determined empirically. Specifically,
inertial sensors’ data from the smartphone are processed by three
convolutional layers composed of $32$, $64$, and $96$ filters with a kernel
size equal to $24$, $16$, and $8$, respectively. These layers are separated by
max pooling layers with a pool size of $4$. After the three convolutional
layers, we add a global max pooling layer, followed by a fully connected layer
that includes $128$ neurons. The smartwatch inertial sensors’ data are
provided to another component of $DNN$ that presents the same sequence of
layers used to automatically extract features from the smartphone’s inertial
data. The only difference is that, in this case, the three convolutional
layers present a kernel size of $16$, $8$, and $4$, respectively. Finally, the
high-level context data is provided to a single fully connected layer composed
of $8$ neurons. The features extracted by these three independent flows are
then combined thanks to a concatenation layer, which is followed by a dropout
layer with a dropout rate of $0.1$, and a fully connected layer with $256$
neurons, useful to extract meaningful correlations between the concatenated
features. The last layer of the network is a softmax layer that is in charge
of providing a probability distribution over the possible activities.

In our experiments, we use this DNN architecture in four different ways:

  * •

As a purely data-driven baseline, without further modifications

  * •

Enhanced with our semantic loss (see Section 4)

  * •

Enhanced by combining in the concatenation layer the symbolic features and the
features automatically extracted from input data (see Section 3.3.2)

  * •

As the DNN module of the context refinement approach (see Section 3.3.1)

####  5.2.3. Knowledge model and Symbolic reasoning

For this work, we extended the knowledge model $K$ proposed in the paper where
the NeSy context refinement method was introduced (Bettini et al., 2020).
Specifically, this knowledge model is an ontology encoding domain-based
relationships between activities and contexts according to common-sense
knowledge. For instance, brushing teeth is defined as an activity that can
only occur in indoor environments (e.g., at home) without any height
variations. On the other hand, on transport is defined as an activity that can
only take place while the user is following a public transportation route. As
Symbolic Reasoning function $SR()$, we use the consistency checking task of
the ontology. In particular, for each activity, we evaluate if it is
consistent considering the available context data. For instance, according to
this ontology, brushing teeth is not consistent with a context in which the
user is in an outdoor location.

####  5.2.4. Cross-Validation

We evaluated the approaches presented in Sections 3.3 and 4 by adopting the
leave-k-users-out cross-validation technique. At each fold, $k$ users are used
to populate the test set, while the remaining users are used to populate
training ($90\%)$ and validation ($10\%)$ sets. We also simulated several data
scarcity scenarios by downsampling the available training data at each fold
(e.g., $1\%$, $50\%$).

Considering the DOMINO dataset, we considered $k=1$ (leave-one-user-out cross-
validation). On the other hand, as also done by other works in the literature
(Cruciani et al., 2020), for the ExtraSensory dataset we choose $k=5$. At each
iteration, we used the test set to evaluate the recognition rate of the
different approaches in terms of the F1 score.

For the sake of robustness, we run each experiment $5$ times, computing the
average f1 score and the $95\%$ confidence interval. Overall, the training
process was based on a maximum of $200$ epochs and a batch size of $32$
samples. We considered an early stopping strategy, stopping the learning
process when the loss computed on the validation set did not improve for $5$
consecutive epochs.

###  5.3. Results

In the following, we show how our semantic loss approach outperforms a purely
data-driven classifier in terms of recognition rate both in scripted and in-
the-wild scenarios. We also compare our method with the Neuro-Symbolic AI
(NeSy) approaches presented in Section 3.3. Although our method does not
include the (computationally expensive) symbolic reasoning during
classification, it often reaches recognition rates that are close (and
sometimes better) to the ones of the other approaches, especially considering
the more realistic scenarios of ExtraSensory.

####  5.3.1. Semantic loss types comparison

Table 3 compares the recognition rates (in terms of overall f1 score) of the
five semantic loss functions presented in Section 4 on DOMINO and
ExtraSensory.

Table 3. Comparison between the Semantic Loss types on the different datasets |  | Dataset
---
(training set percentage)
|  | DOMINO
---
($100\%$)
| ExtraSensory
---
($10\%$)
Baseline | 0.9024 | 0.5199
MinusProb-Prob (-PP) |  | 0.9139
---
$\alpha=5$
| 0.5402
---
$\alpha=4$
Zero-One (01) |  | 0.9042
---
$\alpha=1$
| 0.5270
---
$\alpha=7$
Zero-Prob (0P) |  | 0.9162
---
$\alpha=3$
| 0.5288
---
$\alpha=9$
AllConsistentActs (ALL) |  | 0.9094
---
$\alpha=1$
| 0.5872
---
$\alpha=30$
MinusProb-One (-P1) |  | 0.9261
---
$\alpha=7$
| 0.5298
---
$\alpha=5$

To better emphasize the differences in the recognition rates, on ExtraSensory
we decided to show the results obtained by considering a data scarcity
scenario in which only $10\%$ of the training data are available. Indeed, the
number of training samples in DOMINO is nearly equal to the number contained
in only $10\%$ of the training samples in ExtraSensory. Moreover, Table 3 also
includes the best $\alpha$ value for each semantic loss type222$\alpha$ values
have been determined empirically by performing a grid search in the range
$[1,30]$ and the results obtained by the purely data-driven baseline that is
based on a standard classification loss.

Each semantic loss strategy leads to an improvement in the recognition rates
compared to the baseline, with -P1 achieving the best improvements on DOMINO
($\approx+2.5\%$) and All on ExtraSensory ($\approx+6.5\%$). Before running
the experiments, we expected similar results for 01, -P1, and 0P since all
these strategies aim at maximizing the distance in penalties between
consistent and not-consistent activities. While this insight is confirmed on
ExtraSensory, on DOMINO the 01 approach proved to be not very effective in
improving the recognition rate. On this dataset, we observed that, besides
increasing the difference between the penalties applied to context-consistent
and context-inconsistent predictions, it is also crucial to consider the
probability values emitted by $DNN$, especially in the case of a context-
consistent prediction, as proved by the -P1 semantic loss. Finally, the
improvement of the All strategy on DOMINO is limited, probably because
learning knowledge constraints considering the whole probability distribution
is unnecessarily too hard on simple scripted scenarios. On the other hand,
this strategy significantly outperforms the others in the more realistic
settings included in ExtraSensory.

####  5.3.2. Comparison with other approaches

Table 4. DOMINO: Results in terms of macro f1 score and $95\%$ confidence interval | Training set percentage
---|---
| 10% | 20% | 30% | 40% | 50% | 60% | 70% | 80% | 90% | 100%
Baseline |  | 0.5946
---
($\pm 0.008$)
| 0.7529
---
($\pm 0.010$)
| 0.8268
---
($\pm 0.006$)
| 0.8556
---
($\pm 0.011$)
| 0.8835
---
($\pm 0.011$)
| 0.8917
---
($\pm 0.010$)
| 0.8915
---
($\pm 0.006$)
| 0.9007
---
($\pm 0.007$)
| 0.8965
---
($\pm 0.002$)
0.9024
| Semantic
---
loss -P1
| 0.6144
---
($\pm 0.024$)
$\alpha=7$
| 0.7712
---
($\pm 0.004$)
$\alpha=8$
| 0.8469
---
($\pm 0.002$)
$\alpha=9$
| 0.8679
---
($\pm 0.010$)
$\alpha=7$
| 0.8892
---
($\pm 0.006$)
$\alpha=7$
| 0.8889
---
($\pm 0.007$)
$\alpha=7$
| 0.9049
---
($\pm 0.006$)
$\alpha=8$
| 0.8997
---
($\pm 0.003$)
$\alpha=7$
| 0.9021
---
($\pm 0.008$)
$\alpha=6$
| 0.9261
---
$\alpha=7$
| Symbolic
---
features
| 0.7268
---
($\pm 0.008$)
| 0.8590
---
($\pm 0.012$)
| 0.9107
---
($\pm 0.011$)
| 0.9152
---
($\pm 0.009$)
| 0.9198
---
($\pm 0.011$)
| 0.9237
---
($\pm 0.009$)
| 0.9265
---
($\pm 0.004$)
| 0.9254
---
($\pm 0.008$)
| 0.9277
---
($\pm 0.007$)
0.9408
| Context
---
refinement
| 0.8192
---
($\pm 0.009$)
| 0.8811
---
($\pm 0.007$)
| 0.9078
---
($\pm 0.009$)
| 0.9178
---
($\pm 0.005$)
| 0.9281
---
($\pm 0.012$)
| 0.9305
---
($\pm 0.006$)
| 0.9225
---
($\pm 0.004$)
| 0.9274
---
($\pm 0.005$)
| 0.9232
---
($\pm 0.002$)
0.9221
Table 5. ExtraSensory: Results in terms of macro f1 score and $95\%$ confidence interval | Training set percentage
---|---
| 1% | 2.5% | 5% | 7.5% | 10% | 25% | 50% | 75% | 100%
Baseline |  | 0.3127
---
($\pm 0.023$)
| 0.4279
---
($\pm 0.008$)
| 0.4867
---
($\pm 0.013$)
| 0.5167
---
($\pm 0.016$)
| 0.5199
---
($\pm 0.011$)
| 0.5842
---
($\pm 0.016$)
| 0.6096
---
($\pm 0.007$)
| 0.5813
---
($\pm 0.032$)
0.6053
| Semantic
---
loss All
| 0.3366
---
($\pm 0.027$)
$\alpha=29$
| 0.4895
---
($\pm 0.010$)
$\alpha=30$
| 0.5256
---
($\pm 0.016$)
$\alpha=26$
| 0.5650
---
($\pm 0.016$)
$\alpha=26$
| 0.5872
---
($\pm 0.014$)
$\alpha=30$
| 0.6331
---
($\pm 0.013$)
$\alpha=29$
| 0.6323
---
($\pm 0.011$)
$\alpha=18$
| 0.6131
---
($\pm 0.011$)
$\alpha=16$
| 0.6244
---
$\alpha=17$
| Symbolic
---
features
| 0.3418
---
($\pm 0.010$)
| 0.4720
---
($\pm 0.016$)
| 0.5877
---
($\pm 0.025$)
| 0.6359
---
($\pm 0.008$)
| 0.6534
---
($\pm 0.012$)
| 0.6404
---
($\pm 0.010$)
| 0.6216
---
($\pm 0.007$)
| 0.6268
---
($\pm 0.007$)
0.6205
| Context
---
refinement
| 0.6324
---
($\pm 0.014$)
| 0.6540
---
($\pm 0.003$)
| 0.6797
---
($\pm 0.003$)
| 0.6656
---
($\pm 0.004$)
| 0.6622
---
($\pm 0.007$)
| 0.6483
---
($\pm 0.010$)
| 0.6258
---
($\pm 0.007$)
| 0.6067
---
($\pm 0.023$)
0.6190

Tables 4 and 5 compare our best semantic loss method (i.e., -P1 on DOMINO and
All on ExtraSensory) with: i) the purely data-driven baseline, ii) the
symbolic features strategy, and iii) the context refinement strategy. More
specifically, we considered different percentages of available training data
for each dataset, thus comparing the approaches in different data scarcity
scenarios. Note that, during the experimental evaluation, we empirically
determined the optimal $\alpha$ values of the semantic loss for each training
set percentage.

Overall, on each dataset, the NeSy approaches outperform the baseline,
considering almost all the data scarcity scenarios. This result suggests that
traditional symbolic AI approaches have the potential of enhancing the
predicting capabilities of purely data-driven deep learning models.

Focusing on the scripted scenarios of DOMINO (Table 4), the improvement of the
semantic loss is lower than the other approaches, especially considering data
scarcity scenarios. For instance, considering $10\%$ of training data,
semantic loss leads to a recognition rate boost over the baseline of
$\approx+2\%$ on DOMINO. On the other hand, symbolic features and context
refinement lead to improvements of $\approx+13\%$ and $\approx+22\%$,
respectively. These performance differences become progressively smaller while
increasing training data availability. Indeed, when all the available training
data are considered, both semantic loss and context refinement outperform the
baseline by $\approx+2\%$, while symbolic features leads to an improvement of
$\approx+4\%$.

On the other hand, different insights are observed when focusing on the
realistic scenarios of ExtraSensory (Table 5). Indeed, on this dataset, the
differences between the three NeSy approaches are smaller. For instance,
considering $10\%$ of training data, the recognition rate improvements of
semantic loss, symbolic features, and context refinement are $\approx+7\%$,
$\approx+13\%$, and $\approx+14\%$, respectively.

In general, the semantic loss achieves improvements that lie between
$\approx+2\%$ and $\approx+7\%$, sometimes outperforming the recognition rates
of the other NeSy techniques. Indeed, the semantic loss outperforms context
refinement from $50\%$ to $100\%$ of training data, and it also outperforms
symbolic features on $100\%$ of training data. Overall, context refinement is
more effective than methods based on knowledge infusion (i.e., symbolic
features and semantic loss) when the availability of labeled data is
drastically low. However, when slightly more training data are available
(e.g., $25\%$ on ExtraSensory), all the NeSy approaches lead to similar
improvements.

Our results indicate that our semantic loss is effective in capturing
relationships between high-level context data and activities with respect to
learning them directly from the training set by using purely data-driven
models. This is especially true on the ExtraSensory dataset, where the
improvement of semantic loss compared to the baseline is larger. Indeed,
DOMINO covers a significantly lower variability of context situations compared
to ExtraSensory, and the relationships between context and activities can be
captured more easily by the DNN. On the other hand, the in-the-wild nature of
ExtraSensory implies a significantly more complex learning task that can be
partially lightened by knowledge reasoning.

Note that, due to the complexity of the dataset, we achieved relatively low
recognition rates on ExtraSensory (e.g., the max F1 score is $\approx 0.68$).
As described in Section 5.1, our results are in line with other works on the
same dataset (Cruciani et al., 2020; Tarafdar and Bose, 2021).

Since the computational complexity of symbolic reasoning is not adequate for
real-world deployment on resource-constrained devices like smartphones and
smartwatches, the choice of the optimal solution should consider a trade-off
between recognition rate and efficiency. We believe that our semantic loss
method is a way more promising approach since it still improves the baseline
while not requiring symbolic reasoning at all after training.

####  5.3.3. Activity-level results

Figure 5. Comparison between the confusion matrices of the baseline and the
three considered Neuro-Symbolic AI approaches trained with $10\%$ of training
data on the ExtraSensory dataset

Figure 5 compares the confusion matrices obtained by the three considered NeSy
approaches and the baseline on ExtraSensory, considering the data scarcity
scenario where only $10\%$ of training data are available333We show a
representative run among the $5$ repetitions of the experiment.. From these
confusion matrices, it emerges the contribution of domain knowledge in
improving the recognition of different activities. For instance, the baseline
often confuses on transport with moving by car due to their similar patterns
(in terms of inertial measurements and speed), even though context information
(e.g., whether the user is following a public transportation route) should
help in distinguishing them.

Indeed, even though high-level context data are provided as input to the
baseline, it is not feasible to learn from the training set all the possible
correlations between all the possible context conditions and the performed
activities. Hence, enhancing the data-driven model with symbolic AI approaches
based on domain knowledge has a key role in enhancing the capabilities of the
deep learning model and mitigating this problem, thus significantly reducing
the confusion between these two activities.

##  6\. Strengths and Weaknesses of Neuro-Symbolic Approaches

This section discusses the strengths and weaknesses of the Neuro-Symbolic AI
(NeSy) approaches presented in Sections 3.3 and 4. This information is also
summarized in Table 6.

Table 6. Comparison of pros and cons of NeSy methods |  | context
---
refinement
| symbolic
---
features
| semantic
---
loss
| improving recognition rate
---
x | x | x
| mitigating data scarcity
---
x | x | x
| retraining not required when knowledge changes
---
x |  |
| handling data uncertainty
---
| x | x
| improving DNN’s interpretability
---
| x | x
| knowledge not required after deployment
---
|  | x

Compared to other methods, context refinement often reaches the highest
recognition rates, especially when the amount of available training data is
limited. Moreover, another advantage of this approach is that it does not
require retraining the $DNN$ when the knowledge is updated or changed since
symbolic reasoning is applied only during classification. However, this method
may be less effective when based on an imperfect knowledge model. Indeed,
context refinement always discards activities only relying on the user’s
surrounding context considering rigid constraints. For instance, a user could
ride a bicycle even in unusual context scenarios (e.g., on a pedestrian-only
road). Hence, when the knowledge model does not cover all the possible
contexts in which an activity can be performed, combining the information from
inertial data with knowledge would be more convenient in refining the
probability distribution. While the symbolic features method is sometimes
slightly less accurate than context refinement, it better captures the
intrinsic uncertainty in sensor data, by learning correlations between
features and contexts instead of directly applying rigid rules.

However, both approaches require the use of the symbolic reasoning module also
after the training phase, complicating their deployment on mobile devices.
Indeed, as demonstrated in the literature (Bettini et al., 2020; Bobed et al.,
2015), on mobile devices, ontological reasoning could require significantly
more than one second of processing time for each window. Since classification
is periodically performed every few seconds (e.g., we consider a time window
of $4$ seconds), we believe that this computational effort may not be
reasonable for real-world deployments. The main advantage of semantic loss is
that it could be trained offline on a server with high computational
capabilities and then deployed and used on a mobile device without the need
for computationally expensive symbolic reasoning tasks. Indeed, semantic loss
is still able to significantly improve the recognition rate.

Finally, the integration of the domain knowledge into the deep learning model,
as performed by our semantic loss, has also the potential of making $DNN$ more
transparent. Indeed, the decision process of our approach also relies on
domain knowledge about HAR that can be used to partially understand from a
high-level perspective the overall behavior of the classifier. Clearly, since
the knowledge $K$ only considers high-level context data, we are only able to
interpret the role of such information during classification. Indeed, inertial
sensors’ data are not associated with semantics, and explaining their high-
level role in the classifier decision process is still an open research
problem. However, such interpretability advantages are shared only between the
semantic loss and the symbolic features approaches. Indeed, considering
context refinement, it is only possible to understand why certain activities
are considered consistent or inconsistent. However, the global decision
process made by $DNN$ remains opaque since the common-sense knowledge about
the specific domain is not directly incorporated into the classifier.

##  7\. Conclusion and future work

In this work, we presented a novel Neuro-Symbolic AI approach for context-
aware HAR based on a combination of a classical loss function with a _semantic
loss_. Our method infuses domain knowledge inside a deep learning classifier,
improving its recognition rate. Compared to existing neuro-symbolic
approaches, our method avoids symbolic reasoning during classification, thus
making the model deployment feasible even on devices with limited
computational resources. The advantage of our approach is particularly evident
in realistic in-the-wild settings.

We have several plans for future work. First, context-Aware HAR requires
continuously obtaining context data. However, this may be computationally
intensive since it may involve costly operations on mobile/wearable devices
(e.g., continuously calling web services). Since high-level contexts may not
change so rapidly, we will design strategies to obtain new information
periodically (e.g., with a low periodicity, when GPS data exhibits significant
changes, etcetera). Thanks to these strategies, it could also be possible to
run our method when mobile devices are not connected to the internet for short
periods.

Also, our experiments considered a rigid symbolic formalism based on
ontologies. However, probabilistic logic frameworks (e.g., Markov Logic
Networks) can further improve the results by providing a more flexible
knowledge model that considers uncertainty.

Moreover, we will evaluate how considering knowledge models that encode
different levels of detail affect the performance of the NeSy approaches we
compared in this work. Indeed, we expect to observe different results when
considering a knowledge model that defines only usual contexts in which
activities can take place, compared to a knowledge model that instead
considers both usual and unusual scenarios.

We also want to explore other strategies to infuse knowledge inside deep
learning models. For instance, the symbolic reasoning function may be
approximated by a dedicated deep learning model (e.g., through a Graph Neural
Network that learns domain constraints from a knowledge graph). This model
could significantly reduce symbolic reasoning time, hence making the context
refinement and symbolic features methods more practical in real-world
deployments.

Another interesting line of research is to explore whether our approach can be
adopted in pervasive computing domains different from HAR, where context
information may have a major role (e.g., sensor-based healthcare systems,
emotion recognition, anomaly detection).

Finally, we want to study how to quantitatively evaluate the intrinsic
interpretability of the DNN components of NeSy approaches.

## References

  * (1)
  * Abdallah et al. (2018) Zahraa S Abdallah, Mohamed Medhat Gaber, Bala Srinivasan, and Shonali Krishnaswamy. 2018\.  Activity recognition with evolving data streams: A review.  _ACM Computing Surveys (CSUR)_ 51, 4 (2018), 1–36.
  * Arrotta et al. (2022a) Luca Arrotta, Gabriele Civitarese, and Claudio Bettini. 2022a.  Dexar: deep explainable sensor-based activity recognition in smart-home environments.  _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ 6, 1 (2022), 1–30.
  * Arrotta et al. (2022b) Luca Arrotta, Gabriele Civitarese, and Claudio Bettini. 2022b.  Knowledge Infusion for Context-Aware Sensor-Based Human Activity Recognition. In _2022 IEEE International Conference on Smart Computing (SMARTCOMP)_. IEEE, 1–8.
  * Arrotta et al. (2023) Luca Arrotta, Gabriele Civitarese, Riccardo Presotto, and Claudio Bettini. 2023.  DOMINO: A Dataset for Context-Aware Human Activity Recognition using Mobile Devices.  In Proceedings of the 24th International Conference on Mobile Data Management (MDM), IEEE Computer Society, 2023..
  * Asim et al. (2020) Yusra Asim, Muhammad Awais Azam, Muhammad Ehatisham-ul Haq, Usman Naeem, and Asra Khalid. 2020\.  Context-aware human activity recognition (CAHAR) in-the-Wild using smartphone accelerometer.  _IEEE Sensors Journal_ 20, 8 (2020), 4361–4371.
  * Atzmueller et al. (2018) Martin Atzmueller, Naveed Hayat, Matthias Trojahn, and Dennis Kroll. 2018.  Explicative human activity recognition using adaptive association rule-based classification. In _2018 IEEE International Conference on Future IoT Technologies (Future IoT)_. IEEE, 1–6.
  * Azkune and Almeida (2018) Gorka Azkune and Aitor Almeida. 2018.  A scalable hybrid activity recognition approach for intelligent environments.  _IEEE Access_ 6 (2018), 41745–41759.
  * Bettini et al. (2020) Claudio Bettini, Gabriele Civitarese, and Riccardo Presotto. 2020\.  Caviar: Context-driven active and incremental activity recognition.  _Knowledge-Based Systems_ 196 (2020), 105816.
  * Bobed et al. (2015) Carlos Bobed, Roberto Yus, Fernando Bobillo, and Eduardo Mena. 2015.  Semantic reasoning on mobile devices: Do androids dream of efficient reasoners?  _Journal of Web Semantics_ 35 (2015), 167–183.
  * Cao et al. (2018) Liang Cao, Yufeng Wang, Bo Zhang, Qun Jin, and Athanasios V Vasilakos. 2018.  GCHAR: An efficient Group-based Context—Aware human activity recognition on smartphone.  _J. Parallel and Distrib. Comput._ 118 (2018), 67–80.
  * Chen et al. (2019) Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K Su. 2019\.  This looks like that: deep learning for interpretable image recognition.  _Advances in neural information processing systems_ 32 (2019).
  * Chen et al. (2021) Kaixuan Chen, Dalin Zhang, Lina Yao, Bin Guo, Zhiwen Yu, and Yunhao Liu. 2021\.  Deep learning for sensor-based human activity recognition: Overview, challenges, and opportunities.  _ACM Computing Surveys (CSUR)_ 54, 4 (2021), 1–40.
  * Cruciani et al. (2020) Federico Cruciani, Anastasios Vafeiadis, Chris Nugent, Ian Cleland, Paul McCullagh, Konstantinos Votis, Dimitrios Giakoumis, Dimitrios Tzovaras, Liming Chen, and Raouf Hamzaoui. 2020.  Feature learning for human activity recognition using convolutional neural networks: A case study for inertial measurement unit and audio data.  _CCF Transactions on Pervasive Computing and Interaction_ 2, 1 (2020), 18–32.
  * Gaur et al. (2021) Manas Gaur, Keyur Faldu, and Amit Sheth. 2021.  Semantics of the black-box: Can knowledge graphs help make deep learning systems more interpretable and explainable?  _IEEE Internet Computing_ 25, 1 (2021), 51–59.
  * Gayathri et al. (2017) KS Gayathri, KS Easwarakumar, and Susan Elias. 2017\.  Probabilistic ontology based activity recognition in smart homes using Markov Logic Network.  _Knowledge-Based Systems_ 121 (2017), 173–184.
  * Guesgen (2020) Hans W Guesgen. 2020\.  Using rough sets to improve activity recognition based on sensor data.  _Sensors_ 20, 6 (2020), 1779.
  * Ha et al. (2015) Sojeong Ha, Jeong-Min Yun, and Seungjin Choi. 2015\.  Multi-modal convolutional neural networks for activity recognition. In _2015 IEEE International conference on systems, man, and cybernetics_. IEEE, 3017–3022.
  * Henricksen et al. (2005) Karen Henricksen, Jadwiga Indulska, Ted McFadden, and Sasitharan Balasubramaniam. 2005.  Middleware for distributed context-aware systems. In _On the Move to Meaningful Internet Systems 2005: CoopIS, DOA, and ODBASE: OTM Confederated International Conferences, CoopIS, DOA, and ODBASE 2005, Agia Napa, Cyprus, October 31-November 4, 2005, Proceedings, Part I_. Springer, 846–863.
  * Hiremath et al. (2022) Shruthi K Hiremath, Yasutaka Nishimura, Sonia Chernova, and Thomas Plötz. 2022.  Bootstrapping Human Activity Recognition Systems for Smart Homes from Scratch.  _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ 6, 3 (2022), 1–27.
  * Hitzler and Sarker (2022) P Hitzler and M Sarker. 2022.  Neuro-Symbolic AI= Neural+ Logical+ Probabilistic AI.  _Neuro-Symbolic Artificial Intelligence: The State of the Art_ 342 (2022), 173\.
  * Jain et al. (2022) Yash Jain, Chi Ian Tang, Chulhong Min, Fahim Kawsar, and Akhil Mathur. 2022.  ColloSSL: Collaborative self-supervised learning for human activity recognition.  _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ 6, 1 (2022), 1–28.
  * Kamruzzaman Sarker et al. (2021) Md Kamruzzaman Sarker, Lu Zhou, Aaron Eberhart, and Pascal Hitzler. 2021.  Neuro-Symbolic Artificial Intelligence: Current Trends.  _arXiv e-prints_ (2021), arXiv–2105.
  * Kwon et al. (2014) Yongjin Kwon, Kyuchang Kang, and Changseok Bae. 2014\.  Unsupervised learning for human activity recognition using smartphone sensors.  _Expert Systems with Applications_ 41, 14 (2014), 6067–6074.
  * Li et al. (2022) Xuhong Li, Haoyi Xiong, Xingjian Li, Xuanyu Wu, Xiao Zhang, Ji Liu, Jiang Bian, and Dejing Dou. 2022\.  Interpretable deep learning: Interpretation, interpretability, trustworthiness, and beyond.  _Knowledge and Information Systems_ 64, 12 (2022), 3197–3234.
  * Riboni and Bettini (2011) Daniele Riboni and Claudio Bettini. 2011.  COSAR: hybrid reasoning for context-aware activity recognition.  _Personal and Ubiquitous Computing_ 15 (2011), 271–289.
  * Ronao and Cho (2016) Charissa Ann Ronao and Sung-Bae Cho. 2016.  Human activity recognition with smartphone sensors using deep learning neural networks.  _Expert systems with applications_ 59 (2016), 235–244.
  * Saguna et al. (2013) Saguna Saguna, Arkady Zaslavsky, and Dipanjan Chakraborty. 2013\.  Complex activity recognition using context-driven activity theory and activity signatures.  _ACM Transactions on Computer-Human Interaction (TOCHI)_ 20, 6 (2013), 1–34.
  * Sanabria et al. (2021) Andrea Rosales Sanabria, Franco Zambonelli, and Juan Ye. 2021\.  Unsupervised domain adaptation in activity recognition: A GAN-based approach.  _IEEE Access_ 9 (2021), 19421–19438.
  * Soleimani and Nazerfard (2021) Elnaz Soleimani and Ehsan Nazerfard. 2021.  Cross-subject transfer learning in human activity recognition systems using generative adversarial networks.  _Neurocomputing_ 426 (2021), 26–34.
  * Sukor et al. (2019) Abdul Syafiq Abdull Sukor, Ammar Zakaria, Norasmadi Abdul Rahim, Latifah Munirah Kamarudin, Rossi Setchi, and Hiromitsu Nishizaki. 2019\.  A hybrid approach of knowledge-driven and data-driven reasoning for activity recognition in smart homes.  _Journal of Intelligent & Fuzzy Systems_ 36, 5 (2019), 4177–4188.
  * Tarafdar and Bose (2021) Pratik Tarafdar and Indranil Bose. 2021.  Recognition of human activities for wellness management using a smartphone and a smartwatch: a boosting approach.  _Decision Support Systems_ 140 (2021), 113426.
  * Vaizman et al. (2017) Yonatan Vaizman, Katherine Ellis, and Gert Lanckriet. 2017\.  Recognizing detailed human context in the wild from smartphones and smartwatches.  _IEEE pervasive computing_ 16, 4 (2017), 62–74.
  * Wang et al. (2019) Jindong Wang, Yiqiang Chen, Shuji Hao, Xiaohui Peng, and Lisha Hu. 2019.  Deep learning for sensor-based activity recognition: A survey.  _Pattern recognition letters_ 119 (2019), 3–11.
  * Wu et al. (2019) Di Wu, Si-Jia Zheng, Chang-An Yuan, and De-Shuang Huang. 2019\.  A deep model with combined losses for person re-identification.  _Cognitive Systems Research_ 54 (2019), 74–82.
  * Yang et al. (2015) Jianbo Yang, Minh Nhut Nguyen, Phyo Phyo San, Xiao Li Li, and Shonali Krishnaswamy. 2015.  Deep convolutional neural networks on multichannel time series for human activity recognition. In _Twenty-fourth international joint conference on artificial intelligence_.
  * Zhao et al. (2017) Bendong Zhao, Huanzhang Lu, Shangfeng Chen, Junliang Liu, and Dongya Wu. 2017.  Convolutional neural networks for time series classification.  _Journal of Systems Engineering and Electronics_ 28, 1 (2017), 162–169.