# Minimal $P$-symmetric period problem of first-order autonomous Hamiltonian
Systems

Chungen Liu and Ben-Xing Zhou
School of Mathematics and LPMC, Nankai University,
Tianjin 300071, P. R. China

11footnotetext:  Partially supported by the NSF of China (11471170, 10621101),
973 Program of MOST (2011CB808002) and SRFDP. Email: liucg@nankai.edu.cn(for
Liu) and 1120130022@mail.nankai.edu.cn(for Zhou)

Abstract: Let $P\in Sp(2n)$ satisfying $P^{k}=I_{2n}$, we consider the minimal
$P$-symmetric period problem of the autonomous nonlinear Hamiltonian system

$\dot{x}(t)=JH^{\prime}(x(t)).$

For some symplectic matrices $P$, we show that for any $\tau>0$ the above
Hamiltonian system possesses a $k\tau$ periodic solution $x$ with $k\tau$
being its minimal $P$-symmetric period provided $H$ satisfies the Rabinowitz’s
conditions on the minimal period conjecture, together with that $H$ is convex
and $H(Px)=H(x)$.

Key Words: Maslov $P$-index, Relative Morse index, Minimal $P$-symmetric
period, Hamiltonian system

##  1 Introduction and main result

In this paper, we study the following first-order autonomous Hamiltonian
system with $P$-boundary condition:

$\left\\{\begin{array}[]{ll}\dot{x}=JH^{\prime}(x),x\in\mathbb{R}^{2n}\\\
x(\tau)=Px(0).\end{array}\right.$ (1.1)

where $\tau>0$, $P\in Sp(2n)$, and $H\in C^{2}(\mathbb{R}^{2n},\mathbb{R})$ is
the Hamiltonian function satisfying $H(Px)=H(x)$, $\forall
x\in\mathbb{R}^{2n}$. $H^{\prime}(x)$ denote its gradient,
$J=\left(\begin{array}[]{cc}0&-I_{n}\\\ I_{n}&0\end{array}\right)$ is the
standard symplectic matrix, $I_{n}$ is the identity matrix on
$\mathbb{R}^{n}$. Without confusion, we shall omit the subindex of the
identity matrix.

A solution $(\tau,x)$ of the problem (1.1) is called a $P$-solution of the
Hamiltonian systems. The problem (1.1) has relation with the closed geodesics
on Riemannian manifold (cf.[17]) and symmetric periodic solution or the quasi-
periodic solution problem (cf.[18]). In addition, the first author C. Liu in
[22] transformed some periodic boundary problem for nonlinear delay
differential systems and some nonlinear delay Hamiltonian systems to
$P$-boundary problems of Hamiltonian systems as above, we also refer [6, 14,
16, 19] and references therein for the background of $P$-boundary problems in
$N$-body problems.

Let $P\in Sp(2n)$ and $k\in\mathbb{N}=\\{0,1,2,\cdots\\}$, we say $P$
satisfies $(P)_{k}$ condition, if $P^{k}=I_{2n}$ and for each integer $m$ with
$1\leq m\leq k-1$, $P^{m}\neq I$. If $P$ satisfies $(P)_{k}$ condition, a
$P$-solution $(\tau,x)$ can be extended as a $k\tau$-periodic solution
$(k\tau,x^{k})$. We say that a $T$-periodic solution $(T,x)$ of the
Hamiltonian system in (1.1) is $P$-symmetric if $x(\frac{T}{k})=Px(0)$. $T$ is
the $P$-symmetric period of $x$. $T$ is called the minimal $P$-symmetric
period of $x$ if $T=\min\\{\lambda>0\mid x(t+\frac{\lambda}{k})=Px(t),\forall
t\in\mathbb{R}\\}$.

We assume the following conditions on $H$ in our arguments:

  1. (H0)

$H\in C^{1}(\mathbb{R}^{2n},\mathbb{R})$ $\forall x\in\mathbb{R}^{2n}$;

  2. (H1)

$H\in C^{2}(\mathbb{R}^{2n},\mathbb{R})$ with $H(Px)=H(x)$, $\forall
x\in\mathbb{R}^{2n}$;

  3. (H2)

$H(x)\geq 0$, $\forall x\in\mathbb{R}^{2n}$;

  4. (H3)

$H(x)=o(|x|^{2})$ as $|x|\to 0$;

  5. (H4)

There are constants $\mu>2$ and $R_{0}>0$ such that

$0<\mu H(x)\leq(H^{\prime}(x),x),\ \ \forall\ |x|\geq R_{0};$

  6. (H5)

$H^{\prime\prime}(x)>0$, $\forall x\in\mathbb{R}^{2n}$;

In [35], Rabinowitz proved that the Hamiltonian system in (1.1) possesses a
non-constant prescribed period solution provided $H$ satisfying (H0) and
(H2)-(H4). Because a $\tau/k$-periodic function is also a $\tau$-periodic
function, moreover, in [35] Rabinowitz proposed a conjecture: under the
conditions (H0) and (H2)-(H4), for any $\tau>0$, the Hamiltonian system in
(1.1) possesses a $\tau$-periodic with $\tau$ being its minimal period. Since
then, there were many papers on this minimal period problem (cf. [5], [2],
[1], [10], [31], [32], [33], [9], etc.). In 1997, D. Dong and Y. Long [9]
developed a new method on this prescribed minimal period solution problem and
discovered the intrinsic relationship between the minimal period and the
indices of a solution. Based upon the work of [9], G. Fei, Q. Qiu, T. Wang and
others applied this method to various problems of Rabinowitz’s conjecture (cf.
[12], [13], [24], etc.). In fact, under conditions $H(0)=0$ and $H(x)>0$,
$\forall x\in\mathbb{R}^{2n}\setminus\\{0\\}$; $\frac{H(x)}{\mid
x\mid^{2}}\rightarrow+\infty$ as $x\rightarrow 0$ and $\frac{H(x)}{\mid
x\mid^{2}}\rightarrow 0$ as $x\rightarrow+\infty$, F. Clarke and I. Ekeland
proved a result on the corresponding minimal period problem for some given $T$
in [5]; I. Ekeland and H. Hofer gave a criterion for the conjecture in [10]
which is unfortunately not easy to check.

For $P$-boundary problem, S. Tang [28] and the first author of this paper
proved that for any
$0<\tau<\frac{\pi}{\max_{t\in[0,\tau]}\|J\dot{\gamma}_{P}(t)\gamma_{P}(t)^{-1}\|}$,
there exists a nonconstant $P$-solution with its minimal $P$-symmetric period
$k\tau$ or $\frac{k\tau}{k+1}$ via the iteration theory of Maslov $P$-index.
In [23], the first author of this paper improved the result that for every
$\tau>0$, there exists a nonconstant $P$-solution with its minimal
$P$-symmetric period $k\tau$ or $\frac{k\tau}{k+1}$.

For $n\in\mathbb{N}$, $k>0$, denote by

$Sp(2n)\equiv Sp(2n,\mathbb{R})=\\{M\in\mathcal{L}(\mathbb{R}^{2n})\mid
M^{T}JM=J\\},$ $\mathcal{P}_{\tau}(2n)\equiv\\{\gamma\in
C([0,\tau],Sp(2n))\mid\gamma(0)=I\\},$ $Sp(2n)_{k}\equiv\\{P\in Sp(2n)\mid{P\
satisfies\ (P)_{k}\ condition}\\},$ $\Omega(M)\equiv\\{N\in
Sp(2n)\mid{\rm\sigma(N)\cap\mathbf{U}=\sigma(M)\cap\mathbf{U}\ and\
\nu_{\lambda}(N)=\nu_{\lambda}(M),\forall\
\lambda\in\sigma(M)\cap\mathbf{U}}\\}.$

Denote by $\Omega^{0}(M)$ the path connected component of $\Omega(M)$ which
contains $M$.

######  Lemma 1.1.

If $P\in Sp(2n)_{k}$, then there exists a matrix $I_{2p}\diamond
R(\frac{2\pi}{k})^{\diamond j_{1}}\diamond\cdots\diamond
R(\frac{2r\pi}{k})^{\diamond j_{r}}\in\Omega^{0}(P^{-1})$, with
$p+\sum_{m=1}^{r}j_{m}=n$.

###### Proof.

For $P\in Sp(2n)_{k}$, we have
$\sigma(P^{-1})=\sigma(P)\subseteq\\{1,e^{\frac{2\pi\sqrt{-1}}{k}},e^{\frac{4\pi\sqrt{-1}}{k}},\cdots,e^{\frac{2(k-1)\pi\sqrt{-1}}{k}}\\}\subseteq\mathbf{U}$.
By the Theorem 1.8.10 in [29], there exists $M_{1}(\omega_{1})\diamond
M_{2}(\omega_{2})\diamond\cdots\diamond
M_{s}(\omega_{s})\in\Omega^{0}(P^{-1})$ where $M_{i}(\omega_{i})$ is a basic
normal form of some eigenvalue of $P^{-1}$, $1\leq i\leq s$. And the following
are the basic normal forms for eigenvalues in $\mathbf{U}$.

Case 1. $N_{1}(\lambda,b)=\left(\begin{array}[]{cc}\lambda&b\\\
0&\lambda\end{array}\right)$, $\lambda=\pm 1,b=\pm 1,0$.
Since $P\in Sp(2n)_{k}$, we have $b=0$ and
$\lambda\in\\{-1,1\\}\cap\sigma(P^{-1})$.

Case 2. $R(\theta)=\left(\begin{array}[]{cc}\cos\theta&-\sin\theta\\\
\sin\theta&\cos\theta\end{array}\right)$, $\theta\in(0,\pi)\cup(\pi,2\pi)$.
Since $P\in Sp(2n)_{k}$, we have
$\theta\in\\{\frac{2\pi}{k},\frac{4\pi}{k},\cdots\frac{2(k-1)\pi}{k}\\}$.

Case 3. $N_{2}(\omega,b)=\left(\begin{array}[]{cc}R(\theta)&b\\\
0&R(\theta)\end{array}\right)$, $\theta\in(0,\pi)\cup(\pi,2\pi)$,
$b=\left(\begin{array}[]{cc}b_{1}&b_{2}\\\
b_{3}&b_{4}\end{array}\right),b_{i}\in\mathbb{R},b_{2}\neq b_{3}$.

From direct computation, it is easy to check that the matrix
$T=\left(\begin{array}[]{cc}\frac{1}{\sqrt{2}}&\frac{\sqrt{-1}}{\sqrt{2}}\\\
\frac{\sqrt{-1}}{\sqrt{2}}&\frac{1}{\sqrt{2}}\\\ \end{array}\right)$ satisfies
that $TR(\theta)T^{-1}=\left(\begin{array}[]{cc}e^{\sqrt{-1}\theta}&0\\\
0&e^{-\sqrt{-1}\theta}\\\ \end{array}\right)$. Then
$\left(\begin{array}[]{cc}T&0\\\ 0&T\\\
\end{array}\right)N_{2}(\omega,b)\left(\begin{array}[]{cc}T^{-1}&0\\\
0&T^{-1}\\\
\end{array}\right)=\left(\begin{array}[]{cc}TR(\theta)T^{-1}&TbT^{-1}\\\
0&TR(\theta)T^{-1}\\\ \end{array}\right)$, where

$TbT^{-1}=\left(\begin{array}[]{cc}\frac{1}{2}(b_{1}+b_{4})-\frac{\sqrt{-1}}{2}(b_{2}-b_{3})&\frac{1}{2}(b_{2}+b_{3})-\frac{\sqrt{-1}}{2}(b_{1}-b_{4})\\\
\frac{1}{2}(b_{2}+b_{3})+\frac{\sqrt{-1}}{2}(b_{1}-b_{4})&\frac{1}{2}(b_{1}+b_{4})+\frac{\sqrt{-1}}{2}(b_{2}-b_{3})\\\
\end{array}\right).$ (1.2)

Denoted by

$\left(\begin{array}[]{cc}TR(i\theta)T^{-1}&X(i)\\\ 0&TR(k\theta)T^{-1}\\\
\end{array}\right)=\left(\begin{array}[]{cc}TR(\theta)T^{-1}&TbT^{-1}\\\
0&TR(\theta)T^{-1}\\\ \end{array}\right)^{i},\;i\in\mathbb{N}$ (1.3)

where $X(i)=\left(\begin{array}[]{cc}x_{1}(i)&x_{2}(i)\\\ x_{3}(i)&x_{4}(i)\\\
\end{array}\right)$ and
$X(1)=TbT^{-1}=\left(\begin{array}[]{cc}x_{1}(1)&x_{2}(1)\\\
x_{3}(1)&x_{4}(1)\\\ \end{array}\right)$. By direct computation, we have

$\begin{split}x_{1}(k)&=ke^{\sqrt{-1}(k-1)\theta}x_{1}(1),\\\
x_{4}(k)&=ke^{-\sqrt{-1}(k-1)\theta}x_{4}(1).\end{split}$ (1.4)

Thus, from $P^{k}=I$ we have $X(k)=0$, so $x_{1}(1)=x_{4}(1)=0$, i.e.

$\begin{split}\frac{1}{2}(b_{1}+b_{4})-\frac{\sqrt{-1}}{2}(b_{2}-b_{3})=0,\\\
\frac{1}{2}(b_{1}+b_{4})+\frac{\sqrt{-1}}{2}(b_{2}-b_{3})=0.\end{split}$

Then we have $b_{2}=b_{3}$, which is contradict to the definition of the basic
normal form $N_{2}(\omega,b)$.

Therefore, from (Case1)-(Case3),we get $M_{i}(\omega_{i})=R(\theta_{i})$ where
$\theta_{i}\in\\{0,\frac{2\pi}{k},\frac{4\pi}{k},\cdots\frac{2(k-1)\pi}{k}\\}$,
$1\leq i\leq s$. And the lemma is proved. ∎

For the notations in Lemma 1.1, we define

$\begin{split}Sp(2n)_{k}(r,p;j_{1},j_{2},\cdots,j_{r})\equiv\left\\{P\in
Sp(2n)_{k}\mid k-2\sum_{m=1}^{r}m\cdot
j_{m}>1,r<\frac{k}{2}\right\\}.\end{split}$

Now we state the main result of this paper.

######  Theorem 1.1.

Suppose $P\in Sp(2n)_{k}(r,p;j_{1},j_{2},\cdots,j_{r})$, and the Hamiltonian
function $H$ satisfies (H1)-(H5), then for every $\tau>0$, the system (1.1)
possesses a non-constant $P$-solution $(\tau,x)$ such that the minimal
$P$-symmetric period of the extended $k\tau$-periodic solution $(k\tau,x^{k})$
is $k\tau$.

In order to prove the above result, we need to obtain the relationship between
the Maslov $P$-index and Morse index. Thus we organize this paper as follows,
in Section 2, we recall the definition and properties of the Maslov $P$-index
theory, and we also list out the relationship between the Morse index and the
Maslov $P$-index (see [27],[28], [23], [11] and [12]). In Section 3, we first
study the iteration formila of Maslov index of paths
$\xi\in\mathcal{P}_{\tau}(2n)$ such that $\xi(\tau)=P^{-1}$ in detail, then we
will give the complete proof of the main result.

##  2 Preliminaries

In this section, we give a brief introduction to the Maslov $P$-index and its
iteration properties, and then give the relationship between Maslov $P$-index
and the relative Morse index which is studied by the first author of this
paper in [23].

Maslov $P$-index was first studied in [7] and [21] independently for any
symplectic matrix $P$ with different treatment. The first author and S. Tang
in [27, 28] defined the Maslov $(P,\omega)$-index
$(i_{\omega}^{P}(\gamma),\nu_{\omega}^{P}(\gamma))$ for any symplectic path
$\gamma\in\mathcal{P}_{\tau}(2n)$. And then the first author of this paper
used relative index theory to develop Maslov $P$-index in [23] which is
consistent with the definition in [27, 28]. When the symplectic matrix
$P=diag\\{-I_{n-\kappa},I_{\kappa},-I_{n-\kappa},I_{\kappa}\\}$,
$0\leq\kappa\in\mathbb{N}\leq n$, the $(P,\omega)$-index theory and its
iteration theory were studied in [8] and then be successfully used to study
the multiplicity of closed characteristics on partially symmetric convex
compact hypersurfaces in $\mathbb{R}^{2n}$. Here we use the notions and
results in [21, 27, 28].

For $\omega\in\mathbf{U}$, then the Maslov $(P,\omega)$-index of a symplectic
path $\gamma\in\mathcal{P}_{\tau}(2n)$ is defined as a pair of
integers(cf.[27])

$(i_{\omega}^{P}(\gamma),\nu_{\omega}^{P}(\gamma))\in\mathbb{Z}\times\\{0,1,\cdots,2n\\},$

where the index part

$i_{\omega}^{P}(\gamma)=i_{\omega}(P^{-1}\gamma\ast\xi)-i_{\omega}(\xi),$
(2.1)

$\xi\in\mathcal{P}_{\tau}(2n)$ such that $\xi(\tau)=P^{-1}$ and the nullity

$\nu_{\omega}^{P}(\gamma)=\dim\ker(\gamma(\tau)-\omega P).$ (2.2)

Suppose $B(t)\in C(\mathbb{R},\mathcal{L}_{s}(\mathbb{R}^{2n}))$, if
$\gamma\in\mathcal{P}_{\tau}(2n)$ is the fundamental solution of the linear
Hamiltonian systems

$\dot{y}(t)=JB(t)y,\ \ \ y\in\mathbb{R}^{2n},$ (2.3)

we also call $(i_{\omega}^{P}(\gamma),\nu_{\omega}^{P}(\gamma))$ the Maslov
$(P,\omega)$-index of $B(t)$, denoting
$(i_{\omega}^{P}(B),\nu_{\omega}^{P}(B))=(i_{\omega}^{P}(\gamma),\nu_{\omega}^{P}(\gamma))$,
just as in [21, 27, 28]. If $x$ is a $P$-solution of (1.1), then the Maslov
$(P,\omega)$-index of the solution $x$ is defined to be the Maslov
$(P,\omega)$-index of $B(t)=H^{\prime\prime}(x(t))$ and denoted by
$(i_{\omega}^{P}(x),\nu_{\omega}^{P}(x))$. When $\omega=1$, we omit the
subindex, denoted by $(i^{P}(\gamma),\nu^{P}(\gamma))$ or
$(i^{P}(B),\nu^{P}(B))$ for simplicity.

For $m\in\mathbb{N}$, we extend the definition of $x(t)$ which is the solution
of (1.1) to $[0,+\infty)$ by

$x(t)=P^{j}x(t-j\tau),\ \ \forall j\tau\leq t\leq(j+1)\tau,\ j\in\mathbb{N}$

and define the $m$-th iteration $x^{m}$ of $x$ by

$x^{m}=x|_{[0,m\tau]}.$

If $P$ satisfies $(P)_{k}$ condition, then $x^{k}$ becomes an $k\tau$-periodic
solution of the Hamiltonian system in (1.1). We know that the fundamental
solution $\gamma_{x}\in\mathcal{P}_{\tau}(2n)$ carries significant information
about $x$. For any $\gamma\in\mathcal{P}_{\tau}(2n)$, S. Tang and the first
author of this paper have defined the corresponding $m$-th iteration path
$\gamma^{m}:[0,m\tau]\to Sp(2n)$ of $\gamma$ in [27] by

$\gamma^{m}(t)=\begin{cases}\gamma(t),&t\in[0,\tau],\\\
P\gamma(t-\tau)P^{-1}\gamma(\tau),&t\in[\tau,2\tau],\\\
P^{2}\gamma(t-2\tau)(P^{-1}\gamma(\tau))^{2},&t\in[2\tau,3\tau],\\\
P^{3}\gamma(t-3\tau)(P^{-1}\gamma(\tau))^{3},&t\in[3\tau,4\tau],\\\
\cdots\cdots\\\
P^{m-1}\gamma(t-(m-1)\tau)(P^{-1}\gamma(\tau))^{m-1},&t\in[(m-1)\tau,m\tau].\\\
\end{cases}$ (2.4)

If the matrix function $B(t)$ in the linear Hamiltonian system (2.3) satisfies
$P^{T}B(t+\tau)P=B(t)$, the iteration of its fundamental solution $\gamma$ is
defined in the same way.

Corresponding we set

$i_{\omega}^{P^{m}}(\gamma,m)=i_{\omega}^{P^{m}}(\gamma^{m}),\ \
\nu_{\omega}^{P^{m}}(\gamma,m)=\nu_{\omega}^{P^{m}}(\gamma^{m}).$

If the subindex $\omega=1$, we simply write
$(i^{P^{m}}(\gamma,m),\nu^{P^{m}}(\gamma,m))$, and omit the subindex $1$ when
there is no confusion. In the sequel, we use the notions
$(i(\gamma),\nu(\gamma))$ and $(i(\gamma,m),\nu(\gamma,m))$ to denote the
Maslov-type index and the iterated index of symplectic path $\gamma$ with the
periodic boundary condition which were introduced by Y. Long and his
collaborators (cf.[25], [26], [29], [34], etc.).

In [27, 28], S. Tang and the first author of this paper obtained the important
Bott-type formula and iteration inequalities for Malsov $(P,\omega)$-index as
follows.

######  Lemma 2.1.

([27], Bott-type iteration formula) For any $\tau>0$,
$\gamma\in\mathcal{P}_{\tau}(2n)$ and $m\in\mathbb{N}$, there hold

$i^{P^{m}}_{\omega_{0}}(\gamma,m)=\sum_{\omega^{m}=\omega_{0}}i^{P}_{\omega}(\gamma),\
\
\nu^{P^{m}}_{\omega_{0}}(\gamma,m)=\sum_{\omega^{m}=\omega_{0}}\nu^{P}_{\omega}(\gamma),$
(2.5)

where $P\in Sp(2n)$ and $\omega_{0}\in\bf U$.

######  Lemma 2.2.

([28]) For any path $\gamma\in\mathcal{P}_{\tau}(2n)$, $P\in Sp(2n)$ and
$\omega\in\mathbf{U}\setminus\\{1\\}$, it always holds that

$i^{P}(\gamma,1)+\nu^{P}(\gamma,1)-n+i_{1}(\xi)-i_{\omega}(\xi)\leq
i_{\omega}^{P}(\gamma)\leq
i^{P}(\gamma,1)+n-\nu_{\omega}^{P}(\gamma)+i_{1}(\xi)-i_{\omega}(\xi).$ (2.6)

######  Lemma 2.3.

([28], iteration inequality ) For any path $\gamma\in\mathcal{P}_{\tau}(2n)$,
$P\in Sp(2n)$ and $m\in\mathbf{N}$,

$\begin{split}&m(i^{P}(\gamma,1)+\nu^{P}(\gamma,1)-n)+n-\nu^{P}(\gamma,1)+mi_{1}(\xi)-i(\xi,m)\\\
&\leq i^{P^{m}}(\gamma,m)\\\ &\leq
m(i^{P}(\gamma,1)+n)-n-(\nu^{P^{m}}(\gamma,m)-\nu^{P}(\gamma,1))+mi_{1}(\xi)-i(\xi,m).\end{split}$
(2.7)

Let $e(M)$ be the elliptic height of symplectic matrix $M$ just as the same in
[29], the following lemma is important for the proof of Theorem 1.1.

######  Lemma 2.4 ([28]).

For any path $\gamma\in\mathcal{P}_{\tau}(2n)$, $P\in Sp(2n)$, set
$M=\gamma(\tau)$ and extend $\gamma$ to $[0,\infty)$ by (2.4). Then for any
$m\in\mathrm{N}$ we have

$\displaystyle\nu^{P^{m}}(\gamma,m)-\nu(\xi,1)+\nu(\xi,m+1)-\frac{e(P^{-1}M)}{2}-\frac{e(P^{-1})}{2}$
(2.8) $\displaystyle\leq
i^{P^{(m+1)}}(\gamma,m+1)-i^{P^{m}}(\gamma,m)-i^{P}(\gamma,1)$
$\displaystyle\leq\nu^{P}(\gamma,1)-\nu^{P^{(m+1)}}(\gamma,m+1)-\nu(\xi,m)+\frac{e(P^{-1}M)}{2}+\frac{e(P^{-1})}{2}.$

Let $S_{k\tau}=\mathbb{R}/(k\tau\mathbb{Z})$, $W_{P}=\\{z\in
W^{1/2,2}(S_{k\tau},\mathbb{R}^{2n})\mid z(t+\tau)=Pz(t)\\}$ be a closed
subspace of $W^{1/2,2}(S_{k\tau},\mathbb{R}^{2n})$. It is also a Hilbert space
with norm $\|\cdot\|$ and inner product $\langle\cdot,\cdot\rangle$ as in
$W^{1/2,2}(S_{k\tau},\mathbb{R}^{2n})$. We denote by $\|\cdot\|_{s}$ the
$L^{s}$-norm for $s\geq 1$. By the well-known Sobolev embedding theorem, we
have the following embedding property: for any $s\in[1,+\infty)$, there is a
constant $\alpha_{s}>0$ such that

$\|z\|_{s}\leq\alpha_{s}\|z\|,\ \ \forall z\in W_{P}.$ (2.9)

Let $\mathcal{L}_{s}(W_{P})$ and $\mathcal{L}_{c}(W_{P})$ denote the space of
the bounded self-adjoint linear operator and compact linear operator on
$W_{P}$. We define two operators $A$, $B\in\mathcal{L}_{s}(W_{P})$ by the
following bilinear forms:

$\langle Ax,y\rangle=\int_{0}^{\tau}(-J\dot{x}(t),y(t))dt,\ \ \langle
Bx,y\rangle=\int_{0}^{\tau}(B(t)x(t),y(t))dt.$ (2.10)

Suppose that
$\cdots\leq\lambda_{-j}\leq\cdots\leq\lambda_{-1}<0<\lambda_{1}\leq\cdots\leq\lambda_{j}\leq\cdots$
are all nonzero eigenvalues of $A$ (count with multiplicity), correspondingly,
$e_{j}$ is the eigenvector of $\lambda_{j}$ satisfying $\langle
e_{j},e_{i}\rangle=\delta_{ji}$. We denote the kernel of $A$ by $W_{P}^{0}$
which is exactly the space $\ker_{\mathbb{R}}(P-I)$. For $m\in\mathbb{N}$,
define the finite dimensional subspace of $W_{P}$ by

$W^{m}_{P}=W_{m}^{-}\oplus W_{P}^{0}\oplus W_{m}^{+}$

with $W_{m}^{-}=\\{z\in
W_{P}|z(t)=\sum_{j=1}^{m}a_{-j}e_{-j}(t),a_{-j}\in\mathbb{R}\\}$ and
$W_{m}^{+}=\\{z\in W_{P}|z(t)=\sum_{j=1}^{m}a_{j}e_{j}(t),\\\
a_{j}\in\mathbb{R}\\}$. Suppose $P_{m}$ is the orthogonal projections
$P_{m}:W_{P}\to W_{P}^{m}$ for $m\in\mathbb{N}\cup\\{0\\}$. Then $\\{P_{m}\mid
m=0,1,2,\cdots\\}$ is the Galerkin approximation sequence respect to $A$.

For a self-adjoint operator $T$, we denote by $M^{\ast}(T)$ the eigenspaces of
$T$ with eigenvalues belonging to $(0,+\infty),\\{0\\}$ and $(-\infty,0)$ with
$\ast=+,0$ and $\ast=-$, respectively. And the dimension of eigenspaces
$M^{\ast}(T)$ is denoted by $m^{\ast}(T)=\dim M^{\ast}(T)$. Similarly, we
denote by $M_{d}^{\ast}(T)$ the eigenspaces of $T$ with eigenvalues belonging
to $(d,+\infty),(-d,d)$ and $(-\infty,-d)$ with $\ast=+,0$ and $\ast=-$,
respectively. we denote $m_{d}^{\ast}(T)=\dim M_{d}^{\ast}(T)$. For any
adjoint operator $L$, we denote $L^{\sharp}=(L|_{ImL})^{-1}$.

The following theorem gives the relationship between the Maslov $P$-index and
Morse index for any $P\in Sp(2n)$.

######  Theorem 2.1.

([23], Lemma 3.2 and Theorem 4.6) For $P\in Sp(2n)$, suppose that
$B(t)\in\mathbb{C}(\mathbb{R},\mathcal{L}_{s}(\mathbb{R}^{2n}))$ and
$P^{T}B(t+\tau)P=B(t)$ with the Maslov $P$-index $(i^{P}(B),\nu^{P}(B))$. For
any constant $0<d\leq\frac{1}{4}\|(A-B)^{\sharp}\|^{-1}$, there exists an
$m_{0}>0$ such that for $m\geq m_{0}$, there holds

$\begin{split}m_{d}^{-}(P_{m}(A-B)P_{m})&=m+i^{P}(B),\\\
m_{d}^{0}(P_{m}(A-B)P_{m})&=\nu^{P}(B),\\\
m_{d}^{+}(P_{m}(A-B)P_{m})&=m+\dim\ker_{\mathbb{R}}(P-I)-i^{P}(B)-\nu^{P}(B),\end{split}$
(2.11)

where $B$ is the operator defined by $B(t)$.

For the operators $A$ and $B$ defined in (2.10), there is another description
of the Maslov $P$-index as follows.

######  Lemma 2.5 ([23]).

For any two operators $B_{1},B_{2}\in C(\mathbb{R},\mathcal{L}_{s}(2n))$ with
$B_{i}(t+\tau)=(P^{-1})^{T}B_{i}(t)P^{-1},i=1,2$ and $B_{1}<B_{2}$, there
holds

$i^{P}(B_{2})-i^{P}(B_{1})=\sum_{s\in[0,1)}\nu^{P}((1-s)B_{1}+sB_{2}).$ (2.12)

######  Remark 2.1.

Suppose that $B>0$, we have

$i^{P}(B)=\sum_{s\in[0,1)}\nu^{P}(sB).$ (2.13)

##  3 The proof of Theorem 1.1

In [23], the following result was proved.

######  Theorem 3.1.

([23]) Suppose $P\in Sp(2n)_{k}$, and the Hamiltonian function $H$ satisfies
(H1)-(H4), then for every $\tau>0$, the system (1.1) possesses a nonconstant
$P$-solution $(\tau,x)$ satisfying

$\rm{\text{dim}\ \text{ker}_{\mathbb{R}}}(P-I)+2-\nu^{P}(\it x)\leq i^{P}(\it
x)\leq\rm{\text{dim}\ \text{ker}_{\mathbb{R}}}(P-I)+1.$ (3.1)

Before the proof of Theorem 1.1, we need to get the information about the
iteration Maslov index for paths connecting $I$ and $P^{-1}$. Firstly, from
Lemma 1.1 we recall that for any $P\in Sp(2n)_{k}$, there exists
$(p,j_{1},j_{2},\cdots,j_{r})\in\mathbb{N}^{r+1}$ such that
$p+\sum_{m=1}^{r}j_{m}=n$ and $I_{2p}\diamond R(\frac{2\pi}{k})^{\diamond
j_{1}}\diamond\cdots\diamond R(\frac{2r\pi}{k})^{\diamond
j_{r}}\in\Omega^{0}(P^{-1})$. We remind that here
$\mathbb{N}=\\{0,1,\cdots,\\}$.

######  Lemma 3.1.

For $P\in Sp(2n)_{k}(r,p;j_{1},j_{2},\cdots,j_{r})$ and
$\xi\in\mathcal{P}_{\tau}(2n)$ with $\xi(\tau)=P^{-1}$, there holds

$(k+1)i(\xi)-i(\xi,k+1)=\sum_{m=1}^{r}(k-2m)j_{m}-kp.$ (3.2)

###### Proof.

By Theorem 9.3.1 in [29](also [34]), we have

$\displaystyle i(\xi,k+1)$
$\displaystyle=(k+1)(i(\xi)+S^{+}_{P^{-1}}(1)-C(P^{-1}))$ (3.3)
$\displaystyle\quad+2\sum_{\theta\in(0,2\pi)}E(\frac{(k+1)\theta}{2\pi})S^{-}_{P^{-1}}(e^{\sqrt{-1}\theta})-(S^{+}_{P^{-1}}(1)+C(P^{-1}))$
$\displaystyle=(k+1)i(\xi)+kS^{+}_{P^{-1}}(1)-(k+2)C(P^{-1})+2\sum_{\theta\in(0,2\pi)}E(\frac{(k+1)\theta}{2\pi})S^{-}_{P^{-1}}(e^{\sqrt{-1}\theta}),$

where $S^{\pm}_{M}(\omega)$ denote the splitting number of $M\in Sp(2n)$ at
$\omega\in\mathbf{U}$,
$C(M)=\sum_{\theta\in(0,2\pi)}S^{-}_{M}(e^{\sqrt{-1}\theta})$ and
$E(a)=\min\\{m\in\mathbb{Z}\mid m\geq a\\}$. One can see these notions in
Chapter 9 of [29].

For $P\in Sp(2n)_{k}(r,p;j_{1},j_{2},\cdots,j_{r})$, $I_{2p}\diamond
R(\frac{2\pi}{k})^{\diamond j_{1}}\diamond\cdots\diamond
R(\frac{2r\pi}{k})^{\diamond j_{r}}\in\Omega^{0}(P^{-1})$ with $2r<k$. By
direct computation, for $\theta\in(0,\pi)\cup(\pi,2\pi)$, we have

$\begin{split}S^{+}_{P^{-1}}(1)&=p\ ;\\\ C(P^{-1})&=\sum_{m=1}^{r}j_{m}\
;\end{split}$ (3.4)

$S^{-}_{P^{-1}}(e^{\sqrt{-1}\theta})=\begin{cases}j_{m},&if\
\theta=\frac{2m\pi}{k},1\leq m\leq
r,\;e^{\frac{2m\pi\sqrt{-1}}{k}}\in\sigma(P^{-1})\ ;\\\ \
0,&otherwise.\end{cases}$ (3.5)

So

$\displaystyle\sum_{\theta\in(0,2\pi)}E(\frac{(k+1)\theta}{2\pi})S^{-}_{P^{-1}}(e^{\sqrt{-1}\theta})$
$\displaystyle=E(\frac{k+1}{k})j_{1}+E(\frac{2(k+1)}{k})j_{2}+\cdots+E(\frac{r(k+1)}{k})j_{r}$
(3.6) $\displaystyle=2j_{1}+3j_{2}+\cdots+(r+1)j_{r}.$

Then

$\displaystyle i(\xi,k+1)$
$\displaystyle=(k+1)i(\xi)+kp-(k+2)(j_{1}+j_{2}+\cdots+j_{r})+2(2j_{1}+3j_{2}+\cdots+(r+1)j_{r})$
(3.7) $\displaystyle=(k+1)i(\xi)+kp-(k-2)j_{1}-(k-4)j_{2}-\cdots-(k-2r)j_{r}\
,$

thus $(k+1)i(\xi)-i(\xi,k+1)=\sum_{m=1}^{r}(k-2m)j_{m}-kp$. ∎

Now we are ready to prove Theorem 1.1.

Proof of Theorem 1.1.

Suppose that $(k\tau,x^{k})$ is the $k\tau$-periodic solution extended by
$P$-solution $(\tau,x)$ in Theorem 3.1. If $k\tau$ is not the minimal
$P$-symmetric period of $(k\tau,x^{k})$, i.e., $\tau>\min\\{\lambda>0\mid
x(t+\lambda)=Px(t),\forall t\in\mathbb{R}\\}$, then there exists some
$l\in\mathbb{N}$ such that

$T\equiv\frac{\tau}{l}=\min\\{\lambda>0\mid x(t+\lambda)=Px(t),\forall
t\in\mathbb{R}\\}.$

Thus $x(\tau-T)=x(0)$, both $(l-1)T$ and $kT$ are the period of $x$. Since
$kT$ is the minimal $P$-symmetric period, we obtain $kT\leq(l-1)T$ and then
$k\leq l-1$.

Note that $x|_{[0,kT]}$ is the $k$-th iteration of $x|_{[0,T]}$. Suppose
$\gamma\in\mathcal{P}_{T}(2n)$ is the fundamental solution of the following
linear Hamiltonian system

$\dot{z}(t)=JB(t)z(t)$ (3.8)

with $B(t)=H^{\prime\prime}(x|_{[0,T]}(t))$. Suppose $\xi$ be any symplectic
path in $\mathcal{P}_{T}(2n)$ such that $\xi(T)=P^{-1}$, since $P^{k}=I$, then

$\nu(\xi,1)=\nu(\xi,k+1)=\nu(\xi,l).$ (3.9)

All eigenvalues of $P$ and $P^{-1}$ are on the unit circle, then the elliptic
height

$e(P^{-1})=e(P)=2n.$ (3.10)

Since the system (1.1) is autonomous, we have

$\nu_{1}(x|_{[0,kT]})\geq 1\ \ \text{and}\ \
\nu^{P^{l-1}}(\gamma,l-1)=\nu_{1}(x|_{[0,(l-1)T]})\geq 1.$ (3.11)

By Lemma 2.4, $P^{l-1}=I$ and (3.9)-(3.11), we have

$\begin{split}i^{I}(\gamma,l-1)&=i^{P^{l-1}}(\gamma,l-1)\\\ &\leq
i^{P^{l}}(\gamma,l)-i^{P}(\gamma,1)+\nu(\xi,1)-\nu(\xi,l)+\frac{e(P^{-1}\gamma(T))}{2}+\frac{e(P^{-1})}{2}-\nu^{P^{l-1}}(\gamma,l-1)\\\
&\leq i^{P^{l}}(\gamma,l)-i^{P}(\gamma,1)+\frac{e(P^{-1}\gamma(T))}{2}+n-1\\\
&\leq i^{P^{l}}(\gamma,l)-i^{P}(\gamma,1)+2n-1.\end{split}$ (3.12)

Note that $i^{P^{l}}(\gamma,l)=i^{P}(x\mid_{[0,\tau]})\leq\text{dim\
ker}_{\mathbb{R}}(P-I_{2n})+1$, here we write $i^{P}(x\mid_{[0,\tau]})$ for
$i^{P}(x)$ to remind the solution $x$ is defined in the interval $[0,\tau]$.
By the definition of Maslov $P$-index,

$i^{I}(\gamma,l-1)=i_{1}(\gamma,l-1)+n.$

So we get

$i_{1}(\gamma,l-1)\leq\dim\ker_{\mathbb{R}}(P-I)-i^{P}(\gamma,1)+n.$ (3.13)

By the condition (H5) and Remark 2.1, we have

$i^{P}(\gamma,1)=i^{P}(B)=\sum_{s\in[0,1)}\nu^{P}(sB)=\sum_{s\in[0,1)}\dim\ker_{\mathbb{R}}(\gamma_{B}(sT)-P).$
(3.14)

Here we remind that $B(t)$ and $\gamma_{B}$ are defined in (3.8). Since
$\gamma_{B}(0)=I$, so
$\dim\ker_{\mathbb{R}}(\gamma_{B}(sT)-P)=\dim\ker_{\mathbb{R}}(P-I)$ when
$s=0$. Thus we have

$i^{P}(\gamma,1)\geq\dim\ker_{\mathbb{R}}(P-I).$ (3.15)

From (3.12), it implies

$i_{1}(\gamma,l-1)\leq n.$ (3.16)

By the convex condition (H5), we also have

$i_{1}(x|_{[0,kT]})\geq n\ \ \text{and}\ \ i_{1}(x|_{[0,(l-1)T]})\geq n.$
(3.17)

We set $m=\frac{l-1}{k}$. Note that $x|_{[0,(l-1)T]}$ is the $m$-th iteration
of $x|_{[0,kT]}$. By (3.11), (3.16), (3.17) and Lemma 4.1 in [26], we obtain
$m=1$ and then $k=l-1$. From the above process (3.12)-(3.13) and
(3.15)-(3.17), we obtain $k=l-1$ provided $e(P^{-1}\gamma(T))=2n$, and

$\begin{split}i^{P}(\gamma,1)&=\dim\ker_{\mathbb{R}}(P-I);\\\
i^{P}(\gamma,l)&=i^{P}(\gamma,k+1)=\dim\ker_{\mathbb{R}}(P-I)+1,\\\
\nu^{P^{k}}(\gamma,k)&=\nu^{P}(\gamma,1)=1.\end{split}$ (3.18)

Here we remind that the left inequality in (2.7) of Lemma 2.3 holds
independent of the choice of $\xi\in\mathcal{P}_{\tau}(2n)$, then for any
$\xi\in\mathcal{P}_{\tau}(2n)$ we have

$i^{P}(\gamma,k+1)\geq(k+1)(i^{P}(\gamma,1)+\nu^{P}(\gamma,1)-n)+n-1+(k+1)i_{1}(\xi)-i(\xi,k+1).$
(3.19)

By the condition $P\in Sp(2n)_{k}(r,p;j_{1},j_{2},\cdots,j_{r})$, we get

$\text{dim\ ker}_{\mathbb{R}}(P-I)=2p,$ (3.20) $k-2\sum_{m=1}^{r}m\cdot
j_{m}>1.$ (3.21)

Applying (3.18), (3.20) and Lemma 3.1 to (3.19), we get

$k-2\sum_{m=1}^{r}m\cdot j_{m}\leq 1.$ (3.22)

It is contradict to the inequality (3.21). So the minimal $P$-symmetric period
of $(k\tau,x^{k})$ is $k\tau$.

######  Remark 3.1.

Note that
$e(P^{-1}\gamma(T))=e((P^{-1}\gamma(T))^{l})=e(P^{-1}\gamma(\tau))=2n$ is
required in the above proof. If $e(P^{-1}\gamma(T))\leq 2n-2$, we get
$i_{1}(\gamma,l-1)<n$ by taking the same process as (3.12)-(3.13). It
contradicts to the second inequality of (3.17). At this moment, the minimal
$P$-symmetric period of $(k\tau,x^{k})$ is $k\tau$.

The condition (H5) can be replaced by a weaker condition:
$H^{\prime\prime}(x(t))\geq 0$ and $\int_{0}^{\tau}H^{\prime\prime}(x(t))dt>0$
for the $P$-solutuin $(\tau,x)$ in Theorem 3.1.

## References

  * [1] A.Ambrosetti, V.Coti Zelati, _Solutions with minimal period for Hamiltonian systems in a potential well_ , Ann. Inst. H. Poincar$\acute{e}$ Anal. Non Lin$\acute{e}$aire 4 (1987), 242–275.
  * [2] A.Ambrosetti, G.Mancini, _Solutions of minimal period for a class of convex Hamiltonian systems_ , Math. Ann. 255 (1981), 405–421.
  * [3] K. C.Chang, _Infinite dimensional Morse theory and multiple solution problems_ , in Progress in Nonlinear Differention Equations and Their Application, Vol.6, (1993).
  * [4] K. C. Chang,J. Liu and M. Liu, _Nontrivial periodic solutions for strong resonance Hamiltonian systems_ , Ann. Inst. H. Poincar$\acute{e}$ Anal. Non Lin$\acute{e}$aire 14(1) (1997), 103–117.
  * [5] F. Clarke, I. Ekeland, _Hamiltonian trajectories having prescribed minimal period_ , Comm. Pure. Appl. Math. 33 (1980), no.3, 103–116.
  * [6] A.Chenciner and R.Montgomery, _A remarkable periodic solution of the three body problem in the case of equal masses_ , Ann. of Math. 152: 3 (2000), 881–901.
  * [7] Y. Dong, _$P$ -index theory for linear Hamiltonian systems and multiple solutions for nonlinear Hamiltonian systems_, Nonlinearity 19: 6 (2006), 1275–1294.
  * [8] Y. Dong and Y. Long, Closed characteristics on partially symmetric compact convex hypersurfaces in $R^{2n}$, J. Diff. Equa. 196 (2004), 226-248.
  * [9] D. Dong and Y. Long, _The iteration formula of the Maslov-type index theory with applications to nonlinear Hamiltonian systems_ , Trans. Amer. Math. Soc. 349 (1997), 2619–2661.
  * [10] I. Ekeland, H. Hofer, _Periodic solutions with prescribed minimal period for convex autonomous Hamiltonian sysmtems_ , Invent. Math. 81 (1985), 155–188.
  * [11] G.Fei, _Relative Morse index and its application to Hamiltonian systems in the presence of symmetries_ , J. Diff. Equa. 122 (1995), 302–315.
  * [12] G.Fei and Q.Qiu, _Minimal period solutions of nonlinear Hamiltonian systems_ , Nonlinear Analysis, Theory, Method Applications 27: 7 (1996), 821–839.
  * [13] G.Fei, S. K.Kim, T.Wang _Minimal period estimates of period solutions for superquadratic Hamiltonian systmes_ , J. Math. Anal. Appl. 238 (1999), 216–233.
  * [14] D. Ferrario and S. Terracini, _On the existence of collisionless equivariant minimizers for the classical $n$-body problem_, Invent. Math. 155: 2 (2004), 305–362.
  * [15] N.Ghoussoub, _Location, multiplicity and Morse indices of min-max critical points_ , J. Reine Angew Math. 417 (1991), 27–76.
  * [16] X.Hu and P.Wang, _Conditional Fredholm determinant for the $S$-periodic orbits in Hamiltonian systems_, Joural of Functional Analysis 261 (2011), 3247–3278.
  * [17] X.Hu and S.Sun, _Index and stability of symmetric periodic orbits in Hamiltonian systems with application to figure-eight orbit_ , Comm. Math. Phys. 290 (2009), 737–777.
  * [18] X. Hu and S. Sun, _Morse index and the stability of closed geodesics_ , Sci. China Math. 53: 5 (2010), 1207–1212.
  * [19] X.Hu and S.Sun, _Stability of relative equilibria and Morse index of central configurations_ , C. R. Acad. Sci. Paris 347 (2009), 1309–1312.
  * [20] A.Lazer and S.Solomini, _Nontrivial solution of operator equations and Morse indices of critical points of min-max type_ , Nonlinear Anal. 12 (1988), 761–775.
  * [21] C.Liu, _Maslov P-index theory for a symplectic path with applications_ , Chin. Ann. Math. 4 (2006), 441–458.
  * [22] C.Liu, _Periodic solutions of asymptotically linear delay differential systems via Hamiltonian systems_ , J. Diff. Equa. 252 (2012), 5712–5734.
  * [23] C. Liu, _Relative index theories and applications_ , Top. Meth. Nonl. Anal, to appear.
  * [24] C. Liu, _Minimal period estimates for brake orbits of nonlinear symmetric Hamiltonian systems_ , Discrete and continuous dynamical systems 27(1) (2010), 337–355.
  * [25] C.Liu and Y.Long, _An optimal increasing estimate of the iterated Maslov-type indices_ , Chinese Sci. Bull. 42 (1997), 2275–2277.
  * [26] C.Liu and Y.Long, _Iteration inequalities of the Maslov-type index theory with applications_ , J. Diff. Equa. 165 (2000), 355–376.
  * [27] C.Liu and S.Tang, _Maslov $(P,\omega)$-index theory for symplectic paths_, Advanced Nonlinear Studies 15 (2015), 963–990.
  * [28] C.Liu and S.Tang, _Iteration inequalities of the Maslov $P$-index theory with applications_, Nonlinear Analysis 127 (2015), 215–234.
  * [29] Y.Long, _Index theory for symplectic paths with application_ , Progress in Mathematics, Vol. 207, Birkh$\ddot{a}$user Verlag, 2002.
  * [30] Y.Long, _Bott formula of the Maslov-type index theory_ , Pacific J. Math. 187 (1999), 113–149.
  * [31] Y. Long, _The minimal period problem for classical Hamiltonian systems with even potentials_ , Ann. Inst. H. Poincar$\acute{e}$ Anal. non. lin$\acute{e}$aire 10(1993), 605–626.
  * [32] Y. Long, _The minimal period problem of periodic solutions for autonomous superquadratic second order Hamiltonian systems_ , J. Diff. Equa. 111(1994), 147–174.
  * [33] Y. Long, _On the minimal period for periodic solutions of nonlinear Hamiltonian systmes_ , Chinese Ann. of Math. 18B(1997), 481–484.
  * [34] Y.Long and C.Zhu, _Closed characteristics on compact convex hypersurfaces in $R^{2n}$_, Annals of Math. 155 (2002), 317–368.
  * [35] P. H.Rabinowitz, _Periodic solutions of Hamiltonian systmes_ , Comm. Pure Appl. Math. 31 (1978), 157–184.
  * [36] P. H.Rabinowitz, _Minimax method in critical point theory with applications to differential equations_ , CBMS Regional Conf. Ser. in Math., No.65, A. M. S., Providence(1986).
  * [37] S.Solimini, _Morse index estimates in min-max theorems_ , Manuscription Math. 63 (1989), 421–453.


# Optimizing Layer-Fused Scheduling of Transformer Networks on Multi-
accelerator Platforms

Steven Colleman*, Arne Symons*, Victor J.B. Jung$\dagger$, Marian Verhelst
*Department of Electrical Engineering, MICAS-ESAT, KU Leuven, Belgium.
$\dagger$Integrated Systems Laboratory, ETH Zürich, Switzerland.

###### Abstract

The impact of transformer networks is booming, yet, they come with significant
computational complexity. It is therefore essential to understand how to
optimally map and execute these networks on modern neural processor hardware.
So far, literature on transformer scheduling optimization has been focusing on
deployment on GPU and specific ASICs. This work enables extensive
hardware/mapping exploration by extending the DSE framework Stream towards
support for transformers across a wide variety of hardware architectures and
different execution schedules. After validation, we explore the optimal
schedule for transformer layers/attention heads and investigate whether layer
fusion is beneficial to improve latency, energy or memory requirements. Our
study shows that the memory requirements for active feature data can be
drastically reduced, by adapting the execution schedule based on the size of
the input of the attention head.

###### Index Terms:

CNN, transformer networks, cross-layer, scheduling, hardware modeling and
optimization

##  I Introduction

Nowadays, machine learning algorithms are gaining more and more importance in
a broad range of daily-life applications in fields like natural language and
image processing. Over the past years, transformer networks have been
introduced [1], with a fast growing set of applications [2]. These networks
use the mechanism of self-attention, giving a data-dependent weight to each
element of the network input data.

Figure 1: Operators of a transformer network, highlighting one attention head
containing 5 matrix-matrix multiplications (3x features$\times$weights, 2x
features$\times$features), a transpose and a softmax layer. Stream [3] is
extended with these additional layer types.

These algorithms, however, lead to new challenges in terms of efficient
hardware mapping, as transformer networks contain layer types and sequences
not present in traditional neural networks. Many practical implementations of
transformer-optimized processors have been proposed [4][5][6]. Yet, they are
ad hoc designs, often focusing on a specific network. Alternative studies
explore the optimal execution schedule, assessing the potential benefits of
layer fusion of specific transformer layers [7][8], yet do so for specific
hardware architectures and networks. The work of [9] performs a broader
analysis, but limits the transformer scheduling space to layer-by-layer
execution only. The goal of this paper is, therefore, to approach the design
and scheduling exploration more methodologically, by creating an analytical
model to optimize the mapping efficiency of transformer networks on a wide
variety of neural processing hardware architectures. The framework should not
only support layer-by-layer schedules, but also the emerging scheduling
technique of deep layer-fused execution [3].

For traditional CNNs, execution efficiency has been analyzed and optimized by
making use of analytical frameworks such as Timeloop [10], ZigZag [11], Kwon
et al. [12]. DeFiNES [13] extended the scheduling space to deep layer-fused
execution for single-core architectures and Stream [3] extended this further
towards multi-core execution. However, none of these frameworks support
transformer networks, lacking support for transpose and softmax layers.
Therefore, this work fills this gap by creating an analytical cost estimation
framework which can both handle transformer networks as well as multi-core
layer-fused execution.

Section II first sketches the background on transformers and Stream’s
analytical cost estimation, followed by introducing the core methodology of
this paper: the requirements to accurately model layer-fused execution of
transformers.

Section III contains a validation of the extended framework against real-world
hardware to verify the correct functionality of the modified framework.

Finally, Section IV uses the developed framework to perform explorations on
efficiently mapping transformer networks on various hardware architectures, in
order to explore the optimal schedule for transformer layers/attention heads
and investigate whether layer fusion is beneficial to improve latency, energy
or memory requirements.

##  II Background & Methodology

###  II-A Background: Transformer Networks

Encoder-decoder transformer networks contain several attention heads [1],
illustrated in Fig. 1, in which scale dot-product attention takes place:

An input matrix of size $M\times N$ is multiplied with three different weight
matrices $W_{Q}$, $W_{K}$ and $W_{V}$ of size $N\times N$ to obtain three new
matrices $Q$ (query), $K$ (key) and $V$ (value), respectively, all of size
$M\times N$. Based on these three matrices, the attention output is calculated
as:

$Attention(Q,K,V)=softmax\left(\frac{QK^{T}}{\sqrt{d_{k}}}\right)V$ (1)

Here, $d_{k}$ is a constant and will therefore not be taken into account in
this work as in practice, this division can be done by adjusting the weights
from one of the three mentioned weight matrices. This means that the attention
block can be described with 7 layers: 5 matrix-matrix multiplications, a
transpose layer, and a softmax layer. This softmax operation happens row-wise.
With input matrix $QK^{T}$ of size $M\times M$, the output of the softmax
layer will be defined as:

$softmax(QK^{T}_{(i,j)})=\frac{exp(QK^{T}_{(i,j)})}{\sum_{j=1..M}exp(QK^{T}_{(i,j)})}$
(2)

In practice, transformer networks make use of multi-headed self-attention,
implying that every attention layer consists of multiple previously-described
heads in parallel.

Throughout this text, the left and right input matrices of a matrix-matrix
multiplication will be denoted by $I1$ and $I2$, respectively, with $O$ being
the output matrix, with following dimensions: $I1(R\times S),I2(S\times
T),O(R\times T)$. Therefore, for the computation of $Q$, $K$ and $V$, $R=M$
and $S=T=N$. For the computation of $QK^{T}$, $R=T=M$ and $S=N$. For the
computation of $QK^{T}.V$, $R=S=M$ and $T=N$.

###  II-B Background: Stream

The open-source framework Stream [3] is a hardware estimation framework that
evaluates the (energy, latency, memory requirement) performance of executing a
given neural network, with a specific execution schedule on a specific
hardware architecture.

Figure 2: Illustration of Stream [3], with indicated in red the blocks where
adaptations have been made.

Stream works in 5 steps, illustrated in Fig. 2:

  1. 1.

The framework splits the layers of the network into fine-grained computation
nodes. Without splitting, only layer-by-layer execution would be possible.
These form the basis for the layer-fused scheduling of step 5.

  2. 2.

An R-tree-based dependency generator derives for each computation node on
which computation node(s)’ output of the previous layer(s) it depends. These
dependencies are needed to explore layer-fused scheduling [14], which exploits
the fact that the partial output data of a given layer can already be used in
the execution of a following layer before the execution of the former layer is
completely finished.

  3. 3.

An analytical intra-core mapping where the mapping of each computation node on
each core of the hardware architecture is optimized for latency and/or energy.
The spatial unrolling and temporal mapping of the node are optimized [15].
With spatial unrolling, we mean the physical parallelization over the PE array
for improved utilization within a clock cycle. With temporal mapping, we mean
the order and splitting of the for loops for better reuse across clock cycles.

  4. 4.

A genetic algorithm optimizes which layer should be allocated to which core.

  5. 5.

A scheduling algorithm decides in which order the computation nodes should be
executed to optimize overall latency or memory usage with a configurable cost
function. The framework iterates between steps 4 and 5 to find the optimal
allocation and scheduling.

###  II-C Proposed Transformer Extensions for Stream

Figure 3: Illustration of dependencies in different layer types.

In order to support layer-fused processing of transformer networks, the Stream
framework is extended. To this end, support is added for both transpose and
softmax layers, which are present in the attention heads of transformer
networks. Specifically, this required to modify Step 1 and Step 2 of the
framework.

To support the splitting of transpose and softmax layers into smaller
individually-schedulable computation nodes in Step 1, we create computation
nodes based on the top ’for loop’ of the temporal mapping: one for each $R$ if
the top ’for loop’ is ’for R’ etc.

Also the computation node dependency generation of step 2 is modified to
accurately model the dependencies between the computation nodes of the newly-
introduced layer types in the attention heads, as illustrated in Fig. 3: For
matrix-matrix multiplication, output position $(i,j)$ depends on the $i$-th
row of the left input matrix and the $j$-th column of the right input matrix.
For the transpose and softmax layers, additional dependency modeling is
required. For a transpose layer, for each output location $(i,j)$ a dependency
has to be registered towards input $(j,i)$. For a softmax layer, each output
location $(i,j)$ is dependent on all input positions of row $i$, due to the
summation in the denominator of the fraction. The exponent, however, is an
elementwise operation, which does not need an additional dependency. The
Stream framework is as such extended to automatically extract the dependency
implications these additional layer types have on the individually-schedulable
compute graph. This enables reuse of Stream’s extensive scheduling and
multicore performance modeling for transformer scheduling and mapping
optimizations. No further modifications are needed for step 3, 4 and 5 as
these make use of (a graph of) computation nodes, not taking into account the
layer type.

##  III Validation

We validate the functional correctness and accuracy of the enhanced modeling
framework with the mapping of a CCT-like transformer network [16] on the GAP8
hardware platform, a commercial Parallel Ultra Low Power (PULP [17])-based
microcontroller developed by GreenWaves Technologies. The CCT model aims at
detecting seizures in epileptic patients by analyzing electroencephalography,
encapsulating the Integrized SoftMax, Layer Normalization and GELU from I-BERT
[18] as well as the library of optimized Attention kernel from [19]. For
validation purposes, the model is deployed with a sequence length of 81 and of
128, with for each configuration 32 embedding channels and a projection space
of size 32. The network is quantized to 8 bit fixed-point precision, with a
linear re-quantization after every layer consisting of an element-wise
multiplication, addition, and shift. The memory hierarchy of GAP8 has 4
levels, noted L3 to L0, and each level can hold every operand type. GAP8 uses
a DMA Engine to pipeline data transfers between L1 and L2 through the AXI
crossbar, drastically reducing the latency per element for large transfers.
Each of the 8 cores makes use of 1 MAC. While the L2 to L1 interface is 64-bit
wide, configuration and package size overhead virtually reduce the effective
bandwidth to 51 bits per cycles.

Executing the whole MHSA on a GAP8 hardware prototype @ 100MHz is measured to
take 1.836 MCycles (seq. length 81), resp. 3.905 MCycles (seq. length 128),
reaching an average of 3.2 MAC/cycle. To validate the modified Stream
framework, we model the described hardware architecture in Stream, as well as
the mapped network execution schedules. Stream suggests a layer-fused
execution, just like the used scheduling in the measurements. The modeling
framework estimates a latency of 1.692 MCycles (seq. length 81), resp. 3.540
MCycles (seq. length 128), as shown in Fig. 4, which is a deviation of 8%,
resp. 9% in comparison with the real-world measurements, which is small enough
to distinguish relevant cost differences between various scheduling and
hardware architecture options.

Figure 4: Validation results of the framework with two different sized CCT-
like networks on GAP8.

##  IV Exploration with analysis

---
(a) Optimized scheduling for layer-by-layer

---
(b) Optimized scheduling for layer-fused, with $M<N$

---
(c) Optimized scheduling for layer-fused, with $M>N$

Figure 5: Scheduling and total active features memory usage across time for
memory-optimized layer-by-layer and layer-fused execution, with best possible
latency. For the layer-fused execution, we make a distinction between $M<N$
and $M>N$ as this leads to different optimal schedulings. For each scheduling,
both $M>N$ as $M=N$ as $M<N$ are shown to illustrate why an optimal scheduling
for one input size is not necessarily the best one for another scheduling. The
dot at the end of the plots indicates that the output of the attention head
should remain active to consume as the input of the next attention head. For
a), the computations of $V$ and $Q.K^{T}$ can be swapped, this doesn’t change
latency or maximal memory requirement. For c), it is also an option to fuse
$Q$, $Q.K^{T}$ and $Q.K^{T}.V$, instead of doing first $Q$ completely and then
only fuse $Q.K^{T}$ and $Q.K^{T}.V$.

###  IV-A Introduction

For typical neural networks, layer-fused execution has been shown to
outperform layer-by-layer execution in terms of latency and active feature
memory [3] [14]. In this exploration section, we will use Stream to
investigate whether these benefits are still true for attention heads in
transformer networks. Therefore, we will first explore the optimal layer-by-
layer schedule towards the lowest latency, resp. lowest memory footprint.
Next, results are compared to an optimized schedule using layer-fused
scheduling. The study will first be conducted on a single core hardware
architecture with a 64x64 array of processing elements, after which multi-core
hardware mappings are explored. Memory hierarchy will be optimized to have
optimal latency and memory footprint for the layer-by-layer schedule. The
scheduler is given full flexibility in terms of allowed spatial and temporal
unrolling.

To explore the impact of the attention head’s input matrix size, we will
perform all experiments with three possible input tensor shapes of equal total
size (same $M.N$): square ($M=N$), deep rectangle ($M>N$, here $M=4N$) and
flat rectangle ($M<N$, here $M=(1/4)N$). $M$ and $N$ are here always assumed
to be multiples of the PE array size (64). The conclusions described below are
exactly the same for each set of $M$ and $N$ meeting these requirements.

###  IV-B Layer-by-layer Optimized Scheduling

####  IV-B1 Hardware and Mapping Exploration

The optimal schedule spatially unrolls (parallel hardware mapping) the $S$ and
$T$ dimensions, while it moves the R dimension to the outer temporal unrolling
dimension. In this way, one row of the left input matrix can be discarded and
substituted by one row of the output matrix reducing the algorithm’s memory
footprint. Therefore, we make use of two L1 memories. One for the left input
matrix and output matrix (bandwidth of 64 words sufficient based on
scheduling), and one for the right input matrix with a multi-banked bandwidth
of 4096 words. Each PE also has a small register file for left inputs, right
inputs and outputs, respectively. Connections between these register files are
also placed to make it possible to consume outputs of a given attention head
layer immediately as input of a next attention head layer. A small SIMD core
is placed in parallel with the 64x64 core to compute the output of the softmax
layer.

####  IV-B2 Memory Footprint Analysis

Fig. 5a visualizes the memory footprint of the memory-optimal scheduling
result obtained by the extended version of Stream for various input shapes.
Probing deeper, following analysis can be made. At the beginning of the
schedule, only the input matrix is active data. The memory requirement is
therefore $M.N$ words. During the computation of the $Q$ and $K$ matrix, new
active features are generated without being able to discard features as we
still need the input matrix to compute the $V$ matrix. Therefore, the total
active memory requirement grows to $3M.N$. Next, during the computation of
$V$, each iteration of the ’for R’ loop results in a new row of $V$, such that
a row of the input matrix with the same size can be discarded. Therefore, the
active feature memory remains constant during the execution of this layer.

After consuming one row of the $Q$ matrix, this row can be discarded and
substituted by a row of $Q.K^{T}$. Note that the $K$ or $K^{T}$ matrix has to
be stored completely until the last input row of $Q$ is consumed. The active
feature memory requirement at the end of the computation of the $Q.K^{T}$
matrix is $2M.N+M^{2}$, which is a decrease in memory if $M<N$, a constant if
$M=N$ and an increase if $M>N$. After the computation of $Q.K^{T}$, the $K$
matrix can be discarded. Afterwards, each row of the $Q.K^{T}$ matrix is
substituted by a row of the output matrix. When the output matrix of the
attention head is computed, $V$ can be thrown away and only the $M.N$ words
from the output matrix remain to be used as input of another attention head.

We can hence conclude that in the optimal layer-by-layer schedule, the maximal
total active features memory requirement $A_{LBL}$ equals:

  * •

If $M<=N$: $A_{LBL}=3M.N$

  * •

If $M>N$: $A_{LBL}=2M.N+M^{2}$

###  IV-C Layer-fused Optimized Scheduling

In this section, we use the extended version of Stream to find a layer-fused
scheduling with the same optimal latency and a lower total active features
memory footprint compared to the memory-optimal layer-by-layer schedule.

####  IV-C1 Optimization for $M<N$

Fig. 5b visualizes the optimal execution schedule for the $M<N$ scenario,
together with the corresponding memory footprint. As can be observed, the
optimized schedule fuses the computation of $Q$ and $Q.K^{T}$. This means that
the outputs of $Q$ are immediately consumed by the computation of $Q.K^{T}$
and are not stored in L1 memory.

The start of the memory usage is the same as in the layer-by-layer case with
an increase to $2M.N$ active feature words. Afterwards, the fusion of $Q$ and
$Q.K^{T}$ takes place which means that the total active features memory
requirement increases with the shape of $Q.K^{T}$ (and not $Q$) and becomes
therefore equal to $2M.N+M^{2}$, whereafter the $K^{T}$ matrix can be
discarded. Memory requirements during the next two layer executions can be
derived using the same methodology, but do not become larger than
$2M.N+M^{2}$. Therefore, the maximal total active features memory requirement
for layer-fused execution $A_{LF}=2M.N+M^{2}$.

We can now compare $A_{LBL}$ with $A_{LF}$ and define the relative memory
footprint gain $\alpha=\frac{A_{LF}}{A_{LBL}}$. For $M<N$,

$\alpha=\frac{2M.N+M^{2}}{3M.N}=\frac{2N+M}{3N}<1$ (3)

resulting in a layer-fused scheduling with the same latency and better maximal
active features memory requirement. For example, with a 128x1024 input size,
$\alpha\approx 0.711$, leading to a 29% memory requirement reduction. In the
limit, we obtain

$\lim_{\frac{M}{N}\to 0}\frac{2N+M}{3N}=\frac{2}{3}$ (4)

which means we can reduce the total active features memory requirement with
1/3 with constant latency.
The same schedule, however, does not benefit execution with different input
tensor sizes, as for $M>N$,

$\alpha=\frac{2M.N+M^{2}}{2M.N+M^{2}}=1$ (5)

and for $M=N$,

$\alpha=\frac{2M.N+M^{2}}{3M.N}=\frac{3M^{2}}{3M^{2}}=1$ (6)

Yet, other optimizations allows to execute such attention heads more
efficiently, as discussed in the next paragraph.

####  IV-C2 Optimization for $M>N$

The schedule of Fig. 5c is found when minimizing the feature memory
requirement for attention heads with $M>N$. The memory graph now only contains
one line as memory requirements are the same for all three input cases ($M>N$,
$M=N$ and $M<N$). First, $K$, $V$ and $Q$ are computed in a layer-by-layer
schedule. At this point, the memory requirement is $3M.N$, similar to the
layer-by-layer case. Afterwards, the computations of $Q.K^{T}$ and $Q.K^{T}.V$
are fused, together with the softmax operation in the SIMD core. This means
that features from $Q.K^{T}$ are not stored in local memory and one row of the
$Q$ matrix is substituted by one row of the $Q.K^{T}.V$ matrix, with the same
size. The maximal active features memory requirement, therefore, remains
constant with $A_{LF}=3M.N$ words.

If we now again compute the relative memory footprint gain
$\alpha=\frac{A_{LF}}{A_{LBL}}$ for $M>N$,

$\alpha=\frac{3M.N}{2M.N+M^{2}}=\frac{3N}{2N+M}<1$ (7)

resulting in a more memory efficient layer-fused scheduling. For example, with
a 1024x128 input size, $\alpha=0.3$, leading to a 70% memory requirement
reduction. In the limit, we obtain

$\lim_{\frac{M}{N}\to\infty}\frac{3N}{2N+M}\approx\frac{3N}{M}$ (8)

which means the total active features memory requirement can be reduced to a
third of $\frac{M}{N}$. For $M<=N$,

$\alpha=\frac{3M.N}{3M.N}=1$ (9)

indicating that this schedule is not beneficial over layer-by-layer scheduling
for those input tensor sizes.

####  IV-C3 Take-away Message and State-of-the-Art Comparison

The relative memory footprint gain $\alpha$ between the layer-fused memory
footprint and the layer-by-layer memory footprint is $\frac{2N+M}{3N}$ for
$M<N$ and $\frac{3N}{2N+M}$ for $M>N$. The value of $\alpha$ in function of
$\frac{M}{N}$ is plotted in Fig. 6. Important to note also is that a reduction
in required feature memory size will also result in a potential reduction of
the total energy consumption, as a smaller memory reduces the read/write
energy cost.

Figure 6: Relative memory footprint gain $\alpha=\frac{A_{LF}}{A_{LBL}}$ in
function of $\frac{M}{N}$.

The optimal execution schedule differs for both cases, with a different set of
layers that have to be fused. In both cases, the data of the largest matrix
were immediately consumed as inputs of the next layer, with smaller output
dimensions. For $M<N$, the $Q.K^{T}$ matrix is smaller than the $Q$ matrix.
For $M>N$, the $Q.K^{T}.V$ matrix is smaller than the $Q.K^{T}$ matrix. This
is also the reason why layer-fused execution doesn’t reduce the memory
requirements for the case where $M=N$ as all the output matrices of the 5
matrix-matrix multiplication layers have the same size. For multi-headed
attention blocks on multi-core architectures, the gain is exactly the same as
the one in Fig. 6 because each core executes another attention head in
parallel, as no inputs or weights are typically shared among heads. Table I
compares this work to previous state-of-the-art modelling frameworks, where
our extensions of [3] enable transformer exploration using layer fusion on
multi-accelerator architectures.

| [10] | [11] | [13] | [12] | [3] | This work
---|---|---|---|---|---|---
Layer fusion | $\times$ | $\times$ | ✓ | $\times$ | ✓ | ✓
Multi-accelerator | $\times$ | $\times$ | $\times$ | ✓ | ✓ | ✓
Transformer support | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | ✓
TABLE I: Comparison with state-of-the-art modelling frameworks.

##  V Conclusion

Transformer networks contain new layer types such as matrix-matrix
multiplications with two feature matrices, transpose layers and softmax layers
leading to new hardware acceleration challenges and scheduling opportunities.
To explore these methodologically, this work extends Stream, a state-of-the-
art multi-accelerator mapping optimization framework, with these layer types
and new execution schedules. After validation, a scheduling exploration is
performed from which we conclude that, depending on the size of the input
matrix of the transformer’s attention head, layer fusion can outperform layer-
by-layer scheduling with a drastically reduced feature memory footprint,
without latency penalty. For flat and wide input matrices ($M<<N$), memory
footprint can be reduced up to a third of the original size, while for deep
and shallow input matrices ($M>>N$), benefits can get to a third of
$\frac{M}{N}$. This memory reduction technique can aditionally lead to an
energy reduction as smaller memories reduce the memory access energy. Both
input dimension cases are characterized by a different optimal layer-fused
schedule, found by the extended Stream framework. The framework, including
extensions, is open-sourced at github.com/kuleuven-micas/stream.

##  VI Acknowledgment

This project has been partly funded by the European Research Council (ERC)
under grant agreement No. 101088865, the European Commission through the
project CONVOLVE (101070374), the Flanders AI Research Program and KU Leuven.

## References

  * [1] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [2] “Transformers,” https://huggingface.co/docs/transformers/index, 2023.
  * [3] A. Symons, L. Mei, S. Colleman, P. Houshmand, S. Karl, and M. Verhelst, “Towards heterogeneous multi-core accelerators exploiting fine-grained scheduling of layer-fused deep neural networks,” _arXiv preprint arXiv:2212.10612_ , 2022.
  * [4] Y. Wang, Y. Qin, D. Deng, J. Wei, Y. Zhou, Y. Fan, T. Chen, H. Sun, L. Liu, S. Wei _et al._ , “A 28nm 27.5 tops/w approximate-computing-based transformer processor with asymptotic sparsity speculating and out-of-order computing,” in _2022 IEEE International Solid-State Circuits Conference (ISSCC)_ , vol. 65. IEEE, 2022, pp. 1–3.
  * [5] F. Tu, Z. Wu, Y. Wang, L. Liang, L. Liu, Y. Ding, L. Liu, S. Wei, Y. Xie, and S. Yin, “A 28nm 15.59 $\mu$j/token full-digital bitline-transpose cim-based sparse transformer accelerator with pipeline/parallel reconfigurable modes,” in _2022 IEEE International Solid-State Circuits Conference (ISSCC)_ , vol. 65. IEEE, 2022, pp. 466–468.
  * [6] G. İslamoğlu, M. Scherer, G. Paulin, T. Fischer, V. J. B. Jung, A. Garofalo, and L. Benini, “Ita: An energy-efficient attention and softmax accelerator for quantized transformers,” 2023.
  * [7] S.-C. Kao, S. Subramanian, G. Agrawal, A. Yazdanbakhsh, and T. Krishna, “Flat: An optimized dataflow for mitigating attention bottlenecks,” in _Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2_ , 2023, pp. 295–310.
  * [8] Y. Sheng, L. Zheng, B. Yuan, Z. Li, M. Ryabinin, D. Y. Fu, Z. Xie, B. Chen, C. Barrett, J. E. Gonzalez _et al._ , “High-throughput generative inference of large language models with a single gpu,” _arXiv preprint arXiv:2303.06865_ , 2023.
  * [9] S. Colleman, P. Zhu, W. Sun, and M. Verhelst, “Optimizing accelerator configurability for mobile transformer networks,” in _2022 IEEE 4th International Conference on Artificial Intelligence Circuits and Systems (AICAS)_. IEEE, 2022, pp. 142–145.
  * [10] A. Parashar, P. Raina, Y. S. Shao, Y.-H. Chen, V. A. Ying, A. Mukkara, R. Venkatesan, B. Khailany, S. W. Keckler, and J. Emer, “Timeloop: A systematic approach to dnn accelerator evaluation,” in _2019 IEEE international symposium on performance analysis of systems and software (ISPASS)_. IEEE, 2019, pp. 304–315.
  * [11] L. Mei, P. Houshmand, V. Jain, S. Giraldo, and M. Verhelst, “Zigzag: Enlarging joint architecture-mapping design space exploration for dnn accelerators,” _IEEE Transactions on Computers_ , vol. 70, no. 8, pp. 1160–1174, 2021.
  * [12] H. Kwon, L. Lai, M. Pellauer, T. Krishna, Y.-H. Chen, and V. Chandra, “Heterogeneous dataflow accelerators for multi-dnn workloads,” in _2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA)_. IEEE, 2021, pp. 71–83.
  * [13] L. Mei, K. Goetschalckx, A. Symons, and M. Verhelst, “Defines: Enabling fast exploration of the depth-first scheduling space for dnn accelerators through analytical modeling,” _arXiv preprint arXiv:2212.05344_ , 2022.
  * [14] K. Goetschalckx and M. Verhelst, “Breaking high-resolution cnn bandwidth barriers with enhanced depth-first execution,” _IEEE Journal on Emerging and Selected Topics in Circuits and Systems_ , vol. 9, no. 2, pp. 323–331, 2019.
  * [15] A. Symons, L. Mei, and M. Verhelst, “Loma: Fast auto-scheduling on dnn accelerators through loop-order-based memory allocation,” in _2021 IEEE 3rd International Conference on Artificial Intelligence Circuits and Systems (AICAS)_. IEEE, 2021, pp. 1–4.
  * [16] Q. Hu, F. Jiang, C. Mei, and R. Shen, “Cct: a cross-concat and temporal neural network for multi-label action unit detection,” in _2018 IEEE International Conference on Multimedia and Expo (ICME)_. IEEE, 2018, pp. 1–6.
  * [17] E. Flamand, D. Rossi, F. Conti, I. Loi, A. Pullini, F. Rotenberg, and L. Benini, “Gap-8: A risc-v soc for ai at the edge of the iot,” in _2018 IEEE 29th International Conference on Application-specific Systems, Architectures and Processors (ASAP)_. IEEE, 2018, pp. 1–4.
  * [18] S. Kim, A. Gholami, Z. Yao, M. W. Mahoney, and K. Keutzer, “I-bert: Integer-only bert quantization,” in _International conference on machine learning_. PMLR, 2021, pp. 5506–5518.
  * [19] A. Burrello, M. Scherer, M. Zanghieri, F. Conti, and L. Benini, “A microcontroller is all you need: Enabling transformer execution on low-power iot endnodes,” in _2021 IEEE International Conference on Omni-Layer Intelligent Systems (COINS)_ , 2021, pp. 1–6.


* (97) Z. Zhang, H. Ning, F. Shi, F. Farha, Y. Xu, J. Xu, F. Zhang, and K.-K. R. Choo, “Artificial intelligence in cyber security: research advances, challenges, and opportunities,” _Artificial Intelligence Review_ , pp. 1–25, 2021.
  * (98) T. C. Truong, I. Zelinka, J. Plucar, M. Čandík, and V. Šulc, “Artificial intelligence and cybersecurity: Past, presence, and future,” in _Artificial intelligence and evolutionary computations in engineering systems_. Springer, 2020, pp. 351–363.
  * (99) S. Samtani, M. Kantarcioglu, and H. Chen, “Trailblazing the artificial intelligence for cybersecurity discipline: A multi-disciplinary research roadmap,” pp. 1–19, 2020.
  * (100) K. B. Kelarestaghi, K. Heaslip, M. Khalilikhah, A. Fuentes, and V. Fessmann, “Intelligent transportation system security: hacked message signs,” _SAE International Journal of Transportation Cybersecurity and Privacy_ , vol. 1, no. 11-01-02-0004, pp. 75–90, 2018.
  * (101) J. CAIRE, “Human factors in cybersecurity for transportation systems,” _WIT Transactions on The Built Environment_ , vol. 176, pp. 405–414, 2017\.
  * (102) R. O. Andrade, S. G. Yoo, L. Tello-Oquendo, and I. Ortiz-Garcés, “A comprehensive study of the iot cybersecurity in smart cities,” _IEEE Access_ , vol. 8, pp. 228 922–228 941, 2020.
  * (103) M. Abdel-Basset, N. Moustafa, H. Hawash, I. Razzak, K. M. Sallam, and O. M. Elkomy, “Federated intrusion detection in blockchain-based smart transportation systems,” _IEEE Transactions on Intelligent Transportation Systems_ , 2021.
  * (104) M. Kashef, A. Visvizi, and O. Troisi, “Smart city as a smart service system: Human-computer interaction and smart city surveillance systems,” _Computers in Human Behavior_ , vol. 124, p. 106923, 2021.
  * (105) R. Khan, P. Kumar, D. N. K. Jayakody, and M. Liyanage, “A survey on security and privacy of 5g technologies: Potential solutions, recent advancements, and future directions,” _IEEE Communications Surveys & Tutorials_, vol. 22, no. 1, pp. 196–248, 2019.
  * (106) I. Ahmad, T. Kumar, M. Liyanage, J. Okwuibe, M. Ylianttila, and A. Gurtov, “Overview of 5g security challenges and solutions,” _IEEE Communications Standards Magazine_ , vol. 2, no. 1, pp. 36–43, 2018.
  * (107) M. Kuzlu, C. Fair, and O. Guler, “Role of artificial intelligence in the internet of things (iot) cybersecurity,” _Discover Internet of things_ , vol. 1, no. 1, pp. 1–14, 2021.
  * (108) I. H. Sarker, M. H. Furhad, and R. Nowrozy, “Ai-driven cybersecurity: an overview, security intelligence modeling and research directions,” _SN Computer Science_ , vol. 2, no. 3, pp. 1–18, 2021.
  * (109) A. Rawal, J. Mccoy, D. B. Rawat, B. Sadler, and R. Amant, “Recent advances in trustworthy explainable artificial intelligence: Status, challenges and perspectives,” _IEEE Transactions on Artificial Intelligence_ , vol. 1, no. 01, pp. 1–1, 2021.
  * (110) I. Ahmed, G. Jeon, and F. Piccialli, “From artificial intelligence to explainable artificial intelligence in industry 4.0: A survey on what, how, and where,” _IEEE Transactions on Industrial Informatics_ , 2022.
  * (111) D.-R. N. Arrieta A.B and, D. S. J., B. A., S. B. A. Tabik, S. G.-L. S. Garcia, D. Molina, B. R., and C. R., “Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai,” _Information Fusion_ , vol. 58, pp. 82–115, 2020.