# Chiral Phonon Induced Spin-Polarization

J. Fransson<EMAIL_ADDRESS>Department of Physics and Astronomy,
Box 516, 751 20, Uppsala University, Uppsala, Sweden

###### Abstract

The current understanding of chirality suggests the existence of a connection
between structure and angular momentum, including spin. This is particularly
emphasised in the chiral induced spin selectivity effect, where chiral
structures act as spin filters. However, the recent discovery of chiral
phonons have demonstrated that phonons too may carry angular momentum which
also can be regarded as magnetic moments which add to the total moment. Here,
it is shown that chiral phonons may induce a non-trivial spin-texture in an
otherwise non-magnetic electronic structure. By considering a set-up in which
electrons and phonons are interfaced with each other, it is shown that chiral
phonons may transfer its angular momentum into the electron reservoir which,
thereby, becomes spin-polarized. It is, moreover, shown that an equivalent
mechanism does not exist whenever the electrons are interfaced with achiral
phonons.

Phonons represent the collective nuclear motion within a structure. As such,
phonons are the quantum mechanically defined quantities with which the
mechanical degrees of freedom are effectively incorporated into the framework
of the general quantum field theory. While traditionally being regarded as a
quantity carrying linear momentum, it is only recently that angular momentum
of phonons have been considered. An incomplete list of important results are
phonon Hall, phonon spin Hall, and phonon angular momentum Hall effects [1, 2,
3, 4, 5, 6, 7], phonon contribution to spin-relaxation processes [8, 9, 10,
11] and the Einstein-de Haas effect [12, 10, 13, 14], phononically mediated
spin-spin interactions [15, 16], temperature gradient induced phonon angular
momentum [17], and optically activated chiral phono-magnetic effects [18, 19,
20]. Experimentally, progress has been made in observations of chiral phonons
[21, 22, 23, 24, 25], and phonon induced magneto-thermal properties [26, 27,
28].

The existence of phonon angular momentum opens up the possibility to couple
the electronic spin-degrees of freedom with the mechanical. It is well-
established that spin and nuclear motion are coupled directly through, e.g.,
the Elliot-Yafet mechanism [29, 30, 31, 32, 33, 34], but also indirectly via
the electronic structure [15], and it has been demonstrated that such coupling
opens for a viable explanation of the chiral induced spin selectivity effect
[35, 36, 34, 37].

Hitherto, however, the angular momentum of phonons and electrons have been
considered as separate from one another, where the magnetic moment associated
with the chiral phonons have been studied in its own right. While this is
definitely pertinent, the effects angular momentum transfer between phonons
and electrons has been discussed in a semi-classical model [38], in which a
spin-dependent coupling between phonons and electrons is assumed and it is
demonstrated that chiral phonons may give rise to the spin-Seebck effect,
something that was also recently observed in experiments [39]. Nonetheless,
the mechanism that enables the angular momentum transfer is yet to be
discussed. The purpose with this Letter is to present a coherent theory that
ties the existence of phononic angular momentum to a broken electronic spin-
degeneracy. It is shown that the mechanism is provided through a vibronically
assisted spin-orbit interaction and while this coupling is always present,
chiral phonons are required for inducing a spin-polarization.

Chirality is a geometrical property where the structure lacks both inversion
and reflection symmetries. It is easy to demonstrate that chiral phonons must
carry a non-vanishing angular momentum ${\bf J}_{\text{ph}}$. This can be seen
directly from the definition

$\displaystyle{\bf J}_{\text{ph}}(t)=$ $\displaystyle\int{\bf Q}({\bf
r},t)\times\dot{\bf Q}({\bf r},t)d{\bf r},$ (1)

where ${\bf Q}$ is the nuclear displacement, which is connected to phonons
through the relation ${\bf Q}({\bf
r},t)=\sum_{q}l_{q}{\bm{\epsilon}}_{q}Q_{p}(t)e^{i{\bf q}\cdot{\bf r}}$, where
$Q_{q}=b_{{\bf q}\mu}+b^{\dagger}_{\bar{\bf q}\mu}$, is the quantum phonon
displacement operator. Here, $b_{q}$ and $b^{\dagger}_{q}$ denote the phonon
destruction and creation operators, respectively, with $q=({\bf q},\mu)$
($\bar{q}=(\bar{\bf q},\mu)=(-{\bf q},\mu)$) comprising the wave vector ${\bf
q}$ and normal mode $\mu$, and $l_{q}=\sqrt{\hslash/2\rho v\omega_{q}}$ which
defines a length scale in terms of the phonon energy $\omega_{q}$, density
$\rho$, and volume $v$, whereas ${\bm{\epsilon}}_{q}$ is the displacement
polarization vector.

The expectation value of the angular momentum, hence, assumes the form

$\displaystyle\langle{{\bf J}_{\text{ph}}}\rangle(t)=$
$\displaystyle\lim_{t^{\prime}\rightarrow
t}i{\partial_{t}}\sum_{pq}l_{p}l_{q}{\bm{\epsilon}}_{p}\times{\bm{\epsilon}}_{\bar{q}}D^{>}_{pq}(t^{\prime},t)\int
e^{i({\bf p}-{\bf q})\cdot{\bf r}}d{\bf r},$ (2)

where
$D^{>}_{pq}(t^{\prime},t)=(-i)\langle{Q_{p}(t^{\prime})Q_{\bar{q}}(t)}\rangle$
defines the correlations between the phonons $Q_{p}(t^{\prime})$ and
$Q_{\bar{p}}(t)$. The expression in Eq. (2) shows that a finite phonon angular
momentum requires non-collinear polarisations ${\bm{\epsilon}}_{p}$ and
${\bm{\epsilon}}_{\bar{q}}$.

Non-collinear polarizations can on the one hand be achieved when there is a
mechanism that mixes the phonon modes $p$ and $q$. Such mode mixing may
originate from, e.g., anharmonic effects or scattering off defects acting upon
the otherwise orthogonal modes defined in the harmonic approximation.

In the harmonic approximation, the phonons can, up to a constant, be
summarised the Hamiltonian form as ${\cal
H}_{\text{ph}}=\sum_{q}\omega_{q}b^{\dagger}_{q}b_{q}$. In this form, it is
assumed that the phonon modes do not mix which, therefore, leads to a
vanishing angular momentum. However, the introduction of a component of the
kind $W_{{\bf q}\mu\nu}b^{\dagger}_{{\bf q}\mu}b_{{\bf q}\nu}$ provides a mode
mixing, which can be understood as chirality.

To see this, consider the phonon spinor $\Phi_{\bf q}=\\{b_{{\bf
q}\mu_{i}}\\}_{i=1}^{N}$, for $N$ modes, which enables us to write the phonon
model as

$\displaystyle{\cal H}_{\text{ph}}=$ $\displaystyle\sum_{\bf
q}\Phi^{\dagger}_{\bf q}{\bm{\omega}}_{\bf q}\Phi_{\bf q}.$ (3)

Written like this, the phonon spectrum is defined through the matrix
${\bm{\omega}}_{\bf q}=\omega_{0{\bf q}}\tau^{0}+{\bm{\omega}}_{1{\bf
q}}\cdot{\bm{\tau}}$, where $\omega_{0}$ and ${\bm{\omega}}_{1}$ represent the
mode conservative and mode mixing components, respectively, whereas $\tau^{0}$
and ${\bm{\tau}}$ are the $N$-dimensional identity and vector of spin
matrices. While in this model it is assumed that the mixing only takes place
between modes $\mu_{i}$ and $\mu_{j}$ with the same momentum ${\bf q}$, it is
straight forward to generalise the model to also include mixing between
different momenta.

Considering a structure with two modes, such that ${\bm{\tau}}$ reduces to the
Pauli matrices, the phonon Green function ${\bf D}_{\bf
q}(z)=\langle{\mathop{\langle\Phi_{\bf q}|\Phi_{\bar{\bf
q}}\rangle}}\rangle(z)$, can be written as

$\displaystyle{\bf D}_{\bf p}(z)=$ $\displaystyle 2{\bm{\omega}}_{\bf
q}\frac{z^{2}-\omega_{0{\bf q}}^{2}-\omega_{1{\bf q}}^{2}+2\omega_{0{\bf
q}}{\bm{\omega}}_{1{\bf q}}\cdot{\bm{\tau}}}{(z^{2}-\omega_{0{\bf
q}}^{2}-\omega_{1{\bf q}}^{2})^{2}-4\omega_{0{\bf q}}^{2}\omega_{1{\bf
q}}^{2}}$ (4a) $\displaystyle=$ $\displaystyle\frac{1}{2}\sum_{s=\pm
1}\frac{2\omega_{{\bf q}s}}{z^{2}-\omega_{{\bf
q}s}^{2}}\Bigl{(}\tau^{0}+s\hat{\bm{\omega}}_{1{\bf
q}}\cdot{\bm{\tau}}\Bigr{)},$ (4b)

where $\omega_{{\bf q}s}=\omega_{0{\bf q}}+s\omega_{1{\bf q}}$, $\omega_{1{\bf
q}}=|{\bm{\omega}}_{1{\bf q}}|$ and $\hat{\bm{\omega}}_{1{\bf
q}}={\bm{\omega}}_{1{\bf q}}/\omega_{1{\bf q}}$. The form of this propagator
written in Eq. (4b) explicitly describes two modes with opposite helicity, or,
chirality.

On the other hand, the polarization for a chiral mode is neither reflection
nor inversion symmetric, where the latter condition leads to that
${\bm{\epsilon}}^{*}_{{\bf q}\mu}={\bm{\epsilon}}_{\bar{\bf
q}\mu}\neq{\bm{\epsilon}}_{{\bf q}\mu}$, since equality in the last relation
requires inversion symmetry. For, e.g., a helical structure with transversal
and longitudinal lattice parameters $a$ and $c$, respectively, the
polarization may be expressed as ${\bm{\epsilon}}_{\bf
q}=(a\cos\phi_{q},a\sin\phi_{q},\mathcent\phi_{q})/d(\phi_{q})$, where
$\mathcent=c/2\pi$ and $d(\phi_{q})=\sqrt{a^{2}+\mathcent^{2}\phi_{q}^{2}}$,
which displays a variation of the mode that depends on the azimuthal angle
$\phi_{q}$. Hence, for such a mode, the vector product ${\bm{\epsilon}}_{\bf
q}\times{\bm{\epsilon}}_{\bar{\bf
q}}=(-a\mathcent\phi_{q}\sin\phi_{q},a\mathcent\phi_{q}\cos\phi_{q},-a^{2}\sin
2\phi_{q})/d^{2}(\phi_{q})$, which suggests that a free chiral phonon mode,
for which $D^{>}_{\bf q}(t,t^{\prime})=(-i)[n_{B}(\omega_{\bf
q})e^{-i\omega_{\bf q}(t-t^{\prime})}-n_{B}(-\omega_{\bf q})e^{i\omega_{\bf
q}(t-t^{\prime})}]$, where $n_{B}(\omega)$ is the Bose-Einstein distribution
function, carry the non-vanishing angular momentum

$\displaystyle\langle{{\bf J}_{\text{ph}}}\rangle=$
$\displaystyle\frac{\omega_{\bf q}l_{\bf q}^{2}}{\pi
d^{2}(\phi_{q})}\begin{pmatrix}-a\mathcent\phi_{q}\sin\phi_{q}\\\
a\mathcent\phi_{q}\cos\phi_{q}\\\ -a^{2}\pi\sin 2\phi_{q}\end{pmatrix}.$ (5)

It is important to notice that the off-diagonal components of the phonon
propagator, Eq. (4b), carry the phase factors $e^{\pm i\phi_{p}}$, where
$\phi_{p}$ defines the azimuthal angle of the momentum vector ${\bf p}$. This
phase dependence reoccurs also in the helical polarization vector and is an
essential feature of chirality. Due to this phase dependence, the phonons
necessarily carry angular momentum.

The objective in this Letter, is to show that the phononic angular momentum
can be transferred to the electronic subsystem and, hence, induce a spin-
polarization. For this sake, consider the coupling between electrons and
phonons, which generally can be written as [34]

$\displaystyle{\cal H}_{\text{e-ph}}=$ $\displaystyle\sum_{{\bf
k}q}\psi^{\dagger}_{{\bf p}+{\bf k}}{\bf U}_{{\bf k}q}\psi_{\bf k}Q_{q},$ (6)

where the coupling matrix ${\bf U}_{{\bf k}q}=U_{{\bf k}q}\sigma^{0}+{\bf
J}_{{\bf k}q}\cdot{\bm{\sigma}}$ accounts for a spin-conservative electron-
phonon coupling, $U_{{\bf k}q}$, and an electron-phonon assisted spin-orbit
interaction, ${\bf J}_{{\bf k}q}$.

The spin-polarisation $\langle{{\bf M}_{\bf k}}\rangle$ of the electrons are
calculated using the identity $\langle{{\bf M}_{\bf k}}\rangle=(-i){\rm
sp}{\bm{\sigma}}\int{\bf G}_{{\bf k}{\bf k}}^{<}(\omega)d\omega/4\pi$, where
${\bf G}^{<}_{{\bf k}{\bf k}^{\prime}}(\omega)$ defines the lesser form of the
general single electron Green function ${\bf G}_{{\bf k}{\bf
k}^{\prime}}(z)=\langle{\mathop{\langle\psi_{\bf k}|\psi_{{\bf
k}^{\prime}}^{\dagger}\rangle}}\rangle(z)$, whereas ${\rm sp}$ denotes the
trace over spin 1/2 space. To the second order (Hartree-Fock) approximation in
the electron-phonon coupling, this Green function can be calculated from the
Dyson equation

$\displaystyle{\bf G}_{{\bf k}{\bf k}^{\prime}}(z)=$ $\displaystyle\delta({\bf
k}-{\bf k}^{\prime}){\bf g}_{\bf k}(z)+{\bf g}_{\bf
k}(z)\sum_{{\bm{\kappa}}}{\bm{\Sigma}}^{\text{(HF)}}_{{\bf
k}{\bm{\kappa}}}(z){\bf G}_{{\bm{\kappa}}{\bf k}^{\prime}}(z),$ (7a)
$\displaystyle{\bm{\Sigma}}^{\text{(HF)}}_{{\bf k}{\bf k}^{\prime}}(z)=$
$\displaystyle(-i)\delta({\bf k}-{\bf
k}^{\prime})\sum_{qq^{\prime}}D_{qq^{\prime}}{\bf U}_{{\bf k}q}{\rm
sp}\int{\bf G}^{<}_{{\bf k}{\bf k}}(\omega){\bf U}_{{\bf
k}\bar{q}^{\prime}}\frac{d\omega}{2\pi}$
$\displaystyle-\frac{1}{\beta}\sum_{z_{\nu}qq^{\prime}}{\bf U}_{{\bf k}q}{\bf
G}_{{\bf k}{\bf k}^{\prime}}(z+z_{\nu}){\bf U}_{{\bf
k}^{\prime}\bar{q}^{\prime}}D_{qq^{\prime}}(z_{\nu}),$ (7b)

where $D_{qq^{\prime}}=\int\delta(\omega)D^{a}_{qq^{\prime}}(\omega)d\omega$.

While this equation should be self-consistently solved, for an analysis of the
induced spin symmetries, it is sufficient to replace the electronic Green
functions in these expressions with its unperturbed form ${\bf g}_{\bf
k}(z)=\sigma^{0}g_{\bf k}(z)$, where $g_{\bf k}(z)=1/(z-\varepsilon_{{\bf
k}})$ and ${\bf g}^{<}_{\bf
k}(\omega)=2\pi\sigma^{0}f(\omega)\delta(\omega-\varepsilon_{{\bf k}})$,
whereas $f(\omega)$ is the Fermi-Dirac distribution function. These
replacements lead to the simplified self-energy

$\displaystyle{\bm{\Sigma}}_{\bf k}^{\text{(HF)}}(z)=$ $\displaystyle
2f(\varepsilon_{{\bf k}})\sum_{qq^{\prime}}\delta({\bf q})U_{{\bf
k}q}D_{qq^{\prime}}{\bf U}_{{\bf k}\bar{q}^{\prime}}$
$\displaystyle-\frac{1}{\beta}\sum_{\nu qq^{\prime}}g_{{\bf k}-{\bf
q}}(z-z_{\nu}){\bf U}_{{\bf k}q}D_{qq^{\prime}}(z_{\nu}){\bf U}_{{\bf
k}\bar{q}^{\prime}}.$ (8)

The Hartree (first) contribution is to lowest order linear in the component
${\bf J}_{{\bf k}q}\cdot{\bm{\sigma}}$ which, hence, shows that the coupling
between the electrons and phonons may break time-reversal symmetry. However,
this contribution vanishes in this approximation since there are no phonons at
${\bf p}=0$. By contrast, the exchange (second) contribution does not only
open for breaking the time-reversal symmetry, since in general

$\displaystyle{\bf U}_{{\bf k}p}{\bf U}_{{\bf k}q}=$ $\displaystyle U_{{\bf
k}q}U_{{\bf k}q^{\prime}}+{\bf J}_{{\bf k}q}\cdot{\bf J}_{{\bf k}q^{\prime}}$
$\displaystyle+\Big{(}U_{{\bf k}q}{\bf J}_{{\bf k}q^{\prime}}+{\bf J}_{{\bf
k}q}U_{{\bf k}q^{\prime}}+i{\bf J}_{{\bf k}q}\times{\bf J}_{{\bf
k}q^{\prime}}\Bigr{)}\cdot{\bm{\sigma}},$ (9)

but it also provides a correlation between the electronic and phononic degrees
of freedom.

However, before showing that chiral phonon leads to a breaking of the
electronic spin-degeneracy, it is pertinent to discuss the origin of the
phonon induced spin-polarization and spin-flip processes which are enabled by
${\bf J}_{{\bf k}q}$. For this purpose, it is instructive to consider the
spin-orbit coupling, which can be written as [40]

$\displaystyle H_{\text{SOC}}=$ $\displaystyle\frac{\xi}{2}\Bigl{[}{\bf
E}\times{\bf p}-{\bf p}\times{\bf E}\Bigr{]}\cdot{\bm{\sigma}},$ (10)

where ${\bf E}$ is the total electric field acting on the electrons, whereas
$\xi=1/4c^{2}$ in atomic units. By introducing the field operator $\psi({\bf
r})=\int\psi_{\bf k}e^{i{\bf k}\cdot{\bf r}}d{\bf k}/\Omega$ and its Hermitian
conjugate, and assuming time-independent magnetic fields ${\bf B}$, such that
$\nabla\times{\bf E}=-{\partial_{t}}{\bf B}=0$, the spin-orbit coupling
contribution can in the second quantization be written

$\displaystyle{\cal H}_{\text{SOC}}=$ $\displaystyle{\cal
E}_{S}-i\xi\int\psi_{\bf k}^{\dagger}({\bf k}\times{\bf
k}^{\prime})\cdot{\bm{\sigma}}V({\bf r})e^{-i({\bf k}-{\bf
k}^{\prime})\cdot{\bf r}}\psi_{{\bf k}^{\prime}}d{\bf r}\frac{d{\bf
k}}{\Omega}\frac{d{\bf k}^{\prime}}{\Omega},$ (11)

where ${\cal E}_{S}$ is the boundary surface contribution, and where the
electric field has been identified with a the electronic confinement potential
$V({\bf r}$) using the relation ${\bf E}=-\nabla V({\bf r})$.

Here, ${\cal E}_{S}=\int\psi_{\bf k}{\cal S}_{{\bf k}{\bf
k}^{\prime}}\psi_{{\bf k}^{\prime}}d{\bf k}d{\bf k}^{\prime}/\Omega^{2}$,
where ${\cal S}_{{\bf k}{\bf k}^{\prime}}$ is given by

$\displaystyle{\cal S}_{{\bf k}{\bf k}^{\prime}}=$ $\displaystyle\xi\int_{\cal
S}({\bf k}\times\hat{\bf n})\cdot{\bm{\sigma}}V({\bf r})e^{-i({\bf k}-{\bf
k}^{\prime})\cdot{\bf r}}d{\cal S},$ (12)

where $\hat{\bf n}$ denotes the outward normal on the boundary surface ${\cal
S}$. This contribution can be identified as the Rashba spin-orbit coupling.

The confinement potential $V({\bf r})=\int V({\bf r}-{\bf R})d{\bf R}$, where
${\bf R}={\bf R}_{0}+{\bf Q}({\bf R})$ denotes the coordinate for the nuclei
in terms of the equilibrium position ${\bf R}_{0}$ and displacement ${\bf
Q}({\bf R})$. For small displacements, the potential can be expanded around
the equilibrium positions $V({\bf r}-{\bf R})\approx V({\bf r}-{\bf
R}_{0})-{\bf Q}\cdot[\nabla_{\bf R}V({\bf r}-{\bf R})]_{{\bf R}\rightarrow{\bf
R}_{0}}$, where the first contribution along with the surface integral provide
the static spin-orbit interaction.

The spin-orbit interaction with the lowest order coupling to the nuclear
displacement is given by the Hamiltonian

$\displaystyle{\cal H}^{(1)}_{\text{SOC}}=$
$\displaystyle(-i)\hslash\xi\int\psi^{\dagger}_{\bf k}\Bigl{[}({\bf
k}\times{\bf k}^{\prime})\cdot{\bm{\sigma}}\Bigr{]}\Bigl{[}{\bf
Q}\cdot[\nabla_{\bf R}V({\bf r}-{\bf R})]_{{\bf R}\rightarrow{\bf
R}_{0}}\Bigr{]}$ $\displaystyle\times e^{-i({\bf k}-{\bf k}^{\prime})\cdot{\bf
r}}\psi_{{\bf k}^{\prime}}d{\bf R}_{0}d{\bf r}\frac{d{\bf
k}}{\Omega}\frac{d{\bf k}^{\prime}}{\Omega}.$ (13)

Using $\nabla_{\bf R}V({\bf r}-{\bf R})=(-i)\int{\bf q}V({\bf q})e^{i{\bf
q}\cdot({\bf r}-{\bf R})}d{\bf q}/\Omega$, leads to that the phonon assisted
spin-orbit interaction can be written

$\displaystyle{\cal H}^{(1)}_{\text{SOC}}=$
$\displaystyle\int\psi^{\dagger}_{{\bf k}+{\bf q}}{\bf J}_{{\bf
k}q}\cdot{\bm{\sigma}}Q_{q}\psi_{\bf k}\frac{d{\bf q}}{\Omega}\frac{d{\bf
k}}{\Omega},$ (14)

where ${\bf J}_{{\bf k}q}=i\xi U_{{\bf k}q}({\bf k}\times{\bf q})$, where
$U_{{\bf k}q}=U_{q}=il_{q}{\bm{\epsilon}}_{q}\cdot{\bf q}V({\bf q})$ is the
spin-conservative electron-phonon interaction. This gives the total electron-
phonon coupling ${\bf U}_{{\bf k}q}=U_{q}\sigma^{0}+{\bf J}_{{\bf
k}q}\cdot{\bm{\sigma}}$.

From an order of magnitude estimation, it can be deduced that the ratio
exchange diagram originating from the spin-dependent and spin-conservative
processes, that is, those which are proportional to $U_{{\bf k}q}{\bf J}_{{\bf
k}\bar{q}}$ and $U_{{\bf k}q}U_{{\bf k}\bar{q}}$, respectively, is roughly
$2\xi q_{c}/3$ ($\times{\bf k}$), where $q_{c}$ is the momentum corresponding
to the phononic high energy cut-off.

More importantly, however, is that while $J_{{\bf k}q}$ is linear in the spin-
orbit coupling parameter $\xi$, that is, in the same order as the static spin-
orbit interaction contribution ${\cal H}_{\text{SOC}}^{(0)}$, the phonon
assisted contribution may potentially have a deeply profound influence on the
electronic structure. The reason is that the coupling to phonons by all means
generates many-body effects, that is, electronic correlations, which may lead
to spontaneous symmetry breaking of the electrons.

To see this, consider the influence from the chiral phonons on the magnetic
moment $\langle{{\bf M}}\rangle=\int\langle{{\bf M}_{\bf k}}\rangle d{\bf
k}/\Omega$. In the first order approximation of the lesser Green function,
${\bf G}^{<}_{\bf k}\approx{\bf g}^{<}_{\bf k}+{\bf g}^{r}_{\bf
k}{\bm{\Sigma}}^{\text{(HF)},<}_{\bf k}{\bf g}^{a}_{\bf k}$, the ${\bf
k}$-resolved spin-moment reduces to the expression ($\Omega=(2\pi)^{3}$)

$\displaystyle\langle{{\bf M}_{\bf k}}\rangle=$ $\displaystyle 4\xi\int
f(\varepsilon_{{\bf k}})l_{q}^{2}V^{2}({\bf q}){\bf K}_{{\bf k}{\bf
q}}\frac{d{\bf q}}{(2\pi)^{3}},$ (15a) $\displaystyle{\bf K}_{{\bf k}{\bf
q}}=$ $\displaystyle|{\bm{\epsilon}}_{q}\cdot{\bf q}|^{2}({\bf k}\times{\bf
q})\sum_{ss^{\prime}=\pm 1}s^{\prime}n_{B}(s^{\prime}\omega_{{\bf
q}s})\frac{1+s\hat{\omega}_{\perp{\bf q}}\cos\phi_{q}}{(\varepsilon_{{\bf
q}}+s^{\prime}\omega_{{\bf q}s}-{\bf k}\cdot{\bf q})^{2}},$ (15b)

where $\hat{\omega}_{\perp{\bf q}}=|(\omega_{1{\bf q}}^{(x)},\omega_{1{\bf
q}}^{(y)},0)|/\omega_{1{\bf q}}$.

Figure 1:  Examples of $\tilde{{\bf K}}_{{\bf k}{\bf q}}=(\tilde{K}_{{\bf
k}{\bf q}}^{(x)},\tilde{K}_{{\bf k}{\bf q}}^{(y)},\tilde{K}_{{\bf k}{\bf
q}}^{(z)})$ as function of the polar (horizontal) and azimuthal (vertical)
angles $\theta_{k}$ and $\phi_{k}$, for (a)–(c) achiral and (d)–(f) chiral
phonons. Parameters used are (a.u.) $k=\sqrt{2m_{e}\varepsilon}/\hslash$,
$\varepsilon=0.15$ eV, $\omega_{{\bf q}s}=(\omega_{0}+s\omega_{1})q$,
$\omega_{0}q=0.04$ eV, $\omega_{0}=1$ , $\omega_{1}=\omega_{0}/10$, and
${\bm{\epsilon}}=(0,0,1)$ at $T=300$ K.

By integrating out the phases $\phi_{q}$ and $\theta_{q}$, defining the spin-
texture $\tilde{{\bf K}}_{{\bf k}{\bf q}}=\int_{0}^{\pi}\int_{0}^{2\pi}{\bf
K}_{{\bf k}{\bf q}}d\phi_{q}\sin\theta_{q}d\theta_{q}/(2\pi)^{2}$, under the
assumption that $V({\bf q})=V(|{\bf q}|)$, the resulting kernel is a regular
function of the electronic momentum ${\bf k}$. This is shown in Fig. 1, where
$\tilde{{\bf K}}_{{\bf k}{\bf q}}$ is plotted as function of the polar and
azimuthal angles $\theta_{k}$ and $\phi_{k}$, respectively, for (a)–(c)
achiral and (d)–(f) chiral phonons.

Both for achiral and chiral phonons, there is a vibrationally assisted spin-
orbit coupling induced in the electronic structure, which follows on general
grounds and has been discussed in different context, e.g., Ref. [41]. The
absence of a $z$-component for achiral phonons in the example given in Fig. 1
can be understood through the following observation.

The denominator in ${\bf K}_{{\bf k}{\bf q}}$, see Eq. (15b), depends on the
difference between the phononic and electronic phases $\phi_{q}$ and
$\phi_{k}$, respectively, as $[A_{{\bf k}{\bf q}ss^{\prime}}-B_{{\bf k}{\bf
q}}\cos(\phi_{q}-\phi_{k})]^{2}$, where $A_{{\bf k}{\bf
q}ss^{\prime}}=q^{2}/2+s^{\prime}\omega_{{\bf
q}s}-kp\cos\theta_{k}\cos\theta_{q}$ and $B_{{\bf k}{\bf
q}}=kp\sin\theta_{k}\sin\theta_{q}$. This is an even function of the phase
difference $\phi_{q}-\phi_{k}$ on the interval $[0,2\pi]$, while both
$|{\bm{\epsilon}}_{q}\cdot{\bf q}|^{2}$ and ${\bf k}\times{\bf q}$ can be
written as sum of even and odd functions of $\phi_{q}-\phi_{k}$. For a the
polarization ${\bm{\epsilon}}_{q}=(0,0,1)$, the numerator of the longitudinal
spin-texture $K_{{\bf k}{\bf q}}^{(z)}$ is proportional to
$\sin(\phi_{q}-\phi_{k})$, which is odd on the interval $[0,2\pi]$. Hence,
$\int_{0}^{2\pi}K_{{\bf k}{\bf q}}^{(z)}d\phi_{q}/2\pi=0$. For chiral phonons,
$\hat{\omega}_{\perp{\bf q}}\neq 0$, which leads to that the numerator of
$K_{{\bf k}{\bf q}}^{(z)}$ also contains a term which is proportional to
$\sin(\phi_{q}-\phi_{k})\cos\phi_{q}=\sin(\phi_{q}-\phi_{k})[\cos\phi_{k}\sin(\phi_{q}-\phi_{k})+\sin\phi_{k}\cos(\phi_{q}-\phi_{k})]$,
where the first (second) contribution is an even (odd) function of
$\phi_{q}-\phi_{k}$. Therefore, $\int_{0}^{2\pi}K_{{\bf k}{\bf
q}}^{(z)}d\phi_{q}/2\pi\neq 0$ for chiral phonons.

The above discussion pertains to longitudinally polarized phonons. For
transversely ($xy$-plane) polarized phonons, the vibrationally assisted spin-
orbit coupling generates a non-vanishing spin-texture for all $\tilde{K}_{{\bf
k}{\bf q}}^{(i)}$, $i=x,y,z$. However, the longitudinal spin-texture
$\tilde{K}_{{\bf k}{\bf q}}^{(z)}$ induced for achiral phonons polarized along
some transverse axis, e.g., $\hat{\bf x}$, exactly cancels the spin-texture
induced for phonons polarized along its orthogonal transverse axis, e.g.,
$\hat{\bf y}$. Chiral phonons, on the other hand, breaks this symmetry such
that there remains a net longitudinal spin-texture after summing over all
phonon modes.

The conclusion is, that a vibrationally assisted spin-orbit coupling exists
for any type of phonons which, in turn, gives rise to a transversal spin-
texture. However, a net longitudinal spin-texture only arises in presence of
chiral phonons. Nevertheless, as is illustrated in Fig. 1, no magnetic moment
is generated from this spin-texture since its angular dependencies integrate
to zero.

A manifestation of the chiral phonon induced spin-texture is the presence of
spin-currents. The spin-current can, in the current approximation, be written
as $\mathbb{J}({\bf r})=(-i){\rm sp}{\bm{\sigma}}\int{\bf k}{\bf G}^{<}_{\bf
k}(\omega)d{\bf k}d\omega/4\pi\Omega=\int{\bf k}\langle{{\bf M}_{\bf
k}}\rangle d{\bf k}/\Omega$, where the last expression vividly suggests a
drift of the spin-texture. This drift is a consequence of the spin-orbit
coupling and exists under general conditions, including equilibrium. However,
a net spin flow can only be induced by chiral phonons, not by achiral phonon.
This is illustrated in Fig. 2, where the non-zero components of the spin-
current tensor are plotted as function of the mode mixing parameter
$\omega_{1}$ for different phonon polarizations. Achiral phonons are
represented by $\omega_{1}=0$, at which condition there is no induced spin-
current, regardless of the phonon polarization.

By contrast, chiral phonons induces spin-currents for which the spatial
distribution depends on the polarization. As can be seen in Fig. 2, for
phonons polarized solely in the longitudinal or the transverse directions, the
spin-currents flow between these directions only. For phonons polarized along
some direction with both longitudinal and transverse components, the spin-
currents flow between all these. However, in general, the diagonal spin-
currents, $\mathbb{J}_{ii}$, $i=x,y,z$, are negligible which is expected for a
non-collinear spin-texture.

Figure 2:  Non-zero components of the spin-current tensor as function of the
mixing $\omega_{1}$ for phonons polarized along (a)
${\bm{\epsilon}}_{q}=(0,0,1)$, (b) ${\bm{\epsilon}}_{q}=(1,0,0)$, (c)
${\bm{\epsilon}}_{q}=(1,1,0)/\sqrt{2}$, and (d)
${\bm{\epsilon}}_{q}=(1,1,\sqrt{2})/2$. Other parameters are as in Fig. 1.

The induced spin-currents would be measurable as a spin accumulation at, e.g,
a terminating surface, by exposing the system to an accelerating force. For
instance, a spatially weakly inhomogenous temperature, such that $T=T({\bf
r})\approx T_{0}+({\bf r}-{\bf r}_{0})\cdot\nabla_{\bf r}T({\bf r})_{|{\bf
r}_{0}}$, where $T_{0}=T({\bf r}_{0})$ represents the reference temperature,
provides an accelerating field acting on the electrons via the temperature
gradient $\nabla T$. Then, to linear order in this field, the longitudinal
spin-current partitions into two contributions. The first of these derive from
the electronic temperature variations $k_{B}\beta^{2}([\varepsilon_{{\bf
k}}-\varepsilon_{F}]/4)\cosh^{-2}(\beta[\varepsilon_{{\bf
k}}-\varepsilon_{F}]/2)({\bf r}-{\bf r}_{0})\cdot\nabla T$. This contribution
gives the expected drift of electrons around the Fermi level
$\varepsilon_{F}$.

The second contribution derives from the thermal variations in the phonon
distribution, providing accelerating forces involving the processes
$\sum_{ss^{\prime}}k_{B}\beta^{2}\omega_{{\bf q}s}(1+s\omega_{\perp{\bf
q}}\cos\phi_{q})[2\sinh(s^{\prime}\beta\omega_{{\bf q}s}/2)(\varepsilon_{{\bf
q}}+s^{\prime}\omega_{{\bf q}s}-{\bf k}\cdot{\bf q})]^{-2}({\bf r}-{\bf
r}_{0})\cdot\nabla T$. The factor $s^{\prime}$ makes reference to the
absorption ($s^{\prime}=-1$) and emission ($s^{\prime}=1)$ processes, whereas
$s$ refers to the effective chirality. Ultimately, the phonon driven
contribution to the drift current results from the net of the two opposite
chiralities, c.f. Eq. (4). The theoretical model predicts that the induced
spin-currents should result in a spin accumulation at some interface surface
intersecting the current, in agreement with recent experimental observations
[39].

In conclusion, it has been demonstrated that the angular momentum carried by
chiral phonons may be transferred to electrons via a phase exchange. The
phonon chirality translates to the electron spin in the sense that it
generates a non-trivial spin-texture including circulating spin-currents.

###### Acknowledgements.

The author thanks the CISSors for helpful and constructive comments. The
author, moreover, thanks R. Arouca, A. Bergman, L. Barreto Braz, O. Grånäs, S.
Kahlhöfer, L. Nordström, M. Shiranzaei, A. Sisman, and P. Thunström for
discussions. Financial support from Vetenskapsrådet and Stiftelsen Olle
Engkvist Byggmästare is acknowledged.

## References

  * Strohm  _et al._ [2005] C. Strohm, G. L. J. A. Rikken, and P. Wyder, Phenomenological evidence for the phonon hall effect, Phys. Rev. Lett. 95, 155901 (2005).
  * Sheng  _et al._ [2006] L. Sheng, D. N. Sheng, and C. S. Ting, Theory of the phonon hall effect in paramagnetic dielectrics, Phys. Rev. Lett. 96, 155901 (2006).
  * Qin  _et al._ [2012] T. Qin, J. Zhou, and J. Shi, Berry curvature and the phonon hall effect, Phys. Rev. B 86, 104305 (2012).
  * Kagan and Maksimov [2008] Y. Kagan and L. A. Maksimov, Anomalous hall effect for the phonon heat conductivity in paramagnetic dielectrics, Phys. Rev. Lett. 100, 145902 (2008).
  * Zhang  _et al._ [2010] L. Zhang, J. Ren, J.-S. Wang, and B. Li, Topological nature of the phonon hall effect, Phys. Rev. Lett. 105, 225901 (2010).
  * Mori  _et al._ [2014] M. Mori, A. Spencer-Smith, O. P. Sushkov, and S. Maekawa, Origin of the phonon hall effect in rare-earth garnets, Phys. Rev. Lett. 113, 265901 (2014).
  * Park and Yang [2020] S. Park and B.-J. Yang, Phonon angular momentum hall effect, Nano Lett. 20, 7694 (2020).
  * Garanin and Chudnovsky [2015] D. A. Garanin and E. M. Chudnovsky, Angular momentum in spin-phonon processes, Phys. Rev. B 92, 024421 (2015).
  * Streib  _et al._ [2018] S. Streib, H. Keshtgar, and G. E. W. Bauer, Damping of magnetization dynamics by phonon pumping, Phys. Rev. Lett. 121, 027202 (2018).
  * Nakane and Kohno [2018] J. J. Nakane and H. Kohno, Angular momentum of phonons and its application to single-spin relaxation, Phys. Rev. B 97, 174403 (2018).
  * Bistoni  _et al._ [2021] O. Bistoni, F. Mauri, and M. Calandra, Intrinsic vibrational angular momentum from nonadiabatic effects in noncollinear magnetic molecules, Phys. Rev. Lett. 126, 225703 (2021).
  * Zhang and Niu [2014] L. Zhang and Q. Niu, Angular momentum of phonons and the einstein–de haas effect, Phys. Rev. Lett. 112, 085503 (2014).
  * Mentink  _et al._ [2019] J. H. Mentink, M. I. Katsnelson, and M. Lemeshko, Quantum many-body dynamics of the einstein–de haas effect, Phys. Rev. B 99, 064428 (2019).
  * Dornes  _et al._ [2019] C. Dornes, Y. Acremann, M. Savoini, M. Kubli, M. J. Neugebauer, E. Abreu, L. Huber, G. Lantz, C. A. F. Vaz, H. Lemke, E. M. Bothschafter, M. Porer, V. Esposito, L. Rettig, M. Buzzi, A. Alberca, Y. W. Windsor, P. Beaud, U. Staub, D. Zhu, S. Song, J. M. Glownia, and S. L. Johnson, The ultrafast einstein–de haas effect, Nature 565, 209 (2019).
  * Fransson  _et al._ [2017] J. Fransson, D. Thonig, P. F. Bessarab, S. Bhattacharjee, J. Hellsvik, and L. Nordström, Microscopic theory for coupled atomistic magnetization and lattice dynamics, Phys. Rev. Materials 1, 074404 (2017).
  * Bennett  _et al._ [2013] S. D. Bennett, N. Y. Yao, J. Otterbach, P. Zoller, P. Rabl, and M. D. Lukin, Phonon-induced spin-spin interactions in diamond nanostructures: Application to spin squeezing, Phys. Rev. Lett. 110, 156402 (2013).
  * Hamada  _et al._ [2018] M. Hamada, E. Minamitani, M. Hirayama, and S. Murakami, Phonon angular momentum induced by the temperature gradient, Phys. Rev. Lett. 121, 175301 (2018).
  * Juraschek and Spaldin [2019] D. M. Juraschek and N. A. Spaldin, Orbital magnetic moments of phonons, Phys. Rev. Materials 3, 064405 (2019).
  * Juraschek  _et al._ [2020] D. M. Juraschek, P. Narang, and N. A. Spaldin, Phono-magnetic analogs to opto-magnetic effects, Phys. Rev. Research 2, 043035 (2020).
  * Juraschek  _et al._ [2022] D. M. Juraschek, T. c. v. Neuman, and P. Narang, Giant effective magnetic fields from optically driven chiral phonons in $4f$ paramagnets, Phys. Rev. Research 4, 013129 (2022).
  * Zhu  _et al._ [2018] H. Zhu, J. Yi, M.-Y. Li, J. Xiao, L. Zhang, C.-W. Yang, R. A. Kaindl, L.-J. Li, Y. Wang, and X. Zhang, Observation of chiral phonons, Science 359, 579 (2018).
  * Yin  _et al._ [2021] T. Yin, K. A. Ulman, S. Liu, A. Granados del Águila, Y. Huang, L. Zhang, M. Serra, D. Sedmidubsky, Z. Sofer, S. Y. Quek, and Q. Xiong, Chiral phonons and giant magneto-optical effect in crbr3 2d magnet, Adv. Mater. 33, 2101618 (2021).
  * Jeong  _et al._ [2022] S. G. Jeong, J. Kim, A. Seo, S. Park, H. Y. Jeong, Y.-M. Kim, V. Lauter, T. Egami, J. H. Han, and W. S. Choi, Unconventional interlayer exchange coupling via chiral phonons in synthetic magnetic oxide heterostructures, Sci. Adv. 8, eabm4005 (2022).
  * Chen  _et al._ [2019] X. Chen, X. Lu, S. Dubey, Q. Yao, S. Liu, X. Wang, Q. Xiong, L. Zhang, and A. Srivastava, Entanglement of single-photons and chiral phonons in atomically thin wse2, Nat. Phys. 15, 221 (2019).
  * Baydin  _et al._ [2022] A. Baydin, F. G. G. Hernandez, M. Rodriguez-Vega, A. K. Okazaki, F. Tay, G. T. Noe, I. Katayama, J. Takeda, H. Nojiri, P. H. O. Rappl, E. Abramof, G. A. Fiete, and J. Kono, Magnetic control of soft chiral phonons in pbte, Phys. Rev. Lett. 128, 075901 (2022).
  * Jaworski  _et al._ [2011] C. M. Jaworski, J. Yang, S. Mack, D. D. Awschalom, R. C. Myers, and J. P. Heremans, Spin-seebeck effect: A phonon driven spin distribution, Phys. Rev. Lett. 106, 186601 (2011).
  * Lou  _et al._ [2018] P. C. Lou, L. de Sousa Oliveira, C. Tang, A. Greaney, and S. Kumar, Spin phonon interactions and magneto-thermal transport behavior in p-si, Solid State Communications 283, 37 (2018).
  * Du  _et al._ [2019] L. Du, J. Tang, Y. Zhao, X. Li, R. Yang, X. Hu, X. Bai, X. Wang, K. Watanabe, T. Taniguchi, D. Shi, G. Yu, X. Bai, T. Hasan, G. Zhang, and Z. Sun, Lattice dynamics, phonon chirality, and spin–phonon coupling in 2d itinerant ferromagnet fe3gete2, Adv. Funct. Mater. 29, 1904734 (2019).
  * Overhauser [1953] A. W. Overhauser, Paramagnetic relaxation in metals, Phys. Rev. 89, 689 (1953).
  * Elliott [1954] R. J. Elliott, Theory of the effect of spin-orbit coupling on magnetic resonance in some semiconductors, Phys. Rev. 96, 266 (1954).
  * Andreev and Gerasimenko [1959] V. V. Andreev and V. I. Gerasimenko, On the theory of paramagnetic resonance and paramagnetic relaxation in metals, Sov. Phys. JETP 35, 846 (1959).
  * Yafet [1663] Y. Yafet,  _Solid State Phys._ , edited by F. Seitz and D. Turnball (Academic, 1663) pp. 1–98.
  * Baral  _et al._ [2016] A. Baral, S. Vollmar, S. Kaltenborn, and H. C. Schneider, Re-examination of the elliott–yafet spin-relaxation mechanism, New J. Phys. 18, 023012 (2016).
  * Fransson [2020] J. Fransson, Vibrational origin of exchange splitting and chiral-induced spin selectivity, Phys. Rev. B 102, 235416 (2020).
  * Du  _et al._ [2020] G.-F. Du, H.-H. Fu, and R. Wu, Vibration-enhanced spin-selective transport of electrons in the dna double helix, Phys. Rev. B 102, 035431 (2020).
  * Zhang  _et al._ [2020] L. Zhang, Y. Hao, W. Qin, S. Xie, and F. Qu, Chiral-induced spin selectivity: A polaron transport model, Phys. Rev. B 102, 214303 (2020).
  * Fransson [2021] J. Fransson, Charge redistribution and spin polarization driven by correlation induced electron exchange in chiral molecules, Nano Lett. 21, 3026 (2021).
  * Li  _et al._ [2021] X. Li, J. Zhong, J. Cheng, H. Chen, L. Zhang, and J. Zhou, Chiral phonon activated spin seebeck effect 10.48550/ARXIV.2105.08485 (2021).
  * Kim  _et al._ [2022] K. Kim, E. Vetter, L. Yan, C. Yang, Z. Wang, R. Sun, Y. Yang, A. Comstock, X. Li, J. Zhou, L. Zhang, W. You, D. Sun, and J. Liu, Observation of the chiral phonon activated spin seebeck effect, XXX XX, xxxxxx (2022).
  * Mondal  _et al._ [2015] R. Mondal, M. Berritta, K. Carva, and P. M. Oppeneer, Ab initio investigation of light-induced relativistic spin-flip effects in magneto-optics, Phys. Rev. B 91, 174415 (2015).
  * She  _et al._ [2013] J.-H. She, J. Fransson, A. R. Bishop, and A. V. Balatsky, Inelastic electron tunneling spectroscopy for topological insulators, Phys. Rev. Lett. 110, 026802 (2013).


# RFC-HyPGCN: A Runtime Sparse Feature Compress Accelerator for Skeleton-Based
GCNs Action Recognition Model with Hybrid Pruning

Dong Wen, Jingfei Jiang, Jinwei Xu, Kang Wang, Tao Xiao, Yang Zhao, Yong Dou
National University of Defense Technology, Changsha, China
{wendong19, jingfeijiang, xujinwei13, wangkang, xiaotao19, zhaoyang10,
<EMAIL_ADDRESS>
###### Abstract

Skeleton-based Graph Convolutional Networks (GCNs) models for action
recognition have achieved excellent prediction accuracy in the field. However,
limited by large model and computation complexity, GCNs for action recognition
like 2s-AGCN have insufficient power-efficiency and throughput on GPU. Thus,
the demand of model reduction and hardware acceleration for low-power GCNs
action recognition application becomes continuously higher.

To address challenges above, this paper proposes a runtime sparse feature
compress accelerator with hybrid pruning method: RFC-HyPGCN. First, this
method skips both graph and spatial convolution workloads by reorganizing the
multiplication order. Following spatial convolution s channel-pruning
dataflow, a coarse-grained pruning method on temporal filters is designed,
together with sampling-like fine-grained pruning on time dimension. Later, we
come up with an architecture where all convolutional layers are mapped on chip
to pursue high throughput. To further reduce storage resource utilization,
online sparse feature compress format is put forward. Features are divided and
encoded into several banks according to presented format, then bank storage is
split into depth-variable mini-banks. Furthermore, this work applies
quantization, input-skipping and intra-PE dynamic data scheduling to
accelerate the model. In experiments, proposed pruning method is conducted on
2s-AGCN, acquiring 3.0x-8.4x model compression ratio and 73.20% graph-skipping
efficiency with balancing weight pruning. Implemented on Xilinx XCKU-115 FPGA,
the proposed architecture has the peak performance of 1142 GOP/s and achieves
up to 9.19x and 3.91x speedup over high-end GPU NVIDIA 2080Ti and NVIDIA V100,
respectively. Compared with latest accelerator for action recognition GCNs
models, our design reaches 22.9x speedup and 28.93% improvement on DSP
efficiency.

###### Index Terms:

Graph Neural Network, Action Recognition, Hybrid Pruning, Sparse Data
Compress, Hardware Accelerator, Field-programmable Gate Array (FPGA)

##  I Introduction

Action recognition based on deep learning has great potential to be applied in
kindergartens and hospitals to prevent danger motions. Skeleton-based graph
convolutional networks (GCNs) methods have achieved state-of-the-art (SOTA)
prediction accuracy in the field [1][2][3]. Mature pose estimation algorithms
extract human skeletons from video stream with real-time speed, for example,
OpenPose [4] and Alphapose [5]. GCNs action recognition models and pose
estimation models thus can be combined into an end-to-end system.

Despite skeleton-based GCNs having great advantages, several problems limit
their applications in expected scenarios. Firstly, intensive computation and
large network architectures are embedded in skeleton-based GCNs, causing great
computing cost on GPUs. MobilePose [6] can produce human skeletons on mobile
platform Snapdragon 845 with 60 fps and 44.4 fps/Watt, while 2s-AGCN model
merely has a performance of 28 fps and 0.11fps/Watt on NVIDIA 2080Ti GPU. The
computing speed and power-consumption’s gap indicates a great importance on
accelerating GCNs action recognition algorithms. Secondly, the expected
application environment of action recognition models poses stringent
constraints on power-consumption and throughput. However, the high-performance
GPU cannot meet the power-efficiency demand.

Network pruning and graph sparsification [7][8] are two effective methods to
relieve model complexity. However, these methods are unsuitable for skeleton-
based GCNs. There are two reasons. (i) _Dataflow is transformed_ : Graph
computation changes the convolution dataflow. When being conducted on
different dataflows, traditional pruning methods for CNNs only skip useless
computing in convolution but may not work on graph task. (ii) _Skeleton-
relationship graph is unchangeable and sensitive._ Some works use pooling [7]
or graph sparsification [8] to drop unimportant edges and points in graph.
However, the human skeleton graph cannot be modified for bones connection
being unchangeable. Particularly, there are learnable hidden information graph
[1][9] which lacks sparsity in some GCNs models . The subtle elements in such
graph are proved to be positively associated with prediction performance. For
instance, in 2s-AGCN model, the prediction accuracy decreases by 2.3% without
learnable matrix [9]. Although Ding et al. [10] present a FPGA-based work on
accelerating ST-GCN, a smaller action recognition GCNs, their work falls short
on more complex models for (i) They do not prune the target model. (ii) They
choose to compress sparse static skeleton graph, while skeleton relationship
matrix in some models can be dense. (iii) Only sparsity graph is optimized for
computation, while feature sparsity is not utilized. (iv) Their single-PE
design cannot meet performance requirement of application scenarios. To
summarize, such GCN models with dense matrixes bring new challenges for
architecture.

For these reasons, efficient pruning methods together with specific
accelerator designs are urgently required to accelerate GCNs action
recognition workloads. To this end, we present RFC-HyPGCN: a runtime sparse
feature compress accelerator for skeleton-based GCNs action recognition model
with hybrid pruning in this paper.

A hybrid GCNs’ pruning method is proposed, which reduces network parameters
and skips graph computation efficiently. We reorganize dataflow by changing
the multiplication order of graph workloads and spatial convolution. With new
dataflow, graph computation and spatial convolution are skipped if the
corresponding parameter is pruned as zero. As to the temporal convolution,
mixed-grained pruning method is elaborately designed. Fine-grained pruning
operation can be dealt as sampling in time series, while coarse-grained
pruning is decided by spatial convolution’s pruned dataflow. Experiments
demonstrate that in most cases, better prediction accuracy and more balanced
pruning can be possessed by our model compared with conventional methods.
Additionally, quantization and input-skipping are applied, which are common
means to accelerate neural networks.

We also design an application-specific architecture. Ten convolution blocks
are mapped on FPGA, which is the platform widely used for speeding up deep
neural networks. Different from previous works, in our layer-pipelined
architecture, challenges are not only reflected on different kinds of sparse
tasks, but also on how to efficiently store sparse intermediate results on
chip. Common compact formats like compressed sparse column format (CSC) is not
the optimal resolution due its irregular memory access and extra
encoding/decoding cost. To address these challenges, our sparse-degree-based
runtime sparse feature compress method is proposed, which splits data
encoding/decoding and corresponding storage into fine-grained bank and mini-
bank. Finally, dynamic data scheduling is applied for intra process elements
(PE) to decrease the utilization of DSPs.

In summary, the contributions in this paper are:

  * •

We propose a hybrid pruning method on 2s-AGCN model, which contains dataflow
reorganization and mix-grained pruning method. The experiments show that our
method is better than conventional pruning on computation-skipping and
prediction accuracy.

  * •

A co-designed architecture is implemented, including runtime sparse vector
compress storage and dynamic data scheduling. The proposed online data compact
format reduces the utilization of BRAM blocks as well as keeping regularity of
data-access. Dynamic data scheduling saves DSPs and raises DSPs working
efficiency with tiny delay.

  * •

Our design is implemented on Xilinx XCKU-115 FPGA platforms with 172 MHz. It
achieves 2.61x-9.19x accelerating ratio compared with NVIDIA 2080Ti and
1.36x-3.91x with NVIDIA V100. On contrast to similar work [10], ours exceeds
on both peak performance and DSPs efficiency. It has the potential to apply in
end-to-end and low-power real time environments.

##  II Background on 2s-AGCN Model

Figure 1: Left: Structure of the basic convolutional block. ConvS stands for
spatial convolution and ConvT stands for temporal convolution. Right: Variance
of the feature dimension. There are 25 key joints in human skeleton and 300
skeleton vectors in the original input feature.

The skeleton-based action recognition GCN models regard human skeleton
features as input and predict human action , like waving and drinking. Several
human skeleton datasets have been proposed, for example NTU-RGB+D[11] and
Kinetics [12]. Our experiments on 2s-AGCN are trained and tested on NTU-RGB+D.
There are ten convolutional blocks and one fully-connected (FC) layer in
2s-AGCN model. As shown in the left picture in Fig. 1, the computation in each
block can be divided into five phases: graph computation, self-similarity
computation, spatial convolution, temporal convolution and shortcut
connection. Batch-normalization and ReLU activation follow behind each
convolution operations. With network going deeper, more channels are stacked
on feature. The right picture of Fig. 1 illustrates this tendency in data
dimension.

In each layer, three different graphs are involved in computation: $A_{k}$,
$B_{k}$ and $C_{k}$, $k$ is explained in (2). The first part $A_{k}$ is the
static human skeleton graph, the second part $B_{k}$ is a learnable skeleton
connection graph and $C_{k}$ is a data-dependent graph generated from self-
similarity process. Elements in $B_{k}$ are trained to indicate hidden
relationships between joints and bones. Unlike static graph $A_{k}$, $B_{k}$
is dense and sensitive to numerical changes. In [9], $C_{k}$ is produced via
(1), where high-dimension tensor transposition and multiplication are
conducted on input feature. $W_{\theta}$ represents similarity coefficient. To
sum up, the computation of graph and spatial convolution can be described as
(2). $\bigotimes$ represents convolution operation, $K_{\nu}$ denotes the
neighbour size of the graph computation and is set to 3 in the 2s-AGCN model.
The kernel size of spatial convolution s weight $W_{k}$ is set to 1.

$C_{k}=softmax(f_{in}^{T}W_{\theta}f_{in})$ (1)
$f_{out}=\sum_{k}^{k_{\nu}}f_{in}(A_{k}+B_{k}+C_{k})\bigotimes W_{k}$ (2)

Different from $A_{k}$ and $B_{k}$ which are determined before inference,
$C_{k}$ relies on input feature, thus needs runtime computing for each
prediction. Table. I demonstrates the computing cost of self-similarity. The
running performance of 2s-AGCN with and without $C_{k}$ are tested on NVIDIA
V100. At the cost of computing complexity and longer time-delay, $C_{k}$ only
elevates prediction accuracy by 0.3%. From the view of software-hardware co-
design, dropping $C_{k}$ graph is a reasonable trade-off for workload
reduction.

TABLE I: Model performance with(w/) and without(w/o)$C_{k}$. | accuracy | throughput | power efficiency
---|---|---|---
2sAGCN(w/C) | 93.70% | 69.38 fps | 0.28 fsp/watt
2sAGCN(w/oC) | 93.40% | 98.87 fps | 0.40 fps/watt

Figure 2: The neighbour connection between temporal convolutional output and
spatial convolutional input.

Following the spatial convolution, temporal convolutional layer is set at the
end of each convolutional block. With kernel size of $9\times 1$, temporal
convolution extracts information from nine skeleton vectors in time order.
Despite the insertion of the graph computation, temporal convolution layer in
block $l$ can still be seen as the leading neighbour of spatial convolution
layer in next block because graph computation does not change temporal
convolutional result along its output-channel dimension, and spatial
convolution operates indirectly on temporal convolution’s output in block
$l+1$ [13]. For above reasons, the connection shown in Fig. 2 guides us to
conduct coarse-grained pruning on temporal convolutional filters.

##  III Related Work

CNNs Accelerators on FPGA. Works on FPGA-based acceleration of sparse CNNs can
be categorized by different pruning granularity levels [13]: (i) for coarse-
grained pruned models, (ii) for fine-grained pruned models, (iii) for mixed-
grained pruned models. Zhu et al. [13] presented a zero-skipping dataflow for
feature. Although such method raised computing efficiency, zero elements in
intermediate result still occupied storage resource. Lu et al. [14] proposed a
weight-oriented dataflow for fine-grained pruned CNNs with little decoding
cost. However, 2s-AGCN model differs from above convolutional workloads in
that feature first goes through graph matrix multiplication. This weight-
oriented design cannot skip corresponding graph computation. Li et al. [15]
worked on PCONV pruning [16], a mixed-grained method. With weight-stationary
dataflow designed on FPGA, Li et al. improved the computing efficiency by
14.7%-44%. However, this work still occupied storage space for huge scale of
zero data like Lu et al., and its simple hardware structure could not tackle
complex workloads in our task.

GCN Accelerators on FPGA. Many works on accelerating large graph’s GCNs based
on FPGA are presented in recent time. AWB-GCN [17] combined dynamic hardware
configuration and runtime hardware workloads balancing on several large graph
datasets. Zhang et al. [18] partitioned input data into smaller segments, then
perform graph sparsification and node re-ordering for computation reduction
and data locality. To sum up, above works focus on: (i) Leveraging and
expanding graph adjacency matrixes sparsity, (ii) Avoiding irregularity and
randomness of data distribution in graph computation, (iii) Keeping balanced
workloads between PEs or computing phases, via offline and online ways.
Unfortunately, graph in skeleton-based GCNs for action recognition models is
dense and unchangeable. The data sparsity is embedded in temporal feature and
pruned weights, not the graph. Moreover, action recognition GCNs behave not
only like CNNs, but also like graph processing, leading to graph-specific
design requirements. Therefore, current specialized architectures on CNNs and
GCNs cannot efficiently accelerate target models since they just take one of
the two sides.

##  IV Methodology

This section introduces our hybrid pruning method for action recognition GCNs.
The dataflow reorganization, coarse-grained and fine-grained pruning on
temporal convolution are described respectively.

###  IV-A Dataflow Reorganization

After clipping self-similarity graph, the computing flow between graph and
spatial convolution can be further summarized as (3), where $G_{k}$ denotes
$A_{k}+B_{k}$ from (2). The computing order is first high-dimensional matrix
multiplication with $G_{k}$, then the spatial convolution of $W_{k}$ and
finally the result merging of three loops. In this dataflow, common pruning
methods only functions in second phase but cannot optimize the graph
computation, which occupies 49.83% of total workloads in (3).

$f_{out}=\sum_{k}^{k_{\nu}}f_{in}G_{k}\bigotimes W_{k}$ (3)

To better analyse the dataflow, we extract first two phases and its output X.
A pixel can be described as $X(h,w,oc)$, where h, w, oc represent height,
width and output-channel coordinates respectively. Then (4) can be deduced
from (3) and ic is the acronym of input channel. Under the commutative law of
multiplication, therefore (4) is transformed into (5). By reorganizing the
computing order between graph phase and convolution phase, an opportunity for
graph-skipping pruning is offered here. If the parameter element $W(1,1,i,oc)$
is pruned to zero, the graph matrix multiplication in current output channel
can be ignored. Further, if we set all convolutional parameters in i input
channel as zero, then all graph computation can be skipped in current loops.
The dataflow reorganization is then proposed when we apply above method to
three loops in (3). Unlike conventional structure pruning method which drops
different channels on filters, weights in specific input channels are all set
as zero on every spatial filter in current convolutional blocks. In this way,
not only the convolution workload is reduced, but also the graph computation
is skipped.

$X(h,w,oc)=\sum_{i=1}^{ic}(\sum_{p=1}^{25}f_{in}(h,p,i)\times G(p,w))\times
W(1,1,i,oc)$ (4) $X(h,w,oc)=\sum_{i=1}^{ic}(\sum_{p=1}^{25}G(p,w)\times
f_{in}(h,p,i)\times W(1,1,i,oc))$ (5)

Since the graph-skipping strategy has been determined by dataflow
reorganization, the next step is choosing the input channel to be pruned. Like
other deep neural networks (DNN), features between convolutional layers are
sparse and useful elements are unevenly distributed. Based on the observation
that unstructured pruning method drops weight element with relatively small
absolute value, we cut off the input channels which have least averaging
absolute value. In this way, data reorganization prunes spatial convolutional
weight and skips both graph and convolution computation.

###  IV-B Mixed-grained Pruning Method

In dataflow reorganization method, features in specific channels are not
computed because such channels are pruned. The coarse-grained method then
prunes corresponding temporal filters via connections in Fig. 2 with no extra
accuracy loss. Moreover, this neighbour connection is hardware-friendly for
that the number of pruned channels in spatial filters equals the number of
pruned filters in temporal convolution. This inherent feature supports a
balanced layer-pipelined architecture.

Figure 3: The illustration of fine-grained pruning on temporal convolution.
White elements are pruned while blue ones are kept. Every 9$\times$1 kernel
performs on time series of skeleton vectors, and the blue vectors are sampled
by first pruning scheme.

Coarse-grained pruning can provide 49.83%-88.96% compression ratio on temporal
filters, depending on the pruning scheme in data organization phase. To
further prune temporal convolutional weights, fine-grained pruning is
proposed. The point of fine-grained method is that in temporal convolution,
zero weight means not sampling current vectors in time order. Fig. 3
demonstrates details of sampling-like fine-grained pruning method. Several
pruning schemes with different intervals and offsets are conducted on filters
recurrently. By this means, the design of pruning scheme is turned into a
sampling problem. We can simulate various sampling schemes on filters, with
different sampling frequencies and phases represented by intervals and
offsets. Experiments show that with proper pruning scheme, our fine-grained
method can keep accuracy as well as discarding unimportant weight.

Conventional unstructured pruning methods randomly drop the weight elements
with least absolute value, which are expensive and unbalanced on hardware.
However, with determined cavity schemes, our fine-grained pruned model can be
indicated by structured weight together with masks. Furthermore, we guarantee
the balancing distribution of reserved weight by controlling start-points of
different sampling patterns. Like Fig. 3 shows, in a loop of eight different
pruning modes, weight elements in every position of kernels are evenly kept by
two or three times. Compression ratio can be adjusted via fine-grained pruning
design.

Figure 4: The demonstration of overall architecture.

##  V Architecture

This section introduces the detailed architecture of our accelerator, where
all pruned convolutional blocks are mapped on chip.

_Overview_ : Fig. 4 depicts the overall design of our layer-pipelined
architecture. Conv block module constitutes the whole architecture, containing
one spatial conv module (SCM) and one temporal conv module (TCM). To be more
detailed, one conv block module processes on convolution block. Due that
proposed pruning method reduces model size, all parameters and graph are
stored in ROM storage on chip. Mult-PE is the basic computing unit of SCM,
while Dyn-Mult-PE is the basic unit in TCM. Moreover, runtime sparse feature
compression module (RFC) functions at the junctions of different layers to
compact and store temporal results.

###  V-A Spatial Conv Module

The main task of SCM is performing graph computation and pruned spatial
convolution. Data-fetch controls the address of data-loading and decodes
compact feature into sparse form, which is prepared for computing. Feature
buffer receives and stores decoded data in order. The decoding process will be
explained later. Sparse feature will first multiply with graph vectors, then
conduct convolution with non-zero weight in Mult-PEs. Pruned channels are
skipped and multiplication results are summed up in accumulating buffer on
output channel. After batch-normalization operation, dataflow merges with
original input activation, which is stored in shortcut feature buffer. ReLU
function is combined with encoding function parts.

In order to combine graph computation and pruned convolution workloads,
dataflow is organized as Fig. 5 shows. A line in feature buffer caches 25 data
and the depth equals the number of kept channels in filter. When computing,
buffer offers one line of original feature data. After computation with one
column vector of graph, there generates one valid element $X(h,w,oc)$ in (4).
Afterwards, feature buffer provides next cache line, which continues to
produce $X(h,w,oc+1)$. Following this mode, when all output elements on
current output channel are computed, feature buffer returns to the first line
and graph ROM switches to the vector in next column to prepare for
$X(h,w+1,0:N_{o}c)$. When the workload of one row feature tensor is finished,
feature buffer receives next row of tensor to start a new sub-loop. In this
way, feature is produced in a channel-first order. Our dataflow reorganization
method essentially abandons feature data on specific channels, so we skip
corresponding workloads by not sending them to feature buffer.

Figure 5: Illustration of SCM dataflow organization.

Feature element is broadcast to all Mult-PEs. In the same channel-first order,
weight ROM sends different filters’ parameters into computing units. To match
the pruned model, only non-zero weights are stored. Each Mult-PEs includes
four DSPs, and by adjusting the number of Mult-PE, our design can fit into
different layers. Results from parallel Mult-PEs are accumulated and buffered
on channel direction as well. When the sum counter reaches the number of valid
channels, current data will be transferred into post-processing modules.

###  V-B Temporal Conv Module

TCM is designed to accelerate temporal convolution workloads, whose kernel
size is $9\times 1$. Fig. 6 shows the detailed information of TCM. Similarly,
feature buffer stores decoded data from data-fetch. However, buffer width is
turned from 25-data into 9-data, and the depth is tuned for holding an
$9\times 25\times in\\_c$ area of feature tensor. Additionally, feature’s one-
hot code is sent from data-fetch to feature hot storage as well. Valid weight
together with its masks is stored on chip. As depicted in Fig. 3, several
balanced fine-grained pruning schemes are conducted on leftover filters in
recurrent ways, providing opportunities for structured weight storage. One
temporal filter is divided into several $1\times 1\times 16$ sub-filters,
thus, parameters can be folded into sub-filters format and then be stored in a
recurrent mode. Moreover, Dyn-Mult-PEs are put across input channel and
parallelizes on filter’s rows. There are two reasons: (i) This parallel scheme
can directly skip the abandoned filters in coarse-grained pruning, (ii) Each
row of sub-filters is taken by one Dyn-Mult-PE and each function part handles
one row of weight tensor. In this way, one Dyn-Mult-PE only needs to process
weights derived from static cavity mask, such as four or six weights in Fig.
6. This design further eliminates data irregularity and scheduling
uncontrollability.

Figure 6: Illustration of TCM’s dataflow.

Shown in Fig. 6, one row of a sub-filter is assigned to a Dyn-Mult-PE, which
includes four or six waiting queues and several DSPs. Logic and operantion is
performed first on weight mask and feature mask to skip the zero-feature and
dropped weights. Then, valid feature enters one waiting queue, which is bonded
to a non-zero weight in the sub-filter. To decrease the use of DSPs, dynamic
data scheduling is designed by dispatching data from busy waiting lines to
empty DSPs. Because multiplication in a Dyn-Mult-PE is part of the intra-
filter computation, results need to be summed up and afterwards sent to the
adder-tree. While dynamic data scheduling has advantages on saving DSP
resource, it may increase the working delay at workloads-intensive cases. With
the help of recurrent fine-grained pruning and statistic sparsity, we
calculate the expectation of valid computation in one sub-filter and use it to
guide the DSP occupation. The detailed method is illustrated in (6), where the
number of kept weight in a sub-filter is assumed as six and $s$ stands for
feature sparsity.

$E(D)=\sum d\times p(d)=3(1-s)^{3}+3s^{2}(1-s)+6s(1-s)^{2}$ (6)

###  V-C Runtime Sparse Feature Compress

Despite layer-pipelined architecture poses great advantages on throughput, it
has to store massive temporal computing results for shortcut task. RFC is
presented to address this issue. Encoding, compact storage and decoding are
included in RFC structure. Encoding process is combined with ReLU while
decoding is embedded in data-fetch module. The whole structure of RFC is
displayed in Fig. 7.

Encoding: At first, one feature vector is divided into several banks across
channels. The width of each bank is 16 data-wise. ReLU function parts perform
on banks, providing activation and one 16-bit hot code, which denotes the
positive/zero value. Then valid elements are gathered at higher bits while
unused bits are padded with zero. After that, mini-bank-hot code (mbhot) is
generated according to the number of non-zero data in bank. Mbhot indicates
which mini-banks are used in the bank storage. Encoding parts work in pipeline
and during several working cycles, the whole vector is finally turned into
compact format. Instead of compressing one vector as whole, we lower the
encoding cost by setting bank as the finest grain of ReLU and encoding
process.

Figure 7: Illustration of TCM’s dataflow.

Storage: The compact storage consist of bank storage units, which includes
mini-banks, mini-bank-hot storage, data-hot storage and address controller pt.
The key insight of bank storage design is to keep access regularity on data-
width dimension and reduce useless storage on data-depth dimension. When input
data and hot codes are valid, mbhot sends enabling signal to every mini-banks
and related pts. For example, if the input data-hot code is
0001_1100_0000_0111, meaning there are five non-zero data in the high bits of
input data stream and the mbhot is 1100. The first mini-bank receives and
stores four valid data, the second mini-bank keeps fifth valid data and three
zero data. Their pt will self-add after this data-writing. Other mini-banks
and their pt are not started. Similarly, when we need to load data from bank
storage in order, mbhot enables related mini-banks and pts to output correct
data. The output of disabled mini-banks is covered by zero. Without any random
access schemes, compact data can be both stored and loaded in only one cycle.

Another issue on compact storage is to determine the volume of each mini-bank.
Like deciding DSP’s utilization, we can calculate the expectation of useful
data based on offline sparsity. However, there always exists vectors drifting
away from average sparsity. Denser vectors demand deeper mini-banks on the
tail (the rightmost mini-bank in Fig. 7) while the lighter vectors merely
occupy head mini-banks. In ideal cases, every vector is fit in bank-lines with
no mini-banks unused and no vector truncated, but it is hard to precisely
determine the number of valid data in every vector. Feature s sparsity
distribution of each layer can help us to adjust the depth of mini-banks. For
example, sparsity of feature is 50% and 25% of vectors has sparsity higher
than 75%, 25%’s feature s are between 50%-75%, 25%’s data is between 25%-50%
and 25% vectors’ are below 25%. The mini-bank arrangement in Fig. 7 meets the
demand of different density feature and reduces 37.50% storage resource
compared with sparse format. In actual design, BRAM units have variable
grains, which provides more flexibility.

Decoding: The decoding function is integrated in data-fetch module in SCM and
TCM. Data-fetch not only controls loading address, but also translates compact
data into sparse form. After receiving both data stream and data-hot codes
from bank storages, parallel decoding modules perform on each banks output.
Each translation part processes compressed feature in four pipeline stages,
four data for one stage. Matched with encoding phase, the output of one bank
is seen as the basic decoding grain, which further decreases the complexity of
decoding circuit.

##  VI Experiments

In this section, we evaluate our design on both software and hardware views.
Our pruning method is explored on one V100 GPU using PyTorch, and accelerator
architecture is implemented with Verilog HDL on Vivado 2018.3 IDE.

###  VI-A Validations on Hybrid Pruning Method

In experiments, the proposed hybrid pruning method on 2s-AGCN model is
compared with unstructured pruning on NTU-RGB+D, which contains 37k training
and 18k testing data. We explore the impact of different pruning designs on
accuracy, e.g., various fine-grained pruning schemes for temporal filters and
channel-dropping modes in reorganization phase.

Comparison: Fig. 8 illustrates the contrast between our hybrid pruning methods
and conventional pruning means. Both unstructured pruning and hybrid pruning
can elevate accuracy for deleting some unimportant weights and improving
convergence performance. With same parameters reduction rate, our method
achieves better accuracy performance in most cases. Additionally, we apply
quantization on our pruned models. With negligible accuracy loss, float data
is transformed into fix-point format, where eight bits are allocated to
decimal part and eight to integer part. To further accelerate the proposed
application-specific system, half of input skeleton vectors are skipped.
Although input-skip method lowers prediction accuracy, it brings 50% reduction
on total computation. Besides, the input-skip model with 86% compress ratio
still keeps the accuracy no less than original model, so we choose this model
as final accelerating target.

Figure 8: Comparison results with unstructured pruning.

Exploration: Both data reorganization and fine-grained temporal pruning are
fatal to model accuracy. To find the best pruning scheme, we conduct isolated
experiments respectively. Additionally, based on our dataflow reorganization
method, graph-skipping rate equals channel-dropping rate of this phase. Guided
by feature sparsity in Fig. 9, we first set each layer s channel pruning rate
roughly equals it s sparsity respectively. For higher parameter reduction
ratio, we progressively raise compressing rate on layers and observe effect on
accuracy, as shown in Fig. 9. Drop-1, Drop-2, and Drop-3 stands for different
channel pruning design. The spatial convolutional parameter in block 1 is not
pruned for it only has three input channels. Also, mix-grained pruning on
temporal convolution is excluded to validate data reorganization method. It
reveals that with compress rate shifting away from base sparsity, model
reduction is growing while accuracy is decreasing. We choose the Drop-1, which
keeps the best accuracy, as our base dataflow reorganization design.

Figure 9: Exploration on channel dropping.

The fine-grained method is important in holding accuracy and keeping balanced-
pruning, since coarse-grained means is totally decided by data reorganization.
We carry out several experiments on fine-grained pruning, including different
pruning intervals, offsets and pruning rates. All experiments are based on
Drop-1 model in Fig. 9 and results are shown in Fig. 10. Pruning schemes in
Fig. 10 are named as the combination of cav (cavity), pruning percent (50, 67
for instance) and intra-order. The size of rectangles below is $9\times$8,
which denotes eight $9\times$1 kernels in loop. Cav-70-1 means the first
cavity patterns with 70% reduction rate. With reduction rates expanding, model
bears more accuracy loss in general. However, cavity patterns play an
important role as well. With same compress ratio of 70%, cav-70-1 performs
better than cav-70-2 on accuracy for more balanced weight pruning. Every
weight line in cav-70-1 has two or three sampling chances, while in cav-70-2,
different lines are kept from one time to four times. Balanced pruning schemes
not only provide convenience for hardware, but also ensure the accuracy
performance. The same situation happens between cav-75-1 and cav-75-2 as well.
Taking both compress ratio and accuracy into consideration, cav-70-1 is chosen
to be the final design.

Figure 10: Exploration on fine-grained pruning schemes.

###  VI-B Hardware implement

Dyn-MultPE: Dyn-MultPE works on the cav-70-1 cavity pattern, which means there
are three Dyn-MultPEs needing to six waiting queues and six facing four
waiting queues. Based on (6), different numbers of DSPs are settled in each
layer s Dyn-MultPEs. We also adjust the number of temporal convolutional PE to
keep balance between pipeline stages. We choose some detail information to
show in Table. II, where our dynamic data scheduling trades only 6.48% of
longer delay for DSP reduction of 23.24%.

TABLE II: Utilization, working efficiency and max delay of Dyn-MultPE. layer | DSP in one PE | total DSP | efficiency | max delay
---|---|---|---|---
1 | 4/6 | 63 | 66.79% | 0.00%
2 | 4/6 | 126 | 83.76% | 3.70%
3 | 4/6 | 126 | 80.96% | 0.00%
4 | 2/3 | 126 | 83.46% | 7.40%
total |  | 882 | 75.38% | 6.48%
static |  | 1149 | 57.86% | 0.00%
TABLE III: Feature sparsity distribution of some layers layer | I | II | III | IV
---|---|---|---|---
l1.sconv |  $<$0.01% | 29.35% | 70.64% |  $<$0.01%
l1.tconv | 0.02% | 94.73% | 5.25% | 0.00%
l2.sconv | 0.00% | 0.73% | 75.79% | 23.48%
l2.tconv |  $<$0.01% | 34.24% | 65.76% | 0.00%

RFC: As stated above, RFC design relies on sparsity distribution. To optimize
runtime compress storage, we refer to pruned model’s offline sparsity
distribution, as is partly shown in Table. III. Feature vectors are divided
into four categories by their sparsity: 75%-100% (I), 50%-75% (II), 25%-50%
(III) and 0%-25% (IV). According to our RFC design, vector of first category
occupies one mini-bank, ones in II takes two, III takes three and IV takes
four mini-banks. We can thus get the total BRAM blocks used for RFC structure.
Comparison in Fig. 11 indicates that our RFC design brings 35.93% reduction on
occupied BRAM blocks. Moreover, with almost same amount of used BRAM elements,
RFC can finish data-loading in one cycle and encoding/decoding in four cycles,
while CSC format usually needs 64 cycles to load data or decoding data
serially. With less extra hardware cost and similar storage compress ratio,
RFC structure achieves more regular data-access.

TABLE IV: Utilization & performance comparison between ours and [10].

| dsp | bram blocks | LUT | dsp efficiency | peak perf | frequency | fps
---|---|---|---|---|---|---|---
ours | 3544 | 1806 | 176776 | 0.322GOP/s/DSP | 1142GOP/S | 172Mhz | 271.25
[10] | 228 | 151 | 44457 | 0.202GOP/s/DSP | 46GOP/S | 188Mhz | 11.99

TABLE V: Performance comparison between ours and high-end GPUs.

| ours | 2080Ti-original | V100-original | 2080Ti(w/o C) | V100(w/o C) | 2080Ti-skip | V100-skip
---|---|---|---|---|---|---|---
throughput | 271.25 | 29.53 | 69.38 | 45.42 | 98.87 | 104 | 199.09
speed-up |  | 9.19 | 3.91 | 5.97 | 2.74 | 2.61 | 1.36

Figure 11: Storage cost of three data formats.

Overall performance: Our architecture is implemented on Xilinx XCKU-115 with
frequency of 172MHz. The resource utilization is demonstrated in Table. IV,
together with comparison with [10]. Experiments have proved that our design
has superiority on peak performance, throughput and DSP efficiency. In Table.
V, we compare the peak performance of ours and two GPUs. The original means
testing program is the original version of 2s-AGCN, and the skip means w/oC
together with input-skipping is applied. To fully use the memory in GPUs,
target model runs with 200 or 700 samples in one batch on 2080Ti and V100,
respectively. Although V100 has 14TFLOPS of peak performance, our pruning
methods provides 86% parameter reductions and 88% computation skipping
efficiency. Moreover, our layer-pipelined structure does not need to exchange
data with DRAM except for loading original input, which further boosts
performance. Compared with two main-stream GPUs, our accelerator provides
1.36x-9.19x of speedup, showing competitive performance.

##  VII Conclusion

In this article, we propose a software-hardware co-design for action
recognition GCNs: RFC-HyPGCN, including hybrid pruning method and a runtime
sparse feature compress architecture. Firstly, a hybrid pruning method is
explored on 2s-AGCN. Secondly, we propose an architecture based on the
balanced pruned model. Finally, a runtime sparse feature compact format is
designed to reduce zero-storage between layers. Experiments demonstrate that
compared with conventional unstructured pruning, our method achieves better
accuracy performance in most cases. The accelerator is implemented on Xilinx
XCKU-115 FPGA. At the cost of negligible working delay, RFC reduces 35.93% of
used BRAM and 23.24% of DSPs. Ours provides 22.62x speed-up and 59.41%
elevation on DSP efficiency over another work on accelerating action
recognition GCNs. On contrast to high-end GPUs, RFC-HyPGCN achieves
1.36x-9.47x speed-up on throughput.

## Acknowledgment

The paper is supported by National Natural Science Foundation of China
(No.61732018, No.61802420) and Pre-research Project of China
(No.31513010602-1).

## References

  * [1] F. M. Bianchi, D. Grattarola, and L. Livi, “Hierarchical representation learning in graph neural networks with node decimation pooling,” IEEE Transactions on Neural Networks and Learning Systems, 2020.
  * [2] K. Cheng, Y. Zhang, and X. He, “Skeleton-based action recognition with shift graph convolutional network,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [3] J. Gao, T. Zhang, and C. Xu, “I know the relationships: Zero-shot action recognition via two-stream graph convolutional networks and knowledge graphs,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 8303–8311, Jul. 2019.
  * [4] Z. Cao and T. Simon, “Realtime multi-person 2d pose estimation using part affinity fields,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
  * [5] Y. Xiu, J. Li, and H. Wang, “Pose Flow: Efficient online pose tracking,” in BMVC, 2018.
  * [6] https://github.com/LiQiang0307/MobilePose pytorch
  * [7] F. M. Bianchi and D. Grattarola, “Hierarchical representation learning in graph neural networks with node decimation pooling,” IEEE Transactions on Neural Networks and Learning Systems, pp. 1–13, 2020.
  * [8] J. Li and T. Zhang, “Sgcn: A graph sparsifier based on graph convolutional networks,” in Advances in Knowledge Discovery and Data Mining, pp. 275–287, 2020.
  * [9] L. Shi, Y. Zhang, et al., “Two-stream adaptive graph convolutional networks for skeleton-based action recognition,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12026–12035, 2019.
  * [10] L. Ding, Z. Huang, and G. Chen, “An fpga implementation of gcn with sparse adjacency matrix,” in 2019 IEEE 13th International Conference on ASIC (ASICON), pp. 1–4, IEEE, 2019.
  * [11] A. Shahroudy, J. Liu, T.-T. Ng, and G. Wang, “Ntu rgb+d: A large scale dataset for 3d human activity analysis,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
  * [12] A. Zisserman, J. Carreira, K. Simonyan, W. Kay, B. Zhang, C. Hillier, S. Vijayanarasimhan, F. Viola, T. Green, T. Back, P. Natsev, and M. Suleyman, “The kinetics human action video dataset,” 2017.
  * [13] C. Zhu, K. Huang, S. Yang, Z. Zhu, H. Zhang, and H. Shen, “An efficient hardware accelerator for structured sparse convolutional neural networks on fpgas,” IEEE Transactions on Very Large Scale Integration (VLSI) Systems, vol. 28, no. 9, pp. 1953–1965, 2020.
  * [14] L. Lu, J. Xie, R. Huang, J. Zhang, and W. Lin, “An efficient hardware accelerator for sparse convolutional neural networks on fpgas,” in 2019 IEEE 27th Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM), pp. 17–25, IEEE, 2019.
  * [15] N. Li, L. Liu, S. Wei, and S. Yin, “A high-performance inference accelerator exploiting patterned sparsity in cnns,” in 2020 IEEE 28th Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM), pp. 243–243, IEEE, 2020.
  * [16] X. Ma, F.-M. Guo, W. Niu, X. Lin, J. Tang, K. Ma, B. Ren, and Y. Wang, “Pconv: The missing but desirable sparsity in dnn weight pruning for real-time execution on mobile devices,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 5117–5124, 2020.
  * [17] T. Geng, A. Li, R. Shi, et al., “Awb-gcn: A graph convolutional network accelerator with runtime workload rebalancing,” in 2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), pp. 922–936, IEEE, 2020.
  * [18] B. Zhang, H. Zeng, and V. Prasanna, “Hardware acceleration of large scale gcn inference,” in 2020 IEEE 31st International Conference on Application-specific Systems, Architectures and Processors (ASAP), pp. 61–68, IEEE, 2020.