# Nonlinear Electromagnetic Waves in Magnetosphere of a Magnetar

Dan Mazur and Jeremy S. Heyl†
Department of Physics and Astronomy, University of British Columbia,
Vancouver, British Columbia, Canada, V6T 1Z1
†Email<EMAIL_ADDRESS>Canada Research Chair

(Accepted —. Received —; in original form —)

###### Abstract

We compute electromagnetic wave propagation through the magnetosphere of a
magnetar. The magnetosphere is modeled as the QED vacuum and a cold, strongly
magnetized plasma. The background field and electromagnetic waves are treated
nonperturbatively and can be arbitrarily strong. This technique is
particularly useful for examining non-linear effects in propagating waves.
Waves travelling through such a medium typically form shocks; on the other
hand we focus on the possible existence of waves that travel without evolving.
Therefore, in order to examine the nonlinear effects, we make a travelling
wave ansatz and numerically explore the resulting wave equations. We discover
a class of solutions in a homogeneous plasma which are stabilized against
forming shocks by exciting nonorthogonal components which exhibit strong
nonlinear behaviour. These waves may be an important part of the energy
transmission processes near pulsars and magnetars.

††pagerange: Nonlinear Electromagnetic Waves in Magnetosphere of a
Magnetar–References††pubyear: 2010

##  1 Introduction

The magnetosphere of a magnetar provides a particularly interesting medium for
the propagation of electromagnetic waves. Magnetars are characterized by
exceptionally large magnetic fields that can be several times larger than the
quantum critical field strength (Mereghetti, 2008). Because the magnetic
fields are so large, the fluctuations of the vacuum of quantum electrodynamics
(QED) influence the propagation of light. Specifically, the vacuum effects add
nonlinear terms to the wave equations of light in the presence large magnetic
fields. In addition, the magnetosphere of a magnetar contains a plasma which
alters the dispersion relationship for light. Because of the unique optical
conditions in the magnetospheres of magnetars, they provide excellent arenas
to explore nonlinear vacuum effects arising due to quantum electrodynamics.

The influence of QED vacuum effects from strong magnetic fields in the
vicinities of magnetized stars has previously been studied by several authors.
The combined QED vacuum and plasma medium is discussed in detail in the
context of neutron stars in Mészáros (1992). Vacuum effects have been found to
dominate the polarization properties and transport of X-rays in the strong
magnetic fields near neutron stars (Mészáros & Ventura, 1979, 1978; Meszaros &
Bonazzola, 1981; Meszaros et al., 1980; Galtsov & Nikitina, 1983). Detailed
consideration of magnetic vacuum effects is therefore critical to an
understanding of emissions from highly magnetized stars.

Most studies of waves in systems including plasmas or vacuum effects approach
the problem perturbatively, which limits the applicability of their results.
The purpose of the present paper is to examine the combined impact of the QED
vacuum and a magnetized plasma using nonperturbative methods to fully preserve
the nonlinear interaction between the fields. Neutron stars may be capable of
producing very intense electromagnetic waves, comparable to the ambient
magnetic field. For example, a coupling between plasma waves and seismic
activity in the crust could produce an Alfvén wave with a very large amplitude
(Blaes et al., 1989; Thompson & Duncan, 1995). Even if they may not be
produced directly, electromagnetic waves naturally develop through the
interactions between Alfvén waves. To lowest order in the size of the wave,
Alfvén waves do not suffer from shock formation (Thompson & Blaes, 1998)
whereas electromagnetic waves do (Heyl & Hernquist, 1998, 1999a); therefore,
how to stabilise the propagation of the the latter is the focus of this paper.

If such a magnetospheric disturbance results in electromagnetic waves of
sufficiently large amplitude and low frequency, then nonperturbative
techniques are required to characterize the wave. The importance of studying
such a system nonperturbatively is particularly well illustrated by the fact
that some nonlinear wave behaviour is fundamentally nonperturbative, as is
generally the case with solitons (Rajaraman, 1982). In order to handle the
problem nonperturbatively, we choose to study waves whose spacetime dependence
is described by the parameter $S=x-vt$, where $v$ is a constant speed of
propagation through the medium in the $\hat{\bf x}$-direction. In the study of
waves, one normally chooses the ansatz $e^{i(\omega t-{\bf k}\cdot{\bf x})}$.
However, in this picture, a numerical study would typically treat the self-
interactions of the electromagnetic field by summing the interactions of
finitely many Fourier modes. So, this ansatz conflicts with our goal of
studying the nonlinear interactions to all orders. In contrast, a plane wave
ansatz given by $S=x-vt$ allows us to study a simple wave structure to all
orders without any reference to individual Fourier modes.

We model a magnetar atmosphere by including the effects of arbitrarily strong
electromagnetic fields using a QED one-loop effective Lagrangian approach.
These effects are discussed in section 2.2. Plasma effects are included by
assuming free electrons moving under the Lorentz force without any self-
interactions. The model is that of a cold magnetohydrodynamic plasma and is
discussed in section 2.3. We have also assumed that the medium is homogeneous
in agreement with the travelling-wave ansatz. Of course the actual situation
is more complicated with a thermally excited plasma (e.g. Gill & Heyl, 2009)
and inhomogeneities — the latter can result in a whole slew of interesting
interactions between the wave modes (Heyl & Shaviv, 2000, 2002; Heyl et al.,
2003; Lai & Ho, 2003) that are especially crucial to our understanding of the
thermal radiation from their surfaces, but these are beyond the scope of this
paper.

The formation of electromagnetic shocks is expected to be an important
phenomenon for electromagnetic waves in the magnetized vacuum since
electromagnetic waves can evolve discontinuities under the influence of
nonlinear interactions (Lutzky & Toll, 1959; Zheleznyakov & Fabrikant, 1982;
Heyl & Hernquist, 1998). Such shocks can form even in the presence of a plasma
(Heyl & Hernquist, 1999b). In this study, through our explicit focus on
travelling waves, we examine an alternate class of solutions to the wave
equations which do not suffer this fate. Instead, they are stabilized against
the formation of discontinuities by nonlinear features. These waves travel as
periodic wave trains without any change to their form, such as wave steepening
or shock formation. Waves such as these may contribute to the formation of
pulsar microstructures (Chian & Kennel, 1987; Jenet et al., 2001).

##  2 Wave Equations

###  2.1 The Maxwell’s equations

The vacuum of QED in the presence of large magnetic fields can be described as
a non-linear optical medium (Heyl & Hernquist, 1997b). We also choose to treat
the effect of the plasma on the waves through source terms $\rho_{p}$ and
${\bf J}_{p}$; therefore, we begin by considering Maxwell’s equations in the
presence of a medium and plasma sources. In Heaviside-Lorentz units with
$c=1$, Maxwell’s equations can be used to derive the wave equations

$\displaystyle{\nabla}^{2}{\bf D}-\frac{\partial^{2}{\bf D}}{\partial t^{2}}$
$\displaystyle=$ $\displaystyle-{\nabla}\times({\nabla}\times({\bf D}-{\bf
E}))+$ (1) $\displaystyle\frac{\partial}{\partial t}({\nabla}\times({\bf
B}-{\bf H}))+{\bf\nabla}\rho_{p}+\frac{\partial{\bf J_{p}}}{\partial t}$
$\displaystyle{\nabla}^{2}{\bf H}-\frac{\partial^{2}{\bf B}}{\partial t^{2}}$
$\displaystyle=$ $\displaystyle-\nabla(\nabla\cdot({\bf B}-{\bf H}))-$ (2)
$\displaystyle\frac{\partial}{\partial t}({\nabla}\times({\bf D}-{\bf
E}))-{\nabla}\times{\bf J_{p}}.$

For clarity, we will avoid making cancellations or dropping vanishing terms.
We define the vacuum dielectric and inverse magnetic permeability tensors as
follows (Jackson, 1975)

$D_{i}=\varepsilon_{ij}E_{j},~{}~{}H_{i}=\mu^{-1}_{ij}B_{j}.$ (3)

In the next few sections we build a model describing travelling waves in a
magnetar’s atmosphere from these equations.

###  2.2 Vacuum Dielectric and Inverse Magnetic Permeability Tensors

In this section, we describe our model of the QED vacuum in strong background
fields in terms of vacuum dielectric and inverse magnetic permeability
tensors. These are most conveniently described in terms of two Lorentz
invariant combinations of the fields.

$\displaystyle K$ $\displaystyle=$
$\displaystyle\left(\frac{1}{2}\varepsilon^{\lambda\rho\mu\nu}F_{\lambda\rho}F_{\mu\nu}\right)^{2}=-(4{\bf
E}\cdot{\bf B})^{2}$ (4) $\displaystyle I$ $\displaystyle=$ $\displaystyle
F_{\mu\nu}F^{\mu\nu}=2(|{\bf B}|^{2}-|{\bf E}|^{2})$ (5)

In order to examine the nonlinear effects of the vacuum nonperturbatively, we
wish to use vacuum dielectric and inverse magnetic permeability tensors which
are valid to all orders in the fields. Analytic expressions for these tensors
were derived by Heyl & Hernquist (1997b) for the case of wrenchless fields
($K=-(4{\bf E}\cdot{\bf B})^{2}=0$) from the Heisenberg-Euler-Weisskopf-
Schwinger (Heisenberg & Euler, 1936; Weisskopf, 1936; Schwinger, 1951) one-
loop effective Lagrangian in Heyl & Hernquist (1997a) and expressed in terms
of a set of analytic functions.

$\displaystyle X_{0}(x)$ $\displaystyle=$ $\displaystyle
4\int_{0}^{x/2-1}\ln(\Gamma(v+1)){\rm
d}v+\frac{1}{3}\ln\left(\frac{1}{x}\right)$ (6) $\displaystyle~{}~{}+2\ln
4\pi-4\ln A-\frac{5}{3}\ln 2$ $\displaystyle~{}~{}-\left[\ln
4\pi+1+\ln\left(\frac{1}{x}\right)\right]x$
$\displaystyle~{}~{}+\left[\frac{3}{4}+\frac{1}{2}\ln\left(\frac{2}{x}\right)\right]x^{2}$
$\displaystyle X_{1}(x)$ $\displaystyle=$
$\displaystyle-2X_{0}(x)+xX_{0}^{(1)}(x)+\frac{2}{3}X_{0}^{(2)}(x)-\frac{2}{9}\frac{1}{x^{2}}$
(7) $\displaystyle X_{2}(x)$ $\displaystyle=$
$\displaystyle-24X_{0}(x)+9xX_{0}^{(1)}(x)$ (8)
$\displaystyle~{}~{}+(8+3x^{2})X_{0}^{(2)}(x)+4xX_{0}^{(3)}(x)$
$\displaystyle~{}~{}-\frac{8}{15}X_{0}^{(4)}(x)+\frac{8}{15}\frac{1}{x^{2}}+\frac{16}{15}\frac{1}{x^{4}}$

where

$X_{0}^{(n)}(x)=\frac{{\rm d}^{n}X_{0}(x)}{{\rm d}x^{n}}$ (9)

and

${\rm ln}A=\frac{1}{12}-\zeta^{(1)}(-1)=0.248754477.$ (10)

The tensors we need are derived in Heyl & Hernquist (1997b), except that we
have kept terms up to linear order in the expansion about $K=0$ instead of
dealing with the strictly wrenchless case. Our analysis therefore requires
that $K\ll B_{k}^{4}$.

$\varepsilon^{ij}=\Delta^{ij}-\frac{\alpha}{2\pi}\left[\frac{2}{I}X_{1}\left(\frac{1}{\xi}\right)+\frac{12K}{I^{3}}X_{2}\left(\frac{1}{\xi}\right)\right]B^{i}B^{j}$
(11)

$(\mu^{-1})^{ij}=\Delta^{ij}+\frac{\alpha}{2\pi}\left[\frac{2}{I}X_{1}\left(\frac{1}{\xi}\right)+\frac{K}{12I^{3}}X_{2}\left(\frac{1}{\xi}\right)\right]E^{i}E^{j}$
(12)

where

$\displaystyle\Delta^{ij}$ $\displaystyle=$
$\displaystyle\delta^{ij}\Bigg{[}1+\frac{\alpha}{2\pi}\Bigg{(}-2X_{0}\left(\frac{1}{\xi}\right)+\frac{1}{\xi}X_{0}^{(1)}\left(\frac{1}{\xi}\right)$
(13)
$\displaystyle+\frac{K}{4I^{2}}X_{1}\left(\frac{1}{\xi}\right)+\frac{K}{8I^{2}\xi}X_{1}^{(1)}\left(\frac{1}{\xi}\right)\Bigg{)}\Bigg{]},$

the fine-structure constant is $\alpha=e^{2}/4\pi$ in these units where we
have set $\hbar=c=1$, and

$\xi=\frac{1}{B_{k}}\sqrt{\frac{I}{2}}.$ (14)

Equations (11) and (12) define our model for the QED vacuum in a strong
electromagnetic field. Because we will focus on photon energies much lower
than the rest-mass energy of the electron, we have treated the vacuum as
strictly non-linear. It is not dispersive. The treatment of the dispersive
properties of the vacuum would require an effective action treatment (e.g.
Cangemi et al., 1995a, b) rather than the local effective Lagrangian treatment
used here.

###  2.3 Plasma

To investigate travelling waves, we choose our coordinate system so that the
$\hat{\bf x}$-direction is aligned with the direction of propagation. Then,
the spacetime dependence of the fields and sources is given by a single
parameter $S\equiv x-vt$ where $v$ is the constant phase velocity in the
$\hat{\bf x}$-direction of the travelling wave. At this point, we are choosing
to work in a specific Lorentz frame.

We model the plasma as a free electron plasma which enters the wave equation
through the source terms $\rho_{p}$ and ${\bf J}_{p}$. For electromagnetic
fields obeying the travelling wave ansatz, the sources must also obey the
ansatz. Then, $\rho_{p}$ and ${\bf J}_{p}$ are functions only of $S$. We
therefore treat them as an additional field which is integrated along with the
electromagnetic components of the field.

In order to perform the numerical ODE integration for the currents, we wish to
find expressions for $\frac{{\rm d}\rho_{p}}{{\rm d}S}$ and $\frac{{\rm d}{\bf
J}_{p}}{{\rm d}S}$. For travelling waves, the continuity equation is

$-v\frac{d\rho_{p}}{dS}+\delta^{ix}\frac{dJ_{p}^{i}}{dS}=0.$ (15)

where we are using index notation to label our explicitly cartesian
$\\{x,y,z\\}$ coordinate system. Repeated indices are summed. However,
whenever $x$ or $z$ appears as an index, it is fixed and does not run from 1
to 3.

We use equation (15) to rewrite the source terms from equation (1)

$\partial^{i}\rho_{p}+\frac{\partial{J}^{i}_{p}}{\partial
t}=\frac{1}{v}\frac{dJ_{p}^{i}}{dS}(\delta^{ix}-v^{2}).$ (16)

Similarly, the source term from equation (2) is

$({\nabla}\times
J_{p})^{i}=\varepsilon^{ixk}\left(\frac{dJ_{p}}{dS}\right)^{k}.$ (17)

To find an expression for $\frac{d{\bf J}_{p}}{dS}$ we express the current as
an integral over the phase-space distribution of the electrons and linearize
the plasma density.

${\bf J}_{p}=\int{f({\bf p}_{p})e{\bf v}_{p}d^{3}{\bf
p}_{p}}\approx\bar{\gamma}ne\bar{\bf v}_{p}$ (18)

where $\bar{\bf v}_{p}\equiv\frac{\int f({\bf p}_{p}){\bf v}_{p}d^{3}{\bf
p}_{p}}{\bar{\gamma}n}$ and $n$ is the mean electron density in the plasma in
the reference frame where $\bar{{\bf v}}_{p}=0$. It is important to note the
distinction between the mean plasma speed, $\bar{v}_{p}$, and the propagation
speed of the wave, $v$. The Lorentz factor
$\bar{\gamma}\equiv\frac{1}{\sqrt{1-\bar{v}_{p}^{2}}}$ accounts for a
relativistic increase in the plasma density since $d^{3}{\bf
x}=\gamma^{-1}d^{3}{\bf x}^{\prime}$.

Next, we take a time derivative of the current and express this in terms of a
three-force acting on the plasma.

$\displaystyle\frac{\partial{\bf J}_{p}}{\partial t}$ $\displaystyle=$
$\displaystyle ne\frac{\partial\bar{\gamma}\bar{{\bf v}}_{p}}{\partial t}$
(19) $\displaystyle=$ $\displaystyle\frac{ne}{m}{\bf F}$

Noting that ${\bf J}_{p}$ is a function only of $S$, we insert the Lorentz
force ${\bf F}=e({\bf E}+\bar{{\bf v}}_{p}\times{\bf B})$ and arrive at an
expression that can be substituted into the source terms, equations (16) and
(17), to find ${\bf J}_{p}(S)$.

$\displaystyle\frac{d{\bf J}_{p}}{dS}=-\frac{1}{v}\frac{e}{m}(en{\bf
E}+\bar{\gamma}^{-1}{\bf J}_{p}\times{\bf B})$ (20)

The equations we have given above describe a cold, relativistic,
magnetohydrodynamic plasma. Forces on the plasma arising due to pressure
gradients and gravity are neglected. Moreover, in our simulations, we neglect
the forces on the plasma due to the magnetic field. This approximation is
suitable if the plasma in question is a pair plasma, or for wave frequencies
much less than the cyclotron frequency.

As we will show in section 4 (see figure 6), the field configurations
generated in our simulation vary over timescales similar to the inverse of the
plasma frequency; the latter is about 9 orders of magnitude smaller than the
cyclotron frequency. This observation allows us to justify some aggressive
assumptions regarding the plasma response. As mentioned above, we may neglect
forces on the plasma from the magnetic field, and quantum effects are expected
to be small far away from the cyclotron resonance (Mészáros, 1992).

We also neglect thermal effects since the influence of the electromagnetic
fields will dominate over thermal motion. For strong background magnetic
fields such that $\frac{eB}{m}\gg kT$, the electrons will be confined to the
lowest Landau level, restricting thermal motion perpendicular to the
background magnetic field. As we will elaborate on in section 4, the greatest
nonlinear effects occur for waves with an electric field component oriented
along the background magnetic field. Because we are interested in waves with
amplitudes comparable to $B_{k}$, thermal motion is negligible relative to the
dynamics induced by the wave. We are therefore justified in neglecting thermal
effects in every direction for the cases of greatest interest.

If one combines Eq. (19) with Eq. (1), one sees that any nonlinearity in this
treatment must originate with the dielectric and permeability tensors — any
non-linearity that may originate from the plasma itself has been neglected
(c.f. Kozlov et al., 1979; Cattaert et al., 2005). The plasma is modelled as
strictly dispersive. In section 2.5, we confirm that this method of describing
the plasma is consistent with standard accounts in the weak-field, small-wave
limit.

###  2.4 Travelling Wave ODEs

The wave equations for travelling waves, are found by combining Maxwell’s
equations (1) and (2) with the continuity equation, (15). We also make the
plane-wave approximation, and assume that the fields and sources are described
by the parameter

$S=x-vt.$ (21)

The equations governing travelling wave propagation are

$\frac{d^{2}\psi^{i}(S)}{dS^{2}}=\frac{1}{v}\frac{{\rm d}J^{i}(S)}{{\rm
d}S}(\delta^{ix}-v^{2})$ (22)

$\frac{d^{2}\chi^{i}(S)}{dS^{2}}=-\varepsilon^{ixj}\frac{{\rm d}J^{j}(S)}{{\rm
d}S}$ (23)

The auxiliary vectors $\psi^{i}$ and $\chi^{i}$ are related to the electric
and magnetic fields.

$\displaystyle\psi^{i}(S)$ $\displaystyle=$
$\displaystyle(1-v^{2})\varepsilon^{ij}E^{j}+\delta^{ix}(\varepsilon^{xj}E^{j}-E^{x})-$
(24)
$\displaystyle~{}~{}(\varepsilon^{ij}E^{j}-E^{i})+\varepsilon^{ixk}v(B^{k}-(\mu^{-1})^{kj}B^{j})$
$\displaystyle\chi^{i}(S)$ $\displaystyle=$
$\displaystyle((\mu^{-1})^{ij}B^{j}-v^{2}B^{i})+\delta^{ix}(B^{x}-(\mu^{-1})^{xj}B^{j})-$
(25) $\displaystyle~{}~{}\varepsilon^{ixk}v(\varepsilon^{kj}E^{j}-E^{k})$

Equations (22) through (25) define a set of coupled ordinary differential
equations that can be integrated to solve for the travelling electric and
magnetic fields. Solving these equations requires that we have at hand the
vacuum dielectric and inverse magnetic permeability tensors as well as an
expression for $\frac{d{\bf J}_{p}}{dS}$. These were discussed in sections 2.2
and 2.3 respectively.

###  2.5 Weak field, Small Wave limit

In this section, we would like to demonstrate that our equations reduce to
standard expressions in the case of small background fields and small
electromagnetic waves. To make this comparison, it is also prudent to assume
waves have a spacetime dependence like $e^{i(\omega t-{\bf k}\cdot{\bf x})}$
instead of $x-vt$. By making this change, we can compare our other assumptions
with those made in standard textbook accounts directly.

Under the assumption that the fields and currents have the standard plane-wave
spacetime dependence, we can make the replacements $\frac{\partial}{\partial
t}\rightarrow-i\omega$ and $\nabla\rightarrow i{\bf k}$. We can then write a
second expression for $\frac{\partial{\bf J}}{\partial t}$.

$\frac{\partial{\bf J}_{p}}{\partial t}=-i\omega{\bf J}_{p}$ (26)

Setting equations (26) and (19) equal, we get

$\frac{e}{m}(\varepsilon^{ijk}J^{j}_{p}B^{k})+i\omega
J^{i}_{p}=\frac{e^{2}}{m}nE^{i}$ (27)

where we have used the nonrelativistic approximation, which is appropriate for
small waves. We will now specialize to a background magnetic field pointing in
the $\hat{\bf z}$-direction. We can then write this background field in terms
of the cyclotron frequency $\omega_{c}=\frac{eB}{m}$. The density $n$
determines the plasma frequency $\omega_{p}^{2}=n\frac{e^{2}}{m}$. Then, we
can write equation (27) as

$\left[-\omega_{c}\varepsilon^{ijz}-i\omega\delta^{ji}\right]J_{p}^{j}=\omega_{p}^{2}E^{i}$
(28)

We can solve this for ${\bf J}_{p}$ by writing a matrix equation

${\bf J}_{p}=\left(\begin{array}[]{ccc}-i\omega&-\omega_{c}&0\\\
\omega_{c}&-i\omega&0\\\ 0&0&-i\omega\end{array}\right)^{-1}\omega_{p}^{2}{\bf
E}$ (29)

inverting the matrix gives

${\bf
J}_{p}=\left(\begin{array}[]{ccc}\frac{i\omega}{\omega^{2}-\omega_{c}^{2}}&\frac{-\omega_{c}}{\omega^{2}-\omega_{c}^{2}}&0\\\
\frac{\omega_{c}}{\omega^{2}-\omega_{c}^{2}}&\frac{i\omega}{\omega^{2}-\omega_{c}^{2}}&0\\\
0&0&i/\omega\end{array}\right)\omega_{p}^{2}{\bf E}.$ (30)

Finally, we would like to use equation (30) to write the right hand side of
equation (1) in the small wave limit in a manner we can interpret as a
dielectric tensor.

Ignoring (for now) the contributions from the vacuum, and assuming an
approximately homogeneous plasma density, equation (1) simplifies to

$\displaystyle{\nabla}^{2}{\bf E}-\frac{\partial^{2}{\bf E}}{\partial t^{2}}$
$\displaystyle=$ $\displaystyle\frac{\partial{\bf J}_{p}}{\partial t}$ (31)
$\displaystyle=$ $\displaystyle-i\omega{\bf J}_{p}.$

If our macroscopic field is to obey ${\nabla}^{2}{\bf
D}-\frac{\partial^{2}{\bf D}}{\partial t^{2}}=0$, we can insert equation (30)
into (31) to obtain the following expression for the dielectric tensor due to
plasma effects:

$\varepsilon^{(p)}_{ij}=\left(\begin{array}[]{ccc}1-\frac{\omega_{p}^{2}}{\omega^{2}-\omega_{c}^{2}}&-i\frac{\omega_{c}}{\omega}\frac{\omega_{p}^{2}}{\omega^{2}-\omega_{c}^{2}}&0\\\
i\frac{\omega_{c}}{\omega}\frac{\omega_{p}^{2}}{\omega^{2}-\omega_{c}^{2}}&1-\frac{\omega_{p}^{2}}{\omega^{2}-\omega_{c}^{2}}&0\\\
0&0&1-\left(\frac{\omega_{p}^{2}}{\omega^{2}}\right)\end{array}\right)$ (32)

This expression is in agreement with the cold plasma dielectric tensor given
in Mészáros (1992). As noted above, our analysis neglects the off-diagonal
(Hall) terms, as is appropriate for pair plasmas or waves with frequencies
much less than the cyclotron frequency.

Because the vacuum effects are added explicitly in the form of dielectric and
magnetic permeability tensors, we only need to confirm that the weak field
limits of our expressions agree with the standard results. This confirmation
is done explicitly in Heyl & Hernquist (1997b).

In the weak field limit, the tensors given by equations (11) and (12) are

$\displaystyle\varepsilon_{ij}^{(v)}$ $\displaystyle=$
$\displaystyle\delta_{ij}+\frac{1}{45\pi}\frac{\alpha}{B_{k}^{2}}\left[2(E^{2}-B^{2})\delta_{ij}+7B_{i}B_{j}\right],$
(33) $\displaystyle\mu^{-1(v)}_{ij}$ $\displaystyle=$
$\displaystyle\delta_{ij}+\frac{1}{45\pi}\frac{\alpha}{B_{k}^{2}}\left[2(E^{2}-B^{2})\delta_{ij}-7E_{i}E_{j}\right]$
(34)

In the case of a weak background magnetic field pointing in the $\hat{\bf
z}$-direction, these become

$\displaystyle\varepsilon_{ij}^{(v)}$ $\displaystyle=$
$\displaystyle\left(\begin{array}[]{ccc}1-2\delta&0&0\\\ 0&1-2\delta&0\\\
0&0&1+5\delta\end{array}\right)$ (38) $\displaystyle\mu_{ij}^{-1(v)}$
$\displaystyle=$ $\displaystyle\left(\begin{array}[]{ccc}1-2\delta&0&0\\\
0&1-2\delta&0\\\ 0&0&1-6\delta\end{array}\right)$ (42)

with

$\delta=\frac{\alpha}{45\pi}\left(\frac{B}{B_{k}}\right)^{2}$ (43)

Again, this result agrees with the vacuum tensors given in Mészáros (1992).

In this limit, we may simply add together the contributions to the dielectric
tensor from the plasma and the vacuum according to

$\varepsilon_{ij}=\delta_{ij}+(\varepsilon_{ij}^{(p)}-\delta_{ij})+(\varepsilon_{ij}^{(v)}-\delta_{ij})$
(44)

with $\mu_{ij}^{-1}$ given entirely by the vacuum contribution.

We have thus recovered the standard result for a medium consisting of a plasma
and the QED vacuum in the weak field, small wave limit.

##  3 Solution Procedure

In total, there are 15 coupled non-linear ODEs which must be integrated to
produce a solution. Equations (22) and (23) define the electric and magnetic
fields as functions of $S$. In addition, we must simultaneously integrate
equation (20) which gives the plasma current as a function of $S$. Each of
these equations has three spatial components. Initial conditions are given for
each of the six field components, and the six derivatives of the field with
respect to $S$. The equations describing the electromagnetic fields do not
depend on the initial values of the current, but in principle these are the
three remaining initial conditions.

Figure 1: A comparison between the $\hat{\bf z}$ components of the electric
(solid) and magnetic fields (dashed) showing the nonorthogonal stabilizing
wave. On larger scales, the $B_{z}$ component is seen to have a periodic
envelope structure as in figure 2

.

Figure 2: The $B_{z}$ wave component as shown in figure 1 has an envelope
structure when viewed at larger scales. In this case, the $E_{z}$ wave
component has a nearly constant amplitude of $0.1B_{k}$.

.

The ODEs are solved using a variable stepsize Runge-Kutta method. In order to
translate between the ${\bf E}$-and-${\bf B}$ picture and the
${\bf\psi}$-and-${\bf\chi}$ picture, equations (24) and (25) must be solved
numerically at each time step, including the first step when the initial
conditions are given.

Tables of numerical values of the functions defined by equations (6) to (8),
as well as their derivatives, were computed in advance and these were used to
interpolate the values needed in the simulation using a standard cubic spline
interpolation algorithm. In producing these tables, expressions for the weak
and strong field limits were used in the appropriate regimes as this reduced
the numerical errors.

Aside from the initial conditions for the fields and derivatives, there is one
parameter in the model which must be selected. The density of the plasma,
given by $n$ in equation(18) is chosen to be

$n=10^{13}\mathrm{cm}^{-3}.$ (45)

This value corresponds to the Goldreich-Julian density for a star of period
$P\sim 1$ s, $\dot{P}\sim 10^{-10}$. The uniform background field is taken to
equal the quantum critical field strength

$B_{k}=\frac{m^{2}}{e}=4.413\times 10^{13}{\hbox{\rm G}}.$ (46)

##  4 Results

This study focuses on the case of waves propagating transverse to a large
background magnetic field. We have already chosen the direction of propagation
to be the $\hat{\bf x}$-direction through our definition of $S$ in Eq. (21).
We now choose the background magnetic field to point in the $\hat{\bf
z}$-direction. In this situation, the largest non-linear effects occur when
there is a large amplitude wave in the $\hat{\bf z}$ component of the electric
field. This is quite natural. The values of both the dielectric tensor of the
plasma (Eq. 32) and the weak-field limit of the vacuum dielectric tensor
(Eq.38) differ most from unity for this component; for strong fields the index
of refraction (as well as its derivative with respect to the field strength)
is largest for vacuum propagation in this mode (Heyl & Hernquist, 1997b).
Furthermore, the dominant three-photon interaction couples photons with the
electric field pointed along the global magnetic field direction with photons
whose magnetic field points along this direction. The three-point interaction
for photons whose electric field is perpendicular to the magnetic field with
parallel photons vanishes by the $CP-$invariance of QED (Adler, 1971).
Therefore, we focus on initial ($S=0$) conditions in which the dominant
component of the electric field points along in the $\hat{\bf z}$-direction
and that of the magnetic field along the $\hat{\bf y}$-direction.

In the classical vacuum, these initial conditions correspond to transverse,
linearly polarized sine-wave solutions that travel at the speed of light.
However, when the wave amplitudes are large, and there is a strongly
magnetized plasma, we find that there is a deviation from normal transverse
electromagnetic waves. In particular, in order to remain stable, a wave with
large $E_{z}$ and $B_{y}$ field components must also excite waves in the
$E_{y}$ and $B_{z}$ fields. These stabilizing wave components exhibit strong
non-linear characteristics (see figures 1 and 2). The symmetries of the wave
equation require a close correspondence between the $E_{y}$ and $B_{z}$
waveforms as well as between the $B_{z}$ and $B_{y}$ waveforms. For
simplicity, only one of each is plotted in the figures. The field strengths
are given in units of the quantum critical field strength, $B_{k}$.

As is apparent from Fig. 1 the dominant electric field along the direction of
the external magnetic field is essentially sinusoidal. Subsequent figures will
show that there is a small harmonic component. The waveform for the dominant
magnetic field component is similar. On the other hand, the magnetic field
along the direction of the electric field (the non-orthogonal component) is
smaller by nearly four orders of magnitude and obviously exhibits higher
harmonics. In particular if one expands the scale of interest (Fig. 2), the
magnetic field exhibits beating between two nearby frequencies with similar
power.

In order to examine the harmonic content of the waveforms, we perform fast
Fourier transforms (FFTs) on the signals produced in the simulations. We
present the results in terms of power spectra normalized by the square
amplitudes of the electric field of the waves. In these plots, the horizontal
axis is normalized by the frequency with the greatest power in the electric
field, so that harmonics can be easily identified.

Fig. 3 depicts the power spectra of the electric and magnetic fields along the
$\hat{z}$-direction for the wave depicted in Figs. 1 and 2. The conclusions
gathered from an examination of the waveforms are born out by the power
spectra. In particular the electric field is a pure sinusoidal variation to
about one part in ten thousand – the power spectrum of a pure sinusoid is
given by the dashed curved. The duration of the simulation is not an integral
multiple of the period of the sinusoid, resulting in a broad power spectrum
even for a pure sinusoid.

Figure 3: The solid curve depicts power spectra of the electric (upper) and
magnetic fields (lower) along the $\hat{\bf z}$-direction for the solution
depicted in Figs. 1 and 2. The inset focusses on the fundamental and the first
harmonic. The dashed curve follows the power spectrum of a single sinusoid for
the electric field and three sinusoids for the magnetic field. Near the peaks
the dashed curve is essentially indistinguishable from the solid one.

The power spectrum of the magnetic field follows the expectations gleaned from
the waveforms. In particular the fundamental and the first harmonic are
dominant, with the first harmonic having about one-third the power of the
fundamental. If one focusses on the fundamental, one sees that two frequencies
are involved. The envelope structure is produced by a beating between the
fundamental and a slightly lower frequency with a similar amount of power as
the first harmonic. Over the course of the simulation the envelope exhibits
two apparent oscillations; the lower frequency differs by two frequency bins,
so it is resolved separately from the fundamental as shown in the inset.

As the amplitude of the electric field increases the non-linear and non-
orthogonal features of the travelling wave increase. Fig. 4 shows that the
strength of the non-orthogonal magnetic field increases as the square of the
electric field, a hallmark of the non-linear interaction between the fields.
For the strongest waves studied with $E_{z}\approx 0.2B_{k}$ (the rightmost
point in the figure), the magnetic field, $B_{z}$, is about $10^{-4}B_{k}$
nearly one-percent of the electric field. The amplitude of the non-orthogonal
magnetic field is given by

$B_{z}=0.008B_{k}\left(\frac{E_{z}}{B_{k}}\right)^{2}$ (47)

for $B_{0}=B_{k}$. The coefficient is coincidentally very close to three-
quarters of the value of the fine-structure constant. It increases with the
strength of the background field.

Figure 4: Amplitude of the $B_{z}$ component plotted against the amplitude of
the $E_{z}$ component for a $B_{0}=B_{k}$ background field. The line is the
best-fitting power-law relation. The slope is consistent with a scaling
exponent equal to two.

For the strongest waves even the non-orthogonal magnetic field is strong, so
it can generate non-linearities in the electric field. Although the strongest
effect is around the fundamental, it is completely swamped by the fundamental
of the electric field. On the other hand, the magnetic field drives a first
and second harmonic in the electric field as seen in Fig. 5. The strength of
these harmonics is approximately given by the formula in Eq. (47) or
equivalently Fig. 4 if one substitutes the value of $B_{z}$ for $E_{z}$ and
uses result for $E_{z}$. This is essentially a six-order correction from the
effective Lagrangian. Because we have used the complete Lagrangian rather than
a term-by-term expansion, all of the corrections up to sixth order (and
further) are automatically included in the calculation.

Figure 5: This power spectrum demonstrates the development of nonlinear
effects in the $\hat{\bf z}$-component of the electric field as the amplitude
of the wave is increased in a $B_{0}=B_{k}$ background field. From top to
bottom the curves follow the solutions whose amplitude of $E_{z}$ equals
$0.01,0.08$ and $0.16B_{k}$. The inset focusses on the first and second
harmonics. Figure 6: The frequency (solid curve) and wavenumber (dashed curve)
of the travelling wave as a function of its phase velocity.

Figure 6 demonstrates how the frequency of the solutions varies with the speed
of propagation. Because there is no mode information stored directly in our
numerical solutions, we take the frequency to be rate that local minima in the
electric field pass a fixed observer. In general, we find that the frequency
of traveling waves increases as the phase velocity approaches the speed of
light, very closely following the formula

$\left(\frac{\omega}{\omega_{p}}\right)^{2}=\frac{v^{2}}{v^{2}-1}.$ (48)

This formula also results from an analysis of the dielectric tensor, Eq. (32).
The vacuum makes a small contribution to the wave velocity in this regime.

##  5 Conclusion

We have discussed techniques for computing electromagnetic waves in a strongly
magnetized plasma which nonperturbatively account for the field interactions
arising from QED vacuum effects. We applied these methods to the case of
travelling waves, which have a spacetime dependence given by the parameter
$S=x-vt$. Travelling waves can be described without any decomposition into
Fourier modes and this is ideal for exploring the nonperturbative aspects of
waves.

The main result from this analysis is the observation that electromagnetic
waves in a strongly magnetized plasma can self-stabilize by exciting
additional nonorthogonal wave components. In the cases studied above, a large
amplitude excitation of the electromagnetic field, for example, from the
coupling between Alfvén waves to starquakes, can induce nonlinear waves which
are stabilized against the formation of shocks. The result is a periodic wave
train with distinctly nonlinear characteristics. Such structures may play a
role in forming pulsar microstructures.

This result demonstrates that shock formation is not a necessary outcome for
waves in a critically magnetized plasma. It is possible that nonlinear
features of a wave can stabilize it against shock formation. The shock-wave
solutions generally decrease in magnitude, dissipating energy along the way
unless they travel into a low-field region, so the self-stabilizing nonlinear
waves are the only ones that keep their shape and energy content intact as
they propagate. It is not yet clear what set of conditions will determine if a
particular wave will self-stabilize or collapse to form a shock or how plasma
inhomogenaities will affect the propagation of a travelling wave train. These
are issues which can be clarified in future work.

## References

  * Adler (1971) Adler S. L., 1971, Ann. Phys., 67, 599
  * Blaes et al. (1989) Blaes O., Blandford R., Goldreich P., Madau P., 1989, ApJ, 343, 839
  * Cangemi et al. (1995a) Cangemi D., D’hoker E., Dunne G., 1995a, Phys. Rev. D, 51, 2513
  * Cangemi et al. (1995b) Cangemi D., D’hoker E., Dunne G., 1995b, Phys. Rev. D, 52, 3163
  * Cattaert et al. (2005) Cattaert T., Kourakis I., Shukla P. K., 2005, Physics of Plasmas, 12, 012319
  * Chian & Kennel (1987) Chian A., Kennel C. F., 1987, NASA STI/Recon Technical Report N, 88, 16622
  * Galtsov & Nikitina (1983) Galtsov D. V., Nikitina N. S., 1983, Zhurnal Eksperimental noi i Teoreticheskoi Fiziki, 84, 1217
  * Gill & Heyl (2009) Gill R., Heyl J. S., 2009, Phys. Rev E, 80, 036407 (8 pages)
  * Heisenberg & Euler (1936) Heisenberg W., Euler H., 1936, Z.PHYS., 98, 714
  * Heyl & Hernquist (1998) Heyl J., Hernquist L., 1998, Physical Review D, 58, 43005
  * Heyl & Hernquist (1997a) Heyl J. S., Hernquist L., 1997a, Phys. Rev. D, 55, 2449
  * Heyl & Hernquist (1997b) Heyl J. S., Hernquist L., 1997b, Journal of Physics A Mathematical General, 30, 6485
  * Heyl & Hernquist (1999a) Heyl J. S., Hernquist L., 1999a, Phys. Rev. D, 59, 045005 (5 pages)
  * Heyl & Hernquist (1999b) Heyl J. S., Hernquist L., 1999b, Phys. Rev. D, 59, 045005
  * Heyl & Shaviv (2000) Heyl J. S., Shaviv N. J., 2000, MNRAS, 311, 555
  * Heyl & Shaviv (2002) Heyl J. S., Shaviv N. J., 2002, Phys. Rev. D, 66, 023002 (4 pages)
  * Heyl et al. (2003) Heyl J. S., Shaviv N. J., Lloyd D., 2003, MNRAS, 342, 134
  * Jackson (1975) Jackson J. D., 1975, Classical electrodynamics. Wiley, New York
  * Jenet et al. (2001) Jenet F. A., Anderson S. B., Prince T. A., 2001, ApJ, 558, 302
  * Kozlov et al. (1979) Kozlov V. A., Litvak A. G., Suvorov E. V., 1979, Soviet Journal of Experimental and Theoretical Physics, 49, 75
  * Lai & Ho (2003) Lai D., Ho W. C., 2003, Physical Review Letters, 91, 071101
  * Lutzky & Toll (1959) Lutzky M., Toll J. S., 1959, Phys. Rev., 113, 1649
  * Mereghetti (2008) Mereghetti S., , 2008, The strongest cosmic magnets: Soft Gamma-ray Repeaters and Anomalous X-ray Pulsars
  * Mészáros (1992) Mészáros P., 1992, High-energy radiation from magnetized neutron stars.. University of Chicago Press, Chicago
  * Meszaros & Bonazzola (1981) Meszaros P., Bonazzola S., 1981, ApJ, 251, 695
  * Meszaros et al. (1980) Meszaros P., Nagel W., Ventura J., 1980, ApJ, 238, 1066
  * Mészáros & Ventura (1978) Mészáros P., Ventura J., 1978, Phys. Rev. Lett., 41, 1544
  * Mészáros & Ventura (1979) Mészáros P., Ventura J., 1979, Phys. Rev. D, 19, 3565
  * Rajaraman (1982) Rajaraman R., 1982, Solitons and instantons. North-Holland Amsterdam etc.
  * Schwinger (1951) Schwinger J., 1951, Phys. Rev., 82, 664
  * Thompson & Blaes (1998) Thompson C., Blaes O., 1998, Phys. Rev. D, 57, 3219
  * Thompson & Duncan (1995) Thompson C., Duncan R. C., 1995, MNRAS, 275, 255
  * Weisskopf (1936) Weisskopf V., 1936, Mathematisk-Fysiske Meddelelser, 14
  * Zheleznyakov & Fabrikant (1982) Zheleznyakov V., Fabrikant A., 1982, Name: Zh. Eksp. Teor. Fiz


# Real-Time Online Skeleton Extraction and Gesture Recognition on Pepper

author 1, author 2
Axel Lefrant1,2, Jean-Marc Montanier1
1Protolab, Softbank Robotics Europe, Paris, France
2INSA, University of Lyon, France
<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

We present a multi-stage pipeline for simple gesture recognition. The novelty
of our approach is the association of different technologies, resulting in the
first real-time system as of now to conjointly extract skeletons and recognise
gesture on a Pepper robot. For this task, Pepper has been augmented with an
embedded GPU for running deep CNNs and a fish-eye camera to capture whole
scene interaction. We show in this article that real-case scenarios are
challenging, and the state-of-the-art approaches hardly deal with unknown
human gestures. We present here a way to handle such cases.

##  1 INTRODUCTION

Figure 1: Example of the “clapping” action.

Pepper is an humanoid robot developed as a business welcoming and information
gathering platform 111https://www.softbankrobotics.com/emea/en/robots/pepper.
To achieve his given tasks of helping people, Pepper has a basic face
recognition system making it able to recognise emotions. However,
communication with Pepper is currently limited to interacting with its on-
board tablet. Desired use-case would be direct interaction with the robot. In
this work, we enable Pepper to recognise simple gestures by analysing the
skeleton of human extracted through the camera (Figure 1).

Deep learning techniques recently achieved outstanding result, especially in
the computer vision field. Deep convolutional networks topped the major
competitions like ImageNet [Russakovsky et al., 2014] or COCO [Lin et al.,
2014]. On top of that, deep networks improved robustness and accuracy in
different sub-fields of computer vision (e.g. activity recognition, skeleton
extraction) which were hardly feasible before. Hailed as an essential aspect
of today’s artificial intelligence, we investigate the potential and relevance
of deep networks in the context of image recognition with mainstream robot
like Pepper.

For the system to be intuitive and effective, it requires low-latency gesture
detection and recognition. In the best case, the observational latency [Ellis
et al., 2013], that is the time needed to observe enough frames for an
accurate decision, would be limited to the first frame of the gesture, which
is unfeasible in practise. A reasonable performance latency would be a
detection near the end of the gesture. To achieve this objective, we arbitrary
decide on a system running at around $5$ frames per seconds, in order to
sample enough frames for accurate detection at the end of the gesture while
minimising the computational latency. The speed constraint guides us in the
choice of a solution being fast and accurate.

We aim to deploy the developed system on a Pepper robot. The current hardware
of Pepper is insufficient for the task, which compel us to modify it (Figure
2). Skeleton extractors based on convolutional networks consists of hundreds
of convolutional layers, which cannot be run on the Pepper hardware. We opt
for an embedded GPU of NVIDIA, the Jetson TX2, as this board is a good
compromise between speed, memory and can be integrated inside a Pepper.

Figure 2: Augmented Pepper: (a) DC-DC converter to adapt Pepper’s battery
voltage to Jetson TX2 voltage; (b) Embedded Jetson TX2 board; (c) RGB Fish-eye
rotated of $90^{\circ}$.

Also, both the central and mouth cameras have limited field of view, meaning
that no adult usually fits entirely in the camera frame. Our solution is to
include an additional fish-eye camera below the robot’s Tablet, which allows
both close and far interactions.

The developed system is composed of two hierarchical modules 222source code
available at removedforanonymisation, 1) a deep convolutional network
extracting the skeletons of all humans body in a frame captured by a camera,
and 2) a simultaneous detection of specific individuals and recognition of
their gestures.

The originality of our approach is the association of different technologies,
resulting in the first real-time system as of now to conjointly extract
skeletons and recognise gesture on a Pepper robot. Moreover, we highlight a
specific use-case due to the use of action recognition in the real-world:
handling large portions of gestures and positions not known by the system.

Section 2 discusses our work on the estimation of body-poses. Section 3
explains how the gesture are recognised. Section 4 focuses on handling the
real-case scenario where unknown gestures are performed. Finally, Section 5
concludes.

##  2 SKELETON DETECTION

Table 1: Evaluation of ArtTrack, Openpose and Mask R-CNN for body-pose
estimation on COCO validation set.

|  | Announced AP
---
(on test set)
| Measured AP
---
(on val set)
| Measured AR
---
(on val set)
| Measured
---
time
| CPU
---
Usage
| GPU
---
Usage
ArtTrack [Insafutdinov et al., 2016a] | / | 7.9 % | 33.6 % |  | 637.78s
---
128ms/frame (7.8 fps)
| 25% use
---
2.0 GB memory
| 30-80% use (fluctuating)
---
7.9 GB memory
Openpose [Cao et al., 2016] |  $\sim$57 % | 44.6 % | 53.9 % |  | 611.8s
---
122.4ms/frame (8.2 fps)
| 50% use
---
1.2 GB memory
| 90-95% (stable)
---
2.3 GB memory
Mask R-CNN [He et al., 2017] | 62.7 % | 65.4 % | 71.7 % |  | 1237.7s
---
247.5 ms/frame (4.0 fps)
| 33% use
---
1.6 GB memory
| 40-80% use (fluctuating)
---
3.6 GB memory

###  2.1 Related Works

####  2.1.1 Body-pose Estimation

Following the results of single body-pose estimator [Toshev and Szegedy,
2013], [Tompson et al., 2014], [Wei et al., 2016], [Newell et al., 2016],
body-pose estimation was extended to multiple human pose detection. There are
two different approaches for solving the multiple human pose estimation
problem.

The top-down approach first detects persons in the image, crops the image
around all the persons detected and finally performs human pose estimation on
each cropped image [Papandreou et al., 2017], [He et al., 2017].

Opposed to the top-down approach, the bottom-up approach detects all the body
parts in the image (location and label) and associates these parts to their
respective human instance [Insafutdinov et al., 2016b], [Insafutdinov et al.,
2016a], [Cao et al., 2016].

The task of gesture recognition requires both accurate joint position even in
the case of a distorted image (fish-eye camera) and a fast estimation to
reliably capture the gesture sequence. Also, we would like this step to be as
generic as possible and without prior assumption about the environment (e.g.
location of people, number of people). To fulfil these requirements we are
using multi-person 2D body-pose estimator networks.

###  2.2 Evaluation Method

In this section, we will evaluate which among the major and publicly available
networks (i.e. Openpose [Cao et al., 2016], Mask-RCNN [He et al., 2017] and
ArtTrack [Insafutdinov et al., 2016a]) are best suited for our application on
the Jetson TX2.

####  2.2.1 Dataset

Common Objects in Context (COCO) [Lin et al., 2014] — The COCO dataset was
created for the COCO challenge, including human body-pose estimation. This
dataset contains around $200.000$ labelled images containing $1.7$ million
labelled keypoints from over $150,000$ people instances.

The COCO dataset is using the Average Precision (AP) for evaluating skeleton
proposals. Each skeleton proposal is categorised as either a True-Positive
(TP) (there was a skeleton) and False Positive (FP) (there was no skeleton),
used to compute the AP measure:

$\text{AP}=\frac{|\text{TP}|}{|\text{TP}|+|\text{FP}|}$ (1)

COCO defines the Object Keypoint Similarity (OKS) to measure whether the
skeleton proposal corresponds to a groundtruth skeleton. The OKS measure is
adjusted by a threshold value. The final AP is an average over AP computed for
different OKS threshold, from $0.5$ to $0.95$ with a stepsize of $0.05$.

####  2.2.2 Implementation Details

The three networks, Openpose, ArtTrack and Mask-RCNN, were trained on the COCO
dataset. However, the test set annotations are not public. It appears that the
validation set was not used to train the three networks, so we can safely use
this set to evaluate the networks ($5000$ images in the validation set)
without having to retrain the whole networks.

###  2.3 Results

####  2.3.1 Baseline

The results (Table 1) were obtained on a personal computer, equipped with a
Intel Xeon CPU and a NVIDIA GTX 1070 GPU. The measured AP on ArtTrack is
surprisingly low, although the evaluation method is the same as the two other
ones. Also, no result were reported on COCO dataset on the original paper
(only on MPII Human Pose dataset [Andriluka et al., 2014]), so no comparison
is available. The other surprising result is the GPU memory usage, $7.9$ GB,
which seems too much compared to the network architecture and to the other two
networks. The authors were contacted, but no solution was found to explain or
resolve this surprising result.

Concerning Openpose, the measured AP is slightly below what is announced in
the paper. We only evaluate the ‘vanilla’ Openpose, whereas the announced AP
was obtained with additional multi-scale testing, post-processing with CPM
[Wei et al., 2016], and filtering with an object detector. It is worth noting
that the GPU usage of Openpose is kept almost constantly at top intensity,
meaning that the API efficiently parallelises the workload.

Mask R-CNN displays the best accuracy but runs approximately two times slower
than the others due to the multi-pipeline of the Faster R-CNN architecture.
Looking at the CPU and GPU usage, we can also deduce that the underlying code
can be optimised to further take advantage of the GPU (i.e. parallel image
scaling and non-maxima suppression).

This experiment enables us to validate our experimental procedure and
disregard ArtTrack due to its poor performance.

####  2.3.2 Jetson TX2

As the final system runs on the Jetson TX2 board, performance and time must be
evaluated on it for a relevant choice. Given an equivalent image resolution,
Openpose is faster while being less accurate than Mask R-CNN. From the two
baselines, we draw the conclusion that Openpose is more suited for our
application as we target a minimal framerate of about $5$ fps. Qualitative
evaluation reveals that the skeleton detection is accurate enough for front
face and close interaction with the robot.

Table 2: Evaluation of Openpose and Mask R-CNN on Jetson TX2.

|  | Measured AP
---
on val set
| Measured AR
---
on val set
| Measured
---
time
| Openpose
---
320x176
22.7 % | 30.5 % |  | 240.0 ms/frame
---
(4.2 fps)
| Mask R-CNN
---
-1x176
29.6 % | 36.2 % |  | 587.7 ms/frame
---
(1.7 fps)

###  2.4 Openpose Optimisation

####  2.4.1 TensorRT

We now aim to further improve Openpose speed, by using the NVIDIA TensorRT
technology which takes the embedded board capabilities into account. TensorRT
is a deep learning framework, substituting traditional frameworks for
optimised inference on NVIDIA platform. In addition to optimise model layout,
TensorRT supports quantisation, which is the process of encoding the weight of
the convolutional filter to lower-precision, compressing the size of the
model, and usually accelerating inference since more operations can be
performed. The Jetson TX2 only supports quantising the float on 16 bits
(fp16), instead of the default encoding on 32 bits (fp32).

####  2.4.2 TensorRT porting results

We now evaluate Openpose optimised with the TensorRT framework on two
different image resolution, $320\times 240$ and $160\times 112$, as the image
resolution is a trade-off parameter between accuracy and speed. The results
are presented in Tables 3 and 4. In all conditions, the measured AP and AR
stay the same when using TensorRT with or without quantisation. On the
$320\times 240$ resolution (Table 3), Openpose TensorRT fp32 is twice as fast
as ‘vanilla’ Openpose, and Openpose TensorRT fp16 is thrice as fast ‘vanilla’
Openpose. On the $160\times 112$ (Table 4), Openpose TensorRT fp16 is ‘only’
twice as fast as the ‘vanilla’ Openpose for the same resolution. Also, the
performance of the detector with this resolution is too low to always
accurately detect skeletons for front view interaction, especially with
unusual body-pose.

The difference in the factor of speed-up between the two resolutions is due to
the time taken for network inference in the $160\times 112$ resolution becomes
similar to the time to group the keypoints to form human instances. As
TensorRT only improves the network inference part, the speed-up becomes less
significant.

The final resolution used in the application is $320\times 240$ pixels with
fp16 quantisation because it allows us to accurately detect skeletons during
front-view interaction while respecting our speed target. An additional
advantage is the lack of rescale since the camera natively supports this
resolution.

Table 3: Evaluation of Openpose TensorRT for $320\times 240$ resolution.

|  | Measured AP
---
on val set
| Measured AR
---
on val set
| Measured
---
time
| Openpose Caffe
---
320x240
30.6 % | 39.2 % |  | 737.3 ms/frame
---
1.4 fps
| Openpose TensorRT fp32
---
320x240
30.6 % | 39.2 % |  | 347.9 ms/frame
---
2.9 fps
| Openpose TensorRT fp16
---
320x240
30.9 % | 39.1 % |  | 189.3 ms/frame
---
5.3 fps

Table 4: Evaluation of Openpose TensorRT for $160\times 112$ resolution.

|  | Measured AP
---
on val set
| Measured AR
---
on val set
| Measured
---
time
| Openpose Caffe
---
160x112
10.4 % | 16 % |  | 167.1 ms/frame
---
6.0 fps
| Openpose TensorRT fp32
---
160x112
10.4 % | 16 % |  | 116.9 ms/frame
---
8.55 fps
| Openpose TensorRT fp16
---
160x112
10.4 % | 16 % |  | 72.4 ms/frame
---
13.8 fps

###  2.5 Conclusion

Thanks to state-of-the-art review and evaluation, we were able to find a
network, i.e. Openpose, satisfying our speed constraint while enabling
accurate skeleton detection. Further speed-ups with TensorRT make Openpose
ample for our given task.

##  3 GESTURE RECOGNITION

From the skeletons estimated by Openpose, we investigate on how to efficiently
detect and recognise gestures.

###  3.1 Related Works

####  3.1.1 Deep learning based recognition

Deep learning has been successfully applied for skeleton-based gesture
recognition, to automatically learn human representations from raw skeletal
data. Both skeleton-based RNN [Du et al., 2015], [Shahroudy et al., 2016],
[Liu et al., 2017b] and CNN [Pham et al., 2018], [Yan et al., 2018] have been
researched for gesture recognition, while RNNs have also been applied for
online action detection from streaming skeleton data [Li et al., 2016], [Liu
et al., 2017a]. While being promising, these networks are currently
computationally expensive, especially on our GPU already filled with skeleton
extraction.

####  3.1.2 Skeletal features

Contrary to deep networks that can implicitly learn representations, machine
learning methods [Han et al., 2016], [Presti and Cascia, 2016] require hand-
crafted skeletal features that globally model the spatial structure and
temporal dynamics of human skeleton. Common spatial features include
displacement-based representations [Luo et al., 2013], joint orientations
features [Jin and Choi, 2013], [Xia et al., 2012]. Popular temporal features
are the velocity and the acceleration of the joints and angles since they
encode quite well motions. Fusing these different modalities allow space-time
human representations.

The previous spatial and temporal features are considered low-level, since
they are directly extracted from the skeleton data in euclidean space.
Additional human representations can also be constructed from feature
transition to other topological space, i.e. manifold representations
[Vemulapalli et al., 2014], [Slama et al., 2015]. Another mid-level features
are the body-parts representations that partially take into account the
physical structure of the human body [Evangelidis et al., 2014], [Borghi et
al., 2016].

####  3.1.3 Gesture classification

One approach is to consider action recognition as a time series problem in
which the characteristics of body postures and their dynamics over time are
extracted to represent a human action. Hiden Markov Models (HMMs) were
extensively used in the field of recognition, e.g., hand gesture recognition
[Yin, 2014], [Piyathilaka and Kodagoda, 2013]. The advantage of HMM is its
ability to learn the temporal dependencies of the gesture from a limited
training set. However, HMMs are inherently only able to model dynamic movement
and ignore static poses.

The template-based methods treat gesture and action recognition as a database
query problem which matches data with templates in the database [Evangelidis
et al., 2014], [Fan et al., 2012]. The intuitive idea in the bag of features
is to generate a code-book of features, and represent each video with an
histogram. The visual codebook is generally built by clustering descriptors
extracted from the objects of interest, the centers of the clusters are
considered the visual words. Online detection and recognition has been
achieved using the template-based method [Fan et al., 2012], [Meshry et al.,
2016].

###  3.2 Online Gesture Detection and Recognition

The method selected for the gesture recognition is a template-based method.
Specifically, we decided to implement the algorithm proposed by [Meshry et
al., 2016] because of its performances and applicability to real-time
scenarios.

####  3.2.1 Algorithm

The algorithm is learning from sequence manually annotated and representing
the demonstration of an individual performing a single action or gesture, from
start to finish. The training set is composed of $S$ sequences, and each
sequence is composed of $N$ frames. Given a sequence of $N$ set of body-joints
representing the successive joints position when realising a specific action,
meaningful features are extracted at each frame. The features are composed of
the Moving Pose [Zanfir et al., 2013] features and angle features [Nowozin and
Shotton, 2012]. The final form of the descriptor, called gesturelets, is as
follow:

$g=[P(t),\alpha\frac{P(t)}{dt},\beta\frac{P^{2}(t)}{dt^{2}}]\gamma[\frac{\Theta(t)}{dt}]$
(2)

where $\alpha$, $\beta$ and $\gamma$ are hyper-parameters (detailed in section
3.4.1) weighting the contribution of each feature, and $P(t)$,
$\frac{P(t)}{dt}$, $\frac{P^{2}(t)}{dt^{2}}$ and $\frac{\Theta(t)}{dt}$
respectively refer to relative joint position from hip center, joint speed,
joint acceleration and joint angle speed. This descriptor is normalised to
unit norm for scale invariance.

We cluster the descriptors from all the sequence of the training set to form a
codebook of $K$ centroid. Each action sequence is represented as an histogram,
of size $K$, which counts how many times each cluster has been assigned to
each gesturelets of the sequence. Soft binning can be applied to adjust the
cluster importance depending on its distance from the gesturelet when
performing k-nearest neighbours.

From the $S$ histograms (one per sequence), a multi-class linear SVM
classifier is trained in a one-vs-all approach, although other linear
classifier can be used. From this classifier, we only retain the weight vector
$w$, which characterises the separating hyperplane, i.e. how much the clusters
are important in regard to their related gesture.

Given a sequence of $N$ frames, we extract a descriptor for each frame. Then
for each of these descriptors, we find its k-nearest neighbours with the
clusters of the codebook. We compute an action score for each mapping by
looking at the weight vector of the action. A detected action is triggered
when the score of the maximum subarray, computed in linear time with Kadane’s
algorithm [Bentley, 1984], exceeds a threshold. This threshold is determined
by searching for the score threshold that minimises the binary classification
error for its related action class.

The online procedure is identical, except that the sequence is coming frame by
frame, so we continuously perform maximum subarray search until an action is
detected.

####  3.2.2 Experimentation

A missing element of the original detector is a probability score measuring
the relevance of the detection and confidence on the predicted action class.
To compute a probability score, we store the cluster matching of each
descriptor and when a gesture is triggered, we compute the histogram of the
sequence from the start and end value deduced by the detector. Then this
histogram is classified by the SVM, alongside a calibrated logistic regression
giving a probability score. We call this implementation ‘vanilla’.

A particularity of the method developed is to decide when a detection is
triggered. By default, a detection is triggered only when an action threshold
is overtaken. Another way is to also require that the following $s$
consecutive scores are negatives. The idea is to avoid short detection or
multiple same detection for one gesture. This is done at the expense of the
observational latency, since it should force the system to wait until the end
of the gesture. However, depending on the dataset used for learning, this
method may inhibit some gestures since the start and end of the gesture may be
similar. To overcome this problem, the authors of [Meshry et al., 2016]
proposed a mixing histogram scheme. We call this solution ‘generated’.

###  3.3 Dataset

NTU RGB-D [Shahroudy et al., 2016] — This action recognition dataset consists
of $56.880$ samples of $60$ different actions containing RGB videos, depth map
sequences, 3D skeletal data, and infrared videos for each sample. In our case,
only the 3D skeletal data is necessary for training the classifier. The
dataset is captured by $3$ Microsoft Kinect v.2 cameras concurrently,
capturing accurate 3D skeletal data of $25$ major body joints, at each frame
and from different point of view (front view, +$45^{\circ}$, -$45^{\circ}$).

SBRE neutral poses and detection — When interacting with a robot, the user
often stays motionless in a pose that could be qualified as ‘neutral’. This
category of movement is particularly important to us, but not present in the
original NTU RGB-D dataset. We have therefore generated a new dataset to
fulfil our needs. It consists of samples of individuals standing clearly in
front of the fish-eye camera equipped by Pepper. When capturing the set,
Pepper is in its standing pose, and the subject is positioned at either close
(0.5m), medium (1.5m) and far (3m) distance from the camera. The videos are
recorded at 30fps with the resolution of $640\times 480$ pixels.

Cleaning of the NTU RGB-D dataset has been done to remove some variability in
the action demonstration, enabling the SVM classifier to better learn the
action representations. The SBRE neutral dataset as well as the list of files
sampled from the NTU RGB-D dataset can be freely downloaded
333removedforanonymisation.

###  3.4 Results

####  3.4.1 Hyper-parameters optimisation

The detector is composed of six hyper-parameters, specifically three for
features extraction ($\alpha,\beta,\gamma$), one controlling the size of the
codebook (i.e. number of cluster), one for the number of cluster neighbour in
the soft binning and one for how much we weight the positive scores against
the negative scores when we learn the threshold. The search for good values of
these hyper-parameters is done with cross-validation on the recognition task
only. Our assumption is that the hyper-parameters found in these conditions
are acceptable for the detection task on a fish-eye camera.

The cross-validation result (Table 5) emphasises the angle features, while the
joint acceleration seems to have less influence in the subsequent
classification.

Table 5: Best hyper-parameters found with coarse-grain cross-validation.

$\alpha$ | $\beta$ | $\gamma$ |  | codebook
---
size
| m
---
neighbours
| weight
---
factor
| recognition
---
score
0.8 | 0.4 | 1 | 2000 | 2 | 3 | 0.888

####  3.4.2 Evaluation metric for gesture detection

One challenge is to effectively measure the detection performance of our
detector. We divide the NTU RGB-D dataset of pre-segmented clips into a
training set and a test set, and we randomly concatenate the sequences of the
test set to effectively make long sequences of multiple actions. While this
trick allows us to evaluate the performance of the detector, the final score
will not be totally representative of the performance of the detector in
practical application (e.g. fish-eye distortions, noisy skeletons).

Also, the metric and way to measure the detection are essentials. A simple
solution is to rely on precision and recall measures and finally compute a F1
score. While this approach is often used in other works and make comparisons
straightforward, the details needed for its computation is generally not
provided (e.g. are the computation based on the frames or events) and it
usually does not offer much insight on the quality and failure of the
detector. We adopt the metrics designed specifically for activity recognition
[Ward et al., 2011]. This evaluation procedure introduces a scoring segments
approach, making possible binary comparison between the groundtruth and
prediction for each action. Besides the regular measures of true-positives,
true-negatives, false-positives and false-negatives, additional metrics are
designed to provide better information on the reason of failure (such as
fragmentation, deletion or insertion). Finally, the evaluation’s computation
is fully detailed which allows easy reproduction.

####  3.4.3 Original implementation evaluation

We evaluate first the ‘vanilla’ and ‘generated’ approaches with a set of 5
actions to recognise, ‘bowing’, ‘clapping’, ‘drinking’, ‘jumping’, and
‘waving’. These particular actions were chosen because they use almost all
limb, while being globally different. In total, after cleaning, the training
set contains $1091$ sequences for a total of around $12000$ frames.

For each condition, training has been done 100 times, and each time one
evaluation has been performed. As proposed in [Ward et al., 2011], for each
action, all the measures can be gathered in two groups: one showing the
results obtained in the positive cases (when the class to detect is present)
and the one showing the negative cases (when the class to detect is not
present). Each measure represents the average over all training performed.

The Figure 6 presents these data as pie charts for the ‘bowing’ action. This
action has been chosen because it is the hardest to classify in our set, and
therefore the one that shows more of the classifiers’ performances. The
analysis of a ‘NULL’ action are also presented: these corresponds to lack of
output (i.e. no detection) when a gesture is presented to the classifier. All
results for all actions are available in Appendix 1.

The first line presents the analysis of the classification of the ‘bowing’
action with the ‘vanilla’ approach. The positive and negative rates have been
computed by taking into account only the 5 actions the system has been trained
on. In the second line we include ‘neutral’ actions (from the SBRE Neutral
dataset) to the test. One can see that with this additional action in the test
set, the true negative detection rates ($tnr$) drops from 0.97% to 0.63%,
meaning the ‘vanilla’ approach has difficulty handling real-case scenarios.

The third line presents the results obtained with the ‘generated’ approach
when tested on the 5 actions used for training plus the neutral actions. It
appears that the ‘generated’ approach brings back the rate of true negative
detections ($tnr$) close to the one obtained with the ‘vanilla’ approach.
However this solution comes at the cost of a large reduction of the true
positive detection rate ($tpr$) from 0.653% to 0.527%. Moreover, we can see
that with the ‘generated’ approach the $tnr$ is also strongly reduced when
neutral poses are presented. The fact that the insertion rate ($ir$) has
increased indicates that the detector fired-off less often.

Table 6: Bowing performance on all categories

| Bowing Negative | Bowing Positive | NULL Negative
---|---|---|---
COCO dataset |  |  |
COCO + neutral |  |  |
Generated |  |  |
Neutral |  |  |

###  3.5 Conclusion

We deduce that the ‘vanilla’ approach is sensible to spurious detection. The
‘generated’ approach mitigates the problem, at the expense of detection
performance.

##  4 HANDLING NEUTRAL POSES

###  4.1 Method proposed

To get around this problem, we add ‘neutral’ poses to the training set. We
expect that this will improve the performance for two reasons. First, this
action class frequently resets the system when the subject is not interacting
with the robot, and thus avoid detecting many false-positives coming from
aggregated noise. Indeed, when an action is detected, it clears the cost
matrix, even though it does not allow simultaneous action (e.g. clapping while
sitting). Second, this additional action class would help for the detection
triggering issue. It forces the system to attribute negative score when the
action is finished, because the user comes back to this initial pose, which
corresponds or is contained in the ‘neutral’ class. This implementation is
referred as ‘neutral’.

###  4.2 Results

The last line of the Figure 6 presents the results obtained with the ‘neutral’
approach. This approach keeps the $tnr$ at a value as high as for the
‘vanilla’ approach tested on the 5 actions to detect, but also maintains a
high value for the $tpr$ as well as the $tnr$ for neutral actions.

While the ‘neutral’ approach seems to be the most trivial to handle ‘neutral’
actions, it is the one that obtains the best results in our quantitative
evaluation. It appears that the ‘neutral’ actions are close enough between
each others to be classified together and differentiated from the other
actions. This approach allows performance equal to the ‘vanilla’ approach,
while also handling real-case scenarios.

###  4.3 Conclusion

The three recognition methods presented in this article have been tested in
our complete framework. While a rigorous quantitative assessment was not
possible with this framework, our qualitative evaluation of the running system
confirmed us that the ‘neutral’ approach is the best way to classify neutral
poses.

##  5 CONCLUSION

We have demonstrated an effective system for improving the human-Pepper
interaction. Specifically, we have proposed a multi-step system composed of a
body-pose estimator, a simple tracker and a gesture detection and recognition
module on a super-charged Pepper robot. Our achievement was to run the whole
pipeline with reasonable results in real-time despite the limited
computational power available. Moreover, we noticed the drop of performance on
real-case scenarios, and proposed a simple solution to overcome this problem.
The resulting system is suitable for effective gesture recognition in real-
world scenarios.

To improve each module independently and get even better performance,
additional speed-up can be achieved by tweaking the Openpose architecture for
mobile-friendly device (e.g. MobileNets [Howard et al., 2017]) [Huang et al.,
2016], or by manually pruning filters [Huang et al., 2018] to reduce the
number of parameters and operations.

One recurring problem on gesture recognition is the lack of 3D skeletons,
providing world-frame computation and invariance. A simple solution would be
to actually estimate the 3D body-pose, by using depth sensors or networks able
to infer the 3D location of the joint in the world frame [Güler et al., 2018].

## ACKNOWLEDGEMENTS

We express our most grateful acknowledgement to the members of the Protolab
team of Softbank Robotics Europe (Maxime Busy, Maxime Caniot and Edouard
Lagrue) without whom this work would not have been possible. We also thank
Alexandre Mazel, the director of software innovation of Softbank Robotics
Europe, for his technical knowledge and support on Pepper and the Jetson TX2.

## REFERENCES

  * [Andriluka et al., 2014] Andriluka, M., Pishchulin, L., Gehler, P., and Schiele, B. (2014).  2d human pose estimation: New benchmark and state of the art analysis.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
  * [Bentley, 1984] Bentley, J. (1984).  Programming pearls: Algorithm design techniques.  Commun. ACM, 27(9):865–873.
  * [Borghi et al., 2016] Borghi, G., Vezzani, R., and Cucchiara, R. (2016).  Fast gesture recognition with multiple stream discrete hmms on 3d skeletons.  In 23rd International Conference on Pattern Recognition (ICPR).
  * [Cao et al., 2016] Cao, Z., Simon, T., Wei, S., and Sheikh, Y. (2016).  Realtime multi-person 2d pose estimation using part affinity fields.  arXiv 1611.08050.
  * [Du et al., 2015] Du, Y., Wang, W., and Wang, L. (2015).  Hierarchical recurrent neural network for skeleton based action recognition.  In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1110–1118.
  * [Ellis et al., 2013] Ellis, C., Masood, S. Z., Tappen, M. F., Laviola, Jr., J. J., and Sukthankar, R. (2013).  Exploring the trade-off between accuracy and observational latency in action recognition.  Int. J. Comput. Vision, 101(3):420–436.
  * [Evangelidis et al., 2014] Evangelidis, G., Singh, G., and Horaud, R. (2014).  Continuous gesture recognition from articulated poses.  In ECCV Workshops.
  * [Fan et al., 2012] Fan, Z., Li, G., Haixian, L., Shu, G., and Jinkui, L. (2012).  Star skeleton for human behavior recognition.  In 2012 International Conference on Audio, Language and Image Processing, pages 1046–1050.
  * [Güler et al., 2018] Güler, R. A., Neverova, N., and Kokkinos, I. (2018).  Densepose: Dense human pose estimation in the wild.  arXiv 1802.00434.
  * [Han et al., 2016] Han, F., Reily, B., Hoff, W., and Zhang, H. (2016).  Space-time representation of people based on 3d skeletal data: A review.  arXiv 1601.01006.
  * [He et al., 2017] He, K., Gkioxari, G., Dollár, P., and Girshick, R. B. (2017).  Mask R-CNN.  arXiv 1703.06870.
  * [Howard et al., 2017] Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. (2017).  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  arXiv 1704.04861.
  * [Huang et al., 2016] Huang, J., Rathod, V., Sun, C., Zhu, M., Korattikara, A., Fathi, A., Fischer, I., Wojna, Z., Song, Y., Guadarrama, S., and Murphy, K. (2016).  Speed/accuracy trade-offs for modern convolutional object detectors.  arXiv 1611.10012.
  * [Huang et al., 2018] Huang, Q., Zhou, S. K., You, S., and Neumann, U. (2018).  Learning to prune filters in convolutional neural networks.  arXiv 1801.07365.
  * [Insafutdinov et al., 2016a] Insafutdinov, E., Andriluka, M., Pishchulin, L., Tang, S., Levinkov, E., Andres, B., and Schiele, B. (2016a).  Articulated multi-person tracking in the wild.  arXiv 1612.01465.
  * [Insafutdinov et al., 2016b] Insafutdinov, E., Pishchulin, L., Andres, B., Andriluka, M., and Schiele, B. (2016b).  Deepercut: A deeper, stronger, and faster multi-person pose estimation model.  arXiv 1605.03170.
  * [Jin and Choi, 2013] Jin, S.-Y. and Choi, H.-J. (2013).  Essential body-joint and atomic action detection for human activity recognition using longest common subsequence algorithm.  In Computer Vision - ACCV 2012 Workshops, pages 148–159.
  * [Li et al., 2016] Li, Y., Lan, C., Xing, J., Zeng, W., Yuan, C., and Liu, J. (2016).  Online human action detection using joint classification-regression recurrent neural networks.  European Conference on Computer Vision.
  * [Lin et al., 2014] Lin, T., Maire, M., Belongie, S. J., Bourdev, L. D., Girshick, R. B., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. (2014).  Microsoft COCO: common objects in context.  arXiv 1405.0312.
  * [Liu et al., 2017a] Liu, C., Li, Y., Hu, Y., and Liu, J. (2017a).  Online action detection and forecast via multitask deep recurrent neural networks.  In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1702–1706.
  * [Liu et al., 2017b] Liu, J., Wang, G., Duan, L., Hu, P., and Kot, A. C. (2017b).  Skeleton based human action recognition with global context-aware attention LSTM networks.  arXiv 1707.05740.
  * [Luo et al., 2013] Luo, J., Wang, W., and Qi, H. (2013).  Group sparsity and geometry constrained dictionary learning for action recognition from depth maps.  In 2013 IEEE International Conference on Computer Vision.
  * [Meshry et al., 2016] Meshry, M., Hussein, M. E., and Torki, M. (2016).  Linear-time online action detection from 3d skeletal data using bags of gesturelets.  2016 IEEE Winter Conference on Applications of Computer Vision (WACV).
  * [Newell et al., 2016] Newell, A., Yang, K., and Deng, J. (2016).  Stacked hourglass networks for human pose estimation.  arXiv 1603.06937.
  * [Nowozin and Shotton, 2012] Nowozin, S. and Shotton, J. (2012).  Action points: A representation for low-latency online human action recognition.  Technical report.
  * [Papandreou et al., 2017] Papandreou, G., Zhu, T., Kanazawa, N., Toshev, A., Tompson, J., Bregler, C., and Murphy, K. P. (2017).  Towards accurate multi-person pose estimation in the wild.  arXiv 1701.01779.
  * [Pham et al., 2018] Pham, H.-H., Khoudour, L., Crouzil, A., Zegers, P., and Velastin, S. A. (2018).  Exploiting deep residual networks for human action recognition from skeletal data.  Computer Vision and Image Understanding.
  * [Piyathilaka and Kodagoda, 2013] Piyathilaka, L. and Kodagoda, S. (2013).  Gaussian mixture based hmm for human daily activity recognition using 3d skeleton features.  In 2013 IEEE 8th Conference on Industrial Electronics and Applications (ICIEA).
  * [Presti and Cascia, 2016] Presti, L. L. and Cascia, M. L. (2016).  3d skeleton-based human action classification: A survey.  Pattern Recognition, 53:130 – 147.
  * [Russakovsky et al., 2014] Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M. S., Berg, A. C., and Li, F. (2014).  Imagenet large scale visual recognition challenge.  arXiv 1409.0575.
  * [Shahroudy et al., 2016] Shahroudy, A., Liu, J., Ng, T., and Wang, G. (2016).  NTU RGB+D: A large scale dataset for 3d human activity analysis.  arXiv 1604.02808.
  * [Slama et al., 2015] Slama, R., Wannous, H., Daoudi, M., and Srivastava, A. (2015).  Accurate 3d action recognition using learning on the grassmann manifold.
  * [Tompson et al., 2014] Tompson, J., Jain, A., LeCun, Y., and Bregler, C. (2014).  Joint training of a convolutional network and a graphical model for human pose estimation.  arXiv 1406.2984.
  * [Toshev and Szegedy, 2013] Toshev, A. and Szegedy, C. (2013).  Deeppose: Human pose estimation via deep neural networks.  arXiv 1312.4659.
  * [Vemulapalli et al., 2014] Vemulapalli, R., Arrate, F., and Chellappa, R. (2014).  Human action recognition by representing 3d skeletons as points in a lie group.  In Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, CVPR ’14, pages 588–595.
  * [Ward et al., 2011] Ward, J. A., Lukowicz, P., and Gellersen, H. W. (2011).  Performance metrics for activity recognition.  ACM Trans. Intell. Syst. Technol., 2(1):6:1–6:23.
  * [Wei et al., 2016] Wei, S., Ramakrishna, V., Kanade, T., and Sheikh, Y. (2016).  Convolutional pose machines.  arXiv 1602.00134.
  * [Xia et al., 2012] Xia, L., Chen, C. C., and Aggarwal, J. K. (2012).  View invariant human action recognition using histograms of 3d joints.  In 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pages 20–27.
  * [Yan et al., 2018] Yan, S., Xiong, Y., and Lin, D. (2018).  Spatial temporal graph convolutional networks for skeleton-based action recognition.  arXiv 1801.07455.
  * [Yin, 2014] Yin, Y. (2014).  Real-time continuous gesture recognition for natural multimodal interaction.
  * [Zanfir et al., 2013] Zanfir, M., Leordeanu, M., and Sminchisescu, C. (2013).  The moving pose: An efficient 3d kinematics descriptor for low-latency action recognition and detection.  In 2013 IEEE International Conference on Computer Vision, pages 2752–2759.

## ANNEXE 1

| Vanilla | Vanilla neutral | Generate | Neutral
---|---|---|---|---
Bowing |  tpr: 0.65 uer: 0.15 usr: 0.08 fr: 0.05 dr: 0.06 tnr: 0.97 oer: 0.01 osr: 0.01 mr: 0.00 ir: 0.01 |  tpr: 0.70 uer: 0.12 usr: 0.07 fr: 0.05 dr: 0.05 tnr: 0.62 oer: 0.03 osr: 0.03 mr: 0.00 ir: 0.31 |  tpr: 0.52 uer: 0.20 usr: 0.09 fr: 0.04 dr: 0.14 tnr: 0.98 oer: 0.00 osr: 0.00 mr: 0.00 ir: 0.00 |  tpr: 0.61 uer: 0.17 usr: 0.10 fr: 0.51 dr: 0.65 tnr: 0.98 oer: 0.01 osr: 0.01 mr: 0.00 ir: 0.01
Clapping |  tpr: 0.65 uer: 0.13 usr: 0.07 fr: 0.01 dr: 0.13 tnr: 0.97 oer: 0.01 osr: 0.01 mr: 0.00 ir: 0.02 |  tpr: 0.65 uer: 0.14 usr: 0.06 fr: 0.01 dr: 0.14 tnr: 0.97 oer: 0.00 osr: 0.01 mr: 0.00 ir: 0.02 |  tpr: 0.61 uer: 0.15 usr: 0.06 fr: 0.01 dr: 0.17 tnr: 0.98 oer: 0.01 osr: 0.00 mr: 0.00 ir: 0.00 |  tpr: 0.65 uer: 0.13 usr: 0.07 fr: 0.01 dr: 0.14 tnr: 0.98 oer: 0.01 osr: 0.00 mr: 0.00 ir: 0.01
Drinking |  tpr: 0.73 uer: 0.11 usr: 0.08 fr: 0.03 dr: 0.05 tnr: 0.95 oer: 0.01 osr: 0.01 mr: 0.00 ir: 0.03 |  tpr: 0.74 uer: 0.10 usr: 0.09 fr: 0.02 dr: 0.06 tnr: 0.93 oer: 0.01 osr: 0.01 mr: 0.00 ir: 0.05 |  tpr: 0.64 uer: 0.14 usr: 0.09 fr: 0.03 dr: 0.09 tnr: 0.98 oer: 0.01 osr: 0.01 mr: 0.00 ir: 0.00 |  tpr: 0.72 uer: 0.11 usr: 0.09 fr: 0.02 dr: 0.06 tnr: 0.96 oer: 0.01 osr: 0.01 mr: 0.00 ir: 0.02
Jumping |  tpr: 0.74 uer: 0.08 usr: 0.10 fr: 0.04 dr: 0.03 tnr: 0.97 oer: 0.01 osr: 0.01 mr: 0.00 ir: 0.02 |  tpr: 0.73 uer: 0.09 usr: 0.11 fr: 0.02 dr: 0.04 tnr: 0.97 oer: 0.01 osr: 0.01 mr: 0.00 ir: 0.02 |  tpr: 0.69 uer: 0.10 usr: 0.11 fr: 0.04 dr: 0.05 tnr: 0.98 oer: 0.01 osr: 0.00 mr: 0.00 ir: 0.01 |  tpr: 0.73 uer: 0.08 usr: 0.11 fr: 0.03 dr: 0.04 tnr: 0.97 oer: 0.01 osr: 0.01 mr: 0.00 ir: 0.01
Waving |  tpr: 0.72 uer: 0.10 usr: 0.08 fr: 0.01 dr: 0.10 tnr: 0.98 oer: 0.01 osr: 0.00 mr: 0.00 ir: 0.01 |  tpr: 0.72 uer: 0.10 usr: 0.08 fr: 0.01 dr: 0.10 tnr: 0.98 oer: 0.00 osr: 0.00 mr: 0.00 ir: 0.01 |  tpr: 0.71 uer: 0.11 usr: 0.07 fr: 0.01 dr: 0.11 tnr: 0.99 oer: 0.01 osr: 0.00 mr: 0.00 ir: 0.00 |  tpr: 0.72 uer: 0.10 usr: 0.08 fr: 0.01 dr: 0.09 tnr: 0.99 oer: 0.01 osr: 0.00 mr: 0.00 ir: 0.01
Table 7: Action performance on all categories