# Entropic uncertainty relations for general SIC-POVMs and MUMs

Shan Huang  School of Physics, Nanjing University, Nanjing 210093, China
Institute for Brain Sciences and Kuang Yaming Honors School, Nanjing
University, Nanjing 210023, China  Zeng-Bing Chen<EMAIL_ADDRESS>School of
Physics, Nanjing University, Nanjing 210093, China  Shengjun Wu
<EMAIL_ADDRESS>School of Physics, Nanjing University, Nanjing 210093, China
Institute for Brain Sciences and Kuang Yaming Honors School, Nanjing
University, Nanjing 210023, China

(August 27, 2024)

###### Abstract

We construct inequalities between Rényi $\alpha$-entropy and the indexes of
coincidence of probability distributions, based on which we obtain improved
state-dependent entropic uncertainty relations for general symmetric
informationally complete positive operator-valued measures (SIC-POVMs) and
mutually unbiased measurements (MUMs). We show that our uncertainty relations
for general SIC-POVMs and MUMs can be tight for sufficiently mixed states,
and, moreover, comparisons to the numerically optimal results are made via
information diagrams.

##  I INTRODUCTION

Incompatible observables cannot be measured with certainty simultaneously,
though contrary to the general cognition of the physical world based on
macroscopic experience, this is a fundamental element of quantum mechanics.
Heisenberg made the first statement of this kind of uncertainty of quantum
mechanics heisenberg , and formulated the first uncertainty relation

$\Delta P\Delta Q\geq\frac{\hbar}{2},$ (1)

where $\Delta P$ and $\Delta Q$ denote the standard deviation of momentum and
position along the same direction respectively. Robertson generalized it to
two arbitrary observables Robertson

$\Delta X\Delta Y\geq\frac{1}{2}|\langle\psi|[X,Y]|\psi\rangle|,$ (2)

where [X,Y] denotes the commutator between X and Y.

Though clear and elegant enough, the standard deviation way of measuring
uncertainty sometimes can be quite strange Deutsch ; Dam ; Bia  and it turns
out to be inappropriate in applications of information theory. On the other
hand, entropy is found to be a more universal and effective measure of
uncertainty Deutsch ; Coles ; Friedland ; JJ , and entropic uncertainty
relations (EURs) have many applications in quantum information theory. The
lower bound on conditional min-entropy can characterize how much randomness
one can extract from a source qrandom , and more over, as entanglement between
two systems reduces the uncertainty (lower bound on entropy) of measurements
performed on one system provided that the other one is accessible, EURs are
useful in entanglement witnessing entangle1 ; entangle2 ; entangle3 . EURs are
also important in the security proof of quantum cryptography as they measure
how much information is possibly leaked to an eavesdropper crypt1 ; crypt2 .
(See more applications in the review CBTW  and references therein.)

Any projective measurement made in one base cannot reveal any information
stored in bases that are mutually unbiased to it, and this property endows
mutually unbiased bases (MUBs) with a special role in quantum information
theory. Based on the work of Deutsch Deutsch  and Kraus Kraus , Maassen and
Uffink proved the famous tight state-independent uncertainty relation for two
MUBs in terms of Shannon entropy MU , and a generalization to multiple MUBs
has also been explored ID ; J3 ; J5 ; BW ; WW ; WYM . However, an analytic
construction of more than three MUBs in general dimensions has not been found,
and the existence of complete MUBs in non-prime-power dimensional spaces such
as $d=6$ is still an open question Ben .

While general symmetric informationally complete positive operator-valued
measures (SIC-POVMs) JKAC  and mutually unbiased measurements (MUMs) mum  are
positive-operator-valued measures with interesting properties similar to MUBs,
and a complete set of them can be constructed analytically in all dimensions
gsicexist ; mum , uncertainty relations have been naturally generalized to
take into consideration more generalized measurements Raste ; indexofgsic ;
indexofmum ; WWS  like them. In two recent works EURs are also constructed
from quantum designs design1 ; design2 . In this paper, we focus on
uncertainty relations for SIC-POVMs and MUMs and deal with them under a
unified framework.

This paper is structured as follows. In Sec. II we introduce some necessary
notations and review the concepts of entropy, SIC-POVMs and MUMs. In Sec. III
we propose entropic uncertainty relations for general SIC-POVMs, and in Sec.
IV uncertainty relations for MUMs are constructed. In Sec V, we make further
discussions and draw a brief conclusion.

##  II Preliminaries

A positive operator-valued measure (POVM) $\mathcal{P}$ on a $d$-dimensional
Hilbert space $\mathcal{H}_{d}$ consists of a set of positive semi-definite
operators that sum up to identity: $\mathcal{P}=\big{\\{}P_{i}\big{|}P_{i}\geq
0,\sum_{i}P_{i}={\bf 1}_{d}\big{\\}}$. The probability distribution induced by
performing a POVM measurement $\mathcal{P}$ on a quantum state $\rho$ is
denoted by $\mathscr{P}=\big{(}p_{1},p_{2},\cdots\big{)}$, where
$p_{i}=Tr(P_{i}\rho)$ is the probability of obtaining the $ith$ result; the
corresponding index of coincidence is defined as the sum of the squares of the
probabilities, i.e.,

$I(\mathcal{P}|\rho)=I(\mathscr{P})=\sum_{i}p_{i}^{2}.$ (3)

Table 1: Notations and meaning. 1 | $\rho$ | Density matrix
---|---|---
2 | $L$ | The length of a probability distribution
3 | $\mathcal{H}_{d}$ |  $d$-dimensional Hilbert space
4 | ${\bf 1}_{d}$ |  $d$-dimensional Identity matrix
5 | P | A boldfaced letter is, if not specified
|  | otherwise, a finte set of POVMs
6 | $\mathscr{P}$ | A probability distribution
7 | $I(\mathscr{P})$ | Index of coincidence of $\mathscr{P}$
8 |  $I({\bf P}|\rho$) | The sum of indexes of coincidence
|  | induced by performing P on $\rho$
9 | $H_{\alpha}({\bf P}|\rho)$ | Sum of Rényi-$\alpha$ entropies for the
|  | measurements P performed on $\rho$
|  | Two probability distributions over $L$
10 |  $\mathscr{P}_{x}^{L}[c]$,$\mathscr{P}_{y}^{L}[c]$ | outcomes defined in Eq. (10) and
|  | Eq. (11), and $I(\mathscr{P}_{x}^{L}[c])=I(\mathscr{P}_{y}^{L}[c])=c$

The Shannon entropy of $\mathscr{P}$, defined by
$H(\mathscr{P})=H(\mathcal{P}|\rho)=-\sum_{i=1}^{d}p_{i}\log_{2}{p_{i}}$,
gives a measure of the uncertainty for the measurement outcomes. Rényi
generalized it to a family of entropies Renyi

$\displaystyle
H_{\alpha}(\mathcal{P}|\rho)=\frac{1}{1-\alpha}\log_{2}\left(\sum_{i=1}p_{i}^{\alpha}\right),\hskip
5.0pt(\alpha>0,\alpha\neq 1)$

which reduces to Shannon entropy in the limitation $\lim_{\alpha\to
1}H_{\alpha}(\mathcal{P}|\rho)=H_{1}(\mathcal{P}|\rho)=H(\mathcal{P}|\rho)$.
Following Ref. HT , we call the range of the map
$\mathscr{P}\rightarrow\big{[}I(\mathscr{P}),H_{\alpha}(\mathscr{P})\big{]}$
information diagrams.

For any finite set of POVMs
$\textbf{P}=\\{\mathcal{P}^{1},\mathcal{P}^{2},\cdots\\}$ performed on $\rho$,
we consider the sum of indexes of coincidence

$I(\textbf{P}|\rho)=\sum_{m=1}^{|\bf{P}|}I(\mathcal{P}^{m}|\rho),$ (4)

and the sum of entropies $H_{\alpha}({\bf
P}|\rho)=\sum_{m}H_{\alpha}(\mathcal{P}^{m}|\rho)$.

Table I contains some notations that are frequently used in this paper.

###  II.1 Symmetric informationally complete POVM

A POVM on $\mathcal{H}_{d}$ is said to be symmetric informationally complete
(SIC-POVMs) JKAC  if it consists of $d^{2}$ rank-1 operators
$\textbf{S}=\big{\\{}S_{i}\big{\\}}$ such that
$Tr(S_{i}S_{j})=\frac{d\delta_{ij}+1}{d^{2}(d+1)}$. From the geometric point
of view, with $S_{i}=\frac{1}{d}|\phi_{i}\rangle\langle\phi_{i}|$, SIC-POVM
comprises of $d^{2}$ subnormalized equiangular vectors
$\big{\\{}\frac{1}{d}|\phi_{i}\rangle\big{\\}}$ in $C^{d}$ as
$|\langle\phi_{i}|\phi_{j}\rangle|^{2}=\frac{d\delta_{ij}+1}{d+1}$ and
$\sum_{i=1}^{d^{2}}\frac{1}{d}|\phi_{i}\rangle\langle\phi_{i}|={\bf 1}_{d}$.
Although research is still ongoing to prove or disprove the existence of SIC-
POVMs for general $d$, analytic and numerical results confirmed its existence
for dimensions up to 67 sicexis .

SIC-POVMs is informationally complete, as when performed on a system the
resulting probability distributions fully reveal all the information of the
corresponding density matrix. More concretely, any density matrix $\rho$ can
be constructed from the probabilities $\\{p_{j}\\}$ induced by SIC-POVM, and
with $Tr(\rho S_{j})=p_{j}$ there is $\rho=\sum_{j}p_{j}\left[d(d+1)S_{j}-{\bf
1}_{d})\right]$ indexofgsic .

By generalizing the method proposed in Ref. WYM , Rastegin obtained Raste

$\displaystyle
I(\textbf{S}|\rho)=\sum_{i=1}^{d^{2}}p_{i}^{2}=\frac{1+Tr(\rho^{2})}{d(d+1)},$
(5)

where $p_{i}=Tr(\rho S_{i})$.

Generalizations of SIC-POVM to that with elements of any rank have been
explored in Refs. gsic1 ; gsic2 , and in Ref. gsicexist  the authors proved
the existence of general SIC-POVMs in all dimensions by giving the explicit
construction. Any general SIC-POVM $\textbf{S}_{g}=\big{\\{}S_{i}\big{\\}}$
$(i=1,2\cdots,d^{2})$ is a POVM satisfying

$\displaystyle Tr(S_{i}S_{i})=a,\left(\forall i,1/d^{3}<a\leq 1/d^{2}\right)$
$\displaystyle Tr(S_{i}S_{j})=\frac{1-ad}{d(d^{2}-1)}\hskip 5.0pt(\forall
i\neq j).$

It is shown in Ref. indexofgsic  that

$\displaystyle
I(\textbf{S}_{g}|\rho)=\frac{(ad^{3}-1)Tr(\rho^{2})+d(1-ad)}{d(d^{2}-1)}$ (6)

###  II.2 Mutually unbiased measurements

We say two orthonormal bases $\\{|b_{i}^{1}\rangle\\}$ and
$\\{|b_{j}^{2}\rangle\\}$ $(1\leq i,j\leq d)$ in $\mathcal{H}_{d}$ are
mutually unbiased bases (MUBs) I ; WF ; KR ; PR  if the inner products between
their basis vectors satisfy $|\langle
b^{1}_{i}|b^{2}_{j}\rangle|=\frac{1}{\sqrt{d}}$ $(\forall 1\leq i,j\leq d)$.
For any $d\geq 2$, one can find at least three MUBs and at most $d+1$ MUBs (an
informationally complete set of MUBs). A complete set of MUBs can always be
found if $d$ is the power of a prime number, while it is still an open
question what’s the maximal number of MUBs in general Ben .

According to WYM , for a set B of MUBs in $\mathcal{H}_{d}$,

$I(\textbf{B}|\rho)\leq Tr(\rho^{2})+\frac{|\textbf{B}|-1}{d}.$ (7)

Introduced as generalizations of MUBs, mutually unbiased measurements (MUMs)
mum  are a set of POVMs
$\textbf{P}=\\{\mathcal{P}^{1},\mathcal{P}^{2},\cdots\\}$ with each
$\mathcal{P}^{m}$ containing $d$ elements
$\mathcal{P}^{m}=\big{\\{}P^{m}_{1},\cdots,P^{m}_{d}\big{\\}}$ and satisfy

$\displaystyle Tr\big{(}P^{m}_{i}\big{)}=1,\hskip 10.00002pt$ $\displaystyle
Tr\big{(}P^{m}_{i}P^{m^{\prime}}_{j}\big{)}=\kappa\delta_{ij}\delta_{mm^{\prime}}+$
$\displaystyle(1-\delta_{ij})\delta_{mm^{\prime}}\frac{1-\kappa}{d-1}+(1-\delta_{mm^{\prime}})\frac{1}{d},$

where $\kappa$ $(\frac{1}{d}<\kappa\leq 1)$ is called the efficiency
parameter. Note that the case $\kappa=1$ corresponds with projective
measurements consisting of mutually unbiased bases.

For any set P of MUMs on $\mathcal{H}_{d}$ there is indexofmum ; mum

$I(\textbf{P}|\rho)\leq\frac{|\textbf{P}|}{d}+\frac{\kappa
d-1}{d(d-1)}\left[dTr(\rho^{2})-1\right],$ (8)

and if P is complete,

$\displaystyle I(\textbf{P}|\rho)=\frac{d+1}{d}+\frac{\kappa
d-1}{d(d-1)}\left[dTr(\rho^{2})-1\right].$ (9)

##  III Uncertainty relations for general SIC-POVMs

In the following discussions we always arrange the probabilities in a
probability distribution in descending order and ignore the probabilities
being zero as they do not contribute to entropy, and we will frequently
consider the two kinds of distributions described below. For any integer
$L\geq 2$ and $\forall c\in[\frac{1}{L},1]$, $\mathscr{P}_{x}^{L}[c]$ and
$\mathscr{P}_{y}^{L}[c]$ are two probability distributions over $L$ outcomes,
the indexes of coincidence of which are both $c$:

$\displaystyle\mathscr{P}_{x}^{L}[c]=\Big{(}\frac{1+\sqrt{(Lc-1)(L-1)}}{L},(L-1)\odot\frac{1-\sqrt{(Lc-1)/(L-1)}}{L}\Big{)},\hskip
60.00009ptI(\mathscr{P}_{x}^{L}[c])=c;$ (10)
$\displaystyle\mathscr{P}_{y}^{L}[c]=\Big{(}(\lceil
1/c\rceil-1)\odot\frac{1+\sqrt{(\lceil\frac{1}{c}\rceil
c-1)/(\lceil\frac{1}{c}\rceil-1)}}{\lceil\frac{1}{c}\rceil},\frac{1-\sqrt{(\lceil\frac{1}{c}\rceil
c-1)(\lceil\frac{1}{c}\rceil-1)}}{\lceil\frac{1}{c}\rceil}\Big{)},\hskip
10.00002ptI(\mathscr{P}_{y}^{L}[c])=c$ (11)

where $\lceil\frac{1}{c}\rceil$ is the smallest integer that $\geq\frac{1}{c}$
and $l\odot p$ is shorthand for $l$ probabilities being $p$. Note here the
number of nonzero probabilities in $\mathscr{P}_{y}^{L}[c]$ is $L,L-1,\cdots,$
respectively when $c\in[\frac{1}{L},\frac{1}{L-1})$,
$[\frac{1}{L-1},\frac{1}{L-2}),\cdots,$ i.e., $\lceil 1/c\rceil$. Two examples
of distributions over four outcomes are presented in Fig. 1

We show in Appendix A the following theorem.

_Theorem 1._ For any discrete probability distribution $\mathscr{P}$ over $L$
outcomes there is
$H_{\alpha}\big{(}\mathscr{P}_{y}^{L}[I(\mathscr{P})]\big{)}\leq
H_{\alpha}\big{(}\mathscr{P}\big{)}\leq
H_{\alpha}\big{(}\mathscr{P}_{x}^{L}[I(\mathscr{P})]\big{)}$ for
$\alpha\in(0,2]$, and
$H_{\alpha}\big{(}\mathscr{P}_{x}^{L}[I(\mathscr{P})]\big{)}\leq
H_{\alpha}\big{(}\mathscr{P}\big{)}\leq
H_{\alpha}\big{(}\mathscr{P}_{y}^{L}[I(\mathscr{P})]\big{)}$ for
$\alpha\in[2,+\infty)$, where $H_{\alpha}\big{(}\mathscr{P}\big{)}$ is the
Rényi-$\alpha$ entropy of $\mathscr{P}$ and $I(\mathscr{P})$ is the index of
coincidence of $\mathscr{P}$.

Thus $H_{\alpha}\big{(}\mathscr{P}_{x}^{L}[c]\big{)}$ and
$H_{\alpha}\big{(}\mathscr{P}_{y}^{L}[c]\big{)}$ are boundary curves of the
diagram of $I(\mathscr{P})\text{-}H_{\alpha}(\mathscr{P})$. The case $L=4$ is
shown in Fig. 2 as an example.

The gray (thick) solid line is the graph of
$H_{2}(\mathscr{P})=-\log_{2}[I(\mathscr{P})]$. The upper bound (UB) on
Shannon entropy (blue dashed line) and the lower bound (LB) on Rényi 5-entropy
(orange dashed dotted line) are respectively given by
$H(\mathscr{P}_{x}^{4}[c])$ and $H_{5}(\mathscr{P}_{x}^{4}[c])$. At the same
time, the lower bound on Shannon entropy (blue solid line) and the upper bound
on Rényi 5-entropy (red dotted line) are respectively given by
$H(\mathscr{P}_{y}^{4}[c])$ and $H_{5}(\mathscr{P}_{y}^{4}[c])$.

We should emphasize that Theorem 1 is a generalization of the Shannon entropic
bounds obtained earlier in Refs. J3 ; HT  to Rényi entropy. With Theorem 1 we
immediately have the Rényi $\alpha$-entropy $H_{\alpha}(\textbf{S}_{g}|\rho)$
for performing any general SIC-POVMs with parameter $a$ on $\mathcal{H}_{d}$
would satisfy

Figure 1: Illustrations of distributions over four outcomes

$(2-\alpha)H_{\alpha}(\textbf{S}_{g}|\rho)\leq(2-\alpha)H_{\alpha}(\mathscr{P}_{x}^{d^{2}}\left[I(\textbf{S}_{g}|\rho)\right]),$
(12)
$(2-\alpha)H_{\alpha}(\textbf{S}_{g}|\rho)\geq(2-\alpha)H_{\alpha}(\mathscr{P}_{y}^{d^{2}}\left[I(\textbf{S}_{g}|\rho)\right]),$
(13)

where $IC(\textbf{S}_{g}|\rho)$ is given by (6). This is the best result that
can be obtained based on (6) only, hence uncertainty relations constructed
from (6) such as those proposed in Ref. Raste ; indexofgsic  cannot be
stronger than our results. In the case $\alpha\to\infty$, Eq. (12) reduces to
the result proposed previously by Rastegin indexofgsic ,

$\displaystyle H_{\infty}(\textbf{S}_{g}|\rho)$ $\displaystyle\geq$
$\displaystyle
2\log_{2}d-\log_{2}\Big{[}1+\sqrt{ad^{3}-1}\sqrt{Tr(\rho^{2})d-1}\Big{]}.$
(14)

Now we show (12) and (13) are tight respectively when
$Tr(\rho^{2})\in[\frac{1}{d},d^{2}a]$ and
$Tr(\rho^{2})\in[\frac{1}{d},\frac{d-2+ad^{2}}{(d-1)^{2}}]$. We only need to
show the probability distributions
$\mathscr{P}_{y/x}^{d^{2}}\left[I(\textbf{S}_{g}|\rho)\right]$ can be achieved
by some positive semi definite matrix in the form $\rho=\sum_{i}x_{i}S_{i}$,
where $x_{i}=\frac{d(da-1)+d(d^{2}-1)p_{i}}{d^{3}a-1}$ is the solution to
$Tr(\rho)=1$ and $Tr(\rho
S_{i})=p_{i}\in\mathscr{P}_{y/x}^{d^{2}}\left[I(\textbf{S}_{g}|\rho)\right]$.
For (12), when $Tr(\rho^{2})\leq d^{2}a$ we have $x_{1}\geq
x_{2}=\cdots=x_{d^{2}}\geq 0$, obviously $\rho\geq 0$. As for (13), we have
$x_{1}=\cdots=x_{d^{2}-1}\geq x_{d^{2}}$, as $\sum_{i}S_{i}={\bf 1}_{d}$, then
$\forall|\phi\rangle\in\mathcal{H}_{d}$,
$\langle\phi|\rho|\phi\rangle=\langle\phi|\sum_{i}x_{i}S_{i}|\phi\rangle=x_{1}+(x_{d^{2}}-x_{1})Tr(S_{d^{2}}|\phi\rangle\langle\phi|)\geq
x_{1}+(x_{d^{2}}-x_{1})/d\geq 0$, thus $\rho$ is a density matrix.

By random sampling over density matrices on $\mathcal{H}_{3}$,

Figure 2: Information diagrams of Shannon entropy and Rényi 5-entropy (see
also Refs. J3 ; HT ). Figure 3: Information diagram of Shannon entropy for
SIC-POVM on $\mathcal{H}_{3}$ (cyan region).

we obtain the information diagram shown in Fig. 3. It is not a surprise to see
that our entropic lower bound for SIC-POVM is not tight when
$Tr(\rho^{2})>\frac{1}{2}$ since (12) and (13) are based on Eq. (6) only.
Interestingly, the corresponding tight bound agrees with

$H(\textbf{S}_{g}|\rho)\geq
H\big{(}\mathscr{P}_{y}^{4}[2I(\textbf{S}_{g}|\rho)]\big{)}+1.$ (15)

##  IV Uncertainty relations for MUMs

###  IV.1 Rényi entropy with $\alpha\leq 1$

We show Theorem 2 in Appendix C.

_Theorem 2._ The sum of Shannon entropies for a finite set P of $|{\bf P}|$
MUMs with efficiency parameter $\kappa$ and performed on an arbitrary
$d$-dimensional system $\rho$ is bounded from below by

$H(\mathscr{P}_{y}^{d}[c])+k\log_{2}n+(|\textbf{P}|-k-1)\log_{2}(n+1),$ (16)

with $C(\textbf{P}|\rho)=\frac{|\textbf{P}|}{d}+\frac{\kappa
d-1}{d(d-1)}\left[dTr(\rho^{2})-1\right]$, here
$n=\lfloor\frac{|\textbf{P}|}{C(\textbf{P}|\rho)}\rfloor$,
$k=\lfloor(C(\textbf{P}|\rho)-\frac{M}{n+1})(n+1)n\rfloor$, and
$c=C(\textbf{P}|\rho)-\frac{k}{n}-\frac{|\textbf{P}|-k-1}{n+1}$.

Despite the complex expression, this theorem can be understood in a simple way
as is discussed in Appendix C. When
$Tr(\rho^{2})\in[\frac{1}{d},\frac{d+\kappa-2}{(d-1)^{2}}]$, (16) reduces to

$(|\textbf{P}|-1)\log_{2}d+H\big{(}\mathscr{P}_{y}^{d}\big{[}C(\textbf{P}|\rho)-(|\textbf{P}|-1)/d\big{]}\big{)},$
(17)

which is actually valid for arbitrary Rényi $\alpha$-entropy with
$0<\alpha\leq 1$, and quite similar to (12) it is tight.

We can linearize the first term of Eq.(16) based on its concavity with respect
to $c$ as follows:
$H(\mathscr{P}_{y}^{d}[c])\geq H(\mathscr{P}_{y}^{d}[\frac{1}{n+1}])$
$+n(n+1)(c-\frac{1}{n+1})\left[H(\mathscr{P}_{y}^{d}[\frac{1}{n}])-H(\mathscr{P}_{y}^{d}[\frac{1}{n+1}])\right]$,
which would then reduce to the result of Wu _et al._ WYM  for MUBs,

$\displaystyle H({\bf B}|\rho)\geq$
$\displaystyle\left[|\textbf{B}|-nC(\textbf{B}|\rho)\right](n+1)\log_{2}(n+1)$
$\displaystyle-$
$\displaystyle[|\textbf{B}|-(n+1)C(\textbf{B}|\rho)]n\log_{2}n,$ (18)

Figure 4: Information diagram of Shannon entropy for complete MUBs in
$\mathcal{H}_{3}$.

where $C(\textbf{B}|\rho)$ is given by the right hand side of (7) and
$n=\lfloor\frac{|\textbf{B}|}{C(\textbf{B}|\rho)}\rfloor$. (16) is generally
improved from (18) and they are equivalent only when $c=\frac{1}{n}$ or
$\frac{1}{n+1}$.

As can be seen in Fig. 4, similar to (13), (16) is not tight when
$Tr(\rho^{2})>\frac{1}{2}$ for $d=3$ and the tight lower bound seems to be

$H(\textbf{B}|\rho)\geq
1+3H\big{(}\mathscr{P}_{y}^{3}[\frac{1+Tr(\rho^{2})}{3}]\big{)}.$ (19)

As for the upper bound, with $L=d$ we only propose the following two unproved
approximations

$\displaystyle(d+1)H\big{(}\mathscr{P}_{x}^{d}[I(\textbf{B}|\rho)/(d+1)]\big{)},\hskip
3.99994pt\text{$Tr(\rho^{2})\approx 1$}$ (20) $\displaystyle
d\log_{2}d+H(\mathscr{P}_{x}^{d}\left[I(\textbf{B}|\rho)-1\right]),\hskip
6.00006ptTr(\rho^{2})\approx\frac{1}{d}$ (21)

###  IV.2 Rényi entropy with $\alpha\geq 2$

_Theorem 3._ Let P be a set of mutually unbiased measurements performed on a
$d$-dimensional system $\rho$, then for any $\alpha\geq 2$

$\displaystyle\frac{1}{|\textbf{P}|}H_{\alpha}({\bf
P}|\rho)\geq\frac{\alpha}{1-\alpha}\log_{2}p_{a}+$
$\displaystyle\frac{\log_{2}d}{(1-\alpha)\ln[1+(d-1)^{\frac{2}{\alpha}}]}\ln\left[1+(d-1)^{\frac{2}{\alpha}}\frac{p_{b}^{2}}{p_{a}^{2}}\right],$
(22)

where $p_{a}=\frac{1+\sqrt{(d-1)(dc-1)}}{d}$,
$p_{b}=\frac{1-\sqrt{(dc-1)/(d-1)}}{d}$, and with $C(\textbf{P}|\rho)$ being
the right hand side of Eq. (8), $c=\frac{1}{|\textbf{P}|}C(\textbf{P}|\rho)$.

This inequality is a direct result of the fact that the right hand side of
(22) is convex with respect to $c$. When $\alpha>2$, Eq. (22) is improved from
Rastegin’s lower bounds $L_{Ras1}$ Raste  and $L_{Ras2}$ design2

$\left\\{\begin{aligned} &L_{Ras1}=\frac{\alpha}{2(1-\alpha)}\log_{2}c\\\
&L_{Ras2}=\frac{\alpha-2}{1-\alpha}\log_{2}\Big{(}\frac{1+\sqrt{(dc-1)(d-1)}}{d}\Big{)}\\\
+&\frac{1}{1-\alpha}\log_{2}c,\end{aligned}\right.$ (23)

Figure 5: Lower bound on Rényi-3 entropy of a single probability distribution.

and when $\alpha=2$ they all reduce to $-\log_{2}c$. A comparison between
these results when $\alpha=3$ and $L=8$ is shown in Fig. 5

###  IV.3 Entropy region

The entropies of performing a finite ordered set of generalized measurements
$\textbf{P}=\\{\mathcal{P}^{m}\\}$ on a $d$-dimensional system described by
$\rho$ form an vector, the $m{th}$ element of which is
$H(\mathcal{P}^{m}|\rho)$. The region of all possible entropic vectors induced
by P is called the entropy region of P. The entropy region of a given
measurement set contains much more information besides the entropic lower
bound, and we expect it to be as meaningful in quantum information theory as
in the classical counterpart.

(a) Shannon entropy region for three MUBs.

(b) Shannon entropy region of three probability distributions satisfying Eq.
(24)

Figure 6: Shannon entropy regions for three MUBs

We make a comparison here between the Shannon entropy region for three MUBs in
$\mathcal{H}_{d}$ and that of three probability distributions over $d$
outcomes satisfying

$\frac{3}{d}\leq\sum_{m=1}^{3}I(\mathscr{P}^{m})\leq\max\limits_{\rho}\\{I(\textbf{B}|\rho)\\}=1+\frac{2}{d}$
(24)

Figure 7: Information diagram of Shannon entropy for complete MUBs in
$\mathcal{H}_{4}$.

As can be seen in Fig. 6, the entropy region of probability distributions
satisfying Eq. (24) is the same to that for three MUBs when $d=2$, while in
higher dimensions distinctions show up at places where the sum of entropies is
relatively small, which is in accordance with the information diagrams.

##  V Discussions

We can see from Figs. (3,4, and 7) that the tight lower Rényi entropic
($\alpha<2$) bound curves for both complete MUBs and SIC-POVMs are non-
differentiable at $Tr(\rho^{2})=\frac{1}{k}$ $(\forall k=2,\cdots,d-1)$, which
divide the curves into $d-1$ sections. A natural thought is that different
sections corresponds with density matrices at different boundaries of the set
of positive semi-definite Hermitian matrices, namely, different sections of
the lower bound curve are attained by density matrices of different ranks.

Conjecture. The tight lower bound on Shannon entropy for complete MUBs or SIC-
POVMs on $\mathcal{H}_{d}$ can only be achieved by density matrices satisfying
$(\lambda_{1},\lambda_{2},\cdots)=\mathscr{P}_{y}^{d}[Tr(\rho^{2})]$, where
$\\{\lambda_{i}\\}$ are nonzero eigenvalues of $\rho$ and arranged in
descending order.

We believe this conjecture, if confirmed, will be helpful in searching for
tight state-independent EURs for complete MUBs and SIC-POVMs, which could be
more efficient in applications of quantum information theory.

Based on the conjecture above, we have an alternative form of Eq. (16) for
MUBs when $Tr(\rho^{2})\leq\frac{1}{d-1}$

$\displaystyle H({\bf
B}|\rho)\geq(|\textbf{B}|-1)\log_{2}d-Tr[\rho\log_{2}\rho],$ (25)

which coincides with the uncertainty relation for two observables proposed by
Berta _et al._ berta .

Lastly, we show an application of entropic uncertainty relations in
entanglement detection. Let
$\rho_{AB}=\sum_{i}p_{i}\rho_{i}^{A}\otimes\rho_{i}^{B}$ be an arbitrary
separable state on the bipartite Hilbert space
$\mathcal{H}_{A}\otimes\mathcal{H}_{B}$, and $\rho_{A}=Tr_{B}(\rho_{AB})$ and
$\rho_{B}=Tr_{A}(\rho_{AB})$ are the reduced density matrices. According to
the results shown in Ref. entangle2 , for any nondegenerate observables
$\\{A_{1},A_{2},\cdots\\}$ on $\mathcal{H}_{A}$ and $\\{B_{1},B_{2},\cdots\\}$
on $\mathcal{H}_{B}$, there is

$\displaystyle\sum_{m}H_{\alpha}$
$\displaystyle\big{(}\mathscr{P}(A_{m}\otimes B_{m}|\rho_{AB})\big{)}\geq
D=\max\\{D_{A},D_{B}\\},$ where $\displaystyle
D_{A}=\sum_{m}H_{\alpha}\big{(}\mathscr{P}(A_{m}|\rho_{A})\Big{)},$ (26)
$\displaystyle
D_{B}=\sum_{m}H_{\alpha}\big{(}\mathscr{P}(B_{m}|\rho_{B})\big{)}.\hskip
5.0pt(\alpha>0)$

Here, $\mathscr{P}(A_{m}\otimes B_{m}|\rho_{AB})$ is the probability
distribution induced by measuring $A_{m}\otimes B_{m}$ on $\rho_{AB}$, and
$\mathscr{P}(A_{m}|\rho_{A})$ and $\mathscr{P}(B_{m}|\rho_{B})$ are the two
corresponding marginal probability distributions.

Any bipartite state violating Eq. (26) must be entangled. When $\\{A_{m}\\}$
and $\\{B_{m}\\}$ are complementary observables, $D$ is the sum of entropies
for measuring $\rho_{A}$ or $\rho_{B}$ in MUBs. According to our uncertainty
relations in the previous section, a further lower bound for $D$ can be
obtained directly from (22) when $\alpha\geq 2$ and from (16) when $\alpha=1$.

As an example, consider a pair of qudits in a Werner state
$\mathcal{W}_{AB}=\frac{1-p}{d^{2}}{\bf 1}_{d^{2}}+p|\psi\rangle\langle\psi|$
$(0\leq p\leq 1)$, where
$|\psi\rangle=\frac{1}{\sqrt{d}}(|0\rangle\otimes|0\rangle+\cdots+|d-1\rangle\otimes|d-1\rangle)$,
and $\\{\sigma_{m}\\}$ ($m=1,2,\cdots,M)$ is a set of complementary
observables in $\mathcal{H}_{d}$. The density matrix of a single qudit is then
$\mathcal{W}_{A}=\mathcal{W}_{B}=\frac{1}{d}{\bf 1}_{d}$, which is independent
of $p$. For simplicity, suppose now $d$ is a prime power and $M=d+1$, in which
case $D=(d+1)\log_{2}d$ is state-independent, and moreover, the strongest form
of (26) becomes

$\sum_{m}^{d+1}H_{\infty}\big{(}\mathscr{P}(\sigma_{m}\otimes\sigma_{m}|\mathcal{W}_{AB})\big{)}\geq(d+1)\log_{2}d$
(27)

since Rényi $\alpha$-entropy is a non-increasing function of $\alpha$.
Numerical results show that (27) can be violated for $p>0.33$ when $d=2$ and
for $p>0.46$ when $d=3$. As the bipartite Werner state is entangled if and
only if $p>\frac{1}{d+1}$ sep2 ; sep3 ; sep4 ; sep5 , Eq. (27) is strong
enough when $d=2$ but it is not strong when $d=3$ and fails to detect all
entangled states.

From the above example we know that our EURs can be used to detect
entanglement, and more stronger separability criteria based on our uncertainty
relations are also possible. More works on entropic separability criteria can
be found in Refs. [11, 15, 23].

## CONCLUSION

In this paper we have obtained improved entropic uncertainty relations for
general symmetric informationally complete positive operator-valued measures
and mutually unbiased measurements in terms of Rényi entropy, which are shown
to be tight for sufficiently mixed states. It might be the first time that
tight state-dependent entropic uncertainty relations for multiple generalized
measurements have been obtained. By random sampling density matrices and
calculating the corresponding entropy for a given set of measurements,
comparisons between our entropic bounds and the numerical optimal bounds are
made via information diagrams. Our investigation of entropic uncertainty
relations could provide some insights for further applications of uncertainty
relations in information theory.

###### Acknowledgements.

This work is supported by the National Key R&D Program of China (Grants No.
2017YFA0303703 and No. 2016YFA0301801) and the National Natural Science
Foundation of China (Grant No. 11475084).

##  Appendix A Proof of Theorem 1

We employ Lagrangian multiplier method to find the distributions which make
Rényi $\alpha$-entropy attain local extreme values on the set: ${\bf
S}(L,c)=\left\\{\mathscr{P}|length(\mathscr{P})=L\geq 3,\hskip
5.0ptI(\mathscr{P})=c<1\right\\}$.

$\displaystyle L_{\alpha}$
$\displaystyle=\sum_{i=1}^{L}{p_{i}}^{\alpha}+\lambda(\sum_{i=1}^{L}p_{i}^{2}-c)+\lambda^{\prime}(\sum_{i=1}^{L}p_{i}-1);$
$\displaystyle\frac{\partial{L_{\alpha}}}{\partial{p_{i}}}$
$\displaystyle=\alpha p_{i}^{\alpha-1}+2\lambda
p_{i}+\lambda^{\prime}=0,\hskip 5.0pt1\leq i\leq L.\hskip
10.00002pt(\alpha>0,\alpha\neq 1)$ (28)

where $\lambda$ and $\lambda^{\prime}$ are multipliers. Note the equation
$\alpha x^{\alpha-1}=-2\lambda x-\lambda^{\prime}x$ has at most two different
solutions as $x^{\alpha-1}$ is either concave or convex with respect to $x$
and $-2\lambda x-\lambda^{\prime}x$ describes a line. But (28) is not valid if
there exists $1\leq i\leq L$ such that $p_{i}=0$.

When $\alpha<1$ and $p_{i}\to 0$, in which case $p_{i}^{\alpha-1}\to+\infty$.
This implies on ${\bf S}(L,c)$, Rényi $\alpha$-entropy can only attain local
minimum value at probability distributions whose positive probabilities
satisfy (28), and it can never attain local maximum value at a distribution
the smallest probability in which is 0. When $\alpha>1$,
$\frac{\partial}{\partial p_{i}}(\sum_{i}p_{i}^{2}-c)\big{|}_{p_{i}=0}=0$,
again, (28) is a restriction on the positive probabilities only.

We only need to consider those distributions containing at most two different
positive probabilities, $p_{a}$ and $p_{b}$ say, and let’s parameterize them
with three parameters as follows: N, the number of positive probabilities;
$N_{a}$, the number of probabilities being $p_{a}$; $c$, the index of
coincidence. We arrange the positive probabilities in descending order and
represent the distribution formally as

$\mathscr{P}[c,N,N_{a}]=\Big{(}N_{a}\odot p_{a},(N-N_{a})\odot
p_{b}\Big{)},\hskip 10.00002pt(1/N\leq c\leq 1/N_{a})$ (29)

where $N_{a}\odot p_{a}$ is shorthand for $N_{a}$ probabilities being $p_{a}$.
Combined with the condition that $N_{a}p_{a}+(N-N_{a})p_{b}=1$ and
$I\big{(}\mathscr{P}[c,N,N_{a}]\big{)}=c$, we have $p_{a}=$
$\frac{1+\sqrt{(Nc-1)(N-N_{a})/N_{a}}}{N}$ and
$p_{b}=\frac{1-\sqrt{(Nc-1)N_{a}/(N-N_{a})}}{N}$. It can be checked that
$\mathscr{P}[c_{1},N,N_{a}]$ majorizes $\mathscr{P}[c_{2},N,N_{a}]$ if
$c_{1}>c_{2}$, thus $H_{\alpha}(\mathscr{P}[c,N,N_{a}])$ is a decreasing
function of $c$.

Given the values of $u$, $c$ $(1/N\leq c<\frac{1}{N_{a}})$, and $N$ ($N_{a}$),
the values of $N_{a}$ $(N)$, $p_{a}$, and $p_{b}$, if exist, are uniquely
determined by (30).

$\displaystyle\left\\{\begin{aligned} 0\leq p_{b}<p_{a},N\geq N_{a}\geq
1,0<\alpha\neq 2\hskip 5.0pt\text{and}\hskip 5.0pt\alpha\neq 1,\hskip
156.49014pt\\\ N_{a}p_{a}+(N-N_{a})p_{b}=1;\hskip
10.00002ptN_{a}p_{a}^{2}+(N-N_{a})p_{b}^{2}=c;\hskip
10.00002ptN_{a}p_{a}^{\alpha}+(N-N_{a})p_{b}^{\alpha}=u>0.\\\
\end{aligned}\right.$ (30) $\displaystyle
H_{\alpha}\big{(}\mathscr{P}[1/N+s,N,N_{a}]\big{)}=\log_{2}N-\frac{\alpha
Ns}{2\ln 2}+\frac{\alpha(\alpha-2)N^{\frac{3}{2}}N_{a}s^{\frac{3}{2}}}{4\ln
2(N-N_{a})}+o(s^{2}),\hskip 10.00002pt(0<s\ll\frac{1}{N_{a}}-\frac{1}{N}).$
(31)

Note that $H_{\alpha}\big{(}\mathscr{P}[\frac{1}{N},N,N_{a}]\big{)}=\log_{2}N$
is independent of $N_{a}$ and $\lim_{c\to
1/N_{a}}H_{\alpha}\big{(}\mathscr{P}[c,N,N_{a}]\big{)}=\log_{2}N_{a}$ is
independent of $N$, thus $H_{\alpha}\big{(}\mathscr{P}[c,N,N_{a}]\big{)}$ is
monotonic of $N$ if $c\neq\frac{1}{N}$ and monotonic of $N_{a}$ if
$c\neq\frac{1}{N_{a}}$, more concretely, taking the series expansion of
entropy (31) into consideration we have

$\displaystyle(2-\alpha)H_{\alpha}\big{(}\mathscr{P}[c,N,N_{a}]\big{)}\geq(2-\alpha)H_{\alpha}\big{(}\mathscr{P}[c,N,N_{a}^{\prime}]\big{)},\hskip
5.0pt(N_{a}<N_{a}^{\prime},\frac{1}{N}\leq c\leq\frac{1}{N_{a}^{\prime}});$
(32)
$\displaystyle(2-\alpha)H_{\alpha}\big{(}\mathscr{P}[c,N,N_{a}]\big{)}\leq(2-\alpha)H_{\alpha}\big{(}\mathscr{P}[c,N^{\prime},N_{a}]\big{)},\hskip
5.0pt(N<N^{\prime},\frac{1}{N}\leq c\leq\frac{1}{N_{a}}).$ (33)

We can conclude from (32) and (33) that for any distribution $\mathscr{P}$
over $L$ outcomes with $I(\mathscr{P})=c$ there is

$(2-\alpha)H_{\alpha}\big{(}\mathscr{P}[c,L,1]\big{)}\geq(2-\alpha)H_{\alpha}\big{(}\mathscr{P}\big{)}\geq(2-\alpha)H_{\alpha}\big{(}\mathscr{P}[c,N,N-1]\big{)},$
(34)

where $N$ is an integer such that $\frac{1}{L}\leq\frac{1}{N}\leq
c<\frac{1}{N-1}$, namely, $N=\lceil\frac{1}{c}\rceil$. This completes the
proof of Theorem 1.

##  Appendix B Properties of extreme values

Let’s reparametrize $\mathscr{P}[c,N,N_{a}]$ as $\mathscr{P}^{*}[c,N,\theta]$,
where $\theta=2\arccos\sqrt{N_{a}/N}$ and $\theta\in[0,\pi)$. We have

$\displaystyle H_{\alpha}\big{(}\mathscr{P}^{*}[c,N,\theta]\big{)}$
$\displaystyle=\frac{1}{1-\alpha}\log_{2}\Big{[}N\cos^{2}\frac{\theta}{2}\Big{(}\frac{1+\sqrt{Nc-1}\tan\frac{\theta}{2}}{N}\Big{)}^{\alpha}+N\sin^{2}\frac{\theta}{2}\Big{(}\frac{1-\sqrt{Nc-1}\cot\frac{\theta}{2}}{N}\Big{)}^{\alpha}\Big{]}$
$\displaystyle=\frac{1}{1-\alpha}\log_{2}M_{\alpha}(\mathscr{P}^{*}[c,N,\theta])$
$\displaystyle(\alpha-1)\frac{\partial^{2}}{\partial
c^{2}}H_{\alpha}\big{(}\mathscr{P}^{*}[c,N,\theta]\big{)}$
$\displaystyle=f(\alpha,z,\theta)\frac{\alpha
N^{\alpha+1}\sin\frac{\theta}{2}\cos^{3}\frac{\theta}{2}}{4\ln
2(Nc-1)^{3/2}M_{\alpha}^{2}\big{(}\mathscr{P}^{*}[c,N,\theta]\big{)}}\left(1+\sqrt{Nc-1}\tan\frac{\theta}{2}\right)^{2\alpha-2}$

where, with
$z=\frac{1-\sqrt{Nc-1}\cot\frac{\theta}{2}}{1+\sqrt{Nc-1}\tan\frac{\theta}{2}}$
$(0<z\leq\tan\frac{\theta}{2})$, $f(\alpha,z,\theta)$

$\displaystyle=$
$\displaystyle\frac{2\tan^{2}\frac{\theta}{2}}{1+z\tan^{2}\frac{\theta}{2}}(z^{\alpha-1}-1)^{2}+z^{\alpha-1}\left[-\tan^{2}\frac{\theta}{2}z^{\alpha-1}+z^{1-\alpha}+(\alpha-1)(z\tan^{2}\frac{\theta}{2}-\frac{1}{z})+(2-\alpha)(\tan^{2}\frac{\theta}{2}-1)\right]$
$\displaystyle\geq$ $\displaystyle
z^{\alpha-1}\left[-\tan^{2}\frac{\theta}{2}z^{\alpha-1}+z^{1-\alpha}+(\alpha-1)(z\tan^{2}\frac{\theta}{2}-\frac{1}{z})+(2-\alpha)(\tan^{2}\frac{\theta}{2}-1)\right]$

when $\alpha\in(0,1)\cup[2,+\infty)$ and $0<\tan\frac{\theta}{2}\leq 1$,

$f(\alpha,z,\theta)\geq
z^{\alpha-1}\left[-z^{\alpha-1}+z^{1-\alpha}+(\alpha-1)(z-\frac{1}{z})\right]\geq
0\Longrightarrow(\alpha-1)\frac{\partial^{2}}{\partial
c^{2}}H_{\alpha}\big{(}\mathscr{P}^{*}[c,N,\theta]\big{)}\geq 0$ (35)

As for Shannon entropy, when $0<z<\tan\frac{\theta}{2}\leq 1$

$\displaystyle\frac{\partial^{2}}{\partial
c^{2}}H\big{(}\mathscr{P}^{*}[c,N,\theta]\big{)}=\log_{2}(1+z\tan{\frac{\theta}{2}})-\log_{2}(1-z\cot{\frac{\theta}{2}})-\frac{1}{1-z\cot{\frac{\theta}{2}}}+\frac{1}{1+z\tan{\frac{\theta}{2}}}\leq
0.$ (36)

(35) and (36) imply that when $N_{a}\in[N/2,N]$,
$H_{\alpha}\big{(}\mathscr{P}[c,N,N_{a}]\big{)}$ is concave with respect to
$c$ when $\alpha\leq 1$ and convex with respect to c when $\alpha\geq 2$.

##  Appendix C Proof of Theorem 2

Let g=$\\{\mathscr{P}^{g}\\}$ denote the probability distributions at which
$\sum_{m=1}^{M}H(\mathscr{P}^{m})$ is minimum under the restriction

$\forall 1\leq m\leq M,length(\mathscr{P}^{m})=L;\hskip
5.0pt\sum_{m=1}^{M}I(\mathscr{P}^{m})=\sum_{m}c^{m}=c\hskip 5.0pt\text{($c$ is
a constant, $c\in[\frac{M}{d},M]$)},$ (37)

where $\mathscr{P}^{g}$ is the $gth$ distribution in g. Firstly, according to
(34) (or Theorem 1) and (36) we have the following
Property 1. $\mathscr{P}^{g}$ must be in the form
$\mathscr{P}^{g}=\mathscr{P}_{y}^{L}[c^{g}]$ for any $g$.
Property 2. At most one element in g, $\mathscr{P}^{k}$ say, is not uniform in
its nonzero part.
It can be proved that for any $2\leq m<n$ $(n,m\in N^{+})$,

$\displaystyle\left\\{\begin{aligned} 1.\hskip
5.0ptH\big{(}\mathscr{P}_{y}^{L}[1/n]\big{)}+H\big{(}\mathscr{P}_{y}^{L}[1/m+s]\big{)}>H\big{(}\mathscr{P}_{y}^{L}[1/m]\big{)}+H\big{(}\mathscr{P}_{y}^{L}[1/n+s]\big{)},\hskip
5.0pt0\leq s\leq 1/n/(n-1)\\\ 2.\hskip
5.0ptH\big{(}\mathscr{P}_{y}^{L}[1/n-s]\big{)}+H\big{(}\mathscr{P}_{y}^{L}[1/m]\big{)}\geq
H\big{(}\mathscr{P}_{y}^{L}[1/m-s]\big{)}+H\big{(}\mathscr{P}_{y}^{L}[1/n]\big{)},\hskip
5.0pt0\leq s\leq 1/n/(n+1)\end{aligned}\right.$ (38) $\displaystyle\text{note
here:}\hskip
5.0ptIC(\mathscr{P}_{y}^{L}[1/n])+IC(\mathscr{P}_{y}^{L}[1/m+s])=IC(\mathscr{P}_{y}^{L}[1/n+s])+IC(\mathscr{P}_{y}^{L}[1/m])$
$\displaystyle\hskip
50.00008ptIC(\mathscr{P}_{y}^{L}[1/n-s])+IC(\mathscr{P}_{y}^{L}[1/m])=IC(\mathscr{P}_{y}^{L}[1/m-s])+IC(\mathscr{P}_{y}^{L}[1/n]),$

with $N_{g}$ denoting the number of nonzero probabilities of
$\mathscr{P}^{g}$, a direct result of properties 1-2 and (38) is the following
$\begin{aligned} \text{{\bf Property 3}}.\hskip 10.00002pt1.\hskip
5.0pt&\max\limits_{g,g^{\prime}}|\\{N_{g}-N_{g^{\prime}}\\}|\leq 1;\hskip
20.00003pt2.\hskip 5.0pt\text{if}\hskip
5.0ptN_{k}=\min\limits_{g}\\{N_{g}\\},\hskip 5.0pt\text{then}\hskip
5.0pt\forall g,N_{g}-N_{k}=0\end{aligned}$
With Properties 1-3, it’s enough to determine g (Theorem 2). To show the first
inequality of (38) we only need to show
$\log_{2}N-H\big{(}\mathscr{P}_{y}[1/N+s,N,N-1]\big{)}$ is an increasing
function of N. Under the parametrization introduced in Appendix B we have,

$\displaystyle\log_{2}N-H\big{(}\mathscr{P}^{*}[1/N+s,N,\theta]\big{)}=\cos^{2}\frac{\theta}{2}\Big{(}1+\sqrt{Ns}\tan\frac{\theta}{2}\Big{)}\log_{2}\Big{[}\cos^{2}\frac{\theta}{2}\Big{(}1+\sqrt{Ns}\tan\frac{\theta}{2}\Big{)}\Big{]}$
(39)
$\displaystyle\sin^{2}\frac{\theta}{2}\Big{(}1-\sqrt{Ns}\cot\frac{\theta}{2}\Big{)}\log_{2}\Big{[}\sin^{2}\frac{\theta}{2}\Big{(}1-\sqrt{Ns}\cot\frac{\theta}{2}\Big{)}\Big{]}=h\big{(}s,N,\theta\big{)}$

Let $\theta_{y}(N)=2\arctan\frac{1}{\sqrt{N-1}}$, then
$\mathscr{P}[c,N,N-1]=\mathscr{P}^{*}[c,N,\theta_{y}(N)]$,

$\displaystyle\frac{\partial}{\partial
N}h\big{(}s,N,\theta\big{)}=\frac{s}{N}\frac{\partial}{\partial
s}h\big{(}s,N,\theta\big{)}>0,\hskip
5.0pt\frac{\partial}{\partial\theta}h\big{(}s,N,\theta\big{)}<0,\hskip
5.0pt\frac{d\theta_{y}}{dN}<0$ $\displaystyle\frac{\partial}{\partial
N}h\big{(}s,N,\theta_{y}(N)\big{)}=\Big{[}\frac{\partial}{\partial
N}h\big{(}s,N,\theta\big{)}+\frac{\partial}{\partial\theta}h\big{(}s,N,\theta\big{)}\frac{d\theta_{y}}{dN}\Big{]}\Big{|}_{\theta=\theta_{y}(N)}\geq
0$ (40)

Hence $h(s,N,\theta_{y}(N))$ is an increasing function of N, and the second
inequality of (38) can be proved similarly.

It turns out that g is also the set of probability distributions that descends
entropy the fastest locally. Consider $c=\frac{M}{L}$ (this is when
probability distributions are all uniform) in the beginning and then let $c$
increase, then according to Properties 1, 2 and (31) obviously the steepest
descent of Shannon entropy is given by

$\left.\begin{aligned}
&\Big{\\{}(M-1)\odot\mathscr{P}_{y}^{L}\big{[}\frac{1}{L}\big{]},\mathscr{P}_{y}^{L}\big{[}c-\frac{M-1}{L}\big{]}\Big{\\}},\hskip
90.00014pt\frac{M}{L}\leq c\leq\frac{M-1}{L}+\frac{1}{L-1}\\\
&\Big{\\{}(M-2)\odot\mathscr{P}_{y}^{L}\big{[}\frac{1}{L}\big{]},\mathscr{P}_{y}^{L}\big{[}\frac{1}{L-1}\big{]},\mathscr{P}_{y}^{L}\big{[}c-\frac{M-2}{L}-\frac{1}{L-1}\big{]}\Big{\\}},\frac{M-1}{L}+\frac{1}{L-1}\leq
c\leq\frac{M-2}{L}+\frac{2}{L-1}\\\ &\hskip 40.00006pt\cdots\cdots\\\
&\Big{\\{}(M-1)\odot\mathscr{P}_{y}^{L}\big{[}\frac{1}{L-1}\big{]},\mathscr{P}_{y}^{L}\big{[}c-\frac{M-1}{L-1}\big{]}\Big{\\}},\hskip
75.0001pt\frac{M}{L-1}\leq c\leq\frac{M-1}{L-1}+\frac{1}{L-2}\\\ &\hskip
40.00006pt\cdots\cdots\end{aligned}\right\\}={\bf g}$

where $M\odot\mathscr{P}$ is shorthand for $M$ probability distributions being
$\mathscr{P}$.

## References

  * (1) W. Heisenberg,  Z. Phys. 43, 172 (1927).
  * (2) H. P. Robertson,  Phys. Rev. 34, 163 (1929).
  * (3) D. Deutsch,  Phys. Rev. Lett. 50, 631 (1983).
  * (4) L. Dammeier, R. Schwonnek, and R. F. Werner,  New J. Phys. 17, 093046 (2015).
  * (5) I. Białynicki-Birula, and Ł. Rudnicki, 2011, in Statistical Complexity, edited by K. Sen (Springer Netherlands, Dordrecht), pp. 1-34
  * (6) P. J. Coles, R. Colbeck, L. Yu, and M. Zwolak,  Phys. Rev. Lett. 108, 210405 (2012).
  * (7) S. Friedland, V. Gheorghiu, and G. Gour,  Phys. Rev. Lett. 111, 230401 (2013).
  * (8) J. B. M. Uffink and J. Hilgevoord,  Found. Phys. 15, 925 (1985).
  * (9) G. Vallone, D. G. Marangon, M. Tomasin, and P. Villoresi,  Phys. Rev. A 90, 052327 (2014).
  * (10) V. Giovannetti,  Phys. Rev. A 70, 012102 (2004).
  * (11) O. Gühne, and M. Lewenstein,  Phys. Rev. A 70, 022316 (2004).
  * (12) Yichen Huang,  Phys. Rev. A 82, 012335 (2010).
  * (13) R. König, S. Wehner, and J. Wullschleger,  IEEE Trans. Inf. Theory 58, 1962 (2012).
  * (14) F. Dupuis, O. Fawzi, and S. Wehner,  IEEE Trans. Inf. Theory 61, 1093 (2015).
  * (15) P. J. Coles, M. Berta, M. Tomamichel, and S. Wehner,  Rev. Mod. Phys. 89, 015002 (2017).
  * (16) K. Kraus,  Phys. Rev. D 35 3070 (1987).
  * (17) H. Maassen and J. B. M. Uffink,  Phys. Rev. Lett. 60, 1103 (1988).
  * (18) I. D. Ivonovic,  J. Phys. A 25, L363 (1992).
  * (19) J. Sánchez,  Phys, Lett. A 173, 233 (1993).
  * (20) J. Sánchez-Ruiz,  Phys, Lett. A 201, 125 (1995).
  * (21) M. A. Ballester and S. Wehner,  Phys. Rev. A 75, 022319 (2007).
  * (22) S. Wehner, and A. Winter,  New J. Phys. 12, 025009 (2010).
  * (23) S. Wu, S. Yu, and K. Mølmer,  Phys. Rev. A 79, 022104 (2009).
  * (24) I. Bengtsson et al.,  J. Math. Phys. 48, 052106 (2007).
  * (25) J. M. Renes, R. Blume-Kohout, A. J. Scott, and C. M. Caves,  J. Math. Phys. 45 2171 (2004).
  * (26) A. Kalev and G. Gour,  New J. Phys. 16, 053038 (2014).
  * (27) G. Gour G and A. Kalev,  J. Phys. A: Math. Theor. 47 335302 (2014).
  * (28) A. E. Rastegin,  Eur. Phys. J. D 67, 269 (2013).
  * (29) A. E. Rastegin,  Phys. Scr. 89, 085101 (2014).
  * (30) B. Chen, and S. Fei,  Quantum Inf. Process. 14, 2227-2238 (2015).
  * (31) K. Wang, N. Wu, and F. Song,  Phys. Rev. A 98, 032329 (2018).
  * (32) A. Ketterer and O. Gühne,  Phys. Rev. Research 2, 023130 (2020).
  * (33) A. E. Rastegin,  J. Phys. A: Math. Theor., Vol. 53, 405301 (2020).
  * (34) A. Rényi, 1961, in Proceedings of the 4th Berkeley Symposium on Mathematical Statistics and Probability, Vol. 1 (University of California Press, Berkeley, CA), pp. 547-561.
  * (35) P. Harremoës and F. Topsøe,  IEEE Trans. Inf. Theory 47, 2944 (2001).
  * (36) A. J. Scott, M. Grassl,  J. Math. Phys. 51, 042203 (2010).
  * (37) D. M. Appleby,  Opt. Spectrosc. 103, 416-428 (2007).
  * (38) A. Kalev,  J. Phys. A: Math. Theor. 47 265301 (2014).
  * (39) I. D. Ivonovic,  J. Phys. A 14, 3241 (1981).
  * (40) W. K. Wootters and B. D. Fields,  Ann. Phys. (N.Y.) 191, 363 (1989).
  * (41) A. Klappenecker and M. Rötteler, Finite Fields and Applications (Springer, Berlin-Heidelberg, 2004), pp. 137-144.
  * (42) A. O. Pittenger and M. H. Rubin,  Linear Algebr. Appl. 390, 255 (2004).
  * (43) M. Berta, M. Christandl, R. Colbeck, J. M. Renes, and R. Renner,  Nat. Phys. 6, 659 (2010).
  * (44) A. O. Pittenger and M. H. Rubin  Phys. Rev. A 62, 032313 (2000).
  * (45) P. Rungta, W. J. Munro, K. Nemoto, P. Deuar, G. J. Milburn, and C. M. Caves, in Directions in Quantum Optics, Vol.561, edited by H. J. Carmichael, R. J. Glauber, and M. O. Scully (Springer-Verlag, Berlin-Heidelberg, 2001) pp. 149-164.
  * (46) A. Peres,  Phys. Rev. Lett. 77, 1413 (1996).
  * (47) A. O. Pittenger, M. H. Rubin,  Opt. Commun. 179, 447-449 (2000).


# Towards Multi-User Activity Recognition through Facilitated Training Data
and Deep Learning for Human-Robot Collaboration Applications

Francesco Semeraro  Manchester Centre for Robotics and AI
The University of Manchester
Manchester, UK
<EMAIL_ADDRESS>Jon Carberry  BAE Systems (Operations)
Ltd.
BAE Systems plc.
Warton, UK
<EMAIL_ADDRESS>Angelo Cangelosi  Manchester Centre for Robotics
and AI
The University of Manchester
Manchester, UK
<EMAIL_ADDRESS>
###### Abstract

Human-robot interaction (HRI) research is progressively addressing multi-party
scenarios, where a robot interacts with more than one human user at the same
time. Conversely, research is still at an early stage for human-robot
collaboration. The use of machine learning techniques to handle such type of
collaboration requires data that are less feasible to produce than in a
typical HRC setup. This work outlines scenarios of concurrent tasks for non-
dyadic HRC applications. Based upon these concepts, this study also proposes
an alternative way of gathering data regarding multi-user activity, by
collecting data related to single users and merging them in post-processing,
to reduce the effort involved in producing recordings of pair settings. To
validate this statement, 3D skeleton poses of activity of single users were
collected and merged in pairs. After this, such datapoints were used to
separately train a long short-term memory (LSTM) network and a variational
autoencoder (VAE) composed of spatio-temporal graph convolutional networks
(STGCN) to recognise the joint activities of the pairs of people. The results
showed that it is possible to make use of data collected in this way for pair
HRC settings and get similar performances compared to using training data
regarding groups of users recorded under the same settings, relieving from the
technical difficulties involved in producing these data.

The related code and collected data are publicly available111The code
repository is available at:
https://github.com/francescosemeraro/Multi_User_through_Single_User.git. The
data repository is available at: https://figshare.com/s/64bd023e968e6eb096f3..

###### Index Terms:

multi-user activity recognition, single-user training data, concurrent tasks,
multi-party human-robot collaboration, non-dyadic human-robot collaboration,
deep learning, long short-term memory, variational autoencoder, spatio-
temporal graph convolutional network, transfer learning

##  I Introduction

Human-robot interaction research is constantly progressing [1]. Therefore,
researchers are increasingly addressing scenarios that go beyond the most
commonly explored dyadic interaction between a human and a robot [2, 3]. The
last 15 years have been characterised by dozens of works in human-robot
interaction regarding non-dyadic interactions. They mainly regard measurement
of social state and engagement in crowded environments [4]. Despite the amount
of works, they are outnumbered by the plethora of works related to HRI.
Furthermore, few works tackle a human-robot collaboration (HRC) application in
a multi-party scenario [4, 5, 6, 7], making this a promising field for
scientific investigation. Therefore, in this work, we delineate elements of a
collaborative scenario in which the robot works jointly with two human users
who perform concurrent tasks in a manufacturing application.

There is a growing interest also in using machine learning (ML) to build an
HRC cognitive system [8], especially deep learning (DL) models. Indeed, in HRI
it is difficult to obtain perfectly replicable experimental settings [9]; ML
can constitute an answer to this issue, as it allows the system to be exposed,
during its training, to different behaviours coming from acquisitions
involving different users. Therefore, this work considers a cognitive system,
based on DL models, meant to be deployed in a collaborative scenario with 2
users.

To do so, it is crucial to be able to collect data regarding the experimental
scenario to train the system properly to act in the collaborative setup.
However, in group interactions, this would require to have pairs of people
present in the setup at the same time, introducing a further layer of
complexity to the feasibility of the already demanding process of recruiting
an adequate pool of participants for these studies [10].

In addressing these new challenging scenarios, this work proposes a way of
managing data collection of these interactions in an efficient way, that leads
to promising performance in terms of learning from data. More precisely, this
study demonstrates that it is possible to build a representative training
dataset regarding a non-dyadic interaction, by collecting recordings of users
individually and later merging their acquisitions, to form a single data
point. Achieving such result would allow non-dyadic interactions to be more
easily considered in robotics research by means of machine learning. Indeed,
this would leverage robotics studies from feasibility issues related to lack
of data [11] and recruitment of participants due to safety restrictions [12].
From the application point of view, the study of non-dyadic HRC has the
potential to unlock more complex manufacturing processes in industrial
applications that go beyond the serial chain production, in which a single
worker at a time can perform operations on the manufacturing target. Moreover,
such result could be used for other application uses in non-dyadic HRI [4],
without being specific to collaborations only.

For this purpose, acquisitions related to single users were grouped in post-
processing to obtain training data relatable to the robot’s perception of a
triadic collaborative scenario while being part of it. This allowed us to
collect more data in a less demanding modality than gathering pairs of people
from the beginning.

To test the performance achievable from such dataset under different
configurations of DL models, a VAE and a neural network of LSTMs were trained
separately to learn the joint activity of the users. We demonstrate that it is
possible to achieve performances in the multi-user classification comparable
to when the models are trained with data coming from users recorded at the
same time and in the same setting.

The contributions of this paper are summarised below:

  * •

design outlines of a non-dyadic HRC scenario with concurrent tasks;

  * •

a multi-user activity recognition pipeline, based on DL models trained with
data coming from activities of single users and previous design
considerations.

From this point on, data generated by pairing acquisitions of single users
will be addressed to as grouped, while the ones regarding pairs of them in the
same scene at the same time will be referred to as pair.

This paper is structured as follows. Section II contextualises this work in
the two main fields it contributes to. Section III depicts the design concepts
of concurrent tasks, inside and outside HRC. Section IV provides a breakdown
of the methodology adopted to acquire and learn from the acquired data.
Section V reports the results of the trainings, then discussed in Section VI.
Finally, Section VII sums up the main achievements of this work.

##  II Related work

Because of the dual aim of this paper, recent trends in the development of
non-dyadic HRC and the use of DL models in group activity recognition are
discussed in two different paragraphs.

###  II-A Non-dyadic interactions in human-robot collaborations

Exploring non-dyadic interactions in HRI is not new [4]. Robotic architectures
have been built for the specific purpose of studying multi-party human-machine
interaction [13]. However, they mainly address situations where a shared task
with the user is not considered [14, 15]. Regarding a collaboration, non-
dyadic cases are mentioned in the taxonomy, but not explored in details [16,
17, 18]. Some works argue about holonic cyber-physical systems in which a
scene with multiple users sharing robotic resources is contemplated, but they
do not illustrate the idea with a practical case [19, 17]. Wang et al. [19]
also highlight the increasing need for robotic systems to be capable of
handling these cases in the future.

More recently, Tsamis et al. [20] carried out an experiment in which a non-
dyadic HRC was involved. In this work, two human workers were present in two
different workbenches at the same time, working on separate tasks. The mobile
manipulator present in the validation setup had to move objects from one place
to another, after one of the workers had accomplished their task.

In this work, we set a case scenario of a concurrent collaborative task
between two users and one robot. The proximity and intertwining of the two
roles in the design (see Section IV-A) make it fall under the technical
definition of a human-robot collaboration [21].

###  II-B Multi-user activity recognition through deep learning

Cases of multi-user activity recognition solved through DL are reported in
literature [22]. In 2016, Ibrahim et al. [23] used a LSTM to infer activity of
single users to then infer the pair activity through a second LSTM. Moreover,
Li et al. [24] combined convolutional neural networks (CNNs) and LSTM for the
same purpose. The main reason for this approach is that CNNs have limited
capabilities of incorporating time dependencies [8]. More recently, Shu et al.
[25] implemented graph networks in LSTMs and processed data coming from RGB
cameras.

In [26], a VAE was used to perform multi-user activity recognition from point-
cloud data through a transfer learning approach. Casserfelt et al. [27]
performed transfer learning with CNNs, exploiting data coming from multiple
cameras. Recently, a VAE has been used by Yasar et al. [6] for multi-user
motion prediction.

In this work, a VAE, composed of STGCN layers (see Subsection IV-C), and a
LSTM netowrk were trained separately by making use of 3D skeleton poses coming
from a depth camera to recognise group activity. In doing so, we propose a
novel way of generating training data regarding group activity, by pairing
recordings related to single users to constitute datapoints representing both
of them. The main reason for choosing two models widely used in literature is
to test the suitability of this data collection process to train different
kinds of DL models. Indeed, the LSTM is well-suited to incorporate long-term
time dependencies. Differently, the VAE may also incorporate dependencies of
this kind, but it is significantly more limited in these regards. Besides,
LSTMs learn through supervised learning, while VAEs perform classification
through self-supervised learning (see Subsection IV-C).

##  III Design concepts for concurrent non-dyadic HRC tasks

With so few cases of HRC scenarios of this kind [8], it is important to
outline concepts of the designed novel interaction. First, a contextualisation
of concurrent non-dyadic HRC tasks is provided. Then, a concurrent non-dyadic
HRC case of interest in manufacturing is described, to motivate the setting of
the experimental validation for the multi-user activity recognition system.

###  III-A Concurrency in collaborative tasks

In collaborative tasks, two agents cooperate to achieve a common goal. There
are different possible ways in which this collaboration can unfold. In turn-
taking collaborations, the agents mutually switch between activity and
inactivity while working in the shared workspace (e.g., collaborative
assembly) [28]. In joint tasks, they must act at the same time, due to the
constraints of the task itself (e.g., lifting a heavy load together from
opposite ends) [29]. However, there are concurrent tasks in which the two
agents need to complete a common goal together, but jointly performing
different operations, consequently causing mutual disturbances (e.g., tactical
vehicle control) [30].

The collaborations described so far consider only two agents in the scene. The
inclusion of a third agent makes sense if it performs a role different from
that of the others. If the third agent acts as an entity with its own sub-task
in the same common goal, thus interfering with the other agents, it could be
collapsed into one of the agents [18]. Indeed, in this case, from the
perception of one of the agents, it could be considered as a component of its
counterpart in the collaboration.

A third agent could, instead, perform the role of a mediator in the
collaboration. Through its intervention in the scene, it would not act on the
final goal directly; rather, it would indirectly assist in the collaboration
by mitigating the disturbances the two working agents may cause to each other,
to improve their joint performance. The other agents would both perceive the
third agent not as an extension of the counterpart, but as a different
standalone agent that supports their actions. This case is the one considered
in this work and is further described below.

###  III-B Multi-user activity recognition for concurrent tasks in non-dyadic
human-robot collaboration

Figure 1: A collaborative robot is assisting two human users working
concurrently on a manufacturing target (adapted from [31]).

Based on the previous considerations, a case design of a collaboration between
2 users and 1 robot for manufacturing applications is discussed. The main
reason to look at an industrial application is the recent repeatedly stated
need to consider this type of scenarios in manufacturing [19].

(a)

(b)

Figure 2: Representation of the concurrent non-dyadic human-robot
collaboration discussed, in terms of multiplicity (2a) [18] and roles of
humans and robot in relation to the task and each other (2b) [32]. In 2b, we
consider the humans with an active role, while the robot with an adaptive one.

Secondly, an industrial scenario envisions the workers performing gestures in
a repetitive manner, which allows for the conditions of the setup to be more
controlled. Nevertheless, these considerations do not exclude the potential
for applications outside manufacturing, such as social robotic interactions
[33, 34, 35].

In some production chains, one worker at a time works on the object of
manufacturing. Therefore, the productivity of the process would be enhanced by
enabling two workers to perform manufacturing operations on a product at the
same time. However, by doing so, there is a chance that the workers may
interfere with each other while performing operations on the object (e.g., by
changing the position and orientation of the object to have it in an optimal
configuration for the operation to perform, see Figure 1). Such issue could be
solved by the presence of a robot (e.g., a robotic manipulator endowed with
vision or force sensing) that acts as an intermediary between the two users.
In this way, the workers do not need to interact with each other while
working, as queries regarding the state of the manufacturing target can be
handled by the robot [5] (see Figure 2).

Because of this, team collaboration is replaced by the robot’s mediation and
the interaction between the users is reduced [5]. This allows us to collect
data from the scene by considering the users individually, to then treat the
data coming from pairs of acquisitions as unique data points, without losing
information. Being capable of acquiring data relatable to a pair setting in
this way allows us to save resources, when it comes to recruiting
participants. Indeed, instead of having pairs of people at the same time in
the setup, it is possible to record them individually and obtain good results
in terms of multi-user activity recognition.

To pinpoint which type of activities would be crucial to recognise in such a
scene, it is important to understand the role of the robot in the
collaboration. More precisely, it needs to know how to handle resources of the
setup (e.g., object of manufacturing) between the two users. Additionally, the
robot needs to understand which user requires its attention. When the users
are both working on the manufacturing, the robot needs to address requirements
from both of them. When one of them finishes a sub-step of the manufacturing
and prepares for the next one, the robot can shift all the resources to the
user now working on its own. Finally, a user can query the robot’s attention
through a signal (e.g., specific hand gesture) for the robot to be aware that
resources must be reallocated. The description of the unfolding of this
collaboration allows us to depict three main activities the robot needs to
detect from a single user:

  * •

working on the object of manufacturing;

  * •

preparing for the next task;

  * •

requesting a reallocation of resources from the robot.

Consequently, in terms of pair activity, 9 possible classes, related to the
possible pairings of these 3 states of each user, were considered in the
multi-user activity recognition problem. From this point on, these three
activities will be addressed as Working (W), Preparing (P) and Requesting (R),
respectively.

##  IV Materials and methods

This section delves into the technical design and validation of the multi-user
activity recognition pipeline proposed. Thanks to the specified features of
this novel scenario (see Subsection III-B), the experimental procedure built
to collect the training and test data is described. Then, a breakdown of the
preprocessing operations performed on the data is provided. Finally, the DL
models and methods for the classification problem are explained.

###  IV-A Data collection

Due to the novelty of this research, a dataset was collected to verify the
proposed hypothesis that using training data coming from single users is a
viable option to make inference from pair settings (see Section I). Not only
the scenario itself is new, but also the hypothesis on the training data does
not allow us to use available datasets regarding group activity [23]. Indeed,
by disentangling such data instances to have data from single users to then
merge back would not be effective to validate the hypothesis, which must use
data that are comparable to the target scenario, but not exactly the same.

(a)

(b)

Figure 3: Pictures from the experimental scenarios. In 3a, participants are
individually performing the requested actions (from left to right: Working,
Preparing and Requesting). All three of these instances will then be used to
generate the grouped data. In 3b, participants are both within the field of
view of the Kinect camera and are performing actions at the same time, case
that corresponds to pair data.

To generate the grouped data in post-processing (see Section I), 11
participants were recruited for the data collection. A Microsoft® Azure Kinect
camera [36] (resolution: 720P, frames per second: 30, depth mode:
NFOV_UNBINNED), controlled through a ROS2 Foxy workspace [37] on Ubuntu 20.04
running on a Lenovo® Thinkpad P53 (Intel® Core™ i7-9750H CPU, 2.60GHz), was
placed before the participants. With tools on a working table at their side,
the participants were then individually asked to perform 3 actions, each one
related to one of the aforementioned states (see Figure 3a):

  * •

performing an action relatable to the tool handled to an imaginary item of
cubic shape placed at their chest height (Working);

  * •

fetching an item from the working table, to prepare for the sub-task ahead
(Preparing);

  * •

rising a hand and hold it in place for some seconds (Requesting).

For the pair data collection, instead, 3 participants who did not participate
in the grouped one were recruited. They were asked to place themselves in
pairs in front of the camera controlled with the same previous settings and
asked to perform the aforementioned actions in a similar way (see Figure 3b).
In performing so, there was a greater level of complexity in these
acquisitions. Indeed, these recordings were aimed at reproducing a scene where
users work together. Therefore, instead of having many small recordings of a
single action, like in the grouped data collection, longer recordings were
made with the participants performing all the possible pairings of the three
states. Consequently, the participants were given a sequence of grouped
actions that they had to perform. These were executed naturally, without
giving them any signal of switching between them. During this collection, the
experimenter dynamically labeled the samples while being acquired.

###  IV-B Data preprocessing

Figure 4: Structure of the VAE model used. The Reshape layer does not
contribute to the learning, but it helps visualise that the network models a
distribution of the input data. The STGCN and AvPool2D layers were then used
to perform the classification.

Regarding the grouped data collection, the 3D skeletons underwent a
preprocessing procedure. Here, 11 meaningful joints were retained from the
initial set of 32 joints provided by the Kinect camera. They were the ones
related to head (not including the face), neck, torso (apart from the
clavicles) and arms. Then, they were normalised through the following
procedure [38]. Let us define
$\textit{{J}}=[\textit{{J}}_{0},\dots,\textit{{J}}_{10}]$ the vector of 3D
coordinates of the selected joints and J0 and J1 the vector of spatial
coordinates related to the spine naval and neck joints, respectively. Then,
the i-th normalised joint Ji is obtained as follows

$\textbf{{J}}_{i}=\frac{\textbf{{J}}_{i}-\textbf{{J}}_{0}}{\left\|\textbf{{J}}_{1}-\textbf{{J}}_{0}\right\|}$
(1)

Such operation granted the 3D poses spatial invariance from the acquisition
point [38] (see Figure 5). Furthermore, because of the formula itself, J0 will
always become $(0,0,0)$, losing any informative content. Therefore, after this
operation, the spine naval joint was removed. An instance of a 3D skeleton
pose is consequently composed of a 10x3 vector.

After this, the processed skeletons were grouped into time windows. Upon
inspection of the data, it was decided to have windows of 130 samples, with
104 samples overlapping between adjacent ones, for a time window to be related
to approximately 5 seconds of recordings and to introduce an approximately
1-second long new activity compared to adjacent time windows [39, 40]. Then,
data windows coming from one user’s recordings were concatenated with all the
other users’, generating the final grouped data points, composed of 130x20x3
coordinates (10 3D joint positions for each user in each frame). Finally, the
data subdued a process of min-max normalisation, to make the range of the
coordinates suitable to be an input to a neural network (see Subsection IV-C),
according to the formula

$\begin{split}\textbf{{J}}=&\frac{\textbf{{J}}-\min(\textbf{{J}})}{\max(\textbf{{J}})-\min(\textbf{{J}})}\times\\\
\\\
\times(\operatorname{new}_{-}\max(\textbf{{J}})&-\operatorname{new}_{-}\min(\textbf{{J}}))+\operatorname{new}_{-}\min(\textbf{{J}})\\\
\\\ \end{split}$ (2)

with $\min(\textbf{{J}})=-2.5$, $\max(\textbf{{J}})=1.75$,
$\operatorname{new}_{-}\min(\textbf{{J}})=0$ and
$\operatorname{new}_{-}\max(\textbf{{J}})=1$. These values were chosen by
looking at the ranges of values of the grouped dataset. The same values proved
to be suitable for the pair data, as well.

(a)

(b)

Figure 5: Visual representation of a pair of 3D skeleton poses acquired at the
same time, relatively to the Kinect camera frame, before (5a) and after (5b)
pruning and first normalisation. In both plots, the participant on the left is
working, and the one on the right is preparing for the next task. Figure 6:
Structure of the cascaded LSTM network used.

Regarding the preprocessing of the pair recordings, the normalisations were
applied likewise. There was no need to pair data from different recordings,
because of the experimental procedure itself (see Subsection IV-A). However,
it was crucial to properly discern every one of them, according to the related
participant, from the data flow of skeleton frames broadcast by the camera.
Indeed, because of the devised labeling (see Subsection III-B), the system
will be able to understand not only the couple of actions being performed, but
also which user is executing which. For instance, the cases Working-Preparing
and Preparing-Working were treated as two different cases. Consequently, the
data needed to be allocated properly in the 130x20x3 windows. Regarding the
labeling, a data window was assigned the same label of its last pair of
skeleton poses. Because the labeling was manually performed during the
acquisitions (see Subsection IV-A), some instances related to the transition
from a couple of states to the next one could have been mislabeled.
Consequently, data windows related to the 2 seconds of recording before and
after a registered change of the label were discarded, to get a dataset less
affected by human errors.

###  IV-C Learning framework

To test the hypothesis on the grouped data (see Section I), two DL models were
used. The first one is composed of stacked LSTM networks [23] (see Figure 6),
to which a softmax layer of 9 neurons (related to the 9 classes to recognise,
that were converted to output vectors through a one-hot encoding) was
appended. This network underwent a supervised learning process by means of
backpropagation through time.

The second model was a VAE [26]. This neural network exploits a learning
method related to self-supervised learning. It makes use of unlabeled data,
but it uses them as labels themselves. The main objective of a VAE is to learn
a variational posterior distribution $\textit{q({{z}}/{{x}}})$ of latent
variables z, given the input data x. This is done by the first half of the
network, which is called encoder. Modeling this last probability distribution
is done by comparing it with a prior distribution p(z), which in this work was
a $\mathcal{N}\textit{({{0}},{{I}})}$, with 0 as vector of zeros and I as
identity matrix. The variational posterior then generates instances of z
through sampling. They are used by the second half of the network, that is the
decoder, to learn a likelihood distribution $\textit{p({{x}}/{{z}}})$ of the
input data x, given the latent variables z, which is also learnt by the
network. This network learns by maximizing the evidence lower bound (ELBO)
loss, defined as

$\textit{ELBO}=\mathbb{E}_{q_{\phi}(\textit{{z}}|\textit{{x}})}\left[\log
p_{\theta}(\textit{{x}}|\textit{{z}})\right]-KL(q_{\phi}(\textit{{z}}|\textit{{x}})\|p(\textit{{z}}))$
(3)

where KL is the Kullback-Leibler constrastive divergence [41], measure of the
discrepancy between two distributions, and $\theta$ and $\phi$ are the weights
of the connections of the decoder and encoder, respectively.

The layout of the VAE is shown in Figure 4 [42]. The Spatio-Temporal Graph
Convolutional Network (STGCN) layers are CNNs with adjacency matrices that are
renowned for being able to learn spatial and temporal patterns of data [43].
Once the training of the VAE was over, the trained STGCNs and the average
pooling layer were extracted from the network and, similarly to the previously
mentioned neural network, the softmax layer of 9 neurons was appended to it.
This new neural network was then trained on labeled test data, keeping the
weights and connections of the STGCNs unchanged. Consequently, only the
softmax layer was trained during this phase. For this part of the training, a
categorical crossentropy was used as the loss function. From this point on,
the network composed of LSTMs will be addressed as LSTM, while the STGCNs
layers trained through the VAE by means of transfer learning approach will be
referred as VAE. The training and testing of both networks were performed on a
container to which 2 Nvidia RTX 2080 Ti and 7 virtual CPUs (2.20 GHz), from a
local server, were allocated, and in which distributed learning strategy [44]
by means of Tensorflow 2 GPU and Keras libraries were used [45].

##  V Results

To evaluate whether it is possible to get valuable performances on pair data
when training DL models with grouped data, the following analyses were carried
out. First, training and testing by using pair data only was performed. This
allowed us to get a performance baseline when testing with the pair data.
Secondly, the depicted DL models were trained and tested with grouped data, to
assess whether the models were suitable to learn from the grouped data. When
the models reached a considerably good performance on this test, they then
underwent the last test. In this, the networks were trained with grouped data
and tested with pair data, to assess the validity of the hypothesis. Every
training was performed by implementing a leave-one-subject-out (LOSO)
crossvalidation.

TABLE I: Accuracy and F score values in the analysed cases. Model | Training | Test | Accuracy | F score
---|---|---|---|---
| data | data | Mean | SDa | Mean | SD
LSTM | Pair | Pair | 0.548 | 0.139 | 0.676 | 0.116
LSTM | Grouped | Grouped | 0.786 | 0.085 | 0.776 | 0.094
LSTM | Grouped | Pair | 0.615 | 0.111 | 0.584 | 0.115
VAE | Pair | Pair | 0.614 | 0.114 | 0.586 | 0.119
VAE | Grouped | Grouped | 0.864 | 0.110 | 0.996 | 0.002
VAE | Grouped | Pair | 0.616 | 0.029 | 0.594 | 0.027
aStandard deviation |  |  |  |  |

(a)

(b)

Figure 7: Confusion matrices related to the classification performance of the
LSTM (7a) and VAE (7b) when trained with grouped data and tested with pair
data. The ground truth is plotted along the lines of the matrices, while the
output of the classifier along the columns. The ratio values were obtained by
dividing the number of entries in a cell by the sum of entries of the related
line.

The overall performances across all training instances are reported in Table
I. The reported values always refer to the test performed with the specified
test data. In case the fields Training data and Test data have the same entry,
the test set is the fold at the different iterations of the LOSO
crossvalidation. Otherwise, it is the whole dataset related to that
denomination (see Section I). As can be appreciated in Table I, the
performance of the networks, when trained with grouped data and tested with
pair data, is comparable to the cases in which they were trained and tested
with pair data. Therefore, as a second-level analysis, the specific
performance of the DL models for each class was also surveyed. This allowed us
to collect more detailed information about the whole procedure described, to
find room for future improvements. Figure 7 reports the confusion matrices of
the LSTM and VAE models, using grouped data for the training and pair data for
the test.

##  VI Discussions

The achieved results confirm that it is possible to make use of data collected
from single users to infer related group activity without having loss in
performance. Indeed, as can be seen in Table I, in the VAE case with grouped
data used in the training and pair data in the test, the performance of the
classifiers is similar to the performance achieved by the same DL model
trained and tested with pair data, both in accuracy and F scores. In the LSTM
case, the accuracy value with the use of pair data only during testing is
slightly higher than when the pair data were used both for training and
testing, although this is evened out by the slightly lower value of F score.
Besides, looking at the standard deviation values of accuracy and F scores, it
is possible to point out that the VAE model generally showed more stable
performances than the LSTM model. In the cases just considered, the F score
values are not considerably high. Despite this, the models were still proven
to be able to learn from the provided training data. Indeed, once provided
with grouped data for training and testing, both models are able to achieve
good performances, followed by high F scores.

This demonstrates that it is feasible to train DL models for multi-user
activity recognition, using data from single users grouped at the post-
processing stage – which significantly decreases feasibility issues related to
safety restrictions [12] and recruiting participants for non-dyadic HRC
studies [11]. The performance of this multi-user activity recognition pipeline
has proven successful for the purpose of verifying the hypothesis on grouped
data, but it still has large room for improvement. First, the results showed
higher reliability on the pair data than on the grouped ones. As such, future
related works and experimental settings would benefit from the development of
new procedures to acquire data that would strengthen the already promising
results achieved in this research from this perspective. As can be observed in
Figure 7, distinguishing between the Requesting and Working states has
constituted one of the main challenges for both DL models. This appears to be
mostly due to the hand position in the Requesting state, which may be confused
with that of Working. Future works could use more diverse gestures for these
states, in order to improve the performance of the classifier. Moreover,
further defining the characteristics of the interaction would also simplify
the classification task. Indeed, this work has considered all possible cases
of a concurrent non-dyadic HRC. However, some of them could not happen in a
specific study case, hence they may not be worth learning, decreasing the
complexity of the situations the classifier is exposed to.

##  VII Conclusions

This work has delineated outlines of experimental design for concurrent
collaborations and depicted them in the case of non-dyadic human-robot
collaborations. For these scenarios, this research proposed a novel way to
collect data about pairs of people for training a multi-agent activity
recognition system based on deep learning models. In particular, training data
was realised by collecting users’ activities individually and pairing them in
post-processing. These data were used to separately train a long short-term
memory network and a variational autoencoder to classify activities among 9
possible states of the pair of workers. The layout and training method of the
networks differed from each other to test the suitability of different models
to the classification problem. Both were trained with data related to grouped
settings and proved not to lose in performance when tested on pair data,
compared to when trained uniquely with datapoints from a pair setting. This
result is promising and shows that it is possible to learn from data acquired
in this way, relieving from the organisational burden of setting up multi-
party scenarios for these applications.

## Acknowledgments

Francesco Semeraro’s work was supported by the DTP CASE Conversion award
"Human-Robot Collaboration for Flexible Manufacturing" (Ref. 2480772), funded
by UKRI Engineering and Physical Sciences Research Council and BAE Systems
Plc. Angelo Cangelosi’s work was partially supported by the H2020 project
TRAINCREASE (Ref. 210638710), the UKRI Trustworthy Autonomous Systems Node on
Trust (Ref. EP/V026682/1) and the US Air Force project THRIVE++ (Ref.
FA9550-19-1-7002).

## References

  * [1] Arash Ajoudani et al.  “Progress and prospects of the human–robot collaboration”  In _Autonomous Robots_ 42.5 Springer, 2018, pp. 957–975
  * [2] Eva Hornecker, Antonia Krummheuer, Andreas Bischof and Matthias Rehm  “Beyond dyadic HRI: building robots for society”  In _interactions_ 29.3 ACM New York, NY, USA, 2022, pp. 48–53
  * [3] Eike Schneiders  “Non-Dyadic Human-Robot Interaction: Concepts and Interaction Techniques”  In _HRI’22: ACM/IEEE International Conference on Human-Robot Interaction_ , 2022, pp. 1176–1178  IEEE Press
  * [4] Eike Schneiders et al.  “Non-dyadic interaction: A literature review of 15 years of human-robot interaction conference publications”  In _ACM Transactions on Human-Robot Interaction (THRI)_ 11.2 ACM New York, NY, 2022, pp. 1–32
  * [5] Eike Schneiders and Stanley Celestin  “Non-Dyadic Entrainment for Industrial Tasks”  In _Workshop on Joint Action, Adaptation, and Entrainment in Human-Robot Interaction_ , 2022
  * [6] Mohammad Samin Yasar and Tariq Iqbal  “A scalable approach to predict multi-agent motion for human-robot collaboration”  In _IEEE Robotics and Automation Letters_ 6.2 IEEE, 2021, pp. 1686–1693
  * [7] Francesco Semeraro, Jon Carberry and Angelo Cangelosi  “Simpler Rather than Challenging: Design of Non-Dyadic Human-Robot Collaboration to Mediate Human-Human Concurrent Tasks”  In _Proceedings of the 22nd International Conference on Autonomous Agents and Multiagent Systems_ , 2023
  * [8] Francesco Semeraro, Alexander Griffiths and Angelo Cangelosi  “Human–robot collaboration and machine learning: A systematic review of recent research”  In _Robotics and Computer-Integrated Manufacturing_ 79 Elsevier, 2023, pp. 102432
  * [9] Paul Baxter et al.  “From characterising three years of HRI to methodology and reporting recommendations”  In _2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI)_ , 2016, pp. 391–398  IEEE
  * [10] Priscilla Briggs, Matthias Scheutz and Linda Tickle-Degnen  “Are Robots Ready for Administering Health Status Surveys’ First Results from an HRI Study with Subjects with Parkinson’s Disease”  In _Proceedings of the tenth annual acm/ieee international conference on human-robot interaction_ , 2015, pp. 327–334
  * [11] Serkan Cabi et al.  “A framework for data-driven robotics”  In _arXiv preprint arXiv:1909.12200_ , 2019
  * [12] David Feil-Seifer et al.  “Where to next? The impact of COVID-19 on human-robot interaction research”  In _ACM Transactions on Human-Robot Interaction (THRI)_ 10.1 ACM New York, NY, USA, 2020, pp. 1–7
  * [13] Samer Al Moubayed, Jonas Beskow, Gabriel Skantze and Björn Granström  “Furhat: a back-projected human-like robot head for multiparty human-machine interaction”  In _Cognitive behavioural systems_ Springer, 2012, pp. 114–130
  * [14] Mary Ellen Foster, Simon Keizer, Zhuoran Wang and Oliver Lemon  “Machine learning of social states and skills for multi-party human-robot interaction”  In _Proceedings of the workshop on Machine Learning for Interactive Systems (MLIS 2012)_ , 2012, pp. 9
  * [15] Mary Ellen Foster, Andre Gaschler and Manuel Giuliani  “Automatically classifying user engagement for dynamic multi-party human–robot interaction”  In _International Journal of Social Robotics_ 9.5 Springer, 2017, pp. 659–674
  * [16] Lihui Wang, Sichao Liu, Hongyi Liu and Xi Vincent Wang  “Overview of human-robot collaboration in manufacturing”  In _Proceedings of 5th international conference on the industry 4.0 model for advanced manufacturing_ , 2020, pp. 15–58  Springer
  * [17] Mohammad A Shehadeh, Stefan Schroeder, Anja Richert and Sabina Jeschke  “Hybrid teams of industry 4.0: A work place considering robots as key players”  In _2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC)_ , 2017, pp. 1208–1213  IEEE
  * [18] Holly A Yanco and Jill Drury  “Classifying human-robot interaction: an updated taxonomy”  In _2004 IEEE international conference on systems, man and cybernetics (IEEE Cat. No. 04CH37583)_ 3, 2004, pp. 2841–2846  IEEE
  * [19] Lihui Wang and Azadeh Haghighi  “Combined strength of holons, agents and function blocks in cyber-physical systems”  In _Journal of manufacturing systems_ 40 Elsevier, 2016, pp. 25–34
  * [20] Georgios Tsamis et al.  “Intuitive and Safe Interaction in Multi-User Human Robot Collaboration Environments through Augmented Reality Displays”  In _2021 30th IEEE International Conference on Robot & Human Interactive Communication (RO-MAN)_, 2021, pp. 520–526  IEEE
  * [21] Eloise Matheson et al.  “Human–robot collaboration in manufacturing applications: A review”  In _Robotics_ 8.4 MDPI, 2019, pp. 100
  * [22] Qimeng Li et al.  “Multi-user activity recognition: Challenges and opportunities”  In _Information Fusion_ 63 Elsevier, 2020, pp. 121–135
  * [23] Mostafa S Ibrahim et al.  “A hierarchical deep temporal model for group activity recognition”  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 1971–1980
  * [24] Xinyu Li et al.  “Concurrent activity recognition with multimodal CNN-LSTM structure”  In _arXiv preprint arXiv:1702.01638_ , 2017
  * [25] Xiangbo Shu, Liyan Zhang, Yunlian Sun and Jinhui Tang  “Host–parasite: Graph LSTM-in-LSTM for group activity recognition”  In _IEEE transactions on neural networks and learning systems_ 32.2 IEEE, 2020, pp. 663–674
  * [26] Mohammad Arif Ul Alam, Md Mahmudur Rahman and Jared Q Widberg  “Palmar: Towards adaptive multi-inhabitant activity recognition in point-cloud technology”  In _IEEE INFOCOM 2021-IEEE Conference on Computer Communications_ , 2021, pp. 1–10  IEEE
  * [27] Karl Casserfelt and Radu-Casian Mihailescu  “An investigation of transfer learning for deep architectures in group activity recognition”  In _2019 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops)_ , 2019, pp. 58–64  IEEE
  * [28] Samuele Vinanzi, Angelo Cangelosi and Christian Goerick  “The Role of Social Cues for Goal Disambiguation in Human-Robot Cooperation”  In _2020 29th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)_ , 2020, pp. 971–977  IEEE
  * [29] Loris Roveda et al.  “Model-based reinforcement learning variable impedance control for human-robot collaboration”  In _Journal of Intelligent & Robotic Systems_ 100.2 Springer, 2020, pp. 417–433
  * [30] William J Horrey and Daniel J Simons  “Examining cognitive interference and adaptive safety behaviours in tactical vehicle control”  In _Ergonomics_ 50.8 Taylor & Francis, 2007, pp. 1340–1350
  * [31] Corey Ryan  “The Fine Line between Industrial and Collaborative Robots: It’s Smaller than You Think”, Article, 2020  URL: https://www.qualitymag.com/articles/96119-the-fine-line-between-industrial-and-collaborative-robots-its-smaller-than-you-think
  * [32] Xi Vincent Wang, Zsolt Kemény, József Váncza and Lihui Wang  “Human–robot collaborative assembly in cyber-physical production: Classification framework and implementation”  In _CIRP annals_ 66.1 Elsevier, 2017, pp. 5–8
  * [33] Francesco Semeraro et al.  “Physiological wireless sensor network for the detection of human moods to enhance human-robot interaction”  In _Italian Forum of Ambient Assisted Living_ , 2018, pp. 361–376  Springer
  * [34] Laura Fiorini et al.  “Physiological sensor system for the detection of human moods towards internet of robotic things applications”  In _New Trends in Intelligent Software Methodologies, Tools and Techniques_ IOS Press, 2018, pp. 967–980
  * [35] Filippo Cavallo et al.  “Mood classification through physiological parameters”  In _Journal of Ambient Intelligence and Humanized Computing_ 12.4 Springer, 2021, pp. 4471–4484
  * [36] Michal Tölgyessy, Martin Dekan and L’uboš Chovanec  “Skeleton tracking accuracy and precision evaluation of Kinect V1, Kinect V2, and the azure kinect”  In _Applied Sciences_ 11.12 Multidisciplinary Digital Publishing Institute, 2021, pp. 5756
  * [37] Yuya Maruyama, Shinpei Kato and Takuya Azumi  “Exploring the performance of ROS2”  In _Proceedings of the 13th International Conference on Embedded Software_ , 2016, pp. 1–10
  * [38] Samuele Vinanzi, Christian Goerick and Angelo Cangelosi  “Mindreading for robots: Predicting intentions via dynamical clustering of human postures”  In _2019 Joint IEEE 9th International Conference on Development and Learning and Epigenetic Robotics (ICDL-EpiRob)_ , 2019, pp. 272–277  IEEE
  * [39] Jozsef Suto, Stefan Oniga, Claudiu Lung and Ioan Orha  “Comparison of offline and real-time human activity recognition results using machine learning techniques”  In _Neural computing and applications_ 32.20 Springer, 2020, pp. 15673–15686
  * [40] Andrey Ignatov  “Real-time human activity recognition from accelerometer data using Convolutional Neural Networks”  In _Applied Soft Computing_ 62 Elsevier, 2018, pp. 915–922
  * [41] James M Joyce  “Kullback-leibler divergence”  In _International encyclopedia of statistical science_ Springer, 2011, pp. 720–722
  * [42] Ali Ghadirzadeh et al.  “Human-centered collaborative robots with deep reinforcement learning”  In _IEEE Robotics and Automation Letters_ 6.2 IEEE, 2020, pp. 566–571
  * [43] Sijie Yan, Yuanjun Xiong and Dahua Lin  “Spatial temporal graph convolutional networks for skeleton-based action recognition”  In _Thirty-second AAAI conference on artificial intelligence_ , 2018
  * [44] Nguyen Quang-Hung, Hieu Doan and Nam Thoai  “Performance evaluation of distributed training in Tensorflow 2”  In _2020 International Conference on Advanced Computing and Applications (ACOMP)_ , 2020, pp. 155–159  IEEE
  * [45] Young Jong Mo et al.  “Performance of deep learning computation with TensorFlow software library in GPU-capable multi-core computing platforms”  In _2017 Ninth International Conference on Ubiquitous and Future Networks (ICUFN)_ , 2017, pp. 240–242  IEEE