Restriction of Odd Degree Characters of $\mathfrak{S}_{n}$

Restriction of Odd Degree Characters of $\boldsymbol{\mathfrak{S}_{n}}$††This
paper is a contribution to the Special Issue on the Representation Theory of
the Symmetric Groups and Related Topics. The full collection is available at
https://www.emis.de/journals/SIGMA/symmetric-groups-2018.html

Christine BESSENRODT †, Eugenio GIANNELLI ‡ and Jørn B. OLSSON §

C. Bessenrodt, E. Giannelli and J.B. Olsson

† Institute for Algebra, Number Theory and Discrete Mathematics,
† Leibniz Universität Hannover, Welfengarten 1, D-30167 Hannover, Germany
<EMAIL_ADDRESS>
‡ Department of Pure Mathematics and Mathematical Statistics, University of
Cambridge,
‡ Cambridge CB3 0WA, United Kingdom<EMAIL_ADDRESS>
§ Department of Mathematical Sciences, University of Copenhagen,
§ DK-2100 Copenhagen Ø, Denmark<EMAIL_ADDRESS>
Received May 25, 2017, in final form August 30, 2017; Published online
September 05, 2017

Let $n$ and $k$ be natural numbers such that $2^{k}<n$. We study the
restriction to $\mathfrak{S}_{n-2^{k}}$ of odd-degree irreducible characters
of the symmetric group $\mathfrak{S}_{n}$. This analysis completes the study
begun in [Ayyer A., Prasad A., Spallone S., Sém. Lothar. Combin. 75 (2015),
Art. B75g, 13 pages] and recently developed in [Isaacs I.M., Navarro G.,
Olsson J.B., Tiep P.H., J. Algebra 478 (2017), 271–282].

characters of symmetric groups; hooks in partitions

20C30; 05A17

##  1 Introduction

Let $n$ be a natural number, and let $\chi$ be an irreducible character of odd
degree of the symmetric group ${\mathfrak{S}}_{n}$. Then there exists a unique
odd-degree irreducible constituent of the restriction
$\chi_{{\mathfrak{S}}_{n-1}}$. This interesting fact was discovered recently
in [1]. The result had immediate applications in the study of natural
correspondences of characters of finite groups (see for example [2]). In [3,
Theorem A] the result mentioned above was generalized, by showing that given
any $k\in\mathbb{N}$ such that $2^{k}<n$, there exists a unique odd-degree
irreducible constituent $f_{k}^{n}(\chi)$ of $\chi_{{\mathfrak{S}}_{n-2^{k}}}$
appearing with odd multiplicity. The main goal of this article is to study for
all $n,k\in\mathbb{N}$ the map

$\displaystyle f_{k}^{n}\colon\
\operatorname{Irr}_{2^{\prime}}({\mathfrak{S}}_{n})\longrightarrow\operatorname{Irr}_{2^{\prime}}({\mathfrak{S}}_{n-2^{k}}),$

naturally defined by Theorem A of [3]. All our results are proved using a
description of $f_{k}^{n}$ in terms of the natural partition labels of the
involved irreducible characters.

Before describing the main results of this paper, we introduce some
vocabulary. If $2^{k}$ appears in the binary expansion of $n$ we say that
$2^{k}$ is a binary digit of $n$. Similarly we say that two natural numbers
$m$ and $n$ are $2$-disjoint if they do not have any common binary digit. On
the other hand, if $m\leq n$ and all the binary digits of $m$ appear in the
binary expansion of $n$, then we say that $m$ is a binary subsum of $n$. This
will be denoted by $m\subseteq_{2}n$. Let $\nu_{2}(n)$ be the exponent of the
highest power of 2 dividing the integer $n$.

A question raised in [3] may be phrased as: For which $n$ and $k$ is
$f^{n}_{k}$ surjective? The authors showed that $f^{n}_{k}$ is surjective
whenever $2^{k}$ is a binary digit of $n$, and they observed that otherwise
$f^{n}_{k}$ could be both surjective or not (see [3, Proposition 4.5 and
Remark 4.6]). In this paper we answer the question of surjectivity completely
with the following result.

######  Theorem A.

Let $n\in\mathbb{N}$, $k\in\mathbb{N}_{0}$ be such that $2^{k}<n$. Let
$d(n,k)=\nu_{2}\big{(}\big{\lfloor}\frac{n}{2^{k}}\big{\rfloor}\big{)}$.

  * •

If $k=0$ then $f^{n}_{k}$ is surjective if and only $d(n,k)\leq 2$.

  * •

If $k>0$ then $f^{n}_{k}$ is surjective if and only $d(n,k)\leq 1$.

Theorem A is a consequence of Theorem 3.5 below, which describes the images of
the maps $f^{n}_{k}$.

For all $n\in\mathbb{N}$, $k\in\mathbb{N}_{0}$ with $2^{k}<n$ and any
$\psi\in\operatorname{Irr}_{2^{\prime}}({\mathfrak{S}}_{n-2^{k}})$ we define
the set

$\displaystyle\mathcal{E}\big{(}\psi,2^{k}\big{)}=\big{\\{}\chi\in\operatorname{Irr}_{2^{\prime}}({\mathfrak{S}}_{n})\,|\,f_{k}^{n}(\chi)=\psi\big{\\}},$

and set
$e\big{(}\psi,2^{k}\big{)}=\big{|}\mathcal{E}\big{(}\psi,2^{k}\big{)}\big{|}$.
We show in Corollary 3.8 that the maps $f^{n}_{k}$ are regular on their
images. This means that for any $\psi$ in the image of $f^{n}_{k}$, the number
$e(\psi,2^{k})$ depends only on $n$ and $k$ and not on the specific $\psi$. We
also give a complete description of those
$\psi\in\operatorname{Irr}_{2^{\prime}}({\mathfrak{S}}_{n-2^{k}})$ such that
$e(\psi,2^{k})=0$, in Theorem 3.5.

In the final part of the paper we study commutativity. For convenience, we
sometimes denote $f_{k}^{n}$ just by $f_{k}$, when the natural number $n$ is
clear from the context. Then, for $k,{\ell}\in\mathbb{N}_{0}$, $k<\ell$, such
that $2^{k}+2^{\ell}\leq n$, we may ask: when is
$f_{k}f_{\ell}=f_{\ell}f_{k}$? or more specifically: when is
$f^{n-2^{\ell}}_{k}f^{n}_{\ell}=f^{n-2^{k}}_{\ell}f^{n}_{k}$? In [3,
Proposition 4.3] it was proved that $f_{k}f_{\ell}=f_{\ell}f_{k}$ whenever
$2^{\ell}<n<2^{\ell+1}$. This is the case $\ell=t$ in our second main result,
which answers the question completely.

######  Theorem B.

Let $n=2^{t}+m$ where $0\leq m<2^{t}$. Suppose that $k$, ${\ell}$ satisfy
$0\leq k<{\ell}\leq t$ and $2^{k}+2^{\ell}\leq n$. Then, with the exception of
the case $n=6$, $k=0$, ${\ell}=1$,

$\displaystyle\text{$f_{k}f_{\ell}=f_{\ell}f_{k}$ if and only if $2^{k}>m$ or
$\ell=t$}.$

##  2 Notation and background

Let $n$ be a natural number. We let $\operatorname{Irr}({\mathfrak{S}}_{n})$
denote the set of irreducible characters of ${\mathfrak{S}}_{n}$ and
$\mathcal{P}(n)$ the set of partitions of $n$. The notation
$\lambda\in\mathcal{P}(n)$ is sometimes replaced by $\lambda\vdash n$ and we
write $|\lambda|=n$. There is a natural correspondence
$\lambda\leftrightarrow\chi^{\lambda}$ between $\mathcal{P}(n)$ and
$\operatorname{Irr}({\mathfrak{S}}_{n})$. We say then that $\lambda$ labels
$\chi^{\lambda}$. We denote by
$\operatorname{Irr}_{2^{\prime}}({\mathfrak{S}}_{n})$ the set of irreducible
characters of ${\mathfrak{S}}_{n}$ of odd degree. If
$\chi^{\lambda}\in\operatorname{Irr}_{2^{\prime}}({\mathfrak{S}}_{n})$ we say
that $\chi^{\lambda}$ is an odd character, we call $\lambda$ an odd partition
of $n$ and write $\lambda\vdash_{o}n$. Also the empty partition will be
considered as an odd partition.

######  Remark 2.1.

Let $n,k$ be such that $2^{k}<n$. In [3, Theorem A and Proposition 4.2] it is
shown that the map
$f_{k}^{n}\colon\operatorname{Irr}_{2^{\prime}}({\mathfrak{S}}_{n})\to\operatorname{Irr}_{2^{\prime}}({\mathfrak{S}}_{n-2^{k}})$
may be described in terms of the odd partitions labelling the odd characters
as follows:

$\displaystyle
f_{k}^{n}(\chi^{\lambda})=\chi^{\mu}\Leftrightarrow\mu\vdash_{o}n-2^{k}\text{
can be obtained from }\lambda\vdash_{o}n\text{ by removing a
}2^{k}\text{-hook}.$

Correspondingly we write (by abuse of notation) $f_{k}^{n}(\lambda)=\mu$. In
fact when $\lambda$ is odd, there is only one $2^{k}$-hook of $\lambda$ whose
removal leads again to an odd partition; we will refer to such a hook as an
odd hook of $\lambda$. This combinatorial description of $f^{n}_{k}$ will be
used throughout this paper, and we will regard $f^{n}_{k}$ also as a map
between the corresponding sets of odd partitions. Also, for
$\mu\vdash_{o}n-2^{k}$ we set
$e\big{(}\mu,2^{k}\big{)}=e\big{(}\chi^{\mu},2^{k}\big{)}$.

We need some concepts and basic facts concerning hooks in partitions. For any
integer $e\in\mathbb{N}$ we denote by $C_{e}(\lambda)$ and $Q_{e}(\lambda)$
the $e$-core and the $e$-quotient of $\lambda$, respectively. Then
$Q_{e}(\lambda)=(\lambda_{0},\ldots,\lambda_{e-1})$ is an $e$-tuple of
partitions satisfying
$n=|C_{e}(\lambda)|+e\sum\limits_{i=0}^{e-1}|\lambda_{i}|$. It is well-known
that a partition is uniquely determined by its $e$-core and $e$-quotient (we
refer the reader to [6] or [4, Chapter 2.7] for a detailed discussion on this
topic).

Let $\mathcal{H}_{e}(\lambda)$ be the set of hooks of $\lambda$ having length
divisible by $e$, and let
$\mathcal{H}(Q_{e}(\lambda))=\cup_{i=0}^{e-1}\mathcal{H}(\lambda_{i})$. As
explained in [6, Theorem 3.3], there is a bijection between
$\mathcal{H}_{e}(\lambda)$ and $\mathcal{H}(Q_{e}(\lambda))$ mapping hooks in
$\lambda$ of length $ex$ to hooks in the quotient of length $x$. Moreover, the
bijection respects the process of hook removal. Namely, the partition $\mu$
obtained by removing a $ex$-hook from $\lambda$ is such that
$C_{e}(\mu)=C_{e}(\lambda)$ and the $e$-quotient of $\mu$ is obtained by
removing an $x$-hook from one of the partitions involved in $Q_{e}(\lambda)$.

For $e=2$ we want to repeat the process of taking 2-cores and 2-quotients to
obtain the $2$-quotient tower $\mathcal{Q}_{2}(\lambda)$ and the $2$-core
tower $\mathcal{C}_{2}(\lambda)$ of $\lambda$. They have rows numbered by
$k\geq 0$. The $k$th row $\mathcal{Q}^{(k)}_{2}(\lambda)$ of
$\mathcal{Q}_{2}(\lambda)$ contains $2^{k}$ partitions $\lambda^{(k)}_{i}$,
$0\leq i\leq 2^{k}-1$, and the $k$th row $\mathcal{C}^{(k)}_{2}(\lambda)$ of
$\mathcal{C}_{2}(\lambda)$ contains the $2$-cores of these partitions in the
same order, i.e., $C_{2}\big{(}\lambda^{(k)}_{i}\big{)}$, $0\leq i\leq
2^{k}-1$. The 0th row of $\mathcal{Q}_{2}(\lambda)$ contains
$\lambda=\lambda^{(0)}_{0}$ itself, row 1 contains the partitions
$\lambda^{(1)}_{0}$, $\lambda^{(1)}_{1}$ occurring in the $2$-quotient
$Q_{2}(\lambda)$, row 2 contains the partitions occurring in the $2$-quotients
of partitions occurring in row 1, and so on. Specifically we have
$Q_{2}\big{(}\lambda^{(k)}_{i}\big{)}=\big{(}\lambda^{(k+1)}_{2i},\lambda^{(k+1)}_{2i+1}\big{)}$
for $i\in\big{\\{}0,1,\dots,2^{k}-1\big{\\}}$. We remark that the $2^{k}$
partitions in $\mathcal{Q}^{(k)}_{2}(\lambda)$ are the same as those in the
$2^{k}$-quotient $Q_{2^{k}}(\lambda)$ of $\lambda$, but in a different order
for $k\geq 2$.

We also introduce the $k$-data $\mathcal{D}^{(k)}_{2}(\lambda)$ of $\lambda$.
This is a table containing the following $k+1$ rows: the $k$ rows
$\mathcal{C}^{(j)}_{2}(\lambda)$, $j=0,\dots,k-1$, and in addition the row
$\mathcal{Q}^{(k)}_{2}(\lambda)$.

######  Remark 2.2.

A partition $\lambda$ may be recovered from its $2$-core tower. For $k>0$, it
may also be recovered from the knowledge of the $k$-data
$\mathcal{D}^{(k)}_{2}(\lambda)$ of $\lambda$, because the rows
$\mathcal{C}^{(l)}_{2}(\lambda)$ with $l\geq k$ of $\mathcal{C}_{2}(\lambda)$
consist of the $2$-core towers of the partitions in
$\mathcal{Q}^{(k)}_{2}(\lambda)$.

######  Lemma 2.3.

Suppose that $\lambda\vdash n-2^{k}$ and $\mu\vdash n$. The following are
equivalent.

  * $(i)$

$\lambda$ is obtained from $\mu$ by removing a $2^{k}$-hook.

  * $(ii)$

The $k$-data $\mathcal{D}^{(k)}_{2}(\mu)$ and $\mathcal{D}^{(k)}_{2}(\lambda)$
coincide, except that for one $i\in\big{\\{}0,\ldots,2^{k}-1\big{\\}}$
$\lambda^{(k)}_{i}$ is obtained from $\mu^{(k)}_{i}$ by removing a $1$-hook.

###### Proof.

A $2^{k}$-hook $H_{0}$ in $\mu$ corresponds in a canonical way to a
$2^{k-1}$-hook $H_{1}$ in a partition in $\mathcal{Q}^{(1)}_{2}(\mu)$, i.e.,
in row 1 of the $2$-quotient tower $\mathcal{Q}_{2}(\mu)$. Continuing we see
that $H_{0}$ corresponds in a canonical way to a 1-hook $H_{k}$ in a partition
$\mu^{(k)}_{i}$ in $\mathcal{Q}^{(k)}_{2}(\mu)$, row $k$ of
$\mathcal{Q}_{2}(\mu)$. If $\lambda$ is obtained by removing $H_{0}$ from
$\mu$, this corresponds to $\lambda^{(k)}_{i}$ being obtained by removing the
1-hook $H_{k}$ from $\mu^{(k)}_{i}$ (by repeated applications of [6, Theorem
3.3]). Apart from this the rows $\mathcal{Q}^{(k)}_{2}(\mu)$ and
$\mathcal{Q}^{(k)}_{2}(\lambda)$ coincide. Note also that the rows
$\mathcal{C}^{(j)}_{2}(\mu)$ and $\mathcal{C}^{(j)}_{2}(\lambda)$ coincide for
$j=0,\dots,k-1$, since the removal of the hooks $H_{j}$ of even length do not
change the 2-cores. ∎

Odd-degree characters of ${\mathfrak{S}}_{n}$ and thus odd partitions were
completely described in [5]. We restate this result in a language which is
convenient for our purposes. We let $c_{2}^{(k)}(\lambda)$ be the sum of the
cardinalities of the partitions in the $k$th row
$\mathcal{C}^{(k)}_{2}(\lambda)$ of $\mathcal{C}_{2}(\lambda)$.

######  Lemma 2.4 ([5]).

Let $\lambda$ be a partition. Then $\lambda$ is odd if and only if
$c_{2}^{(k)}(\lambda)\leq 1$ for all $k\geq 0$.

It may be decided from the $k$-data $\mathcal{D}^{(k)}_{2}(\lambda)$ whether
$\lambda$ is odd. The case $k=1$ of the following result appeared in [3, Lemma
4.1] and also in [1, Lemma 6].

######  Theorem 2.5.

Let $\lambda\vdash n$, and let $k\geq 0$ be fixed. Consider
$\mathcal{Q}_{2}^{(k)}(\lambda)=\big{(}\lambda^{(k)}_{i}\big{)}$. Then
$\lambda$ is odd if and only if the following conditions are all fulfilled:

  * $(i)$

$c_{2}^{(j)}(\lambda)\leq 1$ for all $j<k$.

  * $(ii)$

The partitions $\lambda^{(k)}_{i}$, $0\leq i\leq 2^{k}-1$, are all odd.

  * $(iii)$

The numbers $\big{|}\lambda^{(k)}_{i}\big{|}$, $0\leq i\leq 2^{k}-1$, are
pairwise $2$-disjoint.

In this case $\sum\limits_{i\geq
0}\big{|}\lambda^{(k)}_{i}\big{|}=\big{\lfloor}\frac{n}{2^{k}}\big{\rfloor}$.

###### Proof.

This is proved by induction on $k\geq 0$, using Remark 2.2 and Lemma 2.4. ∎

We illustrate the result above by giving an example.

######  Example 2.6.

Let $n=15$ and take $\lambda=\big{(}5,4,2^{2},1^{2}\big{)}\vdash 15$. To
decide whether $\lambda$ is odd, we choose $k=2$ and compute the 2-data
$\mathcal{D}_{2}^{(2)}(\lambda)$. The 2-core is $C_{2}(\lambda)=(1)$, giving
$\mathcal{C}_{2}^{(0)}(\lambda)=((1))$. Furthermore, the 2-quotient is
$Q_{2}(\lambda)=\big{(}\big{(}2^{2},1^{2}\big{)},(1)\big{)}$, and computing
the 2-cores $C_{2}\big{(}\big{(}2^{2},1^{2}\big{)}\big{)}=(0)$,
$C_{2}((1))=(1)$, we obtain the next row:
$\mathcal{C}_{2}^{(1)}(\lambda)=((0),(1))$. The 2-quotients are
$Q_{2}\big{(}\big{(}2^{2},1^{2}\big{)}\big{)}$
$=\big{(}\big{(}1^{2}\big{)},(1)\big{)}$, $Q_{2}((1))=((0),(0))$; hence the
final row of the 2-data table is obtained as
$\mathcal{Q}_{2}^{(2)}(\lambda)=\big{(}\big{(}1^{2}\big{)},(1),(0),(0)\big{)}$.

We visualize $\mathcal{D}_{2}^{(2)}(\lambda)$ like this:

$\displaystyle\begin{array}[]{@{}r c c c c c c c c c c
c}\mathcal{C}_{2}^{(0)}(\lambda)\colon&&&&(1)\\\\[5.0pt]
\mathcal{C}_{2}^{(1)}(\lambda)\colon&&(0)&&&&(1)\\\\[5.0pt]
\mathcal{Q}_{2}^{(2)}(\lambda)\colon&(1^{2})&&(1)&&(0)&&(0)\end{array}$

Theorem 2.5 shows that $\lambda$ is odd and thus it contains a unique odd
4-hook. Again using the theorem, it is clear that removing this 4-hook
corresponds to the second partition $(1)$ in $\mathcal{Q}_{2}^{(2)}$ being
replaced by $(0)$. Thus, removing the corresponding 4-hook of $\lambda$ we
obtain the odd partition $\mu=\big{(}3,2^{3},1^{2}\big{)}\vdash 11$ with the
property that $\mathcal{D}_{2}^{(2)}(\lambda)$ and
$\mathcal{D}_{2}^{(2)}(\mu)$ differ only in their final row.

######  Remark 2.7.

Using the construction of partitions from their 2-cores and 2-quotients
already mentioned, the criterion above can be applied to construct all odd
partitions of $n$ with a specific $k$th row in the 2-quotient tower. For this,
let $n,k\in\mathbf{N}$, and take any sequence of odd partitions $\nu_{i}$,
$0\leq i\leq 2^{k}-1$, such that the numbers $|\nu_{i}|$ are pairwise
2-disjoint, and $\sum\limits_{i\geq
0}|\nu_{i}|=\big{\lfloor}\frac{n}{2^{k}}\big{\rfloor}$. Then there are exactly
$\prod\limits_{{m<k}\atop{2^{m}\subseteq_{2}n}}2^{m}$ odd partitions $\lambda$
of $n$ with $\mathcal{Q}_{2}^{(k)}(\lambda)=(\nu_{i})$, obtained by choosing
one 2-core in row $m$ of the $k$-data table to be $(1)$, for each $m<k$ such
that $2^{m}\subseteq_{2}n$.

The following easy consequence of Theorem 2.5 will be used repeatedly.

######  Lemma 2.8.

Let $2^{t}$ be the largest binary digit of $n$. A partition $\lambda$ of $n$
is odd if and only if $\lambda$ contains a unique $2^{t}$-hook and the
partition obtained from $\lambda$ by removing this $2^{t}$-hook is an odd
partition of $n-2^{t}$.

##  3 Surjectivity and regularity

The aim of this section is to study the images of the maps $f_{k}^{n}$ for all
$n$, $k$ such that $2^{k}\leq n$. For this purpose we introduce the concept of
$d$-good partitions (see Definition 3.1 below). This will allow us to prove
Theorem 3.5 (describing the images) and thus Theorem A (describing exactly
when $f_{k}^{n}$ is surjective) and to show that the maps $f_{k}^{n}$ are
always regular on their image (see Corollary 3.8).

######  Definition 3.1.

Let $d\geq 0$. We call an odd partition $\lambda$ $d$-good, if

  * (i)

$|\lambda|\equiv 2^{d}-1\mod 2^{d+1}$.

  * (ii)

$C_{2^{d}}(\lambda)$ is a hook partition.

Let us remark that condition (i) may be reformulated as

  * (i∗)

$\nu_{2}(|\lambda|+1)=d$.

In particular, if $\lambda$ is $d$-good, then $|\lambda|$ is odd if and only
if $d>0$.

The relevance of $d$-good partitions in our context is illuminated by the
following reformulation of [1, Theorem 2]:

######  Lemma 3.2.

Let $\lambda\vdash_{o}n$. Let $d=\nu_{2}(n+1)$. Then $e(\lambda,1)\neq 0$ if
and only if $\lambda$ is $d$-good. In this case, $e(\lambda,1)=1$ if $d=0$,
and $e(\lambda,1)=2$ if $d>0$.

######  Lemma 3.3.

Let $\lambda$ be an odd partition, and let $d\geq 0$. Then the following hold.

  * $(1)$

For $d\leq 2$, $\lambda$ is $d$-good if and only if $|\lambda|\equiv
2^{d}-1\mod 2^{d+1}$.

  * $(2)$

If $\lambda$ is $d$-good, then $C_{2^{d}}(\lambda)$ is a partition of
$2^{d}-1$.

###### Proof.

If the odd partition $\lambda$ is $d$-good, then
$|\lambda|=\big{(}2^{d}-1\big{)}+m$ where the binary digits of $m$ are at
least $2^{d+1}$. The hooks of $\lambda$ corresponding to the binary digits of
$m$ may be decomposed into $2^{d}$-hooks and thus do not contribute to
$C_{2^{d}}(\lambda)$. Thus $|C_{2^{d}}(\lambda)|=2^{d}-1$. This shows (2). For
$d=0,1,2$ we have $|C_{2^{d}}(\lambda)|=0$, $1$ and $3$, respectively. Since
all partitions of 0, 1 and 3 are hook partitions, (1) follows. ∎

######  Definition 3.4.

If $2^{k}\leq n$, we define
$d(n,k)=\nu_{2}\big{(}\lfloor\frac{n}{2^{k}}\rfloor\big{)}$. Thus $d(n,k)$ is
the smallest integer $d\geq 0$ satisfying the condition
$2^{k+d}\subseteq_{2}n$. In particular, $d(n,k)=0$ if and only if
$2^{k}\subseteq_{2}n$. Moreover, we may write
$\big{\lfloor}\frac{n}{2^{k}}\big{\rfloor}=2^{d(n,k)}+m(n,k)$ where
$2^{d(n,k)+1}\,|\,m(n,k)$.

As mentioned in the introduction, the results in [3] show that $f_{k}^{n}$ is
a surjective $(2^{k}$-to-$1)$-map whenever $2^{k}\subseteq_{2}n$, i.e.,
$d(n,k)=0$. In the spirit of [1, Theorem 2], we now give a characterization of
the image of the map $f_{k}^{n}$ for all $n$, $k$ such that $2^{k}<n$.

######  Theorem 3.5.

Let $n\in\mathbb{N}$, $k\in\mathbb{N}_{0}$ be such that $2^{k}<n$. Let
$\lambda\vdash_{o}n-2^{k}$. Then $e\big{(}\lambda,2^{k}\big{)}\neq 0$ if and
only if there exists a $d(n,k)$-good partition in the $k$th row of
$\mathcal{Q}_{2}(\lambda)$. In this case, $e\big{(}\lambda,2^{k}\big{)}=2^{k}$
if $d(n,k)=0$, and $e\big{(}\lambda,2^{k}\big{)}=2$ if $d(n,k)>0$.

###### Proof.

If $k=0$ then the statement follows from Lemma 3.2. Hence assume that $k\geq
1$. Let $d=d(n,k)$. By assumption
$\big{\lfloor}\frac{n}{2^{k}}\big{\rfloor}=2^{d}+m$, where the binary digits
of $m$ are at least $2^{d+1}$. Thus
$\big{\lfloor}\frac{n-2^{k}}{2^{k}}\big{\rfloor}=\big{(}2^{d}-1\big{)}+m$.

Suppose first that $e\big{(}\lambda,2^{k}\big{)}\neq 0$ and that
$\mu\vdash_{\mathrm{o}}n$ satisfies $f_{k}(\mu)=\lambda$. From Remark 2.1 and
Lemma 2.3 we get that there exists an $i\in\\{0,1,\ldots,2^{k}-1\\}$ such that
$f_{0}\big{(}\mu^{(k)}_{i}\big{)}=\lambda^{(k)}_{i}$. Since $\mu^{(k)}_{i}$
and $\lambda^{(k)}_{i}$ are odd, we get
$e\big{(}\lambda^{(k)}_{i},1\big{)}\neq 0$. We have that
$\big{|}\lambda^{(k)}_{i}\big{|}$ and $\big{|}\mu^{(k)}_{i}\big{|}$ are both
2-disjoint with $m_{1}:=\sum\limits_{j\neq
i}\big{|}\lambda^{(k)}_{j}\big{|}=\sum\limits_{j\neq
i}\big{|}\mu^{(k)}_{j}\big{|}\subseteq_{2}\big{\lfloor}\frac{n-2^{k}}{2^{k}}\big{\rfloor}$,
by Theorem 2.5. Since
$m_{1}\subseteq_{2}\big{\lfloor}\frac{n-2^{k}}{2^{k}}\big{\rfloor}$ and
$m_{1}\subseteq_{2}\big{\lfloor}\frac{n}{2^{k}}\big{\rfloor}$, we get
$m_{1}\subseteq_{2}m$. Thus $\big{|}\lambda^{(k)}_{i}\big{|}=(2^{d}-1)+m_{2}$
and $\big{|}\mu^{(k)}_{i}\big{|}=2^{d}+m_{2}$, where
$m_{2}=m-m_{1}\subseteq_{2}m$. In particular
$\nu_{2}\big{(}\big{|}\lambda^{(k)}_{i}\big{|}+1\big{)}=\nu_{2}\big{(}\big{|}\mu^{(k)}_{i}\big{|}\big{)}=d$.
Then Lemma 3.2 shows that $\lambda^{(k)}_{i}$ is $d$-good.

Conversely, if $\lambda^{(k)}_{i}$ is a $d$-good partition for some
$i\in\big{\\{}0,1,\ldots,2^{k}-1\big{\\}}$, then there exists a
$\mu^{*}\vdash_{o}\big{|}\lambda^{(k)}_{i}\big{|}+1$ such that
$f_{0}(\mu^{*})=\lambda^{(k)}_{i}$, by Lemma 3.2. We let $\mu$ be the
partition where the $k$-data $\mathcal{D}^{(k)}_{2}(\mu)$ and
$\mathcal{D}^{(k)}_{2}(\lambda)$ coincide, except that
$\mu^{(k)}_{i}=\mu^{*}$. Since $\lambda$ is odd and $\lambda^{(k)}_{i}$ is
$d$-good, we know that
$\big{|}\lambda^{(k)}_{i}\big{|}=\big{(}2^{d}-1\big{)}+m^{\prime}$ where
$m^{\prime}\subseteq_{2}m$, and
$\big{|}\lambda^{(k)}_{j}\big{|}\subseteq_{2}m-m^{\prime}$ for all $j\neq i$.
Hence $|\mu^{*}|=\big{|}\lambda^{(k)}_{i}\big{|}+1=2^{d}+m^{\prime}$ is
2-disjoint from all $\big{|}\lambda^{(k)}_{j}\big{|}$, $j\neq i$. Thus $\mu$
is an odd partition of $n$ by Theorem 2.5, and $f_{k}(\mu)=\lambda$ by Lemma
2.3 and Remark 2.1.

We conclude that
$e\big{(}\lambda,2^{k}\big{)}=\sum\limits_{\lambda^{(k)}_{i}d-{\rm
good}}e\big{(}\lambda^{(k)}_{i},1\big{)}$. If $d=0$ then
$\big{\lfloor}\frac{n-2^{k}}{2^{k}}\big{\rfloor}$ is even. This implies that
all $\lambda^{(k)}_{i}$ are of even cardinality and thus $d$-good. Thus
$e\big{(}\lambda^{(k)}_{i},1\big{)}=1$ for all $i$, and we get
$e\big{(}\lambda,2^{k}\big{)}=2^{k}$. If $d>0$ there is exactly one
$\lambda^{(k)}_{i}$ in $\mathcal{Q}^{(k)}_{2}(\lambda)$ of odd cardinality.
Only this $\lambda^{(k)}_{i}$ may be $d$-good and then
$e\big{(}\lambda,2^{k}\big{)}=e\big{(}\lambda^{(k)}_{i},1\big{)}=2$. Otherwise
$e\big{(}\lambda,2^{k}\big{)}=0$. ∎

######  Corollary 3.6.

Let $n\in\mathbb{N}$, $k\in\mathbb{N}_{0}$ be such that $2^{k}<n$, and let
$d=\nu_{2}\big{(}\big{\lfloor}\frac{n}{2^{k}}\big{\rfloor}\big{)}$. Let
$\lambda\vdash_{o}n-2^{k}$. Then $e\big{(}\lambda,2^{k}\big{)}\neq 0$ if and
only if there exists a partition $\lambda^{(k)}_{i}$ in the $k$th row of
$\mathcal{Q}_{2}(\lambda)$ such that $\big{|}\lambda^{(k)}_{i}\big{|}\equiv
2^{d}-1\mod 2^{d+1}$, and $C_{2^{d}}\big{(}\lambda^{(k)}_{i}\big{)}$ is a hook
partition. In this case, $e\big{(}\lambda,2^{k}\big{)}=2^{k}$ if $d=0$, and
$e\big{(}\lambda,2^{k}\big{)}=2$ if $d>0$.

We are now ready to prove Theorem A. In fact, this is a consequence of Theorem
3.5 and it is stated here as the following corollary.

######  Corollary 3.7 (Theorem A).

Let $n\in\mathbb{N}$, $k\in\mathbb{N}_{0}$ be such that $2^{k}<n$.

  * •

If $k=0$ then $f^{n}_{k}$ is surjective if and only if $d(n,k)\leq 2$.

  * •

If $k>0$ then $f^{n}_{k}$ is surjective if and only if $d(n,k)\leq 1$.

###### Proof.

By Theorem 3.5, $f^{n}_{k}$ is surjective if and only if for all
$\lambda\vdash_{o}n-2^{k}$ we have that the $k$th row of
$\mathcal{Q}_{2}(\lambda)$ contains a $d(n,k)$-good partition
$\lambda^{(k)}_{j}$. By Theorem 2.5 and Definition 3.4, for any
$\lambda\vdash_{o}n-2^{k}$ we have $\sum\limits_{j\geq
0}\big{|}\lambda^{(k)}_{j}\big{|}=\big{\lfloor}\frac{n-2^{k}}{2^{k}}\big{\rfloor}=\big{(}2^{d(n,k)}-1\big{)}+m(n,k)$.

If $k=0$ then $\mathcal{Q}^{(0)}_{2}(\lambda)$ contains only
$\lambda=\lambda^{(0)}_{0}$. Hence $f^{n}_{0}$ is surjective if and only all
odd partitions of $n-1$ are $d(n,0)$-good. By Lemma 3.3(1), the latter
condition holds when $d=d(n,0)\leq 2$. On the other hand, if $d=\nu_{2}(n)>2$,
then $\lambda=(n-5,2,2)$ is an odd partition of $n-1$ by Theorem 2.5, but
$C_{8}(\lambda)=(3,2,2)$ is not a hook, and hence $C_{2^{d}}(\lambda)$ is not
a hook. So $\lambda$ is not $d$-good, and thus $f^{n}_{0}$ is not surjective.

Now assume $k\geq 1$. Then $\mathcal{Q}^{(k)}_{2}(\lambda)$ contains at least
two odd partitions. If $d(n,k)\geq 2$ then any $d(n,k)$-good partition $\mu$
satisfies $3\subseteq_{2}2^{d(n,k)}-1\subseteq_{2}|\mu|$. Write
$\big{\lfloor}\frac{n-2^{k}}{2^{k}}\big{\rfloor}=1+m_{1}$ where $m_{1}$ is
even. Applying Remark 2.7, take any $\lambda\vdash_{o}n-2^{k}$ such that
$\big{|}\lambda^{(k)}_{0}\big{|}=1$ and $\lambda^{(k)}_{1}$ is an odd
partition with $\big{|}\lambda^{(k)}_{1}\big{|}=m_{1}$. Then no partition in
$\mathcal{Q}^{(k)}_{2}(\lambda)$ is $d(n,k)$-good. Thus $f^{n}_{k}$ is not
surjective. On the other hand, if $d(n,k)=0$ then $2^{k}\subseteq_{2}n$ and
$f^{n}_{k}$ is surjective [3, Proposition 4.5]. If $d(n,k)=1$ then
$\big{\lfloor}\frac{n-2^{k}}{2^{k}}\big{\rfloor}=1+m(n,k)$, where $4\mid
m(n,k)$. Thus any $\mathcal{Q}^{(k)}_{2}(\lambda)$ contains a partition with
odd cardinality; this partition is 1-good, by Lemma 3.3. Again $f^{n}_{k}$ is
surjective. ∎

It is an immediate consequence of Theorem 3.5 that $f_{k}^{n}$ is regular on
its image for all relevant choices of $n,k$ such that $2^{k}<n$. We have:

######  Corollary 3.8.

Let $n\in\mathbb{N}$, $k\in\mathbb{N}_{0}$ be such that $2^{k}<n$; set
$d=\nu_{2}\big{(}\big{\lfloor}\frac{n}{2^{k}}\big{\rfloor}\big{)}$. Let
$\lambda\vdash_{o}n-2^{k}$. Then

$\displaystyle e\big{(}\lambda,2^{k}\big{)}=\begin{cases}2^{k}&\text{if
}d=0;\\\ 2&\text{if }d>0,\ \text{and the}\ k\text{th row of}\
\mathcal{Q}_{2}(\lambda)\text{ contains a }d\text{-good partition};\\\
0&\text{otherwise. }\\\ \end{cases}$

######  Example 3.9.

For an illustration, we consider odd extensions of odd partitions by a
$4$-hook, i.e., we take $k=2$ above. For $n>2^{2}$ we first compute
$d(n,k)=\nu_{2}\big{(}\big{\lfloor}\frac{n}{2^{k}}\big{\rfloor}\big{)}$, and
then consider odd partitions of $n-4$ and their 4-extensions. For $n=6$,
$d(6,2)=0$. Thus $e((2),4)=4$. The odd 4-extensions of $(2)$ are $(6)$,
$\big{(}3^{2}\big{)}$, $\big{(}2^{2},1^{2}\big{)}$, $\big{(}2,1^{4}\big{)}$.
For $n=10$, $d(10,2)=1$. In this case, $e(\lambda,4)=2$ for all odd partitions
$\lambda$ of 6. For instance, the odd 4-extensions of $(6)$ are $(10)$ and
$(6,3,1)$. For $n=19$, $d(19,2)=2$. Example 2.6 shows that for
$\lambda=\big{(}5,4,2^{2},1^{2}\big{)}\vdash_{o}15$ there is no 2-good
partition in $\mathcal{Q}_{2}^{(2)}(\lambda)$, hence $e(\lambda,4)=0$.

##  4 Deciding commutativity of the maps $\boldsymbol{f_{k}}$ and
$\boldsymbol{f_{\ell}}$

Let $n\in\mathbb{N}$, and suppose that $0\leq k<\ell$ satisfy
$2^{k}+2^{\ell}\leq n$. As stated in the introduction, we want to complete the
discussion of the commutativity of the maps $f_{k}$ and $f_{\ell}$. Since the
relevant $n$ will always be apparent for the maps $f^{n}_{k}$ in this section,
we just write $f_{k}$.

We write $(n;k,\ell)\in\mathcal{T}$ if for all $\lambda\vdash_{o}n$ we have
$f_{k}f_{\ell}(\lambda)=f_{\ell}f_{k}(\lambda)$. Otherwise we write
$(n;k,\ell)\in\mathcal{F}$.

In this section we will prove Theorem B, which may be reformulated as follows.

######  Theorem 4.1.

Let $n=2^{t}+m$ where $0\leq m<2^{t}$. Suppose that $k$, $\ell$ satisfy $0\leq
k<\ell$ and $2^{k}+2^{\ell}\leq n$. Then with the exception of $(6;0,1)$

$\displaystyle\text{$(n;k,\ell)\in\mathcal{F}$ if and only if $\ell<t$ and
$2^{k}\leq m$}.$

The proof of Theorem 4.1 is based on a series of lemmas. The first lemmas
concern two extreme cases, where $f_{k}$ and $f_{\ell}$ commute.

In the case $\ell=t$ we have the following result as a reformulation of [3,
Proposition 4.3].

######  Lemma 4.2.

Let $n=2^{t}+m$ with $0\leq m<2^{t}$. If $2^{k}\leq m$, then
$(n;k,t)\in\mathcal{T}$.

It is also known that in the case where $n$ is a power of $2$, the maps
$f_{k}$ and $f_{\ell}$ commute [3, Remark 4.4], and we include a short proof
here.

######  Lemma 4.3.

If $n=2^{t}$ then $(n;k,\ell)\in\mathcal{T}$ for all $k$, $\ell$.

###### Proof.

If $0\leq b\leq a$ are integers then the binomial coefficient $\binom{a}{b}$
is odd if and only if $b\subseteq_{2}a$, by Lucas’ theorem. The odd partitions
of $2^{t}$ are exactly the hook partitions $\big{(}2^{t}-b,1^{b}\big{)}$,
$0\leq b\leq 2^{t}-1$, of degree $\binom{2^{t}-1}{b}$. Hence for
$k\in\\{0,1,\ldots,t-1\\}$ we have

$\displaystyle
f_{k}(\lambda)=\begin{cases}\big{(}2^{t}-b-2^{k},1^{b}\big{)}&\text{if
}2^{k}\not\subseteq_{2}b,\\\ \big{(}2^{t}-b,1^{b-2^{k}}\big{)}&\text{if
}2^{k}\subseteq_{2}b.\end{cases}$

It follows that for any $k,\ell<t$ and odd partition $\lambda$ of $2^{t}$, we
have $f_{\ell}f_{k}(\lambda)=f_{k}f_{\ell}(\lambda)$. ∎

######  Lemma 4.4.

Let $n=2^{t}+m$ with $0\leq m<2^{t}$. Suppose that $k$, $\ell$ satisfy $0\leq
k<\ell$ and $2^{k}+2^{\ell}\leq n$. If $m<2^{k}$ then
$(n;k,\ell)\in\mathcal{T}$.

###### Proof.

We use induction on $k\geq 0$. For $k=0$ we have $m=0$ and the claim follows
from Lemma 4.3. Suppose that $k\geq 1$ and that the claim has been proved up
to $k-1$. Let $\lambda\vdash_{o}n$. Odd hooks of length $2^{k}$ and $2^{\ell}$
in $\lambda$ correspond to odd hooks of length $2^{k-1}$ and $2^{\ell-1}$ in
the $2$-quotient $Q_{2}(\lambda)=(\lambda_{0},\lambda_{1})$ of $\lambda$. From
Theorem 2.5 we deduce that $|\lambda_{0}|$ and $|\lambda_{1}|$ are 2-disjoint
binary subsums of $\big{\lfloor}\frac{n}{2}\big{\rfloor}$, so one of them
contains $2^{t-1}$, say $|\lambda_{0}|$; then
$|\lambda_{1}|\leq\big{\lfloor}\frac{m}{2}\big{\rfloor}<2^{k-1}<2^{\ell-1}$.
Thus the odd $2^{k-1}$-hook in $Q_{2}(\lambda)$ has to be in $\lambda_{0}$.
Therefore

$\displaystyle Q_{2}(f_{k}(\lambda))=(f_{k-1}(\lambda_{0}),\lambda_{1}).$

Applying $f_{\ell}$, the odd $2^{\ell-1}$-hook cannot be in $\lambda_{1}$,
hence

$\displaystyle
Q_{2}(f_{\ell}f_{k}(\lambda))=(f_{\ell-1}f_{k-1}(\lambda_{0}),\lambda_{1})).$

In particular, we know that $|\lambda_{0}|\geq 2^{\ell-1}+2^{k-1}$. Also
$|\lambda_{0}|+|\lambda_{1}|=\big{\lfloor}\frac{n}{2}\big{\rfloor}=2^{t-1}+\big{\lfloor}\frac{m}{2}\big{\rfloor}$.
We have already seen that $2^{t-1}$ is the largest binary digit of
$|\lambda_{0}|$; furthermore $|\lambda_{0}|-2^{t-1}$ is a binary subsum of
$\big{\lfloor}\frac{m}{2}\big{\rfloor}<2^{k-1}$. We may therefore apply the
inductive hypothesis to $\lambda_{0}$ to get
$f_{\ell-1}f_{k-1}(\lambda_{0})=f_{k-1}f_{\ell-1}(\lambda_{0})$. This implies
that $Q_{2}(f_{k}f_{\ell}(\lambda))=Q_{2}(f_{\ell}f_{k}(\lambda))$ and thus
$f_{k}f_{\ell}(\lambda)=f_{\ell}f_{k}(\lambda)$. ∎

Lemmas 4.2 and 4.4 show that the only if part of the theorem is true. We now
turn to the if part. We start by proving the statement for $k=0$ and use this
as part of an inductive argument.

######  Lemma 4.5.

Let $n=2^{t}+m$ with $0<m<2^{t}$. If $0<\ell<t$ then
$(n;0,\ell)\in\mathcal{F}$, with the exception of $(6;0,1)$.

###### Proof.

The result is easily checked for $n\leq 8$, which includes the exception
$(6;0,1)$. So we assume that $t\geq 3$.

Case 1: $2^{\ell}<m$. Then $m\geq 3$, since $\ell>0$. Consider the partition
$\lambda=\big{(}m,m,1^{a}\big{)}\vdash n$ where $a=n-2m=2^{t}-m$. The
(1,1)-hook length of $\lambda$ is $2^{t}+1$. The (2,1)-hook length of
$\lambda$ is $2^{t}$. Removing the (2,1)-hook hook we get the odd partition
$(m)$, so $\lambda$ is odd, by Lemma 2.8. We claim that

$\displaystyle f_{0}(\lambda)=\big{(}m,m,1^{a-1}\big{)}.$

Indeed we cannot have $f_{0}(\lambda)=\big{(}m,m-1,1^{a}\big{)}$ because this
partition does not have a hook of length $2^{t}$, and thus it is not odd. Now

$\displaystyle
f_{\ell}(f_{0}(\lambda))=f_{\ell}\big{(}m,m,1^{a-1}\big{)}=\big{(}m,m-2^{\ell},1^{a-1}\big{)}$

since $\big{(}m,m,1^{a-1-2^{\ell}}\big{)}$ and
$\big{(}m-1,m-2^{\ell}+1,1^{a-1}\big{)}$ both do not have a hook of length
$2^{t}$ and thus are not odd (again by Lemma 2.8).

On the other hand,

$\displaystyle
f_{\ell}(\lambda)=\big{(}m-1,m-\big{(}2^{\ell}-1\big{)},1^{a}\big{)}.$

Indeed, the other candidates for $f_{\ell}(\lambda)$, which are
$\big{(}m,m-2^{\ell},1^{a}\big{)}$ and $\big{(}m,m,1^{a-2^{\ell}}\big{)}$, do
not have hooks of length $2^{t}$. Then

$\displaystyle
f_{0}(f_{\ell}(\lambda))=f_{0}\big{(}m-1,m-\big{(}2^{\ell}-1\big{)},1^{a}\big{)}=\big{(}m-1,m-2^{\ell},1^{a}\big{)}.$

This follows (again) by observing that all the other partitions of
$n-2^{\ell}-1$ obtained from
$\big{(}m-1,m-\big{(}2^{\ell}-1\big{)},1^{a}\big{)}$ by removing a node do not
have hooks of length $2^{t}$. Thus $f_{0}(f_{\ell}(\lambda))\neq
f_{\ell}(f_{0}(\lambda))$.

Case 2: $m<2^{\ell}$. Consider the partition
$\lambda=\big{(}n-2^{\ell},m+1,1^{a}\big{)}$, where $a=2^{\ell}-(m+1)$. Note
that $n-2^{\ell}\geq m+1$ since $\ell<t$ by assumption, and that $a\geq 0$.
The (1,1)-hook length of $\lambda$ is $n-m=2^{t}$. Removing this hook we get
the odd partition $(m)$, so $\lambda$ is odd. The (2,1)-hook length of
$\lambda$ is $2^{\ell}$. Now

$\displaystyle f_{0}(\lambda)=\big{(}n-2^{\ell},m,1^{a}\big{)}$

since the other candidates do not have hooks of length $2^{t}$. Then

$\displaystyle
f_{\ell}(f_{0}(\lambda))=f_{\ell}\big{(}n-2^{\ell},m,1^{a}\big{)}=\mu,$

where $\mu$ is obtained from $f_{0}(\lambda)$ by removing a $2^{\ell}$-hook in
the first row. (There are only hooks of length $<2^{\ell}$ in the other rows.)
In fact, $\mu=\big{(}n-2^{\ell+1},m,1^{a}\big{)}$ since $n-2^{\ell+1}\geq
n-2^{t}=m$. Thus $f_{\ell}(f_{0}(\lambda))$ has at least 2 parts. On the other
hand

$\displaystyle f_{\ell}(\lambda)=\big{(}n-2^{\ell}\big{)}$

since this odd partition is obtained from the odd partition $\lambda$ by
removing a $2^{\ell}$-hook (the one in $(2,1)$). It follows that

$\displaystyle f_{0}(f_{\ell}(\lambda))=\big{(}n-2^{\ell}-1\big{)}$

and again $f_{0}(f_{\ell}(\lambda))\neq f_{\ell}(f_{0}(\lambda))$.

Case 3: $m=2^{\ell}$. Then $n=2^{t}+2^{\ell}$. If $\ell\geq 2$ then choose
$\lambda=\big{(}2^{t},2^{\ell}-1,1\big{)}$. The $(1,2)$-hook length of
$\lambda$ is $2^{t}$; thus $\lambda$ is an odd partition since removing this
$2^{t}$-hook gives an odd partition $\big{(}2^{\ell}-2,1,1\big{)}$ of
$2^{\ell}$. We have $f_{0}(\lambda)=\big{(}2^{t},2^{\ell}-2,1\big{)}$ since
the other candidates are not odd. Then

$\displaystyle
f_{\ell}(f_{0}(\lambda))=\big{(}2^{t}-2^{\ell},2^{\ell}-2,1\big{)}.$

The $(2,1)$-hook length of $\lambda$ is $2^{\ell}$, so
$f_{\ell}(\lambda)=\big{(}2^{t}\big{)}$ and

$\displaystyle f_{0}(f_{\ell}(\lambda))=\big{(}2^{t}-1\big{)},$

showing $f_{0}(f_{\ell}(\lambda))\neq f_{\ell}(f_{0}(\lambda))$.

On the other hand, if $\ell=1$ then choose
$\lambda=\big{(}2^{t}-2,2,2\big{)}\vdash_{o}2^{t}+2=n$. Since $t\geq 3$, it is
now easy to show that $f_{1}(f_{0}(\lambda))=\big{(}2^{t}-4,2,1\big{)}$. On
the other hand we see that $f_{0}(f_{1}(\lambda))$ is a hook partition of
$2^{t}-1=n-3$ and therefore is not equal to $f_{1}(f_{0}(\lambda))$. ∎

######  Lemma 4.6.

If $(n;k,\ell)\in\mathcal{F}$ then also $(2n;k+1,\ell+1)\in\mathcal{F}$ and
$(2n+1;k+1,\ell+1)\in\mathcal{F}$.

###### Proof.

Let the odd partition $\mu$ of $n$ satisfy $f_{k}f_{\ell}(\mu)\neq
f_{\ell}f_{k}(\mu)$. Let $\lambda$ be a partition of $2n$ or $2n+1$ having
2-quotient $Q_{2}(\lambda)=(\mu,(0))$. Then $\lambda$ is odd, by Theorem 2.5.
We have

$\displaystyle
Q_{2}(f_{k+1}f_{\ell+1}(\lambda))=(f_{k}f_{\ell}(\mu),(0))\neq(f_{\ell}f_{k}(\mu),(0))=Q_{2}(f_{\ell+1}f_{k+1}(\lambda)),$

so that $f_{k+1}f_{\ell+1}(\lambda)\neq f_{\ell+1}f_{k+1}(\lambda)$. ∎

We are now ready to conclude this section with the proof of Theorem B.

###### Proof of Theorem 4.1.

The only if part follows from Lemmas 4.2 and 4.4. To prove the if part we use
induction on $k\geq 0$. If $k=0$, then the statement follows from Lemma 4.5.
Let $k>1$ and suppose that the assertion is true up to and including $k-1$. To
show that $(n;k,\ell)\in\mathcal{F}$ it suffices to prove
$(\lfloor\frac{n}{2}\rfloor;k-1,\ell-1)\in\mathcal{F}$, by Lemma 4.6. We are
assuming $n=2^{t}+m$, $0\leq m<2^{t}$, $0\leq k<\ell\leq t$ and
$2^{k}+2^{\ell}\leq n$. This implies
$\big{\lfloor}\frac{n}{2}\big{\rfloor}=2^{t-1}+\big{\lfloor}\frac{m}{2}\big{\rfloor}$,
$0\leq\big{\lfloor}\frac{m}{2}\big{\rfloor}<2^{t-1}$ and
$2^{k-1}+2^{\ell-1}\leq\big{\lfloor}\frac{n}{2}\big{\rfloor}$. We may apply
the inductive hypothesis to get
$\big{(}\big{\lfloor}\frac{n}{2}\big{\rfloor};k-1,\ell-1\big{)}\in\mathcal{F}$,
and then $(n;k,\ell)\in\mathcal{F}$ except when
$\big{(}\big{\lfloor}\frac{n}{2}\big{\rfloor};k-1,\ell-1\big{)}=(6;0,1)$. In
that case we are considering (12;1,2) or (13;1,2) which are both in
$\mathcal{F}$, by direct computation (consider for example
$(6,4,2)\vdash_{o}12$ and $(6,4,3)\vdash_{o}13$, respectively). ∎

### Acknowledgements

We thank Gabriel Navarro for providing some useful tables which helped us find
the patterns explained by the results of this paper. The second author is
grateful to Trinity Hall, University of Cambridge, for funding his research.
Thanks also go to the referee for helpful comments on a previous version of
the article.

## References

  * [1] Ayyer A., Prasad A., Spallone S., Odd partitions in Young’s lattice, Sém. Lothar. Combin. 75 (2015), Art. B75g, 13 pages, arXiv:1601.01776.
  * [2] Giannelli E., Kleshchev A., Navarro G., Tiep P.H., Restriction of odd degree characters and natural correspondences, Int. Math. Res. Not., to appear, arXiv:1601.04423.
  * [3] Isaacs I.M., Navarro G., Olsson J.B., Tiep P.H., Character restrictions and multiplicities in symmetric groups, J. Algebra 478 (2017), 271–282.
  * [4] James G., Kerber A., The representation theory of the symmetric group, Encyclopedia of Mathematics and its Applications, Vol. 16, Addison-Wesley Publishing Co., Reading, Mass., 1981.
  * [5] Macdonald I.G., On the degrees of the irreducible representations of symmetric groups, Bull. London Math. Soc. 3 (1971), 189–192.
  * [6] Olsson J.B., Combinatorics and representations of finite groups, Vorlesungen aus dem Fachbereich Mathematik der Universität Essen, Heft 20, 1994, available at http://www.math.ku.dk/~olsson/manus/comb_rep_all.pdf.


* [255] Marieke Mur, Mirjam Meys, Jerzy Bodurka, Rainer Goebel, Peter A Bandettini, and Nikolaus Kriegeskorte. Human object-similarity judgments reflect and transcend the primate-IT object representation. _Frontiers in psychology,_ 4:128, 2013.
  * [256] Radoslaw M Cichy, Nikolaus Kriegeskorte, Kamila M Jozwik, Jasper JF van den Bosch, and Ian Charest. The spatiotemporal neural dynamics underlying perceived similarity for real-world objects. _Neuroimage,_ 194:12–24, 2019.
  * [257] James R Golden, Kedarnath P Vilankar, Michael CK Wu, and David J Field. Conjectures regarding the nonlinear geometry of visual neurons. _Vision research,_ 120:74–92, 2016.
  * [258] SueYeon Chung, Daniel D Lee, and Haim Sompolinsky. Classification and geometry of general perceptual manifolds. _Physical Review X,_ 8(3):031003, 2018.
  * [259] Hansem Sohn, Devika Narain, Nicholas Meirhaeghe, and Mehrdad Jazayeri. Bayesian computation through cortical latent dynamics. _Neuron,_ 103(5):934–947, 2019.
  * [260] Olivier J Hénaff, Robbe LT Goris, and Eero P Simoncelli. Perceptual straightening of natural videos. _Nature neuroscience,_ 22(6):984–991, 2019.
  * [261] MohammadMehdi Kafashan, Anna W Jaffe, Selmaan N Chettih, Ramon Nogueira, Iñigo Arandia-Romero, Christopher D Harvey, Rubén Moreno-Bote, and Jan Drugowitsch. Scaling of sensory information in large neural populations shows signatures of information-limiting correlations. _Nature Communications,_ 12(1):1–16, 2021.
  * [262] Ramon Nogueira, Nicole E Peltier, Akiyuki Anzai, Gregory C DeAngelis, Julio Martínez-Trujill, and Rubén Moreno-Bote. The effects of population tuning and trial-by-trial variability on information encoding and behavior. _Journal of Neuroscience,_ 40(5):1066–1083, 2020.
  * [263] Gouki Okazawa, Christina E Hatch, Allan Mancoo, Christian K Machens, and Roozbeh Kiani. The geometry of the representation of decision variable and stimulus difficulty in the parietal cortex. _bioRxiv_ , 2021.
  * [264] Shun-ichi Amari. Information geometry. _Japanese Journal of Mathematics_ , 1–48, 2000.
  * [265] Anil Bhattacharyya. On a measure of divergence between two statistical populations defined by their probability distributions. _Bull. Calcutta Math. Soc._ 35:99–109, 1943
  * [266] Don Johnson, and Sinan Sinanovic. Symmetrizing the Kullback-Leibler distance. _IEEE Transactions on Information Theory_ , 2001.

## Supplementary Information

###  4.1 Rotations, translations, and reflections conserve geometry and
information content

Assume that $p(\mathbf{r}|\theta)$, the probability of response pattern
${\mathbf{r}}$ given stimulus $\theta$, follows an isotropic multivariate
Gaussian distribution $\mathcal{N}(\bf{f}(\theta),I_{n})$. For a particular
response $\mathbf{r}$, the likelihood function can be written as

$\mathcal{L}(\theta|\mathbf{r})\propto
e^{-\frac{1}{2}\|\mathbf{r}-\bf{f}(\theta)\|^{2}}$ (18)

Applying an arbitrary rotation using a rotation matrix $\mathbf{A}$ would lead
to a new representation $\mathcal{N}(\mathbf{A}\cdot\bf{f}(\theta),I_{n})$
with the same geometry. Because the noise is isotropic the rotation of the
tuning curves rigidly rotates the entire joint probability function
$p(\theta,\mathbf{r})$. For each original response pattern $\mathbf{r}$
generated by the original code, there is a new response pattern
$\mathbf{r^{\prime}}=\mathbf{A}\cdot\mathbf{r}$ that is equally probable. The
likelihood function for the transformed code and response pattern can be
expressed as

$\mathcal{L}(\theta|\mathbf{r}^{\prime})\propto
e^{-\frac{1}{2}\|\mathbf{A}\cdot(\mathbf{r}-\bf{f}(\theta))\|^{2}}=e^{-\frac{1}{2}\|\mathbf{r}-\bf{f}(\theta)\|^{2}}$
(19)

For every value of $\theta$, thus,
$\mathcal{L}(\theta|\mathbf{r}^{\prime})=\mathcal{L}(\theta|\mathbf{r})$. This
conservation of the likelihood function holds not just for rotations, but also
for translations and reflections: all transformations that conserve the
geometry. Because the prior is also the same, these transformations have no
influence on the performance of a Bayesian ideal observer. Choosing a
different code with the same geometry yields the same estimation error
distribution.

Figure 5: Encoding 1-d location with grid cells. (a) A grid module for a
single scale. (b) Two grid modules for two scales with a scale ratio of 2. (c)
Two grid modules with a scale ratio of 3. (d) Three grid modules with scales
[2, 5.5, 8.3].

###  4.2 An additional coding example: grid cells

How can a population of neurons encode a position in an extended (and
potentially unbounded) 1D or 2D space, such as the location of a rat in a
corridor or 2D environment? For a finite environment, monotonic tuning
functions could encode the coordinates. However, what if the rat keeps running
once each neuron’s activity has either saturated or gone to zero? It seems
useful for neurons encoding the position in a large space to come on and off
periodically. This principle is familiar from the way we write numbers (with
ten digits advanced in a cycle) and from the Fourier transform. In both cases,
we need multiple scales (digits of different significance and Fourier
components of different frequencies) to disambiguate the code.

An encoding of 2D space is provided by grid cells. Originally discovered in
rats [29, 30] and later in humans and other animal species [184, 185, 186,
187, 188, 189, 190], grid cells provide a code of 2D position. A given grid
cell will fire at locations clustered around the vertices of a triangular
lattice over the environment. Different cells will fire around the vertices of
triangular lattices of different phases and spatial scales [30, 191],
collectively forming a multiscale code for location. To illustrate the
representational geometry resulting from multiscale, periodic tuning
functions, let us consider, for simplicity, a code for a 1D environment, such
as a corridor (Fig. 5).

We assume that each grid cell has a periodic tuning function, and different
grid cells can have different phases and spatial scales. A population of grid
cells with the same scale, where each neuron encodes a different phase,
results in a cyclic representational geometry [192] (Fig. 5a). As the animal
keeps moving in the same direction, the population activity pattern traverses
this cyclic path over and over. We can add another population, a second
module, to encode a second scale. Two modules for different scales already
result in a complex geometry that forms a cycle whose period is the least
common multiple of the periods of the two modules (Fig. 5b, c). The
representational geometry is sensitive to the scale ratio between the
different modules (compare Fig. 5 b and c, which have scale ratios of 2 and 3,
respectively). For these codes, the representational distance is a smooth
(albeit not a monotonic) function of the distance between two locations.

Fig. 5d shows an example with 3 grid modules. The representation is highly
distorted, and the representational distance is no longer a smooth function of
the distance between two locations. Interestingly, most pairs of the points
are well separated (Fig. 5d, right), unlike the previous cases. This is due to
the greater spread (higher entropy) of the response patterns (Fig. 5d, third
column), which increases the informational capacity of the code. As
neighboring points along the manifold (corresponding to neighboring locations
along the corridor) become more distinct in the representation, the FI
increases. As pairs of points, in general, become more distinct, the MI
increases. Maximizing the encoding capacity, thus will require spreading out
the manifold to increase its length, corresponding to the total FI and, in
case we desire high MI, to avoid its coming close to itself. There is likely a
tradeoff between maximizing some notion of encoding capacity (e.g., FI or MI)
and maintaining a simple (e.g., near monotonic) relationship between
environmental and representational distance.


# A Vision-Guided Multi-Robot Cooperation Framework for Learning-by-
Demonstration and Task Reproduction

Bidan Huang, Menglong Ye, Su-Lin Lee, Guang-Zhong Yang B. Huang, M. Ye, S.-L.
Lee and G.-Z. Yang are with the Hamlyn Centre for Robotic Surgery, Imperial
College London, SW7 2AZ, London, UK (e-mail: b.huang@imperial.ac.uk). The
project is supported by the EPSRC (EP/L020688/1).

###### Abstract

This paper presents a vision-based learning-by-demonstration approach to
enable robots to learn and complete a manipulation task cooperatively. With
this method, a vision system is involved in both the task demonstration and
reproduction stages. An expert first demonstrates how to use tools to perform
a task, while the tool motion is observed using a vision system. The
demonstrations are then encoded using a statistical model to generate a
reference motion trajectory. Equipped with the same tools and the learned
model, the robot is guided by vision to reproduce the task. The task
performance was evaluated in terms of both accuracy and speed. However, simply
increasing the robot’s speed could decrease the reproduction accuracy. To this
end, a dual-rate Kalman filter is employed to compensate for latency between
the robot and vision system. More importantly, the sampling rates of the
reference trajectory and the robot speed are optimised adaptively according to
the learned motion model. We demonstrate the effectiveness of our approach by
performing two tasks: a trajectory reproduction task and a bimanual sewing
task. We show that using our vision-based approach, the robots can conduct
effective learning by demonstrations and perform accurate and fast task
reproduction. The proposed approach is generalisable to other manipulation
tasks, where bimanual or multi-robot cooperation is required.

##  I Introduction

Learning-by-demonstration has become one of the standard approaches for
performing a range of tasks [1]. Increasing numbers of industrial robots now
have built-in “record-and-replay” modes, which allow users to physically move
their joints and then replay the same trajectory. This function enables
workers without robotic expertise to program a robot trajectory and complete
simple tasks; however, there are limitations to this kinesthetic teaching
approach. Firstly, it requires a robot with this “record-and-replay” mode
during the demonstration, which might not always be available. Secondly, the
user needs to kinesthetically move the robot arm to demonstrate a task: for a
small and delicate manipulation task such as sewing or surgical subtasks [2,
3, 4], this can be difficult as these tasks are usually demonstrated via
teleoperation or use pre-programmed trajectories. Thirdly, it is difficult to
accurately demonstrate a bimanual task by kinesthetic teaching; typically a
user must use both hands to move a robot arm. In industry, bimanual tasks are
usually demonstrated by moving the robot with the control panel or with pre-
programmed commands [5].

Figure 1: An overview of our system setup. Two Kuka robots hold the motorised
needle drivers to perform bimanual sewing on a target under the observation of
a stereo camera. An addition robot is used to hold the target (mandrel and
fabric) for manipulation.

Transferring skills from users to robots can be performed naturally via
vision-based demonstration [6]. With a vision system observing the user
motion, no robot is required during the demonstration and the user can perform
the task naturally with their own hands, e.g. manipulating an object or using
a tool. This type of vision-based demonstration learning facilitates the
performance of delicate tasks with high precision. Furthermore, bimanual or
multiple tool/robot cooperation tasks require accurate tool/robot co-
localisation. At the demonstration stage, a vision system can provide the
configurations of multiple tools/robots and objects in a unified coordinate
frame, via tracking their poses in real-time. This facilitates finding the
correlation between tools and objects in learning. At the task reproduction
stage, the poses of the robots and objects can be similarly retrieved via
tracking. This then enables the robots to perform accurate task execution via
visual servoing, which mitigates the need for accurate hand-eye calibration.
Despite the benefits of a vision-based learning-by-demonstration approach, two
main problems should be addressed: how to map the user’s behaviour to the
robot behaviour and how to achieve good task performance, e.g. high accuracy
and speed.

To tackle these problems, this paper proposes a vision-based learning by
demonstration approach for task manipulation, which allows easy manipulation
learning and accurate task execution. To learn a manipulation task, we adopt
an “object centric” approach [7, 8]. Instead of modelling the human hand or
robot motion, we model the motion of manipulated objects during demonstration
learning. With this, we do not need to consider a direct mapping from the
human joints motion to the robot joints motion, whilst we focus only on
reproducing the same object behaviour. In our approach, we assume that the
target object is rigidly attached to the robot end-effector and hence its
behaviour can be reproduced by transforming its motion to the end-effector.

As the demonstrations are observed and learned in the coordinate frame of the
vision system, programming the robot to perform the tasks under the same frame
with visual guidance is an intuitive solution. We use the “look-and-move”
visual servoing approach [9] to instruct the robot to complete the learned
task. For reliable visual servoing, vision-based detection and tracking has
been studied in the past decade for industrial [10, 11] and surgical robotic
setups [12, 13]. Accurate pose estimation of tools during manipulation plays
an important role in guiding robots to follow desired trajectories for its
provided advantage of compensating hand-eye calibration or kinesthetic errors
[14].

A robust vision-based method is proposed to guide the robot to perform
reliable task execution. A dual-rate Kalman filter [15] is applied to deal
with the low sampling rate and the latency between vision and kinesthetic
data. With accurate estimation of the visual feedback, robot can perform the
task accurately. To further overcome the slow visual feedback and increase
robot speed, we optimise the reference trajectory based current task context,
and adjust the robot speed accordingly.

Here, learning-by-demonstration is combined with visual servoing. The main
contributions of this work are:

  1. 1.

An easy-to-use method is introduced for users to demonstrate delicate bimanual
tasks by simply using their own hands without the need for using robots. This
allows for more natural demonstrations without requiring the user to directly
adapt complex motions for the robots.

  2. 2.

A robust visual servoing approach is proposed, which uses low-cost cameras to
guide robots for task reproduction. The approach enables robot motion speed to
be adjusted according to the task context, such that the speed of reproduction
can be increased without sacrificing the reproduction accuracy.

The rest of the paper is organized as follows. Section II describes our
system, the hardware, and the software components. Section III shows the
experiments conducted using this system and presents the results. Discussion
and conclusions are presented in Section IV.

Figure 2: An illustration of vision sensing latency and noise.

##  II System Architecture

The main motivation of this work is to present a system to generically solve
the problem of a robot learning a bimanual task via vision. We focus on tasks
of using tools and manipulating objects. Here, a “tool” is a device that can
be either held by a user or attached rigidly to a robot end-effector, whilst
an “object” can be manipulated in 3D space by tools. The user and robot use
the same tools and objects during demonstration and reproduction. Learning is
done using the object centric principle. After modelling the manipulated
objects’ or tools’ motion, these models are applied to robots to reproduce the
behaviour. Section A details our method of tracking tools and the process of
user demonstration with a given bimanual task. Section B shows the encoding of
the demonstrated motion and the optimization of the reference trajectory
according to task contexts. Section C explains our implementation of the
Kalman filter for the vision feedback control.

Without loss of generality, our approach is framed in the context of a
bimanual sewing task for the purpose of personalized stent graft manufacturing
[16]. This task is challenging as it involves complicated movements and fine
manipulation of the needle and requires high reproduction accuracy, e.g.
pierce in and out of fabric within a 2mm slot, passing the needle from one
robot to another. In our previous work, the robot was programmed by
kinesthetically teaching one arm to complete the entire stitch cycle. This
system has an uncertainty: the needle could change its pose when detached from
the needle driver by unpredictable fabric tension. This introduces a large
amount of uncertainty in the sewing process. The bimanual sewing system
presented in this paper is for the purpose of reducing the uncertainty and
increase the robustness of the process.

In this work, the vision system is extended to learn bimanual sewing. There
are three different devices: a mandrel111A mandrel is a hollow cylinder for
bounding the fabric and the stents together. It is design according to
patients’ anatomy. Details about it please see our previous work [16] for
fixing the fabric, a needle for sewing, and needle drivers for gripping the
needle. The proposed platform is shown in Figure 1: one robot arm holds the
mandrel and fabric while two robot arms with needle drivers manipulate a
surgical needle. The needle drivers control the needle to pierce the fabric
and complete a stitch. We focus on learning the trajectories of the needle
drivers (tools) and the needle (object).

###  II-A Vision-based Bimanual Task Demonstration

For both the task demonstration and reproduction, the same stereo vision
system was used to avoid any error caused by different vision systems. The
vision system plays two roles: motion observation during task demonstration
and robot servoing for task reproduction.

####  II-A1 Detection-Tracking for Continuous Tool Pose Estimation

Our stereo vision system was composed of two cameras calibrated using the
OpenCV 222http://opencv.org/ library. To facilitate pose estimation, several
fiducial bar-code markers were placed on each tool such that the entire motion
trajectory can be observed. As shown in Figure 5, a hollow pentagonal adapter
was designed and attached rigidly to each needle driver, with each adapter
face being a 1cm2 square on which the bar-code maker was placed. Each marker
had a different pattern so that the rotation of the needle driver can be
computed correctly. The detected marker pose was transformed to the needle
driver pose according to the pentagonal prism’s size. The rotational axis of
the pentagonal prism was designed to align with the needle driver. For the
mandrel, an octagonal prism was designed.

To allow for continuous 3D pose estimation of the tools in consecutive frames,
a vision-based marker detection-tracking approach was therefore adopted. The
marker detector was an off-the-shelf approach provided in ArUco [17] which
processes only the current frame, whilst the tracker was a forward-backward
(FB) approach [18] based on optical flow that considers temporal information.

More specifically, the tracker was applied on a marker whose previous pose was
available but whose current pose could not be detected in the frame. With
previous locations $\left\\{q_{i}\right\\}_{i=1}^{n}$ of the corner points on
a marker, optical flow was applied to forwards track from the previous to the
current frame, to obtain the corner estimates
$\left\\{q^{+}_{i}\right\\}_{i=1}^{n}$. Then, optical flow was applied again
to backwards track these corner estimates from the current to the previous
frame, thus obtaining additional estimates as
$\left\\{q^{-}_{i}\right\\}_{i=1}^{n}$. With these, to determine if a corner
point estimate $q^{+}_{i}$ was valid, its FB error, defined as Euclidean
distance $\epsilon\left(q_{i},1^{-}_{i}\right)$, was compared to $\tau$. The
value of $\tau$ was chosen as 1 pixel in this work, which helps filtering out
the outlier corner estimates. The remaining estimates were treated as inliers,
which were then used to estimate the 6 d.o.f pose of the marker via
perspective-n-points [19]. Note that, when $\tau$ was larger than $5$ pixels,
the output was empirically ignored from the tracker, as it indicated the
corner estimates were not reliable.

Therefore, this tracker was able to provide pose estimation when the detector
failed, thus providing continuous pose estimation. This detection-tracking
strategy was applied to every marker that was attached to the tools.

####  II-A2 Data Acquisition

In the bimanual sewing task, the user held a needle driver in each hand to
manipulate the needle. The fabric was bound to the mandrel to constrain its
deformation. The stereo camera was placed at the top of the mandrel so that
the sewing behaviour, i.e. the needle drivers and needle motion, could be
observed. The 6 d.o.f poses of both needle drivers were tracked and recorded
by the stereo camera.

The motion of finishing one stitch is treated as one cycle; the steps for this
are shown in Figure 3. The two needle drivers are referred to as “Needle
Driver A” and “Needle Driver B”. A single tip surgical curved needle was used.
The sharp end of the needle is the “needle tip” and the blunt end is the
“needle end”. While Needle Driver B was static in our previous work, here both
needle drivers were mobile. At the beginning of the task, the curved needle
was gripped by Needle Driver A at its end (Step a). Needle Driver A approached
the fabric and pierced it with the needle (Step b). When the needle tip
pierced out of the fabric, Needle Driver B approached the needle tip and
gripped it (Step c). Needle Driver A then opened and released the needle (Step
d). After that, Needle Driver B pulled the needle out of the fabric (Step e).
When the needle was completely out, Needle Driver A approached it and re-
gripped it from the needle end (Step f). Needle Driver B then opened and
released the needle (Step f). Finally Needle Driver A pulled the thread to
tighten the stitch and returned to its initial pose. Hence a full stitch
cycle, i.e. one stitch, was completed.

Figure 3: The key steps involved in one stitch cycle (a-g), of which at the
end the needle is passed back to needle driver A for next cycle.

In this bimanual task, the two needle drivers hold the needle alternately.
When a needle driver was holding the needle, the needle trajectory was
recorded in the world frame; when the needle driver was not holding the
needle, its own trajectory was recorded in the local frame of the needle. The
needle trajectory was not directly observed but was computed as a rigid
transformation of the observed needle driver position; this transformation was
estimated by a needle pose detection method [16]. This detection was performed
at the beginning of each stitching cycle.

###  II-B Task Learning

After multiple user demonstrations, e.g. five times for bimanual sewing, the
different demonstration data were temporally aligned using Dynamic Time
Warping [20]. According to the needle drivers’ open and closed status and
their attachment to the needle, one stitch cycle was segmented into four
motion primitives, as shown in Table I. Each segment was a primitive movement
and encoded by a Gaussian Mixture Model (GMM) for its ability to encode non-
linear data [21, 22].

TABLE I: Motion primitives of stitching Motion Primitives |  Steps in Fig. 3 |  Needle Driver A status |  Needle Driver B status |  Needle status
---|---|---|---|---
1. |  a, b |  Closed |  Open |  With A
2. |  c |  Closed |  Closed |  With A
3. |  d,e |  Open |  Closed |  With B
4. |  f |  Closed |  Closed |  With B
5. |  g |  Closed |  Open |  With A

In this study, each motion primitive was modelled by a 7 d.o.f GMM $\Omega$,
which encoded the time stamp $t$ and the 6 d.o.f pose
$h=\\{x,y,z,\alpha,\beta,\theta\\}$. The probability that a given data point
${t,h}$ belonged to $\Omega$ was computed as:

$p\left(t,h\mid\Omega\right)=\sum_{m=1}^{M}\pi_{m}p_{m}\left(t,h\mid\mu_{m},\Sigma_{m}\right)$
(1)

where $\pi_{m}$, $p_{m}$, ${\mu}_{m}$, and ${\Sigma}_{m}$ were the prior, the
corresponding conditional probability density, mean, and covariance of the
$m$-th Gaussian component, respectively. The number of Gaussian components was
$M$ and was determined by a five-fold cross validation.

For each time step, Gaussian Mixture Regression (GMR) was used to generate
trajectory points and hence form a reference trajectory to the demonstrated
task.

#### Trajectory Optimisation for Task Contexts

The learned reference trajectory must be further optimised to allow the robots
to complete the task with high speed and high accuracy. In a manipulation
task, there are two task contexts: “end point driven” and “contact driven”. In
the “end point driven” context, the robot can be moved freely as long as it
reaches the final destination. In the “contact driven” context, the robot is
in contact with the environment and its motion is constrained; in this
context, the robot movement needs to follow a particular trajectory.

In the case of bimanual sewing, the approaching motion of the needle to the
fabric is end point driven, while the piercing in and out motion is contact
driven. For the approaching motion, the needle does not need to follow the
reference trajectory from point to point and hence the trajectory can be down-
sampled to allow the robot to move faster. For the piercing motion, the needle
needs to follow the reference trajectory and accuracy is the first priority.
Hence the robot must slow down and follow the reference trajectory carefully.

The task context was distinguished by the variance between different
demonstrations. Figure 8 shows the variance of Motion Primitive 1 across the
five demonstrations. The variance of the trajectories is large at the
beginning, i.e. it is end point driven, and low at the end, i.e. it is contact
driven, hence the beginning of this trajectory was down-sampled. The
correlation of the variance and the ratio to the demonstration speed $r$ was:

$r=\begin{cases}0.5,&\text{if }v_{t}>0.01m\text{ $or$ }v_{r}>0.2rad\\\
1&\text{if }0.01m>v_{t}>0.005m\text{ $or$ }\\\ &0.2rad>v_{r}>0.1rad\\\
2&\text{if }v_{t}<0.005m\text{ $or$ }v_{r}<0.1rad\\\ \end{cases}$

where $v_{t}$ and $v_{r}$ are the variance of the translation and the
rotation, respectively. This is decided according to our system and the task
requirement (Session III).

###  II-C Task Reproduction with Vision Guidance

To allow the robot to reproduce the task, the tool was first motorized and
installed on the robot. Figure 4 shows the motorized needle driver designed
for the robot. The visual markers were placed at the same location, such that
the task can be reproduced by following the reference trajectory.

####  II-C1 Visual Servoing

For robots with considerable kinematic errors, online vision feedback is
essential. For high accuracy industrial robots, online vision feedback is also
important, especially for multi-robot operation or bimanual tasks where all
robots need to be registered to the same frame. For delicate tasks such as
sewing or surgical tasks, it is very time-consuming to achieve a good off-line
calibration with adequate precision. Therefore our robots were controlled with
online vision feedback, such that the error of the robot reaching the target
is independent of the registration and the robot kinematic accuracy [9].

Figure 4: The motorised needle driver. It is designed for attachment to the
Kuka robot and has a DC motor that opens and closes the needle driver.

All the robots in this study were registered to the camera frame. This was
performed by computing the transformation between the robot ($r$) base and the
camera ($c$) base $\tensor[^{c}]{H}{{}_{r}}$ according to the pose of the
markers relative to the robot base frame and the camera frame333We denote
$\tensor[^{b}]{H}{{}_{a}}$ as the homogeneous matrix of the pose of an object
a in the frame of an object b.. In the bimanual sewing task, the needle driver
($d$) pose was first computed in the robot end effector ($EE$) frame
$\tensor[^{EE}]{x}{{}_{d}}$ according to the motorized needle driver design.
The needle driver was then moved into the view of the camera and a series of
end effector poses $\tensor[^{r}]x{{}_{EE}}$, as well as the needle driver
pose under the camera frame $\tensor[^{c}]x{{}_{d}}$, was recorded at each
time step. The transformation between the robot base frame and the camera
frame $\tensor[^{c}]{H}{{}_{r}}$ was then estimated by the absolute
orientation algorithm 444From Matlab:
https://uk.mathworks.com/matlabcentral/fileexchange/22422-absolute-
orientation.

The “look-and-move” position based servoing approach was used for our tasks.
In this method, the problem is modelled as moving the robot to reduce the
error between the current pose of the manipulated object and its target pose.
Taking Motion Primitive 1 in our sewing task as an example, the aim is to move
the needle to approach the mandrel and pierce the fabric. The reference
trajectory was expressed in the frame of the mandrel ($m$) as a series of
target poses of the needle ($n$) $\tensor[^{m}]x{{}_{n^{*}}}$. These needle
target poses were transferred to needle driver ($d$) target poses
$\tensor[^{m}]x{{}_{d^{*}}}$ :

${\tensor[^{m}]x{{}_{d}^{*}}=\tensor[^{m}]x{{}_{n^{*}}}\cdot\left(\tensor[^{d}]H{{}_{n}}\right)^{-1}}$
(2)

where $\tensor[^{d}]H{{}_{n}}$ is the needle and needle driver relative pose
detected over the task as explained in Section II-A.

As the needle driver and the mandrel was directly observed by the camera
($c$), the error of the pose was computed as

${\tensor[^{d}]x{{}_{d^{*}}}=\left(\tensor[^{c}]x{{}_{d}}\right)^{-1}\cdot\tensor[^{c}]x{{}_{m}}\cdot\tensor[^{m}]x{{}_{d^{*}}}}$
(3)

This then can be transformed to the error of the robot end effector and hence
generate commands to move towards the target pose.

Figure 5: A 3D printed rigid body for validation, with the Optotrak Certus
infrared markers and the bar-code markers affixed.

####  II-C2 Kalman Filter

In relative terms, camera latency is inevitable compared to other sensors used
in robotic control. To balance these, a dual rate Kalman filter was used to
estimate the vision feedback $\tensor[^{c}]x{{}_{d}}$.

In a standard Kalman filter, the state of a system at a time $t$ predicted
from the previous state at time $t-1$ is modelled as

$\bm{x}_{t}=\bm{F}_{t}\bm{x}_{t-1}+\bm{B}_{t}\bm{u}_{t}+\bm{w}_{t}$ (4)

where $\bm{x}_{t}$, $\bm{F}_{t}$, $\bm{B}_{t}$, $\bm{u}_{t}$, and $\bm{w}_{t}$
are the state vector, the state transition matrix, the control input matrix,
the control inputs, and the process noise term at time $t$, respectively. The
noise term is assumed to be a zero mean Gaussian with covariance matrix
$\bm{Q}_{t}$.

The system measurement is modelled as

$\bm{z}_{t}=\bm{H}_{t}\bm{x}_{t}+\bm{v}_{t}$ (5)

where $\bm{z}_{t}$, $\bm{H}_{t}$, and $\bm{v}_{t}$ are the measurement vector,
the transformation matrix which transforms the system state space to the
measurement space, and the measurement noise at time $t$, respectively. The
noise term is also assumed to be a zero mean Gaussian with covariance matrix
$\bm{R}_{t}$.

In our sewing task, both the system state and measurement are the 6 d.o.f
needle driver pose. Hence the transformation matrix $\bm{H}_{t}$ is an
identity matrix. Similarly, the state transition matrix $\bm{F}_{t}$ is also
an identity matrix. These are omitted in the following equations. The Kalman
filter has two phases: prediction and measurement update. Applying the Kalman
filter to our task, the equations of the prediction phase are

$\hat{\bm{x}}_{t\mid{t-1}}=\hat{\bm{x}}_{{t-1}\mid{t-1}}+\bm{B}_{t}\bm{u}_{t}$
(6)

$\bm{P}_{t\mid{t-1}}=\bm{P}_{{t-1}\mid{t-1}}+\bm{Q}_{t}$ (7)

Figure 6: Trajectory reproduction results demonstrating the relative errors
for each trial compared to the reference.

The symbol $\hat{\bullet}_{a\mid{b}}$ denotes the estimated value of $\bullet$
at time $a$ given its previous value at time $b$. The term $\bm{P}$ is the
covariance matrix of the estimation of the state vector, and the term $\bm{Q}$
is, as mentioned above, the process noise covariance. At every time step, the
robot was commanded to move to the next pose $x^{*}$ in the reference
trajectory. Hence in our task the equation 6 is equivalent to

$\hat{\bm{x}}_{t\mid{t-1}}=\bm{x}^{*}_{t-1}$ (8)

For the update phase, the current system state was estimated as

$\hat{\bm{x}}_{t\mid{t}}=\hat{\bm{x}}_{t\mid{t-1}}+\bm{K}_{t}\left(\hat{\bm{z}}_{t}-\hat{\bm{x}}_{t\mid{t-1}}\right)$
(9) $\bm{P}_{t\mid{t}}=\bm{P}_{{t}\mid{t-1}}+\bm{K}_{t}\bm{P}_{t\mid{t-1}}$
(10)

where the Kalman gain $\bm{K}_{t}$ is

$\bm{K}_{t}=\bm{P}_{t\mid{t-1}}\left(\bm{P}_{t\mid{t-1}}+\bm{R}_{t}\right)^{-1}$
(11)

To compensate for camera latency, an additional prediction step was applied to
estimate the current state measurement $\hat{\bm{z}}_{t}$. In our sewing task,
this was the needle driver pose in the camera frame
$\tensor[^{c}]{\hat{x}}{{}_{d}}$:

${\hat{\bm{z}}_{t\mid{t_{L}}}=\tensor[^{c}]{\hat{x}}{{}_{d}}\left(t\right)=\tensor[^{c}]{{x}}{{}_{d}}\left({t}\right)\cdot\left(\tensor[^{r}]x{{}_{d}}\left({t_{L}}\right)\right)^{-1}\cdot\tensor[^{r}]x{{}_{d}}\left(t\right)}$
(12)

where $t_{L}$ is the current time step minus the camera latency.

The filter runs in the same rate of the camera frame rate to provide visual
feedback. Between the periods of two frames, robot follows the reference
trajectory according to the last estimated needle driver pose.

The evaluation of the system state noise covariance $\bm{Q}$ and the
measurement noise covariance $\bm{R}$ is explained in the next section.

##  III Experiments and results

###  III-A Estimation of Variables

The covariance of the system noise $\bm{Q}$ and the measurement noise $\bm{R}$
were considered to be constant during the whole task. The system noise was
dominated by the precision of the robot. According to the accuracy estimation
[23], the covariance matrix $\bm{Q}$ was estimated to be a diagonal matrix:

$\small\bm{Q}=diag\left(0.25,0.25,0.25,0.02,0.02,0.02\right)$ (13)

An experiment was performed to evaluate the measurement error covariance
matrix $\bm{R}$. In this experiment, the robot was moved slowly in a straight
line while keeping the needle driver in the view of the camera. The needle
driver pose was recorded from the camera and a straight line was fitted to the
pose trajectory. According to the deviation of the detected poses to the
straight line, the covariance matrix $\bm{R}$ was estimated to be a diagonal
matrix:

$\small\bm{R}=diag\left(0.35,0.35,0.35,0.04,0.04,0.04\right)$ (14)

In both matrix, the first three values are in the unit of $mm$ and the later
three are in $rad$.

###  III-B Task 1: Trajectory Reproduction

Figure 7: Velocity vs. task duration for task reproduction.

Detailed experiments with an Optotrak Certus (NDI) tracking device were
conducted to validate the accuracy provided by visual servoing. A 3D printed
rigid body (Fig. 5) was designed to host both the infrared markers and our
barcode markers. During each experimental trial, a ground truth trajectory was
collected from user demonstrations and this trajectory was recorded both by
the tracking device (infrared markers) and the camera (barcode markers). The
robot was first commanded to execute the trajectory in two settings, with and
without visual servoing, for which all actual movements were recorded by the
tracking device. The actual movements resulted from both settings were then
compared to the ground truth trajectory via dynamic time warping [24] and
rotation and translation differences between the trajectories were calculated.
In this work, five trials were conducted and the results are provided in Fig.
6 and Table II. It can been seen from Fig. 6 that the trajectories executed
without visual servoing deviate significantly from the ground truth for on
average 5.23 mm per point while the trajectories generated with visual
servoing are more accurate. Table II presents the quantitative results, from
which we can observe that visual servoing improves the accuracy over no visual
servoing guidance, with average error ranges of $\left[0.80,1.11\right]$ mm in
translation and $\left[0.01,0.02\right]$ degrees in rotation. Note that the
errors presented in our visual servoing framework can result from marker
tracking and pose estimation. This trajectory was reproduced with the same
speed of demonstration. To analyse the correlation between the speed and the
accuracy, another three sets of similar experiments were conducted with
different speeds, i.e. reference trajectory sampling rate. The result is shown
in Fig. 7. According to this result, we determine the sampling rate of the
reference trajectory in different task contexts.

TABLE II: Trajectory reproduction accuracy Error | Translation (mm) | Rotation (degree)
---|---|---
No Visual Servoing | 5.23 | 0.07
Trial 1 | 0.82 | 0.01
Trial 2 | 0.81 | 0.01
Trial 3 | 1.11 | 0.02
Trial 4 | 0.92 | 0.01
Trial 5 | 0.80 | 0.01
Figure 8: The tip trajectories of the needle drivers from the user
demonstrations and the generalized reference trajectories of all motion
primitives in the bimanual sewing task. Top row: Multiple user demonstrations.
Each colour represents one demonstration trajectory. (a)-(c): The grey
cylinder represents the mandrel. (d)-(e): The red half cylinder represents the
needle. Bottom row: 2D representation of the GMM of each primitive. Deep blue
lines are the generalized trajectories, while light blue areas denote the
corresponding variances. Figure 9: Key frames of one stitch cycle of bimanual
sewing. The top row shows the view from the top camera, which is used for
visual servoing. The detected markers are labeled by red boxes and their IDs.
The bottom row shows the corresponding views from the side. The letters at the
bottom show their corresponding steps in Fig. 3

###  III-C Task 2: Robot Bimanual Sewing Task

A bimanual sewing task was conducted to demonstrate the validity of this
approach for teaching a robot a bimanual task. As mentioned in Section II-A,
in each stitching cycle the robot manipulates a needle to pierce the fabric
bound on a mandrel. The stitching slots on the mandrel were of size of 2 by 10
$mm$ and hence high accuracy of reproducing the demonstrated trajectory was
crucial. A small error in reproduction will cause an entire task failure.

Figure 1 shows the setup of our bimanual sewing system with three 7 d.o.f
robots used (KUKA LWR 4+ and iiwa). To register all three robots to the camera
frame, hand eye calibration was conducted as detailed in Section II-A. The
robot system was shown five cycles of stitching. A stereo vision system
composed of two Logitech C920 web cameras retrieves 640$\times 480$-pixel
videos for needle driver motion tracking and pose estimation. The trajectories
of the needle drivers were registered to their corresponding sewing slots. The
sewing slot positions were computed via the detected mandrel pose according to
the mandrel design. A curved surgical curved needle of diameter 8$mm$ was
used. Before each cycle (Figure 3), the needle pose was detected to correct
the needle driver trajectory so that the needle can be delivered to pierce
accurately. At Step g, the robot was required to pull the thread and tighten
the stitch; this pulling motion was performed manually according to an
estimation of the remaining length of the thread. The mandrel is fixed on the
third robot, of which the trajectory is programmed by mandrel’s design, i.e.
the sewing slots’ locations.

Based on the variance of the demonstrations, the speed of the robot was varied
in order to satisfy both the speed and accuracy requirements. Figure 8 shows
the variance among all the demonstrations for Motion Primitives 1-5. It can be
seen in Motion Primitive 1, both from the trajectories plot in the top row and
the 2D representations of the GMM in the bottom row, that the motion of Needle
Driver A has large variance at the beginning and small variance toward the
end. Among all the demonstrations, the needle driver follows the same path to
pierce the needle. For Motion Primitives 2 and 3, the motion of Needle Driver
B has small variance when it is piercing out the needle (end of Motion
Primitive 2 and beginning of Motion Primitive 3) and otherwise large variance.
In Motion Primitive 4, Needle Driver A goes to pick up the needle from Needle
Driver B. Hence the variance of Needle Driver A is large and then small in the
frame of the needle. For the same reason, the variance of Needle Driver A in
Motion Primitive 5 is small and then large. The motion of Needle Driver B in
Motion Primitive 1, 4, 5 and of Needle Driver A in Motion Primitives 2 and 3
are small and hence considered to be static.

Overall, the variance is large for more than half of the task duration and
therefore speeding up these parts of the task effectively speeds up the entire
stitching cycle. The accuracy required in this task is preserved as the robot
slows down at the moments requiring high precision. The robots performed 8
stitches with 6 success and 2 failures: one caused by the entanglement of the
thread and one by the robot joint limit. The overall success rate of this
bimanual sewing task is $75\%$.

##  IV Discussion and Conclusion

In this paper, we present a vision based programming by demonstration approach
that enables users to demonstrate bimanual tasks with tools in their own hands
and enables the robot to execute the task with varying speed according to the
task context. Using the same tools and markers among the task demonstration
and reproduction, the demonstrated motion skills can be transferred to robots
seamlessly. The visually controlled execution allows for execution of a
bimanual task with both robot position calibration and handeye calibration
provided to a rough accuracy.

When the task context is end point driven, i.e. large deviation from the
reference trajectory can be tolerated, the robot speeds up to reach the end
point. When the task context is contact driven, i.e. the difference between
the reproduced trajectory and the reference trajectory has to be minimised,
the robot slows down. A trajectory reproduction task is conducted to evaluate
the accuracy of visual servoing. The results show that as the robot speeds up,
the accuracy decrease dramatically. According to this evaluation, we find out
the speed corresponding to the required accuracy for the next sewing task.

In a typical bimanual sewing task, delicate skill and bimanual cooperations
are involved such as piercing the needle in and out of the fabric via a narrow
sewing slot and passing the needle over from one needle driver to another. The
entire stitch cycle is segmented into five motion primitives and for each, the
variance is computed. The robot reproduces the task with varying speed
according to the corresponding variance. We show that the robot system can
reproduce the original hand stitch with the guidance of a low cost camera.
Despite the low sampling rate and camera latency, the robot system is able to
reproduce the task with user level speed.

It should be noted that the thread has yet to be considered in our bimanual
sewing task. To pull the thread, the needle currently moves out of the view of
the stereo camera and we program this manually. The thread length was
estimated as the original length minus the length of each stitch. The thread
was pulled by moving the needle driver out of the the camera view from the
bottom right and brought back into view from the top right. In this way, the
thread formed a loop under its weight and a blanket stitch was formed, which
is required for sewing a stent graft. To handle the thread properly in the
future, a vision method is required to detect and localize the thread. As the
current camera’s view limits the working space for visual servoing, a multiple
view system will be investigated.

Compared to kinesthetic teaching and tele-operation, our approach allows for a
simple method to program multiple robots and benefits from allowing users to
demonstrate bimanual tasks more naturally, demonstrations without the robot
system, and the use of robots without built-in “record-and-replay” modes to be
programmed. We show that by combining the programming by demonstration and
visual feedback control approaches, a low cost camera can supervise a multi-
robot system to accomplish delicate tasks such as hand sewing. The proposed
method in this paper can be applied to many other tasks involving cooperative
control of multiple robots.

## References

  * [1] Z. Pan, J. Polden, N. Larkin, S. Van Duin, and J. Norrish, “Recent progress on programming methods for industrial robots,” _Robotics and Computer-Integrated Manufacturing_ , vol. 28, no. 2, pp. 87–94, 2012.
  * [2] T. Osa, K. Harada, N. Sugita, and M. Mitsuishi, “Trajectory planning under different initial conditions for surgical task automation by learning from demonstration,” in _Robotics and Automation (ICRA), 2014 IEEE International Conference on_. IEEE, 2014, pp. 6507–6513.
  * [3] A. Murali, S. Sen, B. Kehoe, A. Garg, S. McFarland, S. Patil, W. D. Boyd, S. Lim, P. Abbeel, and K. Goldberg, “Learning by observation for surgical subtasks: Multilateral cutting of 3d viscoelastic and 2d orthotropic tissue phantoms,” in _2015 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2015, pp. 1202–1209.
  * [4] S. Sen, A. Garg, D. V. Gealy, S. McKinley, Y. Jen, and K. Goldberg, “Automating multi-throw multilateral surgical suturing with a mechanical needle guide and sequential convex optimization,” in _Robotics and Automation (ICRA), 2016 IEEE International Conference on_. IEEE, 2016, pp. 4178–4185.
  * [5] S. Makris, P. Tsarouchi, D. Surdilovic, and J. Krüger, “Intuitive dual arm robot programming for assembly operations,” _CIRP Annals-Manufacturing Technology_ , vol. 63, no. 1, pp. 13–16, 2014.
  * [6] J. Bandera, J. Rodriguez, L. Molina-Tanco, and A. Bandera, “A survey of vision-based architectures for robot learning by imitation,” _International Journal of Humanoid Robotics_ , vol. 9, no. 01, p. 1250006, 2012.
  * [7] A. S. Phung, J. Malzahn, F. Hoffmann, and T. Bertram, “Tool centered learning from demonstration for robotic arms with visual feedback,” in _Robotics and Biomimetics (ROBIO), 2012 IEEE International Conference on_. IEEE, 2012, pp. 1117–1122.
  * [8] M. Li, H. Yin, K. Tahara, and A. Billard, “Learning object-level impedance control for robust grasping and dexterous manipulation,” in _Robotics and Automation (ICRA), 2014 IEEE International Conference on_. IEEE, 2014, pp. 6784–6791.
  * [9] S. Hutchinson, G. D. Hager, and P. I. Corke, “A tutorial on visual servo control,” _IEEE transactions on robotics and automation_ , vol. 12, no. 5, pp. 651–670, 1996.
  * [10] V. Lippiello, B. Siciliano, and L. Villani, “Position-based visual servoing in industrial multirobot cells using a hybrid camera configuration,” _IEEE Transactions on Robotics_ , vol. 23, no. 1, pp. 73–86, 2007.
  * [11] C. Cai, E. Dean-León, D. Mendoza, N. Somani, and A. Knoll, “Uncalibrated 3d stereo image-based dynamic visual servoing for robot manipulators,” in _Intelligent Robots and Systems (IROS), 2013 IEEE/RSJ International Conference on_. IEEE, 2013, pp. 63–70.
  * [12] R. Mebarki, A. Krupa, and F. Chaumette, “2-d ultrasound probe complete guidance by visual servoing using image moments,” _IEEE Transactions on Robotics_ , vol. 26, no. 2, pp. 296–306, 2010.
  * [13] L. Zhang, M. Ye, P. Giataganas, M. Hughes, and G. Yang, “Autonomous scanning for endomicroscopic mosaicing and 3d fusion,” _CoRR_ , vol. abs/1604.04137, 2016. [Online]. Available: http://arxiv.org/abs/1604.04137
  * [14] M. Ye, L. Zhang, S. Giannarou, and G.-Z. Yang, “Real-time 3d tracking of articulated tools for robotic surgery,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2016, pp. 386–394.
  * [15] C. Wang, C.-Y. Lin, and M. Tomizuka, “Statistical learning algorithms to compensate slow visual feedback for industrial robots,” _Journal of Dynamic Systems, Measurement, and Control_ , vol. 137, no. 3, p. 031011, 2015.
  * [16] B. Huang, A. Vandini, Y. Hu, S.-L. Lee, and G.-Z. Yang, “A vision-guided dual arm sewing system for stent graft manufacturing,” in _Intelligent Robots and Systems (IROS), 2016 IEEE/RSJ International Conference on_. IEEE, 2016, pp. 751–758.
  * [17] S. Garrido-Jurado, R. M. noz Salinas, F. Madrid-Cuevas, and M. Marín-Jiménez, “Automatic generation and detection of highly reliable fiducial markers under occlusion,” _Pattern Recognition_ , vol. 47, no. 6, pp. 2280 – 2292, 2014. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S0031320314000235
  * [18] Z. Kalal, K. Mikolajczyk, and J. Matas, “Forward-backward error: Automatic detection of tracking failures,” in _Pattern recognition (ICPR), 2010 20th international conference on_. IEEE, 2010, pp. 2756–2759.
  * [19] V. Lepetit, F. Moreno-Noguer, and P. Fua, “Epnp: An accurate o (n) solution to the pnp problem,” _International journal of computer vision_ , vol. 81, no. 2, p. 155, 2009.
  * [20] D. J. Berndt and J. Clifford, “Using dynamic time warping to find patterns in time series.” in _KDD Workshop_ , vol. 10, no. 16. Seattle, WA, 1994, pp. 359–370.
  * [21] S. Calinon, F. Guenter, and A. Billard, “On learning, representing, and generalizing a task in a humanoid robot,” _Systems, Man, and Cybernetics, Part B: Cybernetics, IEEE Transactions on_ , vol. 37, no. 2, pp. 286–298, 2007.
  * [22] B. Huang, S. El-Khoury, M. Li, J. J. Bryson, and A. Billard, “Learning a real time grasping strategy,” in _Robotics and Automation (ICRA), 2013 IEEE International Conference on_ , 2013, pp. 593–600.
  * [23] S. Popić and B. Miloradović, “Light weight robot arms–an overview,” _Proceedings of Infoteh_ , pp. 818–822, 2015.
  * [24] D. J. Berndt and J. Clifford, “Using dynamic time warping to find patterns in time series.” in _KDD Workshop_ , vol. 10, no. 16. Seattle, WA, 1994, pp. 359–370.