# On quadratic conjecture††thanks: This work was supported by NSFC (No.
11971280 & 11771258)

Jingjing Duan and Lijian An
Department of Mathematics, Shanxi Normal University
Taiyuan, Shanxi 030031, P. R. China
Corresponding author. e-mail<EMAIL_ADDRESS>
###### Abstract

Quadratic conjecture is a strengthening of oliver’s $p$-group conjecture. Let
$G$ be a $p$-group of maximal class of order $p^{n}$. We prove that if $n\leq
8$ or $n\geq\max\\{2p-6,p+2\\}$ then $G$ satisfies Quadratic Conjecture. Hence
quadratic conjecture holds if $G$ is a $p$-group of maximal class where $p\leq
7$.

Keywords Quadratic conjecture F-module Quadratic offender Quadratic element
$p$-group of maximal class

2000 Mathematics subject classification: 20D15.

##  1 Introduction

An open question in the theory of $p$-local finite groups claims that every
fusion system has a unique $p$-completed classifying space. In [7], Oliver
introduced the characteristic subgroup $\mathfrak{X}(S)$ for a finite
$p$-group $S$. For odd primes he demonstrated that the conjecture outlined
below would substantiate the unique existence of the classifying space. Recall
that $J(S)$ denotes the Thompson subgroup of $S$ generated by all elementary
abelian subgroups of the greatest rank.

######  Conjecture 1.1.

[7, Conjecture 3.9] For any odd prime $p$ and any $p$-group $S$,
$J(S)\leq\mathfrak{X}(S)$.

Conjecture 1.1 is called Oliver’s p-group conjecture. Let $G$ be a finite
$p$-group and $V$ an elementary abelian $p$-group on which $G$ acts
faithfully. Then $V$ is an F-module for $G$ if there exists a non-trivial
elementary abelian subgroup $E$ of $G$ such that $|E|\cdot|C_{V}(E)|\geq|V|$.
In this case, we call $E$ an offender.

In [1], Green, H$\acute{e}$thelyi and Lilienthal obtained the following
reformulation of Conjecture 1.1.

######  Conjecture 1.2.

[1, Conjecture 1.3] Let $p$ be an odd prime and $G$ a finite $p$-group. If the
faithful $\mathbb{F}_{p}[G]$-module $V$ is an $F$-module, then there is an
element $1\neq g\in\Omega_{1}(Z(G))$ such that the minimal polynomial of the
action of $g$ on $V$ divides $(x-1)^{p-1}$.

An element $g\in G$ is said to be quadratic on the $\mathbb{F}_{p}[G]$-module
$V$ if its action has minimal polynomial $(X-1)^{2}$. Note that if $V$ is
faithful then quadratic elements must have order $p$.

In [3], Green, H$\acute{e}$thelyi and Mazza propose the following
strengthening of Conjecture 1.2.

Quadratic conjecture [3, Conjecture 1.4] Let $p$ be a prime and $G$ a finite
$p$-group. If the faithful $\mathbb{F}_{p}[G]$-module $V$ is an $F$-module,
then there are quadratic elements in $\Omega_{1}(Z(G))$.

Observe that Quardatic Conjecture is trivially true for $p=2$. For $p=3$,
Quardatic Conjecture is just Conjecture 1.2. By [2, Theorem 1.2], Quadratic
Conjecture holds if $G$ is metabelian or of (nilpotence) class at most four.
By [2, Theorem 1.3], Conjecture 1.2 holds if $G$ is a $p$-group of maximal
class. Hence Quadratic Conjecture holds if $G$ is a $3$-group of maximal
class. In this paper, we get the following main result:

######  Theorem 1.3.

Let $p$ be an odd prime and $G$ a $p$-group of maximal class of order $p^{n}$.
If $n\leq 8$ or $n\geq\max\\{2p-6,p+2\\}$ then $G$ satisfies Quadratic
Conjecture.

By Theorem 1.3, quadratic conjecture holds if $G$ is a $p$-group of maximal
class where $p\leq 7$.

##  2 Quadratic Offenders

######  Definition 2.1.

[6, 11, 2.3] Let $G$ be a finite group and $V$ a faithful
$\mathbb{F}_{p}[G]$-module. For a subgroup $H\leq G$ one sets

$j_{H}(V):=\frac{|H||C_{V}(H)|}{|V|}\in\mathbb{Q}.$

Note that $j_{1}(V)=1$. Suppose that $E$ is an element abelian subgroup of
$G$. Then $E$ is an offender on $V$ if and only if $j_{E}(V)\geq 1$.

######  Definition 2.2.

[4, 26.5] Let $G$ be a finite $p$-group and $V$ a faithful
$\mathbb{F}_{p}[G]$-module. We denote by $\mathscr{E}(G)$ the poset of non-
trivial elementary abelian subgroups of $G$. Define

$\mathscr{P}(G,V):=\\{E\leq G\mid E\in\mathscr{E}(G)\ \mbox{and}\ j_{E}(V)\geq
j_{F}(V)\ \forall\ 1\leq F\leq E\\}.$

Note that $V$ is an F-module if and only if $\mathscr{P}(G,V)$ is non-empty.
The subgroups in $\mathscr{P}(G,V)$ are sometimes called best offenders.

A quadratic offender is a subgroup which is both quadratic ($[V,E,E]=0$) and
an offender.

######  Theorem 2.3.

(Timmesfeld’s replacement theorem)[2, Theorem 4.1] Let $V$ be a faithful
$\mathbb{F}_{p}[G]$-module and $E\in\mathscr{P}(G,V)$. Then there is a
quadratic offender $F\in\mathscr{P}(G,V)$ which satisfies $j_{F}(V)=j_{E}(V)$
and $F\leq E$ .

By Timmesfeld’s replacement theorem, A minimal offender must be a quadratic
offender.

Following from [3], an abelian subgroup $A$ of $G$ is said to be weakly closed
if $[A,A^{g}]\neq 1$ holds for every $G$-conjugate $A^{g}\neq A$. The
following theorem show that there is also a weakly closed quadratic offender
on F-module $V$.

######  Theorem 2.4.

[3, Proposition 4.5] Suppose that the faithful $\mathbb{F}_{p}[G]$-module $V$
is an F-module. Set

$j_{0}=\max\\{j_{E}(V)\mid E\ \mbox{an offender}\\}.$

Then there is a weakly closed quadratic offender $E$ with $j_{E}(V)=j_{0}$.
Moreover if $D\leq G$ is any offender with $j_{D}(V)=j_{0}$, then there is
such an $E$ which is a subgroup of the normal closure of $D$.

######  Lemma 2.5.

Let $G$ a finite $p$-group and $V$ an $F$-module. Suppose that there is no
quadratic element in $\Omega_{1}(Z(G))$. Then

  * (1)

$|E|\geq p^{2}$ for any offender $E$ on $V$;

  * (2)

$|E|\geq p^{3}$ for any weakly closed offender $E$ on $V$.

Proof (1) Otherwise, there is an offender $E$ of order $p$. Since $V$ is
faithful, $C_{V}(E)\neq V$. Since $E$ is an offender, $|E||C_{V}(E)|\geq|V|$.
Hence $C_{V}(E)$ is maximal in $V$. Since $C_{V}(E)$ is $Z(G)$-invariant,
$[V,Z(G)]\leq C_{V}(E)$. Since $[V,Z(G)]\leq C_{V}(E^{g})$ for all $g\in G$,
$[V,Z(G)]\leq C_{V}(E^{G})$. Take $1\neq z\in Z(G)\cap E^{G}$. Then
$[V,z,z]=0$. Hence $z$ is a quadratic element in $\Omega_{1}(Z(G))$. This
contradicts the hypothesis.

(2) Otherwise, there is a weakly closed offender $E$ of order $p^{2}$. By (1),
$E$ is a minimal offender. Hence $E$ is also a quadratic offender. If
$E\trianglelefteq G$, then $E\cap Z(G)\neq 1$. Hence there is a quadratic
element in $\Omega_{1}(Z(G))$. This contradicts the hypothesis. In the
following, we may assume that $N_{G}(E)\neq G$. Let $g\in
N_{G}(N_{G}(E))\setminus N_{G}(E)$. Then $N_{G}(E^{g})=N_{G}(E)^{g}=N_{G}(E)$
and $E^{g}\neq E$. Hence $E$ and $E^{g}$ normalize each other. It follows that
$[E,E^{g}]\leq E\cap E^{g}$. Since $E$ is weakly closed, $[E,E^{g}]\neq 1$ and
hence $E\cap E^{g}\neq 1$.

Let $F=E\cap E^{g}$. By (1), $F$ is not an offender. Hence
$|C_{V}(E)|\leq|C_{V}(F)|\leq\frac{1}{p^{2}}|V|$. On the other hand, since $E$
is an offender, $|C_{V}(E)|\geq\frac{1}{p^{2}}|V|$. Hence
$|C_{V}(E)|=\frac{1}{p^{2}}|V|$ and $C_{V}(F)=C_{V}(E)$. The same reason gives
that $C_{V}(F)=C_{V}(E^{g})$. Since $E$ and $E^{g}$ are both quadratic,
$[V,E]\leq C_{V}(E)=C_{V}(F)=C_{V}(E^{g})$ and $[V,E^{g}]\leq
C_{V}(E^{g})=C_{V}(F)=C_{V}(E)$. It follows that $[V,E,E^{g}]=[V,E^{g},E]=0$.
By Three Subgroups Lemma, $[V,[E,E^{g}]]=0$. Hence $V$ is not faithful. This
is a contradiction. $\Box$

##  3 Quadratic elements

For finding quadratic element, Lemma 4.1 of [1] is a key result. In the
following, we refine this lemma, and give an elementary proof.

######  Lemma 3.1.

Suppose that $p$ is an odd prime, that $G$ is a non-trivial $p$-group, and
that $V$ is a faithful $\mathbb{F}_{p}[G]$-module. Suppose that $a,b\in G$ are
such that $c:=[a,b]$ is a non-trivial element of $C_{G}(a,b)$. If $a$ is
quadratic, then $[V,E,E]=0$ where $E=\langle a,c\rangle$.

Proof Let $d=a^{b}=ac$ and $e=a^{b^{2}}=ac^{2}=a^{-1}d^{2}$. Then $E=\langle
a\rangle\times\langle d\rangle$. Since $a$ is quadratic, $d$ and $e$ are also
quadratic. Since $[a,d]=1$, $[v,a,d]=[v,d,a]$ for all $v\in V$. Let $v\in V$
and $i,j,k,l\in\mathbb{Z}$. Then

$[v,a^{i}d^{j},a^{k}d^{l}]=(il+jk)[v,a,d].$

In particular, $[v,e,e]=-4[v,a,d]$. It follows from $[v,e,e]=0$ that
$[v,a,d]=0$. Hence $[v,a^{i}d^{j},a^{k}d^{l}]=0$ for all $v\in V$ and
$i,j,k,l\in\mathbb{Z}$. Thus we have $[V,E,E]=0$. $\Box$

Recall that the following terminology. Given a finite group $G$, the ascending
central series is defined inductively by $Z_{0}(G)=1$ and $Z_{r+1}(G)$ is the
normal subgroup of $G$ containing $Z_{r}(G)$ and such that
$Z_{r+1}(G)/Z_{r}(G)=Z(G/Z_{r}(G))$, for all $r\geq 1$. The class $c$ of $G$
is the smallest number such that $Z_{c}(G)=G$. In this paper we use
$G>G_{2}=G^{\prime}>\cdots>G_{c+1}=1$ to denote the descending central series
of $G$, where $G_{r+1}=[G_{r},G]$ for all $r\geq 2$.

######  Lemma 3.2.

Let $G$ be a $p$-group and $V$ an F-module. $E$ is a weakly closed quadratic
offender. If there is no quadratic element in $\Omega_{1}(Z(G))$, then

  * (1)

there is no quadratic element in $Z_{2}(G)$;

  * (2)

$Z_{2}(G)\cap E=1$, $[Z_{2}(G),E]=1$;

  * (3)

$[Z_{3}(G),E]=1$;

  * (4)

$[Z_{4}(G),E]\leq Z_{3}(G)\cap E$.

Proof (1) Otherwise, let $a$ be a quadratic element in $Z_{2}(G)$. Then
$a\not\in Z(G)$ since there is no quadratic element in $Z(G)$. Hence there is
$b\in G$ such that $c:=[a,b]\neq 1$. Since $c\in Z(G)\leq C_{G}(a,b)$, by
Lemma 3.1, $c$ is also a quadratic element. This is a contradiction.

(2) By (1), $Z_{2}(G)\cap E=1$. Since $[Z_{2}(G),E]\leq Z(G)$, $[E^{g},E]=1$
for all $g\in Z_{2}(G)$. Since $E$ weakly closed, $E^{g}=E$ for all $g\in
Z_{2}(G)$. It follows that $[Z_{2}(G),E]\leq Z(G)\cap E=1$.

(3) Since $[Z_{3}(G),E]\leq Z_{2}(G)$, by (2), $[Z_{3}(G),E,E]=1$. It follows
that $[E^{g},E]=1$ for all $g\in Z_{3}(G)$. Since $E$ weakly closed, $E^{g}=E$
for all $g\in Z_{3}(G)$. Furthermore, $[Z_{3}(G),E]\leq Z_{2}(G)\cap E=1$.

(4) Since $[Z_{4}(G),E]\leq Z_{3}(G)$, by (3), $[Z_{4}(G),E,E]=1$. It follows
that $[E^{g},E]=1$ for all $g\in Z_{4}(G)$. Since $E$ weakly closed, $E^{g}=E$
for all $g\in Z_{4}(G)$. Furthermore, $[Z_{4}(G),E]\leq Z_{3}(G)\cap E$.
$\Box$

######  Corollary 3.3.

[2, Theorem 5.2] Suppose that $G$ is a $p$-group and $V$ is a faithful
$\mathbb{F}_{p}[G]$-module such that $\Omega_{1}(Z(G))$ has no quadratic
elements. If $G$ has class at most four, then $V$ cannot be an F-module.

Proof Otherwise, $V$ is an F-module. By Lemma 3.2 (4), $G=Z_{4}(G)$ normalize
$E$. Hence $E\cap Z(G)\neq 1$, which is a contradiction. $\Box$

######  Theorem 3.4.

[2, Theorem 1.5] Let $G$ be a $p$-group and $V$ a faithful
$\mathbb{F}_{p}[G]$-module such that there is no quadratic element in
$\Omega_{1}(Z(G))$.

If $A$ is an abelian normal subgroup of $G$, then $A$ does not contain any
offender.

Suppose that $E$ is an offender. Then $[G^{\prime},E]\neq 1$.

######  Theorem 3.5.

Let $G$ be a $p$-group and $V$ a faithful $\mathbb{F}_{p}[G]$-module. If there
is a quadratic offender $E$ such that $E^{G}=EK$ where $K$ is an abelian
normal subgroup of $G$, then there is a quadratic element in
$\Omega_{1}(Z(G))$.

Proof Let $K$ be a maximal member such that $E^{G}=EK$ where $K$ is an abelian
normal subgroup of $G$.

If $K=E^{G}$, then $K$ contains a offender $E$. By Theorem 3.4 (1), there is a
quadratic element in $\Omega_{1}(Z(G))$.

If $K\neq E^{G}$, then there is an $a\in E$ such that $N=\langle a\rangle
K\unlhd G$. Hence $N^{\prime}\cap\Omega_{1}(Z(G))\neq 1$. Since $K$ is
abelian, $N^{\prime}=\\{[a,x]\mid x\in K\\}$. Hence there exists an $x\in K$
such that $1\neq[a,x]\in\Omega_{1}(Z(G))$. By Lemma 3.1, $[a,x]$ is quadratic.
$\Box$

##  4 $p$-groups of maximal class

Firstly we recall some terminology. Let $G$ be a $p$-group. The
(characteristic) subgroup of $G$ generated by $\\{g^{p}\mid g\in G\\}$ is
denoted by $\mho_{1}(G)$. Now suppose that $G$ has order $p^{n}$ and is of
maximal class. Set $G_{1}=C_{G}(G_{2}/G_{4})$. One says that $G$ is
exceptional if $n\geq 5$ and there is some $3\leq i\leq n-2$ such that
$C_{G}(G_{i}/G_{i+2})\neq G_{1}$.

######  Definition 4.1.

[5, Definition 3.2.1] The degree of commutativity $l$ of a $p$-group $G$ of
maximal class is defined to be the maximum integer such that
$[G_{i},G_{j}]\leq G_{i+j+l}$ for all $i,j\geq 1$ if $G_{1}$ is not abelian,
and $l=n-3$ if $G_{1}$ is abelian.

######  Theorem 4.2.

[5, Corollary 3.2.7] The degree of commutativity of $G$ is positive if and
only if $G$ is not exceptional.

######  Theorem 4.3.

[5, Corollary 3.3.4 (i)] Let $G$ be a $p$-group of maximal class of order
$p^{n}$ with $n>p+1$. Then $\Omega_{1}(G_{1})=G_{n-p+1}$.

######  Theorem 4.4.

[5, Theorem 3.3.5] Let $G$ be a $p$-group of maximal class of order $p^{n}$
with $n>p+1$. Then $G$ has positive degree of commutativity.

######  Theorem 4.5.

[5, Theorem 3.2.11] Let $G$ be a $p$-group of maximal class of order $p^{n}$
where $n$ is odd and $5\leq n\leq 2p+1$. Then $G$ has positive degree of
commutativity.

######  Theorem 4.6.

Let $p$ be an odd prime and $G$ a $p$-group of maximal class of order $p^{n}$.
If $n\geq\max\\{2p-6,p+2\\}$ then $G$ satisfies Quadratic Conjecture.

Proof Otherwise, there is an F-module $V$ such that there is no quadratic
element in $\Omega_{1}(Z(G))$. By Theorem 2.4, there is a weakly closed
quadratic offender $E$ on $V$. If $E\trianglelefteq G$, then $G_{n-1}=Z(G)\leq
E$ and hence there is a quadratic element in $\Omega_{1}(Z(G))$. In the
following, we may assume that $E<E^{G}$. By Theorem 3.4 (1), we may assume
that $E^{G}$ is not abelian.

By Lemma 3.2 (3), $[Z_{3}(G),E]=1$. Hence $E\leq C_{G}(G_{n-3}/G_{n-1})$.
Since $n\geq p+2$, by Theorem 4.2 $\&$ 4.4, $G$ is not exceptional. Hence
$E\leq G_{1}$. Furthermore, $E\leq\Omega_{1}(G_{1})$. By Theorem 4.3,
$\Omega_{1}(G_{1})=G_{n-p+1}$. It follows that $E\leq G_{n-p+1}$.

Since $E^{G}$ is not abelian, $1\neq[E,E^{G}]$. Since $G$ has positive degree
of commutativity, $[G_{n-p+1},G_{n-p+1}]=[G_{n-p+1},G_{n-p+2}]\leq
G_{2n-2p+4}$. Since $n\geq 2p-6$, $2n-2p+4\geq n-2$. Hence
$1\neq[E,E^{G}]\leq[G_{n-p+1},G_{n-p+1}]\leq G_{n-2}=Z_{2}(G)$. Thus there are
$a\in E$ and $b\in E^{G}$ such that $1\neq c:=[a,b]\in Z_{2}(G)$. By Lemma 3.2
(2), $c\in C_{G}(a,b)$. By Lemma 3.1, $c$ is a quadratic element. This
contradicts Lemma 3.2 (1). $\Box$

By Theorem 4.6, if $G$ is a $5$-group of maximal class of order $5^{n}$ with
$n\geq 7$, then $G$ satisfies Quadratic Conjecture; and if $G$ is a $7$-group
of maximal class of order $7^{n}$ with $n\geq 9$, then $G$ satisfies Quadratic
Conjecture. By Corollary 3.3, if $G$ is a $p$-group of maximal class of order
$p^{n}$ with $n\leq 5$, then $G$ satisfies Quadratic Conjecture. In the
following, we prove that if $G$ is a $p$-group of maximal class of order
$p^{n}$ with $6\leq n\leq 8$, then $G$ satisfies Quadratic Conjecture. It will
deduce that Quadratic Conjecture holds if $G$ is a $p$-group of maximal class
where $p\leq 7$.

######  Theorem 4.7.

Let $G$ be a $p$-group of maximal class with $|G|=p^{6}$. Then $G$ satisfies
Quadratic Conjecture.

Proof Otherwise, there is an F-module $V$ such that there is no quadratic
element in $\Omega_{1}(Z(G))$. By Theorem 2.4, there is a weakly closed
quadratic offender $E$ on $V$. By Lemma 3.2 (4), $G^{\prime}=Z_{4}(G)$
normalize $E$ and $[G^{\prime},E]\leq G_{3}\cap E$. By Theorem 3.4 (2),
$[G^{\prime},E]\neq 1$. Hence $E\cap G_{3}\neq 1$. Since there is no quadratic
element in $Z(G)=G_{5}$, $[E\cap G_{3},G^{\prime}]\leq E\cap G_{5}=1$. It
follows that $E\cap G_{3}\leq Z(G^{\prime})$. By [2, Theorem 6.2],
$G^{\prime}$ is not abelian. Hence $Z(G^{\prime})\leq G_{4}=Z_{2}(G)$. Thus
$E\cap G_{3}\leq Z_{2}(G)$. This contradicts Lemma 3.2 (2). $\Box$

######  Theorem 4.8.

Let $p$ be an odd prime and $G$ a $p$-group of maximal class with $|G|=p^{7}$.
Then $G$ satisfies Quadratic Conjecture.

Proof Otherwise, there is an F-module $V$ such that there is no quadratic
element in $\Omega_{1}(Z(G))$. By Theorem 2.4, there is a weakly closed
quadratic offender $E$ on $V$. By Lemma 3.2 (4), $G_{3}=Z_{4}(G)$ normalize
$E$.

By Lemma 3.2 (3), $[Z_{3}(G),E]=1$. Hence $E\leq C_{G}(G_{4}/G_{6})$. By
Theorem 4.2 $\&$ 4.5, $G$ is not exceptional. Hence $E\leq G_{1}$. It follows
that $[G_{3},E]\leq E\cap[G_{3},G_{1}]\leq E\cap G_{5}=1$.

Since $[G_{3},G_{3}]\leq G_{7}=1$, $G_{3}$ is abelian. By Theorem 3.4 (1),
$E\not\leq G_{3}$. Hence $G^{\prime}\leq E^{G}\leq G_{1}$.

Since $[E,G_{3}]=1$, $[E^{G},G_{3}]=1$. It follows that
$[G^{\prime},G^{\prime}]=[G^{\prime},G_{3}]\leq[E^{G},G_{3}]=1$. Hence
$G^{\prime}$ is abelian. This contradicts [2, Theorem 6.2]. $\Box$

######  Lemma 4.9.

Suppose that $A$ is an abelian normal subgroup of a non-abelian $p$-group $G$,
$G/A=\langle xA\rangle$ is cyclic, $B\leq A$ and $B\unlhd G$. Then $[B,G]\cong
B/B\cap Z(G)$.

Proof Since $G/A$ is cyclic, $G^{\prime}\leq A$. Let $\varphi:B\mapsto[B,G]$
be a mapping defined as follows: $\varphi(b)=[b,x]$ for $b\in B$. If
$b_{1},b_{2}\in B$, then

$\varphi(b_{1}b_{2})=[b_{1}b_{2},x]=[b_{1},x]^{b_{2}}[b_{2},x]=[b_{1},x][b_{2},x]=\varphi(b_{1})\varphi(b_{2}),$

since $[b_{1},x],{b_{2}}\in A$ and $A$ is abelian. Hence $\varphi$ is a
homomorphism. If $b\in B$ and $g\in G$, Then

$[b,x]^{g}=[b^{g},x^{g}]=[b^{g},x[x,g]]=[b^{g},[x,g]][b^{g},x]^{[x,g]}=[b^{g},x],$

since $b^{g},[x,g],[b^{g},x]\in A$ and $A$ is abelian. It follows that
$\hbox{\rm Im}\varphi$ is normal in $G$. Since $B$ centralizes $G/\hbox{\rm
Im}\varphi$, $[B,G]\leq\hbox{\rm Im}\varphi$. Hence $\hbox{\rm
Im}\varphi=[B,G]$. Next, $\hbox{\rm Ker}\varphi=\\{b\in
B\mid[b,x]=1\\}=C_{B}(x)=B\cap Z(G)$. Hence $\hbox{\rm Im}\varphi=[B,G]\cong
B/B\cap Z(G)$. $\Box$

######  Theorem 4.10.

Let $p$ be an odd prime and $G$ a $p$-group of maximal class with $|G|=p^{8}$.
Then $G$ satisfies Quadratic Conjecture.

Proof Otherwise, there is an F-module $V$ such that there is no quadratic
element in $\Omega_{1}(Z(G))$. By Theorem 2.4, there is a weakly closed
quadratic offender $E$ on $V$. By Lemma 3.2 (4), $G_{4}=Z_{4}(G)$ normalize
$E$.

For convenience, let $G^{*}=C_{G}(G_{5}/G_{7})$, a maximal subgroup of $G$. By
Lemma 3.2 (3), $[G_{5},E]=[Z_{3}(G),E]=1$. Hence

$E\leq C_{G}(G_{5}/G_{7})=G^{*}.$ (4.1)

By [2, Theorem 6.2], $G^{\prime}$ is not abelian. Hence

$Z(G^{\prime})\leq G_{4}.$ (4.2)

We claim that $E\not\leq G_{3}$. Otherwise, $E\leq G_{3}$. In this case,
$[E,G_{3}]\leq[G_{3},G_{3}]=[G_{3},G_{4}]\leq G_{7}$. Since $[G_{3},E,E]=1$,
$G_{3}$ normalize $E$. Hence $[E,G_{3}]\leq E\cap G_{7}=1$. It follows that
$E\leq Z(G_{3})$. This contradicts Theorem 3.4 (1).

We claim that $E\not\leq G^{\prime}$. Otherwise, $E^{G}=G^{\prime}$. In this
case, $[E,G_{4}]\leq[G^{\prime},G_{4}]\leq G_{6}$. Since $G_{4}$ normalize
$E$, $[E,G_{4}]\leq E\cap G_{6}=1$. It follows that $[E^{G},G_{4}]=1$. Hence
$G_{4}=Z(G^{\prime})$. Note that $G_{3}$ is abelian and $E^{G}=EG_{3}$. By
Theorem 3.5, there is a quadratic element in $\Omega_{1}(Z(G))$. This
contradicts the hypothesis.

By above argument,

$E^{G}=G^{*}=EG^{\prime}.$ (4.3)

Since $G^{\prime}$ is not abelian,
$[E^{G},G_{3}]\geq[G^{\prime},G_{3}]=G^{\prime\prime}\neq 1$. Hence

$[E,G_{3}]\neq 1.$ (4.4)

Case 1. $[E\cap G^{\prime},G^{\prime}]\neq 1$.

In this case, we claim that

$E\cap G^{\prime}\not\leq G_{4}.$ (4.5)

Otherwise, $1\neq[E\cap G^{\prime},G^{\prime}]\leq[G_{4},G^{\prime}]\leq
G_{6}$. Hence there are $a\in E\cap G^{\prime}$ and $b\in G^{\prime}$ such
that $1\neq c:=[a,b]\in G_{6}=Z_{2}(G)$. Since $c\in C_{G}(a,b)$, by Lemma
3.1, $c$ is a quadratic element. This contradicts Lemma 3.2 (1).

Since $G_{4}$ normalize $E$, $[E\cap G^{\prime},G_{4}]\leq G_{6}\cap E$. By
Lemma 3.2 (2), $G_{6}\cap E=Z_{2}(G)\cap E=1$. Hence $[E\cap
G^{\prime},G_{4}]=1$. Furthermore,

$[(E\cap G^{\prime})^{G},G_{4}]=1.$ (4.6)

We claim that $E\cap G^{\prime}\leq G_{3}$. Otherwise, $(E\cap
G^{\prime})^{G}=G^{\prime}$. By (4.6), $G_{4}\leq Z(G^{\prime})$. It follows
that $G_{3}$ is abelian. By Lemma 4.9,
$G^{\prime\prime}=[G^{\prime},G_{3}]\cong G_{3}/G_{3}\cap
Z(G^{\prime})=G_{3}/G_{4}$. Hence $G^{\prime\prime}=G_{7}$. Since $1\neq[E\cap
G^{\prime},G^{\prime}]=G_{7}=Z(G)$, there are $a\in E\cap G^{\prime}$ and
$b\in G^{\prime}$ such that $1\neq c:=[a,b]\in Z(G)$. By Lemma 3.1, $c$ is a
quadratic element. This contradicts the hypothesis. By (4.5),

$(E\cap G^{\prime})^{G}=G_{3}.$ (4.7)

By (4.6) and (4.7), $[G_{3},G_{3}]=[G_{3},G_{4}]=[(E\cap
G^{\prime})^{G},G_{4}]=1$. Hence $G_{3}$ is abelian. By Lemma 4.9,
$[G^{\prime},G_{4}]\cong G_{4}/G_{4}\cap Z(G^{\prime})=G_{4}/Z(G^{\prime})$.
Since $[G_{5},E]=1$, $[G_{5},E^{G}]=1$. Hence $G_{5}\leq Z(G^{\prime})$ and
$|[G^{\prime},G_{4}]|\leq p$. Thus

$[G^{\prime},G_{4}]\leq G_{7}.$ (4.8)

Let $K=EG_{3}$. Then $K/G_{3}\cong E/E\cap G_{3}=E/E\cap G^{\prime}\cong
EG^{\prime}/G^{\prime}=G^{*}/G^{\prime}$ is cyclic. By Lemma 4.9,
$[K,G_{4}]\cong G_{4}/G_{4}\cap Z(K)$. Since $G_{5}\leq Z(K)$,
$|[K,G_{4}]|\leq p$. It follows that
$[G^{*},G_{4}]=[KG^{\prime},G_{4}]=[K,G_{4}][G^{\prime},G_{4}]$ is of order at
most $p^{2}$. Hence

$[E,G_{4}]\leq[G^{*},G_{4}]\leq G_{6}.$ (4.9)

Furthermore, $[E,G_{4}]\leq G_{6}\cap E=1$. By (4.3), $G_{4}\leq Z(G^{*})\leq
Z(G^{\prime})$. By Lemma 4.9, $G^{\prime\prime}=[G^{\prime},G_{3}]\cong
G_{3}/G_{3}\cap Z(G^{\prime})=G_{3}/G_{4}$. Hence $G^{\prime\prime}=G_{7}$.
Furthermore, $1\neq[E\cap G^{\prime},G^{\prime}]=G^{\prime\prime}=G_{7}=Z(G)$.
Hence there are $a\in E\cap G^{\prime}$ and $b\in G^{\prime}$ such that $1\neq
c:=[a,b]\in Z(G)$. By Lemma 3.1, $c$ is a quadratic element. This contradicts
the hypothesis.

Case 2. $[E\cap G^{\prime},G^{\prime}]=1$.

In this case, $E\cap G^{\prime}\leq Z(G^{*})$ since $[E\cap G^{\prime},E]=1$.
Hence $E\cap G^{\prime}\leq G_{4}$ and

$E\cap G^{\prime}=E\cap G_{4}.$ (4.10)

By Lemma 3.2 (2), $E\cap G_{6}=1$. It follows that $|E\cap G_{4}|\leq p^{2}$.
On the other hand, by Lemma 2.5, $|E|\geq p^{3}$. It follows that $|E\cap
G_{4}|=|E\cap G^{\prime}|\geq p^{2}$. Thus $|E|=p^{3}$, $|E\cap G_{4}|=p^{2}$,
$|E\cap G_{5}|=p$ and $E\cap G^{\prime}\not\leq G_{5}$. Hence

$Z(G^{*})=G_{4}=Z(G^{\prime}),$ (4.11)

and $G_{3}$ is abelian. Note that $EG_{3}/G_{3}\cong E/E\cap G_{3}=E/E\cap
G^{\prime}\cong EG^{\prime}/G^{\prime}$ is cyclic. By Lemma 4.9,
$[EG_{3},G_{3}]\cong G_{3}/G_{3}\cap Z(EG_{3})=G_{3}/G_{4}$, and
$[G^{\prime},G_{3}]\cong G_{3}/G_{3}\cap Z(G^{\prime})=G_{3}/G_{4}$. Hence
$[E,G_{3}]$ and $[G^{\prime},G_{3}]$ are of order $p$. It follows that
$[G^{*},G_{3}]=[EG^{\prime},G_{3}]=[E,G_{3}][G^{\prime},G_{3}]\leq G_{6}$. In
this case, $[G_{3},E,E]=1$. Hence $G_{3}$ normalize $E$. Thus $[E,G_{3}]\leq
E\cap G_{6}=1$. This contradicts (4.4). $\Box$

## References

  * [1] D. J. Green, L. H$\acute{e}$thelyi and M. Lilienthal, On Oliver’s $p$-group conjecture, Alg. Number Theory 2 (2008), 969-977.
  * [2] D. J. Green, L. H$\acute{e}$thelyi and N. Mazza, On Oliver’s $p$-group conjecture: II, Math. Ann. 347 (2010), 111-122.
  * [3] D. J. Green, L. H$\acute{e}$thelyi and N. Mazza, On a strong form of Oliver’s $p$-group conjecture, J. Algebra, 342 (2011), 1-15.
  * [4] Gorenstein, D., Lyons, R., Solomon, R.: The classification of the finite simple groups. Number 2, Mathematical Surveys and Monographs, vol. 40. American Mathematical Society, Providence, RI (1996)
  * [5] Leedham-Green,C.R., McKay, S.: The structure of groups of prime power order, London Mathematical Society Monographs. New Series, vol. 27. Oxford University Press, Oxford (2002)
  * [6] U. Meierfrankenfeld and B. Stellmacher, The other $\mathscr{P}(G,V)$-theorem, Rend. Sem. Mat. Univ. Padova 115, (2006), 41-50.
  * [7] B. Oliver, Equivalences of classifying spaces completed at odd primes, Math. Proc. Cambridge Philos. Soc. 137(2), (2004), 321-347.


# Onsets and Frames: Dual-Objective Piano Transcription

###### Abstract

We advance the state of the art in polyphonic piano music transcription by
using a deep convolutional and recurrent neural network which is trained to
jointly predict onsets and frames. Our model predicts pitch onset events and
then uses those predictions to condition framewise pitch predictions. During
inference, we restrict the predictions from the framewise detector by not
allowing a new note to start unless the onset detector also agrees that an
onset for that pitch is present in the frame. We focus on improving onsets
_and_ offsets together instead of either in isolation as we believe this
correlates better with human musical perception. Our approach results in over
a 100% relative improvement in note F1 score (with offsets) on the MAPS
dataset. Furthermore, we extend the model to predict relative velocities of
normalized audio which results in more natural-sounding transcriptions.

##  1 Introduction

Automatic music transcription (AMT) aims to create a symbolic music
representation (e.g., MIDI) from raw audio. Converting audio recordings of
music into a symbolic form makes many tasks in music information retrieval
(MIR) easier to accomplish, such as searching for common chord progressions or
categorizing musical motifs. Making a larger collection of symbolic music
available also broadens the scope of possible computational musicology studies
[8].

Piano music transcription is a task considered difficult even for humans due
to its inherent polyphonic nature. Accurate note identifications are further
complicated by the way note energy decays after an onset, so a transcription
model needs to adapt to a note with varying amplitude and harmonics.
Nonnegative matrix factorization (NMF) is an early popular method used in the
task of polyphonic music transcription [19]. With recent advancements in deep
learning, neural networks have attracted more and more attention from the AMT
community [18, 13]. In particular, the success of convolutional neural
networks (CNN) for image classification tasks [21] has inspired the use of
CNNs for AMT because two-dimensional time-frequency representations (e.g.,
constant-Q transform [5]) are common input representations for audio. In [13],
the authors demonstrated the potential for a single CNN-based acoustic model
to accomplish polyphonic piano music transcription. [18] considered an
approach inspired by common models used in speech recognition where a CNN
acoustic model and a Recurrent Neural Network (RNN) language model are
combined. In this paper, we investigate improving the acoustic model by
focusing on note onsets.

Note onset detection looks for only the very beginning of a note. Intuitively,
the beginning of a piano note is easier to identify because the amplitude of
that note is at its peak. For piano notes, the onset is also percussive and
has a distinctive broadband spectrum. Once the model has determined onset
events, we can condition framewise note detection tasks on this knowledge.
Previously, [6, 27] demonstrated the promise of modeling onset events
explicitly in both NMF and CNN frameworks. In this work, we demonstrate that a
model conditioned on onsets achieves state of the art performance by a large
margin for all common metrics measuring transcription quality: frame, note,
and note-with-offset.

We also extend our model to predict the relative velocity of each onset.
Velocity captures the speed with which a piano key was depressed and is
directly related to how loud that note sounds. Including velocity information
in a transcription is critical for describing the expressivity of a piano
performance and results in much more natural-sounding transcriptions.

##  2 Dataset and Metrics

We use the MAPS dataset [9] which contains audio and corresponding annotations
of isolated notes, chords, and complete piano pieces. Full piano pieces in the
dataset consist of both pieces rendered by software synthesizers and
recordings of pieces played by a Yamaha Disklavier player piano. We use the
set of synthesized pieces as the training split and the set of pieces played
on the Disklavier as the test split, as proposed in [18]. When constructing
these datasets, we also ensured that the same music piece was not present in
more than one set. Not including the Disklavier recordings, individual notes,
or chords in the training set is closer to a real-world testing environment
because we often do not have access to recordings of a testing piano at
training time. Testing on the Disklavier recordings is also more realistic
because many of the recordings that are most interesting to transcribe are
ones played on real pianos.

When processing the MAPS MIDI files for training and evaluation, we first
translate “sustain pedal” control changes into longer note durations. If a
note is active when sustain goes on, that note will be extended until either
sustain goes off or the same note is played again. This process gives the same
note durations as the text files included with the dataset.

The metrics used to evaluate a model are frame-level and note-level metrics
including precision, recall, and F1 score. We use the _mir_eval_ library [16]
to calculate note-based precision, recall, and F1 scores. As is standard, we
calculate two versions of note metrics: one requiring that onsets be within
$\pm$50ms of ground truth but ignoring offsets and one that also requires
offsets resulting in note durations within 20% of the ground truth or within
50ms, whichever is greater. Frame-based scores are calculated using the
standard metrics as defined in [2]. We also introduce a new note metric for
velocity transcription that is further described in Section 3.1. Both frame
and note scores are calculated per piece and the mean of these per-piece
scores is presented as the final metric for a given collection of pieces.

Our goal is to generate piano transcriptions that contain all perceptually
relevant performance information in an audio recording without prior
information about the recording environment such as characterization of the
instrument. We need a numerical measure that correlates with this perceptual
goal. Poor quality transcriptions can still result in high frame scores due to
short spurious notes and repeated notes that should be held. Note onsets are
important, but a piece played with only onset information would either have to
be entirely staccato or use some kind of heuristic to determine when to
release notes. A high note-with-offset score will correspond to a
transcription that sounds good because it captures the perceptual information
from both onsets and durations. Adding a velocity requirement to this metric
ensures that the dynamics of the piece are captured as well. More perceptually
accurate metrics may be possible and warrant further research. In this work we
focus on improving the note-with-offset score, but also achieve state of the
art results for the more common frame and note scores and extend the model to
transcribe velocity information as well.

##  3 Model Configuration

Framewise piano transcription tasks typically process frames of raw audio and
produce frames of note activations. Previous framewise prediction models [18,
13] have treated frames as both independent and of equal importance, at least
prior to being processed by a separate language model. We propose that some
frames are more important than others, specifically the onset frame for any
given note. Piano note energy decays starting immediately after the onset, so
the onset is both the easiest frame to identify and the most perceptually
significant.

We take advantage of the significance of onset frames by training a dedicated
note onset detector and using the raw output of that detector as additional
input for the framewise note activation detector. We also use the thresholded
output of the onset detector during the inference process, similar to
concurrent research described in [24]. An activation from the frame detector
is only allowed to start a note if the onset detector agrees that an onset is
present in that frame.

Our onset and frame detectors are built upon the convolution layer acoustic
model architecture presented in [13], with some modifications. We use
_librosa_ [15] to compute the same input data representation of mel-scaled
spectrograms with log amplitude of the input raw audio with 229
logarithmically-spaced frequency bins, a hop length of 512, an FFT window of
2048, and a sample rate of 16kHz. We present the network with the entire input
sequence, which allows us to feed the output of the convolutional frontend
into a recurrent neural network (described below).

The onset detector is composed of the acoustic model, followed by a
bidirectional LSTM [17] with 128 units in both the forward and backward
directions, followed by a fully connected sigmoid layer with 88 outputs for
representing the probability of an onset for each of the 88 piano keys.

The frame activation detector is composed of a separate acoustic model,
followed by a fully connected sigmoid layer with 88 outputs. Its output is
concatenated together with the output of the onset detector and followed by a
bidirectional LSTM with 128 units in both the forward and backward directions.
Finally, the output of that LSTM is followed by a fully connected sigmoid
layer with 88 outputs. During inference, we use a threshold of 0.5 to
determine whether the onset detector or frame detector is active.

Figure 1: Diagram of Network Architecture

Training RNNs over long sequences can require large amounts of memory and is
generally faster with larger batch sizes. To expedite training, we split the
training audio into smaller files. However, when we do this splitting we do
not want to cut the audio during notes because the onset detector would miss
an onset while the frame detector would still need to predict the note’s
presence. We found that 20 second splits allowed us to achieve a reasonable
batch size during training of at least 8, while also forcing splits in only a
small number of places where notes are active. When notes are active and we
must split, we choose a zero-crossing of the audio signal. Inference is
performed on the original and un-split audio file.

Our ground truth note labels are in continuous time, but the results from
audio processing are in spectrogram frames. So, we quantize our labels to
calculate our training loss. When quantizing, we use the same frame size as
the output of the spectrogram. However, when calculating metrics, we compare
our inference results against the original, continuous time labels.

Our loss function is the sum of two cross-entropy losses: one from the onset
side and one from the note side.

$L_{total}=L_{onset}+L_{frame}$ (1)
$L_{onset}=\sum_{p=p_{min}}^{p_{max}}\sum_{t=0}^{T}CE\left(\mathbf{I}_{onset}(p,t),\mathbf{P}_{onset}(p,t)\right)$
(2)

where $p_{min/max}$ denote the MIDI pitch range of the piano roll, $T$ is the
number of frames in the example, $\mathbf{I}_{onset}(p,t)$ is an indicator
function that is 1 when there is a ground truth onset at pitch $p$ and frame
$t$, $\mathbf{P}_{onset}(p,t)$ is the probability output by the model at pitch
$p$ and frame $t$ and $CE$ denotes cross entropy. The labels for the onset
loss are created by truncating note lengths to
$min(note\\_length,onset\\_length)$ prior to quantization. We performed a
coarse hyperparameter search over $onset\\_length$ (we tried 16, 32 and 48ms)
and found that 32ms worked best. In hindsight this is not surprising as it is
also the length of our frames and so almost all onsets will end up spanning
exactly two frames. Labeling only the frame that contains the exact beginning
of the onset does not work as well because of possible mis-alignments of the
audio and labels. We experimented with requiring a minimum amount of time a
note had to be present in a frame before it was labeled, but found that the
optimum value was to include any presence.

In addition, within the frame-based loss term $L_{frame}$, we apply a
weighting to encourage accuracy at the start of the note. A note starts at
frame $t_{1}$, completes its onset at $t_{2}$ and ends at frame $t_{3}$.
Because the weight vector assigns higher weights to the early frames of notes,
the model is incentivized to predict the beginnings of notes accurately, thus
preserving the most important musical events of the piece. First, we define a
raw frame loss as:

$L_{frame}=\sum_{p=p_{min}}^{p_{max}}\sum_{t=0}^{T}CE\left(\mathbf{I}_{frame}(p,t),\mathbf{P}_{frame}(p,t)\right)$
(3)

where $\mathbf{I}_{frame}(p,t)$ is 1 when pitch $p$ is active in the ground
truth in frame $t$ and $\mathbf{P}_{frame}(p,t)$ is the probability output by
the model for pitch $p$ being active at frame $t$. Then, we define the
weighted frame loss as:

$L_{frame}(l,p)=\begin{cases}cL^{\prime}_{frame}(l,p)&t_{1}\leq t\leq t_{2}\\\
\frac{c}{t-t_{2}}L^{\prime}_{frame}&t_{2}<t\leq t_{3}\\\
L^{\prime}_{frame}(l,p)&elsewhere\end{cases}$ (4)

where $c=5.0$ as determined with coarse hyperparameter search.

###  3.1 Velocity Estimation

We further extend the model by adding another stack to also predict velocities
for each onset. This stack is similar to the others and consists of the same
layers of convolutions. This stack does not connect to the other two. The
velocity labels are generated by dividing all the velocities by the maximum
velocity present in the piece. The smallest velocity does not go to zero, but
rather to $\frac{v_{min}}{v_{max}}$. The stack is trained with the following
loss averaged across a batch:

$L_{vel}=\sum_{p=p_{min}}^{p_{max}}\sum_{t=0}^{T}\mathbf{I}_{onset}(p,t)(v^{p,t}_{label}-v^{p,t}_{predicted})^{2}$
(5)

At inference time the output is clipped to $[0,1]$ and then transformed to a
midi velocity by the following mapping:

$v_{midi}=80v_{predicted}+10$ (6)

The final mapping is arbitrary, but we found this leads to pleasing audio
renderings.

While various studies have considered the estimation of dynamics (note
intensities or velocities) in a recording given the score [22, 10, 26], to our
knowledge there has been no work in the literature considering estimation of
dynamics alongside pitch and timing information. As a result, as Benetos et
al. [3] noted in their review paper in 2013, “evaluating the performance of
current [automatic music transcription] systems for the estimation of note
dynamics has not yet been addressed.” To evaluate our velocity-aware model, we
therefore propose an additional criterion for the note-level precision,
recall, and F1 scores.

Evaluating velocity predictions is not straightforward because unlike pitch
and timing, velocity has no absolute meaning. For example, if two
transcriptions contained identical velocities except that they were offset or
scaled by a constant factor, they would be effectively equivalent despite
reporting completely different velocities for every note. To address these
issues, we first re-scale all of the ground-truth velocities in a
transcription to be in the range $[0,1]$. After notes are matched according to
their pitch and onset/offset timing, we assemble pairs of the reference
(ground-truth) and estimated velocities for matched notes, referred to as
$v_{r}$ and $v_{e}$ respectively. We then perform a linear regression to
estimate a global scale and offset parameter such that the squared difference
between pairs of reference and estimated velocities is minimized:

$m,b=\operatorname*{arg\,min}_{m,b}\sum_{i=1}^{M}\|v_{r}(i)-(mv_{e}(i)+b)\|^{2}$
(7)

where $M$ is the number of matches (i.e. number of entries in $v_{r}$ and
$v_{e}$). These scalar parameters are used to re-scale the entries of $v_{e}$
to obtain

$\hat{v}_{e}=\\{mv_{e}(i)+b,i\in 1,\ldots,M\\}$ (8)

Finally, a match $i$ is now only considered correct if, in addition to having
its pitch and timing match, it also satisfies $|\hat{v}_{e}(i)-v_{r}(i)|<\tau$
for some threshold $\tau$. We used $\tau=0.1$ in all of our evaluations. The
precision, recall, and F1 scores are then recomputed as normal based on this
newly filtered list of matches.

##  4 Experiments

We trained our onsets and frames model using TensorFlow [1] on the training
dataset described in Section 2 using a batch size of 8, a learning rate of
.0006, and a gradient clipping L2-norm of 3. A hyperparameter search was
conducted to find the optimal learning rate. We use the Adam optimizer [14]
and train for 50,000 steps. Training takes 5 hours on 3 P100 GPUs. The same
hyperparameters were used to train all models, including those from the
ablation study, except when reproducing the results of [18] and [13], where
hyperparameters from the respective papers were used. The source code for our
model is available at https://goo.gl/magenta/onsets-frames-code.

For comparison, we reimplemented the models described in [18, 13] to ensure
evaluation consistency. We also compared against the commercial software
Melodyne version 4.1.1.011111http://www.celemony.com/en/melodyne. We would
have liked to compare against AnthemScore222https://www.lunaverus.com/ as
described in [25] as well, but because it produces a MusicXML score with
quantized note durations instead of a MIDI file with millisecond-scale
timings, an accurate comparison was not possible.

Results from these evaluations are summarized in Table 1. Our onsets and
frames model not only produces better note-based scores (which only take into
account onsets), it also produces the best frame-level scores and note-based
scores that include offsets.

An example input spectrogram, note and onset output posteriorgrams, and
inferred transcription for a recording from outside of the training set is
shown in Figure 2. The importance of restricting frame activations based on
onset predictions during inference is clear: The second-to-bottom image
(“Estimated Onsets and Notes”) shows the results from the frame and onset
predictors. There are several examples of notes that either last for only a
few frames or that reactivate briefly after being active for a while. Frame
results after being restricted by the onset detector are shown in magenta.
Many of the notes that were active for only a few frames did not have a
corresponding onset detection and were removed, shown in cyan. Cases where a
note briefly reactivated were also removed because a corresponding second
onset was not detected.

Despite not optimizing for inference speed, our network performs $70\times$
faster than real time on a Tesla K40c. The MIDI files resulting from our
inference experiments are available at https://goo.gl/magenta/onsets-frames-
examples.

| Frame | Note | Note w/ offset | Note w/ offset & velocity
---|---|---|---|---
| P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1
Sigtia et al., 2016 [18] | 71.99 | 73.32 | 72.22 | 44.97 | 49.55 | 46.58 | 17.64 | 19.71 | 18.38 | — | — | —
Kelz et al., 2016 [13] | 81.18 | 65.07 | 71.60 | 44.27 | 61.29 | 50.94 | 20.13 | 27.80 | 23.14 | — | — | —
Melodyne (decay mode) | 71.85 | 50.39 | 58.57 | 62.08 | 48.53 | 54.02 | 21.09 | 16.56 | 18.40 | 10.43 | 8.15 | 9.08
Onsets and Frames | 88.53 | 70.89 | 78.30 | 84.24 | 80.67 | 82.29 | 51.32 | 49.31 | 50.22 | 35.52 | 30.80 | 35.39
Table 1: Precision, Recall, and F1 Results on MAPS configuration 2 test
dataset (ENSTDkCl and ENSTDkAm full-length .wav files). Note-based scores
calculated by the _mir_eval_ library, frame-based scores as defined in [2].
Final metric is the mean of scores calculated per piece. MIDI files used to
calculate these scores are available at https://goo.gl/magenta/onsets-frames-
examples.

Figure 2: Inference on 6 seconds of MAPS_MUS-mz_331_3_ENSTDkCl.wav (a
recording which is not in the training set). From top to bottom, we show the
log-magnitude mel-frequency spectrogram input, the framewise note probability
and onset probability “posteriorgrams” produced by our model, the
corresponding estimated onsets and notes after thresholding, and finally the
resulting estimated transcription produced by our model alongside the
reference transcription. In the onset and notes plot (second from the bottom),
onset predictions are shown in black. Notes with a corresponding onset
prediction are shown in magenta and notes which are filtered out because no
onset was predicted for the note are shown in cyan. In the bottom plot, the
estimated transcription is shown in blue and the reference is shown in red.
Figure best viewed in color.

##  5 Ablation Study

To understand the individual importance of each piece in our model, we
conducted an ablation study. We consider removing the onset detector entirely
(i.e., using only the frame detector) (a), not using the onset information
during inference (b), making the bi-directional RNNs uni-directional (c,d),
removing the RNN from the onset detector entirely (e), pre-training the onset
detector rather than jointly training it with the frame detector (f),
weighting all frames equally (g), sharing the convolutional features between
both detectors (h), removing the connection between the onset and frame
detectors during training (i), using a Constant Q-Transform (CQT) input
representation instead of mel-scaled spectrograms (j), and finally removing
all the LSTMs and sharing the convolutional features (k).

These results show the importance of the onset information – not using the
onset information during inference (b) results in a significant 18% relative
decrease in the note onset score and a 31% relative decrease in the note-with-
offset score while increasing the frame score slightly. Despite the increased
frame score, the output sounds significantly worse. To our ears, the decrease
in transcription quality is best reflected by the note-with-offset scores.

The model which does not have the onset detector at all (a) – consisting of
convolutions followed by a bi-directional RNN followed by a frame-wise loss –
does the worst on all metrics, although it still outperforms the baseline
model from [13]. The other ablations indicate a small impact for each
component ($<6\%$). It is encouraging that forward-only RNNs have only a small
accuracy impact as they can be used for online piano transcription.

We tried many other architectures and data augmentation strategies not listed
in the table, none of which resulted in any improvement. Significantly,
augmenting the training audio by adding normalization, reverb, compression,
noise, and synthesizing the training MIDI files with other synthesizers made
no difference. We believe these results indicate a need for a much larger
training dataset of real piano recordings that have fully accurate label
alignments. These requirements are not satisfied by the current MAPS dataset
because only 60 of its 270 recordings are from real pianos, and they are also
not satisfied by MusicNet [23] because its alignments are not fully accurate
(e.g., there is an audible time difference between piano audio and MIDI at
1:24 in sequence 2533). Other approaches, such as seq2seq [20] may not require
fully accurate alignments.

| F1
---|---
| Frame | Note | Note
|  |  | with offset
Onset and Frames | 78.30 | 82.29 | 50.22
(a) Frame-only LSTM | 76.12 | 62.71 | 27.89
(b) No Onset Inference | 78.37 | 67.44 | 34.15
(c) Onset forward LSTM | 75.98 | 80.77 | 46.36
(d) Frame forward LSTM | 76.30 | 82.27 | 49.50
(e) No Onset LSTM | 75.90 | 80.99 | 46.14
(f) Pretrain Onsets | 75.56 | 81.95 | 48.02
(g) No Weighted Loss | 75.54 | 80.07 | 48.55
(h) Shared conv | 76.85 | 81.64 | 43.61
(i) Disconnected Detectors | 73.91 | 82.67 | 44.83
(j) CQT Input | 73.07 | 76.38 | 41.14
(k) No LSTM, shared conv | 67.60 | 75.34 | 37.03
Table 2: Ablation Study Results.

##  6 Need for more data, more rigorous evaluation

The most common dataset for evaluation of piano transcription tasks is the
MAPS dataset, in particular the ENSTDkCl and ENSTDkAm renderings of the MUS
collection of pieces. This set has several desirable properties: the pieces
are real music as opposed to randomly-generated sequences, the pieces are
played on a real physical piano as opposed to a synthesizer, and multiple
recording environments are available (“close” and “ambient” configurations).
The main drawback of this dataset is that it contains only 60 recordings. To
best measure transcription quality, we believe a new and much larger dataset
is needed. However, until that exists, evaluations should make full use of the
data that is currently available.

Many papers, for example [27, 18, 12, 7], further restrict the data used in
evaluation by using only the “close” collection and/or only the first 30
seconds or less of each file. We believe this method results in an evaluation
that is not representative of real-world transcription tasks. For example,
evaluating on only the “close” collection raises our note F1 score from 82.29
to 84.34, and evaluating on only the first 30 seconds further raises it to
86.38. For comparison, [27] achieved a note F1 score of 80.23 in this setting.
The model in [12] is also evaluated using 30s clips from the “close”
collection, but it was additionally trained on data from the test piano. This
method limits the generalizability of the model but produced a note F1 score
of 85.06.

In addition to the small number of the MAPS Disklavier recordings, we have
also noticed several cases where the Disklavier appears to skip some notes
played at low velocity. For example, at the beginning of the Beethoven Sonata
No. 9, 2nd movement, several A$\flat$ notes played with MIDI velocities in the
mid-20s are clearly missing from the audio (https://goo.gl/magenta/onsets-
frames-examples). More analysis is needed to determine how frequently missed
notes occur, but we have noticed that our model performs particularly poorly
on notes with ground truth velocities below 30.

Finally, we believe that more strict metrics should be adopted by the
community. As discussed in Section 2, frame and note onset scores are not
enough to determine whether a transcription has captured all musically
relevant information from a performance. We present several audio examples at
https://goo.gl/magenta/onsets-frames-examples to illustrate this point. Of the
metrics currently available, we believe that the note-with-offset and velocity
is the best way to compare models going forward.

Similarly, the current practice of using a 50ms tolerance for note onset
correctness allows for too much timing jitter. An audio example illustrating
this point is available at the above URL. We suggest future work should
evaluate models with tighter timing requirements. Much work remains to be done
here because as observed in [4], achieving high accuracy is increasingly
difficult as timing precision is increased, in part due to the limited timing
accuracy of the datasets currently available [11]. When we trained our model
at a resolution of 24ms, our scores using the existing 50ms metrics were not
always as high: Frame 76.87, Note F1 82.54, Note-with-offset 49.99. Audio
examples of this higher resolution model are also available at the above URL.
In the examples, the higher time resolution is evident, but the model also
produces more extraneous notes.

##  7 Conclusion and Future Work

We presented a jointly-trained onsets and frames model for transcribing
polyphonic piano music which yields significant improvements by using onset
information. This model transfers well between the disparate train and test
distributions. The current quality of our model’s output is on the cusp of
enabling downstream applications such as symbolic MIR and automatic music
generation. To further improve the results we need to create a new dataset
that is much larger and more representative of various piano recording
environments and music genres for both training and evaluation. Combining an
improved acoustic model with a language model is a natural next step. Another
direction is to go beyond traditional spectrogram representations of audio
signals.

It is very much worth listening to the examples of transcription. Consider
Mozart Sonata K331, 3rd movement. Our system does a good job in terms of
capturing harmony, melody, rhythm, and even dynamics. If we compare this
transcription to the other systems, the difference is quite audible. We have
also successfully used the model to transcribe recordings from the Musopen.org
website that are completely unrelated to our training dataset. The model even
works surprisingly well transcribing a harpsichord recording. Audio examples
are available at https://goo.gl/magenta/onsets-frames-examples.

##  8 Acknowledgements

We would like to thank Hans Bernhard for helping with data collection and
Brian McFee and Justin Salamon for consulting on the velocity metric design.

## References

  * [1] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al.  Tensorflow: Large-scale machine learning on heterogeneous distributed systems.  arXiv preprint arXiv:1603.04467, 2016.
  * [2] Mert Bay, Andreas F Ehmann, and J Stephen Downie.  Evaluation of multiple-f0 estimation and tracking systems.  In ISMIR, pages 315–320, 2009.
  * [3] Emmanouil Benetos, Simon Dixon, Dimitrios Giannoulis, Holger Kirchhoff, and Anssi Klapuri.  Automatic music transcription: challenges and future directions.  Journal of Intelligent Information Systems, 41(3):407–434, 2013\.
  * [4] Sebastian Böck and Markus Schedl.  Polyphonic piano note transcription with recurrent neural networks.  In Acoustics, speech and signal processing (ICASSP), 2012 ieee international conference on, pages 121–124. IEEE, 2012.
  * [5] Judith C Brown.  Calculation of a constant q spectral transform.  The Journal of the Acoustical Society of America, 89(1):425–434, 1991.
  * [6] Tian Cheng, Matthias Mauch, Emmanouil Benetos, Simon Dixon, et al.  An attack/decay model for piano transcription.  In Proceedings of the 17th International Society for Music Information Retrieval Conference, ISMIR 2016, New York City, United States, August 7-11, 2016. ISMIR, 2016.
  * [7] Andrea Cogliati, Zhiyao Duan, and Brendt Wohlberg.  Piano transcription with convolutional sparse lateral inhibition.  IEEE Signal Processing Letters, 24(4):392–396, 2017.
  * [8] Michael Scott Cuthbert and Christopher Ariza.  music21: A toolkit for computer-aided musicology and symbolic music data.  In Proceedings of the 11th International Society for Music Information Retrieval Conference, pages 637–642, 2010.
  * [9] Valentin Emiya, Roland Badeau, and Bertrand David.  Multipitch estimation of piano sounds using a new probabilistic spectral smoothness principle.  IEEE Transactions on Audio, Speech, and Language Processing, 18(6):1643–1654, 2010.
  * [10] Sebastian Ewert and Meinard Müller.  Estimating note intensities in music recordings.  In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 385–388, 2011.
  * [11] Sebastian Ewert and Mark Sandler.  Piano transcription in the studio using an extensible alternating directions framework.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 24(11):1983–1997, 2016.
  * [12] Lufei Gao, Li Su, Yi-Hsuan Yang, and Tan Lee.  Polyphonic piano note transcription with non-negative matrix factorization of differential spectrogram.  In Acoustics, Speech and Signal Processing (ICASSP), 2017 IEEE International Conference on, pages 291–295. IEEE, 2017.
  * [13] Rainer Kelz, Matthias Dorfer, Filip Korzeniowski, Sebastian Böck, Andreas Arzt, and Gerhard Widmer.  On the potential of simple framewise approaches to piano transcription.  arXiv preprint arXiv:1612.05153, 2016.
  * [14] Diederik Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [15] Brian McFee.  librosa: v0.4.3, May 2016.
  * [16] Colin Raffel, Brian McFee, Eric J. Humphrey, Justin Salamon, Oriol Nieto, Dawen Liang, and Daniel P. W. Ellis.  mir_eval: A transparent implementation of common mir metrics.  In In Proceedings of the 15th International Society for Music Information Retrieval Conference, ISMIR. Citeseer, 2014.
  * [17] Mike Schuster and Kuldip K Paliwal.  Bidirectional recurrent neural networks.  IEEE Transactions on Signal Processing, 45(11):2673–2681, 1997\.
  * [18] Siddharth Sigtia, Emmanouil Benetos, and Simon Dixon.  An end-to-end neural network for polyphonic piano music transcription.  IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP), 24(5):927–939, 2016.
  * [19] Paris Smaragdis and Judith C Brown.  Non-negative matrix factorization for polyphonic music transcription.  In Applications of Signal Processing to Audio and Acoustics, 2003 IEEE Workshop on., pages 177–180. IEEE, 2003.
  * [20] Ilya Sutskever, Oriol Vinyals, and Quoc V Le.  Sequence to sequence learning with neural networks.  In Advances in neural information processing systems, pages 3104–3112, 2014.
  * [21] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2818–2826, 2016.
  * [22] Wai Man Szeto, Kin Hong Wong, and Chi Hang Wong.  Finding intensities of individual notes in piano music.  In Computer Music Modeling and Retrieval, 2005.
  * [23] John Thickstun, Zaid Harchaoui, and Sham Kakade.  Learning features of music from scratch.  In International Conference on Learning Representations (ICLR), 2017\.
  * [24] Carl Thomé and Sven Ahlbäck.  Polyphonic pitch detection with convolutional recurrent neural networks.  Music Information Retrieval Evaluation eXchange (MIREX), 2017.
  * [25] Daylin Troxel.  Music transcription with a convolutional neural network 2016.  Music Information Retrieval Evaluation eXchange (MIREX), 2016.
  * [26] Sam Van Herwaarden, Maarten Grachten, and Bas De Haas.  Predicting expressive dynamics in piano performances using neural networks.  In Proceedings of the 15th Conference of the International Society for Music Information Retrieval, pages 45–52, 2014.
  * [27] Qi Wang, Ruohua Zhou, and Yonghong Yan.  A two-stage approach to note-level transcription of a specific piano.  Applied Sciences, 7(9):901, 2017.