L[1]¿m#1 C[1]¿m#1 R[1]¿m#1

# Single-Stage 6D Object Pose Estimation

Yinlin Hu, Pascal Fua, Wei Wang, Mathieu Salzmann
CVLab, EPFL, Switzerland
<EMAIL_ADDRESS>

###### Abstract

Most recent 6D pose estimation frameworks first rely on a deep network to
establish correspondences between 3D object keypoints and 2D image locations
and then use a variant of a RANSAC-based Perspective-n-Point (PnP) algorithm.
This two-stage process, however, is suboptimal: First, it is not end-to-end
trainable. Second, training the deep network relies on a surrogate loss that
does not directly reflect the final 6D pose estimation task.

In this work, we introduce a deep architecture that directly regresses 6D
poses from correspondences. It takes as input a group of candidate
correspondences for each 3D keypoint and accounts for the fact that the order
of the correspondences within each group is irrelevant, while the order of the
groups, that is, of the 3D keypoints, is fixed. Our architecture is generic
and can thus be exploited in conjunction with existing correspondence-
extraction networks so as to yield single-stage 6D pose estimation frameworks.
Our experiments demonstrate that these single-stage frameworks consistently
outperform their two-stage counterparts in terms of both accuracy and speed.

##  1 Introduction

Detecting 3D objects in images and computing their 6D pose must be addressed
in a wide range of applications [11, 31, 48, 29], ranging from robotics to
augmented reality. State-of-the-art approaches [39, 41, 32, 16, 13, 36, 54,
34, 24] follow a two-stage paradigm: First use a deep network to establish
correspondences between 3D object points and their 2D image projections, then
use a RANSAC-based Perspective-n-Point (PnP) algorithm to compute the 6 pose
parameters [9, 20, 40, 47, 21, 18, 7, 46].

While effective, this paradigm suffers from several weaknesses. First, the
loss function used to train the deep network does not reflect the true goal of
pose estimation, but encodes a surrogate task, such as minimizing the 2D
errors of the detected image projections. The relationship between such errors
and the pose accuracy, however, is not one-to-one. As shown in Fig. 1 (a) for
the state-of-the-art framework of [13], two sets of correspondences with the
same average 2D error can result in different pose estimates. Second, the two-
stage process is not end-to-end trainable. Finally, the iterative RANSAC is
time-consuming when there are many correspondences that need to be handled.

|
---|---
(a) | (b)
Figure 1: Motivation. Consider the modern 6D pose estimation algorithm of [13]
that uses a deep network to predict several 2D correspondences for each of the
eight 3D corners of the pitcher’s bounding box. (a) Because it minimizes the
average 2D error of these correspondences, two instances of such a framework
could produce correspondences that differ but have the same average accuracy,
such as the green and the red ones. As evidenced by the projected green and
red reference frames, applying a RANSAC-based PnP algorithm to these two sets
of correspondences can yield substantially different poses. (b) Even when
using only the set of green correspondences, simply changing their order
causes a RANSAC-based PnP algorithm to return different solutions.

In principle, an end-to-end framework could be designed by exploiting a deep
version of RANSAC [1, 2], followed by another network performing pose
estimation from correspondences [5]. However, the time-consuming character of
RANSAC in the presence of many outliers, and the poor repeatability of its
solution, arising from the fact that, as shown in Fig. 1 (b), the order of the
correspondences affects the resulting pose, do not make it a good candidate
for inclusion into an end-to-end trainable network. Furthermore, the approach
of [5] relies on using a Direct Linear Transform (DLT) [9] to compute the
pose, which is known to be imprecise and would exacerbate in the poor
repeatability problem.

As a result, there are still no end-to-end frameworks that can handle jointly
keypoint localization and 6D pose estimation. In this paper, we overcome this
by introducing a simple but effective network that directly regresses the 6D
pose from groups of 3D-to-2D correspondences associated to each 3D object
keypoint. Its architecture explicitly encodes that the order of the
correspondences in each group is irrelevant, while exploiting the fact that
the order of the groups is fixed and corresponds to that of the 3D keypoints.

We then demonstrate the generality of this network by combining it with two
state-of-the-art correspondence-extraction frameworks [13, 36]. This yields
end-to-end trainable 6D pose estimation frameworks that are both accurate and
repeatable. We show that these single-stage frameworks systematically
outperform the original two-stage ones [13, 36], in terms of both accuracy and
runtime.

##  2 Related Work

Detecting keypoints in the input image followed by running a RANSAC-based PnP
algorithm on the established 3D-to-2D correspondences is a classical way for
solving the 6D object pose estimation problem. Over the years, many methods
have been proposed to improve 3D-to-2D matching [28, 42, 43, 44, 35, 33],
relying on diverse techniques, such as template-matching [10, 11], edge-
matching [22, 27], and 3D model-based matching [14, 26, 12]. However, these
traditional methods still often fail in the presence of severe occlusions and
cluttered background.

As in many other areas, the modern take on 6D object pose estimation from an
RGB image involves deep neural networks. The simplest approach is to directly
regress from the image to the pose parameters [17, 50]. However, this tends to
be less accurate than first establishing 3D-to-2D correspondences [39, 41, 32,
16, 13, 36, 54, 34, 24] and then running a RANSAC-based Perspective-n-Point
(PnP) algorithm [9] to estimate the object position and orientation given the
camera intrinsic parameters. What these methods all have in common is that the
correspondences are established independently from each other and consistency
is only imposed after the fact by the RANSAC PnP algorithm, which is not part
of the deep network. As shown in [53], albeit in a different context, this
fails to exploit the fact that all correspondences are constrained by the
camera pose and are therefore not independent from each other.

Our goal in this paper is to turn the two-stage process described above into a
single-stage one by implementing the RANSAC-based PnP part of the process as a
deep network that can be combined with the one that establishes the
correspondences. This is not a trivial problem because the standard approach
to PnP involves performing a Singular Value Decomposition (SVD), which can be
embedded in a deep network but often results in numerical instabilities. In
[5], this was addressed by avoiding the explicit use of SVD and instead
treating PnP as a least-square fitting problem via the Direct Linear Transform
(DLT) approach [9]. This, however, does not guarantee that the result
describes a true rotation and further post processing is still needed.

By contrast, the backpropagation-friendly eigendecomposition method of [49]
performs explicit SVD, and could in principle used to perform PnP. Doing so,
however, would fail to account for the RANSAC part of the algorithm to select
the correct correspondences. While RANSAC can be implemented via a deep
network [1, 2], its poor repeatability, evidenced in Fig. 1(b), makes it ill-
suited to train an end-to-end 6D pose estimation network. In short, no one yet
has proposed a satisfying solution to designing a single-stage 6D pose
estimation network, which is the problem we address here.

Our architecture is inspired by PointNet [37, 38]. However, PointNet was
designed to deliver invariance to rigid transformations, which is the opposite
of what we need. Furthermore, we introduce a grouped feature aggregation
scheme to effectively hande correspondence clusters in 6D object pose
estimation.

##  3 Approach

Given an RGB image captured by a calibrated camera, our goal is to
simultaneously detect objects and estimate their 6D pose. We assume them to be
rigid and their 3D model to be available. In this section, we first formalize
the 6D pose estimation problem assuming that sets of 2D correspondence are
given a priori for each 3D keypoint on the target object and propose a network
architecture that yields 6D poses from such inputs. This network is depicted
by Fig. 3. We then discuss how to obtain a single-stage 6D pose estimation
framework when these correspondences are the output of another network.

###  3.1 6D Pose from Correspondence Clusters

Let us assume that we are given the $3\times 3$ camera intrinsic parameter
matrix ${\bf K}$ and $m$ potential 2D correspondences $\mathbf{u}_{ik}$ for
each one of $n$ 3D object keypoints ${\bf p}_{i}$, with $1\leq i\leq n$ and
$1\leq k\leq m$. The ${\bf p}_{i}$ is expressed in a coordinate system linked
to the object, as shown in Fig. 2(a). For each valid 3D to 2D correspondence
${\bf p}_{i}\leftrightarrow\mathbf{u}_{ik}$, we have

$\displaystyle\lambda_{ik}\begin{bmatrix}{\bf u}_{ik}\\\ 1\\\
\end{bmatrix}=\mathbf{K}(\mathbf{R}{\bf p}_{i}+{\bf t}),$ (1)

where $\lambda_{i}$ is a scale factor, and $\mathbf{R}$ and ${\bf t}$ are the
rotation matrix and translation vector that define the camera pose. Because
${\bf R}$ is a rotation, it only has three degrees of freedom and ${\bf t}$
likewise, for a total of 6.

Note that the 3D-to-2D correspondences above are not restricted to 3D point to
2D point correspondences. In particular, as shown in Fig. 2(b), our formalism
can handle 3D point to 2D _vector_ correspondences, which have been shown to
be better-suited to use in conjunction with a deep network [36]. In that case,
the 2D locations can be infered as the crosspoint of two 2D vectors, and Eq. 1
still holds on crosspoints. Our approach as discussed below also still
applies, and we therefore do not explicitly distinguish between these two
types of 3D-to-2D correspondences unless necessary.

Classical PnP methods [21, 7, 46] try to recover $\bf R$ and ${\bf t}$ given
several correspondences, which typically involves using RANSAC to find the
valid ones. In the process, an SVD has to be performed on the many randomly
chosen subsets of correspondences that must be tried before one containing
only valid correspondences is found. In this work, we propose to replace this
cumbersome process by a non-linear regression implemented by an appropriately
designed deep network $g$ with parameters $\Theta$. In other words, we have

$\displaystyle({\bf R},{\bf t})$ $\displaystyle=g(\\{({\bf
p}_{i}\leftrightarrow{\bf u}_{ik})\\}_{1\leq i\leq n,1\leq k\leq
m};\Theta)\;.$ (2)

We now turn to the actual implementation of $g_{\theta}$. In the remainder of
this section, we first discuss the properties of the set of 3D to 2D
correspondences ${\cal C}_{2}^{3}=\\{({\bf p}_{i}\leftrightarrow{\bf
u}_{ik})\\}_{1\leq i\leq n,1\leq k\leq m}$ that the network takes as input and
then the architecture we designed to account for them.

####  3.1.1 Properties of the Correspondence Set

We will refer to all the 2D points associated to a specific 3D point as a
cluster because, assuming that the algorithm used to find them is a good one,
they tend to cluster around the true location of the 3D point’s projection, as
can be seen in Fig. 1. Our implementation choice were driven by the following
considerations:

Cluster ordering. The order of the correspondences within a cluster is
irrelevant and should not affect the result. However, the order of the
clusters corresponds to the order of the 3D points, which is given and fixed.

Interaction within a cluster and across clusters. Although the points in the
same cluster correspond to the same 3D point, the 2D location estimate for
each point should be expected to be noisy. Thus the model needs to capture the
noise distribution within each cluster. More importantly, one single cluster
can tell us nothing about the pose, and the final pose can only be inferred by
capturing the global structure for multiple clusters.

Rigid transformations matter. When processing 3D point clouds with a deep
network, one usually wants the result to be invariant to rigid
transformations. By contrast, here, we want our 2D points to represent
projections of 3D points, and the features that we extract from them should
depend on their absolute positions, which are critical to pose estimation.

|
---|---
(a) | (b)
Figure 2: 3D to 2D correspondences. (a) Given $m$ potential 2D correspondences
$\mathbf{u}_{ik}$ for each one of $n$ 3D object keypoints ${\bf p}_{i}$,
$\\{({\bf p}_{i}\leftrightarrow\mathbf{u}_{ik})\\}_{1\leq i\leq n,1\leq k\leq
m}$, the pose can be computed based on these 3D-to-2D correspondences. Here,
we only show the correspondence cluster for ${\bf p}_{i}$. The camera and
object coordinate systems are denoted by $O$ and $W$ respectively. (b) The
pose can also be obtained from point-to-vector correspondences, in which case
a 3D-to-2D correspondence is defined between a 3D point and a 2D vector. Our
method can handle both cases.  Figure 3: Overall architecture for single-stage
6D object pose estimation. After establishing 3D-to-2D correspondences by some
segmentation-driven CNN for 6D pose [13, 36], we use three main modules to
infer the pose from these correspondence clusters directly: a local feature
extraction module with shared network parameters, a feature aggregation module
operating within the different clusters, and a global inference module
consisting of simple fully-connected layers to estimate the final pose as a
quaternion and a translation. The color in the CNN outputs indicates the
direction of the 2D offset from the grid cell center to the corresponding
projected 3D bounding box corner.

####  3.1.2 Network Architecture

We construct a simple network architecture, depicted by Fig. 3, that utilizes
the properties discussed above to predict the pose from correspondence
clusters. It comprises three main modules: A local feature extraction module
with shared network parameters, a feature aggregation module within individual
clusters, and a global inference module made of simple fully-connected layers.

Local feature extraction. We use an MLP with three layers to extract local
features for each correspondence, with weights shared across the
correspondences and across the clusters.

Grouped feature aggregation. As the order of the clusters is given but the
points within each cluster are orderless, to extract the representation for
each cluster, we design a grouped feature aggregation method that insensitive
to the correspondence order. In theory, we could have used an architecture
similar to that of PointNet [37, 38]. However, PointNet is designed to deliver
invariance to rigid transformations, which is the opposite of what we need.
Instead, given $n$ clusters, each containing $m$ 2D points $\\{{\bf
u}_{ik}\\},1\leq i\leq n,1\leq k\leq m$, we define a set function
$\mathcal{F}:\mathcal{X}\rightarrow\mathbb{R}^{nD}$ that maps correspondences
$\\{{\bf u}_{ik}\\}_{1\leq k\leq m}$ to the $nD$-dimensional vector

$\begin{split}\displaystyle CAT\Big{(}\underset{k}{MAX}(\\{{\bf
f}_{1k}\\}),\underset{k}{MAX}(\\{{\bf f}_{2k}\\}),..,\underset{k}{MAX}(\\{{\bf
f}_{nk}\\})\Big{)},\end{split}$ (3)

where ${\bf f}_{ik}$ is the $D$-dimensional feature representation of ${\bf
u}_{ik}$ obtained via the above-mentioned fully-connected layers, $MAX()$ is
the max-pooling operation and $CAT()$ is the concatenation operation. In our
experiments, we found that neither instance normalization [45, 52] nor batch
normalization [15] improved the performance here. Therefore, we do not use
these operations in our network $g_{\theta}$.

In principle, one could use a single max-pooling operation, without accounting
for the order of the groups, just as PointNet [37] does to achieve permutation
invariance for all points. In our case, however, this would mean ignoring the
property that the order of the groups is fixed. By contrast, Eq. 3 is
invariant to any permutation within a cluster but still accounts for the pre-
defined cluster order. We demonstrate the benefits of this approach in the
results section.

Global inference. We then pass the $nD$-dimensional vector aggregating the
group features through another MLP which outputs the 6D pose. To this end, we
use three fully-connected layers and encode the final pose as a quaternion and
a translation.

###  3.2 Single-Stage 6D Object Pose Estimation

The deep network described above gives us a differentiable way to predict the
6D pose from correspondence clusters for a given object. Given the input
image, we therefore still need to detect each object and establish the 3D to
2D correspondences. To do so, we use another deep regressor $f$ with
parameters $\Phi$, which, for one object, lets us write

$\left[{\bf u}_{i1},\ldots,{\bf u}_{im}\right]=f({\bf p}_{i},{\bf
I};\Phi),\quad 1\leq i\leq n$ (4)

where ${\bf I}$ is the input RGB image. To implement $f$, we use the recent
encoder-decoder architecture of either [13] or [36].

In practice, the $\\{{\bf p}_{i}\\}$ are often taken to be the eight corners
of the 3D bounding box of the object’s 3D model [39, 32, 13], which leads to
different 3D points $\\{{\bf p}_{i}\\}$, for different object types. In our
experiments, we have observed that using the same $\\{{\bf p}_{i}\\}$ for
every object has little impact on the accuracy of $f_{\phi}$ and makes the
subsequent training of $g_{\theta}$ much easier. We therefore use a single
cube for all dataset objects, defined as the largest cube contained by a
sphere whose radius is the average of that of the bounding spheres of all
object 3D models. This means that the 3D keypoint coordinates are implicitly
given by the order of the clusters and do not need to be explicitly specified
as network inputs. We therefore a use of 4D representation for each input
correspondence, which does not include the 3D coordinates. Instead, because
the network of [13] operates on an image grid, when we use it to find the
correspondences, we take the input to be the $x$ and $y$ coordinates of the
center of the grid cell in which the 2D projections are and the $dx$ and $dy$
offsets from that center. In other words, the image coordinates of a 2D
correspondence are $x+dx$ and $y+dy$. We tried using these directly as input
but we found out experimentally that giving the network what amounts to a
first order expansion works better. When using the network of [36] instead of
that of [13] to find the correspondences, we use the same input format but
normalize the $dx$ and $dy$ so that they represent an orientation.

Our complete model can therefore be written as

$\displaystyle({\bf R},{\bf t})$ $\displaystyle=g\big{(}f({\bf p}_{1},{\bf
I};\Phi),\cdots,f({\bf p}_{8},{\bf I};\Phi);\Theta\big{)}\;.$ (5)

To train it, we minimize the loss function

${\mathcal{L}}={\mathcal{L}}_{s}+{\mathcal{L}}_{k}+{\mathcal{L}}_{p}\;,$ (6)

which combines segmentation term ${\mathcal{L}}_{s}$ aiming to assign each
grid cell to an object class of to the background, a keypoint regression term
${\mathcal{L}}_{k}$, and a pose estimation term ${\mathcal{L}}_{p}$. We take
${\mathcal{L}}_{s}$ to be the Focal Loss of [25], and ${\mathcal{L}}_{k}$ to
be the regression term of either [13] or [36] depending on which of the two
architectures we use. As in [50, 23], we take ${\mathcal{L}}_{p}$ to be the 3D
space reconstruction error, that is

${\mathcal{L}}_{p}=\frac{1}{n}\sum_{i=1}^{n}\|(\hat{\bf R}{\bf p}_{i}+\hat{\bf
t})-({\bf R}{\bf p}_{i}+{\bf t})\|\;,$ (7)

where $\hat{\bf R}$ and $\hat{\bf t}$ are the estimated rotation matrix and
translation vector, ${\bf R}$ and ${\bf t}$ are the ground-truth ones. The
rotations are estimated from the estimated and ground-truth quaternions, which
can be done in a differentiable manner [55]. We also normalize the
translations to make sure the regression targets all have a comparable range.

Our architecture simultaneously outputs a segmentation mask and potential 2D
locations for a set of predefined 3D keypoints. More specifically, for a
dataset with $S$ object classes and an input image ${\bf I}$ of size $h\times
w\times 3$, it outputs a 3D tensor of size $H\times W\times C$. The dimensions
$H$ and $W$ are proportional to the input resolution and $C=(S+1)+2*n$ with
$(S+1)$ channels for segmentation, including one for the background class, and
$2*n$ for the 2D locations (or 2D direction vectors) corresponding to the $n$
3D points ${\bf p}_{i}$. To obtain correspondence clusters for a given object,
we randomly sample $m=200$ grid cells on the output feature tensor that fall
under the segmentation mask of a particular class label.

##  4 Experiments

Figure 4: Synthetic data. We create synthetic data by randomly changing the
pose of a unit sphere in 3D space relative to the camera. We capture 20K
images for training and 2K for testing.

We compare our single-stage approach to more traditional but state-of-the-art
two-stage frameworks [13, 36], first on synthetic data and then on real data
from the challenging Occluded-LINEMOD [19] and YCB-Video [50] datasets. Our
source code is publicly available at https://github.com/cvlab-epfl/single-
stage-pose.

###  4.1 Synthetic Data

Figure 5: Generating correspondences. We project each corner of the sphere’s
3D bounding box in the image and, for each grid cell within the object mask,
create a correspondence by recording the center $x,y$ of the grid cell and the
offset $dx,dy$ to the projected corner.

As in [21, 7], we create synthetic 3D-to-2D correspondences using a virtual
calibrated camera, with image size $640\times 480$, focal length $800$, and
principal point at the image center. We take our target object to be a unit 3D
sphere, which we randomly rotate and whose center we randomly translate within
the interval $[-2,2]\times[-2,2]\times[4,8]$ expressed in the camera
coordinate system, as shown in Fig. 4.

Recall from Section 3.2, that $g_{\theta}$, the network that regresses poses
from the correspondence clusters, expects 4D inputs in the form $[x,y,dx,dy]$,
where $x,y$ represent the center of an image grid location and $dx,dy$ a shift
from that center. Here, each one should represent a potential image
correspondence for a specific corner of the sphere’s bounding box for a
particular object. Given the segmentation mask of a particular object obtained
by projecting the object’s 3D model in the image, we create correspondences in
the following manner. We project each corner of the sphere’s 3D bounding box
in the image and, for each grid cell in the segmentation mask, record the cell
center $x,y$ and the displacement $dx,dy$ to the projected corner. We then
take the resulting correspondences from 200 randomly sampled grid cells within
the mask. We add Gaussian noise to their $dx,dy$ values as well as create
outliers by setting some percentage of the $dx,dy$ to values uniformly sampled
in the image. Fig. 5 demonstrates this procedure.

We trained $g_{\theta}$ for 300 epoch on 20K synthetic training images with
batch size 32, and a learning rate of 1e-3 using the Adam optimizer. During
training we randomly add 2D noise with variance $\sigma$ in the range of [0,
15] and create from 0% to 30% of outliers. To test the accuracy obtained with
different noise levels and outlier rates, we use 2K synthetic test images and
report the mean pose accuracy in terms of the ratio of the 3D space
reconstruction error of Eq. 7 to the diameter of the target object.

Comparing with RANSAC PnP. Combining a PnP algorithm with RANSAC is the most
widespread approach to handling noisy correspondences [39, 41, 13, 54]. Fig. 6
shows that RANSAC-based EPnP [21] and RANSAC-based P3P [8] yield similar
performance. While they are more accurate than our learning-based method when
there is very little noise, our method quickly becomes much more accurate when
the noise level increases.

Importance of correspondence clustering. To showcase the importance to
structure our network in the way we did, we implemented a simplified version
that uses a single max-pooling operation to achieve permutation invariance for
all correspondences, without accounting for the order of the clusters that
matches that of the keypoints. To make this work, we had to incorporate
explicitly the 3D keypoint coordinates associated to each correspondence as
input to the network. As shown in Fig. 7, not modeling the fixed order of the
keypoints yields a significant decreases in accuracy.

Comparing with PVNet’s voting-based PnP. In the above experiments, the 2D
correspondences were expressed in terms of 2D locations of image points. Since
one of the best current techniques [36] uses directions instead and infers
poses from those using a voting-based PnP scheme, we feed the same 3D point to
2D vector correspondences to our own network. In this setting, as shown in
Fig. 8, the pose is more sensitive to the correspondence noise. However, as in
the previous case, while voting-based PnP yields more accurate results when
there is little noise, our method is much more robust and accurate when the
noise level increases.

Figure 6:  Comparison with RANSAC PnP. We compare our network with two
classical RANSAC-based PnP methods, EPnP [21] and P3P [8]. The two RANSAC-
based methods have very similar performance. More importantly, our method is
much more accurate and robust when the noise increases. The pose error is
reported as the ratio of the 3D space reconstruction error to the diameter of
the target object.

Figure 7: Importance of correspondence clustering. We compare our network with
one having a single max-pooling operation, thus not accounting for the order
of the clusters. Ignoring this property clearly degrades the performance.

###  4.2 Real Data

We evaluate our method on real data from two challenging datasets, Occluded-
LINEMOD [19] and YCB-Video [50].

Occluded-LINEMOD consists of 8 objects and is a subset of the older LINEMOD
dataset [11]. Unlike LINEMOD in which only one object per image is annotated,
Occluded-LINEMOD features multiple annotated objects. This makes it more
meaningful for evaluating methods that perform both instance detection and
pose estimation. In addition to the cluttered backgrounds, textureless
objects, and changing lighting conditions of LINEMOD, Occluded-LINEMOD also
has severe occlusions between multiple object instances. As there are only
1214 testing images and no explicit training data in Occluded-LINEMOD, we
train our network based on the LINEMOD training data.

YCB-Video is more recent and even more challenging. It features 21 objects
taken from the YCB dataset [4, 3] and comprises about 130K real images from 92
video sequences. It offers all the challenges of Occluded-LINEMOD plus more
diverse object sizes, including several tiny textures-less objects.

Figure 8: Comparison with PVNet’s voting-based PnP [36]. When using 3D point to 2D vector correspondences, we compare our network with the voting-based PnP used by PVNet. Our method is much more robust to noise than voting-based PnP.  | [13] |  [13] \+ Ours |  [36] |  [36] \+ Ours
---|---|---|---|---
Ape | 12.1 | 14.8 |  15.8 | 19.2
Can | 39.9 | 45.5 |  63.3 | 65.1
Cat | 8.2 | 12.1 |  16.7 | 18.9
Driller | 45.2 | 54.6 |  65.7 | 69.0
Duck | 17.2 | 18.3 |  25.2 | 25.3
Eggbox∗ | 22.1 | 30.2 |  50.2 | 52.0
Glue∗ | 35.8 | 45.8 |  49.6 | 51.4
Holepun. | 36.0 | 37.4 |  39.7 | 45.6
Average | 27.0 | 32.3 |  40.8 | 43.3
Table 1: Evaluation with different correspondence-extraction networks on Occluded-LINEMOD. We evaluate two state-of-the-art correspondence-extraction networks: SegDriven [13] and PVNet [36], by replacing their original RANSAC-based post processing with our small network. Our method consistently outperforms the original versions in both cases. Here, we report the ADD-0.1d.  | ADD-0.1d | REP-5px
---|---|---
| PoseCNN | SegDriven | PVNet | Ours |  PoseCNN | SegDriven | PVNet | Ours
Ape | 9.6 | 12.1 | 15.8 | 19.2 |  34.6 | 59.1 | 69.1 | 70.3
Can | 45.2 | 39.9 | 63.3 | 65.1 |  15.1 | 59.8 | 86.1 | 85.2
Cat | 0.9 | 8.2 | 16.7 | 18.9 |  10.4 | 46.9 | 65.1 | 67.2
Driller | 41.4 | 45.2 | 65.7 | 69.0 |  7.4 | 59.0 | 73.1 | 71.8
Duck | 19.6 | 17.2 | 25.2 | 25.3 |  31.8 | 42.6 | 61.4 | 63.6
Eggbox∗ | 22.0 | 22.1 | 50.2 | 52.0 |  1.9 | 11.9 | 8.4 | 12.7
Glue∗ | 38.5 | 35.8 | 49.6 | 51.4 |  13.8 | 16.5 | 55.4 | 56.5
Holepun. | 22.1 | 36.0 | 39.7 | 45.6 |  23.1 | 63.6 | 69.8 | 71.0
Average | 24.9 | 27.0 | 40.8 | 43.3 |  17.2 | 44.9 | 61.1 | 62.3
Table 2: Comparison with the state of the art on Occluded-LINEMOD. We compare
our results with those of PoseCNN [50], SegDriven [13], and PVNet [36] in
terms of both ADD-0.1d and REP-5px. Our method outperforms the state of the
art, especially in ADD-0.1d.

Figure 9: Qualitative results on Occluded-LINEMOD. Our method yields accurate
results even in the presence of large occlusions, as shown in the first three
columns. The last column shows two failure cases, where the target egg box is
occluded too much and the target glue exhibits subtle symmetry ambiguities,
making it not easy for the correspondence-extraction network [36] to establish
stable correspondences. Here, the pose is visualized as the reprojection of
the 3D mesh for each object.

| correspondence
extraction | fusion | total time | FPS
---|---|---|---|---
PoseCNN | - | - |  $>$250 |  $<$4
SegDriven | 30 | 20 | 50 | 20
PVNet | 14 | 26 | 40 | 25
Ours | 14 | 8 | 22 | 45

Table 3: Comparing speed. We compare the running times (in milliseconds) of PoseCNN [50], SegDriven [13], PVNet [36] and our method on a modern GPU (GTX1080 Ti). Except for PoseCNN, these methods first extract correspondences and then fuse them. With the same correspondence-extraction backbone as in PVNet, our method runs about 2 times faster, thanks to our network that prevents the need for RANSAC-based fusion.  | ADD-0.1d | REP-5px
---|---|---
PoseCNN | 21.3 | 3.7
SegDriven | 39.0 | 30.8
PVNet | - | 47.4
Ours | 53.9 | 48.7
Table 4: Comparison with the state of the art on YCB-Video. We compare our
results with those of PoseCNN [50], SegDriven [13], and PVNet [36] in terms of
ADD-0.1d and REP-5px. We denote by “-” the result missing from the original
PVNet paper.

##### Data preparation.

For Occluded-LINEMOD, as in [41, 13, 36], we first use the Cut-and-Paste
synthetic technique [6] to generate 20K images from LINEMOD data and random
background data [51], with 4 to 10 different instances for each image. Then,
we generate 10K rendering images for each object type from the textured 3D
mesh, as in [36]. The pose range during the rendering procedure is the same as
in LINEMOD except for one thing: To handle pose ambiguities when encountering
symmetry objects [30], we restrict the pose range to a subrange according to
the symmetry type of the object during training to avoid confusing the network
[39]. In the end, our training data consists of 20K synthetic images with
multiple instances and 10K rendered images with only one instance for each
object, a total of ($20+10\times 8$)K images.

For YCB-Video, we follow a similar procedure. We render 10K images for each of
the 21 objects using the 3D mesh models that are provided and according to the
pose statistic of the dataset. However, we do not use the Cut-and-Paste
technique to generate images with multiple instances because in the original
YCB-Video images are already annotated with multiple objects and we use that
directly.

##### Training Procedure.

For both datasets, we use an input image resolution of 640 $\times$ 480 for
both training and testing, as in [36]. We use Adam to optimize with the
initial learning rate set to 1e-3 and divided by 10 after processing 50%, 75%,
and 90% of the total number of data samples. We set the batch size to be 8 and
rely on the usual data augmentation techniques, that is as random luminance,
Gaussian noise, translation, scaling, and also occlusions [56]. We train the
network on 5M training samples through online data augmentation.

##### Metrics.

We quantify the pose error in both 3D and 2D as in [50, 13]. In 3D, it use the
average distance between the 3D model points transformed using the predicted
pose and those obtained with the ground-truth one, and we refer to it as ADD
[50]. In 2D, we use the usual 2D reprojection error of the 3D model points,
and we refer it as REP [13]. We measure the pose accuracy in terms the
percentage of recovered poses that are correct. In the tables below, we report
ADD-0.1d and REP-5px, for which the predicted pose are considered to be
correct if ADD is smaller than 10% of the model diameter and REP is below 5
pixel, respectively. For each metric, we use the symmetric version for
symmetric objects, which we denote by a ∗ superscript.

####  4.2.1 Occluded-LINEMOD Results

As discussed before, to demonstrate that our method is generic, we test it in
conjunction with two correspondence-extraction networks SegDriven [13] and
PVNet [36]. Table 1 shows that, by replacing the original RANSAC-based post
processing by our network to turn the approach into a single-stage one we
improve performance in both cases.

In Table 2, we shown that our single-stage network outperform the state-of-
the-art methods, PoseCNN [50], SegDriven [13] and PVNet [36]. Fig. 9 provides
qualitative results. In Table 3, we report runtimes for an input image
containing about 4-5 objects. Our method is also faster than the others
because it does away for the iterative RANSAC procedure.

####  4.2.2 YCB-Video Results

Table 4 summarizes the results comparing against PoseCNN [50], SegDriven [13],
and PVNet [36]. It shows that our method consistently also outperforms the
others on this dataset. Furthermore, note that it runs nearly 10 times faster
than PoseCNN and also nearly 2 times faster than SegDriven and PVNet.

###  4.3 Limitations

While our method is accurate and fast when used in conjunction with state-of-
the-art correspondence-extraction networks [13, 36], the network that
estimates the poses from the correspondences is still not as accurate as
traditional geometry-based PnP algorithms when very precise correspondences
can be obtained by other means, as shown in Fig. 6. Furthermore, it does not
address the generic PnP problem because we only trained it for fixed sets of
3D coordinates. Addressing this will be the focus of our future work.

##  5 Conclusion

We have introduced a single-stage approach approach to 6D detection and pose
estimation. Its key ingredient is a small network that takes candidate
3D-to-2D correspondences and returns a 6D pose. When combined with state-of-
the-art approaches to establish the correspondences, it boosts performance by
allowing end-to-end training and eliminating the cumbersome RANSAC style
procedure that they normally require.

Future work will focus on making the pose estimation network more accurate and
more generic so that it can be used in a broader context.

#### Acknowledgments

This work was supported in part by the Swiss Innovation Agency (Innosuisse).
We would like to thank Vincent Lepetit, Kwang Moo Yi and Eduard Trulls for
helpful discussions.

## References

  * [1] Eric Brachmann, Alexander Krull, Sebastian Nowozin, Jamie Shotton, Frank Michel, Stefan Gumhold, and Carsten Rother.  DSAC – Differentiable RANSAC for Camera Localization.  Conference on Computer Vision and Pattern Recognition, 2017.
  * [2] Eric Brachmann and Carsten Rother.  Neural-Guided RANSAC: Learning Where to Sample Model Hypotheses.  In International Conference on Computer Vision, 2019.
  * [3] Berk Calli, Arjun Singh, James Bruce, Aaron Walsman, Kurt Konolige, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M Dollar.  Yale-Cmu-Berkeley Dataset for Robotic Manipulation Research.  In International Journal of Robotics Research, 2017.
  * [4] Berk Calli, Arjun Singh, Aaron Walsman, Siddhartha Srinivasa, Pieter Abbeel, and Aaron Dollar.  The YCB Object and Model Set: Towards Common Benchmarks for Manipulation Research.  In International Conference on Advanced Robotics, 2015.
  * [5] Zheng Dang, Kwang Moo Yi, Yinlin Hu, Fei Wang, Pascal Fua, and Mathieu Salzmann.  Eigendecomposition-Free Training of Deep Networks with Zero Eigenvalue-Based Losses.  In European Conference on Computer Vision, 2018.
  * [6] Debidatta Dwibedi, Ishan Misra, and Martial Hebert.  Cut, Paste and Learn: Surprisingly Easy Synthesis for Instance Detection.  In International Conference on Computer Vision, 2017.
  * [7] Luis Ferraz, Xavier Binefa, and Francesc Moreno-Noguer.  Very Fast Solution to the PnP Problem with Algebraic Outlier Rejection.  In Conference on Computer Vision and Pattern Recognition, pages 501–508, 2014.
  * [8] Xiao-Shan Gao, Xiao-Rong Hou, Jianliang Tang, and Hang-Fei Cheng.  Complete Solution Classification for the Perspective-Three-Point Problem.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(8):930–943, 2003.
  * [9] Richard Hartley and Andrew Zisserman.  Multiple View Geometry in Computer Vision.  Cambridge University Press, 2000.
  * [10] Stefan Hinterstoißer, Cedric Cagniart, Slobodan Ilic, Peter F. Sturm, Nassir Navab, Pascal Fua, and Vincent Lepetit.  Gradient Response Maps for Real-Time Detection of Textureless Objects.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 34, May 2012.
  * [11] Stefan Hinterstoißer, Vincent Lepetit, Slobodan Ilic, Stefan Holzer, Gary R. Bradski, Kurt Konolige, and Nassir Navab.  Model Based Training, Detection and Pose Estimation of Texture-Less 3D Objects in Heavily Cluttered Scenes.  In Asian Conference on Computer Vision, 2012.
  * [12] Edward Hsiao, Sudipta N. Sinha, Krishnan Ramnath, Simon Baker, C. Lawrence Zitnick, and Richard Szeliski.  Car Make and Model Recognition Using 3D Curve Alignment.  In IEEE Winter Conference on Applications of Computer Vision, 2014\.
  * [13] Yinlin Hu, Joachim Hugonot, Pascal Fua, and Mathieu Salzmann.  Segmentation-Driven 6D Object Pose Estimation.  In Conference on Computer Vision and Pattern Recognition, 2019.
  * [14] Daniel P. Huttenlocher, Gregory A. Klanderman, and William Rucklidge.  Comparing Images Using the Hausdorff Distance.  IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 850–863, 1993.
  * [15] Sergey Ioffe and Christian Szegedy.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift.  In International Conference on Machine Learning, 2015.
  * [16] Omid Hosseini Jafari, Siva Karthik Mustikovela, Karl Pertsch, Eric Brachmann, and Carsten Rother.  Ipose: Instance-Aware 6D Pose Estimation of Partly Occluded Objects.  In Asian Conference on Computer Vision, 2018.
  * [17] Wadim Kehl, Fabian Manhardt, Federico Tombari, Slobodan Ilic, and Nassir Navab.  SSD-6D: Making Rgb-Based 3D Detection and 6D Pose Estimation Great Again.  In International Conference on Computer Vision, 2017.
  * [18] Laurent Kneip, Hongdong Li, and Yongduek Seo.  UPnP: An Optimal O(n) Solution to the Absolute Pose Problem with Universal Applicability.  In European Conference on Computer Vision, 2014.
  * [19] Alexander Krull, Eric Brachmann, Frank Michel, Michael Ying Yang, Stefan Gumhold, and Carsten Rother.  Learning Analysis-By-Synthesis for 6D Pose Estimation in RGB-D Images.  In International Conference on Computer Vision, 2015.
  * [20] Vincent Lepetit and Pascal Fua.  Monocular Model-Based 3D Tracking of Rigid Objects: A Survey.  Now Publishers, September 2005.
  * [21] Vincent Lepetit, Francesc Moreno-Noguer, and Pascal Fua.  EPnP: An Accurate O(n) Solution to the PnP Problem.  International Journal of Computer Vision, 2009.
  * [22] Dengwang Li, Hongjun Wang, Yong Yin, and Xiuying Wang.  Deformable Registration Using Edge-preserving Scale Space for Adaptive Image-guided Radiation Therapy.  In Journal of Applied Clinical Medical Physics, 2011.
  * [23] Yi Li, Gu Wang, Xiangyang Ji, Yu Xiang, and Dieter Fox.  DeepIM: Deep Iterative Matching for 6D Poseestimation.  In European Conference on Computer Vision, 2018.
  * [24] Zhigang Li, Gu Wang, and Xiangyang Ji.  CDPN: Coordinates-Based Disentangled Pose Network for Real-Time RGB-Based 6-DoF Object Pose Estimation.  In International Conference on Computer Vision, 2019.
  * [25] Tsung-Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár.  Focal Loss for Dense Object Detection.  In International Conference on Computer Vision, 2017.
  * [26] Ming-Yu Liu, Oncel Tuzel, Ashok Veeraraghavan, and Rama Chellappa.  Fast Directional Chamfer Matching.  In Conference on Computer Vision and Pattern Recognition, 2010.
  * [27] David G. Lowe.  Fitting Parameterized Three-Dimensional Models to Images.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 13(5):441–450, June 1991.
  * [28] David G. Lowe.  Distinctive Image Features from Scale-Invariant Keypoints.  International Journal of Computer Vision, 20(2):91–110, November 2004.
  * [29] Subhransu Maji and Jitendra Malik.  Object Detection using a Max-margin Hough Transform.  In Conference on Computer Vision and Pattern Recognition, 2009.
  * [30] Fabian Manhardt, Diego Martin Arroyo, Christian Rupprecht, Benjamin Busam, Tolga Birdal, Nassir Navab, and Federico Tombari.  Explaining the Ambiguity of Object Detection and 6D Pose From Visual Data.  In International Conference on Computer Vision, 2019.
  * [31] Frank Michel, Alexander Kirillov, Eric Brachmann, Alexander Krull, Stefan Gumhold, Bogdan Savchynskyy, and Carsten Rother.  Global Hypothesis Generation for 6D Object Pose Estimation.  In Conference on Computer Vision and Pattern Recognition, 2017.
  * [32] Markus Oberweger, Mahdi Rad, and Vincent Lepetit.  Making Deep Heatmaps Robust to Partial Occlusions for 3D Object Pose Estimation.  In European Conference on Computer Vision, 2018.
  * [33] Yuki Ono, Eduard Trulls, Pascal Fua, and Kwang Moo Yi.  LF-Net: Learning Local Features from Images.  In Advances in Neural Information Processing Systems, 2018.
  * [34] Kiru Park, Timothy Patten, and Markus Vincze.  Pix2Pose: Pixel-Wise Coordinate Regression of Objects for 6D Pose Estimation.  In International Conference on Computer Vision, 2019.
  * [35] Georgios Pavlakos, Xiaowei Zhou, Aaron Chan, Konstantinos G. Derpanis, and Kostas Daniilidis.  6-DoF Object Pose from Semantic Keypoints.  In International Conference on Robotics and Automation, 2017.
  * [36] Sida Peng, Yuan Liu, Qixing Huang, Hujun Bao, and Xiaowei Zhou.  PVNet: Pixel-Wise Voting Network for 6DoF Pose Estimation.  In Conference on Computer Vision and Pattern Recognition, 2019.
  * [37] Charles R. Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas.  PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation.  In Conference on Computer Vision and Pattern Recognition, 2017.
  * [38] Charles R. Qi, Li Yi, Hao Su, and Leonidas J. Guibas.  PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space.  In Advances in Neural Information Processing Systems, 2017.
  * [39] Mahdi Rad and Vincent Lepetit.  Bb8: A Scalable, Accurate, Robust to Partial Occlusion Method for Predicting the 3D Poses of Challenging Objects Without Using Depth.  In International Conference on Computer Vision, 2017.
  * [40] Fred Rothganger, Svetlana Lazebnik, Cordelia Schmid, and Jean Ponce.  3D Object Modeling and Recognition Using Local Affine-Invariant Image Descriptors and Multi-View Spatial Constraints.  International Journal of Computer Vision, 66(3), 2006.
  * [41] Bugra Tekin, Sudipta N. Sinha, and Pascal Fua.  Real-Time Seamless Single Shot 6D Object Pose Prediction.  In Conference on Computer Vision and Pattern Recognition, 2018.
  * [42] Engin Tola, Vincent Lepetit, and Pascal Fua.  DAISY: An Efficient Dense Descriptor Applied to Wide Baseline Stereo.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(5):815–830, 2010.
  * [43] Tomasz Trzcinski, C. Mario Christoudias, Vincent Lepetit, and Pascal Fua.  Learning Image Descriptors with the Boosting-Trick.  In Advances in Neural Information Processing Systems, December 2012\.
  * [44] Shubham Tulsiani and Jitendra Malik.  Viewpoints and Keypoints.  In Conference on Computer Vision and Pattern Recognition, 2015.
  * [45] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky.  Instance Normalization: The Missing Ingredient for Fast Stylization.  In arXiv Preprint, 2016.
  * [46] Steffen Urban, Jens Leitloff, and Stefan Hinz.  MLPnP-A Real-Time Maximum Likelihood Solution to the Perspective-N-Point Problem.  arXiv Preprint, 2016.
  * [47] Daniel Wagner, Gerhard Reitmayr, Alessandro Mulloni, Tom Drummond, and Dieter Schmalstieg.  Pose Tracking from Natural Features on Mobile Phones.  In International Symposium on Mixed and Augmented Reality, September 2008.
  * [48] Chen Wang, Danfei Xu, Yuke Zhu, Roberto Martín-Martín, Cewu Lu, Li Fei-Fei, and Silvio Savarese.  DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion.  In Conference on Computer Vision and Pattern Recognition, 2019.
  * [49] Wei Wang, Zheng Dang, Yinlin Hu, Pascal Fua, and Mathieu Salzmann.  Backpropagation-Friendly Eigendecomposition.  In Advances in Neural Information Processing Systems, 2019.
  * [50] Yu Xiang, Tanner Schmidt, Venkatraman Narayanan, and Dieter Fox.  PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes.  In Robotics: Science and Systems Conference, 2018.
  * [51] Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba.  SUN database: Large-scale scene recognition from abbey to zoo.  In Conference on Computer Vision and Pattern Recognition, 2010.
  * [52] Kwang Moo Yi, Eduard Trulls, Yuki Ono, Vincent Lepetit, Mathieu Salzmann, and Pascal Fua.  Learning to Find Good Correspondences.  In Conference on Computer Vision and Pattern Recognition, 2018.
  * [53] Kwang Moo Yi, Yannick Verdie, Pascal Fua, and Vincent Lepetit.  Learning to Assign Orientations to Feature Points.  In Conference on Computer Vision and Pattern Recognition, 2016.
  * [54] Sergey Zakharov, Ivan Shugurov, and Slobodan Ilic.  DPOD: 6D Pose Object Detector and Refiner.  In International Conference on Computer Vision, 2019.
  * [55] Yinqiang Zheng, Yubin Kuang, Shigeki Sugimoto, Kalle Åström, and Masatoshi Okutomi.  Revisiting the PnP Problem: A Fast, General and Optimal Solution.  In International Conference on Computer Vision, 2013.
  * [56] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang.  Random Erasing Data Augmentation.  In arXiv Preprint, 2017.


# On the Potential of Simple Framewise Approaches to Piano Transcription

###### Abstract

In an attempt at exploring the limitations of simple approaches to the task of
piano transcription (as usually defined in MIR), we conduct an in-depth
analysis of neural network-based framewise transcription. We systematically
compare different popular input representations for transcription systems to
determine the ones most suitable for use with neural networks. Exploiting
recent advances in training techniques and new regularizers, and taking into
account hyper-parameter tuning, we show that it is possible, by simple bottom-
up frame-wise processing, to obtain a piano transcriber that outperforms the
current published state of the art on the publicly available MAPS dataset –
without any complex post-processing steps. Thus, we propose this simple
approach as a new baseline for this dataset, for future transcription research
to build on and improve.

##  1 Introduction

Since their tremendous success in computer vision in recent years, neural
networks have been used for a large variety of tasks in the audio, speech and
music domain. They often achieve higher performance than hand-crafted feature
extraction and classification pipelines [20]. Unfortunately, using this model
class brings along considerable computational baggage in the form of hyper-
parameter tuning. These hyper-parameters include architectural choices such as
the number and width of layers and their type (e.g. dense, convolutional,
recurrent), learning rate schedule, other parameters of the optimization
scheme and regularizing mechanisms. Whereas for computer vision these
successes were possible using raw pixels as the input representation, in the
audio domain there seems to be an additional complication. Here the choices
for how to best represent the input range from spectrograms, logarithmically
filtered spectrograms over constant-Q transforms to even the raw audio itself
[10].

This is a tedious problem, and there seem to be only two solutions to it:
manual hyper-parameter selection, where a human expert tries to make decisions
based on her past experience, or automatic hyper-parameter optimization as
discussed in [11, 4, 28]. In this work we pursue a mixed strategy. As a first
step, we systematically find the most suitable input representation, and
progress from there with human expert knowledge to find best performing
architectures for the task of framewise piano transcription.

A variety of neural network architectures has been used specifically for
framewise transcription of piano notes from monaural sources. Some
transcription systems are separated into an acoustic model and a musical
language model, such as [7, 27, 26], whereas in others there is no such
distinction [23, 6, 2]. As shown in [26], models that utilize musical language
models perform better than those without, albeit the differences seem to be
small. We focus on the acoustic model here, neglecting the complementary
language model for now.

##  2 Input, Methods and Parameters

In what follows, we will describe the input representations we compared, and
give a brief overview of techniques for training and regularizing neural
networks.

###  2.1 Input Representation

Time-frequency representations in the form of spectrograms still seem to have
a distinct advantage over the raw audio input, as mentioned in [10]. The exact
parameterization of spectrograms is not entirely clear however, so we try to
address this question in a systematic way. We investigate the suitability of
different types of spectrograms and constant-Q transforms as input
representations for neural networks and compare four types of input
representations: spectrograms with linearly spaced bins S, spectrograms with
logarithmically spaced bins LS, spectrograms with logarithmically spaced bins
and logarithmically scaled magnitude LM, as well as the constant-Q transform
CQT [8]. The filterbank for LS and LM has a linear response (and lower
resolution) for the lower frequencies, and a logarithmic response for the
higher frequencies. We vary the sample rate $sr\in\\{22050,44100\\}$ [Hz],
number of bands per octave $nb\in\\{12,24,36,48\\}$, whether or not frames
undergo circular shift $cs\in\\{\mathrm{on},\mathrm{off}\\}$, the amount of
zero padding $zp\in\\{\times 0,\times 1,\times 2\\}$, and whether or not to
use area normalized filters when filter banks are used
$norm\in\\{\mathrm{yes},\mathrm{no}\\}$. Furthermore, we re-scale the
magnitudes of the spectrogram bins to be in the range $[0,1]$. Table 1
specifies which parameters are varied for which input type. For the
computation of spectrograms we used Madmom [5] and for the constant-Q
transform we used the Yaafe library [21].

| $sr$ | $zp$ | $cs$ | $nb$ | $norm$
---|---|---|---|---|---
CQT | $\times$ |  |  | $\times$ |
S | $\times$ | $\times$ | $\times$ |  |
LS | $\times$ | $\times$ | $\times$ | $\times$ | $\times$
LM | $\times$ | $\times$ | $\times$ | $\times$ | $\times$
Table 1: For each spectrogram type, these are the parameters that were varied.
See text for a description of the value ranges.

###  2.2 Model Class and Suitability

Formally, neural networks are functions with the structure

$\displaystyle\mathrm{net}_{k}(\mathbf{x})$
$\displaystyle=\mathbf{W}_{k}f_{k-1}(\mathbf{x})+\mathbf{b}_{k}$
$\displaystyle f_{k}(\mathbf{x})$
$\displaystyle=\sigma_{k}(\mathrm{net}_{k}(\mathbf{x}))$ $\displaystyle
f_{0}(\mathbf{x})$ $\displaystyle=\mathbf{x}$

where $\mathbf{x}\in\mathbb{R}^{w_{in}}$,
$f_{k}:\mathbb{R}^{w_{k-1}}\rightarrow\mathbb{R}^{w_{k}}$, $\sigma$ is any
element-wise nonlinear function, $\mathbf{W}_{k}$ is a matrix in
$\mathbb{R}^{w_{k}\times w_{k-1}}$ called the weight matrix, and
$\mathbf{b}_{k}\in\mathbb{R}^{w_{k}}$ is a vector called bias. The subscript
$k\in\\{0,\dots,L\\}$ is the index of the layer, with $k=0$ denoting the input
layer.

Choosing a very narrow definition on purpose, what we mean by a model class
$F$ is a fixed number of layers $L$, a fixed number of layer widths
$\\{w_{0},\dots w_{L}\\}$ and fixed types of nonlinearities
$\\{\sigma_{0},\dots\sigma_{L}\\}$. A model means an instance $f$ from this
class, defined by its weights alone. References to the whole collection of
weights will be made with $\Theta$.

For the task of framewise piano transcription we define the suitability of an
input representation in terms of the performance of a simple classifier on
this task, when given exactly this input representation.

Assuming we can reliably mitigate the risk of overfitting, we would like to
argue that this method of determining suitable input representations, and
using them for models with higher capacity, is the best we can do, given a
limited computational budget.

Using a low-variance, high-bias model class, the perceptron, also called
logistic regression or single-layer neural network, we learn a spectral
template per note. To test whether the results stemming from this analysis are
really relevant for higher-variance, lower-bias model classes, we run the same
set of experiments again, employing a multi layer perceptron with exactly one
hidden layer, colloquially called a shallow net. This small extension already
gives the network the possibility to learn a shared, distributed
representation of the input. As we will see, this has a considerable effect on
how suitability is judged.

###  2.3 Nonlinearities and Initialization

Common choices for nonlinearities include the logistic function
$\sigma(a)=\frac{1}{1+e^{-a}}$, hyperbolic tangent $\sigma(a)=\tanh{a}$, and
rectified linear units (ReLU) $\sigma(a)=\max(0,a)$ . Nonlinearities are
necessary to make neural networks universal function approximators [16].
According to [15, 13], using ReLUs as the nonlinearities in neural networks
leads to better behaved gradients and faster convergence because they do not
saturate.

Before training, the weight matrices are initialized randomly. The scale of
this initialization is crucial and depends on the used nonlinearity as well as
the number of weights contributing to the activation [15, 13]. Proper
initialization plays an even bigger role when networks with more than one
hidden layer are trained [31]. This is also important for the transcription
setting we use, as the output layer of our networks uses the logistic
function, which is prone to saturation effects. Thus we decided on using ReLUs
throughout, initialized with a uniform distribution having a scale of
$\pm\sqrt{2}\cdot\sqrt{\frac{2}{w_{k-1}+w_{k}}}$. For the last layer with the
logistic nonlinearity, we omit the gain factor of $\sqrt{2}$, as advised in
[13].

###  2.4 Weight Decay

To reduce overfitting and regularizing the network, different priors can be
imposed on the network weights. Usually a Gaussian or Laplacian prior is
chosen, corresponding to an $L_{2}$ or $L_{1}$ penalty term on connection
weights, added to the cost function
$\mathcal{L}_{reg}=\mathcal{L}+\lambda\sum_{k}\|\mathrm{vec}(\mathbf{W}_{k})\|_{1|2}$
[25, 32], where $\mathcal{L}$ is an arbitrary, unregularized cost function and
$\lambda$ governs the extent of regularization. Adding both of these penalty
terms corresponds to a technique called Elastic Net [33]. It is pointed out in
[1] that using $L_{2}$ regularization plays a similar role as early stopping
and thus might be omitted. An $L_{1}$ penalty on the other hand leads to
sparser weights, as it has a tendency to drive weights with irrelevant
contributions to zero.

###  2.5 Dropout

Applying dropout to a layer zeroes out a fraction of the activations of a
hidden layer of the network. For each training case, a different random
fraction is dropped. This prevents units from co-adapting, and relying too
much on each other’s presence, as reasoned in [30]. Dropout increases
robustness to noise, improves the generalization ability of networks and
mitigates the risk of overfitting. Additionally dropout can be interpreted as
model-averaging of exponentially many models [30].

###  2.6 Batch Normalization

Batch normalization [18] seeks to produce networks whose individual
activations per layer are zero-mean and unit-variance. This is ensured by
normalizing the activations for each mini-batch at each training step. This
effectively limits how far the activation distribution can drift away from
zero-mean, unit-variance during training. Not only does this alleviate the
need of the weights of the subsequent layer to adapt to a changing input
distribution during training, it also keeps the nonlinearities from saturating
and in turn speeds up training. It has additional regularizing effects, which
become more apparent the more layers a network has. After training is stopped,
the normalization is performed for each layer and for the whole training set.

###  2.7 Layer Types

We employ three different types of layer. Their respective functions can all
be viewed as matrix expressions in the end, and thus can be made to fit into
the framework described in Section 2.2. For the sake of readability, we simply
describe their function in a procedural way.

A dense layer consists of a dense matrix - vector pair
$(\mathbf{W},\mathbf{b})$ together with a nonlinearity. The input is
transformed via this affine map, and then passed through a nonlinearity.

A convolutional layer consists of a number $C_{k}$ of convolution kernels of a
certain size $\\{(\mathbf{W}_{c},\mathbf{b}_{c})\\}_{c=0}^{C_{k}}$ together
with a non-linearity. The input is convolved with each convolution kernel,
leading to $C_{k}$ different feature maps to which the same nonlinearity is
applied.

Max pooling layers are used in convolutional networks to provide a small
amount of translational invariance. They select the units with maximal
activation in a local neighborhood $(w_{t},w_{f})$ in time and frequency in a
feature map. This has beneficial effects, as it makes the transcriber
invariant to small changes in tuning.

Global average pooling layers are used in all-convolutional networks to
compute the mean value of feature maps.

###  2.8 Architectures

There is a fundamental choice between a network with all dense layers, a
network with all convolutional layers, and a mixed approach, where usually the
convolutional layers are the first ones after the input layer followed by
dense layers. Pooling layers, batch normalization and dropout application are
interleaved. For all networks we have to choose the number of layers, how many
hidden units per layer to use and when to interleave a regularization layer.
For convolutional networks we have to choose the number of filter kernels and
their extent in time and frequency direction.

###  2.9 Networks for Framewise Polyphonic Piano Transcription

The output layer of all considered model classes has $88$ units, in line with
the playable notes on most modern pianos, and the output nonlinearity is the
logistic function, whose output ranges lie in the interval $[0,1]$.

The loss function being minimized is the frame- and element-wise applied
binary crossentropy

$\displaystyle\mathcal{L}_{bce}^{(t)}(\mathbf{y}_{t},\hat{\mathbf{y}}_{t})=-(\mathbf{y}_{t}\cdot\log(\hat{\mathbf{y}}_{t})+(1-\mathbf{y}_{t})\cdot\log(1-\hat{\mathbf{y}}_{t}))$

where $\hat{\mathbf{y}}_{t}=f_{L}(\mathbf{x}_{t})$ is the output vector of the
network, and $\mathbf{y}_{t}$ the ground truth at time $t$. As the overall
loss over the whole training set we take the mean

$\displaystyle\mathcal{L}=\frac{1}{T}\sum_{t=1}^{T}\mathcal{L}_{bce}^{(t)}$

For the purpose of computing the performance measures, the prediction of the
network is thresholded to obtain a binary prediction
$\bar{\mathbf{y}}_{t}=\hat{\mathbf{y}}_{t}>0.5$.

###  2.10 Optimization

The simplest way to adapt the weights $\Theta$ of the network to minimize the
loss is to take a small step with length $\alpha$ in the direction of steepest
descent:

$\displaystyle\Theta_{i+1}=\Theta_{i}-\alpha\cdot\frac{\partial\mathcal{L}}{\partial\Theta}$

Computing the true gradient
$\frac{\partial\mathcal{L}}{\partial\Theta}=\frac{1}{T}\sum_{t=1}^{T}\frac{\partial\mathcal{L}_{bce}^{(t)}}{\partial\Theta}$
requires a sum over the length of the whole training set, and is
computationally too costly. For this reason, the gradient is usually only
approximated from an i.i.d. random sample of size $M\ll T$. This is called
mini-batch stochastic gradient descent. There are several extensions to this
general framework, such as momentum [24], Nesterov momentum [22] or Adam [19],
which try to smooth the gradient estimate, correct small missteps or adapt the
learning rate dynamically, respectively. Additionally we can set a learning
rate schedule that controls the temporal evolution of the learning rate.

##  3 Dataset and Measures

The computational experiments have been performed with the MAPS dataset [12].
It provides MIDI-aligned recordings of a variety of classical music pieces.
They were rendered using different hi-quality piano sample patches, as well as
real recordings from an upright Disklavier. This ensures clean annotation and
therefore almost no label-noise. For all performance comparisons the following
framewise measures on the validation set are used:

$\displaystyle P$
$\displaystyle=\sum_{t=0}^{T-1}\frac{\mathit{TP}[t]}{\mathit{TP}[t]+\mathit{FP}[t]}$
$\displaystyle R$
$\displaystyle=\sum_{t=0}^{T-1}\frac{\mathit{TP}[t]}{\mathit{TP}[t]+\mathit{FN}[t]}$
$\displaystyle F_{1}$ $\displaystyle=\frac{2\cdot P\cdot R}{P+R}$

The train-test folds are those used in [26] which were published online
111http://www.eecs.qmul.ac.uk/~sss31/TASLP/info.html. For each fold, the
validation set consists of $43$ tracks randomly removed from the train set,
deviating from the $26$ used in [26], and leading to a division of 173-43-54
between the three sets. Note that the test sets are the same, and are referred
to as configuration I in [26]. The exact splits for configuration II were not
published. We had to choose them ourselves, using the same methodology, which
has the additional constraint that only recordings of the real piano are used
for testing, resulting in a division of 180-30-60. This constitutes a more
realistic setting for piano transcription.

##  4 Analysis of relative Hyper-Parameter Importance

To identify and select an appropriate input representation and determine the
most important hyper-parameters responsible for high transcription
performance, a multi-stage study with subsequent fANOVA analysis was
conducted, as described in [17]. This is similar in spirit to [14], albeit on
a smaller scale.

###  4.1 Types of Representation

To isolate the effects of different input representations on the performance
of different model classes, only parameters for the spectrogram were varied
according to Table 1. This leads to $204$ distinct input representations. The
hyper-parameters for the model class as well as the optimization scheme were
held fixed. To make our estimates more robust, we conducted multiple runs for
the same type of input.

The results for each model class are summarized in Table 2, containing the
three most influential hyper-parameters and the percentage of variability in
performance they are responsible for. The most important hyper-parameter for
both model classes is the type of spectrogram used, followed by pairwise
interactions. Please note that the numbers in the percentage column are mainly
useful to judge the relative importance of the parameters. We will see these
relative importances put into a larger context later on.

In Figure 1, we can see the mean performance attainable with different types
of spectrograms for both model classes. The error bars indicate the standard
deviation for the spread in performance, caused by the rest of the varied
parameters. Surprisingly, the spectrogram with logarithmically spaced bins and
logarithmically scaled magnitude, $LM$, enables the shallow net to perform
best, even though it is a clear mismatch for logistic regression. The lower
performance of the constant-Q transform was quite unexpected in both cases and
warrants further investigation.

Model Class | Pct | Parameters
---|---|---
Logistic Regression | 48.6% | Spectrogram Type
| 16.9% | Spectrogram Type
|  |  $\times$ Normed Area Filters
| 10.4% | Spectrogram Type
|  |  $\times$ Sample Rate
Shallow Net | 68.4% | Spectrogram Type
| 20.8% | Spectrogram Type
|  |  $\times$ Sample Rate
| 5.7% | Sample Rate
Table 2: The three most important parameters determining input representation
for different model classes

Figure 1: (a) Mean logistic regression performance dependent on spectrogram
(b) Mean shallow net performance dependent on type of spectrogram

###  4.2 Greater context

Attempting a full grid search on all possible input representation and model
class hyper-parameters described in Section 2 to compute their true
marginalized performance is computationally too costly. It is possible however
to compute the predicted marginalized performance of a hyper-parameter
efficiently from a smaller subsample of the space, as shown in [17]. All
parameters are randomly varied to sample the space as evenly as possible, and
a random forest of $100$ regression trees is fitted to the measured
performance. This allows to predict the marginalized performance of individual
hyper-parameters. Table 3 contains the list of hyper-parameters varied.

Figure 2: Relative importance of the first $10$ hyper-parameters for the
shallow net model class.

The percentage of variance in performance these hyper-parameters are
responsible for, can be seen in Figure 2 for the $10$ most important ones. A
total of $3000$ runs with random parameterizations were made.

Optimizer (Plain SGD, Momentum, Nesterov Momentum, Adam)
---
Learning Rate (0.001, 0.01, 0.1, 0.5, 1.0, 2.0, 10.0, 50.0, 100.0)
Momentum (Off, 0.7, 0.8, 0.9)
Learning rate Scheduler (On, Off)
Batch Normalization (On, Off)
Dropout (Off, 0.1, 0.3, 0.5)
$L_{1}$ Penalty (Off, 1e-07, 1e-08, 1e-09)
$L_{2}$ Penalty (Off, 1e-07, 1e-08, 1e-09)
Table 3: The list of additional hyper-parameters varied, and their ranges.

Analyzing the results of all the runs tells us that the most important hyper-
parameters are Learning Rate (47.11%), and Spectrogram Type (5.28%). The
importance of the learning rate is in line with the findings in [14]. Figure 2
shows the relative importances of the first $10$ hyper-parameters, and Figure
3 shows the predicted marginal performance of the learning rate dependent on
its value (on a logarithmic scale) in greater detail.

Figure 3: Mean predicted performance for the shallow net model class,
dependent on learning rate (on a logarithmic scale). The dark line shows the
mean performance, and the gray area shows the standard deviation.

##  5 State of the Art Models

Having completed the analysis of input representation, more powerful model
classes were tried: a deep neural network consisting entirely of dense layers
(DNN), a mixed network with convolutional layers directly after the input
followed by dense layers (ConvNet), and an all-convolutional network (AllConv
[29]). Their architectures are described in detail in Table 4. To the best of
our knowledge, this is the first time an all-convolutional net has been
proposed for the task of framewise piano transcription.

We computed a logarithmically filtered spectrogram with logarithmic magnitude
from audio with a sample rate of $44.1$ kHz, a filterbank with $48$ bins per
octave, normed area filters, no circular shift and no zero padding. The
choices for circular shift and zero padding ranged very low on the importance
scale, so we simply left them switched off. This resulted in only $229$ bins,
which are logarithmically spaced in the higher frequency regions, and almost
linearly spaced in the lower frequency regions as mentioned in Section 2.1.
The dense network was presented one frame at a time, whereas the convolutional
network was given a context in time of two frames to either side of the
current frame, summing to $5$ frames in total.

| DNN | ConvNet | AllConv
---|---|---|---
| Input 229 | Input 5x229 | Input 5x229
| Dropout 0.1 | Conv 32x3x3 | Conv 32x3x3
| Dense 512 | Conv 32x3x3 | Conv 32x3x3
| BatchNorm | BatchNorm | BatchNorm
| Dropout 0.25 | MaxPool 1x2 | MaxPool 1x2
| Dense 512 | Dropout 0.25 | Dropout 0.25
| BatchNorm | Conv 64x3x3 | Conv 32x1x3
| Dropout 0.25 | MaxPool 1x2 | BatchNorm
| Dense 512 | Dropout 0.25 | Conv 32x1x3
| BatchNorm | Dense 512 | BatchNorm
| Dropout 0.25 | Dropout 0.5 | MaxPool 1x2
| Dense 88 | Dense 88 | Dropout 0.25
|  |  | Conv 64x1x25
|  |  | BatchNorm
|  |  | Conv 128x1x25
|  |  | BatchNorm
|  |  | Dropout 0.5
|  |  | Conv 88x1x1
|  |  | BatchNorm
|  |  | AvgPool 1x6
|  |  | Sigmoid
# Params | 691288 | 1877880 | 284544
Table 4: Model Architectures

All further hyper-parameter tuning and architectural choices have been left to
a human expert. Models within a model class were selected based on average
F-measure across the four validation sets. An automatic search via a hyper-
parameter search algorithm for these larger model classes, as described in
[11, 4, 28] is left for future work (the training time for a convolutional
model is roughly $8-9$ hours on a Tesla K40 GPU, which leaves us with
$204\cdot 3\cdot 4\cdot 8$ hours (variants $\times$ #models $\times$ #folds
$\times$ hours per model), or on the order of $800-900$ days of compute time
to determine the best input representation exactly).

For these powerful models, we followed practical recommendations for training
neural networks via gradient descent found in [1]. Particularly relevant is
the way of setting the initial learning rate. Strategies that dynamically
adapt the learning rate, such as Adam or Nesterov Momentum [19, 22] help to a
certain extent, but still do not spare us from tuning the initial learning
rate and its schedule.

We observed that using a combination of batch normalization and dropout
together with very simple optimization strategies leads to low validation
error fairly quickly, in terms of the number of epochs trained. The strategy
that worked best for determining the learning rate and its schedule was trying
learning rates on a logarithmic scale, starting at $10.0$, until the
optimization did not diverge anymore [1], then training until the validation
error flattened out for a few epochs, then multiplying the learning rate with
a factor from the set $\\{0.1,0.25,0.5,0.75\\}$. The rates and schedules we
finally settled on were:

  * •

DNN: SGD with Momentum, $\alpha=0.1,\mu=0.9$ and halving of $\alpha$ every
$10$ epochs

  * •

ConvNet: SGD with Momentum, $\alpha=0.1,\mu=0.9$ and a halving of $\alpha$
every $5$ epochs

  * •

AllConv: SGD with Momentum, $\alpha=1.0,\mu=0.9$ and a halving of $\alpha$
every $10$ epochs

The results for framewise prediction on the MAPS dataset can be found in Table
5. It should be noted that we compare straightforward, simple, and largely un-
smoothed systems (ours) with hybrid systems [26]. There is a small degree of
temporal smoothing happening when processing spectrograms with convolutional
nets. The term simple is supposed to mean that the resulting models have a
small amount of parameters and the models are composed of a few low-complexity
building blocks. All systems are evaluated on the same train-test splits,
referred to as configuration I in [26] as well as on realistic train-test
splits, that were constructed in the same fashion as configuration II in [26].

Model Class | $P$ | $R$ | $F_{1}$
---|---|---|---
Hybrid DNN [26] | 65.66 | 70.34 | 67.92
Hybrid RNN [26] | 67.89 | 70.66 | 69.25
Hybrid ConvNet [26] | 72.45 | 76.56 | 74.45
DNN | 76.63 | 70.12 | 73.11
ConvNet | 80.19 | 78.66 | 79.33
AllConv | 80.75 | 75.77 | 78.07
Table 5: Results on the MAPS dataset. Test set performance was averaged across $4$ folds as defined in configuration I in [26]. Model Class | $P$ | $R$ | $F_{1}$
---|---|---|---
DNN [26] | - | - | 59.91
RNN [26] | - | - | 57.67
ConvNet [26] | - | - | 64.14
DNN | 75.51 | 57.30 | 65.15
ConvNet | 74.50 | 67.10 | 70.60
AllConv | 76.53 | 63.46 | 69.38
Table 6: Results on the MAPS dataset. Test set performance was averaged across
$4$ folds as defined in configuration II in [26].

##  6 Conclusion

We argue that the results demonstrate: the importance of proper choice of
input representation, and the importance of hyper-parameter tuning, especially
the tuning of learning rate and its schedule; that convolutional networks have
a distinct advantage over their deep and dense siblings, because of their
context window and that all-convolutional networks perform nearly as well as
mixed networks, although they have far fewer parameters. We propose these
straightforward, framewise transcription networks as a new state-of-the art
baseline for framewise piano transcription for the MAPS dataset.

##  7 Acknowledgements

This work is supported by the European Research Council (ERC Grant Agreement
670035, project CON ESPRESSIONE), the Austrian Ministries BMVIT and BMWFW, the
Province of Upper Austria (via the COMET Center SCCH) and the European Union
Seventh Framework Programme FP7 / 2007-2013 through the GiantSteps project
(grant agreement no. 610591). We would like to thank all developers of Theano
[3] and Lasagne [9] for providing comprehensive and easy to use deep learning
frameworks. The Tesla K40 used for this research was donated by the NVIDIA
Corporation.

## References

  * [1] Yoshua Bengio.  Practical recommendations for gradient-based training of deep architectures.  In Neural Networks: Tricks of the Trade, pages 437–478. Springer, 2012.
  * [2] Taylor Berg-Kirkpatrick, Jacob Andreas, and Dan Klein.  Unsupervised Transcription of Piano Music.  In Advances in Neural Information Processing Systems, pages 1538–1546, 2014.
  * [3] James Bergstra, Olivier Breuleux, Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, Guillaume Desjardins, Joseph Turian, David Warde-Farley, and Yoshua Bengio.  Theano: a CPU and GPU Math Expression Compiler.  In Proceedings of the Python for Scientific Computing Conference (SciPy), 2010.
  * [4] James Bergstra, Dan Yamins, and David D. Cox.  Hyperopt: A python library for optimizing the hyperparameters of machine learning algorithms.  In Proceedings of the 12th Python in Science Conference, pages 13–20, 2013.
  * [5] Sebastian Böck, Filip Korzeniowski, Jan Schlüter, Florian Krebs, and Gerhard Widmer.  madmom: a new Python Audio and Music Signal Processing Library.  arXiv preprint arXiv:1605.07008, 2016.
  * [6] Sebastian Böck and Markus Schedl.  Polyphonic piano note transcription with recurrent neural networks.  In Acoustics, Speech and Signal Processing (ICASSP), 2012 IEEE International Conference on, pages 121–124. IEEE, 2012.
  * [7] Nicolas Boulanger-Lewandowski, Yoshua Bengio, and Pascal Vincent.  Modeling temporal dependencies in high-dimensional sequences: Application to polyphonic music generation and transcription.  arXiv preprint arXiv:1206.6392, 2012.
  * [8] Judith C. Brown.  Calculation of a constant Q spectral transform.  The Journal of the Acoustical Society of America, 89(1):425–434, 1991.
  * [9] Sander Dieleman, Jan Schlüter, Colin Raffel, Eben Olson, Søren Kaae Sønderby, Daniel Nouri, Daniel Maturana, Martin Thoma, Eric Battenberg, Jack Kelly, Jeffrey De Fauw, Michael Heilman, diogo149, Brian McFee, Hendrik Weideman, takacsg84, peterderivaz, Jon, instagibbs, Dr. Kashif Rasul, CongLiu, Britefury, and Jonas Degrave.  Lasagne: First release., August 2015.
  * [10] Sander Dieleman and Benjamin Schrauwen.  End-to-end learning for music audio.  In Acoustics, Speech and Signal Processing (ICASSP), 2014 IEEE International Conference on, pages 6964–6968. IEEE, 2014.
  * [11] Katharina Eggensperger, Matthias Feurer, Frank Hutter, James Bergstra, Jasper Snoek, Holger Hoos, and Kevin Leyton-Brown.  Towards an empirical foundation for assessing bayesian optimization of hyperparameters.  In NIPS workshop on Bayesian Optimization in Theory and Practice, 2013.
  * [12] Valentin Emiya, Roland Badeau, and Bertrand David.  Multipitch estimation of piano sounds using a new probabilistic spectral smoothness principle.  Audio, Speech, and Language Processing, IEEE Transactions on, 18(6):1643–1654, 2010.
  * [13] Xavier Glorot and Yoshua Bengio.  Understanding the difficulty of training deep feedforward neural networks.  In International Conference on Artificial Intelligence and Statistics, pages 249–256, 2010.
  * [14] Klaus Greff, Rupesh Kumar Srivastava, Jan Koutník, Bas R. Steunebrink, and Jürgen Schmidhuber.  LSTM: A search space odyssey.  arXiv preprint arXiv:1503.04069, 2015.
  * [15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification.  arXiv preprint arXiv:1502.01852, 2015.
  * [16] Kurt Hornik, Maxwell Stinchcombe, and Halbert White.  Multilayer feedforward networks are universal approximators.  Neural networks, 2(5):359–366, 1989.
  * [17] Frank Hutter, Holger Hoos, and Kevin Leyton-Brown.  An efficient approach for assessing hyperparameter importance.  In Proceedings of the 31st International Conference on Machine Learning (ICML-14), pages 754–762, 2014.
  * [18] Sergey Ioffe and Christian Szegedy.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate shift.  arXiv preprint arXiv:1502.03167, 2015.
  * [19] Diederik Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [20] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton.  Deep learning.  Nature, 521(7553):436–444, May 2015.
  * [21] Benoit Mathieu, Slim Essid, Thomas Fillon, Jacques Prado, and Gaël Richard.  YAAFE, an Easy to Use and Efficient Audio Feature Extraction Software.  In Proceedings of the International Conference on Music Information Retrieval, pages 441–446, 2010.
  * [22] Yurii Nesterov.  A method of solving a convex programming problem with convergence rate $O(1/k^{2})$.  Soviet Mathematics Doklady, 27(2):372––376, 1983.
  * [23] Ken O’Hanlon and Mark D. Plumbley.  Polyphonic piano transcription using non-negative matrix factorisation with group sparsity.  In Acoustics, Speech and Signal Processing (ICASSP), 2014 IEEE International Conference on, pages 3112–3116. IEEE, 2014.
  * [24] Boris T. Polyak.  Some methods of speeding up the convergence of iteration methods.  USSR Computational Mathematics and Mathematical Physics, 4(5):1–17, 1964.
  * [25] David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams.  Learning representations by back-propagating errors.  Nature, 323(6088), 1986.
  * [26] Siddartha Sigtia, Emmanouil Benetos, and Simon Dixon.  An end-to-end neural network for polyphonic piano music transcription.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 24(5):927–939, May 2016.
  * [27] Siddhartha Sigtia, Emmanouil Benetos, Nicolas Boulanger-Lewandowski, Tillman Weyde, Artur S. d’Avila Garcez, and Simon Dixon.  A hybrid recurrent neural network for music transcription.  In Acoustics, Speech and Signal Processing (ICASSP), 2015 IEEE International Conference on, pages 2061–2065. IEEE, 2015.
  * [28] Jasper Snoek, Hugo Larochelle, and Ryan P. Adams.  Practical bayesian optimization of machine learning algorithms.  In Advances in neural information processing systems, pages 2951–2959, 2012.
  * [29] Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller.  Striving for Simplicity: The All Convolutional Net.  arXiv preprint arXiv:1412.6806, 2014.
  * [30] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: A simple way to prevent neural networks from overfitting.  The Journal of Machine Learning Research, 15(1):1929–1958, 2014\.
  * [31] David Sussillo.  Random Walks: Training Very Deep Nonlinear Feed-Forward Networks with Smart Initialization.  arXiv preprint arXiv:1412.6558, 2014.
  * [32] Peter M. Williams.  Bayesian Regularization and Pruning Using a Laplace Prior.  Neural Computation, 7(1):117–143, Jan 1995.
  * [33] Hui Zou and Trevor Hastie.  Regularization and variable selection via the elastic net.  Journal of the Royal Statistical Society: Series B (Statistical Methodology), 67(2):301–320, 2005.