# Towards musically informed evaluation
of piano transcription models

###### Abstract

Automatic piano transcription models are typically evaluated using simple
frame- or note-wise information retrieval (IR) metrics. Such benchmark metrics
do not provide insights into the transcription quality of specific musical
aspects such as articulation, dynamics, or rhythmic precision of the output,
which are essential in the context of expressive performance analysis.
Furthermore, in recent years, MAESTRO has become the de-facto training and
evaluation dataset for such models. However, inference performance has been
observed to deteriorate substantially when applied on out-of-distribution
data, thereby questioning the suitability and reliability of transcribed
outputs from such models for specific MIR tasks. In this work, we investigate
the performance of three state-of-the-art piano transcription models in two
experiments. In the first one, we propose a variety of musically informed
evaluation metrics which, in contrast to the IR metrics, offer more detailed
insight into the musical quality of the transcriptions. In the second
experiment, we compare inference performance on real-world and perturbed audio
recordings, and highlight musical dimensions which our metrics can help
explain. Our experimental results highlight the weaknesses of existing piano
transcription metrics and contribute to a more musically sound error analysis
of transcription outputs.

##  1 Introduction

Automatic Music Transcription (AMT) refers to the task of converting audio
signals into symbolic music representations. The target output format can be a
full symbolic score including quantized rhythm, time signature and pitch
spelling information, or a mid-level physical MIDI(-like) representation,
describing notes in terms of their onset and offset times, pitch and velocity
[1, 2, 3].

AMT methods are typically evaluated using information retrieval (IR) metrics
like precision, recall and F1 score [4]. These IR metrics can be computed at
the level of frames, by comparing binary piano roll-like matrices, or at the
level of note lists, by comparing notes in terms of their onset, offset, pitch
and/or velocity attributes. Each error (i.e. misplaced frame or note activity)
has equal weight, resulting in limited explanatory power of these metrics with
respect to the underlying musical material [1, 5].

As in many other areas in MIR, the current state-of-the-art is defined by deep
neural networks [6, 3, 2]. To a large extent, this progress has been enabled
by the release of the MAESTRO dataset [7], which made well-aligned audio-MIDI
piano performance data available on a large scale. The most up-to-date version
of the MAESTRO dataset111https://magenta.tensorflow.org/datasets/maestro
contains close to 200 hours of performance data from close to 1300 recordings
of Western classical piano repertoire. State-of-the-art piano transcription
systems achieve beyond 90% frame-level, or 80% note-level F1 scores on its
test split [7, 2, 3, 8], and have led to the release of large-scale
transcribed solo piano performance datasets [9, 10].

Although these results are impressive, we believe that two important aspects
have been largely overlooked: first, the validity and (lack of) explanatory
power of the standard evaluation metrics with respect to musically relevant
information, and second, the reliability of these transcription models on out-
of-distribution data. In this work, we address the first problem by proposing
a set of musically informed evaluation metrics that support a more nuanced
understanding of piano transcription errors. The metrics are intended to be
used in the context of computational performance studies, and therefore focus
on musical dimensions that are commonly studied in the context of expressive
piano performance analysis and generation. We demonstrate our metrics on a
subset of the MAESTRO dataset, which we transcribe using three state-of-the-
art transcription models. In particular, we contrast the performance of these
models, as evaluated with the standard IR metrics, with their performance on
musical dimensions such as timing, articulation and dynamics which we can
evaluate using our set of musically informed metrics.

Then, to elucidate the second problem, we re-record a subset of the MAESTRO
dataset on a Yamaha Disklavier grand piano and further manipulate the audio
recordings by adding different levels of noise and reverberation. An analysis
of the outputs of the pre-trained transcription models on these recordings
provides some detailed insights into the lack of generalization on out-of-
distribution data. We make our set of metrics, data and all experimental
results available at https://github.com/CPJKU/mpteval.

##  2 Related Work

This section briefly reviews the standard IR evaluation metrics along with
criticism related to these, followed by a description of the benchmark
datasets typically used for evaluating transcription methods.

Precision, recall and F1 score are the standard evaluation metrics used in AMT
[1, 2, 3, 4]. They can be computed either at the level of frames or at the
level of notes. Frame-level evaluation compares two binary piano roll matrices
$M,\hat{M}\in\\{0,1\\}^{p\times t}$, where $p$ defines the pitch range and $t$
the temporal resolution of a frame (typically 10ms [4]). Both $M$ and
$\hat{M}$ are sparse matrices where 1 at a given index $p,t$ indicates a note
with pitch $p$ is active at time frame $t$. True positives (TP) are then
counted for frames in which $M_{p,t}=\hat{M}_{p,t}=1$, false positives (FP)
for frames where $M_{p,t}=0$ but $\hat{M}_{p,t}=1$, and false negatives (FN)
for frames where $M_{p,t}=1$ but $\hat{M}_{p,t}=0$. Precision, recall and F1
score are then calculated on these counts.

Note-level metrics are computed by comparing lists of notes, in which each
note is described by a tuple describing the onset, offset, pitch, and (where
predicted) velocity. Note-based metrics can be based on onset information
only, onset and offset information (i.e., note durations), or on predicted
onset, offset and velocity. In onset-only note evaluation, a note is
considered correct if its onset falls within a $\pm$50 ms threshold of its
respective target onset. For onset-and-offset note-level evaluation, the note
offset must fall within the greater of either an offset tolerance threshold of
$\pm$50 ms, or a duration threshold 20% of the ground truth duration [4]. If
velocity is included in the evaluation, an estimated note is considered
correct if its velocity (after some normalization and rescaling operations)
falls within a 0.1 tolerance threshold of the velocity of the corresponding
reference note [1].

The need for better (i.e., musically or perceptually sound) transcription
metrics has been expressed by various researchers before. Hawthorne et al. [1]
point out that frame- and onset-only note-level evaluation does not
sufficiently capture musically relevant information. Similarly, Ycart et al.
[5] and Daniel et al. [11] focus on the problem of perceptual saliency of
different kinds of transcription errors and each propose a new, perceptually
(more) valid transcription metric. Finally, McLeod and Steedman [12] focus on
the problem of audio-to-score transcription and propose a new metric that
jointly evaluates voice separation, metrical alignment, note value detection
and harmonic analysis along with multi-pitch detection.

With respect to training and evaluation data for solo piano transcription,
until the introduction of MAESTRO [7], MAPS [13] was used as the standard
dataset. Apart from size, the biggest difference between the two is the
diversity of the captured recording environments: while MAESTRO exclusively
contains Disklavier recordings from the Yamaha International Piano
e-Competition222http://piano-e-competition.com/, MAPS contains Disklavier
recordings and synthesized audio simulating various recording environments.
The prevailing trend in evaluating current piano transcription models centers
around the MAESTRO dataset [2, 3, 6], and most models that do include MAPS in
their evaluation [1, 3] use the split proposed in [14], which only includes
Disklavier recordings in the test split. Both frame- and note-level metrics
are usually computed for each piece in a given test set, and their mean is
subsequently reported as the inference performance for a given model and
dataset/split. Frame-level metrics are typically higher than note-level ones
due to common known transcription errors such as merged or segmented notes.

##  3 Musically informed metrics

In this section we describe our proposed metrics that are meant to capture
different musical dimensions commonly studied in the context of expressive
performance. Each metric compares a ground truth to a predicted MIDI
performance by measuring the Pearson correlation between a performance
parameter computed from the ground truth and from the predicted MIDI,
respectively. We choose a correlational measure to ensure all metrics fall
into the same range. The goal is to quantify dimensions of musical quality of
transcriptions that are otherwise obscured by standard IR metrics. In
particular, we wish to capture dimensions that are important for computational
performance studies that make use of automatically transcribed piano
performances.

###  3.1 Timing

Timing can be described as expressive deviations from the metrical grid. A
common measure of expressive timing in computational performance analysis is
the inter-onset-interval (IOI), that is, the amount of time passed between two
consecutive notes belonging to the same stream.333We use the term _stream_ as
a generalization of the concept of a voice in polyphonic music [15].

To evaluate how well a transcription preserves the micro onset deviations, we
predict a monophonic melody line and the accompaniment part (i.e., all notes
not belonging to the melody line) in a given MIDI using the skyline algorithm
for melody identification [16, 17]. Then we compute the IOIs of these streams
both on the ground truth and predicted performance, and measure their
correlation. Note that for non-strictly monophonic streams (like the
accompaniment part), the IOI between notes that belong to the same onset
(i.e., chords) is zero. The result of this process gives us two measures,
which we call _Melody IOI_ and _Accompaniment IOI_.

###  3.2 Articulation

Articulation in expressive piano performance refers to how (adjacent) notes
are played in terms of their duration, intensity, and clarity, resulting in
expressive strategies such as legato, staccato or marcato. Computationally,
articulation is measured through the key-overlap-ratio (KOR), defined as the
ratio between the time interval between two notes and their inter-onset-
interval (IOI) [18, 19, 20].

We use the skyline algorithm [16] to extract monophonic melody and bass lines
within a performance, and compute a sequence of KOR values, for each pair of
successive note events, for both the target and the predicted performance MIDI
for both streams, and their ratio. We define three metrics for capturing
articulation:

  1. 1.

_Melody KOR_ : the correlation between the KOR sequences of the melody lines
of the ground truth and the predicted performance MIDI.

  2. 2.

_Bass KOR_ : the correlation between the KOR sequences of the bass lines of
the ground truth and the predicted performance MIDI.

  3. 3.

_Ratio KOR_ : for this metric, we consider the ratio of KOR sequences of the
melody to the bass line. A ratio KOR greater than 1 indicates that the melody
voice is played more legato than the bass voice. The Ratio KOR metric is
computed as correlation of this ratio between the ground truth and the
predicted performance MIDI.

###  3.3 Harmony

Harmony and aspects such as harmonic tension have been shown to be determining
factors for various performance decisions (particularly relating to expressive
tempo and dynamics [21, 22]). To quantify how well harmonic tension is
preserved in a transcription, we use two features proposed by Herremans and
Chew [23] based on Chew’s spiral array model [24]. This model is a three
dimensional representation of pitch classes, chords and keys constructed in
such a way that spatial proximity represents close tonal relationships. The
proposed metrics for capturing harmonic tension are:

  1. 1.

_Cloud diameter_. This feature estimates the maximal tonal distance between
notes in a segment of music. We compute this feature on overlapping windows
for both the ground truth and the predicted performance MIDI, and compute the
correlation between them.

  2. 2.

_Cloud momentum_. This feature quantifies the harmonic movement in a piece of
music as the tonal distance between consecutive sections. In a similar way to
the cloud diameter, we measure this feature on overlapping windows for the
ground truth and the predicted performance MIDI, and compute the correlation
between these features.

###  3.4 Dynamics

For comparing the performance of transcription models regarding expressive
dynamics, we use the loudness ratio of the melody and bass lines as a proxy to
identify how well a transcription preserves the dynamics of the performance.
We estimate the loudness as the “energy” of a stream (i.e., melody or bass
line), which is computed using the MIDI velocity following a model proposed by
Dannenberg [25]. The loudness ratio is then computed as follows (cf. Equation
8 in [25]):

$\text{R}(t)=\log\left(\frac{m\cdot\text{vel}_{mel}(t)+b}{m\cdot\text{vel}_{bass}(t)+b}\right)$
(1)

where $\text{vel}_{mel}(t)$ and $\text{vel}_{bass}(t)$ are the MIDI velocities
of the melody and bass lines at time $t$, respectively, and $m$ and $b$ are
constant parameters that depend on the dynamic range of the audio signal. We
compute the loudness ratio for both the ground truth and estimated performance
MIDI, and compute the correlation between these ratios as our metric for
dynamics.444Note that parameters $m$ and $b$ cancel out when computing the
correlation.

##  4 Data

composer | pieces | performances | duration (min)
---|---|---|---
Bach | 1 | 7 | 23.36
Beethoven | 5 | 28 | 285.54
Chopin | 4 | 15 | 150.28
Debussy | 2 | 3 | 32.06
Glinka | 1 | 2 | 10.35
Haydn | 3 | 9 | 90.23
Liszt | 3 | 12 | 58.98
Mozart | 2 | 4 | 29.02
Rachmaninoff | 2 | 3 | 11.87
Schubert | 3 | 17 | 107.27
Scriabin | 1 | 5 | 55.05
Total | 27 | 105 | 854.01
Table 1: Overview of our evaluation set.

For our experiments, we create an evaluation set consisting of the three
subsets: First, we choose a subset of audio recordings from the MAESTRO
dataset that captures a wide range of musical repertoire and style in terms of
composers and performers. Following this rationale, we select recordings from
all (train, validation, and test) splits of the dataset as they are officially
provided by the authors[3]. We make this latter choice by design, to test
whether the split category would influence the generalization capability of
the analysed models.555The official MAPS splits [7] ensure a unique piece-to-
split mapping. An overview of the selected MAESTRO subset is shown in Table 1.

Next, to investigate the inference performance of the analysed models with
respect to the acoustic environment, we create two more subsets:

  1. 1.

We re-record all MIDI performances from our MAESTRO subset on a Yamaha
Disklavier Enspire ST C1X666Note that all recordings in the MAESTRO dataset
likewise stem from performances on a Disklavier. using the Focusrite Scarlett
18i8 and a pair of AKG P420 microphones in a moderately bright, fully carpeted
room with asymmetric geometry and low background noise level. We use
Disklavier to refer to this subset.

  2. 2.

To simulate more challenging real-world environments, we further add
perturbations using different levels of reverberation and noise (see Section
6). We will refer to this subset as revnoise.

We compare three state-of-the-art piano transcription models: Onsets and
Frames [7] and the Transformer transcription model [3] by Google/Magenta
(which we will refer to as OaF and T5 respectively, in the following), and the
high-resolution onset and offset regression model by Bytedance [2] (referred
to as Kong).

We transcribe all the recordings in our MAESTRO, Disklavier, and revnoise
subsets using the (officially provided) trained models, and these
transcriptions then form our _evaluation set_. Note that all audio recordings
from the Disklavier and revnoise subsets are only used for testing; we use the
MAESTRO-trained models as they are provided by the respective authors via
their repositories.

Figure 1: Model performance comparison as evaluated on note-offset F1 score
and our proposed musical metrics, by composer.

##  5 Demonstration of musically informed metrics

We now discuss the experimental results obtained with the three systems and
explain the relation between our metrics and the standard IR metrics as
computed on transcriptions of the MAESTRO subset of our evaluation set. We
focus in particular on the musical dimensions that can be better understood
through our metrics. For the standard evaluation, we include the frame-level
score, and all note-level F1 scores other than the onset-only as they do not
capture offset and velocity information. We compute the note-level metrics
with the official mir_eval python implementation [26].

We start our discussion with Table 2, which summarizes the evaluation results
per model and metric. For comparative reasons, we also include a perceptually
informed piano transcription metric, PEAMT [5].

Metric | OaF | Kong | T5
---|---|---|---
Frame F1 | 0.8710 | 0.9138 | 0.7048
Note Offset F1 | 0.6167 | 0.8736 | 0.6358
Note Offset Velocity F1 | 0.5917 | 0.8587 | 0.6309
Melody IOI | 0.2377 | 0.5481 | 0.2217
Accompaniment IOI | 0.2168 | 0.3679 | 0.4329
Melody KOR | 0.4057 | 0.7415 | 0.2825
Bass KOR | 0.2638 | 0.6967 | 0.2672
Ratio KOR | 0.4247 | 0.6938 | 0.3094
Cloud Diameter | 0.7240 | 0.8301 | 0.7472
Cloud Momentum | 0.2461 | 0.2250 | 0.1671
Dynamics | 0.5501 | 0.6503 | 0.6355
PEAMT [5] | 0.6570 | 0.6241 | 0.5789
Table 2: Performance of the three models using the standard metrics, our
musically informed metrics, and the perceptually informed PEAMT metric [5] as
computed on the MAESTRO subset of our _evaluation set_.

Generally, it can be observed that the Kong model performs the best across
most metrics. This implies that most of our metrics, overall, correlate with
the performance ranking as measured on the standard metrics. Furthermore, it
can be seen that the two Magenta models perform significantly different when
measured against frame-level F1 score, yet this difference becomes less
pronounced when evaluated on note-level metrics, which would suggest superior
performance of the T5 model. Comparing these results to the model performance
as evaluated on our set of metrics, however, reveals that while both models
perform similarly on onset time prediction, the T5 model is worse at
adequately capturing note durations, particularly in lower voices/frequency
ranges, but superior in estimating MIDI velocity and the overall loudness
ratio between voices than the OaF model. Lastly, we can observe that the
perceptually informed PEAMT metric correlates most with the frame-level and
harmony metric _Cloud Momentum_ , which might suggest (if PEAMT is indeed a
veridical listening model) that listeners place relatively high importance on
harmonic context.

Figure 2: Relationship between model performance evaluated on note-offset-velocity F1 score and our proposed dynamics measure. |  | $\mathbf{frame}$ | $\mathbf{note_{off}}$ | $\mathbf{note_{off-vel}}$
---|---|---|---|---
split | model/audio | OaF | Kong | T5 | OaF | Kong | T5 | OaF | Kong | T5
train | MAESTRO | 0.8807 | 0.9207 | 0.7262 | 0.6183 | 0.8899 | 0.6350 | 0.5929 | 0.8756 | 0.6308
Disklavier | 0.8185 | 0.8508 | 0.6157 | 0.5269 | 0.7384 | 0.5132 | 0.4853 | 0.6660 | 0.4663
validation | MAESTRO | 0.8404 | 0.8936 | 0.6546 | 0.6492 | 0.8617 | 0.7117 | 0.6236 | 0.8471 | 0.7063
Disklavier | 0.7696 | 0.8678 | 0.6093 | 0.5539 | 0.8142 | 0.6570 | 0.5161 | 0.7480 | 0.6031
test | MAESTRO | 0.8527 | 0.9002 | 0.6552 | 0.5931 | 0.8215 | 0.5968 | 0.5695 | 0.8041 | 0.5896
Disklavier | 0.8049 | 0.8530 | 0.6048 | 0.4989 | 0.6979 | 0.5135 | 0.4607 | 0.6304 | 0.4624
Table 3: Frame-, note-offset, and note-offset-velocity F1 score results
computed on our _evaluation set_ , grouped per data set split, evaluated model
and piano / audio environment.

We continue our discussion in Figure 1, which compares the note-offset F1
score per composer (averaged over pieces and performers) and model to our
musically informed timing, articulation and harmony metrics. We can see again
that the Kong model performs best across all composers and metrics except for
the _Accompaniment IOI_ timing and the _cloud momentum_ harmony metrics. The
fact that it performs better on the _Bass KOR_ articulation metric, but poorly
on those timing accompaniment and harmony metrics might suggest that this
model detects many out-of-key extra notes that both erroneously influences the
IOI sequence on the accompaniment part, and the estimation of the tonal
context. Interestingly, we can can also observe, as a general trend, that the
F1 score somewhat deteriorates with increasing virtuoso and challenging
musical repertoire. This general deterioration with increasing musical
difficulty is not reflected correspondingly by our metrics, which show more
variation with respect to different composers and aspects of the underlying
music: While the F1 score for Bach, Mozart, Haydn and Schubert all suggest a
near-perfect transcription, our metrics indicate more diverse results, and
e.g., suggest poor(er) accuracy (and therefore reliability in a performance
study context) in the _Melody IOI_ and _Melody KOR_ metrics. Another
illustrative example can be found in the case of Chopin: here again the F1
score (particularly of the Kong model) would suggest a highly accurate
transcription output, while our metrics reveal that the expressive dimension
of articulation is not well captured. Lastly, we can observe again that the
PEAMT and harmony metric _Cloud momentum_ show a similar trend for most
composers, suggesting a greater weight of the harmonic context in that trained
metric.

We conclude our discussion by examining the dynamics aspect. Figure 2
illustrates the relationship between the note-offset-velocity F1 score and our
proposed _dynamics_ metric. While both metrics show a weak correlation
(Pearson $r=0.21$), the figure also indicates that our metric evaluates
dynamics in a more differentiated way and leads to a wider range of evaluation
results than the standard metric. Note that our metric only evaluates the
dynamics aspect, in particular how well the overall balance in loudness
between different voice streams is preserved in a transcription. It does not
account for onset, offset and pitch information, which also explains the
results in the very left part of the figure that score low on F1 score but
high on our metric.

##  6 Out-of-distribution inference

In this section we illustrate the problem of out-of-distribution performance
of the models analysed. We believe that this is an important aspect to
emphasize, as transcription models are ultimately intended to be (and have
been) used on real-world audio performances [9, 10].

We approach the problem in two stages: First, we elucidate the problem by
performing a short evaluation of the three analysed models on real-world
recordings using only the standard IR metrics. Second, we simulate more
challenging real-world environments using different levels of noise and
reverberation, and evaluate the analysed models again using the standard and
our proposed metrics, where we highlight how our musically informed metrics
can reveal aspects that the standard metrics would otherwise have missed.

###  6.1 Generalization on real-world recordings

Table 3 shows the mean frame- and note-level F1 scores per model and per
piano/acoustic recording environment on the three splits of the MAESTRO
dataset (as they are officially defined).777We note that for the two Magenta
models, OaF and T5, our evaluation results do not come close to the reported
ones in [7] and [3]. The differences are particularly pronounced in the note-
level F1 scores.

We group the results per split to test whether the analysed models would
perform worse on out-of-distribution recordings of performances (pieces) from
the test set compared to those from the train/validation sets. Next we conduct
a Kruskal Wallis ANOVA[27] to test for differences between frame-, note-offset
and note-offset-velocity F1 scores, grouping the evaluation scores each by
split and by audio environment and comparing each model separately. For each
ANOVA we use a significance threshold of $\alpha=0.05$. The ANOVA on the audio
environment dimension show a statistically significant difference ($p<0.05$)
between the MAESTRO and Disklavier audio recordings for all three analysed
models.

The ANOVA on the split dimension yields more differentiated results: For all
models, the frame-level F1 scores are significantly different, and there are
no statistically significant differences in the note-offset-velocity F1
scores. For the model by Kong, the note-offset-score is significantly
different depending on the split, whereas the two Magenta models show no
significant differences. This suggests that the most musically meaningful
metric from the current set of standard metrics[1] does not sufficiently
capture overfitting tendencies.

###  6.2 Evaluation on perturbed audio recordings

Similar as in Section 5, we again compare our musically informed metrics to
the standard IR and the PEAMT metrics, however, this time on a set of more
challenging audio recordings. To this end, we choose six (MAESTRO and
Disklavier) audio recordings which we artificially perturb by introducing
reverberation and synthetic noise. We use three Impulse-Response filters,
modelling short, medium and long reverberation times (RT60@1kHz
$\in\\{0.19,1.85,10.5\\}$ seconds) and sourced from the
OpenAIR888https://www.openair.hosted.york.ac.uk database. We further add white
noise into the recordings at three different Signal-to-Noise Ratio levels
($\text{SNR}_{dB}\in\\{24,12,6\\}$). Following a factorial design with these
two independent variables, each with four levels, we first perturb the audio
recordings on all conditions, and transcribe these recordings using all three
analysed models. Following this procedure, we obtain 284 transcribed MIDI
performances.999Note that 6 pieces x 4 noise levels x 4 reverberation levels x
3 models yield 288 transcriptions, but 4 recordings (each at the two higher
most of either reverberation and/or noise levels) resulted in empty
transcriptions (zero predicted note events) by the T5 model, and are therefore
excluded from the evaluation.

Each grid cell in Figure 3 compares the mean note-offset F1 scores per model
to the _Melody IOI_ timing metric and _cloud momentum_ harmony metric, where
grid rows represent increasing reverb levels, and grid columns represent
increasing noise levels. Generally, it can be observed that the performance
range of models as measured by the F1 score is notably reduced compared to our
metrics, indicating that our metrics possess higher discriminative capacity
than the standard ones.

As expected, the inference performance of all three analysed models
deteriorates with increasing noise and reverberation levels, though the
deterioration is less pronounced on the noise than on the reverberation axis.
Furthermore, analysing the results on the timing metric _Melody IOI_ suggests
that the model by Kong predicts onset times worse with increasing noise
levels, while the onset times prediction by OaF seems to be more resistant to
this form of perturbation. Finally, the results measured on the harmony metric
_Cloud momentum_ suggest that the overall harmonic context is relatively well
preserved at higher perturbation levels by the OaF and Kong models, and less
so by the T5 model.

Figure 3: Performance degradation analysis as evaluated on the note-offset F1
score and our proposed _Melody IOI_ timing and _Cloud momentum_ harmony
metric.

##  7 Conclusion

In this study, we investigated two aspects that are commonly neglected in the
evaluation of transcription models: (i) limited explanatory power of the
standard IR evaluation metrics with respect to the underlying musical
material, and (ii) poor inference on out-of-distribution data. We study both
problems in the context of solo piano transcription, and, in addressing the
first aspect, propose a set of musically informed metrics designed to capture
more musically relevant information, particularly for the context of
computational studies of expressive performance.

We demonstrated our metrics on transcriptions obtained by three state-of-the-
art piano transcription models on a subset of the MAESTRO dataset, the de-
facto standard train and test set for current transcription models, and
highlighted musical dimensions for which they provide more informative value
than the standard information retrieval metrics. We have further illustrated
the lack of generalization with respect to the acoustic environment, both on
real-world and perturbed audio recordings. We plan to extend our musically
informed metrics to capture more qualities of expressive performance, possibly
using score alignment information.

##  8 Acknowledgments

This work receives funding from the European Research Council (ERC), under the
European Union’s Horizon 2020 research and innovation programme, grant
agreement No. 101019375 (Whither Music?). The LIT AI Lab is supported by the
Federal State of Upper Austria.

## References

  * [1] C. Hawthorne, E. Elsen, J. Song, A. Roberts, I. Simon, C. Raffel, J. Engel, S. Oore, and D. Eck, “Onsets and frames: Dual-objective piano transcription,” _Proceedings of the International Society for Music Information Retrieval Conference (ISMIR)_ , 2018.
  * [2] Q. Kong, B. Li, X. Song, Y. Wan, and Y. Wang, “High-resolution piano transcription with pedals by regressing onset and offset times,” _IEEE/ACM Transactions on Audio Speech and Language Processing_ , vol. 29, pp. 3707–3717, 2021.
  * [3] C. Hawthorne, I. Simon, R. Swavely, E. Manilow, and J. Engel, “Sequence-to-sequence piano transcription with transformers,” _Proceedings of the International Society for Music Information Retrieval Conference (ISMIR)_ , 2021.
  * [4] M. Bay, A. F. Ehmann, and J. S. Downie, “Evaluation of multiple-f0 estimation and tracking systems.” in _Proceedings of the International Society for Music Information Retrieval Conference (ISMIR)_ , 2009, pp. 315–320.
  * [5] A. Ycart, L. Liu, E. Benetos, and M. Pearce, “Investigating the perceptual validity of evaluation metrics for automatic piano music transcription,” _Transactions of the International Society for Music Information Retrieval (TISMIR)_ , 2020.
  * [6] W.-T. Lu, J.-C. Wang, and Y.-N. Hung, “Multitrack music transcription with a time-frequency perceiver,” in _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2023, pp. 1–5.
  * [7] C. Hawthorne, A. Stasyuk, A. Roberts, I. Simon, C. A. Huang, S. Dieleman, E. Elsen, J. H. Engel, and D. Eck, “Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset,” in _International Conference on Learning Representations, (ICLR)_ , 2019.
  * [8] J. Gardner, I. Simon, E. Manilow, C. Hawthorne, and J. Engel, “MT3: Multi-task multitrack music transcription,” _International Conference on Learning Representations, (ICLR)_ , 2022.
  * [9] Q. Kong, B. Li, J. Chen, and Y. Wang, “Giantmidi-piano: A large-scale midi dataset for classical piano music,” _Transactions of the International Society for Music Information Retrieval (TISMIR)_ , 2022.
  * [10] H. Zhang, J. Tang, S. Rafee, S. Dixon, G. Fazekas, G. Wiggins _et al._ , “Atepp: A dataset of automatically transcribed expressive piano performance,” _Proceedings of the International Society for Music Information Retrieval Conference (ISMIR)_ , 2023.
  * [11] A. Daniel, V. Emiya, and B. David, “Perceptually-based evaluation of the errors usually made when automatically transcribing music,” in _Proceedings of the International Society for Music Information Retrieval Conference (ISMIR)_ , 2008.
  * [12] A. McLeod and M. Steedman, “Evaluating automatic polyphonic music transcription.” in _Proceedings of the International Society for Music Information Retrieval Conference (ISMIR)_ , 2018, pp. 42–49.
  * [13] V. Emiya, N. Bertin, B. David, and R. Badeau, “MAPS - A piano database for multipitch estimation and automatic transcription of music,” Research Report, Jul. 2010. [Online]. Available: https://inria.hal.science/inria-00544155
  * [14] S. Sigtia, E. Benetos, and S. Dixon, “An end-to-end neural network for polyphonic piano music transcription,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 24, no. 5, pp. 927–939, 2016.
  * [15] D. Temperley, “A Unified Probabilistic Model for Polyphonic Music Analysis,” _Journal of New Music Research_ , vol. 38, no. 1, pp. 3–18, 2009. [Online]. Available: https://doi.org/10.1080/09298210902928495
  * [16] A. Uitdenbogerd and J. Zobel, “Melodic matching techniques for large music databases,” in _Proceedings of the seventh ACM international conference on Multimedia (Part 1)_ , 1999, pp. 57–66.
  * [17] F. Simonetta, C. Cancino-Chacón, S. Ntalampiras, and G. Widmer, “A convolutional approach to melody line identification in symbolic scores,” _Proceedings of the International Society for Music Information Retrieval Conference (ISMIR)_ , 2019.
  * [18] R. Bresin and G. Umberto Battel, “Articulation strategies in expressive piano performance analysis of legato, staccato, and repeated notes in performances of the andante movement of mozart’s sonata in g major (k 545),” _Journal of New Music Research_ , vol. 29, no. 3, pp. 211–224, 2000.
  * [19] B. H. Repp, “Acoustics, perception, and production of legato articulation on a computer-controlled grand piano,” _The Journal of the Acoustical Society of America_ , vol. 102, no. 3, pp. 1878–1890, 1995.
  * [20] C. Drake and C. Palmer, “Accent structures in music performance,” _Music perception_ , vol. 10, no. 3, pp. 343–378, 1993.
  * [21] C. Cancino-Chacón and M. Grachten, “A computational study of the role of tonal tension in expressive piano performance,” _arXiv preprint arXiv:1807.01080_ , 2018.
  * [22] D. Herremans and E. Chew, “Towards emotion based music generation: A tonal tension model based on the spiral array,” in _The Annual Meeting of the Cognitive Science Society_ , 2019.
  * [23] D. Herremans, E. Chew _et al._ , “Tension ribbons: Quantifying and visualising tonal tension.” 2016.
  * [24] E. Chew, “Playing with the edge: Tipping points and the role of tonality,” _Music Perception: An Interdisciplinary Journal_ , vol. 33, no. 3, pp. 344–366, 2016.
  * [25] R. B. Dannenberg, “The interpretation of MIDI velocity,” in _ICMC_ , 2006.
  * [26] C. Raffel, B. McFee, E. J. Humphrey, J. Salamon, O. Nieto, D. Liang, D. P. Ellis, and C. C. Raffel, “MIR_EVAL: A Transparent Implementation of Common MIR Metrics.” in _Proceedings of the International Society for Music Information Retrieval Conference (ISMIR)_ , vol. 10, 2014, p. 2014.
  * [27] A. Vargha and H. D. Delaney, “The Kruskal-Wallis test and stochastic homogeneity,” _Journal of Educational and behavioral Statistics_ , vol. 23, no. 2, pp. 170–192, 1998.


# Under-actuated Robotic Gripper with Multiple Grasping Modes Inspired by
Human Finger

Jihao Li1,2, Tingbo Liao3, Hassen Nigatu1,2, Haotian Guo1,2, Guodong Lu2, and
Huixu Dong1,2,∗ 1Grasp Lab, School of Mechanical Engineering, Zhejiang
University, Hangzhou 310027, China.2Robotics Institute of Zhejiang University,
Hangzhou 310027, China.3College of Design and Engineering National University
of Singapore, 117575, Singapore. Email: {huixudong,
<EMAIL_ADDRESS>author

###### Abstract

Under-actuated robot grippers as a pervasive tool of robots have become a
considerable research focus. Despite their simplicity of mechanical design and
control strategy, they suffer from poor versatility and weak adaptability,
making widespread applications limited. To better relieve relevant research
gaps, we present a novel 3-finger linkage-based gripper that realizes
retractable and reconfigurable multi-mode grasps driven by a single motor.
Firstly, inspired by the changes occurred in the contact surface with a human
finger moving, we artfully design a slider-slide rail mechanism as the phalanx
to achieve retraction of each finger, allowing for better performance in the
enveloping grasping mode. Secondly, a reconfigurable structure is constructed
to broaden the grasping range of objects’ dimensions for the proposed gripper.
By adjusting the configuration and gesture of each finger, the gripper can
achieve five grasping modes. Thirdly, the proposed gripper is just actuated by
a single motor, yet it can be capable of grasping and reconfiguring
simultaneously. Finally, various experiments on grasps of slender, thin, and
large-volume objects are implemented to evaluate the performance of the
proposed gripper in practical scenarios, which demonstrates the excellent
grasping capabilities of the gripper.

##  I INTRODUCTION

As grasping is a crucial ability of robots, numerous robot grippers as
grasping executive tools have been developed to be applied in various fields,
such as logistics, warehousing scenarios, industrial settings, as well as
human-computer interaction [1, 2, 3]. Among various grippers, the full-
actuated grippers have improved dexterity, owing to the independent motion of
each joint [4]. However, it brings about issues in the high cost and
complexity of the mechanism and control. Therefore, under-actuated grippers
have attracted substantial attention, being advantageous in simplifying the
mechanical design, control strategy and enhancing grasping adaptability as
well as reducing the cost [5].

Figure 1: The prototype of presented gripper (A, B), grasp slender and thin
objects through the human finger (C) and the presented gripper (D). Figure 2:
A general three-phalanx finger consists of the metacarpophalangeal (MCP)
joint, proximal interphalangeal (PIP) joint and distal interphalangeal (DIP)
joint (A); A full-actuated three-phalanx finger (B); An under-actuated three-
phalanx finger (C); The prototype of the proposed three-phalanx finger in this
work (D). The parallel grasping mode (D-1) and the enveloping grasping mode
(D-2/D-3). The thick dotted line represents the driven linkage, and the dotted
line indicates that the linkage is retractable.

Prevalent under-actuated grippers can be broadly categorized as soft and rigid
types [6, 7]. The soft grippers are typically constructed via lightweight and
flexible materials, driven by tendons or pneumatic/hydraulic systems [8, 9,
10]. They prioritize safety, versatility, and adaptability in robotic
manipulation, offering a stable grasp of objects with complex contours. For
instance, Gunderman et al. [8] presented a tendon-driven soft robotic gripper
with passive compliance, while Wang et al. [9] presented a pneumatic-driven
gripper that is able to vary its stiffness and working length. Nevertheless,
they have a low load capacity and a short lifespan due to material properties.
In contrast, the structure of rigid grippers is dominated by linkage and gear
systems, which are known for their stiffness and precision [11, 12]. Toshihiro
et al. [13] utilized chuck clamping devices to create a gripper. A gripper
with fingers that are scissor linkages was proposed by Kwon et al. [11].
However, these grippers cannot adjust the grasping range and adapt to objects
with irregular shapes, resulting in unstable grasps. To alleviate the above
drawbacks, certain grippers were designed by mimicking the human finger to
flex and relax muscles for modifying contact surface areas [14]. This feature
significantly improves the graspable range and dexterity of the human hand,
allowing it to grasp even slender or thin objects with a single finger.
Notably, Bandara et al. [15] built an under-actuated gripper based on a
modified cross-bar mechanism. Abayasiri et al. [16] proposed a humanoid
robotic hand that employs a 5-linkage mechanism. Yet, these rigid grippers,
which are designed to imitate the motion of human fingers, cannot achieve
changes in contact surface caused by muscle contraction, which limits their
adaptability and envelopment capabilities. To enhance the flexibility and
adaptability of robot grippers, some grippers are designed with
reconfiguration mechanisms [17, 18]. However, most of the reconfigurable
grippers rely heavily on additional motors to rotate or translate the fingers,
resulting in a complex mechatronic system, such as the GTac gripper [17] and
the SMS gripper [18]. Therefore, one current challenge is to develop an under-
actuated gripper that has the capability of grasping objects with a large
range of objects’ dimensions and high adaptability. Using fewer motors to
achieve all the abilities of the gripper is another challenge.

To address the aforementioned issues, we propose a novel under-actuated
gripper with three compressible 3-fingers, as shown in Fig. 1. Firstly, the
proximal phalanx is composed of a parallel four-bar linkage mechanism to
achieve parallel grasping. In particular, each phalanx can realize adaptive
retraction through the designed slider-slide rail mechanism to perform
enveloping grasping mode, ensuring stable grasping of objects of various
volumes and shapes. Secondly, to minimize the number of gripper motors, a
transmission mechanism including worm, gears, and racks was artfully designed
to distribute the output of the motors. In addition, a slotted block
positioned proximal to the finger base is incorporated to constrain the
movement of the finger during reconfiguration. Thus, the proposed gripper can
achieve both grasping and reconfiguration abilities simultaneously through a
single motor. Finally, the presented gripper can actively adjust the grasping
range, and passively switch between parallel grasping and enveloping grasping
according to the shape of the objects, enabling the gripper to achieve up to
five distinct grasping modes. As empirically validated through
experimentation, the gripper exhibits notable traits, including elevated
rigidity, substantial load-bearing capacity, robust adaptability, and a broad
grasping range, which amalgamates the advantageous features of both rigid and
soft grippers.

We highlight the novelties of our work. Foremost, our core contribution to
this work is constructing a novel under-actuated gripper with capabilities of
high adaptability and a large grasping range of objects’ dimensions. The first
novelty is that this work presents an innovative linkage mechanism to achieve
the retraction of the phalanges for changing contact areas between the fingers
and the objects, thereby varying the friction to improve the grasping
stability. Although the proposed gripper is actuated by a single motor, it can
perform grasping and reconfiguring simultaneously, considerably reducing the
control and mechanism complexities, which is attributed to the second novelty.
The third novelty is the capability to achieve five grasping modes by
combining finger adaptability and reconfiguration. Different grasping modes
can be employed to grasp objects with various sizes and contours, enhancing
the stability of the grasp.

Figure 3: Kinematics model of the proximal phalanx and middle phalanx.

##  II METHODOLOGY

###  II-A Kinematic Analysis of the Under-actuated Finger

Three-joint fingers are typically 3 DOF and consist of the MCP joint, PIP
joint and DIP joint, they are individually driven by three active linkages, as
shown in Fig. 2-A. To facilitate the installation of the motors in fully
actuated grippers, the driving linkages are often concentrated at the MCP
joint, which is closer to the gripper’s palm, as illustrated in Fig. 2-B [19].
Besides, torsion springs and mechanical blocks can be placed at
$\textit{O}_{1}$,$\textit{O}_{2}$ and $\textit{O}_{3}$ to achieve an under-
actuated ability, thereby reducing the number of motors (see Fig. 2-C). The
proposed gripper utilizes the retractable linkage
$\textit{L}_{1}$,$\textit{L}_{2}$ and $\textit{L}_{3}$ to constrain the
movement of joints, as depicted in Fig. 2-D.

Without external contact, retractable linkages $\textit{L}_{1}$ and
$\textit{L}_{2}$ are in origin length. In this state, the proximal phalanx
($\textit{O}_{1}$$\textit{O}_{2}$$\textit{O}_{m}$$\textit{O}_{a}$) is
maintained in a flexible parallelogram mechanism by $\textit{L}_{1}$, while
the middle and distal phalanx combine to form a fixed five-bar mechanism
($\textit{O}_{2}$$\textit{O}_{3}$$\textit{O}_{d}$$\textit{O}_{c}$$\textit{O}_{m}$).
Since $\beta$ is a fixed right angle, the orientation of the middle and distal
phalanx will remain perpendicular to $\textit{O}_{2}$$\textit{O}_{m}$ as the
finger moves. This ensures the parallel grasping mode of the gripper, as shown
in Fig. 2-D-1.

Figure 4: Initial state (A); Parallel grasp (B) and Enveloping grasp (C, D);
Maximum flexion state (E); The retractable ability of the distal phalanx (F).

The angle of $\textit{$\theta$}_{1}$ would be fixed whenever the proximal
phalanx makes contact with the objects. The movement of the driven linkage
compresses $\textit{L}_{1}$, causing it to change its length.The shape of the
proximal phalanx changes, resulting in a shift in the orientation of the
middle phalanx. The middle and distal phalanx remain coupled due to the
constraint of the $\textit{L}_{2}$, forming a fixed five-bar mechanism. The
gripper switches to the enveloping grasping mode, as indicated in Fig. 2-D-2.

Similarly, when the middle phalanx comes into contact with the object, the
angle of $\textit{$\theta$}_{2}$ is fixed, while $\textit{L}_{2}$ will be
compressed. The distal phalanx and middle phalanx will be decoupled. Thus, the
distal phalanx can move independently. A finger can bend further to grasp
thinner objects, as demonstrated in Fig. 2-D-3.

Here we construct the mathematical relationship between the angle of each
joint and the length of each phalanx after retracting. Taking the proximal
phalanx shown in Fig. 3-A as an example, we analyze the length of the
retractable linkage $\textit{L}_{1}$ after a finger moves. The vector-loop
equation is provided as

$\displaystyle\overrightarrow{\textit{O}_{1}\textit{O}_{a}}+\overrightarrow{\textit{O}_{a}\textit{O}_{m}}+\overrightarrow{\textit{O}_{m}\textit{O}_{2}}=\overrightarrow{\textit{O}_{1}\textit{O}_{2}}$
(1)

The phalanx is composed of a four-bar linkage. Except for the retractable
linkage $\textit{L}_{1}$, the length of other linkages ($\textit{L}_{1a}$,
$\textit{L}_{1b}$, $\textit{L}_{1c}$) is a constant length. And $\alpha$ can
be calculated based on the rotation angle of the finger,

$\displaystyle\alpha=\beta-\theta_{2}$ (2)

A coordinate is built at $\textit{O}_{1}$, with
$\textit{O}_{2}$$\textit{O}_{1}$ representing the positive direction of the
x-axis. Thus, the coordinates of $\textit{O}_{2}$ ($\textit{o}_{2x}$,
$\textit{o}_{2y}$), $\textit{O}_{m}$ ($\textit{o}_{mx}$, $\textit{o}_{my}$)
and $\textit{O}_{a}$ ($\textit{o}_{ax}$, $\textit{o}_{ay}$) can be obtained
sequentially.

$\displaystyle\textit{o}_{2x}=-\textit{L}_{1},\textit{o}_{2y}=0$ (3)
$\displaystyle\textit{o}_{mx}=-\textit{L}_{1}-\textit{L}_{1c}\cos{(\alpha)},\textit{o}_{my}=\textit{L}_{1c}\sin{(\alpha)}$
(4)
$\displaystyle\textit{o}_{ax}=-\textit{L}_{1a}\cos{({\theta}_{1})},\textit{o}_{ay}=\textit{L}_{1a}\sin{({\theta}_{1})}$
(5)

To determine $\textit{L}_{1}$, we substitute Eq. (3)-(5) into Eq. (1)

$\begin{split}\textit{L}_{1b}^{2}=(\textit{L}_{1}&+\textit{L}_{1c}\cos{(\alpha)}-\textit{L}_{1a}\cos{({\theta}_{1})})^{2}\\\
&+(\textit{L}_{1a}\sin{({\theta}_{1})}-\textit{L}_{1c}\sin{(\alpha)})^{2}\end{split}$
(6)

After algebraic manipulation, the quadratic polynomial representing
$\textit{L}_{1}$ can be derived

$\begin{split}\textit{L}_{1}^{2}+\textit{b}_{1}\textit{L}_{1}&+\textit{c}_{1}=0\\\
\textit{b}_{1}=2\textit{L}_{1c}\cos{(\alpha)}&-2\textit{L}_{1a}\cos{({\theta}_{1})}\\\
\textit{c}_{1}=\textit{L}_{1a}^{2}+\textit{L}_{1b}^{2}+\textit{L}_{1c}^{2}-&2\textit{L}_{1a}\textit{L}_{1c}\cos{(\alpha-{\theta}_{1})}\end{split}$
(7)

Figure 5: Part connected to the slider (A-1). Part connected to the slide rail
(A-2). Retractable linkage at its original length (B-1). Retractable linkage
after being compressed (B-2).

Therefore, the retractable linkage $\textit{L}_{1}$ can be calculated by

$\displaystyle\textit{L}_{1}=-\frac{1}{2}(\textit{b}_{1}\pm\sqrt{\textit{b}_{1}^{2}-4\textit{c}_{1}})$
(8)

In Fig. 3-B, the linkage $\textit{L}_{2a}$, $\textit{L}_{2b}$,
$\textit{L}_{2c}$ and angle $\kappa$ are constant. Similarly, the length of
$\textit{L}_{2}$ can be calculated by

$\begin{split}\delta=\kappa&-\theta_{3}\\\
\textit{b}_{2}=2\textit{L}_{2c}\cos{(\delta)}&-2\textit{L}_{2a}\cos{({\beta})}\\\
\textit{c}_{2}=\textit{L}_{2a}^{2}+\textit{L}_{2b}^{2}+\textit{L}_{2c}^{2}&-2\textit{L}_{2a}\textit{L}_{2c}\cos{(\delta-\beta)}\\\
\textit{L}_{2}=-\frac{1}{2}(\textit{b}_{2}\pm&\sqrt{\textit{b}_{2}^{2}-4\textit{c}_{2}})\end{split}$
(9)

The solutions with negative signs are used for determining the lengths of
$\textit{L}_{1}$ and $\textit{L}_{2}$.

As previously stated, the inside linkage of each phalanx is the retractable
linkage. As shown in Fig. 5-A, each retractable linkage comprises two parts:
one part is connected to the MGN7C slider (17×22.5×6.5 mm) and the other part
is connected to the MGN7 slide rail (7×4.8 mm). The two parts are constrained
by the tension springs. The springs remain unstretched when there is no
external force, and the retractable linkage retains its original length. When
the phalanx is under stress, the tension springs gradually stretch as the
retractable linkage compresses (see Fig. 5-B). To ensure smooth movement of
the slider, the springs are installed symmetrically on both sides of the slide
rail so that the resultant force is always along the direction of the slide
rail and the resultant moment is 0. The types and parameters of the springs at
each phalanx are:

  * •

MCP joint : tension spring, $\textit{K}_{MCP}$ = 1 N/mm

  * •

PIP joint : tension spring, $\textit{K}_{PIP}$ = 0.8 N/mm.

  * •

DIP joint : compression spring, $\textit{K}_{DIP}$ = 0.55 N/mm.

###  II-B Design of the Under-actuated Finger

The linkage length parameters for this finger are shown in Table I. Table II
illustrates the theoretical parameters for the three retractable linkages
after compression. Except for changes in linkage length, the flexion and
mutual occlusion of the linkages will further affect the length of the finger
contact surface. The actual contact surface length parameters of the phalanges
are shown in Table III. It can be found that the maximum compressibility of
the phalanges is 57 mm. Which is almost one-third of the original length. The
actual contact length can be reduced by 74 mm. The minimum contact length is
57.95% of the original length.

TABLE I: Linkage parameter Linkage | Length/mm | Linkage | Length/mm
---|---|---|---
$\textit{D}_{1}$ | 85 | $\textit{L}_{2}$ | 55
$\textit{D}_{2}$ | 68 | $\textit{L}_{2a}$ | 30
$\textit{L}_{1}$ | 70 | $\textit{L}_{2b}$ | 76
$\textit{L}_{1a}$ | 70 | $\textit{L}_{3}$ | 51
$\textit{L}_{1b}$ | 30 | $\textit{L}_{3a}$ | 29
TABLE II: Analysis of linkage length

Linkage |  $\textit{L}_{o}$/mm |  $\textit{L}_{c}$/mm |  $\Delta$L/mm | $\textit{R}_{theory}$
---|---|---|---|---
$\textit{L}_{1}$ | 70 | 46 | 24 | 34.29$\%$
$\textit{L}_{2}$ | 55 | 36 | 18 | 32.73$\%$
$\textit{L}_{3}$ | 51 | 36 | 15 | 29.41$\%$
Total | 176 | 119 | 57 | 32.39$\%$

$\textit{L}_{o}$, $\textit{L}_{c}$, $\Delta$L represent the original length,
the shortest length after compression, and the change in length of each
linkage, respectively. $\textit{R}_{theory}$ represents the theory ratio of
the change in length.

TABLE III: Analysis of contact length

Phalanx |  $\textit{S}_{o}$/mm |  $\textit{S}_{c}$/mm |  $\Delta$S/mm | $\textit{R}_{practice}$
---|---|---|---|---
Proximal | 70 | 40 | 30 | 42.86$\%$
Middle | 55 | 26 | 29 | 52.73$\%$
Distal | 51 | 36 | 15 | 29.41$\%$
Total | 176 | 102 | 74 | 42.05$\%$

$\textit{S}_{o}$, $\textit{S}_{c}$, $\Delta$S represent the original contact
length, the shortest contact length after compression, and the change in
contact length of each phalanx, respectively. $\textit{R}_{practice}$
represents the practice ratio of the change in contact length.

During the actual grasping process, in the absence of external contact, the
finger will keep in parallel grasping mode, as shown in Fig. 4-B. In Fig. 4-C,
until the finger contacts the object, the proximal phalanx will retract first.
As demonstrated in Fig. 4-D, once the middle phalanx contacts the object, it
will start to retract, resulting in the decoupling of the middle phalanx and
distal phalanx. Moreover, the distal phalanx also has a retractable ability
(see Fig. 4-F), which is serving mainly for grasping thin objects and does not
participate in enveloping grasping. When grasping thin objects, this
retractable linkage ensures that the fingertips are always in close contact
with the object placement surface, making it easier to pick up thin objects.

Figure 6: 3D modeling(A) and schematic diagram(B) of gear set.

###  II-C Design of the Reconfiguration Mechanism

The proposed gripper has three fingers arranged crosswise on both sides of the
gripper, allowing the gripper to perform grasping and reconfiguration
abilities simultaneously with a single motor. The detail of the
reconfiguration mechanism is described below:

To drive the fingers distributed on both sides to perform correct motions, a
gear set is introduced which is used to allocate the power, as shown in Fig.
6-A. The gear set consists of a set of worm gears with a reduction ratio of
15:1 and several gears with 15 teeth in 1 mold (represented by g) and 30 teeth
in 1 mold (represented by G). The ultimate goal of the gear set is to achieve
the forward rotation of $\textit{G}_{1}$ and $\textit{G}_{3}$ to drive the two
fingers on one side, and reverse rotation of $\textit{G}_{2}$ to drive the
fingers on the other side. The absolute values of the angular velocities of
the three gears are equal, so that the fingers on both sides can flex toward
each other at the same speed.

Following the schematic diagram of the gear set revealed in Fig. 6-B, these
gears are arranged in three planes. The gears between different planes are
connected by shafts. The same color of the gears implies that the gears are on
the same plane. The same color of the gear axis means that the gears are
coaxial. The $\otimes$ symbol on the axis represents that the gear is fixed to
the shaft and the angular velocity is the same as the shaft, while the $\odot$
symbol on the axis indicates that the gear is only concentric with the shaft
and the rotation state is independent of the axis. The motor (proServo-planet
PP08D) connected to the worm has a maximum speed of 120rpm and a nominal
torque of 6.6Nm. Gear $\textit{g}_{4}$ is connected to the worm gear, allowing
the motor torque to be increased 30 times to 198 Nm. The final motion of each
gear is depicted in the figure.

Figure 7: 3D modeling of the reconfiguration components (A, B), cross-
sectional view of the rack group (C), each part of the rack corresponds to a
specific controlled gripping mode (D), self-locking mechanism of
reconfiguration components (E) and tension spring of reconfiguration
components (F).

Slide rails are installed on the base of the gripper finger. After assembly on
the slide block, the entire translation motion of the finger can be realized,
as shown in Fig. 7-A/B. Each transmission system consists of rack R (tooth
down) and rack r (tooth up), they are installed in line and mesh with G and
$\textit{g}_{F}$, respectively (see Fig. 7-C). G represents the three gears
($\textit{G}_{1}$, $\textit{G}_{2}$, $\textit{G}_{3}$) of the final output of
the gear set, and $\textit{g}_{F}$ indicates the gear installed in each finger
base. The rotation of the motor first drive G to translate the rack set. Then,
the rack set will drive $\textit{g}_{F}$ to rotate. Finally, $\textit{g}_{F}$
will actuate the $\textit{D}_{1}$ (the driven linkage of each finger),
enabling the fingers to move. In addition, a tension spring is installed
between the finger base and the palm to constrain the reconfiguration of the
finger (see Fig. 7-F). The overall prototype of the proposed gripper is
illustrated in Fig. 8. Considering the viewpoint demonstrated in Figure 7-C as
a model, we analyze the opening and closing motion of the gripper.

When G rotates clockwise, it will drive the rack R and r to translate to the
right. Due to the constraint of the tension spring, the finger base will
initially remain stationary. Thus, r will drive $\textit{g}_{F}$ to rotate
counterclockwise and linkage $\textit{D}_{1}$ to rotate clockwise. This
sequence of actions ultimately leads to the opening of the gripper. Similarly,
the counterclockwise motion of G will close the gripper. However, when the
gripper is fully open, the clockwise rotation of linkage $\textit{D}_{1}$ is
limited. Then, r will overcome the constraint of the tension spring and drive
the entire finger base to move to the right, achieving reconfigurability. It
should be noted that after the finger base moves to the far-right end, further
counterclockwise motion of $\textit{g}_{F}$ will: 1) move the finger base to
the left due to the tension of the spring, 2) actuate the finger performing
the closing motion. To achieve the closing action at the far-right end, a
self-locking mechanism is introduced for limiting the reverse movement of the
finger base, as depicted in Fig. 7-E. This mechanism consists of a lock block,
a slotted block and several compression springs. The top and bottom of the
lock block are connected to the compression spring.

Figure 8: The prototype of the proposed gripper (A), perspective view of
driven components (B), the origin length of each phalanx (C), maximum grasping
range of parallel grasping mode (D) and minimum grasping range of enveloping
grasping mode (E). Figure 9: Self-locking flow chart of self-locking
mechanism. The circle represents the part of the lock block that extends into
the slotted block. The arrow indicates the movement direction of the lock
block. The white thin dotted line is the equilibrium position of the lock
block. This figure shows the state of the spring on the upper side of the lock
block.

The specific self-locking process of the mechanism is shown in Fig. 9:

• Locking stage (Fig. 9-A): The lock block is in an equilibrium position
before entering the slotted block. After entering the slotted block, it will
move upward along the upper groove, compressing the spring accordingly. When
the lock block crosses the top of the incline, the spring pushes it down to
the red position. If it changes to move in the opposite direction at this
time, it will be restricted by the slotted block and cannot return.
Correspondingly, if the finger base stops here, even if G rotates
counterclockwise, the finger base will not be able to move to the left due to
the self-locking mechanism. Therefore, r will give priority to driving linkage
$\textit{D}_{1}$, realizing the closing motion of the finger at the
reconfigured position.

• Unlocking stage (Fig. 9-B): When the lock block is at the red position,
continue moving to the right, the lock block will enter the groove below. If
the movement is reversed at this moment, the lock block will move along the
groove below and will no longer be constrained by the slotted block.

• Returning stage (Fig. 9-C): When the lock block escapes the slotted block,
it will return to the equilibrium position with the springs returning to the
original length. Rotating G counterclockwise will return the finger to its
original position.

Figure 10: Grasping range and grasping procedure of each grasp mode: proximal
parallel grasp mode(A), proximal enveloping grasp mode(B), translational grasp
mode(C), remote parallel grasp mode(D) and remote enveloping grasp mode(E).

###  II-D Description of Multi-grasping Mode

Combining the described grasping modes of the finger with the reconfiguration
of the finger base, the proposed gripper can achieve a total of five grasping
modes. As the gripper is actuated by a single motor, the control strategy is
relatively straightforward and does not require an additional motor. Each
grasping mode is described as follows:

• Mode 1 (proximal parallel grasp mode): Parallel grasping of fingers in the
non-reconfigured state, which is suitable for grasping small regular and thin
objects.

• Mode 2 (proximal enveloping grasp mode): Enveloping grasping of the fingers
in the non-reconfigured state, which is suitable for grasping slender and
small irregular objects.

• Mode 3 (translational grasp mode): The finger uses the translation of the
base to achieve grasping. In this mode, the fingertips can only move
horizontally, which is useful for grasping thin objects when the fingertips
need to be close to the work surface.

• Mode 4 (remote parallel grasp mode): Parallel grasping of the fingers after
reconfiguration, which is suitable for grasping large regular objects.

• Mode 5 (remote enveloping grasp mode): Enveloping grasping of the fingers
after reconfiguration, which is suitable for grasping large irregular objects.

When G meshes with part a of the rack in Fig. 7-D, the gripper is in proximal
grasping mode. The forward rotation of the motor will control the gripper to
open, while the reverse rotation will close it. Notably, the adaptability of
fingers enables the gripper to autonomously switch between Mode 1 and 2 based
on the shape of the object.

When the gear meshes with part b-1 of the rack, the motor’s forward rotation
will achieve translational grasping mode (Mode 3).

As soon as the gear G meshes with the red dotted line, the self-locking
mechanism completes the locking and the gripper completes reconfiguration. On
the one hand, the self-locking mechanism unlocks when the motor rotation is
reversed, coinciding with gear G meshing with the far right end of part b-2.
The finger will return to its original state from reconfiguration. On the
other hand, if reverse the motor immediately after the gear meshes with the
red dotted line, the self-locking mechanism will stay locking. Following this,
the gear shifts to mesh with part c of the rack, prompting the gripper to
change to a mode suitable for remote grasping. By keeping the gear meshed with
rack part c, the gripper automatically alternates between Mode 4 and 5 for
grasping, depending on the object’s shape. Fig. 10 shows the grasping range
and the practical application of each grasping mode.

Figure 11: Several demonstrations to verify the grasping performances in
different Mode: grasp objects with regular shapes (A); grasp objects with
regular planes (B); grasp objects with regular arc surfaces (C); grasp objects
with irregular contours (D); grasp slender objects with one finger (E).

##  III EXPERIMENTS

To evaluate the grasping performance of the proposed gripper, a prototype
three-finger gripper was constructed. A series of experimental validations
followed: Firstly, confirmation of the implementation of five grasping modes
was conducted. Subsequently, the adaptability of the fingers and the
reconfiguration ability of the gripper were verified by grasping regular
objects and daily necessities, with the contracted length of each phalanx
being measured.

###  III-A Grasps at Different Grasping Modes

In the first experiment, the gripper was employed to grasp a variety of
objects utilizing five distinct grasping modes. Targets include grasping 60 mm
cylinder, 40 mm cube, 125 mm cube, 120 mm cylinder and 80 mm cube in five
modes. Detailed grasping procedures are demonstrated in Fig. 10.

During the actual grasping process, it was observed that in the enveloping
grasping modes (Mode 1 & Mode 4), all three joints are involved in grasping
objects, resulting in a more stable grasping of the object. Whereas, in the
parallel grasping modes (Mode 2 & Mode 3 & Mode 5), mainly DIP joints and PIP
joints are engaged in grasping objects, leading to precise grasping
performance, particularly in the translational grasping mode where the
fingertips will not change height in the vertical direction, which enables
fixed-point grasping of objects. From another perspective, the switching
between parallel grasping mode and enveloping grasping mode (switching between
Mode 1 & Mode 2 and switching between Mode 4 & Mode 5) is achieved through the
retraction of phalanx, which passively adapts the MCP and DIP joints to the
contour of the object. In contrast, the reconfiguration mechanism can actively
adjust the grasping range by transitioning among the proximal grasping mode,
translational grasping mode, and remote grasping mode.

Eventually, the grasping ranges in the five modes are: [0,127]mm, [16,127]mm,
[127,177]mm, [0,177]mm, and [34,177]mm. It is noteworthy that although both
Mode 1 and Mode 4 fingertips can be completely closed, Mode 4 has a large
hollow area after the gripper closes (as marked in the blue area in Fig.
10-D). As a result, grasping small objects may be inhibited in this Mode 4.

Figure 12: Grasping is achieved with a single finger through the retractable
of the distal phalanx.

###  III-B Grasp Objects with Regular Shapes and Daily Necessities

The proposed gripper was mounted on the 6DOF industrial manipulator-UR3.
Subsequent to this, a sequence of grasping experiments was performed to
validate the versatility and adaptability of the gripper concerning object
manipulation. To evaluate the versatility of the proposed gripper, we applied
it to grasp objects of different sizes, shapes, materials, and weights. Fig.
11-A to Fig. 11-D demonstrate the cases of grasping regular objects with
regular shapes, regular planes, regular arc surfaces, and irregular contours,
respectively. Fig. 11-E illustrates using a single finger to achieve grasping.
In this state, the gripper is capable of gripping a 25 mm diameter cylinder.
These grasping experiments prove that the gripper has a large grasping range
and can adaptively adjust the finger posture according to the contour of the
object.

TABLE IV: Analysis of contact area

Object |  $\textit{L}_{P}$/mm | $\textit{R}_{P}$ |  $\textit{L}_{M}$/mm | $\textit{R}_{M}$ |  $\textit{L}_{D}$/mm | $\textit{R}_{D}$ |  $\textit{L}_{total}$/mm | $\textit{R}_{total}$
---|---|---|---|---|---|---|---|---
25mm Cylinder | 47.2 | 32.57 % | 38.4 | 30.18 % | 51.0 | 0.00 % | 136.6 | 22.39 %
40mm Cylinder | 68.7 | 1.86 % | 40.2 | 26.91 % | 51.0 | 0.00 % | 159.9 | 9.15 %
120mm Cylinder | 68.3 | 2.43 % | 42.1 | 23.45 % | 51.0 | 0.00 % | 161.4 | 8.30 %
Ping-pong ball | 52.3 | 25.29 % | 38.2 | 30.55 % | 51.0 | 0.00 % | 141.5 | 19.60 %
Tennis | 60.2 | 14.00 % | 39.4 | 28.36 % | 51.0 | 0.00 % | 150.6 | 14.43 %
Scotch tape | 64.4 | 8.00 % | 44.8 | 18.55 % | 51.0 | 0.00 % | 160.2 | 8.98 %
Ruler | 69.5 | 0.71 % | 53.1 | 3.45 % | 38.2 | 25.10 % | 160.8 | 8.64 %
Cardboard | 69.7 | 0.43 % | 53.6 | 2.55 % | 39.8 | 21.96 % | 163.1 | 7.33 %

$\textit{L}_{P}$, $\textit{L}_{M}$, $\textit{L}_{D}$, and $\textit{L}_{total}$
represent the actual length of the proximal phalanx, middle phalanx, distal
phalanx and the total length respectively. $\textit{R}_{P}$, $\textit{R}_{M}$,
$\textit{R}_{D}$, and $\textit{R}_{total}$ represent the actual ratio of the
change in proximal phalanx, middle phalanx, distal phalanx and total length,
respectively.

As depicted in Figure 10, the trajectory of the fingertip exhibits an arcuate
pattern during parallel grasping or enveloping grasping mode. When grasping an
object, the relative height between the fingertip and the object placement
surface will change, which can make it hard for the gripper to grasp thin
objects, especially since the gesture and position of the gripper are
critically required. However, with the retractable of the phalanx, thin
objects can be grasped with ease. As the gripper closes and the fingertips
contact the object placement surface, the distal phalanx and middle phalanx
adaptively contract to ensure a smooth grasp. During the grasping process, the
fingers are kept close to the surface on which the object is placed, allowing
the gripper to better pick up thin objects, as shown in Fig. 12.

In Fig. 12-A, the fingertips begin to contact the object placement surface. In
Fig. 12-B, the gripper adapts to the object placement surface during the
closing process. Finally, the picking of thin objects is achieved (see Fig.
12-C/D). If the paper area is suitable, the gripper can even grasp the paper
placed on the desktop.

Table. IV displays the actual length and expansion ratio of each finger
phalanx after the gripper grasps objects in enveloping grasping mode. The
gripper primarily utilizes the retractable of the proximal phalanx and middle
phalanx to adaptively grasp objects, while using the distal phalanx to grasp
thin objects on a desktop. The experiments confirm the feasibility of adaptive
grasping by retracting the phalanges of the gripper. The phalanx can shrink up
to 32.57% of its maximum capacity.

##  IV CONCLUSIONS

In conclusion, this paper investigated a three-finger under-actuated gripper
featuring a retractable phalanx based on the linkage system. The proposed
gripper not only achieved an extensive grasping range through its
reconfiguration ability but also facilitated the automatic switching of
grasping modes via finger retraction. Moreover, the implementation of a
minimal number of motors objectively mitigated the challenges associated with
gripper control. We conducted various grasping experiments to evaluate the
real-world performance of the proposed gripper in diverse scenarios. The
experimental results demonstrated impressive capabilities in terms of grasping
stiffness, versatility, and adaptability. Future work will focus on optimizing
the geometric parameters of the gripper, reducing its size and weight, and
enhancing the dexterous grasping functionality for smaller objects.

## References

  * [1] H. Dong, Y. Feng, C. Qiu, and I.-M. Chen, “Construction of interaction parallel manipulator: towards rehabilitation massage,” IEEE/ASME Transactions on Mechatronics, vol. 28, no. 1, pp. 372–384, 2022.
  * [2] H. Dong, E. Asadi, G. Sun, D. K. Prasad, and I.-M. Chen, “Real-time robotic manipulation of cylindrical objects in dynamic scenarios through elliptic shape primitives,” IEEE Transactions on Robotics, vol. 35, no. 1, pp. 95–113, 2018.
  * [3] H. Dong, E. Asadi, C. Qiu, J. Dai, and I.-M. Chen, “Geometric design optimization of an under-actuated tendon-driven robotic gripper,” Robotics and Computer-Integrated Manufacturing, vol. 50, pp. 80–89, 2018.
  * [4] J. Zhou, J. Yi, X. Chen, Z. Liu, and Z. Wang, “Bcl-13: A 13-dof soft robotic hand for dexterous grasping and in-hand manipulation,” IEEE Robotics and Automation Letters, vol. 3, no. 4, pp. 3379–3386, 2018.
  * [5] L. Birglen, T. Laliberté, and C. M. Gosselin, Underactuated robotic hands, vol. 40.  Springer, 2007.
  * [6] Z. Samadikhoshkho, K. Zareinia, and F. Janabi-Sharifi, “A brief review on robotic grippers classifications,” in 2019 IEEE Canadian Conference of Electrical and Computer Engineering (CCECE), pp. 1–4, IEEE, 2019.
  * [7] H. Dong, C.-Y. Chen, C. Qiu, C.-H. Yeow, and H. Yu, “Gsg: A granary-shaped soft gripper with mechanical sensing via snap-through structure,” IEEE Robotics and Automation Letters, vol. 7, no. 4, pp. 9421–9428, 2022.
  * [8] A. L. Gunderman, J. A. Collins, A. L. Myers, R. T. Threlfall, and Y. Chen, “Tendon-driven soft robotic gripper for blackberry harvesting,” IEEE Robotics and Automation Letters, vol. 7, no. 2, pp. 2652–2659, 2022.
  * [9] Z. Wang and S. Hirai, “A soft gripper with adjustable stiffness and variable working length for handling food material,” in 2018 IEEE International Conference on Real-time Computing and Robotics (RCAR), pp. 25–29, IEEE, 2018.
  * [10] Y. Li, Y. Chen, Y. Yang, and Y. Li, “Soft robotic grippers based on particle transmission,” IEEE/ASME Transactions on Mechatronics, vol. 24, no. 3, pp. 969–978, 2019.
  * [11] J. Kwon, D. Bombara, C. Teeple, J. Lee, C. Hoberman, R. Wood, and J. Werfel, “Transformable linkage-based gripper for multi-mode grasping and manipulation,” IEEE Robotics and Automation Letters, vol. 8, no. 12, pp. 8446–8453, 2023.
  * [12] G. A. Zappatore, G. Reina, A. Messina, et al., “Analysis of a highly underactuated robotic hand,” Int. J. Mech. Control, vol. 18, no. 2, pp. 17–24, 2017.
  * [13] T. Nishimura and T. Watanabe, “Single-motor robotic gripper with three functional modes for grasping in confined spaces,” IEEE Robotics and Automation Letters, 2023.
  * [14] S. J. Yoon, M. Choi, B. Jeong, and Y.-L. Park, “Elongatable gripper fingers with integrated stretchable tactile sensors for underactuated grasping and dexterous manipulation,” IEEE Transactions on Robotics, vol. 38, no. 4, pp. 2179–2193, 2022.
  * [15] D. Bandara, R. Gopura, G. Kajanthan, M. Brunthavan, and H. Abeynayake, “An under-actuated mechanism for a robotic finger,” in The 4th Annual IEEE International Conference on Cyber Technology in Automation, Control and Intelligent, pp. 407–412, IEEE, 2014.
  * [16] R. Abayasiri, R. Abayasiri, R. Gunawardhana, R. Premakumara, S. Mallikarachchi, R. Gopura, T. D. Lalitharatne, and D. Madusanka, “An under-actuated hand prosthesis with finger abduction and adduction for human like grasps,” in 2020 6th International Conference on Control, Automation and Robotics (ICCAR), pp. 574–580, IEEE, 2020.
  * [17] Z. Lu, H. Guo, W. Zhang, and H. Yu, “Gtac-gripper: A reconfigurable under-actuated four-fingered robotic gripper with tactile sensing,” IEEE Robotics and Automation Letters, vol. 7, no. 3, pp. 7232–7239, 2022.
  * [18] T. Nishimura and T. Watanabe, “Single-motor robotic gripper with three functional modes for grasping in confined spaces,” IEEE Robotics and Automation Letters, 2023.
  * [19] L. Kang, Y. Yang, J. Yang, and B.-J. Yi, “A three-fingered adaptive gripper with multiple grasping modes,” in 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 6097–6103, IEEE, 2021.