††thanks: This work has been submitted to the IEEE for possible publication.
Copyright may be transferred without notice, after which this version may no
longer be accessible.

# Deep Metric Learning with Locality Sensitive Angular Loss for Self-
Correcting Source Separation of Neural Spiking Signals

Alexander Kenneth Clarke, Dario Farina D. Farina<EMAIL_ADDRESS>and
A. K. Clarke<EMAIL_ADDRESS>are with the Department of
Bioengineering, Imperial College London, UK.The work was supported by the
EPSRC Centre for Neurotechnology (EP/L016737/1) (AKC) and the EPSRC
Transformative Healthcare Technologies 2050 Grant (EP/T020970/1) (DF).

###### Abstract

Neurophysiological time series, such as electromyographic signal and
intracortical recordings, are typically composed of many individual spiking
sources, the recovery of which can give fundamental insights into the
biological system of interest or provide neural information for man-machine
interfaces. For this reason, source separation algorithms have become an
increasingly important tool in neuroscience and neuroengineering. However, in
noisy or highly multivariate recordings these decomposition techniques often
make a large number of errors, which degrades human-machine interfacing
applications and often requires costly post-hoc manual cleaning of the output
label set of spike timestamps. To address both the need for automated post-hoc
cleaning and robust separation filters we propose a methodology based on deep
metric learning, using a novel loss function which maintains intra-class
variance, creating a rich embedding space suitable for both label cleaning and
the discovery of new activations. We then validate this method with an
artificially corrupted label set based on source-separated high-density
surface electromyography recordings, recovering the original timestamps even
in extreme degrees of feature and class-dependent label noise. This approach
enables a neural network to learn to accurately decode neurophysiological time
series using any imperfect method of labelling the signal.

###### Index Terms:

Deep metric learning, deep learning, blind source separation, surface
electromyography.

##  I Introduction

Time series neurophysiological data is commonly characterised by repetitive
activation events, for example the motor unit activation potentials (MUAP) in
electromyographic (EMG) signals or the spike potentials in microelectrode
cortical recordings[1][2]. The ensemble of these activation events constitute
neural codes that give a direct insight into the target system[3], whilst also
providing an accurate control signal for human-machine interfacing
applications, such as prosthetic control[4][5]. Neurophysiological signals are
typically linear superpositions of many of these spiking sources, and their
extraction from noisy systems has long been a major focus of applied source
separation in neuroscience[6].

Identifying sources from multiunit activity was originally achieved through
manual sorting by trained operators[7], but this tedious process was quickly
supplanted by automated methods using early forms of blind source separation
(BSS)[8]. BSS algorithms have since become extremely effective, able to
automate the recovery of sources in highly noisy and complex systems[8][4][9].
An important trend within the field of applied source separation has been the
increasing availability of highly multivariate data[10][11], as a result
developments in high-density electrode arrays[12][13]. By exploiting the
increased spatial information collected by these systems, BSS pipelines can
yield extremely large numbers of sources[11][14]. A more recent development is
the adoption of deep learning approaches as a replacement for linear
separation vectors, using neural networks to decode signals with a high degree
of robustness to noise and signal non-stationarities[15][16]. These methods
involve an offline supervised training phase using the augmented output of a
BSS algorithm, therefore an important requirement is that the BSS
decomposition contains relatively few errors if the network is to decode with
high accuracy[15].

As the number of sources identified in a manual or automatic decomposition
increases, so does the probability of labelling errors. Noise can be
mistakenly labelled as an activation, an activation can be assigned to the
wrong class or missed entirely, a class might be inappropriately partitioned
or two distinct classes merged into one. These sources of label noise can be
categorised based on whether the factors affecting the likelihood of a label
from one class to flip to that of a different class are shared at the dataset,
class or feature level[17]. As label-flipping is class or even feature-
dependent, it is difficult to identify such errors automatically, and for
automatic decompositions a degree of manual post-hoc cleaning is commonly
employed, often using additional knowledge about the system of interest, such
as temporal statistics in the source activations[18]. The nature of this
manual cleaning generally relates to the mixing system of interest, for
example intracortical and intramuscular EMG (iEMG) decompositions generally
require post-hoc examination of classes due to extensive class-dependent label
noise[8][19]. On the other hand, surface EMG (sEMG) decompositions also
contain a degree of feature-dependent noise and so require further inspection
of specific activations[20]. Whilst accurate, manual post-hoc ”cleaning” is an
extremely time-consuming process and in some cases not feasible because of the
size of the datasets being source-separated[21]. For this reason, modern
source separation pipelines are increasingly using additional automated post-
processing steps in an attempt to reduce the false label burden[18][22][10].
However, these methods only compensate for a relatively small proportion of
incorrect labels, so that there remains a need for new methods of post-hoc
label cleaning. Additionally, if supervised deep learning frameworks are to be
trained using BSS-labelled signals with increasing degrees of label noise,
then they need to be able to detect and manage such errors, i.e. they need to
be designed to be implicitly self-correcting.

Managing label noise is a fast-developing subfield of modern machine learning.
As the size of datasets expand faster than the capacity of domain experts to
screen and label, data scientists are increasingly turning to new labelling
methods that are more scalable at the cost of a greater proportion of label
noise, such as Amazon’s Mechanical Turk[23]. This is particularly true in a
neuroscience setting, where datasets are generally labelled by a small pool of
domain experts who can differ in professional opinion[24]. Approaches to
learning in the presence of label noise can be broadly split into two
categories; methods that aim to select models that are robust to label noise
and methods that attempt to clean the label set prior to training[25].
Contemporary methods based on the latter approach generally rely on additional
models which attempt to identify noisy labels using either a smaller pool of
known correct labels or by comparing a label with other in-class labels using
a similarity metric[26][27][28]. This principle of using similarity metrics to
build embedding spaces that inform intra- and inter-class classification is
closely related to deep metric learning (DML) approaches.

The objective of deep metric learning is the training of a deep neural network
which maps discrete inputs to an embedding space in which positive pairs (two
inputs from the same class) are closer than negative pairs (two inputs from
different classes)[29]. The Euclidean distance is commonly used as the
distance metric, although measures based on the angular difference between
embeddings have become popular due to their inherent rotation and scale
invariance[30]. The network is optimised either using the absolute similarity
between pairs or, more commonly, the relative difference between one or more
positive and negative pairings[31]. During optimisation, most negative pairs
will quite quickly become much further away than positive pairs, thus random
generation of pairs for training is extremely inefficient. For this reason
many implementations of DML seek to select the most informative pairings from
each minibatch by applying some form of selection rule operating on the output
embedding space[32][33]. Most of these approaches are designed purely to
maximise the class separability of the embedding space, leading to dense
clusters in each class[34]. More recent work has attempted to increase intra-
class variability, as such tight embeddings potentially reduce the ability of
the model to generalise to new in-class inputs[35][34][36]. These locality-
sensitive approaches have a less distorting effect on the embedding space,
giving better generalisation performance[36]. Whilst not the primary goal of
these studies, another effect of preserving intra-class variance is a richer
embedding of inputs, with semantically similar events sub-clustering[35]. Such
rich embedding spaces could potentially be of use for detecting class
outliers, perhaps even having utility for label-cleaning operations in event-
driven neurophysiological recordings. DML pipelines have been designed to
operate on noisy label sets for similar tasks such as person reidentification,
however these methods tend to use an external method to modify training rather
than the embedding space itself, for example using label-correction based on
cross-entropy[37].

Motivated by the need to further preserve a rich intra-class embedding, and
inspired by ranking approaches to triplet sampling, such as[36], here we
propose a novel locality-sensitive approach to sampling during DML
optimisation. We use an efficient top-k query to identify the closest in-batch
positive to each event and the N-closest in-batch negatives, which are then
used to calculate an N-pair formulation of the popular angular loss[30]. The
idea of using top-k queries within batch losses has been explored in the
context of binary classification problems[38], however this is, to our
knowledge, the first such implementation within the domain of DML. In this
paper we demonstrate that this simple modification, which we call locality-
sensitive angular loss (LSAL), generates an embedding space which can be used
to detect and classify repetitive events, whilst importantly having the
additional utility of being able to detect label-noise in the data used for
training.

The main contribution of this paper is DeepLSAL, a novel DML pipeline that
leverages LSAL to perform both label-cleaning and the identification of new
activations in unseen data, operating directly on neurophysiological time
series signals. The specific focus is on label sets generated by BSS
algorithms for decoding convolutive mixtures, as this is an area where
supervised deep learning methods are clearly beginning to outperform existing
methods[15][16]; however, in principle the proposed methodology is applicable
to any manual or automatic method of generating imperfect label sets. The
effectiveness of DeepLSAL is validated robustly using an experimentally-
collected high-density (HD) sEMG dataset that had been source-separated into
constituent motor unit activity using the gradient convolutional kernel
compensation (gCKC) algorithm[39]. The scenario of decomposition of HD-sEMG
signals is highly convenient for validation of the proposed approach since
both the generating system and decomposition methodology are well-studied.
Moreover, the sEMG system is characterised by a high degree of feature-
dependent label noise due to the complex superposition of MUAPs caused by
volume-conduction effects[40].

##  II Theory and Algorithm

###  II-A Deep Metric Learning with N-pair Loss

The objective of ranking loss DML, also called triplet loss, is to train a
deep learning function such as a convolutional neural network to map a sample
taken from one of $C$ classes to an embedding vector $\mathbf{x}$, such that
for an arbitrarily selected anchor embedding $\mathbf{x_{a}}$, the embedding
space reduces the relative distance $\mathbf{D}$ between positive samples from
the same class $\mathbf{x_{p}}$ and negative samples from different classes
$\mathbf{x_{n}}$. $\mathbf{D}$ can be a number of different metrics, such as
the Euclidean distance, cosine similarity or Kullback–Leibler divergence[41].
Commonly the loss function is formulated such that the relative difference be
greater than a margin $m$ such that:

$\small\mathbf{D}(\mathbf{x_{a}},\mathbf{x_{n}})+m\leq\mathbf{D}(\mathbf{x_{a}},\mathbf{x_{p}})$
(1)

Figure 1: a shows the principle behind N-pair loss, in this case with an N of
5. As multiple negative embeddings are utilised, the chance of sampling only
uninformative pairings reduces. b illustrates the principle behind vertex
selection in angular loss. Rather than directly using the triplet as vertices,
only the negative embedding is used, whilst the other two vertices are
constructed so as to build a right-angled triangle with the 90°angle at the
midpoint between the anchor and positive embedding. c demonstrates the effect
of random versus locality-sensitive sampling on the intra-class variance.

After a small amount of optimisation, the bulk of negative pairs will be much
further away than the positive pairs, meaning most training examples in a
batch will become uninformative[32]. N-pair loss seeks to avoid this problem
by comparing each positive pair in an $M$-size batch $B$ to $N$ multiple
negative pairs (fig. 1a), which are then combined in a $log-sum-exp$
formulation[33]:

$\small L=\frac{1}{M}\sum_{\mathbf{x}_{a}\in
B}^{M}log\left[1+\sum_{\mathbf{x}_{n}\in
B\setminus(\mathbf{x}_{a},\mathbf{x}_{p})}^{N}\exp(f_{a,p,n})\right]$ (2)

where $f_{a,p,n}$ is usually a hinge function such as
$max(0,\mathbf{D}(\mathbf{x_{a}},\mathbf{x_{n}})-\mathbf{D}(\mathbf{x_{a}},\mathbf{x_{p}})+m)$.
By taking an average across the negative pairs, it is likely that at least
some informative negative pairings will be included in the loss calculation.

###  II-B Angular Loss

Angular loss is based on a geometric reformulation of the distance metric;
rather than minimising the distance of $\mathbf{x_{p}}$ to $\mathbf{x_{a}}$
relative to $\mathbf{x_{n}}$, it instead minimises the angle at
$\mathbf{x_{n}}$ of a triangle formed from the three embeddings. This has the
effect of improving optimisation stability as angles are scale invariant,
whilst using a triangle means all edges of the triplet are taken into
account[30]. However, in certain circumstances the minimisation of the angle
at $\angle\mathbf{x_{n}}$ will push $\mathbf{x_{n}}$ towards $\mathbf{x_{a}}$.
This can be avoided by constructing a right-angled triangle with
$\mathbf{x_{n}}$ and the midpoint between $\mathbf{x_{a}}$ and
$\mathbf{x_{p}}$ (fig. 1b), with the final vertex being the point on the
semicircle joining $\mathbf{x_{a}}$ and $\mathbf{x_{p}}$ which creates a
right-angled triangle[30]. By dropping constant terms, this geometric
relationship can be used for the $f_{a,p,n}$ in equation 2, expressed as:

$\small
f_{a,p,n}=4tan^{2}\alpha(\mathbf{x}_{a}+\mathbf{x}_{p})^{T}\mathbf{x}_{n}-2(1+tan^{2}\alpha)\mathbf{x}_{a}^{T}\mathbf{x}_{p}$
(3)

where $\alpha$ is an angle in radians which sets the upper accepted bounds of
the loss, analogous to $m$ in equation 1.

###  II-C Inducing Rich Embeddings

Whilst a random selection of positive pairings in $B_{ang}$ is suitable if the
objective is to maximise inter-class distance within the embedding space, it
also has the tendency to collapse intra-class distances down to a point, as
illustrated in figure 1c[35]. We theorised that the main explanation for this
compression is that the sampling process is random, meaning that the neural
network is induced during training to bring all embeddings from the same class
together, which, as a complex non-linear function, it is quite capable of
doing given sufficient training steps. We instead elected to sample positive
and negative pairs based on the local neighbourhood of each embedding in the
batch, modifying the selection of $\mathbf{x_{p}}$ and the set of
$\mathbf{x_{n}}$’s for each $\mathbf{x_{a}}$ in the batch $B$, a process we
term locality-sensitive sampling.

$B$ is first selected such that it be large enough to have a diverse
representation of each class. As each embedding vector is L2-normalised, the
tensor formed by finding the inner product of the batch tensor with its
transpose is the pairwise cosine similarity. For each $\mathbf{x_{a}}$ the
$\mathbf{x_{p}}$ selected with a simple argmax, i.e. the closest different
vector from the same class is selected. A similar procedure is used to select
the set of $\mathbf{x_{n}}$’s, using a top-k algorithm to select the $k$ most
similar vectors to $\mathbf{x_{a}}$ that belong to a different class. GPU
implementations of top-k algorithms have become extremely efficient in recent
years, due to their increasing use within machine learning paradigms[42]. With
these easily-implemented changes the N-pair formulation of the angular loss
becomes:

$\displaystyle l_{LSAL}={}$ $\displaystyle\frac{1}{M}\sum_{\mathbf{x}_{a}\in
B_{r}}^{M}\>\sum_{\mathbf{x}_{p}\in
B\setminus(\mathbf{x}_{a},\mathbf{x}_{n})}\mathbf{1}_{\mathbf{x}_{p}\in\Upsilon}\>log\left[1+g\right]$
(4) $\displaystyle g={}$ $\displaystyle\sum_{\mathbf{x}_{n}\in
B\setminus(\mathbf{x}_{a},\mathbf{x}_{p})}\
\mathbf{1}_{\mathbf{x}_{p}\in\Theta}\>exp(f_{a,p,n})$

where $\Upsilon=argmax(\angle(\mathbf{x}_{a},\mathbf{x}_{p}))$ is the argmax
set of the pairwise cosine similarity $\angle$ between $\mathbf{x_{a}}$ and
its associated set $\mathbf{x_{p}}$ in $B$ and
$\Theta=topk(\angle(\mathbf{x}_{a},\mathbf{x}_{n}))$ is the top-k values of
the ordered pairwise cosine similarity $\angle$ between $\mathbf{x_{a}}$ and
its associated set $\mathbf{x_{n}}$ in $B$. $\mathbf{1}$ is the indicator
function.

To stabilise early optimisation we combined the LSAL loss with a categorical
cross-entropy with temperature given by:

$\small l_{cross}=-\frac{1}{M}\sum_{i}^{M}\sum_{c}^{C}\mathbf{1}_{y_{i}\in
C_{c}}log\frac{exp\left[\frac{1}{\gamma}\mathbf{z}_{i}\right]}{\sum_{j}^{C}exp\left[\frac{1}{\gamma}\mathbf{z}_{j}\right]}$
(5)

where $\mathbf{z}=\mathbf{\frac{W}{\left\|W\right\|}}_{2}\mathbf{x}$ and
$\mathbf{W}$ is a trainable matrix that compresses the embedding vector down
to a dimension C vector for comparison with the one-hot encoded class labels.
This gives the final loss function:

$\small l=l_{LSAL}+\tau l_{cross}$ (6)

###  II-D Source Separation

The timestamps used by DeepLSAL can be generated using a wide variety of
manual and automated processes, however for this study the gradient
convolution kernel compensation (gCKC) algorithm was selected due to its
strong performance in HD-sEMG signal decomposition[43][39]. In the gCKC
framework for blind source separation, the vector of spiking sources $s$ at
time $t$ are first extended with $L$ delayed versions of themselves, allowing
the mixing problem, which is convolutive in most neurophysiological settings,
to be written in instantaneous form:

$\small x(t)=\mathbf{H}\tilde{s}(t-l)+\omega(t)$ (7)

where the signal observation vector $x$ at time $t$ is a linear mixture
parameterised by the operation of the mixing matrix $\mathbf{H}$ on the
extended source vector $\tilde{s}$ plus noise $\omega$. In practice both the
observation and source vectors are additionally extended with a further $R$
values for reason of numerical stability during the source separation
procedure.

Unlike independent component analysis methods which seek to directly estimate
a separation vector for each source, gCKC seeks to include the additional
statistical information that the spiking sources generate repetitive events
within the signal. Sources are instead estimated indirectly using a linear
minimum mean square error estimator, with the estimated $j$th source
$\hat{s}_{j}$ at time point $t$ given by:

$\small\hat{s}_{j}(t)=\hat{c}_{\mathbf{s}_{j}\tilde{\mathbf{x}}}^{T}\mathbf{C}_{\tilde{\mathbf{x}}\tilde{\mathbf{x}}}^{-1}\tilde{x}(t)$
(8)

where $\hat{\mathbf{c}}_{\hat{{\mathbf{s}}_{j}\tilde{\mathbf{x}}}}^{T}$ is the
transposed cross-correlation vector between an activation of the $j$th source
and extended HD-sEMG matrix and
$\mathbf{C}_{\tilde{\mathbf{x}}\tilde{\mathbf{x}}}^{-1}$ is the inverted
autocorrelation matrix of the extended HD-sEMG matrix $\tilde{\mathbf{x}}$.

The vector $\hat{c}_{\hat{{\mathbf{s}}_{j}\tilde{\mathbf{x}}}}^{T}$ is usually
initialised with a time point likely to contain a source activation, which can
be estimated by, for example, the Mahalanobis distance calculated on the
signal[39]. Once selected,
$\hat{\mathbf{c}}_{\hat{{\mathbf{s}}_{j}\tilde{\mathbf{x}}}}^{T}$ is then
optimised to find the rest of the source’s signal contributions. This can be
done with either a fixed-point algorithm as in [10] or in the gCKC formulation
by gradient descent:

$\small\mathbf{c}_{\hat{\mathbf{s}}_{j}\tilde{\mathbf{x}}}^{\circ}=\mathbf{c}_{\hat{\mathbf{s}}_{j}\tilde{\mathbf{x}}}-\alpha\sum_{t}\frac{\partial
f\big{(}\hat{\mathbf{s}}_{j}\big{(}t\big{)}\big{)}}{\partial\hat{\mathbf{s}}_{j}\big{(}t\big{)}}\tilde{\mathbf{x}}\big{(}t\big{)}$
(9)

where $\mathbf{c}_{\hat{\mathbf{s}}_{j}\tilde{\mathbf{x}}}^{\circ}$ is the
updated cross-correlation vector, $\alpha$ is the learning rate and
$f\big{(}.\big{)}$ a contrast function designed to estimate the non-
gaussianity of the output source in a similar fashion to independent component
analysis. Optimised sources can then be converted to timestamps using a linear
threshold or a two-class k means clustering algorithm.

Figure 2: a shows the model used in DeepLSAL, which takes the form of an
easily-implemented convolutional neural network trained using a locality-
sensitive angular loss to embed windows of neurophysiological time series into
a low-dimensional space which can be used to both source separate and label
clean. b shows the full DeepLSAL pipeline by which the noisy activation labels
found from source-separating the high-density surface electromyography signal
are cleaned. DeepLSAL is run twice - a cleaning phase to find the false
positive labels and and a refitting phase to find the false negatives. After
the refitting phase the predicted class activity is much cleaner than that of
the original source separation algorithm, as seen in c.

##  III Validation Methodology

###  III-A Experimental Dataset

The HD-sEMG data set consisted of a set of 20-second recordings taken from the
dominant tibialis anterior muscle of 10 men performing an isometric
contraction at 15% of maximal force, previously used to validate source
separation techniques[44]. Maximal contraction was defined as the mean force
of three 5-s maximal contractions separated by 3 min of rest, with force
sampled at 2048Hz by load cells mounted on an isometric brace. Force feedback
was provided to the participants by an oscilloscope. The signal from a
monopolar $12\times 5$ electrode array placed over the main muscle innervation
zone was sampled at 2048Hz having been band-pass filtered at 10-500 Hz.

Gradient convolution kernel compensation with an additional k-means source
refinement step was implemented using the tensorflow machine learning
package[39][10]. As the label set was to be artificially corrupted it was
important that the original be as noise-free as possible, so additional post
hoc steps were taken to maximise the likelihood that the timestamps were
correct. Sources were manually cleaned by examining interspike intervals and
the source-to-noise ratio of each activation. An additional step of validating
decomposition accuracy was implemented by comparing the sources to those found
using the DEMUSE source-separation software package[43][39], with source
cleaning completed by a different trained operator.

###  III-B Label Set Corruption

In experiment 1 we evaluated the ability of DeepLSAL to clean a label set
corrupted by feature-dependent noise, where a label flipping probability is
related to its associated features[17]. In the context of source-separated HD-
sEMG, this most commonly occurs as a false-positive, where a separation vector
incorrectly assigns a high probability of an in-class MUAP being present when
it is not, i.e. a noise class or other MU class label is flipped to the MU
class of interest. To simulate this effect, we corrupted the label set by
generating an artificially-noisy separation vector for each MU class;
randomly-selecting 15 MUAP labels from that class and using the average of the
associated extended HD-sEMG vectors to generate a linear minimum mean square
error prediction on the extended HD-sEMG matrix. A two-class k-means
clustering algorithm was then used to parameterise a linear threshold to find
activations, creating a label set with a high degree of feature-dependent
noise. Five levels of increasing difficulty were generated by taking an amount
of false positives corresponding to 10/20/30/40/50% of the number of true
labels, selected at random from the set of false positives.

In experiment 2 the DeepLSAL algorithm was evaluated on class-dependent label
noise, when the probability of a label flipping to another class is stable
across all labels in the class[17]. In HD-sEMG source separation this error
generally occurs when the separation vectors are very similar, usually due to
similar MUAP waveform shapes between two MU classes. This can be simulated by
transferring a percentage of labels to a similar MU class. This was done by
first averaging the MUAPs of each MU class and then cross-correlating these
averages with the average MUAP of every other class in the recording, with
10/20/30/40% of the class labels transferred to the class with the highest
value. If labels had already been transferred to the closest class then the
next closest class was selected until all classes had had label transfers. A
maximum label corruption of 40% was used to preserve the concept of a majority
true and minority false class.

###  III-C DeepLSAL Pipeline and Training

To convert the source-separated HD-sEMG signal into labelled windows, first
each channel of the HD-sEMG signal was standardised by z-scoring and then cut
into overlapping 80-sample wide windows at a stride of 1. Each window was then
labelled by reference to the predicted source activity at the final sample of
the window. This meant the bulk of windows were labelled as part of the
inactive class due to the sparse nature of motor neuron spiking. Due to this
serious class imbalance, each minibatch was created from the entirety of the
windows labelled as containing a motor neuron spike, with an additional 256
samples from the inactive class. Each class assignment was then converted to a
one-hot representation, the bulk of which had only one class active at any one
time, although rarely two activations would occur simultaneously on the same
time-point. As the richness of the intra-class embedding of the inactive class
windows was not of any great concern, the embeddings of these windows were not
used as anchor vectors when calculating the $l_{LSAL}$ component of the loss,
although they were used as both $\mathbf{x_{n}}$ values and in the calculation
of $l_{cross}$.

Embeddings were calculated with a convolutional neural network implemented
using the tensorflow machine learning library in python, as seen in figure 2a.
Convolution steps used a 1D 3-sample wide kernel, with 32 filters and a drop-
out of 0.2. 1D max-pooling was completed with 2-sample wide kernels. Each
densely-connected layers had 64 neurons and a drop-out percentage of 0.5
during training. Both the convolution and densely-connected layers used ReLU
activation functions. Finally the output of the last densely-connected layer
was densely-connected to a bias and activation-free embedding layer of 8
neurons-wide, which was then divided by its L2 norm. This was an intentionally
low-dimension embedding compared to standard DML due to the desire to avoid
dimensionality issues during the clustering steps in the refitting phase. The
additional matrix $\mathbf{W}$ used in the categorical cross-entropy was
initialised with truncated normal noise, whilst the weights of the neural
network layers was initialised by glorot uniform. The Adam optimisation
algorithm at a learning rate of 0.001 was then used to train the model over
500 epochs for both cleaning and refitting stages.

For both experiments $k$ was set to 5, $\gamma$ was set to 0.1, and $\alpha$
to 0.25 radians in both the cleaning and refitting stages. After the cleaning
stage the labels in the embedding space likely to relate to specific classes
were selected by a simple density-estimator. First a local scale value $v$ was
estimated by finding the mean cosine similarity of the each embedding vector
to its 20 nearest neighbours and taking a median of this value across all
vectors. For each label the number of other labels with a cosine similarity
more than $v$ was found and the label with the highest number of neighbours
was selected as the centre of the cluster. All labels within a cosine
similarity higher than $v$ were then added to the refitting training set. This
simple approach was generally adequate for quickly finding the densest region
of the embedding space, which was usually the cluster of true labels.

After the cleaning phase the set of timestamp labels is generally free of
false positives; if DeepLSAL is retrained with this new label set then it
should effectively generalise to find unlabelled activations in both the
current and future data (fig. 2b). It should be noted that finding unlabelled
activations in the current dataset is in some ways a ”harder” problem than
trying to generalise to completely unseen data, as these activations are
included in training, but mislabelled. However these mislabelled activations
are a very small proportion of the total dataset, meaning they were predicted
not to impact convergence on a model with good generalisation ability,
particularly a heavily regularised model such as that used for DeepLSAL. Once
DeepLSAL was retrained an average embedding vector of the current MUAP
timestamps was found for each class, which was then cross-correlated with the
entire embedded HD-sEMG signal to generate a predicted activity (fig. 2c).
This activity was then timestamped by a linear threshold parameterised by a
two-class k-means clustering algorithm. These labels were compared to the pre-
corrupted data using the rate of agreement (RoA) metric, a percentage defined
as the number of true positive matches divided by the total number of true
positives, false positives and false negatives.

##  IV Results

###  IV-A Feature-dependent Label Noise

In experiment 1, which tested the effect of feature-dependent label noise by
simulating noisy separation vectors, DeepLSAL generated an embedding space
with dense clusters for each class corresponding to the true labels.
Surrounding each cluster is a large sparse periphery of false labels, with no
apparent structure. The efficacy of locality-sensitive sampling at preserving
a rich class embedding is clear when compared to random sampling using a two-
dimensional principal component space, as in figure 3. The simple density-
estimator operating on the cosine similarity between embeddings could then be
used to select a subset of true labels by selecting the area with the highest
density (fig. 4). This algorithm was weighted to favour specificity over
sensitivity, meaning almost no false labels were included in the cleaned
label-set at the cost of losing a percentage of the true labels.

Figure 3: The effect of two different sampling strategies on the embedding
space for two units as shown by the first and second principle components. a
shows the effect of locality-sensitive sampling, with a rich intra-class
embedding that clearly separates the true and false embeddings. In b the same
optimisation was run again, but with the positive and negative pairs randomly
selected, leading to all intra-class embeddings contracting down to a point.

DeepLSAL generated an embedding space with utility for removing false labels
even at the maximum tested value of 50% of total correct values (fig. 5a),
with a median post-cleaning false label retention of 1.3% of the total correct
labels in the class (IQR 0.5 - 1.9). The number of true labels lost during the
cleaning process fell as the pre-cleaning percentage of false labels
increased, but even at the highest false label percentage tested, a median of
74.1% (IQR 68.9 - 80.3) of the true values were still retained (table I).

Figure 4: The first and second principal components of the embedding space after the cleaning phase for all classes found in a single HD-sEMG sample, with labels initially extracted from a noisy separation filter. The samples selected by density analysis are circled, these will form the training labels for the refitting phase. The model is effective at inducing clustering of similar labels, with the area of highest density corresponding to the true labels. Figure 5: Median label accuracy and rate of agreement plots for both experiments across all recordings, with interquartile range. a shows the outcome of the DeepLSAL cleaning phase for experiment 1 simulating a noisy separation filter. This leads to high number of false positives in the labels, plotted here as a fraction of the total correct labels in each class. These are almost completely removed, at the cost of losing a fraction of the true labels. A similar cleaning result was found in experiment 2 (b), where a proportion of labels from each class was transferred to the nearest correlated class. Refitting recovers the bulk of these lost labels, giving good final rates of agreement with the ground truth labels. c shows the RoA from experiment 1 before and after cleaning and refitting, whilst d shows the RoA change in experiment 2. The RoA is returned to values close to pre-corruption levels. TABLE I: Combined median (interquartile range) scores for all classes across all recordings from experiment 1 for different stages of the cleaning and refitting pipeline. False Labels Added as Proportion of Class (%) |  Starting RoA (%) |  Remaining True Labels After Cleaning (%) |  RoA After Cleaning and Refitting (%)
---|---|---|---
10 |  91.5 (91.5 – 91.7) |  84.1 (79.6 - 87.1) |  94.5 (91.2 – 97.2)
20 |  83.9 (83.8 – 83.9) |  78.3 (74.4 - 83.9) |  94.5 (89.8 – 97.3)
30 |  77.4 (77.3 – 77.5) |  80.3 (75.3 - 85.1) |  93.5 (90.2 – 97.7)
40 |  71.8 (71.7 – 71.9) |  76.6 (71.7 - 81.8) |  93.8 (87.7 – 96.3)
50 |  67.0 (66.9 – 67.0) |  74.1 (68.9 - 80.3) |  92.0 (86.7 – 96.4)

###  IV-B Class-dependent Label Noise

In experiment 2, when labels were randomly flipped to the MU class with the
closes average MUAP shape, DeepLSAL again generated embedding spaces with
clear separation between true and false separation (fig. 6). However, unlike
in the first experiment, the false labels formed a second distinct cluster
within the embedding space, again clearly visualised in the first two
principle component dimensions. As the true label cluster always had more
values, it was still clearly identified by the density-estimator.

As in experiment 1, the DeepLSAL cleaning phase was effective at removing
almost all false labels (fig. 5b). Even at a 40% transfer the median post-
cleaning fraction of 0.0% (0 - 0.6) of the total correct labels. As true
labels were lost both to the initial transfer to other classes and to the
cleaning phase, far fewer were retained in the post-cleaning dataset than in
experiment 1 and would need to be recovered in the refitting stage (table II).

Figure 6: Principal component plot of the embedding space after the cleaning
phase for all classes found in a single HD-sEMG sample in experiment 2. 40% of
the labels in each class have been removed and then added to the class with
the closest correlation. The samples selected automatically for the refitting
phase have been circled. Unlike in experiment 1, the false labels are
correlated as they come from the same class. This results in two tight
clusters for both true and false labels, however they still clearly separable.

###  IV-C Rediscovering Unlabelled Activations

An important requirement if the DeepLSAL-cleaned label set is to be useful is
that the cleaning process does not overly bias against true labels that are
lost at this stage, making them difficult to recover using the retained true
labels. Lost labels tend to be more peripheral in the cluster, meaning their
MUAP shapes are likely to be less similar to the MU class average, potentially
due to superposition with a MUAP from a different class or due to a noise
artefact. False negatives are also still used for training, but are labelled
inappropriately, with a possibly detrimental effect of the model to
generalise. To demonstrate that neither of these potential problems actually
impacted training, after the cleaning stage of both experiment 1 and 2
DeepLSAL was refitted with the cleaned label sets. For both experiments the
predicted activity was generally both sparse and clean, with MUAPs easily
identifiable (fig. 7). These labels were compared to the original data using
the rate of agreement (RoA) metric, a percentage defined as the number of true
positive matches divided by the total number of true positives, false
positives and false negatives.

The RoA of the predicted MUAP labels with the original data was generally good
for both experiments at every level of difficulty (tables I and II). In
experiment 1 there was little change in RoA as difficulty increased (fig. 5c),
suggesting that DeepLSAL is able to generalise to unseen activations. This
finding was also replicated in experiment 2 (fig. 5d), and even when the
median post-cleaning training set was just over half of the total class
activations a median RoA of 94% (86.9 - 97.7) was achieved after refitting.

Figure 7: A single channel of unprocessed HD-sEMG and the post-decomposition predicted activity of a single class before and after cleaning and refitting, with true and false labels. a demonstrates the degree of complex superposition inherent to sEMG signal as opposed to ”cleaner” recordings such as those from intracortical sources. A linear separation filter based on an average of only 15 labels is applied to the signal to generate b, which is consequently extremely noisy, simulating a poorly optimised filter. A number of false positives corresponding to 50% of the number of true class labels has been selected. After the cleaning and refitting phases the spiking motor neuron activity in c is clearly identifiable, whilst incorrect labels have been suppressed. TABLE II: Combined median (interquartile range) scores for all classes across all recordings from experiment 2 for different stages of the cleaning and refitting pipeline. Proportion of Class Labels Transferred (%) |  Starting RoA (%) |  Remaining True Labels After Cleaning (%) |  RoA After Cleaning and Refitting (%)
---|---|---|---
10 |  83.2 (82.1 - 83.9) |  84.1 (84.1 - 88.1) |  98.2 (95.4 – 98.8)
20 |  67.0 (66.1 – 69.0) |  75.4 (70.0 – 77.5) |  96.8 (93.3 – 99.0)
30 |  55.0 (53.1 – 56) |  60.9 (57.3 – 64.9) |  95.7 (88.9 – 98.8)
40 |  43.7 (42.1 - 44.9) |  53.1 (49.1 – 56.7) |  94 (86.9 – 97.7)

##  V Discussion

Source separation is often applied to decode the neural information embedded
in neurophysiological data. This approach provides a window into the neural
determinants of behaviour as well as a way to identify neural features for
human machine interfacing. However, conventional source separation approaches
as well as manual decompositions by expert operators often provides noisy
outputs due to decoding errors. In this study we demonstrated that a
supervised DML paradigm can find self-correcting data embeddings that allow
accurate cleaning of labels corrupted by common noise issues, whilst also
predicting new labels not included within the training set. Importantly the
model was not biased against labels not selected during the cleaning phase
when refitting, which is particularly important in neurophysiological signals,
such as HD-sEMG, where the shape of a subset of activations will be distorted
by temporal superpositions. This ability of the model to operate on un-
preprocessed signals and generalise to unseen activations is particularly
relevant to the field of sEMG decomposition for prosthetic control, where a
current focus is on the use of neural networks to directly identify MUAPs in
raw signal and so avoid the latency involved with the current complex
preprocessing pipeline that prevents online implementations[15][16].

Deep metric learning has a number of attractive properties over standard
softmax-based binary classifiers for neurophysiological time series
classification, needing fewer training examples in general and being able to
adapt to new classes easily[45]. DML methods can adapt quickly to the changes
in class activity commonly seen in neural systems over time, such as with
dropped units in intracortical recordings or MU recruitment and derecruitment
in sEMG and iEMG recordings[46][47]. However, the focus of most
implementations of DML is a high degree of inter-class separation[48], rather
than the need for a more descriptive intra-class embedding needed if this
space is to be used for the identification of false labels. An important
result of this study was that a loss function designed to operate only on
samples local to each other in the embedding space can give rise to a richer
intra-class distribution, avoiding the collapse to a dense point commonly seen
in a more traditional triplet learning paradigm. Furthermore, we demonstrate
that the utility of the embedding space generated by DeepLSAL for label
cleaning is unaffected by feature- or class-dependence in the label noise.

Whilst this study focused on source-separated HD-sEMG signal it is important
to emphasise the broader applicability of this approach to any imperfectly-
labelled neurophysiological time series data in which the underlying sources
are repeating events, i.e. to any neural recording. Whilst the study focused
specifically on action potentials, the proposed methodology could also be used
for pattern recognition in bulk neurophysiological signal, supplementing
recently-proposed systems of prosthetic control[49]. Additionally, the
labelling process need not be by a BSS algorithm. For example, a DeepLSAL
could be applied to a dataset for which only a small component of the data has
been manually labelled by an expert-operator to recover the rest of the labels
accurately. In this way, the proposed approach can be viewed as a minimally
supervised method for neurophysiological time series decomposition into
individual cell activities.

A potential limitation of the study is the method of converting the embedding
layer to a cleaned label set. One strength of the DeepLSAL algorithm was that
it was usually quite simple to identify the main cluster of clean labels in
the embedding space during the cleaning phase, meaning a simple density-
estimator was sufficient to set a decision threshold. However, this approach
tended to cut out a larger proportion of true labels than was potentially
necessary, even if the refitting stage was able to rediscover those lost
labels. An improved sorting methodology is even more relevant to the specific
case when two classes were mixed with equal proportions, which meant that
there was no way of identifying of which of the label clusters was ”correct”.
This implies an interesting future direction for a DML-based approach is to
investigate the use of adaptive k-means clustering to find distinct clusters
within the error labels and hence potentially rehabilitate their averages with
their origin class. Whilst it was not a focus of this study, a an additional
interesting future direction might be found in improving the richness of non-
activation embeddings, which may allow the identification of new unlabelled
classes through clustering events.

In summary, we have presented DeepLSAL, a deep metric learning pipeline for
embedding source-separated multivariate neurophysiological time series into a
dimensionally-reduced space suitable for both classification and label
cleaning. Whilst the focus of the demonstration of this approach in this paper
was performed on electromyographic signal decomposition, the method is broadly
applicable to other neuromuscular recordings, such as intracortical or
intraneural signals.

## References

  * [1] R. Merletti and D. Farina, Surface electromyography: physiology, engineering, and applications.  John Wiley & Sons, 2016.
  * [2] E. Stark and M. Abeles, “Predicting movement from multiunit activity,” Journal of Neuroscience, vol. 27, pp. 8387–8394, Aug. 2007.
  * [3] E. Drebitz, B. Schledde, A. K. Kreiter, and D. Wegener, “Optimizing the yield of multi-unit activity by including the entire spiking activity,” Frontiers in Neuroscience, vol. 13, Feb. 2019.
  * [4] D. Farina and A. Holobar, “Characterization of human motor units from surface EMG decomposition,” Proceedings of the IEEE, vol. 104, pp. 353–373, Feb. 2016.
  * [5] T. Kapelner, I. Vujaklija, N. Jiang, F. Negro, O. C. Aszmann, J. Principe, and D. Farina, “Predicting wrist kinematics from motor unit discharge timings for the control of active prostheses,” Journal of NeuroEngineering and Rehabilitation, vol. 16, Apr. 2019.
  * [6] W. H. Calvin, “Some simple spike separation techniques for simultaneously recorded neurons,” Electroencephalography and Clinical Neurophysiology, vol. 34, pp. 94–96, Jan. 1973.
  * [7] W. Simon, “The real-time sorting of neuro-electric action potentials in multiple unit studies,” Electroencephalography and Clinical Neurophysiology, vol. 18, pp. 192–195, Feb. 1965.
  * [8] H. G. Rey, C. Pedreira, and R. Q. Quiroga, “Past, present and future of spike sorting techniques,” Brain Research Bulletin, vol. 119, pp. 106–117, Oct. 2015.
  * [9] J. Kevric and A. Subasi, “Comparison of signal decomposition methods in classification of EEG signals for motor-imagery BCI system,” Biomedical Signal Processing and Control, vol. 31, pp. 398–406, Jan. 2017.
  * [10] F. Negro, S. Muceli, A. M. Castronovo, A. Holobar, and D. Farina, “Multi-channel intramuscular and surface EMG decomposition by convolutive blind source separation,” Journal of Neural Engineering, vol. 13, p. 026027, Feb. 2016.
  * [11] M. Pachitariu, N. Steinmetz, S. Kadir, M. Carandini, and H. K. D., “Kilosort: realtime spike-sorting for extracellular electrophysiology with hundreds of channels,” bioRxiv preprint, June 2016.
  * [12] J. J. Jun, N. A. Steinmetz, J. H. Siegle, D. J. Denman, M. Bauza, B. Barbarits, A. K. Lee, C. A. Anastassiou, A. Andrei, Ç. Aydın, M. Barbic, T. J. Blanche, V. Bonin, J. Couto, B. Dutta, S. L. Gratiy, D. A. Gutnisky, M. Häusser, B. Karsh, P. Ledochowitsch, C. M. Lopez, C. Mitelut, S. Musa, M. Okun, M. Pachitariu, J. Putzeys, P. D. Rich, C. Rossant, W. lung Sun, K. Svoboda, M. Carandini, K. D. Harris, C. Koch, J. O’Keefe, and T. D. Harris, “Fully integrated silicon probes for high-density recording of neural activity,” Nature, vol. 551, pp. 232–236, Nov. 2017.
  * [13] S. Muceli, W. Poppendieck, K.-P. Hoffmann, S. Dosen, J. Benito-León, F. O. Barroso, J. L. Pons, and D. Farina, “A thin-film multichannel electrode for muscle recording and stimulation in neuroprosthetics applications,” Journal of Neural Engineering, vol. 16, p. 026035, Feb. 2019\.
  * [14] A. D. Vecchio and D. Farina, “Interfacing the neural output of the spinal cord: robust and reliable longitudinal identification of motor neurons in humans,” Journal of Neural Engineering, Oct. 2019.
  * [15] A. K. Clarke, S. F. Atashzar, A. D. Vecchio, D. Barsakcioglu, S. Muceli, P. Bentley, F. Urh, A. Holobar, and D. Farina, “Deep learning for robust decomposition of high-density surface EMG signals,” IEEE Transactions on Biomedical Engineering, vol. 68, pp. 526–534, Feb. 2021.
  * [16] Y. Wen, S. Avrillon, J. C. Hernandez-Pavon, S. J. Kim, F. Hug, and J. L. Pons, “A convolutional neural network to identify motor units from high-density surface electromyography signals in real time,” Journal of Neural Engineering, Mar. 2021.
  * [17] G. Algan and I. Ulusoy, “Label noise types and their effects on deep learning,” arXiv preprint arXiv:2003.10471, 2020.
  * [18] R. I. Kumar, M. M. Mallette, S. S. Cheung, D. W. Stashuk, and D. A. Gabriel, “A method for editing motor unit potential trains obtained by decomposition of surface electromyographic signals,” Journal of Electromyography and Kinesiology, vol. 50, p. 102383, Feb. 2020.
  * [19] K. C. McGill, Z. C. Lateva, and H. R. Marateb, “EMGLAB: An interactive EMG decomposition program,” Journal of Neuroscience Methods, vol. 149, pp. 121–133, Dec. 2005.
  * [20] F. Hug, S. Avrillon, A. D. Vecchio, A. Casolo, J. Ibanez, S. Nuccio, J. Rossato, A. Holobar, and D. Farina, “Analysis of motor unit spike trains estimated from high-density surface electromyography is highly reliable across operators,” bioRxiv preprint, Feb. 2021.
  * [21] D. Carlson and L. Carin, “Continuing progress of spike sorting in the era of big data,” Current Opinion in Neurobiology, vol. 55, pp. 90–96, Apr. 2019\.
  * [22] P. Yger, G. L. Spampinato, E. Esposito, B. Lefebvre, S. Deny, C. Gardella, M. Stimberg, F. Jetter, G. Zeck, S. Picaud, J. Duebel, and O. Marre, “A spike sorting toolbox for up to thousands of electrodes validated with ground truth recordings in vitro and in vivo,” eLife, vol. 7, Mar. 2018.
  * [23] H. Song, M. Kim, D. Park, and J.-G. Lee, “Learning from noisy labels with deep neural networks: A survey,” arXiv preprint, 2020.
  * [24] D. Karimi, H. Dou, S. K. Warfield, and A. Gholipour, “Deep learning with noisy labels: exploring techniques and remedies in medical image analysis,” CoRR, vol. abs/1912.02911, 2019.
  * [25] B. Frenay and M. Verleysen, “Classification in the presence of label noise: A survey,” IEEE Transactions on Neural Networks and Learning Systems, vol. 25, pp. 845–869, May 2014.
  * [26] A. Veit, N. Alldrin, G. Chechik, I. Krasin, A. Gupta, and S. J. Belongie, “Learning from noisy large-scale datasets with minimal supervision,” CoRR, vol. abs/1701.01619, 2017.
  * [27] K. Lee, X. He, L. Zhang, and L. Yang, “Cleannet: Transfer learning for scalable image classifier training with label noise,” CoRR, vol. abs/1711.07131, 2017.
  * [28] J. Han, P. Luo, and X. Wang, “Deep self-learning from noisy labels,” CoRR, vol. abs/1908.02160, 2019.
  * [29] S. Chopra, R. Hadsell, and Y. LeCun, “Learning a similarity metric discriminatively, with application to face verification,” in 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05), IEEE, 2005.
  * [30] J. Wang, F. Zhou, S. Wen, X. Liu, and Y. Lin, “Deep metric learning with angular loss,” 2017.
  * [31] F. Schroff, D. Kalenichenko, and J. Philbin, “Facenet: A unified embedding for face recognition and clustering,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015.
  * [32] A. Hermans, L. Beyer, and B. Leibe, “In defense of the triplet loss for person re-identification,” CoRR, vol. abs/1703.07737, 2017.
  * [33] K. Sohn, “Improved deep metric learning with multi-class n-pair loss objective,” in Advances in Neural Information Processing Systems (D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, eds.), vol. 29, Curran Associates, Inc., 2016.
  * [34] C.-Y. Wu, R. Manmatha, A. J. Smola, and P. Krahenbuhl, “Sampling matters in deep embedding learning,” in Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
  * [35] O. Rippel, M. Paluri, P. Dollar, and L. Bourdev, “Metric learning with adaptive density discrimination,” 2016.
  * [36] X. Wang, Y. Hua, E. Kodirov, G. Hu, R. Garnier, and N. M. Robertson, “Ranked list loss for deep metric learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [37] F. Xu, B. Ma, H. Chang, and S. Shan, “PRDP: Person reidentification with dirty and poor data,” IEEE Transactions on Cybernetics, pp. 1–13, 2021\.
  * [38] Y. Fan, S. Lyu, Y. Ying, and B.-G. Hu, “Learning with average top-k loss,” arXiv preprint arXiv:1705.08826, 2017.
  * [39] A. Holobar and D. Zazula, “Multichannel blind source separation using convolution kernel compensation,” IEEE Transactions on Signal Processing, vol. 55, no. 9, pp. 4487–4496, 2007.
  * [40] D. Farina, F. Negro, S. Muceli, and R. M. Enoka, “Principles of motor unit physiology evolve with advances in technology,” Physiology, vol. 31, pp. 83–94, Mar. 2016.
  * [41] S. Ji, Z. Zhang, S. Ying, L. Wang, X. Zhao, and Y. Gao, “Kullback-leibler divergence metric learning,” IEEE Transactions on Cybernetics, pp. 1–12, 2020.
  * [42] N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. V. Le, G. E. Hinton, and J. Dean, “Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,” CoRR, vol. abs/1701.06538, 2017.
  * [43] A. Holobar and D. Zazula, “Gradient convolution kernel compensation applied to surface electromyograms,” in Independent Component Analysis and Signal Separation, pp. 617–624, Springer Berlin Heidelberg, 2007.
  * [44] A. Holobar, M. A. Minetto, A. Botter, F. Negro, and D. Farina, “Experimental analysis of accuracy in the identification of motor unit spike trains from high-density surface emg,” IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 18, no. 3, pp. 221–229, 2010.
  * [45] H. Oh Song, Y. Xiang, S. Jegelka, and S. Savarese, “Deep metric learning via lifted structured feature embedding,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4004–4012, 2016.
  * [46] C. F. Dunlap, S. C. Colachis, E. C. Meyers, M. A. Bockbrader, and D. A. Friedenberg, “Classifying intracortical brain-machine interface signal disruptions based on system performance and applicable compensatory strategies: A review,” Frontiers in Neurorobotics, vol. 14, Oct. 2020.
  * [47] D. Farina, M. Fosci, and R. Merletti, “Motor unit recruitment strategies investigated by surface EMG variables,” Journal of Applied Physiology, vol. 92, pp. 235–247, Jan. 2002.
  * [48] Q. Qian, L. Shang, B. Sun, J. Hu, H. Li, and R. Jin, “Softtriple loss: Deep metric learning without triplet sampling,” 2020.
  * [49] S. Pancholi and A. M. Joshi, “Advanced energy kernel-based feature extraction scheme for improved EMG-PR-based prosthesis control against force variation,” IEEE Transactions on Cybernetics, pp. 1–10, 2020.


# Improving Prototypical Part Networks with Reward Reweighing, Reselection,
and Retraining

Robin Netzorg∗ Jiaxun Li∗ Bin Yu

University of California, Berkeley
∗ Equal Contribution

###### Abstract

In recent years, work has gone into developing deep interpretable methods for
image classification that clearly attributes a model’s output to specific
features of the data. One such of these methods is the prototypical part
network (ProtoPNet), which attempts to classify images based on meaningful
parts of the input. While this method results in interpretable
classifications, it often learns to classify from spurious or inconsistent
parts of the image. Hoping to remedy this, we take inspiration from the recent
developments in Reinforcement Learning with Human Feedback (RLHF) to fine-tune
these prototypes. By collecting human annotations of prototypes quality via a
1-5 scale on the CUB-200-2011 dataset, we construct a reward model that learns
to identify non-spurious prototypes. In place of a full RL update, we propose
the reweighed, reselected, and retrained prototypical part network
(R3-ProtoPNet), which adds an additional three steps to the ProtoPNet training
loop. The first two steps are reward-based reweighting and reselection, which
align prototypes with human feedback. The final step is retraining to realign
the model’s features with the updated prototypes. We find that R3-ProtoPNet
improves the overall consistency and meaningfulness of the prototypes, but
lower the test predictive accuracy when used independently. When multiple
trained R3-ProtoPNets are incorporated into an ensemble, we find an increase
in test predictive performance while maintaining interpretability.

##  1 Introduction

With the widespread use of deep learning, having these models be interpretable
is more important now than ever. As these models continue to see use in high-
stakes situations, practitioners hoping to justify a decision need to
understand how a deep model makes a prediction, and trust that those
explanations are valuable and correct [15]. One such proposed method for image
classification is the prototypical part network (ProtoPNet), which classifies
a given image based on its similarity to prototypical parts of training
images, called prototypes [7]. This model aims to combine the power of deep
learning with an intuitive reasoning module similar to humans.

While ProtoPNet aims to learn meaningful prototypical concepts, in practice,
learned prototypes suffer from learning spurious concepts, such as the
background of an image, from inconsistent concepts, such as learning both the
head and the wing of a bird, and from duplicating concepts, such as having two
prototypes that correspond to the same wing of the same bird [5]. Such
problems are highly detrimental to the efficacy of these models, resulting in
wasted computation at best and incorrect reasoning at worst. Various methods
have been proposed to account for these issues [5, 13, 4], but these methods
involve either costly labelling procedures or fall short of providing a means
of measuring prototype quality.

We seek to increase the performance of the learned prototypes by taking
inspiration from recent advances in reinforcement learning with human feedback
(RLHF) [14] and reward learning [12]. RLHF and reward learning have become
popular approaches for aligning large language models with human preferences,
partially due to the flexibility of learned rewards and feedback collection
methods [2]. While prior work has incorporated human feedback into ProtoPNets
[4, 5], no variation of ProtoPNet has incorporated a cheap and flexible reward
learning fine-tuning framework.

Towards this end, we propose the reward reweighed, reselected, and retrained
prototypical part network (R3-ProtoPNet), which seeks to improve the original
ProtoPNet via fine-tuning with a learned reward model. With minimal human
feedback data on the Caltech-UCSD Birds-200-2011 (CUB-200-211) dataset [20],
we are able to train a high-quality reward model that achieves $91.5\%$ test
accuracy when ranking human preferences, serving as a strong measure for
prototype quality. R3-ProtoPNet is then able to improve the meaningfulness of
prototypes, removing dependence on spurious features, and is able to slightly
decrease inconsistency across images compared to the original ProtoPNet. When
used as base learners in an ensemble, R3-ProtoPNet is able to outperform an
ensemble of ProtoPNets on a held-out test dataset.

In summary, our contributions are as follows. Firstly, we demonstrate that a
reward model trained on small amounts of human feedback data (roughly 300
ratings) can accurately rank human preference data. Secondly, due to the high
performance of the reward model, we propose using the reward model as a
measure of prototype. Thirdly, we introduce the R3-ProtoPNet, which uses
reward-guided fine-tuning to improve prototype meaningfulness and ensemble
performance.

##  2 Related Work

###  2.1 Reinforcement Learning with Human Feedback

Since the success of InstructGPT [14], Reinforcement Learning with Human
Feedback (RLHF) has received a great deal of attention in the machine learning
community. Although this success is recent, incorporating human feedback into
reinforcement learning methods via a learned reward model has a deep history
in reward learning [8, 11]. While works taking inspiration from InstructGPT
have used proximal policy optimization (PPO) to fine-tune networks with human
feedback [3], it is unclear to the extent that formal reinforcement learning
is necessary to improve models via learned reward functions [12], or if the
human feedback needs to follow a particular form [2]. Some prior work
incorporates the reward function as a way to weigh the likelihood term [17,
22]. Keeping this work in mind, we incorporate the reward model into ProtoPNet
as a way to reweigh prototypes post-training.

###  2.2 Example-based Models and Prototypical Part Networks

The field of interpretable deep learning is vast, with a plethora of
explainability and interpretability methods available to the user. For a more
complete overview of interpretable deep learning, please refer to Rudin et al.
[15]. To ground the discussion, we focus primarily on example-based models,
one such example being ProtoPNet. While ProtoPNet is our model of interest,
other example-based methods exist, such as the non-parametric xDNN [1] or
SITE, which performs predictions directly from interpretable prototypes [19].
While other example-based methods exist, we focus on the ProtoPNet due to its
intuitive reasoning structure.

Since its introduction by Chen et al. [7], ProtoPNets have received a great
deal of attention, and various iterations have been developed. Work has
explored extending the ProtoPNet to different architectures such as
transformers ([21]), or sharing class information between prototypes ([16]).
Donnelly et al. [9] increase the spatial flexibility of ProtoPNet, allowing
prototypes to change spatial positions depending on the pose information
available in the image. ProtoPNets and variations have seen success in high-
stakes applications, such as kidney stone identification ([10]) and
mammography ([4]).

Many works have commented on how the original ProtoPNet tends to overemphasize
spurious features, and they have taken different approaches to solving this
issue. Nauta et al. [13] introduce a explainability interface to ProtoPNet,
allowing users to see the dependence of the prototype on certain image
attributes like hue and shape. The authors claim that seemingly dissimilar or
spurious prototypes share certain difficult-to-perceive features, like texture
or contrast. Barnett et al. [4] introduce a variation of the ProtoPNet, IAIA-
BL, which biases prototypes towards expert labelled annotations of
classification-relevant parts of the image.

Similar to how we provide human feedback at the interpretation level,
Bontempelli et al. [5] introduce the ProtoPDebug, where a user labels a
prototype and image pair as "forbidden" or "valid", and a fine-tuning step
maximizes the distance between learned prototypes and patches in the forbidden
set and minimizes the distance between learned prototypes and patches in the
valid set. While also incorporating human feedback, [5] do not ground their
method in RLHF, but instead includes the binary feedback as a supervised
constraint into the ProtoPNet loss function. Learning a reward function via
ratings allows us to simultaneously increase the interpretability of the
prototypes, and develop an evaluation metric for the quality of a particular
prototype. Compared to previous approaches, reward reweighing, reselection,
and retraining allows for fast collection of high-quality human feedback data
and the construction of a reward model that measures prototype quality while
increasing the interpretability and the performance of the model.

##  3 Prototypical Part Network (ProtoPNet)

In this section, we describe the base architecture used in our method, the
Prototypical Part Network (ProtoPNet) introduced in Chen et al. [7]. The
ProtoPNet aims to introduce interpretability to otherwise uninterpretable
image classifiers. In place of predicting from an arbitrary representation,
the model makes a classification based on part attention and similar
prototypical parts of an image. The general reasoning of a model is to
classify an unseen image by finding training images with similar prototypical
parts to those of the unseen image. This approach allows the user to
interrogate the reasoning of the model, and clearly see which parts of the
image led to the model’s classification.

###  3.1 Description

Here we briefly describe the ProtoPNet, adopting the notation used in Chen et
al. [7]. The ProtoPNet architecture builds on a base convolutional neural
network $f$, which is then followed by a prototype layer denoted $g_{p}$, and
a fully connected layer $h$. Typically, the convolutional features are taken
pretrained models like VGG-19, ResNet-34, or DenseNet-121.

The ProtoPNet injects interpretability into these convolutional architectures
with the prototype layer $g_{p}$, consisting of $m$ prototypes
$\boldsymbol{P}=\\{p_{j}\\}^{m}_{j=1}$ typically of size $1\times 1\times D$,
where $D$ is the shape of the convolutional output $f(x)$. By keeping the
depth the same as the output of the convolutional layer, but restricting the
height and width to be smaller than that of the convolutional output, the
learned prototypes select a patch of the convolutional output. Reversing the
convolution leads to recovering a prototypical patch of the original input
image $x$. Using upsampling, the method constructs a activation pattern per
prototype $p_{j}$.

To use the prototypes to make a classification given a convolutional output
$z=f(x)$, ProtoPNet’s prototype layer computes a max pooling over similarity
scores:
$g_{p_{j}}(z)=\max_{\tilde{z}\in\text{patches}(z)}\log((\lVert\tilde{z}-p_{j}\rVert_{2}^{2}+1)(\lVert\tilde{z}-p_{j}\rVert_{2}^{2}+\epsilon))$,
for some small $\epsilon<1$. This function is monotonically decreasing with
respect to the distance, with small values of
$\lVert\tilde{z}-p_{j}\rVert_{2}^{2}$ resulting in a large similarity score
$g_{p_{j}}(z)$. Assigning $m_{k}$ prototypes for all $K$ classes, such that
$\sum_{k=1}^{K}m_{k}=m$, the prototype layer outputs a vector of similarity
scores that matches parts of the latent representation $z$ to prototypical
patches across all classes. The final layer in the model is a linear layer
connecting similarities to class predictions.

In order to ensure that the prototypes match specific parts of training
images, during training the prototype vectors are projected onto the closest
patch in the training set. For the final trained ProtoPNet, every $p_{j}$
corresponds to some patch of a particular image.

###  3.2 Limitations

While ProtoPNet is capable of providing interpretable classifications, the
base training described in Chen et al. [7] results in prototypes that are
inconsistent and represent spurious features of the image ([4, 5]).
Additionally, same-class prototypes will often converge to the same part of
the image, resulting in duplicate prototypes.

Chen et al. [7] note that a prototype whose top $L$ (usually $L=5$) closest
training image patches come from different classes than the target class tend
to be spurious and inconsistent, focusing on features like the background. To
remedy this issue, they introduce a pruning operation, removing these
prototypes entirely. While pruning does remove dependency on some subpar
prototypes, we find that pruning still leaves some prototypes that rely on
spurious and inconsistent features (Table 3) and does not improve accuracy. We
also find that duplicate prototypes still occur after the pruning operation as
well. We visualize subpar prototypes in Figure 1. For more examples of low-
quality prototypes, please see the supplementary material.

##  4 Human Feedback and the Reward Reweighed, Reselected, and Retrained
Prototypical Part Network (R3-ProtoPNet)

Inspired by the recent advances in reinforcement learning with human feedback
(RLHF) [14], the reward reweighed, reselected, and retrained prototypical part
network (R3-ProtoPNet) utilizes a learned reward model to fine-tune
prototypes. In place of pruning prototypes and sacrificing potential
information, we demonstrate that incorporating human feedback into the
training of the ProtoPNet improves prototype quality while increasing ensemble
accuracy. In this section, we describe the collection of high-quality human
feedback data, our reward model, and how we incorporate the reward model into
the training loop via a three-stage training procedure.

###  4.1 Human Feedback Collection

A crucial aspect behind the success of RLHF methods is the collection of high
quality human feedback data. Unclear or homogeneous feedback may result in a
poor performing reward model [8]. The design of human feedback collection is
vitally important to the training of a useful reward model.

The inherent interpretability of ProtoPNet leads to a useful benefit for RLHF.
Given a trained ProtoPNet, it is possible for a knowledgeable user to directly
critique the learned prototypes. Given a particular classification task, a
human with enough expertise should be able to recognize if a particular
prototype is "good" or "bad" [5]. In the case of classifying birds in the
CUB-200-2011 dataset, one of the original classification tasks used in Chen et
al. [7], it is clear that if a prototype gives too much weight to the
background of the image (spurious), or if the prototype corresponds to
different parts of the bird when looking at different images (inconsistency),
the learned prototype is not meaningfully or interpretably contributing to
prediction. Given these prototypes that fail to contribute to prediction, a
knowledgeable human trying to classify birds would rate these prototypes as
"bad".

There are many different ways to elicit this notion of "goodness" from a user
[2]. Although it is possible to incorporate many different forms of feedback
into the R3-ProtoPNet, such as asking a user to compare prototypes to elicit
preferences or ask for a binary value of whether a prototype is "good" or
"bad", we found most success with asking the user to rate a prototype on a
scale from 1 to 5. While scalar ratings can be unstable across different
raters, with a clear, rule-based rating method, rating variance is reduced and
it is possible to generate high-quality labels. An example rating scale on the
CUB-200-2011 dataset is provided in Figure 1.

Figure 1: Rubric used for human feedback on the activation patterns of
predictions for birds from the CUB-200-2011 dataset. Ratings of 4-5 are
correspond to high-quality prototypes, 1-2 to low-quality prototypes, and 3 to
unclear quality prototypes.

###  4.2 Reward Learning

We note that, when a user provides feedback on a prototype, it is not the
training image or the model prediction that the user is providing feedback on,
but the prototype’s resulting interpretation: the activation patterns. Our
task is therefore different from RLHF applied to language modeling or RL tasks
([14], [8]), where human feedback is provided on the model output or resulting
state. We therefore collect a rating dataset
$\mathcal{D}=\\{(x_{i},y_{i},h_{i,j},r_{i,j})\\}_{i=1,j=1}^{n,m}$, where
$x_{i},y_{i}$ are the training image and label, and $h_{i,j},r_{i,j}$ are
prototype $p_{j}$’s activation patterns and user-provided activation patterns
for image $x_{i}$. We note that collecting preferences for this entire dataset
is prohibitive and unnecessary, so we only collect a subset.

Given the dataset $\mathcal{D}$, we generate the induced comparison dataset,
whereby each entry in $\mathcal{D}$ is paired with one another. Given $i\neq
i^{\prime}$ and/or $j\neq j^{\prime}$, we populate a new paired dataset,
$\mathcal{D}_{paired}$, which consists of the entries of $\mathcal{D}$ indexed
by $i,j,i^{\prime},j^{\prime}$, and a comparison $c$, which takes values
$-1,0,1$. If the left-hand sample is greater, and therefore considered higher-
quality, $r_{i,j}>r_{i^{\prime},j^{\prime}}$, then $c=-1$. If the right-hand
sample is greater $r_{i,j}<r_{i^{\prime},j^{\prime}}$, then $c=1$. We note
that, during learning, we exclude entries with $c=0$ to increase the contrast
between pairs. This synthetic construction allows us to model the reward
function, $r(x_{i},h_{i,j})$, via the Bradley-Terry Model for pairwise
preferences [6]. We train this model with the same loss function as in
Christiano et al. [8], a cross-entropy loss over the probabilities of ranking
one pair over the other. This synthetic construction combinatorially increases
the amount of preference data, allowing us to train a high-quality reward
model on relatively small amounts of quality human feedback data.

###  4.3 Reward Reweighed, Reselected, and Retrained Prototypical Part Network
(R3-ProtoPNet)

After having collected high-quality human feedback data and trained a reward
model, we can now incorporate it into a fine-tuning framework to improve the
interpretability of ProtoPNet. We incorporate the reward model via a three
step process consisting of reward weighting, reselection, and retraining. Each
step is described in more detail below.

####  4.3.1 Reward Reweighing

Although PPO is a popular option for RLHF ([14]), there is evidence that
simpler fine-tuning algorithms can lead to similar performance increases
([2]). Inspired by the success and the ease of implementation of reward-
weighted learning [12, 17, 22], we develop a reward-weighted update for the
ProtoPNet:

$\max_{p_{j}}\mathcal{L}_{reweigh}(z_{i}^{*},p_{j})=\max_{p_{j}}\sum_{i\in
I(p_{j})}^{n}r(x_{i},p_{j})\frac{1}{\lambda_{dist}\lVert
z_{i}^{*}-p_{j}\rVert^{2}_{2}+1}$ (1)

where $z_{i}^{*}=\text{argmin}_{z\in\text{patches}(f(x_{i}))}\lVert
z-p_{j}\rVert_{2}^{2}$, $I(p_{j})=\\{i\ \lvert\
y_{i}\in\text{class}(p_{j})\\}$, and $\lambda_{dist}$ is a fixed
hyperparameter. We note that the loss function $\mathcal{L}_{reg}$ is a sum of
the inverse distances weighted by the reward of the prototype on that image.
Since we only update the prototype $p_{j}$, the only way to maximize the loss
is to minimize the distance between prototype and image patches with high
reward $r(x_{i},p_{j})$. This causes the prototype to resemble high reward
image patches, improving the overall quality of the prototypes. Wanting to
preserve prototypes that already have high reward, we only update those
prototypes that have relatively low mean reward less than $\gamma=0.45$.
$\lambda_{dist}$ is included in the loss function to rescale distances, since
the closest distances are near zero. We find best performance with
$\lambda_{dist}=100$.

Practically, we find that optimizing this loss function leads to locally
maximal solutions, resulting in local updates that do not modify prototypes
with low quality values of $1$, but it’s more likely to improve prototypes
with quality values of $2$ or higher. If the prototype $p_{j}$ has high
activation over the background of an image $x_{i}$, for example, the closest
patches $z_{i}^{*}$ in the training data will also be background patches, and
the reward of the prototype will be low, leaving minimal room for change. It
is not possible for this update to dramatically change the location of the
patch in the image via this loss function.

####  4.3.2 Prototype Reselection

In order to improve low quality prototypes that require significant
manipulation, we introduce a reselection procedure based on a reward
threshold. Given a prototype $p_{j}$ and image $x_{i}$, if
$\frac{1}{n_{k}}\sum_{i\in I(p_{j})}r(x_{i},p_{j})<\alpha$, where $\alpha$ is
a pre-determined threshold and $n_{k}$ is the number of training images in
class $k$, we reselect the prototype. The reselection process involves
iterating over patch candidates $z^{\prime}_{i}$ and temporarily setting the
prototype $p^{\prime}_{j}=z^{\prime}_{i}$, where $z^{\prime}_{i}$ is chosen
randomly from the patches of a randomly selected image $x^{\prime}_{i}$ in the
class of $p_{j}$. If $\frac{1}{n_{k}}\sum_{i\in
I(p_{j})}r(x_{i}^{\prime},p^{\prime}_{j})>\beta$, where $\beta$ is an
acceptance threshold, and if none of the prototypes match patch
$p^{\prime}_{j}=z^{\prime}_{j}$, then we accept the patch candidate as the new
prototype. We found that $\alpha=0.15$ and $\beta=0.50$ led to good
performance. We refer to the combination of reweighting and reselection as the
R2 update step, and the corresponding trained model the R2-ProtoPNet.

The reasoning process behind our prototype reselection method takes
inspiration from the original push operation in Chen et al. [7]. Similar to
how ProtoPNet projects prototypes onto a specific training image patch, here
we reselect prototypes to be a particular reward-filtered training image
patch. With a high enough acceptance threshold $\beta$, this forces the
elimination of low reward prototypes while preserving the information gain of
having an additional prototype.

One possible alternative approach is to instead search over the training
patches, and select those patches with the highest reward. We found that
randomly selecting patches, in place of searching for patches with the highest
reward, led to higher prototype diversity and less computation time. As
discussed in Section 6, it is possible that a reward model that more
explicitly accounts for prototype diversity could alleviate the duplicate
issue, but we leave this to future work.

While we do not use a traditional reinforcement learning algorithm to fine-
tune our model as is typically done in RLHF [2], pairing the reselection and
fine-tuning steps together resembles the typical explore-exploit trade-off in
RL problems. We see that fine-tuning with our reward model leads to exploit
behavior, improving upon already high-quality prototypes. At the same time,
the reselection step serves as a form of exploration, drastically increasing
the quality of uninformative prototypes. We find that these similarities are
enough to improve the quality of ProtoPNet, as discussed in the next section.

####  4.3.3 Retraining

A critical step missing in the R2 update is a connection to prediction
accuracy. As discussed in Section 5, without incorporating predictive
information, performing the reward update alone results in lowered test
accuracy. Since the above updates only act on the prototypes themselves, not
the rest of the network, the result is a misalignment between the prototypes
and the model’s base features and final predictive layer. The reward update
guides the model towards more interpretable prototypes, but the reward update
alone fails to use the higher quality prototypes for better prediction.

To account for the lack of predictive performance, the final step of
R3-ProtoPNet is retraining. Simply retraining with the same loss function used
in the original ProtoPNet update results in the realignment of the prototypes
and the rest of the model. Although one could worry that predictive accuracy
would reduce the interpretability of the model [15], we find that retraining
increases predictive accuracy while maintaining the quality increases of the
R2 update. The result is a high accuracy model with higher-quality prototypes.
We explore evidence of this phenomenon and why this is the case in the
following section.

##  5 Experiments

Here we discuss the results of training the R3-ProtoPNet on the CUB-200-2011
dataset, the same dataset as used in Chen et al. [7]. We demonstrate that the
R3-ProtoPNet leads for higher quality prototypes across base model
architectures and prototype configurations while not sacrificing predictive
performance.

###  5.1 Datasets

R3-ProtoPNet requires two datasets: the original dataset for initial training,
and the scalar ratings of activation pattern dataset. Combined, this results
in the dataset described in Section 4. To offer better comparison against the
original ProtoPNet, we use the same dataset for initial training that was used
in Chen et al. [7], the CUB-200-2011 dataset [18]. The CUB-200-2011 dataset
consists of roughly 30 images of 200 different bird species. We employ the
same data augmentation scheme used in Chen et al. [7], which adds additional
training data by applying a collection of rotation, sheer, and skew
perturbations to the images, resulting in a larger augmented dataset.

For the collection of the activation pattern ratings, we only provide
activation patterns overlaid on the original images to the rater. Although it
is possible to crowdsource the collection of human preference data, we found
that it was possible to increase the performance of ProtoPNet with relatively
small amounts human preference data that we ourselves collected. We rated a
total of 700 prototype-image pairs according to the scale approach described
in Figure 1, which we justify in the next subsection.

###  5.2 Architectures and Training

Similar to Chen et al. [7], we study the performance of R3-ProtoPNet across
three different base architectures: VGG-19, ResNet-34, and DenseNet-121. While
the original ProtoPNet sets the number of prototypes per class at $m_{k}=10$,
we additionally run the VGG19 architecture with $m_{k}=5$ prototypes to
explore model performance when the number of prototypes is limited. No other
modifications were made to the original ProtoPNet architecture. We train for
100 epochs and report results for the best performing model.

The reward model $r(x_{i},h_{i})$ is similar to the base architecture of the
ProtoPNet. Two ResNet-50 base architectures take in the input image $x_{i}$
and the associated acticvation pattern $h_{i}$ separately, and both have two
additional convolutional layers. The outputs of the convolutional layers are
concatenated and fed into a final linear layer with sigmoid activation to
predict the Bradley-Terry ranking. Predicted rewards are therefore bound in
the range $(0,1)$. We train the reward model for 5 epochs on a comparison
dataset of 71,875 paired images and preference labels, and evaluate on a
13,831 testing pairs. The reward model achieves 91.54% test accuracy when
trained on the whole dataset, and we additionally find that the reward model
converges to roughly 91% test accuracy on a comparison dataset generated from
at least 300 rated activation patterns.

###  5.3 Evaluation Metrics

To evaluate the performance of R3-ProtoPNet, we compare it to ProtoPNet using
three metrics: test accuracy, reward, and prototype class mismatch. We use
test accuracy to measure the predictive performance of the models. As the
above section demonstrates, the learned reward model achieves high accuracy in
predicting which prototype ranks above another in accordance with human
preferences, so we therefore use it as a measure of prototype quality.
Regarding the class mismatch metric, Chen et al. [7] note that low-quality
prototypes tend to have close training images that come from different
classes. To evaluate the effect of R3 updating, we compute the average class
mismatch across all prototypes for a given model for the Top-5 and Top-10
closest training images.

###  5.4 Results

Base ($m_{k})$ | ProtoPNet | R2-ProtoPNet | R3-ProtoPNet
---|---|---|---
VGG-19 ($5$) | 73.44% | 57.36% | 74.12%
VGG-19 ($10$) | 73.80% | 54.25% | 74.73%
ResNet-34 ($10$) |  77.49% | 52.88% | 76.32%
DenseNet-121 ($10$) |  75.68% | 64.18% | 74.63%
Ensemble (VGG-19(10) + ResNet-34 + DenseNet-121) | 80.53% | 74.89% |  80.86%
Ensemble (All 4 Base Models) | 80.91% | 77.77% |  82.00%
Table 1: Test accuracy of different base architectures during the stages of R3-ProtoPNet training, where $m_{k}$ is the number of prototypes per class. Ensembles consist of the corresponding individually trained networks. Base ($m_{k})$ | ProtoPNet | R2-ProtoPNet | R3-ProtoPNet
---|---|---|---
VGG19 ($5$) | 0.56 | 0.64 | 0.68
VGG19 ($10$) | 0.48 | 0.65 | 0.65
ResNet34 ($10$) | 0.57 | 0.61 | 0.65
DenseNet121 ($10$) | 0.52 | 0.68 | 0.69
Table 2: Average rewards over the test dataset of different base architectures during the stages of R3-ProtoPNet training. Base ($m_{k})$ | ProtoPNet | R2-ProtoPNet | R3-ProtoPNet
---|---|---|---
VGG19 ($5$) | 2.77, 5.75 | 2.59, 5.39 | 1.45, 3.16
VGG19 ($10$) | 3.28, 6.82 | 3.00, 6.30 | 2.77, 5.84
ResNet34 ($10$) | 4.07, 8.52 | 4.05, 8.37 | 4.05, 8.29
DenseNet121 ($10$) | 3.52, 7.49 | 2.78, 6.26 | 3.55, 6.53
Table 3: Average class mismatch of prototypes and the Top-5, Top-10 closest
training image patches across different base architectures during the stages
of R3-ProtoPNet training.

After training ProtoPNet, running the R2 update step, and then performing
retraining, we see several trends across multiple base architectures. In Table
1, we report the test accuracy of the different base architectures across
stages of R3-ProtoPNet training. Generally, the test accuracy from ProtoPNet
substantially decreases after applying the R2 update, but retraining tends to
recover most of the predictive loss. This accuracy maintenance demonstrates
that it is possible to align prototypes with human preferences without
sacrificing predictive power.

In Table 2, we report the average reward of all prototypes on all test images
for a given base architecture. We see that ProtoPNet achieves an average
reward between 0.48 and 0.57 across architectures. Investigating the
distribution of rewards further in Figure 2(a), it is revealed that ProtoPNet
tends to produce a bimodal distribution over prototype rewards, with some bias
towards low-quality and high-quality prototypes. Applying the R2 update
results in the desired behavior, increasing the average reward and shifting
the distribution of rewards upwards. We additionally see that the retraining
step in R3-ProtoPNet actually continues to increase average reward across all
base architectures while slightly increasing the spread of the reward
distribution.

(a) ProtoPNet

(b) R2-ProtoPNet

(c) R3-ProtoPNet

Figure 2: Histograms of the rewards of prototype,image pairs for (a)
ProtoPNet, (b) R2-ProtoPNet, and (c) R3-ProtoPNet using VGG19 with 10
prototypes per class.

Finally, we report the Top-5 and Top-10 class mismatch in Table 3. Here we see
an interesting phenomena. Across all base architectures, ProtoPNet has an
average class mismatch of at least half of the Top-L closest image patches,
for both $L=5,10$. Although performing the R2 greatly increases the average
reward for all base architectures except ResNet-34, we see that class mismatch
is only marginally reduced, with still all of the base architectures resulting
in mismatches for over half of the closest Top-L training image patches. We
see that R3-ProtoPNet greatly reduces class mismatch for the $m_{k}=5$ VGG-19
base architecture, but tends to only marginally reduce class mismatch for the
$m_{k}=10$ case.

###  5.5 Discussion

Given the results, we see that R3-ProtoPNet manages to increase the quality of
learned prototypes without sacrificing predictive performance. While the
ResNet-34 and DenseNet-121 base architectures do see a slight performance
decrease, producing an ensemble of trained R3-ProtoPNets results in an
accuracy increase over an ensemble of the original trained ProtoPNets. We see
that R3-ProtoPNet results in a substantial increase of the average test
reward, verifying that prototype quality is increasing. There is still much
room for improvement, as class mismatch for $10$ prototypes does not decrease
across all architectures, while there is some class mismatch decrease for the
$5$ prototype VGG-19-based ProtoPNet. Overall, these results demonstrate that
incorporating reward information into the ProtoPNet via reweighing,
reselection, and retraining does increase interpretability of ProtoPNets, and,
when incorporated into an ensemble, increases predictive performance.

##  6 Limitations and Future Work

While R3-ProtoPNet improves interpretability and predictiveness in an
ensemble, there is plenty of room for improvement. We note that the reward
model is trained on ratings of a single image and heatmap, highly constrained
to measuring overlap between prototype and the object of interest, but it is
quite possible to extend ratings to multiple images and heatmaps. This would
allow for the reward model to better learn cross-image preferences, such as
consistency. We hope that this could alleviate the duplicate issue as well. We
note that R3-ProtoPNet fails to entirely eliminate duplicates, with several
high-reward prototypes converge to the same part of the image.

While this work investigated increasing the performance of ProtoPNet, it is
possible to extend the R3 update to other extensions of the ProtoPNet. A major
benefit of reward fine-tuning is its flexibility in application, and we expect
that combining the R3 update with other variations of the ProtoPNet would
result in further increased performance gains. Combining multiple feedback
modalities, such as the binary feedback used in ProtoPDebug [5], could further
increase model performance.

A final limitation with R3-ProtoPNet and other methods that rely on human
feedback is that the model itself might be learning features that, while
seemingly confusing to a human, are helpful and meaningful for prediction.
Barnett et al. [4] argue that the ProtoPNet can predict with non-obvious
textures like texture and contrast, which might be penalized via a learned
reward function. Future work is necessary to investigate how ProtoPNet
variants could critique human feedback, and argue against a learned reward
function.

##  7 Conclusion

In this work, we propose the R3-ProtoPNet, a method that uses a learned reward
model of human feedback to improve the meaningfulness of learned prototypical
parts. We find that ensembling multiple R3-ProtoPNets results in increased
performance over original ProtoPNet ensembles. Considering the high
performance of the reward model, we use the reward model as a measure of
prototype quality, allowing us to critique the interpretability of ProtoPNet
along a human lens. The ability of reward learning to quantize qualitative
human preferences make reward-based fine-tuning a promising direction for the
improvement of interpretable deep models.

## References

  * Angelov and Soares [2019] P. Angelov and E. Soares.  Towards explainable deep neural networks (xdnn), 2019.
  * Askell et al. [2021] A. Askell, Y. Bai, A. Chen, D. Drain, D. Ganguli, T. Henighan, A. Jones, N. Joseph, B. Mann, N. DasSarma, et al.  A general language assistant as a laboratory for alignment.  _arXiv preprint arXiv:2112.00861_ , 2021.
  * Bai et al. [2022] Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, N. Joseph, S. Kadavath, J. Kernion, T. Conerly, S. El-Showk, N. Elhage, Z. Hatfield-Dodds, D. Hernandez, T. Hume, S. Johnston, S. Kravec, L. Lovitt, N. Nanda, C. Olsson, D. Amodei, T. Brown, J. Clark, S. McCandlish, C. Olah, B. Mann, and J. Kaplan.  Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022.
  * Barnett et al. [2021] A. J. Barnett, F. R. Schwartz, C. Tao, C. Chen, Y. Ren, J. Y. Lo, and C. Rudin.  Iaia-bl: A case-based interpretable deep learning model for classification of mass lesions in digital mammography, 2021.
  * Bontempelli et al. [2023] A. Bontempelli, S. Teso, K. Tentori, F. Giunchiglia, and A. Passerini.  Concept-level debugging of part-prototype networks, 2023.
  * Bradley and Terry [1952] R. A. Bradley and M. E. Terry.  Rank analysis of incomplete block designs: I. the method of paired comparisons.  _Biometrika_ , 39(3/4):324–345, 1952.
  * Chen et al. [2019] C. Chen, O. Li, D. Tao, A. Barnett, C. Rudin, and J. K. Su.  This looks like that: deep learning for interpretable image recognition.  _Advances in neural information processing systems_ , 32, 2019.
  * Christiano et al. [2017] P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei.  Deep reinforcement learning from human preferences.  _Advances in neural information processing systems_ , 30, 2017.
  * Donnelly et al. [2022] J. Donnelly, A. J. Barnett, and C. Chen.  Deformable protopnet: An interpretable image classifier using deformable prototypes.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 10265–10275, June 2022.
  * Flores-Araiza et al. [2022] D. Flores-Araiza, F. Lopez-Tiro, E. Villalvazo-Avila, J. El-Beze, J. Hubert, G. Ochoa-Ruiz, and C. Daul.  Interpretable deep learning classifier by detection of prototypical parts on kidney stones images, 2022.
  * Jeon et al. [2020] H. J. Jeon, S. Milli, and A. Dragan.  Reward-rational (implicit) choice: A unifying formalism for reward learning.  In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, _Advances in Neural Information Processing Systems_ , volume 33, pages 4415–4426. Curran Associates, Inc., 2020.  URL https://proceedings.neurips.cc/paper_files/paper/2020/file/2f10c1578a0706e06b6d7db6f0b4a6af-Paper.pdf.
  * Lee et al. [2023] K. Lee, H. Liu, M. Ryu, O. Watkins, Y. Du, C. Boutilier, P. Abbeel, M. Ghavamzadeh, and S. S. Gu.  Aligning text-to-image models using human feedback.  _arXiv preprint arXiv:2302.12192_ , 2023.
  * Nauta et al. [2021] M. Nauta, A. Jutte, J. Provoost, and C. Seifert.  This looks like that, because … explaining prototypes for interpretable image recognition.  In _Communications in Computer and Information Science_ , pages 441–456. Springer International Publishing, 2021.  doi: 10.1007/978-3-030-93736-2_34.  URL https://doi.org/10.1007%2F978-3-030-93736-2_34.
  * Ouyang et al. [2022] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe.  Training language models to follow instructions with human feedback, 2022\.
  * Rudin et al. [2021] C. Rudin, C. Chen, Z. Chen, H. Huang, L. Semenova, and C. Zhong.  Interpretable machine learning: Fundamental principles and 10 grand challenges, 2021.
  * Rymarczyk et al. [2021] D. Rymarczyk, L. Struski, J. Tabor, and B. Zieliński.  Protopshare: Prototypical parts sharing for similarity discovery in interpretable image classification.  In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_, KDD ’21, page 1420–1430, New York, NY, USA, 2021\. Association for Computing Machinery.  ISBN 9781450383325.  doi: 10.1145/3447548.3467245.  URL https://doi.org/10.1145/3447548.3467245.
  * Stiennon et al. [2022] N. Stiennon, L. Ouyang, J. Wu, D. M. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. Christiano.  Learning to summarize from human feedback, 2022.
  * Wah et al. [2011] C. Wah, S. Branson, P. Welinder, P. Perona, and S. J. Belongie.  The caltech-ucsd birds-200-2011 dataset.  2011\.
  * Wang and Wang [2021] Y. Wang and X. Wang.  Self-interpretable model with transformationequivariant interpretation, 2021.
  * Welinder et al. [2010] P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona.  Caltech-ucsd birds 200.  Technical Report CNS-TR-201, Caltech, 2010.  URL /se3/wp-content/uploads/2014/09/WelinderEtal10_CUB-200.pdf,http://www.vision.caltech.edu/visipedia/CUB-200.html.
  * Xue et al. [2022] M. Xue, Q. Huang, H. Zhang, L. Cheng, J. Song, M. Wu, and M. Song.  Protopformer: Concentrating on prototypical parts in vision transformers for interpretable image recognition, 2022.
  * Ziegler et al. [2019] D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving.  Fine-tuning language models from human preferences.  _arXiv preprint arXiv:1909.08593_ , 2019.