#  1e0a: A COMPUTATIONAL APPROACH TO RHYTHM TRAINING

###### Abstract

We present a computational assessment system that promotes the learning of
basic rhythmic patterns. The system is capable of generating multiple rhythmic
patterns with increasing complexity within various cycle lengths. For a
generated rhythm pattern the performance assessment of the learner is carried
out through the statistical deviations calculated from the onset detection and
temporal assessment of a learner’s performance. This is compared with the
generated pattern, and their performance accuracy forms the feedback to the
learner. The system proceeds to generate a new pattern of increased complexity
when performance assessment results are within certain error bounds. The
system thus mimics a learner-teacher relationship as the learner progresses in
their feedback-based learning. The choice of progression within a cycle for
each pattern is determined by a predefined complexity metric. This metric is
based on a coded element model for the perceptual processing of sequential
stimuli. The model earlier proposed for a sequence of tones and non-tones, is
now used for onsets and silences. This system is developed into a web-based
application and provides accessibility for learning purposes. Analysis of the
performance assessments shows that the complexity metric is indicative of the
perceptual processing of rhythm patterns and can be used for rhythm learning.

Keywords: Music education, rhythm, onsets, performance assessment, complexity
metrics, coded element processing, web application.

##  1 Introduction

Learning to play a musical instrument is a complex process that requires the
acquisition of many interlinked skills such as tone, intonation, note
accuracy, timing, rhythm precision, clear articulation, and dynamic variation
[1]. An acknowledged important skill is the understanding of rhythm, timing,
and rhythm precision. Rhythm is defined to be the description and
understanding of the duration and durational patterns of musical notes [2].
The word rhythm is used to refer to all of the temporal aspects of a musical
work [3].
Two main contexts within which formal musical learning takes place are the
classroom, through playing and interacting with a teacher, and at home,
practicing alone [1]. From the perspective of a drum kit or percussive
instrument, a typical classroom setting involves the teacher and a learner.
The teacher decides what the learner must practice and the teacher’s feedback
helps them achieve accuracy in timing and quality in performance. The
learner’s practicing habits at home aid in reaching performance goals. A
software application cannot substitute a music teacher but it can be a
compliment in the learning process, especially in the performance assessment
of daily practice at home [1].

###  1.1 Related Literature

Most assessment methods are either the assessment of learning, used to measure
competence, or the assessment for learning used to enhance learning [4-6]. The
rapid development of music technology over the last few decades has
dramatically changed the way that people interact with music today. In [7],
the authors explore the use of Music Information Retrieval (MIR) techniques in
music education and their integration in learning software. They outline
Song2See as a representative example of a music learning system developed
within the MIR community. Songs2See is a music game developed based on pitch
detection, sound separation, music transcription, interface development, and
audio analysis technologies [8]. Despite advancements in these sound analysis
technologies, Eremenko V, Morsi A, et al. in [1], revealed that the
performance assessment tools present in these applications are still not
reliable and effective enough to support the strict requirements of a
professional music education context. In [1], they review some of the work
done in the practice of music assessment and present a proof of concept for a
complete assessment system for supporting guitar exercises. Furthermore, they
identify the challenges that should be addressed to further advance these
assessment technologies and their useful integration into professional
learning contexts. Particularly those, that will help learners to plan,
monitor, and evaluate their learning progression. They also highlight the
importance and requirement of presenting the assessment results to a learner
in a way that promotes learning [1]. Similarly, Lerch, A, et al. in [9]
surveyed the field of Music Performance Analysis (MPA) from the perspective of
performance assessment and its ability to provide insights and interactive
feedback by analyzing and assessing the audio of practice sessions.
Furthermore, they highlight the necessity for individual assessment and
curriculum in the context of a music education tool. The work of Chih-Wei Wu
and Alexander Lerch in [10], presents an unsupervised feature learning
approach to derive features for assessing percussive instrument performances
amongst a large data of audition performances, mimicking a judge in this
scenario. Specifically, they proposed using a histogram-based input
representation to sparse coding to allow the sparse coding to take advantage
of temporal rhythmic information. Various (commercially available) solutions
exist specifically for rhythm education and performance assessment. Examples
for rhythm tutoring applications are Perfect Ear [11], Rhythm Trainer [12],
and Complete Rhythm Trainer [13]. These form a class of mobile applications
that help learners practice and assess their rhythm skills by following a set
curriculum presented in each of these applications and touchscreen-based
feedback for performance assessment. Other apps such as SmartMusic [14],
Yousician [15], Music Prodigy [16], and SingStar [17] are available for music
education purposes however, they do not have a dedicated rhythm training
section. Furthermore, none of the previous works address musical rhythm
complexity measures as a feature to aid in a learner’s progression. Evaluating
various rhythm complexity measures based on how accurately they reflect human-
based measures were studied by Eric Thul in his thesis [3]. Although his
results suggest that none of the measures accurately reflect the difficulty
humans have performing or listening to rhythm, the measures do accurately
reflect how humans recognize a rhythm’s structure.

###  1.2 Our Contribution

In this work, we consider the individual assessment of learning a particular
rhythmic pattern at a fixed tempo. We present the proof of concept for a
system that will assess a learner’s audio recording and provide feedback to
the learner on their performance. We will only focus on assessing the
learner’s temporal proficiency, keeping the performance parameter categories
of tempo, dynamics, pitch, and timbre fixed. Furthermore, we focus on the
progression of a learner within a certain cycle length. We do so with the
introduction of new patterns of increased complexity based on a pre-defined
complexity metric that is similar to a teacher who decides what the learner
must practice.
This manuscript is organized as follows, we first define the terms, notations,
and conventions to be held throughout that will help assess the performance of
an individual. That is followed by the proposed system for the assessment
where we go over the subsequent blocks that make up the assessment system and
understand the coded element model used to define the complexity metric.
Finally, we discuss the learner results and accuracy of this system through
our web application platform, and conclude with future enhancements and
improvements to the system.

##  2 Background

###  2.1 Definitions

Figure 1: Diagrammatic illustration of the terms used in this article

We first provide a brief background of some of the terms used to define a
rhythmic pattern.
In this work we consider the fundamental and individual time unit for a
rhythmic pattern to be a pulse. Pulse is defined to be “one of a series of
regularly recurring, precisely equivalent stimuli” [18]. The duration of a
rhythmic pattern is composed of a number of pulses. Tempo refers to the rate
at which the musical pulses are generated and it also controls the number of
pulses for a given duration. For our proposed system the total duration of a
rhythmic pattern is determined by the total number of samples present in the
audio signal. The uniform number of samples or duration, between two
subsequent pulses, is defined to be the inter-pulse interval (IPI). The IPI
depends on the sampling frequency and the tempo used for the rhythmic pattern.
Figure 1 shows a duration of 300000 samples with pulses generated at a fixed
Inter Pulse Interval (IPI), determined by a tempo of 160 Pulses Per Minute
(PPM).
To generate a rhythmic pattern, we consider instantaneous attacks on the
marked time units (pulses) i.e., striking a drum, clapping, or tapping. The
instantaneous attacks are defined as onsets. The duration that is marked
between two onsets is defined as the inter-onset interval (IOI). The IPI,
onsets, and the IOI together create a rhythmic pattern as shown in figure 1.
A cycle determines the number of pulses beyond which the rhythmic pattern
begins to repeat itself. The cycle represents a structure for the pulses, and
this structure is realised by the rhythmic pattern. For example, we use 7 to
indicate a repeating rhythmic pattern of 7 pulses. Rhythm patterns can be
composed in any predetermined cycle number, for this article we will only be
following cycles of 4,3,5,7, and 16.

###  2.2 Representation

We will now introduce the rhythm pattern representation used in the proposed
system with the help of a famous Afro-Cuban rhythm, the clave son, shown in
figure 2 (a).

(a)

(b)

Figure 2:  (a) The clave son rhythm in music (stave) notation. (b) The clave
son rhythm in binary notation.

Figure. 2 (b) shows the clave son rhythm as a binary string, a notation that
describes the position of the onsets and silences. This notation is used in
the domain of computer science [19]. The onsets are marked by a “1” and the
silent pulses are marked by a “0”. This representation enables faster
computation and integration of rhythm patterns into our system. This
representation of a rhythm pattern henceforth will be referred to as the
binary representation.

###  2.3  Naming 1e0a

The name ‘1e0a’ for the proposed system is derived from a mixture of the
binary representation with its 1’s and 0’s and a western rhythm counting
technique used by a large majority of music educators, called the Harr system.
The system that Harr used assigned a set of numbers and syllables. For a cycle
of length 4, numbers are assigned to onsets that occur at the beginning of the
rhythmic pattern’s duration and its repeats. Syllables are assigned to those
onsets that fall everywhere else. The syllables remain consistent for all the
pulses with only the number for each repeat changing. For example, the rhythm
$[11111111]$ would be read, “1 e & a 2 e & a” [20]. Thereby, giving us the
name ‘1e0a’.

##  3 Proposed System

Figure 3: Block diagram of the proposed system for ‘1e0a.’

Figure 3 shows the block diagram of our proposed system for rhythm training
and progression in learning. This system consists of two distinct blocks: (1)
The rhythm pattern generator with the complexity metric, and, (2) the
performance assessment of the learner input with feedback.

###  3.1 Pattern Generator

####  3.1.1 Pattern Generation

The pattern generator as shown in figure 5 begins with a pulse generator
function. This function takes input parameters that determine the cycle
length, pattern array, tempo (PPM), and the total number of repeats, required
to generate a collection of pulses whose inter-pulse interval (IPI) is
determined by the equation 1.

$IPI=(60/PPM)/(1/fs)$ (1)

$PPM$ represents the pulses per minute of the generated collection of pulses,
$fs$ represents the sampling frequency and $IPI$ represents the fixed inter-
pulse interval.
The pattern array in binary representation determines the location of onsets
and the number of silences to be incorporated into the duration of pulses.
Furthermore, we propose the selection of pattern arrays for generating
subsequent sequences using complexity metrics detailed in section 3.1.2.
For the clave son: [1 0 0 1 0 0 1 0 0 0 1 0 1 0 0 0] , at 160 $PPM$, with 4
repeats, at an $fs$ of 44100. The function generates an array of pulses as
shown in figure 4(a). The fixed inter-pulse interval for 160 $PPM$ as
calculated by equation 1 is 16539 samples.

(a)

(b)

(c)

Figure 4: Pattern Generator plots. (a) Pulse Generator Output; (b) Hi-Hat Drum
sample impulse signal; (c) Convolution result of (a) and (b).

The next stage of the pattern generator block adds tonal quality to the output
of the pulse generator. We achieve this by convolving the pulse generator
output with an appropriate drum sample impulse signal. The drum sample impulse
signal is a short recording of an onset played using a particular drum sound.
The drum sound selected must have an instantaneous attack and a short decay as
shown in figure 4(b).
The convolution process requires the loaded sample impulse to have the same
$fs$ as the pulse generator output. For a pulse generator output and drum
impulse, figure 4(c) shows the output of the convolution function. We make use
of the Discrete Fourier Transform convolution property [21]. This allows our
result to have the same number of samples as the pulse generator output. We
use the Fast Fourier Transform (FFT) [21] algorithm to achieve this result.
The resultant clave son rhythm from the pattern generator block, at 160 $PPM$
and 4 cycles, is an audio signal whose plot is shown in figure 4(c).

####  3.1.2 Complexity Metric

Various measures of musical rhythm complexity have been studied in [3]. Rhythm
complexity measures were divided into:
Rhythm syncopation measures that use metrical weights determined from the
metrical hierarchy to find syncopated onsets in a rhythm [19,22].
Pattern matching measures which chop up a rhythm as determined by the levels
of the metrical hierarchy and search each sub-rhythm for patterns that
indicate the complexity [23,24].
Distance measures, which measure how far away a rhythm is from a more simple
rhythm composed of beats [19,25,26].
Finally, mathematical measures which are based upon Shannon’s information
entropy [27,28]. Shannon’s information entropy uncertainty equations are given
in equations 4,5,6.
We propose to use the rhythmic complexity measure that calculates information
entropy in the sense of Shannon’s entropy for a particular rhythm pattern.
Vitz and Todd in [28], present a method that can be used to calculate the
complexity of cyclic binary patterns. Their work defined that the complexity
measure of a certain periodic or repeating pattern is the sum of the total
amount of uncertainty evaluated in processing the stimulus sequence. In [28],
they achieved this by proposing a set of rules (axioms) describing a
perceptual coding process. As the method implies an emphasis on coded
elements, it is referred to as a coded element processing system (CEPS) [3].
We consider this model for our application due to its algorithmic approach to
measure rhythm complexity, thus making it viable for coding, as against rhythm
syncopation models in [19,21]. CEPS also has an improved ability to
distinguish rhythm patterns within a cycle based on complexity as compared to
pattern matching [23,24] and distance measures [19,25,26].
The axioms in [28], proposed originally for tones and non-tones, break down
sequences into code levels. In our work, we extend it to rhythmic patterns,
composed of onsets and silences. Each code level successfully codes the
pattern from single elements to larger elements which continues until the
entire pattern is constructed. To calculate the complexity of the rhythmic
pattern, we measure two uncertainty values at each Code Level in the algorithm
as proposed in [28]. Uncertainty in this case means information entropy. The
uncertainty values are the maximum uncertainty, $H_{max}$, and the joint
uncertainty, $H_{joint}$ represented in equations 2 and 3 respectively.

$H_{max}(X,Y,Z,W)=H(X)+H(Y)+H(Z)+H(W)$ (2) $\displaystyle H_{joint}(X,Y,Z,W)=$
$\displaystyle H(X)+H(Y|X)+H(Z|X,Y)$ $\displaystyle+H(W|W,Y,Z)$ (3)

where,

$H(X)=-\sum_{x\in{X}}p(x)log_{2}p(x)$ (4)
$H(X,Y)=-\sum_{x\in{X}}\sum_{y\in{Y}}p(x)p(x,y)log_{2}p(x,y)$ (5)
$H(X|Y)=-\sum_{x\in{X}}\sum_{y\in{Y}}p(y,x)log_{2}p(y/x)$ (6)

Let the value $H^{k}$ be the sum of $H^{k}_{max}$ and $H^{k}_{joint}$ at Code
Level k. The total sum of H at each Code Level represents the complexity of
the pattern. This value is termed $H_{code}$.

$H_{code}=\sum_{k=1}^{n}H^{k}=\sum_{k=1}^{n}H_{max}^{k}+H_{joint}^{k}$ (7)

The complexity measure for $[1001001000101000]$ is calculated to be 20.5538.
Similarly, we were able to calculate complexity measures for all the
combinations of patterns within the cycle lengths of 4,3,5, and 7. Upon
calculating the complexity measures, we arrange patterns according to their
increasing order of complexity as shown in
$sheets$111{https://docs.google.com/spreadsheets/d/e/2PACX-1vTSDSqil264H6-M2kLdQUuD6VYnRqYPQePhKl8C_STD1j-54rFdhbLag-
ep_15ggiHpo8-DOpJWyQ-O/pubhtml}. We observed that for patterns that are a
shifted version of another, where 1’s (onsets) replace the 0’s (silences), and
vice versa, the complexity measures are identical. For example, $[1100]$ and
$[0011]$ have a complexity measure of 5.0. To tackle this, we considered those
patterns that begin with a 0 as off-beat patterns and assumed that they
present a higher challenge to learners. Hence, we incremented a constant value
to the complexity measures for all those patterns that begin with a 0
(silence). The constant value within a cycle was determined by the highest
complexity measure for the patterns that begin with 1 (onsets), for that cycle
length. This can be observed in $sheets^{1}$.
Furthermore, this complexity metric does not account for the complexity that
arises with changes in tempo. Therefore in this work, we only assess the
performances at a constant $PPM$.

###  3.2 Performance Assessment

Once the learner listens to the generated pattern their performance is
recorded and loaded onto the system. We rectify the recorded audio signal
$(z)$ with a threshold, $z_{threshold}=z_{max}/4$, to improve the results of
the onset detection function and it also accounts for low magnitude noise in
the live recording environment.

(a) Recorded input.

(b) Onsets from rectified inputs.

Figure 5: Plots of onset detection in one cycle.

Any onset detection function can be used to extract the onsets from the
rectified recorded input. We used the onset detection function defined by the
Librosa python library [29]. A sample of the recorded input and the extracted
onsets are shown in figure 5.

Inter onset intervals are calculated from onsets estimated from both recorded
($IOI_{input}$) and generated signals ($IOI_{generated}$).
The deviation: $IOI_{generated}-IOI_{input}$, for each inter onset interval is
calculated to be the error in temporal performance at each onset.

The error is recorded in a matrix format where the number of columns is
equivalent to the number of onsets in a cycle and the number of rows is
determined by the total number of cycles. For the son clave rhythm, there are
5 ‘1’s’ (onsets) and 11 ‘0’s’ (silence) played for 4 cycles, resulting in a
$4X5$ matrix as shown in 6.

Figure 6: $4X5$ Matrix of error deviation result.

From the error matrix, two sets of averages are calculated. One set of average
values represent the deviation of each inter onset interval within a cycle
(average of all the values in one column of the error matrix). This results in
a $5X1$ average onset error array. This average onset error array is plotted
on a graph with the horizontal axis representing each onset and the vertical
axis representing the percentage of deviation (-100 to 100) This is shown in
figure 7(left). This result indicates to the learner their timing error at
each onset. The negative error implies slowly playing speed and the positive
indicates fast playing speed.
The next set of average values represent the error deviation of a learner
across the cycles (average all the values in one row of the error matrix)
which results in a $4X1$ average result array. This result is plotted on a
graph with the horizontal axis representing the number of cycles and the
vertical axis representing the deviation (-100 to 100) shown in figure
7(right). This result gives the learner information on their consistency
throughout the performance and indicates if the timing error is consistent
across cycles.

Figure 7: Deviation graph results of the performance assessment block.

The graphs in figure 7 give the learner comprehensive performance feedback
concerning their timing and consistency. We achieve this through the graphical
representation wherein, a positive deviation (above 0 line) in figure 7 (left)
infers that the learner is rushing or playing the beat faster than intended
and a negative deviation (below 0 line) infers that the learner is dragging or
playing the beat slower than intended. Each onset in figure 7 (left) is marked
by an x.

##  4 1e0a - The Web Application

For improved accessibility to this system to aid in learning, we developed a
web application hosted at https://1e0a.noelalben.com

Our goal was to implement the proposed system with a simple user interface and
a server to collect learners’ data. We were able to achieve both by using
Flask a micro web framework written in Python [30]. The website was hosted on
AWS [31] with certbot SSL certification [32].

(a) Learning platform

(b) Performance Assessment results

Figure 8: ‘1e0a’ Web Application.

Every new learner must register with the website and provide consent to use
their registration information and performance assessment results for research
purposes. Post-registration the learner is given a login id and password. Upon
logging in to the system they enter the learning platform as shown in figure 8
(a). They must listen to the generated pattern and record their respective
performance. The ‘Analyze’ button conducts performance assessment on the
learner’s recording and takes them to the results as shown in figure 8 (b). If
they choose to progress in their learning by clicking ‘Learn Another’, the
system will generate the next pattern based on the complexity metrics defined
in section 3.1.2 and take the learner back to the learning platform. When they
log out, the most recent generated pattern is stored into the database and
they can continue where they left, upon logging in again.

4(a)

3(a)

5(a)

7(a)

4(b)

3(b)

5(b)

7(b)

4 (c)

3(c)

5(c)

7(c)

Figure 9: Graphical analysis of performance assessment for 6 learners
categorised by professional($\bullet$), amateur($\times$), and
beginner($\star$) on the 1e0a web application at each cycle. (a)Top Panel:
Total number of attempts taken by learners. (b) Center Panel: Absolute average
beat error in performance. (c) Bottom Panel: absolute average error over the
cycle in performance.

##  5 Experiments and Results

###  5.1 Experimental Setup

To assess and quantify the effectiveness of our rhythm learning system, each
learner on the 1e0a web application was required to perform and record 23
distinct patterns over 4 different cycle lengths of (4,3,5 and 7), at a
constant PPM of 160. The learners were asked to use a desktop/laptop to access
the web application and to maintain a constant interface to perform the
rhythmic patterns (clapping or a single percussive instrument). They were not
allowed to use accents in their performance nor take breaks while performing a
given pattern. Throughout the experiment, the application recorded the number
of attempts each learner takes to achieve performance assessment results
within the error bounds presented in section 3.2. Furthermore, we stored the
performance assessment results of the learner for each rhythmic pattern where
their results were within the error bounds. The results of 6 learners are
shown in fig 9. Each learner is categorized based on their years of experience
with music education: Professionals (>10 years), Amateur (>2 years), and
Beginners( 0-2 years).

###  5.2 Results

From the resulting graphs in figure 9, it can be observed that compared to an
amateur or a beginner, for any given pattern the professional requires less
number of attempts to achieve performance accuracy within the error bounds.
This is reflective of a learning system where beginners make more mistakes and
progressively improve in their performance as they spend more time on the
application. We also observe that the learners were able to correct their
mistakes and achieve performance accuracy within the error bounds after a
sufficient number of attempts. This indicates that the performance assessment
feedback of the system aids the learner in understanding and correcting their
performance to achieve accuracy. Furthermore, it can be observed that across
different patterns within the cycle there exists complexity because the number
of attempts taken by the learner increases with the progression of patterns
within a cycle. However, this does not directly reflect the difficulty that
learners have while performing or listening to the rhythm; but does accurately
reflect how beginners and amateurs struggle to recognize a rhythm’s structure
as the complexity increases.

##  6 Conclusion

In this article, we introduce ‘1e0a’, the proof of concept for a rhythm
learning and assessment system that helps learners progress in their learning
with a new pattern generated based on a pre-defined rhythm complexity measure.
We describe the rhythm complexity measure that was used and present the
methodology for rhythm performance assessment. The results of the proposed
system show that we can successfully test a learner’s proficiency on rhythm
patterns over a single PPM and monitor their progression within a cycle of
patterns of increasing complexity values. This system does not test the
learner’s proficiency across different tempos neither does it test for mixed
cycle complexity.
The future work is to implement a complexity measure that extends to various
tempos, accents and mixed cycle length progression. Furthermore, we aim to
have an improved onset detection and noise removal on the recorded audio
signal for a performance assessment that is robust to noise and disturbances.

## References

  * [1] Eremenko V, Morsi A, Narang J, Serra X. _Performance assessment technologies for the support of musical instrument learning. Paper presented at: CSEDU 2020 The 12th International Conference on Computer Supported Education; 629-640, Volume 1, 2020 May._
  * [2] J. London. Rhythm. _In L. Macy, editor, Grove Music Online. Oxford University Press, 2008. Accessed: 20 March 2008._
  * [3] Eric Thul _Measuring the Complexity of Musical Rhythm’, Master’s thesis, McGill University, Montreal, Quebec, 2008._
  * [4] Mantie, R. _The Philosophy of Assessment in Music Education. In The Oxford handbook of assessment policy and practice in music education, (32-55) Volume 1 . Oxford University Press 2019._
  * [5] Schneider, M. C., McDonel, J. S., DePascale, C. A. _Performance Assessment and Rubric Design. In The Oxford handbook of assessment policy and practice in music education. Volume 1, 2019._
  * [6] Wesolowski, B. C. _Understanding and Developing Rubrics for Music Performance Assessment. Music Educators Journal, 98(3), 36-42,2012._
  * [7] Dittmar, Christian, et al _Music information retrieval meets music education." Dagstuhl Follow-Ups. Vol. 3. Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik, 2012._
  * [8] Cano, Estefanía, Christian Dittmar, and Sascha Grollmisch. _"Songs2See: learn to play by playing." 12th International Society for Music Information Retrieval Conference (ISMIR 2011). Miami, 2011._
  * [9] Lerch, A., Arthur, C., Pati, A., & Gururani, S. _Music performance analysis: A survey. arXiv preprint arXiv:1907.00178. 2019._
  * [10] Wu, Chih-Wei, and Alexander Lerch. _"Learned features for the assessment of percussive music performances." 2018 IEEE 12th International Conference on Semantic Computing (ICSC). IEEE, 2018._
  * [11] Crazy Ootka Software AB. _Perfect Ear; version: 3.8.71 https://play.google.com/store/apps/details?id=com.evilduck.musiciankit, last accessed 14/05/2021._
  * [12] Demax. _Rhythm Trainer; version:0.2104261849 https://play.google.com/store/apps/details?id=ru.demax.rhythmerr, last accessed 14/05/2021._
  * [13] Stéphane Dupont. _Complete Rhythm Trainer; version:1.3.10 https://play.google.com/store/apps/details?id=com.binaryguilt.completerhythmtrainer, last accessed 14/05/2021._
  * [14] MakeMusic, Inc. _SmartMusic,April 2019. https://www.smartmusic.com, last accessed 05/14/2021._
  * [15] Yousician Oy. _Yousician, April 2019. https://www.yousician.com, last accessed 05/14/2021._
  * [16] The Way of H, Inc. (dba Music Prodigy). _Music Prodigy, April 2019. http://www.musicprodigy.com, last accessed 05/14/2021._
  * [17] Sony Interactive Entertainment Europe. _SingStar, April 2019. http://www.singstar.com, last accessed 05/14/2021._
  * [18] G. W. Cooper and L. B. Meyer. _The Rhythmic Structure of Music. The University of Chicago Press, 1960._
  * [19] G. T. Toussaint. _A mathematical analysis of African,Brazilian, and Cuban clave rhythms. In BRIDGES: Mathematical Connections in Art, Music and Science, pages 157-168, Jul 2002._
  * [20] Janssen, Brett Allen. _A preliminary comparative study of rhythm systems employed within the first-year college aural skills class. Diss. Kansas State University, 2017._
  * [21] Proakis, John G., and Dimitris G. Manolakis. _"Digital signal processing." PHI Publication: New Delhi, India (2004)._
  * [22] H. Longuet-Higgins and C. Lee. _The rhythmic interpretation of monophonic music. Music Perception, 1(4):424-441, 1984._
  * [23] J. Pressing. _Cognitive complexity and the structure of musical patterns. http://www. psych.unimelb.edu.au/staff/jp/cog-music.pdf, 1999._
  * [24] M. Keith. From Polychords to Póya: Adventures in Musical Combinatorics. Vinculum Press, 1991.
  * [25] G. T. Toussaint. _Classification of phylogenetic analysis of African ternery rhythm timelines. In BRIDGES: Mathematical Connections in Art, Music and Science, pages 23-27, Jul 2003._
  * [26] F. Gómez, A. Melvin, D. Rappaport, and G. Toussaint. _Mathematical measures of syncopation. In BRIDGES: Mathematical Connections in Art, Music and Science, pages 73-84, Jul 2005._
  * [27] P. C. Vitz. _Information, run structure and binary pattern complexity. Perception and Psychophysics, 3(4A):275-280, 1968._
  * [28] P. C. Vitz and T. C. Todd. _A coded element model of the perceptual processing of sequential stimuli. Psycological Review, 75(6):433-449, Sep 1969._
  * [29] McFee, Brian, et al. _"librosa: Audio and music signal analysis in python." Proceedings of the 14th python in science conference. Vol. 8. 2015._
  * [30] Grinberg, Miguel. _Flask web development: developing web applications with python. " O’Reilly Media, Inc.", 2018._
  * [31] Amazon, E. C. _"Amazon web services." Available in: http://aws.amazon.com/es/ec2/(November 2012) 2015._
  * [32] Tiefenau, Christian, et al. _"A Usability Evaluation of Let’s Encrypt and Certbot: Usable Security Done Right." Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security. 2019._


# Can you trust your explanations? A robustness test for feature attribution
methods

Ilaria Vascotto Corresponding Author. Email<EMAIL_ADDRESS>Alex
Rodriguez  Alessandro Bonaita  Luca Bortolussi  Department of Mathematics,
Informatics and Geosciences, University of Trieste  The Abdus Salam
International Center for Theoretical Physics (ICTP)  Assicurazioni Generali
Spa

###### Abstract

The increase of legislative concerns towards the usage of Artificial
Intelligence (AI) has recently led to a series of regulations striving for a
more transparent, trustworthy and accountable AI. Along with these proposals,
the field of Explainable AI (XAI) has seen a rapid growth but the usage of its
techniques has at times led to unexpected results. The robustness of the
approaches is, in fact, a key property often overlooked: it is necessary to
evaluate the stability of an explanation (to random and adversarial
perturbations) to ensure that the results are trustable. To this end, we
propose a test to evaluate the robustness to non-adversarial perturbations and
an ensemble approach to analyse more in depth the robustness of XAI methods
applied to neural networks and tabular datasets. We will show how leveraging
manifold hypothesis and ensemble approaches can be beneficial to an in-depth
analysis of the robustness.

905

##  1 Introduction

The recent popularity of Artificial Intelligence (AI) models in public and
industrial domains has brought up questions on their accountability and
trustworthiness. Ethical and moral concerns have been raised on their
fairness, especially when considering high-risk scenarios, such as self-
driving cars or medical applications. While the AI field has rapidly expanded,
the legal systems have struggled to keep up with the novelties in the field.
Recent proposals of the European Union, such as the AI Act [7] and sections of
the General Data Protection Regulation (GDPR) [6], aim at introducing a first
set of guidelines to be followed in order to regulate the current and future
state of AI systems and encourage a fairer and more transparent approach to
AI. Similarly, the United States of America have proposed the new Blueprint
for an AI bill of rights [19], a conceptually similar set of guidelines not
enforceable by law that aim at guiding a fair development and deployment of AI
systems.

A central point in both the AI Act and the Blueprint for an AI bill of rights
is the transparency of the systems. This is broadly described as the ability
of providing a detailed description in plain language of the model reasoning,
which holds true from a technical standpoint but is also meaningful and useful
to system’s operators or individuals that need to interpret a given outcome.
While it is unclear how to attain the transparency mentioned in such
regulations, as it is hardly translatable into a mathematical concept, the AI
field has concurrently developed tools under the hood of Explainable
Artificial Intelligence (XAI). This field of research focuses on shedding
light on the inner mechanisms of complex machine learning models in the form
of explanations. The usage of such techniques can be beneficial to both
practitioners and end-users. For data scientists, being able to understand in
a clear and concise manner the reasoning behind a model’s outcome can lead to
a deeper understanding of the model itself but also allows the identification
of biases or faulty patterns in the model or even in the dataset. On the other
hand, end-users of AI models, or individuals who may be directly affected by a
model’s output, can benefit from the explanations as they allow for a
problem’s understanding without the need of technical details from a
mathematical perspective.

A key aspect in AI is the robustness of its systems, recognized also in the AI
Act as a desiderata. This property must be, nonetheless, reflected also on the
XAI techniques which are applied to a model’s output. We define robustness as
the ability of an explanator (i.e. any explainability technique) to produce
similar explanations for similar inputs. It is crucial to ensure that the
explanations are robust to perturbations (random or adversarial) of the data
manifold. Only in this scenario, it will be possible to ensure fairness and
increase the trustworthiness of systems.

Let us assume that a user is presented with two similar instances for which
the outcome of a black box model is the same. If the two explanations are
significantly different, the lack of robustness may influence the
trustworthiness of the whole system, as one may question both the explanations
and the predictions.

Our contribution focuses on neural networks for classification and tabular
data. We propose an ensemble approach that leverages information from multiple
XAI approaches and a test to validate the robustness of the produced
explanations, expressed in the form of a ranking of feature importances. The
robustness test is based on the generation of a neighborhood (artificial
instances generated by perturbation of the original ones) that is sampled
within the data manifold and is hence more faithful to the feature space.

Our work is structured as follows: section 2 presents related work on the
robustness of XAI techniques. The following, section 3, presents relevant
terminology and a brief introduction to three XAI approaches for neural
networks we have used for the ensemble. In sections 4 and 5, we will proceed
in detailing our approach and show results on both toy and realistic datasets,
before concluding with a discussion on future developments of this work.

##  2 Related work

Available literature on the topic uses almost interchangeably the terms
robustness and stability when discussing the robustness of explanations as
presented in the Introduction.

A known example in literature of explanation’s instability is the LIME method
developed by Ribeiro et al. [13]. LIME proposes explanations in the form of
linear model’s coefficients, fitted on a neighbourhood of the selected
datapoint. The neighbourhood is sampled from a fixed data distribution, but
the generated datapoints are different at each call of the method. This
translates into different linear models being fitted at each time the method
is applied on a given datapoint, and consequently their coefficients (and
proposed explanations) vary even if the same datapoint is being explained.

Attempts at improving the technique have been proposed in DLIME [22], which
replaces the random generation with a hierarchical clustering step. Visani et
al. [20] proposed OptiLIME, a modified version of LIME which minimises the
instability, with respect to two stability indices defined by the same authors
in [21].

Rosenfeld [14] proposed a metric to evaluate the stability of an explainer’s
output. They suggest the stability test should be performed using
perturbations that don’t change the class label and introduce small amounts of
resampling noise to ensure the stability of the explanations.

Alvarez-Melis and Jaakkola [2] proposed a formalization of local robustness
based on the definition of local Lipschitz continuity. They test their
proposal on images’ explanations and show that gradient-based approaches are
much more robust than their perturbation-based counterparts. In [3] the
authors propose self-explaining neural networks, a class of NNs for which
faithfulness and robustness are enforced via specific regularization.

Ghorbani et al. [8] focuses on images’ explanations and their relationship to
adversarial attacks. In particular, they consider adversarial perturbations to
the neural network’s interpretation: they are defined as small perturbations
that change the model’s explanation while retaining the same prediction
between the two instances. This is in contrast with adversarial attacks to the
net itself, where a small perturbation changes the net’s output.

In [4] the authors note that it is possible, for any classifier $g$, to
construct a classifier $\hat{g}$ with the same behavior on the data, but with
manipulated explanations. To this end, they propose a robust version of the
presented gradient-based explanations.

Gosiewska and Biecek [9] underlines how model-agnostic additive methods, such
as [10, 13], fail at detecting feature interactions, as they assume feature
independence in their theoretical settings.

Nauta et al. [11] presents an in-depth survey on the evaluation of XAI
techniques and highlights different metrics that can be used to assess the
explanation robustness. Similarity can be evaluated with metrics such as: rank
order correlations, top-$k$ intersections, rule matching and structural
similarity index.

##  3 Background

###  3.1 Terminology

We briefly introduce in this section some of the terminology linked to the
field of Explainable AI that will be useful for the understanding of this
article’s proposal. While there is not a common ground on the theory of the
field yet, we agree upon the following taxonomy of XAI approaches [1], which
spans over three axis:

  * •

Scope of the explanation: a local approach focuses on the explanation of a
single (input, output) pair while a global one aims at explaining the whole
model’s reasoning.

  * •

Model of interest: model-specific techniques leverage the structure of a
specific model which is being analysed, while model-agnostic ones can be
applied to any model’s output.

  * •

Transparency: intrinsically transparent models are interpretable by
construction (glass boxes) while post-hoc techniques are applied to already
trained models (black boxes).

As our model of interest are neural networks, we can further distinguish
between perturbation-based and gradient-based approaches [15]. Perturbation-
based methods are often model-agnostic and rely either on neighbourhood
generation or combinatorial aspects, such as LIME proposed by Ribeiro et al.
[13] or SHAP from the work of Lundberg and Lee [10]. Gradient-based methods,
on the other hand, are specific to neural networks and take advantage of their
inner structure, mainly exploiting the backpropagation mechanisms [5, 16, 18].

Feature attribution techniques may be called in different manners according to
the type of data being explained. Most commonly, an image would be explained
feature-wise through heatmaps highlighting the importance of each pixel, while
for text the explanation would either be a word or a bag of words. In the
context of tabular data, feature importance methods return a vector of feature
importances, in which each coefficient represents the relative importance of
the corresponding feature. A positive (negative) sign usually represents a
positive (negative) contribution towards the outcome. In this context, we may
also refer to the feature importance vectors as attributions.

###  3.2 Considered techniques

We focus on local post-hoc approaches, specific for neural network’s
explanations. We consider XAI gradient-based techniques applicable to nets
trained on tabular datasets. More details on the techniques can be found on
the cited papers in which the methods were originally proposed.

Note that neural networks explanations require the selection of an output
neuron to be explained: this is necessary especially in classification
scenarios, in which one may want to investigate the features that have
contributed to any of the classes probability scores. In the following, we
always select as target neuron the one associated with the model’s prediction,
that is, the one with a higher output score.

####  3.2.1 DeepLIFT

DeepLIFT (DeepLearning Important FeaTures) [16] computes feature contributions
with respect to their difference from a given reference. In particular, the
differences between the two outputs are explained in terms of the difference
among the two inputs. Each neuron is analysed with respect to the difference
between its activation and that of the reference input. DeepLIFT makes use of
contribution scores and multipliers to backpropagate the difference in output
through the network. It requires a single forward-backward pass through the
net, rendering it efficient. The propagations are computed through appropriate
chain rules, defined according to the neuron’s type and its activation.

####  3.2.2 Integrated Gradients

Integrated Gradients (IG) [18] satisfies the axioms of sensitivity and
implementation variance. It computes the integral of the gradients along the
straight-line path from a baseline $x^{\prime}$ to the input $x$, considering
a series of instances linearly separated along the path from the baseline to
the point of interest. In practice, it takes advantage of an approximation of
$m$ steps such that for the $i$-th dimension it holds:

$IG_{i}^{\text{approx}}(x)=\cfrac{x_{i}-x^{\prime}_{i}}{m}\
\sum_{k=1}^{m}\cfrac{\partial F(x^{\prime}+(x-x^{\prime})\cdot k/m)}{\partial
x_{i}}$

The authors of [18] found that $m\in(20,300)$ produced satisfactory
approximations but the computation can nonetheless be expensive when the
number of steps is large.

####  3.2.3 Layerwise Relevance Propagation

Layerwise Relevance Propagation (LRP) [5] is based on the backpropagation
principle. It defines a series of rules to propagate the output score (or
relevance) $f(x)$ through the net’s layers, depending on the architecture at
hand. The conservation property holds: with $R_{j}$ the relevance for neuron
$j$, for each pair of layers $\sum_{j}R_{j}=\sum_{k}R_{k}$ and globally
summing over all layers $\sum_{i}R_{i}=f(x)$. Two common propagation rules are
the epsilon and the gamma rules.

$\begin{split}\text{LRP-}{\epsilon}:&R_{j}=\sum_{k}\frac{a_{j}\cdot
w_{jk}}{\epsilon+\sum_{0,j}a_{j}\cdot w_{jk}}R_{k}\\\
\text{LRP-}{\gamma}:&R_{j}=\sum_{k}\frac{a_{j}\cdot(w_{jk}+\gamma\cdot
w_{jk}^{+})}{\sum_{0,j}a_{j}\cdot(w_{jk}+\gamma\cdot
w_{jk}^{+})}R_{k}\end{split}$

where $a_{j}$ is the activation at neuron $j$, $w_{jk}$ is the weight linking
neuron $j$ to neuron $k$ in the following layer ($w_{jk}^{+}$ is a positive
weight), $\sum_{0,j}$ indicates a sum over all lower-layer activations.

##  4 Method

We propose a local robustness test for neural network-based explainability
techniques. We begin this section by introducing an ensemble approach that
merges explanations derived from the previously described methods. The
ensemble is able to better deal with cases in which two methods show a degree
of disagreement over the importance of a feature: by proposing a more robust
explanation, derived in terms of a features’ ranking, we can grasp the overall
importance of a feature comparing multiple perspectives. Following this
proposal, we investigated the role of locality with respect to the robustness.
We first tested whether the methods, including our ensemble, were robust
within a properly constructed neighbourhood. Moreover, we aimed at verifying
whether the robustness of a datapoint’s explanation shows coherent patterns
across its neighbouring points: if that was the case, we could introduce a
local predictor to test whether a new point was robust. In subsection 4.6 we
will present a pipeline that allows us to evaluate whether the explanation is,
in fact, robust or not. Our final aim is that of proposing both a metric to
compute the robustness of a feature attribution ranking’s importance, and
enrich it with a broader trustworthiness analysis. The usage of our pipeline
is intended to aid practitioners in understanding whether an explanation can
be trusted (that is, it is stable within a small neighbourhood) or if the
considered area of the data manifold is linked to unreliable explanations (and
typically unreliable predictions).

Our approach is applied to classification problems and tabular datasets. Let
us introduce the following notation: let
$\mathbf{X}=(x^{1},\dots,x^{i},\dots,x^{n})$ be a dataset of size $n$ with $m$
features such that $x^{i}=(x_{j}^{i})$ with $j\in(1,\dots,m)$ and let each
point be associated to a class label $y^{i}$.

###  4.1 Ensemble

The proposed aggregation focuses on the rankings of the features in terms of
their absolute importance. To this end, we do not discriminate between
positive and negative features but consider them according to their absolute
value. We have derived a weighting schema to merge the different feature
attributions’ rankings. Let the set of weights $\mathbf{w}_{l}$ for the $l$-th
method be such that for $i$-th point and $j$-th feature it holds:

$w_{l}^{ij}=\cfrac{\sigma(\mathbf{a}_{l}^{i})}{\sqrt{|\overline{\mathbf{a}_{l}^{i}}|\cdot|a_{l}^{ij}|}}$

where $\mathbf{a}_{l}^{i}=(a_{l}^{i1},\dots,a_{l}^{ij},\dots,a_{l}^{im})$ is
the feature importance vector and $\overline{\mathbf{a}_{l}^{i}}$,
$\sigma(\mathbf{a}_{l}^{i})$ are its average and standard deviation
respectively. For each of the $L=3$ methods, we define the ranking
$\mathbf{r}_{l}^{i}$ as the vector of size $m$ of the indices that would sort
the array $|\mathbf{a}_{l}^{i}|$ in decreasing order. Let an average
attribution be:

$\hat{a}^{ij}=\cfrac{\sum_{l=1}^{L}r_{l}^{ij}\cdot
w_{l}^{ij}}{\sum_{l=1}^{L}w_{l}^{ij}}\cdot(1+\lambda\cdot(L-n_{S}))$

where $\lambda=0.15$ is a penalization term and $n_{S}$ is the number of
methods for which there is concordance on the sign of the attribution
$a_{l}^{ij}$ (for $l=1,\dots,3$). The ranking associated with this average
attribution, defined as the vector of size $m$ that would sort the array in
increasing order, is the result of our ensemble approach.

The defined set of weights was proposed to deal with some of the issues we
encountered. First of all, the attributions may produce values very close, but
not equal, to zero. It is not often clear when a low value of the attribution
is associated with a non influential feature. We introduced in our weighting
schema the absolute value of the reciprocal of the attribution to penalize
attribution scores too close to zero: the weight in this case would be a large
value, pushing the feature further down in our ranking (i.e. moving the
feature into the set of non-influential ones). The remaining part of the
proposed weights was guided by the need of a normalization among different
instances, as the coefficients can significantly vary in magnitude. The
standard deviation was used as normalization factor and we empirically
verified that the introduction of the average produced better ensembles.

The ensemble is computed over the weighted average of the XAI methods’
rankings and a penalization term is introduce to penalize features for which
the different XAI methods exhibit discordance in the sign of a feature’s
importance. The penalization is aimed at favouring those features for which
the three methods agree over the sign (positive or negative) and when the
attributions’ magnitude is non neglibigle.

Let us recall that this ranking considers the overall importance of a feature,
without explicitly discriminating between positive and negative influences. A
sign can nonetheless be selected depending on the most frequent sign recorded
among the $L$ original attributions for the given feature. It is moreover
important to remember that, while for XAI attributions it usually holds that a
larger value of the attribution is associated with a larger influence in the
model’s prediction, our average attribution favours smaller values: this is
driven by the fact that our interest lies in the ranking and not the
aggregated coefficients.

We have computed the ensemble based on the attributions derived from the
methods Integrated Gradients, DeepLIFT and LRP, but it can be easily extended
to include a higher number of feature attribution methods.

###  4.2 Neighbourhood generation

Figure 1: Neighbourhood generation example. a) A visualization of the swiss
roll dataset, with $n=5000$ points. In blue, the selected datapoint for which
we want to compute a neighbourhood. In red, its 5 closest medoids. b) A
neighbourhood of size $n=500$. c) The same neighbourhood visualized from above
the data structure: the perturbations are still on manifold.

A crucial step in the local robustness analysis is the data point’s
neighbourhood generation. The way neighbourhoods are crafted can be extremely
influential in the robustness score computed as seen in the following
subsection. We are interested in investigating the robustness with respect to
random perturbations, taking into account the data manifold. We propose the
following neighbourhood generation schema that is built upon the data
structure and produces perturbations which are still on-manifold and therefore
more suited for a truthful robustness analysis.

We first perform $k$-medoids clustering on the training set, selecting
$k_{medoids}$ to be such that, on average, each cluster is of size $n_{k}=10$.
This is derived from the intuition that a manifold may present a thickness
layer: we would expect random perturbations to be able to explore the data
manifold, while this hypothesis finds the perturbations not able to escape the
thickness layer. We have empirically tested this intuition and determined
$n_{k}=10$ was a sufficient size to escape the locality for the perturbation
step.

For each cluster, the medoid $x^{c}$ acts as a representative and its $k_{M}$
nearest neighbours among the other cluster centers
$NN^{c}=(x^{1},\dots,x^{k_{M}})$ are retrieved. For each point $x^{i}$ we want
to test, we select the associated cluster, indexed by $c$. From the cluster
center’s neighbours $NN^{c}$, we randomly select one of the medoids, say
$x^{M}$.

Having $\alpha$ and $\alpha_{cat}$ the probabilities of perturbing a numerical
and categorical variable $x_{num}^{i}$ and $x_{cat}^{i}$ respectively, the
perturbed point $x^{P}$ is generated as:

$\begin{cases}x_{num}^{P}=(1-\bar{\alpha})\cdot x_{num}^{i}+\bar{\alpha}\cdot
x_{num}^{M}\\\ \text{with $\bar{\alpha}$ drawn from $\textit{Beta}(\alpha\cdot
100,(1-\alpha)\cdot 100)$ }\\\ \\\ x_{cat}^{P}=\begin{cases}x_{cat}^{i}\text{
with probability }1-\alpha_{cat}\\\ x_{cat}^{M}\text{ with probability
}\alpha_{cat}\\\ \end{cases}\end{cases}$

The generated neighbourhood should be at least of size $n=100$ to ensure
statistical significance. A final filtering step is performed on the generated
data points: only those for which the model’s prediction is the same as that
of the original datapoint $x^{i}$ are kept as valid. Hyperparameter tuning is
performed for $\alpha$, $\alpha_{cat}$ and $k_{M}$ in this step to ensure
that, on average, 97% of the generated points are kept.

###  4.3 Categorical variables

The neighbourhood is fed through the network and its attributions are computed
for all $L=3$ methods. A preprocessing step is performed to deal with
categorical variables, when present. As the network takes as input a one-hot
encoding of the categorical variables, the corresponding attributions are
filled with null values. The methods we are considering satisfy in fact the
so-called missingness property: when the input is null, the corresponding
attribution is a zero coefficient. We can therefore perform a reverse encoding
of the categorical variables: as the only non-null coefficients for a one-hot
encoding are the ones corresponding to the class present in the feature
vector, we can associate its value with the importance of the categorical
variable directly and not only with the specific class. This step allows us to
better grasp the influence of the feature and perform an effective comparison
in terms of robustness, as it reduces the dimensionality of the feature
attribution vector. The reverse encoded feature importance vectors of the $L$
methods are used to derive the ensemble’s ranking, as in 4.1.

###  4.4 Robustness computation

We have selected Spearman’s rank correlation coefficient $\rho$ as the metric
to evaluate the robustness. The metric is computed between the original data
point’s attribution vector and that of a slightly perturbed input. The average
value of the metrics between the original data point and its generated
neighbourhood is the robustness score. Values closer to one indicate higher
rank correlation (and hence, robustness within the neighbourhood) while
smaller values highlight a larger number of differences between the points’
explanations.

As the metric assumes values between 0 and 1, we can define a robustness
threshold to discriminate between robust and non robust points. The selection
of the threshold value is dataset dependant but a default value is set at
$th=0.80$. This worked well across most datasets (see subsection 5) and was
selected considering how the percentage of non-robust points varied over
possible values of the threshold. In particular, the threshold is associated
with the flex point of such curve.

Note that the robustness score can be computed not only for the ensemble, but
also for the $L$ individual attribution approaches, as it only requires the
calculation over the feature importance vector, independently of how they were
derived. For an efficient and truthful comparison, when considering also the
ensemble’s robustness, the robustness for the other methods was computed over
the absolute value of their attributions. This ensures comparability between
the two derived rankings.

###  4.5 Can we trust the explanation?

Evaluating previously unseen datapoints in terms of the trustability of their
explanations is not a trivial task. It is not sufficient to compute the
robustness score for the new datapoint, as it may lay in an unstable area of
the feature space. To tackle this issue, we introduce a $k$-nearest neighbours
regressor fitted on the training set’s robustness scores. For new datapoints,
we compute both the robustness score and the regressor’s prediction. According
to the dataset’s selected threshold of robustness, we can distinguish three
cases:

  * •

when both the robustness score and the regressor’s predicted value are above
the threshold, we can deem the point to be robust.

  * •

when the robustness score is above the threshold but the regressor’s predicted
value is below it, the point lies in an uncertain area of the robustness space
and therefore its robustness must not be presumed.

  * •

when the robustness score is below the threshold, we can consider the point to
be non robust.

The second case highlights a set of conditions under which the robustness is
less apparent. In this case, the choice of trusting the explanation is left to
the practitioner, which is now presented with a broader set of information.

The usage of a $k$nn regressor instead of a $k$nn classifier renders it less
susceptible to the selected threshold: it allows us to consider multiple
thresholds, if required, without the need for a re-computation. The number of
neighbours is selected depending on the goodness of approximation of the
regressor to the robustness scores.

###  4.6 Pipeline

With the considerations presented up to this point, we can provide a complete
pipeline to be followed in order to test the robustness of feature
attributions methods.

  1. 1.

Train a neural network.

  2. 2.

Perform $k$-medoids clustering on the training set and compute the $k_{M}$
nearest neighbours among the medoids.

  3. 3.

Compute the attributions of Integrated Gradients, DeepLIFT and LRP on the
training set, and merge them to retrieve the ensemble’s ranking.

  4. 4.

Compute the robustness score of the training set by first generating a
neighbourhood and comparing the ensemble’s attributions with Spearman’s rank
correlation, averaging over the neighbourhood.

  5. 5.

Use these robustness scores to train a $k$-nearest neighbours regressor.

  6. 6.

On previously unseen datapoints, first compute the $L$ individual
attributions, the respective ensemble and proceed with the generation of a
neighbourhood. Note that in this case, it is first necessary to predict to
which $k$-medoids cluster the new point is assigned to, and then generate the
neighbourhood.

  7. 7.

Compute the robustness score as above and predict the robustness with the
$k$nn regressor.

  8. 8.

Analyse the results as described in subsection 4.5.

###  4.7 Validation

The assessment of quality of our proposal is subject to the lack of a ground
truth for the robustness. We rely on the following intuition to test whether
the robustness holds. We argue that the robustness is related both to the
model under consideration and the specific problem at hand. Our assumption is
that a robust point will be correctly classified also if passed through a
different net, while harder-to-predict points will exhibit more differences
among the models’ predictions and respective explanations.

We have trained three neural networks per dataset, all retaining similar
accuracy, but with different architectures (either varying the number of
hidden layers or the number of neurons per layer). We have investigated our
assumption in relation to the agreement of the three model’s predictions and
their correctness.

##  5 Results

###  5.1 Datasets

We have selected the following datasets from the UC Irvine Machine Learning
Repository: beans, cancer, mushroom, wine, adult and bank marketing. We have
selected two additional datasets, namely heloc and ocean, following the work
of [12] and [17] respectively.

The first four datasets act as toy examples as they represent simpler
classification tasks and have a lower number of both features and datapoints.
The remaining four instead show more constructed examples as the number of
features and datapoints increases. All the datasets propose either a binary or
multiclass classification tasks, and present both numerical and categorical
variables.

General information on the datasets is presented in Table 1, which includes
also information on the hyperparameters set for the experiments.

We have relied on the Python libraries pytorch and captum for the
implementation of our approach. The former was used for the training and
deployment of the neural networks while the latter, a pytorch-compatible
explainability framework developed by Meta, was applied to derive the
attributions of Integrated Gradients, DeepLIFT and LRP.

Table 1: Datasets description and hyperparameters. Dataset | Classes |  | Numerical
---
variables
| Categorical
---
variables
Train size |  | Validation
---
size
Test size | $k_{medoids}$ | $\alpha$ | $\alpha_{cat}$ | $k_{M}$ | $th$ | $k_{R}$
beans | 7 | 7 | 0 | 10888 | 2233 | 500 | 225 | 0.10 | - | 10 | 0.85 | 9
cancer | 2 | 15 | 0 | 397 | 121 | 50 | 10 | 0.10 | - | 4 | 0.85 | 11
mushroom | 2 | 0 | 21 | 6498 | 1225 | 400 | 120 | - | 0.15 | 10 | 0.80 | 11
wine | 7 | 8 | 1 | 5197 | 1000 | 300 | 100 | 0.10 | 0.05 | 10 | 0.85 | 11
adult | 2 | 5 | 7 | 36177 | 8045 | 1000 | 1000 | 0.05 | 0.05 | 5 | 0.85 | 7
bank | 2 | 5 | 9 | 36138 | 8043 | 1000 | 1000 | 0.05 | 0.10 | 5 | 0.85 | 11
heloc | 2 | 14 | 2 | 8367 | 1592 | 500 | 150 | 0.05 | 0.05 | 5 | 0.80 | 15
ocean | 6 | 8 | 0 | 109259 | 30328 | 10000 | 3000 | 0.05 | - | 5 | 0.85 | 5
Figure 2: Distribution of robustness scores for Integrated Gradients,
DeepLIFT, LRP and the ensemble on the validation (left) and test (right) sets
of the adult dataset.

###  5.2 Discussion of experiments

For all datasets, we have taken into account the following preprocessing
steps:

  * •

Removal of correlated features. The correlation was computed with Spearman’s
rank correlation coefficient for the numerical variables and normalized mutual
information criterion for the categorical ones. The removal ensures that the
neighbourhood generation is realistic with respect to the feature space and
that the computed attributions are reliable and not distorted by features
correlation.

  * •

Softmax in the final layer of the net for binary classification. It ensures
stability of attribution’s computations, as higher relevance scores are
backpropagated through the net and it reduces the vanishing gradient problem
(cases in which the attribution is returned as a zero-vector, as the signal is
lost during the backward pass through the net).

  * •

Selection of the gamma rule for LRP’s attributions, as it was the rule
minimising the vanishing gradient problem for the method on the tested
datasets. We have applied the same rule to all layers of the networks as they
were all linear layers.

  * •

We have divided the datasets into a train, validation and test sets. The train
set was used for the neural network’s training. The validation set acts as the
test set on which we have tested the accuracy of the net, computed the
$k$-medoids clusters, the robustness scores and fitted the $k$-nearest
neighbours regressor. Finally, the test set acts as a collection of previously
unseen datapoints on which we want to perform a blind test.

The three trained networks per dataset, say Model 1, 2 and 3, show comparable
accuracy and tend to classify more accurately the easier datasets, as the
trivial nature of the examples is easily overcome by powerful predictive tools
such as neural networks. The remaining datasets include a higher unbalance
ratio between the output classes and have a higher classification error. Model
1 represents our baseline, model 2 has more layers and the initial ones have
more neurons while model 3 is a smaller version with two hidden layers and a
lower number of neurons overall. The activation functions are the same in all
three cases, being ReLU for all datasets except for the ocean dataset, for
which we followed the structure of [17] and used tanh activations.

As per the pipeline detailed in Section 4, we have computed the robustness
score on the validation and test sets with neighbourhoods of size $n=100$.
Hyperparameters $\alpha$, $\alpha_{cat}$ and $k_{M}$ were set through a grid
search (Table 1) such that the neighbourhood retains on average 97% of the
generated datapoints. Our experiments have shown that default value of
$\alpha=0.05$, $\alpha_{cat}=0.05$ and $k_{M}=5$ are a good initial choice for
the following analysis but dataset-specific hyperparameter selection is
nonetheless required. A more restrictive choice of the hyperparameters will
result in a neighbourhood too close to the original datapoint and the
robustness scores will consequently be very high. Likewise, a relaxed choice
of the hyperparameters (in particular those guiding the probability of
perturbation) would generate datapoints further away from the original one,
rendering the comparisons less accurate and overall meaningless.

Figure 2 shows the distribution of the robustness score on both the validation
and test sets of the adult dataset. We computed the individual methods’ and
the ensemble’s robustness scores, taking into account the absolute value of
the attributions to actively compare these results to the ensemble’s. We can
appreciate how the ensemble’s robustness (the red curve) is influenced by all
the methods, proposing on average behaviour among the three. In particular, on
this dataset the Integrated Gradients robustness score is on average lower
than that of the counterparts, and influences the lower values of the
ensemble’s score.

Table 2: Comparison of the concordance between the robustness score and the regressor, with $th=0.80$ for model 1. Dataset | Robust | Uncertain | Non robust
---|---|---|---
adult | 74.6% | 7.2% | 18.2%
beans | 62.0% | 12.0% | 26.0%
mushroom | 56.3% | 12.8% | 31.0%
wine | 73.0% | 5.7% | 21.3%
adult | 78.1% | 5.7% | 16.2%
bank | 92.9% | 0.5% | 6.6%
heloc | 62.8% | 9.2% | 28.0%
ocean | 79.2% | 5.6% | 15.3%
Table 3: Robustness results compared to nets’ predictions with $th=0.80$ for model 1. | Robust | Non robust
---|---|---
Dataset |  | Concordant
---
Correct
Discordant |  | Concordant
---
Incorrect
N. points |  | Concordant
---
Correct
Discordant |  | Concordant
---
Incorrect
N. points
beans | 88.0% | 4.6% | 7.3% | 409 | 85.7% | 4.4% | 9.9% | 91
cancer | 91.9% | 8.1% | 0.0% | 37 | 84.6% | 7.7% | 7.7% | 13
mushroom | 100.0% | 0.0% | 0.0% | 276 | 100.0% | 0.0% | 0.0% | 124
wine | 52.2% | 10.2% | 37.3% | 236 | 53.1% | 7.8% | 39.1% | 64
adult | 82.6% | 6.6% | 10.9% | 838 | 57.4% | 28.4% | 14.2% | 162
bank | 90.1% | 4.6% | 5.2% | 934 | 63.6% | 19.7% | 16.7% | 66
heloc | 64.7% | 17.8% | 17.5% | 360 | 40.0% | 43.6% | 16.4% | 140
ocean | 76.6% | 14.8% | 8.6% | 8475 | 66.9% | 19.7% | 13.4% | 1525

In Table 2 we present the results of our robustness analysis for the derived
ensemble, on all datasets. For the sake of comparison, we have selected the
threshold $th=0.80$ to be equal for all applications and we have used a
structurally-similar model, Model 1, trained on the various datasets.
Nonetheless, the number of neighbours used to train the $k$nn regressor on the
validation set is dataset-specific and is detailed in table 1. In all cases,
the robust points (that is, points for which the explanation is robust within
a neighbourhood), are the majority, but the percentage of non-robust ones is
non negligible. The non-robust datapoints may lie in an uncertain area either
from the point of view of the robustness or of the models’ correctness with
respect to the classification.

Our interest lies in the percentages of uncertain items, for which there is
discordance between the computed robustness score and the regressor’s
prediction. With $th=0.80$, on average $7.3\%$ of the test’s set points are
classified as uncertain, while the average increases to $13.2\%$ when the
selected threshold is set at $th=0.85$. We argue that these cases are the most
sensible ones and must be taken under consideration carefully when presented
in a practical application. Practitioners should be aware of the possible
instability of the explanations and analyse in depth the results.

As mentioned in subsection 4.7, the validation of our results is limited by
the absence of a reference in terms of robustness. In table 3, we propose the
results of the analysis of the robustness with model 1, taking into account
the concordance and correctness of the three models’ predictions. The
robustness label refers to the robustness score computed with model 1 being
greater or equal to the threshold $th=0.80$ on all datasets. The points are
then classified depending on whether the three models’ predictions are equal
(concordant) or not (discordant). In the former case, we further distinguish
the cases in which the prediction is correct or not.

As per our hypothesis, we can see that the relative percentage of discordant
points is higher on average when considering non robust datapoints. This is
more evident in the non trivial datasets, when the ratio between non-robust
and robust discordant points is greater than 2. This trend still persists when
the selected robustness threshold is set at $th=0.85$. Note that, the mushroom
dataset only presents datapoints classified as concordant and correct: this is
due to the fact that all three nets were able to achieve 100% accuracy within
the first ten epochs of training. This is linked to the fact that the dataset
is quite trivial and easily classified even by decision trees as well. We have
proposed this example to show a borderline case of neural networks’ usage:
part of the necessary discussion on machine learning is linked to the correct
usage of simpler and transparent models when the datasets allows it. While
neural networks are a powerful predictive tool, their usage should be avoided
when more interpretable models exhibit a similar performance.

To further examine our proposal, we have investigated the local assumption of
the robustness. In Figure 3, we present a two dimensional visualization of
UMAP projections, clustered with the HDBSCAN algorithm. Each cluster is
colored by its mean robustness score on the validation set.

UMAP is a dimensionality reduction technique that allows to visualize a lower
dimensional projection of the dataset at hand. On such projections, we employ
a density-based clustering technique, HDBSCAN, which allows to derive clusters
without the need to set a hyperparameter for the desired number of clusters,
such as in $k$-means.

As it can be seen from the figure, the projected space is more or less complex
depending on the problem at hand. Simpler datasets like beans are then
clustered in a lower number of clusters, while more complex ones present more
complex patterns. The average robustness score within the retrieved clusters
differs according to both the dataset and the selected neural network being
explained.

As each cluster does present a different average score within, this analysis
gave rise to the intuition that robustness is indeed a local property and
supported our manifold hypothesis, as well as the introduction of the local
$k$nn regressor for the pipeline.

Figure 3: UMAP projections, with HDBSCAN clustering, for beans (above) and
adult (below) dataset, colored by mean robustness score.

##  6 Conclusions and Future Work

We have proposed an ensemble approach and a pipeline to validate the
robustness of neural networks’ feature attributions explainability methods. We
have shown how to validate such proposal and its performance on publicly
available datasets. In practical applications, our approach would aid
practitioners in understanding both the quality of an explanation in terms of
its robustness and to provide the ability of questioning the proposed results
when our pipeline suggests so: this step is essential for the ensurance of
fairness in AI applications.

While we have presented our proposal targeting neural network’s explanations,
the approach is agnostic in nature with respect to both the investigated model
and the applied XAI techniques for explainability. Future applications may
include a generalization of such proposal to a different class of machine
learning models, correlated with the appropriate model-specific feature
attribution methods.

Future work will also focus on the extension of our proposal to include
regression problems and investigate the relationship with adversarial attacks.
In particular, we would like to investigate the robustness of the ensemble to
adversarial attacks targeting one of its composing explanations. We believe
that the ensemble would benefit from the inclusion of multiple methods, which
are not always concordant, to show an increased robustenss against such
attacks.

## References

  * Adadi and Berrada [2018] A. Adadi and M. Berrada.  Peeking inside the black-box: A survey on explainable artificial intelligence (xai).  _IEEE Access_ , 6:52138––52160, 2018.  https://doi.org/10.1109/ACCESS.2018.2870052.
  * Alvarez-Melis and Jaakkola [2018a] D. Alvarez-Melis and T. S. Jaakkola.  On the robustness of interpretability methods, 2018a.
  * Alvarez-Melis and Jaakkola [2018b] D. Alvarez-Melis and T. S. Jaakkola.  Towards robust interpretability with self-explaining neural networks.  In _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ , NIPS’18, page 7786–7795, Red Hook, NY, USA, 2018b. Curran Associates Inc.
  * Anders et al. [2020] C. Anders, P. Pasliev, A.-K. Dombrowski, K.-R. Müller, and P. Kessel.  Fairwashing explanations with off-manifold detergent.  In H. D. III and A. Singh, editors, _Proceedings of the 37th International Conference on Machine Learning_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 314–323. PMLR, 13–18 Jul 2020.  URL https://proceedings.mlr.press/v119/anders20a.html.
  * Bach et al. [2015] S. Bach, A. Binder, G. Montavon, F. Klauschen, K. R. Müller, and W. Samek.  On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation.  _PLoS ONE_ , 10(7), 2015.  https://doi.org/10.1371/journal.pone.0130140.
  * European Commission [2016] European Commission.  Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation) (Text with EEA relevance), 2016.  https://eur-lex.europa.eu/eli/reg/2016/679/oj.
  * European Commission [2021] European Commission.  Proposal for a Regulation of the European Parliament and of the Council laying down harmonised rules on artificial intelligence (Artificial Intelligence Act) and amending certain union legislative acts (COM(2021) 206 final), 2021.  https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex\%3A52021PC0206.
  * Ghorbani et al. [2019] A. Ghorbani, A. Abid, and J. Zou.  Interpretation of neural networks is fragile.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 33(01):3681–3688, Jul. 2019.  10.1609/aaai.v33i01.33013681.  URL https://ojs.aaai.org/index.php/AAAI/article/view/4252.
  * Gosiewska and Biecek [2019] A. Gosiewska and P. Biecek.  Do not trust additive explanations.  _arXiv: Learning_ , 2019.  URL https://api.semanticscholar.org/CorpusID:208512658.
  * Lundberg and Lee [2017] S. M. Lundberg and S. I. Lee.  A unified approach to interpreting model predictions.  _Advances in Neural Information Processing Systems_ , 2017-December:4766––4775, 2017.  https://dl.acm.org/doi/10.5555/3295222.3295230.
  * Nauta et al. [2023] M. Nauta, J. Trienes, S. Pathak, E. Nguyen, M. Peters, Y. Schmitt, J. Schlötterer, M. van Keulen, and C. Seifert.  From anecdotal evidence to quantitative evaluation methods: A systematic review on evaluating explainable ai.  _ACM Computing Surveys_ , 55(13s):1–42, July 2023.  ISSN 1557-7341.  10.1145/3583558.  URL http://dx.doi.org/10.1145/3583558.
  * Pawelczyk et al. [2020] M. Pawelczyk, K. Broelemann, and G. Kasneci.  Learning model-agnostic counterfactual explanations for tabular data.  In _Proceedings of The Web Conference 2020_ , WWW ’20, page 3126–3132, New York, NY, USA, 2020. Association for Computing Machinery.  ISBN 9781450370233.  10.1145/3366423.3380087.  URL https://doi.org/10.1145/3366423.3380087.
  * Ribeiro et al. [2016] M. T. Ribeiro, S. Singh, and C. Guestrin.  "why should i trust you?” explaining the predictions of any classifier.  _NAACL-HLT 2016 - 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Demonstrations Session_ , pages 97––101, 2016.  https://doi.org/10.18653/v1/n16-3020.
  * Rosenfeld [2021] A. Rosenfeld.  Better metrics for evaluating explainable artificial intelligence.  _Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS_ , 1:45–50, 2021.  https://dl.acm.org/doi/abs/10.5555/3463952.3463962.
  * Samek et al. [2021] W. Samek, G. Montavon, S. Lapuschkin, C. J. Anders, and K. R. Müller.  Explaining deep neural networks and beyond: A review of methods and applications.  _Proceedings of the IEEE_ , 109(3):247––278, 2021.  https://doi.org/10.1109/JPROC.2021.3060483.
  * Shrikumar et al. [2017] A. Shrikumar, P. Greenside, and A. Kundaje.  Learning important features through propagating activation differences.  _34th International Conference on Machine Learning, ICML 2017_ , 7:4844–4866, 2017.  https://dl.acm.org/doi/10.5555/3305890.3306006.
  * Sonnewald and Lguensat [2021] M. Sonnewald and R. Lguensat.  Revealing the impact of global heating on north atlantic circulation using transparent machine learning.  _Journal of Advances in Modeling Earth Systems_ , 13, 2021.  URL https://api.semanticscholar.org/CorpusID:237769713.
  * Sundararajan et al. [2017] M. Sundararajan, A. Taly, and Q. Yan.  Axiomatic attribution for deep networks.  _34th International Conference on Machine Learning, ICML 2017_ , 7:5109–5118, 2017.  https://dl.acm.org/doi/10.5555/3305890.3306024.
  * The White House [2022] The White House.  Blueprint for an AI Bill of Rights: Making Automated Systems Work for the American People, 2022.  https://www.whitehouse.gov/ostp/ai-bill-of-rights/.
  * Visani et al. [2020a] G. Visani, E. Bagli, and F. Chesani.  Optilime: Optimized lime explanations for diagnostic computer algorithms.  _ArXiv_ , abs/2006.05714, 2020a.  URL https://api.semanticscholar.org/CorpusID:219559031.
  * Visani et al. [2020b] G. Visani, E. Bagli, F. Chesani, A. Poluzzi, and D. Capuzzo.  Statistical stability indices for lime: Obtaining reliable explanations for machine learning models.  _Journal of the Operational Research Society_ , 73:91 – 101, 2020b.  URL https://api.semanticscholar.org/CorpusID:211004021.
  * Zafar [2019] N. M. Zafar, M. R.and Khan.  Dlime: A deterministic local interpretable model-agnostic explanations approach for computer-aided diagnosis systems.  _arXiv preprint_ , 2019.  https://arxiv.org/abs/1906.10263.