# Explaining the Unique Nature of Individual Gait Patterns with Deep
Learning††footnotetext: This is a pre-print of an article published in
_Scientific Reports_. The final authenticated version is available online at:
https://doi.org/10.1038/s41598-019-38748-8

Fabian Horst  Department of Training and Movement Science, Institute of Sport
Science, Johannes Gutenberg-University Mainz, Mainz, Rhineland-Palatinate,
Germany  these authors contributed equally to this work  Sebastian Lapuschkin
Department of Video Coding & Analytics, Fraunhofer Heinrich Hertz Institute,
Berlin, Germany  these authors contributed equally to this work  Wojciech
Samek  Department of Video Coding & Analytics, Fraunhofer Heinrich Hertz
Institute, Berlin, Germany<EMAIL_ADDRESS>(WIS)  Klaus-
Robert Müller Department of Electrical Engineering & Computer Science,
Technical University Berlin, Berlin, Germany  Department of Brain and
Cognitive Engineering, Korea University, Seoul, Korea  Max Planck Institute
for Informatics, Saarbrücken, Saarland, Germany  wolfgang.schoellhorn@uni-
mainz.de (WIS)  Wolfgang I. Schöllhorn  Department of Training and Movement
Science, Institute of Sport Science, Johannes Gutenberg-University Mainz,
Mainz, Rhineland-Palatinate, Germany<EMAIL_ADDRESS>(WIS)

###### Abstract

Machine learning (ML) techniques such as (deep) artificial neural networks
(DNN) are solving very successfully a plethora of tasks and provide new
predictive models for complex physical, chemical, biological and social
systems. However, in most cases this comes with the disadvantage of acting as
a black box, rarely providing information about what made them arrive at a
particular prediction. This black box aspect of ML techniques can be
problematic especially in medical diagnoses, so far hampering a clinical
acceptance. The present paper studies the uniqueness of individual gait
patterns in clinical biomechanics using DNNs. By attributing portions of the
model predictions back to the input variables (ground reaction forces and
full-body joint angles), the Layer-Wise Relevance Propagation (LRP) technique
reliably demonstrates which variables at what time windows of the gait cycle
are most relevant for the characterisation of gait patterns from a certain
individual. By measuring the time-resolved contribution of each input variable
to the prediction of ML techniques such as DNNs, our method describes the
first general framework that enables to understand and interpret non-linear ML
methods in (biomechanical) gait analysis and thereby supplies a powerful tool
for analysis, diagnosis and treatment of human gait.

## Introduction

The ability to walk is crucial for human mobility and enables to predict
quality of life, morbidity and mortality [69, 68, 60, 63, 62, 25, 17, 40, 49,
21]. Its importance is underlined by the fear of losing the ability to walk,
which is frequently considered to be the first and most significant concern
from individuals that sustain diagnoses like stroke [52, 57] or Parkinson
disease [60, 15]. However, gait and balance are no longer regarded as purely
motor tasks, but are considered as complex sensorimotor behaviours that are
heavily affected by cognitive and affective aspects [23]. This may partially
explain the sensitivity to subtle neuronal dysfunction, and why gait and
postural control can predict the development of disease such as diabetes,
dementia or Parkinson even years before they are diagnosed clinically [23, 70,
6, 10, 67].

In order to prevent, diagnose, or rehabilitate a loss of independence due to
(gait) impairments, gait analysis is common practice to support and
standardise researchers’, clinicians’ and therapists’ decisions when assessing
gait abnormalities and/or identifying changes due to orthopaedic or
physiotherapeutic interventions [5]. But although becoming gradually
established over the past decades, most biomechanical gait analyses have
examined the influence of single time-discrete gait variables, like gait
velocity, step length or range of motion, as risk factor or predictor for
(gait) disease in isolation[42, 73]. While conventional approaches have
addressed successfully many important clinical and scientific questions
related to human gait (impairments), they exhibit some inherent limitations:
When single time-discrete variables (e.g. the range of motion in the knee
joint) are extracted from time-continuous variables (e.g. knee joint angle-
gait stride curve), a large amount of data are discarded. In many cases it
remains unclear, if and to what degree single pre-selected variables are
capable to represent a sufficient description of a whole body movement like
human gait [55, 18, 16, 11, 12]. The a priori selection of single gait
variables relies mostly on the experience and/or subjective opinion of the
analyst, which may lead to a certain risk of investigator bias. Furthermore,
single pre-selected variables might miss potentially meaningful information
that are represented by – or in combination with – other (not selected)
variables. In this context, it seems questionable if the multi-dimensional
interactions between gait characteristics and gait disease or disease that
impair gait can be entirely represented by a subjective selection of single
time-discrete variables [55, 18, 54].

In response to these shortcomings, multivariate statistical analysis [11, 12,
76] and machine learning techniques such as artificial neural networks
(ANN)[8] and support vector machines (SVM)[9, 14, 45, 53] have been used to
examine human locomotion based on time-continuous gait patterns [54].
Significant advances in motion capture equipment and data analysis techniques
have enabled a plethora of studies that have advanced our understanding about
human gait. Due to extensive new datasets [41], the application of machine
learning techniques is becoming increasingly popular in the area of clinical
biomechanics [54, 47, 19] and provided new insights into the nature of human
gait control. The application of ANNs and SVMs highlighted for example that
gait patterns are unique to an individual person [55, 27], exhibited natural
changes within different time-scales [26, 28] and identified that emotional
states [31] and grades of fatigue [32] can be differentiated from human gait
patterns. Furthermore, gender and age-specific gait patterns could be
differentiated [16, 7]. In the context of clinical gait analysis, several
approaches based on machine learning have been published in recent years in
order to support clinicians in identifying and categorizing specific gait
patterns into clinically relevant groups [11, 12, 54, 19]. Previous studies
were able to differentiate gait patterns from healthy individuals and
individuals with (neurological) disorders like Parkinson’s disease [78],
cerebral palsy [1], multiples scleroses [1] or traumatic brain injuries [74]
and pathological gait conditions like lower-limb fractures [19] or acute
anterior cruciate ligament injury [13].

Although machine learning techniques are solving very successfully a variety
of classification tasks and provide new insights from complex physical,
chemical, biological, or social systems, in most cases they go along with the
disadvantage of acting as a black box, rarely providing information about what
made them arrive at a particular decision [4, 44]. This non-transparent
operating and decision-making of most non-linear machine learning methods
leads to the problem that their predictions are not straightforward
understandable and interpretable. This black box manner can be problematic
especially in applications of machine learning in medical diagnosis like gait
analyses and so far strongly hamper their clinical acceptance [76]. The lack
of understanding and interpreting the decision process of machine learning
techniques is a clear drawback and recently attracted attention in the field
of machine learning [4, 44, 22, 59, 77, 3, 81, 48, 79, 58, 20, 43]. In this
context, the so called Layer-wise Relevance Propagation (LRP) technique has
been proposed as general technique for explaining classifier’s decisions by
decomposition, i.e. by measuring the contribution of each input variable to
the overall prediction [3]. LRP has been successfully applied to a number of
technical and scientific tasks such as image classification [50, 37] and text
document classification [2]. Also, interpreting linear and non-linear models
have helped to gain interesting insights in neuroscience [24, 64],
bioinformatics [80, 61, 71] and physics [56].

Due to benefits of machine learning methods in comparison to conventional
approaches in gait analysis [54, 47], LRP appears highly promising to increase
their transparency and therefore make them applicable and reliable for
clinical diagnoses [11, 12, 76]. In the context of personalised medicine, the
aim of the present study was to examine individual gait patterns by:

  1. 1.

demonstrating the uniqueness of gait patterns to the individual by using
(deep) artificial neural networks for predicting identities based on gait;

  2. 2.

verifying that non-linear machine learning methods such as (deep) artificial
neural networks use comprehensible prediction strategies and learn meaningful
gait characteristics by using the Layer-Wise Relevance Propagation; and

  3. 3.

analysing the unique gait signature from an individual by highlighting which
variables at what time windows of the gait cycle are used by the model to
identify an individual.

The presented approach investigates the suitability of understanding and
interpreting the classification of gait patterns using state-of-the-art
machine learning methods. This paper therefore presents a first step towards
establishing a powerful tool that can be used as the basis for future
application of machine learning in (biomechanical) gait analysis and thus
enabling automatic classifications of (neurological) disorders and
pathological gait conditions applicable in clinical diagnoses.

## Results

Figure 1: Overview of data acquisition and data analysis, showing the example
of subject 6. _I:_ The biomechanical gait analysis compromised the recording
of 20 times walking barefoot a distance of 10 m at a self-selected walking
speed. Two force plates and ten infrared cameras recorded the three-
dimensional full-body joint angles and ground reaction forces during a double
step. _II:_ Lower-body joint angles in the sagittal plane (flexion-extension)
(LBJAX) and ground reaction forces (GRF) compromising the fore-aft shear force
(fore-aft), medial-lateral shear force (med-lat) and vertical force (vert)
have been used as time-normalised and concatenated input vectors $\mathbf{x}$
for the prediction of subjects $y$ using deep artificial neural networks
(DNN). Shaded areas for the LBJAX highlight the time where the respective
(left or right) foot is in contact with the ground. _III:_ Decomposition of
input relevance values using the Layer-Wise Relevance Propagation (LRP).
Colour Spectrum for the visualisation of input relevance values of the model
predictions. Throughout this manuscript, we use LRP to exclusively analyse the
prediction for the true class of a sample. Thereby, black line segments are
irrelevant to the model’s prediction. Red and hot colours identify input
segments causing a prediction corresponding to the subject label, while blue
and cold hues are features contradicting the subject label. For subject 6, the
predicting model (CNN-A) achieves true positive rates (TP) of $100\%$ for
LBJAX and $95.23\%$ for GRF.

The uniqueness of human walking to the individual was examined based on time-
continuous kinematic (full-body joint angles) and kinetic (ground reaction
forces) gait patterns (see Methods section for a description of the data).
From a biomechanical gait analysis (Figure 1 _I: Record gait data_), conducted
on 57 healthy subjects, lower-body joint angles (LBJA) and ground reaction
forces (GRF) have been measured as input vectors $\mathbf{x}$ for the
prediction of subjects $y$ using deep artificial neural networks (DNN) (Figure
1 _II: Predict with DNN_). LRP decomposes the prediction $f(\mathbf{x})$ of a
learned function $f$ given an input sample $\mathbf{x}$ into into time-
resolved input relevance values $R_{i}$ for each time-discrete input $x_{i}$,
which enables to explain the prediction of DNNs as partial contributions from
individual input components (Figure 1 _III: Explain prediction using LRP_).
LRP indicates based on which information a model predicts and thereby enables
to interpret the input relevance values and their dynamics as representation
for a certain class (individual). Hence, the input relevance values point out
which gait characteristics were most relevant for the identification of a
certain individual. In the following, input relevance values are visualised by
colour coding, using a diverging and symmetric high contrast colour scheme as
shown in Figure 1 (_III: Explain prediction using LRP_). Here, input elements
neutral to the predictor ($R_{i}\approx 0$) will be shown in black colour,
while warm and hot hues indicate input components supporting the prediction (
$R_{i}\gg 0$ ) of the analysed class and cold hues identify contradictory
inputs ( $R_{i}\ll 0$ ).

As an example, Figure 1 illustrates the unique gait signature from subject 6
by decomposing the input relevance values using LRP (see Figure 2 and
Supplementary Figures SF1 to SF4 for additional subject specific examples).
From the gait feature relevance, we can observe that the extension of the
ankle during the terminal stance phase of the right and left leg and the
flexion of the knee and hip during the initial contact of the right leg is
unique to subject 6. The kinetic data supports this finding, showing the
highest input relevance values for the prediction of subject 6 for the
vertical GRF during the terminal time window of the right and left stance
phase. On these grounds, LRP enables to discover the trial-individual gait
signature from a certain individual. This individual signature can serve as
indicator in clinical diagnoses and starting point for therapeutic
interventions. In our example (Figure 1), the terminal stance phase is unique
to subject 6. While this uniqueness might be interpreted as a reflection of a
highly coordinated individual system, it could also indicate first relevant
information about (forthcoming) complaints or impairments. Clinicians and
researchers are therefore capable to pick up the unique peculiarity during the
terminal stance phase for an individualisation of therapeutic interventions,
e.g. by changing the strength of the responsible muscles for the extension of
the ankle joint or reducing forces during the terminal stance phase during
walking by shoes or insoles.

As discussed earlier, linear classification models have often been used for
the classification of gait patterns [54, 47, 19]. However, other domains in
machine learning have shown that highly non-linear DNNs are capable to
outperform linear and kernel based methods [38, 39, 34, 66, 65, 33]. In this
study, we therefore investigate the applicability of state-of-the-art non-
linear machine learning models such as DNNs for gait analysis and additionally
compare different linear and non-linear machine learning methods in terms of
prediction accuracy, model robustness and decomposition of input relevance
values using the LRP technique (see Methods for detailed description).

### Prediction Accuracy and Model Robustness

The mean prediction accuracy for the subject-classification (on an out of
sample set also denoted as test set) are summarised in Table 1 (and
Supplementary Table ST1). The most striking result to emerge from this table
is that most of the tested models were able to predict the correct class
(subject) with a high accuracy above 95.4% (ground reaction forces), 99.9%
(full-body joint angles) and 99.9% (lower-body joint angles).

Model | Ground | Joint Angles | Joint Angles | Joint Angles | Joint Angles
---|---|---|---|---|---
| Reaction | Full-Body | Full-Body | Lower-Body | Lower-Body
| Forces [%] | [%] | (flex.-ext.) [%] | [%] | (flex.-ext.) [%]
Linear (SGD) | 95.4 (1.7) | 100.0 (0.0) | 96.3 (1.8) | 100.0 (0.0) | 91.5 (2.2)
Linear (SVM) | 100.0 (0.0) | 100.0 (0.0) | 99.7 (0.4) | 100.0 (0.0) | 99.8 (0.6)
MLP (3, 64) | 88.3 (3.7) | 99.9 (0.3) | 84.3 (3.3) | 99.9 (0.3) | 68.5 (8.3)
MLP (3, 256) | 96.6 (0.8) | 100.0 (0.0) | 95.6 (2.6) | 100.0 (0.0) | 89.4 (4.3)
MLP (3, 1024) | 98.8 (1.3) | 100.0 (0.0) | 97.8 (1.0) | 100.0 (0.0) | 96.5 (1.2)
CNN-A | 99.1 (0.8) | 100.0 (0.0) | 95.6 (1.7) | 99.9 (0.3) | 92.0 (3.9)
CNN-C3 | 99.2 (0.6) | 100.0 (0.0) | 97.7 (1.5) | 100.0 (0.0) | 97.0 (1.3)
Table 1: The prediction accuracy of the subject-classification task, reported
as pairs of $mean~{}(standard~{}deviation)$ in percent.

The results in Table 1 (and Supplementary Table ST1) are quite revealing in
several aspects. First, the highest prediction accuracy can be observed
throughout the kinetic and kinematic variables for the linear support vector
machines (Linear (SVM)). Second, the linear one-layer fully-connected neural
network using Stochastic Gradient Descent (Linear (SGD)), fully-connected
neural network (multi layer perceptron (MLP)) using higher number of neurons
(MLP (3, 256) and MLP (3, 1024)) and deep (convolutional) neural network
architectures (CNN-A and CNN-C3) result in similar and throughout high
prediction accuracies, while the prediction accuracy of fully-connected
networks using a lower number of neurons (MLP (3, 64)) is decreased. Third,
surprisingly, even the linear neural network model architecture (Linear (SGD))
was able to predict the correct individual by quite high mean accuracies of
95.4% (ground reaction forces), 100.0% (full-body joint angles) or 100.0%
(lower-body joint angles).

While the prediction accuracy of the linear neural network model (Linear
(SGD)) is comparable to the fully-connected and convolutional neural network
architectures, the robustness of their predictions against noise on the
testing data exhibits considerable differences. As an example, Figure 3 shows
the progression of the mean prediction accuracy of the subject-classification
for the stepwise increase of random noise perturbation on the test data. That
is, for a given input, we compute a random order by which the components of
said input are perturbed, e.g. with the addition of random gaussian noise to
the current input component. After each perturbation step (and executing the
n-th perturbation for all test samples simultaneously), the prediction
performance of a model is re-evaluated. If a model is sensitive to random
noise in the data it will react strongly to the ongoing perturbations. Figure
3 shows for each model the average test set prediction accuracy over 50
consecutive perturbation steps, averaged over 10 repetitions of the
experiment. It is apparent that the prediction accuracy of the Linear (SGD)
model decreases rapidly after only a few perturbation steps, indicating low
robustness and reliability of the model. Variability is an inherent feature of
human movements that occurs not only between but also within individuals.
Therefore robust and reliable model predictions are a mandatory prerequisite
for the development of automatic classification tools in clinical gait
analysis.

Figure 2:  _Left:_ Mean Ground Reaction Force as a line plot, colour coded via
input relevance values for the actual class for subject 21, 28, 39, 42, 55 and
57 using convolutional neural network CNN-A. The highest input relevance
values per body side are highlighted by a red circle. _Right:_ Mean Lower-Body
Joint Angles in the sagittal plane (flexion-extension) as line plot, colour
coded via input relevance values for the actual class for subject 6, 23, 32,
37, 47 and 55 using convolutional neural network CNN-A. The highest input
relevance values per body side are highlighted by a red circle. Shaded areas
for the LBJAX highlight the time where the respective (left or right) foot is
in contact with the ground.

Table 2 (and Supplementary Table ST2) summarises the robustness of the model
predictions and presents the mean area over perturbation curve (AOPC)[50] of
the stepwise increase of random noise perturbation over multiple repetitions
of the perturbation runs on the test data. High AOPC values computed with
above strategy of random perturbations corresponds to high sensitivity to
noise. From the results presented in Table 2 (and Supplementary Table ST2) ,
it is apparent that the stepwise increase of noise on the test data lead to an
abrupt decrease in the prediction accuracy of the linear neural network.
Furthermore, closer inspection shows that fully-connected models based on a
higher number of neurons are more robust against noise perturbations on the
test data than models based on lower numbers of neurons. Even though the
prediction problem at hand seems simple enough such that linear predictors
perform excellently (i.e. the relationship between the input variables and the
prediction target is linearly separable) and the performance-wise gain from
non-linear and networks is minimal, the deeper architectures bring to the
table considerably more robust predictors, which is especially valuable in
application settings like gait analysis, where variability is an important
factor to consider.

Model / | Gaussian | Gaussian | Gaussian | Salt- | Pepper | Salt+ | Shot
---|---|---|---|---|---|---|---
Noise | Random | Random | Random | Random | Random | Random | Random
| $\sigma=0.5$ | $\sigma=1.0$ | $\sigma=2.0$ |  |  |  |
Linear (SGD) | 40.5 (6.0) | 45.6 (3.0) | 47.6 (1.8) | 46.1 (2.7) | 37.1 (7.0) | 46.4 (3.1) | 46.5 (2.9)
Linear (SVM) | 4.0 (3.9) | 18.6 (7.0) | 35.4 (4.7) | 31.3 (7.7) | 2.5 (2.4) | 32.8 (8.7) | 32.4 (8.6)
MLP (3, 64) | 16.2 (14.0) | 27.6 (12.6) | 37.8 (9.0) | 43.5 (7.1) | 14.4 (13.6) | 43.3 (7.4) | 43.3 (7.4)
MLP (3, 256) | 8.9 (9.7) | 20.8 (10.9) | 34.9 (8.4) | 41.7 (6.8) | 7.7 (9.5) | 41.2 (7.4) | 41.1 (7.5)
MLP (3, 1024) | 5.4 (7.0) | 16.7 (9.4) | 32.4 (7.8) | 40.0 (6.6) | 4.3 (6.1) | 39.4 (7.2) | 39.4 (7.2)
CNN-A | 6.1 (6.8) | 18.2 (9.4) | 33.1 (8.0) | 36.9 (6.6) | 8.8 (8.5) | 37.4 (7.5) | 37.4 (7.6)
CNN-C3 | 12.2 (8.6) | 27.5 (9.4) | 38.0 (7.0) | 31.0 (8.5) | 11.3 (8.6) | 37.9 (8.2) | 38.1 (8.1)
Table 2: The area over perturbation curve of the subject-classification of
ground reaction forces for different noise perturbation runs (AOPC)[50].
Values are reported as pairs of $mean~{}(standard~{}deviation)$. Smaller
values correspond to higher robustness.

### Interpreting and Understanding Model Predictions using Layer-Wise
Relevance Propagation

As an example Figure 2 shows relevance attributions for predictions based on
ground reaction force (Figure 2 (left)) and the lower-body joint angles
(Figure 2 (right)) of a certain individual based on his or her gait patterns.

Figure 3: Progression of the mean prediction accuracy of the subject-
classification of ground reaction forces for stepwise random perturbation
using gaussian noise with $\sigma=1$.

The input relevance contributions point out which gait characteristics were
most relevant for the identification of an individual and thereby reveal the
unique gait signature of a certain subject. The comparison of input relevance
values from different subjects indicates that individuals were classified by
both, different gait characteristics and differing magnitudes or shapes of the
same gait characteristic. For example, the highest input relevance values for
the prediction of subject 21 (Figure 2 (left)) can be observed in the
medial/lateral ground reaction force at approximately 10 % of the gait stride
and subject 28 (Figure 2 (left)) in the vertical ground reaction force at
approximately 90 % of the gait stride. While the highest input relevance
values for the prediction of subject 55 (Figure 2 (left)) and subject 57
(Figure 2 (left)) can be both observed in the vertical ground reaction force
at approximately 10 % of the stance phase. It is further interesting that
among all predicted LBJAX curves shown in Figure 2 (right), subject 37 is the
only one that is identified dominantly by gait characteristics during the
swing phase (and not the stance phase). The model has associated the
pronounced flexion of the right (left) ankle joint during the swing phase
unique to subject 37.

In the vast majority of cases, the input relevance values for a certain class
(individual) are comparable between the different model architectures (Figure
4 (left)), i.e. all models pick up on similar features, which are
characteristic to the individual subject in general. It is rather apparent
from the results that most artificial neural networks and the linear SVM are
using a number of different gait characteristics for their predictions, albeit
the vast majority of the inputs seems to be irrelevant to the model’s decision
(see Figure 1; the tight interval ($R\approx 0$) projected onto black), which
are largely based on individual details in a subject’s movement patterns. This
explanatory relevance feedback indicates that (non-linear) machine learning
methods such as (deep) artificial neural networks are not arbitrarily picking
up single randomly distributed input values, but rather learn certain
dynamically meaningful features that can be related to functional gait
characteristics and thus making them applicable in clinical gait analysis.

Figure 4:  _Left:_ Mean Ground Reaction Force as line plot, colour coded via
input relevance for the actual class of different models using artificial
neural networks and the linear SVM model from subject 57. The highest input
relevance values per body side are highlighted by a red circle. _Right:_ Input
relevance as colour coded line plots for the predicted class of different
models using artificial neural networks and linear models of ground reaction
force of the 20 gait trials from subject 28.

When comparing input relevance values from different architectures of
artificial neural networks and the SVM strikingly all model architectures
(except the CNN-C3) show that not a single gait characteristic (certain
variable at a certain time window of the gait cycle) is relevant for the
identification of a certain individual, but rather complex combinations
thereof.

Even more interesting is the observation that input relevance value
attributions appear to be similar between right and the left body side
variables (as an example take Figure 4 (left): Linear (SVM), MLP (3, 1024) and
CNN-A). This indicates the importance of symmetries/asymmetries between left
and right body side variables for the examination of human gait. Note however
that the models all learned without any information about the physiological
meaning of the used variables. While the input relevance attributions are
throughout comparable for all evaluated architectures – i.e. there is, given a
subject, a non-empty set of features recognized as characteristic features to
the individual by all models – some models have learned to ground their
predictions on supplementary gait characteristics (e.g. Figure 4 (left):
CNN-C3 vs CNN-A).

We can further observe that the use of multiple gait characteristics for
prediction can be associated with a model’s robustness to random perturbations
of the input. The observed robustness of the evaluated models is also
reflected by the reliability of the attributed input relevance values: Over
the relevance values of each input component and subject (i.e. 20 relevance
values each), the coefficient of variation[75] was computed in order to prove
their consistency over several trials and cross validation splits. The
coefficient of variation represents the (root mean square) normalized band of
standard deviation around the relevance signal of an input variable, where low
values correspond to high reliability/stability of relevance attributions to a
observed feature between samples and data splits, and thus to the model’s
ability to generalize. A high coefficient of variation indicates that a model
overfits on its respective training split population. As an example, Figure 4
(right) shows the input relevance values for the actual class prediction for
the ground reaction forces. Qualitatively, the highest deviations of input
relevance values between trials appeared in the prediction of the linear
neural network model (Linear (SGD)), which also is most sensitive to even
minute noise added to the test data (Figure 3 and Table 2). It becomes
apparent from Figure 4 (right) that the variance of input relevance is
decreasing in fully-connected neural network architectures composed of
increasing numbers of neurons. However, the lowest variance of the input
relevance values can be observed in the relevance decomposition of the
predictions from the Linear (SVM) and convolutional neural network
architectures, which we attribute for the former to the complexity of the
regularised training regime and complexity of the model itself for the latter.

Table 3 (Supplementary Table ST3) summarises the mean coefficient of variation
for the input relevance values for the subject-classification over all
subjects, expressing that decreasing variance(increasing reliability) goes
along with increasing model complexity, and also model robustness when
compared to Figure 3 and Table 2. Hence, the lowest reliability is present for
the Linear (SGD) model, while reliability is increasing in fully-connected
model architectures composed of increasing numbers of layers and neurons and
discloses the highest reliability for the convolutional neural networks (CNN-A
and CNN-C3). Interestingly, the reliability of the input relevance values from
linear support vector machines (Linear (SVM)) are as well on a high level and
comparable to convolutional neural networks.

Model | Ground | Joint Angles | Joint Angles | Joint Angles | Joint Angles
---|---|---|---|---|---
| Reaction | Full-Body | Full-Body | Lower-Body | Lower-Body
| Forces |  | (flex.-ext.) |  | (flex.-ext.)
Linear (SGD) | 4.31 (0.25) | 4.27 (0.08) | 3.93 (0.12) | 4.16 (0.10) | 3.86 (0.15)
Linear (SVM) | 0.31 (0.08) | 0.56 (0.10) | 0.31 (0.07) | 0.48 (0.09) | 0.26 (0.05)
MLP (3, 64) | 1.31 (0.34) | 2.73 (0.15) | 1.58 (0.17) | 2.37 (0.16) | 1.41 (0.23)
MLP (3, 256) | 0.84 (0.18) | 2.21 (0.13) | 1.05 (0.10) | 1.86 (0.13) | 0.85 (0.12)
MLP (3, 1024) | 0.63 (0.12) | 1.85 (0.10) | 0.77 (0.08) | 1.50 (0.10) | 0.61 (0.09)
CNN-A | 0.30 (0.08) | 0.56 (0.09) | 0.35 (0.06) | 0.49 (0.08) | 0.32 (0.05)
CNN-C3 | 0.35 (0.08) | 0.50 (0.08) | 0.43 (0.08) | 0.48 (0.08) | 0.44 (0.08)
Table 3: The coefficient of variation of the input relevance values of the
subject-classification, reported in pairs of $mean~{}(standard~{}deviation)$
over all subjects.

## Discussion

The present results verified the uniqueness of characteristics for individual
gait patterns based on kinematic and kinetic variables. By decomposing the
prediction of machine learning methods such as (deep) artificial neural
networks back to the input variables (time-continuous ground reaction forces
and full-body joint angles), the LRP technique demonstrated which gait
variables were most relevant for the characterisation of gait patterns from a
certain individual. By measuring the contribution of each input variable to
the prediction of (deep) artificial neural networks, the present paper
describes a procedure that enables to understand and interpret the predictions
of non-linear machine learning methods in (biomechanical) gait analysis. LRP
thereby outlines the first general framework that facilitates to overcome the
inherent black box problem of non-linear machine learning methods and makes
them applicable in clinical gait analysis. In the context of personalised
medicine, the determination of characteristics that are specific for gait
patterns of a certain individual facilitates to support clinicians and
researchers in the individualisation of their analyses, diagnoses and
interventions.

The individual nature of human gait patterns was quantified using different
linear and non-linear machine learning methods. The present results support
previous studies on the individuality of human movements [55, 27, 31, 32] and
provide evidence for gait characteristics that are unique to an individual and
can be clearly differentiated from gait patterns of other individuals. Most of
the artificial neural network architectures classified gait patterns almost
error-free to the corresponding individual and achieved very high prediction
accuracies that are suitable for clinical applications. However, advantages
for more sophisticated model architectures (like fully-connected model using a
higher numbers of neurons or deep convolutional neural networks) can be
observed in higher prediction accuracies (Table 1 and Supplementary Table ST1)
and even more significant in the higher robustness of the model predictions
against noise perturbations on the test data (Figure 3, Table 2 and
Supplementary Table ST2). Because variability within individuals [26, 28] as
well as variability due to differences between individuals [55, 27], genders
[7] and ages [16] is an inherent feature of human motor control, prediction
accuracy and model robustness are both essential for the development of
reliable clinical applications using machine learning. Consequently, the
present results suggest high potential of state-of-the-art non-linear methods
such as DNNs compared to linear methods.

One of the issues that emerges from the evidence that gait patterns are unique
to an individual, is the demand to evaluate clinical approaches for diagnoses
and therapy that consider individual needs[55, 27]. However, previous studies
could not address how an individualisation of diagnoses and therapy could be
obtained. By measuring the contribution of each input variable to the
prediction of machine learning methods, the LRP method enables one for the
first time to describe qualitatively why a certain individual could be
identified based on his/her gait patterns. The LRP technique provides the
possibility to comprehend what a model has learned and to interpret the input
relevance values as representation for a certain class (individual). In the
context of personalised gait analysis (medicine) that means, the decomposition
of input relevance values and their dynamics describe what input variables are
most relevant for the identification of a certain individual and thereby
indicate which input variables are the most characteristic ones for the gait
patterns of a certain individual (Figure 1).

On these grounds, the input relevance values enable clinicians and researchers
to determine the unique gait signature of a certain individual based on single
trials and adjust their analyses, diagnoses and interventions to the specific
needs of this individual (Figure 1, Figure 2).

In addition, explaining the model predictions provide interesting insights
into the analysis of gait patterns. The input relevance values highlight that
in most cases not a single gait characteristic (specific value or shape of a
certain variable at a certain time of the gait cycle) is relevant for the
identification of a certain individual. It is rather apparent that most
artificial neural networks architectures look for the shape of different
variables as well as their interaction at the same time window or at different
time windows of the gait cycle. Similar results have been found on
photographic image data [3, 37]. Interestingly, the prediction of most
artificial neural network architectures (except CNN-C3) trace to input
relevance values that are similar between right and left body side variables
at the same time. That means a certain variable at a certain time window of
the gait cycle of the right and the left body side is relevant for the
prediction of the models (Figure 4 (left)), indicating importance of
symmetries and asymmetries between right and left body movements for the
identification of individuals and probably the examination of human gait in
general.

The input relevance values support that machine learning approaches like
artificial neural networks are able to consider several variables at various
time points of the gait cycle for their predictions. In comparison to most
conventional approaches of gait analysis that are based on single pre-selected
variables, machine learning approaches seem to be promising to represent the
multi-dimensional associations of human locomotion and their connections to
functional and neurological disease [55, 18, 54].

The present results demonstrate in the vast majority of cases that the input
relevance values are similar between different model architectures (Figure 4).
That means all models pick up similar features for the classification of gait
patterns, which are characteristic to the individual subject in general.
However, the LRP technique enables to identify the strategy of a certain model
to classify a class (individual gait patterns) and to compare strategies
between different model architectures [37]. For an implementation of machine
learning in clinical diagnoses and therapeutic interventions, for example in
terms of an automatic classification of gait disorders or (neurological)
disease [19, 78, 1], the understanding about their decisions and decision-
making seems to be inevitable. Since the lack of transparency has so far been
a major drawback of preceding applications of machine learning, e.g. in
medical applications (like gait analysis), further research on explaining,
understanding and interpreting machine learning predictions should get
attention.

Here, the decomposition of input variable relevance values using the LRP was
consistent over multiple test trials and cross-validation splits. Taken
together, the results demonstrate the suitability of the proposed method for
the explanation of machine learning predictions in clinical (biomechanical)
gait analysis. However, higher reliability of input relevance values between
test trials and cross-validation splits indicate advantages for deep
(convolutional) neural networks architectures. These findings are in agreement
with those observed in earlier studies on text [2] or image [35]
classification that indicated more robust and traceable class representations
in deep (convolutional) neural networks.

In conclusion, the present findings underline that methods enabling to
understand and interpret the predictions of machine learning, like the LRP,
are highly promising for the application and implementation of machine
learning in gait analysis. Due to the above discussed advantages of non-linear
machine learning methods such as DNNs for the analysis of human gait [11, 12,
54, 47], the understanding and interpreting of machine learning predictions is
essential in order to overcome one of their major drawbacks (the lack of
transparency) [11, 12, 76]. Using the testbed of uniqueness of individual gait
patterns, the present study proposed a general framework for the understanding
and interpretation of non-linear machine learning methods in gait analysis
thus providing a solid basis for future studies in biomechanical analysis and
clinical diagnosis.

## Methods

### Subjects and ethics statement

Fifty-seven physically active subjects ($29$ female, $28$ male; $23.1\pm 2.7$
years; $1.74\pm 0.10$ m; $67.9\pm 11.3$ kg) without gait pathology and free of
lower extremity injuries participated in the study. The study was carried out
according to the Declaration of Helsinki at the Johannes Gutenberg-University
in Mainz (Germany). All subjects were informed about the experimental protocol
and provided their informed written consent to participate in the study.
Subjects appearing in the figures provided informed written consent to the
publication of identifying images and videos in an online open-access
publication. The approval from the ethical committee of the medical
association Rhineland-Palatinate in Mainz (Germany) was received.

### Experimental protocol and data acquisition

The subjects performed 20 gait trials in a single assessment session, while
they did not undergo any intervention. For each trial upper- and lower-body
joint angles as well as ground reaction forces were measured, while the
subjects walked on a 10 m path. The subjects were instructed to walk barefoot
at a self-selected speed. Kinematic data were recorded using a full-body
marker set consisting of 62 retro reflective markers placed on anatomical
landmarks (Figure 5). Ten Oqus 310 infrared cameras (Qualisys AB, Sweden)
captured the three-dimensional marker trajectories at a sampling frequency of
250 Hz.

Figure 5: Full body marker set in (A) anterior (B) right lateral (C) posterior
view. The markers were placed at os frontale glabella, 7th cervical vertebrae,
sternum jugular notch, sacrum (mid-point between left and right posterior
superior iliac spine) and bilaterally at greater wing of sphenoid bone,
acromion, scapula inferior angle, humerus lateral epicondyle, humerus medial
epicondyle, forearm, radius styloid process, ulna styloid process, head of 3rd
metacarpal, iliac crest tubercle, femur greater trochanter, femur lateral
epicondyle, femur medial epicondyle, fibula apex of lateral malleolus, tibia
apex of medial malleolus, posterior surface of calcaneus, head of 1st
metatarsus, head of 5th metatarsus and clusters with four markers each at the
thigh and shank and clusters of three markers each at the humerus.

The three-dimensional ground reaction forces were recorded by two Kistler
force plates (Type 9287CA) (Kistler, Switzerland) at a frequency of 1000 Hz.
The recording was managed in time-synchronization by the Qualisys Track
Manager 2.7 (Qualisys AB, Sweden). Two experienced assessors attached the
markers and conducted the analysis. Every subject was analysed by the same
assessor only. The laboratory environment was kept constant during the
investigation.

Before the data acquisition, each subject performed 20 test trials to get
accustomed to the experimental setup and to assign a starting point for a walk
over the force plates. This procedure is described to minimize the impact of
targeting on the force plates on the observed gait variables [51, 72].
Additionally, the participants were instructed to watch a neutral symbol
(smiley) on the opposing wall of the laboratory to direct their attention away
from targeting on the force plates and ensure a natural walk with an upright
body position.

### Data processing

The gait analysis was conducted for one gait stride per trial. The stride was
defined from right foot heel strike to left foot toe off and was determined
using a vertical ground reaction force threshold of 10 N. The three-
dimensional marker trajectories and ground reaction forces were filtered using
a second order Butterworth bidirectional low-pass filter at a cut off
frequency of 12 Hz and 50 Hz, respectively. The ground reaction force data
were normalized to the body weight. The computation of the upper- and lower-
body joint angles was conducted by Visual3D Standard v4.86.0 (C-Motion, USA)
for elbow, shoulder, spine, hip, knee and ankle in sagittal, transversal and
coronal plane.

Further data processing was executed by a self-written script within the
software Matlab 2016a (MathWorks, USA). Each variable time course was
normalized to 101 data points, z-transformed and scaled to a range of -1 to
1[30]. The z-transformation was executed for kinematic variables for each
trial separately and for kinetic variables for all trials. The scaling was
carried out in order to prevent numerical difficulties during the calculation
of the artificial neural networks [30] and to ensure an equal contribution of
all variables to the classification rates and thereby avoid that variables in
greater numeric ranges dominate those in smaller numeric ranges[30]. Scaling
is a common procedure for data processing in advance for the classification of
gait data[55, 19].

### Data Analysis

The data analysis was conducted within the software frameworks of Matlab 2016a
(MathWorks, USA) and Python 2.7 (Python Software Foundation, USA). The ability
to distinguish gait patterns of one subject from gait patterns of other
subjects was investigated in a multi-class classification (subject-
classification) using the data from 57 subjects. The classification of gait
patterns, based on time-continuous kinetic and kinematic data, was carried out
by supervised machine learning models using support vector machines (SVM)[9,
14, 45, 53] and artificial neural networks (ANN)[76, 38, 39]. While fully-
connected ANNs such as multi layer perceptions (MLP) and SVM represent
established models for the classification of gait patterns based on joined
input vectors of time-continuous kinematic and kinetic data[54], convolutional
(deep) artificial neural networks (DNN) have not yet been applied for the
biomechanical analysis of human movements. Because DNNs showed superior
prediction accuracies in domains like image[34, 66, 65] or speech
recognition[33], they seem to be promising for the given classification of
human gait patterns.

In the present paper, an SVM and different architectures of fully-connected
and convolutional artificial neural networks were compared in terms of
prediction accuracy, model robustness to noise on the test data and
decomposition of input relevance values.

As a simple baseline, two linear classification models were implemented. A
one-layer fully-connected neural network (Linear (SGD)) and a SVM using a
linear kernel function (Linear (SVM)). Among the considered fully-connected
artificial neural network architectures were all combinations $L\times H$ with
$L\in[2,3]$ describing the number of layers and $H\in[2^{6},\dots,2^{10}]$
describing the number of neurons per hidden layer. For the convolutional
artificial neural network architectures, the number of convolutional layers is
$C\in[1,2,3]$ with the number of hidden neurons depending on the number of
channels in the data as well as the stride and shape of the learned
convolutional filters. All convolutional neural network architectures are
topped off with one linear layer, connecting all neurons of the highest
convolutional layer to the number of classes of the prediction problem. Major
architectural differences between the evaluated convolutional neural network
architectures were the sizes of the input layer filters ( $3\times 3$ and
$6\times 6$ as well as $C\times 3$ , $C\times 6$ and $C\times C$ ) spanning
different amounts of neighbouring channels and time windows in the input
samples.

All artificial neural networks using hidden layers (i.e. all architectures
except the linear classifier Linear (SGD)) have ReLU-nonlinearities after each
linear / convolutional layer as activation functions for the hidden neurons
and a SoftMax activation function for the output layer. Both linear and the
fully-connected classifiers receive as input the channel $\times$ time samples
as row-concatenated channel $\cdot$ time dimensional vectors. The
convolutional models directly operate on the channel $\times$ time shaped
samples. For a detailed description of all evaluated model architectures, see
the Supplement (Supplementary Tables ST4-ST9).

With the exception of the linear support vector machines predictor, all models
have been trained as n-way classifiers using Stochastic Gradient Descent (SGD)
Optimization[39] for up to $3\cdot 10^{4}$ iterations of mini batches of 5
randomly selected training samples and an initial learning rate of $5e^{-3}$.
The learning rate is gradually lowered to $1e^{-3}$ and then $5e^{-4}$ after
every $10^{4}$ training iterations. Model weights are initialized with random
values drawn from normal distributions with $\mu=0$ and
$\sigma=m^{-\frac{1}{2}}$, where $m$ is the number of inputs to each output
neuron of the layer[39]. The linear support vector machine model has been
using regularized quadratic optimization.

For SVM, the multi-class linear support vector classifier of the scikit learn
Toolbox for python [46] was used with a regularisation parameter $C=0.1$.

Prediction accuracies were reported over a ten-fold cross validation
configuration, where eight parts of the data are used for training, one part
is used as a validation set and the remaining part is reserved for testing.
With on average 912 samples per split being reserved for training, a (neural
network) model passes the training set up to 164 (= 30000 iterations $\cdot$ 5
samples per batch / 912 training samples on average) times and each training
stage tied to a given learning rate may be terminated prematurely if the model
performance has converged on the validation hold out set to avoid overfitting
on the training data. For subject-classification, the 20 samples per subject
are uniformly distributed across all data partitions at random.

### Layer-wise Relevance Propagation

One of the main reasons for the wide-spread use of linear models in the (meta)
sciences is the inherent transparency of the prediction function. Given a set
of learned model parameters $\\{\mathbf{w},\mathbf{b}\\}$ where $\mathbf{w}$
is a weight vector matching the dimensionality of the input data and a bias
term $\mathbf{b}$, the (multi-class) prediction function for an arbitrary
input $\mathbf{x}$ evaluates for class $c$ as

$\displaystyle
f_{c}(\mathbf{x})~{}=~{}\mathbf{w}^{T}\mathbf{x}~{}+~{}b_{c}~{}=~{}\sum_{i}~{}w_{ic}x_{i}~{}+~{}b_{c}~{}.$
(1)

It is apparent, that component $i$ of the given input $\mathbf{x}$ contributes
to the evaluation of $f_{c}$ together with the learned parameters as the
quantity $w_{ic}x_{i}$. Each decision made by a linear model is therefore
transparent, while complex non-linear models are generally considered black
box classifiers.

A technique called Layer-Wise Relevance Propagation (LRP)[3] has generalized
the explanation of linear models for non-linear models such as deep
(convolutional) artificial neural networks and arbitrary pipelines of pre-
processing steps and nonlinear predictions. As a principled and general
approach, LRP decomposes the output of a given decision function $f_{c}$ for
an input $\mathbf{x}$ and attributes “relevance scores” $R_{i}$ to all
components $i$ of $\mathbf{x}$, such that $f_{c}(\mathbf{x})=\sum_{i}R_{i}$.
Similarly to how the prediction of a linear model can be “explained” LRP
starts at the model output by selecting a class output $c$ of interest,
initiating $f_{c}(\mathbf{x})=R_{j}$ (and selecting 0 for all other model
outputs) as the initial output neuron relevance value. Note that for two-class
problems, there is often one shared model output, with the predicted class
being determined by the sign of the prediction. Here, $f(\mathbf{x})=R_{j}$
initially.

The method can best be described by considering a single output neuron $j$
anywhere within the model. That neuron receives a quantity of relevance
$R_{j}$ from upper layer neurons (or is initiated with that value in case of a
model output neuron, and redistributes that quantity to its immediate input
neurons $i$ , in proportion to the contribution of the inputs $i$ to the
activation of $j$ in the forward pass:

$\displaystyle R_{i\leftarrow j}=\frac{z_{ij}}{z_{j}}R_{j}~{}.$ (2)

Here, $z_{ij}$ is a quantity measuring the contribution of the input neuron
$i$ to the output neuron $j$ and $z_{j}$ is the aggregation thereof. This
decomposition approach follows the semantic that the output neuron $j$ holds a
certain amount of relevance, due to its activation in the forward pass and its
influence to consecutive layers and finally the model output. This relevance
is then distributed across the neuron’s inputs in proportion of each input’s
contribution to the activation of neuron: If a neuron $i$ contributes as
$z_{ij}$ towards the overall trend $z_{j}$ , it shall receive a positively
weighted fraction of $R_{j}$. If fires against the overall trend, e.g. the
amount or relevance attributed to it will be weighted negatively.

Usually, the layers of an ANN model implement an affine transformation
function $x_{j}=\left(\sum_{i}x_{i}w_{ij}\right)+~{}b_{j}$ or a (component-
wise) non-linearity $x_{j}=\sigma(x_{i})$. In the former case, we then have
$z_{ij}=x_{i}w_{ij}$ , for example, and $z_{j}$ is the output activation
$x_{j}$ . In the latter case, $z_{ij}=\delta_{ij}\sigma(x_{i})$ where
$\delta_{ij}$ is the Kronecker delta, since there is no mixing between inputs
and outputs of different subscripts.

The relevance score $R_{i}$ at input neuron $i$ is then obtained by pooling
all incoming relevance values $R_{i\leftarrow j}$ from the output neurons to
which $i$ contributes in the forward pass:

$\displaystyle R_{i}=\sum_{j}R_{i\leftarrow j}~{}.$ (3)

Together, both above relevance decomposition and pooling steps ensure a local
relevance conservation property, i.e. $\sum_{i}R_{i}=\sum_{j}R_{j}$ and thus
$\sum_{i}R_{i}=f(\mathbf{x})$ for all layers of the model. In case of a
component-wise operating non-linear activation, e.g. a ReLU
($\forall_{i=j}:\mathbf{x}_{j}=\max(0,x_{i})$) or Tanh
($\forall_{i=j}:x_{j}=\tanh(x_{i})$), then $\forall_{i=j}:R_{i}=R_{j}$. since
the top layer relevance values $R_{j}$ only need to be attributed towards one
single respective input $i$ for each output neuron $j$.

After initiating the algorithm at the model output, it iterates over all the
layers of the model towards the input, until relevance scores $R_{i}$ for all
input components $x_{i}$ are obtained. Assuming a (strong) positive model
output represents the predicted presence of a class, then the input level
relevance scores can be interpreted as follows: Values $R_{i}\gg 0$ indicate
components $x_{i}$ of the input which, due to the models’ learned decision
function, represent the presence of the “explained” class, while conversely
$R_{i}\ll 0$ contradict the prediction of that class. $R_{i}\approx 0$
identify inputs $x_{i}$ which have no or only little influence to the model’s
decision.

Applying above decomposition rules to a linear classifier
$f(\mathbf{x})=\sum_{i}x_{i}w_{i}+b$ with only a single output results in
$z_{ij}=x_{i}w_{i}$ and $z_{bj}=b$, since the bias $b$ can be considered a
constant always-on neuron, and $z_{j}=\sum_{i}x_{i}w_{i}+b=f(\mathbf{x})$.
Initiating the (only) model output relevance as $R_{j}=f(\mathbf{x})$ and
substituting both $z_{ij}$ and $z_{j}$ in above relevance decomposition and
pooling rules in Equations (2) and (3) yields:

$\displaystyle R_{i\leftarrow j}$
$\displaystyle=\frac{x_{i}w_{i}}{f(\mathbf{x})}f(\mathbf{x})=x_{i}w_{i}$
$\displaystyle~{};~{}\text{for inputs depending on data}~{}\mathbf{x}$
$\displaystyle R_{b\leftarrow j}$
$\displaystyle=\frac{b}{f(\mathbf{x})}f(\mathbf{x})=b$
$\displaystyle~{};~{}\text{for the relevance quantities for the bias}~{}b~{}.$
(4)

Since the model only has one output, the pooling at each input $x_{i}$ (or the
bias) becomes $R_{i}=R_{i\leftarrow j}$ (or $R_{b}=R_{b\leftarrow j}$). In
short, the application of LRP to a model consisting of only a single linear
layer collapses to $R_{i}=x_{i}w_{i}$, the inherent explanation of the
decision of a linear model in terms of input variables and the bias. For
further details, please refer to [3].

### Data Availability

The datasets generated and analysed during the current study are available in
the Mendeley Data Repository [29] (http://dx.doi.org/10.17632/svx74xcrjr.1).
The Layer-Wise Relevance Propagation Toolbox [36]
(https://github.com/sebastian-lapuschkin/lrp_toolbox) and the experimental
code derivation thereof is available on GitHub (https://github.com/sebastian-
lapuschkin/interpretable-deep-gait).

## References

  * [1] Murad Alaqtash, Thompson Sarkodie-Gyan, Huiying Yu, Olac Fuentes, Richard Brower, and Amr Abdelgawad.  Automatic classification of pathological gait patterns using ground reaction forces and machine learning algorithms.  In Engineering in Medicine and Biology Society, EMBC, 2011 Annual International Conference of the IEEE, pages 453–457. IEEE, 2011.
  * [2] Leila Arras, Franziska Horn, Grégoire Montavon, Klaus-Robert Müller, and Wojciech Samek.  "What is relevant in a text document?": An interpretable machine learning approach.  PloS ONE, 12(8):e0181142, 2017.
  * [3] Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek.  On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation.  PLoS ONE, 10(7):e0130140, 2015.
  * [4] David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus-Robert Müller.  How to explain individual classification decisions.  Journal of Machine Learning Research, 11(Jun):1803–1831, 2010.
  * [5] Richard Baker.  Measuring walking: A handbook of clinical gait analysis.  Mac Keith Press, 2013.
  * [6] Rossitza Baltadjieva, Nir Giladi, Leor Gruendlinger, Chava Peretz, and Jeffrey M Hausdorff.  Marked alterations in the gait timing and rhythmicity of patients with de novo Parkinson’s disease.  European Journal of Neuroscience, 24(6):1815–1820, 2006.
  * [7] Rezaul Begg and Joarder Kamruzzaman.  A machine learning approach for automated recognition of movement patterns using basic, kinetic and kinematic gait data.  Journal of Biomechanics, 38(3):401–408, 2005.
  * [8] Christopher M. Bishop et al.  Neural networks for pattern recognition.  Oxford university press, 1995.
  * [9] Bernhard E Boser, Isabelle M Guyon, and Vladimir N Vapnik.  A training algorithm for optimal margin classifiers.  In Proceedings of the fifth annual workshop on Computational learning theory, pages 144–152. ACM, 1992.
  * [10] Teresa Buracchio, Hiroko H Dodge, Diane Howieson, Dara Wasserman, and Jeffrey Kaye.  The trajectory of gait speed preceding mild cognitive impairment.  Archives of Neurology, 67(8):980–986, 2010.
  * [11] Tom Chau.  A review of analytical techniques for gait data. Part 1: Fuzzy, statistical and fractal methods.  Gait & Posture, 13(1):49–66, 2001.
  * [12] Tom Chau.  A review of analytical techniques for gait data. Part 2: Neural network and wavelet methods.  Gait & Posture, 13(2):102–120, 2001.
  * [13] Josef Christian, Josef Kröll, Gerda Strutzenberger, Nathalie Alexander, Michael Ofner, and Hermann Schwameder.  Computer aided analysis of gait patterns in patients with acute anterior cruciate ligament injury.  Clinical Biomechanics, 33:55–60, 2016.
  * [14] Corinna Cortes and Vladimir Vapnik.  Support-vector networks.  Machine Learning, 20(3):273–297, 1995.
  * [15] Terry Ellis, James T Cavanaugh, Gammon M Earhart, Matthew P Ford, Kenneth B Foreman, and Leland E Dibble.  Which measures of physical function and motor impairment best predict quality of life in Parkinson’s disease?  Parkinsonism & Related Disorders, 17(9):693–697, 2011.
  * [16] Bjoern M Eskofier, Peter Federolf, Patrick F Kugler, and Benno M Nigg.  Marker-based classification of young–elderly gait pattern differences via direct pca feature extraction and svms.  Computer Methods in Biomechanics and Biomedical Engineering, 16(4):435–442, 2013.
  * [17] Cecilia Fagerström and Gunilla Borglin.  Mobility, functional ability and health-related quality of life among people of 60 years or older.  Aging Clinical and Experimental Research, 22(5-6):387–394, 2010\.
  * [18] Peter Federolf, Karelia Tecante, and Benno Nigg.  A holistic approach to study the temporal variability in gait.  Journal of Biomechanics, 45(7):1127–1132, 2012.
  * [19] Joana Figueiredo, Cristina P Santos, and Juan C Moreno.  Automatic recognition of gait patterns in human motor disorders using machine learning: A review.  Medical Engineering & Physics, 2018.
  * [20] Ruth C. Fong and Andrea Vedaldi.  Interpretable explanations of black boxes by meaningful perturbation.  In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 3449–3457, 2017.
  * [21] Roberta Forte, Colin AG Boreham, Giuseppe De Vito, and Caterina Pesce.  Health and quality of life perception in older adults: The joint role of cognitive efficiency and functional mobility.  International Journal of Environmental Research and Public Health, 12(9):11328–11344, 2015.
  * [22] Muriel Gevrey, Ioannis Dimopoulos, and Sovan Lek.  Review and comparison of methods to study the contribution of variables in artificial neural network models.  Ecological Modelling, 160(3):249–264, 2003.
  * [23] Nir Giladi, Fay B Horak, and Jeffrey M Hausdorff.  Classification of gait disturbances: Distinguishing between continuous and episodic changes.  Movement Disorders, 28(11):1469–1473, 2013.
  * [24] Stefan Haufe, Frank Meinecke, Kai Görgen, Sven Dähne, John-Dylan Haynes, Benjamin Blankertz, and Felix Bießmann.  On the interpretation of weight vectors of linear models in multivariate neuroimaging.  Neuroimage, 87:96–110, 2014.
  * [25] Calvin Hayes Hirsch, Petra Bůžková, John A Robbins, Kushang V Patel, and Anne B Newman.  Predicting late-life disability and death by the rate of decline in physical performance measures.  Age and Ageing, 41(2):155–161, 2011.
  * [26] F Horst, F Kramer, B Schäfer, A Eekhoff, P Hegen, BM Nigg, and WI Schöllhorn.  Daily changes of individual gait patterns identified by means of support vector machines.  Gait & Posture, 49:309–314, 2016.
  * [27] F Horst, M Mildner, and WI Schöllhorn.  One-year persistence of individual gait patterns identified in a follow-up study – A call for individualised diagnose and therapy.  Gait & Posture, 58:476–480, 2017.
  * [28] Fabian Horst, Alexander Eekhoff, Karl M Newell, and Wolfgang I Schöllhorn.  Intra-individual gait patterns across different time-scales as revealed by means of a supervised learning model using kernel-based discriminant regression.  PloS ONE, 12(6):e0179738, 2017.
  * [29] Fabian Horst, Sebastian Lapuschkin, Wojciech Samek, Klaus-Robert Müller, and Wolfgang I Schöllhorn.  A public dataset of overground walking kinetics and full-body kinematics in healthy individuals.  Mendeley Data Repository, 2018.  http://dx.doi.org/10.17632/svx74xcrjr.1.
  * [30] Chih-Wei Hsu, Chih-Chung Chang, Chih-Jen Lin, et al.  A practical guide to support vector classification, 2003.
  * [31] Daniel Janssen, Wolfgang I Schöllhorn, Jessica Lubienetzki, Karina Fölling, Henrike Kokenge, and Keith Davids.  Recognition of emotions in gait patterns by means of artificial neural nets.  Journal of Nonverbal Behavior, 32(2):79–92, 2008.
  * [32] Daniel Janssen, Wolfgang I Schöllhorn, Karl M Newell, Jörg M Jäger, Franz Rost, and Katrin Vehof.  Diagnosing fatigue in gait patterns by support vector machines and self-organizing maps.  Human Movement Science, 30(5):966–975, 2011.
  * [33] Yoon Kim.  Convolutional neural networks for sentence classification.  In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1746–1751, 2014.
  * [34] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In Advances in Neural Information Processing Systems (NIPS), pages 1097–1105, 2012.
  * [35] Sebastian Lapuschkin, Alexander Binder, Grégoire Montavon, Klaus-Robert Müller, and Wojciech Samek.  Analyzing classifiers: Fisher vectors and deep neural networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2912–2920, 2016.
  * [36] Sebastian Lapuschkin, Alexander Binder, Grégoire Montavon, Klaus-Robert Müller, and Wojciech Samek.  The layer-wise relevance propagation toolbox for artificial neural networks.  Journal of Machine Learning Research, 17(114):1–5, 2016.
  * [37] Sebastian Lapuschkin, Alexander Binder, Klaus-Robert Müller, and Wojciech Samek.  Understanding and comparing deep neural networks for age and gender classification.  In Proceedings of the ICCV’17 Workshop on Analysis and Modeling of Faces and Gestures (AMFG), 2017.
  * [38] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton.  Deep learning.  Nature, 521(7553):436, 2015.
  * [39] Yann A LeCun, Léon Bottou, Genevieve B Orr, and Klaus-Robert Müller.  Efficient backprop.  In Neural networks: Tricks of the trade, pages 9–48. Springer, 2012\.
  * [40] Philipp Mahlknecht, Stefan Kiechl, Bastiaan R Bloem, Johann Willeit, Christoph Scherfler, Arno Gasperi, Gregorio Rungger, Werner Poewe, and Klaus Seppi.  Prevalence and burden of gait disorders in elderly men and women aged 60–97 years: A population-based study.  PLoS One, 8(7):e69627, 2013.
  * [41] Marnee J McKay, Jennifer N Baldwin, Paulo Ferreira, Milena Simic, Natalie Vanicek, Claire E Hiller, Elizabeth J Nightingale, Niamh A Moloney, Kate G Quinlan, Fereshteh Pourkazemi, et al.  1000 norms project: Protocol of a cross-sectional study cataloging human variation.  Physiotherapy, 102(1):50–56, 2016.
  * [42] Kathryn Mills, Michael A Hunt, and Reed Ferber.  Biomechanical deviations during level walking associated with knee osteoarthritis: A systematic review and meta-analysis.  Arthritis Care & Research, 65(10):1643–1665, 2013.
  * [43] Grégoire Montavon, Sebastian Bach, Alexander Binder, Wojciech Samek, and Klaus-Robert Müller.  Explaining nonlinear classification decisions with deep taylor decomposition.  Pattern Recognition, 65:211–222, 2017.
  * [44] Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller.  Methods for interpreting and understanding deep neural networks.  Digital Signal Processing, 73:1–15, 2018.
  * [45] K-R Müller, Sebastian Mika, Gunnar Ratsch, Koji Tsuda, and Bernhard Schölkopf.  An introduction to kernel-based learning algorithms.  IEEE Transactions on Neural Networks, 12(2):181–201, 2001.
  * [46] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al.  Scikit-learn: Machine learning in python.  Journal of Machine Learning Research, 12(Oct):2825–2830, 2011.
  * [47] Angkoon Phinyomark, Giovanni Petri, Esther Ibáñez-Marcelo, Sean T Osis, and Reed Ferber.  Analysis of big data in gait biomechanics: Current trends and future directions.  Journal of Medical and Biological Engineering, 38(2):244–260, 2018\.
  * [48] Marco Túlio Ribeiro, Sameer Singh, and Carlos Guestrin.  "Why should I trust you?": Explaining the predictions of any classifier.  In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016, pages 1135–1144, 2016.
  * [49] Laurence Z Rubenstein, Christopher M Powers, and Catherine H MacLean.  Quality indicators for the management and prevention of falls and mobility problems in vulnerable elders.  Annals of Internal Medicine, 135(8_Part_2):686–693, 2001.
  * [50] Wojciech Samek, Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, and Klaus-Robert Müller.  Evaluating the visualization of what a deep neural network has learned.  IEEE Transactions on Neural Networks and Learning Systems, 28(11):2660–2673, 2017.
  * [51] David J Sanderson, Ian M Franks, and Digby Elliott.  The effects of targeting on the ground reaction forces during level walking.  Human Movement Science, 12(3):327–337, 1993.
  * [52] Arlene Schmid, Pamela W Duncan, Stephanie Studenski, Sue Min Lai, Lorie Richards, Subashan Perera, and Samuel S Wu.  Improvements in speed-based gait classifications are meaningful.  Stroke, 38(7):2096–2100, 2007.
  * [53] Bernhard Schölkopf, Alexander J Smola, et al.  Learning with kernels: Support vector machines, regularization, optimization, and beyond.  MIT press, 2002.
  * [54] WI Schöllhorn.  Applications of artificial neural nets in clinical biomechanics.  Clinical Biomechanics, 19(9):876–898, 2004.
  * [55] WI Schöllhorn, BM Nigg, DJ Stefanyshyn, and W Liu.  Identification of individual walking patterns using time discrete and time continuous data sets.  Gait & Posture, 15(2):180–186, 2002.
  * [56] Kristof T Schütt, Farhad Arbabzadah, Stefan Chmiela, Klaus R Müller, and Alexandre Tkatchenko.  Quantum-chemical insights from deep tensor neural networks.  Nature Communications, 8:13890, 2017.
  * [57] Jill Seale.  Gait in persons with chronic stroke: An investigation of overall gait and quality of life.  Texas Woman’s University, 2010.
  * [58] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In ICCV, pages 618–626, 2017.
  * [59] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman.  Deep inside convolutional networks: Visualising image classification models and saliency maps.  CoRR, abs/1312.6034, 2013.
  * [60] Sze-Ee Soh, Meg E Morris, and Jennifer L McGinley.  Determinants of health-related quality of life in Parkinson’s disease: A systematic review.  Parkinsonism & Related Disorders, 17(1):1–9, 2011.
  * [61] Oliver Stegle, Leopold Parts, Matias Piipari, John Winn, and Richard Durbin.  Using probabilistic estimation of expression residuals (PEER) to obtain increased power and interpretability of gene expression analyses.  Nature Protocols, 7(3):500, 2012.
  * [62] Stephanie Studenski, Subashan Perera, Kushang Patel, Caterina Rosano, Kimberly Faulkner, Marco Inzitari, Jennifer Brach, Julie Chandler, Peggy Cawthon, Elizabeth Barrett Connor, et al.  Gait speed and survival in older adults.  Jama, 305(1):50–58, 2011.
  * [63] Stephanie Studenski, Subashan Perera, Dennis Wallace, Julie M Chandler, Pamela W Duncan, Earl Rooney, Michael Fox, and Jack M Guralnik.  Physical performance measures in the clinical setting.  Journal of the American Geriatrics Society, 51(3):314–322, 2003\.
  * [64] Irene Sturm, Sebastian Lapuschkin, Wojciech Samek, and Klaus-Robert Müller.  Interpretable deep neural networks for single-trial eeg classification.  Journal of Neuroscience Methods, 274:141–145, 2016.
  * [65] Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A Alemi.  Inception-v4, inception-resnet and the impact of residual connections on learning.  In AAAI, volume 4, page 12, 2017.
  * [66] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions.  In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1–9, 2015.
  * [67] Vyara Valkanova and Klaus P Ebmeier.  What can gait tell us about dementia? Review of epidemiological and neuropsychological evidence.  Gait & Posture, 53:215–223, 2017.
  * [68] Joe Verghese, Roee Holtzer, Richard B Lipton, and Cuiling Wang.  Mobility stress test approach to predicting frailty, disability, and mortality in high-functioning older adults.  Journal of the American Geriatrics Society, 60(10):1901–1905, 2012\.
  * [69] Joe Verghese, Aaron LeValley, Charles B Hall, Mindy J Katz, Anne F Ambrose, and Richard B Lipton.  Epidemiology of gait disorders in community-residing older adults.  Journal of the American Geriatrics Society, 54(2):255–261, 2006\.
  * [70] Joe Verghese, Richard B Lipton, Charles B Hall, Gail Kuslansky, Mindy J Katz, and Herman Buschke.  Abnormality of gait as a predictor of non-Alzheimer’s dementia.  New England Journal of Medicine, 347(22):1761–1768, 2002.
  * [71] Marina M-C Vidovic, Nico Görnitz, Klaus-Robert Müller, Gunnar Rätsch, and Marius Kloft.  Svm2Motif — reconstructing overlapping DNA sequence motifs by mimicking an svm predictor.  PloS ONE, 10(12):e0144782, 2015.
  * [72] Scott C Wearing, Stephen R Urry, and James E Smeathers.  The effect of visual targeting on ground reaction force and temporospatial parameters of gait.  Clinical Biomechanics, 15(8):583–591, 2000.
  * [73] Caleb Wegener, Adrienne E Hunt, Benedicte Vanwanseele, Joshua Burns, and Richard M Smith.  Effect of children’s shoes on gait: A systematic review and meta-analysis.  Journal of Foot and Ankle Research, 4(1):3, 2011.
  * [74] Gavin Williams, Daniel Lai, Anthony Schache, and Meg E Morris.  Classification of gait disorders following traumatic brain injury.  Journal of Head Trauma Rehabilitation, 30(2):E13–E23, 2015.
  * [75] David A Winter.  Kinematic and kinetic patterns in human gait: variability and compensating effects.  Human Movement Science, 3(1-2):51–76, 1984.
  * [76] Sebastian Wolf, Tobias Loose, Matthias Schablowski, Leonhard Döderlein, Rüdiger Rupp, Hans Jürgen Gerner, Georg Bretthauer, and Ralf Mikut.  Automated feature assessment in instrumented gait analysis.  Gait & Posture, 23(3):331–338, 2006.
  * [77] Matthew D Zeiler and Rob Fergus.  Visualizing and understanding convolutional networks.  In Proc. of European Conference on Computer Vision (ECCV), pages 818–833. Springer, 2014.
  * [78] Wei Zeng, Fenglin Liu, Qinghui Wang, Ying Wang, Limin Ma, and Yu Zhang.  Parkinson’s disease classification using gait analysis via deterministic learning.  Neuroscience Letters, 633:268–278, 2016.
  * [79] Jianming Zhang, Zhe Lin, Jonathan Brandt, Xiaohui Shen, and Stan Sclaroff.  Top-down neural attention by excitation backprop.  In European Conference on Computer Vision, pages 543–559. Springer, 2016.
  * [80] Alexander Zien, Nicole Krämer, Sören Sonnenburg, and Gunnar Rätsch.  The feature importance ranking measure.  In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 694–709. Springer, 2009.
  * [81] Luisa M. Zintgraf, Taco S. Cohen, and Max Welling.  A new method to visualize deep neural networks.  CoRR, abs/1603.02518, 2016.

## Acknowledgements

The authors thank all the participating subjects for their time and patience
as well as Christin Rupprecht and Eva Klein for her encouragement and support
during the data collection. No benefits in any form have been received or will
be received from a commercial party related directly or indirectly to the
subject of this article. This work was supported by the Brain Korea 21 Plus
Program through the National Research Foundation of Korea; the Institute for
Information & Communications Technology Promotion (IITP) grant funded by the
Korea government (MSIT) [No. 2017-0-01779]. This work was also supported by
the grant DFG (MU 987/17-1) and by the German Ministry for Education and
Research as Berlin Big Data Centre (BBDC) (01IS14013A) and Berlin Center for
Machine Learning under Grant 01IS18037I. This publication only reflects the
authors views. Funding agencies are not liable for any use that may be made of
the information contained herein. This is a pre-print of an article published
in _Scientific Reports_. The final authenticated version is available online
at: https://doi.org/10.1038/s41598-019-38748-8

## Author Contributions

FH and WIS conceived, designed and performed the experiment. FH, SL, WS, KRM
and WIS analysed the data. SL, WS and KRM contributed analysis tools. FH, SL,
WS, KRM and WIS wrote the paper and drafted the article or revised it
critically for important intellectual content. All authors reviewed the
manuscript.

## Competing Interests

The author(s) declare no competing interests.

## SUPPLEMENTARY MATERIAL

This document contains 3 supplementary tables with results in Tables ST1-ST3,
6 supplementary tables describing neural network architectures in Tables
ST4-ST9 and 4 supplementary subject specific analyses in Figures SF1-SF4.

## Supplementary Results

Model | Ground | Joint Angles | Joint Angles | Joint Angles | Joint Angles
---|---|---|---|---|---
| Reaction | Full-Body | Full-Body | Lower-Body | Lower-Body
| Forces [%] | [%] | (flex.-ext.) [%] | [%] | (flex.-ext.) [%]
Linear (SGD) | 95.4 (1.7) | 100.0 (0.0) | 96.3 (1.8) | 100.0 (0.0) | 91.5 (2.2)
Linear (SVM) | 100.0 (0.0) | 100.0 (0.0) | 99.7 (0.4) | 100.0 (0.0) | 99.8 (0.6)
MLP (2, 64) | 80.7 (3.6) | 100.0 (0.0) | 75.7 (4.7) | 99.9 (0.3) | 55.6 (6.5)
MLP (2, 128) | 89.1 (2.9) | 100.0 (0.0) | 85.7 (4.7) | 99.9 (0.3) | 69.9 (5.7)
MLP (2, 256) | 94.4 (2.3) | 100.0 (0.0) | 92.1 (3.2) | 100.0 (0.0) | 78.6 (4.2)
MLP (2, 512) | 96.6 (0.8) | 100.0 (0.0) | 95.7 (1.2) | 100.0 (0.0) | 87.1 (2.6)
MLP (2, 1024) | 98.6 (1.1) | 100.0 (0.0) | 97.0 (2.0) | 100.0 (0.0) | 91.9 (3.4)
MLP (3, 64) | 88.3 (3.7) | 99.9 (0.3) | 84.3 (3.3) | 99.9 (0.3) | 68.5 (8.3)
MLP (3, 128) | 92.5 (1.9) | 100.0 (0.0) | 90.9 (2.0) | 100.0 (0.0) | 84.2 (4.7)
MLP (3, 256) | 96.6 (0.8) | 100.0 (0.0) | 95.6 (2.6) | 100.0 (0.0) | 89.4 (4.3)
MLP (3, 512) | 98.2 (0.9) | 100.0 (0.0) | 97.0 (1.0) | 100.0 (0.0) | 94.0 (2.4)
MLP (3, 1024) | 98.8 (1.3) | 100.0 (0.0) | 97.8 (1.0) | 100.0 (0.0) | 96.5 (1.2)
CNN–A | 99.1 (0.8) | 100.0 (0.0) | 95.6 (1.7) | 99.9 (0.3) | 92.0 (3.9)
CNN–A3 | 99.6 (0.6) | 99.8 (0.4) | 95.0 (1.7) | 99.9 (0.3) | 94.2 (1.6)
CNN–A6 | 99.0 (0.8) | 99.9 (0.3) | 96.5 (1.3) | 99.8 (0.4) | 97.7 (1.4)
CNN–C3 | 99.2 (0.6) | 100.0 (0.0) | 97.7 (1.5) | 100.0 (0.0) | 97.0 (1.3)
CNN–C3–3 | – | 100.0 (0.0) | – | 99.9 (0.3) | –
CNN–C6 | 99.2 (0.6) | 99.9 (0.3) | 97.2 (1.6) | 100.0 (0.0) | 95.6 (1.8)
Supplemental Material, Table ST1: The classification rates of the subject-classification of different models using artificial neural networks (n=57), reported in pairs of $mean~{}(standard~{}deviation)$ in percent. Model / Noise | Gaussian | Gaussian | Gaussian | Salt- | Pepper | Salt+ | Shot
---|---|---|---|---|---|---|---
| Random | Random | Random | Random | Random | Random | Random
| $\sigma=0.5$ | $\sigma=1$ | $\sigma=2$ |  |  |  |
Linear (SGD) | 40.5 (6.0) | 45.6 (3.0) | 47.6 (1.8) | 46.1 (2.7) | 37.1 (7.0) | 46.4 (3.1) | 46.5 (2.9)
Linear (SVM) | 7.4 (4.6) | 26.2 (5.6) | 39.9 (3.7) | 34.0 (6.9) | 5.0 (3.2) | 35.9 (7.2) | 35.9 (7.5)
MLP (2, 64) | 19.2 (15.9) | 29.3 (13.6) | 38.8 (9.3) | 43.1 (7.3) | 16.6 (15.8) | 42.9 (7.6) | 42.8 (7.7)
MLP (2, 128) | 14.4 (14.2) | 25.9 (13.1) | 36.8 (9.4) | 42.0 (6.9) | 11.7 (13.6) | 41.8 (7.6) | 41.7 (7.7)
MLP (2, 256) | 10.3 (12) | 21.8 (12.4) | 34.9 (9.6) | 40.8 (6.7) | 8.1 (11.1) | 40.7 (7.2) | 40.5 (7.2)
MLP (2, 512) | 7.6 (9.8) | 19.0 (11.5) | 32.5 (9.6) | 39.6 (6.6) | 5.7 (9.0) | 39.4 (7.3) | 39.5 (7.2)
MLP (2, 1024) | 5.9 (8.1) | 17.0 (11.1) | 31.9 (9.2) | 38.8 (6.5) | 4.4 (7.0) | 38.4 (7.2) | 38.6 (7.2)
MLP (3, 64) | 16.2 (14.0) | 27.6 (12.6) | 37.8 (9.0) | 43.5 (7.1) | 14.4 (13.6) | 43.3 (7.4) | 43.3 (7.4)
MLP (3, 128) | 12.5 (12.3) | 24.3 (11.9) | 35.8 (8.9) | 42.7 (6.9) | 10.3 (11.9) | 42.4 (7.4) | 42.5 (7.4)
MLP (3, 256) | 8.9 (9.7) | 20.8 (10.9) | 34.9 (8.4) | 41.7 (6.8) | 7.7 (9.5) | 41.2 (7.4) | 41.1 (7.5)
MLP (3, 512) | 6.9 (8.4) | 18.7 (10.4) | 33.4 (8.2) | 40.8 (6.8) | 5.5 (7.6) | 40.4 (7.1) | 40.4 (7.2)
MLP (3, 1024) | 5.4 (7.0) | 16.7 (9.4) | 32.4 (7.8) | 40.0 (6.6) | 4.3 (6.1) | 39.4 (7.2) | 39.4 (7.2)
CNN–A | 6.1 (6.8) | 18.2 (9.4) | 33.1 (8.0) | 36.9 (6.6) | 8.8 (8.5) | 37.4 (7.5) | 37.4 (7.6)
CNN–A3 | 9.4 (7.7) | 23.4 (9.0) | 36.6 (7.4) | 38.0 (7.4) | 6.6 (6.8) | 39.6 (6.7) | 39.6 (6.7)
CNN–A6 | 8.6 (7.4) | 22.9 (9.3) | 35.9 (7.6) | 38.0 (7.0) | 8.8 (8.3) | 39.3 (7.0) | 39.3 (6.9)
CNN–C3 | 12.2 (8.6) | 27.5 (9.4) | 38.0 (7.0) | 31.0 (8.5) | 11.3 (8.6) | 37.9 (8.2) | 38.1 (8.1)
CNN–C6 | 5.9 (6.5) | 18.3 (9.3) | 33.3 (7.8) | 37.2 (6.6) | 8.5 (8.2) | 37.3 (7.2) | 37.7 (7.2)
Supplemental Material, Table ST2: The area over perturbation curve of the subject-classification of ground reaction forces of different models using artificial neural networks for different noise perturbation runs (n=57). AOPC values are reported in pairs of $mean~{}(standard~{}deviation)$. Gaussian noise types add white noise additively to the input samples. Salt- and Salt+ noise types set input variables to $-1$ and $1$ respectively. Pepper noise replaces input variables with $0$. Shot noise replaces input variables randomly chosen from the set $\\{-1,0,1\\}$.  Model | Ground | Joint Angles | Joint Angles | Joint Angles | Joint Angles
---|---|---|---|---|---
| Reaction | Full-Body | Full-Body | Lower-Body | Lower-Body
| Forces |  | (flex.-ext.) |  | (flex.-ext.)
Linear (SGD) | 4.31 (0.25) | 4.27 (0.08) | 3.93 (0.12) | 4.16 (0.10) | 3.86 (0.15)
Linear (SVM) | 0.31 (0.08) | 0.56 (0.10) | 0.31 (0.07) | 0.48 (0.09) | 0.26 (0.05)
MLP (2, 64) | 1.61 (0.56) | 2.47 (0.12) | 1.96 (0.29) | 2.11 (0.13) | 1.89 (0.38)
MLP (2, 128) | 1.38 (0.46) | 2.27 (0.10) | 1.59 (0.25) | 1.91 (0.11) | 1.48 (0.32)
MLP (2, 256) | 1.12 (0.34) | 2.09 (0.09) | 1.36 (0.21) | 1.74 (0.11) | 1.21 (0.27)
MLP (2, 512) | 0.95 (0.28) | 1.93 (0.08) | 1.10 (0.15) | 1.60 (0.10) | 0.96 (0.21)
MLP (2, 1024) | 0.83 (0.21) | 1.80 (0.08) | 0.94 (0.12) | 1.48 (0.09) | 0.83 (0.18)
MLP (3, 64) | 1.31 (0.34) | 2.73 (0.15) | 1.58 (0.17) | 2.37 (0.16) | 1.41 (0.23)
MLP (3, 128) | 1.00 (0.23) | 2.44 (0.14) | 1.28 (0.13) | 2.05 (0.14) | 1.02 (0.17)
MLP (3, 256) | 0.84 (0.18) | 2.21 (0.13) | 1.05 (0.10) | 1.86 (0.13) | 0.85 (0.12)
MLP (3, 512) | 0.72 (0.15) | 2.02 (0.11) | 0.90 (0.08) | 1.66 (0.11) | 0.70 (0.10)
MLP (3, 1024) | 0.63 (0.12) | 1.85 (0.10) | 0.77 (0.08) | 1.50 (0.10) | 0.61 (0.09)
CNN–A | 0.30 (0.08) | 0.56 (0.09) | 0.35 (0.06) | 0.49 (0.08) | 0.32 (0.05)
CNN–A3 | 0.37 (0.10) | 0.64 (0.11) | 0.37 (0.08) | 0.55 (0.09) | 0.35 (0.06)
CNN–A6 | 0.32 (0.09) | 0.64 (0.12) | 0.38 (0.07) | 0.58 (0.11) | 0.27 (0.05)
CNN–C3 | 0.35 (0.08) | 0.50 (0.08) | 0.43 (0.08) | 0.48 (0.08) | 0.44 (0.08)
CNN–C3–3 | – | 2.74 (0.10) | – | 2.32 (0.13) | –
CNN–C6 | 0.60 (0.12) | 0.59 (0.08) | 0.45 (0.08) | 0.62 (0.11) | 0.27 (0.05)
Supplemental Material, Table ST3: The coefficient of variation of the subject-
classification of different models using artificial neural networks (n=57).
Values are reported in pairs of $mean~{}(standard~{}deviation)$.

## Neural Network Architectures

Complete description of all evaluated artificial neural network architectures.
The complete set of features and their specific input dimensionalities
$D=C\times T$ ($T$ = time points, $C$ = number of channels) for which results
are reported in this study are:

  * •

Ground Reaction Forces (GRF): $D=6\times 101$

  * •

Full-Body Joint Angles (FBJA): $D=33\times 101$

  * •

Full-Body Joint Angles (flexion-extension) (FBJAX): $D=10\times 101$

  * •

Lower-Body Joint Angles (LBJA): $D=18\times 101$

  * •

Lower-Body Joint Angles (flexion-extension) (LBJAX): $D=6\times 101$

All multi-layer perceptron models (MLP ($\cdots$)) and Linear models are –
except for the number of input and hidden neurons – uniform in structure for
all evaluated feature sets. Those models also receive the input as flattened
arrays (vectorization by row concatenation) shaped $T\cdot C$. with $H$ hidden
units per layer, of which the number is uniform across all layers of the
model. The variable $L$ describes the number of output labels. The MLP and
Linear architectures are summarized in Table ST4.

Model | Layer 1 | Layer 2 | Layer 3 | Layer 4 | Layer 5 | Layer 6
---|---|---|---|---|---|---
Linear (SGD) | Dense (D,L) |  |  |  |  |
MLP (2, H) | Dense (D,H) | ReLU | Dense (H,L) | SoftMax |  |
MLP (3, H) | Dense (D,H) | ReLU | Dense (H,H) | ReLU | Dense (H,L) | SoftMax
Supplemental Material, Table ST4: Model architectures for the linear (Linear
(SGD)) and fully-connected (MLP ($\cdots$)) artificial neural networks.)

Where Dense ($N$,$M$) describes a linear / fully-connected / dense layer with
$N$ input neurons and $M$ output neurons.

Due to construction, the output shape of a convolutional layer is determined
via an interplay of layer parameters (filter size and stride parameters) and
the input shapes. Since above features are characterized in a varying number
of data channels, the convolutional neural network (CNN) architectures have
been adapted accordingly. The following tables (Table ST5 \- Table ST9) will
describe the architectures of the evaluated models in details, where
convolutional layers will be described as Conv($Fc$, $Ft$|$Sc$, $St|H$), where
$Fc$ and $Ft$ are the channel axis span and a time axis span of the learned
convolutional filter banks respectively, $Sc$ and $St$ are the channel axis
stride and time axis stride and $H$ the number of learned filters ( = output
depth).

The CNN models can be categorized in the following groups:

  * •

CNN-A: Models which use one layer of convolutions and have square input filter
sizes configured to read all channels at once and use the same scope to read
from the time axis, i.e. $Fc$ = $Ft$ = $C$.

  * •

CNN-A3: These models read all input channels at once ($Fc=C$), but the scope
for reading the time axis is limited to $Ft=6$. Two layers of convolutions.

  * •

CNN-A6: Same as CNN-A3, with the difference of $Ft=3$ for the input layer.

  * •

CNN-C3: Uses convolution filters shaped ($Fc$, $Ft$) = ($3$,$3$)

  * •

CNN-C3-3: Uses convolution filters shaped ($Fc$, $Ft$) = ($3$,$3$) and a
stride pattern of ($Sc$, $St$) = ($3$,$3$), i.e. there is no overlap between
locations where convolutional filters are applied. This model receives inputs
padded with one additional columns of zeros, to extend the time axis to 102
entries.

  * •

CNN-C6: Uses convolution filters shaped ($Fc$, $Ft$) = ($6$,$6$)

Model | Layer 1 | Layer 2 | Layer 3 | Layer 4 | Layer 5 | Layer 6 | Layer 7 | Layer 8 | Layer 9
---|---|---|---|---|---|---|---|---|---
CNN–A | Conv | ReLU | Flatten | Dense | SoftMax |  |  |  |
| (6,6|1,1|32) |  |  | (3072,L) |  |  |  |  |
CNN–A3 | Conv | ReLU | Conv | ReLU | Flatten | Dense | SoftMax |  |
| (6,3|1,1|32) |  | (1,3|1,1|32) |  |  | (3104,L) |  |  |
CNN–A6 | Conv | ReLU | Conv | ReLU | Flatten | Dense | SoftMax |  |
| (6,6|1,1|32) |  | (1,6|1,1|32) |  |  | (2912,L) |  |  |
CNN–C3 | Conv | ReLU | Conv | ReLU | Conv | ReLU | Flatten | Dense | SoftMax
| (3,3|1,1|32) |  | (3,3|1,1|32) |  | (2,2|1,1|32) |  |  | (3072,L) |
CNN–C3–3 | – | – | – | – | – | – | – | – | –
CNN–C6 | Conv | ReLU | Flatten | Dense | SoftMax |  |  |  |
| (6,6|1,1|32) |  |  | (3072,L) |  |  |  |  |
Supplemental Material, Table ST5: Model architectures for the convolutional neural network architectures for ground reaction force variables (GRF). Model | Layer 1 | Layer 2 | Layer 3 | Layer 4 | Layer 5 | Layer 6 | Layer 7 | Layer 8 | Layer 9
---|---|---|---|---|---|---|---|---|---
CNN–A | Conv | ReLU | Flatten | Dense | SoftMax |  |  |  |
| (33,33|1,1|64) |  |  | (4416,L) |  |  |  |  |
CNN–A3 | Conv | ReLU | Conv | ReLU | Flatten | Dense | SoftMax |  |
| (33,3|1,1|64) |  | (1,3|1,1|32) |  |  | (3104,L) |  |  |
CNN–A6 | Conv | ReLU | Conv | ReLU | Flatten | Dense | SoftMax |  |
| (33,6|1,1|32) |  | (1,6|1,1|32) |  |  | (2912,L) |  |  |
CNN–C3 | Conv | ReLU | Conv | ReLU | Conv | ReLU | Flatten | Dense | SoftMax
| (3,3|1,1|32) |  | (3,3|1,1|32) |  | (3,3|1,1|16) |  |  | (41040,L) |
CNN–C3–3 | Conv | ReLU | Conv | ReLU | Conv | ReLU | Flatten | Dense | SoftMax
| (3,3|3,3|64) |  | (3,3|1,1|64) |  | (3,3|1,1|32) |  |  | (6720,L) |
CNN–C6 | Conv | ReLU | Conv | ReLU | Conv | ReLU | Flatten | Dense | SoftMax
| (6,6|1,1|32) |  | (6,6|1,1|32) |  | (6,6|1,1|16) |  |  | (24768,L) |
Supplemental Material, Table ST6: Model architectures for the convolutional neural network architectures for full-body joint angles (FBJA). Model | Layer 1 | Layer 2 | Layer 3 | Layer 4 | Layer 5 | Layer 6 | Layer 7 | Layer 8 | Layer 9
---|---|---|---|---|---|---|---|---|---
CNN–A | Conv | ReLU | Flatten | Dense | SoftMax |  |  |  |
| (10,10|1,1|32) |  |  | (2944,L) |  |  |  |  |
CNN–A3 | Conv | ReLU | Conv | ReLU | Flatten | Dense | SoftMax |  |
| (10,3|1,1|32) |  | (1,3|1,1|32) |  |  | (3104,L) |  |  |
CNN–A6 | Conv | ReLU | Conv | ReLU | Flatten | Dense | SoftMax |  |
| (10,6|1,1|32) |  | (1,6|1,1|32) |  |  | (2912,L) |  |  |
CNN–C3 | Conv | ReLU | Conv | ReLU | Conv | ReLU | Flatten | Dense | SoftMax
| (3,3|1,1|32) |  | (3,3|1,1|32) |  | (3,3|1,1|32) |  |  | (12160,L) |
CNN–C3–3 | – | – | – | – | – | – | – | – | –
CNN–C6 | Conv | ReLU | Conv | ReLU | Conv | ReLU | Flatten | Dense | SoftMax
| (6,6|1,1|32) |  | (3,3|1,1|32) |  | (3,3|1,1|32) |  |  | Dense (2944,L) |
Supplemental Material, Table ST7: Model architectures for the convolutional neural network architectures for full-body joint angles in the sagittal plane (flexion-extension) (FBJAX). Model | Layer 1 | Layer 2 | Layer 3 | Layer 4 | Layer 5 | Layer 6 | Layer 7 | Layer 8 | Layer 9
---|---|---|---|---|---|---|---|---|---
CNN–A | Conv | ReLU | Flatten | Dense | SoftMax |  |  |  |
| (18,18|1,1|64) |  |  | (5376,L) |  |  |  |  |
CNN–A3 | Conv | ReLU | Conv | ReLU | Flatten | Dense | SoftMax |  |
| (18,3|1,1|64) |  | (1,3|1,1|32) |  |  | (3104,L) |  |  |
CNN–A6 | Conv | ReLU | Conv | ReLU | Flatten | Dense | SoftMax |  |
| (18,6|1,1|64) |  | (1,6|1,1|32) |  |  | (2912,L) |  |  |
CNN–C3 | Conv | ReLU | Conv | ReLU | Conv | ReLU | Flatten | Dense | SoftMax
| (3,3|1,1|32) |  | (3,3|1,1|32) |  | (3,3|1,1|16) |  |  | (18240,L) |
CNN–C3–3 | Conv | ReLU | Conv | ReLU | Conv | ReLU | Flatten | Dense | SoftMax
| (3,3|3,3|64) |  | (3,3|1,1|64) |  | (3,3|1,1|32) |  |  | (1920,L) |
CNN–C6 | Conv | ReLU | Conv | ReLU | Conv | ReLU | Flatten | Dense | SoftMax
| (6,6|1,1|32) |  | (6,6|1,1|32) |  | (6,6|1,1|16) |  |  | (4128,L) |
Supplemental Material, Table ST8: Model architectures for the convolutional neural network architectures for lower-body joint angles (LBJA). Model | Layer 1 | Layer 2 | Layer 3 | Layer 4 | Layer 5 | Layer 6 | Layer 7 | Layer 8 | Layer 9
---|---|---|---|---|---|---|---|---|---
CNN–A | Conv | ReLU | Flatten | Dense | SoftMax |  |  |  |
| (6,6|1,1|32) |  |  | (3072,L) |  |  |  |  |
CNN–A3 | Conv | ReLU | Conv | ReLU | Flatten | Dense | SoftMax |  |
| (6,3|1,1|32) |  | (1,3|1,1|32) |  |  | (3104,L) |  |  |
CNN–A6 | Conv | ReLU | Conv | ReLU | Flatten | Dense | SoftMax |  |
| (6,6|1,1|32) |  | (1,6|1,1|32) |  |  | (2912,L) |  |  |
CNN–C3 | Conv | ReLU | Conv | ReLU | Conv | ReLU | Flatten | Dense | SoftMax
| (3,3|1,1|32) |  | (3,3|1,1|32) |  | (2,2|1,1|32) |  |  | (3072,L) |
CNN–C3–3 | – | – | – | – | – | – | – | – | –
CNN–C6 | Conv | ReLU | Flatten | Dense | SoftMax |  |  |  |
| (6,6|1,1|32) |  |  | (3072,L) |  |  |  |  |
Supplemental Material, Table ST9: Model architectures for the convolutional
neural network architectures for lower-body joint angles in the sagittal plane
(flexion-extension) (LBJAX).

## Supplementary Subject Specific Analyses

Supplemental Material, Figure SF1: Subject 21: GRF and JLBAX inputs as line
plots _(top)_. The corresponding relevance maps _(bottom)_ for the CNN-A,
following the procedure outlined in Figure 1 in the paper. _Left:_ Ground
reaction force (GRF). The input relevance values point out that the model has
learned to identify subject 21 by the medial-lateral shear force during the
initial contact of the stance phase. The GRF input balues reveal that the
medial-lateral shear forces during the initial contact of the stance phase are
among the highest in the sample for both foot contacts of subject 21. _Right:_
Lower-body joint angles in the sagittal plane (flexion-extension) (LBJAX). It
can be observed from the input relevance values that the extension of the
ankle is unique to subject 21 during the terminal stance phase of the right
foot, which is identified by the model as an identifying characteristic. For
this subject, the model achieves TP rates of $100\%$ for LBJAX and $95.23\%$
for GRF.

Supplemental Material, Figure SF2: Subject 50: GRF and JLBAX inputs as line
plots _(top)_. The corresponding relevance maps _(bottom)_ for the CNN-A,
following the procedure outlined in Figure 1 in the paper. _Left:_ Ground
reaction force (GRF). The highest input relevance values are appearning for
the vertical GRF during the mid stance phase of the right and left foot
contact. This indicates that the time window of features, where the leg is
located directly over the foot and the other leg is in the middle of the swing
phase, is unique to subject 50. _Right:_ Lower-body joint angles in the
sagittal plane (flexion-extension) (LBJAX).. From the input relevance values,
we can observe that the extension of the ankle during the terminal stance
phase of the left leg as well as the flexion of the hip joint during the
terminal stance phase and the mid swing phase of the right leg is unique to
subject 50. For this subject, the model achieves TP rates of $100\%$ for GRF
but only $80\%$ for LBJAX, for which it consistently mispredicts as subject 4.
The model’s uncertainty is reflected in the comparatively noisy and piece-wise
negative relevance values. Relevance feedback such as for subject 50 and LBJAX
data, providing detailed insight into the reasoning behind the model’s
uncertain prediction, yields information critical for the practical
applicability of gait analysis tools based on machine learning in a clinical
setting.

Supplemental Material, Figure SF3: Subject 54: GRF and JLBAX inputs as line
plots _(top)_. The corresponding relevance maps _(bottom)_ for the CNN-A,
following the procedure outlined in Figure 1 in the paper. _Left:_ Ground
reaction force (GRF). The input relevance values point out that the model has
learned to identify subject 54 based the vertical GRF during the terminal
stance phase of the right and left leg. _Right:_ Lower-body joint angles in
the sagittal plane (flexion-extension) (LBJAX). The LBJAX supports the finding
about the unique nature of the terminal stance phase to subject 54, showing
the highest input relevance values for the extension of the ankle during this
time window of the gait cycle. For this subject, the model achieves TP rates
of $95\%$ for LBJA and $100\%$ for GRF.

Supplemental Material, Figure SF4: Subject 56: GRF and JLBAX inputs as line
plots _(top)_. The corresponding relevance maps _(bottom)_ for the CNN-A,
following the procedure outlined in Figure 1 in the paper. _Left:_ Ground
reaction force (GRF). From the input relevance values, we can observe that the
vertical GRF during the initial and terminal stance phase of the right and
left leg is unique to subject 56. _Right:_ Lower-body joint angles in the
sagittal plane (flexion-extension) (LBJAX). From the input relevance values,
we can observe that the extension of the ankle during the terminal stance
phase of the right is unique to subject 56. For this subject, the model
achieves TP rates of $100\%$ for both LBJAX and GRF.