# Using AI to Measure Parkinson’s Disease Severity at Home

Md Saiful Islam1, *, Wasifur Rahman1, Abdelrahman Abdelkader1, Phillip T.
Yang2, Sangwu Lee1, Jamie L. Adams2, Ruth B. Schneider2, E. Ray Dorsey2, Ehsan
Hoque1

###### Abstract

We present an artificial intelligence system to remotely assess the motor
performance of individuals with Parkinson’s disease (PD). Participants
performed a motor task (i.e., tapping fingers) in front of a webcam, and data
from 250 global participants were rated by three expert neurologists following
the Movement Disorder Society Unified Parkinson’s Disease Rating Scale (MDS-
UPDRS). The neurologists’ ratings were highly reliable, with an intra-class
correlation coefficient (ICC) of 0.88. We developed computer algorithms to
obtain objective measurements that align with the MDS-UPDRS guideline and are
strongly correlated with the neurologists’ ratings. Our machine learning model
trained on these measures outperformed an MDS-UPDRS certified rater, with a
mean absolute error (MAE) of 0.59 compared to the rater’s MAE of 0.79.
However, the model performed slightly worse than the expert neurologists (0.53
MAE). The methodology can be replicated for similar motor tasks, providing the
possibility of evaluating individuals with PD and other movement disorders
remotely, objectively, and in areas with limited access to neurological care.

##  1 Introduction

Parkinson’s disease (PD) is the fastest-growing neurological disease, and
currently, it has no cure. Regular clinical assessments and medication
adjustments can help manage the symptoms and improve the quality of life.
Unfortunately, access to neurological care is limited, and many individuals
with PD do not receive proper treatment or diagnosis. For example, in the
United States, an estimated 40% of individuals aged 65 or older living with PD
do not receive care from a neurologist (Willis et al. 2011). Access to care is
much scarce in developing and underdeveloped regions, where there may be only
one neurologist per millions of people (Kissani et al. 2022). Even for those
with access to care, arranging clinical visits can be challenging, especially
for older individuals living in rural areas with cognitive and driving
impairments.

The finger-tapping task 111The finger-tapping task requires an individual to
repeatedly tap their thumb finger with their index finger as fast and as big
as possible. is commonly used in neurological exams to evaluate bradykinesia
(i.e., slowing of movement) in upper extremities, which is a key symptom of PD
(Hughes et al. 1992). Videos of finger-tapping tasks have been used to analyze
movement disorders like PD in prior research. However, the videos are often
collected from a few participants ($<20$) (Khan et al. 2014), or the studies
only provide binary classification (e.g., slight vs. severe Parkinsonian
symptoms; Parkinsonism vs. non-Parkinsonism) and do not measure PD severity
(Williams et al. 2020; Nunes et al. 2022). Additionally, existing models lack
interpretability, making it difficult to use them in clinical settings. Most
importantly, the videos are noise-free as they are recorded in a clinical
setting with the guidance of experts. Machine learning models trained on clean
data may not perform effectively if the task is recorded in a noisy home
environment due to the models’ susceptibility to data shift (Quinonero-Candela
et al. 2008). Consequently, these models may not enhance access to care for
Parkinson’s disease.

Imagine anyone from anywhere in the world could perform a motor task (i.e.,
finger-tapping) using a computer webcam and get an automated assessment of
their motor performance severity. This presents several challenges: collecting
a large amount of data in the home environment, developing interpretable
computational features that can be used as digital biomarkers to track the
severity of motor functions, and developing a platform where (elderly) people
can complete the tasks without direct supervision. In this paper, we address
these challenges by leveraging AI-driven techniques to derive interpretable
metrics related to motor performance severity and apply them across 250 global
participants performing the task mostly from home. Three experts222All the
experts are US neurologists with at least five years of experience in PD
clinical studies and actively consult PD patients. and a non-expert333The non-
expert is a certified rater of PD clinical studies with a degree in medicine
from a non-US institution but does not actively consult PD patients. rated the
severity of motor performance watching these videos, using the Movement
Disorder Society Unified Parkinson’s Disease Rating Scale (MDS-UPDRS). Our
proposed interpretable, clinically relevant features highly correlate with the
experts’ ratings. An AI-based model was trained on these features to assess
the severity score automatically, and we compared its performance against both
expert and non-expert clinicians. Figure 1 presents an illustrative overview
of our system.

Figure 1: Overview of the AI-based system for assessing the severity of motor
performance. Anyone can perform the finger-tapping task in front of a computer
webcam. The system employs a hand-tracking model to locate the key points of
the hand, enabling a continuous tracking of the finger-tapping angle incident
by the thumb finger-tip, the wrist, and the index finger-tip. After reducing
noise from the time-series data of this angle, the system computes several
objective features associated with motor function severity. The AI-based model
then utilizes these features to assess the severity score automatically.
Figure 2: Data collection. The participants, both those with Parkinson’s
disease (PD) and healthy controls, performed the task primarily in a noisy
home environment without any clinical supervision. The dataset includes blurry
videos caused by poor internet connection, videos where participants had
difficulty following instructions, and videos with overexposed or underexposed
backgrounds. These issues are common when collecting data from home,
particularly from an aged population that may be less familiar with technology
than other age groups.

##  2 Results

### Data

We obtained data from 250 global participants (172 with PD, 78 control) who
completed a finger-tapping task with both hands (see Figure 2 for examples).
Participants used a web-based tool (Langevin et al. 2019) to record themselves
with a webcam primarily from their homes. Demographic information for the
participants is presented in Table 1.

Characteristics | With PD | Without PD | Total
---|---|---|---
| Number of
---
Participants,
n
172 | 78 | 250
Sex, n (%)
| Male | 109 (63.4%) | 28 (35.9%) | 137 (54.8%)
| Female | 63 (36.6%) | 50 (64.1%) | 113 (45.2%)
Age in years (range: 18 - 86, mean: 62.13), n (%)
| <20 | 0 (0.0%) | 3 (3.8%) | 3 (1.2%)
| 20-29 | 0 (0.0%) | 10 (12.8%) | 10 (4.0%)
| 30-39 | 1 (0.6%) | 3 (3.8%) | 4 (1.6%)
| 40-49 | 5 (2.9%) | 6 (7.7%) | 11 (4.4%)
| 50-59 | 34 (19.8%) | 14 (18.0%) | 48 (19.2%)
| 60-69 | 64 (37.2%) | 30 (38.5%) | 94 (37.6%)
| 70-79 | 62 (36.0%) | 12 (15.4%) | 74 (29.6%)
| >=80 | 6 (3.5%) | 0 (0.0%) | 6 (2.4%)
Race, n (%)
| White | 161 (93.6%) | 69 (88.5%) | 230 (92.0%)
| Asian | 2 (1.2%) | 5 (6.4%) | 7 (2.8%)
|  | Black or
---
African
American
1 (0.6%) | 2 (2.6%) | 3 (1.2%)
|  | American
---
Indian or
Alaska
Native
2 (1.2%) | 0 (0.0%) | 2 (0.8%)
| Others | 1 (0.6%) | 0 (0.0%) | 1 (0.4%)
| No mention | 5 (2.9%) | 2 (2.6%) | 7 (2.8%)

Table 1: Demographic characteristics of the participants.

Following the MDS-UPDRS guidelines, we consider each participant’s left and
right-hand finger-tapping as two separate videos. All these $250\times 2=500$
videos are rated by three expert neurologists with extensive experience
providing care to individuals with PD and leading PD research studies.
However, after undertaking manual and automated quality assessments, we
removed 11 videos from the dataset. Ultimately, we had 489 videos for analysis
(244 videos for the left hand and 245 for the right hand). We obtained the
ground truth severity score a) by majority agreement when at least two experts
agreed on their ratings (451 cases), or b) by taking the average of three
ratings and rounding it to the nearest integer when no majority agreement was
found (38 cases).

### Rater Agreement

The three expert neurologists demonstrated good agreement on their ratings, as
measured by a) Krippendorff’s alpha score of 0.69 and b) Intra-class
correlation coefficient (ICC) score of 0.88 (95% confidence interval: [0.86,
0.90]). Figure 3 provides an overview of pair-wise agreement between expert
raters. All three experts agreed in 30.7% of the videos, and at least two
agreed in 93% videos. The three raters showed a difference of less than 1
point from the ground truth in 99.2%, 99.5%, and 98.2% of the cases,
respectively. These metrics suggest that the experts can reliably rate our
videos recorded from home environments.

Figure 3: An overview of how the experts and the non-expert agreed on their
ratings. Green dots indicate two raters having a perfect agreement, while
grey, orange, and red dots imply a difference of 1, 2, and 3 points,
respectively. We did not observe any 4 points rating difference. The high
density of green and gray dots and an ICC score of 0.88 verifies that the
experts demonstrated high inter-rater agreement among themselves, and the
finger-tapping task can be reliably rated when recorded from home. However,
the non-expert was less reliable than the experts, demonstrating moderate
agreement with the three expert raters (the ICC of the non-expert’s ratings
and the ratings from the three experts were 0.66, 0.70, and 0.65,
respectively.)

### Features as Digital Biomarkers

We quantify 42 features measuring several aspects of the finger-tapping task,
including speed, amplitude, hesitations, slowing, and rhythm. We also quantify
how much an individual’s wrist moves using 18 features. For each feature,
Pearson’s correlation coefficient ($r$) is measured to see how the feature is
correlated with the ground truth severity score, along with a statistical
significance test (significance level $\alpha=0.01$). We found that 22
features were significantly correlated with the severity scores, which
reflects their promise for use as digital biomarkers of symptom progression.
Table 2 shows the top 10 features with the highest correlation. These features
are clinically meaningful as they capture several aspects of speed, amplitude,
and rhythm (i.e., regularity) of the finger-tapping task, which are focused on
the MDS-UPDRS guideline for scoring PD severity.

Feature | Statistic | r | p-value | Rank
---|---|---|---|---
speed | inter-quartile range | -0.56 | $10^{-33}$ | 1
median | -0.52 | $10^{-27}$ | 2
maximum | -0.32 | $10^{-10}$ | 6
amplitude | median | -0.50 | $10^{-25}$ | 3
maximum | -0.41 | $10^{-17}$ | 4
frequency | inter-quartile range | 0.32 | $10^{-10}$ | 5
standard deviation | 0.29 | $10^{-8}$ | 8
period | entropy (i.e., irregularity) | 0.32 | $10^{-10}$ | 7
| variance (normalized
---
by the average period)
0.28 | $10^{-8}$ | 9
inter-quartile range | 0.27 | $10^{-7}$ | 10

Table 2: Features most correlated with the ground truth severity scores for
the finger-tapping task. r and p-value indicate Pearson’s correlation
coefficient and significance level, respectively.

Traditionally, human evaluators cannot constantly measure the finger-tapping
speed. Instead, they count the number of taps the participant has completed
within a specific time (e.g., three taps per second). However, in our case,
the videos are collected at 30 frames per second rate, thus allowing us to
track the fingertips 30 times per second and develop a continuous measure of
speed. The former approach, the number of finger taps completed in unit time,
is termed as “frequency” throughout the paper, and “speed” (and
“acceleration”) refers to the continuous measure (i.e., movement per frame).
Similarly, “period” refers to the time it takes to complete a tap, and thus,
is a discrete measure. In addition, finger tapping amplitude is measured by
the maximum distance between the thumb and index-finger tips during each tap.
Since linear distance can vary depending on how far the participant is sitting
from the camera, we approximate amplitude using the maximum angle incident by
three key points: the thumb-tip, the wrist, and the index fingertip. As we see
in Table 2, several statistical measures of continuous speed are significantly
correlated with PD severity. These granular computations are only attainable
using automated video analysis, which, to our knowledge, was missing in prior
literature.

### Performance of a Non-Expert Clinician

(a)

(b)

(c)

(d)

Figure 4: Model performance. (a) We observe good agreement between the
predicted severity and the ground truth scores. Green dots indicate correct
predictions, while grey and orange dots imply a difference of 1 and 2 points
between the predicted and actual scores. We did not observe any 3 points
rating difference. (b) The confusion matrix presents the agreement
numerically. (c) The mean absolute error (MAE) measures the difference between
two ratings. The model incurs slightly higher MAE than an average expert but
substantially lower MAE than the non-expert. (d) Pearson correlation
coefficient (PCC) measures the correlation between two sets of ratings. The
model’s predicted severity ratings are more correlated with the ground truth
scores than the non-expert’s (higher PCC) but less correlated than an average
expert’s (lower PCC) ratings.

When an individual lacks access to a neurologist with expertise in movement
disorders, they may consult with a non-specialist clinician. Thus, it is
critical to assess how a clinician with limited expertise in movement
disorders or Parkinson’s disease may perform compared to experts in this
field. To this end, we recruited an investigator who had completed an MBBS
444MBBS: Bachelor of Medicine, Bachelor of Surgery but not additional medical
training (e.g., residency), was certified to administer the MDS-UPDRS, and has
the experience of rating the severity of PD symptoms in multiple research
studies. We asked the non-expert clinician to rate the same videos that the
three experts had rated. We observed moderate reliability in their ratings,
with an intra-class correlation coefficient (ICC) of 0.70 compared to the
ground truth scores (Figure 3). On average, the non-expert’s ratings deviated
from the ground truth severity score by 0.79 points, and the Pearson’s
correlation coefficient between the non-expert’s ratings and the ground truth
severity scores was 0.55.

### Modeling the Severity Rating

We employed a machine learning model (i.e., XGBoost regressor (Chen and
Guestrin 2016)) that predicts the severity of PD symptoms based on the
extracted features from a finger-tapping video. The severity rating predicted
by the model is a continuous value, ranging from 0 to 3. We measure the mean
absolute error (MAE) and Pearson’s correlation coefficient (PCC) between the
model’s predictions and the ground truth severity scores for every video in
the test set. The continuous prediction is converted to four severity classes
(0, 1, 2, and 3) by rounding it to the closest integer. The classification
result is reported in Figure 4 (a, b), showing that the model’s predictions
largely agree with the ground truth severity scores. The model’s reliability
in rating the videos is moderate, as indicated by an ICC score of 0.74 (95%
C.I.: [0.67, 0.79]). On average, the model predictions deviated from the
ground truth severity scores by 0.59 points, and the Pearson’s correlation
coefficient between the predictions and the ground truth severity scores was
0.65. Since the ground truth scores are derived from the three experts’
ratings, it is natural to find an excellent correlation (PCC = 0.86) and a
minimal difference (MAE = 0.27) between an average expert and the ground
truth. However, this is an unfair baseline to compare the performance of the
model and the non-expert. Instead, we looked at how the expert neurologists
agreed with each other to establish a human-level performance of the rating
task. On average, any pair of experts differed by 0.53 points from each
other’s ratings, and their ratings were correlated with PCC = 0.72. In all the
metrics we tested, the XGBoost regression model outperformed the non-expert
clinician but _was outperformed_ by the experts (Figure 4 (c, d)).

### Interpretability of Model Predictions

We use SHapley Additive exPlanations (SHAP) to interpret the outputs of the
machine learning model. SHAP values provide a way to attribute a prediction to
different features and quantify the impact of each feature on the model output
(Lundberg et al. 2020). This information can be useful for understanding how a
model makes decisions and for identifying which features are most important
for making accurate predictions. We find that important features identified by
SHAP align well with our previously identified significant features (see Table
2). Specifically, SHAP identifies the inter-quartile range (IQR) of finger-
tapping speed as the most important feature driving the model’s predictions,
which is the most correlated feature with the ground-truth severity score.
Top-10 most important features include finger-tapping speed (IQR, median),
amplitude (maximum, median), period (IQR, variance, minimum), frequency (IQR),
and maximum freezing duration, which are all significantly correlated with the
ground-truth severity score (at a significance level, $\alpha=0.01$). The only
top-10 feature that does not correlate significantly with the severity score
is the decrement of amplitude (amplitude at the end - mean amplitude), which
is also underscored in the MDS-UPDRS guideline. These results indicate that
the model is looking at the right features while deciding the finger-tapping
severity scores from the recorded videos, further suggesting the model’s
reliability.

### Analyzing Bias

To better evaluate the performance of our model across various demographic
groups, we conduct a group-wise error analysis that takes into account factors
such as gender, age, and Parkinson’s disease diagnosis status. This approach
allows us to assess any potential biases or inaccuracies in our model’s
predictions and make necessary improvements to ensure equitable and accurate
results for all users. We combined the model predictions of the samples
present only in the test fold across 20 iterations. Additionally, we tracked
the demographic attributes associated with each sample, allowing us to
evaluate the performance of our model across various demographic groups.

Our model achieved a mean absolute error (MAE) of 0.59 for male subjects
($n=176$) and 0.54 for female subjects ($n=155$), indicating relatively
accurate predictions for both genders. Furthermore, we conducted statistical
tests to compare the errors across the two groups and found no significant
difference (p-value = 0.43). We also performed a similar error analysis on our
model’s predictions for subjects with PD ($n=226$) and those without PD
($n=105$). The model had an MAE of 0.54 for subjects with PD and 0.61 for
those without PD. However, we found no significant difference in the errors
between these two groups (p-value = 0.31). These results suggest that our
model does not exhibit gender bias and performs similarly for PD and non-PD
groups. Age had a slight negative correlation with the error of model
predictions (Pearson’s correlation coefficient, $r=-0.09$). However, the
correlation was not statistically significant at $\alpha=0.05$ significance
level ($p$-value = $0.12$). This suggests that the performance of the model is
not significantly different across the younger and older populations.

##  3 Discussion

This paper makes three significant contributions. First, it demonstrates that
the finger-tapping task can be reliably assessed by neurologists from remotely
recorded videos. Second, it suggests that AI-driven models can perform close
to clinicians and often better than non-specialists in assessing the finger-
tapping task. Third, the model is equitable across gender, age, and PD vs.
non-PD groups. These offer new opportunities for utilizing AI to address
movement disorders, extending beyond Parkinson’s disease to encompass other
conditions like ataxia and Huntington’s disease, where finger-tapping provides
valuable insights into the severity of the disease.

Our tool can be expanded to enable longitudinal tracking of symptom
progression to fine-tune the treatment of PD. People with PD (PwP) often
exhibit episodic symptoms, and longitudinal studies require careful management
of variables to ensure accurate temporal responses to individual doses of
medication. It is best practice to conduct repeated ratings under consistent
conditions, such as at the same time of day, the same duration after the last
medication dose, and with the same rater (Perlmutter 2009). However, the
limited availability of neurological care providers and mobility constraints
of elderly PwP make this challenging. In the future, we envision extending our
platform for other neurological tasks (e.g., postural and rest tremors,
speech, facial expression, gait, etc.) so that patients can perform an
extensive suite of neurological tasks in their suitable schedule and from the
comfort of their homes. For this use case, our tool is not intended to replace
clinical visits for individuals who have access to them. Instead, the tool can
be used frequently between clinical visits to track the progression of PD,
augment the neurologists’ capability to analyze the recorded videos with
digital biomarkers, and fine-tune the medications. In healthcare settings with
an extreme scarcity of neurologists, the tool can take a more active role by
automatically assessing the symptoms frequently and referring the patient to a
neurologist if necessary.

We introduce several digital biomarkers of PD – objective features that are
interpretable, clinically useful, and significantly associated with the
clinical ratings. For example, the most significant feature correlated with
the finger-tapping severity score is the inter-quartile range (IQR) of finger-
tapping speed (Table 2). It measures one’s ability to demonstrate a range of
speeds (measured continuously) while performing finger-tapping and is
negatively correlated with PD severity. As the index finger is about to touch
the thumb finger, one needs to decelerate and operate at a low speed.
Conversely, when the index finger moves away from the thumb finger after
tapping, one needs to accelerate and operate at high speed. A higher range
implies a higher difference between the maximum and minimum speed, denoting
someone having more control over the variation of speed and, thus, healthier
motor functions. Moreover, the median and the maximum finger-tapping speed, as
well as the median and the maximum finger-tapping amplitudes have strong
negative correlations with PD severity. Finally, IQR and standard deviation of
finger tapping frequency, as well as entropy, variance, and IQR of tapping
periods, are found to have strong positive correlations with PD severity, as
they all indicate the absence of regularity in the amplitude and periods.
These findings align with prior clinical studies reporting that individuals
with Parkinson’s have a slower and less rhythmic finger tapping than those
without the condition (Kim et al. 2011). These computational features can not
only be used as _digital biomarkers_ to track the symptom progression of PwP
but also explain the model’s predicted severity score (e.g., an increase in
the severity score can be attributed to factors like reduced tapping speed,
smaller amplitude, etc.)

As we prepare to roll out our AI tool in healthcare settings, we must
prioritize ethical considerations such as data security, user privacy, and
algorithmic bias. As AI platforms become increasingly integrated into
healthcare domains, there is growing emphasis on protecting against data
breaches and crafting appropriate regulatory frameworks prioritizing patient
agency and privacy (Murdoch 2021). This ever-evolving landscape will have
significant implications for our future approach. Additionally, algorithmic
bias and its risks in perpetuating healthcare inequalities will present
ongoing challenges (Mhasawade, Zhao, and Chunara 2021). Many AI algorithms
tend to underdiagnose the underserved groups (Seyyed-Kalantari et al. 2021),
and it is critical to evaluate and report the model performance across age,
race, and gender groups. Our proposed model performs equally across the male
and female population, people with and without PD, and does not demonstrate a
bias to a particular age group. However, there is still room for improvement,
and diversifying our training data and gathering feedback from critical
stakeholders (especially from traditionally underrepresented and underserved
communities) will be important first steps for us to take toward building
fair, high-performance algorithms in the future. To that end, we will
diversify our dataset to be representative of the general population. Notably,
92% of participants in this study self-reported as white. Non-white races are
typically underrepresented in clinical research (Clark et al. 2019). Thus,
emphasizing recruitment of these populations through targeted outreach will be
essential, especially considering the risks that homogeneous training data can
pose in furthering healthcare inequalities (Gianfrancesco et al. 2018).

In the future, we plan to improve our model’s performance by building (i) a
larger dataset with a better balance in severity scores and (ii) a gatekeeper
to improve data quality. The proposed AI-driven model (i.e., XGBoost
regressor) was trained with 489 videos from 250 global participants. While
this dataset is the largest in the literature in terms of unique individuals,
it is still a relatively small sample size for training models capable of
capturing the essence of complex diseases such as Parkinson’s. As we see in
the confusion matrix presented in Figure 4(b), the model tends to predict a
severity score near 1 or 2. This is likely due to having fewer data to model
severity scores 0 and 3 accurately. Thus, additional data will be essential in
building more powerful models with potentially better performance.
Furthermore, we can improve our data quality and model performance by
developing “quality control” algorithms to provide users with real-time
feedback on capturing high-quality videos: such as adjusting their positioning
relative to the webcam or moving to areas with better lighting. Developing
user-friendly features to strengthen data collection will be crucial for
collecting videos remotely without direct supervision.

##  4 Methods

### Data Sources

Participants’ data were collected through a publicly accessible web-based
tool555https://parktest.net/. This tool allows individuals to contribute data
from the comfort of their homes, provided they have a computer browser,
internet connection, webcam, and microphone. In addition to the finger-tapping
task, the tool also gathers self-reported demographic information such as age,
gender, race, and whether the participant has been diagnosed with Parkinson’s
disease (PD) or not. Moreover, the tool records other standard neurological
tasks involving speech, facial expressions, and motor functions, which can
help to extend this study in the future.

We collected data from 250 global participants who recorded themselves
completing the finger-tapping task in front of a computer webcam. Data was
collected primarily at participants’ homes; however, a group of individuals
(48) completed the task in a clinic using the same web-based tool. Study
coordinators were available for the latter group if the participants needed
help.

### Clinical Ratings

The finger-tapping task videos were evaluated by a team of four raters,
including a non-specialist and three expert neurologists. The expert
neurologists are all associate or full professors in the Department of
Neurology at a reputable institution in the United States, possess vast
experience in PD-related clinical studies, and actively consult PD patients.
The non-specialist is an MDS-UPDRS certified independent rater and experienced
in conducting PD clinical studies. Despite holding a non-U.S. bachelor’s
degree in medicine (MBBS), the non-specialist is not a neurologist and does
not consult PD patients.

The raters watched the recorded videos of each participant performing the
finger-tapping task and rated the severity score for each hand following the
MDS-UPDRS guideline666https://www.movementdisorders.org/MDS-
Files1/Resources/PDFs/MDS-UPDRS.pdf (Part III, Section 3.4). The severity
rating is an integer ranging from 0 to 4 representing normal (0), slight (1),
mild (2), moderate (3), and severe (4) symptom severity. The rating
instructions emphasize focusing on speed, amplitude, hesitations, and
decrementing amplitude while rating the task. In addition to providing the
ratings, the raters could also mark videos where the task was not properly
performed or when a video was difficult to rate. We excluded these difficult-
to-rate videos when analyzing the performance of the raters.

To compute the ground-truth severity scores, we considered only the ratings
the three expert neurologists provided. If at least two experts agreed on the
severity rating for each recorded video, this was recorded as the ground
truth. If the experts had no consensus, their average rating rounded to the
nearest integer was considered the ground truth. The ratings from the non-
specialist were used solely to compare the machine-learning model’s
performance.

### Feature Extraction

We develop a set of features by analyzing the movements of several key points
of the hand. The feature extraction process is comprised of four stages: (i)
distinguishing left and right-hand finger-tapping from the recorded video,
(ii) quantifying finger-tapping movements by extracting key points on the
hand, (iii) reducing noise, and (iv) computing features that align with
established clinical guidelines, such as MDS-UPDRS.

#### Hand separation:

The finger-tapping task is performed for both hands, one hand at a time.
However, to rate each hand independently, we divided the task video into two
separate videos, one featuring the right hand and the other featuring the left
hand. We manually reviewed each video and marked the transition from one hand
to the other. The data collection framework will be designed to record each
hand separately to avoid manual intervention in the future.

#### Extracting hand key points:

After separating the left and right-hand finger-tapping videos, we apply
MediaPipe Hands 777https://google.github.io/mediapipe/solutions/hands.html to
detect the coordinates of 21 key points on each hand. Among the key points, we
utilized the thumb-tip, index-finger-tip, and the wrist, which are critical to
track the finger-tapping task-specific movements. In cases where multiple
persons are in the frame (i.e., one in the background), we discarded the
smaller hand’s key points as the individual performing the task was typically
closer to the camera, resulting in a larger hand appearance in the recorded
video. For the left-handed finger-tapping video, we only track the key points
on the left hand, and similarly, for the right-handed videos, we only track
the key points on the right hand. Instead of using Euclidean distance between
the thumb-tip and index finger-tip to measure the amplitude, speed, and other
metrics to quantify finger-tapping movements, we use the angle incident by
three key points: thumb-tip, wrist, and index finger-tip. This helps us to
deal with participants sitting at a variable distance from the camera since
the angle is invariant to the camera distance. We compute the angle for each
frame of the recorded video (i.e., if a video was recorded at 30
frames/second, we compute the angle 30 times per second). This helps us assess
the speed and acceleration of the fingers in a continuous manner.

#### Noise reduction:

Figure 5: Data pre-processing. Finger-tapping angles incident by three hand
key points (thumb-tip, wrist, index finger-tip) plotted as a time series.
Figures on the left show the noisy raw signals directly extracted using
Mediapipe. After the noise reduction step, we identify peak angles (red dots)
using a custom peak detection algorithm. Finally, trimming the signal by
removing the first and last tap yields the cleanest signal used for analysis,
as shown on the right. The top figures depict a person with severe tapping
difficulty (severity: 3), resulting in low and irregular amplitudes. The
central figures show a person with moderate tapping ability (severity: 2),
with slow and interrupted tapping and irregular amplitudes. Finally, the
bottom figures show a person with good rhythmic tapping ability, albeit with a
slower tapping speed (severity: 1).

The computed finger-tapping angles are plotted as a time-series signal in
Figure 5 (left). Negative values of the angle are indicative of a missing hand
in the captured frame. For example, at the beginning or at the end of the
recording, the hand might not be visible in the recording frame, as the
participant needs to properly position their hand. However, Mediapipe can also
inaccurately miss the hand in many frames, resulting in a negative value for
the angle. To address this issue, we implemented a strategy to interpolate
missing angle values when the majority of neighboring frames have non-negative
values. Specifically, we looked at the five frames before and after the
missing value. If the majority had non-negative angles, we interpolated the
value using a polynomial fit on the entire signal. Then, we find the largest
consecutive segment of frames where the hand is visible (i.e., the finger-
tapping angle was non-negative) in the signal and remove the frames before and
after that. This helps us to remove the pre and post-task segments where the
participants were not tapping their fingers and ensure that the analysis is
focused on the relevant segment of the signal. Figure 5 (middle) shows how the
raw, noisy signals were converted to cleaner signals after performing this
step.

The participants need to adjust the positioning of their hands before starting
to tap their fingers, and they also need to move their hands after completing
the task, which introduces further noise to the signal. Specifically, it can
impact the first and last tap they undergo. As the task instructs the
participants to tap their fingers ten times, we decided to remove the first
and last tap, hoping to obtain the cleanest signal to analyze. To accomplish
this, we ran a custom peak detection algorithm to find the peaks of the
finger-tapping angle, and we removed the portion of the signal before the
second peak and after the second-last peak. The peak detection algorithm
utilizes some of the unique properties of the task. For example, a peak must
be followed by a bottom (i.e., low value of finger-tapping angle) as the
tapping is repetitive, the duration between two subsequent taps cannot be too
small (i.e., determined by the fastest finger-tapping speed recorded), and the
peaks must be bigger than the smallest 25 percentile values of the signal.
Figure 5 (right) demonstrates the effectiveness of this step, as it helps to
obtain a clean signal that can be used to develop objective measures of the
finger-tapping task.

#### Computational features:

The coordinates of the wrist, thumb-tip, and index finger-tip, and their
incident angle were used to compute the following features:

  * •

Finger-tapping period is measured as the time (in seconds) it took for a
participant to complete a tap.

  * •

Finger-tapping frequency is the inverse of the finger-tapping period,
measuring the number of taps completed per second.

  * •

Amplitude is defined as the maximum angle (in degree) made by the thumb-tip,
wrist, and index finger-tip while completing a tap. As explained earlier, an
angular measurement is more reliable than linear (i.e., Cartesian) distance
since angles are invariant to distance from the camera.

  * •

Finger-tapping speed is a continuous measurement of an individual’s tapping
speed. For each recorded frame, we quantify the change in the finger-tapping
angle compared to the previous frame and multiply this with the recorded
video’s frame rate (so the speed is measured in degree/second unit). The
average frame rate of the recorded videos was 30, meaning that we can measure
an individual’s tapping speed 30 times a second.

  * •

Acceleration is also a continuous measurement, which is the derivative of
finger-tapping speed. Specifically, for each frame, we measure the change in
speed compared to the previous frame to quantify acceleration in
degree/second-square unit.

  * •

Wrist movement measures the stability of the hand when performing the task.
Ideally, the hand of a healthy individual should remain stable, and the wrist
should not move much. However, people with PD may experience hand tremors and
therefore demonstrate wrist movement. We compared the wrist’s current
coordinates for each frame with the previous frame’s coordinates. We measured
the absolute values of the movements along both the X and Y axes separately
and also measured the Cartesian distance. All values were normalized by the
distance between the wrist and the thumb carpometacarpal (CMC) to account for
the distance from the camera.

For each of the features above, we measured the statistical median, inter-
quartile range (IQR), mean, minimum, maximum, standard deviation, and entropy
888Entropy is a measure of uncertainty or randomness in a signal, calculated
using Shannon’s formula., and used them as separate features. Period,
frequency, and amplitude were measured discretely (i.e., for each tap), while
speed, acceleration, and wrist movement were assessed continuously (i.e., at
each frame). The detected peaks were used to separate each tap.

Additionally, we measured the following aggregate features based on the entire
signal to capture the rhythmic aspects of the finger-tapping task:

  * •

Aperiodicity: Periodicity is a concept borrowed from signal processing that
refers to the presence of a repeating pattern or cycle in a signal. For
example, a simple sinusoidal signal (e.g., $f(t)=sin(t)$) will have higher
periodicity compared to a signal that is a combination of several sinusoidal
signals (e.g., $f(t)=sin(t)+sin(2t)$). Aperiodicity measures the absence of
periodicity (i.e., the absence of repeating patterns). To measure
aperiodicity, signals are transformed into the frequency domain using Fast
Fourier Transformation (FFT). The resulting frequency distribution can be used
to calculate the normalized power density distribution, which describes the
energy present at each frequency. The entropy of the power density
distribution is then computed to measure the degree of aperiodicity in the
signal. A higher entropy value indicates greater uncertainty in the frequency
distribution, which corresponds to a more aperiodic signal. A similar measure
was found to be effective in measuring the symptoms of Alzheimer’s disease
(Sharma and Nadkarni 2020).

  * •

Number of interruptions: Interruption is defined as the minimal movement of
the fingers for an extended duration. We calculated a distribution of
continuous finger-tapping speeds across the study population. Our analysis
revealed that over 95 percent of the tapping speeds exceeded 50
degrees/second. As a result, any instance where an individual’s finger-tapping
speed was less than 50 degrees/second for at least ten milliseconds (ms) was
marked as an interruption. The total number of interruptions present in the
recorded video was then computed using this method.

  * •

Number of freezing: In our study, we considered freezing as a prolonged break
in movement. Specifically, any instance where an individual recorded less than
50 degrees/second for over 20 ms was identified as a freezing event, and we
counted the total number of such events.

  * •

Longest freezing duration: We recorded the duration of each freezing event and
calculated the longest duration among them.

  * •

Tapping period linearity: We recorded the tapping period for each tap and
evaluated the possibility of fitting all tapping periods using a linear
regression model based on their degree of fitness ($R^{2}$). Additionally, we
determined the slope of the fitted line. The underlying idea was that if the
tapping periods were uniform or comparable, a straight line with slope=0 would
adequately fit most periods. Conversely, a straight line would not be an
appropriate fit if the periods varied significantly.

  * •

Complexity of fitting periods: The complexity of fitting finger-tapping
periods can provide insights into the variability of these periods. To measure
this complexity, we used regression analysis and increased the degree of the
polynomial from linear (degree 1) up to 10. We recorded the minimum polynomial
degree required to reasonably fit the tapping periods (i.e., $R^{2}\geq 0.9$).

  * •

Decrement of amplitude: Decrement of amplitude is one of the key symptoms of
Parkinsonism. We measured the finger-tapping amplitude for each tap and
quantified how the amplitude at the end differed from the mean amplitude and
amplitude at the beginning. Additionally, we calculate the slope of the linear
regression fit to capture the overall change in amplitude from start to end.

#### Feature selection and significance test:

The abovementioned measurements and some of their statistical aggregates
result in 65 features used to analyze the recorded finger-tapping videos. We
perform a cross-correlation analysis among the features, identify the highly
correlated pair (i.e., Pearson’s correlation coefficient, $r>0.85$), and drop
one from each pair. This helps to remove redundant features and enables
learning the relationship between the features and the ground truth severity
scores using simple models. This is important, as simple models tend not to
over-fit the training data and are more generalizable than complex models
(commonly known as Occam’s razor (Blumer et al. 1987)). After this step, the
number of features was reduced to 53.

For each of the 53 features, we perform a statistical correlation test to
identify the features significantly correlated with the ground-truth severity
score. Specifically, for each feature, we take the feature values for all the
recorded videos in our dataset and the associated ground-truth severity scores
obtained by the majority agreement of three expert neurologists. We measure
the Pearson’s correlation coefficient ($r$) between the feature values and
severity scores and test the significance level of that correlation (i.e.,
$p$-value). We found 18 features to be significantly correlated (at a
significance level, $\alpha=0.01$). The significant features include finger-
tapping speed (inter-quartile range, median, maximum, minimum), acceleration
(minimum), amplitude (median, maximum), frequency (inter-quartile range,
standard deviation), period (entropy, inter-quartile range, minimum), number
of interruptions, number of freezing, longest freezing duration, aperiodicity,
the complexity of fitting periods, and wrist movement (minimum Cartesian
distance). Table 2 reports the correlation’s direction, strength, and
statistical significance level for the most correlated ten features.

### Model Training, Evaluation and Explanation

To model our dataset, we applied a standard set of regressor models like
XGBoost, CatBoost, SVM, etc., and found XGBoost(Chen and Guestrin 2016) to be
the best-performing one. XGBoost – a widely used algorithm for regression,
classification, and ranking problems – stands for eXtreme Gradient Boosting
and implements a gradient boosting decision trees algorithm. XGBoost-regressor
is an implementation of the XGBoost algorithm for regression
problems999https://xgboost.readthedocs.io/en/stable/python/python˙api.html#xgboost.XGBRegressor.
It works by building a series of decision trees where each tree tries to
correct the errors made by the previous tree. In the end, the algorithm
combines the results of all trees to make the final prediction. As mentioned
in Table 3, we conducted an extensive hyper-parameter search of the XGBoost
regressor by experimenting with different learning rates, max depth of the
tree, number of estimators to use, etc.

Hyper-parameter name | Experiment range
---|---
max_depth | [1, 2, 3, 5, 6, 10, 12, 15, 18, 20, 22, 25, 28, 30]
learning_rate | Sampled uniformly in between 0 and 0.1
n_estimators | [1, 2, 5, 8, 10, 12, 15, 18, 20, 22, 25, 28, 30,
32, 35, 38, 40, 43, 46, 50, 53, 56, 60, 63, 66,
70, 73, 76, 80, 83, 86, 90, 93, 96, 100, 110,
120, 130, 140, 150, 160, 170, 180, 190, 200]
sub_sample | [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]
colsample_bytree | [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]
colsample_bylevel | [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]

Table 3: Hyper-parameters used to tune our XgBoost Regressor model.

We have a reasonably balanced representation across all the severity scores
except one: the number of videos with severity scores 0, 1, 2, 3, and 4 was
108, 181, 141, 54, and 5. The neurologists rate a video as 4 only if a
participant can barely perform the task. Therefore, collecting data from such
participants with a remote tool like ours is very difficult. To simplify the
machine learning model, we combined videos with 3 and 4 severity and labeled
them as 3.

To evaluate the model, we implemented a stratified $k$-fold cross-validation
technique with $k=10$. This involved dividing the dataset into ten equally
sized folds, each containing examples from all severity classes in the same
ratio as the entire dataset. 9 out of 10 folds were used to train the XGBoost
model, and the model was evaluated on the remaining fold. The average
performance of 20 random iterations was reported. Stratified cross-validation
ensures that each fold of the cross-validation process has a representative
proportion of examples from each class in the dataset, and the reported
evaluation metrics accurately reflect the model’s performance on the entire
dataset. This is particularly important in imbalanced datasets where the
number of examples in each class can vary widely.

We used two metrics: mean absolute error (MAE) and Pearson’s correlation
coefficient (PCC) to evaluate the performance of the regression model. MAE is
a metric used to measure the average magnitude of the errors in a set of
predictions without considering their direction. It is calculated by taking
the absolute differences between the predicted and actual values and then
averaging those differences. The smaller the MAE, the better the model is
performing. PCC measures the strength and direction of the relationship
between two variables and ranges from -1 to +1. A correlation of -1 indicates
a perfect negative relationship, a correlation of +1 indicates a perfect
positive relationship, and 0 indicates no relationship between the variables.

Some of the videos in the dataset were very noisy and marked as “difficult to
rate” by the expert raters. We ignored these ratings while assessing the
performance of the expert and non-expert raters. However, the raters still
rated these videos, and the XGBoost regressor model was trained with and
without these videos. The model’s performance was similar in both cases – the
model trained with all the videos (including the difficult ones) had an MAE of
0.5903 and a PCC of 0.6541, while the MAE and PCC were 0.5889 and 0.6521,
respectively, when the difficult videos were excluded. These results show that
the model could learn from noisy data, which is critical for the proposed
remote tool. For consistency in reporting results and fair comparison, we
reported the model’s performance based on the dataset excluding the difficult
videos.

To explain the model’s performance, we used SHAP. SHAP (SHapley Additive
exPlanations) is a tool used for explaining the output of any supervised
machine learning model (Lundberg et al. 2020). It is based on Shapley values
from cooperative game theory, which allows us to assign an explanatory value
to each feature in the input data. The main idea behind SHAP is to assign a
contribution score to each feature that represents its impact on the model’s
output. These contribution scores are calculated by considering each feature’s
value in relation to all possible combinations of features in the input data.
By doing this, SHAP can provide a detailed and intuitive explanation of why a
particular model made a certain decision. This makes it easier for human
decision-makers to trust and interpret the output of a machine-learning model.

### Use of Large Language Models

ChatGPT101010https://chat.openai.com/chat – a large language model developed
by OpenAI111111https://openai.com/ that can understand natural language
prompts and generate text – was used to edit some part of the manuscript
(i.e., suggest improvements to the language, grammar, and style). All
suggested edits by ChatGPT were further verified and finally integrated into
the manuscript by an author. Please note that ChatGPT was used only to suggest
edits to existing text, and we did not use it to generate any new content for
the manuscript.

## Code and Data Availability

The recorded videos were collected using a web-based tool. The tool is
publicly accessible at https://parktest.net. The codes for video processing
and feature extraction, as well as the trained model, will be made publicly
available upon the acceptance of this paper. We will provide a link to the
repository containing the codes and models in the paper.

Unfortunately, we are unable to share the raw videos due to the Health
Insurance Portability and Accountability Act (HIPAA) compliance. However, we
are committed to sharing the extracted features upon receiving an email
request at<EMAIL_ADDRESS>The features will be provided in a
structured format that can be easily integrated with existing machine-learning
workflows.

For potential collaboration, we welcome interested individuals or groups to
reach out to us at<EMAIL_ADDRESS>Depending on the specifics of the
collaboration, we may be able to share some additional data beyond the
extracted features.

## Acknowledgement

This work was supported by the U.S. Defense Advanced Research Projects Agency
(DARPA) under grant W911NF19-1-0029, National Science Foundation Award
IIS-1750380, National Institute of Neurological Disorders and Stroke of the
National Institutes of Health under award number P50NS108676 and Moore
Foundation.

## References

  * Blumer et al. (1987) Blumer, A.; Ehrenfeucht, A.; Haussler, D.; and Warmuth, M. K. 1987.  Occam’s razor.  _Information processing letters_ , 24(6): 377–380.
  * Chen and Guestrin (2016) Chen, T.; and Guestrin, C. 2016.  Xgboost: A scalable tree boosting system.  In _Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining_ , 785–794.
  * Clark et al. (2019) Clark, L. T.; Watkins, L.; Piña, I. L.; Elmer, M.; Akinboboye, O.; Gorham, M.; Jamerson, B.; McCullough, C.; Pierre, C.; Polis, A. B.; et al. 2019.  Increasing diversity in clinical trials: overcoming critical barriers.  _Current problems in cardiology_ , 44(5): 148–172.
  * Gianfrancesco et al. (2018) Gianfrancesco, M. A.; Tamang, S.; Yazdany, J.; and Schmajuk, G. 2018.  Potential biases in machine learning algorithms using electronic health record data.  _JAMA internal medicine_ , 178(11): 1544–1547.
  * Hughes et al. (1992) Hughes, A. J.; Daniel, S. E.; Kilford, L.; and Lees, A. J. 1992.  Accuracy of clinical diagnosis of idiopathic Parkinson’s disease: a clinico-pathological study of 100 cases.  _Journal of neurology, neurosurgery & psychiatry_, 55(3): 181–184.
  * Khan et al. (2014) Khan, T.; Nyholm, D.; Westin, J.; and Dougherty, M. 2014.  A computer vision framework for finger-tapping evaluation in Parkinson’s disease.  _Artificial intelligence in medicine_ , 60(1): 27–40.
  * Kim et al. (2011) Kim, J.-W.; Lee, J.-H.; Kwon, Y.; Kim, C.-S.; Eom, G.-M.; Koh, S.-B.; Kwon, D.-Y.; and Park, K.-W. 2011.  Quantification of bradykinesia during clinical finger taps using a gyrosensor in patients with Parkinson’s disease.  _Medical & biological engineering & computing_, 49: 365–371.
  * Kissani et al. (2022) Kissani, N.; Liqali, L.; Hakimi, K.; Mugumbate, J.; Daniel, G. M.; Ibrahim, E. A. A.; Yewnetu, E.; Belo, M.; Wilmshurst, J.; Mbelesso, P.; et al. 2022.  Why does Africa have the lowest number of Neurologists and how to cover the Gap?  _Journal of the Neurological Sciences_ , 434: 120119.
  * Langevin et al. (2019) Langevin, R.; Ali, M. R.; Sen, T.; Snyder, C.; Myers, T.; Dorsey, E. R.; and Hoque, M. E. 2019.  The PARK Framework for Automated Analysis of Parkinson’s Disease Characteristics.  _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ , 3(2): 1–22.
  * Lundberg et al. (2020) Lundberg, S. M.; Erion, G.; Chen, H.; DeGrave, A.; Prutkin, J. M.; Nair, B.; Katz, R.; Himmelfarb, J.; Bansal, N.; and Lee, S.-I. 2020.  From local explanations to global understanding with explainable AI for trees.  _Nature Machine Intelligence_ , 2(1): 2522–5839.
  * Mhasawade, Zhao, and Chunara (2021) Mhasawade, V.; Zhao, Y.; and Chunara, R. 2021.  Machine learning and algorithmic fairness in public and population health.  _Nature Machine Intelligence_ , 3(8): 659–666.
  * Murdoch (2021) Murdoch, B. 2021.  Privacy and artificial intelligence: challenges for protecting health information in a new era.  _BMC Medical Ethics_ , 22(1): 1–5.
  * Nunes et al. (2022) Nunes, A. S.; Kozhemiako, N.; Stephen, C. D.; Schmahmann, J. D.; Khan, S.; and Gupta, A. S. 2022.  Automatic Classification and Severity Estimation of Ataxia From Finger Tapping Videos.  _Frontiers in neurology_ , 12: 2587.
  * Perlmutter (2009) Perlmutter, J. S. 2009.  Assessment of Parkinson disease manifestations.  _Current protocols in neuroscience_ , 49(1): 10–1.
  * Quinonero-Candela et al. (2008) Quinonero-Candela, J.; Sugiyama, M.; Schwaighofer, A.; and Lawrence, N. D. 2008\.  _Dataset shift in machine learning_.  Mit Press.
  * Seyyed-Kalantari et al. (2021) Seyyed-Kalantari, L.; Zhang, H.; McDermott, M. B.; Chen, I. Y.; and Ghassemi, M. 2021.  Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations.  _Nature medicine_ , 27(12): 2176–2182.
  * Sharma and Nadkarni (2020) Sharma, R.; and Nadkarni, S. 2020.  Biophysical basis of alpha rhythm disruption in Alzheimer’s disease.  _Eneuro_ , 7(2).
  * Williams et al. (2020) Williams, S.; Relton, S. D.; Fang, H.; Alty, J.; Qahwaji, R.; Graham, C. D.; and Wong, D. C. 2020.  Supervised classification of bradykinesia in Parkinson’s disease from smartphone videos.  _Artificial Intelligence in Medicine_ , 110: 101966.
  * Willis et al. (2011) Willis, A.; Schootman, M.; Evanoff, B.; Perlmutter, J.; and Racette, B. 2011.  Neurologist care in Parkinson disease: a utilization, outcomes, and survival study.  _Neurology_ , 77(9): 851–857.