# Systematically Assessing the Security Risks of AI/ML-enabled Connected
Healthcare Systems

Mohammed Elnawawy$\dagger$, Mohammadreza Hallajiyan$\dagger$, Gargi
Mitra$\dagger$, Shahrear Iqbal$\ast$, Karthik Pattabiraman$\dagger$
$\dagger$University of British Columbia, $\ast$National Research Council
Canada
Email: {mnawawy, hallaj<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

The adoption of machine-learning-enabled systems in the healthcare domain is
on the rise. While the use of ML in healthcare has several benefits, it also
expands the threat surface of medical systems. We show that the use of ML in
medical systems, particularly connected systems that involve interfacing the
ML engine with multiple peripheral devices, has security risks that might
cause life-threatening damage to a patient’s health in case of adversarial
interventions. These new risks arise due to security vulnerabilities in the
peripheral devices and communication channels. We present a case study where
we demonstrate an attack on an ML-enabled blood glucose monitoring system by
introducing adversarial data points during inference. We show that an
adversary can achieve this by exploiting a known vulnerability in the
Bluetooth communication channel connecting the glucose meter with the ML-
enabled app. We further show that state-of-the-art risk assessment techniques
are not adequate for identifying and assessing these new risks. Our study
highlights the need for novel risk analysis methods for analyzing the security
of AI-enabled connected health devices.

###### Index Terms:

Machine learning, medical system security, risk analysis, multi-vendor
systems.

††publicationid: pubid: 0000–0000/00$00.00 © 2021 IEEE

##  I Introduction

The use of Artificial Intelligence (AI), especially Machine Learning (ML)
techniques, is becoming increasingly popular in medical fields in recent
years. As of October 2022, the U.S. Food and Drug Administration (FDA) has
approved $521$ ML-enabled devices across $15$ different medical disciplines
(e.g., Cardiology, Ophthalmology, and Gastroenterology) [1]. However, despite
its benefits, the use of ML has expanded the threat surface of medical
systems. Many papers [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16] have
shown that ML techniques are vulnerable to cyberattacks.

ML-enabled medical devices are used for performing critical activities such as
remote patient monitoring, controlling surgical equipment, automatic drug
administration, and preliminary/advanced disease diagnosis – tasks that
require high accuracy and reliability [1]. If an adversary compromises such a
device, it can force the ML engine to make incorrect predictions or decisions,
which can have catastrophic consequences, such as wrong treatment leading to
health complications.

An adversary can force an ML engine to generate incorrect predictions or
decisions by injecting carefully crafted malicious data points either during
training or inference. Preventing such attacks in ML-enabled medical devices
is challenging. These ML-enabled devices are typically interconnected with
multiple other peripheral sensor devices that collect physiological data of
patients, which are then processed by the ML engine. Therefore, it is not
enough to secure only the ML-enabled device, since adversaries can exploit
vulnerabilities in the peripheral devices to inject malicious data points into
the ML engine, and cause damage.

To protect the end-to-end system, one must systematically identify and assess
the security risks 111We define risk as the probability of a security
vulnerability getting exploited, and its potential impact or loss. of the
overall system due to vulnerabilities in peripheral devices. _To the best of
our knowledge, there is no systematic technique for identifying and assessing
the end-to-end risks of ML-enabled medical systems._

Identification of risks in ML-enabled connected medical systems has two
challenges. First, at deployment, the ML-enabled device is interfaced with
several other peripheral devices, each of which may be manufactured by a
different company. For instance, a user of the ML-enabled blood glucose
monitoring app Dreamed Advisor Pro [17], needs to install the app on a
smartphone, and then connect to it a smartwatch, a glucose meter, and an
insulin pump, all of which would be manufactured by different companies, and
may have their own security vulnerabilities. Second, each app user may use
peripheral devices from different sets of manufacturers, leading to diverse
vulnerabilities among different users of the same app. For instance, one user
of the app might use a vulnerable smartphone, while another user might use a
vulnerable glucose meter. This makes it challenging for the manufacturer of
the ML-enabled device to estimate the post-deployment security risks during
the design phase.

Furthermore, it is also challenging to assess the _severity_ of these risks.
This is because the severity of a risk posed by a vulnerable peripheral device
might differ when assessed in the context of the individual device (in-silo
assessment) versus when assessed in the context of the entire system. For
instance, consider a user who connects the Dreamed Advisor Pro app to a
glucose meter with a write-access vulnerability, and a smartphone with a read-
access vulnerability. When assessed separately, the glucose meter would have a
higher perceived risk than the smartphone. However, for an adversary who wants
to inject adversarial glucose meter readings into the app, being able to read
data from the smartphone (e.g., meal timings, latest insulin dose,
carbohydrates taken) might be useful for crafting malicious data points that
adhere to physiological constraints. Adhering to the physiological constraints
is important for the adversary to get the malicious data points accepted as
valid inputs by the ML engine. Therefore, we need to holistically consider the
risks from the interplay of vulnerabilities in peripheral devices.

In this paper, we perform a systematic analysis to highlight the security
risks of end-to-end ML-enabled connected medical systems. Our analysis
consists of three steps. First, we conduct a systematic exploration of the
FDA-approved ML-enabled medical devices to understand the ML techniques that
they use, and the damage that can be caused to a patient if the ML technique
mispredicts their case. Second, we conduct an extensive review to identify
possible ways in which adversaries can inject malicious data points into an
ML-enabled medical device at deployment. This involves a cross-domain
analysis, where we map known attacks on ML algorithms with known
vulnerabilities in peripheral devices that would make the attacks practical.
Finally, we perform a critical evaluation of state-of-the-art risk assessment
frameworks used by the ML-enabled medical device manufacturing companies
today. We identify loopholes in these risk assessment strategies that might
make manufacturers miss the risks arising due to vulnerabilities in connected
peripheral devices.

Contributions. The main contributions of this paper are:

  1. 1.

We perform a systematic cross-domain security analysis of commercial ML-
enabled medical devices (Section III), to highlight the security risks of
connected health devices.

  2. 2.

We perform a case study on a realistic ML-enabled blood glucose management
system (BGMS)(Section IV) to demonstrate an attack on the system where the
adversary compromises a communication link in the system.

  3. 3.

Finally, we perform an evaluation of state-of-the-art risk assessment
techniques (Section V). We find that they are inadequate in identifying and
analyzing the severity of security risks in ML-enabled medical systems,
particularly the risks posed to the ML engine by vulnerable peripheral
devices. We also highlight directions for improvement.

##  II Motivation and Background

We first show an example of an AI/ML-enabled medical system, namely, the BGMS,
to highlight the security risks that are caused by vulnerabilities present in
the peripheral components connected to it at deployment. Subsequently, we show
how the risks identified in this scenario are generalizable to any connected
ML-enabled medical system.

###  II-A Blood Glucose Management: Background

Diabetes is a chronic health condition that hinders the ability of the human
body to produce insulin naturally. This increases the blood glucose level of
the patient beyond the normal physiological limits, which has detrimental
effects on their overall health condition. A sudden spike or drop in the blood
glucose level of the patient can even be fatal. Furthermore, normal daily
activities such as having a meal or exercising can cause fluctuations in their
blood glucose levels.

Blood glucose levels can be divided into three ranges, hypoglycemic (less than
70-80 mg/dL), normal (80 mg/dl - 125 mg/dl), and hyperglycemic (above 125
mg/dL while fasting and above 180 mg/dL two hours postprandial)[18, 19]. A
consistently hyperglycemic patient is diabetic and often needs insulin to be
injected to bring their glucose level to the normal range. A hypoglycemic
patient does not need insulin to be injected into the body.

BGMS apps help diabetic patients monitor their blood glucose levels and
administer insulin bolus whenever the glucose levels begin to rise at an
abnormal rate. The patient can either inject the insulin bolus into their
bloodstream manually or use an automated insulin pump that is connected to and
triggered by the BGMS app. Note that, the dose of the insulin bolus needs to
be computed carefully - a dose higher than necessary can cause a sharp drop in
the blood glucose level, whereas a dose lower than necessary would fail to
bring the glucose level down to the normal range.

###  II-B An ML-enabled Blood Glucose Management System

We consider a commercial FDA-approved ML-enabled BGMS app, the Dreamed Advisor
Pro [17]. The app helps diabetic patients keep their blood glucose levels
within the normal physiological range by suggesting the dose of the insulin
bolus to be taken from time to time. Additionally, the app also suggests
personalized meal plans and physical activities to the patients so as to
prevent a spike in their blood glucose levels. However, since the
implementation details of the ML technique used by the Dreamed Advisor Pro app
are not publicly available, we instead use a well-known, public ML-based blood
glucose prediction technique [20].

BGMS apps suggest insulin doses based on the predicted blood glucose level of
the patient in the near future (next $30$ or $60$ minutes) [20]. This
prediction is done by an ML engine that runs at the back-end of the BGMS app,
using other physiological values of the patient recorded in the recent past,
such as blood glucose measurements, insulin doses taken, meal timings,
carbohydrate intake, etc. These values are either entered manually into the
app by the patient or read from other peripheral sensor devices (e.g.,
continuous glucose monitors and smartwatches) that are connected to the
patient’s body. Apart from sensor devices, the app can also be interfaced with
actuator devices (e.g., insulin pumps) that perform actions suggested by the
ML-enabled app. The ML-enabled app, along with the connected peripheral
devices, constitutes the BGMS.

Figure 1 shows an end-to-end schematic representation of an ML-enabled BGMS.
The glucose monitor records the blood glucose levels of the patient at regular
time intervals and transmits them to the app over a Bluetooth communication
channel. The app uses these values for data visualization and also sends them
to a cloud server over the Internet for storage and processing by the AI/ML
engine. Finally, the predicted insulin dose is either displayed on the app or
sent to the automated insulin pump attached to the patient’s body.

Figure 1: A blood glucose management system that uses ML on the cloud, and
interfaces with multiple peripheral devices.

Security Risks.  We consider a scenario where an adversary intends to cause
harm to a targeted user of the BGMS app by forcing the BGMS to inject a high
dose of insulin into the user’s body when it is not supposed to, or vice
versa. We further assume that the insulin pump used by the target is secure
against known vulnerabilities [21] and that the ML engine runs on a secure
cloud server. Under such circumstances, the adversary can still force the ML
engine to mispredict the insulin dose by injecting carefully crafted
adversarial data points into it via the peripheral devices. For instance, the
adversary can modify the blood glucose values sent to the ML engine [10], by
exploiting vulnerabilities in the third-party glucose monitor interfaced with
the BGMS app.

The Dreamed Advisor Pro app is compatible with all non-continuous glucose
meters with regulatory approval and nine different models of continuous
glucose monitors manufactured by different companies [22]. Commercial glucose
meters have been known to have firmware vulnerabilities [23] that would allow
an adversary to change the blood glucose level readings that are sent by the
glucose meter to the BGMS app. Alternatively, an adversary could also exploit
known vulnerabilities in the Bluetooth communication protocol [21] to launch a
man-in-the-middle attack and change the blood glucose level readings.
Furthermore, these glucose monitors are also susceptible to physical attacks
that can be carried out with electromagnetic radiation [24]. Additionally,
vulnerabilities in devices and communication links in the end-to-end data
processing pipeline, such as mobile devices and routers, can be exploited to
manipulate blood glucose measurements.

###  II-C Unique Security Risks in AI/ML-enabled Medical Systems

The security risks discussed in the context of the BGMS app can be generalized
for any ML-enabled medical device that is connected to multiple peripheral
devices (often manufactured by other companies) at deployment. A breach in the
security of any of these peripheral devices could enable an adversary to
manipulate the data sent to the ML engine, which would lead the ML engine to
mispredict the condition of the patient. Such mispredictions can adversely
impact the patient’s health.

There are two challenges for the manufacturer of the ML-enabled device to
foresee the aforementioned security risks and gauge their impact at the time
of design and manufacturing. First, these devices are built to be compatible
with a wide range of peripheral devices for the operational convenience of the
consumers. This makes it difficult for the manufacturer to predict what
peripheral devices the consumer would connect with the ML-enabled device at
deployment, and what vulnerabilities those devices might have. The interplay
of different vulnerabilities would enable an adversary to perform different
types of attacks on the ML engine, which makes it challenging to analyze the
risk via the in-silo testing performed today.

Second, while ML-enabled connected systems are used in many domains such as
smart homes and industrial control systems, performing risk analysis of
medical systems is more difficult as physiological data is much more complex
in nature and varies widely across individuals [25]. Consequently, the impact
of manipulating physiological data might be different for different patients.
Manufacturers of commercial ML-enabled medical devices focus on the accuracy
of prediction of the ML engine and device failures in non-adversarial
scenarios. However, for a comprehensive end-to-end security risk analysis, the
impact of adversarial inputs on different patients, and the feasibility of
attacks should be considered.

For instance, in the BGMS attack in Section II-B, the adversary attempts to
force the ML model to inaccurately predict insulin dosage by introducing
adversarial inputs to the ML engine. Most known attacks on ML engines [2, 3,
4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16] would require the adversary to
observe and manipulate at least a subset of input sensor values to alter the
predicted insulin dose. The more observability the adversary has into
different sensor values, the more efficiently she can craft the adversarial
inputs. We assumed in Section II-B that the adversary can only manipulate the
glucose level readings by exploiting vulnerabilities in the glucose meter or
the Bluetooth communication link. However, if the adversary has visibility
into the commands sent to the insulin pump, she can craft adversarial glucose
meter readings in a way that would yield a higher success rate with an equal
or lower amount of perturbations. On an individual basis, unauthorized read
access to the insulin pump is a low-level risk. However, in the context of the
entire BGMS, this is a high-level risk.

##  III Attacks on ML-Based Systems and Their Relevance in the Healthcare
Domain

Motivated by the BGMS in Section II, we perform a systematic investigation of
a subset of FDA-approved AI/ML-enabled medical devices/software [1] to
identify potential security risks at deployment. We perform this investigation
in two steps. First, we identify the ML techniques used by each of the
devices/software systems. We survey existing work in the domain of AI/ML
security to understand what types of attacks may target these techniques
(Section III-A). Next, for each of these devices/software, we examine the
practicality of the attack scenarios identified in the previous step (
SIII-B).

Selecting medical systems for our investigation. As of December 2022, the U.S.
FDA has approved $521$ ML-enabled medical devices 222As per the terminology
used by the FDA website, the term ‘device’ refers to both physical devices as
well as software solutions across $15$ different physiological panels.
However, there is no automated risk analysis technique today, and analyzing
all the $521$ devices manually would be arduous and time-consuming. Therefore,
we selected a subset of these devices for manual analysis. To ensure a fair
representation of the set of devices, we use the following selection criteria.

  1. 1.

We select at least one device from each physiological panel to study if risks
due to vulnerabilities in peripheral devices are common across all medical
domains;

  2. 2.

Within the same physiological panel, we select devices that perform different
types of diagnosis or treatments, to ensure coverage across different medical
activities.

  3. 3.

We select an equal number of two types of ML-enabled devices - software that
can be installed on the consumer’s pre-existing device, and software that is
sold bundled with proprietary hardware. This would help us understand if one
of these is more secure than the other;

  4. 4.

We select devices that are used in hospitals and clinics under medical
supervision, as well as devices that are used by patients at home without
medical supervision. This would help us understand if the environment in which
the medical device is deployed affects its security.

In addition, we also need to ensure that sufficient information for our
analysis is available for each of the selected devices, e.g., the ML algorithm
used, the type of data processed, etc. We thus select $20$ different devices
across $13$ of the $15$ physiological panels, as shown in Table I.
Unfortunately, due to the lack of sufficient information, we could not select
any device from the Dental and Hematology panels.

###  III-A Known Attacks on ML Algorithms Used by FDA-Approved Medical Devices

Table I presents our study of the ML algorithms used by the devices we
selected for our evaluation. Our goal is to understand if there are known
attacks against these ML algorithms that can be used by adversaries to make
these ML engines mispredict the outcome. We also examine what type of tasks
these ML algorithms are used for in the medical systems and the worst-case
consequences of misprediction by the ML engine. First, we describe how we
conduct our study. Next, we present some interesting insights from this study.

Survey Process. For identifying the ML algorithm used by the devices, we go
through the Premarket Notification summary of each device. The Premarket
Notification summary contains important information about the device,
including the ML algorithm they use, and the type of data they process. This
was submitted by the device manufacturers to the FDA along with a request for
approval (available on the FDA website [1]).

Unfortunately, for some devices, the information in the Premarket notification
summary was inadequate. For instance, for the device GI Genius, we were unable
to find information about the exact ML algorithm used in the Premarket
Notification summary. For such devices, we search the website of the
manufacturer for detailed device information. If that also fails, we attempt
to estimate the nature of the ML algorithm that is used by the device. For
this, we first note the task performed by the device, the type of data it
processes, and the features that it takes as inputs. Then we look for known ML
algorithms that perform the same task using similar input features and the
same data type. For instance, in the case of GI Genius, we found an existing
paper [26] that performs gastrointestinal lesion detection (the same task
performed by GI Genius) with a high accuracy using Convolutional Neural
Networks (CNN).

We also estimate the worst-case impact of mispredictions by these devices from
our understanding of the device functionality and description provided by the
manufacturer, either in the device summary or on their website. We deem the
misprediction to be potentially fatal if the device is used for the treatment
or diagnosis of a patient in a medical emergency, where a medical expert would
not have enough time to assess the correctness of the device output. For
instance, the NuVasive Pulse System is used by surgeons during spinal
surgeries for continuously monitoring the neurophysiological status of the
patient. According to our understanding, during surgery, the medical
practitioner would not have enough time to assess the output of the device to
determine if it is anomalous.

Insights. We obtained the following insights from the information that we
gathered using the aforementioned process.

  1. 1.

We observed that the majority of the ML algorithms are vulnerable to
inference-time attacks, while only a few are vulnerable to training-time
attacks. Both types of attacks can pose major health risks to the targeted
patients. However, inference-time attacks are easier to execute for the
adversary since they require fewer adversarial inputs as compared to training-
time attacks. Most of the devices vulnerable to training-time attacks are
deployed in hospitals or diagnostic centers, where often the same set of
peripheral devices is used for collecting data from multiple patients. If the
adversary succeeds in compromising such peripheral devices over a long time,
they can manipulate enough patient data to poison the training dataset 333Many
systems undergo periodic re-training on recent physiological data. This would
impact a large number of patients. Examples include the Deep Learning Image
Recognition Software, and the Oxehealth Vital Signs monitor.

  2. 2.

Even when the devices are operated by medical practitioners, detecting a
misprediction might be challenging for two reasons. First, physiological data
exhibit a high degree of variance across patients. Two patients having the
same medical condition might exhibit different trends in their physiological
data due to other underlying health conditions and demographic factors [27].
Second, many devices are used for diagnoses/medical procedures that are
performed infrequently on a given patient or are used only for medical
emergencies. Under such circumstances, the absence of the particular patient’s
historical physiological information makes it challenging for the medical
practitioner to detect an anomaly. Examples are Cardiologs ECG Analysis
Platform, GI Genius, ABMD Software, and NuVasive Pulse System.

  3. 3.

Some devices (e.g., the One Drop Blood Glucose Monitoring System), are used by
patients at home without continuous medical supervision. Detecting a
misprediction from such devices would be much more challenging than devices
that are directly operated by medical practitioners.

  4. 4.

Many of these devices are used in clinics for disease diagnosis, some are used
in hospitals for treatment and patient monitoring, while a few are used by
patients at home. Typically, hospitals and clinics have a higher security
budget as compared to individual patients staying at home. Therefore, it would
be challenging to design one security solution that fits all ML-enabled
medical systems. Therefore, while designing security solutions for medical
devices, the implementation effort and cost should also be considered.

  5. 5.

Some of the ML-enabled softwares are sold bundled with proprietary hardware
(i.e., software-in-medical-device), while some can be installed by the user on
any general-purpose computer (i.e., software-as-medical-device). The latter
have a larger threat surface than the former as different general-purpose
computer models would have different combinations of hardware, software, and
OS vulnerabilities, and assessing the severity of the risks they pose is
challenging.

###  III-B Analyzing the Functionality and Vulnerability Landscape of FDA-
approved (AI/ML)-Enabled Medical Devices

| Sl.
---
No.
| Device
---
Name [1]
| Physiological
---
Panel
| Device
---
Functionality
User |  | Type of
---
ML algo
used
| Type of
---
data
processed
| Known
---
attacks
(Attack type)
| Potential
---
impact of
misprediction
1 |  | CardioLogs
---
ECG Analysis
Platform†
Cardiovascular |  | Cardiac
---
arrhythmia
detector
| Medical
---
practitioners
| Deep Neural Network
---
(DNN)
Image |  Chen et al. [2] I |  | Wrong
---
treatment
(Fatal)
2 |  | Oxehealth
---
Vital Signs†
Cardiovascular |  | Camera-based
---
monitor for heart,
pulse, and
respiratory rate
| Medical
---
practitioners
| Hybrid
---
convolutional Long
short term memory
networks (LSTM)
Video |  | Albattah
---
et al. [3] T,I
| Wrong
---
treatment
3 |  GI Genius‡ |  | Gastroenterology/
---
Urology
| Gastro-
---
intestinal
lesion
detection
| Medical
---
practitioners
| Convolutional
---
neural
networks (CNN) ∗
Video |  |  Amin et al. [28] T
---
| Wrong
---
diagnosis
4 |  SOZO‡ |  | Gastroenterology/
---
Urology
| Body fluid
---
analyzer for
assessing
protein-calorie
malnutrition
| Medical
---
practitioners
CNN ∗ | Numeric |  Byra et al. [29] I |  | Wrong
---
diagnosis
5 |  | WellDoc
---
BlueStar†
General hospital |  | Diabetes
---
management
| Medical
---
practitioners,
patients
| Darknet-53
---
CNN
Numeric |  Lal et al. [4] I |  | Wrong
---
diagnosis
6 |  d-Nav System† | General hospital |  | Insulin dose
---
predictor
| Medical
---
practitioners,
patients
| Multi-layer
---
perception (MLP)
and LSTM
Numeric |  Zhou et al. [30] I |  | Wrong
---
treatment
(Fatal)
7 |  | MBT-CA
---
System‡
Microbiology | Spectometry |  | Medical
---
practitioners
DNN ∗ | Numeric |  | Meiseles
---
et al. [5] I
| Wrong
---
diagnosis
(Fatal)
8 |  KIDScore D3† |  | Obstetrics &
---
Gynaecology
| Embryo image
---
assessment
| Medical
---
practitioners
| Decentralized
---
federated
learning
Image |  Nguyen et al. [31] P |  | Wrong
---
diagnosis
9 |  | NuVasive
---
Pulse System‡
Orthopedic |  | Neurological
---
monitoring
| Medical
---
practitioners
CNN ∗ | Image |  Joel et al. [6] I |  | Mistake in
---
surgery (Fatal)
10 |  ABMD Software† | Radiology |  | Bone
---
densitometer
| Medical
---
practitioners
| Inception-v3
---
and
Densenet-121 ∗
Image |  Bortsova et al. [7] I |  | Wrong
---
diagnosis
11 |  | Deep Learning
---
Image
Reconstruction†
Radiology |  | X-ray
---
reconstruction
| Medical
---
practitioners
ResNet-18 | Image |  |  Menon et al. [8] T
---
Paul et al. [32] I
| Wrong
---
diagnosis
12 |  Air Next‡ | Anesthesiology | Spirometer |  | Medical
---
practitioners
| CatBoost
---
ResNet-50 ∗
Image |  Vargas et al. [9] I |  | Wrong
---
diagnosis
13 |  | One Drop
---
Blood Glucose
Monitoring
System‡
| Clinical
---
Chemistry
| Diabetes
---
management
Patients |  | MLP
---
Numeric |  | Levy-Loboda
---
et al. [10] I
| Wrong
---
treatment
(Fatal)
14 |  | OTIS 2.1 and
---
THiA Optical
Coherence
Tomography
System‡
| General and
---
Plastic Surgery
| Human tissue
---
imaging
| Medical
---
practitioners
| Support Vector
---
Machines
(SVM)
Image |  Ma et al. [16] I |  | Wrong
---
diagnosis
15 |  | EarliPoint
---
System‡
Neurology |  | Diagnosis of
---
Pediatric
Autism Spectrum
Disorder
| Medical
---
practitioners
| Graph Neural
---
Network
(GNN)
Image |  Chen et al. [11] T |  | Wrong
---
diagnosis
16 |  BrainScope TBI‡ | Neurology |  | Brain injury
---
assessment
| Medical
---
practitioners
| CNN +
---
Recurrent neural
networks (RNN)
Numeric |  Yu et al. [12] I |  | Wrong
---
treatment
(Fatal)
17 |  IDx-DR v2.3† | Ophthalmic |  | Diabetic
---
Retinopathy
Detection
| Medical
---
practitioners
| Federated
---
learning
Image |  Nielsen et al. [13] I |  | Wrong
---
diagnosis
(loss of vision)
18 |  | Iris Intelligent
---
Retinal Imaging
System†
Ophthalmic |  | Storage,
---
management and
display of retinal
images
| Medical
---
practitioners
DNN | Image |  | Mangaokar
---
et al. [14] I
| Wrong
---
diagnosis
(loss of vision)
19 |  Paige Prostate† | Pathology |  | Cancer
---
diagnosis
| Medical
---
practitioners
CNN | Numeric |  Hu et al. [15] T |  | Wrong
---
treatment (Fatal)
20 |  | Tissue of Origin
---
Test Kit‡
Pathology |  | Malignant Tumor
---
diagnosis
| Medical
---
practitioners
SVM | Image |  Ma et al. [16] I |  | Wrong
---
treatment (Fatal)
TABLE I: A study of different FDA-Approved ML-enabled medical devices and
their security vulnerabilities
$\dagger$: Software as medical device, $\ddagger$: Software in medical device,
∗: Best-guessed ML algorithm,
T: Training-time attack, I: Inference-time attack, P: Privacy attack

We attempt to understand how an adversary can exploit the vulnerabilities
identified in the previous section. As shown in Table I, all the identified ML
attacks require the adversary to manipulate the readings of peripheral sensor
devices either during training or at the time of inference. Therefore, for
each ML-enabled device, we look for compatible peripheral devices and
communication media that would enable an adversary to introduce malicious data
into the ML engine. We also examine the risk assessments performed by the
manufacturers of the ML-enabled devices (mentioned in their Premarket
Notifications) to understand their adequacy (or the lack thereof) in
preventing such security risks. Table II presents this study.

Survey Process. For each ML-approved medical device, we first identify its
compatible peripheral sensor devices, communication media, and other
operational details (e.g., type of device on which the ML-enabled software can
be installed) by manually reading through its Premarket Notification summary
on the FDA website [1] and information brochure on the manufacturer website.
Then, for each potential attack point, we look for known attacks and
vulnerabilities by searching research papers and vulnerability databases [33,
34].

We list at least one vulnerability that allows an adversary to either
eavesdrop or inject malicious data into the ML engine. This list of
vulnerabilities is not comprehensive. Rather, we aim to highlight at least one
vulnerability in the peripheral devices to motivate the risk analysis
technique.

Insights. We summarize the insights from this study below.

  1. 1.

We found known vulnerabilities in the peripheral devices compatible with
several ML-enabled devices. While most vulnerabilities affect only a small
group of devices, a few vulnerabilities affect all devices of a certain type.
For instance, the Conexus telemetry protocol vulnerability [35] only affects
the ECG monitors from Medtronic. However, another attack [36] affects all
infrared-sensitive cameras.

  2. 2.

Some vulnerabilities (e.g., [35] for the Cardiologs ECG Analysis Platform, and
[36] for Oxehealth Vital Signs) require the adversary to execute the attack
locally as they have to be within the Bluetooth [35] communication range, or
within the range of infrared light emission [36]. Such attacks can be executed
by insiders or by breaching the physical security of the hospital or the
patient’s home.

  3. 3.

Many of the vulnerabilities can be exploited remotely (e.g., [37] for
Oxehealth and [38] for the IDx-DR)over the Internet. Since connectivity to the
Internet is mandatory for these devices, preventing remote attacks is
challenging.

  4. 4.

In some cases the attack path is difficult to identify. For instance, the IDx-
DR software takes inputs from the Topcon NW200 Fundus camera. Although we did
not find any known vulnerability in the camera, we found that the camera comes
packaged with a computer that runs Windows 7 OS by default. Thereafter, we
found known vulnerabilities [38] in Windows 7 that would allow an adversary to
inject malicious inputs into the ML engine. While it is likely that updates
for Windows 7 would be issued to patch such vulnerabilities, security updates
are not routinely applied to medical devices. In one case [38], the vendor
decided not to even release a patch since they thought most users would
upgrade to Windows 10.

  5. 5.

We did not find any known vulnerability in the peripheral devices for some
systems (e.g., SOZO, WellDoc Bluestar, and Air Next). However, many of these
systems use Bluetooth, Internet communications, and web services. Adversaries
can exploit vulnerabilities [21] in these communication channels for injecting
adversarial inputs.

  6. 6.

We found that many of the ML-enabled device manufacturers (e.g., the NuVasive
Pulse System) do not perform any security evaluation, and only focus on
accuracy and safe operating conditions (e.g., protecting the devices from
electrical hazards). Even if they consider security, the manufacturers rarely
consider the peripheral devices. For instance, the developers of the IDx-DR
software evaluate the software for various security risks, but not its
peripheral device, the Topcon NW200 Fundus Camera. However, security
evaluation of the software alone is insufficient. This is because an adversary
might execute the inference-time attack [13] shown in Table I by exploiting
the vulnerability in the camera [38] to install malware that manipulates the
images that are sent to the input of the ML engine.

| Sl.
---
No.
| Device
---
Name
| Risk assessment guideline followed
---
| Known attacks and
---
vulnerabilities in compatible
peripheral sensor devices
| Connected to
---
the Internet
or Bluetooth ?
1 |  | CardioLogs ECG Analysis
---
Platform†
| Inadequate information - Acknowledges the need for
---
cybersecurity of cloud-based software
|  Portable ECG Monitors - {[35]} L
---
| Cellular network,
---
Bluetooth
2 |  Oxehealth Vital Signs† |  | Guidance for the Content of Premarket Submissions for
---
Management of Cybersecurity in Medical Devices [39]
| Infra-red sensitive cameras
---
\- [36] L, {[37]} R
Intranet / Internet
3 |  GI Genius‡ |  | Moderate level of concern as defined in the
---
“Guidance for the Content of Premarket Submissions
for Software Contained in Medical Devices.” [39]
|  Endoscope cameras - {[40]} R
---
Intranet / Internet
4 |  SOZO‡ | None |  | No third-party peripheral device used
---
| Bluetooth,
---
Intranet/Internet
5 |  WellDoc BlueStar† |  | Guidance for the Content of Premarket Submissions for
---
Management of Cybersecurity in Medical Devices [39]
| No vulnerability identified in peripheral
---
sensor devices
| Bluetooth,
---
Cloud Service API
6 |  d-Nav System† |  | Guidance for the Content of Premarket Submissions for
---
Management of Cybersecurity in Medical Devices [39]
| No vulnerability identified in peripheral
---
sensor devices
Cloud Service API
7 |  MBT-CA System‡ | None |  | No third-party peripheral device used
---
No
8 |  KIDScore D3† | None |  | No vulnerability identified in peripheral
---
sensor devices
Intranet / Internet
9 |  NuVasive Pulse System‡ | None |  | Infra-red sensitive cameras
---
\- [36] L, {[37]} R
Internet
10 |  ABMD Software† | None |  | No vulnerability identified in peripheral
---
sensor devices
Unknown
11 |  | Deep Learning
---
Image Reconstruction†
None |  |  X-ray machines \- {[41]} R
---
Unknown
12 |  Air Next‡ | None |  | No third-party peripheral device used
---
| Bluetooth,
---
Internet
13 |  | One Drop Blood Glucose
---
Monitoring System
None |  | No third-party peripheral device used
---
| Bluetooth,
---
Internet
14 |  | OTIS 2.1 and THiA Optical
---
Coherence Tomography System
|  ANSI AAMI ISO 14971:2007/(R)2010 [42],
---
IEC 62304:2006/A1:2015 [43]
| No third-party peripheral device used
---
Unknown
15 |  EarliPoint System‡ | None |  | Webcams installed on personal
---
computers - [44] R
Internet
16 |  BrainScope TBI‡ | None | No third-party peripheral devices used | Internet
17 |  IDx-DR v2.3† |  | Considers security concerns related to data
---
confidentiality, integrity, availability, denial of service
attacks and malware. Risks related to the failure of various
software components and their potential impact on
patient reports were also adequately addressed [45].
| This device uses the Topcon NW200
---
Fundus camera, which comes packaged
with a PC running Windows 7 OS. The
Windows 7 OS has known
vulnerabilities. \- {[38]} R
Internet
18 |  | Iris Intelligent Retinal Imaging
---
System†
Ensures HIPAA [46] compliance |  | Retinal cameras such as Topcon
---
NW200 \- Same vulnerable peripherals
as in the case of IDx-DR v2.3
Internet
19 |  Paige Prostate† |  | Considers software security as per “Content of Premarket
---
Submissions for Management of Cybersecurity in
Medical Devices: Guidance for Industry and Food and
Drug Administration Staff". Also encrypts the
communication between the device and servers.
|  Medical scanners \- {[47]} L
---
Internet
20 |  Tissue of Origin Test Kit‡ | None |  | No third-party peripheral device used
---
Internet
TABLE II: Known vulnerabilities in peripheral devices and communication media
compatible with FDA-approved ML-enabled medical devices. L: Locally
exploitable only, R: Remotely exploitable

##  IV Case Study

We present a case study to demonstrate the security risks in the ML-enabled
BGMS described in Section II-B. We show a practical attack on the BGMS in
which the attacker exploits the vulnerabilities in connected devices to
negatively affect the predictions of the ML-enabled decision-making component.

###  IV-A Attack Description

Adversarial Goal. The attacker’s objective is to endanger a targeted patient’s
life by causing the ML model to misdiagnose the patient’s condition, and
thereby suggest a wrong insulin dose, in the near future. While small
prediction errors are harmless, a significant error in prediction might result
in life-threatening consequences. In this case study, we consider an attacker
whose goal is to make the model predict a high blood glucose level
(hyperglycemia) for the targeted patient when in reality the patient has a low
blood glucose level (hypoglycemia) or a normal blood glucose level. If the
attacker succeeds, the BGMS would mistakenly infer that more insulin needs to
be infused into the patient’s body, thus causing the patient’s glucose level
to drop significantly lower than the normal level, with consequences ranging
from wrong diagnosis (e.g., in case of the WellDoc BlueStar system) to wrong
treatment (e.g., in the case of d-Nav and One Drop BGMS systems) which is
potentially fatal.

Adversarial Capabilities. We assume the attacker has reasonable and realistic
capabilities, wherein they can only tamper with the CGM measurements.
Manipulating the manually entered finger-based glucose readings, carbohydrate
intake, and bolus dose is beyond the attacker’s capabilities. Moreover, the
attacker is oblivious of the structure and parameters of the underlying ML
model (black box attack). Further, the attacker does not have access to the
training set. The attacker can exploit the Bluetooth communication stack [21]
to intercept and manipulate the CGM measurements. This is because FDA-approved
diabetes management devices (e.g., One Drop and WellDoc BlueStar) use
Bluetooth communication to transmit the collected measurements.

Attack Strategy. The attacker wants the ML model to misdiagnose the patient as
hyperglycemic by pushing the predicted blood glucose levels toward the
hyperglycemic range. Therefore, the attacker modifies blood glucose levels
that are either hypoglycemic or normal to values that are above 125 mg/dL
(hyperglycemic while fasting) or above 180 mg/dL (hyperglycemic postprandial).
To accomplish this, the attacker manipulates the CGM readings for a time
duration so that the BGMS app misdiagnoses the blood glucose level of the
patient. Deciding the minimum time duration for which the attacker needs to
manipulate the readings and the extent to which she should deviate them,
requires careful consideration.

###  IV-B Experimental Setup

We first present the ML model used in the BGMS setup, followed by a
description of the dataset used for our experiments. Next, we describe the
Universal Robustness Evaluation Toolkit (URET) [48], used for generating the
adversarial inputs.

Targeted ML model. Since the details of the actual ML algorithm used in the
Dreamed Advisor Pro app (described in Section II-B) are confidential, we
approximated it using a time-series prediction model developed by Rubin-
Falcone et al. [20]. This model uses a bidirectional long short-term memory
(LSTM) recurrent neural network (RNN) architecture, and uses root mean square
error (RMSE) and mean absolute error (MAE) to evaluate the prediction
accuracy. Intuitively, both RMSE and MAE represent how far away the predicted
glucose levels are from the actual glucose levels. The farther the distance,
the worse the prediction. Further, we selected a target model that uses a
similar neural network as the FDA-approved d-Nav System [49] in Tables I and
II (i.e., LSTM).

Rubine-Falcone et al. [20] built two models - (i) a personalized model for
each patient trained on the patient’s data, and (ii) an aggregate model
trained on the data of all patients. They exhibit average RMSEs of 18.2 and
31.7 and average MAEs of 12.8 and 23.6 on the 30 and 60-minute horizons,
respectively.

Dataset. To demonstrate the impact of adversarial inputs on the predictions of
the targeted ML model, we use the 2020 OhioT1DM dataset [50]This was used by
the target model developers [20] for evaluating its accuracy. The dataset
comprises physiological measurements of six patients, all of whom have Type-1
diabetes. The main features are CGM blood glucose measurements, finger-based
measurements, basal insulin, bolus dose, carbohydrate intake, heart rate, and
other activity features like sleeping patterns and acceleration, besides other
physiological, and self-reported life-event features. The dataset spans eight
weeks and consists of approximately 10000 samples for training, and 2500 for
testing, both recorded at approximately 5-minute intervals per patient.

Universal Robustness Evaluation Toolkit (URET). In Table I, we found three
diabetes management devices – WellDoc BlueStar, d-Nav, and OneDrop – that are
vulnerable to inference-time attacks. Hence, we decided to launch an evasion
attack against our target model, which manipulates data points at inference
time by introducing slight perturbations to their original values to prevent
detection by the ML model or cause misclassification [51]. To demonstrate an
attack on a BGMS, we use URET which is an evasion attack framework developed
by Eykholt et al. [48] for generating adversarial inputs. Unlike most evasion
attack techniques that focus on the image classification domain [52, 53, 54,
32], URET is a general-purpose framework that can generate adversarial inputs
regardless of the data types or task domain. URET takes in a benign input
instance and a set of pre-defined input transformations and attempts to find a
sequence of transformations (e.g., increment or replace glucose level)
resulting in an adversarial input that is both semantically and functionally
correct. The URET framework is compatible with a wide variety of data types
and domains and allows the user to specify which feature values to manipulate,
and bounds and constraints on the feature values. As URET allows specifying
constraints on the feature values, we can ensure the generated adversarial
samples abide by victims’ physiological limits.

Since we assume the attacker can only modify CGM values, we specify the CGM
feature indices in URET’s configuration file as the only modifiable features.
To ensure that adversarial CGM values respect the physiological levels, we
constrain them to be between 125 and 499 mg/dL for fasting levels, since a
hyperglycemic glucose level in a fasting state should exceed 125 mg/dL, and
between 180 and 499 mg/dL for postprandial levels, since a hyperglycemic
glucose level in a postprandial state should exceed 180 mg/dL (499 mg/dL is
the highest reported glucose level in the OhioT1DM dataset). Further, since
there are missing physiological measurements at specific timestamps in the
dataset, especially CGM measurements, we specify the relationship between CGM
glucose and a feature called “missing” as a dependency. This ensures the ML
model does not interpret missing CGM values as zero values.

In addition, the attacker can observe other feature values (e.g., bolus dose),
which in turn play a role in adversarial data generation, since the attacker
needs read access to those features to use the loss function to rank the
adversarial input.

###  IV-C Results

As mentioned, the target models use RMSE and MAE to evaluate the prediction
accuracy. Because the two metrics show similar behaviors, we only consider the
RMSE results. The RMSE results are shown in Figure 2. The figure shows the
performance of personalized patient models and the aggregate model trained on
all patients’ data (‘All patients’) before and after the URET attacks while
fasting and postprandial. The maximum RMSE value for the benign models is
around 21 mg/dL, while the average RMSE across all 7 benign models is around
18.3 mg/dL. The figure also shows the RMSE values after the attack for both
fasting ($>$125 mg/dL) and postprandial ($>$180 mg/dL) hyperglycemic glucose
levels. Thus, the RMSE values increase regardless of the used threshold since
glucose values are driven further away from the actual values.

Moreover, postprandial RMSE values are consistently higher than fasting RMSE
values due to the use of a higher threshold for the lower bound of the
adversarial input (i.e., 180 mg/dL instead of 125 mg/dL). The hike in RMSE
values between the benign and the attacked models, whether fasting or
postprandial, shows a farther distance between the actual blood glucose levels
and the glucose levels predicted by the ML models, implying less accuracy. A
less accurate insulin dose means more insulin is infused into the patient’s
body than needed, which is potentially fatal.

Figure 2: RMSE of the benign model, the model attacked with fasting
hyperglycemic blood glucose levels, and with postprandial hyperglycemic blood
glucose levels.

Figures 3 and 4 show the percentage of instances that are misdiagnosed as
hyperglycemic while actually being normal and hypoglycemic, respectively, for
both the fasting and postprandial attack scenarios. A higher misclassification
percentage demonstrates more susceptibility to the respective attack scenario,
while a lower percentage implies more difficulty in crafting successful
attacks. We make three observations from the two figures. First, URET achieves
considerably high attack success rates reaching up to 100% in some cases. On
average, URET achieves similar attack success rates in both normal to
hyperglycemic, and hypoglycemic to hyperglycemic attack scenarios. This
indicates that the attack generated by URET remains powerful regardless of the
initial glucose levels. Second, the attack success rate is always higher when
the patient is fasting compared to postprandial. This implies that it is
easier for the attacker to attack a patient in a fasting state. This is
because a smaller amount of perturbation is performed when increasing the CGM
glucose from the original value to fasting hyperglycemic levels as opposed to
postprandial hyperglycemic levels. This confirms that URET is more successful
when the perturbation margin is smaller. Third, some patients are more
resilient to the URET attack than others. For example, Figure 3 shows that the
attack success rate of misdiagnosing a patient from normal glucose to
hyperglycemic is the lowest with patient 2 (67.4% fasting, and 44.2%
postprandial) and is the highest with patient 5 (100.0% fasting, and 97.9%
postprandial) suggesting that it is more challenging for URET to attack
patient 2 compared to patient 5. Similarly, Figure 4 shows that the attack
success rate of misdiagnosing a patient from hypoglycemic to hyperglycemic is
the lowest with patient 3 (72.4% fasting, and 28.0% postprandial) and is the
highest with patient 5 (100.0% fasting, and 100.0% postprandial) suggesting
that patient 5 is more vulnerable compared to patient 3. We hypothesize that
this is because the rich history of some patients leads to a more
generalizable prediction model that is more difficult to attack.

Figure 3: Percentage of originally normal glucose instances that are
misclassified as hyperglycemic. Figure 4: Percentage of originally
hypoglycemic glucose instances that are misclassified as hyperglycemic.

Summary. We demonstrated an attack where an attacker compromises the Bluetooth
communication channel between the CGM peripheral device and the mobile app
enabling them to manipulate the measured glucose levels. The results of our
attacks show that the attacker can cause the ML model to misdiagnose the
patient’s medical condition as hyperglycemic. In best-case scenarios, this
leads to a wrong diagnosis, and in worst-case scenarios, it has fatal
consequences.

##  V Risk Assessment Techniques Evaluation

We assess the state-of-the-art risk assessment techniques, investigating both
their strengths and limitations. We start by delving into how these methods
benefit risk analysis processes while pointing out areas where they fall short
using the BGMS attack example. Finally, we highlight the need for a new risk
assessment framework for ML-enabled connected medical systems.

###  V-A A Survey of Existing Techniques

DREAD. Microsoft introduced a comprehensive risk rating system known as DREAD
[55] for assessing security risks in web applications. It evaluates risks
based on damage potential, reproducibility, exploitability, affected users,
and discoverability. DREAD was primarily designed for conventional web
applications. Using it to assess risks in ML-enabled connected medical systems
is not possible, since DREAD does not describe how security risks in the
individual connected components might pose a risk to the overall system.

STRIDE. Microsoft also introduced another qualitative model known as STRIDE
[55], aimed at identifying and categorizing threats faced by the web
applications. This model covers six attack types: Spoofing, Tampering,
Repudiation, Information Disclosure, Denial of Service (DoS), and Elevation of
Privilege, each with corresponding countermeasures to mitigate risks. However,
STRIDE cannot be used for an end-to-end risk assessment in ML-based medical
systems as it does not consider an adversary who exploits peripheral devices.

FMEA. Failure Modes and Effects Analysis (FMEA) is a foundational analytical
technique to detect and mitigate potential risks [56]. This method involves a
detailed examination of system components to identify potential causes of
failure and their impact on system stability. However, it falls short in
providing end-to-end risk assessment for connected ML-based medical systems as
it primarily focuses on individual component failures, neglecting the broader
systemic implications of how vulnerabilities propagate throughout the entire
pipeline.

Method |  | Damage
---
Potential
Exploitability |  | Affected
---
Users
Detectability |  | Ease of
---
Mitigation
DREAD | ✓ | ✓ | ✓ | ✓ | ✗
STRIDE | ✓ | ✗ | ✗ | ✗ | ✗
FMEA | ✓ | ✗ | ✗ | ✓ | ✗
CVSS | ✓ | ✓ | ✗ | ✗ | ✗
| Other Works
---
[57, 58]
✓ | ✓ | ✗ | ✗ | ✗
TABLE III: Comparison of factors considered in existing risk analysis
frameworks

CVSS. The Common Vulnerability Scoring System (CVSS) [59] is an open framework
designed by FIRST.Org, Inc. to capture the severity level of software
vulnerabilities. CVSS comprises three metric groups: Base, Temporal, and
Environmental, capturing different vulnerability characteristics. It is a
scoring system, not a risk assessment model [59], but helps prioritize threats
across system components through context-specific choices by the risk
management team. However, it lacks uniformity in prioritizing metrics,
especially in ML-enabled medical devices where the impact of availability loss
could be life-threatening, unlike in web applications where it may only result
in user dissatisfaction or financial consequences [60].

FDA-approved Security Standards for Medical Devices. The FDA has formulated a
cybersecurity guideline aimed at assisting the industry in recognizing
cybersecurity-related issues in medical devices [39]; however, their central
emphasis is on the communication between medical devices and IT networks,
along with the attendant risks. As a result, dealing with vulnerabilities in
AI/ML-enabled medical devices that operate independently of IT networks,
remains a challenge.

Academic Research Efforts. There has been a rich literature on performing risk
assessment within the healthcare sector [57, 58]. However, in the context of
ML-enabled connected medical devices, evaluations often lack three key
aspects: (1) impact on affected users (more users affected, higher risk), (2)
ease of vulnerability detection (easier detection, lower risk), and (3) ease
of post-exploitation mitigation, including available remediation levels and
responsible entities. An overview of how the existing methods incorporate
these factors in the risk analysis process is presented in Table III.

Risk assessment of ML algorithms. While companies typically conduct
preliminary risk evaluations for their ML models, they do not analyze the
security risks associated with deploying the models in a connected healthcare
environment [61]. Rather, their concern is ensuring the ML model is trained on
an unbiased dataset, and evaluated using a diverse dataset.

###  V-B Limitations of Existing Techniques

We use the ML-enabled BGMS as an example to demonstrate the limitations of
existing techniques in adequately addressing the factors required for an ideal
risk analysis. We find that all methods fall short in analyzing the risk
factors.

DREAD. In the case of the BGMS, DREAD fails to assess the realistic level of
potential damage if a peripheral device like the glucose meter or smartphone
has been compromised. In these instances, the attacker gains the ability not
only to access the data but also to manipulate it. Consequently, both the
integrity and confidentiality of the system are at risk, but DREAD cannot
differentiate between them accurately.

STRIDE. There are many different methods by which an attacker can manipulate
data before it is fed into the ML engine. Nevertheless, according to the
STRIDE model, all these methods are categorized simply as Tampering, without
providing additional insights into how these manipulations were performed.

FMEA. Using FMEA in the process of addressing the risks in the BGMS could
potentially handle the risk of peripheral devices like glucose meters or
smartphones being compromised. However, FMEA does not offer insights into the
risks associated with the propagation of this vulnerability within the system
or its potential impact on the ML model’s predictions.

CVSS. When a vulnerability impacts the availability of the BGMS, the CVSS
assigns the same risk level as it would for a similar incident in other
domains like web applications. This approach is not appropriate, as an
availability issue in the BGMS could potentially lead to irreversible harm to
patients.

FDA-approved Security Standards for Medical Devices. Encountering physical
attacks like electromagnetic interference [24] is independent of the
connectivity between the glucose meter and the IT network. Hence, the FDA-
approved security standards also fall short of an ideal risk analysis.

###  V-C Need for a New Risk Assessment Framework

Figure 5: Security risk assessment techniques by manufacturers of FDA-approved
ML-enabled medical systems based on [39]

We studied the kinds of risk assessment done by manufacturers of ML-enabled
medical devices - the results are in Figure 5. We find that more than 80% of
manufacturers of ML-enabled medical devices either do not mention anything
about security risk assessment in their documentation, or use inadequate means
to do so. Another 5% use proprietary mechanisms, and hence we cannot judge the
adequacy of their assessment. The remaining 12% of the devices use existing
risk assessment techniques, which as we have seen, are not sufficient for
assessing the severity of the security risks in ML-enabled connected medical
systems. Therefore, risk assessment of ML-enabled medical devices remains an
open challenge. Developing such a risk assessment technique would require
bridging the domains of cybersecurity and medicine.

##  VI Conclusion and Future Research Directions

We presented a detailed study of security risks associated with modern AI/ML-
enabled medical devices, mainly due to vulnerabilities in connected peripheral
devices. We conducted a systematic security analysis of commercial AI/ML-
enabled devices that were approved by the U.S. Food and Drug Administration
(FDA). Our analysis shows that many of these devices are vulnerable to
existing adversarial attacks, which raises concerns about the appropriateness
of using such safety-critical devices on patients in the real world. We
validate our analysis by performing a case study, where we execute a realistic
adversarial attack on an ML-enabled blood glucose monitoring system. Through
this case study, we identify security risks in the glucose monitoring system.
Further, we studied state-of-the-art risk assessment frameworks to highlight
their drawbacks in identifying security risks in connected ML-enabled medical
systems, and the need for a new framework.

Our work opens up three interesting future work directions – (1) Automated
risk identification: Automating the risk identification process at scale would
benefit device manufacturers as well as the security research community. This
would require identifying relevant documents on the web and parsing a huge
volume of unstructured documents, while at the same time being able to relate
various ML concepts; (2) Building personalized spatial and temporal risk
profiles per patient: Our case study shows that attacks on ML-enabled medical
systems cause more damage to certain patients than others. Moreover, a patient
is not equally vulnerable at all points of time. An interesting research
problem is to study patients’ data in more detail to develop customized
spatial and temporal risk profiles for every patient; and, (3) Efficient risk
mitigation techniques: This involves designing attack-resilient ML models,
determining accountable entity and enforcing accountability in risk
mitigation, accounting for the costs and deployment scenario.

NOTE: We will make our code and datasets publicly available if accepted

## References

  * [1] U.S. Food & Drug Administration.  Artificial Intelligence and Machine Learning (AI/ML)-Enabled Medical Devices.  URL: https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-and-machine-learning-aiml-enabled-medical-devices, Last accessed: Nov 30, 2023.
  * [2] Huangxun Chen, Chenyu Huang, Qianyi Huang, Qian Zhang, and Wei Wang.  Ecgadv: Generating adversarial electrocardiogram to misguide arrhythmia classification system.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 3446–3453, 2020.
  * [3] Albatul Albattah and Murad A Rassam.  Detection of adversarial attacks against the hybrid convolutional long short-term memory deep learning technique for healthcare monitoring applications.  Applied Sciences, 13(11):6807, 2023.
  * [4] Sheeba Lal, Saeed Ur Rehman, Jamal Hussain Shah, Talha Meraj, Hafiz Tayyab Rauf, Robertas Damaševičius, Mazin Abed Mohammed, and Karrar Hameed Abdulkareem.  Adversarial attack and defence through adversarial training and feature fusion for diabetic retinopathy recognition.  Sensors, 21(11):3922, 2021.
  * [5] Amiel Meiseles, Ishai Rosenberg, Yair Motro, Lior Rokach, and Jacob Moran-Gilad.  Adversarial vulnerability of deep learning models in analyzing next generation sequencing data.  In 2020 IEEE International Conference on BIBM, pages 464–468, 2020.
  * [6] Marina Z Joel, Sachin Umrao, Enoch Chang, Rachel Choi, Daniel Yang, James Duncan, Antonio Omuro, Roy Herbst, Harlan M Krumholz, Sanjay Aneja, et al.  Adversarial attack vulnerability of deep learning models for oncologic images.  MedRxiv, 2021.
  * [7] Gerda Bortsova, Cristina González-Gonzalo, Suzanne C Wetstein, Florian Dubost, Ioannis Katramados, Laurens Hogeweg, Bart Liefers, Bram van Ginneken, Josien PW Pluim, Mitko Veta, et al.  Adversarial attack vulnerability of medical image analysis systems: Unexplored factors.  Medical Image Analysis, 73:102141, 2021.
  * [8] Karthika Menon, V Khushi Bohra, Lakshana Murugan, Kavya Jaganathan, and Chamundeswari Arumugam.  Covid-19 diagnosis from chest x-ray images using convolutional neural networks and effects of data poisoning.  In Computational Science and Its Applications–ICCSA 2021: 21st International Conference, Cagliari, Italy, September 13–16, 2021, Proceedings, Part IX 21, pages 508–521. Springer, 2021.
  * [9] Danilo Vasconcellos Vargas and Jiawei Su.  Understanding the one-pixel attack: Propagation maps and locality analysis.  In CEUR Workshop Proceedings, volume 2640. CEUR-WS, 2020.
  * [10] Tamar Levy-Loboda, Eitam Sheetrit, Idit F Liberty, Alon Haim, and Nir Nissim.  Personalized insulin dose manipulation attack and its detection using interval-based temporal patterns and machine learning algorithms.  Journal of Biomedical Informatics, 132:104129, 2022.
  * [11] Yuzhong Chen, Jiadong Yan, Mingxin Jiang, Tuo Zhang, Zhongbo Zhao, Weihua Zhao, Jian Zheng, Dezhong Yao, Rong Zhang, Keith M Kendrick, et al.  Adversarial learning based node-edge graph attention networks for autism spectrum disorder identification.  IEEE Transactions on Neural Networks and Learning Systems, 2022.
  * [12] Jianfeng Yu, Kai Qiu, Pengju Wang, Caixia Su, Yufeng Fan, and Yongfeng Cao.  Perturbing BEAMs: EEG adversarial attack to deep learning models for epilepsy diagnosing.  BMC Medical Informatics and Decision Making, 23(1):115, 2023.
  * [13] Christopher Nielsen, Anup Tuladhar, and Nils D Forkert.  Investigating the vulnerability of federated learning-based diabetic retinopathy grade classification to gradient inversion attacks.  In International Workshop on Ophthalmic Medical Image Analysis, pages 183–192. Springer, 2022.
  * [14] Neal Mangaokar, Jiameng Pu, Parantapa Bhattacharya, Chandan K Reddy, and Bimal Viswanath.  Jekyll: Attacking medical image diagnostics using deep generative models.  In 2020 IEEE EuroS&P, pages 139–157. IEEE, 2020.
  * [15] Lei Hu, Da-Wei Zhou, Xiang-Yu Guo, Wen-Hao Xu, Li-Ming Wei, and Jun-Gong Zhao.  Adversarial training for prostate cancer classification using magnetic resonance imaging.  Quantitative Imaging in Medicine and Surgery, 12(6):3276, 2022.
  * [16] Xingjun Ma, Yuhao Niu, Lin Gu, Yisen Wang, Yitian Zhao, James Bailey, and Feng Lu.  Understanding adversarial attacks on deep learning based medical image analysis systems.  Pattern Recognition, 110:107332, 2021.
  * [17] U.S. Food & Drug Administration.  DreaMed Advisor Pro.  URL: https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfpmn/denovo.cfm?id=DEN170043, Last accessed: Nov 30, 2023.
  * [18] Michael Kahn.  Diabetes.  UCI Machine Learning Repository.  DOI: https://doi.org/10.24432/C5T59G.
  * [19] MIchelle Mouri and Madhu Badireddy.  Hyperglycemia.  URL: https://www.ncbi.nlm.nih.gov/books/NBK430900/, Last accessed: Nov 30, 2023.
  * [20] Harry Rubin-Falcone, Ian Fox, and Jenna Wiens.  Deep Residual Time-Series Forecasting: Application to Blood Glucose Prediction.  In KDH@ ECAI, pages 105–109, 2020.
  * [21] Kasper Rasmussen.  BLURtooth: Exploiting Cross- Transport Key Derivation in Bluetooth Classic and Bluetooth Low Energy.  In AsiaCCS. ASIA Conference on Computer and Communications Security, 2022.
  * [22] Dreamed Diabetes Ltd.  DreaMed Advisor Pro: Manual For Personal Use iOS.  URL: https://dreamed-diabetes.com/wp-content/uploads/2019/06/Dreamed-Advisor-Pro-iOS-Patient-IFU.pdf, Last accessed: Nov 30, 2023.
  * [23] Guillaume Dupont, Daniel Ricardo dos Santos, Elisa Costante, Jerry Den Hartog, and Sandro Etalle.  A matter of life and death: analyzing the security of healthcare networks.  In SEC 2020: ICT Systems Security and Privacy Protection, pages 355–369. Springer, 2020.
  * [24] SMJ Mortazavi, M Gholampour, M Haghani, G Mortazavi, and AR Mortazavi.  Electromagnetic radiofrequency radiation emitted from GSM mobile phones decreases the accuracy of home blood glucose monitors.  Journal of Biomedical Physics & Engineering, 4(3):111, 2014.
  * [25] Tim Hulsen, Saumya S Jamuar, Alan R Moody, Jason H Karnes, Orsolya Varga, Stine Hedensted, Roberto Spreafico, David A Hafler, and Eoin F McKinney.  From big data to precision medicine.  Frontiers in medicine, 6:34, 2019.
  * [26] Spiros V Georgakopoulos, Dimitris K Iakovidis, Michael Vasilakakis, Vassilis P Plagianakos, and Anastasios Koulaouzidis.  Weakly-supervised convolutional learning for detection of inflammatory gastrointestinal lesions.  In 2016 IEEE International conference on Imaging Systems and Techniques (IST), pages 510–514. IEEE, 2016.
  * [27] Amalia M Issa.  Personalized medicine and the practice of medicine in the 21st century.  McGill Journal of Medicine: MJM, 10(1):53, 2007.
  * [28] Muhammad Shahid Amin, Jamal Hussain Shah, Mussarat Yasmin, Ghulam Jillani Ansari, Muhamamd Attique Khan, Usman Tariq, Ye Jin Kim, and Byoungchol Chang.  A two stream fusion assisted deep learning framework for stomach diseases classification.  CMC-Comput. Mater. Contin, 73:4423–4439, 2022.
  * [29] Michal Byra, Grzegorz Styczynski, Cezary Szmigielski, Piotr Kalinowski, Lukasz Michalowski, Rafal Paluszkiewicz, Bogna Ziarkiewicz-Wroblewska, Krzysztof Zieniewicz, and Andrzej Nowicki.  Adversarial attacks on deep learning models for fatty liver disease classification by modification of ultrasound image reconstruction method.  In 2020 IEEE International Ultrasonics Symposium (IUS), pages 1–4, 2020.
  * [30] Xugui Zhou, Maxfield Kouzel, and Homa Alemzadeh.  Robustness testing of data and knowledge driven anomaly detection in cyber-physical systems.  In 2022 52nd Annual IEEE/IFIP International Conference on Dependable Systems and Networks Workshops (DSN-W), pages 44–51. IEEE, 2022.
  * [31] TV Nguyen, MA Dakka, SM Diakiw, MD VerMilyea, M Perugini, JMM Hall, and D Perugini.  A novel decentralized federated learning approach to train on globally distributed, poor quality, and protected private medical data.  Scientific Reports, 12(1):8888, 2022.
  * [32] Rahul Paul, Matthew Schabath, Robert Gillies, Lawrence Hall, and Dmitry Goldgof.  Mitigating adversarial attacks on medical image understanding systems.  In 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI), pages 1517–1521. IEEE, 2020.
  * [33] Mitre.  Common Vulnerabilities and Exposures.  URL: https://cve.mitre.org/, Last accessed: Nov 30, 2023.
  * [34] Information Technology Laboratory, USA.  National Vulnerability Database.  URL: https://nvd.nist.gov/vuln/search, Last accessed: Nov 30, 2023.
  * [35] Mitre.  Conexus Telemetry Protocol vulnerability.  URL: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-6538, Last accessed: Nov 30, 2023.
  * [36] Wei Wang, Yao Yao, Xin Liu, Xiang Li, Pei Hao, and Ting Zhu.  I can see the light: Attacks on autonomous vehicles using invisible lights.  In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, pages 1930–1944, 2021.
  * [37] Mitre.  Sony IPELA E Series Camera vulnerability (1).  URL: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-3938, Last accessed: Nov 30, 2023.
  * [38] Mitre.  Windows 7 vulnerability (2).  URL: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-5921, Last accessed: Nov 30, 2023.
  * [39] U.S. Food & Drug Administration.  Content of Premarket Submissions for Management of Cybersecurity in Medical Devices.  URL: https://www.fda.gov/media/86174/download, Last accessed: Nov 30, 2023.
  * [40] Mitre.  Shekar Endoscope vulnerability (1).  URL: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-10722, Last accessed: Nov 30, 2023.
  * [41] Mitre.  GE Healthcare Discovery vulnerability.  URL: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2014-7232, Last accessed: Nov 30, 2023.
  * [42] AAMI.  ANSI/AAMI/ISO 14971: 2007/(R) 2010, Medical devices—Application of risk management to medical devices.
  * [43] International Electrotechnical Commission et al.  Iec 62304: 2006/a1: 2015.  Medical device software-Software life-cycle processes, 2015.
  * [44] Mitre.  H264WebCam vulnerability.  URL: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2010-2349, Last accessed: Nov 30, 2023.
  * [45] U.S. Food & Drug Administration.  IDx-DR v2.3.  URL: https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfpmn/pmn.cfm?ID=K213037, Last accessed: Nov 30, 2023.
  * [46] George J Annas.  Hipaa regulations: a new era of medical-record privacy?  New England Journal of Medicine, 348:1486, 2003.
  * [47] Mitre.  Philips MRI 1.5T and MRI 3T vulnerability (1).  URL: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-26262, Last accessed: Nov 30, 2023.
  * [48] Kevin Eykholt, Taesung Lee, Douglas Schales, Jiyong Jang, and Ian Molloy.  URET: Universal Robustness Evaluation Toolkit (for Evasion).  In USENIX Security 23, pages 3817–3833, Anaheim, CA, August 2023. USENIX Association.
  * [49] U.S. Food & Drug Administration.  d-Nav System.  URL: https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfpmn/pmn.cfm?ID=K181916, Last accessed: Nov 30, 2023.
  * [50] Cindy Marling and Razvan Bunescu.  The OhioT1DM dataset for blood glucose level prediction: Update 2020.  In CEUR workshop proceedings, volume 2675, page 71. NIH Public Access, 2020.
  * [51] Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Šrndić, Pavel Laskov, Giorgio Giacinto, and Fabio Roli.  Evasion attacks against machine learning at test time.  In ECML PKDD 2013, Prague, Czech Republic, September 23-27, 2013, Proceedings, Part III 13, pages 387–402. Springer, 2013.
  * [52] Samuel G Finlayson, John D Bowers, Joichi Ito, Jonathan L Zittrain, Andrew L Beam, and Isaac S Kohane.  Adversarial attacks on medical machine learning.  Science, 363(6433):1287–1289, 2019.
  * [53] Samuel G Finlayson, Hyung Won Chung, Isaac S Kohane, and Andrew L Beam.  Adversarial attacks against medical deep learning systems.  arXiv preprint arXiv:1804.05296, 2018.
  * [54] Nicholas Carlini and David Wagner.  Towards evaluating the robustness of neural networks.  In 2017 IEEE Symposium on Security and Privacy (SP), pages 39–57. IEEE, 2017.
  * [55] J.D. Meier and Microsoft Corporation.  Improving Web Application Security: Threats and Countermeasures.  Patterns & practices. Microsoft, 2003.
  * [56] Hu-Chen Liu, Li-Jun Zhang, Ye-Jia Ping, and Liang Wang.  Failure mode and effects analysis for proactive healthcare risk evaluation: a systematic literature review.  Journal of evaluation in clinical practice, 26(4):1320–1337, 2020.
  * [57] Tom Mahler, Yuval Elovici, and Yuval Shahar.  A new methodology for information security risk assessment for medical devices and its evaluation.  arXiv preprint arXiv:2002.06938, 2020.
  * [58] Tahreem Yaqoob, Haider Abbas, and Narmeen Shafqat.  Integrated security, safety, and privacy risk assessment framework for medical devices.  IEEE journal of biomedical and health informatics, 24(6):1752–1761, 2019.
  * [59] R Abraham, D Arora, M Coles, M Eckert, M Heitman, A Manion, S Moore, S Romanowsky, K Scarfone, J Stuppi, et al.  Common vulnerability scoring system v3. 0: Specification document.  First, 2015.
  * [60] Jonathan Spring, Eric Hatleback, Allen Householder, Art Manion, and Deana Shick.  Time to Change the CVSS?  IEEE Security & Privacy, 19(2):74–78, 2021.
  * [61] Eric Wu, Kevin Wu, Roxana Daneshjou, David Ouyang, Daniel E Ho, and James Zou.  How medical ai devices are evaluated: limitations and recommendations from an analysis of fda approvals.  Nature Medicine, 27(4):582–584, 2021.