# MLHOps: Machine Learning for Healthcare Operations

Faiza Khan Khattak  Vallijah Subasri  Amrit Krishnan  Elham Dolatabadi  Deval
Pandya  Laleh Seyyed-Kalantari  Frank Rudzicz  Vector Institute for Artificial
Intelligence, Toronto, Ontario, Canada  Hospital for Sick Children, Toronto,
Ontario, Canada  University of Toronto, Toronto, Ontario, Canada  York
University, Toronto, Ontario, Canada  Dalhousie University, Halifax, Nova
Scotia, Canada

###### Abstract

Machine Learning Health Operations (MLHOps) is the combination of processes
for reliable, efficient, usable, and ethical deployment and maintenance of
machine learning models in healthcare settings. This paper provides both a
survey of work in this area and guidelines for developers and clinicians to
deploy and maintain their own models in clinical practice. We cover the
foundational concepts of general machine learning operations, describe the
initial setup of MLHOps pipelines (including data sources, preparation,
engineering, and tools). We then describe long-term monitoring and updating
(including data distribution shifts and model updating) and ethical
considerations (including bias, fairness, interpretability, and privacy). This
work therefore provides guidance across the full pipeline of MLHOps from
conception to initial and ongoing deployment.

###### keywords:

MLOps, Healthcare, Responsible AI

††journal: Journal of Biomedical Informatics

additions

##  1 Introduction

Over the last decade, efforts to use health data for solving complex medical
problems have increased significantly. Academic hospitals are increasingly
dedicating resources to bring machine learning (ML) to the bedside and to
addressing issues encountered by clinical staff. These resources are being
utilized across a range of applications including clinical decision support,
early warning, treatment recommendation, risk prediction, image informatics,
tele-diagnosis, drug discovery, and intelligent health knowledge systems.

There are various examples of ML being applied to medical data, including
prediction of sepsis [239], in-hospital mortality, prolonged length-of-stay,
patient deterioration, and unplanned readmission [218]. In particular, sepsis
is one of the leading causes of in-hospital deaths. A large-scale study
demonstrated the impact of an early warning system to reduce the lead time for
detecting the onset of sepsis, and hence allowing more time for clinicians to
prescribe antibiotics [8]. Similarly, deep convolutional neural networks have
been shown to achieve superior performance in detecting pneumonia and other
pathologies from chest X-rays, compared to practicing radiologists [219].
These results highlight the potential of ML models when they are strongly
integrated into clinical workflows.

When deployed successfully, data-driven models can free time for
clinicians[109], improve clinical outcomes [217], reduce costs [28], and
provide improved quality care for patients. However, most studies remain
preliminary, limited to small datasets, and/or implemented in select health
sub-systems. Integrating with clinical workflows remains crucial [278, 266]
but, despite recent computational advances and an explosion of health data,
deploying ML in healthcare responsibly and reliably faces several operational
and engineering challenges, including:

  * •

Standardizing data formats,

  * •

Strengthening methodologies for evaluation, monitoring and updating,

  * •

Building trust with clinicians and hospital staff,

  * •

Adopting interoperability standards, and

  * •

Ensuring that deployed models align with ethical considerations, do not
exacerbate biases, and adhere to privacy and governance policies

In this review, we articulate the challenges involved in implementing
successful Machine Learning Health Operations (MLHOps) pipelines, specific to
clinical use cases. We begin by outlining the foundations of model deployment
in general, and provide a comprehensive study of the emerging discipline [251,
167]. We then provide a detailed review of the different components of
development pipelines specific to healthcare. We discuss data, pipeline
engineering, deployment, monitoring and updating models, and ethical
considerations pertaining to healthcare use cases. While MLHOps often requires
aspects specific to healthcare, best practices and concepts from other
application domains are also relevant. This summarizes the primary outcome of
our review, which is to provide a set of recommendations for implementing
MLHOps pipelines in practice – i.e., a “how-to” guide for practitioners.

##  2 Foundations of MLOps

###  2.1 What is MLOps?

Machine learning operations (MLOps) is a combination of tools, techniques,
standards, and engineering best practices to standardize ML system development
and operations [251]. It is used to streamline and automate the deployment,
monitoring, and maintenance of machine learning models, in order to ensure
they are robust, reliable, and easily updated or upgraded.

###  2.2 MLOps Pipeline

Pipelines are processes of multiple modules that streamline the ML workflow.
Once the project is defined, the MLOps pipeline begins with identifying the
inputs and outputs relevant to the problem, cleaning, and transforming the
data towards useful and efficient representations for machine learning,
training, and evaluating model performance, and deploying selected models in
production while continuing to monitor their performance. Figure 1 illustrates
a general MLOps pipeline. Common types of pipelines include:

  * •

Automated pipelines: An end-to-end pipeline that is automated towards a single
task, e.g., a model training pipeline.

  * •

Orchestrated pipelines: A pipeline that consists of multiple modules, designed
for several automated tasks, and managed and coordinated in a dynamic
workflow, e.g., the pipeline managing MLOps.

Recently, MLOps has become more well-defined and widely implemented due to the
reusability and standardization benefits across various applications [229]. As
a result, the structure and definitions of different components are becoming
quite well-established.

###  2.3 MLOps Components

MLOps pipelines consist of different components and key concepts [134, 108],
stated below (and shown in Figure 1):

Figure 1: MLOps pipeline

  * •

Stores: Stores encapsulate the tools designed to centralize building,
managing, and sharing either features or models across different teams and
applications in an organization.

    * –

Raw data source: A raw data store is a centralized repository that stores data
in its raw, unprocessed form. It is a staging area where data is initially
collected and stored before processing or transformation.

    * –

Feature store: A centralized online repository for storing, managing, and
sharing features used in ML models. These features are acquired by processing
the raw data and are made available for real-time serving through the feature
store.

    * –

ML metadata store: A ML metadata store helps record and retrieve metadata
associated with an ML pipeline including information about various pipeline
components, their executions (e.g. training runs), and resulting artifacts
(e.g. trained models).

  * •

Serving: Serving is the task of hosting ML artifacts (usually models) either
on the cloud or on-premise so that their functions are accessible to multiple
applications through remote function calls (i.e., application programming
interfaces (APIs)).

    * –

In batch serving, the artifact is used by scheduled jobs.

    * –

In online serving, the artifact processes requests in real-time. Communication
and access point channels, traffic management, pre- and post-processing
requests, and performance monitoring should all be considered while serving
artifacts.

  * •

Data query: The component queries the data, processes it and stores it in a
format that models can easily utilize.

  * •

Experimentation: The experimentation component consists of model training,
model evaluation, and model validation.

  * •

Model registry: The model registry is a centralized repository that stores
trained machine learning models, their metadata, and their versions.

  * •

Drift-detection: The drift-detection component is responsible for monitoring
the AI system for potentially harmful drift and issuing an alert when drift is
detected.

  * •

Workflow orchestration: The workflow orchestration component responsible for
the process of automating and managing the end-to-end flow of the ML pipeline.

  * •

Source repository: The source repository is a centralized code repository that
stores the source code (and its history) for ML models and related components.

  * •

Containerization: Containerization involves packaging models with the
components required to run them; this includes libraries and frameworks so
they can run in isolated user spaces with minimal configuration of the
underlying operating system [86]. Sometimes, source code is also included in
these containers.

###  2.4 Levels of MLOps maturity

MLOps practices can be divided into different levels based on the maturity of
the ML system automation process [118, 251], as described below.

  * •

Level 0 – Manual ML pipeline: Every step in the ML pipeline, including data
processing, model building, evaluation, and deployment, are manual processes.
In Level 0, the experimental and operational pipelines are distinct and the
data scientists provide a trained model as an artifact to the engineering team
to deploy on their infrastructure. Here, only the trained model is served for
deployment and there are infrequent model updates. Level 0 processes typically
lack rigorous and continuous performance monitoring capabilities.

  * •

Level 1 – Continuous Model Training and Delivery: Here, the entire ML pipeline
is automated to perform continuous training of the model as well as continuous
delivery of model prediction services. Software orchestrates the execution and
transition between the steps in the pipeline, leading to rapid iteration over
experiments and an automatic process for deploying a selected model into
production. Contrary to Level 0, the entire training pipeline is automated,
and the deployed model can incorporate newer data based on pipeline triggers.
Given the automated nature of Level 1, it is necessary to continuously
monitor, evaluate, and validate models and data to ensure expected performance
during production.

  * •

Level 2 – Continuous Integration and Continuous Delivery: This involves the
highest maturity in automation through enforcing combined practice of
continuous integration and delivery which enables for a rapid and reliable
update of the pipelines in production. Through automated test and deployment
of new pipeline implementations, any rapid changes in data and business
environment can be addressed. In this level, the pipeline and its components
are automatically built, tested, and packaged when new code is committed or
pushed to the source code repository. Moreover, the system continuously
delivers new pipeline implementations to the target environment that in turn
delivers prediction services of the newly trained model.

Ultimately implementation of MLOps leads to many benefits, including better
system quality, increased scalability, simplified management processes,
improved governance and compliance, increased cost savings and improved
collaboration.

##  3 MLHOps Setup

Operationalizing ML models in healthcare is unique among other application
domains. Decisions made in clinical environments have a direct impact on
patient outcomes and, hence, the consequences of integrating ML models into
health systems need to be carefully controlled. For example, early warning
systems might enable clinicians to prescribe treatment plans with increased
lead time [109]; however, these systems might also suffer from a high false
alarm rate, which could result in alarm fatigue and possibly worse outcomes.
The requirements placed on such ML systems are therefore very high and, if
they are not adequately satisfied, the result is diminished adoption and trust
from clinical staff. Rigorous long-term evaluation is needed to validate the
efficacy and to identify and assess risks, and this evaluation needs to be
reported comprehensively and transparently [265].

While most MLOps best practices extend to healthcare settings, the data,
competencies, tools, and model evaluation differ significantly [179, 172, 255,
17]. For example, typical performance metrics (e.g., positive predictive value
and F1-scores) may differ between clinicians and engineers. Therefore, unlike
in other industries, it becomes necessary to evaluate physician experience
when predictions and model performance are presented to clinical staff [272].
In order to build trust in the clinical setting, the interpretability of ML
models is also exceptionally important. As more ML models are integrated into
hospitals, new legal frameworks and standards for evaluation need to be
adopted, and MLHOps tools need to comply with existing standards.

In the following sections, we explore the different components of MLHOps
pipelines.

###  3.1 Data

Successfully digitizing health data has resulted in a prodigious increase in
the volume and complexity of patient data collected [218]. These datasets are
now stored, maintained, and processed by hospital IT infrastructure systems
which in turn use specialized software systems.

####  3.1.1 Data sources

There could be multiple sources of data, which are categorized as follows:

Electronic health records (EHRs) record, analyze, and present information to
clinicians, including:

  1. 1.

Patient demographic data: E.g., age and sex.

  2. 2.

Administrative data: E.g., treatment costs and insurance.

  3. 3.

Patient observations records: E.g., chart events such as lab tests and vitals.
These include a multitude of physiological signals captured using various
methods such as heart rate, blood pressure, skin temperature, and respiratory
rate.

  4. 4.

Interventions: These are steps that significantly alter the course of patient
care, such as mechanical ventilation, dialysis, or blood transfusions.

  5. 5.

Medications information: E.g., medications administered and their dosage.

  6. 6.

Waveform data: This digitizes physiological signals collected from bedside
patient monitors.

  7. 7.

Imaging reports and metadata: E.g., CT scans, MRI, ultrasound, and
corresponding radiology reports.

  8. 8.

Medical notes: These are made by clinical staff on patient condition. These
can also be transcribed text of recorded interactions between the patient and
clinician.

Other sources of health data include primary care data, wearable data (e.g.,
smartwatches), genomics data, video data, surveys, medical claims, billing
data, registry data, and other patient-generated data [216, 30, 45].

Healthcare
dataEHRDemographicWaveformAdministrativeBillingInsuranceObservationsNotesMedicationsInterventionsImagingX-
raysMRIsCT
scansUltrasoundPathologyOmicsGenomicsTranscriptomicsProteomicsEpigenomicsMetabolomicsSensorsWearablesAmbient
sensorsOtherPharmacologicalImmuneMicrobiome

Figure 2: Stratification of health data. Further levels of stratification can
be extended as the data becomes richer. For example, observational EHR data
could include labs, vital measurements, and other recorded observations.

Figure 2 illustrates the heterogeneous nature of health data. The
stratification shown can be extended further to contain more specialized data.
For example, genomics data can be further stratified into different types of
data based on the method of sequencing; observational EHR data can be further
stratified to include labs, vital measurements, and other recorded
observations.

With such large volumes and variability in data, standardization is key to
achieve scalability and interoperability. Figure 3 illustrates the different
levels of standardization that need to be achieved with respect to health
data.

4.4 and 4.0 Variable names and types Concepts and relationships Exchange
formats and protocols Figure 3: The hierarchy of standardization that common
data models and open standards for interoperability address. The lowest level
is about achieving standardization of variable names such as lab test names,
medications and diagnosis codes, as well as the data types used to store these
variables (i.e. integer vs. character). The next level is about having
abstract concepts such that data can be mapped and grouped under these concept
definitions. The top level of standardization is about data exchange formats,
e.g. JSON, XML, protocols, along with protocols for information exchange like
supported RESTful API architectures. This level addresses questions on
interoperability and how data can be exchanged across sites and EHR systems.

####  3.1.2 Common Data Model (CDM)

Despite the widespread adoption of EHR systems, clinical events are not
captured in a standard format across observational databases [195]. For
effective research and implementation, data must be drawn from many sources
and compared and contrasted to be fully understood.

Databases must also support scaling to large numbers of records which can be
processed concurrently. Hence, efficient storage systems along with
computational techniques are needed to facilitate analyses. One of the first
steps towards scalability is to transform the data to a common data standard.
Once available in a common format, the process of extracting, transforming,
and loading (ETL) becomes simplified. In addition to scale, patient data
require a high level of protection with strict data user agreements and access
control. A common data model addresses these challenges by allowing for
downstream functional access points to be designed independent of the data
model. Data that is available in a common format promotes collaboration and
mitigates duplicated effort. Specific implementations and formats of data
should be hidden from users, and only high-level abstractions need to be
visible.

The Systematized Nomenclature of Medicine (SNOMED) was among the first efforts
to standardize clinical terminology, and a corresponding dictionary with a
broad range of clinical terminology is available as part of SNOMED-CT [67].
Several data models use SNOMED-CT as part of their core vocabulary. Converting
datasets to a common data model like the Observational Medical Outcomes
Partnership (OMOP) model involves mapping from a source database to the target
content delivery manager. This process is usually time-consuming and involves
a lot of manual effort undertaken by data scientists. Tools to simplify the
mapping and conversion process can save time and effort and promote adoption.
For OMOP, the ATLAS tool [195] developed by Observational Health Data Sciences
and Informatics (OHDSI) provides such a feature through their web based
interactive analysis platform.

####  3.1.3 Interoperability and open standards

As the volume of data grows in healthcare institutions and applications ingest
data for different use cases, real-time performance and data management is
crucial. To enable real-time operation and easy exchange of health data across
systems, an interoperability standard for data exchange along with protocols
for accessing data through easy-to-use programming interfaces is necessary.
Some of the popular healthcare data standards include Health Level 7 (HL7),
Fast Healthcare Interoperability Resources (FHIR), Health Level 7 v2 (HL7v2),
and Digital Imaging and Communications in Medicine (DICOM).

The FHIR standard [31] is a leading open standard for exchanging health data.
FHIR is developed by Health Level 7 (HL7), a not-for-profit standards
development organization that was established to develop standards for
hospital information systems. FHIR defines the key entities involved in
healthcare information exchange as resources, where each resource is a
distinct identifiable entity. FHIR also defines APIs which conform to the
representational state transfer (REST) architectural style for exchanging
resources, allowing for stateless Hypertext Transfer Protocol (HTTP) methods,
and exposing directory-structure like URIs to resources. RESTful architectures
are light-weight interfaces that allow for faster transmission, which is more
suitable for mobile devices. RESTful interfaces also facilitate faster
development cycles because of their simple structure.

DICOM is the standard for the communication and management of medical imaging
information and related metadata. The DICOM standard specifies the format and
protocol for exchange of digital information between medical imaging equipment
and other systems. Persistent information objects which encode images are
exchanged and an instance of such an information object may be exchanged
across many systems and many organizational contexts, and over time. DICOM has
enabled deep collaboration and standardization across different disciplines
such as radiology, cardiology, pathology, ophthalmology, and related
disciplines.

####  3.1.4 Quality assurance and validation

Data collected in retrospective databases for analysis and ML use cases need
to be checked for quality and consistency. Data validation is an important
step towards ensuring that ML systems developed using the data are highly
performant, and do not incorporate biases from the data. Errors in data
propagate through the MLOps pipeline and hence specialized data quality
assurance tools and checks at various stages of the pipeline are necessary
[223]. A standardized data validation framework that includes i) data element
pre-processing, ii) checks for completeness, conformance, and plausibility,
and iii) a review process by clinicians and other stakeholders should capture
generalizable insight across various clinical investigations [238].

###  3.2 Pipeline Engineering

Data stored in raw formats need to be processed to create feature
representations for ML models. Each transformation is a computation, and a
chain of these processing elements, arranged so that the output of each
element is the input of the next, constitutes a pipeline [134] and using
software tools and workflow practices that enable such pipelines is pipeline
engineering.

There are advantages to using such a pipeline approach, including:

  * •

Modularization: By breaking the chain of transformations into small steps,
modularization is naturally achieved.

  * •

Testing: Each transformation step can be tested independently, which
facilitates quality assurance and testing.

  * •

Debugging: Version controlling the outputs at each step makes it easier to
ensure reproducibility, especially when many steps are involved.

  * •

Parallelism: If any step in the pipeline is easily parallelizable across
multiple compute nodes, the overall processing time can be reduced.

  * •

Automation: By breaking a complex task into a series of smaller tasks, the
completion of each task can be used to trigger the start of the next task, and
this can be automated using continuous integration tools such as Jenkins,
Github actions and Gitlab CI.

In health data processing, the following steps are crucial:

  1. 1.

Cleaning: Formatting values, adjusting data types, checking and fixing issues
with raw data.

  2. 2.

Encoding: Computing word embeddings for clinical text, encoding the text and
raw values into embeddings [127, 15]. Encoding is a general transformation
step that can be used to create vector representations of raw data. For
example, transforming images to numeric representations can also be considered
to be encoding.

  3. 3.

Aggregation: Grouping values into buckets, e.g., for aggregating measurements
into fixed time-intervals, or grouping values by patient ID.

  4. 4.

Normalization: Normalizing values into standard ranges or using statistics of
the data.

  5. 5.

Imputation: Handling missing values in the data. For various clinical data,
‘missingness’ can actually provide valuable contextual information about the
patient’s health and needs to be handled carefully [47].

Multiple data sources such as EHR data, clinical notes and text, imaging data,
and genomics data can be processed independently to create features and they
can be combined to be used as inputs to ML models. Hence, composing pipelines
of these tasks facilitates component reusability [115]. Furthermore, since the
ML development life-cycle constitutes a chain of tasks, the pipelining
approach becomes even more desirable. Some of the high level tasks in the
MLHOps pipeline include feature creation, feature selection, model training,
evaluation, and monitoring. Evaluating models across different slices of data,
hyper-parameters, and other confounding variables is necessary for building
trust.

Table 4 lists popular open-source tools and packages specific to health data
and ML processing. These tools are at different stages of development and
maturity. Some examples of popular tools include MIMIC-Extract [273],
Clairvoyance [115] and CheXstray [245].

###  3.3 Modelling

At this stage, the data has been collected, cleaned, and curated, ready to be
fed to the ML model to accomplish the desired task. The modelling phase
involves choosing the available models that fit the problem, training &
testing the models, and choosing the model with the best performance &
reliability guarantees. Given the the existence of numerous surveys
summarizing machine learning and deep learning algorithms for general
healthcare scenarios [74, 1], as well as specific use cases such as brain
tumor detection [18], COVID-19 prevention[26], and clinical text
representation [127], we omit this discussion and let the reader explore the
surveys relevant to their prediction problem.

###  3.4 Infrastructure and System

Hospitals typically use models developed by their EHR vendor which are
deployed through the native EHR vendor configuration. Often, inference is run
locally or in a cloud instance, and the model outputs are communicated within
the EHR [124]. Predominantly, these models are pre-trained and sometimes fine-
tuned on the specific site’s data.

A feature store is a ML-specific data system used to centralize storage,
processing, and access to frequently used features, making them available for
reuse in the development of future machine learning models. Feature stores
operationalize and streamline the input, tracking, and governance of the data
as part of feature engineering for machine learning [134].

To ensure reliability, the development, staging, and production environments
are separated and have different requirements. The staging and production
environments typically consist of independent virtual machines with adequate
compute and storage, along with reliable and secure connections to the
databases.

The infrastructure and software systems also have to follow and comply with
cybersecurity, medical software design and software testing standards [65].

####  3.4.1 Roles and Responsibilities

Efficient and successful MLHOps requires a collaborative, interdisciplinary
team across a range of expertise and competencies commonly found in data
science, ML, software, operations, production engineering, medicine, and
privacy capabilities [134]. Similar to general MLOps practices, data and ML
scientists, data, DevOps, and ML engineers, solution and data architects, ML
and software fullstack developers, and project managers are needed. In
addition, the following role are required, which are distinct to healthcare
(for more general MLOps roles see Table 5):

  * •

Health AI Project Managers: Responsibilities include panning projects,
establishing guidelines, milestone tracking, managing risk, supporting the
teams and governing partnerships with collaborators from other health
organizations.

  * •

Health AI Implementation Coordinator: Liaison that engages with key
stakeholders to facilitate the implmentation of clinical AI systems.

  * •

Healthcare Operations Manager: Oversees and coordinates quality management,
resource management, process improvement, and patient safety in clinical
settings like hospitals.

  * •

Clinical Researchers & Scientists: Domain experts that provide critical
domain-specific knowledge relevant to model development and implementation.

  * •

Patient-Facing Practitioners: Responsibilities include providing system
requirements, pipeline usage feedback, and perspective about the patient
experience (e.g. clinicians, nurses).

  * •

Ethicists: Provides support regarding ethical implications of clinical AI
systems.

  * •

Privacy Analysts: Provides assessments regarding privacy concerns pertaining
to the usage of patient data.

  * •

Legal Analysts: Works closely with privacy analysts and ethicists to evaluate
the legal vulnerabilities of clinical AI systems.

###  3.5 Reporting Guidelines

Many clinical AI systems do not meet reporting standards because of a failure
to assess for poor quality or unavailable input data, insufficient analysis of
performance errors, or a lack of information regarding code or algorithm
availability [208]. Systematic reviews of clinical AI systems suggest there is
a substantial reporting burden, and additions regarding reliability and
fairness can improve reporting [164]. As a result, guidelines informed by
challenges in existing AI deployments in health settings have become
imperative [57]. Reporting guidelines including CONSORT-AI [158], DECIDE-AI
[265], and SPIRIT-AI [225] were developed by a multidisciplinary group of
international experts using the Delphi process to ensure complete and
transparent reporting of randomized clinical trials (RCT) that evaluate
interventions with an AI model. Broadly these guidelines suggest inclusion of
the following criteria [65]:

  * •

Intended use: Inclusion of the medical problem and context, current standard
practice, intended patient population(s), how the AI system will be integrated
into the care pathway, and the intended patient outcomes.

  * •

Patient and user recruitment: Well-defined inclusion and exclusion criteria.

  * •

Data and outcomes: The use of a representative patient population, data coding
and processing, missing- and low-quality data handling, and sample size
considerations.

  * •

Model: Inclusion of inputs, outputs, training, model selection, parameter
tuning, and performance.

  * •

Implementation: Inclusion of user experience with the AI system, user
adherence to intended implementation, and changes to clinical workflow.

  * •

Modifications: A description protocol for changes made, timing and rationale
for modifications, and outcome changes after each modification.

  * •

Safety and errors: Identification of system errors and malfunctions,
anticipated risks and mitigation strategies, undesirable outcomes, and worst-
case scenarios.

  * •

Ethics and fairness: Inclusion of subgroup analyses, and fairness metrics.

  * •

Human-computer agreement: Report of user agreement with the AI system, reasons
for disagreement, and cases of users changing their mind based on the AI
system.

  * •

Transparency: Inclusion of data and code availability.

  * •

Reliability: Inclusion of uncertainty measures, and performance against
realistic baselines.

  * •

Generalizability: Inclusion of measures taken to reduce overfitting, and
external performance evaluations.

####  3.5.1 Tools and Frameworks

Understanding the MLOps pipeline and required expertise is just the first step
to addressing the problem. Once this has been accomplished, it is necessary to
create and/or adopt appropriate tooling for transforming these principles into
practice. There are seven broad categories of MLOps tools as shown in Table 1
whereby different tools to automate different phase of the workflows involved
in MLOps processes exist. A compiled list of tools within each category is
shown in Table 1

Table 1: MLOps tools 4 Category | Description |  Tooling Examples
---|---|---
4 Model metadata storage and management |  Section 3.1 |  • MLFlow 111https://mlflow.org/docs/latest/index.html • Comet 222https://mlops.community/learn/metadata-storage-and-management/comet-ml/ • Neptune333https://docs.aws.amazon.com/neptune/latest/userguide/feature-overview-storage.html
Data and pipeline versioning |  Section 3.2 |  • DVC444https://dvc.org/doc/start/data-management/data-pipelines • Pachyderm555https://www.jetbrains.com/pycharm/
Model deployment and serving |  Section 3.3 |  • DEPLOYR666https://www.deployr.ai/home-en/[59] • Flyte777https://flyte.org/ • ZenML888https://zenml.io/home
Production model monitoring |  Section 4 |  • MetaFlow999https://metaflow.org/ • Kedro101010https://docs.kedro.org/en/stable/index.html • Seldon Core111111https://www.seldon.io/solutions/open-source-projects/core
Run orchestration and workflow pipelines |  Orchestrating the execution of preprocessing, training, and evaluation pipelines. Section 3.4 & 3.5 |  • Kuberflow121212https://www.kubeflow.org/ • Polyaxon131313https://polyaxon.com/ • MLRun141414https://www.mlrun.org/
Collaboration Tool |  Setting up an MLOps pipeline requires collabora- tion between different people. Section 3.4.1 |  • ChatOps151515https://docs.gitlab.com/ee/ci/chatops/ • Slack161616https://slack.com/ • Trello171717https://trello.com/ • GitLab181818https://about.gitlab.com/ • Rocket Chat 191919https://rocket.chat/

##  4 MLHOps Monitoring and Updating

Once an MLHOps pipeline and required resources are setup and deployed, robust
monitoring protocols are crucial to the safety and longevity of clinical AI
systems. For example, inevitable updates to a model can introduce various
operational issues (and vice versa), including bias (e.g., a new hospital
policy that shifts the nature of new data) and new classes (e.g., new subtypes
in a disease classifier) [287]. Incorporating expert labels can improve model
performance; however, the time, cost, and expertise required to acquire
accurate labels for very large imaging datasets like those used in radiology-
or histology-based classifiers makes this difficult [138].

As a result, there exist monitoring frameworks with policies to determine when
to query experts for labels [300]. These include:

  * •

Periodic Querying, a non-adaptive policy whereby labels are periodically
queried in batches according to a predetermined schedule;

  * •

Request-and-Reverify which sets a predetermined threshold for drift and
queries a batch of labels whenever the drift threshold is exceeded [288];

  * •

MLDemon which follows a periodic query cycle and uses a linear estimate of the
accuracy based on changes in the data [90].

###  4.1 Time-scale windows

Monitoring clinical AI systems requires evaluating robustness to temporal
shifts. Since the time-scale used can change the types of shifts detected
(i.e., gradual versus sudden shifts), multiple time windows should be
considered (e.g., week, month). Moreover, it is important to use both 1)
cumulative statistics, which use a single time window and updates at the
beginning of each window and 2) sliding statistics, which retain previous data
and update with new data.

###  4.2 Appropriate metrics

It is critical to choose evaluation and monitoring metrics optimal for each
clinical context. The quality of labels is highly dependent on the data from
which they are derived and, as such, can possess inherent biases. For
instance, sepsis labels derived from incorrect billing codes will inherently
have a low positive predictive value (PPV). Moreover, clinical datasets are
often imbalanced, consisting of far fewer positive instances of a label than
negative ones. As a result, measures like accuracy that weigh positive and
negative labels equally can be detrimental to monitoring. For instance, in the
context of disease classification, it may be particularly important to monitor
sensitivity, in contrast to more time-sensitive clinical scenarios like the
intensive care unit (ICU) where false positives (FP) can have critical
outcomes [20].

###  4.3 Detecting data distribution shift

Data distribution shift occurs when the underlying distribution of the
training data used to build an ML model differs from the distribution of data
applied to the model during deployment [214]. When the difference between the
probability distributions of these data sets is sufficient to deteriorate the
model’s performance, the shift is considered malignant.

In healthcare, there are multiple sources of data distribution shifts, many of
which can occur concurrently [78, 248]. Common occurrences of malignant shifts
include differences attributed to:

  * •

Institution \- These differences can arise when comparing teaching to non-
teaching hospitals, government-owned to private hospitals, or general to
specialized hospitals (e.g., paediatric, rehabilitation, trauma). These
institutions can have differing local clinical practices, resource allocation
schemes, medical instruments, and data-collection and processing workflows
that can lead to downstream variation. This has previously been reported in
Pneumothorax classifiers when evaluated on external institutions [130].

  * •

Behaviour \- Temporal changes in behaviour at the systemic, physician and
patient levels are unavoidable sources of data drift. These changes include
new healthcare reimbursement incentives, changes in the standard-of-care in
medical practice, novel therapies, and updates to hospital operational
processes. An example of this is the COVID-19 pandemic, which required changes
in resource allocation to cope with hospital bed shortages [132, 201].

  * •

Patient demographics \- Differences in factors like age, race, gender,
religion, and socioeconomic background can arise for various reasons including
epidemiological transitions, gentrification of neighbourhoods around a health
system, and new public health and immigration policies. Distribution shifts
due to demographic differences can disproportionately deteriorate model
performance in specific patient populations. For instance, although Black
women are more likely to develop breast tumours with poor prognosis, many
breast mammography ML classifiers experience deterioration in performance on
this patient population [284]. Similarly, skin-lesion classifiers trained
primarily on images of lighter skin tones may show decreased performance when
evaluated on images of darker skin tones [9, 69].

  * •

Technology \- Data shifts can be attributed to changes in technology between
institutions or over time. This includes chest X-ray classifiers trained on
portable radiographs that are evaluated on stationary radiographs or
deteroriation of clinical AI systems across EHR systems (e.g., Philips Carevue
vs. Metavision) [188].

Although evaluated differently, data shifts are present across various
modalities of clinical data such as medical images [98] and EHR data [70,
201]. In order to effectively prevent these malignant shifts from occurring,
it is necessary to perform prospective evaluation of clinical AI systems [303]
in order to understand the circumstances under which they arise, and to design
strategies that mitigate model biases and improve models for future iterations
[290]. Broadly, these data shifts can be categorized into three groups which
can co-occur or lead to one another:

####  4.3.1 Covariate Shift

Covariate shift is a difference in the distribution of input variables between
source and target data. It can occur due to a lack of randomness, inadequate
sampling, biased sampling, or a non-stationary environment. This can be at the
level of a single input variable (i.e. feature shift) or a group of input
features (i.e. dataset shift). Table 2 contains a list of commonly used
methods used for covariate shift detection.

Feature Shift Detection: Feature shift refers to the change in distribution
between the source and target data for a single input feature. Feature shift
detection can be performed using two-sample univariate tests such as the
Kolmogorov-Smirnov (KS) test [215]. Publicly available tools like TensorFlow
Extended (TFX) apply univariate tests (i.e., $L$-infinity distance for
categorical variables, Jensen-Shannon divergence for continuous variables) to
perform feature shift detection between training and deployment data and
provide users with summary statistics (Table 3). It is also possible to detect
feature shift while conditioning on the other features in a model using
conditional distribution tests [135].

Dataset Shift Detection: Dataset shift refers to the change in the joint
distribution between the source and target data for a group of input features.
Multivariate testing is crucial because input to ML models typically consist
of more than one variable and multiple modalities. In order to test whether
the distribution of the target data has drifted from the source data two main
approaches exist: 1) two-sample testing and 2) classifiers. These approaches
often work better on low-dimensional data compared to high-dimensional data,
therefore dimensionality reduction is typically applied first [215]. For
instance, variational autoencoders (VAE) have been used to reduce chest X-ray
images to a low-dimensional space prior to two-sample testing [245]. In the
context of medical images, including chest X-rays [211] [289], diabetic
retinopathies [41], and histology slides [246], classifier methods have proven
effective. For EHR data, dimensionality reduction using clinically meaningful
patient representations has improved model performance [188]. For clinically
relevant drift detection, it is important to ensure that drift metrics
correlate well with ground truth performance differences.

Method |  Shift |  Test Type
---|---|---
L-infinity distance |  Feature (c) |  2-ST
Cramér-von Mises |  Feature (c) |  2-ST
Fisher’s Exact Test |  Feature (c) |  2-ST
Chi-Squared Test |  Feature (c) |  2-ST
Jensen-Shannon divergence |  Feature (n) |  2-ST
Kolmogorov-Smirnov [174] |  Feature (n) |  2-ST
Feature Shift Detector [135] |  Feature |  Model
Maximum Mean Discrepancy (MMD) [93] |  Dataset |  2-ST
Least Squares Density Difference [37] |  Dataset |  2-ST
Learned Kernel MMD [155] |  Dataset |  2-ST
Context Aware MMD [56] |  Dataset |  2-ST
MMD Aggregated [236] |  Dataset |  2-ST
Classifier [161] |  Dataset |  Model
Spot-the-diff [117] |  Dataset |  Model
Model Uncertainty [240] |  Dataset |  Model
Mahalanobis distance [222] |  Dataset |  Model
Gram matrices [202] [234] |  Dataset |  Model
Energy Based Test [157] |  Dataset |  Model
H-Divergence [299] |  Dataset |  Model
Table 2: Covariate Shift Detection Methods c: categorical; n: numeric; 2-ST:
Two-Sample Test

####  4.3.2 Concept Shift

Concept shift is a difference in the relationship (i.e., joint distribution)
of the variables and the outcome between the source and target data. In
healthcare, concept shift can arise due to changes in symptoms for a disease
or antigenic drift. This has been explored in the context of surgery
prediction [32] and medical triage for emergency and urgent care [112].

Concept Shift Detection: There are three broad categories of concept shift
detection based on their approach.

  1. 1.

Distribution techniques which use a sliding window to divide the incoming data
streams into windows based on data size or time interval and that compare the
performance of the most recent observations with a reference window [84].
ADaptive WINdowing (ADWIN), and its extension ADWIN2, are windowing techniques
which use the Hoeffding bound to examine the change between the means of two
sufficiently large subwindows [106].

  2. 2.

Sequential Analysis strategies use the Sequential Probability Ratio Test
(SPRT) as the basis for their change detection algorithms. A well-known
algorithm is CUMSUM which outputs an alarm when the mean of the incoming data
significantly deviates from zero [29].

  3. 3.

Statistical Process Control (SPC) methods track changes in the online error
rate of classifiers and trigger an update process when there is a
statistically significant change in error rate [163]. Some common SPC methods
include: Drift Detection Method (DDM), Early Drift Detection Method (EDDM),
and Local Drift Detection (LLDD) [23].

####  4.3.3 Label Shift

Label shift is a difference in the distribution of class variables in the
outcome between the source and target data. Label shift may appear when some
concepts are under-sampled or over-sampled in the target domain compared to
the source domain. Label shift arises when class proportions differ between
the source and target, but the feature distributions of each class do not. For
instance, in the context of disease diagnosis, a classifier trained to predict
disease occurrence is subject to drift due to changes in the baseline
prevalence of the disease across various populations.

Label Shift Detection: Label shift can be detected using moment matching-based
estimator methods that leverage model predictions like Black Box Shift
Estimation (BBSE) [151] and Regularized Learning under Label Shift (RLLS)
[22]. Assuming access to a classifier that outputs the true source
distribution conditional probabilities $p_{s}(y|x)$ Expectation Maximization
(EM) algorithms like Maximum Likelihood Label Shift (MLLS) can also be used to
detect label shift [87]. Furthermore, methods using bias-corrected calibration
show promise in correcting label shift [14].

###  4.4 Model Updating and Retraining

As the implementation of ML-enabled tools is realized in the clinic, there is
a growing need for continuous monitoring and updating in order to improve
models over time and adapt to malignant distribution shifts. Retraining of ML
models has demonstrated improved model performance in clinical contexts like
pneumothorax diagnosis [130]. However, proposed modifications can also degrade
performance and introduce bias [149]; as a result it may be preferable to
avoid making a prediction and defer the decision to a downstream expert [186].
When defining a model updating or retraining strategy for clinical AI models
there are several factors to consider [279], we outline they key criteria in
this section.

Name of tool |  Capabilities
---|---
Evidently 202020https://github.com/evidentlyai/evidently |  Interactive reports to analyze ML models during validation or production monitoring.
NannyML212121https://github.com/NannyML/nannyml |  Performance estimation and monitoring, data drift detection and intelligent alerting for deployment.
River [185] |  Online metrics, drift detection and outlier detection for streaming data.
SeldonCore [262] |  Serving, monitoring, explaining, and management of models using advanced metrics, explainers, and outlier detection.
TFX222222https://github.com/tensorflow/tfx |  Explore and validate data used for machine learning models.
TorchDrift232323https://github.com/TorchDrift/TorchDrift |  Covariate and concept drift detection.
deepchecks [54] |  Testing for continuous validation of ML models and data.
EHR OOD Detection [258] |  Uncertainty estimation, OOD detection and (deep) generative modelling for EHRs.
Avalanche [160] |  Prototyping, training and reproducible evaluation of continual learning algorithms.
Giskard242424https://github.com/Giskard-AI/giskard |  Evaluation, monitoring and drift testing.
Table 3: List of open-source tools available on Github that can be used for ML
Monitoring and Updating

####  4.4.1 Quality and Selection of Model Update Data

When updating a model it is important to consider the relevance and size of
the data to be used. This is typically done by defining a window of data to
update the model: i) Fixed window uses a window that remains constant across
time. ii) Dynamic window uses a window that changes in size due to an adaptive
data shift, iii) Representative subsample uses a subsample from a window that
is representative of the entire window distribution.

####  4.4.2 Updating Strategies

There are several ways to update a model including: i) Model recalibration is
the simplest type of model update, where continuous scores (e.g. predicted
risks) produced by the original model are mapped to new values [52]. Some
common methods to achieve this include Platt scaling [209], temperature
scaling, and isotonic regression [191]. ii) Model updating includes changes to
an existing model, for instance, fine-tuning with regularization [139] or
model editing where pre-collected errors are used to train hypernetworks that
can be used to edit a model’s behaviour by predicting new weights or building
a new classifier [182]. iii) Model retraining involves retraining a model from
scratch or fitting an entirely different model.

####  4.4.3 Frequency of Model Updates

In practice, retraining procedures for clinical AI models have generally been
locked after FDA approval [140] or confined to ad-hoc one-time updates [261]
[104]. The timing of when it is necessary to update or retrain a model varies
across use case. As a result, it is imperative to evaluate the appropriate
frequency to update a model. Strategies employed include: i) Periodic training
on a regular schedule (e.g. weekly, monthly). ii) Performance-based trigger in
response to a statistically significant change in performance. iii) Data-based
trigger in response to a statistically significant data distribution shift.
iv) Retraining on demand is not based on a trigger or regular schedule, and
instead initiated based on user prompts.

####  4.4.4 Continual Learning

Continual learning is a strategy used to update models when there is a
continuous stream of input data that may be subject to changes over time.
Prior to deployment, it is crucial to simulate the online learning procedure
on retrospective data to assess robustness to data shifts [51] [198]. When
models are retrained on only the most recent data, this can result in
“catastrophic forgetting” [267] [140], in which the integration of new data
into the model can overwrite knowledge learned in the past and interfere with
what the model has already learned [138]. Contrastingly, procedures that
retrain models on all previously collected data can fail to adapt to important
temporal shifts and are computationally expensive. More recently, strategies
leveraging multi-armed bandits have been utilized to select important samples
or batches of data for retraining [92] [301]. This is an important
consideration in healthcare contexts like radiology, where the labelling of
new data can be a time-consuming bottleneck [100] [206].

To ensure continual learning satisfies performance guarantees, hypothesis
testing can be used for approving proposed modifications [62]. An effective
approach for parametric models include continual updating procedures like
online recalibration/revision [76]. Strategies for continual learning can
broadly be categorized into: 1) Parameter isolation where changes to
parameters that are important for the previous tasks are forbidden e.g. Local
Winner Takes All (LWTA), Incremental Moment Matching (IMM) [260]; 2)
Regularization methods which builds on the observation forgetting can be
reduced by protecting parameters that are important for the previous tasks
e.g. elastic weight consolidation (EWC), Learning Without Forgetting (LWF);
and 3) Replay-based approaches that retain some samples from the previous
tasks and use them for training or as constraints to reduce forgetting e.g.
episodic representation replay (ERR) [66]. Evaluation of several continual
learning methods on ICU data across a large sequence of tasks indicate replay-
based methods achieves more stable long-term performance, compared to
regularization and rehearsal based methods [19]. In the context of chest X-ray
classification, Joint Training (JT) has demonstrated superior model
performance, with LWF as a promising alternative in the event that training
data is unavailable at deployment [141]. For sepsis prediction using EHR data,
a joint framework leveraging EWC and ERR has been proposed [16]. More
recently, continual model editing strategies have shown promise in overcoming
the limitations of continual fine-tuning methods by updating model behavior
with minimal influence on unrelated inputs and maintaining upstream test
performance [105].

####  4.4.5 Domain Generalization and Adaptation

Broadly, domain generalization and adaptation methods are used to improve
clinical AI model stability and robustness to data shifts by reducing
distribution differences between training and test data [293] [95]. However,
it is critical to evaluate several methods over a range of metrics, as the
effectiveness of each method varies based on several factors including the
type of shift and data modality [285].

  * •

Data-based methods perform manipulations based on the patient data to minimize
distribution shifts. This can be done by re-weighting observations during
training based on the target domain [133], upsampling informative training
examples [153] or leveraging a combination of labeled and pseudo-labeled
[147].

  * •

Representation-based methods focus on achieving a feature representation such
that the source classifier performs well on the target domain. In clinical
data this has been explored using strategies including invariant risk
minimization (IRM), distribution matching (e.g. CORAL) and domain-adversarial
adaptation networks (DANN). DANN methods have demonstrated a reduction on the
impact of data shift on cross-institutional transfer performance for
diagnostic prediction [296]. However, it has been shown that for clinical AI
models subject to real life data shifts, in contrast to synthetic
perturbations, empirical risk minimization outperforms domain generalization
and unsupervised domain adaptation methods [97] [294].

  * •

Inference-based methods introduce constraints on the optimization procedure to
reduce domain shift [133]. This can be done by estimating a model’s
performance on the “worst-case” distribution [249] or constraining the
learning objective to enforces closeness between protected groups [237]. Batch
normalization statistics can also be leveraged to build models that are more
robust to covariate shifts [235].

####  4.4.6 Data Deletion and Unlearning

In healthcare there are two primary reasons for wanting to remove data from
models. Firstly, with the growing concerns around privacy and ML in
healthcare, it may become necessary to remove patient data for privacy
reasons. Secondly, it may also be beneficial to a model’s performance to
delete noisy or corrupted training data [35]. The naive approach to data
deletion is to exclude unwanted samples and retrain the model from scratch on
the remaining data, however this approach can quickly become time consuming
and resource-intensive [114]. As a result, more sophisticated approaches have
been proposed for unlearning in linear and logistic models [114], random
forest models [36], and other non-linear models [96].

####  4.4.7 Feedback Loops

Feedback loops that incorporate patient outcomes and clinician decisions are
critical to improving outcomes in future model iterations. However, retraining
feedback loops can also lead to error amplification, and subsequent downstream
increases in false positives [6]. As a result, it is important to consider
model complexity and choose an appropriate classification threshold to ensure
minimization of error amplification [7].

##  5 Responsible MLHOps

AI has surged in healthcare, out of necessity or/and [290, 199], but many
issues still exist. For instance, many sources of bias exist in clinical data,
large models are opaque, and there are malicious actors who damage or pollute
the AI/ML systems. In response, responsible AI and trustworthiness have
together become a growing area of study [176, 264]. Responsible AI, or
trustworthy MLOps, is defined as an ML pipeline that is fair and unbiased,
explainable and interpretable, secure, private, reliable, robust, and
resilient to attacks. In healthcare, trust is critical to ensuring a
meaningful relationship between the healthcare provider and patient [63]. In
this section, we discuss components of responsible and trustworthy AI [142],
which can be applied to the MLHOps pipeline. In Section 5.1, we review the
main concepts of responsible AI and in Section 5.2 we explore how these
concepts can be embedded in the MLHOps pipeline to enable safe deployment of
clinical AI systems.

###  5.1 Responsible AI in healthcare

Ethics in healthcare:

Ethics in healthcare primarily consists of the following criteria [263]:

  1. 1.

Nonmaleficence: Do not harm the patient.

  2. 2.

Beneficence: Act to the benefit of the patient.

  3. 3.

Autonomy: The patient (when able) should have the freedom to make decisions
about his/her body. More specifically, the following aspects should be taken
care of:

     * •

Informed Consent: The patient (when able) should give informed consent for any
medical or surgical procedure, or for research.

     * •

Truth-telling: The patient (when able) should receive full disclosure to
his/her diagnosis and prognosis.

     * •

Confidentiality: The patient’s medical information should not be disclosed to
any third party without the patient’s consent.

  4. 4.

Justice: Ensure fairness to the patient.

To supplement these criteria, guiding principles drawn from surgical settings
[152, 228] include:

  1. 5.

Rescue: A patient surrenders to the healthcare provider’s expertise to be
rescued.

  2. 6.

Proximity: The emotional proximity to the patient should be limited to
maintain self-preservation and stability in case of any failure.

  3. 7.

Ordeal: A patient may have to face an ordeal (i.e., go through painful
procedures) in order to be rescued.

  4. 8.

Aftermath: The physical and psychological aftermath that may occur to the
patient due to any treatment must be acknowledged.

  5. 9.

Presence: An empathetic presence must be provided to the patient.

While some of these criteria relate to the humanity of the healthcare
provider, others relate to the following topics in ML models:

  * $\blacksquare$

Fairness involves the justice component in the healthcare domain [50].

  * $\blacksquare$

Interpretability & explainability relate to explanations and better
understanding of the ML models’ decisions, which can help in achieving
nonmaleficence, beneficence, informed consent, and truth-telling principles in
healthcare. Interpretability can help identify the reasons for a given model
outcome, which can help inform healthcare providers and patients on how to
respond accordingly [179].

  * $\blacksquare$

Privacy and security relate to confidentiality. [126].

  * $\blacksquare$

Reliability, robustness, and resilience addresses rescue [227].

We discuss these concepts further in Sections 5.1.1, 5.1.2, 5.1.3 and 5.1.4.

####  5.1.1 Bias & Fairness

The fairness of AI-based decision support systems have been studied generally
in a variety of applications including occupation classifiers [64], criminal
risk assessments algorithms [55], recommendation systems [71], facial
recognition algorithms [38], search engines [85], and risk score assessment
tools in hospitals [193]. In recent years, the topic of fairness in AI models
in healthcare has received a lot of attention [193, 241, 137, 49, 278, 242].
Unfairness in healthcare manifests as differences in model performance against
or in favour of a sub-population, for a given predictive task. For instance,
disproportionate performance differences for disease diagnosis in Black versus
White patients [241].

#####  5.1.1.1 Causes

A lack of fairness in clinical AI systems may be a result of various
contributing causes:

  * •

Objective:

    * –

Unfair objective functions: The initial objective used in developing a ML
approach may not consider fairness. This does not mean that the developer
explicitly (or implicitly) used an unfair objective function to train the
model, but the oversimplification of that objective can lead to downstream
issues. For example, a model designed to maximize accuracy across all
populations may not inherently provide fairness across different sub-
populations even if it reaches state-of-the-art performance on average, across
the whole population [241, 242].

    * –

Incorrect presumptions: In some instances, the objective function includes
incorrect interpretations of features, which can lead to bias. For instance, a
commercial algorithm used in the USA, used health costs as a proxy for health
needs[193]; however, due to financial limitations, Black patients with the
same need for care as White patients often spend less on healthcare and
therefore have a lower health cost. As a result, the model falsely inferred
that Black patients require less care compared to White patients because they
spend less [193]. Additionally, patients may be charged differently for the
same service based on their insurance, suggesting cost may not be
representative of healthcare needs.

  * •

Data:

    * –

Inclusion and exclusion: It is important to clearly outline the conditions and
procedures utilized for patient data collection, in order to understand
patient inclusion criteria and any potential selection biases that could
occur. For instance, the Chest X-ray dataset [275] was gathered in a research
hospital that does not routinely conduct diagnostic and treatment
procedures252525from https://clinicalcenter.nih.gov/about/welcome/faq.html..
This dataset therefore includes mostly critical cases, and few patients at the
early stages of diagnosis. Moreover, as a specialized hospital, patient
admission is selective and chosen solely by institute physicians based on if
they have an illness being studied by the given institute 262626from
https://clinicalcenter.nih.gov/about/welcome/faq.html.. Such a dataset will
not contain the diversity of disease cases that might be seen in hospitals
specialized across different diseases, or account for patients visiting for
routine treatment services at general hospitals.

    * –

Insufficient sample size: Insufficient sample sizes of under-represented
groups can also result in unfairness [89]. For instance, patients of low
socioeconomic status may use healthcare services less, which reduces their
sample size in the overall dataset, resulting in an unfair model [294, 38,
49]. In another instance, an algorithm that can classify skin cancer [73] with
high accuracy will not be able to generalize to different skin colours if
similar samples have not been represented sufficiently in the training data
[38].

    * –

Missing essential representative features: Sometimes, essential representative
features are missed or not collected during the dataset curation process,
which prohibits downstream fairness analyses. For instance, if the patient’s
race has not been recorded, it is not possible to analyze whether a model
trained on that data is fair with respect to that race [242]. Failure to
include sensitive features can generate discrimination and reduce transparency
[48].

  * •

Labels:

    * –

Social bias reflection on labels: Biases in healthcare systems widely reflect
existing biases in society [168, 250, 269]. For instance, race and sex biases
exist in COPD underdiagnosis [168], in medical risk score analysis (whereby
there exists a higher threshold for Black patients to gain access to clinical
resources) [269], and in the time of diagnosis for cardiovascular disease
(whereby female patients are diagnosed much later compared to the male
patients with similar conditions) [250]. These biases are reflected in the
labels used to train clinical AI systems and, as a result, the model will
learn to replicate this bias.

    * –

Bias of automatic labeling: Due to the high cost and labour-intensive process
of acquiring labels for healthcare data, there has been a shift away from
hand-labelled data, towards automatic labelling [39, 113, 120]. For instance,
instead of expert-labeled radiology images, natural language processing (NLP)
techniques are applied to radiology reports in order to extract labels. This
presents concerns as these techniques have shown racial biases, even after
they have been trained on clinical notes [295]. Therefore, using NLP
techniques for automatic labeling may sometimes amplify the overall bias of
the labels [242].

  * •

Resources:

    * –

Limited computational resources: Not all centers have enough labeled data or
computational resources to train ML models ‘from scratch’ and must use
pretrained models for inference or transfer learning. If the original model
has been trained on biased (or differently distributed) data, it will unfairly
influence the outcome, regardless of the quality of the data at the host
center.

#####  5.1.1.2 Evaluation

To evaluate the fairness of a model, we need to decide which fairness metric
to use and what sensitive attributes to consider in our analysis.

  * •

Fairness metric(s): There are many ways to define fairness metrics. For
instance, [55] and [103] discussed several fairness criteria and suggested
balancing the error rate between different subgroups [58, 292]. However, it is
not always possible to satisfy multiple fairness constraints concurrently
[242]. Jon Kleinberg et al., [131] showed that three fairness conditions
evaluated could not be simultaneously satisfied. As a result, a trade-off
between the different notions of fairness is required, or a single fairness
metric can be chosen based on domain knowledge and the given clinical
application.

  * •

Sensitive attributes: Sensitive attributes are protected groups that we want
to consider when evaluating the fairness of an AI model. Sex and race are two
commonly used sensitive attributes [292, 241, 242, 295]. However, a lack of
fairness in an AI system with respect to other sensitive attributes such as
age [241, 242], socioeconomic status, [241, 242, 295], and spoken language
[295] are also important to consider.

Defining AI fairness is context- and problem-dependent. For instance, if we
build an AI model to support decision making for disease diagnosis with the
goal of using it in the clinic, then it is critical to ensure equal
opportunity in the model is provided; i.e., patients from different races
should have equal opportunity to be accurately diagnosed [241]. However, if an
AI model is to be used to triage patients, then ensuring the system does not
underdiagnose unhealthy patients of a certain group may be of greater concern
compared to the specific disease itself because the patient will lose access
to timely care [242].

####  5.1.2 Interpretability & Explainability

In recent years, interpretability has received a lot of interest from the ML
community [184, 253, 172]. In machine learning, interpretability is defined as
the ability to explain the rationale for an ML model’s predictions in terms
that a human can understand [68] and explainability refers to a detailed
understanding of the model’s internal representations, a priori of any
decision. After other research in this area [170], we use ‘interpretability’
and ‘explainability’ interchangeably.

Interpretability is not a pre-requisite for all AI systems [68, 184],
including in low-risk environments (in which miscalculations have very limited
consequences) and in well-studied problems (which have been tested and
validated extensively according to robust MLOps methods). However,
interpretability can be crucial in many cases, especially for systems deployed
in the healthcare domain [88]. The need for interpretability arises from the
incompleteness of the problem where system results require an accompanying
rationale.

#####  5.1.2.1 Importance of interpretability

Interpretability applied to an ML model can be useful for the following
reasons:

  * •

Trust: Interpretability enhances trust when all components are well-explained.
This builds an understanding of the decisions made by a model and may help
integrate it into the overall workflow.

  * •

Reliability & robustness: Interpretability can help in auditing ML models,
further increasing model reliability.

  * •

Privacy & security: Interpretability can be used to assess if any private
information is leaked from the results. While some researchers claim that
interpretability may hinder privacy [244, 102, 244] as the interpretable
features may leak sensitive information, others have shown that it can help
make the system robust against the adversarial attacks [145, 297].

  * •

Fairness: Interpretability can help in identifying and reducing biases
discussed in Sec. 5.1.1. However, the quality of these explanations can differ
significantly between subgroups and, as such, it is important to test various
explanation models in order to carefully select an equitable model with high
overall fidelity [24].

  * •

Better understanding and knowledge: A good interpretation of the model can
lead to the identification of the factors that most impact the model. This can
also result in a better understanding of the use case itself and enhance
knowledge in that particular area.

  * •

Causality: Interpretability gives a better understanding of the model
decisions and the features and hence can help to identify causal relationships
of the features [43].

#####  5.1.2.2 Types of approaches for interpretability in ML:

Many methods have been developed for better interpretability in ML, such as
explainable AI for trees [165], Tensorflow
Lattice272727https://www.tensorflow.org/lattice, DeepLIFT [143],
InterpretML[192], LIME [224], and SHAP [166]. Some of these have been applied
to healthcare [2, 247]. The methods for interpretability are usually
categorized as:

  * •

Model-based

    * –

Model-specific: Model-specific interpretability can only be used for a
particular model. Usually, this type of interpretability uses the model’s
internal structure to analyze the impact of features, for example.

    * –

Model-agnostic: Interpretability is not restricted to a specific machine
learning model and can be used more generally with several.

  * •

Complexity-based

    * –

Intrinsic: Relatively simple methods, such as height-bound decision trees, are
easier for humans to understand.

    * –

Post-hoc: After the model has produced output, interpretation proceeds for
more complex methods.

  * •

Scope-based

    * –

Locally interpretable: Interprets individual or per-instance predictions of
the model.

    * –

Globally interpretable: Interprets the model’s overall prediction set and
provides insight into how the model works in general.

  * •

Methodology-based approach

    * –

Feature-based: Methods that interpret the models based on the impact of the
features on that model. E.g., weight plot, feature selection, etc.

    * –

Perturbation-based: Methods that interpret the model by perturbing the
settings or features of the model. E.g., LIME [224], SHAP [166] and anchors.

    * –

Rule-based: Methods that apply rules on features to identify their impact on
the model e.g., BETA, MUSE, and decision trees.

    * –

Image-based: Methods where important inputs are shown using images
superimposed over the input e.g., saliency maps [10].

#####  5.1.2.3 Interpretability in healthcare

In recent years, interpretability has become common in healthcare [2, 179,
220]. In particular, Abdullah et al. [2] reported that interpretability
methods (e.g., decision trees, LIME, SHAP) have been applied to extract
insights into different medical conditions including cardiovascular diseases,
eye diseases, cancer, influenza, infection, COVID-19, depression, and autism.
Similarly, Meng et al. [179] performed interpretability of deep learning
mortality prediction models and fairness analysis on the MIMIC-III dataset
[119], showing connections between interpretability methods and fairness
metrics.

####  5.1.3 Privacy & Security

While digitizing healthcare has led to centralized data and improved access
for healthcare professionals, it has also increased risks to data security and
privacy [189]. After previous work [3], privacy is the individual’s ability to
control, interact with, and regulate their personal information and security
is a systemic protection of data from leaks or cyber-attacks.

#####  5.1.3.1 Security & privacy requirements

In order to ensure privacy and security, the following requirements should be
met [189]:

  * •

Authentication: Strong authentication mechanisms for accessing the system.

  * •

Confidentiality: Access to data and devices should be restricted to authorized
users.

  * •

Integrity: Integrity-checking mechanisms should be applied to restrict any
modifications to the data or to the system.

  * •

Non-repudiation: Logs should be maintained to monitor the system. Access to
those logs should be restricted and avoid any tampering.

  * •

Availability: Quick, easy, and fault-tolerant availability should be ensured
at all times.

  * •

Anonymity: Anonymity of the device, data, and communication should be
guaranteed.

  * •

Device unlinkability: An unauthorized person should not be able to establish a
connection between data and the sender.

  * •

Auditability and accountability: It should be possible to trace back the
recording time, recording person, and origins of the data to validate its
authenticity.

#####  5.1.3.2 Types of threats

Violation of privacy & security can occur either due to human error
(unintentional or non-malicious) or an adversarial attack (intentional or
malicious).

  1. 1.

Human error: Human error can cause data leakage through the carelessness or
incompetence of authorized individuals. Most of the literature in this context
[148, 75] divides human error into two types:

    1. (a)

Slip: the wrong execution of correct, intended actions; e.g., incorrect data
entry, forgetting to secure the data, giving access of information to
unauthorized persons using the wrong email address.

    2. (b)

Mistake: the right execution of incorrect, unintended actions; e.g.,
collecting data that is not required, using the same password for different
systems to avoid password recovery, giving access of information to
unauthorized persons assuming they can have access.

While people dealing with data should be trained to avoid such negligence,
some researchers have suggested policies, frameworks, and strategies such as
error avoidance, error interception, or error correction to prevent or
mitigate these issues [148, 75].

  2. 2.

Adversarial attacks: A primary risk for any digital data or system is from
adversarial attackers [99] who can damage, pollute, or leak information from
the system. An adversarial attacker can attack in many ways; e.g., they can be
remote or physically present, they can access the system through a third-party
device, or they can be personified as a patient [189]. The most common types
of attacks are listed below.

     * •

Hardware or software attack: Modifying the hardware or software to use it for
malicious purposes.

     * •

System unavailability: Making the device or data unavailable.

     * •

Communication attack: Interrupting the communication or forcing a device to
communicate with unauthorized external devices.

     * •

Data sniffing: Illegally capturing the communication to get sensitive
information.

     * •

Data modification: Maliciously modifying data.

     * •

Information leakage: Retrieving sensitive information from the system.

#####  5.1.3.3 Healthcare components and security & privacy

Extra care needs to be taken to protect healthcare data [5]. Components [194]
include:

  * •

Electronic health data: This data can be leaked due to human mistakes or
malicious attacks, which can result in tampering or misuse of data. In order
to overcome such risks, measures such as access control, cryptography,
anonymization, blockchain, steganography, or watermarking can be used.

  * •

Medical devices: Medical devices such as smartwatches and sensors are also
another source of information that can be attacked. Secure hardware and
software, authentication and cryptography can be used to avoid such problems.

  * •

Medical network: Data shared across medical professionals and organizations
through a networks may be susceptible to eavesdropping, spoofing,
impersonating, and unavailability attacks. These threats can be reduced by
applying encryption, authentication, access control, and compressed sensing.

  * •

Cloud storage: Cloud computing is becoming widely adopted in healthcare.
However, like any system, it is also prone to unavailability, data breaches,
network attacks, and malicious access. Similar to those above, threats to
cloud services can be avoided through authentication, cryptography, and
decoying (i.e., a method to make an attacker erroneously believe that they
have acquired useful information).

#####  5.1.3.4 Healthcare privacy & security laws

Due to the sensitivity of healthcare data and communication, many countries
have introduced laws and regulations such as the Personal Information
Protection and Electronic Documents Act (PIPEDA) in Canada, the Health
Insurance Portability and Accountability Act (HIPPA) in the USA, and the Data
Protection Directive in the EU [280]. These acts mainly aim at protecting
patient data from being shared or used without their consent but while
allowing them to access to their own data.

#####  5.1.3.5 Attacks on ML pipeline

Any ML model that learns from data can also leak information about the data,
even if it is generalized well; e.g., using membership inference (i.e.,
determining if a particular instance was used to train the model) [178, 111]
or using property inference (i.e., inferring properties of the training
dataset from a given model) [178, 200]. Adversarial attacks in the context of
the MLOps pipeline can occur in the following phases [99]:

  * •

Data collection phase: At this phase, a poisoning attack results in modified
or polluted data, impacting the training of the model and lowering performance
on unmodified data.

  * •

Modelling phase: Here, the Trojan AI attack can modify a model to provide an
incorrect response for specific trigger instances [271] by changing the model
architecture and parameters. Since it is now common to use pre-trained models,
these models can be modified or replaced by attackers.

  * •

Production and deployment phases: At these phases, both Trojan AI attacks and
evasion attacks can occur. Evasion attacks consist, e.g., of modifying test
data to have them misclassified [207].

####  5.1.4 Reliability, robustness and resilience

A trustworthy MLOps system should be reliable, robust, and resilient. These
terms are defined as follows [302]:

  * •

Reliability: The system performs in a satisfactory manner under specific,
unaltered operating conditions.

  * •

Robustness: The system performs in a satisfactory manner despite changes in
operating conditions, e.g., data shift.

  * •

Resilience: The system performs in a satisfactory manner despite a major
disruption in operating conditions; e.g., adversarial attacks.

These aspects have been studied in the healthcare domain [181, 213] and
different approaches such as interpretability, security, privacy, and methods
to deal with data shift (discussed in Sections 5.1.2 and 5.1.3) have been
suggested.

Trade-off between accuracy and trustworthiness: In Section 5.1, we discussed
different important components of trustworthy AI that should be considered
while designing an ML system; however, literature shows that there can be a
trade-off between accuracy, interpretability, and robustness [220, 256]. While
a main reason for the trade-off is that robust models learn a different
feature representation that may decrease accuracy, it is better perceived by
humans [256].

###  5.2 Incorporating Responsibility and Trust into MLHOps

In recent years, responsible and trustworthy AI have gained a lot of attention
in general as well as for healthcare due to its implications on society [220].
There are several definitions of trustworthiness [220], and they are related
to making the system robust, unbiased, generalizable, reproducible,
transparent, explainable, and secure. However, the lack of standardized
practices for applying, explaining, and evaluating trustworthiness in AI for
healthcare makes this very challenging [220]. In this section, we discuss how
we can incorporate all these qualities at each step of the pipeline.

####  5.2.1 Data

The process of a responsible and trustworthy MLOps pipeline starts with data
collection and preparation. The impact of biased or polluted data propagates
through all the subsequent steps of the pipeline [82]. This can be even more
important and challenging in the healthcare domain due to the privacy and
sensitivity of the data [21]. If compromised, this information can be tempered
or misused in various ways (e.g., identity theft, information sold to a third
party) and introduce bias in the healthcare system. Such challenges can also
cause economic harm (such as job loss), psychological harm (e.g., causing
embarrassment due to a medical issue), and social isolation (e.g., due to a
serious illness such as HIV) [187, 4]. It can also impact ML model performance
and trustworthiness [50].

#####  5.2.1.1 Data collection

In healthcare, data can be acquired through multiple sources [257], which
increases the chance of the data being polluted by bias. Bias can concern, for
example, race[284], gender, sexual orientation, gender identity, and
disability. Bias in healthcare data can be mitigated against by increasing
diversity in data, e.g., by including underrepresented minorities (URMs),
which can lead to better outcomes [169]. Debiasing during data collection can
include:

  1. 1.

Identifying & acknowledging potential real-world biases: Bias in healthcare is
introduced long before the data collection stage. Although increasingly less
common in many countries282828https://applymd.utoronto.ca/admission-stats,
bias can still occur in medical school admission, job interviews, patient
care, disease identification, research samples, and case studies. Such biases
lead to the dominance of people from certain communities [169] or in-group vs.
out-group bias [91], which can result in stereotyped and biased data
generation and hence biased data collection.

Bias can be unconscious or conscious [169, 79]. Unconscious bias stems from
implicit or unintentional associations outside conscious awareness resulting
from stereotypical perceptions and experiences. On the other hand, conscious
bias is explicit and intentional and has resulted in abuse and criminal acts
in healthcare; e.g., the Tuskegee study of untreated Syphilis in black men
demonstrated intentional racism [80]. Both conscious and unconscious biases
damage the validity of the data. Since conscious bias is relatively more
visible, it is openly discouraged not only in healthcare but also in all areas
of society. However, unconscious bias is more subtle and not as easy to
identify. In most cases, unconscious bias is not even known to the person
suffering from it.

Different surveys, tests, and studies have found the following types of biases
(conscious or unconscious) common in healthcare [169]:

    1. (a)

Racial bias e.g., Black, Hispanic, and Native American physicians are
underrepresented [197]. According to one study, white males from the upper
classes are preferred by the admission committees [42] (although some other
sources suggest the opposite28).

    2. (b)

Gender bias: e.g., professional women in healthcare being less likely to be
invited to give talks [177], to be introduced using professional titles [77],
to experience harassment or exclusion, to receive insufficient support at work
or negative comparisons with male colleagues, and to be perceived as weak &
less competitive [150, 252].

    3. (c)

Gender minority bias e.g., LGBTQ people receive lower quality healthcare [226]
and faced challenges to get jobs in healthcare [232].

    4. (d)

Disability bias e.g., people with disabilities receive limited accessibility
supports to all facilities and have to work harder to be feel validated or
recognized [175].

Various tests identify the existence of unconscious bias, such as the Implicit
Association Test (IAT), and have been reported to be useful. For example, Race
IAT results detected unintentional bias in $75\%$ of the population taking the
test [25]. While debate continues regarding the degree of usefulness of these
tests [34], they may still capture some subtle human behaviours. Some other
assessment tools (e.g., Diversity Engagement Survey (DES) [203]) have also
been built for successfully measuring inclusion and diversity in medical
institutes.

According to Marcelin et al. [169], the following measures can help in
reducing unintentional bias:

    1. (a)

Using IAT to identify potential biases in admissions or hiring committee
members in advance.

    2. (b)

Promoting equity, diversity, inclusion, and accessibility (EDIA) in teams.
Including more people from underrepresented minorities (URM) in the healthcare
profession, especially in admissions and hiring committees.

    3. (c)

Conducting and analyzing surveys to keep track of the challenges faced by URM
individuals due to the biased perception of them.

    4. (d)

Training to highlight the existence and need for mitigation of bias.

    5. (e)

Self-monitoring bias can be another way to incorporate inclusion and
diversity.

  2. 2.

Debiasing during data collection and annotation:
In addition to human factors, we can take steps to improve the data collection
process itself. In this regard, the following measures can be taken [156]:

    1. (a)

Investigating the exclusion: In dataset creation, an important step is to
carefully investigate which patients are included in the dataset. An exclusion
criterion in dataset creation may be conscious and clinically motivated, but
there are many unintentional exclusion criteria that are not very well visible
and enforce biases. For instance, a dataset that is gathered in a research
hospital that does not routinely provide standard diagnostic and treatment
services and select the patients only because they have an illness being
studied by the Institutes have a different type of patients compared to
clinical hospitals that do not have these limitations [242]. Alternatively,
whether the service delivered to the patient is free or covered by insurance
would change the distribution of the patients and infect biases into the
resulting AI model [241].

    2. (b)

Annotation with explanation: Adding justification for choosing the label by
the human annotators not only helps them identify their own unconscious biases
but also can help in setting standards for unbiased annotations and avoid any
automatic association and stereotyping (e.g., high prevalence HIV in gay men
led to under-diagnosis of this disease in women and children [169]. Moreover,
these explanations can be a good resource for training explainable AI models
[277].

    3. (c)

Data provenance: This involves tracking data lineage through the data source,
dependencies, and data collection process. Healthcare data can come from
multiple sources which increases the chances of it being biased [45]. Data
provenance improves data quality, integrity, audibility, and transparency
[283]. Different tools for data provenance are available including Fast
Healthcare Interoperability Resources (FHIR) [233] and Atmolytics[283]. [171]

    4. (d)

Data security & privacy during data collection: Smart healthcare technologies
have become a common practice [45]. A wide variety of smart devices is
available, including wearable devices (e.g., smartwatches, skin-based
sensors), body area networks (e.g., EEG sensors, blood pressure sensors),
tele-healthcare (e.g., tele-monitoring, tele-treatment), digital healthcare
systems (e.g., electronic health records (EHR), electronic medical records
(EMR)), and health analytics (e.g., medical big-data). While the digitization
of healthcare has improved access to medical facilities, it has increased the
risk of data leakage and malicious attacks. Extra care should be taken while
designing an MLOps pipeline to avoid privacy and security risks, as it can
lead to serious life-threatening consequences. Other issues include the number
of people involved in using the data and proper storage for high volumes of
data. Chaudhry et al. [45] proposed an AI-based framework using 6G-networks
for secure data exchange in digital healthcare devices. In the past decade,
the blockchain has also emerged as a way of ensuring data privacy and
security. Blockchain is a distributed database with unique characteristics
such as immutability, decentralization, and transparency. This is especially
relevant in healthcare because of security and privacy issues [101, 286, 190].
Using blockchain can help in more efficient and secure management of patient’s
health records, transparency, identification of false content, patient
monitoring, and maintaining financial statements [101].

    5. (e)

Data-sheet: Often, creating a dataset that represents the full diversity of a
population is not feasible, especially for very multi-cultural societies.
Additionally, the prevalence of diseases among different sub-populations may
be different [242]. If it is not possible to build an ideal dataset with the
above specifications, the data needs to be delivered by a data-sheet. The
data-sheet is meta-data that helps to analyze and specify the characteristics
of the data, clearly explain exclusion and inclusion criteria, detail
demographic features of the patients, and statistics of the data distribution
over sub-populations, labels and features.

#####  5.2.1.2 Data pre-processing

  1. 1.

Data quality assurance: Sendak et al. [238] argued that clinical researchers
choose data for research very carefully but the machine learning community in
healthcare does not follow this practice. To overcome this gap, they suggest
that data points are identified by the clinicians and extracted into a
project-specific data store. After this, a three-step framework is applied:
(1) use different measures for data pre-processing to ensure the correctness
of all data elements (e.g, converting each lab measurement to the same unit),
(2) ensure completeness, conformance, plausibility, and possible data shifts,
and (3) adjudicate the data with the clinicians.

  2. 2.

Data anonymization: Due to the sensitivity of healthcare data preparation,
data anonymization should minimize the chances of it being de-anonymized.
Olatunji et al. [196] provide a detailed overview of data anonymization models
and techniques in healthcare such as k-anonymity, k-map, l-diversity,
t-closeness, $\delta$-disclosure privacy, $\beta$-likeness, $\delta$-presence,
and ($\epsilon$, $\delta$)-differential privacy. To avoid data leakage, many
tools for data anonymization and its evaluation tools [268] such as SecGraph
[116], ARX- tool for anonymizing biomedical data [212],
Amnesia292929https://www.openaire.eu/item/amnesia-data-anonymization-made-easy
[254], PySyft [230], Synthea [270] and
Anonimatron303030https://realrolfje.github.io/anonimatron/ (open-source data
anonymization tool written in Java) can be incorporated in the MLHOps
pipeline.

  3. 3.

Removing subgroups indicators. Changing the race of the patients can have a
dramatic impact on the outcome of an algorithm that is designed to fill a
prompt [295]. Therefore, the existence of race attributes in the text can
decrease the fairness of the model dramatically. In some specific problems,
removing subgroup indicators such as the sex of a job candidate from their
application has shown to have minimal influence on classifier accuracy while
improving the fairness [64]. This method is applicable mostly in text-based
data where sensitive attributes are easily removable. As a preprocessing step,
one can estimate the effect of keeping or removing such sensitive attributes
on the overall accuracy and fairness of a developed model. At the same time,
it is not always possible to remove the sensitive attributes from the data.
For example, AI models can predict patient race from medical images, but it is
not yet clear how they can do it [276]. In one study [276], researchers did
not provide the patient race during model training, but they also could not
find a particular patch or region in the data for which AI failed to detect
race by removing that part.

  4. 4.

Differential privacy: Differential privacy [61] aims to provide information
about inherent groups while withholding the information about the individuals.
Many algorithms and tools have been developed for this, including CapC[53] and
PySyft [230].

####  5.2.2 Methodology

The following sections overview the steps to put these concepts into practice.

#####  5.2.2.1 Algorithmic fairness

Algorithmic fairness [183, 282, 83] attempts to ensure the unbiased output
across the available classes. Here, we discuss how we can overcome this
challenge at different stages of model training [183, 282].

  1. 1.

Pre-processing

     * •

Choice of sampling & data augmentation: Making sure that the dataset is
balanced (having approximately an equal number of instances from each class)
and all the classes get equal representation in the dataset using simple
under- or over-sampling methods [282]. This can also be done by data
augmentation [180, 81] to improve the counterfactual fairness by
counterfactual text generation and using it to augment data. Augmentation
methods include Synthetic Minority Oversampling Technique (SMOTE) [46] and
Adaptive Synthetic Sampling (ADASYN) [107]. Since synthetic samples may not be
universally beneficial for the healthcare domain, acquiring more data and
undersampling may be the best strategy [282].

     * •

Causal fairness using data pre-processing: Causal fairness is achieved by
reducing the impact of protected or sensitive attributes (e.g., race and
gender) on predicted variables and different methods have been developed to
accomplish this [83, 281]. Kamiran et al. [121] proposed “massaging the data”
before using traditional classification algorithms.

     * •

Re-weighing: In a pre-processing approach, one may re-weight the training
dataset samples or remove features with high correlation to sensitive
attributes as well as the sensitive attribute itself [122], learning
representations that are relatively invariant to sensitive attribute [162].
One might also adjust representation rates of protected groups and achieve
target fairness metrics [44], or utilize optimization to learn a data
transformation that reduce discrimination [40].

  2. 2.

In-processing

     * •

Adversarial learning: It is also possible to enforce fairness during model
training, using adversarial debiasing [291, 221, 274]. Adversarial learning
refers to the methods designed to intentionally confound ML models during
training, through deceptive or misleading inputs, to make those models more
robust. This technique has been used in healthcare to create robust models
[125], and for bias mitigation, by intentionally inputting biased examples
[144, 204].

     * •

Prejudice remover: Another important aspect is prejudice injected into the
features [123]. Prejudice can be (a) Direct prejudice: using a protected
attribute as a prediction variable, (b) Indirect prejudice: statistical
dependence between protected attributes and prediction variables, and (c)
Latent prejudice: statistical dependence between protected attributes and non-
protected attributes. Kamishaima et al. [123] proposed a method to remove
prejudice using regularization. Similarly, Grgic et al. [94] introduced a
method using constraints for classifier optimization objectives to remove
prejudice.

     * •

Enforcing fairness in the model training: Fairness can also be enforced by
making changes to the model through constraint optimization [159], modifying
loss functions to penalize deviation from the general population for
subpopulations [205], regularizing loss function to minimize mutual
information between feature embedding and bias [128], or adding regularizer to
identify and treat latent discriminating features [123].

     * •

Up-weighing: It is possible to improve the outcome on worst case group by up
weighting the groups with the largest loss [292, 231, 173]. However, all these
methods need awareness about the membership of the instance to sensitive
attributes. There are also group un-aware methods where they try to weights
each sample with an adversary that tries to maximize the weighted loss [136],
or trains an additional classier that up-weights samples classified
incorrectly in the last training step [154].

  3. 3.

Post-processing: The post-processing fairness mitigation approaches may target
post-hoc calibration of model predictions. This method has sown impact in bias
mitigation in both non-healthcare [103, 210] and healthcare [129]
applications.

There are some software tools and libraries for algorithmic fairness check,
listed in [282], which can be used by developer and end user to evaluate the
fairness of the AI model outcomes.

####  5.2.3 Development & evaluation

At this stage, the ML system is evaluated to make sure its trustworthiness,
which includes evaluating the evaluation methods [220, 17].

#####  5.2.3.1 Model interpretability & explainability

At this stage, model evaluation can be done through interpretability and
explainability methods to mitigate any potential issues such as possible
anomalies in the data or the model. However, it should be noted that the
methods perform interpretability and explainability should also be evaluated
carefully before relying on them, which can be performed using different
methods such as human evaluation [170, 24].

##  6 Concluding remarks

Machine learning (ML) has been applied to many clinically-relevant tasks and
many relevant datasets in the research domain but, to fully realize the
promise of ML in healthcare, practical considerations that are not typically
necessary or even common in the research community must be carefully designed
and adhered to. We have provided a deep survey into a breadth of these ML
considerations, including infrastructure, human resources, data sources, model
deployment, monitoring and updating, bias, interpretability, privacy and
security.

As there are an increasing number of AI systems being deployed into medical
practice, it is important to standardize and to specify specific engineering
pipelines for medical AI development and deployment, a process we term MLHOps.
To this end, we have outlined the key steps that should be put into practice
by multidisciplinary teams at the cutting-edge of AI in healthcare to ensure
the responsible deployment of clinical AI systems.

##  7 Appendix

Name of tool |  Description
---|---
MIMIC-Extract |  Pipeline to transform data from MIMIC-III into DataFrames that are directly usable for ML modelling
Clairvoyance |  End-to-End AutoML Pipeline for Medical Time Series
Pyhealth |  A python library for health predictive models
ROMOP |  R package to easily interface with OMOP-formatted EHR data
ATLAS |  Research tool to conduct scientific analyses on data available in OMOP format
FIDDLE |  Preprocessing pipeline that transforms structured EHR data into feature vectors for clinical use cases
hi-ml |  Toolbox for deep learning for medical imaging and Azure integration
MedPerf |  An open benchmarking platform for medical artificial intelligence using Federated Evaluation.
MONAI |  AI Toolkit for Healthcare Imaging
TorchXRayVision |  A library of chest X-ray datasets and models
Leaf |  Clinical Data Explorer
Table 4: List of open-source tools available on Github that can be used for ML system development specific to health. Table 5: Key Roles in an MLOps Team 4 Role |  Alternatively |  Description
---|---|---
4 Domain Expert |  • Business Translator • Business Stakeholder • PO/Manager |  An instrumental role in any phase of the MLOps process where a deeper understanding of the data and the domain is required.
Solution Architect |  • IT Architect • ML Architect |  Unifying the work of data scientists, data engineers, and software developers through developing strategies for MLOps processes, defining the project lifecycle, and identifying the best tools and assemble the team of engineers and developers to work on projects.
Data Scientist |  • ML Specialist • ML Developer |  A central player in any MLOps team responsible for creating the data and ML model pipelines. The pipelines include analysing and processing the data as well as building and testing the ML models.
Data Engineer |  • DataOps Engineer • Data Analyst |  Working in coordination with product manager and domain expert to uncover insights from data through data ingestion pipelines.
Software Developer |  • Full-stack engineer |  Focusing on the productionizing of ML models and the supporting infrastructure based on the ML architect’s blueprints. They standardize the code for compatibility and re-usability
DevOps Engineer |  • CI/CD Engineer |  Facilitating access to the specialized tools and high performance computing infrastructure, enabling transition from development to deployment and monitoring, and automating ML lifecycle.
ML Engineer |  • MLOps Engineer |  Highly skilled programmers supporting designing and deploying ML models in close collaboration with Data Scientists and DevOps Engineers.
4 |  |
Tool |  Description
---|---
FairMLHealth313131https://github.com/KenSciResearch/fairMLHealth |  Tools and tutorials for variation analysis in healthcare machine learning.
AIF360 [33] |  An open-source library containing techniques developed by the research community to help detect and mitigate bias in machine learning models throughout the AI application lifecycle.
Fairlearn323232https://fairlearn.org/ |  An open-source, community-driven project to help data scientists improve the fairness of AI systems.
Fairness-comparison333333https://github.com/algofairness/fairness-comparison |  Benchmark fairness-aware machine learning techniques.
Fairness Indicators343434https://www.tensorflow.org/responsible_ai/fairness_indicators/tutorials/Fairness_Indicators_Example_Colab |  Fairness Indicators is a suite of tools built on top of TensorFlow Model Analysis (TFMA) that enable regular evaluation of fairness metrics in product pipelines.
ML-fairness-gym353535https://ai.googleblog.com/2020/02/ml-fairness-gym-tool-for-exploring-long.html |  A tool for exploring long-term impacts of ML systems.
themis-ml [27] |  An open source machine learning library that implements several fairness-aware methods that comply with the sklearn API.
FairML [11] |  ToolBox for diagnosing bias in predictive modelling.
Black Box Auditing [12] |  A toolkit for auditing ML model deviations.
What-If Tool363636https://pair-code.github.io/what-if-tool/ |  Visually probe the behaviour of trained machine learning models, with minimal coding.
Aequitas373737http://www.datasciencepublicpolicy.org/our-work/tools-guides/aequitas/ |  An open source bias audit toolkit for machine learning developers, analysts, and policymakers to audit machine learning models for discrimination and bias, and make informed and equitable decisions around developing and deploying predictive risk-assessment tools.
DECAF [259] |  A fair synthetic data generator for tabular data utilizing GANs and causal models.
REPAIR [146] |  A dataset resampling algorithm to reduce representation bias by reweighting.
CERTIFAI [243] |  Evaluates AI models for robustness, fairness, and explainability, and allows users to compare different models or model versions for these qualities.
FairSight [13] |  A fair decision making pipeline to assist decision makers track fairness throughout a model.
Adv-Demog-Text [72] |  An adversarial network demographic attributes remover from text data.
GN-GloVe [298] |  A framework for generating gender neutral word embeddings.
Tensorflow Constrained Optimization 383838https://github.com/google-research/tensorflow_constrained_optimization |  A library for optimizing inequality-constrained problems using rate helpers.
Responsibly393939https://github.com/ResponsiblyAI/responsibly [60] |  Toolkit for auditing and mitigating bias and fairness of ML systems.
Dataset-Nutrition-Label [110] |  The Data Nutrition Project aims to create a standard label for interrogating datasets.
Table 6: List of open-source tools available on Github that can be used for ML
Monitoring and Updating specific to health.

## References

  * [1] Abdullah A Abdullah, Masoud M Hassan, and Yaseen T Mustafa.  A review on bayesian deep learning in healthcare: Applications and challenges.  IEEE Access, 2022.
  * [2] Talal AA Abdullah, Mohd Soperi Mohd Zahid, and Waleed Ali.  A review of interpretable ml in healthcare: Taxonomy, applications, challenges, and future directions.  Symmetry, 13(12):2439, 2021.
  * [3] Adnan Ahmed Abi Sen and Abdullah M Basahel.  A comparative study between security and privacy.  In 2019 6th International Conference on Computing for Sustainable Global Development (INDIACom), pages 1282–1286. IEEE, 2019.
  * [4] Karim Abouelmehdi, Abderrahim Beni-Hessane, and Hayat Khaloufi.  Big healthcare data: preserving security and privacy.  Journal of big data, 5(1):1–18, 2018.
  * [5] Karim Abouelmehdi, Abderrahim Beni-Hssane, Hayat Khaloufi, and Mostafa Saadi.  Big data security and privacy in healthcare: A review.  Procedia Computer Science, 113:73–80, 2017.
  * [6] George A. Adam, Chun-Hao K. Chang, Benjamin Haibe-Kains, and Anna Goldenberg.  Hidden risks of machine learning applied to healthcare: Unintended feedback loops between models and future data causing model degradation.  Proceedings of Machine Learning Research, (126):710–731, 2020.
  * [7] George A. Adam, Chun-Hao K. Chang, Benjamin Haibe-Kains, and Anna Goldenberg.  Hidden risks of machine learning applied to healthcare: Unintended feedback loops between models and future data causing model degradation.  Proceedings of Machine Learning Research, (182):1–26, 2022.
  * [8] Roy Adams, Katharine E Henry, Anirudh Sridharan, Hossein Soleimani, Andong Zhan, Nishi Rawat, Lauren Johnson, David N Hager, Sara E Cosgrove, Andrew Markowski, et al.  Prospective, multi-site study of patient outcomes after implementation of the trews machine learning-based early warning system for sepsis.  Nature medicine, pages 1–6, 2022.
  * [9] Adewole S. Adamson and Avery Smith.  Machine learning and health care disparities in dermatology.  JAMA Dermatology, 154(11):1247–1248, 2018.
  * [10] Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim.  Sanity checks for saliency maps.  Advances in neural information processing systems, 31, 2018.
  * [11] Julius A Adebayo et al.  FairML: ToolBox for diagnosing bias in predictive modeling.  PhD thesis, Massachusetts Institute of Technology, 2016.
  * [12] Philip Adler, Casey Falk, Sorelle A Friedler, Tionney Nix, Gabriel Rybeck, Carlos Scheidegger, Brandon Smith, and Suresh Venkatasubramanian.  Auditing black-box models for indirect influence.  Knowledge and Information Systems, 54:95–122, 2018.
  * [13] Yongsu Ahn and Yu-Ru Lin.  Fairsight: Visual analytics for fairness in decision making.  IEEE transactions on visualization and computer graphics, 26(1):1086–1095, 2019.
  * [14] Amr M. Alexandari, Anshul Kundaje, and Avanti Shrikumar.  Maximum likelihood with bias-corrected calibration is hard-to-beat at label shift adaptation.  In International Conference on Machine Learning, pages 222–232. PMLR, 2020.
  * [15] Emily Alsentzer, John R Murphy, Willie Boag, Wei-Hung Weng, Di Jin, Tristan Naumann, and Matthew McDermott.  Publicly available clinical bert embeddings.  arXiv preprint arXiv:1904.03323, 2019.
  * [16] Fatemeh Amrollahi, Supreeth P Shashikumar, Andre L Holder, and Shamim Nemati.  Leveraging clinical data across healthcare institutions for continual learning of predictive risk models.  Scientific Reports, 12(1):1–10, 2022.
  * [17] Tony Antoniou and Muhammad Mamdani.  Evaluation of machine learning solutions in medicine.  CMAJ, 193(36):E1425–E1429, 2021.
  * [18] Mahsa Arabahmadi, Reza Farahbakhsh, and Javad Rezazadeh.  Deep learning for smart healthcare—a survey on brain tumor detection from medical imaging.  Sensors, 22(5):1960, 2022.
  * [19] Jacob Armstrong and David A Clifton.  Continual learning of longitudinal health records.  In 2022 IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI), pages 01–06. IEEE, 2022.
  * [20] Anand Avati, Martin Seneviratne, Emily Xue, Zhen Xu, Balaji Lakshminarayanan, and Andrew M. Dai.  Beds-bench: Behavior of ehr-models under distributional shift–a benchmark.  arXiv preprint arXiv:2107.08189, 2021.
  * [21] Joseph Bamidele Awotunde, Rasheed Gbenga Jimoh, Sakinat Oluwabukonla Folorunso, Emmanuel Abidemi Adeniyi, Kazeem Moses Abiodun, and Oluwatobi Oluwaseyi Banjo.  Privacy and security concerns in iot-based healthcare systems.  In The Fusion of Internet of Things, Artificial Intelligence, and Cloud Computing in Health Care, pages 105–134. Springer, 2021.
  * [22] K. Azizzadenesheli, A. Liu, F. Yang, and A. Anandkumar.  Regularized learning for domain adaptation under label shifts.  arXiv preprint arXiv:1903.09734, 2019.
  * [23] Manuel Baena-Garcıa, Jose del Campo-Avila, Raul Fidalgo, Albert Bifet, Ricard Gavalda, and Rafael Morales-Bueno.  Early drift detection method.  In Fourth international workshop on knowledge discovery from data streams, 6:77–86, 2006.
  * [24] Aparna Balagopalan, Haoran Zhang, Kimia Hamidieh, Thomas Hartvigsen, Frank Rudzicz, and Marzyeh Ghassemi.  The road to explainability is paved with bias: Measuring the fairness of explanations.  arXiv preprint arXiv:2205.03295, 2022.
  * [25] MR Banaji and G Greenwald.  Blindspot: Hidden biases of good people [kindle ipad version], 2013.
  * [26] Shahab S Band, Sina Ardabili, Atefeh Yarahmadi, Bahareh Pahlevanzadeh, Adiqa Kausar Kiani, Amin Beheshti, Hamid Alinejad-Rokny, Iman Dehzangi, Arthur Chang, Amir Mosavi, et al.  A survey on machine learning and internet of medical things-based approaches for handling covid-19: Meta-analysis.  Frontiers in Public Health, 10, 2022.
  * [27] Niels Bantilan.  Themis-ml: A fairness-aware machine learning interface for end-to-end discrimination discovery and mitigation.  Journal of Technology in Human Services, 36(1):15–30, 2018.
  * [28] David W Bates, Suchi Saria, Lucila Ohno-Machado, Anand Shah, and Gabriel Escobar.  Big data in health care: using analytics to identify and manage high-risk and high-cost patients.  Health affairs, 33(7):1123–1131, 2014.
  * [29] Firas Bayram, Bestoun S Ahmed, and Andreas Kassler.  From concept drift to model degradation: An overview on performance-aware drift detectors.  Knowledge-Based Systems, page 108632, 2022.
  * [30] Ashwin Belle, Raghuram Thiagarajan, SM Soroushmehr, Fatemeh Navidi, Daniel A Beard, and Kayvan Najarian.  Big data analytics in healthcare.  BioMed research international, 2015, 2015.
  * [31] Duane Bender and Kamran Sartipi.  Hl7 fhir: An agile and restful approach to healthcare information exchange.  In Proceedings of the 26th IEEE International Symposium on Computer-Based Medical Systems, pages 326–331, 2013.
  * [32] Ayne A. Beyene, Tewelle Welemariam, Marie Persson, and Niklas Lavesson.  Improved concept drift handling in surgery prediction and other applications.  Knowledge and Information Systems, 44(1):177–196, 2015.
  * [33] Sarah Bird, Miro Dudík, Richard Edgar, Brandon Horn, Roman Lutz, Vanessa Milan, Mehrnoosh Sameki, Hanna Wallach, and Kathleen Walker.  Fairlearn: A toolkit for assessing and improving fairness in ai.  Microsoft, Tech. Rep. MSR-TR-2020-32, 2020.
  * [34] Hart Blanton, James Jaccard, Jonathan Klick, Barbara Mellers, Gregory Mitchell, and Philip E Tetlock.  Strong claims and weak evidence: reassessing the predictive validity of the iat.  Journal of applied Psychology, 94(3):567, 2009.
  * [35] Lucas Bourtoule, Varun Chandrasekaran, Christopher A. Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot.  Machine unlearning.  In 2021 IEEE Symposium on Security and Privacy (SP), pages 141–159, 2021.
  * [36] Jonathan Brophy and Daniel Lowd.  Machine unlearning for random forests.  In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 1092–1104. PMLR, 18–24 Jul 2021.
  * [37] Li Bu, Cesare Alippi, and Dongbin Zhao.  A pdf-free change detection test based on density difference estimation.  IEEE transactions on neural networks and learning systems, 29(2):324–334, 2016.
  * [38] Joy Buolamwini and Timnit Gebru.  Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classiﬁcation.  In Proceedings of the 1st Conference on Fairness, Accountability and Transparency, volume 81 of FAT*’18, page 15, 2018.
  * [39] Aurelia Bustos, Antonio Pertusa, Jose-Maria Salinas, and Maria de la Iglesia-Vayá.  Padchest: A large chest x-ray image dataset with multi-label annotated reports.  Medical image analysis, 66:101797, 2020.
  * [40] Flavio Calmon, Dennis Wei, Bhanukiran Vinzamuri, Karthikeyan Natesan Ramamurthy, and Kush R Varshney.  Optimized pre-processing for discrimination prevention.  30, 2017.
  * [41] Tianshi Cao, Chinwei Huang, David Yu-Tung Hui, and Joseph Paul Cohen.  A benchmark of medical out of distribution detection.  arXiv preprint arXiv:2007.04250, 2020.
  * [42] Quinn Capers IV, Daniel Clinchot, Leon McDougle, and Anthony G Greenwald.  Implicit racial bias in medical school admissions.  Academic Medicine, 92(3):365–369, 2017.
  * [43] Diogo V Carvalho, Eduardo M Pereira, and Jaime S Cardoso.  Machine learning interpretability: A survey on methods and metrics.  Electronics, 8(8):832, 2019.
  * [44] L. Elisa Celis, Vijay Keswani, and Nisheeth Vishnoi.  Data preprocessing to mitigate bias: A maximum entropy based approach.  119:1349–1359, 2020.
  * [45] Sachi Chaudhary, Riya Kakkar, Nilesh Kumar Jadav, Anuja Nair, Rajesh Gupta, Sudeep Tanwar, Smita Agrawal, Mohammad Dahman Alshehri, Ravi Sharma, Gulshan Sharma, et al.  A taxonomy on smart healthcare technologies: Security framework, case study, and future directions.  Journal of Sensors, 2022, 2022.
  * [46] Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer.  Smote: synthetic minority over-sampling technique.  Journal of artificial intelligence research, 16:321–357, 2002.
  * [47] Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu.  Recurrent neural networks for multivariate time series with missing values.  Scientific reports, 8(1):1–12, 2018.
  * [48] Irene Chen, Fredrik D Johansson, and David Sontag.  Why Is My Classifier Discriminatory?  In Advances in Neural Information Processing Systems 31, pages 3539–3550. Curran Associates, Inc., 2018.
  * [49] Irene Chen, Shalmali Joshi, and Marzyeh Ghassemi.  Treating health disparities with artificial intelligence.  volume 26, page 16–17, 2020.
  * [50] Irene Y Chen, Emma Pierson, Sherri Rose, Shalmali Joshi, Kadija Ferryman, and Marzyeh Ghassemi.  Ethical machine learning in healthcare.  Annual review of biomedical data science, 4:123–144, 2021.
  * [51] Jonathan H Chen, Muthuraman Alagappan, Mary K Goldstein, Steven M Asch, and Russ B Altman.  Decaying relevance of clinical data towards future decisions in data-driven inpatient clinical order sets.  International journal of medical informatics, 102:71–79, 2017.
  * [52] Weijie Chen, Berkman Sahiner, Frank Samuelson, Aria Pezeshk, and Nicholas Petrick.  Calibration of medical diagnostic classifier scores to the probability of disease.  Statistical methods in medical research, 27(5):1394–1409, 2018\.
  * [53] Christopher A Choquette-Choo, Natalie Dullerud, Adam Dziedzic, Yunxiang Zhang, Somesh Jha, Nicolas Papernot, and Xiao Wang.  Capc learning: Confidential and private collaborative learning.  arXiv preprint arXiv:2102.05188, 2021.
  * [54] Shir Chorev, Philip Tannor, Dan Ben Israel, Noam Bressler, Itay Gabbay, Nir Hutnik, Jonatan Liberman, Matan Perlmutter, Yurii Romanyshyn, and Lior Rokach.  Deepchecks: A Library for Testing and Validating Machine Learning Models and Data.
  * [55] Alexandra Chouldechova.  Fair prediction with disparate impact: A study of bias in recidivism prediction instruments.  Big data, 5(2):153–163, 2016.
  * [56] Oliver Cobb and Arnaud Van Looveren.  Context-aware drift detection.  In International Conference on Machine Learning, pages 4087–4111. PMLR, 2022.
  * [57] Gary S Collins, Paula Dhiman, Constanza L Andaur Navarro, Jie Ma, Lotty Hooft, Johannes B Reitsma, Patricia Logullo, Andrew L Beam, Lily Peng, Ben Van Calster, et al.  Protocol for development of a reporting guideline (tripod-ai) and risk of bias tool (probast-ai) for diagnostic and prognostic prediction model studies based on artificial intelligence.  BMJ open, 11(7):e048008, 2021.
  * [58] Sam Corbett-Davies and Sharad Goel.  The measure and mismeasure of fairness: A critical review of fair machine learning.  2018\.
  * [59] Conor K Corbin, Rob Maclay, Aakash Acharya, Sreedevi Mony, Soumya Punnathanam, Rahul Thapa, Nikesh Kotecha, Nigam H Shah, and Jonathan H Chen.  Deployr: A technical framework for deploying custom real-time machine learning models into the electronic medical record.  arXiv preprint arXiv:2303.06269, 2023.
  * [60] Andrew Cotter, Maya Gupta, Heinrich Jiang, Nathan Srebro, Karthik Sridharan, Serena Wang, Blake Woodworth, and Seungil You.  Training well-generalizing classifiers for fairness metrics and other data-dependent constraints.  In International Conference on Machine Learning, pages 1397–1405. PMLR, 2019.
  * [61] Fida Kamal Dankar and Khaled El Emam.  Practicing differential privacy in health care: A review.  Trans. Data Priv., 6(1):35–67, 2013.
  * [62] Sharon E Davis, Robert A Greevy Jr, Christopher Fonnesbeck, Thomas A Lasko, Colin G Walsh, and Michael E Matheny.  A nonparametric updating method to correct clinical prediction model drift.  Journal of the American Medical Informatics Association, 26(12):1448–1457, 2019.
  * [63] Angus Dawson.  Trust, trustworthiness and health.  Forum for Medical Ethics Society, 2015.
  * [64] Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai.  Bias in bios: a case study of semantic representation bias in a high-stakes setting.  In Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT*’19, pages 120–128, USA, 2019.  event-place: Atlanta, GA.
  * [65] Anne AH de Hond, Artuur M Leeuwenberg, Lotty Hooft, Ilse MJ Kant, Steven WJ Nijman, Hendrikus JA van Os, Jiska J Aardoom, Thomas Debray, Ewoud Schuit, Maarten van Smeden, et al.  Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review.  npj Digital Medicine, 5(1):1–13, 2022.
  * [66] Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars.  A continual learning survey: Defying forgetting in classification tasks.  IEEE transactions on pattern analysis and machine intelligence, 44(7):3366–3385, 2021.
  * [67] Kevin Donnelly et al.  Snomed-ct: The advanced terminology and coding system for ehealth.  Studies in health technology and informatics, 121:279, 2006.
  * [68] Finale Doshi-Velez and Been Kim.  Towards a rigorous science of interpretable machine learning.  arXiv preprint arXiv:1702.08608, 2017.
  * [69] Xinyi Du-Harpur, Callum Arthurs, Clarisse Ganier, Rick Woolf, Zainab Laftah, Manpreet Lakhan, Amr Salam, Bo Wan, Fiona M. Watt, Nicholas M. Luscombe, and Magnus D. Lynch.  Clinically relevant vulnerabilities of deep machine learning systems for skin cancer diagnosis.  J Invest Dermatol., 141(4):916–920, 2021.
  * [70] Christopher Duckworth, Francis P. Chmiel, Dan K. Burns, Zlatko D. Zlatev, Neil M. White, Thomas W. V. Daniels, Michael Kiuber, and Michael J. Boniface.  Using explainable machine learning to characterise data drift and detect emergent health risks for emergency department admissions during covid-19.  Sci Rep, 11:23017, 2021.
  * [71] Michael Ekstrand, Robin Burke, and Fernando Diaz.  Fairness and discrimination in recommendation and retrieval.  In RecSys ’19: Proceedings of the 13th ACM Conference on Recommender Systems, RecSys*’19, page 576–577, 2016.
  * [72] Yanai Elazar and Yoav Goldberg.  Adversarial removal of demographic attributes from text data.  arXiv preprint arXiv:1808.06640, 2018.
  * [73] Andre Esteva, Brett Kuprel, Roberto A. Novoa, Justin Ko, Susan M. Swetter, Helen M. Blau, and Sebastian Thrun.  Dermatologist-level classification of skin cancer with deep neural networks.  Nature, 542(7639):115–118, February 2017.
  * [74] Andre Esteva, Alexandre Robicquet, Bharath Ramsundar, Volodymyr Kuleshov, Mark DePristo, Katherine Chou, Claire Cui, Greg Corrado, Sebastian Thrun, and Jeff Dean.  A guide to deep learning in healthcare.  Nature medicine, 25(1):24–29, 2019.
  * [75] Mark Evans, Ying He, Leandros Maglaras, and Helge Janicke.  Heart-is: A novel technique for evaluating human error-related information security incidents.  Computers & Security, 80:74–89, 2019.
  * [76] Jean Feng, Rachael V Phillips, Ivana Malenica, Andrew Bishara, Alan E Hubbard, Leo A Celi, and Romain Pirracchio.  Clinical artificial intelligence quality improvement: towards continual monitoring and updating of ai algorithms in healthcare.  npj Digital Medicine, 5(1):1–9, 2022.
  * [77] Julia A Files, Anita P Mayer, Marcia G Ko, Patricia Friedrich, Marjorie Jenkins, Michael J Bryan, Suneela Vegunta, Christopher M Wittich, Melissa A Lyle, Ryan Melikian, et al.  Speaker introductions at internal medicine grand rounds: forms of address reveal gender bias.  Journal of women’s health, 26(5):413–419, 2017.
  * [78] Samuel G. Finlayson, Adarsh Subbaswamy, Karandeep Singh, John Bowers, Annabel Kupke, Jonathan Zittrain, Isaac S. Kohan, and Suchi Saria.  The clinician and dataset shift in artificial intelligence.  New England Journal of Medicine, 385(3):283–286, 2021.
  * [79] Chloë FitzGerald and Samia Hurst.  Implicit bias in healthcare professionals: a systematic review.  BMC medical ethics, 18(1):1–18, 2017.
  * [80] Charles K Francis.  Medical ethos and social responsibility in clinical medicine.  Journal of Urban Health, 78(1):29–45, 2001.
  * [81] Zee Fryer, Vera Axelrod, Ben Packer, Alex Beutel, Jilin Chen, and Kellie Webster.  Flexible text generation for counterfactual fairness probing.  arXiv preprint arXiv:2206.13757, 2022.
  * [82] Daniel James Fuchs.  The dangers of human-like bias in machine-learning algorithms.  Missouri S&T’s Peer to Peer, 2(1):1, 2018.
  * [83] Sainyam Galhotra, Karthikeyan Shanmugam, Prasanna Sattigeri, Kush R Varshney, Rachel Bellamy, Kuntal Dey, et al.  Causal feature selection for algorithmic fairness.  2022\.
  * [84] João Gama, Indrė Žliobaitė, Albert Bifet, Mykola Pechenizkiy, and Abdelhamid Bouchachia.  A survey on concept drift adaptation.  ACM computing surveys (CSUR), 46(4):1–37, 2014.
  * [85] Ruoyuan Gao and Chirag a Shah.  Toward creating a fairer ranking in search engine results.  In Information Processing & Management, volume 57, 2020.
  * [86] Satvik Garg, Pradyumn Pundir, Geetanjali Rathee, PK Gupta, Somya Garg, and Saransh Ahlawat.  On continuous integration/continuous delivery for automated deployment of machine learning models using mlops.  In 2021 IEEE fourth international conference on artificial intelligence and knowledge engineering (AIKE), pages 25–28. IEEE, 2021.
  * [87] Saurabh Garg, Yifan Wu, Sivaraman Balakrishnan, and Zachary C. Lipton.  A unified view of label shift estimatation.  Advances in Neural Information Processing Systems, 33:3290–3300, 2020.
  * [88] Marzyeh Ghassemi and Shakir Mohamed.  Machine learning and health need better values.  npj Digital Medicine, 5(1):1–4, 2022.
  * [89] Milena A. Gianfrancesco, Suzanne Tamang, Jinoos Yazdany, and Gabriela Schmajuk.  Potential biases in machine learning algorithms using electronic health record data.  JAMA internal medicine, 178(11):1544–1547, 2018.
  * [90] Tony Ginart, Martin Jinye Zhang, and James Zou.  Mldemon:deployment monitoring for machine learning systems.  In Gustau Camps-Valls, Francisco J. R. Ruiz, and Isabel Valera, editors, Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pages 3962–3997. PMLR, 28–30 Mar 2022.
  * [91] Elliot Graham, Samer Halabi, and Arie Nadler.  Ingroup bias in healthcare contexts: Israeli-jewish perceptions of arab and jewish doctors.  Frontiers in psychology, 12, 2021.
  * [92] Alex Graves, Marc G. Bellemare, Jacob Menick, Rémi Munos, and Koray Kavukcuoglu.  Automated curriculum learning for neural networks.  In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1311–1320. PMLR, 06–11 Aug 2017\.
  * [93] Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola.  A kernel two-sample test.  The Journal of Machine Learning Research, 13(1):723–773, 2012.
  * [94] Nina Grgic-Hlaca, Muhammad Bilal Zafar, Krishna P Gummadi, and Adrian Weller.  The case for process fairness in learning: Feature selection for fair decision making.  In NIPS symposium on machine learning and the law, volume 1, page 2. Barcelona, Spain, 2016.
  * [95] Hao Guan and Mingxia Liu.  Domain adaptation for medical image analysis: a survey.  IEEE Transactions on Biomedical Engineering, 69(3):1173–1185, 2021\.
  * [96] Chuan Guo, Tom Goldstein, Awni Hannun, and Laurens Van Der Maaten.  Certified data removal from machine learning models.  arXiv preprint arXiv:1911.03030, 2019.
  * [97] Lin Lawrence Guo, Stephen R. Pfohl, Jason Fries, Alistair E. W. Johnson, Jose Posada, Catherine Aftandilian, Nigam Shah, and Lillian Sung.  Evaluation of domain generalization and adaptation on improving model robustness to temporal dataset shift in clinical medicine.  Sci Rep, page 2726, 2022.
  * [98] Xiaoyuan Guo, Judy Wawira Gichoya, Hari Trivedi, Saptarshi Purkayastha, and Imon Banerjee.  Medshift: identifying shift data for medical dataset curation.  arXiv preprint arXiv:2112.13885, 2021.
  * [99] Kishor Datta Gupta and Dipankar Dasgupta.  Who is responsible for adversarial defense?  arXiv preprint arXiv:2106.14152, 2021.
  * [100] Raia Hadsell, Dushyant Rao, Andrei A. Rusu, and Razvan Pascanu.  Embracing change: Continual learning in deep neural networks.  Trends in cognitive sciences, 24(12):1028–1040, 2020.
  * [101] Abid Haleem, Mohd Javaid, Ravi Pratap Singh, Rajiv Suman, and Shanay Rab.  Blockchain technology applications in healthcare: An overview.  International Journal of Intelligent Networks, 2:130–139, 2021\.
  * [102] Frederik Harder, Matthias Bauer, and Mijung Park.  Interpretable and differentially private predictions.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 4083–4090, 2020.