# Quality Assurance Challenges for Machine Learning Software Applications
During Software Development Life Cycle Phases

###### Abstract

In the past decades, the revolutionary advances of Machine Learning (ML) have
shown a rapid adoption of ML models into software systems of diverse types.
Such Machine Learning Software Applications (MLSAs) are gaining importance in
our daily lives. As such, the Quality Assurance (QA) of MLSAs is of paramount
importance. Several research efforts are dedicated to determining the specific
challenges we can face while adopting ML models into software systems.
However, we are aware of no research that offered a holistic view of the
distribution of those ML quality assurance challenges across the various
phases of software development life cycles (SDLC). This paper conducts an in-
depth literature review of a large volume of research papers that focused on
the quality assurance of ML models. We developed a taxonomy of MLSA quality
assurance issues by mapping the various ML adoption challenges across
different phases of SDLC. We provide recommendations and research
opportunities to improve SDLC practices based on the taxonomy. This mapping
can help prioritize quality assurance efforts of MLSAs where the adoption of
ML models can be considered crucial.

Index Terms—  Machine Learning Software Application (MLSA), SDLC, ML Pipeline,
Challenge, Quality Assurance

##  1 Introduction

Each disrupting change in software development required the software industry
to evolve and adapt a novel strategy. The latest trend is the widespread
interest in the adoption of ML (Machine Learning) capabilities into large
scale Machine Learning software applications (MLSAs) [1]. The market of MLSA
and Artificial Intelligence (AI) is expected to grow at a compound annual
growth rate of 29.7% worldwide. However, quality assurance challenges for
MLSAs are hard to address, leading to deadly errors. For example, recently, a
Uber self-driving car ran into pedestrians because the sensor could not detect
those. There is now a growing concern on the quality assurance of safety-
critical ML applications like self-driving cars [2], healthcare, financial
institutions, etc.

Software quality assurance is a systematic approach to detect defects and it
can improve the reliability and adaptation of MLSA [3]. However, ML models’
stochastic nature and their dependency on data introduce diverse novel
challenges like testing non-determinism in the MLSA outputs [3]. While deep
learning (DL) models have revolutionalized ML models’ performance across
domains, DL models often behave like a black-box, which makes it difficult to
be evaluated and explained in safety-critical applications [4].

In traditional software development, first we gather requirements. We then
design, develop, test, deploy and maintain the application. For ML systems, we
still need to scope out the goal of the application, but instead of designing
the the algorithm we let the ML model learn the desired logic from data [1].
Such observations lead to the question of whether and how ML models can be
adopted without disrupting the software development life cycle (SDLC) of the
MLSAs. Ideally, ML workflow/pipeline and SDLC phases should go hand in hand to
ensure proper quality assurance. However, as we noted above, such expectations
can be unrealistic due to the inherent differences in how ML models are
designed and how traditional software applications are developed. We thus need
a holistic view of the diverse ML adoption challenges we encounter during the
SDLC phases of an MLSA development. Such insights can be used to improve the
ML adoption pipeline and the SDLC phases of MLSA development.

In recent years, significant research efforts are devoted to understanding the
diverse quality assurance challenges while adopting ML models into software
systems [5, 6, 7]. However, we are not aware of any research that specifically
focused on mapping the QA challenges across SDLC phases. In this paper, by
conducting an in-depth literature review of the challenges of ML adoption and
by determining how such challenges may permeate, we have produced a taxonomy
of quality assurance challenges that practitioners face during the adoption of
ML models into the diverse SDLC phases. We present the taxonomy and describe
it with examples. We conclude by offering recommendations for research
opportunities that lie ahead to ensure the quality assurance of MLSAs.

##  2 Background and Related Work

We introduce two concepts (SDLC and ML pipeline stages) on which this paper
builds on. We then briefly discuss related work. More related work are
discussed in Section 3.

$\bullet$ Background. SDLC methodology is embedded in traditional application
development that consists of Requirement analysis, Designing & planning,
Implementation, Quality assurance, Deployment, Maintenance phases. Our
interest is to study the QA challenges faced during different SDLC phases of
MLSA. Machine learning workflow is a little bit different from traditional
application development. It begins with system requirement analysis to data
collection and processing, feature engineering and training, evaluation of
model performance and then model deployment and monitoring. Figure 1 shows a
typical ML model development stages/workflows [1]. It is an iterative process
based on model’s performance.

$\bullet$ Related Work. There are quite a few numbers of research that focus
on the current challenges of machine learning in software engineering[5, 7,
6], empirical studies on best practices of integrating AI capabilities[1],
developers survey on challenges faced during different SDLC[8]. There are also
quite a few research on the quality assurance of ML models like reliability,
transparency, trustworthiness, etc. [9, 10, 11, 6, 10, 12]. Zhang et al.[3]
provides a comprehensive survey on ML testing considering 138 related papers
on four aspects of MLSA testing such as what properties to test, what ML
components to test, testing workflows and test scenario of different type of
ML application. Shafiq et al.[13] conducts a bibliometric study and provides
an MLSA taxonomy of the adaptation of ML techniques across different SDLC
phase. It focuses on the research attention at different SDLC of MLSA. Lack of
modularity in ML models compared to traditional software can make those hard
to debug [3]. Test oracle problem, due to insufficient specification and data
dependency, can introduce a wide range of challenges during software testing
[14]. While the above papers offer information of ML adoption challenges, none
unlike us, has focused on providing a holistic view of the ML adoption
challenges within the standard development life cycle of MLSAs.

##  3 Quality Assurance Challenges in MLSA

Fig. 1:  A typical iterative pipeline (i.e., workflow) to prepare an machine
learning (ML) model for a system [1] Fig. 2: A taxonomy of MLSA quality
assurance challenges across different phases of SDLC

In Figure 2, we present a taxonomy of quality assurance challenges for MLSA
development. The challenges are derived from an in-depth literature review of
quality assurance challenges for ML adoption into software systems. We group
the challenges under six SDLC phases:  (1) Requirement analysis,  (2) design
and planning,  (3) implementation,  (4) testing,  (5) deployment, and  (6)
maintenance.  For each challenge, we also show the specific stages in ML
pipeline that are impacted/consulted to address the challenge. The ML pipeline
stages are derived from Amershi et al. [1].

In total, we found 31 challenges that we group into three higher categories:
(1) Data. It contains QA challenges related to data collection, cleaning,
labelling, management,  (2) Practice. It contains QA challenges that is faced
in practice by SE teams.  (3) Standard. This category of QA challenges
represent the challenges due to the lack of Standard specification or
guidelines.  A challenge can be observed in multiple SDLC phases. For example,
lack of ML standards and metrics can be a problem during MLSA requirement
collection as well as during MLSA implementation phase. However, depending on
the types of SDLC, the same challenge can be observed in varying formats. For
example, while lack of standards in ML model evaluation may result in
ambiguities during MLSA requirement collection, it can also affect the
development team during their analysis of whether an ML model is good enough
with regards to the stated requirements and designs. We now discuss the
challenges per SDLC phases below.

$\bullet$ SDLC Phase 1. MLSA requirement Collection. Two challenges arise due
to the lack of clarity in Standard specification:  (1) Ambiguous requirement
(ML stage = R, i.e., Requirements): MLSAs can have ambiguous specification, as
it is difficult to define the expected behaviour, which can change due to data
[8, 15].  (2) Lack of standards in reporting & metrics (R): MLSAs do not have
a standard reporting specification like other industries (transportation,
aviation).

$\bullet$ SDLC Phase 2. Design and planning. There are four challenges under
three categories: Data, Practice, and Standard. Data category has one
challenge.  (1) Privacy concerns (R): Many companies can not use raw user-data
because of terms and service agreement with the users, organizational legal
and ethical constrains. Inability to understand the internal representation of
the model might cause privacy breach [16]. Federated learning [17] has the
potential to deliver these features properly [18].  Practice category has two
challenges.  (1) Bias and Lack of Fairness (R): An MLSA can be biased if the
training is biased [19].  (2) Communication gap between ML and non-ML
teams/practitioners (ML Stage = A, i.e., all): Qualitative and quantitative
studies suggest that it is important to adopt best of SE and ML research world
such as sharing data [8], complexity reduction, deletion of features,
improving reproducibility [20].  There is one challenge faced due to lack of
Standard specifications: Inadequate standard regulation/compliance (R): During
training we need to consider user data privacy and confidentiality.
Organizations need to comply with user data protection acts HIPPA[21], GDPR
[22].

$\bullet$ SDLC Phase 3. MLSA Implementation. There are eight QA challenges.
The Data category has one challenge: Inadequate/Inefficient data management
and versioning support (for ML stage = DP, i.e., Data Processing). There is a
lack of tools to manage different data attributes (e.g., data freshness). The
Practice category has six challenges:  (1) Lack of modularity &
customizability (ML stage = F, i.e., Feature Engineering): Due to lack of
customizability, it requires a significant effort to reuse a model for a
different task or handle different input format.  (2) Longer/impractical
evaluation process & time of ML models (ML Stage = T, i.e., Testing): The
performance of a model can not be evaluated until training on the whole
dataset is finished, which is computationally expensive and time-consuming.
(3) Non-deterministic outcome due to data dependency (A): ML models are
inherently dependent on training data. However, real-world data distribution
may be different, and the model might misbehave.  (4) Non-deterministic
debugging due to data dependency (A): The debugging strategy of traditional
systems is not quite applicable for many ML-systems: bugs might be in the code
as well as in the data.  (5) Lack of well-defined workflow for integration due
to rapid ML evolution (A): There is a lack of ML-specific process management
tools, or specification to estimate time and resources.  (6) Presence of
additional technical debt due to ML adoption into software (A): A mature ML
systems can be of 95% of glue code that mainly connects different library and
packages [20, 23].  There is 1 QA challenge under Standard category: Lack of
standards in reporting & metrics (E). There is a lack of standards to compare
the performance of different models, guideline to identify & report fault
[24].

$\bullet$ SDLC Phase 4. MLSA Testing. We find eight challenges. Data category
has one challenge: Lack of adversarial robustness due to data dependency (ML
Stage = D, i.e., Deployment). Adversarial example refers to a small malicious
modification of the input that causes the model to erroneous output [25]. The
Practice category has six challenges:  (1) Non-deterministic integration of ML
models (ML stage = M, i.e., monitoring): All the external systems that provide
input or consumes the output of the ML model should be explicitly monitored
because these dependencies can affect the ability to release updates[23, 20].
(2) Increased testing scope due to the needs for both data and code testing
(A): QA for ML systems are difficult as they are inherently non-deterministic
and they self-learn [1].  (3) Predominance of black-box-type testing for
already trained ML models (A): Various phases of a typical MLSA contain highly
coupled components. Faults occurred in one component of this pipeline may
propagate to other phases and thus hard to detect and fix [26, 27].  (4) Lack
of interpretability and explainability of model results (A)is prevalent when
features are too abstract for human/tester to understand [28, 29].  (5) Bias
and Lack of Fairness (A): Model’s fairness and biases should be tested in both
algorithms and data.  (6) Difficulty in reproducibility due to the stochastic
nature of parameter initialization (A): Many ML-libraries use random
initialization of initial states which makes it hard to reproduce issues.
Standard category has one challenge: Lack of standard testing specification
(D). A model’s performance is tested as a whole rather than for a specific
input, making it challenging to design a test oracle [14].

$\bullet$ SDLC Phase 5. MLSA Deployment There are four QA challenges in this
phase and 3 under Practice category:  (1) Insufficient MLOps support into the
traditional DevOps process (D): There is a lack of end to end deployment
pipeline support such as advanced logging, automated rollback, security [30].
(2) Lack of scalability in real-world scenarios (D): Many ML models perform
well in a research setting but are not computationally scalable for large
scale deployment.  (3) Elevated complexity in ML configuration management (A):
Rapid experimentation requires to track code, data, parameters,
hyperparameters to compare the trade-off of different algorithms, model
architecture.  There is one QA challenges under Standard category: Inadequate
standard regulation/compliance (D). In order to ensure public trust the model
needs to be continually checked to determine whether it is following the user
data protection acts (HIPPA[21], GDPR[22]).

$\bullet$ SDLC Phase 6. MLSA Maintenance. The maintenance of MLSA over time is
expensive and challenging[20]. There are 5 QA challenges. Data category has
two challenges:  (1) Gradual drift and decay of ML models (M): ML systems have
a predictable performance degradation over time if the models are not updated
with new data.  (2) Lack of adversarial robustness due to data dependency (M):
The deployed ML-application should perform reasonably in the real world and
prevent adversarial attacks[31, 32].  The Practice category has three
challenges:  (1) Non-deterministic debugging due to hidden feedback loop (M):
An ML application learns from data and hence some data-driven feedback could
be unseen or remain hidden [20].  (2) Reliability monitoring due to the
model’s lack of generalizability (M): A deployed ML model requires constant
monitoring to assess the output against new input and new corner cases.  (3)
Presence of additional technical debt due to ML adoption into software (A):
Many undeclared external systems can consume an ML model’s output and cause
“visibility debt” [33].

##  4 Recommendations and Conclusions

We conclude by summarizing the research opportunities to address the MLSA
quality assurance challenges we categorized across the six SDLC in Figure 2.
We find that MLSAs and traditional software applications share some common QA
challenges like insufficient specifications and privacy. However, as the
performance and robustness of MLSAs depend on the quality of the training
dataset, the scope and the severity of these QA challenges differ like non-
deterministic outcome, implicit biases in data, lack of interpretability,
gradual performance decay, hidden feedback loops pose some of the unique
challenges for MLSAs, and so on. To handle ambiguity and lack of standards in
MLSA requirement analysis, research can focus on developing necessary
specifications, tools, and metrics for MLSA requirement analysis and third-
party verification [34]. For design and planning challenges of MLSA, research
can improve data privacy [18, 17, 35] and develop specifications and tools
[36] to improve model bias and fairness [19]. For implementation-specific
challenges, research can focus on developing debugging tools [37], transfer
learning [38], model modularization [39], and interpretable AI [28, 29]. To
handle the challenges of testing of MLSA, more research is needed on bug
analysis[3], regression testing, reinforced learning [3], novel adversarial
attacks [40, 41, 42, 43] and defence techniques [11, 44]. To address MLSA
deployment and maintenance challenges, future research can focus on developing
configuration management and reliability monitoring specification and tools
[31, 32, 9].

## References

  * [1] Amershi et al.,  “Software engineering for machine learning: A case study,”  in proc ICSE-SEIP. IEEE, 2019, pp. 291–300.
  * [2] Chen et al.,  “Deepdriving: Learning affordance for direct perception in autonomous driving,”  in proc ICCV, 2015, pp. 2722–2730.
  * [3] Zhang et al.,  “Machine learning testing: Survey, landscapes and horizons,”  In TSE, 2020.
  * [4] Willers et al.,  “Safety concerns and mitigation approaches regarding the use of deep learning in safety-critical perception tasks,”  in International Conference on Computer Safety, Reliability, and Security. Springer, 2020, pp. 336–350.
  * [5] Devanbu et al.,  “Deep learning & software engineering: State of research and future directions,”  arXiv preprint arXiv:2009.08525, 2020.
  * [6] Santhanam et al.r,  “Engineering reliable deep learning systems,”  arXiv preprint arXiv:1910.12582, 2019.
  * [7] Feldt et al.,  “Ways of applying artificial intelligence in software engineering,”  in proc RAISE. IEEE, 2018, pp. 35–41.
  * [8] Wan et al.,  “How does machine learning change software development practices?,”  In TSE, 2019.
  * [9] Saria et al.,  “Tutorial: safe and reliable machine learning,”  arXiv preprint arXiv:1904.07204, 2019.
  * [10] Roscher et al.,  “Explainable machine learning for scientific insights and discoveries,”  IEEE Access, vol. 8, pp. 42200–42216, 2020.
  * [11] Goodfellow et al.,  “Explaining and harnessing adversarial examples,”  arXiv preprint arXiv:1412.6572, 2014.
  * [12] Qiu et al.,  “Review of artificial intelligence adversarial attack and defense technologies,”  Applied Sciences, vol. 9, no. 5, pp. 909, 2019.
  * [13] Shafiq et al.,  “Machine learning for software engineering: A systematic mapping,”  arXiv preprint arXiv:2005.13299, 2020.
  * [14] Barr et al.,  “The oracle problem in software testing: A survey,”  IEEE transactions on software engineering, vol. 41, no. 5, pp. 507–525, 2014.
  * [15] Finkelstein et al.,  ““fairness analysis” in requirements assignments,”  in 2008 16th IEEE International Requirements Engineering Conference. IEEE, 2008, pp. 115–124.
  * [16] Davide Castelvecchi,  “Can we open the black box of ai?,”  Nature News, vol. 538, no. 7623, pp. 20, 2016.
  * [17] McMahan et al.,  “Communication-efficient learning of deep networks from decentralized data,”  in AISTATS. PMLR, 2017, pp. 1273–1282.
  * [18] Li et al.,  “Federated learning: Challenges, methods, and future directions,”  IEEE Signal Processing Magazine, vol. 37, no. 3, pp. 50–60, 2020\.
  * [19] Mehrabi et al.,  “A survey on bias and fairness in machine learning,”  arXiv preprint arXiv:1908.09635, 2019.
  * [20] Sculley et al.,  “Hidden technical debt in machine learning systems,”  in Advances in neural information processing systems, 2015, pp. 2503–2511.
  * [21] Annas et al.,  “Hipaa regulations-a new era of medical-record privacy?,”  New England Journal of Medicine, vol. 348, no. 15, pp. 1486–1490, 2003.
  * [22] Voigt et al.,  “The eu general data protection regulation (gdpr),”  A Practical Guide, 1st Ed., Cham: Springer International Publishing, vol. 10, pp. 3152676, 2017.
  * [23] Arpteg et al.,  “Software engineering challenges of deep learning,”  in proc of Software Engineering and Advanced Applications (SEAA). IEEE, 2018, pp. 50–59.
  * [24] Devanbu et al.,  “Deep learning & software engineering: State of research and future directions,”  arXiv preprint arXiv:2009.08525, 2020.
  * [25] Madry et al.,  “Towards deep learning models resistant to adversarial attacks,”  arXiv preprint arXiv:1706.06083, 2017.
  * [26] Sculley et al.,  “Machine learning: The high interest credit card of technical debt,”  2014\.
  * [27] Felderer et al.,  “Quality assurance for ai-based systems: Overview and challenges,”  preprint arXiv:2102.05351, 2021.
  * [28] Zachary C Lipton,  “The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery.,”  Queue, vol. 16, no. 3, pp. 31–57, 2018.
  * [29] Ribeiro et al.,  “Why should i trust you? explaining the predictions of any classifier,”  in proc ACM KDD, 2016, pp. 1135–1144.
  * [30] Paleyes et al.,  “Challenges in deploying machine learning: a survey of case studies,”  preprint arXiv:2011.09926, 2020.
  * [31] Schulam et al.,  “Can you trust this prediction? auditing pointwise reliability after learning,”  in proc of Artificial Intelligence and Statistics. PMLR, 2019, pp. 1022–1031.
  * [32] Jiang et al.,  “To trust or not to trust a classifier.,”  in proc NeurIPS, 2018, pp. 5546–5557.
  * [33] Morgenthaler et al.,  “Searching for build debt: Experiences managing technical debt at google,”  in Workshop on Managing Technical Debt (MTD). IEEE, 2012, pp. 1–6.
  * [34] Vogelsang et al.,  “Requirements engineering for machine learning: Perspectives from data scientists,”  in International Requirements Engineering Conference Workshops (REW). IEEE, 2019, pp. 245–251.
  * [35] Bonawitz et al.,  “Towards federated learning at scale: System design,”  arXiv preprint arXiv:1902.01046, 2019.
  * [36] Galhotra et al.,  “Fairness testing: testing software for discrimination,”  in proc ESEC/FSE, 2017, pp. 498–510.
  * [37] Ma et al.,  “Mode: automated neural network model debugging via state differential analysis and input selection,”  in proc ESEC/FSE, 2018, pp. 175–186.
  * [38] Pan et al.,  “A survey on transfer learning,”  In TKDE, vol. 22, no. 10, pp. 1345–1359, 2009.
  * [39] Pan et al.,  “On decomposing a deep neural network into modules,”  in proc ESEC/FSE, 2020, pp. 889–900.
  * [40] Dong et al.,  “Boosting adversarial attacks with momentum,”  in proc IEEE conference on computer vision and pattern recognition, 2018, pp. 9185–9193.
  * [41] Carlini et al.,  “Towards evaluating the robustness of neural networks,”  in symposium on security and privacy (sp). IEEE, 2017, pp. 39–57.
  * [42] Szegedy et al.,  “Intriguing properties of neural networks,”  arXiv preprint arXiv:1312.6199, 2013.
  * [43] Mei et al.,  “Using machine teaching to identify optimal training-set attacks on machine learners,”  in Proceedings of the AAAI Conference on Artificial Intelligence, 2015, vol. 29.
  * [44] Zhang et al.,  “Defense against adversarial attacks using feature scattering-based adversarial training,”  preprint arXiv:1907.10764, 2019.