Classification, Challenges, and Automated Approaches for Non-Functional Requirements in ML-Enabled Systems]Classification, Challenges, and Automated Approaches to Handle Non-Functional Requirements in ML-Enabled Systems: A Systematic Literature Review

Vincenzo De Martino

Software Engineering (SeSa) Lab, University of Salerno

Fabio Palomba

Software Engineering (SeSa) Lab, University of Salerno

Machine learning (ML) is nowadays so pervasive and diffused that virtually no application can avoid its use. Nonetheless, its enormous potential is constantly threatened by non-functional requirements, such as sustainability.
In particular, we noticed the lack of a comprehensive synthesis of the research efforts done so far and how these may drive further research.
In this paper, we propose a systematic literature review targeting three key aspects such as (1) the classification of the non-functional requirements investigated so far, (2) the challenges to face when dealing with them, and (3) the automated approaches proposed in literature to support practitioners when optimizing them in practice.
Through the combination of well-established guidelines for conducting systematic literature reviews and additional search criteria, we survey a total amount of 69 research articles.
Our findings report that current research identified 30 different non-functional requirements, which can be grouped into six main classes.
We also deliver a catalog of over 23 software engineering challenges that further research should consider, besides an overview of the automated approaches researchers proposed to support practitioners when optimizing non-functional requirements of machine learning-enabled systems.
We conclude our work by distilling implications and a future outlook on the topic.

<concept_desc>Computing methodologies Machine learning</concept_desc>
<concept_desc>Software and its engineering Extra-functional properties</concept_desc>

[500]Computing methodologies Machine learning
[500]Software and its engineering Extra-functional properties

§ INTRODUCTION

Machine learning (ML) is now, more than ever, being used in theory, experiment, and simulation <cit.>. On the one hand, companies and individuals increasingly rely on the outcome of machine learning models to make informed decisions <cit.> or automate tasks that would take substantial human workload <cit.>. On the other hand, machine learning-intensive systems, i.e., systems that embed machine learning solutions, have been recently deployed in multiple domains, with some recent applications showing highly efficient and accurate performance <cit.>. As such, the pervasiveness of machine learning-intensive systems is expected to increase further in the coming years in multiple domains <cit.>.

Nonetheless, such a pervasiveness is constantly threatened by multiple concerns, which are often not related to the specific features made available to users, but to non-functional attributes <cit.>. In particular, a non-functional attribute is defined as a condition that specifies a criterion that may be used to judge the operation of a system rather than specific behaviors <cit.>. In the context of machine learning-enabled systems, non-function attributes may affect the overall level of reliability, trustworthiness, and sustainability of these systems  <cit.>. It is therefore not surprising that the software engineering research community—and specifically the software engineering for artificial intelligence (SE4AI) research branch—has been investing notable efforts in understanding non-functional requirements of machine learning-intensive systems, other than proposing methods and instruments to support practitioners when dealing with them <cit.>. This effort is also stimulated by government and funding agencies, which are more and more willing to invest in the matter, e.g., the European Union has recently approved the so-called Artificial Intelligence Act,[The European Union Artificial Intelligence Act: <https://artificialintelligenceact.eu>.] which aims at promoting research on themes connected to the improvement of non-functional attributes of artificial intelligence-based software systems.

Recent advances in the field of SE4AI contributed to the development of a consistent body of knowledge with respect to the management of multiple non-functional requirements, including fairness <cit.>, security <cit.>, privacy <cit.>, and more <cit.>. While recognizing the relevant advances made over the last years, our research identifies a notable key limitation.

Despite the current, extensive body of knowledge produced by the SE4AI research community with respect to the management of non-functional requirements of ML-intensive systems, there is still a lack of a comprehensive, systematic synthesis of the current knowledge on the non-functional attributes affecting machine learning-enabled systems, the challenges faced when dealing with them, and the automated approaches proposed to support practitioners when optimizing them throughout the software development lifecycle.

An improved understanding of these aspects may have critical implications for researchers and practitioners. First, researchers might learn more about the current state of the art, possibly identifying neglected research angles that would be worth further investigating. At the same time, practitioners may have a comprehensive overview of the instruments that researchers have been providing to support the analysis and optimization of non-functional requirements, possibly accelerating the technological transfer of academic prototypes to industry.

In this paper, we conduct a systematic literature review (SLR) on non-functional requirements of machine learning-intensive software systems. Our work follows well-established guidelines <cit.> and additional search criteria based on seed set identification <cit.> to comprehensively synthesize existing research.
From an initial set composed of over 2,500 hits, and after applying multiple snowballing rounds and additional data collection procedures, we ended up analyzing more than 81,000 research results. Through the application of exclusion/inclusion criteria and a rigorous quality assessment, we finally selected 69 papers.

In addition, we provide a novel catalog composed of more than 23 software engineering challenges to deal with them, other than an overview of the automated approaches proposed by researchers to support practitioners when optimizing non-functional requirements throughout the software lifecycle. We conclude the paper by elaborating on the implications of our results, along with the actionable items that readers of our work may (re-)use to analyze further the problem of non-functional requirements in machine learning-intensive software systems.

Structure of the paper. Section <ref> analyzes the related work. Section <ref> describes the research goals and the methods applied to address them, while Section <ref> discusses the results achieved. The implications and actionable insights are elaborated in Section <ref>, with the limitations reported in Section <ref>. Finally, Section <ref> concludes the paper and outlines our future research agenda.

Comparison with the Closest Related Work.

Related Work     Main Focus     Commonalities and differences

Martinez-Fernandez et al. <cit.>
A systematic mapping review targeting all previous research on software engineering for artificial intelligence. The authors surveyed 248 studies, classifying the available research according to the SWEBOK areas <cit.>.

* Similar research approach to the search, yet with some differences in terms of seed set identification;

* A broader analysis of current research, with no specific focus on non-functional requirements;

* While the work includes part of the research papers we analyze, it does not provide insights into the challenges, practices, and methods to deal with non-functional requirements of machine learning-intensive systems.

Habibullah et al. <cit.>     A non-systematic analysis of non-functional requirements of machine learning-intensive systems. The authors focused on (i) the identification of clusters of non-functional requirements, (ii) the estimation of the amount of relevant studies for a subset of non-functional requirements, and (iii) the definition of the scope of non-functional requirements.

* We approach the literature search with a systematic approach, hence making a more comprehensive analysis;

* We did not limit ourselves to the estimation of the current interest in the matter but provided a systematic investigation;

* We let additional factors arise, including challenges, practices, and methods to deal with non-functional requirements of machine learning-intensive systems.

Horkoff <cit.>
An experience report on the challenges that the Requirements Engineering (RE) research community is called to face when addressing non-functional attributes of machine learning-intensive systems.

* We approach the literature search with a systematic approach, hence making a more comprehensive analysis;

* We let additional factors arise, including a comprehensive set of non-functional requirements for machine learning-intensive systems, other than practices and methods to deal with them;

* We let the challenges to deal with non-functional requirements emerge from the scientific literature on the matter, hence approaching the research differently.

Habibullah and Horkoff <cit.>
An industrial, interview-based study aiming at assessing (i) the identification and measurement of non-functional requirements, (ii) the importance of non-functional requirements in the industry, and (iii) the challenges associated with the identified non-functional requirements.

* We let our findings emerge from the scientific literature on the matter, hence approaching the research differently;

* We let additional factors arise, including a comprehensive set of non-functional requirements for machine learning-intensive systems, other than practices and methods to deal with them;

* We did not limit ourselves to the observations made by the industry practitioners but provided a systematic investigation of the current relevance of non-functional requirements.

Ahmad et al. <cit.>     A systematic mapping study on the requirements specification and modeling approaches and tools.

* The work targets requirements engineering from the perspective of specification and modeling, while ours focuses on non-functional attributes;

* The research method is similar, yet we perform a systematic literature review rather than a mapping study, hence being more restrictive in terms of quality assessment of the primary studies;

§ RELATED WORK AND MOTIVATION

To the best of our knowledge, no systematic literature review has been conducted with the aim of classifying non-functional requirements of machine learning-enabled systems and summarizing the challenges and automated approaches to deal with them. At the same time, it is important to point out that some secondary studies recently attempted to (1) synthesize the research on software engineering for artificial intelligence <cit.>, (2) explore, in a preliminary fashion, the relevance and research interest around non-functional requirements of machine learning-intensive systems <cit.>, and (3) summarize some of the key academic and industrial challenges when dealing with non-functional requirements in industry <cit.>. These works are clearly the most closely related to ours. This section describes the major differences and limitations of current literature that motivated our research. A summary is provided in Table <ref>.

First and foremost, Martinez-Fernandez et al. <cit.> conducted a systematic mapping study of the research on software engineering for artificial intelligence. The main goal of the work was to provide a comprehensive schema representing the elements composing the field of SE4AI, from requirements engineering to verification and validation. In other terms, the systematic mapping study had a pretty broad objective and aimed at covering all the research on the matter. As such, there are multiple differences for our study. While our scope is limited to non-functional requirements, we aim to address the matter in a more detailed fashion by letting emerge a complete set of non-functional requirements discussed in the literature, other than the challenges and automated approaches proposed to deal with them. Secondly, ours is a systematic literature review rather than a mapping study: as such, there are intrinsic, methodical differences in the search process conducted and in the criteria used to select the relevant pieces of research. Third, we additionally tuned the search process to produce an extensive set of seed papers, as further discussed in Section <ref>—hence attempting to strengthen the completeness of the search process.

Habibullah et al. <cit.> recently investigated the topic of non-functional requirements of machine learning-enabled systems under three perspectives such as (1) the clustering of non-functional attributes based on shared characteristics; (2) the estimation of the number of relevant studies that investigated aspects connected to non-functional attributes; and (3) the definition of the scope of non-functional requirements. Habibullah et al. <cit.> share the same overall objective of our paper, i.e., an improved understanding of non-functional requirements of machine learning-intensive systems. Nonetheless, we aimed to conduct a comprehensive, systematic literature review. At the same time, we focused on a larger set of research angles, including an investigation of the challenges and approached proposed to deal with non-functional requirements.

Horkoff <cit.> discussed the challenges that the requirements engineering research community would be called to face to address themes connected to non-functional requirements of machine learning-intensive systems. The author identified several challenges based on the extensive experience accumulated in the industry over the years. With respect to this paper, our systematic literature review aims to collect comprehensive pieces of information coming from the scientific community to provide insights into how researchers have addressed the problem so far. At the same time, it is worth saying that we could corroborate the relevance of some of the challenges identified by Horkoff <cit.> through the evidence coming from our work.

Habibullah and Horkoff <cit.> conducted an interview-based study with ten machine learning engineers to elicit the practices used to face non-functional requirements. In particular, the authors were interested in collecting information about the identification and measurement mechanisms put in place, the importance of various non-functional requirements from their perspective, and the challenges associated with the identified non-functional requirements. This work can be therefore seen as complementary to our systematic literature review. On the one hand, we enlarge the knowledge of non-functional requirements in machine learning-enabled systems by synthesizing the current literature from various perspectives. On the other hand, we could corroborate and further extend the findings on the challenges identified through a different research method.

Finally, Ahmad et al. <cit.> recently performed a systematic mapping of the research on requirements engineering for artificial intelligence. The authors surveyed the approaches to specify requirements and the frameworks/tools/methods to model them. Our work is clearly complementary, as we focus on non-functional attributes, attempting to classify them and synthesize the current knowledge on the challenges and approaches to deal with them.

While the papers discussed above are closely connected to the work proposed herein, it is also worth mentioning the existence of an ever-increasing number of secondary studies targeting multiple aspects of software engineering for artificial intelligence research.

A consistent amount of systematic literature reviews focused on the synergies between the artificial intelligence and software engineering research communities <cit.>, other than on the software engineering challenges and solutions for developing artificial intelligence systems <cit.>. With respect to them, our work focuses on non-functional requirements, hence providing finer-grained pieces of information and insights on the next steps that researchers should consider to better support practitioners.

Other researchers targeted the quality assurance problem, which is deemed one of the most relevant and complex for the SE4AI research community. In particular, we identified systematic literature review and empirical investigations into the field of software quality <cit.>, design patterns <cit.>,
software architecture <cit.>, and testing <cit.>. Finally, recent systematic literature reviews have been conducted with the aim of understanding the deployment strategies for artificial intelligence systems <cit.>, and model-based development approaches <cit.>.

The substantial efforts put by researchers in conducting systematic investigations on software engineering for artificial intelligence further motivate our work, as they let arise the collective, emerging interest around these matters. Our systematic literature review aims, therefore, to provide additional insights to the research community by studying the non-functional requirements of machine learning-enabled systems that have been studied over the last years.

§ RESEARCH METHOD

The ultimate goal of the study was to systematically survey the current research on non-functional requirements of machine learning-enabled systems, with the purpose of providing researchers in the field of SE4AI with actionable items and insights that they can exploit to further explore the matter and improve the support provided to machine learning engineers. Through our work, we therefore provided a synthesis of the state of the art rather than a comparison between the state of the art and the state of the practice: such a comparison would be undoubtedly beneficial, yet is deemed as part of our future research agenda. Specifically, the research was conducted in terms of (1) the classification of the non-functional requirements of ML-enabled systems, (2) the challenges that non-functional requirements pose during the development of ML-enabled systems, and (3) the automated approaches that may support practitioners throughout the development lifecycle. The perspective is of fresh Ph.D. students, senior researchers, and practitioners. The former are interested in mapping the research literature that focuses on addressing non-functional requirements in machine learning-enabled systems so that they can possibly identify neglected research areas to further investigate. Senior researchers are interested in mapping the state of the art in order to identify opportunities for research proposals and industrial collaborations. Finally, practitioners are interested in understanding the challenges to face when optimizing non-functional requirements during the development of ML-enabled systems, other than the current support provided by researchers, hence possibly identifying chances for technological transfer of those approaches in the industry.

To address our goal, we designed and executed a systematic literature review (SLR), which is a process through which the existing scientific papers on a subject of interest are rigorously identified, selected, and analyzed to address one or more research questions <cit.>. To make our SLR as complete as possible, we first followed the well-established guidelines proposed by Kitchenham et al. <cit.>. However, as further elaborated in Section <ref>, we had to preserve the sustainability of the data collection and analysis procedures when designing the search string. Specifically, we defined a search string that looked for articles that explicitly referred to terms like “Non-Functional Requirements”, possibly missing the articles that referred to the specific non-functional requirements considered, e.g., “fairness”. To mitigate threats due to incompleteness of the search, we boosted the search process by means of two additional steps. On the one hand, we systematically screened the research articles published in top-tier software engineering and artificial intelligence venues in an effort to identify a set of seed papers to further process, i.e., the identification of seed papers has been shown to be an effective alternative when investigating research angles for which the standard guidelines do not enable a comprehensive search <cit.>. On the other hand, we provided additional rigor to the analysis by integrating the so-called snowballing procedure <cit.>, i.e., the iterative scanning of the incoming and outcoming references of the primary studies done to identify additional relevant sources of information. Therefore, our systematic literature review can be considered “hybrid” <cit.>, namely a systematic analysis that combines traditional search strategies with additional search steps and snowballing. In terms of reporting, we followed the ACM/SIGSOFT Empirical Standards[The ACM/SIGSOFT Empirical Standards: <https://github.com/acmsigsoft/EmpiricalStandards>.] and, in particular, the “General Standard” and “Systematic Reviews” guidelines.

§.§ Research Objectives and Questions

The Goal-Question-Metric (GQM) approach <cit.> was initially employed to purposefully measure the goals to be achieved and relate them to the data analyzed. More specifically, the overall research objectives established by this systematic literature review were the following:

Objective 1. Providing a systematic classification of the non-functional requirements of ML-enabled software defined in the literature.

Objective 2. Providing a systematic classification of the challenges to face when dealing with non-functional requirements of ML-enabled systems.

Objective 3. Synthesizing the automated approaches proposed to handle non-functional requirements in ML-enabled software throughout the development lifecycle.

Starting from these objectives, we defined more specific research questions (RQs) that could drive our search process and analysis. These are presented in Table <ref>.

Research Questions of the Systematic Literature Review.

Research Question     Description
RQ$_1$     What are the non-functional requirements of ML-enabled software considered by researchers?
RQ$_2$     What are the challenges of dealing with non-functional requirements of ML-enabled software?
RQ$_3$     What automated approaches are proposed to deal with non-functional requirements of ML-enabled systems?

More particularly, RQ$_1$ addressed the first objective of the study and aimed at systematically classifying the non-functional requirements that researchers have defined. By addressing this research angle, we aimed to contribute to previous research on the classification of non-functional requirements of ML-enabled systems <cit.>. Indeed, while previous efforts in this respect have been conducted, there is still not a systematic classification - this represents a key contribution to our work. The second objective was mapped onto RQ$_2$. This research question elicited the major challenges that non-functional requirements pose during the development of ML-enabled systems.
On the one hand, this research question allowed us to understand the rationale behind the existence of the automated approaches proposed in the literature. On the other hand, RQ$_2$ was able to enlarge the body of knowledge on the challenges to face when dealing with non-functional requirements, possibly informing researchers and practitioners on the aspects to further consider during the development of these systems. Finally, the third objective was mapped onto RQ$_3$. The final step of our systematic literature review aimed at classifying which are the existing automated approaches, in an effort to provide an overview of the current status of the research on the matter. First, the results of this research question may inform researchers about neglected research areas to investigate. Second, RQ$_3$ may be useful to practitioners to overview the currently available support, possibly finding technological transfer opportunities.

§.§ Research Method to Conduct the Systematic Literature Search

As a first step to address the goals of our study, we applied the guidelines by Kitchenham et al. <cit.> to identify primary studies targeting non-functional requirements of ML-enabled systems.

§.§.§ Research Query definition

To develop an effective search string, we first extracted relevant terms from the research questions, identifying the keywords from the RQs <cit.>. For all the terms, we then elaborated on the alternative spellings and synonyms. Afterward, we used boolean operators to assemble the search string, i.e., we used the `OR' operator for the concatenation of alternative spellings and synonyms, while the `AND' operator for the concatenation of relevant terms. More specifically, we elaborated on the following search string:

Search String.((“Machine Learning”) OR (“Artificial Intelligence”) OR (“Deep Learning”) OR (“Reinforcement Learning”) OR (“Deep Neural Network”) OR (“ml”) OR (“ai”) OR (“dl”)) AND ((“nfr”) OR (“Non-Functional Requirement*”) OR (“Non Functional Requirement*”))

There are some considerations to make on the definition of the search string. In the first place, it included terms connected to machine, deep, and reinforcement learning, but also to deep neural networks and artificial intelligence. This was done to deal with the lack of a standard terminology: it is indeed possible that researchers used more generic terms, like “Artificial Intelligence”, or more specific terms, like “Deep Neural Networks” to indicate the analysis of ML-enabled systems.

In the second place, the search string did not include terms related to any specific, known non-functional requirements, e.g., fairness, but focused on the more generic concept of non-functional requirement, including keywords like “Non-Functional Requirement”, “Non Functional Requirement”, and “NFR”. We are aware that a search string including the specific non-functional requirements would have been ensured the collection of a larger amount of relevant papers. Nonetheless, we would have faced two issues. First, we would have been bound to the inclusion of the non-functional requirements explicitly classified in previous studies while defining the search string: as a consequence, we could not have satisfactorily addressed RQ$_1$, not being able to classify additional non-functional requirements emerging from primary studies. Perhaps more importantly, the systematic literature review would have become prohibitive in terms of effort. For instance, Habibullah et al. <cit.> estimated the number of hits for a search query that includes all classified non-functional requirements in over 200,000 articles: on the one hand, such a search query would have had a low precision, identifying a high amount of irrelevant resources; on the other hand, the application of exclusion and inclusion criteria over such a large set of candidate articles would have required an excessive effort. As such, we had to identify an alternative solution,  opting for the implementation of a hybrid mechanism to search the additional relevant resources through the processes described in Sections <ref> and <ref>.

In the third place, we did not include terms connected to the development of automated approaches. We made this choice on purpose, as the first two research questions were not strictly connected to the automated approaches proposed in the literature but aimed at providing an overview of the non-functional requirements defined and the challenges they pose to the development of ML-enabled systems. On a similar note, we did not include terms like “data mining”, “software engineering”, “software analytics”, which would have restricted the scope of the search.

Data mining refers to the set of techniques that may extract data <cit.>. On the one hand, its inclusion would have possibly extended the data collection to the scientific literature not related to the development of ML-enabled systems or not in the scope of our work, e.g., a paper discussing a data mining technique for extracting non-functional requirements of traditional software. On the other hand, the keyword might have had a side effect, narrowing too much the scope of the systematic literature review: it would have indeed possibly limited the analyses to the non-functional requirements and automated approaches that can be employed at the time of data mining and engineering, hence not considering those that can be applied through a post-training operation.

The use of the term “software engineering” would have restricted the scope of the search to the papers that explicitly mentioned both “machine learning” and “software engineering”. This would have significantly reduced the effort required for the data collection and analysis. At the same time, we might have missed papers that were in scope but that did not refer to software engineering. This was a concern in our case, as some of the collected papers came from AI venues that do not use the term “software engineering” but mentioned terms like “non-functional requirement”. A similar discussion might be done for the term “software analytics”. By including it in the search string, we might have reduced the scope of the work to the papers that explicitly reported the use of software analytics instruments to handle non-functional requirements of ML-enabled systems. There were two potential issues here. On the one hand, we might have been too restrictive, not allowing the inclusion of articles published in venues different from those of software engineering. On the other hand, the articles collected from the execution of the search string should have addressed all the research questions of the work: two of them were not related to the automated approaches but rather aimed at providing a taxonomy of non-functional requirements of ML-enabled systems and of the challenges faced when addressing them: as such, we might have missed the analysis of relevant papers required to address RQ$_1$ and RQ$_2$.

More in general, we did not include keywords that might have led to the analysis of the broad field of research in artificial intelligence for software engineering (AI4SE) <cit.>. This was done on purpose, as our goal is exactly the opposite, namely that of contributing to the field of software engineering for artificial intelligence (SE4AI). More specifically, our work aims at contributing to the field of research that assesses how to build, maintain, and evolve ML-enabled systems rather than to the research field that explores the use of artificial intelligence techniques to support software engineering tasks. Our approach to the search string design is similar to the one of recent systematic literature reviews in the field of SE4AI proposed by Gezici and Tarhan <cit.> and Martinez-Fernandez et al. <cit.>: also in these cases, the authors did not include keywords that might have led to the analysis of scientific literature in the field of AI4SE.

Based on the arguments made above, we may conclude that we preferred to have a more generic search string in an effort to increase the recall of the search process—being able to collect a larger amount of papers—while preserving the sustainability of the overall data collection and analysis process. Of course, this decision impacted the precision of the search process and, subsequently, the effort required to apply the exclusion and inclusion criteria. Nonetheless, we accepted this compromise to make sure to include all relevant sources in our investigation.

§.§.§ Search Databases

As a second step of our investigation, we selected the databases to collect relevant resources. We applied the search on three well-known databases such as ACM Digital Library,[Link to ACM Digital Library: <https://dl.acm.org>.] Scopus,[Link to Scopus: <www.scopus.com>.] and IEEEXplore.[Link to IEEEXplore: <http://ieeexplore.ieee.org>.] These digital libraries are often used to carry out systematic literature and mapping studies, as they provide a comprehensive set of resources to conduct them. It is worth noticing that Scopus indexes all the papers published by relevant publishers such as Springer and Elsevier - this is the reason why we did not include the SpringerLink[Link to SpringerLink: <https://link.springer.com/>.] and ScienceDirect[Link to ScienceDirect: <https://www.sciencedirect.com>.] databases. At the same time, we still opted for the inclusion of ACM Digital Library and IEEEXplore. This was done on the basis of the fact that the proceedings of some relevant ACM and IEEE conferences might not have been indexed by Scopus and, for this reason, we might have missed relevant resources for our study. These databases only include peer-reviewed articles: as such, we did not need to double-check whether the papers extracted from them were peer-reviewed.

§.§.§ Exclusion and Inclusion criteria

To be considered useful for the goals of the systematic literature review, the papers retrieved from the search process had to be assessed against the following exclusion and inclusion criteria <cit.>.

Exclusion criteria. The resources that met the constraints reported below were excluded:

* Papers not written in English;

* Duplicated papers;

* Papers whose full-text read was not available;

* Paper not published or not peer-reviewed;

* Workshop, systematic, survey papers;

* Short papers, with a page count of less than five pages;

* Papers that did not fall into themes of computer science and computer engineering;

* Papers published before 2012;

* Conference papers which were later extended as journal submissions;

* Papers out of scope;

Inclusion criteria. We included the resources that met at least one of the following constraints:

* Papers defining types of non-functional requirements of ML-enabled systems;

* Papers presenting approaches or tools to analyze non-functional requirements of ML-enabled systems;

* Papers presenting the domain in which non-functional requirements for ML-enabled systems are applied;

* Papers presenting methodologies to optimize non-functional requirements problems in ML-enabled systems.

§.§ Research Method to Conduct the Seed Set Search

As further detailed in Section <ref>, the research method implemented through the guidelines by Kitchenham et al. <cit.> was found to be insufficient in finding relevant articles. Similarly to Habibullah et al. <cit.>, we found out that the relevant papers might have referred to specific non-functional requirements, e.g., “fairness”, rather than to the more generic term “non-functional requirement”. As such, the traditional systematic literature review approach might not have ensured completeness.

Conferences and journals considered in the scope of the seed set identification process.

Type     Name     Ranking
Journal     SE-related     IEEE Transactions on Software Engineering (TSE).      Q1
Journal     SE-related     ACM Transactions on Software Engineering and Methodology (TOSEM).      Q1
Journal     SE-related     Empirical Software Engineering (EMSE).      Q1
Journal     SE-related     Journal of Systems and Software (JSS).      Q1
Journal     SE-related     Information and Software Technology (IST).     Q1
Journal     AI-related     Journal of Engineering Applications of Artificial Intelligence (EAAI).     Q1
Journal     AI-related     IEEE Transactions on Knowledge and Data Engineering (TKDE).     Q1
Journal     AI-related     IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI).     Q1
Journal     AI-related     IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI).     Q1
Conference     SE-related     IEEE/ACM International Conference on Software Engineering (ICSE).      A*
Conference     SE-related     Joint European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE).      A*
Conference     SE-related     IEEE/ACM Automated Software Engineering Conference (ASE).      A*
Conference     SE-related     IEEE International Conference on Software Maintenance and Evolution (ICSME).     A
Conference     SE-related     IEEE International Conference on Software Testing, Verification and Validation (ICST).     A
Conference     SE-related     ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA).     A
Conference     SE-related     IEEE International Requirements Engineering Conference (RE).     A
Conference     SE-related     IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER).     A
Conference     SE-related     IEEE International Conference on Program Comprehension (ICPC).     A
Conference     SE-related     IEEE International Working Conference on Mining Software Repositories (MSR).     A
Conference     AI-related     The Association for the Advancement of Artificial Intelligence (AAAI).     A*
Conference     AI-related     International Joint Conference on Artificial Intelligence (ICAI).     A*
Conference     AI-related     Empirical Methods in Natural Language Processing (EMNLP).     A*

To address this problem, we conducted the so-called seed set identification. This is the process through which a researcher systematically screens the research papers published in top conferences and journals over a given time period in an effort to identify additional resources relevant to the research they are conducting. Wohlin et al. <cit.> recently showed that this approach could be used to effectively complement a systematic literature review in cases where the standard guidelines cannot lead to a comprehensive search. In our context, the seed search process should have considered both software engineering and artificial intelligence venues, as studies targeting the analysis of non-functional requirements and/or the design of automated approaches to deal with them might have been published in both the research areas.

In the case of software engineering venues, we identified the most relevant software engineering conferences and journals, relying on (1) the CORE Ranking system[The Computing Research and Education Association of Australasia: <https://www.core.edu.au/>] to select A* and A conferences, and (2) the Scimago Journal Ranking[The Scimago Journal Ranking: <https://www.scimagojr.com>.] to identify the software engineering journals falling into the first quartile of all journals (Q1) in the `Software' category. As such, we selected the conferences and journals marked as `SE-related' presented in Table <ref>. In the second place, we scanned all the papers published at those conferences starting from 2022 backward until 2012.
As a final step, we analyzed each conference website, and for journals, we searched for papers via the DBLP,[The DBLP: <https://dblp.org/>.] the most extensive computer science bibliography library. We applied the same exclusion and inclusion criteria defined for the search process conducted using the standard guidelines (see Section <ref>).

In the case of artificial intelligence venues, opening up to all the most relevant ones would have been prohibitively expensive in terms of effort. Using the same criteria as for the software engineering venues, we would have selected 70 Q1 journals and more than 40 A* and A conferences, which we would have scanned from 2022 backward until 2012. We estimated the potential hits to analyze in around 200,000, which was deemed overly expensive for a human assessment. Nonetheless, to make our systematic literature process as complete as possible, we narrowed down the scope of the seed search to the artificial intelligence venues having a higher likelihood to publish engineering or empirical approaches to the development of ML-enabled systems, i.e., the venues that are more likely to contain papers relevant to our research questions.

In particular, we first identified an online repository, named `AI Venues',[The `AI Venues' repository: <https://aivenues.github.io/>], which lists the whole set of artificial intelligence conferences and journals along with their ranking and H-index. We then associated to each journal the corresponding rank provided by Scimago and to each conference the rank provided by CORE. Besides discarding the venues that did not meet our selection criteria (rank=A* or A for conferences, rank=Q1 for journals), we decided to filter out the venues that revolved around too specific techniques or technologies, e.g., the IEEE Transactions on Image Processing journal, and favor instead the venues that encompassed a broader spectrum of engineering or empirical approaches applied for the development of ML-enabled systems, e.g., the IEEE Transactions on Neural Networks and Learning Systems. In this way, we could have focused on papers that had higher likelihood to be relevant for our research questions. Based on this process, we selected four Q1 journals and three A* conferences whose themes were either related to the improvement of AI approaches or the use of engineering or empirical approaches to AI. This process led to the selection of the venues marked as `AI-related' reported in Table <ref>. Also in this case, we scanned all the papers published to these venues between 2012 and 2022, applying the same selection process described in Section <ref>.

The seed search identified additional primary studies published to the selected venues. As a consequence, we did not need to double-check whether these were actually published or not.

§.§ Research Method to Conduct the Snowballing Process

The third step conducted to ensure the completeness of the search process revolved around the so-called forward and backward snowballing. This is the procedure through which a researcher systematically scans the incoming and outcoming references of the primary studies with the aim of identifying new relevant resources to address the research questions <cit.>. In our case, the primary studies identified as a consequence of the application of the exclusion and inclusion criteria on both the studies retrieved using the standard guidelines and the seed search were scanned. In particular, we applied an iterative procedure in which all incoming and outcoming references of the primary studies were first considered. Afterward, we reiterated the procedure for the newly acquired studies in an effort to identify additional resources. Overall, four rounds of backward and four rounds of forward were conducted: we stopped at four as we reached saturation, namely, we did not identify any additional papers to include. From a practical perspective, the snowballing steps were conducted through the use of Google Scholar,[Link to Google Scholar: <https://scholar.google.com/>] an academic search engine which simplifies the analysis of incoming and outcoming references of research papers. This was the only step where we actually needed to verify whether the papers cited or citing the primary resources were published. Whenever needed, i.e., whenever we identified a new paper which was not previously identified through the initial or the seed search, we searched the title of the paper on ACM Digital Library, Scopus, and IEEEXplore to verify its publishing status. In the case the paper was published, it was accepted for the subsequent steps of our research method.

§.§ Research Method to Conduct the Quality Assessment

Once we had completed the application of the three complementary search processes described in the previous sections, we conducted a quality assessment of the resources that successfully passed the inclusion criteria. This step is recommended to ensure that the primary studies would have sound and reliable pieces of information to address the research questions of the study, while the papers that are not found to be of sufficient quality are discarded from the analysis <cit.>.

The implementation of the quality assessment process started with the definition of qualitative questions aiming at operationalizing the main pieces of information that a primary study should have had in order to be useful to address our research questions. These were defined according to the research objectives initially defined:

* Q1: Is the non-functional requirement well defined?

* Q2: If the primary study defined a challenge to deal with a non-functional requirement, is this well defined?

* Q3: If the primary study proposes an automated approach to deal with a non-functional requirement, are the data preparation steps well-defined?

* Q4: If the primary study proposes an automated technique to deal with a non-functional requirement, is the pipeline used to optimize it described?

In particular, Q1 was the instrument to ensure a proper answer to RQ$_1$, while Q2 mapped our requirements to address RQ$_2$. Finally, Q3 and Q4 were relevant to understand the way automated approaches were fed and engineered and, as a consequence, to address RQ$_3$.

When assessing the primary studies against each of the qualitative questions, previous systematic literature reviews (e.g., <cit.>) assigned a boolean value to indicate whether a study had or not the quality required with respect to a property considered. However, assessing the primary studies through boolean values might be challenging, other than possibly threatening the validity of the assessment. For instance, there might be cases where the challenges associated with non-functional requirements may be logically elicited from the text, even though not explicitly stated. To deal with this process, we, therefore, opted for the application of a fuzzy linguistic approach <cit.>, which consists of rating each primary study through a continuous variable ranging between 0 and 1. In particular, the scores were assigned as follows:

* 0 => No

* 0.1-0.3 => Rarely

* 0.4-0.6 => Partly

* 0.7-0.9 => Mostly

* 1 => Yes

In other terms, for each qualitative question, the primary studies were assigned a value reporting how explicit and clear the content was in that respect. At the end of the evaluation conducted for each question, the total merit of a primary study was computed as follows:

* 0 => No

* 0.1-1.5 => Rarely

* 1.6-2.8 => Partly

* 2.9-3.6 => Mostly

* 3.7+ => Yes

To be finally accepted as part of our systematic analysis, the primary study should have obtained a final score of more than 1.6, i.e., it should have partly specified the required pieces of information to address the research questions of the study.

§.§ Design of the Data Extraction Form

As a final step of the research method applied to address the goals of our study, we designed the data extraction form, namely the specification of the pieces of information to collect when addressing our research questions. Table <ref> summarizes the data collected, reporting (i) the dimension the attribute group referred to; (ii) the scope where the data has been used; (iii) the description of the dimension considered; and (iv) the specific attributes considered. As shown in the table, we collected six main categories of information. At first, we identified pieces of information that might help statistically describe our sample in terms of bibliometrics, e.g., publication year and venues: we used the knowledge acquired to describe the trends in terms of publication, the most relevant venues accepting research papers on non-functional requirements of ML-enabled systems. Besides these meta-data, we then collected and stored information that may be directly connected to the specific research questions posed in our study and that, therefore, was used in the context of the data analysis and reporting process.

Data extraction Form.

Dimension     Scope     Description     Attribute Collected
Paper Information     Bibliometrics     This component includes general information and criteria used to assess the quality of the selected studies.     Seed set or Systematic Literature Review

Year of publication

Accepted Score

Journal or a conference.
Terms concerning non-functional requirements     RQ$_1$     This component encompasses all the keywords that make it possible to describe non-functional requirements, both those that already exist in the literature and new non-functional requirements that emerged from this study.     Non-Functional Requirements

Domains where non-functional requirements were studied     RQ$_1$     This component is similar to row 2 but contains all the domains in which non-functional requirements have been studied and found to be problematic. This will make it possible to diversify the impact of AI according to context and to identify the least researched areas.     ML Domains

Environmental Domains
Challenges of SE approaches for ML-enabled systems     RQ$_2$     This component contains a list of challenges explicitly stated in primary studies and any future challenges and problems emerging from these studies. This information will help us identify areas where further research is needed to improve SE approaches for ML-enabled systems and improve the existing ones.     Challenges

New Challenges Emerged
Data-driven terms     RQ$_3$     The term data-driven includes all the techniques and technologies used on the datasets before training the models and the ML activities for each dataset the researchers performed.     Datasets

Non-functional requirement considered

Technologies for testing and analyzing the datasets

Pre-processing techniques
SE approaches for ML-enabled systems     RQ$_3$     This component contains the knowledge areas most suitable for SE approaches for ML-enabled systems. We will collect and classify SE approaches according to different domains of ML systems.     Non-functional requirement considered in each domain

Architectures of models.

Type of ML systems

Tools, Frameworks, and Methodologies to optimize Non-Functional Requirements during and after training.

Overview of the process for selecting papers.

§.§ Execution of the Research Methods

Figure <ref> overviews the outputs of the execution of the research methods designed in the previous sections. The systematic literature review search was first conducted by the first author of the paper on September 4, 2023. As summarized in the figure, the application of the search query produced a total amount of 2,571 hits. As expected, most of these hits came from Scopus (1,980), with ACM Digital Library and IEEEXplore reporting a lower amount of results (552 and 39, respectively).

The first author of the paper manually collected all the relevant information on the resources identified and stored them within a sheet that was later shared with the second author. Afterwards, the first author proceeded with the application of exclusion and inclusion criteria: the former led to filtering out 1,088 research results, while the latter had a total of 1,471. The very limited amount of relevant papers identified after the application of the guidelines by Kitchenham et al. <cit.> led the two authors to open a discussion about the method defined and the implications that such an outcome might have had on the completeness of the search. In particular, the two authors scheduled two meetings of one hour each where they jointly reviewed the activities performed in an effort to understand the motivations behind the limited amount of resources retrieved. As a result of this fine-grained investigation, they could draw two main observations:

* The vast majority of the hits were found to be irrelevant to the goals of the study because they mentioned terms connected to non-functional requirements or machine learning either as part of the background or the related work, hence not discussing the challenges posed by non-functional requirements in the context of ML-enabled systems nor proposing automated approaches to cope with them;

* The few relevant hits which passed the inclusion criteria did not only explicitly mention the terms of our search string but also discussed the fine-grained requirements considered.

Based on these observations, the two authors realized the need for boosting the traditional search process with additional steps. While they first considered the re-execution of the systematic search through a search string designed to include all non-functional requirements classified by researchers so far, this would have led to three critical issues. First, the answer to RQ$_1$ would have been biased by design: we would have indeed introduced ourselves to the set of non-functional requirements to look for rather than letting them emerge from the analysis of the available literature. Second, the search string would have included a number of AND/OR conditions that might have increased noise, leading to the retrieval of several resources not connected to SE4AI research. Last but not least, such an increased noise would have led to the identification of a prohibitively expensive amount of resources to analyze—Habibullah et al. <cit.> estimated over 200k hits for a search string including all the non-functional requirements of ML-enabled systems classified in literature.

To cope with these challenges, we opted for complementing the systematic literature search with the seed set identification and the snowballing process. Since our goal was that to analyze the literature available in the field of SE4AI on non-functional requirements, we then proceeded with the analysis of the papers published to top-tier software engineering and artificial intelligence venues. The first author of the paper looked for those resources, extracting information concerned to 30,004 papers from conferences and 18,153 papers from journals.
After applying the exclusion and inclusion criteria, 77 unique new resources were deemed relevant, 71 from conferences and 6 from journals.

Before proceeding with the next steps, the second author of the paper reviewed the activities conducted so far. Also, two one-hour meetings were scheduled to discuss the results achieved. The meetings allowed us to acknowledge the soundness of the decision made when opting for a seed search. Hence, the 89 relevant resources acquired so far were taken as input for the snowballing step. Also, in this case, the first author was responsible for conducting the search process: by iterating multiple times, both backward and forward, the author was able to identify an additional set of 31 primary studies. Finally, the total amount of 120 papers identified through the three-step search process was considered within the scope of the quality assessment. Given the criticality of this step, this was jointly conducted by the authors: after analyzing 81,081 results, 69 primary studies finally contributed to our study.

To make the reader aware of the effort required to conduct the search/analysis process and contribute to the transparency/replicability of our work, we also estimated the number of person-hours invested in the work. As for the first author, the estimation is about 800 person-hours; as for the second, it is about 400 person-hours.

Reproducibility and Replicability of Our Work.
Our online appendix reports all the fine-grained steps described above. The interested reader may use the appendix to replicate our work or build on top of our findings to create additional knowledge on non-functional requirements of ML-enabled systems: <cit.>

§ ANALYSIS OF THE RESULTS

In this section, we report quantitative and qualitative insights coming from the data extraction and analysis phase. We first analyze bibliometric data to describe the sample considered in our research. Afterward, we address the specific research questions of the study.

Results of the quality assessment.

Papers per publication venue.

§.§ Bibliometrics

Figure <ref> overviews the results of the quality assessment procedure. We finally included a total of 69 primary studies. Most of them (31, 45%) received an overall quality score of “Partly”, i.e., they specified the pieces of information required to address our research questions mostly implicitly, yet giving us a chance to elicit them satisfactorily. The other 38 primary studies were more explicit and, indeed, reached an overall quality score of “Mostly” or “Yes”. The primary studies were mostly retrieved through the seed set search (41, 59%), while the other 22 resources (32%) were identified by snowballing the seed primary studies identified. Only 5 papers were retrieved through the systematic literature search and only one additional resource could be identified through the snowballing process conducted on the set of papers identified with the systematic search. These considerations further justified our choice of complementing the traditional systematic literature search with additional instruments.

Papers published per publication venue.

Papers published on the matter over time.

More interesting were the insights coming from the publication venues, which are shown in Figure <ref>. We observed that the vast majority of the primary studies (61, 88%) appeared in conference proceedings, with a limited amount of resources published in journals (8, 12%). This finding seems to suggest that the research on non-functional requirements of ML-enabled systems is still at its early stage, with researchers interested in discussing recent advances in venues that allow discussion and interaction with the research community. On a similar note, the result may stimulate young researchers to enter this growing research area. The analysis of the amount of papers published on the matter per year confirmed the early nature of the research area (Figure <ref>). According to our findings, until 2015, no articles were published, and from 2015 to 2018 only a few papers were published, while a notably increasing trend could be found in the last two years. Performing a systematic synthesis of the knowledge collected so far could provide a relevant boost to the research activities that will be performed in the next years: our work indeed identifies multiple implications and challenges that the software engineering research community will be called to address.

§.§ RQ$_1$ - What are the non-functional requirements of ML-enabled software considered by researchers?

The first goal of our work was to classify the non-functional requirements of machine learning-enabled systems. The primary studies considered were first analyzed in an effort to elicit the non-functional attributes that they aimed to address. Afterward, we provided the extracted non-functional attributes with a (1) name, (2) a description, and (3) a reference class, i.e., a high-level class that may be used to cluster together multiple non-functional requirements based on similarity, i.e., non-functional requirements sharing similar purposes. For instance, requirements such as “privacy” and “security” were both extracted and mapped onto the “Resiliency” class, as they both referred to aspects making an ML-enabled system resilient to unexpected events arising in operation, including external attacks. Whenever possible, i.e., when considering the papers that received a quality score of “Yes”, the extraction was relatively straightforward as these resources explicitly referred to the non-functional requirements considered; as such, they could be easily extracted and mapped onto known classes of non-functional requirements or onto new classes according to the conceptual similarity between the non-functional requirement analyzed and the set of classes identified. In the other cases, relevant terms connected to non-functional aspects were extracted, elaborated, and only later mapped onto the corresponding class of non-functional requirements. For example, the primary study 10.1145/3530019.3530025 reported terms like “memory problems” and “battery drain”, which we used to interpret the context of the study and assigned the non-functional aspect to the class entitled “Sustainability”. As an output of the process, we were able to develop a taxonomy of non-functional attributes of ML-enabled systems.

It is worth pointing out that the names assigned to the classes, the description of each non-functional requirement, and the classification exercise as a whole, were informed by different sources of the existing body of knowledge, i.e., by the primary studies collected in this paper, other resources of the state of the art not directly related to our research goals, and the Systems & Software Quality Requirements & Evaluation standard (ISO/EIC 25010). In particular, the process was initially conducted by the first author of the paper, who (1) analyzed the primary studies to elicit the non-functional attributes considered and (2) provided a preliminary version of the taxonomy. The author exploited the existing body of knowledge in an effort to homogenize the definitions and classification; e.g., he favored the assignment of well-established class names whenever possible in an effort to provide a more comprehensive and usable taxonomy. The preliminary taxonomy was later the subject of evaluation. The second author joined the process at this stage, and a discussion was opened on the taxonomy produced. The two authors discussed about the consistency of the classification performed, other than on the names and descriptions assigned. Whenever needed, they modified the taxonomy—this happened in four cases, in which some non-functional requirements were grouped differently, and the names assigned to the classes were modified.

In the first place, as a result of this procedure we could identify a total amount of 30 non-functional requirements. These were: accuracy usman2021nn,9439863,9508369,9623049,Eykholt_2018_CVPR,10.1145/3132747.3132785,10.1145/2810103.2813687,10.1145/3460120.3484818,8746775,8952197,10.1145/3551349.3556920,10.1145/3540250.3549093,10.1145/3338906.3338954,10.1145/3540250.3549103,10.1145/3510003.3510163,10.1145/3510003.3510087,10.1145/3510003.3510221,9402020,10.1145/3510003.3510231,10.1145/3510003.3510202,8812047,9402039,10.1145/3377811.3380368,10.1145/3510003.3510051,9402057,10.1145/3533767.3534373,10.1145/3533767.3534391,10.1145/3533767.3534408,10.1145/3533767.3534386,10.1145/3460319.3464809,secrypt19,9590404,10.1145/3368089.3409697,10.1145/3468264.3468565,9676691,10.1145/3506695,10.1145/3429444,10.1145/3243734.3243792,10.1145/3540250.3549102,10.1145/3540250.3549169,10.1145/3468264.3468537,10.1145/3368089.3409730,10.1145/3368089.3409668,10.1145/3368089.3409739,10.1145/3368089.3417065,10.1145/3180155.3180220,10.1145/3238147.3238187,10.1145/3510003.3510191,10.1145/3551349.3556906,10.1145/3510003.3510088,10.1145/3551349.3561158,Chen_2022_CVPR,10.1145/3340531.3411980, robustness 7958570,8418593,usman2021nn,9439863,9508369,9623049,Eykholt_2018_CVPR,10.1145/3132747.3132785,huang2017safety,9402124,9451178,9766323,10.1145/3460120.3484818,8746775,10.1145/3551349.3556920,10.1145/3338906.3338937,9402020,10.1145/3510003.3510231,8812047,9402039,10.1145/3377811.3380368,10.1145/3293882.3330580,10.1145/3533767.3534373,10.1145/3533767.3534391,10.1145/3533767.3534408,10.1145/3533767.3534386,10.1145/3460319.3464809,10.1145/3540250.3549102,10.1145/3368089.3409739,10.1145/3368089.3417065,10.1145/3238147.3238165,10.1145/3377811.3380331,10.1145/3238147.3238187,10.1145/3510003.3510191,10.1145/3510003.3510088,Chen_2022_CVPR, security 7958570,8418593,usman2021nn,9439863,10.1145/3132747.3132785,9402124,9451178,9766323,10.1145/3460120.3484818,10.1145/3551349.3556920,10.1145/3338906.3338954,9402020,8812047,9402039,10.1145/3377811.3380368,10.1145/3293882.3330580,10.1145/3533767.3534373,10.1145/3460319.3464809,secrypt19,9590404,9676691,10.1145/3368089.3409739,10.1145/3510003.3510191, performance 10.1145/3530019.3530025,9402124,9451178,9766323,8746775,10.1145/3468264.3468536,10.1145/3540250.3549093,10.1145/3540250.3549103,10.1145/3510003.3510163,10.1145/3510003.3510087,10.1145/3533767.3534391,10.1145/3533767.3534408,10.1145/3533767.3534386,secrypt19,10.1145/3506695,10.1145/3243734.3243792,10.1145/3540250.3549102,10.1145/3368089.3417065,10.1145/3551349.3556906,10.1145/3551349.3561158,Chen_2022_CVPR, fairness 7961993,pmlr-v81-buolamwini18a,10.1145/3338906.3338937,10.1145/3468264.3468536,10.1145/3540250.3549093,10.1145/3540250.3549103,10.1145/3510003.3510087,10.1145/3510003.3510202,9402057,9590404,10.1145/3368089.3409697,10.1145/3468264.3468565,10.1145/3540250.3549169,10.1145/3468264.3468537,10.1145/3238147.3238165,10.1145/3377811.3380331,10.1145/3340531.3411980,10.1145/3447548.3467177,10.1145/3534678.3539145, behavioral 10.1145/3132747.3132785,9451178,10.1145/3533767.3534391,10.1145/3338906.3338937,10.1145/3338906.3338954,10.1145/3510003.3510231,10.1145/3368089.3409730,10.1145/3180155.3180220,10.1145/3238147.3238165,10.1145/3238147.3238187,10.1145/3510003.3510191,

interpretability 9451178,10.1145/3338906.3338954,10.1145/3510003.3510087,10.1145/3533767.3534373,9590404,10.1145/3429444,10.1145/3243734.3243792,10.1145/3368089.3409739,
transferability 7958570,huang2017safety,9402124,9766323,10.1145/3377811.3380368,10.1145/3460319.3464809,10.1145/3510003.3510191, safety Eykholt_2018_CVPR,10.1145/3132747.3132785,huang2017safety,10.1145/3338906.3338954,10.1145/3377811.3380368,10.1145/3180155.3180220,10.1145/3551349.3556906, reliability 9451178,10.1145/3510003.3510231,10.1145/3293882.3330580,10.1145/3533767.3534373,10.1145/3533767.3534391,9590404,10.1145/3238147.3238187, explainability 10.1145/3460120.3484818,10.1145/3338906.3338937,10.1145/3510003.3510202,10.1145/3429444,10.1145/3243734.3243792,10.1145/3447548.3467177,  retrainability 9439863,9623049,9402039,10.1145/3293882.3330580,10.1145/3368089.3417065,10.1145/3377811.3380331, cost 10.1145/3510003.3510221,8812047,9590404,10.1145/3506695,10.1145/3368089.3417065, energy consumption 10.1145/3530019.3530025,10.1145/3510003.3510221,10.1145/3506695,10.1145/3540250.3549102,10.1145/3510003.3510088,10.1145/3551349.3561158, privacy 10.1145/2810103.2813687,10.1145/3540250.3549093,10.1145/3293882.3330580,secrypt19,9590404,9676691,10.1145/3510003.3510088,
transparency pmlr-v81-buolamwini18a,10.1145/3338906.3338937,10.1145/3368089.3409697,10.1145/3243734.3243792, scalability 8418593,huang2017safety,10.1145/3510003.3510051,10.1145/3533767.3534373,
accountability 7961993,pmlr-v81-buolamwini18a,10.1145/3368089.3409697,
Capacity 10.1145/3530019.3530025,10.1145/3506695,10.1145/3551349.3556906, efficiency 10.1145/3540250.3549102,Chen_2022_CVPR, stability 10.1145/3533767.3534373,10.1145/3510003.3510088, replaceability 10.1145/3510003.3510051,10.1145/3368089.3409668, reusability 10.1145/3510003.3510051,10.1145/3368089.3409668, imperceptibility Eykholt_2018_CVPR, ethics 10.1145/3368089.3409697,
flexibility 10.1145/3293882.3330580,  sustainability 10.1145/3510003.3510051,
reproducibility 10.1145/3510003.3510163, and
availability10.1145/3540250.3549102, adaptability 10.1145/3510003.3510088.

Interestingly, our systematic exercise could identify not only non-functional requirements that emerged already in previous studies <cit.>, but also additional categories that were not previously pointed out, i.e., transferability, cost, accountability, energy consumption, sustainability, capacity, stability, behavioral, imperceptibility, efficiency, availability and adaptability.

Source-wise distribution of non-functional requirements in machine learning-enabled systems. A symbol `*' associated to one non-functional requirement denotes that it is a newly classified one.

Figure <ref> depicts the number of primary studies that targeted each of the 30 non-functional requirements identified. In the figure, the symbol `*' denotes the new non-functional requirements classified though our study, namely the ones that did not emerge from previous studies. As somehow expected, non-functional requirements such as accuracy and robustness of ML-enabled systems were those more frequently considered (53 and 36 times, respectively): these two aspects are likely the ones considered as recurrent and fundamental features of any machine learning model. Nonetheless, aspects connected to performance (24), security (24), fairness (19), and behavioral (11) seem to be growing research areas. On the contrary, multiple aspects were little explored, e.g., reusability or scalability, possibly indicating emerging research themes that would be worth further exploring.

Non-Functional Requirements in Ml-enabled systems.

Cluster     NFRs     Definition

Accuracy     Accuracy     The degree to which a model's predictions match the actual values.

Efficiency     Performance     The ability of a system to perform actions within defined time or throughput bounds.

Capacity     The amount of space required to store the model and any associated data.

Stability     Degree to which the output of a model varies as a consequence of perturbations to its input.

Scalability     The capability to handle increased workloads by adding resources while maintaining or enhancing model performance.

Maintainability     Replaceability     The degree to which a model can be replaced or substituted with another model without significant changes to the system.

Retrainability     The degree to which a model can be retrained on new data without significant performance loss.

Reproducibility     The degree to which a model's results can be reproduced by others using different software or hardware.

Transferability     The degree to which a model trained on one data set can be applied to another with similar characteristics.

Reusability     The degree to which a model can be reused in different applications or contexts.

Adaptability     The ability of the model to adapt to changing requirements or environments.

Resiliency     Security     The degree to which a model and its associated data are protected against unauthorized access, modification, or theft.

Safety     The degree to which a model and its outcomes are safe for humans and the environment.

Privacy     The degree to which a model and its associated data protect individuals' privacy rights and comply with data protection regulations.

Robustness     The ability of a model to maintain its performance when faced with uncertainties or adversarial conditions.

Reliability     Degree to which a model is resilient to errors and to variations of the surrounding environment.

Behavioral     The degree to which a model's outcomes align with requirements and expectations.

Flexibility     The degree to which a model can adapt to input data or environment changes without significant performance degradation.

Availability     The degree to which a model is operational and accessible when needed, without significant downtime or interruption.

Sustainability     Fairness     The degree to which a model produces unbiased predictions and decision-making outcomes across different groups of individuals.

Ethics     The degree to which a model mitigates potential societal risk.

Accountability     The degree to which individuals or organizations are held responsible for the actions of the model and its outcomes.

Cost      The overall economic means required to develop and maintain an ML-enabled system.

Energy Consumption     The amount of energy required for training and inference of the model and its impact on the systems.

Usability     Interpretability     The degree to which a model's predictions and decision-making process can be explained in terms of causality or human-understandable concepts.

Imperceptibility     The system's ability to produce outputs that are indistinguishable from what a human would produce in the same scenario.

Explainability     The degree to which a model's predictions can be explained and understood by humans.

Transparency     The degree to which a model's inner workings and decision-making process can be understood and evaluated by humans.

The classification exercise led to the identification of six main classes of non-functional requirements: Table <ref> reports, for each class, the set of non-functional requirements belonging to the class, along with their definition. More specifically, we identified the following classes:

Accuracy. The first class contained only the accuracy requirement, namely the degree to which a model's predictions match the actual values. We considered accuracy as a non-functional property for two main reasons. In the first place, multiple papers included in our systematic literature review, e.g., <cit.>, described accuracy as a non-functional requirement, defining it as the number of correctly predicted data points out of all the data points. As our work aims at synthesizing the current knowledge available on the matter, we preferred to be conservative and considered accuracy as reported in the state of the art. In the second place, the definition is in line with the concept of non-functional requirement, i.e., “a condition that specifies a criterion that may be used to judge the operation of a system rather than specific behaviors” <cit.>: as a matter of fact, accuracy does not define constraints on the specific functionality that a system should enable, but rather question how the system performs in terms of correctness of the predictions made. In other terms, accuracy represents an attribute that can be used to judge the operation of a system rather than its specific behavior, i.e., it is by definition a non-functional requirement. The class was made isolated because of two main reasons: (1) accuracy represents the key feature to optimize by machine learning-enabled systems; (2) the non-functional requirement cannot be conceptually compared to any other, being “unique”.

Efficiency. This was concerned with the overall performance and effectiveness of machine learning-enabled systems. The ISO/IEC 25010 standard includes performance efficiency as one of its main characteristics, encompassing attributes such as time behavior, resource utilization, and capacity. As such, we exploited the same reasoning to cluster non-functional requirements such as performance, capacity, stability, and scalability, under the “Efficiency” class.

Maintainability. Maintainability is chosen as a cluster to represent replaceability, retrainability, reproducibility, transferability, reusability, and adaptability: it emphasizes the ability of a system to adapt and evolve over time. All these non-functional requirements are critical to ensure that the system can be maintained and updated as needed and that it can be easily adapted to new use cases or environments. By prioritizing maintainability, designers and developers can create reliable, efficient, adaptable, flexible, and sustainable systems over the long term.

Resiliency. Resilience was chosen as a cluster to represent robustness, reliability, behavioral, flexibility, security, safety, privacy, and availability because it emphasizes the ability of a system to adapt and recover from adverse events while maintaining its functionality and performance. Robustness, reliability, and behavioral flexibility are essential to ensure the system can operate in various conditions and remain responsive to changing circumstances. Security, reliability, and confidentiality are also critical to ensure the system protects users and stakeholders from harm, including threats to their physical safety, personal information, and data confidentiality. By prioritizing resilience, designers and developers can build reliable and secure systems that can withstand disruptions and threats while maintaining their core functions and services.

Sustainability. Sustainability is chosen as a cluster to represent economic, social, and environmental aspects. It ensures that resources are used efficiently and effectively, reducing the environmental impact, energy consumption and financial costs associated with producing the model and making inferences. Additionally, sustainability is chosen to represent fairness, ethics, and accountability, as these aspects are critical for ensuring that the system benefits all stakeholders, including marginalized and vulnerable communities, and operates in a way that aligns with ethical principles and values. The rationale to classify social and ethical concerns under the “Sustainability” category comes from the increasing recognition that social sustainability is currently experiencing. For instance, the United Nations included the reduction of inequalities among the 17 objectives for sustainable development.[The United Nations Goals for Sustainable Development: <https://sdgs.un.org/goals>.] Researchers have been also arguing that fairness and ethical concerns should be considered as a form of sustainability. McGuire et al. <cit.> advocated that social sustainability refers to multiple dimensions, including pro-social vs. anti-social affordances. On a similar note, various other researchers <cit.> argued to consider social and ethical aspects as sustainability properties. Overall, sustainability can be considered as a guiding principle for designing and developing efficient, cost-effective but also equitable, ethical, and accountable systems.

Usability. The usability class is similar to the one defined by Habibullah et al. <cit.>. More particularly, this class was chosen as the cluster to represent interpretability, imperceptibility, explainability, and transparency: it is concerned with ensuring that a system is easy to use and understand for end users, reducing the cognitive load required to interpret the model's behavior, and making it more transparent and explainable.

Researchers can use the taxonomy built in the context of our work to have a comprehensive mapping of the relevant non-functional requirements to optimize when developing machine learning-enabled systems, other than understanding the quality and efficiency aspects practitioners should focus on when releasing machine learning models.

§.§.§ On the non-functional requirement domains

Distribution of NFRs across ML domains in ML-enabled systems.

Distribution of NFRs across Environment domains in ML-enabled systems.

Figure <ref> overviews the number of primary studies targeting each machine learning domain extracted. As shown, the most frequent are `Computer Vision' (47), `Decision Making' (18), and `Natural Language Processing' (9), while other, less targeted domains pertain to emerging technologies, e.g., `Recommender Systems' or `Automatic Speech Recognition'. These insights raise some key ML domains that researchers have been analyzing in the recent past, but also raise contexts where further research might be worth focusing on. The much larger amount of primary studies targeting `Computer Vision' may indicate the critical nature of such a domain with respect to the management of non-functional requirements. This may be due to the nature of the inputs that the machine learning-enable systems should consider, i.e., images or videos, which may be the subject of multiple concerns such as accuracy, security, ethics, and privacy—this may potentially emphasize the need for novel methods to process images and/or deal with non-functional attributes in the field.

Heatmap of Non-Functional Requirements across Environment domains in ML-enabled systems.

Besides the ML domains, we also analyzed the environments where the primary studies focused on. These are shown in Figure <ref>. The most common environment has been classified as 'General Environment' (39) and indicates that most papers did not explicitly specify the environmental conditions which they experimented with. Other papers were instead more specific, targeting 'Driving Systems' (12), 'Mobile Devices' (7), and 'Critical/Evolving Scenarios' (5). Looking at the figure, we may conclude that most primary studies worked in generalized conditions, with fewer researchers deepening the analysis of non-functional requirements in the context of ML-enabled systems designed to work under specific working conditions. This may represent a challenge for further research in the field: researchers might indeed be interested in further understanding the peculiarities of the various environments, possibly letting emerge novel non-functional requirements to deal with or learning how to prioritize non-functional requirements based on the environmental conditions that the ML-enabled systems are called to face.

Providing an additional perspective, Figure <ref> illustrates a heatmap that shows the relation between the specific non-functional requirements presented in Table <ref> and the corresponding environments depicted in Figure <ref>. Each entry in the heatmap represents a numerical value, indicating the frequency with which the `i-th' non-functional requirement has been considered within the `j-th' environment, with `i' corresponding to the rows and `j' to the columns. By looking at the columns, the heatmap shows which are the non-functional requirements often addressed within the same environment. By looking at the rows, the heatmap shows the environments where each non-functional requirement has been analyzed so far. Through this visualization, we could first observe that there exist a number of environments where the current knowledge seems to be limited: for example, the environments concerned with `Finance', `IoT', and `Cloud', and `Edge Devices' were the least targeted ones, which possibly suggests that further research might consider the impact of non-functional requirements in these contexts.

As expected, accuracy represents the property more frequently addressed in all the environments. Nonetheless, some non-functional requirements like robustness, performance, fairness, and security have been targeted much more than others, possibly indicating that researchers perceived these as essential properties to investigate. At the same time, Figure <ref> could further highlight the non-functional requirements that were somehow neglected so far: for instance, all the properties concerned with maintainability were found to be mostly unexplored independently from the environment considered. These insights may, again, be beneficial for researchers interested in working toward understanding non-functional requirements and their impact on ML-enabled systems working in different environments, and their understanding may help make more informed decisions and improve system development practices.

Finally, Figure <ref> also suggests the existence of possible trade-offs to further consider in future research. Indeed, while the figure does not indicate if more non-functional requirements were simultaneously addressed by researchers, it may still highlight potential relations. As an example, in the `Driving Systems' environment we noticed that accuracy and robustness were more frequently targeted, possibly indicating that these two non-functional requirements should jointly considered by researchers to enlarge the current body of knowledge on the trade-offs among non-functional requirements of ML-enabled systems.

1As a result of RQ$_1$, we could identify a total of 30 relevant non-functional attributes, classified according to six main categories. We also identified the most frequent ML domains and environments where these non-functional attributes have been investigated. The resulting taxonomy maps the current knowledge on the matter and provides insights into the research areas that may be worth exploring in the future.

§.§ RQ$_2$ - What are the challenges of dealing with non-functional requirements of ML-enabled software?

The second research question is concerned with the identification of the challenges faced in dealing with the non-functional requirements of machine learning-enabled software. Taking the primary studies as an input, we processed them individually. We first went through the content of each primary study in search of two pieces of information: (1) the `context' where the non-functional requirement was considered; and (2) the `problem(s)' the primary study aimed at addressing. The information about the context was helpful to elicit where a non-functional requirement should be more carefully taken into account, while the information about the problem considered by the primary study was used to elaborate on the motivations making a non-functional requirement hard to treat. After collecting these pieces of information, we then applied an inductive reasoning to elicit the challenge(s) associated with the management of non-functional requirements. This process was systematic and included (1) the identification of the issues addressed by the primary studies; (2) the thematic analysis of these issues; and (3) the classification of the challenges related to management of a non-functional requirement. As such, the set of challenges described in our work has been derived from the content of the primary studies through the analysis of the contexts and problems considered by these primary studies. In doing so, we focused on the software engineering perspective, hence attempting to produce open challenges that our research community is called to address through further investigations.

Our work identified more than twenty software engineering challenges for our research community. First and foremost, it is worth pointing out that most of the challenges identified pertained to the use/integration of neural networks within more complex software systems. This was somehow expected, as deep learning has become the most widely employed form of machine learning to empower the capabilities of traditional software systems <cit.>. In the second place, it is important to remark that accuracy represents a cross-cutting concern, i.e., all the challenges identified have implications for the accuracy of machine learning-enabled systems. For this reason, we preferred to discuss accuracy while presenting the challenges connected to the other classes of non-functional requirements identified in RQ$_1$. The list of challenges is presented in the following.

Efficiency (E): In terms of overall performance and effectiveness of machine learning-enabled systems, we could elicit three major challenges which are connected to internal errors, high latency, and memory issues. Specifically:

* Challenge E.1 - Dealing with Internal Errors. Neural networks can produce erroneous outputs as a consequence of internal errors due to incorrect parameters, incorrect weight values, uncovered root causes, and incorrect manual labeling. Those internal errors may significantly impact the performance of machine learning-enabled systems, leading to reduced accuracy, slower inference times, and increased energy consumption 9439863,10.1145/3530019.3530025,10.1145/3506695. For example, incorrect weight values or parameters can cause the network to converge slowly or get stuck in suboptimal solutions, resulting in poor performance. Dealing with internal errors has two main connotations such as tracking errors in deep neural network models and addressing them. Our findings suggest that research in terms of bug localization, performance monitoring, and program repair might support the challenges identified.

* Challenge E.2 - Dealing with High Latency. Literature reported that multiple design choices related to hyper-parameter tuning, model optimization, and model architecture can affect the overall efficiency of neural networks, even though those decisions might sometimes lead to increased accuracy 10.1145/3506695. In other terms, the definition of neural networks should be considered as an optimization problem, where designers look for the best trade-off between performance and other quality measures and carry out preliminary studies to find the best configurations to employ within the specific use case. Additionally, the computational efficiency of DNN systems is very sensitive to even slightly different inputs. As a consequence, a change to the inputs may result in a significantly higher computational demand, deteriorating the overall models' efficiency 10.1145/3540250.3549102,10.1145/3551349.3561158,Chen_2022_CVPR.
As such, researchers in the field of software analytics, software architecture, and software quality and optimization might help address this challenge by means of analytical and code quality instruments aiming at finding the right compromise between accuracy and efficiency.

* Challenge E.3 - Dealing with Space. A third critical challenge arising from our review consists of the space required by machine learning models and, particularly, by neural network-based solutions. These may indeed present memory issues due to their large size, which has been particularly relevant in contexts like edge computing and IoT. Deploying large models on such devices can result in memory constraints affecting system performance 10.1145/3530019.3530025,10.1145/3506695,10.1145/3551349.3556906. In response to this challenge, designers should be able to apply policies and strategies to build reduced models before deploying them on devices. These strategies may mitigate memory issues and improve the overall performance of machine learning-enabled systems. In this respect, our findings call for further research on the emerging field of tiny machine learning <cit.>, that is, the combination of hardware, algorithms, and software that support on-device sensor data analytics at low power. While some initial effort has been made in the artificial intelligence research community (e.g., <cit.>), we point out a lack of knowledge of the software engineering side of the matter.

The three challenges identified were discussed in all the domains requiring the definition of large models, i.e., `Computer Vision', `Driving Systems', `Natural Language Processing', `Mobile Devices' and `IoT Devices'. This was somehow expected, as scalability concerns become evident in environments requiring high computational demands.

Maintainability (M): When it turns to the ability of machine learning-enabled systems to be adapted and evolved over time, we could elicit two main challenges connected to model reproducibility and model decomposition and reuse.

* Challenge M.1 - Increasing Model Reproducibility. Machine learning-enabled systems based on deep learning models are known to be complex and difficult to reproduce accurately. One of the primary challenges in reproducing deep learning models concerns with randomness, which can affect the behavior of the algorithms used. Additionally, hardware non-determinism, such as that present in graphics processing units (GPUs), can further complicate the reproducibility of deep learning models 10.1145/3510003.3510163. Our findings seem, therefore, to suggest the need for mechanisms that could support the verification of deep learning models, for instance, by injecting elements of randomness and non-determinism to check how the system may behave in those circumstances.

* Challenge M.2 - Increasing Model Decomposition and Reuse.
A critical challenge when building and improving machine learning-enabled systems based on deep learning models is the need to reuse parts of previously constructed models. This can be difficult, for instance, when attempting to replace potentially defective parts with others.  A possible approach concerns with the identification of the parts responsible for each output class or module in the models: this would allow to selectively reuse or replace only the desired output classes to build the model. This challenge is strictly connected to the properties of randomness and non-determinism already discussed in the previous point. Indeed, increasing decomposition and reuse requires a proper understanding of the potential sources of variability and non-determinism in the system's software and hardware components. Therefore, improvements in terms of reproducibility may also lead to more reusable models 10.1145/3510003.3510163,10.1145/3510003.3510051,10.1145/3368089.3409668. On the one hand, our findings reinforce the need for research in terms of verification. On the other hand, the challenge of identifying reusable components seems to call for research in terms of program slicing and refactoring, which might support machine learning engineers to properly extract parts of the models for reusability purposes.

Challenges connected to maintainability were found to be pervasive in all domains, hence representing concerns that may impact any kind of machine learning-enabled system.

Resiliency (R):
Improving the universal robustness and security of machine learning models present key issues 10.1145/3551349.3556920. This was the class of non-functional requirements where we identified more challenges (8); these were connected to various aspects such as adversarial attacks, theft of model and intellectual property, attack and defense of models, repairing internal models, post-deploy issues, preserving privacy, and vulnerability transferability.

* Challenge R.1 - Resilience to Adversarial Attacks. Machine learning models, and more particularly deep neural networks, are vulnerable to the so-called  adversarial attacks <cit.>, namely attacks that manipulate the input data in a way that causes the neural network to produce an incorrect output and significantly drop accuracy. Different types of adversarial attacks can be targeted or untargeted and can be carried out through various techniques such as adversarial examples <cit.>, adversarial perturbation <cit.>, poisoned data <cit.>, and backdoor samples <cit.>. These types of attacks can negatively affect the overall robustness of the model and decrease its accuracy. As such, a key challenge is represented by the resilience to adversarial attacks usman2021nn,7958570,9402124. More specifically, existing literature highlighted the need for instruments able to design perturbations that may properly generate adversarial examples for different machine learning models, e.g., rotations, smoothing, and erosion operations in machine learning-enable systems targeting computer vision problems 9508369,huang2017safety,Eykholt_2018_CVPR,8746775. Those instruments might support practitioners in terms of an improved comprehension of how models can be affected by vulnerabilities, other than through improved methods to increase appropriate defenses against adversarial attacks. Hence, our findings suggest additional verification and validation research efforts: indeed, while the security perspective has been often targeted by the artificial intelligence community through the definition of new algorithms and techniques <cit.>, software engineering research might be complementary and propose novel verification and validation approaches, which may improve the policies and strategies employed by practitioners when testing for security.

* Challenge R.2 - Exploiting the Sensibility of Adversarial Attacks. A complementary perspective is concerned with the sensibility that certain adversarial attacks may have to input distortions and how such a sensibility can be exploited to improve the security of machine learning models. For example, in computer vision applications, attacks made through non-additive noise or geometric morphing may cause serious security concerns, yet those filters are sensitive to input distortions, meaning that automatic correction instruments may potentially distort the malicious inputs to mitigate their effects on the machine learning model 8418593,8746775,7958570,10.1145/3368089.3409739. Our findings, therefore, point out the need for techniques that may handle perturbations that not only occur naturally in the physical environment but could be maliciously generated by adversary attacks. The perturbations are typically stealthy and undetectable, which poses further challenges for systems based on deep neural networks 9508369: as such, characterizing them would be the first step for empirical software engineering and automatic program repair researchers.

* Challenge R.3 - Security Verification of Pre-Trained Models. Pre-trained models can be beneficial, as they reduce the computational burden of training complex deep learning models through transferability. However, pre-trained models may be vulnerable to attacks due to the displacement of the dataset and the potential for malicious code or exploitation 9766323,9402124. On the one hand, assessing the security risks of these models still represent a key challenge to face through the definition of software analytics instruments able to provide practitioners with security insights and best practices in selecting the most appropriate pre-trained model to use to avoid security concerns. On the other hand, existing literature 9402124,9766323,9402020,10.1145/3510003.3510191 let also emerge additional challenges targeting the definition of design practices that may allow to develop and retrain security-aware fine-tuned models. Our findings, therefore, suggest that researchers in the field of software analytics, software quality, and verification and validation might contribute to addressing the challenges in this set.

* Challenge R.4 - Resilience to Intellectual Property Theft. Machine learning-enabled systems based on neural networks suffer from data breaches and unauthorized access to sensitive data due to training attacks <cit.>, i.e., malicious attacks on training data. According to the existing literature, only a little knowledge is available on how to secure machine learning models against those types of attacks. Greater attention should be paid to strengthening security protocols secrypt19,9402124. While this mostly calls the attention of researchers in the field of networks and security, the software engineering research community might still contribute through the definition of more security best practices able to protect models from intellectual property theft, especially in edge devices and IoT systems <cit.>.

* Challenge R.5 - Diagnosing the Internal Behavior of Models. Understanding the root cause of anomalous behaviors of deep neural networks and how to fix them represent critical aspects to further elaborate. On the one hand, evaluating the reliability implications of small sets of weights assigned to the network may support practitioners toward an improved understanding of how they work. In this respect, the definition of exploration, interpretability, and explainability methods may lead to substantial advances in terms of diagnosis of failures and misclassification, other than of security concerns  10.1145/3460120.3484818,10.1145/3338906.3338937,10.1145/3180155.3180220,10.1145/3238147.3238187—Ji et al. 10.1145/3533767.3534391 also called for novel local models to uncover the root causes of deep neural network failures. On the other hand, existing literature also pointed out the need for novel assessment metrics that, besides accuracy, can provide actionable insights into the overall robustness of the model 8952197,10.1145/3533767.3534373. As such, our findings suggest multiple avenues for software engineering researchers in the field of fault localization, explainable AI, software metrics, and automatic program repair.

* Challenge R.6 - Optimal Post-Deployment Simulation. While some threats to resiliency might be managed at the development time, additional challenges arise at the deployment stage. In particular, machine learning models could not work as expected when deployed. The problem has been mainly pointed out in the context of automatic speech recognition 10.1145/3533767.3534391,10.1145/3533767.3534408, yet it may affect any kind of machine learning solution. The main challenge is related to properly understanding how the model would work in a real-case scenario. Besides software testing <cit.>, literature identified post-deployment simulations as a complementary instrument that might be worth to further investigate: this refers to the definition of agent-based models that may (i) simulate the environment where the machine learning-enabled system would work and (ii) verify the system against a large variety of simulated inputs. This is, again, something that may catch the attention of researchers in the field of verification and validation and machine learning for software engineering.

* Challenge R.7 - Preserving Privacy in Machine Learning-Enabled Systems. The primary studies identified the challenge of developing privacy-preserving deep neural networks and machine learning systems, which may allow algorithms to run securely on distributed data without compromising sensitive information about the subjects of the data, leaving users with the ability to delete their data at any time
leaving users with the ability to delete their data at any time. This would require the definition of strategies to train models without sacrificing accuracy while ensuring the protection of private data 9676691,10.1145/2810103.2813687. As such, this challenge targets multiple software engineering fields, from software quality to software architecture, which may deepen the current knowledge on data encryption, anonymization techniques, federated learning, and differential privacy.

* Challenge R.8 - Improving the Generalizability of Existing Automated Frameworks. Multiple primary studies identified limitations and challenges related to the automated support provided by existing frameworks. Challenges in this category mainly concern with their generalizability. For instance, Plum 9623049, a recommendation system to prioritize model repair strategies, is limited to the analysis of deep learning models and does not provide support for different learning tasks. Similarly, other approaches 10.1145/3510003.3510231,10.1145/3132747.3132785 can only assist practitioners when developing specific types of neural networks. As such, we also identified technological and empirical challenges for our research community, which is called to develop and experiment with automated solutions by keeping generalizability into account.

Challenges in this category were mainly considered in the domains of `Computer Vision', `Driving Systems', `Mobile Devices', `Edge Devices', and `IoT Devices', i.e., those more intensively threatened by security issues. Nonetheless, we observed a rising interest toward `Automatic Speech Recognition'—this is likely due to the growing research on large language models <cit.>.

Sustainability (S): As for the challenges in this category, we could observe that most of the available literature focused on ethics and fairness, somehow neglecting other perspectives of sustainability. More particularly, we identified seven main challenges connected to algorithmic discrimination, model accountability, fairness metrics, low-quality datasets, energy cost, energy and performance aware trade-offs, and generalizability of existing solutions.

* Challenge S.1 - Dealing with Algorithmic Discrimination. Improving performance, accuracy, and fairness simultaneously is currently the main challenge for researchers 10.1145/3540250.3549093,10.1145/3540250.3549103,10.1145/3540250.3549169,10.1145/3468264.3468537. More specifically, the challenge is to study and mitigate the impact of disparate results, offensive labeling, and uneven algorithmic error rates in data-driven applications. In addition, the primary studies investigated highlighted the relevance of dealing with algorithmic discrimination in the context of the pre-processing phase in an effort to improve the trade-off between fairness and performance in ML software. While the software engineering research community has already contributed to addressing this challenge (e.g., 7961993,10.1145/3338906.3338937), our findings suggest that further research in terms of software analytics and verification and validation might be worthwhile.

* Challenge S.2 - Model Accountability: The term `accountability' refers to the model's ability to provide clear explanations of its decisions, its transparency in how it has been trained and makes decisions, and its ability to allow users to provide feedback or challenge its decisions. The primary studies considered in our work remarked that more effort should be given to this aspect to allow models to be deployed with an eye on ethics and fairness. In this respect, major challenges pertain to the definition of verification and validation instruments through which practitioners can verify how accountable their systems actually are 7961993,pmlr-v81-buolamwini18a. For instance, decision making may represent ideal tools to assess the level of accountability of machine learning models.

* Challenge S.3 - Fairness Analytics. Another critical challenge for social sustainability is related to fairness analytics, that is, the definition of metric toolkits that may support practitioners in diagnosing fairness at the development time. However, fairness should not be considered as a non-functional requirement per se, but rather be part of trade-off analyses. More specifically, existing literature called for novel metrics able to provide insights into the compromise between fairness and accuracy 9402057, fairness and privacy 10.1145/2810103.2813687, and fairness and accountability 7961993. As such, the challenge is represented by the diagnosis and multi-objective optimization of data-driven applications in multiple development stages, from requirements engineering to verification and validation 7961993.

* Challenge S.4 - Improving Sustainability Benchmarks. Current literature pmlr-v81-buolamwini18a also highlighted the lack of low-quality of benchmark datasets that may assist the creation of sustainable machine-learning applications. These datasets should indeed be devised by enabling cross-domain and cross-sectional analysis. On the one hand, application domains focusing on the image processing would require high-quality datasets allowing automated pose, illumination, and expression (PIE) analysis. On the other hand, datasets presenting large and diverse information on minorities should be devised to allow the more effective training of machine learning systems.

* Challenge S.5 - Reducing Energy Cost. One of the most important challenges for deep neural networks is concerned with the reduction of energy and computational costs. Indeed, those systems have considerable energy and financial costs, which lead to increase CO2 emissions and memory consumption in both the training and post-release phases 10.1145/3506695,9590404,10.1145/3510003.3510231,10.1145/3510003.3510221,10.1145/3510003.3510088,10.1145/3551349.3561158. More specifically, energy consumption issues significantly impact edge and IoT devices, which indeed represent critical use cases. The software engineering research community is called to contribute in different manners. First, empirical software engineering research is called to assess the impact of different AI containerization strategies on energy consumption and memory utilization. Second, software quality and software architecture researchers might help address the trade-offs between training accuracy and post-deployment consumption. Last but not least, our findings call for further research on the definition of software engineering practices to develop tiny machine learning solutions.

* Challenge S.6 - Increasing the Practitioner's Awareness of Sustainability. Our literature work identified the challenge of increasing the developer's awareness with respect to sustainability concerns. This issue is particularly relevant when considering energy consumption, which is a critical concern in today's world. As such, the definition of novel strategies and instruments to make practitioners aware of the trade-offs between performance and sustainability represents a key challenge for our research community 10.1145/3510003.3510221,10.1145/3530019.3530025. Once again, more research on tiny machine learning might represent a suitable solution, as it would allow to improve the scalability of models without compromising their overall accuracy. At the same time, the empirical software engineering research community might play a key role by understanding the practitioners' needs and practices, other than tailoring (novel) automated approaches on them.

* Challenge S.7 - Improving the Generalizability of Existing Automated Approaches. The last challenge in this category concerns with the generalizability of existing approaches that support sustainability analysis of machine learning-enabled systems. Multiple primary studies identified limitations and challenges that the research community should further consider. Similarly to R.8, current approaches can only provide support for the development of a limited set of deep neural networks 10.1145/3510003.3510202,10.1145/3510003.3510087, are built using limited datasets 10.1145/3368089.3409697, or are not integrated within MLOps pipelines 10.1145/3468264.3468565. In other terms, the challenge emphasizes the need for novel, more accurate, generalizable, and integrated instruments that may support practitioners throughout the software lifecycle.

Since most of the challenges described pertained to social sustainability, these were investigated in the `Decision Making' and `Healthcare' domains, which are the most demanding in terms of ethics and fairness. However, we observed a growing interest in the `Cloud', `IoT Devices', and `Edge Devices' domains, mostly due to the increasing efforts on energy consumption matters.

Usability (U): Usability is the non-functional requirement for which we could identify fewer challenges. Only a few works targeted this matter, hence suggesting that further research might be worthwhile. Indeed, usability, interpretability, and visualization of both shallow and deep learning models pose significant challenges when it turns to assess quality assurance throughout the software lifecycle. For instance, the lack of interpretability increases the effort required to estimate the soundness of machine learning-enabled systems. Current literature somehow neglected this research angle, especially when considering models operating in critical and evolving scenarios. As such, our findings suggest the urgent need for further research on usability concerns 9451178,10.1145/3338906.3338954,10.1145/3243734.3243792,10.1145/3429444,10.1145/3506695,10.1145/3447548.3467177. Interestingly, the available primary studies did not focus on any specific application domain and proposed general-purpose interpretability models. This further suggests the investigation of domain-specific approaches and methods.

Cross-cutting challenges (C): Besides the challenges pertaining to the individual classes of non-functional requirements, we also identified a set of cross-cutting challenges which are related to the management and assessment of non-functional requirements.

* Challenge C.1 - Context-based Trade-off Identification. Finding a trade-off between multiple non-functional requirements represents a key socio-technical and managerial challenge 10.1145/3540250.3549093,10.1145/3510003.3510163,10.1145/3510003.3510163, 10.1145/2810103.2813687,10.1145/3510003.3510087. On the one hand, machine learning-enabled systems should always preserve accuracy. On the other hand, other non-functional requirements may play a key role depending on the specific context where the system should be employed. As an example, robustness and security might be of paramount importance in the context of driving systems, while fairness might be preferred when dealing with healthcare systems. The identification of the non-functional requirements to preserve may indeed depend on contextual analyses performed at both socio-technical and managerial levels. In the former, novel approaches able to (semi-)automatically mine contextual information that provide practitioners with insights into the non-functional requirements to consider in the specific context might be worthwhile. In the latter, novel software project management methods to assess the significance of non-functional requirements might support the managerial decisions on how to design a machine learning-enabled system. This challenge therefore emphasizes the need for novel methods and instruments to identify the non-functional requirements to simultaneously consider within a given context.

* Challenge C.2 - Prioritizing and Balancing Non-Functional Requirements. A second cross-cutting challenge pertains to the balancing of multiple non-functional requirements. While the trade-off identification may provide practitioners with information on the aspects to take into account while developing a ML-enabled system, this would not be enough to prioritize and balance them. Literature argues the need for novel methods to support the technical management of non-functional requirements in terms of prioritization and balancing of non-functional requirements during the optimization process 10.1145/3510003.3510221,10.1145/3530019.3530025,10.1145/3540250.3549093, 10.1145/3338906.3338954,10.1145/3540250.3549103,10.1145/3551349.3556906. As such, the requirements engineering research community might further explore this matter, by proposing guidelines or automated instruments to (1) prioritize non-functional requirements and (2) find an optimal balance among both correlated and contrasting objectives by means of the analysis of contextual factors and practitioners' preferences.

* Challenge C.3 - Software Analytics for Non-Functional Requirement Assessment. As a last challenge of this category, multiple primary studies advocated the need to empower requirements engineering processes with software analytics instruments able to assess the implications that trade-off choices may have on the the development of ML-enabled systems 10.1145/3510003.3510088,10.1145/3551349.3561158,10.1145/3340531.3411980,10.1145/3447548.3467177,10.1145/3534678.3539145. More particularly, current literature argues the need for software metrics able to inform practitioners of how their requirements engineering decisions may influence both the complexity of the development process and the overall quality of the system being developed. As an example, the definition of strategies to recommend the quality assurance mechanisms to put in place based on the trade-off to meet would be desirable 10.1145/3534678.3539145. On a similar note, researchers claimed that the definition of novel predictive analytics instruments might largely improve requirements engineering processes by enhancing the practitioner's capabilities to assess the impact that trade-off analysis may have on the ML-enabled system throughout the software lifecycle. As a consequence, this challenge may be of the interest of requirements engineering and software analytics research communities, which may jointly collaborate toward an improved understanding of the support required by requirements engineers in practice.

2As a result of RQ$_2$, we elicited more than 20 software engineering challenges targeting both the individual categories of non-functional requirements identified in our systematic synthesis work and the cross-cutting aspects affecting non-functional requirements engineering as a whole. We methodically untangled the challenges plaguing machine learning-enabled systems, providing insights into the specific research fields interested in those challenges. According to our results, we call for comprehensive analyses and approaches that may assess the impact and implications of the management of individual and multiple non-functional requirements.

§.§ RQ$_3$ - What automated approaches are proposed to deal with non-functional requirements of ML-enabled systems?

With our last research question (RQ$_3$), we were interested in surveying the automated approaches to handle non-functional requirements of machine learning-enabled systems proposed so far by researchers. On the one hand, this analysis allowed us to collect information on multiple aspects of those approaches, hence providing insights into the current support available and the related limitations. On the other hand, we could also measure the alignment between the classes of non-functional requirements identified and the approaches available in an effort to understand the extent to which the current approaches fulfil the potential practitioner's needs and, at the same time, what are the limitations of the state of the art and the non-functional requirements that researchers should be more closely pay attention to in future research.

To address our goal, we analyzed the primary studies from three main perspectives. As further discussed in the remainder of the section, all the approaches retrieved were based on machine learning. As such, we first explored how non-functional requirements were measured by the proposed approaches. Second, we analyzed how these machine learning solutions were fed in terms of training datasets and architectural decisions. In the third place, we further deepen our investigation and survey, for each approach, the non-functional requirements addressed and the development phase where it may be applied. By means of these three perspectives, we could basically investigate the design decisions made by researchers when devising machine learning solutions able to address non-functional requirements of machine learning-enabled systems.

§.§.§ On the measurement adopted to analyze non-functional requirements

Non-Functional Requirement Measurement employed by the automated approaches in ML-Enabled System.

Non-Functional Requirement     Measurements approaches

Accuracy     Test set

Efficiency     Nvidia’s PyNVML library to measure latency

Maintainability     Software analytics

Resiliency     Perturbed test set, EvalDNN <cit.> to measure the neuron coverage

Sustainability     Intel Power Gadget, Intel RAPL to measure CPU and GPU energy and CO2. The test set for bias.

Usability     Input sample

As a first point of discussion, we analyzed how non-functional requirements were measured by the automated approaches proposed in the literature. Table <ref> summarizes the measurement approaches adopted for each non-functional requirement class identified in RQ$_1$.

We could first observe that existing approaches primarily used software measurements. More particularly, the non-functional requirements pertaining to `Accuracy', `Efficiency', `Resiliency', `Maintainability', and `Usability' were typically assessed by means of software measurements performed on data. On the contrary, the non-functional requirements pertaining to `Sustainability' were also assessed using hardware components or libraries that enable the computation of hardware-specific properties. In this sense, we may conclude that addressing non-functional requirements of ML-enabled systems may require a mixture of measurement approaches: this is especially relevant for researchers interested in devising multi-objective optimization techniques, who might indeed need to acquire specific hardware components or combine software and hardware measurements to enable a proper measurement of the attributes to optimize.

Going deeper on the computation of non-functional requirements, accuracy was assessed by executing a machine learning model against a test set: this enabled the analysis of (1) true positives and negatives, and (2) false positives and negatives, which were combined to compute the overall accuracy of the model. A similar discussion could be made for the non-functional requirements of the `Resiliency' class: to assess properties like security, reliability, and robustness, the approaches proposed in literature proposed to inject perturbations within the dataset, e.g., altered pixels or rotated images for datasets involving images, to assess the model's responsiveness to these diverse inputs. Alternatively, certain approaches assessed the internal behaviour of the model using the EvalDNN framework <cit.> to evaluate neuron coverage.

Among the non-functional requirements of the `Usability', we could observe that only explainability was actually addressed by the currently available automated approaches. To assess explainability, these approaches run the machine learning model against a test set, generating input samples that measured the impact of each feature on the resulting performance. As for the non-functional requirements within the `Maintainability' class, the approaches adopted software analytics measurements, e.g., by analyzing the training data to identify potential maintainability concerns.

Within 'Sustainability', researchers have been employing both software and hardware measurements, depending on the specific non-functional requirement they had to assess. In particular, fairness was measured by means of the analysis of the predictions made by the machine learning model, i.e., through a software-based approach: the model was first assessed against a test set and, afterwards, software analytics instruments were adopted to estimate the bias of the predictions made. As for energy consumption, the automated approaches adopted methodologies grounded in hardware energy measurement tools, e.g., through the so-called Running Average Power Limit (RAPL), which allows monitoring energy consumption of the CPU chip, attached DRAM and on-chip GPU. With these tools, researchers quantify the carbon emissions through mathematical derivations from analysing the actual emissions released by the machine learning model. Finally, when considering the 'Efficiency' class, the available approaches employed software tools to measure the response latency of forecasts and estimate the response time to get a prediction to output.

§.§.§ On the training decisions behind automated approaches

Overview of the datasets employed by the automated approaches proposed in the literature.

Domain     Non-Functional Requirement     Dataset

Computer Vision
Efficiency     CIFAR-10 - CIFAR-100 <cit.>, MNIST <cit.>, UCF101 <cit.>, ImageNet <cit.>, Dataset X, GTSRB <cit.>, SVHN <cit.>

Maintainability     CelebA <cit.>, UnityEyes <cit.>, CIFAR-10 - CIFAR-100 <cit.>, MNIST <cit.>, MIT Indoor Scenes<cit.>, Stanford 40 Actions <cit.>, Stanford Dogs <cit.>

Resiliency     CIFAR-10 - CIFAR-100 <cit.>, MNIST <cit.>, CelebA <cit.>, UnityEyes <cit.>, UCF101 <cit.>, SVHN <cit.>, GTSRB <cit.>, Youtube Faces <cit.>, ImageNet <cit.>, Fashion MNIST<cit.>, SVHN <cit.>, Credit Scoring System, Driving <cit.>, MIT Indoor Scenes<cit.>,Stanford 40 Actions <cit.>, Stanford Dogs <cit.>

Sustainability     CIFAR-10 - CIFAR-100 <cit.>, MNIST <cit.>, Fashion MNIST<cit.>

Usability     CIFAR-10 - CIFAR-100 <cit.>, MNIST <cit.>, , UCF101 <cit.>, GTSRB <cit.>, Youtube Faces <cit.>, Credit Scoring System

Decision Making
Efficiency     Adult <cit.>, Bank Marketing <cit.>, Statlog <cit.>, Compas1, Mep5, CIFAR-10 - CIFAR-100 <cit.>, MNIST <cit.>

Maintainability     Adult <cit.>,Bank Marketing <cit.>

Adult <cit.>, Bank Marketing <cit.>, Statlog <cit.>, Compas1, Mep5, Heart Disease <cit.>, Breast Cancer Wisconsin (Diagnostic) <cit.>, Depresjon <cit.>, Psykose <cit.>, Violent Crime

Adult <cit.>, Bank Marketing <cit.>, Statlog <cit.>,  Compas1, Mep5, CIFAR-10 - CIFAR-100 <cit.>, MNIST <cit.>, Heart Disease <cit.>

Usability     Adult <cit.>, Bank Marketing <cit.>,Statlog <cit.>, CIFAR-10 - CIFAR-100 <cit.>, MNIST <cit.>, Heart Disease <cit.>

Natural Language Processing
Efficiency     MoleculeNet <cit.>, MultiUN dataset <cit.>, AgNews <cit.>

Maintainability     IMDB reviews <cit.>

MoleculeNet <cit.>, Snips <cit.>, AgNews <cit.>, MultiUN dataset <cit.>, IMDB reviews <cit.>, AgNews <cit.>

Sustainability     MoleculeNet <cit.>, Snips <cit.>, AgNews <cit.>, MultiUN dataset <cit.>

Automatic Speech Recognition
Efficiency     AN4 <cit.>, TIMIT <cit.>, TED-LIUM 2 <cit.>, TED-LIUM 3 <cit.>

Resiliency     AN4 <cit.>, TIMIT <cit.>, TED-LIUM 2 <cit.>, TED-LIUM 3 <cit.>

Anomaly Detection
Maintainability     SICS Geek Lounge6

Resiliency     SICS Geek Lounge6

1 Compas: <https://github.com/propublica/compas-analysis>

2 Titanic: <https://www.kaggle.com/c/titanic/data>

3 Bank Execution: <https://data.world/markmarkoh/executions-since-1977>

4 Fraud Detection: <https://www.kaggle.com/c/frauddetection/data     Marketing Campaigns>

5 Mep: <https://meps.ahrq.gov/mepsweb/data_stats/download_data_files_detail.jsp?cboPufNumber=HC-181>

6 4SICS Geek Lounge: <https://www.netresec.com/?page=PCAP4SICS>

7 BTS prediction: <https://github.com/rdsea/IoTCloudSamples/tree/master/MLUnits/>

Overview of Machine Learning Models employed by the automated approaches proposed in the literature.

Domain     Non-Functional Requirement     Model Architecture

Computer Vision     Efficiency     LSTM, VGG16+LSTM, AlexNet, ResNet18, ResNet-20, ResNet-32, ResNet-56, MobileNet, ShuffleNet, ResNet, VGG16

Maintanability     ResNet18, ResNet-20, ResNet-32, ResNet-56, MobileNet, MobileNetV2, DenseNet, LSTM, VGG16+LSTM, AlexNet

Resiliency      AllConvNet, WideResNet, DenseNet, LSTM, ResNet18, ResNet-20, ResNet-32, ResNet50, MobileNet, MobileNetV2, VGG16+LSTM, AlexNet, VGG11, VGG16, ResNet18, GoogleNet, VGG19, LSTM+BiLSTM, LSTM+GRU, MobileNet, ShuffleNet, ResNet-56, DAVE-2,  Decision Tree, Random Forest, XGBoost, Udacity, Autumn, Chauffeur, Rwightman

Sustainability     LSTM+BiLSTM, LSTM+GRU

Usability     LSTM, VGG16+LSTM, VGG11, VGG16, ResNet18, GoogleNet, Decision Tree, Random Forest, XGBoost

Decision Making     Efficiency     ResNet-18, Naive Model, Sample Clustering, Logistic Regression, Support Vector Machine, Random Forest

Resiliency      Distributed: Privacy-Preserving Distributed Extremely Randomized Trees, ID3, Centralized: ERT, Random Forest, XGBoost, Decision Tree, Linear SVM, Logistic Regression, Support Vector Machine, Fair SVM, Ensemble Voting Classifier

Sustainability     ResNet-18, Logistic Regression, Random Forest, Support Vector Machine, Decision Tree, Discriminant Analysis, Naive Model, Sample Clustering, Fair SVM, Ensemble Voting Classifier, KMeans clustering, Agglomerative clustering, Spectral clustering

Usability     ResNet-18, Logistic Regression, Random Forest, Support Vector Machine, Decision Tree, Discriminant Analysis, Naive Model, Sample Clustering

Natural Language Processing     Efficiency     LSTM, H-NLP, AllenAI, T5, BERT, RoBERTa

Maintanability     Bert, RoBERTa

Resiliency      LSTM, BiLSTM+GRU, LSTM+BiLSTM, H-NLP, AllenAI, T5, Bert, RoBERTa

Sustainability     BiLSTM+GRU, LSTM+BiLSTM, H-NLP, AllenAI, T5

Usability     LSTM

Automatic Speech Recognition     Efficiency     LSTM, RNN-RNN, Transformer-AED, Conformer-AED

Resiliency      Bi-LSTM, RNN-RNN, LSTM, Transformer-AED, Conformer-AED

Tables <ref> and <ref> report the results achieved when considering how the automated approaches considered were fed. More particularly, Table <ref> overviews the datasets retrieved in our study. The datasets are first grouped according to the domain in which the primary study focused. The domains are those resulting from RQ$_1$. The datasets are then further grouped by class of non-functional requirements they can be experiment with. Finally, the last column of Table <ref> provides names and references of the datasets—the reader may observe the missing reference to the `Preductive Healthcare', `Violent Crime',`Dataset X', `Credit Scoring System', and `AN4'   datasets: they are currently not accessible or have restricted access. At the same time, Table <ref> maps the model architectures employed by the automated approaches, namely the machine learning algorithms exploited when building them.