[datatype=bibtex] [fieldset=issn, null] [fieldset=month, null]
[fieldset=notes, null] [fieldset=urldate, null]

# Focus Areas, Themes, and Objectives of Non-Functional Requirements in
DevOps: A Systematic Mapping Study

Philipp Haindl  Software Competence Center Hagenberg, Austria  Department of
Business Informatics - Software Engineering, Johannes Kepler University Linz,
Austria  Reinhold Plösch  Department of Business Informatics - Software
Engineering, Johannes Kepler University Linz, Austria

###### Abstract

Context: Software non-functional requirements address a multitude of
objectives, expectations, and even liabilities that must be considered during
development and operation. Typically, these non-functional requirements
originate from different domains and their concrete scope, notion, and
demarcation to functional requirements is often ambiguous. Objective: In this
study we seek to categorize and analyze relevant work related to software
engineering in a DevOps context in order to clarify the different focus areas,
themes, and objectives underlying non-functional requirements and also to
identify future research directions in this field. Method: We conducted a
systematic mapping study, including 142 selected primary studies, extracted
the focus areas, and synthesized the themes and objectives of the described
NFRs. In order to examine non-engineering-focused studies related to non-
functional requirements in DevOps, we conducted a backward snowballing step
and additionally included 17 primary studies. Results: Our analysis revealed 7
recurrent focus areas and 41 themes that characterize NFRs in DevOps, along
with typical objectives for these themes. Overall, the focus areas and themes
of NFRs in DevOps are very diverse and reflect the different perspectives
required to align software engineering with technical quality, business,
compliance, and organizational considerations. Conclusion: The lack of
methodological support for specifying, measuring, and evaluating fulfillment
of these NFRs in DevOps-driven projects offers ample opportunities for future
research in this field. Particularly, there is a need for empirically
validated approaches for operationalizing non-engineering-focused objectives
of software.

_Keywords_ Non-functional Requirements $\cdot$ Software Quality Attributes
$\cdot$ DevOps

## Remark

This paper is an addition to our peer-reviewed publication in [1] and contains
a more elaborate presentation of our findings. When citing our work, please
always cite the peer-reviewed publication; citing this technical report should
always be optional for cases where you explicitly reference aspects that were
not published in the peer-reviewed publication.

##  1 Introduction

The common notion of non-functional requirements (NFRs) of software, such as
e.g., maintainability or performance efficiency, primarily targets
_engineering-focused_ objectives of software. As such they disregard the also
_non-engineering-focused_ objectives pursued with software, e.g., that it is
frequently used by customers, generates revenue aligned with the business
model, and that its monitored usage data allows to ideate new features for the
customers. Due to the lack of awareness for these non-engineering-focused
NFRs, they are often only vaguely elicited and balanced with functional
requirements [2, 3, 4, 5]. This in turn disguises business, staffing, and
organizational considerations that also have an impact on decisions in
software projects. Also, it makes it difficult for software engineers to
appropriately react to deviations of these NFRs in DevOps. Due to the absence
of a comprehensive classification scheme that takes into account the
heterogeneity of domains having an impact on NFRs, it is cumbersome to
properly approach them in software specification documents and meet the
different stakeholders’ objectives. As an example, the objectives towards
exchangeability of software developers or transaction-cost monitoring in
DevOps exceed the primarily engineering-focused understanding of NFRs. These
cases of NFRs are primarily applicable in DevOps as they require a thorough
indentation between the development and operation perspective of software.

Specifically the evolution of BizDevOps [6, 7] emphasizes the necessity for
authors to more intensively integrate business and other non-engineering-
focused NFRs in the DevOps context. Several works have stressed the required
alignment between business processes, strategic objectives, company culture,
and engineering decisions with DevOps [8, 9, 10, 11]. In addition, the
ubiquity and volume of data accruing in DevOps provide ample methods for
evaluating fulfillment of NFRs to continuously improve the software. The
absence of a classification scheme for NFRs in DevOps and the thereof pursued
objectives also hinder the effective formulation of respective measures that
can then be acquired and assessed in the DevOps cycle.

###  1.1 Problem statement and motivation

To increase the value of DevOps in practice and assure a better alignment with
non-engineering-focused objectives in software projects, a comprehensive
classification of NFRs, typical themes, pursued objectives and implementation
practices in DevOps is required. In order to examine the handling of NFRs in
DevOps we conducted a systematic mapping study in combination with backward
snowballing that included 142 primary studies. The following paper presents
the findings from this mapping study.

The focus of this research was on examining

  * •

which NFRs are relevant and typically handled in DevOps, and

  * •

how these NFRs can be classified by focus areas and subordinate themes, and

  * •

what objectives are pursued with these NFRs;

We also analyzed the publication channels, contributions and research facets
as well as the frequency distribution over the years. In order to get a solid
understanding of the current state of practice of NFRs in DevOps, we followed
a snowballing procedure and selected further relevant papers for inclusion in
our analysis. In the context of this paper we refer to DevOps as "a set of
practices intended to reduce the time between committing a change to a system
and the change being placed into normal production while ensuring quality"
[12]. The introduction and practice of DevOps shall however not be seen as an
isolated software engineering activity, but also requires organizational
changes. In many cases this organizational change is much more fundamental
than the changes in the software engineering process, as it additionally
requires significant adaption to other processes, e.g., change management
processes [13].

Our work seeks to augment the common understanding of NFRs by contributing new
facets about the different scopes and objectives behind NFRs, specially in
DevOps. The different perception and understanding of NFRs also becomes
apparent when skimming the definitions of NFRs from other publications. In one
of the first publications in this domain [14], economic aspects of software
are mentioned in the same turn as purely engineering-related aspects such as
reliability, usability or safety. Other authors [15, 16, 17] reflect only on
the structural and behavioral aspects of NFRs and hence also approach them
solely from a technical perspective. In contrast, recent works in this context
[18, 19] have a broader notion of NFRs. These works stress that also
"…environmental factors, such as legislation related to the business branch
and compliance, may introduce vital requirements…" and that "…user needs and
other stakeholders should not be forgotten, either.". To summarize, there
still is no clear demarcation between functional and non-functional
requirements [20, 16] and how to approach and evaluate non-engineering-focused
NFRs that affect the design, implementation, and operation of software.

###  1.2 Contribution

Our study shows that there are multiple perspectives on why and how to
approach NFRs in DevOps. These perspectives are reflected by different points
of focus that are typically (and often tacitly) taken in software projects and
range from engineering-related qualities, e.g., static or dynamic code
analysis, to non-engineering-related qualities addressing business, legal, or
organizational requirements. As an example, these requirements address how to
balance economic requirements of the software, the mapping of software
artefacts and their contribution to revenue streams, or motivational factors
influencing the development and operation of software.

Resulting from our analysis of 142 relevant papers in this systematic mapping
study, we contribute an analysis and thematic classification of the current
state of research in this field. The analysis of the studies takes into
account the research and contribution facets, publication channels, and the
publication frequency of the studies over time. Complementary, the NFRs in
DevOps described in the studies are classified by focus area, typical themes,
and pursued objectives. In particular, our work seeks to extend the primarily
engineering-focused understanding of NFRs by showing their heterogeneity in
the analyzed studies and their relevance specially in DevOps.

###  1.3 Outline

The remainder of the paper is organized as follows: In Section 2 we provide an
overview of work related to our study. Section 3 presents the research
methodology, introducing the research goals and questions and how each method
has been applied in this literature study. The analysis of the study
population showing the different facets of the studies, publication channels,
and the publication frequency over time is elaborated in Section 4. Following,
the focus areas of NFRs in DevOps are presented in Section 5, before the
themes and objectives related to these NFR focus areas in DevOps are portrayed
in Section 6. The findings from our study are discussed in Section 7, followed
by outlining the threats to the validity of our results in Section 8. Finally,
we conclude our paper in Section 9 and attach the details of the used
classification schema in A and the systematic map of all analyzed primary
studies in B.

##  2 Related work

In this section we elaborate on related work in the context of handling NFRs
in DevOps, the necessity for this study and how we demarcate our work from
other authors. Basically, 3 streams of research can be identified:

The first stream of related research deals with the scope and notion of NFRs
towards software. Particularly, this tackles the question where to draw the
demarcation between non-functional and functional requirements and whether the
notion of NFRs also covers non-engineering-related concerns, e.g., business,
organizational or legal concerns. These concerns also affect the design,
development, and operation of the software [20, 3, 16] and may also become
manifest in NFRs. The differentiation between non-functional and functional
requirements is not only prevalent in research, but also affects the
elicitation, documentation, and validation of NFRs in software projects.
Eckhardt et al. [17] analyzed 530 NFRs of 11 industrial requirements
specifications regarding the extent these NFRs describe system behavior. They
concluded that most many so-called NFRs actually are not non-functional, but
instead could be treated as functional requirements. However, the focus of the
analyzed NFRs is primarily engineering-oriented and leaves out non-
engineering-related concerns which may also become manifest in NFRs. Mairiza
et al. [21] examined 182 sources of information elaborating NFRs, ranging from
academic articles to technical reports and white papers, and categorized them
depending on their type of being either business, external, development or
quality related. For each extracted NFR from these sources, they analyzed
whether it adheres to a common definition, has concrete measurable attributes
and a concrete measurement methodology defined for it. They concluded that
only around 20% of NFRs typically arising in software projects are commonly
defined covering attributes and measurement methodology. The remaining 26% of
NFRs at least have a definition, but for the majority of over 53% of common
NFRs neither their attributes nor their measurement methodology is defined,
leaving them factually ignored in the development process. Also, the authors
of this work conclude that the common notion of NFRs leaves out important non-
engineering-related aspects that have an impact specially on the development
of the software and specifically, that further research addressing scope,
notion, and measurement of NFRs shall be done.

The second stream of relevant works comprises systematic literature reviews
and mapping studies. Hasan et al. [22] argue that NFRs are often ignored,
inadequately specified, and rarely treated as first-class elements such as
functional requirements. In the frame of a systematic literature review (SLR)
the authors analyzed 92 describing approaches for handling NFRs in the
software development lifecycle, to examine the documentation approaches and
scopes of NFRs in software projects. The focus of their work is not on the
scope of NFRs itself, but primarily on approaches for their elicitation,
documentation, and handling. The authors categorize these approaches into
goal-, aspect-, or pattern-oriented NFR elicitation approaches. While goal-
oriented NFR elicitation techniques result in higher preciseness and less
ambiguity of the NFR specification, the authors also stress that the plethora
of NFR elicitation techniques is due to the lack of a thorough taxonomy for
NFRs. They also underline that despite the undisputed advantages of these
techniques, most of them can only be used by domain experts, which complicates
NFR elicitation and specification in software projects. Ouhbi et al. [23]
performed a systematic mapping study based on 51 primary studies, mainly to
analyze the population of studies dealing with software quality elicitation
and specification. In their study, the authors solely take into account NFRs
that adhere to the ISO/IEC 25010 specification. Their study summarizes that
NFRs are mainly validated during development and that data from software
operation are often disregarded or not used effectively for NFR validation.
Also, the authors conclude that there is a need for more studies focusing on
the validation of NFRs throughout the development lifecycle.

The last stream of relevant research examines the handling of NFRs in the
DevOps context. In a qualitative multiple-case study, Riungu-Kalliosaari et
al. [24] interviewed practitioners about the challenges and benefits of DevOps
adoption. In the interviews, the respondents mentioned that NFRs in DevOps
primarily focus on product quality, thus neglecting the quality of the
software development, delivery, and operation process. This aligns with the
work of Senapathi et al. [9] who conducted an exploratory case study
interviewing several software engineers to explore the challenges linked with
adopting DevOps practices. In the interviews the experts responded that in
practice they are confronted with non-engineering-focused NFRs such as e.g.,
the alignment of the software with revenue streams or feature ideation from
monitored user behavior. These non-engineering-focused NFRs are not fully
covered by the primarily engineering-oriented notion of NFRs and hence are
often neglected in DevOps. Though, similar to engineering-focused NFRs such as
e.g., maintainability of the source code, they have a direct effect on whether
the objectives pursued with software can be fulfilled and with what efforts
and costs.

It can be summarized that multiple related works stress the solely technical
notion of NFRs and suggest to rethink their scope to more effectively specify,
measure, and evaluate non-engineering-focused NFRs, e.g., business,
organizational, human, and legal requirements. Particularly in DevOps with a
multitude of data originating from different systems, a thorough catalogue of
NFRs along with their focus, typical themes, subordinate objectives, and
measures is the prerequisite for more effectively approaching NFRs. The
presented systematic mapping study aims to close this gap through providing a
detailed classification of NFRs in DevOps – by focus areas, themes, and
pursued objectives.

##  3 Research methodology

The research presented in this paper resulted from a systematic mapping study
conducted over the period of 9 months (December 2018 to August 2019). There is
an important distinction between systematic literature reviews and mapping
studies: Systematic literature reviews (SLRs) intend to "identify best
practice with respect to specific procedures, technologies, methods or tools
by aggregating information from comparative studies" while the result of a
systematic mapping studies is a "classification and thematic analysis
literature on a software engineering topic" [25]. As already outlined in
Section 2, the notion and scope of NFRs still remain vague in academic works
and industry, specifically whether non-engineering-focused concerns are also
covered by this term. Therefore, in our mapping study we augmented the set of
analyzed papers through a snowballing procedure [26] to also capture primary
studies that elaborate on non-engineering-focused NFRs in DevOps. For the sake
of transparency and reproducibility of our study, we aimed to keep the number
of additionally included papers small.

###  3.1 Research goal and questions

The overall research goal for this study can be described through the
following objectives: to (a) analyze the publication population in this field
and particularly the research and contribution facets; (b) provide a
comprehensive catalogue of engineering- as well as non-engineering-focused
NFRs; (c) nurture a broader understanding of the different focus areas,
themes, and subordinate objectives of NFRs in DevOps; and (d) outline
directions for future research. Below we present the research questions that
were derived from these objectives.

  * •

What is the current state of research pertaining to NFRs in the context of
DevOps? (RQ1)

We took common bibliometric perspectives onto the current state of research
pertaining to this field and refined them into corresponding research
questions.

    * –

What is the distribution of research facets in studies related to NFRs in
DevOps? (RQ1.1)

    * –

What is the distribution of contribution facets in these studies? (RQ1.2)

    * –

What are the publication channels and venues used to publish related studies?
(RQ1.3)

    * –

How did the publication frequency of related studies evolve over time? (RQ1.4)

In RQ1.1 and RQ1.2 the perspectives are on the research and contribution
facets of the works respectively. Complementary, RQ1.3 examines the typical
publication channels and venues of works in the field of NFRs in DevOps. The
analysis of the respective publication population is finally supplemented by
RQ1.4, which investigates how the publication frequency of the studies evolved
over time.

  * •

How can NFRs typically relevant in the context of DevOps be characterized?
(RQ2)

The focus of RQ2 is on carving out which NFRs can be discovered recurrently
among the different studies and are typically important in DevOps. Therefore,
the NFRs described in the studies are categorized by their conjoint focus
area, i.e., their spanning characteristics and objectives.

  * •

Which themes and objectives are associated with each focus area of NFRs in
DevOps? (RQ3)

This question analyzes representative themes and objectives that are pursued
with each NFR focus area. If possible, we present them along with tangible
practical examples and associated challenges.

Based on these research questions we defined the data to be extracted from the
primary studies (cf. Section 3.6).

###  3.2 Research process and steps

The overall research design of our mapping study is depicted in Figure 1. We
followed the process as suggested by Petersen et al. [27] but extended it by a
snowballing step (step 4, Figure 1). The process itself was iterative and we
repeated individual steps the more we learned about the data available in the
primary studies and how we could integrate them in our study. Also, we refined
the research questions with increasing knowledge about NFRs, DevOps, and the
challenges that are associated with the combination of these two topics.

Below we describe the design and execution of the systematic mapping study in
a sequential manner: The need for the study emerged from our related
activities in this field [28, 29] pertaining to the extension of an
operational software quality model to evaluate the fulfillment of NFRs on the
level of software features, particularly in DevOps. The research questions
(step 1, Figure 1) are elaborated in Section 3.1. They define the concrete
review scope of the systematic mapping study. Accordingly, the review scope
also influences the search string used to retrieve representative papers for
the topic of interest. We piloted the search string several times until we
ended up with a clear definition of it, which repeatedly returned the most
stable and complete set of relevant papers for this topic.

Figure 1: Research process and steps of the mapping study, modified from
Petersen et al. [27].

Consequently, we searched for publications on selected databases (cf. Section
3.3) using predefined search strings (step 2, Figure 1) following the PICO
schema [30]. The concrete search strategy, the database-dependent search
strings, and the rationales behind the composites of the search strings are
described in Section 3.3. From all papers we excluded duplicate, unrelated,
and non-english works upfront (exclusion criteria 1-2). Then we screened these
candidate papers thoroughly (step 3, Figure 1) and again selected only papers
according to the predefined inclusion and exclusion criteria (tentative
papers). The applied inclusion and exclusion criteria are detailed in Section
3.4.

Several authors emphasize that the scope and notion of NFRs are still vague in
academic literature [20, 3, 16]. To address this limitation and to capture
further papers contributing to a broader notion of NFRs, we subjected the
tentative papers (step 3, Figure 1) a backward-snowballing procedure and
included selected cited papers (step 4, Figure 1) into the final paper set
(relevant papers) for this mapping study. The snowballing procedure [26] is
elaborated in Section 3.5. Consequently, we extracted recurrent keywords and
key themes from the abstracts of the papers to develop and repeatedly refined
our classification schema (step 5, Figure 1). Resulting from this step is a
classification scheme that particularly regards the focus areas of NFRs in
DevOps, which is the prerequisite for the systematic map as one outcome of the
mapping study. Then we analyzed all papers to gain a thorough understanding
about these NFRs (step 6, Figure 1) to synthesize the focus areas, themes, and
objectives associated with NFRs in DevOps. In the last step we used the
classification scheme to extract the metadata, map the papers according to
predefined characteristics, e.g., research and contribution facets, (step 7,
Figure 1). Eventually we built the systematic map to illustrate the
characteristics of the analyzed papers, the extracted focus areas, themes, and
the objectives of NFRs in DevOps.

###  3.3 Search strategy and databases

We conducted several experimental searches from December 2018 to January 2019
to gain a better understanding about the scope of research available on the
topic and the terms that are most frequently in the respective works. For the
experimental searches we used the pilot search string "*DevOps" and ("non-
functional" OR "quality")*" on title, keywords, and abstract and limited the
searches to the IEEE database. Based on this search string, we retrieved 69
papers of which we excluded 24 papers just by reading title or abstract.
Subsequently, we read the remaining 45 papers in detail and discovered that
some of them reflected on quality and NFRs in DevOps derivatives, e.g.,
BizDevOps or Data-Driven DevOps. Thus, to capture a broader notion of the term
"DevOps" in these works, we adjusted the search strings accordingly through
wildcards. When developing the search string, we followed the suggestions by
Kitchenham and Charters [30] and composed the search string of several parts,
representing either the population (of the studied phenomenon) or the
intervention to it. Table 1 shows the parts of the final search string along
with their rationale for including them.

|  Search term |  Rationale
---|---|---
Population |  *DevOps |  Studies discussing DevOps principles and practices. The wildcard operator is used to also match related terms, e.g., in the context of "BizDevOps".
Intervention |  qualit* |  Studies discussing quality in the context of DevOps (e.g., "quality", "qualities").
| OR
|  non-functional* |  Studies discussing non-functional requirements and similar terms in the context of DevOps (e.g., "non-functional attributes", "non-functional requirements", "non-functional concerns").
| OR
|  NFR* |  Studies discussing non-functional requirements using the abbreviation "NFR".
Table 1: Population, intervention, and search string keywords.

During the pilot study we also discovered that though most works share the
common technical notion of NFRs, an ample amount of works elaborates on non-
engineering-focused NFRs. These works intensively contribute to a broader
understanding of NFRs and their objectives in general, which was also
described in Section 1.1 as research motivation for conducting this study.
When analyzing the references of these papers, we also noticed that they often
referred to other relevant papers contributing to a better understanding of
the different types of NFRs. Thus, we decided to subject these primary studies
to a snowballing procedure as described in Section 3.5 and include their
backward references if they meet the defined inclusion requirements [26].

The databases for searching the primary studies were selected considering
their coverage of the software engineering literature and are shown along with
the respective search string, filter, and retrieved paper count in Table 2. In
order to regard the individual syntax of each database and assure we retrieve
the complete set of relevant works from each database, we adapted the search
string accordingly.

Database |  Search string |  Filter |  Papers
---|---|---|---
IEEE |  ("*DevOps") AND ("qualit*" OR "non-functional*" OR "NFR*") |  Only conference papers and journal articles |  76
Springer |  *DevOps ("qualit*" OR "non-functional*" OR "NFR*") |  Only conference papers and journal articles in English in the following subject areas: computer science, software engineering, and information systems applications |  82
ACM Digital Library |  +("*DevOps") +("qualit*" "non-functional*" "NFR*")) |  None |  15
Wiley |  ("*DevOps") AND ("qualit*" OR "non-functional*" OR "NFR*") |  None |  11
Science Direct |  DevOps AND ("qualit" OR "non-functional*" OR "NFR") |  None |  45
Total |  |  |  229
Table 2: Selected databases and retrieved papers.

Whenever possible we also defined filters to narrow the result set according
to the primary study selection criteria described in Section 3.4. We did not
utilize an additional time filter as we were interested in the full time
spectrum of the studies. The top-right column of the table shows the number of
retrieved papers from each database, totaling to 229 papers (up to and
including July 2019).

###  3.4 Primary study selection criteria

In Figure 1, we illustrate the screening procedure, the numbers of retrieved
papers from each database, and the numbers of included and excluded papers
during study selection. The selection of the primary studies was performed by
the two co-authors with one researcher checking the retrieved 229 primary
studies prior to the screening (step 2, Figure 1). This initial review
comprised only title, author, and publication channel of the papers.
Inappropriate or duplicate papers were excluded upfront, resulting in 70 of
the 229 studies being excluded. To mitigate study selection bias, we discussed
disagreements of upfront excluded papers until consensus was reached.
Consequently, the remaining 159 primary studies were reviewed by both
researchers against predefined inclusion/exclusion criteria (step 3, Figure
1). Both researchers reviewed all studies individually, resulting in 23
disagreements regarding the selection of these studies. Again, we discussed
the selection of these studies in detail until we agreed on a final set for
the selection. Below we outline the applied inclusion and exclusion criteria.

Inclusion criteria: Works such as books, book chapters, peer-reviewed studies
published in journals and conferences, and workshop papers reflecting,
investigating, or reporting on

  1. 1.

Elicitation, specification, operationalization, and monitoring of NFRs in
DevOps

  2. 2.

Fulfillment evaluation of NFRs or quality aspects in DevOps

  3. 3.

Different engineering strategies and their effect on software quality and NFRs
in DevOps

  4. 4.

The integration of technical and business objectives and their handling in
DevOps

  5. 5.

Benefits, challenges, and limitations of DevOps and their effect on software
quality

Exclusion criteria:

  1. 1.

Duplicate articles

  2. 2.

Works not written in English

  3. 3.

PhD or Master Theses

  4. 4.

Technical reports or white papers

  5. 5.

Works that not clearly discuss DevOps and NFRs in the frame of software
engineering

Though they are not peer-reviewed works, we also included books and book
chapters retrieved from the academic databases in the initial screening.
However, we only regarded these works if they could be attributed to the
common body of knowledge or have at least one author with a strong and
traceable academic record. In total we excluded 44 primary studies, the
majority of the excluded papers being too generic or non-relevant to the
studied topic. Our review resulted in 115 primary studies being selected for
the subsequent snowballing procedure. Finally, we included additional 27
publications (of which 13 are books and book chapters) referenced by the 115
primary studies, which totals to 142 publications finally selected for the
mapping study.

###  3.5 Snowballing procedure

As we discovered that also the references retrieved through the systematic
search contain valuable information for understanding the notion and scope of
NFRs, we decided to also include a backward snowballing procedure. For the
sake of transparency and repeatability of our study, we paid attention on the
number of papers added through snowballing. Overall, we strived to keeping the
number of thereby added papers as little as possible, but also as extensive as
necessary to contribute to a broader notion specially of non-engineering-
focused NFRs in DevOps.

The snowballing procedure was done by two researchers conjointly, with each
one reviewing each paper separately and deciding about the further handling of
each paper again conjointly. If there were differences about how to proceed
with a paper, we repeated the analysis of the respective paper and discussed
until consensus was reached.

We followed the process as outlined by Wohlin [26] and selected the 115 papers
remaining after exclusion based on abstract, introduction, or conclusion (cf.
Figure 1) as the tentative start set for the snowballing procedure. Thus, this
set originated from the structured search on 5 academic databases and the
inclusion and exclusion of papers adhered to concrete and predefined criteria
described in the previous section. In total, we conducted 4 iterations of the
snowballing process and thereby analyzed each paper in its entirety and not
only selected parts of it. The reason for that being to evaluate if the NFRs
described in an analyzed paper do require additional complementary
information, such as most likely in the case of non-engineering-focused NFRs,
or if the described NFRs rather fit to the common and technical notion of
NFRs, thus not requiring any further elaboration.

As suggested by Wohlin [26], when deciding whether to include a referenced
paper, we specially regarded the place and context in which it was cited in
the original paper, as this might give additional hints about its focus. After
selecting a referenced paper as a candidate for inclusion, we studied it in
detail and evaluated, whether it actually contributes to broadening the notion
of NFRs in DevOps. Specifically, we examined which properties (cf. Table 3,
P5-P7) we can extract from a study and thus how it could contribute to
answering the research questions more thoroughly. We only included papers of
which we could extract at least two properties. Contrarily, we excluded all
papers from being further regarded in the snowballing process if they were
either too generic regarding their NFRs scope, did not provide new insights
into this topic, or because they solely elaborated on basic common knowledge.
Also, we recorded which papers we have already examined and the rationale for
either inclusion or exclusion from snowballing.

Starting with the set of papers retrieved from the structured search we
conducted the backward snowballing process 4 times, resulting in the following
statistics:

  * •

Start set: Out of the references of 115 papers originating from the structured
literature search, we selected 38 candidates of which 9 papers were included,
i.e., efficiency = 9/38 = 23.7%.

  * •

Iteration 1: 26 candidates originating from snowballing from start set and 7
papers were included, i.e., efficiency = 7/26 = 22.2%.

  * •

Iteration 2: 35 candidates originating from snowballing from iteration 1 and 9
papers were included, i.e., efficiency = 9/35 = 25.7%.

  * •

Iteration 3: 12 candidates originating from snowballing from iteration 2 and 2
papers were included, i.e., efficiency = 2/12 = 16.7%.

  * •

Iteration 4: 15 candidates were examined from iteration 3, but no papers were
further included, i.e., efficiency equals to 0%.

We stopped the snowballing procedure after 4 iterations as we did not find any
further papers being relevant for inclusion in the final set. The overall
efficiency of our snowballing procedure, that is calculated on all candidates,
equals to (9+7+9+2+0)/(38+26+35+12+15) = 27/126 = 21.4%. During the
snowballing process we also noticed that each candidate paper must be analyzed
thoroughly, as an analysis spanning just the title and abstract might often be
misleading. Also, we observed that the additional snowballing procedure in
many cases led to relevant papers of other domains being included, e.g., from
the business, product and strategic management domain, which we would not have
without this step. It shall be noted that actually those papers helped us
gaining a better understanding about the scope and notion, particularly, of
non-engineering-focused NFRs in the software lifecycle.

###  3.6 Data extraction

In the next step, we extracted the predefined properties shown in Table 3 from
each selected primary study. This task was done by each researcher separately
using a tool111MAXQDA Plus, https://www.maxqda.com/ for qualitative text
analysis. Overall, we extracted 2 categories of data that also map with the
research questions: properties P1-P5 focus on the analysis of the publication
population, properties P6-P8 capture the overall focus, themes, and objectives
of the NFRs described in a primary study.

ID |  Property |  Research question
---|---|---
P1 |  Research facet |  RQ1.1
P2 |  Contribution facet |  RQ1.2
P3 |  Publication channel |  RQ1.3
P4 |  Publication venue |  RQ1.4
P5 |  Publication year |  RQ1.4
P6 |  Focus area of NFRs in DevOps |  RQ2
P7 |  Themes associated with NFR focus area in DevOps |  RQ3
P8 |  Objectives associated with theme of NFR focus area |  RQ4
Table 3: Data extraction form.

In the following we describe the data extraction for each property from the
primary studies.

####  3.6.1 Primary study properties (P1–P5):

We extracted 4 properties for answering RQ1.1-RQ1.4, which are briefly
described below. The detailed description of the classification schemes and
these properties is given in A, Table LABEL:tab:classification_scheme_facets.

  1. 1.

Research facet (P1): Categorizes the studies by applied research design, i.e.,
how the statements presented in the studies have been evaluated. We used
existing classification schemes [31, 27] and distinguished between experience
reports, evaluation/validation research, philosophical/opinion papers, and
solution proposals.

  2. 2.

Contribution facet (P2): Maps the different outcomes of the studies. Again, we
followed existing classification schemes [32, 33] and defined 6 categories:
model, method, theory, framework, guideline, and lessons learned.

  3. 3.

Publication channel (P3), venue (P4), and year (P5): Captures the publication
channels, venues, and years of the studies.

####  3.6.2 Focus areas, themes, and objectives of NFRs in DevOps (P6–P8):

To examine the focus areas and objectives associated with NFRs in DevOps, we
extracted 3 properties from the studies which help answering RQ2 and RQ3.

  1. 1.

General focus area of NFRs in DevOps (P6): Summarizes the general focus of
NFRs in DevOps described in the studies. To facilitate the inductive coding
necessary to capture the qualitative nature of this property, we setup 7
different focus areas of NFRs ab initio: customer, development, operation,
governance, quality, organization, and business. An elaborate definition of
these focus areas is given in A, Table
LABEL:tab:classification_scheme_focus_areas. We derived these focus areas from
our pilot study when developing the search strings (cf. Section 3.2) and
encountered them recurrently among these papers.

  2. 2.

Themes (P7) and objectives (P8) associated with focus areas of NFRs in DevOps:
We introduced the concept "objectives" to capture the underlying goals and
motivations tacitly associated with NFRs in DevOps and grouped them by
spanning theme. In most studies, these objectives are either expressed
vaguely, outside the research scope of the study, or not clearly distinguished
from functional requirements [16]. If the objective was clearly stated in a
study, we codified the respective statements. Otherwise we codified the origin
and context that an NFR was mentioned in or the stakeholder(s) who explicated
an NFR.

###  3.7 Data analysis and interpretation

We used descriptive statistics to analyze the data relating to RQ1.1-RQ1.4.
Particularly, we analyzed these data regarding the frequencies of research and
contribution facets, publication channels, and years. In addition, we used
logarithmic regression for trend analysis of the publication years. The
analysis of the respective data was done by two researchers separately.

For analyzing the qualitative data relating to RQ2 we used thematic synthesis
[34]. The classification schema for analyzing the extracted properties was
derived from the pilot study during developing the search strings (cf. Section
3.2). Thus, this classification scheme was available right at the beginning of
the data analysis and not developed iteratively. We solely used inductive
coding for analyzing the data of this research question. The main objective of
this analysis was to match the codings, which marked the relevant passages in
the text of the studies, to the given NFR focus areas. We however revised this
classification schema regularly to assure that the labeling of the NFR focus
areas suitably describe the related studies. The analysis of the studies
according to the given classification schema was done by two researchers
separately, followed by a final review of this classification by all
researchers conjointly.

In contrast, for analyzing the data related to RQ3 to identify the themes and
objectives associated with NFRs in DevOps, we used inductive and deductive
coding. First, we labelled the relevant text fragments in the studies using an
inductive approach and derived descriptive codes on the level of individual
studies. Then, after labelling all studies this manner, we iteratively derived
more generic codes that span and suitably describe the more specific codes.
For example, we encountered text fragments in the studies that reflect on the
use of certain versions of libraries, compare advantages of certain frameworks
or elaborate on specific programming patterns to assure compatibility with a
framework. In this case we synthesized these text fragments through the code
framework compliance. After we had developed an initial list of codes, we
revised each code to check whether it suitably matches the content of the
associated study and also that it doesn’t leave out important semantic
subtleties of the encoded text fragments. When reviewing this initial list of
codes, we iteratively refined codes and reviewed if the new codes better fit
to the referred text fragments. We repeated this step multiple times until we
finally agreed upon a consolidated list of codes.

Specifically, when extracting the themes and objectives described in the
studies, we noticed that many studies cannot be attributed to a single focus
area. Instead, they describe multiple focus areas, themes and thus also
different objectives underlying NFRs in DevOps. In these cases, if relevant
for answering the research questions, we separately encoded all described
focus areas, themes, and objectives in a single study.

##  4 Overview of the state-state-of-the-art: NFRs in DevOps (RQ1)

After reviewing 229 primary studies retrieved from the academic databases (cf.
Figure 1) and an additional snowballing step, we selected 142 primary studies
for this mapping study. In this section we present our findings from examining
the bibliometrics of the study population, such as the research and
contribution facets, used publication channels and publication frequency using
descriptive statistics. The primary studies were classified according to the
classification schemes, as defined in A, Tables
LABEL:tab:classification_scheme_facets and
LABEL:tab:classification_scheme_focus_areas. While in this section we present
our summarized findings, the details how we categorized each primary study
according to these facets are elaborated in B, Table LABEL:tab:systematic_map.

###  4.1 Research facets (RQ1.1)

Most primary studies on the state-of-the-art handling of NFRs in DevOps
present practical solutions or experiences (63%, comprising
evaluation/validation research, experience reports, and solution proposals),
contrasted by 37% of studies being solely theoretical (represented through
philosophical/opinion papers and solution proposals) expressing the personal
opinion of the authors and without claiming practical applicability. Figure 2
shows the distribution of the 6 research facets. Interestingly, only 8% of
primary studies share practical experiences regarding handling NFRs in DevOps.
An innovative contribution to this research area can primarily be found by 29%
of the studies, contributed through philosophical papers and solution
proposals. Contrarily to these both types of works, opinion papers having a
share of 17% of the studies do not rely on related work or any sound research
methodology and thus also do not claim for applicability, feasibility or
suitability of the presented ideas.

Figure 2: Distribution of research facets.

To summarize, practically applicable and empirically validated approaches for
handling NFRs in DevOps only constituted 19% of the primary studies,
represented by the studies based on evaluation research. The remaining studies
either are not empirically validated, solely theoretical, or primarily
innovate without any form of structured validation.

###  4.2 Contribution facets (RQ1.2)

In Figure 3 we show the distribution of the contribution facets of the
studies. Many studies contributed to the theory of NFRs in DevOps (26%, e.g.,
requirements prioritization criteria [35], share experiences and lessons
learned in practice (17%) such as the capabilities and challenges of DevOps
faced in industrial settings [9], or describe general advices for handling
NFRs in DevOps (10%), such as the organizational change associated with DevOps
adoption [36]. On the other side, 40% of the studies contribute concrete
approaches, comprised of methods (17%) (e.g., for evaluating the fulfillment
of NFRs towards value streams [37], frameworks (11%), (e.g., for guiding the
evaluation of NFRs specially in DevOps [38]), models (9%) e.g., for feature
ideation from user interactions [39], and tools (3%) for instance to support
customer value estimation during feature ideation [40].

Figure 3: Distribution of contribution facets.

Processes for handling NFRs in DevOps are contributed by only 1% of the
studies. It can be summarized that the emphasis of the studies in this field
is on theoretical and conceptual contributions (60%, comprised of 17% reports
about lessons learned, 6% guidelines, 26% theoretical analyses, 10% advices
and 1% presenting processes).

###  4.3 Publication channels and venues (RQ1.3)

We also analyzed which channels are used for publishing the studies and
illustrate this distribution in Figure 4. The smallest share of studies
dealing with the handling of NFRs in DevOps is published in books (as
chapters), comprising only 11% of all studies. The remaining 89% of studies
are either published on scientific conferences (44%) or journals (45%), which
typically follow a stricter review process than conferences. While the
dominance of journal publications is not a valid measure of the quality of the
studies, it might be an indicator of high research efforts spent in this
field.

Figure 4: Distribution of publication channels.

In addition, we examined the publication venues of the reviewed conference and
journal publications, as these channels build the majority of the studies
published in this field. In Table 4 we outline the channels and their main
publication venues, i.e., where more than 3 studies have been published, and
the concrete number of studies published there.

Publication channel |  Venue |  Studies
---|---|---
Conferences |  International Conference on Software Business (ICSOB) |  6
|  International Working Conference on Requirements Engineering: Foundation for Software Quality (REFSQ) |  5
|  International Conference on Product-Focused Software Process Improvement (PROFES) |  5
|  International Symposium on Empirical Software Engineering and Measurement (ESEM) |  4
|  International Conference on Software Engineering (ICSE) |  4
|  International Workshop on Quality-Aware DevOps (QUDOS) |  3
|  International Conference on Evaluation and Assessment in Software Engineering (EASE) |  3
|  International Workshop on Real-Time Business Intelligence and Analytics (BIRTE) |  3
|  Euromicro Conference on Software Engineering and Advanced Applications (SEAA) |  3
|  International Conference on Agile Software Development (XP) |  3
Journals |  IEEE Software |  19
|  Journal of Systems and Software |  7
|  Journal of Software: Evolution and Process |  4
|  Empirical Software Engineering |  3
|  Journal of Software Maintenance and Evolution: Research and Practice |  3
Table 4: Main publication venues of the studies.

Studies in this field typically are presented on business-oriented (e.g.,
ICSOB and PROFES), requirements (e.g., REFSQ), and software engineering
conferences (e.g., ESEM, EASE, ICSE, and SEAA) or published predominantly in
software-focused journals such as "IEEE Software" or "Journal of Systems and
Software".

###  4.4 Publication frequency over years (RQ1.4)

As the final step of the quantitative description of the study population, we
also analyzed the number of publications per year. As depicted in Figure 5,
the reviewed studies were published between 1998 and 2018. The figure also
shows the steady incline in publication over the years, illustrated through
the dotted regression line.

Figure 5: Publication distribution by year.

Although the first 12 studies (9%) were published between 1998 and 2006, rises
in the publication frequency can be seen in the years 2007 and again in 2015.
A noticeable and steady publication frequency can be observed since 2015 where
the publication frequency almost doubled compared to former years. This also
indicates that the research field examining the handling of NFRs in DevOps is
still in an early stage, compared with other fields in software engineering
but gaining more interest in recent years. The continuous rise between 2007
and 2015 can most probably be attributed to the rising academic interest of
NFRs in software engineering and specially to influencing works in this domain
[41, 20, 3]. The second rise in publication frequency since 2015 may be
explained with the advent of the DevOps culture that triggered respective
academic research in this area [42, 43, 8].

##  5 Focus areas of NFRs in DevOps (RQ2)

The 142 primary studies cover a multitude of different focus areas – software
quality, operational aspects, the structuring of source code to ensure
exchangeability of developers, or the mapping of software artefacts to value
streams. We categorized the described NFRs in DevOps by 7 focus areas,
according to the classification scheme elaborated in A, Table
LABEL:tab:classification_scheme_focus_areas. This classification scheme was
derived from the overall focus areas we repeatedly encountered when reviewing
the primary studies during piloting the search string (cf. Section 3.3). In
this section we illustrate the dominant focus areas of NFRs in DevOps and the
typical research and contribution facets of each focus area. Complementary, we
describe the themes and objectives of each focus area along with examples in
Section 6.

###  5.1 Customer focus area

We selected 38 studies (27%) of the study population to be relevant for this
focus area, as they elaborate on methods for increasing customer experience,
sketching novel features, and monitoring customer-centered improvements
throughout the DevOps cycle. In Figure 6 we show the distribution of the
research and contribution facets of the studies related to this focus area,
along with the 5 main themes recurrently described in the studies.

Figure 6: Facets and themes of studies examining customer-focused NFRs.

The majority of studies in this area follows validation research principles,
i.e., irrespective their novelty these approaches have not been empirically
validated. Only 7 studies in this area are empirically validated (evaluation
research facet). However, 6 studies elaborate on concrete practical
experiences in the context of customer-focused NFRs in DevOps. These are also
supported by 3 theoretical proposals sketching new ideas, mainly for the
prioritization and alignment of agile requirements to increase customer
experience. Considering the contributions of the studies, most studies
contribute theories and concepts or report on lessons learned either from a
software/requirements engineering or product management perspective. Models
and frameworks for customer-centricity, supported through NFRs in DevOps, are
contributed by 13 of the studies.

###  5.2 Development focus area

In total, 26 of the 142 studies (18%) elaborate on NFRs in DevOps that focus
on the efficiency of the software development process, e.g., certain
restrictions that must be obeyed at an early development or requirements
specification stage or particular considerations for continuous integration
and delivery. We again picture the distribution of the research and
contribution facets as well as the 5 main themes of the respective studies in
Figure 7.

Figure 7: Facets and themes of studies examining development-focused NFRs.

Considering the research facets, in this group most studies are empirically (8
studies, evaluation research facet) or theoretically (6 studies, validation
research facet) validated. This most likely can be explained through the high
availability of data accruing in the development process as well as the
availability of experts that can be consulted for empirical studies in this
context [44, 45]. Contrarily, 12 studies (46%, comprised of 2 experience
reports, 5 solution proposals, and 5 opinion papers) of the 26 studies
focusing on development-centered NFRs have not used any form of validation.

The majority of 15 studies (58%) in this focus area contribute practical
experiences, comprised of guidelines (6 studies) and lessons learned (9
studies). Concrete frameworks, methods, and models applicable to development-
focused NFRs are only contributed by 5 studies (19%).

###  5.3 Operation focus area

This focus area comprises studies addressing special characteristics and
requirements towards the software that must be regarded primarily during its
operation. The selected studies of this focus area amount to 20 studies (14%)
of the overall study population and can be split up into 3 prime themes.

Figure 8: Facets and themes of studies examining operation-focused NFRs.

Again considering the research facets, only 2 of the 20 studies (10%) are
empirically validated and the benefits and drawbacks clearly elaborated
(evaluation research facet). Contrarily, 8 of the 20 studies (40%) use
literature reviews or are based on quantitative case studies (validation
research). Compared to these numbers, only 8 of the 20 studies (40%, comprised
of 4 philosophical papers and 4 solution proposals) lack any validation but
contribute innovative new ideas to NFRs related to software operation.

A proportionally large share of 9 of 20 studies (45%) representing this focus
area contribute guidelines or report on lessons learned from practical
software projects. Concrete approaches for handling software operation-focused
NFRs are contributed by 6 studies (30%, comprised of models, methods, and
frameworks). Finally, theoretical inputs to this focus area come from 5
studies (25%).

###  5.4 Governance focus area

Studies elaborating on NFRs that focus on (legal) liabilities, regulatory or
technological compliances that must be obeyed and monitored during DevOps are
captured by the governance focus area. We selected 41 studies (29%) being
representative for this focus area and extracted 3 main themes linked with the
NFRs described in the studies, as illustrated in Figure 9.

Figure 9: Facets and themes of studies examining governance-focused NFRs.

In this focus area, 19 of 36 studies (53%, comprising 11 philosophical and 8
opinion papers) can be categorized as being solely theoretical and thus
dominate among the research facets. Apparent is the small number of experience
reports in this focus area, only being attributed to 2 studies (6%). This is
interesting insofar, as the comparatively high number of theoretical papers
indicates a certain relevance of the related topics, at least from an academic
perspective. Contrarily, practical approaches for handling governance-focused
NFRs in DevOps are presented by 14 studies (39%), comprised of 8 studies being
empirically validated (evaluation research facet) and 6 studies presenting
literature reviews or quantitative, labor-driven experiments.

The dominance of theoretical works can also be observed among the contribution
facets of the works, totaling to 19 studies (53%). Operational approaches for
handling governance-focused NFRs in the DevOps cycle are contributed by 12
studies (33%, comprised of 1 model, 5 methods, 5 frameworks, and 1 tool
described in the studies). Practical guidance for implementing governance-
centered NFRs in DevOps is contributed by 10 of the 36 studies (28%, comprised
of 1 guideline, 8 lessons learned, and 1 advice).

###  5.5 Technical quality focus area

This focus area addresses the common engineering-focused NFRs of software,
ranging from static qualities such as the maintainability of the source code
to dynamic qualities such as the response time or memory consumption. Due to
the existing classification schemes, such as ISO/IEC 25010 [46], the themes
described in the studies were categorized on a much finer level than of the
other focus areas. Figure 10 illustrates the primarily engineering-oriented
themes of this focus area.

Figure 10: Facets and themes of studies examining technical quality-focused
NFRs.

Also, due to the broad coverage of this type of NFRs in academic literature,
the majority of studies retrieved from the databases are applicable to this
focus area. Overall, we selected 59 studies (42%) for this focus area and
extracted 11 representative themes. With 21 respective studies (36%),
validation research takes the greatest share among the research facets,
followed by 10 studies (17%) applying evaluation research methodologies.
Compared with the 47% of studies attributed to other research facets
(comprised of 12 philosophical, 6 solution proposals, 8 opinion papers, and 2
experience reports) the emphasis in this focus area apparently is on validated
research. The majority of research in this focus area is conducted
theoretically through quantitative testing methods or conducting experiments
with prototypes. These studies nonetheless lack a clear elaboration on the
specific benefits and limitations of their approaches in practical contexts.
However, a very high number of studies applied empirical methods additionally.
Specially, these studies examined the use of certain measurement or evaluation
techniques of NFRs in practice or tested novel approaches in practical
settings and industrial contexts followed by e.g., expert interviews from
different domains. Also interesting is the comparatively high number of
philosophical papers, which e.g., sketch possible ways of intertwining
functional and non-functional requirements [47] or challenge the scope and
notion of NFRs and their demarcation from functional requirements [16, 17].

We found 17 studies (29%) contributing concrete methods for handling quality-
related NFRs in DevOps, complemented by 8 studies (14%) describing suitable
models and 5 studies (8%) presenting frameworks for that purpose. Practical
experiences for tackling software quality particularly in DevOps are
contributed by 9 studies (15%) in the form of concrete advices and by 4
studies (7%) elaborating lessons learned in handling quality-related NFRs in
DevOps. Theoretical contributions such as e.g., classification schemes come
from 13 of the 59 studies (22%).

###  5.6 Organization focus area

Studies examining NFRs in DevOps that target the organization level within
companies are summarized in this focus area. We selected 45 of the 142 studies
(32%) to be applicable for this focus area and subdivided their described NFRs
into 7 themes, as illustrated in Figure 11. Implicitly, the NFRs of this focus
area address the organizational capabilities and properties that must be in
place to effectively implement business-driven themes with software in a
company. Thus, they address human factors such as motivation or the required
support for skills growth as well as team/supplier communication
effectiveness. In contrast to other focus areas, the respective NFRs can
mainly be expressed and evaluated on a qualitative basis.

Figure 11: Facets and themes of studies examining organization-focused NFRs.

We categorized 21 studies (47%, comprised of 16 opinion and 5 philosophical
papers) as theoretical research works without empirical validation. On the
contrary side, 16 studies (36%, comprised of 13 validation and 3 evaluation
research works) presented validated research, mainly through systematic or
narrative literature reviews. Due to the applied research methods, these
studies do not claim for practical applicability of their findings. A small
number of 3 studies applied research methods that also allow to examine
specific benefits of their approaches, e.g., through conducting expert
interviews or case study research. Lastly, 8 studies (18%, comprising 1
solution proposal and 7 experience reports) examine the handling of
organization-focused NFRs in practice and report about possible ways for
improvement and practical challenges, without sketching a generally applicable
and validated methodology.

In total, 16 studies (13%) in this focus area contribute theoretical concepts
such as success factors for agile transformations [48], integrations of the
Scaled Agile framework (SAFe) for software development [49, 50] or differences
between quality management strategies [51] on organizational level. Concrete
guidance for implementing NFRs in DevOps that address organization-wide themes
is contributed by 22 studies (49%, comprised of 5 guidelines, 12 studies
elaborating on lessons learned, and 5 giving advices). These contributions
primarily originate from multi-case studies focusing on DevOps capabilities
and picture the benefits and challenges faced by the companies [52, 9].
Practically applicable artefacts, mainly complemented by empirical validations
in the context of case studies, are contributed by 6 studies (13%, comprising
1 model, 2 methods, and 3 frameworks).

###  5.7 Business focus area

Finally, the business focus area covers all NFRs that address strategic,
business, and cost/revenue considerations that manifest in software NFRs and
need to be considered throughout the DevOps cycle. As illustrated in Figure
12, we selected 39 studies (27%) for this focus area and carved out 5 main
themes behind the respective NFRs. The typical themes of this focus area are
closely linked with elements of software business models [53, 54] of the
company. The fulfillment of these NFRs is mainly evaluated through dynamic
measures.

Figure 12: Facets and themes of studies examining business-focused NFRs.

The largest portion of studies in this focus area, totaling 20 studies (51%,
comprising 7 philosophical and 13 opinion papers), deal with business-focused
NFRs in DevOps from a solely theoretical perspective, e.g., to outline
business models specially for software [55], reflect on the suitability of ROI
indicators for software [56], or sketch cost and value estimation techniques
[57]. These theoretical and often biased works express personal opinions or
future research directions and are complemented by 14 studies (36%) following
sound validation methods. These studies have either evaluated a concrete
methodology in practice (8 studies, evaluation research facet) and elaborate
on their benefits and limitations, e.g., to guide value estimation in software
projects [40] or conducted literature reviews in any form (6 studies,
validation research facet) particularly about business models and value
creation from software [39, 58]. Lastly, 5 studies (13%) elaborate on
practical experiences (4 studies) and implementation solutions (1 study) for
handling business-focused NFRs in DevOps. Particularly these studies focus on
the monitoring of the respective NFRs and how to map business-level objectives
to software NFRs.

Considering the contribution facets of the studies, there is an emphasis on
theoretical works (13 studies, 33%) examining representative performance
indicators that can be applied to software products. Research artefacts that
facilitate the practical handling of business-focused NFRs in DevOps, e.g.,
through methods and tools for deriving quantitative software performance
indicators for software products, are contributed by 8 studies (21%, comprised
of 2 models, 3 methods, 1 framework and 2 tools). Best-practices from own or
observed experiences of the authors and implementation guidelines are
presented by 18 studies (46%, comprising 7 studies presenting guidelines, 6
studies reporting on lessons learned and 5 studies giving concrete advices).

##  6 Themes and objectives of NFRs in DevOps (RQ3)

In this section we describe the recurrent themes of NFRs in DevOps that we
identified in the studies and the typical objectives that are linked with
them. We group this description by the focus areas presented in the previous
section.

###  6.1 Themes and objectives behind NFRs of the customer focus area

The customer-focused themes of NFRs in DevOps target on optimizing the
contributed value to the customer, as illustrated in Table
LABEL:tab:customer_themes. An emphasis is on monitoring and analyzing user
interaction with the software for data-driven software improvement and
shortening of release cycles.

Table 5: Themes and objectives of customer-focused NFRs. Theme |  Frequency |  Primary studies
---|---|---
Feature ideation from customer feedback |  15 |  [59, 60, 61, 62, 63, 64, 65, 66, 67, 44, 68, 69, 70, 71, 72]
Requirements validation |  3 |  [73, 61, 74]
Customer value creation |  8 |  [75, 76, 77, 78, 79, 80, 35, 81]
Release cadence |  4 |  [50, 24, 69, 9]
Customer-centered epics prioritization |  8 |  [82, 36, 62, 83, 84, 35, 9, 85]
|  |

  * •

Feature ideation from customer feedback: Gathering continuous feedback from
the customer for software improvement as well as the ideation of new features.
The focus of the improvement mainly is on the functional suitability of the
software. For feature ideation, quantitative user interaction data (from user
monitoring) is complemented with qualitative data from user surveys.

  * •

Requirements validation: Validating the relevance of user requirements
throughout the development cycle, e.g., through frequent validation cycles
with customers.

  * •

Customer value creation: Estimating the contributed value of software
throughout the development cycle, e.g., through a "customer needs scorecard"
[77] or the analysis of user queries [80].

  * •

Release cadence: Shortening the time between planning and provisioning of
software changes.

  * •

Customer-centered epics prioritization: Prioritizing epics and user stories by
contributed value of the software change for the user. Contributed user value
is estimated e.g., through goal-pursuit time from monitored user interaction
[85].

###  6.2 Themes and objectives behind NFRs of the development focus area

The themes associated with development-focused NFRs in DevOps target on
reducing failure rates in the software development lifecycle, mostly through
frequent feedback loops and efficient tool support. We illustrate these themes
in Table LABEL:tab:development_themes and the objectives sequentially.

Table 6: Themes and objectives of development-focused NFRs. Theme |  Frequency |  Primary studies
---|---|---
Delivery pipeline maturity |  8 |  [38, 86, 8, 87, 24, 69, 45, 88]
Continuous integration |  4 |  [73, 89, 78, 45]
Continuous delivery |  4 |  [38, 89, 90, 45]
Development tool versatility |  4 |  [89, 91, 8, 49]
Development task complexity |  6 |  [73, 61, 92, 57, 87, 93, 84]
|  |

  * •

Delivery pipeline maturity: Closing feedback loops to facilitate actionable
analytics of the data accruing on the different systems in DevOps. The degree
of actionability is assessed through concrete metrics and reflects the
maturity of the delivery pipeline.

  * •

Continuous integration: Assuring correct system, software, and service
behavior through continuous integration of smaller code changes into the
codebase of the whole product.

  * •

Continuous delivery: Reducing the failure rate after software changes through
deploying the software in short cycles. Also, the aim is on gaining
understanding of the impacts of the software change on operational
characteristics and improve architectural decisions thereof [38].

  * •

Development tool versatility: Assuring the suitability and flexibility of the
development environments (e.g., IDEs, source code repositories, issue
trackers) to support engineers in their development activities and the
tracking between source code and compiled software on different systems.

  * •

Development task complexity: Reducing complexity of development tasks, e.g.,
through architectural and code blueprints, best practices, and guidelines for
development tasks.

###  6.3 Themes and objectives behind NFRs of the operation focus area

The main objective of the themes linked with operation-focused NFRs in DevOps
address the monitoring, debugging, and tracing of runtime artefacts, as
illustrated in Table LABEL:tab:operation_themes.

Table 7: Themes and objectives of operation-focused NFRs. Theme |  Frequency |  Primary studies
---|---|---
Artefact runtime traceability |  6 |  [73, 94, 87, 95, 96, 35]
Monitoring process maturity |  6 |  [38, 48, 87, 24, 9, 45]
Post-deployment debugging |  8 |  [59, 62, 97, 98, 58, 44, 96, 71]
|  |

  * •

Artefact runtime traceability: Establishing links between source code
artefacts and runtime entities [94]. Specially, this needs to regard different
versions of the software deployed on different systems and allow to relate
monitored operational data back to the originating source code version.

  * •

Monitoring process maturity: Effectively and efficiently monitor the software
in operation and automatize the integration of the monitoring data for
software analytics [24].

  * •

Post-deployment debugging: Facilitating debugging of the operational software,
e.g., upon deviations between expected and actual software behavior.
Particularly in system-of-system architectures and transactional systems this
may lead to specific challenges arising from different network
protocols/architectures, and firewalls [62].

###  6.4 Themes and objectives behind NFRs of the governance focus area

In this focus area, the objectives of the themes behind the NFRs are on
assuring data privacy, enterprise architecture, and regulatory compliance. We
picture the themes and their frequency in the studies in Table
LABEL:tab:governance_themes and the respective objectives afterwards.

Table 8: Themes and objectives of governance-focused NFRs. Theme |  Frequency |  Primary studies
---|---|---
Data privacy |  10 |  [99, 38, 100, 91, 101, 57, 102, 69, 103, 104]
Liabilities and regulatory compliance |  19 |  [105, 59, 106, 48, 89, 107, 55, 20, 57, 78, 108, 95, 80, 35, 69, 109, 110, 111, 112]
Enterprise architecture alignment |  7 |  [18, 113, 19, 44, 103, 114, 115]
|  |

  * •

Data privacy: Assuring privacy of the data accruing on the different systems,
comprised of e.g., interaction traces and processed sensitive data from the
user [100].

  * •

Liabilities and regulatory compliance: Addressing how legal obligations and
duties can be implemented by the software, such as the processing of licensing
data [55], the recording of presented information to customers in financial
advisory processes [35], the calculation scheme for financial ratings of
customers, or the processing of data for identifying customers in accounting
applications [109].

  * •

Enterprise architecture alignment: Aligning and integrating the software into
the enterprise application landscape through architectural guidelines and
frameworks, e.g., TOGAF [116] or ATAM [117].

###  6.5 Themes and objectives behind NFRs of the technical quality focus area

The themes in the quality focus area mainly reflect common static and dynamic
software quality factors, as also defined in e.g., the ISO/IEC 25010 standard
[46]. However, as shown in Table LABEL:tab:quality_themes, the frequency of
studies examining certain themes varies among the studies which indicates that
they are differently important in DevOps. Due to the apparent similarities of
the themes to the ISO 25010 standard, we primarily outline important
differences or additions of the respective NFRs in DevOps.

Table 9: Themes and objectives of technical quality-focused NFRs. Theme |  Frequency |  Primary studies
---|---|---
Usability |  4 |  [118, 119, 120, 81]
Resilience |  3 |  [121, 122, 84]
Scalability |  5 |  [18, 82, 65, 50, 37]
Performance |  10 |  [123, 3, 41, 124, 125, 126, 15, 127, 128, 109]
Framework compliance |  3 |  [129, 130, 131]
Security |  4 |  [75, 132, 64, 133]
Extensibility |  2 |  [79, 47]
Test quality |  2 |  [134, 135]
Maintainability |  4 |  [136, 137, 138, 139]
Data-driven deployment |  3 |  [74, 68, 70]
Safety |  19 |  [140, 141, 142, 143, 16, 144, 17, 66, 145, 146, 147, 148, 149, 150, 151, 104, 152, 153, 112]
|  |

  * •

Usability: Reducing the time and cognitive load to accomplish goals with
software. Task-based methods allow concrete quantification and automatized
evaluation of usability [120, 81].

  * •

Resilience: Supporting flexible adaptions of software to ensure product
robustness and particularly to diminish remediation costs in DevOps [121].

  * •

Scalability: Facilitating automatized deployments to dynamically react on
usage peaks. In the DevOps context this involves inserting probes into the
source code prior to deployment that can trigger automatic deployments thus
ensuring elasticity of the system [18].

  * •

Performance: Providing results to the user of the software in a responsive
fashion independently on whether the provided software is a graphical user
interface (GUI) or a service endpoint (API). In addition, performance
requirements shall be explicated and evaluated on the level of individual
features, balanced with their strategic and economic value [142, 123].

  * •

Framework compliance: Reducing complexity, training efforts, and maintenance
costs by adhering to predetermined and evaluated frameworks. For this purpose,
most companies use library blueprints or ready-to-use build profiles, which
can be integrated with little effort in the DevOps toolchain.

  * •

Security: Specially with the proliferation of software ecosystems shared
between multiple companies, end-to-end encryption of entered user data and
cascaded cryptographic blocks of the user journey are gaining more attention.
Most of these security requirements need to be fulfilled on feature-level and
require operational support for acquiring the measures in a DevOps context.

  * •

Extensibility: Facilitating modification of the software, e.g., through
separation and decomposition of functional blocks into self-contained units of
work. In DevOps, apart from static code metrics, also functional and latent
dependencies between software components can be traced. This can for instance
be achieved through labeling software components and extracting cause-effect-
relationships from operational data.

  * •

Test quality: Assessing test processes, methods, sources, and tools used for
testing in DevOps, i.e., test strategies, the degree of automation, test
coverage, test report dissemination and handling of failed test cases.

  * •

Maintainability: Lowering maintenance costs through adhering e.g., to design
best practices, coding styles and the early elimination of code bad smells.
Due to the availability of operational data in DevOps, maintainability
requirements shall be balanced with actual feature usage [137].

  * •

Data-driven deployment: Evolving development practices to the point where they
could continuously deploy and validate individual features, rather than
preparing larger product releases. Apparently, this requires continuous
collection of post-deployment data for quantitative analysis of this data.
These data allow to derive development activities and suitable indicators of
deployment efficiency [68].

  * •

Safety: Giving guarantees for certain quality attributes under predefined
conditions. Specially with the proliferation of software-centric devices in
new areas such as autonomous driving or drones, the monitoring of safety
quality attributes in a DevOps context gains increasing industry relevance
[144].

###  6.6 Themes and objectives behind NFRs of the organization focus area

The organization-focused themes of NFRs in DevOps target on ensuring
organizational agility through harmonizing team, supplier, and market
requirements. An important and recurrent center point among the themes is the
human factor and its influence on software development and operation, as
illustrated in Table LABEL:tab:organization_themes.

Table 10: Themes and objectives of technical organization-focused NFRs. Theme |  Frequency |  Primary studies
---|---|---
Market responsiveness |  8 |  [154, 155, 123, 48, 156, 49, 50, 157, 72]
Resource fluidity |  3 |  [158, 49, 50]
Motivation |  3 |  [99, 106, 159]
Team-supplier communication |  5 |  [89, 55, 108, 160, 9]
Cross-team integration |  10 |  [56, 136, 123, 36, 89, 62, 156, 90, 50, 161]
Skills growth |  7 |  [142, 48, 162, 62, 148, 24, 9]
Cross-domain convergence |  7 |  [89, 64, 138, 163, 102, 51]
|  |

  * •

Market responsiveness: Decreasing the time to respond to customer requests by
concretizing them on the level of software functionality. This can be achieved
in DevOps e.g., through extracting and mapping operational software
characteristics to metrics for Quality Function Deployment (QFD) [157].

  * •

Resource fluidity: Applying software engineering methods, best practices, and
standardized tools that allow to shuffle work between the organization when
priorities change.

  * •

Motivation: Leveraging internal sources for software improvement through
providing feedback directly and frequently to a certain engineer. In DevOps
this can be accomplished for instance through labelling software artefacts and
analyzing the commit history automatically.

  * •

Team-supplier communication: Empowering teams to drive decisions regarding
development and operation of the software concertedly with outside
stakeholders, such as e.g., suppliers. Apart from consistent communication
channels, in DevOps this requires thorough documentation of the software
artefact, specially of operational characteristics and all runtime metadata
[89].

  * •

Cross-team integration: Establishing inter-company collaboration among teams
of different domains to improve the software value stream [86] for the
company. Apart from cross-discipline training of the software engineers, an
unambiguous labeling of the software artefacts, a RACI (responsibility-
assignment) matrix, and standardized documentation guidelines facilitate
collaboration among teams [62]. Cross-functional teams shall not exceed the
size of 8-12 so that they can maintain a focus on a specific aspect of the
software [36].

  * •

Skills growth: Providing training materials and mentoring to acquire skills
required for accomplishing the DevOps-related tasks. Having a supportive and
visible support infrastructure on the level of concrete features and recurrent
tools is critical for the wholesale success of software product [48].
Particularly for DevOps this requires establishing links between software
artefacts, issue trackers, and deployment tools for capturing and documenting
all relevant data throughout the lifecycle.

  * •

Cross-domain convergence: Designing, developing, operating, and continuously
improving software, so that it can be reused across application domains, e.g.,
mobile and pervasive sensing systems that share and process data among
autonomous systems [144]. Primarily this affects the selection of suitable
network protocols for integrating the applications in the DevOps environment.
Respective environments for this purpose must be able to deal with unreliable
data connections and time-delayed delivery of operational data of the
developed applications [89].

###  6.7 Themes and objectives behind NFRs of the business focus area

Value capture, creation, and optimization throughout the software lifecycle is
the prime objective behind business-focused NFRs in DevOps.

Table 11: Themes and objectives of business-focused NFRs. Theme |  Frequency |  Primary studies
---|---|---
Mean time to value |  3 |  [36, 9, 164]
Transaction costs |  9 |  [105, 141, 39, 62, 49, 58, 80, 165, 35]
Ecosystem valuation |  9 |  [40, 155, 166, 89, 66, 55, 167, 138, 69]
Product market share |  10 |  [168, 136, 86, 55, 77, 146, 169, 170, 157, 35]
Value stream mapping |  8 |  [171, 172, 144, 86, 57, 87, 173, 157]
|  |

  * •

Mean time to value: Reducing the cycle time, i.e., the time between an idea
and its realization as software in production. Operationally, the respective
measures can be acquired from issue trackers and deployment systems [36].

  * •

Transaction costs: Balancing the software development and operation costs for
the manufacturer with the perceived value of using the software for the
customer. Also, customers conceive of costs in terms of the offer price plus
all additional changes they need to undertake in order to use software [105].
The cost for the manufacturer accruing in the DevOps cycle can be calculated
indirectly, e.g., through monitoring the frequency of bugs and code changes of
particular features. Contrarily, the value of the software for the user can be
evaluated using qualitative feedback methods (e.g., in-app surveys, ratings)
that complement quantitative measures such as e.g., the invocation frequency
or cause-effect analysis of customer behavior and offer price [66].

  * •

Ecosystem valuation: Assigning quantitative measures (either monetary or in
form of a company balanced scorecard) to an ecosystem and its services [40]. A
software ecosystem consists of a software platform, internal/external
developers and a community of domain experts. These domain experts help adapt
or modify the software to satisfy the users’ needs [155]. In DevOps, these
quantitative measures needed for thorough ecosystem valuation can be acquired
e.g., from ERP- and billing systems, bug trackers, development and operational
systems. Overall, this allows to dynamically balance the company’s efforts for
participation in the software ecosystem based on actual user behavior and thus
optimize the business value of the software for the company.

  * •

Product market share: Assessing and evaluating the impact of changes of the
provisioned software on its market share, particularly through gap analysis
between potential and actual market share, competitor analysis and value-
estimation of new features [67, 170]. Similar to ecosystem valuation, this
theme requires quantification of external factors e.g., through mining textual
product reviews, designing novel features, and weighting them according to
estimated economic value added (EVA) and return on invest (ROI) [168, 169]. In
DevOps, external factors such as e.g., the results from A/B testing, must then
be considered in conjunction with internal factors such as e.g., business
performance indicators, to decide upon concrete software engineering
activities [136].

  * •

Value stream mapping: Identifying and addressing delays and non-value-added
activities in DevOps to accomplish the shortest sustainable lead time. The
mapping between software artefacts and the business model’s value streams is
the prerequisite for assessing the business value of software features, as
also advocated in the Scaled Agile Framework (SAFe) [173]. For implementing
this theme in DevOps, application lifecycle management (ALM) tools can be used
to acquire the measures of the operational value stream, i.e., the steps used
to provide goods or services to the customers. As an example, the invocation
frequency of a feature in relation to the update cadence and its offer price
can serve as an indicator for its value for the customer. These indicators can
then be analyzed based on the cost and revenue model [174] to derive concrete
activities for the development value stream, i.e., the steps used to create
products, systems, or feature capabilities [86].

##  7 Discussion

In this section we summarize the principle findings from this systematic
mapping study and sketch their implications for researchers and practitioners.

###  7.1 Principle findings

Our work outlines the broad scope and focus areas of NFRs, becoming apparent
through the pursed objectives originating from different domains.
Particularly, our study shows that NFRs to a large extent address non-
engineering qualities of software which are not covered e.g., through the
ISO/IEC [46] which subsequently result in little if any practical support for
them in DevOps. This is problematic insofar, as operational data accruing in
the DevOps context, and generally in the software development lifecycle, that
are possibly suitable for guiding software development is not utilized.
Compared with the other focus areas of NFRs, the majority of analyzed studies
focus on technical qualities, such as e.g., maintainability of the source
code, safety, and deployability. This may be due to the concrete definition of
technically-oriented NFRs in common literature and specifically in the ISO/IEC
specification [46]. However, the importance of aligning software engineering,
organizational, and governance related objectives can also be inferred from
the number of studies elaborating on challenges arising from this misbalance.
As an example, the requirement for collecting operational feature usage data
is a prerequisite for A/B and feature testing [74, 78] which must be
implemented during software development. However, the customer-focused NFR
aiming on feature-ideation from user interaction (cf. Section 6.1) itself
requires an overall product strategy and ecosystem valuation, so that the
acquired measures can be suitably evaluated. In addition, the maintainability
requirements for new software features derived from actual user interaction
might probably be higher than of features having less strategic importance for
a company. This example shows the required indentation of different NFR focus
areas, i.e., business, customer, and development, to effectively build
software in DevOps following the idea of value-based software engineering
[56]. In addition, our study confirms the vagueness of the scope of "non-
functional requirements" itself, which has already been stressed by several
authors [175, 16, 93]. Also, the alignment between governance, regulatory
compliance, and software development is explicated as being crucial in many
studies [55, 105, 48, 135] and the lack of methods, to address these
objectives in a balanced manner, underlined. Surprisingly many studies
elaborate on the human factor in software engineering and its impact on the
final quality of the software product, but neither present applicable means to
express, nor methods to measure and evaluate the respective NFRs in DevOps.

###  7.2 Implications for researchers and practitioners

This study seeks to give an overview about typical focus areas, themes, and
objectives of NFRs in DevOps, to identify existing research gaps and to sketch
possible future research directions. Though NFRs being an established concept
in software engineering, there is a continuous and rising interesting in this
topic when it comes to the handling of NFRs in DevOps, which also became
apparent in the analysis of the publication frequency (cf. Figure 5). While
multiple studies describe (theoretical) methods, models, and frameworks for
handling NFRs in DevOps, few of them are validated empirically. Also, besides
classical engineering-oriented NFRs, the automatic evaluation of NFRs arising
from non-technical focus areas in DevOps, e.g., focusing on business aspects,
customers, and governance requirements, is still hindered by the absence of a
common collection of metrics for these focus areas. These metrics also are the
prerequisite for balancing the objectives of the different NFRs focus areas,
e.g., to quantitatively express the allowed maintenance effort for a software
feature based on its usage. The operationalization of non-engineering-focused
NFRs and their evaluation, e.g., in relation to a company’s business model,
plays a key role when it comes to software-based digitalization in companies
[6, 7, 176]. As this study is based on a literature analysis, the overall
question remains which objectives are actually important from a practical
perspective and also how the results from evaluating the NFRs of the diverse
focus areas can be most effectually utilized in DevOps.

In this frame, we are working on an approach for operationalizing and
evaluating NFRs in DevOps. Our approach is based on an operational software
quality model [177] to specify feature-dependent NFRs, respective measures,
and suitable instruments for their acquisition in an automatable manner. The
quality model is complemented by a constraint definition language that
utilizes these measures to express complex NFRs and evaluate their fulfillment
in DevOps. Accordingly, the classification of focus areas, themes, and
objectives of NFRs presented in this study are the foundation for specifying
NFRs in DevOps through constraints and measures using this operational quality
model.

##  8 Threats

Though we conducted this systematic mapping study and the snowballing
procedure according to the respective guidelines [27, 26], we recognize the
following threats to the validity of the study:

  * •

Publication bias: Literature reviews are influenced by the common bias that
positive research outcomes more likely are reported and published than
negative ones. As the researched topics in this mapping study cannot be
attributed to failure or success, we did not observe any noticeable bias among
the reviewed studies in this regard. We contrarily recognized that multiple
studies reflect on benefits of handling the multiform NFRs in DevOps, but also
on illustrate the challenges and problems associated with these approaches.
Overall, we recognized a well-balanced amount of studies reporting on either
positive or negative aspects towards the examined topics. Hence, we consider
the effect of this bias on our study as neglectable.

  * •

Lack of universal taxonomy: While the term "non-functional requirement" is
widely used in academic literature as well as in practice, in many cases a
clear distinction between non-functional and functional requirements is
difficult [175, 16]. Also, many studies actually describe non-functional
software requirements implicitly, particularly if these studies do not
originate from the software engineering domain. To mitigate this threat, we
conducted a snowballing procedure prior to the data analysis. Only if the
primary studies cited in the initially retrieved papers were relevant for
answering the research questions and adhered to the inclusion/exclusion
criteria we included these studies in the subsequent analysis. Thus, we are
confident that we captured the most relevant studies contributing to a broader
understanding also of non-engineering focused NFRs in software projects.

  * •

Identification of primary studies: We addressed this threat twofold: first, by
composing the search string of multiple keywords each individually expressing
a relevant aspect of NFRs in DevOps [30] and second, by piloting this search
string with different databases multiple times to evaluate the effect of the
keywords onto the result set. However, the precision of the retrieved result
set also depends on the quality of the keywords used in the searched studies.
Also, we only used databases that cover the software engineering research very
well and also read the titles, abstract, and conclusion (if needed) to decide
upon the selection of a study. Thus, we are convinced to have captured the
majority of the relevant studies and missed only little, if any, significant
studies not matched by the search strings.

  * •

Study selection bias: Though we are not aware of any biases regarding the
selection of the studies, we recognize that unsuitable definition of the
inclusion and exclusion criteria may lead to attrition bias. To mitigate this
threat, both researchers defined these criteria upfront in an unambiguous
manner. Also, we revised these criteria during multiple when piloting the
search string to assure to not exclude relevant studies. When deciding upon
the selection of studies, we discussed any disagreements regarding the
inclusion and exclusion of studies until both researchers agreed on a common
decision.

  * •

Data extraction and analysis: Biased judgments are also possible during data
extraction and analysis, i.e., which text passages in the studies are labeled
and which codes are derived and refined thereof. We mitigated this threat by
defining the concrete properties to extract from the studies and the way to
document this data upfront. Also, the thematic refinement of the codes and the
encoded text fragments were discussed between both researchers until we agreed
on a solid set of codes.

##  9 Conclusion

This paper presents the results from a systematic mapping study examining the
focus areas, themes, and objectives of NFRs in the DevOps context. We
retrieved 229 candidate studies from 5 academic databases, of which 114 were
selected as primary studies. Also, we included a snowballing step to capture
studies that elaborate on NFRs which not adhere to the strict technically-
oriented notion of this term but nonetheless have an impact on the design,
development, and operation of a software system. This step again resulted in
27 primary studies being selected for inclusion, with the result set totaling
to 142 primary studies for this mapping study. Data were extracted from these
studies and then categorized by a defined classification schema. Themes and
objectives of NFRs in DevOps described in the studies were iteratively
synthesized from labeled text fragments and then summarized per focus area.
The publication frequency over the years shows a continuous interest in this
field with higher increases since 2015, with the studies being predominantly
published in journals and on conferences. Most studies in this field
contribute theories, methods, models, or present lessons learned from handling
NFRs in DevOps in practical settings. However, the majority of these studies
describe artefacts which are not yet practically implemented and thus lack an
empirical validation. From the studies, we condensed 7 recurrent focus areas
of NFRs in DevOps, ranging from development and operation to governance- and
business-oriented areas. The most pressing objectives pursued with NFRs in
DevOps are to assure safe and responsive operation of the software (technical
quality focus), facilitate cross-team integration of experts from different
domains (organization focus), ascertain regulatory compliance of the software
(governance focus), utilize customer interaction with the software for
prospective feature ideation (customer focus), create and safeguard product
market share (business focus), optimize the delivery pipeline (development
focus), and to facilitate post-deployment debugging (operation focus).

For future work, typical metrics for the NFRs of the different focus areas and
usage scenarios in practice, as well as methods for their operationalization
shall be explored by researchers, at best complemented with an empirical
validation. Furthermore, methodological and tool support is needed to aid
practitioners specifying these NFRs in a way that can be utilized in a DevOps
context.

## Appendices

## Appendix A

Table 12: Classification scheme for research and contribution facets. Category |  Description
---|---
P1: Research facet, adapted from Petersen [27] and Wieringa [31]
Experience report |  Similar to opinion paper, these works express practical experiences of the author with a method or technique. Often these reports come from practitioners who have used a specific method in practice and can thus provide valuable insights into possible ways for improvements.
Evaluation research |  Describes the investigation of a problem or an implementation of a solution for it in practice, along with an evaluation for the solution. It also shows the benefits and drawbacks of the technique.
Validation research |  The presented techniques are novel and have not yet been practically implemented, e.g., work done in laboratory.
Philosophical paper |  The work sketches a new way on looking on things, such as taxonomies or concepts. Evaluation criteria are the originality, soundness, and transparency of the work.
Solution proposal |  A solution technique for a problem is presented along with an argumentation for its relevance, but an evaluation of the solution is missing. The solution must be novel or demonstrate a significant enhancement of a previous approach.
Opinion paper |  The work presents the personal opinion of the author about what is good or bad about a technique and how it can be improved. Such works do not rely on related works or sound research methodologies, but can possibly sketch new ways to address known problems.
P2: Contribution facet, adapted from Shaw [32] and Paternoster [33]
Model |  An abstraction of an observed reality by concepts or related concepts after a conceptualization process.
Method |  Method or technique for handling non-functional requirements in the planning, management, development, or operation of software.
Theory |  Construct of cause-effect relationships of determined results.
Framework |  Combination of methods with distinct conditions which method shall be applied under which circumstances and with defined input and output parameters.
Guideline |  List of advises, compilation, or interpretation of obtained research results.
Lessons Learned |  Commented set of outcomes with fact-based recommendations, directly concluded from the obtained research results.
Advice |  Generic recommendation, influenced by personal opinions from the author.
Tool |  Technology or software application used for handling non-functional requirements in the planning, management, development, or operation of software.
|
Table 13: Classification schema for NFR focus areas. Category |  Description
---|---
P5: Focus area of NFRs described in study
Customer |  Studies describing NFRs related to increasing customer experience or value, e.g., through user-centered value estimation techniques or passive user involvement, e.g., through user monitoring.
Development |  Approaches capturing NFRs related to the software development, integration and delivery process and its tools, artefacts, methods, systems, and stakeholders.
Operation |  Works that address NFRs affecting software operation and specifically includes the tools, artefacts, methods, systems, and involved stakeholders.
Governance |  Presents liabilities, regulatory or technological compliances and requirement alignments that need to be taken account in DevOps and affect the NFRs of the respective software.
Quality |  Works elaborating on or referring to static and dynamic quality factors of software that are linked with NFRs, e.g., the code or design quality, performance behavior or reliability.
Organization |  Tackles NFRs attributed to the needed agility of the organization in the DevOps context to cope with internal or external changes. Particularly this comprises staff fluctuation, training, and (third party) supplier integration.
Business |  Works regarding business-related NFRs that need to be taken into account in the DevOps context, e.g., to measure cost and revenue per user transaction or to evaluate fulfillment of business-driven goals with software.
|

## Appendix B

Table 14: Systematic map overview. Primary Study |  Research facet |  Contribution facet |  Focus area |  Theme
---|---|---|---|---
[67] |  Experience Report |  Lessons Learned |  Customer |  Feature Ideation
[68] |  Solution Proposal |  Framework |  Customer |  Feature Ideation
[61] |  Validation Research |  Model |  Customer |  Feature Ideation
[64] |  Validation Research |  Model |  Customer |  Feature Ideation
[71] |  Validation Research |  Method |  Customer |  Feature Ideation
[65] |  Evaluation Research |  Model |  Customer |  Feature Ideation
[66] |  Validation Research |  Model |  Customer |  Feature Ideation
[62] |  Experience Report |  Lessons Learned |  Customer |  Feature Ideation
[59] |  Validation Research |  Lessons Learned |  Customer |  Feature Ideation
[123] |  Validation Research |  Guideline |  Customer |  Feature Ideation
[44] |  Solution Proposal |  Framework |  Customer |  Feature Ideation
[69] |  Evaluation Research |  Theory |  Customer |  Feature Ideation
[70] |  Validation Research |  Lessons Learned |  Customer |  Feature Ideation
[72] |  Validation Research |  Theory |  Customer |  Feature Ideation
[63] |  Evaluation Research |  Framework |  Customer |  Feature Ideation
[73] |  Validation Research |  Lessons Learned |  Customer |  Requirements Validation
[61] |  Validation Research |  Model |  Customer |  Requirements Validation
[74] |  Solution Proposal |  Model |  Customer |  Requirements Validation
[75] |  Validation Research |  Theory |  Customer |  Value Creation
[77] |  Evaluation Research |  Guideline |  Customer |  Value Creation
[78] |  Evaluation Research |  Lessons Learned |  Customer |  Value Creation
[79] |  Validation Research |  Model |  Customer |  Value Creation
[80] |  Philosophical Paper |  Theory |  Customer |  Value Creation
[76] |  Validation Research |  Theory |  Customer |  Value Creation
[35] |  Validation Research |  Theory |  Customer |  Value Creation
[81] |  Validation Research |  Model |  Customer |  Value Creation
[50] |  Validation Research |  Theory |  Customer |  Release Cadence
[24] |  Validation Research |  Lessons Learned |  Customer |  Release Cadence
[69] |  Evaluation Research |  Theory |  Customer |  Release Cadence
[9] |  Experience Report |  Lessons Learned |  Customer |  Release Cadence
[36] |  Experience Report |  Advice |  Customer |  Customer-Centered Epics Prioritization
[82] |  Validation Research |  Advice |  Customer |  Customer-Centered Epics Prioritization
[62] |  Experience Report |  Lessons Learned |  Customer |  Customer-Centered Epics Prioritization
[83] |  Philosophical Paper |  Theory |  Customer |  Customer-Centered Epics Prioritization
[84] |  Evaluation Research |  Model |  Customer |  Customer-Centered Epics Prioritization
[35] |  Validation Research |  Theory |  Customer |  Customer-Centered Epics Prioritization
[9] |  Experience Report |  Lessons Learned |  Customer |  Customer-Centered Epics Prioritization
[85] |  Philosophical Paper |  Framework |  Customer |  Customer-Centered Epics Prioritization
[141] |  Validation Research |  Theory |  Governance |  Enterprise Architecture
Alignment
[113] |  Philosophical Paper |  Tool |  Governance |  Enterprise Architecture
Alignment
[44] |  Solution Proposal |  Framework |  Governance |  Enterprise Architecture
Alignment
[103] |  Solution Proposal |  Method |  Governance |  Enterprise Architecture
Alignment
[115] |  Validation Research |  Advice |  Governance |  Enterprise Architecture
Alignment
[154] |  Experience Report |  Lessons Learned |  Organization |  Market Responsiveness
[144] |  Opinion Paper |  Advice |  Organization |  Cross-Domain
Convergence
[89] |  Opinion Paper |  Lessons Learned |  Organization |  Cross-Domain
Convergence
[64] |  Validation Research |  Model |  Organization |  Cross-Domain
Convergence
[138] |  Evaluation Research |  Method |  Organization |  Cross-Domain
Convergence
[163] |  Solution Proposal |  Framework |  Organization |  Cross-Domain
Convergence
[102] |  Opinion Paper |  Framework |  Organization |  Cross-Domain
Convergence
[69] |  Evaluation Research |  Theory |  Organization |  Cross-Domain
Convergence
[51] |  Evaluation Research |  Method |  Organization |  Cross-Domain
Convergence
[118] |  Opinion Paper |  Theory |  Quality |  Usability
[119] |  Solution Proposal |  Method |  Quality |  Usability
[120] |  Opinion Paper |  Theory |  Quality |  Usability
[73] |  Validation Research |  Lessons Learned |  Operation |  Artefact
Runtime Traceability
[94] |  Validation Research |  Method |  Operation |  Artefact
Runtime Traceability
[87] |  Solution Proposal |  Guideline |  Operation |  Artefact
Runtime Traceability
[95] |  Evaluation Research |  Method |  Operation |  Artefact
Runtime Traceability
[96] |  Philosophical Paper |  Theory |  Operation |  Artefact
Runtime Traceability
[35] |  Validation Research |  Theory |  Operation |  Artefact
Runtime Traceability
[59] |  Validation Research |  Lessons Learned |  Operation |  Post-Deployment
Debugging
[62] |  Experience Report |  Lessons Learned |  Operation |  Post-Deployment
Debugging
[97] |  Philosophical Paper |  Model |  Operation |  Post-Deployment
Debugging
[98] |  Philosophical Paper |  Theory |  Operation |  Post-Deployment
Debugging
[58] |  Validation Research |  Guideline |  Operation |  Post-Deployment
Debugging
[44] |  Solution Proposal |  Framework |  Operation |  Post-Deployment
Debugging
[96] |  Philosophical Paper |  Theory |  Operation |  Post-Deployment
Debugging
[71] |  Validation Research |  Method |  Operation |  Post-Deployment
Debugging
[38] |  Solution Proposal |  Framework |  Operation |  Monitoring
Process Maturity
[48] |  Validation Research |  Theory |  Operation |  Monitoring
Process Maturity
[87] |  Solution Proposal |  Guideline |  Operation |  Monitoring
Process Maturity
[69] |  Validation Research |  Lessons Learned |  Operation |  Monitoring
Process Maturity
[9] |  Experience Report |  Lessons Learned |  Operation |  Monitoring
Process Maturity
[45] |  Evaluation Research |  Guideline |  Operation |  Monitoring
Process Maturity
[38] |  Solution Proposal |  Framework |  Development |  Development Pipeline
Maturity
[86] |  Opinion Paper |  Guideline |  Development |  Development Pipeline
Maturity
[8] |  Evaluation Research |  Theory |  Development |  Development Pipeline
Maturity
[87] |  Solution Proposal |  Guideline |  Development |  Development Pipeline
Maturity
[69] |  Validation Research |  Lessons Learned |  Development |  Development Pipeline
Maturity
[69] |  Evaluation Research |  Theory |  Development |  Development Pipeline
Maturity
[45] |  Evaluation Research |  Guideline |  Development |  Development Pipeline
Maturity
[88] |  Solution Proposal |  Framework |  Development |  Development Pipeline
Maturity
[73] |  Validation Research |  Lessons Learned |  Development |  Continuous Integration
[89] |  Opinion Paper |  Lessons Learned |  Development |  Continuous Integration
[78] |  Evaluation Research |  Lessons Learned |  Development |  Continuous Integration
[45] |  Evaluation Research |  Guideline |  Development |  Continuous Integration
[90] |  Experience Report |  Process |  Development |  Continuous Integration
[38] |  Solution Proposal |  Framework |  Development |  Continuous Integration
[89] |  Opinion Paper |  Lessons Learned |  Development |  Continuous Integration
[45] |  Evaluation Research |  Guideline |  Development |  Continuous
Delivery
[89] |  Opinion Paper |  Lessons Learned |  Development |  Development Tool
Versatility
[91] |  Experience Report |  Lessons Learned |  Development |  Development Tool
Versatility
[8] |  Evaluation Research |  Theory |  Development |  Development Tool
Versatility
[49] |  Opinion Paper |  Theory |  Development |  Development Tool
Versatility
[73] |  Validation Research |  Lessons Learned |  Development |  Development
Task Complexity
[61] |  Validation Research |  Model |  Development |  Development
Task Complexity
[92] |  Validation Research |  Lessons Learned |  Development |  Development
Task Complexity
[87] |  Solution Proposal |  Guideline |  Development |  Development
Task Complexity
[93] |  Validation Research |  Theory |  Development |  Development
Task Complexity
[44] |  Evaluation Research |  Model |  Development |  Development
Task Complexity
[140] |  Validation Research |  Method |  Technical Quality |  Safety
[141] |  Validation Research |  Theory |  Technical Quality |  Safety
[142] |  Philosophical Paper |  Theory |  Technical Quality |  Safety
[143] |  Opinion Paper |  Advice |  Technical Quality |  Safety
[16] |  Philosophical Paper |  Advice |  Technical Quality |  Safety
[144] |  Opinion Paper |  Advice |  Technical Quality |  Safety
[17] |  Philosophical Paper |  Lessons Learned |  Technical Quality |  Safety
[145] |  Evaluation Research |  Lessons Learned |  Technical Quality |  Safety
[146] |  Evaluation Research |  Method |  Technical Quality |  Safety
[147] |  Philosophical Paper |  Method |  Technical Quality |  Safety
[148] |  Opinion Paper |  Advice |  Technical Quality |  Safety
[149] |  Validation Research |  Framework |  Technical Quality |  Safety
[150] |  Philosophical Paper |  Theory |  Technical Quality |  Safety
[66] |  Validation Research |  Model |  Technical Quality |  Safety
[151] |  Opinion Paper |  Advice |  Technical Quality |  Safety
[152] |  Evaluation Research |  Framework |  Technical Quality |  Safety
[104] |  Validation Research |  Theory |  Technical Quality |  Safety
[153] |  Evaluation Research |  Model |  Technical Quality |  Safety
[112] |  Philosophical Paper |  Theory |  Technical Quality |  Safety
[105] |  Opinion Paper |  Lessons Learned |  Governance |  Regulatory Compliance
[59] |  Validation Research |  Lessons Learned |  Governance |  Regulatory Compliance
[106] |  Opinion Paper |  Guideline |  Governance |  Regulatory Compliance
[110] |  Opinion Paper |  Theory |  Governance |  Regulatory Compliance
[107] |  Solution Proposal |  Method |  Governance |  Regulatory Compliance
[55] |  Opinion Paper |  Theory |  Governance |  Regulatory Compliance
[20] |  Philosophical Paper |  Theory |  Governance |  Regulatory Compliance
[57] |  Philosophical Paper |  Theory |  Governance |  Regulatory Compliance
[95] |  Evaluation Research |  Method |  Governance |  Regulatory Compliance
[80] |  Philosophical Paper |  Theory |  Governance |  Regulatory Compliance
[111] |  Evaluation Research |  Model |  Governance |  Regulatory Compliance
[112] |  Philosophical Paper |  Theory |  Governance |  Regulatory Compliance
[19] |  Experience Report |  Lessons Learned |  Governance |  Enterprise Architecture
Alignment
[114] |  Evaluation Research |  Theory |  Governance |  Enterprise Architecture
Alignment
[99] |  Philosophical Paper |  Framework |  Governance |  Data Privacy
[38] |  Solution Proposal |  Framework |  Governance |  Data Privacy
[100] |  Evaluation Research |  Lessons Learned |  Governance |  Data Privacy
[91] |  Experience Report |  Lessons Learned |  Governance |  Data Privacy
[101] |  Philosophical Paper |  Framework |  Governance |  Data Privacy
[57] |  Philosophical Paper |  Theory |  Governance |  Data Privacy
[102] |  Opinion Paper |  Framework |  Governance |  Data Privacy
[69] |  Evaluation Research |  Theory |  Governance |  Data Privacy
[103] |  Solution Proposal |  Method |  Governance |  Data Privacy
[104] |  Validation Research |  Theory |  Governance |  Data Privacy
[81] |  Validation Research |  Model |  Technical Quality |  Usability
[121] |  Validation Research |  Theory |  Technical Quality |  Resilience
[122] |  Evaluation Research |  Method |  Technical Quality |  Resilience
[178] |  Opinion Paper |  Advice |  Technical Quality |  Resilience
[141] |  Validation Research |  Theory |  Technical Quality |  Scalability
[65] |  Evaluation Research |  Model |  Technical Quality |  Scalability
[82] |  Validation Research |  Advice |  Technical Quality |  Scalability
[50] |  Validation Research |  Theory |  Technical Quality |  Scalability
[37] |  Solution Proposal |  Method |  Technical Quality |  Scalability
[123] |  Validation Research |  Guideline |  Technical Quality |  Performance
[3] |  Validation Research |  Lessons Learned |  Technical Quality |  Performance
[41] |  Validation Research |  Method |  Technical Quality |  Performance
[179] |  Evaluation Research |  Method |  Technical Quality |  Performance
[125] |  Philosophical Paper |  Method |  Technical Quality |  Performance
[126] |  Validation Research |  Model |  Technical Quality |  Performance
[147] |  Philosophical Paper |  Method |  Technical Quality |  Performance
[127] |  Philosophical Paper |  Method |  Technical Quality |  Performance
[128] |  Solution Proposal |  Framework |  Technical Quality |  Performance
[109] |  Solution Proposal |  Method |  Technical Quality |  Performance
[129] |  Validation Research |  Tool |  Technical Quality |  Framework Compliance
[130] |  Experience Report |  Method |  Technical Quality |  Framework Compliance
[131] |  Evaluation Research |  Method |  Technical Quality |  Framework Compliance
[48] |  Validation Research |  Theory |  Governance |  Regulatory Compliance
[89] |  Opinion Paper |  Lessons Learned |  Governance |  Regulatory Compliance
[55] |  Opinion Paper |  Theory |  Governance |  Regulatory Compliance
[20] |  Philosophical Paper |  Theory |  Governance |  Regulatory Compliance
[78] |  Evaluation Research |  Lessons Learned |  Governance |  Regulatory Compliance
[108] |  Opinion Paper |  Theory |  Governance |  Regulatory Compliance
[80] |  Philosophical Paper |  Theory |  Governance |  Regulatory Compliance
[35] |  Validation Research |  Theory |  Governance |  Regulatory Compliance
[69] |  Evaluation Research |  Theory |  Governance |  Regulatory Compliance
[109] |  Solution Proposal |  Method |  Governance |  Regulatory Compliance
[112] |  Philosophical Paper |  Theory |  Governance |  Regulatory Compliance
[75] |  Validation Research |  Theory |  Technical Quality |  Security
[132] |  Philosophical Paper |  Method |  Technical Quality |  Security
[64] |  Validation Research |  Model |  Technical Quality |  Security
[133] |  Philosophical Paper |  Theory |  Technical Quality |  Security
[79] |  Validation Research |  Model |  Technical Quality |  Extensibility
[47] |  Philosophical Paper |  Method |  Technical Quality |  Extensibility
[134] |  Experience Report |  Advice |  Technical Quality |  Test Quality
[135] |  Validation Research |  Theory |  Technical Quality |  Test Quality
[138] |  Evaluation Research |  Method |  Technical Quality |  Maintainability
[136] |  Opinion Paper |  Advice |  Technical Quality |  Maintainability
[137] |  Validation Research |  Lessons Learned |  Technical Quality |  Maintainability
[139] |  Evaluation Research |  Framework |  Technical Quality |  Maintainability
[70] |  Validation Research |  Lessons Learned |  Technical Quality |  Data-Driven Deployment
[74] |  Solution Proposal |  Model |  Technical Quality |  Data-Driven Deployment
[68] |  Solution Proposal |  Framework |  Technical Quality |  Data-Driven Deployment
[156] |  Experience Report |  Process |  Organization |  Market Responsiveness
[123] |  Validation Research |  Guideline |  Organization |  Market Responsiveness
[48] |  Validation Research |  Theory |  Organization |  Market Responsiveness
[155] |  Philosophical Paper |  Theory |  Organization |  Market Responsiveness
[49] |  Opinion Paper |  Theory |  Organization |  Market Responsiveness
[50] |  Validation Research |  Theory |  Organization |  Market Responsiveness
[157] |  Philosophical Paper |  Guideline |  Organization |  Market Responsiveness
[72] |  Validation Research |  Theory |  Organization |  Market Responsiveness
[158] |  Validation Research |  Theory |  Organization |  Resource Fluidity
[49] |  Opinion Paper |  Theory |  Organization |  Resource Fluidity
[50] |  Validation Research |  Theory |  Organization |  Resource Fluidity
[106] |  Opinion Paper |  Guideline |  Organization |  Motivation
[99] |  Philosophical Paper |  Framework |  Organization |  Motivation
[159] |  Philosophical Paper |  Theory |  Organization |  Motivation
[89] |  Opinion Paper |  Lessons Learned |  Organization |  Team-Supplier
Communication
[162] |  Opinion Paper |  Theory |  Organization |  Team-Supplier
Communication
[108] |  Opinion Paper |  Theory |  Organization |  Team-Supplier
Communication
[160] |  Opinion Paper |  Advice |  Organization |  Team-Supplier
Communication
[9] |  Experience Report |  Lessons Learned |  Organization |  Team-Supplier
Communication
[56] |  Validation Research |  Lessons Learned |  Organization |  Cross-Team Integration
[123] |  Validation Research |  Guideline |  Organization |  Cross-Team Integration
[89] |  Opinion Paper |  Lessons Learned |  Organization |  Cross-Team Integration
[62] |  Experience Report |  Lessons Learned |  Organization |  Cross-Team Integration
[36] |  Experience Report |  Advice |  Organization |  Cross-Team Integration
[89] |  Opinion Paper |  Lessons Learned |  Organization |  Cross-Team Integration
[86] |  Opinion Paper |  Guideline |  Organization |  Cross-Team Integration
[136] |  Opinion Paper |  Advice |  Organization |  Cross-Team Integration
[50] |  Validation Research |  Theory |  Organization |  Cross-Team Integration
[161] |  Opinion Paper |  Theory |  Organization |  Cross-Team Integration
[148] |  Opinion Paper |  Advice |  Organization |  Skills Growth
[9] |  Experience Report |  Lessons Learned |  Organization |  Skills Growth
[162] |  Validation Research |  Lessons Learned |  Organization |  Skills Growth
[142] |  Philosophical Paper |  Theory |  Organization |  Skills Growth
[48] |  Validation Research |  Theory |  Organization |  Skills Growth
[62] |  Experience Report |  Lessons Learned |  Organization |  Skills Growth
[69] |  Validation Research |  Lessons Learned |  Organization |  Skills Growth
[36] |  Experience Report |  Advice |  Business |  Mean Time To Value
[9] |  Experience Report |  Lessons Learned |  Business |  Mean Time To Value
[164] |  Opinion Paper |  Advice |  Business |  Mean Time To Value
[105] |  Opinion Paper |  Lessons Learned |  Business |  Costs
[18] |  Validation Research |  Theory |  Business |  Costs
[58] |  Validation Research |  Guideline |  Business |  Costs
[35] |  Validation Research |  Theory |  Business |  Costs
[39] |  Validation Research |  Model |  Business |  Costs
[62] |  Experience Report |  Lessons Learned |  Business |  Costs
[49] |  Opinion Paper |  Theory |  Business |  Costs
[80] |  Philosophical Paper |  Theory |  Business |  Costs
[165] |  Philosophical Paper |  Theory |  Business |  Costs
[40] |  Evaluation Research |  Tool |  Business |  Ecosystem Valuation
[166] |  Philosophical Paper |  Theory |  Business |  Ecosystem Valuation
[89] |  Opinion Paper |  Lessons Learned |  Business |  Ecosystem Valuation
[162] |  Opinion Paper |  Theory |  Business |  Ecosystem Valuation
[138] |  Evaluation Research |  Method |  Business |  Ecosystem Valuation
[167] |  Evaluation Research |  Framework |  Business |  Ecosystem Valuation
[155] |  Philosophical Paper |  Theory |  Business |  Ecosystem Valuation
[69] |  Evaluation Research |  Theory |  Business |  Ecosystem Valuation
[66] |  Validation Research |  Model |  Business |  Ecosystem Valuation
[168] |  Opinion Paper |  Theory |  Business |  Product Market Share
[77] |  Evaluation Research |  Guideline |  Business |  Product Market Share
[86] |  Opinion Paper |  Guideline |  Business |  Product Market Share
[162] |  Opinion Paper |  Theory |  Business |  Product Market Share
[136] |  Opinion Paper |  Advice |  Business |  Product Market Share
[146] |  Evaluation Research |  Method |  Business |  Product Market Share
[170] |  Opinion Paper |  Lessons Learned |  Business |  Product Market Share
[157] |  Philosophical Paper |  Guideline |  Business |  Product Market Share
[35] |  Validation Research |  Theory |  Business |  Product Market Share
[169] |  Evaluation Research |  Method |  Business |  Product Market Share
[171] |  Evaluation Research |  Tool |  Business |  Value Stream Mapping
[172] |  Opinion Paper |  Advice |  Business |  Value Stream Mapping
[144] |  Opinion Paper |  Advice |  Business |  Value Stream Mapping
[86] |  Opinion Paper |  Guideline |  Business |  Value Stream Mapping
[87] |  Solution Proposal |  Guideline |  Business |  Value Stream Mapping
[57] |  Philosophical Paper |  Theory |  Business |  Value Stream Mapping
[173] |  Experience Report |  Lessons Learned |  Business |  Value Stream Mapping
[157] |  Philosophical Paper |  Guideline |  Business |  Value Stream Mapping
|  |  |  |

## Bibliography

  * [1] Philipp Haindl and Reinhold Plösch  “Focus Areas, Themes, and Objectives of Non-Functional Requirements in DevOps: A Systematic Mapping Study”  In _2020 46th Euromicro Conference on Software Engineering and Advanced Applications (SEAA)_ , 2020, pp. 394–403  DOI: 10.1109/SEAA51224.2020.00071
  * [2] L. Chung, B.A. Nixon, E. Yu and J. Mylopoulos  “Non-Functional Requirements in Software Engineering” 5, International Series in Software Engineering  Springer US, 2000
  * [3] Lawrence Chung and Julio Cesar Prado Leite  “On Non-Functional Requirements in Software Engineering”  In _c_ Berlin, Heidelberg: Springer, 2009, pp. 363–379
  * [4] Maya Daneva, Luigi Buglione and Andrea Herrmann  “Software Architects’ Experiences of Quality Requirements: What We Know and What We Do Not Know?”  In _Requirements Engineering: Foundation for Software Quality_ , Lecture Notes in Computer Science  Springer Berlin Heidelberg, 2013, pp. 1–17
  * [5] J. Doerr et al.  “Non-functional requirements in industry - three case studies adopting an experience-based NFR method”  In _13th IEEE International Conference on Requirements Engineering (RE’05)_ , 2005, pp. 373–382  DOI: 10.1109/RE.2005.47
  * [6] Volker Gruhn and Clemens Schäfer  “BizDevOps: Because DevOps is Not the End of the Story”  In _Intelligent Software Methodologies, Tools and Techniques_ , Communications in Computer and Information Science  Springer International Publishing, 2015, pp. 388–398
  * [7] Peter Forbrig and Anke Dittmar  “Integrating HCD into BizDevOps by Using the Subject-Oriented Approach”  In _Human-Centered Software Engineering_ , Lecture Notes in Computer Science  Springer International Publishing, 2019, pp. 327–334
  * [8] Ramtin Jabbari, Nauman Ali, Kai Petersen and Binish Tanveer  “What is DevOps?: A Systematic Mapping Study on Definitions and Practices”  ACM, 2016, pp. 12  DOI: 10.1145/2962695.2962707
  * [9] Mali Senapathi, Jim Buchan and Hady Osman  “DevOps Capabilities, Practices, and Challenges: Insights from a Case Study”  In _Proceedings of the 22Nd International Conference on Evaluation and Assessment in Software Engineering 2018_ , EASE’18  New York, NY, USA: ACM, 2018, pp. 57–67  DOI: 10.1145/3210459.3210465
  * [10] Erik Beulen  “Implementing and Contracting Agile and DevOps: A Survey in the Netherlands”  In _Digital Services and Platforms. Considerations for Sourcing_ Springer, Cham, 2018, pp. 124–146  DOI: 10.1007/978-3-030-15850-7_7
  * [11] Aymeric Hemon, Barbara Lyonnet, Frantz Rowe and Brian Fitzgerald  “From Agile to DevOps: Smart Skills and Collaborations”  In _Information Systems Frontiers_ , 2019, pp. 1–19  DOI: 10.1007/s10796-019-09905-1
  * [12] Len Bass, Ingo Weber and Liming Zhu  “DevOps: A Software Architect’s Perspective”  New York: Addison-Wesley Professional, 2015
  * [13] D.. Tamburri, R. Kazman and H. Fahimi  “The Architect’s Role in Community Shepherding”  In _IEEE Software_ 33.06, 2016, pp. 70–79  DOI: 10.1109/MS.2016.144
  * [14] Peter Freeman  “Software Perspectives: The System is the Message”  Boston, MA, USA: Addison-Wesley Longman Publishing Co., Inc., 1987
  * [15] Y. Liu, Z. Ma and W. Shao  “Integrating Non-functional Requirement Modeling into Model Driven Development Method”  In _2010 Asia Pacific Software Engineering Conference_ , 2010, pp. 98–107  DOI: 10.1109/APSEC.2010.21
  * [16] M. Broy  “Rethinking Nonfunctional Software Requirements”  In _Computer_ 48.5, 2015, pp. 96–99  DOI: 10.1109/MC.2015.139
  * [17] J. Eckhardt, A. Vogelsang and D.. Fernández  “Are "Non-functional" Requirements really Non-functional? An Investigation of Non-functional Requirements in Practice”  In _2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE)_ , 2016, pp. 832–842  DOI: 10.1145/2884781.2884788
  * [18] D. Ameller, X. Franch and J. Cabot  “Dealing with Non-Functional Requirements in Model-Driven Development”  In _2010 18th IEEE International Requirements Engineering Conference_ , 2010, pp. 189–198  DOI: 10.1109/RE.2010.32
  * [19] H. Hiisilä and M. Kujala  “Combining Process Modeling and Requirements Engineering: An Experience Report”  In _2015 IEEE 17th Conference on Business Informatics_ 1, 2015, pp. 242–249  DOI: 10.1109/CBI.2015.20
  * [20] M. Glinz  “On Non-Functional Requirements”  In _15th IEEE International Requirements Engineering Conference (RE 2007)_ , 2007, pp. 21–26  DOI: 10.1109/RE.2007.45
  * [21] Dewi Mairiza, Didar Zowghi and Nurie Nurmuliani  “An Investigation into the Notion of Non-functional Requirements” event-place: Sierre, Switzerland  In _Proceedings of the 2010 ACM Symposium on Applied Computing_ , SAC ’10  New York, NY, USA: ACM, 2010, pp. 311–317  DOI: 10.1145/1774088.1774153
  * [22] M. Hasan, Pericles Loucopoulos and Mara Nikolaidou  “Classification and Qualitative Analysis of Non-Functional Requirements Approaches”  In _Enterprise, Business-Process and Information Systems Modeling_ , Lecture Notes in Business Information Processing  Springer Berlin Heidelberg, 2014, pp. 348–362
  * [23] S. Ouhbi, A. Idri, J.. Fernández-Alemán and A. Toval  “Software Quality Requirements: A Systematic Mapping Study”  In _2013 20th Asia-Pacific Software Engineering Conference (APSEC)_ 1, 2013, pp. 231–238  DOI: 10.1109/APSEC.2013.40
  * [24] Leah Riungu-Kalliosaari et al.  “DevOps Adoption Benefits and Challenges in Practice: A Case Study”  In _Product-Focused Software Process Improvement_ , Lecture Notes in Computer Science  Springer International Publishing, 2016, pp. 590–597
  * [25] Barbara A. Kitchenham, David Budgen and O. Pearl Brereton  “Using Mapping Studies As the Basis for Further Research - A Participant-observer Case Study”  In _Inf. Softw. Technol._ 53.6, 2011, pp. 638–651  DOI: 10.1016/j.infsof.2010.12.011
  * [26] Claes Wohlin  “Guidelines for Snowballing in Systematic Literature Studies and a Replication in Software Engineering” event-place: London, England, United Kingdom  In _Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering_ , EASE ’14  New York, NY, USA: ACM, 2014, pp. 38:1–38:10  DOI: 10.1145/2601248.2601268
  * [27] Kai Petersen, Robert Feldt, Shahid Mujtaba and Michael Mattsson  “Systematic mapping studies in software engineering”  In _EASE’08 Proceedings of the 12th international conference on Evaluation and Assessment in Software Engineering_ , 2008, pp. 68–77
  * [28] Philipp Haindl and Reinhold Plösch  “Towards Continuous Quality: Measuring and Evaluating Feature-Dependent Non-Functional Requirements in DevOps”  In _2019 IEEE International Conference on Software Architecture Companion (ICSA-C)_ , 2019, pp. 91–94  DOI: 10.1109/ICSA-C.2019.00024
  * [29] Philipp Haindl, Reinhold Plösch and Christian Körner  “An Extension of the QUAMOCO Quality Model to Specify and Evaluate Feature-Dependent Non-Functional Requirements”  In _2019 45th Euromicro Conference on Software Engineering and Advanced Applications (SEAA)_ , 2019, pp. 19–28  DOI: 10.1109/SEAA.2019.00012