# Understanding the Issues, Their Causes and Solutions in Microservices
Systems:
An Empirical Study

Muhammad Waseem, Peng Liang, Aakash Ahmad, Arif Ali Khan,
Mojtaba Shahin, Pekka Abrahamsson, Ali Rezaei Nasab, and Tommi Mikkonen
Muhammad Waseem, Peng Liang, and Ali Rezaei Nasab are with the School of
Computer Science, Wuhan University, Wuhan, China. E-mail: {m.waseem,
<EMAIL_ADDRESS><EMAIL_ADDRESS>Aakash Ahmad is with the School
of Computing and Communications, Lancaster University Leipzig, Leipzig,
Germany. E-mail<EMAIL_ADDRESS>Arif Ali Khan is with the M3S
Empirical Software Engineering Research Unit, University of Oulu, Oulu,
Finland. E-mail<EMAIL_ADDRESS>Mojtaba Shahin is with the School of
Computing Technologies, RMIT University, Melbourne, Australia. E-mail:
<EMAIL_ADDRESS>Pekka Abrahamsson is with the Faculty of
Information Technology and Communication Sciences, Tampere University,
Tampere, Finland. E-mail<EMAIL_ADDRESS>Tommi Mikkonen is with the
Faculty of Information Technology, University of Jyväskylä, Jyväskylä,
Finland. E-mail<EMAIL_ADDRESS>Manuscript received; revised.
(Corresponding author: Peng Liang.)

###### Abstract

Many small to large organizations have adopted the Microservices Architecture
(MSA) style to develop and deliver their core businesses. Despite the
popularity of MSA in the software industry, there is a limited evidence-based
and thorough understanding of the types of issues (e.g., errors, faults,
failures, and bugs) that microservices system developers experience, the
causes of the issues, and the solutions as potential fixing strategies to
address the issues. To ameliorate this gap, we conducted a mixed-methods
empirical study that collected data from 2,641 issues from the issue tracking
systems of 15 open-source microservices systems on GitHub, 15 interviews, and
an online survey completed by 150 practitioners from 42 countries across 6
continents. Our analysis led to comprehensive taxonomies for the issues,
causes, and solutions. The findings of this study inform that Technical Debt,
Continuous Integration and Delivery, Exception Handling, Service Execution and
Communication, and Security are the most dominant issues in microservices
systems. Furthermore, General Programming Errors, Missing Features and
Artifacts, and Invalid Configuration and Communication are the main causes
behind the issues. Finally, we found 177 types of solutions that can be
applied to fix the identified issues. Based on our study results, we
formulated future research directions that could help researchers and
practitioners to engineer emergent and next-generation microservices systems.

###### Index Terms:

Microservices System, Microservices Architecture, Issues, Open Source
Software, Empirical Study

##  1 Introduction

The software industry has recently witnessed the growing popularity of the
Microservices Architecture (MSA) style as a promising design approach to
develop applications that consist of multiple small, manageable, and
independently deployable services [1]. Software development organizations may
have adopted or planned to use the MSA style for various reasons.
Specifically, some of them want to increase the scalability of applications
using the MSA style, while others use it to quickly release new products and
services to the customers, whereas it is argued that the MSA style can also
help build autonomous development teams [2, 3]. From an architectural
perspective, a microservices system (a system that adopts the MSA style)
entails a significant degree of complexity both at the design phase as well as
at runtime configuration [4]. This implies that the MSA style brings unique
challenges for software organizations, and many quality attributes may be
(positively or negatively) influenced [5]. For example, service level security
may be impacted because microservices are developed and deployed by various
technologies (e.g., Docker containers [6]) and tools that are potentially
vulnerable to security attacks [4, 7]. Data management is also influenced
because each microservice needs to own its domain data and logic [8]. This
can, for example, challenge achieving and managing data consistency across
multiple microservices.

Zimmermann argues that MSA is not entirely new from Service-Oriented
Architecture (SOA) (e.g., “microservices constitute one particular
implementation approach to SOA – service development and deployment”) [9].
Similarly, Márquez and Astudillo discovered that some existing design
rationale and patterns from SOA fit the context for MSA [10]. However, an
important body of literature (e.g., [11, 3]) has concluded that there are
overwhelming differences between microservices systems, monolithic systems,
and traditional service-oriented systems in terms of design, implementation,
testing, and deployment. Gupta and Palvankar indicated that even having SOA
experience can lead to suboptimal decisions (e.g., excessive service calls) in
microservices systems [12]. Hence, microservices systems may have an
additional and specific set of issues. Borrowing the idea from [13], we define
issues in this study as errors, faults, failures, and bugs that occur in a
microservices system and consequently impact its quality and functionality.
Hence, there is a need to leverage existing methods or derive new practices,
techniques, and tools to address the specific and additional issues in
microservices systems.

Recently, a number of studies have investigated particular issues (e.g., code
smell [14], debugging [15], performance [16]) in microservices systems.
Despite these efforts, there is no in-depth and comprehensive study on the
nature of different types of issues that microservices developers face, the
potential causes of these issues, and possible fixing strategies for these
issues. Jamshidi et al. believed that this can be partially attributed to the
fact that researchers have limited access to industry-scale microservices
systems [3]. The empirical knowledge on the nature of issues occurring in
microservices systems can be useful from the following perspectives: (i)
understanding common issues in the design and implementation of microservices
systems and how to avoid them, (ii) identifying trends in the types of issues
that arise in microservices systems and how to address them effectively, (iii)
experienced microservices developers can be allocated to address the most
frequent and challenging issues, (iv) novice microservices developers can
quickly be informed of empirically-justified issues and avoid common mistakes,
and (v) the industry and academic communities can synergies theory and
practice to develop tools and techniques for the frequently reported issues in
microservices systems.

Motivating Example: We now contextualize the issues, causes, and solutions
based on an example illustrated in Figure 1. The example is taken from the
Spinnaker project, an open-source microservices project hosted on GitHub (see
Table I), and annotated with numbering to represent a sequence among the
reported issue, its cause(s), and the solution(s) to resolve the issue. Figure
1 shows auxiliary information about the Spinnaker project, such as project
description, stars, and contributors. As shown in the example, a contributor,
typically a microservices developer, writes code and may provide additional
details of the code through comments. Once the code is compiled, the
contributor reports permission denied issue highlighting “pipeline save with
Admin account fails with permission denied”. As the next step, the same or
other contributor highlights the cause for such issue as “Spinnaker user does
not have access to the service account”. As the last step, an individual or a
community of developers provides a solution such as “Allow the admin users to
save the accounts” to resolve the issue. We are only interested in analyzing
issues that have been marked as closed to ensure that a solution to resolve
the issue exists. As shown in Table I, the Spinnaker project has 4,595 closed
issues and 121 contributors.

Figure 1: An example of the issue, cause, and their solution

This work aims to systematically and comprehensively study and categorize the
issues that developers face in developing microservices systems, the causes of
the issues, and the solutions (if any). To this end, we conducted a mixed-
methods empirical study following the guideline proposed by Easterbrook and
his colleagues [17]. We collected the data by (i) mining 2,641 issues from the
issue tracking systems of 15 open-source microservices systems on GitHub, (ii)
conducting 15 interviews, and (iii) deploying an online survey completed by
150 practitioners to develop the taxonomies of the issues, their causes and
solutions in microservices systems. The key findings of this study are:

  1. 1.

The issue taxonomy consists of 19 categories, 54 subcategories, and 402 types,
indicating the diversity of issues in microservices systems. The top three
categories of issues are Technical Debt, Continuous Integration and Delivery,
and Exception Handling.

  2. 2.

The cause taxonomy consists of 8 categories, 26 subcategories, and 228 types,
in which General Programming Errors, Missing Features and Artifacts, and
Invalid Configuration and Communication are the most frequently reported
causes.

  3. 3.

The solution taxonomy consists of 8 categories, 32 subcategories, and 177
types of solutions, in which the top three categories of solutions for
microservices issues are Fix Artifacts, Add Artifacts, and Modify Artifacts.

  4. 4.

The overall survey findings confirm the taxonomies of the issues, their causes
and solutions in microservices systems and also indicate no major
statistically significant differences in practitioners’ perspectives on the
developed taxonomies.

The empirically validated taxonomies of issues, causes, and solutions in this
study may greatly benefit software developers. For instance, developers can
gain valuable insights into the diverse range of issues that they may
encounter. These taxonomies can also help developers understand the causes of
issues and implement preventive measures, enabling them to make informed
decisions about addressing the issues during the development process.
Furthermore, the proposed taxonomies in this study can facilitate knowledge
sharing, training, and ensure a shared understanding among developers about
the issues, causes, and solutions in microservices systems.

This paper has extended our previous work [18] by adding two new research
questions (RQ3 and RQ4) and expanding and enhancing the results of RQ1 and RQ2
with increased volume and variety of data and a mixed-methods approach.
Specifically, we explored 10 more open-source microservices systems on GitHub
(now 15 projects), interviewed 15 practitioners, and conducted an online
survey with 150 microservices practitioners for getting their perspectives on
the proposed taxonomies of the issues, their causes and solutions in
microservices systems, as well as the mapping between the issues, causes, and
solutions.

Our study makes the following key contributions:

  1. 1.

We developed the taxonomies of the issues, their causes and solutions in
microservices systems based on a qualitative and quantitative analysis of
2,641 issue discussions among developers on GitHub, 15 interviews, and an
online survey completed by 150 practitioners.

  2. 2.

We provided the mapping between the issues, causes, and solutions in
microservices systems with promising research directions on microservices
systems that require more attention.

  3. 3.

We made the dataset of this study available online [19], which includes the
data collection and analysis from GitHub and microservices practitioners, as
well as detailed hierarchies of the taxonomies of issues, causes, and
solutions, to enable replication of this study and conduct future research.

The remainder of the paper is structured as follows. Section 2 details the
research methodology employed. Section 3 presents the results of our study.
Section 4 discusses the relationship between the issues, causes, and
solutions, along with the implications and the threats to the validity of our
results. Section 6 reviews related work, and Section 7 draws conclusions and
outlines avenues for future work.

##  2 Methodology

The research methodology of this study consists of three phases, as
illustrated in Figure 2. Given the nature of this research and the formulated
research questions (see Section 2.1) – issues, causes, and solutions in
microservices projects, we decided to use a mixed-methods study. Our study
collected data from microservices projects hosted on GitHub, interviews, and a
web-based survey. During Phase 1, we derived the taxonomies of 386 types of
issues, 217 types of causes, and 177 types of solutions by mining and
analyzing microservices practitioners’ discussions in the issue tracking
systems of 15 open-source microservices projects hosted on GitHub. During
Phase 2, we interviewed 15 microservices practitioners to extend and verify
the taxonomies and identified additional 14 types of issues, 20 types of
causes, and 22 types of solutions. During Phase 3, we surveyed 150
microservices practitioners using a Web-based survey to validate the outcomes
of Phase 1 and Phase 2, i.e., using practitioners’ perspectives and feedback
to validate the extracted types of issues, their causes and solutions.

###  2.1 Research Questions

We formulated the following research questions (RQs).

RQ1: What issues do occur in the development of microservice systems?

Rationale: RQ1 aims to systematically identify and taxonomically classify the
types of issues that occur in microservices systems. The answer to RQ1
provides a comprehensive understanding of the issues (e.g., the most frequent
issues) of microservices systems.

RQ2: What are the causes of issues that occur in microservices systems?

Rationale: The aim of RQ2 is to investigate and classify the root causes
behind the issues identified in RQ1 and map causes to issues. The answer to
RQ2 helps practitioners avoid common issues in microservices systems.

RQ3: What solutions are proposed to fix issues that occur in microservices
systems?

Rationale: The aim of RQ3 is to identify the solutions for the issues
according to their causes and to develop the taxonomy of solutions. The answer
to RQ3 helps to understand the fixing strategies for addressing microservices
issues.

RQ4: What are the practitioners’ perspectives on the taxonomies of the
identified issues, causes, and solutions in microservices systems?

Rationale: The taxonomies of issues, causes, and solutions developed in RQ1,
RQ2, and RQ3 are based on 15 open-source microservices systems and
interviewing 15 practitioners. RQ4 aims to evaluate the taxonomies of issues,
causes, and solutions built in RQ1, RQ2, and RQ3 by conducting a relatively
large-scale online survey.

Figure 2: An overview of the research method

###  2.2 Phase 1 - Mining Developer Discussions

This phase aims to systematically identify and synthesize the issues, their
causes and solutions in open-source microservices systems on GitHub. For an
objective and fine granular presentation of methodological details, this phase
is divided into two steps, each elaborated below, based on the illustrative
view in Figure 2.

####  2.2.1 Step A – Identify & Select MSA-based OSS Projects

The specified RQs require us to identify and select MSA-based OSS projects,
representing a repository of developer discussions and knowledge, to extract
the issues, their causes, and solutions. This means that the RQs guided the
development of search strings based on the recommendations and steps from [20]
for string composition to retrieve developer discussions on microservices
projects deployed on GitHub [21]. We formulated the search string using the
format [keyword-1 [OR logic] keyword-2 … [OR logic] … keyword-N], where
keywords represented the synonyms as [‘micro service’ OR ‘micro-service’ OR
‘microservice’ OR ‘Micro service’ OR ‘Micro-service’ OR ‘Microservice’]. To
extract MSA issues, we selected GitHub, which is one of the most popular and
rapidly growing platforms for social coding and community-driven collaborative
development of OSS. GitHub represents a modern genre of software forges that
unifies traditional methods of development (e.g., version control, code
hosting) with features of socio-collaborative development (e.g., issue
tracking, pull requests) [22]. The variety and magnitude of OSS available on
GitHub also inspired our choice to investigate the largest OSS platform in the
world, with approximately 40 million users and 28 million publicly available
project repositories.

Based on the search string, we searched for the title and description of the
OSS projects deployed on the GHTorrent dump hosted on Google Cloud. The search
helped us retrieve a total of 2,690 potentially relevant MSA-based OSS
projects for investigation. To shortlist and eventually select the projects
pertinent to the outlined RQs, we applied multi-criteria filtering [23],
considering a multitude of aspects such as the popularity or perceived
significance of a project in the developers’ community (represented as total
stars), adoption by or interests of developers (total forks), and the total
number of developers involved (total contributors) for the project. As shown
in Figure 2 (Step B), we only selected the projects that have (i) more than 10
stars and forks, (ii) an English description, and (iii) three or more
contributors. This led to a shortlist of 167 microservices projects. To
eliminate the instances of potential false positives, i.e., avoiding bias in
construct validity, such as misleading project names and mockup code, we
contacted the top three contributors of each project via their emails to
clarify about:

  1. 1.

Correct Interpretation of the Project: Please confirm if our interpretation of
your project (Project URL and Name as an identifier) is appropriate for its
design and implementation based on MSA. Also, please help us clarify if this
project (e.g., tool, framework, solution) supports the development of
microservices systems or if this project is developed using MSA.

  2. 2.

MSA-based Characteristics and/or Features of the Project: What features and/or
characteristics of the project reflect MSA being used in the project?

  3. 3.

Optional Question: Could you please help us identify (the names, URLs, etc.
of) any other OSS projects that are designed or developed using MSA?

We contacted a total of 426 contributors, with 39 of them responded to our
query. Based on their confirmation, we selected 15 MSA-based OSS projects (see
Table I). These 15 selected projects, based on the available data at the time
of project selection, represent the highest number of issues, making them
highly relevant to our study.

TABLE I: Identified open-source microservices systems Project Name | #Issues | #Contributors | #Forks | #Stars
---|---|---|---|---
Spinnaker | 4595 | 121 | 1.2K | 8.5K
Cortex | 1120 | 226 | 681 | 4.7K
Jaeger | 995 | 239 | 1.9K | 15.8K
eShopOnContainers | 986 | 157 | 8.9K | 20.7K
Goa | 930 | 97 | 485 | 4.7K
Light-4j | 584 | 37 | 588 | 3.4K
Moleculer | 473 | 102 | 497 | 5.1K
Microservices-demo | 287 | 55 | 2.1K | 3.1K
Cliquet | 207 | 19 | 20 | 65
Deep-framework | 174 | 12 | 75 | 537
Scalecube | 130 | 20 | 90 | 547
Lelylan | 123 | 7 | 93 | 1.5K
Open-loyalty | 175 | 14 | 80 | 300
Spring PetClinic | 69 | 32 | 1.4K | 1.1K
Pitstop | 39 | 15 | 490 | 890

####  2.2.2 Step B – Synthesize Issues, Causes, and Solutions

After the projects were identified, as illustrated in Figure 2, extracting and
synthesising the issues was divided into the following five parts.

Raw Data Collection: We chose 15 microservices projects (see Table I) as the
source for building the dataset to answer our RQs. These 15 projects were
chosen because they are significantly larger than other microservices projects
hosted on GitHub. Hence, it is highly likely that their contributors had more
discussions about the type of issues, causes, and solutions in issue tracking
systems. The discussions relating to a software system can usually be captured
in issue tracking systems [24]. We initially extracted 10,222 issue titles,
issue links, issue opening and closing dates, and the number of contributors
for each issue through our customized Python script (see the Raw Data sheet in
[19]). We stored this information in MySQL and exported it into MS Excel
sheets for further processing. We extracted only closed issues because it
could increase the chances of answering all our RQs (e.g., solutions).

Issues Screening: The first author further scanned the 10,222 issues to check
if an issue has been closed or is still open. All open issues were discarded
because an open issue is ongoing with many of its causes unknown and most
likely solution(s) not found. Furthermore, after selecting only the closed
issues, the first author further eliminated (i) issues without a detailed
description, (ii) general questions, opinions, feedback, and ideas, (iii)
feature requests (e.g., enhancements, proposals), (iv) announcements (e.g.,
about new updates), (v) duplicated issues, (vi) issues that had only one
participant, and (vii) stale issues. After this step, we got 5,115 issues (see
the Selected Issues (Round 1) sheet in [19]). We had second round of screening
on these 5,115 issues to check whether these issues are related to our RQs or
not, and after comprehensively analyzing them we found 2,641 issues that were
related to our RQs (see the Selected Issues (Round 2) sheet in [19]).

Pilot Data Extraction: To gain initial insights into the issues, two authors
(i.e., the first and fourth) performed pilot data extraction based on 150
issues (5.67% of 2,641 issues). The authors focused on issue-specific data,
such as issue description (i.e., textual details specified by contributors),
type of issue (e.g., testing issue, deployment issue), and frequency of issue
(i.e., number of occurrences). Pilot data extraction was checked by the second
and third authors to verify and refine the details before final data
extraction.

Data Extraction: Issues, causes, solutions, and their corresponding data were
extracted based on the guidelines for mining software engineering data from
GitHub [25]. We extracted issue titles, issue links, the number of
participants for each issue, key points of issues, key points of causes, and
key points of solutions (see the Raw Data and Initial Codes sheet in [19]).
Data such as issue title, issue link, and number of participants for each
issue were used to capture general information about developer discussions. On
the other hand, key points of issues, causes, and solutions were used to
document the data for answering RQ1, RQ2, RQ3.

Data Analysis: To synthesize the issues, we used the thematic analysis
approach [26] to identify the categories of issues, causes, and solutions. The
thematic analysis approach is composed of five steps. (i) Familiarizing with
data: The first author repeatedly read the project’s contributor’s discussion
and documented all discussed key points about issues, causes, and solutions.
(ii) Generating initial codes: after data familiarization, the first author
generated an initial list of codes from the extracted data (see the Initial
Codes sheet in [19]). (iii) Searching for the types of issues, causes, and
solutions: The first and second authors analyzed the initially generated codes
and brought them under the specific types of issues,causes, and solutions.
(iv) Reviewing types of issues, causes, and solutions: All the authors
reviewed and refined the coding results with the corresponding types of
issues, causes, and solutions. We separated, merged, and dropped several
issues, causes, and solutions based on a mutual discussion between all the
authors. (v) Defining and naming categories: We defined and further refined
all the types of issues, causes, and solutions under precise and clear
subcategories and categories (see Figure 4). Throughout this process, we
engaged in the collaborative refinement and organization of issues, causes,
and solutions into specific subcategories and categories. The naming and
assignment of issues, causes, and solutions in microservices systems to the
appropriate subcategories were conducted through consensus among all the
authors, ensuring the development of a comprehensive and coherent taxonomy.
Finally, we introduced three levels of categories for managing the identified
issues, causes, and solutions. First, we organized the types of issues,
causes, and solutions under a specific subcategory (e.g., service dependency
in Service Design Debt). Then we arranged the subcategories under a specific
category (e.g., Service Design Debt in Technical Debt).

To minimize any bias during data analysis, each step, i.e., classification,
mapping, and documentation, were cross-checked and verified independent of the
individual(s) involved in data synthesis. Documentation of the results as
answers to RQs is presented as taxonomical classification of issues (Section
3.1), causes of issues (Section 3.2), solutions to address the issues (Section
3.3), and mapping of issues with their causes and solutions (Section 4.1),
complemented by details of validity threats (Section 5).

###  2.3 Phase 2 - Conducting Practitioner Interviews

Since we aim to understand the issues, causes, and solutions of microservices
systems from a practitioners’ perspective, we opted to conduct interviews to
confirm and improve the developed taxonomies. The interview process consists
of the following steps.

####  2.3.1 Preparing a Protocol

The first author conducted 15 online interviews with microservices
practitioners through Zoom, Tencent Meeting, and Microsoft Teams. Before
conducting actual interviews, we also conducted two pilot interviews with
microservices practitioners to check the understandability and
comprehensiveness of interview questions. However, we did not include their
answers in our dataset. In total, we conducted 15 actual interviews, and each
interview took 35-45 minutes. It is argued that conducting 12 to 15 interviews
with homogeneous groups is enough to reach saturation [27]. In our case, after
completing 15 interviews, we noticed a clear saturation in the responses to
our interview questions, indicating that we had gathered enough information to
address our interview objectives effectively. Therefore, we decided not to
conduct any additional interviews. We conducted semi-structured interviews
based on an interview guide (see the Interview Questionnaire sheet in [19]),
which contains closed-ended questions related to participant demographics and
open-ended questions related to general categories of microservices issues,
causes, and solutions.

The interview process was comprised of three sections. In the first section,
we asked 6 demographic questions to understand the interviewee’s background in
microservices. We covered various aspects in this section, including the
country of the practitioner, major responsibilities, overall experience in the
IT industry, experience with implementing microservices systems, the work
domain of the organization, and programming languages for developing
microservices systems. In the second part, we asked three open-ended questions
about the types of issues, causes of issues, and solutions to issues during
the development of microservices systems. The purpose of this part was to
allow the interviewees to spontaneously express their views about the issues
developers face in developing microservices systems, the causes of the issues,
and resolution strategies without the interviewer biasing their responses. In
the third part, we presented the three taxonomies to the interviewees and
asked them to indicate any missing issues, causes, and solutions that have not
been explicitly mentioned. All three taxonomies come from identifying,
analyzing, and synthesizing the developer discussions from 15 open-source
microservices systems. The taxonomy of issues consists of 386 types of issues,
54 issue subcategories, and 18 issue categories. The taxonomy of causes
consists of 217 types of causes, 26 cause subcategories, and 8 cause
categories. The taxonomy of solutions consists of 171 types of solutions, 33
solution subcategories, and 8 solution categories. At the end of each
interview, we thanked the interviewee and briefly informed them of our next
plans.

####  2.3.2 Conducting Interviews

To recruit interviewees, we initially reached out to our professional contacts
in each country via email. We also requested that they disseminate our
interview request among their colleagues who they believed would be suitable
candidates for the study. We informed the possible participants that this
interview was entirely voluntary with no compensation. With this approach, we
recruited 15 microservices practitioners from IT companies in 10 countries:
Australia (1), Canada (3), China (1), Chile (1), India (1), Pakistan (2),
Sweden (2), Norway (1), the United Kingdom (1), and the United States of
America (2). We refer to the interviewees as P1 to P15. Most of the
interviewees are mainly software architects and application developers. Their
average experience in the IT industry is 10.33 years (Minimum: 5, Maximum: 16,
Median: 10, Mode: 9, Standard Deviations: 3.26). The interviewees’ average
experience in microservices is 5.33 years (Minimum: 3, Maximum: 8, Mode: 4,
Median: 5, Standard Deviations: 1.67).

####  2.3.3 Data Analysis

We applied a thematic analysis method [28] to analyze the recorded interviews.
Before applying the thematic analysis method, the first author prepared the
text transcripts from audio recordings. The first author read the interview
transcripts and coded them using the MAXQDA tool. We dropped several sentences
unrelated to “microservices issues, causes, and solutions”. After removing the
extraneous information from the transcribed interviews, the first author coded
the interview transcripts’ contents to get the answers to the interview
questions. To ensure the quality of the codes, the second author verified the
initial codes created by the first author and provided suggestions for
improvement. After the initial coding and subsequent enhancements, all the
authors collaborated in analyzing the codes. Conflicts in the codes were
addressed through collaborative discussions and brainstorming sessions.
Finally, we generated a total of 28 types of issues (classified into 15
subcategories and 11 categories), 28 types of causes (classified into 15
subcategories and 7 categories), and 30 types of solutions (classified into 4
subcategories and 3 categories). During the interviews, we got 48 instances of
issues, 31 instances of causes, and 36 instances of solutions. Among these
instances, we identified 14 types of issues, 21 types of causes, and 23 types
of solutions that were not part of the taxonomies we derived from the 15 open-
source microservices systems. Later, we exported the analyzed interview data
from the MAXQDA tool to an MS Excel sheet (i.e., the Interview Results sheet
in [19]) to make the part of taxonomies of issues, causes, and solutions in
microservices systems from the interview data.

Except for service size, operational and tooling overhead, and team management
issues, all of the issues, causes, and solutions identified through the
interviews can be classified under the existing categories (i.e., the output
of Phase 1). The instances of issues mentioned by the interviewees include
CI/CD Issues (9), Security Issues (6), Service Execution and Communication
Issues (5), Database Issues (5), Organizational Issues (5), Testing Issues
(5), Monitoring Issues (4), Performance Issues (4), Update and Installation
Issues (3), Configuration Issues (1), and Technical Debt (1). The causes
mentioned by the interviewees include Service Design and Implementation
Anomalies (20), Poor Security Management (2), Legacy Versions, Compatibility,
and Dependency Problems (2), Invalid Configuration and Communication Problems
(2), General Programming Errors (2), Fragile Code (2), and Insufficient
Resources (1). The solutions mentioned by the interviewees include Add
Artifacts (32) and Upgrade Tools and Platforms (4).

TABLE II: Overview of the interviewees and their demographic information # |  Responsibilities |  Languages |  Domain |  Overall Exp. |  MSA Exp. |  Country
---|---|---|---|---|---|---
P1 |  Software Architect, Developer |  Python, Java, Node.JS |  E-commerce, Healthcare |  14 Years |  6 Years |  Sweden
P2 |  Developer |  Java with Spark |  E-commerce |  9 Years |  5 Years |  USA
P3 |  Software Architect |  Python, Go, Java |  E-commerce |  13 Years |  6 Years |  UK
P4 |  Software Architect, Developer |  Python, Java |  Educational ERP |  7 Years |  3 Years |  Pakistan
P5 |  Software Architect |  Python, Java |  Internet of Things |  12 Years |  7 Years |  Canada
P6 |  Software Architect, Developer |  Java, Node.JS, Python |  Healthcare |  15 Years |  8 Years |  Australia
P7 |  Software Engineer |  C#.Net |  E-commerce, Banking |  10 Years |  4 Years |  Canada
P8 |  DevOps Consultant |  Kotlin, Python |  Network applications |  9 Years |  6 Years |  Canada
P9 |  Software Architect |  Java |  Education, Healthcare |  5 Years |  4 Years |  Chile
P10 |  Application Developer, Architect |  Java |  Financial (Insurance) |  10 Years |  8 Years |  Sweden
P11 |  DevOps Consultant |  Java, Kotlin,Python |  Telecommunication |  6 Years |  3 Years |  Norway
P12 |  Application Developer |  Angular, Puthon |  Manufacturing |  8 Years |  4 Years |  China
P13 |  Principal Consultant |  Swift |  Transportation |  9 Years |  5 years |  USA
P14 |  Azure Technical Engineer |  JavaScript, Golang |  Embedded systems |  12 Years |  4 Years |  Pakistan
P15 |  Software Architect |  Ruby, UML |  Payment applications |  16 Years |  7 Years |  India

###  2.4 Phase 3 - Conducting a Survey

A questionnaire-based survey approach is used to evaluate the taxonomies of
issues, causes, and solutions based on mining developer discussions and
conducting practitioner interviews. We adopted Kitchenham and Pfleeger’s
guidelines for conducting surveys [29] and used an anonymous survey to
increase response rates [30].

####  2.4.1 Recruitment of Participants and Conducting the Pilot Survey

After the survey design, we needed to (i) select the survey participants and
(ii) conduct a pilot survey for initial assessments (e.g., time taken, clarity
of statements, and add, remove, and refine the questions). To select the
potential respondents, we used the following contact channels to spread the
survey broadly to a wide range of companies from various locations worldwide.
The contact channels to recruit the potential participants included (i)
professional contacts, researchers of industrial track publications, and
authors of web blogs related to microservices, (ii) practitioners and their
communities on social coding platforms (e.g., GitHub, Stack Overflow), and
(iii) social and professional online networks (LinkedIn, Facebook, Twitter,
Google Groups). In the survey invitation email, we also requested the
potential participants to share the survey invitation with individuals or
groups deemed as relevant participants. Before sending out the invitations, we
ensured that we only contact individuals with experience in any aspects of MSA
design, development, and/or engineering based on their professional profiles,
such as code commits, industrial publications, and professional designations.
Based on publicly available email IDs, first, we sent out survey invitations
to only a selected set of 30 participants for a pilot survey. Out of the 30
participants contacted for the pilot survey, 10 replied (response rate 33.33%)
from 7 countries. It should be noted that the results of the pilot survey were
not incorporated into our primary survey data. However, the pilot survey
helped us to refine the survey questionnaire in terms of restructuring the
sections and rephrasing some questions for clarity of the survey to ensure
that (i) the length of the survey is appropriate, (ii) the terms used in the
survey questions are clear and understandable, and (iii) the answers to the
survey questions are meaningful.

Figure 3: Overview of the demographics of survey participants

####  2.4.2 Conducting the Web-based Survey

We adopted a cross-sectional survey design, which is appropriate for
collecting information at one given point in time across a sample population
[29]. Surveys can be conducted in many ways, such as Web-based online
questionnaires and phone surveys [31]. We decided to conduct a Web-based
survey because these surveys can help to (i) minimize the time and cost, (ii)
collect the responses from geographically distributed respondents, (iii)
minimize time zone constraints, and (iv) save the effort of researchers to
collect data in a textual, graphical, or structured format [31]. To document
different types of responses while maintaining the granularity of information,
we structured the questionnaire into a total of 12 questions organized under
four sections (see the Survey Questionnaire sheet in [19]).

Demographics: We asked 6 demographic questions about the background
information of the respondents to identify the (i) country or region, (ii)
major responsibilities, (iii) overall work experience in the IT industry, (iv)
work experience with microservices systems, (v) work domain of the
organization, and (vi) programming languages and implementation technologies
for developing microservices systems. The demographic information has been
collected to (i) identify respondents who do not have sufficient knowledge
about microservices, (ii) divide the results into different groups, and (iii)
generalize the survey findings for the microservices research and practice
community. We received a total of 156 responses, and we excluded 6 responses
that were either randomly filled or filled by research students and professors
who were not practitioners. In the end, we got 150 valid responses.

  * •

Countries: Respondents came from 42 countries of 6 continents (see Figure
3(a)) working in diverse teams and roles to develop microservices systems. The
majority of them are from China (16 out of 150, 10.66%), Pakistan (13 out of
150, 8.66%), and India (9 out of 150, 6.00%).

  * •

Experience: We asked the participants about their experiences in the IT
industry and the development of microservices systems. Figure 3(b) shows that
the majority of the respondents (57 out of 150, 38.00%) have worked in the IT
industry for more than 10 years and around one third of the respondents (44
out of 150, 29.33%) have worked with microservices systems for 1 to 3 years.
We also received a considerable amount of responses in which practitioners
have more than 10 years of experience working with microservices systems (20
out of 150, 13.33%).

  * •

Professional Roles: Figure 3(c) shows that the majority of the participants
were application developer (62 out of 150, 41.33%), architect (40 out of 150,
26.66%), and DevOps engineer (29 out of 150, 19.8%). Note that one participant
may have multiple major responsibilities in the company, and consequently, the
sum of the percentages exceeds 100%.

  * •

Application Domains: Figure 3(d) shows the domains of the participants’
organizations where the microservice practitioners worked. Financial Systems
(57 out of 150, 38.00%), E-commerce (29 out of 150, 19.33%), and Professional
Services (29 out of 150, 19.33%) are the dominant domains. Note that one
organization where a practitioner worked may have one or more application
domains.

  * •

Programming Languages and Implementation Technologies: Figure 3(e) shows that
38 programming languages and technologies were used to develop microservices
systems, in which “Java" (70 out of 150, 46.66%), “Python" (67 out of 150,
44.66%), and “GO" (39 out of 150, 26.00%) are the most frequently used
languages for developing microservices systems.

Microservices Practitioners’ Perspective: To evaluate the taxonomies of
issues, causes, and solutions, we asked six survey questions (both Likert
scale and open-ended, see the Survey Questionnaire sheet in [19]) from
microservices practitioners. We provided a list of 19 issue categories and
asked survey participants to respond to each category on a 5-point Likert
scale (Very Often, Often, Sometimes, Rarely, Never). Similarly, regarding
causes, we provided 8 categories and asked practitioners to respond to each
category on a 5-point Likert scale (Strongly Agree, Agree, Neutral, Disagree,
Strongly Disagree). We also provided 8 categories of solutions and asked
practitioners to respond to each category on a 5-point Likert scale. Along
with these 5-point Likert scales, we added one option to know the familiarity
of the survey respondents with the listed categories. We also asked three
open-ended questions to identify the missing issue, causes, and solutions in
the provided categories. All the open-ended responses were further analyzed
through thematic classification and adjusted in the taxonomies.

####  2.4.3 Data Analysis

We used descriptive statistics and constant comparison techniques [32] to
analyze the quantitative (i.e., closed-ended questions) and qualitative (i.e.,
open-ended questions) responses to the survey questions, respectively. To
better understand practitioners’ perspectives through Likert answers on
issues, causes, and solutions in microservices systems, we used the Wilcoxon
rank-sum test, i.e., participants who have $\leq$ 6 years of experience (49
responses) versus participants who have $\geq$ 6 years of experience with
microservices systems (101 responses). We used 6 years as a breaking point for
separating the groups because 6 years is almost in the middle of
practitioners’ experience with microservices systems. We used the symbol to
indicate a significant difference between participant groups who have
Experience $\leq$ 6 years vs. Experience $\geq$ 6 years. To analyze the
results of open-ended questions, we employed the constant comparison method
from grounded theory [32], which allowed us to systematically identify and
understand various issues, causes, and solutions from practitioners’
perspective. We began the process by coding the responses and assigning
relevant labels. Throughout the analysis, we persistently compared new
responses with existing codes (i.e., types of issues, causes, and solutions
from the taxonomies) and made refinements as needed.

##  3 Results

This section presents the analyzed results of this study, addressing the four
RQs outlined in Section 2.1. The analyzed results are further organized as
categories (e.g., Technical Debt), subcategories (e.g., Code Debt), and types
(e.g., code smell). We present categories in boldface, subcategories in
italic, and types in small capitals. The relevant examples are provided as
quoted messages along with their issue ID numbers to facilitate the
traceability to our dataset (see the Initial Codes sheet in [19]). We report
the types of issues in Section 3.1 (see Figure 4), the types of causes in
Section 3.2 (see Table III), the types of solutions in Section 3.3 (see Table
IV), and practitioners’ perspective on these taxonomies in Section 3.4 (see
Table V).

###  3.1 Types of Issues (RQ1)

The taxonomy of issues in microservices systems is provided in Figure 4. The
taxonomy of issues is derived by mining developer discussions (i.e., 2,641
instances of issues, see Section 2.2.2), conducting practitioner interviews
(i.e., 48 instances of issues, see Section 2.3.3), and conducting a survey (9
instances of issues, see Section 3.4). Therefore, we got a total of 2,698
instances of issues. The results show that Technical Debt (687 out of 2698),
Continues Integration and Delivery (313 out of 2698), and Service Execution
and Communication (219 out of 2698) issues are most frequently discussed. It
is worth noting that one issue may cover more than one topic. However, we
consider the main topic of an issue where there is a high probability of
identifying the causes or solutions. The number of issues in each issue type,
subcategory, and category are also shown in Figure 4.

1\. Technical Debt (687/2698, 25.46%): Technical Debt (TD) is “a metaphor
reflecting technical compromises that can yield short-term benefit but may
hurt the long-term health of a software system” [33]. This is the largest
category in the taxonomy of microservices issues and includes a wide range of
issues related to Code Debt and Service Design Debt. The interviewees also
mentioned several issues regarding this category, and one representative
quotation is depicted below.

“The complexity introduces several types of technical debt both in the design
and development phases of microservices systems” Developer, P2.

We identified and classified 19 types of TD issues in 2 subcategories (see
Figure 4 and the Issue Taxonomy sheet in [19]). Each of them is briefly
described below.

  * •

Code Debt (658, 24.47%) refers to the issues of source code, which could
adversely impact the code quality. This subcategory mainly gathers issues
related to code refactoring, code smell, code formatting, excessive literals,
and duplicate code. For instance, developers refactored the code of the
Spinnaker project by “adding hal command for tweaking the component sizings,
#11”. In addition, several other types of Code Debt, such as code formatting,
excessive literals, and duplicate code also negatively affect the code
legibility of microservices systems.

  * •

Service Design Debt (29, 1.07%) refers to the violation of adopting successful
practices (e.g., MSA patterns) for designing open-source microservices
systems. The issues in this subcategory are mainly related to service
dependency, business logic issue, and design pattern issue. For instance, the
developers of the moleculer project reported the issue of service design debt
in which “Service A requires module A. If service A changed, the runner
reloads, but if module A changed, the runner does not reload, #1873”.

2\. Continuous Integration and Delivery (CI/CD) Issue (313/2698, 11.60%):
CI/CD refers to the automation process that enables development teams to
frequently develop, test, deploy, and modify software systems (e.g.,
microservices systems). Usually, a variety of tools and technologies are used
to implement the CI/CD process. A wide range of CI/CD issues has been
identified by mining microservices systems. However, we also found a few
issues that the interviewees mentioned, and one representative quotation is
depicted below.

“The key issues of CI/CD for practitioners are many small independent code
bases, multiple languages, frameworks, microservices integration, load
testing, managing releases, and continued service updates”, DevOps Consultant,
P8.

We identified and classified 55 types of CI/CD issues in 7 subcategories (see
Figure 4 and the Issue Taxonomy sheet in [19]). Each of them is briefly
described below.

  * •

Deployment and Delivery Issue (105, 3.89%) reports the problems that occur
during the deployment and delivery of microservices systems. We identified 17
types of issues in this subcategory, which are mainly related to cd pipeline
error, cd pipeline stage, halyard deployment, and deployment script errors.
For example, regarding cd pipeline error, one contributor of the
eShopOnContainers project highlighted that “Jenkins pipeline is failing with
error 403, #990”.

  * •

Kubernetes Issue (74, 2.74%) reports the CI/CD issues specific to Kubernetes
which is an open-source system for automatic deployment, scaling, and
management of containerized applications (e.g., microservices systems). Most
problems of the subcategory are related to general kubernetes, helm bake,
kubernetes manifest errors. For example, the contributors of the
eShopOnContainers project were “unable to list Kubernetes resources using
default ASK to create a script, #688”).

  * •

Docker Issue (50, 1.85%): Docker is an open-source platform that helps
practitioners with continuous testing, deployment, executing, and delivering
applications (e.g., microservices systems). Our results indicate that most of
the issues related to Dockers are docker image error, docker configuration
error, and outdated container. For instance, the contributors of the
eShopOnContainers project pointed out that the “building of the solution using
docker-compose was failing, #2464” due to a docker configuration error.

  * •

Amazon Web Services (AWS) Issue (17, 0.63%): AWS provides an on-demand cloud
computing platform for creating, testing, delivering, and managing
applications. The most frequent types of this subcategory are general aws
error and aws jenkins error. As an example, some developers of the Spinnaker
project faced a situation in which “AWS Jenkins multi Debian package jobs fail
to bake, #2429”.

  * •

Version Control Issue (16, 0.69%) is related to version control and management
systems. In general, the major issues in this subcategory are related to Git,
such as git plugin, master branch, and gitlab issues.

  * •

Google Cloud Issue (8, 0.29%): Google Cloud provides infrastructure services
for creating and managing projects (e.g., microservices systems) and
resources. This subcategory contains issues related to the Google Cloud
platform for microservices systems. The most reported issues are gcp error gke
error, and gce clone error. For instance, in the Spinnaker project,
practitioners identified that “GCP: instance group port name Mapping is not
working properly, #2496”.

  * •

Others (38, 1.40%): This subcategory gathers issues related to cloud driver
error, virtual machine error, and spring boot error.

3\. Exception Handling Issue (228/2698, 8.45%): Exception handling is used to
respond the unexpected errors during the running state of software systems
(e.g., microservices systems), and it helps to avoid the software system being
crashed unexpectedly. This category represents the issues practitioners face
when handling various kinds of exceptions in microservices systems. We
identified and classified 44 types of Exception Handling issues in 5
subcategories (see Figure 4 and the Issue Taxonomy sheet in [19]). Each of
them is briefly described below.

  * •

Unchecked Exception (81, 3.00%): These exceptions cannot be checked on the
program’s compile time and throw the errors while executing the program’s
instructions. We identified 10 types of issues in this subcategory, in which
the top three types are null pointer exception, file not found exception, and
runtime exception.

  * •

Checked Exception (77, 2.85%): These exceptions can be checked on the
program’s compile time. Checked exceptions could be fully or partially checked
exceptions. We identified 16 types of issues in this subcategory, mainly
related to io exception, variables are not declared, and error handling.

  * •

Resource not Found Exception (37, 1.33%): These exceptions occur when some
services cannot find the required resources for executing operations. We
identified 8 types of issues in this subcategory. Most of them are related to
attributes do not exist, no server group, and missing library.

  * •

Communication Exception (28, 1.03%): These are exceptions thrown when the
client services cannot communicate with the producer services. We identified 7
types of issues in this subcategory, mainly related to http request exception
and timeout exception.

  * •

Others (5, 0.18%): This subcategory gathers issues related to api exception,
dependency exception, and thrift exception.

4\. Service Execution and Communication Issue (219/2698, 8.11%): Communication
problems are obvious when microservices communicate across multiple servers
and hosts in a distributed environment. Services interact using e.g., HTTP,
AMQP, and TCP protocols depending on the nature of services. The interviewees
also mentioned a few issues regarding this category, and one representative
quotation is depicted below.

“The poor implementation of microservices communication is also a source of
insecure communication, latency, lack of scalability, and errors and fault
identification on runtime” (P7, Software Engineer).

We identified and classified 34 types of service execution and communication
issues in 3 subcategories (see Figure 4 and the Issue Taxonomy sheet in [19]).
Each of them is briefly described below

  * •

Service Communication (166, 6.15%): There are different ways of communication
(e.g., synchronous communication, asynchronous message passing) between
microservices. This subcategory covers the issues of service communication in
which the majority of the issues are related to service discovery failure,
http connection error, and grpc connection error.

  * •

Service Execution (27, 1.00%): This subcategory contains the issues regarding
asynchronous communication, dynamic port binding, rabbitmq messaging, and
service broker during service execution. These issues occur due to various
reasons. For example, a dependency issue between microservices occurred when
“integration commands were sent asynchronously, #1710”. We also found several
issues regarding dynamic port binding. For instance, a server module of the
light-4j project could not dynamically allocate a “port on the same host with
a given range, #1742”.

  * •

Service Management (17, 0.63%): This subcategory covers the issues that occur
in the distributed event store platform, service management platform, and
service networking layer. The majority of the problems that happen in this
subcategory are kafka bug, kafka josn format issue, and eks (elastic
kubernetes service) error.

5\. Security Issue (213/2698, 7.89%): Microservices provide public interfaces,
use network-exposed APIs for communicating with other services, and are
developed by using polyglot technologies and toolsets that may be insecure.
This makes microservices a potential target for cyber-attacks; therefore,
security in microservices systems demands serious attention. Mining
microservices systems have identified a wide range of security issues. Among
those issues, microservices practitioners also mentioned several other
security issues during the interviews. One representative quotation is
depicted below.

“The other problem is securing microservices at different levels.
Specifically, we deal with microservices-based IOT applications that have more
insecure points than traditional ones” (P5, Solution Architect).

We identified and classified 37 types of security issues in 4 subcategories
(see Figure 4 and the Issue Taxonomy sheet in [19]). Each of them is briefly
described below.

  * •

Authentication and Authorization (123, 4.55%): Authentication is the process
of identifying a user, whereas authorization determines the access rights of a
specific user to system resources. We found that the majority of the
authentication and authorization issues are related to handling authorization
header (e.g., Basic Auth, OAuth, OAuth 2.0) and shared authentication issues.
handling authorization header is generally used to implement authorization
mechanisms. Our study found several issues related to Basic Auth, OAuth, and
OAuth 2.0 header failure and the non-availability of these security headers.
For example, the developers of the eShopOnContainers project reported the
issue about “invalid_request on auth from Swagger for Location API, #1990”.
Moreover, we found several issues regarding improper implementation of shared
authentication methods in microservices systems.

  * •

Access Control (64, 2.37%) is a fundamental element in securing the
infrastructure of microservices or any software systems. Access control could
be role- or attribute-based in a microservices system. The major types of
issues in this subcategory are managing credential setup and security policy
violation.

  * •

Secure Certificate and Connection (43, 1.59%): Our study reports several
issues regarding implementing security certificates and standards, such as
SSL, TSL, and JWT, which are used to secure communication between client-
server, service-to-service, or between microservices. For example, we found a
jwt error and an ssl connection issue in the Goa and Spinnaker project
respectively. We also found several other types of secure certificate and
connection issues, such as security token expired, tls certificate issue, and
expired certificate.

  * •

Encryption and Decryption (13, 0.48%) is used to convert plain text into
ciphertext and ciphertext into plain text to secure the information. We
identified three types of issues related to this subcategory are data
encryption, data decryption, and configuration decryption.

6\. Build Issue (210/2698, 7.78%): Build is a process of preparing an
application program for software release by collecting and compiling all
required source files. The outcome of this process could be several types of
artifacts, such as binaries and executable programs. We identified and
classified 20 types of build issues in 3 subcategories (see Figure 4 and the
Issue Taxonomy sheet in [19]). Each of them is briefly described below.

  * •

Build Error (141, 5.22%): We identified several types of build errors which
can interrupt the build process of microservices systems. We found that the
majority of build errors are related to build script, plugin compatibility,
docker build fail, build pipeline error, build file server error, source file
loading, and module resolution.

  * •

Broken and Missing Artifacts (59, 2.18%): These issues occur during the build
process’s parsing stage when the build systems verify the required information
(e.g., files, packages, designated locations) in the build script files before
executing the build tasks. This subcategory mainly covers the issues related
to missing properties, packages, and files, broken files, and missing objects.
We also identified several other types of broken and missing artifacts which
include missing ami (amazon machine image), missing base parameter in the
client, and missing link attributes.

  * •

Others (10, 0.37%): This subcategory mainly gathers issues related to wrong
use of universally unique identifier, and inconsistent data generated.

7\. Configuration Issue (121/2698, 4.48%): Configuration is a process of
controlling and tracking and making consistent all the required instances for
software systems(e.g., microservices systems). Microservices systems have
multiple instances and third party applications to configure. This category
gathers configuration issues during microservices system development,
implementation, and deployment phases. Besides identifying the configuration
issues from the OSS microservices system, microservices practitioners also
indicated configuration issues during the interviews. One representative
quotation is depicted in the following.

“The major challenge for me is the poor microservices’ configuration, which
grows as the application size grows—the configuration effect on implementation
and deployment phases of the microservices systems. The poor configuration of
microservices may lead to increased latency and decrease the speed of
microservices calls between different services” (P12, Application Developer).

We identified and classified 16 types of configuration issues in 2
subcategories (see Figure 4 and the Issue Taxonomy sheet in [19]). Each of
them is briefly reported below.

  * •

Configuration Setting Error (61, 2.26%): This subcategory contains issues
associated with the setup configuration of different types of servers,
databases, and cloud infrastructure platforms. The main types of issues in
this subcategory are server configuration error, database configuration error,
and aks (azure kubernetes service) configuration error.

  * •

Configuration File Error (60, 2.22%): This subcategory covers the issues that
mainly occur due to providing incorrect values in environment setting
variables. The major types of issues in this subcategory are configuration
mismatch, conflict in configuration file names, and incorrect file path.

8\. Monitoring Issue (89/2698, 3.29%): The dynamic nature of microservices
systems needs monitoring infrastructures to diagnose and report errors,
faults, failure, and performance issues. This category reports issues related
to monitoring microservices systems. Several interviewees also mentioned
monitoring issues for microservices systems. One representative quotation is
depicted in the following.

“Microservices systems host containerized or virtualized across distributed
private, public, hybrid, and multi-cloud environments. Monitoring highly
distributed systems like microservices systems through traditional monitoring
tools is a challenging experience because these tools only focus on a specific
component or the overall operational health of the system” (P14, Azure
Technical Engineer).

We identified and classified 17 types of monitoring issues in 3 subcategories
(see Figure 4 and the Issue Taxonomy sheet in [19]). Each of them is briefly
described below.

  * •

Tracing and Logging Management Issue (60, 2.22%): One prominent challenge of
monitoring microservices systems is the collection of logs from containers and
distributed tracing. We identified 7 types of issues in this subcategory,
mainly related to distributed tracing error, logging management error, and
observability issue.

  * •

Health Check Issue (17, 0.63%): This subcategory deals with the problems
related to the health monitoring of microservices systems. We identified six
types of issues, mainly related to health check API error, health check fail,
and health check port error.

  * •

Monitoring Tool Issue (12, 0.44%): We also identified several issue
discussions, in which microservices practitioners discussed the problems of
three monitoring tools, including zipkin issue, jenkins issue, and tcp/tt
health check issue.

9\. Compilation Issue (79/2698, 2.92%): This category reports compilation
errors, which mainly occur when the compiler cannot compile source code due to
errors in the code or errors with the compiler itself. We identified and
classified 9 types of compilation issues in 2 subcategories (see Figure 4 and
the Issue Taxonomy sheet in [19]). Each of them is briefly described below.

  * •

Illogically Symbols (60/2698, 2.22%): These issues occur when developers use
illegal characters or incorrect syntax during the coding, for instance, syntax
error, invalid parent id, and unexpected end of file.

  * •

Wrong Method Call (19/2698, 0.70%): These issues occur when the compiler tries
to search for definitions of methods by invoking them through method calls and
finds wrong parameter, wrong method call, and incorrect values.

10\. Testing Issue (77/2698, 2.85%): Microservices systems pose significant
challenges for testing because of many services, inter-communication
processes, dependencies, network communication, and other factors. Among those
issues, microservices practitioners also mentioned several other testing
issues during the interviews. One representative quotation is depicted in the
following.

“Testing is another issue that I think is more challenging in microservices
systems. I also think deploying each microservice as a singular entity and
testing them is tedious and brings several problems. For example, testing
coordination among multiple microservices when deploying one service as a
singular entity” (P2, Application Developer).

We identified and classified 20 types of testing issues in 3 subcategories
(see Figure 4 and the Issue Taxonomy sheet in [19]). Each of them is briefly
described below.

  * •

Test Case Issue (45, 1,66%): This subcategory covers the problems with test
cases written to evaluate the expected output compliance with specific
requirements for the microservices systems. Most of them are related to faulty
test case, missing test case, and syntax error in test case.

  * •

Code and Component Test (21, 0.77%): This subcategory deals with the issues
mainly related to debugging, api testing, and missing design test of
microservices systems.

  * •

Application Test (11, 0.40%): This subcategory gathers the issues related to
overall microservices application testing, which include load test case,
broken integration test, and application security testing issues.

11\. Documentation Issue (75/2698, 2.77%): Documentation for microservices
systems may suffer from several problems. We identified and classified 8 types
of documentation issues in two subcategories (see Figure 4 and the Issue
Taxonomy sheet in [19]). Each of them is briefly described below.

  * •

Insufficient Document (49, 1.81%): This subcategory covers the problems
related to outdated document, broken images, and inappropriate examples. For
instance, one contributor of the eShopOnContainers project mentioned the issue
of “Out of date wiki guide for vs2015, #2030”.

  * •

Readability Issue (26, 0.95%): This subcategory is related to readability
problems with provided documentation. The leading types of readability issues
are poor readability, missing readme file, and old readme file.

12\. Graphical User Interface (GUI) Issue (70/2698, 2.59%): This category
reports the problems that can wreck the GUI of microservices systems. We
identified and classified 68 types of GUI issues in 3 subcategories (see
Figure 4 and the Issue Taxonomy sheet in [19]). Each of them is briefly
described below.

  * •

Broken User Interface Elements (38, 1.41%) are dysfunctional User Interface
(UI) elements (e.g., buttons or text fields) that can become the reason for
inconsistencies in the page layout across different devices (e.g., mobile and
desktop browsers). This subcategory represents the faults mainly related to
front end crash, uneditable contents, and broken images in ui.

  * •

Missing Information and Legacy UI Artifacts (32, 1.19%): This subcategory
contains the problems of wrong and incomplete information along with outdated
UI artifacts, mainly related to wrong gui display, displaying incomplete
information, and selection not working.

13\. Update and Installation Issue (68/2698, 2.52%): This category gathers the
identified problems related to update and installation of the packages,
libraries, tools, and containerization platforms required to develop and
manage microservices systems. We identified and classified 16 types of update
and installation issues in 2 subcategories (see Figure 4 and the Issue
Taxonomy sheet in [19]). Each of them is briefly described below.

  * •

Update Error (45, 1.67%): This subcategory represents the errors in which
developers face the issues of outdated packages, platforms, technologies, and
backward compatibility. The leading types of issues are outdated install
packages ,backward compatibility issue, and json update error.

  * •

Installation Error (23, 0.85%): The development of microservices systems can
be interrupted because of failure to install required languages, packages, and
platforms. The top three types of issues are language package installation
error, npm (node package manager) error, and gke (google kubernetes engine)
installation error.

14\. Database Issue (65/2698, 2.40%): The ownership of the microservices
system database is usually distributed, and most of the microservices are
autonomous and have a private data store relevant to their functionality. The
distributed nature of microservices systems brings challenges like database
implementation, data accessibility, and database connectivity. The
microservices practitioners also mentioned several other database issues
during the interviews. One representative quotation is depicted below.

“Relational database for microservices systems. This issue occurs when we
migrate from monolithic applications to microservices systems. It was mainly
because of the missing transaction management system for getting data from the
database of the old application (that was a relational database) through the
microservices application” (P6, Software Architect, Developer).

We identified and classified 24 types of database issues in 3 subcategories
(see Figure 4 and the Issue Taxonomy sheet in [19]). Each of them is briefly
described below

  * •

Database Connectivity (26, 0.96%): This subcategory refers to the issues that
occur while establishing a database connection. The leading types of issues in
this subcategory are sql container failure , sql transient connection failure
, and database creation failure.

  * •

Database Query (31, 1.14%): The errors in this subcategory cover search query
issues during the development of microservices systems. Such issues may cause
a database performance bottleneck. The leading types of issues in this
subcategory are wrong query, elasticsearch database error, and database search
error.

  * •

Others (8, 0.29%): Other types of database issues that cannot be classified
into the above subcategories are included in this subcategory, which are
mainly related to database migration, database storage, and database adapter.
textcolorredFor example, one developer of the Spinnaker project mentioned that
“DB adapter issue - can not perform a textual search with a list, #1374”.

15\. Storage Issue (54/2698, 2.00%): This category reports storage space
problems during the development, execution, and management of microservices
systems. We identified and classified 13 types of storage issues in 2
subcategories (see Figure 4 and the Issue Taxonomy sheet in [19]), and each
subcategory is further described below.

  * •

Storage Size Constraints (48, 1.78%): Different microservices have different
data storage requirements. This subcategory covers the storage size
constraints, mainly related to lack of main memory, cache issue, and storage
backend failure.

  * •

Large Data Size (6, 0.21%): This subcategory gathers the issues related to
large data size, including large image size, large message size, and large
file.

16\. Performance Issue (45/2698, 1.67%): Microservices systems offer various
advantages over monolithic systems. However, several types of issues wreak the
performance of microservices systems. The interviewees also mentioned
performance overhead as an issue, and one representative quotation is depicted
below.

“Unlike a monolithic application whose deployment and management are seemingly
easier due to centralized control and monitoring, a microservices-based
application has numerous independent services that may be deployed on
different infrastructures and platforms. Such an aspect increases its
performance overhead. I also think that microservices systems consume more
resources, creating a heavy burden for servers. In the response achieving the
performance goal become questionable” (P15, Software Architect).

We identified and classified 16 types of performance issues in three
subcategories (see Figure 4 and the Issue Taxonomy sheet in [19]). Each of
them is briefly described below.

  * •

Service Response Delay (28, 1.03%): This subcategory gathers the types of
issues regarding delay in service response that ruins the performance of
microservices systems, such as long wait, inconsistent payloads, and slow
query.

  * •

Resource Utilisation (13, 0.48%): In this subcategory, we collected the issues
that degrade the performance of microservices systems due to resource
utilisation. These issues are mainly related to load balancer error, high cpu
usage, and rate limiting error. One example of the load balancer issue
mentioned by a Spinnaker project contributor is “Failed to create a Load
Balancer when creating a new application, #1583”.

  * •

Lack of Scalability (4, 0.14%): Scalability is the system’s ability to respond
the user demands and change workload by adding or removing resources. This
subcategory gathers the issues related to scalability that can hinder
microservices system growth, for instance, scale to cluster error and circuit
breaker issue.

17\. Networking Issue (41/2698, 1.51%): Deploying, executing, and
communicating in microservices systems over the network is complex. It is
observed that many problems may disrupt the network. We identified and
classified 20 types of networking issues in 2 subcategories (see Figure 4 and
the Issue Taxonomy sheet in [19]). Each of them is briefly described below.

  * •

Hosting and Protocols (22, 0.81%): This subcategory represents the issues
related to hosting protocols, ports, and topologies for microservices systems.
The leading types of issues in this subcategory are localhost error, ip
address issue, and udp (user datagram protocol) discovery error.

  * •

Service Accessibility (19, 0.70%): This subcategory of issues represent the
cases where microservices practitioners face service accessibility problems.
The leading types of issues are webhook error, broken urls, and dns (domain
name system) error.

18\. Typecasting Issue (35/2698, 1.29%): This category is related to the
typecasting issues that occur when assigning a value of one primitive data
type to another type. We identified and classified 9 types of typecasting
issues in 2 subcategories (see Figure 4 and the Issue Taxonomy sheet in [19]).
Each of them is briefly described below.

  * •

Type Conversion (20, 0.74%): This subcategory deals with the issues when
variables are not correctly converted from one type to another. The top three
types of conversion issues are identity conversion, boxing conversion, and
enumeration validation issue.

  * •

Narrow/Wide Conversion (15, 0.55%): These issues occur when the compiler
converts variables of a larger type into a smaller type (e.g., double to
float) or a smaller type into a larger type (e.g., float to double). The top
two types of narrow/wide conversion issues are narrowing primitive conversion
and narrowing reference conversion.

19\. Organizational Issue (7/2698, 0.25%): We derived this category based on
the interviewees’ feedback on the taxonomy of the issues (see Figure 4 and the
Issue Taxonomy sheet in [19]). The interviewees only mentioned three types of
issues in this category, including team management, operational and tooling
overhead, and service size. We depicted two representative quotations below.

“One of the critical challenges in organizations is team management according
to available people, their expertise, and their working habits” (P1, Software
Architect, Developer).

“Creating a reasonable size for each microservices’ (I mean, each microservice
should have sufficient responsibilities). This is a bit tricky because most of
the issues are rooted here” (P6, Software Architect, Developer).

Key Findings of RQ1: We identified 2,641 instances of issues by mining
developer discussions in 15 open-source microservices systems with 48
instances of issues mentioned by the interviewees and 9 instances of issues
mentioned by the survey participants, which are 2,698 issues in total. The
issue taxonomy consists of 19 categories, 54 subcategories, and 402 types,
indicating the diversity of the issues in microservices systems. The majority
of issues are related to Technical Debt (25.46%), CI/CD (11.60%), and
Exception Handling (8.45%).

Figure 4: A taxonomy of issues in microservices systems

###  3.2 Causes of Issues (RQ2)

The taxonomy of causes of microservices issues is provided in Table III. It is
worth mentioning that not all the issue discussions provide the information
about their causes. Therefore, we identified 2,225 issue discussions
containing information about the causes. The taxonomy of causes is derived by
mining developer discussions (i.e., 2,225 instances of causes), conducting
practitioner interviews (i.e., 31 instances of causes, see Section 2.3.3), and
conducting a survey (i.e., 11 instances of causes, see Section 3.4). Hence, we
got a total of 2,267 instances of causes. We identified a total of 228 types
of causes that can be classified into 8 categories and 26 subcategories. Due
to space limitations, we only list the top two types of causes for each
subcategory in Table III. The detail of the types of causes can be found in
the dataset [19]. The results show that General Programming Error (860 out of
2267), Missing Features and Artifacts (386 out of 2267), and Invalid
Configuration & Communication Problems (382 out of 2267) are the top three
categories of causes. Each cause category is briefly discussed below.

1\. General Programming Error (GPE) (860/2267, 37.93%): This category captures
the causes that are based on a broad range of errors occurred in different
phases of microservices system development, such as coding (e.g., syntax
errors), testing (e.g., incorrect test cases), and maintenance (e.g., wrong
examples in documentation) phases. We identified and classified 78 types of
GPE causes in six subcategories (see Table III and the Cause Taxonomy sheet in
[19]). Each of them is briefly described below.

  * •

Compile Time Error (377, 16.62%): This subcategory gathers the causes of
issues in which microservices practitioners violate the rules of writing
syntax for microservices systems codes. These causes must be addressed before
the program can be compiled. We identified 16 types of causes in this
subcategory in which the top three are semantic error, syntax error in code,
and variable mutations.

  * •

Erroneous Method Definition and Execution (262, 11.55%): The causes in this
subcategory are related to incorrect or partly correct definitions and
executions of methods associated with object messages. Generally, methods are
referred as class building blocks linked together for sharing and processing
data to produce the desired results. We identified 27 types of causes in this
subcategory in which the top three are lack of cohesion in methods, long
message chain, and wrong parameterization.

  * •

Incorrect Naming and Data Type (157, 6.92%): This subcategory covers the
causes related to choosing incorrect names and data types for identifiers,
methods, packages, and other entities in the source code. Our taxonomy
contains 18 types of causes in this subcategory, and among them wrong data
conversion, wrong data type, and wrong use of data types are the top three
types of causes.

  * •

Testing Error (25, 1.10%): This subcategory covers the causes behind testing
issues in microservices systems. In this subcategory, we identified 6 types of
causes in which the top two types of causes are incorrect test case and
incorrect syntax in test cases.

  * •

Poor Documentation (22, 0.97%): The documentation of software systems may
contain critical information that describes the software product capabilities
for system stakeholders. We identified 7 types of causes in this subcategory,
in which the top three are typo in documents, and wrong example in documents.

  * •

Query and Database Issue (11, 0.48%): This subcategory contains the causes
behind database issues in microservices systems. We identified 4 types of
causes in this subcategory, and the top three types of causes are wrong query
parameters, missing query parameters, and incorrect querying range.

2\. Missing Features and Artifacts (MFA) (386/2267, 17.02%): This category
represents the causes behind microservices systems issues that occur due to
missing required features, packages, files, variables, and documentation and
tool support. We identified and classified 27 types of MFA causes in four
subcategories (see Table III and the Cause Taxonomy sheet in [19]). Each of
them is briefly described below.

  * •

Missing Features (186, 8.20%) denotes the nonexistence of system functionality
in microservices systems. In this subcategory, we identified 14 types of
causes, in which the top two types of causes are missing required system
features and missing security features.

  * •

Missing Documentation and Tool Support (104,4.58%): Proper documentation and
tool support is vital to keep the record of and track changes between system
requirements, architecture, and source code. It also guide various system
stakeholders (e.g., architects, developers, end-users) regarding design,
architecture, and coding standards use in microservices systems. Moreover,
several types of tool support are also necessary for different phases of
microservices system development (e.g., development and deployment). The
absence of proper documentation and tool support can bring several types of
issues to microservices systems. This subcategory contains 4 types of missing
documentation and tool support causes, and mong them missing readme file and
missing development and deployment tool support are identified as the leading
causes.

  * •

Missing Packages and Files (68, 2.99%): This subcategory groups the causes
related to absence of required resources, packages, and files for developing,
deploying, and executing microservices systems. We collected 9 types of
missing packages and files causes, in which the leading three types of causes
are missing resource, missing required package, and missing api.

  * •

Missing Variables (28, 1.23%): A few missing variables are also identified as
the causes for several microservices issues. Compilers throw the error
messages of missing variables if variables are set to a nonexistent directory
or have the wrong names. This subcategory contains 7 types of missing
variables causes, and among them missing environment variable and missing
properties are identified as the leading causes.

3\. Invalid Configuration and Communication (ICC) Problem (382/2267, 16.85%):
Considering a large number of microservices, their distributed nature, and
third-party plugins, microservices systems need to be configured for complete
business operations properly. Each microservice has its instances and process,
and services interact with each other using several inter-service
communication protocols (e.g., HTTP, gRPC, message brokers AMQP). One of the
interviewees also mentioned the following cause regarding invalid
configuration and communication.

“Microservices systems typically use one or more infrastructure and 3rd party
services. Examples of infrastructure services include a service registry, a
message broker, and a database server. During the configuration of
microservices, a service must be provided with configuration data that tells
it how to connect to the external or 3rd party services — for example, the
database network location and credentials” (P1, Software Architect,
Developer).

We identified and classified 29 types of causes in two subcategories (see
Table III and the Cause Taxonomy sheet in [19]). Each of them is briefly
described below.

  * •

Incorrect Configuration (290, 12.79%): Configuration management in
microservices systems is a hefty task because microservices are scattered
across multiple servers, containers, databases, and storage units. Each
microservices may have multiple instances. Therefore, an incorrect
configuration may lead to several types of errors. This subcategory contains
16 causes for different types of microservices issues, in which the leading
two causes are incorrect configuration setting and wrong connection closure.

  * •

Server and Access Problem (92, 4.05%): Each microservices acts as a miniature
application that communicates with each other. We need to configure the
infrastructure layers of the microservice system for sharing different types
of resources. A poor configuration may lead to problems of accessibility for
servers and other resources that bring multiple issues in microservices
systems. In this subcategory, we identified 13 types of causes, in which the
leading three types of causes are transient failure, service registry error,
and wrong communication protocol.

4\. Legacy Versions, Compatibility, and Dependency (LC&D) Problem (222/2267,
9.79%): This category represents a broad range of causes arising from outdated
repositories, applications, documentation versions, development and deployment
platforms, APIs, libraries, and packages. One of the interviewees also
mentioned several causes regarding microservices issues, especially
compatibility and dependency. One representative quotation is depicted below.

“Along with the legacy code version, some of the critical reasons for the
microservices issues are i) no clear strategy for code repository and
branching, the mix of technologies (each team uses its way of development),
dependencies on other services which are not released yet but going for
integration and load testing, development pace varies from team to team, lack
of centralized release manager, incompatibility of a new version of the
service with previous services.” (P8, DevOps Consultant).

We identified and classified 28 types of causes in five subcategories (see
Table III and the Cause Taxonomy sheet in [19]). Each of them is briefly
described below.

  * •

Compatibility and Dependency (59, 2.60%): A typical microservices system
consists of several independent services running on multiple servers or hosts
[34]. However, some microservices also depend on other microservices to
complete business operations. Usually, practitioners ensure the compatibility
(e.g., backward compatibility) of each microservice with previous versions of
the microservices systems during the upgrading. In this subcategory, we
identified 5 types of causes, in which the leading three types of causes are
compatibility error, and outdated dependency.

  * •

Outdated and Inconsistent Repositories (53, 2.33%): This subcategory gathers
the causes which are the source of the issues that occur when the online code
repository of version control systems has updated the local files repository.
The most frequent causes in this subcategory are old repository version, old
dev branch, and version conflicts.

  * •

Outdated Application and Documentation Version (45, 1.98%): We identified
several causes arising from using outdated applications and documentation
versions in the selected systems, which are mainly related to document not
updated, and deprecated software version.

  * •

Outdated Development and Deployment Platforms (38, 1.67%): Development and
deployment platforms help developers build, test, and deploy microservices
systems efficiently. We identified 10 types of causes in this subcategory, and
among them old Kubernetes version and old visual studio version are the top
two leading type of causes.

  * •

Outdated APIs, Libraries, and Packages (27, 1.09%): This subcategory covers
the causes arising from the usage of outdated APIs, libraries, and packages.
The most frequent types of causes in this subcategory are outdated version of
library and old version of package.

5\. Service Design and Implementation Anomalies (SD&IA) (174/2267, 7.67% ):
This category covers the causes of issues when microservices practitioners
cannot address the associated complexity of distributed systems at the design
level. The interviewees also mentioned several causes related to service
design and implementation anomalies, and one representative quotation is
depicted below.

“According to my experience, the primary reason behind the design issues is
the distributed nature of microservices systems, which is becoming
increasingly complicated with growing systems—especially where multiple
subsystems also need to be integrated” (P7, Software Engineer).

We identified and classified 17 types of SD&IA causes in 2 subcategories (see
Table III and the Cause Taxonomy sheet in [19]). Each of them is briefly
described below.

  * •

Code Design Anomaly (130, 5.73%): Code design anomalies refer to poorly
written code that may lead to several problems (e.g., difficulties in
maintenance or future enhancements) in microservices systems. Our taxonomy
gets 12 types of causes in this subcategory and among them poor code
readability, messy code, and data clumps are the top three types of causes.

  * •

System Design Anomaly (44, 1.94%): System design anomalies refer to poorly
designed microservices architecture that may lead to maintainability,
scalability, and performance issues. This subcategory covers 5 types of
causes. Among them, the top three types of causes are wrong dependencies
chain, lack of asrs, and wrong application decomposition.

6\. Poor Security Management (PSM) (126/2267, 5.55%): Microservices systems
are distributed over data centres, cloud providers, and host machines. The
security of microservices systems is a multi-faceted problem that requires a
layered solution to cope with various types of vulnerabilities. The interview
participants also mentioned several causes for security issues, and one
representative quotation is depicted below.

“I think the basic reasons behind the security issues are poor understanding
of microservices architecture, large attack surfaces (many distributed
points), error-prone encryption techniques while services are communicating,
and insecure physical devices” (P5, Solution Architect).

We identified and classified 21 types of PSM causes in 3 subcategories (see
Table III and the Cause Taxonomy sheet in [19]). Each of them is briefly
described below.

  * •

Coding Level (55, 2.42%): This subcategory collects the causes where strict
security principles and practices are not followed to prevent potential
vulnerabilities while writing code of microservices systems. This subcategory
covers 7 types of coding level causes, and among them, the top three types of
causes are unsafe code, malformed input, and wrong implementation of security
api.

  * •

Communication Level (40, 1.76%): Given the polyglot and distributed nature of
microservices systems, practitioners need to secure the inter-microservices
communication. This subcategory covers 10 types of communication level causes,
and among them, the top two types of causes are security dependencies and
wrong access control.

  * •

Application Level (29, 1.27%): The application level of security refers to
security practices implemented at the interface between an application and
various components (e.g., databases, containers) of microservices systems.
This subcategory only contains 3 types of causes that are insecure
configuration management, missing security features, and violation of the
security policies.

7\. Insufficient Resources (IR) (93/2267, 4.10% ): Microservices systems are
at risk of delivering the required outcome without sufficient resources. We
identified and classified 4 types of IR causes in two subcategories (see Table
III and the Cause Taxonomy sheet in [19]). Each of them is briefly described
below.

  * •

Memory Issue (68, 2.99%): Microservices systems are developed by using
multiple languages (e.g., Java, Python, C++) and platforms (e.g., containers,
virtual machines). Some languages and platforms consume more memory than
others. For instance, C/C++ consumes less memory than Java, and Python and
Perl consume less memory than C/C++. This subcategory covers 2 types of causes
that are limited memory for process execution, and ide problem with memory.

  * •

Lack of Human Resources, Tools and Platforms  (25, 1.10%): This subcategory
deals with the causes related to tools and platforms support for microservices
systems. Only 3 types of causes related to this subcategory are identified,
including lack of tool support, anddeployment platform problem.

8\. Fragile Code (FC) (24/2267, 1.05%): Fragile code refers to code that is
difficult to change, and a minor modification in fragile code may break the
service or module. We identified and classified 6 types of FC causes in two
subcategories (see Table III and the Cause Taxonomy sheet in [19]). Each of
them is briefly described below.

  * •

Poor Implementation of Code (20, 0.88%): This subcategory gathers the causes
of poor code quality, which exhibits the buggy behaviour of microservices
systems. We identified 3 types of causes in this subcategory, including poor
object-oriented design, poor code reusability, and unnecessary code.

  * •

Poor Code Flexibility (11, 0.48%): Code flexibility is important for long-
lived microservices project code bases, however, we found a few issues that
occurred due to poor code flexibility. The 3 types of causes in this
subcategory are divergent changes, delayed refactoring, and poorly organized
code.

TABLE III: Taxonomy of causes of issues in microservices systems Category of Causes | Subcategory of Causes | Type of Causes
---|---|---
General Programming Error (GPE) (860) | Compile Time Error (377) | Semantic Error (205)
Syntax Error in Code (112)
Erroneous Method Definition and Execution (262) | Lack of Cohesion in Methods (38)
Long Message Chain (36)
Incorrect Naming and Data Type (157) | Wrong Data Conversion (39)
Wrong Data Type (20)
Testing Error (25) | Incorrect Test Case (20)
Wrong Implementation of Security Pattern (5)
Poor Documentation (22) | Typo in Documents (9)
Wrong Example in Documents (7)
Query and Database Issue (13) | Wrong Query Parameters (6)
Missing Query Parameters (3)
Missing Features and Artifacts (MFA) (386) | Missing Features (186) | Missing Required System Features (111)
Missing Security Features (37)
Missing Documentation and Tool Support (104) | Missing Readme File (88)
Missing Tool Support (14)
Missing Packages and Files (68) | Missing Resource (41)
Missing Required Package (15)
Missing Variables (28) | Missing Environment Variable (14)
Missing Properties (8)
Invalid Configuration and Communication (ICC) Problem (382) | Incorrect Configuration (285) | Incorrect Configuration Setting (227)
Wrong Connection Closure (24)
Server and Access Problem (92) | Transient Failure (21)
Service Registry Error (24)
Legacy Versions, Compatibility, and Dependency (LC&D) Problem (222) | Compatibility and Dependency (59) | Compatibility Error (42)
Outdated Dependency (9)
Outdated and Inconsistent Repositories (53) | Old Repository Version (38)
Old DEV Branch (7)
Outdated Application and Documentation Version (45) | Document not Updated (33)
Deprecated Software Version (6)
Outdated Development and Deployment Platforms (38) | Old Kubernetes Version (7)
Old IDE Version (6)
Outdated APIs, Libraries, and Packages (27) | Outdated Version of Library (15)
Old Version of Required Package (11)
Service Design and Implementation Anomalies (SD&IA) (174) | Code Design Anomaly (125) | Poor Code Readability (54)
Messy Code (35)
System Design Anomaly (44) | Wrong Dependencies Chain (16)
Lack of ASRs (5)
Poor Security Management (PSM) (126) | Coding Level (55) | Unsafe Code (23)
Malformed Input (12)
Communication Level (42) | Security Dependencies (8)
Wrong Access Control (5)
Application Level (29) | Insecure Configuration Management (17)
Missing Security Features (7)
Insufficient Resources (IR) (93) | Memory Issue (68) | Limited Memory for Process Execution (66)
IDE Problem with Memory (2)
Lack of Human Resources,Tools and Platforms (25) | Lack of Tool Support (7)
Deployment Platform Problem (5)
Fragile Code (FC) (24) | Poor Implementation of Code (20) | Poor Object Oriented Design (6)
Poor Code Reusability (5)
Poor Code Flexibility (11) | Tightly Services Components (7)
Divergent Changes (9)

Key Findings of RQ2: We found that not all the issue discussions provide the
information about their causes, and finally we identified 2,225 issue
discussions containing information about the causes with 31 causes mentioned
by the interviewees and 11 causes indicated by the survey participants, which
are 2,267 causes instances in total. The cause taxonomy of microservices
issues consists of 8 categories, 26 subcategories, and 228 types. The majority
of causes are related to General Programming Errors (37.93%), Missing Features
and Artifacts (17.02%), and Invalid Configuration and Communication (16.85%).

###  3.3 Solutions of Issues (RQ3)

The taxonomy of solutions for microservices issues is provided in Table IV. It
is worth mentioning that not all the issue discussions provide the information
about their solutions. Therefore, we identified 1,899 issue discussions
containing the information about the solutions. The taxonomy of solutions is
derived by mining developer discussions (i.e., 1,899 solutions), conducting
practitioner interviews (i.e., 36 solutions, see Section 2.2.2), and
conducting a survey (i.e., one instance of solution, see Section 3.4). Hence,
we got a total of 1,936 solutions. We identified a total of 196 types of
solutions that can be classified in 8 categories and 35 subcategories. Due to
space limitations, we only list the top two types of solutions for each
subcategory in Table IV. The detail of the types of solutions can be found in
the dataset [19]. The results show that Fix Artifacts (1056 out of 1936), Add
Artifacts (360 out of 1936), and Modify Artifacts (210 out of 1936) are the
top three categories of solutions. Each solution category is briefly discussed
below.

1\. Fix Artifacts (1056/1936, 54.54%): During the analysis of developer
discussions about microservices issues, we identified that most developers did
not explicitly mention any solution to the problems. They fixed the issue in
the local repository and send the fixed code (e.g., through a pull request) to
the maintainer of the public repository. In this case, from the developer
discussions, we could not find exactly what they added, removed, or modified
in the project to fix a specific issue. Therefore, we named this category Fix
Artifacts. We identified and classified 25 types of fix artifacts solutions in
4 subcategories (see Table IV and the Solution Taxonomy sheet in [19]). Each
of them is briefly described below.

  * •

Fix Code Issue (912, 47.10%): The solutions in this subcategory are related to
the direct repair of the source code by developers. We identified 14 types of
solutions in this subcategory and among them fix source code of issues, fix
illegal symbols (syntax) in code, and clean code are the top three types of
solutions.

  * •

Fix Testing Issue (107, 5.52%): This subcategory covers the types of solutions
with which testing issues have been fixed. We collected 2 types of solutions
in this subcategory that are debug code and add test services in containers.

  * •

Fix Build Issue (33, 1.70%): Build systems are essential for developing,
deploying, and maintaining microservices systems. In contrast, build failures
frequently occur across the development life cycle, bringing non-negligible
costs in microservices system development. We collected 3 types of solutions,
including fix errors in build files, correct build type, and hide fail status.

  * •

Fix GUI Issue (4, 0.20%): We collected two types of solutions related to
repairing the graphical user interface of microservices systems, and they are
fix backwards incompatible ui and fix broken screenshot.

2\. Add Artifacts (360/1936, 18.59%): This category covers the types of
solutions for addressing missing features, packages, files, variables, and
documentation and tool support issues. The interview participants also
mentioned that they adopted solutions for addressing several types (e.g.,
CI/CD, Security) of issues that have occurred because of missing features, and
one representative quotation is depicted below.

“To address these issues, we recently adopted the ‘service mesh’ approach.
This approach helps address security, latency and scalability, fault
identification, and runtime error detection in microservices systems. A
service mesh pattern can also provide features for a service health check with
the lowest latency to address errors and fault identification issues” (P7,
Software Engineer).

We identified and classified 33 types of solutions in 10 subcategories (see
Table IV and the Solution Taxonomy sheet in [19]). Each of them is briefly
described below.

  * •

Add Features and Services (131, 6.76%): Developers added required features and
services to microservices systems. System features and services refer to a
process that accepts one or more inputs and returns outputs for particular
system functionality [35]. We identified 5 types of solutions in this
subcategory, and among them, add missing features, add security features, and
add communication protocols are the top three types of solutions.

  * •

Add Files, Templates, and Interfaces (39, 2.01%): This subcategory covers the
solutions for adding missing files, templates, and interfaces in microservices
systems. We identified 3 types of solutions, including add files, add
templates, and add interfaces.

  * •

Add Methods and Modules (33, 1.70%): We identified the solutions in which
developers mainly added or repaired the missing methods and modules to address
several types (e.g., Compilation, Service Execution and Communication, Build)
of issues in microservices systems. We identified 3 types of solutions,
including add constructor, add parameters in methods, and add security
certificates.

  * •

Add Test Cases (28, 1.44%): We identified the solutions in which developers
added test cases to address testing issues of microservices systems. We
identified 4 types of solutions in this subcategory in which the top three are
add test cases to validate service, generate correct test cases, and add test
cases to validate data type.

  * •

Add Classes and Packages (27, 1.39%): This subcategory covers the solutions in
which developers added classes and packages to address the microservices
issues. We identified 3 types of solutions, including add packages, add
properties, and add objects.

  * •

Implement Patterns and Strategies (23, 1.18%): The solutions in this
subcategory are based on interviewees’ feedback in which they mentioned 19 MSA
patterns and strategies for addressing microservices design issues. The most
frequently mentioned patterns and strategies are service mesh architecture,
service instance per container, and serverless deployment.

  * •

Add Data Types, Identifiers, and Loops (21, 1.08%): This subcategory contains
the solutions for addressing errors at the program initialization level. We
identified 5 types of solutions in which the top three are add identifiers,
add data types, and add query parameters.

  * •

Add Dependencies and Metrics (7, 0.36%): The issues related to missing metrics
(e.g., monitoring) and dependencies can be resolved by adding the required
metrics and dependencies in the microservices systems. We identified 3 types
of solutions, including add monitoring metrics, add dependencies, and add
stack trace.

  * •

Add APIs, Namespaces, and Plugins (7, 0.36%): APIs, namespaces, and plugins
are the essential part of any microservices systems. This subcategory covers
the solutions for addressing missing APIs, namespaces, and plugins. We
identified three types of solutions in this subcategory, including add apis,
add namespaces, and add plugins.

  * •

Add Logs (4, 0.20%): Logging activity is specifically related to monitoring
microservices systems. We identified 4 types of solutions that have been used
to address monitoring issues (e.g., missing logs), in which the top three are
add trace id, add trace logging, and add header logging.

3\. Modify Artifacts (212/1936, 10.95%): Besides adding new artifacts to the
existing system to address the microservices issues, we also identified a
large number of solutions in which the developers explicitly mentioned how
they modified modules, services, packages, APIs, scripts, methods, objects,
data types, identifiers, databases, and documentation to address the
microservices issues. We identified and classified 31 types of solutions in 5
subcategories (see Table IV and the Solution Taxonomy sheet in [19]). Each of
them is briefly described below.

  * •

Modify Methods and Objects (78, 4.03%): We found that most developers
corrected the proprieties (e.g., method calls, operations, parameters)
associated with methods and objects of classes to address various types of
issues (e.g., code smells, excessive literals) in microservices systems. We
identified 13 types of solutions in this subcategory, among them correct
method definition, redefine method operations, and correct method parameters
are the top three types of solutions.

  * •

Modify Packages, Modules, and Documentation (64, 3.30%): This subcategory
covers the packages, modules, and documentation that are modified to address
the microservices issues. We identified 4 types of solutions in this
subcategory, and among them improve documentation (“Resolve ’Create Stack’
issue and update docs, #336”) and update packages are the two leading types of
solutions.

  * •

Modify Data Types and Identifiers (51, 2.63%): This subcategory covers the
data types and identifiers that are modified to address the microservices
issues. We collected 5 types of solutions, and among them correct naming,
correct data types, and correct nil value are the top three types of
solutions.

  * •

Modify APIs, Services, and Scripts (13, 0.67%): This subcategory covers the
solutions in which APIs, services, and scripts are modified to address the
microservices issues. We identified 6 types of solutions in this subcategory,
and among them update scripts, update syntax, and modify services are the top
three types of solutions.

  * •

Modify Databases (6, 0.31%): This subcategory covers the solutions in which
database query strings and tables are modified to address the microservices
issues. We identified 2 types of solutions related to this subcategory,
including modify query strings and modify database tables.

4\. Remove Artifacts (39/1936, 2.01%): This category covers the solutions in
which artifacts are removed to address several types of microservices issues.
We identified and classified 14 types of solutions in 5 subcategories (see
Table IV and the Solution Taxonomy sheet in [19]). Each of them is briefly
described below.

  * •

Remove Data Types, Methods, Objects, and Plugins (20, 1.03%): This subcategory
covers the solutions in which data types, methods, objects, and plugins are
removed to address the microservices issues. We identified 4 types of
solutions in this subcategory, including remove data types, remove methods,
remove conflicting plugins, and remove objects.

  * •

Remove Dependencies and Databases (8, 0.41%): This subcategory covers the
solutions in which dependencies and database images are removed to address the
microservices issues. We identified 2 types of solutions in this subcategory,
including remove conflicting dependencies and remove database images.

  * •

Remove Logs (5, 0.25%): Logging is required to track the communication and
identify the failure in microservices systems. This subcategory covers the
solutions in which log messages and transaction IDs are removed to address the
microservices issues. We identified 3 types of solutions in this subcategory
to address the microservices issues, including eliminate log messages, remove
transaction id for logging, and unregister from registry.

  * •

Remove Documentation (5, 0.25%): Several microservices issues were addressed
by removing unnecessary or wrong information from project documentation. We
identified 2 types of solutions in this subcategory, including remove
unnecessary information and remove empty tags.

5\. Manage Infrastructure (160/1936, 8.26%): This category captures the
solutions that are based on efficient resource utilization for addressing the
microservices issues. We identified and classified 20 types of solutions in 2
subcategories (see Table IV and the Solution Taxonomy sheet in [19]). Each of
them is briefly described below.

  * •

Manage Storage (136, 7.02%): Typical microservices systems store their data in
dedicated databases for each service. We found several microservices issues
that occurred due to the lack of data storage. We identified 9 types of
solutions in this subcategory, in which the top three are allocate storage ,
clean cache, and extend memory.

  * •

Manage Networking (24, 1.23%): Networking is complicated in microservices
systems due to managing an explosion of service connections over the
distributed network. We collected 10 types of solutions in which the top three
are change proxy settings, server resource management, and disable server
groups.

6\. Manage Configuration and Execution (50/1936, 2.58%): Managing
configuration and execution enables developers to track changes in
microservices systems and their consuming applications over time, for example,
the ability to track the version history of configuration changes for multiple
instances of microservices systems. We identified and classified 13 types of
solutions in 2 subcategories (see Table IV and the Solution Taxonomy sheet in
[19]). Each of them is briefly described below.

  * •

Manage Execution (29, 1.49%): This subcategory collects the solutions for
issues related to managing commands for executing and configuring
microservices systems. We identified 5 types of solutions in this subcategory,
and among them execution and configuration management and execute multiple
commands are the top two types of solutions.

  * •

Manage Configuration (21, 1.08%): Managing configurations of each microservice
and their instances separately is a tedious and time-consuming task. In this
subcategory, we identified 8 types of solutions, and the top three types of
solutions are documentation for configuration management, change configuration
files, and correct uuid.

7\. Upgrade Tools and Platforms (47/1936, 2.42%): The updates of used tools
and platforms to develop and manage microservices systems help developers
address several issues due to old or legacy versions and protect the
microservices systems from security breaches. The interviewees also mentioned
a few solutions to address the microservices issues with the help of tools,
and one representative quotation is mentioned below.

“Normally, we adopt the solutions according to the type of issues. It could
include adding several tools, importing different packages, and adopting
successful practices. For example, we improve security by using several open-
source API gateways such as OKTA, Spring Cloud gateway, JWT token, and Spring
Security. To address service communication issues, we mainly use different
ways of communication according to the needs of projects, such as Kafka,
RabbitMQ, and Service Mesh. In addition, we automated our continuous
integration and delivery process using AWS Code Pipeline. AWS Code Pipeline
automates the project release process’s build, test, and deployment phases”
(P1, Software Architect, Developer).

We identified and classified 21 types of solutions in two subcategories (see
Table IV and the Solution Taxonomy sheet in [19]). Each of them is briefly
described below.

  * •

Upgrade Deployment, Scaling, and Management Platforms (39, 2.01%): This
subcategory gathers the solutions for addressing the microservices issues
related to the deployment, scaling, and management of CI/CD tools and
platforms. We identified 16 types of solutions, and the top 3 types of
solutions are upgrade container logging, upgrade load balancer, and upgrade
docker files.

  * •

Upgrade Development and Monitoring Tool Support (8, 0.41%): Development and
monitoring are crucial tasks for developers due to the distributed nature of
microservices systems. We identified 5 types of solutions in this subcategory,
and the top three types of solutions are upgrade kafka flags, upgrade zipkin
thrift, and disable tracking.

8\. Import/Export Artifacts (12/1936, 0.61%): We found several issues that can
be fixed by importing and exporting various artifacts. We identified and
classified 3 types of solutions in 2 subcategories (see Table IV and the
Solution Taxonomy sheet in [19]). Each of them is briefly described below.

  * •

Import Artifacts (9, 0.46%): This subcategory gathers the solutions related to
importing packages and libraries. We identified 2 types of solutions in this
subcategory, which are import packages, and import libraries to address the
scalability issues in microservices systems.

  * •

Export Artifacts (3, 0.15%): In this subcategory, we identified only one type
of solutions export packages to fix the Configuration issues in microservices
systems (e.g., conflict on configuration file names).

TABLE IV: Taxonomy of solutions of issues in microservices systems Category of Solutions | Subcategory of Solutions | Types of Solutions
---|---|---
Fix Artifacts (1056) | Fix Code Issue (912) | Fix Source Code of Issues (791)
Fix Illegal Symbols (Syntax) in Code (79)
Fix Testing Issue (107) | Debug Code (105)
Add Test Services in Containers (2)
Fix Build Issue (33) | Fix Errors in Build File (17)
Correct Build Type (8)
Fix GUI Issue (4) | Fix Backwards Incompatible UI (3)
Fix Broken Screenshot (1)
Add Artifacts (360) |  Add Features and Services (144) | Add Missing Feature (114)
Add Security Feature (26)
Add Interfaces, Templates, and Files (39) | Add File (37)
Add Templates (1)
Add Methods and Modules (33) | Add constructor (24)
Add Parameters in Method (8)
Add Security Certificates (1)
Add Classes and Packages (27) | Add Packages (15)
Add Properties(11)
Add Test Cases (28) | Add Test Cases to Validate Service (21)
Generate Correct Test Case (3)
Implement Patterns and Strategies (23) | Service Mesh Architecture (2)
Serverless Deployment (2)
Add Data Types, Identifiers, and Loops (21) | Add Identifiers (11)
Add Data Types (5)
Add Dependencies and Metrics (7) | Add Monitoring Metrics (3)
Add Dependencies (2)
Add APIs, Namespaces, and Plugins (7) | Add APIs (4)
Add Namespaces (2)
| Add Logs (4) | Add Trace ID (1)
| Add General Purpose Logger (1)
Modify Artifacts (212) |  Modify Methods and Objects (78) | Correct Method Definition (51)
Redefine Method Operations (8)
Modify Package, Module, and Documentation (64) | Improve Documentation (56)
Update Packages (6)
Modify Data Types and Identifiers (51) | Correct Naming (29)
Correct Data Types (8)
Modify APIs, Services, and Scripts(13) | Update Scripts (5)
Update Syntax (3)
Modify Database (4) | Modify Database Tables (2)
Modify Query Strings (3)
Remove Artifacts (38) |  Remove Data Types, Methods, Objects, and Plugins (20) | Remove Conflicting Dependencies (17)
Remove Database Images (1)
Remove Dependencies and Databases (8) | Remove Conflicting Dependencies (7)
Remove Code Dependencies (1)
Remove Logs (5) | Eliminate Log Messages(3)
Remove Transaction ID for Logging(1)
Remove Documentation (5) | Remove Unnecessary Information (3)
Remove Empty Tags (1)
Manage Infrastructure (160) | Manage Storage (82) | Allocate Storage (77)
Clean Cache (5)
Manage Networking (24) | Change Proxy Setting (8)
Server Resource Management (5)
Manage Configuration and Execution (50) | Manage Execution (29) | Manage Configuration Commands (24)
Execute Multiple Commands (2)
Manage Configuration (21) | Documentation for Configuration Management (7)
Change Configuration Files (4)
Upgrade Tools and Platforms (43) |  Upgrade Deployment, Scaling, and Management Platforms (35) | Upgrade Container Logging (6)
Upgrade Load Balancer (4)
Upgrade Development and Monitoring Tool Support (8) | Upgrade Kafka Flags (2)
Upgrade Zipkin Thrift (2)
Import/Export Artifacts (12) | Import Artifacts (9) | Import Packages (5)
Import Libraries (4)
Export Artifacts (3) | Export Packages (3)

Key Findings of RQ3: We found that not all the issue discussions provide the
information about their solutions, and finally we identified 1,899 issue
discussions containing information about the solutions with 36 solutions
mentioned by the interviewees and 1 solution mentioned by the survey
participants, which are 1,936 solutions in total. The solution taxonomy
consists of 8 categories, 32 subcategories, and 177 types of solutions. The
majority of solutions are related to Fix Artifacts (54.54%), Add Artifacts
(18.59%), and Modify Artifacts (10.95%).

###  3.4 Practitioners’ Perspective (RQ4)

We conducted a cross-sectional survey to evaluate the taxonomies of issues,
causes, and solutions in microservices systems built in RQ1, RQ2, and RQ3. We
provided a list of 19 issue categories and asked survey participants to
respond to each category on a 5-point Likert scale (Very Often, Often,
Sometimes, Rarely, Never). Similarly, regarding causes and solutions, we
provided 8 cause categories and 8 solution categories, and asked practitioners
to respond to each category on a 5-point Likert scale (Strongly Agree, Agree,
Neutral, Disagree, Strongly Disagree). We also asked three open-ended
questions to identify the missing issues, causes, and solutions in the
provided categories. We received 150 valid responses completed by
microservices practitioners from 42 countries across 6 continents. The results
of RQ4 are summarized in two tables (i.e., Table V and Table VI). This section
also provides representative quotations from practitioners for answering open-
ended questions with the sign. The practitioners’ perspectives on the issues,
causes, and solutions categories in microservices systems are presented in
Table V. Due to the limited space, we only presented the ‘percentage’ and
‘mean’ values of the practitioners’ responses to each category of issues,
causes, and solutions. The survey results about the practitioners’
perspectives on microservices issues, causes, and solutions are briefly
reported below.

Practitioners’ Perspective on Issues: We asked the microservices participants
which issue they faced while developing microservices systems. The majority of
the respondents mentioned that they face all of the issues while developing
microservices systems. The practitioners’ responses presented in Table V show
that IC1: Technical Debt (46.67% Very Often, 24.00% Often), IC2: Continuous
Integration and Delivery Issue (26.67% Very Often, 42.67% Often), and IC5:
Security Issue (18.67% Very Often, 64.00% Often) occur most frequently than
other categories of issues. Some practitioners also suggested 6 types of
issues (with 9 instances of issues) that were not part of the initial taxonomy
of issues in microservices systems. We added microservices practitioners’
suggested types of issues in Figure 4 and the Issue Taxonomy sheet in [19].
One representative quotation about other types of issues is depicted below.

“In my understanding, there should be some other issues such as lack of (i)
understanding of the implementation domain, (ii) defined process for
designing, developing, and deploying the projects, and (iii) expertise in
programming languages for developing microservices systems” (Application
Developer, SP2).

Practitioners’ Perspective on Causes: Our survey participants also confirmed
the identified causes that could lead to issues in microservices systems. The
practitioners’ responses presented in Table V show that CC1: General
Programming Error (41.33% Strongly Agree, 38.67% Agree), CC8: Fragile Code
(23.33% Strongly Agree, 46.67% Agree), and CC2: Missing Features and Artifacts
(12.67% Strongly Agree, 55.33% Agree) are the top three cause categories. Some
practitioners also suggested 5 types of causes (with 11 instances of causes)
that were not part of the initial taxonomy of causes in microservices systems.
We added microservices practitioners’ suggested types of causes in Table V and
the Cause Taxonomy sheet in [19]. One representative quotation about suggested
types of causes is depicted below.

“Several microservices issues can be occurred because of (i) separate physical
database, (ii) lack of resilience support for the whole application, (iii)
excessive tooling, (iv) lack of CI/CD (e.g., DevOps) culture in organizations,
and (v) lack of practitioners in teams who have multiple skills” (Business
Analyst, SP120)

Practitioners’ Perspective on Solutions: We also recorded the practitioners’
perspective about the solutions for the issues occurring during microservices
system development. The practitioners’ responses presented in Table V show
that SC1: Add Artifacts (42.00% Strongly Agree, 39.33% Agree), SC3: Modify
Artifact (27.33% Strongly Agree, 37.33% Agree), and SC6: Manage Configuration
and Execution (18.00% Strongly Agree, 48.67% Agree) and are the top three
solution categories that have been used to address the microservices issues.
One practitioner also suggested one type of solution (with one instance of
solution) that was not part of the initial taxonomy of solutions in
microservices systems. We added microservices practitioners’ suggested types
of solutions in Table V and the Solution Taxonomy sheet in [19]. One
representative quotation about the suggested solutions is depicted below.

“Regular training of the employees on the latest technologies and cloud
platforms for developing and managing microservices systems can address
several types of security, communication, and deployment issues” (DevOps &
Cloud Engineer, SP92).

TABLE V: Practitioners’ perspective (in %) on the issue, cause, and solution
categories in microservices systems Microservices Issues (VO-Very Often,
O-Often, S-Sometimes, R-Rarely, N-Never)
---
Categories | VO | O | S | R | N | Mean
Technical Debt | 46.67 | 24.00 | 12.67 | 5.33 | 1.33 | 3.93
Continuous Integration and Delivery (CI/CD) Issue | 26.67 | 42.67 | 15.33 | 7.33 | 8.00 | 3.73
Exception Handling Issue | 14.67 | 29.33 | 27.33 | 13.33 | 14.67 | 3.14
Service Execution and Communication Issue | 15.33 | 44.67 | 22.67 | 10.00 | 7.33 | 3.51
Security Issue | 18.67 | 64.00 | 8.67 | 6.00 | 2.67 | 3.90
Build Issue | 15.33 | 43.33 | 23.33 | 10.00 | 8.00 | 3.48
Configuration Issue | 14.00 | 38.67 | 25.33 | 10.67 | 11.33 | 3.33
Monitoring Issue | 12.00 | 40.67 | 22.67 | 16.67 | 6.67 | 3.31
Compilation Issue | 16.00 | 34.67 | 15.33 | 14.00 | 20.00 | 3.13
Testing Issue | 26.67 | 32.67 | 20.00 | 8.00 | 11.33 | 3.51
Documentation Issue | 13.33 | 42.00 | 14.67 | 11.33 | 18.67 | 3.20
Graphical User Interface (GUI) Issue | 7.33 | 30.67 | 22.00 | 16.67 | 20.67 | 2.79
Update and Installation Issue | 10.67 | 44.00 | 20.67 | 12.00 | 12.00 | 3.27
Database Issue | 18.00 | 40.67 | 18.00 | 16.00 | 7.33 | 3.46
Storage Issue | 12.00 | 38.67 | 22.00 | 17.33 | 10.00 | 3.25
Performance Issue | 17.33 | 46.00 | 14.67 | 14.67 | 6.67 | 3.51
Networking Issue | 12.00 | 34.67 | 22.00 | 20.00 | 10.67 | 3.15
Typecasting Issue | 12.67 | 34.00 | 21.33 | 10.67 | 18.00 | 3.03
Organizational Issue | 18.67 | 64.00 | 8.67 | 6.00 | 2.67 | 3.90
Causes of Issues (SA-Strongly Agree, A-Agree, UD-Undecided, D-Disagree, SD-
Strongly Disagree)
Categories | SA | A | U | D | SD | Mean
General Programming Error | 41.33 | 38.67 | 12.00 | 6.00 | 2.00 | 4.11
Missing Features and Artifacts | 12.67 | 55.33 | 25.33 | 4.67 | 2.67 | 3.70
Invalid Configuration and Communication Problem | 18.00 | 47.33 | 20.00 | 10.00 | 4.67 | 3.64
Legacy Versions, Compatibility, and Dependency Problem | 12.67 | 41.33 | 27.33 | 13.33 | 5.33 | 3.43
Service Design and Implementation Anomaly | 19.33 | 48.00 | 18.67 | 8.67 | 5.33 | 3.67
Poor Security Management | 18.67 | 42.67 | 21.33 | 10.00 | 7.33 | 3.55
Insufficient Resources | 16.00 | 46.00 | 22.67 | 10.00 | 5.33 | 3.57
Fragile Code | 23.33 | 46.67 | 12.67 | 12.67 | 4.67 | 3.71
Solutions for Issues (SA-Strongly Agree, A-Agree, UD-Undecided, D-Disagree,
SD-Strongly Disagree)
Categories | SA | A | U | D | SD | Mean
Add Artifacts | 42.00 | 39.33 | 8.67 | 6.00 | 3.33 | 4.11
Remove Artifacts | 16.00 | 60.67 | 14.00 | 6.67 | 2.00 | 3.83
Modify Artifact | 27.33 | 37.33 | 25.33 | 6.00 | 3.33 | 3.80
Manage Infrastructure | 14.00 | 48.00 | 23.33 | 10.00 | 4.00 | 3.58
Fix Artifacts | 18.00 | 46.67 | 24.00 | 6.00 | 4.67 | 3.68
Manage Configuration and Execution | 18.00 | 48.67 | 18.67 | 9.33 | 4.67 | 3.66
Upgrade Tools and Platforms | 15.33 | 52.00 | 22.00 | 8.00 | 2.00 | 3.71
Import/Export Artifacts | 14.67 | 50.00 | 28.67 | 3.33 | 2.67 | 3.71

Statistical significance on the issue, cause, and solution categories in
microservices systems: We analyzed the practitioners’ responses across one
pair of demographic groups in Table VI. The first column of Table VI lists the
categories of issues, causes, and solutions presented to the survey
participants. The subsequent columns of Table VI show the Likert Distribution,
Mean, p-value, and Effect Size. Analyzing survey data using the Likert
Distribution, Mean, p-value, and Effect Size provides insights into response
patterns, central tendency, statistical significance, and practical
significance, respectively. The practitioners’ responses are grouped into
Experience $\leq$ 6 Years vs. Experience > 6 Years group to check the test
whether there are statistical differences between the two groups on the same
variable or not.

The Likert Distribution shows the level of agreement and importance for each
issue, cause, and solution category. In contrast, the Mean indicates the
average of the Likert distribution for the issue, cause, and solution
categories. The p-value indicates statistical differences between Experience
$\leq$ 6 Years and Experience > 6 Years in the Experience Based Grouping
column. We used the non-parametric Mann–Whitney U test to test the null
hypothesis (i.e., there is no significant difference between the responses in
both groups). We describe the impact of the groups on the responses as
significant if the p-value is less than 0.05 (see in Table VI). The Effect
Size is measured by taking the mean difference of Experience > 6 Years and
Experience $\leq$ 6 Years.

Observations: We made the following observations based on the practitioners’
responses.

  * •

There are no major statistically significant differences in practitioners’
responses on the issue, cause, and solution categories in microservices
systems.

  * •

The observed statistically significant differences between experienced-based
grouping indicate that the experience of microservices practitioners does not
affect the survey responses.

  * •

The survey findings indicate that most issues are related to Technical Debt,
CI/CD, and Security; most causes are associated with General Programming
Errors, Fragile Code, and Missing Features and Artifacts; and most solutions
are associated with Add Artifacts, Modify Artifacts, and Manage Configuration
and Execution.

  * •

More than 50% of the respondents indicated that they frequently (Very Often
and Often) encountered different issues belonging to the given list of issue
categories.

  * •

Our results indicate that many practitioners rarely or never face GUI (16.67%
Rarely, Never 20.67%), Compilation (14.00% Rarely, 20.00% Never), and
Networking issues (20.00% Rarely, 10.66% Never) in microservices system
development.

Key Findings of RQ4: (i) There are no major statistically significant
differences in the issue, cause, and solution categories in practitioners’
responses, (ii) practitioners frequently face Technical Debt, CI/CD, and
Security issues, (iii) most causes are associated with General Programming
Errors, Fragile Code, and Missing Features and Artifacts, and (iv) most
solutions are associated with Add Artifacts, Modify Artifact, and Manage
Configuration and Execution. The survey participants generally confirmed the
categories of issues, causes, and solutions derived from mining developer
discussions in open-source microservices systems. However, the survey
participants also indicated several other types of issues, causes, and
solutions that help improve our taxonomies.

TABLE VI: Statistical significance on the issue, cause, and solution categories in microservices systems Categories | # | Likert Distributions | Experience Based Grouping
---|---|---|---