# Containerization in Multi-Cloud Environment: Roles, Strategies, Challenges,
and Solutions for Effective Implementation

Muhammad Waseem  Faculty of Information Technology, University of
JyväskyläJyväskyläFinland<EMAIL_ADDRESS>,  Aakash Ahmad  School of
Computing and Communications, Lancaster University LeipzigLeipzigGermany
<EMAIL_ADDRESS>,  Peng Liang  School of Computer Science, Wuhan
UniversityWuhanChina<EMAIL_ADDRESS>,  Muhammad Azeem Akbar  Software
Engineering Department, Lappeenranta-Lahti University of
TechnologyLappeenrantaFinland<EMAIL_ADDRESS>,  Arif Ali Khan  M3S
Empirical Software Engineering Research Unit, University of OuluOuluFinland
<EMAIL_ADDRESS>,  Iftikhar Ahmad  TietoEVRY OyTampereFinland
<EMAIL_ADDRESS>,  Manu Setälä  Solita OyTampereFinland
<EMAIL_ADDRESS>and  Tommi Mikkonen  Faculty of Information Technology,
University of JyväskyläJyväskyläFinland<EMAIL_ADDRESS>
###### Abstract.

Containerization in a multi-cloud environment facilitates workload portability
and optimized resource utilization. Containerization in multi-cloud
environments has received significant attention in recent years both from
academic research and industrial development perspectives. However, there
exists no effort to systematically investigate the state of research on this
topic. The aim of this research is to systematically identify and categorize
the multiple aspects of containerization in multi-cloud environment. We
conduct the Systematic Mapping Study (SMS) on the literature published between
January 2013 and March 2023. Eighty-six studies were finally selected and the
key results are: (1) Four leading themes on cloud computing and network
systems research were identified: ‘Scalability and High Availability’,
‘Performance and Optimization’, ‘Security and Privacy’, and ‘Multi-Cloud
Container Monitoring and Adaptation’. (2) Seventy-four patterns and strategies
for containerization in multi-cloud environment were classified across 10
subcategories and 4 categories. (3) Ten quality attributes considered were
identified with 47 associated tactics. (4) Four frameworks were introduced
from the analysis of identified challenges and solutions: security,
automation, deployment, and monitoring challenge-solution frameworks. The
results of this SMS will assist researchers and practitioners in pursuing
further studies on containerization in multi-cloud environment and developing
specialized solutions.

Containerization, Multi-Cloud Environment, Challenges, Solutions, Frameworks,
Systematic Mapping Study

††journal: TOSEM

##  1\. Introduction

The use of containers in multi-cloud environment has been widespread in the
industry for many years (Naik, 2016b). Containers are standalone and
executable packages of software that include everything needed to run an
application, such as code, system tools, libraries, and settings (Merkel et
al., 2014). Containers allow the packaging of an application along with its
required dependencies, which makes it easy to move the application across
different environments with little or no modification (Docker, 2021). On the
other hand, multi-cloud environment is the distribution of cloud assets
(Petcu, 2013). Using containers in multi-cloud environment allows
organizations to achieve flexibility, agility, and cost efficiency. Developers
can build applications in a consistent environment and easily move them
between multiple cloud platforms, thereby leveraging the unique strengths
(e.g., extensive infrastructure, seamless integration, advanced data analytic)
of each platform.

Containerization in multi-cloud is illustrated in Figure 1. This figure
provides an overview of applications deployed within various cloud
configurations including public, private, and hybrid models. Each cloud hosts
multiple applications encapsulated in containers, and completes different
types of tasks with their required binaries and libraries, emphasizing the
portability and isolation that containerization offers. A container platform
layer, which could be exemplified by systems like Kubernetes or Docker,
manages and orchestrates these containers across the different cloud
environments. This architecture enables a flexible and scalable approach,
allowing efficient application deployment and operations in a multi-cloud
setting, while maintaining consistency and resilience across platforms.

Figure 1. Context: Containerization in multi-cloud environment

However, utilizing containerization in multi-cloud environment is not without
challenges that organizations may face (Saif et al., 2022). These challenges
may arise during different phases and activities of container-based
application development including architectural design phase, system
implementation, or when establishing an automated development infrastructure.
Furthermore, the challenges can also present themselves during system testing,
the coding process, and the deployment phase. For instance, during the
architectural design phase, challenges may involve ensuring seamless
integration of containerized components with existing systems and selecting
appropriate container orchestration patterns, strategies, and tools that can
effectively manage containers across multiple clouds (Raj et al., 2018) (Baby
and Vysala, 2015), and while establishing an automated development
infrastructure, organizations may encounter difficulties in creating efficient
Continuous Integration/Continuous Deployment (CI/CD) pipelines that handle
multi-cloud deployment scenarios while maintaining consistent performance and
security standards (Debroy et al., 2018). Similarly, In the system
implementation phase, challenges may include dealing with potential
compatibility issues between containers and various cloud platforms, as well
as optimizing resource utilization to control costs effectively (Helali and
Omri, 2021).

Recent research (e.g., see Table 8) has highlighted the growing significance
of container utilization in multi-cloud environment. These studies have
explored various aspects, including container roles and strategies,
architectural patterns, Quality Attributes (QAs), and the tools and frameworks
employed. Additionally, these studies have explored the challenges associated
with automation, deployment, monitoring, and security of containerization
applications in multi-cloud environment. Despite the breadth of knowledge
available, such valuable information is dispersed across different
publications, spanning scientific research papers and the grey literature.
This fragmentation of knowledge about containerization applications in multi-
cloud environment presents a significant navigational challenge for
practitioners tasked with real-world implementations. They need to thoroughly
scrutinize numerous aspects, such as patterns and strategies for
containerization applications in multi-cloud environment, to locate specific
information relevant to their use cases, whether it be a pattern, a challenge,
or a solution to an existing problem. Consequently, this spreading of
knowledge restricts the development of a holistic understanding of the
containerization applications in multi-cloud environment, leaving
practitioners underprepared to devise comprehensive solutions.

Furthermore, this state of affairs of containerization in multi-cloud also
poses challenges for the academic community. Researchers bear the obligation
of navigating through this vast array of information to uncover the precise
aspects they seek - be it a unique pattern, an unexplored challenge, or an
innovative solution. Moreover, the distributed nature of this knowledge delays
the formation of a cohesive and unified understanding of the subject matter.
Recent studies (e.g., (Bordeleau et al., 2020) (Mihai et al., 2022) (Sarker,
2022)) have notably highlighted that challenges in design, development,
monitoring, and testing of containerization applications in multi-cloud
environment are deeply interconnected with the software development life
cycle.

To address this knowledge gap, we established the objective of this research
is to systematically identify and categorize various facets of container
utilization in multi-cloud environment. These facets encompass the roles of
containers, their implementation strategies, architectural patterns, quality
attributes, as well as the motivations of the customers. Additionally, we
aspire to explore the challenges and solutions related to automation,
deployment, monitoring, and security of containerization applications in
multi-cloud environment. To achieve this objective, we employed a systematic
approach to conduct a mapping study. We finally selected 86 studies that focus
on containerized applications in multi-cloud environment for further analysis.

The empirical findings and frameworks of this Systematic Mapping Study (SMS)
offer significant potential benefits for developers of container-based
applications. By exploring a diverse range of implementation strategies,
architectural patterns, and quality attributes, practitioners can gain
valuable insights to enhance their practices. Moreover, a detailed examination
of challenges and solutions related to automation, deployment, monitoring, and
security further provides practitioners with the knowledge to address problems
effectively and improve dedicated quality attributes. Additionally, these
findings and frameworks offer valuable support to practitioners in
comprehending various patterns, strategies, and tactics specific to
containerized applications in multi-cloud environment. The proposed frameworks
can act as a facilitator of knowledge sharing, training, and ensure a shared
understanding among practitioners regarding the complexities and solutions
within containerized applications in multi-cloud environment. Ultimately, this
SMS serves to empower software developers and practitioners, enabling them to
optimize their approaches and elevate the quality and efficiency of
containerized applications in multi-cloud environment.

Our SMS makes the following key contributions:

  * •

We developed four frameworks to address challenges and solutions in security,
automation, deployment, and monitoring for containerization in multi-cloud
environment.

  * •

We systematically identified and classified an extensive list of patterns and
strategies for container-based applications in multi-cloud environment.

  * •

We systematically identified and classified quality attributes and tactics for
containerized applications in multi-cloud environment.

  * •

We identified and classified a wide range of tools and frameworks for
containerized applications in multi-cloud environment.

  * •

We have publicly released a dataset, which is available online (Waseem et al.,
2024), to enable researchers and practitioners to access all collected data,
replicate, and validate our SMS. This dataset includes detailed hierarchies of
the developed frameworks, identified and classified container implementation
strategies, and the roles of containers in multi-cloud environment. It also
encompasses patterns, strategies, quality attributes, tactics, tools, and
frameworks for containerized-based applications in multi-cloud environment.

The paper is structured as follows: Section 2 describes the research
methodology employed. Section 3 presents the results and findings of this SMS.
Section 4 discusses the implications of the SMS results. Section 5 clarifies
the potential threats to the validity of this SMS. Section 6 reviews relevant
prior work and highlights the differences between our work and previous
research. Section 7 draws conclusions and outlines future research directions.

##  2\. Methodology

We conducted a Systematic Mapping Study (SMS) by following the guidelines in
(Petersen et al., 2008a) and augmenting them with SLR strategies (Keele et
al., 2007). Our SMS consists of three phases: specifying research questions
and search string, conducting the literature search, and performing data
analysis and documentation. Figure 2 illustrates the SMS execution process.

Figure 2. Schematic representation of the research methodology implemented in
this study

###  2.1. Research Questions

To conduct this SMS, we formulated six Research Questions (RQs), as outlined
in Table 1, aligning with the objective of our SMS. The table provides a
structured view of the RQs, their corresponding rationale, and categorization.

Table 1. Research questions and their rationale # |  Research Questions |  Rationale
---|---|---
Container Roles and Implementation
RQ1 |  What roles do containers play, and what strategies can be employed for implementing container-based applications in multi-cloud environment? |  To investigate the role of containers and explore implementation strategies in multi-cloud environment, with the objective of enhancing understanding and facilitating efficient deployment and management of applications across multiple cloud environments.
RQ2 |  What architectural patterns and strategies are utilized in the implementation of container-based applications in multi-cloud environment? |  To explore and document the various architectural patterns and strategies that are utilized in implementing container-based applications in multi-cloud environment, thus providing a roadmap for efficient and scalable application development and deployment.
RQ3 |  What are the quality attributes and the tactics associated with their implementation in container-based applications in multi-cloud environment? |  To examine and analyze the quality attributes required for container-based applications in multi-cloud environment and the corresponding tactics employed in their implementation, enabling the identification of best practices and guidelines for developing robust and reliable applications in such environments.
Challenges and Solutions
RQ4 |  What are the challenges related to security, automation, deployment, and monitoring in container-based applications in multi-cloud environment? |  To comprehensively evaluate and gain insights into the challenges surrounding security, automation, deployment, and monitoring concerning container-based applications in multi-cloud environment, this question aims to provide valuable insights into the obstacles and intricacies that organizations encounter while implementing and managing these applications in such a dynamic setting.
RQ5 |  How to address security, automation, deployment, and monitoring in container-based applications within multi-cloud environment? |  This question seeks to investigate solutions aimed at tackling the challenges of security, automation, deployment, and monitoring within container-based applications operating in multi-cloud environment. By providing actionable guidance and recommendations, the goal is to elevate the overall management and performance of these applications in the context of multi-cloud environment.
Tools and Frameworks
RQ6 |  What tools and frameworks are used to implement container-based multi-cloud applications? |  To investigate and identify the specific tools and frameworks utilized in the implementation of container-based multi-cloud applications, providing an overview of the technological landscape and aiding developers and organizations in making informed decisions when selecting appropriate tools for their application deployment and management.

###  2.2. Study Search and Selection Process

The study search and selection process for this study is divided into two
phases. In the first phase, we conducted a primary search using a specific
search string on various databases (see Table 2 and the Summary sheet in
(Waseem et al., 2024)). In the second phase, we applied the snowballing
technique on primary studies that were selected from the first phase.

Table 2. Search string and targeted search area for databases Search String
---
_((multicloud OR multi-cloud OR multi cloud) AND_
_(container*) AND (automation) And (security) AND (monitoring) AND (deploy*))_
Databases
Database | Targeted search area
ACM Digital Library | Paper title, abstract
IEEE Xplore | Paper title, keywords, abstract
Scopus | Paper title, keywords, abstract
SpringerLink | Paper title, abstract
ScienceDirect | Paper title, keywords, abstract
Wiley Online Library | Paper title, abstract
Engineering Village | Paper title, abstract
Web of Science | Paper title, keywords, abstract

####  2.2.1. Primary Search

The primary search involved querying digital databases (see Table 2) using
customized search string. The search strings were executed concurrently on
eight databases, as illustrated in Figure 2, spanning between January 2013 and
March 2023 when we started this SMS. We chose 2013 as the starting point
because it is the year when container technologies like Docker emerged
(Docker, 2023), marking a key phase in containerization and cloud computing.
This period also saw a rise in relevant research to containers in multi-cloud
environment, making it ideal to understand the development of containerized
applications in multi-cloud environments over the past decade. We followed the
steps mentioned below to select the relevant studies.

Table 3. Inclusion and exclusion criteria for selecting primary studies in this SMS Selection Criteria |  Inclusion Criteria |  Exclusion Criteria
---|---|---
_Language_ |  English |  Non-English
_Study Type_ |  Primary studies including peer-reviewed journal articles, book chapters, conference papers, workshops, and symposium papers |  Secondary studies or non-peer-reviewed content (e.g., blogs, webpages, videos, white papers, technical reports, grey literature)
_Study Focus_ |  Studies addressing container roles and strategies, architectural patterns, quality attributes, adoption drivers, automation, deployment, monitoring, security challenges, solutions, and tools/frameworks for multi-cloud environment. |  Studies that focus solely on traditional cloud computing, single-cloud environments, or unrelated topics
_Study Duration_ |  Studies published between January 2013 and March 2023 |  Studies published before January 2013 or after March 2023

  * •

Step 1: Extraction of Studies: We ran custom search strings in the selected
databases (see Table 2) to retrieve study titles, author names, publication
years, venues, publication types, and abstracts. This initial search yielded
5,676 studies from eight databases.Following initial retrieval, we did not
apply Cohen’s Kappa as this step is automated and does not involve subjective
decision-making.

  * •

Step 2: Title and Keyword Screening: This step involves reviewing the titles
and keywords of the collected studies to assess their relevance to the
research topic. Initially, we removed any duplicated studies obtained from
different databases (such as IEEE Xplore and Scopus) by sorting them in
ascending order. This resulted in the elimination of several hundred duplicate
studies. We divided the remaining studies between two authors (R1, R2), who
independently applied the inclusion and exclusion criteria to identify studies
relevant to our research objectives, as detailed in Table 3. To assess the
consistency of study selection and mitigate bias, a Cohen’s Kappa analysis
(Cohen, 1968) was performed on a random sample of 40 studies, as illustrated
in Formula 1:

(1) $\kappa=\frac{P_{o}-P_{e}}{1-P_{e}}$

where $P_{o}$ represents the relative observed agreement among authors, and
$P_{e}$ the hypothetical probability of chance agreement. The results,
recorded in Table 4.a, indicated a Cohen’s Kappa of 0.531. This value reflects
a moderate level of agreement between R1 and R2. Any disagreements were
systematically resolved through discussions among all authors, leading to a
consensus. Following this process, the number of studies was reduced to 809
for further analysis.

  * •

Step 3: Abstract-based Screening: During this step, we conducted a thorough
review of the abstracts of the collected studies to determine their relevance
to our research topic. Two researchers (R1 and R2) independently examined each
abstract, assigning a status of ”relevant,” ”irrelevant,” or ”doubtful” with
respect to our research goals. To reduce subjective bias and to quantify the
consensus between the researchers, we conducted a Cohen’s Kappa analysis on a
subset of 30 randomly selected studies. The outcomes of this assessment are
depicted in Table 4.b, which show substantial agreement between R1 and R2,
with 25 studies being unanimously considered relevant and only 4 studies
disagree (3 with one researcher favoring relevance and the other not, and 1
where both disagreed) The Kappa value, calculated based on the observed
agreements and disagreements was approximately K= 0.268 which suggests a fair
agreement between the two researchers. Consequently, upon the compilation of
step, we got 303 studies for more detailed scrutiny.

  * •

Step 4: Full-Text Screening: In this step, we reviewed the full texts of 303
studies that had been shortlisted through abstract-based screening. To ensure
consistency in our review process, we randomly chose 20 studies (see Table
4.c) for a Cohen’s Kappa analysis to measure the agreement between two
researchers. The resulting Kappa value was K=0.318, which represents a fair
level of agreement. This measure confirms that our selection criteria were
applied consistently during the full-text screening studies.

Table 4. Kappa Agreement across different screening steps.

(a) Title and Keyword Screening on 40 Randomly Selected Studies | R1 Yes | R1 No
---|---|---
R2 Yes | 35 | 2
R2 No | 1 | 2
Total | 36 | 4

(b) Abstract-based Screening on 30 Randomly Selected Studies | R1 Yes | R1 No
---|---|---
R2 Yes | 25 | 3
R2 No | 1 | 1
Total | 26 | 4

(c) Full-Text Screening on 20 Randomly Selected Studies | R1 Yes | R1 No
---|---|---
R2 Yes | 16 | 2
R2 No | 1 | 1
Total | 17 | 3

####  2.2.2. Snowballing

In Phase 2, we utilized the snowballing technique, as described in (Wohlin,
2014), to examine references within primary studies for identifying additional
relevant studies. This strategy was augmented by forward snowballing, where we
gathered studies that cited the selected studies, and backward snowballing,
which involved using references within the selected studies. Notably, we
encountered several dozen studies during the forward and backward snowballing
that had been excluded in primary search. This phase resulted in the addition
of 8 more studies, bringing the final count to 86.

###  2.3. Data Extraction and Analysis

Data Extraction: The data extraction form was designed based on predefined
data items (see Table 5) that were formulated to address the RQs specified in
Table 1. To ensure the reliability of the extracted data items, the pilot data
extraction was conducted on ten studies by the first author, and all the other
authors assessed the extracted data. Subsequently, the first author employed a
revised set of data items (e.g., D11, D12, D16), determined after evaluating
the extracted data items, for the formal data extraction from the selected
studies. To mitigate the personal bias and ambiguity, all authors engaged in
discussions regarding the extracted data. The data items labeled as D1 to D11
present a summary of the demographics of the primary studies selected, while
D12 to D26 are specifically employed to address RQ1 to RQ6. A concise
description of each data item is also presented in Table 5. Finally, Google
Sheets were used to record and further analysis of the extracted data.

Data Analysis: We employed a descriptive statistics to analyze the
quantitative data from data items D1, D5, D7, D8, and D9. For the remainder of
the data, which primarily consist of qualitative free-text descriptions (e.g.,
study aim, roles of containers, challenges, and solutions), we conducted a
thematic analysis in accordance with the guidelines outlined in (Braun and
Clarke, 2006). Our thematic analysis process consists of the following steps:

  1. (1)

Data Familiarization: We conducted a thorough review of the selected studies
by repeatedly reading through them and meticulously noting key points related
to study aims (D7), contributions (D8), roles of containers (D9),
implementation strategies (D10), architectural patterns (D11), quality
attributes (D12), tactics (D13), motivations (D14), and challenges and
solutions (D15-D21) in automation, deployment, monitoring, and security
challenges and solutions, as well as the tools, languages, and frameworks
(D22) employed.

  2. (2)

Generation of Initial Codes: After developing a thorough understanding of the
data, we created an initial set of codes derived from the information
extracted concerning the data items identified in the previous step.

  3. (3)

Identification of Types and Emerging Themes: In this step, we conducted a two-
tiered analysis. Initially, we examined the codes to ascertain their types.
Subsequently, we developed subcategories based on these types, and then
formulated overarching categories that encompass the related subcategories.

  4. (4)

Critical Evaluation of Types, Subcategories, and Categories: All authors
actively participated in rigorously reviewing and refining the coded data,
including types, subcategories, and categories. During this collaborative
process, we redefined, merged, or dropped certain themes based on collective
input and discussion.

  5. (5)

Defining and Naming Categories: At this point, we provided explicit
definitions for each of the identified themes and further refined them,
ensuring that the terminology used for the categories was precise and
unambiguous.

Table 5. Data items to be extracted in this SMS Code | Data Item | Description | RQ
---|---|---|---
D1 | Index | The ID of the study | Demographics
D2 | Publication Year | Publication year of the study
D3 | Publisher | The publisher of the study
D4 | Venue | The name of the publishing venue
D5 | Publication Type | Journal, conference, workshop, and book chapter
D6 | Authors’ Affiliation | The affiliation of the authors
D7 | Study Aim | The aim of the paper
D8 | Study Contribution | The contributions made by the paper
D9 | Role of Containers | Role of containers in the study | RQ1
D10 | Container Implementation strategy | Strategy used for container implementation
D11 | Architectural Pattern and Strategies | Architectural patterns and strategies reported in the study | RQ2
D12 | Quality Attributes | Quality attributes discussed in the study | RQ3
D13 | Tactics | Tactics used in the study
D14 | Automation Challenges | Challenges in automation discussed in the study | RQ4
D15 | Automation Solutions | Solutions for automation challenges discussed in the study | RQ5
D16 | Deployment Challenges | Challenges in deployment discussed in the study | RQ4
D17 | Deployment Solutions | Solutions for deployment challenges discussed in the study | RQ5
D18 | Monitoring Challenges | Challenges in monitoring discussed in the study | RQ4
D19 | Monitoring Solutions | Solutions for monitoring challenges discussed in the study | RQ6
D20 | Security Challenges | Challenges in security discussed in the study | RQ4
D21 | Security Solutions | Solutions for security challenges discussed in the study | RQ5
D22 | Tools, Languages, and Frameworks | Tools, programming languages, and frameworks | RQ6

The process of obtaining research themes executed in this SMS is shown in
Figure 2. Two researchers participated in the process in order to reduce
personal bias. The most important activity of this process is brainstorming
sessions that were mainly conducted while reviewing, defining, and naming the
research themes. In these sessions, both researchers discussed and validated
the research themes found.

Concerning data item D9 (testing approaches) and D10 (testing challenges), we
used the open coding and constant comparison techniques from Grounded Theory
(Glaser and Strauss, 1967) to analyze the qualitative data extracted from the
selected studies.

Finally, we provided a replication package (Waseem et al., 2024) containing
the results of each phase of the study selection process (e.g., Phase 1, Phase
2) and detailed results (e.g., Contributions, Patterns, QAs, Challenges,
Solutions) for verification and validation purposes of this SMS.

##  3\. Results

This section presents the results of the SMS,derived from analyzing data from
chosen studies. Section 3.1 provide a summary of the demographic data,
classifications, and the mapping of research themes identified in the studies.
Section 3.2 presents the results concerning container roles and implementation
strategies. Section 3.3 reports on the patterns and strategies employed in
containerized-based applications in multi-cloud environment. Section 3.4
presents the QAs and related tactics for containerized applications in multi-
cloud environment. Section 3.5 presents the security challenges and solution
framework, while Section 3.6 presents the automation challenges and solution
framework. Section 3.7 presents the deployment challenges and solution
framework, and Section 3.8 presents the monitoring challenges and solution
framework. Lastly, the tools and frameworks are reported in Section 3.9.

###  3.1. Demographics, Research Aims, and Contributions

Studies Yearly Distribution: Figure 3-a illustrates the annual distribution of
studies found in eight databases from January 2013 to March 2023. After
applying our selection criteria, the final dataset includes studies published
from 2014 to 2022. We found no primary studies on containerization in multi-
cloud environments meeting our criteria in the first three months of 2023. The
bars displays yearly study counts, with a trend showing the overall direction.
A peak in 2017, followed by a decrease and occasional fluctuations, suggests
an early spike in interest then more targeted research.

Figure 3. Demographics of the Selected Studies, Including: a) Frequency and
Categories of Publications; b) Publisher of the Research; c) Affiliations of
the Authors

Publishers Distribution: The pie chart presented in Figure 3-b illustrates the
distribution of identified studies among different publishers. The largest
portion of the studies, accounting for 72.09%, is published by IEEE,
indicating its significant contribution to the field. Scopus follows with
11.62%, while Engineering Village, ACM, and Springer Link represent smaller
portions at 6.97%, 5.81%, and 3.48%, respectively. The chart emphasizes the
dominance of IEEE in the identified studies, suggesting its importance as a
primary source of relevant literature.

Study Types: Figure 3-a provides a breakdown of the types of the selected
studies. It is apparent that conferences dominate as the most common venue,
comprising a substantial 72.09% of the total publications. Journals, which are
often considered as venues for more mature and rigorously peer-reviewed
research, make up 19.76%. This indicates a strong preference for disseminating
information through conferences, which may be due to the faster publication
process and opportunities for networking and collaboration. Symposiums and
workshops account for 6.97% and 1.16% respectively, suggesting that these are
less common avenues, possibly due to their more specialized or focused nature.

Authors’ Affiliations: Figure 3-c illustrates the distribution of authors’
affiliations in the collected publications. A staggering 93.02% of the
affiliations are associated with academia. This overwhelmingly high percentage
emphasizes the central role that academic institutions play in the production
and dissemination of research in the field. Industry affiliations represent a
mere 4.65%, which could imply that industry professionals may be more focused
on practical applications and might not publish frequently in the venues
considered. Mixed affiliations, at 2.32%, indicate a small proportion of
collaborations between academia and industry. This low rate of cross-
affiliation collaboration may suggest that there is potential for increased
synergy between academic and industrial entities in future research endeavors.

Mapping of Research Themes: The thematic analysis of selected studies aims on
containerization in multi-cloud environment, as illustrated in Figure 4,
indicating various significant insights.The prominent research themes
identified include “Scalability and High Availability” with 20 studies, and
three other themes: ’Security and Privacy”m “Multi-Cloud Container Monitoring
and Adaptation”, and “Performance and Optimization”, each attracting attention
in 15 studies. This highlights that the core areas of interest in multi-cloud
environment are proficient container management, scalability, and performance
optimization. “Security and Privacy’ also stands as a considerable theme, as
demonstrated by the 15 studies focusing on areas such as secure container
handling, data security, and blockchain applications. This emphasizes the
necessity of protecting data and applications in the context of containers
spreading across multi-cloud environment.

Moreover, it is noted that several studies encapsulate more than one theme.
For instance, study S19 is catalogued under both ‘Multi-cloud Orchestration
and Management’ and ‘Application Deployment in Multi-cloud Container’,
signifying that it encompasses aspects of both Kubernetes and microservices
architecture.

In essence, the exploration into containerization in multi-cloud environment
is substantially inclined towards, scalability, performance optimization
security, and orchestration, with several studies showcasing inter-thematic
relevance. This is suggestive of the diverse challenges and opportunities
present in deploying containers within multi-cloud contexts.

Major Contributions of the Selected Studies: Table 6 provides an overview of
the contributions made by the selected studies. The contributions are
primarily categorized into five main categories and 22 subcategories (see the
Contributions sheet in (Waseem et al., 2024)), through thematic analysis. The
selected studies primarily focused on proposing and validating reference
architectures, security approaches, cloud management and interoperability
frameworks, orchestration and deployment approaches, as well as approaches and
tools for optimization and deployment of containerized applications in multi-
cloud environment.

The studies related to the reference architecture category mainly contributed
to the understanding and design of architectures specifically tailored for
multi-cloud containerization. They aimed to address the complexities
associated with containerization in multi-cloud environment. On the other
hand, the studies in the security approaches category discuss concerns related
to cloud analysis, security and performance, cybersecurity, and access control
within multi-cloud contexts.

Figure 4. Selected studies research aims in containerization for multi-cloud
environment.

The studies on cloud management and interoperability frameworks primarily aim
to improve resource and service management, monitoring, and deployment in
multi-cloud environment. Their goal is to enhance the performance and
efficiency of managing resources and services across multiple clouds.
Meanwhile, the research in the area of orchestration and deployment approaches
introduces methods for bench-marking, optimizing, and simulating container
orchestration in multi-cloud settings. The objective of these studies is to
streamline the process of deploying and orchestrating containers within multi-
cloud environment.

Lastly, the research on optimization and deployment approaches and tools
focuses on improving container deployment and creating efficient management
solutions for multi-cloud environment. These studies underscore the importance
of optimizing resource allocation and developing practical management tools
for use in multi-cloud settings. Overall, these studies contribute to
proposing reference architectures, security approaches, cloud management, and
interoperability frameworks. These studies also offer strategies and tools for
orchestration, deployment operations, and optimization in multi-cloud
containerization.

Table 6. Selected Studies’ Contributions on Containerization in Multi-Cloud Environment  Category | Subcategory | Study #
---|---|---
Reference Architectures and Models for Cloud | General Cloud Architectures | S26, S33, S35, S42, S76
Fault Tolerance & Self-Healing | S12, S55, S67, S74
Migration & Testing Architecture | S17, S79
Domain-Specific Language | S80, S82
Performance Modeling | S28, S62
Optimization & Reliability Model | S50, S66
IoT & Service Mesh Architecture | S05, S59
Security Approaches | Cloud Analysis Approach | S18, S24, S37, S65, S69, S86
Security and Performance Solution | S10, S23, S38, S75, S77
Cybersecurity & Access Control | S11, S30, S34, S71, S73
Security Approach | S06, S25, S40, S83
Cloud Management and Interoperability Framework | Resource & Service Management Framework | S48, S49, S47, S15, S58
Monitoring & Deployment Framework | S19, S45, S46, S08
Interoperability Framework | S03, S81
Evaluation & Modeling Framework | S20, S85
Orchestration and Deployment Approach | Benchmarking & Optimization Approach | S01, S02, S41, S51, S68
Service Management Approach | S07, S14, S29
Simulation & Integration Approach | S27, S31, S56
Storage & Services Solution | S64, S75
Optimization and Deployment Approaches and Tool | Optimization & Scheduling Approach | S22, S32, S36, S39
Multi-Cloud Platform | S09, S16, S43, S60, S72
Deployment Approach and Tool | S21, S52, S57, S61

Takeaway 1: The research into containerization within multi-cloud environments
emphasizes the critical importance of scalability, security, and performance
optimization. These elements are foundational for proficient container
management and ensuring high availability and efficiency in handling
containerized applications across diverse cloud platforms. Takeaway 2:
Interoperability and orchestration emerge as pivotal themes, highlighting the
necessity for sophisticated strategies that enable seamless integration and
management of containers across multiple cloud environments. This underlines
the evolving complexity and the need for innovative solutions in multi-cloud
orchestration, including the use of Kubernetes and microservices architecture
to address these challenges.

###  3.2. Container Roles and Implementation Strategies (RQ1)

The next two subsections describe the roles and implementation strategiesS
related to containers in multi-cloud environment.

####  3.2.1. Container Roles

Table 7 presents a systematic breakdown of the roles of containers in various
aspects of multi-cloud environment. It is organized by the “Category Name” and
the “Role of Containers” which detail the specific context or usage of
containers in that category.

1\. Container-based Application Deployment: This category discusses the use of
containers for the deployment of various types of applications. This includes
web apps, Platform as a Service (PaaS) solutions, data-intensive applications,
and Internet of Things (IoT) apps, among others. Containers provide a means of
packaging and distributing applications, including their dependencies, across
various environments.

2\. Container-based Service Hosting: This category refers to the use of
containers as a hosting environment for different services. Container-based
service hosting allow for scaling and isolation of these services, creating an
efficient multi-cloud environment.

3\. Container-Based Resource Management: This category discusses the use of
containers to manage computational resources. Containers allow for better
multi-tenancy solutions, resource sharing, and reduction in complexity, among
other benefits.

4\. Container Technology Features: This category discusses the specific
features and advantages of container technology, such as lightweight
virtualization, use in edge computing, and compatibility with orchestration
tools like Kubernetes.

5\. Containerization in App Development and IoT: The focus of this category is
on the use of containers for app development, particularly in the context of
IoT. Containers provide a means of encapsulating software units, increasing
portability, improving efficiency, and integration with DevOps practices,
among other benefits.

6\. Performance and Efficiency through Containers: This category discusses how
containers can enhance performance and efficiency in a cloud environment.
Containers can improve scalability, isolation, and cost-effectiveness, and
also aid in platform optimization.

7\. Workload Management via Containers: This category is about the use of
containers to manage computational workloads across multiple cloud
environments. Containers facilitate the migration and execution of these
workloads across different platforms.

8\. Security and Compliance via Containers: In this category, containers are
discussed as a tool for enhancing security and compliance in multi-cloud
environment.

9\. Other Container Uses: This category includes various other uses of
containers, such as data storage, support in architecture, blockchain
reliability, scheduling algorithm evaluation, and virtual network composition.

Each of these categories demonstrates a dedicated aspect of how containers can
contribute to efficient and effective multi-cloud management. The studies
listed in each row (e.g., S01, S02) provide a more in-depth discussion and
evidence for the roles and applications of containers.

Table 7. Roles of containers in multi-cloud environment Category | Role of Containers | ID
---|---|---
Container-based Application Deployment | Containerized Web-App Deployment | S01
Container-Driven App Deployment | S05, S08
Containerized PaaS Prototyping | S09
Data-Intensive App Deployment via Containers | S28
Containerized App Management | S29
Container-Based Aneka Service Deployment | S51
Container-Driven Microservices Deployment & Scaling | S62
Container-Based Multi-Cloud App Deployment & Scaling | S74
Containerized IoT App Deployment | S85
Container-based Service Hosting | Container-Based App Component Hosting | S02
Containerized Service Hosting | S15
Container-Driven MicroCloud Scaling & Isolation | S42
Container-Based Resource Management | Container-Based Multi-Tenancy Solution | S04
Container-Driven Resource Sharing & Multi-Tenancy | S06
Multi-Tenant Service Management via Containers | S17
Container-Enabled OS-Level Virtualization | S40
Container-Based Complexity Reduction | S41
Container Technology Features | Container Usage in Edge Computing | S22
Container-Enabled Lightweight Virtualization | S24
Key Features of Container Technology | S25
Container-Based Edge Computing Virtualization | S35
Container Leverage in Kubernetes | S70
Containerization in App Development & IoT | App Development & IoT Integration via Containers | S16
Container-Based OpenVAS & Integration of Chef | S30
Containers as Standardized Software Units | S32
Container Placement & Orchestration in Fog Computing | S33
Container Deployment Speed & DevOps Integration | S36
Containerizing Apps for Privacy & Compliance | S37
Container-Driven Portability & Efficiency Improvement | S39
Preventing Vendor Lock-In via Containers | S63
Container-Based Extensible App Development | S76
Performance & Efficiency via Containers | Scalable App Building via Containers | S18
Container-Driven SaaS Performance Isolation | S23
Cloud Performance Enhancement via Containers | S48
Evaluating Performance Impact of Containers | S59
Cost Efficiency of Microservices via Containers | S68
Container-Based Cloud Platform Optimization | S69
Workload Management via Containers | Container-Based Virtual Cluster Implementation | S21
Multi-Cloud Workload Migration via Containers | S60, S65
Seamless Workload Migration via Containers | S65
Container-Based Multi-Cloud Platform Execution | S79
Other Container Uses | Container-Based Azure Data Storage | S54
Container & VM Support in Architecture | S55
Blockchain-Based Multi-Cloud Reliability via Containers | S66
Evaluating Scheduling Algorithm via Containers | S67
Container-Based Virtual Network Composition | S72

Takeaway 3: Containers play multifaceted roles in multi-cloud environment,
encompassing aspects such as application deployment, service hosting, resource
management, and security enhancement. Takeaway 4: Containers can boost
performance and efficiency in cloud settings, facilitating resource sharing,
workload management, platform optimization, and cost savings. Takeaway 5:
Containers significantly aid in IoT and application development, providing
encapsulation of software units, integration with DevOps, and improving
deployment portability and efficiency. Takeaway 6: A variety of strategies
have been employed for container implementation in multi-cloud environment,
focusing on areas like container deployment, orchestration, security,
performance scaling, cloud service networking, and data storage.

####  3.2.2. Implementation Strategies

Table 8 provides an overview of various implementation strategies for
container technology in multi-cloud environment. The strategies are grouped
into several broad categories, each with a unique focus and varying
subcategories. Within these subcategories, each implementation strategy is
listed with a concise and revised name for reference.

1\. Container Deployment and Management: This category focuses on how
containers are deployed and managed in multi-cloud environment. An example
from the “Deployment Strategies” subcategory is “Automated Orchestration for
Web Applications”, where container deployment is automated for web
applications. From the “Container Management” subcategory, the strategy
“Container Image Optimization” highlights how the size and efficiency of a
container can be optimized.

2\. Container Orchestration: In this category, the focus is on coordinating
and managing the life cycle of containers in a multi-cloud setup. An instance
from the “Cloud Orchestration” subcategory is “SLA-Based Container
Strategies”, illustrating how Service Level Agreements (SLAs) can guide the
orchestration of containers. In the “Microservices Orchestration” subcategory,
“Microservices Architecture in IoT Cloud Apps” represents a way to orchestrate
microservices in IoT applications using containers.

3\. Container Security: This category focus on the security of containers. An
example from the “Container Placement” subcategory is “Secure Placement with
Deep Learning”, which employs deep learning techniques to secure the placement
of containers. From the “Security Policies” subcategory, the strategy “Access
Control and Defense for Container Security” suggests ways to control access
and defend containers from security threats.

4\. Performance and Scaling: This category addresses the optimization of
container performance and the scaling of container-based applications. For
example, from the “Performance Optimization” subcategory, “LXC Virtual Cluster
for Performance” shows how Linux Containers (LXC) can be used to enhance the
performance of virtual clusters. An example from the “Scaling Strategies”
subcategory is “Edge Data Streaming and Real-Time Database”, highlighting a
strategy for scaling data streaming and real-time databases at the edge.

5\. Cloud Service and Networking: This category deals with the management of
cloud services and networking in containerized environments. “Multi-Tenant
Service with Docker and Cloudant” from the “Cloud Service Management”
subcategory illustrates how Docker and Cloudant can be utilized to manage
multi-tenant services. From the “Networking” subcategory, the strategy “Edge
Scheduling with Location-Awareness” proposes a way to schedule network
services at the edge based on location.

6\. Data Storage and Persistence: This category discusses strategies related
to data management in containerized environments. For instance, the strategy
“Firebase Cloud for Real-Time Database” presents an approach that uses
Firebase Cloud to manage real-time databases in container-based applications.

Takeaway 7: Containers significantly aid in IoT and application development,
providing encapsulation of software units, integration with DevOps, and
improving deployment portability and efficiency. Takeaway 8: A variety of
strategies have been employed for container implementation in multi-cloud
environment, focusing on areas like container deployment, orchestration,
security, performance scaling, cloud service networking, and data storage.

Table 8. Container implementation strategies in multi-cloud environment Category | Subcategory | Implementation Strategies | ID
---|---|---|---
Container Deployment and Management | Deployment Strategies | Automated Orchestration for Web Applications | S02
Kubernetes for Containerized Applications | S05
Multi-cloud App Development Framework | S07
Modular Approach to Container Implementation | S08
Deployment with Docker Images | S09
Docker Image Creation and Management | S31
Container Management | Container Image Optimization | S32
Efficient Image Reorganization | S36
Orchestrating with Docker and Kubernetes | S37
Executable Images on Multi-cloud Platforms | S79
Container Orchestration | Cloud Orchestration | SLA-Based Container Strategies | S04
Task Division and Scaling in Container Strategy | S12
Location-Aware Service Brokering | S14
Docker Containers for Multi-tenant Services | S17
Resource Acquisition for Aneka Containers | S51
Cross-Cloud Container and VM Management | S60
Container Orchestration Technologies Overview | S33
Microservices Orchestration | Microservices Architecture in IoT Cloud Apps | S16
ADAPT Deployment of Microservices | S27
Microservices Benchmark with Docker and Kubernetes | S62
Container Security | Container Placement | Secure Placement with Deep Learning | S06
Container Placement Strategies | S39
Security Policies | SLA-Oriented Performance Isolation | S23
Access Control and Defense for Container Security | S25
HIP Integration for Docker Security | S77
Secure Connectivity in Multi-tenant Environment | S72
Signed URLs for Access Control in Multi-cloud | S71
Performance and Scaling | Performance Optimization | LXC Virtual Cluster for Performance | S21
Scalable Multi-layer Architecture with Containers | S28
Reactive Auto-scaling and Resource Provisioning | S48
Istio for Control Plane Impact and Latency Reduction | S59
Scaling Strategies | Edge Data Streaming and Real-Time Database | S35
MicroCloud Architecture for Efficient Scaling | S42
Distributed Workloads with Kubernetes | S70
Hybrid Abstractions for Container and Host | S40
Cloud Service and Networking | Cloud Service Management | Multi-Tenant Service with Docker and Cloudant | S17
Multi-Cloud Runtime Environment | S76
Abstraction Layer for Cloud Service Standardization | S63
Networking | Edge Scheduling with Location-Awareness | S22
Openstack Networking with Kuryr | S55
Federated Framework for Cloud Storage | S64
Host Identity Protocol for Docker Networking | S77
Data Storage and Persistence | Data Management | Firebase Cloud for Real-Time Database | S35
High Availability with SpyStorage MCSS | S75
Data Storage with Object Store Service | S76

###  3.3. Pattern and Strategies (RQ2)

Table 9 presents various patterns and strategies employed in containerized-
based applications in multi-cloud environment. These patterns and strategies
are divided into various categories and subcategories, each of which
highlights different aspects of multi-cloud architecture, management,
security, resilience, and migration patterns and strategies. Each category is
briefly reported below.

1\. Cloud Architectural Patterns and Models: This category reports different
architecture patterns, communication and networking patterns, deployment
patterns, and service models for containerized-based applications in multi-
cloud environment. Architecture Patterns involve several design models for
structuring the components of a system, providing a template that guides
system designers in building complex systems. The leading patterns identified
are microservice architecture and service-oriented architecture (SOA), both of
which focus on building applications as suites of independently deployable
services. Communication and Networking Patterns relate to how different
services within the architecture communicate and network with each other.
Service mesh and Service Chaining (SC) are the main patterns observed.
Deployment Patterns pertain to different strategies for deploying software
systems in multi-cloud environment. The two leading patterns are Blue-Green
Deployment and Object Store Service. Service Models involve different ways of
providing cloud services, with Platform-as-a-Service (PaaS) and Software-as-a-
Servic (SaaS) leading the pack.

2\. Cloud Management and Resource Allocation Strategies: This category
presents the patterns and strategies related to the management of multi-cloud
environment, container management, edge computing, IoT strategies, and
resource management. For example, Multi-Cloud Management Strategies include
various approaches for managing multiple cloud environments. Multi-Cloud
Computing Strategy and Hybrid Cloud Architecture are found leading strategies
in this category. Container Management Strategies involve the management of
containers in multi-cloud environment, with the Linux Container (LXC) project
and Container Engine as the primary patterns identified. Edge Computing and
IoT Strategies pertain to the use of edge computing and IoT in multi-cloud
environment. Mobile Edge Computing (MEC) and Fog Computing are leading
strategies for container management in multi-cloud environment. Resource
Management Strategies involve methods for managing resources in the multi-
cloud with the help of Kubernetes and Minikube.

3\. Cloud Security and Resilience Strategies: In this category, the emphasis
is on strategies for ensuring the security and resilience of software systems
in multi-cloud environment. Security and Resiliency Patterns are strategies to
ensure the system is secure and can recover from failures. The two most
prominent patterns are the security-by-design approach and encryption of
files. Fault-tolerance Strategies are methods used to ensure that systems can
continue to function even when there are failures. Fault-tolerance with
Redundant Engines and Multi-cloud Systems Fault-Tolerant Workflow are the
leading patterns.

4\. Cloud Migration Strategies: This category involves patterns and strategies
for moving applications and data between cloud environments and optimizing the
migration process. Image-based Migration, for example, is an efficient way to
migrate applications because it packages the applications along with their
dependencies, configurations, and runtime environment. Migration to a
virtualized container, such as Docker, is increasingly popular because it
guarantees application portability across various environments and scalability
on different cloud platforms.

Table 9. Patterns and strategies for container-based applications a multi-
cloud environment

Categories | Subcategories |  Patterns |  ID
---|---|---|---
Cloud Architectural Patterns and Models | Architecture Patterns |  Microservice Architecture |  S02, S09, S16, S77, S80, S24, S31, S33, S68
Service-Oriented Architecture (SOA) |  S51, S52, S86
Multitier Architecture, Layered Architecture |  S02, S04, S54
Client-Server Architecture |  S03, S54
Component-based Architecture |  S22
Event-driven Architecture |  S44
Multi-agent Architecture |  S74, S48
Master-worker Nodes Architecture |  S33
Netflix Zuul-based Seeker Component |  S17
Multi-tenant Cloud Service Architecture |  S17
Stateful Engine Architecture |  S17
Communication and Networking Patterns |  Service Mesh |  S05
Service Chaining (SC) |  S50
Multi-Cloud APIs |  S71
Third-Party APIs |  S71
Service Request Broker |  S07
Publish and Subscribe Communication Protocol |  S45
Synchronous Network Communication Protocols |  S01
Asynchronous Network Communication Protocols |  S01
Network Function Virtualization (NFV) |  S50
Service Discovery |  S76
Deployment Pattern |  Blue-Green deployment |  S27, S30
Object store service |  S76
Multi-cloud Deployment Model |  S10
Distributed Deployment Model |  S10
Accelerated Particle Swarm Optimization for Deployment |  S16
Service Models |  Platform-as-a-Service (PaaS) |  S24, S41
Software as a Service |  S79
Infrastructure-as-a-Service (IaaS) |  S24
Function-as-a-Service (FaaS) |  S41
Cloud Management and Resource Allocation Strategies |  Multi-Cloud Management Strategies |  Multi-cloud computing strategy |  S78, S81
Hybrid cloud architecture |  S78
Hybrid/Multi-cloud Approach |  S05, S83
Introducing multi-cloud Middleware |  S52, S53
MAPE-K control loop |  S46, S48
Connecting to multiple cloud service providers |  S76
Multi-cloud load balancing |  S23, S76
Multi-Cloud BP provisioning |  S84
Cross-level orchestration of cloud services |  S84
Cross-level monitoring and adaptation of BPs |  S84
Container Management Strategies |  Linux Container (LXC) project |  S24
Container engine |  S24
Docker Container Images |  S24
Lightweight virtualization |  S24
Portable application packaging |  S24
One-container-per-app approach |  S24
Data volumes and data volume containers |  S24
Edge Computing and IoT Strategies |  Mobile Edge Computing (MEC) |  S22, S70, S43, S54
Fog computing |  S32
Edge Services |  S35
Connecting IoT Edge Devices to the Cluster |  S05
Cloud of Things (CoT) |  S05
Resource Management Strategies |  Resource allocation based on different SLA levels |  S23
Container Orchestration, Kubernetes, Minikube |  S24, S29, S33
Cloud Security and Resilience Strategies |  Security and Resiliency Patterns |  Security-by-Design approach |  S51
Encryption of files |  S71
Standardized APIs for file transfer |  S71
Centralized/External ACM Service |  S71
Distributed ACM Service |  S71
Dynamic Switching between Authentication Methods |  S71
ACM based on Signed URLs |  S71
Attribute-Based Encryption and Signature |  S75
Secure data sharing |  S75
Data access control |  S75
Local encryption and signing |  S75
Authorized Tokens |  S75
Byzantine quorum protocol |  S75
Fault-tolerance Strategies |  Fault-tolerance with Redundant Engines |  S17
Multi-cloud systems fault-tolerant workflow |  S67
RAFT Consensus Algorithm |  S13
Cloud Migration Strategies |  |  Service-oriented Migration |  S15
Application-centric Migration |  S15
Image-based Migration |  S15
Migration to a Virtualized Container |  S15

Takeaway 9: Identified a diverse range of architectural patterns and
strategies for container-based applications in multi-cloud environment. MSA
and SOA are prevalent, while a range of communication and networking patterns
such as Service Mesh and Service Chaining are frequently reported. Deployment
of cloud native applications often employs patterns like Blue-Green Deployment
and Object Store Service. Takeaway 10: Several studies are referenced to
highlight the usage of Linux Container (LXC) project, Docker Container Images,
and Lightweight Virtualization.

###  3.4. Quality Attributes and Tactics (RQ3)

Table 10 provides a comprehensive overview of the quality attributes and
related tactics for containerized applications in multi-cloud environment. It
is based on data extracted from 86 selected studies, which are organized
following the ISO 25010 standard. This standard defines high-level QAs along
with their sub-characteristics. Table 10 lists nine QAs, namely Performance
(Efficiency), Security, Compatibility, Scalability, Reliability, Portability,
Flexibility, Maintainability, and Usability. Each QA has its related terms,
associated study numbers, and corresponding tactics to achieve them.

In Table 10, five tactics are provided for each quality attribute, due to the
limited space. The complete set of 195 identified tactics is available in the
provided dataset (Waseem et al., 2024), presenting a comprehensive map of
tactics with QAs. However, it is important to note that these tactics are
scattered across different studies, and only a few studies explicitly
mentioned these tactics for specific QA.

Below we briefly discuss how each QA can be achieved by applying the mentioned
tactics.

Performance (Efficiency): The quality attribute of performance (or efficiency)
in containerized applications can be achieved by employing tactics such as
efficient resource optimization and allocation, use of ML techniques for
performance optimization, utilization of container technology, location-aware
service brokering, and performance-oriented Service Level Agreements (SLA).
For instance, optimizing resources ensures that the system uses the minimum
possible resources while delivering the required output. Using ML techniques
for performance optimization can enhance system performance by learning and
adjusting the operational parameters. Similarly, utilizing container
technology can help maintain optimal performance by encapsulating the
application and its environment.

Security: Security can be enhanced using tactics like encryption and strong
authentication mechanisms, implementation of firewalls, VPNs, or SDNs for
network security, utilization of ML techniques to detect and prevent security
threats and attacks, deployment of applications on diverse cloud providers,
and consideration of users’ security specifications. For example, encryption
provides a secure way of transmitting data, and ML can detect unusual behavior
that could signal a threat or attack.

Compatibility: The tactics that can be employed to achieve compatibility
include implementation of lightweight communication protocols and modes,
standardization of interfaces for seamless integration, componentization and
modular design for interoperability, prototyping and exploration of
interoperability approaches for multi-cloud deployment, and interoperability
standardization and federation between clouds. Standardizing interfaces can
ensure that different software components can interact with each other
seamlessly, enhancing compatibility.

Scalability: Scalability can be achieved through tactics like combining
containerization and microservices for enhanced scalability, implementation of
elastic resource allocation, enabling migration between multi-cloud services,
combining container-based deployment with runtime monitoring and optimization,
and leveraging orchestration, federated networks, and geographic placement.
Elastic resource allocation ensures that the system can seamlessly scale up or
down in response to changing demands.

Reliability: Tactics such as deploying redundant engines for fault tolerance,
distributing resources and replicating applications for improved response
time, enforcing redundancy and distributed services for availability,
deploying parallel search and multi-cloud distribution, and building fault-
tolerant systems can enhance system reliability.

Portability: Portability can be achieved using tactics such as virtual
machine-based packaging, Docker image-based packaging, cloud-portable
containerization, dynamic cross-level adaptation and provisioning, and
adaptive rule-based system modification. These tactics ensure that an
application can be easily transferred from one computing environment to
another.

Flexibility: Flexibility can be improved through tactics like service-oriented
architecture, on-demand dynamic allocation of resources across different cloud
platforms, containers, container orchestration, model-driven development, risk
analysis, and cross-cloud service orchestration. Using multi-cloud
environment, for instance, offers the flexibility to choose services from
different providers as per specific needs.

Maintainability: Tactics like microservices for easier maintenance,
Infrastructure Provisioning as Code (IaC), containerization and image
management, configuration management and templating, version control and
change management, and continuous integration and deployment automation can
enhance maintainability. For example, IaC allows developers to manage
infrastructure more efficiently and minimize human errors, thus increasing
maintainability.

Usability: Usability can be enhanced by employing tactics like a feedback-loop
controller for multi-cloud infrastructure, container orchestration (e.g.,
Kubernetes), user-centric interface design, responsive and adaptive user
experience, accessible design and compliance, and error handling and feedback
mechanisms. For instance, a feedback-loop controller is effective in a multi-
cloud context as it dynamically adjusts resources across different cloud
platforms based on user interaction, ensuring a seamless and responsive user
experience tailored to the multi-cloud setup.

Takeaway 11: Utilization of machine learning, container technology, and
location-aware service brokering are found to be essential for enhancing
performance and efficiency of containerized applications in multi-cloud
environment. Takeaway 12: To maintain security in multi-cloud deployments, a
combination of encryption, network security protocols, machine learning, and
user-specified security measures are commonly used. Also, deployment across
different cloud providers is encouraged. Takeaway 13: In the context of a
multi-cloud environment, compatibility is ensured through the standardization
of interfaces, componentization, and interoperability approaches. Special
emphasis is placed on lightweight communication protocols and interoperability
solutions to facilitate seamless integration across multiple cloud platforms.

Table 10. Quality attributes and tactics for containerized applications in
multi-cloud environment

Quality Attribute | Related Terms | ID | Tactics
---|---|---|---
Performance (Efficiency) |  Time behavior, Resource utilization, Capacity, Throughput, Response time |  S01, S02, S03, S06, S09, S14, S15, S16, S20, S21, S23, S25, S27, S29, S34, S35, S36, S37, S38, S39, S41, S43, S44, S47, S49, S52, S54, S57, S58, S59, S60, S62, S64, S65, S66 | Efficient resource optimization and allocation
Utilizing machine learning techniques for performance optimization
Container technology utilization
Location-aware service brokering
Performance-oriented Service Level Agreements (SLA)
Security |  Confidentiality, Authenticity, Access control, Authorization, Privacy, Trustworthiness, Integrity |  S01,S05, S06, S09, S15, S23, S24, S25, S29, S34, S37, S38, S40, S44, S47, S52, S54, S55, S57, S59, S61, S64, S65, S66, S69, S71, S72, S73, S77, S86 | Encryption and strong authentication mechanisms
Implementation of firewalls, VPNs, or SDNs for network security
| Utilizing machine learning techniques to detect and
---
prevent security threats and attacks
Deployment of cloud applications on diverse cloud providers
| Consideration of users’ security specifications and
---
addressing CSP incompatibilities
Compatibility |  Interoperability, Co-existence |  S15, S16, S21, S24, S27, S42,S44, S45, S49, S63, S64, S71, S77, S80, S82 | Implementing lightweight communication protocols and modes
Standardization of interfaces for seamless integration
Componentization and modular design for interoperability
| Prototyping and exploring interoperability approaches for
---
multi-cloud deployment
Interoperability standardization and federation between clouds
Scalability |  Capacity, Extensibility, Elasticity, Throughput, Responsiveness |  S01, S08, S09, S14, S15, S16, S17, S18,S19, S21, S22, S24, S25, S27, S29, S31, S35, S41, S42, S44, S45, S54, S57, S62, S66, S69, S72, S74, S76, S77, S82, S84 |  | Combining Containerization and Microservices for enhanced scalability
---
Implementing Elastic Resource Allocation
Enabling Migration between Multi-Cloud Services
| Combining Container-based Deployment with Run-time Monitoring
---
and Optimization
| Leveraging Orchestration, Federated Network,and Geographic Placement
---
Reliability |  Maturity, Availability, Fault tolerance, Recoverability |  S01, S02, S04, S10, S13, S17,S19, S20, S22, S25, S27, S32, S33, S35, S37, S44, S47, S51,S53, S54, S55, S56, S57, S60, S64, S65, S66, S67, S71, S73, S74, S75, S82 | Deploying Redundant Engines for Fault-Tolerance
| Distributing Resources and Replicating Applications for Improved
---
Response Time
Enforcing Redundancy and Distributed Service for Availability
Deploying Parallel Search and Multi-Cloud Distribution
Building Fault-Tolerant Systems
Portability |  Installability, Replaceability, Adaptability | S18, S24, S25, S31, S62, S77 | Virtual Machine-Based Packaging
Docker Image-Based Packaging
Cloud-Portable Containerization
| Dynamic Cross-Level Adaptation and Provisioning
---
Adaptive Rule-Based System Modification
Flexibility |  Functional completeness, Functional correctness, Functional appropriateness |  S16, S21, S27, S31, S41, S61, S60, S70, S76, S84 | Service-oriented Architecture
Dynamic allocation of resources across different cloud platforms based on
demand
| Container orchestration
---
Model-Driven Development, Risk Analysis
Cross-cloud service orchestration,
Maintainability |  Modularity, Reusability, Analyzability, Modfiabality, Testability | S08, S17, S25, S27, S62, S74, S82 | Microservices, easier maintenance
Infrastructure Provisioning as Code (IaC)
Containerization and Image Management
Configuration Management and Templating
Version Control and Change Management
| Continuous Integration and Deployment Automation
---
Usability |  Appropriateness recognizability, Learnability, Operability , User interface aesthetics, Accessibility | S09, S17, S27, S27, S37, S74 | Feedback-loop controller for multi-cloud Infrastructure
Container Orchestration (e.g., Kubernetes)
User-Centric Interface Design
Responsive and Adaptive User Experience
Accessible Design and Compliance
Error Handling and Feedback Mechanisms

###  3.5. Security Challenges and Solution Framework (RQ4 and RQ5)

In an era of rapid technological advancement, ensuring data security is of
paramount importance. In this section, we propose a framework designed to
address the security challenges in containerized and multi-cloud environment.
This proposed framework, composed of seven categories of security challenges
and solutions illustrated in Table LABEL:tab:Security-challenges-solutions, is
derived through a comprehensive review of existing literature. It covers a
broad spectrum of categories, each capturing specific security challenges and
their corresponding solutions. Each category is briefly described below:

Data Security in container-based applications category covers a wide range of
challenges and solutions, including data protection, database security, data
compliance, and secure data transfer. Our identified solutions revolve around
the use of encryption and secure container orchestration, complying with
regulatory requirements such as GDPR, HIPAA, and secure data management
strategies. Furthermore, identified solutions emphasizes the role of security
measures such as access control and secure transmission protocols in ensuring
data privacy, service continuity, fault tolerance, and the secure transfer of
data between different cloud environments.

Access and Communication Control category emphasizes maintaining secure multi-
cloud connectivity and managing user access control. The identified solutions
involve implementing encryption and authentication mechanisms, using secure
communication protocols, and executing operations in a provider-agnostic
manner to maintain security.

Container Security category addresses issues like co-resident security,
container isolation, and providing host security. Implementing secure
placement strategies for containers across multi-cloud platforms, utilizing
container isolation techniques, and enhancing manual defenses to mitigate
risks of compromised containers are some of the identified solutions in this
category.

Infrastructure and Deployment Management Security category emphasizes the
significance of infrastructure and deployment management security. It
encompasses aspects such as secure data deployment, infrastructure
orchestration, storage security, and migration security. Essential solutions
encompass the application of encryption techniques, implementation of access
control mechanisms, adoption of standardized and secure infrastructure
orchestration practices, and ensuring the secure migration of containerized
applications across multi-cloud environment.

Security Monitoring and Breach Prevention category addresses a range of
challenges and solutions related to enhancing security. This includes the
implementation of automated security assessments, the use of vulnerability
scanners, and the establishment of automated bootstrapping mechanisms for
Virtual Machines (VMs) that incorporate security configurations and policies.
Additionally, it involves deploying Security Information and Event Management
(SIEM) systems, as well as Intrusion Detection System/Intrusion Prevention
System (IDS/IPS) tools. The category also covers the application of log
analysis solutions to proactively detect and prevent security breaches.

Trust and Compliance Management category identified solutions encompass on
implementation of robust security controls, adherence to industry best
practices and standards, and establishment of trust-based coalitions to foster
secure community formation.

Placement Strategy: It recommends devising a security model for services
hosted on volunteering fogs within multi-cloud environment, including
formulation of secure communication protocols and implementation of access
control mechanisms.

Takeaway 24: Data security in multi-cloud environment can be ensured through
robust encryption and access control mechanisms, adherence to compliance
regulations, and by securing data transfers across different cloud
environments. Takeaway 15: To achieve optimal application security in multi-
cloud deployments, organizations should focus on access control, secure
communication between systems, and container isolation. It is essential that
security measures extend from the user level to the host and infrastructure
levels of operations. Takeaway 16 Organizations can enhance security in a
multi-cloud environment by implementing continuous monitoring, proactive
vulnerability management, and dynamic security adaptation. Maintaining trust
and compliance through standardization, community formation, and strategic
service placement is also a key aspect of this enhancement.

Table 11. Security challenge and solution framework for containerized applications in multi-cloud environment ID |  Challenge |  Challenge Description |  Solution |  Solution Description
---|---|---|---|---
Category 1: Data Security
S17 |  Data Protection |  Protecting data against threats in containerized multi-cloud setups. |  Multi-Cloud Data Protection through Secure Containers |  Deploys sensitive data in encrypted, isolated containers, using secure orchestration tools like Docker and Kubernetes.
S29 |  Database Security |  Securing databases within containerized multi-cloud environments. |  Enhanced multi-cloud Database Security via Containerization |  Secures databases by implementing encryption (e.g., Transparent Data Encryption) and robust access controls (e.g., RBAC, ABAC).
S37 |  Data Compliance |  Ensuring data processing and storage compliance across multi-cloud platforms. |  Multi-cloud Legal Compliance and Data Protection |  Maintains privacy and service quality by adhering to regulations (e.g., GDPR, HIPAA) and employing data protection strategies like encryption and audits.
S38 |  Data Security |  Adhering to data security standards in multi-cloud containerized systems. |  Secure CSP Interoperability and Selection in multi-cloud |  Tackles CSP compatibility issues and enhances security by implementing data handling practices and selecting compliant CSPs.
S53 |  Data Ownership and Privacy |  Tackles data ownership, privacy, and availability issues in multi-cloud setups. |  Multi-cloud Security via NoMISHAP Service Abstraction |  Uses NoMISHAP for service abstraction, enhancing privacy with encryption and pseudonymization, and ensuring availability with fault tolerance strategies like backups and failovers.
S58 |  Data Transfer Security |  Secures data during transfers in multi-cloud environments. |  Secure multi-cloud Integration via OpenStack Standardization |  Standardizes OpenStack APIs to prevent vendor lock-in, ensuring safe data transfers with encryption and secure protocols.
S85 |  Cloud Data Management |  Manages cloud data security, privacy, and compliance. |  Multi-cloud Data Security via DRA Framework |  Implements the DRA framework for comprehensive data management, ensuring encryption, access control, and compliance with standards.
Category 2: Access and Communication Control
S05 |  Connectivity and Access Control |  Ensures secure multi-cloud connectivity and robust user access control. |  Multi-cloud Containerized Application Security |  Implements TLS, JWT, OAuth 2.0 for secure authentication and authorization in multi-cloud setups.
S43 |  Secure Communication |  Secures communications across multi-cloud and MEC platforms. |  Middleware Secure Deployment for multi-cloud |  Uses middleware for secure cloud app deployment, with SSL/TLS and access control for secure communication.
S51 |  Provider-Agnostic Operations |  Facilitates secure, provider-agnostic operations in multi-cloud environments. |  Aneka-Based Security and Performance for Containerized multi-cloud Applications |  Employs Aneka for security and performance, offering transparent access and dynamic provider selection, while ensuring security via secure channels and monitoring.
Category 3: Container Security
S06 |  Co-resident Security |  Addresses security risks from container virtualization in multi-cloud environments. |  Secure Placement Strategy via Deep RL |  Uses deep reinforcement learning, like Kubernetes Cluster Autoscaler with RL, for secure container placement across multi-cloud platforms.
S25 |  Container Isolation |  Aims to prevent information leaks and power attacks through effective container isolation. |  Power and Leakage Management for multi-cloud Security |  Manages security via container isolation (e.g., Docker isolation) and Linux kernel-level resource isolation (e.g., SELinux).
S40 |  Container and Host Security |  Enhances security against compromised containers and strengthens defenses. |  Host-Container Cooperation for multi-cloud Defense |  Develops container-host defense strategies, utilizing tools like Falco, Sysdig, and practices for host hardening.
Category 4: Infrastructure and Deployment Management Security
S20 |  Data Deployment |  Secures private data deployment in multi-cloud environments. |  Secure Private Data Deployment Strategy |  Utilizes encryption, RBAC, and secure protocols (e.g., AES-256, SFTP, HTTPS) for safeguarding data deployment.
S44 |  Infrastructure Orchestration |  Orchestrates secure infrastructure for containerized applications. |  Standard Interface Implementation for Serverless Security |  Uses standard interfaces (e.g., OpenStack APIs) for secure, interoperable serverless application deployment across clouds.
S47 |  Storage Security |  Enhances security for multi-cloud storage in containerized setups. |  Multi-cloud Storage Security with MSSF |  Employs the MSSF for secure storage management, implementing RBAC, data encryption, and regular audits.
S80 |  Migration Security |  Ensures the secure migration of applications in multi-cloud environments. |  Infrastructure-Aware and Agnostic Secure Migration |  Leverages strategies and tools for safe migration, including image signing and data encryption, adhering to best practices like least privilege.
Category 5: Security Monitoring and Breach Prevention
S30 |  System Vulnerabilities |  Addresses vulnerabilities and patching delays in multi-cloud environments. |  Automated Security Evaluation and Bootstrapping |  Utilizes tools like OpenVAS and Nessus for security assessments, and automates VM setup with Terraform, Chef, Ansible, or Puppet.
S45 |  Breach Detection and Prevention |  Boosts breach detection and prevention in multi-cloud infrastructures. |  Metric Filtering and Fault Recovery for multi-cloud Security |  Enhances security with metric filtering, fault recovery, SIEM, IDS/IPS, and log analysis for proactive breach management.
Category 6: Trust and Compliance Management
S52 |  Application Security and Compliance |  Ensures security and compliance for applications across multi-clouds. |  Risk Analysis and Security SLAs with MUSA Framework |  Uses MUSA for risk analysis and security SLA management, incorporating secure coding, vulnerability management, and compliance audits.
S49 |  Data Security and Standardization |  Standardizes data security and cloud interfaces in multi-cloud environments. |  OpenStack API-based multi-cloud Security and Interoperability |  Utilizes OpenStack API for secure, standardized cloud interoperability, applying encryption, access control, and CIS benchmarks.
S57 |  Network Security |  Secures networks and trust in multi-cloud environments. |  Inter-cloud Communication Protection with SFC |  Employs Service Function Chaining (SFC) for secure inter-cloud communications, using encryption, integrity checks, and protocols like IPSec, WireGuard.
S83 |  Trust and Community Formation |  Builds secure communities and trust in multi-cloud settings. |  Trust-based Hedonic Coalitions for multi-cloud Security |  Forms trust-based coalitions for community security, leveraging IAM and trust frameworks for collaboration in multi-clouds.
S84 |  Orchestration and Adaptation |  Manages orchestration and adaptation for security and performance. |  Multi-cloud Service Orchestration and Cross-Level Adaptation |  Implements multi-cloud orchestration with tools like Kubernetes, Istio, and Prometheus for secure, adaptive application management.
Category 7: Placement Strategy
S33 |  Placement Strategy |  Secures service deployment on volunteer fogs in multi-cloud setups. |  Security Model for Volunteering Fog Services |  Creates a security model for fog-deployed services, incorporating SSL/TLS for communication and certificate or token-based access control.

###  3.6. Automation Challenges and Solution Framework (RQ4 and RQ5)

Table LABEL:tab:Automation-solutions presents an overview of the automation
challenges and solution framework for containerized applications in multi-
cloud environment. The figure consists of eight categories, each focusing on
different aspects of automation challenges and their solutions in a multi-
cloud environment.

Multi-Cloud Automation: This category reports challenges and solutions related
to automating processes for or combining and administering various cloud
computing environments. It addresses challenges such as multi-cloud
integration and provisioning, and provides solutions such as proxies for
environment interface and model-driven engineering techniques.

Automation in Deployment and Scaling: This category reports challenges and
solutions related to automating the deployment and scaling processes of
containerized applications across multi-cloud platforms. It tackles challenges
like container placement automation and application runtime switch automation.
Solutions include intelligent container placement algorithms and service
drivers for automated runtime switch.

Resource Management Automation: This category reports challenges and solutions
related to automating the management of resources in multi-cloud environment.
It addresses challenges such as resource overbooking management and
containerized applications management. Solutions include ML-based resource
overbooking detection and custom Kubernetes schedulers.

Data and Application Migration Automation: This category reports challenges
and solutions related to automating the migration of data and applications in
multi-cloud environment. It covers challenges such as data integrity during
migration and legacy web applications migration. Solutions encompass the
implementation of self-sufficient management cycles for streamlined
deployment, alongside the utilization of adaptable architectural patterns
designed for the modernization of legacy applications within a containerized
multi-cloud environment.

Testing and Benchmarking Automation: This category reports challenges and
solutions related to the automation of testing and benchmarking processes for
containerized applications in multi-cloud environment. It addresses challenges
like user-oriented testing automation and benchmarking automation. Solutions
include container-based lightweight virtualization for testing and smart
Docker benchmarking orchestrators.

Standardization and Interoperability Challenges: This category reports
challenges and solutions related to standardization and interoperability in
the automation of containerized applications across multiple cloud
environments. It covers challenges like standardization in automation
processes and cloud provider selection. Solutions include service mesh
networks and AI-driven cloud provider selection tools.

Application and Service Management: This category reports challenges and
solutions related to the automation of application and service management
processes in containerized multi-cloud environment. It addresses challenges
such as 5G ecosystem component selection and end-to-end tail latency
management. Solutions include open-source software components for customizable
orchestration and modeling approaches for performance evaluation.

Runtime and Service Discovery: This category deals with challenges and
solutions related to runtime management and service discovery in multi-cloud
environment, which involves managing the complexity of network control and
administration within containerized applications across multiple cloud
environments.

Table 12. Automation challenge and solution framework for containerized applications in multi-cloud environment ID |  Challenge |  Challenge Description |  Solution |  Solution Description
---|---|---|---|---
Category 1: Multi-Cloud Automation
S10 |  Multi-Cloud Integration |  Automating processes to manage the complexities of integrating multiple cloud environments |  Proxies for Environment Interface |  Proxies for individual environment interface, unified view of resources, shielding from underlying complexity in multi-cloud environment
S20 |  Multi-Cloud Provisioning and Adaptation |  Automating provisioning, deployment, and dynamic adaptation of multi-cloud applications for efficiency and timely response |  Model-driven Engineering Techniques |  Model-driven engineering techniques for dynamic provisioning, deployment, and adaptation in multi-cloud environment
S33 |  Fog Creation in Multi-Cloud |  Automating the dynamic creation of nearby fog in multi-cloud environment |  Dynamic On-demand Fog Computing |  Dynamic on-demand fog computing architecture in multi-cloud environment
S47 |  Intelligent User-Friendly Automation |  Addressing the lack of intelligent, user-friendly automation in delivery platforms |  MSSF |  MSSF (Multi-cloud Storage Selection Framework) for automatic selection of multi-cloud redundancy storage based on user needs and security rules
S44 |  Multi-Stack Orchestration |  Automating the specification, coordination, and orchestration of different infrastructure stacks and cloud services |  TOSCA-based Serverless Deployment |  Topology and Orchestration Specification for Cloud Applications (TOSCA) for automated deployment of multi-cloud serverless applications
S34 |  Secure Multi-Cloud App |  Creation Providing tools and mechanisms for non-security expert developers to automate secure multi-cloud app creation |  MUSA Framework |  MUSA framework for application modeling, risk analysis, cloud service selection, composite SLA generation in multi-cloud environment
S37 |  Unified Abstraction for Multi-Cloud Deployment |  Implementing a common abstraction for products offered in automated multi-cloud deployments |  Application-level Resource Managers |  Application-level resource managers, native multi-cloud support for enterprise applications in multi-cloud environment
S52 |  Multi-Cloud Deployment Integration |  Addressing integration complexity in multi-cloud deployment automation |  MUSA Deployer Tool |  MUSA Deployer tool for automation of deployment process, deployment plan, and security compliance in multi-cloud environment
S57 |  Deployment Across Federated Clouds |  Automating deployment across federated cloud providers |  Orchestrated Deployment Subsystem |  Orchestrated deployment subsystem, Beacon Broker, and federated environments for coordinated application deployment in multi-cloud environment
S58 |  Hybrid/Multi-Cloud Setup |  Automating the setup of hybrid/multi-cloud environment to address security, vendor lock-in, and interoperability issues |  Interoperability Standardization |  Interoperability standardization and federation of clouds in multi-cloud environment
S80 |  Multi-cloud Interoperability |  Facilitating multi-cloud interoperability, application topology definition/composition, and elastic runtime adaptation for containerized services |  Multi-cloud Interoperability |  Multi-cloud interoperability, application topology definition/composition, and elastic runtime adaption in multi-cloud environment
S81 |  Multi-cloud Brokerage System |  Overcoming API heterogeneity and promoting semantic interoperability for a brokerage system in multi-cloud container management. |  Managing Large Number of Cloud Services |  Interoperability and vendor lock-in caused by the diversity of cloud solutions, complexity of multi-cloud systems administration
S84 |  Multi-cloud Orchestration |  Implementing multi-cloud business process provisioning and cross-level orchestration for cloud services in a containerized setting |  Manual Migration to the Cloud |  Manual migration of business processes to the Cloud, limited multi-cloud and cross-cloud orchestration
S85 |  DevOps for Multi-cloud |  Building a DevOps reference architecture with a CI broker for automation within a containerized multi-cloud environment |  Agile Software Development Automation |  Automation of agile software development, application deployment using CI-broker, automated faster application delivery, automated and integrated testing of the application in multi-cloud environment
Category 2: Automation in Deployment and Scaling
S27 |  Multi-Cloud Deployment and Monitoring |  Tackling challenges in automating deployment and monitoring of applications across multi-cloud platforms |  Multi-cloud Strategy |  Multi-cloud strategy for risk distribution and provider dependency reduction in multi-cloud environment
S06 |  Container Placement Automation |  Addressing the complexity of automating container placement in multi-cloud environment |  Intelligent Container Placement |  Develop a machine learning-based optimization algorithm for intelligent container placement across multiple cloud environments, focusing on load balancing and minimizing network latency.
S07 |  Application Runtime Switch Automation |  Enabling automated application runtime switch among various cloud services |  Service Driver |  Service driver, multi-cloud application service access model, runtime service update, development framework in multi-cloud environment
S15 |  Fine-Grained Automation at Component Level |  Implementing autonomic fine-grained solutions at the service component level for better automation |  Mobility Service and Replication |  Mobility service, migration and replication of critical software components, autonomic fine-grained service migration/replication in multi-cloud environment
S28 |  DevOps Automation |  Ensuring effective automation of software development and operations management |  DevOps Development Model |  DevOps development model, continuous development, automated tools and management systems, autonomic management system, performance models, systematic decision-making in multi-cloud environment
S29 |  Service Deployment Automation |  Addressing manual deployment issues by automating service deployment |  Automation Scripts |  Automation scripts, automated deployment, and error reduction in multi-cloud environment
S30 |  Deployment Time Security Automation |  Implementing security automation during deployment time |  Automated Security Measures |  Automated security measures, security evaluation and bootstrap, vulnerability scanning, firewall deployment, Chef integration in multi-cloud environment
S31 |  Distributed Simulation Automation |  Automating distributed simulation environments in cloud infrastructures |  DevOps Methodologies |  DevOps methodologies, automated developer operations, rapid microservices deployment, containerization paradigm, DEVS Simulation Farm, Docker technology, SOA Microservices paradigm, DEVSML Stack in multi-cloud environment
S32 |  Software Deployment Speed Improvement |  Addressing slow software deployment issues by implementing effective automation, enhancing Quality of Experience (QoE) |  Reorganized Container Images |  Reorganized container images, modified Docker deployment process, asynchronous image layer retrieval, profiled container execution, overlay file system in multi-cloud environment
S36 |  Container Startup Automation |  Identifying essential files for container startup automation |  Profiled Container Execution |  Profiled container execution, essential file determination, special image layer creation in multi-cloud environment
S38 |  Dynamic Scaling of Resources |  Automating frequent deployment and dynamic scaling of resources in multi-cloud environment for improved efficiency and performance |  CSP Automation Capability |  CSP automation capability for user agility in containerization of multi-cloud environment
S24 |  Container Deployment and Scaling |  Implementing automated container deployment and scaling based on resource demand across multiple cloud platforms |  Kubernetes |  Kubernetes for automated container deployment and multi-cloud scaling in multi-cloud environment
S48 |  Elastic Resource Provisioning |  Implementing elastic resource provisioning for fluctuating business workloads in a containerized multi-cloud environment through automation |  Elastic Resource Provisioning |  Design a predictive analytics system coupled with automation tools for real-time, elastic resource provisioning, addressing fluctuating business workloads in a containerized multi-cloud environment
S50 |  VF Placement and Service Deployment |  Automating optimal Virtual Function (VF) placement and dynamic service deployment in multi-cloud environment |  VF Placement Optimization |  Optimization model for virtual function (VF) placement and dynamic VF allocation in multi-cloud environment
S68 |  Container and VM Deployment |  Orchestrating and automating the deployment of containers and VMs in multi-cloud environment |  Multi-Cloud Orchestration |  Develop a comprehensive multi-cloud management platform for streamlining orchestration and automation of containers and virtual machine deployment across different cloud environments
Category 3: Resource Management Automation
S04 |  Resource Overbooking Management |  Automating the management of resource overbooking in service containers across multi-cloud platforms |  ML-based Resource Overbooking Detection |  Machine learning-based resource overbooking detection in multi-cloud environment
S21 |  VCluster Management |  Automating cluster creation, deployment, and management for multiple Virtual Clusters (Vclusters) across diverse application frameworks |  ClaaS Service Model |  ClaaS service model, virtualized cluster environment, easy deployment and management in multi-cloud environment
S39 |  Containerized Applications Management |  Automating deployment, scheduling, and management of containerized applications in multi-user and multi-cloud environment |  Custom Kubernetes Scheduler |  Custom Kubernetes scheduler (label-affinity-scheduler) for container orchestration in multi-cloud environment
S41 |  VM Types Selection in Data Centers |  Addressing the large and complex search space of different VM types in various data centers through automation |  GA-based Algorithm |  GA-based algorithm, domain-tailored service clustering method for complexity reduction in multi-cloud environment
S42 |  Resource Commissioning and De-Commissioning |  Speeding up resource commissioning and de-commissioning through automation for timely system scalability |  MicroCloud Architecture |  MicroCloud architecture, fine-grained resource allocation, TOSCA library, meta-workflow, coordinated adaptation workflow in multi-cloud environment
S43 |  Cloud Federation Automation |  Implementing automation support for federation or inter-platform portability in multi-cloud environment |  Middleware Platform |  Middleware platform for security, cost, and performance optimization, data locality, privacy needs in multi-cloud and MEC environments
S45 |  Resource Provisioning and Elasticity Detection |  Automating resource provisioning for cloud services, monitoring runtime configurations, and detecting elasticity actions |  JCatascopia |  JCatascopia, a fully-automated and modular cloud monitoring framework with elasticity actions, metric rule mechanism, and low-level monitoring metrics in multi-cloud environment
S51 |  Resource Acquisition Automation |  Automating resource acquisition processes for applications to function without human intervention |  Aneka Platform |  Aneka platform for automatic acquisition and release of resources from multiple clouds in multi-cloud environment
S74 |  SLA Enforcement Automation |  Automating the testing and corrective actions for SLA enforcement in containerized multi-cloud environment to reduce expense and time consumption |  SLA Compliance Testing |  Manual testing and corrective actions for SLA compliance are expensive and time-consuming in multi-cloud environment
Category 4: Data and Application Migration Automation
S01 |  Data Integrity during Migration |  Safeguarding data integrity during the automation of migration processes in multi-cloud environment |  Autonomous Management Cycle |  Autonomous management cycle for efficient deployment in multi-cloud environment
S17 |  Legacy Web Applications Migration |  Facilitating the automated migration of legacy web applications to cloud services |  Reusable Architectural Pattern |  Reusable architectural pattern for containerization in multi-cloud environment
S69 |  Application Porting to Cloud |  Automating the porting of existing applications to cloud infrastructures |  CloudSME Simulation Platform |  CloudSME Simulation Platform for fast development and deployment of complex multi-cloud applications, commercial accounting, and billing
Category 5: Testing and Banchmakring Automation
S09 |  User-Oriented Testing Automation |  Accelerating user-oriented software testing in diverse and heterogeneous environments through automation S |  Container-based Lightweight Virtualization |  Container-based lightweight virtualization, just-in-time deployment, streamlined multicomponent prototyping in multi-cloud environment
S02 |  Benchmarking Automation |  Implementing an automated benchmarking process for cloud hosts from diverse providers in multi-cloud environment |  Smart Docker Benchmarking Orchestrator |  Smart Docker Benchmarking Orchestrator for containerization in multi-cloud environment
S13 |  CI for Dependable Automation |  Leveraging Computational Intelligence (CI) to enhance dependability in automated multi-cloud containerized environments |  Dependability Enhancement with CI |  Leverage Computational Intelligence (CI) to enhance dependability in automated multi-cloud containerized environments
Category 6: Standardization and Interoperability Challenges
S05 |  Standardization in Automation |  Implementing standardization in the automation processes for containerized applications across multiple cloud environments |  Service Mesh and Multi-cloud Federation |  Service mesh network and multi-cloud cluster federation for containerization in multi-cloud environment
S70 |  Cloud Provider Selection |  Developing optimal strategies for cloud provider and node configuration selection to balance cost and runtime in a multi-cloud, containerized environment |  Optimal Cloud Provider Selection |  Create an AI-driven decision-support tool that automatically selects the optimal cloud provider and node configuration, considering both cost-effectiveness and runtime efficiency in a multi-cloud, containerized environment.
S16 |  Unified Template for Microservices |  Addressing the lack of a unified template and specification for automating microservices deployment across multiple cloud platforms |  Tool Service Model |  Tool service model, invocation relationships, service function chains, objective function optimization, accelerated Particle Swarm Optimization, configurable objective parameters in multi-cloud environment
S64 |  Cloud Interoperability |  Addressing the interoperability issues between different clouds through automation |  multi-cloud Harmony |  Develop an AI-driven multi-cloud management platform to automate cross-cloud interoperability, ensuring seamless data transfer and communication between disparate cloud environments
S63 |  Cloud Applications Standardization |  Promoting the adoption of worldwide standards or interfaces for cloud applications and their management through automation |  Cloud Service Interoperability |  Cloud service providers and middleware for interoperability, standardized interfaces, and hybrid solutions in multi-cloud environment
S18 |  Package Creation and Scaling |  Automating the creation and scaling of packages using VM and Docker based on machine-readable definition files |  Package Automation with VM |  Docker Implement a scriptable automation framework that facilitates creation, scaling, and management of packages using VM and Docker based on machine-readable definition files
Category 7: Application and Service Management
S55 |  5G Ecosystem Component Selection |  Simplifying the complexity of 5G ecosystem component selection through automation |  Open-source Software Components |  Open-source software components for customizable container and virtual machine orchestration in contemporary deployments
S62 |  End-to-End Tail Latency Management |  Automating end-to-end tail latency management in a microservice architecture |  Performance Modeling Approach |  Modeling approach for performance evaluation of container-level and VM-level data in multi-cloud environment
S79 |  Distributed Application Complexity |  Addressing the increasing complexity of managing distributed applications in containerized multi-cloud environment |  Complexity of Distributed Applications |  Increasing complexity of distributed applications and operating multi-cloud management topologies involving CMPs
Category 8: Runtime and Service Discovery
S72 |  Network Complexity |  Managing the complexity of network control and administration within containerized applications across multiple cloud environments. |  Network Management Complexity |  Complexity of network management and control in multi-cloud environment
S83 |  Social Network Service Discovery |  Improving service discovery algorithms for tagging in online social networks within containerized multi-cloud environment |  Automated Discovery and Selection |  Automating discovery and selection of relevant and appropriate services for composition and QoS management in multi-cloud environment
S08 |  Complex Application Administration |  Handling growing application complexity due to new technologies and needs through automation |  Autonomic Computing Systems |  Autonomic Computing Systems with knowledge transfer for containerization in multi-cloud environment

Takeaway 23: In addressing the complex challenges of automating containerized
application management across multi-cloud environments, it is essential to
adopt holistic approaches. These approaches should encompass every aspect,
from multi-cloud integration to resource management, ensuring comprehensive
and effective automation solutions. Takeaway 24: AI and machine learning are
indispensable in overcoming automation challenges within multi-cloud settings.
Their role is critical in devising intelligent algorithms for container
placement, resource management, and cloud provider selection, thereby
enhancing the automation process’s efficiency and effectiveness. Takeaway 25:
Standardization and interoperability are the cornerstones for successful
automation in multi-cloud environments. They are key to ensuring seamless
integration and efficient management of containerized applications across
different cloud platforms, particularly in aspects like environment interface,
application and service management, and runtime control.

###  3.7. Deployment Challenges and Solution Framework (RQ4 and RQ5)

Table LABEL:tab:Deployment-solutions presents a comprehensive overview of the
challenges and corresponding solutions faced during the deployment of
containerized applications in multi-cloud environment. As organizations
increasingly adopt containerization and multi-cloud strategies to enhance
scalability, flexibility, and resource utilization, they encounter various
obstacles that require effective solutions. The figure consists of nine
categories, each representing a specific deployment challenge, along with the
corresponding challenge description and its solution. These categories
encompass a wide range of issues, including deployment complexity and
Orchestration, access and communication control, security and compliance
management, multi-cloud deployment coordination and Integration, monitoring
and scalability challenges, microservices architecture and containerization,
and network connectivity and hybrid cloud integration challenges.

Deployment Complexity and Orchestration: This category focuses on addressing
challenges and providing solutions related to the complexity and orchestration
of container deployment in multi-cloud environment. One of the significant
challenges faced in this context is Performance Testing Deployment. It
involves evaluating testing procedures and assessing performance metrics
during container deployment across multiple cloud environments. To overcome
this challenge, the corresponding solution is Standardized Benchmarking and
Automation. This solution entails adopting Standardized Benchmarking and
automatic systems to ensure consistent and reliable performance evaluation.
Another challenge is Deployment Validation, which requires conducting thorough
validation and artifact checks during container deployment in multi-cloud
environment. To effectively address this challenge, the suggested solution is
the implementation of a Testing Process Management System. This system
enhances and oversees the validation process for deployment, making sure that
all essential criteria are met before deployment proceeds. The challenges
outlined in this category can simplify and optimise container deployment in
complex multi-cloud environment, facilitating efficient and reliable
operations across different cloud service providers.

Access and Communication Control: This category emphasizes maintaining secure
multi-cloud connectivity and managing user access control. Challenges in this
category include ensuring secure access to resources and secure communication
between containers and across clouds. The proposed solutions involve
implementing encryption and authentication mechanisms, using secure
communication protocols, and executing operations in a provider-agnostic
manner to maintain security.

Security and Compliance Management in Multi-cloud Deployment: This category
focuses on security and compliance challenges faced during multi-cloud
container deployment. One challenge, Vendor Lock-in Prevention Deployment,
aims to prevent vendor lock-in and ensure modular and loosely-coupled
deployment in multi-cloud environment. The solution proposed is the Cloud
Modelling Framework, which introduces a framework for specifying provisioning
and deployment to enhance compatibility with existing ACSs and cloud
solutions. Another challenge in this category is Security Deployment in Multi-
container Environment, which addresses security and privacy concerns during
multi-container deployment on the same OS kernel in multi-cloud environment.
The solution is Cross-Container Isolation, which enforces cross-container
isolation to enhance security.

Multi-cloud Deployment Coordination and Integration: This category deals with
challenges related to coordination and integration during multi-cloud
deployment. One challenge, Multi-cloud PaaS Deployment, aims to overcome high
entry barriers for deploying a multi-cloud PaaS infrastructure. The proposed
solution is Lightweight Proxies for PaaS Integration, which utilizes
lightweight proxies to facilitate individual PaaS services integration.
Another challenge is IoT Application Deployment, which addresses challenges
related to deploying IoT applications in multi-cloud environment. The solution
is based on the DRA Framework and CI Broker for Multi-Cloud Deployment, which
leverages the DRA framework and CI broker component for efficient multi-cloud
deployment.

Monitoring and Scalability Challenges: This category focuses on challenges
related to monitoring, performance, and scalability during multi-cloud
container deployment. For instance, Legacy Code Migration Deployment deals
with handling legacy code migration, tenant engine separation, and composition
modeling during container deployment in multi-cloud environment. The proposed
solution is the Reusable Architectural Pattern with Docker and Cloudant, which
advocates for adopting a reusable architectural pattern with Docker containers
and Cloudant persistence layer.

Microservices Architecture and Containerization: This category addresses
challenges associated with microservices architecture and containerization in
multi-cloud environment. For example, Autonomic System Deployment aims to
implement an autonomic management system for deploying multi-layer, multi-tier
web architectures in multi-cloud environment. The solution proposed is the
Self-Tuning Performance Model and Autonomic Management, which introduces a
self-tuning performance model and autonomic management system.

Network Connectivity and Hybrid Cloud Integration: This category focuses on
network connectivity and hybrid cloud integration challenges during multi-
cloud deployment. One challenge, Inter-edge Bandwidth Deployment, aims to
address challenges related to interconnecting distributed localities and
provisioning inter-edge bandwidth.

Cloud Deployment Constraints and Challenges: This category deals with various
constraints and challenges encountered during cloud deployment. For instance,
ACS Compatibility Deployment aims to ensure compatibility with existing ACSs
and cloud solutions during container deployment in multi-cloud environment.
The solution proposed is the Scalable Multi-Cloud Deployment Framework, which
proposes a scalable deployment framework for multi-cloud applications.

Infrastructure Provisioning and Container Deployment Challenges: This category
addresses challenges related to infrastructure provisioning and container
deployment in multi-cloud environment. For example, Fog Computing Deployment
deals with fog computing hardware and container deployment time challenges.
The proposed solution is Reorganized Container Images and Docker Deployment,
which involves reorganizing container images and modifying the Docker
deployment process.

Application Deployment Workflow and Orchestration Challenges: This category
focuses on challenges related to application deployment workflow and
orchestration in multi-cloud environment. One challenge is API Management
Deployment, which deals with effectively managing different cloud providers’
APIs/interfaces for multi-cloud container deployment. The solution proposed is
DevOps Approach for Multi-Cloud Applications, which advocates for adopting a
DevOps approach for managing the lifecycle of multi-cloud applications.

Table 13. Deployment challenge and solution framework for containerized applications in multi-cloud environment ID |  Challenge |  Challenge Description |  Solution |  Solution Description
---|---|---|---|---
Category 1: Deployment Complexity and Orchestration Challenges
S01 |  Performance Testing Deployment |  Evaluating testing and performance during container deployment in multi-cloud environment |  Standardized Benchmarking and Automation |  Adoption of standardized benchmarking applications and automatic systems
S09 |  Deployment Validation |  Conducting rigorous validation and artifact checks during container deployment in multi-cloud environment |  Testing Process Management System |  Implementation of a testing process management system
S02 |  Cost-effective Deployment |  Ensuring cost-effective and flexible container deployment options in multi-cloud environment |  Containerization and Microservice Architecture |  Implementation of containerization and microservice architecture
S38 |  Resource Management Deployment |  Managing diverse resources across multiple cloud service providers during container deployment |  User Preference-Based Multi-Cloud Resource Brokering |  Adoption of multi-cloud resource brokering dependent on user preferences and application requirements
S50 |  NFV Site Deployment |  Managing site selection, dynamic VF allocation, and service chaining during NFV deployment in multi-cloud environment Site |  Selection and VF Allocation Algorithms |  Development of algorithms for site selection and dynamic VF allocation
S79 |  Cloud Management Platform |  Evaluation Deployment Addressing diverging functionality, lack of standard benchmark, and evaluating CMPs during multi-cloud container deployment |  Standardized Output Formats and Evaluation Criteria |  Establishment of standardized output formats and evaluation criteria
S37 |  Standardized Requirement Deployment |  Standardizing the expression of application requirements during multi-cloud container deployments |  Application-Level Resource Managers for Multi-Cloud |  Emergence of application-level resource managers supporting multi-cloud deployment
S42 |  Heterogeneous Node Deployment |  Managing the deployment and control of heterogeneous nodes in multi-cloud environment |  Technology-Independent Multi-Level Adaptation |  Utilization of MicroCloud and TOSCA library for technology independence and fine-grained multi-level adaptation
S52 |  Deployment Plan Generation |  Identifying suitable cloud services and generating deployment plans for multi-cloud environment |  Cloud Services Selection and Deployment Planning |  Utilization of the MUSA Decision Support Tool and Deployer tool for cloud services selection and deployment planning
S57 |  TOSCA-based Deployment Implementing |  TOSCA-based container deployment across multiple cloud providers |  Brokered Multi-Cloud Deployment with TOSCA |  Adoption of a brokered approach for multi-cloud deployment with TOSCA
S58 |  Standardization Deployment |  Addressing compatibility issues and lack of standardization during multi-cloud container deployment |  Proxy Cloud Virtualization for OpenStack |  Framework mapping other clouds as sub-clouds to OpenStack with Proxy Cloud Virtualization layer
Category 2: Security and Compliance Management in Multi-Cloud Deployment
S20 |  Vendor Lock-in Prevention Deployment |  Developing strategies to prevent vendor lock-in and ensure modular and loosely-coupled deployment in multi-cloud environment |  Cloud Modelling Framework |  Introduction of a Cloud Modelling Framework for specifying provisioning and deployment
S25 |  Security Deployment in Multi-container Environment |  Addressing security and privacy concerns during multi-container deployment on the same OS kernel in multi-cloud environment |  Cross-Container Isolation |  Enforcing cross-container isolation
S34 |  Security Deployment |  Ensuring integrated security during the deployment of multi-cloud applications |  Security Control Specification and Deployment Framework |  Utilization of the MUSA framework for specifying, deploying, and assessing security controls
S72 |  Virtual Network Security Deployment |  Providing secure virtual networks during multi-cloud container deployment |  Network Virtualization Platform for Multi-Cloud |  Adoption of Sirius, a network virtualization platform for multi-cloud environment
S83 |  Malicious Service Management Deployment |  Managing malicious services and supporting multi-cloud communities formation during container deployment |  Collusion-Resilient Trust Aggregation Technique |  Implementation of a trust aggregation technique resilient to collusion attacks
Category 3: Multi-Cloud Deployment Coordination and Integration
S10 |  Multi-cloud PaaS Deployment |  Overcoming challenges and high entry barriers for deploying a multi-cloud PaaS infrastructure |  Lightweight Proxies for PaaS Integration |  Usage of lightweight proxies for individual PaaS services integration
S85 |  IoT Application Deployment |  Addressing challenges related to deploying IoT applications in multi-cloud environment |  DRA Framework and CI Broker for Multi-Cloud Deployment |  Utilization of the DRA framework and CI broker component for multi-cloud deployment
Category 4: Monitoring, Performance, and Scalability Challenges
S17 |  Legacy Code Migration Deployment |  Handling legacy code migration, tenant engine separation, and composition modeling during container deployment in multi-cloud environment |  Reusable Architectural Pattern with Docker and Cloudant |  Adoption of a reusable architectural pattern with Docker containers and Cloudant persistence layer
S23 |  Tenant Performance Deployment |  Managing tenant service performance competition during multi-cloud container deployment |  Multi-Tenant and Multi-Instance Hybrid Deployment |  Implementation of a multi-tenant and multi-instance hybrid deployment scheme based on container technology
S45 |  Monitoring Deployment |  Overcoming the limitations of current monitoring tools during multi-cloud container deployment |  Platform-Independent Monitoring System |  Implementation of JCatascopia, a platform-independent and interoperable monitoring system
Category 5: Microservices Architecture and Containerization Challenges
S28 |  Autonomic System Deployment |  Implementing an autonomic management system for deploying multi-layer, multi-tier web architectures in multi-cloud environment |  Self-Tuning Performance Model and Autonomic Management |  Development of a self-tuning performance model and autonomic management system
S46 |  Microservice Deployment |  Managing the complexity of deploying microservice architectures in multi-cloud environment |  Metrics and Requirements Framework for Microservices |  Framework for establishing requirements and relevant metrics for the deployment of microservice-based applications
S76 |  Multi-cloud Application Configuration Deployment |  Overcoming challenges related to configuring applications during multi-cloud deployment |  Runtime Environment with Object Store and Artifact Repository |  Deployment of a runtime environment with an object store service and artifact repository
S80 |  Elastic Container Platform Dependency Deployment |  Addressing vendor lock-in and dependency on specific elastic container platforms during multi-cloud deployment |  Separation of Elastic Platform and Cloud Application |  Separation of elastic platform definition from cloud application definition
Category 6: Network Connectivity and Hybrid Cloud Integration
S35 |  Inter-edge Bandwidth Deployment |  Addressing challenges related to interconnecting distributed localities, provisioning inter-edge bandwidth, and centralized edge nodes during multi-cloud deployment |  Modular Edge Cloud Computing Architecture |  Designing a modular edge cloud computing system architecture using containerization techniques
S41 |  Network Latency Deployment |  Minimizing the impact of network latency on application performance during multi-cloud deployment |  VM Selection for Composite Applications |  Selection of proper VMs from the global cloud marketplace to support multiple composite applications
S43 |  Latency Reduction Deployment |  Tackling the issue of high end-to-end latencies during multi-cloud deployment |  Edge-Cellular Hybrid Infrastructure Provisioning |  Provisioning infrastructure at the edge of the cellular network combined with traditional cloud infrastructures
S63 |  Vendor Lock-in Prevention Deployment |  Preventing negative effects of vendor lock-in during multi-cloud container deployment |  Interoperability Layer Above Cloud Infrastructure |  Implementation of a layer above the cloud service provider’s infrastructure for interoperability
Category 7:Cloud Deployment Constraints and Challenges
S08 |  ACS Compatibility Deployment |  Ensuring compatibility with existing ACSs and cloud solutions during container deployment in multi-cloud environment |  Scalable Multi-Cloud Deployment Framework |  Proposal and development of a scalable deployment framework for multi-cloud applications
S30 |  Manual Intervention Risk Deployment |  Managing risks related to manual intervention, vulnerability, and exposed services during multi-cloud deployment |  Automated Security Measures at Deployment |  Automated implementation of required security measures at the point of deployment
S44 |  Multi-provider Deployment |  Handling the challenges of using multiple providers and technologies during multi-cloud container deployment |  TOSCA-Based Deployment Modeling Approach |  Adoption of a TOSCA-based deployment modeling approach
S81 |  Cloud Provider Transition Deployment |  Addressing incompatibilities of cloud solution features during multi-cloud deployment |  Semantic Interoperability in Multi-Clouds |  Adoption of the FCLOUDS framework for semantic interoperability in multi-clouds
Category 8: Infrastructure Provisioning and Container Deployment Challenges
S32 |  Fog Computing Deployment |  Addressing fog computing hardware and container deployment time challenges in multi-cloud environment |  Reorganized Container Images and Docker Deployment |  Reorganization of container images and modification of the Docker deployment process
S33 |  Fog Device Deployment |  Managing fog device availability and services during multi-cloud container deployment |  On-Demand Fog and Microservices Deployment |  Enabling on-demand creation of fogs and deployment of microservices using Docker and Kubeadm
S36 |  Fog Computing Container |  Deployment Overcoming low rates of container deployment, especially in fog computing environments |  FogDocker for Container Deployment |  Deployment of containers using FogDocker after essential file downloads
S39 |  Distributed Compute Node Deployment |  Effectively managing containerized applications across distributed compute nodes in multi-cloud environment |  Label-Based Scheduling Strategy |  Development of a label-based scheduling strategy
S40 |  Kernel Security Deployment |  Addressing security risks related to kernel sharing, isolation, and OS-level security during multi-cloud container deployment |  Enhanced Container Security |  Enhancing container security from the host OS level
Category 9: Application Deployment Workflow and Orchestration Challenges
S27 |  API Management Deployment |  Effectively managing different cloud providers’ APIs/interfaces for multi-cloud container deployment |  DevOps Approach for Multi-Cloud Applications |  Adoption of a DevOps approach for managing the lifecycle of multi-cloud applications
S29 |  Web Service Deployment |  Ensuring correct and efficient deployment of web services across various computing environments |  Kubernetes-Based Containerized Deployment |  Deployment of containerized environments using Kubernetes
S51 |  Complex Application Deployment |  Overcoming complexities associated with application deployment in multi-cloud environment |  Aneka Platform for Distributed Applications |  Adoption of the Aneka platform for programming and deploying distributed applications
S53 |  High-availability Deployment |  Ensuring high availability and preventing outages during deployment in multi-cloud environment |  Middleware Support for High Availability in Multi-Cloud PaaS |  Introduction of novel Middleware Support for High Availability in multi-cloud PaaS
S69 |  Complex Workflow Deployment |  Utilizing multiple cloud providers for deploying complex application workflows CloudSME Simulation Platform |  Utilization of the CloudSME |  Simulation Platform for access to various resources and workflow support
S84 |  Business Process Provisioning Deployment |  Orchestrating multi-cloud business process provisioning during deployment |  Multi-Cloud Service Orchestration Frameworks |  Adoption of multi-cloud service orchestration frameworks

Takeaway 26: Multi-cloud deployment complexity requires standardized
procedures and tools for consistency. Many approaches, like TOSCA-based
Deployment and Proxy Cloud Virtualization, focus on standardizing deployment
practices across diverse cloud environments. Takeaway 27: Security concerns
rise with multi-cloud deployments, demanding protection against vendor lock-in
and malicious services. Innovations like Cross-Container Isolation and
Collusion-Resilient Trust Aggregation tackle security threats, ensuring
integrated protection. Takeaway 26: Multi-cloud deployment demands efficient
tools and strategies for success. Tools like FogDocker and DevOps approaches
introduce novel techniques to streamline microservices, monitoring, and
overall deployment processes.

###  3.8. Monitoring Challenges and Solution Framework (RQ4 and RQ5)

Table LABEL:tab:monitoring-solutions provides an extensive overview of the
monitoring challenges and solution framework for containerized applications in
multi-cloud environment. It encompasses diverse challenges related to
monitoring containerized applications in the complex multi-cloud landscape.
Each category in the figure targets specific monitoring aspects and offers
solutions to address these challenges.

Performance, Consistency, and Variability Monitoring: This category emphasizes
sustaining performance monitoring in dynamically varying cloud environments
for containerized applications. It tackles the challenge of ensuring
consistent performance monitoring and proposes solutions like performance
variation monitoring and benchmarking for optimized performance.

Resource and Infrastructure Monitoring: In this category, the focus is on