# Machine Learning-based Orchestration of Containers: A Taxonomy and Future
Directions

Zhiheng Zhong  The University of MelbourneGrattan Street,
ParkvilleMelbourneVictoriaAustraliaVIC 3010 ,  Minxian Xu  Shenzhen Institute
of Advanced Technology, Chinese Academy of Sciences1068 Xueyuan Avenue,
Shenzhen University Town,ShenzhenGuangdongChina518055 ,  Maria Alejandra
Rodriguez  The University of MelbourneGrattan Street,
ParkvilleMelbourneVictoriaAustraliaVIC 3010 ,  Chengzhong Xu  The University
of MacauAvenida da UniversidadeTaipaMacauChina999078 and  Rajkumar Buyya  The
University of MelbourneGrattan Street, ParkvilleMelbourneVictoriaAustraliaVIC
3010

(2021)

###### Abstract.

Containerization is a lightweight application virtualization technology,
providing high environmental consistency, operating system distribution
portability, and resource isolation. Existing mainstream cloud service
providers have prevalently adopted container technologies in their distributed
system infrastructures for automated application management. To handle the
automation of deployment, maintenance, autoscaling, and networking of
containerized applications, container orchestration is proposed as an
essential research problem. However, the highly dynamic and diverse feature of
cloud workloads and environments considerably raises the complexity of
orchestration mechanisms. Machine learning algorithms are accordingly employed
by container orchestration systems for behavior modelling and prediction of
multi-dimensional performance metrics. Such insights could further improve the
quality of resource provisioning decisions in response to the changing
workloads under complex environments. In this paper, we present a
comprehensive literature review of existing machine learning-based container
orchestration approaches. Detailed taxonomies are proposed to classify the
current researches by their common features. Moreover, the evolution of
machine learning-based container orchestration technologies from the year 2016
to 2021 has been designed based on objectives and metrics. A comparative
analysis of the reviewed techniques is conducted according to the proposed
taxonomies, with emphasis on their key characteristics. Finally, various open
research challenges and potential future directions are highlighted.

Container Orchestration, Machine Learning, Cloud Computing, Resource
Provisioning, Systematic Review

††copyright: acmcopyright††journalyear: 2021††doi: 0††journal:
JACM††journalvolume: 0††journalnumber: 0††article: 0††publicationmonth: 0

##  1\. Introduction

Our era has witnessed the trend of cloud computing becoming the mainstream
industrial computing paradigm, providing stable service delivery with high
cost-efficiency, scalability, availability, and accessibility (cloudcom, ).
Existing mainstream cloud service providers, including Amazon Web Services
(AWS) (aws, ), Google (borg, ), and Alibaba (alibabasemi, ), prevalently adopt
virtualization technologies including virtual machines (VM) and containers in
their distributed system infrastructures for automated application deployment.
In recent years, their infrastructures are evolving from VM centric to
container centric (soatcontainertech, ).

Containers employ a logical packing mechanism that binds both software and
dependencies together for application abstraction (costeffi, ). Unlike VMs
that support hardware-level resource virtualization where each VM has to
maintain its own operating system (OS), containers virtualize resources at the
OS level where all the containers share the same OS with less overhead.
Therefore, containers are more lightweight in nature with high application
portability, resource efficiency, and environmental consistency. They define a
standardized unit for application deployment under an isolated runtime system.
Thanks to these features, we have observed the prevalence of container
technologies for automatic application deployment under diverse cloud
environments.

Following this trend of containerization, container orchestration techniques
are proposed for the management of containerized applications. Container
orchestration is the automated management process of a container lifecycle,
including resource allocation, deployment, autoscaling, health monitoring,
migration, load balance, security, and network configuration. For cloud
service providers which have to handle hundreds or thousands of containers
simultaneously, a refined and robust container orchestration system is the key
factor in controlling overall resource utilization, energy efficiency, and
application performance. Under the surge of cloud workloads in terms of
resource demands, bandwidth consumption, and quality of service (QoS)
requirements, the traditional cloud computing environment is extended to fog
and edge infrastructures that are close to end users with extra computational
power (edgecomputing, ). Consequently, this also requires current container
orchestration systems to be further enhanced in response to the rising
resource heterogeneity, application distribution, workload diversity, and
security requirements across hybrid cloud infrastructures.

###  1.1. Needs for Machine Learning-based Container Orchestration

Considering the increasingly diverse and dynamic cloud workloads processed by
existing cloud service providers, it is still unclear how to automate the
orchestration process for complex heterogeneous workloads under large-scale
cloud computing systems (htas, ; microservicereview, ).

In traditional cloud computing platforms, Container Orchestrators are usually
designed with heuristic policies without consideration of the diversity of
workload scenarios and QoS requirements (containerbasedsystem, ). Their main
drawbacks are listed as follows:

  1. (1)

Most of these policies are static heuristic methods configured offline
according to certain workload scenarios at a limited scale. For instance,
threshold-based autoscaling strategies can only be suitable for managing a set
of pre-defined workloads (costeffi, ; googletrace, ; stratus, ). Such policies
can not handle highly dynamic workloads where applications need to be scaled
in/out at runtime according to specific behavior patterns.

  2. (2)

The performance of heuristic methods could dramatically degrade when the
system scales up. For example, bin-packing algorithms, such as best fit or
least fit algorithms, are widely utilised for solving task scheduling and
resource allocation (fitnessaware, ; containerpowereffi, ; video, ). However,
such algorithms could perform poorly with high task scheduling delays in
large-scale compute clusters.

  3. (3)

Resource contention and performance interference between co-located
applications are usually ignored. Co-located applications could compete for
shared resources, which may cause application performance downgrade, extra
maintenance costs, and service-level agreement (SLA) violations (borg, ;
alibabasemi, ; htas, ).

  4. (4)

The dependency structures between containerized application components are not
considered during resource provisioning. Although some existing studies
(networkawareapp, ; powerawarescheduling, ; dynamicplacementmobile, ;
multicompo, ) address this issue to a certain degree by leveraging machine
learning (ML) methods, their ML models are only feasible for traditional
cloud-based applications and relatively simple for containerized workload
scenarios. For instance, containerized microservice-based applications are
more lightweight and decentralized in nature, compared with traditional
monolithic applications. There are internal connections between different
microservice units within an application. Critical components in a
microservice architecture are the dependencies of most other microservices and
more likely to suffer from service level objectives (SLO) violations due to
higher resource demands and communication costs (sinan, ; firm, ). These
factors should all be taken into account during resource provisioning.

  5. (5)

Current container orchestration methodologies mostly emphasize the evaluation
of infrastructure-level metrics, while application-level metrics and specific
QoS requirements are not receiving sufficient consideration. For example,
containerized workloads may be attached with stricter time constraints
compared with traditional cloud workloads, such as task deployment delays,
task completion time, and communication delays (news, ; jobplacementdmlc, ).

With such fast-growing complexity of application management in cloud
platforms, cloud service providers have a strong motivation to optimize their
container orchestration policies by leveraging machine ML techniques
(towardmlcentric, ). Through applying mathematical methods to training data,
ML algorithms manage to build sophisticated analytic models that can precisely
understand the behavior patterns in data flows or impacts of system
operations. Thus, ML approaches could be adopted for modelling and prediction
of both infrastructure-level or application-level metrics, such as application
performance, workload characteristics, and system resource utilization. These
insights could further assist the Container Orchestrators to improve the
quality of resource provisioning decisions. On the other hand, ML algorithms
could directly produce resource management decisions instead of heuristic
methods, offering higher accuracy and lower time overhead in large-scale
systems (sarsa, ; negotiation, ; FScaler, ; migrationfog, ).

###  1.2. Motivation of Research

Machine learning-based container orchestration technologies have been
leveraged in cloud computing environments for various purposes, such as
resource efficiency, load balance, energy efficiency, and SLA assurance.
Therefore, we aim to investigate them in depth in this paper:

  1. (1)

The machine learning-based container orchestration technologies have shown
promise in application deployment and resource management in cloud computing
environments. Hence, we aim to outline the evolution and principles of machine
learning-based container orchestration technologies in existing studies.

  2. (2)

We recognize the need for a literature review to address the status of machine
learning-based container orchestration researches in such fast-evolving and
challenging scenarios. Furthermore, we investigate and classify the relevant
papers by their key features. Our goal is to identify potential future
directions and encourage more efforts in advanced research.

  3. (3)

Although an innovative review is proposed on container orchestration in
Reference (containerbasedsystem, ), the research on this field is growing
continually by leveraging machine learning models. Therefore, there is a need
for fresh reviews of machine learning-based container orchestration approaches
to find out the advanced research challenges and potential future directions.

###  1.3. Our Contributions

The main contributions of this work are:

  1. (1)

We introduce a taxonomy of the most common ML algorithms used in the field of
container orchestration.

  2. (2)

We present a comprehensive literature review of the state-of-the-art machine
learning-based container orchestration approaches.

  3. (3)

We classify the reviewed orchestration methods by their key characteristics
and conditions.

  4. (4)

We identify the future directions of machine learning-based container
orchestration technologies.

###  1.4. Related Surveys

Table 1. A Comparison of Our Work with Existing Surveys Based on Key Parameters Ref. |  | Application Deployment
---
Unit
Infrastructure | Classification Schemes
VM | Container | Cloud | Fog | Edge |  | Application
---
Architecture
| Behavior Modelling
---
and Prediction
| Resource
---
Provisioning
| ML-based
---
Orchestration Strategies
(cloudresource, ) | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗
(brownout, ) | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗
(mlsurvey, ) | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | ✓ | ✓
(qosawarecloud, ) | ✓ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗
(soatcontainertech, ) | ✗ | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | ✓ | ✗
(cloudconainertechsoat, ) | ✗ | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | ✗
(containerbasedsystem, ) | ✗ | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗
This review | ✗ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓

As summarized in Table 1, some previous surveys have already explored the
field of application management in cloud computing environments. Weerasiri et
al. (cloudresource, ) introduce a comprehensive classification framework for
analysis and evaluation of cloud resource orchestration techniques. Xu and
Buyya (brownout, ) survey brownout technologies for adaptive application
maintenance in cloud computing systems. Duc et al. (mlsurvey, ) look into the
research challenge of resource management and performance optimization under
edge-cloud platforms, with emphasis on workload modelling and resource
management through ML techniques. Singh and Chana (qosawarecloud, ) depict a
classification of QoS-aware autonomic resource provisioning under cloud
computing environments through the methodical analysis of related research.
However, these works are designed for managing general cloud workloads without
adequate analysis of the key characteristics of containerized applications.

Furthermore, some recent studies also investigate different scopes of
container orchestration techniques. Casalicchio and Iannucci
(soatcontainertech, ) conduct an extensive literature review of the state-of-
the-art container technologies, focusing on three main issues, including
performance evaluation, orchestration, and cyber-security. Pahl et al.
(cloudconainertechsoat, ) present a survey and taxonomy of cloud container
technologies with a systematic classification of the existing researches.
Rodriguez and Buyya (containerbasedsystem, ) propose a systematic review and
taxonomy of the mainstream container orchestration systems by classifying
their features, system architectures, and management strategies. Nonetheless,
the research direction of machine learning-based orchestration for
containerized applications has not been explored with systematic categories in
any existing survey and taxonomy.

Therefore, this paper extends the previous surveys by focusing on how machine
learning algorithms could be applied to solve complex research problems from a
container orchestration perspective, such as multi-dimensional workload
characterization or autoscaling in hybrid clouds. We emphasis the diversity
and complexity of orchestration schemes under specific application
architectures and cloud infrastructures. Furthermore, it also specifies the
extensive research questions and potential future directions of machine
learning-based container orchestration techniques.

###  1.5. Article Structure

The rest of this paper is organized as follows: Section 2 introduces the
background of machine learning and container orchestration. Section 3
describes an overview of machine learning-based container orchestration
technologies, followed by the categories and taxonomy of the existing
approaches. A description of the reviewed approaches and their mappings to the
categories are presented in Section 4. Section 5 summarizes a discussion the
future directions and open challenges. Finally, the conclusion of this work is
given in Section 6.

##  2\. Background

In this section, we present a comprehensive taxonomy of the existing ML models
utilised in the area of container orchestration and a brief introduction of
the high-level container orchestration framework.

###  2.1. Machine Learning

Figure 1. Machine Learning Taxonomy in the Context of Container Orchestration
by Optimization Objectives

Machine Learning

Machine learning is a general concept describing a set of computing algorithms
that can learn from data and automatically build analytical models through
data analysis (mlreview, ). One of its fundamental objectives is to build
mathematical models that can emulate and predict the behavior patterns of
various applications through training data. Machine learning has gained
immense popularity through the past decades, widely accepted in many research
areas such as image recognition, speech recognition, medical diagnose, and
smart building (mlstartbuilding, ; breastcancer, ). With the continuously
growing computational power and adoption of GPUs in warehouse-scale
datacenters, the capability and data scale of machine learning technologies
are still soaring (deeplearning, ).

As depicted in Fig. 1, we design a machine learning taxonomy through
classifying some of the most popular ML models in the field of container
orchestration by their optimization objectives:

  1. (1)

Regression algorithms predict a continuous output variable through analysis of
the relationship between the output variable and one or multiple input
variables. Popular regression algorithms (e.g., support vector regression,
random forest, and polynomial regression) are usually used to explore and
understand the relation between different performance metrics.

  2. (2)

Classification methods categorize training data into a series of classes. The
use cases of classification mainly include anomaly detection and dependency
analysis. For example, K-means is broadly adopted for the identification of
abnormal system behaviors or components (iotcontainer, ;
containeranomalydetection, ), while support vector machine (SVM) can be
leveraged for decomposing the internal structure of containerized applications
and identifying key application components (firm, ).

  3. (3)

Decision making models generate resolutions by simulating the decision making
process and identifying the choices with the maximized cumulative rewards
(rlsurvey, ). As the most common algorithms in this category, reinforcement
learning (RL) models, including model-free (e.g., Q-Learning and Actor-Critic)
and model-based RL, have been widely employed for decision making in resource
provisioning.

  4. (4)

Time series analysis achieves pattern recognition of time series data and
forecast of future time series values from past values. Ranging from
autoregressive integrated moving average (ARIMA) to more advanced recurrent
neural network (RNN) models, such algorithms are useful for behavior modelling
of sequential data, including workload arrival rates or resource utilization.

To be noted, some ML models can be utilised under multiple optimization
scenarios. For example, artificial neural network (ANN) models can be applied
for time series analysis of resource utilization or regression of application
performance metrics (sarsa, ; performancedocker, ).

###  2.2. Container Orchestration

Figure 2. A High-level Machine Learning-based Container Orchestration
Framework Reference Architecture

A High-level Machine Learning-based Container Orchestration Framework
Reference Architecture

Container orchestration empowers cloud service providers to decide how
containerized applications are configured, deployed, and maintained under
cloud computing environments (soatcontainertech, ). It is targeted at
automatic application deployment and dynamic configuration adjustment at
runtime for a diverse range of workloads. Fig. 2 demonstrates a reference
architecture of machine learning-based container orchestration frameworks,
where the components included are common to most existing systems. A detailed
description of each layer is given as below:

####  2.2.1. Application Workloads

Workloads are generally defined as the requests submitted by users to an
application or the Orchestrator. Workloads could be classified into many
categories from different perspectives according to their unique features,
such as long-running services or short-living batch jobs (classified by
execution time); computation-intensive, data-intensive, or memory-sensitive
workloads (classified by resource usage pattern) (borg, ; alibabasemi, ;
mixbatch, ). Each workload is also defined with multi-dimensional resource
requirements (e.g., CPU, memory, disk, network, etc.) and QoS requirements
(e.g., time constraints, priorities, throughput, etc). The extremely dynamic
and unpredictable feature of heterogeneous workloads greatly increases the
complexity of orchestration mechanisms.

####  2.2.2. Compute Cluster

A compute cluster is a group of virtual or physical computational nodes that
run in a shared network. As the core component in a containerized cluster, the
Container Orchestrator is responsible for assigning containerized application
components to worker nodes where containers are actually hosted and executed.
Its major functionalities involve:

  1. (1)

Resource allocation assigns a specific amount of resources to a container.
Such configurations and limitations are basic metrics in managing container
placement and resource isolation control between containers.

  2. (2)

Scheduling defines the policy for the initial placement for one or a group of
containers, by considering conditions such as resource constraints,
communication costs, and QoS requirements.

  3. (3)

Scaling is the resource configuration adjustment of containerized applications
or computational nodes in response to any potential workload fluctuations.

  4. (4)

Migration is designed as a complementary mechanism to prevent severe
infrastructure-level resource overloading or resource contention between co-
located applications by relocating one or a group of containers from one node
to another.

  5. (5)

Infrastructure monitoring keeps the track of infrastructure-level metrics of
the cluster, such as the number of nodes, node states, resource usage of each
node, and network throughput. Such metrics are the critical information for
evaluating the health conditions of the cluster and making precise decisions
in the above resource source provisioning layers.

  6. (6)

Application monitoring does not only periodically check the application states
but also records their resource consumption and performance metrics (e.g.,
response time, completion time, throughput, and error rates). This information
serves as the crucial reference data in anomaly detection and SLA violation
measurement.

  7. (7)

Load distribution, as the core mechanism for load balancing under
containerized environments, distributes network traffic between containers
evenly with policies such as Round-Robin (roundrobin, ). It plays an important
role in improving system scalability, availability, and network performance.

  8. (8)

Fault tolerance ensures the high availability of the system through replica
control. Each container is maintained with a configurable number of replicas
across multiple nodes in case of a single point of failure. It is also
possible to have multiple Orchestrators to deal with unexpected node crashes
or system overloading.

####  2.2.3. Infrastructure

Thanks to the high portability, flexibility, and lightweight nature of
containers, it allows containers to be deployed across a multitude of
infrastructures, including private clouds, public clouds, fog, and edge
devices. Similar to traditional server farms, private clouds provide exclusive
resource sharing within a single organization based on its internal datacenter
(cloudcom, ). By contrast, public cloud service providers support on-demand
resource renting from their warehouse-scale cloud datacenters.

Since applications and workloads are all deployed and processed at cloud
datacenters in traditional cloud computing, this requires a massive volume of
data transferred to cloud servers. As a consequence, it leads to significant
propagation delays, communication costs, bandwidth and energy consumption.
Through moving computation and storage facilities to the edge of a network,
fog and edge infrastructures can achieve higher performance in a delay-
sensitive, QoS-aware, and cost-saving manner (edgesurvey, ; fogsurvey, ). With
the requests or data directly received from users, fog or edge nodes (e.g.,
IoT-enabled devices, routers, gateways, switches, and access points) can
decide whether to host them locally or send them to the cloud.

####  2.2.4. Machine Learning-based Optimization Engine

An ML-based optimization engine builds certain ML models for workload
characterization and performance analysis, based on analysis of monitoring
data and system logs received from the Orchestrator. Furthermore, it can
produce future resource provisioning decisions relying on the generated
behavior models and prediction results. The engine can be entirely integrated
or independent from the Orchestrator. Its major components are listed as
follows:

  1. (1)

Workload Modeller is designed for ML-based workload characterization through
analyzing the input application workloads and identifying their key
characteristics.

  2. (2)

Performance Analyzer generates application and system behavior models through
applying ML algorithms to application and infrastructure-level monitoring data
acquired from the Orchestrator.

  3. (3)

Predictor forms forecasts of workload volumes or application/system behaviors,
relying on the models obtained from Workload Modeller and Performance
Analyzer. The prediction results could be sent either to the Orchestrator or
Decision Maker.

  4. (4)

Decision Maker combines the behavior models and prediction results received
from the above components with certain ML-based optimization methods/schemes
to further generate precise resource provisioning decisions that are fed back
to the Orchestrator.

These components are common to most existing ML-based approaches, however, not
all of them have to be implemented to make the whole system functional.
According to different design principles, the engine can alternatively produce
multi-level prediction results to assist the original Orchestrator in making
high-quality decisions in resource provisioning, or directly generate such
decisions instead of the Orchestrator.

##  3\. Taxonomy of Machine Learning-based Container Orchestration
Technologies

Figure 3. Machine Learning-based Container Orchestration Taxonomy

Container Orchestration

Fig. 3 presents a taxonomic classification of the literature reviewed in our
work. Application Architecture describes the behaviors and internal structures
of containerized application components that together perform as a whole unit.
Infrastructure indicates the environments or platforms where the applications
operate. Objectives are the improvements that the proposed ML-based approaches
attempt to achieve. Behavior Modelling and Prediction leverage ML models for
pattern recognition and simulation of system and application behaviors, as
well as forecasting future trends according to previously collected data.
Resource Provisioning specifies the ML-based or heuristic policies for
resource management of containerized applications at different phases in a
container lifecycle under various scenarios.

Figure 4. An Example of the Microservice Architecture

Microservice

###  3.1. Application Architecture

This section presents the most common application architectures that define
the composition of containerized applications and how they are deployed,
executed, and maintained.

####  3.1.1. Monolithic

Monolithic applications follow an all-in-one architecture where all the
functional modules are developed and configured into exactly one deployment
unit, namely one container. Such applications could be initially easy to
develop and maintain at a small scale. For example, Microsoft Azure (azure, )
still supports automated single-container-based application deployment for
enterprise solutions where the business logic is not feasible for building
complex multi-component models. However, the consistent development and
enrichment of monolithic applications would inevitably lead to incremental
application sizes and complexity (dsormono, ). Consequently, the maintenance
costs can dramatically grow in continuous deployment. Even modification of a
single module requires retesting and redeployment of the whole application.
Furthermore, scaling of monolithic applications means replication of the
entire deployment unit containing all the modules (webmigration, ). In most
scenarios, only a proportion of the modules need to be scaled due to resource
shortage. Therefore, scaling the whole application would lead to poor overall
resource efficiency and reliability. Thus, the monolithic architecture is only
suitable for small-scale applications with simple internal structures.

####  3.1.2. Microservice

To address the problem of high development and maintenance costs caused by
colossal application sizes, the microservice architecture (MSA) is proposed to
split single-component applications into multiple loosely coupled and self-
contained microservice components (micromono, ). An example of MSA is given in
Fig. 4. Each microservice unit can be deployed and operate independently for
different functionalities and business objectives. Furthermore, they can
interact with each other and collaborate as a whole application through
lightweight communication methods such as representational state transfer
(REST). Through decomposing applications into a group of lightweight and
independent microservice units, MSA has significantly reduced the costs and
complexity of application development and deployment. Nonetheless, the growing
number of components and dynamic inter-dependencies between microservices in
MSA raises the problem of load distribution and resource management at the
infrastructure level. A well-structured orchestration framework for MSA should
be able to maintain multiple parts of an application with SLA assurance,
granting more control over individual components.

####  3.1.3. Serverless

Figure 5. An Example of the Serverless Architecture

Serverless

The serverless architecture defines an event-driven application paradigm that
undertakes stateless computational tasks, namely serverless functions.
Designed to perform certain user-defined functionalities, functions are
usually short pieces of code hosted in function containers with specific
configurations and limited execution time. To ensure the successful completion
and performance requirements of functions, serverless platforms are
responsible for managing the function execution environments in terms of
resource provisioning, energy consumption, SLA assurance, and security. On the
one hand, this grants the platforms more control over application deployment
and maintenance at the infrastructure level, while clients can focus on the
development of application features and business logic. On the other hand,
clients only have to pay for the resource consumption generated during
function invocation. Overall, it provides clients with a cost-efficient pay-
for-value billing standard as well as higher resource availability,
scalability, and reliability. Thus, the serverless architecture has been
adopted by most mainstream cloud service providers to automate workload-aware
application deployment (amazonlambda, ; serverlessreview, ).

As depicted in Fig. 5, a typical serverless application is represented in the
form of a function chain consisting of a workflow of individual functions.
Within a function chain, interactions and transitions between functions are
conducted through a centralized messaging service, such as a message queue or
event bus (serverlesscoldstart, ). The orchestration problem under such
context translates into managing the invocation of function chains in terms of
function initialization, execution, transition, and data flow control. Several
previous studies have proved that the significant time overhead of function
initialization is currently the major performance bottleneck in function
invocation (catalyzer, ; firecracker, ). Therefore, the current research focus
of ML-based solutions in this field is on minimization of function invocation
delays and resource wastage (fifer, ; functionserverless, ; rlserverless, ).

###  3.2. Infrastructure

A cloud infrastructure consists of a set of hardware resources and
virtualization software to deliver virtualized resources to users for
application deployment. In general, we have identified three types of cloud
infrastructures in the context of container orchestration.

  1. (1)

A single cloud environment is built on resources from only one cloud service
provider (either a private or public cloud) to host and serve all the
applications.

  2. (2)

A multi-cloud environment includes multiple cloud services (e.g., private
clouds, public clouds, or a mixture of both). As different cloud service
providers may differ in many aspects, such as resource configurations, price,
network latency, and geographic locations, this allows more choices for
optimization of application deployment.

  3. (3)

A hybrid cloud environment is composed of a mixture of private clouds, public
clouds, fog, or edge devices. It is not always efficient to deploy all the
applications and data to public/private clouds, considering the data
transmission time and network latency from end users to cloud servers. To
solve this issue, the hybrid cloud enables applications and data to be
deployed and processed at fog or edge devices that are close to end users.

###  3.3. Objectives

In light of the diversity of application architectures and cloud
infrastructures, many types of metrics have been considered as optimization
objectives during the behavior modelling and resource provisioning of
containerized applications. As presented in Fig. 3, we group the existing
objective metrics into four major categories. Since an orchestration solution
usually needs to achieve multiple objectives according to specific user
requirements, balancing the trade-off between different optimization
objectives remains a key concern in automated application deployment and
maintenance.

  1. (1)

Resource Efficiency. Infrastructure-level resource usage metrics are usually
treated as key application performance indicators for energy and cost
efficiency. They are the fundamental data source of most behavior modelling
schemes, such as prediction of the resource demands of coming workloads, or
discovery of the relationship between resource usage patterns and other
performance metrics. Such insights could be further applied in decision making
of resource provisioning to improve the overall resource efficiency and
application performance.

  2. (2)

Energy Efficiency. Under the continuously growing scale of cloud data centers,
the tremendous electricity usage consumed by cloud infrastructures has emerged
as a critical concern in the field of cloud computing (energyeffi, ).
Therefore, various approaches have been proposed to minimize energy
consumption and optimize energy efficiency (iotcontainer, ; brownout, ; fifer,
; kuberknots, ). As the overall electricity usage of a system is estimated as
the summation of the consumption by each physical machine (PM) where its
energy usage is directly related to its resource utilization, an essential way
to control energy efficiency is to adjust and balance the resource utilization
of physical machines during resource provisioning.

  3. (3)

Cost Efficiency. Following the pay-as-you-go payment model of mainstream cloud
service providers, market-based solutions regard cost efficiency as one of
their principal targets (costeffi, ; stratus, ; negotiation, ; selfadptive, ).
Through evaluation and selection of diverse cloud services according to their
pricing models and computing capability, an optimized orchestration solution
aims to minimize the overall financial costs while satisfying the QoS
requirements defined by users.

  4. (4)

SLA Assurance. Containerized applications are mostly configured with specific
performance requirements, such as response time, initialization time,
completion time, and throughput. These constraints are mostly expressed as SLA
contracts, while their violations could lead to certain penalties. Because of
the dynamic and unpredictable feature of cloud workloads, autoscaling is
usually leveraged to automate application maintenance with SLA assurance
(rlserverless, ; htas, ; sarsa, ; microsacler, ; negotiation, ; Probabilistic,
; aiautoscale, ), in response to the frequently changing workloads.

###  3.4. Behavior Modelling and Prediction

Behavior modelling is a fundamental step in understanding the overall
application or system behavior patterns through analysis of
application/infrastructure-level metrics. The variety of multi-layer metrics
related to workloads, application performance, and system states significantly
complicates the modelling process. Nonetheless, a well-structured behavior
model that can produce precise prediction results is apparently useful for
achieving certain optimization objectives during orchestration.

  1. (1)

Workload characterization captures the key features of application workloads.
Because of the dynamic and decentralized nature of containerized applications
like microservices, the received workloads may differ in many ways, such as
task structures, resource demands, arrival rates, and distributed locations.
These factors make it hard to define a robust method for characterization and
categorization of all the different workloads within an orchestration system.
However, the knowledge of workload behaviors is necessary to improve the
quality of resource provisioning decisions by making precise resource
assignments in response to any incoming workloads.

  2. (2)

Performance analysis discovers the relation among infrastructure (e.g.,
resource utilization and energy consumption) or application-level (e.g.,
response time, execution time, and throughput) metrics to depict the system
states and application performance. These insights are important in managing
the trade-off between different optimization objectives.

  3. (3)

Anomaly detection classifies and identifies abnormal system behaviors,
including security threats, instance failure, workload spikes, performance
downgrade, and resource overloading. Such anomalies could severely harm the
system availability and reliability. Therefore, fast and accurate localization
of their root causes could prevent SLA violations or system crashes.

  4. (4)

Dependency analysis looks into the graph-based internal dependencies between
containerized application components. It helps to understand the workload
distribution/pressure among application components and make precise resource
configurations. Since the dependencies may be dynamically updated at runtime,
it requires an incremental model that can consistently adjust itself and
address the chain reactions of individual components to the overall
application performance.

###  3.5. Resource Provisioning

Figure 6. Resource Provisioning Taxonomy

RPT

Considering the various application architectures, infrastructures, and
optimization objectives mentioned in the above sections, resource provisioning
for containerized applications has become much more challenging. The diversity
of cloud workloads, resource heterogeneity within hybrid cloud environments,
and complex application internal structures should be all assessed during
resource provisioning. Therefore, the state-of-the-art resource provisioning
strategies are commonly relying on behavior modelling and prediction, to
achieve higher accuracy and shorter computation delays. As addressed in Fig.
6, resource provisioning operations can be classified into the following
categories:

  1. (1)

Scheduling decides the initial placement of a containerized task unit, which
could consist of a single task, a group of independent tasks, or a group of
dependent tasks in graph-based structures. Due to the variety of application
architectures and task structures, application deployment policies should
consider a wide range of placement schemes. The quality of scheduling
decisions has a direct impact on the overall application performance and
resource efficiency (microservicereview, ).

  2. (2)

Scaling is the size adjustment of containerized applications or computational
nodes in response to any potential workload fluctuations, which ensures the
applications supported with enough resources to minimize SLA violations.
Horizontal scaling adjusts the number of container replicas belonging to the
applications or the number of nodes. By contrast, vertical scaling only
updates the amount of resources assigned to existing containers or nodes.
Moreover, hybrid scaling combines both horizontal and vertical scaling to
produce an optimized solution.

  3. (3)

Migration is the relocation of one or a group of tasks from one node to
another. When resource contention, overloading, or underloading occur between
co-located applications due to poor scheduling decisions, rescheduling is
triggered for load balancing through task migration within a single cloud. On
the other hand, computation offloading manages the migration of computation-
intensive tasks that are experiencing resource bottlenecks to devices with
enough demanded resources across hybrid cloud environments. Targeted to
improve the application performance, a refined offloading policy should be
able to pick a suitable relocation destination that minimizes the migration
costs in terms of execution time, bandwidth and energy consumption.

###  3.6. Evolution of Machine Learning-based Container Orchestration
technologies

Figure 7. Evolution of Machine Learning-based Container Orchestration
Technologies

Evolution of Machine Learning-based Container Orchestration Approaches

This section describes how ML-based methods are applied in the context of
container orchestration. Fig. 7 demonstrates the evolution of ML-based models
since 2016, with emphasis on their objectives and training metrics.

In 2016, the ARIMA (CRUPA, ) and nearest neighbor (NN) (video, ) algorithms
were already leveraged for resource utilization prediction of containerized
applications. At this stage, the application models were relatively simple,
which only considered the time series pattern of infrastructure-level resource
metrics.

In 2017, Shah et al. (denpendencyana, ) first adopted the long short-term
memory (LSTM) model for dependency analysis of microservices. Their model
evaluated both the internal connections between microservice units and the
time series pattern of resource metrics. Furthermore, anomaly detection was
built on top of the LSTM model for the identification of abnormal behaviors in
resource utilization or application performance. Besides, Cheng et al.
(highutiautoscaling, ) used Gradient Boosting Regression (GBR) for resource
demand prediction in workload characterization.

Xu et al. (negotiation, ) leveraged a model-free RL method, namely Q-Learning,
to produce vertical scaling plans. An optimal scaling decision was targeted to
minimize resource wasting and computation costs under the assumption of SLA
assurance.

In 2018, Tang et al. (fisher, ) employed the bidirectional LSTM (Bi-LSTM)
model for the prediction of workload arrival rates and application throughput.
Their training module had demonstrated significant accuracy improvement over
ARIMA and LSTM models in terms of time series prediction. Ye et al.
(performancedocker, ) applied a series of regression methods, including
support vector regression (SVR), linear regression (LR), and ANN, to conduct
performance analysis of relevant resource metrics. They attempted to evaluate
the relationship between resource allocation and application performance.
However, only single-component applications with limited performance
benchmarks were considered within the scope of their work.

Du et al. (containeranomalydetection, ) designed an anomaly detection engine
composed of k-nearest neighbors (KNN), SVM, Naive Bayes (NB), and random
forest (RF), to classify and diagnose abnormal resource usage patterns in
containerized applications. Orhean et al. (news, ) utilised
state–action–reward–state–action (SARSA), a model-free RL algorithm, to manage
the graph-based task scheduling problem in directed acyclic graph (DAG)
structures, aiming at minimizing the overall DAG execution time.

In 2019, Cheng et al. (gruesprediction, ) proposed a hybrid gated recurrent
unit (GRU) model to further reduce the computational costs and error rates of
resource usage prediction of cloud workloads. Since LSTM models are relatively
complex with high computational costs and data processing time, GRU is
considered as an optimized model of LSTM (gru, ). An LSTM cell structure
consists of three gates, including the input gate, the forget gate, and the
output gate. GRU simplifies this structure and achieves higher computational
efficiency by integrating the input gate and the forget gate into one update
gate.

Performance analysis of containerized applications was also explored in depth.
Venkateswaran and Sarkar (fitnessaware, ) leveraged the K-means clustering and
polynomial regression (PR) to classify the multi-layer container execution
structures under multi-cloud environments by their feasibility to the
application performance requirements. According to workload arrival rates and
resource metrics, Podolskiy et al. (slocloudnative, ) applied Lasso regression
(LASSO) to forecast service level indicators (SLI) in terms of application
response time and throughput. Dartois et al. (ssdidperformance, ) used the
decision tree (DT) regression algorithm to analyze the solid state drive (SSD)
I/O performance under interference between applications.

As for resource provisioning, deep reinforcement learning (DRL) was first
applied in the context of task scheduling (jobplacementdmlc, ; migrationfog,
). Bao et al. (jobplacementdmlc, ) designed a DRL framework for the placement
of batch processing jobs, where an ANN model represented the mapping
relationship between workload features, system states, and corresponding job
placement decisions. The Actor-Critic RL algorithm was selected to train the
ANN model and generate optimal scheduling decisions that minimized the
performance interference between co-located batch jobs. Compared with
traditional heuristic scheduling policies like bin packing, their solution
demonstrated remarkable performance improvement regarding overall job
execution time. Moreover, DRL was also employed to solve the problem of
computation offloading under fog-cloud environments in Reference
(migrationfog, ). On top of a Markov decision process (MDP) model that
simulated the interactions of the offloading process at a large scale, the
deep Q-Learning method optimized the migration decisions by minimization of
the time overhead, energy usage, and computational costs. To explore the
efficiency of hybrid scaling mechanisms, Rossi et al. (hybridscaling, ;
Geocontainer, ) leveraged model-based RL models to compose a mixture of
horizontal and vertical scaling operations for monolithic applications, aiming
at minimizing the resource usage, performance degradation, and adaption costs.

In 2020, several RL-based scaling approaches were proposed in the form of
hybrid ML models (firm, ; FScaler, ; sarsa, ). Qiu et al. (firm, ) adopted the
SVM model for dependency analysis of microservices and recognition of the key
components that are highly likely to experience resource bottlenecks and
performance downgrade. To prevent severe service level objectives (SLO)
violations, the Actor-Critic method was utilised to generate the appropriate
resource assignment decisions for these components through horizontal scaling.
Besides, Sami et al. (FScaler, ) combined MDP and SARSA models to build a
horizontal scaling solution for monolithic applications under fog-cloud
environments. SARAS produced the optimized scaling decisions through model
training relying on the MDP model that simulated the scaling scenarios with
the fluctuating workloads and resource availability in fog taken into account.

In 2021, Zhang et al. (sinan, ) proposed a novel approach composed of a
convolutional neural network (CNN) and boosted trees (BT) for dependency and
performance analysis of microservices. Their CNN model did not only analyze
the inter-dependencies between microservice units for system complexity
navigation, but also the time series metrics related to application
performance. Furthermore, the BT model is responsible for the prediction of
long-term QoS violations. To further improve the speed and efficiency of RL-
based scaling approaches for microservices under hybrid cloud environments,
Yan et al. (HANSEL, ) developed a multi-agent parallel training module based
on SARSA, supported by the microservice workload prediction results generated
by Bi-LSTM.

Overall, diverse ML algorithms have been utilised in the context of container
orchestration, ranging from workload modelling to decision making through RL.
However, there are not many new ML models adopted in the area of container
orchestration in recent years. To further improve prediction accuracy and
computational efficiency, the emerging trend of hybrid ML-based solutions
targets to combine multiple existing ML methods to form a complete
orchestration pipeline, including multi-dimensional behavior modelling and
resource provisioning. The evolution of ML models also contributes to the
extension of various application architectures and cloud infrastructures.

##  4\. State-of-the-Art in Machine Learning for Orchestration of Containers

In this section, we introduce a literature review of machine learning-based
container orchestration approaches. To stress the key features in the
evaluated studies, we use the taxonomy in Section 3 to outline the key
characteristics of the approaches designed for behavior modelling and
prediction, as well as resource provisioning. The articles discussed in each
class are key representative works selected from literature by several
evaluation criteria, including robustness of application models, coverage of
critical challenges in the field, accurate depiction of real-world
orchestration scenarios, high scalability under complex cloud environments,
novelty of orchestration approaches, refined combinations of ML-based models,
and extensibility to hybrid clouds.

###  4.1. Behavior Modelling and Prediction

This section presents the approaches related to behavior modelling and
prediction.

####  4.1.1. Workload Characterization

Table 2. Summary of Workload Characterization Approaches Ref. |  | Application
---
Architecture
Methods | Type | Infrastructure | Objectives | Advantages | Limitations
(fifer, ) | Serverless | LSTM |  | Time series
---
analysis
Single cloud |  | Request arrival rate
---
prediction
High prediction accuracy | Simplicity of application models
(htas, ) | Monolithic | K-means++ | Classification | Single cloud |  | Resource demand
---
prediction
High scalability |  | Limited accuracy under high load
---
variance
(sarsa, ) | Monolithic | ARIMA |  | Time series
---
analysis
Single cloud |  | Request arrival rate
---
prediction
Capability of large data scales | Inaccuracy under trend turning
(hybridarima, ) | Monolithic | ARIMA-TES |  | Time series
---
analysis
Single cloud |  | Multi-dimensional
---
workload prediction
| High robustness, accuracy,
---
and anti-interference
Higher time overhead
(hybridworkloadprediction, ) | Monolithic |  | LSTM,
---
Bi-LSTM
| Time series
---
analysis
Single cloud |  | Resource demand
---
prediction
High prediction accuracy |  | Lack of consideration for anomaly
---
detection
(gruesprediction, ) | Monolithic | GRU-ES |  | Time series
---
analysis
Single cloud |  | Resource demand
---
prediction
Low error rates |  | Ignorance of potential resource
---
allocation strategies based on
the prediction model
(video, ) | Microservice | TSNNR | Regression | Single cloud |  | Resource demand
---
prediction
Improved prediction accuracy | Limited workload scenarios
(highutiautoscaling, ) | Microservice | GBR | Regression | Single cloud |  | Resource demand
---
prediction
High prediction accuracy | Limited workload scenarios
(mlbasedautoscaling, ) | Microservice | LSTM |  | Time series
---
analysis
Single cloud |  | Request arrival rate
---
prediction
Low time overhead | High computational expense
(gruprediction, ) | Microservice | IGRU-SD |  | Time series
---
analysis
Hybrid cloud |  | Resource demand
---
prediction
Low error rates |  | Unclear demonstration of the
---
relationship between resource
allocation and energy efficiency
(aiautoscale, ) | Microservice |  | AR, LSTM,
---
HTM
| Time series
---
analysis
Single cloud |  | Request arrival rate
---
prediction
Multi-model optimization | High computational costs
(fisher, ) | Microservice | Bi-LSTM |  | Time series
---
analysis
Single cloud |  | Prediction of request
---
arrival rate and
application throughput
Improved prediction accuracy | Implicit time overhead analysis
(HANSEL, ) | Microservice | Bi-LSTM |  | Time series
---
analysis
Hybrid cloud |  | Task arrival rate
---
prediction
Performance improvement | Inaccuracy in long-term forecasts

The knowledge of workload characteristics and behavior patterns offers
important reference data for the estimation of resource provisioning and load
distribution. Table 2 shows the existing studies that leverage ML techniques
in workload characterization.

Many previous studies have tried to model the time series pattern of request
arrival rates in containerized applications through various algorithms. ARIMA,
as a classic algorithm for analyzing time series data, was utilised in
Reference (sarsa, ). Compared with other linear models such as autoregressive
(AR), moving average (MA), autoregressive moving average (ARMA), and
exponentially weighted moving average (EWMA), ARIMA enjoys high accuracy even
under unstable time series. As the workload scenario of ARIMA is limited to
linear models, it is usually referenced as a baseline approach by many studies
included in our literature review.

To further speed up the data training process, LSTM models are adopted in
References (fifer, ; mlbasedautoscaling, ; aiautoscale, ) to predict the
request arrival rates under large dynamic variations and avoid unnecessary
resource provisioning operations. Unlike general feedforward neural networks,
LSTM has feedback connections. It produces prediction results by analyzing the
whole data sequences and are more accurate at identifying new patterns.
However, LSTM models only train the data in one direction. Bi-LSTM models can
process the data sequence from both forward and backward directions.
Therefore, Bi-LSTM models are proposed in References (fisher, ; HANSEL, ) to
capture more key metrics and improve the prediction accuracy.

Another direction in ML-based workload characterization is resource demand
modelling and prediction. Zhong et al. (htas, ) leverage the K-means++
algorithm for task classification and identification based on the resource
usage (e.g., CPU and memory) patterns of different workloads. Zhang et al.
(video, ) introduce the Time Series Nearest Neighbor Regression (TSNNR)
algorithm for prediction of future workload resource requirements by matching
the recent time series data trend to similar historical data. To enhance the
ARIMA model in analysis of nonlinear relationship and trend turning points
within data sequences, Xie et al. (hybridarima, ) apply ARIMA with triple
exponential smoothing (ARIMA-TES) in the prediction of container workload
resource usage.

Besides, A hybrid association learning architecture is designed in Reference
(hybridworkloadprediction, ) through combining the LSTM and Bi-LSTM models. A
multi-layer structure is built to find the inter-dependencies and relationship
between various resource metrics, which are generally classified into three
distinct groups, including CPU, memory, and I/O. To further reduce the error
rates of resource usage prediction and data training time, GRU-based models
are utilised due to their high computational efficiency and prediction
accuracy. Lu et al. (gruprediction, ) develop a hybrid prediction framework
consisting of a GRU and a straggler detection model (IGRU-SD) to predict the
periodical resource demand patterns of cloud workloads on a long-term basis.
Likewise, Cheng et al. (gruesprediction, ) propose a GRU-ES model where the
exponential smoothing method is used to update the resource usage prediction
results generated by GRU and reduce prediction errors.

In summary, the majority of the reviewed literature in workload
characterization is focusing on the analysis and prediction of request arrival
rates and resource usage patterns through time series analysis or regression
models.

####  4.1.2. Performance Analysis

Table 3. Summary of Performance Analysis Approaches Ref. |  | Application
---
Architecture
Methods | Type | Infrastructure | Objectives | Advantages | Limitations
(functionserverless, ) | Serverless | LSTM, LR |  | Time series
---
analysis,
regression
Single cloud |  | Prediction of function
---
invoking time
| Online prediction of
---
function chains
| Additional resource consumption
---
for model training
(edgecloudserverless, ) | Serverless | GBR | Regression | Hybrid cloud |  | Prediction of costs and
---
end-to-end latency
High prediction accuracy | High computational expenses
(cose, ) | Serverless | BO | Regression | Hybrid cloud |  | Prediction of costs and
---
execution time based on
function configurations
High prediction accuracy | High computational complexity
(fitnessaware, ) | Monolithic | K-means, PR |  | Classification,
---
regression
Hybrid cloud |  | Classification of cloud
---
service providers and
container clusters
| High prediction accuracy
---
and effectiveness
Simplicity of application models
(sarsa, ) | Monolithic | ANN |  | Time series
---
analysis
Single cloud |  | Resource utilization and
---
response time prediction
Incremental modelling | Long model training time
(kuberknots, ) | Monolithic | ARIMA |  | Time series
---
analysis
Single cloud |  | GPU resource utilization
---
prediction
| Discovery of the peak of
---
resource consumption
| Implicit accuracy and time
---
overhead evaluation
(performancedocker, ) | Monolithic |  | SVR, LR,
---
ANN
Regression | Single cloud |  | Prediction of application
---
performance based on
resource metrics
Low error rates | High computational expenses
(miras, ) | Microservice | ANN | Regression | Single cloud |  | Performance modelling
---
of microservice workflow
systems
| Sample complexity
---
reduction
| Randomness due to boundary
---
effects
(loadbalancemicro, ) | Microservice | SVR, ANN | Regression | Multi-cloud |  | Performance modelling
---
of microservice response
time
| Time saving for route
---
searching
| Dependency on offline training
---
of historical data
(containeroffloading, ) | Microservice |  | LR, PR, RF,
---
SVR
Regression | Hybrid cloud |  | Prediction of offloading
---
execution time
High prediction accuracy | High computational costs
(Probabilistic, ) | Microservice | GP | Regression | Single cloud |  | Prediction of end-to-end
---
latency
High prediction accuracy | High computational complexity
(slocloudnative, ) | Microservice |  | LR, RF,
---
LASSO
Regression | Single cloud | Prediction of SLI |  | High prediction accuracy
---
by removing anomalies
High computational complexity
(sinan, ) | Microservice | CNN, BT |  | Time series
---
analysis,
regression
Single cloud |  | End-to-end latency and
---
QoS violations prediction
| Online prediction with high
---
resource efficiency
Overfitting and misprediction
(ssdidperformance, ) | Microservice |  | DT, MARS,
---
boosting, RF
Regression | Single cloud |  | Modelling and prediction
---
of SSD I/O performance
Short data training time | Simplicity of application models

Performance analysis captures the key infrastructure and application-level
metrics for evaluation of the overall system status or application
performance. A summary of the ML-based performance analysis approaches is
given in Table 3.

Das et al. (edgecloudserverless, ) design a performance modeller based on GBR
for predicting the costs and end-to-end latency of input serverless functions
based on their configurations under edge-cloud environments. Such metrics are
the key factors in the estimation of the efficiency of function scheduling
decisions. Similarly, Akhtar et al. (cose, ) leverage the Bayesian
Optimization (BO) function to achieve the same purpose. The prediction results
will be used to estimate the optimal function configurations that meet the
time constraints in function deployment with the lowest costs.

To estimate the cold start latency in serverless computing platforms, Xu et
al. (functionserverless, ) propose a two-phase approach. LSTM is used in the
first phase to predict the invoking time of the first function in a function
chain, while the rest of the functions is processed through LR. This two-phase
approach achieves a significant reduction of prediction error rates and
resource wasting in the container pool.

Venkateswaran and Sarkar(fitnessaware, ) try to classify the cloud service
providers and container cluster configurations under multi-cloud environments
through a two-level approach. K-means is employed in the first level to
precisely classify the comparable container cluster compositions by their
performance data, while PR is applied in the second level for analyzing the
relationship between container strength and container system performance.

Some research works have investigated the issue of infrastructure-level
resource utilization prediction (sarsa, ; kuberknots, ; ssdidperformance, ).
Zhange et al. (sarsa, ) leverage the ANN model to predict the CPU utilization
and request response time by modelling a series of metrics, including CPU and
memory usage, response time, and the request arrival rates mentioned in
Section 4.1.1. Besides, Dartois et al. (ssdidperformance, ) look into the
research topic of SSD I/O performance modelling and interference prevention
with a series of regression techniques, including boosting, RF, DT, and
Multivariate adaptive regression splines (MARS). As SSDs are prevalently used
by large-scale datacenter for data storage due to their high performance and
energy efficiency, their internal mechanisms could directly impact
application-level behaviors and cause potential SLO violations.

On the other hand, some previous studies focus on behavior modelling of
application-level metrics (slocloudnative, ; sinan, ; Probabilistic, ; miras,
). RScale (Probabilistic, ) is implemented as a robust scaling system
leveraging Gaussian process (GP) regression for investigation of the
interconnections between end-to-end tail latency of microservice workloads and
internal performance dependencies. Likewise, Sinan (sinan, ), as an ML-based
and QoS-aware cluster manager for containerized microservices, combines the
CNN and BT model for the prediction of end-to-end latency and QoS violations.
Taking both the microservice inter-dependencies and the time series pattern of
application-level metrics into account, Sinan evaluates the efficiency of
short-term resource allocation decisions as well as long-term application QoS
performance.

Overall, most studies in this section are concerned with prediction of time
constraints or resource usage patterns through regression or time series
analysis techniques.

####  4.1.3. Anomaly Detection

Table 4. Summary of Anomaly Detection Approaches Ref. |  | Application
---
Architecture
Methods | Type | Infrastructure | Objectives | Advantages | Limitations
(iotcontainer, ) | Monolithic |  | K-means, ensemble,
---
hierarchical
Classification | Hybrid cloud |  | Identification of overloaded
---
or underloaded nodes
| Short data training
---
time
| Limited workload scenarios,
---
high space complexity
(containerdetection, ) | Monolithic |  | K-means, KNN,
---
self-organizing map
Classification | Single cloud |  | Container vulnerability
---
detection
| High detection
---
accuracy
Insufficient anomaly cases
(containeranomalydetection, ) | Microservice | KNN, SVM, NB, RF | Classification | Single cloud |  | Anomaly detection of
---
microservices according to
real-time performance metrics
| Various monitoring
---
metrics
| Insufficient evaluation of
---
application-level anomalies
(denpendencyana, ) | Microservice | LSTM |  | Time series
---
analysis
Single cloud |  | Anomaly detection and
---
prediction on application
or infrastructure-level metrics
| Improved prediction
---
accuracy
Static models
(dockercontainer, ) | Monolithic | Isolation forest | Classification | Single cloud |  | Identification of abnormal
---
resource metrics
| Improved detection
---
accuracy
Unstable monitoring delays

Anomaly detection is a critical mechanism for identifying abnormal behaviors
in system states or application performance. Table 4 describes the reviewed
approaches regarding anomaly detection.

Due to the application-centric and decentralized features of containers, they
are more likely to experience a wide range of security threats, which may lead
to potential propagation delays in container image dependency management or
security attacks (dockerhub, ). Tunde-Onadele et al. (containerdetection, )
classify 28 container vulnerability scenarios into six categories and develop
a detection model, including both dynamic and static anomaly detection schemes
with KNN, K-means, self-organization map algorithms, which could reach
detection coverage up to 86%.

To balance the energy efficiency and resource utilization of a containerized
computing system under hybrid cloud environments, Chhikara et al.
(iotcontainer, ) employ K-means, Hierarchical clustering algorithms, and
ensemble learning for identification and classification of underloaded and
overloaded hosts. Further container migration operations will be conducted
between these two groups for load balancing and energy consumption reduction.

Shah et al. (denpendencyana, ) extend the LSTM model to analyze the long-term
dependencies of real-time performance metrics among microservice units.
Relying on the constructed model, they manage to identify critical performance
indicators that support anomaly detection of various application and
infrastructure-level metrics, including network throughput and CPU
utilization.

####  4.1.4. Dependency Analysis

Table 5. Summary of Dependency Analysis Approaches Ref. |  | Application
---
Architecture
Methods | Type | Infrastructure | Objectives | Advantages | Limitations
(loadbalancemicro, ) | Microservice | BO, GP | Regression | Multi-cloud |  | Discovery of optimal route
---
of individual microservice
Load balance |  | Poor performance under highly
---
dynamic environment
(firm, ) | Microservice | SVM | Classification | Single cloud |  | Identification of potential
---
heavy-loaded microservice
units
High accuracy |  | Implicit explanation of the time
---
overhead and computational costs
(sinan, ) | Microservice | CNN | Classification | Single cloud |  | Dependency analysis of
---
microservices between
pipelined-tiers
| Navigation of
---
system complexity
Overfitting and misprediction
(denpendencyana, ) | Microservice | LSTM |  | Time series
---
analysis
Single cloud |  | Dependency analysis
---
among microservice units
| Improved prediction
---
accuracy
Ignorance of dependency updates

Table 5 summarizes the recent studies in service dependency analysis of
containerized applications. As serverless function chains or workflows are
usually pre-defined by users (serverlessworkflow, ), current ML-based
dependency analysis solutions only focus on decomposing the internal
structures of microservice units and monitoring any dynamic structural
updates.

SVM is utilised by Qiu et al. (firm, ) to find the microservice units with
higher risks causing SLO violations through analysis of the performance
metrics related to the critical path (CP) of each individual microservice. CP
is defined as the longest path between the client request and the underlying
microservice in the execution history graph. Since CP can change dynamically
at runtime in response to potential resource contention or performance
interference, the SVM classifier is implemented with incremental learning for
dependency analysis in a dynamic and consistent manner.

Load balancing between microservices under the multi-cloud environment could
be rather complex, because of the unstable network latency, dynamic service
configuration, and fluctuating application workloads. To address this
challenge, Cui et al. (loadbalancemicro, ) leverage the BO search algorithm
with GP to produce the optimal load-balanced request chain for each
microservice unit. Then all the individual request chains are consolidated
into a tree structure dependency model that can be dynamically updated in case
of potential environmental changes.

###  4.2. Resource Provisioning

In this section, we discuss various resource provisioning techniques,
including scheduling, scaling, and migration.

####  4.2.1. Scheduling

Table 6. Summary of Scheduling Approaches Ref. |  | Application
---
Architecture
Methods |  | Task
---
Structure
Infrastructure | Objectives | Advantages | Limitations
(fifer, ) | Serverless | Heuristic |  | Multiple
---
independent
Single cloud |  | Resource utilization
---
improvement and
energy saving
| Reduction of cold start and
---
response latency
| Poor efficiency for tasks
---
with long lifetimes
(edgecloudserverless, ) | Serverless | Heuristic | Single | Hybrid cloud |  | Cost and latency
---
minimization
| Multi-objective task
---
placement
| Limited accuracy under
---
high load variance
(cose, ) | Serverless | Heuristic | Graph-based | Hybrid cloud | Cost minimization | SLA assurance |  | Simplicity of application
---
workloads
(htas, ) | Monolithic | Heuristic | Single | Single cloud |  | Resource utilization
---
optimization
Load balance | High scheduling delays
(fitnessaware, ) | Monolithic | Heuristic | Single | Hybrid cloud |  | Automated task
---
deployment
| Optimized container build
---
time and provisioning time
| High computational
---
expenses and time
overhead
(kuberknots, ) | Monolithic | Heuristic | Single | Single cloud |  | Resource utilization
---
and energy efficiency
optimization
QoS improvement |  | Insufficient analysis of
---
computational costs and
time complexity
(miras, ) | Microservice | Actor-Critic | Graph-based | Single cloud | Cost saving |  | Training time reduction
---
and accuracy improvement
| Limitation caused by
---
scarce data
(video, ) | Microservice | Heuristic | Graph-based | Single cloud |  | Resource utilization
---
optimization
Cost saving | High scheduling delays
(news, ) | Microservice |  | Q-Learning,
---
SARSA
Graph-based | Single cloud |  | Minimization of
---
task execution time
SLA assurance | Limited scalability
(jobplacementdmlc, ) | Microservice |  | Actor-Critic,
---
ANN
Single | Single cloud |  | Minimization of task
---
completion time
| Performance interference
---
awareness
| Implicit description of
---
the space and time
complexity

As shown in Table 6, various algorithms have been proposed to solve the
scheduling issue for improving system and application performance.

Most of the reviewed approaches follow a design pattern of combining ML-based
Workload Modellers or Performance Analyzers with a heuristic scheduling
Decision Maker, as discussed in Section 2.2.4. As the system complexity has
been navigated by prediction models, bin packing and approximation algorithms
such as best fit or least fit are commonly adopted to make scheduling
decisions with improved resource utilization and energy efficiency (htas, ;
video, ; edgecloudserverless, ; fifer, ; fitnessaware, ; cose, ). For example,
Venkateswaran and Sarkar (fitnessaware, ) manage to significantly reduce the
complexity of selecting the best-fit container system and cluster compositions
under multi-cloud environments, standing on their prediction model described
in Section 4.1.2. To mitigate the resource contention between co-located tasks
deployed at the same host, Thinakaran et al. (kuberknots, ) implement a
correlation-based scheduler for handling task co-location by measuring GPU
consumption correlation metrics, especially consecutive peak resource demand
patterns recognized by its ARIMA prediction model.

The rest of the studies choose RL models as the core component in their
scheduling engine. Orhean et al. (news, ) choose two classic model-free RL
models, namely Q-Learning and SARSA, to schedule a group of tasks in a DAG
structure. To reduce the overall DAG execution time, the internal tasks in a
DAG categorized by their key features and priorities are scheduled under
consideration of the dynamic cluster state and machine performance. To address
the limitation of high sample complexity of model-free RL approaches, Zhang et
al. (miras, ) attempt to handle scientific workflows under microservice
architectures through model-based RL. An ANN model is trained to emulate the
system behavior by identification of key performance metrics collected from
the microservice infrastructure, so that the synthetic interactions generated
by ANN could directly be involved in the policy training process with Actor-
Critical to generate scheduling decisions. In such a way, it simplifies the
system model and avoids the time consuming and computationally expensive
interactions within the real microservice environment.

In conclusion, heuristic or RL models are applied in most of the investigated
works for decision making in task scheduling, to achieve resource utilization
improvement and task completion time minimization.

####  4.2.2. Scaling

Table 7. Summary of Scaling Approaches Ref. |  | Application
---
Architecture
Methods | Mechanism | Infrastructure | Objectives | Advantages | Limitations
(functionserverless, ) | Serverless | Heuristic | Horizontal | Single cloud |  | Reduction of execution
---
latency and resource
consumption
Alleviation of cold starts |  | Instability under changing
---
workloads
(rlserverless, ) | Serverless | Q-Learning | Horizontal | Single cloud | SLA assurance |  | Reduction of time overhead
---
of cold starts
Simplicity of training model
(autopilot, ) | Monolithic | Ensemble | Hybrid | Single cloud | Resource saving |  | High resource efficiency
---
and reliability
Limited flexibility
(htas, ) | Monolithic | Heuristic | Horizontal | Single cloud |  | Resource utilization and SLA
---
assurance
High resource efficiency | Frequent cluster resizing
(sarsa, ) | Monolithic |  | SARSA,
---
Q-Learning
Horizontal | Single cloud |  | Resource utilization
---
optimization
SLA violation reduction | Inaccuracy due to cold starts
(hybridscaling, ) | Monolithic |  | Model-based
---
RL
Hybrid | Single cloud |  | Minimization of application
---
performance penalty,
adaption costs, and resource
usage
Improved training speed |  | Simplicity of application
---
models
(Geocontainer, ) | Monolithic |  | Model-based
---
RL, heuristic
Hybrid | Multi-cloud |  | Optimal application
---
performance and adaption
time
Cost saving |  | Simplicity of application
---
structures and QoS
requirements
(FScaler, ) | Monolithic | MDP, SARSA | Horizontal | Hybrid Cloud |  | Minimization of resource
---
consumption and response
time
High scalability | Limited dimensionality
(microsacler, ) | Microservice | BO, GP | Horizontal | Single cloud | SLA assurance |  | High precision and short
---
training time
| Poor performance and
---
sub-optimal decisions under
workload spikes
(mlmicro, ) | Microservice | RF | Horizontal | Single cloud | SLA assurance |  | Container expansion time
---
reduction
| Limited QoS and workload
---
scenarios
(negotiation, ) | Microservice | Q-Learning | Vertical | Single cloud |  | Optimization of resource
---
configurations and costs
SLA assurance | Limited workload scenarios
(highutiautoscaling, ) | Microservice | Heuristic | Horizontal | Single cloud | Cost minimization |  | resource utilization
---
improvement
High time complexity
(mlbasedautoscaling, ) | Microservice | Heuristic | Horizontal | Single cloud | Resource efficiency |  | Avoidance of oscillations
---
under unexpected workload
spikes
Rigid scaling mechanisms
(Probabilistic, ) | Microservice | Heuristic | Horizontal | Single cloud | SLA assurance |  | Resource utilization
---
optimization
High time complexity
(aiautoscale, ) | Microservice | Heuristic | Horizontal | Single cloud | SLA assurance | Lower request loss | High resource usage
(slocloudnative, ) | Microservice | Heuristic | Vertical | Single cloud | SLO assurance |  | Resource utilization
---
optimization
| Simplicity of application
---
datasets
(HANSEL, ) | Microservice | SARSA | Horizontal | Hybrid cloud |  | Resource utilization
---
optimization
SLA assurance |  | Capability limitation of edge
---
devices
(sinan, ) | Microservice | Heuristic | Horizontal | Single cloud | QoS assurance |  | Resource utilization
---
optimization
| Implicit evaluation of
---
computational costs and
time complexity
(firm, ) | Microservice | Actor-critic | Hybrid | Single cloud |  | Resource utilization
---
optimization
SLO violation mitigation |  | Limited scalability and
---
anomaly detection

Scaling could dynamically adjust the system states in response to the changing
workloads and cloud environments. The research works related to scaling are
given in Table 7.

To improve the decision quality and accuracy of RL-based autoscalers for
monolithic applications with SLA assurance, Zhang et al. (sarsa, ) build a
horizontal scaling approach through the SARSA algorithm, based on analysis of
the application workloads and CPU usage predicted by the ARIMA and ANN models.
For addressing the same research questions under fog-cloud environments, Sami
et al. (FScaler, ) simulate the horizontal scaling of containers as an MDP
model that considers both the changing workloads and free resource capacity in
fogs. Then SARSA is chosen for finding the optimal scaling strategy on top of
the MDP model through online training at a small data scale.

Further considering the possibility of hybrid scaling of monolithic
applications, Rossi et al. (hybridscaling, ) propose a model-based RL
approach, targeting to find a combination of horizontal and vertical scaling
decisions that meet the QoS requirements with the lowest adaption costs and
resource wastage. Furthermore, the authors extend this model with a network-
aware heuristic method for container placement in geographically distributed
clouds (Geocontainer, ). Although the approaches mentioned above can improve
resource utilization and QoS to a certain degree, their application workloads
and QoS scenarios are too simple, without enough consideration of the
diversity and complexity of cloud workloads.

Plenty of previous studies (highutiautoscaling, ; mlbasedautoscaling, ;
aiautoscale, ; Probabilistic, ; slocloudnative, ) have tried to leverage
heuristic methods for microservice scaling, assisted by ML-based workload
modelling and performance analysis. However, such approaches underestimate the
inter-dependencies between microservices that are updated dynamically. On the
other hand, model-based RL algorithms are usually unsuitable for microservice-
based applications for the same reason. As microservice dependencies could
potentially change at runtime, the simulation of state transitions could then
be invalid.

Therefore, model-free RL algorithms are more common in the application scaling
of microservices, as they do not rely on transition models. Qiu et al. (firm,
) implement an SLO violation alleviation mechanism using Actor-Critic to scale
the critical microservices detected by SVM as mentioned in Section 4.1.4. The
Actor-Critic model produces a horizontal scaling decision to minimize the SLO
violations through evaluating three crucial features, including SLO
maintenance ratio, workload changes, and request composition. To speed up the
model training process of RL methods, Yan et al. (HANSEL, ) design a multi-
agent parallel training model based on SARSA for horizontal scaling of
microservices under hybrid clouds. Assisted with the workload prediction
results generated by Bi-LSTM , their elastic scaling approach could make a
more accurate scaling decision of when, where, and how many microservice
instances should be scaled up/down. In such a way, it achieves significant
resource utilization improvement and cost reduction under SLA assurance.

Cold starts during the invocation of serverless functions are a serious
performance bottleneck of serverless computing platforms. Cold starts are
defined as the time overhead of environment setup and function initialization.
Xu et al. (functionserverless, ) present a container pool scaling strategy,
where function containers are pre-initialized by evaluating the first function
invocation time (predicted by LSTM as discussed in Section 4.1.2) and the
number of containers in each function category. Similarly, Agarwal et al.
(rlserverless, ) introduce a Q-Learning agent to summarize the function
invocation patterns and make the optimal scaling decisions of function
containers in advance. A series of metrics, including the number of available
function containers, per-container CPU utilization, and success/failure rates,
are selected to represent the system states in the Q-Learning model. Due to
the nature of model-free RL methods, prior information of the input function
is not necessary.

In summary, there has been a large body of literature in the area of scaling
using diverse solutions, covering all types of application architectures and
cloud infrastructures. The majority of the reviewed works are related to
autoscaling of microservice-based applications, with the model-free RL models
as the latest resolution to process the dynamically changing workloads and
microservice dependencies (negotiation, ; HANSEL, ; firm, ). Their common
targets mainly focus on SLA assurance and resource efficiency optimization.

####  4.2.3. Migration

Table 8. Summary of Migration Approaches Ref. |  | Application
---
Architecture
Methods | Mechanism | Infrastructure | Objectives | Advantages | Limitations
(htas, ) | Monolithic | Heuristic | Rescheduling | Single cloud | SLA assurance |  | Resource contention
---
alleviation
Long execution delays
(migrationfog, ) | Monolithic |  | MDP, DNN,
---
Q-Learning
Offloading | Hybrid cloud |  | Reduction of communication
---
delays, power consumption
Low migration costs | High space complexity
(iotcontainer, ) | Monolithic | Heuristic | Offloading | Hybrid cloud | Energy efficiency improvement | Load balance |  | High time and space
---
complexity
(video, ) | Microservice | Heuristic | Rescheduling | Single cloud | Resource utilization optimization | Cost saving | SLA violations
(microservicecoordination, ) | Microservice |  | Q-Learning,
---
MDP
Offloading | Hybrid cloud |  | Reduction of service delays and
---
migration cost
Optimized performance |  | Lack of consideration on
---
load balancing

As depicted in Table 8, Migration is a complementary mechanism for load
balancing across cloud environments.

Some reviewed approaches manage to bind ML-based behavior models with
heuristic migration algorithms. Zhong et al. (htas, ) develop a least-fit
rescheduling algorithm to evict and relocate a set of lower-priority
containers with the least QoS impact when unexpected resource contention
occurs between co-located containers. The rescheduling algorithm readjusts the
container configuration based on runtime performance metrics and selects the
node with the most available resources evaluated by K-means for relocation.
Besides, Chhikara et al. (iotcontainer, ) introduce an energy-efficient
offloading model with a set of heuristic methods, including random placement,
first-fit, best-fit, and correlation threshold-based placement algorithms. It
is aimed at resource load balancing under hybrid cloud environments by
migrating containers from overloaded nodes to underloaded nodes that are
identified through classification as described in Section 4.1.3. However,
these approaches may come with high execution delays in large-scale computing
systems.

A fog-cloud container offloading prototype system is presented by Tang et al.
(migrationfog, ). The container offloading process is considered as a multi-
dimensional MDP model. To reduce the network delays and computation costs
under potentially unstable environments, the deep Q-Learning algorithm
combines the deep neural network (DNN) and Q-Learning model to quickly produce
an efficient offloading plan. Wang et al. (microservicecoordination, ) further
extends the combination of MDP and Q-Learning in the context of microservice
coordination under edge-cloud environments. The process of microservice
coordination is assumed as a sequential decision scheme and formulated as an
MDP model. On top of the MDP model, Q-Learning is used to find the optimal
solution for service migration or offloading, in light of long-term
performance metrics, including overall migration delays and costs.

###  4.3. Summary

In this section, we summarize the discussed researches based on three
important categories, namely application architecture, infrastructure, and
hybrid machine learning model.

####  4.3.1. Application Architecture

Figure 8. Study Distribution of Application Architectures

Application Archi

As described in Fig. 8, most of the reviewed papers (68%) are concerned with
modelling and management of microservice-based applications. In light of the
dynamic and decentralized nature of microservices, diverse ML algorithms have
been investigated for capturing the key characteristic of microservice
workloads, performance, and inter-dependencies, such as BGR, LSTM, GRU, SVM,
GP, ANN, CNN, and DT (highutiautoscaling, ; mlbasedautoscaling, ;
gruprediction, ; miras, ; loadbalancemicro, ; sinan, ; denpendencyana, ). As
the inter-dependencies of microservice units could dynamically update at
runtime, the key concern of microservice resource provisioning is the chain
reactions caused by microservice unit scaling operations. By combining ML-
based behavior models with heuristic or model-free RL methods for scaling, the
most recent studies manage to achieve high resource utilization optimization,
cost reduction, and SLA assurance (highutiautoscaling, ; mlbasedautoscaling, ;
aiautoscale, ; Probabilistic, ; slocloudnative, ; firm, ; HANSEL, ).

The research works related to the orchestration of single-component
containerized applications hold a study distribution of 22%. As the workload
scenarios of these applications are relatively simple, their core drive of
workload/behavior modelling is to predict their resource demands under certain
QoS requirements. Assisted with such prediction results, heuristic and RL
methods (both model-free and model-based) are adopted for optimization of the
resource allocation process regarding cost, energy, and resource efficiency
(htas, ; sarsa, ; hybridarima, ; hybridworkloadprediction, ; fitnessaware, ;
kuberknots, ; performancedocker, ).

Although the serverless architecture is currently having the lowest study
distribution (10%), it is enjoying a growing popularity and becoming a
prevalent application architecture in cloud computing. Most of the existing
ML-based researches in this field are trying to alleviate the performance
downgrade caused by function cold starts. LSTM and GBR models have been
employed to estimate the function invocation time in serverless function
chains, while the allocation of functions and scaling of function containers
are mainly solved by heuristic methods and Q-Learning (edgecloudserverless, ;
functionserverless, ; fifer, ; rlserverless, ).

####  4.3.2. Infrastructure

Figure 9. Study Distribution of Cloud Infrastructures

Cloud Infra

The majority of the included researches (71%) only consider application
deployment under single cloud environments as demonstrated in Fig. 9. Since
single cloud environments are easier to manage, it is not necessary to raise
the system complexity for most applications. Only 5% of studies have attempted
behavior modelling and scaling under geographic-distributed multi-cloud
environments (loadbalancemicro, ; Geocontainer, ), because the interactions
under such context are usually time-consuming and computation-intensive.

As traditional cloud computing commonly causes significant propagation delays,
bandwidth and energy consumption by hosting all the applications and data in
cloud servers, edge and fog computing is emerging as mainstream computing
paradigms. Therefore, the latest studies have investigated the possibility of
container orchestration under hybrid cloud environments, where the crucial
research question is to decide where to host the containerized applications
among cloud/edge devices and the cloud. The proposed solutions for scheduling
and offloading of containerized applications under hybrid clouds, including
MDP, RL, DRL and heuristic methods (edgecloudserverless, ; fitnessaware, ;
FScaler, ; HANSEL, ; migrationfog, ; iotcontainer, ), aim to reduce end-to-end
latency and optimize resource/energy efficiency.

####  4.3.3. Hybrid Machine Learning Model

Figure 10. Study Distribution of Machine Learning Models between 2016 and 2021

Machine Learning Model

Fig. 10 shows the study distribution of ML models between 2016 and 2021, where
we can observe a rising trend of hybrid ML models. A single ML model
consisting of merely one ML algorithm is designed to solve a specific
container orchestration problem, either data analysis or resource
provisioning. By 2018, only single ML models had been adopted in this field.
As the internal structures of containerized applications like microservices
are becoming rather complex and dynamic with continuously growing demands of
higher modelling/prediction accuracy and lower response time, this requires ML
models to be more robust and efficient with even lower error rates,
computation costs, and data training time. Therefore, most studies have been
investigating hybrid ML models composed of a mixture of multiple ML algorithms
to solve one or more orchestration problems from 2019 (hybridarima, ;
hybridworkloadprediction, ; gruesprediction, ; gruprediction, ; microsacler, ;
migrationfog, ; microservicecoordination, ).

##  5\. Future Directions

Although the existing studies have covered diverse orchestration schemes,
application architectures, and cloud infrastructures, some research gaps and
challenges are only partially addressed. In this section, we discuss a series
of open research opportunities and potential future directions :

  1. (1)

Workload Distribution in Microservices. The current workload characterization
methods are mainly focusing on modelling and prediction of the request arrival
rates and resource usage patterns. Very few works try to address the issue of
workload distribution across microservice units. The changing workload
distribution on individual microservices could potentially cause a chain
reaction and further impact the overall application performance. Therefore,
how to simulate and standardize workload distribution between microservices
for load balance and performance optimization remains an unsolved research
question.

  2. (2)

Microservice Dependency Analysis. The dynamic inter-dependencies between
microservices is a crucial part of application complexity navigation. Though
some works have attempted to predict application performance or identity
critical microservice units through dependency analysis, there is no existing
solution to explicitly address the relationship between the status of
individual microservices and overall application performance metrics. Such
analysis models are necessary for scheduling and scaling microservice-based
applications for performance optimization and SLA assurance.

  3. (3)

Systematic Anomaly Detection and Recovery. The current ML-based anomaly
detection methods for containerized applications are mostly based on
resource/performance metrics or security threats. There is a need for a
systematic approach for anomaly detection, root cause categorization, and
recovery in a timely and accurate fashion, under different application
architectures and cloud infrastructures.

  4. (4)

Graph-based Task Scheduling. Batch processing jobs consisting of a group of
dependent tasks in DAG structures are common in containerized applications,
but the literature related to the scheduling problem of such applications is
very limited. Some previous studies manage to resolve this problem under a
simplified condition of homogeneous job structures where each task in a DAG is
configured similarly with the same execution time. This kind of assumptions
are rather unrealistic considering the complex and heterogeneous nature of DAG
jobs. A sophisticated scheduling strategy should not only consider the overall
DAG structure and different task configurations (e.g., resource demands,
execution time, and replica sizes), but also the runtime performance metrics
and fault tolerance.

  5. (5)

Management of Function Chains in Serverless Architectures. The latest ML-based
studies on serverless architectures are all related to the alleviation of
function cold starts, with many other research directions left to be
discovered. As functions are submitted without any prior application-level
knowledge, it is tricky to implement a robust and efficient solution for
workload classification, resource demand estimation, and autoscaling. How to
optimize the invocation of function chains in an SLO and cost-aware manner is
also a crucial research question, especially under hybrid cloud environments.

  6. (6)

Microservices in Hybrid Clouds. Under the emerging trend of edge and fog
computing, the most recent studies have made efforts to move microservices to
edge/fog devices for communication delays reduction and cost saving. However,
the extremely heterogeneous and geographically distributed computation
resources within hybrid clouds significantly raise the complexity of
application and resource management. A resource provisioning solution under
such environments must consider the long-term impacts of each microservice
placement or adjustment decision, regarding multi-dimensional optimization
objectives, such as costs, energy, and end-to-end latency. Therefore, there is
great potential for ML-based solutions to simulate the process of scheduling,
scaling, and offloading for microservices under hybrid clouds.

  7. (7)

Energy-aware Resource Management. Through combing ML-based workload
characterization and performance analysis models, the overall energy
consumption of a system could be precisely predicted based on the resource
utilization of each PM. Accordingly, these insights grant us more options for
developing energy-aware orchestration frameworks. For instance, brownout
technologies could be utilised to activate/deactivate optional microservices
for energy efficiency optimization, according to the predicted trends in
resource utilization and SLA violations (brownout, ). Besides, resource
provisioning of containerized applications could also be conducted in a load
balancing way to reduce energy consumption.

  8. (8)

Multi-dimensional Performance Optimization. One of the essential optimization
problems in container orchestration is to manage the trade-off between
different performance metrics, especially SLA violations and financial costs.
Though some previous studies (fifer, ; edgecloudserverless, ; kuberknots, )
address this issue to a certain degree by balancing several optimization
objectives during resource provisioning, a standard performance analysis
benchmark should be designed to accurately decompose the relation between a
set of pre-defined key performance metrics. Such knowledge could be further
applied in resource provisioning to produce optimal orchestration decisions
with multiple performance requirements taken into account.

  9. (9)

Fully Integrated ML-based Optimization Engine. Considering the long data
training time and large data volumes requested by ML models, a partially
integrated ML engine enjoys high popularity, where ML models are only utilised
for behavior modelling to assist the original Container Orchestrator in online
resource provisioning. Following the reference architecture of ML-based
optimization engine in Section 2.2.4, a fully integrated ML engine should be
capable of combining multiple ML models for both offline training of behavior
models and fast online decision making.

  10. (10)

Edge Intelligence. Due to the extreme data scales and geographical
distribution of edge devices, data transition to a centralized ML-based
Optimization Engine across edge networks is potentially time-consuming and
expensive. Hence, the next-generation edge intelligence agents should follow a
decentralized architecture where each independent agent is deployed at a
distributed location close to end users for data collection, analytical model
construction, and maintenance of serverless or microservice-based applications
under low propagation delays.

##  6\. Conclusion

In this work, we have conducted an extensive literature review on how
container orchestration is managed using diverse machine learning-based
approaches in the state-of-the-art studies, with emphasis on the specific
application architectures and cloud infrastructures. The last few years have
witnessed a continuously growing trend of the adoption of machine learning
methods in containerized application management systems for behavior
modelling, as well as resource provisioning of complicated decentralized
applications. Compared with most traditional heuristic methods, ML-based
models could produce more accurate orchestration decisions with shorter
computation delays, under complex and dynamic cloud environments consisting of
highly heterogeneous and geographically distributed computation resources.

We have observed machine learning-based methods applied under a wide range of
scenarios, but there is no systematic approach to build a complete machine
learning-based optimization framework in charge of the whole orchestration
process. Under the emerging trend of moving microservice and serverless
applications to hybrid clouds, such frameworks are necessary to handle the
sustainably growing system complexity and balance multi-dimensional
optimization objectives.

## References

  * [1] Qi Zhang, Lu Cheng, and Raouf Boutaba.  Cloud computing: state-of-the-art and research challenges.  Journal of internet services and applications, 1(1):7–18, 2010\.
  * [2] Saakshi Narula, Arushi Jain, and Prachi.  Cloud computing security: Amazon web service.  In Proceedings of the 2015 International Conference on Advanced Computing Communication Technologies, pages 501–505, 2015.
  * [3] Abhishek Verma, Luis Pedrosa, Madhukar Korupolu, David Oppenheimer, Eric Tune, and John Wilkes.  Large-scale cluster management at google with borg.  In Proceedings of the 2015 European Conference on Computer Systems, pages 1–18, 2015.
  * [4] Qixiao Liu and Zhibin Yu.  The elasticity and plasticity in semi-containerized co-locating cloud workload: A view from alibaba trace.  In Proceedings of the 2018 ACM Symposium on Cloud Computing, pages 347–360, 2018.
  * [5] Emiliano Casalicchio and Stefano Iannucci.  The state-of-the-art in container technologies: Application, orchestration and security.  Concurrency and Computation: Practice and Experience, 32(17):1–21, 2020.
  * [6] Zhiheng Zhong and Rajkumar Buyya.  A cost-efficient container orchestration strategy in kubernetes-based cloud computing infrastructures with heterogeneous resources.  ACM Transactions on Internet Technology, 20(2):1–24, 2020.
  * [7] Weisong Shi, Jie Cao, Quan Zhang, Youhuizi Li, and Lanyu Xu.  Edge computing: Vision and challenges.  IEEE Internet of Things Journal, 3(5):637–646, 2016.
  * [8] Zhiheng Zhong, Jiabo He, Maria A. Rodriguez, Sarah Erfani, Ramamohanarao Kotagiri, and Rajkumar Buyya.  Heterogeneous task co-location in containerized cloud computing environments.  In Proceedings of the 2020 IEEE International Symposium on Real-Time Distributed Computing (ISORC), pages 79–88, 2020.
  * [9] Christina Terese Joseph and K. Chandrasekaran.  Straddling the cstraddling the crevasse: A review of microservice software architecture foundations and recent advancementsrevasse: A review of microservice software architecture foundations and recent advancements.  Software: Practice and Experience, 49(10):1448–1484, 2019.
  * [10] Maria A. Rodriguez and Rajkumar Buyya.  Container‐based cluster orchestration systems: A taxonomy and future directions.  Software: Practice and Experience, 49(5):698–719, 2019.
  * [11] Charles Reiss, Alexey Tumanov, Gregory R. Ganger, Randy H. Katz, and Michael A. Kozuch.  Heterogeneity and dynamicity of clouds at scale: Google trace analysis.  In Proceedings of the 2012 ACM Symposium on Cloud Computing, pages 1–13, 2012.
  * [12] Andrew Chung, Jun Woo Park, and Gregory R. Ganger.  Stratus: Cost-aware container scheduling in the public cloud.  In Proceedings of the 2018 ACM Symposium on Cloud Computing, page 121–134, 2018.
  * [13] Sreekrishnan Venkateswaran and Santonu Sarkar.  Fitness-aware containerization service leveraging machine learning.  IEEE Transactions on Services Computing, 1(1):1–14, 2019.
  * [14] Tarek Menouer, Otman Manad, Christophe Cérin, and Patrice Darmon.  Power efficiency containers scheduling approach based on machine learning technique for cloud computing environment.  In Proceedings of the 2019 International Symposium on Pervasive Systems, Algorithms and Networks, pages 193–206, 2019.
  * [15] Haitao Zhang, Huadong Ma, Guangping Fu, Xianda Yang, Zhe Jiang, and Yangyang Gao.  Container based video surveillance cloud service with fine-grained resource provisioning.  In Proceedings of the 2016 IEEE International Conference on Cloud Computing (CLOUD), pages 758–765, 2016.
  * [16] Maryam Barshan, Hendrik Moens, Steven Latre, Bruno Volckaert, and Filip De Turck.  Algorithms for network-aware application component placement for cloud resource allocation.  Journal of Communications and Networks, 19(5):493–508, 2017.
  * [17] Josep Ll. Berral, Ricard Gavalda, and Jordi Torres.  Adaptive scheduling on power-aware managed data-centers using machine learning.  In Proceedings of the 2011 IEEE/ACM International Conference on Grid Computing, pages 66–73, 2011.
  * [18] William Trneberg, Amardeep Mehta, Eddie Wadbro, Johan Tordsson, Johan Eker, Maria Kihl, and Erik Elmroth.  Dynamic application placement in the mobile cloud network.  Future Generation Computer Systems, 70(1):163–177, 2017.
  * [19] Shiqiang Wang, Murtaza Zafer, and Kin K. Leung.  Online placement of multi-component applications in edge computing environments.  IEEE Access, 5(1):2514–2533, 2017.
  * [20] Yanqi Zhang, Weizhe Hua, Zhuangzhuang Zhou, G. Edward Suh, and Christina Delimitrou.  Sinan: Ml-based and qos-aware resource management for cloud microservices.  In Proceedings of the 2021 ACM International Conference on Architectural Support for Programming Languages and Operating Systems, page 167–181, 2021.
  * [21] Haoran Qiu, Subho S. Banerjee, Saurabh Jha, Zbigniew T. Kalbarczyk, and Ravishankar K. Iyer.  FIRM: An intelligent fine-grained resource management framework for slo-oriented microservices.  In Proceedings of the 2020 USENIX Symposium on Operating Systems Design and Implementation (OSDI 20), pages 805–825, 2020.
  * [22] Alexandru Iulian Orhean, Florin Pop, and Ioan Raicu.  New scheduling approach using reinforcement learning for heterogeneous distributed systems.  Journal of Parallel and Distributed Computing, 117(1):292–302, 2018\.
  * [23] Yixin Bao, Yanghua Peng, and Chuan Wu.  Deep learning-based job placement in distributed machine learning clusters.  In Proceedings of the 2019 IEEE INFOCOM - IEEE Conference on Computer Communications, pages 505–513, 2019.
  * [24] Ricardo Bianchini, Marcus Fontoura, Eli Cortez, Anand Bonde, Alexandre Muzio, Ana-Maria Constantin, Thomas Moscibroda, Gabriel Magalhaes, Girish Bablani, and Mark Russinovich.  Toward ml-centric cloud platforms.  Communications of the ACM, 63(2):50–59, 2020.
  * [25] Shubo Zhang, Tianyang Wu, Maolin Pan, Chaomeng Zhang, and Yang Yu.  A-sarsa: A predictive container auto-scaling algorithm based on reinforcement learning.  In Proceedings of the 2020 IEEE International Conference on Web Services (ICWS), pages 489–497, 2020.
  * [26] Yu Xu, Jianguo Yao, Hans-Arno Jacobsen, and Haibing Guan.  Cost-efficient negotiation over multiple resources with reinforcement learning.  In Proceedings of the 2017 IEEE/ACM International Symposium on Quality of Service (IWQoS), pages 1–6, 2017.
  * [27] Hani Sami, Azzam Mourad, Hadi Otrok, and Jamal Bentahar.  Fscaler: Automatic resource scaling of containers in fog clusters using reinforcement learning.  In Proceedings of the 2020 International Wireless Communications and Mobile Computing (IWCMC), pages 1824–1829, 2020.
  * [28] Zhiqing Tang, Xiaojie Zhou, Fuming Zhang, Weijia Jia, and Wei Zhao.  Migration modeling and learning algorithms for containers in fog computing.  IEEE Transactions on Services Computing, 12(5):712–725, 2019.
  * [29] Denis Weerasiri, Moshe Chai Barukh, Boualem Benatallah, Quan Z. Sheng, and Rajiv Ranjan.  A taxonomy and survey of cloud resource orchestration techniques.  ACM Computing Surveys, 50(2):1–41, 2017.
  * [30] Minxian Xu and Rajkumar Buyya.  Brownout approach for adaptive management of resources and applications in cloud computing systems: A taxonomy and future directions.  ACM Computing Surveys, 52(1):1–27, 2019.
  * [31] Thang Le Duc, Rafael García Leiva, Paolo Casari, and Per-Olov Östberg.  Machine learning methods for reliable resource provisioning in edge-cloud computing: A survey.  ACM Computing Surveys, 52(5):1–39, 2019.
  * [32] Sukhpal Singh and Inderveer Chana.  Qos-aware autonomic resource management in cloud computing: A systematic review.  ACM Computing Surveys, 48(3):1–46, 2015.
  * [33] Claus Pahl, Antonio Brogi, Jacopo Soldani, and Pooyan Jamshidi.  Cloud container technologies: a state-of-the-art review.  IEEE Transactions on Cloud Computing, 7(3):677–692, 2019.
  * [34] A. Brnabic and L. M. Hess.  Systematic literature review of machine learning methods used in the analysis of real-world data for patient-provider decision making.  BMC Medical Informatics and Decision Making, 21(1):1–19, 2021.
  * [35] Djamel Djenouri, Roufaida Laidi, Youcef Djenouri, and Ilangko Balasingham.  Machine learning for smart building applications: Review and taxonomy.  ACM Computing Surveys, 52(2):1–36, 2019.
  * [36] Pedro Henriques Abreu, Miriam Seoane Santos, Miguel Henriques Abreu, Bruno Andrade, and Daniel Castro Silva.  Predicting breast cancer recurrence using machine learning techniques: A systematic review.  ACM Computing Surveys, 49(3):1–40, 2016.
  * [37] Samira Pouyanfar, Saad Sadiq, Yilin Yan, Haiman Tian, Yudong Tao, Maria Presa Reyes, Mei-Ling Shyu, Shu-Ching Chen, and S. S. Iyengar.  A survey on deep learning: Algorithms, techniques, and applications.  ACM Computing Surveys, 51(5):1–36, 2018.
  * [38] Prateek Chhikara, Rajkumar Tekchandani, Neeraj Kumar, and Mohammad S. Obaidat.  An efficient container management scheme for resource constrained intelligent iot devices.  IEEE Internet of Things Journal, 1(1):1–13, 2020.
  * [39] Qingfeng Du, Tiandi Xie, and Yu He.  Anomaly detection and diagnosis for container-based microservices with performance monitoring.  In Proceedings of the 2018 International Conference on Algorithms and Architectures for Parallel Processing, pages 560–572, 2018.
  * [40] Kok-Lim Alvin Yau, Junaid Qadir, Hooi Ling Khoo, Mee Hong Ling, and Peter Komisarczuk.  A survey on reinforcement learning models and algorithms for traffic signal control.  ACM Computing Surveys, 50(3):1–38, 2017.
  * [41] Kejiang Ye, Yanmin Kou, Chengzhi Lu, Yang Wang, and Cheng-Zhong Xu.  Modeling application performance in docker containers using machine learning techniques.  In Proceedings of the 2018 IEEE International Conference on Parallel and Distributed Systems (ICPADS), pages 1–6, 2018.
  * [42] David Carrera, Malgorzata Steinder, Ian Whalley, Jordi Torres, and Eduard Ayguade.  Autonomic placement of mixed batch and transactional workloads.  IEEE Transactions on Parallel and Distributed Systems, 23(2):219–231, 2012.
  * [43] Rasmus V Rasmussen and Michael A Trick.  Round robin scheduling–a survey.  European Journal of Operational Research, 188(3):617–636, 2008\.
  * [44] Rodrigo Roman, Javier Lopez, and Masahiro Mambo.  Mobile edge computing, fog et al.: A survey and analysis of security threats and challenges.  Future Generation Computer Systems, 78(1):680–698, 2018.
  * [45] Mithun Mukherjee, Lei Shu, and Di Wang.  Survey of fog computing: Fundamental, network applications, and research challenges.  IEEE Communications Surveys Tutorials, 20(3):1826–1857, 2018.
  * [46] Ricardo Bianchini, Marcus Fontoura, Eli Cortez, Anand Bonde, Alexandre Muzio, Ana-Maria Constantin, Thomas Moscibroda, Gabriel Magalhaes, Girish Bablani, and Mark Russinovich.  Toward ml-centric cloud platforms.  Communications of the ACM, 63(2):50–59, 2020.
  * [47] Mohsen Mosleh, Kia Dalili, and Babak Heydari.  Distributed or monolithic? a computational architecture decision framework.  IEEE Systems Journal, 12(1):125–136, 2018.
  * [48] Zhongshan Ren, Wei Wang, Guoquan Wu, Chushu Gao, Wei Chen, Jun Wei, and Tao Huang.  Migrating web applications from monolithic structure to microservices architecture.  In Proceedings of the 2018 Asia-Pacific Symposium on Internetware, pages 1–10, 2018.
  * [49] Gabor Kecskemeti, Attila Csaba Marosi, and Attila Kertesz.  The entice approach to decompose monolithic services into microservices.  In Proceedings of the 2016 International Conference on High Performance Computing Simulation (HPCS), pages 591–596, 2016.
  * [50] Mariam Kiran, Peter Murphy, Inder Monga, Jon Dugan, and Sartaj Singh Baveja.  Lambda architecture for cost-effective batch and speed big data processing.  In Proceedings of the 2015 IEEE International Conference on Big Data (Big Data), pages 2785–2792, 2015.
  * [51] Theo Lynn, Pierangelo Rosati, Arnaud Lejeune, and Vincent Emeakaroha.  A preliminary review of enterprise serverless cloud computing (function-as-a-service) platforms.  In Proceedings of the 2017 IEEE International Conference on Cloud Computing Technology and Science (CloudCom), pages 162–169, 2017.
  * [52] Nilanjan Daw, Umesh Bellur, and Purushottam Kulkarni.  Xanadu: Mitigating cascading cold starts in serverless function chain deployments.  In Proceedings of the 2020 International Middleware Conference, page 356–370, 2020.
  * [53] Dong Du, Tianyi Yu, Yubin Xia, Binyu Zang, Guanglu Yan, Chenggang Qin, Qixuan Wu, and Haibo Chen.  Catalyzer: Sub-millisecond startup for serverless computing with initialization-less booting.  In Proceedings of the 2020 International Conference on Architectural Support for Programming Languages and Operating Systems, page 467–481, 2020.
  * [54] Alexandru Agache, Marc Brooker, Alexandra Iordache, Anthony Liguori, Rolf Neugebauer, Phil Piwonka, and Diana-Maria Popa.  Firecracker: Lightweight virtualization for serverless applications.  In Proceedings of the 17th $\\{$usenix$\\}$ symposium on networked systems design and implementation ($\\{$nsdi$\\}$ 20), pages 419–434, 2020.
  * [55] Jashwant Raj Gunasekaran, Prashanth Thinakaran, Nachiappan C. Nachiappan, Mahmut Taylan Kandemir, and Chita R. Das.  Fifer: Tackling resource underutilization in the serverless era.  In Proceedings of the 2020 International Middleware Conference, page 280–295, 2020.
  * [56] Zhengjun Xu, Haitao Zhang, Xin Geng, Qiong Wu, and Huadong Ma.  Adaptive function launching acceleration in serverless computing platforms.  In Proceedings of the 2019 IEEE International Conference on Parallel and Distributed Systems (ICPADS), pages 9–16, 2019.
  * [57] Siddharth Agarwal, Maria Alejandra Rodriguez, and Rajkumar Buyya.  A reinforcement learning approach to reduce serverless function cold start frequency.  In Proceedings of the 2021 IEEE/ACM International Symposium on Cluster, Cloud, and Internet Computing (CCGrid), pages 797–803, 2021.
  * [58] Toni Mastelic, Ariel Oleksiak, Holger Claussen, Ivona Brandic, Jean-Marc Pierson, and Athanasios V. Vasilakos.  Cloud computing: Survey on energy efficiency.  ACM Computing Surveys, 47(2):1–36, 2014.
  * [59] Prashanth Thinakaran, Jashwant Raj Gunasekaran, Bikash Sharma, Mahmut Taylan Kandemir, Chita R. Das.  Kube-knots: Resource harvesting through dynamic container orchestration in gpu-based datacenters.  In Proceedings of the 2019 IEEE International Conference on Cluster Computing (CLUSTER), pages 1–13, 2019.
  * [60] Minxian Xu, Adel N. Toosi, and Rajkumar Buyya.  A self-adaptive approach for managing applications and harnessing renewable energy for sustainable cloud computing.  IEEE Transactions on Sustainable Computing, 1(1):1–15, 2020.
  * [61] Guangba Yu, Pengfei Chen, and Zibin Zheng.  Microscaler: Automatic scaling for microservices with an online learning approach.  In Proceedings of the 2019 IEEE International Conference on Web Services (ICWS), pages 68–75, 2019.
  * [62] Peng Kang and Palden Lama.  Robust resource scaling of containerized microservices with probabilistic machine learning.  In Proceedings of the 2020 IEEE/ACM International Conference on Utility and Cloud Computing (UCC), pages 122–131, 2020.
  * [63] Laszlo Toka, Gergely Dobreff, Balazs Fodor, and Balazs Sonkoly.  Adaptive ai-based auto-scaling for kubernetes.  In Proceedings of the 2020 IEEE/ACM International Symposium on Cluster, Cloud and Internet Computing (CCGRID), pages 599–608, 2020.
  * [64] Yang Meng, Ruonan Rao, Xin Zhang, and Pei Hong.  Crupa: A container resource utilization prediction algorithm for auto-scaling based on time series analysis.  In Proceedings of the 2016 International Conference on Progress in Informatics and Computing (PIC), pages 468–472, 2016.
  * [65] Syed Yousaf Shah, Zengwen Yuan, Songwu Lu, and Petros Zerfos.  Dependency analysis of cloud applications for performance monitoring using recurrent neural networks.  In Proceedings of the 2017 IEEE International Conference on Big Data (Big Data), pages 1534–1543, 2017.
  * [66] Yi-Lin Cheng, Ching-Chi Lin, Pangfeng Liu, and Jan-Jan Wu.  High resource utilization auto-scaling algorithms for heterogeneous container configurations.  In Proceedings of the 2017 IEEE International Conference on Parallel and Distributed Systems (ICPADS), pages 143–150, 2017.
  * [67] Xuehai Tang, Qiuyang Liu, Yangchen Dong, Jizhong Han, and Zhiyuan Zhang.  Fisher: An efficient container load prediction model with deep neural network in clouds.  In Proceedings of the 2018 IEEE Intl Conf on Parallel Distributed Processing with Applications, Ubiquitous Computing Communications, Big Data Cloud Computing, Social Computing Networking, Sustainable Computing Communications (ISPA/IUCC/BDCloud/SocialCom/SustainCom), pages 199–206, 2018.
  * [68] Yuming Cheng, Chao Wang, Huihuang Yu, Yahui Hu, and Xuehai Zhou.  Gru-es: Resource usage prediction of cloud workloads using a novel hybrid method.  In Proceedings of the 2019 IEEE International Conference on High Performance Computing and Communications; the 2019 IEEE International Conference on Smart City; the 2019 IEEE International Conference on Data Science and Systems (HPCC/SmartCity/DSS), pages 1249–1256, 2019.
  * [69] Rui Fu, Zuo Zhang, and Li Li.  Using lstm and gru neural network methods for traffic flow prediction.  In Proceedings of the 2016 Youth Academic Annual Conference of Chinese Association of Automation (YAC), pages 324–328, 2016.
  * [70] Vladimir Podolskiy, Michael Mayo, Abigail Koay, Michael Gerndt, and Panos Patros.  Maintaining slos of cloud-native applications via self-adaptive resource sharing.  In Proceedings of the 2019 IEEE International Conference on Self-Adaptive and Self-Organizing Systems (SASO), pages 72–81, 2019.
  * [71] Jean-Emile Dartois, Jalil Boukhobza, Anas Knefati, and Olivier Barais.  Investigating machine learning algorithms for modeling ssd i/o performance for container-based virtualization.  IEEE Transactions on Cloud Computing, 1(1):1–14, 2019.
  * [72] Fabiana Rossi, Matteo Nardelli, and Valeria Cardellini.  Horizontal and vertical scaling of container-based applications using reinforcement learning.  In Proceedings of the 2019 IEEE International Conference on Cloud Computing (CLOUD), pages 329–338, 2019.
  * [73] Fabiana Rossi, Valeria Cardellini, and Francesco Lo Presti.  Elastic deployment of software containers in geo-distributed computing environments.  In Proceedings of the 2019 IEEE Symposium on Computers and Communications (ISCC), pages 1–7, 2019.
  * [74] Ming Yan, XiaoMeng Liang, ZhiHui Lu, Jie Wu, and Wei Zhang .  Hansel: Adaptive horizontal scaling of microservices using bi-lstm.  Applied Soft Computing, 105(1):107216–107230, 2021.
  * [75] Yulai Xie, Minpeng Jin, Zhuping Zou, Gongming Xu, Dan Feng, Wenmao Liu, and Darrell Long.  Real-time prediction of docker container resource load based on a hybrid model of arima and triple exponential smoothing.  IEEE Transactions on Cloud Computing, 1(1):1–17, 2020.
  * [76] Siddhant Kumar, Neha Muthiyan, Shaifu Gupta, A. D. Dileep, and Aditya Nigam.  Association learning based hybrid model for cloud workload prediction.  In Proceedings of the 2018 International Joint Conference on Neural Networks (IJCNN), pages 1–8, 2018.
  * [77] Mahmoud Imdoukh, Imtiaz Ahmad, and Mohammad Gh Alfailakawi.  Machine learning-based auto-scaling for containerized applications.  Neural Computing and Applications, 1(1):1–16, 2019.
  * [78] Yao Lu, Lu Liu, John Panneerselvam, Bo Yuan, Jiayan Gu, and Nick Antonopoulos.  A gru-based prediction framework for intelligent resource management at cloud data centres in the age of 5g.  IEEE Transactions on Cognitive Communications and Networking, 6(2):486–498, 2020.
  * [79] Anirban Das, Shigeru Imai, Stacy Patterson, and Mike P Wittie.  Performance optimization for edge-cloud serverless platforms via dynamic task placement.  In Proceedings of the 2021 IEEE/ACM International Symposium on Cluster, Cloud, and Internet Computing (CCGrid), pages 41–50, 2020.
  * [80] Nabeel Akhtar, Ali Raza, Vatche Ishakian, and Ibrahim Matta.  Cose: Configuring serverless functions using statistical learning.  In Proceedings of the IEEE INFOCOM 2020 - IEEE Conference on Computer Communications, pages 129–138, 2020.
  * [81] Zhe Yang, Phuong Nguyen, Haiming Jin, and Klara Nahrstedt.  Miras: Model-based reinforcement learning for microservice resource allocation over scientific workflows.  In Proceedings of the 2019 IEEE International Conference on Distributed Computing Systems (ICDCS), pages 122–132, 2019.
  * [82] Jieqi Cui, Pengfei Chen, and Guangba Yu.  A learning-based dynamic load balancing approach for microservice systems in multi-cloud environment.  In Proceedings of the 2020 IEEE International Conference on Parallel and Distributed Systems (ICPADS), pages 334–341, 2020.
  * [83] Ayesha Abdul Majeed, Peter Kilpatrick, Ivor Spence, and Blesson Varghese.  Performance estimation of container-based cloud-to-fog offloading.  In Proceedings of the 2019 IEEE/ACM International Conference on Utility and Cloud Computing Companion, page 151–156, 2019.
  * [84] Olufogorehan Tunde-Onadele, Jingzhu He, Ting Dai, and Xiaohui Gu.  A study on container vulnerability exploit detection.  In Proceedings of the 2019 IEEE International Conference on Cloud Engineering (IC2E), pages 121–127, 2019.
  * [85] Zhuping Zou, Yulai Xie, Kai Huang, Gongming Xu, Dan Feng, and Darrell Long.  A docker container anomaly monitoring system based on optimized isolation forest.  IEEE Transactions on Cloud Computing, 1(1):1534–1543, 2019.
  * [86] Rui Shu, Xiaohui Gu, and William Enck.  A study of security vulnerabilities on docker hub.  In Proceedings of the 2017 ACM on Conference on Data and Application Security and Privacy, page 269–280, 2017.
  * [87] Maciej Malawski, Adam Gajek, Adam Zima, Bartosz Balis, and Kamil Figiela.  Serverless execution of scientific workflows: Experiments with hyperflow, aws lambda and google cloud functions.  Future Generation Computer Systems, 110(1):502–514, 2020.
  * [88] Krzysztof Rzadca, Pawel Findeisen, Jacek Swiderski, Przemyslaw Zych, Przemyslaw Broniek, Jarek Kusmierek, Pawel Nowak, Beata Strack, Piotr Witusowski, Steven Hand, and John Wilkes.  Autopilot: workload autoscaling at google.  In Proceedings of the 2020 European Conference on Computer Systems, pages 1–16, 2020.
  * [89] Jingze Lv, Mingchang Wei, and Yang Yu.  A container scheduling strategy based on machine learning in microservice architecture.  In Proceedings of the 2019 IEEE International Conference on Services Computing (SCC), pages 65–71, 2019.
  * [90] Shangguang Wang, Yan Guo, Ning Zhang, Peng Yang, Ao Zhou, and Xuemin Shen.  Delay-aware microservice coordination in mobile edge computing: A reinforcement learning approach.  IEEE Transactions on Mobile Computing, 20(3):939–951, 2021.