# Cloud-Native Computing: A Survey from the Perspective of Services

Shuiguang Deng, Hailiang Zhao, Binbin Huang, Cheng Zhang, Feiyi Chen, Yinuo
Deng, Jianwei Yin, Schahram Dustdar, , and Albert Y. Zomaya S. Deng, H. Zhao,
C. Zhang, F. Chen, Y. Deng, and J. Yin are with the College of Computer
Science and Technology, Zhejiang University, Hangzhou 310027, China. E-mail:
dengsg, hliangzhao, coolzc, chenfeiyi, yinuo<EMAIL_ADDRESS>Huang is
with the College of Computer Science and Technology, Hangzhou Dianzi
University, Hangzhou 310012, China. E-mail<EMAIL_ADDRESS>Dustdar
is with the Distributed Systems Group, Technische Universität Wien, 1040
Vienna, Austria. E-mail<EMAIL_ADDRESS>Y. Zomaya is with the
School of Computer Science, University of Sydney, Sydney, NSW 2006, Australia.
e-mail<EMAIL_ADDRESS>
###### Abstract

The development of cloud computing delivery models inspires the emergence of
cloud-native computing. Cloud-native computing, as the most influential
development principle for web applications, has already attracted increasingly
more attention in both industry and academia. Despite the momentum in the
cloud-native industrial community, a clear research roadmap on this topic is
still missing. As a contribution to this knowledge, this paper surveys key
issues during the life-cycle of cloud-native applications, from the
perspective of services. Specifically, we elaborate the research domains by
decoupling the life-cycle of cloud-native applications into four states:
building, orchestration, operate, and maintenance. We also discuss the
fundamental necessities and summarize the key performance metrics that play
critical roles during the development and management of cloud-native
applications. We highlight the key implications and limitations of existing
works in each state. The challenges, future directions, and research
opportunities are also discussed.

###### Index Terms:

Cloud-native applications, survey, service life-cycle management, research
roadmap, microservice.

##  I Introduction

Services are self-describing and technology-neutral computation entities that
support rapid and low-cost composition of web applications in distributed
network systems [1]. Service-Oriented Architecture (SOA) is the principle to
design the software systems by $(i)$ provisioning independent, reusable, and
automated functions as reusable services and $(ii)$ providing a robust and
secure foundation for leveraging these services [2]. In recent years, the most
influential variant of SOA is the microservices architecture, which decouples
a monolithic application into a collection of loosely-coupled, fine-grained
microservices, communicating through lightweight protocols [3]. Over the last
decade, the microservices approach is more and more appealing, as it allows
teams and software organizations to be more productive to build continuously
deployed systems with the support of DevOps [4] and continuous
integration/continuous delivery (CI/CD) pipelines [5].

Accompanied with the development of microservices, a new terminology, cloud-
native, or cloud-native computing, is attracting increasingly more attention
in academia. In accordance with Cloud Native Computing Foundation (CNCF), the
open source, vendor-neutral hub of cloud-native
computing111https://www.cncf.io/about/who-we-are/, cloud-native is the
collection of technologies that break down applications into microservices and
package them in lightweight containers to be deployed and orchestrated across
a variety of servers222https://github.com/cncf/toc/blob/main/DEFINITION.md. In
addition to the microservices architecture, cloud-native is also characterized
by the following terminologies:

  * •

Containerization. Containerization is a function isolation mechanism which
leverages the Linux kernel to isolate resources, and creating containers as
different processes in Host OS [6, 7]. Docker, with a ten-year development, is
the most popular implementation of the containerization techniques [8].
Combing containerization with the microservices architecture, each part of an
application, including processes, libraries, etc., is packaged into its own
container. This facilitates reproducibility, transparency, and resource
isolation.

  * •

Orchestration. Orchestration is the automated configuration, management, and
coordination of the inter-related microservices to build the elastic and
scalable functionalities. Because microservices are deployed in the way of
containers, orchestration reduces to the automation of the operational effort
to manage the containers’ life-cycle, including resource provisioning,
deployment, scheduling, scaling (up and down), networking, load balancing,
etc, in order to execute the applications’ workflows or processes. Kubernetes
[9], originated from Google’s Borg cluster manager [10], is the most popular
open-source container orchestration software.

Figure 1: The research roadmap for cloud-native computing, designed from the
perspective of services.

As a conclusion, a cloud-native application can be viewed as a distributed,
elastic and horizontal scalable system composed of inter-related
microservices, which isolates state in a minimum of stateful components [11].
Applications are built with cloud-native technologies by the following steps:
$(i)$ Separating the monolith into self-deployed, function-explicit
microservices and letting them communicate with each other through REST APIs
(for synchronous communication) and lightweight messaging protocols (for
asynchronous communication); $(ii)$ Using lightweight operating system
virtualization technology, i.e., containerization, to pack each microservice
into a container; $(iii)$ Orchestrating these containers into a organic whole
for functionalities with automatic configuration and management throughout
their life-cycle; $(iv)$ Using DevOps and CI/CDs to deliver the cloud-native
applications with reliability and scalability.

Considering that cloud-native is better known in industry, in this paper, we
try to survey the past and present of cloud-native applications w.r.t. the key
problems during their life-cycle from a research perspective. We attempt to
merge the industrial popularity, including the widely used open-source
software and platforms, with the trending researches, either theoretical or
systematic, from the perspective of services computing. We divide the life-
cycle of a cloud-native application, which is viewed as a service in the field
of service-oriented computing (SOC), into four states: building,
orchestration, operate, and maintenance. As Fig. 1 shows, different key
problems are emphasized in different states. In addition, we also collect the
performance metrics which are frequently mentioned when building the cloud-
native applications, and analyse them from three levels: infrastructure,
platform, and software. Apart from the service life-cycle and the performance
metrics, Fig. 1 also demonstrates the foundation of cloud-native computing. In
our opinion, the fundamental issue to be dealt with for building cloud-native
applications is resource management, i.e., resource virtualization and elastic
provisioning, such that business agility can come true. Besides, as the
foundation, it should provide general function modules for the building and
running of cloud-native applications. We divide the functions into three
cases: security, performance, and efficiency, and demonstrate the
corresponding popular platforms and tools. The following sections will be
presented surrounding the roadmap.

TABLE I: Summary of important Acronyms. Acronym | Definition | Acronym | Definition
---|---|---|---
SOC | Services-Oriented Computing | SOA | Service-Oriented Architecture
DevOps | Combination of development and operations | CI/CD | continuous integration & continuous delivery
CNCF | Cloud Native Computing Foundation | API | application programming interfere
SDN | software-defined network | NFV | network funciton virtualization
CNF | cloud-native network function | VXLAN | Virtual eXtensible Local Area Network
GRE | Generic Routing Encapsulation | MPLS | Multiprotocol Label Switching
LVM | logical volume manager | RAID | redundant array of independent disks
CNI | container network interface | CSI | container storage interface
SDK | software development kit | TLS | transport layer security
NAT | network address translation | SaaS | Software as a Service
PaaS | Platform as a Service | IaaS | Infrastructure as a Service
JCT | job completion time | CRUD | create, read, update and deletion
REST | Representational State Transfer | HPC | high-performance computing
GA | Genetic algorithm | QoE | Quality of Experience
QoS | Quality of Service | SLO | Service-Level Objective
ANN | Artificial Neural Network | MARL | Multi-Agent reinforcement learning
VM | Virtual Machine | ISP | Internet Service Provider

To the best of our knowledge, this is the first survey focusing on key issues
during the life-cycle of cloud-native applications from the perspective of
services. There are some studies on the origin, status quo, challenges and
opportunities of cloud-native applications [11, 12, 13]. However, they mainly
provide high-level opinions and ignore the review on the state-of-the-art
research works. As a result, researchers may find it struggling to grasp and
comprehend each issue in the development and management of cloud-native
applications. It is worth mentioning that, there are surveys focusing on
specific topics in cloud-native computing. For example, the author of [14]
discusses the recent developments of architectural frameworks for intelligent
and autonomous management for cloud-native networks. This paper
comprehensively reviewed the technical trend toward cloud-native network
design and network-cloud/edge convergence. Moreover, the scheduling of
microservices and containers, which has a strong connection with the
orchestration platform Kubernetes, is reviewed in [15] and [16]. However, in
the lack of systematic knowledge, challenges and proposed solutions will lack
high portability and compatibility for various cloud-native applications. To
this end, this survey is inspired to propose a pipelined design and summarize
the research domains from different views. It can help researchers and
practitioners to further understand the nature of cloud-native applications.
As shown in Fig. 1, we analyze the key problems across the whole life-cycle of
cloud-native applications, from the building state to the maintenance state.
We also discuss the performance metrics and the fundamental necessities when
developing cloud-native applications.

The rest of the survey is organized as follows: Sec. II-VII introduce the
foundation, performance metrics, and the four states of demonstrated in this
roadmap. Sec. VIII discusses the issues, challenges, limitations, and
opportunities of cloud-native. We conclude this paper in Sec. IX. We summarize
the definitions of the acronyms that will be frequently used in this paper in
Table I for ease of reference.

##  II Foundation of Cloud-Native Computing

In this section, we demonstrate the fundamental necessities in cloud-native.
From the perspective of clusters, we demonstrate the central issue of the
foundation and general function modules which play critical roles when
building cloud-native applications.

###  II-A The Hierarchical Structure of Infrastructure

A cluster is a set of computing nodes which are connected to each other
through fast local area networks. On top of the physical nodes, OS-level
virtualization techniques are utilized to create virtual machines (VMs) such
that operating costs and downtime can be minimized. By maintaining a pool of
VMs, fast provisioning of resources can be realized with cluster management
softwares. Cluster management is always tightly coupled with resource
management and task scheduling. Kubernetes, as we have mentioned before, is
widely adopted across industries and has become a de facto standard.
Kubernetes has the ability of managing nodes (including both physical servers
and VMs) with the module Node
Controller333https://kubernetes.io/docs/concepts/architecture/nodes/, which is
responsible for node registration, keeping the nodes up-to-date, and
monitoring their health. In addition to Kubernetes, Mesos [17], Docker Swarm
[18], and Hadoop YARN [19] are also widely used cluster managers.

  * •

Kubernetes. Kubernetes, abbreviated as K8s, is the most influential open-
source platform in cloud-native since its release in 2014. K8s originates in
Google’s Borg [10], which has been used for managing containerized workloads
in Google’s inner clusters for more than a decade. K8s is a distributed
software deployed across nodes, whose target is the automation of the
deployment, management, and scaling of containerized applications by
efficiently managing heterogeneous resources444The architecture of Kubernetes
can be found in the official document:
https://kubernetes.io/docs/concepts/overview/components/. With the centre
being K8s, an open-source ecosystem gradually forms to improve and facilitate
the management of cloud-native applications. In the ever-growing K8s
ecosystem, publishing, installing, removing, and upgrading of cloud-native
applications can be managed with Helm555https://helm.sh/. The traffic between
internal microservices within an application can be managed with
Istio666http://istio.io/, which is the most influential implementation of
Service Mesh[20]. Istio extends K8s to establish a programmable, application-
aware network by taking Envoy777https://www.envoyproxy.io/ as the proxy. The
monitoring of nodes and applications can be realized with
Prometheus888https://prometheus.io and Grafana999https://grafana.com while
logging can be managed with Kibana101010https://www.elastic.co/kibana/.

  * •

Mesos. Mesos is implemented in a two-level architecture. The first level is a
master resource manager that dynamically controls which resources each
framework scheduler owns. Correspondingly, in the second level, each framework
scheduler, such as hadoop, MPI, and Mesos Marathon, are responsible for
scheduling tasks at the application level [17].

  * •

YARN. YARN manages Hadoop clusters. It consists of clients, container,
resource manager (RM), node manager (NM), and application master (AM). Here
container is defined as a collection of physical resources such as RAM, CPU
cores and disk on a single node. It is NM that takes care of each node and
manages applications and workflows on that particular node. When a job is
submitted by a client, the corresponding AM negotiates resources with RM and
requests Container from the NM [19].

  * •

Docker Swarm. Docker Swarm is the native inbuilt orchestration tool for
Docker, which is called “swarm mode”. Docker Swarm is composed of Docker Node,
Docker Services, and Docker Tasks. There are two kinds of Docker Nodes,
Manager and Worker, which are similar to the Master/Worker in K8s. The
Manager, as the name suggests, is responsible for maintaining the cluster
status, scheduling the services, and serving swarm mode HTTP API endpoints. By
constrast, the Workers are nothing but the instances of Docker Engine for
running Docker containers [21].

The following contents are mainly focusing on K8s since in cloud-native, many
rigor progress are achieved based on K8s. Apart from the above cluster
managers, it is worth pointing out that there are orchestration frameworks
that extend the native capabilities of K8s to the network edge.
KubeEdge111111https://kubeedge.io/en/ is a representative one. Edge computing
has a three-level hierarchy: Cloud-Edge-Device [22]. To take the full
advantage of this hierarchy, KueEdge divides its components into two parts:
CloudCore and EdgeCore. CloudCore $(i)$ handles the communication between it
and api-server of a K8s cluster and $(ii)$ communicates with the edge nodes.
EdgeCore is responsible for $(i)$ communicating with CloudCore and $(ii)$
manages the containers, services that are deployed on the edge
devices121212The architecture of KubeEdge can be found in the official
document: https://kubeedge.io/en/docs/kubeedge/. In a recent project, KubeEdge
is reported to stably support 100,000 concurrent edge nodes and manage more
than one million Pods [23].

On top of cluster managers, development tools including SDKs and middlewares
are developed as the building blocks for cloud-native applications. The
hierarchical structure of hardwares, OS-level softwares, cluster managers, and
middlewares construct the fundamental necessities for building cloud-native
applications.

###  II-B Resource Provisioning and Management

Virtualization refers to a collection of techniques for building and managing
virtual resources on top of actual hardware, with key benefits including high
redundancy, unified interfaces for users, and highly efficient resource
utilization. It is the infrastructural foundation of today’s cloud-native
orchestration at all scales. When provisioning resources at a large scale,
virtualization manages all low-level and possibly heterogeneous resources,
such that better global efficiency can be reached in comparison to the
traditional way of letting service users decide on their own, since service
users often do not have access to the global resource utilization information.
Typical computation virtualization technologies are summarized in Table II.

There are multiple types of virtualization in the computing field, among which
computation virtualization, storage virtualization, and network virtualization
are most commonly used in cloud-native design and deployment. Due to the
diversity of resources that can be virtualized, orchestrating all these
heterogeneous types of devices is challenging. In this section, we will review
virtualization technologies of different resources, their latest development,
as well as the role they play when building cloud-native applications.

####  II-B1 Computation Virtualization

Computation virtualization refers to creating an abstraction layer over
computation, often in the form of virtual machines or containers. This is the
core of all cloud-native deployments. The properties and efficiency of a
specific virtualization technology deeply influence the entire deployment.

TABLE II: Computation Virtualization Technologies. Name | Advantages | Disadvantages | Comments | Examples
---|---|---|---|---
Type 1 hypervisor |  High efficiency, unmodified OS | Low flexibility |  Also called "bare metal" |  VMware ESXi, Microsoft Hyper-V
Type 2 hypervisor |  High compatibility, high flexibility, unmodified OS |  Slightly lower efficiency than Type 1 hypervisor |  Runs on OS |  Oracle VirtualBox, VMware Workstation
Container |  Low overhead, flexible deployment |  No vendored kernel |  Foundation of cloud-native systems |  Docker, FreeBSD jail

From the perspective of where the hypervisor resides, virtualization
technologies can be divided into Hypervisor Type 1 and Type 2. A hypervisor is
a software layer that controls the creation and execution of VMs. Type 1
hypervisors, a.k.a. “bare-metal” hypervisors, directly run on hardware, while
Type 2 hypervisors rely on an OS. Both hypervisors support unmodified guest
OSs. Since Type 1 hypervisors directly communicate with hardware, they offer
better performance and efficiency. Type 2 hypervisors, on the other hand,
offer the best flexibility and compatibility, at the cost of a small portion
of performance loss.

Another commonly used hypervisor in today’s server hosting industry is KVM
[24]. It is worth noting that KVM cannot be simply put into Type 1 or Type 2
hypervisor. While KVM runs in the Linux kernel and turns the kernel into a
Type 1 hypervisor, the entire set of solution does operate on an existing
operating system, making it Type 2 by definition.

Instead of running the entire OS, containers choose another approach to
virtualize computation resources. This new approach has been highly successful
in today’s cloud-native scenarios due to its high efficiency[25]. Containers
share the OS kernel with the host OS but have a dedicated userland filesystem.
Moreover, the filesystem only contains necessary binaries, libraries, and
resource files, so the final image could be as low as a few hundred kilobytes.
In many applications, having a completely isolated environment and a dedicated
kernel is, in fact, a huge overkill. Currently, most cloud-native
orchestration implementations, including Kubernetes, are based on Docker or
other container engines[26, 27]. This is mainly due to several unique
advantages containers possess: simplicity, low overhead, fast deployment, and
ease to design and build.

####  II-B2 Network Virtualization

Network virtualization is another important level of virtualization in cloud-
native scenarios. With network virtualization, traditional switches and
routers are replaced with programmable devices, therefore allowing smarter
operation. In this section, we will make a brief introduction to several key
network virtualization technologies in the cloud-native context, including
software-defined network (SDN), network function virtualization (NFV), Service
Mesh, and overlay networks, to understand how they enable efficient and
intelligent network management.

Software-defined Network (SDN) includes a set of techniques to decouple the
data plane and control plane to enable programmatical a dynamical management
of network forwarding devices like routers and switches. Traditionally,
network operators leverage white-label devices from vendors to run their
networks. This approach does not fit current quickly developing cloud-native
environments due to the inflexibility and high price of vendor-made devices. A
typical SDN system consists of several controllers and more forwarders.
Traditional route or forward tables are replaced by unified flow tables, which
are dynamically calculated by controllers and sent to forwarders. Forwarders
then simply forward or drop traffic by looking up relevant table items from
flow tables. The logically centralized control plane provides good visibility
of the entire network, easing the management of network resources [28].

Network Function Virtualization (NFV) is another layer of virtualization in
networking technologies. While SDN decouples the data plane and control plane,
NFV focuses on decoupling software and hardware. With the quick development of
computation virtualization, using virtualized software to replace network
devices has become possible. Together with SDN, there have been several
solutions, including firewall[29] and router[30]. Furthermore, Cloud-native
Network Function (CNF), a new cloud-native aware trend of Virtual Network
Function has emerged. It is designed to run in containers instead of VMs, with
the advantages of cloud-native fully leveraged. To sum up, By utilizing NFV
and SDN, network operators like ISPs and cloud computing companies will
benefit from reduced cost and improved flexibility to keep up with today’s
fast-evolving cloud-based trends.

####  II-B3 Storage Virtualization

Storage virtualization is the technique of creating an abstraction layer over
storage devices, to provide large, fast and redundant storage pools across
multiple hard disks. As I/O operations take a large portion of the entire
turnaround time, storage virtualization deeply affects the efficiency of the
entire system. Furthermore, data redundancy and safety is an inherent
requirement of cloud-native systems, which is often offered by storage. We
identify three major layers of storage virtualization: Host-based
Virtualization, Storage Device-based Virtualization, and Network-based
Virtualization.

Host-based Virtualization is building the storage pool on the end host. An
example of host-based virtualization is Logical Volume Manager (LVM). LVM is
installed onto the OS, and creates a storage pool using storage devices
connected to the host. Device-based Virtualization moves the virtualization
layer from the OS to the device itself. A well-known example of this is
Redundant Array of Independent Disks (RAID). There has been multiple
combinations of RAID technologies (e.g. RAID-0, RAID-1, RAID-10, and RAID-60),
for different requirements in regard to speed, redundancy and other
specialized needs. Finally, Network-based Virtualization is often used in data
centers. Network-based storage pool is built as a dedicated cluster of storage
devices, and is connected to the end host using fast network links. As the
storage cluster often live in the same data center as the hosts, optimal
performance can still be achieved. All these three solutions do not need any
modification on high level applications, as they have the same behavior as
regular disk partitions. Therefore, good compatibility can be guaranteed.

To integrate low level storage systems into containers, Container Storage
Interface (CSI)131313https://kubernetes-csi.github.io/docs/ is proposed and
introduced since Kubernetes v1.9. The CSI is a standard of exposing arbitrary
low level storage system to containerized applications orchestrated by Cloud
Orchestration systems like Kubernetes. To start using a new type of storage
system, developers of the storage system are able to create a CSI plugin,
without modifying the core of the Cloud Orchestration system in use. This is
helpful in today’s customized cloud-native deployments.

###  II-C General Supporting Function Modules

In this section, we demonstrate the general function modules provided by the
K8s ecosystem that play critical roles in the building and managing of cloud-
native applications.

####  II-C1 Security

K8s is designed with several security mechanisms to ensure the safety and
confidentiality of data and resources within the cluster.

  * •

Role-Based Access Control (RBAC).
RBAC141414https://kubernetes.io/docs/reference/access-authn-authz/rbac/ is a
security feature in K8s that enables system administrators to define specific
access levels and permissions for each user or group of users within the
cluster. With RBAC, it’s possible to restrict certain privileges to only
authorized entities.

  * •

Pod Security Policies. K8s provides Pod Security Policies that restrict the
behavior of containers running inside the pods. These policies can prevent
containers from executing privileged actions and running as root. By default,
k8s also isolates pods from one another, which adds an additional layer of
protection.

  * •

Secrets Management. K8s offers a facility, named as
secret151515https://kubernetes.io/docs/concepts/configuration/secret/, for
securely storing and managing sensitive data like passwords, certificates,
keys, etc. The secret data is encrypted at rest and in transit, and the access
to this data is restricted using RBAC.

####  II-C2 Performance

Since the performance of containerization is mainly guaranteed by the
underlying container engines, here we mainly discuss the performance of pod-
to-pod (container-to-container, pod-to-service, etc.) communications.

While SDN and NFV offer flexible network environments, they focus more on low-
level communication, which is not easy to integrate with microservices.
Ideally, microservices should focus on application logic, rather than low-
level communications. Additionally, with hundreds even thousands of
microservices cooperating with each other, it is harder to manage network
communications as the number grows. K8s’s inbuilt network support is able to
provide basic network connectivity. Nevertheless, it is more common to use
third-party network implementations that plug into K8s using the CNI
(Container Network Interface) APIs. Typical implementations include
Flannel161616https://github.com/coreos/flannel,
Calico171717https://github.com/projectcalico/cni-plugin,
Weave181818https://www.weave.works/oss/net/, etc. Flannel is the most popular
implementation to configure a layer 3 network fabric for K8s. By using
Flannel, each node will be installed a binary called flanneld, which is
responsible for allocating a subnet lease to each node out of a larger,
preconfigured address space. The network built by Flannel uses VXLAN and many
other cloud integrations for package forwarding [31]. Different
implementations of CNI are compared in [32], [33] and [34]. K8s also has an
inbuilt DNS such that Pods and Services can be discovered and visited though
their domain names.

Figure 2: The sidecar proxies along with each microservice form a mesh-like
network.

Note that the implementation of CNI is mandatory for building a working K8s
cluster. However, to have a reliable, observable, and secure communication,
the vanilla network is far from enough. Under the circumstances, service mesh
is proposed, which is a software infrastructure working in the application
layer for controlling and monitoring internal, service-to-service traffic in
microservice-based applications [20]. Fig. 2 illustrates why Service mesh is
called a mesh. Service mesh provides dynamic discovery of services,
intelligent load balancing across services, security features with encryption
and authentication, and observability tracing by leveraging a so-called
Sidecar Design Pattern, where a sidecar proxy is dynamically injected into
each Pod for handling incoming requests. In service mesh, the control plane
manages and configures proxies to route traffic, and collects and consolidates
data plane telemetry. Correspondingly, the data plane is implemented as
sidecar proxies. As we have mentioned before, Envoy is the most popular high-
performance open-source implementation of sidecar proxy.

####  II-C3 Efficiency

DevOps, as a portmanteau of development and operation, is a collaborative and
multidisciplinary effort within an organization to automate continuous
delivery of new software versions, while guaranteeing their correctness and
reliability [35]. With DevOps, development and operations teams work together
across the entire cloud-native applications’ life-cycle. DevOps is strongly
connected with CI/CD, which is a multi-stage pipeline of continuous
integration (build $\to$ test $\to$ merge), continuous delivery (automatically
release the code to repository), and continuous deployment (automatically
deploy the application in production). DevOps has already become the
cornerstone of developing microservice-based cloud-native applications
nowadays.

In the K8s ecosystem, KubeSphere DevOps is a powerful CI/CD platform that has
attracted many attentions from industry [36]. KubeSphere DevOps provides CI/CD
pipelines based on Jenkins191919https://www.jenkins.io/. It offers automation
toolkits including Binary-to-Image (B2I) and Source-to-Image (S2I), and boosts
continuous delivery across K8s clusters. A key character of KubeSphere DevOps
is that it scales Jenkins Agents dynamically such that the CI/CD workflows can
be accelerated flexibly.

##  III Performance Metrics

By the top-down decomposition, the cloud native architecture is mainly
composed of the application layer, the platform layer and the infrastructure
layer. Different layers provides different type services from different
perspective. First, at application layer, various software are deployed as
different service that can be accessed by customers over network. The
performance metrics related to the application contextual are latency,
response time, completion time, makespan and so on. Then, platform layer
provides the facilities and APIs to support the building and delivering of
various services. The performance indicators related to the platform layer are
scalability, stability, availability and so on. Finally, infrastructure layer
provides the raw computing, storage, and network resources required by the
service providers. The main performance metrics related to the infrastructure
contextual are resource utilization, resource failure, energy consumption and
so on. All in all, the service provision should be determined by single
performance metric or jointly considering multi-criteria. We describe these
performance metrics in detail as follows.

###  III-A Performance Metrics in Application Level

Various applications are encapsulated to be different containers and provides
services to customers by deploying these containers to physical hosts in a
cloud environment. Benefited from the convenience without installing and
running the application programs. In application layer, the key problem is to
guarantee the quality of service (QoS) for applications which can be measured
by multiple performance metrics such as latency, response time, completion
time, makespan. These performance metrics are very similar, and are often
easily confused. But in fact, they have different meanings. In this paper, we
distinguish these performance indicators and review these related work
optimizing these indicators.

  * •

Latency: it is usually referred to the time during between when something
happens and when it is perceived. Some efforts presented in [37, 38] take
latency as an important metric. For example, the authors in [38] extend
Kubernetes mechanisms to support multi-tenant at the cost of introducing the
moderate latency and throughput overheads.

  * •

Response Time: it is referred to the time during from the request submission
to the result return. Response time usually consists of the transmission time
of data required by the request, the queuing time, the processing time and the
result return time. Many works also consider to optimize response time. For
example, the authors presented in [39] extend Kubernetes mechanism to schedule
pod according to dynamic network metrics, the goal of which is to reduce the
application response time.

  * •

Job Completion Time (JCT): it is the time during from the start time of the
entry task in job to the finish time of the last task. Different from response
time, job completion time mainly concentrates on its processing time. Some
efforts presented in [40, 41, 42, 43] explicitly consider the job completion
time.

  * •

Throughput: it is the ratio of processed requests to the total number of
arrived requests at the system. A higher throughput indicates that much more
requests are processed in unit time and much less response time are incurred
by each request. Hence, optimizing throughput essentially is to optimize
response time as well. Many works take the throughput maximization as an
optimization objective [44, 45].

  * •

Mobility: the mobility of terminals also bring great challenges to service
performance assurance. Specially, the service provision problem in the driver-
less scenario has attracted extensive attention from academic and industry.
Many works explore a series problems of service provision in terms of service
placing, service scheduling, service migration, aiming at guaranteeing the
service performance in the case of terminal moving. For instances, the authors
presented in [37] explicitly investigate the influence of the terminal
mobility on service performance, and design some strategies to guarantee the
service performance.

  * •

SLO: Service Level Objective (SLO) is a key performance indicator that defines
the level of service. It typically defined the minimum level of service that a
provider must deliver to its customers and can be used to set expectations and
establish accountability. A number of works presented in [46, 47] tries to
guarantee SLO for an application.

###  III-B Performance Metrics in Platform Level

Platform as a service provides all facilities and APIs to build and deliver
various services conveniently, which efficiently avoid tedious overhead
incurred by downloading and installing the required software. The metrics to
measure the platform service mainly include scalability, stability,
reliability and availability. These metrics characterize the performances of
platform service from different perspective. We describe these metrics and
review these related studies optimizing these indicators in detail.

  * •

Scalability: it is the ability of a system enabling to dynamically adjust the
amount of resources allocated to containerized applications according to their
potential workload fluctuations, which ensures the applications supported with
enough resources to minimize SLA violations. Scaling can be performed
vertically, horizontally, or both [48]. At present, some efforts presented in
[48, 49, 50, 51, 52] explicitly consider horizontal autoscaling, vertical
autoscaling, and both. For example, in [53], it can create much more container
replicas to meet more application resource requirement. In [54], it
reallocates the amount of resource to the existing containerized application
to best utilize the new hardware resource capacity.

  * •

Stability: it is a system’s ability to keep a quantity of required properties
(e.g., queue length, waiting time, etc) within a bounded region when the
system encounter some disturbances. To guarantee the stability of a system is
a fundamental issue to ensure service performance. Therefore, some works
investigate system stability problem and design various container deployment
or placement approaches to guarantee service performance. For example, in
[55], an adapted reinforcement learning algorithm is adopted to achieve
horizontal and vertical elasticity of cloud application for increasing the
flexibility to cope with varying workloads and guarantee performance
stability. In [56], a two-step algorithm is designed to solve the container
deployment problem in a geo-distributed computing environment. In the first
step, a reinforcement learning approach is adopted to dynamically controls the
number of replicas of individual containers on the basis of the application
response time. In the second step, a network-aware heuristic algorithm is
designed to place containers on geo-distributed computing resources. Its main
goal is to satisfy Quality of Service requirements of latency-sensitive
applications.

  * •

Reliability: it is referred to the system’s ability to deliver services
without service disruption, errors, or significant reductions in performance
even when one or several of its software of hardware components fail. System
reliability is also very important performance metrics. Many research efforts
investigate the system reliability problem and design different software and
hardware schemes to optimize this performance metric. To improve the
reliability of the system and reduce makespan, a heuristic algorithm is
proposed to balance load among virtual machines in [57]. To maintain
reliability and elasticity for the system, a dynamic scheduling algorithm is
proposed to balance the workload of virtual machines in a cloud environment
elastically based on resource provisioning and de-provisioning methods. The
above works mainly concentrate on solving software component failure to
guarantee system reliability. Different from these above works, other works
concentrate on solving hard component failure problem to guarantee system
reliability. The author presented in [58] predict the disk drives failure and
overlap the time of regular data operation and data restoring to significantly
improve service reliability and reduce data center downtime.

  * •

Availability: it is the proportion of time a system is in a functioning
condition. Along with scalability, stability, reliability, availability is
also a prevailing issues for platform service. To cope with possible failure
caused by the mobility of parked vehicles and improve the service
availability, the author presented in [59] design the dual cost and utility-
aware heuristic algorithm to solve the problem of multi-replica task
scheduling in a collaborative computing paradigm consisting parked vehicles.

###  III-C Performance Metrics in Infrastructure Level

Infrastructure as a Service provides the raw computing, network and storage
resources and corresponding operating middleware software to customers on
demand. One of the main benefits for infrastructure as a Service is free from
the burden of infrastructure maintenance. In contrast to application as a
service and platform as a service, infrastructure as a Service mainly provide
to the resource service at lowest level. The performance metrics to measure
the resource service mainly include resource utilization (computation,
storage, network), failure rate, interference, or energy. We describe these
performance metrics and review these related researches optimizing these
indicators.

  * •

Resource Utilization: It is an important performance metric used to describe
the percentage of a system available resource, such computation resource,
storage resource and network resource, that is occupied over an amount of
available time (or capacity). In recent years, some efforts presented in [60,
61, 62, 63, 64] explore resource planning and resource scheduling problems
with the maximization of the CPU and RAM utilization. Specifically, the
authors presented in [61] extend the Kubernetes mechanisms to fairly allocate
multi-resource (such as CPU, memory, and disk) for containerized workloads of
multi tenants. In [65], a storage service orchestration platform is designed
and implemented to support the stateful applications. In [66], a workload
orchestration framework is proposed to match infrastructure owner and tenants,
aiming at optimizing the use of infrastructure while satisfying the
application requirements. Not only that, but the network traffic is also taken
as an optimization indicator [67, 68, 69]. For example, the authors in [69]
and [70] design a network-aware scheduler to automatically manage and deploy
containerized applications, aiming at for reducing the network latency.

  * •

Interference: In cloud native environments, various types of workloads are
encapsulated in the form of containers. However, the isolation of container is
weaker than that of virtual machine. Multiple containerized workloads (such as
computing intensive, storage intensive) co-located on the same server can
interfere with each other, which seriously affect system performance. The
interference issue incurred by co-locating different type workloads become a
pressing issue. Therefore, many works presented in [40, 44, 71, 72, 73]
explicitly consider the inference between co-locate containerized jobs. For
instance, the authors propose in [40] a container placement scheme that
balances the resource contention on the worker nodes.

  * •

Energy Consumption: It is the amount of energy used. The significant amount of
energy consumed by data centers can incur high cost and environmental
pollution. Moreover, the energy consumption problem is also very important for
resource-constrained terminals, due to their limited battery capacity. In
recent years, there exist research efforts on designing various energy-
efficient schedulers [71, 74, 75, 76, 77]. For example, the authors presented
in [74] propose an energy-efficient container migration scheme to migrate
containers for reducing the energy consumption. The authors presented in [71]
design a scheduler to minimize energy consumption and interference.

  * •

Cost: Currently, the big infrastructure service providers such as AWS, Azure
and Alibaba mainly adopt the pay-as-you-go payment model. Therefore, the
financial costs for renting infrastructure resource is also very import
performance metric. In recent years, some research efforts presented in [78]
[79] [80] [81] take financial costs as an optimization goal, and find an
optimal orchestration solution by selecting diverse cloud services according
to their pricing models and computing capability. Their main goals are to
minimize the overall financial costs while satisfying the QoS requirements.

  * •

Fault Tolerance: It is the ability of a system to behave in a well-defined
manner once faults occur. Failures could occur due to dynamic changes in the
execution environment. The failures in the IaaS layer or the physical hardware
has heavily negative effect to the system. Hence, it is import to design
various fault tolerance mechanism to cope with this problem and to minimize
the risk of failure. For instances, in [82], the authors develop a new
container storage driver to solve the global failures and bundled performance
problem.

##  IV Service Building

In service building state, the key steps are $(i)$ architecture selection and
$(ii)$ code development & packing.

###  IV-A From Monolith to Microservices

Before the rise of the microservices architecture, many traditional
applications adopt monolithic architectures. In this case, the application is
deployed in the shape of a single-tiered monolith, which combines different
components into a single program. Typical components are listed as follows:

  * •

Business Logic. The application’s core business logic. For example, in
e-commerce websites, the logic of inventory and shipping management.

  * •

Database. The data access objects responsible for the CRUD of data.

  * •

Interaction and Presentation. The component responsible for handling HTTP
requests and responding with either HTML or JSON/XML (for web services APIs)
objects.

  * •

Integration. The component responsible for the integration with other internal
or external services though message protocols or REST APIs.

With monolithic architectures, all components are tightly coupled and run as a
single service. As a result, any component of the application experiences a
spike in demand, the entire architecture has to be scaled. Besides, adding or
improving a monolithic application’s features becomes complex as the code base
grows. This greatly increase the risk for availability since many dependent
and tightly coupled components increase the impact of a single failure.

To solve the above problems, the microservices architecture is proposed and it
becomes the dominant architectural style choice for service-oriented software
[83]. With a microservices architecture, an application is built as
independent components that run separately as a single service. These services
communicate via a well-defined interface using lightweight APIs. Services are
built for business capabilities and each service performs a single function
module. Since they are independently run, each service can be updated,
deployed, and scaled to meet demand for specific functions of an application
[84, 85, 86]. The microservices architecture brings in many benefits, such as
agility, flexible scaling, easy deployment, resilience, etc. When developing
the cloud-native applications, the primary task is to select the appropriate
architecture (monolith or microservice) based on specific business logic.

###  IV-B Packing Microservices into Containers

Microservices are usually packaged as container images using container
technologies such as Docker and then published to an image registry. The most
popular choice for deploying these container images on a container
orchestration platform like Kubernetes is Helm. Helm charts contain references
to the publicly accessible container registry in order to pull the necessary
container images. Nevertheless, certain companies and organizations uphold
their own private cloud infrastructure. In such cases, accessing public
container image registries or the internet from within the private cloud is
restricted. To deploy an application in such a limited environment, it becomes
necessary to bundle all the required artifacts, including container images,
Helm charts, documentation, etc., into an archive.

It worth mentioning that, if the cloud-native application is published through
serverless functions, the developer only need to upload the code to the
serverless platform, and the containerization and orchestration is
automatically executed by the underlying middlewares and tools. Serverless
computing is a method of providing backend services on an as-used basis [87].
A serverless provider allows users to write and deploy code without the hassle
of worrying about the underlying infrastructure. A company that gets backend
services from a serverless vendor is charged based on their computation and do
not have to reserve and pay for a fixed amount of bandwidth or number of
servers, as the service is auto-scaling. Note that despite the name
serverless, physical servers are still used but developers do not need to be
aware of them. Serverless computing allows developers to purchase backend
services on a flexible “pay-as-you-go” basis, meaning that developers only
have to pay for the services they use. Detailed reviews of recent works on
serverless computing will be given in Sec. V-A3.

##  V Service Orchestration

Service orchestration is the automated configuration, management, and
coordination of multiple microservices to deliver the end-to-end services.
Since microservices are encapsulated in form of container, service
orchestration is essentially container orchestration. As a popular open-source
container orchestration tool, Kubernetes is able to automatically deploy a
large number of containers, and coordinate them to work together in
congruence, thereby greatly reducing operational burdens. The key technology
to support service orchestration lies in effective service placement and
dynamic service scheduling [45, 88]. In the cloud native context, these two
key technologies are investigated widely by a plenty of studies. The service
orchestration solution is mainly affected by the characteristics of the
applications and the computational architectures. Hence, these orchestration
solutions can be classified based on the type of applications and the
computational architectures. The subcategories match the following questions.
Representative works are listed in Table III.

  * •

What type of applications are orchestrated in cloud native system?

  * •

What computational architectures are used in the service orchestration?

TABLE III: Representative works in service orchestration. Work | App. Type | Resource | Cloud Type | Metric | Implementation
---|---|---|---|---|---
CPU | Storage | B.W. | Single-Cloud | Multi-Cloud | Cloud-Edge
[89] | ML | ✓ | ✓ | ✓ |  |  | ✓ | JCT | simulation
[90] | ✓ | ✓ | ✓ |  |  | ✓ | JCT | simulation
[91] | ✓ | ✓ | ✓ | ✓ |  |  | training time | simulation
[92] | ✓ | ✓ | ✓ | ✓ |  |  | JCT & makespan | system
[93] |  | ✓ |  | ✓ |  |  | makespan | system
[94] | ✓ |  |  | ✓ |  |  | JCT | simulation
[95] | ✓ |  |  | ✓ |  |  | latency & res. utilization | simulation
[hbb-yeung2020horus] | ✓ |  |  | ✓ |  |  | inference time | simulation
[96] | ✓ | ✓ |  | ✓ |  |  | response time | system
[97] | HPC | ✓ | ✓ |  | ✓ |  |  | response time | system
[98] | ✓ | ✓ | ✓ | ✓ |  |  | response time | simulation
[99] | ✓ | ✓ |  | ✓ |  |  | througput | simulation
[100, 101] | ✓ | ✓ | ✓ | ✓ |  |  | response time | system
[41] | ✓ | ✓ |  |  |  | ✓ | makespane | simulation
[102] | ✓ |  |  | ✓ |  |  | prediction accuracy | simulation
[103] | serverless | ✓ | ✓ | ✓ | ✓ |  |  | response time | simulation
[104] | ✓ | ✓ |  | ✓ |  |  | scalability | system
[105] | ✓ | ✓ | ✓ | ✓ |  |  | JCT | simulation
[106] | ✓ | ✓ |  | ✓ |  |  | latency & throughput | simulation
[107] |  | ✓ |  | ✓ |  |  | JCT | simulation
[108] | ✓ | ✓ |  | ✓ |  |  | JCT & res. utilization | system
[109] | batch job | ✓ | ✓ |  | ✓ |  |  | SLO & res. utilization | simulation
[110] | ✓ | ✓ | ✓ | ✓ |  |  | res. utilization | simulation
[111] | ✓ | ✓ | ✓ |  | ✓ |  | JCT | simulation
[112] | ✓ | ✓ | ✓ | ✓ |  |  | res. utilization & cost | simulation
[113] | ✓ | ✓ | ✓ | ✓ |  |  | JCT & res. utilization | simulation

###  V-A Application Types

Different types application have significantly distinct characteristics, such
as their Quality of Service requirements, the type, the structure and so on.
These characteristics of the applications have a significant impact on the
service orchestration. A wide range of applications from High-Performance
Computing (HPC), machine learning, batch, web service or serverless, are
handled in cloud native system. In the following, we review these research
studies of service orchestration for different types of applications. Fig. 3
outlines the structure of this section.

Figure 3: Service orchestration under different application types.

####  V-A1 Machine Learning

As a subfield of machine learning, deep learning has become a very popular
research topic due to its advancement in various applications. A standard deep
learning development pipeline consist of model training and model inference
two stages. These two stage tasks are characterized by unique and complicated
features. Specifically, the model training task is long-lived offline task,
while the model inference task is short-lived online task. Moreover, these two
stage tasks focus different performance metrics, in which the model training
task focus on achieving high performance and the model inference task pay more
attention on the response latency and inference accuracy. Their unique
characteristics and different performance requirements impose some specific
challenges to orchestrate model training tasks and model inference tasks. We
comprehensively review and summarize these studies related to model training
task orchestration and model inference task, respectively.

Model training task orchestration. Model training is the process of learning a
model over a large dataset using a machine learning algorithm. Due to
increasingly complicated model and larger datasets, model training is an
extremely time consuming and resource consuming task. Thus, it is urgent to
parallelize model training task on multiple distributed workers. There are
mainly four types of parallelism model training: data parallelism [114, 115,
116, 117, 89], model parallelism[90], pipeline parallelism [91, 118, 119, 120]
and mixed parallelism [121, 92]. Data parallelism refers to place multiple
replicas of a model on multiple workers, and divide the datasets into many
subsets to feed to these multiple workers. These multipe workers
simultaneously perform the model training tasks, and synchronize their
training results in the form of parameter servers or All-Reduce, etc. By data
parallelism, the speed for model training can be greatly accelerated and the
performance for model training can be enhanced. For example, in [89], a novel
online preemptive scheduling framework is designed to dispatch machine
learning jobs to workers and parameter server for reducing the average job
completion time. Analogously, in [93], the authors present a heterogeneity-
aware scheduler which can efficiently collocate deep learning jobs on GPUs by
exploiting the predicting information of GPU memory demand and job completion
times. Its main goal is to improve the GPU resource utilization and reducing
the makespan. However, with the increase of model complexity, a model with the
large number of parameters cannot be launched on a single worker. Thus, model
parallelism is proposed. Model parallelism refer to divide a model into
multiple disjoint partitions and place these partitions on multiple workers.
Since each worker only have one part of the model, only one worker is
performing the model training task at any one time. One of the problems for
the model parallelism is the long training latency incurred by the
communication among multiple workers. To address this problem, a novel
parallelism model training, called pipeline parallelism, is proposed. Pipeline
parallelism divides a model into multiple stages and place multiple stages on
multiple workers; Beside, pipeline parallelism further divide the datasets
into multiple micro-batches. Multiple workers can process multiple micro-
batches simultaneously. Thus, pipeline parallelism training greatly reduce the
training latency. For example, to achieve an efficient and task-independent
model parallelism, the authors in [91] introduce a pipeline parallelism
library to partition a deep neural network across multiple workers and split
the datasets into mini-batches. Finally, mixed parallelism with data and model
parallelism is proposed to reduce the training latency and resource
consumption. For example, in [92], the authors design a job scheduling system
which enable machine learning jobs to be implemented with data parallelism and
model parallelism in clusters. The proposed system greatly reduces the job
completion time and improves accuracy.

Model inference task orchestration. Model inference is the service ability to
make predictions on new data based a trained model. Model inference tasks are
usually deployed as online short-lived services (e.g., automatic driving, face
recognition). How to deploy and orchestrate model inference tasks has
attracted much attention in industry and academia. For industry, many
mainstream deep learning frameworks, such as TensorFlow Serving [122] and
MXNet Model Server [123], have implemented the orchestration function for
model inference task. For academia, there are also a lot of studies about
model inference task orchestration. These studies mainly can be classified two
types: the individual model inference task orchestration and multiple model
inference task orchestration. For the single model inference task
orchestration, the authors in [124, 125] design some optimization techniques
to efficiently orchestrate model inference task. However, the execution of a
single model inference task not only fail to meet the requirement of
application scenarios, but also causes the waste of the resources. Thus, many
researchers further study multiple model inference task orchestration. They
design different heuristic-based, modeling-based or prediction-based
mechanisms to orchestrate multiple model inference tasks[126, 127, 94].
Specifically, the authors in [95] adopt a heuristic approach to select the
requests to be co-located on the same GPU. First, they assign the optimal
batch sizes, given the latency requirement of existing inference task
requests. Finally, they establish the node runtime cycles, aiming at
maximizing the resource utilization while satisfying the latency requirement.
Analogously, the authors in [94] colocate these model inference tasks where
the total of their peak GPU requirement does not exceed the capacity, and
heuristically schedule the newly arrived model inference task to the worker
with smallest completion time, aiming at reducing the total delay. However,
all of the above studies mainly exploit the heuristic method to orchestrate
the certain deep learning model scenarios at a limited scale. The performance
of these heuristic methods could dramatically degrade when the deep learning
models vary. Thus, these heuristic methods can not be applied to cope with
dynamic colocation mechanisms for managing the inference workloads. With
complexity and dynamics of model inference task, many researchers turn to
learning-based methods such as multi-armed bandit, reinforcement learning,
etc. For instance, in [44], the authors investigate the job completion time
slowdown problem caused by the interference between co-located deep learning
jobs. To address this problem, an interference-aware resource manager is
designed to effectively co-locate heterogenous deep learning jobs for
improving resource utilization and job throughput.

####  V-A2 High Performance Computing

High Performance Computing (HPC) jobs are usually large workloads such as
large-scale financial, scientific computing and engineering simulation. To
execute these HPC jobs, an amount of computing power, memory and network
speeds tend to be required. HPC jobs are often submitted to an HPC cluster and
wait to be scheduled by a HPC job scheduler. However, the existing HPC job
schedulers lack micro-service support and container management capacities.
Therefore, it is a challenge about how to efficiently support HPC workloads on
Kubernetes. In recent years, there exist research efforts on efficiently
orchestrating HPC jobs on cloud clusters [128, 129, 130, 131]. These state-of-
the-art studies on HPC job orchestration can be divided to four categories:
added functionalities to HPC workload managers [132, 133, 134], connector
between cloud and HPC [135, 96, 136, 137], cohabitation [99, 138, 139], meta-
orchestration [140, 141, 142]. For added functionalities to HPC workload
managers, it mainly extends HPC workload manager to support container
orchestrator for HPC application. For example, in [97, 96], the authors
investigate the problem of running HPC workloads efficiently on Kubernetes
clusters, and implement a plug-in to efficiently schedule the HPC workloads.
The benefit of this HPC job orchestration approach is less intrusive. However,
its disadvantage is that the added functionalities are limited. To address the
shortcoming for added functionalities to HPC workload managers, the connector
between cloud and HPC is proposed. The connector enable to bridge the gap
between HPC and cloud systems and achieve HPC job orchestration on cloud
platform. For example, in [98], a scheduler plug-in for Kubernetes is
implemented to efficiently schedule the HPC applications on cloud platforms
managed by Kubernetes. In [143], a workflow management system is implemented
to schedule the containerized HPC applications such as nextflow, which greatly
improve the numerical instability incurred by variations across computational
platforms. The benefit of the connector is non-intrusive and enable to exploit
orchestration strategies of orchestration platforms. However, its disadvantage
is that the network latency between cloud and HPC is high. Therefore, a HPC
job orchestration approach, called cohabitaion, is proposed. The cohabitaion
is to coexist HPC workloads manager and cloud orchestrators on an HPC cluster.
For instances, in [99], the authors investigate the service orchestration
problem towards HPC workloads. To address this problem, the authors modify the
configuration and setup of Kubernetes to support HPC workloads, and evaluate
the performance of HPC workloads. The cohabitaion has an advantage of fully
exploiting the functionalities of orchestration platforms. However, the
cohabitaion is extremely intrusive. Therefore, a HPC job orchestration
approach, called meta-orchestration, is designed. The meta-orchestration
approach is to implement an additional orchestrator on top of the cloud
orchestrator and HPC workload manager. For example, in [144], a framework
called Kube-batch is designed to enable HPC workloads execution on Kubernetes.
In [100], an open source tool is designed to manage the full life cycle of HPC
workloads in cloud architectures. In [101], a framework which is compatibility
with Prometheus is proposed to automatically deploy the benchmarking workload
for containerized HPC applications and analyze their performances. The
advantage of the meta-orchestration approach is less intrusive. However, its
disadvantage is to increase the complexity of the architecture and the efforts
of maintenance.

####  V-A3 Serverless Computing

Severless computing is a new execution model of cloud computing which is a
integration of both function as a service (FaaS) and backend as a service
(BaaS). Serverless computing is characterized by the automatic management and
lightweight features. These characteristics of serverless computing enable
developers to focus on the business logic, with no need worry about
infrastructure provisioning and management. Benefiting from its advantages,
serverless computing recently attracts a lot of attentions in both industry
and academia. However, the inextricable dependencies between massive functions
pose a great challenge to serverless orchestration. In recent years, there are
a plenty of research efforts on serverless orchestration. In the industry
community, several open-source platforms and serverless computing frameworks,
such as Kubeless [145], OpenFaas [146], OpenWhisk [147] or Fission [148], are
designed to support the serverless computing orchestration. These open source
frameworks with different architectures enable to dynamically manage, scale,
and provide different types of resources for serverless applicaitons. In the
academic community, some research efforts presented in [102, 103] design
diverse scheduling scheme for serverless applications. These strategies can
roughly be divided two categories: centralized scheduling [104, 105, 106] and
distributed scheduling [107, 149]. For the centralized scheduler, the authors
in [102] present a double exponential smoothing approach to calculate the
optimal number of pods for serverless applications. Analogously, in [103], a
serverless computing frameworks, called Pigeon, is presented to schedule the
FaaS funcation to pre-warmed containers. Moreover, the framework introduces a
static pre-warmed container pool to cope with the burst function arrival. Both
of novelty mechanism can greatly reducing the response time for serverless
application and improve the system performance. Moreover, in [41], the authors
investigate the influence of the composite property of services on scheduling
scheme at serverless edge. To address this problem, a dependent function
embedding algorithm is designed to get the optimal edge server for each
function, aiming to minimize its completion time. All above these approaches
are centralized. The centralized schedulers are vulnerable to a single point
of failure and high communication overhead. To address these problem of the
centralized scheduler, some distributed scheduling strategies are proposed.
For example, the authors in [107] design a scheduler based on deep
reinforcement learning to dynamically make decision on the number of functions
and their resources, aiming at making a trade-off between cost and
performance.

####  V-A4 Batch Jobs

More and more diverse tasks are running on cloud data centers, of which batch
jobs account for a large proportion. There exist many works dealing with batch
job scheduling. These works are mainly carried from the system implementation
and algorithm optimization two aspects. For the works on system
implementation, the authors in [108], design a cloud-native platform called
Fluid which can co-orchestrate the data cache and deep learning jobs to
improve the overall performance of multiple deep learning jobs. Analogously,
in [109], a scheduling system based on the real server utilization and a
sliding window-based algorithm are designed to schedule and reschedule batch
jobs, and thereby effectively improve the resource utilization in Kubernetes.
For the works on optimization algorithm, there are mainly three kinds of
methods to solve it: sophisticated heuristics, meta-heuristic algorithms [110,
111, 112, 113] and reinforcement learning [150, 151]. The sophisticated
heuristics, such fair scheduling [152], first-fit [153], simple packing
strategies [154] are usually easy to understand and implement. However, it
needs manual adjustment to gradually improve the algorithm. Therefore,the
meta-heuristic algorithms, such genetic algorithm or ant colony algorithm, are
proposed to orchestrate batch jobs. For example, in [110], the authors adopt
the meta-heuristics optimization algorithm to schedule batch jobs for achieve
higher resource utilization. In [111], a redundant placement problem for
microservice-based applications is formulated to be a stochastic optimization
problem. To address this problem, a GA-based server selection algorithm is
designed to efficiently decide about how many instances as well as on which
edge sites to place them for each microservice. Its main goal is to reduce
service execution latency and improve the service availability. Analogously,
in [112], a stochastic hybrid workflow scheduling algorithm is design to
jointly schedule offline batch workflows and online stream workflows in cloud
container services. Its main goal is to minimize the cost and improving
resource utilization in cloud container services. Moreover, in [113], the
authors formulate the concurrent container scheduling problem to be a minimum
cost flow problem. To address this problem, an efficient solution is designed
to lower the average container completion time and improve resource
utilization. However, the batch job orchestrations based on meta-heuristic
algorithms can not efficiently cope with the dynamics of the batch jobs and
the variety of the execution environment. To address this problem,
reinforcement learning methods are adopted to handle dynamic orchestration
problems of batch jobs. For instances, in [150], the authors adopt a deep
reinforcement learning algorithm to schedule independent batch jobs among
multiple clusters adaptively. Analogously, the authors in [151] propose a
graph learning approach to discover the insightful properties and patterns of
batch jobs. Based on these characteristics, the batch jobs can be better
scheduled in production cloud computing environment.

###  V-B Cloud Types

Existing mainstream computing paradigms include single-cloud, multi-cloud and
cloud-edge synergy. Different computing paradigms have different
characteristics, which have an important impact on service orchestration.
Plenty of research studies have investigated the service orchestration under
three different computing paradigms. We categorize them by the mainstream
computing paradigms and overview them. Fig. 4 outlines the structure of this
section.

Figure 4: Service orchestration under different computation paradigms.

####  V-B1 Single-Cloud

The single-cloud paradigm is a new service model which delivers complex
hardware and software services to external customers through the Internet
[110]. In order to improve the utilization of cloud resource and reduce the
response time of cloud service, the efficient cloud service orchestration is
key. Currently, a large number of approaches are proposed to handle service
orchestration [155, 156, 72, 63, 157, 62, 158, 78, 79, 159, 159]. These
approaches can be classified three types: static heuristic based service
orchestration [155, 156], machine learning based service orchestration [72,
63, 157]. For static heuristic based service orchestration, various heuristic
algorithms including bin-packing algorithms, genetic algorithm, particle swarm
optimization, etc., are adopted to orchestrate service in certain workload
scenarios at a limited scale. For example, in [155], the authors formulate the
placement of containers to be a variable-sized bin packing problem. To address
this problem, an elastic scheduling algorithm for microservices in clouds is
proposed. Its main goal is to minimize the cost of virtual machines while
meeting deadline constraints. Analogously, in [156], the authors design some
heuristic algorithms to schedule container cloud services to implement load
balance and improve application performance. However, the performance of
static heuristic algorithms could dramatically degrade when the system scales
up. They cannot cope with the increasingly diverse and dynamic workloads and
environments. To address this problem, machine learning algorithms including
reinforcement learning, K-means, recurrent Neural Network, etc., are
accordingly employed to orchestrate service. For example, in [72], a
containerized task scheduler employs K-means++ algorithms to characterize the
workload features and identify their behavior, and accurately co-locate
heterogeneous workloads in an interference-aware manner. This scheme greatly
improves resource utilization and reduces the rescheduling rate. Moreover, in
[63], a Kubernetes scheduler extension which adopt machine learning algorithm
to predict Quality of Experience (QoE) and schedule the resource based on the
predicted QoE, is designed to improve the average QoE and eliminate over-
provisioning altogether in the cloud. Also, in [157], a self-adaptive
Kubernetes scheduler (re-)deploys these time-sensitive applications by
predicting their required resource in the cloud. These machine learning based
service orchestration schemes can build certain machine learning model for
diverse and dynamic workloads and environments and predict multi-dimensional
performance metrics. These schemes could further improve the quality of
resource provisioning decisions in response to the changing workloads under
complex environments.

####  V-B2 Multi-Cloud

With the surge of cloud workloads, single-cloud paradigm cannot meet with
their various requirements, such as resource requirement, cost requirement,
reliability requirement and so on. Therefore, multi-cloud paradigm is
proposed. The multi-cloud paradigm enables resources among different clouds to
be shared to cope with a burst of incoming tasks. In addition, the multi-cloud
paradigm can efficiently improve service reliability and reduce service cost.
Although benefiting from these advantages of the multi-cloud paradigm, the
heterogeneity of the underlying resources and services for different cloud
systems brings some new challenges to service orchestration in multi-cloud
paradigm. To cope with these new challenges, some related research works about
service orchestration in multi-cloud paradigm [160, 161, 162, 163, 164, 165]
are conducted. Their main optimization objectives are service cost and service
performance. According to these two optimization objectives, these research
works can be divided into two types: cost-optimal service orchestration and
performance-optimal service orchestration. For cost-optimal service
orchestration, the authors in [160, 161, 162] evaluate and select diverse
cloud services according to their pricing models and computing capacity, and
design various service orchestration strategies to minimize the financial
costs. For performance-optimal service orchestration, the authors in [163],
adopt a meta-heuristic algorithm to continuously make elastic container
deployment plans in geographically distributed clouds, aim to maintain
performance while minimizing the operating costs. Also, the authors in [165],
propose a hybrid GA-based approach to deploy a new type of composite
application in multi-cloud. Its main goal is to optimize the performance and
control the budget. However, these heuristic algorithms rely on the prior
knowledge of the system, and cannot cope with the high variable workloads.
Thus, the authors in [164] turn to learning-based method. They adopt a deep
reinforcement learning to dispatch the new arriving requests for applications
in multi-cloud, the goal of which is to minimize the network latency and
satisfy the budget satisfaction.

####  V-B3 Cloud-Edge Synergy

With the explosive growth of data generated by the terminal devices of IoT,
transmitting these massive data to remote cloud to process commonly leads to
significant propagation delays, bandwidth and energy consumption. It drives
the centralized cloud to sink their computation and storage resources down to
the network edge to process data, which is called the cloud-edge synergy
paradigm. The cloud-edge synergy paradigm has the characteristics of resource
heterogeneity, device mobility and connection uncertainty. These
characteristics bring some new challenges to the service orchestration in
cloud-edge synergy paradigm. There are a plenty of researcher studies to
investigate these challenges [166, 74, 167, 161, 45, 168, 169, 170, 171, 172].
Their optimization objectives mainly include response time and energy
consumption. Base on their optimization objectives, we classify these studies
into two types: latency-aware service orchestration and energy-efficient
service orchestration. For latency-aware service orchestration, the authors in
[45, 169, 166, 173, 167, 161], adopt Markov decision process, reinforcement
learning, deep reinforcement learning and heuristic methods to offload the
containerized applications in cloud-edge synergy paradigm. Their main goals is
to optimize latency. Specifically, in [45], a learning-based scheduling
framework for edge-cloud systems is designed to dispatch service request and
orchestrate multiple microservices instances, the goal of which is to improve
the long-term system throughput rate. Analogously, in [168], a network-aware
scheduler plugin is designed to place containerized applications on
distributed cloud-edge clusters. The placement strategy of these applications
considers both current network conditions and communication requirements
between microservices, which is suitable for the placement of time critical
applications. For energy-efficient service orchestration, the authors in [74]
adopt best-fit algorithms to place the containers, aiming to reduce the energy
consumption. Also, in [71], a competent controller is presented to schedule
containerized applications in edge-cloud system, aiming at minimizing the
interference and the energy consumption.

##  VI Service Operate

Service operation, which encompasses load balancing, service migration, and
resource auto-scaling, is crucial for maintaining a high-performing and
efficient system infrastructure. By integrating load balancing, service
migration, and resource auto-scaling in the operational state, we can enable
robust and efficient service management dynamically and at scale.
Representative works are listed in Table IV for summarization.

TABLE IV: Representative works in service operation. Work | Metric | Resource | Algorithm
---|---|---|---
QoS | Throughput | Makespan | Response time | Revenue | CPU | RAM | B.W. | Storage
[174] |  |  | ✓ |  |  | ✓ |  | ✓ |  | Heuristic
[156] |  | ✓ |  |  |  | ✓ | ✓ | ✓ | ✓ | Heuristic
[175] |  |  |  |  | ✓ | ✓ |  |  | ✓ | Game theory
[57, 176] |  |  | ✓ |  |  | ✓ |  |  |  | Heuristic
[177] | ✓ | ✓ |  |  |  | ✓ |  |  |  | LP
[178] | ✓ |  |  |  |  |  |  | ✓ | ✓ | Heuristic
[179] | ✓ |  |  |  |  | ✓ |  | ✓ |  | Approx. algorithm
[180] |  |  |  | ✓ |  | ✓ | ✓ |  |  | Heuristic
[181] | ✓ |  |  |  |  | ✓ |  |  |  | Epidemic
[182] | ✓ |  |  |  |  | ✓ |  | ✓ |  | Game theory
[183] |  |  | ✓ |  |  | ✓ |  |  |  | MARL
[184] |  |  |  | ✓ |  | ✓ |  |  |  | DQN
[185] |  | ✓ |  | ✓ |  | ✓ |  | ✓ |  | MARL
[186] |  |  |  | ✓ |  | ✓ | ✓ |  |  | Heuristic
[187, 188] |  |  |  | ✓ |  | ✓ | ✓ |  |  | -
[189] |  |  |  | ✓ |  | ✓ | ✓ | ✓ |  | Linear regression
[190] | ✓ |  |  |  | ✓ | ✓ |  | ✓ |  | MIP
[191] |  |  |  |  | ✓ | ✓ |  | ✓ |  | -
[192] |  |  |  | ✓ |  | ✓ |  | ✓ |  | -
[193] |  | ✓ |  | ✓ |  | ✓ |  | ✓ |  | -
[194] | ✓ |  |  | ✓ | ✓ | ✓ |  |  |  | RL
[195] |  | ✓ |  | ✓ |  | ✓ | ✓ | ✓ |  | -
[196] |  |  |  | ✓ |  |  | ✓ | ✓ |  | -
[197] | ✓ |  |  | ✓ |  |  | ✓ | ✓ |  | -
[198] |  |  |  | ✓ |  | ✓ |  | ✓ |  | -
[199, 200] |  | ✓ |  | ✓ | ✓ | ✓ |  | ✓ |  | Heuristic
[201] |  | ✓ |  | ✓ |  | ✓ | ✓ | ✓ |  | -
[202] |  | ✓ |  | ✓ |  | ✓ |  |  |  | PID
[203] |  |  |  | ✓ |  | ✓ |  |  |  | MAP
[204] |  | ✓ |  | ✓ |  | ✓ | ✓ |  |  | MPC
[205] |  |  |  | ✓ |  | ✓ | ✓ |  |  | PID
[206] | ✓ |  |  | ✓ |  | ✓ |  |  |  | Fuzzy logic
[207] | ✓ | ✓ |  |  | ✓ | ✓ |  | ✓ |  | FTRL
[208] | ✓ |  |  | ✓ |  | ✓ | ✓ |  |  | ILP
[209] | ✓ |  |  | ✓ |  | ✓ | ✓ | ✓ |  | Heuristic
[210] |  | ✓ |  | ✓ |  | ✓ |  | ✓ |  | RL

###  VI-A Load Balancing

Load balancing is a technique used to address the problem of workload
imbalance across multiple containers. It enables optimal utilization of
resources, improves throughput, and reduces response time and makespan. In
cloud-native environments, the primary goal of load balancing is to prevent
overloading of a single container or cluster while keeping other containers
idle. Cloud-native applications with high throughput and parallel computing
architectures require effective load balancing techniques. One such technique
involves redistributing heavy workloads from a single virtual server to
multiple virtual servers, ensuring optimal resource utilization. In Sec.
VI-A1, we will provide a comprehensive analysis of load-balancing algorithms.
Sec. VI-A2 introduces the current techniques for implementing load balancing
in cloud-native.

####  VI-A1 Algorithm Design and Analysis

Load balancing can be divided into two categories: centralized and
distributed, as illustrated in Fig. 5. Centralized load balancing can further
be classified into static and dynamic algorithms based on whether the
algorithm incorporates prior knowledge of the system. On the other hand,
distributed load balancing employs multi-agent algorithms that offer greater
flexibility and scalability than centralized algorithms.

Figure 5: Algorithms for load-balancing.

Static algorithms. In cloud-native environments, static load-balancing
algorithms are widely adopted due to their ability to leverage prior knowledge
of system states. These algorithms, such as Round-robin, FIFO, MIN-MIN, and
others, do not require detailed information about the current workload running
in the system. Instead, they rely on factors such as CPU usage, memory usage,
storage availability, or network bandwidth utilization. Implementing static
algorithms in a cloud-native system is generally straightforward. Heuristic
algorithms play a significant role in developing load-balancing strategies for
cloud systems under fixed conditions. For example, Zhang et al. [174] propose
a static algorithm called APDPSO that utilizes a particle swarm optimization
method. They treat the allocation of suitable hosts as a discrete optimization
problem. Similarly, Zhao et al. [156] propose a load-balancing solution based
on service performance. They employ a statistical method to optimize the
collaboration problem heuristically.

While static load-balancing algorithms are efficient and easy to implement, it
is essential to consider server performance and continuously monitor the
current load status to prevent exacerbating load imbalances during long-term
execution. Regular maintenance and adjustment of the load-balancing algorithm
are necessary to ensure optimal performance and resource utilization.

Dynamic algorithms. Dynamic load-balancing algorithms offer superior
performance in adaptively executing load balancing, especially when dealing
with sudden changes in workload. They can effectively operate on a cloud-
native platform without prior knowledge of the system or workload. These
algorithms adjust the allocation of resources dynamically based on real-time
information, such as server availability and network bandwidth utilization,
ensuring optimal resource utilization and reduced response times.

Load balancing involves transferring tasks to appropriate servers to alleviate
the burden on overloaded servers. One crucial aspect is designing an efficient
scheduling strategy to minimize the average load across all servers and reduce
the makespan of the system. Hongli et al. [175] propose a theoretical game
algorithm that balances tasks by offloading them to edge servers while
ensuring Service Level Objectives (SLOs). Another approach to dynamic load
balancing revolves around resource allocation. Fatemeh et al. [57] design a
heuristic algorithm-based optimization algorithm that evaluates overloaded,
loaded, and balanced virtual machines to achieve load balancing. However,
these efficiency-focused load-balancing algorithms may not be suitable for
large-scale virtual machine environments.

To meet the requirements of reliability and elasticity in large-scale systems,
Mohit et al. [176] propose a dynamic scheduling algorithm based on the last
optimal $k$-interval virtual machines strategy that balances workload through
resource provisioning and de-provisioning methods. This algorithm effectively
balances the workload of virtual machines in a cloud environment through
resource provisioning and de-provisioning methods. Regarding load balancing
among microservices, Ruozhou et al. [177] introduce a graph-based model to
analyze dependencies among microservices and adopted a polynomial
approximation method to solve the QoS-aware load-balancing optimization
problem. Network traffic is another critical metric for monitoring the state
of cloud-native systems, often triggering load-balancing operations. Lemei et
al. [178] address service unreliability and dynamic network traffic challenges
by designing a load-balancing strategy based on traffic allocation consistency
and DNS granularity, aiming to achieve an approximate solution to the QoS
optimization problem. To efficiently route network traffic, Jingzhou et al.
[179] propose an approximation algorithm with a polynomial-time complexity
that follows a two-step process to implement service deployment.

Multi-agent algorithms. In large-scale clusters, centralized load-balancing
solutions can become time-consuming due to the reliance on a single machine
for decision-making. These solutions gather system information from the
involved servers, which introduces delays in the decision-making process. On
the other hand, distributed load-balancing schemes offer advantages in terms
of scalability and flexibility, particularly in cloud-native environments.
Imbalanced workloads among heterogeneous servers in the cloud can result in
performance degradation within the cloud platform. To address this challenge,
Gutierrez et al. [180] design a distributed approach that focuses on migrating
virtual machines (VMs) to achieve load balancing. Their approach outperforms
the centralized load-balancing method. However, it should be noted that
collecting global information for load-balancing decisions in each agent can
still be time-consuming. To tackle this issue, Harshitha et al. [181] propose
a distributed load-balancing scheme that leverages partial information about
the global state of the cloud system. Their scheme involves two steps: global
information propagation and workload transfer. By utilizing partial
information, the load-balancing process can be expedited while still achieving
effective load distribution. Another proposed scheme, F-TORA, by Xu et al.
[182], focuses on task load balancing. It utilizes fuzzy neural networks and
game theory to optimize task allocation and resource utilization. F-TORA aims
to ensure timely and high-quality services by intelligently distributing tasks
among available resources. These distributed load-balancing schemes offer
advantages over centralized solutions in cloud-native environments. They
provide scalability, flexibility, and improved performance by efficiently
distributing workloads across servers or VMs. However, it’s important to
consider the specific requirements and characteristics of the system before
choosing the most suitable load-balancing scheme.

The advent of intelligent algorithms, such as reinforcement learning, has
opened up new possibilities for distributed load balancing. Reinforcement
learning techniques, including Q-learning and multi-agent reinforcement
learning (MARL), have gained popularity in this domain. Zhiyuan et al. [183]
propose a MARL framework that specifically addresses the dynamics of arrival
workload. This framework overcomes the limitations of independent and selfish
algorithms commonly used in load-balancing schemes. By leveraging MARL, the
proposed approach enables agents to collaborate and make coordinated
decisions, leading to more effective load balancing in dynamic workload
scenarios. Ali et al. [184] design a multi-agent deep Q-network with coral
reefs optimization (MDQ-CR) to minimize the energy consumption of cloud
computing. This approach combines the power of deep Q-networks, a variant of
reinforcement learning, with coral reefs optimization, a nature-inspired
optimization algorithm. The combination of these techniques enables efficient
load balancing while considering energy consumption as a critical factor. Omar
et al. [185] utilize a graph neural network (GNN)-based method to model the
network as a graph and apply MARL techniques to tackle the load-balancing
problem while scheduling traffic flow. By representing the network as a graph,
the authors capture the dependencies between nodes and leverage GNNs to
process and aggregate information effectively. MARL techniques are then used
to optimize load balancing and traffic scheduling based on the learned graph
representations. These studies highlight the application of reinforcement
learning, particularly MARL, in distributed load balancing. These intelligent
algorithms provide a promising avenue for addressing load-balancing challenges
and optimizing various aspects such as workload dynamics, energy consumption,
and traffic flow in cloud computing environments.

####  VI-A2 Tools and Systems

The most widely used load balancing techniques possess several desirable
characteristics, including scalability, flexibility, low cost, simple
deployment, and security. The load balancer allows the system to adapt to
dynamic workloads by scaling in or out as needed. It should work seamlessly
with various operating systems, cloud environments, and virtual machines and
can be easily deployed. Additionally, the load balancer should provide a
secure environment for the system and its users. Some popular load balancing
solutions include Nginx[211], a widely-deployed reverse proxy server, and
HAProxy[212], a fast and efficient reverse proxy software. Recent advancements
in load balancing technology include Maglev[213], which is able to balance
sudden spikes in network traffic based on ECMP rules. Maglev is Google’s
production load balancer, which fully utilizes multiple networking techniques
to achieve flexible and scalable load balancing. Specifically, Maglev utilizes
Google’s global backbone to announce IP prefixes at the same cost so that BGP
routers can provide the first layer of load balancing. Then, IP packets are
evenly distributed among service endpoints, providing another layer of load
balancing. Since Maglev is entirely software-based, adding more load-balancing
capability is simple as long as the backbone or service endpoints are not
saturated. CHEETAH[214] is another load balancer that supports uniform load
distribution with per-connection consistency.

###  VI-B Service Migration

Service migration refers to moving the service application from the original
clouds or machines to the destination. The host transfers all system states,
including the memory, file system, and network connectivity profiles, to the
destination host, keeping conditions without changes.

####  VI-B1 Algorithm Design and Analysis

Service migration is a critical aspect of cloud-native environments,
encompassing live migration, VM-based migration, and container-based
migration. Live migration offers minimal impact on running services and
preserves memory data. VM-based migration focuses on optimizing the process
through modeling, prediction, and analysis. Container-based migration benefits
from efficient migration techniques and tools, enabling seamless migration of
container-based services. Evaluating migration performance under various
conditions is essential. Service migration enhances the flexibility,
efficiency, and reliability of cloud-native systems. We introduce service
migration in the following aspects as shown in Fig. 6.

Figure 6: Classification of service migration.

Live vs. non-live service migration. The main difference between live and non-
live migration is as follows: non-live migration requires the system to be
shut down during the migration process, while live migration involves
migrating running systems. Non-live migration is a simpler approach but does
not support the preservation of memory data, leading to memory data loss. On
the other hand, live migration offers the advantage of minimal impact on
running services, with low interruption time and the ability to preserve data
in memory. Additionally, applications running in the system remain unaware of
the migration. However, live migration can be a complex operation, and the
migration process may encounter interruptions. Despite the challenges
associated with live migration, it is widely used in cloud-native environments
due to the flexibility it provides through its techniques [215].

Live migration predominantly employs two methods: pre-copy and post-copy
[186]. The pre-copy migration algorithm involves iterative copy operations,
which can sometimes result in the migration process failing to converge,
leading to prolonged overall migration time. On the other hand, the post-copy
migration algorithm offers shorter overall migration time. However, this
approach can cause page faults during the migration process, resulting in
degraded performance and reduced stability of the virtual machine [216]. To
address the challenges of post-copy migration, fault tolerance becomes
necessary after a failure during the recovery of a virtual machine. One
method, called PostCopyFT, tackles this issue by utilizing an efficient
reverse incremental checkpoint mechanism. This approach resolves the problem
without increasing the total migration time [187]. Additionally, an optimized
post-copy mechanism based on Fabric-Attached Memories (FAMs) has been
proposed. This mechanism, which is FAM-aware and employs system-level
checkpointing, reduces both the migration time and the system’s busy time
[188]. These advancements in live migration techniques aim to improve the
efficiency, reliability, and performance of the migration process. The choice
between pre-copy and post-copy methods depends on factors such as migration
time, system stability, fault tolerance requirements, and the impact on the
virtual machine’s performance.

VM-based vs. container-based service migration. Existing works have optimized
the VM-based migration process from various perspectives including modeling
[189, 190] , prediction [189, 191] , latency[192, 193], energy consumption
[194], etc. The dynamics of workloads make live migration modeling
challenging. Jo et al. propose a machine learning-based model to enhance the
prediction accuracy, considering critical characteristics of live migration
[189]. Nguyen et al. develop a two-phase migration optimization model aimed at
optimizing VM movement. The first phase computes an optimal embedding strategy
to reduce demands on other virtual networks, while the second phase executes
the migration using this strategy [190]. Maintaining uninterrupted uptime is
crucial for live migration, particularly in large-scale systems with frequent
infrastructure changes. Adam et al. propose a live VM migration scheme that
minimizes the impact on users while addressing version updates and security
concerns [191]. Bandwidth-Aware Compression (BAC) focuses on the trade-off
between VM compression and transmission during migration [192]. The
utilization of multi-page compression techniques enables an efficient
migration scheme, reducing total migration latency while maintaining
performance comparable to benchmarks. To enhance performance during live
migration, Franck et al. propose a new Multi-Path TCP method over WAN, which
significantly decreases round-trip latency and improves responsiveness and
user engagement [193]. For energy consumption reduction and resource
allocation in cloud-native environments, Basu et al. adopt a reinforcement
learning algorithm to make optimal decisions regarding virtual machine
migration [194]. These research efforts aim to address various challenges and
optimize live migration processes in terms of prediction accuracy, network
optimization, uninterrupted uptime, bandwidth management, performance
improvement, and resource allocation.

Container-based migration has gained popularity in cloud-native environments
compared to VM-based migration [217, 195, 196, 197, 198]. Cloud-native
platforms like K8s and Docker Swarm offer efficient handoff capabilities
during container migration. To reduce handoff latency during migration, Lele
et al. propose a framework that enables mobile users to offload their tasks to
edge servers through seamless migration of container-based services [217]. In
order to provide users with the freedom to choose cloud-native platforms, Thad
et al. introduce a tool called CloudHopper, which facilitates the movement of
containers between different platforms [195]. Live migration is widely
utilized in cloud-native platforms, but the cost of copying numerous memory
pages from a source to a destination server can be high. To tackle this
challenge, Piush et al. present mWarp, a live container migration tool that
efficiently remaps the physical memory of containers [196]. Bo proposes an
efficient live migration system called Sledge, which integrates images and
management context to reduce migration overhead and improve quality of service
(QoS) with minimal downtime. The system employs a dynamic context-loading
mechanism to minimize downtime during migration [197]. Although containers
boot faster than VMs, their behavior during live migration under non-ideal
conditions remains a question. Roberto et al. develop a testbed to evaluate
latency and downtime during live container migration in adjusted conditions.
They find that network overload significantly impacts migration performance,
while stressing a container within a host has minimal effect [198]. These
advancements in container-based migration address various challenges and offer
solutions for efficient handoff, provider flexibility, memory optimization,
migration overhead reduction, and evaluation of migration performance in
different conditions.

####  VI-B2 Tools and Systems

Service migration aims to solve problems such as the upgrade during service
operations, load balancing between clusters, and service deployment between
cloud vendors. The most popular hypervisors used for migration in cloud-native
are as follows.

  * •

Kernel-based Virtual Machine (KVM) is a module in the Linux kernel used to
virtualize physical machines. It enables the host machine to turn into a
hypervisor running multiple isolated virtual environments. KVM was first
announced in 2006 and merged into Linux kernel releases a year later[24].

  * •

Xen focuses on the virtualization technology that supports multiple cloud
platforms. The most significant feature of Xen is that it can support multiple
guest operating systems, for instance, Linux, Windows, NetBSD, and FreeBSD,
etc. It allows live migration between multiple hosts seamlessly[218].

  * •

OpenVZ is a virtualization technology for Linux based on an operating system
level. It can support multiple operating systems and allow live container
migration using checkpointing features with little delay[219].

  * •

Checkpoint/Restore In Userspace (CRIU) is a software tool for Linux to freeze
the system states by a checkpoint technology. With CRIU, we can operate live
migration in user space which is mainly distinctive to other migration tools.
During live migration, CRIU can convert the frozen running applications into a
collection of files and then restore them in the checkpoint frozen[220].

###  VI-C Resource Auto-Scaling

With the pay-as-you-go principle, a cloud vendor allows applications to
dynamically acquire or release their resources on their demands. Thus, the
application provider can leverage the auto-scaling method to efficiently
utilize the elastic feature of resources according to its budget and profit.
This section introduces auto-scaling in three categories, i.e., vertical
scaling, horizontal scaling, and hybrid auto-scaling in Sec. VI-C1. Tools and
systems are presented in Sec. VI-C2.

####  VI-C1 Algorithm Design and Analysis

According to different policies adopted by auto-scaling, we summarise auto-
scaling into three categories as shown in Fig. 7.

Figure 7: Techniques for Resource Auto-Scaling.

Horizontal. Horizontal scaling is a widely adopted approach for auto-scaling
in cloud-native environments, enabling applications to dynamically adjust the
number of virtual machines (VMs) to scale resources. Researchers have explored
cost-efficient methods [199, 200] and fast scaling approaches [221, 201] in
this domain. Cost-effectiveness is a crucial consideration in implementing
horizontal scaling in cloud-native systems. Romero et al. introduce INFaaS,
which optimizes resource cost efficiency for machine learning inference
applications with evolving dynamic requirements [199]. Zhang et al. propose a
solution for auto-scaling in ML-as-a-Service using an LSTM network for
workload prediction and a heuristic method for optimal instance provisioning
decisions [200]. The ability to respond quickly with guaranteed response times
is a key requirement in cloud-native platforms. Somma et al. present a fast
resource provisioning method consisting of deploying containers responsible
for application services and auto-scaling resource allocations among
containers [221]. Shillaker et al. design Faaslets, a lightweight horizontal-
scaling approach for containers in clusters [201]. In the context of
virtualized network functions (VNFs) in cloud-native systems, auto-scaling
techniques have a significant impact on on-the-fly provisioned infrastructure
performance. Salhab et al. propose a framework to address resource
provisioning on-demand for the 5G core network through auto-scaling of
constrained resources [222]. Akhtar et al. design a horizontal scaling manager
on virtualized infrastructure to balance traffic workload for security network
functions [202]. These research efforts contribute to addressing the
challenges of cost-effectiveness, fast scaling, and resource provisioning in
horizontal scaling for cloud-native environments, benefiting applications with
improved efficiency, responsiveness, and performance.

Vertical. Vertical scaling, which involves adjusting resources within an
individual VM such as CPU, RAM, and storage, allows applications to modify
their serviceability. Recent studies have focused on optimizing the cost and
efficiency of vertical scaling in cloud-native environments [203, 204, 205,
206]. To achieve cost-effective resource allocation in vertical scaling, Russo
et al. propose MEAD, which utilizes a prediction algorithm based on Markovian
Arrival Processes to handle bursty workloads, along with an auto-scaling
module for resource allocation [203]. Lakew et al. address the resource
allocation problem by employing a fine-grained vertical scaling technique that
adapts to varying workloads in cloud-native systems [204]. Efficiency
improvement in vertical scaling has also been a focus of research. Sfakianakis
et al. introduce LatEst, a vertical scaling strategy that predicts bursts in
serverless cloud systems and allocates resources efficiently within minimal
time [205]. Tesfatsion et al. aim to increase resource usage and reduce energy
consumption through long-term optimization using vertical scaling techniques
[206]. In the context of avoiding overloaded Virtualized Network Functions
(VNFs), Fei et al. propose an approximation algorithm that minimizes the
prediction error caused by VNF workload, followed by the implementation of a
vertical scaling technique to achieve load balancing for VNFs [207]. These
studies contribute to the optimization of cost, efficiency, and workload
management in vertical scaling, enabling applications to adapt their resource
allocation dynamically within individual VMs in cloud-native environments.

Hybrid. Hybrid scaling is a method commonly used in cloud-native networks that
combines horizontal and vertical scaling mechanisms simultaneously. This
approach leverages the advantages of both horizontal scaling and vertical
scaling, making it more flexible and robust in managing resource provisioning.
To achieve efficient resource provisioning in hybrid scaling, Shahidinejad et
al. utilize the Imperialist Competition Algorithm (ICA) and K-means methods to
evaluate the workload from users. Based on these evaluations, they make
optimal decisions using a combination of horizontal and vertical scaling
techniques [223]. Avgeris et al. employ a control-theoretic approach to
establish a hybrid scaling method that maximizes the number of offloading
requests in a cloud-native edge network, aiming for efficient resource
allocation [208]. In terms of minimizing resource costs, Mahmud et al. propose
a framework that integrates a latency-aware and deadline-satisfied strategy in
a hybrid scaling approach. This framework optimizes the number of edge nodes
required to meet application requirements while minimizing resource expenses
[209]. Schuler et al. introduce a reinforcement learning-based algorithm to
minimize resource provisioning in serverless environments. By adopting a
hybrid scaling method, they dynamically adjust resources to meet the dynamic
demands of users while optimizing resource allocation [210]. These studies
demonstrate the benefits of hybrid scaling in cloud-native networks, allowing
for efficient and cost-effective resource provisioning by combining horizontal
and vertical scaling techniques.

####  VI-C2 Tools and Systems

The typical tools, plugins, and systems that are used for auto-scaling are
listed as follows.

  * •

HPA [224] is a fundamental horizontal-scaling strategy in k8s framework, with
the target of re-allocating resources for the dynamic workload to satisfy its
demand.HPA can respond to the increasing workload by running more Pods to
support overloaded traffic. On the contrary, due to the decreasing workload,
HPA releases its Pods to the configured minimum.

  * •

AWS Lambda function scaling[225] supports a commercial scaling method in the
service of serverless function. Lambda can invoke a scaling strategy to avoid
an overloaded service supply when the incoming traffic increases.

  * •

Knative Pod Autoscaler(KPA)[226] is an auto-scaling method supported in the
recently popular framework Knative. KPA offers the automated scaling of
applications to fit incoming demand, even for the clusters.

###  VI-D Challenges and Research Opportunities

In cloud-native environments, load balancing, service migration, and auto-
scaling are essential. Load balancing optimizes resource utilization, prevents
congestion, and manages workloads efficiently by considering factors such as
resource allocation granularity, migration time, workload detection, and
algorithm efficiency. Service migration in edge-cloud environments focuses on
improving QoS, ensuring network connection continuity, and overall efficiency.
Auto-scaling in cloud-native platforms involves determining the optimal
monitoring interval, selecting appropriate metrics for scaling decisions, and
making accurate and efficient decisions based on system states and workload
predictions. We summarize the main challenges in these three key problems as
follows.

  * •

Heterogeneous workloads. Different workload has different resource demand for
computing and bandwidth. The resource allocation granularity is a key for the
performance of load balancing. Allocating too many resources leads to a waste
while allocating too few resources causes congestion.

  * •

Congestion detection. Developing efficacious algorithms to predict the unknown
workload is a vital issue in cloud-native. Efficient load detection can avoid
network resource congestion, especially in a resource-constraint environment.

  * •

Configuration management. Migrating services often involves configuring
multiple components (e.g., databases, web servers) to work together
seamlessly. Keeping track of configurations and ensuring they are properly
migrated can be challenging.

##  VII Service Maintenance

Service maintenance collects and analyzes service and system indicators,
adjusts and develop strategies, and performs fault recovery. This section is
organized as Fig. 8 shows. After introducing the data collection, we describe
the research status of data analysis of cloud and the evolution based on data
analysis. We summarize and classify the representative works based on the main
purpose of them, as shown in Table V.

TABLE V: Representative works in service maintenance. Work | QoS | Server Utilization | Risk Control | Cost and Profit
---|---|---|---|---
[227, 228, 229] | ✓ |  | ✓ |
[230] |  | ✓ |  | ✓
[231] | ✓ | ✓ |  | ✓
[232] | ✓ | ✓ |  |
[233, 234] | ✓ | ✓ |  |
[235, 58, 236, 237, 238] | ✓ |  | ✓ |
[239, 240] |  |  | ✓ | ✓
[241] | ✓ |  | ✓ |
[242, 243] |  |  | ✓ |
[244, 245, 246] |  |  | ✓ | ✓
Figure 8: Organization of service maintenance.

###  VII-A Data Collection

Data collection aims to collect metrics of cloud service and physical
infrastructure to guide strategy development. There are mainly two aspects to
collect and analyze the performance of cloud service: firstly, monitor the
whole physical infrastructure of cloud environment. Secondly, monitor the
performance of each service. For the former one, the healthy state of disks,
the usage of memory, processors, network bandwidth are usually focused on.
There are lots of mature tools to detect the metrics of cloud infrastructures:
such as DX [247], CA [248], etc. When analysing the performance of cloud
services, there are mainly two kinds of contexts: the temporal context and the
operation context [232]. The operation context means the metrics directly
related to the service at each level, from software to the kernel. For
example, the cache, processors, memory usage directly caused by the monitoring
service. Besides, the temporal context is also important. The temporal context
is the behavior of the services which have resource competition relationship
with monitoring service. The challenge for collecting the operation context is
that it is difficult to trace the the operation from layer to layer (i.e. from
service layer to kernel layer). DTrace can solve this problem by instrumenting
all code [249]. Besides, Ardelean et al. propagate the operation by system
call _getid_. _getid_ will ignore all the arguments passed to it, but the
kernel trace will record the arguments. Thus, they use the arguments to inject
operation information. As for temporal context, Magpie [250] collects all the
requests across multiple nodes. But it is not suitable for the cloud
environment with billions users. To reduce the overhead of Magpie, a common
way is to use bursty tracing [251], which just samples partial temporal
context.

###  VII-B Data Analysis

Data analysis at service maintenance state is to mine the the property and
regular patterns of running jobs from monitoring indexes and guide jobs
allocation and scaling. Generally, there are three main research directions
for data analysis: analyze and summarize the characteristic of running jobs
and anomalous events to provide better understanding of cloud services,
predict the future state of the cloud services, detect cloud service anomaly
and find root cause automatically. We organize this section as following: in
section VII-B1, we try to answer the two questions: what kinds of data is
worth and suitable to analyze, what kinds of analysis are useful for cloud
providers? In section VII-B2, we introduce the research status of system state
prediction, which includes the prediction of workload, prediction of healthy
state of disk, prediction of service failures. In section VII-B3, we introduce
the main challenges of anomaly detection for cloud services and the research
states for each challenges.

####  VII-B1 Summarize Cloud Services Characteristic

It is important to summarize the characteristic of cloud systems and the
running cloud jobs, as it provides valuable insight to improve utilization of
servers and quality of services. There are two important questions to
summarize the characteristics. Firstly, what kind of data would contain
valuable information and suitable to be analyzed? Secondly, researchers and
cloud service providers are interested on what kind of characteristics? For
the first question, Hauswirth et al. claimed the virtualization introduced by
cloud native environment provides a significant challenge to understanding
complete system performance, not found in traditionally compiled languages
[252]. Thus, they proposes vertical profiling to provide profiling of all
level of the execution stack. Vertical profiling can just apply to java-based
applications. To make it more general, Ardelean et al. extend it to
application based on any language [232]. Besides, code snippets [253],
functional-level variance [254], control flow [255] are also suitable data.
For the second question, characteristic summarizations usually concentrate on
reducing the cost of providers, improving the quality of services and the
healthy state of cloud systems. We list some of the most popular topics below:
the different and similar impacts of different failures [227], the causes of
failures [228, 229], the causes of low utilizations [230], pricing strategy
analysis [231], the analysis of time-varying mixture load [232].

####  VII-B2 Future State Prediction

The cloud system and service future state prediction on the one hand can alarm
forthcoming failures and risks so as to prevent them. On the other hand, it
can provide basic information for task scheduling, auto-scaling, service
migration and so on to improve the utilization of cloud infrastructures. The
prediction of future system state focuses on three aspects: the workload, the
healthy state of cloud servers, the forthcoming service failures.

Workload prediction. Workload prediction focus on predicting the future
processor usage, memory usage and bandwidth usage, which can help to improve
the quality of service(Qos) as well as improve the utilization of cloud
servers. There are mainly two kinds of workload prediction methods: the
statistical methods and neural-network based methods. For statistical methods,
AR [256], MA [256], ARIMA [257], Bayesian models [258] are used to predict the
future workloads. Among them, ARIMA is one of the most popular and classical
methods, which assumes the value at present is affected by the trend
information, the history values and some noises. One of the problem of using
ARIMA is that the workload of different jobs can have different regular
patterns and it will lead to low accuracy to predict the workload by a single
model. Thus, an adaptive statistic model [259] is proposed to solve this
problem, which combines linear regression, ARIMA, and support vector
regression.

Recently, it is reported that these statistic methods rely on strong
mathematical assumptions (e.g. ARIMA is based on the assumption that the time
series should be stationary after difference), and predict inaccurately when
the workload is highly variable [234]. Thus, many researchers turn to neural-
network based methods such as RNN [260], LSTM [261], etc. LSTM is one of the
classical time series prediction methods, which can capture both the long-term
dependent information and the short-term dependent information. But these
recurrent network based methods give the same weights to the workload in
observing window, while the history workload has different impact on
predicting workload. Thus, a method combines LSTM and attention mechanism is
proposed to put different weight on history workload [233]. Besides, another
problem of using recurrent network based methods is the forgetting effect
[262] when extracting long-term dependent information. Thus, a method [234]
combines top-sparse auto-encoder and GRU is proposed to effectively extract
the essential representations of workloads from the original high-dimensional
workload data and predict highly variable workload accurately.

Healthy state of cloud servers prediction. Researchers in this domain mainly
focus on disk dives failure prediction, as it can dramatically reduce data
restoring time to predict disk failures in advance. Disk drives failure
prediction plays a very important and crucial role in reducing data center
downtime and significantly improving service reliability [58], as it alarms
forthcoming disk drives failure and the system can overlap the time of regular
data operation and the time of data restoring. At the beginning, the task of
disk drives failure prediction is regarded as a binary classification problem
and lots of classification models are used to predict the disk failure, such
as Bayesian models [263], Wilcoxon rank-sum test [264], support vector
machines (SVM) [265] artificial neural network (ANN) [266], etc.

However, these methods have reported poor performance on real-world
environment. Firstly, the status of disk drives corrupt gradually and is not
only either good or bad [235] [58]. Thus, Aniello et al. propose a method
firstly divide the healthy status of disks drives into seven levels by
regression tree, and then use LSTM to predict the disk healthy status in the
future [58]. The fine-grained disk drivers healthy status support more
flexible data-restore mechanism, which can plan data restoring in advance
according to the different prediction of fine-grained disk drivers healthy
status. However, LSTM used in this work is recurrent networks and it is
reported to be vulnerable to he highly variant interval between triggering
events and hardware failures [236]. Thus, Sun et al. use the temporal
Convolution Neural Network (CNN) to leverage CNN’s characteristic of
translation invariance, which can make the CNN insensitive to various delays
between triggering-and failure-events in the time dimension [236]. Secondly,
the data imbalance between disk failure data and normal data hinders the
models to predict accurately. Thus, Sun et al. also design a new loss function
to prevent the gradient vanishing in front of the huge data imbalance [236].
Moreover, the above methods are based on offline training and can not adapt to
the continuous update systems [237]. Thus, Xiao et al. proposes a method based
on online random forest algorithm to maintain stable predicting accuracy for
long-term usage.

Service failure prediction. Different from the above sections, service failure
prediction focus on service Qos and predict the failure from the level of
service. The service failure can lead to penalty payments, profit margin
reduction, reputation degradation, customer churn and service interruptions
[238]. Thus, it is worthwhile to know the possible failure in advance. By
doing so the cloud systems can take steps to prevent the predicting failures.

Generally, service failure prediction can be divided to three categories: rule
based methods, statistic methods, deep learning methods. Rule based methods
rely on manually defined rules to predict the failures, which are limited to
the human experience and its adaptability is poor. The other two methods are
data-driven method, compared with rule based methods, they are more flexible
and convenient.

The rule based methods require experts define specific rules in advance. For
example, PerfAugur [267] is designed to predict failures by specified
features. Generally, these methods are accurate but just suitable for specific
scenarios.

The statistic methods include ARMA [268], ARIMA [269], SVM [270], Hidden Semi-
markov Model [271], etc. Among the classical statistic methods, Cavallo et al.
[272] have claimed that ARIMA forecasting has the best compromise in ensuring
a good prediction error, being sensible to outliers, and being able to predict
likely violations of QoS constraints [273]. However, Amin et al. [273] point
out the traditional ARIMA model can not deal with the high volatility of
quality of service (QoS) properly. Thus, they propose a model integrate GARCH
and ARIMA to solve this problem.

However, statistic methods rely on some mathematics assumptions and can not
work well on high dimension feature and dependent sequence data. Thus, many
researchers apply the deep learning models to service failure prediction. Chen
et al. [274] propose a deep learning methods based recurrent neural networks
(RNN) to predict task-level failures. However, the drawback of RNN is that it
will definitely forget the information in long distance, which will degrade
the predicting accuracy. Although some modified recurrent neural networks,
such as LSTM [275], can mitigate this effect, the weights put on each value in
observing window is unequal and degrade as the distance goes farther. Thus,
Gao et al. [276] propose a method based on bi-direction LSTM to further
improve the accuracy.

####  VII-B3 Anomaly Detection

In the cloud native scene, researchers detect anomaly from different aspects:
components anomalous usage (e.g. anomalous processor and memory usage, network
attacks, disk drives failures), service anomalous Qos (e.g. high latency and
low throughput). It is worth to notice that though anomaly detection and
system state prediction both study different component usages and Qos, the
prediction is to infer the forthcoming system states, while anomaly detection
is to discover the anomaly already happens. The organization of this
subsection is illustrated in Fig.9. There are three main challenges when
detecting anomaly: labeled data obtaining issues, high variance of cloud
environment, alert storm. The labeled data obtaining issues is caused by the
requirements of expertise experience and lots of efforts to label the
anomalous data. The high variance of cloud environment suggests that the
detecting patterns the models learned from history data may become outdated
frequently. Besides, there are lots of APIs and components in cloud
environment. Lots of them are relevant to each other. When anomaly occurs to
one part, the related parts will also be abnormal. Thus, when an anomaly
occurs, the system always suffers from a alerting storm. This phenomena
suggests the necessity of root cause finding.

Figure 9: Organization of anomaly detection.

Data labeling issue. The anomalous data labeling in cloud is different from
labeling tasks in many other domains (e.g. image recognition), as it requires
expertise experience to label the anomaly, which makes the labeled data rarer.
There are mainly four kinds of methods to solve it: semi-supervised learning
[277, 278, 279, 280], unsupervised learning [281, 282], transfer learning
[239, 283, 284], active learning [239, 240]. Semi-supervised learning is
suitable for the datasets with small amount of labeled data. It firstly uses
the labeled data to train a primary model. After that, it uses the primary
model to assign pseudo label to the remaining unlabeled data. Then, it uses
the data with pseudo label and data with label to retrain the primary model.
If we not only have a dataset with small amount of labeled data, but also have
experts to label some data in the process of model training, we can use the
active learning to further improve the detecting accuracy. Active learning
firstly uses the labeled data to train a primary model. After that, it uses
the primary model to pick a small part of unlabeled data which need label most
and label the remaining data with pseudo labels. Then, experts label the data
picked by the primary model. Then, it retrains the primary model with newly
labeled data, labeled data, pseudo labeled data. However, if we have none
labeled data at all, we can use transfer learning or unsupervised learning.
Both of them rely on assumptions: transfer learning assume there are some
similarities between source dataset and target dataset, unsupervised learning
assume the normal state has some unified and invariant latent regularity that
can be learned by models. Autoencoder [285] is one of the most classical
unsupervised learning models in anomaly detection. It firstly compresses the
features into a smaller latent variable. After that, it reconstructs the
features from the latent variable. In training process, it makes the
reconstructed features as similar as possible to the original features. It
assumes in the training process, autoencoder can learn the reconstructing
patterns of normal data and when the autoencoder meets anomalous data in
inferring process, it will fail to reconstruct. By then, autoencoder can
detect anomalies. As each feature in cloud anomaly detection is time
dependent, recently, some researchers combine the recurrent neural network
with the autoencoder to capture the correlation between features as well as
the time dependence [286, 287].

Generally speaking, the detecting accuracy of active learning, semi-supervised
learning, unsupervised learning and transfer learning decreases one by one, as
well as the inputting human efforts. Therefore, the choice of model depends on
the constraints and main objectives in the practical environment.

Cloud environment high variance issue. The cloud environment is highly
variable. For example, Google and baidu are reported thousands of software
changes are deployed everyday [241]. These changes will continuously degrade
the predicting accuracy of anomaly detection models trained by outdated data.
Frequent model retraining is costly and impractical. This problem calls for
lightweight and self-evolving algorithms. The data-driven anomaly detection
methods on the cloud scene can be roughly divided to two streams: one is based
on time dependent information, the other is based on the correlation between
different metrics. The former one is to capture the time-dependent normal
pattern from time series and compare the tested data with the normal pattern,
while the latter one is to capture the normal correlation of different metrics
and compare the correlation of tested metrics with the normal correlation. For
the former anomaly detection methods, Xu et al. [243] propose a method based
on online machine learning to mitigate the negative effect of cloud
environment variance. Besides, they also propose a reduction method to further
reduce the overhead of retraining and inferring. For the latter anomaly
detection methods, Peng et al. [242] propose a method uses cuboid structure to
store the relationship of different metrics, which significantly reduce the
storing overhead. Besides, they also propose a incremental learning method
suitable for cuboid structure, which significantly reduce the retraining
overhead. However, it is reported the incremental retraining methods need lots
of data points to converge [288] and still need more data to reach steady
state [289], which requires the cloud system collect enough data points before
retraining the models. The data collecting time is long (generally tens of
days) and there will be a period called initial time [241] when the accuracy
of old model is low and there is no enough data to apply the incremental
learning. To reduce the initial time, Ma et al. [241] propose a quickly start
method which can work well without relying on as big amount of data as
incremental learning and report high accuracy.

Alert storm issue. Cloud system is consisted of thousands of components with
extraordinary complex dependency [244]. Besides, business transactions in a
cloud native system usually have a much longer calling path with dozens of
distributed microservices participating [245]. Thus, an anomaly in one part
will trigger lots of anomaly alerts in related components and tasks calling
the anomalous task, which is called Alert storm [245]. This problem calls for
root cause finding techniques. There are mainly three kinds of root cause
finding techniques: manual-rule based methods, causal inference based methods,
reconstruction based methods. Manual-rule based methods are based on expertise
experience, which is accurate though human-work costly and easily-outdated.
For example, Demirbaga et al. [246] defined three kinds anomaly cause: data
locality (i.e. data needed is not at the same server as task), resource
heterogeneity (i.e. jobs are scheduled to machine remaining few computing
resource), network failure (i.e. network disconnection). They detect these
anomaly by defining threshold of several metrics, when the defining combined
metrics exceed the thresholds, then corresponding anomaly is detected. Due to
the limitations of manual-rule based methods, researchers also pay attention