# Deep Reinforcement Learning-based Methods for Resource Scheduling in Cloud
Computing: A Review and Future Directions

Guangyao Zhou<EMAIL_ADDRESS>School of Information and
Software Engineering, University of Electronic Science and Technology of
ChinaCheng DuChina ,  Wenhong Tian<EMAIL_ADDRESS>School of
Information and Software Engineering, University of Electronic Science and
Technology of ChinaCheng DuChina and  Rajkumar Buyya<EMAIL_ADDRESS>ICloud Computing and Distributed Systems (CLOUDS) Laboratory, Department of
Computing and Information Systems, The University of
MelbourneMelbourneAustralia

###### Abstract.

As the quantity and complexity of information processed by software systems
increase, large-scale software systems have an increasing requirement for
high-performance distributed computing systems. With the acceleration of the
Internet in Web 2.0, Cloud computing as a paradigm to provide dynamic,
uncertain and elastic services has shown superiorities to meet the computing
needs dynamically. Without an appropriate scheduling approach, extensive Cloud
computing may cause high energy consumptions and high cost, in addition that
high energy consumption will cause massive carbon dioxide emissions. Moreover,
inappropriate scheduling will reduce the service life of physical devices as
well as increase response time to users’ request. Hence, efficient scheduling
of resource or optimal allocation of request, that usually a NP-hard problem,
is one of the prominent issues in emerging trends of Cloud computing. Focusing
on improving quality of service (QoS), reducing cost and abating
contamination, researchers have conducted extensive work on resource
scheduling problems of Cloud computing over years. Nevertheless, growing
complexity of Cloud computing, that the super-massive distributed system, is
limiting the application of scheduling approaches. Machine learning, a utility
method to tackle problems in complex scenes, is used to resolve the resource
scheduling of Cloud computing as an innovative idea in recent years. Deep
reinforcement learning (DRL), a combination of deep learning (DL) and
reinforcement learning (RL), is one branch of the machine learning and has a
considerable prospect in resource scheduling of Cloud computing. This paper
surveys the methods of resource scheduling with focus on DRL-based scheduling
approaches in Cloud computing, also reviews the application of DRL as well as
discusses challenges and future directions of DRL in scheduling of Cloud
computing.

Cloud Computing, Deep Reinforcement Learning, Resource Scheduling

††copyright: none††journalyear:  ††booktitle:

##  1\. Introduction

Cloud environment is general accepted as a set of distributed systems linked
by high-speed network including the applications delivered as services over
the Internet, the hardware and systems software that can dynamically provide
services to users(b21, ; b30, ; b80, ). As a paradigm that provides services
to users according to users’ demand in pay-as-you-go(b73, ; b76, ; b81, )
manner or pay-per-use(b97, ; b145, ) to the general public, Cloud computing
usually has four forms that include three traditional types of services,
Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software
as a Service (SaaS)(b21, ; b80, ; b97, ; b125, ; b128, ) as well as a new form
of serverless computing(b32, ; b80, ).

As an elastic, reliable, dynamic services provider, Cloud computing provisions
computing resources on the basis of CPU (Central Processing Unit)(b80, ; b71,
; b169, ), RAM (Random Access Memory)(b103, ; b138, ), GPU (Graphics
Processing Unit)(b181, ; b83, ), Disk Capacity(b80, ; b169, ) and Network
Bandwidth(b103, ; b139, ). From another perspective, ”time”, that the whole
service life cycle of Cloud computing platform, and ”space”, that the real
physical place to emplace Cloud computing physical devices, are also two
pivotal resources of Cloud computing. Each electrical components of Cloud
computing’s devices, driven by electric energy and working at the time and the
space, constitutes the real resources assemble of Cloud computing. Hence in
fact, real natural resources provided by Cloud computing are effective
electric energy conversion per unit time and per unit space, regarding energy,
time and space as the most essential resources. Therefore, the limited
resource utilization capacity of Cloud computing will raise the cost of Cloud
providers and energy consumption of Cloud computing system. Moreover,
considering that user’s time and resource’s usage time are both crucial
components of society, long response time, long queuing time and high delay
rate will direct energy consumption of another social form. Consequently, how
to schedule components of Cloud computing in an efficient, energy-saving,
balanced method, is proved to be a critical factor influencing the orientation
of Cloud computing in society. Whereas, the huge scale of Cloud computing, the
complexity of scenarios, the unpredictability of user requests, the randomness
of electronic components, and uncertain temperature of various components
presented in the running process, pose challenges to efficient and effective
scheduling of Cloud computing in the emerging trend(b69, ; b71, ; b81, ; b105,
; b137, ; b138, ; b139, ; b193, ). To lead more precise comprehension of Cloud
computing, we will briefly review the development history of Cloud and its
scheduling approaches.

Additionally, Multi-phase approach(b56, ; b71, ; b104, ; b146, ; b161, ),
Virtual machine migration(b76, ; b102, ; b105, ; b137, ), Queuing model(b46, ;
b58, ; b73, ; b81, ; b101, ; b105, ; b110, ; b139, ; b167, ), service
migration(b113, ; b187, ; b97, ), workload migration(b67, ), application
migration(b97, ; b81, ), task migration(b68, ; b76, ; b90, ) and scheduling
algorithm of scheduler are current common strategies to resolve the scheduling
of Cloud computing’s components. Even so, the core of solution is still the
design of the scheduler on the basis of scheduling algorithm. Fig.1 shows a
resource management and task allocation process with scheduler as the core.
Hence, scheduling algorithms of Cloud computing have attracted a great
quantity of researchers. The scheduling problem in distributed systems is
usually a NP-complete problem(b76, ; b44, ) or a NP-hard problem(b80, ; b144,
; b139, ). Existing methods to resolve scheduling problems contain six
categories including Dynamic programming, Probability algorithm, Heuristic
method, Meta-Heuristic algorithm, Hybrid algorithm and Machine learning
method.

Figure 1. Resource management and task allocation process based on central
scheduler

There are abundant discussions about the application of machine learning in
cloud computing resource scheduling such as work from Microsoft(b201, ),
CLOUDS Laboratory of The University of Melbourne(b193, ; b202, ), and other
reseach(b81, ; b92, ; b186, ; b198, ; b112, ). Deep reinforcement learning
(DRL), a novel approach belonging to ML and combined with advantages of deep
neural network and reinforcement learning, is utilized to solve the scheduling
problems of Cloud computing in recent years, which has been proved to occupy
strong superiorities in many scenarios especially in complex scenes of Cloud
computing(b107, ; b150, ; b153, ; b161, ; b164, ; b168, ). As the application
of DRL in resource scheduling of Cloud computing began in recent years, these
surveys(b42, ; b76, ; b80, ; b81, ; b92, ; b94, ; b97, ; b98, ; b99, ; b102, ;
b104, ; b119, ; b182, ; b183, ; b184, ; b185, ; b186, ; b198, ), provided
detail, comprehensive and valuable reviews of Cloud computing, yet did not
specifically discuss the application of DRL in resource scheduling of Cloud
computing, although some surveys have discussed the application of machine
learning in cloud computing(b198, ; b81, ; b92, ; b186, ). Researchers are
still exploring the application pattern of RL especially DRL in scheduling of
Cloud computing(b150, ; b151, ; b176, ; b177, ; b169, ; b107, ; b153, ; b83, ;
b161, ; b168, ; b203, ). Similarly, RL is also applied to solve scheduling
problems in other field(b195, ; b199, ). Aiming at providing a specific vision
for scheduling methods of Cloud computing and provisioning integrated
information reference of DRL in resource scheduling of Cloud computing, we
focus on the reviews of research using RL especially the DRL to solve
scheduling problems of Cloud computing and finally target challenges and
future directions using DRL to adapt more realistic scenarios of resource
scheduling in Cloud computing.

The main contributions in this paper can be summarized as follows:

  1. (1)

We briefly review the development of Cloud computing and categorize scheduling
methods and optimization objectives in Cloud computing.

  2. (2)

We summarize models in the scheduling of Cloud computing for various
objectives.

  3. (3)

We discuss the application of DRL in scheduling of Cloud computing compared
with non-DRL methods through reviewing applications of DRL in Cloud computing.

  4. (4)

We identify and propose challenges and future directions of DRL in scheduling
problems of Cloud computing and target a specific idea to use DRL to solve
resource scheduling.

The rest of the paper is organized as follows: Section 2 briefly reviews the
development history of Cloud computing and its scheduling. Section 3 discusses
the terminologies and establishes the mathematical models for various
objectives of scheduling problems in Cloud computing. According to the
classification of classic methods and machine learning methods, Section 4
sorts out the existing approaches utilized in scheduling of Cloud computing.
Structure analysis of RL and DRL applied in the scheduling of Cloud computing
is shown in Section 5. Literature review utilizing RL especially DRL to
address resource scheduling of Cloud computing is presented in Section 6.
Future directions of DRL applied in scheduling of Cloud computing are
discussed in Section 7. Finally, Section 8 summarizes and concludes the paper.

##  2\. Overview of Cloud computing and its scheduling

###  2.1. Concept development of Cloud computing

The concept of Clouds distributed system can be found in some papers of 1980s
to 1990s(b2, ; b3, ; b4, ; b5, ; b7, ). In 1983, Allchin presented a nested
action management algorithm considering that constructing reliable programs
for distributed processing systems is a task with difficulty(b1, ). This
marked an architecture was described, which proposed the construction of
reliable and distributed operating systems. Theoretically based on Allchin’s
reliable algorithm, Dasgupta et al. provided a functional description of
Clouds distributed system in 1988(b2, ). In (b2, ), Dasgupta et al. mentioned
that ”Clouds is designed to run on a set of general purpose computers
(uniprocessors or multiprocessors) that are connected via a medium-to-high
speed local area network”. In 1989, Bernabeu-Auban et al. described the
architecture of Ra which is a kernel for Clouds and can be used to support the
development of an object-based distributed operating system(b3, ). In (b3, ),
Ra virtual space was proposed to structure Clouds objects and distributed
operating systems which used the object-thread model of computation. In 1991,
Dasgupta et al. expounded Clouds distributed operating system based on object-
thread model, regarding Clouds as a paradigm, also a general-purpose operating
system for structuring distributed operating systems as well as described
potential and implications of this paradigm(b4, ). Same in 1991, Raymond C.
Chen et al. presented a kernel-level consistency control mechanism called
Invocation-Based Consistency Control (IBBC) as part of Clouds distributed
operating system to support general-purpose persistent object-based
distributed computing(b5, ). As a set of flexible mechanisms that controls
recovery and synchronization, IBBC had three types of consistencies as
defaults: global, local, and standard. The same year, Pearson et al. presented
Clouds LISP distributed environments and proposed CLIDE environment
structure(b6, ). In 1992, Gunaseelan et al. designed a language, that
Distributed Eiffel, for programming multi-granular distributed objects on
Clouds operating system(b7, ). The design in (b7, ) addressed a number of
issues such as parameter passing, asynchronous invocations and result
claiming, and concurrency control, which makes it possible to combine and
control both data migration and computation migration effectively at the
language level.

With the exponential development of the Internet, its users have an increasing
demand for computing. More concepts of distributed computing have been
proposed or been developed such as grid computing(b10, ; b11, ), peer to peer
computing(b12, ; b13, ), and utility computing(b15, ; b16, ). With the advent
of network information age, Web 2.0(b28, ; b29, ), network information flow
presents the characteristics of big data, high speed and multi-function. The
concept of Cloud computing was in formally presented in the search engine
conference in August 2006. Amazon, Google, IBM, Microsoft and Yahoo, who are
the forerunners that provide Cloud computing services, contributed to
demonstrate Cloud computing(b21, ; b25, ; b26, ). In 2008, Microsoft released
Windows Azure Platform(b21, ; b27, ). Until the International conference
Cloudcom 2009(b18, ) started, three categories of services, IAAS, PAAS, and
SAAS provided by Cloud computing, had been recognized by researchers(b19, ;
b20, ).

In reference (b28, ) which gave a Berkeley view of Cloud computing in 2009,
Armando Fox et al. deemed the conditions, that are widespread broadband
Internet access, the capacity of providers, the suitable billing model and
development of efficient virtualization techniques, would make Cloud computing
available. Meanwhile, experts began to discuss the relationships and
differences between Cloud computing and grid computing(b10, ; b31, ; b194, ).
In (b10, ), Foster et al. compared Cloud computing and grid computing in
various aspects including business model, architecture, resource management,
programming model, application model and security model. Foster et al. added a
definition for Cloud computing: a large-scale distributed computing paradigm
driven by economies of scale, in which a pool of abstracted, virtualized,
dynamically-scalable, managed computing power, storage, platforms, and
services are delivered on demand to external customer over the Internet(b10,
). Before this definition, there had been many definitions of Cloud computing
but there still was little consensus on how to define the Cloud(b30, ). In
Foster’s definition, Cloud computing differs from traditional distributed
computing paradigms in that it has massive scalability, can be encapsulated as
an abstract entity that delivers different levels of services to customers, is
driven by economies of scale and has dynamical services(b10, ). In (b31, ),
Klems et al. held that Cloud computing is an emerging trend of provisioning
scalable and reliable services over Internet as computing utilities as they
deemed that the methods and the business models introduced for grid computing
from the previous works of distributed computing do not consider all economic
drivers which are identified relevant for Cloud computing, such as pushing for
short time to market in the context of organization inertia or low entry
barriers for start-up companies.

A definition of Cloud computing that ”Cloud computing refers to both the
applications delivered as services over the Internet and the hardware and
systems software in the data centers that provide those services” was gave in
(b21, ), where Armbrust et al. demonstrated the advantages of public Cloud
comparing with private data centers. From Armbrust et al., public Cloud has
the characteristics including appearance of infinite computing resources on
demand, elimination of an up-front commitment by Cloud users, ability to pay
for use of computing resources on a short-term basis as needed, economies of
scale due to very large data centers, simplifying operation and increasing
utilization via resource virtualization that the Conventional Data Center does
not or usually not possess(b21, ). In (b194, ), Buyya et. al. proposed one of
the popular definitions of Cloud computing and demonstrated how it enabled
emergence of ”computing as 5th utility”. A High-level Market-oriented Cloud
architecture, consisting of users/brokers, SLA resource allocator, VMs and
physical machines, was presented in (b194, ).

###  2.2. Introduction of scheduling in Cloud computing

In distributed systems, the scheduling problem is NP-complete generally(b44,
). Some of the primitive research about task allocation, resource scheduling
and resource management of distributed systems or multiprocessor systems had
appeared in 1970s-1980s(b38, ; b41, ; b42, ). The early methods of specifying
a schedule include Graph theoretic approach(b41, ; b38, ; b42, ), minimal-
length preemptive schedule(b41, ), Meta-Heuristic Models(b47, ; b48, ; b42, ),
zero-one integer programming approach(b38, ), Branch-and-bound techniques(b43,
), maximum flow method(b42, ), Gomory-Hu cut tree technique(b42, ) etc..

Table 1. A summary of objectives discussed in reviewed papers Objectives |  References
---|---
Energy consumption minimization (Energy efficiency) |  (b72, ; b165, ; b114, ; b135, ; b113, ; b151, ; b153, ; b58, ; b91, ; b82, ; b57, ; b157, ; b137, ; b71, ; b142, ; b136, ; b147, ; b89, ; b88, ; b68, ; b74, ; b125, ; b67, ; b65, ; b64, ; b124, ; b123, ; b62, ; b61, ; b170, ; b189, ; b176, ; b177, ; b190, )
Makespan minimization (execution time minimization) |  (b162, ; b72, ; b163, ; b164, ; b93, ; b165, ; b60, ; b114, ; b135, ; b84, ; b117, ; b56, ; b83, ; b82, ; b57, ; b71, ; b148, ; b89, ; b79, ; b75, ; b78, ; b74, ; b121, ; b130, ; b131, ; b120, ; b64, ; b124, ; b122, ; b59, ; b132, ; b96, ; b95, ; b85, )
Delay time (or delayed services) minimization |  (b93, ; b135, ; b73, ; b84, ; b56, ; b91, ; b90, ; b88, ; b168, )
Response (or running) time minimization |  (b72, ; b113, ; b111, ; b107, ; b58, ; b69, ; b101, ; b79, ; b125, ; b65, ; b169, ; b203, )
Load balancing maximization |  (b162, ; b112, ; b84, ; b151, ; b56, ; b83, ; b82, ; b68, ; b65, ; b85, ; b190, )
Reliability or Utilization maximization |  (b72, ; b93, ; b113, ; b84, ; b117, ; b108, ; b151, ; b58, ; b82, ; b157, ; b142, ; b69, ; b147, ; b89, ; b127, ; b121, ; b122, ; b59, ; b123, ; b169, ; b168, )
Profit maximization (Economic Efficiency, cost minimization) |  (b162, ; b163, ; b60, ; b135, ; b113, ; b111, ; b110, ; b108, ; b151, ; b82, ; b57, ; b157, ; b138, ; b139, ; b140, ; b141, ; b70, ; b69, ; b145, ; b147, ; b148, ; b89, ; b75, ; b78, ; b87, ; b128, ; b65, ; b116, ; b131, ; b59, ; b132, )
Task completion ratio maximization |  (b72, ; b113, ; b101, ; b122, )
SLA Violation minimization |  (b89, ; b65, ; b61, ; b157, ; b113, ; b108, ; b203, )
Throughput maximization |  (b88, ; b79, ; b136, ; b114, ; b111, ; b117, )
Multi-objectives |  (b163, ; b165, ; b114, ; b112, ; b84, ; b117, ; b82, ; b157, ; b158, ; b105, ; b145, ; b147, ; b148, ; b78, ; b74, ; b125, ; b130, ; b131, ; b59, ; b123, ; b132, )

As far as we can investigate, the early research of Cloud computing’s resource
scheduling emerged in about 2008-2009(b44, ; b45, ; b46, ; b47, ; b48, ; b49,
; b50, ), before that research of resource scheduling of distributed computing
systems mainly focused on grid computing(b53, ; b55, ). In (b44, ), Xu et al.
proposed a Multiple QoS Constrained Scheduling Strategy of Multi-Workflows
(MQMW) for Cloud computing. In (b45, ), Li built the system cost function and
non-preemptive priority M/G/1 queuing model for the jobs to aid the strategy
and algorithm to get the approximate optimistic value of service for each job.
In (b46, ), Caron et al. proposed the use of the DIET-Solve Grid middleware on
top of the EUCALYPTUS Cloud system to manage the resources of Cloud platforms.
Garg et al.(b47, ) proposed near-optimal scheduling policies, Meta-Scheduling
Policies, to address the high increase in the energy consumption considering
various contributing factors such energy cost, CO2 emission rate, HPC
workload, and CPU power efficiency. Garg et al. illustrated in (b47, ) that
their carbon/energy-based scheduling policies are able to achieve on average
up to 30% of energy savings in comparison to profit based scheduling policies.
Sadhasivam et al.(b49, ) designed an efficient Two-level Scheduler, Intra
VMScheduler, for Cloud computing environment where the first level was a queue
model and the second was a scheduler. Unlike First Come First Serve (FCFS)
Policy and Round-robin(RR), the VMScheduler was implemented by a typical
queue-based policy known as Conservative Backfilling(b49, ). The approach of
two levels/phases or multi-phases to model the scheduling of Cloud computing
can be seen in recent research(b56, ; b57, ; b60, ). Grounds et al. proposed
Cost-Minimizing Scheduling Algorithm (CMSA) to minimize the cost function
values for all workflows on a Cloud of Memory Managed Multicore Machines(b50,
).

From 2010 to 2020, resource scheduling strategies for Cloud computing has
attracted scholars to research. Some of the mainstreams of the resource
scheduling strategies for Cloud computing focus on objectives of minimizing
energy consumption, minimizing makespan, minimizing delay time, reducing
response time, load balancing, increasing reliability, increasing the
utilization of resources, maximizing the profit of providers, maximizing task
completion ratio, minimizing Service Level Agreement (SLA) Violation,
maximizing throughput, and multi-objectives. Table 1 provides a summary of
objectives addressed in surveyed papers. The above objectives can be
summarized as three aspects, reducing cost and increasing profit of Cloud
providers, improving quality of services (Qos) provisioned to users, and
greening Cloud computing.

##  3\. Terminologies and Different Optimization Objectives of Scheduling
Problems in Cloud computing

###  3.1. Terminologies of scheduling problems in Cloud computing

References (b56, ; b57, ; b150, ; b105, ; b158, ; b139, ; b140, ; b141, ;
b142, ; b70, ; b144, ; b103, ; b167, ; b110, ; b168, ; b73, ) have
demonstrated the description of the symbols and terms related to Cloud
computing and Edge computing to a certain extent. Surveying these references
and combining various scenarios, this paper gives an integration description
of notations and terminologies of scheduling in Cloud computing as shown in
Table LABEL:table2. When the scene becomes complex, many parameters will turn
to time-varying or random. Therefore, this paper uses time-related parameters
to represent various elements. Then considering future modeling, we add the
dimension of parameters.

###  3.2. Modeling of resource scheduling problems in Cloud computing with
different optimization objectives

In surveyed literature, energy consumption minimization, makespan
minimization, delay time (or delayed services) minimization, response time
minimization, load balancing maximization, utilization maximization, cost
minimization and multi-objectives are the main optimization objectives of most
of the scheduling algorithms as shown in Table 1. These objectives have close
affect to the performance of Cloud computing system including quality of
services, total profit of Cloud computing providers and carbon emission. In
general, energy consumption, time-cost and load balancing are the major
foundations to evaluate the performance of a scheduling algorithm in Cloud
computing.

####  3.2.1. Energy consumption optimization problems

Energy consumption optimization problems can be summarized into two types,
total energy consumption and utilization of energy (energy efficiency). The
total energy consumption contains the energies consumed by physical resources
during stand-by time, consumed in processing tasks, consumed in hanging tasks,
and dissipated in the form of heat exchange. In realistic, it is not easy to
measure the energy consumption of each partition especially when there are
multiple tasks on a physical resource at the same time. A more intuitive
measurement index is electricity consumption. Another approach is to treat
tasks that are simultaneously on the same physical resource as mutually
coupled objects and to decoupled by statistic the power consumption under
various scenes.

In addition, the energy consumed by the task can also be approximately
calculated according to the task’s lifetime(b58, ) or can also be regard as a
given value(b80, ). It will be one of the far-reaching prospects of Cloud
computing to study the measurement of energy consumption and the coupling
relationship between multiple tasks. In view of the fact that the power
consumption of a specific task is variable at different times and also
variable at different physical resources, we regard the power consumption of a
task as a time-varying function $PJ(x,t)$ in paper, which can adapt to various
forms of energy calculation in future. For the same consideration, $EPP(y,t)$
and $EPP(y,t)$ of physical resource are assumed as time-varying functions.

Table 2. A list of Notations with Their Meaning Description |  Notations
---|---
Number of physical resources |  $PN$
Physical resource numbered $i$ |  $P_{i}$
Resources Set: $P=\left\\{{{P_{1}},{P_{2}},\ldots,{P_{PN}}}\right\\}$ |  $P$
Capacity of ${P_{i}}$ including CPU, GPU, RAM (b103, ) |  ${C_{i}}$
Number of tasks (requests) |  $JN$
Task numbered $i$ where $i\in[1,JN]\cap\mathbb{N}$ |  ${J_{i}}$
Set of tasks (requests) |  $J$
Initial time of task $x$ |  $J{T_{0}}(x)$
Start time to process task $x$ |  $J{T_{1}}(x)$
End time to process task $x$ |  $J{T_{2}}(x)$
Finish time of task $x$ |  $J{T_{3}}(x)$
Execution time of $x$: $EJ{T_{1}}(x)=J{T_{2}}(x)-J{T_{1}}(x)$ |  $EJ{T_{1}}(x)$
Life time(b58, ) of $x$: $EJ{T_{0}}(x)=J{T_{3}}(x)-J{T_{0}}(x)$ |  $EJ{T_{0}}(x)$
Pre-executing waiting time of $x$: $WJ{T_{0}}(x)=J{T_{1}}(x)-J{T_{0}}(x)$ |  $WJ{T_{0}}(x)$
Epi-executing waiting time of $x$: $WJ{T_{1}}(x)=J{T_{3}}(x)-J{T_{2}}(x)$ |  $WJ{T_{1}}(x)$
Volume of task $x$ that |  $JV(x)$
Size of task $x$ |  $JSZ(x)$
Resource assigned to task $x$ |  $p(x)$
Set of tasks on resource $y$ at time $t$: $PJS(y,t)=\left\\{{\forall\alpha|p(\alpha)=y\wedge\left({J{T_{1}}(\alpha)\leq t\leq J{T_{2}}(\alpha)}\right)}\right\\}$ |  $PJS(y,t)$
Set of tasks on all resources at time $t$: $JS(t)=\bigcup_{\alpha\in P}{PJS(\alpha,t)}$ |  $JS(t)$
Total loads of resource $y$ at time $t$: $\sum_{\forall\alpha\in C(y,t)}{JV(\alpha)}$ |  $PJC(y,t)$
Total number of tasks on resource $y$ at time $t$ and $PJN(y,t)=card\left({PJS(x,t)}\right)=\sum_{\forall\alpha\in C(y,t)}1$ |  $PJN(y,t)$
Total tasks volume constraint of resource $y$ where $PJC(y,t)$ must satisfy $PJC(y,t)\leq CPV(y)$ |  $CPV(y)$
Total tasks number constraint of $y$ where $PJN(y,t)$ must satisfy $PJN(y,t)\leq CPN(y)$ |  $CPN(y)$
Processing speed of resource $y$ |  $PV(y)$
Average load of all resources at time $t$ |  $\mu(t)$
Deadline of task $x$ where $x\in J$ |  $DJ(x)$
Effective Power of resource $y$ at time $t$: $\sum_{p(x,t)=y}{PJ(x,t)}$ |  $EPP(y,t)$
Consumption Power of resource $y$ at time $t$ |  $CPP(y,t)$
Total consumption energy of resource $y$ in $({t_{1}},t_{2})$ |  $EP(y,{t_{1}},{t_{2}})$
Effective energy of resource $y$ in $({t_{1}},t_{2})$: $\int_{{t_{1}}}^{{t_{2}}}{EPP(y,t)dt}$ |  $EE(y,{t_{1}},{t_{2}})$
Energy Utilization Rate of resource in $({t_{1}},t_{2})$ |  $EUPP(y,{t_{1}},{t_{2}})$
Average Power constraint of resource $y$ |  $APCR(y)$
$C{O_{2}}$ emission rate(b80, ) of resource $y$ at time $t$ |  $ERCOP(y,t)$
$C{O_{2}}$ emission of resources in $({t_{1}},t_{2})$: $\sum\limits_{y\in P}{\int_{{t_{1}}}^{{t_{2}}}{ERCOP(y,t)dt}}$ |  $ECOP(y,{t_{1}},{t_{2}})$
Power consumption to process task $x$ in resource $y$ at time $t$ |  $PJP(x,y,t)$
Power consumption to hang the task $x$ in resource $y$ at time $t$ |  $PJPQ(x,y,t)$
Total Energy consumption to process task $x$ |  $ECPJ(x)$
Total Energy consumption to hang task $x$ |  $ECPQ(x)$
Total Energy consumption to hang task $x$ |  $ECPR(x)$
Total Energy consumption of task $x$ |  $EJ(x)$
Total Energy consumption in $(t_{1},t_{2})$ |  $TEC(t_{1},t_{2})$
Power consumption of task $x$ at time $t$ |  $PJ(x,t)$
Delay time of task $x$: $DTJ(x)=\max\left(J{T_{3}}(x)-DL(x),0\right)$ |  $DTJ(x)$
Delay time constraint of task |  $DTCJ$
Delay number constraint of task |  $DNCJ$
Number of delay tasks, $DJN=\sum_{DTJ(x)>0}1$ |  $DJN$
The maximum work time of resources in $(t_{1},t_{2})$ |  $MWTP(t_{1},t_{2})$
The average execution time of tasks |  $MAETJ$
The total utilization of occupied time in $(t_{1},t_{2})$ |  $TUOT(t_{1},t_{2})$
The average delay time of delay tasks |  $MADTJ$
The number of delay tasks |  $DJN$
The average response time of tasks |  $MARTJ$
Utilization of time for all tasks |  $PPTJ$

Hence from the perspective of resources, tasks and workflows, the problem of
minimizing total energy consumption from time $t_{1}$ to $t_{2}$ time can be
modeled as

(1) $\min TEC({t_{1}},{t_{2}})=\sum\limits_{y\in P}{EP(y,{t_{1}},{t_{2}})}.$

The problem of maximizing energy utilization from time $t_{1}$ to $t_{2}$ time
can be modeled as

(2) $\max EUPP({t_{1}},{t_{2}})={{\sum\limits_{y\in
P}{EE(y,{t_{1}},{t_{2}})}}}/{{\sum\limits_{y\in P}{EP(y,{t_{1}},{t_{2}})}}}.$

The public constraint is

(3) $PublicConstraint:y\in P,x\in J,t\in\left[{{t_{1}},{t_{2}}}\right].$

The constraint of energy consumption optimization problems is the combination
of _Public Constrains_ , _Constraint1_ and _Constraint2_ seen in Equation (3),
Equation (4) and Equation (5).

(4) $Constraint\emph{1}:\frac{{TEC({t_{1}},{t_{2}})}}{{{t_{2}}-{t_{1}}}}\leq
APCR(y)$

(5) $Constraint\emph{2}:\left\\{\begin{array}[]{c}PJC(y,t)\leq CPV(y)\\\
PJN(y,t)\leq CPN(y)\end{array}\right.$

####  3.2.2. Time related optimization problem

Time index of scheduling in Cloud computing can be summarized as four types
including makespan (execution time), delay time, response time and waiting
time. In some cases, the capacity of physical resources and the size of tasks
can be used to calculate the execution time of tasks(b80, ). While, in
realistic scenes, the capacity of physical resources will vary with the change
of load, temperature and working time. The processing speed of the same
resource for tasks with different properties will be different. For example, a
resource with better GPU performance can process a task with multiple matrix
operations faster than a resource without better GPU performance, and a
resource with better CPU performance can process a task with numerous commands
transfers faster. The most practical approach is to classify tasks according
to their specific properties to estimate their executed time on different
types of resources. Also considering the complex practical situation, we can
treat them as time-varying parameters.

The objectives of makespan, also known as total execution time(b80, ), can be
divided into four aspects that minimizing the maximum execution time of
resources, minimizing average makespan of workflows, minimizing average
execution time of tasks and maximizing total utilization of occupied time. The
problems of makespan can be formulated as follows.

Minimizing the maximum work time of resources:

(6) $\min
MWTP({t_{1}},{t_{2}})=\min\left({\max\int_{{t_{1}}}^{{t_{2}}}{sig\left({PJN(y,t)}\right)dt}}\right)$

where $sig(x)=\left\\{{\begin{array}[]{*{20}{c}}0&{x\leq 0}\\\
1&{x>0}\end{array}}\right.$.

Minimizing average execution time of tasks:

(7) $\min MAETJ={\left({\sum\limits_{x\in
J}{EJ{T_{0}}(x)}}\right)}/{{card(J)}}.$

Maximizing total utilization of occupied time:

(8) $\max TUOT({t_{1}},{t_{2}})=\frac{{\sum\limits_{y\in
J}{\int_{{t_{1}}}^{{t_{2}}}{sig\left({PJN(y,t)}\right)dt}}}}{{({t_{2}}-{t_{1}})card(J)}}$

The indexes to evaluate the delay tasks can be divided into three aspects that
minimizing the average delay time of delay tasks or workflows, minimizing the
sum of delay time of all tasks or all workflows and minimizing the number of
delay tasks or delay workflows.

Minimizing the average delay time of delay tasks:

(9) $\min MADTJ={{\sum\limits_{x\in J}{DTJ(x)}}}/{{\sum\limits_{x\in
J}{sig\left({DTJ(x)}\right)}}}.$

The constraints of Equation (6) to Equation (9) can be given as the
combination of _Public Constraint_ , _Constraint2_ and _Constraint3_(or
_Constraint4_) seen as Equation (3), Equation (5) and Equation (10).

(10) $\left\\{\begin{array}[]{l}Constraint3:DTJ(x)\leq DTCJ\\\
Constraint4:DJN\leq DNCJ\end{array}\right.$

Minimizing the sum of delay time of all tasks:

(11) $\min MSDTJ=\sum\limits_{x\in J}{DTJ(x)}.$

Minimizing the number of delay tasks:

(12) $\min DJN=\sum\limits_{x\in
J}{sig\left({DTJ(x)}\right)}=\sum\limits_{DTJ(x)>0}1.$

Additionally, the objectives of delay workflows are similar to those of delay
tasks. For the delay problems, the constraints of delay can be removed and be
simplified to the combination of _Public Constraint_ and _Constraint2_ as
Equation (3) and Equation (5).

Response time equals to the sum of transmission time and queuing time.
Minimizing the average response time of tasks:

(13) $\min MARTJ={\left({\sum\limits_{x\in
J}{\left(WJ{T_{0}}(x)+TTJ(x)\right)}}\right)}/{{card(J)}}.$

Another index to evaluate time performance is the utilization of time for all
tasks. Maximizing percentage of processing time of all tasks (utilization of
time for all tasks):

(14) $\max PPTJ={{\sum\limits_{x\in J}{EJ{T_{1}}(x)}}}/{{\sum\limits_{x\in
J}{EJ{T_{0}}(x)}}}$

The constraints of Equation (13) to Equation (14) can be expressed with the
combination of _Public Constraint_ and _Constraint2_ where _Constraint3_ or
_Constraint4_ is optional.

####  3.2.3. Load balancing optimization problem

Load balancing index can be divided into two types including load balancing of
tasks volume and load balancing of tasks number according to the loaded
objects. The degree of load balancing can also be divided into two types
including cumulative load balancing and real-time load balancing according to
the time period. There are various functions to calculate the degree of load
balancing in surveyed references such as variance or standard deviation(b162,
; b84, ; b83, ), average success rate(b101, ), coefficient of variance
(CV)(b112, ), load value(b151, ) and degree of imbalance(b82, ). We may as
well assume the degree of load balancing as a function $LoadDegree(X)$ where
$X$ is a vector related to the load object and time. Hence, the load balancing
optimization problem can be written as $\max LoadDegree(X)$.

Other indexes such as Simpson’s index(b191, ) and Shannon-Wiener index (b192,
) can also be used as a measure of load balancing in addition to variance,
standard derivation, coefficient of variance, degree of imbalance etc.. In
realistic application, load balancing may not be final goal and is usually an
avenue to achieve other objectives such as maximizing utilization of
resources, reducing energy consumption or prolong service life of devices. The
index to evaluate the degree of balancing should be determined by the final
target.

Table 3. A summary of Meta-Heuristic approaches Subcategories |  Algorithm |  References
---|---|---
Ant Colony Algorithm |  Ant Lion Optimizer Algorithm |  (b85, )
HGA-ACO |  (b106, )
DAAGA |  (b160, )
Two ant colony-based algorithms |  (b137, )
S-MOAL |  (b135, )
Genetic algorithm |  NSGA-II |  (b74, ; b56, )
TS-NSGA-II |  (b130, )
NSGA-II-based MOGA |  (b131, )
Parallel multi-objective genetic algorithm |  (b162, )
MOGA |  (b57, ; b124, )
NSGA-III |  (b142, ; b56, )
CMI |  (b138, )
MOEA/D-based genetic algorithm |  (b123, ; b56, )
Hybrid Whale Genetic Algorithm |  (b163, )
Particle Swarm Algorithm |  MOPSO or MPSO |  (b65, ; b96, )
TSPSO |  (b134, )
HAPSO |  (b125, )
PSO-based MOS algorithm |  (b78, )
Firefly Algorithm |  Firefly algorithm (FA) |  (b84, )
FIMPSOA |  (b117, )
Others |  Improved differential evolution algorithm |  (b133, )
Multi-heuristic resource algorithm |  (b64, )
EPRD algorithm |  (b73, )
Whale Optimization Algorithm (WOA) |  (b147, )
MuSIC: an efficient heuristic algorithm |  (b88, )
CSSA |  (b157, )
Hybrid artificial bee colony algorithm |  (b158, )

####  3.2.4. Other optimization problems

Except to energy consumption optimization problems, time optimization problems
and load balancing problems, the other optimization problems contain
maximizing utilization of resources, minimizing $C{O_{2}}$ emission,
minimizing temperature of components(b193, ) and minimizing tasks loss rate.

The objective of minimizing $C{O_{2}}$ emission can be modeled as $\min
ECOP(t_{1},t_{2})$.

The maximizing utilization of resources can be divided into energy utilization
seen in Equation (2), total utilization of occupied time seen in Equation (8),
and utilization of time for all tasks seen in Equation (14).

When the queuing line or response time is too long, users may choose to give
up executing a task on a Cloud platform with high probability, which will
cause loss of tasks and users. In some extreme cases when Cloud platforms are
unable to provide new services such as when severe overload, physical device
failure, device attacked to failure by network, excessive energy consumption
and so on, Cloud computing providers have no choice but only to suspend
service. Thereupon, tasks loss rate is also an urgent factor to be studied in
building a stable Cloud computing platform. Back and forth, stable groups of
Cloud environment and users will also debase the difficulty to allocate tasks
or schedule resources in Cloud computing. Modeling the minimization of tasks
loss rate is complex considering the users’ psychology, users’ characteristics
and randomness of task executions.

###  3.3. Summary

So far, this section provides models for various common scheduling
optimization problems in Cloud computing. The constraints of optimization
problem in Cloud computing can vary with actual scenarios. In some specific
scenarios, we can give weight to constrains to meet complex scenes. For multi-
objectives problems, the objectives can be integrated based on their weights
in realistic scenarios or can be evaluated by Pareto efficiency(b165, ; b60, ;
b56, ).

Table 4. A summary of Heuristic algorithms Heuristic Algorithm |  References
---|---
Jacobi Best-response Algorithm |  (b87, )
Efficient wholesale and buyback scheme (EWBS) |  (b127, )
Distributed Geographical Load Balancing (DGLB) |  (b116, )
Adapting Johnson’s model-based algorithm |  (b120, )
Lagrange Relaxation based Aggregated Cost (LARAC) |  (b63, )
Dynamic Bipartition-First-Fit (BFF) |  (b62, )
MS and normalized water-filling-based MSNWF |  (b136, )
QoS-Aware Distributed Algorithm ( FCFI+ACTI) |  (b141, )
AFCFS (Adaptive First Comes First Served) |  (b146, )
FCFS (First Come First Serve) |  (b148, )
LLIF (A 2-approximation algorithm) |  (b68, )
RR (Round-Robin) algorithm |  (b154, )

##  4\. Major approaches for scheduling in Cloud computing

Table 5. A summary of other classic approaches Categories |  Algorithm |  References
---|---|---
DP |  ADP algorithm |  (b105, )
Random |  OCRP algorithm |  (b128, )
Hybrid |  FDHEFT(a fuzzy dominance sort) |  (b75, )
EWBS algorithm |  (b70, )
Other |  CWSA |  (b121, )
PBTS algorithm |  (b72, )
Lyaponuv Optimization Approach |  (b91, )
CHOPPER |  (b89, )
Dynamic replica allocation algorithm |  (b108, )
LPOD algorithm |  (b155, )
DMFIC algorithm |  (b93, )
LPGWO algorithm |  (b165, )

Approaches for scheduling in Cloud computing can be classified into six
categories including Dynamic Programming(DP), Probability algorithm (Random),
Heuristic method, Meta-Heuristic algorithm, Hybrid algorithms and Machine
Learning. In this section, we introduce these approaches applied in surveyed
literature from two aspects, classic approaches and machine learning method.
In this paper, we regard Dynamic programming, Randomization, Heuristic method,
Meta-Heuristic algorithm, and Hybrid algorithms as classic approaches.

###  4.1. Classic approaches for scheduling in Cloud computing

In classic approaches, the most commonly utilized methods in surveyed
literature are Meta-Heuristic algorithm and Heuristic method. In skeleton,
Meta-Heuristic, the combination of Heuristic and Randomization(b76, ),
contains Ant Colony Algorithm, Particle Swarm Optimization, Genetic algorithm,
Firefly algorithm and other Meta-Heuristic algorithms. Some common Heuristic
methods are Johnson’s model, FF (first fit), BF (best fit), RR (Round-robin),
FFD (first fit decreasing), BFD (best fit decreasing) and their variants.
Table LABEL:table3, Table LABEL:table4, and Table LABEL:table5 provide
summaries of classic methods from surveyed literature, aiming for an intuitive
reference and summary for the research of scheduling problems in Cloud
computing.

###  4.2. Machine learning for scheduling in Cloud computing

In practice, Cloud system has several characteristics: large scale and
complexity of systems that make it impossible to model accurately; timeliness
of scheduling decisions that demands the high-speed scheduling algorithm; and
randomness of tasks (or requests) including randomness of task numbers,
arrival time and sizes.

Table 6. Machine learning methods used in scheduling of Cloud Subcategories |  Algorithm |  References
---|---|---
Deep learning (without RL) |  DERP |  (b149, )
NN-DNSGA-II algorithm |  (b82, )
DLSC framework |  (b111, )
RL |  QEEC |  (b58, )
RLVMrB |  (b112, )
RL+Belief-Learning-Based Algorithm |  (b167, )
Revised reinforcement learning |  (b170, )
URL: A unified reinforcement learning |  (b171, )
Q-value Learning Algorithm |  (b172, )
A Bare-Bones Reinforcement Learning |  (b173, )
Adaptive reinforcement learning |  (b174, )
Rethinking reinforcement learning |  (b175, )
ML+RM resource manager |  (b201, )
RL-based ADEC |  (b203, )
DRL |  A3C RL algorithm |  (b150, ; b113, )
The RL-based DPM framework |  (b177, )
Deep Q Network (DQN) |  (b152, ; b164, )
ADRL |  (b169, )
DeepRM-Plus |  (b107, )
Modified DRL algorithm |  (b153, )
DQTS |  (b83, )
MRLCO |  (b161, )
Multiagent DRL |  (b168, )
Other |  Multi-Agent Imitation Learning |  (b95, )

These characteristics are challenging for the research of resource scheduling
in Cloud computing. It is tough for one specific Meta-Heuristic or Heuristic
algorithm to fully adapt the real dynamic Cloud computing systems or Edge-
Cloud computing systems. A novel type of machine learning policy for resource
scheduling in Cloud computing is the combination of deep neural network (DNN)
and reinforcement learning (RL), called deep reinforcement learning (DRL).
Integrating the advantages of RL and DNN, DRL has following benefits. They are
Modeling ability: it can model complex systems and decision-making policies
with DNN; Adaptability for optimization objectives: Training progress based on
gradient descent algorithm makes it possible to search the optimization
solution for various objectives; Adaptability for environment: DRL can adjust
parameters to adapt to various environments; Possibilities for further growth:
DRL can grow over time to process large-scale tasks; Adaptability for state
space: DRL can process continuous states or multi-dimensional states; Memory
of experience: DRL possess capacity to memory experience with experience
replay.

Before providing detailed introduction of RL-based algorithms in resource
scheduling of Cloud computing, we give a summary of machine learning methods
used in scheduling of Cloud computing by reviewed literature in Table
LABEL:table6.

##  5\.  Analysis of RL and DRL Frameworks in scheduling of Cloud computing

Machine learning is the discipline of teaching computer to predict outcomes or
classify objects without explicit programming(b92, ). Machine learning is also
an artificial intelligence discipline of studying how computers simulate or
implement human learning behavior so that computers can gain new knowledge and
skills. Based on learning methods, machine learning can be divided into
supervised learning, unsupervised learning and semi-supervised learning(b92,
). RL is one of the unsupervised learning(b203, ). Based on learning strategy,
machine learning contains Symbol learning, Artificial Neural Networks
learning, Statistical machine learning, Bionic machine learning etc.. Deep
learning on the basis of deep artificial neural networks and reinforcement
learning (RL) are two subsets with intersection of machine learning, where the
intersection between deep learning and RL is deep reinforcement learning
(DRL). DRL combined with the perception of deep learning and with decision-
making ability of RL, has been applied in robot control, computer vision,
natural language processing and some Go sports(b196, ; b200, ). DRL and Cloud
computing, two emerging technologies, do not have sufficient intersection
currently. The utilization of DRL to resolve the scheduling problems in Cloud
computing emerged in recent years. Afterwards, DRL has shown superior
performance in the current research on the application in resource scheduling
of Cloud computing. We discuss the evolution of RL and DRL frameworks in this
section, in order to provide a comprehensive sight for the discuss of RL and
DRL in resource scheduling of Cloud computing.

###  5.1.  Analysis of RL framework

RL, based on the interaction model between agent and environment, instructs
agent to learn the optimal action strategy by the feedback from environment
corresponding to the agent’s action. The RL model can update the action
strategy according to the timely feedback and long-term feedback. Agent will
choose the action on the basis of action strategy. State space, action space,
environment, feedback, and strategy are five basic elements of RL. Fig.2 shows
a fundamental structure of RL.

Figure 2. A fundamental framework of RL

In some of reviewed literature related to RL(b164, ; b150, ; b168, ; b167, ;
b151, ; b58, ), the feedback is described as reward. In this paper, we regard
it as feedback considering that both positive feedback and negative feedback
will affect the learning progress of agent’s action strategy(b83, ; b107, ;
b151, ).

In another standpoint, agent learns the strategy by trailing error, which
requires agent to maintain balance between exploration and exploitation.
Greedy method, Random method and Meta-Heuristics method will be used to
simulate the decision progress between exploration and exploitation. Markov
decision process (MDP) is a common model to express the action choose process
and Bellman Equation, a dynamic programming equation, is a common function to
update the action strategy. Hence, a framework of RL containing action
selection and strategy update is shown as Fig.3.

Figure 3. A framework of RL with action selection and strategy update

In complex scenarios, state and agent are varying with time. In addition,
decision should be made according to the state and agent at real-time and the
feedback from environment will affect strategy directly. Hence, an agent-state
based structure of RL can be shown as Fig.4.

Figure 4. A complex Framework of RL based on varying agent-state

In most of realistic scenes, a system is often not completely independent and
will alter with extrinsic stimulus. The environment in Fig.4 is actually the
internal environment of system which cannot express the overall interferences
from other systems to this internal system. Hence, the system of RL in Fig.4
should be regarded as an autonomous system because agent and environment
evolute on the autonomous rules. A computer game, a Go sport and a language
processing problem that covers a large enough amount of data may be regard as
an autonomous system, because the regulation of them is quite stable without
external modification of regulation. The movement of vehicles and ships,
antagonistic sports like basketball and football are usually affected by
external incentives. Then, a Cloud computing system, with time-varying
constructive demands, optimization objectives and users’ requests, is not an
autonomous system. On the other hand, the update function of internal
environment for agent-state and the decision-making function are also time-
varying such as the revenue ratio of Cloud computing is variable in different
periods of the same day. Regarding the decision-making process as an ensemble,
a framework of RL with time-varying extrinsic stimulus can be shown as Fig.5.

Figure 5. A framework of RL with extrinsic stimulus

###  5.2.  Analysis of DRL framework

In Fig.5, decision maker, that is a complex mapping from agent-state to
action, is integrated as an ensemble. Some patterns of RL focus on the
expression of this mapping relationship such as Q-table, Advantage Function,
Policy Gradient, and Hidden Markov Chain. However, as the sizes and dimensions
of state space and action space increase, the computational complexity and
storage space of these patterns will grow exponentially. Furthermore, when the
state space is non discrete which appears in the resource scheduling problem
usually, it is difficult to express the mapping relationship in the general
methods of RL. Nonhomogeneous Markov process-based RL, that one of method to
express the process of time-varying continuous time space and continuous state
space, requires to solve differential equations with variable coefficients
however, which astricts the application of nonhomogeneous Markov process in RL
to solve the problem with time-varying continuous time space and continuous
state space. Hence for various reasons, deep artificial neural network with
excellent performance in the establishment of mapping relationship is a fairly
good choice to be a mapper of strategy between state-agent and action. Fig.6
shows a framework using deep neural network to express the decision process.

Figure 6. A framework of RL with neural network-based decision maker

Fig.6 regards decision process as a mapping process, which can enlighten us to
reconstruct the structure of Fig.5 according to the mapping relation. Thus, we
can construct the framework in Fig.5 as five mappers including mapper of time
varying, mapper of stimulus evolution, mapper of decision, environment and
mapper of feedback as Fig.7.

Figure 7. A framework of RL with segment of system

The details of each mapper are as follows: Mapper of time varying refers to
the relationship between agent-state and time with stimulus from extrinsic or
internal space where time and update are input, stimulus force is the output.
Mapper of stimulus evolution refers to the stimulus evolution in agent and
state as agent and state are usually variable with stimulus where stimulus
force is the input and the set of agent-state at real-time is output. Mapper
of decision is responsible for calculating the next action according to the
current state of agent where set of agent-state at this time is input and
action at next time is output. Environment receives actions that the result of
mapper of decision provides, evolves according to the action of agent and then
outputs environment’s state at next time. The output of environment enters
mapper of feedback as its input on one hand, and enters mapper of time varying
as internal stimulus for agent on the other hand. Mapper of feedback receives
the environment’s state, then stores it as replay storage in preparation for
long-term feedback in future and takes it as timely feedback simultaneously.
Long-term feedback and real-time feedback will update the parameters of
decision maker. The framework in Fig.7 is a generalized RL based on the
integration of mappers. In some of experiments, mapper of time varying, mapper
of stimulus evolution and environment are usually simulated by program or
observed in real scenes. Mapper of decision and mapper of feedback can be
constructed with neural networks. As mapper of feedback is aimed at updating
the parameters of decision maker, thus the mapper of feedback can be designed
as a neural network to calculate the loss function of the neural network in
mapper of decision, hence Nature DQN or Double DQN(b152, ; b153, ; b164, ;
b189, ) where the neural network of feedback is called as target network with
a same structure of decision’s network. While inherently, the five mappers in
Fig.7 can be represented by neural networks respectively. In some scenes, it
is difficult to simulate or observe the realistic process of a complex system,
and the neural network can be used as an end-to-end alternative. An extreme
example is that five mappers are all expressed with deep neural networks.
However, existing research, using DRL to resolve the scheduling problems in
Cloud computing surveyed in this paper, are carried out by replacing one or
several of the five mappers with neural networks and have performed well in
experiments according to their results.

###  5.3. Summary

Cloud environment is a complex and random system with large-scale user
requests and complex physical environment, and these user requests and
extrinsic physical environment can be regarded as time-varying stimulus. In
addition, the actual running processes of electronic components and software
programs are hard to express using simulation. Crucially, the high
dimensionality and the continuity of state space make mapper of decision and
mapper of feedback difficult to be modeled with conventional methods. In
summary, the five mappers may have demand to be modeled with implicit
expression functions, while deep neural network is a practical method
currently to deliver implicit relation on the basis of sufficient data and
sufficient training time.

Moreover, the literature adjusts the structure of neural network (CNN, LSTM,
full connection, etc.), increase the strategy of initialization of neural
network parameters, the training strategy of neural network, prediction or
simulation of internal and external environment, and assist with other Meta-
Heuristic algorithms as appropriate.

##  6\. A review of DRL-based resource scheduling in Cloud computing

Based on the location of neural network, we review frameworks in surveyed
papers. In RL, the central component is the mapper of decision which can
conduct scheduler of Cloud computing. In scheduling of Cloud computing using
RL, the mapper of decision is usually represented by deep neural network or
Q-table. In order to deeply analyze and macroscopically summarize the
application of RL in resource scheduling of Cloud computing, we organize the
information of literature structurally. In addition to the information of the
literature themselves, we also reorganize the possible future work of some
literature to provide another probability.

QEEC(b58, ) is a Q-learning based task scheduling framework for energy-
efficient Cloud computing using Q-value table to express the decision maker of
action. The DeepRM_Plus (b107, ) uses a neural network that has convolution
neural network (CNN) of six layers to describe the mapper of decision based on
the great success of DNN in image processing. Data center cluster, waiting
queues, and backlog queue compose the state of environment which is
represented by image.

Table 7. Category and Objectives of RL-based algorithms Algorithm |  Category |  Objectives
---|---|---
QEEC(b58, ) |  QL |  Response time, CPU utilization
MDP_DT(b174, ) |  QL |  Cost
AGH+QL(b170, ) |  QL |  Energy consumption, QoS
MRLCO (b161, ) |  MRL |  Network traffic, service latency
DeepRM_Plus(b107, ) |  DRL |  Turnaround time, cycling time
DERP(b149, ) |  DRL |  Automatic elasticity
DPM (b177, ) |  DRL |  Task latency, energy consumption
DQST(b83, ) |  DQL |  Makespan, load balancing
MDRL(b153, ) |  DDQN |  Energy consumption, response time
RLTS (b164, ) |  DDQN |  Makespan
DRL-Cloud (b189, ) |  DDQN |  Energy consumption, cost
ADRL(b169, ) |  DDQN |  Resource Utilization, response time
IDRQN(b151, ) |  DDQN |  Energy consumption, service latency
MADRL(b168, ) |  DDQN |  Computation delay, channel utilization
DDQN(b152, ) |  DDQN |  Service latency, system rewards
DRL+FL(b190, ) |  DDQN |  Energy consumption, load balancing

AGH+QL(b170, ), a novel revised Q-learning-based model, takes hash codes as
input states with a reduced size of state space. DQST(b83, ), deep Q-learning
task scheduling, uses fully connected network to calculate the Q-values which
can express the mapper of action decision. DERP(b149, ) uses three different
approaches of a DRL agent to handle the multi-dimensional state and to provide
elastic VM resource. Modified DRL(b153, ), RLTS(b164, ), DRL-Cloud(b189, ),
and ADRL(b169, ) also use the structure of action-value Q network (or called
evaluate Q-network(b164, )) and target-Q network. Then, their similarities and
differences are as follows. IDRQN(b151, ) is a fine-grained task offloading
scheme based on DRL with Q-network and Target net where LSTM network layer is
used in Q-network and candidate network is used to update Target Net. DPM
framework(b177, ) based on RL, adopts the long short-term memory (LSTM)
network to capture the prediction results and uses DRL to train the strategy
of resource allocation aimed at reducing energy consumption in Cloud
environment. The LSTM network used to predict the state of environment can be
regarded as a mapper of time varying in Fig.7. DDQN(b152, ), Dueling deep
Q-network, contains a set of convolutional neural networks and full connected
layer to achieve higher efficiency of data processing, lower network cost, and
better security of data interaction. MRLCO (b161, ), a Meta Reinforcement
Learning-based method, contains seq2seq neural network to represent the
policy. MADRL(b168, ), a novel multiagent DRL, contains actor network and
critic network to generate Q value. Actor network with two layer fully
connected network is a mapper from state to action, and critic network with
two fully connected network hidden layers and an output layer with one node is
a mapper from state and action to Q-value. DRL+FL(b190, ), based on DDQN, uses
Federal Learning to accelerate the training of DRL agents. MDP_DT(b174, ), a
novel full-model based RL for elastic resource management, employs adaptive
state space partitioning. Table LABEL:table7 provides a summary of multi-
aspects including category, system type and objectives of RL-based algorithms,
Table LABEL:table8 provides the summary of scenario and task/server nature, as
well as Table LABEL:table9 provides the summary of experimental data and
compared baselines.

Table 8. Scenario and Task/Server Nature of RL-based algorithms  Algorithm |  Scenario |  Task/Server Nature
---|---|---
QEEC(b58, ) |  Online scheduling |  Heterogeneous servers
MDP_DT(b174, ) |  Dynamic scheduling |  Dynamic tasks
AGH+QL(b170, ) |  Scheduling in C-RANs |  Traffic demand
MRLCO (b161, ) |  Adaptive offloading |  Multiple tasks
DeepRM_Plus(b107, ) |  Online scheduling |  Independent tasks
DERP(b149, ) |  Dynamic scheduling |  Dynamic tasks
DPM (b177, ) |  Online scheduling |  Dynamic tasks
DQST(b83, ) |  Dynamic scheduling |  Non-preemptive task
MDRL(b153, ) |  Dynamic scheduling |  Heterogeneous servers
RLTS (b164, ) |  Dynamical scheduling |  Heterogeneous servers
DRL-Cloud (b189, ) |  Resource provisioning |  Depended Tasks
ADRL(b169, ) |  On-time scheduling |  Dynamic tasks
IDRQN(b151, ) |  Task offloading |  Heterogeneous servers
MADRL(b168, ) |  Multichannel access |  Joint multichannel
DDQN(b152, ) |  Online scheduling |  Delay-tolerant tasks
DRL+FL(b190, ) |  Dynamic scheduling |  Dynamic tasks

In reviewed literature, strategies to queue, to accelerate training, to
partite state space of agent, capture resource state, to keep stability of
rewards etc. are proposed to optimize the performance of algorithms, and their
details are listed in Table 10.

Combined with the results of literature research, future work of RL-based
algorithms in reviewed literature are discussed in Table 11.

Based on the review of RL-based resource scheduling in Cloud computing, the
current situation of RL in resource scheduling of Cloud computing is
summarized as follows.

DRL has strong adaptability to continuous or high dimensional state space;
adaptability for scheduling scenarios and various optimization objectives of
Cloud computing. The main scenario closest to realistic scene used RL to solve
in reviewed literature is dynamic online multi resources scheduling problem in
Cloud computing environment or Edge-Cloud computing environment which can
contain dependent or independent tasks, workflows, and homogeneous or
heterogeneous servers. In reviewed literature, experiment results showed DRL
can achieve better performance than various common compared algorithms such as
Randomization, FCFS, Round-robin, Greedy, Q learning, MDP, QDT, FIFO, HEFT,
FA, SDR. And these algorithms together with Conventional DQN can be regarded
as baselines to evaluate other algorithms in future.

Table 9. Compared baselines of RL-based algorithms Algorithm |  Compared Baselines
---|---
QEEC(b58, ) |  RANDOM, FAIR,GREEDY, QL
MDP_DT(b174, ) |  MDP, QDT, Q-learning
AGH+QL(b170, ) |  Pure Q-learning, DRL
MRLCO (b161, ) |  Fine-tuning DRL, HEFT-based, Greedy
DeepRM_Plus(b107, ) |  Random, FCFS, SJF, HRRN, Tetris, DeepRM
DERP(b149, ) |  MDP, Q-learning, MDDPT, QDT
DPM (b177, ) |  DRL, Round-robin
DQST(b83, ) |  FCFS, MAXMIN, MCT, MINMIN, RR
MDRL(b153, ) |  FIFO, Greedy algorithm
RLTS (b164, ) |  HEFT, CPOP, Lookahead, PEFT
DRL-Cloud (b189, ) |  Greedy, FERPTS, Round-robin
ADRL(b169, ) |  Over-utilized, Under-utilized, DRL
IDRQN(b151, ) |  DQN, HERDQN, IDQN, DRQN
MADRL(b168, ) |  Actor-critic; DQN; Greedy
DDQN(b152, ) |  Conventional DQN, Greedy, Random
DRL+FL(b190, ) |  Centralized DDQN, DRLRA, SDR, LOBO
Table 10. Strategies and Advantages of RL-based algorithms Algorithm |  Strategies and Advantages
---|---
QEEC(b58, ) |  M/M/S to reduce the average waiting time of task; dynamic task ordering strategy to promote the quality of Cloud services
MDP_DT(b174, ) |  Adaptively partitions the state space utilizing novel statistical criteria and strategies to perform accurate splits
AGH+QL(b170, ) |  Anchor graph hashing can accelerate training; hash codes can reduce size of state space
MRLCO (b161, ) |  Seq2seq neural network to represent the offloading policy; new training method combining the first-order approximation
DeepRM_Plus(b107, ) |  Imitating learning to accelerate convergence and CNN to capture the state of resource
DERP(b149, ) |  DERP does not demand space Partitioning; DERP with three aspects manages to collect rewards
DPM(b177, ) |  Using LSTM Network to predict workload which can eliminate the vanishing gradient problem
DQST(b83, ) |  Entropy weight method to produce a high-quality solution of bi-objective optimization
MDRL(b153, ) |  DRL can adapt to scalable state space; fair resource allocation helps reduce the underlying practical problems
RLTS (b164, ) |  Utilization of DQN to describe the relationship between state-agent and action
DRL-Cloud (b189, ) |  experience replay, target networks as well as exploration and exploitation can accelerate converge speed
ADRL(b169, ) |  Using an anomaly detection model to identify performance problems and to increase awareness of the environment
IDRQN(b151, ) |  LSTM to estimated value; candidate networks to decouple the action selection and action value evaluation
MADRL(b168, ) |  Combination of actor-critic and DQN can improve performance of algorithm
DDQN(b152, ) |  DDQN can keep stability of reward
DRL+FL(b190, ) |  Combination of DRL and FL can improve the performance in training

DDQN, duel deep Q network also called double deep Q network, is the most
commonly used model to solve the scheduling problem in Cloud computing as well
as in some Edge-Cloud computing of reviewed literature(b153, ; b164, ; b189, ;
b151, ; b176, ; b169, ; b152, ; b168, ; b190, ). Its common framework can be
summarized as Fig.8. The DDQN contains two networks, action-value Q network
and target-Q network, with same structure(b200, ). The action-value Q network
can generate Q-value of action corresponding current state. Additionally,
target-Q network can generate target value based on real-time feedback and
long-term feedback to obtain the loss function to train the action-value Q
network.

Figure 8. A framework of duel deep Q network of DRL Table 11. Future work of RL-based algorithms Algorithm |  Future Work
---|---
QEEC(b58, ) |  To investigate Meta-Heuristic to increase the performance; to establish various queueing model to satisfy realistic scenes
MDP_DT(b174, ) |  To combine the strategies of MDP_DT with DQN to solve complex scenarios
AGH+QL(b170, ) |  To combine anchor graph hashing with DRL
MRLCO (b161, ) |  To apply an adaptive client selection algorithm to automatically filter out stragglers
DeepRM_Plus(b107, ) |  To apply other policies such as Actor-Critic network and DDPG; to analyze the state recognition analytically
DERP(b149, ) |  To combine DERP with federate learning; to design intercommunicate framework of simple DRL, full DRL and DDRL
DPM (b177, ) |  To combine LSTM predictor and CNN to reduce energy
DQST(b83, ) |  To establish model of energy consumption or multi-objective
MDRL(b153, ) |  To consider dependent tasks and workflow
DRL-Cloud (b189, ) |  To utilize it in dynamic scheduling and static scheduling
ADRL(b169, ) |  To applied parameter initialization strategy and combination of DRL and semi-supervised learning to accelerate training
IDRQN(b151, ) |  To apply transfer learning to heterogeneous MEC ; to utilize federate learning to solve multi-objectives problems
MADRL(b168, ) |  To use gated recurrent units of the network to predict channel conditions
DDQN(b152, ) |  To apply it in other issues such as energy efficiency
DRL+FL(b190, ) |  To utilize the combination of FL and DRL in other scenarios

There are few frameworks using DRL to deal with multi-objective resource
scheduling problems, while Meta-Heuristic algorithms are major methods to
resolve multi-objective resource scheduling problems. The major policies in
reviewed literature using RL contain several aspects: adjusting the structure
of decision mapper to DNN or Q-table; strategies to accelerate training of
(deep) RL such as periodical update; partition strategies for state space;
federal learning to improve convergence and stability; strategies to perceive
current states or to predict subsequent states of agent in RL; policies to
provide loss function to train main-net in DRL.

##  7\. Challenges and Future Directions for using DRL to resolve scheduling
problems in Cloud computing

Although DRL-based scheduling algorithms have performed advantages in reviewed
literature, they are only tested at the laboratory simulation experiments
where tasks’ order of magnitude is far less than that in real Cloud
environment. DRL, as a complex, non-analytic and time-costing algorithm(b196,
), has inevitable challenges to address the scheduling problems in real large-
scale Cloud computing systems. The challenges using DRL to solve scheduling
problems in Cloud computing of realistic scenarios are mainly as:

  1. (1)

DRL consumes large computing power and occupies prodigious complexity in the
progress of training and computation especially for multi-clusters or large-
scale system. Computational complexity require to be optimized.

  2. (2)

The scheduling results based on DRL are still unpredictable so that the
performance of the worst case is hard to evaluate.

  3. (3)

Real scheduling also depends on the prediction of dynamic tasks without
preemptive and priori knowledge.

  4. (4)

Gradient descent algorithm used in DRL or Bellman Equation used in QL have
inherent restrictions which will lead local optimization rather than global
optimization.

  5. (5)

Unexplainability of training process challenges for the theoretical derivation
based on mathematics techniques. Modeling and theoretical derivation of high
dimensional continuous state space demand development of mathematics.

  6. (6)

Antagonism network, federal learning, mathematical logic, Nonhomogeneous
Markov process, hidden Markov process and other policies can be utilized in
DRL or RL.

The solution of its challenges requires more pursuit of theoretical research
based on improvement of mathematical theory such as Markov Decision
Processes(b196, ; b170, ), Gradient Descent Theory(b200, ; b107, ), Matrix
theory(b170, ) and Discrete Mathematics, as well as requires more research of
real objects in realistic scenarios such as thermal conversion process,
calculation process, driving process by electric signal and voltage switching
process over the time of components. Nevertheless, research of DRL utilizing
in scheduling problems in Cloud computing still has considerable potential,
which can advance modeling for complex scenes, theoretical modeling for
machine learning, reduction of computational complexity, flexibility of
scheduling algorithm and theoretical research on existing algorithms in
principle. In addition, in cloud computing resource scheduling, DRL can be
used not only as a resource scheduling algorithm directly like reviewed
literature in Section 6, but also as a strategy to select scheduling algorithm
like Figure 9.

According to the previous review and analysis, some of future directions
utilizing RL especially DRL in resource scheduling of Cloud computing can be
summarized as follows.

  1. (1)

DRL can combine other policies or approaches to meet complex scenes and multi-
objectives, which has be verified in experiments of reviewed literature.
Therefore, its application mode in realistic engineering environment to reduce
the risk of excessive computational complexity is a noteworthy direction.

  2. (2)

As the RL is one category of non-supervised learning, one of the crucial
issues is how to train main-net of decision mapper in DRL. The main-net in DRL
can be represented by CNN, LSTM and multi-networks.

  3. (3)

Other policies to improve the performance of DRL contain utilization of Meta-
Heuristic method and other machine learning such as imitating learning.

  4. (4)

Queue model is also a crucial aspect to increase the performance of schedulers
such as M/M/S queueing model(b58, ) and M/G/1 queueing model(b45, ).

  5. (5)

Application pattern of DRL to assist other analyzable scheduling algorithms
such as FCFS, Hungarian algorithm, LPT algorithm, Johnson’s algorithm and
more.

  6. (6)

Additional basis for application of DRL in scheduling problems are baselines
and benchmarks. We summarize characteristics of baselines and benchmarks as
follows.

A baseline should possess following properties.

    1. (a)

The baseline is easy to construct and implement;

    2. (b)

It has reproducibility and performance stability;

    3. (c)

It can adapt to a variety of scenes, objectives and experiments with variable
scales;

    4. (d)

It should include various types of algorithms, and each algorithm can
represent the characteristics of its type;

    5. (e)

Its performance can be proved theoretically or has accepted conclusion;

    6. (f)

It should have been optimized to a certain extent and should contain state-of-
the-art;

    7. (g)

The compared experiments should reduce the influence of parameter adjustment
as much as possible and restore the inherent performance of algorithm itself.

A benchmark should possess following properties.

    1. (a)

It should be easy to reproduce and calculate;

    2. (b)

It should contain data of multiple scenarios and be as close as possible to
real scenarios;

    3. (c)

It can be applied to the experimental verification of a variety of
optimization objectives;

    4. (d)

It should have a dynamic scale rather than a single scale which can avoid the
performance optimization caused by adjusting parameters;

    5. (e)

If there is a random experiment, a benchmark should have enough sampling
times;

    6. (f)

It should have certain control variables to verify the advantage of local
strategy;

    7. (g)

It should contain enough extreme scenarios especially on some parameter
boundaries;

    8. (h)

It can test the algorithm running under a variety of devices and components.

  7. (7)

Another emergency trend of resource scheduling in Cloud computing is the
application of algorithms in real complex and random scenarios.

  8. (8)

The exploration of more novel roles of DRL is also a practical research
direction such as DRL can perform as system strategist to booster the process
of selecting methods, as specific approaches are able to adapt specific
scenarios. To illustrate its possible novel role, a Deep Q-learning-based
framework of scheduler is presented in Fig.9 used to schedule various
scheduling algorithms for different scenarios, which can be called a scheduler
of scheduling algorithm aiming to give full play to the superiorities of all
scheduling algorithms and considering that all the algorithms are part of
anthroposophy. In this framework, the scheduling algorithms are regarded as
resources that can be automatically selected and DRL-based algorithms are not
only components of resource scheduling algorithm but also strategy to guide
the selection of specific scheduling algorithm.

Figure 9. Two phases Q learning-based Scheduler used to schedule various
scheduling algorithms

  9. (9)

Other directions used DRL to solve resource scheduling problems in Cloud
computing still demand further research, which can be demonstrated as follows.

    1. (a)

How to construct the framework of RL or DRL and how to construct DNN in DRL?

    2. (b)

How to accelerate the training or reduce calculation complexity of (deep) RL?

    3. (c)

How to ensure the stability of the results under application of DRL to resolve
scheduling problems in large-scale Cloud computing?

    4. (d)

How to construct the deducible optimization theory?

    5. (e)

How to predict the running state of Cloud systems?

    6. (f)

How to build a flexible scheduling system to cope with time-varying
objectives?

    7. (g)

How to capture agent-state of DRL correctly?

    8. (h)

How to construct the communication between multi-DRL agents of federal
learning?

    9. (i)

How to improve other category of method to address pervasive scheduling
problem not only in Cloud computing but also in other dynamic distributed
systems?

##  8\. Summary and Conclusions

Resource scheduling in Cloud computing is a crucial and emergent research
area. In this paper, we reviewed the development of Cloud computing and
resource scheduling in Cloud computing. We describe the terminologies in
resource scheduling problem of Cloud computing to restore this problem in
real-world. According to the optimization objectives, the scheduling problem
is divided into energy consumption optimization problem, time optimization
problem, load balancing optimization problem and other problems. On the basis
of reviewed literature, we classify existing methods used to solve the
resource scheduling problems in Cloud computing, which provides a macro
perspective for the selection of scheduling methods.

Additionally, DRL-based resource scheduling of Cloud computing successfully
integrates the areas of Cloud computing and DRL. From the reviewed literature,
DRL is one of effective methods to solve the dynamic resource scheduling of
large-scale Cloud computing. Therefore, we collate and research the
architecture of RL and DRL, and based on the mapping process of mathematics,
we gradually analyze and construct various architectures of RL and DRL,
providing a unified architecture view for the subsequent RL or DRL in resource
scheduling of Cloud computing. Finally, based on the framework of DRL model
and the perspective of mapper, we focus on reviewing the information of given
literature on resource scheduling of Cloud computing using RL or DRL. We also
discuss the current status and future directions of RL especially DRL in
resource scheduling of Cloud computing.

## Acknowledgments

This research is partially supported by Nature Science Foundation of China
with ID 61672136 and by National Key Research and Development Program of China
with ID 2018AAA0103203.2.

## References

  * (1) M. Armbrust, A. Fox, R. Griffith, A. D. Joseph, R. H. Katz, A. Konwinski, G. Lee, D. A. Patterson, A. Rabkin, I. Stoica, and M. Zaharia, “A view of cloud computing,” _Commun. ACM_ , vol. 53, no. 4, pp. 50–58, 2010.
  * (2) J. Geelan _et al._ , “Twenty-one experts define cloud computing,” _Cloud Computing Journal_ , vol. 4, pp. 1–5, 2009.
  * (3) M. Adhikari, T. Amgoth, and S. N. Srirama, “A survey on scheduling strategies for workflows in cloud environment and emerging trends,” _ACM Comput. Surv._ , vol. 52, no. 4, pp. 68:1–68:36, 2019.
  * (4) L. Zhang, L. Zhou, and A. Salah, “Efficient scientific workflow scheduling for deadline-constrained parallel tasks in cloud computing environments,” _Inf. Sci._ , vol. 531, pp. 31–46, 2020.
  * (5) M. Kumar, S. C. Sharma, A. Goel, and S. P. Singh, “A comprehensive survey for scheduling techniques in cloud computing,” _J. Netw. Comput. Appl._ , vol. 143, pp. 1–33, 2019.
  * (6) T. L. Duc, R. A. G. Leiva, P. Casari, and P. Östberg, “Machine learning methods for reliable resource provisioning in edge-cloud computing: A survey,” _ACM Comput. Surv._ , vol. 52, no. 5, pp. 94:1–94:39, 2019.
  * (7) Z. Zhan, X. F. Liu, Y. Gong, J. Zhang, H. S. Chung, and Y. Li, “Cloud computing resource scheduling and a survey of its evolutionary approaches,” _ACM Comput. Surv._ , vol. 47, no. 4, pp. 63:1–63:33, 2015.
  * (8) Q. Wu, M. Zhou, Q. Zhu, Y. Xia, and J. Wen, “MOELS: multiobjective evolutionary list scheduling for cloud workflows,” _IEEE Trans Autom. Sci. Eng._ , vol. 17, no. 1, pp. 166–176, 2020.
  * (9) S. Midya, A. Roy, K. Majumder, and S. Phadikar, “Multi-objective optimization technique for resource allocation and task scheduling in vehicular cloud architecture: A hybrid adaptive nature inspired approach,” _J. Netw. Comput. Appl._ , vol. 103, pp. 58–84, 2018.
  * (10) J. Chase and D. Niyato, “Joint optimization of resource provisioning in cloud computing,” _IEEE Trans. Serv. Comput._ , vol. 10, no. 3, pp. 396–409, 2017.
  * (11) T. Rings, G. Caryer, J. R. Gallop, J. Grabowski, T. Kovacikova, S. Schulz, and I. Stokes-Rees, “Grid and cloud computing: Opportunities for integration with the next generation network,” _J. Grid Comput._ , vol. 7, no. 3, pp. 375–393, 2009.
  * (12) S. Guo, J. Liu, Y. Yang, B. Xiao, and Z. Li, “Energy-efficient dynamic computation offloading and cooperative task scheduling in mobile cloud computing,” _IEEE Trans. Mob. Comput._ , vol. 18, no. 2, pp. 319–333, 2019\.
  * (13) S. Kardani-Moghaddam, R. Buyya, and K. Ramamohanarao, “ADRL: A hybrid anomaly-aware deep reinforcement learning-based resource scaling in clouds,” _IEEE Trans. Parallel Distributed Syst._ , vol. 32, no. 3, pp. 514–526, 2021.
  * (14) G. Rjoub, J. Bentahar, and O. A. Wahab, “Bigtrustscheduling: Trust-aware big data task scheduling approach in cloud computing environments,” _Future Gener. Comput. Syst._ , vol. 110, pp. 1079–1097, 2020.
  * (15) D. A. Monge, E. Pacini, C. Mateos, E. Alba, and C. G. Garino, “CMI: an online multi-objective genetic autoscaler for scientific and engineering workflows in cloud infrastructures with unreliable virtual machines,” _J. Netw. Comput. Appl._ , vol. 149, 2020.
  * (16) J. Shao, J. Ma, Y. Li, B. An, and D. Cao, “GPU scheduling for short tasks in private cloud,” in _13th IEEE International Conference on Service-Oriented System Engineering, SOSE 2019, San Francisco, CA, USA, April 4-9, 2019_. IEEE, 2019.
  * (17) Z. Tong, H. Chen, X. Deng, K. Li, and K. Li, “A scheduling scheme in the cloud computing environment using deep _Q_ -learning,” _Inf. Sci._ , vol. 512, pp. 1170–1191, 2020.
  * (18) J. Mei, K. Li, Z. Tong, Q. Li, and K. Li, “Profit maximization for cloud brokers in cloud computing,” _IEEE Trans. Parallel Distributed Syst._ , vol. 30, no. 1, pp. 190–203, 2019.
  * (19) K. Xie, X. Wang, G. Xie, D. Xie, J. Cao, Y. Ji, and J. Wen, “Distributed multi-dimensional pricing for efficient application offloading in mobile cloud computing,” _IEEE Trans. Serv. Comput._ , vol. 12, no. 6, pp. 925–940, 2019.
  * (20) P. Zhang, X. Ma, Y. Xiao, W. Li, and C. Lin, “Two-level task scheduling with multi-objectives in geo-distributed and large-scale saas cloud,” _World Wide Web_ , vol. 22, no. 6, pp. 2291–2319, 2019.
  * (21) S. C. A, C. Sudhakar, and T. Ramesh, “Energy efficient VM scheduling and routing in multi-tenant cloud data center,” _Sustain. Comput. Informatics Syst._ , vol. 22, pp. 139–151, 2019.
  * (22) S. Ilager, K. Ramamohanarao, and R. Buyya, “Thermal prediction for efficient energy management of clouds using machine learning,” _IEEE Trans. Parallel Distributed Syst._ , vol. 32, no. 5, pp. 1044–1056, 2021.
  * (23) Y. Laili, S. Lin, and D. Tang, “Multi-phase integrated scheduling of hybrid tasks in cloud manufacturing environment,” _Robotics and Computer-Integrated Manufacturing_ , vol. 61, p. 101850, 2020.
  * (24) M. Xu and R. Buyya, “Brownout approach for adaptive management of resources and applications in cloud computing systems: A taxonomy and future directions,” _ACM Comput. Surv._ , vol. 52, no. 1, pp. 8:1–8:27, 2019\.
  * (25) J. Li, “Resource management for moldable parallel tasks supporting slot time in the cloud,” _KSII Trans. Internet Inf. Syst._ , vol. 13, no. 9, pp. 4349–4371, 2019.
  * (26) J. Wang, J. Hu, G. Min, A. Y. Zomaya, and N. Georgalas, “Fast adaptive task offloading in edge computing based on meta reinforcement learning,” _IEEE Trans. Parallel Distributed Syst._ , vol. 32, no. 1, pp. 242–253, 2021.
  * (27) J. Ren, D. Zhang, S. He, Y. Zhang, and T. Li, “A survey on end-edge-cloud orchestrated network computing paradigms: Transparent computing, mobile edge computing, fog computing, and cloudlet,” _ACM Comput. Surv._ , vol. 52, no. 6, pp. 125:1–125:36, 2020.
  * (28) E. Caron, F. Desprez, D. Loureiro, and A. Muresan, “Cloud computing resource management through a grid middleware: A case study with DIET and eucalyptus,” in _IEEE International Conference on Cloud Computing, CLOUD 2009, Bangalore, India, 21-25 September, 2009_. IEEE Computer Society, 2009, pp. 151–154.
  * (29) D. Ding, X. Fan, Y. Zhao, K. Kang, Q. Yin, and J. Zeng, “Q-learning based dynamic task scheduling for energy-efficient cloud computing,” _Future Gener. Comput. Syst._ , vol. 108, pp. 361–371, 2020.
  * (30) V. Priya, C. S. Kumar, and R. Kannan, “Resource scheduling algorithm with load balancing for cloud service provisioning,” _Appl. Soft Comput._ , vol. 76, pp. 416–424, 2019.
  * (31) B. Wan, J. Dang, Z. Li, H. Gong, F. Zhang, and S. Oh, “Modeling analysis and cost-performance ratio optimization of virtual machine scheduling in cloud computing,” _IEEE Trans. Parallel Distributed Syst._ , vol. 31, no. 7, pp. 1518–1532, 2020.
  * (32) Q. Li, H. Yao, T. Mai, C. Jiang, and Y. Zhang, “Reinforcement-learning- and belief-learning-based double auction mechanism for edge computing resource allocation,” _IEEE Internet Things J._ , vol. 7, no. 7, pp. 5976–5985, 2020.
  * (33) S. Tuli, S. Ilager, K. Ramamohanarao, and R. Buyya, “Dynamic scheduling for stochastic edge-cloud computing environments using a3c learning and residual recurrent neural networks,” _IEEE Transactions on Mobile Computing_ , vol. PP, no. 99, pp. 1–1, 2020.
  * (34) H. Ren, Y. Wang, C. Xu, and X. Chen, “Smig-rl: An evolutionary migration framework for cloud services based on deep reinforcement learning,” _ACM Trans. Internet Techn._ , vol. 20, no. 4, pp. 43:1–43:18, 2020.
  * (35) C. Fiandrino, D. Kliazovich, P. Bouvry, and A. Y. Zomaya, “Performance and energy efficiency metrics for communication systems of cloud computing data centers,” _IEEE Trans. Cloud Comput._ , vol. 5, no. 4, pp. 738–750, 2017\.
  * (36) W. Tian, M. He, W. Guo, W. Huang, X. Shi, M. Shang, A. N. Toosi, and R. Buyya, “On minimizing total energy consumption in the scheduling of virtual machine reservations,” _J. Netw. Comput. Appl._ , vol. 113, pp. 64–74, 2018.
  * (37) Y. Miao, G. Wu, M. Li, A. Ghoneim, M. Al-Rakhami, and M. S. Hossain, “Intelligent task prediction and computation offloading based on mobile-edge cloud computing,” _Future Gener. Comput. Syst._ , vol. 102, pp. 925–931, 2020.
  * (38) M. Xu, L. Cui, H. Wang, and Y. Bi, “A multiple qos constrained scheduling strategy of multiple workflows for cloud computing,” in _IEEE International Symposium on Parallel and Distributed Processing with Applications, ISPA 2009, Chengdu, Sichuan, China, 10-12 August 2009_. IEEE Computer Society, 2009, pp. 629–634.
  * (39) W. Chen, D. Wang, and K. Li, “Multi-user multi-task computation offloading in green mobile edge cloud computing,” _IEEE Trans. Serv. Comput._ , vol. 12, no. 5, pp. 726–738, 2019.
  * (40) R. Bianchini, M. Fontoura, E. Cortez, A. Bonde, A. Muzio, A. Constantin, T. Moscibroda, G. Magalhães, G. Bablani, and M. Russinovich, “Toward ml-centric cloud platforms,” _Commun. ACM_ , vol. 63, no. 2, pp. 50–59, 2020.
  * (41) S. Ilager, R. Muralidhar, and R. Buyya, “Artificial intelligence (ai)-centric management of resources in modern distributed computing systems,” _CoRR_ , vol. abs/2006.05075, 2020.
  * (42) T. K. Rodrigues, K. Suto, H. Nishiyama, J. Liu, and N. Kato, “Machine learning meets computation and communication control in evolving edge and cloud: Challenges and future perspective,” _IEEE Commun. Surv. Tutorials_ , vol. 22, no. 1, pp. 38–67, 2020.
  * (43) M. Demirci, “A survey of machine learning applications for energy-efficient resource management in cloud computing environments,” in _14th IEEE International Conference on Machine Learning and Applications, ICMLA 2015, Miami, FL, USA, December 9-11, 2015_. IEEE, 2015, pp. 1185–1190.
  * (44) S. Goodarzy, M. Nazari, R. Han, E. Keller, and E. Rozner, “Resource management in cloud computing using machine learning: A survey,” in _19th IEEE International Conference on Machine Learning and Applications, ICMLA 2020, Miami, FL, USA, December 14-17, 2020_ , M. A. Wani, F. Luo, X. A. Li, D. Dou, and F. Bonchi, Eds. IEEE, 2020, pp. 811–816.
  * (45) A. Ghasemi and A. T. Haghighat, “A multi-objective load balancing algorithm for virtual machine placement in cloud data centers based on machine learning,” _Computing_ , vol. 102, no. 9, pp. 2049–2072, 2020.
  * (46) W. Guo, W. Tian, Y. Ye, L. Xu, and K. Wu, “Cloud resource scheduling with deep reinforcement learning and imitation learning,” _IEEE Internet Things J._ , vol. 8, no. 5, pp. 3576–3586, 2021.
  * (47) J. Feng, F. R. Yu, Q. Pei, X. Chu, J. Du, and L. Zhu, “Cooperative computation offloading and resource allocation for blockchain-enabled mobile-edge computing: A deep reinforcement learning approach,” _IEEE Internet Things J._ , vol. 7, no. 7, pp. 6214–6228, 2020.
  * (48) K. Karthiban and J. S. Raj, “An efficient green computing fair resource allocation in cloud computing using modified deep reinforcement learning algorithm,” _Soft Comput._ , vol. 24, no. 19, pp. 14 933–14 942, 2020\.
  * (49) T. Dong, F. Xue, C. Xiao, and J. Li, “Task scheduling based on deep reinforcement learning in a cloud manufacturing environment,” _Concurr. Comput. Pract. Exp._ , vol. 32, no. 11, 2020.
  * (50) Z. Cao, P. Zhou, R. Li, S. Huang, and D. O. Wu, “Multiagent deep reinforcement learning for joint multichannel access and task offloading of mobile-edge computing in industry 4.0,” _IEEE Internet Things J._ , vol. 7, no. 7, pp. 6201–6213, 2020.
  * (51) C. C. Price, “Task allocation in distributed systems: A survey of practical strategies,” in _Proceedings of the ACM’82 conference_ , 1982, pp. 176–181.
  * (52) P. Cong, J. Zhou, L. Li, K. Cao, T. Wei, and K. Li, “A survey of hierarchical energy optimization for mobile edge computing: A perspective from end devices to the cloud,” _ACM Comput. Surv._ , vol. 53, no. 2, pp. 38:1–38:44, 2020.
  * (53) S. Bera, S. Misra, and J. J. P. C. Rodrigues, “Cloud computing applications for smart grid: A survey,” _IEEE Trans. Parallel Distributed Syst._ , vol. 26, no. 5, pp. 1477–1494, 2015.
  * (54) M. Xu, W. Tian, and R. Buyya, “A survey on load balancing algorithms for virtual machines placement in cloud computing,” _Concurr. Comput. Pract. Exp._ , vol. 29, no. 12, 2017.
  * (55) T. Welsh and E. Benkhelifa, “On resilience in cloud computing: A survey of techniques across the cloud domain,” _ACM Comput. Surv._ , vol. 53, no. 3, pp. 59:1–59:36, 2020.
  * (56) P. Cong, G. Xu, T. Wei, and K. Li, “A survey of profit optimization techniques for cloud providers,” _ACM Comput. Surv._ , vol. 53, no. 2, pp. 26:1–26:35, 2020.
  * (57) K. Braiki and H. Youssef, “Resource management in cloud data centers: A survey,” in _15th International Wireless Communications & Mobile Computing Conference, IWCMC 2019, Tangier, Morocco, June 24-28, 2019_. IEEE, 2019, pp. 1007–1012.
  * (58) B. Jennings and R. Stadler, “Resource management in clouds: Survey and research challenges,” _J. Netw. Syst. Manag._ , vol. 23, no. 3, pp. 567–619, 2015.
  * (59) A. R. Arunarani, D. Manjula, and V. Sugumaran, “Task scheduling techniques in cloud computing: A literature survey,” _Future Gener. Comput. Syst._ , vol. 91, pp. 407–415, 2019.
  * (60) H. Lu, C. Gu, F. Luo, W. Ding, and X. Liu, “Optimization of lightweight task offloading strategy for mobile edge computing based on deep reinforcement learning,” _Future Gener. Comput. Syst._ , vol. 102, pp. 847–861, 2020.
  * (61) Z. Xu, Y. Wang, J. Tang, J. Wang, and M. C. Gursoy, “A deep reinforcement learning based framework for power-efficient resource allocation in cloud rans,” in _IEEE International Conference on Communications, ICC 2017, Paris, France, May 21-25, 2017_. IEEE, 2017, pp. 1–6.
  * (62) N. Liu, Z. Li, J. Xu, Z. Xu, S. Lin, Q. Qiu, J. Tang, and Y. Wang, “A hierarchical framework of cloud resource allocation and power management using deep reinforcement learning,” in _37th IEEE International Conference on Distributed Computing Systems, ICDCS 2017, Atlanta, GA, USA, June 5-8, 2017_. IEEE Computer Society, 2017, pp. 372–382.
  * (63) S. M. R. Nouri, H. Li, S. Venugopal, W. Guo, M. He, and W. Tian, “Autonomic decentralized elasticity based on a reinforcement learning controller for cloud applications,” _Future Gener. Comput. Syst._ , vol. 94, pp. 765–780, 2019.
  * (64) X. Ni, J. Li, M. Yu, W. Zhou, and K. Wu, “Generalizable resource allocation in stream processing via deep reinforcement learning,” in _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020_. AAAI Press, 2020, pp. 857–864.
  * (65) E. Baccour, A. Erbad, A. Mohamed, F. Haouari, M. Guizani, and M. Hamdi, “RL-OPRA: reinforcement learning for online and proactive resource allocation of crowdsourced live videos,” _Future Gener. Comput. Syst._ , vol. 112, pp. 982–995, 2020.
  * (66) P. Dasgupta, R. J. LeBlanc, and W. F. Appelbe, “The clouds distributed operating system: Functional description, implementation details and related work,” in _The 8th International Conference on Distributed_. IEEE Computer Society, 1988, pp. 2–3.
  * (67) J. M. Bernabéu-Aubán, P. W. Hutto, M. Y. A. Khalidi, M. Ahamad, W. F. Appelbe, P. Dagupta, R. LeBlanc, and U. Ramachandran, “The architecture of ra: a kernel for clouds,” in _Proceedings of the Twenty-Second Annual Hawaii International Conference on System Sciences. Volume II: Software Track_ , vol. 2. IEEE Computer Society, 1989, pp. 936–937.
  * (68) P. Dasgupta, R. J. LeBlanc, M. Ahamad, and U. Ramachandran, “The clouds distributed operating system,” _Computer_ , vol. 24, no. 11, pp. 34–44, 1991\.
  * (69) R. C. Chen and P. Dasgupta, “Implementing consistency control mechanisms in the clouds distributed operating system,” in _Proceedings 11th International Conference on Distributed Computing Systems_. IEEE Computer Society, 1991, pp. 10–11.
  * (70) L. Gunaseelan and J. Richard Jr, “Leblanc: Distributed eiffel: a language for programming multi-granular distributed objects on the clouds operating system,” 1991.
  * (71) J. E. Allchin, “An architecture for reliable decentralized systems,” 1983.
  * (72) M. P. Pearson and P. Dasgupta, “Clide: A distributed, symbolic programming system based on large-grained persistent objects,” in _Proceedings 11th International Conference on Distributed Computing Systems_. IEEE Computer Society, 1991, pp. 626–627.
  * (73) I. Foster, Y. Zhao, I. Raicu, and S. Lu, “Cloud computing and grid computing 360-degree compared,” in _2008 grid computing environments workshop_. Ieee, 2008, pp. 1–10.
  * (74) S. Garfinkel, “An evaluation of amazon’s grid computing services: Ec2, s3, and sqs,” 2007.
  * (75) N. Minar, K. H. Kramer, and P. Maes, “Cooperating mobile agents for dynamic network routing,” in _Software agents for future communication systems_. Springer, 1999, pp. 287–304.
  * (76) J. van der Merwe, D. S. Dawoud, and S. McDonald, “A survey on peer-to-peer key management for mobile ad hoc networks,” _ACM Comput. Surv._ , vol. 39, no. 1, p. 1, 2007.
  * (77) P. Padala, K. G. Shin, X. Zhu, M. Uysal, Z. Wang, S. Singhal, A. Merchant, and K. Salem, “Adaptive control of virtualized resources in utility computing environments,” in _Proceedings of the 2007 EuroSys Conference, Lisbon, Portugal, March 21-23, 2007_. ACM, 2007, pp. 289–302.
  * (78) Y. Fu, J. S. Chase, B. N. Chun, S. Schwab, and A. Vahdat, “SHARP: an architecture for secure resource peering,” in _Proceedings of the 19th ACM Symposium on Operating Systems Principles 2003, SOSP 2003, Bolton Landing, NY, USA, October 19-22, 2003_. ACM, 2003, pp. 133–148.
  * (79) A. Fox, R. Griffith, A. Joseph, R. Katz, A. Konwinski, G. Lee, D. Patterson, A. Rabkin, I. Stoica _et al._ , “Above the clouds: A berkeley view of cloud computing,” _Dept. Electrical Eng. and Comput. Sciences, University of California, Berkeley, Rep. UCB/EECS_ , vol. 28, no. 13, p. 2009, 2009\.
  * (80) T. V. Raman, “Cloud computing and equal access for all,” in _Proceedings of the International Cross-Disciplinary Conference on Web Accessibility, W4A 2008, Beijing, China, April 21-22, 2008_ , ser. ACM International Conference Proceeding Series. ACM, 2008, pp. 1–4.
  * (81) M. Helft, “Google confirms problems with reaching its services,” _The New York Times_ , 2009.
  * (82) J. Markoff, “Software via the internet: Microsoft in cloud computing,” _New York Times_ , vol. 3, 2007.
  * (83) B. Nemade, S. Moorthy, and O. Kadam, “Cloud computing: Windows azure platform,” in _Proceedings of the ICWET ’11 International Conference & Workshop on Emerging Trends in Technology, Mumbai, Maharashtra, India, February 25 - 26, 2011_. ACM, 2011, pp. 1361–1362.
  * (84) D. R. Avresky, M. Diaz, A. Bode, B. Ciciani, and E. Dekel, Eds., _Cloud Computing - First International Conference, CloudComp 2009, Munich, Germany, October 19-21, 2009 Revised Selected Papers_ , ser. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol. 34. Springer, 2010.
  * (85) R. Buyya, S. Pandey, and C. Vecchiola, “Cloudbus toolkit for market-oriented cloud computing,” in _Cloud Computing, First International Conference, CloudCom 2009, Beijing, China, December 1-4, 2009. Proceedings_ , ser. Lecture Notes in Computer Science, vol. 5931. Springer, 2009, pp. 24–44.
  * (86) Y. Dai, Y. Xiang, and G. Zhang, “Self-healing and hybrid diagnosis in cloud computing,” in _Cloud Computing, First International Conference, CloudCom 2009, Beijing, China, December 1-4, 2009. Proceedings_ , ser. Lecture Notes in Computer Science, vol. 5931. Springer, 2009, pp. 45–56.
  * (87) M. Klems, J. Nimis, and S. Tai, “Do clouds compute? A framework for estimating the value of cloud computing,” in _Designing E-Business Systems. Markets, Services, and Networks - 7th Workshop on E-Business, WEB 2008, Paris, France, December 13, 2008, Revised Selected Papers_ , ser. Lecture Notes in Business Information Processing, vol. 22. Springer, 2008, pp. 110–123.
  * (88) R. Buyya, C. S. Yeo, S. Venugopal, J. Broberg, and I. Brandic, “Cloud computing and emerging IT platforms: Vision, hype, and reality for delivering computing as the 5th utility,” _Future Gener. Comput. Syst._ , vol. 25, no. 6, pp. 599–616, 2009.
  * (89) W. W. Chu, L. J. Holloway, M. Lan, and K. Efe, “Task allocation in distributed data processing,” _Computer_ , vol. 13, no. 11, pp. 57–69, 1980.
  * (90) R. R. Muntz and E. G. C. Jr., “Preemptive scheduling of real-time tasks on multiprocessor systems,” _J. ACM_ , vol. 17, no. 2, pp. 324–338, 1970\.
  * (91) S. K. Garg, C. S. Yeo, A. Anandasivam, and R. Buyya, “Energy-efficient scheduling of HPC applications in cloud computing environments,” _CoRR_ , vol. abs/0909.1146, 2009.
  * (92) Y. Yang, K. Liu, J. Chen, X. Liu, D. Yuan, and H. Jin, “An algorithm in swindew-c for scheduling transaction-intensive cost-constrained cloud workflows,” in _Fourth International Conference on e-Science, e-Science 2008, 7-12 December 2008, Indianapolis, IN, USA_. IEEE Computer Society, 2008, pp. 374–375.
  * (93) P. R. Ma, E. Y. S. Lee, and M. Tsuchiya, “A task allocation model for distributed computing systems,” _IEEE Trans. Computers_ , vol. 31, no. 1, pp. 41–47, 1982.
  * (94) J. Praveenchandar and A. Tamilarasi, “Dynamic resource allocation with optimized task scheduling and improved power management in cloud computing,” _Journal of Ambient Intelligence and Humanized Computing_ , pp. 1–13, 2020\.
  * (95) M. Gokuldhev, G. Singaravel, and N. R. R. Mohan, “Multi-objective local pollination-based gray wolf optimizer for task scheduling heterogeneous cloud environment,” _J. Circuits Syst. Comput._ , vol. 29, no. 7, pp. 2 050 100:1–2 050 100:24, 2020.
  * (96) S. K. Mishra and R. Manjula, “A meta-heuristic based multi objective optimization for load distribution in cloud data center under varying workloads,” _Clust. Comput._ , vol. 23, no. 4, pp. 3079–3093, 2020.
  * (97) A. Belgacem, K. B. Bey, H. Nacer, and S. Bouznad, “Efficient dynamic resource allocation method for cloud computing environment,” _Clust. Comput._ , vol. 23, no. 4, pp. 2871–2889, 2020.
  * (98) Q. Zhang, L. Gui, F. Hou, J. Chen, S. Zhu, and F. Tian, “Dynamic task offloading and resource allocation for mobile-edge computing in dense cloud RAN,” _IEEE Internet Things J._ , vol. 7, no. 4, pp. 3282–3299, 2020\.
  * (99) G. Ismayilov and H. R. Topcuoglu, “Neural network based multi-objective evolutionary algorithm for dynamic workflow scheduling in cloud computing,” _Future Gener. Comput. Syst._ , vol. 102, pp. 307–322, 2020.
  * (100) A. Rehman, S. S. Hussain, Z. ur Rehman, S. Zia, and S. Shamshirband, “Multi-objective approach of energy efficient workflow scheduling in cloud environments,” _Concurr. Comput. Pract. Exp._ , vol. 31, no. 8, 2019.
  * (101) M. Sanaj and P. J. Prathap, “Nature inspired chaotic squirrel search algorithm (cssa) for multi objective task scheduling in an iaas cloud computing atmosphere,” _Engineering Science and Technology, an International Journal_ , vol. 23, no. 4, pp. 891–902, 2020.
  * (102) X. Xu, Q. Liu, Y. Luo, K. Peng, X. Zhang, S. Meng, and L. Qi, “A computation offloading method over big data for iot-enabled cloud-edge computing,” _Future Gener. Comput. Syst._ , vol. 95, pp. 522–533, 2019.
  * (103) X. Zhang, M. Jia, X. Gu, and Q. Guo, “An energy efficient resource allocation scheme based on cloud-computing in H-CRAN,” _IEEE Internet Things J._ , vol. 6, no. 3, pp. 4968–4976, 2019.
  * (104) G. N. Reddy and S. P. Kumar, “Multi objective task scheduling algorithm for cloud computing using whale optimization technique,” in _International Conference on Next Generation Computing Technologies_. Springer, 2017, pp. 286–297.
  * (105) S. S. Gill, I. Chana, M. Singh, and R. Buyya, “CHOPPER: an intelligent qos-aware autonomic resource management approach for cloud computing,” _Clust. Comput._ , vol. 21, no. 2, pp. 1203–1241, 2018.
  * (106) M. R. Rahimi, N. Venkatasubramanian, S. Mehrotra, and A. V. Vasilakos, “On optimal and fair service allocation in mobile cloud computing,” _IEEE Trans. Cloud Comput._ , vol. 6, no. 3, pp. 815–828, 2018.
  * (107) A. S. Sofia and P. Ganeshkumar, “Multi-objective task scheduling to minimize energy consumption and makespan of cloud computing using NSGA-II,” _J. Netw. Syst. Manag._ , vol. 26, no. 2, pp. 463–485, 2018.
  * (108) H. Li, G. Zhu, Y. Zhao, Y. Dai, and W. Tian, “Energy-efficient and qos-aware model based resource consolidation in cloud data centers,” _Clust. Comput._ , vol. 20, no. 3, pp. 2793–2803, 2017.
  * (109) F. Juarez, J. Ejarque, and R. M. Badia, “Dynamic energy-aware scheduling for parallel task-based application in cloud computing,” _Future Gener. Comput. Syst._ , vol. 78, pp. 257–271, 2018.
  * (110) F. Ramezani, J. Lu, J. Taheri, and F. K. Hussain, “Evolutionary algorithm-based multi-objective task scheduling optimization model in cloud environments,” _World Wide Web_ , vol. 18, no. 6, pp. 1737–1757, 2015.
  * (111) X. Wang, Y. Wang, and Y. Cui, “A new multi-objective bi-level programming model for energy and locality aware multi-job scheduling in cloud computing,” _Future Gener. Comput. Syst._ , vol. 36, pp. 91–101, 2014.
  * (112) W. Tian, Q. Xiong, and J. Cao, “An online parallel scheduling method with application to energy-efficiency in cloud computing,” _J. Supercomput._ , vol. 66, no. 3, pp. 1773–1790, 2013.
  * (113) A. Beloglazov, J. H. Abawajy, and R. Buyya, “Energy-aware resource allocation heuristics for efficient management of data centers for cloud computing,” _Future Gener. Comput. Syst._ , vol. 28, no. 5, pp. 755–768, 2012.
  * (114) G. Sun, T. Zhan, G. O. Boateng, D. Ayepah-Mensah, G. Liu, and W. Jiang, “Revised reinforcement learning based on anchor graph hashing for autonomous cell activation in cloud-rans,” _Future Gener. Comput. Syst._ , vol. 104, pp. 60–73, 2020.
  * (115) M. Cheng, J. Li, and S. Nazarian, “Drl-cloud: Deep reinforcement learning-based resource provisioning and task scheduling for cloud service providers,” in _23rd Asia and South Pacific Design Automation Conference, ASP-DAC 2018, Jeju, Korea (South), January 22-25, 2018_. IEEE, 2018, pp. 129–134.
  * (116) N. Shan, X. Cui, and Z. Gao, “”drl + fl”: An intelligent resource allocation model based on deep reinforcement learning for mobile edge computing,” _Comput. Commun._ , vol. 160, pp. 14–24, 2020.
  * (117) M. Sardaraz and M. Tahir, “A parallel multi-objective genetic algorithm for scheduling scientific workflows in cloud computing,” _International Journal of Distributed Sensor Networks_ , vol. 16, no. 8, p. 1550147720949142, 2020\.
  * (118) G. Natesan and A. Chokkalingam, “Multi-objective task scheduling using hybrid whale genetic optimization algorithm in heterogeneous computing environment,” _Wirel. Pers. Commun._ , vol. 110, no. 4, pp. 1887–1913, 2020\.
  * (119) S. Pandiyan, T. S. Lawrence, V. Sathiyamoorthi, M. Ramasamy, Q. Xia, and Y. Guo, “A performance-aware dynamic scheduling algorithm for cloud-based iot applications,” _Comput. Commun._ , vol. 160, pp. 512–520, 2020.
  * (120) D. Gabi, A. S. Ismail, A. Zainal, Z. Zakaria, A. Abraham, and N. M. Dankolo, “Cloud customers service selection scheme based on improved conventional cat swarm optimization,” _Neural Comput. Appl._ , vol. 32, no. 18, pp. 14 817–14 838, 2020.
  * (121) M. Adhikari, T. Amgoth, and S. N. Srirama, “Multi-objective scheduling strategy for scientific workflows in cloud environment: A firefly-based approach,” _Appl. Soft Comput._ , vol. 93, p. 106411, 2020.
  * (122) A. F. S. Devaraj, M. Elhoseny, S. Dhanasekaran, E. L. Lydia, and K. Shankar, “Hybridization of firefly and improved multi-objective particle swarm optimization algorithm for energy efficient load balancing in cloud computing environments,” _J. Parallel Distributed Comput._ , vol. 142, pp. 36–45, 2020\.
  * (123) A. Narwal and S. Dhingra, “Enhanced task scheduling algorithm using multi-objective function for cloud computing framework,” in _International Conference on Next Generation Computing Technologies_. Springer, 2017, pp. 110–121.
  * (124) Y. Zhao, C. Tian, Z. Zhu, J. Cheng, C. Qiao, and A. X. Liu, “Minimize the make-span of batched requests for FPGA pooling in cloud computing,” _IEEE Trans. Parallel Distributed Syst._ , vol. 29, no. 11, pp. 2514–2527, 2018.
  * (125) X. Zhou, G. Zhang, J. Sun, J. Zhou, T. Wei, and S. Hu, “Minimizing cost and makespan for workflow scheduling in cloud using fuzzy dominance sort based HEFT,” _Future Gener. Comput. Syst._ , vol. 93, pp. 278–289, 2019.
  * (126) H. Hu, Z. Li, H. Hu, J. Chen, J. Ge, C. Li, and V. I. Chang, “Multi-objective scheduling for scientific workflow in multicloud environment,” _J. Netw. Comput. Appl._ , vol. 114, pp. 108–122, 2018.
  * (127) B. P. Rimal and M. Maier, “Workflow scheduling in multi-tenant cloud computing environments,” _IEEE Trans. Parallel Distributed Syst._ , vol. 28, no. 1, pp. 290–304, 2017.
  * (128) Q. Liu, W. Cai, J. Shen, Z. Fu, X. Liu, and N. Linge, “A speculative approach to spatial-temporal efficiency with multi-objective optimization in a heterogeneous cloud environment,” _Secur. Commun. Networks_ , vol. 9, no. 17, pp. 4002–4012, 2016.
  * (129) H. Jiang, J. Yi, S. Chen, and X. Zhu, “A multi-objective algorithm for task scheduling and resource allocation in cloud-based disassembly,” _Journal of Manufacturing Systems_ , vol. 41, pp. 239–255, 2016.
  * (130) W. Tian, G. Li, W. Yang, and R. Buyya, “Hscheduler: an optimal approach to minimize the makespan of multiple mapreduce jobs,” _J. Supercomput._ , vol. 72, no. 6, pp. 2376–2393, 2016.
  * (131) W. Wang, B. Liang, and B. Li, “Multi-resource fair allocation in heterogeneous cloud computing systems,” _IEEE Trans. Parallel Distributed Syst._ , vol. 26, no. 10, pp. 2822–2835, 2015.
  * (132) S. K. Panda and P. K. Jana, “A multi-objective task scheduling algorithm for heterogeneous multi-cloud environment,” in _2015 International Conference on Electronic Design, Computer Networks & Automated Verification (EDCAV)_. IEEE, 2015, pp. 82–87.
  * (133) F. Zhang, J. Cao, K. Li, S. U. Khan, and K. Hwang, “Multi-objective scheduling of many tasks in cloud platforms,” _Future Gener. Comput. Syst._ , vol. 37, pp. 309–320, 2014.
  * (134) F. Ramezani, J. Lu, and F. K. Hussain, “Task scheduling optimization in cloud computing applying multi-objective particle swarm optimization,” in _Service-Oriented Computing - 11th International Conference, ICSOC 2013, Berlin, Germany, December 2-5, 2013, Proceedings_ , ser. Lecture Notes in Computer Science, vol. 8274. Springer, 2013, pp. 237–251.
  * (135) X. Wang, Z. Ning, and S. Guo, “Multi-agent imitation learning for pervasive edge computing: A decentralized computation offloading algorithm,” _IEEE Trans. Parallel Distributed Syst._ , vol. 32, no. 2, pp. 411–425, 2021.
  * (136) L. Abualigah and A. Diabat, “A novel hybrid antlion optimization algorithm for multi-objective task scheduling problems in cloud computing environments,” _Cluster Computing_ , pp. 1–19, 2020.
  * (137) S. S. Haytamy and F. A. Omara, “A deep learning based framework for optimizing cloud consumer qos-based service composition,” _Computing_ , vol. 102, no. 5, pp. 1117–1137, 2020.
  * (138) C. Li, J. Bai, Y. Chen, and Y. Luo, “Resource and replica management strategy for optimizing financial cost and user experience in edge cloud computing system,” _Inf. Sci._ , vol. 516, pp. 33–55, 2020.
  * (139) J. Zhao, Y. Xiang, T. Lan, H. H. Huang, and S. Subramaniam, “Elastic reliability optimization through peer-to-peer checkpointing in cloud computing,” _IEEE Trans. Parallel Distributed Syst._ , vol. 28, no. 2, pp. 491–502, 2017.
  * (140) Y. Jiao, P. Wang, D. Niyato, and K. Suankaewmanee, “Auction mechanisms in cloud/fog computing resource allocation for public blockchain networks,” _IEEE Trans. Parallel Distributed Syst._ , vol. 30, no. 9, pp. 1975–1989, 2019.
  * (141) Z. Hong, W. Chen, H. Huang, S. Guo, and Z. Zheng, “Multi-hop cooperative computation offloading for industrial iot-edge-cloud computing environments,” _IEEE Trans. Parallel Distributed Syst._ , vol. 30, no. 12, pp. 2759–2774, 2019.
  * (142) Y. Zhang, X. Lan, Y. Li, L. Cai, and J. Pan, “Efficient computation resource management in mobile edge-cloud computing,” _IEEE Internet Things J._ , vol. 6, no. 2, pp. 3455–3466, 2019.
  * (143) Z. Guan and T. Melodia, “The value of cooperation: Minimizing user costs in multi-broker mobile cloud computing networks,” _IEEE Trans. Cloud Comput._ , vol. 5, no. 4, pp. 780–791, 2017.
  * (144) T. Chen, A. G. Marqués, and G. B. Giannakis, “DGLB: distributed stochastic geographical load balancing over cloud networks,” _IEEE Trans. Parallel Distributed Syst._ , vol. 28, no. 7, pp. 1866–1880, 2017.
  * (145) J. Li and Y. Han, “A hybrid multi-objective artificial bee colony algorithm for flexible task scheduling problems in cloud computing system,” _Clust. Comput._ , vol. 23, no. 4, pp. 2483–2499, 2020.
  * (146) L. Li, “An optimistic differentiated service job scheduling system for cloud computing service users and providers,” in _2009 Third International Conference on Multimedia and Ubiquitous Engineering, MUE 2009, Qingdao, China, June 4-6, 2009_. IEEE Computer Society, 2009, pp. 295–299.
  * (147) R. Jayarani, R. V. Ram, S. Sadhasivam, and N. Nagaveni, “Design and implementation of an efficient two-level scheduler for cloud computing environment,” in _ARTCom 2009, International Conference on Advances in Recent Technologies in Communication and Computing, Kottayam, Kerala, India, 27-28 October 2009_. IEEE Computer Society, 2009, pp. 884–886.
  * (148) N. G. Grounds, J. K. Antonio, and J. T. Muehring, “Cost-minimizing scheduling of workflows on a cloud of memory managed multicore machines,” in _Cloud Computing, First International Conference, CloudCom 2009, Beijing, China, December 1-4, 2009. Proceedings_ , ser. Lecture Notes in Computer Science, vol. 5931. Springer, 2009, pp. 435–450.
  * (149) V. Hamscher, U. Schwiegelshohn, A. Streit, and R. Yahyapour, “Evaluation of job-scheduling strategies for grid computing,” in _Grid Computing - GRID 2000, First IEEE/ACM International Workshop, Bangalore, India, December 17, 2000, Proceedings_ , ser. Lecture Notes in Computer Science, R. Buyya and M. Baker, Eds., vol. 1971. Springer, 2000, pp. 191–202.
  * (150) A. Afzal, A. S. McGough, and J. Darlington, “Capacity planning and scheduling in grid computing environments,” _Future Gener. Comput. Syst._ , vol. 24, no. 5, pp. 404–414, 2008.
  * (151) M. O. Hill, “Diversity and evenness: a unifying notation and its consequences,” _Ecology_ , vol. 54, no. 2, pp. 427–432, 1973.
  * (152) R. H. Whittaker, “Evolution and measurement of species diversity,” _Taxon_ , vol. 21, no. 2-3, pp. 213–251, 1972.
  * (153) A. M. S. Kumar and M. Venkatesan, “Multi-objective task scheduling using hybrid genetic-ant colony optimization algorithm in cloud environment,” _Wirel. Pers. Commun._ , vol. 107, no. 4, pp. 1835–1848, 2019.
  * (154) Y. Yang, B. Yang, S. Wang, F. Liu, Y. Wang, and X. Shu, “A dynamic ant-colony genetic algorithm for cloud service composition optimization,” _The International Journal of Advanced Manufacturing Technology_ , vol. 102, no. 1-4, pp. 355–368, 2019.
  * (155) R. Jena, “Multi objective task scheduling in cloud environment using nested pso framework,” _Procedia Computer Science_ , vol. 57, pp. 1219–1227, 2015\.
  * (156) J. Tsai, J. Fang, and J. Chou, “Optimized task scheduling and resource allocation on cloud computing environment using improved differential evolution algorithm,” _Comput. Oper. Res._ , vol. 40, no. 12, pp. 3045–3055, 2013.
  * (157) W. Zhang and Y. Wen, “Energy-efficient task execution for application as a general topology in mobile cloud computing,” _IEEE Trans. Cloud Comput._ , vol. 6, no. 3, pp. 708–719, 2018.
  * (158) P. Pradhan, P. K. Behera, and B. Ray, “Modified round robin algorithm for resource allocation in cloud computing,” _Procedia Computer Science_ , vol. 85, pp. 878–890, 2016.
  * (159) C. Bai, S. Lu, I. Ahmed, D. Che, and A. Mohan, “LPOD: A local path based optimized scheduling algorithm for deadline-constrained big data workflows in the cloud,” in _2019 IEEE International Congress on Big Data, BigData Congress 2019, Milan, Italy, July 8-13, 2019_. IEEE, 2019, pp. 35–44.
  * (160) C. Bitsakos, I. Konstantinou, and N. Koziris, “DERP: A deep reinforcement learning cloud system for elastic resource provisioning,” in _2018 IEEE International Conference on Cloud Computing Technology and Science, CloudCom 2018, Nicosia, Cyprus, December 10-13, 2018_. IEEE Computer Society, 2018, pp. 21–29.
  * (161) C. Xu, J. Rao, and X. Bu, “URL: A unified reinforcement learning approach for autonomic cloud management,” _J. Parallel Distributed Comput._ , vol. 72, no. 2, pp. 95–105, 2012.
  * (162) Z. Peng, D. Cui, J. Zuo, Q. Li, B. Xu, and W. Lin, “Random task scheduling scheme based on reinforcement learning in cloud computing,” _Clust. Comput._ , vol. 18, no. 4, pp. 1595–1607, 2015.
  * (163) A. Kontarinis, V. Kantere, and N. Koziris, “Cloud resource allocation from the user perspective: A bare-bones reinforcement learning approach,” in _Web Information Systems Engineering - WISE 2016 - 17th International Conference, Shanghai, China, November 8-10, 2016, Proceedings, Part I_ , ser. Lecture Notes in Computer Science, vol. 10041, 2016, pp. 457–469.
  * (164) K. Lolos, I. Konstantinou, V. Kantere, and N. Koziris, “Elastic management of cloud applications using adaptive reinforcement learning,” in _2017 IEEE International Conference on Big Data, BigData 2017, Boston, MA, USA, December 11-14, 2017_. IEEE Computer Society, 2017, pp. 203–212.
  * (165) ——, “Rethinking reinforcement learning for cloud elasticity,” in _Proceedings of the 2017 Symposium on Cloud Computing, SoCC 2017, Santa Clara, CA, USA, September 24-27, 2017_. ACM, 2017, p. 648.
  * (166) M. Li, F. R. Yu, P. Si, W. Wu, and Y. Zhang, “Resource optimization for delay-tolerant data in blockchain-enabled iot with edge computing: A deep reinforcement learning approach,” _IEEE Internet Things J._ , vol. 7, no. 10, pp. 9399–9412, 2020.
  * (167) N. C. Luong, D. T. Hoang, S. Gong, D. Niyato, P. Wang, Y. Liang, and D. I. Kim, “Applications of deep reinforcement learning in communications and networking: A survey,” _IEEE Commun. Surv. Tutorials_ , vol. 21, no. 4, pp. 3133–3174, 2019.
  * (168) H. Wang, N. Liu, Y. Zhang, D. Feng, F. Huang, D. S. Li, and Y. Zhang, “Deep reinforcement learning: a survey,” _Frontiers Inf. Technol. Electron. Eng._ , vol. 21, no. 12, pp. 1726–1744, 2020.