# A Survey on the Use of Preferences for Virtual Machine Placement in Cloud
Data Centers

Abdulaziz Alashaikh  Eisa Alanazi  Ala Al-Fuqaha111 Abdulaziz Alashaikh is
with the Computer& Networks Engineering Department, University of Jeddah,
Saudi Arabia (asalashaikh@uj.edu.sa). Eisa Alanazi with Computer Science
Department, Umm Al-Qura University, Saudi Arabia (eaanazi@uqu.edu.sa). Ala Al-
Fuqaha is with the Computer Science Department, Western Michigan University,
USA (aalfuqaha@wmich.edu).

###### Abstract

With the rapid development of virtualization techniques, cloud data centers
allow for cost effective, flexible, and customizable deployments of
applications on virtualized infrastructure. Virtual machine (VM) placement
aims to assign each virtual machine to a server in the cloud environment. VM
Placement is of paramount importance to the design of cloud data centers.
Typically, VM placement involves complex relations and multiple design factors
as well as local policies that govern the assignment decisions. It also
involves different constituents including cloud administrators and customers
that might have disparate preferences while opting for a placement solution.
Thus, it is often valuable to not only return an optimized solution to the VM
placement problem but also a solution that reflects the given preferences of
the constituents. In this paper, we provide a detailed review on the role of
preferences in the recent literature on VM placement. We further discuss key
challenges and identify possible research opportunities to better incorporate
preferences within the context of VM placement.

##  1 Introduction

Today, virtualization technology has facilitated the hosting of millions of
services and applications on the cloud and simplified their deployments and
accesses. Many small, medium, and large organizations have been moving their
applications or even their entire data centers to the cloud. This is mainly
motivated to their economical advantage. in terms of real-estate, equipment
costs and manpower compared to private data centers. Cloud data centers also
maintain high degree of flexibility where services and applications can be
deployed, terminated, migrated, and replicated on-demand [91, 114]. In
addition, cloud data centers offer a wide range of customizable settings and
add-on features to suit the requirements and non-functional needs of different
customers.

Essentially, virtualization enables dynamic sharing of physical resources such
as processors, memory, and storage of the cloud infrastructure. An application
is installed on a logically isolated Virtual Machine (VM) that has limited and
pre-configured access to the shared resources of Physical Machines (PM) and
networking devices. The process of mapping the VMs to the existing PMs is
commonly known as the Virtual Machine Placement (VMP) problem. The VMP problem
is of paramount importance to the design and operation of cloud data centers.
Throughout the last decade, there have been many works addressing the VMP from
different perspectives. The majority of these works are concerned about the
profitability of data centers and the energy bill as well as other operational
aspects. While many proposals share similar objectives, they may have taken
different approaches, yielding a wide range of frameworks, models, and
algorithms.

A pre-requisite to the success of existing approaches is handling customer and
administrator preferences when looking for a placement solution. This is due
to the fact that cloud administrators and customers are often not satisfied
with _any_ placement solution. Rather, they are interested in the best, or at
least good, solution that meets their preferences. As a motivating example,
consider Bob who wants to place a VM that requires one unit of CPU and two
units of RAM. Alice, on the other hand, has a VM that requires two units of
CPU and one unit of RAM. The cloud data center, which is administrated by
Charlie, has two physical machines $P_{1}$ with 3 CPU units and 4 RAM units
and $P_{2}$ with 4 CPU units and 3 RAM units. Assume the objective is to
minimize the total wastage of RAM and CPU units. Obviously, the four possible
placements $(P_{1},P_{1})$,$(P_{2},P_{2})$,$(P_{1},P_{2})$, and
$(P_{2},P_{1})$ are all Pareto optimal with total wastage of 8.

Now, assume both Alice and Bob prefer placing their VMs in a PM with faster
CPU. In such case,the solution $(P_{2},P_{2})$ is preferred to all other
solutions. Furthermore, in case this solution is unattainable (e.g., violating
a constraint), $(P_{1},P_{2})$ and $(P_{2},P_{1})$ are both preferred to
$(P_{1},P_{2})$. These placements represent the most preferred feasible
options given both Bob and Alice preferences. Finally, Charlie prefers Bob’s
VM be placed on $P_{1}$ to better meet Bob’s service level agreement (SLA) and
therefore, $(P_{1},P_{2})$ is the optimal placement given the three
preferences. This example demonstrates that cloud customers and administrators
may have preferences that need to be considered while consolidating customer
VMs (e.g., customers prefer servers with faster CPUs) and solutions are no
longer indifferent but some are preferred to others.

There has been a number of attempts trying to capture preferences during the
VMP process. For instance, statements of the form “I prefer $X$ and $Y$ to be
placed on a different geographic location", “I prefer $X$ not to be co-located
with other VMs", or "I prefer $X$ and $Y$ to be placed on the same PM" are
particularly popular in the literature as customer preferences [138, 120, 121,
43, 40, 42, 13].

Furthermore, the cloud administrator is usually overwhelmed with a large set
of potential placements and has several preferences on the deployment process.
While any solution in the set is optimal to the underlying constraints and
objectives, some solutions are expected to be preferred (by the administrator
or customer) compared to others [61, 59, 127, 123, 121]. Neglecting the
preferences while finding a solution would discourage the customers from
subscribing to the cloud and administrators from using the proposed approach.
Hence, proper handling of preferences is a critical issue to address in the
design of a cloud.

The relevant literature is rich with survey papers on the VMP problem. There
are multiple VMP survey papers that interested readers may refer to, such as
[128, 80, 93, 91, 114, 81, 110, 85, 19, 139, 116]. Most of these surveys
provide multiple classifications and reviews of approaches to handle the VMP
problem including modeling, the objectives and constraints representation and
formulation, the techniques used to solve the problems, the scope and the
context in which the placement is taking place and more profoundly to its
variants. The focus on this paper is different as it aims to look deeper into
the notion of preferences in the VMP context. We begin with a clarification of
what is meant by preferences in the context of an optimization problem. Figure
1 shows a conceptual difference between the three types of knowledge.
Basically, the objectives and constraints of the problem instantiate its
optimality and feasibility regions, whereas satisfaction is commonly
attributed to preferences [112]. Indeed, one can promote a preference to an
objective or constraint. Many attempts in the literature adopt this
methodology and add preferences as an objective to the problem. We believe
such transformation albeit perhaps efficient, has its own drawbacks as it is
not always possible to construct objectives that reflect the preferences
without losing some information. Especially when the customer or administrator
pose complex preference statements over different aspects of the network.

ConstraintsObjectivesPreferences

Must be satisfied

Figure 1: The satisfaction level expected for different types of information:
Constraints are requirements that every solution needs to satisfy; Objectives
are the deriving force in searching and choosing one feasible solution over
another in the feasible region; Preferences represent wishes and desires.
Objectives and constraints are part of the problem representation and thus
need to be taken into account in any algorithm while preferences are the key
concept in personalizing an approach to the customers and administrators. VMP
with Preferences Source Type Approach Problem DM Customer Cloud Admin VM PM
Utility Importance Binary Relation Soft Constraints Metaheuristics Matching
MCDM Heuristics Placement Re-Placement Migration Figure 2: General
Classification of preferences based approaches on Virtualization

In this paper, we provide a detailed survey of VMP proposals that incorporate
soft constraints and preferences highlighting their types, implications, and
the algorithms used to handle them. We also discuss some challenges and
identify possible research opportunities to better incorporate preferences in
the context of VMP. A classification of the proposed approaches is given in
Figure 2. Some of the important aspects that need to be addressed when
proposing an approach include:

Source

identifies who initiates the preferences, the customer who owns the VMs, the
cloud administrator who owns the PMs, or both.

Type

depicts how the preferences are represented. (i.e., via utility function,
binary relation, etc.)

Approach

classifies the algorithm family used to develop an algorithm. In a broad
sense, they can be classified to heuristic, metaheuristic, and exact
algorithms.

Problem

describes the VMP problem context as there are different variations of the
problem, such as migration and re-placement.

It is important to realize that the different considerations of these aspects
impose some limitations on how the preferences are represented and processed.
More details about these aspects and their implications will be discussed
throughout the paper.

The rest of the paper is organized as follows: Section 2 presents background
regarding the VMP problem and preferences. We then review the recent
literature on the VMP problem with preferences in Section 3. Section 4
provides our insights on the potential benefits for the use of preferences in
the context of the VMP problem, followed by a discussion over current
limitations and future research opportunities. Section 5 concludes this work.

##  2 Background

###  2.1 VMP design problem

Figure 3: A 3-layer tree topology of a data center.

VMP is an essential process for the design and operation of cloud data
centers. VMP is concerned with the assignment of virtual machines to physical
machines (servers) in the data center. The decision of such placement highly
impacts the performance of the data center in terms of utilization levels,
traffic congestion level, and energy consumption, all of which the Decision
Makers (DMs) continuously try to optimize.

A wide range for data center network topologies have been studied extensively
in the literature. Basically, the architecture tells us how routers, switches,
and servers are connected, and what is the number of ports required on these
devices to implement the network. The choice of a topology has a great impact
on the cost, scalability, resiliency (in terms of alternative paths),
complexity in terms of number wires, and bandwidth and traffic distribution
within the data center. As an example, consider the tree topology shown in
Figure 3. It is comprised of three layers; namely, Core, Aggregation, and
Edge. The core layer is the data center gateway to the public Internet,
whereas the switches in the edge layer interconnect the servers (called
physical machines) to each other and to the gateway through the middle layer
switches. This architecture is a classic one that is commonly used as a
reference or baseline for evaluating other proposals. It is widely adopted due
to its simplicity. Despite that, it raises a number of technical concerns
regarding its scalability and resiliency. Hence, modern data centers have
evolved to utilize more sophisticated architectures such as Clos tree,
recursive and random topologies. More about alternative architectures and
their feature can be found in [94, 137, 33, 109].

Many virtualization tool developers have realized the importance of the
customizability of the virtual services and also enabled VMP based on multiple
attributes that also consider non-technical ones, e.g., Profile-Driven Storage
in vSphere 5.0 [9, 11], HP Moab [6], Server affinity policies in Oracle VM
server [8], and many others [15].

In realization to consumers varying needs, cloud service providers have
designed multiple options for VMs owners to choose from, in which an owner can
choose from a set of pre-defined VM types (i.e., a combination of disk space,
allocated memory, bandwidth, etc.) [3, 1]. On top of that, many providers went
to offer added-value or policy-based options, such as colocation, geo-
redundancy, availability, etc (see for instance, placement policies and
proximity placement group of Microsoft Azure [5, 2], phoenixNAP colocation and
security [4], IBM High availability [7] , and most recent mobile core network
virtualization on AWS [10]).

A basic VMP problem can be defined as follows: Given
$\mathcal{PM}=\\{P_{1},P_{2},...,P_{m}\\}$, a set of $m$ physical machines
($P_{k}$) and $\mathcal{VM}=\\{V_{1},V_{2},...,V_{n}\\}$, a set of $n$ virtual
machines ($V_{i}$), how to place each virtual machine into a physical machine
while optimizing a set of objectives and satisfying a set of constraints?

####  2.1.1 Objectives

Basically, the utmost goal when solving a VMP problem is to optimize some
metrics of interest. These metrics are identified as objectives. From a cloud
administrator’s perspective, the main objectives considered in the VMP are
highly related to maintaining profitable operation of the cloud. On top of the
list of these objectives is energy consumption. Typically, energy consumption
is modeled in terms of CPU utilization, and a possible minimization approach
is to distribute the load among existing PMs to reduce CPU usage on each PM.
Alternatively, turning off underutilized PMs and move their VMs to other PMs
appeared to be more a effective strategy. However, this approach brings the
issue of wear-and-tear to the surface as PMs maintaining higher resource
utilization are expected to fail sooner than others. In addition to energy
consumption, efficient packing of VMs with low defragmentation (resource
wastage index) increases the ability to accommodate more VMs in the cloud
which is an important feature from the profitability perspective. Moreover,
VMs may need to communicate with each other to perform a certain task. If two
communicating VMs are hosted on the same PM, then no traffic will go through
the switches. However, if they reside on two different machines, traffic has
to go through intermediate switches. This kind of traffic is called intra-data
center traffic as opposed to external traffic from/to the Internet. Excessive
intra-data center traffic may disrupt traffic between VMs and Internet as
intermediate switches may become congested. In order to prevent this from
happening, the traffic between peer VMs needs to be localized as much as
possible. This can be achieved by placing VMs that are highly interdependent
close to each other. However, this is mainly constrained by the capacities of
the PMs.

The objectives may also involve other metrics that contribute to the
profitability index. For example, traffic minimization to avoid
unpredicted/undesired congestion that may degrade performance, QoS
satisfaction, infrastructure reliability, VM availability, energy consumption
of switches, routers, and auxiliary equipment (e.g., cooling systems), etc.
For more examples on alternative objectives the reader is referred to [93, 80,
19, 110, 85, 139].

Generally, a single objective or multiple objectives can be involved in making
the placement decision. However, due to the nature of the VMP problem,
involving multiple dimensions, factors, stakeholders, and influential metrics,
researchers usually consider multiple objectives. As discussed above, these
objectives are highly interdependent and likely conflicting with each other.

####  2.1.2 Constraints

Mostly, any VMP problem can be seen as a pin packing problem. Hence, the
constraints are more or less similar to the pin packing problem: the
assignment constraint ensures that each VM is assigned to one PM, and one PM
can host multiple VMs, and the capacity constraint ensures the availability of
enough resources on the PM to utilize and fulfill VM requirements. Typically,
each VM requires resources described in terms of CPU cycles, RAM space, disk
space, network bandwidth, etc. While a server $P_{k}$, leveraging
virtualization techniques, can host more than a single VM, the resource
utilization (i.e., load) should not exceed its resource capacities. In
addition, other design considerations can be imposed as constraints. For
example, a cloud administrator may be interested in setting a limit on the
number of VMs migrations to avoid instability and interruption of the cloud
operation, or impose a constraint on security levels for some VMs.

####  2.1.3 Variants

There are multiple variants of the basic VMP problem that appeared in the
literature. The basic VMP problem may assume that there are some VMs hosted in
a DC and new VMs are to be placed on PMs based on the current utilization and
operational status of the DC. On the long-run, this might lead to
defragmentation of the resources. Therefore, rearrangement of the VM-PM
mapping is desired in which multiple VMs are consolidated into a PM to enhance
the mapping efficiency [115]. Migration is the technique used to move a
running VM from its current PM and re-assign it to a new PM. It is a powerful
technique that enables the cloud operator to reallocate resources and
reconfigure assignment based on current status of the machines [139]. However,
there are critical considerations when making the migration decisions,
particularly, related to the timing, frequency, and the cost of migrations
[124]. Hence, these factors exacerbate the complexity of problem in which
adopting soft rules or preferences to handle them and introduce a meaningful
trade-off is advisable. In our study, we refer to these VMP problem variants
when classifying the research efforts that appeared in the literature.

VMP can also be classified based on the number of the clouds involved [80]. A
single-cloud (private or public) VMP problem is concerned with the placement
of VMs into PMs within the same data center. Multiple clouds VMP, typically,
entails solving two assignment problems in which the system has to distribute
incoming VM requests over the different clouds and then optimize each cloud
individually. There are multiple architectures and models for multi-cloud VMP.
For example, hybrid cloud [122], multiple cloud with multiple administrators
[103], and others [82, 49, 58, 56]. Another related but different problem is
the cloud administrators selection problem where it is assumed that customers
express preferences and intermediate brokers negotiate for the best offer [14,
65, 79, 83].

###  2.2 Preferences

Preferences are ubiquitous. Whether we want to choose a restaurant for lunch,
watch a movie or purchase an airline ticket, we often have preferences over
the set of available alternatives. Thus, preferences play a key role in the
decision making process and have been an active topic of research in many
fields such as artificial intelligence, economics, and operations research
[64]. Preferences can be broadly classified based on their representation as
quantitative and qualitative. In quantitative preferences, users express
preferences with numerical values representing the utility of having one
alternative. On the other hand, qualitative preferences are expressed based on
comparative assessments of alternatives. Figure 4 shows a general
classification of preferences. The graphical models refer to a class of models
that have been introduced lately in the literature to represent the
preferences in a compact way. They rely on the notion of (conditional)
preferential independency among attributes and can be described in terms of
vertices and edges of a graph [16]. In what follows, we review the basic
methods used to represent preferences (interested readers can refer to [45]
for a comprehensive introduction to the field). Informally, preferences are
any additional knowledge that is not part of the problem requirements and
objectives, but represent the subjective judgments of the users in terms of
liking or disliking an alternative.

Preferences Qualitative Graphical Models Quantitative Binary relation
Lexicographic GAI-nets CP-nets Utility function Weights Soft constraints
Figure 4: A General classification of Preferences from representation point of
view.

In decision theory, preferences provide a way to evaluate alternatives based
on subjective judgments. It is typically assumed that every alternative $\bf
x$ is a vector over some $k$ attributes ${\bf x}=(x_{1},x_{2},\dots,x_{k})$
that represent the world. For instance, in the context of VMP, an alternative
can be represented by a possible placement of the VMs to PMs and the set of
alternatives represent the placements that are possible given the problem
constraints and objectives. Given a set of alternatives $S$, there are two
important questions that need to be answered by the preference model:

  * •

(optimality) What is the best alternative given the preferences?

  * •

(dominance) Given two possible alternatives $\bf x$ and $\bf y$, which one is
more preferred?

One of the simplest models for preferences is the weighted sum model obtained
by assigning an importance weight $w_{i}$ to every attribute $i$. Then
alternatives are assessed based on their value i.e., $w({\bf
x})=\sum_{i=1}^{k}w_{i}x_{i}$ where $w_{i}\geq 0$ and $\sum_{i=1}^{k}w_{i}=1$.
For instance, consider three possible alternatives $x_{1}=(10,20,30)$,
$x_{2}=(30,10,20)$, and $x_{3}=(20,5,25)$ defined on a set of three attributes
$a_{1},a_{2},a_{3}$ with importance weights $0.6$ for $a_{1}$ and $0.2$ for
both $a_{2}$ and $a_{3}$. Using the weighted sum model, $w(x_{1})=0.6\times
10+0.2\times 20+0.2\times 30=16$, similarly $w(x_{2})=24$ and $w(x_{3})=18$.
Given these values, one can conclude that $x_{2}$ is the best alternative, as
it has the highest value among other alternatives and also $x_{3}$ is
preferred $x_{1}$ since its value is higher.

Another representation is the value (or utility) function where alternatives
are mapped to real numbers $\mu:S\rightarrow\mathbb{R}$. Here, every
alternative $\bf x$ is mapped to a real number $\mu({\bf x})$ that represents
the desirability of the user for such alternative. Considering the above
example in the weighted sum model, we can have $S=\\{x_{1},x_{2},x_{3}\\}$ and
the alternative utility is simply its sum, i.e., $\mu(x_{1})=16$,
$\mu(x_{2})=24$, and $\mu(x_{3})=18$. In general, an arbitrary numerical value
can be associated with every possible alternative expressing its preference to
the decision maker.

Generally, a preference over a set of alternatives $S$, can be defined as a
binary relation on $S$ [64]. Several properties arise to different relations.
We start by general definition of preference relations.

###### Definition 1

A preference relation $\succeq$ is a preorder if $\succeq$ is reflexive and
transitive.

A preorder is called a partial order if it is also antisymmetric. A statement
of the form $x\succeq x^{\prime}$ is interpreted as $x$ is at least as good as
$x^{\prime}$. Given two alternatives $x,x^{\prime}$, one can derive some
useful conclusions between the two as follows

  1. 1.

$x$ is strictly preferred to $x^{\prime}$ iff $x\succeq x^{\prime}$ and
$x^{\prime}\not\succeq x$.

  2. 2.

$x$ is indifferent from $x^{\prime}$ iff $x\succeq x^{\prime}$ and
$x^{\prime}\succeq x$.

  3. 3.

$x$ is incomparable with $x^{\prime}$ iff $x\not\succeq x^{\prime}$ and
$x^{\prime}\not\succeq x$.

Incomparability captures the scenario where there is no possible information
in $\succeq$ to assess the two alternatives. However, being indifferent means
the user actually prefers both alternatives to the same level. One important
property that is usually assumed in practice is the completeness of the
partial order.

###### Definition 2

A partial order $\succeq$ is said to be complete iff $x\succeq x^{\prime}$ or
$x^{\prime}\succeq x$ for any $x,x^{\prime}\in S$.

A complete partial order is also known as a linear order or a ranking over a
set $S$. In such case, any two alternatives are comparable and one is
preferred to the other. Lexicographic preference relations are also common in
practice where an importance order over attributes are assumed.

Another model of preferences that has gained much attention in recent years is
the soft constraint model. In such model, constraints are associated with
numerical values representing the cost, importance, or preference and unlike
hard constraints, alternatives are not required to satisfy all of them. A
formal representation for soft constraints is Valued Constraint Satisfaction
Problem (VCSP) [105]. Lastly, the concept of reference point is widely used in
the evolutionary algorithms to capture users’ idealized vision of the solution
[37] and define a distance to relatively evaluate the closeness of a given
alternative to the reference point.

###  2.3 Applications of preferences in communications problems

Given the rise of multiple aspects involved in the design of communications
system that are of interest to decision makers (e.g., delay, availability,
utilization level, hop-count etc.) as well as concepts or design features that
are desired to achieve (e.g., load balancing, fairness, human-centric design,
etc.), many communications problems have been regarded today as multi-
dimensional or multi-criteria problems. Hence, incorporating the preference or
soft constraint concept into the problem can be useful to many applications in
communications beyond the VMP problem.

There are many problems in the communications and networking field where it is
advisable to consider such preferences and/or soft constraints. Such problems
include multiple decision makers in which each entity has its own requirements
(solid constraints) and preferences, or a problem where multiple metrics are
involved in the design with some sort of priority or importance (e.g., delay
is more important than reliability) and require insightful handling of
preferences in the design [30].

Consider for example, a multi-criteria routing problem. Typically, one metric
is regarded as an objective while others are given as constraints, or
alternatively the multiple metrics are aggregated into a single objective
function with some defined weights. In a multi-criteria approach, the problem
is approached from a different perspective where metrics to be optimized are
handled as objectives. This promotes optimization in all dimensions of the
problem which results in multiple solutions scattered in hybrid-planes. Using
preferences (e.g., bounds, sequences, etc.), solution space can be partitioned
into smaller regions each has a subset of the solutions with different
desirability level [31].

Similarly, a telecom network planning with multiple cost types (e.g.,
investment, maintenance, management, extension, etc.) and different design
criteria (e.g., capacities, availabilities, diameter, edge degree, etc.) is
another example where the adoption of preferences can help achieve more
desirable outcome [38]. This also can be extended to multilayer network design
to perform mapping between different layers or design overlay networks [100],
traffic grooming [102], or resource provisioning considering the cost of each
layer independently [133].

Another example in communication networks is the design of wireless sensor
networks which entails multiple design sub-problems such as sensor placement,
data aggregation, power assignment, etc. Typically, each considers a set of
objectives to be optimized (e.g., detection probability, latency, deployment
cost), and a proper definition of objective preferences or layout preferences
would improve our perception to the design [67, 97, 86].

Furthermore, the rapid evolution in many technological fields in
communications as well as the complexity associated with design decisions
stems from the nature of these problems, being multi-criteria, multi-
attribute, multi-stakeholders, make the adoption of the preference concept a
more viable and promising option. There are many problems where the
application of the preference concept can be seen advantageous in modeling and
solving the problem. For example, in the multiobjective SDN controllers
placement problem, trade-offs between objective metrics are needed [104] [57].
Another example is the resource scheduling problem in mobile networks where
multiple users of different requirements and classes of service are being
served with limited radio resources and changing conditions.

Functional51.0%Non-Functional30.6%Mixed18.4% Figure 5: The distribution of the
preference types in the literature from functional and non-functional
perspectives. 51% of the proposed approaches deal only with functional
preferences neglecting users’ genuine preferences on the problem.

##  3 Literature Survey

Pietri and Sakellariou, [91] surveyed the literature addressing the VMP
problem, and provided a set of common optimization techniques used to achieve
the desired objectives. In this section, we survey existing techniques that
incorporate the notion of preference and its variants in the context of the
VMP problem.

In the context of the VMP problem, preferences have been tackled by means of
hints, soft constraints, soft SLAs, relaxation bounds, weights on the
objectives, weights on the constraints, and affinity. One natural
classification is to distinguish between preferences that are issued directly
by the user (i.e., cloud administrator or customer) and preferences that are
provided implicitly from a certain measurement (e.g., ranking VMs based on
current communication cost). The later are related to measurable system
attributes. For example, CPU usage index, memory usage index, and inter-VM
traffic are operational factors that can be translated into preference
relations or rankings. A commonly proposed approach of this type is the
affinity-based VMP problem. It adopts the notion of affinity to evaluate which
VM is preferred to be hosted on a PM, and which PM a VM prefers [25, 112].

The former preferences are derived from people interacting with the system
including cloud administrators and customers. For example, a cloud admin may
prefer to assign some VMs to newer PMs due to cost considerations, or a
customer prefers to have his VMs located on the same PM or without sharing
other VMs [121, 98, 13]. Figure 5 shows the distribution of the approaches
from functional and non-functional perspectives.

Table 1 lists some examples of preference uses reported in the recent
literature. Clearly, the literature shows a wide variety of applications. For
instance, preferences have been utilized to maintain higher degree of non-
functional aspects such as SLA satisfaction and availability. Also, they have
utilized to represent the desire to assign sub-systems to servers with higher
resources, or to servers in certain geographic locations. One notable
application is ranking PMs based on their available resources and ranking VMs
based on their required resources. Such ranking is often used in the matching
algorithms.

Preference |  References |  Usage example
---|---|---
Resources |  [18, 20, 25, 29, 89, 35, 34, 47, 62, 65, 68, 69, 72, 119, 118, 126, 135, 136, 14] |  • A VM ranks PMs based on available resources or resources utilization ratio. • A PM ranks VMs based on required resource/s. • A VM prefers a PM with faster CPU.
Location |  [74, 92, 113, 117] |  • A customer prefers a geographical location over another for security or reliability concerns. • A cloud administrator prefers to place a group of VMs in a specific location for organizational purposes.
Inter-VM dependency |  [89, 76, 75, 112, 136, 99] |  • Preference information is derived from expected/measured traffic between VMs. • VMs collaborating in performing specific tasks or services.
Colocation |  [26, 41, 42, 112, 117, 122, 43] |  • VMs belonging to the same customer are preferred to be placed close to each other. • Flexible split preferences i.e., allow group VMs to be placed on different clouds or in the same cloud.
Anti-colocation |  [26, 42, 47] |  • Place VMs disjointly for security, reliability, or administrative purposes.
Sharability |  [79, 115] |  • A customer prefers to not share a PM with others.
Migration cost/overhead |  [65, 72, 134, 142] |  • VMs with lower migration overhead are more preferred to migrate. • VMs with migration cost (e.g., susceptible to SLA violation due to migration) are less preferred to migrate.
Availability/ Reliability |  [142, 17] |  • Include the degree of PM wear-and-tear (e.g., due to frequent PM ON/OFF cycles) in weighted function. • Replicate a VM to improve its availability.
Cost |  [20, 17, 56] |  • A customer may prefer a lower cost PM. • A cloud admin may prefer to place VMs based on their payment scale and the cost of the infrastructure. • Typically, the cost computed as utility function composing multiple attributes or just monetary value.
Ordering |  [13, 52, 54] |  • Preference order of preferred PM to host a VM or a set of VM.
Distance |  [29, 66, 134] |  • An application prefers computing and storage PMs pair with smaller distance. • An application prefers to be placed near a specific PM.
Objectives |  [12, 22, 39, 77, 103, 125, 54] |  • The extent to which an objective is prioritized over other objectives.
SLA |  [108, 142] |  • Soft SLA tolerates some violations.
Security |  [20, 141] |  • Security preferences can be modeled as a function of the system diversity. • Place VMs according to their security preferences and PMs capabilities.
Other/ Unspecified |  [98, 120, 121, 130, 87, 74, 106, 44] |  • a generic model in which preferences information are customized/defined by the user/DM. • Place a VM in a specific PM [74, 106].
Table 1: Existing application scenarios for preferences

In the remainder of this section, we review the literature based on the
proposed approaches. We classify existing works into five broad classes of
algorithms: Metaheuristics, Matching, MCDM, Heuristics, and others. Table 2
shows an overview of the existing works based on this classification. At the
end of this section, we compararatively discuss the advantages and
shortcomings of the five classes.

Table 2: Existing Approaches that handle preferences in VMP Method |  References
---|---
Metaheuristics |  Simulated annealing [120, 121, 12], NSGA-II [22, 13], Improved grouping genetic algorithm [39], Memetic Algorithm [54, 77] Ant-colony [76, 75], heuristics [79] Hybrid algorithm [103]
Matching |  Stable matching [69, 41, 135, 66, 35, 34, 126, 68, 72, 56], Matching game [20, 99], Three-sided stable matching [29], Markov approximation & stable matching [89], College admission problem [65, 136], Resource Balancing Placement matchmaking [74] Uncertainty principle of game theory [92], Egalitarian stable Matching [134], Game theory [141]
MCDM |  TOPSIS [119, 118, 18, 44], PROMETHEE [62], General MCDM [98], Fuzzy MCDM [17], AHP [111]
Heuristic |  First-Fit [108], Next-Fit and First-Fit Decreasing [42], Opportunistic-Fit [47], Multiple Heuristics [115]
Other |  Fair proportional scheduler [25] Bin packing [26, 52], ILP repacking algorithm [106], Distributed bartering algorithm [112] Random search (biased statistical sampling) [117] Random search (biased statistical sampling) [122] Brute-force [125], Constraints-based model [130], NA [142], A framework [87, 82], cloud broker [14]

###  3.1 Metaheuristic-based Approaches

####  3.1.1 Genetic & Evolutionary Algorithms

Evolutionary Algorithms (EAs) [32] have proven to be a successful approach in
solving many optimization problems including the VMP problem. EAs search for
solutions by producing a population of candidates in every run. The population
_evolves_ by applying crossover and mutation operators. In the past decade,
several preference-based EAs have been proposed (see for example [96] for a
survey on preference-based EAs) where the goal is to incorporate users’
preferences and desires when solving the problem. Thus, returning solutions
that are not only optimized but also preferred by the user.

For VMP, preferences have been expressed as a utility function of the
administrator for different combinations of SLA satisfaction, reliability and
energy [39] and as tie breaking mechanism for final solutions of the MOP [76].
Also, preferences have been used to correlate two VMs to be placed on the same
PM based on historical experience; thus, minimizing number of active PMs [75].
Indeed, one of the most well-suited roles of preferences in EA is to reduce
the final Pareto set. Thus, the work of Ihara et al., 2015b [54], Blagodurov
et al., [22] and López-Pires and Barán, [77] considered administrator
preferences to reduce the final solutions of the VMP problem. A decentralized
data center over different local departments have been considered in Saber et
al., [103] where it assumes every department administrator has a (possibly
conflicting) preference. Alashaikh and Alanazi, [13] considered the case where
some VMs have a ranking over which PM to be placed and used a variant of NSGA-
II to return preferred solutions based on the notion of ceteris paribus
preferences.

Table 3: Detailed overview of the existing literature (Metaheuristics)

Reference | Perspective |  Problem Type |  Objectives |  Algorithm |  Preferences |  Comments
---|---|---|---|---|---|---
Addya et al., [12] | cloud provider |  placement |  Energy – Revenue |  Simulated annealing |  Objectives |  A bi-objective optimization model to maximize revenue and minimize energy consumption. A trade-off between the two objectives is maintained based on DM’s preference.
Alashaikh and Alanazi, [13] | cloud provider |  placement |  Traffic – Energy – Resource wastage |  MOEA |  Ordering |  Placement decision preferences are given a priori as a total order.
Blagodurov et al., [22] | cloud provider |  placement |  Traffic – Energy – Resource consumption |  NSGA-II |  Objectives |  Posteriori preferences on objective space.
Deng et al., [39] | cloud provider |  consolidation |  Energy – Resource consumption – Availability/Reliability |  MOEA - improved grouping genetic algorithm |  Objectives |  Defines a weighted utility functions as a weighted combination of SLA satisfaction, reliability and energy impacts, where the weights on each metric represents the DM preferences.
Ihara et al., 2015b [54] | cloud provider |  placement migration |  Traffic Energy – Revenue – No. of migrations – Migration cost |  Metaheuristics Memetic Algorithm |  Ordering |  Set of Pareto solutions with preferences applied on objective space. Multiple selection strategies such as distance to origin and Lexicographic Order (Each objective function is given in an order of evaluation)
Liu et al., [76] | cloud provider |  consolidation |  Energy – No. of migrations |  Ant-colony |  Inter-VM dependency |  Obtains Pareto-based solutions and then the trade-off between the two objectives is adjusted based on the DM’s preferences.
Liu et al., [75] | cloud provider |  consolidation |  Resource consumption – min. active PMs |  Ant-colony |  Inter-VM dependency |  The ant-colony algorithm deposits the pheromone between VMs rather than between a VM and a PM and represents the preference between VM and PM. The preference is based on the historical experience of packing the VM together with those VMs that have already been placed on the PM.
López-Pires and Barán, [77] | cloud provider |  placement |  Traffic – Energy – load Balancing – Revenue – QoS priority |  Metaheuristics Memetic Algorithm |  Objectives |  Preference knowledge is utilized to reduce search space by imposing upper and lower bounds on objectives.
Mahalingam and Sengottaiyan, [79] | user – multi-cloud |  placement |  satisfaction |  Simulated Annealing-based |  Sharing |  The model allows customers to specify their preferences (along with other QoS requirements) as hints based on whether their VMs can be located with other users? VMs or alone. Further, it considers soft constraints that only contribute to the satisfaction score and not enforced as the hard ones. (Uses Nefeli [121])
Saber et al., [103] | multi-cloud |  re-placement |  Energy – Cost – Migration cost – Availability |  Metaheuristics |  Objectives |  Hybrid cloud managed by multiple entities, each of which has different requirements and preferences in maintaining its virtual cloud. Preferences are incorporated as weights on the objectives. Three algorithms are utilized greedy algorithm to find initial solutions, NSGA to diversify, and a local search to obtain more non-dominated solutions.
Tsakalozos et al., [120] | user |  migration |  other/unspecified |  Simulated annealing-based & 2-phase heuristics |  other/unspecified |  Adopts hints to represents user preferences, internal physical cloud infrastructure, and high-level management goals that the DM seek. Fine-grained constraints refer to a) all specifics regarding the hosting capacity including features and resource availability in PMs, b) user-provided deployment hints and c) cloud administrative goals. A ranking algorithm finds different sets (Cohorts) of PMs that satisfy VM requests in terms of resources. Simulated Annealing algorithm to select a single cohort that satisfy preferences.
Tsakalozos et al., [121] | user – cloud provider |  placement |  other/unspecified |  Simulated annealing |  other/unspecified |  proposes Nefeli, a hint-based VM placement algorithm. In Nefeli, a predefined set of preferences are mapped onto a utility function with some weight (a score) on each preference criterion. The model also utilizes soft and hard constraints for both consumer and provider.

####  3.1.2 Simulated Annealing

Another common technique to solve the VMP problem is Simulated Annealing, a
single-based metaheuristic technique to solve optimization problems [60]. A
number of proposals have developed simulated annealing-based heuristics to
solve the VMP problem. Tsakalozos et al., [120] developed a two-phase VM-to-PM
placement scheme in which a group of PMs, termed cohort, are first identified
as potential host based on some properties, then the final constrained mapping
is resolved taking into account customers’ special requirements and
preferences. The goal of selecting a subset of all available PMs in the first
phase is to reduce the number of constraints and the search space during the
second phase. In addition to resource and cloud specifications constraints,
the scheme considers customers’ deployment hints such as colocation of
multiple VMs in a single PM and reserving a whole PM for a single VM. The
authors proposed two heuristics for filtering cohort PMs and placement based
on simulated annealing.

Moreover, Tsakalozos et al., [121] developed a cloud gateway, Nefeli, that
takes customers’ deployment hints when mapping incoming VMs to PMs. The hints
realize several preferences related to VMs deployment such as colocation,
availability, proximity and other information.

Their objective is to optimize cloud energy efficiency and load balancing
while meeting cloud admin constraints. At the same time, Nefeli tries to
satisfy customers’ preferences but it may only satisfy parts of them and
ignore the rest based on the available resources. For evaluating multiple
placement solutions, Nefeli adopts a weighted score function on constraints,
where a weight derived from customers’ hints is associated with each
constraint. The set of constraints is further decoupled into hard and soft
constraints. While the first is strict and has to be met, the second only
contributes to the score function. Eventually, the optimal solution is defined
as the one with the highest score. The algorithm adopts a simulated annealing-
based algorithm to solve the placement problem and a search space reduction
method was used for large scale problems.

Mahalingam and Sengottaiyan, [79] adopted Nefeli to develop a cloud service
selection mechanism that takes into account VMs’ preferences for price and QoS
and search for acceptable offers. Lastly, Addya et al., [12] developed a bi-
objective optimization model to maximize revenue and minimize energy
consumption and solve it using a simulated annealing technique. The proposed
model attempts to maximize the revenue and minimize the power budget. A trade-
off between the two objectives is maintained based on administrator’s
preference.

Table 3 provides a summary to the works presented here.

###  3.2 Matching

VMP can be naturally described as a matching problem. The set of participants
in this matching are often the available VMs and PMs where every participant
expresses their preferences over who to partner with. Such preference
relations are due to some metrics such as transmission cost or processing
capabilities and requirements which can be measured or estimated. The result
of the matching is that every VM is placed on exactly one PM. Stability is an
important property where a matching is considered _stable_ when no machine (VM
or PM) would prefer another machine to its current partner [90].

The literature shows different roles of preferences when matching VMs to PMs.
VMs could rank PMs based on some properties such as transmission cost [134],
migration time and cost [65] , processing capabilities [136] , and security
[20] while PMs rank VMs based on their migration overhead [65] , CPU and
resource utilization [136, 29], and risk [89, 20]. Kim and Lee, [66] also
considered the case where preferences are not necessarily a ranking and ties
are allowed. The goal is often to minimize the overall dissatisfaction score
which is defined as sum of ranks. This score can be intuitively seen as the
average happiness of VMs and PMs.

Preferences have been utilized to place high throughput applications near
computation nodes to minimize the cost of communication [69]. Dhillon et al.,
[41] addressed consolidating multiple VMs on the same PM while minimizing
degradation performance. It assumed that every VM acts as a selfish agent and
has a preference over which VMs to partner with on the same PM. The preference
is based on some hidden cost function related to the VM. Kella and Belalem,
[65] considered the case of multiple cloud providers with intermediate brokers
where customers express preferences. The live migration problem has been
formulated as a matching problem with the goal of reducing the overall energy
consumption without degrading QoS. Cui and Tso, [35] considered networks where
the policies, which reside on middle boxes such as firewalls and load
balancers, etc., are migrated first then a matching between PMs and VMs based
on the communication cost and resource usage is applied. In [74], a
descriptive language based on XML for the preferences of data center managers
was proposed. The goal is to automate the matching procedure while classifying
preferences into two classes; namely, flexible and inflexible. Wang et al.,
[126] addressed the problem when VMs and PMs cooperate on a mutual objective
(reducing the overall energy consumption). Chu et al., [29] added storage PMs
as a third set into the matching where storage PMs prefer smaller VMs. Pillai
and Rao, [92] considered the case where IaaS request is beyond the capability
of a single PM and therefore developed a mechanism to form coalition with the
goal of minimizing the total execution time based on uncertainty principle in
game theory. The work in [141] addressed the problem of migrating VMs for
security preferences to make it harder for attackers to locate targeted VMs.
Lastly, Korasidis et al., [68] enhanced the scalability of the stable matching
technique for the VMP problem by considering only the top-k alternatives in
the preference list. Table 4 summarizes the presented works of this method.

Table 4: Detailed overview of the existing literature (Matching)

Reference | Perspective |  Problem Type |  Objectives |  Algorithm |  Preferences |  Comments
---|---|---|---|---|---|---
Azougaghe et al., [20] | cloud provider |  migration |  Energy – Resource consumption |  Matching game |  Resource – Cost – Security |  VMs preferences and PMs are represented as utility functions that take into considerations multiple attributes such as resources and security level for VMs and risk, energy, and revenue for PMs.
Chu et al., [29] | cloud provider |  placement |  Cost |  Three-sided Stable matching |  Resource – Distance |  PM nodes are decoupled as computing and storage nodes. three-sided matching among computing, storage resources and applications, each with its preference list
Pham et al., [89] | cloud provider |  placement |  Traffic Energy |  Markov approx. and Stable matching |  Resource – Inter-VM dependency |  vNFs of the same service chain maintain a preference list on PMs, while each PM maintains a preference list on vNF instances.
Cui and Tso, [35] | cloud provider |  migration consolidation |  Resource consumption – Policies |  Stable matching |  Resource |  preference knowledge is the cost of paths and links for VMs traffic.
Cui et al., [34] | cloud provider |  migration consolidation |  Resource consumption – Policies |  Stable matching |  Resource |  same as above.
Dhillon et al., [41] | cloud provider |  placement |  Satisfaction |  Stable matching |  Colocation |  Every VM acts as selfish agent and has a preference (strict total order) over other VMs to partner with. The system tries to minimize the degradation of VMs placement problem and find a solution that minimizes the global performance degradation.
Ranjana et al., [99] | cloud provider |  placement |  Energy –comm. cost – min. active PMs |  Stable matching |  VM dependency |  Affinity-based algorithm.
Jain et al., [56] | multi-cloud |  placement |  |  Stable matching |  Cost |  multiple cloud providers, intermediate brokers and end users
Kella and Belalem, [65] | multi-cloud |  migration |  Energy |  Stable matching /college admission problem |  Resource – Migration cost |  VMs preference list contain PMs put in an ascending order according to the transfer time. PMs preference list contain VMs put in an ascending order according to their impact to cpu utilization.
Kim and Lee, [66] | cloud provider |  migration |  |  Stable matching |  Distance |  the model allows ties in the preference list ordering (i.e., preferences are not given in a strict order)
Korasidis et al., [68] | user – cloud provider |  placement |  other/unspecified |  Stable matching |  other/unspecified |  Addresses algorithm scalability.
Korupolu et al., [69] | cloud provider |  placement |  Cost |  Stable matching |  Resource |  One of the first proposals that applied stable matching to the VM placement problem. It considers compute and storage PMs, and applications of different computation and data needs. Requirements and other applications preferences can be integrated into a weighted utility function.
Li et al., [74] | user |  consolidation |  min. active PMs – Resource wastage |  Stable matching (resource balancing and matchmaking) |  Location – Specific PM |  Defined normal and categorized types of preferences and flexible and inflexible preferences.
Li et al., [72] | cloud provider |  consolidation |  Energy |  Stable matching |  Migration cost – Resource |  Modeled as a matching problem on a weighted bipartite graph, with VM migration cost used as a weight on edge between VM and PM. Each VM candidate for migration has a preference list on a subset of PMs chosen to accommodate migrated VMs. The list is ordered based on weights.
Pillai and Rao, [92] | cloud provider |  placement |  Revenue |  Stable matching and uncertainty principle of game theory |  Location |  The proposal considers forming coalition between PMs to satisfy requirements that are beyond single machine capabilities. The PM has preference over which coalition of machines to go with based on expected payoff.
Wang et al., [126] | cloud provider |  migration |  Energy |  stable matching |  Resource |  PMs and VMs have mutual objective of freeing overloaded PMs and utilizing underutilized PMs in order to reduce overall energy consumption. The proposed algorithm tries to achieve a reasonable trade-off between power consumption and SLA violation. Preference list is computed accordingly.
Xu and Li, [134] | cloud provider |  migration |  other/unspecified |  Egalitarian stable Matching |  other/unspecified |  A VM or a server ranks the agents on the other side based on some criteria, which is its preference list. An algorithm then computes a matching that cannot be improved by any pair of VM and server. VMs rank servers in an ascending order of the transmission cost. Servers rank VMs in an ascending order of the migration overhead.
Xu and Li, [135] | user cloud provider |  placement |  Resource consumption – other/unspecified |  Stable matching |  other/unspecified |  A VM preference list ranks servers according to their resources. PMs may prefer packing larger or smaller VMs recognizing DM’s preference of consolidation or load balancing, respectively. Proposed a revised DA algorithm that finds weakly stable matching which can be an effective approach for handling large-scale problems.
Xu et al., [136] | user cloud provider |  placement |  Resource consumption |  Stable matching/ College admission problem |  Resource – Inter-VM dependency |  The stable matching theory is applied to generate an optimal mapping from containers to physical servers.
Zhang et al., [141] | cloud provider |  migration |  Cost |  game theory |  Security |  Users security preferences are modeled as a linear function of the system diversity (using weighted function of the total number possible VM-placements). The cost accounts for migration time and number of migrations.

###  3.3 MCDM

Multi Criteria Decision Making (MCDM) [64] is a field of study to find and
analyze preferred options in complex problems under conflicting criteria. Two
well-known techniques in MCDM are AHP and TOPSIS [128]. Araujo et al., [17]
presented an MCDM method to rank the best cloud infrastructure based on a
customized dependability metric and the cost. The customized metric takes
customer service constraints such as reliability, downtime, and cost into
consideration. The model allows users to express their preferences on cost
versus availability, through a planning and analysis tool, to obtain the best
configuration for them. Tarighi et al., 2010b [119], Tarighi et al., 2010a
[118] presented a method for VM migration between cluster nodes (a group of
PMs) using TOPSIS algorithm The aim of their algorithm is to achieve load
balancing among PMs by moving load from the most overloaded PMs to the least-
overloaded PMs and attempting to minimize migration cost defined in terms of
amount of data copied and transferred. The TOPSIS method is a multi criteria
decision making technique. It stands for order preference by similarity to
ideal solution. It is based on selecting an alternative solution, from a
finite set of alternatives, that is closest to the ideal solution and farthest
from the negative ideal solution. The ideal solution is the solution that
maximizes the benefits and minimizes the cost, whereas the negative ideal
solution does the opposite. Typically, each alternative is evaluated with
respect to some attributes (decision criteria), and each attribute is assigned
a weight based on its importance (i.e., given by the user).

The alternatives are ranked according to their relative closeness to the ideal
solution. The closer the alternative, the better.

PM nodes are sorted from the highest overloaded to the idle ones based on some
criteria such as CPU, RAM, NET, etc. Each attribute is given a weight. Then
after finding the most saturated PM, the algorithm tries to find the best VM
candidate for migration based on similar criteria w.r.t. VMs. The proposed
TOPSIS algorithm can receive three types of information including
deterministic, linguistic, and fuzzy information.

Arianyan et al., [18] proposed a TOPSIS algorithm aiming to optimize energy,
SLA, and number of migrations in a cloud data center. The algorithm selects
VMs to be migrated when overloaded PMs are predicted. It considers a decision
matrix of the resource utilizations and capacities of VMs. Each VM is assigned
a score for each resource criterion based on the ideal points concept. The
total score of a VM is the weighted combination of the scores of each
criterion. The weights for each VM are set independently. Similarly, Feng et
al., [44] developed a TOPSIS algorithm that incorporates AHP to find weights
of coefficients of the resources decision matrix. Rane and Srivastava, [98]
introduced a cloud broker architecture to help application customers select
their cloud provider considering both functional and non-functional
requirements on the service. While functional requirements must be satisfied,
non-functional requirements are associated with weights to specify their
priority on these requirements. Further, an MCDM constraint significance
factor realizing the preference of each constraint is utilized. Then, the
method tries to maximize customers’ satisfaction. Kabir et al., [62] proposed
a two phase VMP algorithm that first selects a cluster of PMs and then a PM
inside the cluster to place a VM. The proposed algorithm is based on the MCDM
method PROMETHEE. Overall, the method is based on outranking alternatives.
Preferences on each criterion are quantified as real numbers and aggregated
onto a weighted preference index. Then, each alternative is evaluated by the
number of alternatives that outrank it and the number of alternatives it
outranks based on the preference index. The alternative with the largest net
outranking is the most preferred one.

Table 5 summarizes the works presented here.

Table 5: Detailed overview of the existing literature (MCDM)

Reference | Perspective |  Problem Type |  Objectives |  Algorithm |  Preferences |  Comments
---|---|---|---|---|---|---
Araujo et al., [17] | user |  placement |  Availability – Cost |  Stochastic model & MCDM |  Objectives |  The authors use stochastic models to assess and evaluate different clouds from user’s perspective. Assessment results are then used to rank solutions, taking into account the user’s preference of each criterion (set as a weight).
Arianyan et al., [18] | cloud provider |  consolidation |  other/unspecified |  TOPSIS |  other/unspecified |  Preferences are expressed as weights for each criteria. The weights are computed based the average utilization of all system resources in the data center (i.e., functional). The utmost objective is to remove hotspots.
Feng et al., [44] | cloud provider |  consolidation |  Cost |  TOPSIS/AHP |  Resources |  The algorithm is mainly concerned about the migration decision.
Kabir et al., [62] | multi-cloud |  placement |  Resource consumption |  PROMETHEE |  other/unspecified |  a preference function is used to denote the preference of solution over another for each criterion. Then, each solution is evaluated by the number of alternatives outrank it and the number of alternatives it outranks based on the preference index. The solution with the highest net outranking flow is considered as the most preferred.
Rane and Srivastava, [98] | user |  re-placement |  Satisfaction |  MCDM |  other/unspecified |  Defines a constraint significance factor that signifies the preference of customer corresponding to each constraint.
Son and Huh, [111] | user – cloud provider |  placement/ migration |  Resource utilization – QoS – no. of migrations – Energy |  MCDM/ AHP |  Objectives |  Proposes to use Analytical Hierarchy process (AHP) coupled with a fuzzy interferer system to solve the VMP problem. By monitoring resources and analyzing service requirements, a placement decision can be obtained from the fuzzy system. Then through AHP algorithm, a placement action is performed. The AHP is managed to provide analysis of the weights for each metrics involved in the decision.
Tarighi et al., 2010b [119] | cloud provider |  migration |  load Balancing Migration cost |  Fuzzy TOPSIS |  Resource |  deterministic, linguistic, and fuzzy preference information.
Tarighi et al., 2010a [118] | cloud provider |  migration |  load Balancing Migration cost |  TOPSIS |  Resource |  same as other one

###  3.4 Heuristics

Shi and Hong, [108] proposed a First-Fit algorithm aiming to maximize the
profit without violating the SLA and the power budget. The algorithm considers
two kinds of SLAs: the hard SLA and the soft SLA. A hard SLA is not tolerable
and violating any of its terms yields no revenue. For a soft SLA, application
customers can tolerate performance degradation to some extent and provider
revenue suffers certain loss accordingly. The more elastic the VMs are, the
more VMs can be consolidated onto a PM. Thus, this elastic model allows
providers to manage revenue based on SLA. Moreover, Dow, [42] used modified
Next-Fit and First-Fit heuristics to solve the VMP pin packing problem. They
proposed to decompose the VM consolidation into two phases; namely, an
abstract packing phase that satisfies constraints, and a concrete assignment
phase based on preferential colocation and anti-colocation. Alternatively,
Georgiou et al., [47] developed an Opportunistic-Fit algorithm addressing the
problem of placing VMs with an Infrasturctue-as-a-service (IaaS) request, in
which a group of VMs are jointly consolidated in the cloud. Considering DC
PortLand architecture, they proposed two VM placement algorithms that exploit
users’ hints about the desired features of the requested IaaS. Users specify
their requirements on the VMs and their colocation preferences through an XML
document. The hints considered here are concerned with the bandwidth needs for
specific pairs of VMs and the anti-colocation constraints for pairs of VMs. In
the first algorithm, a greedy algorithm places requested VMs as close as
possible to each other while satisfying colocation requirements as much as
possible; otherwise, a path with sufficient bandwidth is established between
them. The second algorithm tries to minimize the time expended for the
placement decision by identifying promising neighborhoods of PMs for deploying
the IaaS first before applying the greedy placement algorithm.

###  3.5 Other Methods

In this part, we review several works that adopt exact methods or general
customized heuristics.

Viswanathan et al., [125] addressed the placement of High-Performance
Computing (HPC) applications in clouds. They proposed a VM consolidation
strategy that aims at optimizing resource utilization and energy efficiency
while satisfying application’s QoS. A brute-force search algorithm is used to
optimize energy efficiency and performance with a parameter utilized to adjust
the possible trade-off between them. He et al., [52] proposed a matching
algorithm for VMs migration addressing IaaS management in clouds. For incoming
IaaS request, the algorithm finds a suitable PM based on available resources.
If there are multiple suitable PMs, the system gives preferential attention to
more important customers. A PM is recognized by the different properties it
has, and a customer may express different preferences over these properties.
The preferences are given in values indicating their importance to the
customer. Then, the algorithm chooses the most suitable PM that satisfies the
preference order the most. Wright et al., [130] developed a two phase resource
selection model that can handle a combination of hard and soft constraints.
The model intends to enable users to match their application requirements to
infrastructure resources. A direct search algorithm is proposed to solve the
problem.

Chen et al., [25] considered the problem of mapping virtual CPUs of VMs to
physical ones. A proportional scheduler (PS) is proposed to provide fair
allocation of CPU resources among virtual CPUs. It assigns a weight to every
VM based on its computational requirements. CPU-affinity is defined as a
property of a virtual CPU that describes which physical CPU it can run on. In
their study, they illustrated that the inappropriate assignment of weights may
produce unfair proportional sharing of CPU. Sonnek et al., [112] presented a
decentralized affinity-aware migration technique that dynamically migrates VMs
based on network affinity between pairs of VMs in order to minimize intra-
cloud traffic. Affinity here is defined as VMs interdependency (i.e., volume
of traffic monitored between pairs of VMs). The authors developed a
distributed bartering algorithm that strives to place VMs on a PM that is
closer to the data they need. VMs have preference ordering on which PM they
migrate to based on the monitored intra-traffic. The VM is responsible for
negotiating the placement with PMs which improves the scalability of the
algorithm and minimizes the overhead. In a similar vein, Ranjana et al., [99]
classify PMs based on CPU and communication bandwidth and then apply affinity-
based matching to minimize energy consumption, number of active PMs, and
communication cost. Zhou et al., [142] developed three live VM storage
migration mechanisms. The mechanisms introduce a trade-off between the
efficiency and cost of VM migration. They also consider user experience (IO
penalty), cluster management (migration time) and device usage (degree of
wear) in the migration decision by utilizing a migration cost (utility
function) in which weights reflect different design preferences.

Tantawi, [117] proposed a biased random search method based on biased
statistical sampling. It maps VMs and other workloads into PMs and other
physical entities so that an objective function is optimized. The algorithm
takes into account both administrators’ and users’ requirements and allows
users to specify their preferences over the constraints (hard/soft
constraints) over some criteria such as location. The biasing pushes the
solution towards satisfying constraints and preferences. Unuvar et al., [122]
adopted the same method to address VMP in hybrid clouds. The placement
decision takes into account whether private or public cloud is preferred for a
customer and a customer preference on splitting VMs over private and public
clouds. Sotiriadis et al., [113] presented VMs migration within OpenStack
cloud platforms considering the FIWARE platform 222FIWARE is developed within
the FI-STAR FP7 project that utilizes cloud technologies and IoT to support
healthcare services and applications, http://www.fi-star.eu.. The migration
tool is developed following the FI-WARE conceptual model and offers a set of
software modules, called Specific Enablers (SEs), that allow flexible
development of healthcare applications. Within the deployment process of VMs,
the system stores and retrieves users’ preferences including geographic
location preference. This is done utilizing a graphical interface to configure
customer preferences. Chen et al., [26] proposed a Joint Affinity Aware
Grouping and Bin Packing (JAGBP) method that aims to minimize application
performance cost and maximize resource utilization. The affinity relationships
of VMs define colocation and disperse placement preference based on VMs
interdependency. They define three affinity types and affinity relations;
namely, colocation placement affinity/relation (co-affinity/relation), no-
colocation placement affinity/relation (no co-affinity/relation) and fixed
placement affinity/relation (fixed-affinity/relation). JAGBP was empirically
found to improve the above objectives compared to non-affinity aware
placement.

Table 6 summarizes the works presented in sections 3.4 and 3.5.

Table 6: Detailed overview of existing literature (Heuristics & others)

Reference | Perspective |  Problem Type |  Objectives |  Algorithm |  Preferences |  Comments
---|---|---|---|---|---|---
Chen et al., [25] | cloud provider |  placement |  Resource consumption |  Heuristics |  Resource |  VMs are classified into domains based on their CPU-affinity.
Chen et al., [26] | cloud provider |  placement |  min. active PMs |  Bin packing |  Colocation – Anti-colocation |  Affinity-aware placement – hard constraint
Dow, [42] | cloud provider |  consolidation |  min. active PMs |  Heuristics (pin-packing using next-fit and ffd) |  Colocation – Anti-colocation |  decomposed into two phases; resources computation then colocation.
Georgiou et al., [47] | cloud provider |  placement |  other/unspecified |  Heuristics (Opportunistic-fit greedy algorithm) |  Resource – Anti-colocation |  IaaS requests. Used the notion of hints to allow users specify their bandwidth needs for specific pairs of VMs and anti-collocation constraints for pairs of VMs. aim to faster solution time. Flexibility of constraints.
He et al., [52] | user – cloud provider |  migration |  Resource consumption – Cost |  Heuristics |  Ordering |  Each PM machine has a set of properties associated with its resources and attributes (e.g., RAM, type of OS, etc). If multiple PMs are suitable for hosting a VM (i.e., satisfy the requirements), then the system looks for user’s preferences.
Ooi et al., [87] | cloud provider |  placement – replication |  Availability |  Team formation |  other/unspecified |  A framework for resource evaluation to identify preferable resources and prepare for service placement.
Sedaghat et al., [106] | cloud provider |  repacking |  Resource utilization |  ILP |  other/unspecified |  Repacking by changing the configuration of VMs (either number or sizes) that reside in a server, as the load changes. The goal is minimizes resource utilization cost accounting for both price and performance issues. A decision of repacking or not can be based on the calculated cost and other factors e.g., expected time to next change, interruption time, etc.
Shi and Hong, [108] | cloud provider |  placement |  Energy – Revenue |  First-fit Heuristic |  SLA |  Hard SLA and soft SLA.
Sonnek et al., [112] | cloud provider |  migration |  Traffic |  Heuristics |  Inter-VM dependency – Colocation |  affinity; satisfy it as much as possible
Sotiriadis et al., [113] | user – multi-cloud |  migration |  other/unspecified |  Migration tool |  location – other/unspecified |  migration of VMs in FIWARE systems
Sunil Rao and Santhi Thilagam, [115] |  |  migration |  min. active PMs – Resource wastage |  Heuristics |  Sharing |  not implemented/discussion only.
Tantawi, [117] | user – cloud provider |  placement |  Load Balancing |  Heuristics/random search (biased statistical sampling) |  Location – Colocation |  location constraints can be hard or soft.
Unuvar et al., [122] | user – multi-cloud |  placement |  Traffic load Balancing Cost |  Heuristics - Biased Importance Sampling |  Resource – Cost – other/unspecified |  Introduced a hybrid cloud placement algorithm that incorporates two policy parameter: reservation and cost weight and addresses the trade-off between deployment cost and request rejection rate.
Viswanathan et al., [125] | cloud provider |  placement |  Energy – Completion time |  Brute-force + set partitioning algorithm |  Objectives |  Addresses HPC applications placement in DC.
Wright et al., [130] | user |  placement |  other/unspecified |  Heuristics |  other/unspecified |  user perspective. measured/evaluated in cost
Zhou et al., [142] | cloud provider |  migration |  Cost – Migration cost |  Heuristics |  other/unspecified |  Incorporates user experience (IO penalty), cluster management (migration time) and device usage (degree of wear). Utilizes a weighted cost function in which the weights on each metric reflects different design preferences.

###  3.6 Discussion

It is clear from the five classes of approaches mentioned above that there
exists different ways to incorporate preferences within the problem with
different goals as some methods try to reduce the search space (e.g., bounds
on objectives [53], preference order on the assignment [111]) while others
generate a single solution or an ordered list of solutions (e.g., most of
matching algorithms). Metaheuristics are viable option in tackling complex
domains with conflicting objectives. For instance, when reducing the energy
while maximizing the revenue during virtualization. However, they are prone to
returning suboptimal solutions and require good tradeoff between exploration
and exploitation that, in practice, is hard to construct efficiently. In case
the preferences are user-centric, i.e., they represent provider or user, there
is a need for interactive metaheuristics approaches that consult the user
during the search process [77].

Indeed, the heuristic approaches require domain knowledge to construct an
effective heuristic function with which promising search space is discovered
first. Such domain knowledge is sometimes prohibitive when multiple
stakeholders’ preferences are involved. One needs to assume preferences follow
certain structure and often construct a heuristic function for a simplified
version of the original problem. Hence, it is obvious that the proposed
heuristic approaches are general and usually domain independent [12].

Most of the matching methods consider resource as the attribute to base their
strategy on. This is a reasonable consideration as physical machines have
limited resources and utilizing them efficiently when looking for a stable
matching between VMs and PMs is of great importance. Stability of the match
guarantee the average happiness among all participants, i.e., that there
exists no pair $(X,Y)$ of virtual machine and physical machine that prefer to
be with each other than their current partner (e.g., the physical machine
hosting $X$, and the VMs placed on $Y$).

MCDM approaches such as AHP and TOPSIS are known to require large number of
pairwise comparisons and do not show attribute interdependencies. They are
also time consuming and have scalability issues when the number of attributes
is large. In such case, preferential dependencies among attributes can lower
the time needed to assess preference between alternatives [63].

##  4 Benefits, Research Challenges, and Trends

###  4.1 Potential Benefits

After the detailed review of preference applications, here we draw insights
regarding the benefits of incorporating preferences into the VMP problem.

User Satisfaction

Indeed one of the main benefits of considering preferences is to find
solutions that are not only optimized but also preferred by the user or
decision maker. Apart from setting firm and solid requirements, a customer or
a cloud administrator can express additional soft requirements through
preferences. Considering such preferences would increase the satisfaction of
both parties and usually a graphical interface is utilized to specify the
preferences [113, 121, 120].

Reducing the Complexity

VMP is a complex problem given all the possible conflicting objectives,
constraints from multiple entities, functional and non-functional properties,
random nature of workload changes, and managerial and geographical policies.
It is remarkably noticed in the literature that considering preferences not
only handles users’ and provider’s fine details but also introduces some sort
of flexibility to the problem. In addition, the problem of optimizing VMP is
known to be NP-complete and even sub-optimal solutions needs to be obtained in
a timely manner. Preferences can effectively not only provide but also be the
tool to control the margin of flexibility considering performance, efficiency,
solution time [101], and violation rate [28].

For example, Ye et al., [138] reported in their study that the number of PMs
required to host VMs with colocation constraints decreases when the
constraints are softened. In addition, the gain of incorporating preferences
improves the scalability to large-scale problems as in [68].

Moreover, preference information can be utilized to set criteria for managing
complex processes. As an example, consider the dynamic PM consolidation
process. Typically, there exists a trade-off between the goodness of the
process and completion time due the complexity of the process [85].
Preferences can be utilized to resolve this trade-off in an efficient manner.
In addition, cloud providers typically offer predefined instances of virtual
resources with different capabilities as well as value-added features and
services (e.g., choice of OS, traffic on a private VLAN, etc). The customer
choices impact the pricing of such leasing. Wright et al., [130] defined
application soft and hard constraints to reduce the complexity of choosing
suitable deployment.

Besides, there are cases that inspire to use predefined soft constraints. Hao
et al., [51] considered bounded constraint violation to maintain and fix the
feasibility of the LP problem under study. The selection of constraints and
the extent to which they are softened is critical to the quality of the
solution and should be set based on administrator’s preferences rather than
arbitrarily.

Reducing Search Space

Multiobjective problems typically obtain multiple Pareto solutions, and the
number of these solutions grows significantly as the number of objectives
grows. Thus, preferences have been utilized to help mitigate this burden [39,
54, 75, 103, 13, 77]. The same can be said about other algorithms (e.g.,
alternative based algorithms). For example, the work in [120] considered a two
phase approach and applied non-functional preferences in the second phase to
reduce the number of candidate PMs.

Another example, there can be the case where two or more solutions are
available (e.g., ties in preference list [90]), and favoring such an option
might have some advantages that the DM was not aware of.

Adaptation of Specialized Services

Preferences and soft constraints can be utilized within VMP schemes to enable
the hosting and QoS guarantees of specialized services (e.g., mapping special
services to a PM or a set of PMs). Security concerns [141], NFV chaining [89],
HPC placement [125], mobile applications [140], smart TV services [66], and
Web services [82] are examples of VMP for specialized services that
incorporate the notion of preferences in their approaches.

Improving Performance

Preferences can be used as a guideline for optimizing the performance of a
system. Such preferences are usually inspired from meta-analysis of the
problem and identified as preferred settings or configurations to the machine.
The work in [124] for instance, provides a detailed experimental study and
identifies the parameters that influence the cost of migrating a VM and
provides some recommended rules to improve VM migration efficiency. We expect
that these rules (e.g., migrate applications with small active memory first,
migrate low priority applications first) can suitably be identified as
preferences rather than strict constraints or objectives.

Furthermore, constraints and objectives are considered known, hard and strict.
preferences are soft, and can be given on top of the problem in a way that
does not impact constraints and objectives. However, preferences can be known
in advance, known a posteriori, or learned throughout the system’s life. The
later can be useful in the VMP context as the system preferences can be
learned and adapted to the operational status of the cloud [21]. This gives a
wider perspective to administrators on getting to know how different unknown
options arise from the problem and to what extent they can change the output
[21, 87]

###  4.2 Trends in the cloud

Software-defined Networking (SDN) and Network Function Virtualization (NFV)
are two of the key technologies in the cloud and 5G networks. The SDN
architecture provides flexible and programmable network control by decoupling
the control plane from the data plane. This enables the use of simple network
devices (i.e., switches) to create and manage networks for different purposes
on the fly via a logically centralized SDN controller.

NFV enables deploying network functions (e.g., routing, firewalling, load
balancing, etc.) as software applications (i.e., VM) stored and run on generic
servers (i.e., PM). It also introduces flexibility in terms of location, in
which functions can be placed anywhere in the network satisfying users and
provider requirements. However for a specific task that requires more than
function, there is an order sequence that must be followed [71].

The advent of SDN/NFV technologies has facilitated the _cloudification_ of
many communication systems [50, 84, 36, 71]. Here we shed light on a number of
trends that leverage virtualization and cloud computing technologies in order
to improve overall system performance.

Internet of Spectrum Devices

aims to expand the radio spectrum of wireless mobile networks in which
devices/elements of the network (e.g., base stations) have no restriction on
spectrum use and can utilize multiple shared resources of the spectrum
dynamically. A cloud is responsible for gathering spectrum status/info. from
monitoring devices and distributing them to spectrum utilizing devices. There
is a virtual DM for each entity in the spectrum cloud that is responsible for
making spectrum decisions. Various cloud computing operations can be applied
based on DM request and users’ preferences [132].

Cloud-Radio Access Networks (C-RANs)

offer cloud-based architecture in which all control algorithms are placed and
executed by data centers [131].

Specifically, in a C-RAN, traditional base stations functionalities are
decoupled into radio functions that are stored in the remote radio unit (RRU)
and digital and higher layer functions that are stored in the baseband unit
(BBU). The RRU remains at the base station site, while the BBU is moved to the
cloud. C-RANs extend the mobile radio access capacity and coverage by pooling
BBU resources from multiple cells. These cells can be heterogenous and may
belong to multiple collaborating and competing carriers. Following NFV
standards, each BBU functionality is represented by one or more VMs which can
be enabled or disabled for each base station. Further, these VMs have to be
placed in PMs that satisfy their requirements and should be dynamically
migrated to where they are needed. The sequence and order of execution of
network functions as well as the colocation requirements have to be taken into
account when making a placement decision.

Content delivery and Caching

In modern ISP networks, content (videos and cached pages) can be stored in
VMs, and the CDN is configured adaptively by SDN controllers to account for
traffic loads on the ISP network [129].

Small Cells

One of the challenges to small cell technologies (e.g., WiFi and Femtocells)
is high-level of interference between cells due to the lack of channel
coordination. Cloud-based control is a promising method to manage channel
assignment between different, multi-vendor, small cells. There are many other
technologies that rely on cloud computing concepts such as mobile cloud, edge
Computing, cognitive IoT, etc. In general, cloud-based architectures can
provide more flexibility and centralized control and management of the
participating entities/devices, in which, coordination and performance can be
optimized. Furthermore, proper understanding and translation of the
requirements of each VM (e.g., application) will lead to desired placement
decisions.

VMP with Preferences on the Edge

The problem of solving VMP with preferences on the edge brings several
challenges on top of the VMP challenges. One needs compact models to represent
and reason about preferences. Such models need to be compact and succinct in
their preference language to handle the possibly large number of attributes
and limited resources on the edge. Even in the case of successfully stored,
preferences on the edge may be insecure for the genuine user’s preferences and
selection criteria and, therefore, privacy-preserving techniques for storing
and dealing with preferences on the edge are required. Recently, Ouyang et
al., [88] proposed an adaptive algorithm for service placement on the edge,
which jointly optimizes latency and migration cost, weighted by the on-line
learned user’s preferences. Qian et al., [95] developed an privacy-aware
algorithm for service placement in edge cloud, and adopting a user preference
model to learn about user preferences toward services. In addition, Chiti et
al., [27] tackled the problem of virtual function placement in fog-cloud by
applying matching game algorithm to fog nodes and virtual functions.

Automation of Cloud Operations

The scale and the frequent changes of cloud data center networks and
configurations necessitate faster handling of cloud operations (i.e., service
management, deployment, and configuration). This includes the provisioning of
resources and instantiation, configuration, and delivery of services at
acceptable service level and without human intervention. Thus, orchestration
and automation of cloud services have been of high interest to both academia
and industry which gave rise to new concepts addressing these concerns. For
instance, Zero-Touch networking is an emerging concept that leverages SDN and
NFV capabilities and aims to automate cloud-based/virtual networks management
and operation. A related emerging concept is intent-driven networking. It is
an end-to-end network configuration automation solution that aims to optimize
performance and user experience. It leverages big data and AI technologies to
achieve such a goal. Preferences and non-functional parameters can be a
helpful tool in the design of automated control algorithms. [106]

###  4.3 Limitations and Research Opportunities

Compactness and Cognitive Burden

As the number of alternatives becomes large, direct assessment of preferences
becomes problematic. VMP indeed has huge space of possible alternatives and
directly evaluating them would be a demanding cognitive task. For example,
consider a customer expressing her preferences for the placement over 10 PMs.
She needs to express her preferences in ordinal way through pairwise
comparisons of every pair of PMs. This is indeed problematic as the users
would lose focus let alone being cognitively able to accurately specify their
preferences.

The vast majority of VMP approaches assume the existence of a utility
function. Accurately specifying utility functions by end-users has been shown
to be a difficult and tedious task [46]. In many situations, users cannot
afford to specify more than qualitative statements of the form “$x$ is better
than $y$". The literature lacks concrete attempts that address user’s policies
and preferences qualitatively. Furthermore, it is usually assumed that a
utility function of one attribute is independent from other attributes. This
assumption is too strong to be met in practice as the utility of one attribute
may depend on other attributes’ values. Generalized Additive Independence
Networks (GAI-net) [48] weaken this assumption by allowing the utility
function of one attribute to depend on other attributes. On the qualitative
side, Conditional Preference Networks (CP-nets) [23] are a prominent graphical
model to represent conditional preferences.

Such compact models have been proposed in the last decade to avoid the
cognitive burden of specifying preferences in multiattribute domains. They
require reasonable effort from the user and adopt the conditional preferential
independency notion. This allows for the decomposition of the preferences into
smaller but compact components while being expressive enough to capture many
scenarios in practice [64, 16].

Uncertainty, Vagueness, and Risk

Recently, there have been several works on dynamic and uncertain VMP [78, 24,
53, 80] where workloads and operational statuses may change frequently over
time. However, it is important to tackle the problem from the decision maker
perspective and consider preferences in uncertain situations. Current
approaches usually assume preferences occur with certainty where, in practice,
preferences occur with uncertainty over the set of possible alternatives. This
is evident in VMP as customers are unaware of possible complexities on the
cloud. In such cases, preferences are no longer deterministic choices but
rather a _lottery_ over the alternatives. A lottery is a probability
distribution over alternatives [64]. Preferences are then represented over
lotteries and known concepts such as expected utility can be used to rank the
alternatives.

Learning and Revision

Machine learning has been recognized as an enabler for building and designing
cloud data centers [73, 70, 107]. The VMP problem is dynamic in nature and
preferences often change over time. This urges for approaches to not only
handle uncertain preferences but also learn to revise them in light of new
information. Instead of insisting on having the preferences upfront, it is
better to elicit them during the solving process. Such elicitation holds the
promise of proposing interactive learning strategies that consider humans in
the loop when solving the VMP; thus, lowering the cognitive burden in
specifying preferences. Another approach is to learn customers’ and
administrators’ preferences from historical data (i.e., learning preferences
passively from past deployments).

Group Decision Making

The VMP problem is inherently a group decision making problem. A group of
customers and cloud administrators issue their preferences on different
aspects of the problem. It is likely that the issued preferences are
conflicting with each other and further aggregation and reconciliation methods
are required. For instance, consider the scenario of Ilkhechi et al., [55]
where each group of VMs relies heavily on some PMs (named sinks). These sinks
(e.g., supercomputers) are a subset of the PMs and it is likely that
customers’ preferences would be conflicting on how to utilize them. Thus, the
aggregation of preferences needs to ensure certain properties like _fairness_
of the deployment among customers.

##  5 Conclusions

Virtualization technology has enabled cloud data centers to become cost
effective, flexible and customizable to host diverse enterprise services and
applications at different scales. The VMP is a problem of significant
importance to the cloud design for sustaining its profit and popularity. The
literature shows a wide variety of approaches to solve the problem and its
variants. The problem is also full of preferences of the involved parties
including cloud administrators and customers. Therefore, the proper handling
of preferences is crucial to the success of proposed approaches.

In this paper, we discussed the current approaches for handling preferences in
the context of VMP. We highlighted certain limitations and identified research
opportunities. Analyzing the VMP problem from a decision making point of view
gives rise to several interesting research challenges. Adopting compact models
of preferences such as CP-nets and GAI-net is particularly promising given the
large number of attributes involved in typical VMP problems. This has the
potential of reducing the cognitive overhead of the problem; thus, increase
its applicability. Another important research to be explored is interactively
learning the constraints and preferences of the cloud admin and/or customers
as well as passively learning the soft constraints among VMs and PMs from
historical data. Lastly, proposals based on group decision making techniques
hold the promise of solving the problem taking everyone’s preferences in a
fair and transparent manner. Addressing such issues may lead to the
development of user-centric VMP that is widely adopted by the stakeholders.

## References

  * [1] Amazon ec2 instance types.  https://aws.amazon.com/ec2/instance-types/.  Accessed: 2019-12-11.
  * [2] Introducing proximity placement groups.  https://azure.microsoft.com/en-us/blog/introducing-proximity-placement-groups/.  Accessed: 2019-12-11.
  * [3] Machine types.  https://cloud.google.com/compute/docs/machine-types.  Accessed: 2019-12-09.
  * [4] Phoenixnap.  https://phoenixnap.com.  Accessed: 2019-12-12.
  * [5] Placement policies for service fabric services.  https://docs.microsoft.com/en-us/azure/service-fabric/service-fabric-cluster-resource-manager-advanced-placement
-rules-placement-policies.  Accessed: 2019-12-11.
  * [6] Policy-based optimization.  Technical report.
  * [7] Solving the ha challenge: Placement groups for virtual servers.  https://www.ibm.com/cloud/blog/announcements/ha-challenge-placement-groups-virtual-servers.  Accessed: 2019-12-10.
  * ora, [2011] (2011).  Oracle optimized solution for enterprise cloud infrastructure.  Technical report.
  * vmw, [2012] (2012).  Cloud infrastructure architecture case study.  Technical report.
  * aws, [2019] (2019).  Carrier-grade mobile packet core network on aws.  Technical report.
  * vmw, [2019] (2019).  Performance of vsphere 6.7 scheduling options.  Technical report.
  * Addya et al., [2017] Addya, S. K., Turuk, A. K., Sahoo, B., Sarkar, M., and Biswash, S. K. (2017).  Simulated annealing based vm placement strategy to maximize the profit for cloud service providers.  Engineering Science and Technology, an International Journal, 20(4):1249 – 1259.
  * Alashaikh and Alanazi, [2019] Alashaikh, A. S. and Alanazi, E. A. (2019).  Incorporating ceteris paribus preferences in multiobjective virtual machine placement.  IEEE Access, 7:59984–59998.
  * Amato and Venticinque, [2013] Amato, A. and Venticinque, S. (2013).  Multi-objective decision support for brokering of cloud SLA.  Proceedings - 27th International Conference on Advanced Information Networking and Applications Workshops, WAINA 2013, pages 1241–1246.
  * Amazon Web Services, [2018] Amazon Web Services, I. (2018).  Architecting for the Cloud.  Technical Report November.
  * Amor et al., [2016] Amor, N. B., Dubois, D., Gouider, H., and Prade, H. (2016).  Graphical models for preference representation: An overview.  In International Conference on Scalable Uncertainty Management, pages 96–111. Springer.
  * Araujo et al., [2018] Araujo, J., Maciel, P., Andrade, E., Callou, G., Alves, V., and Cunha, P. (2018).  Decision making in cloud environments: an approach based on multiple-criteria decision analysis and stochastic models.  Journal of Cloud Computing, 7(1).
  * Arianyan et al., [2016] Arianyan, E., Taheri, H., and Sharifian, S. (2016).  Novel heuristics for consolidation of virtual machines in cloud data centers using multi-criteria resource management solutions.  Journal of Supercomputing, 72(2):688–717.
  * Attaoui and Sabir, [2018] Attaoui, W. and Sabir, E. (2018).  Multi-criteria virtual machine placement in cloud computing environments: A literature review.  CoRR.
  * Azougaghe et al., [2017] Azougaghe, A., Oualhaj, O. A., Hedabou, M., Belkasmi, M., and Kobbane, A. (2017).  Many-to-one matching game towards secure virtual machines migration in cloud computing.  2016 International Conference on Advanced Communication Systems and Information Security, ACOSIS 2016 - Proceedings, pages 1–7.
  * Battiti and Passerini, [2010] Battiti, R. and Passerini, A. (2010).  Brain-computer evolutionary multiobjective optimization: A genetic algorithm adapting to the decision maker.  IEEE Transactions on Evolutionary Computation, 14(5):671–687.
  * Blagodurov et al., [2015] Blagodurov, S., Fedorova, A., Vinnik, E., Dwyer, T., and Hermenier, F. (2015).  Multi-objective job placement in clusters.  In SC ’15: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–12.
  * Boutilier et al., [2004] Boutilier, C., Brafman, R. I., Domshlak, C., Hoos, H. H., and Poole, D. (2004).  Cp-nets: A tool for representing and reasoning with conditional ceteris paribus preference statements.  J. Artif. Intell. Res. (JAIR), 21:135–191.
  * Chamas et al., [2017] Chamas, N., López-Pires, F., and Baran, B. (2017).  Two-phase virtual machine placement algorithms for cloud computing: An experimental evaluation under uncertainty.  In 2017 XLIII Latin American Computer Conference (CLEI), pages 1–10. IEEE.
  * Chen et al., [2010] Chen, H., Jin, H., and Hu, K. (2010).  Affinity-aware proportional share scheduling for virtual machine system.  Proceedings - 9th International Conference on Grid and Cloud Computing, GCC 2010, pages 75–80.
  * Chen et al., [2017] Chen, J., He, Q., Ye, D., Chen, W., Xiang, Y., and Chiew, K. (2017).  Joint affinity aware grouping and virtual machine placement.  Microprocessors and Microsystems, 52:365–380.
  * Chiti et al., [2019] Chiti, F., Fantacci, R., Paganelli, F., and Picano, B. (2019).  Virtual Functions Placement With Time Constraints in Fog Computing: A Matching Theory Perspective.  IEEE Transactions on Network and Service Management, 16(3):980–989.
  * Chowdhury et al., [2015] Chowdhury, M. R., Mahmud, M. R., and Rahman, R. M. (2015).  Implementation and performance analysis of various VM placement strategies in CloudSim.  Journal of Cloud Computing, 4(1):20.
  * Chu et al., [2017] Chu, Q., Cui, L., and Zhang, Y. (2017).  Joint Computing and Storage Resource Allocation Based on Stable Matching in Data Centers.  Proceedings - 3rd IEEE International Conference on Big Data Security on Cloud, BigDataSecurity 2017, 3rd IEEE International Conference on High Performance and Smart Computing, HPSC 2017 and 2nd IEEE International Conference on Intelligent Data and Security, IDS 2017, pages 207–212.
  * Clímaco et al., [2016] Clímaco, J., Craveirinha, J., and Girão-Silva, R. (2016).  Multicriteria Analysis in Telecommunication Network Planning and Design: A Survey.  In Greco, S., Ehrgott, M., and Figueira, J. R., editors, Multiple Criteria Decision Analysis, volume 233 of International Series in Operations Research & Management Science, pages 1167–1233. Springer New York, New York, NY.
  * Clímaco et al., [2019] Clímaco, J., Craveirinha, J., and Martins, L. (2019).  Comparison of Routing Methods in Telecommunication Networks—An Overview and a New Proposal Using aMulti-criteria Approach Dealing with Imprecise Information.  In Doumpos, M., Figueira, J. R., Greco, S., and Zopounidis, C., editors, New Perspectives in Multiple Criteria Decision Making, Multiple Criteria Decision Making, pages 397–427. Springer International Publishing, Cham.
  * Coello et al., [2007] Coello, C. A. C., Lamont, G. B., Van Veldhuizen, D. A., et al. (2007).  Evolutionary algorithms for solving multi-objective problems, volume 5.  Springer.
  * Colman-meixner et al., [2016] Colman-meixner, C., Develder, C., Member, S., Tornatore, M., Member, S., and Mukherjee, B. (2016).  A Survey on Resiliency Techniques in Cloud Computing Infrastructures and Applications.  IEEE Communications Surveys & Tutorials, 18(3):2244–2281.
  * Cui et al., [2016] Cui, L., Cziva, R., Tso, F. P., and Pezaros, D. P. (2016).  Synergistic policy and virtual machine consolidation in cloud data centers.  Proceedings - IEEE INFOCOM, 2016-July(May):10–15.
  * Cui and Tso, [2015] Cui, L. and Tso, F. P. (2015).  Joint virtual machine and network policy consolidation in cloud data centers.  2015 IEEE 4th International Conference on Cloud Networking, CloudNet 2015, pages 153–158.
  * Dai et al., [2017] Dai, B., Xu, G., Huang, B., Qin, P., and Xu, Y. (2017).  Enabling network innovation in data center networks with software defined networking: A survey.  Journal of Network and Computer Applications, 94:33–49.
  * Deb and Sundar, [2006] Deb, K. and Sundar, J. (2006).  Reference point based multi-objective optimization using evolutionary algorithms.  In Proceedings of the 8th annual conference on Genetic and evolutionary computation, pages 635–642. ACM.
  * Decouchon-Brochet, [2004] Decouchon-Brochet, C. (2004).  Case study 2: study of the extension of a telecommunication network.  In Collette, Y. and Siarry, P., editors, Multiobjective Optimization Principles and Case Studies, Decision Engineering, chapter 11, pages 237–248. Springer Berlin Heidelberg, Berlin, Heidelberg.
  * Deng et al., [2014] Deng, W., Liu, F., Jin, H., Liao, X., and Liu, H. (2014).  Reliability-aware server consolidation for balancing energy-lifetime tradeoff in virtualized cloud datacenters.  International Journal of Communication Systems, 27(4):623–642.
  * Deshpande and Keahey, [2017] Deshpande, U. and Keahey, K. (2017).  Traffic-sensitive Live Migration of Virtual Machines.  Future Generation Computer Systems, 72:118–128.
  * Dhillon et al., [2013] Dhillon, J. S., Purini, S., and Kashyap, S. (2013).  Virtual machine coscheduling: A game theoretic approach.  In Proceedings - 2013 IEEE/ACM 6th International Conference on Utility and Cloud Computing, UCC 2013, pages 227–234. IEEE.
  * Dow, [2016] Dow, E. M. (2016).  Decomposed multi-objective bin-packing for virtual machine consolidation.  PeerJ Computer Science, 2:e47.
  * Espling et al., [2016] Espling, D., Larsson, L., Li, W., Tordsson, J., and Elmroth, E. (2016).  Modeling and Placement of Cloud Services with Internal Structure.  IEEE Transactions on Cloud Computing, 4(4):429–439.
  * Feng et al., [2019] Feng, D., Wu, Z., Zuo, D., and Zhang, Z. (2019).  A multiobjective migration algorithm as a resource consolidation strategy in cloud computing.  PLOS ONE, 14(2):e0211729.
  * Fishburn, [1970] Fishburn, P. C. (1970).  Utility theory for decision making.  Technical report, Research analysis corp McLean VA.
  * French, [1986] French, S., editor (1986).  Decision Theory: An Introduction to the Mathematics of Rationality.  Halsted Press, New York, NY, USA.
  * Georgiou et al., [2013] Georgiou, S., Tsakalozos, K., and Delis, A. (2013).  Exploiting network-topology awareness for vm placement in iaas clouds.  In 2013 International Conference on Cloud and Green Computing, pages 151–158.
  * Gonzales and Perny, [2004] Gonzales, C. and Perny, P. (2004).  Gai networks for utility elicitation.  KR, 4:224–234.
  * Grozev and Buyya, [2014] Grozev, N. and Buyya, R. (2014).  Inter-Cloud architectures and application brokering: Taxonomy and survey.  Software - Practice and Experience, 44(3):369–390.
  * Han et al., [2015] Han, B., Gopalakrishnan, V., Ji, L., and Lee, S. (2015).  Network function virtualization: Challenges and opportunities for innovations.  IEEE Communications Magazine, 53(2):90–97.
  * Hao et al., [2017] Hao, F., Kodialam, M., Lakshman, T. V., and Mukherjee, S. (2017).  Online allocation of virtual machines in a distributed cloud.  IEEE/ACM Transactions on Networking, 25(1):238–249.
  * He et al., [2011] He, S., Guo, L., and Guo, Y. (2011).  Real time elastic cloud management for limited resources.  In 2011 IEEE 4th International Conference on Cloud Computing, pages 622–629.
  * [53] Ihara, D., López-Pires, F., and Baran, B. (2015a).  Many-objective virtual machine placement for dynamic environments.  In 2015 IEEE/ACM 8th International Conference on Utility and Cloud Computing (UCC), pages 75–79. IEEE.
  * [54] Ihara, D., Pirez, F. L., and Barán, B. (2015b).  Many-Objective Virtual Machine Placement for Dynamic Environments.  Proceedings - 2015 IEEE/ACM 8th International Conference on Utility and Cloud Computing, UCC 2015, (September 2017):75–79.
  * Ilkhechi et al., [2015] Ilkhechi, A. R., Korpeoglu, I., and Ulusoy, Ö. (2015).  Network-aware virtual machine placement in cloud data centers with multiple traffic-intensive components.  Computer Networks, 91:508–527.
  * Jain et al., [2018] Jain, S., Purini, S., and Reddy, P. V. (2018).  A multi-cloud marketplace model with multiple brokers for iaas layer and generalized stable matching.  2018 IEEE/ACM 11th International Conference on Utility and Cloud Computing (UCC), pages 257–266.
  * Jalili et al., [2019] Jalili, A., Keshtgari, M., Akbari, R., and Javidan, R. (2019).  Multi criteria analysis of Controller Placement Problem in Software Defined Networks.  Computer Communications, 133(August 2018):115–128.
  * Jamshidi et al., [2017] Jamshidi, P., Pahl, C., and Mendonça, N. C. (2017).  Pattern-based multi-cloud architecture migration.  Software - Practice and Experience, 47(9):1159–1184.
  * Jayasinghe et al., [2011] Jayasinghe, D., Pu, C., Eilam, T., Steinder, M., Whally, I., and Snible, E. (2011).  Improving performance and availability of services hosted on iaas clouds with structural constraint-aware virtual machine placement.  In 2011 IEEE International Conference on Services Computing, pages 72–79. IEEE.
  * Johnson et al., [1989] Johnson, D. S., Aragon, C. R., McGeoch, L. A., and Schevon, C. (1989).  Optimization by simulated annealing: An experimental evaluation; part i, graph partitioning.  Operations research, 37(6):865–892.
  * Jung et al., [2010] Jung, G., Joshi, K. R., Hiltunen, M. A., Schlichting, R. D., and Pu, C. (2010).  Performance and availability aware regeneration for cloud based multitier applications.  In 2010 IEEE/IFIP International Conference on Dependable Systems & Networks (DSN), pages 497–506. IEEE.
  * Kabir et al., [2015] Kabir, M. H., Shoja, G. C., and Ganti, S. (2015).  VM placement algorithms for hierarchical cloud infrastructure.  Proceedings of the International Conference on Cloud Computing Technology and Science, CloudCom, 2015-February(February):656–659.
  * Keeney and Raiffa, [1976] Keeney, R. and Raiffa, H. (1976).  Decisions with multiple objectives: Preferences and value tradeoffs.  J. Wiley, New York.
  * Keeney and Raiffa, [1993] Keeney, R. L. and Raiffa, H. (1993).  Decisions with multiple objectives: preferences and value trade-offs.  Cambridge university press.
  * Kella and Belalem, [2014] Kella, A. and Belalem, G. (2014).  VM live migration algorithm based on stable matching model to improve energy consumption and quality of service.  CLOSER 2014 - Proceedings of the 4th International Conference on Cloud Computing and Services Science, pages 118–128.
  * Kim and Lee, [2014] Kim, G. and Lee, W. (2014).  Stable matching with ties for cloud-assisted smart TV services.  Digest of Technical Papers - IEEE International Conference on Consumer Electronics, pages 558–559.
  * Konstantinidis et al., [2010] Konstantinidis, A., Yang, K., Zhang, Q., and Zeinalipour-Yazti, D. (2010).  A multi-objective evolutionary algorithm for the deployment and power assignment problem in wireless sensor networks.  Computer Networks, 54(6):960–976.
  * Korasidis et al., [2017] Korasidis, N., Giannakopoulos, I., Doka, K., Tsoumakos, D., and Koziris, N. (2017).  Fair, fast and frugal large-scale matchmaking for vm placement.  In Sellis, T. and Oikonomou, K., editors, Algorithmic Aspects of Cloud Computing, pages 131–145, Cham. Springer International Publishing.
  * Korupolu et al., [2009] Korupolu, M., Singh, A., and Bamba, B. (2009).  Coupled placement in modern data centers.  IPDPS 2009 - Proceedings of the 2009 IEEE International Parallel and Distributed Processing Symposium.
  * Kundu et al., [2012] Kundu, S., Rangaswami, R., Gulati, A., Zhao, M., and Dutta, K. (2012).  Modeling virtualized applications using machine learning techniques.  In ACM Sigplan Notices, volume 47, pages 3–14. ACM.
  * Laghrissi, [2019] Laghrissi, A. (2019).  A Survey on the Placement of Virtual Resources and Virtual Network Functions.  IEEE Communications Surveys & Tutorials, 21(2):1409–1434.
  * Li et al., [2017] Li, W., Wang, Y., Wang, Y., Xia, Y., Luo, X., and Wu, Q. (2017).  An Energy-Aware and Under-SLA-Constraints VM Consolidation Strategy Based on the Optimal Matching Method.  Int. J. Web Serv. Res., 14(4):75–89.
  * Li et al., [2013] Li, X., Qian, Z., Lu, S., and Wu, J. (2013).  Energy efficient virtual machine placement algorithm with balanced and improved resource utilization in a data center.  Mathematical and Computer Modelling, 58(5-6):1222–1235.
  * Li et al., [2016] Li, X., Ventresque, A., Murphy, J., and Thorburn, J. (2016).  SOC: Satisfaction-Oriented Virtual Machine Consolidation in Enterprise Data Centers.  International Journal of Parallel Programming, 44(1):130–150.
  * Liu et al., [2018] Liu, X. F., Zhan, Z. H., Deng, J. D., Li, Y., Gu, T., and Zhang, J. (2018).  An Energy Efficient Ant Colony System for Virtual Machine Placement in Cloud Computing.  IEEE Transactions on Evolutionary Computation, 22(1):113–128.
  * Liu et al., [2017] Liu, X. F., Zhan, Z. H., and Zhang, J. (2017).  An energy aware unified ant colony system for dynamic virtual machine placement in cloud computing.  Energies, 10(5).
  * López-Pires and Barán, [2017] López-Pires, F. and Barán, B. (2017).  Many-Objective Virtual Machine Placement.  Journal of Grid Computing, 15(2):161–176.
  * López-Pires et al., [2018] López-Pires, F., Barán, B., Benítez, L., Zalimben, S., and Amarilla, A. (2018).  Virtual machine placement for elastic infrastructures in overbooked cloud computing datacenters under uncertainty.  Future Generation Computer Systems, 79:830–848.
  * Mahalingam and Sengottaiyan, [2015] Mahalingam, S. K. and Sengottaiyan, N. (2015).  A QoS Guaranteed Selection of Efficient Cloud Services.  Indian Journal of Science and Technology, 8(S9):103.
  * Mann, [2015] Mann, Z. Á. (2015).  Allocation of Virtual Machines in Cloud Data Centers—A Survey of Problem Models and Optimization Algorithms.  ACM Computing Surveys, 48(1):1–34.
  * Masdari et al., [2016] Masdari, M., Nabavi, S. S., and Ahmadi, V. (2016).  An overview of virtual machine placement schemes in cloud computing.  J. Netw. Comput. Appl., 66(C):106–127.
  * Menzel and Ranjan, [2012] Menzel, M. and Ranjan, R. (2012).  Cloudgenius: Decision support for web server cloud migration.  CoRR, abs/1203.3997.
  * Mezni and Abdeljaoued, [2018] Mezni, H. and Abdeljaoued, T. (2018).  A cloud services recommendation system based on Fuzzy Formal Concept Analysis.  Data and Knowledge Engineering, 116(May):100–123.
  * Mijumbi et al., [2016] Mijumbi, R., Serrat, J., Gorricho, J.-L., Bouten, N., De Turck, F., and Boutaba, R. (2016).  Network Function Virtualization: State-of-the-Art and Research Challenges.  IEEE Communications Surveys & Tutorials, 18(1):236–262.
  * Mohamadi Bahram Abadi et al., [2018] Mohamadi Bahram Abadi, R., Rahmani, A. M., and Alizadeh, S. H. (2018).  Server consolidation techniques in virtualized data centers of cloud environments: A systematic literature review.  Software - Practice and Experience, 48(9):1688–1726.
  * Murugeswari et al., [2016] Murugeswari, R., Radhakrishnan, S., and Devaraj, D. (2016).  A multi-objective evolutionary algorithm based QoS routing in wireless mesh networks.  Applied Soft Computing Journal, 40:517–525.
  * Ooi et al., [2012] Ooi, B. Y., Chan, H. Y., and Cheah, Y. N. (2012).  Dynamic service placement and replication framework to enhance service availability using team formation algorithm.  Journal of Systems and Software, 85(9):2048–2062.
  * Ouyang et al., [2019] Ouyang, T., Li, R., Chen, X., Zhou, Z., and Tang, X. (2019).  Adaptive User-managed Service Placement for Mobile Edge Computing: An Online Learning Approach.  In IEEE INFOCOM 2019 - IEEE Conference on Computer Communications, pages 1468–1476. IEEE.
  * Pham et al., [2017] Pham, C., Tran, N. H., Ren, S., Saad, W., and Hong, C. S. (2017).  Traffic-aware and Energy-efficient vNF Placement for Service Chaining: Joint Sampling and Matching Approach.  IEEE Transactions on Services Computing, pages 1–1.
  * Phuke et al., [2018] Phuke, N., Gharote, M., Patil, R., and Lodha, S. (2018).  Multi-objective stable matching with ties.  IEEE International Conference on Industrial Engineering and Engineering Management, 2017-December:964–968.
  * Pietri and Sakellariou, [2016] Pietri, I. and Sakellariou, R. (2016).  Mapping virtual machines onto physical machines in cloud computing: A survey.  ACM Comput. Surv., 49(3):49:1–49:30.
  * Pillai and Rao, [2016] Pillai, P. S. and Rao, S. (2016).  Resource Allocation in Cloud Computing Using the Uncertainty Principle of Game Theory.  IEEE Systems Journal, 10(2):637–648.
  * Pires and Barán, [2015] Pires, F. L. and Barán, B. (2015).  Virtual machine placement literature review.  CoRR, abs/1506.01509.
  * Qi et al., [2014] Qi, H., Shiraz, M., Liu, J.-y., Gani, A., Abdul Rahman, Z., and Altameem, T. A. (2014).  Data center network architecture in cloud computing: review, taxonomy, and open research issues.  Journal of Zhejiang University SCIENCE C, 15(9):776–793.
  * Qian et al., [2019] Qian, Y., Hu, L., Chen, J., Guan, X., Hassan, M. M., and Alelaiwi, A. (2019).  Privacy-aware service placement for mobile edge computing via federated learning.  Information Sciences, 505:562–570.
  * Rachmawati and Srinivasan, [2006] Rachmawati, L. and Srinivasan, D. (2006).  Preference Incorporation in Multi-objective Evolutionary Algorithms: A Survey.  2006 IEEE International Conference on Evolutionary Computation, pages 962–968.
  * Rajagopalan et al., [2011] Rajagopalan, R., Mohan, C. K., Mehrotra, K. G., and Varshney, P. K. (2011).  Multi-Objective Evolutionary Algorithms for Sensor Network Design.  Multi-Objective Optimization in Computational Intelligence, pages 208–238.
  * Rane and Srivastava, [2015] Rane, D. and Srivastava, A. (2015).  Cloud Brokering Architecture for Dynamic Placement of Virtual Machines.  Proceedings - 2015 IEEE 8th International Conference on Cloud Computing, CLOUD 2015, pages 661–668.
  * Ranjana et al., [2019] Ranjana, R., Radha, S., and Raja, J. (2019).  Network affinity aware energy efficient virtual machine placement algorithm.  International Journal of Business Intelligence and Data Mining, 14(1/2):40.
  * Risso et al., [2018] Risso, C., Nesmachnow, S., and Robledo, F. (2018).  Metaheuristic approaches for IP/MPLS network design.  International Transactions in Operational Research, 25(2):599–625.
  * Roytman et al., [2013] Roytman, A., Kansal, A., Govindan, S., Liu, J., and Nath, S. (2013).  Algorithm Design for Performance Aware VM Consolidation.  Technical Report MSR-TR-2013-28, Microsoft Research.
  * Rubio-Largo et al., [2013] Rubio-Largo, A., Vega-Rodriguez, M. A., Gomez-Pulido, J. A., and Sanchez-Perez, J. M. (2013).  Multiobjective Metaheuristics for Traffic Grooming in Optical Networks.  IEEE Transactions on Evolutionary Computation, 17(4):457–473.
  * Saber et al., [2018] Saber, T., Thorburn, J., Murphy, L., and Ventresque, A. (2018).  VM reassignment in hybrid clouds for large decentralised companies: A multi-objective challenge.  Future Generation Computer Systems, 79:751–764.
  * Sahoo et al., [2018] Sahoo, K. S., Puthal, D., Obaidat, M. S., Sarkar, A., Mishra, S. K., and Sahoo, B. (2018).  On the placement of controllers in software-Defined-WAN using meta-heuristic approach.  Journal of Systems and Software, 145(January):180–194.
  * Schiex et al., [1995] Schiex, T., Fargier, H., Verfaillie, G., et al. (1995).  Valued constraint satisfaction problems: Hard and easy problems.  IJCAI (1), 95:631–639.
  * Sedaghat et al., [2013] Sedaghat, M., Hernandez-Rodriguez, F., and Elmroth, E. (2013).  A virtual machine re-packing approach to the horizontal vs. vertical elasticity trade-off for cloud autoscaling.  In Proceedings of the 2013 ACM Cloud and Autonomic Computing Conference, CAC ’13, pages 6:1–6:10, New York, NY, USA. ACM.
  * Shi et al., [2015] Shi, R., Zhang, J., Chu, W., Bao, Q., Jin, X., Gong, C., Zhu, Q., Yu, C., and Rosenberg, S. (2015).  Mdp and machine learning-based cost-optimization of dynamic resource allocation for network function virtualization.  In 2015 IEEE International Conference on Services Computing, pages 65–73. IEEE.
  * Shi and Hong, [2011] Shi, W. and Hong, B. (2011).  Towards profitable virtual machine placement in the data center.  Proceedings - 2011 4th IEEE International Conference on Utility and Cloud Computing, UCC 2011, pages 138–145.
  * Shooshtarian et al., [2018] Shooshtarian, L., Safaei, F., and Tizghadam, A. (2018).  Scaling-up versus scaling-out networking in data centers : a comparative robustness analysis.  The Journal of Supercomputing, 74(8):3950–3974.
  * Silva Filho et al., [2018] Silva Filho, M. C., Monteiro, C. C., Inácio, P. R., and Freire, M. M. (2018).  Approaches for optimizing virtual machine placement and migration in cloud environments: A survey.  Journal of Parallel and Distributed Computing, 111:222–250.
  * Son and Huh, [2019] Son, A.-Y. and Huh, E.-N. (2019).  Multi-Objective Service Placement Scheme Based on Fuzzy-AHP System for Distributed Cloud Computing.  Applied Sciences, 9(17):3550.
  * Sonnek et al., [2010] Sonnek, J., Greensky, J., Reutiman, R., and Chandra, A. (2010).  Starling: Minimizing communication overhead in virtualized computing platforms using decentralized affinity-aware migration.  Proceedings of the International Conference on Parallel Processing, pages 228–237.
  * Sotiriadis et al., [2016] Sotiriadis, S., Vakanas, L., Petrakis, E., Zampognaro, P., and Bessis, N. (2016).  Automatic migration and deployment of cloud services for healthcare application development in FIWARE.  Proceedings - IEEE 30th International Conference on Advanced Information Networking and Applications Workshops, WAINA 2016, pages 416–419.
  * Sun et al., [2016] Sun, X., Ansari, N., and Wang, R. (2016).  Optimizing resource utilization of a data center.  IEEE Communications Surveys Tutorials, 18(4):2822–2846.
  * Sunil Rao and Santhi Thilagam, [2015] Sunil Rao, K. and Santhi Thilagam, P. (2015).  Heuristics based server consolidation with residual resource defragmentation in cloud data centers.  Future Generation Computer Systems, 50:87–98.
  * Talebian et al., [2019] Talebian, H., Gani, A., Sookhak, M., Abdelatif, A. A., Yousafzai, A., Vasilakos, A. V., and Yu, F. R. (2019).  Optimizing virtual machine placement in IaaS data centers: taxonomy, review and open issues.  Cluster Computing, 0123456789.
  * Tantawi, [2016] Tantawi, A. N. (2016).  Solution biasing for optimized cloud workload placement.  Proceedings - 2016 IEEE International Conference on Autonomic Computing, ICAC 2016, pages 105–110.
  * [118] Tarighi, M., Motamedi, S. A., and Arianyan, E. (2010a).  Performance improvement of virtualized cluster computing system using TOPSIS algorithm.  40th International Conference on Computers and Industrial Engineering: Soft Computing Techniques for Advanced Manufacturing and Service Systems, CIE40 2010, pages 1–6.
  * [119] Tarighi, M., Motamedi, S. A. A. A., and Sharifian, S. (2010b).  A new model for virtual machine migration in virtualized cluster server based on Fuzzy Decision Making.  Journal of Telecommunications, 1(1):40–51.
  * Tsakalozos et al., [2011] Tsakalozos, K., Roussopoulos, M., and Delis, A. (2011).  VM placement in non-homogeneous IaaS-clouds.  Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 7084 LNCS:172–187.
  * Tsakalozos et al., [2013] Tsakalozos, K., Roussopoulos, M., and Delis, A. (2013).  Hint-based execution of workloads in clouds with Nefeli.  IEEE Transactions on Parallel and Distributed Systems, 24(7):1331–1340.
  * Unuvar et al., [2014] Unuvar, M., Steinder, M., and Tantawi, A. N. (2014).  Hybrid cloud placement algorithm.  In 2014 IEEE 22nd International Symposium on Modelling, Analysis Simulation of Computer and Telecommunication Systems, pages 197–206.
  * Van et al., [2009] Van, H. N., Tran, F. D., and Menaud, J.-M. (2009).  Autonomic virtual resource management for service hosting platforms.  In 2009 ICSE Workshop on Software Engineering Challenges of Cloud Computing, pages 1–8. IEEE.
  * Verma et al., [2010] Verma, A., Kumar, G., and Koller, R. (2010).  The cost of reconfiguration in a cloud.  In Proceedings of the 11th International Middleware Conference Industrial Track, Middleware Industrial Track ’10, pages 11–16, New York, NY, USA. ACM.
  * Viswanathan et al., [2011] Viswanathan, H., Lee, E. K., Rodero, I., Pompili, D., Parashar, M., and Gamell, M. (2011).  Energy-aware application-centric vm allocation for hpc workloads.  In 2011 IEEE International Symposium on Parallel and Distributed Processing Workshops and Phd Forum, pages 890–897.
  * Wang et al., [2016] Wang, J. V., Fok, K. Y., Cheng, C. T., and Tse, C. K. (2016).  A Stable Matching-Based Virtual Machine Allocation Mechanism for Cloud Data Centers.  Proceedings - 2016 IEEE World Congress on Services, SERVICES 2016, pages 103–106.
  * Wang et al., [2007] Wang, X., Lan, D., Wang, G., Fang, X., Ye, M., Chen, Y., and Wang, Q. (2007).  Appliance-based autonomic provisioning framework for virtualized outsourcing data center.  In Fourth International Conference on Autonomic Computing (ICAC’07), pages 29–29. IEEE.
  * Whaiduzzaman et al., [2014] Whaiduzzaman, M., Gani, A., Anuar, N. B., Shiraz, M., Haque, M. N., and Haque, I. T. (2014).  Cloud Service Selection Using Multicriteria Decision Analysis.  The Scientific World Journal, 2014:1–10.
  * Wichtlhuber et al., [2015] Wichtlhuber, M., Reinecke, R., and Hausheer, D. (2015).  An SDN-based CDN/ISP collaboration architecture for managing high-volume flows.  IEEE Transactions on Network and Service Management, 12(1):48–60.
  * Wright et al., [2012] Wright, P., Sun, Y. L., Harmer, T., Keenan, A., Stewart, A., and Perrott, R. (2012).  A constraints-based resource discovery model for multi-provider cloud environments.  Journal of Cloud Computing, 1(1):1–14.
  * Wu et al., [2015] Wu, J., Zhang, Z., Hong, Y., and Wen, Y. (2015).  Cloud radio access network (C-RAN): a primer.  IEEE Network, 29(1):35–41.
  * Wu et al., [2016] Wu, Q., Ding, G., Du, Z., Sun, Y., Jo, M., and Vasilakos, A. V. (2016).  A Cloud-Based Architecture for the Internet of Spectrum Devices Over Future Wireless Networks.  IEEE Access, 4:2854–2862.
  * Xing et al., [2019] Xing, C., Addie, R. G., Peng, Y., Lin, R., Li, F., Hu, W., Abramov, V. M., and Zukerman, M. (2019).  Resource Provisioning for a Multi-Layered Network.  IEEE Access, 7:16226–16245.
  * Xu and Li, [2011] Xu, H. and Li, B. (2011).  Egalitarian stable matching for VM migration in cloud computing.  2011 IEEE Conference on Computer Communications Workshops, INFOCOM WKSHPS 2011, (May 2011):631–636.
  * Xu and Li, [2013] Xu, H. and Li, B. (2013).  Anchor: A versatile and efficient framework for resource management in the cloud.  IEEE Transactions on Parallel and Distributed Systems, 24(6):1066–1076.
  * Xu et al., [2014] Xu, X., Yu, H., and Pei, X. (2014).  A novel resource scheduling approach in container based clouds.  In 2014 IEEE 17th International Conference on Computational Science and Engineering, pages 257–264. IEEE.
  * Yao et al., [2014] Yao, F., Wu, J., Venkataramani, G., and Subramaniam, S. (2014).  A comparative analysis of data center network architectures.  In 2014 IEEE International Conference on Communications (ICC), pages 3106–3111.
  * Ye et al., [2015] Ye, K., Wu, Z., Wang, C., Zhou, B. B., Si, W., Jiang, X., and Zomaya, A. Y. (2015).  Profiling-based workload consolidation and migration in virtualized data centers.  IEEE Transactions on Parallel and Distributed Systems, 26(3):878–890.
  * Zhang et al., [2018] Zhang, F., Liu, G., Fu, X., and Yahyapour, R. (2018).  A survey on virtual machine migration: Challenges, techniques, and open issues.  IEEE Communications Surveys Tutorials, 20(2):1206–1243.
  * Zhang et al., [2016] Zhang, W., Tan, S., Xia, F., Chen, X., Li, Z., Lu, Q., and Yang, S. (2016).  A survey on decision making for task migration in mobile cloud environments.  Personal and Ubiquitous Computing, 20(3):295–309.
  * Zhang et al., [2012] Zhang, Y., Li, M., Bai, K., Yu, M., and Zang, W. (2012).  Incentive compatible moving target defense against vm-colocation attacks in clouds.  In Gritzalis, D., Furnell, S., and Theoharidou, M., editors, Information Security and Privacy Research, pages 388–399, Berlin, Heidelberg. Springer Berlin Heidelberg.
  * Zhou et al., [2013] Zhou, R., Liu, F., Li, C., and Li, T. (2013).  Optimizing virtual machine live storage migration in heterogeneous storage environment.  ACM SIGPLAN Notices, 48(7):73.