#  Jiagu: Optimizing Serverless Computing Resource Utilization with Harmonized
Efficiency and Practicability

Qingyuan Liu  Shanghai Jiao Tong UniversityShanghai200240China ,  Yanning Yang
Shanghai Jiao Tong UniversityShanghai200240China ,  Dong Du  Shanghai Jiao
Tong UniversityShanghai200240China ,  Yubin Xia  Shanghai Jiao Tong
UniversityShanghai200240China ,  Ping Zhang  Huawei CloudChina ,  Jia Feng
Huawei CloudChina ,  James Larus  EPFLSwitzerland and  Haibo Chen  Shanghai
Jiao Tong UniversityShanghai200240China

###### Abstract.

Current serverless platforms struggle to optimize resource utilization due to
their dynamic and fine-grained nature. Conventional techniques like
overcommitment and autoscaling fall short, often sacrificing utilization for
practicability or incurring performance trade-offs. Overcommitment requires
predicting performance to prevent QoS violation, introducing trade-off between
prediction accuracy and overheads. Autoscaling requires scaling instances in
response to load fluctuations quickly to reduce resource wastage, but more
frequent scaling also leads to more cold start overheads. This paper
introduces Jiagu to harmonize efficiency with practicability through two novel
techniques. First, pre-decision scheduling achieves accurate prediction while
eliminating overheads by decoupling prediction and scheduling. Second, dual-
staged scaling achieves frequent adjustment of instances with minimum
overhead. We have implemented a prototype and evaluated it using real-world
applications and traces from the public cloud platform. Our evaluation shows a
54.8% improvement in deployment density over commercial clouds (with
Kubernetes) while maintaining QoS, and 81.0%–93.7% lower scheduling costs and
a 57.4%–69.3% reduction in cold start latency compared to existing QoS-aware
schedulers in research work.

††copyright: rightsretained

##  1\. Introduction

Serverless computing (Jonas et al., 2019) simplifies the management of cloud
applications with features like on-demand execution and autoscaling. Through
autoscaling, cloud providers are able to dynamically deploy zero to numerous
instances to run a specific function in response to real-time demands,
optimizing performance and cost efficiency. Critical to this process are two
main components: the autoscaler and the scheduler. The autoscaler utilizes a
predefined threshold for each function, called saturated value (e.g., requests
per second, or RPS), and triggers the creation of new instances as needed when
the live load approaches this threshold (kna, 2023; ope, 2023b)111This
approach typically ensures individual instance loads do not exceed the
saturated value, as doing so would lead to the instantiation of additional
instances, thereby redistributing the load.. By doing so, the autoscaler
prevents instances from becoming overloaded and guarantees performance, which
is generally measured in terms of Quality-of-Service (QoS). When the
autoscaler triggers the creation of a new instance, the scheduler is
responsible for scheduling the instance to a suitable node. In making these
assignments, the scheduler accounts for various aspects, including whether a
node can meet a function’s resource requirements which developers manually
define.

Figure 1. Opportunities of improving resource utilization. “Configured
resources” means the allocated resources can be fully utilized. The blue line
demonstrates the resource utilization of instances that are serving a
saturated value of load, which fluctuates due to the variation in resource
interference.

However, current serverless systems still fall short on efficiently utilizing
resources. According to traces from AliCloud (Tian et al., 2022) and our
public cloud (i.e., Huawei Cloud, a top-five IaaS provider (gar, 2023)), most
serverless functions can only utilize a small portion of allocated resources.
As illustrated in Figure 1, the issue mainly results from two factors. First,
to guarantee performance, users usually consider the worst case, and thus
specify excessive resources to instances. Even if instances are processing
saturated load of requests, they cannot utilize the allocated resources. This
causes the resource wastage of part ① in Figure 1. Second, in practice, user
loads served by each instance continuously fluctuate. Under-loaded instances
usually require fewer resources and are less able to utilize the resources,
resulting in resource wastage of part ② in Figure 1.

To mitigate the two issues, nowadays serverless systems can leverage
overcommitment (Tian et al., 2022; Li et al., 2023) and autoscaling (Qiu et
al., 2023). First, for part ① in Figure 1, overcommitment is an intuitive
solution, i.e., deploy more instances on a server. This approach could be
implemented by the scheduler, as the scheduler is responsible for deciding how
instances are deployed. Second, for part ②, it could be helpful to reduce the
load fluctuations within instances, trying to keep the load close to the
saturated load for each instance. This can be achieved by the autoscaler,
which is responsible for observing load fluctuations and adjusting the number
of instances accordingly.

Nevertheless, it is challenging to apply the two approaches while achieving
effectiveness with practical cost. First, since overcommitment may lead to
performance degradation and QoS violation, it is a common practice to predict
performance before scheduling an instance to avoid QoS violation. However, it
is challenging to accurately predict instances’ performance with a practical
cost. Although prior predictor-based schedulers have made great progress (Tian
et al., 2022; Zhao et al., 2021; Delimitrou and Kozyrakis, 2013; Mars et al.,
2011; Mishra et al., 2017; Xu et al., 2018; Mars and Tang, 2013), they fall
short in achieving both high accuracy and low cost, since the two goals can be
a trade-off: making accurate prediction requires considering complex
interference on multiple resources imposed by highly heterogeneous colocated
functions, but the complex computation inevitably introduces high overheads.
For example, some prior works predict with specially designed machine learning
models (Zhao et al., 2021), which is costly for scheduling. Since cold start
latency mainly consists of scheduling latency and instance initialization
latency, with initialization being optimized by a lot of work to even $<$1ms,
scheduling with inference ($\sim$20ms or higher) could become a new bottleneck
(Liu et al., 2023). To compromise, nowadays public clouds still use heuristic
policies for scheduling, thus failing to accurately predict performance and
providing limited improvement in resource utilization.

Second, to efficiently utilize resources under load fluctuation, the
autoscaler needs to quickly respond to the load variation, evict instances and
free the resources when the load drops. Intuitively, the faster autoscaling
can react to load fluctuations, the more efficiently resources can be used.
However, more sensitive and frequent scaling also means more additional cold
starts, resulting in the trade-off between resource utilization and cold start
overheads. It is challenging to achieve both goals simultaneously.

We present two key insights to tackle the aforementioned challenges. First,
the trade-off between scheduling costs and efficiency is due to the fact that
these complex decisions need to be made at the time of scheduling, which
involves costly predictions. Therefore, it can be beneficial if we decouple
prediction and decision making. Specifically, we can predict in advance the
performance of possible incoming instances and save the results. If the
incoming instance matches the advance predicted scenario, the scheduling can
be made by directly checking the prepared decisions without model inference.
This provides a scheduling fast path.

Second, the trade-off between resource utilization and cold starts arises from
that instances do not release resources until they are evicted. Instead, we
can decouple resource releasing and instance eviction. Specifically,
operations like sending requests to fewer instances could achieve similar
effect of releasing resources as eviction. Releasing resources without
instance eviction can be less costly, and thus can be applied with higher
sensitivity for better resource utilization.

Inspired by the two insights, we propose Jiagu, a serverless system that
harmonizes efficiency and practicability in improving resource utilization,
tackling the challenges with two techniques. First, Jiagu achieves accurate
and low cost performance prediction with pre-decision scheduling, which
decouples prediction and decision making, providing a scheduling fast path
without inference. For every deployed function on a server, Jiagu predicts in
advance their capacities on that server with a model. A function’s capacity
means the maximum number of its instances can be deployed on the server under
the interference of current neighbors without violating everyone’s QoS. To
schedule a new instance of a deployed function, Jiagu only needs to compare
the function’s capacity with the number of its instances to determine whether
the scheduling can be successful or not. Moreover, Jiagu designs an
asynchronous update approach, which keeps the capacities up-to-date without
incurring model inference to the scheduling critical path, and applies a
concurrency-aware scheduling to batch the scheduling of concurrent incoming
instances when load spikes occur. Results with real-world patterns show that
¿80% scheduling goes through the fast path.

Second, Jiagu designs a dual-staged scaling method. When load drops, before
instances are evicted, Jiagu first adjusts the routing with a higher
sensitivity, sending requests to fewer instances, thus releasing some of the
instances’ resources. Then, if the load rises again before idle instances are
evicted, Jiagu can get the instances working again by re-routing with minimum
overhead. Third, to cope with the case where a node is full and causes idle
instances on it can not be converted to the saturated state, Jiagu migrates
idle instances to other nodes in advance to hide the overhead of the required
cold starts.

We present a prototype of Jiagu based on an open-source serverless system,
OpenFaaS (ope, 2023a). We evaluate it using ServerlessBench (Yu et al., 2020),
FunctionBench (Kim and Lee, 2019), and applications with real-world traces
from Huawei Cloud. The results show that on real-world traces, compared to a
state-of-the-art model-based serverless scheduler (Zhao et al., 2021), Jiagu
incurs 81.0%–93.7% lower scheduling costs and 57.4%–69.3% lower cold start
latency with cfork (Du et al., 2022). It also achieves the highest resource
utilization compared to all baseline schedulers (54.8% higher instance density
than Kubernetes).

##  2\. Motivation

###  2.1. Background and Motivation

Notations and terms. The term function denotes the basic scheduling unit
within a serverless platform. A function might comprise multiple instances to
serve requests. Each instance is allocated with specified resources by user
configurations. Quality-of-Service (QoS) is utilized to describe the
performance targets for functions. In our platform, QoS is primarily
characterized by tail latency; however, the definition can be extended to
other metrics.

Components for load and utilization management in serverless platforms. First,
router dispatches user requests to available instances of a function,
typically employing a load balancing strategy to ensure equal distribution of
requests among instances (Figure 2). Second, autoscaler (kna, 2023; ope,
2023b) is responsible for determining the number of instances. It uses a pre-
determined RPS threshold, referred to as the saturated load, to determine
whether it is necessary to create or evict instances, to avoid instance
overloaded or resource waste. Different from creating instances, the
autoscaler evicts instances with a keep-alive approach, which does not
immediately evict instances when the expected number of instances drops, but
waits for a “keep-alive duration”. The keep-alive duration can reflect the
sensitivity of the autoscaler — shorter duration means higher sensitivity and
more frequent evictions. Last, when a new instance is created, a scheduler is
responsible for assigning a specific node for it to deploy the instance, which
will consider several factors like the resource requirements.

Figure 2. Schedulers and routers.

Low resource utilization in serverless platforms. Optimizing resource usage is
one of the top cloud initiatives (fle, 2023), as every 1% increase in
utilization could mean more than millions of dollars a year in cost savings.
However, production traces show that the resources of serverless platforms are
still severely under-utilized (Figure 4). In general, the wastage mainly
arises from two parts: part ① and ② in Figure 1.

Part ① is because the resources allocated by the user are usually
conservative, considering the worst cases that suffer the severest resource
interference. Therefore, the allocated resources could be even higher than the
actual demand of instances that are processing saturated load. Part ② is
because the load served by each instance constantly fluctuates, and under-
loaded instances are less able to fully utilize the resources. For example,
Figure 3 shows the average RPS served by an instance for one of the most
popular functions in the trace of Huawei Cloud. If the instance is always
considered saturated, 51% of resources could possibly be wasted.

Opportunities for improving resource utilization. To mitigate the issues,
serverless systems could adopt two approaches respectively. First is
overcommitment for part ①, i.e., the scheduler could deploy more instances on
a server. Second is autoscaling for part ②, i.e., the autoscaler dynamically
evicts instances when the load drops, reducing the overestimation of resource
demands for under-loaded instances. Applying the two approaches, our objective
is to simultaneously achieve effectiveness in improving resource utilization
with practical cost.

Figure 3. Fluctuation of user load per instance.

Figure 4. Statistics on the ratio of actual resource usage to allocated resources. The data was collected on Huawei Cloud over one month from a region. About 80% of servers only use 8% CPU and 64% memory resources. Table 1. A comparison between Jiagu and previously developed methods. Scalable profiling requires no more than O($n$) complexity, where $n$ represents the number of functions, $k$ denotes to the number of instances that can be deployed on a server. Pythia, due to have a model for each function, requires O($n^{2}$) complexity. Whare-map requires O($n^{k}$) complexity to profile all colocation combinations where $k$ denotes to the number of co-runners allowed on a server. Owl profiles the combination of two functions with $k$ instances, requiring O($n^{2}k$) complexity. “Fast scheduling” implies that the scheduler makes an decision within a few milliseconds ($\sim$1ms). Gsight requires more than 20ms for scheduling (§7.2).  Workload | System | Year | Efficient QoS-Aware Scheduler with Low Cost? |  Handling Load Dynamicity?
---|---|---|---|---
Prediction model? |  Accurate Prediction? | Overhead?
Profiling cost? | Fast scheduling?
Serverless | Jiagu |  | Global statistical model | ✓ | O($n$) | ✓ | Dual-staged scaling
Owl (Tian et al., 2022) | 2022 | Historical information | Limited | O($n^{2}k$) | ✓ | Autoscaling
Gsight (Zhao et al., 2021) | 2021 | Global statistical model | ✓ | O($n$) | ✗ | Autoscaling
Monolithic Service |  Pythia (Xu et al., 2018) | 2018 |  | Per-function Linear model
---
✓ | O($n^{2}$) | ✓ | Autoscaling
Paragon (Delimitrou and Kozyrakis, 2013) | 2013 | Heuristic | NA | O($n$) | ✓ | Autoscaling
Whare-map (Mars and Tang, 2013) | 2013 | Historical information | Limited | O($n^{k}$) | ✓ | Autoscaling
Bubble-up (Mars et al., 2011) | 2011 | Heuristic | Inaccurate | O($n$) | ✓ | Autoscaling
| Resource
---
Schedulers
|  Aquatope (Zhou et al., 2022), Cilantro (Bhardwaj et al., 2023), Sinan
(Zhang et al., 2021b)
---
FIRM (Qiu et al., 2020), Parties (Chen et al., 2019a), Heracles (Lo et al.,
2015), …
Resilient to runtime variations with limitations (§8).

###  2.2. Challenges

However, achieving both efficiency and practicability can be challenging,
since the two goals usually have trade-offs.

####  2.2.1. Challenges for the Scheduler

Although overcommitment helps improve resource utilization, it could also
cause performance degradation, possibly resulting in QoS violation. Therefore,
a common approach for the scheduler is to predict instances’ performances when
scheduling, maximizing instance deployment density while not violating
instances’ QoS. The goal means that the scheduler should simultaneously
achieve: accurate QoS violation prediction (with scalable profiling overhead)
and low scheduling latency. Although prior schedulers have made great progress
for the serverless scenario (Tian et al., 2022; Zhao et al., 2021) compared
with schedulers (Delimitrou and Kozyrakis, 2013; Mars et al., 2011; Mishra et
al., 2017; Xu et al., 2018; Mars and Tang, 2013) designed for monolithic
services, they fall short of achieving both goals. Table 1 summarizes the key
differences of prior schedulers against our objectives.

Accurate QoS prediction. The key to accurately predicting the QoS violation is
to quantify the resource interference by colocated function instances, which
is challenging. First, in the serverless scenario, the number of colocation
combinations grows exponentially with the number of functions, and ultimately
becomes practically infinite. Therefore, the profiling overhead should be
scalable, otherwise it could be impossible to profile such a large number of
colocation combinations. Second, resource interference can be complex, while
plentiful and heterogeneous functions exert various pressures on multiple
resources. The large number of colocation combinations further amplifies the
interference complexity.

Some prior schedulers do not meet the goal yet. For example, some schedulers
predict based on historical performance information (Tian et al., 2022; Mars
and Tang, 2013), so that their predictive ability is limited and can only
predict the performance of profiled colocated instance combinations. Their
profiling cost is also unscalable (shown in Table 1). Schedulers that train a
model for each function has unacceptable profiling and training overhead (Xu
et al., 2018). Schedulers using heuristic algorithms are limited in their
ability to accurately predict performance due to overly simplistic models
(Mars et al., 2011), or does not actually predict performance at all
(Delimitrou and Kozyrakis, 2013).

Table 2. State-of-the-art serverless systems with startup optimizations. “Container startup” presents the runtime startup latency reported by public reports or papers. “Scheduling overhead” presents the overhead of scheduling in a cold startup with model-based methods. We use 21.78ms as the model-based scheduling costs, the average result of our ported Gsight (Zhao et al., 2021) model.  System | Container startup | Scheduling overhead
---|---|---
AWS Snapstart (Brooker, 2022) |  $\sim$100ms |  $\sim$21.8%
Replayable (Wang et al., 2019) | 54ms | 40.3%
Fireworks (Shin et al., 2022) |  $\sim$50ms |  $\sim$43.6%
SOCK (Oakes et al., 2018) | 20ms | 108.9%
Molecule (Du et al., 2022) | 8.4ms | 2.6x
SEUSS (Cadden et al., 2020) | 7.5ms | 2.9x
Catalyzer (Du et al., 2020) | 0.97ms | 22.5x
Faasm (Shillaker and Pietzuch, 2020) | 0.5ms | 43.6x

Low scheduling latency. The scheduling process is involved in every instance’s
cold start. Given the recent advancements in optimizing the instance
initialization cost to a few milliseconds or even sub-milliseconds (Wang et
al., 2021; Akkus et al., 2018; Wang et al., 2019; Du et al., 2020; Oakes et
al., 2018; Jia and Witchel, 2021b; Shin et al., 2022; Du et al., 2022; Cadden
et al., 2020; Shillaker and Pietzuch, 2020), it becomes imperative to ensure
that the scheduling cost remains low as well. Otherwise, the scheduling cost
can become the new bottleneck in the startup cost (Liu et al., 2023). As shown
in Table 2, we use the average scheduling costs of Gsight (Zhao et al., 2021)
(our ported version) to illustrate the overhead of scheduling over the total
cold startup costs in these systems. Although it can accurately predict the
performance and consider serverless-specific features like partial
interference, it can incur non-trivial overhead even in commercial systems
like AWS Lambda with Snapstart (Brooker, 2022) (¿20%). Scheduling cost can be
dominant for cold starts with state-of-the-art optimizations like Catalyzer
(Du et al., 2020) and Faasm (Shillaker and Pietzuch, 2020). In short,
schedulers need to make decisions within a few milliseconds to avoid hindering
the startup process.

####  2.2.2. Challenges for Autoscaling

The autoscaling feature commonly helps mitigate the defect. When load drops,
the autoscaler would dynamically evict instances, thus freeing space for
deploying new instances, preventing those under-loaded instances from wasting
resources (typically every instance can be under-loaded because of load
balancing). Intuitively, a more sensitive autoscaler could better utilize
resources in response to load fluctuation. However, more frequent scaling
would also result in more additional cold starts. It means the trade-off
between resource utilization and cold start costs.

The irregularity of load fluctuations further complicates the problem. Prior
works have analyzed production traces and tried to predict the invocation
patterns (Fuerst and Sharma, 2021; Shahrad et al., 2020; Roy et al., 2022;
Zhou et al., 2022), pre-warming instances accordingly to reduce cold starts.
However, user loads exhibit only moderate regularity over extended periods of
time (e.g., diurnal patterns), but are extremely unpredictable over shorter
intervals. For example, the average coefficient of variable (CV) over the
number of requests in a minute could be more than 10 (Zhang et al., 2023) in
the Azure trace (Shahrad et al., 2020). The more frequent scaling also means a
finer-grained pre-warming prediction and a worse prediction accuracy.

###  2.3. Insights

We observe two insights that help to tackle the two challenges.

Insight-1: Prediction and decision making can be decoupled. The trade-off
between prediction accuracy and scheduling cost arises from the coupling of
the prediction and decision making, i.e., the costly model inference is
usually made during scheduling. We can decouple prediction and decision
making. Specifically, the scheduler can assume the interference environment
before the new instance arrives, and make advance predictions based on the
assumptions. When a new instance arrives and the interference environment at
the time of scheduling matches our previous assumption, the scheduling can be
made by directly checking the prepared decisions. This provides a scheduling
fast path without inference.

Insight-2: Resource releasing and instance eviction can be decoupled. The
trade-off between resource utilization and cold start overheads arises from
the coupling of resource releasing and instance eviction. Instead, we can
decouple resource releasing and instance eviction. Specifically, even if an
instance is not evicted, we could adjust the routing and do not send requests
to it. It consolidates the loads of under-loaded instances to fewer instances
to reduce the waste of resources caused by treating under-loaded instances as
saturated instances, achieving a similar resource releasing effect as an
actual eviction. The overhead of adjusting the routing is much smaller than
actual cold start overheads. Therefore, in this way, we could release/reclaim
resources with higher sensitivity to cope with load fluctuations, while
avoiding excessive additional cold start overheads.

##  3\. Design Overview

We propose Jiagu, an efficient and practical QoS-aware serverless system that
tackles the two challenges of improving resource utilization with the
insights. The overall design is shown in Figure 5. First, Jiagu designs a pre-
decision scheduling that could make accurate predictions with low scheduling
latency (§4). When an instance is created, Jiagu predicts its performance
after deployment, trying to increase the instance deployment density without
violating QoS to fully utilize the resources. Second, Jiagu adopts a dual-
staged scaling design that efficiently utilizes resources under load
fluctuation with minimum overhead (§5).

Cluster setting. To apply Jiagu, our cluster is partitioned into three types
of nodes. A small proportion of nodes are profiling nodes and training nodes,
deploying instances whose runtime information is collected to construct
function profiles and the dataset to train the model used by Jiagu’s
scheduler. Jiagu schedules instances that actually handle user requests to
high-density nodes, maximizing the resource utilization of these nodes. The
training, profiling and high-density nodes are homogeneous.

User configurations and QoS. In our platform, when uploading a function, a
user is expected to specify the allocated resources (CPU and memory) and the
saturated load of the function (optional with default value). The QoS
requirement of a function is established by the provider according to the
function’s previously monitored performances, as an important reference metric
for overcommitment. Moreover, in Huawei Cloud, we can establish specific QoS
agreements with top-tier customers, since their workload could be the majority
in the cloud.

Figure 5. Jiagu overview.

##  4\. Pre-decision Scheduling

###  4.1. Jiagu’s Prediciton Model

Prediction model. Jiagu’s ability of QoS-aware scheduling requires accurate
prediction of the functions’ performance under resource interference. The
prediction model is based on Random Forest Regression (RFR). Specifically,
Jiagu’s regression model has the following form:

$P_{A\cup\\{B,C,...\\}}=RFR\\{P_{A},R_{A},C_{A},R_{B},C_{B},R_{C},C_{C},...\\}$

where, $P_{A\cup\\{B,C,...\\}}$ is function $A$’s performance222 Although we
use P90 tail latency as our performance metrics, it is a general model that
can be extended to other metrics like costs. under the interference of
functions $B$ and $C$. $R_{A}$, $R_{B}$ and $R_{C}$ is the profile matrix of
function $A$, $B$ and $C$ respectively. Jiagu uses a single instance’s
mutliple resource’s utilization as its profile (Table 3), and adopts a solo-
run approach to collect the profile metrics of functions (details in §6).
$C_{A}$, $C_{B}$ and $C_{C}$ are concurrency information of the three
functions. The concurrency information includes two parts: the number of
saturated instances and the number of cached instances (detailed in §5).
$P_{A}$ is the solo-run performance of function $A$.

The model is proven effective in our public cloud, and also utilized in prior
systems (Zhao et al., 2021; Li et al., 2023). Moreover, compared to prior
models that predict the performance in the instance granularity (Zhao et al.,
2021; Xu et al., 2018), since different instances of the same function are
homogeneous and performs similarly on the same server, the model predicts the
performance in function-granularity. We merge the features of the same
function’s instances and introduce concurrency as a new feature for a
function. This effectively reduces the input dimensions, resulting in less
training overhead and possible mitigation of the “curse of dimensionality”
(Bellman, 2003). Jiagu is also flexible to utilize other prediction models.

Table 3. Profiling metrics. Metrics |  Description
---|---
mCPU |  CPU utilization
Instructions |  Instructions retired
IPC |  Instructions per cycle
Context switches |  Switching between privilege modes
MLP |  Efficiency of concurrent access indicated by Memory Level Parallelism (MLP)
L1d/L1i/L2/LLC MPKI |  Cache locality indicated by misses per thousand instructions (MPKI)
TLB data/inst. MPKI |  TLB locality indicated by MPKI
Branch MPKI |  Branch predictor locality indicated by MPKI
Memory bandwidth |  Memory usage and performance

Inference overhead. In practice, a policy may require predicting different
functions’ performances under various colocation environments, and the
colocation environments can be described by multiple inputs. Notably, learning
frameworks (e.g., sklearn(sk-, 2023)) have optimizations to infer multiple
simultaneous inputs, which incurs trivial additional cost (§7). Therefore, in
the descriptions that follow, if multiple predictions can be merged in the
form of multiple inputs, we refer to such inference overhead as “once”
inference overhead.

###  4.2. Capacity and Capacity Table

Inspired by the first insight, to accurately predict instances’ performances
without incurring excessive cost to scheduling, we can decouple prediction and
decision making. However, making predictions requires knowledge of what the
incoming instance is and what the interference environment on the server is,
which is not known until scheduling (decision making). Considering there are
infinite number of functions, it is also impossible to traverse all possible
interference environments. Therefore, the decoupling can be challenging.

(a) Weighted instance concurrencies of functions.

(b) Weighted instance concurrencies (¡13) of functions.

Figure 6. Analysis of highly-replicated characteristic.

Serverless features that help tackle the challenge. We realize that the
highly-replicated feature of serverless may help address the issue. The
feature can be illustrated with the following statistical analysis on real-
world traces collected from Huawei Cloud’s production environment.
Specifically, we mainly analyze the concurrency of function instances, i.e.,
the number of concurrently running instances of a function in a certain time
window, rather than the invocation concurrencies in previous studies (Roy et
al., 2022; Shahrad et al., 2020; Zhang et al., 2021a; Zhou et al., 2022). We
weight the ratio of every concurrency value by itself, e.g., if there are 100
functions that have only 1 concurrency but 1 function whose concurrency is
100, the CDF point should be (1, 0.5) and (100, 1). The distribution is shown
in Figure 6-a. Each point (e.g., ($x$,$y$)) in the figure indicates that
($y$$\times$100)% instances belong to functions whose concurrencies are $\leq
x$. For example, 56% of instances are from functions whose concurrency is
$>$12, which is shown more clearly in Figure 6-b. Instances of functions with
only one concurrent instance are just 23% of all instances. In conclusion,
such analysis shows serverless’s highly-replicated feature — most serverless
functions have many replicated instances.

Introducing capacity for decoupling. The feature means, for a deployed
instance on a server, it is likely that more instances of the same function
would come in the future. Moreover, considering locality when deploying
instances has also been proven beneficial by plentiful prior works (Schall et
al., 2022; Abdi et al., 2023). Therefore, to decouple prediction and decision
making, we can predict in advance whether the next incoming instance of
existing functions can be deployed. With this idea, for every server, Jiagu
calculates capacity for each existing function on that server. Jiagu predicts
in advance the function’s new instance’s performance after deployment, and
sets the capacity value “true” if the predicted performance meets its QoS. The
capacities for all functions on a server form the server’s capacity table.
After that, when the scheduler is making scheduling decisions, it can judge
QoS violation by simply checking the table without model inference. It is the
scheduling “fast path”. In addition, it is only necessary to make predictions
on the critical path if the function to which the incoming instance belongs is
not in the capacity table. It is the scheduling “slow path”.

###  4.3. Asynchronous Update

Although introducing capacity could predict the QoS violation of new instances
in advance, the deployment of a new instance could impose resource
interference to its neighbors, possibly causing their QoS violations.
Therefore, it is necessary to introduce a validation process, which predicts
and checks whether neighbors will violate their QoS after deploying the new
instance. However, such validation introduces costly inference for cold starts
if done before deployment, or the risk of QoS violation if done after
deployment.

To mitigate the defect, Jiagu applies an asynchronous update approach. To
avoid the cost, asynchronous update refines the way to calculate the capacity,
predicting all colocated functions’ performances, and setting the capacity
“true” if the predicted performance of each function meets its own QoS. In
this way, after deploying the new instance, the predicted performances of
either the new instance itself or the neighbors would not violate the QoS.
This prevents the validation from affecting the scheduling latency.

Furthermore, when a new instance is scheduled to a server, it triggers
updating the capacity table. Now that the validation can be delayed, and this
update can be done asynchronously, outside the scheduling critical path. By
asynchronous update, the capacity table is always up-to-date with minimum
overhead, ensure that the capacity table always reflects real time
interference on servers when scheduling.

###  4.4. Concurrency-aware Scheduling

Decoupling prediction efficiently reduces model inference for individual
schedulings. Moreover, one of the extreme invocation patterns in the cloud is
the load spike, where the load rises rapidly so that multiple instances are
created simultaneously. In such case, the asynchronous update of the previous
instance could possibly block the scheduling of the next instance, as shown in
Figure 8-a. To mitigate, Jiagu further adopts concurrency-aware scheduling,
which batches the scheduling of concurrent incoming instances at load spikes.

Jiagu further refines the way of calculating capacity, considering not only
whether the next instance can be deployed, but also how many next instances
can be deployed, as shown in Figure 7. The capacity value of each function is
no longer a bool value, but a specific numerical value indicating how many
instances of the function can be deployed with current neighbors. For example,
for a server where there are 2 f1 and 3 f2 instances, with f1 and f2’s
capacities are 4 and 6 respectively. It means that 4 f1 instances can be
deployed with 3 f2 instances, while 6 f2 instances can be deployed with 2 f1
instances. Then, as shown in Figure 8-b, when multiple instances of the same
function arrive, if the function’s capacity on the server is sufficient to
accommodate those new instances, the scheduling and the asynchronous update
can be done once for scheduling multiple instances.

Figure 7. Calculating capacity. It finds the maximum concurrency value as the
capacity which every colocated function’s predicted performance can guarantee
its QoS.

(a) Without batching.

(b) With batching.

Figure 8. Concurrency-aware scheduling enables batch scheduling for one
function’s multiple instances.

###  4.5. Put It All Together: Scheduling Example

Now we describe the complete scheduling process and provide an example. As
shown in Figure 5, when a function’s new instance is created, Jiagu will first
select a node for it using a node filter (detailed in §6). Then, based on the
node’s capacity table built in advance, the scheduler decides whether to
schedule the instance via a fast path or a slow path.

Figure 9 demonstrates an example. First, an instance of f3 incomes and the
node filter selects the node in Figure 9 for it. The scheduler checks the
capacity table and finds that there is no entry for f3. Therefore, the
scheduler calculates f3’s capacity using the prediction model (as shown in
Figure 7). The result means that 3 f3’s instances can be deployed under
current numbers of f1 and f2 instances, so the new instance of f3 can be
deployed on that server. It is the scheduling slow path involving a costly
model inference.

Then, when load spike occurs, two instances of f3 are created and come to the
node. The scheduler checks the capacity table that the number of f3’s
instances will not exceed its capacity after deployment, so that the
scheduling decision is made quickly without model inference. This is the
scheduling fast path. The deployment also triggers an asynchronous update of
the capacity table, which can be done outside the scheduling critical path.
The scheduling of two instances are batched, triggering only one update of the
capacity table.

When load drops and an instance of f3 is evicted, such event will also trigger
an update to the capacity table. The increased capacities of f1 and f2 mean
that the resources can be reutilized by the scheduler to deploy more instances
later. Of course, this update would hardly affect the scheduling.

Figure 9. Scheduling example on a node.

##  5\. Dual-staged Scaling

(a) Existing eviction.

(b) Dual-staged eviction.

Figure 10. Dual-staged scaling. In the example, the release and keep-alive
duration are 10 and 60 seconds respectively.

Motivated by the second insight, Jiagu designs dual-staged scaling to
efficiently utilize resources under load fluctuation while minimizing cold
start overheads.

Dual-staged eviction. The key technique of the scaling approach is dual-staged
eviction. When load drops and so as the expected number of instances
decreases, instead of directly evicting instances after the keep-alive
duration as shown in Figure 10-a, Jiagu introduces a “release” duration
(Figure 10-b). The “release” duration is (much) shorter than the keep-alive
duration, i.e., the “release” operation is more sensitively triggered before
eviction. After the expected number of instances drops for the release
duration, the autoscaler first triggers the “release” operation, which changes
the routing rules, sending requests to fewer instances, but does not actually
evict any instances. The instances that do not actually process requests are
called cached instances, while other instances that are still processing
requests are called saturated instances, as shown in Figure 10-b. In the
example, the load of four instances are consolidated to three instances.
Therefore, when considering resource interference, the scheduler would
consider only three saturated instances instead of four, thus reducing
resource wastage caused by overestimation of loads. After the “release”
operation, the decrease in the number of saturated instances will also trigger
an asynchronous update and may increase the capacities of other functions on
the server. It means that the resources can be reutilized by the scheduler to
deploy more instances.

Logical cold start. If the load rises again and the expected number of
instances exceeds the current number of saturated instances, if there are
cached instances at this point, Jiagu triggers a “logical cold start”.
Specifically, Jiagu’s scheduler chooses a cached instance, then the router re-
routes requests to the selected instance, letting it return to a saturated
instance. This “logical cold start” is fast since the re-routing only takes
$<1ms$, while a real instance initialization is far more costly.

On-demand migration. If the number of saturated instances of a function on a
server drops and new instances of other functions are deployed to the server,
the function’s capacity could be updated and reduced. Thus, after a while, the
server could possibly be “full”, i.e., the conversion of the function’s cached
instances to saturated instances on that server could result in the number of
saturated instances surpassing its capacity. At this moment, if the load rises
again, Jiagu has to re-initialize instances on other feasible nodes, rather
than merely trigger “logical cold starts”. To avoid incurring such additional
“real cold starts”, Jiagu learns in advance how many cached instances are no
longer converted to saturated instances by comparing the capacity with the sum
of the numbers of both types of instances. Then, it migrates these cached
instances to other feasible servers in advance. Therefore, such additional
cold start overheads can be hidden right away and will not affect the running
of instances.

Real eviction. If the load does not rise again after the initial drop, finally
after the keep-alive duration, the cached instances are actually evicted.
After a while, if the load rises again and there are no cached instances, it
will trigger the autoscaler to initialize new instances, which involves “real
cold starts”. It is consistent with the traditional autoscaling approach.

##  6\. Implementation

We implement Jiagu based on OpenFaaS (ope, 2023a), with 4.5k+ LoCs changes.
Jiagu is also developed and discussed at Huawei Cloud.

OpenFaaS architecture. OpenFaaS is a high-level controller based on Kubernetes
(K8s), which schedules instances with the K8s scheduler and deploys instances
with K8s resources (e.g., K8s Deployment and Service (k8s, 2023b, a)). It
receives user requests from its gateway, and makes instance creation/eviction
decisions with its autoscaling logic. OpenFaaS relies on Prometheus (pro,
2023) to monitor the RPS of user requests and autoscale instances accordingly.
The autoscaler calculates the expected number of instances of a function by
dividing the overall RPS by the predefined saturated load. By default, it
applies a 60-second keep-alive duration.

Jiagu’s scheduling controller. Jiagu’s scheduling logic is implemented as a
centralized controller, which manages the model, capacity tables,and other
information required by the scheduling. It works together with a plugin for
Kubernetes scheduler in the prototype, allowing for interaction between the
scheduler and the controller and enabling the scheduler to obtain placement
decisions for incoming instances. It also implements asynchronous update,
where the scheduling results are returned first before performing the update.
Moreover, it is responsible for choosing instances for “logical scaling” for
dual-staged scaling with its scheduling algorithm.

Implementing dual-staged scaling. To implement dual-staged scaling, Jiagu adds
new rules for Prometheus to monitor RPS, and calculates the expected number of
saturated instances according to the monitored values. It applies a 45 (or 30,
this sensitivity can be configured) seconds “release” duration, i.e., if the
expected number of saturated instances decreases and last for 45 (or 30)
seconds, it notifies the router for changing its routing rules. To re-route,
the scheduling controller chooses an instance and labels it as “cached”. The
“cached” instances are excluded by the function’s corresponding K8s Service
(k8s, 2023b) so that new requests will not be routed to them. When load rises
and the expected number of saturated instances increases, it immediately
triggers a logical cold start, i.e., the scheduler chooses a cached instance
and unlabels it.

Profiling and model training. Jiagu adopts a solo-run approach motivated by
Gsight (Zhao et al., 2021) to collect the profile metrics of functions. To
profile a function, Jiagu addtionally deploys an instance to the profiling
node exclusively, executes the function by the instance under the saturated
load, and profiles the resource utilizations using tools such as perf for a
duration. Moreover, to construct and maintain a training dataset, Jiagu
further collects the performance metrics of various colocation combinations at
runtime.

User configurations for functions. How users properly configure their
functions (Mahgoub et al., 2022; Qiu et al., 2023) (e.g., configure resources
and the saturated value) is out of the scope of the paper. Even if a user can
wisely configure their functions, the traditional approach of scheduling and
autoscaling could still suffer from interference and load fluctuations (shown
in Figure 1), leading to resource under-utilization.

Node filter and management. The node filter described in Figure 5 prioritizes
worker nodes in Jiagu. For a new instance, it grants higher priority to nodes
that have deployed instances of the same function, as scheduling on those
nodes possibly follows a fast path. While the current implementation focuses
on this simple priority scheme, future work may explore extending it with
additional scheduling policies, such as placing multiple instances of the same
function on a single node to benefit from locality (Schall et al., 2022; Abdi
et al., 2023). In cases where no usable nodes are available for an instance,
Jiagu will request the addition of a new server to the cluster. Similarly, an
empty server will be evicted to optimize costs.

Hyperparameter configuration. The implementation involves configuring several
specific parameters, including the profiling time and the definition of model
convergence. Setting these parameters requires careful consideration of trade-
offs based on the actual scenario. For example, collecting more samples when
new functions arrive may result in a more accurate model but also incur higher
overhead. In our evaluation, all parameters remain consistent and aligned with
our practices in Huawei Cloud.

Performance predictability in the cloud. QoS violations are mostly
predictable, especially when using tail latency as the QoS metric, which
considers the overall performance of multiple requests over a period of time.
It is consistent with the observations in our production environment and prior
works (Mishra et al., 2017; Xu et al., 2018; Zhao et al., 2021; Li et al.,
2023). In Huawei Cloud’s production environment, Jiagu keeps monitoring the
QoS violation and adopts two ways to handle the incidental unpredictability.
First, the random forest model is naturally feasible for incremental learning,
and we continuously collect runtime performance metrics and retrain the model
periodically with the up-to-date training set in case the behavior of
functions changes. Second, if the prediction error does not converge after
several iterations, Jiagu disables overcommitment and uses traditional
conservative QoS-unaware policy to schedule the instances of the unpredictable
function on separate nodes. Jiagu can perform well and improve resource
utilization in real world.

##  7\. Evaluation

###  7.1. Methodology

Experimental setup. We use a cluster of 24 machines for evaluation. Each
machine is equipped with an Intel Xeon E5-2650 CPU (2.20GHz 48 logical cores
in total) and 128GB memory, running Ubuntu LTS 18.04 with Linux 4.15 kernel.
One machine is dedicated to OpenFaaS control plane components (gateway,
scheduler, Jiagu’s controller, etc.). The machine is also responsible for
sending requests and collecting QoS and utilization results. The other
machines are worker machines responsible for executing or profiling function
instances.

Baseline systems. We compare Jiagu with three baseline systems: Kubernetes,
Gsight (Zhao et al., 2021) and Owl (Tian et al., 2022). Kubernetes scheduler
is one of the mostly used serverless scheuduler systems in production (kub,
2023; Ope, 2023). Gsight (Zhao et al., 2021) and Owl (Tian et al., 2022)
represent state-of-the-art predictor-based and historical information-based
serverless scheduling systems. Since they do not open source the
implementation, we implement comparable prototypes of them. Moreover, we
evaluate three versions of Jiagu. Jiagu-30 and Jiagu-45 denote prototypes with
30 and 45 seconds “release duration” of dual-staged scaling respectively,
while Jiagu-NoDS disables dual-staged scaling.

Scheduling effect metrics. To evaluate the scheduling effect, we compare Jiagu
with our baseline scheduling algorithms by two metrics. First is the QoS
violation rate, which is the percentage of requests that violate QoS in all
requests to all functions. The QoS constraint is chosen to be 120% of the tail
latency when the instance is saturated and suffers no interference, consistent
with previous work (Tian et al., 2022; Zhao et al., 2021; Zhou et al., 2022;
Chen et al., 2019a) and in-production practice. In the evaluation, we aim to
achieve a QoS violation rate of less than 10% so we predict the p90 tail
latency accordingly. Second is the function density, higher density means
better resource utilization. We normalize traditional Kubernetes scheduler’s
function density to one, meaning that instances are deployed with exactly the
amount of configured resources. One of the goals of Jiagu is to increase
function density ($>$1) as much as possible while achieving an acceptable QoS
violation rate ($<$10%).

(a) Scheduling cost and model inferences.

(b) Cold start optimization with cfork.

(c) Cold start optimization with docker.

Figure 11. Performance analysis under extreme scenarios. It includes
scheduling cost, number of model inferences per schedule and cold start
latency optimizations. We normalize results and label concrete numbers (ms) in
the figure.

(a) Scheduling cost and model inferences.

(b) Cold start optimization with cfork.

(c) Cold start optimization with docker.

Figure 12. Performance analysis on real-world traces. It includes scheduling
cost, number of model inferences per schedule and cold start latency
optimizations. We normalize results and label concrete numbers (ms) in the
figure.

Workloads and traces. The workloads and traces for our evaluation are
consistent with prior work (Tian et al., 2022; Zhou et al., 2022; Zhao et al.,
2021; Chen et al., 2019a). We use six representative functions in
ServerlessBench (Yu et al., 2020) and FunctionBench (Kim and Lee, 2019) for
evaluation, including model inference (rnn), batch processing applications
(image resize, linpack), log processing, chameleon and file processing (gzip).
All functions are configured with the same amount of resources. To ensure the
robustness of our system to perturbations in the benchmark inputs, we
optionally add zero-mean Gaussian noise to the inputs.

We use various traces for evaluation. First, we use real-world traces since
they possess a similar level of complexity as in the production environment.
For each set of real-world traces, we randomly select six invocation patterns
from real-world Huawei Cloud traces and map each pattern to a function with
similar execution time. We generate a total of four different sets of real-
world traces to prevent the evaluation results from being influenced by
randomness in the traces. The four traces are also selected from real-world
traces from various regions. Second, we use specially constructed traces to
evaluate the scheduling performance in extreme scenarios.

###  7.2. Scheduling Performance Analysis

In this section, we evaluate the scheduling costs on OpenFaaS with different
traces in two aspects. First, we evaluate the average scheduling costs of the
two algorithms. Second, we simulate how Jiagu can optimize the cold start in
conjunction with one of the state-of-the-art instance initialization
optimizations, container fork (cfork) (Du et al., 2022), which can create an
instance in about 8.4ms. We also analyze the cold start with Docker (about
85.5ms). In each test, we compare Jiagu’s default version (Jiagu-45) with
Gsight, whose performance is normalized to 1.

Performance under extreme scenarios. We first analyze the extreme performance
of Jiagu using specific traces. For the best case, we use a timer trace with
only one function. It assumes that the function is invoked and instances are
scaled at a fixed frequency. The results (Figure 11-a) show that Gsight
suffers a scheduling overhead 11.9x larger than Jiagu. This is because almost
all scheduling decisions of Jiagu go through the fast path in this case.
Considering cold start latency (Figure 11-b), Gsight’s is 126.3% longer than
Jiagu when using cfork because of its costly scheduling. For the worst case,
we construct a rare trace where the function concurrencies recurrently change
from 0 and 1, so that every scheduling process will go through the slow path.
The results show that Jiagu’s performance degraded to a similar level as
Gsight.

When using Docker in both cases, the cost of instance initialization becomes
the bottleneck. We believe that as a growing number of efforts have limited
initialization overhead to $<$10ms (Wang et al., 2021; Akkus et al., 2018;
Wang et al., 2019; Du et al., 2020; Oakes et al., 2018; Jia and Witchel,
2021b; Shin et al., 2022; Du et al., 2022; Cadden et al., 2020; Shillaker and
Pietzuch, 2020), the reduction in scheduling costs will be increasingly
meaningful.

Performance with real-world traces. We then analyze Jiagu’s scheduling costs
with four real-world traces. The results (Figure 12) show that for real-world
traces, Jiagu achieves 81.0%–93.7% lower scheduling costs than Gsight. This is
because Jiagu’s scheduling policy can drastically reduce the number of model
inferences (83.8%–92.1%), so that the inference overhead is amortized over
multiple cold starts. For cold start latency, when applying Jiagu with cfork,
the reduced scheduling costs lead to 57.4%–69.3% lower cold start latency than
Gsight.

In conclusion, Jiagu’s scheduling algorithm can significantly reduce model
inference times and result in lower scheduling costs, leading to better cold
start latency.

Figure 13. Normalized function density. Density on K8s is normalized to 100%.
Jiagu achieves the highest density with all of the real-world traces.

(a) QoS violation of Trace A.

(b) Reduced cold starts.

Figure 14. QoS violations and reduced cold starts. Applications are chameleon
(Cha), gzip, image resize (Img), linpack (Lin), log processing (Log) and RNN.

###  7.3. Scheduling Effect on OpenFaaS

This section describes how Jiagu improves resource utilization while
guaranteeing QoS. We analyze Jiagu and the baseline systems by the two
metrics, function density and QoS violation rate, with the four real-world
traces. The density result is the average value during the evaluation weighted
by duration.

According to Figure 13, all QoS-aware schedulers improve function density
compared with Kubernetes (which is normalized to 1). Without dual-staged
scaling, Gsight and Jiagu (Jiagu-NoDS) achieve higher function density than
Owl. It shows Owl’s limitation of allowing only two colocated functions
prevents optimal scheduling decisions. With dual-staged scaling, Jiagu further
optimizes resource utilization. Higher sensitivity performs better in
improving the instance density. Specifically, Jiagu-30 achieves up to 54.8%
higher function density than Kubernetes, 22.0% higher than Gsight and 38.3%
higher than Owl, while the QoS violation rate is comparable with Gsight and
Jiagu-NoDS. It is because when user load drops, Jiagu can quickly utilize the
resources of the unsaturated instances. Moreover, on four traces, all
schedulers achieve an acceptable QoS violation rate of less than 10%, and the
four results are similar. Therefore, we only show the result on Trace A
(Figure 14-a).

Reducing cold starts by migration of cached instances. This section describes
how many additional cold starts can be avoided by migrating cached instances.
According to Figure 14-b, for 45 seconds sensitivity, all re-routing
operations are logical cold starts, needless to migrate instances. With
30-second sensitivity, only a small proportion of re-routing, i.e., $<20\%$
for four traces and $0$ for Trace B, requires “real” rather than “logical”
cold starts, which can be avoided by migrating cached instances in advance. It
illustrates that with dual-staged scaling, Jiagu has the additional choice of
greatly improving resource utilization at minimum cost.

In conclusion, with the prediction-based scheduler and dual-staged scaling,
Jiagu can achieve high resource utilization while limiting QoS violations with
less scheduling overhead than state-of-the-art model-based scheduling policy.

Figure 15. Model accuracy. (a) is the error rate of the prediction models. (b)
shows that the prediction error drops with new samples.

Figure 16. Prediction errors of various models.

Figure 17. Model performance. (a) is the training time and the total number of
dimensions of the two models. (b) Jiagu’s model inference overhead with the
number of inputs increases.

###  7.4. Prediction Analysis

Prediction accuracy. We evaluate the prediction accuracy of our model, as
shown in Figure 15. We define the prediction error rate of a model as
$\frac{|\hat{P}-P|}{P}$, where $\hat{P}$ and $P$ are predicted performances
and real performances, respectively. First, we analyze the accuracy of the
model used by previous evaluations. The model could accurately predict the
performance of the six functions, with similar prediction error to Gsight (Jg
and Gs in the Figure). Second, we evaluate the overfitting of the model by
splitting the test set into two equal-sized sets (Jg-1 and Jg-2). The model
achieves similar prediction error on the two different sets, showing that it
does not overfit to a specific test set. Third, we evaluate the model’s
scalability, i.e., how the model can predict the performance of increasing
number of functions (30 and 60 functions). The result shows that more
functions do not affect the prediction precision. Fourth, we evaluate how the
model is resilient to the changes in function’s behavior. Each time, we train
the model with five functions, and add the remaining function to the system,
collect performance metrics as new samples and retrain the model once a sample
is added. Figure 15-b shows that the performance prediction error of the new
function drops rapidly under increasing number of samples, and converges after
about 5–30 samples. It means that it takes limited time to retrain an accurate
model after collecting a couple of samples.

Finally, we evaluate the model choices, comparing Jiagu’s random forest
regression (RFR) with ESP (Mishra et al., 2017), XGBoost, linear regression,
and three multi-layer perception (MLP) models with 2,3,4 layers respectively.
The result (Figure 16) confirms our choice of RFR model considering its high
accuracy, low training overhead and capability of incremental learning, etc.
Possible other models or algorithms (e.g., ridge regression) can be explored
to improve Jiagu in future work.

Model performance. We evaluate Jiagu’s model training and inference costs.
Figure 17-a shows that Jiagu’s model has evidently better training performance
and lower number of input dimensions. Moreover, Figure 17-b demonstrates that
the model inference costs increase only 2 milliseconds even when the number of
inputs increases to 100. Therefore, although Jiagu batches multiple inputs to
the prediction model when calculating the capacity, it will not significantly
increase the inference overhead.

In conclusion, Jiagu’s model can predict the performance accurately with a
much lower number of input dimensions and training overhead.

##  8\. Related Work

Resource schedulers. Another widely studied (and orthogonal) scheduler is the
online resource scheduler (Bhardwaj et al., 2023; Rzadca et al., 2020; Gan et
al., 2021; Li et al., 2021; Zhang et al., 2014; Yang et al., 2013; Chen et
al., 2019b; Saha and Jindal, 2018; Zhang et al., 2013; Gan et al., 2019; Lo et
al., 2015; Chen et al., 2019a; Patel and Tiwari, 2020; Qiu et al., 2020; Zhang
et al., 2021c; Fried et al., 2020; Romero et al., 2021; Maji et al., 2014;
Akhtar et al., 2020; Zhou et al., 2022; Wang et al., 2022; Delimitrou et al.,
2015; Delimitrou and Kozyrakis, 2014; Novak et al., 2019, 2020; Chen et al.,
2022; Kulkarni et al., 2020), which tunes the allocated resources according to
functions’ online performance to guarantee QoS and improve resource
efficiency. Many of them are designed for long running services (Rzadca et
al., 2020; Gan et al., 2021; Zhang et al., 2014; Yang et al., 2013; Chen et
al., 2019b; Zhang et al., 2013; Gan et al., 2019; Lo et al., 2015; Chen et
al., 2019a; Patel and Tiwari, 2020; Qiu et al., 2020; Zhang et al., 2021c;
Fried et al., 2020; Romero et al., 2021; Maji et al., 2014; Wang et al.,
2022), which are challenging to apply directly to transient serverless
instances. In addition, some serverless resource schedulers (Akhtar et al.,
2020; Zhou et al., 2022; Saha and Jindal, 2018) use algorithms like Bayesian
Optimization to search the resource configuration space and make optimal
decisions based on runtime performance. They are resilient to runtime
performance variations but still fail to guarantee QoS before convergence.

Request schedulers. Rather than scheduling instances, many prior works tackle
the problem of assigning instances for requests. Atoll (Singhvi et al., 2021)
overcomes many latency challenges in the serverless platform via a ground-up
redesign of system control and data planes, utilizing deadline-aware
scheduling. Hermod (Kaffes et al., 2022) analyzes and tunes different design
choices on OpenWhisk and improves function performance. Fifer (Gunasekaran et
al., 2020) queues requests to warm containers to reduce the number of
instances and prewarms instances with LSTM resource interference to optimize
the cold starts. Sequoia (Tariq et al., 2020) is deployed as a proxy to
control user requests and enabling QoS. Neither of them considers resource
interference. Golgi (Li et al., 2023) applies a model for the router,
predicting requests’ performance and sending them to overcommitted instances
if the performance does not violate QoS, or otherwise to non-overcommitted
instances. It can avoid QoS violation for a fixed number of instances, but
does not guide for how many overcommitted instances need to be deployed, while
excessive number of overcommitted instances could cause resource under-
utilization.

Other optimizations. There are many related serverless optimizations (Mahgoub
et al., 2022; Roy et al., 2022; Li et al., 2022; Klimovic et al., 2018; Romero
et al., 2021; Tariq et al., 2020; Sreekanti et al., 2020; Jia and Witchel,
2021a). For example, Orion (Mahgoub et al., 2022) proposes a system optimized
for serverless DAG by proposing a new performance model and a method for
colocating parallel invocations. FaasFlow (Li et al., 2022) proposes a worker-
side workflow schedule pattern that enables better scalability and
performance. All of these systems can benefit from Jiagu with a better and
more effective scheduler to also improve the resource utilization for cloud
vendors without introducing high scheduling costs.

##  9\. Conclusion

This paper presents our experience in optimizing resource utilization in
Huawei Cloud for serverless computing with Jiagu. Jiagu decouples prediction
and decision making to achieve efficient and fast scheduling, and decouples
resource releasing and instance eviction to achieve fast reaction to load
fluctuation without incurring overmuch cold start overhead. Our results show
Jiagu brings significant benefits for resource efficiency using real-world
traces.

## References

  * (1)
  * Ope (2023) 2023\.  Apache OpenWhisk is a serverless, open source cloud platform.  http://openwhisk.apache.org/.  Referenced 2023.
  * fle (2023) 2023\.  Cloud computing trends and statistics: Flexera 2023 State of the Cloud Report.  https://www.flexera.com/blog/cloud/cloud-computing-trends-flexera-2023-state-of-the-cloud-report/.
  * k8s (2023a) 2023a.  Deployments — Kubernetes.  https://kubernetes.io/docs/concepts/services-networking/service/.
  * gar (2023) 2023\.  Gartner Says Worldwide IaaS Public Cloud Services Market Grew 41.4% in 2021.  https://www.gartner.com/en/newsroom/press-releases/2022-06-02-gartner-says-worldwide-iaas-public-cloud-services-market-grew-41-percent-in-2021.
  * kna (2023) 2023\.  Knative Autoscaling.  https://knative.dev/docs/serving/autoscaling/.
  * kub (2023) 2023\.  Kubernetes Scheduler.  https://kubernetes.io/docs/concepts/scheduling-eviction/kube-scheduler/.
  * ope (2023a) 2023a.  OpenFaaS - Serverless Functions Made Simple.  https://www.openfaas.com.
  * ope (2023b) 2023b.  Openfaas Autoscaling.  https://docs.openfaas.com/architecture/autoscaling/.
  * pro (2023) 2023\.  Prometheus - Monitoring system and time series database.  https://prometheus.io/.
  * sk- (2023) 2023\.  scikit-learn: machine learning in Python.  https://scikit-learn.org/.
  * k8s (2023b) 2023b.  Service — Kubernetes.  https://kubernetes.io/docs/concepts/services-networking/service/.
  * Abdi et al. (2023) Mania Abdi, Samuel Ginzburg, Xiayue Charles Lin, Jose Faleiro, Gohar Irfan Chaudhry, Inigo Goiri, Ricardo Bianchini, Daniel S Berger, and Rodrigo Fonseca. 2023\.  Palette Load Balancing: Locality Hints for Serverless Functions. In Proceedings of the Eighteenth European Conference on Computer Systems (Rome, Italy) (EuroSys ’23). Association for Computing Machinery, New York, NY, USA, 365–380.  https://doi.org/10.1145/3552326.3567496
  * Akhtar et al. (2020) Nabeel Akhtar, Ali Raza, Vatche Ishakian, and Ibrahim Matta. 2020\.  COSE: Configuring Serverless Functions using Statistical Learning. In IEEE INFOCOM 2020 - IEEE Conference on Computer Communications. 129–138.  https://doi.org/10.1109/INFOCOM41043.2020.9155363
  * Akkus et al. (2018) Istemi Ekin Akkus, Ruichuan Chen, Ivica Rimac, Manuel Stein, Klaus Satzke, Andre Beck, Paarijaat Aditya, and Volker Hilt. 2018.  $SAND$: Towards High-Performance Serverless Computing. In 2018 $USENIX$ Annual Technical Conference ($USENIX$ $ATC$ 18). 923–935.
  * Bellman (2003) R.E. Bellman. 2003\.  Dynamic Programming.  Dover Publications.  https://books.google.com.hk/books?id=fyVtp3EMxasC
  * Bhardwaj et al. (2023) Romil Bhardwaj, Kirthevasan Kandasamy, Asim Biswal, Wenshuo Guo, Benjamin Hindman, Joseph Gonzalez, Michael Jordan, and Ion Stoica. 2023.  Cilantro: Performance-Aware Resource Allocation for General Objectives via Online Feedback. In 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23). USENIX Association, Boston, MA, 623–643.  https://www.usenix.org/conference/osdi23/presentation/bhardwaj
  * Brooker (2022) Marc Brooker. 2022\.  Lambda Snapstart, and snapshots as a tool for system builders.  https://brooker.co.za/blog/2022/11/29/snapstart.html.  Referenced Dec 2022.
  * Cadden et al. (2020) James Cadden, Thomas Unger, Yara Awad, Han Dong, Orran Krieger, and Jonathan Appavoo. 2020\.  SEUSS: skip redundant paths to make serverless fast. In Proceedings of the Fifteenth European Conference on Computer Systems. 1–15.
  * Chen et al. (2019b) Quan Chen, Zhenning Wang, Jingwen Leng, Chao Li, Wenli Zheng, and Minyi Guo. 2019b.  Avalon: Towards QoS Awareness and Improved Utilization through Multi-Resource Management in Datacenters. In Proceedings of the ACM International Conference on Supercomputing (Phoenix, Arizona) (ICS ’19). Association for Computing Machinery, New York, NY, USA, 272–283.  https://doi.org/10.1145/3330345.3330370
  * Chen et al. (2019a) Shuang Chen, Christina Delimitrou, and José F. Martínez. 2019a.  PARTIES: QoS-Aware Resource Partitioning for Multiple Interactive Services. In Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems (Providence, RI, USA) (ASPLOS ’19). Association for Computing Machinery, New York, NY, USA, 107–120.  https://doi.org/10.1145/3297858.3304005
  * Chen et al. (2022) Shuang Chen, Angela Jin, Christina Delimitrou, and José F. Martínez. 2022.  ReTail: Opting for Learning Simplicity to Enable QoS-Aware Power Management in the Cloud. In 2022 IEEE International Symposium on High-Performance Computer Architecture (HPCA). 155–168.  https://doi.org/10.1109/HPCA53966.2022.00020
  * Delimitrou and Kozyrakis (2013) Christina Delimitrou and Christos Kozyrakis. 2013.  Paragon: QoS-Aware Scheduling for Heterogeneous Datacenters. In Proceedings of the Eighteenth International Conference on Architectural Support for Programming Languages and Operating Systems (Houston, Texas, USA) (ASPLOS ’13). Association for Computing Machinery, New York, NY, USA, 77–88.  https://doi.org/10.1145/2451116.2451125
  * Delimitrou and Kozyrakis (2014) Christina Delimitrou and Christos Kozyrakis. 2014.  Quasar: Resource-Efficient and QoS-Aware Cluster Management. In Proceedings of the 19th International Conference on Architectural Support for Programming Languages and Operating Systems (Salt Lake City, Utah, USA) (ASPLOS ’14). Association for Computing Machinery, New York, NY, USA, 127–144.  https://doi.org/10.1145/2541940.2541941
  * Delimitrou et al. (2015) Christina Delimitrou, Daniel Sanchez, and Christos Kozyrakis. 2015.  Tarcil: Reconciling Scheduling Speed and Quality in Large Shared Clusters. In Proceedings of the Sixth ACM Symposium on Cloud Computing (Kohala Coast, Hawaii) (SoCC ’15). Association for Computing Machinery, New York, NY, USA, 97–110.  https://doi.org/10.1145/2806777.2806779
  * Du et al. (2022) Dong Du, Qingyuan Liu, Xueqiang Jiang, Yubin Xia, Binyu Zang, and Haibo Chen. 2022\.  Serverless Computing on Heterogeneous Computers. In Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (Lausanne, Switzerland) (ASPLOS 2022). Association for Computing Machinery, New York, NY, USA, 797–813.  https://doi.org/10.1145/3503222.3507732
  * Du et al. (2020) Dong Du, Tianyi Yu, Yubin Xia, Binyu Zang, Guanglu Yan, Chenggang Qin, Qixuan Wu, and Haibo Chen. 2020\.  Catalyzer: Sub-Millisecond Startup for Serverless Computing with Initialization-Less Booting. In Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems (Lausanne, Switzerland) (ASPLOS ’20). Association for Computing Machinery, New York, NY, USA, 467–481.  https://doi.org/10.1145/3373376.3378512
  * Fried et al. (2020) Joshua Fried, Zhenyuan Ruan, Amy Ousterhout, and Adam Belay. 2020\.  Caladan: Mitigating Interference at Microsecond Timescales.  USENIX Association, USA.
  * Fuerst and Sharma (2021) Alexander Fuerst and Prateek Sharma. 2021.  FaasCache: keeping serverless computing alive with greedy-dual caching. In Proceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems. 386–400.
  * Gan et al. (2021) Yu Gan, Mingyu Liang, Sundar Dev, David Lo, and Christina Delimitrou. 2021.  Sage: Practical and Scalable ML-Driven Performance Debugging in Microservices. In Proceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (Virtual, USA) (ASPLOS ’21). Association for Computing Machinery, New York, NY, USA, 135–151.  https://doi.org/10.1145/3445814.3446700
  * Gan et al. (2019) Yu Gan, Yanqi Zhang, Kelvin Hu, Dailun Cheng, Yuan He, Meghna Pancholi, and Christina Delimitrou. 2019.  Seer: Leveraging Big Data to Navigate the Complexity of Performance Debugging in Cloud Microservices. In Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems (Providence, RI, USA) (ASPLOS ’19). Association for Computing Machinery, New York, NY, USA, 19–33.  https://doi.org/10.1145/3297858.3304004
  * Gunasekaran et al. (2020) Jashwant Raj Gunasekaran, Prashanth Thinakaran, Nachiappan C. Nachiappan, Mahmut Taylan Kandemir, and Chita R. Das. 2020.  Fifer: Tackling Resource Underutilization in the Serverless Era. In Proceedings of the 21st International Middleware Conference (Delft, Netherlands) (Middleware ’20). Association for Computing Machinery, New York, NY, USA, 280–295.  https://doi.org/10.1145/3423211.3425683
  * Jia and Witchel (2021a) Zhipeng Jia and Emmett Witchel. 2021a.  Boki: Stateful Serverless Computing with Shared Logs. In Proceedings of the ACM SIGOPS 28th Symposium on Operating Systems Principles (Virtual Event, Germany) (SOSP ’21). Association for Computing Machinery, New York, NY, USA, 691–707.  https://doi.org/10.1145/3477132.3483541
  * Jia and Witchel (2021b) Zhipeng Jia and Emmett Witchel. 2021b.  Nightcore: Efficient and Scalable Serverless Computing for Latency-Sensitive, Interactive Microservices. In Proceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (Virtual, USA) (ASPLOS 2021). Association for Computing Machinery, New York, NY, USA, 152–166.  https://doi.org/10.1145/3445814.3446701
  * Jonas et al. (2019) Eric Jonas, Johann Schleier-Smith, Vikram Sreekanti, Chia-Che Tsai, Anurag Khandelwal, Qifan Pu, Vaishaal Shankar, Joao Carreira, Karl Krauth, Neeraja Yadwadkar, et al. 2019\.  Cloud programming simplified: A berkeley view on serverless computing.  arXiv preprint arXiv:1902.03383 (2019).
  * Kaffes et al. (2022) Kostis Kaffes, Neeraja J. Yadwadkar, and Christos Kozyrakis. 2022\.  Hermod: Principled and Practical Scheduling for Serverless Functions. In Proceedings of the 13th Symposium on Cloud Computing (San Francisco, California) (SoCC ’22). Association for Computing Machinery, New York, NY, USA, 289–305.  https://doi.org/10.1145/3542929.3563468
  * Kim and Lee (2019) Jeongchul Kim and Kyungyong Lee. 2019.  Practical cloud workloads for serverless faas. In Proceedings of the ACM Symposium on Cloud Computing. 477–477.
  * Klimovic et al. (2018) Ana Klimovic, Yawen Wang, Patrick Stuedi, Animesh Trivedi, Jonas Pfefferle, and Christos Kozyrakis. 2018.  Pocket: Elastic Ephemeral Storage for Serverless Analytics. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18). USENIX Association, Carlsbad, CA, 427–444.  https://www.usenix.org/conference/osdi18/presentation/klimovic
  * Kulkarni et al. (2020) Neeraj Kulkarni, Gonzalo Gonzalez-Pumariega, Amulya Khurana, Christine Shoemaker, Christina Delimitrou, and David Albonesi. 2020\.  CuttleSys: Data-Driven Resource Management for Interactive Applications on Reconfigurable Multicores. In 53rd IEEE/ACM International Symposium on Microarchitecture (MICRO).
  * Li et al. (2021) Qian Li, Bin Li, Pietro Mercati, Ramesh Illikkal, Charlie Tai, Michael Kishinevsky, and Christos Kozyrakis. 2021.  RAMBO: Resource Allocation for Microservices Using Bayesian Optimization.  IEEE Computer Architecture Letters 20, 1 (2021), 46–49.  https://doi.org/10.1109/LCA.2021.3066142
  * Li et al. (2023) Suyi Li, Wei Wang, Jun Yang, Guangzhen Chen, and Daohe Lu. 2023.  Golgi: Performance-Aware, Resource-Efficient Function Scheduling for Serverless Computing. In Proceedings of the 2023 ACM Symposium on Cloud Computing (¡conf-loc¿, ¡city¿Santa Cruz¡/city¿, ¡state¿CA¡/state¿, ¡country¿USA¡/country¿, ¡/conf-loc¿) (SoCC ’23). Association for Computing Machinery, New York, NY, USA, 32–47.  https://doi.org/10.1145/3620678.3624645
  * Li et al. (2022) Zijun Li, Yushi Liu, Linsong Guo, Quan Chen, Jiagan Cheng, Wenli Zheng, and Minyi Guo. 2022.  FaaSFlow: Enable Efficient Workflow Execution for Function-as-a-Service. In Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (Lausanne, Switzerland) (ASPLOS ’22). Association for Computing Machinery, New York, NY, USA, 782–796.  https://doi.org/10.1145/3503222.3507717
  * Liu et al. (2023) Qingyuan Liu, Dong Du, Yubin Xia, Ping Zhang, and Haibo Chen. 2023.  The Gap Between Serverless Research and Real-World Systems. In Proceedings of the 2023 ACM Symposium on Cloud Computing (, Santa Cruz, CA, USA,) (SoCC ’23). Association for Computing Machinery, New York, NY, USA, 475–485.  https://doi.org/10.1145/3620678.3624785
  * Lo et al. (2015) David Lo, Liqun Cheng, Rama Govindaraju, Parthasarathy Ranganathan, and Christos Kozyrakis. 2015\.  Heracles: Improving Resource Efficiency at Scale. In Proceedings of the 42nd Annual International Symposium on Computer Architecture (Portland, Oregon) (ISCA ’15). Association for Computing Machinery, New York, NY, USA, 450–462.  https://doi.org/10.1145/2749469.2749475
  * Mahgoub et al. (2022) Ashraf Mahgoub, Edgardo Barsallo Yi, Karthick Shankar, Sameh Elnikety, Somali Chaterji, and Saurabh Bagchi. 2022.  ORION and the Three Rights: Sizing, Bundling, and Prewarming for Serverless DAGs. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). USENIX Association, Carlsbad, CA, 303–320.  https://www.usenix.org/conference/osdi22/presentation/mahgoub
  * Maji et al. (2014) Amiya K. Maji, Subrata Mitra, Bowen Zhou, Saurabh Bagchi, and Akshat Verma. 2014.  Mitigating Interference in Cloud Services by Middleware Reconfiguration. In Proceedings of the 15th International Middleware Conference (Bordeaux, France) (Middleware ’14). Association for Computing Machinery, New York, NY, USA, 277–288.  https://doi.org/10.1145/2663165.2663330
  * Mars and Tang (2013) Jason Mars and Lingjia Tang. 2013.  Whare-Map: Heterogeneity in ”Homogeneous” Warehouse-Scale Computers. In Proceedings of the 40th Annual International Symposium on Computer Architecture (Tel-Aviv, Israel) (ISCA ’13). Association for Computing Machinery, New York, NY, USA, 619–630.  https://doi.org/10.1145/2485922.2485975
  * Mars et al. (2011) Jason Mars, Lingjia Tang, Robert Hundt, Kevin Skadron, and Mary Lou Soffa. 2011.  Bubble-Up: Increasing Utilization in Modern Warehouse Scale Computers via Sensible Co-Locations. In Proceedings of the 44th Annual IEEE/ACM International Symposium on Microarchitecture (Porto Alegre, Brazil) (MICRO-44). Association for Computing Machinery, New York, NY, USA, 248–259.  https://doi.org/10.1145/2155620.2155650
  * Mishra et al. (2017) Nikita Mishra, John D. Lafferty, and Henry Hoffmann. 2017\.  ESP: A Machine Learning Approach to Predicting Application Interference. In 2017 IEEE International Conference on Autonomic Computing (ICAC). 125–134.  https://doi.org/10.1109/ICAC.2017.29
  * Novak et al. (2020) Joe Novak, Sneha Kumar Kasera, and Ryan Stutsman. 2020\.  Auto-Scaling Cloud-Based Memory-Intensive Applications. In 2020 IEEE 13th International Conference on Cloud Computing (CLOUD). 229–237.  https://doi.org/10.1109/CLOUD49709.2020.00042
  * Novak et al. (2019) Joe H. Novak, Sneha Kumar Kasera, and Ryan Stutsman. 2019\.  Cloud Functions for Fast and Robust Resource Auto-Scaling. In 2019 11th International Conference on Communication Systems & Networks (COMSNETS). 133–140.  https://doi.org/10.1109/COMSNETS.2019.8711058
  * Oakes et al. (2018) Edward Oakes, Leon Yang, Dennis Zhou, Kevin Houck, Tyler Harter, Andrea Arpaci-Dusseau, and Remzi Arpaci-Dusseau. 2018\.  $SOCK$: Rapid Task Provisioning with Serverless-Optimized Containers. In 2018 $USENIX$ Annual Technical Conference ($USENIX$ $ATC$ 18). 57–70.
  * Patel and Tiwari (2020) Tirthak Patel and Devesh Tiwari. 2020.  CLITE: Efficient and QoS-Aware Co-Location of Multiple Latency-Critical Jobs for Warehouse Scale Computers. In 2020 IEEE International Symposium on High Performance Computer Architecture (HPCA). 193–206.  https://doi.org/10.1109/HPCA47549.2020.00025
  * Qiu et al. (2020) Haoran Qiu, Subho S. Banerjee, Saurabh Jha, Zbigniew T. Kalbarczyk, and Ravishankar K. Iyer. 2020\.  FIRM: An Intelligent Fine-Grained Resource Management Framework for SLO-Oriented Microservices.  USENIX Association, USA.
  * Qiu et al. (2023) Haoran Qiu, Weichao Mao, Chen Wang, Hubertus Franke, Alaa Youssef, Zbigniew T. Kalbarczyk, Tamer Başar, and Ravishankar K. Iyer. 2023.  AWARE: Automate Workload Autoscaling with Reinforcement Learning in Production Cloud Systems. In 2023 USENIX Annual Technical Conference (USENIX ATC 23). USENIX Association, Boston, MA, 387–402.  https://www.usenix.org/conference/atc23/presentation/qiu-haoran
  * Romero et al. (2021) Francisco Romero, Mark Zhao, Neeraja J. Yadwadkar, and Christos Kozyrakis. 2021.  Llama: A Heterogeneous & Serverless Framework for Auto-Tuning Video Analytics Pipelines. In Proceedings of the ACM Symposium on Cloud Computing (Seattle, WA, USA) (SoCC ’21). Association for Computing Machinery, New York, NY, USA, 1–17.  https://doi.org/10.1145/3472883.3486972
  * Roy et al. (2022) Rohan Basu Roy, Tirthak Patel, and Devesh Tiwari. 2022\.  IceBreaker: Warming Serverless Functions Better with Heterogeneity. In Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (Lausanne, Switzerland) (ASPLOS ’22). Association for Computing Machinery, New York, NY, USA, 753–767.  https://doi.org/10.1145/3503222.3507750
  * Rzadca et al. (2020) Krzysztof Rzadca, Pawel Findeisen, Jacek Swiderski, Przemyslaw Zych, Przemyslaw Broniek, Jarek Kusmierek, Pawel Nowak, Beata Strack, Piotr Witusowski, Steven Hand, and John Wilkes. 2020.  Autopilot: Workload Autoscaling at Google. In Proceedings of the Fifteenth European Conference on Computer Systems (Heraklion, Greece) (EuroSys ’20). Association for Computing Machinery, New York, NY, USA, Article 16, 16 pages.  https://doi.org/10.1145/3342195.3387524
  * Saha and Jindal (2018) Aakanksha Saha and Sonika Jindal. 2018.  EMARS: Efficient Management and Allocation of Resources in Serverless. In 2018 IEEE 11th International Conference on Cloud Computing (CLOUD). 827–830.  https://doi.org/10.1109/CLOUD.2018.00113
  * Schall et al. (2022) David Schall, Artemiy Margaritov, Dmitrii Ustiugov, Andreas Sandberg, and Boris Grot. 2022\.  Lukewarm Serverless Functions: Characterization and Optimization. In Proceedings of the 49th Annual International Symposium on Computer Architecture (New York, New York) (ISCA ’22). Association for Computing Machinery, New York, NY, USA, 757–770.  https://doi.org/10.1145/3470496.3527390
  * Shahrad et al. (2020) Mohammad Shahrad, Rodrigo Fonseca, Íñigo Goiri, Gohar Chaudhry, Paul Batum, Jason Cooke, Eduardo Laureano, Colby Tresness, Mark Russinovich, and Ricardo Bianchini. 2020.  Serverless in the wild: Characterizing and optimizing the serverless workload at a large cloud provider. In 2020 $USENIX$ Annual Technical Conference ($USENIX$ $ATC$ 20). 205–218.
  * Shillaker and Pietzuch (2020) Simon Shillaker and Peter Pietzuch. 2020.  Faasm: lightweight isolation for efficient stateful serverless computing. In 2020 $USENIX$ Annual Technical Conference ($USENIX$ $ATC$ 20). 419–433.
  * Shin et al. (2022) Wonseok Shin, Wook-Hee Kim, and Changwoo Min. 2022.  Fireworks: A Fast, Efficient, and Safe Serverless Framework Using VM-Level Post-JIT Snapshot. In Proceedings of the Seventeenth European Conference on Computer Systems (Rennes, France) (EuroSys ’22). Association for Computing Machinery, New York, NY, USA, 663–677.  https://doi.org/10.1145/3492321.3519581
  * Singhvi et al. (2021) Arjun Singhvi, Arjun Balasubramanian, Kevin Houck, Mohammed Danish Shaikh, Shivaram Venkataraman, and Aditya Akella. 2021\.  Atoll: A Scalable Low-Latency Serverless Platform. In Proceedings of the ACM Symposium on Cloud Computing (Seattle, WA, USA) (SoCC ’21). Association for Computing Machinery, New York, NY, USA, 138–152.  https://doi.org/10.1145/3472883.3486981
  * Sreekanti et al. (2020) Vikram Sreekanti, Chenggang Wu, Xiayue Charles Lin, Johann Schleier-Smith, Joseph E. Gonzalez, Joseph M. Hellerstein, and Alexey Tumanov. 2020.  Cloudburst: Stateful Functions-as-a-Service.  Proc. VLDB Endow. 13, 12 (sep 2020), 2438–2452.  https://doi.org/10.14778/3407790.3407836
  * Tariq et al. (2020) Ali Tariq, Austin Pahl, Sharat Nimmagadda, Eric Rozner, and Siddharth Lanka. 2020.  Sequoia: Enabling Quality-of-Service in Serverless Computing. In Proceedings of the 11th ACM Symposium on Cloud Computing (Virtual Event, USA) (SoCC ’20). Association for Computing Machinery, New York, NY, USA, 311–327.  https://doi.org/10.1145/3419111.3421306
  * Tian et al. (2022) Huangshi Tian, Suyi Li, Ao Wang, Wei Wang, Tianlong Wu, and Haoran Yang. 2022\.  Owl: Performance-Aware Scheduling for Resource-Efficient Function-as-a-Service Cloud. In Proceedings of the 13th Symposium on Cloud Computing (San Francisco, California) (SoCC ’22). Association for Computing Machinery, New York, NY, USA, 78–93.  https://doi.org/10.1145/3542929.3563470
  * Wang et al. (2021) Ao Wang, Shuai Chang, Huangshi Tian, Hongqi Wang, Haoran Yang, Huiba Li, Rui Du, and Yue Cheng. 2021\.  FaaSNet: Scalable and Fast Provisioning of Custom Serverless Container Runtimes at Alibaba Cloud Function Compute. In 2021 USENIX Annual Technical Conference, USENIX ATC 2021, July 14-16, 2021, Irina Calciu and Geoff Kuenning (Eds.). USENIX Association, 443–457.  https://www.usenix.org/conference/atc21/presentation/wang-ao
  * Wang et al. (2019) Kai-Ting Amy Wang, Rayson Ho, and Peng Wu. 2019.  Replayable execution optimized for page sharing for a managed runtime environment. In Proceedings of the Fourteenth EuroSys Conference 2019. 1–16.
  * Wang et al. (2022) Ziliang Wang, Shiyi Zhu, Jianguo Li, Wei Jiang, K. K. Ramakrishnan, Yangfei Zheng, Meng Yan, Xiaohong Zhang, and Alex X. Liu. 2022.  DeepScaling: Microservices Autoscaling for Stable CPU Utilization in Large Scale Cloud Systems. In Proceedings of the 13th Symposium on Cloud Computing (San Francisco, California) (SoCC ’22). Association for Computing Machinery, New York, NY, USA, 16–30.  https://doi.org/10.1145/3542929.3563469
  * Xu et al. (2018) Ran Xu, Subrata Mitra, Jason Rahman, Peter Bai, Bowen Zhou, Greg Bronevetsky, and Saurabh Bagchi. 2018.  Pythia: Improving Datacenter Utilization via Precise Contention Prediction for Multiple Co-Located Workloads. In Proceedings of the 19th International Middleware Conference (Rennes, France) (Middleware ’18). Association for Computing Machinery, New York, NY, USA, 146–160.  https://doi.org/10.1145/3274808.3274820
  * Yang et al. (2013) Hailong Yang, Alex Breslow, Jason Mars, and Lingjia Tang. 2013.  Bubble-Flux: Precise Online QoS Management for Increased Utilization in Warehouse Scale Computers.  SIGARCH Comput. Archit. News 41, 3 (jun 2013), 607–618.  https://doi.org/10.1145/2508148.2485974
  * Yu et al. (2020) Tianyi Yu, Qingyuan Liu, Dong Du, Yubin Xia, Binyu Zang, Ziqian Lu, Pingchao Yang, Chenggang Qin, and Haibo Chen. 2020.  Characterizing Serverless Platforms with ServerlessBench. In Proceedings of the ACM Symposium on Cloud Computing (SoCC ’20). Association for Computing Machinery.  https://doi.org/10.1145/3419111.3421280
  * Zhang et al. (2023) Hong Zhang, Yupeng Tang, Anurag Khandelwal, and Ion Stoica. 2023\.  SHEPHERD: Serving DNNs in the Wild. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23). USENIX Association, Boston, MA, 787–808.  https://www.usenix.org/conference/nsdi23/presentation/zhang-hong
  * Zhang et al. (2013) Xiao Zhang, Eric Tune, Robert Hagmann, Rohit Jnagal, Vrigo Gokhale, and John Wilkes. 2013\.  CPI2: CPU Performance Isolation for Shared Compute Clusters. In Proceedings of the 8th ACM European Conference on Computer Systems (Prague, Czech Republic) (EuroSys ’13). Association for Computing Machinery, New York, NY, USA, 379–391.  https://doi.org/10.1145/2465351.2465388
  * Zhang et al. (2021a) Yanqi Zhang, Íñigo Goiri, Gohar Irfan Chaudhry, Rodrigo Fonseca, Sameh Elnikety, Christina Delimitrou, and Ricardo Bianchini. 2021a.  Faster and Cheaper Serverless Computing on Harvested Resources. In Proceedings of the ACM SIGOPS 28th Symposium on Operating Systems Principles (Virtual Event, Germany) (SOSP ’21). Association for Computing Machinery, New York, NY, USA, 724–739.  https://doi.org/10.1145/3477132.3483580
  * Zhang et al. (2021b) Yanqi Zhang, Weizhe Hua, Zhuangzhuang Zhou, G Edward Suh, and Christina Delimitrou. 2021b.  Sinan: ML-based and QoS-aware resource management for cloud microservices. In Proceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems. 167–181.
  * Zhang et al. (2021c) Yanqi Zhang, Weizhe Hua, Zhuangzhuang Zhou, G. Edward Suh, and Christina Delimitrou. 2021c.  Sinan: ML-Based and QoS-Aware Resource Management for Cloud Microservices. In Proceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (Virtual, USA) (ASPLOS 2021). Association for Computing Machinery, New York, NY, USA, 167–181.  https://doi.org/10.1145/3445814.3446693
  * Zhang et al. (2014) Yunqi Zhang, Michael A. Laurenzano, Jason Mars, and Lingjia Tang. 2014.  SMiTe: Precise QoS Prediction on Real-System SMT Processors to Improve Utilization in Warehouse Scale Computers. In Proceedings of the 47th Annual IEEE/ACM International Symposium on Microarchitecture (Cambridge, United Kingdom) (MICRO-47). IEEE Computer Society, USA, 406–418.  https://doi.org/10.1109/MICRO.2014.53
  * Zhao et al. (2021) Laiping Zhao, Yanan Yang, Yiming Li, Xian Zhou, and Keqiu Li. 2021.  Understanding, Predicting and Scheduling Serverless Workloads under Partial Interference. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (St. Louis, Missouri) (SC ’21). Association for Computing Machinery, New York, NY, USA, Article 22, 15 pages.  https://doi.org/10.1145/3458817.3476215
  * Zhou et al. (2022) Zhuangzhuang Zhou, Yanqi Zhang, and Christina Delimitrou. 2022\.  AQUATOPE: QoS-and-Uncertainty-Aware Resource Management for Multi-Stage Serverless Workflows. In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1 (Vancouver, BC, Canada) (ASPLOS 2023). Association for Computing Machinery, New York, NY, USA, 1–14.  https://doi.org/10.1145/3567955.3567960