# Why does Prediction Accuracy Decrease over Time?
Uncertain Positive Learning for Cloud Failure Prediction

Haozhe Li*, Minghua Ma, Yudong Liu, Pu Zhao, Lingling Zheng, Ze Li, Yingnong
Dang,
Murali Chintalapati, Saravan Rajmohan, Qingwei Lin, Dongmei Zhang  Microsoft

(2023)

###### Abstract.

With the rapid growth of cloud computing, a variety of software services have
been deployed in the cloud. To ensure the reliability of cloud services, prior
studies focus on failure instance (disk, node, and switch, etc.) prediction.
Once the output of prediction is positive, mitigation actions are taken to
rapidly resolve the underlying failure. According to our real-world practice
in Microsoft Azure, we find that the prediction accuracy may decrease by about
$9\%$ after retraining the models. Considering that the mitigation actions may
result in uncertain positive instances since they cannot be verified after
mitigation, which may introduce more noise while updating the prediction
model. To the best of our knowledge, we are the first to identify this
Uncertain Positive Learning (UPLearning) issue in the real-world cloud failure
prediction scenario. To tackle this problem, we design an Uncertain Positive
Learning Risk Estimator (Uptake) approach. Using two real-world datasets of
disk failure prediction and conducting node prediction experiments in
Microsoft Azure, which is a top-tier cloud provider that serves millions of
users, we demonstrate Uptake can significantly improve the failure prediction
accuracy by 5% on average.

Cloud Failure Prediction, Model Updating, Machine Learning

* The work is done during Haozhe Li’s internship at Microsoft.

††copyright: acmcopyright††journalyear: 2023††doi: XXXXXXX.XXXXXXX††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06††copyright: acmcopyright††ccs: Computer
systems organization Cloud computing††ccs: Software and its engineering
Maintaining software

##  1\. Introduction

The IT industry has seen a significant trend towards migrating large workloads
to the cloud, such as Microsoft Azure, Amazon Web Services, and Google Cloud
Platform. These providers monitor and analyze thousands of metrics from their
cloud infrastructures to ensure high-quality service for billions of users
worldwide (3; 35; 2). These metrics can help detect and prevent failures in
various components of the cloud system, such as memory (8), disk (24; 22),
node (19), or switch (37), etc. Machine learning or deep learning techniques
have been applied to these metrics to predict failures and take proactive
actions to mitigate them, which can improve the availability and performance
of cloud-based software systems while reducing operational costs and risks.

A lot of previous studies aim to design better machine learning models to
improve the performance of cloud failure prediction tasks. For example, RNN
(32), LSTM (36), Transformer (24), and TCNN (30) models are used to predict
cloud failures. When a prediction indicates a problem, we take quick action to
fix the underlying issue. For example, if a computing node fails, we might
move the running virtual machine to another node with minimal disruption or
preserve the virtual machine’s state during a node reboot (15). Although these
actions change the node’s status, they do not offer a detailed analysis of the
root cause of the failure. Consequently, we cannot be certain whether the
predicted failure will actually occur, which we call an “uncertain positive”
instance.

According to our experience in deploying cloud failure prediction models in
Microsoft Azure, we notice that the model updating meets great challenges in
real-world online usage. Model updating is the process of retraining the
machine learning model over time (weekly or monthly) to adapt to the changing
cloud environment (new hardware and software) (25; 15). An empirical study
conducted on both open datasets and Microsoft Azure (see Section 3), suggests
that the prediction accuracy of updated models may decrease by about 9% over
time because the uncertain positive instances introduce noise to the model
updating. This effect is compounded when using a continuously updated model
that accumulates uncertain positive instances over time.

Figure 1. Toy example of cloud failure prediction model (binary classification
for normal or failure instances) with offline updating and online updating.
The prediction model (M) is updated by training with data in the dashed box in
each time stage (T). The number on the top of each sub-figure illustrates the
instance ID.

More specifically, Figure 1 illustrates a toy example of the machine learning
model updating scenario in the cloud failure prediction. The figure has two
parts, i.e., the offline updating and the online updating. The offline
updating is an ideal scenario in which we can access the oracle of cloud
instance status (shown as the disk icons in black color). We may collect the
monitoring metrics and status of the cloud instances in each period (T1, T2,
T3, etc.). Then we train the model (M1, M2, M3, etc.) in each period without
any concern. When it comes to the online scenario, however, we cannot access
the oracle of instances status after T1 (they are identical to the offline
scenario). In real practice, we train a model in the T1 and make online
inferences in the T2 stage, some instances (#7, #8, and #9) are predicted as
failure (in blue color). Then, these instances are mitigated and cannot access
their actual status, which is noted as uncertain positive instances. Finally,
together with the known failure instance (#10), we may retrain the model M2.
Clearly, the #7 instance is a False Positive one that may introduce noise to
model M2. Similarly, we may obtain a less accurate model (#11, #12 are
uncertain positive instances, and #14, #15 are true positive instances) in the
T3 stage since the model noise may be accumulated.

To the best of our knowledge, we are the first to identify the Uncertain
Positive Learning (UPLearning) challenge in the real-world scenario of cloud
failure prediction. Addressing this challenge is crucial for ensuring cloud
reliability by enabling accurate failure prediction. The most closely related
research topic to UPLearning is the Positive Unlabeled Learning (PULearning)
(11; 26; 18; 21; 17; 14; 20; 9; 13). PULearning is a machine learning scenario
for binary classification where the training set consists of a set of
positively labeled instances and an additional unlabeled set that contains
positive and negative instances in unknown proportions (so no training
instances are explicitly labeled as negative). The risk estimator-based
approach, which estimates the distribution of negative instances in the
unlabeled set, is widely used to solve the PULearning problem (13; 18; 26).
However, our UPLearning involves uncertain positive instances, which are
similar to the unlabeled set in the PULearning scenario, but also includes
both positive and negative instances that system operators investigate after
failures occur (see Section 2). As a result, it is not feasible to directly
apply the risk estimator approach used in PULearning.

In this paper, we propose an Uncertain Positive Learning Risk Estimator
(Uptake), a novel approach for various cloud failure prediction models to
achieve high prediction accuracy even with uncertain positive instances.
Uptake regards uncertain positive instances as both positive and negative
through a specially designed risk estimator during the model updating
procedure. Besides, Uptake only improves the loss function which makes it easy
to integrate with various machine learning models, such as RNN, LSTM,
Transformer, and TCNN. To evaluate the effectiveness of our proposed Uptake
approach, we conduct extensive experiments to compare Uptake against three
model updating approaches on two public disk datasets, Alibaba and Backblaze,
which both contain tens of thousands of disks over months. We also apply
Uptake to the scenario of node failure prediction in Microsoft Azure, which is
a top-tier cloud provider and serves millions of customers around the world.
The experiment results on those three datasets demonstrate Uptake outperforms
any other model updating approaches over different prediction models, i.e.,
RNN, LSTM, Transformer, and TCNN. The F1-scores of Uptake are $45.26\%$,
$70.16\%$, and $69.33\%$ in Alibaba, Backblaze, and Microsoft Azure scenarios,
which are $4.85\%$, $4.17\%$, and $5.13\%$ better than those achieved by the
best baseline approaches, respectively.

To sum up, this work has the following contributions:

  * •

To the best of our knowledge, we identify the Uncertain Positive Learning
(UPLearning) problem in the scenario of cloud failure prediction for the first
time, which is an important issue to tackle for machine learning model usage
in the real-world scenario.

  * •

To solve the UPLearning, we propose an uncertain positive learning risk
estimator approach, dubbed Uptake, which is easy to integrate with various
machine learning models.

  * •

To illustrate the generality and effectiveness of Uptake, we conduct
experiments on different cloud failure prediction scenarios, i.e., two public
disk failure datasets and the node failure from Microsoft Azure. Uptake
outperforms any other online model updating approach over four widely used
failure prediction models. Moreover, Uptake has been successfully applied to
Microsoft Azure and improved the reliability of cloud platforms.

##  2\. Background

Table 1. Examples of node & disk monitoring metrics.

Type | Description | Examples (Feature belongs to Disk/Node)
---|---|---
Operation Timer |  | The implementation of a timer for various operations of hard disk drives,
---
e.g., magnetic head seeking and spindle activation, plays a crucial role in
optimizing the performance of these devices.
|  SpinUpTimeNorm (Disk)
---
SpinRetryCountRaw (Disk)
Spin_Retry_Count_VALUE (Disk)
Spin_Up_Time_VALUE (Disk)
Operation Counter |  | Count of common operations, e.g., CPU utilization and power
---
consumption, etc.
|  PowerCycle (Node)
---
PowerOnHours (Node)
PowerConsumption (Node)
Physical Characteristics |  | The measurement of physical characteristics, e.g., temperature and
---
humidity, etc.
|  AirflowTempRaw (Node)
---
TemperatureRaw (Node)
Temperature (Node)
Error Counter |  | A count of specific, identifiable errors is maintained. This count typically
---
remains unchanged unless an error occurs, at which point the count is
updated to reflect the occurrence of the error.
|  Program_Fail_Cnt_Total_RAW_VALUE (Disk)
---
Seek_Error_Rate_RAW_VALUE (Disk)
TotalErrors (Node)
IOEDCErrorCountRaw (Node)
MediaErrors (Node)
ErrorInfoLogEntryCount (Node)
EVENT |  | Windows event are typically used for indicating exception handling for
---
Windows servers.
|  WindosStorportEvent_534 (Node)
---
WindosStorportEvent_554 (Node)

Cloud system. Cloud systems, such as Amazon Web Services (AWS), Microsoft
Azure, and Alibaba Cloud, provide pay-as-you-go services for companies and
developers, making it convenient for them to deploy their web applications
through the Internet. These cloud systems serve millions of users around the
world, and any failure in the cloud will have a significant impact. Therefore,
it is of great importance to keep the reliability of cloud systems. Among the
reliability efforts in cloud systems, failure prediction, which can detect
failure before it actually happens, is one of the most effective solutions and
is well-studied in the literature.

Cloud failure prediction. Cloud failure refers to a state where cloud
instances become unavailable due to hardware interruptions, code bugs, or high
workload demands. Cloud failure can be categorized based on the affected
instances, such as node failure (19), network issue (6), disk failure (24),
service overloading (38), etc. In the following, we introduce two typical
cloud failure prediction scenarios, i.e., disk failure prediction and node
failure prediction.

  * •

Node failures may occur for a variety of reasons (e.g., OS crashes,
application bugs, misconfigurations, memory leaks, software incompatibility,
overheating, and service exceptions). The node failure can be predicted by a
wide range of monitoring metrics (signals indicating their temporal and
spatial information). MING (19) predicts node failure by taking into account
both spatial and temporal signals. Table 1 presents several monitoring metrics
of a node, which are classified into six distinct categories: operation timer,
physical characteristics, operation counter, error counter, logical
input/output, and EVENT data. A detailed explanation of these categories,
including examples, is illustrated in Table 1. The collection of these metrics
is conducted at regular intervals, except for EVENT data. To analyze these
metrics, a sliding window technique is employed in the prediction model,
resulting in the transformation of the metrics into a feature vector.

  * •

Disk failure in cloud systems is a type of hardware failure that can lead to
service downtime. Disk failure can be predicted based on its status data,
known as the SMART (Self-Monitoring, Analysis and Reporting Technology) (31;
23). The monitoring metrics of a disk are SMART (Self-Monitoring, Analysis,
and Reporting Technology) (31; 23). These attributes, which include metrics
such as temperature, spin-up time, and reallocated sectors (examples shown in
Table 1), are employed to predict potential failures. By providing early
warning of such failures, SMART technology enables proactive maintenance and
replacement to be performed prior to a failure occurring.

Based on previous studies, we introduce and implement four widely used machine
learning models for cloud failure prediction.

  * •

RNN (32): Recurrent Neural Network is a widely-used deep learning model
designed for sequential data. During prediction, RNN uses its recurrent unit
to feature the difference between the normal state and failure state from the
input sequential data.

  * •

LSTM (36): Long Short-Term Memory is an advanced version of recurrent neural
network architecture that was designed to model chronological sequences and
their long-range dependencies more precisely than conventional RNNs. With
long-term features, LSTM can always perform better than RNN and can typically
class difficult examples in RNN.

  * •

Transformer (24): The Transformer Model is a novel deep learning-based
approach to failure prediction tasks with the attention mechanism, which can
capture the temporal nature from instance status data. Transformer utilizes
not only a single instance’s status data but also considers its neighbors’
status data to optimize its prediction performance.

  * •

TCNN (30): Temporal Convolutional Network is a variation of Convolutional
Neural Networks for sequence modeling tasks, by combining aspects of RNN and
CNN architectures.

Model updating. The distribution of online monitoring metrics changes with
dynamic software and hardware updates in cloud systems, which causes the
distribution learned by the previous model to deviate significantly from the
online distribution (25). As a result, the prediction performance of the model
may degrade. To ensure prediction performance, machine learning models
deployed online need to be updated over time (weekly or monthly). Model
updating needs a considerable number of failure instances. Updating cloud
failure prediction models using pre-training and fine-tuning (33) strategies
[32] is not feasible due to the imbalance between positive and negative
instances, as well as the insufficient number of positive instances in real-
world scenarios.

##  3\. An Empirical Study of Prediction Accuracy over Time

We conduct an empirical study on failure prediction. In this study, we aim to
uncover the problem based on our experience in deploying failure prediction
models in Microsoft Azure. We address the following research questions:

  * •

RQ1: How does the failure prediction accuracy change over time?

  * •

RQ2: Why does the prediction accuracy decrease over time?

###  3.1. Subjects

We adopt two public datasets (24; 22) for disk-level failure prediction, i.e.,
Alibaba Cloud and Backblaze datasets. Besides, Microsoft Azure provides large-
scale datasets for node-level failure prediction.

Alibaba is collected from large-scale data centers and published by Alibaba
Cloud for PAKDD 2020 Alibaba AIOps Competition (29), which contains millions
of disks with a period of more than 16 months. In our experiment, we adopt the
dataset within 10 months and split it into five continuous time phases of
equal length. Each time phase contains more than 8,000 disk records with a
period of two months and each record has 30 days long monitoring metrics. We
use 17-dimension features for failure prediction.

Backblaze is a public dataset published by Backblaze, based on the hard drives
in Backblaze data center (4). Each disk of the dataset has a label indicating
its status. In our experiment, we use five months from 2021Q4 to 2022Q1, and
it contains over 90,000 disks in total. We split the dataset into five
continuous time phases of equal length. Each time phase contains approximately
16,000 disk records and each record contains 38 dimension features with a
period of 30 days.

Microsoft Azure is a large-scale commercial cloud system that includes node-
level monitoring metrics. Over 500,000 node recordings overall from a period
of 35 days are employed in our experiment. Additionally, we divide all records
into five continuous time phases of equal length. Each time phase comprises
more than 100,000 node records. The feature input is the 23-dimension
monitoring metrics of nodes in a 48-hour time window.

To demonstrate the generality and robustness of our investigation, we employed
different periods for model updating: two months for Alibaba, one month for
Backblaze, and one week for Microsoft Azure. These periods were selected based
on the number of positive instances available in each, as positive instances
are significantly less frequent than negative ones in real-world datasets, as
noted by Ntam(24). We aim to ensure that each period contains a relatively
sufficient number of positive instances, maintaining an imbalance rate
(#Positive to #Negative ratio) of approximately 1%.

###  3.2. RQ1: Prediction Accuracy over Time

Figure 2. Cloud failure prediction average F1-score over time.

We undertake an investigation on prediction performance trends across a
continuous long range of time, which is divided into 5 equal-length time
phases. At the start of each phase (except for the first training phase T1),
model updating is performed using monitoring data collected from the previous
phase, then deployed for online prediction. The average prediction performance
of the model is calculated at each time phase.

The prediction F1-score (see Section 5.1.4) on 3 datasets across 4-time phases
is illustrated in Figure 2. The time in the figure starts from T2, which is
because monitoring data in the T1 phase is used for the initial model
training. Overall, there is a significant and consistent decrease in the
accuracy of the failure prediction model over time for all datasets.
Performance on Microsoft Azure decreases from 78.22% on T2 to 57.46% on T5, a
decrease of 20.66%. Backblaze and Alibaba also show decreases of 8.34% and
13.77%, respectively. The results on Alibaba do not show a decreasing trend in
performance on T3 and T4 because the percentage and performance of failures
are not completely consistent across time, but the performance of failure
prediction on Alibaba also shows a significant decreasing trend of 13.77% when
observed over the entire experimental time.

The decrease in prediction accuracy over time can be attributed to changes in
the distribution of online data. However, it is important to note that this
accuracy decrease is observed consistently across all three cloud systems and
time phases. Furthermore, the retrained model is trained using the latest
collected data, which has the least bias compared to the online data. We refer
to this decrease in prediction accuracy after retraining as the UPLearning
(Uncertain Positive Learning) problem.

###  3.3. RQ2: Explanation of Accuracy Decrease

Table 2. Mitigation actions to cloud failures.

Action |  Description
---|---
Live Migration |  Move running VMs from to other failure-free nodes
VM Preserving Soft Reboot |  Reboot the host OS kernel and preserve the VM states
Service Healing |  Disconnect current VM and generate new assignment of the VM to healthy nodes
Mark Unallocatable |  Block allocation of new VMs to a node
Avoid |  Reduce the weight to allocate a new VM to a node

It is mitigation actions that contaminate the training data and lead to a
decrease in prediction accuracy (i.e., UPLearning) during model retraining. In
the context of failure prediction in cloud systems, when a cloud component is
anticipated to fail, mitigation actions(15) in Table 2 are promptly undertaken
to either resolve the underlying failure or minimize the impact on services.
The effectiveness of these mitigation actions, however, makes it impractical
to accurately verify the accuracy of the predictions. For instance, in the
case of applying live migration to a predicted failing node, all run-time
state is migrated to a new node, resulting in the release of workload on the
original node(7). This operation alters the original state, rendering it
unverifiable whether the failure would have indeed occurred in the original
state.

Figure 3. Example of positive, negative, and uncertain positive in the
scenario of online cloud failure prediction.

UPLearning problem. As we discussed before, the Uncertain Positive Learning
(UPLearning) problem in cloud systems is mainly due to mitigation actions,
which change the state of the instances and make it impossible to verify the
accuracy of the predictions. In the context of cloud failure prediction, an
online model (M) is initially trained using historical data with true labels
(oracle labels) during period T1. This model predicts instances that are
likely to fail, represented by blue-filled instances in period T2, applying
mitigation actions to address the potential failure. However, it remains
unknown whether these predicted instances will actually fail or not, as
indicated by the black-bordered instances (#6-#10). Uncertain Positive
instances are those predicted as failures by the online model, such as
instances #7, #8, and #9 in Figure 3. However, the true labels (oracle labels)
for these instances (#7: Negative, #8 and #9: Positive) cannot be verified
after applying failure mitigation actions, because mitigation actions change
the original state of the instances.

##  4\. Approach

We propose a novel model updating approach to solve the UPLearning problem,
named Uncertain Positive Learning Risk Estimator (Uptake), which still
consumes uncertain positive instances during model updating and significantly
improves the online model updating performance for cloud failure prediction.

In this section,

we first present the problem formulation of UPLearning in Section 4.1. Then we
introduce the technical details of Uptake in Section 4.2.

###  4.1. Problem Settings

Before diving into the approach details, we present a formal definition of the
cloud failure prediction task. We define the cloud failure prediction task as
a binary classification problem based on monitoring metrics. Essentially, we
collect feature vectors with $d$ dimensions, which contain different types of
monitoring metrics from cloud infrastructures, at regular intervals (e.g.
hourly or daily). These feature vectors are used to create a continuous time
series of features, denoted as $X=[x_{1},x_{2},...,x_{l}]\in\mathbb{R}^{d}$,
where $l$ is the number of timestamps in the feature $X$. For each feature
$X$, we assign a label $Y\in{0,1}$ based on whether it represents a failure
instance $(Y=1)$ or a normal instance $(Y=0)$ in the cloud system. The goal of
cloud failure prediction is to train a classifier $f_{X}\to Y$ that can
predict the class label $Y$ when given a feature $X$ with uncertain positive
labels.

The training set is a collection of features $X$ with their corresponding
label $Y$, denoted as $\mathcal{N}=\\{(X_{1},Y_{1}),...,(X_{n},Y_{n})\\}$,
where $n=\left|\mathcal{N}\right|$. Our target is to minimize prediction loss
$\mathcal{L}$, (e.g., the binary cross-entropy loss), on the training set.

In the scenario of cloud failure prediction, some labels
$Y_{i}(i\in\\{1,...,n\\})$ in the training set are uncertain, specifically, we
have to use the training set which contains uncertain label $\hat{Y}$ to train
and update the online model. The training loss to be minimized is represented
as:

(1) $\mathcal{L}=-\frac{1}{n}\sum^{n}_{i=1}l(f(X_{i}),\hat{Y_{i}})$

where $l$ is the loss function, $f$ is the binary classifier, and
$(X_{i},\hat{Y}_{i}),\quad i\in\\{1,...n\\}$ is the sample from the training
set. $\hat{Y}$ is the uncertain label (i.e., the predicted label from the
online model).

###  4.2. Uptake

Typically, the model’s performance is assessed using the risk function (10). A
precise risk function enhances the performance of the failure prediction
model. In this section, we introduce a new method for cloud failure prediction
called Uptake (Uncertain Positive Learning Risk Estimator). This approach
ensures high prediction accuracy, especially when handling uncertain positive
instances, and effectively addresses the UPLearning problem.

Risk estimator. In this section, we describe the binary classification
representation of the risk function (10; 28). Consider a feature set
$X\in\mathbb{R}^{d}$ and its corresponding label $Y\in\\{0,1\\}$. Let $p(x,y)$
be the underlying joint density of $(X,Y)$, $p_{\rm p}(x)=p(x|Y=1)$, and
$p_{\rm n}(x)=p(x|Y=0)$ represent the positive and negative marginals,
respectively. Additionally, define $\pi_{\rm p}=p(Y=1)$ as the class-prior
probability and $\pi_{\rm n}=p(Y=0)=1-\pi_{\rm p}$.

Let $f_{X}\to Y$ be the decision function, and
$l:\mathbb{R}\times\\{0,1\\}\to\mathbb{R}$ be the loss function, where
$l(f(X),Y)$ measures the loss between the predicted output $f(X)$ and the
actual label $Y$.

Define $R(f)$ as the risk function for the given decision function $f_{X}\to
Y$, $R^{+}_{\rm p}(f)=\mathbb{E}_{\rm p}[l(f(X),1)]$, and $R^{-}_{\rm
n}(f)=\mathbb{E}_{\rm n}[l(f(X),0)]$, where $\mathbb{E}_{\rm
p}[\cdot]=\mathbb{E}_{X\sim p_{\rm p}}[\cdot]$ and $\mathbb{E}_{\rm
n}[\cdot]=\mathbb{E}_{X\sim p_{\rm n}}[\cdot]$, denoting the expectations of
positive and negative variables.

In standard binary classification, the risk of the decision function $f_{X}\to
Y$ is denoted as:

(2) $R(f)=\mathbb{E}_{(X,Y)\sim p(x,y)}[l(f(X),Y)]=\pi_{\rm p}R^{+}_{\rm
p}(f)+\pi_{\rm n}R^{-}_{\rm n}(f)$

Uncertain Positive Learning risk estimator. In the context of cloud failure
prediction, we encounter instances not only labeled as positive or negative
but also instances with uncertain positive labels predicted by the online
model. Uptake treats uncertain positive instances as having both positive and
negative labels with different weights. The risk function for this scenario is
represented as follows:

(3) $R(f)=\pi_{\rm p}R^{+}_{\rm p}(f)+\pi_{\rm n}R^{-}_{\rm n}(f)+(\pi_{\rm
p}R^{+}_{\rm u}(f)+\pi_{\rm n}R^{-}_{\rm u}(f))$

Here, $R_{\rm u}^{+}(f)=\mathbb{E}_{\rm u}[l(f(X),1)]$ and $R_{\rm
u}^{-}(f)=\mathbb{E}_{\rm u}[l(f(X),0)]$ represent the expectations of the
decision function $f(X)$ when dealing with uncertain positive instances.

It’s crucial to note that $\pi_{\rm p}$ and $\pi_{\rm n}$ are hyper-parameters
in this study. $\pi_{\rm p}$ estimates the proportion of real positive
instances among uncertain positive instances, while $\pi_{\rm n}=1-\pi_{\rm
p}$ estimates the percentage of negative instances. In this paper, $\pi_{\rm
p}$ is set to the Precision value obtained during the T1 validation stage of
hyper-parameter tuning (refer to Section 5.6). When $\pi_{\rm p}$ approximates
the proportion of uncertain positive instances with true positive labels,
Uptake demonstrates improved performance. The Precision value from the T1
validation stage offers an accurate estimate of the proportion of uncertain
positive instances with positive labels.

Implementation in various models. Uptake is a generic approach that can be
integrated with any machine learning model for cloud failure prediction.

Algorithm 1 illustrates the core implementation of Uptake. The input training
data subscripted with $\rm p$, $\rm n$, and $\rm u$ are the instances labeled
with positive, negative, and uncertain positive, respectively, and $\theta$
indicates the model parameter for decision function $f(X;\theta)$. To reduce
the interference of uncertain positive instances to the direction of gradient
descent, Uptake improves the loss function in the model updating procedure by
adjusting the direction of the gradient.

Since Uptake only improves the original loss function without changing its
property, it is suitable for various machine learning models.

Algorithm 1 Introducing Uptake in the gradient descent

0: training data ($X_{\rm p},X_{\rm n},X_{\rm u}$); hyperparameter
$0\leq\pi_{\rm p}\leq 1$, and $\pi_{\rm n}=1-\pi_{\rm p}$;decision function
$f(X;\theta)$; loss function $l(f(X),Y)$

0: gradient $\nabla_{\theta}$ to update $\theta$ for $f(X;\theta)$

1: Improved loss $\mathcal{L}=\pi_{\rm p}l(f([X_{\rm p},X_{\rm
u}]),1)+\pi_{\rm n}l(f([X_{\rm n},X_{\rm u}]),0)$

2: Set gradient $\nabla_{\theta}\mathcal{L}$ and update $\theta$

##  5\. Experiment

In this section, we conduct extensive experiments to demonstrate the
effectiveness of our Uptake approach when integrated with existing failure
prediction models. First, we present the experimental settings, including
datasets, the failure prediction models, and three compared approaches. Then,
we conduct experiments to verify the following research questions:

  * •

RQ3: How does Uptake perform for UPLearning problem in cloud failure
prediction?

  * •

RQ4: How does Uptake perform in the scenario of online cloud systems?

  * •

RQ5: Does Uptake exhibit consistent performance across different base models?

  * •

RQ6: What is the impact of Uptake on predicting efficiency?

  * •

RQ7: How Uptake’s parameters impact its prediction performance?

RQ3 quantitatively measures the performance of Uptake and compares it with
several in the public disk prediction datasets. RQ4 applies Uptake to a large-
scale commercial cloud system to evaluate its performance in a real-world
setting.

###  5.1. Settings

####  5.1.1. Datasets

To evaluate the reliability and applicability of Uptake, we employ two
publicly accessible datasets: the Alibaba Cloud and Backblaze datasets (24;
22), for predicting disk failures. Furthermore, we integrate a vast industrial
cloud system dataset to predict node failures. These datasets together form
the foundation of our assessment and analysis. The specific configuration and
details of these datasets align with what we outlined in Section 3.1.

####  5.1.2. Implementations and Environments

We implement all four deep learning models, i.e., RNN, LSTM, Transformer, and
TCNN, based on Python 3.8.13 and PyTorch 1.11.0 (27). We use identical
parameter values as in the previous works for the compared approaches. The
model was selected based on the epoch that achieved the highest F1-score on
the validation set.

We conduct all the experiments on Linux Dev Node with Ubuntu 20.04 LTS 64-bit
operating system, 24 cores AMD Epyc 7V13, 220 GB memory, and NVIDIA Tesla A100
with 80 GB GPU memory.

####  5.1.3. Important Hyper-parameters

As explained in Section 4.2, the parameter $\pi_{\rm p}$ plays a crucial role
in this study. Its value is derived from the Precision metric obtained during
the T1 validation stage.

####  5.1.4. Evaluation metrics

The cloud failure prediction task is a binary classification task, aligning
with existing research. The performance of Uptake is assessed using Precision,
Recall, and F1-score, following standard conventions. Instances are labeled as
either normal or failure based on their practical performance, with failure
considered positive and normal considered negative. True Positive (TP)
represents the number of correctly predicted failure instances, while True
Negative (TN) represents the accurately predicted normal instances.
Conversely, False Positive (FP) and False Negative (FN) indicate instances
that were incorrectly predicted as normal and failure, respectively.

####  5.1.5. Compared Approaches

In the experiment section, we compare Uptake with two model updating
approaches, which are named offline updating (offline),

and online updating with certain positive labels (certain). To the best of our
knowledge, UPLearning has not been proposed before. Therefore, there are no
state-of-the-art solutions for model updating. We design the two model
updating approaches below as compared with Uptake.

  * •

Offline updating employs ground truth labels for model training, which contain
no uncertain labels. However, this approach is not feasible in practice
because part of the ground truth labels are uncertain positive labels. It
serves as an upper bound of the performance for Uptake and other subsequent
approaches.

  * •

Online updating with certain positive labels accumulates all records with
certain labels for model updating. For instance, when updating the model for
predicting phase T4, the certain approach collects all records with certain
labels from phase T1 to phase T3. This includes all records in phase T1, as
well as true negative (TN) and false negative (FN) records in phases T2 and
T3.

Table 3. Performance comparison of Uptake and its three compared approaches on
Alibaba dataset. P, R, and F1 denote precision, recall and F1-score,
respectively.

Model | Approach | T2 | T3 | T4 | T5 | Avg.
---|---|---|---|---|---|---
P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | F1
RNN | Offline | 57.84 | 33.33 | 42.29 | 40.12 | 50.00 | 44.52 | 76.92 | 35.71 | 48.78 | 43.89 | 49.07 | 46.33 | 46.54
2-26-15 | Certain | 35.95 | 42.31 | 38.87 | 79.59 | 23.21 | 35.94 | 49.12 | 17.39 | 25.69 | 33.50
| Uptake | 40.30 | 41.54 | 40.91 | 47.65 | 42.26 | 44.79 | 35.96 | 39.75 | 37.76 | 41.15
LSTM | Offline | 65.31 | 36.16 | 46.55 | 65.82 | 40.00 | 49.76 | 74.71 | 38.69 | 50.98 | 63.11 | 40.37 | 49.24 | 49.99
2-26-15 | Certain | 54.22 | 34.62 | 42.25 | 80.43 | 22.02 | 34.58 | 59.09 | 24.22 | 34.36 | 37.06
| Uptake | 56.18 | 38.46 | 45.66 | 64.76 | 40.48 | 49.82 | 58.72 | 39.75 | 47.41 | 47.63
Trans -former | Offline | 62.41 | 46.89 | 53.55 | 61.80 | 42.31 | 50.23 | 68.47 | 45.24 | 54.48 | 61.42 | 48.45 | 54.17 | 52.96
2-26-15 | Certain | 53.92 | 42.31 | 47.41 | 79.66 | 27.98 | 41.41 | 61.76 | 26.09 | 36.68 | 41.83
| Uptake | 53.04 | 46.92 | 49.80 | 46.70 | 50.60 | 48.57 | 41.24 | 49.69 | 45.07 | 47.81
TCNN | Offline | 57.72 | 40.11 | 47.33 | 43.59 | 52.31 | 47.55 | 60.38 | 38.10 | 46.72 | 58.88 | 39.13 | 47.01 | 47.09
2-26-15 | Certain | 46.25 | 28.46 | 35.24 | 72.41 | 25.00 | 37.17 | 42.86 | 22.36 | 29.39 | 33.93
| Uptake | 63.24 | 33.08 | 43.43 | 36.84 | 54.16 | 43.85 | 68.29 | 34.78 | 46.09 | 44.46
Avg. | Offline | 60.82 | 39.12 | 47.43 | 52.83 | 46.16 | 48.02 | 70.12 | 39.44 | 50.24 | 56.83 | 44.26 | 49.19 | 49.15
2-26-15 | Certain | 47.59 | 36.93 | 40.94 | 78.02 | 24.55 | 37.28 | 53.21 | 22.52 | 31.53 | 36.58
| Uptake | 53.19 | 40.00 | 44.95 | 48.99 | 46.88 | 46.76 | 51.05 | 40.99 | 44.08 | 45.26

Table 4. Performance comparison of Uptake and its two compared approaches on
Backblaze dataset. P, R, and F1 denote precision, recall and F1-score,
respectively.

Model | Approach | T2 | T3 | T4 | T5 | Avg.
---|---|---|---|---|---|---
P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | F1
RNN | Offline | 50.88 | 84.67 | 63.56 | 62.75 | 73.56 | 67.73 | 74.29 | 73.24 | 73.76 | 76.33 | 62.32 | 68.62 | 70.04
2-26-15 | Certain | 57.89 | 50.57 | 53.99 | 80.91 | 41.78 | 55.11 | 83.67 | 39.61 | 53.77 | 54.29
15-15 | Uptake | 59.31 | 69.54 | 64.02 | 62.21 | 87.32 | 72.66 | 57.73 | 81.16 | 67.47 | 68.05
LSTM | Offline | 63.59 | 85.40 | 72.90 | 70.79 | 82.18 | 76.06 | 76.71 | 78.87 | 77.78 | 70.71 | 81.64 |  75.78 | 76.54
2-26-15 | Certain | 71.81 | 61.49 | 66.25 | 86.79 | 43.19 | 57.68 | 82.86 | 42.03 | 55.77 | 59.90
| Uptake | 63.32 | 83.33 | 71.96 | 61.72 | 74.18 | 67.38 | 54.79 | 80.19 | 65.10 | 68.15
Trans -former | Offline | 71.60 | 84.67 | 77.59 | 74.87 | 85.63 | 79.89 | 76.23 | 79.81 | 77.98 | 72.44 | 78.74 | 75.46 | 77.78
2-26-15 | Certain | 69.54 | 69.54 | 69.54 | 83.48 | 45.07 | 58.54 | 82.41 | 43.00 | 56.51 | 61.53
| Uptake | 71.36 | 87.36 | 78.55 | 71.31 | 81.69 | 76.15 | 64.77 | 82.61 | 72.61 | 75.77
TCNN | Offline | 63.54 | 83.94 | 72.33 | 67.35 | 75.86 | 71.35 | 74.24 | 79.81 | 76.92 | 79.63 | 62.32 | 69.92 | 72.73
2-26-15 | Certain | 77.45 | 45.40 | 57.25 | 79.07 | 47.89 | 59.65 | 80.92 | 51.21 | 62.72 | 59.87
| Uptake | 59.26 | 73.56 | 65.64 | 62.45 | 81.22 | 70.61 | 63.49 | 77.29 | 69.72 | 68.66
Avg. | Offline | 62.40 | 84.67 | 71.60 | 68.94 | 79.31 | 73.76 | 75.37 | 77.93 | 76.61 | 74.78 | 71.26 | 72.45 | 74.27
2-26-15 | Certain | 69.17 | 56.75 | 61.76 | 82.56 | 44.48 | 57.75 | 82.47 | 43.96 | 57.19 | 58.90
| Uptake | 63.31 | 78.45 | 70.04 | 64.42 | 81.10 | 71.70 | 60.20 | 80.31 | 68.73 | 70.16

###  5.2. RQ3: Performance of Uptake

Table 5. Performance comparison of Uptake and its two compared approaches on
Microsoft Azure. P, R, and F1 denote precision, recall and F1-score,
respectively.

Model | Approach | T2 | T3 | T4 | T5 | Avg.
---|---|---|---|---|---|---
P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | F1
RNN | Offline | 88.31 | 86.96 | 75.16 | 88.59 | 67.11 | 76.36 | 71.43 | 59.14 | 64.71 | 82.24 | 68.75 | 74.89 | 71.99
2-26-15 | Certain | 93.94 | 46.62 | 62.31 | 83.16 | 43.82 | 57.39 | 93.50 | 48.70 | 64.04 | 61.25
| Uptake | 94.62 | 49.62 | 65.10 | 84.65 | 51.88 | 64.33 | 93.28 | 57.81 | 71.38 | 66.94
LSTM | Offline | 80.60 | 79.67 | 80.13 | 66.19 | 78.76 | 71.93 | 77.44 | 68.28 | 72.57 | 80.79 | 63.54 | 71.14 | 71.88
2-26-15 | Certain | 47.45 | 61.28 | 53.49 | 78.95 | 56.45 | 65.83 | 68.65 | 93.69 | 68.65 | 62.66
| Uptake | 59.58 | 79.51 | 68.12 | 65.05 | 75.54 | 69.90 | 84.00 | 60.16 | 70.11 | 69.38
Trans -former | Offline | 65.60 | 96.55 | 78.12 | 75.27 | 78.95 | 77.06 | 76.45 | 67.20 | 71.53 | 71.03 | 73.44 | 72.22 | 73.60
2-26-15 | Certain | 80.54 | 56.02 | 66.08 | 82.63 | 52.42 | 64.14 | 91.28 | 51.82 | 66.11 | 65.44
| Uptake | 72.38 | 77.82 | 75.54 | 65.37 | 72.04 | 68.54 | 71.74 | 68.75 | 70.21 | 71.43
TCNN | Offline | 80.87 | 75.70 | 78.20 | 67.89 | 79.89 | 73.40 | 84.69 | 66.94 | 74.77 | 64.25 | 76.30 | 69.76 | 72.64
2-26-15 | Certain | 45.24 | 57.14 | 50.50 | 86.51 | 50.00 | 63.37 | 91.08 | 50.52 | 64.99 | 59.62
| Uptake | 63.72 | 77.26 | 69.84 | 71.28 | 73.39 | 72.32 | 57.77 | 78.39 | 66.52 | 69.56
Avg. | Offline | 78.85 | 84.72 | 77.90 | 74.49 | 76.18 | 74.69 | 77.50 | 65.39 | 70.90 | 74.58 | 70.51 | 72.00 | 72.53
2-26-15 | Certain | 66.79 | 55.27 | 58.10 | 82.81 | 50.67 | 62.68 | 86.13 | 61.18 | 65.95 | 62.24
| Uptake | 72.58 | 71.05 | 69.65 | 71.59 | 68.21 | 68.77 | 76.70 | 66.28 | 69.56 | 69.33

We assess our proposed approach, Uptake, using two public datasets referred to
as Alibaba and Backblaze. The results are presented in Table 3 and Table 4
respectively. In these tables, the last column displays the average F1-score
for each row. The final three rows depict the average results of all four
models across different time phases and using different approaches.

In the rest of the table, every set of three columns showcases the performance
metrics (Precision, Recall, and F1-score) for different time phases. Notably,
the time phases in Table 3 and Table 4 commence from T2, as the data in T1 is
utilized for the initial model training. It’s important to observe that the
three approaches yield identical results at T2 since it marks the first stage
to obtain model inference results without encountering any UPLearning problem
(as illustrated in Figure 1).

The performance of these models remains competitive with previous work (22),
considering the inherent challenges of cloud failure prediction tasks in real-
world scenarios. It’s worth mentioning that online disk failure prediction
often faces limitations due to low F1-scores. This challenge largely stems
from the prevalence of missing data and significant imbalances in the data
distribution, posing significant obstacles to improvement.

Upon analyzing Table 3 and Table 4, we observe that Uptake performs admirably
on these two public datasets. The average F1-scores are $45.26\%$ for Alibaba
and $70.16\%$ for Backblaze. According to the results shared above, we have
the following findings. The results of various methods applied to Alibaba’s
data demonstrate low performance due to the complexity of the dataset. The
highest F1-score achieved on the leader-board is only $49.07\%$.
111https://tianchi.aliyun.com/competition/entrance/231775/rankingList

Comparison with different updating approaches. When examining the last column
of Table 3 and Table 4, it becomes evident that offline updating consistently
outperforms other approaches. It achieves an average F1-score of $49.15\%$ in
Alibaba and $74.27\%$ in Backblaze. However, it’s important to note that
offline updating, while showing the best performance, is not practically
applicable as it requires access to all certain labels, which is often
unavailable in real-world scenarios. Therefore, we present this approach as
the ”upper bound” of the prediction model, representing the best possible
performance using all certain labels. Among the practical and feasible
approaches, Uptake demonstrates the best performance. It achieves an average
F1-score of $45.26\%$ in Alibaba and $70.16\%$ in Backblaze. In contrast,
Certain performs the worst, with an average F1-score of $36.58\%$ in Alibaba
and $58.90\%$ in Backblaze. We attribute the lower accuracy of Certain to the
accumulation of phases, which causes the proportion of positive and negative
instances to deviate from the actual distribution. Specifically, the Certain
method disregards all uncertain positive samples, even though only an average
of $60.84\%$ in Alibaba and $19.27\%$ in Backblaze of positive labels are
certain positive. This discrepancy in handling uncertain positive samples
results in a distribution drift, leading to worse performance.

Model comparison. Analyzing the Uptake results for each model displayed in
Table 3 and Table 4, we observe that Uptake consistently demonstrates
different performance improvements compared to the Certain approach across
various models. In general, each model in our experiment experiences
enhancements when using Uptake. Specifically, RNN exhibits the most
substantial improvement, achieving an average F1-score increase of $7.02\%$
across all datasets, while LSTM shows the smallest improvement, with an
average F1-score increase of $1.64\%$. However, when compared with offline
updating, there isn’t a consistent pattern across all datasets concerning
model differences. This discrepancy can be attributed to biases introduced in
the data distribution among different stages, affecting the performance of
various models differently. In summary, Uptake consistently outperforms
Certain model updating approaches and approaches the performance of the
offline (i.e., the upper bound) across different prediction models. This
consistency underscores its generality and robustness in enhancing model
performance.

###  5.3. RQ4: Online Performance

We have deployed Uptake to one of the top-tier cloud systems in the world,
Microsoft Azure, which suffer from the UPLearning problem before using our
approach. We have conducted an online experiment (in the A/B testing
environment to obtain ground truth) for a period of time over five weeks from
July 2022 to August 2022.

Table 5 presents a comparison of model performance for the mentioned model
updating approaches, following the same format as the tables in Section 5.2.
Uptake outperforms Certain, achieving an average F1-score of $69.33\%$. This
performance surpasses Certain by an average of $7.09\%$ and falls slightly
below the Offline performance by $3.2\%$ in terms of F1-score.

The accumulation of uncertain positive labels significantly impacts the
performance of node failure prediction. Unlike failure prediction on public
disk datasets, the node failure prediction model is more concerned with the
quality of labels rather than the proportion of positive samples to negative
samples. In this context, the accuracy and reliability of labels play a
crucial role in determining the effectiveness of the prediction model.

In summary, Uptake excels in handling the UPLearning problem compared to
previous online updating approaches. Additionally, it proves effective in
mitigating performance degradation over time, showcasing its robustness and
reliability in real-world applications.

###  5.4. RQ5: Robustness

Figure 4. The average F1-score comparison of Uptake and its two compared
approaches on three datasets: Alibaba, Backblaze, and Microsoft Azure. Each
sub-figure corresponds to a specific type of base model. The error bar in the
figure represents the upper and lower bounds of the results, indicating the
robustness of Uptake across different base models.

To further empirically evaluate the performance of Uptake, we counted the
results of Uptake for different base models (i.e., the model in Section 5.1.2)
on different cloud systems (i.e., Alibaba, Backblaze and Microsoft Azure).

The provided Figure 4 displays the average F1-score across all time phases for
different model update strategies on diverse base models and datasets. It is
essential to clarify that ”Offline” denotes the theoretical optimal
performance of the current system at a given time, assuming ideal conditions.
However, achieving this theoretical best performance in practical applications
is hindered by the UPLearning problem.

Figure 4 clearly demonstrates Uptake’s superior performance over other model
update strategies across all scenarios and models. Notably, it closely
approaches the theoretical best performance (Offline). Uptake exhibits greater
stability over time, as indicated by the smaller fluctuations in performance
metrics, highlighting its robustness and reliability compared to other model
updating strategies.

Table 6. Efficiency comparison of Uptake and its two compared approaches on
Alibaba, Backblaze and Microsoft Azure dataset. Each value indicates the
average time (s) taken of the training epoch in the model updating process.

| Offline | Certain | Uptake
---|---|---|---
Alibaba | 5.38 | 12.58 | 4.31
Backblaze | 4.38 | 12.75 | 4.12
Microsoft Azure | 11.54 | 27.90 | 10.95

###  5.5. RQ6: Efficiency

Besides predicting performance, efficiency is also a critical metric for
online model updating. Consequently, we compare the running time of Uptake
with other approaches proposed in Section 5.1.5. Since the difference between
different approaches only exists in the training step, we compared the average
time cost of an epoch with the same hyper-parameters on each time phase.

From the results in Table 6, these approaches perform almost similarly in
efficiency except Certain, and Uptake take the least time to train during an
epoch with $4.31$s in Alibaba, $4.12$s in Backblaze, and $10.95$s in Microsoft
Azure. The Certain approach is slowest because Certain accumulates instances
with certain labels in all former phases, which leads to a larger training
set. The larger the dataset it uses, the longer time it takes to train.

(a) Alibaba (b) Backblaze

Figure 5. F1-score of Uptake under different $\pi_{\rm p}$ on two public
datasets. The dots are the parameters chosen by our solution.

###  5.6. RQ7: Parameter Sensitivity

We investigate the impact of the only parameter $\pi_{\rm p}$ used in Uptake.
It reflects the ratio of positive instances in all uncertain positive
instances. Our algorithm suggests using the Precision value on the T1 stage to
set $\pi_{\rm p}$, which is an estimation using historical information.

In this experiment, we grid search the value of $\pi_{\rm p}$ from 0 to 1 on a
0.1 base step. Figure 5 shows the effectiveness of different $\pi_{\rm p}$ in
terms of F1-score. In this figure, each line is the average F1-score of the
prediction model (RNN, LSTM, Transformer, and TCNN) on the Alibaba and
Backblaze datasets, respectively. The dots represent the $\pi_{\rm p}$ chosen
by our solution. Although there exists a small perturbation under different
$\pi_{\rm p}$, the overall performance is stable, and our selected $\pi_{\rm
p}$ always leads to almost the best performance, which indicates Uptake is
robust in practice without the need to tune parameters carefully.

##  6\. Discussions

###  6.1. Threats to validity

Internal threats. Our implementation choices are a potential internal threat.
To address this, we utilize established implementations for our deep learning
models, detailed in Section 5.1.2. Moreover, our implementation is openly
accessible, ensuring transparency and enabling future research replication.

External threats. External threats stem from model and data selection, as well
as comparisons with other methods. We mitigate selection bias by incorporating
diverse datasets from three distinct cloud systems, covering both node and
disk-level data. These datasets align with industry standards, enhancing the
representativeness of our results. Additionally, this study pioneers
investigations into the UPLearning problem, selecting standard model updating
strategies and a theoretical optimal strategy, as explained in Section 5.1.5.
We remain open to exploring new cloud component data and model updating
techniques in future research to enhance our study’s depth.

Construct threats. Construct threats concern our chosen metrics and
parameters. We utilize widely-accepted metrics such as precision, recall, and
F1-score for effectiveness evaluation, and training time for efficiency
measurement. Future evaluations will incorporate additional metrics for a more
comprehensive assessment. Parameters in Uptake are defined based on
established rules (Section 5.1.3), with detailed discussions about the primary
parameter, $\pi_{\rm p}$, in Section 5.6, ensuring a well-informed evaluation
framework.

###  6.2. Deployment

Our Uptake framework is deployed on Microsoft Azure, a platform with millions
of nodes serving a wide customer base. The process consists of three core
phases: data preparation, model retraining, and model deployment.

Data preparation. During this phase, collected data is cleaned and engineered
to ensure completeness and quality. These steps are essential for enhancing
the reliability and effectiveness of subsequent model retraining.

Model retraining. In this stage, the failure prediction model undergoes
retraining. To address challenges related to the UPLearning problem, Uptake is
integrated into the retraining process, mitigating issues associated with the
UPLearning problem.

Model deployment. The retrained model is deployed on AzureML, a platform
designed for seamless management and deployment of online models. Once
deployed, the model actively performs online cloud failure prediction,
providing insights into potential failure events within the cloud system.

To gauge Uptake’s effectiveness in terms of business impact, we conduct A/B
testing, measuring reduced mitigation action times and improved service
availability. Compared to the online retraining strategy discussed in Section
3, Uptake significantly reduced required mitigation actions and enhanced
service availability. These results demonstrate Uptake’s substantial benefits
for online failure prediction models, highlighting its effectiveness and
positive business impact.

##  7\. Related work

Failure prediction. In recent years, various approaches for predicting cloud
failure have appeared, including hard drive disk failure and node failure
prediction. As a binary classification problem, machine learning and deep
learning mechanisms are widely used for failure prediction. Machine learning
approaches for cloud failure prediction, such as support vector machine (36)
and tree models (16; 5; 12; 34), use several monitoring metrics collected in a
time window to predict whether the cloud component will fail soon. However,
these machine learning approaches struggle to handle the complex temporal
information of cloud systems (30). Deep learning approaches such as RNN (32),
LSTM (19) and TCNN (30) can better capture the temporal correlation of the
complex monitoring metrics than classical machine learning approaches. In
recent years, Transformers have outperformed conventional deep-learning
approaches. The state-of-the-art performance is achieved by NTAM (24), which
incorporates both temporal and spatial information into failure prediction.
Our research is orthogonal to previous failure prediction approaches since we
aim to solve the model updating issue and boost the overall performance of
failure prediction.

Uncertain Labels. Positive-Unlabeled Learning (PULearning) also solves
uncertain/unlabeled problems for classification tasks (11; 26; 18; 21; 17; 14;
20). PULearning defines positive samples and unlabeled ones for the training
phase. For example, an unbiased risk estimator (9) is proposed to solve this
problem. Different from PULearning, the UPLearning is identified in the model
updating phase, where the samples for retraining may have three classes:
positive samples, negative samples, and uncertain positive samples.

##  8\. Conclusion

Cloud failure prediction is an important task to ensure the reliability of
cloud systems. According to our real-world practice of deploying cloud failure
prediction models, we identify a problem, Uncertain Positive Learning
(UPLearning), during the model updating procedure. This problem is of great
importance to tackling since it downgrades the prediction performance
significantly in the real-world scenario. In this paper, we propose a novel
model updating approach, Uncertain Positive Learning risk estimator, dubbed
Uptake, to improve the performance of any prediction model, such as RNN, LSTM,
Transformer, and TCNN. Our experiments on both public and real-world cloud
datasets demonstrate that Uptake can robustly achieve much better performance
than the baseline approaches. More encouragingly, Uptake has been successfully
applied to our cloud platforms and obtained benefits in real practice.

## References

  * (1)
  * Alhilal et al. (2022) Ahmad Alhilal, Tristan Braud, Bo Han, and Pan Hui. 2022.  Nebula: Reliable low-latency video transmission for mobile cloud gaming. In _Proceedings of the international conference on World Wide Web (WWW)_. 3407–3417.
  * Babaioff et al. (2022) Moshe Babaioff, Ronny Lempel, Brendan Lucier, Ishai Menache, Aleksandrs Slivkins, and Sam Chiu-wai Wong. 2022.  Truthful Online Scheduling of Cloud Workloads under Uncertainty. In _Proceedings of the international conference on World Wide Web (WWW)_. 151–161.
  * Backblaze (2022) Backblaze. 2022.  The Backblzae Hard Drive Data and Stats.  https://www.backblaze.com/b2/hard-drive-test-data.html.
  * Botezatu et al. (2016) Mirela Madalina Botezatu, Ioana Giurgiu, Jasmina Bogojeska, and Dorothea Wiesmann. 2016.  Predicting Disk Replacement towards Reliable Data Centers. In _Proceedings of the Knowledge Discovery and Data Mining (SIGKDD)_. ACM, 39–48.
  * Chen et al. (2019) Yujun Chen, Xian Yang, Qingwei Lin, Hongyu Zhang, Feng Gao, Zhangwei Xu, Yingnong Dang, Dongmei Zhang, Hang Dong, Yong Xu, Hao Li, and Yu Kang. 2019.  Outage Prediction and Diagnosis for Cloud Service Systems. In _Proceedings of the International World Wide Web Conferences (WWW)_. ACM, 2659–2665.
  * Clark et al. (2005) Christopher Clark, Keir Fraser, Steven Hand, Jacob Gorm Hansen, Eric Jul, Christian Limpach, Ian Pratt, and Andrew Warfield. 2005.  Live migration of virtual machines. In _Proceedings of the 2nd conference on Symposium on Networked Systems Design & Implementation-Volume 2_. 273–286.
  * Du and Li (2018) Xiaoming Du and Cong Li. 2018.  Memory Failure Prediction Using Online Learning. In _Proceedings of the International Symposium on Memory Systems (MEMSYS)_ (Alexandria, Virginia, USA) _(MEMSYS ’18)_. ACM, New York, NY, USA, 38–49.  https://doi.org/10.1145/3240302.3240309
  * Du Plessis et al. (2014) Marthinus C Du Plessis, Gang Niu, and Masashi Sugiyama. 2014.  Analysis of learning from positive and unlabeled data.  _Proceedings of the Advances in Neural Information Processing Systems (NeurIPS)_ 27 (2014).
  * du Plessis et al. (2014) Marthinus C du Plessis, Gang Niu, and Masashi Sugiyama. 2014.  Analysis of Learning from Positive and Unlabeled Data. In _Proceedings of the Advances in Neural Information Processing Systems (NeurIPS)_ , Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger (Eds.), Vol. 27. MIT Press.
  * Elkan and Noto (2008) Charles Elkan and Keith Noto. 2008.  Learning classifiers from only positive and unlabeled data. In _Proceedings of the SIGKDD international conference on Knowledge discovery and data mining (SIGKDD)_. ACM, 213–220.
  * Huang (2017) Xiaohong Huang. 2017.  _Hard Drive Failure Prediction for Large Scale Storage System_.  Ph. D. Dissertation. UCLA.
  * Kiryo et al. (2017) Ryuichi Kiryo, Gang Niu, Marthinus C Du Plessis, and Masashi Sugiyama. 2017.  Positive-unlabeled learning with non-negative risk estimator.  _Proceedings of the Advances in Neural Information Processing Systems (NeurIPS)_ 30 (2017).
  * Lee and Liu (2003) Wee Sun Lee and Bing Liu. 2003.  Learning with positive and unlabeled examples using weighted logistic regression. In _Proceedings of the International Conference on Machine Learning (ICML)_ , Vol. 3. 448–455.
  * Levy et al. (2020) Sebastien Levy, Randolph Yao, Youjiang Wu, Yingnong Dang, Peng Huang, Zheng Mu, Pu Zhao, Tarun Ramani, Naga Govindaraju, Xukun Li, et al. 2020\.  Predictive and Adaptive Failure Mitigation to Avert Production Cloud $\\{$VM$\\}$ Interruptions. In _Proceedings of the Operating Systems Design and Implementation (OSDI)_. USENIX, 1155–1170.
  * Li et al. (2014) Jing Li, Xinpu Ji, Yuhan Jia, Bingpeng Zhu, Gang Wang, Zhongwei Li, and Xiaoguang Liu. 2014.  Hard Drive Failure Prediction Using Classification and Regression Trees. In _Proceedings of the International Conference on Dependable Systems and Networks (DSN)_. IEEE/IFIP, 383–394.
  * Li and Liu (2003) Xiaoli Li and Bing Liu. 2003.  Learning to classify texts using positive and unlabeled data. In _Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence (IJCAI)_ , Vol. 3. 587–592.
  * Li et al. (2009) Xiao-Li Li, Philip S Yu, Bing Liu, and See-Kiong Ng. 2009.  Positive unlabeled learning for data stream classification. In _Proceedings of the SIAM international conference on data mining (SDM)_. SIAM, 259–270.
  * Lin et al. (2018) Qingwei Lin, Ken Hsieh, Yingnong Dang, Hongyu Zhang, Kaixin Sui, Yong Xu, Jian-Guang Lou, Chenggang Li, Youjiang Wu, Randolph Yao, et al. 2018\.  Predicting node failure in cloud service systems. In _Proceedings of the Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE)_. ACM, 480–490.
  * Liu et al. (2003) Bing Liu, Yang Dai, Xiaoli Li, Wee Sun Lee, and Philip S Yu. 2003.  Building text classifiers using positive and unlabeled examples. In _Proceedings of the International Conference on Data Mining (ICDM)_. IEEE, 179–186.
  * Liu et al. (2002) Bing Liu, Wee Sun Lee, Philip S Yu, and Xiaoli Li. 2002.  Partially supervised classification of text documents. In _Proceedings of the International Conference on Machine Learning (ICML)_ , Vol. 2. Sydney, NSW, 387–394.
  * Liu et al. (2022) Yudong Liu, Hailan Yang, Pu Zhao, Minghua Ma, Chengwu Wen, Hongyu Zhang, Chuan Luo, Qingwei Lin, Chang Yi, Jiaojian Wang, et al. 2022\.  Multi-task Hierarchical Classification for Disk Failure Prediction in Online Service Systems. In _Proceedings of the SIGKDD Conference on Knowledge Discovery and Data Mining (SIGKDD)_. 3438–3446.
  * Lu et al. (2020) Sidi Lu, Bing Luo, Tirthak Patel, Yongtao Yao, Devesh Tiwari, and Weisong Shi. 2020.  Making Disk Failure Predictions SMARTer!. In _Proceedings of the Conference on File and Storage Technologies (FAST)_. USENIX, 151–167.
  * Luo et al. (2021) Chuan Luo, Pu Zhao, Bo Qiao, Youjiang Wu, Hongyu Zhang, Wei Wu, Weihai Lu, Yingnong Dang, Saravanakumar Rajmohan, Qingwei Lin, and Dongmei Zhang. 2021.  NTAM: Neighborhood-Temporal Attention Model for Disk Failure Prediction in Cloud Platforms. In _Proceedings of the International World Wide Web Conferences (WWW)_. ACM, 1181–1191.
  * Ma et al. (2018) Minghua Ma, Shenglin Zhang, Dan Pei, Xin Huang, and Hongwei Dai. 2018.  Robust and rapid adaption for concept drift in software system anomaly detection. In _Proceedings of the International Symposium on Software Reliability Engineering (ISSRE)_. IEEE, 13–24.
  * Nguyen et al. (2011) Minh Nhut Nguyen, Xiao-Li Li, and See-Kiong Ng. 2011.  Positive Unlabeled Learning for Time Series Classification. In _Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence (IJCAI)_ (Barcelona, Catalonia, Spain) _(IJCAI’11)_. AAAI Press, 1421–1426.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019.  PyTorch: An Imperative Style, High-Performance Deep Learning Library.  In _Proceedings of the Advances in Neural Information Processing Systems (NeurIPS)_. Vol. 32. MIT Press, 8024–8035.
  * Plessis et al. (2015) Marthinus Du Plessis, Gang Niu, and Masashi Sugiyama. 2015.  Convex Formulation for Learning from Positive and Unlabeled Data. In _Proceedings of the 32nd International Conference on Machine Learning (ICML)_ _(Proceedings of Machine Learning Research, Vol. 37)_ , Francis Bach and David Blei (Eds.). PMLR, Lille, France, 1386–1394.
  * Sets (2022) Tianchi: Data Sets. 2022.  Large-scale Disk Failure Prediciton Dataset.  https://tianchi.aliyun.com/dataset/dataDetail?dataId=70251.
  * Sun et al. (2019) Xiaoyi Sun, Krishnendu Chakrabarty, Ruirui Huang, Yiquan Chen, Bing Zhao, Hai Cao, Yinhe Han, Xiaoyao Liang, and Li Jiang. 2019.  System-Level Hardware Failure Prediction using Deep Learning. In _Proceedings of the Design Automation Conference (DAC)_. ACM, 20.
  * Wikipedia contributors (2022) Wikipedia contributors. 2022.  S.M.A.R.T. — Wikipedia, The Free Encyclopedia.  https://en.wikipedia.org/w/index.php?title=S.M.A.R.T.&oldid=1109378579 [Online; accessed 28-September-2022].
  * Xu et al. (2016) Chang Xu, Gang Wang, Xiaoguang Liu, Dongdong Guo, and Tie-Yan Liu. 2016.  Health Status Assessment and Failure Prediction for Hard Drives with Recurrent Neural Networks.  _Transactions on Computers_ 65, 11 (2016), 3502–3508.
  * Xu et al. (2021) Hang Xu, Ning Kang, Gengwei Zhang, Chuanlong Xie, Xiaodan Liang, and Zhenguo Li. 2021.  NASOA: Towards Faster Task-Oriented Online Fine-Tuning With a Zoo of Models. In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_. 5097–5106.
  * Xu et al. (2018) Yong Xu, Kaixin Sui, Randolph Yao, Hongyu Zhang, Qingwei Lin, Yingnong Dang, Peng Li, Keceng Jiang, Wenchi Zhang, Jian-Guang Lou, Murali Chintalapati, and Dongmei Zhang. 2018.  Improving Service Availability of Cloud Systems by Predicting Disk Error. In _Proceedings of the USENIX Annul Technical Conference (ATC)_. USENIX, 481–494.
  * Zeng et al. (2022) Liekang Zeng, Peng Huang, Ke Luo, Xiaoxi Zhang, Zhi Zhou, and Xu Chen. 2022.  Fograph: Enabling Real-Time Deep Graph Inference with Fog Computing. In _Proceedings of the international conference on World Wide Web (WWW)_. 1774–1784.
  * Zhang et al. (2018b) Jianguo Zhang, Ji Wang, Lifang He, Zhao Li, and Philip S. Yu. 2018b.  Layerwise Perturbation-Based Adversarial Training for Hard Drive Health Degree Prediction. In _Proceedings of the International Conference on Data Mining (ICDM)_. IEEE, 1428–1433.
  * Zhang et al. (2018a) Shenglin Zhang, Ying Liu, Weibin Meng, Zhiling Luo, Jiahao Bu, Sen Yang, Peixian Liang, Dan Pei, Jun Xu, Yuzhi Zhang, et al. 2018a.  Prefix: Switch failure prediction in datacenter networks.  _Proceedings of the ACM on Measurement and Analysis of Computing Systems (SIGMETRICS)_ 2, 1 (2018), 1–29.
  * Zhou et al. (2018) Hao Zhou, Ming Chen, Qian Lin, Yong Wang, Xiaobin She, Sifan Liu, Rui Gu, Beng Chin Ooi, and Junfeng Yang. 2018.  Overload Control for Scaling WeChat Microservices. In _Proceedings of the Symposium on Cloud Computing (SoCC)_ (Carlsbad, CA, USA) _(SoCC ’18)_. ACM, New York, NY, USA, 149–161.  https://doi.org/10.1145/3267809.3267823