# vPALs: Towards Verified Performance-aware Learning System
For Resource Management

Guoliang He111Work done as Intern at Huawei Edinburgh Research Centre.,2,
Gingfung Yeung1, Sheriffo Ceesay1, Adam Barker1,3

###### Abstract

Accurately predicting task performance at runtime in a cluster is advantageous
for a resource management system to determine whether a task should be
migrated due to performance degradation caused by interference. This is
beneficial for both cluster operators and service owners. However, deploying
performance prediction systems with learning methods requires sophisticated
safeguard mechanisms due to the inherent stochastic and black-box natures of
these models, such as Deep Neural Networks (DNNs). Vanilla Neural Networks
(NNs) can be vulnerable to out-of-distribution data samples that can lead to
sub-optimal decisions. To take a step towards a safe learning system in
performance prediction, We propose vPALs that leverage well-correlated system
metrics, and verification to produce safe performance prediction at runtime,
providing an extra layer of safety to integrate learning techniques to cluster
resource management systems. Our experiments show that vPALs can outperform
vanilla NNs across our benchmark workload.

## Introduction

Cluster resource management systems should ensure users’ applications run with
a satisfactory level of performance. However, the inherent conflicting goal of
cloud operators to improve hardware resource utilization, such as sharing the
underlying hardware with applications running on the same hardware, ultimately
leads to interference between applications, which could lead to significant
performance degradation (Zhang et al. 2013; Delimitrou and Kozyrakis 2013;
Shahrad, Elnikety, and Bianchini 2021; Wang et al. 2021b).

The interference sources can originate from hardware resource contention such
as CPU caches, GPU resources, I/0 and network bandwidth (Yeung et al. 2022;
Romero and Delimitrou 2018; Chen, Delimitrou, and Martínez 2019; Fried et al.
2020). To anticipate performance degradation at runtime, there has been a
large interest in applying machine learning methods to performance prediction
(Bhardwaj et al. 2023; Delimitrou and Kozyrakis 2013; Grohmann et al. 2019;
Zhang et al. 2021b; Liu et al. 2022) for timely performance degradation
detection. In particular, Deep Neural Networks (DNNs) can be utilized offline
to learn application performance using historic data (Mendoza et al. 2021;
Zhang et al. 2021a).

To ensure accurate prediction, useful features should be selected to construct
the dataset for the training phase. In the context of a cluster resource
management system, useful feature sets should contain the system state (usage
metrics) collected via observability tools. Hardware usage metrics can provide
insights into resource bottlenecks but at the costs of higher overheads (Ren
et al. 2010) and delayed aggregation (Kanev et al. 2015).

Alternatively, system metrics such as CPU and memory usage are proxies of
resource bottlenecks and have a lower overhead of collection (Zhang et al.
2020; Bashir et al. 2021). Recent works explore CPU, Memory, and Network usage
to predict application performance (Grohmann et al. 2019; Zhang et al. 2021b).

Pressure Stall Information (PSI) is a new system-level metric which quantifies
in real time how contended a particular CPU, memory or IO resource is. This
allows the resource scheduler to maximise the hardware utilization without
sacrificing the health of a workload through resource contention. PSI is an
interesting candidate metric to predict an application’s performance (Weiner
et al. 2022; Lu et al. 2023), however, it remains unexplored to leverage PSI
as a feature set to predict an application’s performance.

Real computer systems which incorporate DNN predictions can inevitably produce
undesirable results due to the probabilistic nature of DNNs (Wang et al.
2018), and require system operators to propose safeguard mechanisms in
addition to ensuring desirable behavior (Wang et al. 2021b; Li et al. 2023).
It is, therefore, extremely beneficial to deploy a DNN model that is safe and
verified, in which the DNN performance prediction outputs correctness are
guaranteed.

To address the aforementioned challenges, this paper proposes vPALs, a
systematic approach that leverages appropriate system metrics to predict
application performance at runtime and combines DNN verification to produce a
trustworthy performance prediction model.

This paper makes the following key research contributions:

  * •

We have used PSI system-level metrics to learn and accurately predict the
performance of various applications.

  * •

We present a novel system that can automate the performance learning and
verification process at scale.

  * •

We investigate the effectiveness of verified DNNs performance learning in
cluster resource management systems.

## Background

Existing large-scale cluster management systems such as Kubernetes (Burns et
al. 2016) provide mechanisms to collect metrics related to both system and
hardware. However, the resource management system does not provide fine-
grained hardware metrics information in real-time due to the high overhead of
low-level profiling (Ren et al. 2010; Wang et al. 2022). Furthermore,
application performance data is not readily available to end users as it is
not provided automatically by the cloud provider, and if it is set up within
the application, the metrics themselves are often delayed due to the
monitoring intervals.

This calls for a lightweight prediction approach that leverages coarse-grained
system metrics that correlate well with application performance. The remainder
of this section details the background and related work relevant to vPALS.

### Definitions

A cluster resource management system e.g., Kubernetes is a software framework
that manages a cluster of connected machines
$\mathcal{N}=\\{n_{1},...,n_{k}\\}$. A Job (application) $j$ $\in$
$\mathcal{J}$ comprises one or many tasks $\tau_{j}$. Each task can have one
or many replicas $r_{\tau,j}$. A cluster resource management system is
responsible for assigning a task replica $r_{\tau,j}$ to a machine $n$.

During the task replica $r_{\tau,j}$ execution lifetime, the monitoring
subsystem collects a set of metrics $M$ from the machine $n$ per timestep $t$,
$M_{n,t}$. Our goal is to find a subset $m$ $\subset$ $M$ that can give us a
reasonable prediction output that maps $m_{n,t}$ to a task replica performance
$p_{r,t}$. Specifically, we aim to learn a function $f$ where
$f(m_{n,t})\approx p_{r,t}$. One of the ongoing challenges of performance
prediction is to identify a suitable subset of $m$.

### Pressure Stall Information (PSI)

A set of well-correlated metrics $m$ should clearly reflect resource
saturation in a system at time $t$. The Linux OS kernel exposes a specific set
of resource saturation metrics called the Pressure Stall Information (PSI)
(Weiner 2018). PSI metrics represent the amount of lost work due to resource
pressure or contention in a time interval $t_{1}\ldots t_{h}$, i.e., no
resources are available within that time window. These metrics can be measured
for a single task replica $r_{\tau}$ or the entire machine $n$. PSI
particularly measures the extent to which various resources, such as CPU,
memory, and I/O, are being utilized and whether there is resource contention.

In addition, the PSI metrics have the following sub-measurements, some
$PSI_{s}$ and full $PSI_{f}$. For example, the some CPU metric $PSI^{cpu}_{s}$
tracks the percentage of time at least one (runnable) task is stalled due to
lack of CPU resources. The full memory $PSI^{mem}_{f}$ metric tracks the
percentage of time when all tasks are stalled due to a lack of memory
resources. Note that, CPU PSI metric does not contain full measurements at the
system level but exists at the cgroup level because it is impractical that at
the system level, all tasks are blocked simultaneously due to lack of CPU
resources. Numerous studies show $PSI$ metrics can correctly reflect system
pressures in real systems and thus be leveraged as an important metric for
various resource management techniques (Weiner et al. 2022; Lu et al. 2023;
Bannon et al. 2018). We, therefore, investigate PSI metrics correlation to
application performance in the motivation sectionPerformance Correlation
Analysis.

Figure 1: Neural Network Training and Verification System Overview.
Performance data and verification bounds are required to produce a
satisfactory DNN.

### Neural Networks Formal Verification

Deep neural networks (DNNs) are powerful function approximators (Bishop and
Nasrabadi 2006; Beneventano et al. 2021), and therefore they are widely
applied to prediction tasks such as classification (He et al. 2016), language
modelling (Touvron et al. 2023) and regression (Lam et al. 2022). However, the
DNNs performance is usually evaluated via its statistical properties. For
example, the percentage of classification accuracy or the mean-square-loss
(MSE) of regression tasks is commonly used to train and evaluate the
prediction performance. While these criteria are statistically appropriate, it
may be frowned upon when system developers or operators try to integrate the
DNNs into existing computer systems, where stability and availability are
highly expected. For example, 95% classification accuracy in assigning CPU
cores to Virtual Machines indicates 5% of the time, CPU resource bottlenecks
can manifest at run time, resulting in performance degradation. As such,
significant system components rely on human heuristics (Wang et al. 2021b; Li
et al. 2023) and the prediction of DNNs can only serve as references with
confidence upper and lower bounds to guarantee certain performance (Bhardwaj
et al. 2023).

Fortunately, recent advances in formal analysis suggest an alternative way to
validate DNNs for safe deployment - referred to as formal verification (Wang
et al. 2018; Liu et al. 2021). A verifier takes as input the verification
boundaries, the verification specification, and a DNN to analyze whether the
candidate DNN violates the specification in an iterative process as shown in
Figure 1. In the case of violation, counterexamples are produced and given to
the DNN for re-training. Otherwise, the candidate DNN is guaranteed to not
violate the specification. With such a verifier, a trained DNN can be safely
deployed with trust on its performance, in scenarios such as robotics and
computer systems.

Specifically, formal verification for neural networks consists of many
specifications, and the three common specifications are i) reachability - our
common interpretation of how a neural network should behave, i.e., given an
input satisfies a condition, then the corresponding output should satisfy the
condition; ii) probabilistic - where a neural network should in addition to
obey rules with some given probability, and iii) monotonicity - where the
neural network predictions should be able to map the relationships between
inputs, this is explained below. In the following definition, the neural
network is denoted as a function $f$, its inputs are $x\in\mathcal{D}_{x}$ and
outputs are $y\in\mathcal{D}_{y}$. For a pair of given input bounds,
$\mathcal{X}\subset\mathcal{D}_{x}$ and $\mathcal{Y}\subset\mathcal{D}_{y}$,
we have

$\forall{x_{0},x_{1}}\in\mathcal{X},\forall{i},x_{0}[i]\geq
x_{1}[i]\rightarrow f(x_{0})[j]\geq f(x_{1})[j]$ (1)

In this paper, we mainly focus on the monotonicity specification, as it is the
most relevant to our use case - our verification process is based on Ouroboros
(Tan, Zhu, and Guo 2021) a system designed to train verified DNNs.

## Performance Correlation Analysis

Our goal is to find an appropriate subset of metrics $m$ that correlate with a
task replica performance $p_{r,t}$. In light of recent publications that
reveal the correlation between application performance and PSI metrics, we
perform benchmarking experiments to verify the correlation relationship in our
compute cluster. Our machines are homogeneous, where each machine runs Ubuntu
20.04 Operating Systems with Linux Kernel 5.4.0-42-generic. Table 1 describes
our machine hardware setup. Our workload consists of five common benchmark
applications (Cooper et al. 2010; Chen, Delimitrou, and Martínez 2019;
Sfakianakis, Marazakis, and Bilas 2021), namely Mindspore, MySQL, Nginx,
Redis, and Solr. The workload setup and configurable properties are described
in Table 2.

Hardware | Infomation
---|---
CPU | Intel(R) Xeon(R) Gold 6266C CPU
| @ 3.00GHz
Memory | DDR4 768GiB
Disk | NVMe 3.2 TB
Table 1: Machine setup in our compute cluster. Software |  Benchmark | Performance | Configurable
---|---|---|---
|  | Metric (ms) | Properties
Nginx |  Wrk2 | Request | CPU cores, Memory
|  | Latency | Rate, Num. Connection
Mindspore |  Resnet50 | Step Time | CPU cores
Training |  Cifar10 |  |
MySQL |  Sysbench | Query | CPU cores, Memory,
|  | Latency | Num. Table, Table size
Redis |  YCSB | Query | CPU cores, Memory
|  | Latency |
Solr |  YCSB | Query | CPU cores, Memory
|  | Latency |
Table 2: Benchmark workload used in our experiments.

We need to choose the right level of PSI metric to verify the relationship
between application performance and PSI metrics. PSI metrics are measured
across CPU, Memory, IO dimensions but can be on a single process level or at
the machine $n$. Since we do not want to incur high overhead at runtime during
performance prediction, we therefore consider the metrics at the machine $n$
level, $PSI^{res}_{n}$. In addition to the PSI metrics, we also collect the
traditional system metrics such as CPU, Network and IO usage for performance
learning similar to previous works. Our benchmarking experiment methodology is
explained in the Methodology section. In total, we have generated 293 unique
colocation combinations of workload for benchmarking. Here we detail our
observation on metrics correlation across our applications.

Across our benchmark, we observe that the most important metrics are CPU, IO,
and Disk usage as listed in Table 3 and Figure 2. We omit the other metrics,
such as $PSI^{Mem}$ and Network usages, as the variation is too small to be
useful. We quantify these remaining metrics relationships via Pearson
correlation (Benesty et al. 2009) as it is the most common metric to quantify
any linear relationships between variables. Intuitively, by using this metric,
we can quantitatively investigate if there exists a positive relationship,
i.e., the higher the latency, the higher the system metrics value.

System metrics | ID
---|---
CPU waiting ($PSI^{CPU}_{s}$) | 1
IO stalled ($PSI^{IO}_{f}$) | 2
IO waiting ($PSI^{IO}_{s}$) | 3
Disk IO time ($disk$) | 4
CPU time ($cpu$) | 5
Table 3: The most significant metrics quantified by Pearson correlation.

(a) Mindspore correlation.

(b) Nginx correlation.

(c) MySQL correlation.

(d) Redis correlation.

(e) Solr correlation.

Figure 2: Mindspore and Nginx demonstrate a strong correlation between
application performance and the system metrics. Solr demonstrates a medium
correlation, while Redis and Mysql show a weak correlation between application
performance and system metrics. The red line indicates where the Pearson
coefficient is $0.5$ and the blue line is $0$.

Figure 2 plots the correlation between the workload application performance
and PSI system metrics, respectively, averaged over the runs in our compute
cluster. We can observe that $two$ out of $five$ (Mindspore, Nginx) workloads
demonstrate strong positive performance-to-system correlation (correlation
coefficient $>0.5$). While Mysql, Redis show weak performance-to-system
correlation and Solr shows a medium correlation relationship. We observe that
multiple features (CPU time $cpu$ and IO PSI $PSI^{IO}$) of Mindspore and
Nginx have $>$ or $\sim 0.5$ correlation coefficients. The observation shows
that application performance can correlate with multiple system metrics
instead of only a single type, and a weighting mechanism could be developed to
take all system metrics into account for performance estimation. Similarly,
for applications like Mysql and Redis, an individual type of system metrics is
a weak indicator to serve as a proxy for the performance metric. The
observation implies that a single type of metric to serve as a proxy for the
application performance metrics is simply not enough. The performance
predictor should consider a significant subset of system metrics $m$ at run
time. We note that there are multiple correlation methods to quantify
relationships between variables, however, our goal is to demonstrate that PSI
+ system metrics can be useful for performance learning, hence we leave the
discussion and exploration of various correlation methods to future work.

The observation inspires performance learning with multiple types of system
metrics. Given that deep neural networks are powerful black-box predictors, we
therefore want to learn the application performance end-to-end without
developing heuristics to weigh those system metrics.

At the same time, unlike existing works whose performance learners output
prediction with upper and lower bounds (Bhardwaj et al. 2023), we choose to
formally verify our performance learners. This is because verification
specification allows us to align our expectation with DNNs prediction, safely
deployed into the system, and therefore we do not need to develop extra
heuristics to deal with prediction upper and lower confidence bounds.

## vPAL System Design

In this section, we will outline the design and implementation of vPALs. We
present our data processing pipeline, and how we design our verification
specification for the DNNs.

Figure 3: vPALs data pipeline.

### Data processing pipeline

To generate offline data for training the performance learners, we implement a
data processing pipeline to perform Extract-Transform-Load data preprocessing.
Note that this is the same processing pipeline for our performance-metrics
correlation presented above. The data processing pipeline is general and can
be extended to support arbitrary workloads in the future. Currently, we
integrate five common workloads as described in Table 2. To manage workload
and monitor the applications, we leverage existing open-sourced software for
the experiment compute cluster, our software system setup is detailed in Table
4.

Software | Description | Version
---|---|---
Kubernetes | Resource Management | 1.26.2
(Burns et al. 2016) | Framework |
cAdvisor | Monitoring Agent | 0.47.2
(Google 2023) |  |
NodeExporter | Monitoring Agent | 1.5.0
(Prometheus 2023a) |  |
Prometheus | Monitoring Server | 2.42.0
(Prometheus 2023b) |  |
PromTails | Logging Agent | 2.8.3
(Grafana 2023) |  |
VictoriaLogs | Logs Storage | 0.3.0
(VictoriaMetrics 2023) |  |
Table 4: Our compute cluster software and framework setup.

#### Experiment Pipeline.

Figure 3 depicts our data processing pipeline workflow. We implement a
workload generator in Python that interacts with the Kubernetes API server to
continuously submit workload to the cluster under certain policies. An example
policy includes a modified Monte Carlo simulation (Verma, Korupolu, and Wilkes
2014), which submits tasks to the cluster as much as possible until the
cluster no longer fits or SLA violation. An alternate policy is to co-locate
multiple workloads up to a threshold $\theta$ on a compute node to investigate
the interference and performance degradation. We use the latter policy to
generate benchmarking data, as co-locating workloads up to a certain number is
a common approach in the production environment and will create pressure on
the compute node. For our experiments, we set $\theta$ to five, as it is a
common colocation factor for interference study (Xu et al. 2018).

Our metrics monitoring system includes application logs collection and is
deployed to continuously record both application performance and machine
system metrics during the experiment process with a configured collection
interval. DNN training requires a well-prepared dataset. We discuss our
preparation for the dataset below. Specifically, we implement a data cleaning
and transformation pipeline to extract data from the metrics logging system.

#### Data Cleaning.

Compute cluster workloads typically have variable lifetime (Barbalho et al.
2023). Reasons such as workloads may not get scheduled and assigned as soon as
they are submitted to the compute cluster scheduler due to lack of resources
or constraints (Zaharia et al. 2010); containerized application cold start
time which is the initial time to boot up could vary based on the application
footprint (Wang et al. 2021a). In our experiments, we try to align the
benchmark workload lifetime and limit the cold start time by pre-downloading
the images to our benchmark machines. To address the challenges and ensure the
data quality, we only keep the data points from the intervals when all
workloads are present. We also discard the data points at the beginning of the
experiment to account for warm-up time.

#### Data Transformation.

The data logging intervals have different granularity and timestamps. As a
result, we choose to average data points at per-minute intervals, because too
fine-grained intervals will introduce system noise, while too coarse-grained
intervals may reduce the available data points. In total, the cleansed dataset
contains $293$ various application configurations, and $26157$ unique training
samples. We perform MinMax scaling to pre-process data points and leverage the
standard train (80%) test (20%) split with random shuffling to prepare the
dataset. Formally, each sample data point $x_{i}$ consists of the subset of
metrics $m$ we identified in Table 3, and the target $y_{i}$ is the target
performance.

Algorithm 1 Training verified DNNs

Initial dataset $\mathcal{D}$

Initial DNNs

Successful $\leftarrow$ false

while iteration $<\epsilon$ do

loss $\leftarrow$ train(DNNs, $\mathcal{D}$)

if loss $<$ theshold & verifier(DNNs) then

Successful $\leftarrow$ True

break

else

$\mathcal{D}\leftarrow$ $\mathcal{D}+$ counterexamples

end if

end while

### Verification Specification

We aim to conduct DNN verification before deployment to introduce an extra
layer of safety. We leverage an existing DNN verifier, Ouroboros (Tan et al.
2023) to train and formally verify our performance learners. We introduce a
learner per application as it is a common practice in computing clusters for
performance learning (Bhardwaj et al. 2023). We leave the unified prediction
approach to future work, as our primary goal is to study the DNN verification
effectiveness.

An important aspect of DNN verification is the specification. Specifically, we
expect the CPU PSI and IO PSI should monotonically decrease with low
application latency. Intuitively, in a computer system, the execution of an
application should have lower latency when there are more available system
resources and the environment remains unchanged. Therefore, if the PSI system
metrics value is low, we expect the application to have lower latency.

The monotonicity specification essentially aligns the inference outputs of
DNNs according to expert knowledge. The monotonicity verification is defined
previously by Equation 1. As another example, if two SQL queries process the
same database, and one queries a subset of the other, then it is always true
that its latency should be smaller given the same environment. In our case, we
align the performance learners with our reasoning of application execution and
system resources in computing clusters. Based on the correlation and domain
knowledge, we design the verification specs tailored to each workload. For
example, for Solr, its latency is expected to monotonically increase with
respect to the second input($PSI^{IO}_{f}$) and the fourth input($disk$). The
rest of the bounds are presented in Table 5.

Software | Monotonic features | Increase
---|---|---
Mindspore |  $PSI^{CPU}_{s}$, $PSI^{IO}_{f}$, $PSI^{IO}_{s}$ | ✓
MySQL |  $PSI^{CPU}_{s}$, $PSI^{IO}_{f}$, $PSI^{IO}_{s}$ | ✓
Solr |  $PSI^{IO}_{f}$, $cpu$ | ✓
Table 5: The detailed verification specification for each workload. Redis and
Nginx are not considered for reasons listed in the Evaluation section.

We encode Table 5 in Ouroboros and leverage its specs-aware mechanism to train
verified performance learners. Recall that the verification process
iteratively adds counterexamples if the verification fails for a candidate
DNN. The addition of counterexamples to the training dataset is the same as in
Ouroboros. Having defined the verification specification, we can train and
produce verified DNNs according to the procedures listed in Algorithm 1.

Figure 4: Training Overview. The input feature vector consists of a
concatenation of system metrics and workload configurations, such as CPU cores
and memory requested.

(a) Mindspore Trainning Loss.

(b) MySQL Training Loss.

(c) Solr Training Loss.

Figure 5: Training loss versus epochs. The yellow interval is when the
verifier activates and performs verification.

#### DNN Training.

Our candidate DNNs are trained using both system metrics and workload
metadata. The input features are concatenated as a feature vector. We follow
the practice of incorporating domain-specific knowledge, as it has been shown
that leveraging domain-specific features can outperform generic feature set
(Liu et al. 2022; Mendoza et al. 2021). Furthermore, the application-specific
DNNs are designed to be efficient and simple, this is due to the design
consideration that for any performance learners that should run in real-time
should be as lightweight as possible and not incur high overhead to the
system, i.e., creating more resource contention. The application-specific DNNs
only differ across the input dimensions where the metadata features are
different. Specifically, our application-specific DNN model’s intermediate
layers and hidden dimensions are described in Table 6. We leverage ReLU
(Krizhevsky, Sutskever, and Hinton 2012) activation function and Adam (Kingma
and Ba 2014) Optimizer with a learning rate of $1e^{-4}$ for our DNNs.

# Layers | Hidden Dimensions | Workloads
---|---|---
2 | 300 | Mindspore
|  | MySQL, Redis
3 | 300 | Nginx
3 | 512 | Solr
Table 6: Fully Connected DNNs Definition.

Figure 6: Accuracy comparison between verified and unverified DNNs. Verified
DNNs outperform unverified DNNs by a margin for counterexamples set and $\sim
3\%$ on the test set.

## Evaluation

We evaluate vPALs with the specified specification bounds (Table 5), to answer
the following questions:

  * •

How does verification help performance learners?

  * •

What are the limitations of performance prediction using verification?

### Experimental Setup

#### Comparison Baselines.

We follow existing verification literature where we compare our verified DNNs
with DNNs without verification to demonstrate potential benefits.
Specifically, after we obtain a verified DNN for a specific application at a
particular epoch $e$, we also train a corresponding candidate DNN without
verification up until the specific epoch regardless of the training loss. Note
that, we set the same random seed for both training processes. Finally, we
also set a $e_{lim}$ for the verification process to ensure only reasonable
efforts are spent on it. The limit is set to 5000. Thus, we can faithfully
compare the two candidate DNNs with the same training process but only differ
in whether they participate in verification.

#### Metrics.

We use Mean-Square-Error (MSE) as the loss function to train our DNNs, as our
performance learner predicts application latency directly, i.e., a regression
task. However, MSE is not very intuitive when it comes to reasoning about the
performance of prediction. As a result, we report an additional accuracy $acc$
metric to quantify the result, this is shown as the following:

$acc:=abs(f(x)-y)<0.05$ (2)

Therefore, the overall accuracy metric is simply the percentage of correct
prediction in the data set. The accuracy threshold ($0.05$) is a
hyperparameter, and we use the same value as leveraged in Ouroboros for
regression tasks.

### Training process and verification

Figure 5 shows the training loss, and the verification stages, denoted by the
yellow interval. Note that, in our benchmarking, the performance of Nginx is
stable across our data samples with practically no differences in latencies,
hence we omit the DNN performance learning. For Redis, the training loss
cannot converge, and therefore we also omit the figure. A potential reason
could be due to the metrics we selected are of low correlation, which is
observed in Figure 2.

We can observe that as soon as the verifier is activated, the training loss
spikes. This is because counterexamples are added to the training set, and as
the name suggested, the performance learner mistakenly predicts the latency of
counterexamples, resulting in the increment of training loss. As more training
iterations proceed, the training loss gradually decreases again, indicating
the performance learner learns to predict the latency correctly even with
counterexamples. Note that the verifier is activated as soon as the training
accuracy is lower than a pre-defined threshold. We set a threshold per
application, following the same practice as Ouroboros.

We can observe that when there are stronger correlations with the subset of
metrics we identified, it the easier to verify the DNNs. As we demonstrated in
Figure 2, Mindspore has a strong performance-to-system correlation, and as a
result, the verifier successfully verifies within a few iterations. On the
other hand, MySQL has a weak performance-to-system correlation, and therefore
it takes a few hundred verification iterations to verify, while Solr needs
tens of epochs to be verified. As an extreme example, Redis cannot converge
within a reasonable number of epochs. We discuss the limitations of our
approach in a later section.

This phenomenon may present a new future direction to understanding the
verifiability of DNNs. High-quality data with strongly correlated features
update the DNNs towards a region where fewer counterexamples are located. As a
result, the verifier does not need to add many counterexamples to adjust the
dataset and the performance learner can be verified easily. This is intuitive,
as a well-balanced and well-correlated dataset that covers the entire
distribution will inevitably cover the verification specification.

### Verified Neural Networks Performance

We compare the traditional (unverified) DNNs and verified DNNs performance on
the training, test, and counterexample sets, respectively. To ensure fair
evaluation, we prepare the datasets similarly and train both networks with the
same number of epochs. Figure 6 shows the result of the comparison. We observe
that the two DNNs types’ performances in the training set and test set are
similar, with the verified DNNs having slightly higher accuracy in the test
set $\sim 3\%$, which shows the verification does not degrade the model
performance but improves it slightly.

On the other hand, the verified DNNs achieve $100\%$ accuracy in the
counterexamples set, which is reasonable because the train-and-verify loop
guarantees that the trained performance learner always satisfies the
verification specification. However, for unverified DNNs, the Mindspore
performance learner violates all counterexamples, while the MySQL performance
learner only predicts $12.5\%$ counterexamples correctly. Surprisingly, the
Solr learner achieves $82.3\%$. This indicates if we deploy unverified DNNs to
online systems, there will be cases where even though the system resources are
limited, the performance learner could predict a low latency, which may result
in violation of SLAs or inefficient resource management.

## Limitations

There are two main limitations of vPALs.

  * •

Existing verifiers cannot scale beyond several layers of MLP. While this is
not an issue at the moment, as larger networks may consume more compute
resources and bring inference overhead, if we were to leverage state-of-the-
art models (e.g., large language models) for performance learning in the
future, the verifier will not be able to verify within a finite amount of
time.

  * •

The performance learners assume to access workload metadata and use it to
construct the input vector. While this is achievable at runtime, the workload
configuration may be out-of-distribution, and we rely on the generalization
ability of DNNs to make predictions. We leave the generalizability
investigation to future work.

## Future Work and Conclusion

#### Unified learner.

Our initial approach requires a learner per application, we plan to
investigate a unified approach to enhance generalizability. One potential
approach to develop a unified learner by leveraging a shared backbone that
takes in intermediate features from domain-specific heads.

#### Uncertainty-aware.

Uncertainty-aware learning approaches are beneficial in producing outputs
containing both the upper and lower bound values (Gal and Ghahramani 2016). A
potential direction is to include verification for uncertainty-aware learning
methods, i.e., guarantee the produced upper and lower bound values do not
fluctuate. We plan to also compare against uncertainty-aware learning
approaches or active learning methods to aid the safe deployment of AI in the
future.

#### Conclusion.

We present vPALs, a step towards performance learning for resource management
systems. We show that PSI metrics are indeed an important set of features for
performance learning. We show that verification does not degrade performance
and even slightly improves accuracy. We show that vanilla DNNs without
verification will fail in most counterexamples in our dataset, thus
verification for performance learning can add another safety layer in
deploying AI in resource management systems.

## References

  * Bannon et al. (2018) Bannon, D.; Moen, E.; Schwartz, M.; Borba, E.; Cui, S.; Huang, K.; Camplisson, I.; Koe, N.; Kyme, D.; Kudo, T.; et al. 2018.  Dynamic allocation of computational resources for deep learning-enabled cellular image analysis with Kubernetes.  _BioRxiv_ , 505032.
  * Barbalho et al. (2023) Barbalho, H.; Kovaleski, P.; Li, B.; Marshall, L.; Molinaro, M.; Pan, A.; Cortez, E.; Leao, M.; Patwari, H.; Tang, Z.; et al. 2023.  Virtual Machine Allocation with Lifetime Predictions.  _Proceedings of Machine Learning and Systems_ , 5.
  * Bashir et al. (2021) Bashir, N.; Deng, N.; Rzadca, K.; Irwin, D.; Kodak, S.; and Jnagal, R. 2021.  Take it to the limit: peak prediction-driven resource overcommitment in datacenters.  In _Proceedings of the Sixteenth European Conference on Computer Systems_ , 556–573.
  * Benesty et al. (2009) Benesty, J.; Chen, J.; Huang, Y.; and Cohen, I. 2009.  Pearson correlation coefficient.  In _Noise reduction in speech processing_ , 37–40. Springer.
  * Beneventano et al. (2021) Beneventano, P.; Cheridito, P.; Graeber, R.; Jentzen, A.; and Kuckuck, B. 2021.  Deep neural network approximation theory for high-dimensional functions.  arXiv:2112.14523.
  * Bhardwaj et al. (2023) Bhardwaj, R.; Kandasamy, K.; Biswal, A.; Guo, W.; Hindman, B.; Gonzalez, J.; Jordan, M.; and Stoica, I. 2023.  Cilantro: Performance-Aware Resource Allocation for General Objectives via Online Feedback.  In _17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23)_ , 623–643. Boston, MA: USENIX Association.  ISBN 978-1-939133-34-2.
  * Bishop and Nasrabadi (2006) Bishop, C. M.; and Nasrabadi, N. M. 2006.  _Pattern recognition and machine learning_ , volume 4.  Springer.
  * Burns et al. (2016) Burns, B.; Grant, B.; Oppenheimer, D.; Brewer, E.; and Wilkes, J. 2016.  Borg, Omega, and Kubernetes: Lessons learned from three container-management systems over a decade.  _Queue_ , 14(1): 70–93.
  * Chen, Delimitrou, and Martínez (2019) Chen, S.; Delimitrou, C.; and Martínez, J. F. 2019.  Parties: Qos-aware resource partitioning for multiple interactive services.  In _Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems_ , 107–120.
  * Cooper et al. (2010) Cooper, B. F.; Silberstein, A.; Tam, E.; Ramakrishnan, R.; and Sears, R. 2010.  Benchmarking cloud serving systems with YCSB.  In _Proceedings of the 1st ACM symposium on Cloud computing_ , 143–154.
  * Delimitrou and Kozyrakis (2013) Delimitrou, C.; and Kozyrakis, C. 2013.  Paragon: QoS-aware scheduling for heterogeneous datacenters.  _ACM SIGPLAN Notices_ , 48(4): 77–88.
  * Fried et al. (2020) Fried, J.; Ruan, Z.; Ousterhout, A.; and Belay, A. 2020.  Caladan: Mitigating interference at microsecond timescales.  In _14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20)_ , 281–297.
  * Gal and Ghahramani (2016) Gal, Y.; and Ghahramani, Z. 2016.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In _international conference on machine learning_ , 1050–1059. PMLR.
  * Google (2023) Google. 2023.  cAdvisor: Analyzes resource usage and performance characteristics of running containers.  Accessed: 2023-11-20.
  * Grafana (2023) Grafana. 2023.  Promtail: An agent which ships the contents of local logs to a private Grafana Loki instance or Grafana Cloud.  Accessed: 2023-11-20.
  * Grohmann et al. (2019) Grohmann, J.; Nicholson, P. K.; Iglesias, J. O.; Kounev, S.; and Lugones, D. 2019\.  Monitorless: Predicting performance degradation in cloud applications with machine learning.  In _Proceedings of the 20th international middleware conference_ , 149–162.
  * He et al. (2016) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 770–778.
  * Kanev et al. (2015) Kanev, S.; Darago, J. P.; Hazelwood, K.; Ranganathan, P.; Moseley, T.; Wei, G.-Y.; and Brooks, D. 2015.  Profiling a warehouse-scale computer.  In _Proceedings of the 42nd Annual International Symposium on Computer Architecture_ , 158–169.
  * Kingma and Ba (2014) Kingma, D. P.; and Ba, J. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_.
  * Krizhevsky, Sutskever, and Hinton (2012) Krizhevsky, A.; Sutskever, I.; and Hinton, G. E. 2012.  ImageNet Classification with Deep Convolutional Neural Networks.  _Advances in neural information processing systems_ , 25.
  * Lam et al. (2022) Lam, R.; Sanchez-Gonzalez, A.; Willson, M.; Wirnsberger, P.; Fortunato, M.; Pritzel, A.; Ravuri, S.; Ewalds, T.; Alet, F.; Eaton-Rosen, Z.; Hu, W.; Merose, A.; Hoyer, S.; Holland, G.; Stott, J.; Vinyals, O.; Mohamed, S.; and Battaglia, P. 2022.  GraphCast: Learning skillful medium-range global weather forecasting.
  * Li et al. (2023) Li, H.; Berger, D. S.; Hsu, L.; Ernst, D.; Zardoshti, P.; Novakovic, S.; Shah, M.; Rajadnya, S.; Lee, S.; Agarwal, I.; et al. 2023.  Pond: CXL-based memory pooling systems for cloud platforms.  In _Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2_ , 574–587.
  * Liu et al. (2021) Liu, C.; Arnon, T.; Lazarus, C.; Strong, C.; Barrett, C.; Kochenderfer, M. J.; et al. 2021.  Algorithms for verifying deep neural networks.  _Foundations and Trends® in Optimization_ , 4(3-4): 244–404.
  * Liu et al. (2022) Liu, L.; Shen, M.; Gong, R.; Yu, F.; and Yang, H. 2022.  Nnlqp: A multi-platform neural network latency query and prediction system with an evolving database.  In _Proceedings of the 51st International Conference on Parallel Processing_ , 1–14.
  * Lu et al. (2023) Lu, C.; Xu, H.; Ye, K.; Xu, G.; Zhang, L.; Yang, G.; and Xu, C. 2023.  Understanding and Optimizing Workloads for Unified Resource Management in Large Cloud Platforms.  In _Proceedings of the Eighteenth European Conference on Computer Systems_ , 416–432.
  * Mendoza et al. (2021) Mendoza, D.; Romero, F.; Li, Q.; Yadwadkar, N. J.; and Kozyrakis, C. 2021.  Interference-aware scheduling for inference serving.  In _Proceedings of the 1st Workshop on Machine Learning and Systems_ , 80–88.
  * Prometheus (2023a) Prometheus. 2023a.  Node Exporter: Prometheus exporter for hardware and OS metrics exposed by *NIX kernels.  Accessed: 2023-11-20.
  * Prometheus (2023b) Prometheus. 2023b.  Prometheus: An open-source systems monitoring and alerting toolkit.  Accessed: 2023-11-20.
  * Ren et al. (2010) Ren, G.; Tune, E.; Moseley, T.; Shi, Y.; Rus, S.; and Hundt, R. 2010.  Google-Wide Profiling: A Continuous Profiling Infrastructure for Data Centers.  _IEEE Micro_ , 65–79.
  * Romero and Delimitrou (2018) Romero, F.; and Delimitrou, C. 2018.  Mage: Online and interference-aware scheduling for multi-scale heterogeneous systems.  In _Proceedings of the 27th International Conference on Parallel Architectures and Compilation Techniques_ , 1–13.
  * Sfakianakis, Marazakis, and Bilas (2021) Sfakianakis, Y.; Marazakis, M.; and Bilas, A. 2021.  Skynet: Performance-driven resource management for dynamic workloads.  In _2021 IEEE 14th International Conference on Cloud Computing (CLOUD)_ , 527–539. IEEE.
  * Shahrad, Elnikety, and Bianchini (2021) Shahrad, M.; Elnikety, S.; and Bianchini, R. 2021.  Provisioning Differentiated Last-Level Cache Allocations to VMs in Public Clouds.  In _Proceedings of the ACM Symposium on Cloud Computing_ , SoCC ’21, 319–334. New York, NY, USA: Association for Computing Machinery.  ISBN 9781450386388.
  * Tan et al. (2023) Tan, C.; Liu, C.; Jia, Z.; and Wei, T. 2023.  Building Verified Neural Networks for Computer Systems with Ouroboros.  _Proceedings of Machine Learning and Systems_ , 5.
  * Tan, Zhu, and Guo (2021) Tan, C.; Zhu, Y.; and Guo, C. 2021.  Building Verified Neural Networks with Specifications for Systems.  In _Proceedings of the 12th ACM SIGOPS Asia-Pacific Workshop on Systems_ , APSys ’21, 42–47. New York, NY, USA: Association for Computing Machinery.  ISBN 9781450386982.
  * Touvron et al. (2023) Touvron, H.; Martin, L.; Stone, K.; Albert, P.; Almahairi, A.; Babaei, Y.; Bashlykov, N.; Batra, S.; Bhargava, P.; Bhosale, S.; et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Verma, Korupolu, and Wilkes (2014) Verma, A.; Korupolu, M.; and Wilkes, J. 2014.  Evaluating job packing in warehouse-scale computing.  In _2014 IEEE International Conference on Cluster Computing (CLUSTER)_ , 48–56.
  * VictoriaMetrics (2023) VictoriaMetrics. 2023.  VictoriaLogs: An open source user-friendly database for logs from VictoriaMetrics.  Accessed: 2023-11-20.
  * Wang et al. (2021a) Wang, A.; Chang, S.; Tian, H.; Wang, H.; Yang, H.; Li, H.; Du, R.; and Cheng, Y. 2021a.  FaaSNet: Scalable and Fast Provisioning of Custom Serverless Container Runtimes at Alibaba Cloud Function Compute.  In _2021 USENIX Annual Technical Conference (USENIX ATC 21)_ , 443–457. USENIX Association.  ISBN 978-1-939133-23-6.
  * Wang et al. (2022) Wang, K.; Li, Y.; Wang, C.; Jia, T.; Chow, K.; Wen, Y.; Dou, Y.; Xu, G.; Hou, C.; Yao, J.; et al. 2022.  Characterizing Job Microarchitectural Profiles at Scale: Dataset and Analysis.  In _Proceedings of the 51st International Conference on Parallel Processing_ , 1–11.
  * Wang et al. (2018) Wang, S.; Pei, K.; Whitehouse, J.; Yang, J.; and Jana, S. 2018.  Efficient Formal Safety Analysis of Neural Networks.  In _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ , NIPS’18, 6369–6379. Red Hook, NY, USA: Curran Associates Inc.
  * Wang et al. (2021b) Wang, Y.; Arya, K.; Kogias, M.; Vanga, M.; Bhandari, A.; Yadwadkar, N. J.; Sen, S.; Elnikety, S.; Kozyrakis, C.; and Bianchini, R. 2021b.  Smartharvest: Harvesting idle cpus safely and efficiently in the cloud.  In _Proceedings of the Sixteenth European Conference on Computer Systems_ , 1–16.
  * Weiner (2018) Weiner, J. 2018.  PSI - Pressure Stall Information.  https://www.kernel.org/doc/html/latest/accounting/psi.html.  Accessed: 19-11-2023.
  * Weiner et al. (2022) Weiner, J.; Agarwal, N.; Schatzberg, D.; Yang, L.; Wang, H.; Sanouillet, B.; Sharma, B.; Heo, T.; Jain, M.; Tang, C.; et al. 2022.  Tmo: transparent memory offloading in datacenters.  In _Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems_ , 609–621.
  * Xu et al. (2018) Xu, R.; Mitra, S.; Rahman, J.; Bai, P.; Zhou, B.; Bronevetsky, G.; and Bagchi, S. 2018.  Pythia: Improving datacenter utilization via precise contention prediction for multiple co-located workloads.  In _Proceedings of the 19th International Middleware Conference_ , 146–160.
  * Yeung et al. (2022) Yeung, G.; Borowiec, D.; Yang, R.; Friday, A.; Harper, R.; and Garraghan, P. 2022\.  Horus: Interference-Aware and Prediction-Based Scheduling in Deep Learning Systems.  _IEEE Transactions on Parallel and Distributed Systems_ , 33(1): 88–100.
  * Zaharia et al. (2010) Zaharia, M.; Borthakur, D.; Sen Sarma, J.; Elmeleegy, K.; Shenker, S.; and Stoica, I. 2010.  Delay scheduling: a simple technique for achieving locality and fairness in cluster scheduling.  In _Proceedings of the 5th European conference on Computer systems_ , 265–278.
  * Zhang et al. (2021a) Zhang, L. L.; Han, S.; Wei, J.; Zheng, N.; Cao, T.; Yang, Y.; and Liu, Y. 2021a.  Nn-meter: Towards accurate latency prediction of deep-learning model inference on diverse edge devices.  In _Proceedings of the 19th Annual International Conference on Mobile Systems, Applications, and Services_ , 81–93.
  * Zhang et al. (2020) Zhang, W.; Zheng, N.; Chen, Q.; Yang, Y.; Song, Z.; Ma, T.; Leng, J.; and Guo, M. 2020.  URSA: Precise Capacity Planning and Fair Scheduling Based on Low-Level Statistics for Public Clouds.  In _Proceedings of the 49th International Conference on Parallel Processing_ , ICPP ’20. New York, NY, USA: Association for Computing Machinery.  ISBN 9781450388160.
  * Zhang et al. (2013) Zhang, X.; Tune, E.; Hagmann, R.; Jnagal, R.; Gokhale, V.; and Wilkes, J. 2013.  CPI2: CPU performance isolation for shared compute clusters.  In _Proceedings of the 8th ACM European Conference on Computer Systems_ , 379–391.
  * Zhang et al. (2021b) Zhang, Y.; Hua, W.; Zhou, Z.; Suh, G. E.; and Delimitrou, C. 2021b.  Sinan: ML-based and QoS-aware resource management for cloud microservices.  In _Proceedings of the 26th ACM international conference on architectural support for programming languages and operating systems_ , 167–181.


# Metrics and Benchmarks for Remote Shared Controllers
in Industrial Applications

Claudio Zito1, Maxime Adjigble1, Brice D. Denoun2, Lorenzo Jamone2, Miles
Hansard2 and Rustam Stolkin1

###### Abstract

Remote manipulation is emerging as one of the key robotics tasks needed in
extreme environments. Several researchers have investigated how to add AI
components into shared controllers to improve their reliability. Nonetheless,
the impact of novel research approaches in real-world applications can have a
very slow in-take. We propose a set of benchmarks and metrics to evaluate how
the AI components of remote shared control algorithms can improve the
effectiveness of such frameworks for real industrial applications. We also
present an empirical evaluation of a simple intelligent share controller
against a manually operated manipulator in a tele-operated grasping scenario.

##  I Introduction

The exposure of humans to hostile work environments can be reduced by means of
shared controlled systems, in which the operator remotely controls a robotic
platform to perform a task in such environments, e.g. [1]. One of the first
industries which initially introduced the use of shared control systems was
the nuclear industry but, over time, many other industries have adopted these
technologies including health care, mining, military, firefighting, undersea,
construction, and space [3].

Telepresence is achieved by sensing through digital sensors information about
the task environment, and feeding back this information to the human operator
on a remote site [7]. The level of automation of the system typically depends
on the area of application. Some requires only supervision from the operator,
but in the majority of cases it requires direct manual manipulation via a
master device. In fact, the current state-of-the-art controllers in extreme
environments still heavily relies on the user-in-the-loop for cognitive
reasoning, while the robot device merely reproduces the master’s movements.
However, direct control is typically non-intuitive for the human operator and
yields to an increase of his cognitive burden, which has led many researchers
to investigate possible alternatives towards more efficient and intuitive
frameworks, e.g. [4, 1, 3].

Figure 1: The slave robot and an example of industrial objects from our
dataset. The robot is an KUKA IIWA equipped with an eye-on-hand Ensenso RGB-D
camera and a Schunk Parallel Gripper. The object are randomly selected from
different categories: deformable, transparent, metallic, combined, and
primitive 3D printed shapes.

The key insight is to add an active AI component which is context- and user-
aware so to make better decision on how to assist the operator. Context-
awareness is typically provided by reconstructing and understanding the scene,
in terms of the objects the robot has to manipulate. User-awareness is
obtained by providing as input the operator’s task to the AI component, so to
enable a more efficient interpretation of his/her inputs through the master
device.

Despite the advancements in technologies and algorithms in autonomous
robotics, e.g. [2, 11, 8, 10, 12, 9, 6], and shared controllers, e.g. [4, 1,
3], many industries has not yet embraced these new approaches, but rather
prefer to maintain out-of-date but reliable systems. This is due to a very
simple fact: the risks for the operators and the money to be invested are not
worth the benefits that a novel approach may have on paper but which has never
been properly tested on an uniform and standardised benchmark. Therefore we
argue that providing such a benchmark, a benchmark approved and standardised
by a consortium of research institutions and industries, will encourage
industrial partners to invest of the new technologies, which will lead to a
safer and efficient environments.

The benchmarks and metrics we propose in this paper are designed to evaluate
mainly two aspects of a share control algorithm: i) The ability of
extrapolating contextual information from sensing the environment to be of any
support of the operator, and ii) how the (visual, haptic) feedback are used to
influence the operator’s response. The combination of these two components
should, in fact, improve the task efficiency (number of successful executions
of the task), reduce the task effort (how long it will take to execute the
task), and robot demand attention (the time the user utilises to interact with
the interface instead of focusing on the task at hand).

Figure 2: A four-stage model of human/robot information processing. The top
row (grey) shows a classical, non-intelligent shared controller, which simply
maps the master’s movements into the slave’s movements. No context- or user-
awareness is provided. The bottom row (yellow) shows the same model for an
intelligent shared controller that is context-aware thanks to a scene
reconstruction of the task space, and user-aware thanks to a shared task which
will allowed the algorithm of interpreting the operator’s intentions, through
his/her responses, in a more helpful way. Modified from [5].

##  II Background

The main task of a shared control interface is to sense appropriate
information about the environment and provide them to the human operator. The
operator will take his or her decisions upon the information received. Figure
2 (blue model) shows a simple four-stage model of information processing for
the operator. The first stage refers to the acquisition and registration of
multiple sources of information. The second stage involves conscious
perception and retrieving processed information from memory. The third stage
is where the decision on how to act is made. The decision is obviously
influenced by the task at hand. The fourth and final stage is the
implementation of the chosen response, typically as a movement on a master
device.

A non-intelligent shared control system would receive the response of the
operator via the master device and mimic (the direction of) the movement on
the slave robot, as shown in Fig. 2 (top grey region). The shared controller
has no knowledge of the working environment and it is not aware of the task at
hand. Hence it is not capable to interpret the operator’s intentions, and the
best it can do is to reproduce the master’s movements and provide some low-
level, haptic feedback, e.g. vibrations when hitting an external surface. To
reproduce the master’s movements, the simplest interface would require the
operator to control one joint of the manipulator at the time. First the
operator manually chooses the appropriate joint to move and then increases or
decreases the joint angle via the master device. However, this type of
interfaces increase the operator’s cognitive burden and are not very efficient
especially for complex robots with high degrees of freedom (DOF) [7]. Other
interfaces control the manipulator in Cartesian space, where the master’s
movements are replicated by the robot’s end-effector w.r.t. a chosen world’s
coordinate frame. Although it is more efficient than controlling single
joints, this process still heavily relies on a visual servoing from the
operator.

Figure 3: Graphical representation of a remote shared control setup. The
operator receives a visual feedback of the task environment (shaded blue area
above the tabletop) and controls the slave robot via a master device. The
operator input is translated in movements for the slave robot according to the
shared control algorithm. Haptic feedback can also be provided to guide the
operator according to the shared control algorithm.

Figure 2 (bottom row, green model) shows the relative four-stage model for an
intelligent shared controller and how the operator/robot share such
information (black dotted arrows). The first stage involves capturing one or
multiple view of the task space. Typically this is done by acquiring RGB-D
images from pre-selected poses from the eye-on-hand camera. If multiple images
are taken, the system registers them to create a single dense point cloud. The
second stage pre-process the point cloud to remove unnecessary surfaces, such
as the tabletop, or outliers. A decision is then made given the contextual
information available. Assuming that the AI system and the operator share the
same task, such as pick and place the objects from the tabletop, the former
can assist the decision-making process of the latter by offering available
actions. In the pick-and-place example, the system could compute a set of
candidate grasping trajectories and visualise them on the augmented feed of
the external camera, as shown in Fig 2 by the black dotted arrow from the
shared controller’s Decision Making module to the operator’s Sensory
Processing. This would provide a way to influence the operator into select the
preferred grasp. Once this process is complete, the operator and the shared
controller are working to reach and execute a selected action, of which they
are both aware. The system will interpret the operator’s commands to assist
him/her to accomplish the goal, e.g. moving along the trajectory. Finally, the
fourth stage of the shared control involves visual or haptic feedback that can
be used to communicate with the human to affect his/her response, e.g.
creating a field force on the master haptic device if the robot is moving away
from an optimal alignment with the object, jeopardising the grasping success.
An example of an intelligent shared control is presented in Fig. 4.

Figure 4: An example of an intelligent shared controller. First image on the
left: a point cloud is collected to process the workspace. Second image: a
grasp suggestion with a relative trajectory is visualised to the operator.
Third and fourth images: the operator drives the robot towards the object
following the suggested trajectory until the operator decides to send the
command to close the gripper and grasp the object.

##  III Benchmark I: Grasp Efficiency

###  III-A Objective

Our main objective in this benchmark is to evaluate how AI algorithms can
improve the efficiency of reach and pick up objects in a tele-operated
framework.

###  III-B Dataset

The dataset is composed of three sets of 3D printed primitive shaped objects,
such as cubes, pyramids, and spheres. The first set is designed to offer the
best possible scenario, in which the object should be clearly visible from a
depth camera and made of non-slippery material so not to challenge the grasps.
The second set of primitive shapes should be made of transparent plastic to
challenge the perception abilities of the system, and therefore the ability of
the algorithm to provide a candidate grasp. The third set should be covered
with a shiny and low-friction material (e.g. wax) so to challenge the
perception as well as the grasping execution.

###  III-C Task

We aim to evaluate if the algorithm can lead the operator to robustly grasps
objects. Each of the object will be presented to the robot attached to a base
via a spring. By lifting the object of ten centimeters the external force
generated by the spring will challenge the grasp.

###  III-D Metrics

Three metrics will be measured for this benchmark, as follows.

  1. 1.

Task efficiency: the efficiency of the human-robot team to perform the task,
measured as pick-and-hold success rate.

  2. 2.

Task effort: the required time to complete the task.

  3. 3.

Robot demand attention: the total amount of time that the operator has to
spend to align the end-effector with the object before making a grasp.

##  IV Benchmark II: Pick & Place

###  IV-A Objective

Our main objective for this benchmark is to evaluate how AI algorithms can
assist the operator in pick and place tasks.

###  IV-B Dataset

The dataset is composed of three sets of objects, as follows.

  * •

Soft & deformable: such as objects as gloves, wires, plastic tubes.

  * •

Metallic-like & slippery: this are objects that can be 3D printed and sprayed
so to result difficult to be sense by a depth camera. Additionally the surface
will be spread with a substance to reduce the friction and thus making the
grasping more challenging.

  * •

Composed objects: such as L-joint tubes or a net filled up with bolts. This
dataset will challenge the grasping algorithm by adding a dynamic component to
the testbed.

###  IV-C Task I

We will present a single object from the dataset to the robot. A random
position in the robot’s workspace will be chosen, but maintained constant
through the comparisons with the algorithms. A point cloud from a pre-defined
single view will be collected from the eye-in-hand camera. The operator will
need to pick the object and place it to a basket area situated at the side of
the workspace.

###  IV-D Task II

We will present a clutter scene with objects from the same class of the
dataset to the robot. A random position in the robot’s workspace will be
chosen for each object, but maintained constant through the comparisons with
the algorithms. A point cloud from a pre-defined single view will be collected
from the eye-in-hand camera. The operator will need to pick an object at the
time and place it to a basket area situated at the side of the workspace until
there are no more objects.

###  IV-E Task III

Similarly to Task II described in Sec IV-D, we will present a clutter scene to
the robot, but we will allow sampling across categories. Again a random
position in the robot’s workspace will be chosen for each object, but
maintained constant through the comparisons with the algorithms. A point cloud
from a pre-defined single view will be collected from the eye-in-hand camera.
The operator will need to pick an object at the time and place it to a basket
area situated at the side of the workspace until there are no more objects.

###  IV-F Metrics

Three metrics will be measured for this benchmark, as follows.

  1. 1.

Task efficiency: the efficiency of the human-robot team to perform the task,
measured as pick-and-place success rate.

  2. 2.

Task effort: the required time to complete the task.

  3. 3.

Robot demand attention: the total amount of time that the operator has to
spend to align the end-effector with the object before making a grasp.

##  V Benchmark III: Assembly

###  V-A Objective

Our main objective for this benchmark is to evaluate how AI algorithms can
assist the operator in manipulating objects.

###  V-B Dataset

The dataset is composed of a set of 3D printed primitive shapes and a peg
board for the respective shapes.

###  V-C Task

A CAD model of the peg board will be available to the algorithm and its pose
will be kept fixed throughout the experiments. We will present a single object
from the dataset to the robot in a pre-defined region of the dexterous
workspace of the robot manipulator. A point cloud from a pre-defined single
view will be collected from the eye-in-hand camera to localise the object in
an expected region. The operator will need to pick the object and push it
through the correct hole in the peg board.

###  V-D Metrics

Three metrics will be measured for this benchmark, as follows.

  1. 1.

Task efficiency: the efficiency of the human-robot team to perform the task,
measured as peg-in-the-hole success rate.

  2. 2.

Task effort: the required time to complete the task.

  3. 3.

Robot demand attention: the total amount of time that the operator has to
spend to align the grasped object with the correct hole on the peg board.

##  VI Experiments

###  VI-A Setup

To simulate many real-world environments in which the user is back in safety
away from where the robot operates, we proposed a setup in which the robot
manipulator and its workspace is hidden from the user. The user will be
provided with a 2D/3D visual feedback from a set of cameras and a mean to
control the manipulator. As shown in Fig. 3, two cameras will be placed in the
remote environment. One RGB-D camera is mounted on the wrist of the robot for
a so called _eye-on-hand_ prospective. A second 4K RGB camera will be placed
such that the entire dexterous workspace of the robot will be visible. The
second camera should be calibrated with the framework so that it is possible
to augment the feed from the camera with critical information from the
algorithm, e.g. a candidate grasp location or trajectory. A 6D haptic device
is used as master controller and to receive haptic feedback. The robot
manipulator is composed by a 7 degrees of freedom (DOF) robot arm mounted on a
solid base and equipped with a parallel gripper.

###  VI-B Baseline Controller

Our framework also employs a standard manually tele-operated controller in
Cartesian space, as in Fig 2 (grey section), which is commonly used in
industrial setups. The master’s movements are directly maps into the slave’s
end-effector movements in 6D with respect to a chosen world’s reference frame.
The robot gripper is controlled with an extra ON/OFF button for opening and
closing. No haptic feedback is provided to the operator. This will provide a
beseline of comparison for all the intelligent shared controller algorithm
that will employ our proposed benchmarks.

###  VI-C Benchmark I

Each algorithms, included the baseline controller, will be tested in the same
conditions. For each object in the dataset described in Sec. III-B, ten random
positions that span the entire workspace will be selected and recorded. Each
algorithm will be tested five times for each pose. This would provide a set of
fifty trials per object, and it guarantees that the performance is not biased
by the configuration of the object. The metrics presented in Sec. III-D will
be computed at the end of the experiments as average per class (i.e. standard
3D printed object, transparent, and shiny-slippery) and across classes.

###  VI-D Benchmark II

Again, each algorithms, included the baseline controller, will be tested in
the same conditions. For _Task I_ (Sec IV-C) each object in the dataset (Sec.
IV-B) will be placed in ten random positions that span the entire workspace
and the positions recorded. For _Task II_ (Sec IV-D) and _Task III_ (Sec IV-E)
ten random clutter scenes with ten objects each will be selected and recorded.
Each algorithm will be tested five times for each object or scene. This would
provide a set of fifty trials per object/scene, and it guarantees that the
performance is not biased by the configuration of the object or the selected
objects in the scene. The metrics presented in Sec. IV-F will be computed for
each task individually at the end of the experiments as average per class
(i.e. soft & deformable, metallic-like & slippery, and composed) and across
classes.

###  VI-E Benchmark III

Similarly, each algorithms, included the baseline controller, will be tested
in the same conditions. Each trial will be composed by twelve objects (i.e.
three for each shape) from the dataset described in Sec. V-B. The objects will
be presented to the operator in a random order which will be recorded. Ten
trials per algorithm will be performed. The metrics presented in Sec. V-D will
be computed as an average over the ten trials.

Figure 5: Time effort (in seconds) for teleoperating a robot manipulator for a
grasping task. We tested 10 participants. The plot shows the average
completion time to reach and grasp 5 objects with standard Cartesian
controller (blue) and a simple shared controller (red). The empirical results
show that the shared controller outperforms the baseline controller in guiding
the robot towards the grasp.

##  VII Conclusion and Future Work

We have proposed a set of benchmarks and metrics to evaluate algorithms for
intelligent remote control in industrial applications. Our proposed design
address mainly two aspects of the algorithms: i) how they map the operator’s
response into a more intuitive controller, and ii) how information are
shared/feedback with the operator. To demonstrate the benefits of these two
components, we present an empirical evaluation between a non-intelligent
Cartesian controller and a simple algorithm that follows a given grasp
trajectory for assisting the operator in pick-up tasks. The intelligent
algorithm provides a visual feedback of the proposed grasping trajectory and a
haptic feedback to constrain the movements of the operator in the direction of
the selected grasp. The operator is only able to move forward or backward
along the direction of the selected trajectory, while all the other directions
of movement are blocked. The algorithm reduces the operator’s cognitive burden
in the sense that with a degree of freedom in the master device, the operator
is controlling the slave robot in a 6D trajectory. Fig. 5 shows some
preliminary results that show how effective an intelligent shared control can
be. We measured for ten participants the task effort in a tele-operated
grasping scenario. We asked each participant to guide the slave robot into
grasping a single object presented on the tabletop in front of the robot.
Before enabling the telo-operation, the tabletop was captured by the eye-on-
hand depth camera from a single frontal view, as shown in Fig. 1. We tested a
total of five industrial objects placed in a random position, but constant
over the comparison between the algorithms and for each participant. The
objects have been randomly selected from the one shown in Fig 1. The empirical
results show that in terms of completion time the shared control outperforms
the baseline controller.

As future work, we aim to implement all the benchmarks and metrics in an open
source framework and to release the relative datasets of objects. Each dataset
will be composed of dense and complete RGB-D scan of each object.

## References

  * [1] Ron Fulbright and Larry M. Stephens.  Swami: An autonomous mobile robot for inspection of nuclear waste storage facilities.  Autonomous Robots, 2(3):225–235, 1995.
  * [2] Marek Kopicki, Renaud Detry, Maxime Adjigble, Rustam Stolkin, Ales Leonardis, and Jeremy L. Wyatt.  One-shot learning and generation of dexterous grasps for novel objects.  The International Journal of Robotics Research, 35(8):959–976, 2015\.
  * [3] Kent Yee Lui, Hyunjun Cho, ChangSu Ha, and Dongjun Lee.  First-person view semi-autonomous teleoperation of cooperative wheeled mobile robots with visuo-haptic feedback.  The International Journal of Robotics Research, 36(5-7):840–860, 2017.
  * [4] Naresh Marturi, Alireza Rastegarpanah, Chie Takahashi, Maxime Adjigble, Rustam Stolkin, Sebastian Zurek, Marek Kopicki, Mohammed Talha, Jeffrey A Kuo, and Yasemin Bekiroglu.  Towards advanced robotic manipulation for nuclear decommissioning: A pilot study on tele-operation and autonomy.  In Proc. of International Conference on Robotics and Automation for Humanitarian Applications (RAHA), 2016.
  * [5] Raja Parasuraman, Thomas B. Sheridan, and Christopher D. Wickens.  A model for types and levels of human interaction with automation.  In IEEE Transactions on systems, man, and cybernetics-Part A: Systems and Humans, pages 286–297, 2000.
  * [6] Carlos J. Rosales, Federico Spinelli, Claudio Zito, Marco Gabiccini, and Jeremy L. Wyatt.  Gpatlasrrt: A tactile exploration strategy for novel object shape modeling.  International Journal of Humanoid Robotics, Special Issue ’Tactile perception for manipulation: new progress and challenges’, 15(1), 2018\.
  * [7] Thomas B. Sheridan.  Telerobotics, Automation, and Human Supervision Control.  PhD thesis, Cambridge, MA, 1992.
  * [8] Jochen Stüber, Claudio Zito, and Rustam Stolkin.  Let’s push things forward: A survey on robot pushing.  arXiv preprint arXiv:1905.05138 [cs.RO] (cs.AI), 2019.
  * [9] Claudio Zito, Marek Kopicki, Rustam Stolkin, Christopher Borst, Florian Schmidt, Maximo A. Roa, and Jeremy L. Wyatt.  Sequential re-planning for dextrous grasping under object-pose uncertainty.  In Workshop on Manipulation with Uncertain Models, Robotics: Science and Systems (RSS), 2013.
  * [10] Claudio Zito, Marek Kopicki, Rustam Stolkin, Christopher Borst, Florian Schmidt, Maximo A. Roa, and Jeremy L. Wyatt.  Sequential trajectory re-planning with tactile information gain for dextrous grasping under object-pose uncertainty.  In IEEE Proc. Intelligent Robots and Systems (IROS), 2013.
  * [11] Claudio Zito, V. Ortienzi, M. Adjigble, Marek Kopicki, Rustam Stolkin, and Jeremy L. Wyatt.  Hypothesis-based belief planning for dexterous grasping.  arXiv preprint arXiv:1903.05517 [cs.RO] (cs.AI), 2019.
  * [12] Claudio Zito, Rustam Stolkin, Marek Kopicki, Massimiliano Di Luca, and Jeremy L. Wyatt.  Sequential re-planning for dextrous grasping under object-pose uncertainty.  In Proc. Workshop on Beyond Robot Grasping: Modern Approaches for Dynamic Manipulation. Intelligent Robots and Systems (IROS), 2012.