# Learning Interpretable Scheduling Algorithms for Data Processing Clusters

Zhibo Hu
The University of New South Wales
CSIRO Data61
Australia
<EMAIL_ADDRESS>
&Chen Wang
CSIRO Data61
The University of New South Wales
Australia
<EMAIL_ADDRESS>
&Helen Hye-Young Paik
The University of New South Wales
Australia
<EMAIL_ADDRESS>
Yanfeng Shu
CSIRO Data61
Australia
<EMAIL_ADDRESS>
&Liming Zhu
CSIRO Data61
The University of New South Wales
Australia
<EMAIL_ADDRESS>

###### Abstract

Workloads in data processing clusters are often represented in the form of DAG
(Directed Acyclic Graph) jobs. Scheduling DAG jobs is challenging. Simple
heuristic scheduling algorithms are often adopted in practice in production
data centres. There is much room for scheduling performance optimisation for
cost saving. Recently, reinforcement learning approaches (like decima) have
been attempted to optimise DAG job scheduling and demonstrate clear
performance gain in comparison to traditional algorithms. However,
reinforcement learning (RL) approaches face their own problems in real-world
deployment. In particular, their black-box decision making processes and
generalizability in unseen workloads may add a non-trivial burden to the
cluster administrators. Moreover, adapting RL models on unseen workloads often
requires significant amount of training data, which leaves edge cases run in a
sub-optimal mode. To fill the gap, we propose a new method to distill a simple
scheduling policy based on observations of the behaviours of a complex deep
learning model. The simple model not only provides interpretability of
scheduling decisions, but also adaptive to edge cases easily through tuning.
We show that our method achieves high fidelity to the decisions made by deep
learning models and outperforms these models when additional heuristics are
taken into account.

##  1 Introduction

In many data processing clusters, data analytics software stack often provides
a programming model for developers to express data parallelism so that job
scheduling algorithms can optimise the resource allocation (e.g., SparkSQL
(Armbrust et al., 2015) on Spark (Zaharia et al., 2012), Dryad (Isard et al.,
2007) and TEZ (project, 2019)).

Commonly, user applications are broken into a series of “compute” jobs where
each job contains a number of interdependent tasks. DAGs, or direct acyclic
graphs are used to represent the input-output dependency among these tasks.
Each vertex is a node that contains a number of homogeneous tasks and the edge
between two nodes shows one requires the other’s output.

Grandl et al. (2016b) have shown that in production clusters, jobs often have
large and complex DAGs. Improving the efficiency of a scheduler could save
operation costs, even a small amount of improvement in utilization can save
millions of dollars at scale (Barroso et al., 2013). However, how to schedule
these jobs for efficient resource utilization is still challenging and
conventional approaches rely on heuristic algorithms. For instance, Dryad
(Isard et al., 2007) adopts a simple greedy algorithm that allocates a vertex
that is ready to run to an available job executor under the assumption that
the job is the only workload in the cluster. Capacity Schedulers (Scheduler,
2022) simply picks a DAG that is ready to run to fit the available executor
resources. GRAPHENE (Grandl et al., 2016a) uses a two-level scheduling
strategy with the intra-DAG scheduler. It divides a DAG into sub-graphs which
are then ranked according to a heuristic scoring method that considers task
arrival times, remaining work, locality and fairness constraints.

In recent years, machine learning based methods have been increasingly used to
optimise complex operations. In database systems, SageDB(Kraska et al., 2021))
and Neo (Marcus et al., 2019)) learn optimal data access methods and database
query plans and Krishnan et al. (2018) learns to optimise join queries. In
networks, RL-cache (Kirilin et al., 2019) learns caching admission policies
for content delivery networks (CDNs), while deep reinforcement learning is
used for network planing (Zhu et al., 2021). Very recently, reinforcement
learning methods have discovered new sorting algorithms that outperform known
human benchmarks (Mankowitz et al., 2023).

Similarly, deep learning methods (Peng et al., 2021; Mao et al., 2019) have
been exploited for the resource scheduling problem. Notably, Mao et al. (2019)
develops Decima scheduler which uses

Graph Neural Networks (GNN) to learn representations of DAG jobs so that a
Deep Reinforcement Learning (DRL) framework can obtain a scheduling policy for
the complex jobs that are difficult to charaterise using traditional methods.
These deep learning based methods have shown superior performance compared to
traditional methods.

However, there is a significant gap to be filled for Reinforcement learning
(RL) based schedulers to be deployed in practice. The main problem is that the
learned models are complex and the scheduling decisions are difficult to
understand. Further, it is challenging to patch a model when problems occur on
edge cases. We observe heuristic algorithms outperform the RL model on some
individual cases, but there is no easy solution to tune an RL model to
accommodate these cases. It is difficult to know why the RL model fails on
these cases, therefore unrealistic to provide a certain number of similar
cases for training. Directly tuning a complex model may incur problems such as
"forgetting of pre-trained capabilities" (Wołczyk et al., 2024).

In this paper, we intend to address these problems by leveraging interpretable
models derived from complex RL models. We give a method to distill
interpretable, or decision tree based schedulers from the RL scheduler
behaviours and then using the interpretable schedulers for scheduling and
patching for edge cases. As RL can be viewed as doing supervised learning on
the “good data” (optimized data) (Eysenbach et al., 2020), an interpretable
scheduler is a supervised model trained on RL’s decisions made on those data.
Importantly, we show that multiple interpretable models can be derived from
the same RL model and a dynamically selection algorithm for these models leads
to the good scheduling adaptability to edge cases.

Figure 1: Example on the scheduling quality of different schedulers: Decima
and its corresponding decision tree scheduler have the same sub-optimal
scheduling trace (top right), while the decision tree scheduler fine-tuned on
additional traces such as edge cases has the improved scheduling trace for
this job DAG (bottom right).

A decision tree is sensitive to data distribution change (Liao et al., 2024),
however, this property also makes a decision tree adaptive when adding edge
cases into training. Fig. 1 shows an example of scheduling decisions made by
three models. The fine-tuned decision tree scheduler produces the best
performance for the workload.

Specifically, this paper makes the following contributions: (1) We propose a
DNN model interpretability method that utilises human knowledge to derive a
decision tree model that clearly shows how the DNN model makes the performance
gain; (2) We design a scheduler for data processing clusters that outperforms
the state-of-the-art scheduling algorithm on job completion time; (3) We
provide a solution to the problem where the RL model cannot be efficiently
fine-tuned to accommodate edge cases. By transferring the RL setting to a
supervised learning on optimized data, we are able to select and tune the
interpretable models efficiently.

##  2 Related Work

#### Data parallel cluster scheduling.

Many DAG job scheduling research either focuses on the theoretical aspect
(Shmoys et al., 1994; Chekuri et al., 2004), or adopts simple greedy
algorithms that schedules a task once its dependencies are solved and the
necessary resources are available (Hindman et al., 2011; Isard et al., 2007;
Hadoop, 2022). Decima (Mao et al., 2019) uses GNN to learn embeddings of job
DAGs and feed them to a reinforcement learning (RL) framework to obtain a
scheduling policy that fully leverages the dependency graph information.
Decima shows a clear performance advantage over greedy and fair scheduling
algorithms. LSched (Sabek et al., 2022) offers efficient inter-query and
intra-query scheduling for dynamic analytical workloads. Despite this, RL-
based schedulers like Decima and LShed are computationally expensive,
highlighting the need for more efficient ML schedulers. Recently, Jeon et al.
(2022) propose a ML scheduler that samples node priorities using a one-shot
neural network encoder and then utilizes list scheduling to create the final
schedules. This scheduler runs much faster than existing ML baselines because
the one-shot encoder can efficiently sample priorities in parallel.

#### Model interpretation.

There are two main approaches developed to explain and understand deep
learning models: post-hoc explanations and built-in explainable models. Post-
hoc explanation techniques seek insight into a trained DNN model to find links
to prediction results (Adadi and Berrada, 2018), while build-in explainable
models intentionally guide model decisions to focus certain part of the input
or relate to certain “prototype” in reasoning to achieve end-to-end
interpretability (Chen et al., 2019). Post-hoc approaches are criticised for
not faithfully reflecting what the original models do (Rudin, 2019),
undermining user trust Chu et al. (2020). Rudin (2019) suggests that high-
stakes applications should use built-in explainable models when possible.
However, a build-in explainable model, such as decision trees and rule-based
models, can only deal with problems with relatively low complexity. Recent
approaches show certain success in injecting reasoning processes into complex
models. Li et al. (2018) introduces a prototype layer in a DNN model to find
parts of training images that represent a specific class. Another approach,
_mimic learning_ (Che et al., 2016), distills soft labels from deep networks
to learn interpretable models such as Gradient Boosting Trees. Similarly, Meng
et al. (2020) uses a teacher-student training approach to obtain an
interpretable model based on a DNN model for networking systems. Compared to
networking systems, a DAG job scheduling system deals with more complex inputs
with dependency and a larger space for resource mapping. Moreover, none of
these interpretability work focus on correcting problems in original DNN
models.

##  3 Decision Tree Scheduler

As shown in Fig. 2, we use a teacher-student framework to train a simple
decision tree scheduler. The DNN-based scheduler is the teacher and the
decision tree scheduler is the student. The student model learns from the
scheduling trace of the teacher model with an objective of matching the
scheduling decisions. We construct the feature set of the decision tree using
knowledge accumulated in existing heuristic DAG scheduling literature to
achieve interpretability.

After obtaining the decision tree, we are able to improve it further by
integrating the interpretable tree with a fair scheduling strategy and further
tune the tree to adapt to jobs scheduled in a sub-optimal way. The resulting
new scheduler shows surprising performance gain against the DNN-based
scheduler.

In this section, we first describe the process of collecting the prediction
sequence/logs from the DNN model, and then detail the process of constructing
a decision tree and how to tune it for edge cases.

Figure 2: The process of extracting scheduling strategies from a DNN-based
scheduler and constructing decision trees. In the decision tree construction
example, nodes in the trace are assigned to groups and node features are
constructed based on human knowledge. The winning nodes (nodes selected to
schedule) are given by the DNN model’s scheduling decisions in each stage.

###  3.1 DNN scheduling traces

As mentioned earlier, a typical DNN scheduler for DAG jobs uses a graph neural
network to obtain representations of the DAGs. Then a policy network is
considered to obtain a schedule that minimizes a given metric such as average
JCT (job completion time). A job is represented as a set of inter-dependent
nodes. Each node contains a number of homogeneous tasks that run on data
partitions. The learned model makes scheduling decision based on scheduling
stages. A scheduling stage is triggered by a scheduling event like an executor
becoming idle, an arrival of a new job, or a resolution of a dependency.

Using Decima as an example, a scheduling decision in the trace can be
described as “node $v$ of job $i$ being selected to run on $l_{i}$ executors”.
$l_{i}$ is also referred to as the maximum parallelism for job $i$. At each
scheduling stage, we record the following information:
$(i,v,l_{i},I\setminus\\{i\\},R)$, in which $I$ is the sequence of jobs ready
to schedule at this stage. $I$ is maintained by the _job manager_ of the
scheduling system. $R$ is the executor status in the cluster. The feature set
of job $i$, denoted by $x_{i}$ can be retrieved from the _job manager_.
$x_{i}$ contains the DAG with each node in the DAG associated with attributes
like the number of remaining tasks of the node and expected execution time of
each task. It also contains the information of executors allocated to the job.
A _resource manager_ monitors the status of executors in the cluster and
handles the dispatch requests from the job manager.

A DNN scheduler computes the node embedding and DAG embedding based on the raw
features. The scheduling policy is trained to select a node to run on a number
of executors to minimise the average number of jobs in the system, hence the
average JCT of jobs. How a node is selected is not transparent in the process
as the mapping between raw features to embedding is non-linear and it is
difficult to attribute the decision to job or node level features through
dynamic embedding. To address the problem, we first construct a set of
features used in traditional scheduling heuristics, and then build an
interpretable model to mimic the DNN scheduler decisions based on these
features. The question we are asking is whether the DNN scheduling decisions
can be explained using commonly used features.

The feature set of a node includes two parts: raw features and composite
features. The raw features include the following: (1) the current number of
executors assigned to the job of the node(denoted by F1); (2) the flag
indicating whether there are executors just finish executing a task of the job
(F2); (3) the number of idle executors (F3);(4) the remaining workload of this
node (F4); (5) the number of remaining tasks of the node (F5).

The composite features aggregate information from multiple nodes starting from
a node in the DAG. These features include the following:(1) the total number
of tasks at the nodes along the longest path (or a critical path), starting
from the current node on the DAG including the current node. We apply
_Topological Sorting_ to find the longest path (F6); (2) the total remaining
workload of the nodes on the longest path starting from the current node (F7);
(3) the total number of remaining tasks of the job of the node (F8); (4) the
total remaining workload of the job to which the node belongs (F9); (5)
whether the node scheduled in the previous stage belongs to the same job of
the current node (F10). This is to exploit the locality of resource allocation
as nodes of the same job often uses the same dataset.

These features are commonly used in heuristic scheduling algorithms to
characterise the size of jobs. This is a key step to understand the DNN
scheduling strategy, which can be seen as expressing the DNN decisions with
the known vocabulary of heuristic scheduling. The values of these features are
normalised against standard computing capacity in the DNN model training and
our experiments. We will show in section 4.1 that this helps reveal the
scheduling strategy of the DNN model in a human understandable way.

###  3.2 Trace mimicking strategies

With the trace, $\\{(i,v,l_{i},I\setminus\\{i\\},R)\\}$ collected from each
scheduling stage, we can develop a model to mimic the scheduling decisions in
the trace.

The knowledge distillation method (Hinton et al., 2015) uses a teacher-student
training framework to learn a simple model based on the numerical information
in the last few layers in the DNN model. However, there are two challenges
with this approach. A well-designed DNN scheduler such as Decima (Mao et al.,
2019) often introduces certain randomness in the selection of top-$k$
candidate nodes with the highest softmax scores. The purpose is to improve the
robustness of scheduling decisions. Experiments indeed show significant
scheduling performance improvement with randomness. However, the randomness
introduces noise in the student model training, which makes it difficult to
interpret the model decisions in a consistent manner.

We therefore introduce a method that decomposes the trace mimicking task into
two parts: first, we assign nodes of jobs to fixed-sized groups (node
grouping) and build a decision tree to learn the node selection strategy
within these groups; second, we use the decision tree as a comparator and
apply it to nodes of all jobs in the queue to select the next node to run in a
scheduling stage. By doing so, we are able to achieve the interpretability
goal and the scheduling mimicking goal simultaneously. For the
interpretability, the decision tree explains why a node is selected in a given
group. For scheduling mimicking, the optimisation process minimises the
difference of node selection between the DNN-based scheduler and the student
scheduler in all scheduling stages.

1

Input : Scheduling trace containing the following: $\mathcal{J}$ – A list of
schedulable jobs; $\mathcal{V}$ – the schedulable nodes; $\mathcal{Y}_{V}$ –
the DNN model output about the selection probabilities of schedulable nodes;
$i_{prev}$ – job scheduled previously; $|g|$ – the group size.

Output : the decision tree for node selection

2

//initialise the training set

3 $L\leftarrow\\{\\}$

4 for  _each scheduling stage in the trace_ do

5 Find the node with the highest selection probability from $\mathcal{Y}_{V}$,
denoted by $v_{w}$ and its corresponding job by $i_{w}$.

6 Obtain the composite features for $v_{w}$

7 for  _$i\in\mathcal{J}\setminus\\{i_{w}\\}$_ do

//set the locality flag

8 if  _$i_{prev}==i$_ then

9 set scheduling locality of $v\in i$ as true

10 else

11 set scheduling locality of $v\in i$ as false

12 end if

13

14 end for

//add nodes to the group and add the group to the training set

15 for  _each combination $\\{v_{k}|v_{k}\in\mathcal{V}\\}|_{k=1}^{|g|-1}$_ do

16 obtain composite node features for $v_{k}$

17 add $((v_{1},...,v_{|g|-1},v_{w}),v_{w})$ to $L$

18 end for

19

20 end for

21fit a decision tree classifier $f$ on $L$ where $v_{w}$ is used as the label
for a record in $L$

22 return _the decision tree $f$_.

Algorithm 1 Decision tree construction

###  3.3 Decision tree as a node comparator

#### Decision Tree Construction

Algorithm 1 details how the decision tree is learned. The key idea is to form
groups of contrastive nodes in training. The groups are formed in fixed sizes,
denoted by $|g|$. Each group contains one node with the highest probability to
be selected by the DNN model for scheduling and $|g|-1$ nodes that are not
selected for scheduling. We call the former “winning node” and the latter
“losing nodes”. By doing so, we reduce the complexity of predicting the node
to schedule next so that an interpretable model can learn the difference
between the “winning” and “losing” class. To ensure the interpretability of
the simple model, we use a small group size of 2 (pair) or 3 (triplet). The
group size is configurable for different application needs. To reduce the
total number of groups to compare during scheduling, we adopt a pruning
strategy as follows: when a node $v$ fails to be the “winning node” in $m$
groups, it is removed from the ready node and the groups are re-assigned. With
this strategy, the running time of the scheduler will not exponentially
increase with the number of ready nodes as most of nodes are excluded during
the group assignment phase.

The feature vectors (F1 to F10) of all the nodes in each group are
concatenated together. Then the decision tree is trained by fitting the
concatenated feature vector of each group with the winner’s label in that
group. In addition, the depth of the learned decision tree is also adjustable
for obtaining different views of the DNN schduler behaviours. We keep a given
number of trees that best fit the DNN model decisions in a pool for edge case
handling to be described in Section 3.5.

#### Scheduler Construction

After training such decision trees, we integrating the best-fit interpretable
tree with a fair scheduling strategy and we call the resulting new scheduler
as _decision tree scheduler_. In a data processing cluster, the job manager
and resource manager feed job DAGs and executor status to the scheduler. For
each scheduling stage, the scheduler assigns nodes with resolved dependency
into groups. Each group contains a node selected to schedule and $|g|-1$ nodes
that are not selected to schedule in this stage. The learned decision tree
predicts the winning node in each group. A “winning counter” is maintained to
track the number a node is selected in different groups. The process repeats
until each node has been compared with other nodes in groups. The final winner
is decided by the “winning counter”. The node that receives the highest
winning count is scheduled to run on available executors. When multiple nodes
have the same number of “winning counts”, depending on the optimisation
objective, we may choose the one based on the rank of a specific feature,
e.g., when the optimisation objective is to minimise the JCT, we select a node
with the least work load to schedule among nodes with equal winning count. In
this process, the decision tree plays a role of node comparator that decides
which node to schedule.

###  3.4 Allocation of executors

In addition to predict which node to schedule, a job scheduler for data
processing cluster also needs to allocate a specific number of executors to
the node. For a Spark job, this is often done through setting the upper limit
of the number of executors allocated to a node. The executor allocation can be
learned based on the DNN scheduler trace in a similar way as the node
selection. It can also incorporate a different executor allocation strategy.
We adopt a fair scheduling strategy Ghodsi et al. (2011), which splits the
total executor number among the current remaining jobs in the system equally.
The design choice is based on our observation from the experiment. We find
that for a DNN scheduler, simultaneously learning node selection and executor
allocation does not lead to a superior scheduling performance. It is likely
due to a significantly larger search space to find an optimal combination of
node and executor number. By replacing the executor allocation with a fair
scheduling strategy, we achieve a significantly better performance than the
original DNN scheduler. Such a surprising result indicates that decomposing a
complex DNN scheduler into an interpretable scheduler and a well-understood
resource allocation strategy not only helps the deployment of the scheduler in
practice, but also realises the potential of improving the whole scheduling
performance.

###  3.5 Multiple decision trees and edge case tuning

As mentioned, we keep multiple decision trees that fit the DNN scheduler
decisions the best. Each tree contains a slightly different explanation to the
DNN scheduler’s decision making process. These interpretable trees enable
intervention to scheduling strategies when sub-optimal cases are identified. A
sub-optimal case indicates that the tree does not provide a satisfactory
explanation for the scheduling decision for a specific job. We show a better
explanation can be found through tuning the tree with the job.

The tuning process is simple: we first use a heuristic scheduler to schedule
jobs in the trace containing the specific job and obtain an improved
scheduling decisions; we then feed the trace to tune the decision trees in the
pool and evaluate their performance on the specific job; finally we select the
tree that achieves the best performance for the job and use the tree for
subsequent scheduling. Our experiment results show that the tuned tree often
accommodates the specific case by growing branches without affecting the
overall performance of other jobs.

##  4 Experiments

In this section, we evaluate our decision tree scheduler by addressing the
following questions: (1) How interpretable are the decision trees learned from
DNN models? (2) How does the scheduler perform compared to Decima and
heuristic scheduling strategies? and (3) How well does our scheduler adapt to
edge cases? For our evaluation, we use TPC-H (TPC-H, 2018) and a production
workload from Alibaba job-trace (Alibaba, 2018) as the benchmark workloads. We
also use a faithful simulator (Mao et al., 2019) as the Spark cluster testing
environment.

###  4.1 Interpretability of Decision Trees Learned from DNN

Fidelity to DNN node selection decisions. We first evaluate the fidelity of
the decision tree in node selection to the original DNN model. To ensure that
the jobs in the test set are not seen by the decision tree scheduler, we
separate jobs into two non-overlapping partitions. We run Decima on these two
partitions and from the logs, we obtain a training set and a test set for the
decision tree scheduler. Details of the datasets generated can be found in
Appendix A.1.

Table 1 shows the fidelity of the decision tree’s “winning node” choices to
those selected by Decima. We measure fidelity by accuracy, and compare the
fidelity in node selection within a group and across the entire test trace. As
shown in the table, the accuracy is high for TPC-H jobs. However, accuracy is
lower with a group size of 3, primarily because there are more candidates to
select from as the group size increases. This demonstrates that our decision
tree method can learn Decima’s scheduling decision strategy with high
fidelity, providing a largely faithful interpretation of the decisions made by
the blackbox model.

Table 1: The accuracy of selecting “winning nodes” on TPC-H. |g| | within group | across test trace
---|---|---
2 | 0.943 | 0.918
3 | 0.903 | 0.899

Job decision path distribution in decision trees. We show the decision
trajectory distribution in Fig. 3 as a use-case about how decision trees
assist users’ understanding of the scheduling decisions. Even though the trees
themselves can be complex, the trajectory distribution shows certain heavy-
tailed characteristics of decision paths. Fig. 4 gives the details of the most
frequently traversed paths for TPC-H jobs and Alibaba jobs respectively, from
which one can infer which job node features the scheduler prioritizes. For
example, while the scheduler tends to prioritize workload than executor
features for most TPC-H jobs, it prioritizes executor over workload features
for most Alibaba jobs. This level of explanation provides meaningful
information for diagnosing problems and understanding workload patterns.

Figure 3: Decision tree trajectory distribution of TPC-H jobs and Alibaba
jobs. X-axis represents the path IDs and y-axis is the number of times a
decision path is taken.

(a) TPC-H jobs

(b) Alibaba jobs

Figure 4: Most frequent decision tree paths.

###  4.2 Overall Scheduling Performance

To evaluate scheduling performance, we consider batched arrivals, where
multiple jobs start simultaneously and run to completion, and continuous
arrivals, where jobs arrive based on stochastic interarrival distributions or
a specific trace.

Batched arrivals. For TPC-H, we randomly sample 20 jobs from seven different
data sizes (2, 5, 10, 20, 50, 80, and 100GB) and 22 different DAG structures,
and run them on a Spark cluster with 20 executors; for Alibaba production
trace, we sample 10 jobs and run them on the cluster with 50 executors. Both
TPC-H and Alibaba jobs arrive in a batch, and we measure their average JCT.

Table 2: Job Completion Time (JCT) for different schedulers (in s) Scheduler | FIFO | Fair | Decima | Decision Tree $|g|=2$ | Decision Tree $|g|=3$
---|---|---|---|---|---
JCT (s) | 363.11 $\pm$ 81.08 | 263.71 $\pm$ 67.86 | 198.89 $\pm$ 46.10 | 191.87 $\pm$ 44.57 | 191.48 $\pm$ 44.71

Table 2 shows the executor allocation along time and job completion for TPC-H,
which compares the following schedulers: (a) Spark’s default FIFO scheduling;
(b) The fair scheduler that dynamically partitions executors equally between
jobs; (c) Decima; (d) The decision tree scheduler with $|g|=2$; and (e) The
decision tree scheduler with $|g|=3$. Results for the Alibaba trace are
provided in Appendix A.2.

As shown in the table, decision tree schedulers outperform non-decision tree
ones. The decision tree scheduler with $|g|=3$ improves the average JCT by 47%
compared to FIFO, 27% compared to the fair scheduler, and 4% compared to
Decima. The group size impacts performance, with a larger group size enhancing
generalizability and reducing JCT. Additionally, the executor allocation
strategy significantly influences the decision tree scheduler’s performance.
Replacing the fair allocation with Decima’s parallelism limit selection
significantly increases the average JCT. Our method combines Decima’s node
selection strategy with a heuristic-based executor allocation strategy and
achieves better performance.

We further show the results of the average JCT breakdown with the job sizes
measured by TPC-H data sizes. In addition, we train another decision tree
model on a trace that contains all job DAG structures to evaluate the
generalizability of our method.

Figure 5: Batched arrivals results on TPC-H: |g| = 2; decision tree-all uses
all DAG types for training.

Fig. 5 shows the average JCT of different models over 20 experiments. We
observe the following: 1) The JCT of the decision tree scheduler is slightly
lower than that of the fair scheduler when the sizes of TPC-H jobs are small.
The decision tree, however, shows increasing advantage when the job size
increases. It outperforms the fair scheduler by 131% with large job size. The
decision tree scheduler prioritizes small jobs, the early completion of small
jobs makes room for large jobs in the batch arrival case and leads to a
significant performance gain measured by JCT. 2) The decision tree scheduler
outperforms Decima on small jobs. On the ’2G’ and ’5G’ TPC-H queries, the
decision tree scheduler outperforms Decima by 116%. The advantage is mainly
due to that the decision tree scheduler incorporates the fair executor
allocation strategy, which is superior to the parallelism limit prediction
strategy in Decima. 3) There is no much obvious performance gap between the
decision tree trained on the whole job DAG types and the decision tree trained
on partial job DAG types. They have similar job JCTs on different query sizes.
This shows a good generalizability of our decision tree method and indicates
that a small amount of trace data can train a decision tree with good
performance.

Continuous arrivals. We also evaluate the performance of the decision tree
scheduler on streaming jobs with continuous arrivals. The result is in
Appendix A.3

###  4.3 Adapting to Edge Cases

Finally, we examine how well our decision tree scheduler adapts to edge cases
compared to Decima. We train Decima with 50 executors and 200 streaming jobs
following a 25 second Poisson job arrival interval. Using two Decima models,
one performing better than the other, we train the decision tree scheduler
($DT_{\text{base}}$) with the logs of the less-performing Decima model to
identify edge cases. Both Decima (the better performing model) and
$DT_{\text{base}}$ are then fine-tuned with edge cases over 100,000 epochs,
resulting in the fine-tuned schedulers $Decima_{\text{ft}}$ and
$DT_{\text{ft}}$ respectively. Note $DT^{*}_{\text{ft}}$ is obtained by fine-
tuning ($DT_{\text{base}}$) using scheduling traces from a fair scheduler, on
edge cases. The experiment considers 10 edge cases: 8 from TPC-H and 2 from
the Alibaba trace, detailed in Appendix A.4. Tabel 3 presents the results on
edge cases. It is evident that both Decima and $Decima^{*}_{\text{ft}}$ do not
perform well, indicating that Decima does not handle or adapt to edge cases
effectively. On the other hand, the fair scheduler performs well on all edge
cases. While $DT_{\text{base}}$ does not perform well on edge cases, its fine-
tuned schedulers yield better performance, comparable to or even surpassing
that of the fair scheduler. The table shows the performance of fine-tuned
decision tree $(DT^{*}_{\text{ft}})$ in different sizes, using $d$ to
represent the maximum tree depth and $l$ the maximum number of leaves, along
with the percentage improvement in average JCT. Decima has more randomness,
result in a larger error bar.

Table 3: Fine-tuned decision tree performance on edge cases (average JCT in
seconds, 50 times)

Edge cases | $Decima$ | $Decima^{*}_{\text{ft}}$ | Fair scheduler |  $DT_{\text{base}}$ (d: 30; l: 1000) |  $DT^{*}_{\text{ft}}$ (d: 5; l: 50) |  $DT^{*}_{\text{ft}}$ (d: 10; l: 100) |  $DT^{*}_{\text{ft}}$ (d: 20; l: 500) |  $DT^{*}_{\text{ft}}$ (d: 30; l: 1000) | JCT $\downarrow$
---|---|---|---|---|---|---|---|---|---
1 (Appendix fig 9) | 21.58 $\pm$ 3.45 | 21.62 $\pm$ 2.32 | 16.58 $\pm$ 0.32 | 18.03 $\pm$ 0.56 | 16.52 $\pm$ 0.24 | 16.20 $\pm$ 0.22 | 16.61 $\pm$ 0.39 | 16.65 $\pm$ 0.30 | 10.14%
2 (Appendix fig 10) | 22.33 $\pm$ 2.41 | 21.62 $\pm$ 3.55 | 17.66 $\pm$ 0.28 | 20.59 $\pm$ 0.85 | 17.64 $\pm$ 0.36 | 17.75 $\pm$ 0.32 | 17.97 $\pm$ 0.35 | 17.90 $\pm$ 0.38 | 14.34%
3 (Appendix fig 11) | 28.36 $\pm$ 0.41 | 28.44 $\pm$ 2.34 | 19.79 $\pm$ 0.27 | 20.86 $\pm$ 0.39 | 20.12 $\pm$ 0.32 | 19.94 $\pm$ 0.3 | 21.38 $\pm$ 0.36 | 21.26 $\pm$ 0.32 | 4.42%
4 (Appendix fig 12) | 19.68 $\pm$ 2.03 | 18.88 $\pm$ 3.33 | 15.55 $\pm$ 0.39 | 20.04 $\pm$ 0.72 | 17.74 $\pm$ 0.71 | 15.81 $\pm$ 0.48 | 15.54 $\pm$ 0.45 | 15.54 $\pm$ 0.44 | 22.49%
5 (Appendix fig 13) | 18.89 $\pm$ 1.76 | 18.72 $\pm$ 2.27 | 15.88 $\pm$ 0.33 | 17.70 $\pm$ 0.48 | 16.54 $\pm$ 0.57 | 15.98 $\pm$ 0.38 | 15.96 $\pm$ 0.56 | 15.90 $\pm$ 0.36 | 10.17%
6 (Appendix fig 14) | 20.03 $\pm$ 2.48 | 19.53 $\pm$ 0.39 | 15.43 $\pm$ 0.26 | 17.29 $\pm$ 0.41 | 15.77 $\pm$ 0.28 | 15.47 $\pm$ 0.32 | 15.51 $\pm$ 0.24 | 15.58 $\pm$ 0.23 | 10.50%
7 (Appendix fig 15) | 38.93 $\pm$ 3.65 | 39.92 $\pm$ 2.09 | 35.60 $\pm$ 0.62 | 39.43 $\pm$ 1.03 | 35.61 $\pm$ 0.35 | 35.48 $\pm$ 0.66 | 35.46 $\pm$ 0.72 | 35.67 $\pm$ 0.72 | 10.07%
8 (Appendix fig 16) | 18.63 $\pm$ 0.36 | 18.25 $\pm$ 3.41 | 15.13 $\pm$ 0.36 | 17.52 $\pm$ 0.56 | 15.29 $\pm$ 0.55 | 15.79 $\pm$ 0.35 | 15.15 $\pm$ 0.45 | 15.17 $\pm$ 0.46 | 13.57%
9 (Appendix fig 17) | 17.70 $\pm$ 1.76 | 17.22 $\pm$ 1.15 | 15.89 $\pm$ 0.43 | 16.40 $\pm$ 0.25 | 16.25 $\pm$ 0.65 | 15.89 $\pm$ 0.42 | 15.90 $\pm$ 0.26 | 15.89 $\pm$ 0.31 | 3.09%
10 (Appendix fig 18) | 3.77 $\pm$ 0.23 | 3.76 $\pm$ 0.30 | 3.39 $\pm$ 0.18 | 3.41 $\pm$ 0.12 | 3.401 $\pm$ 0.28 | 3.40 $\pm$ 0.22 | 3.38 $\pm$ 0.27 | 3.39 $\pm$ 0.12 | 0.878%

For batched arrivals, the decision tree scheduler fine-tuned with edge cases
achieves nearly performance with an average JCT of 194202.30 ms as the non-
fine-tuned version with an average JCT of 191871.99 ms of Decision tree |g|=2
for the overall workload, indicating the scheduler’s adaptability and
generalizabilty. For continuous arrivals, the fine-tuned decision tree
scheduler (|g|=2) achieves an average JCT of 56779.28ms, better than the non-
fine-tuned scheduler’s 57670.62ms (Fig. 7).

##  5 Conclusion

As deep learning is utilised increasingly in system research, it is worth
noting the significant gap between the strategy learned by a DNN model and its
deployment in practice. Specifically, the DNN model predictions are difficult
to interpret and hard to patch when problems occur. In this paper, we proposed
a method to distill interpretable models from a complex DNN model to address
the challenge in developing optimised scheduling algorithm in data processing
clusters. More importantly, we gave a method to efficiently improve the
scheduler’s performance on edge cases through interpretable model tuning. Our
decision tree scheduler outperforms the state-of-the-art DNN-based scheduler,
esp. with large jobs.

#### Limitations

Even though a decision tree is easy to tune to adapt to edge cases, we rely on
heuristic scheduling strategies to produce a reference better schedule for
edge cases. This may limit the tuning performance.

## References

  * Adadi and Berrada [2018] Amina Adadi and Mohammed Berrada.  Peeking inside the black-box: A survey on explainable artificial intelligence (xai).  _IEEE Access_ , 6:52138–52160, 2018.  doi: 10.1109/ACCESS.2018.2870052.
  * Alibaba [2018] Alibaba.  Cluster data collected from production clusters in Alibaba for cluster management research., 2018.  URL https://github.com/alibaba/clusterdata/tree/master/cluster-trace-v2018.
  * Armbrust et al. [2015] Michael Armbrust, Reynold S Xin, Cheng Lian, Yin Huai, Davies Liu, Joseph K Bradley, Xiangrui Meng, Tomer Kaftan, Michael J Franklin, Ali Ghodsi, et al.  Spark sql: Relational data processing in spark.  In _Proceedings of the 2015 ACM SIGMOD international conference on management of data_ , pages 1383–1394, 2015.
  * Barroso et al. [2013] Luiz André Barroso, Jimmy Clidaras, and Urs Hölzle.  The datacenter as a computer: An introduction to the design of warehouse-scale machines.  _Synthesis lectures on computer architecture_ , 8(3):1–154, 2013.
  * Che et al. [2016] Zhengping Che, Sanjay Purushotham, Robinder Khemani, and Yan Liu.  Interpretable deep models for icu outcome prediction.  In _AMIA annual symposium proceedings_ , volume 2016, page 371. American Medical Informatics Association, 2016.
  * Chekuri et al. [2004] Chandra Chekuri, Ashish Goel, Sanjeev Khanna, and Amit Kumar.  Multi-processor scheduling to minimize flow time with $\varepsilon$ resource augmentation.  In _Proceedings of the thirty-sixth annual ACM symposium on Theory of computing_ , pages 363–372, 2004.
  * Chen et al. [2019] Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K Su.  This looks like that: deep learning for interpretable image recognition.  _Advances in neural information processing systems_ , 32, 2019.
  * Chu et al. [2020] Eric Chu, Deb Roy, and Jacob Andreas.  Are visual explanations useful? a case study in model-in-the-loop prediction.  _arXiv preprint arXiv:2007.12248_ , 2020.
  * Eysenbach et al. [2020] Ben Eysenbach, Aviral Kumar, and Abhishek Gupta.  Reinforcement learning is supervised learning on optimized data, 2020.
  * Ghodsi et al. [2011] Ali Ghodsi, Matei Zaharia, Benjamin Hindman, Andy Konwinski, Scott Shenker, and Ion Stoica.  Dominant resource fairness: Fair allocation of multiple resource types.  In _8th USENIX Symposium on Networked Systems Design and Implementation (NSDI 11)_ , 2011.
  * Grandl et al. [2016a] Robert Grandl, Srikanth Kandula, Sriram Rao, Aditya Akella, and Janardhan Kulkarni.  G: Packing and dependency-aware scheduling for data-parallel clusters.  In _Proceedings of OSDI16: 12th USENIX Symposium on Operating Systems Design and Implementation_ , page 81, 2016a.
  * Grandl et al. [2016b] Robert Grandl, Srikanth Kandula, Sriram Rao, Aditya Akella, and Janardhan Kulkarni.  $\\{$GRAPHENE$\\}$: Packing and $\\{$Dependency-Aware$\\}$ scheduling for $\\{$Data-Parallel$\\}$ clusters.  In _12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16)_ , pages 81–97, 2016b.
  * Hadoop [2022] Hadoop.  Yarn fair scheduler, 2022.  URL https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/FairScheduler.html.
  * Hindman et al. [2011] Benjamin Hindman, Andy Konwinski, Matei Zaharia, Ali Ghodsi, Anthony D Joseph, Randy Katz, Scott Shenker, and Ion Stoica.  Mesos: A platform for $\\{$Fine-Grained$\\}$ resource sharing in the data center.  In _8th USENIX Symposium on Networked Systems Design and Implementation (NSDI 11)_ , 2011.
  * Hinton et al. [2015] Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_ , 2(7), 2015.
  * Isard et al. [2007] Michael Isard, Mihai Budiu, Yuan Yu, Andrew Birrell, and Dennis Fetterly.  Dryad: distributed data-parallel programs from sequential building blocks.  In _Proceedings of the 2nd ACM SIGOPS/EuroSys European Conference on Computer Systems 2007_ , pages 59–72, 2007.
  * Jeon et al. [2022] Wonseok Jeon, Mukul Gagrani, Burak Bartan, Weiliang Will Zeng, Harris Teague, Piero Zappi, and Christopher Lott.  Neural dag scheduling via one-shot priority sampling.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Kirilin et al. [2019] Vadim Kirilin, Aditya Sundarrajan, Sergey Gorinsky, and Ramesh K Sitaraman.  Rl-cache: Learning-based cache admission for content delivery.  In _Proceedings of the 2019 Workshop on Network Meets AI & ML_, pages 57–63, 2019.
  * Kraska et al. [2021] Tim Kraska, Mohammad Alizadeh, Alex Beutel, Ed H Chi, Jialin Ding, Ani Kristo, Guillaume Leclerc, Samuel Madden, Hongzi Mao, and Vikram Nathan.  Sagedb: A learned database system.  2021\.
  * Krishnan et al. [2018] Sanjay Krishnan, Zongheng Yang, Ken Goldberg, Joseph Hellerstein, and Ion Stoica.  Learning to optimize join queries with deep reinforcement learning.  _arXiv preprint arXiv:1808.03196_ , 2018.
  * Li et al. [2018] Oscar Li, Hao Liu, Chaofan Chen, and Cynthia Rudin.  Deep learning for case-based reasoning through prototypes: A neural network that explains its predictions.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 32, 2018.
  * Liao et al. [2024] Yufan Liao, Qi Wu, and Xing Yan.  Invariant random forest: Tree-based model solution for ood generalization.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 38, pages 13772–13781, 2024.
  * Mankowitz et al. [2023] Daniel J. Mankowitz, Andrea Michi, Anton Zhernov, Marco Gelmi, Marco Selvi, Cosmin Paduraru, Edouard Leurent, Shariq Iqbal, Jean-Baptiste Lespiau, Alex Ahern, Thomas KÃ¶ppe, Kevin Millikin, Stephen Gaffney, Sophie Elster, Jackson Broshear, Chris Gamble, Kieran Milan, Robert Tung, Minjae Hwang, Taylan Cemgil, Mohammadamin Barekatain, Yujia Li, Amol Mandhane, Thomas Hubert, Julian Schrittwieser, Demis Hassabis, Pushmeet Kohli, Martin Riedmiller, Oriol Vinyals, and David Silver.  Faster sorting algorithms discovered using deep reinforcement learning.  _Nature_ , 618(7964):257–263, 2023.  doi: 10.1038/s41586-023-06004-9.  URL https://doi.org/10.1038/s41586-023-06004-9.
  * Mao et al. [2019] Hongzi Mao, Malte Schwarzkopf, Shaileshh Bojja Venkatakrishnan, Zili Meng, and Mohammad Alizadeh.  Learning scheduling algorithms for data processing clusters.  In _Proceedings of the ACM special interest group on data communication_ , pages 270–288. 2019.
  * Marcus et al. [2019] Ryan Marcus, Parimarjan Negi, Hongzi Mao, Chi Zhang, Mohammad Alizadeh, Tim Kraska, Olga Papaemmanouil, and Nesime Tatbul.  Neo: a learned query optimizer.  _Proceedings of the VLDB Endowment_ , 12(11):1705–1718, 2019.
  * Meng et al. [2020] Zili Meng, Minhu Wang, Jiasong Bai, Mingwei Xu, Hongzi Mao, and Hongxin Hu.  Interpreting deep learning-based networking systems.  In _Proceedings of the Annual conference of the ACM Special Interest Group on Data Communication on the applications, technologies, architectures, and protocols for computer communication_ , pages 154–171, 2020.
  * Peng et al. [2021] Yanghua Peng, Yixin Bao, Yangrui Chen, Chuan Wu, Chen Meng, and Wei Lin.  Dl2: A deep learning-driven scheduler for deep learning clusters.  _IEEE Transactions on Parallel and Distributed Systems_ , 32(8):1947–1960, 2021.
  * project [2019] Apache TEZ project, 2019.  URL https://tez.apache.org/.
  * Rudin [2019] Cynthia Rudin.  Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead.  _Nature Machine Intelligence_ , 1(5):206–215, 2019.
  * Sabek et al. [2022] Ibrahim Sabek, Tenzin Samten Ukyab, and Tim Kraska.  Lsched: A workload-aware learned query scheduler for analytical database systems.  In _Proceedings of the 2022 International Conference on Management of Data_ , pages 1228–1242, 2022.
  * Scheduler [2022] Apache Capacity Scheduler, 2022.  URL https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html.
  * Shmoys et al. [1994] David B Shmoys, Clifford Stein, and Joel Wein.  Improved approximation algorithms for shop scheduling problems.  _SIAM Journal on Computing_ , 23(3):617–632, 1994.
  * TPC-H [2018] TPC-H.  The TPC-H Benchmarks., 2018.  URL https://www.tpc.org/tpch/.
  * Wołczyk et al. [2024] Maciej Wołczyk, Bartłomiej Cupiał, Mateusz Ostaszewski, Michał Bortkiewicz, Michał Zając, Razvan Pascanu, Łukasz Kuciński, and Piotr Miłoś.  Fine-tuning reinforcement learning models is secretly a forgetting mitigation problem.  _arXiv preprint arXiv:2402.02868_ , 2024.
  * Zaharia et al. [2012] Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy McCauly, Michael J Franklin, Scott Shenker, and Ion Stoica.  Resilient distributed datasets: A $\\{$Fault-Tolerant$\\}$ abstraction for $\\{$In-Memory$\\}$ cluster computing.  In _9th USENIX Symposium on Networked Systems Design and Implementation (NSDI 12)_ , pages 15–28, 2012.
  * Zhu et al. [2021] Hang Zhu, Varun Gupta, Satyajeet Singh Ahuja, Yuandong Tian, Ying Zhang, and Xin Jin.  Network planning with deep reinforcement learning.  In _Proceedings of the 2021 ACM SIGCOMM 2021 Conference_ , pages 258–271, 2021.

##  Appendix A Appendix / supplemental material

###  A.1 Dataset Generation for Interpretability Study

For TPC-H, we randomly sample jobs from seven different data sizes (2, 5, 10,
20, 50, 80, and 100GB) and all 22 TPC-H queries or DAG structures, creating
two non-overlapping partitions of 500 jobs each. We run Decima on these
partitions, and use the logs to obtain training and test sets for the decision
tree scheduler. Both the training and test sets consist of the combined
features of node comparators and the selected nodes for each scheduling event.
By setting the group size, $|g|$ to 2, we have 252,472 training pairs. Each
set includes the combined features of node comparators and selected nodes for
each scheduling event. With a group size $|g|$ of 2, we obtain 252,472
training pairs. With $|g|$ set to 3, we get 1,796,180 training triplets. The
test set contains 119,414 pairs and 456,420 triplets.

###  A.2 Batched Arrivals: Alibaba Trace

Figure 6: Model generalisability results (job arrival starts from time 2500):
the Decision tree scheduler trained on TPC-H improves average JCT of 10 random
Alibaba production jobs by 44.8 $\%$ over Spark’s FIFO scheduler, by 30.4 $\%$
over shortest job first, by 6.5 $\%$ over a fair scheduler, and 8.7 $\%$ over
Decima on a cluster with 50 task slots (executors). Different queries in
different colors; vertical red lines are job completions; purple means idle.
Different to the experiment on TPC-H jobs, we additionally visualize
scheduling details under Shortest job first (SJF) for comparing more
algorithms on a real job trace.

We apply the model trained on TPC-H on the Alibaba production job trace to
evaluate the generalizability of our decision tree model. Fig. 6 shows that
our decision tree scheduler consistently outperforms the baselines. In Fig. 6
(e) and (f), we compare the performance difference between a model trained on
TPC-H and a model trained on Alibaba production trace, using test data
containing workload from the Alibaba trace only. The model trained on TPC-H
slightly outperforms the one trained on Alibaba trace. It is mainly because
the TPC-H jobs are more diverse, which enables the model to select optimised
decision paths for jobs in the test data. The results show the good
generalizability of our decision tree scheduler on unseen job patterns.

###  A.3 Continuous Arrivals

Figure 7: Decision tree scheduler performance comparison on TPC-H streaming
jobs: executor number – 50. The triplet decision tree model improves average
JCT of 5000 streaming TPC-H queries by 97$\%$ over Spark’s FIFO scheduler, by
9$\%$ over the Fair scheduler, by 6$\%$ over Decima. The maximum executor
utilization is 50. The number of concurrent jobs in the system is collected at
each time point.

Continuous arrivals. We also evaluate the performance of the decision tree
scheduler on streaming jobs with continuous arrivals. We randomly sample 5,000
TPC-H jobs from seven different sizes and 5000 Alibaba’s production jobs which
are unseen in training jobs. We generate their arrival intervals using a
Poisson process with a mean of 25 seconds for TPC-H jobs and 2.5 seconds for
Alibaba’s production jobs (as the Alibaba jobs’ sizes are smaller).

Figure 7 shows that the decision tree scheduler outperforms heuristic-based
schedulers and Decima on TPC-H jobs. It maintains a lower maximum number of
concurrent jobs in the system and a lower variance in the number of concurrent
jobs at each time point. The average maximum number of concurrent jobs in the
system at each time under the fair scheduler and Decima is 19 and 15
respectively while the decision tree scheduler has only 12.5 concurrent jobs
in the system in average at each time point. This is an indication of its
efficient resource use.

Similar to Decima, the performance gain of the decision tree schedulers also
comes from early completion of smaller jobs. But unlike Decima, our decision
tree schedulers allocate executors fairly among all concurrent jobs rather
than allocate extra executors to the small jobs as Decima does. It is clear
that Decima fails to learn an optimal executor allocation strategy, which
indicates the limitation of a purely data-driven approach for learning
scheduling policies. Our decision tree scheduler instead allows the injection
of heuristic strategies to an interpretable model for further performance
improvement. We achieve this by distilling node selection strategy from the
DNN model in Decima and replacing its executor allocation strategy with fair
scheduling. To demonstrate the difference, we downgrade our executor
allocation strategy in the triplet decision tree ($|g|=3$) to the one used by
Decima. Fig. 7(d) shows the resulting JCT increases to a level close to the
one produced by the original Decima scheduler in Fig. 7(c), in contrast to the
significant lower JCT achieved in Fig. 7(e) and (f). This clearly shows the
importance to develop an interpretable model to enable the use of heuristics
or domain knowledge for improvement and problem diagnostics.

Figure 8: Model generalisability results for streaming jobs on Alibaba jobs:
executor number – 50. the decision tree model trained on Alibaba jobs improves
average JCT of 5000 streaming Alibaba queries by 99 $\%$ over Spark’s FIFO
scheduler, by 43 $\%$ over the SJF (Shortest Job First), by 7.8 $\%$ over the
fair scheduler, by 2.8 $\%$ over Decima. The model trained on TPC-H trace
slightly outperforms the one trained on Alibaba trace, which indicates the
good generalizability of our decision tree scheduler on unseen job patterns.

On Alibaba’s production workload, we test the Decision tree scheduler obtained
by two different ways. Fig. 8(e) is the decision tree scheduler trained from
the scheduling trace of Decima on TPC-H jobs and Fig. 8(f) is the decision
tree scheduler trained from the scheduling trace of Decima on Alibaba’s
production jobs. Although they have different data distributions, the decision
tree scheduler obtained by simulating Decima’s trace on TPC-H jobs also
performs well on Alibaba’s production jobs. There is no obvious performance
gap between these two Decision tree schedulers, which indicates the good
generalizability of our decision tree scheduler.

We did our experiments on M2 CPU.

###  A.4 Edge Cases

Figure 9: Case 1.  Figure 10: Case 2.  Figure 11: Case 3.  Figure 12: Case 4.
Figure 13: Case 5.  Figure 14: Case 6.  Figure 15: Case 7.  Figure 16: Case 8.
Figure 17: Case 9.  Figure 18: Case 10.