# A case for disaggregation of ML data processing

Andrew Audibert
Google  Yang Chen
Google  Dan Graur
ETH Zurich  Ana Klimovic
ETH Zurich  Jiří Šimša
Google  Chandramohan A. Thekkath
Google

###### Abstract

Machine Learning (ML) computation requires feeding input data for the models
to ingest. Traditionally, input data processing happens on the same host as
the ML computation [8, 25]. The input data processing can however become a
bottleneck of the ML computation if there are insufficient resources to
process data quickly enough. This slows down the ML computation and wastes
valuable and scarce ML hardware (e.g. GPUs and TPUs) used by the ML
computation.

In this paper, we present tf.data service, a disaggregated input data
processing service built on top of tf.data. Our work goes beyond describing
the design and implementation of a new system which disaggregates
preprocessing from ML computation and presents: (1) empirical evidence based
on production workloads for the need of disaggregation, as well as
quantitative evaluation of the impact disaggregation has on the performance
and cost of production workloads, (2) benefits of disaggregation beyond
horizontal scaling, (3) analysis of tf.data service’s adoption at Google, the
lessons learned during building and deploying the system and potential future
lines of research opened up by our work.

We demonstrate that horizontally scaling data processing using tf.data service
helps remove input bottlenecks, achieving speedups of up to 110$\times$ and
job cost reductions of up to 89$\times$. We further show that tf.data service
can support computation reuse through data sharing across ML jobs with
identical data processing pipelines (e.g. hyperparameter tuning jobs),
incurring no performance penalty and reducing overall resource cost. Finally,
we show that tf.data service advanced features can benefit performance of non-
input bound jobs; in particular, coordinated data reads through tf.data
service can yield up to 2$\times$ speedups and job cost savings for NLP jobs.

##  1 Introduction

(a) CPU usage

(b) RAM usage

(c) Memory bandwidth usage

Figure 1: CDFs of normalized host resource usage for ML jobs that perform
colocated data ingestion and processing. The data is based on approximately
73k ML jobs running in Google data centers collected through random sampling
during a 24 hour period. The main takeaway from the CDFs is that the host
resource demand of ML computations follows a long tail distribution.

The success of machine learning (ML) in the past decade has led to the
widespread commercial deployment of specialized hardware for ML (e.g., GPUs
and TPUs). The cost of ML hardware dwarfs that of other datacenter resources,
such as CPU and RAM [2, 14]. Hence, to make cost-effective use of ML
accelerators, it is essential to feed the accelerators data as fast as the ML
computation they execute demands.

To enable high utilization of ML hardware, Google built and open-sourced the
tf.data framework [25]. tf.data provides an efficient runtime to execute ML
input data pipelines and a convenient API to express input data
transformations. Since its launch in 2017, tf.data has grown in adoption to
become the predominant solution for data ingestion and processing of ML
computations at Google. All Google-based submissions to the ML Perf training
competition [22] in recent years have relied on tf.data to achieve high
performance. The framework is also widely used by open-source Tensorflow [1]
programs.

However, tf.data could not meet the needs of all Tensorflow programs. The
original design colocated data ingestion and processing with the ML
computations. For some Tensorflow programs, host resources used for colocated
data processing (CPU, RAM, and I/O bandwidth) became the bottleneck, leaving
expensive ML hardware underutilized. This increases the end-to-end execution
time and cost of ML jobs.

The fundamental challenge is that ML jobs have a wide spectrum of CPU and
memory requirements, which make it impossible to right-size host CPU and
memory resources (for data processing) colocated with specialized ML
accelerators (for ML computations). Evidence of this is shown in Figure 1. By
pre-provisioning colocated preprocessing resources, a one-size-fits-all
resource deployment is imposed on ML preprocessing which is only optimal for a
narrow subset of all potential ML jobs. Most jobs will either end up using a
fraction of their assigned resources (taking away useful resources from other
jobs), or end up not having sufficient resources to keep up with the model’s
data needs (idling ML hardware).

A general approach for imbalanced resource utilization is disaggregation [12,
18, 29]. Disaggregation fits well into the context of ML jobs, as it naturally
enables scaling CPU, RAM, and I/O bandwidth independently from specialized ML
hardware. By disaggregating and distributing data processing across remote CPU
hosts, we can adjust the resource ratio for individual ML jobs and avoid input
data stalls [36]. However, adopting a disaggregated data processing solution
for ML jobs also has several potential concerns. Distributing data processing
across multiple nodes introduces new failure domains. The extra CPU, memory,
and network bandwidth needed for remote data processing may also add
significant costs, particularly since existing ML server configurations often
already contain local CPU and RAM.

In this paper, we present tf.data service, a disaggregated input data
processing service built on top of tf.data. Launched internally in 2020 and
open-sourced the same year, the service currently accounts for approximately
$10\%$ of all of tf.data computations at Google. This paper goes beyond
presenting tf.data service’s disaggregated system design and implementation
with three additional key contributions.

First, besides the quantitative-based motivation derived from production
workloads for the need of disaggregation, we provide the first quantitative
analysis for the performance and cost benefits of disaggregated data
processing on production ML workloads, showing training job speedups and cost
reductions of up to two orders of magnitude. With the particular set of models
we chose for this study, we are able to obtain up to 110x job time speedups,
and up to 89x job cost reductions.

Second, we show that the disaggregated tf.data service design provides
benefits that extend beyond the horizontal scaling of data processing. While
originally built to minimize input bottlenecks for ML jobs with data-intensive
modalities (e.g., image and video), over time we extended tf.data service with
functionality that helped improve performance of large-scale NLP jobs
(traditionally not considered data intensive). We also achieved CPU and RAM
savings by using tf.data service as an in-memory cache for concurrently
executing ML jobs consuming the same data. Based on this, we emphasize the
need for disaggregation not only for the sake of efficiency, but also for the
flexibility it introduces in ML workloads, enabling new and essential
preprocessing features and allowing them to be more easily implemented.

Finally, we analyze the fleetwide usage and adoption of tf.data service at
Google. We reflect on our experience creating and adapting tf.data service to
address the evolving needs of a wide variety of ML workloads, identifying
important lessons we have learned and directions for future work.

Similar to tf.data, tf.data service is an open-source project, available
through the Tensorflow GitHub project [31].

##  2 ML Input Data Processing

Input data processing can be view as an “Extract Transform Load” (ETL)
pipeline. Jobs read source data from storage, process data on-the-fly, and
load batches of transformed data to ML accelerators for model training or
inference. Common data transformations include decompressing data, parsing
file formats, extracting features, and batching elements. It is also common to
add randomness to input data (e.g., randomly sample, augment, and shuffle
elements) to improve model generalization [9, 10, 11, 30]. While model
training or inference typically executes on specialized hardware, such as GPUs
or TPUs, input data processing involves user-defined transformations which are
executed on general purpose processors. The high cost of accelerators compared
to their CPU hosts makes it particularly important to ensure that accelerators
operate at high utilization [2, 14]. If the ML computation does not have to
wait for data, the job is considered _model bound_. Otherwise, the job is
considered _input bound_. Input-bound jobs are much less desirable as they
leave valuable ML hardware underutilized (see Figure 2).

Figure 2: Depiction of input- and model-bound jobs. Input-bound jobs are
characterized by greater batch preprocessing times compared to the training
time on a batch. In model-bound jobs, training takes longer than
preprocessing.

##### tf.data framework

Users can express and execute input data pipelines using a variety of
libraries that integrate with ML frameworks like PyTorch [27] and Tensorflow
[1]. Examples include PyTorch DataLoader [8], NVIDIA DALI [17], and tf.data
[25]. Without loss of generality, we focus on tf.data as it is widely used
internally at Google as well as in open-source Tensorflow programs. tf.data
provides an API and a runtime for creating and executing efficient ML data
processing pipelines. The tf.data API provides generic operators that can be
parameterized by user-defined functions, composed, and reused across ML
domains. For example, the tf.data pipeline depicted in Figure 3 reads elements
from files, applies user-defined parsing logic to each element, combines the
processed elements into mini-batches, and overlaps data processing and ML
computation using software pipelining. Some tf.data operators expose knobs for
tuning performance, e.g., the map transformation allows users to set the
number of parallel calls and the prefetch transformation allows users to set
the buffer size. Users can rely on tf.data autotuning functionality [25] or
adjust these parameters manually to tune input pipeline throughput [13].

⬇

1def make_dataset() -> tf.data.Dataset:

2 files = tf.data.Dataset.list_files(

3 [".../*.tfrecord"])

4 ds = files.interleave(

5 tf.data.TFRecordDataset)

6 ds = ds.map(parse_fn,

7 num_parallel_calls=tf.data.AUTOTUNE)

8 ds = ds.batch(batch_size=256)

9 ds = ds.prefetch(tf.data.AUTOTUNE)

10 return ds

Figure 3: tf.data pipeline example.

##### Colocated vs. disaggregated data processing

Traditionally, ML frameworks have tightly coupled input data processing and ML
computations [8, 26, 32]. We refer to this mode of operation as colocated
processing, since data processing and ML computations execute on the same
machine (see Figure 4(a)). However, as ML hardware accelerators continue to
provide more FLOPS, feeding accelerators with data at sufficient throughput
requires increasingly more CPU and memory resources. For many jobs, the CPU
and RAM required for high-throughput data processing often exceed the
resources available on an ML training machine [36]. This motivates a
disaggregated processing mode, as shown in Figure 4(b), where data processing
executes on a separate machine(s) whose resources can be scaled independently
from expensive ML accelerators. The ability to right-size resource allocations
is critical to satisfy the wide variety of resource requirements in ML jobs
and ensure that all allocated resources — especially costly ML accelerators —
remain highly utilized. Figure 1 shows the heavy-tail distribution of CPU and
memory requirements across ML jobs in the Google fleet using tf.data.

(a) Colocated job

(b) Disaggregated job

Figure 4: Depictions of colocated and disaggregated jobs. Colocated jobs
preprocess and train on the same machine, whereas disaggregated jobs place the
two components on different machines.

##### Prior work on optimizing input processing

Several approaches have been proposed to alleviate input data stalls in ML
training jobs. Plumber [19] and tf.data’s autotuning harness [32] dynamically
tune software parallelism and memory buffer sizes to maximize performance on a
given training node. However, this tuning does not scale resources beyond a
single node. NVIDIA DALI supports offloading data processing to GPUs [17].
This alleviates CPU bottlenecks but may lead to GPU resource contention among
input data transformation tasks and model training tasks. Other works cache
and reuse (transformed) data across epochs [7, 21] or across jobs [16, 20,
24], trading storage capacity to save CPU cycles for repeated data processing.
However, caching solutions are still not guaranteed to eliminate data stalls.
In-memory caching solutions have capacity limitations since ML datasets often
exceed the size of RAM on a training machine [32]. SSD-based caching requires
high parallelism to avoid I/O bottlenecks when reading large volumes of data
from storage.

To address the above concerns, Meta’s Distributed Data Processing (DPP)
service [36] proposes horizontally scaling data workers with disaggregated
data processing. We take a similar approach, however we focus on addressing
the needs of a broader range of ML jobs, whereas Zhao et al.’s work on DPP
focused on the unique demands of ML recommender system jobs. Compared to DPP,
we demonstrate how to leverage disaggregation for more than just horizontal
scaling of data workers; tf.data service can also be used to avoid stragglers
and share input data and data transformations efficiently. It is open-sourced;
any user can deploy it to resolve input bottlenecks in their training job. We
also provide the first quantitative analysis of how much disaggregated data
processing improves the overall performance and cost of real production ML
workloads, addressing potential downsides of disaggregation, such as the extra
cost of remote data processing resources and the addition of new failure
domains.

##  3 tf.data service Design

In this section we present the design of tf.data service, a system that
enables disaggregation, distribution, and horizontal scaling of data
processing for ML workloads. tf.data service provides a simple API based on
tf.data for users to scale out their data processing. Besides horizontal
scaling, the API supports advanced features such as data sharing across
concurrent ML jobs and coordination of data consumption by tf.data service
clients.

###  3.1 System Architecture

Figure 5 shows the tf.data service architecture. tf.data service uses a
centralized distributed system design with a single _dispatcher_ and a pool of
_workers_. An ML job registers a dataset definition with the _dispatcher_. The
dispatcher distributes the dataset definition across the pool of the _workers_
and sends the list of worker addresses to the ML job tasks, which act as the
tf.data service _clients_. The clients read data directly from the workers.
The clients also periodically send heartbeats to the dispatcher to be notified
of changes in the worker pool.

Figure 5: tf.data service architecture. Solid lines correspond to the data
path, dashed lines correspond to the control path.

The dispatcher manages a pool of workers and metadata about registered
datasets and clients. It receives heartbeats from clients and workers,
notifies clients of worker pool updates, and assigns dataset processing tasks
to workers. It also keeps track of dataset processing task states and notifies
clients of completed tasks. Notably, the dispatcher does not perform any data
processing to avoid becoming a bottleneck.

The workers execute data processing tasks defined in a tf.data computation
graph. They read data from remote storage, apply requested transformations,
and store the elements in a buffer. The buffered elements are returned to the
clients when requested via RPCs. By default, the workers compress the elements
before sending them over RPC. The dispatcher may shard the dataset according
to a sharding policy, as described in Section 3.2. The number of workers and
computing resources may be specified manually or determined automatically by
integrating with an auto-scaling framework [16, 28].

The clients send RPCs to workers to fetch processed data. To speed up the
reads, clients may query data from multiple workers in parallel. The results
are stored in a client-side buffer and returned to the downstream ML
computation when requested. Without tf.data service, each client would run its
own local tf.data input pipeline to process the dataset. By offloading input-
processing to the workers, tf.data service can horizontally scale the
resources used for data ingestion and processing.

###  3.2 API

To indicate that tf.data datasets should use tf.data service, users can apply
the distribute transformation to a dataset definition. Figure 6 shows a
minimal example.

⬇

1ds = make_dataset()

2ds = ds.distribute(

3 processing_mode=ShardingPolicy.OFF,

4 service_address=...)

5for batch in ds:

6 train_step(batch)

Figure 6: tf.data service API example.

In this example, make_dataset() represents a user-defined tf.data dataset.
During input pipeline construction, the distribute transformation serializes
the upstream dataset definition and sends it to the dispatcher. The dispatcher
shares the definition with the pool of workers and sends the worker addresses
to the client. During input pipeline execution – using the for loop – the
client sends RPCs to the tf.data service workers to request data. Notably, the
communication between clients and workers is abstracted away from the Python
API. User programs simply iterate over the Python dataset object the same way
they would during colocated processing.

#### Data Sharding

Users can specify a sharding policy for the distribute transformation. The OFF
sharding policy offers no sharding. Under this policy, each worker processes
the entire dataset independently, typically randomizing the order of elements
across workers. The benefit of this approach is that it does not require
dataset partitioning and coordinated processing of the partitions across
workers. The downside of this approach is that the relaxed _visitation
guarantee_ can negatively affect the model quality for ML computations that
are sensitive to ordering guarantees.

If ML computation, such as evaluation, requires visitation guarantees, it can
use the DYNAMIC sharding policy. With dynamic sharding, the dispatcher
partitions the dataset into _splits_. The workers request splits from the
dispatcher and process one split at a time. Each split is dispatched to at
most one worker. By using a larger number of splits than the number of
workers, the dispatcher can load balance the workloads across the workers. If
workers are preempted, dynamic sharding provides _at-most-once_ visitation
guarantee. In the absence of preemptions, dynamic sharding provides _exactly-
once_ visitation guarantee.

The sharding policy also affects how new or restarted workers process data.
With no sharding, new workers process data from the first element in the
dataset while with dynamic sharding, workers start from the split assigned to
them.

The tf.data service API provides additional configuration, such as static
sharding, the compression method, and the data transfer protocol. We refer
interested readers to the tf.data service API documentation [15] for further
details.

###  3.3 Fault tolerance

While disaggregating data processing allows tf.data service to horizontally
scale data processing across multiple machines, this introduces a new failure
domain. The likelihood of a data worker failure increases as we distribute
data processing to more workers. tf.data service handles worker failures
according to the sharding policy. With no sharding, it assumes the client
tolerates repeated training data, and the restarted workers process the
dataset from the beginning. With dynamic sharding, tf.data service provides
at-most-once visitation guarantee.

##### Dispatcher fault-tolerance

The dispatcher writes its internal state changes to a write-ahead journal,
including registered datasets, active workers, and active clients. On restart,
the dispatcher replays the journal to restore its previous state.

##### Worker fault-tolerance

To handle input data workers failures, tf.data service is designed such that
workers are stateless. Hence, a restarted worker will register with the
dispatcher and retrieve the dataset definition it needs to process, like any
new worker would. In the no sharding case, the restarted worker processes the
dataset from the beginning. In the dynamic sharding case, the restarted worker
queries the dispatcher for the next split to process. The dispatcher assigns
one split to at most one worker. If the worker fails when it is processing a
split, the remaining data in that split is lost. We opted for a design that
provides at-most-once (as opposed to exactly-once) visitation guarantees
because of its simplicity and performance. We may extend tf.data service
design with support for exactly-once visitation guarantees in the future if we
register a growing demand for it among our users.

###  3.4 Data Sharing

Figure 7: Data service sharing: Workers serve requests from multiple training
jobs using a sliding window of cached data.

1 if not next_element_ready(experiment_id) then

2 extend_cache(experiment_id) // may discard old elements

3 end if

return next_element(experiment_id)

Algorithm 1 Cache-Get(experiment_id)

An additional benefit of disaggregating input processing is that the service
can be shared across multiple training clients and reuse the training data.
Input data pipelines are frequently re-executed. In Google for instance, 10%
of unique input data pipelines account for 77% of input pipeline executions
[25]. A common type of this usage is _hyperparameter tuning_ , during which
the state space of possible model parameters, such as the learning rate, is
explored to identify parameters that yield the best performance. The same
model is trained with different hyperparameters using the same data and each
such computation is referred to as an _experiment_.

Traditionally, each experiment would read and process the same data over and
over. If the data ingestion and processing requires significant amounts of CPU
and RAM, this can limit the number of experiments that can be executed.

To mitigate this inefficiency, we extended tf.data service with a _data
sharing_ functionality that allows concurrently executing experiments to use a
shared tf.data service instance in order to avoid having to read and process
data independently. When concurrent experiments read from the shared tf.data
service instance, tf.data service workers store processed elements in a
sliding-window cache that enables reuse across experiments. tf.data service
workers keep track of which elements have been read by each experiment to
avoid serving the same element to the same experiment multiple times. If an
experiment requests data and the cache contains an element that it has not yet
read, the element is served from the cache. Otherwise, the next element of the
dataset is materialized and added to the cache, evicting the oldest elements
if necessary to limit the RAM used for the cache. Each tf.data service worker
runs Algorithm 1 to provide data for the experiments. Figure 7 illustrates a
scenario with $k$ concurrently executing experiments. The experiments read
data from a shared tf.data service instance with $n$ workers.

To analyze the benefits of data sharing, let us assume that the data
processing cost for each experiment is $C$. If all experiments run at the same
speed, then the data processing cost with sharing is $C$ while the data
processing cost without sharing is $k\times C$. If the experiments run at
different speeds, slower experiments may miss some elements due to cache
evictions needed to accommodate faster experiments. In the worst case, the
experiments run sequentially and each experiment only shares the final window
populated by the previous experiment. In this case, the data processing cost
with sharing is $k\times C-(k-1)\times\frac{\textrm{cache
size}}{\textrm{dataset size}}\times C$ (assuming constant processing cost per
element).

As the algorithm can skip over certain elements, the dataset definition should
be resilient to this. One strategy is to make use of the repeat()
transformation to repeat the dataset infinitely, and applying random
augmentation to the examples. This way, each experiment eventually gets the
required number of random examples regardless of their training speed.

###  3.5 Coordinated Reads

Figure 8: Coordinated reads example with three tf.data service workers and two
clients.

⬇

1ds = ds.bucket_by_sequence_length(

2 bucket_boundaries=[128, 256, 512])

3ds = ds.group_by_window(window_size=num_consumers)

4ds = ds.flat_map(lambda x: x)

5ds = ds.distribute(

6 processing_mode=ShardingPolicy.OFF,

7 service=tf_data_service_address,

8 job_name="coordinated_read_job",

9 num_consumers=num_consumers,

10 consumer_index=i)

Figure 9: Dynamic-sequence-length training with tf.data service and
coordinated read.

Another scenario where disaggregating input processing benefits distributed
training clients is when the clients need to coordinate the reading in a
certain order. For example, in synchronous training of NLP models, batches of
variable-length sequences are consumed at each step. During training, clients
should ideally consume sequences of similar lengths at any given step.
Otherwise, due to the synchronous nature of training, clients consuming
shorter sequences would finish early and have to wait for clients with longer
sequences to finish.

To address this use case, tf.data service provides the _coordinated reads_
functionality. Instead of producing data in FCFS order, workers generate
groups of $n$ batches of similar-length sequences, where $n$ is the number of
clients. Workers then assign batches from each group to different clients.
Clients coordinate the order they read from workers, so that during each step,
the data used for training by every client comes from the same worker. Figure
8 illustrates coordinated reading. At the first step, the clients read 0 and 1
from worker 1. At the next step, they read 0 and 1 from worker 2, and so on.

The responsibility for making sure each worker produces batches of sequence of
similar length at each step falls onto the user. This can be achieved through
the bucket_by_sequence_length and group_by_window transformations as
illustrated in Figure 9. Users define the desired bucket boundaries and call
bucket_by_sequence_length to bucketize the training data by sequence length.
Then they group every num_consumers elements into a window, which will thus
contain elements of similar sequence length.

When the user applies the distribute transformation to distribute the
processing of a dataset, tf.data service workers generate windows consisting
of num_consumers batches. At each step, the workers wait until receiving
requests from all consumers before returning data. These consumers will
receive data from the same window, hence with batches of the similar sequence
length.

##  4 Evaluation

In this section we present experiments based on production ML workloads that
quantify the benefits tf.data service provides. Our experiments demonstrate
that tf.data service can save both time and cost for both input and model
bound ML jobs, in the latter case via its advanced features such as data
sharing and coordinated reads.

###  4.1 Methodology

##### Workloads

For this study, we employ a total of nine production models, which we
anonymize and identify as $M_{1},M_{2},...,M_{9}$. Models $M_{1},...,M_{4}$
are from the computer vision domain; models $M_{5},...,M_{9}$ are from the NLP
domain. Models $M_{1}$, $M_{2}$ and $M_{3}$ are input bound, whereas the other
models are not. Preprocessing and training are always colocated in the same
datacenter. Unless explicitly mentioned, the storage location is not a factor
on the performance of a job.

Computer vision models often have computationally intensive input pipelines,
which is why we select models $M_{1},M_{2},$ and $M_{3}$ to show the benefits
of disaggregation. We choose model $M_{4}$ to emphasize the benefits of data
sharing as internally it is a prime beneficiary of this feature. NLP models
generally posses less compute intensive input pipelines, hence we show the
potential of advanced features such as coordinated reads on models
$M_{5},...,M_{9}$. While we exemplify the benefits of certain features on
certain categories of models, we emphasize that these benefits can extend
beyond the types of models we sample in this study.

##### Frameworks

We use Tensorflow [1] both for defining the logic of the input pipelines and
of the models, although tf.data service can be used with other frameworks,
such as Jax [4] or PyTorch [27]. We conduct our evaluation using Borg [33],
Google’s internal cluster management system. The vertical and horizontal
autoscaling of tf.data service resources is handled by Autopilot [28] and is
based on hardware-based utilization metrics.

##### Baselines

The colocated time for a model is obtained by running the model in tf.data
without service and observing the job time. For input bound jobs, the ideal
job speedups are collected by observing the job times when the input
bottleneck is alleviated entirely by using a take(1).cache().repeat()
operation at the end of the input pipeline. This command retrieves the first
element produced by the input pipeline, caches it, and returns it without any
recomputation whenever the model requires a new batch. To avoid potential
network overheads, the ideal speedup is obtained using tf.data without
service. We identify this time as ideal, because from the preprocessing point
of view, no amount of resources assigned to the input pipeline could speed up
the job beyond this time.

##### Metrics

We calculate speedups by measuring the average steps per second of each
experiment and dividing the value with the average steps per second of the
baseline. We also compute standard deviations for the averages. We elide error
bars in our plots as the standard deviations are negligible. For computing the
cost savings of a given experiment, we take the reference experiment’s cost
and divide it by the cost of the experiment in question.

##### Cost Model

To gain an understanding of job costs and enable cost comparisons between
different setups, we compute the cost $\mathcal{C}$ of an ML job using the
following formula:

$\begin{split}\mathcal{C}=&~{}t(\mathcal{C}_{CPU}(n_{W}\cdot\overline{CPU}^{W}_{u}+n_{T}\cdot
CPU^{T}_{a})\\\
&+\mathcal{C}_{MEM}(n_{W}\cdot\overline{MEM}^{W}_{u}+n_{T}\cdot
MEM_{a}^{T})\\\ &+\mathcal{C}_{ACC}\cdot n_{T}\cdot n_{ACC/T})\end{split}$ (1)

where $t$ represents the job execution time, $n_{W}$, $n_{T}$ and $n_{ACC/T}$
represent the number of tf.data service workers, training workers and
accelerators per training workers respectively. $\mathcal{C}_{CPU}$,
$\mathcal{C}_{MEM}$ and $\mathcal{C}_{ACC}$ represent the normalized cost of
one “unit” of CPU, memory and accelerator per unit of time respectively.
$\overline{CPU}^{W}_{u}$ and $\overline{MEM}^{W}_{u}$ represent the average
CPU and memory utilization across all tf.data service workers per unit of
time. Finally, $CPU^{T}_{a}$ and $MEM_{a}^{T}$ represent the available CPU and
memory for a training worker. We use CPU and memory utilization (as opposed to
reservation) for the tf.data service workers, as reserved but unused CPU and
memory resources are released back to the global resource pool. In contrast,
the CPU and memory of ML hosts are non-fungible, regardless of their true
utilization, and as a consequence they are charged altogether. We do not
disclose the absolute (or relative) values of $\mathcal{C}_{CPU}$,
$\mathcal{C}_{MEM}$, and $\mathcal{C}_{ACC}$ but they are consistent with the
pricing of cloud VMs [2, 14].

Setup details pertaining to particular experiments are discussed in the
experiments’ respective sections below.

###  4.2 Disaggregation

To demonstrate the importance of disaggregation for production ML workloads,
we present results for models $M_{1}$, $M_{2}$ and $M_{3}$. The reported
results use 32, 8 and 16 accelerators to train the models respectively. When
trained without tf.data service, models $M_{1}$ and $M_{2}$ fully utilize
locally available CPU resources for data ingestion and processing, while
$M_{3}$ makes only partial use of the locally available CPU resources. We
present this mix of models to demonstrate that tf.data service can address
both hardware and software input bottlenecks.

#### Speedup

We measured the speedup obtained by using disaggregated tf.data service while
enabling Autopilot. Figure 10 shows our results. With the use of disaggregated
tf.data service we were able to obtain speedups of 11.7x, 110.3x and 2.9x
respectively. For models $M_{1}$ and $M_{3}$ tf.data service was able to reach
the ideal speedup, while for $M_{2}$ it fell 8% short. Upon closer inspection
into model $M_{2}$’s disaggregated performance, we found that the model
receives data from tf.data service without delay upon request, suggesting that
the penalty in performance stems from hardware limitations on the side of the
trainer nodes. We believe that the remarkably high ingestion rate requirements
of this model (up to 580 batches per second) in concert with the need to
decompress and copy all batches received from tf.data service introduce
resource contention on the trainer nodes. This contrasts with the ideal
speedup, which is obtained when tf.data is deployed locally where no
decompression and additional data copies are required.

To make sure disaggregating input processing achieved the same model quality
as colocated processing, we compared the loss and accuracy of the trained
models. We verified that the models achieved the same loss and accuracy with
and without tf.data service.

Figure 10: End-to-end job time speedups with disaggregated tf.data service
compared to colocated processing and the ideal speedup. The $Y$ axis
represents the speedup relative to the colocated processing baseline and uses
logarithmic scale.

To further illustrate the benefits of disaggregated tf.data service, we
evaluated model $M_{3}$ in a scenario where the data to be ingested is stored
in a different geographical region from the tf.data service and ML hosts. This
setting is often encountered in practice, for example due to resource scarcity
or financial constraints. Such situations can incur large performance
penalties on the job time due to increased latency.

Figure 11 shows our results for both the “in-region” and “out-of-region”
scenarios. tf.data service is able to reach the ideal speedup and ensure no
performance is lost due to ingesting and processing regardless of the
geographical location of the data. For the out-of-region data, the ideal
speedup is 13.3x. This contrasts with the in-region scenario where the ideal
speedup is 2.9x. The difference being due to the additional wait time incurred
by the increased latency of the remote storage, which affects the reference
colocated time.

Figure 11: End-to-end job time speedups for $M_{3}$ when using tf.data service
to read data from the same vs. different geographical region. In both cases
the ideal job time is the same, but the change in colocated time affects the
ideal speedup.

#### Cost Reduction

From an economical perspective, disaggregated tf.data service uses additional
resources to speed up end-to-end time of ML workloads. To better understand
the cost effectiveness of disaggregated tf.data service, we measured the
impact its use has on the overall resource cost of the ML workload.

The impact of using disaggregated tf.data service on cost is depicted in
Figure 12. Notably, the cost reductions are comparable to the speedups,
realizing reductions of 10.8x, 89.3x and 2.8x for models $M_{1}$, $M_{2}$, and
$M_{3}$ respectively. The cost reduction stems directly from the smaller job
times which imply shorter leases on the valuable ML hardware.

Figure 12: Comparing total resource cost savings of disaggregated tf.data
service versus colocated processing. The $Y$ axis represents the cost savings
relative to the colocated processing baseline and uses logarithmic scale.

Note that our cost equation does not capture possibly the most important cost
– the opportunity cost. By allowing ML workloads to finish faster, the
valuable and scarce ML hardware is released back into the general pool sooner
and can be used by other ML computations. In other words, the cost reductions
estimated using Equation 1 are likely an underestimate of the actual
economical impact of using disaggregated tf.data service to remove input
bottlenecks of ML workloads.

##### Takeaways

Disaggregated tf.data service can help eliminate both software and hardware
input bottlenecks for ML workloads, speeding up production ML workloads by up
to two orders of magnitude. The speedup of ML workloads realized through the
use of additional CPU and RAM resources results in overall cost reduction of
production ML workloads.

###  4.3 Horizontal Scaling

To further quantify the impact of disaggregation on input bound ML workloads,
we used model $M_{1}$ to train across a range of various sizes of the tf.data
service worker pool – 8, 16, 32, 64, 128, 256, 512, and 640. The lower bound –
8 workers – is chosen such that it was comparable to the resources available
for colocated processing. The upper bound – 640 workers – is chosen such that
it lies comfortably above the number of workers required to meet the model’s
ingestion rate requirements, and hence alleviate the input bottleneck.

Figure 13 shows our results. Figure 13(a) plots the job time speedup and
Figure 13(b) job cost savings for each of the configurations. The
normalization is done against the job time and cost of the colocated
processing baseline. The figure also plots a horizontal line that indicates
the ideal job time speedup that can be achieved when there are not input
bottlenecks.

(a) Job speedup

(b) Job cost savings

Figure 13: Job time speedup and cost savings for model $M_{1}$ across several
tf.data service worker counts.

At the lower bound of the sweep, the job time and costs increased by 83%
compared to the colocated processing time and cost due to a slowdown in the
processing throughput. This is because the processing resources were
comparable to the colocated case but when using disaggregated tf.data service
the processed data must be obtained via RPC calls which involves additional
data copies. Additionally, tf.data service also compresses data before sending
it over the wire and decompresses it on the client side.

An addition of 8 more workers, bringing the total to 16, begins to improve job
time and cost, cutting both by 14% compared to the baseline. The further
addition of workers sees significant improvements to both the time and the
cost of the job. At around 512 nodes, tf.data service meets the ideal time,
achieving a training time speedup of 12.3x and job cost savings of 11.4x
compared to the baseline. The job cost increases marginally at the 640 worker
count due to the higher processing costs caused by the superfluous 128 tf.data
service workers, however the job time remains unaffected. In other words,
over-provisioning processing resources has minimal impact on the job time.
These observations were consistent for models $M_{2}$ and $M_{3}$ (not
plotted).

(a) Setup with data sharing

(b) Setup with no data sharing

(c) Setup with dedicated services

Figure 14: tf.data service setups used in the data sharing experiments.

(a) Job time comparison

(b) Preprocessing cost comparison

Figure 15: Job time and preprocessing cost of hyperparameter tuning jobs on
model $M_{4}$ using different preprocessing techniques.

###  4.4 Data Sharing

To understand the performance of tf.data service data sharing feature we
evaluated a set of hyperparameter tuning jobs across three different
deployment setups: (A) all hyperparameter tuning jobs sharing a single tf.data
service instance with data sharing enabled, (B) all hyperparameter tuning jobs
sharing a single tf.data service instance with no data sharing enabled, (C)
each hyperparameter tuning job using a dedicated tf.data service instance.
Figures 14(a), 14(b) and 14(c) provide an architectural diagram for each of
the three modes.

For modes (A) and (B) we deployed tf.data service with 128 workers, whereas
for mode (C) each individual tf.data service deployment had 128 workers. We
ranged the number of tuning jobs over the set $\\{1,2,4,8,16\\}$. For this
experiment we used production model $M_{4}$. A single tuning job for this
model makes use of 16 accelerators. It should be noted that the model is not
input bound when there are 128 tf.data service workers. Figure 15 presents the
normalized job time and cost of each hyperparameter tuning job. The
normalization is done relative to a training run of the model without any
hyperparameter tuning and with a dedicated tf.data service deployment. Figures
15(a) and 15(b) show the results in terms of normalized job times and
preprocessing costs respectively.

Mode (A) of employing a shared tf.data service with shared data incurs no
performance overhead on the job time regardless of the number of tuning jobs,
completing in the same amount of time as a regular training run. This
assertion holds even up to 32 and 64 tuning jobs (not plotted). Moreover, the
processing cost remains the same thanks to the constant job time. The same
cannot be said about mode (B) which employs a shared tf.data service with no
data sharing. Here the processing resources are sufficient to support up to
four tuning jobs. For more concurrent tuning jobs, the job time gradually
increases by 1.75x for 8 tuning and to 3x for 16 tuning jobs. The longer job
time maps directly to increased cost. For mode (C) with a dedicated tf.data
service deployment, the job time is not affected, however the processing cost
grows linearly in the number of tuning jobs.

Besides the job time and cost benefits, tf.data service with data sharing
helps reduce storage and network load by keeping the number of connections to
and the number of bytes read from storage layer constant with respect to the
number of tuning jobs. This contrasts with the other two modes in which the
number of connections and bytes read scales linearly in the number of tuning
jobs.

tf.data service with data sharing also has significant benefits in scheduling
jobs internally. In particular, high CPU and memory data processing
requirements can limit the number of tuning jobs that can be scheduled
concurrently. Data sharing significantly reduces CPU and memory requirements
by only deploying a single tf.data service instance, helping hyperparameter
tuning searches to complete faster and allowing other jobs to make use of the
saved CPU and memory resources.

##### Takeaways

Data sharing is able to save significant amounts of preprocessing resources
without affecting the job performance in the context of concurrent jobs with
identical input data pipelines (e.g. hyperparameter tuning or model selection
tasks). Our experiments on model $M_{4}$ show this to be the case for scales
as large as 64 concurrently running hyperparameter tuning jobs. Data sharing
helps reduce storage and network load by avoiding redundant transfers of
identical data. Data sharing internally helps schedule tuning jobs more easily
by significantly reducing the compute and memory requirements of
preprocessing.

###  4.5 Coordinated Reads

To quantify the benefits of coordinating reads from tf.data service, we
evaluated the performance of models $M_{5}$, $M_{6}$, $M_{7}$, $M_{8}$, and
$M_{9}$. None of these models were input bound hence any observed performance
benefits stem from reducing the imbalance of lengths of batches of variable-
length sequences. The maximal sequence length for each of the models is 512,
512, 128, 256 and 512 respectively. The non-coordinated reads time is obtained
using tf.data with no service, while the coordinated reads time is obtained
using tf.data service with a similar amount of preprocessing resources as in
the colocated case. The bucket size for all models is either 32 or 64. Figure
16 shows our results.

Figure 16: Job time speedup in NLP production models with dynamic sequence
length support.

With the use of tf.data service coordinated reads, we were able to realize
speedups ranging from 1.1x to 2x, which translate directly into similar job
cost reductions. Note that the realized speedups are correlated with the
maximal sequence length of the models. The intuition behind this is that the
longer the maximal sequence length, the greater the difference in processing
time between a batch of short vs. long sequences and thus the higher the
impact of stragglers in the absence of coordination.

##### Takeaways

tf.data service’s coordinated reads make it possible to perform global
bucketization in a scalable fashion, realizing a speedup for NLP workloads.
The global nature of bucketization is important to minimize stragglers –
different training workers operating on batches of different sizes during the
same training step – while the scalability aspect is important to avoid input
bottlenecks.

(a) Growth of tf.data service adoption, measured by accelerator chips fed.

(b) Distribution of the number of tf.data service workers used by production
jobs.

(c) Normalized client node CPU limit and tf.data worker CPU usage.

Figure 17: tf.data service fleetwide usage.

##  5 Experience

We launched tf.data service internally in Q2 2020 and open-sourced it in Q3
2020. Its adoption has since been growing steadily both internally and
externally (e.g. for Cloud TPU workloads). This section provides insights
about the internal adoption of tf.data service and shares our experience
helping users improve performance of their ML workloads.

###  5.1 Fleetwide Usage

Similar to tf.data [25], we instrument tf.data service to collect metrics such
as the data sources, the set of transformations applied in each job’s input
pipeline, the accelerator types and topology, or the CPU and memory usage.
Figure 17(a) shows the growth in adoption of tf.data service over time,
measured by the number of accelerator chips fed and normalized to the adoption
in Q3 2022. The figure shows that the number of accelerator chips reading data
from tf.data service has been steadily growing since the initial launch.

As shown in Section 4, tf.data service can speed up execution of input bound
jobs by up to two orders of magnitude. The speedup stems from ingesting and
processing data using multiple tf.data service workers in parallel. The
optimal amount of processing resources depends on the model. Figure 17(b)
shows the distribution of tf.data service deployment sizes (i.e. the number of
workers) for the internal ML workloads over the last year. While most training
jobs deploy between 2 and 32 workers, the largest model uses more than 5K
workers. Users can either configure a fixed size deployment or use Autopilot
[28] to automatically determine the optimal configuration of the tf.data
service deployment. Autopilot supports horizontal scaling to adjust the number
of workers and vertical scaling to adjust the CPU and RAM resources of the
workers. Autopilot helps users adapt their tf.data service deployments to
match the needs of their ML workload. In contrast, without tf.data service,
tf.data could only use the fixed resources of the accelerator host.

As a result of horizontal scaling, clients can request more workers, hence
more CPU resources, to match the accelerator training speed. We profiled the
top 10 tf.data service deployments in terms of the overall CPU usage of their
tf.data service workers. The result is shown in Figure 17(c). The left bars
are the normalized CPU _limits_ of the clients. The right bars are the CPU
_usage_ of tf.data service workers. In the profiled jobs, the tf.data service
workers use up to $25\times$ of CPUs available on the ML hosts. Without
tf.data service, running data processing on these would lead to large input
bottlenecks, slowing down the ML computation and limiting utilization of
valuable ML hardware.

###  5.2 Key Insights and Research Opportunities

We have helped tens of internal teams deploy tf.data service and optimize
their ML workload performance. In this section, we share the lessons we
learned from this experience and highlight potential future lines of research.

##### Keep accelerators utilized

We have shown quantitative evidence on the importance of keeping accelerators
utilized and avoiding input bottlenecks, by scaling out processing resources
when necessary. Avoiding input bottlenecks implies faster job execution, which
in turn results in shorter leases on ML and non-ML resources, thus reducing
costs. These benefits grow proportionally with the scale of the ML workload.
Another reason why ML workloads should not under-utilize ML hardware stems
from the scarcity of resources in the context of an organization. Teams within
an organization often draw from a shared resource pool. Jobs are oftentimes
scheduled with delays as they await for the required hardware (CPU, RAM, or
accelerators) to become available [34]. ML workloads that make inefficient use
of ML hardware are not only under-utilizing a valuable resource, but are
preventing other ML workloads from being scheduled.

##### Flexibility for the future

Disaggregation offers the flexibility to scale-out ML preprocessing to avoid
input data bottlenecks. It also offers the flexibility to adjust data
processing resources over time during a job’s lifetime. For example, in a
shared cluster with resource contention, the availability of CPU and memory
resources for data processing may vary over time and a disaggregated data
service can adapt to dynamic resource availability. Finally, we demonstrated
that disaggregation also enables features such as data sharing and coordinated
reads. We argue that this new level of flexibility is imperative for future ML
workloads, as it decouples ML preprocessing from the evolving ML landscape
(e.g. new requirements, model architectures, training algorithms, and
accelerator hardware). Disaggregation enables adjusting ML job resource
allocation across flexible pools of CPU, RAM, and ML accelerators, rather than
being constrained by the physical hardware configurations on individual
trainer nodes.

##### ML datacenter hardware design

While trainer nodes currently come equipped with a fixed pre-assigned quota of
CPU, RAM and other hardware resources, disaggregating ML preprocessing at the
software framework level also enables the separation of data preprocessing and
ML computation resources at the hardware level. Just as disaggregating data
storage from computation in cloud applications has led to new hardware trends
such as SmartNIC-attached storage device arrays [5, 23], we envision an
exploration of hardware deployments that completely separate ML accelerators
from the CPU and RAM resources used for traditional data processing. How to
design SmartNIC-attached hardware accelerator arrays and whether such hardware
can improve the performance and cost of ML jobs remain open questions.

##### Leveraging knowledge across jobs

Disaggregation enables deploying ML preprocessing as a dedicated service which
is shared by independent clients at an organization level. Such a deployment
provides a bird’s-eye view of all jobs executed within an organization, which
helps improve job performance and cluster resource efficiency. For example,
Cachew [16] is a system built on top of tf.data service that caches common
input data pipelines across jobs from different clients to optimize training
throughput. A bird’s eye view of the source data ingested and the data
processing operations applied across ML jobs in an organization can also be
useful for building a dataset discovery service [35, 6] or automating input
data augmentations [9]. Ultimately, the advantage of features that exploit the
organization’s view is that they can help make globally optimal decisions, in
contrast to practitioners which have a limited view of only their jobs.

##  6 Conclusion

In this paper, we presented tf.data service. By disaggregating data processing
to dedicated workers, tf.data service can use horizontal scaling to speed up
input-bound models, for which colocated data processing cannot keep up with
the ML computation. Beyond horizontal scaling, we developed additional
features of tf.data service for advanced use cases, including data sharing and
coordinated reading. Data sharing saves computing resources when
hyperparameter tuning jobs use one tf.data service cluster to share input.
Coordinated reading speeds up training when the models require input data of
the same size at each training step.

We launched tf.data service at Google in Q2 2020 and open-sourced it in Q3
2020. We showed that horizontally scaling data workers to remove input
bottlenecks in production ML workloads at Google can achieve speedups of up to
$110\times$ and reduce job costs by up to $89\times$. Data sharing and
coordinated reading can achieve speedups and resource savings even in the
absence of input bottlenecks. While our experimental evaluation focused on
internal Google production workloads, our experience deploying tf.data service
for external Cloud TPU customers suggests that similar benefits of tf.data
service can be achieved by external users.

## Acknowledgements

This work represents a combined effort of a large group of people. First, we
would like to acknowledge Battulga Bayarsaikhan, Emanuel Taropa, Rohan Anil,
and Ryan Doherty for their work on DataGuzzler [3], a precursor to tf.data
service. Next, we would like to thank everyone who contributed to the
development of tf.data service, most notably Derek Murray and Rohan Jain, and
everyone who provided feedback to help us improve tf.data service, including
Aleksandr Zaks, Aniruddh Nath, Deping Xie, Hao Wu, Huan Gui, Josh Cai, Lin
Yuan, Mahesh Sathiamoorthy, Ruoxin Sang, Yingpei Wang, and Zhaoqi Leng. Last
but not least, we would like to thank Abhishek Gupta, Carrie Grimes Bostock,
and Steven Hand for reviewing the paper and helping us improve it.

## References

  * [1] Martin Abadi et al.  “TensorFlow: A system for large-scale machine learning”  In _Proc. of OSDI_ , 2016
  * [2] Amazon  “Amazon EC2 Pricing”, https://aws.amazon.com/ec2/pricing/, 2020
  * [3] Rohan Anil, Battulga Bayarsaikhan, Ryan Doherty and Emanuel Taropa  “Distributed computing pipeline processing”, https://patents.google.com/patent/WO2021177976A1, 2021
  * [4] James Bradbury et al.  “JAX: composable transformations of Python+NumPy programs”, 2018  URL: http://github.com/google/jax
  * [5] Broadcom  “Broadcom Stingray PS250 SmartNIC”, 2019  URL: https://docs.broadcom.com/doc/PS250-PB
  * [6] Tianshi Cao, Sasha (Alexandre) Doubov, David Acuna and Sanja Fidler  “Scalable Neural Data Server: A Data Recommender for Transfer Learning”  In _Advances in Neural Information Processing Systems_ , 2021
  * [7] Dami Choi, Alexandre Passos, Christopher J. Shallue and George E. Dahl  “Faster Neural Network Training with Data Echoing”, 2019  arXiv:1907.05550 [cs.LG]
  * [8] Torch Contributors  “PyTorch Docs: torch.utils.data”, https://pytorch.org/docs/stable/data.html, 2019
  * [9] Ekin D. Cubuk, Barret Zoph, Dandelion Mané, Vijay Vasudevan and Quoc V. Le  “AutoAugment: Learning Augmentation Strategies From Data”  In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR_ , 2019
  * [10] Ekin Dogus Cubuk, Barret Zoph, Jon Shlens and Quoc Le  “RandAugment: Practical Automated Data Augmentation with a Reduced Search Space”  In _Advances in Neural Information Processing Systems_ , 2020, pp. 18613–18624
  * [11] Jia Deng, W. Dong, R. Socher, L.-J. Li, K. Li and L. Fei-Fei  “ImageNet: A Large-Scale Hierarchical Image Database”  In _Proc. of CVPR_ , 2009
  * [12] Peter X. Gao et al.  “Network Requirements for Resource Disaggregation”  In _Proc. of OSDI_ , 2016
  * [13] Google  “Better performance with the tf.data API”  URL: https://www.tensorflow.org/guide/data_performance
  * [14] Google  “Google Cloud: All Pricing”, https://cloud.google.com/compute/all-pricing, 2020
  * [15] Google  “tf.data service API documentation”  URL: https://www.tensorflow.org/api_docs/python/tf/data/experimental/service
  * [16] Dan Graur, Damien Aymon, Dan Kluser, Tanguy Albrici, Chandramohan A Thekkath and Ana Klimovic  “Cachew: Machine Learning Input Data Processing as a Service”  In _Proc. of USENIX ATC_ , 2022
  * [17] Joaquin Anton Guirao et al.  “Fast AI Data Preprocessing with NVIDIA DALI”, https://devblogs.nvidia.com/fast-ai-data-preprocessing-with-nvidia-dali, 2019
  * [18] Ana Klimovic, Christos Kozyrakis, Eno Thereska, Binu John and Sanjeev Kumar  “Flash Storage Disaggregation”  In _Proc. EuroSys_ , EuroSys ’16, 2016
  * [19] Michael Kuchnik, Ana Klimovic, Jiri Simsa, Virginia Smith and George Amvrosiadis  “Plumber: Diagnosing and Removing Performance Bottlenecks in Machine Learning Data Pipelines”  In _Proc. of Machine Learning and Systems_ 4, 2022, pp. 33–51
  * [20] Abhishek Vijaya Kumar and Muthian Sivathanu  “Quiver: An Informed Storage Cache for Deep Learning”  In _Proc. of FAST_ , 2020
  * [21] Gyewon Lee et al.  “Refurbish Your Training Data: Reusing Partially Augmented Samples for Faster Deep Neural Network Training”  In _Proc. of USENIX ATC_ , 2021
  * [22] Peter Mattson et al.  “MLPerf training benchmark”  In _arXiv preprint arXiv:1910.01500_ , 2019
  * [23] Jaehong Min et al.  “Gimbal: Enabling Multi-Tenant Storage Disaggregation on SmartNIC JBOFs”  In _Proc. of ACM SIGCOMM_ , SIGCOMM ’21, 2021, pp. 106–122
  * [24] Jayashree Mohan, Amar Phanishayee, Ashish Raniwala and Vijay Chidambaram  “Analyzing and Mitigating Data Stalls in DNN Training”, 2021  arXiv:2007.06775 [cs.DC]
  * [25] Derek G. Murray, Jiří Šimša, Ana Klimovic and Ihor Indyk  “tf.data: A Machine Learning Data Processing Framework”  In _Proc. VLDB Endow._ 14.12, 2021
  * [26] MXNET  “Designing Efficient Data Loaders for Deep Learning”, https://mxnet.apache.org/api/architecture/note_data_loading, 2018
  * [27] Adam Paszke et al.  “PyTorch: An Imperative Style, High-Performance Deep Learning Library”  In _Advances in Neural Information Processing Systems 32_ Curran Associates, Inc., 2019, pp. 8024–8035
  * [28] Krzysztof Rzadca et al.  “Autopilot: workload autoscaling at google”  In _Proc. of the Fifteenth European Conference on Computer Systems_ , 2020
  * [29] Yizhou Shan, Yutong Huang, Yilun Chen and Yiying Zhang  “LegoOS: A Disseminated, Distributed OS for Hardware Resource Disaggregation”  In _Proc. of OSDI_ , 2018
  * [30] Patrice Y. Simard, Dave Steinkraus and John C. Platt  “Best Practices for Convolutional Neural Networks Applied to Visual Document Analysis”  In _Proc. of ICDAR_ IEEE Computer Society, 2003
  * [31] TensorFlow  “Tensorflow”, https://github.com/tensorflow/tensorflow
  * [32] TensorFlow  “tf.data: Build TensorFlow input pipelines”, https://www.tensorflow.org/guide/data, 2020
  * [33] Abhishek Verma, Luis Pedrosa, Madhukar Korupolu, David Oppenheimer, Eric Tune and John Wilkes  “Large-scale cluster management at Google with Borg”  In _Proc. of EuroSys_ , 2015
  * [34] Qizhen Weng et al.  “MLaaS in the Wild: Workload Analysis and Scheduling in Large-Scale Heterogeneous GPU Clusters”  In _Proc. of NSDI_ , 2022
  * [35] Xi Yan, David Acuna and Sanja Fidler  “Neural Data Server: A Large-Scale Search Engine for Transfer Learning Data”  In _Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020
  * [36] Mark Zhao et al.  “Understanding Data Storage and Ingestion for Large-Scale Deep Recommendation Model Training: Industrial Product”  In _Proc. of ISCA_ , ISCA ’22, 2022