#  InTune: Reinforcement Learning-based Data Pipeline Optimization for Deep
Recommendation Models

Kabir Nagrecha<EMAIL_ADDRESS>Netflix, Inc.Los Gatos, CAUSA ,  Lingyi
Liu<EMAIL_ADDRESS>Netflix, Inc.Los Gatos, CAUSA ,  Pablo Delgado
<EMAIL_ADDRESS>Netflix, Inc.Los Gatos, CAUSA and  Prasanna Padmanabhan
<EMAIL_ADDRESS>Netflix, Inc.Los Gatos, CAUSA

(2023)

###### Abstract.

Deep learning-based recommender models (DLRMs) have become an essential
component of many modern recommender systems. Several companies are now
building large compute clusters reserved only for DLRM training, driving new
interest in cost- & time- saving optimizations. The systems challenges faced
in this setting are unique; while typical deep learning (DL) training jobs are
dominated by model execution times, the most important factor in DLRM training
performance is often online data ingestion.

In this paper, we explore the unique characteristics of this data ingestion
problem and provide insights into the specific bottlenecks and challenges of
the DLRM training pipeline at scale. We study real-world DLRM data processing
pipelines taken from our compute cluster at Netflix to both observe the
performance impacts of online ingestion and to identify shortfalls in existing
data pipeline optimizers. We find that current tooling either yields sub-
optimal performance, frequent crashes, or else requires impractical cluster
re-organization to adopt. Our studies lead us to design and build a new
solution for data pipeline optimization, InTune. InTune employs a
reinforcement learning (RL) agent to learn how to distribute the CPU resources
of a trainer machine across a DLRM data pipeline to more effectively
parallelize data-loading and improve throughput. Our experiments show that
InTune can build an optimized data pipeline configuration within only a few
minutes, and can easily be integrated into existing training workflows. By
exploiting the responsiveness and adaptability of RL, InTune achieves
significantly higher online data ingestion rates than existing optimizers,
thus reducing idle times in model execution and increasing efficiency. We
apply InTune to our real-world cluster, and find that it increases data
ingestion throughput by as much as 2.29X versus current state-of-the-art data
pipeline optimizers while also improving both CPU & GPU utilization.

data processing, recommendation systems, deep learning, parallel computing,
resource allocation

††journalyear: 2023††copyright: rightsretained††conference: Seventeenth ACM
Conference on Recommender Systems; September 18–22, 2023; Singapore,
Singapore††booktitle: Seventeenth ACM Conference on Recommender Systems
(RecSys ’23), September 18–22, 2023, Singapore, Singapore††doi:
10.1145/3604915.3608778††isbn: 979-8-4007-0241-9/23/09††ccs: Theory of
computation Reinforcement learning††ccs: Information systems Recommender
systems††ccs: Computing methodologies Parallel computing methodologies††ccs:
Computing methodologies Machine learning††ccs: Applied computing E-commerce
infrastructure††ccs: Computer systems organization

##  1\. Introduction

Recommendation systems now underpin many essential components of the web
ecosystem, including search result ranking, e-commerce product placement, and
media suggestions in streaming services. Over the last several years, many of
these services have begun to employ deep learning (DL) models in their
recommendation infrastructure, to better exploit historical patterns in their
data. In turn, DL-based product recommendation has quickly become one of the
most commercially significant applications of DL. Companies have begun to
invest heavily in DL recommendation infrastructure, often maintaining entire
datacenters and super-clusters for the sole purpose of recommender model
training (Acun et al., 2021). But in many cases, these infrastructural
investments have run into critical hurdles (Gupta et al., 2020). Practitioners
and cluster administrators are discovering that the training optimization
challenges faced with DL recommender models differ significantly from those
seen in historical practice with other DL model types (Wu et al., 2021). In
particular, recent studies of industry clusters have found that the unique
design of recommender model architectures has left training pipelines
susceptible to inefficiencies in data ingestion (Zhao et al., 2022).

Most DL architectures are dominated by high-intensity matrix operators, and
standard tooling for DL training optimization has evolved to support models
that fit this pattern (Shoeybi et al., 2019; Harlap et al., 2018; Jia et al.,
2018; Lepikhin et al., 2020; Ren et al., 2021; Huang et al., 2018; Nagrecha,
2023). In such cases, model execution usually dominates training times to such
a degree that data ingestion procedures (e.g. disk loading, shuffling, etc)
can be overlapped with and hidden underneath the matrix operation times.
Unfortunately, however, DL-based recommender models (DLRMs) are atypical in
this regard.

Recommender datasets are generally composed of both sparse (categorical) and
dense (continuous) features, and joining information across features requires
transforming these two representations into a common format. To this end, DLRM
architectures use embedding tables to transform categorical inputs into dense
embedding vectors through a hash-table lookup. These can then be combined with
the dense vectors and fed through some secondary DL model to produce user-item
probability ratings (Wei et al., 2022). Figure 1 illustrates a typical
architecture.

Figure 1. A typical DLRM architecture (Cheng et al., 2016; Acun et al., 2021).
The model uses an embedding table to convert sparse categorical data to dense
vectors that can then be merged with dense features in some overlaid DNN.
Adapted from a similar illustration in prior art (Wei et al., 2022).

[The DLRM architecture has a table in it.]Illustration of a DLRM architecture
shows that the single most important component is the embedding table.

The embedding tables, which are the typically single largest component of the
DLRM architecture, use a key-value lookup rather than dense matrix
multiplication. For this reason, DLRM models are often less compute-intensive
than other architectures of a comparable size. Figure 2 charts the differences
in computational intensity between large-scale recommendation models versus
language models and computer vision architectures, illustrating that DLRM
models require orders of magnitude fewer operations than comparably-sized
Transformers or Convolutional Neural Networks. This uniquely light
computational footprint can lead to unexpected system optimization challenges.

Figure 2. Approximate parameter & FLOP counts for popular architectures in
language modeling and image recognition contrasted against DLRM models drawn
from a recent paper (Mudigere et al., 2021). FLOPs are reported on single-
element batches (single-token for language models). We also report averaged
FLOPs per parameter, derived from the previous two charts. Y-axis is set to
log-scale for all charts.

[DLRMs are much less computationally intensive than other
architectures.]Comparison of FLOPs & parameter counts across model
architecture types shows that DLRMs are far less computationally intensive for
their size than other architectures.

Challenge & Motivation. The low computational intensity of DLRMs generally
translates to low model latencies, which fail to mask the cost of data-loading
and transformation. Improved GPU hardware and new model acceleration
techniques have only exacerbated this issue by reducing model runtimes and
increasing the requisite data-loading throughput to keep the model fed during
training. The fault is not only with the models, however; the problem is
aggravated further still by the generally high demands of online data
processing, i.e. data transformation at ingestion time, for recommendation
applications. In other domains (e.g. language modeling, computer vision) not
only are training times dominated by model execution, so that data processing
latencies can be more effectively hidden, but it is also practical to push the
heaviest data transformation steps to an offline pre-compute phase. This
greatly reduces the need to optimize data-loading. By contrast, recommendation
datasets are uniquely reliant on the online step, which must be done alongside
model execution. We attribute this to three characteristics of recommendation
data: scale, reusability, and volatility.

First, scale. A recommender dataset for a popular application might span
billions of interactions and require terabytes (or even petabytes!) of disk
space. Offline data transformation can bloat these already high storage costs
further still. Consider, for example, a common data processing operation such
as augmentation, which randomly modifies various aspects of a data sample to
produce an altogether new sample. Applying this operation offline might double
or triple the size of an already massive dataset; the only practical way to
run such transformations would be to do them online so that augmented samples
can be discarded as soon as they are consumed. Furthermore, this scale issue
often makes caching, which might otherwise help to mitigate processing
challenges, impractical.

Second, reusability. A single core dataset might be reused for multiple
different DLRM architectures. In a movie recommendation system, one DLRM might
be used to rank rows, another might be used to rank search results, while yet
another might rank genres. Each model would likely require different data
transformation operations and feature extraction procedures. Pushing data
transformation to the offline phase would require replicating and re-
processing the original dataset dozens of times, again bloating storage and
compute costs.

Third, volatility. Recommendation datasets are updated frequently as new
interactions are recorded. In addition, ephemeral IDs often lead to dataset
changes in domains such as e-commerce, e.g. when a product is added or removed
from the platform. Any offline transformations would have to be re-run
frequently as the dataset evolves. Incremental transformation is not always
practical; some operations such as shuffling require the whole dataset to be
present.

Prior analyses (Zhao et al., 2022; Mohan et al., 2020) of DLRM training have
recorded the impacts of these issues in practice, suggesting that online data
ingestion optimization is critical to improving DLRM training performance.
This new and emerging problem lacks a satisfactory solution. Table
LABEL:tb:prior_art provides an overview of existing tooling, but none of these
prior systems can effectively tackle this issue. Generic pipeline tools such
as AUTOTUNE & Plumber (Murray et al., 2021; Kuchnik et al., 2022) often lead
to sub-optimal performance for DLRM jobs, or can even cause fatal out-of-
memory errors. GPU data-loaders can be situationally useful, but cannot be
recommended for general use due to concerns over processor cycle contention
between the model and pipeline (Zhao et al., 2022). The only CPU-based DLRM
data pipeline work we are aware of (Zhao et al., 2022; Mudigere et al., 2021)
relies on a specialized cluster architecture design, and is not feasible to
adopt for typical users. We expand on these in Section 6.

Table 1. Overview of existing tooling. |  Name |  Description
---|---|---
Generic Pipelines |  AUTOTUNE (Murray et al., 2021) |  TensorFlow’s built-in tool for optimizing tf.data pipelines, considered to be a state-of-the-art optimizer (Kuchnik et al., 2022).
Plumber (Kuchnik et al., 2022) |  AUTOTUNE alternative with roughly equivalent performance.
DLRM Pipelines |  Data PreProcessing Service (Zhao et al., 2022) |  Meta’s internal service for data ingestion. Replicates data pipelines across machines and wraps them behind a singular entry-point. Tailored for Meta’s cluster; adoption would require a cluster re-design to match their architecture.
GPU Data-Loading |  DALI |  Nvidia’s tool for GPU-accelerated data-loading primitives, targets image processing operations (rotations, resizing, etc).
NVTabular |  Nvidia’s tool for GPU-accelerated data-loading primitives, focuses on tabular data. Introduces GPU resource contention between the model and data pipeline; not always practical to use.

We seek a new system — one which can improve data-loading throughput in a
general, scalable fashion without disrupting practitioner workflows or
requiring large-scale cluster changes.

Approach & Contributions. In order to reason about the data-ingestion problem
from first principles, we study traces taken from our internal DLRM training
cluster. We focus on the outcomes observed by real-world DLRM practitioners,
and observe shortfalls in generic data pipeline optimizers. We study training
times and processor utilization to better understand how poorly optimized data
ingestion pipelines increase costs and reduce efficiency.

From our studies, we find that a lack of adaptability and feedback is the
primary missing piece in generic data pipeline optimization tools. Out-of-
memory errors, under-optimized user-defined-functions (UDFs), and poor
responsiveness to dynamic machine re-sizing are the three main symptoms we
observe. Addressing the first symptom requires incorporating feedback from the
system’s memory usage monitor, the second requires actively adapting the
optimizer’s performance model of black-box UDFs, and the third requires
adaptability under changing hardware conditions.

We use this key finding to motivate the design of a new data pipeline
optimization tool for our cluster users at Netflix. We build a feedback-
driven, adaptive tool to optimize data ingestion pipelines that we name
InTune. InTune serves as a drop-in replacement for industry-standard
optimizers such as tf.data’s AUTOTUNE, requiring no large-scale cluster
redesigns or workflow disruptions. It can be applied to any data pipeline
framework, including tf.data, PyTorch Datasets, and Ray Datasets. At the core
of InTune is a reinforcement learning (RL) agent trained on historical job
traces and tuned online to understand how to distribute computational
resources across the data pipeline. RL provides the adaptability we need to
mazimize performance. The idea of a “DL-for-systems-for-DL” loop has recently
gained traction in the systems world (Olukotun, 2022); InTune provides a
complete example of this loop in practice.

We apply InTune to jobs on our real-world training cluster and see 1.18-2.29X
speedups in training times versus current tooling. We observe that InTune can
converge on an optimized resource distribution within only a few minutes, even
on complex real-world pipelines. Our tests show that InTune is both practical
and effective in improving DLRM training efficiency. We run scaling studies to
test InTune’s performance further, and find that it achieves good scaling
performance with respect to both workload size and machine size.

Our contributions can be summarized as follows:

  1. (1)

We provide in-depth analyses of DLRM model training job traces taken from our
real-world compute cluster, highlighting the critical and unique problem of
data pipeline optimization.

  2. (2)

We identify and study a new gap in the DL systems landscape, and evaluate the
weaknesses of state-of-the-art tooling for data ingestion during model
training.

  3. (3)

We propose a novel automated data-pipeline optimizer motivated by our cluster
studies, InTune. To the best of our knowledge, InTune is the first system to
use RL for data pipeline optimization. It is also an instance of the emerging
“DL-for-systems-for-DL” loop.

  4. (4)

We run comprehensive evaluations of InTune against state-of-the-art baselines
on real-world workloads. We find that InTune significantly outperforms the
baselines by a factor of 1.18-2.29X, providing significant speedups and
training cost reductions.

The remainder of the paper is structured as follows. Section 2 dives into the
fundamentals of DL recommender training, data processing, and RL. Section 3
analyzes real job traces from a compute cluster to provide motivation for
InTune’s development. Section 4 goes into the details of InTune and describes
how it conceptually addresses each of the challenges we identify. We show the
results of our experimental studies in Section 5, where we benchmark our
system’s performance on a variety of workloads. Section 6 describes some
existing tools for data processing and other related areas. Finally, we
provide our concluding remarks in Section 7.

##  2\. Background

We now provide background on the basics of DLRMs and online data pre-
processing to provide context for the rest of the paper. We then go into the
basics of RL to clarify the principles that underly our proposed solution.

###  2.1. Deep Learning Recommender Models

Deep learning has becoming an increasingly popular approach to tackle
recommendation problems in both industry and academia (Cheng et al., 2016).
These model architectures aim to bring the success that DL has seen in other
domains (e.g. language modeling, object recognition) to the recommender
systems space.

A typical DL model consists of a chained sequence of matrix transformations,
known as layers, combined with non-linear activation functions. The model’s
matrix entries, or parameters, are tuned using a historical dataset of sample-
label pairs. Each sample typically consists of multiple features, each
capturing a different aspect of the historical record. In an e-commerce
dataset, for example, each record might maintain features such as item ID, the
user ID, the item price, etc. The label might be a binary indicator reflecting
whether or not the user purchased the item.

The model is tuned to fit the dataset in a procedure known as stochastic
gradient descent (SGD). In SGD, batches of samples are pulled from the
dataset, then fed into the model to produce predictions. The predictions are
compared to the known ground-truth labels to produce an error value. The
derivative chain rule is used to compute a set of parameter updates to would
minimize the error, which are then multiplied by a learning rate factor and
applied to the model.

Recommendation data can be problematic in this context. Tabular datasets often
include categorical features (e.g. user ID, or product ID). Such arbitrary
identifiers do not carry any inherent meaning for matrix operations. Instead,
an embedding table is used to extract meaning from these categorical
identifiers.

The embedding table maps a categorical ID to a vector of continuous values.
These can be combined with any continuous sample features through an
interaction procedure (e.g. concatenation). The resultant vector can then be
fed through a standard training process. During the SGD parameter updates, the
embedding vectors will be updated as though they were matrix parameters. In
theory, the embedding table is equivalent to transforming input IDs into one-
hot vectors and feeding them into a standard DL model; the embedding table
simply provides a more efficient representation of this same procedure.

Thus, the embedding table is the key to enabling personalized applications
using DL. Multiple major web companies now employ versions of this embedding-
table DL design (e.g. Meta’s DLRM (Acun et al., 2021) and Google’s Deep & Wide
(Cheng et al., 2016)). It adapts the unique challenges of recommendation
problems into a format amenable to DL processing.

But the embedding table introduces scaling challenges. Consider a social media
company with 1B users. They want to be able to recommend posts to users using
a DLRM model. They select a fairly typical embedding vector size, e.g. 128. To
accommodate their 1B users, they build an embedding table with 1B entries.
Each vector entry is a 4-byte float. The resultant table would require more
than 512GB of memory! DL training is typically done on a GPU to accelerate
operations, but a table of this size is too large to fit into the memory of
even state-of-the-art GPUs.

Various techniques have been proposed to tackle this issue, each with their
own tradeoffs. Embedding table compression, for example, where multiple inputs
map to the same vector, tends to degrade accuracy. Another popular solution is
model parallelism, where the table is sharded over multiple GPUs (Wang et al.,
2022). Others have proposed hybrid compute, where the embedding table is split
between system DRAM and GPU memory (Sethi et al., 2022; Adnan et al., 2022).
These scale-out solutions demand powerful and expensive hardware — recent
studies have shown that effective DLRM training requires significant
infrastructure investment in both hardware and software. (Acun et al., 2021).

Figure 3. A data processing pipeline drawn from real practice. We include the
percentage of pipeline latency attributable to each stage to demonstrate the
differing costs of processing.

[A data pipeline illustration with associated stage latencies. The UDF stage
dominates.]Illustration demonstrating what percentage of pipeline latency is
attributable to each processing stage. Disk loading takes 14%, batching 28%,
shuffling 11.7%, UDFs 42.3%, and prefetching 4%.

###  2.2. DLRM Data Processing

The data processing challenges faced in the recommender setting differ from
those seen in other domains (Wang et al., 2021b; Penedo et al., 2023). A
typical recommender dataset is composed of historical user interactions with
the target application. A streaming service, for example, might record user
interactions (i.e. plays, ratings) with movies and shows. Millions of such
interactions could be recorded every day. Recommender models must be retrained
regularly to account for the dataset updates. Each model might target a
different aspect of the application — for example one model might be used for
ordering rows in the UI, while another might be used for video ordering.
Individual models need different features (i.e. columns) of the base dataset
and might require custom preprocessing pipelines. Thus, it is generally
impractical to push preprocessing to the offline stage; per-model
customization encourages online transformation of the same base dataset.

We will now describe a typical online transformation pipeline for one such
model. Figure 3 illustrates, along with estimated latencies drawn from real
industry pipelines.

Samples are loaded from the base dataset in a disk read operation. Each sample
is represented as a dictionary of key-value pairs, mapping feature names to
values. These samples are used to fill up a batch for SGD training. This is
repeated until some significant number of batches are in memory. They are then
shuffled to encourage some randomness in the SGD procedure to improve model
robustness (Goodfellow et al., 2016).

For each batch, a custom dictionary lookup operation is used to extract
relevant feature columns. In this case, we will say that product ID, user ID,
user country, and total product view time are the relevant columns. Note that
this dictionary lookup could be fairly expensive on a feature-rich dataset.

The first three columns are categorical, while the fourth is continuous. Some
random noise is applied to the continuous variable to augment the data and
improve model robustness. To improve training times, several batches will be
“prefetched” at once into GPU memory to overlap the next pipeline loading
phase with model execution, trading memory for performance. At this point, the
pipeline has finished producing a training batch for model consumption.

Millions (or even billions) of recorded interactions will have to run through
these stages to feed and train the model. The throughput rate needed from this
pipeline is dependent on the GPU-driven model execution speed — in the DLRM
case, this typically yields a very high rate requirement. To improve pipeline
performance, two levels of parallelism are possible. First, pipelining. This
simply exploits the stage-by-stage processing structure. Stages can be
overlapped in a similar way to CPU instruction pipelining (Crawford, 1990) to
improve throughput. Maximizing pipeline performance requires a delicate
balancing act. Each transformation stage within the pipeline must take the
same amount of time to avoid idling (Li et al., 2021) . Second, per-stage
replication. Replicating pipeline stages across multiple processors can
improve per-stage throughput significantly. The effect of this replication
interplays with the balance of stage performances, thus impacting pipeline-
parallel throughput. Solving this complex, joint optimization problem
effectively can yield significant performance benefits. At a coarse-grained
level, the entire pipeline itself could be copied across multiple machines
(Acun et al., 2021), but this would discard the opportunity presented by the
joint optimization problem.

These challenges are complicated further by the possibility of machine
resizing. Many clusters now use techniques such as auto-scaling, interruption
& reassignment, or even machine multi-tenancy. In such cases, external
decision-making may cause a job to actually receive new or different resources
across the course of its lifecycle. This setting has become increasingly
popular in recent years as new multi-model training tools (Nagrecha and Kumar,
2022; Moritz et al., 2018; Nagrecha, 2021) have emerged. Effectively
parallelizing such jobs even as the underlying resource pool is actively
shifting requires a level of adaptability and flexibility not present in
existing tooling.

Figure 4. (A) Our study of real job traces shows that compute time is
dominated by data processing rather than model execution, even on the most
compute-intensive models. Jobs using AUTOTUNE are marked in black, jobs using
human-selected distributions are marked in blue, and unoptimized pipelines are
marked in red. (B) Breakdown of individual pipeline stage latencies when using
AUTOTUNE. For each stage, we provide a scatter-plot the percentage of pipeline
time taken up.

[Deepdive into a pipeline, illustrating that human-set approaches are better
than current automated ones.].

###  2.3. Reinforcement Learning

Next, we describe the basics of deep RL that underpin InTune.

The general aim of RL is to train an “agent”, or actor, using data collected
from exploring an environment. The agent can choose from a set of actions in
the environment based on the current state. The state is updated as a result
of the action and a reward is computed to reflect the benefit produced as a
result of the agent’s action. The new state and reward are used to modify the
agent in a way that encourages reward-positive actions and discourages reward-
negative actions.

A variety of techniques can be used to construct this feedback loop. The Deep
Q-Network (DQN) approach uses a DL model as its agent with and SGD for the
feedback loop.

DQN Technique. The agent model is trained to approximate an unknown function
$Q$, where $Q(s,a)$ yields the reward for execution action $a$ in the
environment state $state$. Then, this DL model can compute an expected total
reward for all possible $a$’s at a given state $s$, then select the action
that maximizes the expected reward. The action space should be relatively
small to make this search feasible, as excessively large action spaces are
known to reduce model accuracy (Thrun and Schwartz, 1993). It has become
common practice to employ action space shaping (Kanervisto et al., 2020),
reducing and combining actions to simplify the space. In multi-discrete action
spaces (e.g. a keyboard), wherein multiple simultaneous actions can be taken
at once, the potential action space is exponential with a degree of the
maximum number of simultaneous actions.

Selecting an action from a space requires understanding both the immediate and
long-term reward. To predict “overall” reward of an action, the Optimal Action
Value Function is used (Boyan and Moore, 1994) to shape the agent’s behaviors
and teach it expected rewards over time. Thus, the agent learns a model of its
environment and how its actions will change its state and impact its rewards.

This design works well in settings where responsiveness and adaptability are
important. The agent can actively make decisions in response to environmental
changes, a positive contrast against static one-shot optimizers. We make use
of these properties in our system design to tackle the complex and dynamic
problem of data pipeline optimization.

Online vs Offline RL. An RL agent can be built in either the “offline” or
“online” setting. Offline RL agents are trained in a simulation environment to
understand how the various factors of their environment impact performance.
They rely on the assumption that the final, live environment will be
reasonably similar to the offline simulation settings.

Online RL, by contrast, tunes the agent as it actively interacts with the
target application. This is more flexible and adaptive, but historically, long
convergence times have been a significant concern. Some recent works have
proposed a hybrid of the two, initially pre-training the RL model on offline
simulation data then re-tuning it online (Peng et al., 2021a). The
effectiveness of this hybrid is largely dependent on the specifics of the
target application.

##  3\. Cluster Study

We now analyze training jobs from our real-world DLRM compute cluster to
better understand the key data pipeline challenges faced by practitioners.

###  3.1. Motivation

TensorFlow, a popular DL framework, provides the tf.data API for users to
build input data pipelines from the primitive operations we have discussed
thus far (batch, UDF, shuffle, etc). The new torchdata pipeline composition
tool introduces similar functionality for PyTorch, though the TensorFlow data
pipeline ecosystem is relatively more mature and more appropriate for our
evaluations. The de-facto standard tool for tf.data data pipeline optimization
is AUTOTUNE, which is built-in to TensorFlow (Murray et al., 2021). tf.data is
commonly considered to be one of the most advanced data pipeline construction
tools available to practitioners, and AUTOTUNE is generally accepted to be
state-of-the-art in pipeline optimization (Kuchnik et al., 2022). Due to its
popularity and widespread adoption, we will take it as the standard benchmark
for automated tooling in our cluster study. Historical practice on our
production cluster has surfaced three issues with AUTOTUNE.

  1. (1)

Low efficiency on DLRM pipelines. Tools like AUTOTUNE often produce suboptimal
configurations in practice, bloating runtimes and costs.

  2. (2)

High failure rates. AUTOTUNE has shown a tendency to trigger costly out-of-
memory errors, typically caused by resource-overallocations.

  3. (3)

Poor support for rescaling. Cluster techniques such as machine multi-tenancy
or virtualization can add new resources to jobs over time. Unfortunately,
AUTOTUNE does not take full advantage of the new resources without human
intervention.

We can validate these three points through quantitative analyses of DLRM job
traces on our cluster. We recorded jobs run over a period of two weeks for our
study.

Figure 5. Deepdive into a case-study pipeline. (A) The percentage of target
throughput achieved with different approaches. (B) Illustration of AUTOTUNE’s
versus human-set alternatives. (C) Performance of various tools when the
machine is re-sized during job execution. Approaches with manual intervention
use the -Adaptive suffix, and results are normalized to the Single-CPU-per-
stage baseline.

[Deepdive into a pipeline, illustrating that human-set approaches are better
than current automated ones.].

###  3.2. Cluster Trace Analyses

We take data from a large GPU cluster reserved only for DL recommendation
model training. A broad mix of job-types are present; both exploratory ad-hoc
experimentation as well as automated production pipelines. Our results show
that as much as 60% of time is spent on data ingestion rather than model
execution, even when AUTOTUNE is applied. Manually-optimized jobs tend to
perform somewhat better, while unoptimized jobs see the worst skew towards
data processing. In all cases, we see a significant opportunity for
improvement — reducing data processing times would provide significant cost
savings and efficiency improvements. Figure 4(A) presents our data.

Next, we dive into the fine-grained stages of the data pipeline, to better
understand the composition of the end-to-end costs. All data pipelines in this
analysis use AUTOTUNE, and follow a standard order of disk load $\rightarrow$
batch $\rightarrow$ shuffle $\rightarrow$ UDF $\rightarrow$ prefetch. These
pipelines do not include a “cache” stage due to memory constraints; these jobs
operate with high-dimensional features & very large datasets. Figure 4(B)
shows the results.

Maximizing pipeline throughput requires achieving equal latencies across each
stage (Li et al., 2021). But AUTOTUNE is known to struggle with irregular
stages such as UDFs or varied-size disk loads (Kuchnik et al., 2022). Our
empirical study confirms this issue at a mass-scale. On average, UDF mappings
and disk loading dominate runtimes, and the skew towards UDFs tends to grow as
overall data pipeline latencies increase. The proportion of time spent on
shuffling or batching tends to stay mostly consistent regardless of overall
pipeline latency, further pointing to UDFs as the primary stumbling-block for
AUTOTUNE. Unfortunately, UDFs are a key piece of most of DLRM data pipelines,
covering basic tasks such as feature extraction, re-mapping, and
transformation. A previous study (Zhao et al., 2022) of DLRM training
describes 16 common preprocessing operations; we found that 14 of these 16
required UDF implementations! Poor UDF optimization alone is sufficient to
discourage AUTOTUNE adoption among our users.

###  3.3. Pipeline Deep Dive

To gain more detailed insights, we will now analyze a singular production
training pipeline from Netflix as a case study. This job is rerun on a regular
basis, multiple times per day, allowing us to collect a rich history of
statistics. Training jobs are run on machines with 128 Intel Xeon 3.0GHz CPUs,
and datasets are stored on a remote network filesystem. One of our primary
aims in this section is to demonstrate how current state-of-the-art tooling
fails to serve our needs. To illustrate this, we labeled jobs according to
whether they used AUTOTUNE, human-set configurations, or else no optimization
at all (i.e. one CPU per stage). We contrast these approaches in our
experiments.

The model is relatively small — under 10M parameters. The model latency is
very low, so to avoid idle times, the data pipeline must offer a high
throughput rate. The data processing pipeline requires: (1) loading data from
disk, (2) shuffling it in a fixed buffer, (3) applying a UDF to extract and
convert categorical features to standard mappings, then (4) batching the data
before (5) prefetching it to the GPU. If only one CPU is given to each stage,
pipeline throughput is 11% of the data-loading rate needed to keep the model
served at all times (i.e. no idling), as shown in Figure 5(A). After AUTOTUNE
distributes all these processors, pipeline throughput is increased by 2.81X to
31% of the target rate.

We contrast this against manually chosen allocations, which increases the
pipeline throughput to 41% of the target rate. Further improvements (e.g. to
100% of the rate, with no idle times in model execution) would require scaling
beyond the machine’s current resources. But even within the machine, we found
scope for a 1.34X speedup versus AUTOTUNE! Ideally, we should be able to
produce this configuration automatically, without manual intervention.

Another serious issue we observe in applying AUTOTUNE to this example pipeline
is overallocation. If we allow AUTOTUNE to take control of the prefetch stage,
it tries to improve performance by maximizing prefetches. This bloats memory
usage, often causing OOM errors. Figure 5(B) illustrates the frequency of OOM
errors produced by applying AUTOTUNE to this pipeline. Recovering from these
errors requires a teardown and reset, leading to significant downtime.

An increasingly popular technique in large-scale compute clusters is machine
resizing (Narayanan et al., 2020; Nagrecha and Kumar, 2022; Moritz et al.,
2018), either from scheduler interruption & re-assignment (Xiao et al., 2018,
2020), or job completions on a multi-tenant machine (Nagrecha and Kumar,
2022). Rescaling the data pipeline to make use of new resources requires the
optimizer to actively respond to hardware changes. Figure 5(C) charts the
performance of various approaches in this setting. We see that out-of-the-box
AUTOTUNE does not respond effectively to machine re-sizing, and does not
increase pipeline throughput even when new CPUs are provided to the job. If
human intervention is used to reset and re-launch AUTOTUNE, the scaling
improves but is still significantly under-performs the fully-human
alternative. This reliance on human decision-making and intervention is
impractical.

Now that we have observed DLRM data pipelining problems in a real training
cluster, we seek a solution that can address these challenges.

##  4\. System Design

Our studies in Section 3 surfaced three issues with current tooling: (1)
suboptimal performance on DLRM pipelines due to UDFs, (2) inability to scale
as resource caps are changed, and (3) a tendency to overallocate resources,
leading to OOMs. We propose that all three of these problems would be resolved
by a feedback-driven tool. A tool that actively evaluates its environment and
collects feedback live would be able to (1) fine-tune its understanding of UDF
performance throughout training, (2) actively respond to changing resource
caps, and (3) manage memory usage.

With this in mind, we turn to RL to design our tool for data pipeline
optimization — InTune. As we discussed in Section 2, building an RL agent for
data pipeline allocation requires us an environment, an agent, and an action
space. We now discuss these RL system components.

Table 2. RL environment factors. |  Factor |  Motivation
---|---|---
Agent-Modified Factors |  Pipeline Latency |  Allows agent to understand the performance of the current configuration. May change based on agent actions.
Free CPUs |  Allows agent to see how many extra CPUs it can allocate. May change based on (1) agent action or (2) machine resizing.
Free Memory (bytes) |  Allows agent to see how much memory it can use to increase prefetch levels. May change based on agent actions.
Uncorrelated Factors |  Model Latency |  The actual model execution time. Updated regularly to improve estimation accuracy. Unrelated to agent actions.
Static Factors |  DRAM-CPU Bandwidth (MB/s) |  Interconnect speed can impact the value of prefetching. Found up front and unchanged throughout training.
CPU Processing Speed (GHz) |  CPU processing speed can impact decision-making on resource allocation. Found up front and unchanged throughout training.

###  4.1. Environment

The environment in our setting should reflect the data pipeline state and
available hardware. Certain aspects of the environment are static (e.g. DRAM-
CPU bandwidth), others are uncorrelated to the agent’s actions (e.g. model
latency), while others are directly impacted by its actions (memory usage, CPU
usage). We model our environment with the aim of providing the RL agent with
any and all information it might need to make an informed decision. Our
finalized list of factors is shown in in Table LABEL:tb:factors.

Figure 6. InTune’s RL data pipeline system architecture.

[InTune’s RL agent optimizes the user-specified data pipeline, which is then
used to load samples for the model. The training performance is profiled and
used as feedback for the agent.].

These details are sufficient for the agent to quickly grasp its problem
setting. The static factors will provide some “immediate” information while
the other aspects will help it to learn how its actions impact data pipeline
performance. Our agent reward is directly based on data pipeline latency and
memory usage. Equation 1 shows the function.

(1) $R=throughput\times(1-\frac{memory_{used}}{memory_{total}})$

If prefetch is not used excessively, then the memory usage portion of the
equation is largely irrelevant. But to avoid OOM outcomes like those seen with
AUTOTUNE, we ensure that InTune’s reward approaches 0 as memory consumption
nears 100%.

###  4.2. Agent Model

We aim to build a low-cost, lightweight model architecture for the RL agent.
Since InTune runs in parallel with the target DL job, we do not want to over-
consume resources. To minimize computational demands, InTune’s DQN agent is a
simple three-layer MLP architecture using the ReLU activation function, built
in PyTorch. It can be run on either CPU or GPU resources, or even as a remote
service interacting with the target job over the network.

If the action space consists of ¡256 possible choices, this model only
requires ¡200FLOPs per iteration, which should not interfere excessively with
the actual model training job. We train different versions of the agent in
offline simulations to prepare them for live deployment/tuning. Each version
is built for a different common pipeline length on our clusters (e.g. one
agent for 4-stage pipelines, one for 5-stage, etc). During actual data
ingestion, the model is fine-tuned using live feedback to adapt it for the
current job. We report on convergence behaviors in Section 5.1.

###  4.3. Action Space

As we discussed in Section 2.3, it is common practice to reshape the agent
action space to improve accuracy. If we allowed our agent to directly select
any distribution of resources, the size of InTune’s action space would be
$n+r-1\choose r-1$ where $n$ is the number of CPUs and $r$ is the number of
pipeline stages. On a typical setup (e.g. 128 CPUs over 5 stages), this would
yield $1.2e7$ possible actions — which is entirely impractical. Based on the
agent we described in Section 4.2, this would increase iteration compute costs
to more than 6.1GFLOPs!

Instead, we use action-space reshaping, and design an incremental action
space. At every step, InTune’s agent can choose to “raise-by-one”, “maintain”,
or “lower-by-one” the allocation of each stage. Memory-bound factors use a
megabyte unit while processing-bound factors use a CPU unit. On its own, this
is somewhat inefficient. In order for the system to allocate 128 CPUs, a
minimum of $128/n$ iterations would be required. To improve search and
convergence times, we give the agent additional options of “raise-by-five” and
“lower-by-five”. This yields an action space of $5^{r}$ options. Since $r$ is
typically $<=5$, this is entirely manageable. Increasing the action space by
adding new options (e.g. “raise-by-ten”) could be used to further modify the
convergence behaviors, but we found that increment options of 1 and 5 are
sufficient for InTune to rapidly converge on a performant solution.

These three components — environment, model, and action space — provide the
basis for InTune.

###  4.4. Interface & Usage

We aim to make InTune easy to integrate into existing user code, without
disrupting workflows or requiring cluster architecture changes. Users design
their data ingestion pipelines in standard framework code (e.g. PyTorch,
tf.data), then wrap their pipeline under InTune, specifying any tunable
performance knobs. Performance monitoring and value adjustment is all handled
automatically by InTune. Listing 1 provides an example.

⬇

1pipeline = create_pipeline()

2system_pipeline = intune.pipeline_wrapper(pipeline, knobs=[pipeline.stage_1,
pipeline.stage_2...])

3...

4train(model, system_pipeline) # replace references to pipeline with
system_pipeline

Listing 1: InTune usage.

Figure 7. All figures use the legend in the leftmost chart. (A) Pipeline
throughput over time for each approach, normalized to the Unoptimized
baseline. (B) CPU utilization over time for each approach. Only active CPUs
are considered, to prevent confounding system behaviors with the separate
impact of rescaling. (C) GPU Utilization over time for each approach.

[In the first figure, we demonstrate throughput on the Netflix & Criteo
pipelines. InTunesignificantly outperforms all other baselines. In the second
and third figures, we depict CPU & GPU utilization numbers, where InTune again
produces the best results.].

We illustrate the overall design in Figure 6. Note the generality of InTune’s
design; nothing about it is tied to a specific data pipeline framework. So
long as the framework exposes optimization knobs (e.g. for CPU assignment),
this approach is applicable.

##  5\. Evaluation

We now provide a thorough evaluation of InTune. Our aim is to answer the
following questions.

  1. (1)

Does InTune capable of achieving higher pipeline throughput than standard
tools such as AUTOTUNE?

  2. (2)

Is InTune less susceptible to issues such as out-of-memory errors than
standard tooling?

  3. (3)

Is InTune capable of responding to resource rescaling?

  4. (4)

Does InTune converge on an optimized solution quickly?

Workloads. We use two workloads, one drawn from an internal recommender model
and dataset and one built using Meta’s open-source DLRM code and the Criteo
dataset. The custom dataset task focuses on product recommendations while the
Criteo dataset is used in a click-through-rate prediction task.

Datasets and Pipelines. The custom dataset uses dozens of sparse features, and
fewer than 5 continuous features, with a batch size in the tens of thousands.
The Criteo dataset consists of 26 sparse categorical features and 13
continuous features and a batch size of 24,096. We initialize the dataloader
allocations with a simple “even division” of CPUs across stages. The RL agent
then modifies the distribution provided by this heuristic. We do not consider
a cache stage in the current version, since most of the relevant jobs on our
cluster do not use one, but there is no reason InTune could not be extended to
manage the resource allocations of cache stages as well.

Models. The model taken from our production pipeline is fairly small — ¡5M
parameters, most of which are contained in the embedding tables. We make a
large model for the Criteo dataset, with 25B+ parameters, most of which are in
the embedding tables. In both cases, the model latency is sufficiently low
such that training times are dominated by data processing.

Hardware Setup. The production model is trained on a single 40GB A100 GPU. We
initially provide the data pipeline with 32 Intel Xeon 3.0GHz CPUs, then on
regular intervals double the CPU count up to a limit of 128. Then, we halve
the allocation repeatedly until we reach 32 CPUs again. Approaches other than
InTune rescale via manual intervention and re-launching; InTune will naturally
adapt to the environmental change and so does not require this intervention
step. The Criteo model is too large for a single A100 to train, so we use a
standard hybrid parallelism approach (Wang et al., 2022) to distribute memory
demands and accelerate training. We use the same CPU scaling procedure applied
to the custom model (i.e. $32\rightarrow 64\rightarrow 128\rightarrow
64\rightarrow 32$). The datasets are stored on a high-bandwidth network-
mounted filesystem, a common approach for large-scale recommendation datasets.

Baselines: We compare InTune to the following baselines.

  1. (1)

Unoptimized. In this version, only a single CPU is allocated per stage such
that no parallelism is possible.

  2. (2)

AUTOTUNE. AUTOTUNE is a standard TensorFlow offering, commonly used to
optimize tf.data pipelines.

  3. (3)

AUTOTUNE-Adaptive. We checkpoint and re-launch AUTOTUNE on machine rescaling
intervals to allow it to adapt to the new machine resources.

  4. (4)

Plumber-Adaptive. Plumber (Kuchnik et al., 2022) is a more user-friendly
alternative to AUTOTUNE that uses an MILP to distribute resources. We apply
the same checkpointing approach as in AUTOTUNE-adaptive. Plumber offers
additional auto-caching optimizations; we disable these since they are
orthogonal to our resource-distribution work with InTune and could be
integrated with InTune in the future.

  5. (5)

Heuristic. This version simply distributes CPUs evenly to emulate what a human
user’s best guess distribution might look like. InTune’s RL agent is
initialized from this state as well.

These baselines cover most typical configurations, both manual and automated.

###  5.1. End-to-End Performance

Pipeline Performance. We compare achieved training throughput for all
approaches on both the real-world and Criteo pipelines. We initiate rescales
at regular intervals to evaluate how each system “responds” to the new
resource availability. We normalize throughput to the Unoptimized baseline in
all our analyses. Figure 7 (A) presents the results.

InTune provides significantly better throughput and hardware utilization than
the strongest competitors on both pipelines. Accounting for flexible
rescaling, the average marginal gain versus standard AUTOTUNE tooling
increases to 2.05X and 2.29X on the custom pipeline and Criteo pipeline
respectively. Against the alternatives which employ human intervention, the
marginal improvement is still significant, ranging between 10-20%. Not only
does our approach eliminate the headache of manual intervention, but it also
achieves lower compute times and higher utilization. In each experiment, we
observe that InTune achieves a stable throughput rate within about 10 minutes.
The Plumber baseline also requires some tens of iterations to converge, but
this period is so short it does not register on the chart. On long-running
jobs, InTune’s 10-minute optimization time is insignificant, but it may be
problematic for short ad-hoc experiments. But in such cases, fine-tuned
performance optimization is rarely important.

We also observe significantly lower failure rates than AUTOTUNE. On average,
AUTOTUNE caused an 8% OOM failure rate on both pipelines, whereas InTune did
not cause even a single crash. This improved robustness makes InTune an
attractive option for failure-sensitive jobs.

InTunealso achieves higher processor utilization, illustrated in Figures 7 (B)
& (C) likely due to reduced idling from more effective resource allocations.
Some part of the CPU utilization increase can also be attributed to the
overhead of maintaining a secondary RL model; unfortunately it is difficult
for us to separate the two. The improved GPU utilization follows directly from
the higher data throughput, since the GPU & model are fed faster.

Intuition on Effectiveness. InTune’s ability to map out and tune its
performance estimates over time allow it to adapt and outperform other
baselines on both pipelines. No other system can adapt as effectively to
machine re-sizing out-of-the-box. The improvements against baselines which
employ human intervention can be attributed to one of the other primary
weaknesses we observed in Section 3, UDF performance modeling. As we will show
in Section 5.2.1, InTune proves to be significantly better than the strongest
baseline — AUTOTUNE — in optimizing UDF pipeline stages.

###  5.2. Drilldown Studies

We now dive into InTune’s scaling behaviors on the real-world data pipeline.
We aim to answer the following questions.

  1. (1)

How does InTune’s performance change as the pipeline’s complexity is changed?

  2. (2)

How does InTune’s performance change as CPU counts are changed?

  3. (3)

How does InTune’s performance change as batch size changes?

Figure 8. Performance scaling with respect to (A) pipeline complexity, (B) CPU
count, and (C) batch size.

[We determine how InTune’s performance scales with pipeline complexity, CPU
count, and batch size. In general, we find that InTune’s marginal improvements
over baselines like AUTOTUNE scale up with pipeline complexity and are
relatively consistent versus CPU count and batch size.].

####  5.2.1. Pipeline Complexity Scaling

We report on performance normalized to the AUTOTUNE baseline on the same
pipeline. All settings use the same machine, with 128 Intel Xeon 3.0GHz CPUs,
and a constant model latency of 0s (to encourage maximal pipeline
optimization). Pipeline “complexity” is adjusted by increasing/decreasing
pipeline length (e.g. \+ batching, \+ shuffling). Figure 8(A) presents the
results. We see that our system’s marginal improvement over the AUTOTUNE
baseline grows as pipeline complexity increases, with a spike when UDFs are
introduced. This corroborates earlier studies which found that AUTOTUNE
underperforms on more complex, UDF-driven pipelines (Kuchnik et al., 2022).

####  5.2.2. Machine Size Scaling

We now study the scaling efficacy of InTune. We increase CPU count in
increments of 2, ranging from 8 to 128. AUTOTUNE is re-launched at each
machine size to rebase the relative performance. Figure 8(B) presents the
results. InTune’s relative improvements over AUTOTUNE tend to grow as the
valid configuration space increases, but then flattens out to a constant
outperformance of roughly 20%. This flattening should be expected; AUTOTUNE is
a strong baseline and even a 20% performance margin is significant (Kuchnik et
al., 2022).

####  5.2.3. Batch Size Scaling

In our final scaling study, we evaluate our system’s performance with respect
to batch size. Like the pipeline complexity study, we evaluate our system’s
ability to respond to varied workload intensity. Larger batch sizes increase
demands on specific stages (i.e. the batch stage, prefetch stage, and possibly
the UDF stage). Since end-system users might wish to deploy the system on any
range of batch sizes, we implement this study to give a more thorough
understanding of our system’s offerings to users. Figure 8(C) presents the
results. We see that our system manages to maintain (and even improve) average
sample throughput even as batch size increases.

##  6\. Related Work

We now discuss related work & prior art. We can divide these works into three
major categories: data pipeline tools, DL resource allocation tools, and works
on RL for systems.

###  6.1. Data Pipeline Optimizers

AUTOTUNE is generally considered to be the gold-standard data pipeline
optimizer for tf.data pipelines (Murray et al., 2021). Built-in to TensorFlow,
the tool offers users with a seamless optimization experience that can be
added to existing pipelines in just a single line of code. This design
philosophy of abstraction has its detractors; recent works (Kuchnik et al.,
2022) have criticized its black-box approach to optimization. AUTOTUNE is
designed to support any tf.data pipeline, but its generality leaves it
vulnerable to task-specific issues, such as those we outline in Section 1 and
explore in Section 3. InTune is more narrowly focused than AUTOTUNE in target
workloads, but is also more general in its support for non-tf.data pipelines.

Plumber was introduced as a more user-friendly alternative to AUTOTUNE (but
still restricted to tf.data). It uses a linear programming solver to determine
a resource allocation. However, in practice it often underperforms AUTOTUNE
(Kuchnik et al., 2022) in its allocations. It does offer the ability to
automatically inject caching into the pipeline to improve performance. A
future version of our system could borrow this optimization from Plumber and
add caching as an action for the agent.

DALI & NVTabular offer GPU-accelerated data-loader primitives for image and
tabular data modalities respectively In practice, we find that NVTabular is
more suited to offline feature engineering, since using the GPU for online
data ingestion can lead to contention over cycles between the pipeline and the
model. As a result, practitioners on our cluster have generally found it
impractical to adopt NVTabular for our target setting.

CoorDL proposed a set of carefully-designed techniques to eliminate data
stalls, including sophisticated caching procedures (Mohan et al., 2020). We
address the related, but orthogonal, problem of data pipeline parallelization
and throughput optimization. We leave it to future work to combine their
results with our own. Another recent work (Isenko et al., 2022) focused on
analyzing various training pipelines and identifying typical bottlenecks. They
characterize general pipeline design spaces; by contrast, our work focuses
specifically on DLRM challenges.

Meta’s Data PreProcessing Service (Zhao et al., 2022) tackles a similar
problem to ours — online data ingestion pipelines for large-scale DLRM
training. Their work primarily focuses on understanding performance issues in
Meta’s cluster, and describing their “disaggregated data service” approach.
The idea of the service is to place replicas of the data ingestion pipeline on
additional, separate CPU servers, which feed the trainer machine the data
samples over the network. The opportunity to scale beyond trainer machine CPUs
is attractive, but adopting this approach requires significant cluster
redesigns, altering user workflows, and creating new machine scheduling
schemes. In addition, replicating the entire pipeline over multiple machines
can be wasteful if the bottleneck is only at a single processing stage. By
contrast, our work focuses on maximizing the effectiveness of CPUs already
present on the trainer machine, so that no alterations to the user workflow or
cluster setup are necessary. In theory, the two approaches are orthogonal —
our CPU-distribution scheme could be applied to each of the machine replicas
used in Meta’s disaggregated data-loading design.

Other query optimization tools (Nakandala et al., 2020; Kumar et al., 2021;
Nakandala and Kumar, 2022; Huang et al., 2019; Wang et al., 2021a) exist, but
target settings other than data pipeline optimization.

###  6.2. DL Resource Allocation

Some works (e.g. Saturn (Nagrecha and Kumar, 2023), Pollux (Qiao et al.,
2021), Optimus (Peng et al., 2018) and DL2 (Peng et al., 2021b)) have tackled
GPU apportioning in the scheduling setting. Still other works (Li et al.,
2018) consider general resource apportionment for hyperparameter tuning. Like
InTune, these tools reduce the manual configuration burden in the DL training
process.

###  6.3. Deep Reinforcement Learning for Systems

Deep RL has become increasingly popular in recent years for various systems
applications (Haj-Ali et al., 2019). Several works have tackled resource
allocation using RL (Peng et al., 2021a; He et al., 2017; Tesauro et al.,
2005). They aim to use the flexibility of RL to tackle the complexity and
dynamic nature of intractable, online problems. Similarly, our work exploits
the flexibility of RL to meet the needs of recommender data ingestion
pipelines that are unaddressed by existing systems. Others have applied RL for
SQL optimization (Zhong et al., 2017; Marcus et al., 2019; Marcus and
Papaemmanouil, 2018; Ortiz et al., 2018; Krishnan et al., 2018). The use of a
learned algorithm helps relax the need for exact information that may be
impractical to obtain in large RDBMSs. Our work also uses RL to relax the need
for exact profiling of blackbox UDFs.

##  7\. Conclusion

DLRM training costs are often dominated by online data ingestion rather than
model execution. The primary throughput bottleneck in this setting is CPU-
driven data processing rather than GPU-accelerated model operations. Thus,
optimizing the data ingestion phase is critical to ensuring cost- & time-
effective model development. Unfortunately, existing tooling for DL data
ingestion pipelines does not support DLRM setting effectively. We draw on
lessons learned from analyses of real DLRM workloads from our training cluster
at Netflix to motivate the design of a novel RL-based system we name InTune.
InTune dynamically allocates CPUs & memory across stages of the online data
ingestion pipeline, significantly improving efficiency over industry-standard
baselines without requiring changes to the cluster architecture or user
workflows. Benchmarks on real & synthetic training pipelines show that our
system outperforms the strongest out-of-the-box tools by ¿2X, and human-
managed baselines by up to 20%. Overall, InTune offers significant performance
and cost benefits for recommender training pipelines and should serve to
encourage further training optimization works customized for the unique needs
of DLRM training. Future extensions could extend InTune to other decisions in
the DLRM data pipeline; e.g. intermediate caching, or how to scale the
pipeline across multiple machines.

## References

  * (1)
  * Acun et al. (2021) Bilge Acun, Matthew Murphy, Xiaodong Wang, Jade Nie, Carole-Jean Wu, and Kim Hazelwood. 2021\.  Understanding training efficiency of deep learning recommendation models at scale. In _2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA)_. IEEE, 802–814.
  * Adnan et al. (2022) Muhammad Adnan, Yassaman Ebrahimzadeh Maboud, Divya Mahajan, and Prashant J Nair. 2022\.  Heterogeneous Acceleration Pipeline for Recommendation System Training.  _arXiv preprint arXiv:2204.05436_ (2022).
  * Boyan and Moore (1994) Justin Boyan and Andrew Moore. 1994.  Generalization in reinforcement learning: Safely approximating the value function.  _Advances in neural information processing systems_ 7 (1994).
  * Cheng et al. (2016) Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. 2016\.  Wide & deep learning for recommender systems. In _Proceedings of the 1st workshop on deep learning for recommender systems_. 7–10.
  * Crawford (1990) John H Crawford. 1990\.  The i486 CPU: Executing instructions in one clock cycle.  _IEEE Micro_ 10, 1 (1990), 27–36.
  * Goodfellow et al. (2016) Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016\.  _Deep Learning_.  MIT Press.  http://www.deeplearningbook.org.
  * Gupta et al. (2020) Udit Gupta, Carole-Jean Wu, Xiaodong Wang, Maxim Naumov, Brandon Reagen, David Brooks, Bradford Cottel, Kim Hazelwood, Mark Hempstead, Bill Jia, et al. 2020\.  The architectural implications of facebook’s dnn-based personalized recommendation. In _2020 IEEE International Symposium on High Performance Computer Architecture (HPCA)_. IEEE, 488–501.
  * Haj-Ali et al. (2019) Ameer Haj-Ali, Nesreen K. Ahmed, Ted Willke, Joseph Gonzalez, Krste Asanovic, and Ion Stoica. 2019\.  A View on Deep Reinforcement Learning in System Optimization.  https://doi.org/10.48550/ARXIV.1908.01275
  * Harlap et al. (2018) Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil Devanur, Greg Ganger, and Phil Gibbons. 2018.  PipeDream: Fast and Efficient Pipeline Parallel DNN Training.  https://doi.org/10.48550/ARXIV.1806.03377
  * He et al. (2017) Ying He, F. Richard Yu, Nan Zhao, Victor C. M. Leung, and Hongxi Yin. 2017.  Software-Defined Networks with Mobile Edge Computing and Caching for Smart Cities: A Big Data Deep Reinforcement Learning Approach.  _IEEE Communications Magazine_ 55, 12 (2017), 31–37.
  * Huang et al. (2019) Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. 2019\.  Gpipe: Efficient training of giant neural networks using pipeline parallelism.  _Advances in neural information processing systems_ 32 (2019).
  * Huang et al. (2018) Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Mia Xu Chen, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. 2018.  GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism.  https://doi.org/10.48550/ARXIV.1811.06965
  * Isenko et al. (2022) Alexander Isenko, Ruben Mayer, Jeffrey Jedele, and Hans-Arno Jacobsen. 2022.  Where Is My Training Bottleneck? Hidden Trade-Offs in Deep Learning Preprocessing Pipelines. In _Proceedings of the 2022 International Conference on Management of Data_. ACM.  https://doi.org/10.1145/3514221.3517848
  * Jia et al. (2018) Zhihao Jia, Matei Zaharia, and Alex Aiken. 2018\.  Beyond Data and Model Parallelism for Deep Neural Networks.  https://doi.org/10.48550/ARXIV.1807.05358
  * Kanervisto et al. (2020) Anssi Kanervisto, Christian Scheller, and Ville Hautamäki. 2020.  Action Space Shaping in Deep Reinforcement Learning.  https://doi.org/10.48550/ARXIV.2004.00980
  * Krishnan et al. (2018) Sanjay Krishnan, Zongheng Yang, Ken Goldberg, Joseph Hellerstein, and Ion Stoica. 2018.  Learning to Optimize Join Queries With Deep Reinforcement Learning.  https://doi.org/10.48550/ARXIV.1808.03196
  * Kuchnik et al. (2022) Michael Kuchnik, Ana Klimovic, Jiri Simsa, Virginia Smith, and George Amvrosiadis. 2022.  Plumber: Diagnosing and removing performance bottlenecks in machine learning data pipelines.  _Proceedings of Machine Learning and Systems_ 4 (2022), 33–51.
  * Kumar et al. (2021) Arun Kumar, Supun Nakandala, Yuhao Zhang, Side Li, Advitya Gemawat, and Kabir Nagrecha. 2021\.  Cerebro: A layered data platform for scalable deep learning. In _11th Annual Conference on Innovative Data Systems Research (CIDR ‘21)_.
  * Lepikhin et al. (2020) Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2020\.  GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding.  https://doi.org/10.48550/ARXIV.2006.16668
  * Li et al. (2018) Liam Li, Kevin Jamieson, Afshin Rostamizadeh, Ekaterina Gonina, Moritz Hardt, Benjamin Recht, and Ameet Talwalkar. 2018.  Massively parallel hyperparameter tuning.  _arXiv preprint arXiv:1810.05934_ 5 (2018).
  * Li et al. (2021) Zhuohan Li, Siyuan Zhuang, Shiyuan Guo, Danyang Zhuo, Hao Zhang, Dawn Song, and Ion Stoica. 2021.  TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models.  https://doi.org/10.48550/ARXIV.2102.07988
  * Marcus et al. (2019) Ryan Marcus, Parimarjan Negi, Hongzi Mao, Chi Zhang, Mohammad Alizadeh, Tim Kraska, Olga Papaemmanouil, and Nesime Tatbul. 2019\.  Neo.  _Proceedings of the VLDB Endowment_ 12, 11 (jul 2019), 1705–1718.  https://doi.org/10.14778/3342263.3342644
  * Marcus and Papaemmanouil (2018) Ryan Marcus and Olga Papaemmanouil. 2018.  Deep Reinforcement Learning for Join Order Enumeration. In _Proceedings of the First International Workshop on Exploiting Artificial Intelligence Techniques for Data Management_. ACM.  https://doi.org/10.1145/3211954.3211957
  * Mohan et al. (2020) Jayashree Mohan, Amar Phanishayee, Ashish Raniwala, and Vijay Chidambaram. 2020.  Analyzing and Mitigating Data Stalls in DNN Training.  https://doi.org/10.48550/ARXIV.2007.06775
  * Moritz et al. (2018) Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I. Jordan, and Ion Stoica. 2018.  Ray: A Distributed Framework for Emerging AI Applications.  arXiv:1712.05889 [cs.DC]
  * Mudigere et al. (2021) Dheevatsa Mudigere et al. 2021\.  Software-Hardware Co-design for Fast and Scalable Training of Deep Learning Recommendation Models.  https://doi.org/10.48550/ARXIV.2104.05158
  * Murray et al. (2021) Derek G Murray, Jiri Simsa, Ana Klimovic, and Ihor Indyk. 2021\.  tf. data: A machine learning data processing framework.  _arXiv preprint arXiv:2101.12127_ (2021).
  * Nagrecha (2021) Kabir Nagrecha. 2021\.  Model-parallel model selection for deep learning systems. In _Proceedings of the 2021 International Conference on Management of Data_. 2929–2931.
  * Nagrecha (2023) Kabir Nagrecha. 2023\.  Systems for Parallel and Distributed Large-Model Deep Learning Training.
  * Nagrecha and Kumar (2022) Kabir Nagrecha and Arun Kumar. 2022.  Hydra: A System for Large Multi-Model Deep Learning.  arXiv:2110.08633 [cs.DC]
  * Nagrecha and Kumar (2023) Kabir Nagrecha and Arun Kumar. 2023.  Saturn: An Optimized Data System for Multi-Large-Model Deep Learning Workloads (Information System Architectures).  (2023).
  * Nakandala and Kumar (2022) Supun Nakandala and Arun Kumar. 2022.  Nautilus: An Optimized System for Deep Transfer Learning over Evolving Training Datasets. In _Proceedings of the 2022 International Conference on Management of Data_. 506–520.
  * Nakandala et al. (2020) Supun Nakandala, Kabir Nagrecha, Arun Kumar, and Yannis Papakonstantinou. 2020.  Incremental and approximate computations for accelerating deep CNN inference.  _ACM Transactions on Database Systems (TODS)_ 45, 4 (2020), 1–42.
  * Narayanan et al. (2020) Deepak Narayanan, Keshav Santhanam, Fiodar Kazhamiaka, Amar Phanishayee, and Matei Zaharia. 2020\.  Heterogeneity-aware cluster scheduling policies for deep learning workloads. In _Proceedings of the 14th USENIX Conference on Operating Systems Design and Implementation_. 481–498.
  * Olukotun (2022) Kunle Olukotun. 2022\.  Systems for ML and ML for Systems: A Virtuous Cycle. (2022).  https://mlsys.org/virtual/2022/invited-talk/2065 MLSys.
  * Ortiz et al. (2018) Jennifer Ortiz, Magdalena Balazinska, Johannes Gehrke, and S. Sathiya Keerthi. 2018.  Learning State Representations for Query Optimization with Deep Reinforcement Learning.  https://doi.org/10.48550/ARXIV.1803.08604
  * Penedo et al. (2023) Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023\.  The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only.  arXiv:2306.01116 [cs.CL]
  * Peng et al. (2018) Yanghua Peng, Yixin Bao, Yangrui Chen, Chuan Wu, and Chuanxiong Guo. 2018.  Optimus: an efficient dynamic resource scheduler for deep learning clusters. In _Proceedings of the Thirteenth EuroSys Conference_. 1–14.
  * Peng et al. (2021a) Yanghua Peng, Yixin Bao, Yangrui Chen, Chuan Wu, Chen Meng, and Wei Lin. 2021a.  DL2: A Deep Learning-Driven Scheduler for Deep Learning Clusters.  _IEEE Transactions on Parallel and Distributed Systems_ 32, 8 (2021), 1947–1960.  https://doi.org/10.1109/TPDS.2021.3052895
  * Peng et al. (2021b) Yanghua Peng, Yixin Bao, Yangrui Chen, Chuan Wu, Chen Meng, and Wei Lin. 2021b.  Dl2: A deep learning-driven scheduler for deep learning clusters.  _IEEE Transactions on Parallel and Distributed Systems_ 32, 8 (2021), 1947–1960.
  * Qiao et al. (2021) Aurick Qiao, Sang Keun Choe, Suhas Jayaram Subramanya, Willie Neiswanger, Qirong Ho, Hao Zhang, Gregory R Ganger, and Eric P Xing. 2021\.  Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learning.. In _OSDI_ , Vol. 21. 1–18.
  * Ren et al. (2021) Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, and Yuxiong He. 2021.  ZeRO-Offload: Democratizing Billion-Scale Model Training.  https://doi.org/10.48550/ARXIV.2101.06840
  * Sethi et al. (2022) Geet Sethi, Bilge Acun, Niket Agarwal, Christos Kozyrakis, Caroline Trippel, and Carole-Jean Wu. 2022\.  RecShard: statistical feature-based memory optimization for industry-scale neural recommendation. In _Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems_. 344–358.
  * Shoeybi et al. (2019) Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019\.  Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism.  https://doi.org/10.48550/ARXIV.1909.08053
  * Tesauro et al. (2005) Gerald Tesauro, Rajarshi Das, and Nicholas K. Jong. 2005\.  Online Performance Management Using Hybrid Reinforcement Learning.
  * Thrun and Schwartz (1993) Sebastian Thrun and Anton Schwartz. 1993.  Issues in using function approximation for reinforcement learning. In _Proceedings of the Fourth Connectionist Models Summer School_ , Vol. 255. Hillsdale, NJ, 263.
  * Wang et al. (2021a) Guanhua Wang, Zhuang Liu, Brandon Hsieh, Siyuan Zhuang, Joseph Gonzalez, Trevor Darrell, and Ion Stoica. 2021a.  sensai: Convnets decomposition via class parallelism for fast inference on live data.  _Proceedings of Machine Learning and Systems_ 3 (2021), 664–679.
  * Wang et al. (2021b) Pei Wang, Kabir Nagrecha, and Nuno Vasconcelos. 2021b.  Gradient-based algorithms for machine teaching. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 1387–1396.
  * Wang et al. (2022) Zehuan Wang, Yingcan Wei, Minseok Lee, Matthias Langer, Fan Yu, Jie Liu, Shijie Liu, Daniel G Abel, Xu Guo, Jianbing Dong, et al. 2022\.  Merlin hugectr: Gpu-accelerated recommender system training and inference. In _Proceedings of the 16th ACM Conference on Recommender Systems_. 534–537.
  * Wei et al. (2022) Yingcan Wei, Matthias Langer, Fan Yu, Minseok Lee, Jie Liu, Ji Shi, and Zehuan Wang. 2022.  A GPU-specialized Inference Parameter Server for Large-Scale Deep Recommendation Models. In _Sixteenth ACM Conference on Recommender Systems_. ACM.  https://doi.org/10.1145/3523227.3546765
  * Wu et al. (2021) Carole-Jean Wu, Ramya Raghavendra, Udit Gupta, Bilge Acun, Newsha Ardalani, Kiwan Maeng, Gloria Chang, Fiona Aga Behram, James Huang, Charles Bai, Michael Gschwind, Anurag Gupta, Myle Ott, Anastasia Melnikov, Salvatore Candido, David Brooks, Geeta Chauhan, Benjamin Lee, Hsien-Hsin S. Lee, Bugra Akyildiz, Maximilian Balandat, Joe Spisak, Ravi Jain, Mike Rabbat, and Kim Hazelwood. 2021.  Sustainable AI: Environmental Implications, Challenges and Opportunities.  https://doi.org/10.48550/ARXIV.2111.00364
  * Xiao et al. (2018) Wencong Xiao, Romil Bhardwaj, Ramachandran Ramjee, Muthian Sivathanu, Nipun Kwatra, Zhenhua Han, Pratyush Patel, Xuan Peng, Hanyu Zhao, Quanlu Zhang, et al. 2018\.  Gandiva: Introspective cluster scheduling for deep learning. In _13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18)_. 595–610.
  * Xiao et al. (2020) Wencong Xiao, Shiru Ren, Yong Li, Yang Zhang, Pengyang Hou, Zhi Li, Yihui Feng, Wei Lin, and Yangqing Jia. 2020.  $\\{$AntMan$\\}$: Dynamic Scaling on $\\{$GPU$\\}$ Clusters for Deep Learning. In _14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20)_. 533–548.
  * Zhao et al. (2022) Mark Zhao, Niket Agarwal, Aarti Basant, Buğra Gedik, Satadru Pan, Mustafa Ozdal, Rakesh Komuravelli, Jerry Pan, Tianshu Bao, Haowei Lu, Sundaram Narayanan, Jack Langman, Kevin Wilfong, Harsha Rastogi, Carole-Jean Wu, Christos Kozyrakis, and Parik Pol. 2022.  Understanding data storage and ingestion for large-scale deep recommendation model training. In _Proceedings of the 49th Annual International Symposium on Computer Architecture_. ACM.  https://doi.org/10.1145/3470496.3533044
  * Zhong et al. (2017) Victor Zhong, Caiming Xiong, and Richard Socher. 2017\.  Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning.  https://doi.org/10.48550/ARXIV.1709.00103