#  Rinas: Training with Dataset Shuffling Can Be General and Fast

Tianle Zhong<EMAIL_ADDRESS>University of Virginia ,
Jiechen Zhao<EMAIL_ADDRESS>University of Toronto ,  Xindi Guo
<EMAIL_ADDRESS>University of Virginia ,  Qiang Su∗
<EMAIL_ADDRESS>City University of Hong Kong and  Geoffrey Fox∗
<EMAIL_ADDRESS>University of Virginia

###### Abstract.

Deep learning datasets are expanding at an unprecedented pace, creating new
challenges for data processing in model training pipelines. A crucial aspect
of these pipelines is dataset shuffling, which significantly improves unbiased
learning and convergence accuracy by adhering to the principles of random
sampling. However, loading shuffled data for large datasets incurs significant
overhead in the deep learning pipeline and severely impacts the end-to-end
training throughput. To mitigate this, current deep learning systems often
resort to partial dataset shuffling, sacrificing global randomness to maintain
acceptable training throughput on large datasets, still leaving global
shuffling efficiency issues not fully explored.

In this work, we present Rinas, a data loading framework that systematically
addresses the performance bottleneck of loading global shuffled datasets. Our
key contribution is to offer an intra-batch unordered data fetching approach,
which unleashes unexplored parallelism of data loading. We implement Rinas
under the PyTorch framework for common dataset libraries HuggingFace and
TorchVision. Our experimental results show that Rinas improves the throughput
of general language model training and vision model training by up to 59% and
89%, respectively.

∗Qiang Su and Geoffrey Fox are corresponding authors. The Univesity of
Virginia team thanks NSF Grant 2210266 and DOE Grant DE-SC0023452 for partial
support. We acknowledge the excellent work of the Rivanna HPC Cluster team.

##  1\. Introduction

Shuffling large datasets remains a critical issue in data processing systems,
impacting a wide spectrum of applications (dean2008mapreduce, ;
10.1145/2934664, ; nicolae2016towards, ; shen2020magnet, ; hadoop, ). In the
realm of deep learning, dataset shuffling is not merely a procedural task but
a fundamental aspect of the data loading pipeline. It is instrumental in
avoiding overfitting better (ying2019overview, ; li2019research, ) and
benefiting convergence accuracy based on the theoretical foundation
(meng2019convergence, ).

Ideally, the training accuracy improvement derived from data shuffling should
introduce minimal shuffling overheads on overall training throughput.
Unfortunately, as the sizes of deep learning datasets expand rapidly, the
shuffling overhead is significantly growing. This is because the datasets
nowadays are towards tens of TBs, largely exceeding the system DRAM capacity
and thus leading to the use of slower disk I/O becoming an inevitable
bottleneck (lobster, ; sun2022solar, ; gu2022fluid, ). Worse, it is
challenging for current systems to effectively manage the shuffling overhead
on training throughput without sacrificing accuracy (nguyen2022globally, ;
exoshuffle, ).

The impact of large dataset shuffling on training throughput is substantial
and far from optimal. For example, we observe that data loading I/O for
shuffling can consume up to 85% of the total training time for models such as
ResNet-152 on the ImageNet dataset ($\sim$140 GB). This phenomenon
corroborates with the prior art (10.1145/3458817.3476181, ). For language
model training, the slowdown caused by shuffled loading can also lead to 30%
to 50% of training throughput degradation. This overhead persists as a
dominant bottleneck in training efficiency, even when deep learning pipelines
are integrated with advanced data processing systems and databases (deeplake,
).

In the practice of typical deep learning systems, shuffling for larger-than-
memory datasets is usually abstracted by a shuffled loading operation,
fetching a batch of data samples randomly from the dataset in a on-demand
fashion, which avoids pre-loading the entire dataset into DRAM for shuffling.
To fetch the data from the disk to the system DRAM, each data sample needs to
be indexed inside the dataset which enforces the data loading to perform
random disk I/O operations (Index, ; hf_map_vs_iter, ).

Amidst this backdrop, contemporary research has made strides in two
directions: (1) refining the shuffled dataset loading pipeline to conceal its
impact on end-to-end training throughput; and (2) identifying an equilibrium
between shuffling thoroughness and converged accuracy. However, these
advancements come with concessions: (1) hiding data loading overhead is
ineffective when it becomes the predominant factor in training performance;
(2) the complexity of the data loading pipeline demands extensive
modifications to existing software infrastructure; (3) the equilibrium point
is not universally applicable across the spectrum of datasets and learning
models; (4) the scope of trade-offs is constrained by system capabilities,
leaving practitioners to contend with a compromise that sacrifices either
accuracy or speed, since a not thorough shuffle harnesses the convergence
accuracy (xu2022stochastic, ).

The fundamental question is, can we devise a universal framework that
accelerates the loading of large, shuffled datasets without compromising
accuracy and training speed across diverse datasets and learning models?

To address this problem, we introduce Rinas, a comprehensive data loading
framework designed for efficient model training on large shuffled datasets. We
pinpoint the frequent random disk I/O on individual data samples as the
principal bottleneck in loading shuffled datasets. Under this observation, we
propose a novel data preparation method for model training processes: intra-
batch unordered data fetching.

Rinas is predicated on a key insight: within a training iteration, the
sequence of computing the average loss from a batch of randomly sampled data
does not influence the learning outcome. This insight suggests that the data
retrieval order for intra-batch samples does not affect the learning process.
Fundamentally, it allows us to shift from a strict global sample order to a
more flexible intra-batch unordered manner.

This insight brings in several benefits. First, relaxing such orders enables
parallel data retrieval without stringent ordering constraints. Second, Rinas
fundamentally negates the need to balance between accuracy and speed. Third,
Rinas is versatile enough to cater to various datasets and learning models
because of the guarantee of equivalent learning outcomes.

We architect Rinas with a data-agnostic control plane and a data plane for on-
demand and parallelized data fetching. The control plane offers an execution
model for loading samples and generating batches with them in an unordered
fashion. The data plane fully exploits the advantages of unordered parallel
data retrieval across diverse dataset structures.

Our implementation showcases Rinas’s applicability across major learning tasks
from computer vision to language models. We demonstrate our prototype within
the PyTorch DataLoader (Index, ), as well as TorchVision (torchvision2016, )
and HuggingFace Datasets libraries (lhoest-etal-2021-datasets, ). Our
experiments evaluate standard training workloads on large datasets, spanning
from computer vision model training to language model pretraining, within
typical deep learning training clusters. Our assessments cover the typical
training setup, revealing Rinas’s minimal loading-related overheads at
different scales, delivering up to 59% and 89% speed increases in training for
computer vision and language models, respectively.

The ensuing sections will delve into the background (§2) and motivation (§3),
review related work and its limitations, and then articulate the design (§4)
and implementation (§5) of Rinas. Finally, we will present our evaluation
findings (§6) and engage in a discussion (§7).

##  2\. Background

Dataset | Size | Description
---|---|---
ImageNet (deng2009imagenet, ) | 140 GB | For image classification
ImageNet-21k (deng2009imagenet, ) | 1.8 TB | Extended version of ImageNet
RedPajama (together2023redpajama, ) | 5 TB | Language modeling
C4 (2019t5, ) | 7 TB
CosmoFlow (mathuriya2018cosmoflow, ) | 10 TB | Cosmological simulations

Table 1. Large datasets for model training in computer vision and language
modeling.

###  2.1. Storing Emerging Very Large Datasets on Disks

ML model training has witnessed increasingly huge datasets that are beyond the
system’s memory capacity (DRAM), and it becomes impractical to preload the
entire dataset before training. Therefore, data preparation emerges as a
critical path for model training: loading large datasets into memory is often
non-trivial. Table 1 presents typical datasets for computer vision models and
large language models. The computer vision dataset collects a huge amount of
image files, and accessing these images typically requires indexing into a
structured file system; Large language models (devlin2019bert, ;
thoppilan2022lamda, ; touvron2023llama, ) are trained on vast text corpora,
with huge data volumes and complicated structures (together2023redpajama, ;
2019t5, ; OpenOrca, ). These datasets are often segmented into multiple files,
each packed with text entries. Retrieving specific samples requires effective
parsing or database systems with sophisticated indexing for efficient search
and access (lhoest-etal-2021-datasets, ; torchvision2016, ). Specifically,
image datasets are usually composed of individual image files on disk as data
samples, while text datasets are usually composed of a series of large files
on disk, each containing rows of text as data samples.

###  2.2. Dataset Shuffling in Model Training Pipeline

Figure 1. The logical data path of shuffling in end-to-end training.

Figure 1 presents the typical training pipeline upon the training datasets,
where the random sampling (olken1995random, ) on the datasets serves as a
critical operation. The dataset on the storage is randomly sampled into DRAM
space, then the sampled data are partitioned into batches to be fed into model
training iterations.

In practice, the operations to achieve random sampling upon the dataset are
generally described as dataset shuffling (liu2017shuffle_spark, ). By
shuffling the dataset at the beginning of each training epoch, the data sample
order to be exposed to the model training process is randomized. Typically,
there are two ways of dataset shuffling: shuffling the dataset in the storage
space (i.e., in-place shuffling), or loading the dataset into DRAM space with
a random order (i.e., shuffled loading).

In-place shuffling is a common operation in large-scale batch processing
systems (dean2008mapreduce, ; hadoop, ; 10.1145/2934664, ). However, unlike
smaller in-memory collections, shuffling large datasets that reside on
persistent storage devices imposes a considerable overhead (iShuffle, ;
liu2017shuffle_spark, ), and involves more than mere in-place reordering. It
requires careful consideration of the I/O throughput, storage latency, and
computational load on the system (welton2011improving, ; zhang2006storage, ;
deeplake, ; gupta2015amazon, ).

Unfortunately, systems highly optimized for in-place shuffling like Hadoop
(hadoop, ) and Spark (spark, ) prevent reading shuffled results until the
shuffle is fully completed, making model training systems hard to pipeline the
shuffling process with model training. On the other hand, shuffled loading
avoids expensive and complicated in-place reordering of the datasets in the
storage space, able to be pipelined with the model training process. As a
result, shuffled loading is our paper’s focus and many other papers’ focus is
how to pipeline the in-place shuffling with model training (ray, ; exoshuffle,
). Typically, there are two ways of shuffled loading: Buffered shuffling and
indices mapping. They are both popular choices and already supported by many
frameworks like PyTorch (Index, ; ofeidis2022overview, ).

Buffered shuffling. Figure 2 presents an example workflow of buffered
shuffling, which involves two steps to improve the efficiency of shuffled
loading. First, it leverages partial shuffling, sequentially loading a subset
of the dataset from the disk into a memory buffer allocated in the system
DRAM. Second, it performs the shuffle operation in this constrained space,
followed by the formation of batches from this shuffled subset.

This approach strikes a balance between the need for random access and the
performance limitations imposed by disk-based storage, aiming to provide a
partially shuffled dataset without the prohibitive overhead of random disk I/O
(DeepIO_buffer_shuffle, ). However, buffered shuffling cannot achieve true
random sampling due to the limited shuffling space compared to global
shuffling, which may inadvertently compromise convergence accuracy
(nguyen2022globally, ; xu2022stochastic, ).

Indices mapping. To avoid the loss of convergence accuracy due to the
compromised shuffle quality by partial shuffling, the dataset should be
globally shuffled (meng2019convergence, ). Figure 3 depicts an example of
indices mapping workflow. In this strategy, the dataset indices are shuffled
rather than data, and the data is read into DRAM following the order of
shuffled indices.

This approach successfully performs a shuffled loading strategy that fully
respects the random sampling principles but creates a sequence of random IO
that the storage system is typically less efficient at.

Since indices mapping guarantees true random sampling to benefit model
training convergence accuracy at most, our work focuses on addressing the
performance issue of indices mapping.

Figure 2. An example of buffered shuffle workflow. Figure 3. An example of
indices mapping workflow.

##  3\. Motivation

In this section, we begin with analyzing the current problems that indices
mapping incurs. Next, we discuss existing solutions and their limitations.

###  3.1. Inefficiency with Indices Mapping for Very Large Datasets

As the previous section discussed, although indices mapping reserves the
global randomness of the dataset, this approach is typically much slower than
buffered shuffling due to the necessity of non-contiguous data retrieval from
storage. Next, we explain the relationship between such a slowdown and the
dataset size.

Training slowdown. Figure 4 presents the end-to-end training throughput of
training RoBERTa-base model (liu2019roberta, ) at different batch sizes when
dataset size increases on a single NVIDIA A100 GPU. While the cleaned English
branch of C4 dataset tokenized by RoBERTa has $\sim 3.6\times 10^{8}$ rows, we
synthesize four different sizes of its subsets by choosing its first $10^{5}$,
$10^{6}$, $10^{7}$, and $10^{8}$ rows. We benchmark the training throughput
with all five sizes of datasets to show the throughput changes when dataset
size increases. Observe that there is a marked reduction in training
efficiency when the dataset size increases, leading to a 30% to 50% decrease
in speed.

This reveals that the primary cause of the training deceleration is the
overhead associated with shuffling by indices mapping against the large
datasets, as opposed to a scenario without shuffling. Notably, the negative
impact of this overhead is observed across both large and small batch sizes,
underscoring the pervasive influence of shuffling-related delays.

This indicates that in extensive large-scale scenarios, I/O overhead by
indices mapping can become the major factor of the total training time,
leading to substantial GPU idle time and data starvation. This phenomenon has
been corroborated by other research on other training scenarios and systems as
well (10.1145/3458817.3476181, ; sun2022solar, ).

This also indicates why shuffled data loading overhead for the large dataset
cannot be easily hidden by overlapping with computation: under this large
extent of degradation, the data loading overhead has dominated the overall
training time.

Figure 4. Training throughput for the RoBERTa-base model for varying batch
sizes within the C4 subsets of different sizes. A discernible trend emerges,
showing a decrement in end-to-end training throughput as the number of samples
in the dataset increases.  Figure 5. Data loading throughput comparison for
varying batch sizes within the C4 subsets of different sizes.

Throughput degradation. The reason for the training slowdown is shuffled data
loading throughput degradation with large datasets. Figure 5 presents the pure
data loading throughput under the same settings as Figure 6 by excluding the
model training process and only performing data loading operations. By further
looking into the measured data loading throughput, a parallel decrease in data
loading throughput, akin to the trend seen in end-to-end training performance,
underscores the impact of data loading efficiency on overall training time.

When employing a batch size of 32, data loading throughput for small datasets
using indices mapping can capitalize on system I/O capabilities, achieving a
data loading throughput of $\sim$1000 samples per second. However, this
throughput significantly diminishes when applied to larger datasets with
indices mapping, with performance dropping to $\sim$50 samples per second,
primarily due to the random dataset indexing overhead which is positive
correlated with the dataset size.

This relationship indicates that throughput degradation during data loading is
a primary contributor to the observed training slowdown. This observation
underscores the prominence of data loading as a substantial bottleneck in the
training process, particularly when dealing with extensive datasets. This
phenomenon necessitates a reevaluation and optimization of data loading
practices to alleviate the undue time expenditure and enhance overall training
performance.

###  3.2. Existing Solutions

Prior work proposes solutions to mitigating the performance issue with large
shuffled datasets. There are three categories of solutions: balanced trade-off
between shuffle quality and speed, scheduled data prefetching, and taking
advantage of the data-parallel training.

Shuffle balancing. Considering the inefficiency of indices mapping, various
research endeavors have experimented with adjusting the degree of shuffling to
strike a balance between converged accuracy and training efficiency
(nguyen2022globally, ; sun2022solar, ). Nevertheless, the process of
identifying this equilibrium can be both time-consuming and highly dependent
on the specifics of the model and dataset, limiting its applicability to a
wider range of models and datasets. Furthermore, the trade-off space is
usually limited by hardware resources which can leave practitioners to contend
with a compromise that sacrifices either accuracy or speed (exoshuffle, ). For
example, for training the ResNet-50 model on the ImageNet dataset, the limited
shuffle can result in $\sim$20% of accuracy drop compared to global shuffled
learning (nguyen2022globally, ).

Data prefetch scheduling. Given that the sequence of shuffled indices is
predetermined, concurrently with the model’s computation on the current batch,
the read operation for the next batch can be initiated, allowing data
preloading to take place in an overlapped fashion, thereby enhancing
performance. There have been efforts such as NoPFS (10.1145/3458817.3476181, )
and ExoShuffle (exoshuffle, ) which integrate data loading pipeline with data
prefetching scheduling and attempt to overlap data loading with model training
computations. Those solutions are very effective when the shuffled loading
latency is at a relatively low level such as when employing buffered shuffling
or dealing with smaller datasets. However, as discussed in §3.1, when data
loading significantly overshadows training time, the benefits of such
overlapping become marginal and less effective. Moreover, such data prefetch
scheduling usually needs extensive modification on the underlying software
infrastructure like the file system (i.e, NoPFS) and data processing system
(i.e, ExoShuffle).

Figure 6. Illustration of distributed training throughput for the RoBERTa-base
model on various subsets of the C4 dataset, differentiated by batch size. 4
NVIDIA A100 (80 GB).

Data-parallel training. By replicating learners across multiple GPUs, each
learner has its own data loading process thus achieving parallel dataset
indexing in the global view (parameter_server, ; li2020pytorch_distributed, ).
To explore the effectiveness of data-parallel training with large shuffled
datasets, we conduct the distributed version of the end-to-end training
throughput experiment, shown as Figure 6. We can see that although data-
parallel training can improve the global end-to-end training throughput
effectively compared to non-distributed training, data-parallel training still
suffers from training throughput degradation when dataset size increases. This
is because each learner’s data loading procedure is still degraded at larger
datasets and hence becomes the bottleneck of its training process, resulting
the end-to-end throughput degradation like non-distributed training. Moreover,
utilizing multiple GPUs or hardware resources can be costly, both in terms of
initial investment and operational expenses. Also, distributing data and
aggregating results across GPUs introduce a communication overhead. This can
sometimes offset the training throughput benefits of parallel processing
(nguyen2022globally, ; dp_communication, ).

##  4\.  Rinas Design

In this section, we first show the design principle of Rinas as a new paradigm
(4.1). Then, we introduce the design goals of this work (4.2), followed by an
analysis of rethinking the randomness in the general learning process (4.3).
Next, we describe the novel execution model under unordered batch generation
(4.4) and its requirements on dataset representation and indexing interface
(4.5). Finally, this section demonstrates the end-to-end system overview of
Rinas (4.6).

###  4.1. Towards Overcoming Throughput Degradation: A Paradigm Shift

Our approach aims at addressing the critical challenge of loading large
shuffled datasets and the resultant throughput degradation inherent in indices
mapping. By leveraging the deep learning-specific domain knowledge, our
approach unleashes unexplored parallelism in the data loading process.

By accelerating the data loading with indices mapping, one of the key benefits
is simplicity; programmers don’t need to bother to solve the above issues in
existing solutions described in 3.2. Practitioners can confidently apply
global shuffling to their model training process to maximize converged
accuracy without the concern of training slowdown caused by indices mapping.
This key distinction sets our work apart from existing solutions, focusing on
a fundamental shift in how intra-batch data is loaded for batch generation
during the training process.

Moreover, since our method is under the framework of indices mapping, the
integration with existing model training environments is straightforward.

In essence, Rinas proposes a paradigm shift in how data is prepared and
managed at batch generation stage to achieve dataset shuffling. Rinas paves
the way for a more easy-to-manage and efficient deep learning process with
high accuracy. Next, we discuss Rinas’s design goals in detail.

###  4.2. Design Goals

Our design objectives are twofold:

  * •

Performance at scale. Mitigate the data loading bottleneck of indices mapping:
the extensive disk random I/Os due to frequent, non-contiguous data indexing,
which can significantly hamper performance. This is an even more challenging
goal for TB-scale datasets since the random dataset indexing overhead is
positively correlated with dataset size (recall 3.1).

  * •

Agnostic to the learning process. Rinas has to reserve and guarantee the
global randomness of shuffling for model training process, which can make sure
that our approach is not specific to datasets or models.

The first goal addresses the performance bottleneck challenge we aim to
overcome, while the second goal serves as a constraint to ensure that our
solution remains versatile, and capable of being seamlessly incorporated into
existing deep learning systems for a variety of tasks. To achieve these goals,
Rinas leverages the deep learning-domain knowledge obtained by rethinking the
randomness in the general learning process.

###  4.3. Rethinking Intra-Batch Sample Randomness

A conventional principle in data loading is to adhere strictly to the order of
shuffled datasets. This practice is commonplace in data processing and
analysis, ensuring complete randomness in the data presented to the model. In
model training, the concept of batch size is introduced to specify the number
of samples processed in a single iteration of model training.

The update of the model parameters in one training step can be expressed as:

(1)
$\theta_{\text{new}}=\theta_{\text{old}}-\eta\cdot\nabla_{\theta}\mathcal{L}\left(\frac{1}{N}\sum_{i=1}^{N}\ell(x_{i},y_{i};\theta_{\text{old}})\right)$

In this equation, $\theta$ represents the model parameters, $\eta$ is the
learning rate, $\nabla_{\theta}\mathcal{L}$ denotes the gradient of the loss
function $\mathcal{L}$ with respect to the parameters, and $\ell$ is the per-
sample loss function. $x_{i}$ and $y_{i}$ are the input and target of the
$i$-th sample in the batch, and $N$ is the batch size.

We can see that the loss is computed as the average of individual sample
losses, suggesting that the order of samples within the same batch does not
influence the outcome. This leads us to a deep learning-specific insight: the
intra-batch sample order does not impact the learning outcome, opening
potential avenues for optimization in data loading and training efficiency.

To be specific, this insight releases the design of the data loading pipeline
from strictly respecting sample order randomness to an intra-batch sample
unordered manner. This enables additional parallelism space of intra-batch
data indexing. By out-of-order retrieval, we can parallelize the intra-batch
data indexing without worrying about the order of data arrival.

###  4.4. Unordered Batch Generation

Based on the previous observation on model training procedure, we can
conceptualize unordered batch generation, a versatile data loading pipeline
that permits batch generation with unordered intra-batch sample retrieval.

Figure 7. A comparative illustration of execution models between conventional
and Rinas’s unordered batch generation.

Figure 7 presents the execution model of both conventional method and
unordered batch generation. There are two major parts of unordered batch
generation: parallel dataset indexing and overlapped preprocessing.

Parallel dataset indexing. By eliminating the necessity to maintain intra-
batch sample order, data retrieval operations for distinct samples within a
batch can be executed in parallel through asynchronous threading. The data
arrival order is changed with such execution but would not affect the learning
outcome (recall 4.3).

Overlapped preprocessing. After data is fetched from the disk, it usually
needs to go through the user-defined preprocessing pipeline before being
passed into the model training process. Taking preprocessing as a part of the
batch generation process, we can safely overlap data preprocessing tasks with
data retrieval, thereby achieving additional performance enhancements.

###  4.5. Dataset Representation and Indexing Interface

The proposed execution model based on unordered batch generation comes with
three requirements on dataset representation and indexing interface.

Indexable dataset representation. Since Rinas is under the framework of
indices mapping, the dataset representation should be indexable, which
distinguishes Rinas from the non-indexable iterative-style datasets
representation like PyTorch iterative datasets and Ray Data (moritz2018ray, ).
This enables the on-demand indexing to retrieve an arbitrary sample of the
dataset at any time.

Interference-free retrieval. The unordered batch generation necessitates the
parallel execution of dataset indexing, which requires the data retrieval
process to be interference-free with each other. If the dataset indexing
procedure prevents another dataset indexing procedure from running
concurrently, it would fully force the retrieval process back to the one-by-
one manner, eliminating the benefits of unordered batch generation.

Reusing existing facility. Considering that there are already many dataset
abstractions satisfactory for the above requirements like map-style image
datasets in TorchVision, Rinas can directly employ them seamlessly. For
unsatisfactory datasets, we provide a case study in 5 to demonstrate how to
convert them into the needed fashion.

###  4.6. End-to-end View of Rinas

We now describe the end-to-end view of our approach. Figure 8 provides a
system overview of Rinas. The dataset representation against the dataset
storage provides a dataset indexing interface for data retrieval which serves
as a data plane. Rinas’s unordered batch generation serves as a control plane
to parallelize the given data retrieval for intra-batch data.

Figure 8. A system overview of Rinas.

Dataset initialization stage. Data representation needs to contain the mapping
from indices to actual data locations. Such information can be stored
externally and read at the initialization stage or created at runtime,
depending on the storage method of the datasets. The dataset representation
also exposes a dataset indexing interface as the data plane to be leveraged by
the control plane which is the unordered batch generation module.

Batch generation stage. The unordered batch generation module executes
parallel dataset indexing for data within the same batch and pipelines the
preprocessing and IO of different samples. Finally, a batch is generated and
fed into the model training process.

Scope of Rinas. Unlike other existing solutions (ray, ; exoshuffle, ), Rinas
is not designed for general dataset shuffling applications beyond model
training due to the fact that Rinas relies on the deep learning-specific
insight (recall 4.3). Any application that requires the intra-batch sample
order to be aligned with the shuffled indices is out of Rinas’s scope.

##  5\. Implementation

We implement a prototype of Rinas by extending the PyTorch framework and
HuggingFace Datasets library with $\sim$400 lines of Python code. Our
prototype involves an unordered batch generation control plane and an on-
demand, parallelizable data plane, each working as a standalone module.

Specifically, we override the _MapDatasetFetcher class to enforce the
unordered batch generation (4.4), and an asynchronous thread pool is created
to fetch data samples in parallel. Note that the index order is changed
according to the intra-batch fetch scheduling. Once the data sample is
fetched, it is immediately sent to the user-defined preprocessing pipeline,
and different data samples are processed in parallel.

###  5.1. A Case Study: Converting HuggingFace Datasets

We provide a case study for how to convert the dataset representation
unsatisfactory to our control plane in the needed fashion. The example case
here is the HuggingFace datasets, which are based on Apache Arrow format
(pyarrow, ), a columnar memory format for flat and hierarchical data,
organized for efficient analytic operations on modern hardware like CPUs and
GPUs.

Dataset storage. Specifically, HuggingFace stores datasets as memory-mapped
arrow stream files for efficient stream processing (arrow_flight, ). Arrow
stream format partitions the whole dataset into small data chunks on the
storage and enables users to efficiently iterate through the dataset at the
unit of data chunks. However, the arrow stream file format lacks data chunk
indices which is necessary for index-based loading. To address this,
HuggingFace chooses to iterate through the entire dataset at dataset
initialization to create a table that contains all the metadata and locations
of data chunks on the storage. Unfortunately, this method leads to two major
drawbacks:

  1. (1)

Long dataset initialization time: the iterating procedure at dataset
initialization has a time cost that is linear to the dataset size. For a
dataset that is around 1 TB, it can take up to 10 minutes to finish the
dataset initialization.

  2. (2)

Frequent page swaps during shuffled loading. The dataset initialization
process creates a memory-mapped arrow table instance in DRAM, hiding data
chunk access details from dataset developers. The data chunk accessing and
mapping is fully managed by the operating systems. Due to the limited DRAM
space compared to the dataset size, most data chunks are paged out during
dataset initialization and need to be re-mapped into DRAM at runtime, which
causes frequent page swaps and blocks the parallel access to data chunks.

Format conversion. Therefore, we need to optimize the HuggingFace
implementation to support the on-demand and parallelized data indexing.
Specifically, we convert the files from the arrow stream format to an arrow
indexable format, utilizing the PyArrow library (pyarrow, ).

Figure 9 presents the difference between accessing data chunks in arrow stream
file format and arrow indexable file format. The arrow stream file format
opens a message stream and sends data chunks in a sequence of messages. The
first message is the data schema, which contains the metadata for data chunks
in the file. The stream reader iterates through data chunks by read_next()
method without knowing the locations of data chunks. On the other hand, when
opening the file, the arrow indexable file format in our prototype reads the
data schema containing metadata, and file layout containing data chunk
locations. With file layout loaded at the beginning of file reading, the
indexable format allows accessing data chunks by its index in the file with
get_batch(index).

With this format, the data chunks can be accessed without page swaps or memory
re-mapping since the data chunk access and mapping becomes on-demand and
manageable, facilitating parallel reading of data chunks and aligning with the
efficient and scalable batch generation execution model in Rinas (recall 4.4).

Figure 9. A comparative illustration of arrow stream and indexable formats.

How to implement Rinas over other deep learning frameworks? In addition to
PyTorch, Rinas can be implemented on various deep learning frameworks
(abadi2016tensorflow, ; jax2018github, ; paszke2019pytorch, ). Specifically,
We provide some tips to enforce Rinas on TensorFlow and JAX.

TensorFlow (abadi2016tensorflow, ) utilizes the tf.data module for the data
input pipeline, but it does not support indices mapping. Therefore, to
implement Rinas, first, we need to modify the Dataset.batch function to create
the batched shuffled indices. After the shuffled indices are available for
TensorFlow datasets, we can force the tf.data.Iterator to iterate through the
dataset based on the shuffled indices. When the iterator iterates at the level
of batches, the intra-batch data can be fetched in parallel to achieve Rinas’s
unordered batch generation (4.4).

Since JAX can directly leverage PyTorch DataLoader for the data input
pipeline, our implementation can be naturally supported by adapting our
modifications on PyTorch DataLoader.

##  6\. Evaluation

We now present the evaluation of Rinas by answering the following questions.

  1. (1)

What’s the improvements Rinas provides on the end-to-end model training
throughput? (6.1)

  2. (2)

How do the control and data plane contribute to Rinas’s performance in various
scenarios? (6.2)

  3. (3)

How does the global randomness brought by Rinas benefit the learning outcome?
(6.3)

  4. (4)

What kinds of overhead Rinas introduces and how much is it? (6.4)

Testbed. We set up a testbed on a standard computing node equipped with an AMD
EPYC 7742 CPU with 96 GB RAM and 4 NVIDIA A100 GPUs. Each GPU has 80 GB
memory. In addition, we rely on the cluster-wide WEKA file system for dataset
storage (weka, ), which is commonly adopted in real-world and supported by
high-demand computing platforms like Amazon AWS and Google Cloud. For software
configurations, we utilize CUDA 11.7, PyTorch 2.0.1, HuggingFace Datasets
2.14.6.dev0, and Transfomers 4.34.0.dev0.

Methodology. We evaluate Rinas using two datasets: C4 and ImageNet, which are
typical text and image datasets for large language models and computer vision
models. Correspondingly, we train a typical large language model RoBERTa and a
computer vision model ResNet-152 to demonstrate Rinas’s benefits. To evaluate
the end-to-end training throughput, we conduct a series of model training
iterations, spanning 300 steps. The throughput is calculated by dividing the
total number of samples processed during these iterations by the total time
taken to complete them. The results are averaged over 3 runs. To avoid startup
interference, we run enough warm-up rounds before collecting the results.

Large language model. We train a RoBERTa-base model based on the C4 dataset.
RoBERTa (Robustly Optimized BERT Approach) is an advanced variant of the BERT
model, and the cleaned English branch of C4 dataset has $\sim$305 GB raw text
data and $\sim$1.1 TB after being tokenized by RoBERTa. We compare Rinas
against HuggingFace, which is the default approach to store datasets and
execute language model training (liu2019roberta, ; devlin2019bert, ;
touvron2023llama, ).

Computer vision model. We train the ResNet-152 model using the ImageNet
dataset that has $\sim$140 GB raw image data. The baseline is the training
pipeline that employs the official PyTorch DataLoader and the ImageNet dataset
implementation by TorchVision (torchvision2016, ), which is widely used for
computer vision models (simonyan2015deep, ; he2016deep, ;
dosovitskiy2021image, ).

Additionally, we also test the baselines when supercharged by Ray for both
text and image model training. Notice that we could not convert the used
datasets with Ray Data (ExoShuffle) due to the extensive DRAM usage for such
transformation (6.4). The only option is to use Ray Train’s wrappers for
HuggingFace and PyTorch DataLoaders.

###  6.1. End-to-end Training Throughput

RoBERTa-base training upon C4. Figure 10 presents the end-to-end distributed
training throughput of the RoBERTa-base model when the batch size increases.
We can see a general trend where Rinas can achieve larger throughput at larger
batch sizes; this is due to the higher parallelism of Rinas’s unordered batch
generation at larger batch sizes.

To further analyze the performance gain provided by Rinas, we measure the
training throughput of employing original HuggingFace pipeline as a baseline
and show the speedup of Rinas at different batch sizes as the Figure 11 shows.
As Figure 11 shows, Rinas can improve the end-to-end training throughput by
1.54-1.59$\times$, demonstrating the efficiency of Rinas at a wide range of
batch sizes. Rinas achieves higher speedup ratios at larger batch sizes,
suggesting Rinas’s better scalability than HuggingFace in terms of batch
sizes.

In addition, when the HuggingFace baseline is supercharged by Ray, the
training throughput has a slight drop. This is because Ray only takes the data
loading process as its actor’s application workload and does not optimize the
inner pipeline. With additional overhead from Ray’s processes, it is
reasonable to see a slight performance drop.

Figure 10.  RoBERTa-base distributed training throughput comparison between
HuggingFace and Rinas.  Figure 11.  RoBERTa-base distributed training speedup
over HuggingFace at different batch sizes.  Figure 12. ResNet-152 distributed
training throughput comparison between using PyTorch dataloader and Rinas.
Figure 13.  ResNet-152 distributed training speedup at different batch sizes.

ResNet-152 training on ImageNet. Figure 12 presents the end-to-end distributed
training throughput of the ResNet-152 model when the batch size increases. We
can observe the same trend with RoBERTa training: Rinas can achieve higher
training throughput when batch size increases at larger batch sizes. For
comparison, training throughput using PyTorch DataLoader remains at a
relatively low level, suggesting the effectiveness of Rinas in image datasets
like ImageNet.

To further analyze the performance gain by Rinas, Figure 13 shows the speedup
ratios compared to the PyTorch DataLoader baseline at different batch sizes.
Similar to Figure 11, We can also see that Rinas can achieve higher speedup
ratios by up to 1.89$\times$ at larger batch sizes, demonstrating Rinas’s
scalability in the image dataset in terms of batch sizes as well.

We can also notice the slight performance drop when PyTorch DataLoader is
supercharged by Ray which is due to the same reason as explained in the
language model training experiment.

###  6.2. Breakdown Analysis

Figure 14. RoBERTa training throughput improvement breakdown under a batch
size of 32.

In order to analyze the benefits of Rinas’s control plane and data plane, we
analyze their contributions to the aforementioned experiments: On one hand, we
measure the RoBERTa-base end-to-end distributed training throughput at a batch
size of 32 by disabling the control plane to showcase the contribution from
solely data plane. Note that removing the control plane does not require any
modification on the side of the data plane for Rinas. Figure 14 displays the
results. We can see that Rinas’s data plane can improve RoBERTa-base
distributed training throughput by 49.4% when the batch size is 32. When the
control plane is further enforced, we can achieve a 58.9% cumulative
performance improvement, highlighting the great contribution from the Rinas
data plane. On the other hand, in the case of ResNet-152 training on the
ImageNet dataset, both Rinas and PyTorch DataLoader operate on the same
dataset implementation, differing only in their respective control planes
within the DataLoader. This highlights the importance of Rinas control plane
in image datasets like ImageNet. As for the speedup ratios difference between
text and image datasets, we would like to

Model | Dataset | Acc. w/ shuffle | Imps.
---|---|---|---
limited | global
TabNet | HIGGS (7.5 GB) |  $\sim$70% |  $\sim$76% |  1.09$\times$
ResNet-50 | DeepCAM (8.2 TB) |  $\sim$78% |  $\sim$83% |  1.06$\times$
ImageNet-21k (1.1 TB) |  $\sim$37% |  $\sim$45% |  1.22$\times$
ImageNet-1k (140 GB) |  $\sim$50% |  $\sim$70% |  1.40$\times$
ResNet-18 | criteo (1.3 TB) |  $\sim$30% |  $\sim$90% |  3.00$\times$
VGG-19 | criteo (1.3 TB) |  $\sim$20% |  $\sim$90% |  4.50$\times$
Table 2. Model training accuracy discrepancies between limited shuffle and
global shuffle.

###  6.3. Convergence Benefits

As Rinas guarantees the global shuffling against the datasets, global
shuffling benefits all gradient-decent-based trainers based on the theoretical
foundation (meng2019convergence, ).

Due to the extensive long time to train a model on large datasets until
convergence, Table 2 collects and presents the model accuracy discrepancies
discovered by other studies (nguyen2022globally, ; exoshuffle, ;
xu2022stochastic, ). The studied models contain TabNet (arik2020tabnet, ),
ResNet, and VGG and the studied datasets include HIGGS (misc_higgs_280, ),
ImageNet, DeepCAM (nguyen2023deepcam, ) and criteo (criteo, ). We can observe
that the accuracy discrepancies vary a lot depending on different models and
datasets; no matter what models and datasets the benefits of global shuffling
are always obvious. Besides vision models, in practice tabular data and models
such as language models and temporal series models are more sensitive to the
shuffle quality (tabular_shuffle, ; ray_global_shuffle, ; schluter-
varab-2018-data, ), further highlighting the necessities of global shuffling
in model training.

###  6.4. Resource Overhead

We consider Rinas’s CPU memory usage overhead at dataset preparation and both
CPU and GPU memory overhead at runtime.

Dataset preparation. For PyTorch Datasets, Rinas can directly optimize the
shuffled loading performance upon PyTorch Datasets. Thus, there is no
additional memory usage overhead for PyTorch Datasets. As for Rinas’s file
format transformation required for HuggingFace datasets, our transformation
process is based on the PyArrow stream processing which only requires less
than $\sim$100 MB DRAM capacity.

For comparison, Ray Data module provides a conversion method from PyTorch and
HuggingFace Datasets to Ray Datasets, which can be stored and shuffled in
Ray’s shared-memory object store. However, their implementations do not allow
us to finish such a transformation for either ImageNet or C4 on our testbed,
due to the out-of-memory errors caused by extensive memory usage of such a
transformation. This results from the fact that Ray needs to load the entire
dataset into its local object store before shuffling (exoshuffle, ), while
Rinas follows an indices mapping manner and only loads data into RAM on
demand.

Runtime. At runtime, another CPU memory usage overhead comes from the
additional thread pool employed by Rinas’s control plane implementation, which
relies on the multi-threaded asynchronous intra-batch data fetching.

Since each intra-batch data is mapped and fetched by a separate thread, the
number of threads needs to match the batch size, which requires a large number
of threads to create when the batch size increases. In the case of distributed
ResNet-152 training with a batch size of 256 on 4 GPUs described in Sec. 6.1,
the entire system needs to create a thread pool of 256 threads on each
learner’s process, cumulatively 1024 threads at a time in total.

This can potentially stress the CPU and the storage system while other data
loaders without such design typically only require a single thread for each
data loader process. However, considering that the modern training clusters
are typically equipped with many-core CPUs and usually not fully exploited
during model training, Rinas’s additional stress on the CPU can be handled.

Also, since the parallelism degree of Rinas’s unordered batch generation is
positively correlated with batch size, a larger batch size per GPU is
necessary to have a significant speedup which leads to larger CPU and GPU
memory consumption. However, based on the theoretical foundation
(gao2020study, ) and the results of practice (KANDEL2020312, ) , a large batch
size benefits model’s generalization ability. Due to this reason, it has been
a common practice to pursue larger batch sizes for model training with various
memory optimization techniques (korthikanti2022reducing, ; zhao2023fsdp, ;
chen2016training, ).

##  7\. Discussion

This section discusses some immediate concerns one may have about Rinas.

Does Rinas provide high-level dataset representations? Rinas introduces a new
tradeoff between the performance benefits and the dataset representations in
its implementation. Ideally, a unified dataset representation can facilitate
the programming for datasets, such as the memory-mapped Arrow table in
HuggingFace, with which developers don’t need to know where the dataset is
located. However, to gain Rinas’s performance benefits (recall 6), the dataset
developers should manipulate the data retrieval process, majorly involving
locating the data segments on the storage (recall 4.5 and 5 ).

Is Rinas able to be extended to other distributed computing systems? Rinas
introduces a novel shuffled loading paradigm for batch generation models. The
key operation is the unordered batch generation, which mainly relies on
asynchronous random IO to parallelize the dataset indexing (recall 4.4).
Therefore, it is feasible to extend Rinas by enforcing unordered batch
generation by integrating the asynchronous random IO into the dataset
iterators supported by a variety of data processing systems, such as Ray,
Spark, and Twister (ray, ; 10.1145/1851476.1851593, ; spark, ). However, since
Rinas is designed with deep learning-specific domain knowledge (recall 4.4),
the use of Rinas needs the consideration of whether their applications are
also applicable.

How should Rinas’s performance to be expected on even larger datasets? In
general, the size of the dataset to be used for training should be aligned
with the model size. To investigate Rinas’s real benefits with larger dataset
sizes, we also need to adjust the model we train. It would be very interesting
to see how it performs on larger datasets to train a truly large foundation
model, but it is beyond our capacity in computational resource. Investigating
such problem can be one of our future directions.

##  8\. Related Work

Shuffling in data processing systems.  Since MapReduce (dean2008mapreduce, )
and Hadoop (hadoop, ), there are plenty of solutions with a focus on
optimizing disk I/O efficiency and pipelining for in-place shuffle operation
in data processing systems (iShuffle, ; hadoop_shuffle, ; ownership, ).
However, under model training workloads, prior approaches are usually
computationally heavy, and hard to be pipelined (recall 2).

To address this issue, Ray (ray, ) and ExoShuffle (exoshuffle, ) propose a
distributed futures-based shuffle system to provide better flexibility and
interoperability with model training workloads. However, those two proposals
mandate loading the entire dataset into its object store before they read and
shuffle the data, causing infeasible demands on DRAM resources for large
datasets (recall 6.4).

Shuffling in deep learning systems.  Due to the inefficiency of shuffled
loading with indices mapping in large datasets (recall 3.1), the programmers
may have to use buffered shuffling at the cost of model convergence accuracy.
This inspires some deep learning systems to modify the global access order and
the buffer eviction scheme accordingly (sun2022solar, ; DeepIO_buffer_shuffle,
) to maximize the data reuse and the buffer hit rate. However, similar to
partial shuffling, this breaks the true randomness of global shuffling no
matter modifying inter-batch or inter-epoch order, which limits the applicable
scope to specific models and datasets. Similar to NoPFS
(10.1145/3458817.3476181, ), they usually also involve significant efforts in
modifying the underlying software infrastructure.

Notably, all these solutions for reducing shuffle-related overheads in model
training can leverage Rinas’s unordered batch generation to further enhance
the performance. Our insight is obtained from an observation of the general
model training process.

Shuffle pattern exploration for ML.  Some researchers work on exploring
theoretical convergence bound of different local shuffle patterns
(meng2019convergence, ; Nguyen2020AUC, ; Ahn2020SGDWS, ;
Rajput2021PermutationBasedSI, ), with a hope to find an alternative to the
expensive global shuffling. However, their analysis is usually limited to
convex problems with low-dimensional data, leaving their effectiveness on
real-world problems and datasets not fully explored.

Data loading parallelism.  Previous proposals also leverage multi-core
parallelism to uncover data loading parallelism in different types of systems.
First, while our paper focuses on faster data loading from local SSDs, some
works focus on faster data loading from a remote datastore or file system
through the network. Barclay et al. (barclay1994loading, ) leverage dataflow
parallelism for memory-to-memory data loading via parallel TCP connections.
Yang et al. (yang2019accelerating, ) propose to accelerate data loading for
PyTorch-based distributed training systems from network-attached remote
storage. Second, some other works focus on local data loading, the same as
this work, leveraging asynchronously I/O to unleash unordered parallelism on
in-memory (lim2014mica, ; zhao2022altocumulus, ) or on-disk databases
(cheng2014scanraw, ; dziedzic2017dbms, ). ScanRaw speculatively uses more
cores to load data when additional disk bandwidth is available
(cheng2014scanraw, ). Dziedzic et al. (dziedzic2017dbms, ) have a similar
observation of this paper that file format is important on data loading
performance and draws the conclusion that data loading is CPU-bound. We
inherit previous work’s insights on leveraging asynchronous I/Os to solve an
unexplored problem: releasing the data loading order of each batch in model
training.

Preprocessing acceleration.  Some researchers are focusing on the
preprocessing acceleration for higher batch generation efficiency. For
example, NVIDIA Data Loading Library (DALI) (DALI, ) accelerates data loading
for image datasets by moving the image preprocessing workload to GPU. Since we
focus on different stages of the data loading pipeline and thus orthogonal to
each other, our work can be combined with DALI when working with image
datasets.

##  9\. Conclusions

Loading shuffled large datasets has become the key bottleneck in model
training throughput. However, existing solutions face different limitations
and usually leave programmers to face the trade-off between training speed and
convergence accuracy. In this work, we propose Rinas, a data loading framework
with the design of a novel execution model with unordered batch generation. By
leveraging the deep learning-specific domain knowledge, Rinas unleashes the
additional parallelism space in the shuffled loading process. Our evaluation
shows that Rinas accelerates model training substantially under the guarantee
of global randomness in shuffling. This work does not raise any ethical
issues.

## References

  * [1] Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng.  Tensorflow: A system for large-scale machine learning, 2016.
  * [2] Tanveer Ahmad.  Benchmarking apache arrow flight - a wire-speed protocol for data transfer, querying and microservices.  In Benchmarking in the Data Center: Expanding to the Cloud, BID’22, New York, NY, USA, 2022. Association for Computing Machinery.
  * [3] Kwangjun Ahn, Chulhee Yun, and Suvrit Sra.  Sgd with shuffling: optimal rates without component convexity and large epoch requirements.  arXiv: Optimization and Control, 2020.
  * [4] Sercan O. Arik and Tomas Pfister.  Tabnet: Attentive interpretable tabular learning, 2020.
  * [5] Tom Barclay, Robert Barnes, Jim Gray, and Prakash Sundaresan.  Loading databases using dataflow parallelism.  ACM Sigmod Record, 23(4):72–83, 1994.
  * [6] Koh Bee Hock David, Chin Lim, Hasnae Rahimi, and Wai Lok Woo.  Deep temporal convolution network for time series classification.  Sensors, 21:603, 01 2021.
  * [7] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang.  JAX: composable transformations of Python+NumPy programs, 2018.
  * [8] Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin.  Training deep nets with sublinear memory cost, 2016.
  * [9] Yu Cheng and Florin Rusu.  Parallel in-situ data processing with speculative loading.  In Proceedings of the 2014 ACM SIGMOD international conference on Management of data, pages 1287–1298, 2014.
  * [10] Together Computer.  Redpajama: An open source recipe to reproduce llama training dataset.  https://github.com/togethercomputer/RedPajama-Data, 2023.  Accessed: 2023-11-27.
  * [11] LIBSVM Data.  Criteo dataset.  https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/., 2011.  Accessed: 2023-11-27.
  * [12] Jeffrey Dean and Sanjay Ghemawat.  Mapreduce: simplified data processing on large clusters.  Communications of the ACM, 51(1):107–113, 2008.
  * [13] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [14] PyTorch developers and maintainers.  PyTorch dataloader.  https://pytorch.org/docs/stable/data.html#map-style-datasets.  Accessed: 2023-11-08.
  * [15] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
  * [16] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale, 2021.
  * [17] Nikoli Dryden, Roman Böhringer, Tal Ben-Nun, and Torsten Hoefler.  Clairvoyant prefetching for distributed machine learning i/o.  In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC ’21, New York, NY, USA, 2021. Association for Computing Machinery.
  * [18] Nikoli Dryden, Tim Moon, Sam Ade Jacobs, and Brian Van Essen.  Communication quantization for data-parallel training of deep neural networks.  In 2016 2nd Workshop on Machine Learning in HPC Environments (MLHPC), pages 1–8, 2016.
  * [19] Adam Dziedzic, Manos Karpathiotakis, Ioannis Alagiannis, Raja Appuswamy, and Anastasia Ailamaki.  Dbms data loading: An analysis on modern hardware.  In Data Management on New Hardware: 7th International Workshop on Accelerating Data Analysis and Data Management Systems Using Modern Processor and Storage Architectures, ADMS 2016 and 4th International Workshop on In-Memory Data Management and Analytics, IMDM 2016, New Delhi, India, September 1, 2016, Revised Selected Papers 4, pages 95–117. Springer, 2017.
  * [20] Jaliya Ekanayake, Hui Li, Bingjing Zhang, Thilina Gunarathne, Seung-Hee Bae, Judy Qiu, and Geoffrey Fox.  Twister: A runtime for iterative mapreduce.  In Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing, HPDC ’10, page 810–818, New York, NY, USA, 2010. Association for Computing Machinery.
  * [21] Fengli Gao and Huicai Zhong.  Study on the large batch size training of neural networks based on the second order gradient, 2020.
  * [22] Rong Gu, Kai Zhang, Zhihao Xu, Yang Che, Bin Fan, Haojun Hou, Haipeng Dai, Li Yi, Yu Ding, Guihai Chen, and Yihua Huang.  Fluid: Dataset abstraction and elastic acceleration for cloud-native deep learning training jobs.  In 2022 IEEE 38th International Conference on Data Engineering (ICDE), pages 2182–2195, 2022.
  * [23] Yanfei Guo, Jia Rao, Dazhao Cheng, and Xiaobo Zhou.  ishuffle: Improving hadoop performance with shuffle-on-write.  IEEE Transactions on Parallel and Distributed Systems, 28(6):1649–1662, 2017.
  * [24] Anurag Gupta, Deepak Agarwal, Derek Tan, Jakub Kulesza, Rahul Pathak, Stefano Stefani, and Vidhya Srinivasan.  Amazon redshift and the case for simpler data warehouses.  In Proceedings of the 2015 ACM SIGMOD international conference on management of data, pages 1917–1923, 2015.
  * [25] Sasun Hambardzumyan, Abhinav Tuli, Levon Ghukasyan, Fariz Rahman, Hrant Topchyan, David Isayan, Mikayel Harutyunyan, Tatevik Hakobyan, Ivo Stranic, and Davit Buniatyan.  Deep lake: a lakehouse for deep learning.  Proceedings of CIDR, 2023.
  * [26] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [27] HuggingFace.  Differences between dataset and iterabledataset.  https://huggingface.co/docs/datasets/about_mapstyle_vs_iterable.  Accessed: 2023-11-27.
  * [28] AnyScale Inc.  When to global shuffle.  https://docs.ray.io/en/latest/data/performance-tips.html#when-should-you-use-global-per-epoch-shuffling.  Accessed: 2023-11-27.
  * [29] WEKA Inc.  Weka file system.  https://docs.weka.io/overview/about, 2023.  Accessed: 2023-11-27.
  * [30] Ibrahem Kandel and Mauro Castelli.  The effect of batch size on the generalizability of the convolutional neural networks on a histopathology dataset.  ICT Express, 6(4):312–315, 2020.
  * [31] Vijay Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catanzaro.  Reducing activation recomputation in large transformer models, 2022.
  * [32] Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf.  Datasets: A community library for natural language processing.  In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 175–184, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
  * [33] Haidong Li, Jiongcheng Li, Xiaoming Guan, Binghao Liang, Yuting Lai, and Xinglong Luo.  Research on overfitting of deep learning.  In 2019 15th international conference on computational intelligence and security (CIS), pages 78–81. IEEE, 2019.
  * [34] Jingui Li, Xuelian Lin, Xiaolong Cui, and Yue Ye.  Improving the shuffle of hadoop mapreduce.  In 2013 IEEE 5th International Conference on Cloud Computing Technology and Science, volume 1, pages 266–273, 2013.
  * [35] Mu Li, David G. Andersen, Jun Woo Park, Alexander J. Smola, Amr Ahmed, Vanja Josifovski, James Long, Eugene J. Shekita, and Bor-Yiing Su.  Scaling distributed machine learning with the parameter server.  In 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI 14), pages 583–598, Broomfield, CO, October 2014. USENIX Association.
  * [36] Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, and Soumith Chintala.  Pytorch distributed: Experiences on accelerating data parallel training, 2020.
  * [37] Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and ”Teknium”.  Openorca: An open dataset of gpt augmented flan reasoning traces.  https://https://huggingface.co/Open-Orca/OpenOrca, 2023.  Accessed: 2023-11-27.
  * [38] Hyeontaek Lim, Dongsu Han, David G Andersen, and Michael Kaminsky.  $\\{$MICA$\\}$: A holistic approach to fast $\\{$In-Memory$\\}$$\\{$Key-Value$\\}$ storage.  In Proceedings of the 11th USENIX Symposium on Networked Systems Design and Implementation (NSDI), pages 429–444, 2014.
  * [39] Jie Liu, Bogdan Nicolae, and Dong Li.  Lobster: Load balance-aware i/o for distributed dnn training.  In Proceedings of the 51st International Conference on Parallel Processing, ICPP ’22, New York, NY, USA, 2023. Association for Computing Machinery.
  * [40] Shuhao Liu, Hao Wang, and Baochun Li.  Optimizing shuffle in wide-area data analytics.  In 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS), pages 560–571, 2017.
  * [41] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach, 2019.
  * [42] Frank Sifei Luan, Stephanie Wang, Samyukta Yagati, Sean Kim, Kenneth Lien, Isaac Ong, Tony Hong, Sangbin Cho, Eric Liang, and Ion Stoica.  Exoshuffle: An extensible shuffle architecture.  In Proceedings of the ACM SIGCOMM 2023 Conference, ACM SIGCOMM ’23, page 564–577, New York, NY, USA, 2023. Association for Computing Machinery.
  * [43] TorchVision maintainers and contributors.  Torchvision: Pytorch’s computer vision library.  https://github.com/pytorch/vision, 2016.  Accessed: 2023-11-27.
  * [44] Amrita Mathuriya, Deborah Bard, Peter Mendygral, Lawrence Meadows, James Arnemann, Lei Shao, Siyu He, Tuomas Karna, Daina Moise, Simon J. Pennycook, Kristyn Maschoff, Jason Sewall, Nalini Kumar, Shirley Ho, Mike Ringenburg, Prabhat, and Victor Lee.  Cosmoflow: Using deep learning to learn the universe at scale, 2018.
  * [45] Qi Meng, Wei Chen, Yue Wang, Zhi-Ming Ma, and Tie-Yan Liu.  Convergence analysis of distributed stochastic gradient descent with shuffling.  Neurocomputing, 337:46–57, 2019.
  * [46] Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I. Jordan, and Ion Stoica.  Ray: A distributed framework for emerging AI applications.  In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18), pages 561–577, Carlsbad, CA, October 2018. USENIX Association.
  * [47] Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I. Jordan, and Ion Stoica.  Ray: A distributed framework for emerging ai applications, 2018.
  * [48] Duy-Thanh Nguyen, Abhiroop Bhattacharjee, Abhishek Moitra, and Priyadarshini Panda.  Deepcam: A fully cam-based inference accelerator with variable hash lengths for energy-efficient deep neural networks, 2023.
  * [49] Lam M. Nguyen, Quoc Tran-Dinh, D. Phan, Phuong Ha Nguyen, and Marten van Dijk.  A unified convergence analysis for shuffling-type gradient methods.  J. Mach. Learn. Res., 22:207:1–207:44, 2020.
  * [50] Truong Thao Nguyen, François Trahay, Jens Domke, Aleksandr Drozd, Emil Vatai, Jianwei Liao, Mohamed Wahib, and Balazs Gerofi.  Why globally re-shuffle? revisiting data shuffling in large scale deep learning.  In 2022 IEEE International Parallel and Distributed Processing Symposium (IPDPS), pages 1085–1096. IEEE, 2022.
  * [51] Bogdan Nicolae, Carlos Costa, Claudia Misale, Kostas Katrinis, and Yoonho Park.  Towards memory-optimized data shuffling patterns for big data analytics.  In 2016 16th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid), pages 409–412. IEEE, 2016.
  * [52] NVIDIA.  Nvidia data loading library.  https://docs.nvidia.com/deeplearning/dali/user-guide/docs/.  Accessed: 2023-11-27.
  * [53] Iason Ofeidis, Diego Kiedanski, and Leandros Tassiulas.  An overview of the data-loader landscape: Comparative performance analysis.  arXiv preprint arXiv:2209.13705, 2022.
  * [54] Frank Olken and Doron Rotem.  Random sampling from databases: a survey.  Statistics and Computing, 5:25–42, 1995.
  * [55] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library, 2019.
  * [56] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  arXiv e-prints, 2019.
  * [57] Shashank Rajput, Kangwook Lee, and Dimitris Papailiopoulos.  Permutation-based sgd: Is random optimal?  ArXiv, abs/2102.09718, 2021.
  * [58] Neal Richardson, Ian Cook, Nic Crane, Dewey Dunnington, Romain François, Jonathan Keane, Dragoș Moldovan-Grünfeld, Jeroen Ooms, Jacob Wujciak-Jens, and Apache Arrow.  arrow: Integration to ’Apache’ ’Arrow’, 2023.  R package version 14.0.1, https://arrow.apache.org/docs/r/.
  * [59] Natalie Schluter and Daniel Varab.  When data permutations are pathological: the case of neural natural language inference.  In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4935–4939, Brussels, Belgium, October-November 2018. Association for Computational Linguistics.
  * [60] Min Shen, Ye Zhou, and Chandni Singh.  Magnet: push-based shuffle service for large-scale data processing.  Proceedings of the VLDB Endowment, 13(12):3382–3395, 2020.
  * [61] Konstantin Shvachko, Hairong Kuang, Sanjay Radia, and Robert Chansler.  The hadoop distributed file system.  In 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST), pages 1–10, 2010.
  * [62] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition, 2015.
  * [63] Baixi Sun, Xiaodong Yu, Chengming Zhang, Jiannan Tian, Sian Jin, Kamil Iskra, Tao Zhou, Tekin Bicer, Pete Beckman, and Dingwen Tao.  Solar: A highly optimized data loading framework for distributed training of cnn-based scientific surrogates, 2022.
  * [64] The Apache Software Foundation.  SparkR: R Front End for ’Apache Spark’, 2023.  https://www.apache.org https://spark.apache.org.
  * [65] Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Vincent Zhao, Yanqi Zhou, Chung-Ching Chang, Igor Krivokon, Will Rusch, Marc Pickett, Pranesh Srinivasan, Laichee Man, Kathleen Meier-Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas, Claire Cui, Marian Croak, Ed Chi, and Quoc Le.  Lamda: Language models for dialog applications, 2022.
  * [66] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.  Llama: Open and efficient foundation language models, 2023.
  * [67] Stephanie Wang, Eric Liang, Edward Oakes, Ben Hindman, Frank Sifei Luan, Audrey Cheng, and Ion Stoica.  Ownership: A distributed futures system for Fine-Grained tasks.  In 18th USENIX Symposium on Networked Systems Design and Implementation (NSDI 21), pages 671–686. USENIX Association, April 2021.
  * [68] Benjamin Welton, Dries Kimpe, Jason Cope, Christina M Patrick, Kamil Iskra, and Robert Ross.  Improving i/o forwarding throughput with data compression.  In 2011 IEEE International Conference on Cluster Computing, pages 438–445. IEEE, 2011.
  * [69] Daniel Whiteson.  HIGGS.  UCI Machine Learning Repository, 2014.  DOI: https://doi.org/10.24432/C5V312.
  * [70] Lijie Xu, Shuang Qiu, Binhang Yuan, Jiawei Jiang, Cedric Renggli, Shaoduo Gan, Kaan Kara, Guoliang Li, Ji Liu, Wentao Wu, Jieping Ye, and Ce Zhang.  In-database machine learning with corgipile: Stochastic gradient descent without full data shuffle.  In Proceedings of the 2022 International Conference on Management of Data, SIGMOD ’22, page 1286–1300, New York, NY, USA, 2022. Association for Computing Machinery.
  * [71] Chih-Chieh Yang and Guojing Cong.  Accelerating data loading in deep neural network training.  In Proceedings of the 26th IEEE International Conference on High Performance Computing, Data, and Analytics (HiPC), pages 235–245. IEEE, 2019.
  * [72] Xue Ying.  An overview of overfitting and its solutions.  In Journal of physics: Conference series, volume 1168, page 022022. IOP Publishing, 2019.
  * [73] Matei Zaharia, Reynold S. Xin, Patrick Wendell, Tathagata Das, Michael Armbrust, Ankur Dave, Xiangrui Meng, Josh Rosen, Shivaram Venkataraman, Michael J. Franklin, Ali Ghodsi, Joseph Gonzalez, Scott Shenker, and Ion Stoica.  Apache spark: A unified engine for big data processing.  Commun. ACM, 59(11):56–65, oct 2016.
  * [74] Jianyong Zhang, Anand Sivasubramaniam, Qian Wang, Alma Riska, and Erik Riedel.  Storage performance virtualization via throughput and latency control.  ACM Transactions on Storage (TOS), 2(3):283–308, 2006.
  * [75] Jiechen Zhao, Iris Uwizeyimana, Karthik Ganesan, Mark C Jeffrey, and Natalie Enright Jerger.  Altocumulus: Scalable scheduling for nanosecond-scale remote procedure calls.  In Proceedings of the 55th IEEE/ACM International Symposium on Microarchitecture (MICRO), pages 423–440. IEEE, 2022.
  * [76] Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li.  Pytorch fsdp: Experiences on scaling fully sharded data parallel, 2023.
  * [77] Yue Zhu, Fahim Chowdhury, Huansong Fu, Adam Moody, Kathryn Mohror, Kento Sato, and Weikuan Yu.  Entropy-aware i/o pipelining for large-scale deep learning on hpc systems.  In 2018 IEEE 26th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS), pages 145–156, 2018.