cslet cslet

#  Heterogeneous Acceleration Pipeline for Recommendation System Training

Muhammad Adnan∗ Yassaman Ebrahimzadeh Maboud∗ Divya Mahajan+ Prashant J. Nair∗
The University of British Columbia∗ Microsoft+
{adnan  yassaman<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Recommendation systems are unique as they show a conflation of compute and
memory intensity due to their deep learning and massive embedding tables.
Training these models typically involve a hybrid CPU-GPU mode, where GPUs
accelerate the deep learning portion and the CPUs store and process the
memory-intensive embedding tables. The hybrid mode incurs a substantial CPU-
to-GPU transfer time and relies on main memory bandwidth to feed embeddings to
GPU for deep learning acceleration. Alternatively, we can store the entire
embeddings across GPUs to avoid the transfer time and utilize the GPU’s High
Bandwidth Memory (HBM). This approach requires GPU-to-GPU backend
communication and scales the number of GPUs with the size of the embedding
tables. To overcome these concerns, this paper offers a heterogeneous
acceleration pipeline, called Hotline.

Hotline leverages the insight that only a _small number_ of embedding entries
are _accessed frequently_ , and can easily fit in a single GPU’s HBM. Hotline
implements a data-aware and model-aware scheduling pipeline that utilizes the
(1) CPU main memory for not-frequently-accessed embeddings and (2) GPUs’ local
memory for frequently-accessed embeddings. Hotline improves the training
throughput by dynamically stitching the execution of popular and not-popular
inputs through a novel hardware accelerator and feeding to the GPUs. Results
on real-world datasets and recommender models show that Hotline reduces the
average training time by 3$\times$ and 1.8$\times$ in comparison to Intel-
optimized CPU-GPU DLRM and HugeCTR-optimized GPU-only baseline, respectively.
Hotline increases the overall training throughput to 35.7 epochs/hour in
comparison to 5.3 epochs/hour for the Intel-optimized DLRM baseline.

##  1 Introduction

#####

Recommendation models are increasingly becoming one of the deployed Machine
Learning (ML) workload. Meta reports that 50% of their training cycles are
devoted to training deep learning based recommendation models [6]. Recommender
models are a disaggregated conflation of compute-intensive neural networks and
memory-intensive embedding tables [21]. The embedding tables store the user
and item features. The size of the embedding tables is expected to increase as
more users and items interact. For instance, a recent work from Baidu [57]
shows that production-scale recommendation models are reaching 10 TBs in size.

Deep Learning (DLRM) [40] and Time-Based Sequence (TBSM) [23] Models are often
trained in a hybrid setup of CPUs and GPUs. Figure 1 shows the training flow
of a recommender model in the hybrid mode with the CPU handling the embedding
data by offering its high memory capacity. The GPUs offer high compute
throughput and perform data parallel execution of the DNN phase [26]. A
training iteration includes reading the inputs, fetching corresponding
embedding data, neural network and embedding computations, and any data
transfer required to update the model.

The hybrid execution has low efficiency as (1) GPUs rely on the CPU to provide
all the embeddings to execute the DNN; (2) the low-bandwidth main memory acts
as a bottleneck during embedding-bag operation and embedding-weights’
optimization; and (3) it does not fully utilize the GPU for the entirety of
the training process – as the embeddings are executed on the CPU.
Alternatively, one can also use multiple GPUs to store a single-copy of the
embeddings and train in a model-parallel fashion. For instance, HugeCTR [2]
uses GPU memories to store the embeddings. Such techniques require the number
of GPUs to scale with the embedding size while the GPUs remain under-utilized
by the neural portion.

Figure 1: The execution flow of a typical recommendation model in the hybrid
CPU-GPU mode. Due to their large sizes, the embedding tables are stored and
processed on CPUs. The GPUs process the neural layers.

Instead in this work we offer a novel _throughput-optimized_ heterogeneous
acceleration pipeline called Hotline. Hotline aims to to efficiently use the
system resources - memory and compute - to pipeline the end-to-end training of
large-scale recommender models. _Hotline is the first work to enable the GPU-
only execution of the entire training of the recommender model, i.e, forward
pass, backward pass, loss function, and optimizer for both embeddings and DNN
portion, without requiring to store the entirety of embeddings on GPUs._ As
Hotline does not require each GPU to act as a bank for _all_ embeddings the
number of GPU devices does not scale with the model size. The pipeline also
mitigates the inefficiencies of the CPU-GPU hybrid execution mode by hiding
the latency of gathering embeddings from the main memory.

Access-aware memory layout for embeddings. To offer Hotline, we leverage the
insight that real-world recommender models train on inputs that exhibit
_extremely_ high popularity skew [7, 48, 12, 44, 8]. This causes certain
embedding entries to be accessed several orders-of-magnitude more frequently
than others. We refer to these embeddings as _frequently-accessed entries_.
The frequently-accessed entries have a small memory footprint while being
computationally significant. Keeping this access property in mind, Hotline
foremost offers an optimized _access-aware memory layout for embeddings_ that
is utilized by a novel _runtime scheduler to perform mini-batch scheduling
onto GPUs to ensure the parameter gathering of embeddings is pipelined with
compute._ Our framework stores the frequently-accessed embeddings on GPUs to
mitigate their transfer overheads.

Layout-aware runtime scheduling. To utilize this memory placement, Hotline
requires a runtime scheduler that can tackle the challenge that a minibatch of
inputs (2K-32K) can access a combination of frequently-accessed and not-
frequently-accessed embeddings. While frequently-accessed embeddings are on
GPU, accessing the not-frequently-accessed embeddings from main memory can
stall the pipeline. The runtime scheduler mitigates this by reforming a small
set of mini-batches into two classes. The first class only accesses
frequently-accessed embeddings and are directly scheduled onto GPUs. This is
because the local GPU memory contains the required working parameters. The
second class contains the remaining inputs that access not-frequently-accessed
embeddings. To hide the data transfer latency, the scheduler gathers the
required working parameters from the CPU memory while the frequently-accessed
inputs are being executed on GPUs. The second task needs to be performed in a
high-throughput manner to ensure that GPU is not starving during the non-
popular inputs. The scheduler overcomes this by offering a novel runtime
_accelerator_ that sits between the GPUs and the main memory. Thus the
accelerator schedules the input mini-batches in a data and model-aware manner
and by efficiently gathering embedding parameters from both the GPU
(frequently-accessed) and CPU-based main (not-frequently-accessed) memory.

Contributions: We make the following contributions:

  1. 1.

Identify the frequently-accessed embeddings _dynamically_ at runtime with
negligible overhead.

  2. 2.

Offer a dynamic data and model-aware scheduler to efficiently pipeline the
mini-batch dispatch onto GPUs while concurrently obtaining not-frequently-
accessed embeddings from the CPU-based main memory.

  3. 3.

Offer a runtime framework that increases the training throughput by stitching
the recommender model execution onto GPUs – _without_ impacting fidelity.

We evaluate Hotline with publicly available deep learning (DLRM) and time-
sequence (TBSM) based recommendation models. We compare against two baselines.
Namely, a hybrid CPU-GPU baseline where CPUs store and compute on embeddings
and a GPU-only embedding processing baseline. In both these cases, multiple
GPUs parallelize the neural network computations,

For the CPU-GPU hybrid baseline, we compare Hotline against state-of-the-art
deep learning framework XDL [27] and Intel [30] optimized DLRM. On average,
the Hotline acceleration pipeline provides 3.75$\times$ speedup over the 4-GPU
XDL and 3$\times$ speedup over Intel optimized DLRM while maintaining the
baseline accuracy. Hotline is able to perform training of larger models such
as Terabyte even while using a single GPU, whereas GPU-only baseline requires
at least 4 GPUs to store its embeddings. On average, Hotline obtains
1.8$\times$ performance benefits over the GPU-only baseline, as it eliminates
data movement between the GPU devices.

##  2 Background and Challenges

###  2.1 A Primer on Recommendation Systems

####  2.1.1 Machine Learning Model:

Recommender models use categorical features and continuous features.
Typically, large embedding tables store the categorical features for each user
and item. On the other hand, the continuous features are processed by
reasonably-sized neural network layers which range from Multi-layer
Perceptrons (MLP) to DNNs. Recommendation models are currently among the
largest ML models that are being executed in data centers. A recently released
recommendation model by Meta contains 12 Trillion parameters [37] in
comparison to GPT-3 with 175 Billion parameters [10].

####  2.1.2 Training Setup:

Large recommender models can be trained in two distributed modes. First, the
embeddings are stored and gathered on CPU and the neural networks are executed
on GPUs in a data-parallel fashion. Now, the training throughput is throttled
by large data transfer and low bandwidth CPU main memory. Figure 2 shows the
training time being spent on these operations in 4 real-world models and
datasets. We observe that the communication and embedding-bag operation can be
as high as 75% for large datasets such as Criteo Terabyte. This presents an
opportunity for acceleration.

Figure 2: The breakdown of the training time for an Intel-optimized DLRM with
1 GPU in a hybrid CPU-GPU training setup. The dotted parts of the bar are
executed on CPU and present an opportunity for GPU-based acceleration.

Second, we can use as many GPUs as required to store the embeddings fully
while enabling data-parallel neural-network execution. However, this mode has
low compute utilization, because the number of GPUs scale based on embedding
sizes and not with the requirements of the neural network. On the contrary,
our paper exploits the semantics of user behaviour, i.e. training input, in
recommendation models to efficiently use the memory capacity and improve
compute throughput.

####  2.1.3 Popularity in Training Inputs:

Hotline exploits a key property of recommendation system inputs – that certain
users and items observe very high popularity. This implies that some embedding
entries are accessed much more frequently. Figure 3 shows that, across real-
world datasets, a small number of frequently-accessed embeddings have
$>$100$\times$ more accesses than others. Our experiments show that simply
tracking up to 512 MB of embeddings can help cater to $>$75% of training
inputs. Tracking beyond 512 MB of frequently-accessed embeddings only provides
diminishing returns. However, exploiting input popularity at a fine
granularity is non-trivial.

Figure 3: The frequency of accesses per embedding entry for one training
epoch. The frequently-accessed embeddings tend to have more than 100$\times$
more accesses compared to others. Inputs with rate of 1-in-every-100000
embedding accesses are labeled as frequently-accessed.

###  2.2 Challenges in Exploiting Input Popularity

Limitation – Segregation and Scheduling with CPU: One could simply employ CPU-
based multi-processing to segregate frequently-accessed and not-frequently-
accessed embeddings and schedule the corresponding mini-batches. Figure 4
shows that an Intel-Xeon CPU, even after using _all its cores_ for multi-
processing, has up to 2.5$\times$ higher latency as compared to NVIDIA-V100
GPU(s) executing 4 training iterations. Additionally, as we execute larger
mini-batch sizes, the CPU processing overhead and its latency increases
proportionately. Therefore, typical CPUs _cannot_ complete segregating popular
and not-popular mini-batches and schedule them before the GPU(s) finish
execution.

Figure 4: Comparison of multi-process CPU-based segregation and scheduling
time using an Intel Xeon CPU for 4 training iterations with NVIDIA-V100
GPU(s). We use mini-batches of 1K, 2K, and 4K for 1, 2, and 4-GPU execution
respectively. Even after using multiprocessing, the CPU has a higher latency
and cannot finish segregating and scheduling popular and not-popular mini-
batches before the GPU(s) finish execution.

Challenge 1 – Tackle Changing Access Frequency: Prior works use an _offline_
profiler to identify the frequently-accessed embeddings. They use two
suppositions [7, 48]. First, that training data is fully available before
training. Second, that frequently-accessed embeddings do not change over time.

We observed that training data is typically updated on a daily basis. Thus,
one cannot access the full training dataset ahead in time. Furthermore, as the
dataset evolves daily and a recent work, called BigPipe [8], has shown the set
of frequently-accessed embeddings also rapidly change over time.

Challenge 2 – Dynamically Capture the Access Skew: We need to dynamically
determine the frequently-accessed embedding entries at very low latency. This
helps avoid extra overheads in the case of online training where training
samples could arrive sporadically. We must also adapt to the changing
popularity of inputs and update the frequently-accessed embedding entries
continuously with very low overheads.

Our Approach: Hotline dynamically determines the frequently-accessed
embeddings by processing only a small portion of the training inputs. Hotline
uses a novel accelerator to enable fine-grained parallel embedding index
tracking across mini-batches. The Hotline accelerator can re-calibrate the
frequently-accessed embeddings as the training data changes over time.

Challenge 3 – Training with Mixed Mini-Batches: Each mini-batch can require a
mix of frequently-accessed and other embeddings. Thus, the working parameters
can be spread across the CPU main memory (for not-frequently-accessed
embeddings) and GPU’s (for frequently-accessed embeddings) local memory.
Consequently, embeddings from the main memory would need to be collected and
sent to the GPU before computation. Gathering working parameters using GPUs is
not ideal as they are not only inefficient at collecting irregularly scattered
embeddings, but would also waste their precious compute cycles on intricate
fine-grained operations.

Our Approach: The Hotline accelerator uses immensely parallel compute engines
that gather several inputs. The accelerator then creates mini-batches of
inputs that only access frequently-accessed embeddings and dispatches them to
GPUs. While the frequently-accessed inputs are being executed, the accelerator
gathers the embeddings from CPU main memory for the remaining not-frequently-
accessed embeddings inputs thereby hiding their communication overheads.

Figure 5: Hotline’s heterogeneous system architecture. The accelerator sits
between the CPU and the GPU(s). It can directly access the main memory to
obtain the training inputs and not-frequently-accessed embeddings and relay
them to the GPU. Figure 6: An example execution pipeline of Hotline training
with two mini-batch working set. The accelerator classifies inputs into
popular and non-popular mini-batches, schedules the popular one directly on
GPU, and in the meantime gathers the working parameters for non-popular one to
schedule onto GPU. The accelerator hides the data gather latency behind GPU
execution.

##  3 System Design and Execution

The baseline system comprises a server multi-core CPU connected to multiple
GPU devices via the PCIe-link. The inter-GPU communication happens through
NVLink [3]. Figure 5 shows the Hotline system; it places the accelerator on
the _low profile PCIe slot_ not being used by a GPU. The accelerator can
access the DMA engine via the PCIe switch and can directly communicate with
the main memory whilst entirely avoiding the high-overhead CPU pipeline.
Hotline does not require any changes to CPU and GPU devices. The accelerator
employs two phases during training: (1) The ‘access learning’ phase identifies
the frequently-accessed embeddings.(2) The ‘runtime accelerated’ phase
pipelines popular and non-popular mini-batches to the GPU.

###  3.1 Hotline Training Execution Phases

####  3.1.1 Access Learning Phase

During training runtime, in the first phase, called the access learning phase,
the Hotline accelerator samples mini-batches to determine which embeddings are
frequently-accessed. This is performed _only_ in the first epoch _only_ for a
% of mini-batches. Our experiments show that, the accelerator only needs to
sample 5% of the mini-batches for datasets like Criteo Kaggle [14] and Criteo
Terabyte [15]. It only needs to sample 15% and 20% of the mini-batches for
Avazu [29] and Taobao Alibaba [9] datasets, respectively. Such low-overhead
sampling can identify $>$90% of frequently-accessed embeddings.

During this phase, all the embedding entries are obtained from the CPU main
memory and gradually some of these entries are classified as frequently-
accessed. Once the first phase completes, the contents of the frequently-
accessed embeddings are replicated across all the GPUs. The Hotline
accelerator only stores the _indices_ of the frequently-accessed embeddings
and does not need to store the data contents of frequently-accessed
embeddings. The frequently-accessed embedding indices are used by the
accelerator to classify popular and non-popular inputs into mini-batches. The
detailed execution of the accelerator during this phase is described in
Section 4.

####  3.1.2 Runtime Accelerated Phase

Once the frequently-accessed embeddings are replicated on each GPU, the
_runtime accelerated phase_ commences. The Hotline accelerator reorders
popular and non-popular inputs across a collection of mini-batches. We refer
to these mini-batches as the working set of the accelerator. The Hotline
accelerator creates _new_ mini-batches from this working set.

Now a mini-batch contains inputs that only accesses the frequently-accessed
embeddings (popular mini-batch) or contain inputs that access both frequently-
accessed and not-frequently-accessed embeddings. As the GPU memories contain
the frequently-accessed embeddings, the popular mini-batches can be directly
scheduled onto the GPUs. The accelerator obtains the embeddings from both the
CPU and GPU memories while GPUs are executing popular minibatches and hides
the embedding gather latency. CPU and GPU memories have the most updated not-
frequently-accessed and frequently-accessed embeddings respectively. Hotline
does not encounter any synchronization or cache coherence overheads because
the not-frequently-accessed embeddings and frequently-accessed embeddings are
always accessed from the CPU main memory and GPUs respectively.

###  3.2 Pipelined Execution at Runtime

Figure 6 shows an example of the pipelined execution across the CPU, the
Hotline accelerator, and the GPU. All the training inputs are obtained from
the CPU main memory. If we have a working set of four mini-batches, on
average, three mini-batches worth of inputs would be popular. This is because,
on average, $>$75% of the inputs are popular and they can be directly
scheduled onto the GPUs. While the popular mini-batches are executing on the
GPU, the accelerator obtains the embeddings for the non-popular minibatch from
CPU and GPU memories. Once the GPU is done executing popular mini-batches, the
accelerator schedules the non-popular mini-batch to the GPUs along with its
working parameters.

The forward and backward pass for the entirety of the recommender model
executes on GPUs for both the popular and non-popular mini-batches This is
because, now the required parameters are always in the local GPU memory. As
the non-popular mini-batch is executing on the GPU, the accelerator begins to
read the next set of mini-batches from the CPU main memory. The details of how
the accelerator segregates mini-batches and gathers embeddings are provided in
Section 4.4.

##  4 Design: Hotline Accelerator

Figure 7 shows the block diagram of the Hotline accelerator. We describe the
micro-architectural details of each component and then provide an example
execution flow.

Figure 7: Block diagram of the Hotline hardware accelerator. The hardware
accelerator connects to the low-profile PCIe slot via a Streaming Interface.

###  4.1 Hotline Accelerator: High-Level Flow

We describe the high-level flow during the access learning and the runtime
accelerated phases:

Access learning phase. The data dispatcher block reads input mini-batches from
CPU main memory via the streaming interface, sends them to the lookup engine
block, and updates the embedding access logger (EAL) block. The role of the
EAL block is to keep track of the amount of times an embedding index has been
accessed across mini-batches. Based on the access pattern of the embedding
indices across first few mini-batches, the Hotline accelerator can determine
which entries are frequently-accessed. Once this phase completes, all entries
in the EAL block becomes read-only.

Runtime accelerated phase. The data dispatcher and the scheduler play a vital
role. The data dispatcher directs each input in the mini-batch to the lookup
engine. The output of the lookup engine then flows into the input classifier.
The input classifier marks each incoming input as either popular or non-
popular. The non-popular inputs are collected in the _input eDRAM_ while the
popular inputs are dispatched to the GPU(s). Once a mini-batch worth of non-
popular inputs are collected and their working parameters are fetched from GPU
and CPU memories, the scheduler dispatches this mini-batch to the GPU devices.

###  4.2 Hotline Accelerator: Components

####  4.2.1 Data Dispatcher

Figure 8 shows the block diagram of the the Data Dispatcher block. The
_Address Registers_ store the base address of each embedding table in the main
memory and GPU memory. The _Memory Controller_ generates the address using the
Address Registers to access the embeddings within the CPU and GPU memories.
The _Input Classifier_ segregates incoming inputs as popular and non-popular
based on availability of embeddings within EAL. While the popular mini-batches
are being executed on GPU(s), the dispatcher issues the non-popular mini-batch
from the input eDRAM to the Lookup Engine. This enables the Hotline
accelerator determine the working parameters of the non-popular mini-batch at
runtime.

Figure 8: The Address Registers, Memory Controller, and Input Classifier
constitute the Data Dispatcher block.

A non-popular minibatch consists of inputs that require a mixture of not-
frequently-accessed and frequently-accessed embeddings. For accessing the not-
frequently-accessed embeddings (present in main memory), the Memory Controller
uses the Address Registers to calculate the embedding address based on the
embedding row index and its sparse dimension size. The memory controller then
issues a direct memory access (DMA) request to the DMA-engine for not-
frequently-accessed embeddings. While for frequently-accessed embeddings, a
GPU read memory request is issued. Thereafter, the working parameters obtained
from CPU and GPU memory are processed by the Reducer block. This block reduces
multiple embedding rows into a single embedding vector and stores it in the
_Embedding Vector Buffer_.

####  4.2.2 Embedding Access Logger (EAL)

During the access learning phase, the _Embedding Access Logger_ (EAL) uses
counters to track the access frequencies of embedding entries. The logger only
stores the indices and not the contents of the embedding entries. As Figure 9
shows EAL consists of a Multi-Banked SRAM, Controller, and an input/output
Queue.

Figure 9: The block diagram of the components within the Embedding Access
Logger (EAL). It consists of an Multi-Banked SRAM, Controller, and Queue sub-
blocks.

Naive Embedding Tracking. A naive way to track frequency of accesses into each
embedding entry is to use a separate counter for each sparse parameter. As
number of sparse parameters can be in the range of billions, this method of
tracking would require gigabytes of on-chip storage. Alternatively, storing
the frequencies in the CPU/GPU memory will incur 3 accesses - one to obtain
the embeddings, one to read the frequency, and another to update the
frequency.

Efficient embedding frequency tracking: EAL is designed to dynamically track
only the frequently-accessed embeddings by using their indices. Our design is
motivated by two observations. First, compared to the overall embedding size
(10-100 GBs), the size of frequently-accessed embeddings is under 512 MB.
Second, the access skews for these entries is extremely high ($>$ 100$\times$
accesses). Thus, EAL is designed as a cache-like structure that keeps track of
frequently-accessed embedding indices. For the purpose of this paper we assume
a 4-way set associative SRAM cache. Assuming an embedding dimension of 64,
with each entry being 4 Bytes, 512 MB worth of frequently-accessed embeddings
contains 2 million indices. Thus, the Hotline accelerator uses a 4 MB SRAM
cache with 2 million blocks. As we need to classify embeddings based on their
access frequency, we could use the Least Frequently Used (LFU) replacement
policy. However, this would would require significant area overheads, as each
of these 2 million cache blocks would require 24-bit counters (Figure 3 shows
embeddings can have up to 10 million accesses).

To minimize area overheads, EAL uses the static Re-reference Interval
Predictor (SRRIP) replacement policy with a 2-bit Re-reference Predictor Value
(RRPV) counter [25]. As frequently-accessed embeddings have $>$100$\times$
more frequent accesses, a 2-bit RRPV counter with insertions at RRPV-1 value
captures $>$ 99% of the frequently-accessed embeddings. Figure 10 compares
SRRIP based optimal logger to an Oracle which has infinite-sized access
counters for each embedding entry.

Figure 10: SRRIP-based tracker as compared to the Oracle LFU scheme. On
average, the SRRIP-based tracker can track 70% of the frequently-accessed
embeddings.

The SRRIP based EAL SRAM cache has 70% tracking capability. Note that, even if
a non-frequently-accessed embedding is mis-classified as frequently-accessed
or vice versa, model correctness does not change.

Multi-Banked SRAM for multi-embedding lookup. Each input accesses multiple
embeddings within each embedding table. Hotline enables these parallel lookups
by dividing EAL into multiple banks. Figure 11 shows our empirical design
space exploration wherein the average number of requests issued as the number
of banks (n) increase while varying input queue size (m). On average, a
512-sized queue with 64-banks enables 60 parallel requests into unique banks
per iteration. The EAL uses a controller to schedule the incoming requests
from lookup engine block to the 512-entry queue.

Figure 11: Impact of Queue Size and Banks on # of parallel requests per
iteration. Embedding Access Logger uses a 512 queue with 64-banks to enable 60
parallel requests.

Once the frequently-accessed embeddings are determined, the EAL is switches to
stops updating and provides these embedding indices during the runtime
accelerated phase. EAL periodically switches back to the access learning phase
and captures the changes in popularity,

####  4.2.3 Lookup Engine

Lookup engine is a parallel 2D compute network that processes on every
training input and extracts their embedding entries. For instance, if an input
requires 26 distinct embedding tables, the accelerator can achieve 26x
throughput per input. Furthermore, the 2D compute network enables exploiting
parallelism across training inputs within the mini-batch working set. The
Lookup engine, during the access learning phase, provides EAL with the indices
accessed by each input for its updates. During the runtime accelerated phase,
the Lookup Engine tags inputs as popular if all embedding indices are in EAL
as frequently-accessed.

Figure 12: Architecture of the Lookup Engine. The Lookup engine array
determines the embedding entries that are used by a training input. Figure 13:
Dataflow through the Hotline accelerator. The accelerator is responsible for
re-arranging popular/non-popular inputs, gathering their embeddings, and
scheduling them on the GPU.

Figure 12 shows the components within a single lookup engine. Each engine
consists of registers containing embedding table number, embedding index, hot
embedding index register and a randomizer. The embedding table number and
index for the input are merged and sent to the randomizer. The randomizer
creates a hash of the (Embedding Index, Embedding Table) tuple. This hash
scatters embedding index values across EAL to avoid trashing during the access
learning phase. We implement our randomizer block using a low-latency Fiestal
Network implementation [35].

####  4.2.4 Reducer

The Reducer performs sparse length sum element-wise operation using an array
of simple arithmetic units. It reduces multiple embedding rows a table into a
single vector (embedding pooling operation) and stores the result in embedding
vector buffer.

###  4.3 Hotline Accelerator: Instruction Set Architecture

As the Hotline accelerator interacts with both the main memory and GPU via the
PCIe link, it requires a brief set of instructions (Table 1) to read/write the
required data into these devices. The Hotline ISA facilitates communication
with the DMA engine and GPUs.

Table 1: Hotline Instructions Set

Instruction | Operand 1 | Operand 2 | Description
---|---|---|---
dma_rd(op1, op2) | mem start idx | # bytes | DMA read request
dma_wr(op1, op2) | mem start idx | # bytes | DMA write request
v_add(op1, op2) | input vector | emb vec buff | element wise addition
v_mul(op1, op2) | input vector | emb vec buff | element wise dot product
s_wr(op1, op2) | reg idx | base addr | write emb base addr
gpu_rd(op1, op2) | gpu device id | sparse idx | read emb idx from GPU device

###  4.4 Hotline Accelerator: Tying it together

Figure 13 shows an example execution of the accelerator during the runtime
accelerated phase. The Hotline accelerator reads mini-batches from the main
memory and segregates them into popular and non-popular sets. In this example
we show a working set of 4. Input eDRAM just stores the non-popular mini-batch
while popular mini-batches are sent to the GPU(s). The _Scheduler_ block
organizes the timeline to issue non-popular mini-batch after the popular mini-
batches complete execution. Popular mini-batches execute in a data parallel
fashion as all the model parameters are present in the local memory of each
GPU. Simultaneously, the scheduler sends the inputs of the non-popular mini-
batch from _input eDRAM_ to the _Lookup Engine_ block. The _Lookup Engine_
block checks whether the indices are located in the EAL, if so, they are
classified as frequently-accessed and obtained from the GPU(s), else are
obtained from the main memory.

Each embedding entry obtained from main or GPU memory goes through the
_Reducer_ block. For multiple embedding lookups within an embedding table, the
_Reducer_ block applies embedding pooling operation to convert all the lookups
into a single embedding vector. Thereafter, the embedding vector is
temporarily stored in the _embedding vector buffer_. Once the all the working
parameters of non-popular mini-batch have been fetched, the non-popular mini-
batch is dispatched along with the embedding vectors. The entire execution
including the forward, backward pass, and the optimizer execute on the GPU for
all the mini-batches. Once the embedding updates are generated by the
optimizer, the updated not-popular embeddings are stored are written to the
CPU main memory. Thus, the CPU maintains the most up-to-date non-frequently-
accessed embeddings whereas the GPU stores the frequently-accessed embeddings
for the entirety of the training process.

Table 2: Recommender Model Architecture and Parameters

Model | Dataset | Time | Features | Parameters | Neural Network Configuration | Size
---|---|---|---|---|---|---
Series | Dense | Sparse | Dense | Sparse | Sparse Dim | Bottom MLP | Top MLP | DNN | (GB)
RM1 | Taobao Alibaba [9] | 21 | 1 | 3 | 7.3 k | 5.1 M | 16 | 1-16 | 30-60-1 | Attn. Layer | 0.3
RM2 | Criteo Kaggle [14] | 1 | 13 | 26 | 287.5 k | 33.8 M | 16 | 13-512-256-64-16 | 512-256-1 | - | 2
RM3 | Criteo Terabyte [15] | 1 | 13 | 26 | 549.1 k | 266 M | 64 | 13-512-256-64 | 512-512-256-1 | - | 63
RM4 | Avazu [29] | 1 | 1 | 21 | 281.4 k | 9.3 M | 16 | 1-512-256-64-16 | 512-256-1 | - | 0.55

(a) Criteo Kaggle

(b) Taobao Alibaba

(c) Criteo Terabyte

(d) Avazu

Figure 14: AUC comparison of Hotline with a baseline system while performing
full-precision training. Hotline maintains the training fidelity throughout
execution.

##  5 Evaluation Methodology

###  5.1 Recommender Models

Table 2 illustrates the specifications of four well-established recommender
models used to evaluate Hotline. The number of sparse parameters varies from
5.1M (RM1) to 266M (RM3). The number of dense parameters is around 5 orders of
magnitude less than the number of sparse parameters. The bottom MLP computes
on dense inputs, whereas the top MLP generates the final recommendation for
the user. RM1 is a neural network dominated model with top and bottom MLP
layers and a deep learning attention layer. RM2 and RM3 models, instead, are
embedding dominated as they consist of more sparse features and bigger
embedding tables. RM4 has an average size of dense neural network and sparse
embedding tables. We use open source implementation of Deep Learning (DLRM)
[40] and Time-based Sequence Models (TBSM) [23] benchmarks to train the above
mentioned models. TBSM trains the RM1 model and DLRM is used to train RM2, RM3
and RM4 models. TBSM consists of an embedding layer implemented using DLRM and
a time-series layer resembles an attention mechanism and contains its own
neural networks.

###  5.2 Datasets

We use four real-world datasets to train the above mentioned models. Table 2
shows the details of each dataset. Taobao Alibaba [9] is a user behaviors
dataset from Taobao, for recommendation problem with implicit feedback offered
by Alibaba. Criteo Kaggle [14] dataset is obtained from Display Advertising
Challenge and contains advertising data from Criteo to capture user’s
preference by predicting click-through-rate. Criteo Terabyte [15] is the
largest publicly available dataset for users click logs. Avazu [29] dataset is
taken from click-through-rate prediction competition by Kaggle.

###  5.3 Software libraries and setup

DLRM and TBSM are configured using Pytorch-1.9 using Python-3. We use the
_torch.distributed_ backend to support scalable distributed training and
performance optimization [46]. For all the backend GPU-to-GPU communication
such as gather, scatter, and all-reduce collective calls, we use the fast
NVIDIA Collective Communication Library (NCCL) [41] on NVLink-2.0. We compare
against DLRM and TBSM implementation on XDL [27], Intel-optimized DLRM [30]
and FAE [7]. We use Tensorflow-1.2 [5] as computation backend for XDL-based
implementation.

Figure 15: The performance comparisons Hotline with XDL, Intel optimized DLRM,
and FAE implementations (normalized to a 1-GPU XDL). On average, even a 1-GPU
Hotline provides 3.5$\times$ higher performance as compared to the baseline.

###  5.4 Server Architecture

Table 3 shows the details of the server. They comprise 24-core Intel Xeon
Silver 4116 (2.1 GHz) processor with Skylake architecture. Each server is
connected to 4 NVIDIA Tesla-V100 GPUs. Every GPU and the Hotline accelerator
communicates with the rest of the system via a 16x PCIe Gen3 bus. We perform
experiments on a single server.

Table 3: System Specifications

Device | Architecture | Memory | Storage
---|---|---|---
CPU | Intel Xeon | 192 GB | 1.9 TB
| Silver 4116 (2.1 GHz) | DDR4 (2.7 GB/s) | NVMe SSD
GPU | Nvidia Tesla | 16 GB | -
| V100 (1.2 GHz) | HBM-2.0 (900 GB/s) |

###  5.5 Runtime measurements.

For the CPU and GPU executions we measure the wall clock time to train the
entire model based on its convergence criterion. For Hotline accelerator, we
synthesize the architecture implemented in Verilog RTL using Synopsys DC with
45$nm$ technology and validated 350 MHz as the frequency. We obtain the access
time of the memory components in the accelerator using Cacti [38]. We used a
RTL-simulation for compute cycles and access cycles from Synopsys DC and Cacti
to calculate the accelerator runtime. Real system DMA latency and HBM latency
are being used for estimating working parameters gathering time. Also we
empirically observed that fetching frequently-accessed embeddings
alternatively from each GPU device would rule out the GPU HBM contention and
results in balanced memory load on each GPU device. The details of the
accelerator are provided in Table 4. We observe that a size of 2.5 MB for the
input eDRAM is sufficient to store 16 K input non-popular mini-batch. The 0.5
KB embedding vector buffer can store an embedding vector with a dimension of
128. We chose the eDRAM [52] technology because of its high storage density.
We use a 4 MB embedding access logger (EAL).

Table 4: Accelerator Specifications

Parameters | Settings | Parameters | Settings
---|---|---|---
Frequency | 350 MHz | EAL size | 4 MB
No of Reducer ALU Units | 16 | No of Lookup Engines | 64
Input eDRAM size | 2.5 MB | Embedding Vector Buffer | 0.5 kB
Total Area | 7.01 $mm^{2}$ | Average Energy | 132 mJ

###  5.6 Area/Energy measurements of Hotline

We use Cacti [38] to estimate the area/energy of memory components (40 $nm$
technology). We implemented the Verilog RTL and synthesized with Synopsys
Design Compiler (45 $nm$ FreePDK process) for other components.

Figure 16: Latency breakdown of 1, 2, and 4 GPU implementations of software
frameworks and Hotline. The overhead of CPU-GPU communication time increases
as number of GPUs scale because of inter-GPU communication. Note that, unlike
Hotline, FAE incurs a reasonably high overhead from embedding synchronization.

##  6 Results and Analysis

###  6.1 Training Accuracy

Hotline segregates the mini-batches into popular and non-popular ones within
the working set. Hence, we evaluate the accuracy implications of Hotline.
Figure 14 illustrates the Area Under Curve (AUC) accuracy metric for Criteo
Kaggle, Criteo Terabyte, and Avazu datasets as established by MLPerf [1, 53].
For the Taobao dataset, we use the test accuracy – as AUC metric is not
offered for the TBSM model. We use full-precision DLRM and TBSM model
implementations using Intel’s optimized DLRM [30] for accuracy comparisons. We
observe that Hotline completely follows the baseline test and train accuracy.
Table 5 compares the testing accuracy, AUC, and cross-entropy loss for all
workloads.

Table 5: Accuracy Metric Comparisons

Dataset | DLRM | Hotline
---|---|---
Accuracy (%) | AUC | Logloss | Accuracy (%) | AUC | Logloss
Criteo Kaggle | 78.64 | 0.798 | 0.456 | 78.64 | 0.798 | 0.456
Taobao Alibaba | 89.11 | - | 0.270 | 89.34 | - | 0.264
Criteo Terabyte | 81.20 | 0.792 | 0.421 | 81.21 | 0.792 | 0.421
Avazu | 83.61 | 0.766 | 0.387 | 83.61 | 0.768 | 0.386

###  6.2 End-to-End Performance

####  6.2.1 Performance benefits over hybrid CPU-GPU Baselines

Figure 15 demonstrates the end-to-end training benefits obtained due to
Hotline in comparison to three state of the art software implementations with
varying number of GPUs. First, XDL [27] is an Industrial Deep Learning
Framework that uses a parameter server approach. Here the GPU improves the
efficiency of Advanced Model Server by using a faster embedding dictionary
lookup on GPU while CPU is used as backend workers. Second, an Intel-Optimized
DLRM [30] implementation, where the embeddings are executed on CPU using lock
free updates and neural networks on GPUs in data parallel fashion. Third, a
software framework called FAE [7], that uses input popularity but combines
offline pre-processing with CPU based scheduling without pipelining. For the
data-parallel approach, we deploy the weak scaling technique where we scale
the mini-batch size with number of GPUs, i.e., 1K for 1-GPU setup, 2K for
2-GPU setup, and 4K for 4-GPU. All the numbers are normalized to XDL 1-GPU
setup.

As shown in Figure 15, Hotline across all the models, datasets, and settings,
reduces the overall training time as it performs the entire recommender model
compute on the GPU(s). On average Hotline obtains a speedup of 3.5$\times$ and
3.5$\times$ for 1-GPU, 3.15$\times$ and 3.11$\times$ for 2-GPU, 3.75$\times$
and 3$\times$ for 4-GPU over XDL and optimized DLRM implementations. As
compared to a recent contemporary work called FAE [7], Hotline obtains a
speedup of 2.3$\times$, 2.9$\times$, and 3.4$\times$ for the 1-GPU, 2-GPU, and
4-GPU implementations respectively.

###  6.3 Latency Breakdown

Figure 16 shows the latency breakdown of the two hybrid baselines and Hotline.
Hotline incurs a negligible overhead due to its efficient pipelining. For the
Criteo Kaggle and Terabyte datasets, which are more embedding and memory
intensive, Hotline significantly reduces the CPU-GPU communication time. For
the Taobao dataset, which is neural network centric, the deep learning
execution overshadows the communication time. Broadly, Hotline always reduces
the total compute time.

###  6.4 Comparison Studies

We compare Hotline against a GPU-only Model-Parallel and a CPU-based Hotline
implementation.

####  6.4.1 GPU-only Model-Parallel Baseline:

We use HugeCTR [2] by Nvidia as the GPU-only baseline. HugeCTR scales the
number of GPUs to fit the entire model within GPUs’ local memories and
performs model parallel training for embeddings while using data parallel
training for the neural network portion. As shown in Figure 17, we compare the
performance of Hotline with HugeCTR for two models, Criteo Kaggle and Criteo
Terabyte – the only ones offered by the current HugeCTR evaluation. We observe
that for a small model like Criteo Kaggle, HugeCTR is able to fit and train
the model on a single GPU. The Criteo Kaggle results are thus normalized to
1-GPU HugeCTR. For a large model like Criteo Terabyte, HugeCTR is unable to
fit the model within 1 or 2 GPUs; throws an Out of Memory (OOM) error, and
requires a minimum of 4 GPUs. Hence, Criteo Terabyte results are normalized to
4-GPUs.

Figure 17: The speedup of Hotline normalized to HugeCTR. Hotline provides a
speedup of up to 1.8$\times$.

As shown in Figure 17, Hotline removes all-to-all communication associated
with model parallel embedding layer of HugeCTR and outperforms in all the
cases. Moreover, for popular minibatches Hotline completely execute embeddings
and neural layers in data parallel fashion, which mitigates any CPU-GPU
communication overheads. For non-popular mini-batches it incurs an overhead of
obtaining embeddings from CPU memory. However, this latency is hidden behind
the execution of popular mini-batches.

####  6.4.2 Comparison over CPU-based Hotline:

CPU can be used as an alternative to the Hotline hardware accelerator for
reordering mini-batches, gathering working set parameters, and as a pipeline
scheduler. To enable this, we map the Hotline computation (forward, backward
and optimizer) to high throughput GPUs and try to provide working parameters
to these GPUs at a rate faster than GPU execution – to avoid stalls. We
observe that mini-batch re-ordering and working parameter gathering are memory
intensive operations. This results in data stalls within the CPU it acts as
bottleneck.

Figure 18: Hotline speedup normalized to CPU-based Hotline implementation. The
Hotline hardware accelerator provides up to 3.5$\times$ speedup as compared to
a CPU-based implementation.

Figure 18 shows the performance benefits of Hotline accelerator over a CPU
that implements the features of Hotline using multi-processing. As the mini-
batch and number of GPUs scale, performance of CPU-based Hotline degrades
because of increased data stalls.

###  6.5 Throughput Improvements

Hotline improves the end-to-end training throughput by processing a pipeline
of mini-batches. As Figure 19 shows, for a 4-GPU system Hotline can process,
on average, 4$\times$ more epochs/hour as compared to optimized DLRM baseline.
Hotline throughput improves at a higher rate with increasing mini-batch than
the baseline, because with a larger mini-batch, Hotline can create bigger
popular mini-batches.

Figure 19: Varying throughput with 4-GPU executions.

###  6.6 CPU to GPU embedding transfer latency

Table 6 shows the absolute time spent by baseline and Hotline to perform CPU
to GPU embedding transfers. Hotline reduces the transfer time, on average
(4-GPU), by 2.7$\times$, as popular inputs do not need embeddings from CPU
memory.

Table 6: CPU-GPU data transfer time - 10 Epochs (mins)

Dataset | 1-GPU | 2-GPU | 4-GPU
---|---|---|---
DLRM | Hotline | DLRM | Hotline | DLRM | Hotline
Criteo Kaggle | 17.04 | 3.16 | 17.91 | 3.30 | 16.89 | 5.66
Taobao Alibaba | 15.65 | 4.81 | 15.21 | 3.30 | 11.71 | 3.10
Criteo Terabyte | 62.47 | 12.44 | 54.27 | 10.77 | 47.27 | 9.21
Avazu | 11.96 | 1.84 | 14.71 | 2.13 | 11.51 | 2.79

###  6.7 Performance improvement for synthetic models

To understand the efficacy of Hotline as the size recommendation models
increase, we generated synthetic models and datasets. Figure 20 shows the
performance of Hotline across two synthetic models. Hotline continues to
provide up to 2.5$\times$ speedup even as model sizes scale.

Figure 20: Performance of Hotline versus Intel-Optimized DLRM across synthetic
models for a 1-GPU system.

###  6.8 Design Space Exploration

Varying the number of inputs in a mini-batch: Figure 21 shows the performance
benefits of Hotline are accentuated with larger mini-batch sizes. This is due
to two reasons, scheduler has to issue input dispatch commands fewer times.
Secondly, larger mini-batches offer more parallelism opportunities, and
Hotline enables more compute on GPU(s).

Figure 21: Hotline speedup with varying mini-batch sizes.

Mini-batch working set: Figure 22 compares the impact of the mini-batch
working set on the execution time. For a small working set, e.g. 1, even if
there is a single input which accesses not-frequently-accessed embeddings, the
accelerator will have to gather the working parameters for the whole mini-
batch. This data transfer latency cannot be hidden behind the popular input
execution and causes a bottleneck. We observe that a 4 mini-batch working set
is sufficient to completely fill the pipeline and extract maximum throughput.

Figure 22: Effect of varying working set on unit iteration time.

Embedding Access logger size: Figure 23 shows the percentage of inputs
accessing frequently-accessed embeddings captured by varying the logger size.
For Criteo and Avazu datasets with more skewness, even a small 2MB logger can
capture most of the frequently-accessed indices. However for Taobao dataset,
that has a lower skew, once the logger size is increased beyond 4 MB we
observe diminishing returns.

Figure 23: Percentage of inputs accessing frequently-accessed embeddings captured with varying logger size. Table 7: Area and Power Breakdown Component | Area | Power
---|---|---
EAL | 65.72% | 67.31%
Input eDRAM | 23.44% | 26.75%
Lookup Engine | 5.80% | 2.97%
Emb Vec Buffer | 0.66% | 0.99%
Others | 4.38% | 1.98%

Hotline Accelerator Area and Power Breakdown. Table 7 shows the area and power
division across different components of Hotline accelerator. Embedding access
logger consumes the most area and power as it is a 4-way set associative
structure with replacement policy logic. The next significant area consuming
component is Input eDRAM as this memory structure is used to store non-popular
inputs. Finally, the lookup engine consumes the least percentage of area and
power as it only performs lookup logic.

##  7 Related Work

Industry-scale recommender datasets show that accesses depict a Power or
Zipfian distribution [56, 7].

Recommendation models and designs: In this work we focus on deep learning
based recommender models as they are commonly used in production and consume
large number of compute cycles in data centres [39, 19]. Even with their
popularity, much of the research has focused on optimizing the inference phase
of these models [22, 20, 42, 33, 19]. Unlike inference, training is throughput
optimized and tends to be more compute and memory intensive. Certain solutions
that offer training optimizations [18, 50] or acceleration [36, 31, 45] do not
maximize throughput by facilitating better memory and bandwidth utilization of
a distributed GPU system.

Embedding parameter placement: Work in [57, 6] offers different embedding
table placements either through hierarchical parameter server or on different
devices, however, treat all embedding entries equally, without the concept of
frequently-accessed embeddings and do not deal with perceptive input pre-
processing to reduce the overhead of communication between devices. Work in
[16] propose techniques to store embedding tables in non-volatile memories and
allocate a certain portion of DRAM for caching. Recent work in [18, 31, 50,
37] has also proposed solutions to accelerate near-memory processing for
embedding tables. All of these works do not facilitate distributed training of
entire recommender models using GPUs. Hotline, instead offers an entirely
runtime solution to efficiently pipeline embedding gather and store operations
from external memory.

Optimizations to mitigate memory intensive training: Prior work [43],
optimizes training by modifying the model either through mixed-precision [55]
training or eliminating rare categorical variables to reduce the embedding
table size. Such approaches that change the data representation and/or
embedding tables, require accuracy re-validation across a variety of models
and datasets. More work has been done on compression [51, 24, 54] and sparsity
[17] to reduce the memory footprint of machine learning models in general.
Hotline, on the other hand, performs full-precision training without employing
overheads such as compression/decompression and sparse operations by
leveraging the skewed access patterns for embeddings. Hotline is orthogonal to
these techniques and can be used with them to improve the memory efficiency.

Machine learning accelerators: There has been a plethora of work on designing
accelerators to execute compute portion of deep learning models [28, 36, 45,
32, 47, 11, 13, 34, 49]. Certain works have also offered accelerators for
recommender models, especially collaborative filtering based models [36, 45].
Hotline does not aim to design a specialized architecture to optimize the
compute of deep learning based recommender models. Hotline facilitates
training by efficiently dispatching and gathering working parameters and input
mini-batches to an already established GPU based system. Nevertheless, Hotline
accelerator can be used to enhance the overall benefits of deep learning
accelerators by pipelining their execution efficiently.

##  8 Conclusions

Production-scale recommendation models deploy DNNs along with capacity and
bandwidth-constrained embedding tables. In this paper, we present Hotline, a
heterogeneous acceleration pipeline based on the insight that small portion of
embedding tables are frequently accessed. Hotline aims to dispatch popular
inputs that only access these frequently-accessed embeddings directly to GPU,
while obtaining the not-frequently-accessed embedding data for non-popular
inputs. To perform the dynamic segregation and dispatching, we offer a novel
accelerator that can perform the fine grained tasks of determining which
inputs require what embedding entries, and if needed gather them from main
memory and dispatch the inputs. As such, Hotline hides the data transfer
latency of obtaining non-frequently-accessed embedding data behind GPU
execution of popular inputs. Results on real-world datasets and recommender
models show that Hotline reduces the average training time by 3$\times$ and
1.8$\times$ in comparison to Intel optimized CPU-GPU DLRM and HugeCTR
optimized GPU-only baseline, respectively.

## Acknowledgements

This project is part of the STAR Lab at the University of British Columbia
(UBC). We express our thanks to the entire team of the Advanced Research
Computing center at UBC [4]. This work was partially supported by the Natural
Sciences and Engineering Research Council of Canada (NSERC) [funding reference
number RGPIN-2019-05059]. The views and conclusions contained herein are those
of the authors and should not be interpreted as necessarily representing the
official policies or endorsements, either expressed or implied, of NSERC, the
Canadian Government, Microsoft, or UBC.

## References

  * [1] MLPerf Benchmarks.  https://mlcommons.org/en/training-normal-10/.
  * [2] NVIDIA Merlin: HugeCTR.  https://github.com/NVIDIA-Merlin/HugeCTR.
  * [3] NVlink.
  * [4] UBC Advanced Research Computing, "UBC ARC Sockeye." UBC Advanced Research Computing, 2019, doi: 10.14288/SOCKEYE.
  * [5] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng.  TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems, 2015.
  * [6] Bilge Acun, Matthew Murphy, Xiaodong Wang, Jade Nie, Carole-Jean Wu, and Kim Hazelwood.  Understanding Training Efficiency of Deep Learning Recommendation Models at Scale, 2020.
  * [7] Muhammad Adnan, Yassaman Ebrahimzadeh Maboud, Divya Mahajan, and Prashant Nair.  Accelerating Recommendation System Trainingby Leveraging Popular Choices.  In VLDB, 2022.
  * [8] Saurabh Agarwal, Ziyi Zhang, and Shivaram Venkataraman.  Bagpipe: Accelerating deep recommendation model training, 2022.
  * [9] Alibaba.  User Behavior Data from Taobao for Recommendation.  https://tianchi.aliyun.com/dataset/dataDetail?dataId=649userId=1.
  * [10] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners, 2020.
  * [11] Yu-Hsin Chen, Joel Emer, and Vivienne Sze.  Eyeriss: A Spatial Architecture for Energy-Efficient Dataflow for Convolutional Neural Networks.  In ISCA, 2016.
  * [12] K. Cho, M. Lee, K. Park, T. T. Kwon, Y. Choi, and Sangheon Pack.  WAVE: Popularity-based and collaborative in-network caching for content-oriented networks.  In 2012 Proceedings IEEE INFOCOM Workshops, pages 316–321, 2012\.
  * [13] Eric Chung, Jeremy Fowers, Kalin Ovtcharov, , Adrian Caulfield, Todd Massengill, Ming Liu, Mahdi Ghandi, Daniel Lo, Steve Reinhardt, Shlomi Alkalay, Hari Angepat, Derek Chiou, Alessandro Forin, Doug Burger, Lisa Woods, Gabriel Weisz, Michael Haselman, and Dan Zhang.  Serving DNNs in Real Time at Datacenter Scale with Project Brainwave.  IEEE Micro, 38:8–20, March 2018.
  * [14] CriteoLabs.  Criteo Display Ad Challenge.  https://www.kaggle.com/c/criteo-display-ad-challenge.
  * [15] CriteoLabs.  Terabyte Click Logs.  https://labs.criteo.com/2013/12/download-terabyte-click-logs.
  * [16] Assaf Eisenman, Maxim Naumov, Darryl Gardner, Misha Smelyanskiy, Sergey Pupyrev, Kim Hazelwood, Asaf Cidon, and Sachin Katti.  Bandana: Using non-volatile memory for storing deep learning models.  Proceedings of Machine Learning and Systems, 1:40–52, 2019.
  * [17] Jeremy Fowers, Kalin Ovtcharov, Karin Strauss, Eric Chung, and Greg Stitt.  A High Memory Bandwidth FPGA Accelerator for Sparse Matrix-Vector Multiplication.  In International Symposium on Field-Programmable Custom Computing Machines. IEEE, May.
  * [18] A. Ginart, M. Naumov, D. Mudigere, Jiyan Yang, and J. Zou.  Mixed Dimension Embeddings with Application to Memory-Efficient Recommendation Systems.  ArXiv, abs/1909.11810, 2019.
  * [19] U. Gupta, C. Wu, X. Wang, M. Naumov, B. Reagen, D. Brooks, B. Cottel, K. Hazelwood, M. Hempstead, B. Jia, H. S. Lee, A. Malevich, D. Mudigere, M. Smelyanskiy, L. Xiong, and X. Zhang.  The Architectural Implications of Facebook’s DNN-Based Personalized Recommendation.  In 2020 IEEE International Symposium on High Performance Computer Architecture (HPCA), pages 488–501, 2020.
  * [20] Udit Gupta, Samuel Hsia, Vikram Saraph, Xiaodong Wang, Brandon Reagen, Gu-Yeon Wei, Hsien-Hsin S Lee, David Brooks, and Carole-Jean Wu.  DeepRecSys: A System for Optimizing End-To-End At-scale Neural Recommendation Inference.  arXiv preprint arXiv:2001.02772, 2020.
  * [21] Jui-Ting Huang, Ashish Sharma, Shuying Sun, Li Xia, David Zhang, Philip Pronin, Janani Padmanabhan, Giuseppe Ottaviano, and Linjun Yang.  Embedding-Based Retrieval in Facebook Search, page 2553–2561.  Association for Computing Machinery, New York, NY, USA, 2020.
  * [22] Ranggi Hwang, Taehun Kim, Youngeun Kwon, and Minsoo Rhu.  Centaur: A Chiplet-based, Hybrid Sparse-Dense Accelerator for Personalized Recommendations.  arXiv preprint arXiv:2005.05968, 2020.
  * [23] T. Ishkhanov, M. Naumov, X. Chen, Y. Zhu, Y. Zhong, A. G. Azzolini, C. Sun, F. Jiang, A. Malevich, and L. Xiong.  Time-based Sequence Model for Personalization and Recommendation Systems.  CoRR, abs/2008.11922, 2020.
  * [24] A. Jain, A. Phanishayee, J. Mars, L. Tang, and G. Pekhimenko.  Gist: Efficient Data Encoding for Deep Neural Network Training.  In 2018 ACM/IEEE 45th Annual International Symposium on Computer Architecture (ISCA), pages 776–789, 2018.
  * [25] Aamer Jaleel, Kevin B. Theobald, Simon C. Steely, and Joel Emer.  High Performance Cache Replacement Using Re-Reference Interval Prediction (RRIP).  In Proceedings of the 37th Annual International Symposium on Computer Architecture, ISCA ’10, page 60–71, New York, NY, USA, 2010. Association for Computing Machinery.
  * [26] Myeongjae Jeon, Shivaram Venkataraman, Amar Phanishayee, unjie Qian, Wencong Xiao, and Fan Yang.  Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads.  In Proceedings of the 2019 USENIX Conference on Usenix Annual Technical Conference, USENIX ATC ’19, page 947–960, USA, 2019. USENIX Association.
  * [27] Biye Jiang, Chao Deng, Huimin Yi, Zelin Hu, Guorui Zhou, Yang Zheng, Sui Huang, Xinyang Guo, Dongyue Wang, Yue Song, Liqin Zhao, Zhi Wang, Peng Sun, Yu Zhang, Di Zhang, Jinhui Li, Jian Xu, Xiaoqiang Zhu, and Kun Gai.  XDL: An Industrial Deep Learning Framework for High-Dimensional Sparse Data.  DLP-KDD ’19, New York, NY, USA, 2019. Association for Computing Machinery.
  * [28] Norman P. Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, Rick Boyle, Pierre-luc Cantin, Clifford Chao, Chris Clark, Jeremy Coriell, Mike Daley, Matt Dau, Jeffrey Dean, Ben Gelb, Tara Vazir Ghaemmaghami, Rajendra Gottipati, William Gulland, Robert Hagmann, C. Richard Ho, Doug Hogberg, John Hu, Robert Hundt, Dan Hurt, Julian Ibarz, Aaron Jaffey, Alek Jaworski, Alexander Kaplan, Harshit Khaitan, Daniel Killebrew, Andy Koch, Naveen Kumar, Steve Lacy, James Laudon, James Law, Diemthu Le, Chris Leary, Zhuyuan Liu, Kyle Lucke, Alan Lundin, Gordon MacKean, Adriana Maggiore, Maire Mahony, Kieran Miller, Rahul Nagarajan, Ravi Narayanaswami, Ray Ni, Kathy Nix, Thomas Norrie, Mark Omernick, Narayana Penukonda, Andy Phelps, Jonathan Ross, Matt Ross, Amir Salek, Emad Samadiani, Chris Severn, Gregory Sizikov, Matthew Snelham, Jed Souter, Dan Steinberg, Andy Swing, Mercedes Tan, Gregory Thorson, Bo Tian, Horia Toma, Erick Tuttle, Vijay Vasudevan, Richard Walter, Walter Wang, Eric Wilcox, and Doe Hyun Yoon.  In-Datacenter Performance Analysis of a Tensor Processing Unit.  In Proceedings of the 44th Annual International Symposium on Computer Architecture, ISCA ’17, page 1–12, New York, NY, USA, 2017. Association for Computing Machinery.
  * [29] Kaggle.  Avazu mobile ads CTR.  https://www.kaggle.com/c/avazu-ctr-prediction.
  * [30] Dhiraj Kalamkar, Evangelos Georganas, Sudarshan Srinivasan, Jianping Chen, Mikhail Shiryaev, and Alexander Heinecke.  Optimizing Deep Learning Recommender Systems Training on CPU Cluster Architectures.  In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC ’20. IEEE Press, 2020\.
  * [31] L. Ke, U. Gupta, B. Y. Cho, D. Brooks, V. Chandra, U. Diril, A. Firoozshahian, K. Hazelwood, B. Jia, H. S. Lee, M. Li, B. Maher, D. Mudigere, M. Naumov, M. Schatz, M. Smelyanskiy, X. Wang, B. Reagen, C. Wu, M. Hempstead, and X. Zhang.  RecNMP: Accelerating Personalized Recommendation with Near-Memory Processing.  In 2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA), pages 790–803, 2020.
  * [32] Hyoukjun Kwon, Ananda Samajdar, and Tushar Krishna.  MAERI: Enabling Flexible Dataflow Mapping over DNN Accelerators via Reconfigurable Interconnects.  In Proceedings of the Twenty-Third International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS ’18, page 461–475, New York, NY, USA, 2018. Association for Computing Machinery.
  * [33] Youngeun Kwon, Yunjae Lee, and Minsoo Rhu.  Tensordimm: A practical near-memory processing architecture for embeddings and tensor operations in deep learning.  In Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture, pages 740–753, 2019.
  * [34] Daofu Liu, Tianshi Chen, Shaoli Liu, Jinhong Zhou, Shengyuan Zhou, Olivier Teman, Xiaobing Feng, Xuehai Zhou, and Yunji Chen.  PuDianNao: A Polyvalent Machine Learning Accelerator.  In Proceedings of the Twentieth International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS ’15, pages 369–381, New York, NY, USA, 2015. ACM.
  * [35] Michael Luby and Charles Rackoff.  How to construct pseudorandom permutations from pseudorandom functions.  SIAM Journal on Computing, 17(2):373–386, 1988.
  * [36] Divya Mahajan, Jongse Park, Emmanuel Amaro, Hardik Sharma, Amir Yazdanbakhsh, Joon Kim, and Hadi Esmaeilzadeh.  Tabla: A unified template-based framework for accelerating statistical machine learning.  March 2016.
  * [37] Dheevatsa Mudigere, Yuchen Hao, Jianyu Huang, Zhihao Jia, Andrew Tulloch, Srinivas Sridharan, Xing Liu, Mustafa Ozdal, Jade Nie, Jongsoo Park, Liang Luo, Jie Amy Yang, Leon Gao, Dmytro Ivchenko, Aarti Basant, Yuxi Hu, Jiyan Yang, Ehsan K. Ardestani, Xiaodong Wang, Rakesh Komuravelli, Ching-Hsiang Chu, Serhat Yilmaz, Huayu Li, Jiyuan Qian, Zhuobo Feng, Yinbin Ma, Junjie Yang, Ellie Wen, Hong Li, Lin Yang, Chonglin Sun, Whitney Zhao, Dimitry Melts, Krishna Dhulipala, KR Kishore, Tyler Graf, Assaf Eisenman, Kiran Kumar Matam, Adi Gangidi, Guoqiang Jerry Chen, Manoj Krishnan, Avinash Nayak, Krishnakumar Nair, Bharath Muthiah, Mahmoud khorashadi, Pallab Bhattacharya, Petr Lapukhov, Maxim Naumov, Ajit Mathews, Lin Qiao, Mikhail Smelyanskiy, Bill Jia, and Vijay Rao.  Software-Hardware Co-design for Fast and Scalable Training of Deep Learning Recommendation Models, 2021.
  * [38] Naveen Muralimanohar, Rajeev Balasubramonian, and Norm Jouppi.  Optimizing NUCA Organizations and Wiring Alternatives for Large Caches with CACTI 6.0.  In MICRO, 2007.
  * [39] Maxim Naumov, John Kim, Dheevatsa Mudigere, Srinivas Sridharan, Xiaodong Wang, Whitney Zhao, Serhat Yilmaz, Changkyu Kim, Hector Yuen, Mustafa Ozdal, Krishnakumar Nair, Isabel Gao, Bor-Yiing Su, Jiyan Yang, and Mikhail Smelyanskiy.  Deep Learning Training in Facebook Data Centers: Design of Scale-up and Scale-out Systems.  arXiv e-prints, page arXiv:2003.09518, March 2020.
  * [40] Maxim Naumov, Dheevatsa Mudigere, Hao-Jun Michael Shi, Jianyu Huang, Narayanan Sundaraman, Jongsoo Park, Xiaodong Wang, Udit Gupta, Carole-Jean Wu, Alisson G. Azzolini, Dmytro Dzhulgakov, Andrey Mallevich, Ilia Cherniavskii, Yinghai Lu, Raghuraman Krishnamoorthi, Ansha Yu, Volodymyr Kondratenko, Stephanie Pereira, Xianjie Chen, Wenlin Chen, Vijay Rao, Bill Jia, Liang Xiong, and Misha Smelyanskiy.  Deep Learning Recommendation Model for Personalization and Recommendation Systems.  CoRR, abs/1906.00091, 2019.
  * [41] Nvidia.  NVIDIA Collective Communications Library (NCCL).  https://docs.nvidia.com/deeplearning/nccl/index.html.
  * [42] Nvidia.  Accelerating wide deep recommender inference on gpus, 2017.  https://developer.nvidia.com/blog/accelerating-wide-deep-recommender-inference-on-gpus/.
  * [43] Mengdi Huang Nvidia Inc. Vinh Nguyen, Tomasz Grel.  Optimizing the Deep Learning Recommendation Model on NVIDIA GPUs.  https://developer.nvidia.com/blog/optimizing-dlrm-on-nvidia-gpus.
  * [44] Fragkiskos Papadopoulos, Maksim Kitsak, M. A. Serrano, Marian Boguna, and Dmitri Krioukov.  Popularity versus similarity in growing networks.  Nature, 489(7417):537–40, Sep 27 2012.
  * [45] Jongse Park, Hardik Sharma, Divya Mahajan, Joon Kyung Kim, Preston Olds, and Hadi Esmaeilzadeh.  Scale-Out Acceleration for Machine Learnng.  October 2017.
  * [46] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer.  Automatic differentiation in PyTorch.  2017\.
  * [47] B. Reagen, P. Whatmough, R. Adolf, S. Rama, H. Lee, S. K. Lee, J. M. Hernández-Lobato, G. Wei, and D. Brooks.  Minerva: Enabling Low-Power, Highly-Accurate Deep Neural Network Accelerators.  In 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA), pages 267–278, June 2016.
  * [48] Geet Sethi, Bilge Acun, Niket Agarwal, Christos Kozyrakis, Caroline Trippel, and Carole-Jean Wu.  Recshard: Statistical feature-based memory optimization for industry-scale neural recommendation, 2022.
  * [49] Hardik Sharma, Jongse Park, Divya Mahajan, Emmanuel Amaro, Joon Kyung Kim, Chenkai Shao, Asit Mishra, and Hadi Esmaeilzadeh.  From High-Level Deep Neural Models to FPGAs.  October 2016.
  * [50] Hao-Jun Michael Shi, Dheevatsa Mudigere, Maxim Naumov, and Jiyan Yang.  Compositional Embeddings Using Complementary Partitions for Memory-Efficient Recommendation Systems, page 165–175.  Association for Computing Machinery, New York, NY, USA, 2020.
  * [51] Yang Sun, Fajie Yuan, Min Yang, Guoao Wei, Zhou Zhao, and Duo Liu.  A Generic Network Compression Framework for Sequential Recommender Systems.  In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’20, page 1299–1308, New York, NY, USA, 2020. Association for Computing Machinery.
  * [52] G. Wang, D. Anand, N. Butt, A. Cestero, M. Chudzik, J. Ervin, S. Fang, G. Freeman, H. Ho, B. Khan, B. Kim, W. Kong, R. Krishnan, S. Krishnan, O. Kwon, J. Liu, K. McStay, E. Nelson, K. Nummy, P. Parries, J. Sim, R. Takalkar, A. Tessier, R.M. Todi, R. Malik, S. Stiffler, and S.S. Iyer.  Scaling deep trench based eDRAM on SOI to 32nm and Beyond.  In 2009 IEEE International Electron Devices Meeting (IEDM), pages 1–4, 2009.
  * [53] Carole-Jean Wu, Robin Burke, Ed H. Chi, Joseph Konstan, Julian McAuley, Yves Raimond, and Hao Zhang.  Developing a Recommendation Benchmark for MLPerf Training and Inference, 2020.
  * [54] Xiaorui Wu, Hong Xu, Honglin Zhang, Huaming Chen, and Jian Wang.  Saec: similarity-aware embedding compression in recommendation systems.  In Proceedings of the 11th ACM SIGOPS Asia-Pacific Workshop on Systems, pages 82–89, 2020.
  * [55] Jie Amy Yang, Jianyu Huang, Jongsoo Park, Ping Tak Peter Tang, and Andrew Tulloch.  Mixed-Precision Embedding Using a Cache, 2020.
  * [56] Chunxing Yin, Bilge Acun, Xing Liu, and Carole-Jean Wu.  TT-Rec: Tensor Train Compression for Deep Learning Recommendation Models, 2021.
  * [57] Weijie Zhao, Deping Xie, Ronglai Jia, Yulei Qian, Ruiquan Ding, Mingming Sun, and Ping Li.  Distributed Hierarchical GPU Parameter Server for Massive Scale Deep Learning Ads Systems, 2020.